2024-01-11T07:53:45.617 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-11T07:53:45.618 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi --verbose --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512563 -- /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512563/orig.config.yaml 2024-01-11T07:53:45.665 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-11T07:53:45.750 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512563 branch: wip-vshankar-testing1-quincy-2024-01-10-2010-quincy description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} email: yweinste@redhat.com first_in_suite: false job_id: '7512563' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-vshankar-testing1-quincy-2024-01-10-2010-quincy ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 4bcedb9fc11bed1fd6fd0c4fd59187401f309680 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 4bcedb9fc11bed1fd6fd0c4fd59187401f309680 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing1-quincy-2024-01-10-2010-quincy sha1: 4bcedb9fc11bed1fd6fd0c4fd59187401f309680 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 419 sha1: 4bcedb9fc11bed1fd6fd0c4fd59187401f309680 sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-vshankar-testing1-quincy-2024-01-10-2010-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 4bcedb9fc11bed1fd6fd0c4fd59187401f309680 targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEunaG6cGUEWWP4tba6VzAu7nUPYDutCiMscx2EbA8khWok9fdqlJ8otWGFY17YR4vwhjOiDGke6+1VbYkIjus8= smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNi4VBy2oxJo5MOLjqpm4HK6B6up8DRtTrBVz5TABoJt5pvN7mongi/6XvIYziSsfdzl9BonN62qXSr5Gn3hSHk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-10_19:20:36 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-11T07:53:45.750 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa; will attempt to use it 2024-01-11T07:53:45.751 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa/tasks 2024-01-11T07:53:45.751 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-11T07:53:45.752 INFO:teuthology.task.internal:Checking packages... 2024-01-11T07:53:45.777 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '4bcedb9fc11bed1fd6fd0c4fd59187401f309680' 2024-01-11T07:53:45.777 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-11T07:53:45.777 INFO:teuthology.packaging:ref: None 2024-01-11T07:53:45.777 INFO:teuthology.packaging:tag: None 2024-01-11T07:53:45.777 INFO:teuthology.packaging:branch: wip-vshankar-testing1-quincy-2024-01-10-2010-quincy 2024-01-11T07:53:45.777 INFO:teuthology.packaging:sha1: 4bcedb9fc11bed1fd6fd0c4fd59187401f309680 2024-01-11T07:53:45.777 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-vshankar-testing1-quincy-2024-01-10-2010-quincy 2024-01-11T07:53:45.942 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-299.g4bcedb9f 2024-01-11T07:53:45.944 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-11T07:53:45.951 INFO:teuthology.task.internal:no buildpackages task found 2024-01-11T07:53:45.951 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-11T07:53:45.996 INFO:teuthology.task.internal:Saving configuration 2024-01-11T07:53:46.013 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-11T07:53:46.029 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-11T07:53:46.052 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512563', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-11 07:48:12.485847', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEunaG6cGUEWWP4tba6VzAu7nUPYDutCiMscx2EbA8khWok9fdqlJ8otWGFY17YR4vwhjOiDGke6+1VbYkIjus8='} 2024-01-11T07:53:46.072 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512563', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-11 07:48:12.487316', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNi4VBy2oxJo5MOLjqpm4HK6B6up8DRtTrBVz5TABoJt5pvN7mongi/6XvIYziSsfdzl9BonN62qXSr5Gn3hSHk='} 2024-01-11T07:53:46.073 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-11T07:53:46.079 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-01-11T07:53:46.079 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-01-11T07:53:46.079 INFO:teuthology.run_tasks:Running task console_log... 2024-01-11T07:53:46.171 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff7efcfc0d0>, signals=[15]) 2024-01-11T07:53:46.171 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-11T07:53:46.177 INFO:teuthology.task.internal:Opening connections... 2024-01-11T07:53:46.178 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T07:53:46.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T07:53:46.261 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2024-01-11T07:53:46.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T07:53:46.336 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-11T07:53:46.342 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2024-01-11T07:53:46.366 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2024-01-11T07:53:46.366 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2024-01-11T07:53:46.431 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream" 2024-01-11T07:53:46.432 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="8" 2024-01-11T07:53:46.432 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos" 2024-01-11T07:53:46.432 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora" 2024-01-11T07:53:46.432 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="8" 2024-01-11T07:53:46.432 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el8" 2024-01-11T07:53:46.432 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-11T07:53:46.432 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2024-01-11T07:53:46.432 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-11T07:53:46.432 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/" 2024-01-11T07:53:46.432 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-11T07:53:46.432 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-11T07:53:46.432 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-11T07:53:46.433 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2024-01-11T07:53:46.456 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2024-01-11T07:53:46.479 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2024-01-11T07:53:46.479 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2024-01-11T07:53:46.539 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="CentOS Stream" 2024-01-11T07:53:46.539 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="8" 2024-01-11T07:53:46.539 INFO:teuthology.orchestra.run.smithi077.stdout:ID="centos" 2024-01-11T07:53:46.539 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="rhel fedora" 2024-01-11T07:53:46.539 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="8" 2024-01-11T07:53:46.540 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el8" 2024-01-11T07:53:46.540 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-11T07:53:46.540 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2024-01-11T07:53:46.540 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-11T07:53:46.540 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://centos.org/" 2024-01-11T07:53:46.540 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-11T07:53:46.540 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-11T07:53:46.540 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-11T07:53:46.540 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2024-01-11T07:53:46.565 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-11T07:53:46.575 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-11T07:53:46.613 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-11T07:53:46.613 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2024-01-11T07:53:46.616 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2024-01-11T07:53:46.636 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-11T07:53:46.647 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-11T07:53:46.647 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2024-01-11T07:53:46.681 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2024-01-11T07:53:46.780 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-11T07:53:46.834 INFO:teuthology.run_tasks:Running task kernel... 2024-01-11T07:53:46.849 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-01-11T07:53:46.849 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-01-11T07:53:46.849 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-01-11T07:53:46.849 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-11T07:53:46.850 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-11T07:53:46.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T07:53:46.871 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2024-01-11T07:53:46.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T07:53:46.875 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2024-01-11T07:53:47.126 DEBUG:teuthology.task.kernel:sha1 for client.1 is c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b 2024-01-11T07:53:47.126 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b"... 2024-01-11T07:53:47.126 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-01-11T07:53:47.137 DEBUG:teuthology.task.kernel:sha1 for client.0 is c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b 2024-01-11T07:53:47.137 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b"... 2024-01-11T07:53:47.137 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-01-11T07:53:47.148 INFO:teuthology.orchestra.run.smithi077.stdout:4.18.0-489.el8.x86_64 2024-01-11T07:53:47.148 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-489.el8.x86_64 vs c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b 2024-01-11T07:53:47.149 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-01-11T07:53:47.149 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum upgrade -y linux-firmware 2024-01-11T07:53:47.159 INFO:teuthology.orchestra.run.smithi027.stdout:4.18.0-489.el8.x86_64 2024-01-11T07:53:47.159 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-489.el8.x86_64 vs c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b 2024-01-11T07:53:47.159 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-01-11T07:53:47.160 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum upgrade -y linux-firmware 2024-01-11T07:53:51.999 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - AppStream 17 MB/s | 35 MB 00:02 2024-01-11T07:53:52.642 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - AppStream 12 MB/s | 35 MB 00:02 2024-01-11T07:53:59.949 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - BaseOS 55 MB/s | 58 MB 00:01 2024-01-11T07:53:59.992 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - BaseOS 56 MB/s | 58 MB 00:01 2024-01-11T07:54:08.776 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - Extras 33 kB/s | 18 kB 00:00 2024-01-11T07:54:10.424 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - Extras 11 kB/s | 18 kB 00:01 2024-01-11T07:54:11.533 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - Extras common packages 7.6 kB/s | 6.9 kB 00:00 2024-01-11T07:54:11.870 INFO:teuthology.orchestra.run.smithi027.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-01-11T07:54:12.584 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - Extras common packages 1.9 kB/s | 6.9 kB 00:03 2024-01-11T07:54:13.036 INFO:teuthology.orchestra.run.smithi077.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2024-01-11T07:54:13.185 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-01-11T07:54:14.755 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2024-01-11T07:54:17.260 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 424 kB/s | 24 kB 00:00 2024-01-11T07:54:18.586 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 397 kB/s | 24 kB 00:00 2024-01-11T07:54:21.532 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-01-11T07:54:21.532 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-01-11T07:54:21.532 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-01-11T07:54:21.532 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-01-11T07:54:21.533 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2024-01-11T07:54:21.533 INFO:teuthology.orchestra.run.smithi027.stdout: linux-firmware noarch 20231121-120.git9552083a.el8 baseos 347 M 2024-01-11T07:54:21.533 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T07:54:21.533 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-01-11T07:54:21.533 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-01-11T07:54:21.533 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 1 Package 2024-01-11T07:54:21.533 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T07:54:21.533 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 347 M 2024-01-11T07:54:21.534 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-01-11T07:54:22.721 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-01-11T07:54:22.722 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-01-11T07:54:22.722 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repo Size 2024-01-11T07:54:22.722 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-01-11T07:54:22.722 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2024-01-11T07:54:22.722 INFO:teuthology.orchestra.run.smithi077.stdout: linux-firmware noarch 20231121-120.git9552083a.el8 baseos 347 M 2024-01-11T07:54:22.722 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T07:54:22.723 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-01-11T07:54:22.723 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-01-11T07:54:22.723 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 1 Package 2024-01-11T07:54:22.723 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T07:54:22.723 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 347 M 2024-01-11T07:54:22.723 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-01-11T07:54:26.012 INFO:teuthology.orchestra.run.smithi027.stdout:linux-firmware-20231121-120.git9552083a.el8.noa 81 MB/s | 347 MB 00:04 2024-01-11T07:54:26.012 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-01-11T07:54:26.012 INFO:teuthology.orchestra.run.smithi027.stdout:Total 77 MB/s | 347 MB 00:04 2024-01-11T07:54:26.788 INFO:teuthology.orchestra.run.smithi077.stdout:linux-firmware-20231121-120.git9552083a.el8.noa 86 MB/s | 347 MB 00:04 2024-01-11T07:54:26.788 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-01-11T07:54:26.789 INFO:teuthology.orchestra.run.smithi077.stdout:Total 85 MB/s | 347 MB 00:04 2024-01-11T07:54:28.170 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-01-11T07:54:28.230 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-01-11T07:54:28.230 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-01-11T07:54:28.840 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-01-11T07:54:28.895 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-01-11T07:54:28.895 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-01-11T07:54:30.461 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-01-11T07:54:30.498 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-01-11T07:54:31.054 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-01-11T07:54:31.090 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-01-11T07:54:53.339 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-01-11T07:54:53.837 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : linux-firmware-20231121-120.git9552083a.el8.noarch 1/2 2024-01-11T07:54:54.072 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-01-11T07:54:54.311 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-01-11T07:54:54.311 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : linux-firmware-20231121-120.git9552083a.el8.noarch 1/2 2024-01-11T07:54:54.317 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-01-11T07:54:54.706 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : linux-firmware-20231121-120.git9552083a.el8.noarch 1/2 2024-01-11T07:54:54.974 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-01-11T07:54:55.155 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-01-11T07:54:55.155 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T07:54:55.155 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2024-01-11T07:54:55.155 INFO:teuthology.orchestra.run.smithi077.stdout: linux-firmware-20231121-120.git9552083a.el8.noarch 2024-01-11T07:54:55.155 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T07:54:55.156 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-01-11T07:54:55.211 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-01-11T07:54:55.211 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : linux-firmware-20231121-120.git9552083a.el8.noarch 1/2 2024-01-11T07:54:55.338 INFO:teuthology.task.kernel:Downloading kernel c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b on client.1... 2024-01-11T07:54:55.338 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b 2024-01-11T07:54:55.561 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b/centos/8/flavors/default/x86_64/ 2024-01-11T07:54:55.562 DEBUG:teuthology.orchestra.run.smithi077:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_gc719ee96ba78-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b/centos/8/flavors/default/x86_64/ --input-file=- 2024-01-11T07:54:55.564 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi077.front.sepia.ceph.com to complete... 2024-01-11T07:54:56.089 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-01-11T07:54:56.089 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T07:54:56.089 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2024-01-11T07:54:56.089 INFO:teuthology.orchestra.run.smithi027.stdout: linux-firmware-20231121-120.git9552083a.el8.noarch 2024-01-11T07:54:56.089 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T07:54:56.089 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-01-11T07:54:56.268 INFO:teuthology.task.kernel:Downloading kernel c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b on client.0... 2024-01-11T07:54:56.268 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b 2024-01-11T07:54:56.486 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b/centos/8/flavors/default/x86_64/ 2024-01-11T07:54:56.487 DEBUG:teuthology.orchestra.run.smithi027:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_gc719ee96ba78-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b/centos/8/flavors/default/x86_64/ --input-file=- 2024-01-11T07:54:56.490 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi027.front.sepia.ceph.com to complete... 2024-01-11T07:55:14.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11 07:55:14 URL:https://3.chacra.ceph.com/r/kernel/testing/c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b/centos/8/flavors/default/x86_64/kernel-6.6.0_gc719ee96ba78-1.x86_64.rpm [383182948/383182948] -> "/tmp/kernel.x86_64.rpm" [1] 2024-01-11T07:55:14.392 INFO:teuthology.orchestra.run.smithi077.stderr:FINISHED --2024-01-11 07:55:14-- 2024-01-11T07:55:14.392 INFO:teuthology.orchestra.run.smithi077.stderr:Total wall clock time: 19s 2024-01-11T07:55:14.392 INFO:teuthology.orchestra.run.smithi077.stderr:Downloaded: 1 files, 365M in 18s (20.1 MB/s) 2024-01-11T07:55:14.394 INFO:teuthology.task.kernel:Installing kernel c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b on client.1... 2024-01-11T07:55:14.395 DEBUG:teuthology.orchestra.run.smithi077:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-01-11T07:55:16.078 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying... ######################################## 2024-01-11T07:55:16.121 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing... ######################################## 2024-01-11T07:55:16.230 INFO:teuthology.orchestra.run.smithi077.stdout:Updating / installing... 2024-01-11T07:55:24.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11 07:55:24 URL:https://3.chacra.ceph.com/r/kernel/testing/c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b/centos/8/flavors/default/x86_64/kernel-6.6.0_gc719ee96ba78-1.x86_64.rpm [383182948/383182948] -> "/tmp/kernel.x86_64.rpm" [1] 2024-01-11T07:55:24.602 INFO:teuthology.orchestra.run.smithi027.stderr:FINISHED --2024-01-11 07:55:24-- 2024-01-11T07:55:24.602 INFO:teuthology.orchestra.run.smithi027.stderr:Total wall clock time: 28s 2024-01-11T07:55:24.602 INFO:teuthology.orchestra.run.smithi027.stderr:Downloaded: 1 files, 365M in 27s (13.3 MB/s) 2024-01-11T07:55:24.605 INFO:teuthology.task.kernel:Installing kernel c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b on client.0... 2024-01-11T07:55:24.605 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-01-11T07:55:26.374 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying... ######################################## 2024-01-11T07:55:26.419 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing... ######################################## 2024-01-11T07:55:26.535 INFO:teuthology.orchestra.run.smithi027.stdout:Updating / installing... 2024-01-11T07:55:54.403 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-6.6.0_gc719ee96ba78-1 ######################################## 2024-01-11T07:55:59.129 INFO:teuthology.orchestra.run.smithi077.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-11T07:56:07.412 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-6.6.0_gc719ee96ba78-1 ######################################## 2024-01-11T07:56:08.838 INFO:teuthology.orchestra.run.smithi027.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-11T07:56:18.312 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi077.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-01-11T07:56:18.313 DEBUG:teuthology.orchestra.run.smithi077:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-01-11T07:56:18.378 INFO:teuthology.orchestra.run.smithi077.stdout:/boot/System.map-6.6.0-gc719ee96ba78 2024-01-11T07:56:18.378 INFO:teuthology.orchestra.run.smithi077.stdout:/boot/config-6.6.0-gc719ee96ba78 2024-01-11T07:56:18.378 INFO:teuthology.orchestra.run.smithi077.stdout:/boot/vmlinuz-6.6.0-gc719ee96ba78 2024-01-11T07:56:18.378 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78 2024-01-11T07:56:18.378 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel 2024-01-11T07:56:18.378 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch 2024-01-11T07:56:18.378 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86 2024-01-11T07:56:18.378 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto 2024-01-11T07:56:18.379 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-01-11T07:56:18.379 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-01-11T07:56:18.379 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-01-11T07:56:18.379 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-01-11T07:56:18.379 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-01-11T07:56:18.379 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-01-11T07:56:18.379 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-01-11T07:56:18.379 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/crc32c-intel.ko 2024-01-11T07:56:18.379 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-01-11T07:56:18.379 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-01-11T07:56:18.379 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-01-11T07:56:18.380 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-01-11T07:56:18.380 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/serpent-avx2.ko 2024-01-11T07:56:18.380 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-01-11T07:56:18.380 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-01-11T07:56:18.380 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-01-11T07:56:18.380 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-01-11T07:56:18.380 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-01-11T07:56:18.380 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-01-11T07:56:18.380 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-01-11T07:56:18.380 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/kernel 2024-01-11T07:56:18.380 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/kernel/cpu 2024-01-11T07:56:18.380 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/kernel/cpu/mce 2024-01-11T07:56:18.381 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-01-11T07:56:18.381 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/kvm 2024-01-11T07:56:18.381 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/kvm/kvm-amd.ko 2024-01-11T07:56:18.381 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/kvm/kvm-intel.ko 2024-01-11T07:56:18.381 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/kvm/kvm.ko 2024-01-11T07:56:18.381 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto 2024-01-11T07:56:18.381 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/ansi_cprng.ko 2024-01-11T07:56:18.381 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/anubis.ko 2024-01-11T07:56:18.381 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/arc4.ko 2024-01-11T07:56:18.381 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/async_tx 2024-01-11T07:56:18.381 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/async_tx/async_memcpy.ko 2024-01-11T07:56:18.381 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/async_tx/async_pq.ko 2024-01-11T07:56:18.382 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/async_tx/async_raid6_recov.ko 2024-01-11T07:56:18.382 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/async_tx/async_tx.ko 2024-01-11T07:56:18.382 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/async_tx/async_xor.ko 2024-01-11T07:56:18.382 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/async_tx/raid6test.ko 2024-01-11T07:56:18.382 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/authenc.ko 2024-01-11T07:56:18.382 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/authencesn.ko 2024-01-11T07:56:18.382 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/blake2b_generic.ko 2024-01-11T07:56:18.382 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/blowfish_common.ko 2024-01-11T07:56:18.382 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/blowfish_generic.ko 2024-01-11T07:56:18.382 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/camellia_generic.ko 2024-01-11T07:56:18.382 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/cast5_generic.ko 2024-01-11T07:56:18.382 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/cast6_generic.ko 2024-01-11T07:56:18.383 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/cast_common.ko 2024-01-11T07:56:18.383 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/ccm.ko 2024-01-11T07:56:18.383 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/cmac.ko 2024-01-11T07:56:18.383 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/crc32_generic.ko 2024-01-11T07:56:18.383 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/crypto_engine.ko 2024-01-11T07:56:18.383 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/crypto_user.ko 2024-01-11T07:56:18.383 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/cts.ko 2024-01-11T07:56:18.383 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/des_generic.ko 2024-01-11T07:56:18.383 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/dh_generic.ko 2024-01-11T07:56:18.383 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/echainiv.ko 2024-01-11T07:56:18.383 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/essiv.ko 2024-01-11T07:56:18.383 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/fcrypt.ko 2024-01-11T07:56:18.384 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/khazad.ko 2024-01-11T07:56:18.384 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/lz4.ko 2024-01-11T07:56:18.384 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/lz4hc.ko 2024-01-11T07:56:18.384 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/md4.ko 2024-01-11T07:56:18.384 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/michael_mic.ko 2024-01-11T07:56:18.384 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/pcbc.ko 2024-01-11T07:56:18.384 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/pcrypt.ko 2024-01-11T07:56:18.384 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/rmd160.ko 2024-01-11T07:56:18.384 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/seed.ko 2024-01-11T07:56:18.384 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/serpent_generic.ko 2024-01-11T07:56:18.384 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/tcrypt.ko 2024-01-11T07:56:18.384 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/tea.ko 2024-01-11T07:56:18.385 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/twofish_common.ko 2024-01-11T07:56:18.385 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/twofish_generic.ko 2024-01-11T07:56:18.385 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/vmac.ko 2024-01-11T07:56:18.385 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/wp512.ko 2024-01-11T07:56:18.385 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/xcbc.ko 2024-01-11T07:56:18.385 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/xor.ko 2024-01-11T07:56:18.385 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/xxhash_generic.ko 2024-01-11T07:56:18.385 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers 2024-01-11T07:56:18.385 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/acpi 2024-01-11T07:56:18.385 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/acpi/acpi_ipmi.ko 2024-01-11T07:56:18.385 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/acpi/acpi_pad.ko 2024-01-11T07:56:18.385 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/acpi/custom_method.ko 2024-01-11T07:56:18.386 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/acpi/ec_sys.ko 2024-01-11T07:56:18.386 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/acpi/platform_profile.ko 2024-01-11T07:56:18.386 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/acpi/sbs.ko 2024-01-11T07:56:18.386 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/acpi/sbshc.ko 2024-01-11T07:56:18.386 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/acpi/video.ko 2024-01-11T07:56:18.386 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata 2024-01-11T07:56:18.386 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/acard-ahci.ko 2024-01-11T07:56:18.386 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/ahci_platform.ko 2024-01-11T07:56:18.386 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/ata_generic.ko 2024-01-11T07:56:18.386 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/libahci_platform.ko 2024-01-11T07:56:18.386 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_acpi.ko 2024-01-11T07:56:18.386 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_ali.ko 2024-01-11T07:56:18.387 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_amd.ko 2024-01-11T07:56:18.387 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_artop.ko 2024-01-11T07:56:18.387 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_atiixp.ko 2024-01-11T07:56:18.387 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_atp867x.ko 2024-01-11T07:56:18.387 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_cmd640.ko 2024-01-11T07:56:18.387 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_cmd64x.ko 2024-01-11T07:56:18.387 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_cypress.ko 2024-01-11T07:56:18.387 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_efar.ko 2024-01-11T07:56:18.387 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_hpt366.ko 2024-01-11T07:56:18.387 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_hpt37x.ko 2024-01-11T07:56:18.387 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_hpt3x2n.ko 2024-01-11T07:56:18.387 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_hpt3x3.ko 2024-01-11T07:56:18.388 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_it8213.ko 2024-01-11T07:56:18.388 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_it821x.ko 2024-01-11T07:56:18.388 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_jmicron.ko 2024-01-11T07:56:18.388 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_marvell.ko 2024-01-11T07:56:18.388 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_mpiix.ko 2024-01-11T07:56:18.388 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_netcell.ko 2024-01-11T07:56:18.388 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_ninja32.ko 2024-01-11T07:56:18.388 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_ns87410.ko 2024-01-11T07:56:18.388 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_ns87415.ko 2024-01-11T07:56:18.388 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_oldpiix.ko 2024-01-11T07:56:18.388 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_opti.ko 2024-01-11T07:56:18.388 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_optidma.ko 2024-01-11T07:56:18.389 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_pcmcia.ko 2024-01-11T07:56:18.389 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_pdc2027x.ko 2024-01-11T07:56:18.389 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-01-11T07:56:18.389 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_piccolo.ko 2024-01-11T07:56:18.389 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_rdc.ko 2024-01-11T07:56:18.389 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_sch.ko 2024-01-11T07:56:18.389 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_serverworks.ko 2024-01-11T07:56:18.389 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_sil680.ko 2024-01-11T07:56:18.389 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_sis.ko 2024-01-11T07:56:18.389 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_sl82c105.ko 2024-01-11T07:56:18.389 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_triflex.ko 2024-01-11T07:56:18.389 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_via.ko 2024-01-11T07:56:18.390 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pdc_adma.ko 2024-01-11T07:56:18.390 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_inic162x.ko 2024-01-11T07:56:18.390 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_mv.ko 2024-01-11T07:56:18.390 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_nv.ko 2024-01-11T07:56:18.390 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_promise.ko 2024-01-11T07:56:18.390 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_qstor.ko 2024-01-11T07:56:18.390 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_sil.ko 2024-01-11T07:56:18.390 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_sil24.ko 2024-01-11T07:56:18.390 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_sis.ko 2024-01-11T07:56:18.390 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_svw.ko 2024-01-11T07:56:18.390 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_sx4.ko 2024-01-11T07:56:18.390 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_uli.ko 2024-01-11T07:56:18.390 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_via.ko 2024-01-11T07:56:18.391 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_vsc.ko 2024-01-11T07:56:18.391 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/auxdisplay 2024-01-11T07:56:18.391 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/auxdisplay/ks0108.ko 2024-01-11T07:56:18.391 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/base 2024-01-11T07:56:18.391 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/base/regmap 2024-01-11T07:56:18.391 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/base/regmap/regmap-i2c.ko 2024-01-11T07:56:18.391 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/base/regmap/regmap-mmio.ko 2024-01-11T07:56:18.391 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/bcma 2024-01-11T07:56:18.391 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/bcma/bcma.ko 2024-01-11T07:56:18.391 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block 2024-01-11T07:56:18.391 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/aoe 2024-01-11T07:56:18.392 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/aoe/aoe.ko 2024-01-11T07:56:18.392 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/brd.ko 2024-01-11T07:56:18.392 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/drbd 2024-01-11T07:56:18.392 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/drbd/drbd.ko 2024-01-11T07:56:18.392 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/floppy.ko 2024-01-11T07:56:18.392 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/loop.ko 2024-01-11T07:56:18.392 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/mtip32xx 2024-01-11T07:56:18.392 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-01-11T07:56:18.392 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/nbd.ko 2024-01-11T07:56:18.392 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/null_blk 2024-01-11T07:56:18.392 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/null_blk/null_blk.ko 2024-01-11T07:56:18.392 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/pktcdvd.ko 2024-01-11T07:56:18.393 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/rbd.ko 2024-01-11T07:56:18.393 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/virtio_blk.ko 2024-01-11T07:56:18.393 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/xen-blkback 2024-01-11T07:56:18.393 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-01-11T07:56:18.393 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/xen-blkfront.ko 2024-01-11T07:56:18.393 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/zram 2024-01-11T07:56:18.393 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/zram/zram.ko 2024-01-11T07:56:18.393 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char 2024-01-11T07:56:18.393 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/hangcheck-timer.ko 2024-01-11T07:56:18.393 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/hw_random 2024-01-11T07:56:18.393 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/hw_random/amd-rng.ko 2024-01-11T07:56:18.394 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/hw_random/intel-rng.ko 2024-01-11T07:56:18.394 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-01-11T07:56:18.394 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/hw_random/via-rng.ko 2024-01-11T07:56:18.394 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/hw_random/virtio-rng.ko 2024-01-11T07:56:18.394 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/ipmi 2024-01-11T07:56:18.394 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-01-11T07:56:18.394 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-01-11T07:56:18.394 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-01-11T07:56:18.394 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/ipmi/ipmi_si.ko 2024-01-11T07:56:18.394 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-01-11T07:56:18.394 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-01-11T07:56:18.394 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/lp.ko 2024-01-11T07:56:18.395 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/mwave 2024-01-11T07:56:18.395 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/mwave/mwave.ko 2024-01-11T07:56:18.395 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/ppdev.ko 2024-01-11T07:56:18.395 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/tlclk.ko 2024-01-11T07:56:18.395 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/tpm 2024-01-11T07:56:18.395 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/tpm/tpm.ko 2024-01-11T07:56:18.395 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/tpm/tpm_atmel.ko 2024-01-11T07:56:18.395 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/tpm/tpm_infineon.ko 2024-01-11T07:56:18.395 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/tpm/tpm_nsc.ko 2024-01-11T07:56:18.395 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/tpm/tpm_tis.ko 2024-01-11T07:56:18.395 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-01-11T07:56:18.395 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/uv_mmtimer.ko 2024-01-11T07:56:18.395 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/virtio_console.ko 2024-01-11T07:56:18.396 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/cpufreq 2024-01-11T07:56:18.396 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-01-11T07:56:18.396 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-01-11T07:56:18.396 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/cpufreq/p4-clockmod.ko 2024-01-11T07:56:18.396 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-01-11T07:56:18.396 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/cpufreq/powernow-k8.ko 2024-01-11T07:56:18.396 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/cpufreq/speedstep-lib.ko 2024-01-11T07:56:18.396 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto 2024-01-11T07:56:18.396 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/ccp 2024-01-11T07:56:18.396 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-01-11T07:56:18.396 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/ccp/ccp.ko 2024-01-11T07:56:18.397 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/intel 2024-01-11T07:56:18.397 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/intel/qat 2024-01-11T07:56:18.397 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/intel/qat/qat_common 2024-01-11T07:56:18.397 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-01-11T07:56:18.397 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-01-11T07:56:18.397 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-01-11T07:56:18.397 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/padlock-aes.ko 2024-01-11T07:56:18.397 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/padlock-sha.ko 2024-01-11T07:56:18.397 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/virtio 2024-01-11T07:56:18.397 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-01-11T07:56:18.397 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/dca 2024-01-11T07:56:18.397 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/dca/dca.ko 2024-01-11T07:56:18.397 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/devfreq 2024-01-11T07:56:18.398 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-01-11T07:56:18.398 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/dma 2024-01-11T07:56:18.398 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/dma/dw 2024-01-11T07:56:18.398 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/dma/dw/dw_dmac.ko 2024-01-11T07:56:18.398 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/dma/ioat 2024-01-11T07:56:18.398 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/dma/ioat/ioatdma.ko 2024-01-11T07:56:18.398 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac 2024-01-11T07:56:18.398 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/amd64_edac.ko 2024-01-11T07:56:18.398 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/e752x_edac.ko 2024-01-11T07:56:18.398 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/edac_mce_amd.ko 2024-01-11T07:56:18.398 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/i3000_edac.ko 2024-01-11T07:56:18.398 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/i3200_edac.ko 2024-01-11T07:56:18.398 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/i5100_edac.ko 2024-01-11T07:56:18.399 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/i5400_edac.ko 2024-01-11T07:56:18.399 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/i7300_edac.ko 2024-01-11T07:56:18.399 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/i7core_edac.ko 2024-01-11T07:56:18.399 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/i82975x_edac.ko 2024-01-11T07:56:18.399 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/ie31200_edac.ko 2024-01-11T07:56:18.399 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/sb_edac.ko 2024-01-11T07:56:18.399 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/x38_edac.ko 2024-01-11T07:56:18.399 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/firmware 2024-01-11T07:56:18.399 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/firmware/edd.ko 2024-01-11T07:56:18.399 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/firmware/iscsi_ibft.ko 2024-01-11T07:56:18.399 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/gpio 2024-01-11T07:56:18.400 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/gpio/gpio-ich.ko 2024-01-11T07:56:18.400 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/gpio/gpio-viperboard.ko 2024-01-11T07:56:18.400 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid 2024-01-11T07:56:18.400 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-appleir.ko 2024-01-11T07:56:18.400 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-aureal.ko 2024-01-11T07:56:18.400 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-axff.ko 2024-01-11T07:56:18.400 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-dr.ko 2024-01-11T07:56:18.400 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-elecom.ko 2024-01-11T07:56:18.400 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-elo.ko 2024-01-11T07:56:18.400 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-emsff.ko 2024-01-11T07:56:18.400 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-gaff.ko 2024-01-11T07:56:18.400 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-gt683r.ko 2024-01-11T07:56:18.401 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-gyration.ko 2024-01-11T07:56:18.401 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-holtek-kbd.ko 2024-01-11T07:56:18.401 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-holtek-mouse.ko 2024-01-11T07:56:18.401 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-holtekff.ko 2024-01-11T07:56:18.401 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-hyperv.ko 2024-01-11T07:56:18.401 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-icade.ko 2024-01-11T07:56:18.401 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-keytouch.ko 2024-01-11T07:56:18.401 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-kye.ko 2024-01-11T07:56:18.401 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-lcpower.ko 2024-01-11T07:56:18.401 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-led.ko 2024-01-11T07:56:18.401 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-lenovo.ko 2024-01-11T07:56:18.401 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-logitech-dj.ko 2024-01-11T07:56:18.401 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-01-11T07:56:18.402 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-multitouch.ko 2024-01-11T07:56:18.402 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-ortek.ko 2024-01-11T07:56:18.402 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-penmount.ko 2024-01-11T07:56:18.402 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-petalynx.ko 2024-01-11T07:56:18.402 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-picolcd.ko 2024-01-11T07:56:18.402 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-pl.ko 2024-01-11T07:56:18.402 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-plantronics.ko 2024-01-11T07:56:18.402 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-primax.ko 2024-01-11T07:56:18.402 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-rmi.ko 2024-01-11T07:56:18.402 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-arvo.ko 2024-01-11T07:56:18.402 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-common.ko 2024-01-11T07:56:18.403 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-isku.ko 2024-01-11T07:56:18.403 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-kone.ko 2024-01-11T07:56:18.403 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-01-11T07:56:18.403 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-konepure.ko 2024-01-11T07:56:18.403 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-01-11T07:56:18.403 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-lua.ko 2024-01-11T07:56:18.403 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-pyra.ko 2024-01-11T07:56:18.403 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-ryos.ko 2024-01-11T07:56:18.403 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-savu.ko 2024-01-11T07:56:18.403 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat.ko 2024-01-11T07:56:18.403 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-saitek.ko 2024-01-11T07:56:18.403 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-samsung.ko 2024-01-11T07:56:18.403 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-sensor-hub.ko 2024-01-11T07:56:18.403 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-sjoy.ko 2024-01-11T07:56:18.404 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-sony.ko 2024-01-11T07:56:18.404 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-speedlink.ko 2024-01-11T07:56:18.404 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-steelseries.ko 2024-01-11T07:56:18.404 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-sunplus.ko 2024-01-11T07:56:18.404 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-thrustmaster.ko 2024-01-11T07:56:18.404 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-tivo.ko 2024-01-11T07:56:18.404 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-tmff.ko 2024-01-11T07:56:18.404 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-topseed.ko 2024-01-11T07:56:18.404 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-twinhan.ko 2024-01-11T07:56:18.404 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-uclogic.ko 2024-01-11T07:56:18.404 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-waltop.ko 2024-01-11T07:56:18.405 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-wiimote.ko 2024-01-11T07:56:18.405 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-xinmo.ko 2024-01-11T07:56:18.405 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-zpff.ko 2024-01-11T07:56:18.405 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-zydacron.ko 2024-01-11T07:56:18.405 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/uhid.ko 2024-01-11T07:56:18.405 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/wacom.ko 2024-01-11T07:56:18.405 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hv 2024-01-11T07:56:18.405 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hv/hv_balloon.ko 2024-01-11T07:56:18.405 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hv/hv_utils.ko 2024-01-11T07:56:18.405 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hv/hv_vmbus.ko 2024-01-11T07:56:18.405 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon 2024-01-11T07:56:18.405 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/abituguru.ko 2024-01-11T07:56:18.405 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/abituguru3.ko 2024-01-11T07:56:18.405 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/acpi_power_meter.ko 2024-01-11T07:56:18.406 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ad7414.ko 2024-01-11T07:56:18.406 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ad7418.ko 2024-01-11T07:56:18.406 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adc128d818.ko 2024-01-11T07:56:18.406 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adm1025.ko 2024-01-11T07:56:18.406 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adm1026.ko 2024-01-11T07:56:18.406 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adm1029.ko 2024-01-11T07:56:18.406 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adm1031.ko 2024-01-11T07:56:18.406 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adm9240.ko 2024-01-11T07:56:18.406 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ads7828.ko 2024-01-11T07:56:18.406 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adt7410.ko 2024-01-11T07:56:18.406 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adt7411.ko 2024-01-11T07:56:18.407 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adt7462.ko 2024-01-11T07:56:18.407 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adt7470.ko 2024-01-11T07:56:18.407 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adt7475.ko 2024-01-11T07:56:18.407 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adt7x10.ko 2024-01-11T07:56:18.407 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/amc6821.ko 2024-01-11T07:56:18.407 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/applesmc.ko 2024-01-11T07:56:18.407 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/asb100.ko 2024-01-11T07:56:18.407 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/asc7621.ko 2024-01-11T07:56:18.407 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/asus_atk0110.ko 2024-01-11T07:56:18.407 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/atxp1.ko 2024-01-11T07:56:18.407 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/coretemp.ko 2024-01-11T07:56:18.407 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-01-11T07:56:18.408 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/dme1737.ko 2024-01-11T07:56:18.408 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ds1621.ko 2024-01-11T07:56:18.408 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ds620.ko 2024-01-11T07:56:18.408 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/emc1403.ko 2024-01-11T07:56:18.408 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/emc6w201.ko 2024-01-11T07:56:18.408 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/f71805f.ko 2024-01-11T07:56:18.408 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/f71882fg.ko 2024-01-11T07:56:18.408 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/f75375s.ko 2024-01-11T07:56:18.408 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/fam15h_power.ko 2024-01-11T07:56:18.408 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/fschmd.ko 2024-01-11T07:56:18.408 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/g760a.ko 2024-01-11T07:56:18.408 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/g762.ko 2024-01-11T07:56:18.409 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/gl518sm.ko 2024-01-11T07:56:18.409 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/gl520sm.ko 2024-01-11T07:56:18.409 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/hwmon-vid.ko 2024-01-11T07:56:18.409 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/i5500_temp.ko 2024-01-11T07:56:18.409 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/i5k_amb.ko 2024-01-11T07:56:18.409 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ibmaem.ko 2024-01-11T07:56:18.409 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ibmpex.ko 2024-01-11T07:56:18.409 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ina209.ko 2024-01-11T07:56:18.409 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ina2xx.ko 2024-01-11T07:56:18.409 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/it87.ko 2024-01-11T07:56:18.409 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/k10temp.ko 2024-01-11T07:56:18.409 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/k8temp.ko 2024-01-11T07:56:18.410 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lineage-pem.ko 2024-01-11T07:56:18.410 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm63.ko 2024-01-11T07:56:18.410 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm73.ko 2024-01-11T07:56:18.410 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm75.ko 2024-01-11T07:56:18.410 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm77.ko 2024-01-11T07:56:18.410 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm78.ko 2024-01-11T07:56:18.410 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm80.ko 2024-01-11T07:56:18.410 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm83.ko 2024-01-11T07:56:18.410 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm85.ko 2024-01-11T07:56:18.410 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm87.ko 2024-01-11T07:56:18.411 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm90.ko 2024-01-11T07:56:18.411 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm92.ko 2024-01-11T07:56:18.411 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm93.ko 2024-01-11T07:56:18.411 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm95234.ko 2024-01-11T07:56:18.411 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm95241.ko 2024-01-11T07:56:18.411 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm95245.ko 2024-01-11T07:56:18.411 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ltc2945.ko 2024-01-11T07:56:18.411 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ltc4151.ko 2024-01-11T07:56:18.411 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ltc4215.ko 2024-01-11T07:56:18.411 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ltc4222.ko 2024-01-11T07:56:18.412 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ltc4245.ko 2024-01-11T07:56:18.412 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ltc4260.ko 2024-01-11T07:56:18.412 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ltc4261.ko 2024-01-11T07:56:18.412 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/max16065.ko 2024-01-11T07:56:18.412 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/max1619.ko 2024-01-11T07:56:18.412 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/max1668.ko 2024-01-11T07:56:18.412 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/max197.ko 2024-01-11T07:56:18.412 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/max6639.ko 2024-01-11T07:56:18.412 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/max6650.ko 2024-01-11T07:56:18.412 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/max6697.ko 2024-01-11T07:56:18.412 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/mcp3021.ko 2024-01-11T07:56:18.412 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/nct6683.ko 2024-01-11T07:56:18.413 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/nct6775-core.ko 2024-01-11T07:56:18.413 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/nct6775.ko 2024-01-11T07:56:18.413 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/nct7802.ko 2024-01-11T07:56:18.413 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pc87360.ko 2024-01-11T07:56:18.413 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pc87427.ko 2024-01-11T07:56:18.413 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pcf8591.ko 2024-01-11T07:56:18.413 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus 2024-01-11T07:56:18.413 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-01-11T07:56:18.413 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-01-11T07:56:18.413 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-01-11T07:56:18.413 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/max16064.ko 2024-01-11T07:56:18.413 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/max34440.ko 2024-01-11T07:56:18.413 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/max8688.ko 2024-01-11T07:56:18.414 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-01-11T07:56:18.414 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-01-11T07:56:18.414 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-01-11T07:56:18.414 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-01-11T07:56:18.414 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-01-11T07:56:18.414 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-01-11T07:56:18.414 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/powr1220.ko 2024-01-11T07:56:18.414 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/sch5627.ko 2024-01-11T07:56:18.414 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/sch5636.ko 2024-01-11T07:56:18.414 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/sch56xx-common.ko 2024-01-11T07:56:18.414 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/sht15.ko 2024-01-11T07:56:18.414 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/sht21.ko 2024-01-11T07:56:18.415 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/shtc1.ko 2024-01-11T07:56:18.415 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/sis5595.ko 2024-01-11T07:56:18.415 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/smsc47b397.ko 2024-01-11T07:56:18.415 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/smsc47m1.ko 2024-01-11T07:56:18.415 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/smsc47m192.ko 2024-01-11T07:56:18.415 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/thmc50.ko 2024-01-11T07:56:18.415 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/tmp102.ko 2024-01-11T07:56:18.415 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/tmp103.ko 2024-01-11T07:56:18.415 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/tmp401.ko 2024-01-11T07:56:18.415 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/tmp421.ko 2024-01-11T07:56:18.415 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/via-cputemp.ko 2024-01-11T07:56:18.415 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/via686a.ko 2024-01-11T07:56:18.415 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/vt1211.ko 2024-01-11T07:56:18.416 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/vt8231.ko 2024-01-11T07:56:18.416 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/w83627ehf.ko 2024-01-11T07:56:18.416 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/w83627hf.ko 2024-01-11T07:56:18.416 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/w83781d.ko 2024-01-11T07:56:18.416 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/w83791d.ko 2024-01-11T07:56:18.416 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/w83792d.ko 2024-01-11T07:56:18.416 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/w83793.ko 2024-01-11T07:56:18.416 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/w83795.ko 2024-01-11T07:56:18.416 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/w83l785ts.ko 2024-01-11T07:56:18.416 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/w83l786ng.ko 2024-01-11T07:56:18.416 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c 2024-01-11T07:56:18.416 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/algos 2024-01-11T07:56:18.417 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-01-11T07:56:18.417 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-01-11T07:56:18.417 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses 2024-01-11T07:56:18.417 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-01-11T07:56:18.417 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-01-11T07:56:18.417 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-01-11T07:56:18.417 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-01-11T07:56:18.417 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-01-11T07:56:18.417 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-01-11T07:56:18.417 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-01-11T07:56:18.417 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-01-11T07:56:18.417 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-i801.ko 2024-01-11T07:56:18.417 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-isch.ko 2024-01-11T07:56:18.418 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-01-11T07:56:18.418 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-01-11T07:56:18.418 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-01-11T07:56:18.418 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-parport.ko 2024-01-11T07:56:18.418 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-01-11T07:56:18.418 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-01-11T07:56:18.418 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-01-11T07:56:18.418 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-01-11T07:56:18.418 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-01-11T07:56:18.418 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-01-11T07:56:18.418 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-via.ko 2024-01-11T07:56:18.418 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-01-11T07:56:18.418 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-01-11T07:56:18.419 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/i2c-dev.ko 2024-01-11T07:56:18.419 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/i2c-mux.ko 2024-01-11T07:56:18.419 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-01-11T07:56:18.419 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/i2c-smbus.ko 2024-01-11T07:56:18.419 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/i2c-stub.ko 2024-01-11T07:56:18.419 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband 2024-01-11T07:56:18.419 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/core 2024-01-11T07:56:18.419 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/core/ib_cm.ko 2024-01-11T07:56:18.419 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/core/ib_core.ko 2024-01-11T07:56:18.419 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/core/ib_umad.ko 2024-01-11T07:56:18.419 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-01-11T07:56:18.419 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/core/iw_cm.ko 2024-01-11T07:56:18.420 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/core/rdma_cm.ko 2024-01-11T07:56:18.420 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-01-11T07:56:18.420 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/hw 2024-01-11T07:56:18.420 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/hw/cxgb4 2024-01-11T07:56:18.420 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-01-11T07:56:18.420 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/hw/mlx4 2024-01-11T07:56:18.420 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-01-11T07:56:18.420 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/hw/mlx5 2024-01-11T07:56:18.420 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-01-11T07:56:18.420 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/hw/mthca 2024-01-11T07:56:18.420 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-01-11T07:56:18.420 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp 2024-01-11T07:56:18.421 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/ipoib 2024-01-11T07:56:18.421 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-01-11T07:56:18.421 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/iser 2024-01-11T07:56:18.421 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-01-11T07:56:18.421 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/isert 2024-01-11T07:56:18.421 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-01-11T07:56:18.421 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/srp 2024-01-11T07:56:18.421 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-01-11T07:56:18.421 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/srpt 2024-01-11T07:56:18.421 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-01-11T07:56:18.421 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input 2024-01-11T07:56:18.421 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/gameport 2024-01-11T07:56:18.421 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-01-11T07:56:18.422 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/gameport/fm801-gp.ko 2024-01-11T07:56:18.422 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/gameport/gameport.ko 2024-01-11T07:56:18.422 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/joydev.ko 2024-01-11T07:56:18.422 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/keyboard 2024-01-11T07:56:18.422 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/keyboard/gpio_keys.ko 2024-01-11T07:56:18.422 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/mouse 2024-01-11T07:56:18.422 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/mouse/appletouch.ko 2024-01-11T07:56:18.422 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/mouse/bcm5974.ko 2024-01-11T07:56:18.422 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/mouse/cyapatp.ko 2024-01-11T07:56:18.422 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/mouse/elan_i2c.ko 2024-01-11T07:56:18.422 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/mouse/sermouse.ko 2024-01-11T07:56:18.422 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-01-11T07:56:18.423 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/mouse/synaptics_usb.ko 2024-01-11T07:56:18.423 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/mouse/vsxxxaa.ko 2024-01-11T07:56:18.423 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/rmi4 2024-01-11T07:56:18.423 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/rmi4/rmi_core.ko 2024-01-11T07:56:18.423 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/serio 2024-01-11T07:56:18.423 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/serio/altera_ps2.ko 2024-01-11T07:56:18.423 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/serio/arc_ps2.ko 2024-01-11T07:56:18.423 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-01-11T07:56:18.423 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/serio/serio_raw.ko 2024-01-11T07:56:18.423 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/sparse-keymap.ko 2024-01-11T07:56:18.423 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/iommu 2024-01-11T07:56:18.423 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/iommu/amd 2024-01-11T07:56:18.423 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/iommu/amd/iommu_v2.ko 2024-01-11T07:56:18.424 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds 2024-01-11T07:56:18.424 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/leds-blinkm.ko 2024-01-11T07:56:18.424 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/leds-lm3530.ko 2024-01-11T07:56:18.424 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/leds-lp3944.ko 2024-01-11T07:56:18.424 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/leds-ss4200.ko 2024-01-11T07:56:18.424 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/trigger 2024-01-11T07:56:18.424 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-01-11T07:56:18.424 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-01-11T07:56:18.424 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-01-11T07:56:18.424 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-01-11T07:56:18.424 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-01-11T07:56:18.424 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-01-11T07:56:18.425 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-01-11T07:56:18.425 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-01-11T07:56:18.425 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md 2024-01-11T07:56:18.425 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/bcache 2024-01-11T07:56:18.425 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/bcache/bcache.ko 2024-01-11T07:56:18.425 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-bio-prison.ko 2024-01-11T07:56:18.425 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-cache-smq.ko 2024-01-11T07:56:18.425 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-cache.ko 2024-01-11T07:56:18.425 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-crypt.ko 2024-01-11T07:56:18.425 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-delay.ko 2024-01-11T07:56:18.425 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-flakey.ko 2024-01-11T07:56:18.425 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-log-userspace.ko 2024-01-11T07:56:18.425 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-multipath.ko 2024-01-11T07:56:18.426 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-queue-length.ko 2024-01-11T07:56:18.426 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-raid.ko 2024-01-11T07:56:18.426 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-round-robin.ko 2024-01-11T07:56:18.426 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-service-time.ko 2024-01-11T07:56:18.426 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-switch.ko 2024-01-11T07:56:18.426 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-thin-pool.ko 2024-01-11T07:56:18.426 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-verity.ko 2024-01-11T07:56:18.426 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/faulty.ko 2024-01-11T07:56:18.426 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/linear.ko 2024-01-11T07:56:18.426 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/multipath.ko 2024-01-11T07:56:18.426 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/persistent-data 2024-01-11T07:56:18.426 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-01-11T07:56:18.426 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/raid0.ko 2024-01-11T07:56:18.427 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/raid1.ko 2024-01-11T07:56:18.427 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/raid10.ko 2024-01-11T07:56:18.427 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/raid456.ko 2024-01-11T07:56:18.427 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/message 2024-01-11T07:56:18.427 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/message/fusion 2024-01-11T07:56:18.427 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/message/fusion/mptbase.ko 2024-01-11T07:56:18.427 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/message/fusion/mptctl.ko 2024-01-11T07:56:18.427 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/message/fusion/mptfc.ko 2024-01-11T07:56:18.427 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/message/fusion/mptlan.ko 2024-01-11T07:56:18.427 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/message/fusion/mptsas.ko 2024-01-11T07:56:18.427 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/message/fusion/mptscsih.ko 2024-01-11T07:56:18.427 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/message/fusion/mptspi.ko 2024-01-11T07:56:18.428 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/mfd 2024-01-11T07:56:18.428 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/mfd/lpc_ich.ko 2024-01-11T07:56:18.428 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/mfd/lpc_sch.ko 2024-01-11T07:56:18.428 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/mfd/mfd-core.ko 2024-01-11T07:56:18.428 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/mfd/sm501.ko 2024-01-11T07:56:18.428 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/mfd/viperboard.ko 2024-01-11T07:56:18.428 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/mfd/vx855.ko 2024-01-11T07:56:18.428 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/mfd/wl1273-core.ko 2024-01-11T07:56:18.428 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc 2024-01-11T07:56:18.428 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/altera-stapl 2024-01-11T07:56:18.428 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-01-11T07:56:18.428 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/apds9802als.ko 2024-01-11T07:56:18.428 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/apds990x.ko 2024-01-11T07:56:18.429 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/bh1770glc.ko 2024-01-11T07:56:18.429 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/cb710 2024-01-11T07:56:18.429 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/cb710/cb710.ko 2024-01-11T07:56:18.429 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/eeprom 2024-01-11T07:56:18.429 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/eeprom/at24.ko 2024-01-11T07:56:18.429 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/eeprom/eeprom.ko 2024-01-11T07:56:18.429 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-01-11T07:56:18.429 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/eeprom/max6875.ko 2024-01-11T07:56:18.429 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/enclosure.ko 2024-01-11T07:56:18.429 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/hpilo.ko 2024-01-11T07:56:18.429 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/isl29003.ko 2024-01-11T07:56:18.429 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/isl29020.ko 2024-01-11T07:56:18.430 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/lis3lv02d 2024-01-11T07:56:18.430 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-01-11T07:56:18.430 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-01-11T07:56:18.430 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/mei 2024-01-11T07:56:18.430 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/mei/mei-me.ko 2024-01-11T07:56:18.430 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/mei/mei-txe.ko 2024-01-11T07:56:18.430 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/mei/mei.ko 2024-01-11T07:56:18.430 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/sgi-gru 2024-01-11T07:56:18.430 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/sgi-gru/gru.ko 2024-01-11T07:56:18.430 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/sgi-xp 2024-01-11T07:56:18.430 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/sgi-xp/xp.ko 2024-01-11T07:56:18.430 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/sgi-xp/xpc.ko 2024-01-11T07:56:18.431 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-01-11T07:56:18.431 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/tifm_7xx1.ko 2024-01-11T07:56:18.431 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/tifm_core.ko 2024-01-11T07:56:18.431 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/tsl2550.ko 2024-01-11T07:56:18.431 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/vmw_balloon.ko 2024-01-11T07:56:18.431 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/vmw_vmci 2024-01-11T07:56:18.431 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-01-11T07:56:18.431 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net 2024-01-11T07:56:18.431 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/bonding 2024-01-11T07:56:18.431 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/bonding/bonding.ko 2024-01-11T07:56:18.431 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/dsa 2024-01-11T07:56:18.431 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/dsa/b53 2024-01-11T07:56:18.431 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/dsa/b53/b53_common.ko 2024-01-11T07:56:18.432 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/dsa/bcm-sf2.ko 2024-01-11T07:56:18.432 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/dsa/mv88e6060.ko 2024-01-11T07:56:18.432 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/dsa/mv88e6xxx 2024-01-11T07:56:18.432 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-01-11T07:56:18.432 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/dummy.ko 2024-01-11T07:56:18.432 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/eql.ko 2024-01-11T07:56:18.432 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet 2024-01-11T07:56:18.432 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/3com 2024-01-11T07:56:18.432 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-01-11T07:56:18.432 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-01-11T07:56:18.432 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-01-11T07:56:18.432 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-01-11T07:56:18.433 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/8390 2024-01-11T07:56:18.433 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/8390/8390.ko 2024-01-11T07:56:18.433 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-01-11T07:56:18.433 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-01-11T07:56:18.433 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-01-11T07:56:18.433 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/adaptec 2024-01-11T07:56:18.433 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-01-11T07:56:18.433 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/agere 2024-01-11T07:56:18.433 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/agere/et131x.ko 2024-01-11T07:56:18.433 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/alteon 2024-01-11T07:56:18.433 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-01-11T07:56:18.434 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/altera 2024-01-11T07:56:18.434 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-01-11T07:56:18.434 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/amd 2024-01-11T07:56:18.434 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-01-11T07:56:18.434 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-01-11T07:56:18.434 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-01-11T07:56:18.434 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros 2024-01-11T07:56:18.434 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros/alx 2024-01-11T07:56:18.434 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-01-11T07:56:18.434 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros/atl1c 2024-01-11T07:56:18.434 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-01-11T07:56:18.434 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros/atl1e 2024-01-11T07:56:18.435 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-01-11T07:56:18.435 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros/atlx 2024-01-11T07:56:18.435 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-01-11T07:56:18.435 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-01-11T07:56:18.435 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/broadcom 2024-01-11T07:56:18.435 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-01-11T07:56:18.435 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-01-11T07:56:18.435 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-01-11T07:56:18.435 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-01-11T07:56:18.435 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-01-11T07:56:18.435 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/broadcom/genet 2024-01-11T07:56:18.435 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-01-11T07:56:18.436 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-01-11T07:56:18.436 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/brocade 2024-01-11T07:56:18.436 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/brocade/bna 2024-01-11T07:56:18.436 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-01-11T07:56:18.436 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio 2024-01-11T07:56:18.436 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/cxgb 2024-01-11T07:56:18.436 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-01-11T07:56:18.436 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-01-11T07:56:18.436 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-01-11T07:56:18.436 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-01-11T07:56:18.436 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-01-11T07:56:18.436 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-01-11T07:56:18.436 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-01-11T07:56:18.437 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-01-11T07:56:18.437 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-01-11T07:56:18.437 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/cisco 2024-01-11T07:56:18.437 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/cisco/enic 2024-01-11T07:56:18.437 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-01-11T07:56:18.437 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dec 2024-01-11T07:56:18.437 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dec/tulip 2024-01-11T07:56:18.437 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-01-11T07:56:18.437 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-01-11T07:56:18.437 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-01-11T07:56:18.437 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-01-11T07:56:18.437 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-01-11T07:56:18.438 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-01-11T07:56:18.438 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dlink 2024-01-11T07:56:18.438 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-01-11T07:56:18.438 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-01-11T07:56:18.438 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dnet.ko 2024-01-11T07:56:18.438 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/emulex 2024-01-11T07:56:18.438 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/emulex/benet 2024-01-11T07:56:18.438 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-01-11T07:56:18.438 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/ethoc.ko 2024-01-11T07:56:18.438 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/fealnx.ko 2024-01-11T07:56:18.438 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel 2024-01-11T07:56:18.439 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/e100.ko 2024-01-11T07:56:18.439 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/e1000 2024-01-11T07:56:18.439 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-01-11T07:56:18.439 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/e1000e 2024-01-11T07:56:18.439 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-01-11T07:56:18.439 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/fm10k 2024-01-11T07:56:18.439 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-01-11T07:56:18.439 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/i40e 2024-01-11T07:56:18.439 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-01-11T07:56:18.439 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/iavf 2024-01-11T07:56:18.439 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-01-11T07:56:18.439 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/igb 2024-01-11T07:56:18.440 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-01-11T07:56:18.440 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/igbvf 2024-01-11T07:56:18.440 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-01-11T07:56:18.440 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/ixgbe 2024-01-11T07:56:18.440 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-01-11T07:56:18.440 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/ixgbevf 2024-01-11T07:56:18.440 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-01-11T07:56:18.440 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/jme.ko 2024-01-11T07:56:18.440 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/marvell 2024-01-11T07:56:18.440 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-01-11T07:56:18.440 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/marvell/skge.ko 2024-01-11T07:56:18.440 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-01-11T07:56:18.441 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/mellanox 2024-01-11T07:56:18.441 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/mellanox/mlx4 2024-01-11T07:56:18.441 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-01-11T07:56:18.441 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-01-11T07:56:18.441 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/mellanox/mlx5 2024-01-11T07:56:18.441 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-01-11T07:56:18.441 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-01-11T07:56:18.441 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/micrel 2024-01-11T07:56:18.441 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-01-11T07:56:18.441 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/myricom 2024-01-11T07:56:18.441 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/myricom/myri10ge 2024-01-11T07:56:18.441 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-01-11T07:56:18.442 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/natsemi 2024-01-11T07:56:18.442 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-01-11T07:56:18.442 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-01-11T07:56:18.442 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/neterion 2024-01-11T07:56:18.442 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-01-11T07:56:18.442 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/nvidia 2024-01-11T07:56:18.442 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-01-11T07:56:18.442 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/packetengines 2024-01-11T07:56:18.442 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-01-11T07:56:18.442 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-01-11T07:56:18.442 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/qlogic 2024-01-11T07:56:18.442 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/qlogic/netxen 2024-01-11T07:56:18.443 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-01-11T07:56:18.443 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-01-11T07:56:18.443 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-01-11T07:56:18.443 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-01-11T07:56:18.443 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/rdc 2024-01-11T07:56:18.443 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-01-11T07:56:18.443 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/realtek 2024-01-11T07:56:18.443 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-01-11T07:56:18.443 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-01-11T07:56:18.443 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/realtek/atp.ko 2024-01-11T07:56:18.443 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-01-11T07:56:18.443 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/rocker 2024-01-11T07:56:18.443 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-01-11T07:56:18.444 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sfc 2024-01-11T07:56:18.444 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-01-11T07:56:18.444 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/silan 2024-01-11T07:56:18.444 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-01-11T07:56:18.444 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sis 2024-01-11T07:56:18.444 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sis/sis190.ko 2024-01-11T07:56:18.444 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sis/sis900.ko 2024-01-11T07:56:18.444 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/smsc 2024-01-11T07:56:18.444 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-01-11T07:56:18.444 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-01-11T07:56:18.444 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-01-11T07:56:18.445 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-01-11T07:56:18.445 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/stmicro 2024-01-11T07:56:18.445 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/stmicro/stmmac 2024-01-11T07:56:18.445 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-01-11T07:56:18.445 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-01-11T07:56:18.445 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sun 2024-01-11T07:56:18.445 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sun/cassini.ko 2024-01-11T07:56:18.445 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sun/niu.ko 2024-01-11T07:56:18.445 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sun/sungem.ko 2024-01-11T07:56:18.445 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-01-11T07:56:18.445 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/tehuti 2024-01-11T07:56:18.445 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-01-11T07:56:18.445 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/ti 2024-01-11T07:56:18.446 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/ti/tlan.ko 2024-01-11T07:56:18.446 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/via 2024-01-11T07:56:18.446 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-01-11T07:56:18.446 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-01-11T07:56:18.446 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/wiznet 2024-01-11T07:56:18.446 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-01-11T07:56:18.446 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-01-11T07:56:18.446 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/xircom 2024-01-11T07:56:18.446 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-01-11T07:56:18.446 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/geneve.ko 2024-01-11T07:56:18.446 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/hyperv 2024-01-11T07:56:18.446 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-01-11T07:56:18.447 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ieee802154 2024-01-11T07:56:18.447 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ieee802154/fakelb.ko 2024-01-11T07:56:18.447 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ifb.ko 2024-01-11T07:56:18.447 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ipvlan 2024-01-11T07:56:18.447 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ipvlan/ipvlan.ko 2024-01-11T07:56:18.447 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/macvlan.ko 2024-01-11T07:56:18.447 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/macvtap.ko 2024-01-11T07:56:18.447 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/mdio 2024-01-11T07:56:18.447 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/mdio.ko 2024-01-11T07:56:18.447 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-01-11T07:56:18.447 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-01-11T07:56:18.447 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/mdio/mdio-regmap.ko 2024-01-11T07:56:18.447 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/mii.ko 2024-01-11T07:56:18.448 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/net_failover.ko 2024-01-11T07:56:18.448 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/netconsole.ko 2024-01-11T07:56:18.448 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/nlmon.ko 2024-01-11T07:56:18.448 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/pcs 2024-01-11T07:56:18.448 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/pcs/pcs-lynx.ko 2024-01-11T07:56:18.448 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-01-11T07:56:18.448 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy 2024-01-11T07:56:18.448 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/amd.ko 2024-01-11T07:56:18.448 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-01-11T07:56:18.448 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-01-11T07:56:18.448 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/bcm7xxx.ko 2024-01-11T07:56:18.448 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/bcm87xx.ko 2024-01-11T07:56:18.449 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/broadcom.ko 2024-01-11T07:56:18.449 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/cicada.ko 2024-01-11T07:56:18.449 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/davicom.ko 2024-01-11T07:56:18.449 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/dp83640.ko 2024-01-11T07:56:18.449 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/et1011c.ko 2024-01-11T07:56:18.449 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/icplus.ko 2024-01-11T07:56:18.449 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/lxt.ko 2024-01-11T07:56:18.449 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/marvell.ko 2024-01-11T07:56:18.449 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/micrel.ko 2024-01-11T07:56:18.449 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/national.ko 2024-01-11T07:56:18.449 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/phylink.ko 2024-01-11T07:56:18.449 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/qsemi.ko 2024-01-11T07:56:18.450 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/realtek.ko 2024-01-11T07:56:18.450 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/smsc.ko 2024-01-11T07:56:18.450 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/ste10Xp.ko 2024-01-11T07:56:18.450 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/vitesse.ko 2024-01-11T07:56:18.450 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp 2024-01-11T07:56:18.450 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp/bsd_comp.ko 2024-01-11T07:56:18.450 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp/ppp_async.ko 2024-01-11T07:56:18.450 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp/ppp_deflate.ko 2024-01-11T07:56:18.450 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp/ppp_generic.ko 2024-01-11T07:56:18.450 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp/ppp_mppe.ko 2024-01-11T07:56:18.450 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp/ppp_synctty.ko 2024-01-11T07:56:18.450 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp/pppoe.ko 2024-01-11T07:56:18.450 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp/pppox.ko 2024-01-11T07:56:18.451 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp/pptp.ko 2024-01-11T07:56:18.451 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/slip 2024-01-11T07:56:18.451 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/slip/slhc.ko 2024-01-11T07:56:18.451 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/slip/slip.ko 2024-01-11T07:56:18.451 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/sungem_phy.ko 2024-01-11T07:56:18.451 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/tap.ko 2024-01-11T07:56:18.451 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/team 2024-01-11T07:56:18.451 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/team/team.ko 2024-01-11T07:56:18.451 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/team/team_mode_activebackup.ko 2024-01-11T07:56:18.451 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/team/team_mode_broadcast.ko 2024-01-11T07:56:18.451 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-01-11T07:56:18.451 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/team/team_mode_random.ko 2024-01-11T07:56:18.451 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-01-11T07:56:18.452 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/tun.ko 2024-01-11T07:56:18.452 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/veth.ko 2024-01-11T07:56:18.452 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/virtio_net.ko 2024-01-11T07:56:18.452 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/vmxnet3 2024-01-11T07:56:18.452 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-01-11T07:56:18.452 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/vxlan 2024-01-11T07:56:18.452 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/vxlan/vxlan.ko 2024-01-11T07:56:18.452 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/xen-netback 2024-01-11T07:56:18.452 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/xen-netback/xen-netback.ko 2024-01-11T07:56:18.452 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/xen-netfront.ko 2024-01-11T07:56:18.452 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ntb 2024-01-11T07:56:18.452 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ntb/ntb.ko 2024-01-11T07:56:18.453 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/nvme 2024-01-11T07:56:18.453 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/nvme/host 2024-01-11T07:56:18.453 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/nvme/host/nvme-core.ko 2024-01-11T07:56:18.453 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/nvme/host/nvme.ko 2024-01-11T07:56:18.453 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/parport 2024-01-11T07:56:18.453 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/parport/parport.ko 2024-01-11T07:56:18.453 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/parport/parport_cs.ko 2024-01-11T07:56:18.453 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/parport/parport_pc.ko 2024-01-11T07:56:18.453 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/parport/parport_serial.ko 2024-01-11T07:56:18.453 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pci 2024-01-11T07:56:18.453 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pci/hotplug 2024-01-11T07:56:18.453 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-01-11T07:56:18.453 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pci/pcie 2024-01-11T07:56:18.454 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pci/pcie/aer_inject.ko 2024-01-11T07:56:18.454 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pci/xen-pcifront.ko 2024-01-11T07:56:18.454 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pcmcia 2024-01-11T07:56:18.454 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pcmcia/i82092.ko 2024-01-11T07:56:18.454 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pcmcia/pd6729.ko 2024-01-11T07:56:18.454 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pcmcia/yenta_socket.ko 2024-01-11T07:56:18.454 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pinctrl 2024-01-11T07:56:18.454 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pinctrl/intel 2024-01-11T07:56:18.454 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-01-11T07:56:18.454 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform 2024-01-11T07:56:18.454 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86 2024-01-11T07:56:18.454 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/acer-wmi.ko 2024-01-11T07:56:18.454 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/acerhdf.ko 2024-01-11T07:56:18.455 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/apple-gmux.ko 2024-01-11T07:56:18.455 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/asus-laptop.ko 2024-01-11T07:56:18.455 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-01-11T07:56:18.455 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/asus-wmi.ko 2024-01-11T07:56:18.455 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/classmate-laptop.ko 2024-01-11T07:56:18.455 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-01-11T07:56:18.455 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-01-11T07:56:18.455 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-01-11T07:56:18.455 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-01-11T07:56:18.455 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/hdaps.ko 2024-01-11T07:56:18.455 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/intel 2024-01-11T07:56:18.455 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-01-11T07:56:18.455 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/intel_ips.ko 2024-01-11T07:56:18.456 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/msi-wmi.ko 2024-01-11T07:56:18.456 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/mxm-wmi.ko 2024-01-11T07:56:18.456 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-01-11T07:56:18.456 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/samsung-laptop.ko 2024-01-11T07:56:18.456 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/samsung-q10.ko 2024-01-11T07:56:18.456 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/topstar-laptop.ko 2024-01-11T07:56:18.456 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-01-11T07:56:18.456 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/toshiba_haps.ko 2024-01-11T07:56:18.456 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/wmi-bmof.ko 2024-01-11T07:56:18.456 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/wmi.ko 2024-01-11T07:56:18.456 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/powercap 2024-01-11T07:56:18.456 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/powercap/intel_rapl_common.ko 2024-01-11T07:56:18.457 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/powercap/intel_rapl_msr.ko 2024-01-11T07:56:18.457 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pps 2024-01-11T07:56:18.457 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pps/clients 2024-01-11T07:56:18.457 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pps/clients/pps-gpio.ko 2024-01-11T07:56:18.457 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pps/clients/pps-ldisc.ko 2024-01-11T07:56:18.457 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pps/clients/pps_parport.ko 2024-01-11T07:56:18.457 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pps/pps_core.ko 2024-01-11T07:56:18.457 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ptp 2024-01-11T07:56:18.457 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ptp/ptp.ko 2024-01-11T07:56:18.457 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ptp/ptp_kvm.ko 2024-01-11T07:56:18.457 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc 2024-01-11T07:56:18.457 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-bq32k.ko 2024-01-11T07:56:18.457 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-ds1286.ko 2024-01-11T07:56:18.458 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-ds1307.ko 2024-01-11T07:56:18.458 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-ds1374.ko 2024-01-11T07:56:18.458 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-ds1511.ko 2024-01-11T07:56:18.458 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-ds1553.ko 2024-01-11T07:56:18.458 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-ds1672.ko 2024-01-11T07:56:18.458 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-ds1742.ko 2024-01-11T07:56:18.458 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-ds2404.ko 2024-01-11T07:56:18.458 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-ds3232.ko 2024-01-11T07:56:18.458 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-em3027.ko 2024-01-11T07:56:18.458 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-fm3130.ko 2024-01-11T07:56:18.458 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-isl12022.ko 2024-01-11T07:56:18.458 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-isl1208.ko 2024-01-11T07:56:18.458 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-m41t80.ko 2024-01-11T07:56:18.459 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-m48t35.ko 2024-01-11T07:56:18.459 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-m48t59.ko 2024-01-11T07:56:18.459 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-max6900.ko 2024-01-11T07:56:18.459 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-msm6242.ko 2024-01-11T07:56:18.459 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-pcf2127.ko 2024-01-11T07:56:18.459 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-pcf85063.ko 2024-01-11T07:56:18.459 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-pcf8523.ko 2024-01-11T07:56:18.459 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-pcf8563.ko 2024-01-11T07:56:18.459 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-pcf8583.ko 2024-01-11T07:56:18.459 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-rp5c01.ko 2024-01-11T07:56:18.459 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-rs5c372.ko 2024-01-11T07:56:18.459 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-01-11T07:56:18.460 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-rx8025.ko 2024-01-11T07:56:18.460 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-rx8581.ko 2024-01-11T07:56:18.460 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-01-11T07:56:18.460 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-x1205.ko 2024-01-11T07:56:18.460 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi 2024-01-11T07:56:18.460 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/3w-9xxx.ko 2024-01-11T07:56:18.460 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/3w-sas.ko 2024-01-11T07:56:18.460 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/3w-xxxx.ko 2024-01-11T07:56:18.460 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/BusLogic.ko 2024-01-11T07:56:18.460 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/a100u2w.ko 2024-01-11T07:56:18.460 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/aacraid 2024-01-11T07:56:18.460 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/aacraid/aacraid.ko 2024-01-11T07:56:18.460 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/advansys.ko 2024-01-11T07:56:18.461 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/aic7xxx 2024-01-11T07:56:18.461 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-01-11T07:56:18.461 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-01-11T07:56:18.461 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/am53c974.ko 2024-01-11T07:56:18.461 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/arcmsr 2024-01-11T07:56:18.461 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-01-11T07:56:18.461 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/atp870u.ko 2024-01-11T07:56:18.461 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/be2iscsi 2024-01-11T07:56:18.461 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-01-11T07:56:18.461 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/bfa 2024-01-11T07:56:18.461 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/bfa/bfa.ko 2024-01-11T07:56:18.461 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/bnx2fc 2024-01-11T07:56:18.461 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-01-11T07:56:18.462 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/bnx2i 2024-01-11T07:56:18.462 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-01-11T07:56:18.462 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/ch.ko 2024-01-11T07:56:18.462 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/csiostor 2024-01-11T07:56:18.462 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/csiostor/csiostor.ko 2024-01-11T07:56:18.462 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/cxgbi 2024-01-11T07:56:18.462 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/cxgbi/cxgb3i 2024-01-11T07:56:18.462 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-01-11T07:56:18.462 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/cxgbi/cxgb4i 2024-01-11T07:56:18.462 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-01-11T07:56:18.462 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-01-11T07:56:18.462 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/dc395x.ko 2024-01-11T07:56:18.463 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/device_handler 2024-01-11T07:56:18.463 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-01-11T07:56:18.463 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-01-11T07:56:18.463 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-01-11T07:56:18.463 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-01-11T07:56:18.463 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/dmx3191d.ko 2024-01-11T07:56:18.463 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/esas2r 2024-01-11T07:56:18.463 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/esas2r/esas2r.ko 2024-01-11T07:56:18.463 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/esp_scsi.ko 2024-01-11T07:56:18.463 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/fcoe 2024-01-11T07:56:18.463 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/fcoe/fcoe.ko 2024-01-11T07:56:18.463 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-01-11T07:56:18.463 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/fnic 2024-01-11T07:56:18.464 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/fnic/fnic.ko 2024-01-11T07:56:18.464 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/hpsa.ko 2024-01-11T07:56:18.464 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/hptiop.ko 2024-01-11T07:56:18.464 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/hv_storvsc.ko 2024-01-11T07:56:18.464 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/initio.ko 2024-01-11T07:56:18.464 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/ipr.ko 2024-01-11T07:56:18.464 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/ips.ko 2024-01-11T07:56:18.464 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/isci 2024-01-11T07:56:18.464 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/isci/isci.ko 2024-01-11T07:56:18.464 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-01-11T07:56:18.464 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/iscsi_tcp.ko 2024-01-11T07:56:18.464 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/libfc 2024-01-11T07:56:18.465 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/libfc/libfc.ko 2024-01-11T07:56:18.465 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/libiscsi.ko 2024-01-11T07:56:18.465 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/libiscsi_tcp.ko 2024-01-11T07:56:18.465 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/libsas 2024-01-11T07:56:18.465 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/libsas/libsas.ko 2024-01-11T07:56:18.465 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/lpfc 2024-01-11T07:56:18.465 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/lpfc/lpfc.ko 2024-01-11T07:56:18.465 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/megaraid 2024-01-11T07:56:18.465 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/megaraid.ko 2024-01-11T07:56:18.465 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-01-11T07:56:18.465 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-01-11T07:56:18.465 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-01-11T07:56:18.465 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/mpt3sas 2024-01-11T07:56:18.466 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-01-11T07:56:18.466 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/mvsas 2024-01-11T07:56:18.466 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/mvsas/mvsas.ko 2024-01-11T07:56:18.466 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/mvumi.ko 2024-01-11T07:56:18.466 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/pm8001 2024-01-11T07:56:18.466 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-01-11T07:56:18.466 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/pmcraid.ko 2024-01-11T07:56:18.466 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/qla1280.ko 2024-01-11T07:56:18.466 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/qla2xxx 2024-01-11T07:56:18.466 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-01-11T07:56:18.466 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-01-11T07:56:18.466 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/qla4xxx 2024-01-11T07:56:18.466 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-01-11T07:56:18.467 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/raid_class.ko 2024-01-11T07:56:18.467 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/scsi_debug.ko 2024-01-11T07:56:18.467 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/scsi_transport_fc.ko 2024-01-11T07:56:18.467 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-01-11T07:56:18.467 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/scsi_transport_sas.ko 2024-01-11T07:56:18.467 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/scsi_transport_spi.ko 2024-01-11T07:56:18.467 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/scsi_transport_srp.ko 2024-01-11T07:56:18.467 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/ses.ko 2024-01-11T07:56:18.467 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/st.ko 2024-01-11T07:56:18.467 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/stex.ko 2024-01-11T07:56:18.467 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/sym53c8xx_2 2024-01-11T07:56:18.467 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-01-11T07:56:18.468 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/virtio_scsi.ko 2024-01-11T07:56:18.468 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/vmw_pvscsi.ko 2024-01-11T07:56:18.468 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/wd719x.ko 2024-01-11T07:56:18.468 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ssb 2024-01-11T07:56:18.468 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ssb/ssb.ko 2024-01-11T07:56:18.468 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target 2024-01-11T07:56:18.468 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/iscsi 2024-01-11T07:56:18.468 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-01-11T07:56:18.468 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/loopback 2024-01-11T07:56:18.468 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/loopback/tcm_loop.ko 2024-01-11T07:56:18.468 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/target_core_file.ko 2024-01-11T07:56:18.468 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/target_core_iblock.ko 2024-01-11T07:56:18.468 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/target_core_mod.ko 2024-01-11T07:56:18.469 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/target_core_pscsi.ko 2024-01-11T07:56:18.469 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/target_core_user.ko 2024-01-11T07:56:18.469 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/tcm_fc 2024-01-11T07:56:18.469 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-01-11T07:56:18.469 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal 2024-01-11T07:56:18.469 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel 2024-01-11T07:56:18.469 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal 2024-01-11T07:56:18.469 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-01-11T07:56:18.469 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-01-11T07:56:18.469 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-01-11T07:56:18.469 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-01-11T07:56:18.469 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-01-11T07:56:18.469 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-01-11T07:56:18.470 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-01-11T07:56:18.470 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-01-11T07:56:18.470 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-01-11T07:56:18.470 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-01-11T07:56:18.470 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-01-11T07:56:18.470 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-01-11T07:56:18.470 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-01-11T07:56:18.470 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-01-11T07:56:18.470 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-01-11T07:56:18.470 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty 2024-01-11T07:56:18.470 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/ipwireless 2024-01-11T07:56:18.470 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-01-11T07:56:18.471 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/n_gsm.ko 2024-01-11T07:56:18.471 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/n_hdlc.ko 2024-01-11T07:56:18.471 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/nozomi.ko 2024-01-11T07:56:18.471 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/serial 2024-01-11T07:56:18.471 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/serial/8250 2024-01-11T07:56:18.471 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-01-11T07:56:18.471 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/serial/arc_uart.ko 2024-01-11T07:56:18.471 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/serial/jsm 2024-01-11T07:56:18.471 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/serial/jsm/jsm.ko 2024-01-11T07:56:18.471 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ufs 2024-01-11T07:56:18.471 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ufs/core 2024-01-11T07:56:18.471 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ufs/core/ufshcd-core.ko 2024-01-11T07:56:18.471 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ufs/host 2024-01-11T07:56:18.472 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-01-11T07:56:18.472 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/uio 2024-01-11T07:56:18.472 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/uio/uio.ko 2024-01-11T07:56:18.472 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/uio/uio_aec.ko 2024-01-11T07:56:18.472 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/uio/uio_cif.ko 2024-01-11T07:56:18.472 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/uio/uio_pci_generic.ko 2024-01-11T07:56:18.472 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/uio/uio_sercos3.ko 2024-01-11T07:56:18.472 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb 2024-01-11T07:56:18.472 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/class 2024-01-11T07:56:18.472 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/class/cdc-acm.ko 2024-01-11T07:56:18.472 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/class/cdc-wdm.ko 2024-01-11T07:56:18.473 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/class/usblp.ko 2024-01-11T07:56:18.473 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/class/usbtmc.ko 2024-01-11T07:56:18.473 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/host 2024-01-11T07:56:18.473 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/host/sl811-hcd.ko 2024-01-11T07:56:18.473 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/image 2024-01-11T07:56:18.473 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/image/mdc800.ko 2024-01-11T07:56:18.473 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/image/microtek.ko 2024-01-11T07:56:18.473 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc 2024-01-11T07:56:18.473 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/adutux.ko 2024-01-11T07:56:18.473 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/appledisplay.ko 2024-01-11T07:56:18.473 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/emi26.ko 2024-01-11T07:56:18.473 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/emi62.ko 2024-01-11T07:56:18.473 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/ezusb.ko 2024-01-11T07:56:18.474 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/idmouse.ko 2024-01-11T07:56:18.474 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/iowarrior.ko 2024-01-11T07:56:18.474 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/isight_firmware.ko 2024-01-11T07:56:18.474 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/ldusb.ko 2024-01-11T07:56:18.474 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/legousbtower.ko 2024-01-11T07:56:18.474 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/sisusbvga 2024-01-11T07:56:18.474 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-01-11T07:56:18.474 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/trancevibrator.ko 2024-01-11T07:56:18.474 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/usb3503.ko 2024-01-11T07:56:18.474 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/usblcd.ko 2024-01-11T07:56:18.474 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/usbsevseg.ko 2024-01-11T07:56:18.474 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/uss720.ko 2024-01-11T07:56:18.475 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/yurex.ko 2024-01-11T07:56:18.475 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/phy 2024-01-11T07:56:18.475 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/phy/phy-generic.ko 2024-01-11T07:56:18.475 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial 2024-01-11T07:56:18.475 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/aircable.ko 2024-01-11T07:56:18.475 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/ark3116.ko 2024-01-11T07:56:18.475 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/belkin_sa.ko 2024-01-11T07:56:18.475 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/ch341.ko 2024-01-11T07:56:18.475 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/cp210x.ko 2024-01-11T07:56:18.475 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/cyberjack.ko 2024-01-11T07:56:18.475 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/cypress_m8.ko 2024-01-11T07:56:18.475 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/digi_acceleport.ko 2024-01-11T07:56:18.475 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/empeg.ko 2024-01-11T07:56:18.476 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/ftdi_sio.ko 2024-01-11T07:56:18.476 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/garmin_gps.ko 2024-01-11T07:56:18.476 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/io_edgeport.ko 2024-01-11T07:56:18.476 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/io_ti.ko 2024-01-11T07:56:18.476 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/ipaq.ko 2024-01-11T07:56:18.476 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/ipw.ko 2024-01-11T07:56:18.476 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/ir-usb.ko 2024-01-11T07:56:18.476 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-01-11T07:56:18.476 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/keyspan.ko 2024-01-11T07:56:18.476 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/keyspan_pda.ko 2024-01-11T07:56:18.476 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/kl5kusb105.ko 2024-01-11T07:56:18.476 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/kobil_sct.ko 2024-01-11T07:56:18.477 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/mct_u232.ko 2024-01-11T07:56:18.477 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/mos7720.ko 2024-01-11T07:56:18.477 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/mos7840.ko 2024-01-11T07:56:18.477 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/navman.ko 2024-01-11T07:56:18.477 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/omninet.ko 2024-01-11T07:56:18.477 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/opticon.ko 2024-01-11T07:56:18.477 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/option.ko 2024-01-11T07:56:18.477 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/oti6858.ko 2024-01-11T07:56:18.477 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/pl2303.ko 2024-01-11T07:56:18.477 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/qcaux.ko 2024-01-11T07:56:18.477 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/qcserial.ko 2024-01-11T07:56:18.477 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/quatech2.ko 2024-01-11T07:56:18.478 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/safe_serial.ko 2024-01-11T07:56:18.478 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/sierra.ko 2024-01-11T07:56:18.478 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/spcp8x5.ko 2024-01-11T07:56:18.478 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/ssu100.ko 2024-01-11T07:56:18.478 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/symbolserial.ko 2024-01-11T07:56:18.478 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-01-11T07:56:18.478 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-01-11T07:56:18.478 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/usb_debug.ko 2024-01-11T07:56:18.478 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/usb_wwan.ko 2024-01-11T07:56:18.478 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/visor.ko 2024-01-11T07:56:18.478 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/whiteheat.ko 2024-01-11T07:56:18.478 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/xsens_mt.ko 2024-01-11T07:56:18.478 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage 2024-01-11T07:56:18.478 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/uas.ko 2024-01-11T07:56:18.479 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-alauda.ko 2024-01-11T07:56:18.479 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-cypress.ko 2024-01-11T07:56:18.479 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-datafab.ko 2024-01-11T07:56:18.479 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-01-11T07:56:18.479 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-freecom.ko 2024-01-11T07:56:18.479 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-isd200.ko 2024-01-11T07:56:18.479 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-01-11T07:56:18.479 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-karma.ko 2024-01-11T07:56:18.479 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-onetouch.ko 2024-01-11T07:56:18.479 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-realtek.ko 2024-01-11T07:56:18.479 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-sddr09.ko 2024-01-11T07:56:18.479 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-sddr55.ko 2024-01-11T07:56:18.480 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-usbat.ko 2024-01-11T07:56:18.480 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/usb-storage.ko 2024-01-11T07:56:18.480 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/usbip 2024-01-11T07:56:18.480 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/usbip/usbip-core.ko 2024-01-11T07:56:18.480 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/usbip/usbip-host.ko 2024-01-11T07:56:18.480 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-01-11T07:56:18.480 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vfio 2024-01-11T07:56:18.480 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vfio/pci 2024-01-11T07:56:18.480 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-01-11T07:56:18.480 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vfio/pci/vfio-pci.ko 2024-01-11T07:56:18.480 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vfio/vfio.ko 2024-01-11T07:56:18.481 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-01-11T07:56:18.481 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vhost 2024-01-11T07:56:18.481 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vhost/vhost.ko 2024-01-11T07:56:18.481 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vhost/vhost_iotlb.ko 2024-01-11T07:56:18.481 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vhost/vhost_net.ko 2024-01-11T07:56:18.481 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vhost/vhost_scsi.ko 2024-01-11T07:56:18.481 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/video 2024-01-11T07:56:18.481 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/video/backlight 2024-01-11T07:56:18.481 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/video/backlight/apple_bl.ko 2024-01-11T07:56:18.481 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/video/backlight/lcd.ko 2024-01-11T07:56:18.481 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/video/backlight/platform_lcd.ko 2024-01-11T07:56:18.481 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/virtio 2024-01-11T07:56:18.481 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/virtio/virtio.ko 2024-01-11T07:56:18.482 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/virtio/virtio_balloon.ko 2024-01-11T07:56:18.482 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/virtio/virtio_mmio.ko 2024-01-11T07:56:18.482 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/virtio/virtio_pci.ko 2024-01-11T07:56:18.482 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-01-11T07:56:18.482 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-01-11T07:56:18.482 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/virtio/virtio_ring.ko 2024-01-11T07:56:18.482 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1 2024-01-11T07:56:18.482 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/masters 2024-01-11T07:56:18.482 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/masters/ds2482.ko 2024-01-11T07:56:18.482 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/masters/ds2490.ko 2024-01-11T07:56:18.482 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves 2024-01-11T07:56:18.483 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-01-11T07:56:18.483 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-01-11T07:56:18.483 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-01-11T07:56:18.483 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-01-11T07:56:18.483 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-01-11T07:56:18.483 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-01-11T07:56:18.483 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-01-11T07:56:18.483 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-01-11T07:56:18.483 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-01-11T07:56:18.483 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_smem.ko 2024-01-11T07:56:18.483 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_therm.ko 2024-01-11T07:56:18.483 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/wire.ko 2024-01-11T07:56:18.484 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog 2024-01-11T07:56:18.484 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/alim1535_wdt.ko 2024-01-11T07:56:18.484 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/alim7101_wdt.ko 2024-01-11T07:56:18.484 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/f71808e_wdt.ko 2024-01-11T07:56:18.484 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/hpwdt.ko 2024-01-11T07:56:18.484 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/i6300esb.ko 2024-01-11T07:56:18.484 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-01-11T07:56:18.484 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/iTCO_wdt.ko 2024-01-11T07:56:18.484 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/ib700wdt.ko 2024-01-11T07:56:18.484 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/ibmasr.ko 2024-01-11T07:56:18.484 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-01-11T07:56:18.484 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/it8712f_wdt.ko 2024-01-11T07:56:18.485 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/it87_wdt.ko 2024-01-11T07:56:18.485 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/machzwd.ko 2024-01-11T07:56:18.485 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/nv_tco.ko 2024-01-11T07:56:18.485 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/pcwd_pci.ko 2024-01-11T07:56:18.485 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/pcwd_usb.ko 2024-01-11T07:56:18.485 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-01-11T07:56:18.485 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/sch311x_wdt.ko 2024-01-11T07:56:18.485 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/softdog.ko 2024-01-11T07:56:18.485 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/sp5100_tco.ko 2024-01-11T07:56:18.485 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/via_wdt.ko 2024-01-11T07:56:18.485 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-01-11T07:56:18.485 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/w83877f_wdt.ko 2024-01-11T07:56:18.485 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/w83977f_wdt.ko 2024-01-11T07:56:18.486 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/wdt_pci.ko 2024-01-11T07:56:18.486 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/xen_wdt.ko 2024-01-11T07:56:18.486 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen 2024-01-11T07:56:18.486 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen/xen-acpi-processor.ko 2024-01-11T07:56:18.486 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen/xen-evtchn.ko 2024-01-11T07:56:18.486 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen/xen-gntalloc.ko 2024-01-11T07:56:18.486 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen/xen-gntdev.ko 2024-01-11T07:56:18.486 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen/xen-pciback 2024-01-11T07:56:18.486 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-01-11T07:56:18.486 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen/xen-privcmd.ko 2024-01-11T07:56:18.486 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen/xenfs 2024-01-11T07:56:18.486 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen/xenfs/xenfs.ko 2024-01-11T07:56:18.486 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs 2024-01-11T07:56:18.487 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/9p 2024-01-11T07:56:18.487 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/9p/9p.ko 2024-01-11T07:56:18.487 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/affs 2024-01-11T07:56:18.487 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/affs/affs.ko 2024-01-11T07:56:18.487 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/befs 2024-01-11T07:56:18.487 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/befs/befs.ko 2024-01-11T07:56:18.487 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/binfmt_misc.ko 2024-01-11T07:56:18.487 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/btrfs 2024-01-11T07:56:18.487 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/btrfs/btrfs.ko 2024-01-11T07:56:18.487 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/cachefiles 2024-01-11T07:56:18.487 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/cachefiles/cachefiles.ko 2024-01-11T07:56:18.488 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ceph 2024-01-11T07:56:18.488 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ceph/ceph.ko 2024-01-11T07:56:18.488 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/coda 2024-01-11T07:56:18.488 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/coda/coda.ko 2024-01-11T07:56:18.488 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/cramfs 2024-01-11T07:56:18.488 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/cramfs/cramfs.ko 2024-01-11T07:56:18.488 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/dlm 2024-01-11T07:56:18.488 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/dlm/dlm.ko 2024-01-11T07:56:18.488 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ecryptfs 2024-01-11T07:56:18.488 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ecryptfs/ecryptfs.ko 2024-01-11T07:56:18.488 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/f2fs 2024-01-11T07:56:18.488 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/f2fs/f2fs.ko 2024-01-11T07:56:18.489 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/fat 2024-01-11T07:56:18.489 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/fat/fat.ko 2024-01-11T07:56:18.489 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/fat/msdos.ko 2024-01-11T07:56:18.489 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/fat/vfat.ko 2024-01-11T07:56:18.489 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/fscache 2024-01-11T07:56:18.489 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/fscache/fscache.ko 2024-01-11T07:56:18.489 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/fuse 2024-01-11T07:56:18.489 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/fuse/cuse.ko 2024-01-11T07:56:18.489 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/fuse/fuse.ko 2024-01-11T07:56:18.489 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/gfs2 2024-01-11T07:56:18.489 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/gfs2/gfs2.ko 2024-01-11T07:56:18.489 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/hfs 2024-01-11T07:56:18.489 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/hfs/hfs.ko 2024-01-11T07:56:18.490 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/hfsplus 2024-01-11T07:56:18.490 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/hfsplus/hfsplus.ko 2024-01-11T07:56:18.490 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/isofs 2024-01-11T07:56:18.490 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/isofs/isofs.ko 2024-01-11T07:56:18.490 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/jfs 2024-01-11T07:56:18.490 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/jfs/jfs.ko 2024-01-11T07:56:18.490 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/lockd 2024-01-11T07:56:18.490 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/lockd/lockd.ko 2024-01-11T07:56:18.490 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/minix 2024-01-11T07:56:18.490 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/minix/minix.ko 2024-01-11T07:56:18.490 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/netfs 2024-01-11T07:56:18.490 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/netfs/netfs.ko 2024-01-11T07:56:18.490 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs 2024-01-11T07:56:18.491 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs/blocklayout 2024-01-11T07:56:18.491 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-01-11T07:56:18.491 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs/filelayout 2024-01-11T07:56:18.491 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-01-11T07:56:18.491 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs/flexfilelayout 2024-01-11T07:56:18.491 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-01-11T07:56:18.491 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs/nfs.ko 2024-01-11T07:56:18.491 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs/nfsv3.ko 2024-01-11T07:56:18.491 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs/nfsv4.ko 2024-01-11T07:56:18.491 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs_common 2024-01-11T07:56:18.491 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs_common/grace.ko 2024-01-11T07:56:18.491 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs_common/nfs_acl.ko 2024-01-11T07:56:18.492 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfsd 2024-01-11T07:56:18.492 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfsd/nfsd.ko 2024-01-11T07:56:18.492 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nilfs2 2024-01-11T07:56:18.492 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nilfs2/nilfs2.ko 2024-01-11T07:56:18.492 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls 2024-01-11T07:56:18.492 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-celtic.ko 2024-01-11T07:56:18.492 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-centeuro.ko 2024-01-11T07:56:18.492 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-croatian.ko 2024-01-11T07:56:18.492 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-cyrillic.ko 2024-01-11T07:56:18.492 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-gaelic.ko 2024-01-11T07:56:18.492 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-greek.ko 2024-01-11T07:56:18.492 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-iceland.ko 2024-01-11T07:56:18.492 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-inuit.ko 2024-01-11T07:56:18.493 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-roman.ko 2024-01-11T07:56:18.493 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-romanian.ko 2024-01-11T07:56:18.493 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-turkish.ko 2024-01-11T07:56:18.493 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp1250.ko 2024-01-11T07:56:18.493 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp1251.ko 2024-01-11T07:56:18.493 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp1255.ko 2024-01-11T07:56:18.493 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp737.ko 2024-01-11T07:56:18.493 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp775.ko 2024-01-11T07:56:18.493 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp850.ko 2024-01-11T07:56:18.493 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp852.ko 2024-01-11T07:56:18.493 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp855.ko 2024-01-11T07:56:18.493 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp857.ko 2024-01-11T07:56:18.493 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp860.ko 2024-01-11T07:56:18.494 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp861.ko 2024-01-11T07:56:18.494 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp862.ko 2024-01-11T07:56:18.494 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp863.ko 2024-01-11T07:56:18.494 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp864.ko 2024-01-11T07:56:18.494 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp865.ko 2024-01-11T07:56:18.494 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp866.ko 2024-01-11T07:56:18.494 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp869.ko 2024-01-11T07:56:18.494 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp874.ko 2024-01-11T07:56:18.494 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp932.ko 2024-01-11T07:56:18.494 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp936.ko 2024-01-11T07:56:18.494 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp949.ko 2024-01-11T07:56:18.494 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp950.ko 2024-01-11T07:56:18.494 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_euc-jp.ko 2024-01-11T07:56:18.495 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-1.ko 2024-01-11T07:56:18.495 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-13.ko 2024-01-11T07:56:18.495 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-14.ko 2024-01-11T07:56:18.495 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-15.ko 2024-01-11T07:56:18.495 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-2.ko 2024-01-11T07:56:18.495 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-3.ko 2024-01-11T07:56:18.495 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-4.ko 2024-01-11T07:56:18.495 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-5.ko 2024-01-11T07:56:18.495 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-6.ko 2024-01-11T07:56:18.495 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-7.ko 2024-01-11T07:56:18.495 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-9.ko 2024-01-11T07:56:18.495 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_koi8-r.ko 2024-01-11T07:56:18.495 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_koi8-ru.ko 2024-01-11T07:56:18.496 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_koi8-u.ko 2024-01-11T07:56:18.496 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_ucs2_utils.ko 2024-01-11T07:56:18.496 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_utf8.ko 2024-01-11T07:56:18.496 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2 2024-01-11T07:56:18.496 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/cluster 2024-01-11T07:56:18.496 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-01-11T07:56:18.496 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/dlm 2024-01-11T07:56:18.496 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-01-11T07:56:18.496 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/dlmfs 2024-01-11T07:56:18.496 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-01-11T07:56:18.496 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/ocfs2.ko 2024-01-11T07:56:18.496 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-01-11T07:56:18.497 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-01-11T07:56:18.497 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-01-11T07:56:18.497 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/overlayfs 2024-01-11T07:56:18.497 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/overlayfs/overlay.ko 2024-01-11T07:56:18.497 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/pstore 2024-01-11T07:56:18.497 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/pstore/ramoops.ko 2024-01-11T07:56:18.497 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/reiserfs 2024-01-11T07:56:18.497 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/reiserfs/reiserfs.ko 2024-01-11T07:56:18.497 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/romfs 2024-01-11T07:56:18.497 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/romfs/romfs.ko 2024-01-11T07:56:18.497 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/smb 2024-01-11T07:56:18.497 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/smb/client 2024-01-11T07:56:18.497 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/smb/client/cifs.ko 2024-01-11T07:56:18.498 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/smb/common 2024-01-11T07:56:18.498 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/smb/common/cifs_arc4.ko 2024-01-11T07:56:18.498 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/smb/common/cifs_md4.ko 2024-01-11T07:56:18.498 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/squashfs 2024-01-11T07:56:18.498 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/squashfs/squashfs.ko 2024-01-11T07:56:18.498 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/sysv 2024-01-11T07:56:18.498 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/sysv/sysv.ko 2024-01-11T07:56:18.498 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/udf 2024-01-11T07:56:18.498 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/udf/udf.ko 2024-01-11T07:56:18.498 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ufs 2024-01-11T07:56:18.498 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ufs/ufs.ko 2024-01-11T07:56:18.498 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/xfs 2024-01-11T07:56:18.498 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/xfs/xfs.ko 2024-01-11T07:56:18.499 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/kernel 2024-01-11T07:56:18.499 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/kernel/rcu 2024-01-11T07:56:18.499 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/kernel/rcu/rcutorture.ko 2024-01-11T07:56:18.499 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/kernel/torture.ko 2024-01-11T07:56:18.499 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/kernel/trace 2024-01-11T07:56:18.499 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-01-11T07:56:18.499 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib 2024-01-11T07:56:18.499 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/asn1_encoder.ko 2024-01-11T07:56:18.499 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/crc-itu-t.ko 2024-01-11T07:56:18.499 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/crc8.ko 2024-01-11T07:56:18.499 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/crypto 2024-01-11T07:56:18.500 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/crypto/libarc4.ko 2024-01-11T07:56:18.500 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/crypto/libdes.ko 2024-01-11T07:56:18.500 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/libcrc32c.ko 2024-01-11T07:56:18.500 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/lru_cache.ko 2024-01-11T07:56:18.500 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/lz4 2024-01-11T07:56:18.500 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/lz4/lz4_compress.ko 2024-01-11T07:56:18.500 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/lz4/lz4hc_compress.ko 2024-01-11T07:56:18.500 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/math 2024-01-11T07:56:18.500 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/math/cordic.ko 2024-01-11T07:56:18.500 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/raid6 2024-01-11T07:56:18.500 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/raid6/raid6_pq.ko 2024-01-11T07:56:18.500 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/reed_solomon 2024-01-11T07:56:18.500 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/reed_solomon/reed_solomon.ko 2024-01-11T07:56:18.501 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/ts_bm.ko 2024-01-11T07:56:18.501 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/ts_fsm.ko 2024-01-11T07:56:18.501 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/ts_kmp.ko 2024-01-11T07:56:18.501 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/zstd 2024-01-11T07:56:18.501 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/zstd/zstd_compress.ko 2024-01-11T07:56:18.501 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/mm 2024-01-11T07:56:18.501 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/mm/hwpoison-inject.ko 2024-01-11T07:56:18.501 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net 2024-01-11T07:56:18.501 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/6lowpan 2024-01-11T07:56:18.501 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/6lowpan/6lowpan.ko 2024-01-11T07:56:18.501 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/6lowpan/nhc_dest.ko 2024-01-11T07:56:18.501 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/6lowpan/nhc_fragment.ko 2024-01-11T07:56:18.501 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/6lowpan/nhc_hop.ko 2024-01-11T07:56:18.502 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/6lowpan/nhc_ipv6.ko 2024-01-11T07:56:18.502 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/6lowpan/nhc_mobility.ko 2024-01-11T07:56:18.502 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/6lowpan/nhc_routing.ko 2024-01-11T07:56:18.502 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/6lowpan/nhc_udp.ko 2024-01-11T07:56:18.502 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/802 2024-01-11T07:56:18.502 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/802/garp.ko 2024-01-11T07:56:18.502 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/802/mrp.ko 2024-01-11T07:56:18.502 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/802/p8022.ko 2024-01-11T07:56:18.502 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/802/psnap.ko 2024-01-11T07:56:18.502 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/802/stp.ko 2024-01-11T07:56:18.502 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/8021q 2024-01-11T07:56:18.503 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/8021q/8021q.ko 2024-01-11T07:56:18.503 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/9p 2024-01-11T07:56:18.503 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/9p/9pnet.ko 2024-01-11T07:56:18.503 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/9p/9pnet_fd.ko 2024-01-11T07:56:18.503 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/9p/9pnet_rdma.ko 2024-01-11T07:56:18.503 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/9p/9pnet_virtio.ko 2024-01-11T07:56:18.503 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/batman-adv 2024-01-11T07:56:18.503 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/batman-adv/batman-adv.ko 2024-01-11T07:56:18.503 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge 2024-01-11T07:56:18.503 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/br_netfilter.ko 2024-01-11T07:56:18.503 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/bridge.ko 2024-01-11T07:56:18.503 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter 2024-01-11T07:56:18.503 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-01-11T07:56:18.504 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_among.ko 2024-01-11T07:56:18.504 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_arp.ko 2024-01-11T07:56:18.504 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-01-11T07:56:18.504 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-01-11T07:56:18.504 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_ip.ko 2024-01-11T07:56:18.504 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-01-11T07:56:18.504 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_limit.ko 2024-01-11T07:56:18.504 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_log.ko 2024-01-11T07:56:18.504 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_mark.ko 2024-01-11T07:56:18.504 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-01-11T07:56:18.505 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-01-11T07:56:18.505 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-01-11T07:56:18.505 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-01-11T07:56:18.505 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_snat.ko 2024-01-11T07:56:18.505 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_stp.ko 2024-01-11T07:56:18.505 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-01-11T07:56:18.505 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-01-11T07:56:18.505 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-01-11T07:56:18.505 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-01-11T07:56:18.505 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebtables.ko 2024-01-11T07:56:18.505 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-01-11T07:56:18.506 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-01-11T07:56:18.506 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-01-11T07:56:18.506 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ceph 2024-01-11T07:56:18.506 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ceph/libceph.ko 2024-01-11T07:56:18.506 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/core 2024-01-11T07:56:18.506 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/core/failover.ko 2024-01-11T07:56:18.506 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/core/pktgen.ko 2024-01-11T07:56:18.506 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dccp 2024-01-11T07:56:18.506 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dccp/dccp.ko 2024-01-11T07:56:18.506 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dccp/dccp_diag.ko 2024-01-11T07:56:18.506 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dccp/dccp_ipv4.ko 2024-01-11T07:56:18.506 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dccp/dccp_ipv6.ko 2024-01-11T07:56:18.506 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dns_resolver 2024-01-11T07:56:18.507 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dns_resolver/dns_resolver.ko 2024-01-11T07:56:18.507 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dsa 2024-01-11T07:56:18.507 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dsa/dsa_core.ko 2024-01-11T07:56:18.507 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dsa/tag_brcm.ko 2024-01-11T07:56:18.507 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dsa/tag_dsa.ko 2024-01-11T07:56:18.507 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dsa/tag_none.ko 2024-01-11T07:56:18.507 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dsa/tag_trailer.ko 2024-01-11T07:56:18.507 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ieee802154 2024-01-11T07:56:18.507 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ieee802154/6lowpan 2024-01-11T07:56:18.507 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-01-11T07:56:18.508 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ieee802154/ieee802154.ko 2024-01-11T07:56:18.508 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ieee802154/ieee802154_socket.ko 2024-01-11T07:56:18.508 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4 2024-01-11T07:56:18.508 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/ah4.ko 2024-01-11T07:56:18.508 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/esp4.ko 2024-01-11T07:56:18.508 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/fou.ko 2024-01-11T07:56:18.508 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/gre.ko 2024-01-11T07:56:18.508 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/inet_diag.ko 2024-01-11T07:56:18.508 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/ip_gre.ko 2024-01-11T07:56:18.508 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/ip_tunnel.ko 2024-01-11T07:56:18.508 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/ip_vti.ko 2024-01-11T07:56:18.509 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/ipcomp.ko 2024-01-11T07:56:18.509 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/ipip.ko 2024-01-11T07:56:18.509 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter 2024-01-11T07:56:18.509 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/arp_tables.ko 2024-01-11T07:56:18.509 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-01-11T07:56:18.509 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-01-11T07:56:18.509 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-01-11T07:56:18.509 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-01-11T07:56:18.510 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-01-11T07:56:18.510 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-01-11T07:56:18.510 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-01-11T07:56:18.510 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-01-11T07:56:18.510 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-01-11T07:56:18.510 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/iptable_security.ko 2024-01-11T07:56:18.510 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-01-11T07:56:18.510 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-01-11T07:56:18.510 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-01-11T07:56:18.510 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-01-11T07:56:18.510 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-01-11T07:56:18.511 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-01-11T07:56:18.511 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-01-11T07:56:18.511 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-01-11T07:56:18.511 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-01-11T07:56:18.511 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-01-11T07:56:18.511 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_bic.ko 2024-01-11T07:56:18.511 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_dctcp.ko 2024-01-11T07:56:18.511 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_diag.ko 2024-01-11T07:56:18.512 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_highspeed.ko 2024-01-11T07:56:18.512 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_htcp.ko 2024-01-11T07:56:18.512 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_hybla.ko 2024-01-11T07:56:18.512 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_illinois.ko 2024-01-11T07:56:18.512 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_lp.ko 2024-01-11T07:56:18.512 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_scalable.ko 2024-01-11T07:56:18.512 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_vegas.ko 2024-01-11T07:56:18.512 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_veno.ko 2024-01-11T07:56:18.512 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_westwood.ko 2024-01-11T07:56:18.512 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_yeah.ko 2024-01-11T07:56:18.512 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tunnel4.ko 2024-01-11T07:56:18.512 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/udp_diag.ko 2024-01-11T07:56:18.513 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/udp_tunnel.ko 2024-01-11T07:56:18.513 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/xfrm4_tunnel.ko 2024-01-11T07:56:18.513 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6 2024-01-11T07:56:18.513 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/ah6.ko 2024-01-11T07:56:18.513 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/esp6.ko 2024-01-11T07:56:18.513 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/fou6.ko 2024-01-11T07:56:18.513 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/ip6_tunnel.ko 2024-01-11T07:56:18.513 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-01-11T07:56:18.513 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/ip6_vti.ko 2024-01-11T07:56:18.513 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/ipcomp6.ko 2024-01-11T07:56:18.513 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter 2024-01-11T07:56:18.514 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-01-11T07:56:18.514 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-01-11T07:56:18.514 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-01-11T07:56:18.514 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-01-11T07:56:18.514 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-01-11T07:56:18.514 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-01-11T07:56:18.514 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-01-11T07:56:18.514 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-01-11T07:56:18.514 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-01-11T07:56:18.514 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-01-11T07:56:18.514 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-01-11T07:56:18.514 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-01-11T07:56:18.515 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-01-11T07:56:18.515 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-01-11T07:56:18.515 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-01-11T07:56:18.515 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-01-11T07:56:18.515 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-01-11T07:56:18.515 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-01-11T07:56:18.515 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-01-11T07:56:18.515 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-01-11T07:56:18.515 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-01-11T07:56:18.515 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-01-11T07:56:18.515 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-01-11T07:56:18.515 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-01-11T07:56:18.516 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/sit.ko 2024-01-11T07:56:18.516 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/tunnel6.ko 2024-01-11T07:56:18.516 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/xfrm6_tunnel.ko 2024-01-11T07:56:18.516 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/key 2024-01-11T07:56:18.516 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/key/af_key.ko 2024-01-11T07:56:18.516 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/l2tp 2024-01-11T07:56:18.516 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/l2tp/l2tp_core.ko 2024-01-11T07:56:18.516 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/l2tp/l2tp_debugfs.ko 2024-01-11T07:56:18.516 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/l2tp/l2tp_eth.ko 2024-01-11T07:56:18.516 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/l2tp/l2tp_ip.ko 2024-01-11T07:56:18.516 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/l2tp/l2tp_ip6.ko 2024-01-11T07:56:18.516 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/l2tp/l2tp_netlink.ko 2024-01-11T07:56:18.516 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/l2tp/l2tp_ppp.ko 2024-01-11T07:56:18.517 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/llc 2024-01-11T07:56:18.517 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/llc/llc.ko 2024-01-11T07:56:18.517 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/mac802154 2024-01-11T07:56:18.517 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/mac802154/mac802154.ko 2024-01-11T07:56:18.517 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/mpls 2024-01-11T07:56:18.517 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/mpls/mpls_gso.ko 2024-01-11T07:56:18.517 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter 2024-01-11T07:56:18.517 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset 2024-01-11T07:56:18.517 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set.ko 2024-01-11T07:56:18.517 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-01-11T07:56:18.517 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-01-11T07:56:18.517 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-01-11T07:56:18.518 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-01-11T07:56:18.518 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-01-11T07:56:18.518 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-01-11T07:56:18.518 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-01-11T07:56:18.518 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-01-11T07:56:18.518 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-01-11T07:56:18.518 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-01-11T07:56:18.518 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-01-11T07:56:18.518 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-01-11T07:56:18.518 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-01-11T07:56:18.518 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-01-11T07:56:18.518 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-01-11T07:56:18.519 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs 2024-01-11T07:56:18.519 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs.ko 2024-01-11T07:56:18.519 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-01-11T07:56:18.519 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-01-11T07:56:18.519 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-01-11T07:56:18.519 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-01-11T07:56:18.519 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-01-11T07:56:18.519 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-01-11T07:56:18.519 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-01-11T07:56:18.519 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-01-11T07:56:18.519 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-01-11T07:56:18.519 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-01-11T07:56:18.519 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-01-11T07:56:18.520 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-01-11T07:56:18.520 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-01-11T07:56:18.520 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conncount.ko 2024-01-11T07:56:18.520 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack.ko 2024-01-11T07:56:18.520 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-01-11T07:56:18.520 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-01-11T07:56:18.520 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-01-11T07:56:18.520 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_h323.ko 2024-01-11T07:56:18.520 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_irc.ko 2024-01-11T07:56:18.520 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-01-11T07:56:18.520 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-01-11T07:56:18.521 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-01-11T07:56:18.521 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_sane.ko 2024-01-11T07:56:18.521 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_sip.ko 2024-01-11T07:56:18.521 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-01-11T07:56:18.521 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-01-11T07:56:18.521 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_dup_netdev.ko 2024-01-11T07:56:18.521 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_flow_table.ko 2024-01-11T07:56:18.521 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_flow_table_inet.ko 2024-01-11T07:56:18.521 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_log_syslog.ko 2024-01-11T07:56:18.521 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_nat.ko 2024-01-11T07:56:18.521 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_nat_amanda.ko 2024-01-11T07:56:18.521 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_nat_ftp.ko 2024-01-11T07:56:18.521 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_nat_irc.ko 2024-01-11T07:56:18.522 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_nat_sip.ko 2024-01-11T07:56:18.522 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_nat_tftp.ko 2024-01-11T07:56:18.522 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_synproxy_core.ko 2024-01-11T07:56:18.522 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_tables.ko 2024-01-11T07:56:18.522 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nfnetlink.ko 2024-01-11T07:56:18.522 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nfnetlink_acct.ko 2024-01-11T07:56:18.522 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nfnetlink_log.ko 2024-01-11T07:56:18.522 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nfnetlink_osf.ko 2024-01-11T07:56:18.522 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nfnetlink_queue.ko 2024-01-11T07:56:18.522 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_chain_nat.ko 2024-01-11T07:56:18.522 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_compat.ko 2024-01-11T07:56:18.522 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_ct.ko 2024-01-11T07:56:18.523 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_dup_netdev.ko 2024-01-11T07:56:18.523 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_fib.ko 2024-01-11T07:56:18.523 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_fwd_netdev.ko 2024-01-11T07:56:18.523 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_hash.ko 2024-01-11T07:56:18.523 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_limit.ko 2024-01-11T07:56:18.523 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_log.ko 2024-01-11T07:56:18.523 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_masq.ko 2024-01-11T07:56:18.523 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_nat.ko 2024-01-11T07:56:18.523 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_numgen.ko 2024-01-11T07:56:18.523 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_queue.ko 2024-01-11T07:56:18.523 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_quota.ko 2024-01-11T07:56:18.523 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_redir.ko 2024-01-11T07:56:18.524 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_reject.ko 2024-01-11T07:56:18.524 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_reject_inet.ko 2024-01-11T07:56:18.524 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_socket.ko 2024-01-11T07:56:18.524 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_tunnel.ko 2024-01-11T07:56:18.524 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_AUDIT.ko 2024-01-11T07:56:18.524 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_CHECKSUM.ko 2024-01-11T07:56:18.524 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_CLASSIFY.ko 2024-01-11T07:56:18.524 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-01-11T07:56:18.524 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_CT.ko 2024-01-11T07:56:18.524 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_DSCP.ko 2024-01-11T07:56:18.524 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_HL.ko 2024-01-11T07:56:18.524 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_HMARK.ko 2024-01-11T07:56:18.524 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_IDLETIMER.ko 2024-01-11T07:56:18.525 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_LED.ko 2024-01-11T07:56:18.525 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_LOG.ko 2024-01-11T07:56:18.525 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_MASQUERADE.ko 2024-01-11T07:56:18.525 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_NETMAP.ko 2024-01-11T07:56:18.525 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_NFLOG.ko 2024-01-11T07:56:18.525 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_NFQUEUE.ko 2024-01-11T07:56:18.525 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_RATEEST.ko 2024-01-11T07:56:18.525 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_REDIRECT.ko 2024-01-11T07:56:18.525 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_SECMARK.ko 2024-01-11T07:56:18.525 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_TCPMSS.ko 2024-01-11T07:56:18.525 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-01-11T07:56:18.525 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_TEE.ko 2024-01-11T07:56:18.526 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_TPROXY.ko 2024-01-11T07:56:18.526 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_TRACE.ko 2024-01-11T07:56:18.526 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_addrtype.ko 2024-01-11T07:56:18.526 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_bpf.ko 2024-01-11T07:56:18.526 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_cgroup.ko 2024-01-11T07:56:18.526 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_cluster.ko 2024-01-11T07:56:18.526 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_comment.ko 2024-01-11T07:56:18.526 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_connbytes.ko 2024-01-11T07:56:18.526 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_connlabel.ko 2024-01-11T07:56:18.526 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_connlimit.ko 2024-01-11T07:56:18.526 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_connmark.ko 2024-01-11T07:56:18.526 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_conntrack.ko 2024-01-11T07:56:18.526 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_cpu.ko 2024-01-11T07:56:18.527 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_dccp.ko 2024-01-11T07:56:18.527 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_devgroup.ko 2024-01-11T07:56:18.527 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_dscp.ko 2024-01-11T07:56:18.527 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_ecn.ko 2024-01-11T07:56:18.527 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_esp.ko 2024-01-11T07:56:18.527 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_hashlimit.ko 2024-01-11T07:56:18.527 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_helper.ko 2024-01-11T07:56:18.527 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_hl.ko 2024-01-11T07:56:18.527 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_ipcomp.ko 2024-01-11T07:56:18.527 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_iprange.ko 2024-01-11T07:56:18.527 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_ipvs.ko 2024-01-11T07:56:18.527 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_l2tp.ko 2024-01-11T07:56:18.527 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_length.ko 2024-01-11T07:56:18.528 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_limit.ko 2024-01-11T07:56:18.528 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_mac.ko 2024-01-11T07:56:18.528 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_mark.ko 2024-01-11T07:56:18.528 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_multiport.ko 2024-01-11T07:56:18.528 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_nat.ko 2024-01-11T07:56:18.528 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_nfacct.ko 2024-01-11T07:56:18.528 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_osf.ko 2024-01-11T07:56:18.528 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_owner.ko 2024-01-11T07:56:18.528 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_physdev.ko 2024-01-11T07:56:18.528 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_pkttype.ko 2024-01-11T07:56:18.528 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_policy.ko 2024-01-11T07:56:18.528 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_quota.ko 2024-01-11T07:56:18.529 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_rateest.ko 2024-01-11T07:56:18.529 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_realm.ko 2024-01-11T07:56:18.529 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_recent.ko 2024-01-11T07:56:18.529 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_sctp.ko 2024-01-11T07:56:18.529 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_set.ko 2024-01-11T07:56:18.529 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_socket.ko 2024-01-11T07:56:18.529 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_state.ko 2024-01-11T07:56:18.529 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_statistic.ko 2024-01-11T07:56:18.529 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_string.ko 2024-01-11T07:56:18.529 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_tcpmss.ko 2024-01-11T07:56:18.529 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_time.ko 2024-01-11T07:56:18.529 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_u32.ko 2024-01-11T07:56:18.530 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netlink 2024-01-11T07:56:18.530 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netlink/netlink_diag.ko 2024-01-11T07:56:18.530 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/nsh 2024-01-11T07:56:18.530 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/nsh/nsh.ko 2024-01-11T07:56:18.530 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/openvswitch 2024-01-11T07:56:18.530 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/openvswitch/openvswitch.ko 2024-01-11T07:56:18.530 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/openvswitch/vport-geneve.ko 2024-01-11T07:56:18.530 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/openvswitch/vport-gre.ko 2024-01-11T07:56:18.530 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/openvswitch/vport-vxlan.ko 2024-01-11T07:56:18.530 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/packet 2024-01-11T07:56:18.530 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/packet/af_packet_diag.ko 2024-01-11T07:56:18.530 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/rds 2024-01-11T07:56:18.531 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/rds/rds.ko 2024-01-11T07:56:18.531 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/rds/rds_rdma.ko 2024-01-11T07:56:18.531 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/rds/rds_tcp.ko 2024-01-11T07:56:18.531 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched 2024-01-11T07:56:18.531 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_csum.ko 2024-01-11T07:56:18.531 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_gact.ko 2024-01-11T07:56:18.531 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_ipt.ko 2024-01-11T07:56:18.531 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_mirred.ko 2024-01-11T07:56:18.531 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_nat.ko 2024-01-11T07:56:18.531 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_pedit.ko 2024-01-11T07:56:18.531 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_police.ko 2024-01-11T07:56:18.531 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_simple.ko 2024-01-11T07:56:18.531 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_skbedit.ko 2024-01-11T07:56:18.532 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_vlan.ko 2024-01-11T07:56:18.532 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/cls_basic.ko 2024-01-11T07:56:18.532 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/cls_bpf.ko 2024-01-11T07:56:18.532 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/cls_flow.ko 2024-01-11T07:56:18.532 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/cls_fw.ko 2024-01-11T07:56:18.532 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/cls_route.ko 2024-01-11T07:56:18.532 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/cls_u32.ko 2024-01-11T07:56:18.532 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/em_cmp.ko 2024-01-11T07:56:18.532 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/em_ipset.ko 2024-01-11T07:56:18.532 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/em_meta.ko 2024-01-11T07:56:18.532 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/em_nbyte.ko 2024-01-11T07:56:18.532 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/em_text.ko 2024-01-11T07:56:18.533 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/em_u32.ko 2024-01-11T07:56:18.533 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_choke.ko 2024-01-11T07:56:18.533 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_codel.ko 2024-01-11T07:56:18.533 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_drr.ko 2024-01-11T07:56:18.533 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_fq.ko 2024-01-11T07:56:18.533 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_gred.ko 2024-01-11T07:56:18.533 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_hfsc.ko 2024-01-11T07:56:18.533 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_hhf.ko 2024-01-11T07:56:18.533 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_htb.ko 2024-01-11T07:56:18.533 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_ingress.ko 2024-01-11T07:56:18.533 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_mqprio.ko 2024-01-11T07:56:18.533 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_mqprio_lib.ko 2024-01-11T07:56:18.534 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_multiq.ko 2024-01-11T07:56:18.534 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_netem.ko 2024-01-11T07:56:18.534 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_pie.ko 2024-01-11T07:56:18.534 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_plug.ko 2024-01-11T07:56:18.534 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_prio.ko 2024-01-11T07:56:18.534 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_qfq.ko 2024-01-11T07:56:18.534 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_red.ko 2024-01-11T07:56:18.534 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_sfb.ko 2024-01-11T07:56:18.534 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_sfq.ko 2024-01-11T07:56:18.534 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_tbf.ko 2024-01-11T07:56:18.534 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_teql.ko 2024-01-11T07:56:18.534 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sctp 2024-01-11T07:56:18.534 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sctp/sctp.ko 2024-01-11T07:56:18.534 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sctp/sctp_diag.ko 2024-01-11T07:56:18.535 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sunrpc 2024-01-11T07:56:18.535 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sunrpc/auth_gss 2024-01-11T07:56:18.535 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-01-11T07:56:18.535 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-01-11T07:56:18.535 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sunrpc/sunrpc.ko 2024-01-11T07:56:18.535 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sunrpc/xprtrdma 2024-01-11T07:56:18.535 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-01-11T07:56:18.535 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/tipc 2024-01-11T07:56:18.535 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/tipc/diag.ko 2024-01-11T07:56:18.535 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/tipc/tipc.ko 2024-01-11T07:56:18.535 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/unix 2024-01-11T07:56:18.535 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/unix/unix_diag.ko 2024-01-11T07:56:18.536 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/vmw_vsock 2024-01-11T07:56:18.536 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-01-11T07:56:18.536 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-01-11T07:56:18.536 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/vmw_vsock/vsock.ko 2024-01-11T07:56:18.536 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/vmw_vsock/vsock_diag.ko 2024-01-11T07:56:18.536 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/vmw_vsock/vsock_loopback.ko 2024-01-11T07:56:18.536 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/xfrm 2024-01-11T07:56:18.536 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/xfrm/xfrm_ipcomp.ko 2024-01-11T07:56:18.536 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/security 2024-01-11T07:56:18.536 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/security/keys 2024-01-11T07:56:18.536 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/security/keys/encrypted-keys 2024-01-11T07:56:18.536 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-01-11T07:56:18.537 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/security/keys/trusted-keys 2024-01-11T07:56:18.537 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/security/keys/trusted-keys/trusted.ko 2024-01-11T07:56:18.537 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/virt 2024-01-11T07:56:18.537 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/virt/lib 2024-01-11T07:56:18.537 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/virt/lib/irqbypass.ko 2024-01-11T07:56:18.537 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.alias 2024-01-11T07:56:18.537 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.alias.bin 2024-01-11T07:56:18.537 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.builtin 2024-01-11T07:56:18.537 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.builtin.bin 2024-01-11T07:56:18.537 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.builtin.modinfo 2024-01-11T07:56:18.537 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.dep 2024-01-11T07:56:18.537 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.dep.bin 2024-01-11T07:56:18.537 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.devname 2024-01-11T07:56:18.538 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.order 2024-01-11T07:56:18.538 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.softdep 2024-01-11T07:56:18.538 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.symbols 2024-01-11T07:56:18.538 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.symbols.bin 2024-01-11T07:56:18.538 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-gc719ee96ba78 2024-01-11T07:56:18.539 DEBUG:teuthology.orchestra.run.smithi077:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-01-11T07:56:18.563 INFO:teuthology.orchestra.run.smithi077.stdout:postinstall scriptlet (using /bin/sh): 2024-01-11T07:56:18.564 INFO:teuthology.orchestra.run.smithi077.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-gc719ee96ba78 -a -r /boot/System.map-6.6.0-gc719ee96ba78 ]; then 2024-01-11T07:56:18.564 INFO:teuthology.orchestra.run.smithi077.stdout:cp /boot/vmlinuz-6.6.0-gc719ee96ba78 /boot/.vmlinuz-6.6.0-gc719ee96ba78-rpm 2024-01-11T07:56:18.564 INFO:teuthology.orchestra.run.smithi077.stdout:cp /boot/System.map-6.6.0-gc719ee96ba78 /boot/.System.map-6.6.0-gc719ee96ba78-rpm 2024-01-11T07:56:18.564 INFO:teuthology.orchestra.run.smithi077.stdout:rm -f /boot/vmlinuz-6.6.0-gc719ee96ba78 /boot/System.map-6.6.0-gc719ee96ba78 2024-01-11T07:56:18.564 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/installkernel 6.6.0-gc719ee96ba78 /boot/.vmlinuz-6.6.0-gc719ee96ba78-rpm /boot/.System.map-6.6.0-gc719ee96ba78-rpm 2024-01-11T07:56:18.564 INFO:teuthology.orchestra.run.smithi077.stdout:rm -f /boot/.vmlinuz-6.6.0-gc719ee96ba78-rpm /boot/.System.map-6.6.0-gc719ee96ba78-rpm 2024-01-11T07:56:18.564 INFO:teuthology.orchestra.run.smithi077.stdout:fi 2024-01-11T07:56:18.564 INFO:teuthology.orchestra.run.smithi077.stdout:preuninstall scriptlet (using /bin/sh): 2024-01-11T07:56:18.564 INFO:teuthology.orchestra.run.smithi077.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-01-11T07:56:18.564 INFO:teuthology.orchestra.run.smithi077.stdout:new-kernel-pkg --remove 6.6.0-gc719ee96ba78 --rminitrd --initrdfile=/boot/initramfs-6.6.0-gc719ee96ba78.img 2024-01-11T07:56:18.564 INFO:teuthology.orchestra.run.smithi077.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-01-11T07:56:18.564 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-install remove 6.6.0-gc719ee96ba78 2024-01-11T07:56:18.565 INFO:teuthology.orchestra.run.smithi077.stdout:fi 2024-01-11T07:56:18.565 INFO:teuthology.orchestra.run.smithi077.stdout:postuninstall scriptlet (using /bin/sh): 2024-01-11T07:56:18.565 INFO:teuthology.orchestra.run.smithi077.stdout:if [ -x /sbin/update-bootloader ]; then 2024-01-11T07:56:18.565 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/update-bootloader --remove 6.6.0-gc719ee96ba78 2024-01-11T07:56:18.565 INFO:teuthology.orchestra.run.smithi077.stdout:fi 2024-01-11T07:56:18.566 DEBUG:teuthology.orchestra.run.smithi077:> sudo rpm -qi grub2-tools 2024-01-11T07:56:18.640 INFO:teuthology.orchestra.run.smithi077.stdout:Name : grub2-tools 2024-01-11T07:56:18.640 INFO:teuthology.orchestra.run.smithi077.stdout:Epoch : 1 2024-01-11T07:56:18.640 INFO:teuthology.orchestra.run.smithi077.stdout:Version : 2.02 2024-01-11T07:56:18.641 INFO:teuthology.orchestra.run.smithi077.stdout:Release : 129.el8 2024-01-11T07:56:18.641 INFO:teuthology.orchestra.run.smithi077.stdout:Architecture: x86_64 2024-01-11T07:56:18.641 INFO:teuthology.orchestra.run.smithi077.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-11T07:56:18.641 INFO:teuthology.orchestra.run.smithi077.stdout:Group : System Environment/Base 2024-01-11T07:56:18.641 INFO:teuthology.orchestra.run.smithi077.stdout:Size : 9265586 2024-01-11T07:56:18.641 INFO:teuthology.orchestra.run.smithi077.stdout:License : GPLv3+ 2024-01-11T07:56:18.641 INFO:teuthology.orchestra.run.smithi077.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-11T07:56:18.641 INFO:teuthology.orchestra.run.smithi077.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-11T07:56:18.641 INFO:teuthology.orchestra.run.smithi077.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-11T07:56:18.641 INFO:teuthology.orchestra.run.smithi077.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-11T07:56:18.641 INFO:teuthology.orchestra.run.smithi077.stdout:Relocations : (not relocatable) 2024-01-11T07:56:18.641 INFO:teuthology.orchestra.run.smithi077.stdout:Packager : CentOS BuildSystem 2024-01-11T07:56:18.641 INFO:teuthology.orchestra.run.smithi077.stdout:Vendor : CentOS 2024-01-11T07:56:18.642 INFO:teuthology.orchestra.run.smithi077.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-11T07:56:18.642 INFO:teuthology.orchestra.run.smithi077.stdout:Summary : Support tools for GRUB. 2024-01-11T07:56:18.642 INFO:teuthology.orchestra.run.smithi077.stdout:Description : 2024-01-11T07:56:18.642 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T07:56:18.642 INFO:teuthology.orchestra.run.smithi077.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-11T07:56:18.642 INFO:teuthology.orchestra.run.smithi077.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-11T07:56:18.642 INFO:teuthology.orchestra.run.smithi077.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-11T07:56:18.642 INFO:teuthology.orchestra.run.smithi077.stdout:hardware devices. 2024-01-11T07:56:18.642 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T07:56:18.642 INFO:teuthology.orchestra.run.smithi077.stdout:This subpackage provides tools for support of all platforms. 2024-01-11T07:56:18.643 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-11T07:56:18.643 INFO:teuthology.task.kernel:Updating grub on smithi077 to boot 6.6.0-gc719ee96ba78 2024-01-11T07:56:18.643 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-11T07:56:23.060 INFO:teuthology.orchestra.run.smithi077.stderr:Generating grub configuration file ... 2024-01-11T07:56:25.150 INFO:teuthology.orchestra.run.smithi077.stderr:done 2024-01-11T07:56:25.152 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2024-01-11T07:56:25.175 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.QogYOGbgrB 2024-01-11T07:56:25.175 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QogYOGbgrB 2024-01-11T07:56:25.257 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.QogYOGbgrB 2024-01-11T07:56:25.406 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.QogYOGbgrB is 6KB 2024-01-11T07:56:25.455 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.QogYOGbgrB 2024-01-11T07:56:25.472 DEBUG:teuthology.orchestra.run.smithi077:> sudo /bin/ls /boot/loader/entries || true 2024-01-11T07:56:25.537 INFO:teuthology.orchestra.run.smithi077.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-11T07:56:25.537 INFO:teuthology.orchestra.run.smithi077.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-11T07:56:25.537 INFO:teuthology.orchestra.run.smithi077.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-11T07:56:25.537 INFO:teuthology.orchestra.run.smithi077.stdout:a1b29d6c289747bdb22930598d39874a-6.6.0-gc719ee96ba78.conf 2024-01-11T07:56:25.538 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-6.6.0-gc719ee96ba78 2024-01-11T07:56:25.689 DEBUG:teuthology.orchestra.run.smithi077:> sudo shutdown -r now 2024-01-11T07:56:33.382 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi027.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-01-11T07:56:33.383 DEBUG:teuthology.orchestra.run.smithi027:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-01-11T07:56:33.445 INFO:teuthology.orchestra.run.smithi027.stdout:/boot/System.map-6.6.0-gc719ee96ba78 2024-01-11T07:56:33.445 INFO:teuthology.orchestra.run.smithi027.stdout:/boot/config-6.6.0-gc719ee96ba78 2024-01-11T07:56:33.445 INFO:teuthology.orchestra.run.smithi027.stdout:/boot/vmlinuz-6.6.0-gc719ee96ba78 2024-01-11T07:56:33.445 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78 2024-01-11T07:56:33.445 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel 2024-01-11T07:56:33.445 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch 2024-01-11T07:56:33.445 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86 2024-01-11T07:56:33.445 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto 2024-01-11T07:56:33.445 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-01-11T07:56:33.446 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-01-11T07:56:33.446 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-01-11T07:56:33.446 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-01-11T07:56:33.446 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-01-11T07:56:33.446 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-01-11T07:56:33.446 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-01-11T07:56:33.446 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/crc32c-intel.ko 2024-01-11T07:56:33.446 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-01-11T07:56:33.446 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-01-11T07:56:33.446 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-01-11T07:56:33.446 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-01-11T07:56:33.446 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/serpent-avx2.ko 2024-01-11T07:56:33.446 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-01-11T07:56:33.446 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-01-11T07:56:33.446 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-01-11T07:56:33.447 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-01-11T07:56:33.447 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-01-11T07:56:33.447 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-01-11T07:56:33.447 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-01-11T07:56:33.447 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/kernel 2024-01-11T07:56:33.447 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/kernel/cpu 2024-01-11T07:56:33.447 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/kernel/cpu/mce 2024-01-11T07:56:33.447 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-01-11T07:56:33.447 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/kvm 2024-01-11T07:56:33.447 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/kvm/kvm-amd.ko 2024-01-11T07:56:33.447 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/kvm/kvm-intel.ko 2024-01-11T07:56:33.447 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/arch/x86/kvm/kvm.ko 2024-01-11T07:56:33.447 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto 2024-01-11T07:56:33.447 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/ansi_cprng.ko 2024-01-11T07:56:33.448 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/anubis.ko 2024-01-11T07:56:33.448 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/arc4.ko 2024-01-11T07:56:33.448 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/async_tx 2024-01-11T07:56:33.448 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/async_tx/async_memcpy.ko 2024-01-11T07:56:33.448 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/async_tx/async_pq.ko 2024-01-11T07:56:33.448 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/async_tx/async_raid6_recov.ko 2024-01-11T07:56:33.448 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/async_tx/async_tx.ko 2024-01-11T07:56:33.448 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/async_tx/async_xor.ko 2024-01-11T07:56:33.448 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/async_tx/raid6test.ko 2024-01-11T07:56:33.448 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/authenc.ko 2024-01-11T07:56:33.448 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/authencesn.ko 2024-01-11T07:56:33.448 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/blake2b_generic.ko 2024-01-11T07:56:33.448 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/blowfish_common.ko 2024-01-11T07:56:33.448 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/blowfish_generic.ko 2024-01-11T07:56:33.448 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/camellia_generic.ko 2024-01-11T07:56:33.449 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/cast5_generic.ko 2024-01-11T07:56:33.449 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/cast6_generic.ko 2024-01-11T07:56:33.449 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/cast_common.ko 2024-01-11T07:56:33.449 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/ccm.ko 2024-01-11T07:56:33.449 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/cmac.ko 2024-01-11T07:56:33.449 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/crc32_generic.ko 2024-01-11T07:56:33.449 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/crypto_engine.ko 2024-01-11T07:56:33.449 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/crypto_user.ko 2024-01-11T07:56:33.449 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/cts.ko 2024-01-11T07:56:33.449 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/des_generic.ko 2024-01-11T07:56:33.449 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/dh_generic.ko 2024-01-11T07:56:33.449 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/echainiv.ko 2024-01-11T07:56:33.449 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/essiv.ko 2024-01-11T07:56:33.449 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/fcrypt.ko 2024-01-11T07:56:33.450 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/khazad.ko 2024-01-11T07:56:33.450 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/lz4.ko 2024-01-11T07:56:33.450 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/lz4hc.ko 2024-01-11T07:56:33.450 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/md4.ko 2024-01-11T07:56:33.450 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/michael_mic.ko 2024-01-11T07:56:33.450 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/pcbc.ko 2024-01-11T07:56:33.450 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/pcrypt.ko 2024-01-11T07:56:33.450 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/rmd160.ko 2024-01-11T07:56:33.450 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/seed.ko 2024-01-11T07:56:33.450 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/serpent_generic.ko 2024-01-11T07:56:33.450 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/tcrypt.ko 2024-01-11T07:56:33.450 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/tea.ko 2024-01-11T07:56:33.450 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/twofish_common.ko 2024-01-11T07:56:33.450 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/twofish_generic.ko 2024-01-11T07:56:33.450 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/vmac.ko 2024-01-11T07:56:33.451 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/wp512.ko 2024-01-11T07:56:33.451 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/xcbc.ko 2024-01-11T07:56:33.451 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/xor.ko 2024-01-11T07:56:33.451 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/crypto/xxhash_generic.ko 2024-01-11T07:56:33.451 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers 2024-01-11T07:56:33.451 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/acpi 2024-01-11T07:56:33.451 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/acpi/acpi_ipmi.ko 2024-01-11T07:56:33.451 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/acpi/acpi_pad.ko 2024-01-11T07:56:33.451 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/acpi/custom_method.ko 2024-01-11T07:56:33.451 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/acpi/ec_sys.ko 2024-01-11T07:56:33.451 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/acpi/platform_profile.ko 2024-01-11T07:56:33.451 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/acpi/sbs.ko 2024-01-11T07:56:33.451 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/acpi/sbshc.ko 2024-01-11T07:56:33.451 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/acpi/video.ko 2024-01-11T07:56:33.451 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata 2024-01-11T07:56:33.452 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/acard-ahci.ko 2024-01-11T07:56:33.452 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/ahci_platform.ko 2024-01-11T07:56:33.452 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/ata_generic.ko 2024-01-11T07:56:33.452 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/libahci_platform.ko 2024-01-11T07:56:33.452 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_acpi.ko 2024-01-11T07:56:33.452 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_ali.ko 2024-01-11T07:56:33.452 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_amd.ko 2024-01-11T07:56:33.452 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_artop.ko 2024-01-11T07:56:33.452 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_atiixp.ko 2024-01-11T07:56:33.452 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_atp867x.ko 2024-01-11T07:56:33.452 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_cmd640.ko 2024-01-11T07:56:33.452 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_cmd64x.ko 2024-01-11T07:56:33.452 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_cypress.ko 2024-01-11T07:56:33.452 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_efar.ko 2024-01-11T07:56:33.452 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_hpt366.ko 2024-01-11T07:56:33.453 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_hpt37x.ko 2024-01-11T07:56:33.453 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_hpt3x2n.ko 2024-01-11T07:56:33.453 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_hpt3x3.ko 2024-01-11T07:56:33.453 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_it8213.ko 2024-01-11T07:56:33.453 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_it821x.ko 2024-01-11T07:56:33.453 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_jmicron.ko 2024-01-11T07:56:33.453 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_marvell.ko 2024-01-11T07:56:33.453 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_mpiix.ko 2024-01-11T07:56:33.453 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_netcell.ko 2024-01-11T07:56:33.453 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_ninja32.ko 2024-01-11T07:56:33.453 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_ns87410.ko 2024-01-11T07:56:33.453 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_ns87415.ko 2024-01-11T07:56:33.453 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_oldpiix.ko 2024-01-11T07:56:33.453 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_opti.ko 2024-01-11T07:56:33.453 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_optidma.ko 2024-01-11T07:56:33.454 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_pcmcia.ko 2024-01-11T07:56:33.454 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_pdc2027x.ko 2024-01-11T07:56:33.454 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-01-11T07:56:33.454 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_piccolo.ko 2024-01-11T07:56:33.454 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_rdc.ko 2024-01-11T07:56:33.454 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_sch.ko 2024-01-11T07:56:33.454 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_serverworks.ko 2024-01-11T07:56:33.454 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_sil680.ko 2024-01-11T07:56:33.454 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_sis.ko 2024-01-11T07:56:33.454 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_sl82c105.ko 2024-01-11T07:56:33.454 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_triflex.ko 2024-01-11T07:56:33.454 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pata_via.ko 2024-01-11T07:56:33.454 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/pdc_adma.ko 2024-01-11T07:56:33.454 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_inic162x.ko 2024-01-11T07:56:33.454 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_mv.ko 2024-01-11T07:56:33.455 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_nv.ko 2024-01-11T07:56:33.455 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_promise.ko 2024-01-11T07:56:33.455 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_qstor.ko 2024-01-11T07:56:33.455 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_sil.ko 2024-01-11T07:56:33.455 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_sil24.ko 2024-01-11T07:56:33.455 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_sis.ko 2024-01-11T07:56:33.455 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_svw.ko 2024-01-11T07:56:33.455 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_sx4.ko 2024-01-11T07:56:33.455 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_uli.ko 2024-01-11T07:56:33.455 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_via.ko 2024-01-11T07:56:33.455 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ata/sata_vsc.ko 2024-01-11T07:56:33.455 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/auxdisplay 2024-01-11T07:56:33.455 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/auxdisplay/ks0108.ko 2024-01-11T07:56:33.455 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/base 2024-01-11T07:56:33.455 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/base/regmap 2024-01-11T07:56:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/base/regmap/regmap-i2c.ko 2024-01-11T07:56:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/base/regmap/regmap-mmio.ko 2024-01-11T07:56:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/bcma 2024-01-11T07:56:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/bcma/bcma.ko 2024-01-11T07:56:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block 2024-01-11T07:56:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/aoe 2024-01-11T07:56:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/aoe/aoe.ko 2024-01-11T07:56:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/brd.ko 2024-01-11T07:56:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/drbd 2024-01-11T07:56:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/drbd/drbd.ko 2024-01-11T07:56:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/floppy.ko 2024-01-11T07:56:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/loop.ko 2024-01-11T07:56:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/mtip32xx 2024-01-11T07:56:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-01-11T07:56:33.456 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/nbd.ko 2024-01-11T07:56:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/null_blk 2024-01-11T07:56:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/null_blk/null_blk.ko 2024-01-11T07:56:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/pktcdvd.ko 2024-01-11T07:56:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/rbd.ko 2024-01-11T07:56:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/virtio_blk.ko 2024-01-11T07:56:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/xen-blkback 2024-01-11T07:56:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-01-11T07:56:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/xen-blkfront.ko 2024-01-11T07:56:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/zram 2024-01-11T07:56:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/block/zram/zram.ko 2024-01-11T07:56:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char 2024-01-11T07:56:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/hangcheck-timer.ko 2024-01-11T07:56:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/hw_random 2024-01-11T07:56:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/hw_random/amd-rng.ko 2024-01-11T07:56:33.457 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/hw_random/intel-rng.ko 2024-01-11T07:56:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-01-11T07:56:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/hw_random/via-rng.ko 2024-01-11T07:56:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/hw_random/virtio-rng.ko 2024-01-11T07:56:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/ipmi 2024-01-11T07:56:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-01-11T07:56:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-01-11T07:56:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-01-11T07:56:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/ipmi/ipmi_si.ko 2024-01-11T07:56:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-01-11T07:56:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-01-11T07:56:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/lp.ko 2024-01-11T07:56:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/mwave 2024-01-11T07:56:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/mwave/mwave.ko 2024-01-11T07:56:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/ppdev.ko 2024-01-11T07:56:33.458 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/tlclk.ko 2024-01-11T07:56:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/tpm 2024-01-11T07:56:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/tpm/tpm.ko 2024-01-11T07:56:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/tpm/tpm_atmel.ko 2024-01-11T07:56:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/tpm/tpm_infineon.ko 2024-01-11T07:56:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/tpm/tpm_nsc.ko 2024-01-11T07:56:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/tpm/tpm_tis.ko 2024-01-11T07:56:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-01-11T07:56:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/uv_mmtimer.ko 2024-01-11T07:56:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/char/virtio_console.ko 2024-01-11T07:56:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/cpufreq 2024-01-11T07:56:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-01-11T07:56:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-01-11T07:56:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/cpufreq/p4-clockmod.ko 2024-01-11T07:56:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-01-11T07:56:33.459 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/cpufreq/powernow-k8.ko 2024-01-11T07:56:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/cpufreq/speedstep-lib.ko 2024-01-11T07:56:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto 2024-01-11T07:56:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/ccp 2024-01-11T07:56:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-01-11T07:56:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/ccp/ccp.ko 2024-01-11T07:56:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/intel 2024-01-11T07:56:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/intel/qat 2024-01-11T07:56:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/intel/qat/qat_common 2024-01-11T07:56:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-01-11T07:56:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-01-11T07:56:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-01-11T07:56:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/padlock-aes.ko 2024-01-11T07:56:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/padlock-sha.ko 2024-01-11T07:56:33.460 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/virtio 2024-01-11T07:56:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-01-11T07:56:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/dca 2024-01-11T07:56:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/dca/dca.ko 2024-01-11T07:56:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/devfreq 2024-01-11T07:56:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-01-11T07:56:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/dma 2024-01-11T07:56:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/dma/dw 2024-01-11T07:56:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/dma/dw/dw_dmac.ko 2024-01-11T07:56:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/dma/ioat 2024-01-11T07:56:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/dma/ioat/ioatdma.ko 2024-01-11T07:56:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac 2024-01-11T07:56:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/amd64_edac.ko 2024-01-11T07:56:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/e752x_edac.ko 2024-01-11T07:56:33.461 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/edac_mce_amd.ko 2024-01-11T07:56:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/i3000_edac.ko 2024-01-11T07:56:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/i3200_edac.ko 2024-01-11T07:56:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/i5100_edac.ko 2024-01-11T07:56:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/i5400_edac.ko 2024-01-11T07:56:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/i7300_edac.ko 2024-01-11T07:56:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/i7core_edac.ko 2024-01-11T07:56:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/i82975x_edac.ko 2024-01-11T07:56:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/ie31200_edac.ko 2024-01-11T07:56:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/sb_edac.ko 2024-01-11T07:56:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/edac/x38_edac.ko 2024-01-11T07:56:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/firmware 2024-01-11T07:56:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/firmware/edd.ko 2024-01-11T07:56:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/firmware/iscsi_ibft.ko 2024-01-11T07:56:33.462 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/gpio 2024-01-11T07:56:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/gpio/gpio-ich.ko 2024-01-11T07:56:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/gpio/gpio-viperboard.ko 2024-01-11T07:56:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid 2024-01-11T07:56:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-appleir.ko 2024-01-11T07:56:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-aureal.ko 2024-01-11T07:56:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-axff.ko 2024-01-11T07:56:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-dr.ko 2024-01-11T07:56:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-elecom.ko 2024-01-11T07:56:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-elo.ko 2024-01-11T07:56:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-emsff.ko 2024-01-11T07:56:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-gaff.ko 2024-01-11T07:56:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-gt683r.ko 2024-01-11T07:56:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-gyration.ko 2024-01-11T07:56:33.463 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-holtek-kbd.ko 2024-01-11T07:56:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-holtek-mouse.ko 2024-01-11T07:56:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-holtekff.ko 2024-01-11T07:56:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-hyperv.ko 2024-01-11T07:56:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-icade.ko 2024-01-11T07:56:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-keytouch.ko 2024-01-11T07:56:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-kye.ko 2024-01-11T07:56:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-lcpower.ko 2024-01-11T07:56:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-led.ko 2024-01-11T07:56:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-lenovo.ko 2024-01-11T07:56:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-logitech-dj.ko 2024-01-11T07:56:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-01-11T07:56:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-multitouch.ko 2024-01-11T07:56:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-ortek.ko 2024-01-11T07:56:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-penmount.ko 2024-01-11T07:56:33.464 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-petalynx.ko 2024-01-11T07:56:33.465 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-picolcd.ko 2024-01-11T07:56:33.465 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-pl.ko 2024-01-11T07:56:33.465 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-plantronics.ko 2024-01-11T07:56:33.465 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-primax.ko 2024-01-11T07:56:33.465 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-rmi.ko 2024-01-11T07:56:33.465 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-arvo.ko 2024-01-11T07:56:33.465 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-common.ko 2024-01-11T07:56:33.465 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-isku.ko 2024-01-11T07:56:33.465 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-kone.ko 2024-01-11T07:56:33.465 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-01-11T07:56:33.465 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-konepure.ko 2024-01-11T07:56:33.465 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-01-11T07:56:33.465 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-lua.ko 2024-01-11T07:56:33.465 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-pyra.ko 2024-01-11T07:56:33.465 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-ryos.ko 2024-01-11T07:56:33.466 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat-savu.ko 2024-01-11T07:56:33.466 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-roccat.ko 2024-01-11T07:56:33.466 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-saitek.ko 2024-01-11T07:56:33.466 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-samsung.ko 2024-01-11T07:56:33.466 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-sensor-hub.ko 2024-01-11T07:56:33.466 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-sjoy.ko 2024-01-11T07:56:33.466 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-sony.ko 2024-01-11T07:56:33.466 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-speedlink.ko 2024-01-11T07:56:33.466 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-steelseries.ko 2024-01-11T07:56:33.466 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-sunplus.ko 2024-01-11T07:56:33.466 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-thrustmaster.ko 2024-01-11T07:56:33.466 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-tivo.ko 2024-01-11T07:56:33.466 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-tmff.ko 2024-01-11T07:56:33.467 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-topseed.ko 2024-01-11T07:56:33.467 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-twinhan.ko 2024-01-11T07:56:33.467 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-uclogic.ko 2024-01-11T07:56:33.467 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-waltop.ko 2024-01-11T07:56:33.467 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-wiimote.ko 2024-01-11T07:56:33.467 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-xinmo.ko 2024-01-11T07:56:33.467 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-zpff.ko 2024-01-11T07:56:33.467 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/hid-zydacron.ko 2024-01-11T07:56:33.467 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/uhid.ko 2024-01-11T07:56:33.467 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hid/wacom.ko 2024-01-11T07:56:33.467 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hv 2024-01-11T07:56:33.467 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hv/hv_balloon.ko 2024-01-11T07:56:33.467 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hv/hv_utils.ko 2024-01-11T07:56:33.467 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hv/hv_vmbus.ko 2024-01-11T07:56:33.467 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon 2024-01-11T07:56:33.467 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/abituguru.ko 2024-01-11T07:56:33.468 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/abituguru3.ko 2024-01-11T07:56:33.468 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/acpi_power_meter.ko 2024-01-11T07:56:33.468 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ad7414.ko 2024-01-11T07:56:33.468 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ad7418.ko 2024-01-11T07:56:33.468 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adc128d818.ko 2024-01-11T07:56:33.468 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adm1025.ko 2024-01-11T07:56:33.468 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adm1026.ko 2024-01-11T07:56:33.468 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adm1029.ko 2024-01-11T07:56:33.468 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adm1031.ko 2024-01-11T07:56:33.468 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adm9240.ko 2024-01-11T07:56:33.468 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ads7828.ko 2024-01-11T07:56:33.468 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adt7410.ko 2024-01-11T07:56:33.468 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adt7411.ko 2024-01-11T07:56:33.468 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adt7462.ko 2024-01-11T07:56:33.469 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adt7470.ko 2024-01-11T07:56:33.469 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adt7475.ko 2024-01-11T07:56:33.469 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/adt7x10.ko 2024-01-11T07:56:33.469 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/amc6821.ko 2024-01-11T07:56:33.469 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/applesmc.ko 2024-01-11T07:56:33.469 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/asb100.ko 2024-01-11T07:56:33.469 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/asc7621.ko 2024-01-11T07:56:33.469 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/asus_atk0110.ko 2024-01-11T07:56:33.469 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/atxp1.ko 2024-01-11T07:56:33.469 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/coretemp.ko 2024-01-11T07:56:33.469 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-01-11T07:56:33.469 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/dme1737.ko 2024-01-11T07:56:33.469 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ds1621.ko 2024-01-11T07:56:33.469 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ds620.ko 2024-01-11T07:56:33.469 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/emc1403.ko 2024-01-11T07:56:33.470 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/emc6w201.ko 2024-01-11T07:56:33.470 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/f71805f.ko 2024-01-11T07:56:33.470 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/f71882fg.ko 2024-01-11T07:56:33.470 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/f75375s.ko 2024-01-11T07:56:33.470 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/fam15h_power.ko 2024-01-11T07:56:33.470 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/fschmd.ko 2024-01-11T07:56:33.470 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/g760a.ko 2024-01-11T07:56:33.470 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/g762.ko 2024-01-11T07:56:33.470 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/gl518sm.ko 2024-01-11T07:56:33.470 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/gl520sm.ko 2024-01-11T07:56:33.470 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/hwmon-vid.ko 2024-01-11T07:56:33.470 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/i5500_temp.ko 2024-01-11T07:56:33.470 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/i5k_amb.ko 2024-01-11T07:56:33.470 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ibmaem.ko 2024-01-11T07:56:33.471 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ibmpex.ko 2024-01-11T07:56:33.471 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ina209.ko 2024-01-11T07:56:33.471 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ina2xx.ko 2024-01-11T07:56:33.471 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/it87.ko 2024-01-11T07:56:33.471 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/k10temp.ko 2024-01-11T07:56:33.471 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/k8temp.ko 2024-01-11T07:56:33.471 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lineage-pem.ko 2024-01-11T07:56:33.471 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm63.ko 2024-01-11T07:56:33.471 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm73.ko 2024-01-11T07:56:33.471 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm75.ko 2024-01-11T07:56:33.471 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm77.ko 2024-01-11T07:56:33.471 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm78.ko 2024-01-11T07:56:33.471 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm80.ko 2024-01-11T07:56:33.471 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm83.ko 2024-01-11T07:56:33.471 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm85.ko 2024-01-11T07:56:33.472 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm87.ko 2024-01-11T07:56:33.472 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm90.ko 2024-01-11T07:56:33.472 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm92.ko 2024-01-11T07:56:33.472 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm93.ko 2024-01-11T07:56:33.472 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm95234.ko 2024-01-11T07:56:33.472 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm95241.ko 2024-01-11T07:56:33.472 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/lm95245.ko 2024-01-11T07:56:33.472 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ltc2945.ko 2024-01-11T07:56:33.472 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ltc4151.ko 2024-01-11T07:56:33.472 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ltc4215.ko 2024-01-11T07:56:33.472 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ltc4222.ko 2024-01-11T07:56:33.472 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ltc4245.ko 2024-01-11T07:56:33.472 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ltc4260.ko 2024-01-11T07:56:33.472 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/ltc4261.ko 2024-01-11T07:56:33.473 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/max16065.ko 2024-01-11T07:56:33.473 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/max1619.ko 2024-01-11T07:56:33.473 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/max1668.ko 2024-01-11T07:56:33.473 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/max197.ko 2024-01-11T07:56:33.473 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/max6639.ko 2024-01-11T07:56:33.473 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/max6650.ko 2024-01-11T07:56:33.473 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/max6697.ko 2024-01-11T07:56:33.473 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/mcp3021.ko 2024-01-11T07:56:33.473 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/nct6683.ko 2024-01-11T07:56:33.473 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/nct6775-core.ko 2024-01-11T07:56:33.473 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/nct6775.ko 2024-01-11T07:56:33.473 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/nct7802.ko 2024-01-11T07:56:33.473 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pc87360.ko 2024-01-11T07:56:33.473 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pc87427.ko 2024-01-11T07:56:33.473 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pcf8591.ko 2024-01-11T07:56:33.473 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus 2024-01-11T07:56:33.474 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-01-11T07:56:33.474 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-01-11T07:56:33.474 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-01-11T07:56:33.474 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/max16064.ko 2024-01-11T07:56:33.474 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/max34440.ko 2024-01-11T07:56:33.474 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/max8688.ko 2024-01-11T07:56:33.474 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-01-11T07:56:33.474 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-01-11T07:56:33.474 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-01-11T07:56:33.474 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-01-11T07:56:33.474 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-01-11T07:56:33.474 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-01-11T07:56:33.474 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/powr1220.ko 2024-01-11T07:56:33.474 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/sch5627.ko 2024-01-11T07:56:33.474 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/sch5636.ko 2024-01-11T07:56:33.475 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/sch56xx-common.ko 2024-01-11T07:56:33.475 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/sht15.ko 2024-01-11T07:56:33.475 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/sht21.ko 2024-01-11T07:56:33.475 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/shtc1.ko 2024-01-11T07:56:33.475 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/sis5595.ko 2024-01-11T07:56:33.475 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/smsc47b397.ko 2024-01-11T07:56:33.475 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/smsc47m1.ko 2024-01-11T07:56:33.475 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/smsc47m192.ko 2024-01-11T07:56:33.475 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/thmc50.ko 2024-01-11T07:56:33.475 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/tmp102.ko 2024-01-11T07:56:33.475 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/tmp103.ko 2024-01-11T07:56:33.475 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/tmp401.ko 2024-01-11T07:56:33.475 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/tmp421.ko 2024-01-11T07:56:33.475 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/via-cputemp.ko 2024-01-11T07:56:33.475 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/via686a.ko 2024-01-11T07:56:33.476 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/vt1211.ko 2024-01-11T07:56:33.476 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/vt8231.ko 2024-01-11T07:56:33.476 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/w83627ehf.ko 2024-01-11T07:56:33.476 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/w83627hf.ko 2024-01-11T07:56:33.476 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/w83781d.ko 2024-01-11T07:56:33.476 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/w83791d.ko 2024-01-11T07:56:33.476 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/w83792d.ko 2024-01-11T07:56:33.476 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/w83793.ko 2024-01-11T07:56:33.476 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/w83795.ko 2024-01-11T07:56:33.476 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/w83l785ts.ko 2024-01-11T07:56:33.476 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/hwmon/w83l786ng.ko 2024-01-11T07:56:33.476 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c 2024-01-11T07:56:33.476 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/algos 2024-01-11T07:56:33.476 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-01-11T07:56:33.477 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-01-11T07:56:33.477 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses 2024-01-11T07:56:33.477 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-01-11T07:56:33.477 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-01-11T07:56:33.477 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-01-11T07:56:33.477 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-01-11T07:56:33.477 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-01-11T07:56:33.477 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-01-11T07:56:33.477 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-01-11T07:56:33.477 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-01-11T07:56:33.477 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-i801.ko 2024-01-11T07:56:33.477 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-isch.ko 2024-01-11T07:56:33.477 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-01-11T07:56:33.477 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-01-11T07:56:33.478 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-01-11T07:56:33.478 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-parport.ko 2024-01-11T07:56:33.478 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-01-11T07:56:33.478 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-01-11T07:56:33.478 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-01-11T07:56:33.478 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-01-11T07:56:33.478 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-01-11T07:56:33.478 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-01-11T07:56:33.478 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-via.ko 2024-01-11T07:56:33.478 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-01-11T07:56:33.478 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-01-11T07:56:33.478 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/i2c-dev.ko 2024-01-11T07:56:33.478 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/i2c-mux.ko 2024-01-11T07:56:33.478 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-01-11T07:56:33.479 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/i2c-smbus.ko 2024-01-11T07:56:33.479 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/i2c/i2c-stub.ko 2024-01-11T07:56:33.479 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband 2024-01-11T07:56:33.479 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/core 2024-01-11T07:56:33.479 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/core/ib_cm.ko 2024-01-11T07:56:33.479 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/core/ib_core.ko 2024-01-11T07:56:33.479 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/core/ib_umad.ko 2024-01-11T07:56:33.479 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-01-11T07:56:33.479 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/core/iw_cm.ko 2024-01-11T07:56:33.479 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/core/rdma_cm.ko 2024-01-11T07:56:33.479 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-01-11T07:56:33.479 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/hw 2024-01-11T07:56:33.479 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/hw/cxgb4 2024-01-11T07:56:33.479 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-01-11T07:56:33.480 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/hw/mlx4 2024-01-11T07:56:33.480 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-01-11T07:56:33.480 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/hw/mlx5 2024-01-11T07:56:33.480 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-01-11T07:56:33.480 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/hw/mthca 2024-01-11T07:56:33.480 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-01-11T07:56:33.480 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp 2024-01-11T07:56:33.480 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/ipoib 2024-01-11T07:56:33.480 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-01-11T07:56:33.480 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/iser 2024-01-11T07:56:33.480 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-01-11T07:56:33.480 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/isert 2024-01-11T07:56:33.480 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-01-11T07:56:33.480 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/srp 2024-01-11T07:56:33.480 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-01-11T07:56:33.481 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/srpt 2024-01-11T07:56:33.481 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-01-11T07:56:33.481 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input 2024-01-11T07:56:33.481 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/gameport 2024-01-11T07:56:33.481 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-01-11T07:56:33.481 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/gameport/fm801-gp.ko 2024-01-11T07:56:33.481 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/gameport/gameport.ko 2024-01-11T07:56:33.481 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/joydev.ko 2024-01-11T07:56:33.481 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/keyboard 2024-01-11T07:56:33.481 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/keyboard/gpio_keys.ko 2024-01-11T07:56:33.481 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/mouse 2024-01-11T07:56:33.481 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/mouse/appletouch.ko 2024-01-11T07:56:33.481 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/mouse/bcm5974.ko 2024-01-11T07:56:33.481 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/mouse/cyapatp.ko 2024-01-11T07:56:33.481 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/mouse/elan_i2c.ko 2024-01-11T07:56:33.481 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/mouse/sermouse.ko 2024-01-11T07:56:33.481 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-01-11T07:56:33.482 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/mouse/synaptics_usb.ko 2024-01-11T07:56:33.482 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/mouse/vsxxxaa.ko 2024-01-11T07:56:33.482 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/rmi4 2024-01-11T07:56:33.482 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/rmi4/rmi_core.ko 2024-01-11T07:56:33.482 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/serio 2024-01-11T07:56:33.482 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/serio/altera_ps2.ko 2024-01-11T07:56:33.482 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/serio/arc_ps2.ko 2024-01-11T07:56:33.482 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-01-11T07:56:33.482 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/serio/serio_raw.ko 2024-01-11T07:56:33.482 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/input/sparse-keymap.ko 2024-01-11T07:56:33.482 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/iommu 2024-01-11T07:56:33.482 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/iommu/amd 2024-01-11T07:56:33.482 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/iommu/amd/iommu_v2.ko 2024-01-11T07:56:33.482 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds 2024-01-11T07:56:33.482 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/leds-blinkm.ko 2024-01-11T07:56:33.482 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/leds-lm3530.ko 2024-01-11T07:56:33.483 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/leds-lp3944.ko 2024-01-11T07:56:33.483 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/leds-ss4200.ko 2024-01-11T07:56:33.483 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/trigger 2024-01-11T07:56:33.483 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-01-11T07:56:33.483 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-01-11T07:56:33.483 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-01-11T07:56:33.483 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-01-11T07:56:33.483 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-01-11T07:56:33.483 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-01-11T07:56:33.483 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-01-11T07:56:33.483 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-01-11T07:56:33.483 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md 2024-01-11T07:56:33.483 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/bcache 2024-01-11T07:56:33.483 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/bcache/bcache.ko 2024-01-11T07:56:33.483 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-bio-prison.ko 2024-01-11T07:56:33.483 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-cache-smq.ko 2024-01-11T07:56:33.484 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-cache.ko 2024-01-11T07:56:33.484 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-crypt.ko 2024-01-11T07:56:33.484 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-delay.ko 2024-01-11T07:56:33.484 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-flakey.ko 2024-01-11T07:56:33.484 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-log-userspace.ko 2024-01-11T07:56:33.484 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-multipath.ko 2024-01-11T07:56:33.484 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-queue-length.ko 2024-01-11T07:56:33.484 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-raid.ko 2024-01-11T07:56:33.484 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-round-robin.ko 2024-01-11T07:56:33.484 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-service-time.ko 2024-01-11T07:56:33.484 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-switch.ko 2024-01-11T07:56:33.484 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-thin-pool.ko 2024-01-11T07:56:33.484 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/dm-verity.ko 2024-01-11T07:56:33.484 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/faulty.ko 2024-01-11T07:56:33.485 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/linear.ko 2024-01-11T07:56:33.485 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/multipath.ko 2024-01-11T07:56:33.485 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/persistent-data 2024-01-11T07:56:33.485 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-01-11T07:56:33.485 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/raid0.ko 2024-01-11T07:56:33.485 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/raid1.ko 2024-01-11T07:56:33.485 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/raid10.ko 2024-01-11T07:56:33.485 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/md/raid456.ko 2024-01-11T07:56:33.485 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/message 2024-01-11T07:56:33.485 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/message/fusion 2024-01-11T07:56:33.485 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/message/fusion/mptbase.ko 2024-01-11T07:56:33.485 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/message/fusion/mptctl.ko 2024-01-11T07:56:33.485 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/message/fusion/mptfc.ko 2024-01-11T07:56:33.485 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/message/fusion/mptlan.ko 2024-01-11T07:56:33.485 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/message/fusion/mptsas.ko 2024-01-11T07:56:33.485 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/message/fusion/mptscsih.ko 2024-01-11T07:56:33.486 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/message/fusion/mptspi.ko 2024-01-11T07:56:33.486 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/mfd 2024-01-11T07:56:33.486 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/mfd/lpc_ich.ko 2024-01-11T07:56:33.486 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/mfd/lpc_sch.ko 2024-01-11T07:56:33.486 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/mfd/mfd-core.ko 2024-01-11T07:56:33.486 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/mfd/sm501.ko 2024-01-11T07:56:33.486 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/mfd/viperboard.ko 2024-01-11T07:56:33.486 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/mfd/vx855.ko 2024-01-11T07:56:33.486 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/mfd/wl1273-core.ko 2024-01-11T07:56:33.486 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc 2024-01-11T07:56:33.486 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/altera-stapl 2024-01-11T07:56:33.486 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-01-11T07:56:33.486 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/apds9802als.ko 2024-01-11T07:56:33.486 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/apds990x.ko 2024-01-11T07:56:33.486 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/bh1770glc.ko 2024-01-11T07:56:33.486 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/cb710 2024-01-11T07:56:33.487 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/cb710/cb710.ko 2024-01-11T07:56:33.487 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/eeprom 2024-01-11T07:56:33.487 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/eeprom/at24.ko 2024-01-11T07:56:33.487 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/eeprom/eeprom.ko 2024-01-11T07:56:33.487 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-01-11T07:56:33.487 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/eeprom/max6875.ko 2024-01-11T07:56:33.487 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/enclosure.ko 2024-01-11T07:56:33.487 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/hpilo.ko 2024-01-11T07:56:33.487 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/isl29003.ko 2024-01-11T07:56:33.487 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/isl29020.ko 2024-01-11T07:56:33.487 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/lis3lv02d 2024-01-11T07:56:33.487 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-01-11T07:56:33.487 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-01-11T07:56:33.487 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/mei 2024-01-11T07:56:33.487 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/mei/mei-me.ko 2024-01-11T07:56:33.487 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/mei/mei-txe.ko 2024-01-11T07:56:33.488 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/mei/mei.ko 2024-01-11T07:56:33.488 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/sgi-gru 2024-01-11T07:56:33.488 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/sgi-gru/gru.ko 2024-01-11T07:56:33.488 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/sgi-xp 2024-01-11T07:56:33.488 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/sgi-xp/xp.ko 2024-01-11T07:56:33.488 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/sgi-xp/xpc.ko 2024-01-11T07:56:33.488 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-01-11T07:56:33.488 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/tifm_7xx1.ko 2024-01-11T07:56:33.488 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/tifm_core.ko 2024-01-11T07:56:33.488 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/tsl2550.ko 2024-01-11T07:56:33.488 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/vmw_balloon.ko 2024-01-11T07:56:33.488 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/vmw_vmci 2024-01-11T07:56:33.488 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-01-11T07:56:33.489 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net 2024-01-11T07:56:33.489 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/bonding 2024-01-11T07:56:33.489 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/bonding/bonding.ko 2024-01-11T07:56:33.489 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/dsa 2024-01-11T07:56:33.489 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/dsa/b53 2024-01-11T07:56:33.489 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/dsa/b53/b53_common.ko 2024-01-11T07:56:33.489 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/dsa/bcm-sf2.ko 2024-01-11T07:56:33.489 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/dsa/mv88e6060.ko 2024-01-11T07:56:33.489 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/dsa/mv88e6xxx 2024-01-11T07:56:33.489 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-01-11T07:56:33.489 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/dummy.ko 2024-01-11T07:56:33.489 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/eql.ko 2024-01-11T07:56:33.489 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet 2024-01-11T07:56:33.489 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/3com 2024-01-11T07:56:33.490 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-01-11T07:56:33.490 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-01-11T07:56:33.490 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-01-11T07:56:33.490 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-01-11T07:56:33.490 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/8390 2024-01-11T07:56:33.490 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/8390/8390.ko 2024-01-11T07:56:33.490 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-01-11T07:56:33.490 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-01-11T07:56:33.490 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-01-11T07:56:33.490 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/adaptec 2024-01-11T07:56:33.490 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-01-11T07:56:33.490 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/agere 2024-01-11T07:56:33.490 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/agere/et131x.ko 2024-01-11T07:56:33.490 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/alteon 2024-01-11T07:56:33.491 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-01-11T07:56:33.491 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/altera 2024-01-11T07:56:33.491 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-01-11T07:56:33.491 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/amd 2024-01-11T07:56:33.491 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-01-11T07:56:33.491 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-01-11T07:56:33.491 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-01-11T07:56:33.491 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros 2024-01-11T07:56:33.491 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros/alx 2024-01-11T07:56:33.491 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-01-11T07:56:33.491 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros/atl1c 2024-01-11T07:56:33.491 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-01-11T07:56:33.491 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros/atl1e 2024-01-11T07:56:33.491 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-01-11T07:56:33.491 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros/atlx 2024-01-11T07:56:33.492 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-01-11T07:56:33.492 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-01-11T07:56:33.492 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/broadcom 2024-01-11T07:56:33.492 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-01-11T07:56:33.492 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-01-11T07:56:33.492 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-01-11T07:56:33.492 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-01-11T07:56:33.492 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-01-11T07:56:33.492 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/broadcom/genet 2024-01-11T07:56:33.492 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-01-11T07:56:33.492 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-01-11T07:56:33.492 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/brocade 2024-01-11T07:56:33.492 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/brocade/bna 2024-01-11T07:56:33.492 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-01-11T07:56:33.493 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio 2024-01-11T07:56:33.493 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/cxgb 2024-01-11T07:56:33.493 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-01-11T07:56:33.493 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-01-11T07:56:33.493 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-01-11T07:56:33.493 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-01-11T07:56:33.493 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-01-11T07:56:33.493 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-01-11T07:56:33.493 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-01-11T07:56:33.493 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-01-11T07:56:33.493 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-01-11T07:56:33.493 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/cisco 2024-01-11T07:56:33.493 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/cisco/enic 2024-01-11T07:56:33.493 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-01-11T07:56:33.494 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dec 2024-01-11T07:56:33.494 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dec/tulip 2024-01-11T07:56:33.494 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-01-11T07:56:33.494 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-01-11T07:56:33.494 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-01-11T07:56:33.494 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-01-11T07:56:33.494 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-01-11T07:56:33.494 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-01-11T07:56:33.494 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dlink 2024-01-11T07:56:33.494 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-01-11T07:56:33.494 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-01-11T07:56:33.494 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/dnet.ko 2024-01-11T07:56:33.494 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/emulex 2024-01-11T07:56:33.494 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/emulex/benet 2024-01-11T07:56:33.495 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-01-11T07:56:33.495 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/ethoc.ko 2024-01-11T07:56:33.495 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/fealnx.ko 2024-01-11T07:56:33.495 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel 2024-01-11T07:56:33.495 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/e100.ko 2024-01-11T07:56:33.495 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/e1000 2024-01-11T07:56:33.495 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-01-11T07:56:33.495 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/e1000e 2024-01-11T07:56:33.495 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-01-11T07:56:33.495 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/fm10k 2024-01-11T07:56:33.495 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-01-11T07:56:33.495 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/i40e 2024-01-11T07:56:33.495 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-01-11T07:56:33.495 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/iavf 2024-01-11T07:56:33.495 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-01-11T07:56:33.496 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/igb 2024-01-11T07:56:33.496 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-01-11T07:56:33.496 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/igbvf 2024-01-11T07:56:33.496 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-01-11T07:56:33.496 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/ixgbe 2024-01-11T07:56:33.496 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-01-11T07:56:33.496 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/ixgbevf 2024-01-11T07:56:33.496 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-01-11T07:56:33.496 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/jme.ko 2024-01-11T07:56:33.496 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/marvell 2024-01-11T07:56:33.496 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-01-11T07:56:33.496 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/marvell/skge.ko 2024-01-11T07:56:33.496 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-01-11T07:56:33.497 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/mellanox 2024-01-11T07:56:33.497 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/mellanox/mlx4 2024-01-11T07:56:33.497 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-01-11T07:56:33.497 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-01-11T07:56:33.497 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/mellanox/mlx5 2024-01-11T07:56:33.497 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-01-11T07:56:33.497 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-01-11T07:56:33.497 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/micrel 2024-01-11T07:56:33.497 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-01-11T07:56:33.497 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/myricom 2024-01-11T07:56:33.497 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/myricom/myri10ge 2024-01-11T07:56:33.497 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-01-11T07:56:33.497 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/natsemi 2024-01-11T07:56:33.497 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-01-11T07:56:33.498 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-01-11T07:56:33.498 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/neterion 2024-01-11T07:56:33.498 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-01-11T07:56:33.498 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/nvidia 2024-01-11T07:56:33.498 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-01-11T07:56:33.498 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/packetengines 2024-01-11T07:56:33.498 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-01-11T07:56:33.498 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-01-11T07:56:33.498 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/qlogic 2024-01-11T07:56:33.498 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/qlogic/netxen 2024-01-11T07:56:33.498 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-01-11T07:56:33.498 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-01-11T07:56:33.498 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-01-11T07:56:33.498 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-01-11T07:56:33.499 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/rdc 2024-01-11T07:56:33.499 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-01-11T07:56:33.499 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/realtek 2024-01-11T07:56:33.499 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-01-11T07:56:33.499 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-01-11T07:56:33.499 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/realtek/atp.ko 2024-01-11T07:56:33.499 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-01-11T07:56:33.499 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/rocker 2024-01-11T07:56:33.499 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-01-11T07:56:33.499 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sfc 2024-01-11T07:56:33.499 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-01-11T07:56:33.499 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/silan 2024-01-11T07:56:33.499 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-01-11T07:56:33.499 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sis 2024-01-11T07:56:33.499 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sis/sis190.ko 2024-01-11T07:56:33.500 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sis/sis900.ko 2024-01-11T07:56:33.500 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/smsc 2024-01-11T07:56:33.500 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-01-11T07:56:33.500 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-01-11T07:56:33.500 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-01-11T07:56:33.500 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-01-11T07:56:33.500 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/stmicro 2024-01-11T07:56:33.500 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/stmicro/stmmac 2024-01-11T07:56:33.500 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-01-11T07:56:33.500 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-01-11T07:56:33.500 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sun 2024-01-11T07:56:33.500 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sun/cassini.ko 2024-01-11T07:56:33.500 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sun/niu.ko 2024-01-11T07:56:33.500 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sun/sungem.ko 2024-01-11T07:56:33.501 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-01-11T07:56:33.501 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/tehuti 2024-01-11T07:56:33.501 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-01-11T07:56:33.501 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/ti 2024-01-11T07:56:33.501 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/ti/tlan.ko 2024-01-11T07:56:33.501 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/via 2024-01-11T07:56:33.501 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-01-11T07:56:33.501 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-01-11T07:56:33.501 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/wiznet 2024-01-11T07:56:33.501 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-01-11T07:56:33.501 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-01-11T07:56:33.501 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/xircom 2024-01-11T07:56:33.501 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-01-11T07:56:33.501 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/geneve.ko 2024-01-11T07:56:33.501 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/hyperv 2024-01-11T07:56:33.502 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-01-11T07:56:33.502 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ieee802154 2024-01-11T07:56:33.502 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ieee802154/fakelb.ko 2024-01-11T07:56:33.502 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ifb.ko 2024-01-11T07:56:33.502 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ipvlan 2024-01-11T07:56:33.502 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ipvlan/ipvlan.ko 2024-01-11T07:56:33.502 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/macvlan.ko 2024-01-11T07:56:33.502 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/macvtap.ko 2024-01-11T07:56:33.502 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/mdio 2024-01-11T07:56:33.502 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/mdio.ko 2024-01-11T07:56:33.502 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-01-11T07:56:33.502 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-01-11T07:56:33.502 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/mdio/mdio-regmap.ko 2024-01-11T07:56:33.503 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/mii.ko 2024-01-11T07:56:33.503 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/net_failover.ko 2024-01-11T07:56:33.503 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/netconsole.ko 2024-01-11T07:56:33.503 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/nlmon.ko 2024-01-11T07:56:33.503 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/pcs 2024-01-11T07:56:33.503 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/pcs/pcs-lynx.ko 2024-01-11T07:56:33.503 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-01-11T07:56:33.503 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy 2024-01-11T07:56:33.503 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/amd.ko 2024-01-11T07:56:33.503 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-01-11T07:56:33.503 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-01-11T07:56:33.503 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/bcm7xxx.ko 2024-01-11T07:56:33.503 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/bcm87xx.ko 2024-01-11T07:56:33.503 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/broadcom.ko 2024-01-11T07:56:33.503 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/cicada.ko 2024-01-11T07:56:33.504 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/davicom.ko 2024-01-11T07:56:33.504 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/dp83640.ko 2024-01-11T07:56:33.504 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/et1011c.ko 2024-01-11T07:56:33.504 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/icplus.ko 2024-01-11T07:56:33.504 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/lxt.ko 2024-01-11T07:56:33.504 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/marvell.ko 2024-01-11T07:56:33.504 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/micrel.ko 2024-01-11T07:56:33.504 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/national.ko 2024-01-11T07:56:33.504 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/phylink.ko 2024-01-11T07:56:33.504 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/qsemi.ko 2024-01-11T07:56:33.504 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/realtek.ko 2024-01-11T07:56:33.504 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/smsc.ko 2024-01-11T07:56:33.504 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/ste10Xp.ko 2024-01-11T07:56:33.505 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/phy/vitesse.ko 2024-01-11T07:56:33.505 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp 2024-01-11T07:56:33.505 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp/bsd_comp.ko 2024-01-11T07:56:33.505 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp/ppp_async.ko 2024-01-11T07:56:33.505 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp/ppp_deflate.ko 2024-01-11T07:56:33.505 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp/ppp_generic.ko 2024-01-11T07:56:33.505 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp/ppp_mppe.ko 2024-01-11T07:56:33.505 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp/ppp_synctty.ko 2024-01-11T07:56:33.505 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp/pppoe.ko 2024-01-11T07:56:33.505 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp/pppox.ko 2024-01-11T07:56:33.505 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/ppp/pptp.ko 2024-01-11T07:56:33.505 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/slip 2024-01-11T07:56:33.505 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/slip/slhc.ko 2024-01-11T07:56:33.505 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/slip/slip.ko 2024-01-11T07:56:33.505 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/sungem_phy.ko 2024-01-11T07:56:33.506 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/tap.ko 2024-01-11T07:56:33.506 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/team 2024-01-11T07:56:33.506 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/team/team.ko 2024-01-11T07:56:33.506 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/team/team_mode_activebackup.ko 2024-01-11T07:56:33.506 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/team/team_mode_broadcast.ko 2024-01-11T07:56:33.506 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-01-11T07:56:33.506 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/team/team_mode_random.ko 2024-01-11T07:56:33.506 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-01-11T07:56:33.506 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/tun.ko 2024-01-11T07:56:33.506 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/veth.ko 2024-01-11T07:56:33.506 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/virtio_net.ko 2024-01-11T07:56:33.506 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/vmxnet3 2024-01-11T07:56:33.506 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-01-11T07:56:33.506 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/vxlan 2024-01-11T07:56:33.507 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/vxlan/vxlan.ko 2024-01-11T07:56:33.507 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/xen-netback 2024-01-11T07:56:33.507 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/xen-netback/xen-netback.ko 2024-01-11T07:56:33.507 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/net/xen-netfront.ko 2024-01-11T07:56:33.507 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ntb 2024-01-11T07:56:33.507 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ntb/ntb.ko 2024-01-11T07:56:33.507 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/nvme 2024-01-11T07:56:33.507 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/nvme/host 2024-01-11T07:56:33.507 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/nvme/host/nvme-core.ko 2024-01-11T07:56:33.507 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/nvme/host/nvme.ko 2024-01-11T07:56:33.507 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/parport 2024-01-11T07:56:33.507 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/parport/parport.ko 2024-01-11T07:56:33.507 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/parport/parport_cs.ko 2024-01-11T07:56:33.507 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/parport/parport_pc.ko 2024-01-11T07:56:33.508 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/parport/parport_serial.ko 2024-01-11T07:56:33.508 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pci 2024-01-11T07:56:33.508 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pci/hotplug 2024-01-11T07:56:33.508 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-01-11T07:56:33.508 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pci/pcie 2024-01-11T07:56:33.508 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pci/pcie/aer_inject.ko 2024-01-11T07:56:33.508 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pci/xen-pcifront.ko 2024-01-11T07:56:33.508 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pcmcia 2024-01-11T07:56:33.508 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pcmcia/i82092.ko 2024-01-11T07:56:33.508 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pcmcia/pd6729.ko 2024-01-11T07:56:33.508 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pcmcia/yenta_socket.ko 2024-01-11T07:56:33.508 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pinctrl 2024-01-11T07:56:33.508 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pinctrl/intel 2024-01-11T07:56:33.508 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-01-11T07:56:33.508 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform 2024-01-11T07:56:33.509 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86 2024-01-11T07:56:33.509 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/acer-wmi.ko 2024-01-11T07:56:33.509 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/acerhdf.ko 2024-01-11T07:56:33.509 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/apple-gmux.ko 2024-01-11T07:56:33.509 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/asus-laptop.ko 2024-01-11T07:56:33.509 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-01-11T07:56:33.509 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/asus-wmi.ko 2024-01-11T07:56:33.509 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/classmate-laptop.ko 2024-01-11T07:56:33.509 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-01-11T07:56:33.509 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-01-11T07:56:33.509 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-01-11T07:56:33.509 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-01-11T07:56:33.509 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/hdaps.ko 2024-01-11T07:56:33.509 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/intel 2024-01-11T07:56:33.509 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-01-11T07:56:33.509 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/intel_ips.ko 2024-01-11T07:56:33.510 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/msi-wmi.ko 2024-01-11T07:56:33.510 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/mxm-wmi.ko 2024-01-11T07:56:33.510 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-01-11T07:56:33.510 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/samsung-laptop.ko 2024-01-11T07:56:33.510 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/samsung-q10.ko 2024-01-11T07:56:33.510 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/topstar-laptop.ko 2024-01-11T07:56:33.510 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-01-11T07:56:33.510 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/toshiba_haps.ko 2024-01-11T07:56:33.510 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/wmi-bmof.ko 2024-01-11T07:56:33.510 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/platform/x86/wmi.ko 2024-01-11T07:56:33.510 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/powercap 2024-01-11T07:56:33.510 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/powercap/intel_rapl_common.ko 2024-01-11T07:56:33.510 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/powercap/intel_rapl_msr.ko 2024-01-11T07:56:33.510 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pps 2024-01-11T07:56:33.510 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pps/clients 2024-01-11T07:56:33.511 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pps/clients/pps-gpio.ko 2024-01-11T07:56:33.511 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pps/clients/pps-ldisc.ko 2024-01-11T07:56:33.511 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pps/clients/pps_parport.ko 2024-01-11T07:56:33.511 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/pps/pps_core.ko 2024-01-11T07:56:33.511 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ptp 2024-01-11T07:56:33.511 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ptp/ptp.ko 2024-01-11T07:56:33.511 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ptp/ptp_kvm.ko 2024-01-11T07:56:33.511 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc 2024-01-11T07:56:33.511 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-bq32k.ko 2024-01-11T07:56:33.511 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-ds1286.ko 2024-01-11T07:56:33.511 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-ds1307.ko 2024-01-11T07:56:33.511 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-ds1374.ko 2024-01-11T07:56:33.511 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-ds1511.ko 2024-01-11T07:56:33.511 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-ds1553.ko 2024-01-11T07:56:33.512 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-ds1672.ko 2024-01-11T07:56:33.512 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-ds1742.ko 2024-01-11T07:56:33.512 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-ds2404.ko 2024-01-11T07:56:33.512 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-ds3232.ko 2024-01-11T07:56:33.512 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-em3027.ko 2024-01-11T07:56:33.512 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-fm3130.ko 2024-01-11T07:56:33.512 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-isl12022.ko 2024-01-11T07:56:33.512 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-isl1208.ko 2024-01-11T07:56:33.512 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-m41t80.ko 2024-01-11T07:56:33.512 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-m48t35.ko 2024-01-11T07:56:33.512 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-m48t59.ko 2024-01-11T07:56:33.512 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-max6900.ko 2024-01-11T07:56:33.512 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-msm6242.ko 2024-01-11T07:56:33.512 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-pcf2127.ko 2024-01-11T07:56:33.513 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-pcf85063.ko 2024-01-11T07:56:33.513 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-pcf8523.ko 2024-01-11T07:56:33.513 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-pcf8563.ko 2024-01-11T07:56:33.513 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-pcf8583.ko 2024-01-11T07:56:33.513 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-rp5c01.ko 2024-01-11T07:56:33.513 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-rs5c372.ko 2024-01-11T07:56:33.513 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-01-11T07:56:33.513 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-rx8025.ko 2024-01-11T07:56:33.513 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-rx8581.ko 2024-01-11T07:56:33.513 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-01-11T07:56:33.513 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/rtc/rtc-x1205.ko 2024-01-11T07:56:33.513 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi 2024-01-11T07:56:33.513 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/3w-9xxx.ko 2024-01-11T07:56:33.513 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/3w-sas.ko 2024-01-11T07:56:33.514 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/3w-xxxx.ko 2024-01-11T07:56:33.514 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/BusLogic.ko 2024-01-11T07:56:33.514 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/a100u2w.ko 2024-01-11T07:56:33.514 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/aacraid 2024-01-11T07:56:33.514 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/aacraid/aacraid.ko 2024-01-11T07:56:33.514 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/advansys.ko 2024-01-11T07:56:33.514 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/aic7xxx 2024-01-11T07:56:33.514 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-01-11T07:56:33.514 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-01-11T07:56:33.514 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/am53c974.ko 2024-01-11T07:56:33.514 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/arcmsr 2024-01-11T07:56:33.514 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-01-11T07:56:33.514 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/atp870u.ko 2024-01-11T07:56:33.514 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/be2iscsi 2024-01-11T07:56:33.514 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-01-11T07:56:33.515 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/bfa 2024-01-11T07:56:33.515 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/bfa/bfa.ko 2024-01-11T07:56:33.515 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/bnx2fc 2024-01-11T07:56:33.515 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-01-11T07:56:33.515 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/bnx2i 2024-01-11T07:56:33.515 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-01-11T07:56:33.515 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/ch.ko 2024-01-11T07:56:33.515 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/csiostor 2024-01-11T07:56:33.515 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/csiostor/csiostor.ko 2024-01-11T07:56:33.515 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/cxgbi 2024-01-11T07:56:33.515 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/cxgbi/cxgb3i 2024-01-11T07:56:33.515 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-01-11T07:56:33.515 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/cxgbi/cxgb4i 2024-01-11T07:56:33.515 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-01-11T07:56:33.515 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-01-11T07:56:33.516 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/dc395x.ko 2024-01-11T07:56:33.516 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/device_handler 2024-01-11T07:56:33.516 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-01-11T07:56:33.516 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-01-11T07:56:33.516 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-01-11T07:56:33.516 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-01-11T07:56:33.516 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/dmx3191d.ko 2024-01-11T07:56:33.516 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/esas2r 2024-01-11T07:56:33.516 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/esas2r/esas2r.ko 2024-01-11T07:56:33.516 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/esp_scsi.ko 2024-01-11T07:56:33.516 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/fcoe 2024-01-11T07:56:33.516 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/fcoe/fcoe.ko 2024-01-11T07:56:33.516 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-01-11T07:56:33.516 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/fnic 2024-01-11T07:56:33.516 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/fnic/fnic.ko 2024-01-11T07:56:33.517 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/hpsa.ko 2024-01-11T07:56:33.517 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/hptiop.ko 2024-01-11T07:56:33.517 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/hv_storvsc.ko 2024-01-11T07:56:33.517 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/initio.ko 2024-01-11T07:56:33.517 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/ipr.ko 2024-01-11T07:56:33.517 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/ips.ko 2024-01-11T07:56:33.517 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/isci 2024-01-11T07:56:33.517 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/isci/isci.ko 2024-01-11T07:56:33.517 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-01-11T07:56:33.517 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/iscsi_tcp.ko 2024-01-11T07:56:33.517 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/libfc 2024-01-11T07:56:33.517 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/libfc/libfc.ko 2024-01-11T07:56:33.517 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/libiscsi.ko 2024-01-11T07:56:33.517 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/libiscsi_tcp.ko 2024-01-11T07:56:33.518 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/libsas 2024-01-11T07:56:33.518 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/libsas/libsas.ko 2024-01-11T07:56:33.518 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/lpfc 2024-01-11T07:56:33.518 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/lpfc/lpfc.ko 2024-01-11T07:56:33.518 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/megaraid 2024-01-11T07:56:33.518 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/megaraid.ko 2024-01-11T07:56:33.518 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-01-11T07:56:33.518 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-01-11T07:56:33.518 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-01-11T07:56:33.518 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/mpt3sas 2024-01-11T07:56:33.518 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-01-11T07:56:33.518 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/mvsas 2024-01-11T07:56:33.518 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/mvsas/mvsas.ko 2024-01-11T07:56:33.518 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/mvumi.ko 2024-01-11T07:56:33.518 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/pm8001 2024-01-11T07:56:33.519 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-01-11T07:56:33.519 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/pmcraid.ko 2024-01-11T07:56:33.519 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/qla1280.ko 2024-01-11T07:56:33.519 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/qla2xxx 2024-01-11T07:56:33.519 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-01-11T07:56:33.519 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-01-11T07:56:33.519 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/qla4xxx 2024-01-11T07:56:33.519 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-01-11T07:56:33.519 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/raid_class.ko 2024-01-11T07:56:33.519 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/scsi_debug.ko 2024-01-11T07:56:33.519 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/scsi_transport_fc.ko 2024-01-11T07:56:33.519 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-01-11T07:56:33.519 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/scsi_transport_sas.ko 2024-01-11T07:56:33.519 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/scsi_transport_spi.ko 2024-01-11T07:56:33.520 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/scsi_transport_srp.ko 2024-01-11T07:56:33.520 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/ses.ko 2024-01-11T07:56:33.520 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/st.ko 2024-01-11T07:56:33.520 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/stex.ko 2024-01-11T07:56:33.520 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/sym53c8xx_2 2024-01-11T07:56:33.520 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-01-11T07:56:33.520 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/virtio_scsi.ko 2024-01-11T07:56:33.520 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/vmw_pvscsi.ko 2024-01-11T07:56:33.520 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/scsi/wd719x.ko 2024-01-11T07:56:33.520 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ssb 2024-01-11T07:56:33.520 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ssb/ssb.ko 2024-01-11T07:56:33.520 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target 2024-01-11T07:56:33.520 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/iscsi 2024-01-11T07:56:33.520 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-01-11T07:56:33.521 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/loopback 2024-01-11T07:56:33.521 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/loopback/tcm_loop.ko 2024-01-11T07:56:33.521 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/target_core_file.ko 2024-01-11T07:56:33.521 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/target_core_iblock.ko 2024-01-11T07:56:33.521 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/target_core_mod.ko 2024-01-11T07:56:33.521 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/target_core_pscsi.ko 2024-01-11T07:56:33.521 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/target_core_user.ko 2024-01-11T07:56:33.521 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/tcm_fc 2024-01-11T07:56:33.521 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-01-11T07:56:33.521 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal 2024-01-11T07:56:33.521 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel 2024-01-11T07:56:33.521 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal 2024-01-11T07:56:33.521 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-01-11T07:56:33.521 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-01-11T07:56:33.521 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-01-11T07:56:33.522 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-01-11T07:56:33.522 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-01-11T07:56:33.522 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-01-11T07:56:33.522 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-01-11T07:56:33.522 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-01-11T07:56:33.522 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-01-11T07:56:33.522 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-01-11T07:56:33.522 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-01-11T07:56:33.522 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-01-11T07:56:33.522 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-01-11T07:56:33.522 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-01-11T07:56:33.522 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-01-11T07:56:33.522 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty 2024-01-11T07:56:33.522 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/ipwireless 2024-01-11T07:56:33.522 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-01-11T07:56:33.523 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/n_gsm.ko 2024-01-11T07:56:33.523 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/n_hdlc.ko 2024-01-11T07:56:33.523 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/nozomi.ko 2024-01-11T07:56:33.523 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/serial 2024-01-11T07:56:33.523 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/serial/8250 2024-01-11T07:56:33.523 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-01-11T07:56:33.523 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/serial/arc_uart.ko 2024-01-11T07:56:33.523 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/serial/jsm 2024-01-11T07:56:33.523 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/tty/serial/jsm/jsm.ko 2024-01-11T07:56:33.523 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ufs 2024-01-11T07:56:33.523 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ufs/core 2024-01-11T07:56:33.523 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ufs/core/ufshcd-core.ko 2024-01-11T07:56:33.523 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ufs/host 2024-01-11T07:56:33.523 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-01-11T07:56:33.523 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/uio 2024-01-11T07:56:33.524 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/uio/uio.ko 2024-01-11T07:56:33.524 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/uio/uio_aec.ko 2024-01-11T07:56:33.524 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/uio/uio_cif.ko 2024-01-11T07:56:33.524 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/uio/uio_pci_generic.ko 2024-01-11T07:56:33.524 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/uio/uio_sercos3.ko 2024-01-11T07:56:33.524 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb 2024-01-11T07:56:33.524 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/class 2024-01-11T07:56:33.524 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/class/cdc-acm.ko 2024-01-11T07:56:33.524 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/class/cdc-wdm.ko 2024-01-11T07:56:33.524 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/class/usblp.ko 2024-01-11T07:56:33.524 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/class/usbtmc.ko 2024-01-11T07:56:33.524 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/host 2024-01-11T07:56:33.524 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/host/sl811-hcd.ko 2024-01-11T07:56:33.524 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/image 2024-01-11T07:56:33.525 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/image/mdc800.ko 2024-01-11T07:56:33.525 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/image/microtek.ko 2024-01-11T07:56:33.525 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc 2024-01-11T07:56:33.525 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/adutux.ko 2024-01-11T07:56:33.525 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/appledisplay.ko 2024-01-11T07:56:33.525 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/emi26.ko 2024-01-11T07:56:33.525 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/emi62.ko 2024-01-11T07:56:33.525 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/ezusb.ko 2024-01-11T07:56:33.525 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/idmouse.ko 2024-01-11T07:56:33.525 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/iowarrior.ko 2024-01-11T07:56:33.525 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/isight_firmware.ko 2024-01-11T07:56:33.525 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/ldusb.ko 2024-01-11T07:56:33.525 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/legousbtower.ko 2024-01-11T07:56:33.525 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/sisusbvga 2024-01-11T07:56:33.525 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-01-11T07:56:33.525 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/trancevibrator.ko 2024-01-11T07:56:33.526 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/usb3503.ko 2024-01-11T07:56:33.526 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/usblcd.ko 2024-01-11T07:56:33.526 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/usbsevseg.ko 2024-01-11T07:56:33.526 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/uss720.ko 2024-01-11T07:56:33.526 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/misc/yurex.ko 2024-01-11T07:56:33.526 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/phy 2024-01-11T07:56:33.526 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/phy/phy-generic.ko 2024-01-11T07:56:33.526 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial 2024-01-11T07:56:33.526 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/aircable.ko 2024-01-11T07:56:33.526 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/ark3116.ko 2024-01-11T07:56:33.526 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/belkin_sa.ko 2024-01-11T07:56:33.526 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/ch341.ko 2024-01-11T07:56:33.526 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/cp210x.ko 2024-01-11T07:56:33.526 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/cyberjack.ko 2024-01-11T07:56:33.526 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/cypress_m8.ko 2024-01-11T07:56:33.527 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/digi_acceleport.ko 2024-01-11T07:56:33.527 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/empeg.ko 2024-01-11T07:56:33.527 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/ftdi_sio.ko 2024-01-11T07:56:33.527 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/garmin_gps.ko 2024-01-11T07:56:33.527 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/io_edgeport.ko 2024-01-11T07:56:33.527 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/io_ti.ko 2024-01-11T07:56:33.527 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/ipaq.ko 2024-01-11T07:56:33.527 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/ipw.ko 2024-01-11T07:56:33.527 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/ir-usb.ko 2024-01-11T07:56:33.527 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-01-11T07:56:33.527 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/keyspan.ko 2024-01-11T07:56:33.527 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/keyspan_pda.ko 2024-01-11T07:56:33.527 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/kl5kusb105.ko 2024-01-11T07:56:33.527 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/kobil_sct.ko 2024-01-11T07:56:33.528 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/mct_u232.ko 2024-01-11T07:56:33.528 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/mos7720.ko 2024-01-11T07:56:33.528 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/mos7840.ko 2024-01-11T07:56:33.528 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/navman.ko 2024-01-11T07:56:33.528 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/omninet.ko 2024-01-11T07:56:33.528 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/opticon.ko 2024-01-11T07:56:33.528 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/option.ko 2024-01-11T07:56:33.528 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/oti6858.ko 2024-01-11T07:56:33.528 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/pl2303.ko 2024-01-11T07:56:33.528 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/qcaux.ko 2024-01-11T07:56:33.528 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/qcserial.ko 2024-01-11T07:56:33.528 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/quatech2.ko 2024-01-11T07:56:33.528 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/safe_serial.ko 2024-01-11T07:56:33.528 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/sierra.ko 2024-01-11T07:56:33.528 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/spcp8x5.ko 2024-01-11T07:56:33.529 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/ssu100.ko 2024-01-11T07:56:33.529 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/symbolserial.ko 2024-01-11T07:56:33.529 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-01-11T07:56:33.529 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-01-11T07:56:33.529 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/usb_debug.ko 2024-01-11T07:56:33.529 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/usb_wwan.ko 2024-01-11T07:56:33.529 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/visor.ko 2024-01-11T07:56:33.529 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/whiteheat.ko 2024-01-11T07:56:33.529 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/serial/xsens_mt.ko 2024-01-11T07:56:33.529 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage 2024-01-11T07:56:33.529 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/uas.ko 2024-01-11T07:56:33.529 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-alauda.ko 2024-01-11T07:56:33.529 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-cypress.ko 2024-01-11T07:56:33.529 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-datafab.ko 2024-01-11T07:56:33.529 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-01-11T07:56:33.530 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-freecom.ko 2024-01-11T07:56:33.530 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-isd200.ko 2024-01-11T07:56:33.530 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-01-11T07:56:33.530 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-karma.ko 2024-01-11T07:56:33.530 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-onetouch.ko 2024-01-11T07:56:33.530 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-realtek.ko 2024-01-11T07:56:33.530 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-sddr09.ko 2024-01-11T07:56:33.530 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-sddr55.ko 2024-01-11T07:56:33.530 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/ums-usbat.ko 2024-01-11T07:56:33.530 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/storage/usb-storage.ko 2024-01-11T07:56:33.530 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/usbip 2024-01-11T07:56:33.530 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/usbip/usbip-core.ko 2024-01-11T07:56:33.530 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/usbip/usbip-host.ko 2024-01-11T07:56:33.530 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-01-11T07:56:33.531 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vfio 2024-01-11T07:56:33.531 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vfio/pci 2024-01-11T07:56:33.531 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-01-11T07:56:33.531 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vfio/pci/vfio-pci.ko 2024-01-11T07:56:33.531 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vfio/vfio.ko 2024-01-11T07:56:33.531 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-01-11T07:56:33.531 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vhost 2024-01-11T07:56:33.531 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vhost/vhost.ko 2024-01-11T07:56:33.531 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vhost/vhost_iotlb.ko 2024-01-11T07:56:33.531 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vhost/vhost_net.ko 2024-01-11T07:56:33.531 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/vhost/vhost_scsi.ko 2024-01-11T07:56:33.531 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/video 2024-01-11T07:56:33.531 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/video/backlight 2024-01-11T07:56:33.531 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/video/backlight/apple_bl.ko 2024-01-11T07:56:33.531 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/video/backlight/lcd.ko 2024-01-11T07:56:33.531 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/video/backlight/platform_lcd.ko 2024-01-11T07:56:33.532 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/virtio 2024-01-11T07:56:33.532 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/virtio/virtio.ko 2024-01-11T07:56:33.532 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/virtio/virtio_balloon.ko 2024-01-11T07:56:33.532 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/virtio/virtio_mmio.ko 2024-01-11T07:56:33.532 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/virtio/virtio_pci.ko 2024-01-11T07:56:33.532 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-01-11T07:56:33.532 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-01-11T07:56:33.532 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/virtio/virtio_ring.ko 2024-01-11T07:56:33.532 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1 2024-01-11T07:56:33.532 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/masters 2024-01-11T07:56:33.532 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/masters/ds2482.ko 2024-01-11T07:56:33.532 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/masters/ds2490.ko 2024-01-11T07:56:33.532 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves 2024-01-11T07:56:33.532 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-01-11T07:56:33.532 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-01-11T07:56:33.533 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-01-11T07:56:33.533 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-01-11T07:56:33.533 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-01-11T07:56:33.533 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-01-11T07:56:33.533 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-01-11T07:56:33.533 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-01-11T07:56:33.533 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-01-11T07:56:33.533 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_smem.ko 2024-01-11T07:56:33.533 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/slaves/w1_therm.ko 2024-01-11T07:56:33.533 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/w1/wire.ko 2024-01-11T07:56:33.533 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog 2024-01-11T07:56:33.533 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/alim1535_wdt.ko 2024-01-11T07:56:33.533 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/alim7101_wdt.ko 2024-01-11T07:56:33.533 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/f71808e_wdt.ko 2024-01-11T07:56:33.533 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/hpwdt.ko 2024-01-11T07:56:33.534 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/i6300esb.ko 2024-01-11T07:56:33.534 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-01-11T07:56:33.534 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/iTCO_wdt.ko 2024-01-11T07:56:33.534 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/ib700wdt.ko 2024-01-11T07:56:33.534 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/ibmasr.ko 2024-01-11T07:56:33.534 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-01-11T07:56:33.534 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/it8712f_wdt.ko 2024-01-11T07:56:33.534 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/it87_wdt.ko 2024-01-11T07:56:33.534 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/machzwd.ko 2024-01-11T07:56:33.534 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/nv_tco.ko 2024-01-11T07:56:33.534 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/pcwd_pci.ko 2024-01-11T07:56:33.534 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/pcwd_usb.ko 2024-01-11T07:56:33.534 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-01-11T07:56:33.535 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/sch311x_wdt.ko 2024-01-11T07:56:33.535 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/softdog.ko 2024-01-11T07:56:33.535 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/sp5100_tco.ko 2024-01-11T07:56:33.535 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/via_wdt.ko 2024-01-11T07:56:33.535 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-01-11T07:56:33.535 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/w83877f_wdt.ko 2024-01-11T07:56:33.535 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/w83977f_wdt.ko 2024-01-11T07:56:33.535 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/wdt_pci.ko 2024-01-11T07:56:33.535 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/watchdog/xen_wdt.ko 2024-01-11T07:56:33.535 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen 2024-01-11T07:56:33.535 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen/xen-acpi-processor.ko 2024-01-11T07:56:33.535 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen/xen-evtchn.ko 2024-01-11T07:56:33.535 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen/xen-gntalloc.ko 2024-01-11T07:56:33.535 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen/xen-gntdev.ko 2024-01-11T07:56:33.536 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen/xen-pciback 2024-01-11T07:56:33.536 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-01-11T07:56:33.536 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen/xen-privcmd.ko 2024-01-11T07:56:33.536 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen/xenfs 2024-01-11T07:56:33.536 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/drivers/xen/xenfs/xenfs.ko 2024-01-11T07:56:33.536 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs 2024-01-11T07:56:33.536 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/9p 2024-01-11T07:56:33.536 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/9p/9p.ko 2024-01-11T07:56:33.536 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/affs 2024-01-11T07:56:33.536 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/affs/affs.ko 2024-01-11T07:56:33.536 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/befs 2024-01-11T07:56:33.536 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/befs/befs.ko 2024-01-11T07:56:33.536 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/binfmt_misc.ko 2024-01-11T07:56:33.536 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/btrfs 2024-01-11T07:56:33.536 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/btrfs/btrfs.ko 2024-01-11T07:56:33.537 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/cachefiles 2024-01-11T07:56:33.537 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/cachefiles/cachefiles.ko 2024-01-11T07:56:33.537 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ceph 2024-01-11T07:56:33.537 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ceph/ceph.ko 2024-01-11T07:56:33.537 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/coda 2024-01-11T07:56:33.537 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/coda/coda.ko 2024-01-11T07:56:33.537 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/cramfs 2024-01-11T07:56:33.537 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/cramfs/cramfs.ko 2024-01-11T07:56:33.537 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/dlm 2024-01-11T07:56:33.537 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/dlm/dlm.ko 2024-01-11T07:56:33.537 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ecryptfs 2024-01-11T07:56:33.537 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ecryptfs/ecryptfs.ko 2024-01-11T07:56:33.537 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/f2fs 2024-01-11T07:56:33.537 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/f2fs/f2fs.ko 2024-01-11T07:56:33.538 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/fat 2024-01-11T07:56:33.538 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/fat/fat.ko 2024-01-11T07:56:33.538 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/fat/msdos.ko 2024-01-11T07:56:33.538 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/fat/vfat.ko 2024-01-11T07:56:33.538 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/fscache 2024-01-11T07:56:33.538 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/fscache/fscache.ko 2024-01-11T07:56:33.538 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/fuse 2024-01-11T07:56:33.538 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/fuse/cuse.ko 2024-01-11T07:56:33.538 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/fuse/fuse.ko 2024-01-11T07:56:33.538 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/gfs2 2024-01-11T07:56:33.538 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/gfs2/gfs2.ko 2024-01-11T07:56:33.538 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/hfs 2024-01-11T07:56:33.538 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/hfs/hfs.ko 2024-01-11T07:56:33.538 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/hfsplus 2024-01-11T07:56:33.538 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/hfsplus/hfsplus.ko 2024-01-11T07:56:33.539 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/isofs 2024-01-11T07:56:33.539 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/isofs/isofs.ko 2024-01-11T07:56:33.539 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/jfs 2024-01-11T07:56:33.539 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/jfs/jfs.ko 2024-01-11T07:56:33.539 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/lockd 2024-01-11T07:56:33.539 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/lockd/lockd.ko 2024-01-11T07:56:33.539 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/minix 2024-01-11T07:56:33.539 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/minix/minix.ko 2024-01-11T07:56:33.539 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/netfs 2024-01-11T07:56:33.539 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/netfs/netfs.ko 2024-01-11T07:56:33.539 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs 2024-01-11T07:56:33.539 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs/blocklayout 2024-01-11T07:56:33.539 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-01-11T07:56:33.539 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs/filelayout 2024-01-11T07:56:33.539 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-01-11T07:56:33.540 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs/flexfilelayout 2024-01-11T07:56:33.540 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-01-11T07:56:33.540 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs/nfs.ko 2024-01-11T07:56:33.540 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs/nfsv3.ko 2024-01-11T07:56:33.540 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs/nfsv4.ko 2024-01-11T07:56:33.540 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs_common 2024-01-11T07:56:33.540 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs_common/grace.ko 2024-01-11T07:56:33.540 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfs_common/nfs_acl.ko 2024-01-11T07:56:33.540 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfsd 2024-01-11T07:56:33.540 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nfsd/nfsd.ko 2024-01-11T07:56:33.540 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nilfs2 2024-01-11T07:56:33.540 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nilfs2/nilfs2.ko 2024-01-11T07:56:33.540 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls 2024-01-11T07:56:33.540 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-celtic.ko 2024-01-11T07:56:33.541 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-centeuro.ko 2024-01-11T07:56:33.541 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-croatian.ko 2024-01-11T07:56:33.541 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-cyrillic.ko 2024-01-11T07:56:33.541 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-gaelic.ko 2024-01-11T07:56:33.541 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-greek.ko 2024-01-11T07:56:33.541 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-iceland.ko 2024-01-11T07:56:33.541 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-inuit.ko 2024-01-11T07:56:33.541 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-roman.ko 2024-01-11T07:56:33.541 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-romanian.ko 2024-01-11T07:56:33.541 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/mac-turkish.ko 2024-01-11T07:56:33.541 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp1250.ko 2024-01-11T07:56:33.541 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp1251.ko 2024-01-11T07:56:33.541 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp1255.ko 2024-01-11T07:56:33.541 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp737.ko 2024-01-11T07:56:33.541 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp775.ko 2024-01-11T07:56:33.542 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp850.ko 2024-01-11T07:56:33.542 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp852.ko 2024-01-11T07:56:33.542 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp855.ko 2024-01-11T07:56:33.542 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp857.ko 2024-01-11T07:56:33.542 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp860.ko 2024-01-11T07:56:33.542 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp861.ko 2024-01-11T07:56:33.542 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp862.ko 2024-01-11T07:56:33.542 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp863.ko 2024-01-11T07:56:33.542 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp864.ko 2024-01-11T07:56:33.542 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp865.ko 2024-01-11T07:56:33.542 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp866.ko 2024-01-11T07:56:33.542 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp869.ko 2024-01-11T07:56:33.542 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp874.ko 2024-01-11T07:56:33.542 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp932.ko 2024-01-11T07:56:33.542 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp936.ko 2024-01-11T07:56:33.542 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp949.ko 2024-01-11T07:56:33.543 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_cp950.ko 2024-01-11T07:56:33.543 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_euc-jp.ko 2024-01-11T07:56:33.543 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-1.ko 2024-01-11T07:56:33.543 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-13.ko 2024-01-11T07:56:33.543 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-14.ko 2024-01-11T07:56:33.543 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-15.ko 2024-01-11T07:56:33.543 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-2.ko 2024-01-11T07:56:33.543 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-3.ko 2024-01-11T07:56:33.543 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-4.ko 2024-01-11T07:56:33.543 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-5.ko 2024-01-11T07:56:33.543 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-6.ko 2024-01-11T07:56:33.543 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-7.ko 2024-01-11T07:56:33.543 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_iso8859-9.ko 2024-01-11T07:56:33.543 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_koi8-r.ko 2024-01-11T07:56:33.544 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_koi8-ru.ko 2024-01-11T07:56:33.544 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_koi8-u.ko 2024-01-11T07:56:33.544 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_ucs2_utils.ko 2024-01-11T07:56:33.544 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/nls/nls_utf8.ko 2024-01-11T07:56:33.544 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2 2024-01-11T07:56:33.544 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/cluster 2024-01-11T07:56:33.544 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-01-11T07:56:33.544 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/dlm 2024-01-11T07:56:33.544 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-01-11T07:56:33.544 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/dlmfs 2024-01-11T07:56:33.544 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-01-11T07:56:33.544 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/ocfs2.ko 2024-01-11T07:56:33.544 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-01-11T07:56:33.544 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-01-11T07:56:33.544 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-01-11T07:56:33.545 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/overlayfs 2024-01-11T07:56:33.545 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/overlayfs/overlay.ko 2024-01-11T07:56:33.545 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/pstore 2024-01-11T07:56:33.545 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/pstore/ramoops.ko 2024-01-11T07:56:33.545 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/reiserfs 2024-01-11T07:56:33.545 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/reiserfs/reiserfs.ko 2024-01-11T07:56:33.545 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/romfs 2024-01-11T07:56:33.545 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/romfs/romfs.ko 2024-01-11T07:56:33.545 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/smb 2024-01-11T07:56:33.545 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/smb/client 2024-01-11T07:56:33.545 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/smb/client/cifs.ko 2024-01-11T07:56:33.545 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/smb/common 2024-01-11T07:56:33.545 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/smb/common/cifs_arc4.ko 2024-01-11T07:56:33.545 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/smb/common/cifs_md4.ko 2024-01-11T07:56:33.545 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/squashfs 2024-01-11T07:56:33.546 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/squashfs/squashfs.ko 2024-01-11T07:56:33.546 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/sysv 2024-01-11T07:56:33.546 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/sysv/sysv.ko 2024-01-11T07:56:33.546 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/udf 2024-01-11T07:56:33.546 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/udf/udf.ko 2024-01-11T07:56:33.546 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ufs 2024-01-11T07:56:33.546 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/ufs/ufs.ko 2024-01-11T07:56:33.546 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/xfs 2024-01-11T07:56:33.546 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/fs/xfs/xfs.ko 2024-01-11T07:56:33.546 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/kernel 2024-01-11T07:56:33.546 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/kernel/rcu 2024-01-11T07:56:33.546 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/kernel/rcu/rcutorture.ko 2024-01-11T07:56:33.546 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/kernel/torture.ko 2024-01-11T07:56:33.546 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/kernel/trace 2024-01-11T07:56:33.547 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-01-11T07:56:33.547 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib 2024-01-11T07:56:33.547 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/asn1_encoder.ko 2024-01-11T07:56:33.547 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/crc-itu-t.ko 2024-01-11T07:56:33.547 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/crc8.ko 2024-01-11T07:56:33.547 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/crypto 2024-01-11T07:56:33.547 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/crypto/libarc4.ko 2024-01-11T07:56:33.547 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/crypto/libdes.ko 2024-01-11T07:56:33.547 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/libcrc32c.ko 2024-01-11T07:56:33.547 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/lru_cache.ko 2024-01-11T07:56:33.547 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/lz4 2024-01-11T07:56:33.547 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/lz4/lz4_compress.ko 2024-01-11T07:56:33.547 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/lz4/lz4hc_compress.ko 2024-01-11T07:56:33.547 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/math 2024-01-11T07:56:33.547 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/math/cordic.ko 2024-01-11T07:56:33.548 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/raid6 2024-01-11T07:56:33.548 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/raid6/raid6_pq.ko 2024-01-11T07:56:33.548 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/reed_solomon 2024-01-11T07:56:33.548 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/reed_solomon/reed_solomon.ko 2024-01-11T07:56:33.548 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/ts_bm.ko 2024-01-11T07:56:33.548 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/ts_fsm.ko 2024-01-11T07:56:33.548 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/ts_kmp.ko 2024-01-11T07:56:33.548 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/zstd 2024-01-11T07:56:33.548 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/lib/zstd/zstd_compress.ko 2024-01-11T07:56:33.548 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/mm 2024-01-11T07:56:33.548 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/mm/hwpoison-inject.ko 2024-01-11T07:56:33.548 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net 2024-01-11T07:56:33.548 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/6lowpan 2024-01-11T07:56:33.548 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/6lowpan/6lowpan.ko 2024-01-11T07:56:33.548 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/6lowpan/nhc_dest.ko 2024-01-11T07:56:33.549 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/6lowpan/nhc_fragment.ko 2024-01-11T07:56:33.549 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/6lowpan/nhc_hop.ko 2024-01-11T07:56:33.549 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/6lowpan/nhc_ipv6.ko 2024-01-11T07:56:33.549 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/6lowpan/nhc_mobility.ko 2024-01-11T07:56:33.549 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/6lowpan/nhc_routing.ko 2024-01-11T07:56:33.549 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/6lowpan/nhc_udp.ko 2024-01-11T07:56:33.549 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/802 2024-01-11T07:56:33.549 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/802/garp.ko 2024-01-11T07:56:33.549 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/802/mrp.ko 2024-01-11T07:56:33.549 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/802/p8022.ko 2024-01-11T07:56:33.549 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/802/psnap.ko 2024-01-11T07:56:33.549 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/802/stp.ko 2024-01-11T07:56:33.549 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/8021q 2024-01-11T07:56:33.549 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/8021q/8021q.ko 2024-01-11T07:56:33.549 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/9p 2024-01-11T07:56:33.550 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/9p/9pnet.ko 2024-01-11T07:56:33.550 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/9p/9pnet_fd.ko 2024-01-11T07:56:33.550 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/9p/9pnet_rdma.ko 2024-01-11T07:56:33.550 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/9p/9pnet_virtio.ko 2024-01-11T07:56:33.550 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/batman-adv 2024-01-11T07:56:33.550 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/batman-adv/batman-adv.ko 2024-01-11T07:56:33.550 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge 2024-01-11T07:56:33.550 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/br_netfilter.ko 2024-01-11T07:56:33.550 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/bridge.ko 2024-01-11T07:56:33.550 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter 2024-01-11T07:56:33.550 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-01-11T07:56:33.550 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_among.ko 2024-01-11T07:56:33.550 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_arp.ko 2024-01-11T07:56:33.550 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-01-11T07:56:33.550 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-01-11T07:56:33.551 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_ip.ko 2024-01-11T07:56:33.551 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-01-11T07:56:33.551 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_limit.ko 2024-01-11T07:56:33.551 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_log.ko 2024-01-11T07:56:33.551 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_mark.ko 2024-01-11T07:56:33.551 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-01-11T07:56:33.551 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-01-11T07:56:33.551 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-01-11T07:56:33.551 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-01-11T07:56:33.551 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_snat.ko 2024-01-11T07:56:33.551 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_stp.ko 2024-01-11T07:56:33.551 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-01-11T07:56:33.551 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-01-11T07:56:33.551 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-01-11T07:56:33.551 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-01-11T07:56:33.552 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/ebtables.ko 2024-01-11T07:56:33.552 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-01-11T07:56:33.552 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-01-11T07:56:33.552 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-01-11T07:56:33.552 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ceph 2024-01-11T07:56:33.552 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ceph/libceph.ko 2024-01-11T07:56:33.552 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/core 2024-01-11T07:56:33.552 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/core/failover.ko 2024-01-11T07:56:33.552 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/core/pktgen.ko 2024-01-11T07:56:33.552 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dccp 2024-01-11T07:56:33.552 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dccp/dccp.ko 2024-01-11T07:56:33.552 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dccp/dccp_diag.ko 2024-01-11T07:56:33.552 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dccp/dccp_ipv4.ko 2024-01-11T07:56:33.552 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dccp/dccp_ipv6.ko 2024-01-11T07:56:33.552 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dns_resolver 2024-01-11T07:56:33.553 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dns_resolver/dns_resolver.ko 2024-01-11T07:56:33.553 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dsa 2024-01-11T07:56:33.553 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dsa/dsa_core.ko 2024-01-11T07:56:33.553 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dsa/tag_brcm.ko 2024-01-11T07:56:33.553 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dsa/tag_dsa.ko 2024-01-11T07:56:33.553 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dsa/tag_none.ko 2024-01-11T07:56:33.553 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/dsa/tag_trailer.ko 2024-01-11T07:56:33.553 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ieee802154 2024-01-11T07:56:33.553 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ieee802154/6lowpan 2024-01-11T07:56:33.553 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-01-11T07:56:33.553 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ieee802154/ieee802154.ko 2024-01-11T07:56:33.553 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ieee802154/ieee802154_socket.ko 2024-01-11T07:56:33.553 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4 2024-01-11T07:56:33.553 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/ah4.ko 2024-01-11T07:56:33.553 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/esp4.ko 2024-01-11T07:56:33.554 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/fou.ko 2024-01-11T07:56:33.554 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/gre.ko 2024-01-11T07:56:33.554 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/inet_diag.ko 2024-01-11T07:56:33.554 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/ip_gre.ko 2024-01-11T07:56:33.554 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/ip_tunnel.ko 2024-01-11T07:56:33.554 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/ip_vti.ko 2024-01-11T07:56:33.554 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/ipcomp.ko 2024-01-11T07:56:33.554 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/ipip.ko 2024-01-11T07:56:33.554 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter 2024-01-11T07:56:33.554 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/arp_tables.ko 2024-01-11T07:56:33.554 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-01-11T07:56:33.554 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-01-11T07:56:33.554 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-01-11T07:56:33.554 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-01-11T07:56:33.554 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-01-11T07:56:33.555 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-01-11T07:56:33.555 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-01-11T07:56:33.555 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-01-11T07:56:33.555 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-01-11T07:56:33.555 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/iptable_security.ko 2024-01-11T07:56:33.555 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-01-11T07:56:33.555 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-01-11T07:56:33.555 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-01-11T07:56:33.555 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-01-11T07:56:33.555 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-01-11T07:56:33.555 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-01-11T07:56:33.555 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-01-11T07:56:33.555 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-01-11T07:56:33.555 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-01-11T07:56:33.555 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-01-11T07:56:33.555 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_bic.ko 2024-01-11T07:56:33.556 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_dctcp.ko 2024-01-11T07:56:33.556 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_diag.ko 2024-01-11T07:56:33.556 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_highspeed.ko 2024-01-11T07:56:33.556 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_htcp.ko 2024-01-11T07:56:33.556 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_hybla.ko 2024-01-11T07:56:33.556 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_illinois.ko 2024-01-11T07:56:33.556 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_lp.ko 2024-01-11T07:56:33.556 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_scalable.ko 2024-01-11T07:56:33.556 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_vegas.ko 2024-01-11T07:56:33.556 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_veno.ko 2024-01-11T07:56:33.556 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_westwood.ko 2024-01-11T07:56:33.556 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tcp_yeah.ko 2024-01-11T07:56:33.556 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/tunnel4.ko 2024-01-11T07:56:33.556 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/udp_diag.ko 2024-01-11T07:56:33.557 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/udp_tunnel.ko 2024-01-11T07:56:33.557 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv4/xfrm4_tunnel.ko 2024-01-11T07:56:33.557 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6 2024-01-11T07:56:33.557 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/ah6.ko 2024-01-11T07:56:33.557 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/esp6.ko 2024-01-11T07:56:33.557 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/fou6.ko 2024-01-11T07:56:33.557 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/ip6_tunnel.ko 2024-01-11T07:56:33.557 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-01-11T07:56:33.557 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/ip6_vti.ko 2024-01-11T07:56:33.557 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/ipcomp6.ko 2024-01-11T07:56:33.557 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter 2024-01-11T07:56:33.557 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-01-11T07:56:33.557 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-01-11T07:56:33.557 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-01-11T07:56:33.557 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-01-11T07:56:33.558 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-01-11T07:56:33.558 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-01-11T07:56:33.558 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-01-11T07:56:33.558 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-01-11T07:56:33.558 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-01-11T07:56:33.558 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-01-11T07:56:33.558 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-01-11T07:56:33.558 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-01-11T07:56:33.558 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-01-11T07:56:33.558 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-01-11T07:56:33.558 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-01-11T07:56:33.558 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-01-11T07:56:33.558 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-01-11T07:56:33.558 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-01-11T07:56:33.558 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-01-11T07:56:33.559 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-01-11T07:56:33.559 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-01-11T07:56:33.559 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-01-11T07:56:33.559 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-01-11T07:56:33.559 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-01-11T07:56:33.559 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/sit.ko 2024-01-11T07:56:33.559 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/tunnel6.ko 2024-01-11T07:56:33.559 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/ipv6/xfrm6_tunnel.ko 2024-01-11T07:56:33.559 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/key 2024-01-11T07:56:33.559 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/key/af_key.ko 2024-01-11T07:56:33.559 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/l2tp 2024-01-11T07:56:33.559 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/l2tp/l2tp_core.ko 2024-01-11T07:56:33.559 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/l2tp/l2tp_debugfs.ko 2024-01-11T07:56:33.559 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/l2tp/l2tp_eth.ko 2024-01-11T07:56:33.559 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/l2tp/l2tp_ip.ko 2024-01-11T07:56:33.560 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/l2tp/l2tp_ip6.ko 2024-01-11T07:56:33.560 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/l2tp/l2tp_netlink.ko 2024-01-11T07:56:33.560 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/l2tp/l2tp_ppp.ko 2024-01-11T07:56:33.560 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/llc 2024-01-11T07:56:33.560 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/llc/llc.ko 2024-01-11T07:56:33.560 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/mac802154 2024-01-11T07:56:33.560 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/mac802154/mac802154.ko 2024-01-11T07:56:33.560 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/mpls 2024-01-11T07:56:33.560 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/mpls/mpls_gso.ko 2024-01-11T07:56:33.560 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter 2024-01-11T07:56:33.560 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset 2024-01-11T07:56:33.560 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set.ko 2024-01-11T07:56:33.560 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-01-11T07:56:33.560 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-01-11T07:56:33.561 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-01-11T07:56:33.561 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-01-11T07:56:33.561 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-01-11T07:56:33.561 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-01-11T07:56:33.561 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-01-11T07:56:33.561 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-01-11T07:56:33.561 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-01-11T07:56:33.561 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-01-11T07:56:33.561 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-01-11T07:56:33.561 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-01-11T07:56:33.561 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-01-11T07:56:33.561 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-01-11T07:56:33.561 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-01-11T07:56:33.562 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs 2024-01-11T07:56:33.562 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs.ko 2024-01-11T07:56:33.562 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-01-11T07:56:33.562 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-01-11T07:56:33.562 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-01-11T07:56:33.562 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-01-11T07:56:33.562 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-01-11T07:56:33.562 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-01-11T07:56:33.562 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-01-11T07:56:33.562 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-01-11T07:56:33.562 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-01-11T07:56:33.562 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-01-11T07:56:33.562 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-01-11T07:56:33.562 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-01-11T07:56:33.562 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-01-11T07:56:33.563 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conncount.ko 2024-01-11T07:56:33.563 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack.ko 2024-01-11T07:56:33.563 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-01-11T07:56:33.563 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-01-11T07:56:33.563 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-01-11T07:56:33.563 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_h323.ko 2024-01-11T07:56:33.563 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_irc.ko 2024-01-11T07:56:33.563 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-01-11T07:56:33.563 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-01-11T07:56:33.563 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-01-11T07:56:33.563 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_sane.ko 2024-01-11T07:56:33.563 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_sip.ko 2024-01-11T07:56:33.563 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-01-11T07:56:33.563 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-01-11T07:56:33.563 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_dup_netdev.ko 2024-01-11T07:56:33.563 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_flow_table.ko 2024-01-11T07:56:33.564 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_flow_table_inet.ko 2024-01-11T07:56:33.564 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_log_syslog.ko 2024-01-11T07:56:33.564 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_nat.ko 2024-01-11T07:56:33.564 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_nat_amanda.ko 2024-01-11T07:56:33.564 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_nat_ftp.ko 2024-01-11T07:56:33.564 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_nat_irc.ko 2024-01-11T07:56:33.564 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_nat_sip.ko 2024-01-11T07:56:33.564 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_nat_tftp.ko 2024-01-11T07:56:33.564 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_synproxy_core.ko 2024-01-11T07:56:33.564 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nf_tables.ko 2024-01-11T07:56:33.564 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nfnetlink.ko 2024-01-11T07:56:33.564 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nfnetlink_acct.ko 2024-01-11T07:56:33.564 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nfnetlink_log.ko 2024-01-11T07:56:33.564 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nfnetlink_osf.ko 2024-01-11T07:56:33.565 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nfnetlink_queue.ko 2024-01-11T07:56:33.565 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_chain_nat.ko 2024-01-11T07:56:33.565 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_compat.ko 2024-01-11T07:56:33.565 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_ct.ko 2024-01-11T07:56:33.565 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_dup_netdev.ko 2024-01-11T07:56:33.565 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_fib.ko 2024-01-11T07:56:33.565 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_fwd_netdev.ko 2024-01-11T07:56:33.565 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_hash.ko 2024-01-11T07:56:33.565 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_limit.ko 2024-01-11T07:56:33.565 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_log.ko 2024-01-11T07:56:33.565 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_masq.ko 2024-01-11T07:56:33.565 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_nat.ko 2024-01-11T07:56:33.565 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_numgen.ko 2024-01-11T07:56:33.565 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_queue.ko 2024-01-11T07:56:33.566 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_quota.ko 2024-01-11T07:56:33.566 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_redir.ko 2024-01-11T07:56:33.566 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_reject.ko 2024-01-11T07:56:33.566 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_reject_inet.ko 2024-01-11T07:56:33.566 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_socket.ko 2024-01-11T07:56:33.566 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/nft_tunnel.ko 2024-01-11T07:56:33.566 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_AUDIT.ko 2024-01-11T07:56:33.566 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_CHECKSUM.ko 2024-01-11T07:56:33.566 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_CLASSIFY.ko 2024-01-11T07:56:33.566 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-01-11T07:56:33.566 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_CT.ko 2024-01-11T07:56:33.566 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_DSCP.ko 2024-01-11T07:56:33.566 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_HL.ko 2024-01-11T07:56:33.567 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_HMARK.ko 2024-01-11T07:56:33.567 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_IDLETIMER.ko 2024-01-11T07:56:33.567 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_LED.ko 2024-01-11T07:56:33.567 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_LOG.ko 2024-01-11T07:56:33.567 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_MASQUERADE.ko 2024-01-11T07:56:33.567 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_NETMAP.ko 2024-01-11T07:56:33.567 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_NFLOG.ko 2024-01-11T07:56:33.567 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_NFQUEUE.ko 2024-01-11T07:56:33.567 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_RATEEST.ko 2024-01-11T07:56:33.567 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_REDIRECT.ko 2024-01-11T07:56:33.567 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_SECMARK.ko 2024-01-11T07:56:33.567 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_TCPMSS.ko 2024-01-11T07:56:33.567 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-01-11T07:56:33.567 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_TEE.ko 2024-01-11T07:56:33.567 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_TPROXY.ko 2024-01-11T07:56:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_TRACE.ko 2024-01-11T07:56:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_addrtype.ko 2024-01-11T07:56:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_bpf.ko 2024-01-11T07:56:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_cgroup.ko 2024-01-11T07:56:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_cluster.ko 2024-01-11T07:56:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_comment.ko 2024-01-11T07:56:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_connbytes.ko 2024-01-11T07:56:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_connlabel.ko 2024-01-11T07:56:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_connlimit.ko 2024-01-11T07:56:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_connmark.ko 2024-01-11T07:56:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_conntrack.ko 2024-01-11T07:56:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_cpu.ko 2024-01-11T07:56:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_dccp.ko 2024-01-11T07:56:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_devgroup.ko 2024-01-11T07:56:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_dscp.ko 2024-01-11T07:56:33.569 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_ecn.ko 2024-01-11T07:56:33.569 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_esp.ko 2024-01-11T07:56:33.569 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_hashlimit.ko 2024-01-11T07:56:33.569 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_helper.ko 2024-01-11T07:56:33.569 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_hl.ko 2024-01-11T07:56:33.569 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_ipcomp.ko 2024-01-11T07:56:33.569 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_iprange.ko 2024-01-11T07:56:33.569 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_ipvs.ko 2024-01-11T07:56:33.569 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_l2tp.ko 2024-01-11T07:56:33.569 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_length.ko 2024-01-11T07:56:33.569 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_limit.ko 2024-01-11T07:56:33.569 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_mac.ko 2024-01-11T07:56:33.569 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_mark.ko 2024-01-11T07:56:33.569 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_multiport.ko 2024-01-11T07:56:33.569 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_nat.ko 2024-01-11T07:56:33.570 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_nfacct.ko 2024-01-11T07:56:33.570 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_osf.ko 2024-01-11T07:56:33.570 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_owner.ko 2024-01-11T07:56:33.570 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_physdev.ko 2024-01-11T07:56:33.570 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_pkttype.ko 2024-01-11T07:56:33.570 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_policy.ko 2024-01-11T07:56:33.570 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_quota.ko 2024-01-11T07:56:33.570 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_rateest.ko 2024-01-11T07:56:33.570 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_realm.ko 2024-01-11T07:56:33.570 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_recent.ko 2024-01-11T07:56:33.570 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_sctp.ko 2024-01-11T07:56:33.570 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_set.ko 2024-01-11T07:56:33.570 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_socket.ko 2024-01-11T07:56:33.570 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_state.ko 2024-01-11T07:56:33.570 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_statistic.ko 2024-01-11T07:56:33.571 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_string.ko 2024-01-11T07:56:33.571 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_tcpmss.ko 2024-01-11T07:56:33.571 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_time.ko 2024-01-11T07:56:33.571 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netfilter/xt_u32.ko 2024-01-11T07:56:33.571 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netlink 2024-01-11T07:56:33.571 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/netlink/netlink_diag.ko 2024-01-11T07:56:33.571 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/nsh 2024-01-11T07:56:33.571 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/nsh/nsh.ko 2024-01-11T07:56:33.571 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/openvswitch 2024-01-11T07:56:33.571 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/openvswitch/openvswitch.ko 2024-01-11T07:56:33.571 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/openvswitch/vport-geneve.ko 2024-01-11T07:56:33.571 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/openvswitch/vport-gre.ko 2024-01-11T07:56:33.571 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/openvswitch/vport-vxlan.ko 2024-01-11T07:56:33.571 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/packet 2024-01-11T07:56:33.571 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/packet/af_packet_diag.ko 2024-01-11T07:56:33.572 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/rds 2024-01-11T07:56:33.572 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/rds/rds.ko 2024-01-11T07:56:33.572 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/rds/rds_rdma.ko 2024-01-11T07:56:33.572 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/rds/rds_tcp.ko 2024-01-11T07:56:33.572 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched 2024-01-11T07:56:33.572 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_csum.ko 2024-01-11T07:56:33.572 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_gact.ko 2024-01-11T07:56:33.572 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_ipt.ko 2024-01-11T07:56:33.572 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_mirred.ko 2024-01-11T07:56:33.572 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_nat.ko 2024-01-11T07:56:33.572 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_pedit.ko 2024-01-11T07:56:33.572 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_police.ko 2024-01-11T07:56:33.572 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_simple.ko 2024-01-11T07:56:33.572 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_skbedit.ko 2024-01-11T07:56:33.573 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/act_vlan.ko 2024-01-11T07:56:33.573 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/cls_basic.ko 2024-01-11T07:56:33.573 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/cls_bpf.ko 2024-01-11T07:56:33.573 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/cls_flow.ko 2024-01-11T07:56:33.573 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/cls_fw.ko 2024-01-11T07:56:33.573 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/cls_route.ko 2024-01-11T07:56:33.573 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/cls_u32.ko 2024-01-11T07:56:33.573 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/em_cmp.ko 2024-01-11T07:56:33.573 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/em_ipset.ko 2024-01-11T07:56:33.573 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/em_meta.ko 2024-01-11T07:56:33.573 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/em_nbyte.ko 2024-01-11T07:56:33.573 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/em_text.ko 2024-01-11T07:56:33.573 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/em_u32.ko 2024-01-11T07:56:33.573 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_choke.ko 2024-01-11T07:56:33.573 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_codel.ko 2024-01-11T07:56:33.574 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_drr.ko 2024-01-11T07:56:33.574 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_fq.ko 2024-01-11T07:56:33.574 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_gred.ko 2024-01-11T07:56:33.574 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_hfsc.ko 2024-01-11T07:56:33.574 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_hhf.ko 2024-01-11T07:56:33.574 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_htb.ko 2024-01-11T07:56:33.574 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_ingress.ko 2024-01-11T07:56:33.574 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_mqprio.ko 2024-01-11T07:56:33.574 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_mqprio_lib.ko 2024-01-11T07:56:33.574 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_multiq.ko 2024-01-11T07:56:33.574 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_netem.ko 2024-01-11T07:56:33.574 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_pie.ko 2024-01-11T07:56:33.574 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_plug.ko 2024-01-11T07:56:33.574 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_prio.ko 2024-01-11T07:56:33.574 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_qfq.ko 2024-01-11T07:56:33.575 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_red.ko 2024-01-11T07:56:33.575 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_sfb.ko 2024-01-11T07:56:33.575 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_sfq.ko 2024-01-11T07:56:33.575 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_tbf.ko 2024-01-11T07:56:33.575 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sched/sch_teql.ko 2024-01-11T07:56:33.575 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sctp 2024-01-11T07:56:33.575 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sctp/sctp.ko 2024-01-11T07:56:33.575 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sctp/sctp_diag.ko 2024-01-11T07:56:33.575 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sunrpc 2024-01-11T07:56:33.575 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sunrpc/auth_gss 2024-01-11T07:56:33.575 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-01-11T07:56:33.575 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-01-11T07:56:33.575 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sunrpc/sunrpc.ko 2024-01-11T07:56:33.575 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sunrpc/xprtrdma 2024-01-11T07:56:33.575 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-01-11T07:56:33.576 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/tipc 2024-01-11T07:56:33.576 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/tipc/diag.ko 2024-01-11T07:56:33.576 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/tipc/tipc.ko 2024-01-11T07:56:33.576 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/unix 2024-01-11T07:56:33.576 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/unix/unix_diag.ko 2024-01-11T07:56:33.576 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/vmw_vsock 2024-01-11T07:56:33.576 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-01-11T07:56:33.576 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-01-11T07:56:33.576 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/vmw_vsock/vsock.ko 2024-01-11T07:56:33.576 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/vmw_vsock/vsock_diag.ko 2024-01-11T07:56:33.576 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/vmw_vsock/vsock_loopback.ko 2024-01-11T07:56:33.576 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/xfrm 2024-01-11T07:56:33.576 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/net/xfrm/xfrm_ipcomp.ko 2024-01-11T07:56:33.577 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/security 2024-01-11T07:56:33.577 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/security/keys 2024-01-11T07:56:33.577 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/security/keys/encrypted-keys 2024-01-11T07:56:33.577 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-01-11T07:56:33.577 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/security/keys/trusted-keys 2024-01-11T07:56:33.577 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/security/keys/trusted-keys/trusted.ko 2024-01-11T07:56:33.577 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/virt 2024-01-11T07:56:33.577 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/virt/lib 2024-01-11T07:56:33.577 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/kernel/virt/lib/irqbypass.ko 2024-01-11T07:56:33.577 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.alias 2024-01-11T07:56:33.577 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.alias.bin 2024-01-11T07:56:33.577 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.builtin 2024-01-11T07:56:33.577 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.builtin.bin 2024-01-11T07:56:33.578 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.builtin.modinfo 2024-01-11T07:56:33.578 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.dep 2024-01-11T07:56:33.578 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.dep.bin 2024-01-11T07:56:33.578 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.devname 2024-01-11T07:56:33.578 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.order 2024-01-11T07:56:33.578 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.softdep 2024-01-11T07:56:33.578 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.symbols 2024-01-11T07:56:33.578 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.6.0-gc719ee96ba78/modules.symbols.bin 2024-01-11T07:56:33.579 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-gc719ee96ba78 2024-01-11T07:56:33.579 DEBUG:teuthology.orchestra.run.smithi027:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-01-11T07:56:33.608 INFO:teuthology.orchestra.run.smithi027.stdout:postinstall scriptlet (using /bin/sh): 2024-01-11T07:56:33.609 INFO:teuthology.orchestra.run.smithi027.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-gc719ee96ba78 -a -r /boot/System.map-6.6.0-gc719ee96ba78 ]; then 2024-01-11T07:56:33.609 INFO:teuthology.orchestra.run.smithi027.stdout:cp /boot/vmlinuz-6.6.0-gc719ee96ba78 /boot/.vmlinuz-6.6.0-gc719ee96ba78-rpm 2024-01-11T07:56:33.609 INFO:teuthology.orchestra.run.smithi027.stdout:cp /boot/System.map-6.6.0-gc719ee96ba78 /boot/.System.map-6.6.0-gc719ee96ba78-rpm 2024-01-11T07:56:33.609 INFO:teuthology.orchestra.run.smithi027.stdout:rm -f /boot/vmlinuz-6.6.0-gc719ee96ba78 /boot/System.map-6.6.0-gc719ee96ba78 2024-01-11T07:56:33.609 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/installkernel 6.6.0-gc719ee96ba78 /boot/.vmlinuz-6.6.0-gc719ee96ba78-rpm /boot/.System.map-6.6.0-gc719ee96ba78-rpm 2024-01-11T07:56:33.609 INFO:teuthology.orchestra.run.smithi027.stdout:rm -f /boot/.vmlinuz-6.6.0-gc719ee96ba78-rpm /boot/.System.map-6.6.0-gc719ee96ba78-rpm 2024-01-11T07:56:33.609 INFO:teuthology.orchestra.run.smithi027.stdout:fi 2024-01-11T07:56:33.609 INFO:teuthology.orchestra.run.smithi027.stdout:preuninstall scriptlet (using /bin/sh): 2024-01-11T07:56:33.609 INFO:teuthology.orchestra.run.smithi027.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-01-11T07:56:33.609 INFO:teuthology.orchestra.run.smithi027.stdout:new-kernel-pkg --remove 6.6.0-gc719ee96ba78 --rminitrd --initrdfile=/boot/initramfs-6.6.0-gc719ee96ba78.img 2024-01-11T07:56:33.609 INFO:teuthology.orchestra.run.smithi027.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-01-11T07:56:33.609 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-install remove 6.6.0-gc719ee96ba78 2024-01-11T07:56:33.609 INFO:teuthology.orchestra.run.smithi027.stdout:fi 2024-01-11T07:56:33.610 INFO:teuthology.orchestra.run.smithi027.stdout:postuninstall scriptlet (using /bin/sh): 2024-01-11T07:56:33.610 INFO:teuthology.orchestra.run.smithi027.stdout:if [ -x /sbin/update-bootloader ]; then 2024-01-11T07:56:33.610 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/update-bootloader --remove 6.6.0-gc719ee96ba78 2024-01-11T07:56:33.610 INFO:teuthology.orchestra.run.smithi027.stdout:fi 2024-01-11T07:56:33.611 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -qi grub2-tools 2024-01-11T07:56:33.690 INFO:teuthology.orchestra.run.smithi027.stdout:Name : grub2-tools 2024-01-11T07:56:33.691 INFO:teuthology.orchestra.run.smithi027.stdout:Epoch : 1 2024-01-11T07:56:33.691 INFO:teuthology.orchestra.run.smithi027.stdout:Version : 2.02 2024-01-11T07:56:33.691 INFO:teuthology.orchestra.run.smithi027.stdout:Release : 129.el8 2024-01-11T07:56:33.691 INFO:teuthology.orchestra.run.smithi027.stdout:Architecture: x86_64 2024-01-11T07:56:33.691 INFO:teuthology.orchestra.run.smithi027.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-11T07:56:33.691 INFO:teuthology.orchestra.run.smithi027.stdout:Group : System Environment/Base 2024-01-11T07:56:33.691 INFO:teuthology.orchestra.run.smithi027.stdout:Size : 9265586 2024-01-11T07:56:33.691 INFO:teuthology.orchestra.run.smithi027.stdout:License : GPLv3+ 2024-01-11T07:56:33.691 INFO:teuthology.orchestra.run.smithi027.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-11T07:56:33.691 INFO:teuthology.orchestra.run.smithi027.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-11T07:56:33.691 INFO:teuthology.orchestra.run.smithi027.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-11T07:56:33.691 INFO:teuthology.orchestra.run.smithi027.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-11T07:56:33.692 INFO:teuthology.orchestra.run.smithi027.stdout:Relocations : (not relocatable) 2024-01-11T07:56:33.692 INFO:teuthology.orchestra.run.smithi027.stdout:Packager : CentOS BuildSystem 2024-01-11T07:56:33.692 INFO:teuthology.orchestra.run.smithi027.stdout:Vendor : CentOS 2024-01-11T07:56:33.692 INFO:teuthology.orchestra.run.smithi027.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-11T07:56:33.692 INFO:teuthology.orchestra.run.smithi027.stdout:Summary : Support tools for GRUB. 2024-01-11T07:56:33.692 INFO:teuthology.orchestra.run.smithi027.stdout:Description : 2024-01-11T07:56:33.692 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T07:56:33.692 INFO:teuthology.orchestra.run.smithi027.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-11T07:56:33.692 INFO:teuthology.orchestra.run.smithi027.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-11T07:56:33.692 INFO:teuthology.orchestra.run.smithi027.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-11T07:56:33.692 INFO:teuthology.orchestra.run.smithi027.stdout:hardware devices. 2024-01-11T07:56:33.692 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T07:56:33.693 INFO:teuthology.orchestra.run.smithi027.stdout:This subpackage provides tools for support of all platforms. 2024-01-11T07:56:33.693 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-11T07:56:33.693 INFO:teuthology.task.kernel:Updating grub on smithi027 to boot 6.6.0-gc719ee96ba78 2024-01-11T07:56:33.694 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-11T07:56:36.387 INFO:teuthology.orchestra.run.smithi027.stderr:Generating grub configuration file ... 2024-01-11T07:56:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:done 2024-01-11T07:56:38.317 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-01-11T07:56:38.350 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.8IR6VNVOsO 2024-01-11T07:56:38.350 DEBUG:teuthology.orchestra.run.smithi027:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.8IR6VNVOsO 2024-01-11T07:56:38.422 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.8IR6VNVOsO 2024-01-11T07:56:38.596 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.8IR6VNVOsO is 6KB 2024-01-11T07:56:38.645 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.8IR6VNVOsO 2024-01-11T07:56:38.663 DEBUG:teuthology.orchestra.run.smithi027:> sudo /bin/ls /boot/loader/entries || true 2024-01-11T07:56:38.735 INFO:teuthology.orchestra.run.smithi027.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-11T07:56:38.735 INFO:teuthology.orchestra.run.smithi027.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-11T07:56:38.735 INFO:teuthology.orchestra.run.smithi027.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-11T07:56:38.735 INFO:teuthology.orchestra.run.smithi027.stdout:a1b29d6c289747bdb22930598d39874a-6.6.0-gc719ee96ba78.conf 2024-01-11T07:56:38.736 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-6.6.0-gc719ee96ba78 2024-01-11T07:56:38.923 DEBUG:teuthology.orchestra.run.smithi027:> sudo shutdown -r now 2024-01-11T07:56:55.708 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-01-11T07:56:55.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-11T07:56:55.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T07:57:08.940 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-01-11T07:57:08.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-01-11T07:57:08.941 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T07:57:14.172 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-11T07:57:23.180 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-11T07:57:23.181 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T07:57:26.236 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-01-11T07:57:27.484 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-01-11T07:57:36.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-01-11T07:57:36.493 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T07:57:38.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-11T07:57:38.240 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T07:57:39.548 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-01-11T07:57:51.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-01-11T07:57:51.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T07:57:54.625 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-01-11T07:58:09.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-01-11T07:58:09.637 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T07:58:10.016 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-01-11T07:58:10.449 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-11T07:58:10.449 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.6.0-gc719ee96ba78"... 2024-01-11T07:58:10.450 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-01-11T07:58:10.494 INFO:teuthology.orchestra.run.smithi077.stdout:6.6.0-gc719ee96ba78 2024-01-11T07:58:10.495 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-gc719ee96ba78 vs 6.6.0-gc719ee96ba78 2024-01-11T07:58:10.495 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-11T07:58:10.495 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-11T07:58:11.496 DEBUG:teuthology.parallel:result is None 2024-01-11T07:58:28.156 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-01-11T07:58:46.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-01-11T07:58:46.158 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T07:58:46.412 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-01-11T07:58:46.812 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-01-11T07:58:46.812 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.6.0-gc719ee96ba78"... 2024-01-11T07:58:46.813 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-01-11T07:58:46.859 INFO:teuthology.orchestra.run.smithi027.stdout:6.6.0-gc719ee96ba78 2024-01-11T07:58:46.859 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-gc719ee96ba78 vs 6.6.0-gc719ee96ba78 2024-01-11T07:58:46.859 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-11T07:58:46.859 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-11T07:58:47.860 DEBUG:teuthology.parallel:result is None 2024-01-11T07:58:47.860 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-11T07:58:47.871 INFO:teuthology.task.internal:Creating test directory... 2024-01-11T07:58:47.871 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-11T07:58:47.876 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-11T07:58:47.927 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-11T07:58:47.934 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-11T07:58:47.961 INFO:teuthology.task.internal:Creating archive directory... 2024-01-11T07:58:47.961 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-11T07:58:47.965 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-11T07:58:48.036 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-11T07:58:48.043 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-11T07:58:48.044 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-11T07:58:48.048 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-11T07:58:48.172 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-11T07:58:48.173 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-11T07:58:48.211 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-11T07:58:48.235 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-11T07:58:48.238 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-11T07:58:48.245 INFO:teuthology.task.internal:Configuring sudo... 2024-01-11T07:58:48.245 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-11T07:58:48.250 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-11T07:58:48.320 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-11T07:58:48.329 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-11T07:58:48.330 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-11T07:58:48.334 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-11T07:58:48.379 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-11T07:58:48.494 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-11T07:58:48.586 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-11T07:58:48.662 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-11T07:58:48.752 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T07:58:48.752 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-11T07:58:48.824 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-11T07:58:48.900 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-11T07:58:48.981 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-11T07:58:49.055 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-11T07:58:49.124 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T07:58:49.124 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-11T07:58:49.194 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2024-01-11T07:58:49.199 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2024-01-11T07:58:49.309 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-11T07:58:49.362 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-11T07:58:49.794 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-11T07:58:49.801 INFO:teuthology.task.internal:Starting timer... 2024-01-11T07:58:49.801 INFO:teuthology.run_tasks:Running task pcp... 2024-01-11T07:58:49.839 INFO:teuthology.run_tasks:Running task selinux... 2024-01-11T07:58:49.873 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-11T07:58:49.873 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2024-01-11T07:58:50.019 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2024-01-11T07:58:50.022 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2024-01-11T07:58:50.159 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: 2024-01-11T07:58:50.162 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-11T07:58:50.162 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2024-01-11T07:58:50.217 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2024-01-11T07:58:50.217 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2024-01-11T07:58:50.265 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2024-01-11T07:58:50.265 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi027.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi077.front.sepia.ceph.com': 'permissive'} 2024-01-11T07:58:50.266 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-11T07:58:50.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T07:58:50.339 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-11T07:58:50.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T07:58:50.409 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-11T07:58:50.409 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2024-01-11T07:58:50.481 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2024-01-11T07:58:50.560 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-11T07:58:50.571 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-11T07:58:50.589 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-11T07:58:50.591 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com,smithi077.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-11T08:06:24.219 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi077.front.sepia.ceph.com')] 2024-01-11T08:06:24.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-01-11T08:06:24.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T08:06:24.317 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-01-11T08:06:24.434 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-01-11T08:06:24.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-11T08:06:24.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T08:06:24.529 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-01-11T08:06:24.641 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-11T08:06:24.642 INFO:teuthology.run_tasks:Running task clock... 2024-01-11T08:06:24.653 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-11T08:06:24.653 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-11T08:06:24.653 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-11T08:06:24.658 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-11T08:06:24.658 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-11T08:06:24.733 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-11T08:06:24.734 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-11T08:06:24.766 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-11T08:06:24.768 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-11T08:06:24.856 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2024-01-11T08:06:24.862 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2024-01-11T08:06:24.883 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2024-01-11T08:06:24.891 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2024-01-11T08:06:24.916 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-11T08:06:24.926 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-11T08:06:24.948 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-11T08:06:24.960 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-11T08:06:25.032 INFO:teuthology.orchestra.run.smithi077.stderr:bash: ntpq: command not found 2024-01-11T08:06:25.039 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-11T08:06:25.039 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-01-11T08:06:25.039 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T08:06:25.039 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T08:06:25.039 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T08:06:25.039 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T08:06:25.063 INFO:teuthology.orchestra.run.smithi027.stderr:bash: ntpq: command not found 2024-01-11T08:06:25.070 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-11T08:06:25.071 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-01-11T08:06:25.071 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T08:06:25.071 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T08:06:25.071 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T08:06:25.071 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T08:06:25.072 INFO:teuthology.run_tasks:Running task install... 2024-01-11T08:06:25.081 DEBUG:teuthology.task.install:project ceph 2024-01-11T08:06:25.081 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4bcedb9fc11bed1fd6fd0c4fd59187401f309680'}} 2024-01-11T08:06:25.081 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '4bcedb9fc11bed1fd6fd0c4fd59187401f309680'} 2024-01-11T08:06:25.081 INFO:teuthology.task.install:Using flavor: default 2024-01-11T08:06:25.088 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-11T08:06:25.088 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-01-11T08:06:25.088 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4bcedb9fc11bed1fd6fd0c4fd59187401f309680', 'tag': None, 'wait_for_package': False} 2024-01-11T08:06:25.089 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4bcedb9fc11bed1fd6fd0c4fd59187401f309680 2024-01-11T08:06:25.091 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4bcedb9fc11bed1fd6fd0c4fd59187401f309680', 'tag': None, 'wait_for_package': False} 2024-01-11T08:06:25.091 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4bcedb9fc11bed1fd6fd0c4fd59187401f309680 2024-01-11T08:06:25.241 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing1-quincy-2024-01-10-2010-quincy/4bcedb9fc11bed1fd6fd0c4fd59187401f309680/centos/8/flavors/default/ 2024-01-11T08:06:25.241 INFO:teuthology.task.install.rpm:Package version is 17.2.7-299.g4bcedb9f 2024-01-11T08:06:25.489 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing1-quincy-2024-01-10-2010-quincy/4bcedb9fc11bed1fd6fd0c4fd59187401f309680/centos/8/flavors/default/ 2024-01-11T08:06:25.489 INFO:teuthology.task.install.rpm:Package version is 17.2.7-299.g4bcedb9f 2024-01-11T08:06:25.538 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing1-quincy-2024-01-10-2010-quincy/4bcedb9fc11bed1fd6fd0c4fd59187401f309680/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing1-quincy-2024-01-10-2010-quincy/4bcedb9fc11bed1fd6fd0c4fd59187401f309680/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing1-quincy-2024-01-10-2010-quincy/4bcedb9fc11bed1fd6fd0c4fd59187401f309680/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-11T08:06:25.538 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:06:25.538 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-11T08:06:25.608 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2024-01-11T08:06:25.609 DEBUG:teuthology.orchestra.run.smithi077:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4bcedb9fc11bed1fd6fd0c4fd59187401f309680/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-11T08:06:25.719 DEBUG:teuthology.orchestra.run.smithi077:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-11T08:06:25.786 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing1-quincy-2024-01-10-2010-quincy/4bcedb9fc11bed1fd6fd0c4fd59187401f309680/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing1-quincy-2024-01-10-2010-quincy/4bcedb9fc11bed1fd6fd0c4fd59187401f309680/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing1-quincy-2024-01-10-2010-quincy/4bcedb9fc11bed1fd6fd0c4fd59187401f309680/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-11T08:06:25.786 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:06:25.786 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-11T08:06:25.815 DEBUG:teuthology.orchestra.run.smithi077:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-11T08:06:25.862 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2024-01-11T08:06:25.862 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4bcedb9fc11bed1fd6fd0c4fd59187401f309680/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-11T08:06:25.884 INFO:teuthology.orchestra.run.smithi077.stdout:check_obsoletes = 1 2024-01-11T08:06:25.887 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum clean all 2024-01-11T08:06:25.975 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-11T08:06:26.075 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-11T08:06:26.148 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2024-01-11T08:06:26.152 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2024-01-11T08:06:26.242 INFO:teuthology.orchestra.run.smithi077.stdout:76 files removed 2024-01-11T08:06:26.281 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2024-01-11T08:06:26.537 INFO:teuthology.orchestra.run.smithi027.stdout:76 files removed 2024-01-11T08:06:26.575 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2024-01-11T08:06:27.118 INFO:teuthology.orchestra.run.smithi077.stdout:ceph packages for x86_64 266 kB/s | 78 kB 00:00 2024-01-11T08:06:27.793 INFO:teuthology.orchestra.run.smithi077.stdout:ceph noarch packages 14 kB/s | 9.2 kB 00:00 2024-01-11T08:06:27.802 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 123 kB/s | 78 kB 00:00 2024-01-11T08:06:28.086 INFO:teuthology.orchestra.run.smithi077.stdout:ceph source packages 5.5 kB/s | 1.5 kB 00:00 2024-01-11T08:06:28.286 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 21 kB/s | 9.2 kB 00:00 2024-01-11T08:06:28.870 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 2.6 kB/s | 1.5 kB 00:00 2024-01-11T08:06:30.144 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - AppStream 17 MB/s | 35 MB 00:02 2024-01-11T08:06:31.228 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - AppStream 15 MB/s | 35 MB 00:02 2024-01-11T08:06:38.970 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - BaseOS 30 MB/s | 58 MB 00:01 2024-01-11T08:06:41.479 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - BaseOS 19 MB/s | 58 MB 00:03 2024-01-11T08:06:48.237 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - Extras 37 kB/s | 18 kB 00:00 2024-01-11T08:06:48.838 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2024-01-11T08:06:49.431 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - PowerTools 15 MB/s | 6.7 MB 00:00 2024-01-11T08:06:51.032 INFO:teuthology.orchestra.run.smithi077.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-01-11T08:06:51.171 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - Extras 33 kB/s | 18 kB 00:00 2024-01-11T08:06:51.746 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2024-01-11T08:06:52.398 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - PowerTools 14 MB/s | 6.7 MB 00:00 2024-01-11T08:06:52.718 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-01-11T08:06:54.007 INFO:teuthology.orchestra.run.smithi027.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-01-11T08:06:54.950 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 21 MB/s | 16 MB 00:00 2024-01-11T08:06:57.266 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 415 kB/s | 24 kB 00:00 2024-01-11T08:06:59.817 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 439 kB/s | 24 kB 00:00 2024-01-11T08:07:01.878 INFO:teuthology.orchestra.run.smithi077.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-11T08:07:01.879 INFO:teuthology.orchestra.run.smithi077.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-11T08:07:01.893 INFO:teuthology.orchestra.run.smithi077.stdout:Package NetworkManager-1:1.40.16-4.el8.x86_64 is already installed. 2024-01-11T08:07:01.893 INFO:teuthology.orchestra.run.smithi077.stdout:Package NetworkManager-initscripts-updown-1:1.40.16-4.el8.noarch is already installed. 2024-01-11T08:07:01.894 INFO:teuthology.orchestra.run.smithi077.stdout:Package iproute-6.2.0-1.el8.x86_64 is already installed. 2024-01-11T08:07:01.896 INFO:teuthology.orchestra.run.smithi077.stdout:Package util-linux-2.32.1-41.el8.x86_64 is already installed. 2024-01-11T08:07:01.897 INFO:teuthology.orchestra.run.smithi077.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-01-11T08:07:01.899 INFO:teuthology.orchestra.run.smithi077.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-01-11T08:07:01.902 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-01-11T08:07:01.903 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-01-11T08:07:02.168 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-01-11T08:07:02.182 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================================ 2024-01-11T08:07:02.182 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-01-11T08:07:02.182 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================================ 2024-01-11T08:07:02.183 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-01-11T08:07:02.183 INFO:teuthology.orchestra.run.smithi077.stdout: bison x86_64 3.0.4-10.el8 appstream 688 k 2024-01-11T08:07:02.183 INFO:teuthology.orchestra.run.smithi077.stdout: ceph x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 6.4 k 2024-01-11T08:07:02.183 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 5.2 M 2024-01-11T08:07:02.183 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 875 k 2024-01-11T08:07:02.183 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 161 k 2024-01-11T08:07:02.183 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 1.6 M 2024-01-11T08:07:02.183 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 121 k 2024-01-11T08:07:02.183 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 1.7 M 2024-01-11T08:07:02.183 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 7.4 M 2024-01-11T08:07:02.183 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 47 k 2024-01-11T08:07:02.183 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 17 M 2024-01-11T08:07:02.183 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 52 M 2024-01-11T08:07:02.184 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 267 k 2024-01-11T08:07:02.184 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 80 k 2024-01-11T08:07:02.184 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 255 k 2024-01-11T08:07:02.184 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 18 k 2024-01-11T08:07:02.184 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel x86_64 0.190-2.el8 baseos 62 k 2024-01-11T08:07:02.184 INFO:teuthology.orchestra.run.smithi077.stdout: flex x86_64 2.6.1-9.el8 appstream 320 k 2024-01-11T08:07:02.184 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel x86_64 0.3.112-1.el8 baseos 19 k 2024-01-11T08:07:02.184 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 25 k 2024-01-11T08:07:02.184 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2 x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 796 k 2024-01-11T08:07:02.184 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 130 k 2024-01-11T08:07:02.184 INFO:teuthology.orchestra.run.smithi077.stdout: libtool x86_64 2.4.6-25.el8 appstream 709 k 2024-01-11T08:07:02.184 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel x86_64 2.32.1-43.el8 baseos 100 k 2024-01-11T08:07:02.184 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel x86_64 1:1.1.1k-11.el8 baseos 3.2 M 2024-01-11T08:07:02.184 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 190 k 2024-01-11T08:07:02.185 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 370 k 2024-01-11T08:07:02.185 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 359 k 2024-01-11T08:07:02.185 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 107 k 2024-01-11T08:07:02.185 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 91 k 2024-01-11T08:07:02.185 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 3.5 M 2024-01-11T08:07:02.185 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 174 k 2024-01-11T08:07:02.185 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 baseos 74 k 2024-01-11T08:07:02.185 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2024-01-11T08:07:02.185 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-14.el8 baseos 144 k 2024-01-11T08:07:02.185 INFO:teuthology.orchestra.run.smithi077.stdout: acl x86_64 2.2.53-3.el8 baseos 83 k 2024-01-11T08:07:02.185 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-debuginfod-client x86_64 0.190-2.el8 baseos 77 k 2024-01-11T08:07:02.185 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf x86_64 0.190-2.el8 baseos 233 k 2024-01-11T08:07:02.185 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libs x86_64 0.190-2.el8 baseos 305 k 2024-01-11T08:07:02.186 INFO:teuthology.orchestra.run.smithi077.stdout: iproute x86_64 6.2.0-5.el8 baseos 854 k 2024-01-11T08:07:02.186 INFO:teuthology.orchestra.run.smithi077.stdout: libacl x86_64 2.2.53-3.el8 baseos 35 k 2024-01-11T08:07:02.186 INFO:teuthology.orchestra.run.smithi077.stdout: libacl-devel x86_64 2.2.53-3.el8 baseos 85 k 2024-01-11T08:07:02.186 INFO:teuthology.orchestra.run.smithi077.stdout: libblkid x86_64 2.32.1-43.el8 baseos 220 k 2024-01-11T08:07:02.186 INFO:teuthology.orchestra.run.smithi077.stdout: libfdisk x86_64 2.32.1-43.el8 baseos 254 k 2024-01-11T08:07:02.186 INFO:teuthology.orchestra.run.smithi077.stdout: libmount x86_64 2.32.1-43.el8 baseos 236 k 2024-01-11T08:07:02.186 INFO:teuthology.orchestra.run.smithi077.stdout: librados2 x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 3.5 M 2024-01-11T08:07:02.186 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1 x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 3.6 M 2024-01-11T08:07:02.186 INFO:teuthology.orchestra.run.smithi077.stdout: libsmartcols x86_64 2.32.1-43.el8 baseos 179 k 2024-01-11T08:07:02.186 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid x86_64 2.32.1-43.el8 baseos 99 k 2024-01-11T08:07:02.186 INFO:teuthology.orchestra.run.smithi077.stdout: openssl x86_64 1:1.1.1k-11.el8 baseos 737 k 2024-01-11T08:07:02.186 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-libs x86_64 1:1.1.1k-11.el8 baseos 1.5 M 2024-01-11T08:07:02.187 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux x86_64 2.32.1-43.el8 baseos 2.5 M 2024-01-11T08:07:02.187 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-user x86_64 2.32.1-43.el8 baseos 103 k 2024-01-11T08:07:02.187 INFO:teuthology.orchestra.run.smithi077.stdout: xfsdump x86_64 3.1.8-7.el8 baseos 334 k 2024-01-11T08:07:02.187 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs x86_64 5.0.0-12.el8 baseos 1.1 M 2024-01-11T08:07:02.187 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-01-11T08:07:02.187 INFO:teuthology.orchestra.run.smithi077.stdout: automake noarch 1.16.1-8.el8 appstream 740 k 2024-01-11T08:07:02.187 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 23 M 2024-01-11T08:07:02.187 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 23 k 2024-01-11T08:07:02.187 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 2.3 M 2024-01-11T08:07:02.187 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 239 k 2024-01-11T08:07:02.187 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 3.9 M 2024-01-11T08:07:02.187 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 17 M 2024-01-11T08:07:02.187 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 14 k 2024-01-11T08:07:02.187 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 24 k 2024-01-11T08:07:02.188 INFO:teuthology.orchestra.run.smithi077.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-11T08:07:02.188 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-11T08:07:02.188 INFO:teuthology.orchestra.run.smithi077.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-11T08:07:02.188 INFO:teuthology.orchestra.run.smithi077.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 baseos 48 k 2024-01-11T08:07:02.188 INFO:teuthology.orchestra.run.smithi077.stdout: krb5-devel x86_64 1.18.2-22.el8 baseos 561 k 2024-01-11T08:07:02.188 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 178 k 2024-01-11T08:07:02.188 INFO:teuthology.orchestra.run.smithi077.stdout: libcom_err-devel x86_64 1.45.6-5.el8 baseos 39 k 2024-01-11T08:07:02.188 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-11T08:07:02.188 INFO:teuthology.orchestra.run.smithi077.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-11T08:07:02.188 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-11T08:07:02.188 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2024-01-11T08:07:02.188 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1 x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 507 k 2024-01-11T08:07:02.188 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-11T08:07:02.189 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2 x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 5.6 M 2024-01-11T08:07:02.189 INFO:teuthology.orchestra.run.smithi077.stdout: libselinux-devel x86_64 2.9-8.el8 baseos 200 k 2024-01-11T08:07:02.189 INFO:teuthology.orchestra.run.smithi077.stdout: libsepol-devel x86_64 2.9-3.el8 baseos 87 k 2024-01-11T08:07:02.189 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-11T08:07:02.189 INFO:teuthology.orchestra.run.smithi077.stdout: libverto-devel x86_64 0.3.2-2.el8 baseos 18 k 2024-01-11T08:07:02.189 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel x86_64 1.4.4-1.el8 baseos 44 k 2024-01-11T08:07:02.189 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-11T08:07:02.189 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2024-01-11T08:07:02.189 INFO:teuthology.orchestra.run.smithi077.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-11T08:07:02.189 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-11T08:07:02.189 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-devel x86_64 10.32-3.el8 baseos 605 k 2024-01-11T08:07:02.189 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf16 x86_64 10.32-3.el8 baseos 229 k 2024-01-11T08:07:02.189 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf32 x86_64 10.32-3.el8 baseos 220 k 2024-01-11T08:07:02.190 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-01-11T08:07:02.190 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-11T08:07:02.190 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-11T08:07:02.190 INFO:teuthology.orchestra.run.smithi077.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-11T08:07:02.190 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-11T08:07:02.190 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 45 k 2024-01-11T08:07:02.190 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 115 k 2024-01-11T08:07:02.190 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-11T08:07:02.190 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-11T08:07:02.190 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-11T08:07:02.190 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-11T08:07:02.190 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-11T08:07:02.190 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-11T08:07:02.191 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-11T08:07:02.191 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-11T08:07:02.191 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-11T08:07:02.191 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-11T08:07:02.191 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-11T08:07:02.191 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-11T08:07:02.191 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-11T08:07:02.191 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-11T08:07:02.191 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-11T08:07:02.191 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-11T08:07:02.191 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-11T08:07:02.191 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-01-11T08:07:02.191 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-11T08:07:02.191 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-11T08:07:02.191 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-11T08:07:02.192 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-11T08:07:02.192 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-11T08:07:02.192 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-11T08:07:02.192 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-11T08:07:02.192 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-11T08:07:02.192 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-11T08:07:02.192 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-11T08:07:02.192 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-11T08:07:02.192 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-11T08:07:02.192 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-11T08:07:02.192 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-11T08:07:02.192 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-11T08:07:02.192 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-11T08:07:02.193 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-11T08:07:02.193 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-11T08:07:02.193 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-11T08:07:02.193 INFO:teuthology.orchestra.run.smithi077.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-11T08:07:02.193 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-11T08:07:02.193 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-11T08:07:02.193 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-11T08:07:02.193 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-11T08:07:02.193 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-11T08:07:02.193 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-11T08:07:02.193 INFO:teuthology.orchestra.run.smithi077.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-11T08:07:02.193 INFO:teuthology.orchestra.run.smithi077.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-01-11T08:07:02.193 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-11T08:07:02.193 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel x86_64 1.2.11-21.el8 baseos 58 k 2024-01-11T08:07:02.193 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:07:02.194 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-01-11T08:07:02.194 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================================ 2024-01-11T08:07:02.194 INFO:teuthology.orchestra.run.smithi077.stdout:Install 123 Packages 2024-01-11T08:07:02.194 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 21 Packages 2024-01-11T08:07:02.194 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:07:02.194 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 213 M 2024-01-11T08:07:02.194 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-01-11T08:07:03.851 INFO:teuthology.orchestra.run.smithi077.stdout:(1/144): ceph-17.2.7-299.g4bcedb9f.el8.x86_64.r 26 kB/s | 6.4 kB 00:00 2024-01-11T08:07:04.143 INFO:teuthology.orchestra.run.smithi077.stdout:(2/144): ceph-base-17.2.7-299.g4bcedb9f.el8.x86 9.7 MB/s | 5.2 MB 00:00 2024-01-11T08:07:04.244 INFO:teuthology.orchestra.run.smithi077.stdout:(3/144): ceph-immutable-object-cache-17.2.7-299 1.6 MB/s | 161 kB 00:00 2024-01-11T08:07:04.510 INFO:teuthology.orchestra.run.smithi077.stdout:(4/144): ceph-mds-17.2.7-299.g4bcedb9f.el8.x86_ 8.7 MB/s | 2.3 MB 00:00 2024-01-11T08:07:04.544 INFO:teuthology.orchestra.run.smithi077.stdout:(5/144): ceph-fuse-17.2.7-299.g4bcedb9f.el8.x86 1.2 MB/s | 875 kB 00:00 2024-01-11T08:07:04.663 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-11T08:07:04.664 INFO:teuthology.orchestra.run.smithi027.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-11T08:07:04.679 INFO:teuthology.orchestra.run.smithi027.stdout:Package NetworkManager-1:1.40.16-4.el8.x86_64 is already installed. 2024-01-11T08:07:04.679 INFO:teuthology.orchestra.run.smithi027.stdout:Package NetworkManager-initscripts-updown-1:1.40.16-4.el8.noarch is already installed. 2024-01-11T08:07:04.680 INFO:teuthology.orchestra.run.smithi027.stdout:Package iproute-6.2.0-1.el8.x86_64 is already installed. 2024-01-11T08:07:04.681 INFO:teuthology.orchestra.run.smithi027.stdout:Package util-linux-2.32.1-41.el8.x86_64 is already installed. 2024-01-11T08:07:04.682 INFO:teuthology.orchestra.run.smithi027.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-01-11T08:07:04.685 INFO:teuthology.orchestra.run.smithi027.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-01-11T08:07:04.688 INFO:teuthology.orchestra.run.smithi027.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-01-11T08:07:04.689 INFO:teuthology.orchestra.run.smithi027.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-01-11T08:07:04.702 INFO:teuthology.orchestra.run.smithi077.stdout:(6/144): ceph-mgr-17.2.7-299.g4bcedb9f.el8.x86_ 8.2 MB/s | 1.6 MB 00:00 2024-01-11T08:07:04.984 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-01-11T08:07:04.999 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================================ 2024-01-11T08:07:04.999 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-01-11T08:07:04.999 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================================ 2024-01-11T08:07:04.999 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-01-11T08:07:04.999 INFO:teuthology.orchestra.run.smithi027.stdout: bison x86_64 3.0.4-10.el8 appstream 688 k 2024-01-11T08:07:05.000 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 6.4 k 2024-01-11T08:07:05.000 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 5.2 M 2024-01-11T08:07:05.000 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 875 k 2024-01-11T08:07:05.000 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 161 k 2024-01-11T08:07:05.000 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 1.6 M 2024-01-11T08:07:05.000 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 121 k 2024-01-11T08:07:05.000 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 1.7 M 2024-01-11T08:07:05.000 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 7.4 M 2024-01-11T08:07:05.000 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 47 k 2024-01-11T08:07:05.000 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 17 M 2024-01-11T08:07:05.000 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 52 M 2024-01-11T08:07:05.000 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 267 k 2024-01-11T08:07:05.001 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 80 k 2024-01-11T08:07:05.001 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-mirror x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 255 k 2024-01-11T08:07:05.001 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-top noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 18 k 2024-01-11T08:07:05.001 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-devel x86_64 0.190-2.el8 baseos 62 k 2024-01-11T08:07:05.001 INFO:teuthology.orchestra.run.smithi027.stdout: flex x86_64 2.6.1-9.el8 appstream 320 k 2024-01-11T08:07:05.001 INFO:teuthology.orchestra.run.smithi027.stdout: libaio-devel x86_64 0.3.112-1.el8 baseos 19 k 2024-01-11T08:07:05.001 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 25 k 2024-01-11T08:07:05.001 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 796 k 2024-01-11T08:07:05.001 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 130 k 2024-01-11T08:07:05.001 INFO:teuthology.orchestra.run.smithi027.stdout: libtool x86_64 2.4.6-25.el8 appstream 709 k 2024-01-11T08:07:05.001 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-devel x86_64 2.32.1-43.el8 baseos 100 k 2024-01-11T08:07:05.001 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-devel x86_64 1:1.1.1k-11.el8 baseos 3.2 M 2024-01-11T08:07:05.001 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 190 k 2024-01-11T08:07:05.001 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 370 k 2024-01-11T08:07:05.001 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 359 k 2024-01-11T08:07:05.001 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 107 k 2024-01-11T08:07:05.002 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 91 k 2024-01-11T08:07:05.002 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 3.5 M 2024-01-11T08:07:05.002 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 174 k 2024-01-11T08:07:05.002 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 baseos 74 k 2024-01-11T08:07:05.002 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2024-01-11T08:07:05.002 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-14.el8 baseos 144 k 2024-01-11T08:07:05.002 INFO:teuthology.orchestra.run.smithi027.stdout: acl x86_64 2.2.53-3.el8 baseos 83 k 2024-01-11T08:07:05.002 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-debuginfod-client x86_64 0.190-2.el8 baseos 77 k 2024-01-11T08:07:05.002 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf x86_64 0.190-2.el8 baseos 233 k 2024-01-11T08:07:05.002 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libs x86_64 0.190-2.el8 baseos 305 k 2024-01-11T08:07:05.002 INFO:teuthology.orchestra.run.smithi027.stdout: iproute x86_64 6.2.0-5.el8 baseos 854 k 2024-01-11T08:07:05.002 INFO:teuthology.orchestra.run.smithi027.stdout: libacl x86_64 2.2.53-3.el8 baseos 35 k 2024-01-11T08:07:05.002 INFO:teuthology.orchestra.run.smithi027.stdout: libacl-devel x86_64 2.2.53-3.el8 baseos 85 k 2024-01-11T08:07:05.002 INFO:teuthology.orchestra.run.smithi027.stdout: libblkid x86_64 2.32.1-43.el8 baseos 220 k 2024-01-11T08:07:05.003 INFO:teuthology.orchestra.run.smithi027.stdout: libfdisk x86_64 2.32.1-43.el8 baseos 254 k 2024-01-11T08:07:05.003 INFO:teuthology.orchestra.run.smithi027.stdout: libmount x86_64 2.32.1-43.el8 baseos 236 k 2024-01-11T08:07:05.003 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 3.5 M 2024-01-11T08:07:05.003 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 3.6 M 2024-01-11T08:07:05.003 INFO:teuthology.orchestra.run.smithi027.stdout: libsmartcols x86_64 2.32.1-43.el8 baseos 179 k 2024-01-11T08:07:05.003 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid x86_64 2.32.1-43.el8 baseos 99 k 2024-01-11T08:07:05.003 INFO:teuthology.orchestra.run.smithi027.stdout: openssl x86_64 1:1.1.1k-11.el8 baseos 737 k 2024-01-11T08:07:05.003 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-libs x86_64 1:1.1.1k-11.el8 baseos 1.5 M 2024-01-11T08:07:05.003 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux x86_64 2.32.1-43.el8 baseos 2.5 M 2024-01-11T08:07:05.003 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux-user x86_64 2.32.1-43.el8 baseos 103 k 2024-01-11T08:07:05.003 INFO:teuthology.orchestra.run.smithi027.stdout: xfsdump x86_64 3.1.8-7.el8 baseos 334 k 2024-01-11T08:07:05.003 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs x86_64 5.0.0-12.el8 baseos 1.1 M 2024-01-11T08:07:05.003 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-01-11T08:07:05.004 INFO:teuthology.orchestra.run.smithi027.stdout: automake noarch 1.16.1-8.el8 appstream 740 k 2024-01-11T08:07:05.004 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 23 M 2024-01-11T08:07:05.004 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 23 k 2024-01-11T08:07:05.004 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 2.3 M 2024-01-11T08:07:05.004 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 239 k 2024-01-11T08:07:05.004 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 3.9 M 2024-01-11T08:07:05.004 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 17 M 2024-01-11T08:07:05.004 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts noarch 2:17.2.7-299.g4bcedb9f.el8 ceph-noarch 14 k 2024-01-11T08:07:05.004 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 24 k 2024-01-11T08:07:05.004 INFO:teuthology.orchestra.run.smithi027.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-11T08:07:05.004 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-11T08:07:05.004 INFO:teuthology.orchestra.run.smithi027.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-11T08:07:05.004 INFO:teuthology.orchestra.run.smithi027.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 baseos 48 k 2024-01-11T08:07:05.004 INFO:teuthology.orchestra.run.smithi027.stdout: krb5-devel x86_64 1.18.2-22.el8 baseos 561 k 2024-01-11T08:07:05.004 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 178 k 2024-01-11T08:07:05.005 INFO:teuthology.orchestra.run.smithi027.stdout: libcom_err-devel x86_64 1.45.6-5.el8 baseos 39 k 2024-01-11T08:07:05.005 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-11T08:07:05.005 INFO:teuthology.orchestra.run.smithi027.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-11T08:07:05.005 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-11T08:07:05.005 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2024-01-11T08:07:05.005 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 507 k 2024-01-11T08:07:05.005 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-11T08:07:05.005 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 5.6 M 2024-01-11T08:07:05.005 INFO:teuthology.orchestra.run.smithi027.stdout: libselinux-devel x86_64 2.9-8.el8 baseos 200 k 2024-01-11T08:07:05.005 INFO:teuthology.orchestra.run.smithi027.stdout: libsepol-devel x86_64 2.9-3.el8 baseos 87 k 2024-01-11T08:07:05.005 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-11T08:07:05.005 INFO:teuthology.orchestra.run.smithi027.stdout: libverto-devel x86_64 0.3.2-2.el8 baseos 18 k 2024-01-11T08:07:05.005 INFO:teuthology.orchestra.run.smithi027.stdout: libzstd-devel x86_64 1.4.4-1.el8 baseos 44 k 2024-01-11T08:07:05.005 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-11T08:07:05.006 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2024-01-11T08:07:05.006 INFO:teuthology.orchestra.run.smithi027.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-11T08:07:05.006 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-11T08:07:05.006 INFO:teuthology.orchestra.run.smithi027.stdout: pcre2-devel x86_64 10.32-3.el8 baseos 605 k 2024-01-11T08:07:05.006 INFO:teuthology.orchestra.run.smithi027.stdout: pcre2-utf16 x86_64 10.32-3.el8 baseos 229 k 2024-01-11T08:07:05.006 INFO:teuthology.orchestra.run.smithi027.stdout: pcre2-utf32 x86_64 10.32-3.el8 baseos 220 k 2024-01-11T08:07:05.006 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-01-11T08:07:05.006 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-11T08:07:05.006 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-11T08:07:05.006 INFO:teuthology.orchestra.run.smithi027.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-11T08:07:05.006 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-11T08:07:05.006 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 45 k 2024-01-11T08:07:05.006 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:17.2.7-299.g4bcedb9f.el8 ceph 115 k 2024-01-11T08:07:05.006 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-11T08:07:05.006 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-11T08:07:05.007 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-11T08:07:05.007 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-11T08:07:05.007 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-11T08:07:05.007 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-11T08:07:05.007 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-11T08:07:05.007 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-11T08:07:05.007 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-11T08:07:05.007 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-11T08:07:05.007 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-11T08:07:05.007 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-11T08:07:05.007 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-11T08:07:05.007 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-11T08:07:05.007 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-11T08:07:05.007 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-11T08:07:05.007 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-11T08:07:05.007 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-01-11T08:07:05.008 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-11T08:07:05.008 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-11T08:07:05.008 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-11T08:07:05.008 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-11T08:07:05.008 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-11T08:07:05.008 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-11T08:07:05.008 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-11T08:07:05.008 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-11T08:07:05.008 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-11T08:07:05.008 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-11T08:07:05.008 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-11T08:07:05.008 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-11T08:07:05.008 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-11T08:07:05.008 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-11T08:07:05.009 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-11T08:07:05.009 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-11T08:07:05.009 INFO:teuthology.orchestra.run.smithi027.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-11T08:07:05.009 INFO:teuthology.orchestra.run.smithi027.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-11T08:07:05.009 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-11T08:07:05.009 INFO:teuthology.orchestra.run.smithi027.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-11T08:07:05.009 INFO:teuthology.orchestra.run.smithi027.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-11T08:07:05.009 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-11T08:07:05.009 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-11T08:07:05.009 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-11T08:07:05.009 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-11T08:07:05.009 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-11T08:07:05.009 INFO:teuthology.orchestra.run.smithi027.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-11T08:07:05.009 INFO:teuthology.orchestra.run.smithi027.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-01-11T08:07:05.009 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-11T08:07:05.009 INFO:teuthology.orchestra.run.smithi027.stdout: zlib-devel x86_64 1.2.11-21.el8 baseos 58 k 2024-01-11T08:07:05.010 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:07:05.010 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-01-11T08:07:05.010 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================================ 2024-01-11T08:07:05.010 INFO:teuthology.orchestra.run.smithi027.stdout:Install 123 Packages 2024-01-11T08:07:05.010 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 21 Packages 2024-01-11T08:07:05.010 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:07:05.010 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 213 M 2024-01-11T08:07:05.010 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-01-11T08:07:05.568 INFO:teuthology.orchestra.run.smithi027.stdout:(1/144): ceph-17.2.7-299.g4bcedb9f.el8.x86_64.r 25 kB/s | 6.4 kB 00:00 2024-01-11T08:07:05.711 INFO:teuthology.orchestra.run.smithi077.stdout:(7/144): ceph-common-17.2.7-299.g4bcedb9f.el8.x 11 MB/s | 23 MB 00:02 2024-01-11T08:07:06.194 INFO:teuthology.orchestra.run.smithi077.stdout:(8/144): ceph-osd-17.2.7-299.g4bcedb9f.el8.x86_ 11 MB/s | 17 MB 00:01 2024-01-11T08:07:06.218 INFO:teuthology.orchestra.run.smithi027.stdout:(2/144): ceph-fuse-17.2.7-299.g4bcedb9f.el8.x86 1.3 MB/s | 875 kB 00:00 2024-01-11T08:07:06.261 INFO:teuthology.orchestra.run.smithi077.stdout:(9/144): ceph-mon-17.2.7-299.g4bcedb9f.el8.x86_ 2.3 MB/s | 3.9 MB 00:01 2024-01-11T08:07:06.286 INFO:teuthology.orchestra.run.smithi077.stdout:(10/144): ceph-selinux-17.2.7-299.g4bcedb9f.el8 265 kB/s | 24 kB 00:00 2024-01-11T08:07:06.302 INFO:teuthology.orchestra.run.smithi027.stdout:(3/144): ceph-base-17.2.7-299.g4bcedb9f.el8.x86 5.2 MB/s | 5.2 MB 00:00 2024-01-11T08:07:06.327 INFO:teuthology.orchestra.run.smithi027.stdout:(4/144): ceph-immutable-object-cache-17.2.7-299 1.5 MB/s | 161 kB 00:00 2024-01-11T08:07:06.328 INFO:teuthology.orchestra.run.smithi077.stdout:(11/144): cephfs-mirror-17.2.7-299.g4bcedb9f.el 6.0 MB/s | 255 kB 00:00 2024-01-11T08:07:06.436 INFO:teuthology.orchestra.run.smithi077.stdout:(12/144): libcephfs-devel-17.2.7-299.g4bcedb9f. 231 kB/s | 25 kB 00:00 2024-01-11T08:07:06.612 INFO:teuthology.orchestra.run.smithi077.stdout:(13/144): libcephfs2-17.2.7-299.g4bcedb9f.el8.x 4.4 MB/s | 796 kB 00:00 2024-01-11T08:07:06.619 INFO:teuthology.orchestra.run.smithi027.stdout:(5/144): ceph-mds-17.2.7-299.g4bcedb9f.el8.x86_ 7.3 MB/s | 2.3 MB 00:00 2024-01-11T08:07:06.663 INFO:teuthology.orchestra.run.smithi077.stdout:(14/144): libcephsqlite-17.2.7-299.g4bcedb9f.el 3.5 MB/s | 178 kB 00:00 2024-01-11T08:07:06.746 INFO:teuthology.orchestra.run.smithi077.stdout:(15/144): librados-devel-17.2.7-299.g4bcedb9f.e 1.5 MB/s | 130 kB 00:00 2024-01-11T08:07:06.905 INFO:teuthology.orchestra.run.smithi077.stdout:(16/144): libradosstriper1-17.2.7-299.g4bcedb9f 3.1 MB/s | 507 kB 00:00 2024-01-11T08:07:06.995 INFO:teuthology.orchestra.run.smithi027.stdout:(6/144): ceph-mgr-17.2.7-299.g4bcedb9f.el8.x86_ 2.4 MB/s | 1.6 MB 00:00 2024-01-11T08:07:07.145 INFO:teuthology.orchestra.run.smithi027.stdout:(7/144): ceph-mon-17.2.7-299.g4bcedb9f.el8.x86_ 7.5 MB/s | 3.9 MB 00:00 2024-01-11T08:07:07.880 INFO:teuthology.orchestra.run.smithi077.stdout:(17/144): ceph-radosgw-17.2.7-299.g4bcedb9f.el8 7.7 MB/s | 17 MB 00:02 2024-01-11T08:07:07.964 INFO:teuthology.orchestra.run.smithi077.stdout:(18/144): librgw2-17.2.7-299.g4bcedb9f.el8.x86_ 5.3 MB/s | 5.6 MB 00:01 2024-01-11T08:07:07.981 INFO:teuthology.orchestra.run.smithi077.stdout:(19/144): python3-ceph-argparse-17.2.7-299.g4bc 446 kB/s | 45 kB 00:00 2024-01-11T08:07:08.014 INFO:teuthology.orchestra.run.smithi077.stdout:(20/144): python3-ceph-common-17.2.7-299.g4bced 2.3 MB/s | 115 kB 00:00 2024-01-11T08:07:08.048 INFO:teuthology.orchestra.run.smithi077.stdout:(21/144): python3-cephfs-17.2.7-299.g4bcedb9f.e 2.8 MB/s | 190 kB 00:00 2024-01-11T08:07:08.148 INFO:teuthology.orchestra.run.smithi077.stdout:(22/144): python3-rbd-17.2.7-299.g4bcedb9f.el8. 3.5 MB/s | 359 kB 00:00 2024-01-11T08:07:08.199 INFO:teuthology.orchestra.run.smithi077.stdout:(23/144): python3-rados-17.2.7-299.g4bcedb9f.el 2.0 MB/s | 370 kB 00:00 2024-01-11T08:07:08.224 INFO:teuthology.orchestra.run.smithi077.stdout:(24/144): python3-rgw-17.2.7-299.g4bcedb9f.el8. 1.4 MB/s | 107 kB 00:00 2024-01-11T08:07:08.266 INFO:teuthology.orchestra.run.smithi077.stdout:(25/144): rbd-fuse-17.2.7-299.g4bcedb9f.el8.x86 1.3 MB/s | 91 kB 00:00 2024-01-11T08:07:08.350 INFO:teuthology.orchestra.run.smithi077.stdout:(26/144): rbd-nbd-17.2.7-299.g4bcedb9f.el8.x86_ 2.0 MB/s | 174 kB 00:00 2024-01-11T08:07:08.433 INFO:teuthology.orchestra.run.smithi077.stdout:(27/144): ceph-grafana-dashboards-17.2.7-299.g4 280 kB/s | 23 kB 00:00 2024-01-11T08:07:08.517 INFO:teuthology.orchestra.run.smithi077.stdout:(28/144): ceph-mgr-cephadm-17.2.7-299.g4bcedb9f 1.4 MB/s | 121 kB 00:00 2024-01-11T08:07:08.784 INFO:teuthology.orchestra.run.smithi077.stdout:(29/144): rbd-mirror-17.2.7-299.g4bcedb9f.el8.x 6.2 MB/s | 3.5 MB 00:00 2024-01-11T08:07:08.820 INFO:teuthology.orchestra.run.smithi027.stdout:(8/144): ceph-radosgw-17.2.7-299.g4bcedb9f.el8. 10 MB/s | 17 MB 00:01 2024-01-11T08:07:08.826 INFO:teuthology.orchestra.run.smithi077.stdout:(30/144): ceph-mgr-dashboard-17.2.7-299.g4bcedb 5.5 MB/s | 1.7 MB 00:00 2024-01-11T08:07:08.887 INFO:teuthology.orchestra.run.smithi027.stdout:(9/144): ceph-selinux-17.2.7-299.g4bcedb9f.el8. 365 kB/s | 24 kB 00:00 2024-01-11T08:07:09.247 INFO:teuthology.orchestra.run.smithi077.stdout:(31/144): ceph-mgr-modules-core-17.2.7-299.g4bc 567 kB/s | 239 kB 00:00 2024-01-11T08:07:09.352 INFO:teuthology.orchestra.run.smithi077.stdout:(32/144): ceph-mgr-rook-17.2.7-299.g4bcedb9f.el 455 kB/s | 47 kB 00:00 2024-01-11T08:07:09.443 INFO:teuthology.orchestra.run.smithi077.stdout:(33/144): ceph-prometheus-alerts-17.2.7-299.g4b 159 kB/s | 14 kB 00:00 2024-01-11T08:07:09.610 INFO:teuthology.orchestra.run.smithi077.stdout:(34/144): ceph-volume-17.2.7-299.g4bcedb9f.el8. 1.6 MB/s | 267 kB 00:00 2024-01-11T08:07:09.702 INFO:teuthology.orchestra.run.smithi077.stdout:(35/144): cephadm-17.2.7-299.g4bcedb9f.el8.noar 877 kB/s | 80 kB 00:00 2024-01-11T08:07:09.786 INFO:teuthology.orchestra.run.smithi077.stdout:(36/144): cephfs-top-17.2.7-299.g4bcedb9f.el8.n 217 kB/s | 18 kB 00:00 2024-01-11T08:07:09.953 INFO:teuthology.orchestra.run.smithi077.stdout:(37/144): ceph-mgr-diskprediction-local-17.2.7- 6.3 MB/s | 7.4 MB 00:01 2024-01-11T08:07:10.612 INFO:teuthology.orchestra.run.smithi077.stdout:(38/144): automake-1.16.1-8.el8.noarch.rpm 897 kB/s | 740 kB 00:00 2024-01-11T08:07:10.637 INFO:teuthology.orchestra.run.smithi077.stdout:(39/144): bison-3.0.4-10.el8.x86_64.rpm 1.0 MB/s | 688 kB 00:00 2024-01-11T08:07:10.712 INFO:teuthology.orchestra.run.smithi077.stdout:(40/144): flex-2.6.1-9.el8.x86_64.rpm 3.1 MB/s | 320 kB 00:00 2024-01-11T08:07:10.737 INFO:teuthology.orchestra.run.smithi077.stdout:(41/144): jq-1.6-8.el8.x86_64.rpm 2.0 MB/s | 203 kB 00:00 2024-01-11T08:07:10.821 INFO:teuthology.orchestra.run.smithi077.stdout:(42/144): librdkafka-1.6.1-1.el8.x86_64.rpm 6.1 MB/s | 677 kB 00:00 2024-01-11T08:07:10.913 INFO:teuthology.orchestra.run.smithi077.stdout:(43/144): libtool-2.4.6-25.el8.x86_64.rpm 3.9 MB/s | 709 kB 00:00 2024-01-11T08:07:10.938 INFO:teuthology.orchestra.run.smithi077.stdout:(44/144): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.2 MB/s | 259 kB 00:00 2024-01-11T08:07:11.030 INFO:teuthology.orchestra.run.smithi077.stdout:(45/144): oniguruma-6.8.2-2.el8.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-01-11T08:07:11.247 INFO:teuthology.orchestra.run.smithi077.stdout:(46/144): openblas-0.3.15-4.el8.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-01-11T08:07:11.473 INFO:teuthology.orchestra.run.smithi077.stdout:(47/144): openblas-threads-0.3.15-4.el8.x86_64. 11 MB/s | 4.9 MB 00:00 2024-01-11T08:07:11.529 INFO:teuthology.orchestra.run.smithi027.stdout:(10/144): ceph-common-17.2.7-299.g4bcedb9f.el8. 3.8 MB/s | 23 MB 00:06 2024-01-11T08:07:11.598 INFO:teuthology.orchestra.run.smithi077.stdout:(48/144): python3-jinja2-2.10.1-3.el8.noarch.rp 4.2 MB/s | 538 kB 00:00 2024-01-11T08:07:11.662 INFO:teuthology.orchestra.run.smithi027.stdout:(11/144): cephfs-mirror-17.2.7-299.g4bcedb9f.el 1.9 MB/s | 255 kB 00:00 2024-01-11T08:07:11.673 INFO:teuthology.orchestra.run.smithi077.stdout:(49/144): python3-babel-2.5.1-7.el8.noarch.rpm 11 MB/s | 4.8 MB 00:00 2024-01-11T08:07:11.698 INFO:teuthology.orchestra.run.smithi077.stdout:(50/144): python3-jsonpatch-1.21-2.el8.noarch.r 265 kB/s | 27 kB 00:00 2024-01-11T08:07:11.746 INFO:teuthology.orchestra.run.smithi027.stdout:(12/144): libcephfs-devel-17.2.7-299.g4bcedb9f. 299 kB/s | 25 kB 00:00 2024-01-11T08:07:11.782 INFO:teuthology.orchestra.run.smithi077.stdout:(51/144): python3-jsonpointer-1.10-11.el8.noarc 184 kB/s | 20 kB 00:00 2024-01-11T08:07:11.807 INFO:teuthology.orchestra.run.smithi077.stdout:(52/144): python3-mako-1.0.6-14.el8.noarch.rpm 1.4 MB/s | 157 kB 00:00 2024-01-11T08:07:11.891 INFO:teuthology.orchestra.run.smithi077.stdout:(53/144): python3-markupsafe-0.23-19.el8.x86_64 357 kB/s | 39 kB 00:00 2024-01-11T08:07:11.913 INFO:teuthology.orchestra.run.smithi027.stdout:(13/144): ceph-osd-17.2.7-299.g4bcedb9f.el8.x86 3.4 MB/s | 17 MB 00:04 2024-01-11T08:07:12.005 INFO:teuthology.orchestra.run.smithi027.stdout:(14/144): libcephfs2-17.2.7-299.g4bcedb9f.el8.x 3.0 MB/s | 796 kB 00:00 2024-01-11T08:07:12.008 INFO:teuthology.orchestra.run.smithi077.stdout:(54/144): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.9 MB/s | 225 kB 00:00 2024-01-11T08:07:12.030 INFO:teuthology.orchestra.run.smithi027.stdout:(15/144): libcephsqlite-17.2.7-299.g4bcedb9f.el 1.5 MB/s | 178 kB 00:00 2024-01-11T08:07:12.097 INFO:teuthology.orchestra.run.smithi027.stdout:(16/144): librados-devel-17.2.7-299.g4bcedb9f.e 1.4 MB/s | 130 kB 00:00 2024-01-11T08:07:12.125 INFO:teuthology.orchestra.run.smithi077.stdout:(55/144): python3-prettytable-0.7.2-14.el8.noar 376 kB/s | 44 kB 00:00 2024-01-11T08:07:12.172 INFO:teuthology.orchestra.run.smithi027.stdout:(17/144): libradosstriper1-17.2.7-299.g4bcedb9f 3.5 MB/s | 507 kB 00:00 2024-01-11T08:07:12.183 INFO:teuthology.orchestra.run.smithi077.stdout:(56/144): python3-numpy-1.14.3-10.el8.x86_64.rp 9.9 MB/s | 3.7 MB 00:00 2024-01-11T08:07:12.217 INFO:teuthology.orchestra.run.smithi077.stdout:(57/144): python3-pyOpenSSL-19.0.0-1.el8.noarch 1.1 MB/s | 103 kB 00:00 2024-01-11T08:07:12.256 INFO:teuthology.orchestra.run.smithi027.stdout:(18/144): python3-ceph-argparse-17.2.7-299.g4bc 536 kB/s | 45 kB 00:00 2024-01-11T08:07:12.326 INFO:teuthology.orchestra.run.smithi077.stdout:(58/144): python3-pyasn1-0.3.7-6.el8.noarch.rpm 890 kB/s | 126 kB 00:00 2024-01-11T08:07:12.351 INFO:teuthology.orchestra.run.smithi077.stdout:(59/144): python3-pyasn1-modules-0.3.7-6.el8.no 822 kB/s | 110 kB 00:00 2024-01-11T08:07:12.356 INFO:teuthology.orchestra.run.smithi027.stdout:(19/144): python3-ceph-common-17.2.7-299.g4bced 1.1 MB/s | 115 kB 00:00 2024-01-11T08:07:12.460 INFO:teuthology.orchestra.run.smithi077.stdout:(60/144): python3-pytz-2017.2-11.el8.noarch.rpm 427 kB/s | 57 kB 00:00 2024-01-11T08:07:12.473 INFO:teuthology.orchestra.run.smithi027.stdout:(20/144): python3-cephfs-17.2.7-299.g4bcedb9f.e 1.6 MB/s | 190 kB 00:00 2024-01-11T08:07:12.602 INFO:teuthology.orchestra.run.smithi077.stdout:(61/144): python3-werkzeug-0.12.2-4.el8.noarch. 3.2 MB/s | 457 kB 00:00 2024-01-11T08:07:12.632 INFO:teuthology.orchestra.run.smithi027.stdout:(21/144): python3-rados-17.2.7-299.g4bcedb9f.el 2.3 MB/s | 370 kB 00:00 2024-01-11T08:07:12.752 INFO:teuthology.orchestra.run.smithi077.stdout:(62/144): socat-1.7.4.1-1.el8.x86_64.rpm 2.1 MB/s | 323 kB 00:00 2024-01-11T08:07:13.191 INFO:teuthology.orchestra.run.smithi027.stdout:(22/144): ceph-test-17.2.7-299.g4bcedb9f.el8.x8 12 MB/s | 52 MB 00:04 2024-01-11T08:07:13.224 INFO:teuthology.orchestra.run.smithi027.stdout:(23/144): python3-rbd-17.2.7-299.g4bcedb9f.el8. 606 kB/s | 359 kB 00:00 2024-01-11T08:07:13.283 INFO:teuthology.orchestra.run.smithi027.stdout:(24/144): python3-rgw-17.2.7-299.g4bcedb9f.el8. 1.1 MB/s | 107 kB 00:00 2024-01-11T08:07:13.325 INFO:teuthology.orchestra.run.smithi027.stdout:(25/144): rbd-fuse-17.2.7-299.g4bcedb9f.el8.x86 906 kB/s | 91 kB 00:00 2024-01-11T08:07:13.417 INFO:teuthology.orchestra.run.smithi027.stdout:(26/144): rbd-nbd-17.2.7-299.g4bcedb9f.el8.x86_ 1.9 MB/s | 174 kB 00:00 2024-01-11T08:07:13.453 INFO:teuthology.orchestra.run.smithi077.stdout:(63/144): xmlstarlet-1.6.1-20.el8.x86_64.rpm 101 kB/s | 71 kB 00:00 2024-01-11T08:07:13.525 INFO:teuthology.orchestra.run.smithi027.stdout:(27/144): ceph-grafana-dashboards-17.2.7-299.g4 215 kB/s | 23 kB 00:00 2024-01-11T08:07:13.609 INFO:teuthology.orchestra.run.smithi027.stdout:(28/144): ceph-mgr-cephadm-17.2.7-299.g4bcedb9f 1.4 MB/s | 121 kB 00:00 2024-01-11T08:07:13.760 INFO:teuthology.orchestra.run.smithi027.stdout:(29/144): rbd-mirror-17.2.7-299.g4bcedb9f.el8.x 7.3 MB/s | 3.5 MB 00:00 2024-01-11T08:07:14.126 INFO:teuthology.orchestra.run.smithi027.stdout:(30/144): ceph-mgr-dashboard-17.2.7-299.g4bcedb 3.3 MB/s | 1.7 MB 00:00 2024-01-11T08:07:14.128 INFO:teuthology.orchestra.run.smithi077.stdout:(64/144): python3-scipy-1.0.0-21.module_el8+762 8.0 MB/s | 14 MB 00:01 2024-01-11T08:07:14.210 INFO:teuthology.orchestra.run.smithi027.stdout:(31/144): librgw2-17.2.7-299.g4bcedb9f.el8.x86_ 2.6 MB/s | 5.6 MB 00:02 2024-01-11T08:07:14.235 INFO:teuthology.orchestra.run.smithi027.stdout:(32/144): ceph-mgr-modules-core-17.2.7-299.g4bc 2.2 MB/s | 239 kB 00:00 2024-01-11T08:07:14.294 INFO:teuthology.orchestra.run.smithi027.stdout:(33/144): ceph-mgr-rook-17.2.7-299.g4bcedb9f.el 567 kB/s | 47 kB 00:00 2024-01-11T08:07:14.328 INFO:teuthology.orchestra.run.smithi027.stdout:(34/144): ceph-prometheus-alerts-17.2.7-299.g4b 157 kB/s | 14 kB 00:00 2024-01-11T08:07:14.428 INFO:teuthology.orchestra.run.smithi027.stdout:(35/144): cephadm-17.2.7-299.g4bcedb9f.el8.noar 806 kB/s | 80 kB 00:00 2024-01-11T08:07:14.453 INFO:teuthology.orchestra.run.smithi027.stdout:(36/144): ceph-volume-17.2.7-299.g4bcedb9f.el8. 1.6 MB/s | 267 kB 00:00 2024-01-11T08:07:14.587 INFO:teuthology.orchestra.run.smithi027.stdout:(37/144): ceph-mgr-diskprediction-local-17.2.7- 8.9 MB/s | 7.4 MB 00:00 2024-01-11T08:07:14.612 INFO:teuthology.orchestra.run.smithi027.stdout:(38/144): cephfs-top-17.2.7-299.g4bcedb9f.el8.n 98 kB/s | 18 kB 00:00 2024-01-11T08:07:15.128 INFO:teuthology.orchestra.run.smithi077.stdout:(65/144): elfutils-libelf-devel-0.190-2.el8.x86 37 kB/s | 62 kB 00:01 2024-01-11T08:07:15.212 INFO:teuthology.orchestra.run.smithi077.stdout:(66/144): keyutils-libs-devel-1.5.10-9.el8.x86_ 579 kB/s | 48 kB 00:00 2024-01-11T08:07:15.329 INFO:teuthology.orchestra.run.smithi027.stdout:(39/144): automake-1.16.1-8.el8.noarch.rpm 845 kB/s | 740 kB 00:00 2024-01-11T08:07:15.454 INFO:teuthology.orchestra.run.smithi027.stdout:(40/144): jq-1.6-8.el8.x86_64.rpm 1.6 MB/s | 203 kB 00:00 2024-01-11T08:07:15.521 INFO:teuthology.orchestra.run.smithi027.stdout:(41/144): bison-3.0.4-10.el8.x86_64.rpm 737 kB/s | 688 kB 00:00 2024-01-11T08:07:15.528 INFO:teuthology.orchestra.run.smithi077.stdout:(67/144): fuse-2.9.7-17.el8.x86_64.rpm 59 kB/s | 83 kB 00:01 2024-01-11T08:07:15.595 INFO:teuthology.orchestra.run.smithi077.stdout:(68/144): libaio-devel-0.3.112-1.el8.x86_64.rpm 280 kB/s | 19 kB 00:00 2024-01-11T08:07:15.662 INFO:teuthology.orchestra.run.smithi077.stdout:(69/144): libcom_err-devel-1.45.6-5.el8.x86_64. 583 kB/s | 39 kB 00:00 2024-01-11T08:07:15.704 INFO:teuthology.orchestra.run.smithi077.stdout:(70/144): krb5-devel-1.18.2-22.el8.x86_64.rpm 1.1 MB/s | 561 kB 00:00 2024-01-11T08:07:15.713 INFO:teuthology.orchestra.run.smithi027.stdout:(42/144): librdkafka-1.6.1-1.el8.x86_64.rpm 2.6 MB/s | 677 kB 00:00 2024-01-11T08:07:15.738 INFO:teuthology.orchestra.run.smithi027.stdout:(43/144): flex-2.6.1-9.el8.x86_64.rpm 284 kB/s | 320 kB 00:01 2024-01-11T08:07:15.847 INFO:teuthology.orchestra.run.smithi027.stdout:(44/144): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.9 MB/s | 259 kB 00:00 2024-01-11T08:07:15.938 INFO:teuthology.orchestra.run.smithi077.stdout:(71/144): libquadmath-8.5.0-21.el8.x86_64.rpm 739 kB/s | 172 kB 00:00 2024-01-11T08:07:15.971 INFO:teuthology.orchestra.run.smithi077.stdout:(72/144): libgfortran-8.5.0-21.el8.x86_64.rpm 2.0 MB/s | 645 kB 00:00 2024-01-11T08:07:15.989 INFO:teuthology.orchestra.run.smithi027.stdout:(45/144): libtool-2.4.6-25.el8.x86_64.rpm 1.5 MB/s | 709 kB 00:00 2024-01-11T08:07:16.030 INFO:teuthology.orchestra.run.smithi077.stdout:(73/144): librabbitmq-0.9.0-4.el8.x86_64.rpm 512 kB/s | 47 kB 00:00 2024-01-11T08:07:16.056 INFO:teuthology.orchestra.run.smithi077.stdout:(74/144): libselinux-devel-2.9-8.el8.x86_64.rpm 2.3 MB/s | 200 kB 00:00 2024-01-11T08:07:16.139 INFO:teuthology.orchestra.run.smithi077.stdout:(75/144): libuuid-devel-2.32.1-43.el8.x86_64.rp 1.2 MB/s | 100 kB 00:00 2024-01-11T08:07:16.164 INFO:teuthology.orchestra.run.smithi077.stdout:(76/144): libsepol-devel-2.9-3.el8.x86_64.rpm 651 kB/s | 87 kB 00:00 2024-01-11T08:07:16.164 INFO:teuthology.orchestra.run.smithi027.stdout:(46/144): oniguruma-6.8.2-2.el8.x86_64.rpm 440 kB/s | 187 kB 00:00 2024-01-11T08:07:16.281 INFO:teuthology.orchestra.run.smithi077.stdout:(77/144): libverto-devel-0.3.2-2.el8.x86_64.rpm 127 kB/s | 18 kB 00:00 2024-01-11T08:07:16.306 INFO:teuthology.orchestra.run.smithi077.stdout:(78/144): libzstd-devel-1.4.4-1.el8.x86_64.rpm 309 kB/s | 44 kB 00:00 2024-01-11T08:07:16.715 INFO:teuthology.orchestra.run.smithi077.stdout:(79/144): pcre2-devel-10.32-3.el8.x86_64.rpm 1.4 MB/s | 605 kB 00:00 2024-01-11T08:07:16.891 INFO:teuthology.orchestra.run.smithi077.stdout:(80/144): pcre2-utf16-10.32-3.el8.x86_64.rpm 1.3 MB/s | 229 kB 00:00 2024-01-11T08:07:16.923 INFO:teuthology.orchestra.run.smithi027.stdout:(47/144): openblas-0.3.15-4.el8.x86_64.rpm 4.4 MB/s | 4.7 MB 00:01 2024-01-11T08:07:16.999 INFO:teuthology.orchestra.run.smithi077.stdout:(81/144): openssl-devel-1.1.1k-11.el8.x86_64.rp 4.4 MB/s | 3.2 MB 00:00 2024-01-11T08:07:17.057 INFO:teuthology.orchestra.run.smithi027.stdout:(48/144): python3-jinja2-2.10.1-3.el8.noarch.rp 3.9 MB/s | 538 kB 00:00 2024-01-11T08:07:17.149 INFO:teuthology.orchestra.run.smithi077.stdout:(82/144): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.6 MB/s | 249 kB 00:00 2024-01-11T08:07:17.182 INFO:teuthology.orchestra.run.smithi027.stdout:(49/144): python3-jsonpatch-1.21-2.el8.noarch.r 213 kB/s | 27 kB 00:00 2024-01-11T08:07:17.191 INFO:teuthology.orchestra.run.smithi077.stdout:(83/144): pcre2-utf32-10.32-3.el8.x86_64.rpm 734 kB/s | 220 kB 00:00 2024-01-11T08:07:17.292 INFO:teuthology.orchestra.run.smithi077.stdout:(84/144): python3-jwt-1.6.1-2.el8.noarch.rpm 427 kB/s | 43 kB 00:00 2024-01-11T08:07:17.315 INFO:teuthology.orchestra.run.smithi027.stdout:(50/144): python3-jsonpointer-1.10-11.el8.noarc 150 kB/s | 20 kB 00:00 2024-01-11T08:07:17.375 INFO:teuthology.orchestra.run.smithi077.stdout:(85/144): python3-oauthlib-2.1.0-1.el8.noarch.r 1.8 MB/s | 155 kB 00:00 2024-01-11T08:07:17.441 INFO:teuthology.orchestra.run.smithi027.stdout:(51/144): python3-mako-1.0.6-14.el8.noarch.rpm 1.2 MB/s | 157 kB 00:00 2024-01-11T08:07:17.459 INFO:teuthology.orchestra.run.smithi077.stdout:(86/144): python3-cryptography-3.2.1-7.el8.x86_ 2.0 MB/s | 641 kB 00:00 2024-01-11T08:07:17.484 INFO:teuthology.orchestra.run.smithi077.stdout:(87/144): python3-pycparser-2.14-14.el8.noarch. 1.0 MB/s | 109 kB 00:00 2024-01-11T08:07:17.535 INFO:teuthology.orchestra.run.smithi077.stdout:(88/144): python3-requests-oauthlib-1.0.0-1.el8 568 kB/s | 43 kB 00:00 2024-01-11T08:07:17.583 INFO:teuthology.orchestra.run.smithi027.stdout:(52/144): python3-markupsafe-0.23-19.el8.x86_64 274 kB/s | 39 kB 00:00 2024-01-11T08:07:17.594 INFO:teuthology.orchestra.run.smithi077.stdout:(89/144): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 679 kB/s | 74 kB 00:00 2024-01-11T08:07:17.644 INFO:teuthology.orchestra.run.smithi077.stdout:(90/144): zlib-devel-1.2.11-21.el8.x86_64.rpm 535 kB/s | 58 kB 00:00 2024-01-11T08:07:17.835 INFO:teuthology.orchestra.run.smithi077.stdout:(91/144): liboath-2.6.2-3.el8.x86_64.rpm 307 kB/s | 59 kB 00:00 2024-01-11T08:07:17.878 INFO:teuthology.orchestra.run.smithi077.stdout:(92/144): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2024-01-11T08:07:17.936 INFO:teuthology.orchestra.run.smithi077.stdout:(93/144): libunwind-1.3.1-3.el8.x86_64.rpm 749 kB/s | 75 kB 00:00 2024-01-11T08:07:18.036 INFO:teuthology.orchestra.run.smithi077.stdout:(94/144): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 437 kB/s | 44 kB 00:00 2024-01-11T08:07:18.062 INFO:teuthology.orchestra.run.smithi077.stdout:(95/144): python3-asyncssh-2.7.0-2.el8.noarch.r 2.5 MB/s | 476 kB 00:00 2024-01-11T08:07:18.128 INFO:teuthology.orchestra.run.smithi077.stdout:(96/144): python3-cachetools-3.1.1-4.el8.noarch 497 kB/s | 33 kB 00:00 2024-01-11T08:07:18.154 INFO:teuthology.orchestra.run.smithi077.stdout:(97/144): python3-beautifulsoup4-4.6.3-2.el8.1. 1.5 MB/s | 185 kB 00:00 2024-01-11T08:07:18.196 INFO:teuthology.orchestra.run.smithi077.stdout:(98/144): python3-certifi-2018.10.15-7.el8.noar 243 kB/s | 16 kB 00:00 2024-01-11T08:07:18.221 INFO:teuthology.orchestra.run.smithi077.stdout:(99/144): python3-cheroot-8.5.2-1.el8.noarch.rp 2.5 MB/s | 173 kB 00:00 2024-01-11T08:07:18.233 INFO:teuthology.orchestra.run.smithi027.stdout:(53/144): python3-numpy-1.14.3-10.el8.x86_64.rp 5.7 MB/s | 3.7 MB 00:00 2024-01-11T08:07:18.288 INFO:teuthology.orchestra.run.smithi077.stdout:(100/144): python3-google-auth-1.1.1-10.el8.noa 1.2 MB/s | 81 kB 00:00 2024-01-11T08:07:18.338 INFO:teuthology.orchestra.run.smithi077.stdout:(101/144): python3-cherrypy-18.4.0-1.el8.noarch 2.7 MB/s | 384 kB 00:00 2024-01-11T08:07:18.358 INFO:teuthology.orchestra.run.smithi027.stdout:(54/144): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.8 MB/s | 225 kB 00:00 2024-01-11T08:07:18.363 INFO:teuthology.orchestra.run.smithi077.stdout:(102/144): python3-jaraco-6.2-6.el8.noarch.rpm 148 kB/s | 11 kB 00:00 2024-01-11T08:07:18.405 INFO:teuthology.orchestra.run.smithi077.stdout:(103/144): python3-jaraco-functools-2.0-4.el8.n 263 kB/s | 18 kB 00:00 2024-01-11T08:07:18.484 INFO:teuthology.orchestra.run.smithi027.stdout:(55/144): python3-prettytable-0.7.2-14.el8.noar 351 kB/s | 44 kB 00:00 2024-01-11T08:07:18.497 INFO:teuthology.orchestra.run.smithi077.stdout:(104/144): python3-logutils-0.3.5-11.el8.noarch 532 kB/s | 49 kB 00:00 2024-01-11T08:07:18.531 INFO:teuthology.orchestra.run.smithi077.stdout:(105/144): python3-kubernetes-11.0.0-6.el8.noar 7.0 MB/s | 1.2 MB 00:00 2024-01-11T08:07:18.565 INFO:teuthology.orchestra.run.smithi077.stdout:(106/144): python3-more-itertools-7.2.0-3.el8.n 880 kB/s | 59 kB 00:00 2024-01-11T08:07:18.598 INFO:teuthology.orchestra.run.smithi077.stdout:(107/144): python3-natsort-7.1.1-2.el8.noarch.r 905 kB/s | 60 kB 00:00 2024-01-11T08:07:18.609 INFO:teuthology.orchestra.run.smithi027.stdout:(56/144): python3-pyOpenSSL-19.0.0-1.el8.noarch 829 kB/s | 103 kB 00:00 2024-01-11T08:07:18.665 INFO:teuthology.orchestra.run.smithi077.stdout:(108/144): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2024-01-11T08:07:18.682 INFO:teuthology.orchestra.run.smithi077.stdout:(109/144): python3-pecan-1.3.2-9.el8.noarch.rpm 2.4 MB/s | 283 kB 00:00 2024-01-11T08:07:18.718 INFO:teuthology.orchestra.run.smithi027.stdout:(57/144): openblas-threads-0.3.15-4.el8.x86_64. 1.8 MB/s | 4.9 MB 00:02 2024-01-11T08:07:18.732 INFO:teuthology.orchestra.run.smithi077.stdout:(110/144): python3-repoze-lru-0.7-6.el8.noarch. 504 kB/s | 34 kB 00:00 2024-01-11T08:07:18.743 INFO:teuthology.orchestra.run.smithi027.stdout:(58/144): python3-pyasn1-0.3.7-6.el8.noarch.rpm 947 kB/s | 126 kB 00:00 2024-01-11T08:07:18.757 INFO:teuthology.orchestra.run.smithi077.stdout:(111/144): python3-routes-2.4.1-12.el8.noarch.r 2.6 MB/s | 196 kB 00:00 2024-01-11T08:07:18.791 INFO:teuthology.orchestra.run.smithi077.stdout:(112/144): python3-rsa-4.9-2.el8.noarch.rpm 1.1 MB/s | 64 kB 00:00 2024-01-11T08:07:18.833 INFO:teuthology.orchestra.run.smithi077.stdout:(113/144): python3-simplegeneric-0.8.1-17.el8.n 248 kB/s | 19 kB 00:00 2024-01-11T08:07:18.843 INFO:teuthology.orchestra.run.smithi027.stdout:(59/144): python3-pyasn1-modules-0.3.7-6.el8.no 880 kB/s | 110 kB 00:00 2024-01-11T08:07:18.867 INFO:teuthology.orchestra.run.smithi077.stdout:(114/144): python3-singledispatch-3.4.0.3-18.el 318 kB/s | 24 kB 00:00 2024-01-11T08:07:18.876 INFO:teuthology.orchestra.run.smithi027.stdout:(60/144): python3-pytz-2017.2-11.el8.noarch.rpm 427 kB/s | 57 kB 00:00 2024-01-11T08:07:18.926 INFO:teuthology.orchestra.run.smithi077.stdout:(115/144): python3-tempora-1.14.1-5.el8.noarch. 318 kB/s | 29 kB 00:00 2024-01-11T08:07:18.951 INFO:teuthology.orchestra.run.smithi077.stdout:(116/144): python3-trustme-0.6.0-4.el8.noarch.r 330 kB/s | 27 kB 00:00 2024-01-11T08:07:19.018 INFO:teuthology.orchestra.run.smithi077.stdout:(117/144): python3-webob-1.8.5-1.el8.1.noarch.r 3.7 MB/s | 251 kB 00:00 2024-01-11T08:07:19.018 INFO:teuthology.orchestra.run.smithi027.stdout:(61/144): python3-werkzeug-0.12.2-4.el8.noarch. 3.2 MB/s | 457 kB 00:00 2024-01-11T08:07:19.043 INFO:teuthology.orchestra.run.smithi077.stdout:(118/144): python3-waitress-1.4.3-1.el8.noarch. 2.1 MB/s | 255 kB 00:00 2024-01-11T08:07:19.068 INFO:teuthology.orchestra.run.smithi077.stdout:(119/144): python3-websocket-client-0.56.0-5.el 1.2 MB/s | 61 kB 00:00 2024-01-11T08:07:19.128 INFO:teuthology.orchestra.run.smithi027.stdout:(62/144): socat-1.7.4.1-1.el8.x86_64.rpm 2.9 MB/s | 323 kB 00:00 2024-01-11T08:07:19.228 INFO:teuthology.orchestra.run.smithi027.stdout:(63/144): xmlstarlet-1.6.1-20.el8.x86_64.rpm 709 kB/s | 71 kB 00:00 2024-01-11T08:07:19.277 INFO:teuthology.orchestra.run.smithi077.stdout:(120/144): python3-webtest-2.0.33-1.el8.noarch. 365 kB/s | 85 kB 00:00 2024-01-11T08:07:19.302 INFO:teuthology.orchestra.run.smithi077.stdout:(121/144): python3-zc-lockfile-2.0-2.el8.noarch 98 kB/s | 23 kB 00:00 2024-01-11T08:07:19.469 INFO:teuthology.orchestra.run.smithi077.stdout:(122/144): thrift-0.13.0-2.el8.x86_64.rpm 9.1 MB/s | 1.7 MB 00:00 2024-01-11T08:07:19.586 INFO:teuthology.orchestra.run.smithi027.stdout:(64/144): elfutils-libelf-devel-0.190-2.el8.x86 174 kB/s | 62 kB 00:00 2024-01-11T08:07:19.678 INFO:teuthology.orchestra.run.smithi077.stdout:(123/144): librados2-17.2.7-299.g4bcedb9f.el8.x 9.2 MB/s | 3.5 MB 00:00 2024-01-11T08:07:19.712 INFO:teuthology.orchestra.run.smithi027.stdout:(65/144): fuse-2.9.7-17.el8.x86_64.rpm 664 kB/s | 83 kB 00:00 2024-01-11T08:07:19.778 INFO:teuthology.orchestra.run.smithi027.stdout:(66/144): keyutils-libs-devel-1.5.10-9.el8.x86_ 727 kB/s | 48 kB 00:00 2024-01-11T08:07:19.836 INFO:teuthology.orchestra.run.smithi077.stdout:(124/144): librbd1-17.2.7-299.g4bcedb9f.el8.x86 9.8 MB/s | 3.6 MB 00:00 2024-01-11T08:07:19.870 INFO:teuthology.orchestra.run.smithi077.stdout:(125/144): NetworkManager-initscripts-updown-1. 752 kB/s | 144 kB 00:00 2024-01-11T08:07:19.920 INFO:teuthology.orchestra.run.smithi077.stdout:(126/144): elfutils-debuginfod-client-0.190-2.e 1.5 MB/s | 77 kB 00:00 2024-01-11T08:07:19.979 INFO:teuthology.orchestra.run.smithi077.stdout:(127/144): acl-2.2.53-3.el8.x86_64.rpm 582 kB/s | 83 kB 00:00 2024-01-11T08:07:20.037 INFO:teuthology.orchestra.run.smithi027.stdout:(67/144): krb5-devel-1.18.2-22.el8.x86_64.rpm 2.1 MB/s | 561 kB 00:00 2024-01-11T08:07:20.088 INFO:teuthology.orchestra.run.smithi077.stdout:(128/144): elfutils-libelf-0.190-2.el8.x86_64.r 1.4 MB/s | 233 kB 00:00 2024-01-11T08:07:20.104 INFO:teuthology.orchestra.run.smithi027.stdout:(68/144): libaio-devel-0.3.112-1.el8.x86_64.rpm 281 kB/s | 19 kB 00:00 2024-01-11T08:07:20.148 INFO:teuthology.orchestra.run.smithi077.stdout:(129/144): elfutils-libs-0.190-2.el8.x86_64.rpm 1.8 MB/s | 305 kB 00:00 2024-01-11T08:07:20.163 INFO:teuthology.orchestra.run.smithi027.stdout:(69/144): libcom_err-devel-1.45.6-5.el8.x86_64. 666 kB/s | 39 kB 00:00 2024-01-11T08:07:20.238 INFO:teuthology.orchestra.run.smithi077.stdout:(130/144): libacl-2.2.53-3.el8.x86_64.rpm 381 kB/s | 35 kB 00:00 2024-01-11T08:07:20.314 INFO:teuthology.orchestra.run.smithi077.stdout:(131/144): libacl-devel-2.2.53-3.el8.x86_64.rpm 1.1 MB/s | 85 kB 00:00 2024-01-11T08:07:20.439 INFO:teuthology.orchestra.run.smithi077.stdout:(132/144): libblkid-2.32.1-43.el8.x86_64.rpm 1.7 MB/s | 220 kB 00:00 2024-01-11T08:07:20.521 INFO:teuthology.orchestra.run.smithi027.stdout:(70/144): libgfortran-8.5.0-21.el8.x86_64.rpm 1.8 MB/s | 645 kB 00:00 2024-01-11T08:07:20.564 INFO:teuthology.orchestra.run.smithi077.stdout:(133/144): iproute-6.2.0-5.el8.x86_64.rpm 1.8 MB/s | 854 kB 00:00 2024-01-11T08:07:20.589 INFO:teuthology.orchestra.run.smithi077.stdout:(134/144): libfdisk-2.32.1-43.el8.x86_64.rpm 1.7 MB/s | 254 kB 00:00 2024-01-11T08:07:20.647 INFO:teuthology.orchestra.run.smithi027.stdout:(71/144): libquadmath-8.5.0-21.el8.x86_64.rpm 1.3 MB/s | 172 kB 00:00 2024-01-11T08:07:20.681 INFO:teuthology.orchestra.run.smithi077.stdout:(135/144): libsmartcols-2.32.1-43.el8.x86_64.rp 1.9 MB/s | 179 kB 00:00 2024-01-11T08:07:20.698 INFO:teuthology.orchestra.run.smithi077.stdout:(136/144): libmount-2.32.1-43.el8.x86_64.rpm 1.7 MB/s | 236 kB 00:00 2024-01-11T08:07:20.773 INFO:teuthology.orchestra.run.smithi077.stdout:(137/144): libuuid-2.32.1-43.el8.x86_64.rpm 1.1 MB/s | 99 kB 00:00 2024-01-11T08:07:20.949 INFO:teuthology.orchestra.run.smithi077.stdout:(138/144): openssl-1.1.1k-11.el8.x86_64.rpm 2.9 MB/s | 737 kB 00:00 2024-01-11T08:07:20.964 INFO:teuthology.orchestra.run.smithi027.stdout:(72/144): librabbitmq-0.9.0-4.el8.x86_64.rpm 148 kB/s | 47 kB 00:00 2024-01-11T08:07:20.982 INFO:teuthology.orchestra.run.smithi077.stdout:(139/144): openssl-libs-1.1.1k-11.el8.x86_64.rp 7.1 MB/s | 1.5 MB 00:00 2024-01-11T08:07:21.041 INFO:teuthology.orchestra.run.smithi077.stdout:(140/144): util-linux-user-2.32.1-43.el8.x86_64 1.7 MB/s | 103 kB 00:00 2024-01-11T08:07:21.064 INFO:teuthology.orchestra.run.smithi027.stdout:(73/144): libselinux-devel-2.9-8.el8.x86_64.rpm 2.0 MB/s | 200 kB 00:00 2024-01-11T08:07:21.156 INFO:teuthology.orchestra.run.smithi027.stdout:(74/144): libsepol-devel-2.9-3.el8.x86_64.rpm 951 kB/s | 87 kB 00:00 2024-01-11T08:07:21.200 INFO:teuthology.orchestra.run.smithi077.stdout:(141/144): xfsdump-3.1.8-7.el8.x86_64.rpm 2.1 MB/s | 334 kB 00:00 2024-01-11T08:07:21.240 INFO:teuthology.orchestra.run.smithi027.stdout:(75/144): libuuid-devel-2.32.1-43.el8.x86_64.rp 1.2 MB/s | 100 kB 00:00 2024-01-11T08:07:21.365 INFO:teuthology.orchestra.run.smithi027.stdout:(76/144): libverto-devel-0.3.2-2.el8.x86_64.rpm 144 kB/s | 18 kB 00:00 2024-01-11T08:07:21.515 INFO:teuthology.orchestra.run.smithi027.stdout:(77/144): libzstd-devel-1.4.4-1.el8.x86_64.rpm 292 kB/s | 44 kB 00:00 2024-01-11T08:07:21.608 INFO:teuthology.orchestra.run.smithi077.stdout:(142/144): xfsprogs-5.0.0-12.el8.x86_64.rpm 2.7 MB/s | 1.1 MB 00:00 2024-01-11T08:07:22.108 INFO:teuthology.orchestra.run.smithi077.stdout:(143/144): util-linux-2.32.1-43.el8.x86_64.rpm 2.1 MB/s | 2.5 MB 00:01 2024-01-11T08:07:22.682 INFO:teuthology.orchestra.run.smithi027.stdout:(78/144): openssl-devel-1.1.1k-11.el8.x86_64.rp 2.7 MB/s | 3.2 MB 00:01 2024-01-11T08:07:22.757 INFO:teuthology.orchestra.run.smithi027.stdout:(79/144): python3-babel-2.5.1-7.el8.noarch.rpm 744 kB/s | 4.8 MB 00:06 2024-01-11T08:07:22.866 INFO:teuthology.orchestra.run.smithi027.stdout:(80/144): pcre2-devel-10.32-3.el8.x86_64.rpm 3.2 MB/s | 605 kB 00:00 2024-01-11T08:07:23.083 INFO:teuthology.orchestra.run.smithi027.stdout:(81/144): pcre2-utf32-10.32-3.el8.x86_64.rpm 1.0 MB/s | 220 kB 00:00 2024-01-11T08:07:23.192 INFO:teuthology.orchestra.run.smithi027.stdout:(82/144): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.2 MB/s | 249 kB 00:00 2024-01-11T08:07:23.300 INFO:teuthology.orchestra.run.smithi027.stdout:(83/144): pcre2-utf16-10.32-3.el8.x86_64.rpm 422 kB/s | 229 kB 00:00 2024-01-11T08:07:23.393 INFO:teuthology.orchestra.run.smithi027.stdout:(84/144): python3-cryptography-3.2.1-7.el8.x86_ 3.1 MB/s | 641 kB 00:00 2024-01-11T08:07:23.420 INFO:teuthology.orchestra.run.smithi027.stdout:(85/144): python3-jwt-1.6.1-2.el8.noarch.rpm 365 kB/s | 43 kB 00:00 2024-01-11T08:07:23.459 INFO:teuthology.orchestra.run.smithi027.stdout:(86/144): python3-oauthlib-2.1.0-1.el8.noarch.r 2.3 MB/s | 155 kB 00:00 2024-01-11T08:07:23.535 INFO:teuthology.orchestra.run.smithi027.stdout:(87/144): python3-requests-oauthlib-1.0.0-1.el8 568 kB/s | 43 kB 00:00 2024-01-11T08:07:23.577 INFO:teuthology.orchestra.run.smithi027.stdout:(88/144): python3-pycparser-2.14-14.el8.noarch. 686 kB/s | 109 kB 00:00 2024-01-11T08:07:23.619 INFO:teuthology.orchestra.run.smithi027.stdout:(89/144): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 895 kB/s | 74 kB 00:00 2024-01-11T08:07:23.719 INFO:teuthology.orchestra.run.smithi027.stdout:(90/144): zlib-devel-1.2.11-21.el8.x86_64.rpm 410 kB/s | 58 kB 00:00 2024-01-11T08:07:24.136 INFO:teuthology.orchestra.run.smithi027.stdout:(91/144): liboath-2.6.2-3.el8.x86_64.rpm 143 kB/s | 59 kB 00:00 2024-01-11T08:07:24.237 INFO:teuthology.orchestra.run.smithi027.stdout:(92/144): gperftools-libs-2.7-9.el8.x86_64.rpm 496 kB/s | 306 kB 00:00 2024-01-11T08:07:24.287 INFO:teuthology.orchestra.run.smithi027.stdout:(93/144): libunwind-1.3.1-3.el8.x86_64.rpm 501 kB/s | 75 kB 00:00 2024-01-11T08:07:24.412 INFO:teuthology.orchestra.run.smithi027.stdout:(94/144): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 349 kB/s | 44 kB 00:00 2024-01-11T08:07:24.437 INFO:teuthology.orchestra.run.smithi027.stdout:(95/144): python3-asyncssh-2.7.0-2.el8.noarch.r 2.3 MB/s | 476 kB 00:00 2024-01-11T08:07:24.546 INFO:teuthology.orchestra.run.smithi027.stdout:(96/144): python3-cachetools-3.1.1-4.el8.noarch 304 kB/s | 33 kB 00:00 2024-01-11T08:07:24.621 INFO:teuthology.orchestra.run.smithi027.stdout:(97/144): python3-beautifulsoup4-4.6.3-2.el8.1. 888 kB/s | 185 kB 00:00 2024-01-11T08:07:24.647 INFO:teuthology.orchestra.run.smithi027.stdout:(98/144): python3-certifi-2018.10.15-7.el8.noar 163 kB/s | 16 kB 00:00 2024-01-11T08:07:24.764 INFO:teuthology.orchestra.run.smithi027.stdout:(99/144): python3-cheroot-8.5.2-1.el8.noarch.rp 1.2 MB/s | 173 kB 00:00 2024-01-11T08:07:24.797 INFO:teuthology.orchestra.run.smithi027.stdout:(100/144): python3-cherrypy-18.4.0-1.el8.noarch 2.5 MB/s | 384 kB 00:00 2024-01-11T08:07:24.881 INFO:teuthology.orchestra.run.smithi027.stdout:(101/144): python3-google-auth-1.1.1-10.el8.noa 697 kB/s | 81 kB 00:00 2024-01-11T08:07:24.898 INFO:teuthology.orchestra.run.smithi027.stdout:(102/144): python3-jaraco-6.2-6.el8.noarch.rpm 110 kB/s | 11 kB 00:00 2024-01-11T08:07:24.973 INFO:teuthology.orchestra.run.smithi027.stdout:(103/144): python3-jaraco-functools-2.0-4.el8.n 192 kB/s | 18 kB 00:00 2024-01-11T08:07:25.073 INFO:teuthology.orchestra.run.smithi027.stdout:(104/144): python3-logutils-0.3.5-11.el8.noarch 488 kB/s | 49 kB 00:00 2024-01-11T08:07:25.174 INFO:teuthology.orchestra.run.smithi027.stdout:(105/144): python3-more-itertools-7.2.0-3.el8.n 588 kB/s | 59 kB 00:00 2024-01-11T08:07:25.208 INFO:teuthology.orchestra.run.smithi027.stdout:(106/144): python3-kubernetes-11.0.0-6.el8.noar 3.8 MB/s | 1.2 MB 00:00 2024-01-11T08:07:25.275 INFO:teuthology.orchestra.run.smithi027.stdout:(107/144): python3-natsort-7.1.1-2.el8.noarch.r 600 kB/s | 60 kB 00:00 2024-01-11T08:07:25.316 INFO:teuthology.orchestra.run.smithi027.stdout:(108/144): python3-pecan-1.3.2-9.el8.noarch.rpm 2.5 MB/s | 283 kB 00:00 2024-01-11T08:07:25.367 INFO:teuthology.orchestra.run.smithi027.stdout:(109/144): python3-portend-2.6-1.el8.noarch.rpm 170 kB/s | 16 kB 00:00 2024-01-11T08:07:25.408 INFO:teuthology.orchestra.run.smithi027.stdout:(110/144): python3-repoze-lru-0.7-6.el8.noarch. 367 kB/s | 34 kB 00:00 2024-01-11T08:07:25.475 INFO:teuthology.orchestra.run.smithi027.stdout:(111/144): python3-routes-2.4.1-12.el8.noarch.r 1.8 MB/s | 196 kB 00:00 2024-01-11T08:07:25.509 INFO:teuthology.orchestra.run.smithi027.stdout:(112/144): python3-rsa-4.9-2.el8.noarch.rpm 639 kB/s | 64 kB 00:00 2024-01-11T08:07:25.568 INFO:teuthology.orchestra.run.smithi027.stdout:(113/144): python3-simplegeneric-0.8.1-17.el8.n 203 kB/s | 19 kB 00:00 2024-01-11T08:07:25.601 INFO:teuthology.orchestra.run.smithi027.stdout:(114/144): python3-singledispatch-3.4.0.3-18.el 261 kB/s | 24 kB 00:00 2024-01-11T08:07:25.660 INFO:teuthology.orchestra.run.smithi027.stdout:(115/144): python3-tempora-1.14.1-5.el8.noarch. 318 kB/s | 29 kB 00:00 2024-01-11T08:07:25.694 INFO:teuthology.orchestra.run.smithi027.stdout:(116/144): python3-trustme-0.6.0-4.el8.noarch.r 300 kB/s | 27 kB 00:00 2024-01-11T08:07:25.786 INFO:teuthology.orchestra.run.smithi027.stdout:(117/144): python3-waitress-1.4.3-1.el8.noarch. 2.0 MB/s | 255 kB 00:00 2024-01-11T08:07:25.811 INFO:teuthology.orchestra.run.smithi027.stdout:(118/144): python3-webob-1.8.5-1.el8.1.noarch.r 2.1 MB/s | 251 kB 00:00 2024-01-11T08:07:25.953 INFO:teuthology.orchestra.run.smithi027.stdout:(119/144): python3-websocket-client-0.56.0-5.el 367 kB/s | 61 kB 00:00 2024-01-11T08:07:25.978 INFO:teuthology.orchestra.run.smithi027.stdout:(120/144): python3-webtest-2.0.33-1.el8.noarch. 511 kB/s | 85 kB 00:00 2024-01-11T08:07:26.054 INFO:teuthology.orchestra.run.smithi027.stdout:(121/144): python3-zc-lockfile-2.0-2.el8.noarch 230 kB/s | 23 kB 00:00 2024-01-11T08:07:26.154 INFO:teuthology.orchestra.run.smithi027.stdout:(122/144): thrift-0.13.0-2.el8.x86_64.rpm 9.9 MB/s | 1.7 MB 00:00 2024-01-11T08:07:26.680 INFO:teuthology.orchestra.run.smithi027.stdout:(123/144): librbd1-17.2.7-299.g4bcedb9f.el8.x86 6.8 MB/s | 3.6 MB 00:00 2024-01-11T08:07:26.830 INFO:teuthology.orchestra.run.smithi027.stdout:(124/144): NetworkManager-initscripts-updown-1. 964 kB/s | 144 kB 00:00 2024-01-11T08:07:26.888 INFO:teuthology.orchestra.run.smithi027.stdout:(125/144): acl-2.2.53-3.el8.x86_64.rpm 1.4 MB/s | 83 kB 00:00 2024-01-11T08:07:26.939 INFO:teuthology.orchestra.run.smithi027.stdout:(126/144): elfutils-debuginfod-client-0.190-2.e 1.5 MB/s | 77 kB 00:00 2024-01-11T08:07:26.997 INFO:teuthology.orchestra.run.smithi027.stdout:(127/144): librados2-17.2.7-299.g4bcedb9f.el8.x 3.7 MB/s | 3.5 MB 00:00 2024-01-11T08:07:27.064 INFO:teuthology.orchestra.run.smithi027.stdout:(128/144): elfutils-libelf-0.190-2.el8.x86_64.r 1.8 MB/s | 233 kB 00:00 2024-01-11T08:07:27.290 INFO:teuthology.orchestra.run.smithi027.stdout:(129/144): iproute-6.2.0-5.el8.x86_64.rpm 3.7 MB/s | 854 kB 00:00 2024-01-11T08:07:27.348 INFO:teuthology.orchestra.run.smithi027.stdout:(130/144): elfutils-libs-0.190-2.el8.x86_64.rpm 871 kB/s | 305 kB 00:00 2024-01-11T08:07:27.374 INFO:teuthology.orchestra.run.smithi027.stdout:(131/144): libacl-2.2.53-3.el8.x86_64.rpm 416 kB/s | 35 kB 00:00 2024-01-11T08:07:27.432 INFO:teuthology.orchestra.run.smithi027.stdout:(132/144): libacl-devel-2.2.53-3.el8.x86_64.rpm 1.0 MB/s | 85 kB 00:00 2024-01-11T08:07:27.457 INFO:teuthology.orchestra.run.smithi027.stdout:(133/144): libblkid-2.32.1-43.el8.x86_64.rpm 2.6 MB/s | 220 kB 00:00 2024-01-11T08:07:27.558 INFO:teuthology.orchestra.run.smithi027.stdout:(134/144): libmount-2.32.1-43.el8.x86_64.rpm 2.3 MB/s | 236 kB 00:00 2024-01-11T08:07:27.650 INFO:teuthology.orchestra.run.smithi027.stdout:(135/144): libfdisk-2.32.1-43.el8.x86_64.rpm 1.1 MB/s | 254 kB 00:00 2024-01-11T08:07:27.675 INFO:teuthology.orchestra.run.smithi027.stdout:(136/144): libsmartcols-2.32.1-43.el8.x86_64.rp 1.5 MB/s | 179 kB 00:00 2024-01-11T08:07:27.768 INFO:teuthology.orchestra.run.smithi077.stdout:(144/144): ceph-test-17.2.7-299.g4bcedb9f.el8.x 2.4 MB/s | 52 MB 00:21 2024-01-11T08:07:27.770 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-01-11T08:07:27.771 INFO:teuthology.orchestra.run.smithi077.stdout:Total 8.3 MB/s | 213 MB 00:25 2024-01-11T08:07:27.833 INFO:teuthology.orchestra.run.smithi027.stdout:(137/144): python3-scipy-1.0.0-21.module_el8+76 1.6 MB/s | 14 MB 00:08 2024-01-11T08:07:27.859 INFO:teuthology.orchestra.run.smithi027.stdout:(138/144): libuuid-2.32.1-43.el8.x86_64.rpm 474 kB/s | 99 kB 00:00 2024-01-11T08:07:27.917 INFO:teuthology.orchestra.run.smithi027.stdout:(139/144): openssl-1.1.1k-11.el8.x86_64.rpm 3.0 MB/s | 737 kB 00:00 2024-01-11T08:07:27.976 INFO:teuthology.orchestra.run.smithi027.stdout:(140/144): util-linux-user-2.32.1-43.el8.x86_64 1.7 MB/s | 103 kB 00:00 2024-01-11T08:07:28.060 INFO:teuthology.orchestra.run.smithi027.stdout:(141/144): xfsdump-3.1.8-7.el8.x86_64.rpm 3.9 MB/s | 334 kB 00:00 2024-01-11T08:07:28.185 INFO:teuthology.orchestra.run.smithi027.stdout:(142/144): openssl-libs-1.1.1k-11.el8.x86_64.rp 4.2 MB/s | 1.5 MB 00:00 2024-01-11T08:07:28.244 INFO:teuthology.orchestra.run.smithi027.stdout:(143/144): xfsprogs-5.0.0-12.el8.x86_64.rpm 5.9 MB/s | 1.1 MB 00:00 2024-01-11T08:07:28.810 INFO:teuthology.orchestra.run.smithi027.stdout:(144/144): util-linux-2.32.1-43.el8.x86_64.rpm 2.6 MB/s | 2.5 MB 00:00 2024-01-11T08:07:28.815 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-01-11T08:07:28.815 INFO:teuthology.orchestra.run.smithi027.stdout:Total 8.9 MB/s | 213 MB 00:23 2024-01-11T08:07:29.635 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-01-11T08:07:30.083 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-01-11T08:07:30.083 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-01-11T08:07:30.746 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-01-11T08:07:31.233 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-01-11T08:07:31.233 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-01-11T08:07:31.537 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-01-11T08:07:31.541 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-01-11T08:07:32.744 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-01-11T08:07:32.748 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-01-11T08:07:37.153 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-01-11T08:07:37.341 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openssl-libs-1:1.1.1k-11.el8.x86_64 1/1 2024-01-11T08:07:37.547 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : openssl-libs-1:1.1.1k-11.el8.x86_64 1/165 2024-01-11T08:07:37.627 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openssl-libs-1:1.1.1k-11.el8.x86_64 1/165 2024-01-11T08:07:37.776 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 2/165 2024-01-11T08:07:37.912 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 2/165 2024-01-11T08:07:38.118 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 3/165 2024-01-11T08:07:38.209 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 3/165 2024-01-11T08:07:38.436 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-01-11T08:07:38.492 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/165 2024-01-11T08:07:38.633 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openssl-libs-1:1.1.1k-11.el8.x86_64 1/1 2024-01-11T08:07:38.682 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 5/165 2024-01-11T08:07:38.825 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : openssl-libs-1:1.1.1k-11.el8.x86_64 1/165 2024-01-11T08:07:38.912 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openssl-libs-1:1.1.1k-11.el8.x86_64 1/165 2024-01-11T08:07:39.041 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 6/165 2024-01-11T08:07:39.061 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 2/165 2024-01-11T08:07:39.166 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 2/165 2024-01-11T08:07:39.304 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/165 2024-01-11T08:07:39.411 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 3/165 2024-01-11T08:07:39.503 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 3/165 2024-01-11T08:07:39.552 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-libelf-0.190-2.el8.x86_64 8/165 2024-01-11T08:07:39.755 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/165 2024-01-11T08:07:39.933 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 5/165 2024-01-11T08:07:39.942 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 9/165 2024-01-11T08:07:40.270 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 6/165 2024-01-11T08:07:40.372 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 9/165 2024-01-11T08:07:40.548 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/165 2024-01-11T08:07:40.551 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librados2-2:17.2.7-299.g4bcedb9f.el8.x86_64 10/165 2024-01-11T08:07:40.835 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : elfutils-libelf-0.190-2.el8.x86_64 8/165 2024-01-11T08:07:40.998 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:17.2.7-299.g4bcedb9f.el8.x86_64 10/165 2024-01-11T08:07:41.002 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 9/165 2024-01-11T08:07:41.235 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librbd1-2:17.2.7-299.g4bcedb9f.el8.x86_64 11/165 2024-01-11T08:07:41.395 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:17.2.7-299.g4bcedb9f.el8.x86_64 11/165 2024-01-11T08:07:41.472 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 9/165 2024-01-11T08:07:41.592 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs2-2:17.2.7-299.g4bcedb9f.el8.x86_64 12/165 2024-01-11T08:07:41.652 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:17.2.7-299.g4bcedb9f.el8.x86_64 10/165 2024-01-11T08:07:41.704 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs2-2:17.2.7-299.g4bcedb9f.el8.x86_64 12/165 2024-01-11T08:07:42.127 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:17.2.7-299.g4bcedb9f.el8.x86_64 10/165 2024-01-11T08:07:42.327 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:17.2.7-299.g4bcedb9f.el8.x86_64 11/165 2024-01-11T08:07:42.417 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rados-2:17.2.7-299.g4bcedb9f.el8.x86_64 13/165 2024-01-11T08:07:42.485 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:17.2.7-299.g4bcedb9f.el8.x86_64 11/165 2024-01-11T08:07:42.584 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librgw2-2:17.2.7-299.g4bcedb9f.el8.x86_64 14/165 2024-01-11T08:07:42.669 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs2-2:17.2.7-299.g4bcedb9f.el8.x86_64 12/165 2024-01-11T08:07:42.793 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:17.2.7-299.g4bcedb9f.el8.x86_64 12/165 2024-01-11T08:07:42.821 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librgw2-2:17.2.7-299.g4bcedb9f.el8.x86_64 14/165 2024-01-11T08:07:43.042 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 15/165 2024-01-11T08:07:43.134 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 15/165 2024-01-11T08:07:43.333 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 16/165 2024-01-11T08:07:43.497 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 17/165 2024-01-11T08:07:43.558 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rados-2:17.2.7-299.g4bcedb9f.el8.x86_64 13/165 2024-01-11T08:07:43.626 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephsqlite-2:17.2.7-299.g4bcedb9f.el8.x86_64 18/165 2024-01-11T08:07:43.744 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librgw2-2:17.2.7-299.g4bcedb9f.el8.x86_64 14/165 2024-01-11T08:07:43.755 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephsqlite-2:17.2.7-299.g4bcedb9f.el8.x86_64 18/165 2024-01-11T08:07:43.909 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libradosstriper1-2:17.2.7-299.g4bcedb9f.el8.x86_ 19/165 2024-01-11T08:07:43.970 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:17.2.7-299.g4bcedb9f.el8.x86_64 14/165 2024-01-11T08:07:44.002 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libradosstriper1-2:17.2.7-299.g4bcedb9f.el8.x86_ 19/165 2024-01-11T08:07:44.208 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libacl-2.2.53-3.el8.x86_64 20/165 2024-01-11T08:07:44.218 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 15/165 2024-01-11T08:07:44.295 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 15/165 2024-01-11T08:07:44.464 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 21/165 2024-01-11T08:07:44.537 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 16/165 2024-01-11T08:07:44.698 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : zlib-devel-1.2.11-21.el8.x86_64 22/165 2024-01-11T08:07:44.869 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 17/165 2024-01-11T08:07:44.900 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 23/165 2024-01-11T08:07:44.993 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephsqlite-2:17.2.7-299.g4bcedb9f.el8.x86_64 18/165 2024-01-11T08:07:45.066 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 23/165 2024-01-11T08:07:45.127 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:17.2.7-299.g4bcedb9f.el8.x86_64 18/165 2024-01-11T08:07:45.267 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 24/165 2024-01-11T08:07:45.285 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libradosstriper1-2:17.2.7-299.g4bcedb9f.el8.x86_ 19/165 2024-01-11T08:07:45.370 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:17.2.7-299.g4bcedb9f.el8.x86_ 19/165 2024-01-11T08:07:45.616 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libacl-2.2.53-3.el8.x86_64 20/165 2024-01-11T08:07:45.820 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 21/165 2024-01-11T08:07:45.926 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 24/165 2024-01-11T08:07:45.997 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : zlib-devel-1.2.11-21.el8.x86_64 22/165 2024-01-11T08:07:46.177 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 23/165 2024-01-11T08:07:46.192 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-0.3.15-4.el8.x86_64 25/165 2024-01-11T08:07:46.347 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 23/165 2024-01-11T08:07:46.726 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 24/165 2024-01-11T08:07:46.871 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 25/165 2024-01-11T08:07:47.058 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 26/165 2024-01-11T08:07:47.415 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 24/165 2024-01-11T08:07:47.610 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-0.3.15-4.el8.x86_64 25/165 2024-01-11T08:07:47.867 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 26/165 2024-01-11T08:07:48.222 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 27/165 2024-01-11T08:07:48.329 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 25/165 2024-01-11T08:07:48.433 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 28/165 2024-01-11T08:07:48.476 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 26/165 2024-01-11T08:07:48.665 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 29/165 2024-01-11T08:07:48.884 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 30/165 2024-01-11T08:07:49.141 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-common-2:17.2.7-299.g4bcedb9f.el8.x 31/165 2024-01-11T08:07:49.270 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-argparse-2:17.2.7-299.g4bcedb9f.el8 32/165 2024-01-11T08:07:49.349 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 26/165 2024-01-11T08:07:49.517 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cephfs-2:17.2.7-299.g4bcedb9f.el8.x86_64 33/165 2024-01-11T08:07:49.715 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 27/165 2024-01-11T08:07:49.919 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 28/165 2024-01-11T08:07:50.092 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 29/165 2024-01-11T08:07:50.289 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 30/165 2024-01-11T08:07:50.534 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-common-2:17.2.7-299.g4bcedb9f.el8.x 31/165 2024-01-11T08:07:50.637 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 34/165 2024-01-11T08:07:50.723 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-argparse-2:17.2.7-299.g4bcedb9f.el8 32/165 2024-01-11T08:07:51.014 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cephfs-2:17.2.7-299.g4bcedb9f.el8.x86_64 33/165 2024-01-11T08:07:51.025 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 35/165 2024-01-11T08:07:51.375 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 36/165 2024-01-11T08:07:51.558 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/165 2024-01-11T08:07:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 34/165 2024-01-11T08:07:52.503 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 35/165 2024-01-11T08:07:52.893 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 36/165 2024-01-11T08:07:53.084 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/165 2024-01-11T08:07:53.612 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/165 2024-01-11T08:07:53.920 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 38/165 2024-01-11T08:07:54.156 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 39/165 2024-01-11T08:07:54.379 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rsa-4.9-2.el8.noarch 40/165 2024-01-11T08:07:54.629 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rgw-2:17.2.7-299.g4bcedb9f.el8.x86_64 41/165 2024-01-11T08:07:54.875 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rbd-2:17.2.7-299.g4bcedb9f.el8.x86_64 42/165 2024-01-11T08:07:55.135 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librados-devel-2:17.2.7-299.g4bcedb9f.el8.x86_64 43/165 2024-01-11T08:07:55.258 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/165 2024-01-11T08:07:55.402 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-libs-0.190-2.el8.x86_64 44/165 2024-01-11T08:07:55.598 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 38/165 2024-01-11T08:07:55.649 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el8.x86_64 45/165 2024-01-11T08:07:55.826 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 39/165 2024-01-11T08:07:55.899 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 46/165 2024-01-11T08:07:55.978 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 46/165 2024-01-11T08:07:56.014 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rsa-4.9-2.el8.noarch 40/165 2024-01-11T08:07:56.216 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 47/165 2024-01-11T08:07:56.224 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rgw-2:17.2.7-299.g4bcedb9f.el8.x86_64 41/165 2024-01-11T08:07:56.304 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 47/165 2024-01-11T08:07:56.435 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rbd-2:17.2.7-299.g4bcedb9f.el8.x86_64 42/165 2024-01-11T08:07:56.500 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 48/165 2024-01-11T08:07:56.710 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librados-devel-2:17.2.7-299.g4bcedb9f.el8.x86_64 43/165 2024-01-11T08:07:56.730 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 49/165 2024-01-11T08:07:56.941 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 50/165 2024-01-11T08:07:56.948 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : elfutils-libs-0.190-2.el8.x86_64 44/165 2024-01-11T08:07:57.154 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el8.x86_64 45/165 2024-01-11T08:07:57.383 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 46/165 2024-01-11T08:07:57.485 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 46/165 2024-01-11T08:07:57.549 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 50/165 2024-01-11T08:07:57.692 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 47/165 2024-01-11T08:07:57.767 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 47/165 2024-01-11T08:07:57.833 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 51/165 2024-01-11T08:07:57.862 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 51/165 2024-01-11T08:07:57.966 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 48/165 2024-01-11T08:07:58.098 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 52/165 2024-01-11T08:07:58.152 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 49/165 2024-01-11T08:07:58.342 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 50/165 2024-01-11T08:07:58.372 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 53/165 2024-01-11T08:07:58.599 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 54/165 2024-01-11T08:07:58.804 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 55/165 2024-01-11T08:07:58.945 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 50/165 2024-01-11T08:07:58.978 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 56/165 2024-01-11T08:07:59.203 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 57/165 2024-01-11T08:07:59.259 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 51/165 2024-01-11T08:07:59.288 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 51/165 2024-01-11T08:07:59.432 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 58/165 2024-01-11T08:07:59.525 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 52/165 2024-01-11T08:07:59.706 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 59/165 2024-01-11T08:07:59.812 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 53/165 2024-01-11T08:07:59.887 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 60/165 2024-01-11T08:08:00.040 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 61/165 2024-01-11T08:08:00.058 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 54/165 2024-01-11T08:08:00.255 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 55/165 2024-01-11T08:08:00.263 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 62/165 2024-01-11T08:08:00.438 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 56/165 2024-01-11T08:08:00.472 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 63/165 2024-01-11T08:08:00.649 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 57/165 2024-01-11T08:08:00.667 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-portend-2.6-1.el8.noarch 64/165 2024-01-11T08:08:00.851 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 58/165 2024-01-11T08:08:00.885 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 65/165 2024-01-11T08:08:01.115 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 59/165 2024-01-11T08:08:01.141 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 66/165 2024-01-11T08:08:01.312 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 60/165 2024-01-11T08:08:01.395 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 67/165 2024-01-11T08:08:01.474 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 61/165 2024-01-11T08:08:01.657 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 68/165 2024-01-11T08:08:01.674 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 62/165 2024-01-11T08:08:01.957 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 63/165 2024-01-11T08:08:02.008 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 69/165 2024-01-11T08:08:02.161 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-portend-2.6-1.el8.noarch 64/165 2024-01-11T08:08:02.276 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 70/165 2024-01-11T08:08:02.388 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 65/165 2024-01-11T08:08:02.529 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 71/165 2024-01-11T08:08:02.613 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 66/165 2024-01-11T08:08:02.751 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 72/165 2024-01-11T08:08:02.942 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 67/165 2024-01-11T08:08:03.008 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 73/165 2024-01-11T08:08:03.193 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 68/165 2024-01-11T08:08:03.476 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 74/165 2024-01-11T08:08:03.565 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 69/165 2024-01-11T08:08:03.796 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 75/165 2024-01-11T08:08:03.852 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 70/165 2024-01-11T08:08:04.004 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 76/165 2024-01-11T08:08:04.082 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 71/165 2024-01-11T08:08:04.290 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 77/165 2024-01-11T08:08:04.321 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 72/165 2024-01-11T08:08:04.533 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 78/165 2024-01-11T08:08:04.562 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 73/165 2024-01-11T08:08:04.996 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 74/165 2024-01-11T08:08:05.257 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 75/165 2024-01-11T08:08:05.491 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 76/165 2024-01-11T08:08:05.784 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 77/165 2024-01-11T08:08:05.838 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 79/165 2024-01-11T08:08:06.003 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 78/165 2024-01-11T08:08:06.244 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 80/165 2024-01-11T08:08:06.482 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 81/165 2024-01-11T08:08:06.733 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 82/165 2024-01-11T08:08:07.079 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 83/165 2024-01-11T08:08:07.342 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 79/165 2024-01-11T08:08:07.401 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 84/165 2024-01-11T08:08:07.718 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 85/165 2024-01-11T08:08:07.773 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 80/165 2024-01-11T08:08:07.944 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : pcre2-utf32-10.32-3.el8.x86_64 86/165 2024-01-11T08:08:07.993 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 81/165 2024-01-11T08:08:08.324 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : pcre2-utf16-10.32-3.el8.x86_64 87/165 2024-01-11T08:08:08.389 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 82/165 2024-01-11T08:08:08.595 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : pcre2-devel-10.32-3.el8.x86_64 88/165 2024-01-11T08:08:08.738 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 83/165 2024-01-11T08:08:08.794 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 89/165 2024-01-11T08:08:08.975 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libverto-devel-0.3.2-2.el8.x86_64 90/165 2024-01-11T08:08:09.011 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 84/165 2024-01-11T08:08:09.271 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 85/165 2024-01-11T08:08:09.363 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 91/165 2024-01-11T08:08:09.489 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : pcre2-utf32-10.32-3.el8.x86_64 86/165 2024-01-11T08:08:09.653 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libselinux-devel-2.9-8.el8.x86_64 92/165 2024-01-11T08:08:09.822 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcom_err-devel-1.45.6-5.el8.x86_64 93/165 2024-01-11T08:08:09.865 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : pcre2-utf16-10.32-3.el8.x86_64 87/165 2024-01-11T08:08:10.083 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 94/165 2024-01-11T08:08:10.138 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : pcre2-devel-10.32-3.el8.x86_64 88/165 2024-01-11T08:08:10.316 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 89/165 2024-01-11T08:08:10.511 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libverto-devel-0.3.2-2.el8.x86_64 90/165 2024-01-11T08:08:10.885 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : krb5-devel-1.18.2-22.el8.x86_64 95/165 2024-01-11T08:08:10.887 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 91/165 2024-01-11T08:08:11.151 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fuse-2.9.7-17.el8.x86_64 96/165 2024-01-11T08:08:11.179 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libselinux-devel-2.9-8.el8.x86_64 92/165 2024-01-11T08:08:11.332 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcom_err-devel-1.45.6-5.el8.x86_64 93/165 2024-01-11T08:08:11.366 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 97/165 2024-01-11T08:08:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 94/165 2024-01-11T08:08:11.564 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 98/165 2024-01-11T08:08:11.777 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : krb5-devel-1.18.2-22.el8.x86_64 95/165 2024-01-11T08:08:12.320 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fuse-2.9.7-17.el8.x86_64 96/165 2024-01-11T08:08:12.551 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 97/165 2024-01-11T08:08:12.690 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 98/165 2024-01-11T08:08:14.538 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:17.2.7-299.g4bcedb9f.el8.x86_64 99/165 2024-01-11T08:08:14.839 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-common-2:17.2.7-299.g4bcedb9f.el8.x86_64 99/165 2024-01-11T08:08:15.477 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:17.2.7-299.g4bcedb9f.el8.x86_64 99/165 2024-01-11T08:08:15.801 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-base-2:17.2.7-299.g4bcedb9f.el8.x86_64 100/165 2024-01-11T08:08:15.852 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:17.2.7-299.g4bcedb9f.el8.x86_64 99/165 2024-01-11T08:08:16.100 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-base-2:17.2.7-299.g4bcedb9f.el8.x86_64 100/165 2024-01-11T08:08:16.107 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-common-2:17.2.7-299.g4bcedb9f.el8.x86_64 99/165 2024-01-11T08:08:16.747 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-selinux-2:17.2.7-299.g4bcedb9f.el8.x86_64 101/165 2024-01-11T08:08:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:17.2.7-299.g4bcedb9f.el8.x86_64 99/165 2024-01-11T08:08:18.212 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-base-2:17.2.7-299.g4bcedb9f.el8.x86_64 100/165 2024-01-11T08:08:18.444 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:17.2.7-299.g4bcedb9f.el8.x86_64 100/165 2024-01-11T08:08:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-selinux-2:17.2.7-299.g4bcedb9f.el8.x86_64 101/165 2024-01-11T08:08:37.554 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-selinux-2:17.2.7-299.g4bcedb9f.el8.x86_64 101/165 2024-01-11T08:08:37.555 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /sys 2024-01-11T08:08:37.555 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /proc 2024-01-11T08:08:37.555 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /dev 2024-01-11T08:08:37.555 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /run 2024-01-11T08:08:37.555 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /mnt 2024-01-11T08:08:37.555 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /var/tmp 2024-01-11T08:08:37.555 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /home 2024-01-11T08:08:37.555 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /tmp 2024-01-11T08:08:37.555 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /dev 2024-01-11T08:08:37.555 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:08:38.011 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-volume-2:17.2.7-299.g4bcedb9f.el8.noarch 102/165 2024-01-11T08:08:39.616 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-volume-2:17.2.7-299.g4bcedb9f.el8.noarch 102/165 2024-01-11T08:08:39.882 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-osd-2:17.2.7-299.g4bcedb9f.el8.x86_64 103/165 2024-01-11T08:08:40.148 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-osd-2:17.2.7-299.g4bcedb9f.el8.x86_64 103/165 2024-01-11T08:08:40.353 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mds-2:17.2.7-299.g4bcedb9f.el8.x86_64 104/165 2024-01-11T08:08:40.452 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:17.2.7-299.g4bcedb9f.el8.x86_64 101/165 2024-01-11T08:08:40.452 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /sys 2024-01-11T08:08:40.452 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /proc 2024-01-11T08:08:40.452 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /dev 2024-01-11T08:08:40.452 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /run 2024-01-11T08:08:40.453 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /mnt 2024-01-11T08:08:40.453 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /var/tmp 2024-01-11T08:08:40.453 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /home 2024-01-11T08:08:40.453 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /tmp 2024-01-11T08:08:40.453 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /dev 2024-01-11T08:08:40.453 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:08:40.811 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mds-2:17.2.7-299.g4bcedb9f.el8.x86_64 104/165 2024-01-11T08:08:40.854 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-volume-2:17.2.7-299.g4bcedb9f.el8.noarch 102/165 2024-01-11T08:08:41.061 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mon-2:17.2.7-299.g4bcedb9f.el8.x86_64 105/165 2024-01-11T08:08:41.111 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mon-2:17.2.7-299.g4bcedb9f.el8.x86_64 105/165 2024-01-11T08:08:41.369 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 106/165 2024-01-11T08:08:41.607 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 107/165 2024-01-11T08:08:41.827 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 108/165 2024-01-11T08:08:41.911 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 108/165 2024-01-11T08:08:42.230 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : jq-1.6-8.el8.x86_64 109/165 2024-01-11T08:08:42.494 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : automake-1.16.1-8.el8.noarch 110/165 2024-01-11T08:08:42.559 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-volume-2:17.2.7-299.g4bcedb9f.el8.noarch 102/165 2024-01-11T08:08:42.842 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-osd-2:17.2.7-299.g4bcedb9f.el8.x86_64 103/165 2024-01-11T08:08:43.126 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:17.2.7-299.g4bcedb9f.el8.x86_64 103/165 2024-01-11T08:08:43.136 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:17.2.7-299.g4bcedb9f.el8.noarch 111/165 2024-01-11T08:08:43.342 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephadm-2:17.2.7-299.g4bcedb9f.el8.noarch 111/165 2024-01-11T08:08:43.345 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mds-2:17.2.7-299.g4bcedb9f.el8.x86_64 104/165 2024-01-11T08:08:43.476 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-299.g4bcedb9f.el 112/165 2024-01-11T08:08:43.701 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-299.g4bcedb9f.e 113/165 2024-01-11T08:08:43.829 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:17.2.7-299.g4bcedb9f.el8.x86_64 104/165 2024-01-11T08:08:43.968 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-299.g4bcedb9f.el8.noar 114/165 2024-01-11T08:08:44.028 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mon-2:17.2.7-299.g4bcedb9f.el8.x86_64 105/165 2024-01-11T08:08:44.081 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:17.2.7-299.g4bcedb9f.el8.x86_64 105/165 2024-01-11T08:08:44.221 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-299.g4bcedb9f.el8.noar 114/165 2024-01-11T08:08:44.279 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 106/165 2024-01-11T08:08:44.522 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-299.g4bcedb9f.el8.no 115/165 2024-01-11T08:08:44.525 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 107/165 2024-01-11T08:08:44.719 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 108/165 2024-01-11T08:08:44.816 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 108/165 2024-01-11T08:08:45.137 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : jq-1.6-8.el8.x86_64 109/165 2024-01-11T08:08:45.378 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : automake-1.16.1-8.el8.noarch 110/165 2024-01-11T08:08:45.605 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-299.g4bcedb9f.el8.no 115/165 2024-01-11T08:08:45.860 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-299.g4bce 116/165 2024-01-11T08:08:45.988 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-299.g4bce 116/165 2024-01-11T08:08:46.170 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:17.2.7-299.g4bcedb9f.el8.noarch 111/165 2024-01-11T08:08:46.412 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-299.g4bcedb9f.el8 117/165 2024-01-11T08:08:46.412 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephadm-2:17.2.7-299.g4bcedb9f.el8.noarch 111/165 2024-01-11T08:08:46.578 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-299.g4bcedb9f.el 112/165 2024-01-11T08:08:46.664 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-2:17.2.7-299.g4bcedb9f.el8.x86_64 118/165 2024-01-11T08:08:46.730 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-2:17.2.7-299.g4bcedb9f.el8.x86_64 118/165 2024-01-11T08:08:46.780 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-299.g4bcedb9f.e 113/165 2024-01-11T08:08:46.918 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-rook-2:17.2.7-299.g4bcedb9f.el8.noarch 119/165 2024-01-11T08:08:46.945 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-299.g4bcedb9f.el8.noarch 119/165 2024-01-11T08:08:46.985 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-299.g4bcedb9f.el8.noar 114/165 2024-01-11T08:08:47.199 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-2:17.2.7-299.g4bcedb9f.el8.x86_64 120/165 2024-01-11T08:08:47.255 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-299.g4bcedb9f.el8.noar 114/165 2024-01-11T08:08:47.376 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libtool-2.4.6-25.el8.x86_64 121/165 2024-01-11T08:08:47.533 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-299.g4bcedb9f.el8.no 115/165 2024-01-11T08:08:48.692 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-299.g4bcedb9f.el8.no 115/165 2024-01-11T08:08:48.919 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-299.g4bce 116/165 2024-01-11T08:08:49.057 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-299.g4bce 116/165 2024-01-11T08:08:49.509 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-299.g4bcedb9f.el8 117/165 2024-01-11T08:08:49.700 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-2:17.2.7-299.g4bcedb9f.el8.x86_64 118/165 2024-01-11T08:08:49.768 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:17.2.7-299.g4bcedb9f.el8.x86_64 118/165 2024-01-11T08:08:50.002 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-rook-2:17.2.7-299.g4bcedb9f.el8.noarch 119/165 2024-01-11T08:08:50.031 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-299.g4bcedb9f.el8.noarch 119/165 2024-01-11T08:08:50.275 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-2:17.2.7-299.g4bcedb9f.el8.x86_64 120/165 2024-01-11T08:08:50.443 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libtool-2.4.6-25.el8.x86_64 121/165 2024-01-11T08:08:52.656 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 121/165 2024-01-11T08:08:54.537 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-test-2:17.2.7-299.g4bcedb9f.el8.x86_64 122/165 2024-01-11T08:08:54.780 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-radosgw-2:17.2.7-299.g4bcedb9f.el8.x86_64 123/165 2024-01-11T08:08:54.906 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-299.g4bcedb9f.el8.x86_64 123/165 2024-01-11T08:08:55.151 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-299.g4bcedb 124/165 2024-01-11T08:08:55.221 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-299.g4bcedb 124/165 2024-01-11T08:08:55.450 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-mirror-2:17.2.7-299.g4bcedb9f.el8.x86_64 125/165 2024-01-11T08:08:55.852 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephfs-mirror-2:17.2.7-299.g4bcedb9f.el8.x86_64 125/165 2024-01-11T08:08:56.082 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 121/165 2024-01-11T08:08:56.125 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-mirror-2:17.2.7-299.g4bcedb9f.el8.x86_64 126/165 2024-01-11T08:08:56.260 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-mirror-2:17.2.7-299.g4bcedb9f.el8.x86_64 126/165 2024-01-11T08:08:57.547 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-fuse-2:17.2.7-299.g4bcedb9f.el8.x86_64 127/165 2024-01-11T08:08:57.922 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openssl-devel-1:1.1.1k-11.el8.x86_64 128/165 2024-01-11T08:08:58.059 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-test-2:17.2.7-299.g4bcedb9f.el8.x86_64 122/165 2024-01-11T08:08:58.183 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : elfutils-libelf-devel-0.190-2.el8.x86_64 129/165 2024-01-11T08:08:58.266 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-radosgw-2:17.2.7-299.g4bcedb9f.el8.x86_64 123/165 2024-01-11T08:08:58.373 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 130/165 2024-01-11T08:08:58.394 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-299.g4bcedb9f.el8.x86_64 123/165 2024-01-11T08:08:58.579 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 131/165 2024-01-11T08:08:58.610 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-299.g4bcedb 124/165 2024-01-11T08:08:58.684 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-299.g4bcedb 124/165 2024-01-11T08:08:58.793 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs-devel-2:17.2.7-299.g4bcedb9f.el8.x86_6 132/165 2024-01-11T08:08:58.910 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-mirror-2:17.2.7-299.g4bcedb9f.el8.x86_64 125/165 2024-01-11T08:08:59.091 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : acl-2.2.53-3.el8.x86_64 133/165 2024-01-11T08:08:59.340 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephfs-mirror-2:17.2.7-299.g4bcedb9f.el8.x86_64 125/165 2024-01-11T08:08:59.343 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libacl-devel-2.2.53-3.el8.x86_64 134/165 2024-01-11T08:08:59.576 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-mirror-2:17.2.7-299.g4bcedb9f.el8.x86_64 126/165 2024-01-11T08:08:59.589 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : xfsdump-3.1.8-7.el8.x86_64 135/165 2024-01-11T08:08:59.719 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:17.2.7-299.g4bcedb9f.el8.x86_64 126/165 2024-01-11T08:08:59.792 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-top-2:17.2.7-299.g4bcedb9f.el8.noarch 136/165 2024-01-11T08:09:00.059 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-fuse-2:17.2.7-299.g4bcedb9f.el8.x86_64 137/165 2024-01-11T08:09:00.471 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-nbd-2:17.2.7-299.g4bcedb9f.el8.x86_64 138/165 2024-01-11T08:09:00.886 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : iproute-6.2.0-5.el8.x86_64 139/165 2024-01-11T08:09:01.136 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-fuse-2:17.2.7-299.g4bcedb9f.el8.x86_64 127/165 2024-01-11T08:09:01.177 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : openssl-1:1.1.1k-11.el8.x86_64 140/165 2024-01-11T08:09:01.383 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-initscripts-updown-1:1.40.16-14.e 141/165 2024-01-11T08:09:01.410 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-14.e 141/165 2024-01-11T08:09:01.515 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openssl-devel-1:1.1.1k-11.el8.x86_64 128/165 2024-01-11T08:09:01.631 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 142/165 2024-01-11T08:09:01.764 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : elfutils-libelf-devel-0.190-2.el8.x86_64 129/165 2024-01-11T08:09:01.941 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flex-2.6.1-9.el8.x86_64 143/165 2024-01-11T08:09:01.959 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 130/165 2024-01-11T08:09:02.047 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 143/165 2024-01-11T08:09:02.172 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 131/165 2024-01-11T08:09:02.299 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : bison-3.0.4-10.el8.x86_64 144/165 2024-01-11T08:09:02.327 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 144/165 2024-01-11T08:09:02.416 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs-devel-2:17.2.7-299.g4bcedb9f.el8.x86_6 132/165 2024-01-11T08:09:02.585 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-debuginfod-client-0.189-1.el8.x86_64 145/165 2024-01-11T08:09:02.715 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : acl-2.2.53-3.el8.x86_64 133/165 2024-01-11T08:09:02.808 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-libs-0.189-1.el8.x86_64 146/165 2024-01-11T08:09:02.988 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libacl-devel-2.2.53-3.el8.x86_64 134/165 2024-01-11T08:09:03.060 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : openssl-1:1.1.1k-9.el8.x86_64 147/165 2024-01-11T08:09:03.223 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : xfsdump-3.1.8-7.el8.x86_64 135/165 2024-01-11T08:09:03.238 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 148/165 2024-01-11T08:09:03.238 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 149/165 2024-01-11T08:09:03.452 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-top-2:17.2.7-299.g4bcedb9f.el8.noarch 136/165 2024-01-11T08:09:03.538 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 149/165 2024-01-11T08:09:03.538 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 150/165 2024-01-11T08:09:03.695 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-fuse-2:17.2.7-299.g4bcedb9f.el8.x86_64 137/165 2024-01-11T08:09:03.812 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 150/165 2024-01-11T08:09:03.959 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libacl-devel-2.2.53-1.el8.x86_64 151/165 2024-01-11T08:09:03.977 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-4.el 152/165 2024-01-11T08:09:04.072 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-nbd-2:17.2.7-299.g4bcedb9f.el8.x86_64 138/165 2024-01-11T08:09:04.089 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-initscripts-updown-1:1.40.16-4.el 152/165 2024-01-11T08:09:04.089 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 153/165 2024-01-11T08:09:04.395 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 153/165 2024-01-11T08:09:04.508 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : iproute-6.2.0-5.el8.x86_64 139/165 2024-01-11T08:09:04.548 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : acl-2.2.53-1.el8.x86_64 154/165 2024-01-11T08:09:04.735 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : iproute-6.2.0-1.el8.x86_64 155/165 2024-01-11T08:09:04.819 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : openssl-1:1.1.1k-11.el8.x86_64 140/165 2024-01-11T08:09:04.961 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : util-linux-user-2.32.1-41.el8.x86_64 156/165 2024-01-11T08:09:04.975 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : NetworkManager-initscripts-updown-1:1.40.16-14.e 141/165 2024-01-11T08:09:05.004 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-14.e 141/165 2024-01-11T08:09:05.184 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : util-linux-2.32.1-41.el8.x86_64 157/165 2024-01-11T08:09:05.185 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libfdisk-2.32.1-41.el8.x86_64 158/165 2024-01-11T08:09:05.223 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 142/165 2024-01-11T08:09:05.453 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libfdisk-2.32.1-41.el8.x86_64 158/165 2024-01-11T08:09:05.453 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libmount-2.32.1-41.el8.x86_64 159/165 2024-01-11T08:09:05.525 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flex-2.6.1-9.el8.x86_64 143/165 2024-01-11T08:09:05.648 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 143/165 2024-01-11T08:09:05.711 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libmount-2.32.1-41.el8.x86_64 159/165 2024-01-11T08:09:05.711 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libblkid-2.32.1-41.el8.x86_64 160/165 2024-01-11T08:09:05.892 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : bison-3.0.4-10.el8.x86_64 144/165 2024-01-11T08:09:05.921 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 144/165 2024-01-11T08:09:05.936 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libblkid-2.32.1-41.el8.x86_64 160/165 2024-01-11T08:09:05.936 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libuuid-2.32.1-41.el8.x86_64 161/165 2024-01-11T08:09:06.127 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : elfutils-debuginfod-client-0.189-1.el8.x86_64 145/165 2024-01-11T08:09:06.211 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libuuid-2.32.1-41.el8.x86_64 161/165 2024-01-11T08:09:06.211 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libsmartcols-2.32.1-41.el8.x86_64 162/165 2024-01-11T08:09:06.379 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : elfutils-libs-0.189-1.el8.x86_64 146/165 2024-01-11T08:09:06.419 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libsmartcols-2.32.1-41.el8.x86_64 162/165 2024-01-11T08:09:06.602 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : openssl-1:1.1.1k-9.el8.x86_64 147/165 2024-01-11T08:09:06.625 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-libelf-0.189-1.el8.x86_64 163/165 2024-01-11T08:09:06.760 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libacl-2.2.53-1.el8.x86_64 164/165 2024-01-11T08:09:06.760 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : openssl-libs-1:1.1.1k-9.el8.x86_64 165/165 2024-01-11T08:09:06.789 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 148/165 2024-01-11T08:09:06.789 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 149/165 2024-01-11T08:09:07.063 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 149/165 2024-01-11T08:09:07.064 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 150/165 2024-01-11T08:09:07.320 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openssl-libs-1:1.1.1k-9.el8.x86_64 165/165 2024-01-11T08:09:07.321 INFO:teuthology.orchestra.run.smithi077.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-01-11T08:09:07.321 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:07.363 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 150/165 2024-01-11T08:09:07.510 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libacl-devel-2.2.53-1.el8.x86_64 151/165 2024-01-11T08:09:07.529 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-4.el 152/165 2024-01-11T08:09:07.666 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : NetworkManager-initscripts-updown-1:1.40.16-4.el 152/165 2024-01-11T08:09:07.666 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 153/165 2024-01-11T08:09:07.921 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 153/165 2024-01-11T08:09:08.030 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-2:17.2.7-299.g4bcedb9f.el8.x86_64 1/165 2024-01-11T08:09:08.030 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-base-2:17.2.7-299.g4bcedb9f.el8.x86_64 2/165 2024-01-11T08:09:08.030 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-common-2:17.2.7-299.g4bcedb9f.el8.x86_64 3/165 2024-01-11T08:09:08.030 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-fuse-2:17.2.7-299.g4bcedb9f.el8.x86_64 4/165 2024-01-11T08:09:08.030 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-299.g4bcedb 5/165 2024-01-11T08:09:08.030 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mds-2:17.2.7-299.g4bcedb9f.el8.x86_64 6/165 2024-01-11T08:09:08.030 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-2:17.2.7-299.g4bcedb9f.el8.x86_64 7/165 2024-01-11T08:09:08.030 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mon-2:17.2.7-299.g4bcedb9f.el8.x86_64 8/165 2024-01-11T08:09:08.030 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-osd-2:17.2.7-299.g4bcedb9f.el8.x86_64 9/165 2024-01-11T08:09:08.030 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-radosgw-2:17.2.7-299.g4bcedb9f.el8.x86_64 10/165 2024-01-11T08:09:08.031 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-selinux-2:17.2.7-299.g4bcedb9f.el8.x86_64 11/165 2024-01-11T08:09:08.031 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-test-2:17.2.7-299.g4bcedb9f.el8.x86_64 12/165 2024-01-11T08:09:08.031 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-mirror-2:17.2.7-299.g4bcedb9f.el8.x86_64 13/165 2024-01-11T08:09:08.031 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs-devel-2:17.2.7-299.g4bcedb9f.el8.x86_6 14/165 2024-01-11T08:09:08.031 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs2-2:17.2.7-299.g4bcedb9f.el8.x86_64 15/165 2024-01-11T08:09:08.031 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephsqlite-2:17.2.7-299.g4bcedb9f.el8.x86_64 16/165 2024-01-11T08:09:08.031 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados-devel-2:17.2.7-299.g4bcedb9f.el8.x86_64 17/165 2024-01-11T08:09:08.031 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libradosstriper1-2:17.2.7-299.g4bcedb9f.el8.x86_ 18/165 2024-01-11T08:09:08.031 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librgw2-2:17.2.7-299.g4bcedb9f.el8.x86_64 19/165 2024-01-11T08:09:08.031 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-argparse-2:17.2.7-299.g4bcedb9f.el8 20/165 2024-01-11T08:09:08.031 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-common-2:17.2.7-299.g4bcedb9f.el8.x 21/165 2024-01-11T08:09:08.031 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cephfs-2:17.2.7-299.g4bcedb9f.el8.x86_64 22/165 2024-01-11T08:09:08.031 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rados-2:17.2.7-299.g4bcedb9f.el8.x86_64 23/165 2024-01-11T08:09:08.031 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rbd-2:17.2.7-299.g4bcedb9f.el8.x86_64 24/165 2024-01-11T08:09:08.031 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rgw-2:17.2.7-299.g4bcedb9f.el8.x86_64 25/165 2024-01-11T08:09:08.032 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-fuse-2:17.2.7-299.g4bcedb9f.el8.x86_64 26/165 2024-01-11T08:09:08.032 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-mirror-2:17.2.7-299.g4bcedb9f.el8.x86_64 27/165 2024-01-11T08:09:08.032 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-nbd-2:17.2.7-299.g4bcedb9f.el8.x86_64 28/165 2024-01-11T08:09:08.032 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-299.g4bcedb9f.e 29/165 2024-01-11T08:09:08.044 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-299.g4bcedb9f.el8.noar 30/165 2024-01-11T08:09:08.044 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-299.g4bcedb9f.el8.no 31/165 2024-01-11T08:09:08.044 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-299.g4bce 32/165 2024-01-11T08:09:08.045 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-299.g4bcedb9f.el8 33/165 2024-01-11T08:09:08.045 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-rook-2:17.2.7-299.g4bcedb9f.el8.noarch 34/165 2024-01-11T08:09:08.045 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-299.g4bcedb9f.el 35/165 2024-01-11T08:09:08.045 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-volume-2:17.2.7-299.g4bcedb9f.el8.noarch 36/165 2024-01-11T08:09:08.045 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephadm-2:17.2.7-299.g4bcedb9f.el8.noarch 37/165 2024-01-11T08:09:08.045 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-top-2:17.2.7-299.g4bcedb9f.el8.noarch 38/165 2024-01-11T08:09:08.045 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : automake-1.16.1-8.el8.noarch 39/165 2024-01-11T08:09:08.045 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : bison-3.0.4-10.el8.x86_64 40/165 2024-01-11T08:09:08.045 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flex-2.6.1-9.el8.x86_64 41/165 2024-01-11T08:09:08.045 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : jq-1.6-8.el8.x86_64 42/165 2024-01-11T08:09:08.045 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 43/165 2024-01-11T08:09:08.045 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 44/165 2024-01-11T08:09:08.045 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 45/165 2024-01-11T08:09:08.046 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 46/165 2024-01-11T08:09:08.046 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 47/165 2024-01-11T08:09:08.046 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 48/165 2024-01-11T08:09:08.046 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 49/165 2024-01-11T08:09:08.046 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 50/165 2024-01-11T08:09:08.046 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 51/165 2024-01-11T08:09:08.046 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 52/165 2024-01-11T08:09:08.046 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 53/165 2024-01-11T08:09:08.046 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 54/165 2024-01-11T08:09:08.046 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 55/165 2024-01-11T08:09:08.046 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 56/165 2024-01-11T08:09:08.046 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 57/165 2024-01-11T08:09:08.046 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 58/165 2024-01-11T08:09:08.046 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 59/165 2024-01-11T08:09:08.046 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 60/165 2024-01-11T08:09:08.047 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 61/165 2024-01-11T08:09:08.047 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 62/165 2024-01-11T08:09:08.047 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 63/165 2024-01-11T08:09:08.047 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 64/165 2024-01-11T08:09:08.047 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 65/165 2024-01-11T08:09:08.047 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-devel-0.190-2.el8.x86_64 66/165 2024-01-11T08:09:08.047 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 67/165 2024-01-11T08:09:08.047 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 68/165 2024-01-11T08:09:08.047 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : krb5-devel-1.18.2-22.el8.x86_64 69/165 2024-01-11T08:09:08.047 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 70/165 2024-01-11T08:09:08.047 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcom_err-devel-1.45.6-5.el8.x86_64 71/165 2024-01-11T08:09:08.047 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 72/165 2024-01-11T08:09:08.047 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 73/165 2024-01-11T08:09:08.047 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 74/165 2024-01-11T08:09:08.047 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libselinux-devel-2.9-8.el8.x86_64 75/165 2024-01-11T08:09:08.048 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 76/165 2024-01-11T08:09:08.048 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 77/165 2024-01-11T08:09:08.048 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libverto-devel-0.3.2-2.el8.x86_64 78/165 2024-01-11T08:09:08.048 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 79/165 2024-01-11T08:09:08.048 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-devel-1:1.1.1k-11.el8.x86_64 80/165 2024-01-11T08:09:08.048 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : pcre2-devel-10.32-3.el8.x86_64 81/165 2024-01-11T08:09:08.048 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : pcre2-utf16-10.32-3.el8.x86_64 82/165 2024-01-11T08:09:08.048 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : pcre2-utf32-10.32-3.el8.x86_64 83/165 2024-01-11T08:09:08.048 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 84/165 2024-01-11T08:09:08.048 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 85/165 2024-01-11T08:09:08.048 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 86/165 2024-01-11T08:09:08.048 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 87/165 2024-01-11T08:09:08.048 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 88/165 2024-01-11T08:09:08.049 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 89/165 2024-01-11T08:09:08.049 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 90/165 2024-01-11T08:09:08.049 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : zlib-devel-1.2.11-21.el8.x86_64 91/165 2024-01-11T08:09:08.049 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 92/165 2024-01-11T08:09:08.049 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 93/165 2024-01-11T08:09:08.049 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 94/165 2024-01-11T08:09:08.049 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 95/165 2024-01-11T08:09:08.049 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 96/165 2024-01-11T08:09:08.049 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 97/165 2024-01-11T08:09:08.049 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 98/165 2024-01-11T08:09:08.049 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 99/165 2024-01-11T08:09:08.049 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 100/165 2024-01-11T08:09:08.049 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 101/165 2024-01-11T08:09:08.049 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 102/165 2024-01-11T08:09:08.049 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 103/165 2024-01-11T08:09:08.050 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 104/165 2024-01-11T08:09:08.050 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 105/165 2024-01-11T08:09:08.050 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 106/165 2024-01-11T08:09:08.050 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 107/165 2024-01-11T08:09:08.050 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 108/165 2024-01-11T08:09:08.050 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 109/165 2024-01-11T08:09:08.050 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-portend-2.6-1.el8.noarch 110/165 2024-01-11T08:09:08.050 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 111/165 2024-01-11T08:09:08.050 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 112/165 2024-01-11T08:09:08.050 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 113/165 2024-01-11T08:09:08.050 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 114/165 2024-01-11T08:09:08.050 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 115/165 2024-01-11T08:09:08.050 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 116/165 2024-01-11T08:09:08.050 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 117/165 2024-01-11T08:09:08.050 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 118/165 2024-01-11T08:09:08.051 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 119/165 2024-01-11T08:09:08.051 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 120/165 2024-01-11T08:09:08.051 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 121/165 2024-01-11T08:09:08.051 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 122/165 2024-01-11T08:09:08.051 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 123/165 2024-01-11T08:09:08.051 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:17.2.7-299.g4bcedb9f.el8.x86_64 124/165 2024-01-11T08:09:08.051 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 125/165 2024-01-11T08:09:08.051 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:17.2.7-299.g4bcedb9f.el8.x86_64 126/165 2024-01-11T08:09:08.051 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 127/165 2024-01-11T08:09:08.051 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-14.e 128/165 2024-01-11T08:09:08.051 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-4.el 129/165 2024-01-11T08:09:08.051 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : acl-2.2.53-3.el8.x86_64 130/165 2024-01-11T08:09:08.051 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : acl-2.2.53-1.el8.x86_64 131/165 2024-01-11T08:09:08.051 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el8.x86_64 132/165 2024-01-11T08:09:08.051 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-debuginfod-client-0.189-1.el8.x86_64 133/165 2024-01-11T08:09:08.052 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-0.190-2.el8.x86_64 134/165 2024-01-11T08:09:08.052 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-0.189-1.el8.x86_64 135/165 2024-01-11T08:09:08.052 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libs-0.190-2.el8.x86_64 136/165 2024-01-11T08:09:08.052 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libs-0.189-1.el8.x86_64 137/165 2024-01-11T08:09:08.052 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : iproute-6.2.0-5.el8.x86_64 138/165 2024-01-11T08:09:08.052 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : iproute-6.2.0-1.el8.x86_64 139/165 2024-01-11T08:09:08.052 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-2.2.53-3.el8.x86_64 140/165 2024-01-11T08:09:08.052 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-2.2.53-1.el8.x86_64 141/165 2024-01-11T08:09:08.052 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-devel-2.2.53-3.el8.x86_64 142/165 2024-01-11T08:09:08.052 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-devel-2.2.53-1.el8.x86_64 143/165 2024-01-11T08:09:08.052 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 144/165 2024-01-11T08:09:08.052 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libblkid-2.32.1-41.el8.x86_64 145/165 2024-01-11T08:09:08.052 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 146/165 2024-01-11T08:09:08.052 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libfdisk-2.32.1-41.el8.x86_64 147/165 2024-01-11T08:09:08.053 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 148/165 2024-01-11T08:09:08.053 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libmount-2.32.1-41.el8.x86_64 149/165 2024-01-11T08:09:08.053 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 150/165 2024-01-11T08:09:08.053 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libsmartcols-2.32.1-41.el8.x86_64 151/165 2024-01-11T08:09:08.053 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 152/165 2024-01-11T08:09:08.053 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-2.32.1-41.el8.x86_64 153/165 2024-01-11T08:09:08.053 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-1:1.1.1k-11.el8.x86_64 154/165 2024-01-11T08:09:08.053 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-1:1.1.1k-9.el8.x86_64 155/165 2024-01-11T08:09:08.053 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-libs-1:1.1.1k-11.el8.x86_64 156/165 2024-01-11T08:09:08.053 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-libs-1:1.1.1k-9.el8.x86_64 157/165 2024-01-11T08:09:08.053 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 158/165 2024-01-11T08:09:08.053 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-2.32.1-41.el8.x86_64 159/165 2024-01-11T08:09:08.053 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 160/165 2024-01-11T08:09:08.053 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-user-2.32.1-41.el8.x86_64 161/165 2024-01-11T08:09:08.053 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsdump-3.1.8-7.el8.x86_64 162/165 2024-01-11T08:09:08.054 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 163/165 2024-01-11T08:09:08.054 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 164/165 2024-01-11T08:09:08.149 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : acl-2.2.53-1.el8.x86_64 154/165 2024-01-11T08:09:08.344 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : iproute-6.2.0-1.el8.x86_64 155/165 2024-01-11T08:09:08.517 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : util-linux-user-2.32.1-41.el8.x86_64 156/165 2024-01-11T08:09:08.760 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : util-linux-2.32.1-41.el8.x86_64 157/165 2024-01-11T08:09:08.760 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libfdisk-2.32.1-41.el8.x86_64 158/165 2024-01-11T08:09:08.988 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libfdisk-2.32.1-41.el8.x86_64 158/165 2024-01-11T08:09:08.989 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libmount-2.32.1-41.el8.x86_64 159/165 2024-01-11T08:09:09.220 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libmount-2.32.1-41.el8.x86_64 159/165 2024-01-11T08:09:09.220 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libblkid-2.32.1-41.el8.x86_64 160/165 2024-01-11T08:09:09.437 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libblkid-2.32.1-41.el8.x86_64 160/165 2024-01-11T08:09:09.437 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libuuid-2.32.1-41.el8.x86_64 161/165 2024-01-11T08:09:09.812 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libuuid-2.32.1-41.el8.x86_64 161/165 2024-01-11T08:09:09.812 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libsmartcols-2.32.1-41.el8.x86_64 162/165 2024-01-11T08:09:10.045 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libsmartcols-2.32.1-41.el8.x86_64 162/165 2024-01-11T08:09:10.243 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : elfutils-libelf-0.189-1.el8.x86_64 163/165 2024-01-11T08:09:10.386 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libacl-2.2.53-1.el8.x86_64 164/165 2024-01-11T08:09:10.386 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : openssl-libs-1:1.1.1k-9.el8.x86_64 165/165 2024-01-11T08:09:10.403 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 165/165 2024-01-11T08:09:10.403 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:10.403 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2024-01-11T08:09:10.403 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-initscripts-updown-1:1.40.16-14.el8.noarch 2024-01-11T08:09:10.403 INFO:teuthology.orchestra.run.smithi077.stdout: acl-2.2.53-3.el8.x86_64 2024-01-11T08:09:10.403 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-debuginfod-client-0.190-2.el8.x86_64 2024-01-11T08:09:10.404 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-0.190-2.el8.x86_64 2024-01-11T08:09:10.404 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libs-0.190-2.el8.x86_64 2024-01-11T08:09:10.404 INFO:teuthology.orchestra.run.smithi077.stdout: iproute-6.2.0-5.el8.x86_64 2024-01-11T08:09:10.404 INFO:teuthology.orchestra.run.smithi077.stdout: libacl-2.2.53-3.el8.x86_64 2024-01-11T08:09:10.404 INFO:teuthology.orchestra.run.smithi077.stdout: libacl-devel-2.2.53-3.el8.x86_64 2024-01-11T08:09:10.404 INFO:teuthology.orchestra.run.smithi077.stdout: libblkid-2.32.1-43.el8.x86_64 2024-01-11T08:09:10.404 INFO:teuthology.orchestra.run.smithi077.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-01-11T08:09:10.404 INFO:teuthology.orchestra.run.smithi077.stdout: libmount-2.32.1-43.el8.x86_64 2024-01-11T08:09:10.404 INFO:teuthology.orchestra.run.smithi077.stdout: librados2-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.404 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.404 INFO:teuthology.orchestra.run.smithi077.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-01-11T08:09:10.404 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-2.32.1-43.el8.x86_64 2024-01-11T08:09:10.404 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-1:1.1.1k-11.el8.x86_64 2024-01-11T08:09:10.404 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-libs-1:1.1.1k-11.el8.x86_64 2024-01-11T08:09:10.405 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-2.32.1-43.el8.x86_64 2024-01-11T08:09:10.405 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-01-11T08:09:10.405 INFO:teuthology.orchestra.run.smithi077.stdout: xfsdump-3.1.8-7.el8.x86_64 2024-01-11T08:09:10.405 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-01-11T08:09:10.405 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-01-11T08:09:10.405 INFO:teuthology.orchestra.run.smithi077.stdout: automake-1.16.1-8.el8.noarch 2024-01-11T08:09:10.405 INFO:teuthology.orchestra.run.smithi077.stdout: bison-3.0.4-10.el8.x86_64 2024-01-11T08:09:10.405 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.405 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.405 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.405 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.405 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:10.405 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.406 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.406 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.406 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:10.406 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:10.406 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:10.406 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:10.406 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:10.406 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.406 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.406 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:10.406 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.406 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.406 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.406 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:10.407 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:10.407 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.407 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:10.407 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel-0.190-2.el8.x86_64 2024-01-11T08:09:10.407 INFO:teuthology.orchestra.run.smithi077.stdout: flex-2.6.1-9.el8.x86_64 2024-01-11T08:09:10.407 INFO:teuthology.orchestra.run.smithi077.stdout: fuse-2.9.7-17.el8.x86_64 2024-01-11T08:09:10.407 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-11T08:09:10.407 INFO:teuthology.orchestra.run.smithi077.stdout: jq-1.6-8.el8.x86_64 2024-01-11T08:09:10.407 INFO:teuthology.orchestra.run.smithi077.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-01-11T08:09:10.407 INFO:teuthology.orchestra.run.smithi077.stdout: krb5-devel-1.18.2-22.el8.x86_64 2024-01-11T08:09:10.407 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-01-11T08:09:10.407 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.407 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.407 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.407 INFO:teuthology.orchestra.run.smithi077.stdout: libcom_err-devel-1.45.6-5.el8.x86_64 2024-01-11T08:09:10.408 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-11T08:09:10.408 INFO:teuthology.orchestra.run.smithi077.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-11T08:09:10.408 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-11T08:09:10.408 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-11T08:09:10.408 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.408 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.408 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-11T08:09:10.408 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.408 INFO:teuthology.orchestra.run.smithi077.stdout: libselinux-devel-2.9-8.el8.x86_64 2024-01-11T08:09:10.408 INFO:teuthology.orchestra.run.smithi077.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-01-11T08:09:10.408 INFO:teuthology.orchestra.run.smithi077.stdout: libtool-2.4.6-25.el8.x86_64 2024-01-11T08:09:10.408 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-11T08:09:10.408 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-01-11T08:09:10.409 INFO:teuthology.orchestra.run.smithi077.stdout: libverto-devel-0.3.2-2.el8.x86_64 2024-01-11T08:09:10.409 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-01-11T08:09:10.409 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-11T08:09:10.409 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-11T08:09:10.409 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-11T08:09:10.409 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-11T08:09:10.409 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel-1:1.1.1k-11.el8.x86_64 2024-01-11T08:09:10.409 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-devel-10.32-3.el8.x86_64 2024-01-11T08:09:10.409 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf16-10.32-3.el8.x86_64 2024-01-11T08:09:10.409 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf32-10.32-3.el8.x86_64 2024-01-11T08:09:10.409 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-01-11T08:09:10.409 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-11T08:09:10.409 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-11T08:09:10.409 INFO:teuthology.orchestra.run.smithi077.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-11T08:09:10.409 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-11T08:09:10.410 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.410 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.410 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.410 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-11T08:09:10.410 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-11T08:09:10.410 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-11T08:09:10.410 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-11T08:09:10.410 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-11T08:09:10.410 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-11T08:09:10.410 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-11T08:09:10.410 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-11T08:09:10.410 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-11T08:09:10.410 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-11T08:09:10.410 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-11T08:09:10.411 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-11T08:09:10.411 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-11T08:09:10.411 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-11T08:09:10.411 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-11T08:09:10.411 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-11T08:09:10.411 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-11T08:09:10.411 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-01-11T08:09:10.411 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-11T08:09:10.411 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-11T08:09:10.411 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-11T08:09:10.411 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-11T08:09:10.411 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend-2.6-1.el8.noarch 2024-01-11T08:09:10.411 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-11T08:09:10.411 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-11T08:09:10.411 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-11T08:09:10.412 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-11T08:09:10.412 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-11T08:09:10.412 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-11T08:09:10.412 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.412 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.412 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-11T08:09:10.412 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-11T08:09:10.412 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.412 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-11T08:09:10.412 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-11T08:09:10.412 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-11T08:09:10.412 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-11T08:09:10.412 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-11T08:09:10.412 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-11T08:09:10.413 INFO:teuthology.orchestra.run.smithi077.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-11T08:09:10.413 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-11T08:09:10.413 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-11T08:09:10.413 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-11T08:09:10.413 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-11T08:09:10.413 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-11T08:09:10.413 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-11T08:09:10.413 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.413 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.413 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:10.413 INFO:teuthology.orchestra.run.smithi077.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-11T08:09:10.413 INFO:teuthology.orchestra.run.smithi077.stdout: thrift-0.13.0-2.el8.x86_64 2024-01-11T08:09:10.413 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-01-11T08:09:10.413 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-11T08:09:10.414 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel-1.2.11-21.el8.x86_64 2024-01-11T08:09:10.414 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:10.414 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-01-11T08:09:10.717 DEBUG:teuthology.parallel:result is None 2024-01-11T08:09:10.879 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openssl-libs-1:1.1.1k-9.el8.x86_64 165/165 2024-01-11T08:09:10.879 INFO:teuthology.orchestra.run.smithi027.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-01-11T08:09:10.879 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:11.656 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:17.2.7-299.g4bcedb9f.el8.x86_64 1/165 2024-01-11T08:09:11.656 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:17.2.7-299.g4bcedb9f.el8.x86_64 2/165 2024-01-11T08:09:11.656 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:17.2.7-299.g4bcedb9f.el8.x86_64 3/165 2024-01-11T08:09:11.656 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:17.2.7-299.g4bcedb9f.el8.x86_64 4/165 2024-01-11T08:09:11.656 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-299.g4bcedb 5/165 2024-01-11T08:09:11.656 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:17.2.7-299.g4bcedb9f.el8.x86_64 6/165 2024-01-11T08:09:11.657 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:17.2.7-299.g4bcedb9f.el8.x86_64 7/165 2024-01-11T08:09:11.657 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:17.2.7-299.g4bcedb9f.el8.x86_64 8/165 2024-01-11T08:09:11.657 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:17.2.7-299.g4bcedb9f.el8.x86_64 9/165 2024-01-11T08:09:11.657 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:17.2.7-299.g4bcedb9f.el8.x86_64 10/165 2024-01-11T08:09:11.657 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:17.2.7-299.g4bcedb9f.el8.x86_64 11/165 2024-01-11T08:09:11.657 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:17.2.7-299.g4bcedb9f.el8.x86_64 12/165 2024-01-11T08:09:11.657 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-mirror-2:17.2.7-299.g4bcedb9f.el8.x86_64 13/165 2024-01-11T08:09:11.657 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:17.2.7-299.g4bcedb9f.el8.x86_6 14/165 2024-01-11T08:09:11.657 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:17.2.7-299.g4bcedb9f.el8.x86_64 15/165 2024-01-11T08:09:11.657 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:17.2.7-299.g4bcedb9f.el8.x86_64 16/165 2024-01-11T08:09:11.657 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:17.2.7-299.g4bcedb9f.el8.x86_64 17/165 2024-01-11T08:09:11.657 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:17.2.7-299.g4bcedb9f.el8.x86_ 18/165 2024-01-11T08:09:11.657 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:17.2.7-299.g4bcedb9f.el8.x86_64 19/165 2024-01-11T08:09:11.657 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:17.2.7-299.g4bcedb9f.el8 20/165 2024-01-11T08:09:11.658 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:17.2.7-299.g4bcedb9f.el8.x 21/165 2024-01-11T08:09:11.658 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:17.2.7-299.g4bcedb9f.el8.x86_64 22/165 2024-01-11T08:09:11.658 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:17.2.7-299.g4bcedb9f.el8.x86_64 23/165 2024-01-11T08:09:11.658 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:17.2.7-299.g4bcedb9f.el8.x86_64 24/165 2024-01-11T08:09:11.658 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:17.2.7-299.g4bcedb9f.el8.x86_64 25/165 2024-01-11T08:09:11.658 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:17.2.7-299.g4bcedb9f.el8.x86_64 26/165 2024-01-11T08:09:11.658 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:17.2.7-299.g4bcedb9f.el8.x86_64 27/165 2024-01-11T08:09:11.658 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:17.2.7-299.g4bcedb9f.el8.x86_64 28/165 2024-01-11T08:09:11.658 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-299.g4bcedb9f.e 29/165 2024-01-11T08:09:11.658 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-299.g4bcedb9f.el8.noar 30/165 2024-01-11T08:09:11.658 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-299.g4bcedb9f.el8.no 31/165 2024-01-11T08:09:11.658 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-299.g4bce 32/165 2024-01-11T08:09:11.658 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-299.g4bcedb9f.el8 33/165 2024-01-11T08:09:11.658 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-rook-2:17.2.7-299.g4bcedb9f.el8.noarch 34/165 2024-01-11T08:09:11.659 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-299.g4bcedb9f.el 35/165 2024-01-11T08:09:11.659 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-volume-2:17.2.7-299.g4bcedb9f.el8.noarch 36/165 2024-01-11T08:09:11.659 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephadm-2:17.2.7-299.g4bcedb9f.el8.noarch 37/165 2024-01-11T08:09:11.659 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-top-2:17.2.7-299.g4bcedb9f.el8.noarch 38/165 2024-01-11T08:09:11.659 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : automake-1.16.1-8.el8.noarch 39/165 2024-01-11T08:09:11.659 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : bison-3.0.4-10.el8.x86_64 40/165 2024-01-11T08:09:11.659 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flex-2.6.1-9.el8.x86_64 41/165 2024-01-11T08:09:11.659 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : jq-1.6-8.el8.x86_64 42/165 2024-01-11T08:09:11.659 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 43/165 2024-01-11T08:09:11.659 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 44/165 2024-01-11T08:09:11.659 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 45/165 2024-01-11T08:09:11.659 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 46/165 2024-01-11T08:09:11.659 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 47/165 2024-01-11T08:09:11.659 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 48/165 2024-01-11T08:09:11.659 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 49/165 2024-01-11T08:09:11.660 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 50/165 2024-01-11T08:09:11.660 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 51/165 2024-01-11T08:09:11.660 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 52/165 2024-01-11T08:09:11.660 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 53/165 2024-01-11T08:09:11.660 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 54/165 2024-01-11T08:09:11.660 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 55/165 2024-01-11T08:09:11.660 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 56/165 2024-01-11T08:09:11.660 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 57/165 2024-01-11T08:09:11.660 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 58/165 2024-01-11T08:09:11.660 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 59/165 2024-01-11T08:09:11.660 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 60/165 2024-01-11T08:09:11.660 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 61/165 2024-01-11T08:09:11.660 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 62/165 2024-01-11T08:09:11.660 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 63/165 2024-01-11T08:09:11.661 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 64/165 2024-01-11T08:09:11.661 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 65/165 2024-01-11T08:09:11.661 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libelf-devel-0.190-2.el8.x86_64 66/165 2024-01-11T08:09:11.661 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 67/165 2024-01-11T08:09:11.661 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 68/165 2024-01-11T08:09:11.661 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : krb5-devel-1.18.2-22.el8.x86_64 69/165 2024-01-11T08:09:11.661 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 70/165 2024-01-11T08:09:11.661 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcom_err-devel-1.45.6-5.el8.x86_64 71/165 2024-01-11T08:09:11.661 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 72/165 2024-01-11T08:09:11.661 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 73/165 2024-01-11T08:09:11.661 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 74/165 2024-01-11T08:09:11.661 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libselinux-devel-2.9-8.el8.x86_64 75/165 2024-01-11T08:09:11.661 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 76/165 2024-01-11T08:09:11.661 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 77/165 2024-01-11T08:09:11.661 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libverto-devel-0.3.2-2.el8.x86_64 78/165 2024-01-11T08:09:11.662 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 79/165 2024-01-11T08:09:11.662 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-devel-1:1.1.1k-11.el8.x86_64 80/165 2024-01-11T08:09:11.662 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : pcre2-devel-10.32-3.el8.x86_64 81/165 2024-01-11T08:09:11.662 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : pcre2-utf16-10.32-3.el8.x86_64 82/165 2024-01-11T08:09:11.662 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : pcre2-utf32-10.32-3.el8.x86_64 83/165 2024-01-11T08:09:11.662 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 84/165 2024-01-11T08:09:11.662 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 85/165 2024-01-11T08:09:11.662 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 86/165 2024-01-11T08:09:11.662 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 87/165 2024-01-11T08:09:11.662 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 88/165 2024-01-11T08:09:11.662 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 89/165 2024-01-11T08:09:11.662 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 90/165 2024-01-11T08:09:11.662 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : zlib-devel-1.2.11-21.el8.x86_64 91/165 2024-01-11T08:09:11.662 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 92/165 2024-01-11T08:09:11.662 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 93/165 2024-01-11T08:09:11.663 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 94/165 2024-01-11T08:09:11.663 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 95/165 2024-01-11T08:09:11.663 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 96/165 2024-01-11T08:09:11.663 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 97/165 2024-01-11T08:09:11.663 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 98/165 2024-01-11T08:09:11.663 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 99/165 2024-01-11T08:09:11.663 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 100/165 2024-01-11T08:09:11.663 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 101/165 2024-01-11T08:09:11.663 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 102/165 2024-01-11T08:09:11.663 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 103/165 2024-01-11T08:09:11.663 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 104/165 2024-01-11T08:09:11.663 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 105/165 2024-01-11T08:09:11.663 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 106/165 2024-01-11T08:09:11.663 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 107/165 2024-01-11T08:09:11.664 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 108/165 2024-01-11T08:09:11.664 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 109/165 2024-01-11T08:09:11.664 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-portend-2.6-1.el8.noarch 110/165 2024-01-11T08:09:11.664 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 111/165 2024-01-11T08:09:11.664 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 112/165 2024-01-11T08:09:11.664 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 113/165 2024-01-11T08:09:11.664 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 114/165 2024-01-11T08:09:11.664 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 115/165 2024-01-11T08:09:11.664 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 116/165 2024-01-11T08:09:11.664 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 117/165 2024-01-11T08:09:11.665 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 118/165 2024-01-11T08:09:11.665 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 119/165 2024-01-11T08:09:11.665 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 120/165 2024-01-11T08:09:11.665 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 121/165 2024-01-11T08:09:11.665 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 122/165 2024-01-11T08:09:11.665 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 123/165 2024-01-11T08:09:11.665 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:17.2.7-299.g4bcedb9f.el8.x86_64 124/165 2024-01-11T08:09:11.665 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 125/165 2024-01-11T08:09:11.665 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:17.2.7-299.g4bcedb9f.el8.x86_64 126/165 2024-01-11T08:09:11.665 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 127/165 2024-01-11T08:09:11.665 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-14.e 128/165 2024-01-11T08:09:11.665 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-4.el 129/165 2024-01-11T08:09:11.665 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : acl-2.2.53-3.el8.x86_64 130/165 2024-01-11T08:09:11.665 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : acl-2.2.53-1.el8.x86_64 131/165 2024-01-11T08:09:11.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el8.x86_64 132/165 2024-01-11T08:09:11.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-debuginfod-client-0.189-1.el8.x86_64 133/165 2024-01-11T08:09:11.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libelf-0.190-2.el8.x86_64 134/165 2024-01-11T08:09:11.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libelf-0.189-1.el8.x86_64 135/165 2024-01-11T08:09:11.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libs-0.190-2.el8.x86_64 136/165 2024-01-11T08:09:11.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libs-0.189-1.el8.x86_64 137/165 2024-01-11T08:09:11.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : iproute-6.2.0-5.el8.x86_64 138/165 2024-01-11T08:09:11.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : iproute-6.2.0-1.el8.x86_64 139/165 2024-01-11T08:09:11.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libacl-2.2.53-3.el8.x86_64 140/165 2024-01-11T08:09:11.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libacl-2.2.53-1.el8.x86_64 141/165 2024-01-11T08:09:11.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libacl-devel-2.2.53-3.el8.x86_64 142/165 2024-01-11T08:09:11.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libacl-devel-2.2.53-1.el8.x86_64 143/165 2024-01-11T08:09:11.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 144/165 2024-01-11T08:09:11.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libblkid-2.32.1-41.el8.x86_64 145/165 2024-01-11T08:09:11.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 146/165 2024-01-11T08:09:11.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libfdisk-2.32.1-41.el8.x86_64 147/165 2024-01-11T08:09:11.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 148/165 2024-01-11T08:09:11.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libmount-2.32.1-41.el8.x86_64 149/165 2024-01-11T08:09:11.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 150/165 2024-01-11T08:09:11.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libsmartcols-2.32.1-41.el8.x86_64 151/165 2024-01-11T08:09:11.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 152/165 2024-01-11T08:09:11.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libuuid-2.32.1-41.el8.x86_64 153/165 2024-01-11T08:09:11.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-1:1.1.1k-11.el8.x86_64 154/165 2024-01-11T08:09:11.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-1:1.1.1k-9.el8.x86_64 155/165 2024-01-11T08:09:11.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-libs-1:1.1.1k-11.el8.x86_64 156/165 2024-01-11T08:09:11.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-libs-1:1.1.1k-9.el8.x86_64 157/165 2024-01-11T08:09:11.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 158/165 2024-01-11T08:09:11.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-2.32.1-41.el8.x86_64 159/165 2024-01-11T08:09:11.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 160/165 2024-01-11T08:09:11.668 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-user-2.32.1-41.el8.x86_64 161/165 2024-01-11T08:09:11.668 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsdump-3.1.8-7.el8.x86_64 162/165 2024-01-11T08:09:11.668 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 163/165 2024-01-11T08:09:11.668 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 164/165 2024-01-11T08:09:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 165/165 2024-01-11T08:09:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:14.493 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2024-01-11T08:09:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-initscripts-updown-1:1.40.16-14.el8.noarch 2024-01-11T08:09:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: acl-2.2.53-3.el8.x86_64 2024-01-11T08:09:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-debuginfod-client-0.190-2.el8.x86_64 2024-01-11T08:09:14.493 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-0.190-2.el8.x86_64 2024-01-11T08:09:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libs-0.190-2.el8.x86_64 2024-01-11T08:09:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: iproute-6.2.0-5.el8.x86_64 2024-01-11T08:09:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: libacl-2.2.53-3.el8.x86_64 2024-01-11T08:09:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: libacl-devel-2.2.53-3.el8.x86_64 2024-01-11T08:09:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: libblkid-2.32.1-43.el8.x86_64 2024-01-11T08:09:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-01-11T08:09:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: libmount-2.32.1-43.el8.x86_64 2024-01-11T08:09:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-01-11T08:09:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-2.32.1-43.el8.x86_64 2024-01-11T08:09:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-1:1.1.1k-11.el8.x86_64 2024-01-11T08:09:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-libs-1:1.1.1k-11.el8.x86_64 2024-01-11T08:09:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux-2.32.1-43.el8.x86_64 2024-01-11T08:09:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-01-11T08:09:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: xfsdump-3.1.8-7.el8.x86_64 2024-01-11T08:09:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-01-11T08:09:14.495 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-01-11T08:09:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: automake-1.16.1-8.el8.noarch 2024-01-11T08:09:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: bison-3.0.4-10.el8.x86_64 2024-01-11T08:09:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.495 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.496 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-mirror-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-top-2:17.2.7-299.g4bcedb9f.el8.noarch 2024-01-11T08:09:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-devel-0.190-2.el8.x86_64 2024-01-11T08:09:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: flex-2.6.1-9.el8.x86_64 2024-01-11T08:09:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: fuse-2.9.7-17.el8.x86_64 2024-01-11T08:09:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-11T08:09:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: jq-1.6-8.el8.x86_64 2024-01-11T08:09:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-01-11T08:09:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: krb5-devel-1.18.2-22.el8.x86_64 2024-01-11T08:09:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-01-11T08:09:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.497 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: libcom_err-devel-1.45.6-5.el8.x86_64 2024-01-11T08:09:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-11T08:09:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-11T08:09:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-11T08:09:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-11T08:09:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-11T08:09:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: libselinux-devel-2.9-8.el8.x86_64 2024-01-11T08:09:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-01-11T08:09:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: libtool-2.4.6-25.el8.x86_64 2024-01-11T08:09:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-11T08:09:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-01-11T08:09:14.498 INFO:teuthology.orchestra.run.smithi027.stdout: libverto-devel-0.3.2-2.el8.x86_64 2024-01-11T08:09:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-01-11T08:09:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-11T08:09:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-11T08:09:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-11T08:09:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-11T08:09:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-devel-1:1.1.1k-11.el8.x86_64 2024-01-11T08:09:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: pcre2-devel-10.32-3.el8.x86_64 2024-01-11T08:09:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: pcre2-utf16-10.32-3.el8.x86_64 2024-01-11T08:09:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: pcre2-utf32-10.32-3.el8.x86_64 2024-01-11T08:09:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-01-11T08:09:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-11T08:09:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-11T08:09:14.499 INFO:teuthology.orchestra.run.smithi027.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-11T08:09:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-11T08:09:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-11T08:09:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-11T08:09:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-11T08:09:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-11T08:09:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-11T08:09:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-11T08:09:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-11T08:09:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-11T08:09:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-11T08:09:14.500 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-11T08:09:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-11T08:09:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-11T08:09:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-11T08:09:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-11T08:09:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-11T08:09:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-11T08:09:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-11T08:09:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-01-11T08:09:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-11T08:09:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-11T08:09:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-11T08:09:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-11T08:09:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend-2.6-1.el8.noarch 2024-01-11T08:09:14.501 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-11T08:09:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-11T08:09:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-11T08:09:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-11T08:09:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-11T08:09:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-11T08:09:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-11T08:09:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-11T08:09:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-11T08:09:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-11T08:09:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-11T08:09:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-11T08:09:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-11T08:09:14.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-11T08:09:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-11T08:09:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-11T08:09:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-11T08:09:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-11T08:09:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-11T08:09:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-11T08:09:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-11T08:09:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:17.2.7-299.g4bcedb9f.el8.x86_64 2024-01-11T08:09:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-11T08:09:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: thrift-0.13.0-2.el8.x86_64 2024-01-11T08:09:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-01-11T08:09:14.503 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-11T08:09:14.504 INFO:teuthology.orchestra.run.smithi027.stdout: zlib-devel-1.2.11-21.el8.x86_64 2024-01-11T08:09:14.504 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:14.504 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-01-11T08:09:14.969 DEBUG:teuthology.parallel:result is None 2024-01-11T08:09:14.970 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4bcedb9fc11bed1fd6fd0c4fd59187401f309680 2024-01-11T08:09:15.114 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-11T08:09:15.172 INFO:teuthology.orchestra.run.smithi027.stdout:17.2.7-299.g4bcedb9f.el8 2024-01-11T08:09:15.173 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-299.g4bcedb9f.el8 2024-01-11T08:09:15.173 INFO:teuthology.task.install:The correct ceph version 17.2.7-299.g4bcedb9f is installed. 2024-01-11T08:09:15.175 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4bcedb9fc11bed1fd6fd0c4fd59187401f309680 2024-01-11T08:09:15.320 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-11T08:09:15.380 INFO:teuthology.orchestra.run.smithi077.stdout:17.2.7-299.g4bcedb9f.el8 2024-01-11T08:09:15.380 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-299.g4bcedb9f.el8 2024-01-11T08:09:15.380 INFO:teuthology.task.install:The correct ceph version 17.2.7-299.g4bcedb9f is installed. 2024-01-11T08:09:15.382 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-11T08:09:15.383 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:15.383 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-11T08:09:15.460 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:09:15.460 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-11T08:09:15.537 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-11T08:09:15.537 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:15.537 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2024-01-11T08:09:15.612 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-11T08:09:15.727 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:09:15.727 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2024-01-11T08:09:15.800 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-11T08:09:15.915 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-11T08:09:15.916 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:15.916 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-11T08:09:15.991 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-11T08:09:16.105 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:09:16.105 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-11T08:09:16.175 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-11T08:09:16.286 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-11T08:09:16.286 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:16.286 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/stdin-killer 2024-01-11T08:09:16.360 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-11T08:09:16.435 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:09:16.435 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/stdin-killer 2024-01-11T08:09:16.504 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-11T08:09:16.615 INFO:teuthology.run_tasks:Running task ceph... 2024-01-11T08:09:16.683 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-11T08:09:16.683 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /var/log/ceph 2024-01-11T08:09:16.688 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /var/log/ceph 2024-01-11T08:09:16.758 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-11T08:09:16.758 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-11T08:09:16.762 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-11T08:09:16.831 INFO:tasks.ceph:Creating extra log directories... 2024-01-11T08:09:16.832 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-11T08:09:16.836 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-11T08:09:16.912 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-01-11T08:09:16.912 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-01-11T08:09:16.912 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-01-11T08:09:16.912 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:16.913 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-11T08:09:16.961 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-11T08:09:17.219 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-11T08:09:17.292 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:09:17.292 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-11T08:09:17.340 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-11T08:09:18.210 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-11T08:09:18.292 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-11T08:09:18.292 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-11T08:09:18.292 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512563', 'branch': 'wip-vshankar-testing1-quincy-2024-01-10-2010-quincy', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7512563', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing1-quincy-2024-01-10-2010-quincy'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '4bcedb9fc11bed1fd6fd0c4fd59187401f309680'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4bcedb9fc11bed1fd6fd0c4fd59187401f309680'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing1-quincy-2024-01-10-2010-quincy', 'sha1': '4bcedb9fc11bed1fd6fd0c4fd59187401f309680'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 419, 'sha1': '4bcedb9fc11bed1fd6fd0c4fd59187401f309680', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing1-quincy-2024-01-10-2010-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '4bcedb9fc11bed1fd6fd0c4fd59187401f309680', 'targets': {'smithi027.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEunaG6cGUEWWP4tba6VzAu7nUPYDutCiMscx2EbA8khWok9fdqlJ8otWGFY17YR4vwhjOiDGke6+1VbYkIjus8=', 'smithi077.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNi4VBy2oxJo5MOLjqpm4HK6B6up8DRtTrBVz5TABoJt5pvN7mongi/6XvIYziSsfdzl9BonN62qXSr5Gn3hSHk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '4bcedb9fc11bed1fd6fd0c4fd59187401f309680'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '4bcedb9fc11bed1fd6fd0c4fd59187401f309680', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-10_19:20:36', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-11T08:09:18.293 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-11T08:09:18.298 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-11T08:09:18.345 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/run/ceph 2024-01-11T08:09:18.349 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/run/ceph 2024-01-11T08:09:18.422 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:18.422 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-01-11T08:09:18.469 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-11T08:09:18.469 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2024-01-11T08:09:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-11T08:09:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T08:09:18.558 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 445 Links: 1 2024-01-11T08:09:18.558 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T08:09:18.558 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-01-11T08:09:18.558 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-01-11 08:09:11.112035564 +0000 2024-01-11T08:09:18.559 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-01-11 08:05:44.783318806 +0000 2024-01-11T08:09:18.559 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-01-11 08:05:44.783318806 +0000 2024-01-11T08:09:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-01-11 08:05:44.754319545 +0000 2024-01-11T08:09:18.559 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-11T08:09:18.628 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-01-11T08:09:18.628 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-01-11T08:09:18.629 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000329802 s, 1.6 MB/s 2024-01-11T08:09:18.631 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-11T08:09:18.681 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2024-01-11T08:09:18.731 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-11T08:09:18.731 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T08:09:18.731 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 451 Links: 1 2024-01-11T08:09:18.731 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T08:09:18.731 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-01-11T08:09:18.731 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-01-11 08:09:11.112035564 +0000 2024-01-11T08:09:18.731 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-01-11 08:05:45.287305971 +0000 2024-01-11T08:09:18.731 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-01-11 08:05:45.287305971 +0000 2024-01-11T08:09:18.731 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-01-11 08:05:45.257306735 +0000 2024-01-11T08:09:18.732 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-11T08:09:18.804 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-01-11T08:09:18.804 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-01-11T08:09:18.804 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000291161 s, 1.8 MB/s 2024-01-11T08:09:18.806 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-11T08:09:18.858 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2024-01-11T08:09:18.907 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-11T08:09:18.907 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T08:09:18.908 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 457 Links: 1 2024-01-11T08:09:18.908 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T08:09:18.908 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-01-11T08:09:18.908 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-01-11 08:09:11.112035564 +0000 2024-01-11T08:09:18.908 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-01-11 08:05:45.797292982 +0000 2024-01-11T08:09:18.908 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-01-11 08:05:45.797292982 +0000 2024-01-11T08:09:18.908 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-01-11 08:05:45.768293721 +0000 2024-01-11T08:09:18.908 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-11T08:09:18.981 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-01-11T08:09:18.982 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-01-11T08:09:18.982 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000272884 s, 1.9 MB/s 2024-01-11T08:09:18.984 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-11T08:09:19.037 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2024-01-11T08:09:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-11T08:09:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T08:09:19.086 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 463 Links: 1 2024-01-11T08:09:19.086 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T08:09:19.086 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-01-11T08:09:19.087 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-01-11 08:09:11.113035539 +0000 2024-01-11T08:09:19.087 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-01-11 08:05:46.281280656 +0000 2024-01-11T08:09:19.087 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-01-11 08:05:46.281280656 +0000 2024-01-11T08:09:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-01-11 08:05:46.257281267 +0000 2024-01-11T08:09:19.087 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-11T08:09:19.163 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-01-11T08:09:19.163 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-01-11T08:09:19.163 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000291155 s, 1.8 MB/s 2024-01-11T08:09:19.166 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-11T08:09:19.219 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-11T08:09:19.219 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:09:19.219 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2024-01-11T08:09:19.267 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-11T08:09:19.267 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2024-01-11T08:09:19.354 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-11T08:09:19.354 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T08:09:19.354 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 447 Links: 1 2024-01-11T08:09:19.354 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T08:09:19.354 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-01-11T08:09:19.354 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-01-11 08:09:07.539536183 +0000 2024-01-11T08:09:19.354 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-01-11 08:05:26.871965712 +0000 2024-01-11T08:09:19.354 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-01-11 08:05:26.871965712 +0000 2024-01-11T08:09:19.354 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-01-11 08:05:26.845966276 +0000 2024-01-11T08:09:19.355 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-11T08:09:19.421 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-01-11T08:09:19.422 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-01-11T08:09:19.422 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000302619 s, 1.7 MB/s 2024-01-11T08:09:19.424 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-11T08:09:19.472 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2024-01-11T08:09:19.519 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-11T08:09:19.519 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T08:09:19.519 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 453 Links: 1 2024-01-11T08:09:19.519 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T08:09:19.519 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-01-11T08:09:19.519 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-01-11 08:09:07.539536183 +0000 2024-01-11T08:09:19.519 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-01-11 08:05:27.353955269 +0000 2024-01-11T08:09:19.519 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-01-11 08:05:27.353955269 +0000 2024-01-11T08:09:19.519 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-01-11 08:05:27.327955832 +0000 2024-01-11T08:09:19.520 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-11T08:09:19.586 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-01-11T08:09:19.586 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-01-11T08:09:19.586 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000287183 s, 1.8 MB/s 2024-01-11T08:09:19.588 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-11T08:09:19.636 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2024-01-11T08:09:19.682 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-11T08:09:19.682 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T08:09:19.682 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 459 Links: 1 2024-01-11T08:09:19.682 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T08:09:19.682 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-01-11T08:09:19.682 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-01-11 08:09:07.539536183 +0000 2024-01-11T08:09:19.682 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-01-11 08:05:27.850944501 +0000 2024-01-11T08:09:19.682 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-01-11 08:05:27.850944501 +0000 2024-01-11T08:09:19.682 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-01-11 08:05:27.824945064 +0000 2024-01-11T08:09:19.683 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-11T08:09:19.788 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-01-11T08:09:19.788 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-01-11T08:09:19.788 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000299292 s, 1.7 MB/s 2024-01-11T08:09:19.790 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-11T08:09:19.838 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2024-01-11T08:09:19.884 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-11T08:09:19.884 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T08:09:19.884 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 465 Links: 1 2024-01-11T08:09:19.884 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T08:09:19.884 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-01-11T08:09:19.884 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-01-11 08:09:07.540536162 +0000 2024-01-11T08:09:19.884 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-01-11 08:05:28.332934057 +0000 2024-01-11T08:09:19.884 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-01-11 08:05:28.332934057 +0000 2024-01-11T08:09:19.884 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-01-11 08:05:28.305934642 +0000 2024-01-11T08:09:19.885 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-11T08:09:19.990 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-01-11T08:09:19.991 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-01-11T08:09:19.991 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000307974 s, 1.7 MB/s 2024-01-11T08:09:19.993 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-11T08:09:20.040 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-11T08:09:20.041 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-11T08:09:20.041 INFO:tasks.ceph:Generating config... 2024-01-11T08:09:20.042 INFO:tasks.ceph:[client] client mount timeout = 600 2024-01-11T08:09:20.042 INFO:tasks.ceph:[client] debug client = 20 2024-01-11T08:09:20.042 INFO:tasks.ceph:[client] debug ms = 1 2024-01-11T08:09:20.042 INFO:tasks.ceph:[client] pid file = 2024-01-11T08:09:20.042 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-01-11T08:09:20.042 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-01-11T08:09:20.042 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-01-11T08:09:20.042 INFO:tasks.ceph:[mds] debug mds = 20 2024-01-11T08:09:20.042 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-01-11T08:09:20.042 INFO:tasks.ceph:[mds] debug ms = 1 2024-01-11T08:09:20.042 INFO:tasks.ceph:[mds] mds debug frag = True 2024-01-11T08:09:20.042 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-01-11T08:09:20.043 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-01-11T08:09:20.043 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-01-11T08:09:20.043 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-01-11T08:09:20.043 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-01-11T08:09:20.043 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-01-11T08:09:20.043 INFO:tasks.ceph:[mgr] debug client = 20 2024-01-11T08:09:20.043 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-01-11T08:09:20.043 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-11T08:09:20.043 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-11T08:09:20.043 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-11T08:09:20.043 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-11T08:09:20.043 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-11T08:09:20.043 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-01-11T08:09:20.043 INFO:tasks.ceph:[osd] bdev async discard = True 2024-01-11T08:09:20.043 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-01-11T08:09:20.043 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-01-11T08:09:20.043 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-11T08:09:20.044 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-11T08:09:20.044 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-01-11T08:09:20.044 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-01-11T08:09:20.044 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-11T08:09:20.044 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-11T08:09:20.044 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-01-11T08:09:20.044 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-11T08:09:20.044 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-11T08:09:20.044 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-11T08:09:20.044 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-11T08:09:20.044 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-11T08:09:20.044 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-01-11T08:09:20.044 INFO:tasks.ceph:Setting up mon.a... 2024-01-11T08:09:20.045 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-11T08:09:20.150 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.keyring 2024-01-11T08:09:20.155 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-11T08:09:20.257 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-11T08:09:20.332 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.27'), ('mon.b', '172.21.15.77'), ('mon.c', '[v2:172.21.15.77:3301,v1:172.21.15.77:6790]')] 2024-01-11T08:09:20.333 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.27,172.21.15.77,[v2:172.21.15.77:3301,v1:172.21.15.77:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-01-11T08:09:20.334 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:20.334 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-11T08:09:20.381 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.27 --add b 172.21.15.77 --addv c '[v2:172.21.15.77:3301,v1:172.21.15.77:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-01-11T08:09:20.508 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-11T08:09:20.508 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: generated fsid 8fddb4a7-9d66-40f1-892d-6fe86c728ee4 2024-01-11T08:09:20.508 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = octopus 2024-01-11T08:09:20.508 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 0 2024-01-11T08:09:20.508 INFO:teuthology.orchestra.run.smithi027.stdout:fsid 8fddb4a7-9d66-40f1-892d-6fe86c728ee4 2024-01-11T08:09:20.508 INFO:teuthology.orchestra.run.smithi027.stdout:last_changed 2024-01-11T08:09:20.509390+0000 2024-01-11T08:09:20.509 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-01-11T08:09:20.509390+0000 2024-01-11T08:09:20.509 INFO:teuthology.orchestra.run.smithi027.stdout:min_mon_release 15 (octopus) 2024-01-11T08:09:20.509 INFO:teuthology.orchestra.run.smithi027.stdout:election_strategy: 1 2024-01-11T08:09:20.509 INFO:teuthology.orchestra.run.smithi027.stdout:0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-01-11T08:09:20.509 INFO:teuthology.orchestra.run.smithi027.stdout:1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.b 2024-01-11T08:09:20.509 INFO:teuthology.orchestra.run.smithi027.stdout:2: [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] mon.c 2024-01-11T08:09:20.509 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-01-11T08:09:20.510 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-11T08:09:20.557 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8fddb4a7-9d66-40f1-892d-6fe86c728ee4... 2024-01-11T08:09:20.558 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-11T08:09:20.602 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-11T08:09:20.720 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-01-11T08:09:20.720 INFO:teuthology.orchestra.run.smithi027.stdout: chdir = "" 2024-01-11T08:09:20.720 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-11T08:09:20.720 INFO:teuthology.orchestra.run.smithi027.stdout: auth supported = cephx 2024-01-11T08:09:20.720 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.720 INFO:teuthology.orchestra.run.smithi027.stdout: filestore xattr use omap = true 2024-01-11T08:09:20.720 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.720 INFO:teuthology.orchestra.run.smithi027.stdout: mon clock drift allowed = 1.000 2024-01-11T08:09:20.721 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.721 INFO:teuthology.orchestra.run.smithi027.stdout: osd crush chooseleaf type = 0 2024-01-11T08:09:20.721 INFO:teuthology.orchestra.run.smithi027.stdout: auth debug = true 2024-01-11T08:09:20.721 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.721 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on old message = true 2024-01-11T08:09:20.721 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on bug = true 2024-01-11T08:09:20.721 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.721 INFO:teuthology.orchestra.run.smithi027.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-11T08:09:20.721 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn max object skew = 0 2024-01-11T08:09:20.721 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.721 INFO:teuthology.orchestra.run.smithi027.stdout: # disable pg_autoscaler by default for new pools 2024-01-11T08:09:20.721 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-11T08:09:20.722 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.722 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default size = 2 2024-01-11T08:09:20.722 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.722 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow primary affinity = true 2024-01-11T08:09:20.722 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow pg remap = true 2024-01-11T08:09:20.722 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on legacy crush tunables = false 2024-01-11T08:09:20.722 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on crush straw calc version zero = false 2024-01-11T08:09:20.722 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on no sortbitwise = false 2024-01-11T08:09:20.722 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = false 2024-01-11T08:09:20.722 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on too few osds = false 2024-01-11T08:09:20.722 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-11T08:09:20.722 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-11T08:09:20.722 INFO:teuthology.orchestra.run.smithi027.stdout: mon_allow_pool_size_one = true 2024-01-11T08:09:20.722 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.723 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-11T08:09:20.723 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.723 INFO:teuthology.orchestra.run.smithi027.stdout: osd default data pool replay window = 5 2024-01-11T08:09:20.723 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.723 INFO:teuthology.orchestra.run.smithi027.stdout: mon allow pool delete = true 2024-01-11T08:09:20.723 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.723 INFO:teuthology.orchestra.run.smithi027.stdout: mon cluster log file level = debug 2024-01-11T08:09:20.723 INFO:teuthology.orchestra.run.smithi027.stdout: debug asserts on shutdown = true 2024-01-11T08:09:20.723 INFO:teuthology.orchestra.run.smithi027.stdout: mon health detail to clog = false 2024-01-11T08:09:20.723 INFO:teuthology.orchestra.run.smithi027.stdout: mon host = "172.21.15.27,172.21.15.77,[v2:172.21.15.77:3301,v1:172.21.15.77:6790]" 2024-01-11T08:09:20.723 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on skipped message = False 2024-01-11T08:09:20.723 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = 8fddb4a7-9d66-40f1-892d-6fe86c728ee4 2024-01-11T08:09:20.723 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.723 INFO:teuthology.orchestra.run.smithi027.stdout:[osd] 2024-01-11T08:09:20.724 INFO:teuthology.orchestra.run.smithi027.stdout: osd journal size = 100 2024-01-11T08:09:20.724 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.724 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub load threshold = 5.0 2024-01-11T08:09:20.724 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub max interval = 600 2024-01-11T08:09:20.724 INFO:teuthology.orchestra.run.smithi027.stdout: osd mclock profile = high_recovery_ops 2024-01-11T08:09:20.724 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.724 INFO:teuthology.orchestra.run.smithi027.stdout: osd recover clone overlap = true 2024-01-11T08:09:20.724 INFO:teuthology.orchestra.run.smithi027.stdout: osd recovery max chunk = 1048576 2024-01-11T08:09:20.724 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.724 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug shutdown = true 2024-01-11T08:09:20.724 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug op order = true 2024-01-11T08:09:20.724 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify stray on activate = true 2024-01-11T08:09:20.724 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.724 INFO:teuthology.orchestra.run.smithi027.stdout: osd open classes on start = true 2024-01-11T08:09:20.725 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug pg log writeout = true 2024-01-11T08:09:20.725 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.725 INFO:teuthology.orchestra.run.smithi027.stdout: osd deep scrub update digest min age = 30 2024-01-11T08:09:20.725 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.725 INFO:teuthology.orchestra.run.smithi027.stdout: osd map max advance = 10 2024-01-11T08:09:20.725 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.725 INFO:teuthology.orchestra.run.smithi027.stdout: journal zero on create = true 2024-01-11T08:09:20.725 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.725 INFO:teuthology.orchestra.run.smithi027.stdout: filestore ondisk finisher threads = 3 2024-01-11T08:09:20.725 INFO:teuthology.orchestra.run.smithi027.stdout: filestore apply finisher threads = 3 2024-01-11T08:09:20.725 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.725 INFO:teuthology.orchestra.run.smithi027.stdout: bdev debug aio = true 2024-01-11T08:09:20.725 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug misdirected ops = true 2024-01-11T08:09:20.725 INFO:teuthology.orchestra.run.smithi027.stdout: bdev async discard = True 2024-01-11T08:09:20.726 INFO:teuthology.orchestra.run.smithi027.stdout: bdev enable discard = True 2024-01-11T08:09:20.726 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore allocator = bitmap 2024-01-11T08:09:20.726 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore block size = 96636764160 2024-01-11T08:09:20.726 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore fsck on mount = True 2024-01-11T08:09:20.726 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluefs = 1/20 2024-01-11T08:09:20.726 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluestore = 1/20 2024-01-11T08:09:20.726 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-01-11T08:09:20.726 INFO:teuthology.orchestra.run.smithi027.stdout: debug osd = 20 2024-01-11T08:09:20.726 INFO:teuthology.orchestra.run.smithi027.stdout: debug rocksdb = 4/10 2024-01-11T08:09:20.726 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd backfillfull_ratio = 0.85 2024-01-11T08:09:20.726 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd full ratio = 0.9 2024-01-11T08:09:20.726 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd nearfull ratio = 0.8 2024-01-11T08:09:20.726 INFO:teuthology.orchestra.run.smithi027.stdout: osd failsafe full ratio = 0.95 2024-01-11T08:09:20.726 INFO:teuthology.orchestra.run.smithi027.stdout: osd objectstore = bluestore 2024-01-11T08:09:20.727 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-01-11T08:09:20.727 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.727 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr] 2024-01-11T08:09:20.727 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-01-11T08:09:20.727 INFO:teuthology.orchestra.run.smithi027.stdout: debug mgr = 20 2024-01-11T08:09:20.727 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-01-11T08:09:20.727 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-01-11T08:09:20.727 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-01-11T08:09:20.727 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-01-11T08:09:20.727 INFO:teuthology.orchestra.run.smithi027.stdout: mgr/telemetry/nag = false 2024-01-11T08:09:20.727 INFO:teuthology.orchestra.run.smithi027.stdout: debug client = 20 2024-01-11T08:09:20.727 INFO:teuthology.orchestra.run.smithi027.stdout: debug finisher = 20 2024-01-11T08:09:20.727 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.727 INFO:teuthology.orchestra.run.smithi027.stdout:[mon] 2024-01-11T08:09:20.728 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-01-11T08:09:20.728 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-01-11T08:09:20.728 INFO:teuthology.orchestra.run.smithi027.stdout: debug paxos = 20 2024-01-11T08:09:20.728 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-01-11T08:09:20.728 INFO:teuthology.orchestra.run.smithi027.stdout: mon data avail warn = 5 2024-01-11T08:09:20.728 INFO:teuthology.orchestra.run.smithi027.stdout: mon mgr mkfs grace = 240 2024-01-11T08:09:20.728 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-01-11T08:09:20.728 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd reporter subtree level = osd 2024-01-11T08:09:20.728 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd prime pg temp = true 2024-01-11T08:09:20.728 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-01-11T08:09:20.729 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.729 INFO:teuthology.orchestra.run.smithi027.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-11T08:09:20.729 INFO:teuthology.orchestra.run.smithi027.stdout: auth mon ticket ttl = 660 # 11m 2024-01-11T08:09:20.729 INFO:teuthology.orchestra.run.smithi027.stdout: auth service ticket ttl = 240 # 4m 2024-01-11T08:09:20.729 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.729 INFO:teuthology.orchestra.run.smithi027.stdout: # don't complain about insecure global_id in the test suite 2024-01-11T08:09:20.729 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-11T08:09:20.729 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-11T08:09:20.729 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.729 INFO:teuthology.orchestra.run.smithi027.stdout: # 1m isn't quite enough 2024-01-11T08:09:20.729 INFO:teuthology.orchestra.run.smithi027.stdout: mon_down_mkfs_grace = 2m 2024-01-11T08:09:20.729 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.729 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_filestore_osds = false 2024-01-11T08:09:20.729 INFO:teuthology.orchestra.run.smithi027.stdout: mon op complaint time = 120 2024-01-11T08:09:20.730 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:09:20.730 INFO:teuthology.orchestra.run.smithi027.stdout:[client] 2024-01-11T08:09:20.730 INFO:teuthology.orchestra.run.smithi027.stdout: rgw cache enabled = true 2024-01-11T08:09:20.730 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable ops log = true 2024-01-11T08:09:20.730 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable usage log = true 2024-01-11T08:09:20.730 INFO:teuthology.orchestra.run.smithi027.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-11T08:09:20.730 INFO:teuthology.orchestra.run.smithi027.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-11T08:09:20.730 INFO:teuthology.orchestra.run.smithi027.stdout: client mount timeout = 600 2024-01-11T08:09:20.730 INFO:teuthology.orchestra.run.smithi027.stdout: debug client = 20 2024-01-11T08:09:20.730 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-01-11T08:09:20.730 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = "" 2024-01-11T08:09:20.730 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-01-11T08:09:20.730 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-01-11T08:09:20.730 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.a] 2024-01-11T08:09:20.731 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.b] 2024-01-11T08:09:20.731 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.c] 2024-01-11T08:09:20.731 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a] 2024-01-11T08:09:20.731 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c] 2024-01-11T08:09:20.731 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b] 2024-01-11T08:09:20.731 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d] 2024-01-11T08:09:20.731 INFO:teuthology.orchestra.run.smithi027.stdout:[mds] 2024-01-11T08:09:20.731 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds = 20 2024-01-11T08:09:20.731 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds balancer = 20 2024-01-11T08:09:20.731 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-01-11T08:09:20.731 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug frag = True 2024-01-11T08:09:20.731 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug scatterstat = True 2024-01-11T08:09:20.731 INFO:teuthology.orchestra.run.smithi027.stdout: mds op complaint time = 180 2024-01-11T08:09:20.731 INFO:teuthology.orchestra.run.smithi027.stdout: mds verify scatter = True 2024-01-11T08:09:20.731 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-01-11T08:09:20.731 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-01-11T08:09:20.732 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-01-11T08:09:20.732 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2024-01-11T08:09:20.732 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2024-01-11T08:09:20.732 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-11T08:09:20.732 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2024-01-11T08:09:20.732 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.732 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2024-01-11T08:09:20.732 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.733 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2024-01-11T08:09:20.733 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.733 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2024-01-11T08:09:20.735 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2024-01-11T08:09:20.735 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.735 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2024-01-11T08:09:20.735 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2024-01-11T08:09:20.735 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.735 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-11T08:09:20.735 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2024-01-11T08:09:20.735 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.735 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2024-01-11T08:09:20.735 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-11T08:09:20.735 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.735 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2024-01-11T08:09:20.736 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.736 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2024-01-11T08:09:20.736 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2024-01-11T08:09:20.736 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2024-01-11T08:09:20.736 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2024-01-11T08:09:20.736 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2024-01-11T08:09:20.736 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2024-01-11T08:09:20.736 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2024-01-11T08:09:20.736 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-11T08:09:20.736 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-11T08:09:20.736 INFO:teuthology.orchestra.run.smithi077.stdout: mon_allow_pool_size_one = true 2024-01-11T08:09:20.737 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.737 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-11T08:09:20.737 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.737 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2024-01-11T08:09:20.737 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.737 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2024-01-11T08:09:20.737 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.737 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2024-01-11T08:09:20.737 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2024-01-11T08:09:20.737 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2024-01-11T08:09:20.737 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = "172.21.15.27,172.21.15.77,[v2:172.21.15.77:3301,v1:172.21.15.77:6790]" 2024-01-11T08:09:20.737 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on skipped message = False 2024-01-11T08:09:20.737 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = 8fddb4a7-9d66-40f1-892d-6fe86c728ee4 2024-01-11T08:09:20.737 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.738 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2024-01-11T08:09:20.738 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2024-01-11T08:09:20.738 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.738 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2024-01-11T08:09:20.738 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2024-01-11T08:09:20.738 INFO:teuthology.orchestra.run.smithi077.stdout: osd mclock profile = high_recovery_ops 2024-01-11T08:09:20.738 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.738 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2024-01-11T08:09:20.738 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2024-01-11T08:09:20.738 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.738 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2024-01-11T08:09:20.738 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2024-01-11T08:09:20.738 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2024-01-11T08:09:20.738 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.739 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2024-01-11T08:09:20.739 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2024-01-11T08:09:20.739 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.739 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2024-01-11T08:09:20.739 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.739 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2024-01-11T08:09:20.739 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.739 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2024-01-11T08:09:20.739 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.739 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2024-01-11T08:09:20.739 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2024-01-11T08:09:20.740 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.740 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2024-01-11T08:09:20.740 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2024-01-11T08:09:20.740 INFO:teuthology.orchestra.run.smithi077.stdout: bdev async discard = True 2024-01-11T08:09:20.740 INFO:teuthology.orchestra.run.smithi077.stdout: bdev enable discard = True 2024-01-11T08:09:20.740 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore allocator = bitmap 2024-01-11T08:09:20.740 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore block size = 96636764160 2024-01-11T08:09:20.740 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore fsck on mount = True 2024-01-11T08:09:20.740 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluefs = 1/20 2024-01-11T08:09:20.740 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluestore = 1/20 2024-01-11T08:09:20.740 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-01-11T08:09:20.740 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 20 2024-01-11T08:09:20.740 INFO:teuthology.orchestra.run.smithi077.stdout: debug rocksdb = 4/10 2024-01-11T08:09:20.740 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd backfillfull_ratio = 0.85 2024-01-11T08:09:20.741 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd full ratio = 0.9 2024-01-11T08:09:20.741 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd nearfull ratio = 0.8 2024-01-11T08:09:20.741 INFO:teuthology.orchestra.run.smithi077.stdout: osd failsafe full ratio = 0.95 2024-01-11T08:09:20.741 INFO:teuthology.orchestra.run.smithi077.stdout: osd objectstore = bluestore 2024-01-11T08:09:20.741 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-01-11T08:09:20.741 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.741 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2024-01-11T08:09:20.741 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-01-11T08:09:20.741 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2024-01-11T08:09:20.741 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-01-11T08:09:20.741 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-01-11T08:09:20.741 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-01-11T08:09:20.741 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-01-11T08:09:20.741 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2024-01-11T08:09:20.741 INFO:teuthology.orchestra.run.smithi077.stdout: debug client = 20 2024-01-11T08:09:20.742 INFO:teuthology.orchestra.run.smithi077.stdout: debug finisher = 20 2024-01-11T08:09:20.742 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.742 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2024-01-11T08:09:20.742 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-01-11T08:09:20.742 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-01-11T08:09:20.742 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2024-01-11T08:09:20.742 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-01-11T08:09:20.742 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2024-01-11T08:09:20.742 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2024-01-11T08:09:20.742 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-01-11T08:09:20.742 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2024-01-11T08:09:20.742 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2024-01-11T08:09:20.742 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-01-11T08:09:20.742 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.743 INFO:teuthology.orchestra.run.smithi077.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-11T08:09:20.743 INFO:teuthology.orchestra.run.smithi077.stdout: auth mon ticket ttl = 660 # 11m 2024-01-11T08:09:20.743 INFO:teuthology.orchestra.run.smithi077.stdout: auth service ticket ttl = 240 # 4m 2024-01-11T08:09:20.743 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.743 INFO:teuthology.orchestra.run.smithi077.stdout: # don't complain about insecure global_id in the test suite 2024-01-11T08:09:20.743 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-11T08:09:20.743 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-11T08:09:20.743 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.743 INFO:teuthology.orchestra.run.smithi077.stdout: # 1m isn't quite enough 2024-01-11T08:09:20.743 INFO:teuthology.orchestra.run.smithi077.stdout: mon_down_mkfs_grace = 2m 2024-01-11T08:09:20.743 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.743 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_filestore_osds = false 2024-01-11T08:09:20.743 INFO:teuthology.orchestra.run.smithi077.stdout: mon op complaint time = 120 2024-01-11T08:09:20.743 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-11T08:09:20.744 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2024-01-11T08:09:20.744 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2024-01-11T08:09:20.744 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2024-01-11T08:09:20.744 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2024-01-11T08:09:20.744 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-11T08:09:20.744 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-11T08:09:20.744 INFO:teuthology.orchestra.run.smithi077.stdout: client mount timeout = 600 2024-01-11T08:09:20.744 INFO:teuthology.orchestra.run.smithi077.stdout: debug client = 20 2024-01-11T08:09:20.744 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-01-11T08:09:20.744 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = "" 2024-01-11T08:09:20.744 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-01-11T08:09:20.744 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-01-11T08:09:20.744 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2024-01-11T08:09:20.744 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.b] 2024-01-11T08:09:20.745 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.c] 2024-01-11T08:09:20.745 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a] 2024-01-11T08:09:20.745 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c] 2024-01-11T08:09:20.745 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b] 2024-01-11T08:09:20.745 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d] 2024-01-11T08:09:20.745 INFO:teuthology.orchestra.run.smithi077.stdout:[mds] 2024-01-11T08:09:20.745 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds = 20 2024-01-11T08:09:20.745 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds balancer = 20 2024-01-11T08:09:20.745 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-01-11T08:09:20.745 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug frag = True 2024-01-11T08:09:20.745 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug scatterstat = True 2024-01-11T08:09:20.745 INFO:teuthology.orchestra.run.smithi077.stdout: mds op complaint time = 180 2024-01-11T08:09:20.745 INFO:teuthology.orchestra.run.smithi077.stdout: mds verify scatter = True 2024-01-11T08:09:20.745 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2024-01-11T08:09:20.746 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2024-01-11T08:09:20.746 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2024-01-11T08:09:20.753 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-11T08:09:20.753 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-11T08:09:20.863 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-11T08:09:20.863 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:20.863 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-11T08:09:20.910 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:20.910 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-11T08:09:20.996 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T08:09:20.996 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:20.996 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-11T08:09:20.996 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-11T08:09:21.134 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:21.134 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-11T08:09:21.221 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi077.front.sepia.ceph.com 2024-01-11T08:09:21.221 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:09:21.221 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-11T08:09:21.221 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-11T08:09:21.316 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:09:21.316 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-11T08:09:21.402 INFO:tasks.ceph:Setting up mon nodes... 2024-01-11T08:09:21.402 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-11T08:09:21.402 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-01-11T08:09:21.529 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-01-11T08:09:21.535 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-11T08:09:21.658 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-11T08:09:21.664 INFO:tasks.ceph:Setting up mds nodes... 2024-01-11T08:09:21.664 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-01-11T08:09:21.786 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-01-11T08:09:21.791 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-01-11T08:09:21.863 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-01-11T08:09:21.984 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-01-11T08:09:21.989 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-01-11T08:09:22.061 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-01-11T08:09:22.178 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-01-11T08:09:22.183 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-01-11T08:09:22.253 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-01-11T08:09:22.369 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-01-11T08:09:22.374 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-01-11T08:09:22.443 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-11T08:09:22.443 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-11T08:09:22.538 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-11T08:09:22.569 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-01-11T08:09:22.664 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-01-11T08:09:22.696 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-11T08:09:22.787 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-01-11T08:09:22.817 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-01-11T08:09:22.907 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-01-11T08:09:22.936 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-11T08:09:22.937 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-11T08:09:22.937 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-11T08:09:23.009 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-11T08:09:23.009 INFO:tasks.ceph:role: osd.0 2024-01-11T08:09:23.009 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T08:09:23.009 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-11T08:09:23.088 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-11T08:09:23.088 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-11T08:09:23.088 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-11T08:09:23.088 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-11T08:09:23.088 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-11T08:09:23.088 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-01-11T08:09:23.088 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-11T08:09:23.088 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-11T08:09:23.088 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-11T08:09:23.088 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-11T08:09:23.524 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-01-11T08:09:23.528 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-01-11T08:09:23.528 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-11T08:09:23.722 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-11T08:09:23.803 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-11T08:09:23.875 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-11T08:09:23.875 INFO:tasks.ceph:role: osd.1 2024-01-11T08:09:23.875 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T08:09:23.875 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-11T08:09:23.954 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-11T08:09:23.954 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-11T08:09:23.954 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-11T08:09:23.954 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-11T08:09:23.954 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-11T08:09:23.954 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-01-11T08:09:23.954 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-11T08:09:23.954 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-11T08:09:23.954 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-11T08:09:23.954 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-11T08:09:24.368 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-01-11T08:09:24.372 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-01-11T08:09:24.372 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-11T08:09:24.509 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-11T08:09:24.588 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-11T08:09:24.661 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-11T08:09:24.661 INFO:tasks.ceph:role: osd.2 2024-01-11T08:09:24.661 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T08:09:24.661 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-11T08:09:24.740 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-11T08:09:24.741 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-11T08:09:24.741 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-11T08:09:24.741 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-11T08:09:24.741 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-11T08:09:24.741 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-01-11T08:09:24.741 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-11T08:09:24.741 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-11T08:09:24.741 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-11T08:09:24.741 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-11T08:09:25.190 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-01-11T08:09:25.195 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-01-11T08:09:25.195 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-11T08:09:25.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-11T08:09:25.474 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-01-11T08:09:25.545 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-11T08:09:25.545 INFO:tasks.ceph:role: osd.3 2024-01-11T08:09:25.545 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T08:09:25.545 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-11T08:09:25.624 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-11T08:09:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-11T08:09:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-11T08:09:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-11T08:09:25.624 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-11T08:09:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-01-11T08:09:25.624 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-11T08:09:25.624 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-11T08:09:25.624 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-11T08:09:25.624 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-11T08:09:26.055 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-01-11T08:09:26.060 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-01-11T08:09:26.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-01-11T08:09:26.196 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-01-11T08:09:26.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-11T08:09:26.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:26.966+0000 7fb47c529540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-11T08:09:26.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:26.966+0000 7fb47c529540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-11T08:09:26.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:26.966+0000 7fb47c529540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-11T08:09:26.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:26.966+0000 7fb47c529540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-11T08:09:26.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:26.966+0000 7fb47c529540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-11T08:09:26.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:26.969+0000 7fb47c529540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-11T08:09:27.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:27.512+0000 7fb47c529540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:28.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:28.272+0000 7fb47c529540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:29.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:29.036+0000 7fb47c529540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:29.853 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-11T08:09:29.931 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-11T08:09:30.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:30.037+0000 7fda535f2540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-11T08:09:30.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:30.038+0000 7fda535f2540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-11T08:09:30.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:30.038+0000 7fda535f2540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-11T08:09:30.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:30.038+0000 7fda535f2540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-11T08:09:30.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:30.038+0000 7fda535f2540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-11T08:09:30.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:30.039+0000 7fda535f2540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-11T08:09:30.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:30.576+0000 7fda535f2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:31.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:31.339+0000 7fda535f2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:32.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:32.104+0000 7fda535f2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:32.919 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-11T08:09:32.998 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-11T08:09:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:33.103+0000 7f1c6d608540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-11T08:09:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:33.104+0000 7f1c6d608540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-11T08:09:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:33.104+0000 7f1c6d608540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-11T08:09:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:33.104+0000 7f1c6d608540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-11T08:09:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:33.104+0000 7f1c6d608540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-11T08:09:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:33.105+0000 7f1c6d608540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-11T08:09:33.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:33.654+0000 7f1c6d608540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:34.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:34.419+0000 7f1c6d608540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:35.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:35.180+0000 7f1c6d608540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:35.993 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-11T08:09:36.071 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-11T08:09:36.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:36.176+0000 7f63ece16540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-01-11T08:09:36.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:36.177+0000 7f63ece16540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-01-11T08:09:36.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:36.177+0000 7f63ece16540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-11T08:09:36.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:36.177+0000 7f63ece16540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-11T08:09:36.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:36.177+0000 7f63ece16540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-11T08:09:36.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:36.178+0000 7f63ece16540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-01-11T08:09:36.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:36.722+0000 7f63ece16540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:37.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:37.481+0000 7f63ece16540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:38.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:38.245+0000 7f63ece16540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:39.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-01-11T08:09:39.139 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-01-11T08:09:39.216 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-11T08:09:39.216 INFO:tasks.ceph:role: osd.4 2024-01-11T08:09:39.216 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com 2024-01-11T08:09:39.217 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-11T08:09:39.295 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-11T08:09:39.295 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-11T08:09:39.295 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-11T08:09:39.296 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-11T08:09:39.296 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-11T08:09:39.296 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-01-11T08:09:39.296 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-11T08:09:39.296 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-11T08:09:39.296 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-11T08:09:39.296 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-11T08:09:39.729 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-01-11T08:09:39.733 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-01-11T08:09:39.733 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-01-11T08:09:39.939 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-01-11T08:09:40.020 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-01-11T08:09:40.089 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-11T08:09:40.089 INFO:tasks.ceph:role: osd.5 2024-01-11T08:09:40.089 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com 2024-01-11T08:09:40.089 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-11T08:09:40.165 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-11T08:09:40.165 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-11T08:09:40.165 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-11T08:09:40.165 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-11T08:09:40.165 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-11T08:09:40.165 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-01-11T08:09:40.165 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-11T08:09:40.165 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-11T08:09:40.165 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-11T08:09:40.165 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-11T08:09:40.572 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-01-11T08:09:40.575 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-01-11T08:09:40.575 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-01-11T08:09:40.704 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-01-11T08:09:40.781 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-01-11T08:09:40.850 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-11T08:09:40.850 INFO:tasks.ceph:role: osd.6 2024-01-11T08:09:40.850 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com 2024-01-11T08:09:40.850 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-11T08:09:40.928 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-11T08:09:40.928 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-11T08:09:40.929 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-11T08:09:40.929 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-11T08:09:40.929 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-11T08:09:40.929 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-01-11T08:09:40.929 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-11T08:09:40.929 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-11T08:09:40.929 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-11T08:09:40.929 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-11T08:09:41.381 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-01-11T08:09:41.385 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-01-11T08:09:41.385 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-01-11T08:09:41.566 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-01-11T08:09:41.644 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-01-11T08:09:41.713 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-11T08:09:41.713 INFO:tasks.ceph:role: osd.7 2024-01-11T08:09:41.713 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi077.front.sepia.ceph.com 2024-01-11T08:09:41.714 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-11T08:09:41.789 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-11T08:09:41.789 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-11T08:09:41.789 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-11T08:09:41.789 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-11T08:09:41.789 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-11T08:09:41.789 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-01-11T08:09:41.789 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-11T08:09:41.790 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-11T08:09:41.790 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-11T08:09:41.790 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-11T08:09:42.221 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-01-11T08:09:42.225 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-01-11T08:09:42.225 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-01-11T08:09:42.359 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-01-11T08:09:42.435 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-11T08:09:42.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:42.552+0000 7fca8478f540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-01-11T08:09:42.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:42.552+0000 7fca8478f540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-01-11T08:09:42.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:42.552+0000 7fca8478f540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-11T08:09:42.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:42.552+0000 7fca8478f540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-11T08:09:42.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:42.552+0000 7fca8478f540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-11T08:09:42.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:42.554+0000 7fca8478f540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-01-11T08:09:43.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:43.091+0000 7fca8478f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:43.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:43.861+0000 7fca8478f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:44.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:44.623+0000 7fca8478f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:45.446 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-01-11T08:09:45.521 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-11T08:09:45.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:45.621+0000 7f5214f33540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-01-11T08:09:45.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:45.621+0000 7f5214f33540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-01-11T08:09:45.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:45.622+0000 7f5214f33540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-11T08:09:45.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:45.622+0000 7f5214f33540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-11T08:09:45.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:45.622+0000 7f5214f33540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-11T08:09:45.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:45.623+0000 7f5214f33540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-01-11T08:09:46.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:46.165+0000 7f5214f33540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:46.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:46.931+0000 7f5214f33540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:47.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:47.694+0000 7f5214f33540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:48.294 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:09:48.299 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:09:48.508 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-01-11T08:09:48.578 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-11T08:09:48.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:48.677+0000 7f9fbfde2540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-01-11T08:09:48.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:48.678+0000 7f9fbfde2540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-01-11T08:09:48.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:48.678+0000 7f9fbfde2540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-11T08:09:48.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:48.678+0000 7f9fbfde2540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-11T08:09:48.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:48.678+0000 7f9fbfde2540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-11T08:09:48.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:48.679+0000 7f9fbfde2540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-01-11T08:09:49.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:49.219+0000 7f9fbfde2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:49.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:49.987+0000 7f9fbfde2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:50.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:50.744+0000 7f9fbfde2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:51.562 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-01-11T08:09:51.645 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-11T08:09:51.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:51.745+0000 7f3821955540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-01-11T08:09:51.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:51.745+0000 7f3821955540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-01-11T08:09:51.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:51.745+0000 7f3821955540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-11T08:09:51.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:51.745+0000 7f3821955540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-11T08:09:51.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:51.745+0000 7f3821955540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-11T08:09:51.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:51.747+0000 7f3821955540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-01-11T08:09:52.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:52.289+0000 7f3821955540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:53.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:53.046+0000 7f3821955540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:53.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:09:53.812+0000 7f3821955540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-01-11T08:09:54.631 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-01-11T08:09:54.704 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-11T08:09:54.704 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:54.704 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-01-11T08:09:54.784 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:54.784 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-01-11T08:09:54.858 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:54.859 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-01-11T08:09:54.970 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:54.970 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-11T08:09:55.081 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:55.082 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-11T08:09:55.193 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:55.193 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-11T08:09:55.304 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:55.304 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-01-11T08:09:55.416 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:09:55.416 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-11T08:09:55.491 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:09:55.491 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-01-11T08:09:55.563 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:09:55.564 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-01-11T08:09:55.672 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:09:55.672 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-01-11T08:09:55.781 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:09:55.781 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-01-11T08:09:55.890 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:09:55.890 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-01-11T08:09:56.000 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:09:56.000 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-01-11T08:09:56.109 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:56.109 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-11T08:09:56.160 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:09:56.160 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-01-11T08:09:56.209 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:09:56.209 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-01-11T08:09:56.255 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:09:56.255 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-01-11T08:09:56.341 INFO:tasks.ceph:Adding keys to all mons... 2024-01-11T08:09:56.341 DEBUG:teuthology.orchestra.run.smithi027:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-11T08:09:56.346 DEBUG:teuthology.orchestra.run.smithi077:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-11T08:09:56.412 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.y] 2024-01-11T08:09:56.412 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQAxop9l3ZGoHxAANqzytTTvWYSomG+RFsFDyg== 2024-01-11T08:09:56.412 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a] 2024-01-11T08:09:56.412 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQAxop9lQm3yLhAAv6erz5hMfat8EUzRA5luMA== 2024-01-11T08:09:56.413 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c] 2024-01-11T08:09:56.413 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQAxop9l1NnEOhAALHXsc5es7hufCthGdRzkyw== 2024-01-11T08:09:56.413 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.0] 2024-01-11T08:09:56.413 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQA2op9ltZygORAAZr1X1nCN0ce8rIVaGDpFyQ== 2024-01-11T08:09:56.413 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.1] 2024-01-11T08:09:56.413 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQA6op9lqSRPAhAAmtyCsyK5ZVGX2wfhOCMuFQ== 2024-01-11T08:09:56.413 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.2] 2024-01-11T08:09:56.413 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQA9op9lZcs8BhAA8j+U+jPOoy+fliVWZAPFYg== 2024-01-11T08:09:56.413 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.3] 2024-01-11T08:09:56.413 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQBAop9lFuCUChAAP6si/2W1mxH4ecQJvuumkw== 2024-01-11T08:09:56.413 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.x] 2024-01-11T08:09:56.413 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQAxop9lXBZLJxAAXa1S8Vkm3MiV4igtNbg/1Q== 2024-01-11T08:09:56.413 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b] 2024-01-11T08:09:56.413 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQAyop9l+FG8ChAACHndtVLx7qS3sidPniSycA== 2024-01-11T08:09:56.414 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d] 2024-01-11T08:09:56.414 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQAyop9lruoWFhAAm8AmrZtTJhYI1+19F79RPQ== 2024-01-11T08:09:56.414 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.4] 2024-01-11T08:09:56.414 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQBGop9lOpwBIRAAcpcsjwmX2IldcKHJWAgHuA== 2024-01-11T08:09:56.414 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.5] 2024-01-11T08:09:56.414 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQBJop9l4q8hJRAAK8mAYVRQMoWS6wBOU69HmA== 2024-01-11T08:09:56.414 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.6] 2024-01-11T08:09:56.414 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQBMop9lqMF9KBAAFgFFI2GWt9AGif4AoScThw== 2024-01-11T08:09:56.414 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.7] 2024-01-11T08:09:56.414 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQBPop9lN7eDLBAAKswPZFfs0g2AaSe6V+pbbQ== 2024-01-11T08:09:56.414 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-01-11T08:09:56.414 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQAyop9lT5okIBAABy8PbJJbA062VNmb1Y8qhg== 2024-01-11T08:09:56.414 INFO:teuthology.orchestra.run.smithi027.stdout:[client.2] 2024-01-11T08:09:56.414 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQAyop9lWayxJxAAcalM8QSOGtbCXxjYZGaNQA== 2024-01-11T08:09:56.414 INFO:teuthology.orchestra.run.smithi027.stdout:[client.1] 2024-01-11T08:09:56.415 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQAyop9lDf37LhAAqV+EabKXJEW1sS5xtu1tXw== 2024-01-11T08:09:56.415 INFO:teuthology.orchestra.run.smithi027.stdout:[client.3] 2024-01-11T08:09:56.415 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQAyop9lHD4tNhAAdmnFBmMrK+QhQYdpbzW0mg== 2024-01-11T08:09:56.447 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr.y] 2024-01-11T08:09:56.447 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAxop9l3ZGoHxAANqzytTTvWYSomG+RFsFDyg== 2024-01-11T08:09:56.447 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a] 2024-01-11T08:09:56.447 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAxop9lQm3yLhAAv6erz5hMfat8EUzRA5luMA== 2024-01-11T08:09:56.447 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c] 2024-01-11T08:09:56.447 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAxop9l1NnEOhAALHXsc5es7hufCthGdRzkyw== 2024-01-11T08:09:56.447 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.0] 2024-01-11T08:09:56.448 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQA2op9ltZygORAAZr1X1nCN0ce8rIVaGDpFyQ== 2024-01-11T08:09:56.448 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.1] 2024-01-11T08:09:56.448 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQA6op9lqSRPAhAAmtyCsyK5ZVGX2wfhOCMuFQ== 2024-01-11T08:09:56.448 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.2] 2024-01-11T08:09:56.448 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQA9op9lZcs8BhAA8j+U+jPOoy+fliVWZAPFYg== 2024-01-11T08:09:56.448 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.3] 2024-01-11T08:09:56.448 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBAop9lFuCUChAAP6si/2W1mxH4ecQJvuumkw== 2024-01-11T08:09:56.448 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr.x] 2024-01-11T08:09:56.448 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAxop9lXBZLJxAAXa1S8Vkm3MiV4igtNbg/1Q== 2024-01-11T08:09:56.448 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b] 2024-01-11T08:09:56.448 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAyop9l+FG8ChAACHndtVLx7qS3sidPniSycA== 2024-01-11T08:09:56.448 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d] 2024-01-11T08:09:56.448 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAyop9lruoWFhAAm8AmrZtTJhYI1+19F79RPQ== 2024-01-11T08:09:56.448 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.4] 2024-01-11T08:09:56.449 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBGop9lOpwBIRAAcpcsjwmX2IldcKHJWAgHuA== 2024-01-11T08:09:56.449 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.5] 2024-01-11T08:09:56.449 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBJop9l4q8hJRAAK8mAYVRQMoWS6wBOU69HmA== 2024-01-11T08:09:56.449 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.6] 2024-01-11T08:09:56.449 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBMop9lqMF9KBAAFgFFI2GWt9AGif4AoScThw== 2024-01-11T08:09:56.449 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.7] 2024-01-11T08:09:56.449 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBPop9lN7eDLBAAKswPZFfs0g2AaSe6V+pbbQ== 2024-01-11T08:09:56.449 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-01-11T08:09:56.449 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAyop9lT5okIBAABy8PbJJbA062VNmb1Y8qhg== 2024-01-11T08:09:56.449 INFO:teuthology.orchestra.run.smithi077.stdout:[client.2] 2024-01-11T08:09:56.449 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAyop9lWayxJxAAcalM8QSOGtbCXxjYZGaNQA== 2024-01-11T08:09:56.449 INFO:teuthology.orchestra.run.smithi077.stdout:[client.1] 2024-01-11T08:09:56.449 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAyop9lDf37LhAAqV+EabKXJEW1sS5xtu1tXw== 2024-01-11T08:09:56.449 INFO:teuthology.orchestra.run.smithi077.stdout:[client.3] 2024-01-11T08:09:56.450 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAyop9lHD4tNhAAdmnFBmMrK+QhQYdpbzW0mg== 2024-01-11T08:09:56.450 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-11T08:09:56.460 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-11T08:09:56.594 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-11T08:09:56.603 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-11T08:09:56.710 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-11T08:09:56.716 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-11T08:09:56.822 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T08:09:56.828 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T08:09:56.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T08:09:56.935 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T08:09:57.039 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T08:09:57.044 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T08:09:57.147 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T08:09:57.151 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T08:09:57.255 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-11T08:09:57.259 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-11T08:09:57.362 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-11T08:09:57.366 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-11T08:09:57.473 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-11T08:09:57.478 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-11T08:09:57.584 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T08:09:57.589 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T08:09:57.692 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T08:09:57.697 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T08:09:57.800 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T08:09:57.805 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T08:09:57.908 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T08:09:57.912 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T08:09:58.016 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-11T08:09:58.020 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-11T08:09:58.126 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-11T08:09:58.130 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-11T08:09:58.236 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-11T08:09:58.241 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-11T08:09:58.347 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-11T08:09:58.352 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-11T08:09:58.458 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-11T08:09:58.458 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-11T08:09:58.530 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-11T08:09:58.652 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-11T08:09:58.725 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-01-11T08:09:58.796 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-11T08:09:58.912 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-01-11T08:09:58.981 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-01-11T08:09:59.051 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-11T08:09:59.166 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-01-11T08:09:59.235 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-11T08:09:59.240 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-11T08:09:59.285 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-11T08:09:59.285 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-11T08:09:59.285 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-11T08:09:59.290 INFO:tasks.ceph.mon.a:Started 2024-01-11T08:09:59.290 INFO:tasks.ceph.mon.b:Restarting daemon 2024-01-11T08:09:59.290 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-01-11T08:09:59.329 INFO:tasks.ceph.mon.b:Started 2024-01-11T08:09:59.330 INFO:tasks.ceph.mon.c:Restarting daemon 2024-01-11T08:09:59.330 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-01-11T08:09:59.334 INFO:tasks.ceph.mon.c:Started 2024-01-11T08:09:59.334 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-11T08:09:59.334 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-01-11T08:09:59.334 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-01-11T08:09:59.338 INFO:tasks.ceph.mgr.y:Started 2024-01-11T08:09:59.339 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-11T08:09:59.339 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-11T08:09:59.343 INFO:tasks.ceph.mgr.x:Started 2024-01-11T08:09:59.343 INFO:tasks.ceph:Setting crush tunables to default 2024-01-11T08:09:59.343 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd crush tunables default 2024-01-11T08:09:59.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:59.542+0000 7f3c9e3ff700 1 Processor -- start 2024-01-11T08:09:59.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:59.542+0000 7f3c9e3ff700 1 -- start start 2024-01-11T08:09:59.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:59.543+0000 7f3c9e3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 0x7f3c9815a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:09:59.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:59.544+0000 7f3c9e3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c980cf850 con 0x7f3c9815ae70 2024-01-11T08:09:59.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:59.544+0000 7f3c9e3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c980cec40 con 0x7f3c980ccd70 2024-01-11T08:09:59.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:59.544+0000 7f3c9e3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c980cef20 con 0x7f3c9815a1b0 2024-01-11T08:09:59.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:59.544+0000 7f3c9c99c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3c9815ae70 0x7f3c9815b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42010/0 (socket says 172.21.15.27:42010) 2024-01-11T08:09:59.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:59.544+0000 7f3c9c99c700 1 -- 172.21.15.27:0/698657770 learned_addr learned my addr 172.21.15.27:0/698657770 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:09:59.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:59.544+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 0x7f3c9815a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:09:59.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:59.544+0000 7f3c97fff700 1 -- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 msgr2=0x7f3c9815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-11T08:09:59.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:59.544+0000 7f3c97fff700 1 -- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 msgr2=0x7f3c9815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-11T08:09:59.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:59.544+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 0x7f3c9815a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-01-11T08:09:59.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:59.544+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 0x7f3c9815a580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:09:59.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:59.746+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 0x7f3c9815a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:09:59.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:59.746+0000 7f3c97fff700 1 -- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 msgr2=0x7f3c9815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-11T08:09:59.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:59.746+0000 7f3c97fff700 1 -- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 msgr2=0x7f3c9815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-11T08:09:59.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:59.746+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 0x7f3c9815a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-01-11T08:09:59.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:09:59.746+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 0x7f3c9815a580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:10:00.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:00.147+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 0x7f3c9815a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:00.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:00.147+0000 7f3c97fff700 1 -- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 msgr2=0x7f3c9815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-11T08:10:00.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:00.147+0000 7f3c97fff700 1 -- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 msgr2=0x7f3c9815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-11T08:10:00.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:00.147+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 0x7f3c9815a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-01-11T08:10:00.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:00.147+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 0x7f3c9815a580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-11T08:10:00.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:00.949+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 0x7f3c9815a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:00.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:00.949+0000 7f3c97fff700 1 -- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 msgr2=0x7f3c9815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-11T08:10:00.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:00.949+0000 7f3c97fff700 1 -- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 msgr2=0x7f3c9815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-11T08:10:00.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:00.949+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 0x7f3c9815a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-01-11T08:10:00.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:00.949+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 0x7f3c9815a580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-11T08:10:02.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.545+0000 7f3c95ffb700 1 -- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 msgr2=0x7f3c9815a580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-11T08:10:02.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.545+0000 7f3c95ffb700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c9815a1b0 0x7f3c9815a580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:02.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.545+0000 7f3c95ffb700 1 -- 172.21.15.27:0/698657770 >> v1:172.21.15.77:6789/0 conn(0x7f3c980ccd70 legacy=0x7f3c980cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:02.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.545+0000 7f3c95ffb700 1 -- 172.21.15.27:0/698657770 >> v1:172.21.15.27:6789/0 conn(0x7f3c9815ae70 legacy=0x7f3c9815b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:02.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.545+0000 7f3c95ffb700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 0x7f3c84005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:02.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.545+0000 7f3c95ffb700 1 --2- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 0x7f3c84008650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:02.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.545+0000 7f3c95ffb700 1 -- 172.21.15.27:0/698657770 --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c980cef20 con 0x7f3c84003710 2024-01-11T08:10:02.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.545+0000 7f3c95ffb700 1 -- 172.21.15.27:0/698657770 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c84008d70 con 0x7f3c840061c0 2024-01-11T08:10:02.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.545+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 0x7f3c84005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:02.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.545+0000 7f3c977fe700 1 --2- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 0x7f3c84008650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:02.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.545+0000 7f3c97fff700 1 -- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 msgr2=0x7f3c84005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-11T08:10:02.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.546+0000 7f3c97fff700 1 -- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 msgr2=0x7f3c84005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-11T08:10:02.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.546+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 0x7f3c84005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-01-11T08:10:02.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.546+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 0x7f3c84005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:10:02.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.546+0000 7f3c977fe700 1 -- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 msgr2=0x7f3c84008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-11T08:10:02.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.546+0000 7f3c977fe700 1 -- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 msgr2=0x7f3c84008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-11T08:10:02.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.546+0000 7f3c977fe700 1 --2- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 0x7f3c84008650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-01-11T08:10:02.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.546+0000 7f3c977fe700 1 --2- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 0x7f3c84008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:10:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.747+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 0x7f3c84005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.747+0000 7f3c97fff700 1 -- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 msgr2=0x7f3c84005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-11T08:10:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.747+0000 7f3c97fff700 1 -- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 msgr2=0x7f3c84005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-11T08:10:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.747+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 0x7f3c84005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-01-11T08:10:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.747+0000 7f3c977fe700 1 --2- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 0x7f3c84008650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.747+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 0x7f3c84005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:10:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.747+0000 7f3c977fe700 1 -- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 msgr2=0x7f3c84008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-11T08:10:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.747+0000 7f3c977fe700 1 -- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 msgr2=0x7f3c84008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-11T08:10:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.747+0000 7f3c977fe700 1 --2- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 0x7f3c84008650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-01-11T08:10:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:02.747+0000 7f3c977fe700 1 --2- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 0x7f3c84008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:10:03.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.148+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 0x7f3c84005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:03.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.148+0000 7f3c97fff700 1 -- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 msgr2=0x7f3c84005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-11T08:10:03.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.148+0000 7f3c97fff700 1 -- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 msgr2=0x7f3c84005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-11T08:10:03.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.148+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 0x7f3c84005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-01-11T08:10:03.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.148+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 0x7f3c84005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-11T08:10:03.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.148+0000 7f3c977fe700 1 --2- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 0x7f3c84008650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:03.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.149+0000 7f3c977fe700 1 -- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 msgr2=0x7f3c84008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-11T08:10:03.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.149+0000 7f3c977fe700 1 -- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 msgr2=0x7f3c84008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-11T08:10:03.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.149+0000 7f3c977fe700 1 --2- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 0x7f3c84008650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-01-11T08:10:03.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.149+0000 7f3c977fe700 1 --2- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 0x7f3c84008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-11T08:10:03.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.950+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 0x7f3c84005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:03.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.950+0000 7f3c97fff700 1 -- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 msgr2=0x7f3c84005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-11T08:10:03.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.950+0000 7f3c97fff700 1 -- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 msgr2=0x7f3c84005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-11T08:10:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.950+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 0x7f3c84005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-01-11T08:10:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.950+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 0x7f3c84005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-11T08:10:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.950+0000 7f3c977fe700 1 --2- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 0x7f3c84008650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.951+0000 7f3c977fe700 1 -- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 msgr2=0x7f3c84008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-11T08:10:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.951+0000 7f3c977fe700 1 -- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 msgr2=0x7f3c84008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-11T08:10:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.951+0000 7f3c977fe700 1 --2- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 0x7f3c84008650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-01-11T08:10:03.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:03.951+0000 7f3c977fe700 1 --2- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 0x7f3c84008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-11T08:10:04.856 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:04.857+0000 7f78c72551c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:10:04.869 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:04.869+0000 7f62ffb111c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:10:05.062 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:05.063+0000 7f78c72551c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:10:05.085 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:05.086+0000 7f62ffb111c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:10:05.456 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:05.457+0000 7f78c72551c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:10:05.487 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:05.487+0000 7f62ffb111c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:10:05.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.545+0000 7f3c95ffb700 1 -- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 msgr2=0x7f3c84008650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-11T08:10:05.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.545+0000 7f3c95ffb700 1 --2- 172.21.15.27:0/698657770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c840061c0 0x7f3c84008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:05.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.545+0000 7f3c95ffb700 1 -- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 msgr2=0x7f3c84005b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-11T08:10:05.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.545+0000 7f3c95ffb700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84003710 0x7f3c84005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:05.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.545+0000 7f3c9c99c700 1 -- 172.21.15.27:0/698657770 reap_dead start 2024-01-11T08:10:05.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.545+0000 7f3c95ffb700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84009170 0x7f3c8400b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:05.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.545+0000 7f3c95ffb700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c84007d90 0x7f3c84008210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:05.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.545+0000 7f3c95ffb700 1 -- 172.21.15.27:0/698657770 --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c980cf200 con 0x7f3c84003ba0 2024-01-11T08:10:05.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.545+0000 7f3c95ffb700 1 -- 172.21.15.27:0/698657770 --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c980cef20 con 0x7f3c84009170 2024-01-11T08:10:05.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.545+0000 7f3c95ffb700 1 -- 172.21.15.27:0/698657770 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c84008d70 con 0x7f3c84007d90 2024-01-11T08:10:05.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.546+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.27:3300/0 conn(0x7f3c84009170 0x7f3c8400b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:05.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.546+0000 7f3c9c99c700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c84007d90 0x7f3c84008210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:05.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.546+0000 7f3c96ffd700 1 -- 172.21.15.27:0/698657770 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235563517 0 0) 0x7f3c980cf200 con 0x7f3c84003ba0 2024-01-11T08:10:05.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.546+0000 7f3c96ffd700 1 -- 172.21.15.27:0/698657770 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c80003570 con 0x7f3c84003ba0 2024-01-11T08:10:05.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.546+0000 7f3c97fff700 1 -- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c84007d90 msgr2=0x7f3c84008210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:05.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.546+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c84007d90 0x7f3c84008210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:05.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.546+0000 7f3c97fff700 1 -- 172.21.15.27:0/698657770 >> v1:172.21.15.27:6789/0 conn(0x7f3c84003ba0 legacy=0x7f3c84006640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:05.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.546+0000 7f3c97fff700 1 -- 172.21.15.27:0/698657770 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c840089c0 con 0x7f3c84009170 2024-01-11T08:10:05.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.547+0000 7f3c97fff700 1 --2- 172.21.15.27:0/698657770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c84009170 0x7f3c8400b550 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3c84007d60 tx=0x7f3c88008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b0b5b2b20f76a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:05.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.547+0000 7f3c96ffd700 1 -- 172.21.15.27:0/698657770 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8801e070 con 0x7f3c84009170 2024-01-11T08:10:05.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.547+0000 7f3c96ffd700 1 -- 172.21.15.27:0/698657770 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c88019710 con 0x7f3c84009170 2024-01-11T08:10:05.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.547+0000 7f3c96ffd700 1 -- 172.21.15.27:0/698657770 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c88019ab0 con 0x7f3c84009170 2024-01-11T08:10:05.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.548+0000 7f3c9e3ff700 1 -- 172.21.15.27:0/698657770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c84009170 msgr2=0x7f3c8400b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:05.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.548+0000 7f3c9e3ff700 1 --2- 172.21.15.27:0/698657770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c84009170 0x7f3c8400b550 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3c84007d60 tx=0x7f3c88008040 comp rx=0 tx=0).stop 2024-01-11T08:10:05.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.548+0000 7f3c9e3ff700 1 -- 172.21.15.27:0/698657770 shutdown_connections 2024-01-11T08:10:05.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.548+0000 7f3c9e3ff700 1 --2- 172.21.15.27:0/698657770 >> v2:172.21.15.77:3300/0 conn(0x7f3c84007d90 0x7f3c84008210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:05.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.548+0000 7f3c9e3ff700 1 --2- 172.21.15.27:0/698657770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c84009170 0x7f3c8400b550 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:05.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.548+0000 7f3c9e3ff700 1 -- 172.21.15.27:0/698657770 >> 172.21.15.27:0/698657770 conn(0x7f3c9806ed40 msgr2=0x7f3c9806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:05.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.548+0000 7f3c9e3ff700 1 -- 172.21.15.27:0/698657770 shutdown_connections 2024-01-11T08:10:05.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.548+0000 7f3c9e3ff700 1 -- 172.21.15.27:0/698657770 wait complete. 2024-01-11T08:10:05.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.550+0000 7f3c9e3ff700 1 Processor -- start 2024-01-11T08:10:05.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.550+0000 7f3c9e3ff700 1 -- start start 2024-01-11T08:10:05.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.551+0000 7f3c9e3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c84003ba0 0x7f3c98156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:05.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.551+0000 7f3c9e3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c84007d90 0x7f3c98155970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:05.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.551+0000 7f3c9e3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c84009170 0x7f3c98157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:05.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.552+0000 7f3c9e3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9816e110 con 0x7f3c84009170 2024-01-11T08:10:05.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.552+0000 7f3c9e3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c980b4610 con 0x7f3c84003ba0 2024-01-11T08:10:05.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.552+0000 7f3c9e3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c980ca210 con 0x7f3c84007d90 2024-01-11T08:10:05.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.552+0000 7f3c9c99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c84009170 0x7f3c98157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:05.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.552+0000 7f3c9c99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c84009170 0x7f3c98157c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57256/0 (socket says 172.21.15.27:57256) 2024-01-11T08:10:05.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.552+0000 7f3c9c99c700 1 -- 172.21.15.27:0/3397414122 learned_addr learned my addr 172.21.15.27:0/3397414122 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:05.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.552+0000 7f3c97fff700 1 --2- 172.21.15.27:0/3397414122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c84003ba0 0x7f3c98156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:05.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.552+0000 7f3c977fe700 1 --2- 172.21.15.27:0/3397414122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c84007d90 0x7f3c98155970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:05.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.552+0000 7f3c9c99c700 1 -- 172.21.15.27:0/3397414122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c84007d90 msgr2=0x7f3c98155970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:05.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.552+0000 7f3c9c99c700 1 --2- 172.21.15.27:0/3397414122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c84007d90 0x7f3c98155970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.552+0000 7f3c9c99c700 1 -- 172.21.15.27:0/3397414122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c84003ba0 msgr2=0x7f3c98156210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.552+0000 7f3c9c99c700 1 --2- 172.21.15.27:0/3397414122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c84003ba0 0x7f3c98156210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.552+0000 7f3c9c99c700 1 -- 172.21.15.27:0/3397414122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8800e040 con 0x7f3c84009170 2024-01-11T08:10:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.552+0000 7f3c9c99c700 1 --2- 172.21.15.27:0/3397414122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c84009170 0x7f3c98157c20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3c98089840 tx=0x7f3c90001a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.553+0000 7f3c7ffff700 1 -- 172.21.15.27:0/3397414122 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c90018030 con 0x7f3c84009170 2024-01-11T08:10:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.553+0000 7f3c7ffff700 1 -- 172.21.15.27:0/3397414122 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9000f970 con 0x7f3c84009170 2024-01-11T08:10:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.553+0000 7f3c9e3ff700 1 -- 172.21.15.27:0/3397414122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c980ca4b0 con 0x7f3c84009170 2024-01-11T08:10:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.553+0000 7f3c7ffff700 1 -- 172.21.15.27:0/3397414122 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9000fcf0 con 0x7f3c84009170 2024-01-11T08:10:05.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.553+0000 7f3c9e3ff700 1 -- 172.21.15.27:0/3397414122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c980c8010 con 0x7f3c84009170 2024-01-11T08:10:05.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.553+0000 7f3c7ffff700 1 -- 172.21.15.27:0/3397414122 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f3c9001b070 con 0x7f3c84009170 2024-01-11T08:10:05.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.553+0000 7f3c7ffff700 1 -- 172.21.15.27:0/3397414122 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f3c90019070 con 0x7f3c84009170 2024-01-11T08:10:05.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.554+0000 7f3c9e3ff700 1 -- 172.21.15.27:0/3397414122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c980b47a0 con 0x7f3c84009170 2024-01-11T08:10:05.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.557+0000 7f3c7ffff700 1 -- 172.21.15.27:0/3397414122 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f3c90013ae0 con 0x7f3c84009170 2024-01-11T08:10:05.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.636+0000 7f3c9e3ff700 1 -- 172.21.15.27:0/3397414122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f3c9804c3c0 con 0x7f3c84009170 2024-01-11T08:10:05.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.642+0000 7f3c7ffff700 1 -- 172.21.15.27:0/3397414122 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f3c90013c80 con 0x7f3c84009170 2024-01-11T08:10:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.643+0000 7f3c9e3ff700 1 -- 172.21.15.27:0/3397414122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c84009170 msgr2=0x7f3c98157c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.643+0000 7f3c9e3ff700 1 --2- 172.21.15.27:0/3397414122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c84009170 0x7f3c98157c20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3c98089840 tx=0x7f3c90001a90 comp rx=0 tx=0).stop 2024-01-11T08:10:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.643+0000 7f3c9e3ff700 1 -- 172.21.15.27:0/3397414122 shutdown_connections 2024-01-11T08:10:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.643+0000 7f3c9e3ff700 1 --2- 172.21.15.27:0/3397414122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c84009170 0x7f3c98157c20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.643+0000 7f3c9e3ff700 1 --2- 172.21.15.27:0/3397414122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c84003ba0 0x7f3c98156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.643+0000 7f3c9e3ff700 1 --2- 172.21.15.27:0/3397414122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c84007d90 0x7f3c98155970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.643+0000 7f3c9e3ff700 1 -- 172.21.15.27:0/3397414122 >> 172.21.15.27:0/3397414122 conn(0x7f3c9806ed40 msgr2=0x7f3c980c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.643+0000 7f3c9e3ff700 1 -- 172.21.15.27:0/3397414122 shutdown_connections 2024-01-11T08:10:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:05.643+0000 7f3c9e3ff700 1 -- 172.21.15.27:0/3397414122 wait complete. 2024-01-11T08:10:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2024-01-11T08:10:05.660 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-11T08:10:05.661 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:10:05.661 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-11T08:10:05.734 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:10:05.734 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-11T08:10:05.842 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:10:05.842 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-11T08:10:05.916 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-11T08:10:05.917 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-01-11T08:10:05.989 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:10:05.989 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-01-11T08:10:06.060 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:10:06.060 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-01-11T08:10:06.080 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:06.081+0000 7f78c72551c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:10:06.152 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:10:06.152 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-01-11T08:10:06.167 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:06.168+0000 7f62ffb111c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:10:06.261 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-11T08:10:06.261 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-01-11T08:10:06.266 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:06.267+0000 7f78c72551c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:10:06.338 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd new 21720a56-9ef9-4541-acb7-dcb4fcfbd10d 0 2024-01-11T08:10:06.346 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:06.346+0000 7f62ffb111c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:10:06.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.538+0000 7f30d55ff700 1 Processor -- start 2024-01-11T08:10:06.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.538+0000 7f30d55ff700 1 -- start start 2024-01-11T08:10:06.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.541+0000 7f30d55ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f30d0142dc0 0x7f30d00cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:06.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.541+0000 7f30d55ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f30d0143190 0x7f30d00cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:06.541 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.541+0000 7f30d55ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d015d320 con 0x7f30d00cc020 2024-01-11T08:10:06.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.541+0000 7f30d55ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30d015df30 con 0x7f30d0142dc0 2024-01-11T08:10:06.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.541+0000 7f30d55ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30d015d590 con 0x7f30d0143190 2024-01-11T08:10:06.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.541+0000 7f30ce7fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f30d0143190 0x7f30d00cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:06.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.541+0000 7f30ce7fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f30d0143190 0x7f30d00cb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56668/0 (socket says 172.21.15.77:56668) 2024-01-11T08:10:06.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.541+0000 7f30cf7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f30d00cc020 0x7f30d00cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54194/0 (socket says 172.21.15.77:54194) 2024-01-11T08:10:06.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.541+0000 7f30ce7fc700 1 -- 172.21.15.77:0/1384161326 learned_addr learned my addr 172.21.15.77:0/1384161326 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-11T08:10:06.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.541+0000 7f30ceffd700 1 --2- 172.21.15.77:0/1384161326 >> v2:172.21.15.27:3300/0 conn(0x7f30d0142dc0 0x7f30d00cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:06.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.541+0000 7f30ce7fc700 1 -- 172.21.15.77:0/1384161326 >> v2:172.21.15.27:3300/0 conn(0x7f30d0142dc0 msgr2=0x7f30d00cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:06.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.541+0000 7f30ce7fc700 1 --2- 172.21.15.77:0/1384161326 >> v2:172.21.15.27:3300/0 conn(0x7f30d0142dc0 0x7f30d00cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:06.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.541+0000 7f30ce7fc700 1 -- 172.21.15.77:0/1384161326 >> v1:172.21.15.77:6789/0 conn(0x7f30d00cc020 legacy=0x7f30d00cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:06.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.542+0000 7f30ce7fc700 1 -- 172.21.15.77:0/1384161326 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d015d870 con 0x7f30d0143190 2024-01-11T08:10:06.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.542+0000 7f30cdffb700 1 -- 172.21.15.77:0/1384161326 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1613061611 0 0) 0x7f30d015d320 con 0x7f30d00cc020 2024-01-11T08:10:06.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.542+0000 7f30ce7fc700 1 --2- 172.21.15.77:0/1384161326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30d0143190 0x7f30d00cb2e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f30d006c480 tx=0x7f30c4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7639a2d5ab250e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:06.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.542+0000 7f30cdffb700 1 -- 172.21.15.77:0/1384161326 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c4009e70 con 0x7f30d0143190 2024-01-11T08:10:06.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.542+0000 7f30cdffb700 1 -- 172.21.15.77:0/1384161326 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c400b860 con 0x7f30d0143190 2024-01-11T08:10:06.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.542+0000 7f30cdffb700 1 -- 172.21.15.77:0/1384161326 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c400bc00 con 0x7f30d0143190 2024-01-11T08:10:06.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.543+0000 7f30d55ff700 1 -- 172.21.15.77:0/1384161326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30d0143190 msgr2=0x7f30d00cb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:06.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.543+0000 7f30d55ff700 1 --2- 172.21.15.77:0/1384161326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30d0143190 0x7f30d00cb2e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f30d006c480 tx=0x7f30c4009250 comp rx=0 tx=0).stop 2024-01-11T08:10:06.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.543+0000 7f30d55ff700 1 -- 172.21.15.77:0/1384161326 shutdown_connections 2024-01-11T08:10:06.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.543+0000 7f30d55ff700 1 --2- 172.21.15.77:0/1384161326 >> v2:172.21.15.27:3300/0 conn(0x7f30d0142dc0 0x7f30d00cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:06.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.543+0000 7f30d55ff700 1 --2- 172.21.15.77:0/1384161326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30d0143190 0x7f30d00cb2e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:06.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.543+0000 7f30d55ff700 1 -- 172.21.15.77:0/1384161326 >> 172.21.15.77:0/1384161326 conn(0x7f30d0063ca0 msgr2=0x7f30d006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:06.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.543+0000 7f30d55ff700 1 -- 172.21.15.77:0/1384161326 shutdown_connections 2024-01-11T08:10:06.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.543+0000 7f30d55ff700 1 -- 172.21.15.77:0/1384161326 wait complete. 2024-01-11T08:10:06.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.544+0000 7f30d55ff700 1 Processor -- start 2024-01-11T08:10:06.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.544+0000 7f30d55ff700 1 -- start start 2024-01-11T08:10:06.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.545+0000 7f30d55ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f30d00cc020 0x7f30d00c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:06.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.545+0000 7f30d55ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30d0142dc0 0x7f30d00c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:06.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.545+0000 7f30d55ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30d0143190 0x7f30d00c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:06.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.545+0000 7f30d55ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d0142260 con 0x7f30d0143190 2024-01-11T08:10:06.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.545+0000 7f30d55ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d00b2be0 con 0x7f30d0142dc0 2024-01-11T08:10:06.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.545+0000 7f30d55ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d00b22b0 con 0x7f30d00cc020 2024-01-11T08:10:06.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.545+0000 7f30ceffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f30d00cc020 0x7f30d00c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:06.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.545+0000 7f30ce7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30d0142dc0 0x7f30d00c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:06.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.546+0000 7f30ce7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30d0142dc0 0x7f30d00c5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56684/0 (socket says 172.21.15.77:56684) 2024-01-11T08:10:06.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.546+0000 7f30ceffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f30d00cc020 0x7f30d00c9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.77:54042/0 (socket says 172.21.15.77:54042) 2024-01-11T08:10:06.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.546+0000 7f30ce7fc700 1 -- 172.21.15.77:0/265557648 learned_addr learned my addr 172.21.15.77:0/265557648 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-11T08:10:06.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.546+0000 7f30cf7fe700 1 --2- 172.21.15.77:0/265557648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30d0143190 0x7f30d00c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:06.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.546+0000 7f30ce7fc700 1 -- 172.21.15.77:0/265557648 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f30d00cc020 msgr2=0x7f30d00c9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:06.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.546+0000 7f30ce7fc700 1 --2- 172.21.15.77:0/265557648 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f30d00cc020 0x7f30d00c9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:06.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.546+0000 7f30ce7fc700 1 -- 172.21.15.77:0/265557648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30d0143190 msgr2=0x7f30d00c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:06.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.546+0000 7f30ce7fc700 1 --2- 172.21.15.77:0/265557648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30d0143190 0x7f30d00c59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:06.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.546+0000 7f30ce7fc700 1 -- 172.21.15.77:0/265557648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c40094f0 con 0x7f30d0142dc0 2024-01-11T08:10:06.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.546+0000 7f30ce7fc700 1 --2- 172.21.15.77:0/265557648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30d0142dc0 0x7f30d00c5490 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f30c40094c0 tx=0x7f30c4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:06.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.546+0000 7f30b6ffd700 1 -- 172.21.15.77:0/265557648 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c4000c90 con 0x7f30d0142dc0 2024-01-11T08:10:06.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.546+0000 7f30b6ffd700 1 -- 172.21.15.77:0/265557648 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c400b9d0 con 0x7f30d0142dc0 2024-01-11T08:10:06.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.546+0000 7f30b6ffd700 1 -- 172.21.15.77:0/265557648 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c4025420 con 0x7f30d0142dc0 2024-01-11T08:10:06.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.546+0000 7f30d55ff700 1 -- 172.21.15.77:0/265557648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d00b2590 con 0x7f30d0142dc0 2024-01-11T08:10:06.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.547+0000 7f30d55ff700 1 -- 172.21.15.77:0/265557648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d00ca350 con 0x7f30d0142dc0 2024-01-11T08:10:06.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.547+0000 7f30b6ffd700 1 -- 172.21.15.77:0/265557648 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f30c4023070 con 0x7f30d0142dc0 2024-01-11T08:10:06.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.547+0000 7f30b6ffd700 1 -- 172.21.15.77:0/265557648 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f30c402f440 con 0x7f30d0142dc0 2024-01-11T08:10:06.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.548+0000 7f30d55ff700 1 -- 172.21.15.77:0/265557648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30d00b2d70 con 0x7f30d0142dc0 2024-01-11T08:10:06.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.552+0000 7f30b6ffd700 1 -- 172.21.15.77:0/265557648 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f30c402a070 con 0x7f30d0142dc0 2024-01-11T08:10:06.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.637+0000 7f30d55ff700 1 -- 172.21.15.77:0/265557648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd new", "uuid": "21720a56-9ef9-4541-acb7-dcb4fcfbd10d", "id": 0} v 0) v1 -- 0x7f30d004c3c0 con 0x7f30d0142dc0 2024-01-11T08:10:06.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.643+0000 7f30b6ffd700 1 -- 172.21.15.77:0/265557648 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "21720a56-9ef9-4541-acb7-dcb4fcfbd10d", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f30c4015020 con 0x7f30d0142dc0 2024-01-11T08:10:06.643 INFO:teuthology.orchestra.run.smithi077.stdout:0 2024-01-11T08:10:06.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.644+0000 7f30d55ff700 1 -- 172.21.15.77:0/265557648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30d0142dc0 msgr2=0x7f30d00c5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:06.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.644+0000 7f30d55ff700 1 --2- 172.21.15.77:0/265557648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30d0142dc0 0x7f30d00c5490 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f30c40094c0 tx=0x7f30c4017000 comp rx=0 tx=0).stop 2024-01-11T08:10:06.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.644+0000 7f30d55ff700 1 -- 172.21.15.77:0/265557648 shutdown_connections 2024-01-11T08:10:06.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.644+0000 7f30d55ff700 1 --2- 172.21.15.77:0/265557648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30d0143190 0x7f30d00c59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:06.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.644+0000 7f30d55ff700 1 --2- 172.21.15.77:0/265557648 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f30d00cc020 0x7f30d00c9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:06.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.644+0000 7f30d55ff700 1 --2- 172.21.15.77:0/265557648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30d0142dc0 0x7f30d00c5490 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:06.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.644+0000 7f30d55ff700 1 -- 172.21.15.77:0/265557648 >> 172.21.15.77:0/265557648 conn(0x7f30d0063ca0 msgr2=0x7f30d0068720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:06.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.645+0000 7f30d55ff700 1 -- 172.21.15.77:0/265557648 shutdown_connections 2024-01-11T08:10:06.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.645+0000 7f30d55ff700 1 -- 172.21.15.77:0/265557648 wait complete. 2024-01-11T08:10:06.661 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd new 2de9c7c8-0153-4483-a1c5-0a70f8d6660b 1 2024-01-11T08:10:06.725 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:06.725+0000 7f78c72551c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:10:06.795 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:06.795+0000 7f62ffb111c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:10:06.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.853+0000 7fef6fdff700 1 Processor -- start 2024-01-11T08:10:06.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.853+0000 7fef6fdff700 1 -- start start 2024-01-11T08:10:06.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.855+0000 7fef6fdff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fef680cb4b0 0x7fef680cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.855+0000 7fef6fdff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef680ccc70 con 0x7fef68143190 2024-01-11T08:10:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.855+0000 7fef6fdff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef6815d320 con 0x7fef68142dc0 2024-01-11T08:10:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.855+0000 7fef6fdff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef6815df30 con 0x7fef680cb4b0 2024-01-11T08:10:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.855+0000 7fef6e39c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fef680cb4b0 0x7fef680cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.855+0000 7fef6db9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fef68142dc0 0x7fef680cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54200/0 (socket says 172.21.15.77:54200) 2024-01-11T08:10:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.855+0000 7fef6db9b700 1 -- 172.21.15.77:0/2131818150 learned_addr learned my addr 172.21.15.77:0/2131818150 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-11T08:10:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.855+0000 7fef6cb99700 1 -- 172.21.15.77:0/2131818150 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250566910 0 0) 0x7fef6815d320 con 0x7fef68142dc0 2024-01-11T08:10:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.856+0000 7fef6cb99700 1 -- 172.21.15.77:0/2131818150 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef54003570 con 0x7fef68142dc0 2024-01-11T08:10:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.856+0000 7fef6e39c700 1 -- 172.21.15.77:0/2131818150 >> v1:172.21.15.77:6789/0 conn(0x7fef68142dc0 legacy=0x7fef680cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.856+0000 7fef6cb99700 1 -- 172.21.15.77:0/2131818150 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865680189 0 0) 0x7fef680ccc70 con 0x7fef68143190 2024-01-11T08:10:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.856+0000 7fef6e39c700 1 -- 172.21.15.77:0/2131818150 >> v1:172.21.15.27:6789/0 conn(0x7fef68143190 legacy=0x7fef680cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.856+0000 7fef6e39c700 1 -- 172.21.15.77:0/2131818150 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef6815d590 con 0x7fef680cb4b0 2024-01-11T08:10:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.856+0000 7fef6e39c700 1 --2- 172.21.15.77:0/2131818150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef680cb4b0 0x7fef680cc730 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fef60009e60 tx=0x7fef60009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79a95840531e492f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.856+0000 7fef6cb99700 1 -- 172.21.15.77:0/2131818150 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6000e070 con 0x7fef680cb4b0 2024-01-11T08:10:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.856+0000 7fef6cb99700 1 -- 172.21.15.77:0/2131818150 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6000bb60 con 0x7fef680cb4b0 2024-01-11T08:10:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.856+0000 7fef6cb99700 1 -- 172.21.15.77:0/2131818150 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef60012430 con 0x7fef680cb4b0 2024-01-11T08:10:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.857+0000 7fef6fdff700 1 -- 172.21.15.77:0/2131818150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef680cb4b0 msgr2=0x7fef680cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.857+0000 7fef6fdff700 1 --2- 172.21.15.77:0/2131818150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef680cb4b0 0x7fef680cc730 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fef60009e60 tx=0x7fef60009780 comp rx=0 tx=0).stop 2024-01-11T08:10:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.857+0000 7fef6fdff700 1 -- 172.21.15.77:0/2131818150 shutdown_connections 2024-01-11T08:10:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.857+0000 7fef6fdff700 1 --2- 172.21.15.77:0/2131818150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef680cb4b0 0x7fef680cc730 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.857+0000 7fef6fdff700 1 -- 172.21.15.77:0/2131818150 >> 172.21.15.77:0/2131818150 conn(0x7fef68063ca0 msgr2=0x7fef6806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.857+0000 7fef6fdff700 1 -- 172.21.15.77:0/2131818150 shutdown_connections 2024-01-11T08:10:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.857+0000 7fef6fdff700 1 -- 172.21.15.77:0/2131818150 wait complete. 2024-01-11T08:10:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.858+0000 7fef6fdff700 1 Processor -- start 2024-01-11T08:10:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.859+0000 7fef6fdff700 1 -- start start 2024-01-11T08:10:06.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.859+0000 7fef6fdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef680cb4b0 0x7fef680c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:06.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.859+0000 7fef6fdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef68142dc0 0x7fef68140b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:06.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.859+0000 7fef6fdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef68143190 0x7fef6815bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:06.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.859+0000 7fef6fdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef680b2070 con 0x7fef68143190 2024-01-11T08:10:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.859+0000 7fef6fdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef680c7010 con 0x7fef68142dc0 2024-01-11T08:10:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.859+0000 7fef6fdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef680c6750 con 0x7fef680cb4b0 2024-01-11T08:10:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.860+0000 7fef6db9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef680cb4b0 0x7fef680c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.860+0000 7fef6d39a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef68142dc0 0x7fef68140b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.860+0000 7fef6db9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef680cb4b0 0x7fef680c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.77:54058/0 (socket says 172.21.15.77:54058) 2024-01-11T08:10:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.860+0000 7fef6db9b700 1 -- 172.21.15.77:0/1627556046 learned_addr learned my addr 172.21.15.77:0/1627556046 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-11T08:10:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.860+0000 7fef6e39c700 1 --2- 172.21.15.77:0/1627556046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef68143190 0x7fef6815bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.860+0000 7fef6db9b700 1 -- 172.21.15.77:0/1627556046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef68142dc0 msgr2=0x7fef68140b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.860+0000 7fef6db9b700 1 --2- 172.21.15.77:0/1627556046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef68142dc0 0x7fef68140b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.860+0000 7fef6db9b700 1 -- 172.21.15.77:0/1627556046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef68143190 msgr2=0x7fef6815bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.860+0000 7fef6db9b700 1 --2- 172.21.15.77:0/1627556046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef68143190 0x7fef6815bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.860+0000 7fef6db9b700 1 -- 172.21.15.77:0/1627556046 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef60009a20 con 0x7fef680cb4b0 2024-01-11T08:10:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.860+0000 7fef6d39a700 1 --2- 172.21.15.77:0/1627556046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef68142dc0 0x7fef68140b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.860+0000 7fef6db9b700 1 --2- 172.21.15.77:0/1627556046 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef680cb4b0 0x7fef680c74a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fef6400c7f0 tx=0x7fef64007d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:06.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.861+0000 7fef5dffb700 1 -- 172.21.15.77:0/1627556046 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef64005c00 con 0x7fef680cb4b0 2024-01-11T08:10:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.861+0000 7fef5dffb700 1 -- 172.21.15.77:0/1627556046 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef64005d60 con 0x7fef680cb4b0 2024-01-11T08:10:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.861+0000 7fef5dffb700 1 -- 172.21.15.77:0/1627556046 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef64007300 con 0x7fef680cb4b0 2024-01-11T08:10:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.861+0000 7fef6fdff700 1 -- 172.21.15.77:0/1627556046 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef680c69c0 con 0x7fef680cb4b0 2024-01-11T08:10:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.861+0000 7fef6fdff700 1 -- 172.21.15.77:0/1627556046 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef680b4020 con 0x7fef680cb4b0 2024-01-11T08:10:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.861+0000 7fef5dffb700 1 -- 172.21.15.77:0/1627556046 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fef64012400 con 0x7fef680cb4b0 2024-01-11T08:10:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.862+0000 7fef5dffb700 1 -- 172.21.15.77:0/1627556046 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7fef64012a70 con 0x7fef680cb4b0 2024-01-11T08:10:06.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.862+0000 7fef6fdff700 1 -- 172.21.15.77:0/1627556046 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef680c72f0 con 0x7fef680cb4b0 2024-01-11T08:10:06.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.865+0000 7fef5dffb700 1 -- 172.21.15.77:0/1627556046 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fef64021060 con 0x7fef680cb4b0 2024-01-11T08:10:06.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.951+0000 7fef6fdff700 1 -- 172.21.15.77:0/1627556046 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2de9c7c8-0153-4483-a1c5-0a70f8d6660b", "id": 1} v 0) v1 -- 0x7fef6804c3c0 con 0x7fef680cb4b0 2024-01-11T08:10:06.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.958+0000 7fef5dffb700 1 -- 172.21.15.77:0/1627556046 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2de9c7c8-0153-4483-a1c5-0a70f8d6660b", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fef64017070 con 0x7fef680cb4b0 2024-01-11T08:10:06.958 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-01-11T08:10:06.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.960+0000 7fef6fdff700 1 -- 172.21.15.77:0/1627556046 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef680cb4b0 msgr2=0x7fef680c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:06.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.960+0000 7fef6fdff700 1 --2- 172.21.15.77:0/1627556046 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef680cb4b0 0x7fef680c74a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fef6400c7f0 tx=0x7fef64007d90 comp rx=0 tx=0).stop 2024-01-11T08:10:06.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.960+0000 7fef6fdff700 1 -- 172.21.15.77:0/1627556046 shutdown_connections 2024-01-11T08:10:06.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.960+0000 7fef6fdff700 1 --2- 172.21.15.77:0/1627556046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef68143190 0x7fef6815bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:06.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.960+0000 7fef6fdff700 1 --2- 172.21.15.77:0/1627556046 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef680cb4b0 0x7fef680c74a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:06.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.960+0000 7fef6fdff700 1 --2- 172.21.15.77:0/1627556046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef68142dc0 0x7fef68140b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:06.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.960+0000 7fef6fdff700 1 -- 172.21.15.77:0/1627556046 >> 172.21.15.77:0/1627556046 conn(0x7fef68063ca0 msgr2=0x7fef680697f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:06.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.960+0000 7fef6fdff700 1 -- 172.21.15.77:0/1627556046 shutdown_connections 2024-01-11T08:10:06.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:06.960+0000 7fef6fdff700 1 -- 172.21.15.77:0/1627556046 wait complete. 2024-01-11T08:10:06.976 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd new bb18c833-73e9-4afd-96a2-103eb308745b 2 2024-01-11T08:10:07.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.162+0000 7ff679dff700 1 Processor -- start 2024-01-11T08:10:07.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.162+0000 7ff679dff700 1 -- start start 2024-01-11T08:10:07.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.164+0000 7ff679dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff6740ccd70 0x7ff6740cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:07.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.164+0000 7ff679dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff67415ae70 0x7ff67415b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:07.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.164+0000 7ff679dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6740cf850 con 0x7ff67415a1b0 2024-01-11T08:10:07.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.164+0000 7ff679dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff67415ac90 con 0x7ff67415ae70 2024-01-11T08:10:07.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.164+0000 7ff679dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6740cec40 con 0x7ff6740ccd70 2024-01-11T08:10:07.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.164+0000 7ff672ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff6740ccd70 0x7ff6740cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:07.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.164+0000 7ff672ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff6740ccd70 0x7ff6740cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56712/0 (socket says 172.21.15.77:56712) 2024-01-11T08:10:07.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.164+0000 7ff672ffd700 1 -- 172.21.15.77:0/1426289504 learned_addr learned my addr 172.21.15.77:0/1426289504 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-11T08:10:07.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.164+0000 7ff673fff700 1 --2- 172.21.15.77:0/1426289504 >> v2:172.21.15.27:3300/0 conn(0x7ff67415ae70 0x7ff67415b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:07.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.164+0000 7ff672ffd700 1 -- 172.21.15.77:0/1426289504 >> v2:172.21.15.27:3300/0 conn(0x7ff67415ae70 msgr2=0x7ff67415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:07.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.164+0000 7ff672ffd700 1 --2- 172.21.15.77:0/1426289504 >> v2:172.21.15.27:3300/0 conn(0x7ff67415ae70 0x7ff67415b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.164+0000 7ff672ffd700 1 -- 172.21.15.77:0/1426289504 >> v1:172.21.15.27:6789/0 conn(0x7ff67415a1b0 legacy=0x7ff67415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:07.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.164+0000 7ff672ffd700 1 -- 172.21.15.77:0/1426289504 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6740cef20 con 0x7ff6740ccd70 2024-01-11T08:10:07.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.165+0000 7ff672ffd700 1 --2- 172.21.15.77:0/1426289504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6740ccd70 0x7ff6740cd210 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff6740696f0 tx=0x7ff668009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e26a199bbde4e29 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:07.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.165+0000 7ff6727fc700 1 -- 172.21.15.77:0/1426289504 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66800b860 con 0x7ff6740ccd70 2024-01-11T08:10:07.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.165+0000 7ff6727fc700 1 -- 172.21.15.77:0/1426289504 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff66800b9c0 con 0x7ff6740ccd70 2024-01-11T08:10:07.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.165+0000 7ff6727fc700 1 -- 172.21.15.77:0/1426289504 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66800bd60 con 0x7ff6740ccd70 2024-01-11T08:10:07.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.165+0000 7ff679dff700 1 -- 172.21.15.77:0/1426289504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6740ccd70 msgr2=0x7ff6740cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:07.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.165+0000 7ff679dff700 1 --2- 172.21.15.77:0/1426289504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6740ccd70 0x7ff6740cd210 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff6740696f0 tx=0x7ff668009250 comp rx=0 tx=0).stop 2024-01-11T08:10:07.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.165+0000 7ff679dff700 1 -- 172.21.15.77:0/1426289504 shutdown_connections 2024-01-11T08:10:07.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.165+0000 7ff679dff700 1 --2- 172.21.15.77:0/1426289504 >> v2:172.21.15.27:3300/0 conn(0x7ff67415ae70 0x7ff67415b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.165+0000 7ff679dff700 1 --2- 172.21.15.77:0/1426289504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6740ccd70 0x7ff6740cd210 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.165+0000 7ff679dff700 1 -- 172.21.15.77:0/1426289504 >> 172.21.15.77:0/1426289504 conn(0x7ff67406ed40 msgr2=0x7ff67406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:07.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.166+0000 7ff679dff700 1 -- 172.21.15.77:0/1426289504 shutdown_connections 2024-01-11T08:10:07.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.166+0000 7ff679dff700 1 -- 172.21.15.77:0/1426289504 wait complete. 2024-01-11T08:10:07.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.167+0000 7ff679dff700 1 Processor -- start 2024-01-11T08:10:07.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.167+0000 7ff679dff700 1 -- start start 2024-01-11T08:10:07.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.168+0000 7ff679dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff6740ccd70 0x7ff6740b5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:07.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.168+0000 7ff679dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff67415a1b0 0x7ff6740ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:07.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.168+0000 7ff679dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67415ae70 0x7ff6741692d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:07.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.168+0000 7ff679dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6740a7240 con 0x7ff67415ae70 2024-01-11T08:10:07.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.168+0000 7ff679dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67416a610 con 0x7ff67415a1b0 2024-01-11T08:10:07.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.168+0000 7ff679dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6741699a0 con 0x7ff6740ccd70 2024-01-11T08:10:07.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.168+0000 7ff6737fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff6740ccd70 0x7ff6740b5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:07.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.168+0000 7ff672ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff67415a1b0 0x7ff6740ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:07.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.168+0000 7ff6737fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff6740ccd70 0x7ff6740b5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.77:54070/0 (socket says 172.21.15.77:54070) 2024-01-11T08:10:07.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.169+0000 7ff6737fe700 1 -- 172.21.15.77:0/4180072997 learned_addr learned my addr 172.21.15.77:0/4180072997 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-11T08:10:07.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.169+0000 7ff673fff700 1 --2- 172.21.15.77:0/4180072997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67415ae70 0x7ff6741692d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:07.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.169+0000 7ff672ffd700 1 -- 172.21.15.77:0/4180072997 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff6740ccd70 msgr2=0x7ff6740b5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:07.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.169+0000 7ff672ffd700 1 --2- 172.21.15.77:0/4180072997 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff6740ccd70 0x7ff6740b5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.169+0000 7ff672ffd700 1 -- 172.21.15.77:0/4180072997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67415ae70 msgr2=0x7ff6741692d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:07.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.169+0000 7ff672ffd700 1 --2- 172.21.15.77:0/4180072997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67415ae70 0x7ff6741692d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.169+0000 7ff672ffd700 1 -- 172.21.15.77:0/4180072997 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6680094f0 con 0x7ff67415a1b0 2024-01-11T08:10:07.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.169+0000 7ff672ffd700 1 --2- 172.21.15.77:0/4180072997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff67415a1b0 0x7ff6740ca0c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff6740d16f0 tx=0x7ff668017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:07.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.169+0000 7ff65b7fe700 1 -- 172.21.15.77:0/4180072997 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff668024440 con 0x7ff67415a1b0 2024-01-11T08:10:07.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.169+0000 7ff65b7fe700 1 -- 172.21.15.77:0/4180072997 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6680245a0 con 0x7ff67415a1b0 2024-01-11T08:10:07.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.169+0000 7ff679dff700 1 -- 172.21.15.77:0/4180072997 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff674169c80 con 0x7ff67415a1b0 2024-01-11T08:10:07.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.169+0000 7ff65b7fe700 1 -- 172.21.15.77:0/4180072997 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff668024940 con 0x7ff67415a1b0 2024-01-11T08:10:07.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.170+0000 7ff679dff700 1 -- 172.21.15.77:0/4180072997 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff67416a240 con 0x7ff67415a1b0 2024-01-11T08:10:07.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.170+0000 7ff65b7fe700 1 -- 172.21.15.77:0/4180072997 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7ff66801a6a0 con 0x7ff67415a1b0 2024-01-11T08:10:07.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.170+0000 7ff65b7fe700 1 -- 172.21.15.77:0/4180072997 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7ff6680233e0 con 0x7ff67415a1b0 2024-01-11T08:10:07.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.171+0000 7ff679dff700 1 -- 172.21.15.77:0/4180072997 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff67416a780 con 0x7ff67415a1b0 2024-01-11T08:10:07.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.173+0000 7ff65b7fe700 1 -- 172.21.15.77:0/4180072997 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7ff67416a780 con 0x7ff67415a1b0 2024-01-11T08:10:07.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.256+0000 7ff679dff700 1 -- 172.21.15.77:0/4180072997 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bb18c833-73e9-4afd-96a2-103eb308745b", "id": 2} v 0) v1 -- 0x7ff67404c3c0 con 0x7ff67415a1b0 2024-01-11T08:10:07.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.264+0000 7ff65b7fe700 1 -- 172.21.15.77:0/4180072997 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bb18c833-73e9-4afd-96a2-103eb308745b", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff668035020 con 0x7ff67415a1b0 2024-01-11T08:10:07.264 INFO:teuthology.orchestra.run.smithi077.stdout:2 2024-01-11T08:10:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.264+0000 7ff679dff700 1 -- 172.21.15.77:0/4180072997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff67415a1b0 msgr2=0x7ff6740ca0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.264+0000 7ff679dff700 1 --2- 172.21.15.77:0/4180072997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff67415a1b0 0x7ff6740ca0c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff6740d16f0 tx=0x7ff668017000 comp rx=0 tx=0).stop 2024-01-11T08:10:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.265+0000 7ff679dff700 1 -- 172.21.15.77:0/4180072997 shutdown_connections 2024-01-11T08:10:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.265+0000 7ff679dff700 1 --2- 172.21.15.77:0/4180072997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67415ae70 0x7ff6741692d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.265+0000 7ff679dff700 1 --2- 172.21.15.77:0/4180072997 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff6740ccd70 0x7ff6740b5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.265+0000 7ff679dff700 1 --2- 172.21.15.77:0/4180072997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff67415a1b0 0x7ff6740ca0c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.265+0000 7ff679dff700 1 -- 172.21.15.77:0/4180072997 >> 172.21.15.77:0/4180072997 conn(0x7ff67406ed40 msgr2=0x7ff674158a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.265+0000 7ff679dff700 1 -- 172.21.15.77:0/4180072997 shutdown_connections 2024-01-11T08:10:07.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.265+0000 7ff679dff700 1 -- 172.21.15.77:0/4180072997 wait complete. 2024-01-11T08:10:07.281 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd new cef59f93-06ff-469c-a433-41cea02e7c62 3 2024-01-11T08:10:07.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.467+0000 7f74fbfff700 1 Processor -- start 2024-01-11T08:10:07.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.467+0000 7f74fbfff700 1 -- start start 2024-01-11T08:10:07.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.468+0000 7f74fbfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f74f415a1b0 0x7f74f415a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:07.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.468+0000 7f74fbfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74f415ae70 0x7f74f415b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:07.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.468+0000 7f74fbfff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f40cf850 con 0x7f74f40ccd70 2024-01-11T08:10:07.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.468+0000 7f74fbfff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74f40cec40 con 0x7f74f415a1b0 2024-01-11T08:10:07.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.468+0000 7f74fbfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f40cef20 con 0x7f74f415ae70 2024-01-11T08:10:07.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.469+0000 7f74f9d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f74f415a1b0 0x7f74f415a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:07.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.469+0000 7f74f9d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f74f415a1b0 0x7f74f415a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56722/0 (socket says 172.21.15.77:56722) 2024-01-11T08:10:07.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.469+0000 7f74fa59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74f415ae70 0x7f74f415b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:07.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.469+0000 7f74f9d9b700 1 -- 172.21.15.77:0/2079700904 learned_addr learned my addr 172.21.15.77:0/2079700904 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-11T08:10:07.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.469+0000 7f74f9d9b700 1 -- 172.21.15.77:0/2079700904 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74f415ae70 msgr2=0x7f74f415b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:07.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.469+0000 7f74f9d9b700 1 --2- 172.21.15.77:0/2079700904 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74f415ae70 0x7f74f415b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.469+0000 7f74f9d9b700 1 -- 172.21.15.77:0/2079700904 >> v1:172.21.15.27:6789/0 conn(0x7f74f40ccd70 legacy=0x7f74f40cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:07.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.469+0000 7f74f9d9b700 1 -- 172.21.15.77:0/2079700904 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f40cf200 con 0x7f74f415a1b0 2024-01-11T08:10:07.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.469+0000 7f74f9d9b700 1 --2- 172.21.15.77:0/2079700904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74f415a1b0 0x7f74f415a580 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f74f00099b0 tx=0x7f74f0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=402c4cc4b81806b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:07.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.470+0000 7f74f8d99700 1 -- 172.21.15.77:0/2079700904 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f0009ea0 con 0x7f74f415a1b0 2024-01-11T08:10:07.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.470+0000 7f74f8d99700 1 -- 172.21.15.77:0/2079700904 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f000b820 con 0x7f74f415a1b0 2024-01-11T08:10:07.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.470+0000 7f74f8d99700 1 -- 172.21.15.77:0/2079700904 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f000bbc0 con 0x7f74f415a1b0 2024-01-11T08:10:07.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.470+0000 7f74fbfff700 1 -- 172.21.15.77:0/2079700904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74f415a1b0 msgr2=0x7f74f415a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:07.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.470+0000 7f74fbfff700 1 --2- 172.21.15.77:0/2079700904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74f415a1b0 0x7f74f415a580 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f74f00099b0 tx=0x7f74f0009250 comp rx=0 tx=0).stop 2024-01-11T08:10:07.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.471+0000 7f74fbfff700 1 -- 172.21.15.77:0/2079700904 shutdown_connections 2024-01-11T08:10:07.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.471+0000 7f74fbfff700 1 --2- 172.21.15.77:0/2079700904 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74f415ae70 0x7f74f415b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.471+0000 7f74fbfff700 1 --2- 172.21.15.77:0/2079700904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74f415a1b0 0x7f74f415a580 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.471+0000 7f74fbfff700 1 -- 172.21.15.77:0/2079700904 >> 172.21.15.77:0/2079700904 conn(0x7f74f406ed40 msgr2=0x7f74f406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:07.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.471+0000 7f74fbfff700 1 -- 172.21.15.77:0/2079700904 shutdown_connections 2024-01-11T08:10:07.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.471+0000 7f74fbfff700 1 -- 172.21.15.77:0/2079700904 wait complete. 2024-01-11T08:10:07.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.472+0000 7f74fbfff700 1 Processor -- start 2024-01-11T08:10:07.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.472+0000 7f74fbfff700 1 -- start start 2024-01-11T08:10:07.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.473+0000 7f74fbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74f40ccd70 0x7f74f40af810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:07.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.473+0000 7f74fbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74f415a1b0 0x7f74f41554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:07.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.473+0000 7f74fbfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74f415ae70 0x7f74f40c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:07.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.473+0000 7f74fbfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f40b3380 con 0x7f74f40ccd70 2024-01-11T08:10:07.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.473+0000 7f74fbfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f40c7ae0 con 0x7f74f415a1b0 2024-01-11T08:10:07.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.473+0000 7f74fbfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f40b3560 con 0x7f74f415ae70 2024-01-11T08:10:07.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.473+0000 7f74f959a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74f415a1b0 0x7f74f41554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:07.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.474+0000 7f74fa59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74f415ae70 0x7f74f40c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:07.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.474+0000 7f74f9d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74f40ccd70 0x7f74f40af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:07.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.474+0000 7f74f959a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74f415a1b0 0x7f74f41554f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56730/0 (socket says 172.21.15.77:56730) 2024-01-11T08:10:07.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.474+0000 7f74f959a700 1 -- 172.21.15.77:0/793293140 learned_addr learned my addr 172.21.15.77:0/793293140 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-11T08:10:07.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.474+0000 7f74f959a700 1 -- 172.21.15.77:0/793293140 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74f415ae70 msgr2=0x7f74f40c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:07.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.474+0000 7f74f959a700 1 --2- 172.21.15.77:0/793293140 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74f415ae70 0x7f74f40c6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.474+0000 7f74f959a700 1 -- 172.21.15.77:0/793293140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74f40ccd70 msgr2=0x7f74f40af810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:07.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.474+0000 7f74f959a700 1 --2- 172.21.15.77:0/793293140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74f40ccd70 0x7f74f40af810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.474+0000 7f74f959a700 1 -- 172.21.15.77:0/793293140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f00094f0 con 0x7f74f415a1b0 2024-01-11T08:10:07.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.474+0000 7f74fa59c700 1 --2- 172.21.15.77:0/793293140 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74f415ae70 0x7f74f40c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:07.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.474+0000 7f74f959a700 1 --2- 172.21.15.77:0/793293140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74f415a1b0 0x7f74f41554f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f74e400b600 tx=0x7f74e40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:07.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.474+0000 7f74e9ffb700 1 -- 172.21.15.77:0/793293140 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e400f510 con 0x7f74f415a1b0 2024-01-11T08:10:07.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.474+0000 7f74e9ffb700 1 -- 172.21.15.77:0/793293140 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e400f690 con 0x7f74f415a1b0 2024-01-11T08:10:07.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.475+0000 7f74fbfff700 1 -- 172.21.15.77:0/793293140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f40b3840 con 0x7f74f415a1b0 2024-01-11T08:10:07.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.475+0000 7f74e9ffb700 1 -- 172.21.15.77:0/793293140 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e400fa50 con 0x7f74f415a1b0 2024-01-11T08:10:07.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.475+0000 7f74fbfff700 1 -- 172.21.15.77:0/793293140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f40afd50 con 0x7f74f415a1b0 2024-01-11T08:10:07.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.475+0000 7f74e9ffb700 1 -- 172.21.15.77:0/793293140 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f74e401a680 con 0x7f74f415a1b0 2024-01-11T08:10:07.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.475+0000 7f74e9ffb700 1 -- 172.21.15.77:0/793293140 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f74e4011400 con 0x7f74f415a1b0 2024-01-11T08:10:07.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.476+0000 7f74fbfff700 1 -- 172.21.15.77:0/793293140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f404c3c0 con 0x7f74f415a1b0 2024-01-11T08:10:07.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.478+0000 7f74e9ffb700 1 -- 172.21.15.77:0/793293140 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f74e401a930 con 0x7f74f415a1b0 2024-01-11T08:10:07.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.561+0000 7f74fbfff700 1 -- 172.21.15.77:0/793293140 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cef59f93-06ff-469c-a433-41cea02e7c62", "id": 3} v 0) v1 -- 0x7f74f40b2d60 con 0x7f74f415a1b0 2024-01-11T08:10:07.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.569+0000 7f74e9ffb700 1 -- 172.21.15.77:0/793293140 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cef59f93-06ff-469c-a433-41cea02e7c62", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f74e401ab90 con 0x7f74f415a1b0 2024-01-11T08:10:07.569 INFO:teuthology.orchestra.run.smithi077.stdout:3 2024-01-11T08:10:07.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.570+0000 7f74fbfff700 1 -- 172.21.15.77:0/793293140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74f415a1b0 msgr2=0x7f74f41554f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:07.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.570+0000 7f74fbfff700 1 --2- 172.21.15.77:0/793293140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74f415a1b0 0x7f74f41554f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f74e400b600 tx=0x7f74e40091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.570+0000 7f74fbfff700 1 -- 172.21.15.77:0/793293140 shutdown_connections 2024-01-11T08:10:07.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.570+0000 7f74fbfff700 1 --2- 172.21.15.77:0/793293140 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74f415ae70 0x7f74f40c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.570+0000 7f74fbfff700 1 --2- 172.21.15.77:0/793293140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74f40ccd70 0x7f74f40af810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.570+0000 7f74fbfff700 1 --2- 172.21.15.77:0/793293140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74f415a1b0 0x7f74f41554f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.570+0000 7f74fbfff700 1 -- 172.21.15.77:0/793293140 >> 172.21.15.77:0/793293140 conn(0x7f74f406ed40 msgr2=0x7f74f40c5ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:07.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.570+0000 7f74fbfff700 1 -- 172.21.15.77:0/793293140 shutdown_connections 2024-01-11T08:10:07.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.570+0000 7f74fbfff700 1 -- 172.21.15.77:0/793293140 wait complete. 2024-01-11T08:10:07.586 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd new 4cfd29cd-a247-49ca-b1d9-a56b522d54a7 4 2024-01-11T08:10:07.636 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:07.636+0000 7f78c72551c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:10:07.715 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:07.715+0000 7f62ffb111c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:10:07.730 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:07.731+0000 7f78c72551c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:10:07.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.774+0000 7f31f71ff700 1 Processor -- start 2024-01-11T08:10:07.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.774+0000 7f31f71ff700 1 -- start start 2024-01-11T08:10:07.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.775+0000 7f31f71ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f31f015a1b0 0x7f31f015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:07.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.775+0000 7f31f71ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f31f00ccd70 0x7f31f00cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:07.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.775+0000 7f31f71ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f00685c0 0x7f31f0068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:07.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.775+0000 7f31f71ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f00cf850 con 0x7f31f015a1b0 2024-01-11T08:10:07.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.775+0000 7f31f71ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f00cec40 con 0x7f31f00ccd70 2024-01-11T08:10:07.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.775+0000 7f31f71ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f00cef20 con 0x7f31f00685c0 2024-01-11T08:10:07.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.775+0000 7f31f579c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f00685c0 0x7f31f0068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:07.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.775+0000 7f31effff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f31f00ccd70 0x7f31f00cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:07.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.775+0000 7f31f579c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f00685c0 0x7f31f0068a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.77:54094/0 (socket says 172.21.15.77:54094) 2024-01-11T08:10:07.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.775+0000 7f31f579c700 1 -- 172.21.15.77:0/2449651169 learned_addr learned my addr 172.21.15.77:0/2449651169 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-11T08:10:07.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.775+0000 7f31f4f9b700 1 --2- 172.21.15.77:0/2449651169 >> v2:172.21.15.27:3300/0 conn(0x7f31f015a1b0 0x7f31f015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:07.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.776+0000 7f31f579c700 1 -- 172.21.15.77:0/2449651169 >> v2:172.21.15.77:3300/0 conn(0x7f31f00ccd70 msgr2=0x7f31f00cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:07.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.776+0000 7f31f579c700 1 --2- 172.21.15.77:0/2449651169 >> v2:172.21.15.77:3300/0 conn(0x7f31f00ccd70 0x7f31f00cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.776+0000 7f31f579c700 1 -- 172.21.15.77:0/2449651169 >> v2:172.21.15.27:3300/0 conn(0x7f31f015a1b0 msgr2=0x7f31f015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:07.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.776+0000 7f31f579c700 1 --2- 172.21.15.77:0/2449651169 >> v2:172.21.15.27:3300/0 conn(0x7f31f015a1b0 0x7f31f015a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.776+0000 7f31f579c700 1 -- 172.21.15.77:0/2449651169 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f00cf200 con 0x7f31f00685c0 2024-01-11T08:10:07.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.776+0000 7f31f579c700 1 --2- 172.21.15.77:0/2449651169 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f00685c0 0x7f31f0068a40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f31f00696f0 tx=0x7f31e8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2af19f6c0eab178 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:07.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.776+0000 7f31ef7fe700 1 -- 172.21.15.77:0/2449651169 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e800e070 con 0x7f31f00685c0 2024-01-11T08:10:07.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.776+0000 7f31ef7fe700 1 -- 172.21.15.77:0/2449651169 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e800b970 con 0x7f31f00685c0 2024-01-11T08:10:07.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.776+0000 7f31ef7fe700 1 -- 172.21.15.77:0/2449651169 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e80092e0 con 0x7f31f00685c0 2024-01-11T08:10:07.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.777+0000 7f31f71ff700 1 -- 172.21.15.77:0/2449651169 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f00685c0 msgr2=0x7f31f0068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:07.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.777+0000 7f31f71ff700 1 --2- 172.21.15.77:0/2449651169 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f00685c0 0x7f31f0068a40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f31f00696f0 tx=0x7f31e8009840 comp rx=0 tx=0).stop 2024-01-11T08:10:07.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.777+0000 7f31f71ff700 1 -- 172.21.15.77:0/2449651169 shutdown_connections 2024-01-11T08:10:07.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.777+0000 7f31f71ff700 1 --2- 172.21.15.77:0/2449651169 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f00685c0 0x7f31f0068a40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.777+0000 7f31f71ff700 1 --2- 172.21.15.77:0/2449651169 >> v2:172.21.15.27:3300/0 conn(0x7f31f015a1b0 0x7f31f015a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.777+0000 7f31f71ff700 1 --2- 172.21.15.77:0/2449651169 >> v2:172.21.15.77:3300/0 conn(0x7f31f00ccd70 0x7f31f00cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.777+0000 7f31f71ff700 1 -- 172.21.15.77:0/2449651169 >> 172.21.15.77:0/2449651169 conn(0x7f31f006ed40 msgr2=0x7f31f006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:07.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.777+0000 7f31f71ff700 1 -- 172.21.15.77:0/2449651169 shutdown_connections 2024-01-11T08:10:07.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.777+0000 7f31f71ff700 1 -- 172.21.15.77:0/2449651169 wait complete. 2024-01-11T08:10:07.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.779+0000 7f31f71ff700 1 Processor -- start 2024-01-11T08:10:07.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.779+0000 7f31f71ff700 1 -- start start 2024-01-11T08:10:07.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.780+0000 7f31f71ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f00685c0 0x7f31f00b5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:07.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.780+0000 7f31f71ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31f00ccd70 0x7f31f00c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:07.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.780+0000 7f31f71ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f015a1b0 0x7f31f0142e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:07.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.780+0000 7f31f71ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f00b4170 con 0x7f31f00ccd70 2024-01-11T08:10:07.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.780+0000 7f31f71ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f00b4450 con 0x7f31f015a1b0 2024-01-11T08:10:07.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.780+0000 7f31f71ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f00b4730 con 0x7f31f00685c0 2024-01-11T08:10:07.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.780+0000 7f31f4f9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f00685c0 0x7f31f00b5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:07.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.780+0000 7f31f579c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f015a1b0 0x7f31f0142e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:07.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.780+0000 7f31f4f9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f00685c0 0x7f31f00b5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.77:54104/0 (socket says 172.21.15.77:54104) 2024-01-11T08:10:07.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.780+0000 7f31f4f9b700 1 -- 172.21.15.77:0/2757482525 learned_addr learned my addr 172.21.15.77:0/2757482525 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-11T08:10:07.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.780+0000 7f31effff700 1 --2- 172.21.15.77:0/2757482525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31f00ccd70 0x7f31f00c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.780+0000 7f31f4f9b700 1 -- 172.21.15.77:0/2757482525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f015a1b0 msgr2=0x7f31f0142e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.780+0000 7f31f4f9b700 1 --2- 172.21.15.77:0/2757482525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f015a1b0 0x7f31f0142e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.780+0000 7f31f4f9b700 1 -- 172.21.15.77:0/2757482525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31f00ccd70 msgr2=0x7f31f00c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.780+0000 7f31f4f9b700 1 --2- 172.21.15.77:0/2757482525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31f00ccd70 0x7f31f00c9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.780+0000 7f31f4f9b700 1 -- 172.21.15.77:0/2757482525 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e8009ae0 con 0x7f31f00685c0 2024-01-11T08:10:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.780+0000 7f31effff700 1 --2- 172.21.15.77:0/2757482525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31f00ccd70 0x7f31f00c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.780+0000 7f31f579c700 1 --2- 172.21.15.77:0/2757482525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f015a1b0 0x7f31f0142e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.781+0000 7f31f4f9b700 1 --2- 172.21.15.77:0/2757482525 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f00685c0 0x7f31f00b5eb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f31e400ec80 tx=0x7f31e40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.781+0000 7f31ecff9700 1 -- 172.21.15.77:0/2757482525 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e400c850 con 0x7f31f00685c0 2024-01-11T08:10:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.781+0000 7f31ecff9700 1 -- 172.21.15.77:0/2757482525 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e400c9b0 con 0x7f31f00685c0 2024-01-11T08:10:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.781+0000 7f31f71ff700 1 -- 172.21.15.77:0/2757482525 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f00b4a10 con 0x7f31f00685c0 2024-01-11T08:10:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.781+0000 7f31ecff9700 1 -- 172.21.15.77:0/2757482525 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e400cd50 con 0x7f31f00685c0 2024-01-11T08:10:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.781+0000 7f31f71ff700 1 -- 172.21.15.77:0/2757482525 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f00b3f20 con 0x7f31f00685c0 2024-01-11T08:10:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.781+0000 7f31ecff9700 1 -- 172.21.15.77:0/2757482525 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f31e4005c80 con 0x7f31f00685c0 2024-01-11T08:10:07.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.781+0000 7f31ecff9700 1 -- 172.21.15.77:0/2757482525 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f31e4013b80 con 0x7f31f00685c0 2024-01-11T08:10:07.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.782+0000 7f31f71ff700 1 -- 172.21.15.77:0/2757482525 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f00b4ba0 con 0x7f31f00685c0 2024-01-11T08:10:07.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.785+0000 7f31ecff9700 1 -- 172.21.15.77:0/2757482525 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f31e4021060 con 0x7f31f00685c0 2024-01-11T08:10:07.809 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:07.809+0000 7f62ffb111c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:10:07.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.870+0000 7f31f71ff700 1 -- 172.21.15.77:0/2757482525 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4cfd29cd-a247-49ca-b1d9-a56b522d54a7", "id": 4} v 0) v1 -- 0x7f31f00c7850 con 0x7f31f00685c0 2024-01-11T08:10:07.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.874+0000 7f31ecff9700 1 -- 172.21.15.77:0/2757482525 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4cfd29cd-a247-49ca-b1d9-a56b522d54a7", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f31e4017070 con 0x7f31f00685c0 2024-01-11T08:10:07.873 INFO:teuthology.orchestra.run.smithi077.stdout:4 2024-01-11T08:10:07.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.874+0000 7f31f71ff700 1 -- 172.21.15.77:0/2757482525 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f00685c0 msgr2=0x7f31f00b5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:07.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.874+0000 7f31f71ff700 1 --2- 172.21.15.77:0/2757482525 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f00685c0 0x7f31f00b5eb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f31e400ec80 tx=0x7f31e40091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.875+0000 7f31f71ff700 1 -- 172.21.15.77:0/2757482525 shutdown_connections 2024-01-11T08:10:07.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.875+0000 7f31f71ff700 1 --2- 172.21.15.77:0/2757482525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f015a1b0 0x7f31f0142e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.875+0000 7f31f71ff700 1 --2- 172.21.15.77:0/2757482525 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f00685c0 0x7f31f00b5eb0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.875+0000 7f31f71ff700 1 --2- 172.21.15.77:0/2757482525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31f00ccd70 0x7f31f00c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.875+0000 7f31f71ff700 1 -- 172.21.15.77:0/2757482525 >> 172.21.15.77:0/2757482525 conn(0x7f31f006ed40 msgr2=0x7f31f0068990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.875+0000 7f31f71ff700 1 -- 172.21.15.77:0/2757482525 shutdown_connections 2024-01-11T08:10:07.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:07.875+0000 7f31f71ff700 1 -- 172.21.15.77:0/2757482525 wait complete. 2024-01-11T08:10:07.890 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd new ea8d9836-29b2-4a44-8510-0373c8be8878 5 2024-01-11T08:10:07.906 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:07.906+0000 7f78c72551c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:10:07.984 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:07.985+0000 7f62ffb111c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:10:07.996 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:07.997+0000 7f78c72551c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:10:08.075 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:08.075+0000 7f62ffb111c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:10:08.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.078+0000 7fbf715ff700 1 Processor -- start 2024-01-11T08:10:08.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.078+0000 7fbf715ff700 1 -- start start 2024-01-11T08:10:08.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.080+0000 7fbf715ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf6c1577e0 0x7fbf6c0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:08.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.080+0000 7fbf715ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbf6c069540 0x7fbf6c0ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:08.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.080+0000 7fbf715ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6c0699d0 con 0x7fbf6c157410 2024-01-11T08:10:08.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.080+0000 7fbf715ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf6c069cb0 con 0x7fbf6c069540 2024-01-11T08:10:08.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.080+0000 7fbf715ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c069f90 con 0x7fbf6c1577e0 2024-01-11T08:10:08.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.080+0000 7fbf6b7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbf6c069540 0x7fbf6c0ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:08.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.080+0000 7fbf6a7fc700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf6c1577e0 0x7fbf6c0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:08.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.080+0000 7fbf6a7fc700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf6c1577e0 0x7fbf6c0ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.77:54114/0 (socket says 172.21.15.77:54114) 2024-01-11T08:10:08.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.080+0000 7fbf6b7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbf6c069540 0x7fbf6c0ae540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56756/0 (socket says 172.21.15.77:56756) 2024-01-11T08:10:08.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.080+0000 7fbf6a7fc700 1 -- 172.21.15.77:0/84315861 learned_addr learned my addr 172.21.15.77:0/84315861 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-11T08:10:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.080+0000 7fbf6affd700 1 --1- 172.21.15.77:0/84315861 >> v1:172.21.15.27:6789/0 conn(0x7fbf6c157410 0x7fbf6c0ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.77:45444/0 (socket says 172.21.15.77:45444) 2024-01-11T08:10:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.080+0000 7fbf6a7fc700 1 -- 172.21.15.77:0/84315861 >> v2:172.21.15.77:3300/0 conn(0x7fbf6c069540 msgr2=0x7fbf6c0ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.080+0000 7fbf6a7fc700 1 --2- 172.21.15.77:0/84315861 >> v2:172.21.15.77:3300/0 conn(0x7fbf6c069540 0x7fbf6c0ae540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.080+0000 7fbf6a7fc700 1 -- 172.21.15.77:0/84315861 >> v1:172.21.15.27:6789/0 conn(0x7fbf6c157410 legacy=0x7fbf6c0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.081+0000 7fbf6a7fc700 1 -- 172.21.15.77:0/84315861 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6c06a270 con 0x7fbf6c1577e0 2024-01-11T08:10:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.081+0000 7fbf6b7fe700 1 --2- 172.21.15.77:0/84315861 >> v2:172.21.15.77:3300/0 conn(0x7fbf6c069540 0x7fbf6c0ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.081+0000 7fbf69ffb700 1 -- 172.21.15.77:0/84315861 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470372727 0 0) 0x7fbf6c0699d0 con 0x7fbf6c157410 2024-01-11T08:10:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.081+0000 7fbf6a7fc700 1 --2- 172.21.15.77:0/84315861 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf6c1577e0 0x7fbf6c0ae000 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbf600099b0 tx=0x7fbf60009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd89c54109c6bb85 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.081+0000 7fbf69ffb700 1 -- 172.21.15.77:0/84315861 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf60009ea0 con 0x7fbf6c1577e0 2024-01-11T08:10:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.081+0000 7fbf69ffb700 1 -- 172.21.15.77:0/84315861 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6000b860 con 0x7fbf6c1577e0 2024-01-11T08:10:08.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.081+0000 7fbf69ffb700 1 -- 172.21.15.77:0/84315861 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6000bc00 con 0x7fbf6c1577e0 2024-01-11T08:10:08.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.082+0000 7fbf715ff700 1 -- 172.21.15.77:0/84315861 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf6c1577e0 msgr2=0x7fbf6c0ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:08.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.082+0000 7fbf715ff700 1 --2- 172.21.15.77:0/84315861 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf6c1577e0 0x7fbf6c0ae000 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbf600099b0 tx=0x7fbf60009250 comp rx=0 tx=0).stop 2024-01-11T08:10:08.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.082+0000 7fbf715ff700 1 -- 172.21.15.77:0/84315861 shutdown_connections 2024-01-11T08:10:08.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.082+0000 7fbf715ff700 1 --2- 172.21.15.77:0/84315861 >> v2:172.21.15.77:3300/0 conn(0x7fbf6c069540 0x7fbf6c0ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.082+0000 7fbf715ff700 1 --2- 172.21.15.77:0/84315861 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf6c1577e0 0x7fbf6c0ae000 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.082+0000 7fbf715ff700 1 -- 172.21.15.77:0/84315861 >> 172.21.15.77:0/84315861 conn(0x7fbf6c0680a0 msgr2=0x7fbf6c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:08.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.082+0000 7fbf715ff700 1 -- 172.21.15.77:0/84315861 shutdown_connections 2024-01-11T08:10:08.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.082+0000 7fbf715ff700 1 -- 172.21.15.77:0/84315861 wait complete. 2024-01-11T08:10:08.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.083+0000 7fbf715ff700 1 Processor -- start 2024-01-11T08:10:08.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.083+0000 7fbf715ff700 1 -- start start 2024-01-11T08:10:08.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.084+0000 7fbf715ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf6c069540 0x7fbf6c0c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:08.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.084+0000 7fbf715ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6c157410 0x7fbf6c1478a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:08.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.084+0000 7fbf715ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf6c1577e0 0x7fbf6c147de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:08.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.084+0000 7fbf715ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c16a470 con 0x7fbf6c157410 2024-01-11T08:10:08.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.084+0000 7fbf715ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c169860 con 0x7fbf6c1577e0 2024-01-11T08:10:08.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.084+0000 7fbf715ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c169b40 con 0x7fbf6c069540 2024-01-11T08:10:08.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.084+0000 7fbf6affd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf6c069540 0x7fbf6c0c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:08.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.085+0000 7fbf6affd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf6c069540 0x7fbf6c0c8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.77:54120/0 (socket says 172.21.15.77:54120) 2024-01-11T08:10:08.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.085+0000 7fbf6affd700 1 -- 172.21.15.77:0/878373167 learned_addr learned my addr 172.21.15.77:0/878373167 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-11T08:10:08.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.085+0000 7fbf6b7fe700 1 --2- 172.21.15.77:0/878373167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf6c1577e0 0x7fbf6c147de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:08.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.085+0000 7fbf6a7fc700 1 --2- 172.21.15.77:0/878373167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6c157410 0x7fbf6c1478a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:08.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.085+0000 7fbf6affd700 1 -- 172.21.15.77:0/878373167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf6c1577e0 msgr2=0x7fbf6c147de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:08.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.085+0000 7fbf6affd700 1 --2- 172.21.15.77:0/878373167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf6c1577e0 0x7fbf6c147de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.085+0000 7fbf6affd700 1 -- 172.21.15.77:0/878373167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6c157410 msgr2=0x7fbf6c1478a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:08.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.085+0000 7fbf6affd700 1 --2- 172.21.15.77:0/878373167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6c157410 0x7fbf6c1478a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.085+0000 7fbf6affd700 1 -- 172.21.15.77:0/878373167 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf64009a20 con 0x7fbf6c069540 2024-01-11T08:10:08.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.085+0000 7fbf6affd700 1 --2- 172.21.15.77:0/878373167 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf6c069540 0x7fbf6c0c8a70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c00c810 tx=0x7fbf5c007de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:08.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.085+0000 7fbf52ffd700 1 -- 172.21.15.77:0/878373167 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c005b90 con 0x7fbf6c069540 2024-01-11T08:10:08.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.085+0000 7fbf52ffd700 1 -- 172.21.15.77:0/878373167 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5c005cf0 con 0x7fbf6c069540 2024-01-11T08:10:08.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.085+0000 7fbf715ff700 1 -- 172.21.15.77:0/878373167 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf600094f0 con 0x7fbf6c069540 2024-01-11T08:10:08.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.085+0000 7fbf52ffd700 1 -- 172.21.15.77:0/878373167 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c007320 con 0x7fbf6c069540 2024-01-11T08:10:08.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.086+0000 7fbf715ff700 1 -- 172.21.15.77:0/878373167 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf6c16a100 con 0x7fbf6c069540 2024-01-11T08:10:08.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.086+0000 7fbf52ffd700 1 -- 172.21.15.77:0/878373167 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fbf5c01c6c0 con 0x7fbf6c069540 2024-01-11T08:10:08.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.086+0000 7fbf52ffd700 1 -- 172.21.15.77:0/878373167 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7fbf5c007710 con 0x7fbf6c069540 2024-01-11T08:10:08.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.087+0000 7fbf715ff700 1 -- 172.21.15.77:0/878373167 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf6c04c3c0 con 0x7fbf6c069540 2024-01-11T08:10:08.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.089+0000 7fbf52ffd700 1 -- 172.21.15.77:0/878373167 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fbf5c017070 con 0x7fbf6c069540 2024-01-11T08:10:08.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.175+0000 7fbf715ff700 1 -- 172.21.15.77:0/878373167 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ea8d9836-29b2-4a44-8510-0373c8be8878", "id": 5} v 0) v1 -- 0x7fbf6c0cb2c0 con 0x7fbf6c069540 2024-01-11T08:10:08.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.181+0000 7fbf52ffd700 1 -- 172.21.15.77:0/878373167 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ea8d9836-29b2-4a44-8510-0373c8be8878", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbf5c005e60 con 0x7fbf6c069540 2024-01-11T08:10:08.181 INFO:teuthology.orchestra.run.smithi077.stdout:5 2024-01-11T08:10:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.182+0000 7fbf715ff700 1 -- 172.21.15.77:0/878373167 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf6c069540 msgr2=0x7fbf6c0c8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.182+0000 7fbf715ff700 1 --2- 172.21.15.77:0/878373167 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf6c069540 0x7fbf6c0c8a70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c00c810 tx=0x7fbf5c007de0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.182+0000 7fbf715ff700 1 -- 172.21.15.77:0/878373167 shutdown_connections 2024-01-11T08:10:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.182+0000 7fbf715ff700 1 --2- 172.21.15.77:0/878373167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf6c1577e0 0x7fbf6c147de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.182+0000 7fbf715ff700 1 --2- 172.21.15.77:0/878373167 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf6c069540 0x7fbf6c0c8a70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.182+0000 7fbf715ff700 1 --2- 172.21.15.77:0/878373167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6c157410 0x7fbf6c1478a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.182+0000 7fbf715ff700 1 -- 172.21.15.77:0/878373167 >> 172.21.15.77:0/878373167 conn(0x7fbf6c0680a0 msgr2=0x7fbf6c0c23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.182+0000 7fbf715ff700 1 -- 172.21.15.77:0/878373167 shutdown_connections 2024-01-11T08:10:08.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.182+0000 7fbf715ff700 1 -- 172.21.15.77:0/878373167 wait complete. 2024-01-11T08:10:08.199 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd new 9f418630-4faf-41b2-9a23-ac50e6177719 6 2024-01-11T08:10:08.224 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:08.225+0000 7f78c72551c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:10:08.303 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:08.303+0000 7f62ffb111c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:10:08.325 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:08.325+0000 7f78c72551c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:10:08.403 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:08.403+0000 7f62ffb111c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:10:08.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.411+0000 7fab0c3ff700 1 Processor -- start 2024-01-11T08:10:08.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.411+0000 7fab0c3ff700 1 -- start start 2024-01-11T08:10:08.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.413+0000 7fab0c3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fab040ccd70 0x7fab040cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:08.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.413+0000 7fab0c3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab040cf850 con 0x7fab0415a1b0 2024-01-11T08:10:08.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.413+0000 7fab0c3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab040cec40 con 0x7fab0415ae70 2024-01-11T08:10:08.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.413+0000 7fab0c3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab0415ac90 con 0x7fab040ccd70 2024-01-11T08:10:08.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.413+0000 7fab0999a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fab040ccd70 0x7fab040cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:08.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.413+0000 7fab0a99c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fab0415ae70 0x7fab0415b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54210/0 (socket says 172.21.15.77:54210) 2024-01-11T08:10:08.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.413+0000 7fab0a99c700 1 -- 172.21.15.77:0/2845516141 learned_addr learned my addr 172.21.15.77:0/2845516141 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-11T08:10:08.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.413+0000 7fab0999a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fab040ccd70 0x7fab040cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56762/0 (socket says 172.21.15.77:56762) 2024-01-11T08:10:08.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.413+0000 7fab0999a700 1 -- 172.21.15.77:0/2845516141 >> v1:172.21.15.77:6789/0 conn(0x7fab0415ae70 legacy=0x7fab0415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:08.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.413+0000 7fab0999a700 1 -- 172.21.15.77:0/2845516141 >> v1:172.21.15.27:6789/0 conn(0x7fab0415a1b0 legacy=0x7fab0415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:08.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.413+0000 7fab0999a700 1 -- 172.21.15.77:0/2845516141 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab040cef20 con 0x7fab040ccd70 2024-01-11T08:10:08.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.413+0000 7fab09199700 1 -- 172.21.15.77:0/2845516141 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746106658 0 0) 0x7fab040cec40 con 0x7fab0415ae70 2024-01-11T08:10:08.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.414+0000 7fab0999a700 1 --2- 172.21.15.77:0/2845516141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab040ccd70 0x7fab040cd210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7faaf40098e0 tx=0x7faaf40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36d36f6b0b22b177 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.414+0000 7fab09199700 1 -- 172.21.15.77:0/2845516141 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf4009e80 con 0x7fab040ccd70 2024-01-11T08:10:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.414+0000 7fab09199700 1 -- 172.21.15.77:0/2845516141 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf400b860 con 0x7fab040ccd70 2024-01-11T08:10:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.414+0000 7fab09199700 1 -- 172.21.15.77:0/2845516141 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf400bbe0 con 0x7fab040ccd70 2024-01-11T08:10:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.414+0000 7fab0c3ff700 1 -- 172.21.15.77:0/2845516141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab040ccd70 msgr2=0x7fab040cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.414+0000 7fab0c3ff700 1 --2- 172.21.15.77:0/2845516141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab040ccd70 0x7fab040cd210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7faaf40098e0 tx=0x7faaf40091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.414+0000 7fab0c3ff700 1 -- 172.21.15.77:0/2845516141 shutdown_connections 2024-01-11T08:10:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.414+0000 7fab0c3ff700 1 --2- 172.21.15.77:0/2845516141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab040ccd70 0x7fab040cd210 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.414+0000 7fab0c3ff700 1 -- 172.21.15.77:0/2845516141 >> 172.21.15.77:0/2845516141 conn(0x7fab0406ed40 msgr2=0x7fab0406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.415+0000 7fab0c3ff700 1 -- 172.21.15.77:0/2845516141 shutdown_connections 2024-01-11T08:10:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.415+0000 7fab0c3ff700 1 -- 172.21.15.77:0/2845516141 wait complete. 2024-01-11T08:10:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.416+0000 7fab0c3ff700 1 Processor -- start 2024-01-11T08:10:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.416+0000 7fab0c3ff700 1 -- start start 2024-01-11T08:10:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.417+0000 7fab0c3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab040ccd70 0x7fab04157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.417+0000 7fab0c3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab0415a1b0 0x7fab040af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.417+0000 7fab0c3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fab0415ae70 0x7fab040b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.417+0000 7fab0c3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab040afc30 con 0x7fab0415a1b0 2024-01-11T08:10:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.417+0000 7fab0c3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab040b3790 con 0x7fab040ccd70 2024-01-11T08:10:08.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.417+0000 7fab0c3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab0415e690 con 0x7fab0415ae70 2024-01-11T08:10:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.417+0000 7fab0a19b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab040ccd70 0x7fab04157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.417+0000 7fab0a19b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab040ccd70 0x7fab04157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56766/0 (socket says 172.21.15.77:56766) 2024-01-11T08:10:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.417+0000 7fab0a19b700 1 -- 172.21.15.77:0/1093179368 learned_addr learned my addr 172.21.15.77:0/1093179368 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-11T08:10:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.417+0000 7fab0a99c700 1 --2- 172.21.15.77:0/1093179368 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fab0415ae70 0x7fab040b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.417+0000 7fab0999a700 1 --2- 172.21.15.77:0/1093179368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab0415a1b0 0x7fab040af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.418+0000 7fab0a19b700 1 -- 172.21.15.77:0/1093179368 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fab0415ae70 msgr2=0x7fab040b49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.418+0000 7fab0a19b700 1 --2- 172.21.15.77:0/1093179368 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fab0415ae70 0x7fab040b49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.418+0000 7fab0a19b700 1 -- 172.21.15.77:0/1093179368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab0415a1b0 msgr2=0x7fab040af6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.418+0000 7fab0a19b700 1 --2- 172.21.15.77:0/1093179368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab0415a1b0 0x7fab040af6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.418+0000 7fab0a19b700 1 -- 172.21.15.77:0/1093179368 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf4009420 con 0x7fab040ccd70 2024-01-11T08:10:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.418+0000 7fab0a19b700 1 --2- 172.21.15.77:0/1093179368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab040ccd70 0x7fab04157fd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fab0000c590 tx=0x7fab0000c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.418+0000 7faafa7fc700 1 -- 172.21.15.77:0/1093179368 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab00011590 con 0x7fab040ccd70 2024-01-11T08:10:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.418+0000 7faafa7fc700 1 -- 172.21.15.77:0/1093179368 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab00011710 con 0x7fab040ccd70 2024-01-11T08:10:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.418+0000 7fab0c3ff700 1 -- 172.21.15.77:0/1093179368 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab0415e970 con 0x7fab040ccd70 2024-01-11T08:10:08.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.418+0000 7faafa7fc700 1 -- 172.21.15.77:0/1093179368 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab00011ad0 con 0x7fab040ccd70 2024-01-11T08:10:08.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.418+0000 7fab0c3ff700 1 -- 172.21.15.77:0/1093179368 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab0415ec50 con 0x7fab040ccd70 2024-01-11T08:10:08.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.418+0000 7faafa7fc700 1 -- 172.21.15.77:0/1093179368 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fab0001c6c0 con 0x7fab040ccd70 2024-01-11T08:10:08.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.419+0000 7faafa7fc700 1 -- 172.21.15.77:0/1093179368 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7fab00012a20 con 0x7fab040ccd70 2024-01-11T08:10:08.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.419+0000 7fab0c3ff700 1 -- 172.21.15.77:0/1093179368 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab0404c3c0 con 0x7fab040ccd70 2024-01-11T08:10:08.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.421+0000 7faafa7fc700 1 -- 172.21.15.77:0/1093179368 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fab00017070 con 0x7fab040ccd70 2024-01-11T08:10:08.426 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:08.427+0000 7f78c72551c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:10:08.504 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:08.504+0000 7f62ffb111c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:10:08.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.505+0000 7fab0c3ff700 1 -- 172.21.15.77:0/1093179368 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9f418630-4faf-41b2-9a23-ac50e6177719", "id": 6} v 0) v1 -- 0x7fab040afdf0 con 0x7fab040ccd70 2024-01-11T08:10:08.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.510+0000 7faafa7fc700 1 -- 172.21.15.77:0/1093179368 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9f418630-4faf-41b2-9a23-ac50e6177719", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fab0001c970 con 0x7fab040ccd70 2024-01-11T08:10:08.509 INFO:teuthology.orchestra.run.smithi077.stdout:6 2024-01-11T08:10:08.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.511+0000 7fab0c3ff700 1 -- 172.21.15.77:0/1093179368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab040ccd70 msgr2=0x7fab04157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:08.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.511+0000 7fab0c3ff700 1 --2- 172.21.15.77:0/1093179368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab040ccd70 0x7fab04157fd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fab0000c590 tx=0x7fab0000c8a0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.511+0000 7fab0c3ff700 1 -- 172.21.15.77:0/1093179368 shutdown_connections 2024-01-11T08:10:08.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.511+0000 7fab0c3ff700 1 --2- 172.21.15.77:0/1093179368 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fab0415ae70 0x7fab040b49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.511+0000 7fab0c3ff700 1 --2- 172.21.15.77:0/1093179368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab040ccd70 0x7fab04157fd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.511+0000 7fab0c3ff700 1 --2- 172.21.15.77:0/1093179368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab0415a1b0 0x7fab040af6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.511+0000 7fab0c3ff700 1 -- 172.21.15.77:0/1093179368 >> 172.21.15.77:0/1093179368 conn(0x7fab0406ed40 msgr2=0x7fab040b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:08.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.511+0000 7fab0c3ff700 1 -- 172.21.15.77:0/1093179368 shutdown_connections 2024-01-11T08:10:08.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.511+0000 7fab0c3ff700 1 -- 172.21.15.77:0/1093179368 wait complete. 2024-01-11T08:10:08.527 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd new 90fddfc3-73d2-4217-a305-73658fd056bb 7 2024-01-11T08:10:08.545 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:08.546+0000 7f78c72551c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:10:08.619 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:08.619+0000 7f62ffb111c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:10:08.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.715+0000 7f4c7a5ff700 1 Processor -- start 2024-01-11T08:10:08.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.715+0000 7f4c7a5ff700 1 -- start start 2024-01-11T08:10:08.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.716+0000 7f4c7a5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c7415a1b0 0x7f4c7415a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:08.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.716+0000 7f4c7a5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4c7415ae70 0x7f4c7415b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:08.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.716+0000 7f4c7a5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c740cf850 con 0x7f4c740ccd70 2024-01-11T08:10:08.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.716+0000 7f4c7a5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c740cec40 con 0x7f4c7415ae70 2024-01-11T08:10:08.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.716+0000 7f4c7a5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c740cef20 con 0x7f4c7415a1b0 2024-01-11T08:10:08.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.717+0000 7f4c73fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c7415a1b0 0x7f4c7415a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:08.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.717+0000 7f4c6bfff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4c740ccd70 0x7f4c740cd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:54222/0 (socket says 172.21.15.77:54222) 2024-01-11T08:10:08.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.717+0000 7f4c6bfff700 1 -- 172.21.15.77:0/3187980589 learned_addr learned my addr 172.21.15.77:0/3187980589 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-01-11T08:10:08.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.717+0000 7f4c78b9c700 1 --2- 172.21.15.77:0/3187980589 >> v2:172.21.15.27:3300/0 conn(0x7f4c7415ae70 0x7f4c7415b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:08.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.717+0000 7f4c73fff700 1 -- 172.21.15.77:0/3187980589 >> v2:172.21.15.27:3300/0 conn(0x7f4c7415ae70 msgr2=0x7f4c7415b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:08.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.717+0000 7f4c73fff700 1 --2- 172.21.15.77:0/3187980589 >> v2:172.21.15.27:3300/0 conn(0x7f4c7415ae70 0x7f4c7415b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.717+0000 7f4c73fff700 1 -- 172.21.15.77:0/3187980589 >> v1:172.21.15.77:6789/0 conn(0x7f4c740ccd70 legacy=0x7f4c740cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:08.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.717+0000 7f4c73fff700 1 -- 172.21.15.77:0/3187980589 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c740cf200 con 0x7f4c7415a1b0 2024-01-11T08:10:08.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.717+0000 7f4c737fe700 1 -- 172.21.15.77:0/3187980589 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342929082 0 0) 0x7f4c740cf850 con 0x7f4c740ccd70 2024-01-11T08:10:08.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.717+0000 7f4c73fff700 1 --2- 172.21.15.77:0/3187980589 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c7415a1b0 0x7f4c7415a580 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4c640098e0 tx=0x7f4c640091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b32c2b786505ff4f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:08.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.718+0000 7f4c737fe700 1 -- 172.21.15.77:0/3187980589 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6400b640 con 0x7f4c7415a1b0 2024-01-11T08:10:08.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.718+0000 7f4c737fe700 1 -- 172.21.15.77:0/3187980589 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6400b7a0 con 0x7f4c7415a1b0 2024-01-11T08:10:08.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.718+0000 7f4c737fe700 1 -- 172.21.15.77:0/3187980589 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6400bb40 con 0x7f4c7415a1b0 2024-01-11T08:10:08.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.718+0000 7f4c7a5ff700 1 -- 172.21.15.77:0/3187980589 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c7415a1b0 msgr2=0x7f4c7415a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:08.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.718+0000 7f4c7a5ff700 1 --2- 172.21.15.77:0/3187980589 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c7415a1b0 0x7f4c7415a580 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4c640098e0 tx=0x7f4c640091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.718+0000 7f4c7a5ff700 1 -- 172.21.15.77:0/3187980589 shutdown_connections 2024-01-11T08:10:08.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.718+0000 7f4c7a5ff700 1 --2- 172.21.15.77:0/3187980589 >> v2:172.21.15.27:3300/0 conn(0x7f4c7415ae70 0x7f4c7415b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.718+0000 7f4c7a5ff700 1 --2- 172.21.15.77:0/3187980589 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c7415a1b0 0x7f4c7415a580 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.718+0000 7f4c7a5ff700 1 -- 172.21.15.77:0/3187980589 >> 172.21.15.77:0/3187980589 conn(0x7f4c7406ed40 msgr2=0x7f4c7406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:08.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.718+0000 7f4c7a5ff700 1 -- 172.21.15.77:0/3187980589 shutdown_connections 2024-01-11T08:10:08.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.718+0000 7f4c7a5ff700 1 -- 172.21.15.77:0/3187980589 wait complete. 2024-01-11T08:10:08.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.720+0000 7f4c7a5ff700 1 Processor -- start 2024-01-11T08:10:08.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.720+0000 7f4c7a5ff700 1 -- start start 2024-01-11T08:10:08.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.721+0000 7f4c7a5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c740ccd70 0x7f4c740c5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:08.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.721+0000 7f4c7a5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c7415a1b0 0x7f4c7416a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:08.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.721+0000 7f4c7a5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c7415ae70 0x7f4c741576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:08.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.721+0000 7f4c7a5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7416a9e0 con 0x7f4c7415ae70 2024-01-11T08:10:08.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.721+0000 7f4c7a5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7416a7f0 con 0x7f4c740ccd70 2024-01-11T08:10:08.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.721+0000 7f4c7a5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c740a6fb0 con 0x7f4c7415a1b0 2024-01-11T08:10:08.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.721+0000 7f4c73fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c740ccd70 0x7f4c740c5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:08.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.721+0000 7f4c6bfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c7415a1b0 0x7f4c7416a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:08.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.721+0000 7f4c73fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c740ccd70 0x7f4c740c5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56774/0 (socket says 172.21.15.77:56774) 2024-01-11T08:10:08.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.721+0000 7f4c73fff700 1 -- 172.21.15.77:0/1279426536 learned_addr learned my addr 172.21.15.77:0/1279426536 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-01-11T08:10:08.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.722+0000 7f4c78b9c700 1 --2- 172.21.15.77:0/1279426536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c7415ae70 0x7f4c741576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:08.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.722+0000 7f4c6bfff700 1 -- 172.21.15.77:0/1279426536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c740ccd70 msgr2=0x7f4c740c5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:08.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.722+0000 7f4c6bfff700 1 --2- 172.21.15.77:0/1279426536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c740ccd70 0x7f4c740c5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.722+0000 7f4c6bfff700 1 -- 172.21.15.77:0/1279426536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c7415ae70 msgr2=0x7f4c741576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:08.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.722+0000 7f4c6bfff700 1 --2- 172.21.15.77:0/1279426536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c7415ae70 0x7f4c741576d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.722+0000 7f4c6bfff700 1 -- 172.21.15.77:0/1279426536 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c64009420 con 0x7f4c7415a1b0 2024-01-11T08:10:08.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.722+0000 7f4c73fff700 1 --2- 172.21.15.77:0/1279426536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c740ccd70 0x7f4c740c5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:08.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.722+0000 7f4c78b9c700 1 --2- 172.21.15.77:0/1279426536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c7415ae70 0x7f4c741576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:08.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.722+0000 7f4c6bfff700 1 --2- 172.21.15.77:0/1279426536 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c7415a1b0 0x7f4c7416a2b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00c520 tx=0x7f4c5c007f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:08.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.722+0000 7f4c70ff9700 1 -- 172.21.15.77:0/1279426536 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c005b30 con 0x7f4c7415a1b0 2024-01-11T08:10:08.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.722+0000 7f4c70ff9700 1 -- 172.21.15.77:0/1279426536 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5c005c90 con 0x7f4c7415a1b0 2024-01-11T08:10:08.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.722+0000 7f4c70ff9700 1 -- 172.21.15.77:0/1279426536 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c007320 con 0x7f4c7415a1b0 2024-01-11T08:10:08.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.722+0000 7f4c7a5ff700 1 -- 172.21.15.77:0/1279426536 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c740a7250 con 0x7f4c7415a1b0 2024-01-11T08:10:08.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.722+0000 7f4c7a5ff700 1 -- 172.21.15.77:0/1279426536 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c740c62b0 con 0x7f4c7415a1b0 2024-01-11T08:10:08.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.722+0000 7f4c70ff9700 1 -- 172.21.15.77:0/1279426536 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f4c5c01c6c0 con 0x7f4c7415a1b0 2024-01-11T08:10:08.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.723+0000 7f4c70ff9700 1 -- 172.21.15.77:0/1279426536 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7f4c5c005e00 con 0x7f4c7415a1b0 2024-01-11T08:10:08.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.723+0000 7f4c7a5ff700 1 -- 172.21.15.77:0/1279426536 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c7404c3c0 con 0x7f4c7415a1b0 2024-01-11T08:10:08.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.726+0000 7f4c70ff9700 1 -- 172.21.15.77:0/1279426536 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f4c5c017070 con 0x7f4c7415a1b0 2024-01-11T08:10:08.779 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:08.780+0000 7f78c72551c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:10:08.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.810+0000 7f4c7a5ff700 1 -- 172.21.15.77:0/1279426536 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd new", "uuid": "90fddfc3-73d2-4217-a305-73658fd056bb", "id": 7} v 0) v1 -- 0x7f4c7416ab20 con 0x7f4c7415a1b0 2024-01-11T08:10:08.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.816+0000 7f4c70ff9700 1 -- 172.21.15.77:0/1279426536 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "90fddfc3-73d2-4217-a305-73658fd056bb", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4c5c007710 con 0x7f4c7415a1b0 2024-01-11T08:10:08.816 INFO:teuthology.orchestra.run.smithi077.stdout:7 2024-01-11T08:10:08.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.817+0000 7f4c7a5ff700 1 -- 172.21.15.77:0/1279426536 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c7415a1b0 msgr2=0x7f4c7416a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:08.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.817+0000 7f4c7a5ff700 1 --2- 172.21.15.77:0/1279426536 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c7415a1b0 0x7f4c7416a2b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00c520 tx=0x7f4c5c007f50 comp rx=0 tx=0).stop 2024-01-11T08:10:08.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.817+0000 7f4c7a5ff700 1 -- 172.21.15.77:0/1279426536 shutdown_connections 2024-01-11T08:10:08.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.817+0000 7f4c7a5ff700 1 --2- 172.21.15.77:0/1279426536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c7415ae70 0x7f4c741576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.817+0000 7f4c7a5ff700 1 --2- 172.21.15.77:0/1279426536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c740ccd70 0x7f4c740c5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.817+0000 7f4c7a5ff700 1 --2- 172.21.15.77:0/1279426536 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c7415a1b0 0x7f4c7416a2b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:08.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.817+0000 7f4c7a5ff700 1 -- 172.21.15.77:0/1279426536 >> 172.21.15.77:0/1279426536 conn(0x7f4c7406ed40 msgr2=0x7f4c74158960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:08.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.818+0000 7f4c7a5ff700 1 -- 172.21.15.77:0/1279426536 shutdown_connections 2024-01-11T08:10:08.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-01-11T08:10:08.818+0000 7f4c7a5ff700 1 -- 172.21.15.77:0/1279426536 wait complete. 2024-01-11T08:10:08.834 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-11T08:10:08.835 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-01-11T08:10:08.841 INFO:tasks.ceph.osd.0:Started 2024-01-11T08:10:08.841 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-11T08:10:08.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-01-11T08:10:08.848 INFO:tasks.ceph.osd.1:Started 2024-01-11T08:10:08.848 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-11T08:10:08.848 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-01-11T08:10:08.854 INFO:tasks.ceph.osd.2:Started 2024-01-11T08:10:08.854 INFO:tasks.ceph.osd.3:Restarting daemon 2024-01-11T08:10:08.854 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-01-11T08:10:08.861 INFO:tasks.ceph.osd.3:Started 2024-01-11T08:10:08.861 INFO:tasks.ceph.osd.4:Restarting daemon 2024-01-11T08:10:08.861 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-01-11T08:10:08.861 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:08.859+0000 7f62ffb111c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:10:08.880 INFO:tasks.ceph.osd.4:Started 2024-01-11T08:10:08.880 INFO:tasks.ceph.osd.5:Restarting daemon 2024-01-11T08:10:08.880 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-01-11T08:10:08.886 INFO:tasks.ceph.osd.5:Started 2024-01-11T08:10:08.886 INFO:tasks.ceph.osd.6:Restarting daemon 2024-01-11T08:10:08.886 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-01-11T08:10:08.887 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:08.885+0000 7f78c72551c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:10:08.892 INFO:tasks.ceph.osd.6:Started 2024-01-11T08:10:08.892 INFO:tasks.ceph.osd.7:Restarting daemon 2024-01-11T08:10:08.892 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-01-11T08:10:08.898 INFO:tasks.ceph.osd.7:Started 2024-01-11T08:10:08.899 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:10:09.028 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:09.028+0000 7f62ffb111c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:10:09.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.164+0000 7fb7557ff700 1 Processor -- start 2024-01-11T08:10:09.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.164+0000 7fb7557ff700 1 -- start start 2024-01-11T08:10:09.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.165+0000 7fb7557ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7500b2b90 0x7fb750159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:09.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.165+0000 7fb7557ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb7500b07b0 0x7fb750159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:09.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.165+0000 7fb7557ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7500cd730 con 0x7fb7500b03e0 2024-01-11T08:10:09.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.165+0000 7fb7557ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7500ccb20 con 0x7fb7500b2b90 2024-01-11T08:10:09.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.165+0000 7fb7557ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7500cce00 con 0x7fb7500b07b0 2024-01-11T08:10:09.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.166+0000 7fb74e7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7500b2b90 0x7fb750159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:09.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.166+0000 7fb74e7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7500b2b90 0x7fb750159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57286/0 (socket says 172.21.15.27:57286) 2024-01-11T08:10:09.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.166+0000 7fb74e7fc700 1 -- 172.21.15.27:0/306629959 learned_addr learned my addr 172.21.15.27:0/306629959 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:09.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.166+0000 7fb74f7fe700 1 --2- 172.21.15.27:0/306629959 >> v2:172.21.15.77:3300/0 conn(0x7fb7500b07b0 0x7fb750159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:09.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.166+0000 7fb74e7fc700 1 -- 172.21.15.27:0/306629959 >> v2:172.21.15.77:3300/0 conn(0x7fb7500b07b0 msgr2=0x7fb750159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:09.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.166+0000 7fb74e7fc700 1 --2- 172.21.15.27:0/306629959 >> v2:172.21.15.77:3300/0 conn(0x7fb7500b07b0 0x7fb750159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:09.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.166+0000 7fb74e7fc700 1 -- 172.21.15.27:0/306629959 >> v1:172.21.15.77:6789/0 conn(0x7fb7500b03e0 legacy=0x7fb7500b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:09.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.166+0000 7fb74e7fc700 1 -- 172.21.15.27:0/306629959 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7500cd0e0 con 0x7fb7500b2b90 2024-01-11T08:10:09.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.166+0000 7fb74e7fc700 1 --2- 172.21.15.27:0/306629959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7500b2b90 0x7fb750159200 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb7500b1d10 tx=0x7fb744009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8e52b37f75f9528 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:09.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.166+0000 7fb74dffb700 1 -- 172.21.15.27:0/306629959 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb744009e70 con 0x7fb7500b2b90 2024-01-11T08:10:09.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.167+0000 7fb74dffb700 1 -- 172.21.15.27:0/306629959 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74400b860 con 0x7fb7500b2b90 2024-01-11T08:10:09.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.167+0000 7fb74dffb700 1 -- 172.21.15.27:0/306629959 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74400bc00 con 0x7fb7500b2b90 2024-01-11T08:10:09.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.168+0000 7fb7557ff700 1 -- 172.21.15.27:0/306629959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7500b2b90 msgr2=0x7fb750159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:09.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.168+0000 7fb7557ff700 1 --2- 172.21.15.27:0/306629959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7500b2b90 0x7fb750159200 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb7500b1d10 tx=0x7fb744009250 comp rx=0 tx=0).stop 2024-01-11T08:10:09.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.168+0000 7fb7557ff700 1 -- 172.21.15.27:0/306629959 shutdown_connections 2024-01-11T08:10:09.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.168+0000 7fb7557ff700 1 --2- 172.21.15.27:0/306629959 >> v2:172.21.15.77:3300/0 conn(0x7fb7500b07b0 0x7fb750159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:09.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.168+0000 7fb7557ff700 1 --2- 172.21.15.27:0/306629959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7500b2b90 0x7fb750159200 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:09.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.168+0000 7fb7557ff700 1 -- 172.21.15.27:0/306629959 >> 172.21.15.27:0/306629959 conn(0x7fb75006ed40 msgr2=0x7fb75006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:09.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.168+0000 7fb7557ff700 1 -- 172.21.15.27:0/306629959 shutdown_connections 2024-01-11T08:10:09.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.168+0000 7fb7557ff700 1 -- 172.21.15.27:0/306629959 wait complete. 2024-01-11T08:10:09.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.169+0000 7fb7557ff700 1 Processor -- start 2024-01-11T08:10:09.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.170+0000 7fb7557ff700 1 -- start start 2024-01-11T08:10:09.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.170+0000 7fb7557ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7500b03e0 0x7fb750158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:09.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.170+0000 7fb7557ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7500b07b0 0x7fb7500c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:09.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.170+0000 7fb7557ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7500b2b90 0x7fb7500ae460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:09.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.170+0000 7fb7557ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7500c4e20 con 0x7fb7500b03e0 2024-01-11T08:10:09.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.170+0000 7fb7557ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7500c5100 con 0x7fb7500b07b0 2024-01-11T08:10:09.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.170+0000 7fb7557ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7500c53e0 con 0x7fb7500b2b90 2024-01-11T08:10:09.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.171+0000 7fb74effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7500b03e0 0x7fb750158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:09.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.171+0000 7fb74effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7500b03e0 0x7fb750158bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57290/0 (socket says 172.21.15.27:57290) 2024-01-11T08:10:09.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.171+0000 7fb74effd700 1 -- 172.21.15.27:0/401189080 learned_addr learned my addr 172.21.15.27:0/401189080 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:09.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.171+0000 7fb74e7fc700 1 --2- 172.21.15.27:0/401189080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7500b07b0 0x7fb7500c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:09.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.171+0000 7fb74f7fe700 1 --2- 172.21.15.27:0/401189080 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7500b2b90 0x7fb7500ae460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:09.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.171+0000 7fb74effd700 1 -- 172.21.15.27:0/401189080 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7500b2b90 msgr2=0x7fb7500ae460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:09.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.171+0000 7fb74effd700 1 --2- 172.21.15.27:0/401189080 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7500b2b90 0x7fb7500ae460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:09.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.171+0000 7fb74effd700 1 -- 172.21.15.27:0/401189080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7500b07b0 msgr2=0x7fb7500c5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:09.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.171+0000 7fb74effd700 1 --2- 172.21.15.27:0/401189080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7500b07b0 0x7fb7500c5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:09.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.171+0000 7fb74effd700 1 -- 172.21.15.27:0/401189080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7440094f0 con 0x7fb7500b03e0 2024-01-11T08:10:09.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.171+0000 7fb74effd700 1 --2- 172.21.15.27:0/401189080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7500b03e0 0x7fb750158bd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb74000c590 tx=0x7fb74000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:09.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.172+0000 7fb736ffd700 1 -- 172.21.15.27:0/401189080 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb740011590 con 0x7fb7500b03e0 2024-01-11T08:10:09.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.172+0000 7fb736ffd700 1 -- 172.21.15.27:0/401189080 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb740011710 con 0x7fb7500b03e0 2024-01-11T08:10:09.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.172+0000 7fb736ffd700 1 -- 172.21.15.27:0/401189080 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb740011ad0 con 0x7fb7500b03e0 2024-01-11T08:10:09.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.172+0000 7fb7557ff700 1 -- 172.21.15.27:0/401189080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7500c56c0 con 0x7fb7500b03e0 2024-01-11T08:10:09.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.172+0000 7fb7557ff700 1 -- 172.21.15.27:0/401189080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7500c4860 con 0x7fb7500b03e0 2024-01-11T08:10:09.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.172+0000 7fb736ffd700 1 -- 172.21.15.27:0/401189080 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fb74001c6c0 con 0x7fb7500b03e0 2024-01-11T08:10:09.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.172+0000 7fb736ffd700 1 -- 172.21.15.27:0/401189080 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7fb740012b00 con 0x7fb7500b03e0 2024-01-11T08:10:09.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.173+0000 7fb7557ff700 1 -- 172.21.15.27:0/401189080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7500718d0 con 0x7fb7500b03e0 2024-01-11T08:10:09.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.176+0000 7fb736ffd700 1 -- 172.21.15.27:0/401189080 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fb740017070 con 0x7fb7500b03e0 2024-01-11T08:10:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.251+0000 7fb7557ff700 1 -- 172.21.15.27:0/401189080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7500c75c0 con 0x7fb7500b03e0 2024-01-11T08:10:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.252+0000 7fb736ffd700 1 -- 172.21.15.27:0/401189080 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7fb74001c970 con 0x7fb7500b03e0 2024-01-11T08:10:09.251 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:09.252 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":10,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:08.812242+0000","last_up_change":"0.000000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:10:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.253+0000 7fb7557ff700 1 -- 172.21.15.27:0/401189080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7500b03e0 msgr2=0x7fb750158bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.253+0000 7fb7557ff700 1 --2- 172.21.15.27:0/401189080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7500b03e0 0x7fb750158bd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb74000c590 tx=0x7fb74000c8a0 comp rx=0 tx=0).stop 2024-01-11T08:10:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.253+0000 7fb7557ff700 1 -- 172.21.15.27:0/401189080 shutdown_connections 2024-01-11T08:10:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.253+0000 7fb7557ff700 1 --2- 172.21.15.27:0/401189080 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7500b2b90 0x7fb7500ae460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.253+0000 7fb7557ff700 1 --2- 172.21.15.27:0/401189080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7500b03e0 0x7fb750158bd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.253+0000 7fb7557ff700 1 --2- 172.21.15.27:0/401189080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7500b07b0 0x7fb7500c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.253+0000 7fb7557ff700 1 -- 172.21.15.27:0/401189080 >> 172.21.15.27:0/401189080 conn(0x7fb75006ed40 msgr2=0x7fb7500d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.253+0000 7fb7557ff700 1 -- 172.21.15.27:0/401189080 shutdown_connections 2024-01-11T08:10:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:09.253+0000 7fb7557ff700 1 -- 172.21.15.27:0/401189080 wait complete. 2024-01-11T08:10:09.270 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-11T08:10:09.271 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-11T08:10:09.291 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:09.291+0000 7f78c72551c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:10:09.379 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:09.380+0000 7f78c72551c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:10:09.403 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:09.403+0000 7f62ffb111c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:10:09.494 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:09.495+0000 7f62ffb111c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:10:09.565 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-01-11T08:10:09.565+0000 7ff7d6636540 -1 Falling back to public interface 2024-01-11T08:10:09.568 INFO:tasks.ceph.osd.4.smithi077.stderr:2024-01-11T08:10:09.569+0000 7f987c6d4540 -1 Falling back to public interface 2024-01-11T08:10:09.576 INFO:tasks.ceph.osd.6.smithi077.stderr:2024-01-11T08:10:09.577+0000 7fc09c5fa540 -1 Falling back to public interface 2024-01-11T08:10:09.577 INFO:tasks.ceph.osd.5.smithi077.stderr:2024-01-11T08:10:09.577+0000 7fb1047a5540 -1 Falling back to public interface 2024-01-11T08:10:09.582 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-01-11T08:10:09.582+0000 7f5de4fc4540 -1 Falling back to public interface 2024-01-11T08:10:09.582 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-01-11T08:10:09.583+0000 7f43fe751540 -1 Falling back to public interface 2024-01-11T08:10:09.583 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-01-11T08:10:09.583+0000 7f5ddc1fd700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6809/0: (98) Address already in use 2024-01-11T08:10:09.583 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-01-11T08:10:09.584+0000 7f71a0ada540 -1 Falling back to public interface 2024-01-11T08:10:09.627 INFO:tasks.ceph.osd.7.smithi077.stderr:2024-01-11T08:10:09.628+0000 7fd5cfbd1540 -1 Falling back to public interface 2024-01-11T08:10:09.996 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:09.996+0000 7f78c72551c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:10:10.077 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-11T08:10:10.152 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:10.152+0000 7f62ffb111c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:10:10.254 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.254+0000 7f174a5ff700 1 Processor -- start 2024-01-11T08:10:10.254 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.254+0000 7f174a5ff700 1 -- start start 2024-01-11T08:10:10.255 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.255+0000 7f174a5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f17440ce040 0x7f174415a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:10.255 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.255+0000 7f174a5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f174413cb00 con 0x7f174406bf60 2024-01-11T08:10:10.255 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.255+0000 7f174a5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17440ce4e0 con 0x7f174413c730 2024-01-11T08:10:10.256 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.255+0000 7f174a5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17440686d0 con 0x7f17440ce040 2024-01-11T08:10:10.256 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.256+0000 7f17437fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f174406bf60 0x7f1744159d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57322/0 (socket says 172.21.15.27:57322) 2024-01-11T08:10:10.256 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.256+0000 7f17437fe700 1 -- 172.21.15.27:0/3604238635 learned_addr learned my addr 172.21.15.27:0/3604238635 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:10.256 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.256+0000 7f1748b9c700 1 --2- 172.21.15.27:0/3604238635 >> v2:172.21.15.77:3300/0 conn(0x7f17440ce040 0x7f174415a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:10.257 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.256+0000 7f1742ffd700 1 -- 172.21.15.27:0/3604238635 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576982569 0 0) 0x7f174413cb00 con 0x7f174406bf60 2024-01-11T08:10:10.257 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.256+0000 7f1742ffd700 1 -- 172.21.15.27:0/3604238635 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f172c003570 con 0x7f174406bf60 2024-01-11T08:10:10.258 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.256+0000 7f1742ffd700 1 -- 172.21.15.27:0/3604238635 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084408120 0 0) 0x7f17440ce4e0 con 0x7f174413c730 2024-01-11T08:10:10.258 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.256+0000 7f1748b9c700 1 -- 172.21.15.27:0/3604238635 >> v1:172.21.15.77:6789/0 conn(0x7f174413c730 legacy=0x7f174406b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:10.258 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.256+0000 7f1748b9c700 1 -- 172.21.15.27:0/3604238635 >> v1:172.21.15.27:6789/0 conn(0x7f174406bf60 legacy=0x7f1744159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:10.258 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.256+0000 7f1748b9c700 1 -- 172.21.15.27:0/3604238635 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17440ce7b0 con 0x7f17440ce040 2024-01-11T08:10:10.258 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.257+0000 7f1748b9c700 1 --2- 172.21.15.27:0/3604238635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17440ce040 0x7f174415a440 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f173c009e60 tx=0x7f173c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7b19875fa5d3db9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:10.258 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.257+0000 7f1742ffd700 1 -- 172.21.15.27:0/3604238635 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c00e070 con 0x7f17440ce040 2024-01-11T08:10:10.258 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.257+0000 7f1742ffd700 1 -- 172.21.15.27:0/3604238635 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173c00bb60 con 0x7f17440ce040 2024-01-11T08:10:10.258 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.257+0000 7f1742ffd700 1 -- 172.21.15.27:0/3604238635 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c012430 con 0x7f17440ce040 2024-01-11T08:10:10.258 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.257+0000 7f174a5ff700 1 -- 172.21.15.27:0/3604238635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17440ce040 msgr2=0x7f174415a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:10.258 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.257+0000 7f174a5ff700 1 --2- 172.21.15.27:0/3604238635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17440ce040 0x7f174415a440 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f173c009e60 tx=0x7f173c009780 comp rx=0 tx=0).stop 2024-01-11T08:10:10.258 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.257+0000 7f174a5ff700 1 -- 172.21.15.27:0/3604238635 shutdown_connections 2024-01-11T08:10:10.258 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.257+0000 7f174a5ff700 1 --2- 172.21.15.27:0/3604238635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17440ce040 0x7f174415a440 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:10.258 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.257+0000 7f174a5ff700 1 -- 172.21.15.27:0/3604238635 >> 172.21.15.27:0/3604238635 conn(0x7f17440632c0 msgr2=0x7f17440636c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:10.258 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.258+0000 7f174a5ff700 1 -- 172.21.15.27:0/3604238635 shutdown_connections 2024-01-11T08:10:10.259 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.258+0000 7f174a5ff700 1 -- 172.21.15.27:0/3604238635 wait complete. 2024-01-11T08:10:10.259 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.259+0000 7f174a5ff700 1 Processor -- start 2024-01-11T08:10:10.259 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.259+0000 7f174a5ff700 1 -- start start 2024-01-11T08:10:10.260 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.260+0000 7f174a5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f174406bf60 0x7f17440c5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:10.260 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.260+0000 7f174a5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17440ce040 0x7f174413bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:10.260 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.260+0000 7f174a5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f174413c730 0x7f17440b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:10.261 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.260+0000 7f174a5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17440c6a00 con 0x7f174413c730 2024-01-11T08:10:10.261 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.260+0000 7f174a5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17440c5df0 con 0x7f17440ce040 2024-01-11T08:10:10.261 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.260+0000 7f174a5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17440c60d0 con 0x7f174406bf60 2024-01-11T08:10:10.261 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.260+0000 7f1748b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f174413c730 0x7f17440b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:10.261 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.261+0000 7f17437fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17440ce040 0x7f174413bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:10.261 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.261+0000 7f1748b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f174413c730 0x7f17440b26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57300/0 (socket says 172.21.15.27:57300) 2024-01-11T08:10:10.261 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.261+0000 7f1748b9c700 1 -- 172.21.15.27:0/3769484961 learned_addr learned my addr 172.21.15.27:0/3769484961 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:10.261 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.261+0000 7f1743fff700 1 --2- 172.21.15.27:0/3769484961 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f174406bf60 0x7f17440c5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:10.263 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.261+0000 7f1748b9c700 1 -- 172.21.15.27:0/3769484961 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f174406bf60 msgr2=0x7f17440c5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:10.263 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.261+0000 7f1748b9c700 1 --2- 172.21.15.27:0/3769484961 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f174406bf60 0x7f17440c5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:10.263 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.261+0000 7f1748b9c700 1 -- 172.21.15.27:0/3769484961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17440ce040 msgr2=0x7f174413bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:10.263 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.261+0000 7f1748b9c700 1 --2- 172.21.15.27:0/3769484961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17440ce040 0x7f174413bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:10.263 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.261+0000 7f1748b9c700 1 -- 172.21.15.27:0/3769484961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f173c009a20 con 0x7f174413c730 2024-01-11T08:10:10.263 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.261+0000 7f17437fe700 1 --2- 172.21.15.27:0/3769484961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17440ce040 0x7f174413bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:10.263 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.261+0000 7f1748b9c700 1 --2- 172.21.15.27:0/3769484961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f174413c730 0x7f17440b26e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f17440c5400 tx=0x7f173c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:10.263 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.262+0000 7f172bfff700 1 -- 172.21.15.27:0/3769484961 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c026480 con 0x7f174413c730 2024-01-11T08:10:10.264 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.262+0000 7f172bfff700 1 -- 172.21.15.27:0/3769484961 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173c026640 con 0x7f174413c730 2024-01-11T08:10:10.264 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.262+0000 7f172bfff700 1 -- 172.21.15.27:0/3769484961 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c026a00 con 0x7f174413c730 2024-01-11T08:10:10.264 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.262+0000 7f174a5ff700 1 -- 172.21.15.27:0/3769484961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17440c63b0 con 0x7f174413c730 2024-01-11T08:10:10.264 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.262+0000 7f174a5ff700 1 -- 172.21.15.27:0/3769484961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17440c59c0 con 0x7f174413c730 2024-01-11T08:10:10.264 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.262+0000 7f172bfff700 1 -- 172.21.15.27:0/3769484961 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f173c01c400 con 0x7f174413c730 2024-01-11T08:10:10.264 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.262+0000 7f172bfff700 1 -- 172.21.15.27:0/3769484961 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f173c026db0 con 0x7f174413c730 2024-01-11T08:10:10.264 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.263+0000 7f174a5ff700 1 -- 172.21.15.27:0/3769484961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f174404c3c0 con 0x7f174413c730 2024-01-11T08:10:10.265 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.266+0000 7f172bfff700 1 -- 172.21.15.27:0/3769484961 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f173c02b020 con 0x7f174413c730 2024-01-11T08:10:10.340 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.339+0000 7f174a5ff700 1 -- 172.21.15.27:0/3769484961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17440c6b40 con 0x7f174413c730 2024-01-11T08:10:10.340 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.340+0000 7f172bfff700 1 -- 172.21.15.27:0/3769484961 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f173c01c6b0 con 0x7f174413c730 2024-01-11T08:10:10.340 INFO:teuthology.misc.health.smithi027.stdout: 2024-01-11T08:10:10.340 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":10,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:08.812242+0000","last_up_change":"0.000000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:10:10.342 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.341+0000 7f174a5ff700 1 -- 172.21.15.27:0/3769484961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f174413c730 msgr2=0x7f17440b26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:10.342 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.341+0000 7f174a5ff700 1 --2- 172.21.15.27:0/3769484961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f174413c730 0x7f17440b26e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f17440c5400 tx=0x7f173c019000 comp rx=0 tx=0).stop 2024-01-11T08:10:10.342 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.341+0000 7f174a5ff700 1 -- 172.21.15.27:0/3769484961 shutdown_connections 2024-01-11T08:10:10.342 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.341+0000 7f174a5ff700 1 --2- 172.21.15.27:0/3769484961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f174413c730 0x7f17440b26e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:10.342 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.341+0000 7f174a5ff700 1 --2- 172.21.15.27:0/3769484961 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f174406bf60 0x7f17440c5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:10.342 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.341+0000 7f174a5ff700 1 --2- 172.21.15.27:0/3769484961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17440ce040 0x7f174413bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:10.342 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.341+0000 7f174a5ff700 1 -- 172.21.15.27:0/3769484961 >> 172.21.15.27:0/3769484961 conn(0x7f17440632c0 msgr2=0x7f1744069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:10.342 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.341+0000 7f174a5ff700 1 -- 172.21.15.27:0/3769484961 shutdown_connections 2024-01-11T08:10:10.342 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:10.341+0000 7f174a5ff700 1 -- 172.21.15.27:0/3769484961 wait complete. 2024-01-11T08:10:10.355 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-11T08:10:10.375 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:10.375+0000 7f78c72551c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:10:10.553 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:10.554+0000 7f62ffb111c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:10:10.554 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:10.555+0000 7f78c72551c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:10:10.739 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:10.739+0000 7f62ffb111c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:10:10.993 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:10.994+0000 7f78c72551c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:10:11.174 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:11.175+0000 7f78c72551c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:10:11.199 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:11.200+0000 7f62ffb111c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:10:11.366 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:11.367+0000 7f78c72551c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:10:11.388 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:11.388+0000 7f62ffb111c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:10:11.491 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:11.492+0000 7f78c72551c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:10:11.592 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:11.592+0000 7f62ffb111c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:10:11.723 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:10:11.723+0000 7f62ffb111c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:10:12.519 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:12.519+0000 7f78865fa700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-01-11T08:10:12.520 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:10:12.520+0000 7f78865fa700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-01-11T08:10:13.103 INFO:tasks.ceph.osd.4.smithi077.stderr:2024-01-11T08:10:13.104+0000 7f987c6d4540 -1 osd.4 0 log_to_monitors true 2024-01-11T08:10:13.104 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-01-11T08:10:13.105+0000 7ff7d6636540 -1 osd.0 0 log_to_monitors true 2024-01-11T08:10:13.147 INFO:tasks.ceph.osd.5.smithi077.stderr:2024-01-11T08:10:13.148+0000 7fb1047a5540 -1 osd.5 0 log_to_monitors true 2024-01-11T08:10:13.164 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-01-11T08:10:13.164+0000 7f43fe751540 -1 osd.2 0 log_to_monitors true 2024-01-11T08:10:13.164 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-01-11T08:10:13.164+0000 7f71a0ada540 -1 osd.1 0 log_to_monitors true 2024-01-11T08:10:13.167 INFO:tasks.ceph.osd.6.smithi077.stderr:2024-01-11T08:10:13.168+0000 7fc09c5fa540 -1 osd.6 0 log_to_monitors true 2024-01-11T08:10:13.167 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-01-11T08:10:13.168+0000 7f5de4fc4540 -1 osd.3 0 log_to_monitors true 2024-01-11T08:10:13.178 INFO:tasks.ceph.osd.7.smithi077.stderr:2024-01-11T08:10:13.178+0000 7fd5cfbd1540 -1 osd.7 0 log_to_monitors true 2024-01-11T08:10:16.977 INFO:tasks.ceph.osd.7.smithi077.stderr:2024-01-11T08:10:16.978+0000 7fd5c35f6700 -1 osd.7 0 waiting for initial osdmap 2024-01-11T08:10:16.986 INFO:tasks.ceph.osd.7.smithi077.stderr:2024-01-11T08:10:16.987+0000 7fd5bc5e8700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-11T08:10:17.017 INFO:tasks.ceph.osd.4.smithi077.stderr:2024-01-11T08:10:17.017+0000 7f98701f6700 -1 osd.4 0 waiting for initial osdmap 2024-01-11T08:10:17.023 INFO:tasks.ceph.osd.4.smithi077.stderr:2024-01-11T08:10:17.024+0000 7f98691e8700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-11T08:10:17.038 INFO:tasks.ceph.osd.6.smithi077.stderr:2024-01-11T08:10:17.039+0000 7fc08fff6700 -1 osd.6 0 waiting for initial osdmap 2024-01-11T08:10:17.043 INFO:tasks.ceph.osd.6.smithi077.stderr:2024-01-11T08:10:17.044+0000 7fc088fe8700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-11T08:10:17.046 INFO:tasks.ceph.osd.5.smithi077.stderr:2024-01-11T08:10:17.047+0000 7fb0f81f6700 -1 osd.5 0 waiting for initial osdmap 2024-01-11T08:10:17.054 INFO:tasks.ceph.osd.5.smithi077.stderr:2024-01-11T08:10:17.054+0000 7fb0f11e8700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-11T08:10:17.077 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-01-11T08:10:17.078+0000 7f5dda1f9700 -1 osd.3 0 waiting for initial osdmap 2024-01-11T08:10:17.090 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-01-11T08:10:17.091+0000 7f5dd19e8700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-11T08:10:17.122 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-01-11T08:10:17.123+0000 7f7195df9700 -1 osd.1 0 waiting for initial osdmap 2024-01-11T08:10:17.123 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-01-11T08:10:17.123+0000 7f43f21f6700 -1 osd.2 0 waiting for initial osdmap 2024-01-11T08:10:17.131 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-01-11T08:10:17.132+0000 7f718d5e8700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-11T08:10:17.132 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-01-11T08:10:17.132+0000 7f43eb1e8700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-11T08:10:17.157 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-01-11T08:10:17.157+0000 7ff7ca1f6700 -1 osd.0 0 waiting for initial osdmap 2024-01-11T08:10:17.161 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-11T08:10:17.166 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-01-11T08:10:17.164+0000 7ff7c31e8700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-11T08:10:17.336 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.336+0000 7f85003ff700 1 Processor -- start 2024-01-11T08:10:17.336 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.336+0000 7f85003ff700 1 -- start start 2024-01-11T08:10:17.338 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.337+0000 7f85003ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f84f80cb100 0x7f84f80af920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:17.338 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.337+0000 7f85003ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f80ccd00 0x7f84f80c9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:17.338 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.338+0000 7f85003ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f80accc0 con 0x7f84f80b0600 2024-01-11T08:10:17.338 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.338+0000 7f85003ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84f80ad560 con 0x7f84f80cb100 2024-01-11T08:10:17.338 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.338+0000 7f85003ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f8156870 con 0x7f84f80ccd00 2024-01-11T08:10:17.338 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.338+0000 7f84fe19b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f84f80cb100 0x7f84f80af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:17.338 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.338+0000 7f84fe19b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f84f80cb100 0x7f84f80af920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52762/0 (socket says 172.21.15.27:52762) 2024-01-11T08:10:17.338 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.338+0000 7f84fe19b700 1 -- 172.21.15.27:0/1635438827 learned_addr learned my addr 172.21.15.27:0/1635438827 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:17.338 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.338+0000 7f84fd99a700 1 --2- 172.21.15.27:0/1635438827 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f80ccd00 0x7f84f80c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:17.338 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.338+0000 7f84fe19b700 1 -- 172.21.15.27:0/1635438827 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f80ccd00 msgr2=0x7f84f80c9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:17.339 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.338+0000 7f84fe19b700 1 --2- 172.21.15.27:0/1635438827 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f80ccd00 0x7f84f80c9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:17.339 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.338+0000 7f84fe19b700 1 -- 172.21.15.27:0/1635438827 >> v1:172.21.15.77:6789/0 conn(0x7f84f80b0600 legacy=0x7f84f80ad9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:17.339 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.338+0000 7f84fd199700 1 -- 172.21.15.27:0/1635438827 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409050145 0 0) 0x7f84f80accc0 con 0x7f84f80b0600 2024-01-11T08:10:17.339 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.338+0000 7f84fe19b700 1 -- 172.21.15.27:0/1635438827 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f80cb4d0 con 0x7f84f80cb100 2024-01-11T08:10:17.340 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.339+0000 7f84fe19b700 1 --2- 172.21.15.27:0/1635438827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f80cb100 0x7f84f80af920 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f84f400bc50 tx=0x7f84f400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bea3262c97b4e0bf server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:17.340 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.339+0000 7f84fd199700 1 -- 172.21.15.27:0/1635438827 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f400dbf0 con 0x7f84f80cb100 2024-01-11T08:10:17.340 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.339+0000 7f84fd199700 1 -- 172.21.15.27:0/1635438827 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f400dd50 con 0x7f84f80cb100 2024-01-11T08:10:17.340 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.339+0000 7f84fd199700 1 -- 172.21.15.27:0/1635438827 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f40196b0 con 0x7f84f80cb100 2024-01-11T08:10:17.340 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.339+0000 7f85003ff700 1 -- 172.21.15.27:0/1635438827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f80cb100 msgr2=0x7f84f80af920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:17.340 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.339+0000 7f85003ff700 1 --2- 172.21.15.27:0/1635438827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f80cb100 0x7f84f80af920 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f84f400bc50 tx=0x7f84f400b4f0 comp rx=0 tx=0).stop 2024-01-11T08:10:17.340 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.339+0000 7f85003ff700 1 -- 172.21.15.27:0/1635438827 shutdown_connections 2024-01-11T08:10:17.340 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.339+0000 7f85003ff700 1 --2- 172.21.15.27:0/1635438827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f80cb100 0x7f84f80af920 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:17.340 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.339+0000 7f85003ff700 1 --2- 172.21.15.27:0/1635438827 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f80ccd00 0x7f84f80c9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:17.341 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.339+0000 7f85003ff700 1 -- 172.21.15.27:0/1635438827 >> 172.21.15.27:0/1635438827 conn(0x7f84f8068040 msgr2=0x7f84f80cc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:17.341 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.340+0000 7f85003ff700 1 -- 172.21.15.27:0/1635438827 shutdown_connections 2024-01-11T08:10:17.341 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.340+0000 7f85003ff700 1 -- 172.21.15.27:0/1635438827 wait complete. 2024-01-11T08:10:17.341 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.341+0000 7f85003ff700 1 Processor -- start 2024-01-11T08:10:17.341 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.341+0000 7f85003ff700 1 -- start start 2024-01-11T08:10:17.343 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.342+0000 7f85003ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84f80b0600 0x7f84f815b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:17.343 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.342+0000 7f85003ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f80cb100 0x7f84f8145be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:17.343 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.342+0000 7f85003ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f80ccd00 0x7f84f806e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:17.343 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.342+0000 7f85003ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f815b810 con 0x7f84f80cb100 2024-01-11T08:10:17.343 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.342+0000 7f85003ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f815c600 con 0x7f84f80b0600 2024-01-11T08:10:17.344 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.342+0000 7f85003ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f815b9f0 con 0x7f84f80ccd00 2024-01-11T08:10:17.344 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.342+0000 7f84fd99a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f80cb100 0x7f84f8145be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:17.344 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.342+0000 7f84fd99a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f80cb100 0x7f84f8145be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52772/0 (socket says 172.21.15.27:52772) 2024-01-11T08:10:17.344 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.342+0000 7f84fd99a700 1 -- 172.21.15.27:0/535902373 learned_addr learned my addr 172.21.15.27:0/535902373 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:17.344 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.343+0000 7f84fe99c700 1 --2- 172.21.15.27:0/535902373 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f80ccd00 0x7f84f806e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:17.344 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.343+0000 7f84fe19b700 1 --2- 172.21.15.27:0/535902373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84f80b0600 0x7f84f815b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:17.344 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.343+0000 7f84fd99a700 1 -- 172.21.15.27:0/535902373 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f80ccd00 msgr2=0x7f84f806e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:17.344 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.343+0000 7f84fd99a700 1 --2- 172.21.15.27:0/535902373 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f80ccd00 0x7f84f806e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:17.344 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.343+0000 7f84fd99a700 1 -- 172.21.15.27:0/535902373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84f80b0600 msgr2=0x7f84f815b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:17.344 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.343+0000 7f84fd99a700 1 --2- 172.21.15.27:0/535902373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84f80b0600 0x7f84f815b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:17.344 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.343+0000 7f84fd99a700 1 -- 172.21.15.27:0/535902373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f400b790 con 0x7f84f80cb100 2024-01-11T08:10:17.344 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.343+0000 7f84fe19b700 1 --2- 172.21.15.27:0/535902373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84f80b0600 0x7f84f815b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:17.344 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.343+0000 7f84fe99c700 1 --2- 172.21.15.27:0/535902373 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f80ccd00 0x7f84f806e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:17.344 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.343+0000 7f84fd99a700 1 --2- 172.21.15.27:0/535902373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f80cb100 0x7f84f8145be0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f84e800e930 tx=0x7f84e80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:17.345 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.343+0000 7f84ee7fc700 1 -- 172.21.15.27:0/535902373 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e800eed0 con 0x7f84f80cb100 2024-01-11T08:10:17.345 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.343+0000 7f84ee7fc700 1 -- 172.21.15.27:0/535902373 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e800c850 con 0x7f84f80cb100 2024-01-11T08:10:17.345 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.343+0000 7f85003ff700 1 -- 172.21.15.27:0/535902373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f815bcd0 con 0x7f84f80cb100 2024-01-11T08:10:17.345 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.343+0000 7f84ee7fc700 1 -- 172.21.15.27:0/535902373 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e800cbf0 con 0x7f84f80cb100 2024-01-11T08:10:17.345 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.343+0000 7f85003ff700 1 -- 172.21.15.27:0/535902373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f815c290 con 0x7f84f80cb100 2024-01-11T08:10:17.345 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.344+0000 7f85003ff700 1 -- 172.21.15.27:0/535902373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f815c790 con 0x7f84f80cb100 2024-01-11T08:10:17.345 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.345+0000 7f84ee7fc700 1 -- 172.21.15.27:0/535902373 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f84e8013470 con 0x7f84f80cb100 2024-01-11T08:10:17.346 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.346+0000 7f84ee7fc700 1 --2- 172.21.15.27:0/535902373 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f84dc066ea0 0x7f84dc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:17.346 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.346+0000 7f84ee7fc700 1 -- 172.21.15.27:0/535902373 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2227+0+0 (secure 0 0 0) 0x7f84e8083990 con 0x7f84f80cb100 2024-01-11T08:10:17.346 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.346+0000 7f84fe19b700 1 --2- 172.21.15.27:0/535902373 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f84dc066ea0 0x7f84dc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:17.347 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.347+0000 7f84fe19b700 1 --2- 172.21.15.27:0/535902373 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f84dc066ea0 0x7f84dc069350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f84f400bc50 tx=0x7f84f401b000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:17.351 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.352+0000 7f84ee7fc700 1 -- 172.21.15.27:0/535902373 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f84e80519d0 con 0x7f84f80cb100 2024-01-11T08:10:17.555 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.555+0000 7f85003ff700 1 -- 172.21.15.27:0/535902373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84f804c3c0 con 0x7f84f80cb100 2024-01-11T08:10:17.555 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.556+0000 7f84ee7fc700 1 -- 172.21.15.27:0/535902373 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f84e8054ff0 con 0x7f84f80cb100 2024-01-11T08:10:17.556 INFO:teuthology.misc.health.smithi027.stdout: 2024-01-11T08:10:17.556 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":12,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:14.719818+0000","last_up_change":"0.000000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:10:17.558 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.558+0000 7f85003ff700 1 -- 172.21.15.27:0/535902373 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f84dc066ea0 msgr2=0x7f84dc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:17.558 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.558+0000 7f85003ff700 1 --2- 172.21.15.27:0/535902373 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f84dc066ea0 0x7f84dc069350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f84f400bc50 tx=0x7f84f401b000 comp rx=0 tx=0).stop 2024-01-11T08:10:17.558 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.558+0000 7f85003ff700 1 -- 172.21.15.27:0/535902373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f80cb100 msgr2=0x7f84f8145be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:17.559 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.558+0000 7f85003ff700 1 --2- 172.21.15.27:0/535902373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f80cb100 0x7f84f8145be0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f84e800e930 tx=0x7f84e80091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:17.559 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.558+0000 7f85003ff700 1 -- 172.21.15.27:0/535902373 shutdown_connections 2024-01-11T08:10:17.559 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.558+0000 7f85003ff700 1 --2- 172.21.15.27:0/535902373 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f84dc066ea0 0x7f84dc069350 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:17.559 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.558+0000 7f85003ff700 1 --2- 172.21.15.27:0/535902373 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f80ccd00 0x7f84f806e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:17.559 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.558+0000 7f85003ff700 1 --2- 172.21.15.27:0/535902373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84f80b0600 0x7f84f815b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:17.559 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.558+0000 7f85003ff700 1 --2- 172.21.15.27:0/535902373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f80cb100 0x7f84f8145be0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:17.559 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.558+0000 7f85003ff700 1 -- 172.21.15.27:0/535902373 >> 172.21.15.27:0/535902373 conn(0x7f84f8068040 msgr2=0x7f84f80abbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:17.559 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.558+0000 7f85003ff700 1 -- 172.21.15.27:0/535902373 shutdown_connections 2024-01-11T08:10:17.559 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:17.559+0000 7f85003ff700 1 -- 172.21.15.27:0/535902373 wait complete. 2024-01-11T08:10:17.573 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-11T08:10:18.424 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:10:18.429 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:10:24.380 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-11T08:10:24.563 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.563+0000 7ff7f3dff700 1 Processor -- start 2024-01-11T08:10:24.563 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.563+0000 7ff7f3dff700 1 -- start start 2024-01-11T08:10:24.566 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.565+0000 7ff7f3dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff7ec0cd740 0x7ff7ec0c60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:24.567 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.565+0000 7ff7f3dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff7ec0cf920 0x7ff7ec0c6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:24.567 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.565+0000 7ff7f3dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7ec0cbf90 con 0x7ff7ec159830 2024-01-11T08:10:24.567 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.565+0000 7ff7f3dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7ec0c6b40 con 0x7ff7ec0cf920 2024-01-11T08:10:24.567 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.565+0000 7ff7f3dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7ec0cc230 con 0x7ff7ec0cd740 2024-01-11T08:10:24.567 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.565+0000 7ff7f1b9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff7ec159830 0x7ff7ec0cd030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37008/0 (socket says 172.21.15.27:37008) 2024-01-11T08:10:24.567 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.565+0000 7ff7f1b9b700 1 -- 172.21.15.27:0/2453842897 learned_addr learned my addr 172.21.15.27:0/2453842897 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:24.567 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.565+0000 7ff7f239c700 1 --2- 172.21.15.27:0/2453842897 >> v2:172.21.15.27:3300/0 conn(0x7ff7ec0cf920 0x7ff7ec0c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:24.567 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.565+0000 7ff7f139a700 1 --2- 172.21.15.27:0/2453842897 >> v2:172.21.15.77:3300/0 conn(0x7ff7ec0cd740 0x7ff7ec0c60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:24.567 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.565+0000 7ff7f0b99700 1 -- 172.21.15.27:0/2453842897 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278648123 0 0) 0x7ff7ec0cbf90 con 0x7ff7ec159830 2024-01-11T08:10:24.567 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.565+0000 7ff7f0b99700 1 -- 172.21.15.27:0/2453842897 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7d8003570 con 0x7ff7ec159830 2024-01-11T08:10:24.567 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.566+0000 7ff7f239c700 1 -- 172.21.15.27:0/2453842897 >> v2:172.21.15.77:3300/0 conn(0x7ff7ec0cd740 msgr2=0x7ff7ec0c60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:24.568 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.566+0000 7ff7f239c700 1 --2- 172.21.15.27:0/2453842897 >> v2:172.21.15.77:3300/0 conn(0x7ff7ec0cd740 0x7ff7ec0c60c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:24.568 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.566+0000 7ff7f239c700 1 -- 172.21.15.27:0/2453842897 >> v1:172.21.15.27:6789/0 conn(0x7ff7ec159830 legacy=0x7ff7ec0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:24.568 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.566+0000 7ff7f239c700 1 -- 172.21.15.27:0/2453842897 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ec0cc510 con 0x7ff7ec0cf920 2024-01-11T08:10:24.568 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.566+0000 7ff7f139a700 1 --2- 172.21.15.27:0/2453842897 >> v2:172.21.15.77:3300/0 conn(0x7ff7ec0cd740 0x7ff7ec0c60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:24.568 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.566+0000 7ff7f239c700 1 --2- 172.21.15.27:0/2453842897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7ec0cf920 0x7ff7ec0c6600 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec0b6630 tx=0x7ff7e4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd75719cfd19919f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:24.568 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.566+0000 7ff7f0b99700 1 -- 172.21.15.27:0/2453842897 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e400e070 con 0x7ff7ec0cf920 2024-01-11T08:10:24.568 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.566+0000 7ff7f0b99700 1 -- 172.21.15.27:0/2453842897 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e400bab0 con 0x7ff7ec0cf920 2024-01-11T08:10:24.568 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.566+0000 7ff7f0b99700 1 -- 172.21.15.27:0/2453842897 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e400be70 con 0x7ff7ec0cf920 2024-01-11T08:10:24.568 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.567+0000 7ff7f3dff700 1 -- 172.21.15.27:0/2453842897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7ec0cf920 msgr2=0x7ff7ec0c6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:24.568 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.567+0000 7ff7f3dff700 1 --2- 172.21.15.27:0/2453842897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7ec0cf920 0x7ff7ec0c6600 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec0b6630 tx=0x7ff7e4009780 comp rx=0 tx=0).stop 2024-01-11T08:10:24.569 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.567+0000 7ff7f3dff700 1 -- 172.21.15.27:0/2453842897 shutdown_connections 2024-01-11T08:10:24.569 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.567+0000 7ff7f3dff700 1 --2- 172.21.15.27:0/2453842897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7ec0cf920 0x7ff7ec0c6600 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:24.569 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.567+0000 7ff7f3dff700 1 --2- 172.21.15.27:0/2453842897 >> v2:172.21.15.77:3300/0 conn(0x7ff7ec0cd740 0x7ff7ec0c60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:24.569 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.567+0000 7ff7f3dff700 1 -- 172.21.15.27:0/2453842897 >> 172.21.15.27:0/2453842897 conn(0x7ff7ec0998e0 msgr2=0x7ff7ec06f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:24.569 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.567+0000 7ff7f3dff700 1 -- 172.21.15.27:0/2453842897 shutdown_connections 2024-01-11T08:10:24.569 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.567+0000 7ff7f3dff700 1 -- 172.21.15.27:0/2453842897 wait complete. 2024-01-11T08:10:24.571 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.568+0000 7ff7f3dff700 1 Processor -- start 2024-01-11T08:10:24.571 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.568+0000 7ff7f3dff700 1 -- start start 2024-01-11T08:10:24.572 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.569+0000 7ff7f3dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff7ec0cd740 0x7ff7ec09c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:24.572 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.569+0000 7ff7f3dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7ec0cf920 0x7ff7ec09a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:24.572 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.569+0000 7ff7f3dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7ec159830 0x7ff7ec1472e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:24.572 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.569+0000 7ff7f3dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ec09dba0 con 0x7ff7ec0cf920 2024-01-11T08:10:24.572 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.569+0000 7ff7f3dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ec09a950 con 0x7ff7ec159830 2024-01-11T08:10:24.572 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.569+0000 7ff7f3dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ec09cf90 con 0x7ff7ec0cd740 2024-01-11T08:10:24.572 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.570+0000 7ff7f139a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7ec0cf920 0x7ff7ec09a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:24.572 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.570+0000 7ff7f139a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7ec0cf920 0x7ff7ec09a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39982/0 (socket says 172.21.15.27:39982) 2024-01-11T08:10:24.572 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.570+0000 7ff7f139a700 1 -- 172.21.15.27:0/3766446460 learned_addr learned my addr 172.21.15.27:0/3766446460 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:24.573 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.570+0000 7ff7f239c700 1 --2- 172.21.15.27:0/3766446460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7ec159830 0x7ff7ec1472e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:24.573 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.570+0000 7ff7f1b9b700 1 --2- 172.21.15.27:0/3766446460 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff7ec0cd740 0x7ff7ec09c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:24.573 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.570+0000 7ff7f139a700 1 -- 172.21.15.27:0/3766446460 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff7ec0cd740 msgr2=0x7ff7ec09c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:24.573 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.570+0000 7ff7f139a700 1 --2- 172.21.15.27:0/3766446460 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff7ec0cd740 0x7ff7ec09c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:24.573 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.570+0000 7ff7f139a700 1 -- 172.21.15.27:0/3766446460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7ec159830 msgr2=0x7ff7ec1472e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:24.573 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.570+0000 7ff7f139a700 1 --2- 172.21.15.27:0/3766446460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7ec159830 0x7ff7ec1472e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:24.573 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.570+0000 7ff7f139a700 1 -- 172.21.15.27:0/3766446460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7dc009420 con 0x7ff7ec0cf920 2024-01-11T08:10:24.573 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.570+0000 7ff7f139a700 1 --2- 172.21.15.27:0/3766446460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7ec0cf920 0x7ff7ec09a410 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc00ec80 tx=0x7ff7dc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:24.574 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.570+0000 7ff7e1ffb700 1 -- 172.21.15.27:0/3766446460 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc00c850 con 0x7ff7ec0cf920 2024-01-11T08:10:24.574 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.571+0000 7ff7e1ffb700 1 -- 172.21.15.27:0/3766446460 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7dc00c9b0 con 0x7ff7ec0cf920 2024-01-11T08:10:24.574 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.571+0000 7ff7f3dff700 1 -- 172.21.15.27:0/3766446460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e4009a20 con 0x7ff7ec0cf920 2024-01-11T08:10:24.574 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.571+0000 7ff7e1ffb700 1 -- 172.21.15.27:0/3766446460 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc00cd30 con 0x7ff7ec0cf920 2024-01-11T08:10:24.574 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.571+0000 7ff7f3dff700 1 -- 172.21.15.27:0/3766446460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7ec09d550 con 0x7ff7ec0cf920 2024-01-11T08:10:24.574 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.572+0000 7ff7f3dff700 1 -- 172.21.15.27:0/3766446460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7ec0cddf0 con 0x7ff7ec0cf920 2024-01-11T08:10:24.575 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.572+0000 7ff7e1ffb700 1 -- 172.21.15.27:0/3766446460 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff7dc013680 con 0x7ff7ec0cf920 2024-01-11T08:10:24.575 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.574+0000 7ff7e1ffb700 1 --2- 172.21.15.27:0/3766446460 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff7d0066f30 0x7ff7d00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:24.575 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.574+0000 7ff7e1ffb700 1 -- 172.21.15.27:0/3766446460 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7ff7dc0844f0 con 0x7ff7ec0cf920 2024-01-11T08:10:24.575 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.574+0000 7ff7f1b9b700 1 --2- 172.21.15.27:0/3766446460 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff7d0066f30 0x7ff7d00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:24.575 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.575+0000 7ff7f1b9b700 1 --2- 172.21.15.27:0/3766446460 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff7d0066f30 0x7ff7d00693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8001360 tx=0x7ff7e8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:24.579 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.579+0000 7ff7e1ffb700 1 -- 172.21.15.27:0/3766446460 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff7dc055540 con 0x7ff7ec0cf920 2024-01-11T08:10:24.780 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.781+0000 7ff7f3dff700 1 -- 172.21.15.27:0/3766446460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7ec04c3c0 con 0x7ff7ec0cf920 2024-01-11T08:10:24.781 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.781+0000 7ff7e1ffb700 1 -- 172.21.15.27:0/3766446460 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12079 (secure 0 0 0) 0x7ff7dc0550d0 con 0x7ff7ec0cf920 2024-01-11T08:10:24.782 INFO:teuthology.misc.health.smithi027.stdout: 2024-01-11T08:10:24.782 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":17,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:22.473493+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:10:24.784 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.784+0000 7ff7f3dff700 1 -- 172.21.15.27:0/3766446460 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff7d0066f30 msgr2=0x7ff7d00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:24.784 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.784+0000 7ff7f3dff700 1 --2- 172.21.15.27:0/3766446460 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff7d0066f30 0x7ff7d00693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8001360 tx=0x7ff7e8009040 comp rx=0 tx=0).stop 2024-01-11T08:10:24.784 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.784+0000 7ff7f3dff700 1 -- 172.21.15.27:0/3766446460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7ec0cf920 msgr2=0x7ff7ec09a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:24.785 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.784+0000 7ff7f3dff700 1 --2- 172.21.15.27:0/3766446460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7ec0cf920 0x7ff7ec09a410 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc00ec80 tx=0x7ff7dc0091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:24.785 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.784+0000 7ff7f3dff700 1 -- 172.21.15.27:0/3766446460 shutdown_connections 2024-01-11T08:10:24.785 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.784+0000 7ff7f3dff700 1 --2- 172.21.15.27:0/3766446460 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff7d0066f30 0x7ff7d00693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:24.785 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.784+0000 7ff7f3dff700 1 --2- 172.21.15.27:0/3766446460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7ec159830 0x7ff7ec1472e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:24.785 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.784+0000 7ff7f3dff700 1 --2- 172.21.15.27:0/3766446460 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff7ec0cd740 0x7ff7ec09c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:24.785 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.784+0000 7ff7f3dff700 1 --2- 172.21.15.27:0/3766446460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7ec0cf920 0x7ff7ec09a410 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:24.785 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.784+0000 7ff7f3dff700 1 -- 172.21.15.27:0/3766446460 >> 172.21.15.27:0/3766446460 conn(0x7ff7ec0998e0 msgr2=0x7ff7ec155200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:24.785 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.784+0000 7ff7f3dff700 1 -- 172.21.15.27:0/3766446460 shutdown_connections 2024-01-11T08:10:24.785 INFO:teuthology.misc.health.smithi027.stderr:2024-01-11T08:10:24.784+0000 7ff7f3dff700 1 -- 172.21.15.27:0/3766446460 wait complete. 2024-01-11T08:10:24.799 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-01-11T08:10:24.799 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-11T08:10:24.800 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-11T08:10:24.800 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-11T08:10:24.805 INFO:tasks.ceph.mds.a:Started 2024-01-11T08:10:24.805 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-11T08:10:24.805 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-11T08:10:24.809 INFO:tasks.ceph.mds.c:Started 2024-01-11T08:10:24.809 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-11T08:10:24.809 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-11T08:10:24.814 INFO:tasks.ceph.mds.b:Started 2024-01-11T08:10:24.814 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-11T08:10:24.814 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-11T08:10:24.818 INFO:tasks.ceph.mds.d:Started 2024-01-11T08:10:24.818 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-01-11T08:10:24.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:10:24.959 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T08:10:24.959 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T08:10:24.961 INFO:tasks.ceph.mds.b.smithi077.stdout:starting mds.b at 2024-01-11T08:10:24.961 INFO:tasks.ceph.mds.d.smithi077.stdout:starting mds.d at 2024-01-11T08:10:25.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.036+0000 7fdb335ff700 1 Processor -- start 2024-01-11T08:10:25.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.036+0000 7fdb335ff700 1 -- start start 2024-01-11T08:10:25.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.038+0000 7fdb335ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdb2c0b2b90 0x7fdb2c159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:25.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.038+0000 7fdb335ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb2c0cd730 con 0x7fdb2c0b07b0 2024-01-11T08:10:25.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.038+0000 7fdb335ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb2c0ccb20 con 0x7fdb2c0b03e0 2024-01-11T08:10:25.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.038+0000 7fdb335ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb2c0cce00 con 0x7fdb2c0b2b90 2024-01-11T08:10:25.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.038+0000 7fdb30b9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdb2c0b2b90 0x7fdb2c159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:25.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.038+0000 7fdb31b9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdb2c0b07b0 0x7fdb2c159740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37014/0 (socket says 172.21.15.27:37014) 2024-01-11T08:10:25.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.038+0000 7fdb30b9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdb2c0b2b90 0x7fdb2c159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40032/0 (socket says 172.21.15.27:40032) 2024-01-11T08:10:25.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.038+0000 7fdb31b9c700 1 -- 172.21.15.27:0/1450561964 learned_addr learned my addr 172.21.15.27:0/1450561964 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:25.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.038+0000 7fdb30b9a700 1 -- 172.21.15.27:0/1450561964 >> v1:172.21.15.77:6789/0 conn(0x7fdb2c0b03e0 legacy=0x7fdb2c0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:25.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.038+0000 7fdb30b9a700 1 -- 172.21.15.27:0/1450561964 >> v1:172.21.15.27:6789/0 conn(0x7fdb2c0b07b0 legacy=0x7fdb2c159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:25.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.038+0000 7fdb30b9a700 1 -- 172.21.15.27:0/1450561964 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2c0cd0e0 con 0x7fdb2c0b2b90 2024-01-11T08:10:25.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.038+0000 7fdb23fff700 1 -- 172.21.15.27:0/1450561964 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362187909 0 0) 0x7fdb2c0cd730 con 0x7fdb2c0b07b0 2024-01-11T08:10:25.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.039+0000 7fdb30b9a700 1 --2- 172.21.15.27:0/1450561964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb2c0b2b90 0x7fdb2c159200 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c0098e0 tx=0x7fdb1c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d43bd1f1f5bcf024 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:25.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.039+0000 7fdb23fff700 1 -- 172.21.15.27:0/1450561964 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c009e80 con 0x7fdb2c0b2b90 2024-01-11T08:10:25.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.039+0000 7fdb23fff700 1 -- 172.21.15.27:0/1450561964 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1c00b860 con 0x7fdb2c0b2b90 2024-01-11T08:10:25.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.039+0000 7fdb23fff700 1 -- 172.21.15.27:0/1450561964 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c00bbe0 con 0x7fdb2c0b2b90 2024-01-11T08:10:25.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.039+0000 7fdb335ff700 1 -- 172.21.15.27:0/1450561964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb2c0b2b90 msgr2=0x7fdb2c159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:25.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.039+0000 7fdb335ff700 1 --2- 172.21.15.27:0/1450561964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb2c0b2b90 0x7fdb2c159200 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c0098e0 tx=0x7fdb1c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.039+0000 7fdb335ff700 1 -- 172.21.15.27:0/1450561964 shutdown_connections 2024-01-11T08:10:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.039+0000 7fdb335ff700 1 --2- 172.21.15.27:0/1450561964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb2c0b2b90 0x7fdb2c159200 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.039+0000 7fdb335ff700 1 -- 172.21.15.27:0/1450561964 >> 172.21.15.27:0/1450561964 conn(0x7fdb2c06ed40 msgr2=0x7fdb2c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.040+0000 7fdb335ff700 1 -- 172.21.15.27:0/1450561964 shutdown_connections 2024-01-11T08:10:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.040+0000 7fdb335ff700 1 -- 172.21.15.27:0/1450561964 wait complete. 2024-01-11T08:10:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.042+0000 7fdb335ff700 1 Processor -- start 2024-01-11T08:10:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.042+0000 7fdb335ff700 1 -- start start 2024-01-11T08:10:25.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.043+0000 7fdb335ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb2c0b03e0 0x7fdb2c0ca480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:25.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.043+0000 7fdb335ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb2c0b07b0 0x7fdb2c169660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:25.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.043+0000 7fdb335ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb2c0b2b90 0x7fdb2c0b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:25.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.043+0000 7fdb335ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c16e510 con 0x7fdb2c0b07b0 2024-01-11T08:10:25.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.043+0000 7fdb335ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c0c9ff0 con 0x7fdb2c0b03e0 2024-01-11T08:10:25.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.043+0000 7fdb335ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c16e230 con 0x7fdb2c0b2b90 2024-01-11T08:10:25.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.043+0000 7fdb30b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb2c0b07b0 0x7fdb2c169660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:25.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.043+0000 7fdb30b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb2c0b07b0 0x7fdb2c169660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40038/0 (socket says 172.21.15.27:40038) 2024-01-11T08:10:25.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.043+0000 7fdb30b9a700 1 -- 172.21.15.27:0/2273856704 learned_addr learned my addr 172.21.15.27:0/2273856704 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:25.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.043+0000 7fdb3139b700 1 --2- 172.21.15.27:0/2273856704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb2c0b03e0 0x7fdb2c0ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:25.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.044+0000 7fdb31b9c700 1 --2- 172.21.15.27:0/2273856704 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb2c0b2b90 0x7fdb2c0b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:25.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.044+0000 7fdb30b9a700 1 -- 172.21.15.27:0/2273856704 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb2c0b2b90 msgr2=0x7fdb2c0b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:25.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.044+0000 7fdb30b9a700 1 --2- 172.21.15.27:0/2273856704 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb2c0b2b90 0x7fdb2c0b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.044+0000 7fdb30b9a700 1 -- 172.21.15.27:0/2273856704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb2c0b03e0 msgr2=0x7fdb2c0ca480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:25.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.044+0000 7fdb30b9a700 1 --2- 172.21.15.27:0/2273856704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb2c0b03e0 0x7fdb2c0ca480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.044+0000 7fdb30b9a700 1 -- 172.21.15.27:0/2273856704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1c009420 con 0x7fdb2c0b07b0 2024-01-11T08:10:25.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.044+0000 7fdb3139b700 1 --2- 172.21.15.27:0/2273856704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb2c0b03e0 0x7fdb2c0ca480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:25.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.044+0000 7fdb30b9a700 1 --2- 172.21.15.27:0/2273856704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb2c0b07b0 0x7fdb2c169660 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c069540 tx=0x7fdb1c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:25.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.044+0000 7fdb217fa700 1 -- 172.21.15.27:0/2273856704 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c01f070 con 0x7fdb2c0b07b0 2024-01-11T08:10:25.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.044+0000 7fdb217fa700 1 -- 172.21.15.27:0/2273856704 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1c000c90 con 0x7fdb2c0b07b0 2024-01-11T08:10:25.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.044+0000 7fdb217fa700 1 -- 172.21.15.27:0/2273856704 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c00bbe0 con 0x7fdb2c0b07b0 2024-01-11T08:10:25.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.044+0000 7fdb335ff700 1 -- 172.21.15.27:0/2273856704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb2c0c99a0 con 0x7fdb2c0b07b0 2024-01-11T08:10:25.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.045+0000 7fdb335ff700 1 -- 172.21.15.27:0/2273856704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb2c169ba0 con 0x7fdb2c0b07b0 2024-01-11T08:10:25.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.045+0000 7fdb335ff700 1 -- 172.21.15.27:0/2273856704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb2c0ca180 con 0x7fdb2c0b07b0 2024-01-11T08:10:25.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.046+0000 7fdb217fa700 1 -- 172.21.15.27:0/2273856704 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fdb1c011880 con 0x7fdb2c0b07b0 2024-01-11T08:10:25.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.047+0000 7fdb217fa700 1 --2- 172.21.15.27:0/2273856704 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdb10066f30 0x7fdb100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:25.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.047+0000 7fdb217fa700 1 -- 172.21.15.27:0/2273856704 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fdb1c0855d0 con 0x7fdb2c0b07b0 2024-01-11T08:10:25.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.048+0000 7fdb3139b700 1 --2- 172.21.15.27:0/2273856704 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdb10066f30 0x7fdb100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:25.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.048+0000 7fdb3139b700 1 --2- 172.21.15.27:0/2273856704 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdb10066f30 0x7fdb100693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdb28007fd0 tx=0x7fdb28009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:25.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.052+0000 7fdb217fa700 1 -- 172.21.15.27:0/2273856704 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdb1c052ba0 con 0x7fdb2c0b07b0 2024-01-11T08:10:25.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.254+0000 7fdb335ff700 1 -- 172.21.15.27:0/2273856704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb2c0718d0 con 0x7fdb2c0b07b0 2024-01-11T08:10:25.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.255+0000 7fdb217fa700 1 -- 172.21.15.27:0/2273856704 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12079 (secure 0 0 0) 0x7fdb1c015090 con 0x7fdb2c0b07b0 2024-01-11T08:10:25.254 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:25.254 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:22.473493+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:10:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.257+0000 7fdb335ff700 1 -- 172.21.15.27:0/2273856704 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdb10066f30 msgr2=0x7fdb100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.257+0000 7fdb335ff700 1 --2- 172.21.15.27:0/2273856704 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdb10066f30 0x7fdb100693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdb28007fd0 tx=0x7fdb28009040 comp rx=0 tx=0).stop 2024-01-11T08:10:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.257+0000 7fdb335ff700 1 -- 172.21.15.27:0/2273856704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb2c0b07b0 msgr2=0x7fdb2c169660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.257+0000 7fdb335ff700 1 --2- 172.21.15.27:0/2273856704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb2c0b07b0 0x7fdb2c169660 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c069540 tx=0x7fdb1c017000 comp rx=0 tx=0).stop 2024-01-11T08:10:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.257+0000 7fdb335ff700 1 -- 172.21.15.27:0/2273856704 shutdown_connections 2024-01-11T08:10:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.257+0000 7fdb335ff700 1 --2- 172.21.15.27:0/2273856704 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdb10066f30 0x7fdb100693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.257+0000 7fdb335ff700 1 --2- 172.21.15.27:0/2273856704 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb2c0b2b90 0x7fdb2c0b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.257+0000 7fdb335ff700 1 --2- 172.21.15.27:0/2273856704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb2c0b03e0 0x7fdb2c0ca480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.257+0000 7fdb335ff700 1 --2- 172.21.15.27:0/2273856704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb2c0b07b0 0x7fdb2c169660 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.257+0000 7fdb335ff700 1 -- 172.21.15.27:0/2273856704 >> 172.21.15.27:0/2273856704 conn(0x7fdb2c06ed40 msgr2=0x7fdb2c0ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.257+0000 7fdb335ff700 1 -- 172.21.15.27:0/2273856704 shutdown_connections 2024-01-11T08:10:25.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.257+0000 7fdb335ff700 1 -- 172.21.15.27:0/2273856704 wait complete. 2024-01-11T08:10:25.275 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-11T08:10:25.275 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:10:25.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.476+0000 7f1e3a1ff700 1 Processor -- start 2024-01-11T08:10:25.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.476+0000 7f1e3a1ff700 1 -- start start 2024-01-11T08:10:25.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.477+0000 7f1e3a1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e340cd220 0x7f1e3415d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:25.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.477+0000 7f1e3a1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1e341598f0 0x7f1e340cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:25.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.477+0000 7f1e3a1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e3415aef0 con 0x7f1e3415d860 2024-01-11T08:10:25.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.477+0000 7f1e3a1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e340cee00 con 0x7f1e340cd220 2024-01-11T08:10:25.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.477+0000 7f1e3a1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e3415dd70 con 0x7f1e341598f0 2024-01-11T08:10:25.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.478+0000 7f1e337fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e340cd220 0x7f1e3415d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:25.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.478+0000 7f1e337fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e340cd220 0x7f1e3415d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40044/0 (socket says 172.21.15.27:40044) 2024-01-11T08:10:25.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.478+0000 7f1e337fe700 1 -- 172.21.15.27:0/2392316452 learned_addr learned my addr 172.21.15.27:0/2392316452 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:25.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.478+0000 7f1e33fff700 1 --2- 172.21.15.27:0/2392316452 >> v2:172.21.15.77:3300/0 conn(0x7f1e341598f0 0x7f1e340cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:25.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.478+0000 7f1e337fe700 1 -- 172.21.15.27:0/2392316452 >> v2:172.21.15.77:3300/0 conn(0x7f1e341598f0 msgr2=0x7f1e340cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.478+0000 7f1e337fe700 1 --2- 172.21.15.27:0/2392316452 >> v2:172.21.15.77:3300/0 conn(0x7f1e341598f0 0x7f1e340cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.478+0000 7f1e337fe700 1 -- 172.21.15.27:0/2392316452 >> v1:172.21.15.77:6789/0 conn(0x7f1e3415d860 legacy=0x7f1e341591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.478+0000 7f1e337fe700 1 -- 172.21.15.27:0/2392316452 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e3415e050 con 0x7f1e340cd220 2024-01-11T08:10:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.478+0000 7f1e33fff700 1 --2- 172.21.15.27:0/2392316452 >> v2:172.21.15.77:3300/0 conn(0x7f1e341598f0 0x7f1e340cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.479+0000 7f1e337fe700 1 --2- 172.21.15.27:0/2392316452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e340cd220 0x7f1e3415d320 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1e3415ae80 tx=0x7f1e24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d925e3ed28ad161 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.479+0000 7f1e327fc700 1 -- 172.21.15.27:0/2392316452 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2400b860 con 0x7f1e340cd220 2024-01-11T08:10:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.479+0000 7f1e327fc700 1 -- 172.21.15.27:0/2392316452 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2400b9c0 con 0x7f1e340cd220 2024-01-11T08:10:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.479+0000 7f1e327fc700 1 -- 172.21.15.27:0/2392316452 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2400bd60 con 0x7f1e340cd220 2024-01-11T08:10:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.479+0000 7f1e3a1ff700 1 -- 172.21.15.27:0/2392316452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e340cd220 msgr2=0x7f1e3415d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.479+0000 7f1e3a1ff700 1 --2- 172.21.15.27:0/2392316452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e340cd220 0x7f1e3415d320 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1e3415ae80 tx=0x7f1e24009250 comp rx=0 tx=0).stop 2024-01-11T08:10:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.480+0000 7f1e3a1ff700 1 -- 172.21.15.27:0/2392316452 shutdown_connections 2024-01-11T08:10:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.480+0000 7f1e3a1ff700 1 --2- 172.21.15.27:0/2392316452 >> v2:172.21.15.77:3300/0 conn(0x7f1e341598f0 0x7f1e340cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.480+0000 7f1e3a1ff700 1 --2- 172.21.15.27:0/2392316452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e340cd220 0x7f1e3415d320 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.480+0000 7f1e3a1ff700 1 -- 172.21.15.27:0/2392316452 >> 172.21.15.27:0/2392316452 conn(0x7f1e3406ed40 msgr2=0x7f1e3406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:25.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.480+0000 7f1e3a1ff700 1 -- 172.21.15.27:0/2392316452 shutdown_connections 2024-01-11T08:10:25.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.480+0000 7f1e3a1ff700 1 -- 172.21.15.27:0/2392316452 wait complete. 2024-01-11T08:10:25.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.481+0000 7f1e3a1ff700 1 Processor -- start 2024-01-11T08:10:25.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.481+0000 7f1e3a1ff700 1 -- start start 2024-01-11T08:10:25.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.482+0000 7f1e3a1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e340cd220 0x7f1e34158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:25.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.482+0000 7f1e3a1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1e341598f0 0x7f1e340b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:25.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.482+0000 7f1e3a1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e3415d860 0x7f1e340c8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:25.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.482+0000 7f1e3a1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e340b2c10 con 0x7f1e340cd220 2024-01-11T08:10:25.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.482+0000 7f1e3a1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e340b2650 con 0x7f1e3415d860 2024-01-11T08:10:25.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.482+0000 7f1e3a1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e340b2930 con 0x7f1e341598f0 2024-01-11T08:10:25.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.482+0000 7f1e337fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e340cd220 0x7f1e34158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:25.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.483+0000 7f1e337fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e340cd220 0x7f1e34158220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40058/0 (socket says 172.21.15.27:40058) 2024-01-11T08:10:25.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.483+0000 7f1e337fe700 1 -- 172.21.15.27:0/4283645978 learned_addr learned my addr 172.21.15.27:0/4283645978 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:25.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.483+0000 7f1e33fff700 1 --2- 172.21.15.27:0/4283645978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e3415d860 0x7f1e340c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:25.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.483+0000 7f1e32ffd700 1 --2- 172.21.15.27:0/4283645978 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1e341598f0 0x7f1e340b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:25.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.483+0000 7f1e337fe700 1 -- 172.21.15.27:0/4283645978 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1e341598f0 msgr2=0x7f1e340b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:25.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.483+0000 7f1e337fe700 1 --2- 172.21.15.27:0/4283645978 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1e341598f0 0x7f1e340b1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.483+0000 7f1e337fe700 1 -- 172.21.15.27:0/4283645978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e3415d860 msgr2=0x7f1e340c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:25.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.483+0000 7f1e337fe700 1 --2- 172.21.15.27:0/4283645978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e3415d860 0x7f1e340c8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.483+0000 7f1e337fe700 1 -- 172.21.15.27:0/4283645978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e240094f0 con 0x7f1e340cd220 2024-01-11T08:10:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.483+0000 7f1e337fe700 1 --2- 172.21.15.27:0/4283645978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e340cd220 0x7f1e34158220 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1e24009290 tx=0x7f1e24017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.483+0000 7f1e1b7fe700 1 -- 172.21.15.27:0/4283645978 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e24026480 con 0x7f1e340cd220 2024-01-11T08:10:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.484+0000 7f1e1b7fe700 1 -- 172.21.15.27:0/4283645978 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2401f070 con 0x7f1e340cd220 2024-01-11T08:10:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.484+0000 7f1e1b7fe700 1 -- 172.21.15.27:0/4283645978 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e24026850 con 0x7f1e340cd220 2024-01-11T08:10:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.484+0000 7f1e3a1ff700 1 -- 172.21.15.27:0/4283645978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e340b4410 con 0x7f1e340cd220 2024-01-11T08:10:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.484+0000 7f1e3a1ff700 1 -- 172.21.15.27:0/4283645978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e34158760 con 0x7f1e340cd220 2024-01-11T08:10:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.485+0000 7f1e3a1ff700 1 -- 172.21.15.27:0/4283645978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e340b4270 con 0x7f1e340cd220 2024-01-11T08:10:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.486+0000 7f1e1b7fe700 1 -- 172.21.15.27:0/4283645978 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1e2401a430 con 0x7f1e340cd220 2024-01-11T08:10:25.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.486+0000 7f1e1b7fe700 1 --2- 172.21.15.27:0/4283645978 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1e20066f30 0x7f1e200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:25.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.487+0000 7f1e1b7fe700 1 -- 172.21.15.27:0/4283645978 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f1e24021070 con 0x7f1e340cd220 2024-01-11T08:10:25.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.487+0000 7f1e32ffd700 1 --2- 172.21.15.27:0/4283645978 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1e20066f30 0x7f1e200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:25.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.487+0000 7f1e32ffd700 1 --2- 172.21.15.27:0/4283645978 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1e20066f30 0x7f1e200693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1e340cecf0 tx=0x7f1e28007ec0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:25.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.491+0000 7f1e1b7fe700 1 -- 172.21.15.27:0/4283645978 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1e340b4270 con 0x7f1e340cd220 2024-01-11T08:10:25.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.692+0000 7f1e3a1ff700 1 -- 172.21.15.27:0/4283645978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1e340b4270 con 0x7f1e340cd220 2024-01-11T08:10:25.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.692+0000 7f1e1b7fe700 1 -- 172.21.15.27:0/4283645978 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1e2402b020 con 0x7f1e340cd220 2024-01-11T08:10:25.692 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:10:25.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.694+0000 7f1e3a1ff700 1 -- 172.21.15.27:0/4283645978 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1e20066f30 msgr2=0x7f1e200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:25.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.694+0000 7f1e3a1ff700 1 --2- 172.21.15.27:0/4283645978 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1e20066f30 0x7f1e200693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1e340cecf0 tx=0x7f1e28007ec0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.694+0000 7f1e3a1ff700 1 -- 172.21.15.27:0/4283645978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e340cd220 msgr2=0x7f1e34158220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:25.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.694+0000 7f1e3a1ff700 1 --2- 172.21.15.27:0/4283645978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e340cd220 0x7f1e34158220 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1e24009290 tx=0x7f1e24017000 comp rx=0 tx=0).stop 2024-01-11T08:10:25.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.694+0000 7f1e3a1ff700 1 -- 172.21.15.27:0/4283645978 shutdown_connections 2024-01-11T08:10:25.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.694+0000 7f1e3a1ff700 1 --2- 172.21.15.27:0/4283645978 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1e20066f30 0x7f1e200693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.694+0000 7f1e3a1ff700 1 --2- 172.21.15.27:0/4283645978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e3415d860 0x7f1e340c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.694+0000 7f1e3a1ff700 1 --2- 172.21.15.27:0/4283645978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e340cd220 0x7f1e34158220 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.694+0000 7f1e3a1ff700 1 --2- 172.21.15.27:0/4283645978 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1e341598f0 0x7f1e340b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.694+0000 7f1e3a1ff700 1 -- 172.21.15.27:0/4283645978 >> 172.21.15.27:0/4283645978 conn(0x7f1e3406ed40 msgr2=0x7f1e340cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:25.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.695+0000 7f1e3a1ff700 1 -- 172.21.15.27:0/4283645978 shutdown_connections 2024-01-11T08:10:25.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.695+0000 7f1e3a1ff700 1 -- 172.21.15.27:0/4283645978 wait complete. 2024-01-11T08:10:25.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:10:25.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.912+0000 7f4d06dff700 1 Processor -- start 2024-01-11T08:10:25.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.912+0000 7f4d06dff700 1 -- start start 2024-01-11T08:10:25.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.913+0000 7f4d06dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4d000b2b90 0x7f4d00159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:25.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.913+0000 7f4d06dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4d000b07b0 0x7f4d00159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.913+0000 7f4d06dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d000cd730 con 0x7f4d000b03e0 2024-01-11T08:10:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.913+0000 7f4d06dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d000ccb20 con 0x7f4d000b07b0 2024-01-11T08:10:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.913+0000 7f4d06dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d000cce00 con 0x7f4d000b2b90 2024-01-11T08:10:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.913+0000 7f4d0539c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4d000b07b0 0x7f4d00159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.913+0000 7f4d0539c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4d000b07b0 0x7f4d00159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40062/0 (socket says 172.21.15.27:40062) 2024-01-11T08:10:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.913+0000 7f4d0539c700 1 -- 172.21.15.27:0/3274428170 learned_addr learned my addr 172.21.15.27:0/3274428170 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.914+0000 7f4cfffff700 1 --2- 172.21.15.27:0/3274428170 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4d000b2b90 0x7f4d00159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.914+0000 7f4d0539c700 1 -- 172.21.15.27:0/3274428170 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4d000b2b90 msgr2=0x7f4d00159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.914+0000 7f4d0539c700 1 --2- 172.21.15.27:0/3274428170 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4d000b2b90 0x7f4d00159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.914+0000 7f4d0539c700 1 -- 172.21.15.27:0/3274428170 >> v1:172.21.15.77:6789/0 conn(0x7f4d000b03e0 legacy=0x7f4d000b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.914+0000 7f4d0539c700 1 -- 172.21.15.27:0/3274428170 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d000cd0e0 con 0x7f4d000b07b0 2024-01-11T08:10:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.914+0000 7f4d0539c700 1 --2- 172.21.15.27:0/3274428170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d000b07b0 0x7f4d00159740 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8009f20 tx=0x7f4cf8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bfef4e54f04fbae server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:25.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.914+0000 7f4cff7fe700 1 -- 172.21.15.27:0/3274428170 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf800e070 con 0x7f4d000b07b0 2024-01-11T08:10:25.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.914+0000 7f4cff7fe700 1 -- 172.21.15.27:0/3274428170 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf800ba80 con 0x7f4d000b07b0 2024-01-11T08:10:25.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.914+0000 7f4cff7fe700 1 -- 172.21.15.27:0/3274428170 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf80092e0 con 0x7f4d000b07b0 2024-01-11T08:10:25.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.915+0000 7f4d06dff700 1 -- 172.21.15.27:0/3274428170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d000b07b0 msgr2=0x7f4d00159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:25.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.915+0000 7f4d06dff700 1 --2- 172.21.15.27:0/3274428170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d000b07b0 0x7f4d00159740 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8009f20 tx=0x7f4cf8009840 comp rx=0 tx=0).stop 2024-01-11T08:10:25.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.915+0000 7f4d06dff700 1 -- 172.21.15.27:0/3274428170 shutdown_connections 2024-01-11T08:10:25.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.915+0000 7f4d06dff700 1 --2- 172.21.15.27:0/3274428170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d000b07b0 0x7f4d00159740 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.915+0000 7f4d06dff700 1 --2- 172.21.15.27:0/3274428170 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4d000b2b90 0x7f4d00159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.915+0000 7f4d06dff700 1 -- 172.21.15.27:0/3274428170 >> 172.21.15.27:0/3274428170 conn(0x7f4d0006ed40 msgr2=0x7f4d0006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:25.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.915+0000 7f4d06dff700 1 -- 172.21.15.27:0/3274428170 shutdown_connections 2024-01-11T08:10:25.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.915+0000 7f4d06dff700 1 -- 172.21.15.27:0/3274428170 wait complete. 2024-01-11T08:10:25.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.917+0000 7f4d06dff700 1 Processor -- start 2024-01-11T08:10:25.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.917+0000 7f4d06dff700 1 -- start start 2024-01-11T08:10:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.918+0000 7f4d06dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d000b03e0 0x7f4d00169850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.918+0000 7f4d06dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d000b07b0 0x7f4d000cf100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.918+0000 7f4d06dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4d000b2b90 0x7f4d000c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.918+0000 7f4d06dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d000ca620 con 0x7f4d000b07b0 2024-01-11T08:10:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.918+0000 7f4d06dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d000c6a80 con 0x7f4d000b03e0 2024-01-11T08:10:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.918+0000 7f4d06dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d000c3ea0 con 0x7f4d000b2b90 2024-01-11T08:10:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.918+0000 7f4cfffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d000b07b0 0x7f4d000cf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.918+0000 7f4cfffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d000b07b0 0x7f4d000cf100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40066/0 (socket says 172.21.15.27:40066) 2024-01-11T08:10:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.918+0000 7f4cfffff700 1 -- 172.21.15.27:0/1758506775 learned_addr learned my addr 172.21.15.27:0/1758506775 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.918+0000 7f4d04b9b700 1 --2- 172.21.15.27:0/1758506775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d000b03e0 0x7f4d00169850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.918+0000 7f4cfffff700 1 -- 172.21.15.27:0/1758506775 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4d000b2b90 msgr2=0x7f4d000c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.918+0000 7f4cfffff700 1 --2- 172.21.15.27:0/1758506775 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4d000b2b90 0x7f4d000c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.918+0000 7f4cfffff700 1 -- 172.21.15.27:0/1758506775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d000b03e0 msgr2=0x7f4d00169850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.918+0000 7f4cfffff700 1 --2- 172.21.15.27:0/1758506775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d000b03e0 0x7f4d00169850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.918+0000 7f4cfffff700 1 -- 172.21.15.27:0/1758506775 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf8009ae0 con 0x7f4d000b07b0 2024-01-11T08:10:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.919+0000 7f4cfffff700 1 --2- 172.21.15.27:0/1758506775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d000b07b0 0x7f4d000cf100 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4cec00d860 tx=0x7f4cec008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.919+0000 7f4cfcff9700 1 -- 172.21.15.27:0/1758506775 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec0125f0 con 0x7f4d000b07b0 2024-01-11T08:10:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.919+0000 7f4cfcff9700 1 -- 172.21.15.27:0/1758506775 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cec012750 con 0x7f4d000b07b0 2024-01-11T08:10:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.919+0000 7f4cfcff9700 1 -- 172.21.15.27:0/1758506775 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec012b10 con 0x7f4d000b07b0 2024-01-11T08:10:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.919+0000 7f4d06dff700 1 -- 172.21.15.27:0/1758506775 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d000c4180 con 0x7f4d000b07b0 2024-01-11T08:10:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.919+0000 7f4d06dff700 1 -- 172.21.15.27:0/1758506775 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d000c4740 con 0x7f4d000b07b0 2024-01-11T08:10:25.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.920+0000 7f4d06dff700 1 -- 172.21.15.27:0/1758506775 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d000718d0 con 0x7f4d000b07b0 2024-01-11T08:10:25.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.921+0000 7f4cfcff9700 1 -- 172.21.15.27:0/1758506775 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4cec01e460 con 0x7f4d000b07b0 2024-01-11T08:10:25.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.922+0000 7f4cfcff9700 1 --2- 172.21.15.27:0/1758506775 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4ce4066f30 0x7f4ce40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:25.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.922+0000 7f4cfcff9700 1 -- 172.21.15.27:0/1758506775 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f4cec0848a0 con 0x7f4d000b07b0 2024-01-11T08:10:25.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.922+0000 7f4d04b9b700 1 --2- 172.21.15.27:0/1758506775 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4ce4066f30 0x7f4ce40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:25.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.923+0000 7f4d04b9b700 1 --2- 172.21.15.27:0/1758506775 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4ce4066f30 0x7f4ce40693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4001760 tx=0x7f4cf4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:25.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:25.926+0000 7f4cfcff9700 1 -- 172.21.15.27:0/1758506775 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4cec0558f0 con 0x7f4d000b07b0 2024-01-11T08:10:26.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.178+0000 7f4d06dff700 1 -- 172.21.15.27:0/1758506775 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d000cd3c0 con 0x7f4d000b07b0 2024-01-11T08:10:26.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.179+0000 7f4cfcff9700 1 -- 172.21.15.27:0/1758506775 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3521 (secure 0 0 0) 0x7f4cec055480 con 0x7f4d000b07b0 2024-01-11T08:10:26.179 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:26.179 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3168764915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3168764915},{"type":"v1","addr":"172.21.15.27:6834","nonce":3168764915}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2607167395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2607167395},{"type":"v1","addr":"172.21.15.27:6835","nonce":2607167395}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2123423970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2123423970},{"type":"v1","addr":"172.21.15.77:6837","nonce":2123423970}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4291,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/269464429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":269464429},{"type":"v1","addr":"172.21.15.77:6836","nonce":269464429}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-01-11T08:10:26.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.181+0000 7f4d06dff700 1 -- 172.21.15.27:0/1758506775 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4ce4066f30 msgr2=0x7f4ce40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:26.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.181+0000 7f4d06dff700 1 --2- 172.21.15.27:0/1758506775 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4ce4066f30 0x7f4ce40693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4001760 tx=0x7f4cf4009040 comp rx=0 tx=0).stop 2024-01-11T08:10:26.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.181+0000 7f4d06dff700 1 -- 172.21.15.27:0/1758506775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d000b07b0 msgr2=0x7f4d000cf100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:26.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.181+0000 7f4d06dff700 1 --2- 172.21.15.27:0/1758506775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d000b07b0 0x7f4d000cf100 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4cec00d860 tx=0x7f4cec008040 comp rx=0 tx=0).stop 2024-01-11T08:10:26.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.181+0000 7f4d06dff700 1 -- 172.21.15.27:0/1758506775 shutdown_connections 2024-01-11T08:10:26.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.181+0000 7f4d06dff700 1 --2- 172.21.15.27:0/1758506775 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4ce4066f30 0x7f4ce40693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.181+0000 7f4d06dff700 1 --2- 172.21.15.27:0/1758506775 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4d000b2b90 0x7f4d000c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.181+0000 7f4d06dff700 1 --2- 172.21.15.27:0/1758506775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d000b03e0 0x7f4d00169850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.181+0000 7f4d06dff700 1 --2- 172.21.15.27:0/1758506775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d000b07b0 0x7f4d000cf100 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.181+0000 7f4d06dff700 1 -- 172.21.15.27:0/1758506775 >> 172.21.15.27:0/1758506775 conn(0x7f4d0006ed40 msgr2=0x7f4d000cb400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:26.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.181+0000 7f4d06dff700 1 -- 172.21.15.27:0/1758506775 shutdown_connections 2024-01-11T08:10:26.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.181+0000 7f4d06dff700 1 -- 172.21.15.27:0/1758506775 wait complete. 2024-01-11T08:10:26.183 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 3 2024-01-11T08:10:26.199 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:10:26.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.398+0000 7f538fdff700 1 Processor -- start 2024-01-11T08:10:26.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.398+0000 7f538fdff700 1 -- start start 2024-01-11T08:10:26.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.399+0000 7f538fdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53880ca3e0 0x7f53880cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:26.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.399+0000 7f538fdff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f53880ca7b0 0x7f5388159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:26.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.399+0000 7f538fdff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f53880c8a20 0x7f53880c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:26.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.399+0000 7f538fdff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f538815ae70 con 0x7f53880c8a20 2024-01-11T08:10:26.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.399+0000 7f538fdff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f538815a3c0 con 0x7f53880ca7b0 2024-01-11T08:10:26.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.399+0000 7f538fdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53880cc4f0 con 0x7f53880ca3e0 2024-01-11T08:10:26.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.400+0000 7f538e39c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f53880c8a20 0x7f53880c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:26.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.400+0000 7f538e39c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f53880c8a20 0x7f53880c8e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40078/0 (socket says 172.21.15.27:40078) 2024-01-11T08:10:26.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.400+0000 7f538e39c700 1 -- 172.21.15.27:0/631577845 learned_addr learned my addr 172.21.15.27:0/631577845 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:26.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.400+0000 7f538db9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53880ca3e0 0x7f53880cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:26.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.400+0000 7f538d39a700 1 --2- 172.21.15.27:0/631577845 >> v2:172.21.15.77:3300/0 conn(0x7f53880ca7b0 0x7f5388159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:26.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.400+0000 7f538e39c700 1 -- 172.21.15.27:0/631577845 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53880ca3e0 msgr2=0x7f53880cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:26.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.400+0000 7f538e39c700 1 --2- 172.21.15.27:0/631577845 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53880ca3e0 0x7f53880cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.400+0000 7f538e39c700 1 -- 172.21.15.27:0/631577845 >> v2:172.21.15.77:3300/0 conn(0x7f53880ca7b0 msgr2=0x7f5388159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.400+0000 7f538e39c700 1 --2- 172.21.15.27:0/631577845 >> v2:172.21.15.77:3300/0 conn(0x7f53880ca7b0 0x7f5388159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.400+0000 7f538e39c700 1 -- 172.21.15.27:0/631577845 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538815d640 con 0x7f53880c8a20 2024-01-11T08:10:26.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.400+0000 7f538e39c700 1 --2- 172.21.15.27:0/631577845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53880c8a20 0x7f53880c8e60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f538806f320 tx=0x7f5380009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e810243558c72e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:26.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.401+0000 7f538cb99700 1 -- 172.21.15.27:0/631577845 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538000e070 con 0x7f53880c8a20 2024-01-11T08:10:26.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.401+0000 7f538cb99700 1 -- 172.21.15.27:0/631577845 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538000b970 con 0x7f53880c8a20 2024-01-11T08:10:26.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.401+0000 7f538cb99700 1 -- 172.21.15.27:0/631577845 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53800092e0 con 0x7f53880c8a20 2024-01-11T08:10:26.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.401+0000 7f538fdff700 1 -- 172.21.15.27:0/631577845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53880c8a20 msgr2=0x7f53880c8e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:26.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.401+0000 7f538fdff700 1 --2- 172.21.15.27:0/631577845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53880c8a20 0x7f53880c8e60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f538806f320 tx=0x7f5380009840 comp rx=0 tx=0).stop 2024-01-11T08:10:26.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.401+0000 7f538fdff700 1 -- 172.21.15.27:0/631577845 shutdown_connections 2024-01-11T08:10:26.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.401+0000 7f538fdff700 1 --2- 172.21.15.27:0/631577845 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53880ca3e0 0x7f53880cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.401+0000 7f538fdff700 1 --2- 172.21.15.27:0/631577845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53880c8a20 0x7f53880c8e60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.401+0000 7f538fdff700 1 --2- 172.21.15.27:0/631577845 >> v2:172.21.15.77:3300/0 conn(0x7f53880ca7b0 0x7f5388159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.401+0000 7f538fdff700 1 -- 172.21.15.27:0/631577845 >> 172.21.15.27:0/631577845 conn(0x7f53880ce3e0 msgr2=0x7f53880ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:26.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.401+0000 7f538fdff700 1 -- 172.21.15.27:0/631577845 shutdown_connections 2024-01-11T08:10:26.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.401+0000 7f538fdff700 1 -- 172.21.15.27:0/631577845 wait complete. 2024-01-11T08:10:26.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.403+0000 7f538fdff700 1 Processor -- start 2024-01-11T08:10:26.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.403+0000 7f538fdff700 1 -- start start 2024-01-11T08:10:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.404+0000 7f538fdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53880c8a20 0x7f5388147eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.404+0000 7f538fdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53880ca3e0 0x7f538815e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.404+0000 7f538fdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53880ca7b0 0x7f538815eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:26.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.404+0000 7f538fdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53881517c0 con 0x7f53880c8a20 2024-01-11T08:10:26.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.404+0000 7f538fdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53880c3970 con 0x7f53880ca7b0 2024-01-11T08:10:26.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.404+0000 7f538fdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538816a970 con 0x7f53880ca3e0 2024-01-11T08:10:26.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.404+0000 7f538db9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53880c8a20 0x7f5388147eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:26.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.404+0000 7f538db9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53880c8a20 0x7f5388147eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40086/0 (socket says 172.21.15.27:40086) 2024-01-11T08:10:26.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.404+0000 7f538db9b700 1 -- 172.21.15.27:0/3331303576 learned_addr learned my addr 172.21.15.27:0/3331303576 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:26.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.404+0000 7f538e39c700 1 --2- 172.21.15.27:0/3331303576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53880ca7b0 0x7f538815eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:26.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.404+0000 7f538d39a700 1 --2- 172.21.15.27:0/3331303576 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53880ca3e0 0x7f538815e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:26.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.404+0000 7f538db9b700 1 -- 172.21.15.27:0/3331303576 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53880ca3e0 msgr2=0x7f538815e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:26.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.404+0000 7f538db9b700 1 --2- 172.21.15.27:0/3331303576 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53880ca3e0 0x7f538815e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.404+0000 7f538db9b700 1 -- 172.21.15.27:0/3331303576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53880ca7b0 msgr2=0x7f538815eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:26.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.404+0000 7f538db9b700 1 --2- 172.21.15.27:0/3331303576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53880ca7b0 0x7f538815eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.405+0000 7f538db9b700 1 -- 172.21.15.27:0/3331303576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5380009ae0 con 0x7f53880c8a20 2024-01-11T08:10:26.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.405+0000 7f538db9b700 1 --2- 172.21.15.27:0/3331303576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53880c8a20 0x7f5388147eb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f538400ec80 tx=0x7f53840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.405+0000 7f537dffb700 1 -- 172.21.15.27:0/3331303576 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538400c9d0 con 0x7f53880c8a20 2024-01-11T08:10:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.405+0000 7f537dffb700 1 -- 172.21.15.27:0/3331303576 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538400cb50 con 0x7f53880c8a20 2024-01-11T08:10:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.405+0000 7f538fdff700 1 -- 172.21.15.27:0/3331303576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53881514e0 con 0x7f53880c8a20 2024-01-11T08:10:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.405+0000 7f537dffb700 1 -- 172.21.15.27:0/3331303576 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5384013470 con 0x7f53880c8a20 2024-01-11T08:10:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.405+0000 7f538fdff700 1 -- 172.21.15.27:0/3331303576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53880c3600 con 0x7f53880c8a20 2024-01-11T08:10:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.406+0000 7f538fdff700 1 -- 172.21.15.27:0/3331303576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53880c3180 con 0x7f53880c8a20 2024-01-11T08:10:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.407+0000 7f537dffb700 1 -- 172.21.15.27:0/3331303576 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5384013820 con 0x7f53880c8a20 2024-01-11T08:10:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.408+0000 7f537dffb700 1 --2- 172.21.15.27:0/3331303576 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f536c066f30 0x7f536c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.408+0000 7f537dffb700 1 -- 172.21.15.27:0/3331303576 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f5384084560 con 0x7f53880c8a20 2024-01-11T08:10:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.408+0000 7f538d39a700 1 --2- 172.21.15.27:0/3331303576 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f536c066f30 0x7f536c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.409+0000 7f538d39a700 1 --2- 172.21.15.27:0/3331303576 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f536c066f30 0x7f536c0693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f538814bf10 tx=0x7f5378008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:26.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.412+0000 7f537dffb700 1 -- 172.21.15.27:0/3331303576 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f53880c3180 con 0x7f53880c8a20 2024-01-11T08:10:26.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.616+0000 7f538fdff700 1 -- 172.21.15.27:0/3331303576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53880c3180 con 0x7f53880c8a20 2024-01-11T08:10:26.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.616+0000 7f537dffb700 1 -- 172.21.15.27:0/3331303576 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12079 (secure 0 0 0) 0x7f53880c3180 con 0x7f53880c8a20 2024-01-11T08:10:26.617 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:26.617 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:22.473493+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:10:26.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.619+0000 7f538fdff700 1 -- 172.21.15.27:0/3331303576 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f536c066f30 msgr2=0x7f536c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:26.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.619+0000 7f538fdff700 1 --2- 172.21.15.27:0/3331303576 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f536c066f30 0x7f536c0693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f538814bf10 tx=0x7f5378008040 comp rx=0 tx=0).stop 2024-01-11T08:10:26.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.619+0000 7f538fdff700 1 -- 172.21.15.27:0/3331303576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53880c8a20 msgr2=0x7f5388147eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:26.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.619+0000 7f538fdff700 1 --2- 172.21.15.27:0/3331303576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53880c8a20 0x7f5388147eb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f538400ec80 tx=0x7f53840091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.619+0000 7f538fdff700 1 -- 172.21.15.27:0/3331303576 shutdown_connections 2024-01-11T08:10:26.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.619+0000 7f538fdff700 1 --2- 172.21.15.27:0/3331303576 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f536c066f30 0x7f536c0693e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.619+0000 7f538fdff700 1 --2- 172.21.15.27:0/3331303576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53880ca7b0 0x7f538815eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.619+0000 7f538fdff700 1 --2- 172.21.15.27:0/3331303576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53880c8a20 0x7f5388147eb0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.619+0000 7f538fdff700 1 --2- 172.21.15.27:0/3331303576 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53880ca3e0 0x7f538815e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.619+0000 7f538fdff700 1 -- 172.21.15.27:0/3331303576 >> 172.21.15.27:0/3331303576 conn(0x7f53880ce3e0 msgr2=0x7f53880693f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:26.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.619+0000 7f538fdff700 1 -- 172.21.15.27:0/3331303576 shutdown_connections 2024-01-11T08:10:26.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.619+0000 7f538fdff700 1 -- 172.21.15.27:0/3331303576 wait complete. 2024-01-11T08:10:26.637 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-11T08:10:26.637 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:10:26.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.839+0000 7f3f05bff700 1 Processor -- start 2024-01-11T08:10:26.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.839+0000 7f3f05bff700 1 -- start start 2024-01-11T08:10:26.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.840+0000 7f3f05bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3f000687c0 0x7f3f000b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:26.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.840+0000 7f3f05bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f0015dff0 0x7f3f001591d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:26.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.840+0000 7f3f05bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f000cbfe0 con 0x7f3f000b2a00 2024-01-11T08:10:26.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.840+0000 7f3f05bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f000cb3d0 con 0x7f3f0015dff0 2024-01-11T08:10:26.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.840+0000 7f3f05bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f000cb6b0 con 0x7f3f000687c0 2024-01-11T08:10:26.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.840+0000 7f3efeffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f0015dff0 0x7f3f001591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:26.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.840+0000 7f3efeffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f0015dff0 0x7f3f001591d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40092/0 (socket says 172.21.15.27:40092) 2024-01-11T08:10:26.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.840+0000 7f3efeffd700 1 -- 172.21.15.27:0/4109809695 learned_addr learned my addr 172.21.15.27:0/4109809695 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:26.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.841+0000 7f3eff7fe700 1 --2- 172.21.15.27:0/4109809695 >> v2:172.21.15.77:3300/0 conn(0x7f3f000687c0 0x7f3f000b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:26.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.841+0000 7f3efeffd700 1 -- 172.21.15.27:0/4109809695 >> v2:172.21.15.77:3300/0 conn(0x7f3f000687c0 msgr2=0x7f3f000b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:26.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.841+0000 7f3efeffd700 1 --2- 172.21.15.27:0/4109809695 >> v2:172.21.15.77:3300/0 conn(0x7f3f000687c0 0x7f3f000b24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.841+0000 7f3efeffd700 1 -- 172.21.15.27:0/4109809695 >> v1:172.21.15.77:6789/0 conn(0x7f3f000b2a00 legacy=0x7f3f00159710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:26.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.841+0000 7f3efeffd700 1 -- 172.21.15.27:0/4109809695 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f000cb990 con 0x7f3f0015dff0 2024-01-11T08:10:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.841+0000 7f3efe7fc700 1 -- 172.21.15.27:0/4109809695 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810412414 0 0) 0x7f3f000cbfe0 con 0x7f3f000b2a00 2024-01-11T08:10:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.841+0000 7f3efeffd700 1 --2- 172.21.15.27:0/4109809695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f0015dff0 0x7f3f001591d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3f00089430 tx=0x7f3ef4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f8a7f579d3349db server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.841+0000 7f3efe7fc700 1 -- 172.21.15.27:0/4109809695 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4009e70 con 0x7f3f0015dff0 2024-01-11T08:10:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.842+0000 7f3efe7fc700 1 -- 172.21.15.27:0/4109809695 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef400b860 con 0x7f3f0015dff0 2024-01-11T08:10:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.842+0000 7f3efe7fc700 1 -- 172.21.15.27:0/4109809695 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef400bc00 con 0x7f3f0015dff0 2024-01-11T08:10:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.842+0000 7f3f05bff700 1 -- 172.21.15.27:0/4109809695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f0015dff0 msgr2=0x7f3f001591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.842+0000 7f3f05bff700 1 --2- 172.21.15.27:0/4109809695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f0015dff0 0x7f3f001591d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3f00089430 tx=0x7f3ef4009250 comp rx=0 tx=0).stop 2024-01-11T08:10:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.842+0000 7f3f05bff700 1 -- 172.21.15.27:0/4109809695 shutdown_connections 2024-01-11T08:10:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.842+0000 7f3f05bff700 1 --2- 172.21.15.27:0/4109809695 >> v2:172.21.15.77:3300/0 conn(0x7f3f000687c0 0x7f3f000b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.842+0000 7f3f05bff700 1 --2- 172.21.15.27:0/4109809695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f0015dff0 0x7f3f001591d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.842+0000 7f3f05bff700 1 -- 172.21.15.27:0/4109809695 >> 172.21.15.27:0/4109809695 conn(0x7f3f0006ed40 msgr2=0x7f3f0006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.842+0000 7f3f05bff700 1 -- 172.21.15.27:0/4109809695 shutdown_connections 2024-01-11T08:10:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.842+0000 7f3f05bff700 1 -- 172.21.15.27:0/4109809695 wait complete. 2024-01-11T08:10:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.844+0000 7f3f05bff700 1 Processor -- start 2024-01-11T08:10:26.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.844+0000 7f3f05bff700 1 -- start start 2024-01-11T08:10:26.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.845+0000 7f3f05bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f000687c0 0x7f3f00158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:26.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.845+0000 7f3f05bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f000b2a00 0x7f3f00169b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:26.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.845+0000 7f3f05bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0015dff0 0x7f3f0016a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:26.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.845+0000 7f3f05bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f00159750 con 0x7f3f000b2a00 2024-01-11T08:10:26.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.845+0000 7f3f05bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f00159a30 con 0x7f3f0015dff0 2024-01-11T08:10:26.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.845+0000 7f3f05bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f00159d10 con 0x7f3f000687c0 2024-01-11T08:10:26.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.845+0000 7f3efeffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f000b2a00 0x7f3f00169b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:26.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.845+0000 7f3efeffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f000b2a00 0x7f3f00169b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40098/0 (socket says 172.21.15.27:40098) 2024-01-11T08:10:26.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.845+0000 7f3efeffd700 1 -- 172.21.15.27:0/1564065921 learned_addr learned my addr 172.21.15.27:0/1564065921 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:26.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.845+0000 7f3eff7fe700 1 --2- 172.21.15.27:0/1564065921 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f000687c0 0x7f3f00158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:26.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.845+0000 7f3efffff700 1 --2- 172.21.15.27:0/1564065921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0015dff0 0x7f3f0016a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:26.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.845+0000 7f3efeffd700 1 -- 172.21.15.27:0/1564065921 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f000687c0 msgr2=0x7f3f00158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:26.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.845+0000 7f3efeffd700 1 --2- 172.21.15.27:0/1564065921 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f000687c0 0x7f3f00158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.846+0000 7f3efeffd700 1 -- 172.21.15.27:0/1564065921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0015dff0 msgr2=0x7f3f0016a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:26.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.846+0000 7f3efeffd700 1 --2- 172.21.15.27:0/1564065921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0015dff0 0x7f3f0016a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:26.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.846+0000 7f3efeffd700 1 -- 172.21.15.27:0/1564065921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef40094f0 con 0x7f3f000b2a00 2024-01-11T08:10:26.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.846+0000 7f3efffff700 1 --2- 172.21.15.27:0/1564065921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0015dff0 0x7f3f0016a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:26.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.846+0000 7f3efeffd700 1 --2- 172.21.15.27:0/1564065921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f000b2a00 0x7f3f00169b40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3ef40094c0 tx=0x7f3ef4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:26.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.846+0000 7f3ee77fe700 1 -- 172.21.15.27:0/1564065921 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef400b9d0 con 0x7f3f000b2a00 2024-01-11T08:10:26.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.846+0000 7f3ee77fe700 1 -- 172.21.15.27:0/1564065921 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef400bb50 con 0x7f3f000b2a00 2024-01-11T08:10:26.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.846+0000 7f3ee77fe700 1 -- 172.21.15.27:0/1564065921 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4025670 con 0x7f3f000b2a00 2024-01-11T08:10:26.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.846+0000 7f3f05bff700 1 -- 172.21.15.27:0/1564065921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f000c9090 con 0x7f3f000b2a00 2024-01-11T08:10:26.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.846+0000 7f3f05bff700 1 -- 172.21.15.27:0/1564065921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f000c36a0 con 0x7f3f000b2a00 2024-01-11T08:10:26.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.847+0000 7f3f05bff700 1 -- 172.21.15.27:0/1564065921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f000c8ef0 con 0x7f3f000b2a00 2024-01-11T08:10:26.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.848+0000 7f3ee77fe700 1 -- 172.21.15.27:0/1564065921 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3ef4023070 con 0x7f3f000b2a00 2024-01-11T08:10:26.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.849+0000 7f3ee77fe700 1 --2- 172.21.15.27:0/1564065921 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3eec066f30 0x7f3eec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:26.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.849+0000 7f3ee77fe700 1 -- 172.21.15.27:0/1564065921 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f3ef4085780 con 0x7f3f000b2a00 2024-01-11T08:10:26.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.850+0000 7f3eff7fe700 1 --2- 172.21.15.27:0/1564065921 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3eec066f30 0x7f3eec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:26.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.850+0000 7f3eff7fe700 1 --2- 172.21.15.27:0/1564065921 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3eec066f30 0x7f3eec0693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3f000cd740 tx=0x7f3ef00091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:26.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:26.854+0000 7f3ee77fe700 1 -- 172.21.15.27:0/1564065921 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3f000c8ef0 con 0x7f3f000b2a00 2024-01-11T08:10:27.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.057+0000 7f3f05bff700 1 -- 172.21.15.27:0/1564065921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3f000c95b0 con 0x7f3f000b2a00 2024-01-11T08:10:27.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.058+0000 7f3ee77fe700 1 -- 172.21.15.27:0/1564065921 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3f000c95b0 con 0x7f3f000b2a00 2024-01-11T08:10:27.058 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:10:27.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.060+0000 7f3f05bff700 1 -- 172.21.15.27:0/1564065921 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3eec066f30 msgr2=0x7f3eec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:27.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.060+0000 7f3f05bff700 1 --2- 172.21.15.27:0/1564065921 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3eec066f30 0x7f3eec0693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3f000cd740 tx=0x7f3ef00091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.060+0000 7f3f05bff700 1 -- 172.21.15.27:0/1564065921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f000b2a00 msgr2=0x7f3f00169b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:27.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.060+0000 7f3f05bff700 1 --2- 172.21.15.27:0/1564065921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f000b2a00 0x7f3f00169b40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3ef40094c0 tx=0x7f3ef4017000 comp rx=0 tx=0).stop 2024-01-11T08:10:27.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.060+0000 7f3f05bff700 1 -- 172.21.15.27:0/1564065921 shutdown_connections 2024-01-11T08:10:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.060+0000 7f3f05bff700 1 --2- 172.21.15.27:0/1564065921 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3eec066f30 0x7f3eec0693e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.060+0000 7f3f05bff700 1 --2- 172.21.15.27:0/1564065921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f0015dff0 0x7f3f0016a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.060+0000 7f3f05bff700 1 --2- 172.21.15.27:0/1564065921 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f000687c0 0x7f3f00158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.060+0000 7f3f05bff700 1 --2- 172.21.15.27:0/1564065921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f000b2a00 0x7f3f00169b40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.060+0000 7f3f05bff700 1 -- 172.21.15.27:0/1564065921 >> 172.21.15.27:0/1564065921 conn(0x7f3f0006ed40 msgr2=0x7f3f0015c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.060+0000 7f3f05bff700 1 -- 172.21.15.27:0/1564065921 shutdown_connections 2024-01-11T08:10:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.060+0000 7f3f05bff700 1 -- 172.21.15.27:0/1564065921 wait complete. 2024-01-11T08:10:27.078 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-11T08:10:27.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.277+0000 7f2d873ff700 1 Processor -- start 2024-01-11T08:10:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.277+0000 7f2d873ff700 1 -- start start 2024-01-11T08:10:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.278+0000 7f2d873ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2d800b03e0 0x7f2d800b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.278+0000 7f2d873ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d800b29c0 0x7f2d80159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.278+0000 7f2d873ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d800cd730 con 0x7f2d800b07b0 2024-01-11T08:10:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.278+0000 7f2d873ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d800b2de0 con 0x7f2d800b03e0 2024-01-11T08:10:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.278+0000 7f2d873ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d800ccb20 con 0x7f2d800b29c0 2024-01-11T08:10:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.279+0000 7f2d8519b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2d800b03e0 0x7f2d800b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.279+0000 7f2d8599c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d800b29c0 0x7f2d80159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.279+0000 7f2d8599c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d800b29c0 0x7f2d80159910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:47664/0 (socket says 172.21.15.27:47664) 2024-01-11T08:10:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.279+0000 7f2d8599c700 1 -- 172.21.15.27:0/609701301 learned_addr learned my addr 172.21.15.27:0/609701301 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.279+0000 7f2d8519b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2d800b03e0 0x7f2d800b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:45238/0 (socket says 172.21.15.27:45238) 2024-01-11T08:10:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.279+0000 7f2d8499a700 1 --1- 172.21.15.27:0/609701301 >> v1:172.21.15.77:6789/0 conn(0x7f2d800b07b0 0x7f2d80159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:47894/0 (socket says 172.21.15.27:47894) 2024-01-11T08:10:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.279+0000 7f2d8599c700 1 -- 172.21.15.27:0/609701301 >> v2:172.21.15.77:3300/0 conn(0x7f2d800b03e0 msgr2=0x7f2d800b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.279+0000 7f2d8599c700 1 --2- 172.21.15.27:0/609701301 >> v2:172.21.15.77:3300/0 conn(0x7f2d800b03e0 0x7f2d800b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.279+0000 7f2d8599c700 1 -- 172.21.15.27:0/609701301 >> v1:172.21.15.77:6789/0 conn(0x7f2d800b07b0 legacy=0x7f2d80159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.279+0000 7f2d8599c700 1 -- 172.21.15.27:0/609701301 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d800cce00 con 0x7f2d800b29c0 2024-01-11T08:10:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.279+0000 7f2d8599c700 1 --2- 172.21.15.27:0/609701301 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d800b29c0 0x7f2d80159910 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009f20 tx=0x7f2d78009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dddd04c63b5868f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.280+0000 7f2d77fff700 1 -- 172.21.15.27:0/609701301 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7800e070 con 0x7f2d800b29c0 2024-01-11T08:10:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.280+0000 7f2d77fff700 1 -- 172.21.15.27:0/609701301 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7800b9d0 con 0x7f2d800b29c0 2024-01-11T08:10:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.280+0000 7f2d77fff700 1 -- 172.21.15.27:0/609701301 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d780092e0 con 0x7f2d800b29c0 2024-01-11T08:10:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.280+0000 7f2d873ff700 1 -- 172.21.15.27:0/609701301 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d800b29c0 msgr2=0x7f2d80159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.280+0000 7f2d873ff700 1 --2- 172.21.15.27:0/609701301 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d800b29c0 0x7f2d80159910 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009f20 tx=0x7f2d78009840 comp rx=0 tx=0).stop 2024-01-11T08:10:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.280+0000 7f2d873ff700 1 -- 172.21.15.27:0/609701301 shutdown_connections 2024-01-11T08:10:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.280+0000 7f2d873ff700 1 --2- 172.21.15.27:0/609701301 >> v2:172.21.15.77:3300/0 conn(0x7f2d800b03e0 0x7f2d800b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.280+0000 7f2d873ff700 1 --2- 172.21.15.27:0/609701301 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d800b29c0 0x7f2d80159910 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.280+0000 7f2d873ff700 1 -- 172.21.15.27:0/609701301 >> 172.21.15.27:0/609701301 conn(0x7f2d8006ed40 msgr2=0x7f2d8006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.280+0000 7f2d873ff700 1 -- 172.21.15.27:0/609701301 shutdown_connections 2024-01-11T08:10:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.280+0000 7f2d873ff700 1 -- 172.21.15.27:0/609701301 wait complete. 2024-01-11T08:10:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.282+0000 7f2d873ff700 1 Processor -- start 2024-01-11T08:10:27.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.282+0000 7f2d873ff700 1 -- start start 2024-01-11T08:10:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.283+0000 7f2d873ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d800b03e0 0x7f2d80169760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.283+0000 7f2d873ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d800b07b0 0x7f2d800cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.283+0000 7f2d873ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d800b29c0 0x7f2d800c9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.283+0000 7f2d873ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d800a6f40 con 0x7f2d800b03e0 2024-01-11T08:10:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.283+0000 7f2d873ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d800a7150 con 0x7f2d800b29c0 2024-01-11T08:10:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.283+0000 7f2d873ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d800a7430 con 0x7f2d800b07b0 2024-01-11T08:10:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.283+0000 7f2d8519b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d800b03e0 0x7f2d80169760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.283+0000 7f2d8519b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d800b03e0 0x7f2d80169760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40106/0 (socket says 172.21.15.27:40106) 2024-01-11T08:10:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.283+0000 7f2d8499a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d800b07b0 0x7f2d800cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.283+0000 7f2d8519b700 1 -- 172.21.15.27:0/3116436872 learned_addr learned my addr 172.21.15.27:0/3116436872 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.283+0000 7f2d8599c700 1 --2- 172.21.15.27:0/3116436872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d800b29c0 0x7f2d800c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.284+0000 7f2d8519b700 1 -- 172.21.15.27:0/3116436872 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d800b07b0 msgr2=0x7f2d800cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.284+0000 7f2d8519b700 1 --2- 172.21.15.27:0/3116436872 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d800b07b0 0x7f2d800cfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.284+0000 7f2d8519b700 1 -- 172.21.15.27:0/3116436872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d800b29c0 msgr2=0x7f2d800c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.284+0000 7f2d8519b700 1 --2- 172.21.15.27:0/3116436872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d800b29c0 0x7f2d800c9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.284+0000 7f2d8519b700 1 -- 172.21.15.27:0/3116436872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d78009ae0 con 0x7f2d800b03e0 2024-01-11T08:10:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.284+0000 7f2d8499a700 1 --2- 172.21.15.27:0/3116436872 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d800b07b0 0x7f2d800cfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.284+0000 7f2d8519b700 1 --2- 172.21.15.27:0/3116436872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d800b03e0 0x7f2d80169760 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c00ec80 tx=0x7f2d7c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:27.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.284+0000 7f2d757fa700 1 -- 172.21.15.27:0/3116436872 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c00c850 con 0x7f2d800b03e0 2024-01-11T08:10:27.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.284+0000 7f2d757fa700 1 -- 172.21.15.27:0/3116436872 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7c00c9b0 con 0x7f2d800b03e0 2024-01-11T08:10:27.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.284+0000 7f2d873ff700 1 -- 172.21.15.27:0/3116436872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d800a7710 con 0x7f2d800b03e0 2024-01-11T08:10:27.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.284+0000 7f2d757fa700 1 -- 172.21.15.27:0/3116436872 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c00cd50 con 0x7f2d800b03e0 2024-01-11T08:10:27.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.284+0000 7f2d873ff700 1 -- 172.21.15.27:0/3116436872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d80169ca0 con 0x7f2d800b03e0 2024-01-11T08:10:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.285+0000 7f2d873ff700 1 -- 172.21.15.27:0/3116436872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d800a7bb0 con 0x7f2d800b03e0 2024-01-11T08:10:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.286+0000 7f2d757fa700 1 -- 172.21.15.27:0/3116436872 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2d7c013680 con 0x7f2d800b03e0 2024-01-11T08:10:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.287+0000 7f2d757fa700 1 --2- 172.21.15.27:0/3116436872 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d64066f30 0x7f2d640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.287+0000 7f2d757fa700 1 -- 172.21.15.27:0/3116436872 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f2d7c084410 con 0x7f2d800b03e0 2024-01-11T08:10:27.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.288+0000 7f2d8499a700 1 --2- 172.21.15.27:0/3116436872 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d64066f30 0x7f2d640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:27.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.288+0000 7f2d8499a700 1 --2- 172.21.15.27:0/3116436872 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d64066f30 0x7f2d640693e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2d70001710 tx=0x7f2d70007eb0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:27.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.291+0000 7f2d757fa700 1 -- 172.21.15.27:0/3116436872 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2d7c0519d0 con 0x7f2d800b03e0 2024-01-11T08:10:27.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.497+0000 7f2d873ff700 1 -- 172.21.15.27:0/3116436872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2d800718d0 con 0x7f2d800b03e0 2024-01-11T08:10:27.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.497+0000 7f2d757fa700 1 -- 172.21.15.27:0/3116436872 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2d7c054ff0 con 0x7f2d800b03e0 2024-01-11T08:10:27.497 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:27.497 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T08:10:27.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:10:27.498 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-11T08:10:27.498 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-11T08:10:27.498 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:10:27.498 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T08:10:27.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.499+0000 7f2d873ff700 1 -- 172.21.15.27:0/3116436872 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d64066f30 msgr2=0x7f2d640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:27.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.499+0000 7f2d873ff700 1 --2- 172.21.15.27:0/3116436872 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d64066f30 0x7f2d640693e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2d70001710 tx=0x7f2d70007eb0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.499+0000 7f2d873ff700 1 -- 172.21.15.27:0/3116436872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d800b03e0 msgr2=0x7f2d80169760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:27.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.499+0000 7f2d873ff700 1 --2- 172.21.15.27:0/3116436872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d800b03e0 0x7f2d80169760 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c00ec80 tx=0x7f2d7c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.500+0000 7f2d873ff700 1 -- 172.21.15.27:0/3116436872 shutdown_connections 2024-01-11T08:10:27.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.500+0000 7f2d873ff700 1 --2- 172.21.15.27:0/3116436872 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d64066f30 0x7f2d640693e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.500+0000 7f2d873ff700 1 --2- 172.21.15.27:0/3116436872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d800b29c0 0x7f2d800c9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.500+0000 7f2d873ff700 1 --2- 172.21.15.27:0/3116436872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d800b03e0 0x7f2d80169760 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.500+0000 7f2d873ff700 1 --2- 172.21.15.27:0/3116436872 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d800b07b0 0x7f2d800cfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.500+0000 7f2d873ff700 1 -- 172.21.15.27:0/3116436872 >> 172.21.15.27:0/3116436872 conn(0x7f2d8006ed40 msgr2=0x7f2d80159ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:27.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.500+0000 7f2d873ff700 1 -- 172.21.15.27:0/3116436872 shutdown_connections 2024-01-11T08:10:27.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.500+0000 7f2d873ff700 1 -- 172.21.15.27:0/3116436872 wait complete. 2024-01-11T08:10:27.518 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-11T08:10:27.518 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-11T08:10:27.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.722+0000 7f00dd5ff700 1 Processor -- start 2024-01-11T08:10:27.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.723+0000 7f00dd5ff700 1 -- start start 2024-01-11T08:10:27.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.724+0000 7f00dd5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f00d80685e0 0x7f00d8068a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:27.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.724+0000 7f00dd5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d815c4b0 con 0x7f00d8159e60 2024-01-11T08:10:27.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.724+0000 7f00dd5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d815d050 con 0x7f00d815a7a0 2024-01-11T08:10:27.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.724+0000 7f00dd5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d8068f40 con 0x7f00d80685e0 2024-01-11T08:10:27.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.724+0000 7f00d67fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f00d8159e60 0x7f00d815aec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37016/0 (socket says 172.21.15.27:37016) 2024-01-11T08:10:27.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.724+0000 7f00d67fc700 1 -- 172.21.15.27:0/3961716937 learned_addr learned my addr 172.21.15.27:0/3961716937 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:27.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.724+0000 7f00d77fe700 1 --2- 172.21.15.27:0/3961716937 >> v2:172.21.15.77:3300/0 conn(0x7f00d80685e0 0x7f00d8068a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:27.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.725+0000 7f00d5ffb700 1 -- 172.21.15.27:0/3961716937 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929103677 0 0) 0x7f00d815c4b0 con 0x7f00d8159e60 2024-01-11T08:10:27.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.725+0000 7f00d5ffb700 1 -- 172.21.15.27:0/3961716937 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00c0003570 con 0x7f00d8159e60 2024-01-11T08:10:27.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.725+0000 7f00d5ffb700 1 -- 172.21.15.27:0/3961716937 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1507230602 0 0) 0x7f00c0003570 con 0x7f00d8159e60 2024-01-11T08:10:27.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.725+0000 7f00d5ffb700 1 -- 172.21.15.27:0/3961716937 >> v2:172.21.15.77:3300/0 conn(0x7f00d80685e0 msgr2=0x7f00d8068a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:27.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.725+0000 7f00d5ffb700 1 --2- 172.21.15.27:0/3961716937 >> v2:172.21.15.77:3300/0 conn(0x7f00d80685e0 0x7f00d8068a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.725+0000 7f00d5ffb700 1 -- 172.21.15.27:0/3961716937 >> v1:172.21.15.77:6789/0 conn(0x7f00d815a7a0 legacy=0x7f00d815c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.725+0000 7f00d5ffb700 1 -- 172.21.15.27:0/3961716937 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d815b5d0 con 0x7f00d8159e60 2024-01-11T08:10:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.725+0000 7f00d5ffb700 1 -- 172.21.15.27:0/3961716937 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f00cc001470 con 0x7f00d8159e60 2024-01-11T08:10:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.725+0000 7f00d77fe700 1 --2- 172.21.15.27:0/3961716937 >> v2:172.21.15.77:3300/0 conn(0x7f00d80685e0 0x7f00d8068a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.725+0000 7f00d5ffb700 1 -- 172.21.15.27:0/3961716937 >> v1:172.21.15.27:6789/0 conn(0x7f00d8159e60 legacy=0x7f00d815aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.725+0000 7f00d5ffb700 1 --2- 172.21.15.27:0/3961716937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00c0003ab0 0x7f00c0003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.725+0000 7f00d5ffb700 1 --2- 172.21.15.27:0/3961716937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00c0004810 0x7f00c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.725+0000 7f00d5ffb700 1 --2- 172.21.15.27:0/3961716937 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00c0005210 0x7f00c000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.725+0000 7f00d5ffb700 1 -- 172.21.15.27:0/3961716937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d8068f40 con 0x7f00c0004810 2024-01-11T08:10:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.725+0000 7f00d5ffb700 1 -- 172.21.15.27:0/3961716937 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c000c130 con 0x7f00c0003ab0 2024-01-11T08:10:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.725+0000 7f00d5ffb700 1 -- 172.21.15.27:0/3961716937 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c000c2f0 con 0x7f00c0005210 2024-01-11T08:10:27.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.725+0000 7f00d67fc700 1 --2- 172.21.15.27:0/3961716937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00c0004810 0x7f00c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.726+0000 7f00d6ffd700 1 --2- 172.21.15.27:0/3961716937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00c0003ab0 0x7f00c0003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.726+0000 7f00d77fe700 1 --2- 172.21.15.27:0/3961716937 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00c0005210 0x7f00c000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.726+0000 7f00d67fc700 1 -- 172.21.15.27:0/3961716937 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00c0005210 msgr2=0x7f00c000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.726+0000 7f00d67fc700 1 --2- 172.21.15.27:0/3961716937 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00c0005210 0x7f00c000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.726+0000 7f00d67fc700 1 -- 172.21.15.27:0/3961716937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00c0003ab0 msgr2=0x7f00c0003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.726+0000 7f00d67fc700 1 --2- 172.21.15.27:0/3961716937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00c0003ab0 0x7f00c0003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.726+0000 7f00d67fc700 1 -- 172.21.15.27:0/3961716937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c0003570 con 0x7f00c0004810 2024-01-11T08:10:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.726+0000 7f00d77fe700 1 -- 172.21.15.27:0/3961716937 reap_dead start 2024-01-11T08:10:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.726+0000 7f00d67fc700 1 --2- 172.21.15.27:0/3961716937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00c0004810 0x7f00c0004c80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f00cc00c2d0 tx=0x7f00cc001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65fcc0ecd1ca7ab8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.726+0000 7f00d5ffb700 1 -- 172.21.15.27:0/3961716937 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00cc00cba0 con 0x7f00c0004810 2024-01-11T08:10:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.726+0000 7f00d5ffb700 1 -- 172.21.15.27:0/3961716937 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00cc00cd20 con 0x7f00c0004810 2024-01-11T08:10:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.726+0000 7f00d5ffb700 1 -- 172.21.15.27:0/3961716937 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00cc005b70 con 0x7f00c0004810 2024-01-11T08:10:27.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.727+0000 7f00dd5ff700 1 -- 172.21.15.27:0/3961716937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00c0004810 msgr2=0x7f00c0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:27.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.727+0000 7f00dd5ff700 1 --2- 172.21.15.27:0/3961716937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00c0004810 0x7f00c0004c80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f00cc00c2d0 tx=0x7f00cc001d80 comp rx=0 tx=0).stop 2024-01-11T08:10:27.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.727+0000 7f00dd5ff700 1 -- 172.21.15.27:0/3961716937 shutdown_connections 2024-01-11T08:10:27.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.727+0000 7f00dd5ff700 1 --2- 172.21.15.27:0/3961716937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00c0004810 0x7f00c0004c80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.727+0000 7f00dd5ff700 1 -- 172.21.15.27:0/3961716937 >> 172.21.15.27:0/3961716937 conn(0x7f00d806ed40 msgr2=0x7f00d806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:27.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.727+0000 7f00dd5ff700 1 -- 172.21.15.27:0/3961716937 shutdown_connections 2024-01-11T08:10:27.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.727+0000 7f00dd5ff700 1 -- 172.21.15.27:0/3961716937 wait complete. 2024-01-11T08:10:27.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.729+0000 7f00dd5ff700 1 Processor -- start 2024-01-11T08:10:27.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.730+0000 7f00dd5ff700 1 -- start start 2024-01-11T08:10:27.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.731+0000 7f00dd5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00c0004810 0x7f00d80b5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:27.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.731+0000 7f00dd5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00d80c9c30 0x7f00d814fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:27.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.731+0000 7f00dd5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00d814df20 0x7f00d814e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:27.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.731+0000 7f00dd5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d80caf90 con 0x7f00c0004810 2024-01-11T08:10:27.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.731+0000 7f00dd5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d814e900 con 0x7f00d814df20 2024-01-11T08:10:27.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.731+0000 7f00dd5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d80cb420 con 0x7f00d80c9c30 2024-01-11T08:10:27.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.731+0000 7f00d6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00c0004810 0x7f00d80b5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:27.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.731+0000 7f00d6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00c0004810 0x7f00d80b5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40124/0 (socket says 172.21.15.27:40124) 2024-01-11T08:10:27.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.731+0000 7f00d6ffd700 1 -- 172.21.15.27:0/249525817 learned_addr learned my addr 172.21.15.27:0/249525817 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:27.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.731+0000 7f00d67fc700 1 --2- 172.21.15.27:0/249525817 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00d80c9c30 0x7f00d814fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:27.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.731+0000 7f00d77fe700 1 --2- 172.21.15.27:0/249525817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00d814df20 0x7f00d814e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:27.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.731+0000 7f00d6ffd700 1 -- 172.21.15.27:0/249525817 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00d80c9c30 msgr2=0x7f00d814fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:27.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.731+0000 7f00d6ffd700 1 --2- 172.21.15.27:0/249525817 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00d80c9c30 0x7f00d814fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.731+0000 7f00d6ffd700 1 -- 172.21.15.27:0/249525817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00d814df20 msgr2=0x7f00d814e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:27.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.731+0000 7f00d6ffd700 1 --2- 172.21.15.27:0/249525817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00d814df20 0x7f00d814e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:27.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.731+0000 7f00d6ffd700 1 -- 172.21.15.27:0/249525817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d80ca380 con 0x7f00c0004810 2024-01-11T08:10:27.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.732+0000 7f00d6ffd700 1 --2- 172.21.15.27:0/249525817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00c0004810 0x7f00d80b5360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f00c80028a0 tx=0x7f00c8007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:27.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.732+0000 7f00beffd700 1 -- 172.21.15.27:0/249525817 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c8003430 con 0x7f00c0004810 2024-01-11T08:10:27.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.732+0000 7f00beffd700 1 -- 172.21.15.27:0/249525817 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00c800f6a0 con 0x7f00c0004810 2024-01-11T08:10:27.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.732+0000 7f00dd5ff700 1 -- 172.21.15.27:0/249525817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d80ca660 con 0x7f00c0004810 2024-01-11T08:10:27.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.732+0000 7f00beffd700 1 -- 172.21.15.27:0/249525817 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c800fa40 con 0x7f00c0004810 2024-01-11T08:10:27.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.732+0000 7f00dd5ff700 1 -- 172.21.15.27:0/249525817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d80cac20 con 0x7f00c0004810 2024-01-11T08:10:27.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.733+0000 7f00dd5ff700 1 -- 172.21.15.27:0/249525817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d80cb270 con 0x7f00c0004810 2024-01-11T08:10:27.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.734+0000 7f00beffd700 1 -- 172.21.15.27:0/249525817 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f00c801c400 con 0x7f00c0004810 2024-01-11T08:10:27.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.735+0000 7f00beffd700 1 --2- 172.21.15.27:0/249525817 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f00c4066f30 0x7f00c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:27.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.735+0000 7f00beffd700 1 -- 172.21.15.27:0/249525817 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f00c8084670 con 0x7f00c0004810 2024-01-11T08:10:27.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.736+0000 7f00d67fc700 1 --2- 172.21.15.27:0/249525817 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f00c4066f30 0x7f00c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:27.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.736+0000 7f00d67fc700 1 --2- 172.21.15.27:0/249525817 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f00c4066f30 0x7f00c40693e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f00d8159190 tx=0x7f00cc015000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:27.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.739+0000 7f00beffd700 1 -- 172.21.15.27:0/249525817 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f00c8051b90 con 0x7f00c0004810 2024-01-11T08:10:27.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:27.941+0000 7f00dd5ff700 1 -- 172.21.15.27:0/249525817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f00d804c3c0 con 0x7f00c0004810 2024-01-11T08:10:28.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.645+0000 7f00beffd700 1 -- 172.21.15.27:0/249525817 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7f00c80551b0 con 0x7f00c0004810 2024-01-11T08:10:28.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.650+0000 7f00dd5ff700 1 -- 172.21.15.27:0/249525817 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f00c4066f30 msgr2=0x7f00c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:28.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.650+0000 7f00dd5ff700 1 --2- 172.21.15.27:0/249525817 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f00c4066f30 0x7f00c40693e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f00d8159190 tx=0x7f00cc015000 comp rx=0 tx=0).stop 2024-01-11T08:10:28.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.650+0000 7f00dd5ff700 1 -- 172.21.15.27:0/249525817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00c0004810 msgr2=0x7f00d80b5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:28.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.650+0000 7f00dd5ff700 1 --2- 172.21.15.27:0/249525817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00c0004810 0x7f00d80b5360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f00c80028a0 tx=0x7f00c8007d30 comp rx=0 tx=0).stop 2024-01-11T08:10:28.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.650+0000 7f00dd5ff700 1 -- 172.21.15.27:0/249525817 shutdown_connections 2024-01-11T08:10:28.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.650+0000 7f00dd5ff700 1 --2- 172.21.15.27:0/249525817 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f00c4066f30 0x7f00c40693e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:28.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.650+0000 7f00dd5ff700 1 --2- 172.21.15.27:0/249525817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00d814df20 0x7f00d814e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:28.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.650+0000 7f00dd5ff700 1 --2- 172.21.15.27:0/249525817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00c0004810 0x7f00d80b5360 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:28.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.650+0000 7f00dd5ff700 1 --2- 172.21.15.27:0/249525817 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00d80c9c30 0x7f00d814fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:28.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.650+0000 7f00dd5ff700 1 -- 172.21.15.27:0/249525817 >> 172.21.15.27:0/249525817 conn(0x7f00d806ed40 msgr2=0x7f00d814f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:28.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.651+0000 7f00dd5ff700 1 -- 172.21.15.27:0/249525817 shutdown_connections 2024-01-11T08:10:28.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.652+0000 7f00dd5ff700 1 -- 172.21.15.27:0/249525817 wait complete. 2024-01-11T08:10:28.655 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-11T08:10:28.675 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-11T08:10:28.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.884+0000 7f892cbff700 1 Processor -- start 2024-01-11T08:10:28.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.884+0000 7f892cbff700 1 -- start start 2024-01-11T08:10:28.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.885+0000 7f892cbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f892806bba0 0x7f89280cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:28.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.885+0000 7f892cbff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f892806bf70 0x7f8928159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:28.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.885+0000 7f892cbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f892815db00 0x7f892815df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:28.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.885+0000 7f892cbff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8928069fc0 con 0x7f892806bf70 2024-01-11T08:10:28.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.885+0000 7f892cbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89280cce10 con 0x7f892806bba0 2024-01-11T08:10:28.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.885+0000 7f892cbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892815aea0 con 0x7f892815db00 2024-01-11T08:10:28.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.885+0000 7f8925d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f892806bf70 0x7f8928159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:28.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.885+0000 7f8925d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f892806bf70 0x7f8928159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40134/0 (socket says 172.21.15.27:40134) 2024-01-11T08:10:28.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.885+0000 7f8925d9b700 1 -- 172.21.15.27:0/1219609379 learned_addr learned my addr 172.21.15.27:0/1219609379 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:28.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.886+0000 7f8926d9d700 1 --2- 172.21.15.27:0/1219609379 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f892815db00 0x7f892815df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:28.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.886+0000 7f892659c700 1 --2- 172.21.15.27:0/1219609379 >> v2:172.21.15.77:3300/0 conn(0x7f892806bba0 0x7f89280cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:28.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.886+0000 7f8925d9b700 1 -- 172.21.15.27:0/1219609379 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f892815db00 msgr2=0x7f892815df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:28.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.886+0000 7f8925d9b700 1 --2- 172.21.15.27:0/1219609379 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f892815db00 0x7f892815df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:28.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.886+0000 7f8925d9b700 1 -- 172.21.15.27:0/1219609379 >> v2:172.21.15.77:3300/0 conn(0x7f892806bba0 msgr2=0x7f89280cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:28.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.886+0000 7f8925d9b700 1 --2- 172.21.15.27:0/1219609379 >> v2:172.21.15.77:3300/0 conn(0x7f892806bba0 0x7f89280cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:28.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.886+0000 7f8925d9b700 1 -- 172.21.15.27:0/1219609379 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892815d670 con 0x7f892806bf70 2024-01-11T08:10:28.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.886+0000 7f8926d9d700 1 --2- 172.21.15.27:0/1219609379 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f892815db00 0x7f892815df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:28.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.886+0000 7f892659c700 1 --2- 172.21.15.27:0/1219609379 >> v2:172.21.15.77:3300/0 conn(0x7f892806bba0 0x7f89280cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:28.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.886+0000 7f8925d9b700 1 --2- 172.21.15.27:0/1219609379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892806bf70 0x7f8928159e40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f89280adc70 tx=0x7f891c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4893d6f1a1bb075a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:28.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.886+0000 7f892559a700 1 -- 172.21.15.27:0/1219609379 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c00b820 con 0x7f892806bf70 2024-01-11T08:10:28.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.886+0000 7f892559a700 1 -- 172.21.15.27:0/1219609379 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c00b980 con 0x7f892806bf70 2024-01-11T08:10:28.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.886+0000 7f892559a700 1 -- 172.21.15.27:0/1219609379 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c00bd20 con 0x7f892806bf70 2024-01-11T08:10:28.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.887+0000 7f892cbff700 1 -- 172.21.15.27:0/1219609379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892806bf70 msgr2=0x7f8928159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:28.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.887+0000 7f892cbff700 1 --2- 172.21.15.27:0/1219609379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892806bf70 0x7f8928159e40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f89280adc70 tx=0x7f891c009250 comp rx=0 tx=0).stop 2024-01-11T08:10:28.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.887+0000 7f892cbff700 1 -- 172.21.15.27:0/1219609379 shutdown_connections 2024-01-11T08:10:28.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.887+0000 7f892cbff700 1 --2- 172.21.15.27:0/1219609379 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f892815db00 0x7f892815df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:28.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.887+0000 7f892cbff700 1 --2- 172.21.15.27:0/1219609379 >> v2:172.21.15.77:3300/0 conn(0x7f892806bba0 0x7f89280cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:28.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.887+0000 7f892cbff700 1 --2- 172.21.15.27:0/1219609379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892806bf70 0x7f8928159e40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:28.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.887+0000 7f892cbff700 1 -- 172.21.15.27:0/1219609379 >> 172.21.15.27:0/1219609379 conn(0x7f89280ce3e0 msgr2=0x7f89280ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:28.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.887+0000 7f892cbff700 1 -- 172.21.15.27:0/1219609379 shutdown_connections 2024-01-11T08:10:28.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.887+0000 7f892cbff700 1 -- 172.21.15.27:0/1219609379 wait complete. 2024-01-11T08:10:28.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.889+0000 7f892cbff700 1 Processor -- start 2024-01-11T08:10:28.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.889+0000 7f892cbff700 1 -- start start 2024-01-11T08:10:28.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.890+0000 7f892cbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f892806bba0 0x7f89280c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:28.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.890+0000 7f892cbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892806bf70 0x7f892814e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:28.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.890+0000 7f892cbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f892815db00 0x7f89280c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:28.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.890+0000 7f892cbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89280cb1a0 con 0x7f892806bf70 2024-01-11T08:10:28.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.890+0000 7f892cbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89280c5650 con 0x7f892806bba0 2024-01-11T08:10:28.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.890+0000 7f892cbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892814e660 con 0x7f892815db00 2024-01-11T08:10:28.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.890+0000 7f8925d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892806bf70 0x7f892814e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:28.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.890+0000 7f8925d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892806bf70 0x7f892814e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40142/0 (socket says 172.21.15.27:40142) 2024-01-11T08:10:28.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.890+0000 7f8925d9b700 1 -- 172.21.15.27:0/939093659 learned_addr learned my addr 172.21.15.27:0/939093659 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:28.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.890+0000 7f8926d9d700 1 --2- 172.21.15.27:0/939093659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f892815db00 0x7f89280c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:28.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.890+0000 7f892659c700 1 --2- 172.21.15.27:0/939093659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f892806bba0 0x7f89280c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:28.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.890+0000 7f8925d9b700 1 -- 172.21.15.27:0/939093659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f892815db00 msgr2=0x7f89280c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:28.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.890+0000 7f8925d9b700 1 --2- 172.21.15.27:0/939093659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f892815db00 0x7f89280c9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:28.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.890+0000 7f8925d9b700 1 -- 172.21.15.27:0/939093659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f892806bba0 msgr2=0x7f89280c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:28.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.890+0000 7f8925d9b700 1 --2- 172.21.15.27:0/939093659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f892806bba0 0x7f89280c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:28.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.890+0000 7f8925d9b700 1 -- 172.21.15.27:0/939093659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891c0094f0 con 0x7f892806bf70 2024-01-11T08:10:28.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.890+0000 7f892659c700 1 --2- 172.21.15.27:0/939093659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f892806bba0 0x7f89280c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.891+0000 7f8925d9b700 1 --2- 172.21.15.27:0/939093659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892806bf70 0x7f892814e120 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f891c0090b0 tx=0x7f891c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.891+0000 7f89167fc700 1 -- 172.21.15.27:0/939093659 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c011e50 con 0x7f892806bf70 2024-01-11T08:10:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.891+0000 7f89167fc700 1 -- 172.21.15.27:0/939093659 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c01f070 con 0x7f892806bf70 2024-01-11T08:10:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.891+0000 7f89167fc700 1 -- 172.21.15.27:0/939093659 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c0266d0 con 0x7f892806bf70 2024-01-11T08:10:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.891+0000 7f892cbff700 1 -- 172.21.15.27:0/939093659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89280ca900 con 0x7f892806bf70 2024-01-11T08:10:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.891+0000 7f892cbff700 1 -- 172.21.15.27:0/939093659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89280caec0 con 0x7f892806bf70 2024-01-11T08:10:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.892+0000 7f892cbff700 1 -- 172.21.15.27:0/939093659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89280c57e0 con 0x7f892806bf70 2024-01-11T08:10:28.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.893+0000 7f89167fc700 1 -- 172.21.15.27:0/939093659 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f891c026a80 con 0x7f892806bf70 2024-01-11T08:10:28.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.894+0000 7f89167fc700 1 --2- 172.21.15.27:0/939093659 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8908066f30 0x7f89080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:28.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.894+0000 7f89167fc700 1 -- 172.21.15.27:0/939093659 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5223+0+0 (secure 0 0 0) 0x7f891c021070 con 0x7f892806bf70 2024-01-11T08:10:28.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.895+0000 7f892659c700 1 --2- 172.21.15.27:0/939093659 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8908066f30 0x7f89080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:28.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.895+0000 7f892659c700 1 --2- 172.21.15.27:0/939093659 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8908066f30 0x7f89080693e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8918006f00 tx=0x7f8918008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:28.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:28.898+0000 7f89167fc700 1 -- 172.21.15.27:0/939093659 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f891c053800 con 0x7f892806bf70 2024-01-11T08:10:29.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.102+0000 7f892cbff700 1 -- 172.21.15.27:0/939093659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f892804c3c0 con 0x7f892806bf70 2024-01-11T08:10:29.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.647+0000 7f89167fc700 1 -- 172.21.15.27:0/939093659 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7f891c056e20 con 0x7f892806bf70 2024-01-11T08:10:29.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.652+0000 7f892cbff700 1 -- 172.21.15.27:0/939093659 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8908066f30 msgr2=0x7f89080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:29.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.652+0000 7f892cbff700 1 --2- 172.21.15.27:0/939093659 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8908066f30 0x7f89080693e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8918006f00 tx=0x7f8918008040 comp rx=0 tx=0).stop 2024-01-11T08:10:29.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.652+0000 7f892cbff700 1 -- 172.21.15.27:0/939093659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892806bf70 msgr2=0x7f892814e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:29.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.652+0000 7f892cbff700 1 --2- 172.21.15.27:0/939093659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892806bf70 0x7f892814e120 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f891c0090b0 tx=0x7f891c017000 comp rx=0 tx=0).stop 2024-01-11T08:10:29.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.652+0000 7f892cbff700 1 -- 172.21.15.27:0/939093659 shutdown_connections 2024-01-11T08:10:29.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.652+0000 7f892cbff700 1 --2- 172.21.15.27:0/939093659 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8908066f30 0x7f89080693e0 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8918006f00 tx=0x7f8918008040 comp rx=0 tx=0).stop 2024-01-11T08:10:29.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.652+0000 7f892cbff700 1 --2- 172.21.15.27:0/939093659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f892815db00 0x7f89280c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:29.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.652+0000 7f892cbff700 1 --2- 172.21.15.27:0/939093659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f892806bba0 0x7f89280c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:29.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.652+0000 7f892cbff700 1 --2- 172.21.15.27:0/939093659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892806bf70 0x7f892814e120 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:29.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.653+0000 7f892cbff700 1 -- 172.21.15.27:0/939093659 >> 172.21.15.27:0/939093659 conn(0x7f89280ce3e0 msgr2=0x7f892814e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:29.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.653+0000 7f892cbff700 1 -- 172.21.15.27:0/939093659 shutdown_connections 2024-01-11T08:10:29.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.654+0000 7f892cbff700 1 -- 172.21.15.27:0/939093659 wait complete. 2024-01-11T08:10:29.657 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-11T08:10:29.688 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-11T08:10:29.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.892+0000 7fea781ff700 1 Processor -- start 2024-01-11T08:10:29.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.892+0000 7fea781ff700 1 -- start start 2024-01-11T08:10:29.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.893+0000 7fea781ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea70142dc0 0x7fea700cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:29.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.893+0000 7fea781ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea700cb2e0 0x7fea700cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:29.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.893+0000 7fea781ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea700ccc70 con 0x7fea70143190 2024-01-11T08:10:29.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.893+0000 7fea781ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea7015df30 con 0x7fea700cb2e0 2024-01-11T08:10:29.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.893+0000 7fea781ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea7015d320 con 0x7fea70142dc0 2024-01-11T08:10:29.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.894+0000 7fea7679c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea700cb2e0 0x7fea700cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:29.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.894+0000 7fea7679c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea700cb2e0 0x7fea700cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40154/0 (socket says 172.21.15.27:40154) 2024-01-11T08:10:29.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.894+0000 7fea75f9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea70142dc0 0x7fea700cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:29.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.894+0000 7fea7679c700 1 -- 172.21.15.27:0/2295569482 learned_addr learned my addr 172.21.15.27:0/2295569482 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:29.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.894+0000 7fea7679c700 1 -- 172.21.15.27:0/2295569482 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea70142dc0 msgr2=0x7fea700cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:29.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.894+0000 7fea7679c700 1 --2- 172.21.15.27:0/2295569482 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea70142dc0 0x7fea700cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:29.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.894+0000 7fea7679c700 1 -- 172.21.15.27:0/2295569482 >> v1:172.21.15.27:6789/0 conn(0x7fea70143190 legacy=0x7fea700cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:29.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.894+0000 7fea7679c700 1 -- 172.21.15.27:0/2295569482 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea700cb700 con 0x7fea700cb2e0 2024-01-11T08:10:29.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.894+0000 7fea75f9b700 1 --2- 172.21.15.27:0/2295569482 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea70142dc0 0x7fea700cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:29.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.894+0000 7fea74f99700 1 -- 172.21.15.27:0/2295569482 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557966914 0 0) 0x7fea700ccc70 con 0x7fea70143190 2024-01-11T08:10:29.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.894+0000 7fea7679c700 1 --2- 172.21.15.27:0/2295569482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea700cb2e0 0x7fea700cc730 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fea68009f20 tx=0x7fea68009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a51d3417ac2f8ad server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:29.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.895+0000 7fea74f99700 1 -- 172.21.15.27:0/2295569482 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6800e070 con 0x7fea700cb2e0 2024-01-11T08:10:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.895+0000 7fea74f99700 1 -- 172.21.15.27:0/2295569482 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6800b9d0 con 0x7fea700cb2e0 2024-01-11T08:10:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.895+0000 7fea74f99700 1 -- 172.21.15.27:0/2295569482 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea680092e0 con 0x7fea700cb2e0 2024-01-11T08:10:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.895+0000 7fea781ff700 1 -- 172.21.15.27:0/2295569482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea700cb2e0 msgr2=0x7fea700cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.895+0000 7fea781ff700 1 --2- 172.21.15.27:0/2295569482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea700cb2e0 0x7fea700cc730 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fea68009f20 tx=0x7fea68009840 comp rx=0 tx=0).stop 2024-01-11T08:10:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.895+0000 7fea781ff700 1 -- 172.21.15.27:0/2295569482 shutdown_connections 2024-01-11T08:10:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.895+0000 7fea781ff700 1 --2- 172.21.15.27:0/2295569482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea700cb2e0 0x7fea700cc730 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.895+0000 7fea781ff700 1 --2- 172.21.15.27:0/2295569482 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea70142dc0 0x7fea700cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.895+0000 7fea781ff700 1 -- 172.21.15.27:0/2295569482 >> 172.21.15.27:0/2295569482 conn(0x7fea70063ca0 msgr2=0x7fea7006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.895+0000 7fea781ff700 1 -- 172.21.15.27:0/2295569482 shutdown_connections 2024-01-11T08:10:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.895+0000 7fea781ff700 1 -- 172.21.15.27:0/2295569482 wait complete. 2024-01-11T08:10:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.897+0000 7fea781ff700 1 Processor -- start 2024-01-11T08:10:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.897+0000 7fea781ff700 1 -- start start 2024-01-11T08:10:29.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.898+0000 7fea781ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea700cb2e0 0x7fea70141630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:29.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.898+0000 7fea781ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea70142dc0 0x7fea700c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.898+0000 7fea781ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea70143190 0x7fea700c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.898+0000 7fea781ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea700b2840 con 0x7fea70142dc0 2024-01-11T08:10:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.898+0000 7fea781ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea700c6150 con 0x7fea700cb2e0 2024-01-11T08:10:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.898+0000 7fea781ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea700c7500 con 0x7fea70143190 2024-01-11T08:10:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.898+0000 7fea7579a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea70142dc0 0x7fea700c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.898+0000 7fea7579a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea70142dc0 0x7fea700c6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40162/0 (socket says 172.21.15.27:40162) 2024-01-11T08:10:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.898+0000 7fea7579a700 1 -- 172.21.15.27:0/3722911359 learned_addr learned my addr 172.21.15.27:0/3722911359 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.899+0000 7fea75f9b700 1 --2- 172.21.15.27:0/3722911359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea700cb2e0 0x7fea70141630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.899+0000 7fea7679c700 1 --2- 172.21.15.27:0/3722911359 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea70143190 0x7fea700c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.899+0000 7fea7579a700 1 -- 172.21.15.27:0/3722911359 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea70143190 msgr2=0x7fea700c5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.899+0000 7fea7579a700 1 --2- 172.21.15.27:0/3722911359 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea70143190 0x7fea700c5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.899+0000 7fea7579a700 1 -- 172.21.15.27:0/3722911359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea700cb2e0 msgr2=0x7fea70141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.899+0000 7fea7579a700 1 --2- 172.21.15.27:0/3722911359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea700cb2e0 0x7fea70141630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.899+0000 7fea7579a700 1 -- 172.21.15.27:0/3722911359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6c009420 con 0x7fea70142dc0 2024-01-11T08:10:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.899+0000 7fea7579a700 1 --2- 172.21.15.27:0/3722911359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea70142dc0 0x7fea700c6e40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fea6000c530 tx=0x7fea6000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.899+0000 7fea65ffb700 1 -- 172.21.15.27:0/3722911359 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea60011590 con 0x7fea70142dc0 2024-01-11T08:10:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.899+0000 7fea65ffb700 1 -- 172.21.15.27:0/3722911359 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea60011710 con 0x7fea70142dc0 2024-01-11T08:10:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.899+0000 7fea781ff700 1 -- 172.21.15.27:0/3722911359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea68009ae0 con 0x7fea70142dc0 2024-01-11T08:10:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.899+0000 7fea65ffb700 1 -- 172.21.15.27:0/3722911359 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea60011ad0 con 0x7fea70142dc0 2024-01-11T08:10:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.900+0000 7fea781ff700 1 -- 172.21.15.27:0/3722911359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea700b21f0 con 0x7fea70142dc0 2024-01-11T08:10:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.900+0000 7fea781ff700 1 -- 172.21.15.27:0/3722911359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea7004c3c0 con 0x7fea70142dc0 2024-01-11T08:10:29.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.902+0000 7fea65ffb700 1 -- 172.21.15.27:0/3722911359 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fea60001010 con 0x7fea70142dc0 2024-01-11T08:10:29.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.903+0000 7fea65ffb700 1 --2- 172.21.15.27:0/3722911359 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fea54066f30 0x7fea540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:29.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.903+0000 7fea65ffb700 1 -- 172.21.15.27:0/3722911359 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5605+0+0 (secure 0 0 0) 0x7fea60083900 con 0x7fea70142dc0 2024-01-11T08:10:29.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.904+0000 7fea75f9b700 1 --2- 172.21.15.27:0/3722911359 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fea54066f30 0x7fea540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:29.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.904+0000 7fea75f9b700 1 --2- 172.21.15.27:0/3722911359 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fea54066f30 0x7fea540693e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fea7015a150 tx=0x7fea6c0091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:29.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:29.908+0000 7fea65ffb700 1 -- 172.21.15.27:0/3722911359 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fea60050b30 con 0x7fea70142dc0 2024-01-11T08:10:30.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.159+0000 7fea781ff700 1 -- 172.21.15.27:0/3722911359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fea700b2980 con 0x7fea70142dc0 2024-01-11T08:10:30.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.665+0000 7fea65ffb700 1 -- 172.21.15.27:0/3722911359 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fea60017070 con 0x7fea70142dc0 2024-01-11T08:10:30.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.668+0000 7fea781ff700 1 -- 172.21.15.27:0/3722911359 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fea54066f30 msgr2=0x7fea540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:30.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.668+0000 7fea781ff700 1 --2- 172.21.15.27:0/3722911359 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fea54066f30 0x7fea540693e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fea7015a150 tx=0x7fea6c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:30.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.668+0000 7fea781ff700 1 -- 172.21.15.27:0/3722911359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea70142dc0 msgr2=0x7fea700c6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:30.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.668+0000 7fea781ff700 1 --2- 172.21.15.27:0/3722911359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea70142dc0 0x7fea700c6e40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fea6000c530 tx=0x7fea6000c840 comp rx=0 tx=0).stop 2024-01-11T08:10:30.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.668+0000 7fea781ff700 1 -- 172.21.15.27:0/3722911359 shutdown_connections 2024-01-11T08:10:30.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.668+0000 7fea781ff700 1 --2- 172.21.15.27:0/3722911359 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fea54066f30 0x7fea540693e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:30.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.668+0000 7fea781ff700 1 --2- 172.21.15.27:0/3722911359 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea70143190 0x7fea700c5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:30.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.668+0000 7fea781ff700 1 --2- 172.21.15.27:0/3722911359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea700cb2e0 0x7fea70141630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:30.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.668+0000 7fea781ff700 1 --2- 172.21.15.27:0/3722911359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea70142dc0 0x7fea700c6e40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:30.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.669+0000 7fea781ff700 1 -- 172.21.15.27:0/3722911359 >> 172.21.15.27:0/3722911359 conn(0x7fea70063ca0 msgr2=0x7fea7006cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:30.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.669+0000 7fea781ff700 1 -- 172.21.15.27:0/3722911359 shutdown_connections 2024-01-11T08:10:30.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.669+0000 7fea781ff700 1 -- 172.21.15.27:0/3722911359 wait complete. 2024-01-11T08:10:30.670 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 2 and data pool 3 2024-01-11T08:10:30.693 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:10:30.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.892+0000 7f5d0d1ff700 1 Processor -- start 2024-01-11T08:10:30.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.892+0000 7f5d0d1ff700 1 -- start start 2024-01-11T08:10:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.893+0000 7f5d0d1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5d080b03e0 0x7f5d080b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.893+0000 7f5d0d1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d080cd730 con 0x7f5d080b07b0 2024-01-11T08:10:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.893+0000 7f5d0d1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d080b2de0 con 0x7f5d080b29c0 2024-01-11T08:10:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.893+0000 7f5d0d1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d080ccb20 con 0x7f5d080b03e0 2024-01-11T08:10:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.894+0000 7f5d06d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5d080b03e0 0x7f5d080b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.894+0000 7f5d06d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5d080b03e0 0x7f5d080b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40168/0 (socket says 172.21.15.27:40168) 2024-01-11T08:10:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.894+0000 7f5d06d9d700 1 -- 172.21.15.27:0/1053333439 learned_addr learned my addr 172.21.15.27:0/1053333439 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.894+0000 7f5d06d9d700 1 -- 172.21.15.27:0/1053333439 >> v1:172.21.15.77:6789/0 conn(0x7f5d080b29c0 legacy=0x7f5d080aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.894+0000 7f5d06d9d700 1 -- 172.21.15.27:0/1053333439 >> v1:172.21.15.27:6789/0 conn(0x7f5d080b07b0 legacy=0x7f5d08159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.894+0000 7f5d06d9d700 1 -- 172.21.15.27:0/1053333439 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d080cce00 con 0x7f5d080b03e0 2024-01-11T08:10:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.894+0000 7f5d05d9b700 1 -- 172.21.15.27:0/1053333439 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1304888016 0 0) 0x7f5d080cd730 con 0x7f5d080b07b0 2024-01-11T08:10:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.894+0000 7f5d06d9d700 1 --2- 172.21.15.27:0/1053333439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080b03e0 0x7f5d080b2480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5cf80099b0 tx=0x7f5cf8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3a0052ccf057a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.895+0000 7f5d05d9b700 1 -- 172.21.15.27:0/1053333439 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf800b860 con 0x7f5d080b03e0 2024-01-11T08:10:30.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.895+0000 7f5d05d9b700 1 -- 172.21.15.27:0/1053333439 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf800b9c0 con 0x7f5d080b03e0 2024-01-11T08:10:30.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.895+0000 7f5d05d9b700 1 -- 172.21.15.27:0/1053333439 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf800bd60 con 0x7f5d080b03e0 2024-01-11T08:10:30.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.895+0000 7f5d0d1ff700 1 -- 172.21.15.27:0/1053333439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080b03e0 msgr2=0x7f5d080b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:30.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.895+0000 7f5d0d1ff700 1 --2- 172.21.15.27:0/1053333439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080b03e0 0x7f5d080b2480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5cf80099b0 tx=0x7f5cf8009250 comp rx=0 tx=0).stop 2024-01-11T08:10:30.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.895+0000 7f5d0d1ff700 1 -- 172.21.15.27:0/1053333439 shutdown_connections 2024-01-11T08:10:30.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.895+0000 7f5d0d1ff700 1 --2- 172.21.15.27:0/1053333439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080b03e0 0x7f5d080b2480 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:30.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.895+0000 7f5d0d1ff700 1 -- 172.21.15.27:0/1053333439 >> 172.21.15.27:0/1053333439 conn(0x7f5d0806ed40 msgr2=0x7f5d0806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:30.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.895+0000 7f5d0d1ff700 1 -- 172.21.15.27:0/1053333439 shutdown_connections 2024-01-11T08:10:30.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.895+0000 7f5d0d1ff700 1 -- 172.21.15.27:0/1053333439 wait complete. 2024-01-11T08:10:30.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.897+0000 7f5d0d1ff700 1 Processor -- start 2024-01-11T08:10:30.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.897+0000 7f5d0d1ff700 1 -- start start 2024-01-11T08:10:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.898+0000 7f5d0d1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d080b03e0 0x7f5d0815bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.898+0000 7f5d0d1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080b07b0 0x7f5d080a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.898+0000 7f5d0d1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d080b29c0 0x7f5d0816a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.898+0000 7f5d0d1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d080c4ae0 con 0x7f5d080b07b0 2024-01-11T08:10:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.898+0000 7f5d0d1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0815c140 con 0x7f5d080b29c0 2024-01-11T08:10:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.898+0000 7f5d0d1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d080c3ed0 con 0x7f5d080b03e0 2024-01-11T08:10:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.898+0000 7f5d0659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080b07b0 0x7f5d080a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.898+0000 7f5d0659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080b07b0 0x7f5d080a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40182/0 (socket says 172.21.15.27:40182) 2024-01-11T08:10:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.898+0000 7f5d0659c700 1 -- 172.21.15.27:0/2668761326 learned_addr learned my addr 172.21.15.27:0/2668761326 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.898+0000 7f5d06d9d700 1 --2- 172.21.15.27:0/2668761326 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d080b03e0 0x7f5d0815bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.898+0000 7f5d0759e700 1 --2- 172.21.15.27:0/2668761326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d080b29c0 0x7f5d0816a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.899+0000 7f5d0659c700 1 -- 172.21.15.27:0/2668761326 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d080b03e0 msgr2=0x7f5d0815bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.899+0000 7f5d0659c700 1 --2- 172.21.15.27:0/2668761326 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d080b03e0 0x7f5d0815bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.899+0000 7f5d0659c700 1 -- 172.21.15.27:0/2668761326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d080b29c0 msgr2=0x7f5d0816a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.899+0000 7f5d0659c700 1 --2- 172.21.15.27:0/2668761326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d080b29c0 0x7f5d0816a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:30.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.899+0000 7f5d0659c700 1 -- 172.21.15.27:0/2668761326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf80094f0 con 0x7f5d080b07b0 2024-01-11T08:10:30.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.899+0000 7f5d0659c700 1 --2- 172.21.15.27:0/2668761326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080b07b0 0x7f5d080a7a10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00c7e0 tx=0x7f5cfc007da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:30.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.899+0000 7f5ceeffd700 1 -- 172.21.15.27:0/2668761326 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc011590 con 0x7f5d080b07b0 2024-01-11T08:10:30.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.899+0000 7f5ceeffd700 1 -- 172.21.15.27:0/2668761326 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cfc011710 con 0x7f5d080b07b0 2024-01-11T08:10:30.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.899+0000 7f5d0d1ff700 1 -- 172.21.15.27:0/2668761326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d080c41b0 con 0x7f5d080b07b0 2024-01-11T08:10:30.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.899+0000 7f5ceeffd700 1 -- 172.21.15.27:0/2668761326 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc011ad0 con 0x7f5d080b07b0 2024-01-11T08:10:30.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.900+0000 7f5d0d1ff700 1 -- 172.21.15.27:0/2668761326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d080c4770 con 0x7f5d080b07b0 2024-01-11T08:10:30.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.900+0000 7f5d0d1ff700 1 -- 172.21.15.27:0/2668761326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d080718d0 con 0x7f5d080b07b0 2024-01-11T08:10:30.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.901+0000 7f5ceeffd700 1 -- 172.21.15.27:0/2668761326 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5cfc01b3b0 con 0x7f5d080b07b0 2024-01-11T08:10:30.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.902+0000 7f5ceeffd700 1 --2- 172.21.15.27:0/2668761326 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5cf4066f30 0x7f5cf40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:30.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.902+0000 7f5ceeffd700 1 -- 172.21.15.27:0/2668761326 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f5cfc083a30 con 0x7f5d080b07b0 2024-01-11T08:10:30.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.903+0000 7f5d06d9d700 1 --2- 172.21.15.27:0/2668761326 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5cf4066f30 0x7f5cf40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:30.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.903+0000 7f5d06d9d700 1 --2- 172.21.15.27:0/2668761326 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5cf4066f30 0x7f5cf40693e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5cf80099b0 tx=0x7f5cf8017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:30.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:30.907+0000 7f5ceeffd700 1 -- 172.21.15.27:0/2668761326 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5cfc050c00 con 0x7f5d080b07b0 2024-01-11T08:10:31.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.112+0000 7f5d0d1ff700 1 -- 172.21.15.27:0/2668761326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d080cd0e0 con 0x7f5d080b07b0 2024-01-11T08:10:31.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.113+0000 7f5ceeffd700 1 -- 172.21.15.27:0/2668761326 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+15315 (secure 0 0 0) 0x7f5cfc017070 con 0x7f5d080b07b0 2024-01-11T08:10:31.113 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:31.113 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:30.160445+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:10:27.993251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-11T08:10:29.153959+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:10:31.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.115+0000 7f5d0d1ff700 1 -- 172.21.15.27:0/2668761326 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5cf4066f30 msgr2=0x7f5cf40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:31.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.115+0000 7f5d0d1ff700 1 --2- 172.21.15.27:0/2668761326 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5cf4066f30 0x7f5cf40693e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5cf80099b0 tx=0x7f5cf8017000 comp rx=0 tx=0).stop 2024-01-11T08:10:31.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.116+0000 7f5d0d1ff700 1 -- 172.21.15.27:0/2668761326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080b07b0 msgr2=0x7f5d080a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:31.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.116+0000 7f5d0d1ff700 1 --2- 172.21.15.27:0/2668761326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080b07b0 0x7f5d080a7a10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00c7e0 tx=0x7f5cfc007da0 comp rx=0 tx=0).stop 2024-01-11T08:10:31.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.116+0000 7f5d0d1ff700 1 -- 172.21.15.27:0/2668761326 shutdown_connections 2024-01-11T08:10:31.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.116+0000 7f5d0d1ff700 1 --2- 172.21.15.27:0/2668761326 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5cf4066f30 0x7f5cf40693e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:31.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.116+0000 7f5d0d1ff700 1 --2- 172.21.15.27:0/2668761326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d080b29c0 0x7f5d0816a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:31.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.116+0000 7f5d0d1ff700 1 --2- 172.21.15.27:0/2668761326 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d080b03e0 0x7f5d0815bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:31.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.116+0000 7f5d0d1ff700 1 --2- 172.21.15.27:0/2668761326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080b07b0 0x7f5d080a7a10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:31.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.116+0000 7f5d0d1ff700 1 -- 172.21.15.27:0/2668761326 >> 172.21.15.27:0/2668761326 conn(0x7f5d0806ed40 msgr2=0x7f5d080c95d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:31.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.116+0000 7f5d0d1ff700 1 -- 172.21.15.27:0/2668761326 shutdown_connections 2024-01-11T08:10:31.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.116+0000 7f5d0d1ff700 1 -- 172.21.15.27:0/2668761326 wait complete. 2024-01-11T08:10:31.133 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:10:31.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.331+0000 7f1398dff700 1 Processor -- start 2024-01-11T08:10:31.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.331+0000 7f1398dff700 1 -- start start 2024-01-11T08:10:31.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.332+0000 7f1398dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13940b2b90 0x7f1394159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:31.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.332+0000 7f1398dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13940b07b0 0x7f1394159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:31.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.333+0000 7f1398dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13940cd730 con 0x7f13940b03e0 2024-01-11T08:10:31.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.333+0000 7f1398dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13940ccb20 con 0x7f13940b2b90 2024-01-11T08:10:31.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.333+0000 7f1398dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13940cce00 con 0x7f13940b07b0 2024-01-11T08:10:31.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.333+0000 7f1391d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13940b2b90 0x7f1394159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:31.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.333+0000 7f139259c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f13940b03e0 0x7f13940b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37042/0 (socket says 172.21.15.27:37042) 2024-01-11T08:10:31.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.333+0000 7f139259c700 1 -- 172.21.15.27:0/2937837147 learned_addr learned my addr 172.21.15.27:0/2937837147 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:31.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.333+0000 7f1392d9d700 1 --2- 172.21.15.27:0/2937837147 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13940b07b0 0x7f1394159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:31.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.333+0000 7f1391d9b700 1 -- 172.21.15.27:0/2937837147 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13940b07b0 msgr2=0x7f1394159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:31.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.333+0000 7f1391d9b700 1 --2- 172.21.15.27:0/2937837147 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13940b07b0 0x7f1394159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:31.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.333+0000 7f1391d9b700 1 -- 172.21.15.27:0/2937837147 >> v1:172.21.15.27:6789/0 conn(0x7f13940b03e0 legacy=0x7f13940b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:31.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.333+0000 7f1391d9b700 1 -- 172.21.15.27:0/2937837147 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13940cd0e0 con 0x7f13940b2b90 2024-01-11T08:10:31.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.333+0000 7f139159a700 1 -- 172.21.15.27:0/2937837147 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444171471 0 0) 0x7f13940cd730 con 0x7f13940b03e0 2024-01-11T08:10:31.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.334+0000 7f1391d9b700 1 --2- 172.21.15.27:0/2937837147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13940b2b90 0x7f1394159200 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f13880098e0 tx=0x7f13880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ad4fce91f287d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:31.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.334+0000 7f139159a700 1 -- 172.21.15.27:0/2937837147 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138800b640 con 0x7f13940b2b90 2024-01-11T08:10:31.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.334+0000 7f139159a700 1 -- 172.21.15.27:0/2937837147 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138800b7a0 con 0x7f13940b2b90 2024-01-11T08:10:31.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.334+0000 7f139159a700 1 -- 172.21.15.27:0/2937837147 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138800bb40 con 0x7f13940b2b90 2024-01-11T08:10:31.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.334+0000 7f1398dff700 1 -- 172.21.15.27:0/2937837147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13940b2b90 msgr2=0x7f1394159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:31.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.334+0000 7f1398dff700 1 --2- 172.21.15.27:0/2937837147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13940b2b90 0x7f1394159200 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f13880098e0 tx=0x7f13880091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:31.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.334+0000 7f1398dff700 1 -- 172.21.15.27:0/2937837147 shutdown_connections 2024-01-11T08:10:31.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.335+0000 7f1398dff700 1 --2- 172.21.15.27:0/2937837147 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13940b07b0 0x7f1394159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:31.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.335+0000 7f1398dff700 1 --2- 172.21.15.27:0/2937837147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13940b2b90 0x7f1394159200 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:31.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.335+0000 7f1398dff700 1 -- 172.21.15.27:0/2937837147 >> 172.21.15.27:0/2937837147 conn(0x7f139406ed40 msgr2=0x7f139406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:31.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.335+0000 7f1398dff700 1 -- 172.21.15.27:0/2937837147 shutdown_connections 2024-01-11T08:10:31.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.335+0000 7f1398dff700 1 -- 172.21.15.27:0/2937837147 wait complete. 2024-01-11T08:10:31.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.336+0000 7f1398dff700 1 Processor -- start 2024-01-11T08:10:31.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.336+0000 7f1398dff700 1 -- start start 2024-01-11T08:10:31.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.337+0000 7f1398dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13940b03e0 0x7f139414a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:31.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.337+0000 7f1398dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13940b07b0 0x7f13940c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:31.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.337+0000 7f1398dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13940b2b90 0x7f13940c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:31.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.337+0000 7f1398dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13940c85b0 con 0x7f13940b2b90 2024-01-11T08:10:31.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.337+0000 7f1398dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13940c79a0 con 0x7f13940b03e0 2024-01-11T08:10:31.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.337+0000 7f1398dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13940c7c80 con 0x7f13940b07b0 2024-01-11T08:10:31.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.337+0000 7f1392d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13940b2b90 0x7f13940c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:31.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.337+0000 7f1392d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13940b2b90 0x7f13940c8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40192/0 (socket says 172.21.15.27:40192) 2024-01-11T08:10:31.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.337+0000 7f1392d9d700 1 -- 172.21.15.27:0/1961939508 learned_addr learned my addr 172.21.15.27:0/1961939508 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:31.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.338+0000 7f1391d9b700 1 --2- 172.21.15.27:0/1961939508 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13940b07b0 0x7f13940c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:31.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.338+0000 7f139259c700 1 --2- 172.21.15.27:0/1961939508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13940b03e0 0x7f139414a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:31.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.338+0000 7f1392d9d700 1 -- 172.21.15.27:0/1961939508 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13940b07b0 msgr2=0x7f13940c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:31.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.338+0000 7f1392d9d700 1 --2- 172.21.15.27:0/1961939508 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13940b07b0 0x7f13940c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:31.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.338+0000 7f1392d9d700 1 -- 172.21.15.27:0/1961939508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13940b03e0 msgr2=0x7f139414a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:31.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.338+0000 7f1392d9d700 1 --2- 172.21.15.27:0/1961939508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13940b03e0 0x7f139414a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:31.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.338+0000 7f1392d9d700 1 -- 172.21.15.27:0/1961939508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1388009420 con 0x7f13940b2b90 2024-01-11T08:10:31.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.338+0000 7f1392d9d700 1 --2- 172.21.15.27:0/1961939508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13940b2b90 0x7f13940c8a40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f138c009e90 tx=0x7f138c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:31.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.338+0000 7f13827fc700 1 -- 172.21.15.27:0/1961939508 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138c00cb70 con 0x7f13940b2b90 2024-01-11T08:10:31.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.338+0000 7f13827fc700 1 -- 172.21.15.27:0/1961939508 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138c00f040 con 0x7f13940b2b90 2024-01-11T08:10:31.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.338+0000 7f13827fc700 1 -- 172.21.15.27:0/1961939508 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138c014480 con 0x7f13940b2b90 2024-01-11T08:10:31.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.338+0000 7f1398dff700 1 -- 172.21.15.27:0/1961939508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13940c7f60 con 0x7f13940b2b90 2024-01-11T08:10:31.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.339+0000 7f1398dff700 1 -- 172.21.15.27:0/1961939508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f139414ae10 con 0x7f13940b2b90 2024-01-11T08:10:31.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.340+0000 7f1398dff700 1 -- 172.21.15.27:0/1961939508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13940c8720 con 0x7f13940b2b90 2024-01-11T08:10:31.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.341+0000 7f13827fc700 1 -- 172.21.15.27:0/1961939508 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f138c014830 con 0x7f13940b2b90 2024-01-11T08:10:31.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.342+0000 7f13827fc700 1 --2- 172.21.15.27:0/1961939508 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1374066f30 0x7f13740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:31.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.342+0000 7f13827fc700 1 -- 172.21.15.27:0/1961939508 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f138c084f00 con 0x7f13940b2b90 2024-01-11T08:10:31.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.342+0000 7f139259c700 1 --2- 172.21.15.27:0/1961939508 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1374066f30 0x7f13740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:31.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.343+0000 7f139259c700 1 --2- 172.21.15.27:0/1961939508 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1374066f30 0x7f13740693e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1384007fd0 tx=0x7f1384007d30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:31.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.346+0000 7f13827fc700 1 -- 172.21.15.27:0/1961939508 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f138c052170 con 0x7f13940b2b90 2024-01-11T08:10:31.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.552+0000 7f1398dff700 1 -- 172.21.15.27:0/1961939508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13940718d0 con 0x7f13940b2b90 2024-01-11T08:10:31.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.553+0000 7f13827fc700 1 -- 172.21.15.27:0/1961939508 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15302 (secure 0 0 0) 0x7f138c055790 con 0x7f13940b2b90 2024-01-11T08:10:31.554 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:31.554 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:31.464323+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:10:27.993251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-11T08:10:29.153959+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:10:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.555+0000 7f1398dff700 1 -- 172.21.15.27:0/1961939508 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1374066f30 msgr2=0x7f13740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.555+0000 7f1398dff700 1 --2- 172.21.15.27:0/1961939508 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1374066f30 0x7f13740693e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1384007fd0 tx=0x7f1384007d30 comp rx=0 tx=0).stop 2024-01-11T08:10:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.555+0000 7f1398dff700 1 -- 172.21.15.27:0/1961939508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13940b2b90 msgr2=0x7f13940c8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.555+0000 7f1398dff700 1 --2- 172.21.15.27:0/1961939508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13940b2b90 0x7f13940c8a40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f138c009e90 tx=0x7f138c00b040 comp rx=0 tx=0).stop 2024-01-11T08:10:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.555+0000 7f1398dff700 1 -- 172.21.15.27:0/1961939508 shutdown_connections 2024-01-11T08:10:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.555+0000 7f1398dff700 1 --2- 172.21.15.27:0/1961939508 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1374066f30 0x7f13740693e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.555+0000 7f1398dff700 1 --2- 172.21.15.27:0/1961939508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13940b2b90 0x7f13940c8a40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.555+0000 7f1398dff700 1 --2- 172.21.15.27:0/1961939508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13940b03e0 0x7f139414a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.555+0000 7f1398dff700 1 --2- 172.21.15.27:0/1961939508 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13940b07b0 0x7f13940c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.555+0000 7f1398dff700 1 -- 172.21.15.27:0/1961939508 >> 172.21.15.27:0/1961939508 conn(0x7f139406ed40 msgr2=0x7f13940d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.555+0000 7f1398dff700 1 -- 172.21.15.27:0/1961939508 shutdown_connections 2024-01-11T08:10:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.555+0000 7f1398dff700 1 -- 172.21.15.27:0/1961939508 wait complete. 2024-01-11T08:10:31.574 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-11T08:10:31.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.777+0000 7f02e63ff700 1 Processor -- start 2024-01-11T08:10:31.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.778+0000 7f02e63ff700 1 -- start start 2024-01-11T08:10:31.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.779+0000 7f02e63ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02e01598f0 0x7f02e01646c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:31.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.779+0000 7f02e63ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e015aef0 con 0x7f02e00cd220 2024-01-11T08:10:31.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.779+0000 7f02e63ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e00cee00 con 0x7f02e00cd5f0 2024-01-11T08:10:31.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.779+0000 7f02e63ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e015dd70 con 0x7f02e01598f0 2024-01-11T08:10:31.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.779+0000 7f02dffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f02e00cd220 0x7f02e015d320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37044/0 (socket says 172.21.15.27:37044) 2024-01-11T08:10:31.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.779+0000 7f02dffff700 1 -- 172.21.15.27:0/1917406846 learned_addr learned my addr 172.21.15.27:0/1917406846 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:31.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.779+0000 7f02e499c700 1 --2- 172.21.15.27:0/1917406846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02e01598f0 0x7f02e01646c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:31.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.780+0000 7f02deffd700 1 -- 172.21.15.27:0/1917406846 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1657989683 0 0) 0x7f02e015aef0 con 0x7f02e00cd220 2024-01-11T08:10:31.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.780+0000 7f02deffd700 1 -- 172.21.15.27:0/1917406846 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c8003570 con 0x7f02e00cd220 2024-01-11T08:10:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.780+0000 7f02deffd700 1 -- 172.21.15.27:0/1917406846 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635925448 0 0) 0x7f02e00cee00 con 0x7f02e00cd5f0 2024-01-11T08:10:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.780+0000 7f02deffd700 1 -- 172.21.15.27:0/1917406846 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e015aef0 con 0x7f02e00cd5f0 2024-01-11T08:10:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.780+0000 7f02e499c700 1 -- 172.21.15.27:0/1917406846 >> v1:172.21.15.77:6789/0 conn(0x7f02e00cd5f0 legacy=0x7f02e01591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.780+0000 7f02deffd700 1 -- 172.21.15.27:0/1917406846 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3082555255 0 0) 0x7f02c8003570 con 0x7f02e00cd220 2024-01-11T08:10:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.780+0000 7f02e499c700 1 -- 172.21.15.27:0/1917406846 >> v1:172.21.15.27:6789/0 conn(0x7f02e00cd220 legacy=0x7f02e015d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.780+0000 7f02e499c700 1 -- 172.21.15.27:0/1917406846 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e015e050 con 0x7f02e01598f0 2024-01-11T08:10:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.780+0000 7f02e499c700 1 --2- 172.21.15.27:0/1917406846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02e01598f0 0x7f02e01646c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f02d8009e90 tx=0x7f02d8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3070d8553f52fe34 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.780+0000 7f02deffd700 1 -- 172.21.15.27:0/1917406846 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d800e070 con 0x7f02e01598f0 2024-01-11T08:10:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.780+0000 7f02deffd700 1 -- 172.21.15.27:0/1917406846 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d800b870 con 0x7f02e01598f0 2024-01-11T08:10:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.781+0000 7f02deffd700 1 -- 172.21.15.27:0/1917406846 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d0001470 con 0x7f02e01598f0 2024-01-11T08:10:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.781+0000 7f02e63ff700 1 -- 172.21.15.27:0/1917406846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02e01598f0 msgr2=0x7f02e01646c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.781+0000 7f02e63ff700 1 --2- 172.21.15.27:0/1917406846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02e01598f0 0x7f02e01646c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f02d8009e90 tx=0x7f02d8009780 comp rx=0 tx=0).stop 2024-01-11T08:10:31.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.781+0000 7f02e63ff700 1 -- 172.21.15.27:0/1917406846 shutdown_connections 2024-01-11T08:10:31.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.781+0000 7f02e63ff700 1 --2- 172.21.15.27:0/1917406846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02e01598f0 0x7f02e01646c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:31.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.781+0000 7f02e63ff700 1 -- 172.21.15.27:0/1917406846 >> 172.21.15.27:0/1917406846 conn(0x7f02e006ed40 msgr2=0x7f02e006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:31.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.781+0000 7f02e63ff700 1 -- 172.21.15.27:0/1917406846 shutdown_connections 2024-01-11T08:10:31.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.781+0000 7f02e63ff700 1 -- 172.21.15.27:0/1917406846 wait complete. 2024-01-11T08:10:31.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.783+0000 7f02e63ff700 1 Processor -- start 2024-01-11T08:10:31.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.783+0000 7f02e63ff700 1 -- start start 2024-01-11T08:10:31.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.784+0000 7f02e63ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02e00cd220 0x7f02e014d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:31.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.784+0000 7f02e63ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02e00cd5f0 0x7f02e0152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:31.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.784+0000 7f02e63ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02e01598f0 0x7f02e015c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:31.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.784+0000 7f02e63ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e00b20c0 con 0x7f02e00cd5f0 2024-01-11T08:10:31.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.784+0000 7f02e63ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e00b2550 con 0x7f02e00cd220 2024-01-11T08:10:31.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.784+0000 7f02e63ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e00b14b0 con 0x7f02e01598f0 2024-01-11T08:10:31.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.784+0000 7f02df7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02e00cd5f0 0x7f02e0152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:31.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.784+0000 7f02df7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02e00cd5f0 0x7f02e0152330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40200/0 (socket says 172.21.15.27:40200) 2024-01-11T08:10:31.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.784+0000 7f02df7fe700 1 -- 172.21.15.27:0/251600314 learned_addr learned my addr 172.21.15.27:0/251600314 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:31.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.784+0000 7f02e499c700 1 --2- 172.21.15.27:0/251600314 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02e01598f0 0x7f02e015c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.784+0000 7f02dffff700 1 --2- 172.21.15.27:0/251600314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02e00cd220 0x7f02e014d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.784+0000 7f02df7fe700 1 -- 172.21.15.27:0/251600314 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02e01598f0 msgr2=0x7f02e015c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.784+0000 7f02df7fe700 1 --2- 172.21.15.27:0/251600314 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02e01598f0 0x7f02e015c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.784+0000 7f02df7fe700 1 -- 172.21.15.27:0/251600314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02e00cd220 msgr2=0x7f02e014d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.784+0000 7f02df7fe700 1 --2- 172.21.15.27:0/251600314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02e00cd220 0x7f02e014d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.784+0000 7f02df7fe700 1 -- 172.21.15.27:0/251600314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d8009a20 con 0x7f02e00cd5f0 2024-01-11T08:10:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.784+0000 7f02dffff700 1 --2- 172.21.15.27:0/251600314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02e00cd220 0x7f02e014d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.785+0000 7f02df7fe700 1 --2- 172.21.15.27:0/251600314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02e00cd5f0 0x7f02e0152330 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f02d400c7c0 tx=0x7f02d4007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.785+0000 7f02c7fff700 1 -- 172.21.15.27:0/251600314 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d4001520 con 0x7f02e00cd5f0 2024-01-11T08:10:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.785+0000 7f02c7fff700 1 -- 172.21.15.27:0/251600314 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d40016a0 con 0x7f02e00cd5f0 2024-01-11T08:10:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.785+0000 7f02c7fff700 1 -- 172.21.15.27:0/251600314 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d4001a60 con 0x7f02e00cd5f0 2024-01-11T08:10:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.785+0000 7f02e63ff700 1 -- 172.21.15.27:0/251600314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e00b1790 con 0x7f02e00cd5f0 2024-01-11T08:10:31.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.785+0000 7f02e63ff700 1 -- 172.21.15.27:0/251600314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e00b1d50 con 0x7f02e00cd5f0 2024-01-11T08:10:31.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.786+0000 7f02e63ff700 1 -- 172.21.15.27:0/251600314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e004c3c0 con 0x7f02e00cd5f0 2024-01-11T08:10:31.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.787+0000 7f02c7fff700 1 -- 172.21.15.27:0/251600314 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f02d401b3b0 con 0x7f02e00cd5f0 2024-01-11T08:10:31.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.788+0000 7f02c7fff700 1 --2- 172.21.15.27:0/251600314 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f02cc066f30 0x7f02cc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:31.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.788+0000 7f02c7fff700 1 -- 172.21.15.27:0/251600314 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f02d4083f10 con 0x7f02e00cd5f0 2024-01-11T08:10:31.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.789+0000 7f02dffff700 1 --2- 172.21.15.27:0/251600314 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f02cc066f30 0x7f02cc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:31.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.789+0000 7f02dffff700 1 --2- 172.21.15.27:0/251600314 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f02cc066f30 0x7f02cc0693e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f02d0001660 tx=0x7f02d00019d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:31.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:31.792+0000 7f02c7fff700 1 -- 172.21.15.27:0/251600314 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f02d40510e0 con 0x7f02e00cd5f0 2024-01-11T08:10:32.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.042+0000 7f02e63ff700 1 -- 172.21.15.27:0/251600314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f02e00b2200 con 0x7f02e00cd5f0 2024-01-11T08:10:32.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.680+0000 7f02c7fff700 1 -- 172.21.15.27:0/251600314 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f02d4017020 con 0x7f02e00cd5f0 2024-01-11T08:10:32.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.683+0000 7f02e63ff700 1 -- 172.21.15.27:0/251600314 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f02cc066f30 msgr2=0x7f02cc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:32.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.683+0000 7f02e63ff700 1 --2- 172.21.15.27:0/251600314 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f02cc066f30 0x7f02cc0693e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f02d0001660 tx=0x7f02d00019d0 comp rx=0 tx=0).stop 2024-01-11T08:10:32.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.683+0000 7f02e63ff700 1 -- 172.21.15.27:0/251600314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02e00cd5f0 msgr2=0x7f02e0152330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:32.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.683+0000 7f02e63ff700 1 --2- 172.21.15.27:0/251600314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02e00cd5f0 0x7f02e0152330 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f02d400c7c0 tx=0x7f02d4007dd0 comp rx=0 tx=0).stop 2024-01-11T08:10:32.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.683+0000 7f02e63ff700 1 -- 172.21.15.27:0/251600314 shutdown_connections 2024-01-11T08:10:32.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.683+0000 7f02e63ff700 1 --2- 172.21.15.27:0/251600314 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f02cc066f30 0x7f02cc0693e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:32.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.683+0000 7f02e63ff700 1 --2- 172.21.15.27:0/251600314 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02e01598f0 0x7f02e015c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:32.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.683+0000 7f02e63ff700 1 --2- 172.21.15.27:0/251600314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02e00cd220 0x7f02e014d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:32.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.683+0000 7f02e63ff700 1 --2- 172.21.15.27:0/251600314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02e00cd5f0 0x7f02e0152330 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:32.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.683+0000 7f02e63ff700 1 -- 172.21.15.27:0/251600314 >> 172.21.15.27:0/251600314 conn(0x7f02e006ed40 msgr2=0x7f02e015d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:32.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.683+0000 7f02e63ff700 1 -- 172.21.15.27:0/251600314 shutdown_connections 2024-01-11T08:10:32.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.683+0000 7f02e63ff700 1 -- 172.21.15.27:0/251600314 wait complete. 2024-01-11T08:10:32.704 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-11T08:10:32.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.911+0000 7fb1fc9ff700 1 Processor -- start 2024-01-11T08:10:32.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.911+0000 7fb1fc9ff700 1 -- start start 2024-01-11T08:10:32.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.912+0000 7fb1fc9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1f813ff80 0x7fb1f815a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:32.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.912+0000 7fb1fc9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb1f815a560 0x7fb1f80c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:32.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.912+0000 7fb1fc9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f80c6b30 con 0x7fb1f813fbb0 2024-01-11T08:10:32.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.912+0000 7fb1fc9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f80c6800 con 0x7fb1f813ff80 2024-01-11T08:10:32.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.912+0000 7fb1fc9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f815a960 con 0x7fb1f815a560 2024-01-11T08:10:32.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.913+0000 7fb1f5d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1f813ff80 0x7fb1f815a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:32.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.913+0000 7fb1f5d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1f813ff80 0x7fb1f815a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40212/0 (socket says 172.21.15.27:40212) 2024-01-11T08:10:32.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.913+0000 7fb1f5d9b700 1 -- 172.21.15.27:0/1827120839 learned_addr learned my addr 172.21.15.27:0/1827120839 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:32.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.913+0000 7fb1f6d9d700 1 --2- 172.21.15.27:0/1827120839 >> v2:172.21.15.77:3300/0 conn(0x7fb1f815a560 0x7fb1f80c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.913+0000 7fb1f5d9b700 1 -- 172.21.15.27:0/1827120839 >> v2:172.21.15.77:3300/0 conn(0x7fb1f815a560 msgr2=0x7fb1f80c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.913+0000 7fb1f5d9b700 1 --2- 172.21.15.27:0/1827120839 >> v2:172.21.15.77:3300/0 conn(0x7fb1f815a560 0x7fb1f80c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.913+0000 7fb1f5d9b700 1 -- 172.21.15.27:0/1827120839 >> v1:172.21.15.77:6789/0 conn(0x7fb1f813fbb0 legacy=0x7fb1f80ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.913+0000 7fb1f5d9b700 1 -- 172.21.15.27:0/1827120839 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f80cb240 con 0x7fb1f813ff80 2024-01-11T08:10:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.913+0000 7fb1f6d9d700 1 --2- 172.21.15.27:0/1827120839 >> v2:172.21.15.77:3300/0 conn(0x7fb1f815a560 0x7fb1f80c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.913+0000 7fb1f5d9b700 1 --2- 172.21.15.27:0/1827120839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1f813ff80 0x7fb1f815a020 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb1f806fe10 tx=0x7fb1ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d611f3e91115989 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:32.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.914+0000 7fb1f559a700 1 -- 172.21.15.27:0/1827120839 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec00b860 con 0x7fb1f813ff80 2024-01-11T08:10:32.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.914+0000 7fb1f559a700 1 -- 172.21.15.27:0/1827120839 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1ec00b9c0 con 0x7fb1f813ff80 2024-01-11T08:10:32.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.914+0000 7fb1f559a700 1 -- 172.21.15.27:0/1827120839 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec00bd60 con 0x7fb1f813ff80 2024-01-11T08:10:32.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.914+0000 7fb1fc9ff700 1 -- 172.21.15.27:0/1827120839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1f813ff80 msgr2=0x7fb1f815a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:32.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.914+0000 7fb1fc9ff700 1 --2- 172.21.15.27:0/1827120839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1f813ff80 0x7fb1f815a020 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb1f806fe10 tx=0x7fb1ec009250 comp rx=0 tx=0).stop 2024-01-11T08:10:32.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.914+0000 7fb1fc9ff700 1 -- 172.21.15.27:0/1827120839 shutdown_connections 2024-01-11T08:10:32.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.914+0000 7fb1fc9ff700 1 --2- 172.21.15.27:0/1827120839 >> v2:172.21.15.77:3300/0 conn(0x7fb1f815a560 0x7fb1f80c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:32.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.914+0000 7fb1fc9ff700 1 --2- 172.21.15.27:0/1827120839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1f813ff80 0x7fb1f815a020 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:32.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.914+0000 7fb1fc9ff700 1 -- 172.21.15.27:0/1827120839 >> 172.21.15.27:0/1827120839 conn(0x7fb1f80680a0 msgr2=0x7fb1f80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:32.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.914+0000 7fb1fc9ff700 1 -- 172.21.15.27:0/1827120839 shutdown_connections 2024-01-11T08:10:32.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.915+0000 7fb1fc9ff700 1 -- 172.21.15.27:0/1827120839 wait complete. 2024-01-11T08:10:32.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.916+0000 7fb1fc9ff700 1 Processor -- start 2024-01-11T08:10:32.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.916+0000 7fb1fc9ff700 1 -- start start 2024-01-11T08:10:32.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.917+0000 7fb1fc9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f813fbb0 0x7fb1f80b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:32.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.917+0000 7fb1fc9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1f813ff80 0x7fb1f80af990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:32.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.917+0000 7fb1fc9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb1f815a560 0x7fb1f8069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:32.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.917+0000 7fb1fc9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f8156a90 con 0x7fb1f813ff80 2024-01-11T08:10:32.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.917+0000 7fb1fc9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f80afed0 con 0x7fb1f813fbb0 2024-01-11T08:10:32.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.917+0000 7fb1fc9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f80b0720 con 0x7fb1f815a560 2024-01-11T08:10:32.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.917+0000 7fb1f5d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1f813ff80 0x7fb1f80af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:32.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.917+0000 7fb1f5d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1f813ff80 0x7fb1f80af990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40222/0 (socket says 172.21.15.27:40222) 2024-01-11T08:10:32.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.917+0000 7fb1f659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f813fbb0 0x7fb1f80b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:32.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.917+0000 7fb1f5d9b700 1 -- 172.21.15.27:0/4145064111 learned_addr learned my addr 172.21.15.27:0/4145064111 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:32.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.918+0000 7fb1f6d9d700 1 --2- 172.21.15.27:0/4145064111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb1f815a560 0x7fb1f8069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:32.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.918+0000 7fb1f5d9b700 1 -- 172.21.15.27:0/4145064111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb1f815a560 msgr2=0x7fb1f8069c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:32.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.918+0000 7fb1f5d9b700 1 --2- 172.21.15.27:0/4145064111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb1f815a560 0x7fb1f8069c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:32.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.918+0000 7fb1f5d9b700 1 -- 172.21.15.27:0/4145064111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f813fbb0 msgr2=0x7fb1f80b0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:32.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.918+0000 7fb1f5d9b700 1 --2- 172.21.15.27:0/4145064111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f813fbb0 0x7fb1f80b0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:32.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.918+0000 7fb1f5d9b700 1 -- 172.21.15.27:0/4145064111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ec0094f0 con 0x7fb1f813ff80 2024-01-11T08:10:32.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.918+0000 7fb1f5d9b700 1 --2- 172.21.15.27:0/4145064111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1f813ff80 0x7fb1f80af990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8069490 tx=0x7fb1ec017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:32.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.918+0000 7fb1e67fc700 1 -- 172.21.15.27:0/4145064111 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec024440 con 0x7fb1f813ff80 2024-01-11T08:10:32.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.918+0000 7fb1e67fc700 1 -- 172.21.15.27:0/4145064111 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1ec0245a0 con 0x7fb1f813ff80 2024-01-11T08:10:32.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.918+0000 7fb1e67fc700 1 -- 172.21.15.27:0/4145064111 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec024960 con 0x7fb1f813ff80 2024-01-11T08:10:32.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.919+0000 7fb1fc9ff700 1 -- 172.21.15.27:0/4145064111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f80b0a00 con 0x7fb1f813ff80 2024-01-11T08:10:32.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.919+0000 7fb1fc9ff700 1 -- 172.21.15.27:0/4145064111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f8156fc0 con 0x7fb1f813ff80 2024-01-11T08:10:32.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.920+0000 7fb1fc9ff700 1 -- 172.21.15.27:0/4145064111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f81571b0 con 0x7fb1f813ff80 2024-01-11T08:10:32.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.921+0000 7fb1e67fc700 1 -- 172.21.15.27:0/4145064111 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb1ec01a420 con 0x7fb1f813ff80 2024-01-11T08:10:32.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.922+0000 7fb1e67fc700 1 --2- 172.21.15.27:0/4145064111 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb1d8066f30 0x7fb1d80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:32.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.922+0000 7fb1e67fc700 1 -- 172.21.15.27:0/4145064111 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fb1ec086360 con 0x7fb1f813ff80 2024-01-11T08:10:32.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.922+0000 7fb1f659c700 1 --2- 172.21.15.27:0/4145064111 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb1d8066f30 0x7fb1d80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:32.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.923+0000 7fb1f659c700 1 --2- 172.21.15.27:0/4145064111 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb1d8066f30 0x7fb1d80693e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8007fd0 tx=0x7fb1e8007eb0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:32.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:32.926+0000 7fb1e67fc700 1 -- 172.21.15.27:0/4145064111 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb1ec0535d0 con 0x7fb1f813ff80 2024-01-11T08:10:33.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.177+0000 7fb1fc9ff700 1 -- 172.21.15.27:0/4145064111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb1f804c3c0 con 0x7fb1f813ff80 2024-01-11T08:10:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.688+0000 7fb1e67fc700 1 -- 172.21.15.27:0/4145064111 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb1ec056bf0 con 0x7fb1f813ff80 2024-01-11T08:10:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.691+0000 7fb1fc9ff700 1 -- 172.21.15.27:0/4145064111 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb1d8066f30 msgr2=0x7fb1d80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.691+0000 7fb1fc9ff700 1 --2- 172.21.15.27:0/4145064111 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb1d8066f30 0x7fb1d80693e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8007fd0 tx=0x7fb1e8007eb0 comp rx=0 tx=0).stop 2024-01-11T08:10:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.691+0000 7fb1fc9ff700 1 -- 172.21.15.27:0/4145064111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1f813ff80 msgr2=0x7fb1f80af990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.691+0000 7fb1fc9ff700 1 --2- 172.21.15.27:0/4145064111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1f813ff80 0x7fb1f80af990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8069490 tx=0x7fb1ec017000 comp rx=0 tx=0).stop 2024-01-11T08:10:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.691+0000 7fb1fc9ff700 1 -- 172.21.15.27:0/4145064111 shutdown_connections 2024-01-11T08:10:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.691+0000 7fb1fc9ff700 1 --2- 172.21.15.27:0/4145064111 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb1d8066f30 0x7fb1d80693e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.691+0000 7fb1fc9ff700 1 --2- 172.21.15.27:0/4145064111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb1f815a560 0x7fb1f8069c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.691+0000 7fb1fc9ff700 1 --2- 172.21.15.27:0/4145064111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f813fbb0 0x7fb1f80b0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.691+0000 7fb1fc9ff700 1 --2- 172.21.15.27:0/4145064111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1f813ff80 0x7fb1f80af990 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.691+0000 7fb1fc9ff700 1 -- 172.21.15.27:0/4145064111 >> 172.21.15.27:0/4145064111 conn(0x7fb1f80680a0 msgr2=0x7fb1f80704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.691+0000 7fb1fc9ff700 1 -- 172.21.15.27:0/4145064111 shutdown_connections 2024-01-11T08:10:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.691+0000 7fb1fc9ff700 1 -- 172.21.15.27:0/4145064111 wait complete. 2024-01-11T08:10:33.710 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:10:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.909+0000 7f766c1ff700 1 Processor -- start 2024-01-11T08:10:33.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.909+0000 7f766c1ff700 1 -- start start 2024-01-11T08:10:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.910+0000 7f766c1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76640b2b90 0x7f7664159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.910+0000 7f766c1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f76640b07b0 0x7f7664159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:33.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.910+0000 7f766c1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76640cd730 con 0x7f76640b03e0 2024-01-11T08:10:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.910+0000 7f766c1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76640ccb20 con 0x7f76640b07b0 2024-01-11T08:10:33.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.910+0000 7f766c1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76640cce00 con 0x7f76640b2b90 2024-01-11T08:10:33.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.911+0000 7f766979a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76640b2b90 0x7f7664159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:33.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.911+0000 7f766a79c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f76640b07b0 0x7f7664159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:33.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.911+0000 7f7669f9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f76640b03e0 0x7f76640b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:60676/0 (socket says 172.21.15.27:60676) 2024-01-11T08:10:33.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.911+0000 7f7669f9b700 1 -- 172.21.15.27:0/1491512945 learned_addr learned my addr 172.21.15.27:0/1491512945 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:33.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.911+0000 7f766979a700 1 -- 172.21.15.27:0/1491512945 >> v2:172.21.15.77:3300/0 conn(0x7f76640b07b0 msgr2=0x7f7664159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:33.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.911+0000 7f766979a700 1 --2- 172.21.15.27:0/1491512945 >> v2:172.21.15.77:3300/0 conn(0x7f76640b07b0 0x7f7664159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:33.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.911+0000 7f766979a700 1 -- 172.21.15.27:0/1491512945 >> v1:172.21.15.77:6789/0 conn(0x7f76640b03e0 legacy=0x7f76640b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:33.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.912+0000 7f766979a700 1 -- 172.21.15.27:0/1491512945 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76640cd0e0 con 0x7f76640b2b90 2024-01-11T08:10:33.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.912+0000 7f766a79c700 1 --2- 172.21.15.27:0/1491512945 >> v2:172.21.15.77:3300/0 conn(0x7f76640b07b0 0x7f7664159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:33.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.912+0000 7f766979a700 1 --2- 172.21.15.27:0/1491512945 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76640b2b90 0x7f7664159200 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f76540098e0 tx=0x7f76540091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb6a9e7c3bee52ec server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:33.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.912+0000 7f7668f99700 1 -- 172.21.15.27:0/1491512945 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765400b640 con 0x7f76640b2b90 2024-01-11T08:10:33.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.912+0000 7f7668f99700 1 -- 172.21.15.27:0/1491512945 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765400b7a0 con 0x7f76640b2b90 2024-01-11T08:10:33.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.912+0000 7f7668f99700 1 -- 172.21.15.27:0/1491512945 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765400bb40 con 0x7f76640b2b90 2024-01-11T08:10:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.913+0000 7f766c1ff700 1 -- 172.21.15.27:0/1491512945 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76640b2b90 msgr2=0x7f7664159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.913+0000 7f766c1ff700 1 --2- 172.21.15.27:0/1491512945 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76640b2b90 0x7f7664159200 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f76540098e0 tx=0x7f76540091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.913+0000 7f766c1ff700 1 -- 172.21.15.27:0/1491512945 shutdown_connections 2024-01-11T08:10:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.913+0000 7f766c1ff700 1 --2- 172.21.15.27:0/1491512945 >> v2:172.21.15.77:3300/0 conn(0x7f76640b07b0 0x7f7664159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.913+0000 7f766c1ff700 1 --2- 172.21.15.27:0/1491512945 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76640b2b90 0x7f7664159200 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.913+0000 7f766c1ff700 1 -- 172.21.15.27:0/1491512945 >> 172.21.15.27:0/1491512945 conn(0x7f766406ed40 msgr2=0x7f766406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.913+0000 7f766c1ff700 1 -- 172.21.15.27:0/1491512945 shutdown_connections 2024-01-11T08:10:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.913+0000 7f766c1ff700 1 -- 172.21.15.27:0/1491512945 wait complete. 2024-01-11T08:10:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.914+0000 7f766c1ff700 1 Processor -- start 2024-01-11T08:10:33.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.914+0000 7f766c1ff700 1 -- start start 2024-01-11T08:10:33.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.915+0000 7f766c1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76640b03e0 0x7f766415ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:33.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.915+0000 7f766c1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76640b07b0 0x7f766415a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:33.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.915+0000 7f766c1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76640b2b90 0x7f76641589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:33.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.915+0000 7f766c1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766416a180 con 0x7f76640b07b0 2024-01-11T08:10:33.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.915+0000 7f766c1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766416a450 con 0x7f76640b03e0 2024-01-11T08:10:33.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.915+0000 7f766c1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766415af70 con 0x7f76640b2b90 2024-01-11T08:10:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.916+0000 7f766979a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76640b07b0 0x7f766415a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.916+0000 7f766a79c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76640b2b90 0x7f76641589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.916+0000 7f7669f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76640b03e0 0x7f766415ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.916+0000 7f766979a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76640b07b0 0x7f766415a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53074/0 (socket says 172.21.15.27:53074) 2024-01-11T08:10:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.916+0000 7f766979a700 1 -- 172.21.15.27:0/2309727374 learned_addr learned my addr 172.21.15.27:0/2309727374 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.916+0000 7f766a79c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76640b2b90 0x7f76641589d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:59384/0 (socket says 172.21.15.27:59384) 2024-01-11T08:10:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.916+0000 7f766a79c700 1 -- 172.21.15.27:0/2309727374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76640b03e0 msgr2=0x7f766415ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.916+0000 7f766a79c700 1 --2- 172.21.15.27:0/2309727374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76640b03e0 0x7f766415ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:33.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.916+0000 7f766a79c700 1 -- 172.21.15.27:0/2309727374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76640b07b0 msgr2=0x7f766415a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:33.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.916+0000 7f766a79c700 1 --2- 172.21.15.27:0/2309727374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76640b07b0 0x7f766415a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:33.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.916+0000 7f766a79c700 1 -- 172.21.15.27:0/2309727374 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765c009a20 con 0x7f76640b2b90 2024-01-11T08:10:33.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.916+0000 7f766979a700 1 --2- 172.21.15.27:0/2309727374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76640b07b0 0x7f766415a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:33.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.916+0000 7f766a79c700 1 --2- 172.21.15.27:0/2309727374 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76640b2b90 0x7f76641589d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f765c009f10 tx=0x7f765c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:33.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.917+0000 7f7659ffb700 1 -- 172.21.15.27:0/2309727374 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c00cb70 con 0x7f76640b2b90 2024-01-11T08:10:33.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.917+0000 7f7659ffb700 1 -- 172.21.15.27:0/2309727374 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765c00f040 con 0x7f76640b2b90 2024-01-11T08:10:33.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.917+0000 7f7659ffb700 1 -- 172.21.15.27:0/2309727374 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c014480 con 0x7f76640b2b90 2024-01-11T08:10:33.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.917+0000 7f766c1ff700 1 -- 172.21.15.27:0/2309727374 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7654009420 con 0x7f76640b2b90 2024-01-11T08:10:33.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.917+0000 7f766c1ff700 1 -- 172.21.15.27:0/2309727374 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f766415b530 con 0x7f76640b2b90 2024-01-11T08:10:33.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.918+0000 7f766c1ff700 1 -- 172.21.15.27:0/2309727374 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766415b750 con 0x7f76640b2b90 2024-01-11T08:10:33.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.919+0000 7f7659ffb700 1 -- 172.21.15.27:0/2309727374 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f765c003020 con 0x7f76640b2b90 2024-01-11T08:10:33.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.920+0000 7f7659ffb700 1 --2- 172.21.15.27:0/2309727374 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7648066f30 0x7f76480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:33.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.920+0000 7f7659ffb700 1 -- 172.21.15.27:0/2309727374 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f765c084d80 con 0x7f76640b2b90 2024-01-11T08:10:33.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.920+0000 7f7669f9b700 1 --2- 172.21.15.27:0/2309727374 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7648066f30 0x7f76480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:33.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.921+0000 7f7669f9b700 1 --2- 172.21.15.27:0/2309727374 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7648066f30 0x7f76480693e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7660004fc0 tx=0x7f7660007eb0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:33.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:33.926+0000 7f7659ffb700 1 -- 172.21.15.27:0/2309727374 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f765c051f50 con 0x7f76640b2b90 2024-01-11T08:10:34.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.175+0000 7f766c1ff700 1 -- 172.21.15.27:0/2309727374 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76640718d0 con 0x7f76640b2b90 2024-01-11T08:10:34.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.177+0000 7f7659ffb700 1 -- 172.21.15.27:0/2309727374 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4588 (secure 0 0 0) 0x7f765c055570 con 0x7f76640b2b90 2024-01-11T08:10:34.177 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:34.177 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3168764915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3168764915},{"type":"v1","addr":"172.21.15.27:6834","nonce":3168764915}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2607167395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2607167395},{"type":"v1","addr":"172.21.15.27:6835","nonce":2607167395}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2123423970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2123423970},{"type":"v1","addr":"172.21.15.77:6837","nonce":2123423970}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:10:30.160827+0000","modified":"2024-01-11T08:10:33.681210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6836/269464429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":269464429},{"type":"v1","addr":"172.21.15.77:6836","nonce":269464429}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2024-01-11T08:10:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.178+0000 7f766c1ff700 1 -- 172.21.15.27:0/2309727374 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7648066f30 msgr2=0x7f76480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.178+0000 7f766c1ff700 1 --2- 172.21.15.27:0/2309727374 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7648066f30 0x7f76480693e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7660004fc0 tx=0x7f7660007eb0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.178+0000 7f766c1ff700 1 -- 172.21.15.27:0/2309727374 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76640b2b90 msgr2=0x7f76641589d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.178+0000 7f766c1ff700 1 --2- 172.21.15.27:0/2309727374 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76640b2b90 0x7f76641589d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f765c009f10 tx=0x7f765c00b040 comp rx=0 tx=0).stop 2024-01-11T08:10:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.178+0000 7f766c1ff700 1 -- 172.21.15.27:0/2309727374 shutdown_connections 2024-01-11T08:10:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.178+0000 7f766c1ff700 1 --2- 172.21.15.27:0/2309727374 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7648066f30 0x7f76480693e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.178+0000 7f766c1ff700 1 --2- 172.21.15.27:0/2309727374 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76640b2b90 0x7f76641589d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.178+0000 7f766c1ff700 1 --2- 172.21.15.27:0/2309727374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76640b03e0 0x7f766415ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.178+0000 7f766c1ff700 1 --2- 172.21.15.27:0/2309727374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76640b07b0 0x7f766415a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.178+0000 7f766c1ff700 1 -- 172.21.15.27:0/2309727374 >> 172.21.15.27:0/2309727374 conn(0x7f766406ed40 msgr2=0x7f76640b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.178+0000 7f766c1ff700 1 -- 172.21.15.27:0/2309727374 shutdown_connections 2024-01-11T08:10:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.178+0000 7f766c1ff700 1 -- 172.21.15.27:0/2309727374 wait complete. 2024-01-11T08:10:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 8 2024-01-11T08:10:34.197 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:10:34.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.395+0000 7fc7ab9ff700 1 Processor -- start 2024-01-11T08:10:34.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.395+0000 7fc7ab9ff700 1 -- start start 2024-01-11T08:10:34.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.396+0000 7fc7ab9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7a40b03e0 0x7fc7a40b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:34.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.396+0000 7fc7ab9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7a40b07b0 0x7fc7a40b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:34.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.396+0000 7fc7ab9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc7a4159200 0x7fc7a4159640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:34.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.396+0000 7fc7ab9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a40cd730 con 0x7fc7a40b07b0 2024-01-11T08:10:34.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.396+0000 7fc7ab9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a40ccb20 con 0x7fc7a4159200 2024-01-11T08:10:34.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.396+0000 7fc7ab9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a40cce00 con 0x7fc7a40b03e0 2024-01-11T08:10:34.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.397+0000 7fc7a8f9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7a40b07b0 0x7fc7a40b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:34.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.397+0000 7fc7a8f9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7a40b07b0 0x7fc7a40b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53078/0 (socket says 172.21.15.27:53078) 2024-01-11T08:10:34.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.397+0000 7fc7a8f9a700 1 -- 172.21.15.27:0/1318390014 learned_addr learned my addr 172.21.15.27:0/1318390014 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:34.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.397+0000 7fc7a979b700 1 --2- 172.21.15.27:0/1318390014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7a40b03e0 0x7fc7a40b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:34.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.397+0000 7fc7a9f9c700 1 --2- 172.21.15.27:0/1318390014 >> v2:172.21.15.77:3300/0 conn(0x7fc7a4159200 0x7fc7a4159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:34.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.397+0000 7fc7a8f9a700 1 -- 172.21.15.27:0/1318390014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7a40b03e0 msgr2=0x7fc7a40b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:34.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.397+0000 7fc7a8f9a700 1 --2- 172.21.15.27:0/1318390014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7a40b03e0 0x7fc7a40b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.397+0000 7fc7a8f9a700 1 -- 172.21.15.27:0/1318390014 >> v2:172.21.15.77:3300/0 conn(0x7fc7a4159200 msgr2=0x7fc7a4159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:34.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.397+0000 7fc7a8f9a700 1 --2- 172.21.15.27:0/1318390014 >> v2:172.21.15.77:3300/0 conn(0x7fc7a4159200 0x7fc7a4159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.397+0000 7fc7a8f9a700 1 -- 172.21.15.27:0/1318390014 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a40cd0e0 con 0x7fc7a40b07b0 2024-01-11T08:10:34.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.397+0000 7fc7a9f9c700 1 --2- 172.21.15.27:0/1318390014 >> v2:172.21.15.77:3300/0 conn(0x7fc7a4159200 0x7fc7a4159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:34.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.397+0000 7fc7a8f9a700 1 --2- 172.21.15.27:0/1318390014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a40b07b0 0x7fc7a40b29c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc7a40b5f00 tx=0x7fc794009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e55251ab6a2d4e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:34.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.398+0000 7fc79bfff700 1 -- 172.21.15.27:0/1318390014 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc794009e70 con 0x7fc7a40b07b0 2024-01-11T08:10:34.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.398+0000 7fc79bfff700 1 -- 172.21.15.27:0/1318390014 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79400b860 con 0x7fc7a40b07b0 2024-01-11T08:10:34.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.398+0000 7fc79bfff700 1 -- 172.21.15.27:0/1318390014 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79400bc00 con 0x7fc7a40b07b0 2024-01-11T08:10:34.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.398+0000 7fc7ab9ff700 1 -- 172.21.15.27:0/1318390014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a40b07b0 msgr2=0x7fc7a40b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:34.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.398+0000 7fc7ab9ff700 1 --2- 172.21.15.27:0/1318390014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a40b07b0 0x7fc7a40b29c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc7a40b5f00 tx=0x7fc794009250 comp rx=0 tx=0).stop 2024-01-11T08:10:34.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.398+0000 7fc7ab9ff700 1 -- 172.21.15.27:0/1318390014 shutdown_connections 2024-01-11T08:10:34.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.398+0000 7fc7ab9ff700 1 --2- 172.21.15.27:0/1318390014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7a40b03e0 0x7fc7a40b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.398+0000 7fc7ab9ff700 1 --2- 172.21.15.27:0/1318390014 >> v2:172.21.15.77:3300/0 conn(0x7fc7a4159200 0x7fc7a4159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.398+0000 7fc7ab9ff700 1 --2- 172.21.15.27:0/1318390014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a40b07b0 0x7fc7a40b29c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.398+0000 7fc7ab9ff700 1 -- 172.21.15.27:0/1318390014 >> 172.21.15.27:0/1318390014 conn(0x7fc7a406ed40 msgr2=0x7fc7a406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:34.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.398+0000 7fc7ab9ff700 1 -- 172.21.15.27:0/1318390014 shutdown_connections 2024-01-11T08:10:34.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.399+0000 7fc7ab9ff700 1 -- 172.21.15.27:0/1318390014 wait complete. 2024-01-11T08:10:34.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.400+0000 7fc7ab9ff700 1 Processor -- start 2024-01-11T08:10:34.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.400+0000 7fc7ab9ff700 1 -- start start 2024-01-11T08:10:34.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.401+0000 7fc7ab9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a40b03e0 0x7fc7a415a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:34.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.401+0000 7fc7ab9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7a40b07b0 0x7fc7a4157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:34.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.401+0000 7fc7ab9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7a4159200 0x7fc7a415bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:34.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.401+0000 7fc7ab9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a40c6330 con 0x7fc7a40b03e0 2024-01-11T08:10:34.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.401+0000 7fc7ab9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a40c6050 con 0x7fc7a40b07b0 2024-01-11T08:10:34.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.401+0000 7fc7ab9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a4159770 con 0x7fc7a4159200 2024-01-11T08:10:34.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.401+0000 7fc7a979b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a40b03e0 0x7fc7a415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:34.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.401+0000 7fc7a979b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a40b03e0 0x7fc7a415a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53088/0 (socket says 172.21.15.27:53088) 2024-01-11T08:10:34.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.401+0000 7fc7a979b700 1 -- 172.21.15.27:0/4237209737 learned_addr learned my addr 172.21.15.27:0/4237209737 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:34.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.402+0000 7fc7a979b700 1 -- 172.21.15.27:0/4237209737 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7a4159200 msgr2=0x7fc7a415bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:34.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.402+0000 7fc7a979b700 1 --2- 172.21.15.27:0/4237209737 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7a4159200 0x7fc7a415bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.402+0000 7fc7a979b700 1 -- 172.21.15.27:0/4237209737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7a40b07b0 msgr2=0x7fc7a4157bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:34.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.402+0000 7fc7a8f9a700 1 --2- 172.21.15.27:0/4237209737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7a40b07b0 0x7fc7a4157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:34.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.402+0000 7fc7a979b700 1 --2- 172.21.15.27:0/4237209737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7a40b07b0 0x7fc7a4157bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.402+0000 7fc7a979b700 1 -- 172.21.15.27:0/4237209737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7940094f0 con 0x7fc7a40b03e0 2024-01-11T08:10:34.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.402+0000 7fc7a8f9a700 1 --2- 172.21.15.27:0/4237209737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7a40b07b0 0x7fc7a4157bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:34.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.402+0000 7fc7a979b700 1 --2- 172.21.15.27:0/4237209737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a40b03e0 0x7fc7a415a030 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000e950 tx=0x7fc7a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:34.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.402+0000 7fc7997fa700 1 -- 172.21.15.27:0/4237209737 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a000c850 con 0x7fc7a40b03e0 2024-01-11T08:10:34.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.402+0000 7fc7997fa700 1 -- 172.21.15.27:0/4237209737 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a000c9b0 con 0x7fc7a40b03e0 2024-01-11T08:10:34.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.402+0000 7fc7997fa700 1 -- 172.21.15.27:0/4237209737 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a000cdb0 con 0x7fc7a40b03e0 2024-01-11T08:10:34.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.402+0000 7fc7ab9ff700 1 -- 172.21.15.27:0/4237209737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a4159a50 con 0x7fc7a40b03e0 2024-01-11T08:10:34.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.403+0000 7fc7ab9ff700 1 -- 172.21.15.27:0/4237209737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a415a570 con 0x7fc7a40b03e0 2024-01-11T08:10:34.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.404+0000 7fc7ab9ff700 1 -- 172.21.15.27:0/4237209737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a40718d0 con 0x7fc7a40b03e0 2024-01-11T08:10:34.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.404+0000 7fc7997fa700 1 -- 172.21.15.27:0/4237209737 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc7a0013680 con 0x7fc7a40b03e0 2024-01-11T08:10:34.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.405+0000 7fc7997fa700 1 --2- 172.21.15.27:0/4237209737 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc788066f30 0x7fc7880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:34.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.405+0000 7fc7997fa700 1 -- 172.21.15.27:0/4237209737 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fc7a0084860 con 0x7fc7a40b03e0 2024-01-11T08:10:34.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.406+0000 7fc7a8f9a700 1 --2- 172.21.15.27:0/4237209737 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc788066f30 0x7fc7880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:34.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.406+0000 7fc7a8f9a700 1 --2- 172.21.15.27:0/4237209737 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc788066f30 0x7fc7880693e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc7a40b5f00 tx=0x7fc794017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:34.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.409+0000 7fc7997fa700 1 -- 172.21.15.27:0/4237209737 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc7a0051a30 con 0x7fc7a40b03e0 2024-01-11T08:10:34.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.615+0000 7fc7ab9ff700 1 -- 172.21.15.27:0/4237209737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7a40cd3c0 con 0x7fc7a40b03e0 2024-01-11T08:10:34.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.616+0000 7fc7997fa700 1 -- 172.21.15.27:0/4237209737 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15302 (secure 0 0 0) 0x7fc7a0055050 con 0x7fc7a40b03e0 2024-01-11T08:10:34.617 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:34.617 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:31.464323+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:10:27.993251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-11T08:10:29.153959+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:10:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.618+0000 7fc7ab9ff700 1 -- 172.21.15.27:0/4237209737 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc788066f30 msgr2=0x7fc7880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.618+0000 7fc7ab9ff700 1 --2- 172.21.15.27:0/4237209737 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc788066f30 0x7fc7880693e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc7a40b5f00 tx=0x7fc794017000 comp rx=0 tx=0).stop 2024-01-11T08:10:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.618+0000 7fc7ab9ff700 1 -- 172.21.15.27:0/4237209737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a40b03e0 msgr2=0x7fc7a415a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.618+0000 7fc7ab9ff700 1 --2- 172.21.15.27:0/4237209737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a40b03e0 0x7fc7a415a030 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000e950 tx=0x7fc7a00091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.618+0000 7fc7ab9ff700 1 -- 172.21.15.27:0/4237209737 shutdown_connections 2024-01-11T08:10:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.618+0000 7fc7ab9ff700 1 --2- 172.21.15.27:0/4237209737 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc788066f30 0x7fc7880693e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.618+0000 7fc7ab9ff700 1 --2- 172.21.15.27:0/4237209737 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7a4159200 0x7fc7a415bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.618+0000 7fc7ab9ff700 1 --2- 172.21.15.27:0/4237209737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7a40b03e0 0x7fc7a415a030 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.618+0000 7fc7ab9ff700 1 --2- 172.21.15.27:0/4237209737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7a40b07b0 0x7fc7a4157bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.618+0000 7fc7ab9ff700 1 -- 172.21.15.27:0/4237209737 >> 172.21.15.27:0/4237209737 conn(0x7fc7a406ed40 msgr2=0x7fc7a4158a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.618+0000 7fc7ab9ff700 1 -- 172.21.15.27:0/4237209737 shutdown_connections 2024-01-11T08:10:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.618+0000 7fc7ab9ff700 1 -- 172.21.15.27:0/4237209737 wait complete. 2024-01-11T08:10:34.637 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-11T08:10:34.637 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:10:34.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.836+0000 7ff610dff700 1 Processor -- start 2024-01-11T08:10:34.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.837+0000 7ff610dff700 1 -- start start 2024-01-11T08:10:34.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.838+0000 7ff610dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff60c0b2b90 0x7ff60c159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:34.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.838+0000 7ff610dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff60c0b07b0 0x7ff60c159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:34.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.838+0000 7ff610dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff60c0cd730 con 0x7ff60c0b03e0 2024-01-11T08:10:34.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.838+0000 7ff610dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff60c0ccb20 con 0x7ff60c0b2b90 2024-01-11T08:10:34.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.838+0000 7ff610dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0cce00 con 0x7ff60c0b07b0 2024-01-11T08:10:34.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.838+0000 7ff609d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff60c0b2b90 0x7ff60c159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:34.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.838+0000 7ff609d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff60c0b2b90 0x7ff60c159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53092/0 (socket says 172.21.15.27:53092) 2024-01-11T08:10:34.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.838+0000 7ff609d9b700 1 -- 172.21.15.27:0/3309522040 learned_addr learned my addr 172.21.15.27:0/3309522040 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:34.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.838+0000 7ff60ad9d700 1 --2- 172.21.15.27:0/3309522040 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff60c0b07b0 0x7ff60c159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:34.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.839+0000 7ff609d9b700 1 -- 172.21.15.27:0/3309522040 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff60c0b07b0 msgr2=0x7ff60c159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:34.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.839+0000 7ff609d9b700 1 --2- 172.21.15.27:0/3309522040 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff60c0b07b0 0x7ff60c159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.839+0000 7ff609d9b700 1 -- 172.21.15.27:0/3309522040 >> v1:172.21.15.77:6789/0 conn(0x7ff60c0b03e0 legacy=0x7ff60c0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:34.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.839+0000 7ff609d9b700 1 -- 172.21.15.27:0/3309522040 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c0cd0e0 con 0x7ff60c0b2b90 2024-01-11T08:10:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.839+0000 7ff60ad9d700 1 --2- 172.21.15.27:0/3309522040 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff60c0b07b0 0x7ff60c159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.839+0000 7ff609d9b700 1 --2- 172.21.15.27:0/3309522040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff60c0b2b90 0x7ff60c159200 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff6000099b0 tx=0x7ff600009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5afb899a0420f870 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.839+0000 7ff60959a700 1 -- 172.21.15.27:0/3309522040 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60000b860 con 0x7ff60c0b2b90 2024-01-11T08:10:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.839+0000 7ff60959a700 1 -- 172.21.15.27:0/3309522040 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60000b9c0 con 0x7ff60c0b2b90 2024-01-11T08:10:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.839+0000 7ff60959a700 1 -- 172.21.15.27:0/3309522040 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60000bd60 con 0x7ff60c0b2b90 2024-01-11T08:10:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.840+0000 7ff610dff700 1 -- 172.21.15.27:0/3309522040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff60c0b2b90 msgr2=0x7ff60c159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.840+0000 7ff610dff700 1 --2- 172.21.15.27:0/3309522040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff60c0b2b90 0x7ff60c159200 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff6000099b0 tx=0x7ff600009250 comp rx=0 tx=0).stop 2024-01-11T08:10:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.840+0000 7ff610dff700 1 -- 172.21.15.27:0/3309522040 shutdown_connections 2024-01-11T08:10:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.840+0000 7ff610dff700 1 --2- 172.21.15.27:0/3309522040 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff60c0b07b0 0x7ff60c159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.840+0000 7ff610dff700 1 --2- 172.21.15.27:0/3309522040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff60c0b2b90 0x7ff60c159200 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.840+0000 7ff610dff700 1 -- 172.21.15.27:0/3309522040 >> 172.21.15.27:0/3309522040 conn(0x7ff60c06ed40 msgr2=0x7ff60c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.840+0000 7ff610dff700 1 -- 172.21.15.27:0/3309522040 shutdown_connections 2024-01-11T08:10:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.840+0000 7ff610dff700 1 -- 172.21.15.27:0/3309522040 wait complete. 2024-01-11T08:10:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.841+0000 7ff610dff700 1 Processor -- start 2024-01-11T08:10:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.841+0000 7ff610dff700 1 -- start start 2024-01-11T08:10:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.842+0000 7ff610dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff60c0b03e0 0x7ff60c14dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.842+0000 7ff610dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff60c0b07b0 0x7ff60c0c3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.842+0000 7ff610dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff60c0b2b90 0x7ff60c0c4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.842+0000 7ff610dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0c7ad0 con 0x7ff60c0b07b0 2024-01-11T08:10:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.842+0000 7ff610dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c14e100 con 0x7ff60c0b2b90 2024-01-11T08:10:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.842+0000 7ff610dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0c4880 con 0x7ff60c0b03e0 2024-01-11T08:10:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.843+0000 7ff609d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff60c0b07b0 0x7ff60c0c3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.843+0000 7ff609d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff60c0b07b0 0x7ff60c0c3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53104/0 (socket says 172.21.15.27:53104) 2024-01-11T08:10:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.843+0000 7ff609d9b700 1 -- 172.21.15.27:0/549924378 learned_addr learned my addr 172.21.15.27:0/549924378 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.843+0000 7ff60a59c700 1 --2- 172.21.15.27:0/549924378 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff60c0b03e0 0x7ff60c14dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.843+0000 7ff60ad9d700 1 --2- 172.21.15.27:0/549924378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff60c0b2b90 0x7ff60c0c4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.843+0000 7ff609d9b700 1 -- 172.21.15.27:0/549924378 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff60c0b03e0 msgr2=0x7ff60c14dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.843+0000 7ff609d9b700 1 --2- 172.21.15.27:0/549924378 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff60c0b03e0 0x7ff60c14dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.843+0000 7ff609d9b700 1 -- 172.21.15.27:0/549924378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff60c0b2b90 msgr2=0x7ff60c0c4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.843+0000 7ff609d9b700 1 --2- 172.21.15.27:0/549924378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff60c0b2b90 0x7ff60c0c4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.843+0000 7ff609d9b700 1 -- 172.21.15.27:0/549924378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6000094f0 con 0x7ff60c0b07b0 2024-01-11T08:10:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.843+0000 7ff609d9b700 1 --2- 172.21.15.27:0/549924378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff60c0b07b0 0x7ff60c0c3ea0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff6000090b0 tx=0x7ff600017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.844+0000 7ff5fa7fc700 1 -- 172.21.15.27:0/549924378 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600011dc0 con 0x7ff60c0b07b0 2024-01-11T08:10:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.844+0000 7ff5fa7fc700 1 -- 172.21.15.27:0/549924378 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60001f070 con 0x7ff60c0b07b0 2024-01-11T08:10:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.844+0000 7ff610dff700 1 -- 172.21.15.27:0/549924378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60c0c4b60 con 0x7ff60c0b07b0 2024-01-11T08:10:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.844+0000 7ff5fa7fc700 1 -- 172.21.15.27:0/549924378 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6000266d0 con 0x7ff60c0b07b0 2024-01-11T08:10:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.844+0000 7ff610dff700 1 -- 172.21.15.27:0/549924378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60c0c9380 con 0x7ff60c0b07b0 2024-01-11T08:10:34.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.845+0000 7ff610dff700 1 -- 172.21.15.27:0/549924378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff60c0c75c0 con 0x7ff60c0b07b0 2024-01-11T08:10:34.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.846+0000 7ff5fa7fc700 1 -- 172.21.15.27:0/549924378 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff600026a40 con 0x7ff60c0b07b0 2024-01-11T08:10:34.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.847+0000 7ff5fa7fc700 1 --2- 172.21.15.27:0/549924378 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff5ec066f30 0x7ff5ec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:34.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.847+0000 7ff5fa7fc700 1 -- 172.21.15.27:0/549924378 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7ff600021070 con 0x7ff60c0b07b0 2024-01-11T08:10:34.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.848+0000 7ff60a59c700 1 --2- 172.21.15.27:0/549924378 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff5ec066f30 0x7ff5ec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:34.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.848+0000 7ff60a59c700 1 --2- 172.21.15.27:0/549924378 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff5ec066f30 0x7ff5ec0693e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc003a80 tx=0x7ff5fc007ec0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:34.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:34.852+0000 7ff5fa7fc700 1 -- 172.21.15.27:0/549924378 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff60c0c75c0 con 0x7ff60c0b07b0 2024-01-11T08:10:35.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.052+0000 7ff610dff700 1 -- 172.21.15.27:0/549924378 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff60c0c75c0 con 0x7ff5ec066f30 2024-01-11T08:10:35.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.062+0000 7ff5fa7fc700 1 -- 172.21.15.27:0/549924378 <== mgr.4099 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+253965 (secure 0 0 0) 0x7ff60c0c75c0 con 0x7ff5ec066f30 2024-01-11T08:10:35.066 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:35.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.065+0000 7ff610dff700 1 -- 172.21.15.27:0/549924378 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff5ec066f30 msgr2=0x7ff5ec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:35.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.065+0000 7ff610dff700 1 --2- 172.21.15.27:0/549924378 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff5ec066f30 0x7ff5ec0693e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc003a80 tx=0x7ff5fc007ec0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.065+0000 7ff610dff700 1 -- 172.21.15.27:0/549924378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff60c0b07b0 msgr2=0x7ff60c0c3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:35.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.065+0000 7ff610dff700 1 --2- 172.21.15.27:0/549924378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff60c0b07b0 0x7ff60c0c3ea0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff6000090b0 tx=0x7ff600017000 comp rx=0 tx=0).stop 2024-01-11T08:10:35.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.065+0000 7ff610dff700 1 -- 172.21.15.27:0/549924378 shutdown_connections 2024-01-11T08:10:35.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.065+0000 7ff610dff700 1 --2- 172.21.15.27:0/549924378 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff5ec066f30 0x7ff5ec0693e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.065+0000 7ff610dff700 1 --2- 172.21.15.27:0/549924378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff60c0b2b90 0x7ff60c0c4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.065+0000 7ff610dff700 1 --2- 172.21.15.27:0/549924378 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff60c0b03e0 0x7ff60c14dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.065+0000 7ff610dff700 1 --2- 172.21.15.27:0/549924378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff60c0b07b0 0x7ff60c0c3ea0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.065+0000 7ff610dff700 1 -- 172.21.15.27:0/549924378 >> 172.21.15.27:0/549924378 conn(0x7ff60c06ed40 msgr2=0x7ff60c0b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:35.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.066+0000 7ff610dff700 1 -- 172.21.15.27:0/549924378 shutdown_connections 2024-01-11T08:10:35.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.066+0000 7ff610dff700 1 -- 172.21.15.27:0/549924378 wait complete. 2024-01-11T08:10:35.069 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:10:35.084 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-11T08:10:33.509350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381400,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752593320,"statfs":{"total":773094113280,"available":770655559680,"internally_reserved":0,"allocated":2531328,"data_stored":1582288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.865796"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.652703+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:25:36.497829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.647578+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:54.150785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.656896+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:50:14.474575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.655490+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:54.150785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.647695+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:25:36.497829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.653102+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:11:42.997404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.647372+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:11:42.997404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.649378+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:11:42.997404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.658221+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:46:57.523656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.650016+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:33.696922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.651649+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:50:14.474575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.657417+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:54.150785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.652601+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:46:33.761348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.650279+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:21.132838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.653028+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:25:36.497829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.655143+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:54.150785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.654433+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:25:36.497829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.650721+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:21.132838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.649708+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:54.150785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.653751+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:25:36.497829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.649301+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:25:36.497829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":160,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T08:10:23.931139+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:10:23.931139+0000","last_peered":"2024-01-11T08:10:23.931139+0000","last_clean":"2024-01-11T08:10:23.931139+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:10:23.931139+0000","last_undegraded":"2024-01-11T08:10:23.931139+0000","last_fullsized":"2024-01-11T08:10:23.931139+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.655612+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:21.132838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.654857+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:33.696922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.650930+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:33.696922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.655261+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:33.696922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.656173+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:46:57.523656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.649549+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:11:42.997404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.649838+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:54.150785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.654674+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:11:42.997404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.656298+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:21.132838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.649549+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:25:36.497829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:10:28.640227+0000","last_change":"2024-01-11T08:10:28.654363+0000","last_active":"2024-01-11T08:10:28.640227+0000","last_peered":"2024-01-11T08:10:28.640227+0000","last_clean":"2024-01-11T08:10:28.640227+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:28.640227+0000","last_undegraded":"2024-01-11T08:10:28.640227+0000","last_fullsized":"2024-01-11T08:10:28.640227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:11:42.997404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:10:29.642935+0000","last_change":"2024-01-11T08:10:29.642935+0000","last_active":"2024-01-11T08:10:29.642935+0000","last_peered":"2024-01-11T08:10:29.642935+0000","last_clean":"2024-01-11T08:10:29.642935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:10:29.642935+0000","last_undegraded":"2024-01-11T08:10:29.642935+0000","last_fullsized":"2024-01-11T08:10:29.642935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298032,"kb_used_data":744,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073808,"statfs":{"total":96636764160,"available":96331579392,"internally_reserved":0,"allocated":761856,"data_stored":640562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297452,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074388,"statfs":{"total":96636764160,"available":96332173312,"internally_reserved":0,"allocated":167936,"data_stored":50194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297452,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074388,"statfs":{"total":96636764160,"available":96332173312,"internally_reserved":0,"allocated":167936,"data_stored":50194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298656,"kb_used_data":744,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073184,"statfs":{"total":96636764160,"available":96330940416,"internally_reserved":0,"allocated":761856,"data_stored":640562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.163}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297452,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074388,"statfs":{"total":96636764160,"available":96332173312,"internally_reserved":0,"allocated":167936,"data_stored":50194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92800000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297452,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074388,"statfs":{"total":96636764160,"available":96332173312,"internally_reserved":0,"allocated":167936,"data_stored":50194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297452,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074388,"statfs":{"total":96636764160,"available":96332173312,"internally_reserved":0,"allocated":167936,"data_stored":50194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297452,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074388,"statfs":{"total":96636764160,"available":96332173312,"internally_reserved":0,"allocated":167936,"data_stored":50194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:10:35.089 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:10:35.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.323+0000 7f26f25ff700 1 Processor -- start 2024-01-11T08:10:35.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.323+0000 7f26f25ff700 1 -- start start 2024-01-11T08:10:35.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.324+0000 7f26f25ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26ec0ca3e0 0x7f26ec0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:35.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.324+0000 7f26f25ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f26ec0ca7b0 0x7f26ec159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:35.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.324+0000 7f26f25ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ec15d640 con 0x7f26ec0c8a20 2024-01-11T08:10:35.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.324+0000 7f26f25ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26ec15ae70 con 0x7f26ec0ca3e0 2024-01-11T08:10:35.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.324+0000 7f26f25ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26ec15a3c0 con 0x7f26ec0ca7b0 2024-01-11T08:10:35.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.324+0000 7f26ebfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26ec0ca3e0 0x7f26ec0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:35.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.325+0000 7f26ebfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26ec0ca3e0 0x7f26ec0cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53120/0 (socket says 172.21.15.27:53120) 2024-01-11T08:10:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.325+0000 7f26ebfff700 1 -- 172.21.15.27:0/2548177153 learned_addr learned my addr 172.21.15.27:0/2548177153 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.325+0000 7f26f0b9c700 1 --1- 172.21.15.27:0/2548177153 >> v1:172.21.15.27:6789/0 conn(0x7f26ec0c8a20 0x7f26ec15dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37102/0 (socket says 172.21.15.27:37102) 2024-01-11T08:10:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.325+0000 7f26eb7fe700 1 --2- 172.21.15.27:0/2548177153 >> v2:172.21.15.77:3300/0 conn(0x7f26ec0ca7b0 0x7f26ec159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.325+0000 7f26ebfff700 1 -- 172.21.15.27:0/2548177153 >> v2:172.21.15.77:3300/0 conn(0x7f26ec0ca7b0 msgr2=0x7f26ec159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.325+0000 7f26ebfff700 1 --2- 172.21.15.27:0/2548177153 >> v2:172.21.15.77:3300/0 conn(0x7f26ec0ca7b0 0x7f26ec159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.325+0000 7f26ebfff700 1 -- 172.21.15.27:0/2548177153 >> v1:172.21.15.27:6789/0 conn(0x7f26ec0c8a20 legacy=0x7f26ec15dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.325+0000 7f26ebfff700 1 -- 172.21.15.27:0/2548177153 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec15a5d0 con 0x7f26ec0ca3e0 2024-01-11T08:10:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.325+0000 7f26eaffd700 1 -- 172.21.15.27:0/2548177153 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430226591 0 0) 0x7f26ec15d640 con 0x7f26ec0c8a20 2024-01-11T08:10:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.325+0000 7f26ebfff700 1 --2- 172.21.15.27:0/2548177153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26ec0ca3e0 0x7f26ec0cca20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0099b0 tx=0x7f26dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e9fc5cefc6a78ad server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.325+0000 7f26eaffd700 1 -- 172.21.15.27:0/2548177153 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc009ea0 con 0x7f26ec0ca3e0 2024-01-11T08:10:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.326+0000 7f26eaffd700 1 -- 172.21.15.27:0/2548177153 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26dc00b820 con 0x7f26ec0ca3e0 2024-01-11T08:10:35.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.326+0000 7f26eaffd700 1 -- 172.21.15.27:0/2548177153 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc00bbc0 con 0x7f26ec0ca3e0 2024-01-11T08:10:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.326+0000 7f26f25ff700 1 -- 172.21.15.27:0/2548177153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26ec0ca3e0 msgr2=0x7f26ec0cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.326+0000 7f26f25ff700 1 --2- 172.21.15.27:0/2548177153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26ec0ca3e0 0x7f26ec0cca20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0099b0 tx=0x7f26dc009250 comp rx=0 tx=0).stop 2024-01-11T08:10:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.326+0000 7f26f25ff700 1 -- 172.21.15.27:0/2548177153 shutdown_connections 2024-01-11T08:10:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.326+0000 7f26f25ff700 1 --2- 172.21.15.27:0/2548177153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26ec0ca3e0 0x7f26ec0cca20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.326+0000 7f26f25ff700 1 --2- 172.21.15.27:0/2548177153 >> v2:172.21.15.77:3300/0 conn(0x7f26ec0ca7b0 0x7f26ec159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.326+0000 7f26f25ff700 1 -- 172.21.15.27:0/2548177153 >> 172.21.15.27:0/2548177153 conn(0x7f26ec0ce3e0 msgr2=0x7f26ec0ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.326+0000 7f26f25ff700 1 -- 172.21.15.27:0/2548177153 shutdown_connections 2024-01-11T08:10:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.326+0000 7f26f25ff700 1 -- 172.21.15.27:0/2548177153 wait complete. 2024-01-11T08:10:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.328+0000 7f26f25ff700 1 Processor -- start 2024-01-11T08:10:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.328+0000 7f26f25ff700 1 -- start start 2024-01-11T08:10:35.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.329+0000 7f26f25ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26ec0c8a20 0x7f26ec1698b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:35.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.329+0000 7f26f25ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26ec0ca3e0 0x7f26ec0b2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:35.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.329+0000 7f26f25ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26ec0ca7b0 0x7f26ec151c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:35.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.329+0000 7f26f25ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec169df0 con 0x7f26ec0ca3e0 2024-01-11T08:10:35.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.329+0000 7f26f25ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec148800 con 0x7f26ec0c8a20 2024-01-11T08:10:35.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.329+0000 7f26f25ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec0b2a60 con 0x7f26ec0ca7b0 2024-01-11T08:10:35.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.329+0000 7f26eb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26ec0ca3e0 0x7f26ec0b2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:35.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.329+0000 7f26eb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26ec0ca3e0 0x7f26ec0b2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53122/0 (socket says 172.21.15.27:53122) 2024-01-11T08:10:35.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.329+0000 7f26eb7fe700 1 -- 172.21.15.27:0/935812204 learned_addr learned my addr 172.21.15.27:0/935812204 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:35.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.329+0000 7f26eb7fe700 1 -- 172.21.15.27:0/935812204 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26ec0ca7b0 msgr2=0x7f26ec151c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:35.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.329+0000 7f26eb7fe700 1 --2- 172.21.15.27:0/935812204 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26ec0ca7b0 0x7f26ec151c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.329+0000 7f26eb7fe700 1 -- 172.21.15.27:0/935812204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26ec0c8a20 msgr2=0x7f26ec1698b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:35.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.329+0000 7f26eb7fe700 1 --2- 172.21.15.27:0/935812204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26ec0c8a20 0x7f26ec1698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.329+0000 7f26eb7fe700 1 -- 172.21.15.27:0/935812204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc0094f0 con 0x7f26ec0ca3e0 2024-01-11T08:10:35.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.330+0000 7f26eb7fe700 1 --2- 172.21.15.27:0/935812204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26ec0ca3e0 0x7f26ec0b2520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f26e000e930 tx=0x7f26e00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:35.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.330+0000 7f26d3fff700 1 -- 172.21.15.27:0/935812204 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e000eed0 con 0x7f26ec0ca3e0 2024-01-11T08:10:35.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.330+0000 7f26d3fff700 1 -- 172.21.15.27:0/935812204 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e000c850 con 0x7f26ec0ca3e0 2024-01-11T08:10:35.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.330+0000 7f26d3fff700 1 -- 172.21.15.27:0/935812204 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e000cbf0 con 0x7f26ec0ca3e0 2024-01-11T08:10:35.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.330+0000 7f26f25ff700 1 -- 172.21.15.27:0/935812204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26ec148e20 con 0x7f26ec0ca3e0 2024-01-11T08:10:35.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.330+0000 7f26f25ff700 1 -- 172.21.15.27:0/935812204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26ec0b2f50 con 0x7f26ec0ca3e0 2024-01-11T08:10:35.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.331+0000 7f26f25ff700 1 -- 172.21.15.27:0/935812204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26ec0718d0 con 0x7f26ec0ca3e0 2024-01-11T08:10:35.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.332+0000 7f26d3fff700 1 -- 172.21.15.27:0/935812204 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f26e0013470 con 0x7f26ec0ca3e0 2024-01-11T08:10:35.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.333+0000 7f26d3fff700 1 --2- 172.21.15.27:0/935812204 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f26d8066f30 0x7f26d80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:35.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.333+0000 7f26d3fff700 1 -- 172.21.15.27:0/935812204 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f26e00847b0 con 0x7f26ec0ca3e0 2024-01-11T08:10:35.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.333+0000 7f26ebfff700 1 --2- 172.21.15.27:0/935812204 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f26d8066f30 0x7f26d80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:35.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.334+0000 7f26ebfff700 1 --2- 172.21.15.27:0/935812204 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f26d8066f30 0x7f26d80693e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0099b0 tx=0x7f26dc017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:35.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.337+0000 7f26d3fff700 1 -- 172.21.15.27:0/935812204 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f26e0051980 con 0x7f26ec0ca3e0 2024-01-11T08:10:35.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.535+0000 7f26f25ff700 1 -- 172.21.15.27:0/935812204 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f26ec06c840 con 0x7f26d8066f30 2024-01-11T08:10:35.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.544+0000 7f26d3fff700 1 -- 172.21.15.27:0/935812204 <== mgr.4099 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264634 (secure 0 0 0) 0x7f26ec06c840 con 0x7f26d8066f30 2024-01-11T08:10:35.549 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:35.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.548+0000 7f26f25ff700 1 -- 172.21.15.27:0/935812204 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f26d8066f30 msgr2=0x7f26d80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:35.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.548+0000 7f26f25ff700 1 --2- 172.21.15.27:0/935812204 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f26d8066f30 0x7f26d80693e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0099b0 tx=0x7f26dc017000 comp rx=0 tx=0).stop 2024-01-11T08:10:35.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.548+0000 7f26f25ff700 1 -- 172.21.15.27:0/935812204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26ec0ca3e0 msgr2=0x7f26ec0b2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:35.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.548+0000 7f26f25ff700 1 --2- 172.21.15.27:0/935812204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26ec0ca3e0 0x7f26ec0b2520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f26e000e930 tx=0x7f26e00091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.548+0000 7f26f25ff700 1 -- 172.21.15.27:0/935812204 shutdown_connections 2024-01-11T08:10:35.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.548+0000 7f26f25ff700 1 --2- 172.21.15.27:0/935812204 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f26d8066f30 0x7f26d80693e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.548+0000 7f26f25ff700 1 --2- 172.21.15.27:0/935812204 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26ec0ca7b0 0x7f26ec151c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.548+0000 7f26f25ff700 1 --2- 172.21.15.27:0/935812204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26ec0c8a20 0x7f26ec1698b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.548+0000 7f26f25ff700 1 --2- 172.21.15.27:0/935812204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26ec0ca3e0 0x7f26ec0b2520 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.548+0000 7f26f25ff700 1 -- 172.21.15.27:0/935812204 >> 172.21.15.27:0/935812204 conn(0x7f26ec0ce3e0 msgr2=0x7f26ec068780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:35.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.548+0000 7f26f25ff700 1 -- 172.21.15.27:0/935812204 shutdown_connections 2024-01-11T08:10:35.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.548+0000 7f26f25ff700 1 -- 172.21.15.27:0/935812204 wait complete. 2024-01-11T08:10:35.552 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:10:35.567 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-11T08:10:35.509736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381224,"kb_used_data":2920,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752593496,"statfs":{"total":773094113280,"available":770655739904,"internally_reserved":0,"allocated":2990080,"data_stored":1787852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001325"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.498429+0000","last_change":"2024-01-11T08:10:30.495770+0000","last_active":"2024-01-11T08:10:30.498429+0000","last_peered":"2024-01-11T08:10:30.498429+0000","last_clean":"2024-01-11T08:10:30.498429+0000","last_became_active":"2024-01-11T08:10:30.495355+0000","last_became_peered":"2024-01-11T08:10:30.495355+0000","last_unstale":"2024-01-11T08:10:30.498429+0000","last_undegraded":"2024-01-11T08:10:30.498429+0000","last_fullsized":"2024-01-11T08:10:30.498429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:02.887025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.493541+0000","last_change":"2024-01-11T08:10:30.492484+0000","last_active":"2024-01-11T08:10:30.493541+0000","last_peered":"2024-01-11T08:10:30.493541+0000","last_clean":"2024-01-11T08:10:30.493541+0000","last_became_active":"2024-01-11T08:10:30.491796+0000","last_became_peered":"2024-01-11T08:10:30.491796+0000","last_unstale":"2024-01-11T08:10:30.493541+0000","last_undegraded":"2024-01-11T08:10:30.493541+0000","last_fullsized":"2024-01-11T08:10:30.493541+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:21.127779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.493213+0000","last_change":"2024-01-11T08:10:30.489955+0000","last_active":"2024-01-11T08:10:30.493213+0000","last_peered":"2024-01-11T08:10:30.493213+0000","last_clean":"2024-01-11T08:10:30.493213+0000","last_became_active":"2024-01-11T08:10:30.489649+0000","last_became_peered":"2024-01-11T08:10:30.489649+0000","last_unstale":"2024-01-11T08:10:30.493213+0000","last_undegraded":"2024-01-11T08:10:30.493213+0000","last_fullsized":"2024-01-11T08:10:30.493213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:08:10.140431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.495948+0000","last_change":"2024-01-11T08:10:30.489768+0000","last_active":"2024-01-11T08:10:30.495948+0000","last_peered":"2024-01-11T08:10:30.495948+0000","last_clean":"2024-01-11T08:10:30.495948+0000","last_became_active":"2024-01-11T08:10:30.489392+0000","last_became_peered":"2024-01-11T08:10:30.489392+0000","last_unstale":"2024-01-11T08:10:30.495948+0000","last_undegraded":"2024-01-11T08:10:30.495948+0000","last_fullsized":"2024-01-11T08:10:30.495948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:08:10.140431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.494482+0000","last_change":"2024-01-11T08:10:30.490346+0000","last_active":"2024-01-11T08:10:30.494482+0000","last_peered":"2024-01-11T08:10:30.494482+0000","last_clean":"2024-01-11T08:10:30.494482+0000","last_became_active":"2024-01-11T08:10:30.490042+0000","last_became_peered":"2024-01-11T08:10:30.490042+0000","last_unstale":"2024-01-11T08:10:30.494482+0000","last_undegraded":"2024-01-11T08:10:30.494482+0000","last_fullsized":"2024-01-11T08:10:30.494482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:57:19.989587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.492082+0000","last_change":"2024-01-11T08:10:30.487385+0000","last_active":"2024-01-11T08:10:30.492082+0000","last_peered":"2024-01-11T08:10:30.492082+0000","last_clean":"2024-01-11T08:10:30.492082+0000","last_became_active":"2024-01-11T08:10:30.487027+0000","last_became_peered":"2024-01-11T08:10:30.487027+0000","last_unstale":"2024-01-11T08:10:30.492082+0000","last_undegraded":"2024-01-11T08:10:30.492082+0000","last_fullsized":"2024-01-11T08:10:30.492082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:12.649092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.498136+0000","last_change":"2024-01-11T08:10:30.494485+0000","last_active":"2024-01-11T08:10:30.498136+0000","last_peered":"2024-01-11T08:10:30.498136+0000","last_clean":"2024-01-11T08:10:30.498136+0000","last_became_active":"2024-01-11T08:10:30.494227+0000","last_became_peered":"2024-01-11T08:10:30.494227+0000","last_unstale":"2024-01-11T08:10:30.498136+0000","last_undegraded":"2024-01-11T08:10:30.498136+0000","last_fullsized":"2024-01-11T08:10:30.498136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:31:36.455594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.484735+0000","last_change":"2024-01-11T08:10:30.477637+0000","last_active":"2024-01-11T08:10:30.484735+0000","last_peered":"2024-01-11T08:10:30.484735+0000","last_clean":"2024-01-11T08:10:30.484735+0000","last_became_active":"2024-01-11T08:10:30.477267+0000","last_became_peered":"2024-01-11T08:10:30.477267+0000","last_unstale":"2024-01-11T08:10:30.484735+0000","last_undegraded":"2024-01-11T08:10:30.484735+0000","last_fullsized":"2024-01-11T08:10:30.484735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:47.753678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.496708+0000","last_change":"2024-01-11T08:10:30.494031+0000","last_active":"2024-01-11T08:10:30.496708+0000","last_peered":"2024-01-11T08:10:30.496708+0000","last_clean":"2024-01-11T08:10:30.496708+0000","last_became_active":"2024-01-11T08:10:30.493769+0000","last_became_peered":"2024-01-11T08:10:30.493769+0000","last_unstale":"2024-01-11T08:10:30.496708+0000","last_undegraded":"2024-01-11T08:10:30.496708+0000","last_fullsized":"2024-01-11T08:10:30.496708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:14:36.595264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.502209+0000","last_change":"2024-01-11T08:10:30.498360+0000","last_active":"2024-01-11T08:10:30.502209+0000","last_peered":"2024-01-11T08:10:30.502209+0000","last_clean":"2024-01-11T08:10:30.502209+0000","last_became_active":"2024-01-11T08:10:30.498064+0000","last_became_peered":"2024-01-11T08:10:30.498064+0000","last_unstale":"2024-01-11T08:10:30.502209+0000","last_undegraded":"2024-01-11T08:10:30.502209+0000","last_fullsized":"2024-01-11T08:10:30.502209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:08:10.140431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.501156+0000","last_change":"2024-01-11T08:10:30.493099+0000","last_active":"2024-01-11T08:10:30.501156+0000","last_peered":"2024-01-11T08:10:30.501156+0000","last_clean":"2024-01-11T08:10:30.501156+0000","last_became_active":"2024-01-11T08:10:30.492806+0000","last_became_peered":"2024-01-11T08:10:30.492806+0000","last_unstale":"2024-01-11T08:10:30.501156+0000","last_undegraded":"2024-01-11T08:10:30.501156+0000","last_fullsized":"2024-01-11T08:10:30.501156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:21.127779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.503398+0000","last_change":"2024-01-11T08:10:30.499317+0000","last_active":"2024-01-11T08:10:30.503398+0000","last_peered":"2024-01-11T08:10:30.503398+0000","last_clean":"2024-01-11T08:10:30.503398+0000","last_became_active":"2024-01-11T08:10:30.497953+0000","last_became_peered":"2024-01-11T08:10:30.497953+0000","last_unstale":"2024-01-11T08:10:30.503398+0000","last_undegraded":"2024-01-11T08:10:30.503398+0000","last_fullsized":"2024-01-11T08:10:30.503398+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:27:24.758750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.490479+0000","last_change":"2024-01-11T08:10:30.488701+0000","last_active":"2024-01-11T08:10:30.490479+0000","last_peered":"2024-01-11T08:10:30.490479+0000","last_clean":"2024-01-11T08:10:30.490479+0000","last_became_active":"2024-01-11T08:10:30.488244+0000","last_became_peered":"2024-01-11T08:10:30.488244+0000","last_unstale":"2024-01-11T08:10:30.490479+0000","last_undegraded":"2024-01-11T08:10:30.490479+0000","last_fullsized":"2024-01-11T08:10:30.490479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:21.127779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.491292+0000","last_change":"2024-01-11T08:10:30.486156+0000","last_active":"2024-01-11T08:10:30.491292+0000","last_peered":"2024-01-11T08:10:30.491292+0000","last_clean":"2024-01-11T08:10:30.491292+0000","last_became_active":"2024-01-11T08:10:30.485871+0000","last_became_peered":"2024-01-11T08:10:30.485871+0000","last_unstale":"2024-01-11T08:10:30.491292+0000","last_undegraded":"2024-01-11T08:10:30.491292+0000","last_fullsized":"2024-01-11T08:10:30.491292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:22:43.961595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.487083+0000","last_change":"2024-01-11T08:10:30.484060+0000","last_active":"2024-01-11T08:10:30.487083+0000","last_peered":"2024-01-11T08:10:30.487083+0000","last_clean":"2024-01-11T08:10:30.487083+0000","last_became_active":"2024-01-11T08:10:30.483788+0000","last_became_peered":"2024-01-11T08:10:30.483788+0000","last_unstale":"2024-01-11T08:10:30.487083+0000","last_undegraded":"2024-01-11T08:10:30.487083+0000","last_fullsized":"2024-01-11T08:10:30.487083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:08:10.140431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.494360+0000","last_change":"2024-01-11T08:10:30.487817+0000","last_active":"2024-01-11T08:10:30.494360+0000","last_peered":"2024-01-11T08:10:30.494360+0000","last_clean":"2024-01-11T08:10:30.494360+0000","last_became_active":"2024-01-11T08:10:30.486728+0000","last_became_peered":"2024-01-11T08:10:30.486728+0000","last_unstale":"2024-01-11T08:10:30.494360+0000","last_undegraded":"2024-01-11T08:10:30.494360+0000","last_fullsized":"2024-01-11T08:10:30.494360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:33:25.110688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.494728+0000","last_change":"2024-01-11T08:10:30.490019+0000","last_active":"2024-01-11T08:10:30.494728+0000","last_peered":"2024-01-11T08:10:30.494728+0000","last_clean":"2024-01-11T08:10:30.494728+0000","last_became_active":"2024-01-11T08:10:30.489527+0000","last_became_peered":"2024-01-11T08:10:30.489527+0000","last_unstale":"2024-01-11T08:10:30.494728+0000","last_undegraded":"2024-01-11T08:10:30.494728+0000","last_fullsized":"2024-01-11T08:10:30.494728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:36:17.244182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.489961+0000","last_change":"2024-01-11T08:10:30.488678+0000","last_active":"2024-01-11T08:10:30.489961+0000","last_peered":"2024-01-11T08:10:30.489961+0000","last_clean":"2024-01-11T08:10:30.489961+0000","last_became_active":"2024-01-11T08:10:30.487511+0000","last_became_peered":"2024-01-11T08:10:30.487511+0000","last_unstale":"2024-01-11T08:10:30.489961+0000","last_undegraded":"2024-01-11T08:10:30.489961+0000","last_fullsized":"2024-01-11T08:10:30.489961+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:56:13.553335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.494305+0000","last_change":"2024-01-11T08:10:30.489544+0000","last_active":"2024-01-11T08:10:30.494305+0000","last_peered":"2024-01-11T08:10:30.494305+0000","last_clean":"2024-01-11T08:10:30.494305+0000","last_became_active":"2024-01-11T08:10:30.489020+0000","last_became_peered":"2024-01-11T08:10:30.489020+0000","last_unstale":"2024-01-11T08:10:30.494305+0000","last_undegraded":"2024-01-11T08:10:30.494305+0000","last_fullsized":"2024-01-11T08:10:30.494305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:33:25.110688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.680471+0000","last_change":"2024-01-11T08:10:29.680721+0000","last_active":"2024-01-11T08:10:30.680471+0000","last_peered":"2024-01-11T08:10:30.680471+0000","last_clean":"2024-01-11T08:10:30.680471+0000","last_became_active":"2024-01-11T08:10:29.680457+0000","last_became_peered":"2024-01-11T08:10:29.680457+0000","last_unstale":"2024-01-11T08:10:30.680471+0000","last_undegraded":"2024-01-11T08:10:30.680471+0000","last_fullsized":"2024-01-11T08:10:30.680471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:54.150785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.496156+0000","last_change":"2024-01-11T08:10:30.494749+0000","last_active":"2024-01-11T08:10:30.496156+0000","last_peered":"2024-01-11T08:10:30.496156+0000","last_clean":"2024-01-11T08:10:30.496156+0000","last_became_active":"2024-01-11T08:10:30.494471+0000","last_became_peered":"2024-01-11T08:10:30.494471+0000","last_unstale":"2024-01-11T08:10:30.496156+0000","last_undegraded":"2024-01-11T08:10:30.496156+0000","last_fullsized":"2024-01-11T08:10:30.496156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:50:04.791490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.684107+0000","last_change":"2024-01-11T08:10:29.665411+0000","last_active":"2024-01-11T08:10:30.684107+0000","last_peered":"2024-01-11T08:10:30.684107+0000","last_clean":"2024-01-11T08:10:30.684107+0000","last_became_active":"2024-01-11T08:10:29.665100+0000","last_became_peered":"2024-01-11T08:10:29.665100+0000","last_unstale":"2024-01-11T08:10:30.684107+0000","last_undegraded":"2024-01-11T08:10:30.684107+0000","last_fullsized":"2024-01-11T08:10:30.684107+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:33.696922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.493355+0000","last_change":"2024-01-11T08:10:30.490972+0000","last_active":"2024-01-11T08:10:30.493355+0000","last_peered":"2024-01-11T08:10:30.493355+0000","last_clean":"2024-01-11T08:10:30.493355+0000","last_became_active":"2024-01-11T08:10:30.490679+0000","last_became_peered":"2024-01-11T08:10:30.490679+0000","last_unstale":"2024-01-11T08:10:30.493355+0000","last_undegraded":"2024-01-11T08:10:30.493355+0000","last_fullsized":"2024-01-11T08:10:30.493355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:31:43.759712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.683204+0000","last_change":"2024-01-11T08:10:29.682048+0000","last_active":"2024-01-11T08:10:30.683204+0000","last_peered":"2024-01-11T08:10:30.683204+0000","last_clean":"2024-01-11T08:10:30.683204+0000","last_became_active":"2024-01-11T08:10:29.678360+0000","last_became_peered":"2024-01-11T08:10:29.678360+0000","last_unstale":"2024-01-11T08:10:30.683204+0000","last_undegraded":"2024-01-11T08:10:30.683204+0000","last_fullsized":"2024-01-11T08:10:30.683204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:56:12.550627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.481573+0000","last_change":"2024-01-11T08:10:30.477253+0000","last_active":"2024-01-11T08:10:30.481573+0000","last_peered":"2024-01-11T08:10:30.481573+0000","last_clean":"2024-01-11T08:10:30.481573+0000","last_became_active":"2024-01-11T08:10:30.476942+0000","last_became_peered":"2024-01-11T08:10:30.476942+0000","last_unstale":"2024-01-11T08:10:30.481573+0000","last_undegraded":"2024-01-11T08:10:30.481573+0000","last_fullsized":"2024-01-11T08:10:30.481573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:47:17.705767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.680175+0000","last_change":"2024-01-11T08:10:29.682657+0000","last_active":"2024-01-11T08:10:30.680175+0000","last_peered":"2024-01-11T08:10:30.680175+0000","last_clean":"2024-01-11T08:10:30.680175+0000","last_became_active":"2024-01-11T08:10:29.680246+0000","last_became_peered":"2024-01-11T08:10:29.680246+0000","last_unstale":"2024-01-11T08:10:30.680175+0000","last_undegraded":"2024-01-11T08:10:30.680175+0000","last_fullsized":"2024-01-11T08:10:30.680175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:48.964847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.480068+0000","last_change":"2024-01-11T08:10:30.476899+0000","last_active":"2024-01-11T08:10:30.480068+0000","last_peered":"2024-01-11T08:10:30.480068+0000","last_clean":"2024-01-11T08:10:30.480068+0000","last_became_active":"2024-01-11T08:10:30.475555+0000","last_became_peered":"2024-01-11T08:10:30.475555+0000","last_unstale":"2024-01-11T08:10:30.480068+0000","last_undegraded":"2024-01-11T08:10:30.480068+0000","last_fullsized":"2024-01-11T08:10:30.480068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:51:50.621267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.683162+0000","last_change":"2024-01-11T08:10:29.670617+0000","last_active":"2024-01-11T08:10:30.683162+0000","last_peered":"2024-01-11T08:10:30.683162+0000","last_clean":"2024-01-11T08:10:30.683162+0000","last_became_active":"2024-01-11T08:10:29.670316+0000","last_became_peered":"2024-01-11T08:10:29.670316+0000","last_unstale":"2024-01-11T08:10:30.683162+0000","last_undegraded":"2024-01-11T08:10:30.683162+0000","last_fullsized":"2024-01-11T08:10:30.683162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:50:14.474575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.486483+0000","last_change":"2024-01-11T08:10:30.477966+0000","last_active":"2024-01-11T08:10:30.486483+0000","last_peered":"2024-01-11T08:10:30.486483+0000","last_clean":"2024-01-11T08:10:30.486483+0000","last_became_active":"2024-01-11T08:10:30.477699+0000","last_became_peered":"2024-01-11T08:10:30.477699+0000","last_unstale":"2024-01-11T08:10:30.486483+0000","last_undegraded":"2024-01-11T08:10:30.486483+0000","last_fullsized":"2024-01-11T08:10:30.486483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:06.679100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.680666+0000","last_change":"2024-01-11T08:10:29.669295+0000","last_active":"2024-01-11T08:10:30.680666+0000","last_peered":"2024-01-11T08:10:30.680666+0000","last_clean":"2024-01-11T08:10:30.680666+0000","last_became_active":"2024-01-11T08:10:29.668947+0000","last_became_peered":"2024-01-11T08:10:29.668947+0000","last_unstale":"2024-01-11T08:10:30.680666+0000","last_undegraded":"2024-01-11T08:10:30.680666+0000","last_fullsized":"2024-01-11T08:10:30.680666+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:50:14.474575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.496590+0000","last_change":"2024-01-11T08:10:30.494173+0000","last_active":"2024-01-11T08:10:30.496590+0000","last_peered":"2024-01-11T08:10:30.496590+0000","last_clean":"2024-01-11T08:10:30.496590+0000","last_became_active":"2024-01-11T08:10:30.493875+0000","last_became_peered":"2024-01-11T08:10:30.493875+0000","last_unstale":"2024-01-11T08:10:30.496590+0000","last_undegraded":"2024-01-11T08:10:30.496590+0000","last_fullsized":"2024-01-11T08:10:30.496590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:47.753678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.466964+0000","last_change":"2024-01-11T08:10:29.673258+0000","last_active":"2024-01-11T08:10:30.466964+0000","last_peered":"2024-01-11T08:10:30.466964+0000","last_clean":"2024-01-11T08:10:30.466964+0000","last_became_active":"2024-01-11T08:10:29.672939+0000","last_became_peered":"2024-01-11T08:10:29.672939+0000","last_unstale":"2024-01-11T08:10:30.466964+0000","last_undegraded":"2024-01-11T08:10:30.466964+0000","last_fullsized":"2024-01-11T08:10:30.466964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:21.132838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.487736+0000","last_change":"2024-01-11T08:10:30.486579+0000","last_active":"2024-01-11T08:10:30.487736+0000","last_peered":"2024-01-11T08:10:30.487736+0000","last_clean":"2024-01-11T08:10:30.487736+0000","last_became_active":"2024-01-11T08:10:30.486332+0000","last_became_peered":"2024-01-11T08:10:30.486332+0000","last_unstale":"2024-01-11T08:10:30.487736+0000","last_undegraded":"2024-01-11T08:10:30.487736+0000","last_fullsized":"2024-01-11T08:10:30.487736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:03:43.208019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.471082+0000","last_change":"2024-01-11T08:10:29.676952+0000","last_active":"2024-01-11T08:10:30.471082+0000","last_peered":"2024-01-11T08:10:30.471082+0000","last_clean":"2024-01-11T08:10:30.471082+0000","last_became_active":"2024-01-11T08:10:29.676624+0000","last_became_peered":"2024-01-11T08:10:29.676624+0000","last_unstale":"2024-01-11T08:10:30.471082+0000","last_undegraded":"2024-01-11T08:10:30.471082+0000","last_fullsized":"2024-01-11T08:10:30.471082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:05.676391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.487063+0000","last_change":"2024-01-11T08:10:30.481939+0000","last_active":"2024-01-11T08:10:30.487063+0000","last_peered":"2024-01-11T08:10:30.487063+0000","last_clean":"2024-01-11T08:10:30.487063+0000","last_became_active":"2024-01-11T08:10:30.481619+0000","last_became_peered":"2024-01-11T08:10:30.481619+0000","last_unstale":"2024-01-11T08:10:30.487063+0000","last_undegraded":"2024-01-11T08:10:30.487063+0000","last_fullsized":"2024-01-11T08:10:30.487063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:03:43.208019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.485859+0000","last_change":"2024-01-11T08:10:29.679415+0000","last_active":"2024-01-11T08:10:30.485859+0000","last_peered":"2024-01-11T08:10:30.485859+0000","last_clean":"2024-01-11T08:10:30.485859+0000","last_became_active":"2024-01-11T08:10:29.679087+0000","last_became_peered":"2024-01-11T08:10:29.679087+0000","last_unstale":"2024-01-11T08:10:30.485859+0000","last_undegraded":"2024-01-11T08:10:30.485859+0000","last_fullsized":"2024-01-11T08:10:30.485859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:05.676391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.497485+0000","last_change":"2024-01-11T08:10:30.495369+0000","last_active":"2024-01-11T08:10:30.497485+0000","last_peered":"2024-01-11T08:10:30.497485+0000","last_clean":"2024-01-11T08:10:30.497485+0000","last_became_active":"2024-01-11T08:10:30.495037+0000","last_became_peered":"2024-01-11T08:10:30.495037+0000","last_unstale":"2024-01-11T08:10:30.497485+0000","last_undegraded":"2024-01-11T08:10:30.497485+0000","last_fullsized":"2024-01-11T08:10:30.497485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:21:29.220934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.472907+0000","last_change":"2024-01-11T08:10:29.681233+0000","last_active":"2024-01-11T08:10:30.472907+0000","last_peered":"2024-01-11T08:10:30.472907+0000","last_clean":"2024-01-11T08:10:30.472907+0000","last_became_active":"2024-01-11T08:10:29.680606+0000","last_became_peered":"2024-01-11T08:10:29.680606+0000","last_unstale":"2024-01-11T08:10:30.472907+0000","last_undegraded":"2024-01-11T08:10:30.472907+0000","last_fullsized":"2024-01-11T08:10:30.472907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:12:48.941888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.491516+0000","last_change":"2024-01-11T08:10:30.487651+0000","last_active":"2024-01-11T08:10:30.491516+0000","last_peered":"2024-01-11T08:10:30.491516+0000","last_clean":"2024-01-11T08:10:30.491516+0000","last_became_active":"2024-01-11T08:10:30.486641+0000","last_became_peered":"2024-01-11T08:10:30.486641+0000","last_unstale":"2024-01-11T08:10:30.491516+0000","last_undegraded":"2024-01-11T08:10:30.491516+0000","last_fullsized":"2024-01-11T08:10:30.491516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:01:02.902411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.678617+0000","last_change":"2024-01-11T08:10:29.666353+0000","last_active":"2024-01-11T08:10:30.678617+0000","last_peered":"2024-01-11T08:10:30.678617+0000","last_clean":"2024-01-11T08:10:30.678617+0000","last_became_active":"2024-01-11T08:10:29.665995+0000","last_became_peered":"2024-01-11T08:10:29.665995+0000","last_unstale":"2024-01-11T08:10:30.678617+0000","last_undegraded":"2024-01-11T08:10:30.678617+0000","last_fullsized":"2024-01-11T08:10:30.678617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:12:48.941888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.503379+0000","last_change":"2024-01-11T08:10:30.499297+0000","last_active":"2024-01-11T08:10:30.503379+0000","last_peered":"2024-01-11T08:10:30.503379+0000","last_clean":"2024-01-11T08:10:30.503379+0000","last_became_active":"2024-01-11T08:10:30.497567+0000","last_became_peered":"2024-01-11T08:10:30.497567+0000","last_unstale":"2024-01-11T08:10:30.503379+0000","last_undegraded":"2024-01-11T08:10:30.503379+0000","last_fullsized":"2024-01-11T08:10:30.503379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:49.967555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.685960+0000","last_change":"2024-01-11T08:10:29.671697+0000","last_active":"2024-01-11T08:10:30.685960+0000","last_peered":"2024-01-11T08:10:30.685960+0000","last_clean":"2024-01-11T08:10:30.685960+0000","last_became_active":"2024-01-11T08:10:29.671395+0000","last_became_peered":"2024-01-11T08:10:29.671395+0000","last_unstale":"2024-01-11T08:10:30.685960+0000","last_undegraded":"2024-01-11T08:10:30.685960+0000","last_fullsized":"2024-01-11T08:10:30.685960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:12:48.941888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.494324+0000","last_change":"2024-01-11T08:10:30.489779+0000","last_active":"2024-01-11T08:10:30.494324+0000","last_peered":"2024-01-11T08:10:30.494324+0000","last_clean":"2024-01-11T08:10:30.494324+0000","last_became_active":"2024-01-11T08:10:30.489290+0000","last_became_peered":"2024-01-11T08:10:30.489290+0000","last_unstale":"2024-01-11T08:10:30.494324+0000","last_undegraded":"2024-01-11T08:10:30.494324+0000","last_fullsized":"2024-01-11T08:10:30.494324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:50:04.791490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.678500+0000","last_change":"2024-01-11T08:10:29.682961+0000","last_active":"2024-01-11T08:10:30.678500+0000","last_peered":"2024-01-11T08:10:30.678500+0000","last_clean":"2024-01-11T08:10:30.678500+0000","last_became_active":"2024-01-11T08:10:29.682646+0000","last_became_peered":"2024-01-11T08:10:29.682646+0000","last_unstale":"2024-01-11T08:10:30.678500+0000","last_undegraded":"2024-01-11T08:10:30.678500+0000","last_fullsized":"2024-01-11T08:10:30.678500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:11:42.997404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.496569+0000","last_change":"2024-01-11T08:10:30.493989+0000","last_active":"2024-01-11T08:10:30.496569+0000","last_peered":"2024-01-11T08:10:30.496569+0000","last_clean":"2024-01-11T08:10:30.496569+0000","last_became_active":"2024-01-11T08:10:30.493678+0000","last_became_peered":"2024-01-11T08:10:30.493678+0000","last_unstale":"2024-01-11T08:10:30.496569+0000","last_undegraded":"2024-01-11T08:10:30.496569+0000","last_fullsized":"2024-01-11T08:10:30.496569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:03:43.208019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.471218+0000","last_change":"2024-01-11T08:10:29.670911+0000","last_active":"2024-01-11T08:10:30.471218+0000","last_peered":"2024-01-11T08:10:30.471218+0000","last_clean":"2024-01-11T08:10:30.471218+0000","last_became_active":"2024-01-11T08:10:29.670615+0000","last_became_peered":"2024-01-11T08:10:29.670615+0000","last_unstale":"2024-01-11T08:10:30.471218+0000","last_undegraded":"2024-01-11T08:10:30.471218+0000","last_fullsized":"2024-01-11T08:10:30.471218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:46:57.523656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.500438+0000","last_change":"2024-01-11T08:10:30.497306+0000","last_active":"2024-01-11T08:10:30.500438+0000","last_peered":"2024-01-11T08:10:30.500438+0000","last_clean":"2024-01-11T08:10:30.500438+0000","last_became_active":"2024-01-11T08:10:30.496966+0000","last_became_peered":"2024-01-11T08:10:30.496966+0000","last_unstale":"2024-01-11T08:10:30.500438+0000","last_undegraded":"2024-01-11T08:10:30.500438+0000","last_fullsized":"2024-01-11T08:10:30.500438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:47:17.705767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.483926+0000","last_change":"2024-01-11T08:10:29.680426+0000","last_active":"2024-01-11T08:10:30.483926+0000","last_peered":"2024-01-11T08:10:30.483926+0000","last_clean":"2024-01-11T08:10:30.483926+0000","last_became_active":"2024-01-11T08:10:29.680133+0000","last_became_peered":"2024-01-11T08:10:29.680133+0000","last_unstale":"2024-01-11T08:10:30.483926+0000","last_undegraded":"2024-01-11T08:10:30.483926+0000","last_fullsized":"2024-01-11T08:10:30.483926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:21.132838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.500317+0000","last_change":"2024-01-11T08:10:30.494824+0000","last_active":"2024-01-11T08:10:30.500317+0000","last_peered":"2024-01-11T08:10:30.500317+0000","last_clean":"2024-01-11T08:10:30.500317+0000","last_became_active":"2024-01-11T08:10:30.494564+0000","last_became_peered":"2024-01-11T08:10:30.494564+0000","last_unstale":"2024-01-11T08:10:30.500317+0000","last_undegraded":"2024-01-11T08:10:30.500317+0000","last_fullsized":"2024-01-11T08:10:30.500317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:37:25.109490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.687461+0000","last_change":"2024-01-11T08:10:29.671294+0000","last_active":"2024-01-11T08:10:30.687461+0000","last_peered":"2024-01-11T08:10:30.687461+0000","last_clean":"2024-01-11T08:10:30.687461+0000","last_became_active":"2024-01-11T08:10:29.670997+0000","last_became_peered":"2024-01-11T08:10:29.670997+0000","last_unstale":"2024-01-11T08:10:30.687461+0000","last_undegraded":"2024-01-11T08:10:30.687461+0000","last_fullsized":"2024-01-11T08:10:30.687461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:46:57.523656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.502943+0000","last_change":"2024-01-11T08:10:30.498224+0000","last_active":"2024-01-11T08:10:30.502943+0000","last_peered":"2024-01-11T08:10:30.502943+0000","last_clean":"2024-01-11T08:10:30.502943+0000","last_became_active":"2024-01-11T08:10:30.497993+0000","last_became_peered":"2024-01-11T08:10:30.497993+0000","last_unstale":"2024-01-11T08:10:30.502943+0000","last_undegraded":"2024-01-11T08:10:30.502943+0000","last_fullsized":"2024-01-11T08:10:30.502943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:51:50.621267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.676301+0000","last_change":"2024-01-11T08:10:29.678760+0000","last_active":"2024-01-11T08:10:30.676301+0000","last_peered":"2024-01-11T08:10:30.676301+0000","last_clean":"2024-01-11T08:10:30.676301+0000","last_became_active":"2024-01-11T08:10:29.678519+0000","last_became_peered":"2024-01-11T08:10:29.678519+0000","last_unstale":"2024-01-11T08:10:30.676301+0000","last_undegraded":"2024-01-11T08:10:30.676301+0000","last_fullsized":"2024-01-11T08:10:30.676301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:11:42.997404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.505207+0000","last_change":"2024-01-11T08:10:30.504188+0000","last_active":"2024-01-11T08:10:30.505207+0000","last_peered":"2024-01-11T08:10:30.505207+0000","last_clean":"2024-01-11T08:10:30.505207+0000","last_became_active":"2024-01-11T08:10:30.503805+0000","last_became_peered":"2024-01-11T08:10:30.503805+0000","last_unstale":"2024-01-11T08:10:30.505207+0000","last_undegraded":"2024-01-11T08:10:30.505207+0000","last_fullsized":"2024-01-11T08:10:30.505207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:36:17.244182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.473300+0000","last_change":"2024-01-11T08:10:29.671224+0000","last_active":"2024-01-11T08:10:30.473300+0000","last_peered":"2024-01-11T08:10:30.473300+0000","last_clean":"2024-01-11T08:10:30.473300+0000","last_became_active":"2024-01-11T08:10:29.670078+0000","last_became_peered":"2024-01-11T08:10:29.670078+0000","last_unstale":"2024-01-11T08:10:30.473300+0000","last_undegraded":"2024-01-11T08:10:30.473300+0000","last_fullsized":"2024-01-11T08:10:30.473300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:50:45.947278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.486167+0000","last_change":"2024-01-11T08:10:30.479486+0000","last_active":"2024-01-11T08:10:30.486167+0000","last_peered":"2024-01-11T08:10:30.486167+0000","last_clean":"2024-01-11T08:10:30.486167+0000","last_became_active":"2024-01-11T08:10:30.477788+0000","last_became_peered":"2024-01-11T08:10:30.477788+0000","last_unstale":"2024-01-11T08:10:30.486167+0000","last_undegraded":"2024-01-11T08:10:30.486167+0000","last_fullsized":"2024-01-11T08:10:30.486167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:50:04.791490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.471183+0000","last_change":"2024-01-11T08:10:29.670241+0000","last_active":"2024-01-11T08:10:30.471183+0000","last_peered":"2024-01-11T08:10:30.471183+0000","last_clean":"2024-01-11T08:10:30.471183+0000","last_became_active":"2024-01-11T08:10:29.669919+0000","last_became_peered":"2024-01-11T08:10:29.669919+0000","last_unstale":"2024-01-11T08:10:30.471183+0000","last_undegraded":"2024-01-11T08:10:30.471183+0000","last_fullsized":"2024-01-11T08:10:30.471183+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:50:14.474575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.498632+0000","last_change":"2024-01-11T08:10:30.490994+0000","last_active":"2024-01-11T08:10:30.498632+0000","last_peered":"2024-01-11T08:10:30.498632+0000","last_clean":"2024-01-11T08:10:30.498632+0000","last_became_active":"2024-01-11T08:10:30.490710+0000","last_became_peered":"2024-01-11T08:10:30.490710+0000","last_unstale":"2024-01-11T08:10:30.498632+0000","last_undegraded":"2024-01-11T08:10:30.498632+0000","last_fullsized":"2024-01-11T08:10:30.498632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:56:13.553335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.486574+0000","last_change":"2024-01-11T08:10:30.483950+0000","last_active":"2024-01-11T08:10:30.486574+0000","last_peered":"2024-01-11T08:10:30.486574+0000","last_clean":"2024-01-11T08:10:30.486574+0000","last_became_active":"2024-01-11T08:10:30.483643+0000","last_became_peered":"2024-01-11T08:10:30.483643+0000","last_unstale":"2024-01-11T08:10:30.486574+0000","last_undegraded":"2024-01-11T08:10:30.486574+0000","last_fullsized":"2024-01-11T08:10:30.486574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:47.753678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.472149+0000","last_change":"2024-01-11T08:10:29.667798+0000","last_active":"2024-01-11T08:10:30.472149+0000","last_peered":"2024-01-11T08:10:30.472149+0000","last_clean":"2024-01-11T08:10:30.472149+0000","last_became_active":"2024-01-11T08:10:29.667455+0000","last_became_peered":"2024-01-11T08:10:29.667455+0000","last_unstale":"2024-01-11T08:10:30.472149+0000","last_undegraded":"2024-01-11T08:10:30.472149+0000","last_fullsized":"2024-01-11T08:10:30.472149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:33.696922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.480195+0000","last_change":"2024-01-11T08:10:30.477204+0000","last_active":"2024-01-11T08:10:30.480195+0000","last_peered":"2024-01-11T08:10:30.480195+0000","last_clean":"2024-01-11T08:10:30.480195+0000","last_became_active":"2024-01-11T08:10:30.476896+0000","last_became_peered":"2024-01-11T08:10:30.476896+0000","last_unstale":"2024-01-11T08:10:30.480195+0000","last_undegraded":"2024-01-11T08:10:30.480195+0000","last_fullsized":"2024-01-11T08:10:30.480195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:22:43.961595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.472646+0000","last_change":"2024-01-11T08:10:29.666032+0000","last_active":"2024-01-11T08:10:30.472646+0000","last_peered":"2024-01-11T08:10:30.472646+0000","last_clean":"2024-01-11T08:10:30.472646+0000","last_became_active":"2024-01-11T08:10:29.665708+0000","last_became_peered":"2024-01-11T08:10:29.665708+0000","last_unstale":"2024-01-11T08:10:30.472646+0000","last_undegraded":"2024-01-11T08:10:30.472646+0000","last_fullsized":"2024-01-11T08:10:30.472646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:50:45.947278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.476809+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:10:30.476809+0000","last_peered":"2024-01-11T08:10:30.476809+0000","last_clean":"2024-01-11T08:10:30.476809+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:10:30.476809+0000","last_undegraded":"2024-01-11T08:10:30.476809+0000","last_fullsized":"2024-01-11T08:10:30.476809+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.495364+0000","last_change":"2024-01-11T08:10:30.491665+0000","last_active":"2024-01-11T08:10:30.495364+0000","last_peered":"2024-01-11T08:10:30.495364+0000","last_clean":"2024-01-11T08:10:30.495364+0000","last_became_active":"2024-01-11T08:10:30.491329+0000","last_became_peered":"2024-01-11T08:10:30.491329+0000","last_unstale":"2024-01-11T08:10:30.495364+0000","last_undegraded":"2024-01-11T08:10:30.495364+0000","last_fullsized":"2024-01-11T08:10:30.495364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:26:51.888257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.476968+0000","last_change":"2024-01-11T08:10:29.676382+0000","last_active":"2024-01-11T08:10:30.476968+0000","last_peered":"2024-01-11T08:10:30.476968+0000","last_clean":"2024-01-11T08:10:30.476968+0000","last_became_active":"2024-01-11T08:10:29.674690+0000","last_became_peered":"2024-01-11T08:10:29.674690+0000","last_unstale":"2024-01-11T08:10:30.476968+0000","last_undegraded":"2024-01-11T08:10:30.476968+0000","last_fullsized":"2024-01-11T08:10:30.476968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:12:48.941888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.479210+0000","last_change":"2024-01-11T08:10:29.680063+0000","last_active":"2024-01-11T08:10:30.479210+0000","last_peered":"2024-01-11T08:10:30.479210+0000","last_clean":"2024-01-11T08:10:30.479210+0000","last_became_active":"2024-01-11T08:10:29.677976+0000","last_became_peered":"2024-01-11T08:10:29.677976+0000","last_unstale":"2024-01-11T08:10:30.479210+0000","last_undegraded":"2024-01-11T08:10:30.479210+0000","last_fullsized":"2024-01-11T08:10:30.479210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:50:45.947278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.486597+0000","last_change":"2024-01-11T08:10:30.483048+0000","last_active":"2024-01-11T08:10:30.486597+0000","last_peered":"2024-01-11T08:10:30.486597+0000","last_clean":"2024-01-11T08:10:30.486597+0000","last_became_active":"2024-01-11T08:10:30.482693+0000","last_became_peered":"2024-01-11T08:10:30.482693+0000","last_unstale":"2024-01-11T08:10:30.486597+0000","last_undegraded":"2024-01-11T08:10:30.486597+0000","last_fullsized":"2024-01-11T08:10:30.486597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:22:43.961595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.474674+0000","last_change":"2024-01-11T08:10:29.673099+0000","last_active":"2024-01-11T08:10:30.474674+0000","last_peered":"2024-01-11T08:10:30.474674+0000","last_clean":"2024-01-11T08:10:30.474674+0000","last_became_active":"2024-01-11T08:10:29.672796+0000","last_became_peered":"2024-01-11T08:10:29.672796+0000","last_unstale":"2024-01-11T08:10:30.474674+0000","last_undegraded":"2024-01-11T08:10:30.474674+0000","last_fullsized":"2024-01-11T08:10:30.474674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:05.676391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.490638+0000","last_change":"2024-01-11T08:10:30.489252+0000","last_active":"2024-01-11T08:10:30.490638+0000","last_peered":"2024-01-11T08:10:30.490638+0000","last_clean":"2024-01-11T08:10:30.490638+0000","last_became_active":"2024-01-11T08:10:30.488728+0000","last_became_peered":"2024-01-11T08:10:30.488728+0000","last_unstale":"2024-01-11T08:10:30.490638+0000","last_undegraded":"2024-01-11T08:10:30.490638+0000","last_fullsized":"2024-01-11T08:10:30.490638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:01:02.902411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.680479+0000","last_change":"2024-01-11T08:10:29.678413+0000","last_active":"2024-01-11T08:10:30.680479+0000","last_peered":"2024-01-11T08:10:30.680479+0000","last_clean":"2024-01-11T08:10:30.680479+0000","last_became_active":"2024-01-11T08:10:29.678023+0000","last_became_peered":"2024-01-11T08:10:29.678023+0000","last_unstale":"2024-01-11T08:10:30.680479+0000","last_undegraded":"2024-01-11T08:10:30.680479+0000","last_fullsized":"2024-01-11T08:10:30.680479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:50:45.947278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.491408+0000","last_change":"2024-01-11T08:10:30.486306+0000","last_active":"2024-01-11T08:10:30.491408+0000","last_peered":"2024-01-11T08:10:30.491408+0000","last_clean":"2024-01-11T08:10:30.491408+0000","last_became_active":"2024-01-11T08:10:30.486002+0000","last_became_peered":"2024-01-11T08:10:30.486002+0000","last_unstale":"2024-01-11T08:10:30.491408+0000","last_undegraded":"2024-01-11T08:10:30.491408+0000","last_fullsized":"2024-01-11T08:10:30.491408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:21.127779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.476646+0000","last_change":"2024-01-11T08:10:29.674369+0000","last_active":"2024-01-11T08:10:30.476646+0000","last_peered":"2024-01-11T08:10:30.476646+0000","last_clean":"2024-01-11T08:10:30.476646+0000","last_became_active":"2024-01-11T08:10:29.673110+0000","last_became_peered":"2024-01-11T08:10:29.673110+0000","last_unstale":"2024-01-11T08:10:30.476646+0000","last_undegraded":"2024-01-11T08:10:30.476646+0000","last_fullsized":"2024-01-11T08:10:30.476646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:27:23.756042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.483307+0000","last_change":"2024-01-11T08:10:30.476023+0000","last_active":"2024-01-11T08:10:30.483307+0000","last_peered":"2024-01-11T08:10:30.483307+0000","last_clean":"2024-01-11T08:10:30.483307+0000","last_became_active":"2024-01-11T08:10:30.474988+0000","last_became_peered":"2024-01-11T08:10:30.474988+0000","last_unstale":"2024-01-11T08:10:30.483307+0000","last_undegraded":"2024-01-11T08:10:30.483307+0000","last_fullsized":"2024-01-11T08:10:30.483307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:12:49.944597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.472984+0000","last_change":"2024-01-11T08:10:29.675819+0000","last_active":"2024-01-11T08:10:30.472984+0000","last_peered":"2024-01-11T08:10:30.472984+0000","last_clean":"2024-01-11T08:10:30.472984+0000","last_became_active":"2024-01-11T08:10:29.675005+0000","last_became_peered":"2024-01-11T08:10:29.675005+0000","last_unstale":"2024-01-11T08:10:30.472984+0000","last_undegraded":"2024-01-11T08:10:30.472984+0000","last_fullsized":"2024-01-11T08:10:30.472984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:27:23.756042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.479995+0000","last_change":"2024-01-11T08:10:30.475525+0000","last_active":"2024-01-11T08:10:30.479995+0000","last_peered":"2024-01-11T08:10:30.479995+0000","last_clean":"2024-01-11T08:10:30.479995+0000","last_became_active":"2024-01-11T08:10:30.475243+0000","last_became_peered":"2024-01-11T08:10:30.475243+0000","last_unstale":"2024-01-11T08:10:30.479995+0000","last_undegraded":"2024-01-11T08:10:30.479995+0000","last_fullsized":"2024-01-11T08:10:30.479995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:37:39.503131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.473867+0000","last_change":"2024-01-11T08:10:29.672091+0000","last_active":"2024-01-11T08:10:30.473867+0000","last_peered":"2024-01-11T08:10:30.473867+0000","last_clean":"2024-01-11T08:10:30.473867+0000","last_became_active":"2024-01-11T08:10:29.671820+0000","last_became_peered":"2024-01-11T08:10:29.671820+0000","last_unstale":"2024-01-11T08:10:30.473867+0000","last_undegraded":"2024-01-11T08:10:30.473867+0000","last_fullsized":"2024-01-11T08:10:30.473867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:46:57.523656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.490433+0000","last_change":"2024-01-11T08:10:30.488516+0000","last_active":"2024-01-11T08:10:30.490433+0000","last_peered":"2024-01-11T08:10:30.490433+0000","last_clean":"2024-01-11T08:10:30.490433+0000","last_became_active":"2024-01-11T08:10:30.487592+0000","last_became_peered":"2024-01-11T08:10:30.487592+0000","last_unstale":"2024-01-11T08:10:30.490433+0000","last_undegraded":"2024-01-11T08:10:30.490433+0000","last_fullsized":"2024-01-11T08:10:30.490433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:57:19.989587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.678978+0000","last_change":"2024-01-11T08:10:29.683377+0000","last_active":"2024-01-11T08:10:30.678978+0000","last_peered":"2024-01-11T08:10:30.678978+0000","last_clean":"2024-01-11T08:10:30.678978+0000","last_became_active":"2024-01-11T08:10:29.680358+0000","last_became_peered":"2024-01-11T08:10:29.680358+0000","last_unstale":"2024-01-11T08:10:30.678978+0000","last_undegraded":"2024-01-11T08:10:30.678978+0000","last_fullsized":"2024-01-11T08:10:30.678978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:05.676391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.502306+0000","last_change":"2024-01-11T08:10:30.499934+0000","last_active":"2024-01-11T08:10:30.502306+0000","last_peered":"2024-01-11T08:10:30.502306+0000","last_clean":"2024-01-11T08:10:30.502306+0000","last_became_active":"2024-01-11T08:10:30.499676+0000","last_became_peered":"2024-01-11T08:10:30.499676+0000","last_unstale":"2024-01-11T08:10:30.502306+0000","last_undegraded":"2024-01-11T08:10:30.502306+0000","last_fullsized":"2024-01-11T08:10:30.502306+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:47.753678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.471402+0000","last_change":"2024-01-11T08:10:29.665360+0000","last_active":"2024-01-11T08:10:30.471402+0000","last_peered":"2024-01-11T08:10:30.471402+0000","last_clean":"2024-01-11T08:10:30.471402+0000","last_became_active":"2024-01-11T08:10:29.664493+0000","last_became_peered":"2024-01-11T08:10:29.664493+0000","last_unstale":"2024-01-11T08:10:30.471402+0000","last_undegraded":"2024-01-11T08:10:30.471402+0000","last_fullsized":"2024-01-11T08:10:30.471402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:12:48.941888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.495532+0000","last_change":"2024-01-11T08:10:30.491426+0000","last_active":"2024-01-11T08:10:30.495532+0000","last_peered":"2024-01-11T08:10:30.495532+0000","last_clean":"2024-01-11T08:10:30.495532+0000","last_became_active":"2024-01-11T08:10:30.491167+0000","last_became_peered":"2024-01-11T08:10:30.491167+0000","last_unstale":"2024-01-11T08:10:30.495532+0000","last_undegraded":"2024-01-11T08:10:30.495532+0000","last_fullsized":"2024-01-11T08:10:30.495532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:12.649092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.678720+0000","last_change":"2024-01-11T08:10:29.669311+0000","last_active":"2024-01-11T08:10:30.678720+0000","last_peered":"2024-01-11T08:10:30.678720+0000","last_clean":"2024-01-11T08:10:30.678720+0000","last_became_active":"2024-01-11T08:10:29.669042+0000","last_became_peered":"2024-01-11T08:10:29.669042+0000","last_unstale":"2024-01-11T08:10:30.678720+0000","last_undegraded":"2024-01-11T08:10:30.678720+0000","last_fullsized":"2024-01-11T08:10:30.678720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:48.964847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.494014+0000","last_change":"2024-01-11T08:10:30.486285+0000","last_active":"2024-01-11T08:10:30.494014+0000","last_peered":"2024-01-11T08:10:30.494014+0000","last_clean":"2024-01-11T08:10:30.494014+0000","last_became_active":"2024-01-11T08:10:30.482784+0000","last_became_peered":"2024-01-11T08:10:30.482784+0000","last_unstale":"2024-01-11T08:10:30.494014+0000","last_undegraded":"2024-01-11T08:10:30.494014+0000","last_fullsized":"2024-01-11T08:10:30.494014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:37:39.503131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.687135+0000","last_change":"2024-01-11T08:10:29.668553+0000","last_active":"2024-01-11T08:10:30.687135+0000","last_peered":"2024-01-11T08:10:30.687135+0000","last_clean":"2024-01-11T08:10:30.687135+0000","last_became_active":"2024-01-11T08:10:29.666406+0000","last_became_peered":"2024-01-11T08:10:29.666406+0000","last_unstale":"2024-01-11T08:10:30.687135+0000","last_undegraded":"2024-01-11T08:10:30.687135+0000","last_fullsized":"2024-01-11T08:10:30.687135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:50:14.474575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.503635+0000","last_change":"2024-01-11T08:10:30.501614+0000","last_active":"2024-01-11T08:10:30.503635+0000","last_peered":"2024-01-11T08:10:30.503635+0000","last_clean":"2024-01-11T08:10:30.503635+0000","last_became_active":"2024-01-11T08:10:30.501255+0000","last_became_peered":"2024-01-11T08:10:30.501255+0000","last_unstale":"2024-01-11T08:10:30.503635+0000","last_undegraded":"2024-01-11T08:10:30.503635+0000","last_fullsized":"2024-01-11T08:10:30.503635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:18:09.736814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.491229+0000","last_change":"2024-01-11T08:10:30.488971+0000","last_active":"2024-01-11T08:10:30.491229+0000","last_peered":"2024-01-11T08:10:30.491229+0000","last_clean":"2024-01-11T08:10:30.491229+0000","last_became_active":"2024-01-11T08:10:30.488681+0000","last_became_peered":"2024-01-11T08:10:30.488681+0000","last_unstale":"2024-01-11T08:10:30.491229+0000","last_undegraded":"2024-01-11T08:10:30.491229+0000","last_fullsized":"2024-01-11T08:10:30.491229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:26:51.888257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.481588+0000","last_change":"2024-01-11T08:10:30.477289+0000","last_active":"2024-01-11T08:10:30.481588+0000","last_peered":"2024-01-11T08:10:30.481588+0000","last_clean":"2024-01-11T08:10:30.481588+0000","last_became_active":"2024-01-11T08:10:30.477029+0000","last_became_peered":"2024-01-11T08:10:30.477029+0000","last_unstale":"2024-01-11T08:10:30.481588+0000","last_undegraded":"2024-01-11T08:10:30.481588+0000","last_fullsized":"2024-01-11T08:10:30.481588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:49.967555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.483210+0000","last_change":"2024-01-11T08:10:30.478289+0000","last_active":"2024-01-11T08:10:30.483210+0000","last_peered":"2024-01-11T08:10:30.483210+0000","last_clean":"2024-01-11T08:10:30.483210+0000","last_became_active":"2024-01-11T08:10:30.477815+0000","last_became_peered":"2024-01-11T08:10:30.477815+0000","last_unstale":"2024-01-11T08:10:30.483210+0000","last_undegraded":"2024-01-11T08:10:30.483210+0000","last_fullsized":"2024-01-11T08:10:30.483210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:14:36.595264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.491328+0000","last_change":"2024-01-11T08:10:30.486406+0000","last_active":"2024-01-11T08:10:30.491328+0000","last_peered":"2024-01-11T08:10:30.491328+0000","last_clean":"2024-01-11T08:10:30.491328+0000","last_became_active":"2024-01-11T08:10:30.485979+0000","last_became_peered":"2024-01-11T08:10:30.485979+0000","last_unstale":"2024-01-11T08:10:30.491328+0000","last_undegraded":"2024-01-11T08:10:30.491328+0000","last_fullsized":"2024-01-11T08:10:30.491328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:37:25.109490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.485482+0000","last_change":"2024-01-11T08:10:30.480947+0000","last_active":"2024-01-11T08:10:30.485482+0000","last_peered":"2024-01-11T08:10:30.485482+0000","last_clean":"2024-01-11T08:10:30.485482+0000","last_became_active":"2024-01-11T08:10:30.478542+0000","last_became_peered":"2024-01-11T08:10:30.478542+0000","last_unstale":"2024-01-11T08:10:30.485482+0000","last_undegraded":"2024-01-11T08:10:30.485482+0000","last_fullsized":"2024-01-11T08:10:30.485482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:21:29.220934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.487827+0000","last_change":"2024-01-11T08:10:30.486206+0000","last_active":"2024-01-11T08:10:30.487827+0000","last_peered":"2024-01-11T08:10:30.487827+0000","last_clean":"2024-01-11T08:10:30.487827+0000","last_became_active":"2024-01-11T08:10:30.482118+0000","last_became_peered":"2024-01-11T08:10:30.482118+0000","last_unstale":"2024-01-11T08:10:30.487827+0000","last_undegraded":"2024-01-11T08:10:30.487827+0000","last_fullsized":"2024-01-11T08:10:30.487827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:37:39.503131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.502193+0000","last_change":"2024-01-11T08:10:30.498251+0000","last_active":"2024-01-11T08:10:30.502193+0000","last_peered":"2024-01-11T08:10:30.502193+0000","last_clean":"2024-01-11T08:10:30.502193+0000","last_became_active":"2024-01-11T08:10:30.498032+0000","last_became_peered":"2024-01-11T08:10:30.498032+0000","last_unstale":"2024-01-11T08:10:30.502193+0000","last_undegraded":"2024-01-11T08:10:30.502193+0000","last_fullsized":"2024-01-11T08:10:30.502193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:22:43.961595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.480236+0000","last_change":"2024-01-11T08:10:30.477044+0000","last_active":"2024-01-11T08:10:30.480236+0000","last_peered":"2024-01-11T08:10:30.480236+0000","last_clean":"2024-01-11T08:10:30.480236+0000","last_became_active":"2024-01-11T08:10:30.476728+0000","last_became_peered":"2024-01-11T08:10:30.476728+0000","last_unstale":"2024-01-11T08:10:30.480236+0000","last_undegraded":"2024-01-11T08:10:30.480236+0000","last_fullsized":"2024-01-11T08:10:30.480236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:08:10.140431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.485414+0000","last_change":"2024-01-11T08:10:30.481145+0000","last_active":"2024-01-11T08:10:30.485414+0000","last_peered":"2024-01-11T08:10:30.485414+0000","last_clean":"2024-01-11T08:10:30.485414+0000","last_became_active":"2024-01-11T08:10:30.477464+0000","last_became_peered":"2024-01-11T08:10:30.477464+0000","last_unstale":"2024-01-11T08:10:30.485414+0000","last_undegraded":"2024-01-11T08:10:30.485414+0000","last_fullsized":"2024-01-11T08:10:30.485414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:02.887025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.485358+0000","last_change":"2024-01-11T08:10:30.480047+0000","last_active":"2024-01-11T08:10:30.485358+0000","last_peered":"2024-01-11T08:10:30.485358+0000","last_clean":"2024-01-11T08:10:30.485358+0000","last_became_active":"2024-01-11T08:10:30.479594+0000","last_became_peered":"2024-01-11T08:10:30.479594+0000","last_unstale":"2024-01-11T08:10:30.485358+0000","last_undegraded":"2024-01-11T08:10:30.485358+0000","last_fullsized":"2024-01-11T08:10:30.485358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:14:36.595264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.490024+0000","last_change":"2024-01-11T08:10:30.486964+0000","last_active":"2024-01-11T08:10:30.490024+0000","last_peered":"2024-01-11T08:10:30.490024+0000","last_clean":"2024-01-11T08:10:30.490024+0000","last_became_active":"2024-01-11T08:10:30.486679+0000","last_became_peered":"2024-01-11T08:10:30.486679+0000","last_unstale":"2024-01-11T08:10:30.490024+0000","last_undegraded":"2024-01-11T08:10:30.490024+0000","last_fullsized":"2024-01-11T08:10:30.490024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:57:19.989587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.497053+0000","last_change":"2024-01-11T08:10:30.495904+0000","last_active":"2024-01-11T08:10:30.497053+0000","last_peered":"2024-01-11T08:10:30.497053+0000","last_clean":"2024-01-11T08:10:30.497053+0000","last_became_active":"2024-01-11T08:10:30.495590+0000","last_became_peered":"2024-01-11T08:10:30.495590+0000","last_unstale":"2024-01-11T08:10:30.497053+0000","last_undegraded":"2024-01-11T08:10:30.497053+0000","last_fullsized":"2024-01-11T08:10:30.497053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:37:39.503131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.502287+0000","last_change":"2024-01-11T08:10:30.499796+0000","last_active":"2024-01-11T08:10:30.502287+0000","last_peered":"2024-01-11T08:10:30.502287+0000","last_clean":"2024-01-11T08:10:30.502287+0000","last_became_active":"2024-01-11T08:10:30.499391+0000","last_became_peered":"2024-01-11T08:10:30.499391+0000","last_unstale":"2024-01-11T08:10:30.502287+0000","last_undegraded":"2024-01-11T08:10:30.502287+0000","last_fullsized":"2024-01-11T08:10:30.502287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:37:39.503131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298084,"kb_used_data":796,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073756,"statfs":{"total":96636764160,"available":96331526144,"internally_reserved":0,"allocated":815104,"data_stored":666235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":76849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":225280,"data_stored":75473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298092,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073748,"statfs":{"total":96636764160,"available":96331517952,"internally_reserved":0,"allocated":823296,"data_stored":666291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":217088,"data_stored":75313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":76849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1439999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":75439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":221184,"data_stored":75403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:10:35.571 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-11T08:10:35.571 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-11T08:10:35.571 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-11T08:10:35.572 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-11T08:10:35.572 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-11T08:10:35.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.806+0000 7f55f71ff700 1 Processor -- start 2024-01-11T08:10:35.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.807+0000 7f55f71ff700 1 -- start start 2024-01-11T08:10:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.808+0000 7f55f71ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f55f00b03e0 0x7f55f00b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.808+0000 7f55f71ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f55f00b07b0 0x7f55f00b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.808+0000 7f55f71ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55f00cd730 con 0x7f55f0159200 2024-01-11T08:10:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.808+0000 7f55f71ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55f0159d50 con 0x7f55f00b03e0 2024-01-11T08:10:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.808+0000 7f55f71ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f00ccb20 con 0x7f55f00b07b0 2024-01-11T08:10:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.808+0000 7f55f4f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f55f00b03e0 0x7f55f00b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.808+0000 7f55f4f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f55f00b03e0 0x7f55f00b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53126/0 (socket says 172.21.15.27:53126) 2024-01-11T08:10:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.808+0000 7f55f4f9b700 1 -- 172.21.15.27:0/64856473 learned_addr learned my addr 172.21.15.27:0/64856473 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.808+0000 7f55effff700 1 --2- 172.21.15.27:0/64856473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f55f00b07b0 0x7f55f00b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.808+0000 7f55f4f9b700 1 -- 172.21.15.27:0/64856473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f55f00b07b0 msgr2=0x7f55f00b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.808+0000 7f55f4f9b700 1 --2- 172.21.15.27:0/64856473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f55f00b07b0 0x7f55f00b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.808+0000 7f55f4f9b700 1 -- 172.21.15.27:0/64856473 >> v1:172.21.15.77:6789/0 conn(0x7f55f0159200 legacy=0x7f55f0159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.808+0000 7f55f4f9b700 1 -- 172.21.15.27:0/64856473 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f00cce00 con 0x7f55f00b03e0 2024-01-11T08:10:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.809+0000 7f55f4f9b700 1 --2- 172.21.15.27:0/64856473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55f00b03e0 0x7f55f00b2480 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f55e40099b0 tx=0x7f55e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48efea5d5a48a74b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.809+0000 7f55ef7fe700 1 -- 172.21.15.27:0/64856473 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e4009ea0 con 0x7f55f00b03e0 2024-01-11T08:10:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.809+0000 7f55ef7fe700 1 -- 172.21.15.27:0/64856473 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e400b860 con 0x7f55f00b03e0 2024-01-11T08:10:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.809+0000 7f55ef7fe700 1 -- 172.21.15.27:0/64856473 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e400bc00 con 0x7f55f00b03e0 2024-01-11T08:10:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.809+0000 7f55f71ff700 1 -- 172.21.15.27:0/64856473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55f00b03e0 msgr2=0x7f55f00b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.809+0000 7f55f71ff700 1 --2- 172.21.15.27:0/64856473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55f00b03e0 0x7f55f00b2480 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f55e40099b0 tx=0x7f55e4009250 comp rx=0 tx=0).stop 2024-01-11T08:10:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.810+0000 7f55f71ff700 1 -- 172.21.15.27:0/64856473 shutdown_connections 2024-01-11T08:10:35.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.810+0000 7f55f71ff700 1 --2- 172.21.15.27:0/64856473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55f00b03e0 0x7f55f00b2480 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.810+0000 7f55f71ff700 1 --2- 172.21.15.27:0/64856473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f55f00b07b0 0x7f55f00b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.810+0000 7f55f71ff700 1 -- 172.21.15.27:0/64856473 >> 172.21.15.27:0/64856473 conn(0x7f55f006ed40 msgr2=0x7f55f006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:35.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.810+0000 7f55f71ff700 1 -- 172.21.15.27:0/64856473 shutdown_connections 2024-01-11T08:10:35.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.810+0000 7f55f71ff700 1 -- 172.21.15.27:0/64856473 wait complete. 2024-01-11T08:10:35.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.811+0000 7f55f71ff700 1 Processor -- start 2024-01-11T08:10:35.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.811+0000 7f55f71ff700 1 -- start start 2024-01-11T08:10:35.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.812+0000 7f55f71ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55f00b03e0 0x7f55f00ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:35.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.812+0000 7f55f71ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55f00b07b0 0x7f55f00b4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:35.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.812+0000 7f55f71ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f55f0159200 0x7f55f016cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:35.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.812+0000 7f55f71ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f0158350 con 0x7f55f00b07b0 2024-01-11T08:10:35.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.812+0000 7f55f71ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f00ca040 con 0x7f55f00b03e0 2024-01-11T08:10:35.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.812+0000 7f55f71ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f00c9770 con 0x7f55f0159200 2024-01-11T08:10:35.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.812+0000 7f55effff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55f00b07b0 0x7f55f00b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:35.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.812+0000 7f55effff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55f00b07b0 0x7f55f00b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53134/0 (socket says 172.21.15.27:53134) 2024-01-11T08:10:35.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.813+0000 7f55effff700 1 -- 172.21.15.27:0/3434092561 learned_addr learned my addr 172.21.15.27:0/3434092561 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:35.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.813+0000 7f55f579c700 1 --2- 172.21.15.27:0/3434092561 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f55f0159200 0x7f55f016cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:35.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.813+0000 7f55f4f9b700 1 --2- 172.21.15.27:0/3434092561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55f00b03e0 0x7f55f00ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:35.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.813+0000 7f55effff700 1 -- 172.21.15.27:0/3434092561 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f55f0159200 msgr2=0x7f55f016cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:35.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.813+0000 7f55effff700 1 --2- 172.21.15.27:0/3434092561 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f55f0159200 0x7f55f016cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.813+0000 7f55effff700 1 -- 172.21.15.27:0/3434092561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55f00b03e0 msgr2=0x7f55f00ca4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:35.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.813+0000 7f55effff700 1 --2- 172.21.15.27:0/3434092561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55f00b03e0 0x7f55f00ca4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:35.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.813+0000 7f55effff700 1 -- 172.21.15.27:0/3434092561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e40094f0 con 0x7f55f00b07b0 2024-01-11T08:10:35.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.813+0000 7f55f4f9b700 1 --2- 172.21.15.27:0/3434092561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55f00b03e0 0x7f55f00ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:35.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.813+0000 7f55effff700 1 --2- 172.21.15.27:0/3434092561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55f00b07b0 0x7f55f00b4730 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f55dc00ec80 tx=0x7f55dc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:35.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.813+0000 7f55ecff9700 1 -- 172.21.15.27:0/3434092561 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc00c850 con 0x7f55f00b07b0 2024-01-11T08:10:35.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.813+0000 7f55ecff9700 1 -- 172.21.15.27:0/3434092561 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55dc00c9b0 con 0x7f55f00b07b0 2024-01-11T08:10:35.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.814+0000 7f55f71ff700 1 -- 172.21.15.27:0/3434092561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f00caa10 con 0x7f55f00b07b0 2024-01-11T08:10:35.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.814+0000 7f55ecff9700 1 -- 172.21.15.27:0/3434092561 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc00cd50 con 0x7f55f00b07b0 2024-01-11T08:10:35.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.814+0000 7f55f71ff700 1 -- 172.21.15.27:0/3434092561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f00c9cd0 con 0x7f55f00b07b0 2024-01-11T08:10:35.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.815+0000 7f55f71ff700 1 -- 172.21.15.27:0/3434092561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f00ca1d0 con 0x7f55f00b07b0 2024-01-11T08:10:35.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.816+0000 7f55ecff9700 1 -- 172.21.15.27:0/3434092561 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f55dc013680 con 0x7f55f00b07b0 2024-01-11T08:10:35.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.817+0000 7f55ecff9700 1 --2- 172.21.15.27:0/3434092561 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f55d4066f30 0x7f55d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:35.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.817+0000 7f55ecff9700 1 -- 172.21.15.27:0/3434092561 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f55dc084890 con 0x7f55f00b07b0 2024-01-11T08:10:35.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.817+0000 7f55f4f9b700 1 --2- 172.21.15.27:0/3434092561 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f55d4066f30 0x7f55d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:35.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.818+0000 7f55f4f9b700 1 --2- 172.21.15.27:0/3434092561 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f55d4066f30 0x7f55d40693e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f55e40099b0 tx=0x7f55e4017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:35.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:35.821+0000 7f55ecff9700 1 -- 172.21.15.27:0/3434092561 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f55dc051a60 con 0x7f55f00b07b0 2024-01-11T08:10:35.981 INFO:tasks.ceph.mds.d.smithi077.stderr:2024-01-11T08:10:35.981+0000 7fda615f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T08:10:36.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.077+0000 7f55f71ff700 1 -- 172.21.15.27:0/3434092561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f55f00718d0 con 0x7f55f00b07b0 2024-01-11T08:10:36.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.081+0000 7f55ecff9700 1 -- 172.21.15.27:0/3434092561 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+163700 (secure 0 0 0) 0x7f55dc055080 con 0x7f55f00b07b0 2024-01-11T08:10:36.084 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.084+0000 7f55f71ff700 1 -- 172.21.15.27:0/3434092561 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f55d4066f30 msgr2=0x7f55d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.084+0000 7f55f71ff700 1 --2- 172.21.15.27:0/3434092561 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f55d4066f30 0x7f55d40693e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f55e40099b0 tx=0x7f55e4017000 comp rx=0 tx=0).stop 2024-01-11T08:10:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.084+0000 7f55f71ff700 1 -- 172.21.15.27:0/3434092561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55f00b07b0 msgr2=0x7f55f00b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.084+0000 7f55f71ff700 1 --2- 172.21.15.27:0/3434092561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55f00b07b0 0x7f55f00b4730 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f55dc00ec80 tx=0x7f55dc0091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.084+0000 7f55f71ff700 1 -- 172.21.15.27:0/3434092561 shutdown_connections 2024-01-11T08:10:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.084+0000 7f55f71ff700 1 --2- 172.21.15.27:0/3434092561 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f55d4066f30 0x7f55d40693e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.084+0000 7f55f71ff700 1 --2- 172.21.15.27:0/3434092561 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f55f0159200 0x7f55f016cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.084+0000 7f55f71ff700 1 --2- 172.21.15.27:0/3434092561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55f00b03e0 0x7f55f00ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.084+0000 7f55f71ff700 1 --2- 172.21.15.27:0/3434092561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55f00b07b0 0x7f55f00b4730 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.084+0000 7f55f71ff700 1 -- 172.21.15.27:0/3434092561 >> 172.21.15.27:0/3434092561 conn(0x7f55f006ed40 msgr2=0x7f55f00cfca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.084+0000 7f55f71ff700 1 -- 172.21.15.27:0/3434092561 shutdown_connections 2024-01-11T08:10:36.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.084+0000 7f55f71ff700 1 -- 172.21.15.27:0/3434092561 wait complete. 2024-01-11T08:10:36.102 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":41381},{"type":"v1","addr":"172.21.15.77:6833","nonce":41381}]},"active_addr":"172.21.15.77:6833/41381","active_change":"2024-01-11T08:10:11.495126+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":3211563793}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":4222476855}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":2257956400}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":969431132}]}]} 2024-01-11T08:10:36.106 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-11T08:10:36.106 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-11T08:10:36.107 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:10:36.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.305+0000 7ff0e95ff700 1 Processor -- start 2024-01-11T08:10:36.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.305+0000 7ff0e95ff700 1 -- start start 2024-01-11T08:10:36.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.307+0000 7ff0e95ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff0e406a700 0x7ff0e406ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:36.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.307+0000 7ff0e95ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e415d860 con 0x7ff0e4159670 2024-01-11T08:10:36.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.307+0000 7ff0e95ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e406b050 con 0x7ff0e4159a40 2024-01-11T08:10:36.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.307+0000 7ff0e95ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e406ecd0 con 0x7ff0e406a700 2024-01-11T08:10:36.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.308+0000 7ff0e37fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff0e406a700 0x7ff0e406ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:36.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.308+0000 7ff0e2ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff0e4159670 0x7ff0e415dcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37114/0 (socket says 172.21.15.27:37114) 2024-01-11T08:10:36.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.308+0000 7ff0e2ffd700 1 -- 172.21.15.27:0/882637714 learned_addr learned my addr 172.21.15.27:0/882637714 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.308+0000 7ff0e37fe700 1 -- 172.21.15.27:0/882637714 >> v1:172.21.15.77:6789/0 conn(0x7ff0e4159a40 legacy=0x7ff0e406e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.308+0000 7ff0e1ffb700 1 -- 172.21.15.27:0/882637714 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54195071 0 0) 0x7ff0e415d860 con 0x7ff0e4159670 2024-01-11T08:10:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.308+0000 7ff0e37fe700 1 -- 172.21.15.27:0/882637714 >> v1:172.21.15.27:6789/0 conn(0x7ff0e4159670 legacy=0x7ff0e415dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.308+0000 7ff0e37fe700 1 -- 172.21.15.27:0/882637714 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e406efa0 con 0x7ff0e406a700 2024-01-11T08:10:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.308+0000 7ff0e37fe700 1 --2- 172.21.15.27:0/882637714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e406a700 0x7ff0e406ab10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc009e60 tx=0x7ff0dc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12cfaf8eb6df4149 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.308+0000 7ff0e1ffb700 1 -- 172.21.15.27:0/882637714 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc00e070 con 0x7ff0e406a700 2024-01-11T08:10:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.309+0000 7ff0e1ffb700 1 -- 172.21.15.27:0/882637714 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0dc00bc30 con 0x7ff0e406a700 2024-01-11T08:10:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.309+0000 7ff0e1ffb700 1 -- 172.21.15.27:0/882637714 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc012430 con 0x7ff0e406a700 2024-01-11T08:10:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.309+0000 7ff0e95ff700 1 -- 172.21.15.27:0/882637714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e406a700 msgr2=0x7ff0e406ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.309+0000 7ff0e95ff700 1 --2- 172.21.15.27:0/882637714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e406a700 0x7ff0e406ab10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc009e60 tx=0x7ff0dc009780 comp rx=0 tx=0).stop 2024-01-11T08:10:36.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.309+0000 7ff0e95ff700 1 -- 172.21.15.27:0/882637714 shutdown_connections 2024-01-11T08:10:36.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.309+0000 7ff0e95ff700 1 --2- 172.21.15.27:0/882637714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e406a700 0x7ff0e406ab10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.309+0000 7ff0e95ff700 1 -- 172.21.15.27:0/882637714 >> 172.21.15.27:0/882637714 conn(0x7ff0e40cd2b0 msgr2=0x7ff0e40cd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:36.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.309+0000 7ff0e95ff700 1 -- 172.21.15.27:0/882637714 shutdown_connections 2024-01-11T08:10:36.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.309+0000 7ff0e95ff700 1 -- 172.21.15.27:0/882637714 wait complete. 2024-01-11T08:10:36.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.311+0000 7ff0e95ff700 1 Processor -- start 2024-01-11T08:10:36.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.311+0000 7ff0e95ff700 1 -- start start 2024-01-11T08:10:36.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.312+0000 7ff0e95ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0e406a700 0x7ff0e40b1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:36.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.312+0000 7ff0e95ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0e4159670 0x7ff0e40c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:36.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.312+0000 7ff0e95ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e4159a40 0x7ff0e4157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.312+0000 7ff0e95ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e40b1170 con 0x7ff0e4159a40 2024-01-11T08:10:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.312+0000 7ff0e95ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e40b0560 con 0x7ff0e406a700 2024-01-11T08:10:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.312+0000 7ff0e95ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e40b0840 con 0x7ff0e4159670 2024-01-11T08:10:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.312+0000 7ff0e37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e4159a40 0x7ff0e4157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.312+0000 7ff0e2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0e406a700 0x7ff0e40b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.312+0000 7ff0e37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e4159a40 0x7ff0e4157780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53158/0 (socket says 172.21.15.27:53158) 2024-01-11T08:10:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.312+0000 7ff0e37fe700 1 -- 172.21.15.27:0/3525360 learned_addr learned my addr 172.21.15.27:0/3525360 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.312+0000 7ff0e27fc700 1 --2- 172.21.15.27:0/3525360 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0e4159670 0x7ff0e40c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.312+0000 7ff0e37fe700 1 -- 172.21.15.27:0/3525360 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0e4159670 msgr2=0x7ff0e40c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.312+0000 7ff0e37fe700 1 --2- 172.21.15.27:0/3525360 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0e4159670 0x7ff0e40c6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.312+0000 7ff0e37fe700 1 -- 172.21.15.27:0/3525360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0e406a700 msgr2=0x7ff0e40b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.312+0000 7ff0e37fe700 1 --2- 172.21.15.27:0/3525360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0e406a700 0x7ff0e40b1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.313+0000 7ff0e37fe700 1 -- 172.21.15.27:0/3525360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0dc009a20 con 0x7ff0e4159a40 2024-01-11T08:10:36.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.313+0000 7ff0e37fe700 1 --2- 172.21.15.27:0/3525360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e4159a40 0x7ff0e4157780 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc013000 tx=0x7ff0dc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:36.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.313+0000 7ff0caffd700 1 -- 172.21.15.27:0/3525360 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc01b420 con 0x7ff0e4159a40 2024-01-11T08:10:36.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.313+0000 7ff0caffd700 1 -- 172.21.15.27:0/3525360 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0dc01b580 con 0x7ff0e4159a40 2024-01-11T08:10:36.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.313+0000 7ff0caffd700 1 -- 172.21.15.27:0/3525360 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc01b920 con 0x7ff0e4159a40 2024-01-11T08:10:36.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.313+0000 7ff0e95ff700 1 -- 172.21.15.27:0/3525360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e40b0b20 con 0x7ff0e4159a40 2024-01-11T08:10:36.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.313+0000 7ff0e95ff700 1 -- 172.21.15.27:0/3525360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e40b1b40 con 0x7ff0e4159a40 2024-01-11T08:10:36.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.314+0000 7ff0e95ff700 1 -- 172.21.15.27:0/3525360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e40b12e0 con 0x7ff0e4159a40 2024-01-11T08:10:36.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.315+0000 7ff0caffd700 1 -- 172.21.15.27:0/3525360 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff0dc01bc90 con 0x7ff0e4159a40 2024-01-11T08:10:36.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.316+0000 7ff0caffd700 1 --2- 172.21.15.27:0/3525360 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff0d0066f30 0x7ff0d00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:36.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.316+0000 7ff0caffd700 1 -- 172.21.15.27:0/3525360 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7ff0dc0863e0 con 0x7ff0e4159a40 2024-01-11T08:10:36.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.317+0000 7ff0e2ffd700 1 --2- 172.21.15.27:0/3525360 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff0d0066f30 0x7ff0d00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:36.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.317+0000 7ff0e2ffd700 1 --2- 172.21.15.27:0/3525360 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff0d0066f30 0x7ff0d00693e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4001360 tx=0x7ff0d4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:36.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.321+0000 7ff0caffd700 1 -- 172.21.15.27:0/3525360 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff0dc0535b0 con 0x7ff0e4159a40 2024-01-11T08:10:36.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.523+0000 7ff0e95ff700 1 -- 172.21.15.27:0/3525360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0e40718d0 con 0x7ff0e4159a40 2024-01-11T08:10:36.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.524+0000 7ff0caffd700 1 -- 172.21.15.27:0/3525360 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15302 (secure 0 0 0) 0x7ff0dc009880 con 0x7ff0e4159a40 2024-01-11T08:10:36.525 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:36.525 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:31.464323+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:10:27.993251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-11T08:10:29.153959+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:10:36.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.526+0000 7ff0e95ff700 1 -- 172.21.15.27:0/3525360 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff0d0066f30 msgr2=0x7ff0d00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:36.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.526+0000 7ff0e95ff700 1 --2- 172.21.15.27:0/3525360 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff0d0066f30 0x7ff0d00693e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4001360 tx=0x7ff0d4009040 comp rx=0 tx=0).stop 2024-01-11T08:10:36.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.526+0000 7ff0e95ff700 1 -- 172.21.15.27:0/3525360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e4159a40 msgr2=0x7ff0e4157780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:36.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.526+0000 7ff0e95ff700 1 --2- 172.21.15.27:0/3525360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e4159a40 0x7ff0e4157780 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc013000 tx=0x7ff0dc019000 comp rx=0 tx=0).stop 2024-01-11T08:10:36.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.527+0000 7ff0e95ff700 1 -- 172.21.15.27:0/3525360 shutdown_connections 2024-01-11T08:10:36.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.527+0000 7ff0e95ff700 1 --2- 172.21.15.27:0/3525360 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff0d0066f30 0x7ff0d00693e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.527+0000 7ff0e95ff700 1 --2- 172.21.15.27:0/3525360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e4159a40 0x7ff0e4157780 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.527+0000 7ff0e95ff700 1 --2- 172.21.15.27:0/3525360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0e406a700 0x7ff0e40b1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.527+0000 7ff0e95ff700 1 --2- 172.21.15.27:0/3525360 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0e4159670 0x7ff0e40c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.527+0000 7ff0e95ff700 1 -- 172.21.15.27:0/3525360 >> 172.21.15.27:0/3525360 conn(0x7ff0e40cd2b0 msgr2=0x7ff0e41586b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:36.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.527+0000 7ff0e95ff700 1 -- 172.21.15.27:0/3525360 shutdown_connections 2024-01-11T08:10:36.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.527+0000 7ff0e95ff700 1 -- 172.21.15.27:0/3525360 wait complete. 2024-01-11T08:10:36.546 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-11T08:10:36.546 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:10:36.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.745+0000 7f96b61ff700 1 Processor -- start 2024-01-11T08:10:36.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.745+0000 7f96b61ff700 1 -- start start 2024-01-11T08:10:36.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.746+0000 7f96b61ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96b00b03e0 0x7f96b00b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:36.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.746+0000 7f96b61ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f96b00b07b0 0x7f96b00b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.747+0000 7f96b61ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b00cd730 con 0x7f96b0159200 2024-01-11T08:10:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.747+0000 7f96b61ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96b0159d50 con 0x7f96b00b07b0 2024-01-11T08:10:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.747+0000 7f96b61ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b00ccb20 con 0x7f96b00b03e0 2024-01-11T08:10:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.747+0000 7f96aeffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f96b00b07b0 0x7f96b00b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.747+0000 7f96aeffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f96b00b07b0 0x7f96b00b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53170/0 (socket says 172.21.15.27:53170) 2024-01-11T08:10:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.747+0000 7f96aeffd700 1 -- 172.21.15.27:0/2893720283 learned_addr learned my addr 172.21.15.27:0/2893720283 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.747+0000 7f96affff700 1 --1- 172.21.15.27:0/2893720283 >> v1:172.21.15.27:6789/0 conn(0x7f96b0159200 0x7f96b0159640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37128/0 (socket says 172.21.15.27:37128) 2024-01-11T08:10:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.747+0000 7f96af7fe700 1 --2- 172.21.15.27:0/2893720283 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96b00b03e0 0x7f96b00b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.747+0000 7f96aeffd700 1 -- 172.21.15.27:0/2893720283 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96b00b03e0 msgr2=0x7f96b00b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.747+0000 7f96aeffd700 1 --2- 172.21.15.27:0/2893720283 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96b00b03e0 0x7f96b00b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.747+0000 7f96aeffd700 1 -- 172.21.15.27:0/2893720283 >> v1:172.21.15.27:6789/0 conn(0x7f96b0159200 legacy=0x7f96b0159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.747+0000 7f96aeffd700 1 -- 172.21.15.27:0/2893720283 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b00cce00 con 0x7f96b00b07b0 2024-01-11T08:10:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.747+0000 7f96ae7fc700 1 -- 172.21.15.27:0/2893720283 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650116675 0 0) 0x7f96b00cd730 con 0x7f96b0159200 2024-01-11T08:10:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.748+0000 7f96aeffd700 1 --2- 172.21.15.27:0/2893720283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b07b0 0x7f96b00b29c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f96a40099b0 tx=0x7f96a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50978f485a22ddb9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.748+0000 7f96ae7fc700 1 -- 172.21.15.27:0/2893720283 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a4009ea0 con 0x7f96b00b07b0 2024-01-11T08:10:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.748+0000 7f96ae7fc700 1 -- 172.21.15.27:0/2893720283 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a400b860 con 0x7f96b00b07b0 2024-01-11T08:10:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.748+0000 7f96ae7fc700 1 -- 172.21.15.27:0/2893720283 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a400bc00 con 0x7f96b00b07b0 2024-01-11T08:10:36.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.748+0000 7f96b61ff700 1 -- 172.21.15.27:0/2893720283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b07b0 msgr2=0x7f96b00b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:36.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.748+0000 7f96b61ff700 1 --2- 172.21.15.27:0/2893720283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b07b0 0x7f96b00b29c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f96a40099b0 tx=0x7f96a4009250 comp rx=0 tx=0).stop 2024-01-11T08:10:36.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.748+0000 7f96b61ff700 1 -- 172.21.15.27:0/2893720283 shutdown_connections 2024-01-11T08:10:36.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.748+0000 7f96b61ff700 1 --2- 172.21.15.27:0/2893720283 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96b00b03e0 0x7f96b00b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.749+0000 7f96b61ff700 1 --2- 172.21.15.27:0/2893720283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b07b0 0x7f96b00b29c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.749+0000 7f96b61ff700 1 -- 172.21.15.27:0/2893720283 >> 172.21.15.27:0/2893720283 conn(0x7f96b006ed40 msgr2=0x7f96b006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:36.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.749+0000 7f96b61ff700 1 -- 172.21.15.27:0/2893720283 shutdown_connections 2024-01-11T08:10:36.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.749+0000 7f96b61ff700 1 -- 172.21.15.27:0/2893720283 wait complete. 2024-01-11T08:10:36.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.750+0000 7f96b61ff700 1 Processor -- start 2024-01-11T08:10:36.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.750+0000 7f96b61ff700 1 -- start start 2024-01-11T08:10:36.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.751+0000 7f96b61ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b03e0 0x7f96b00cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:36.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.751+0000 7f96b61ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96b00b07b0 0x7f96b00a7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:36.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.751+0000 7f96b61ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b0159200 0x7f96b0157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:36.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.751+0000 7f96b61ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b00c75c0 con 0x7f96b00b03e0 2024-01-11T08:10:36.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.751+0000 7f96b61ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b00c7810 con 0x7f96b0159200 2024-01-11T08:10:36.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.751+0000 7f96b61ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b00c7af0 con 0x7f96b00b07b0 2024-01-11T08:10:36.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.751+0000 7f96af7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b03e0 0x7f96b00cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:36.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.751+0000 7f96af7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b03e0 0x7f96b00cbf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53184/0 (socket says 172.21.15.27:53184) 2024-01-11T08:10:36.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.751+0000 7f96af7fe700 1 -- 172.21.15.27:0/3723039058 learned_addr learned my addr 172.21.15.27:0/3723039058 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:36.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.752+0000 7f96aeffd700 1 --2- 172.21.15.27:0/3723039058 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96b00b07b0 0x7f96b00a7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:36.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.752+0000 7f96affff700 1 --2- 172.21.15.27:0/3723039058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b0159200 0x7f96b0157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:36.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.752+0000 7f96af7fe700 1 -- 172.21.15.27:0/3723039058 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96b00b07b0 msgr2=0x7f96b00a7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:36.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.752+0000 7f96af7fe700 1 --2- 172.21.15.27:0/3723039058 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96b00b07b0 0x7f96b00a7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.752+0000 7f96af7fe700 1 -- 172.21.15.27:0/3723039058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b0159200 msgr2=0x7f96b0157740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:36.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.752+0000 7f96af7fe700 1 --2- 172.21.15.27:0/3723039058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b0159200 0x7f96b0157740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.752+0000 7f96af7fe700 1 -- 172.21.15.27:0/3723039058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a40094f0 con 0x7f96b00b03e0 2024-01-11T08:10:36.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.752+0000 7f96affff700 1 --2- 172.21.15.27:0/3723039058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b0159200 0x7f96b0157740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:36.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.752+0000 7f96af7fe700 1 --2- 172.21.15.27:0/3723039058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b03e0 0x7f96b00cbf40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f96a000ec80 tx=0x7f96a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:36.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.752+0000 7f96977fe700 1 -- 172.21.15.27:0/3723039058 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a000c9d0 con 0x7f96b00b03e0 2024-01-11T08:10:36.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.752+0000 7f96977fe700 1 -- 172.21.15.27:0/3723039058 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a000cb30 con 0x7f96b00b03e0 2024-01-11T08:10:36.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.752+0000 7f96b61ff700 1 -- 172.21.15.27:0/3723039058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b00c7dd0 con 0x7f96b00b03e0 2024-01-11T08:10:36.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.752+0000 7f96977fe700 1 -- 172.21.15.27:0/3723039058 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a0013470 con 0x7f96b00b03e0 2024-01-11T08:10:36.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.752+0000 7f96b61ff700 1 -- 172.21.15.27:0/3723039058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b00c8390 con 0x7f96b00b03e0 2024-01-11T08:10:36.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.754+0000 7f96b61ff700 1 -- 172.21.15.27:0/3723039058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b0169bc0 con 0x7f96b00b03e0 2024-01-11T08:10:36.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.754+0000 7f96977fe700 1 -- 172.21.15.27:0/3723039058 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f96a0013820 con 0x7f96b00b03e0 2024-01-11T08:10:36.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.755+0000 7f96977fe700 1 --2- 172.21.15.27:0/3723039058 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f969c066f30 0x7f969c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:36.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.755+0000 7f96977fe700 1 -- 172.21.15.27:0/3723039058 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f96a0084920 con 0x7f96b00b03e0 2024-01-11T08:10:36.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.756+0000 7f96aeffd700 1 --2- 172.21.15.27:0/3723039058 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f969c066f30 0x7f969c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:36.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.756+0000 7f96aeffd700 1 --2- 172.21.15.27:0/3723039058 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f969c066f30 0x7f969c0693e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f96a40099b0 tx=0x7f96a4017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:36.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.759+0000 7f96977fe700 1 -- 172.21.15.27:0/3723039058 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f96a0051b90 con 0x7f96b00b03e0 2024-01-11T08:10:36.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.961+0000 7f96b61ff700 1 -- 172.21.15.27:0/3723039058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96b00718d0 con 0x7f96b00b03e0 2024-01-11T08:10:36.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.962+0000 7f96977fe700 1 -- 172.21.15.27:0/3723039058 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15302 (secure 0 0 0) 0x7f96a00551b0 con 0x7f96b00b03e0 2024-01-11T08:10:36.963 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:36.963 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:31.464323+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:10:27.993251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-11T08:10:29.153959+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:10:36.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.965+0000 7f96b61ff700 1 -- 172.21.15.27:0/3723039058 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f969c066f30 msgr2=0x7f969c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:36.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.965+0000 7f96b61ff700 1 --2- 172.21.15.27:0/3723039058 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f969c066f30 0x7f969c0693e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f96a40099b0 tx=0x7f96a4017000 comp rx=0 tx=0).stop 2024-01-11T08:10:36.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.965+0000 7f96b61ff700 1 -- 172.21.15.27:0/3723039058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b03e0 msgr2=0x7f96b00cbf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:36.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.965+0000 7f96b61ff700 1 --2- 172.21.15.27:0/3723039058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b03e0 0x7f96b00cbf40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f96a000ec80 tx=0x7f96a00091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.965+0000 7f96b61ff700 1 -- 172.21.15.27:0/3723039058 shutdown_connections 2024-01-11T08:10:36.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.965+0000 7f96b61ff700 1 --2- 172.21.15.27:0/3723039058 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f969c066f30 0x7f969c0693e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.965+0000 7f96b61ff700 1 --2- 172.21.15.27:0/3723039058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b0159200 0x7f96b0157740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.965+0000 7f96b61ff700 1 --2- 172.21.15.27:0/3723039058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b03e0 0x7f96b00cbf40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.965+0000 7f96b61ff700 1 --2- 172.21.15.27:0/3723039058 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96b00b07b0 0x7f96b00a7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:36.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.965+0000 7f96b61ff700 1 -- 172.21.15.27:0/3723039058 >> 172.21.15.27:0/3723039058 conn(0x7f96b006ed40 msgr2=0x7f96b0159640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:36.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.965+0000 7f96b61ff700 1 -- 172.21.15.27:0/3723039058 shutdown_connections 2024-01-11T08:10:36.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:36.965+0000 7f96b61ff700 1 -- 172.21.15.27:0/3723039058 wait complete. 2024-01-11T08:10:36.984 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-11T08:10:36.985 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-11T08:10:36.985 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-11T08:10:36.985 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-01-11T08:10:36.986 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-01-11T08:10:36.986 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-01-11T08:10:36.986 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-01-11T08:10:36.987 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-01-11T08:10:37.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.355+0000 7fa5449ff700 1 Processor -- start 2024-01-11T08:10:37.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.355+0000 7fa5449ff700 1 -- start start 2024-01-11T08:10:37.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.357+0000 7fa5449ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa540069a40 0x7fa5400cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.357+0000 7fa5449ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5400ae800 con 0x7fa5401571d0 2024-01-11T08:10:37.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.357+0000 7fa5449ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5400adbf0 con 0x7fa5400ad730 2024-01-11T08:10:37.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.357+0000 7fa5449ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5400ccb90 con 0x7fa540069a40 2024-01-11T08:10:37.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.359+0000 7fa53e59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa5401571d0 0x7fa540069330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37134/0 (socket says 172.21.15.27:37134) 2024-01-11T08:10:37.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.359+0000 7fa53e59c700 1 -- 172.21.15.27:0/137297853 learned_addr learned my addr 172.21.15.27:0/137297853 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:37.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.360+0000 7f60071ff700 1 Processor -- start 2024-01-11T08:10:37.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.360+0000 7fa53d59a700 1 -- 172.21.15.27:0/137297853 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826881538 0 0) 0x7fa5400ae800 con 0x7fa5401571d0 2024-01-11T08:10:37.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.360+0000 7fa53d59a700 1 -- 172.21.15.27:0/137297853 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa528003570 con 0x7fa5401571d0 2024-01-11T08:10:37.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.361+0000 7fa53d59a700 1 -- 172.21.15.27:0/137297853 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2859504814 0 0) 0x7fa528003570 con 0x7fa5401571d0 2024-01-11T08:10:37.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.361+0000 7fa53d59a700 1 -- 172.21.15.27:0/137297853 >> v2:172.21.15.27:3300/0 conn(0x7fa540069a40 msgr2=0x7fa5400cc650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-11T08:10:37.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.360+0000 7f60071ff700 1 -- start start 2024-01-11T08:10:37.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.361+0000 7fb296b9d700 1 Processor -- start 2024-01-11T08:10:37.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.361+0000 7fa53d59a700 1 --2- 172.21.15.27:0/137297853 >> v2:172.21.15.27:3300/0 conn(0x7fa540069a40 0x7fa5400cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.362+0000 7fb296b9d700 1 -- start start 2024-01-11T08:10:37.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.362+0000 7fa53d59a700 1 -- 172.21.15.27:0/137297853 >> v1:172.21.15.77:6789/0 conn(0x7fa5400ad730 legacy=0x7fa54006d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.362+0000 7fa53d59a700 1 -- 172.21.15.27:0/137297853 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5400aded0 con 0x7fa5401571d0 2024-01-11T08:10:37.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.362+0000 7fa53d59a700 1 -- 172.21.15.27:0/137297853 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fa538001650 con 0x7fa5401571d0 2024-01-11T08:10:37.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.362+0000 7fa53d59a700 1 -- 172.21.15.27:0/137297853 >> v1:172.21.15.27:6789/0 conn(0x7fa5401571d0 legacy=0x7fa540069330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.362+0000 7fa53d59a700 1 --2- 172.21.15.27:0/137297853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa528003ae0 0x7fa528003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.362+0000 7fa53d59a700 1 --2- 172.21.15.27:0/137297853 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa528004810 0x7fa528004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.362+0000 7fa53d59a700 1 --2- 172.21.15.27:0/137297853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5280053c0 0x7fa52800ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.362+0000 7fa53d59a700 1 -- 172.21.15.27:0/137297853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400ccb90 con 0x7fa528003ae0 2024-01-11T08:10:37.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.362+0000 7fa53d59a700 1 -- 172.21.15.27:0/137297853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5280106d0 con 0x7fa5280053c0 2024-01-11T08:10:37.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.362+0000 7fa53d59a700 1 -- 172.21.15.27:0/137297853 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa528010890 con 0x7fa528004810 2024-01-11T08:10:37.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.363+0000 7ff0b49ff700 1 Processor -- start 2024-01-11T08:10:37.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.363+0000 7ff0b49ff700 1 -- start start 2024-01-11T08:10:37.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.364+0000 7fa53dd9b700 1 --2- 172.21.15.27:0/137297853 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa528004810 0x7fa528004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.364+0000 7fb296b9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb288033ac0 0x7fb288033e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.364+0000 7fa53dd9b700 1 -- 172.21.15.27:0/137297853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5280053c0 msgr2=0x7fa52800ba80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-11T08:10:37.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.364+0000 7fa53dd9b700 1 --2- 172.21.15.27:0/137297853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5280053c0 0x7fa52800ba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.364+0000 7fa53dd9b700 1 -- 172.21.15.27:0/137297853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa528003ae0 msgr2=0x7fa528003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.364+0000 7fa53dd9b700 1 --2- 172.21.15.27:0/137297853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa528003ae0 0x7fa528003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.365+0000 7fa53dd9b700 1 -- 172.21.15.27:0/137297853 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5400aded0 con 0x7fa528004810 2024-01-11T08:10:37.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.364+0000 7fb296b9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb2880352d0 0x7fb288005f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.364+0000 7fb296b9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb2880dfaf0 0x7fb2880dff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.364+0000 7fb296b9d700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb288006450 con 0x7fb288033ac0 2024-01-11T08:10:37.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.364+0000 7fb296b9d700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb288007270 con 0x7fb2880352d0 2024-01-11T08:10:37.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.364+0000 7fb296b9d700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb288035770 con 0x7fb2880dfaf0 2024-01-11T08:10:37.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.365+0000 7fa53ed9d700 1 -- 172.21.15.27:0/137297853 reap_dead start 2024-01-11T08:10:37.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.365+0000 7f87a0bff700 1 Processor -- start 2024-01-11T08:10:37.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.366+0000 7fa53dd9b700 1 --2- 172.21.15.27:0/137297853 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa528004810 0x7fa528004c80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa5300027c0 tx=0x7fa530007d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3791da4532db64e4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.365+0000 7fb29539a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb2880352d0 0x7fb288005f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.365+0000 7fb29539a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb2880352d0 0x7fb288005f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:58422/0 (socket says 172.21.15.27:58422) 2024-01-11T08:10:37.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.365+0000 7fb29539a700 1 -- 172.21.15.27:0/1470317892 learned_addr learned my addr 172.21.15.27:0/1470317892 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.365+0000 7fb29639c700 1 --2- 172.21.15.27:0/1470317892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb2880dfaf0 0x7fb2880dff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.365+0000 7ff0b49ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0b00ad840 0x7ff0b00cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.366+0000 7ff0b49ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b0157a00 con 0x7ff0b00b19a0 2024-01-11T08:10:37.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.366+0000 7ff0b49ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b0157c50 con 0x7ff0b00adc10 2024-01-11T08:10:37.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.366+0000 7ff0b49ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b0157f30 con 0x7ff0b00ad840 2024-01-11T08:10:37.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.366+0000 7f60071ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f60000d06b0 0x7f60000ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.367+0000 7f60071ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f600015dd70 0x7f60000b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.367+0000 7f60071ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60000cf660 con 0x7f60000d0a80 2024-01-11T08:10:37.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.367+0000 7f60071ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60000cd650 con 0x7f60000d06b0 2024-01-11T08:10:37.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.367+0000 7f60071ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60000ced30 con 0x7f600015dd70 2024-01-11T08:10:37.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.367+0000 7f6004f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f60000d06b0 0x7f60000ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.366+0000 7fb295b9b700 1 --2- 172.21.15.27:0/1470317892 >> v2:172.21.15.27:3300/0 conn(0x7fb288033ac0 0x7fb288033e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.367+0000 7fb29539a700 1 -- 172.21.15.27:0/1470317892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb2880dfaf0 msgr2=0x7fb2880dff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.367+0000 7fb29539a700 1 --2- 172.21.15.27:0/1470317892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb2880dfaf0 0x7fb2880dff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.367+0000 7fb29539a700 1 -- 172.21.15.27:0/1470317892 >> v2:172.21.15.27:3300/0 conn(0x7fb288033ac0 msgr2=0x7fb288033e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.367+0000 7fb29539a700 1 --2- 172.21.15.27:0/1470317892 >> v2:172.21.15.27:3300/0 conn(0x7fb288033ac0 0x7fb288033e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.367+0000 7fb29539a700 1 -- 172.21.15.27:0/1470317892 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb288006660 con 0x7fb2880352d0 2024-01-11T08:10:37.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.367+0000 7fb295b9b700 1 --2- 172.21.15.27:0/1470317892 >> v2:172.21.15.27:3300/0 conn(0x7fb288033ac0 0x7fb288033e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:37.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.367+0000 7f6004f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f60000d06b0 0x7f60000ccf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53210/0 (socket says 172.21.15.27:53210) 2024-01-11T08:10:37.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.367+0000 7f6004f9b700 1 -- 172.21.15.27:0/580068377 learned_addr learned my addr 172.21.15.27:0/580068377 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.368+0000 7fa53d59a700 1 -- 172.21.15.27:0/137297853 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530006820 con 0x7fa528004810 2024-01-11T08:10:37.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.367+0000 7ff0ae59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0b00ad840 0x7ff0b00cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.367+0000 7ff0ae59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0b00ad840 0x7ff0b00cc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:59512/0 (socket says 172.21.15.27:59512) 2024-01-11T08:10:37.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.367+0000 7ff0ae59c700 1 -- 172.21.15.27:0/2725117461 learned_addr learned my addr 172.21.15.27:0/2725117461 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.368+0000 7fa53d59a700 1 -- 172.21.15.27:0/137297853 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa530006980 con 0x7fa528004810 2024-01-11T08:10:37.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.368+0000 7fa53d59a700 1 -- 172.21.15.27:0/137297853 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530006d40 con 0x7fa528004810 2024-01-11T08:10:37.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.366+0000 7f87a0bff700 1 -- start start 2024-01-11T08:10:37.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.368+0000 7fb29539a700 1 --2- 172.21.15.27:0/1470317892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2880352d0 0x7fb288005f10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb2880e4dc0 tx=0x7fb280009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ebb8d5d1b8e792d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.368+0000 7fb294b99700 1 -- 172.21.15.27:0/1470317892 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28000b640 con 0x7fb2880352d0 2024-01-11T08:10:37.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.368+0000 7fb294b99700 1 -- 172.21.15.27:0/1470317892 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb28000b7a0 con 0x7fb2880352d0 2024-01-11T08:10:37.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.368+0000 7fb294b99700 1 -- 172.21.15.27:0/1470317892 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28000bb60 con 0x7fb2880352d0 2024-01-11T08:10:37.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.367+0000 7ff0add9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff0b00adc10 0x7ff0b00b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:60726/0 (socket says 172.21.15.27:60726) 2024-01-11T08:10:37.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.368+0000 7f600579c700 1 --2- 172.21.15.27:0/580068377 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f600015dd70 0x7f60000b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.369+0000 7f87a0bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f87940e0aa0 0x7f87940ea1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.369+0000 7fa5449ff700 1 -- 172.21.15.27:0/137297853 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa528004810 msgr2=0x7fa528004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.369+0000 7f879a59c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f87940e0aa0 0x7f87940ea1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.369+0000 7f6004f9b700 1 -- 172.21.15.27:0/580068377 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f600015dd70 msgr2=0x7f60000b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.369+0000 7f6004f9b700 1 --2- 172.21.15.27:0/580068377 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f600015dd70 0x7f60000b3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.369+0000 7f879a59c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f87940e0aa0 0x7f87940ea1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:58428/0 (socket says 172.21.15.27:58428) 2024-01-11T08:10:37.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.370+0000 7f879a59c700 1 -- 172.21.15.27:0/3511724944 learned_addr learned my addr 172.21.15.27:0/3511724944 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.369+0000 7f8799d9b700 1 --1- 172.21.15.27:0/3511724944 >> v1:172.21.15.77:6789/0 conn(0x7f87940e0e70 0x7f87940067a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:60744/0 (socket says 172.21.15.27:60744) 2024-01-11T08:10:37.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.369+0000 7ff0ad59a700 1 -- 172.21.15.27:0/2725117461 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201424022 0 0) 0x7ff0b0157a00 con 0x7ff0b00b19a0 2024-01-11T08:10:37.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.370+0000 7f87a0bff700 1 --2- 172.21.15.27:0/3511724944 >> v2:172.21.15.27:3300/0 conn(0x7f8794006eb0 0x7f87940ea6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.369+0000 7fa5449ff700 1 --2- 172.21.15.27:0/137297853 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa528004810 0x7fa528004c80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa5300027c0 tx=0x7fa530007d30 comp rx=0 tx=0).stop 2024-01-11T08:10:37.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.370+0000 7f87a0bff700 1 -- 172.21.15.27:0/3511724944 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87940375f0 con 0x7f87940e0e70 2024-01-11T08:10:37.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.370+0000 7ff0ad59a700 1 -- 172.21.15.27:0/2725117461 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff09c003570 con 0x7ff0b00b19a0 2024-01-11T08:10:37.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.370+0000 7ff0ad59a700 1 -- 172.21.15.27:0/2725117461 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051214011 0 0) 0x7ff0b0157c50 con 0x7ff0b00adc10 2024-01-11T08:10:37.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.370+0000 7ff0ad59a700 1 -- 172.21.15.27:0/2725117461 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0b0157a00 con 0x7ff0b00adc10 2024-01-11T08:10:37.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.370+0000 7f87a0bff700 1 -- 172.21.15.27:0/3511724944 --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87940eb040 con 0x7f8794006eb0 2024-01-11T08:10:37.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.370+0000 7f87a0bff700 1 -- 172.21.15.27:0/3511724944 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87940534d0 con 0x7f87940e0aa0 2024-01-11T08:10:37.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.370+0000 7fa5449ff700 1 -- 172.21.15.27:0/137297853 shutdown_connections 2024-01-11T08:10:37.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.370+0000 7f879ad9d700 1 --2- 172.21.15.27:0/3511724944 >> v2:172.21.15.27:3300/0 conn(0x7f8794006eb0 0x7f87940ea6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.370+0000 7fa5449ff700 1 --2- 172.21.15.27:0/137297853 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa528004810 0x7fa528004c80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.370+0000 7fa5449ff700 1 -- 172.21.15.27:0/137297853 >> 172.21.15.27:0/137297853 conn(0x7fa5400680a0 msgr2=0x7fa5400cbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.370+0000 7f879959a700 1 -- 172.21.15.27:0/3511724944 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054757741 0 0) 0x7f8790001110 con 0x7f87940e0e70 2024-01-11T08:10:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7fa5449ff700 1 -- 172.21.15.27:0/137297853 shutdown_connections 2024-01-11T08:10:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7f879959a700 1 -- 172.21.15.27:0/3511724944 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8784003570 con 0x7f87940e0e70 2024-01-11T08:10:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7f879959a700 1 -- 172.21.15.27:0/3511724944 <== mon.1 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2802713352 0 0) 0x7f8784003570 con 0x7f87940e0e70 2024-01-11T08:10:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7f879959a700 1 -- 172.21.15.27:0/3511724944 >> v2:172.21.15.77:3300/0 conn(0x7f87940e0aa0 msgr2=0x7f87940ea1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7f879959a700 1 --2- 172.21.15.27:0/3511724944 >> v2:172.21.15.77:3300/0 conn(0x7f87940e0aa0 0x7f87940ea1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7f879959a700 1 -- 172.21.15.27:0/3511724944 >> v2:172.21.15.27:3300/0 conn(0x7f8794006eb0 msgr2=0x7f87940ea6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7f879959a700 1 --2- 172.21.15.27:0/3511724944 >> v2:172.21.15.27:3300/0 conn(0x7f8794006eb0 0x7f87940ea6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7f879959a700 1 -- 172.21.15.27:0/3511724944 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8794054970 con 0x7f87940e0e70 2024-01-11T08:10:37.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7f879959a700 1 -- 172.21.15.27:0/3511724944 <== mon.1 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f8790001600 con 0x7f87940e0e70 2024-01-11T08:10:37.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7f879959a700 1 -- 172.21.15.27:0/3511724944 >> v1:172.21.15.77:6789/0 conn(0x7f87940e0e70 legacy=0x7f87940067a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7ff0ae59c700 1 -- 172.21.15.27:0/2725117461 >> v1:172.21.15.77:6789/0 conn(0x7ff0b00adc10 legacy=0x7ff0b00b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7ff0ae59c700 1 -- 172.21.15.27:0/2725117461 >> v1:172.21.15.27:6789/0 conn(0x7ff0b00b19a0 legacy=0x7ff0b00aba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7ff0ae59c700 1 -- 172.21.15.27:0/2725117461 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b0158210 con 0x7ff0b00ad840 2024-01-11T08:10:37.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7f879959a700 1 --2- 172.21.15.27:0/3511724944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8784003a60 0x7f8784003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7f879959a700 1 --2- 172.21.15.27:0/3511724944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8784004810 0x7f8784004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7f879959a700 1 --2- 172.21.15.27:0/3511724944 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87840051c0 0x7f878400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7f879959a700 1 -- 172.21.15.27:0/3511724944 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87940eb040 con 0x7f8784004810 2024-01-11T08:10:37.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7f879959a700 1 -- 172.21.15.27:0/3511724944 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87940534d0 con 0x7f8784003a60 2024-01-11T08:10:37.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7f879959a700 1 -- 172.21.15.27:0/3511724944 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f878400c080 con 0x7f87840051c0 2024-01-11T08:10:37.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.372+0000 7f879a59c700 1 --2- 172.21.15.27:0/3511724944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8784003a60 0x7f8784003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7ff0ae59c700 1 --2- 172.21.15.27:0/2725117461 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0b00ad840 0x7ff0b00cc8d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff0980099e0 tx=0x7ff098009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3adc05bf1a8b6788 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.372+0000 7f8799d9b700 1 --2- 172.21.15.27:0/3511724944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8784004810 0x7f8784004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.371+0000 7fa5449ff700 1 -- 172.21.15.27:0/137297853 wait complete. 2024-01-11T08:10:37.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.372+0000 7f879a59c700 1 -- 172.21.15.27:0/3511724944 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87840051c0 msgr2=0x7f878400ba60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-11T08:10:37.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.372+0000 7f879a59c700 1 --2- 172.21.15.27:0/3511724944 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87840051c0 0x7f878400ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.372+0000 7f879a59c700 1 -- 172.21.15.27:0/3511724944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8784004810 msgr2=0x7f8784004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.372+0000 7f879a59c700 1 --2- 172.21.15.27:0/3511724944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8784004810 0x7f8784004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.372+0000 7f879a59c700 1 -- 172.21.15.27:0/3511724944 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8784003570 con 0x7f8784003a60 2024-01-11T08:10:37.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.372+0000 7ff0ad59a700 1 -- 172.21.15.27:0/2725117461 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09800b640 con 0x7ff0b00ad840 2024-01-11T08:10:37.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.372+0000 7f5ffffff700 1 -- 172.21.15.27:0/580068377 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789352678 0 0) 0x7f60000cf660 con 0x7f60000d0a80 2024-01-11T08:10:37.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.372+0000 7f6004f9b700 1 -- 172.21.15.27:0/580068377 >> v1:172.21.15.77:6789/0 conn(0x7f60000d0a80 legacy=0x7f60000cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.373+0000 7f6004f9b700 1 -- 172.21.15.27:0/580068377 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60000cf010 con 0x7f60000d06b0 2024-01-11T08:10:37.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.372+0000 7fb296b9d700 1 -- 172.21.15.27:0/1470317892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2880352d0 msgr2=0x7fb288005f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.373+0000 7fb296b9d700 1 --2- 172.21.15.27:0/1470317892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2880352d0 0x7fb288005f10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb2880e4dc0 tx=0x7fb280009250 comp rx=0 tx=0).stop 2024-01-11T08:10:37.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.373+0000 7f6004f9b700 1 --2- 172.21.15.27:0/580068377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60000d06b0 0x7f60000ccf50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00099b0 tx=0x7f5ff0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f754efdcf54f3cb8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.374+0000 7f5ffffff700 1 -- 172.21.15.27:0/580068377 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0009ea0 con 0x7f60000d06b0 2024-01-11T08:10:37.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.374+0000 7fa5449ff700 1 Processor -- start 2024-01-11T08:10:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.373+0000 7fb296b9d700 1 -- 172.21.15.27:0/1470317892 shutdown_connections 2024-01-11T08:10:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.373+0000 7fb296b9d700 1 --2- 172.21.15.27:0/1470317892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb2880dfaf0 0x7fb2880dff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.373+0000 7fb296b9d700 1 --2- 172.21.15.27:0/1470317892 >> v2:172.21.15.27:3300/0 conn(0x7fb288033ac0 0x7fb288033e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.373+0000 7fb296b9d700 1 --2- 172.21.15.27:0/1470317892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2880352d0 0x7fb288005f10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.373+0000 7fb296b9d700 1 -- 172.21.15.27:0/1470317892 >> 172.21.15.27:0/1470317892 conn(0x7fb2880517b0 msgr2=0x7fb288051bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.373+0000 7fb296b9d700 1 -- 172.21.15.27:0/1470317892 shutdown_connections 2024-01-11T08:10:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.373+0000 7fb296b9d700 1 -- 172.21.15.27:0/1470317892 wait complete. 2024-01-11T08:10:37.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.373+0000 7f879a59c700 1 --2- 172.21.15.27:0/3511724944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8784003a60 0x7f8784003e70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f878c006fa0 tx=0x7f878c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f870dd5311e543f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.373+0000 7f879ad9d700 1 -- 172.21.15.27:0/3511724944 reap_dead start 2024-01-11T08:10:37.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.373+0000 7f879959a700 1 -- 172.21.15.27:0/3511724944 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c00e740 con 0x7f8784003a60 2024-01-11T08:10:37.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.373+0000 7f879959a700 1 -- 172.21.15.27:0/3511724944 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878c00e8a0 con 0x7f8784003a60 2024-01-11T08:10:37.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.373+0000 7f879959a700 1 -- 172.21.15.27:0/3511724944 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c00ec20 con 0x7f8784003a60 2024-01-11T08:10:37.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.373+0000 7ff0ad59a700 1 -- 172.21.15.27:0/2725117461 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09800b7a0 con 0x7ff0b00ad840 2024-01-11T08:10:37.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.373+0000 7ff0ad59a700 1 -- 172.21.15.27:0/2725117461 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09800bb40 con 0x7ff0b00ad840 2024-01-11T08:10:37.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.374+0000 7fa5449ff700 1 -- start start 2024-01-11T08:10:37.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.375+0000 7f5ffffff700 1 -- 172.21.15.27:0/580068377 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff000b860 con 0x7f60000d06b0 2024-01-11T08:10:37.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.375+0000 7f5ffffff700 1 -- 172.21.15.27:0/580068377 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0009ea0 con 0x7f60000d06b0 2024-01-11T08:10:37.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.375+0000 7f87a0bff700 1 -- 172.21.15.27:0/3511724944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8784003a60 msgr2=0x7f8784003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.375+0000 7f87a0bff700 1 --2- 172.21.15.27:0/3511724944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8784003a60 0x7f8784003e70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f878c006fa0 tx=0x7f878c009250 comp rx=0 tx=0).stop 2024-01-11T08:10:37.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.374+0000 7ff0b49ff700 1 -- 172.21.15.27:0/2725117461 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0b00ad840 msgr2=0x7ff0b00cc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.374+0000 7ff0b49ff700 1 --2- 172.21.15.27:0/2725117461 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0b00ad840 0x7ff0b00cc8d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff0980099e0 tx=0x7ff098009250 comp rx=0 tx=0).stop 2024-01-11T08:10:37.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.375+0000 7f87a0bff700 1 -- 172.21.15.27:0/3511724944 shutdown_connections 2024-01-11T08:10:37.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.375+0000 7f87a0bff700 1 --2- 172.21.15.27:0/3511724944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8784003a60 0x7f8784003e70 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.375+0000 7f87a0bff700 1 -- 172.21.15.27:0/3511724944 >> 172.21.15.27:0/3511724944 conn(0x7f8794054c20 msgr2=0x7f8794055020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.375+0000 7fb296b9d700 1 Processor -- start 2024-01-11T08:10:37.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.375+0000 7fa5449ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa528004810 0x7fa540157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7fb296b9d700 1 -- start start 2024-01-11T08:10:37.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7ff0b49ff700 1 -- 172.21.15.27:0/2725117461 shutdown_connections 2024-01-11T08:10:37.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7fa5449ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa540159e20 0x7fa5400c3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7fa5449ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5400c39b0 0x7fa5400c5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7ff0b49ff700 1 --2- 172.21.15.27:0/2725117461 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0b00ad840 0x7ff0b00cc8d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7ff0b49ff700 1 -- 172.21.15.27:0/2725117461 >> 172.21.15.27:0/2725117461 conn(0x7ff0b00680a0 msgr2=0x7ff0b00cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7f60071ff700 1 -- 172.21.15.27:0/580068377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60000d06b0 msgr2=0x7f60000ccf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7f60071ff700 1 --2- 172.21.15.27:0/580068377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60000d06b0 0x7f60000ccf50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00099b0 tx=0x7f5ff0009250 comp rx=0 tx=0).stop 2024-01-11T08:10:37.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7ff0b49ff700 1 -- 172.21.15.27:0/2725117461 shutdown_connections 2024-01-11T08:10:37.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7f60071ff700 1 -- 172.21.15.27:0/580068377 shutdown_connections 2024-01-11T08:10:37.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7fa5449ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400cb4c0 con 0x7fa5400c39b0 2024-01-11T08:10:37.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7fa5449ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54015aef0 con 0x7fa528004810 2024-01-11T08:10:37.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7fa5449ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa540165520 con 0x7fa540159e20 2024-01-11T08:10:37.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7fa53e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa528004810 0x7fa540157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7fa53dd9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa540159e20 0x7fa5400c3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7f60071ff700 1 --2- 172.21.15.27:0/580068377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60000d06b0 0x7f60000ccf50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7fa53dd9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa540159e20 0x7fa5400c3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:59536/0 (socket says 172.21.15.27:59536) 2024-01-11T08:10:37.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7fa53dd9b700 1 -- 172.21.15.27:0/1352661556 learned_addr learned my addr 172.21.15.27:0/1352661556 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7f60071ff700 1 --2- 172.21.15.27:0/580068377 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f600015dd70 0x7f60000b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.376+0000 7f60071ff700 1 -- 172.21.15.27:0/580068377 >> 172.21.15.27:0/580068377 conn(0x7f600006ed40 msgr2=0x7f600006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7f60071ff700 1 -- 172.21.15.27:0/580068377 shutdown_connections 2024-01-11T08:10:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7f87a0bff700 1 -- 172.21.15.27:0/3511724944 shutdown_connections 2024-01-11T08:10:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7ff0b49ff700 1 -- 172.21.15.27:0/2725117461 wait complete. 2024-01-11T08:10:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7f87a0bff700 1 -- 172.21.15.27:0/3511724944 wait complete. 2024-01-11T08:10:37.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7f60071ff700 1 -- 172.21.15.27:0/580068377 wait complete. 2024-01-11T08:10:37.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7fa53ed9d700 1 --2- 172.21.15.27:0/1352661556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5400c39b0 0x7fa5400c5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7fa53dd9b700 1 -- 172.21.15.27:0/1352661556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa528004810 msgr2=0x7fa540157f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7fa53dd9b700 1 --2- 172.21.15.27:0/1352661556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa528004810 0x7fa540157f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7fa53dd9b700 1 -- 172.21.15.27:0/1352661556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5400c39b0 msgr2=0x7fa5400c5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7fa53dd9b700 1 --2- 172.21.15.27:0/1352661556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5400c39b0 0x7fa5400c5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7fa53dd9b700 1 -- 172.21.15.27:0/1352661556 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa540165800 con 0x7fa540159e20 2024-01-11T08:10:37.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7fa53ed9d700 1 --2- 172.21.15.27:0/1352661556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5400c39b0 0x7fa5400c5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:37.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7fb296b9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb288033ac0 0x7fb28804ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7fb296b9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2880352d0 0x7fb28804da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7fb296b9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2880dfaf0 0x7fb2880f6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7fb296b9d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb288038960 con 0x7fb2880352d0 2024-01-11T08:10:37.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7fb296b9d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28804f1c0 con 0x7fb2880dfaf0 2024-01-11T08:10:37.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7fb296b9d700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb28804efb0 con 0x7fb288033ac0 2024-01-11T08:10:37.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7fb295b9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb288033ac0 0x7fb28804ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7fb295b9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb288033ac0 0x7fb28804ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:59538/0 (socket says 172.21.15.27:59538) 2024-01-11T08:10:37.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.377+0000 7fb295b9b700 1 -- 172.21.15.27:0/3107442394 learned_addr learned my addr 172.21.15.27:0/3107442394 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.378+0000 7fb29639c700 1 --2- 172.21.15.27:0/3107442394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2880dfaf0 0x7fb2880f6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.378+0000 7fa53dd9b700 1 --2- 172.21.15.27:0/1352661556 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa540159e20 0x7fa5400c3470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa530002db0 tx=0x7fa530015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.378+0000 7fb29539a700 1 --2- 172.21.15.27:0/3107442394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2880352d0 0x7fb28804da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.379+0000 7fb29639c700 1 -- 172.21.15.27:0/3107442394 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb288033ac0 msgr2=0x7fb28804ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.379+0000 7ff0b49ff700 1 Processor -- start 2024-01-11T08:10:37.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.379+0000 7fa52e7fc700 1 -- 172.21.15.27:0/1352661556 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530013860 con 0x7fa540159e20 2024-01-11T08:10:37.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.379+0000 7fb29639c700 1 --2- 172.21.15.27:0/3107442394 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb288033ac0 0x7fb28804ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.379+0000 7fb29639c700 1 -- 172.21.15.27:0/3107442394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2880352d0 msgr2=0x7fb28804da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.379+0000 7fa5449ff700 1 -- 172.21.15.27:0/1352661556 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa540165ae0 con 0x7fa540159e20 2024-01-11T08:10:37.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.379+0000 7fa5449ff700 1 -- 172.21.15.27:0/1352661556 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa540158440 con 0x7fa540159e20 2024-01-11T08:10:37.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.379+0000 7fb29639c700 1 --2- 172.21.15.27:0/3107442394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2880352d0 0x7fb28804da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.379+0000 7fb29639c700 1 -- 172.21.15.27:0/3107442394 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb28c0093c0 con 0x7fb2880dfaf0 2024-01-11T08:10:37.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.380+0000 7f87a0bff700 1 Processor -- start 2024-01-11T08:10:37.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.380+0000 7f87a0bff700 1 -- start start 2024-01-11T08:10:37.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.380+0000 7f60071ff700 1 Processor -- start 2024-01-11T08:10:37.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.380+0000 7f60071ff700 1 -- start start 2024-01-11T08:10:37.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.381+0000 7fb29639c700 1 --2- 172.21.15.27:0/3107442394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2880dfaf0 0x7fb2880f6ee0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb29004d1b0 tx=0x7fb2900668e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.379+0000 7ff0b49ff700 1 -- start start 2024-01-11T08:10:37.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.381+0000 7ff0b49ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0b00ad840 0x7ff0b0146bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.381+0000 7ff0b49ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0b00adc10 0x7ff0b006f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.381+0000 7ff0b49ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0b00b19a0 0x7ff0b014b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.381+0000 7ff0b49ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b0147190 con 0x7ff0b00ad840 2024-01-11T08:10:37.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.381+0000 7ff0b49ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b0147470 con 0x7ff0b00b19a0 2024-01-11T08:10:37.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.381+0000 7ff0b49ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b0147750 con 0x7ff0b00adc10 2024-01-11T08:10:37.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.380+0000 7fa52e7fc700 1 -- 172.21.15.27:0/1352661556 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5300139c0 con 0x7fa540159e20 2024-01-11T08:10:37.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.380+0000 7fa52e7fc700 1 -- 172.21.15.27:0/1352661556 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530013860 con 0x7fa540159e20 2024-01-11T08:10:37.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.381+0000 7fb285ffb700 1 -- 172.21.15.27:0/3107442394 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2900649c0 con 0x7fb2880dfaf0 2024-01-11T08:10:37.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.381+0000 7fb285ffb700 1 -- 172.21.15.27:0/3107442394 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb290068cd0 con 0x7fb2880dfaf0 2024-01-11T08:10:37.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.381+0000 7fb285ffb700 1 -- 172.21.15.27:0/3107442394 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb290090600 con 0x7fb2880dfaf0 2024-01-11T08:10:37.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.381+0000 7f7ed91ff700 1 Processor -- start 2024-01-11T08:10:37.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.382+0000 7f7ed91ff700 1 -- start start 2024-01-11T08:10:37.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.382+0000 7ff0aed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0b00b19a0 0x7ff0b014b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.382+0000 7f60071ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f60000d06b0 0x7f600015a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.382+0000 7f60071ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60000d0a80 0x7f6000150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.382+0000 7f60071ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f600015dd70 0x7f60000afe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.382+0000 7f60071ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600015aa00 con 0x7f600015dd70 2024-01-11T08:10:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.382+0000 7f60071ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600015abf0 con 0x7f60000d0a80 2024-01-11T08:10:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.382+0000 7f60071ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600015aed0 con 0x7f60000d06b0 2024-01-11T08:10:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.382+0000 7f6004f9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f60000d06b0 0x7f600015a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.383+0000 7f5ff7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60000d0a80 0x7f6000150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.383+0000 7f5ff7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60000d0a80 0x7f6000150110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:58484/0 (socket says 172.21.15.27:58484) 2024-01-11T08:10:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.383+0000 7f5ff7fff700 1 -- 172.21.15.27:0/209996211 learned_addr learned my addr 172.21.15.27:0/209996211 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.382+0000 7f87a0bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8784003a60 0x7f87940f6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.382+0000 7f87a0bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8794036830 0x7f87940dd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.382+0000 7f87a0bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87940e2290 0x7f87940e8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.382+0000 7f87a0bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87940f7a30 con 0x7f8784003a60 2024-01-11T08:10:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.382+0000 7f87a0bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87940f6f70 con 0x7f8794036830 2024-01-11T08:10:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.382+0000 7f87a0bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8794034c30 con 0x7f87940e2290 2024-01-11T08:10:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.383+0000 7fb296b9d700 1 -- 172.21.15.27:0/3107442394 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2800094f0 con 0x7fb2880dfaf0 2024-01-11T08:10:37.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.383+0000 7fa5449ff700 1 -- 172.21.15.27:0/1352661556 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa524000fd0 con 0x7fa540159e20 2024-01-11T08:10:37.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.383+0000 7fb296b9d700 1 -- 172.21.15.27:0/3107442394 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb288039480 con 0x7fb2880dfaf0 2024-01-11T08:10:37.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.383+0000 7f5ff7fff700 1 -- 172.21.15.27:0/209996211 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f60000d06b0 msgr2=0x7f600015a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.383+0000 7f8799d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8794036830 0x7f87940dd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.383+0000 7f8799d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8794036830 0x7f87940dd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:58494/0 (socket says 172.21.15.27:58494) 2024-01-11T08:10:37.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.383+0000 7f8799d9b700 1 -- 172.21.15.27:0/1560388346 learned_addr learned my addr 172.21.15.27:0/1560388346 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.383+0000 7f5ff7fff700 1 --2- 172.21.15.27:0/209996211 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f60000d06b0 0x7f600015a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.383+0000 7f600579c700 1 --2- 172.21.15.27:0/209996211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f600015dd70 0x7f60000afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.383+0000 7f5ff7fff700 1 -- 172.21.15.27:0/209996211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f600015dd70 msgr2=0x7f60000afe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.383+0000 7f5ff7fff700 1 --2- 172.21.15.27:0/209996211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f600015dd70 0x7f60000afe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.383+0000 7f5ff7fff700 1 -- 172.21.15.27:0/209996211 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff00094f0 con 0x7f60000d0a80 2024-01-11T08:10:37.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.384+0000 7f5ff7fff700 1 --2- 172.21.15.27:0/209996211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60000d0a80 0x7f6000150110 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5fe800cb60 tx=0x7f5fe80017a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.384+0000 7f879ad9d700 1 --2- 172.21.15.27:0/1560388346 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87940e2290 0x7f87940e8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.384+0000 7f879a59c700 1 --2- 172.21.15.27:0/1560388346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8784003a60 0x7f87940f6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.384+0000 7f5ffd7fa700 1 -- 172.21.15.27:0/209996211 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe800a7e0 con 0x7f60000d0a80 2024-01-11T08:10:37.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.384+0000 7f5ffd7fa700 1 -- 172.21.15.27:0/209996211 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe800a960 con 0x7f60000d0a80 2024-01-11T08:10:37.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.384+0000 7f5ffd7fa700 1 -- 172.21.15.27:0/209996211 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe800ad20 con 0x7f60000d0a80 2024-01-11T08:10:37.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.382+0000 7ff0ae59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0b00ad840 0x7ff0b0146bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.383+0000 7ff0ae59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0b00ad840 0x7ff0b0146bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53242/0 (socket says 172.21.15.27:53242) 2024-01-11T08:10:37.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.383+0000 7ff0ae59c700 1 -- 172.21.15.27:0/1726869050 learned_addr learned my addr 172.21.15.27:0/1726869050 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.382+0000 7ff0add9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0b00adc10 0x7ff0b006f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.384+0000 7f60071ff700 1 -- 172.21.15.27:0/209996211 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600015b1b0 con 0x7f60000d0a80 2024-01-11T08:10:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.385+0000 7f60071ff700 1 -- 172.21.15.27:0/209996211 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f600015b770 con 0x7f60000d0a80 2024-01-11T08:10:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.385+0000 7fa52e7fc700 1 -- 172.21.15.27:0/1352661556 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa530029460 con 0x7fa540159e20 2024-01-11T08:10:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.386+0000 7f60071ff700 1 -- 172.21.15.27:0/209996211 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f60000d1140 con 0x7f60000d0a80 2024-01-11T08:10:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.386+0000 7ff0ae59c700 1 -- 172.21.15.27:0/1726869050 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0b00adc10 msgr2=0x7ff0b006f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.386+0000 7fb285ffb700 1 -- 172.21.15.27:0/3107442394 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb290090970 con 0x7fb2880dfaf0 2024-01-11T08:10:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.386+0000 7ff0ae59c700 1 --2- 172.21.15.27:0/1726869050 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0b00adc10 0x7ff0b006f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.386+0000 7ff0ae59c700 1 -- 172.21.15.27:0/1726869050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0b00b19a0 msgr2=0x7ff0b014b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.386+0000 7ff0ae59c700 1 --2- 172.21.15.27:0/1726869050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0b00b19a0 0x7ff0b014b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.387+0000 7ff0ae59c700 1 -- 172.21.15.27:0/1726869050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0980094f0 con 0x7ff0b00ad840 2024-01-11T08:10:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.387+0000 7ff0add9b700 1 --2- 172.21.15.27:0/1726869050 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0b00adc10 0x7ff0b006f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:37.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.384+0000 7f7ed91ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7ed40b4630 0x7f7ed40ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.384+0000 7f7ed91ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed40cf660 con 0x7f7ed40d0a80 2024-01-11T08:10:37.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.384+0000 7f7ed91ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed40ced30 con 0x7f7ed40d06b0 2024-01-11T08:10:37.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.384+0000 7f7ed91ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ed40cd650 con 0x7f7ed40b4630 2024-01-11T08:10:37.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.387+0000 7f7ed259c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7ed40d0a80 0x7f7ed40b3f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37152/0 (socket says 172.21.15.27:37152) 2024-01-11T08:10:37.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.387+0000 7f7ed259c700 1 -- 172.21.15.27:0/324938996 learned_addr learned my addr 172.21.15.27:0/324938996 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:37.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.387+0000 7ff0aed9d700 1 --2- 172.21.15.27:0/1726869050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0b00b19a0 0x7ff0b014b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:37.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.387+0000 7fb296b9d700 1 -- 172.21.15.27:0/3107442394 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb2880053d0 con 0x7fb2880dfaf0 2024-01-11T08:10:37.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.388+0000 7f7ed359e700 1 --2- 172.21.15.27:0/324938996 >> v2:172.21.15.77:3300/0 conn(0x7f7ed40b4630 0x7f7ed40ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.387+0000 7f879a59c700 1 -- 172.21.15.27:0/1560388346 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87940e2290 msgr2=0x7f87940e8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.387+0000 7f879a59c700 1 --2- 172.21.15.27:0/1560388346 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87940e2290 0x7f87940e8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.387+0000 7f879a59c700 1 -- 172.21.15.27:0/1560388346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8794036830 msgr2=0x7f87940dd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.387+0000 7f879a59c700 1 --2- 172.21.15.27:0/1560388346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8794036830 0x7f87940dd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.387+0000 7f879a59c700 1 -- 172.21.15.27:0/1560388346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87940dde10 con 0x7f8784003a60 2024-01-11T08:10:37.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.388+0000 7ff0ae59c700 1 --2- 172.21.15.27:0/1726869050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0b00ad840 0x7ff0b0146bb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff098009930 tx=0x7ff098016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.388+0000 7f7ed1d9b700 1 -- 172.21.15.27:0/324938996 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909317669 0 0) 0x7f7ed40cf660 con 0x7f7ed40d0a80 2024-01-11T08:10:37.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.388+0000 7fb285ffb700 1 --2- 172.21.15.27:0/3107442394 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb274067080 0x7fb274069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.388+0000 7f7ed1d9b700 1 -- 172.21.15.27:0/324938996 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ebc003570 con 0x7f7ed40d0a80 2024-01-11T08:10:37.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.389+0000 7f3fb8fff700 1 Processor -- start 2024-01-11T08:10:37.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.389+0000 7f7ed1d9b700 1 -- 172.21.15.27:0/324938996 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444908451 0 0) 0x7f7ed40ced30 con 0x7f7ed40d06b0 2024-01-11T08:10:37.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.388+0000 7f5ffd7fa700 1 -- 172.21.15.27:0/209996211 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5fe8003030 con 0x7f60000d0a80 2024-01-11T08:10:37.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.389+0000 7f7ed1d9b700 1 -- 172.21.15.27:0/324938996 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ed40cf660 con 0x7f7ed40d06b0 2024-01-11T08:10:37.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.389+0000 7f3fb8fff700 1 -- start start 2024-01-11T08:10:37.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.389+0000 7fa52e7fc700 1 --2- 172.21.15.27:0/1352661556 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa520067010 0x7fa5200694c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.389+0000 7ff0a67fc700 1 -- 172.21.15.27:0/1726869050 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0980119c0 con 0x7ff0b00ad840 2024-01-11T08:10:37.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.390+0000 7f879a59c700 1 --2- 172.21.15.27:0/1560388346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8784003a60 0x7f87940f6a30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f878c019000 tx=0x7f878c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.390+0000 7ff0a67fc700 1 -- 172.21.15.27:0/1726869050 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff098020070 con 0x7ff0b00ad840 2024-01-11T08:10:37.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.390+0000 7ff0a67fc700 1 -- 172.21.15.27:0/1726869050 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff098011d90 con 0x7ff0b00ad840 2024-01-11T08:10:37.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.390+0000 7f5ffd7fa700 1 --2- 172.21.15.27:0/209996211 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5fe0066fb0 0x7f5fe0069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.389+0000 7f7ed359e700 1 -- 172.21.15.27:0/324938996 >> v1:172.21.15.77:6789/0 conn(0x7f7ed40d06b0 legacy=0x7f7ed40cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.389+0000 7f7ed359e700 1 -- 172.21.15.27:0/324938996 >> v1:172.21.15.27:6789/0 conn(0x7f7ed40d0a80 legacy=0x7f7ed40b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.389+0000 7f7ed359e700 1 -- 172.21.15.27:0/324938996 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed40cf010 con 0x7f7ed40b4630 2024-01-11T08:10:37.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.389+0000 7fb295b9b700 1 --2- 172.21.15.27:0/3107442394 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb274067080 0x7fb274069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.389+0000 7fb285ffb700 1 -- 172.21.15.27:0/3107442394 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fb2900fb280 con 0x7fb2880dfaf0 2024-01-11T08:10:37.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.390+0000 7fb285ffb700 1 --2- 172.21.15.27:0/3107442394 >> [v2:172.21.15.27:6809/41833,v1:172.21.15.27:6811/41833] conn(0x7fb27406f0a0 0x7fb2740714b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.390+0000 7fa52e7fc700 1 -- 172.21.15.27:0/1352661556 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa530086210 con 0x7fa540159e20 2024-01-11T08:10:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.390+0000 7fb285ffb700 1 -- 172.21.15.27:0/3107442394 --> [v2:172.21.15.27:6809/41833,v1:172.21.15.27:6811/41833] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb274071b60 con 0x7fb27406f0a0 2024-01-11T08:10:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.390+0000 7fb285ffb700 1 -- 172.21.15.27:0/3107442394 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2900c8580 con 0x7fb2880dfaf0 2024-01-11T08:10:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.390+0000 7f7ed359e700 1 --2- 172.21.15.27:0/324938996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed40b4630 0x7f7ed40ccf50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc0098e0 tx=0x7f7ecc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25ec7e75fb139a4f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.391+0000 7fa52e7fc700 1 --2- 172.21.15.27:0/1352661556 >> [v2:172.21.15.27:6813/41835,v1:172.21.15.27:6816/41835] conn(0x7fa52006f220 0x7fa520071630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.390+0000 7f7ed1d9b700 1 -- 172.21.15.27:0/324938996 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc011630 con 0x7f7ed40b4630 2024-01-11T08:10:37.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.391+0000 7f7ed1d9b700 1 -- 172.21.15.27:0/324938996 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ecc011790 con 0x7f7ed40b4630 2024-01-11T08:10:37.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.391+0000 7f7ed1d9b700 1 -- 172.21.15.27:0/324938996 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc011b30 con 0x7f7ed40b4630 2024-01-11T08:10:37.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.391+0000 7fb29539a700 1 --2- 172.21.15.27:0/3107442394 >> [v2:172.21.15.27:6809/41833,v1:172.21.15.27:6811/41833] conn(0x7fb27406f0a0 0x7fb2740714b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.391+0000 7fa52e7fc700 1 -- 172.21.15.27:0/1352661556 --> [v2:172.21.15.27:6813/41835,v1:172.21.15.27:6816/41835] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa520071ce0 con 0x7fa52006f220 2024-01-11T08:10:37.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.391+0000 7fa52e7fc700 1 -- 172.21.15.27:0/1352661556 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5300887b0 con 0x7fa540159e20 2024-01-11T08:10:37.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.391+0000 7fa53ed9d700 1 --2- 172.21.15.27:0/1352661556 >> [v2:172.21.15.27:6813/41835,v1:172.21.15.27:6816/41835] conn(0x7fa52006f220 0x7fa520071630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.391+0000 7f3fb8fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3fb4156ab0 0x7f3fb40ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.392+0000 7fb295b9b700 1 --2- 172.21.15.27:0/3107442394 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb274067080 0x7fb274069530 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009fa0 tx=0x7fb28c009120 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.391+0000 7f3fb8fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3fb40b0550 0x7f3fb4160070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.392+0000 7ff0b49ff700 1 -- 172.21.15.27:0/1726869050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b0147a30 con 0x7ff0b00ad840 2024-01-11T08:10:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.392+0000 7f878a7fc700 1 -- 172.21.15.27:0/1560388346 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c000bc0 con 0x7f8784003a60 2024-01-11T08:10:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.392+0000 7f3fb8fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb40adc80 con 0x7f3fb40cc9b0 2024-01-11T08:10:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.392+0000 7f3fb8fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fb40adf60 con 0x7f3fb4156ab0 2024-01-11T08:10:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.392+0000 7f3fb8fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb40ae240 con 0x7f3fb40b0550 2024-01-11T08:10:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.392+0000 7f878a7fc700 1 -- 172.21.15.27:0/1560388346 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878c00ea10 con 0x7f8784003a60 2024-01-11T08:10:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.392+0000 7f878a7fc700 1 -- 172.21.15.27:0/1560388346 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c000d20 con 0x7f8784003a60 2024-01-11T08:10:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.392+0000 7f4a2e1ff700 1 Processor -- start 2024-01-11T08:10:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.392+0000 7f3fb259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3fb4156ab0 0x7f3fb40ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.392+0000 7fb29539a700 1 --2- 172.21.15.27:0/3107442394 >> [v2:172.21.15.27:6809/41833,v1:172.21.15.27:6811/41833] conn(0x7fb27406f0a0 0x7fb2740714b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.392+0000 7f3fb259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3fb4156ab0 0x7f3fb40ad650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53272/0 (socket says 172.21.15.27:53272) 2024-01-11T08:10:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.392+0000 7f3fb259c700 1 -- 172.21.15.27:0/394158902 learned_addr learned my addr 172.21.15.27:0/394158902 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.393+0000 7ff0b49ff700 1 -- 172.21.15.27:0/1726869050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b016e820 con 0x7ff0b00ad840 2024-01-11T08:10:37.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.392+0000 7f7ed91ff700 1 -- 172.21.15.27:0/324938996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed40b4630 msgr2=0x7f7ed40ccf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.392+0000 7f7ed91ff700 1 --2- 172.21.15.27:0/324938996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed40b4630 0x7f7ed40ccf50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc0098e0 tx=0x7f7ecc0091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.393+0000 7f7ed91ff700 1 -- 172.21.15.27:0/324938996 shutdown_connections 2024-01-11T08:10:37.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.393+0000 7f7ed91ff700 1 --2- 172.21.15.27:0/324938996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed40b4630 0x7f7ed40ccf50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.393+0000 7f7ed91ff700 1 -- 172.21.15.27:0/324938996 >> 172.21.15.27:0/324938996 conn(0x7f7ed406ed40 msgr2=0x7f7ed406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.390+0000 7f5ffd7fa700 1 -- 172.21.15.27:0/209996211 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f5fe8084690 con 0x7f60000d0a80 2024-01-11T08:10:37.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.391+0000 7f5ffd7fa700 1 --2- 172.21.15.27:0/209996211 >> [v2:172.21.15.27:6800/41829,v1:172.21.15.27:6801/41829] conn(0x7f5fe006f1c0 0x7f5fe00715d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.391+0000 7f5ffd7fa700 1 -- 172.21.15.27:0/209996211 --> [v2:172.21.15.27:6800/41829,v1:172.21.15.27:6801/41829] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5fe0071c80 con 0x7f5fe006f1c0 2024-01-11T08:10:37.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.391+0000 7f5ffd7fa700 1 -- 172.21.15.27:0/209996211 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fe8086c30 con 0x7f60000d0a80 2024-01-11T08:10:37.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.392+0000 7f600579c700 1 --2- 172.21.15.27:0/209996211 >> [v2:172.21.15.27:6800/41829,v1:172.21.15.27:6801/41829] conn(0x7f5fe006f1c0 0x7f5fe00715d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.392+0000 7f6004f9b700 1 --2- 172.21.15.27:0/209996211 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5fe0066fb0 0x7f5fe0069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.394+0000 7f3fb259c700 1 -- 172.21.15.27:0/394158902 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3fb40b0550 msgr2=0x7f3fb4160070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-11T08:10:37.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.392+0000 7f4a2e1ff700 1 -- start start 2024-01-11T08:10:37.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.394+0000 7fb285ffb700 1 -- 172.21.15.27:0/3107442394 <== osd.2 v2:172.21.15.27:6809/41833 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fb274071b60 con 0x7fb27406f0a0 2024-01-11T08:10:37.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.392+0000 7fa53ed9d700 1 --2- 172.21.15.27:0/1352661556 >> [v2:172.21.15.27:6813/41835,v1:172.21.15.27:6816/41835] conn(0x7fa52006f220 0x7fa520071630 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.393+0000 7fa53e59c700 1 --2- 172.21.15.27:0/1352661556 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa520067010 0x7fa5200694c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.394+0000 7f600579c700 1 --2- 172.21.15.27:0/209996211 >> [v2:172.21.15.27:6800/41829,v1:172.21.15.27:6801/41829] conn(0x7f5fe006f1c0 0x7f5fe00715d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.394+0000 7f87a0bff700 1 -- 172.21.15.27:0/1560388346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87940f7190 con 0x7f8784003a60 2024-01-11T08:10:37.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.394+0000 7f7ed91ff700 1 -- 172.21.15.27:0/324938996 shutdown_connections 2024-01-11T08:10:37.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.394+0000 7fa52e7fc700 1 -- 172.21.15.27:0/1352661556 <== osd.1 v2:172.21.15.27:6813/41835 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa520071ce0 con 0x7fa52006f220 2024-01-11T08:10:37.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.394+0000 7f3fb259c700 1 --2- 172.21.15.27:0/394158902 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3fb40b0550 0x7f3fb4160070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.394+0000 7f3fb259c700 1 -- 172.21.15.27:0/394158902 >> v1:172.21.15.27:6789/0 conn(0x7f3fb40cc9b0 legacy=0x7f3fb4156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.395+0000 7f87a0bff700 1 -- 172.21.15.27:0/1560388346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87940f7750 con 0x7f8784003a60 2024-01-11T08:10:37.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.394+0000 7f3fb1d9b700 1 -- 172.21.15.27:0/394158902 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9265326 0 0) 0x7f3fb40adc80 con 0x7f3fb40cc9b0 2024-01-11T08:10:37.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.394+0000 7f3fb259c700 1 -- 172.21.15.27:0/394158902 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb40ae520 con 0x7f3fb4156ab0 2024-01-11T08:10:37.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.394+0000 7f7ed91ff700 1 -- 172.21.15.27:0/324938996 wait complete. 2024-01-11T08:10:37.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.395+0000 7f4a2e1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4a2815d9c0 0x7f4a28161770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.396+0000 7fa53e59c700 1 --2- 172.21.15.27:0/1352661556 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa520067010 0x7fa5200694c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa538000b60 tx=0x7fa538001c90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.395+0000 7f4a2e1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4a280ccb70 0x7f4a280af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.395+0000 7f4a2e1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2815f4d0 con 0x7f4a280af7d0 2024-01-11T08:10:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.395+0000 7f4a2e1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a28162680 con 0x7f4a2815d9c0 2024-01-11T08:10:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.395+0000 7f4a2e1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a280afc00 con 0x7f4a280ccb70 2024-01-11T08:10:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.395+0000 7f4a27fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4a280af7d0 0x7f4a2815dd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37170/0 (socket says 172.21.15.27:37170) 2024-01-11T08:10:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.395+0000 7f4a27fff700 1 -- 172.21.15.27:0/3567483909 learned_addr learned my addr 172.21.15.27:0/3567483909 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.396+0000 7f4a1effd700 1 --2- 172.21.15.27:0/3567483909 >> v2:172.21.15.77:3300/0 conn(0x7f4a280ccb70 0x7f4a280af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.396+0000 7f4a1effd700 1 -- 172.21.15.27:0/3567483909 >> v2:172.21.15.27:3300/0 conn(0x7f4a2815d9c0 msgr2=0x7f4a28161770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.396+0000 7f4a1effd700 1 --2- 172.21.15.27:0/3567483909 >> v2:172.21.15.27:3300/0 conn(0x7f4a2815d9c0 0x7f4a28161770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.396+0000 7f4a1effd700 1 -- 172.21.15.27:0/3567483909 >> v1:172.21.15.27:6789/0 conn(0x7f4a280af7d0 legacy=0x7f4a2815dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.396+0000 7f4a1effd700 1 -- 172.21.15.27:0/3567483909 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a28161de0 con 0x7f4a280ccb70 2024-01-11T08:10:37.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.397+0000 7f5ffd7fa700 1 -- 172.21.15.27:0/209996211 <== osd.0 v2:172.21.15.27:6800/41829 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f5fe0071c80 con 0x7f5fe006f1c0 2024-01-11T08:10:37.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.397+0000 7f4a1effd700 1 --2- 172.21.15.27:0/3567483909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a280ccb70 0x7f4a280af290 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4a280b1140 tx=0x7f4a180091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3f381617f9b000c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.397+0000 7f7ed91ff700 1 Processor -- start 2024-01-11T08:10:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.398+0000 7f4a26ffd700 1 -- 172.21.15.27:0/3567483909 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1800b6b0 con 0x7f4a280ccb70 2024-01-11T08:10:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.398+0000 7f4a26ffd700 1 -- 172.21.15.27:0/3567483909 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a1800b810 con 0x7f4a280ccb70 2024-01-11T08:10:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.398+0000 7f4a26ffd700 1 -- 172.21.15.27:0/3567483909 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1800bbb0 con 0x7f4a280ccb70 2024-01-11T08:10:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.396+0000 7ff0a67fc700 1 -- 172.21.15.27:0/1726869050 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff098022030 con 0x7ff0b00ad840 2024-01-11T08:10:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.396+0000 7ff0b49ff700 1 -- 172.21.15.27:0/1726869050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff0b01475e0 con 0x7ff0b00ad840 2024-01-11T08:10:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.398+0000 7ff0a67fc700 1 --2- 172.21.15.27:0/1726869050 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff090066f30 0x7ff0900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.397+0000 7f7ed91ff700 1 -- start start 2024-01-11T08:10:37.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.398+0000 7f6004f9b700 1 --2- 172.21.15.27:0/209996211 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5fe0066fb0 0x7f5fe0069460 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00099b0 tx=0x7f5ff0017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.399+0000 7ff0add9b700 1 --2- 172.21.15.27:0/1726869050 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff090066f30 0x7ff0900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.399+0000 7ff0a67fc700 1 -- 172.21.15.27:0/1726869050 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7ff098057830 con 0x7ff0b00ad840 2024-01-11T08:10:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.399+0000 7f4a2e1ff700 1 -- 172.21.15.27:0/3567483909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a280ccb70 msgr2=0x7f4a280af290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.399+0000 7f4a2e1ff700 1 --2- 172.21.15.27:0/3567483909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a280ccb70 0x7f4a280af290 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4a280b1140 tx=0x7f4a180091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.397+0000 7f87a0bff700 1 -- 172.21.15.27:0/1560388346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8794034a00 con 0x7f8784003a60 2024-01-11T08:10:37.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.399+0000 7ff0a67fc700 1 --2- 172.21.15.27:0/1726869050 >> [v2:172.21.15.77:6800/42095,v1:172.21.15.77:6801/42095] conn(0x7ff09006fe20 0x7ff0900722b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.399+0000 7f878a7fc700 1 -- 172.21.15.27:0/1560388346 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f878c026070 con 0x7f8784003a60 2024-01-11T08:10:37.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.399+0000 7ff0a67fc700 1 -- 172.21.15.27:0/1726869050 --> [v2:172.21.15.77:6800/42095,v1:172.21.15.77:6801/42095] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff090072980 con 0x7ff09006fe20 2024-01-11T08:10:37.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.400+0000 7ff0a67fc700 1 -- 172.21.15.27:0/1726869050 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff098056f00 con 0x7ff0b00ad840 2024-01-11T08:10:37.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.400+0000 7f7ed91ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed40b4630 0x7f7ed4156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.400+0000 7f7ed91ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ed40d06b0 0x7f7ed406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.397+0000 7f3fb259c700 1 --2- 172.21.15.27:0/394158902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fb4156ab0 0x7f3fb40ad650 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3fa40099b0 tx=0x7f3fa4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d957174ae41b8c1b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.400+0000 7f3fb1d9b700 1 -- 172.21.15.27:0/394158902 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa400b860 con 0x7f3fb4156ab0 2024-01-11T08:10:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.400+0000 7f3fb1d9b700 1 -- 172.21.15.27:0/394158902 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa400b9c0 con 0x7f3fb4156ab0 2024-01-11T08:10:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.400+0000 7f3fb1d9b700 1 -- 172.21.15.27:0/394158902 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa400bd60 con 0x7f3fb4156ab0 2024-01-11T08:10:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.400+0000 7ff0aed9d700 1 --2- 172.21.15.27:0/1726869050 >> [v2:172.21.15.77:6800/42095,v1:172.21.15.77:6801/42095] conn(0x7ff09006fe20 0x7ff0900722b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.400+0000 7f7ed91ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ed40d0a80 0x7f7ed406f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.400+0000 7f7ed91ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed406b940 con 0x7f7ed40d06b0 2024-01-11T08:10:37.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.400+0000 7f7ed91ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed406a930 con 0x7f7ed40b4630 2024-01-11T08:10:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.400+0000 7f7ed91ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed406ab80 con 0x7f7ed40d0a80 2024-01-11T08:10:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.400+0000 7f7ed259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ed40d06b0 0x7f7ed406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.400+0000 7f7ed259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ed40d06b0 0x7f7ed406f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53290/0 (socket says 172.21.15.27:53290) 2024-01-11T08:10:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.400+0000 7f7ed259c700 1 -- 172.21.15.27:0/4173508689 learned_addr learned my addr 172.21.15.27:0/4173508689 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.401+0000 7f4a2e1ff700 1 -- 172.21.15.27:0/3567483909 shutdown_connections 2024-01-11T08:10:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.401+0000 7f4a2e1ff700 1 --2- 172.21.15.27:0/3567483909 >> v2:172.21.15.27:3300/0 conn(0x7f4a2815d9c0 0x7f4a28161770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.401+0000 7f4a2e1ff700 1 --2- 172.21.15.27:0/3567483909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a280ccb70 0x7f4a280af290 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.401+0000 7f4a2e1ff700 1 -- 172.21.15.27:0/3567483909 >> 172.21.15.27:0/3567483909 conn(0x7f4a280680b0 msgr2=0x7f4a280cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.401+0000 7ff0aed9d700 1 --2- 172.21.15.27:0/1726869050 >> [v2:172.21.15.77:6800/42095,v1:172.21.15.77:6801/42095] conn(0x7ff09006fe20 0x7ff0900722b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.401+0000 7f7ed359e700 1 --2- 172.21.15.27:0/4173508689 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ed40d0a80 0x7f7ed406f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.401+0000 7f7ed2d9d700 1 --2- 172.21.15.27:0/4173508689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed40b4630 0x7f7ed4156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.401+0000 7f4a2e1ff700 1 -- 172.21.15.27:0/3567483909 shutdown_connections 2024-01-11T08:10:37.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.401+0000 7f4a2e1ff700 1 -- 172.21.15.27:0/3567483909 wait complete. 2024-01-11T08:10:37.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.401+0000 7f7ed359e700 1 -- 172.21.15.27:0/4173508689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed40b4630 msgr2=0x7f7ed4156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.401+0000 7f7ed359e700 1 --2- 172.21.15.27:0/4173508689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed40b4630 0x7f7ed4156060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.401+0000 7f7ed359e700 1 -- 172.21.15.27:0/4173508689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ed40d06b0 msgr2=0x7f7ed406f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.401+0000 7f7ed359e700 1 --2- 172.21.15.27:0/4173508689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ed40d06b0 0x7f7ed406f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.401+0000 7f7ed359e700 1 -- 172.21.15.27:0/4173508689 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ecc009420 con 0x7f7ed40d0a80 2024-01-11T08:10:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.401+0000 7f7ed259c700 1 --2- 172.21.15.27:0/4173508689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ed40d06b0 0x7f7ed406f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.402+0000 7ff0add9b700 1 --2- 172.21.15.27:0/1726869050 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff090066f30 0x7ff0900693e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0146b60 tx=0x7ff0a0007d30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.402+0000 7f7ed359e700 1 --2- 172.21.15.27:0/4173508689 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ed40d0a80 0x7f7ed406f650 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc016000 tx=0x7f7ecc017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.403+0000 7ff0a67fc700 1 -- 172.21.15.27:0/1726869050 <== osd.4 v2:172.21.15.77:6800/42095 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7ff090072980 con 0x7ff09006fe20 2024-01-11T08:10:37.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.403+0000 7f7ebaffd700 1 -- 172.21.15.27:0/4173508689 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc011980 con 0x7f7ed40d0a80 2024-01-11T08:10:37.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.403+0000 7f7ebaffd700 1 -- 172.21.15.27:0/4173508689 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ecc011b20 con 0x7f7ed40d0a80 2024-01-11T08:10:37.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.403+0000 7f7ed91ff700 1 -- 172.21.15.27:0/4173508689 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed406ae60 con 0x7f7ed40d0a80 2024-01-11T08:10:37.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.403+0000 7f3fb8fff700 1 -- 172.21.15.27:0/394158902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fb4156ab0 msgr2=0x7f3fb40ad650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.403+0000 7f3fb8fff700 1 --2- 172.21.15.27:0/394158902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fb4156ab0 0x7f3fb40ad650 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3fa40099b0 tx=0x7f3fa4009250 comp rx=0 tx=0).stop 2024-01-11T08:10:37.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.403+0000 7f878a7fc700 1 --2- 172.21.15.27:0/1560388346 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f877c067080 0x7f877c069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.403+0000 7f3fb8fff700 1 -- 172.21.15.27:0/394158902 shutdown_connections 2024-01-11T08:10:37.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.403+0000 7f3fb8fff700 1 --2- 172.21.15.27:0/394158902 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3fb40b0550 0x7f3fb4160070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.403+0000 7f3fb8fff700 1 --2- 172.21.15.27:0/394158902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fb4156ab0 0x7f3fb40ad650 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.403+0000 7f3fb8fff700 1 -- 172.21.15.27:0/394158902 >> 172.21.15.27:0/394158902 conn(0x7f3fb40680b0 msgr2=0x7f3fb40cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.403+0000 7f4a2e1ff700 1 Processor -- start 2024-01-11T08:10:37.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.403+0000 7f4a2e1ff700 1 -- start start 2024-01-11T08:10:37.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.403+0000 7f878a7fc700 1 -- 172.21.15.27:0/1560388346 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f878c085b40 con 0x7f8784003a60 2024-01-11T08:10:37.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.404+0000 7f878a7fc700 1 --2- 172.21.15.27:0/1560388346 >> [v2:172.21.15.27:6808/41831,v1:172.21.15.27:6810/41831] conn(0x7f877c06f290 0x7f877c0716a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.404+0000 7f878a7fc700 1 -- 172.21.15.27:0/1560388346 --> [v2:172.21.15.27:6808/41831,v1:172.21.15.27:6810/41831] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f877c071d50 con 0x7f877c06f290 2024-01-11T08:10:37.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.404+0000 7f878a7fc700 1 -- 172.21.15.27:0/1560388346 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f878c08a220 con 0x7f8784003a60 2024-01-11T08:10:37.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.404+0000 7f8799d9b700 1 --2- 172.21.15.27:0/1560388346 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f877c067080 0x7f877c069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.403+0000 7f3fb8fff700 1 -- 172.21.15.27:0/394158902 shutdown_connections 2024-01-11T08:10:37.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.405+0000 7f4a2e1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a280af7d0 0x7f4a28154eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.403+0000 7f3fb8fff700 1 -- 172.21.15.27:0/394158902 wait complete. 2024-01-11T08:10:37.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.406+0000 7f4a2e1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a280ccb70 0x7f4a280adc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.403+0000 7f7ed91ff700 1 -- 172.21.15.27:0/4173508689 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed406b420 con 0x7f7ed40d0a80 2024-01-11T08:10:37.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.406+0000 7f7ebaffd700 1 -- 172.21.15.27:0/4173508689 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc025660 con 0x7f7ed40d0a80 2024-01-11T08:10:37.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.405+0000 7f879ad9d700 1 --2- 172.21.15.27:0/1560388346 >> [v2:172.21.15.27:6808/41831,v1:172.21.15.27:6810/41831] conn(0x7f877c06f290 0x7f877c0716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.406+0000 7f4a277fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a280af7d0 0x7f4a28154eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.406+0000 7f4a277fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a280af7d0 0x7f4a28154eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:58512/0 (socket says 172.21.15.27:58512) 2024-01-11T08:10:37.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.407+0000 7f4a277fe700 1 -- 172.21.15.27:0/962413262 learned_addr learned my addr 172.21.15.27:0/962413262 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.407+0000 7f4a2e1ff700 1 --2- 172.21.15.27:0/962413262 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4a2815d9c0 0x7f4a2816f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.406+0000 7f3fb8fff700 1 Processor -- start 2024-01-11T08:10:37.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.406+0000 7f3fb8fff700 1 -- start start 2024-01-11T08:10:37.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.407+0000 7f4a2e1ff700 1 -- 172.21.15.27:0/962413262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a28154a20 con 0x7f4a280ccb70 2024-01-11T08:10:37.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.408+0000 7f4a27fff700 1 --2- 172.21.15.27:0/962413262 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4a2815d9c0 0x7f4a2816f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.408+0000 7f4a2e1ff700 1 -- 172.21.15.27:0/962413262 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a281553f0 con 0x7f4a280af7d0 2024-01-11T08:10:37.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.408+0000 7f4a2e1ff700 1 -- 172.21.15.27:0/962413262 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a28153e10 con 0x7f4a2815d9c0 2024-01-11T08:10:37.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.408+0000 7f7ed91ff700 1 -- 172.21.15.27:0/4173508689 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7eb4000fd0 con 0x7f7ed40d0a80 2024-01-11T08:10:37.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.406+0000 7f8799d9b700 1 --2- 172.21.15.27:0/1560388346 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f877c067080 0x7f877c069530 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8790000c20 tx=0x7f8790007040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.406+0000 7f879ad9d700 1 --2- 172.21.15.27:0/1560388346 >> [v2:172.21.15.27:6808/41831,v1:172.21.15.27:6810/41831] conn(0x7f877c06f290 0x7f877c0716a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.408+0000 7f878a7fc700 1 -- 172.21.15.27:0/1560388346 <== osd.3 v2:172.21.15.27:6808/41831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f877c071d50 con 0x7f877c06f290 2024-01-11T08:10:37.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.409+0000 7f7ebaffd700 1 -- 172.21.15.27:0/4173508689 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7ecc023070 con 0x7f7ed40d0a80 2024-01-11T08:10:37.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.409+0000 7f4a27fff700 1 -- 172.21.15.27:0/962413262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a280af7d0 msgr2=0x7f4a28154eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.409+0000 7f4a27fff700 1 --2- 172.21.15.27:0/962413262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a280af7d0 0x7f4a28154eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.409+0000 7f4a27fff700 1 -- 172.21.15.27:0/962413262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a280ccb70 msgr2=0x7f4a280adc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.409+0000 7f4a27fff700 1 --2- 172.21.15.27:0/962413262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a280ccb70 0x7f4a280adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.409+0000 7f4a27fff700 1 -- 172.21.15.27:0/962413262 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a18009420 con 0x7f4a2815d9c0 2024-01-11T08:10:37.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.410+0000 7f4a27fff700 1 --2- 172.21.15.27:0/962413262 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4a2815d9c0 0x7f4a2816f7b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000cb90 tx=0x7f4a20001960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.410+0000 7f7ebaffd700 1 --2- 172.21.15.27:0/4173508689 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7ec0067000 0x7f7ec00694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.410+0000 7f7ebaffd700 1 -- 172.21.15.27:0/4173508689 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f7ecc0859e0 con 0x7f7ed40d0a80 2024-01-11T08:10:37.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.411+0000 7f4a1ffff700 1 -- 172.21.15.27:0/962413262 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2000a8b0 con 0x7f4a2815d9c0 2024-01-11T08:10:37.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.410+0000 7f7ebaffd700 1 --2- 172.21.15.27:0/4173508689 >> [v2:172.21.15.77:6810/42096,v1:172.21.15.77:6812/42096] conn(0x7f7ec006f190 0x7f7ec00715a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.410+0000 7f7ebaffd700 1 -- 172.21.15.27:0/4173508689 --> [v2:172.21.15.77:6810/42096,v1:172.21.15.77:6812/42096] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ec0071c50 con 0x7f7ec006f190 2024-01-11T08:10:37.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.410+0000 7f7ebaffd700 1 -- 172.21.15.27:0/4173508689 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ecc0092a0 con 0x7f7ed40d0a80 2024-01-11T08:10:37.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.411+0000 7f4a1ffff700 1 -- 172.21.15.27:0/962413262 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2000aa30 con 0x7f4a2815d9c0 2024-01-11T08:10:37.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.411+0000 7f4a1ffff700 1 -- 172.21.15.27:0/962413262 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2000adf0 con 0x7f4a2815d9c0 2024-01-11T08:10:37.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.411+0000 7f4a2e1ff700 1 -- 172.21.15.27:0/962413262 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a281540f0 con 0x7f4a2815d9c0 2024-01-11T08:10:37.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.411+0000 7f4a2e1ff700 1 -- 172.21.15.27:0/962413262 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a281546b0 con 0x7f4a2815d9c0 2024-01-11T08:10:37.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.412+0000 7f3fb8fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fb40b0550 0x7f3fb4141af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.412+0000 7f3fb8fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fb40cc9b0 0x7f3fb40ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.412+0000 7f3fb8fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3fb4156ab0 0x7f3fb40ad370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.412+0000 7f3fb8fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb414b570 con 0x7f3fb40cc9b0 2024-01-11T08:10:37.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.412+0000 7f3fb8fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb414a960 con 0x7f3fb40b0550 2024-01-11T08:10:37.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.412+0000 7f3fb8fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb414ac40 con 0x7f3fb4156ab0 2024-01-11T08:10:37.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.412+0000 7f7ed2d9d700 1 --2- 172.21.15.27:0/4173508689 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7ec0067000 0x7f7ec00694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.412+0000 7f3fb2d9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3fb4156ab0 0x7f3fb40ad370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.412+0000 7f3fb2d9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3fb4156ab0 0x7f3fb40ad370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:59604/0 (socket says 172.21.15.27:59604) 2024-01-11T08:10:37.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.412+0000 7f7ed259c700 1 --2- 172.21.15.27:0/4173508689 >> [v2:172.21.15.77:6810/42096,v1:172.21.15.77:6812/42096] conn(0x7f7ec006f190 0x7f7ec00715a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.412+0000 7f3fb2d9d700 1 -- 172.21.15.27:0/1354074089 learned_addr learned my addr 172.21.15.27:0/1354074089 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.413+0000 7f3fb2d9d700 1 -- 172.21.15.27:0/1354074089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fb40b0550 msgr2=0x7f3fb4141af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:10:37.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.413+0000 7f7ed2d9d700 1 --2- 172.21.15.27:0/4173508689 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7ec0067000 0x7f7ec00694b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7ed40af700 tx=0x7f7ec4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.413+0000 7f3fb2d9d700 1 --2- 172.21.15.27:0/1354074089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fb40b0550 0x7f3fb4141af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.413+0000 7f3fb2d9d700 1 -- 172.21.15.27:0/1354074089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fb40cc9b0 msgr2=0x7f3fb40ace30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:10:37.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.413+0000 7f7ed259c700 1 --2- 172.21.15.27:0/4173508689 >> [v2:172.21.15.77:6810/42096,v1:172.21.15.77:6812/42096] conn(0x7f7ec006f190 0x7f7ec00715a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.413+0000 7f3fa9d9b700 1 --2- 172.21.15.27:0/1354074089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fb40cc9b0 0x7f3fb40ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.413+0000 7f3fb2d9d700 1 --2- 172.21.15.27:0/1354074089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fb40cc9b0 0x7f3fb40ace30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.413+0000 7f3fb2d9d700 1 -- 172.21.15.27:0/1354074089 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa40094f0 con 0x7f3fb4156ab0 2024-01-11T08:10:37.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.414+0000 7f3fa9d9b700 1 --2- 172.21.15.27:0/1354074089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fb40cc9b0 0x7f3fb40ace30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:37.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.414+0000 7f3fb2d9d700 1 --2- 172.21.15.27:0/1354074089 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3fb4156ab0 0x7f3fb40ad370 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3fac00b5e0 tx=0x7f3fac009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.414+0000 7f7ebaffd700 1 -- 172.21.15.27:0/4173508689 <== osd.5 v2:172.21.15.77:6810/42096 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f7ec0071c50 con 0x7f7ec006f190 2024-01-11T08:10:37.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.414+0000 7f3faaffd700 1 -- 172.21.15.27:0/1354074089 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac00f510 con 0x7f3fb4156ab0 2024-01-11T08:10:37.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.414+0000 7f3faaffd700 1 -- 172.21.15.27:0/1354074089 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fac00f670 con 0x7f3fb4156ab0 2024-01-11T08:10:37.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.414+0000 7f3faaffd700 1 -- 172.21.15.27:0/1354074089 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac00fa10 con 0x7f3fb4156ab0 2024-01-11T08:10:37.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.415+0000 7f4a1ffff700 1 -- 172.21.15.27:0/962413262 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4a20007300 con 0x7f4a2815d9c0 2024-01-11T08:10:37.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.415+0000 7f4a2e1ff700 1 -- 172.21.15.27:0/962413262 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4a2804c3c0 con 0x7f4a2815d9c0 2024-01-11T08:10:37.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.415+0000 7f3fb8fff700 1 -- 172.21.15.27:0/1354074089 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb414af20 con 0x7f3fb4156ab0 2024-01-11T08:10:37.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.415+0000 7f3fb8fff700 1 -- 172.21.15.27:0/1354074089 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fb4142030 con 0x7f3fb4156ab0 2024-01-11T08:10:37.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.416+0000 7f3fb8fff700 1 -- 172.21.15.27:0/1354074089 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3fb404c3c0 con 0x7f3fb4156ab0 2024-01-11T08:10:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.416+0000 7f4a1ffff700 1 --2- 172.21.15.27:0/962413262 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4a08066f30 0x7f4a080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.417+0000 7f4a1ffff700 1 -- 172.21.15.27:0/962413262 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f4a20084810 con 0x7f4a2815d9c0 2024-01-11T08:10:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.417+0000 7f4a1ffff700 1 --2- 172.21.15.27:0/962413262 >> [v2:172.21.15.77:6824/42140,v1:172.21.15.77:6825/42140] conn(0x7f4a0806f140 0x7f4a08071550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.417+0000 7f4a1ffff700 1 -- 172.21.15.27:0/962413262 --> [v2:172.21.15.77:6824/42140,v1:172.21.15.77:6825/42140] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a08071c00 con 0x7f4a0806f140 2024-01-11T08:10:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.417+0000 7f4a1ffff700 1 -- 172.21.15.27:0/962413262 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a20086d70 con 0x7f4a2815d9c0 2024-01-11T08:10:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.417+0000 7f4a277fe700 1 --2- 172.21.15.27:0/962413262 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4a08066f30 0x7f4a080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.418+0000 7f4a1effd700 1 --2- 172.21.15.27:0/962413262 >> [v2:172.21.15.77:6824/42140,v1:172.21.15.77:6825/42140] conn(0x7f4a0806f140 0x7f4a08071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.418+0000 7f4a277fe700 1 --2- 172.21.15.27:0/962413262 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4a08066f30 0x7f4a080693e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4a280b0b80 tx=0x7f4a100091d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.421+0000 7f3faaffd700 1 -- 172.21.15.27:0/1354074089 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3fac024430 con 0x7f3fb4156ab0 2024-01-11T08:10:37.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.421+0000 7f4a1effd700 1 --2- 172.21.15.27:0/962413262 >> [v2:172.21.15.77:6824/42140,v1:172.21.15.77:6825/42140] conn(0x7f4a0806f140 0x7f4a08071550 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.422+0000 7f4a1ffff700 1 -- 172.21.15.27:0/962413262 <== osd.7 v2:172.21.15.77:6824/42140 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f4a08071c00 con 0x7f4a0806f140 2024-01-11T08:10:37.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.422+0000 7f3faaffd700 1 --2- 172.21.15.27:0/1354074089 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3f98067000 0x7f3f980694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.423+0000 7f3faaffd700 1 -- 172.21.15.27:0/1354074089 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f3fac084960 con 0x7f3fb4156ab0 2024-01-11T08:10:37.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.423+0000 7f3faaffd700 1 --2- 172.21.15.27:0/1354074089 >> [v2:172.21.15.77:6808/42097,v1:172.21.15.77:6809/42097] conn(0x7f3f9806f210 0x7f3f98071620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.423+0000 7f3faaffd700 1 -- 172.21.15.27:0/1354074089 --> [v2:172.21.15.77:6808/42097,v1:172.21.15.77:6809/42097] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f98071cd0 con 0x7f3f9806f210 2024-01-11T08:10:37.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.423+0000 7f3faaffd700 1 -- 172.21.15.27:0/1354074089 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3fac084c50 con 0x7f3fb4156ab0 2024-01-11T08:10:37.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.424+0000 7f3fa9d9b700 1 --2- 172.21.15.27:0/1354074089 >> [v2:172.21.15.77:6808/42097,v1:172.21.15.77:6809/42097] conn(0x7f3f9806f210 0x7f3f98071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.424+0000 7f3fb259c700 1 --2- 172.21.15.27:0/1354074089 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3f98067000 0x7f3f980694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.424+0000 7f3fa9d9b700 1 --2- 172.21.15.27:0/1354074089 >> [v2:172.21.15.77:6808/42097,v1:172.21.15.77:6809/42097] conn(0x7f3f9806f210 0x7f3f98071620 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.425+0000 7f3fb259c700 1 --2- 172.21.15.27:0/1354074089 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3f98067000 0x7f3f980694b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3fa40099b0 tx=0x7f3fa4019000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.425+0000 7f3faaffd700 1 -- 172.21.15.27:0/1354074089 <== osd.6 v2:172.21.15.77:6808/42097 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f3f98071cd0 con 0x7f3f9806f210 2024-01-11T08:10:37.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.427+0000 7fb296b9d700 1 -- 172.21.15.27:0/3107442394 --> [v2:172.21.15.27:6809/41833,v1:172.21.15.27:6811/41833] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb288006940 con 0x7fb27406f0a0 2024-01-11T08:10:37.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.427+0000 7fb285ffb700 1 -- 172.21.15.27:0/3107442394 <== osd.2 v2:172.21.15.27:6809/41833 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb288006940 con 0x7fb27406f0a0 2024-01-11T08:10:37.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.429+0000 7fb296b9d700 1 -- 172.21.15.27:0/3107442394 >> [v2:172.21.15.27:6809/41833,v1:172.21.15.27:6811/41833] conn(0x7fb27406f0a0 msgr2=0x7fb2740714b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.429+0000 7fb296b9d700 1 --2- 172.21.15.27:0/3107442394 >> [v2:172.21.15.27:6809/41833,v1:172.21.15.27:6811/41833] conn(0x7fb27406f0a0 0x7fb2740714b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.432+0000 7fb296b9d700 1 -- 172.21.15.27:0/3107442394 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb274067080 msgr2=0x7fb274069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.432+0000 7fb296b9d700 1 --2- 172.21.15.27:0/3107442394 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb274067080 0x7fb274069530 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009fa0 tx=0x7fb28c009120 comp rx=0 tx=0).stop 2024-01-11T08:10:37.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.432+0000 7fb296b9d700 1 -- 172.21.15.27:0/3107442394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2880dfaf0 msgr2=0x7fb2880f6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.432+0000 7fb296b9d700 1 --2- 172.21.15.27:0/3107442394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2880dfaf0 0x7fb2880f6ee0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb29004d1b0 tx=0x7fb2900668e0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.432+0000 7fb29639c700 1 -- 172.21.15.27:0/3107442394 reap_dead start 2024-01-11T08:10:37.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.432+0000 7fb296b9d700 1 -- 172.21.15.27:0/3107442394 shutdown_connections 2024-01-11T08:10:37.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.432+0000 7fb296b9d700 1 -- 172.21.15.27:0/3107442394 >> 172.21.15.27:0/3107442394 conn(0x7fb2880517b0 msgr2=0x7fb288036d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.433+0000 7f60071ff700 1 -- 172.21.15.27:0/209996211 --> [v2:172.21.15.27:6800/41829,v1:172.21.15.27:6801/41829] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f600004c3c0 con 0x7f5fe006f1c0 2024-01-11T08:10:37.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.433+0000 7fb296b9d700 1 -- 172.21.15.27:0/3107442394 shutdown_connections 2024-01-11T08:10:37.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.433+0000 7fb296b9d700 1 -- 172.21.15.27:0/3107442394 wait complete. 2024-01-11T08:10:37.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.433+0000 7fa5449ff700 1 -- 172.21.15.27:0/1352661556 --> [v2:172.21.15.27:6813/41835,v1:172.21.15.27:6816/41835] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa524002d30 con 0x7fa52006f220 2024-01-11T08:10:37.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.433+0000 7f5ffd7fa700 1 -- 172.21.15.27:0/209996211 <== osd.0 v2:172.21.15.27:6800/41829 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f600004c3c0 con 0x7f5fe006f1c0 2024-01-11T08:10:37.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.434+0000 7f60071ff700 1 -- 172.21.15.27:0/209996211 >> [v2:172.21.15.27:6800/41829,v1:172.21.15.27:6801/41829] conn(0x7f5fe006f1c0 msgr2=0x7f5fe00715d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.434+0000 7f60071ff700 1 --2- 172.21.15.27:0/209996211 >> [v2:172.21.15.27:6800/41829,v1:172.21.15.27:6801/41829] conn(0x7f5fe006f1c0 0x7f5fe00715d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.434+0000 7fa52e7fc700 1 -- 172.21.15.27:0/1352661556 <== osd.1 v2:172.21.15.27:6813/41835 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa524002d30 con 0x7fa52006f220 2024-01-11T08:10:37.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.435+0000 7f60071ff700 1 -- 172.21.15.27:0/209996211 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5fe0066fb0 msgr2=0x7f5fe0069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.435+0000 7f60071ff700 1 --2- 172.21.15.27:0/209996211 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5fe0066fb0 0x7f5fe0069460 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00099b0 tx=0x7f5ff0017000 comp rx=0 tx=0).stop 2024-01-11T08:10:37.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.435+0000 7f60071ff700 1 -- 172.21.15.27:0/209996211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60000d0a80 msgr2=0x7f6000150110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.435+0000 7f60071ff700 1 --2- 172.21.15.27:0/209996211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60000d0a80 0x7f6000150110 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5fe800cb60 tx=0x7f5fe80017a0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.435+0000 7f600579c700 1 -- 172.21.15.27:0/209996211 reap_dead start 2024-01-11T08:10:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.435+0000 7f60071ff700 1 -- 172.21.15.27:0/209996211 shutdown_connections 2024-01-11T08:10:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.435+0000 7f60071ff700 1 -- 172.21.15.27:0/209996211 >> 172.21.15.27:0/209996211 conn(0x7f600006ed40 msgr2=0x7f60000b5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.435+0000 7f60071ff700 1 -- 172.21.15.27:0/209996211 shutdown_connections 2024-01-11T08:10:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.435+0000 7f60071ff700 1 -- 172.21.15.27:0/209996211 wait complete. 2024-01-11T08:10:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.437+0000 7fa5449ff700 1 -- 172.21.15.27:0/1352661556 >> [v2:172.21.15.27:6813/41835,v1:172.21.15.27:6816/41835] conn(0x7fa52006f220 msgr2=0x7fa520071630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.437+0000 7fa5449ff700 1 --2- 172.21.15.27:0/1352661556 >> [v2:172.21.15.27:6813/41835,v1:172.21.15.27:6816/41835] conn(0x7fa52006f220 0x7fa520071630 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.438+0000 7fa5449ff700 1 -- 172.21.15.27:0/1352661556 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa520067010 msgr2=0x7fa5200694c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.438+0000 7fa5449ff700 1 --2- 172.21.15.27:0/1352661556 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa520067010 0x7fa5200694c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa538000b60 tx=0x7fa538001c90 comp rx=0 tx=0).stop 2024-01-11T08:10:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.438+0000 7fa5449ff700 1 -- 172.21.15.27:0/1352661556 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa540159e20 msgr2=0x7fa5400c3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.438+0000 7fa5449ff700 1 --2- 172.21.15.27:0/1352661556 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa540159e20 0x7fa5400c3470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa530002db0 tx=0x7fa530015000 comp rx=0 tx=0).stop 2024-01-11T08:10:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.438+0000 7fa53ed9d700 1 -- 172.21.15.27:0/1352661556 reap_dead start 2024-01-11T08:10:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.438+0000 7fa5449ff700 1 -- 172.21.15.27:0/1352661556 shutdown_connections 2024-01-11T08:10:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.438+0000 7fa5449ff700 1 -- 172.21.15.27:0/1352661556 >> 172.21.15.27:0/1352661556 conn(0x7fa5400680a0 msgr2=0x7fa5400692c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.439+0000 7fa5449ff700 1 -- 172.21.15.27:0/1352661556 shutdown_connections 2024-01-11T08:10:37.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.439+0000 7fa5449ff700 1 -- 172.21.15.27:0/1352661556 wait complete. 2024-01-11T08:10:37.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.439+0000 7ff0b49ff700 1 -- 172.21.15.27:0/1726869050 --> [v2:172.21.15.77:6800/42095,v1:172.21.15.77:6801/42095] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff0b004c3c0 con 0x7ff09006fe20 2024-01-11T08:10:37.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.440+0000 7ff0a67fc700 1 -- 172.21.15.27:0/1726869050 <== osd.4 v2:172.21.15.77:6800/42095 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff0b004c3c0 con 0x7ff09006fe20 2024-01-11T08:10:37.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.441+0000 7ff0b49ff700 1 -- 172.21.15.27:0/1726869050 >> [v2:172.21.15.77:6800/42095,v1:172.21.15.77:6801/42095] conn(0x7ff09006fe20 msgr2=0x7ff0900722b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.441+0000 7ff0b49ff700 1 --2- 172.21.15.27:0/1726869050 >> [v2:172.21.15.77:6800/42095,v1:172.21.15.77:6801/42095] conn(0x7ff09006fe20 0x7ff0900722b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.441+0000 7ff0b49ff700 1 -- 172.21.15.27:0/1726869050 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff090066f30 msgr2=0x7ff0900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.442+0000 7ff0b49ff700 1 --2- 172.21.15.27:0/1726869050 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff090066f30 0x7ff0900693e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0146b60 tx=0x7ff0a0007d30 comp rx=0 tx=0).stop 2024-01-11T08:10:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.442+0000 7ff0b49ff700 1 -- 172.21.15.27:0/1726869050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0b00ad840 msgr2=0x7ff0b0146bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.442+0000 7ff0b49ff700 1 --2- 172.21.15.27:0/1726869050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0b00ad840 0x7ff0b0146bb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff098009930 tx=0x7ff098016000 comp rx=0 tx=0).stop 2024-01-11T08:10:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.442+0000 7ff0aed9d700 1 -- 172.21.15.27:0/1726869050 reap_dead start 2024-01-11T08:10:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.442+0000 7ff0b49ff700 1 -- 172.21.15.27:0/1726869050 shutdown_connections 2024-01-11T08:10:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.442+0000 7ff0b49ff700 1 -- 172.21.15.27:0/1726869050 >> 172.21.15.27:0/1726869050 conn(0x7ff0b00680a0 msgr2=0x7ff0b00c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.443+0000 7ff0b49ff700 1 -- 172.21.15.27:0/1726869050 shutdown_connections 2024-01-11T08:10:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.443+0000 7ff0b49ff700 1 -- 172.21.15.27:0/1726869050 wait complete. 2024-01-11T08:10:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.443+0000 7f87a0bff700 1 -- 172.21.15.27:0/1560388346 --> [v2:172.21.15.27:6808/41831,v1:172.21.15.27:6810/41831] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f87940052b0 con 0x7f877c06f290 2024-01-11T08:10:37.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.444+0000 7f878a7fc700 1 -- 172.21.15.27:0/1560388346 <== osd.3 v2:172.21.15.27:6808/41831 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f87940052b0 con 0x7f877c06f290 2024-01-11T08:10:37.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.445+0000 7f87a0bff700 1 -- 172.21.15.27:0/1560388346 >> [v2:172.21.15.27:6808/41831,v1:172.21.15.27:6810/41831] conn(0x7f877c06f290 msgr2=0x7f877c0716a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.445+0000 7f87a0bff700 1 --2- 172.21.15.27:0/1560388346 >> [v2:172.21.15.27:6808/41831,v1:172.21.15.27:6810/41831] conn(0x7f877c06f290 0x7f877c0716a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.445+0000 7f87a0bff700 1 -- 172.21.15.27:0/1560388346 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f877c067080 msgr2=0x7f877c069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.445+0000 7f87a0bff700 1 --2- 172.21.15.27:0/1560388346 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f877c067080 0x7f877c069530 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8790000c20 tx=0x7f8790007040 comp rx=0 tx=0).stop 2024-01-11T08:10:37.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.445+0000 7f87a0bff700 1 -- 172.21.15.27:0/1560388346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8784003a60 msgr2=0x7f87940f6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.445+0000 7f87a0bff700 1 --2- 172.21.15.27:0/1560388346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8784003a60 0x7f87940f6a30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f878c019000 tx=0x7f878c01a000 comp rx=0 tx=0).stop 2024-01-11T08:10:37.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.445+0000 7f879ad9d700 1 -- 172.21.15.27:0/1560388346 reap_dead start 2024-01-11T08:10:37.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.445+0000 7f87a0bff700 1 -- 172.21.15.27:0/1560388346 shutdown_connections 2024-01-11T08:10:37.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.445+0000 7f87a0bff700 1 -- 172.21.15.27:0/1560388346 >> 172.21.15.27:0/1560388346 conn(0x7f8794054c20 msgr2=0x7f87940ea100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.446+0000 7f87a0bff700 1 -- 172.21.15.27:0/1560388346 shutdown_connections 2024-01-11T08:10:37.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.446+0000 7f87a0bff700 1 -- 172.21.15.27:0/1560388346 wait complete. 2024-01-11T08:10:37.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.453+0000 7f7ed91ff700 1 -- 172.21.15.27:0/4173508689 --> [v2:172.21.15.77:6810/42096,v1:172.21.15.77:6812/42096] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7eb4002d30 con 0x7f7ec006f190 2024-01-11T08:10:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.456+0000 7f4a2e1ff700 1 -- 172.21.15.27:0/962413262 --> [v2:172.21.15.77:6824/42140,v1:172.21.15.77:6825/42140] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4a2806a4d0 con 0x7f4a0806f140 2024-01-11T08:10:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.456+0000 7f7ebaffd700 1 -- 172.21.15.27:0/4173508689 <== osd.5 v2:172.21.15.77:6810/42096 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7eb4002d30 con 0x7f7ec006f190 2024-01-11T08:10:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.457+0000 7f4a1ffff700 1 -- 172.21.15.27:0/962413262 <== osd.7 v2:172.21.15.77:6824/42140 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4a2806a4d0 con 0x7f4a0806f140 2024-01-11T08:10:37.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.457+0000 7f7ed91ff700 1 -- 172.21.15.27:0/4173508689 >> [v2:172.21.15.77:6810/42096,v1:172.21.15.77:6812/42096] conn(0x7f7ec006f190 msgr2=0x7f7ec00715a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.457+0000 7f7ed91ff700 1 --2- 172.21.15.27:0/4173508689 >> [v2:172.21.15.77:6810/42096,v1:172.21.15.77:6812/42096] conn(0x7f7ec006f190 0x7f7ec00715a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.457+0000 7f7ed91ff700 1 -- 172.21.15.27:0/4173508689 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7ec0067000 msgr2=0x7f7ec00694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.457+0000 7f7ed91ff700 1 --2- 172.21.15.27:0/4173508689 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7ec0067000 0x7f7ec00694b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7ed40af700 tx=0x7f7ec4009040 comp rx=0 tx=0).stop 2024-01-11T08:10:37.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.457+0000 7f7ed91ff700 1 -- 172.21.15.27:0/4173508689 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ed40d0a80 msgr2=0x7f7ed406f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.457+0000 7f7ed91ff700 1 --2- 172.21.15.27:0/4173508689 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ed40d0a80 0x7f7ed406f650 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc016000 tx=0x7f7ecc017000 comp rx=0 tx=0).stop 2024-01-11T08:10:37.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.457+0000 7f7ed359e700 1 -- 172.21.15.27:0/4173508689 reap_dead start 2024-01-11T08:10:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.457+0000 7f7ed91ff700 1 -- 172.21.15.27:0/4173508689 shutdown_connections 2024-01-11T08:10:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.457+0000 7f7ed91ff700 1 -- 172.21.15.27:0/4173508689 >> 172.21.15.27:0/4173508689 conn(0x7f7ed406ed40 msgr2=0x7f7ed4158360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.458+0000 7f7ed91ff700 1 -- 172.21.15.27:0/4173508689 shutdown_connections 2024-01-11T08:10:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.458+0000 7f7ed91ff700 1 -- 172.21.15.27:0/4173508689 wait complete. 2024-01-11T08:10:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.458+0000 7f4a2e1ff700 1 -- 172.21.15.27:0/962413262 >> [v2:172.21.15.77:6824/42140,v1:172.21.15.77:6825/42140] conn(0x7f4a0806f140 msgr2=0x7f4a08071550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.458+0000 7f4a2e1ff700 1 --2- 172.21.15.27:0/962413262 >> [v2:172.21.15.77:6824/42140,v1:172.21.15.77:6825/42140] conn(0x7f4a0806f140 0x7f4a08071550 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.458+0000 7f4a2e1ff700 1 -- 172.21.15.27:0/962413262 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4a08066f30 msgr2=0x7f4a080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.458+0000 7f4a2e1ff700 1 --2- 172.21.15.27:0/962413262 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4a08066f30 0x7f4a080693e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4a280b0b80 tx=0x7f4a100091d0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.459+0000 7f4a2e1ff700 1 -- 172.21.15.27:0/962413262 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4a2815d9c0 msgr2=0x7f4a2816f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.459+0000 7f4a2e1ff700 1 --2- 172.21.15.27:0/962413262 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4a2815d9c0 0x7f4a2816f7b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000cb90 tx=0x7f4a20001960 comp rx=0 tx=0).stop 2024-01-11T08:10:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.459+0000 7f4a27fff700 1 -- 172.21.15.27:0/962413262 reap_dead start 2024-01-11T08:10:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.459+0000 7f4a2e1ff700 1 -- 172.21.15.27:0/962413262 shutdown_connections 2024-01-11T08:10:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.459+0000 7f4a2e1ff700 1 -- 172.21.15.27:0/962413262 >> 172.21.15.27:0/962413262 conn(0x7f4a280680b0 msgr2=0x7f4a28151740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.460+0000 7f4a2e1ff700 1 -- 172.21.15.27:0/962413262 shutdown_connections 2024-01-11T08:10:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.460+0000 7f4a2e1ff700 1 -- 172.21.15.27:0/962413262 wait complete. 2024-01-11T08:10:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.460+0000 7f3fb8fff700 1 -- 172.21.15.27:0/1354074089 --> [v2:172.21.15.77:6808/42097,v1:172.21.15.77:6809/42097] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3fb414ef90 con 0x7f3f9806f210 2024-01-11T08:10:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.461+0000 7f3faaffd700 1 -- 172.21.15.27:0/1354074089 <== osd.6 v2:172.21.15.77:6808/42097 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3fb414ef90 con 0x7f3f9806f210 2024-01-11T08:10:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.462+0000 7f3fb8fff700 1 -- 172.21.15.27:0/1354074089 >> [v2:172.21.15.77:6808/42097,v1:172.21.15.77:6809/42097] conn(0x7f3f9806f210 msgr2=0x7f3f98071620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.462+0000 7f3fb8fff700 1 --2- 172.21.15.27:0/1354074089 >> [v2:172.21.15.77:6808/42097,v1:172.21.15.77:6809/42097] conn(0x7f3f9806f210 0x7f3f98071620 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.463+0000 7f3fb8fff700 1 -- 172.21.15.27:0/1354074089 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3f98067000 msgr2=0x7f3f980694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.463+0000 7f3fb8fff700 1 --2- 172.21.15.27:0/1354074089 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3f98067000 0x7f3f980694b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3fa40099b0 tx=0x7f3fa4019000 comp rx=0 tx=0).stop 2024-01-11T08:10:37.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.463+0000 7f3fb8fff700 1 -- 172.21.15.27:0/1354074089 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3fb4156ab0 msgr2=0x7f3fb40ad370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.463+0000 7f3fb8fff700 1 --2- 172.21.15.27:0/1354074089 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3fb4156ab0 0x7f3fb40ad370 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3fac00b5e0 tx=0x7f3fac009120 comp rx=0 tx=0).stop 2024-01-11T08:10:37.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.463+0000 7f3fb2d9d700 1 -- 172.21.15.27:0/1354074089 reap_dead start 2024-01-11T08:10:37.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.464+0000 7f3fb8fff700 1 -- 172.21.15.27:0/1354074089 shutdown_connections 2024-01-11T08:10:37.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.464+0000 7f3fb8fff700 1 -- 172.21.15.27:0/1354074089 >> 172.21.15.27:0/1354074089 conn(0x7f3fb40680b0 msgr2=0x7f3fb414bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.464+0000 7f3fb8fff700 1 -- 172.21.15.27:0/1354074089 shutdown_connections 2024-01-11T08:10:37.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.464+0000 7f3fb8fff700 1 -- 172.21.15.27:0/1354074089 wait complete. 2024-01-11T08:10:37.501 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2024-01-11T08:10:37.501 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-11T08:10:37.502 INFO:teuthology.orchestra.run.smithi027.stdout:55834574854 2024-01-11T08:10:37.503 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-11T08:10:37.504 INFO:teuthology.orchestra.run.smithi027.stdout:55834574856 2024-01-11T08:10:37.504 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-11T08:10:37.505 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2024-01-11T08:10:37.506 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-11T08:10:37.506 INFO:teuthology.orchestra.run.smithi027.stdout:55834574854 2024-01-11T08:10:37.507 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-11T08:10:37.507 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2024-01-11T08:10:37.508 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-11T08:10:37.508 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2024-01-11T08:10:37.509 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-11T08:10:37.510 INFO:teuthology.orchestra.run.smithi027.stdout:55834574854 2024-01-11T08:10:37.510 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-11T08:10:37.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.915+0000 7f5ddf5ff700 1 Processor -- start 2024-01-11T08:10:37.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.915+0000 7f5ddf5ff700 1 -- start start 2024-01-11T08:10:37.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.917+0000 7f5ddf5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5dd815cde0 0x7f5dd815a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.917+0000 7f5ddf5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd80b4bd0 con 0x7f5dd815ca10 2024-01-11T08:10:37.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.917+0000 7f5ddf5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd815ac70 con 0x7f5dd8068cd0 2024-01-11T08:10:37.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.917+0000 7f5ddf5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd80b3fc0 con 0x7f5dd815cde0 2024-01-11T08:10:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.918+0000 7f5ddd39b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5dd815ca10 0x7f5dd80685c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37180/0 (socket says 172.21.15.27:37180) 2024-01-11T08:10:37.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.919+0000 7f5ddd39b700 1 -- 172.21.15.27:0/2504180226 learned_addr learned my addr 172.21.15.27:0/2504180226 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:37.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.919+0000 7f5dddb9c700 1 --2- 172.21.15.27:0/2504180226 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5dd815cde0 0x7f5dd815a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.920+0000 7f5dcbfff700 1 -- 172.21.15.27:0/2504180226 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276496045 0 0) 0x7f5dd815ac70 con 0x7f5dd8068cd0 2024-01-11T08:10:37.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.920+0000 7f5dddb9c700 1 -- 172.21.15.27:0/2504180226 >> v1:172.21.15.77:6789/0 conn(0x7f5dd8068cd0 legacy=0x7f5dd80b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.920+0000 7f5dddb9c700 1 -- 172.21.15.27:0/2504180226 >> v1:172.21.15.27:6789/0 conn(0x7f5dd815ca10 legacy=0x7f5dd80685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.920+0000 7f5dddb9c700 1 -- 172.21.15.27:0/2504180226 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd80b42a0 con 0x7f5dd815cde0 2024-01-11T08:10:37.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.920+0000 7f5ddd39b700 1 --1- 172.21.15.27:0/2504180226 >> v1:172.21.15.27:6789/0 conn(0x7f5dd815ca10 0x7f5dd80685c0 :-1 s=CLOSED pgs=0 cs=0 l=0).handle_server_banner_and_identify state changed while learned_addr, mark_down or replacing must be happened just now 2024-01-11T08:10:37.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.921+0000 7f5dddb9c700 1 --2- 172.21.15.27:0/2504180226 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5dd815cde0 0x7f5dd815a730 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc009940 tx=0x7f5dcc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0708194b2d7c84f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.922+0000 7f5dcbfff700 1 -- 172.21.15.27:0/2504180226 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc00b7e0 con 0x7f5dd815cde0 2024-01-11T08:10:37.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.922+0000 7f5dcbfff700 1 -- 172.21.15.27:0/2504180226 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dcc00b940 con 0x7f5dd815cde0 2024-01-11T08:10:37.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.922+0000 7f5dcbfff700 1 -- 172.21.15.27:0/2504180226 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc00b7e0 con 0x7f5dd815cde0 2024-01-11T08:10:37.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.923+0000 7f64017ff700 1 Processor -- start 2024-01-11T08:10:37.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.923+0000 7f64017ff700 1 -- start start 2024-01-11T08:10:37.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.924+0000 7f5ddf5ff700 1 -- 172.21.15.27:0/2504180226 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5dd815cde0 msgr2=0x7f5dd815a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.924+0000 7f5ddf5ff700 1 --2- 172.21.15.27:0/2504180226 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5dd815cde0 0x7f5dd815a730 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc009940 tx=0x7f5dcc0091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.924+0000 7f6555b9d700 1 Processor -- start 2024-01-11T08:10:37.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.925+0000 7f64017ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f63fc153cd0 0x7f63fc15a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.926+0000 7f64017ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63fc15d300 0x7f63fc0ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.926+0000 7f64017ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f63fc0cd310 0x7f63fc068570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.926+0000 7f64017ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63fc0cebd0 con 0x7f63fc0cd310 2024-01-11T08:10:37.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.926+0000 7f64017ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63fc0ce8d0 con 0x7f63fc153cd0 2024-01-11T08:10:37.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.926+0000 7f64017ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc0cd790 con 0x7f63fc15d300 2024-01-11T08:10:37.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.926+0000 7f63f27fc700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63fc15d300 0x7f63fc0ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.926+0000 7f63f27fc700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63fc15d300 0x7f63fc0ccdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:59620/0 (socket says 172.21.15.27:59620) 2024-01-11T08:10:37.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.927+0000 7f63f27fc700 1 -- 172.21.15.27:0/3566855160 learned_addr learned my addr 172.21.15.27:0/3566855160 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.927+0000 7f63faffd700 1 --2- 172.21.15.27:0/3566855160 >> v2:172.21.15.77:3300/0 conn(0x7f63fc153cd0 0x7f63fc15a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.927+0000 7f63fb7fe700 1 --2- 172.21.15.27:0/3566855160 >> v2:172.21.15.27:3300/0 conn(0x7f63fc0cd310 0x7f63fc068570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.925+0000 7f6555b9d700 1 -- start start 2024-01-11T08:10:37.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.927+0000 7f6555b9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65480e0bc0 0x7f65480ea2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.927+0000 7f6555b9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f65480e0f90 0x7f65480ea800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.927+0000 7f6555b9d700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6548037710 con 0x7f65480068c0 2024-01-11T08:10:37.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.927+0000 7f6555b9d700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65480eb160 con 0x7f65480e0f90 2024-01-11T08:10:37.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.927+0000 7f6555b9d700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65480535f0 con 0x7f65480e0bc0 2024-01-11T08:10:37.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.927+0000 7f63f27fc700 1 -- 172.21.15.27:0/3566855160 >> v2:172.21.15.77:3300/0 conn(0x7f63fc153cd0 msgr2=0x7f63fc15a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.927+0000 7f63f27fc700 1 --2- 172.21.15.27:0/3566855160 >> v2:172.21.15.77:3300/0 conn(0x7f63fc153cd0 0x7f63fc15a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.927+0000 7f63f27fc700 1 -- 172.21.15.27:0/3566855160 >> v2:172.21.15.27:3300/0 conn(0x7f63fc0cd310 msgr2=0x7f63fc068570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.927+0000 7f45ac3ff700 1 Processor -- start 2024-01-11T08:10:37.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.928+0000 7f45ac3ff700 1 -- start start 2024-01-11T08:10:37.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.928+0000 7f5ddf5ff700 1 -- 172.21.15.27:0/2504180226 shutdown_connections 2024-01-11T08:10:37.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.928+0000 7f5ddf5ff700 1 --2- 172.21.15.27:0/2504180226 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5dd815cde0 0x7f5dd815a730 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.928+0000 7f5ddf5ff700 1 -- 172.21.15.27:0/2504180226 >> 172.21.15.27:0/2504180226 conn(0x7f5dd806ed40 msgr2=0x7f5dd806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.928+0000 7f654ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f65480e0f90 0x7f65480ea800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.928+0000 7f63f27fc700 1 --2- 172.21.15.27:0/3566855160 >> v2:172.21.15.27:3300/0 conn(0x7f63fc0cd310 0x7f63fc068570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.928+0000 7f63f27fc700 1 -- 172.21.15.27:0/3566855160 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63fc15db50 con 0x7f63fc15d300 2024-01-11T08:10:37.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.928+0000 7f655539c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f65480068c0 0x7f6548006d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37186/0 (socket says 172.21.15.27:37186) 2024-01-11T08:10:37.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.929+0000 7f655539c700 1 -- 172.21.15.27:0/2166438318 learned_addr learned my addr 172.21.15.27:0/2166438318 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:37.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.929+0000 7f5ddf5ff700 1 -- 172.21.15.27:0/2504180226 shutdown_connections 2024-01-11T08:10:37.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.929+0000 7f63fb7fe700 1 --2- 172.21.15.27:0/3566855160 >> v2:172.21.15.27:3300/0 conn(0x7f63fc0cd310 0x7f63fc068570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:37.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.929+0000 7f5ddf5ff700 1 -- 172.21.15.27:0/2504180226 wait complete. 2024-01-11T08:10:37.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.928+0000 7f6554b9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65480e0bc0 0x7f65480ea2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.930+0000 7f654ffff700 1 -- 172.21.15.27:0/2166438318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65480e0bc0 msgr2=0x7f65480ea2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.930+0000 7f654ffff700 1 --2- 172.21.15.27:0/2166438318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65480e0bc0 0x7f65480ea2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.930+0000 7f654ffff700 1 -- 172.21.15.27:0/2166438318 >> v1:172.21.15.27:6789/0 conn(0x7f65480068c0 legacy=0x7f6548006d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.930+0000 7f654ffff700 1 -- 172.21.15.27:0/2166438318 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6548054a90 con 0x7f65480e0f90 2024-01-11T08:10:37.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.930+0000 7f63f27fc700 1 --2- 172.21.15.27:0/3566855160 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63fc15d300 0x7f63fc0ccdd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f63fc0d16b0 tx=0x7f63ec009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7741164ac73a9540 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.930+0000 7f6554b9b700 1 --2- 172.21.15.27:0/2166438318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65480e0bc0 0x7f65480ea2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:37.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.931+0000 7f654ffff700 1 --2- 172.21.15.27:0/2166438318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65480e0f90 0x7f65480ea800 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f65440098e0 tx=0x7f65440091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a49604085772059 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.931+0000 7f654f7fe700 1 -- 172.21.15.27:0/2166438318 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6544009e80 con 0x7f65480e0f90 2024-01-11T08:10:37.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.931+0000 7f654f7fe700 1 -- 172.21.15.27:0/2166438318 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654400b860 con 0x7f65480e0f90 2024-01-11T08:10:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.931+0000 7f654f7fe700 1 -- 172.21.15.27:0/2166438318 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654400bbe0 con 0x7f65480e0f90 2024-01-11T08:10:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.931+0000 7f63fa7fc700 1 -- 172.21.15.27:0/3566855160 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec009e70 con 0x7f63fc15d300 2024-01-11T08:10:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.931+0000 7f63fa7fc700 1 -- 172.21.15.27:0/3566855160 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63ec00b860 con 0x7f63fc15d300 2024-01-11T08:10:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.931+0000 7f63fa7fc700 1 -- 172.21.15.27:0/3566855160 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec00bc00 con 0x7f63fc15d300 2024-01-11T08:10:37.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.932+0000 7f55625ff700 1 Processor -- start 2024-01-11T08:10:37.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.932+0000 7f5ddf5ff700 1 Processor -- start 2024-01-11T08:10:37.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.933+0000 7f55625ff700 1 -- start start 2024-01-11T08:10:37.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.930+0000 7f45ac3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45a40ad770 0x7f45a4164620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.930+0000 7f45ac3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a40ace10 con 0x7f45a4159cf0 2024-01-11T08:10:37.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.930+0000 7f45ac3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a40a99d0 con 0x7f45a4142e50 2024-01-11T08:10:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.930+0000 7f45ac3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a40ccb80 con 0x7f45a40ad770 2024-01-11T08:10:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.931+0000 7f45aa19b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f45a4142e50 0x7f45a40abd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:60758/0 (socket says 172.21.15.27:60758) 2024-01-11T08:10:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.931+0000 7f45aa99c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45a40ad770 0x7f45a4164620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.931+0000 7f45aa19b700 1 -- 172.21.15.27:0/3865167418 learned_addr learned my addr 172.21.15.27:0/3865167418 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.932+0000 7f45a9199700 1 -- 172.21.15.27:0/3865167418 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862667727 0 0) 0x7f45a40a99d0 con 0x7f45a4142e50 2024-01-11T08:10:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.932+0000 7f45a9199700 1 -- 172.21.15.27:0/3865167418 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4594003570 con 0x7f45a4142e50 2024-01-11T08:10:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.932+0000 7f45a9199700 1 -- 172.21.15.27:0/3865167418 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390078142 0 0) 0x7f45a40ace10 con 0x7f45a4159cf0 2024-01-11T08:10:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.932+0000 7f45a9199700 1 -- 172.21.15.27:0/3865167418 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45a40a99d0 con 0x7f45a4159cf0 2024-01-11T08:10:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.932+0000 7f45a9199700 1 -- 172.21.15.27:0/3865167418 <== mon.1 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 871979465 0 0) 0x7f4594003570 con 0x7f45a4142e50 2024-01-11T08:10:37.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.933+0000 7f1884dff700 1 Processor -- start 2024-01-11T08:10:37.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.932+0000 7f45a9199700 1 -- 172.21.15.27:0/3865167418 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45a40ad770 msgr2=0x7f45a4164620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.932+0000 7f45a9199700 1 --2- 172.21.15.27:0/3865167418 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45a40ad770 0x7f45a4164620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.932+0000 7f45a9199700 1 -- 172.21.15.27:0/3865167418 >> v1:172.21.15.27:6789/0 conn(0x7f45a4159cf0 legacy=0x7f45a40ad060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.932+0000 7f45a9199700 1 -- 172.21.15.27:0/3865167418 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a40ac570 con 0x7f45a4142e50 2024-01-11T08:10:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.932+0000 7f45a9199700 1 -- 172.21.15.27:0/3865167418 <== mon.1 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f4598001830 con 0x7f45a4142e50 2024-01-11T08:10:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.932+0000 7f45a9199700 1 -- 172.21.15.27:0/3865167418 >> v1:172.21.15.77:6789/0 conn(0x7f45a4142e50 legacy=0x7f45a40abd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.932+0000 7f45a9199700 1 --2- 172.21.15.27:0/3865167418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4594003bf0 0x7f4594003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.932+0000 7f45a9199700 1 --2- 172.21.15.27:0/3865167418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4594004810 0x7f4594004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.932+0000 7f45a9199700 1 --2- 172.21.15.27:0/3865167418 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4594005210 0x7f459401ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.932+0000 7f45a9199700 1 -- 172.21.15.27:0/3865167418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a40ccb80 con 0x7f4594004810 2024-01-11T08:10:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.932+0000 7f45a9199700 1 -- 172.21.15.27:0/3865167418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459401d140 con 0x7f4594003bf0 2024-01-11T08:10:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.932+0000 7f45a9199700 1 -- 172.21.15.27:0/3865167418 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459401d300 con 0x7f4594005210 2024-01-11T08:10:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.933+0000 7f45aa99c700 1 --2- 172.21.15.27:0/3865167418 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4594005210 0x7f459401ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.933+0000 7f45aa19b700 1 --2- 172.21.15.27:0/3865167418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4594003bf0 0x7f4594003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.934+0000 7f45aa19b700 1 -- 172.21.15.27:0/3865167418 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4594005210 msgr2=0x7f459401ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.933+0000 7f5ddf5ff700 1 -- start start 2024-01-11T08:10:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.934+0000 7f5ddf5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dd8068cd0 0x7f5dd8148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.934+0000 7f5ddf5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dd815ca10 0x7f5dd80c6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.934+0000 7f5ddf5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5dd815cde0 0x7f5dd80c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.934+0000 7f5ddf5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd80c7520 con 0x7f5dd815ca10 2024-01-11T08:10:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.934+0000 7f5ddf5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd81559c0 con 0x7f5dd8068cd0 2024-01-11T08:10:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.934+0000 7f5ddf5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd81485c0 con 0x7f5dd815cde0 2024-01-11T08:10:37.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.934+0000 7f64017ff700 1 -- 172.21.15.27:0/3566855160 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63fc15d300 msgr2=0x7f63fc0ccdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.934+0000 7f64017ff700 1 --2- 172.21.15.27:0/3566855160 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63fc15d300 0x7f63fc0ccdd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f63fc0d16b0 tx=0x7f63ec009250 comp rx=0 tx=0).stop 2024-01-11T08:10:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.934+0000 7f45aa19b700 1 --2- 172.21.15.27:0/3865167418 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4594005210 0x7f459401ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.934+0000 7f45aa19b700 1 -- 172.21.15.27:0/3865167418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4594004810 msgr2=0x7f4594004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.934+0000 7f45a999a700 1 --2- 172.21.15.27:0/3865167418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4594004810 0x7f4594004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.934+0000 7f45aa19b700 1 --2- 172.21.15.27:0/3865167418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4594004810 0x7f4594004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.934+0000 7f45aa19b700 1 -- 172.21.15.27:0/3865167418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a40ac570 con 0x7f4594003bf0 2024-01-11T08:10:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.934+0000 7f45a999a700 1 --2- 172.21.15.27:0/3865167418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4594004810 0x7f4594004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.934+0000 7f45aa19b700 1 --2- 172.21.15.27:0/3865167418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4594003bf0 0x7f4594003fe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f45980099c0 tx=0x7f4598009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f27fec5b8f0c1e9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.933+0000 7f1884dff700 1 -- start start 2024-01-11T08:10:37.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.935+0000 7f45a9199700 1 -- 172.21.15.27:0/3865167418 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459800d2f0 con 0x7f4594003bf0 2024-01-11T08:10:37.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.935+0000 7f45a9199700 1 -- 172.21.15.27:0/3865167418 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459800e4b0 con 0x7f4594003bf0 2024-01-11T08:10:37.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.935+0000 7f45a9199700 1 -- 172.21.15.27:0/3865167418 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459800e850 con 0x7f4594003bf0 2024-01-11T08:10:37.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.935+0000 7f5ddd39b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dd8068cd0 0x7f5dd8148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.935+0000 7f55625ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f555c06f570 0x7f555c0afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.935+0000 7f55625ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f555c0cc960 0x7f555c0adb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.935+0000 7f55625ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f555c0ae070 0x7f555c155a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.935+0000 7f55625ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f555c0ad6a0 con 0x7f555c0cc960 2024-01-11T08:10:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.935+0000 7f55625ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f555c0ae510 con 0x7f555c0ae070 2024-01-11T08:10:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.935+0000 7f55625ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f555c15b2b0 con 0x7f555c06f570 2024-01-11T08:10:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.935+0000 7f555bfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f555c06f570 0x7f555c0afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.935+0000 7f555bfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f555c06f570 0x7f555c0afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:59652/0 (socket says 172.21.15.27:59652) 2024-01-11T08:10:37.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.935+0000 7f555bfff700 1 -- 172.21.15.27:0/1864898408 learned_addr learned my addr 172.21.15.27:0/1864898408 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f5560b9c700 1 --2- 172.21.15.27:0/1864898408 >> v2:172.21.15.77:3300/0 conn(0x7f555c0ae070 0x7f555c155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.935+0000 7f5ddd39b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dd8068cd0 0x7f5dd8148080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:58554/0 (socket says 172.21.15.27:58554) 2024-01-11T08:10:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f555bfff700 1 -- 172.21.15.27:0/1864898408 >> v2:172.21.15.77:3300/0 conn(0x7f555c0ae070 msgr2=0x7f555c155a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f555bfff700 1 --2- 172.21.15.27:0/1864898408 >> v2:172.21.15.77:3300/0 conn(0x7f555c0ae070 0x7f555c155a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f55537fe700 1 --2- 172.21.15.27:0/1864898408 >> v2:172.21.15.27:3300/0 conn(0x7f555c0cc960 0x7f555c0adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f555bfff700 1 -- 172.21.15.27:0/1864898408 >> v2:172.21.15.27:3300/0 conn(0x7f555c0cc960 msgr2=0x7f555c0adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f555bfff700 1 --2- 172.21.15.27:0/1864898408 >> v2:172.21.15.27:3300/0 conn(0x7f555c0cc960 0x7f555c0adb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f555bfff700 1 -- 172.21.15.27:0/1864898408 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555c0ad050 con 0x7f555c06f570 2024-01-11T08:10:37.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f55537fe700 1 --2- 172.21.15.27:0/1864898408 >> v2:172.21.15.27:3300/0 conn(0x7f555c0cc960 0x7f555c0adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f1884dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18800af9a0 0x7f1880161770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f1884dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f188015eda0 0x7f18800ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f1884dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18800cd0b0 con 0x7f188015d9c0 2024-01-11T08:10:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f1884dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1880162680 con 0x7f18800af9a0 2024-01-11T08:10:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f1884dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f188015f4d0 con 0x7f188015eda0 2024-01-11T08:10:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f5ddcb9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dd815ca10 0x7f5dd80c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f5ddcb9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dd815ca10 0x7f5dd80c6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53324/0 (socket says 172.21.15.27:53324) 2024-01-11T08:10:37.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f5ddcb9a700 1 -- 172.21.15.27:0/2650736049 learned_addr learned my addr 172.21.15.27:0/2650736049 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f5dddb9c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5dd815cde0 0x7f5dd80c6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.934+0000 7f6555b9d700 1 -- 172.21.15.27:0/2166438318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65480e0f90 msgr2=0x7f65480ea800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.934+0000 7f6555b9d700 1 --2- 172.21.15.27:0/2166438318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65480e0f90 0x7f65480ea800 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f65440098e0 tx=0x7f65440091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f555bfff700 1 --2- 172.21.15.27:0/1864898408 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f555c06f570 0x7f555c0afa90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f555c0aeff0 tx=0x7f5544009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20be89e8f7a67b97 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.937+0000 7f555b7fe700 1 -- 172.21.15.27:0/1864898408 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554400b8f0 con 0x7f555c06f570 2024-01-11T08:10:37.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.937+0000 7f555b7fe700 1 -- 172.21.15.27:0/1864898408 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554400ba50 con 0x7f555c06f570 2024-01-11T08:10:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.937+0000 7f555b7fe700 1 -- 172.21.15.27:0/1864898408 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554400bdf0 con 0x7f555c06f570 2024-01-11T08:10:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.937+0000 7f5ddcb9a700 1 -- 172.21.15.27:0/2650736049 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5dd815cde0 msgr2=0x7f5dd80c6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.937+0000 7f5ddcb9a700 1 --2- 172.21.15.27:0/2650736049 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5dd815cde0 0x7f5dd80c6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.937+0000 7f187e59c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f188015d9c0 0x7f18800af290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:60764/0 (socket says 172.21.15.27:60764) 2024-01-11T08:10:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.937+0000 7f187e59c700 1 -- 172.21.15.27:0/3706582037 learned_addr learned my addr 172.21.15.27:0/3706582037 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.937+0000 7f1875d9b700 1 --2- 172.21.15.27:0/3706582037 >> v2:172.21.15.27:3300/0 conn(0x7f18800af9a0 0x7f1880161770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.937+0000 7f187ed9d700 1 --2- 172.21.15.27:0/3706582037 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f188015eda0 0x7f18800ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.937+0000 7f5ddcb9a700 1 -- 172.21.15.27:0/2650736049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dd8068cd0 msgr2=0x7f5dd8148080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f45ac3ff700 1 -- 172.21.15.27:0/3865167418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4594003bf0 msgr2=0x7f4594003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.936+0000 7f45ac3ff700 1 --2- 172.21.15.27:0/3865167418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4594003bf0 0x7f4594003fe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f45980099c0 tx=0x7f4598009060 comp rx=0 tx=0).stop 2024-01-11T08:10:37.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.937+0000 7f45aa99c700 1 -- 172.21.15.27:0/3865167418 reap_dead start 2024-01-11T08:10:37.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.937+0000 7f5ddcb9a700 1 --2- 172.21.15.27:0/2650736049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dd8068cd0 0x7f5dd8148080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f55625ff700 1 -- 172.21.15.27:0/1864898408 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f555c06f570 msgr2=0x7f555c0afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f55625ff700 1 --2- 172.21.15.27:0/1864898408 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f555c06f570 0x7f555c0afa90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f555c0aeff0 tx=0x7f5544009250 comp rx=0 tx=0).stop 2024-01-11T08:10:37.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.937+0000 7f187dd9b700 1 -- 172.21.15.27:0/3706582037 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316820753 0 0) 0x7f1870001110 con 0x7f188015d9c0 2024-01-11T08:10:37.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f45ac3ff700 1 -- 172.21.15.27:0/3865167418 shutdown_connections 2024-01-11T08:10:37.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f45ac3ff700 1 -- 172.21.15.27:0/3865167418 >> 172.21.15.27:0/3865167418 conn(0x7f45a40680a0 msgr2=0x7f45a40cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f55625ff700 1 -- 172.21.15.27:0/1864898408 shutdown_connections 2024-01-11T08:10:37.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f55625ff700 1 --2- 172.21.15.27:0/1864898408 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f555c06f570 0x7f555c0afa90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f64017ff700 1 -- 172.21.15.27:0/3566855160 shutdown_connections 2024-01-11T08:10:37.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f64017ff700 1 --2- 172.21.15.27:0/3566855160 >> v2:172.21.15.27:3300/0 conn(0x7f63fc0cd310 0x7f63fc068570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f64017ff700 1 --2- 172.21.15.27:0/3566855160 >> v2:172.21.15.77:3300/0 conn(0x7f63fc153cd0 0x7f63fc15a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f55625ff700 1 --2- 172.21.15.27:0/1864898408 >> v2:172.21.15.77:3300/0 conn(0x7f555c0ae070 0x7f555c155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f55625ff700 1 --2- 172.21.15.27:0/1864898408 >> v2:172.21.15.27:3300/0 conn(0x7f555c0cc960 0x7f555c0adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f64017ff700 1 --2- 172.21.15.27:0/3566855160 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63fc15d300 0x7f63fc0ccdd0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f64017ff700 1 -- 172.21.15.27:0/3566855160 >> 172.21.15.27:0/3566855160 conn(0x7f63fc06ed40 msgr2=0x7f63fc06f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f45ac3ff700 1 -- 172.21.15.27:0/3865167418 shutdown_connections 2024-01-11T08:10:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f55625ff700 1 -- 172.21.15.27:0/1864898408 >> 172.21.15.27:0/1864898408 conn(0x7f555c0680a0 msgr2=0x7f555c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f6555b9d700 1 -- 172.21.15.27:0/2166438318 shutdown_connections 2024-01-11T08:10:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f6555b9d700 1 --2- 172.21.15.27:0/2166438318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65480e0bc0 0x7f65480ea2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f6555b9d700 1 --2- 172.21.15.27:0/2166438318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65480e0f90 0x7f65480ea800 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f6555b9d700 1 -- 172.21.15.27:0/2166438318 >> 172.21.15.27:0/2166438318 conn(0x7f6548054d40 msgr2=0x7f6548055140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f45ac3ff700 1 -- 172.21.15.27:0/3865167418 wait complete. 2024-01-11T08:10:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f187dd9b700 1 -- 172.21.15.27:0/3706582037 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f186c003570 con 0x7f188015d9c0 2024-01-11T08:10:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f5ddcb9a700 1 -- 172.21.15.27:0/2650736049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dcc009420 con 0x7f5dd815ca10 2024-01-11T08:10:37.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f55625ff700 1 -- 172.21.15.27:0/1864898408 shutdown_connections 2024-01-11T08:10:37.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f64017ff700 1 -- 172.21.15.27:0/3566855160 shutdown_connections 2024-01-11T08:10:37.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f18b3fff700 1 Processor -- start 2024-01-11T08:10:37.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f187ed9d700 1 -- 172.21.15.27:0/3706582037 >> v2:172.21.15.27:3300/0 conn(0x7f18800af9a0 msgr2=0x7f1880161770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f187ed9d700 1 --2- 172.21.15.27:0/3706582037 >> v2:172.21.15.27:3300/0 conn(0x7f18800af9a0 0x7f1880161770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f187ed9d700 1 -- 172.21.15.27:0/3706582037 >> v1:172.21.15.77:6789/0 conn(0x7f188015d9c0 legacy=0x7f18800af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f187ed9d700 1 -- 172.21.15.27:0/3706582037 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1880161de0 con 0x7f188015eda0 2024-01-11T08:10:37.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f55625ff700 1 -- 172.21.15.27:0/1864898408 wait complete. 2024-01-11T08:10:37.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f6555b9d700 1 -- 172.21.15.27:0/2166438318 shutdown_connections 2024-01-11T08:10:37.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f5ddd39b700 1 --2- 172.21.15.27:0/2650736049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dd8068cd0 0x7f5dd8148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:37.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.939+0000 7f18b3fff700 1 -- start start 2024-01-11T08:10:37.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.939+0000 7f5ddcb9a700 1 --2- 172.21.15.27:0/2650736049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dd815ca10 0x7f5dd80c6980 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5dd400c810 tx=0x7f5dd4007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f6555b9d700 1 -- 172.21.15.27:0/2166438318 wait complete. 2024-01-11T08:10:37.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.938+0000 7f64017ff700 1 -- 172.21.15.27:0/3566855160 wait complete. 2024-01-11T08:10:37.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.940+0000 7f5dc97fa700 1 -- 172.21.15.27:0/2650736049 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4011590 con 0x7f5dd815ca10 2024-01-11T08:10:37.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.940+0000 7f187ed9d700 1 --2- 172.21.15.27:0/3706582037 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f188015eda0 0x7f18800ccb70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1878009e60 tx=0x7f1878009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=690f476b2ff819f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.941+0000 7f64017ff700 1 Processor -- start 2024-01-11T08:10:37.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.941+0000 7f187dd9b700 1 -- 172.21.15.27:0/3706582037 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187800e070 con 0x7f188015eda0 2024-01-11T08:10:37.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.941+0000 7f18b3fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18ac15d9c0 0x7f18ac161770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.941+0000 7f18b3fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac15f4d0 con 0x7f18ac0ccb70 2024-01-11T08:10:37.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.941+0000 7f18b3fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac161de0 con 0x7f18ac0af9a0 2024-01-11T08:10:37.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.941+0000 7f18b3fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac162680 con 0x7f18ac15d9c0 2024-01-11T08:10:37.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.941+0000 7f187dd9b700 1 -- 172.21.15.27:0/3706582037 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f187800ba70 con 0x7f188015eda0 2024-01-11T08:10:37.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.941+0000 7f187dd9b700 1 -- 172.21.15.27:0/3706582037 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187800be30 con 0x7f188015eda0 2024-01-11T08:10:37.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.941+0000 7f6555b9d700 1 Processor -- start 2024-01-11T08:10:37.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.941+0000 7f64017ff700 1 -- start start 2024-01-11T08:10:37.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.941+0000 7f5dc97fa700 1 -- 172.21.15.27:0/2650736049 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd4011730 con 0x7f5dd815ca10 2024-01-11T08:10:37.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.941+0000 7f5dc97fa700 1 -- 172.21.15.27:0/2650736049 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4011af0 con 0x7f5dd815ca10 2024-01-11T08:10:37.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.941+0000 7f18b159a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f18ac0ccb70 0x7f18ac0af290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37208/0 (socket says 172.21.15.27:37208) 2024-01-11T08:10:37.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.941+0000 7f18b159a700 1 -- 172.21.15.27:0/4145156416 learned_addr learned my addr 172.21.15.27:0/4145156416 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:37.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.942+0000 7f55625ff700 1 Processor -- start 2024-01-11T08:10:37.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.942+0000 7f55625ff700 1 -- start start 2024-01-11T08:10:37.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.942+0000 7f6555b9d700 1 -- start start 2024-01-11T08:10:37.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.941+0000 7f45ac3ff700 1 Processor -- start 2024-01-11T08:10:37.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.943+0000 7f1884dff700 1 -- 172.21.15.27:0/3706582037 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f188015eda0 msgr2=0x7f18800ccb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.943+0000 7f1884dff700 1 --2- 172.21.15.27:0/3706582037 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f188015eda0 0x7f18800ccb70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1878009e60 tx=0x7f1878009780 comp rx=0 tx=0).stop 2024-01-11T08:10:37.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.943+0000 7f18b1d9b700 1 --2- 172.21.15.27:0/4145156416 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18ac15d9c0 0x7f18ac161770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.943+0000 7f45ac3ff700 1 -- start start 2024-01-11T08:10:37.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.943+0000 7f6555b9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65480068c0 0x7f65480e48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.943+0000 7f6555b9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65480e0bc0 0x7f65480e4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.943+0000 7f1884dff700 1 -- 172.21.15.27:0/3706582037 shutdown_connections 2024-01-11T08:10:37.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.943+0000 7f1884dff700 1 --2- 172.21.15.27:0/3706582037 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f188015eda0 0x7f18800ccb70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.943+0000 7f1884dff700 1 --2- 172.21.15.27:0/3706582037 >> v2:172.21.15.27:3300/0 conn(0x7f18800af9a0 0x7f1880161770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.943+0000 7f1884dff700 1 -- 172.21.15.27:0/3706582037 >> 172.21.15.27:0/3706582037 conn(0x7f18800680b0 msgr2=0x7f18800cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.943+0000 7f6555b9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65480cec80 0x7f65480cf100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.943+0000 7f6555b9d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65480cf990 con 0x7f65480068c0 2024-01-11T08:10:37.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.943+0000 7f6555b9d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65480cc580 con 0x7f65480e0bc0 2024-01-11T08:10:37.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.943+0000 7f6555b9d700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65480cd100 con 0x7f65480cec80 2024-01-11T08:10:37.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.943+0000 7f62917ff700 1 Processor -- start 2024-01-11T08:10:37.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.943+0000 7f62917ff700 1 -- start start 2024-01-11T08:10:37.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.944+0000 7f1884dff700 1 -- 172.21.15.27:0/3706582037 shutdown_connections 2024-01-11T08:10:37.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.944+0000 7f18b1d9b700 1 -- 172.21.15.27:0/4145156416 >> v1:172.21.15.77:6789/0 conn(0x7f18ac0af9a0 legacy=0x7f18ac15f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.944+0000 7f18b1d9b700 1 -- 172.21.15.27:0/4145156416 >> v1:172.21.15.27:6789/0 conn(0x7f18ac0ccb70 legacy=0x7f18ac0af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.944+0000 7f18b1d9b700 1 -- 172.21.15.27:0/4145156416 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ac1620c0 con 0x7f18ac15d9c0 2024-01-11T08:10:37.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.944+0000 7f55625ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c06f570 0x7f555c16dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.944+0000 7f654ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65480e0bc0 0x7f65480e4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.944+0000 7f654ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65480e0bc0 0x7f65480e4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:58578/0 (socket says 172.21.15.27:58578) 2024-01-11T08:10:37.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.944+0000 7f654ffff700 1 -- 172.21.15.27:0/3719447329 learned_addr learned my addr 172.21.15.27:0/3719447329 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.944+0000 7f18b1d9b700 1 --2- 172.21.15.27:0/4145156416 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18ac15d9c0 0x7f18ac161770 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f18a8009940 tx=0x7f18a80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a2f0be4a4affbb5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.944+0000 7f55625ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f555c0ae070 0x7f555c16c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.944+0000 7f655539c700 1 --2- 172.21.15.27:0/3719447329 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65480cec80 0x7f65480cf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.944+0000 7f45ac3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4594003bf0 0x7f45a40a6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.944+0000 7f55625ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f555c0cc960 0x7f555c13eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f5ddf5ff700 1 -- 172.21.15.27:0/2650736049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd816d030 con 0x7f5dd815ca10 2024-01-11T08:10:37.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f6554b9b700 1 --2- 172.21.15.27:0/3719447329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65480068c0 0x7f65480e48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f5ddf5ff700 1 -- 172.21.15.27:0/2650736049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd816d5f0 con 0x7f5dd815ca10 2024-01-11T08:10:37.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f55625ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555c156800 con 0x7f555c06f570 2024-01-11T08:10:37.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f55625ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555c13d450 con 0x7f555c0cc960 2024-01-11T08:10:37.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f55625ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f555c156520 con 0x7f555c0ae070 2024-01-11T08:10:37.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f555bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c06f570 0x7f555c16dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f555bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c06f570 0x7f555c16dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53340/0 (socket says 172.21.15.27:53340) 2024-01-11T08:10:37.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f5560b9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f555c0cc960 0x7f555c13eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f555bfff700 1 -- 172.21.15.27:0/2370316653 learned_addr learned my addr 172.21.15.27:0/2370316653 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f55537fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f555c0ae070 0x7f555c16c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f64017ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63fc0cd310 0x7f63fc143f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f64017ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc153cd0 0x7f63fc16ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f64017ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63fc15d300 0x7f63fc0c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f64017ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc16ae30 con 0x7f63fc15d300 2024-01-11T08:10:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f64017ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc16d7f0 con 0x7f63fc153cd0 2024-01-11T08:10:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f64017ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc16dad0 con 0x7f63fc0cd310 2024-01-11T08:10:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.946+0000 7f63faffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63fc0cd310 0x7f63fc143f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f654ffff700 1 -- 172.21.15.27:0/3719447329 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65480cec80 msgr2=0x7f65480cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f654ffff700 1 --2- 172.21.15.27:0/3719447329 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65480cec80 0x7f65480cf100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f654ffff700 1 -- 172.21.15.27:0/3719447329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65480068c0 msgr2=0x7f65480e48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f654ffff700 1 --2- 172.21.15.27:0/3719447329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65480068c0 0x7f65480e48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f654ffff700 1 -- 172.21.15.27:0/3719447329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6544009420 con 0x7f65480e0bc0 2024-01-11T08:10:37.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.944+0000 7f45ac3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45a40a8f20 0x7f45a413fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f45ac3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45a40c3050 0x7f45a40c34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f45ac3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a40c3a00 con 0x7f45a40c3050 2024-01-11T08:10:37.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f45ac3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a4165840 con 0x7f4594003bf0 2024-01-11T08:10:37.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f45ac3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a4165b20 con 0x7f45a40a8f20 2024-01-11T08:10:37.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.946+0000 7f62917ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f628c1616d0 0x7f628c0d0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.946+0000 7f62917ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c0cea20 con 0x7f628c15dd20 2024-01-11T08:10:37.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.946+0000 7f62917ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c161b60 con 0x7f628c15d950 2024-01-11T08:10:37.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.946+0000 7f62917ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628c0d0fa0 con 0x7f628c1616d0 2024-01-11T08:10:37.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f18b0d99700 1 -- 172.21.15.27:0/4145156416 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a8009ee0 con 0x7f18ac15d9c0 2024-01-11T08:10:37.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.945+0000 7f18b0d99700 1 -- 172.21.15.27:0/4145156416 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a800b760 con 0x7f18ac15d9c0 2024-01-11T08:10:37.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.946+0000 7f18b0d99700 1 -- 172.21.15.27:0/4145156416 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a800bb00 con 0x7f18ac15d9c0 2024-01-11T08:10:37.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.946+0000 7f63faffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63fc0cd310 0x7f63fc143f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:59684/0 (socket says 172.21.15.27:59684) 2024-01-11T08:10:37.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.946+0000 7f63faffd700 1 -- 172.21.15.27:0/964937385 learned_addr learned my addr 172.21.15.27:0/964937385 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.946+0000 7f5560b9c700 1 -- 172.21.15.27:0/2370316653 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f555c0ae070 msgr2=0x7f555c16c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.946+0000 7f5560b9c700 1 --2- 172.21.15.27:0/2370316653 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f555c0ae070 0x7f555c16c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.946+0000 7f5560b9c700 1 -- 172.21.15.27:0/2370316653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c06f570 msgr2=0x7f555c16dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.946+0000 7f5560b9c700 1 --2- 172.21.15.27:0/2370316653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c06f570 0x7f555c16dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.946+0000 7f5560b9c700 1 -- 172.21.15.27:0/2370316653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55440094f0 con 0x7f555c0cc960 2024-01-11T08:10:37.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f654ffff700 1 --2- 172.21.15.27:0/3719447329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65480e0bc0 0x7f65480e4e20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f654800dae0 tx=0x7f6544017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.944+0000 7f1884dff700 1 -- 172.21.15.27:0/3706582037 wait complete. 2024-01-11T08:10:37.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.946+0000 7f628affd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f628c15d950 0x7f628c0ccb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:60784/0 (socket says 172.21.15.27:60784) 2024-01-11T08:10:37.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f5ddf5ff700 1 -- 172.21.15.27:0/2650736049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd804c3c0 con 0x7f5dd815ca10 2024-01-11T08:10:37.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f628affd700 1 -- 172.21.15.27:0/1336135081 learned_addr learned my addr 172.21.15.27:0/1336135081 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:37.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f628a7fc700 1 --2- 172.21.15.27:0/1336135081 >> v2:172.21.15.77:3300/0 conn(0x7f628c1616d0 0x7f628c0d0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f18b3fff700 1 -- 172.21.15.27:0/4145156416 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18ac15d9c0 msgr2=0x7f18ac161770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f654cff9700 1 -- 172.21.15.27:0/3719447329 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654400b9d0 con 0x7f65480e0bc0 2024-01-11T08:10:37.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f45aa19b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4594003bf0 0x7f45a40a6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f45a999a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45a40a8f20 0x7f45a413fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f45aa99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45a40c3050 0x7f45a40c34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f45aa99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45a40c3050 0x7f45a40c34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53358/0 (socket says 172.21.15.27:53358) 2024-01-11T08:10:37.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f45aa99c700 1 -- 172.21.15.27:0/414922584 learned_addr learned my addr 172.21.15.27:0/414922584 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:37.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f6289ffb700 1 -- 172.21.15.27:0/1336135081 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578394990 0 0) 0x7f628c161b60 con 0x7f628c15d950 2024-01-11T08:10:37.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f6289ffb700 1 -- 172.21.15.27:0/1336135081 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6274003570 con 0x7f628c15d950 2024-01-11T08:10:37.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.948+0000 7f18b3fff700 1 --2- 172.21.15.27:0/4145156416 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18ac15d9c0 0x7f18ac161770 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f18a8009940 tx=0x7f18a80091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f63fb7fe700 1 --2- 172.21.15.27:0/964937385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63fc15d300 0x7f63fc0c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.946+0000 7f63f27fc700 1 --2- 172.21.15.27:0/964937385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc153cd0 0x7f63fc16ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:37.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f63faffd700 1 -- 172.21.15.27:0/964937385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc153cd0 msgr2=0x7f63fc16ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f63faffd700 1 --2- 172.21.15.27:0/964937385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc153cd0 0x7f63fc16ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f63faffd700 1 -- 172.21.15.27:0/964937385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63fc15d300 msgr2=0x7f63fc0c5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f63faffd700 1 --2- 172.21.15.27:0/964937385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63fc15d300 0x7f63fc0c5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f63faffd700 1 -- 172.21.15.27:0/964937385 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f4009a20 con 0x7f63fc0cd310 2024-01-11T08:10:37.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.948+0000 7f654cff9700 1 -- 172.21.15.27:0/3719447329 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654400bb30 con 0x7f65480e0bc0 2024-01-11T08:10:37.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.948+0000 7f654cff9700 1 -- 172.21.15.27:0/3719447329 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6544025670 con 0x7f65480e0bc0 2024-01-11T08:10:37.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.948+0000 7f1884dff700 1 Processor -- start 2024-01-11T08:10:37.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.947+0000 7f5560b9c700 1 --2- 172.21.15.27:0/2370316653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f555c0cc960 0x7f555c13eb20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5554007ed0 tx=0x7f5554009850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.948+0000 7f628a7fc700 1 -- 172.21.15.27:0/1336135081 >> v1:172.21.15.77:6789/0 conn(0x7f628c15d950 legacy=0x7f628c0ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.948+0000 7f628a7fc700 1 -- 172.21.15.27:0/1336135081 >> v1:172.21.15.27:6789/0 conn(0x7f628c15dd20 legacy=0x7f628c0cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:37.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.948+0000 7f5558ff9700 1 -- 172.21.15.27:0/2370316653 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5554018070 con 0x7f555c0cc960 2024-01-11T08:10:37.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.948+0000 7f63faffd700 1 --2- 172.21.15.27:0/964937385 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63fc0cd310 0x7f63fc143f40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f63e400d890 tx=0x7f63e4008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.948+0000 7f6555b9d700 1 -- 172.21.15.27:0/3719447329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65480cd590 con 0x7f65480e0bc0 2024-01-11T08:10:37.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.948+0000 7f6555b9d700 1 -- 172.21.15.27:0/3719447329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65480ccab0 con 0x7f65480e0bc0 2024-01-11T08:10:37.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.949+0000 7f45a999a700 1 -- 172.21.15.27:0/414922584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4594003bf0 msgr2=0x7f45a40a6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.949+0000 7f45a999a700 1 --2- 172.21.15.27:0/414922584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4594003bf0 0x7f45a40a6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.949+0000 7f45a999a700 1 -- 172.21.15.27:0/414922584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45a40c3050 msgr2=0x7f45a40c34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.949+0000 7f45a999a700 1 --2- 172.21.15.27:0/414922584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45a40c3050 0x7f45a40c34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.949+0000 7f45a999a700 1 -- 172.21.15.27:0/414922584 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a40a73b0 con 0x7f45a40a8f20 2024-01-11T08:10:37.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.948+0000 7f1884dff700 1 -- start start 2024-01-11T08:10:37.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.948+0000 7f628a7fc700 1 -- 172.21.15.27:0/1336135081 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c161e40 con 0x7f628c1616d0 2024-01-11T08:10:37.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.948+0000 7f628a7fc700 1 --2- 172.21.15.27:0/1336135081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f628c1616d0 0x7f628c0d0a60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f62800098e0 tx=0x7f62800091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e166d9a2c932545 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.948+0000 7f18b3fff700 1 -- 172.21.15.27:0/4145156416 shutdown_connections 2024-01-11T08:10:37.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.948+0000 7f18b3fff700 1 --2- 172.21.15.27:0/4145156416 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18ac15d9c0 0x7f18ac161770 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.948+0000 7f18b3fff700 1 -- 172.21.15.27:0/4145156416 >> 172.21.15.27:0/4145156416 conn(0x7f18ac0680b0 msgr2=0x7f18ac0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:37.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.949+0000 7f6289ffb700 1 -- 172.21.15.27:0/1336135081 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6280011630 con 0x7f628c1616d0 2024-01-11T08:10:37.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.949+0000 7f6289ffb700 1 -- 172.21.15.27:0/1336135081 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6280011790 con 0x7f628c1616d0 2024-01-11T08:10:37.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.949+0000 7f6289ffb700 1 -- 172.21.15.27:0/1336135081 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6280011b30 con 0x7f628c1616d0 2024-01-11T08:10:37.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.948+0000 7f55625ff700 1 -- 172.21.15.27:0/2370316653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f555c13ce00 con 0x7f555c0cc960 2024-01-11T08:10:37.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.948+0000 7f55625ff700 1 -- 172.21.15.27:0/2370316653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f555c1561f0 con 0x7f555c0cc960 2024-01-11T08:10:37.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.949+0000 7f5558ff9700 1 -- 172.21.15.27:0/2370316653 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555400eb90 con 0x7f555c0cc960 2024-01-11T08:10:37.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.949+0000 7f5558ff9700 1 -- 172.21.15.27:0/2370316653 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5554014420 con 0x7f555c0cc960 2024-01-11T08:10:37.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.949+0000 7f63f37fe700 1 -- 172.21.15.27:0/964937385 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e400b600 con 0x7f63fc0cd310 2024-01-11T08:10:37.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.949+0000 7f63f37fe700 1 -- 172.21.15.27:0/964937385 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e400b760 con 0x7f63fc0cd310 2024-01-11T08:10:37.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.949+0000 7f63f37fe700 1 -- 172.21.15.27:0/964937385 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e400bae0 con 0x7f63fc0cd310 2024-01-11T08:10:37.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.949+0000 7f45a999a700 1 --2- 172.21.15.27:0/414922584 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45a40a8f20 0x7f45a413fbb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f459000d830 tx=0x7f4590009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:37.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.949+0000 7f64017ff700 1 -- 172.21.15.27:0/964937385 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63ec0094f0 con 0x7f63fc0cd310 2024-01-11T08:10:37.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.950+0000 7f64017ff700 1 -- 172.21.15.27:0/964937385 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63fc144480 con 0x7f63fc0cd310 2024-01-11T08:10:37.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.950+0000 7f1884dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18800af9a0 0x7f188016b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.950+0000 7f1884dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f188015d9c0 0x7f18800caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.950+0000 7f1884dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f188015eda0 0x7f188015c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:37.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.950+0000 7f1884dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1880177040 con 0x7f18800af9a0 2024-01-11T08:10:37.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.951+0000 7f1884dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1880176730 con 0x7f188015d9c0 2024-01-11T08:10:37.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.951+0000 7f64017ff700 1 -- 172.21.15.27:0/964937385 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63fc16a920 con 0x7f63fc0cd310 2024-01-11T08:10:37.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.951+0000 7f62917ff700 1 -- 172.21.15.27:0/1336135081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f628c1616d0 msgr2=0x7f628c0d0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:37.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.951+0000 7f62917ff700 1 --2- 172.21.15.27:0/1336135081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f628c1616d0 0x7f628c0d0a60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f62800098e0 tx=0x7f62800091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:37.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.951+0000 7f1884dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f188016bba0 con 0x7f188015eda0 2024-01-11T08:10:37.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.951+0000 7f187ed9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f188015eda0 0x7f188015c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.951+0000 7f6555b9d700 1 -- 172.21.15.27:0/3719447329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65480cd270 con 0x7f65480e0bc0 2024-01-11T08:10:38.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.951+0000 7f187ed9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f188015eda0 0x7f188015c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:59698/0 (socket says 172.21.15.27:59698) 2024-01-11T08:10:38.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.951+0000 7f187ed9d700 1 -- 172.21.15.27:0/2716276892 learned_addr learned my addr 172.21.15.27:0/2716276892 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:38.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.952+0000 7f45a27fc700 1 -- 172.21.15.27:0/414922584 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4590006820 con 0x7f45a40a8f20 2024-01-11T08:10:38.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.952+0000 7f18b3fff700 1 -- 172.21.15.27:0/4145156416 shutdown_connections 2024-01-11T08:10:38.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.952+0000 7f45a27fc700 1 -- 172.21.15.27:0/414922584 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45900069a0 con 0x7f45a40a8f20 2024-01-11T08:10:38.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.952+0000 7f45a27fc700 1 -- 172.21.15.27:0/414922584 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4590006d60 con 0x7f45a40a8f20 2024-01-11T08:10:38.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.952+0000 7f187e59c700 1 --2- 172.21.15.27:0/2716276892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18800af9a0 0x7f188016b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.952+0000 7f187ed9d700 1 -- 172.21.15.27:0/2716276892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f188015d9c0 msgr2=0x7f18800caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.952+0000 7f187ed9d700 1 --2- 172.21.15.27:0/2716276892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f188015d9c0 0x7f18800caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.952+0000 7f187ed9d700 1 -- 172.21.15.27:0/2716276892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18800af9a0 msgr2=0x7f188016b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.952+0000 7f187ed9d700 1 --2- 172.21.15.27:0/2716276892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18800af9a0 0x7f188016b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.952+0000 7f187ed9d700 1 -- 172.21.15.27:0/2716276892 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1878009a20 con 0x7f188015eda0 2024-01-11T08:10:38.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.952+0000 7f45ac3ff700 1 -- 172.21.15.27:0/414922584 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a40a7630 con 0x7f45a40a8f20 2024-01-11T08:10:38.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.952+0000 7f45ac3ff700 1 -- 172.21.15.27:0/414922584 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a40a7bf0 con 0x7f45a40a8f20 2024-01-11T08:10:38.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.952+0000 7f187e59c700 1 --2- 172.21.15.27:0/2716276892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18800af9a0 0x7f188016b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:38.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.953+0000 7f62917ff700 1 -- 172.21.15.27:0/1336135081 shutdown_connections 2024-01-11T08:10:38.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.953+0000 7f62917ff700 1 --2- 172.21.15.27:0/1336135081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f628c1616d0 0x7f628c0d0a60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.953+0000 7f62917ff700 1 -- 172.21.15.27:0/1336135081 >> 172.21.15.27:0/1336135081 conn(0x7f628c06ed40 msgr2=0x7f628c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:38.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.952+0000 7f18b3fff700 1 -- 172.21.15.27:0/4145156416 wait complete. 2024-01-11T08:10:38.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.953+0000 7f654cff9700 1 -- 172.21.15.27:0/3719447329 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6544023070 con 0x7f65480e0bc0 2024-01-11T08:10:38.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.953+0000 7f62917ff700 1 -- 172.21.15.27:0/1336135081 shutdown_connections 2024-01-11T08:10:38.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.954+0000 7f62917ff700 1 -- 172.21.15.27:0/1336135081 wait complete. 2024-01-11T08:10:38.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.954+0000 7f45ac3ff700 1 -- 172.21.15.27:0/414922584 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a40a7e10 con 0x7f45a40a8f20 2024-01-11T08:10:38.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.953+0000 7f187ed9d700 1 --2- 172.21.15.27:0/2716276892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f188015eda0 0x7f188015c4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1878003000 tx=0x7f1878018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:38.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.954+0000 7f5dc97fa700 1 -- 172.21.15.27:0/2650736049 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5dd4012420 con 0x7f5dd815ca10 2024-01-11T08:10:38.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.954+0000 7f1876ffd700 1 -- 172.21.15.27:0/2716276892 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187801e430 con 0x7f188015eda0 2024-01-11T08:10:38.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.954+0000 7f18b3fff700 1 Processor -- start 2024-01-11T08:10:38.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.956+0000 7f62917ff700 1 Processor -- start 2024-01-11T08:10:38.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.954+0000 7f1876ffd700 1 -- 172.21.15.27:0/2716276892 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f187801e5b0 con 0x7f188015eda0 2024-01-11T08:10:38.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.954+0000 7f1876ffd700 1 -- 172.21.15.27:0/2716276892 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187801e970 con 0x7f188015eda0 2024-01-11T08:10:38.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.955+0000 7f1884dff700 1 -- 172.21.15.27:0/2716276892 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18801769f0 con 0x7f188015eda0 2024-01-11T08:10:38.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.955+0000 7f1884dff700 1 -- 172.21.15.27:0/2716276892 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f188016b7e0 con 0x7f188015eda0 2024-01-11T08:10:38.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.954+0000 7f63f37fe700 1 -- 172.21.15.27:0/964937385 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f63e401e460 con 0x7f63fc0cd310 2024-01-11T08:10:38.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.956+0000 7f63f37fe700 1 --2- 172.21.15.27:0/964937385 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f63dc067080 0x7f63dc069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:38.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.956+0000 7f63f37fe700 1 -- 172.21.15.27:0/964937385 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f63e4084bf0 con 0x7f63fc0cd310 2024-01-11T08:10:38.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.955+0000 7f18b3fff700 1 -- start start 2024-01-11T08:10:38.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.956+0000 7f18b3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18ac0af9a0 0x7f18ac157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:38.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.956+0000 7f18b3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18ac0ccb70 0x7f18ac1584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:38.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.956+0000 7f18b3fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18ac15d9c0 0x7f18ac15b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:38.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.956+0000 7f654cff9700 1 --2- 172.21.15.27:0/3719447329 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6534067000 0x7f65340694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:38.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.956+0000 7f654cff9700 1 -- 172.21.15.27:0/3719447329 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f65440858e0 con 0x7f65480e0bc0 2024-01-11T08:10:38.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.956+0000 7f5dc97fa700 1 --2- 172.21.15.27:0/2650736049 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5dbc067080 0x7f5dbc069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:38.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.956+0000 7f5dc97fa700 1 -- 172.21.15.27:0/2650736049 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f5dd40841f0 con 0x7f5dd815ca10 2024-01-11T08:10:38.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.957+0000 7f5ddd39b700 1 --2- 172.21.15.27:0/2650736049 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5dbc067080 0x7f5dbc069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.957+0000 7f62917ff700 1 -- start start 2024-01-11T08:10:38.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.957+0000 7f63f27fc700 1 --2- 172.21.15.27:0/964937385 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f63dc067080 0x7f63dc069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.956+0000 7f45a27fc700 1 -- 172.21.15.27:0/414922584 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4590007240 con 0x7f45a40a8f20 2024-01-11T08:10:38.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.958+0000 7f6554b9b700 1 --2- 172.21.15.27:0/3719447329 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6534067000 0x7f65340694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.958+0000 7f1884dff700 1 -- 172.21.15.27:0/2716276892 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18640052b0 con 0x7f188015eda0 2024-01-11T08:10:38.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.957+0000 7f18b3fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac1598c0 con 0x7f18ac0af9a0 2024-01-11T08:10:38.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.957+0000 7f18b3fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac154b30 con 0x7f18ac0ccb70 2024-01-11T08:10:38.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.957+0000 7f18b3fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac1589e0 con 0x7f18ac15d9c0 2024-01-11T08:10:38.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.958+0000 7f63f27fc700 1 --2- 172.21.15.27:0/964937385 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f63dc067080 0x7f63dc069530 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f63fc0d16b0 tx=0x7f63ec017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:38.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.958+0000 7f6554b9b700 1 --2- 172.21.15.27:0/3719447329 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6534067000 0x7f65340694b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f65500614f0 tx=0x7f655006a040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:38.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f5dc97fa700 1 -- 172.21.15.27:0/2650736049 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5dd4051570 con 0x7f5dd815ca10 2024-01-11T08:10:38.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.958+0000 7f18b159a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18ac0ccb70 0x7f18ac1584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.958+0000 7f18b159a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18ac0ccb70 0x7f18ac1584a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:58636/0 (socket says 172.21.15.27:58636) 2024-01-11T08:10:38.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.958+0000 7f18b159a700 1 -- 172.21.15.27:0/511246927 learned_addr learned my addr 172.21.15.27:0/511246927 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:38.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f18b259c700 1 --2- 172.21.15.27:0/511246927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18ac15d9c0 0x7f18ac15b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f18b1d9b700 1 --2- 172.21.15.27:0/511246927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18ac0af9a0 0x7f18ac157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f62917ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f628c15d950 0x7f628c153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:38.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f45a27fc700 1 --2- 172.21.15.27:0/414922584 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4588067080 0x7f4588069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:38.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f18b159a700 1 -- 172.21.15.27:0/511246927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18ac15d9c0 msgr2=0x7f18ac15b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f18b159a700 1 --2- 172.21.15.27:0/511246927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18ac15d9c0 0x7f18ac15b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f45a27fc700 1 -- 172.21.15.27:0/414922584 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f4590084440 con 0x7f45a40a8f20 2024-01-11T08:10:38.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f62917ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f628c15dd20 0x7f628c13f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:38.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f62917ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f628c1616d0 0x7f628c152070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:38.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f62917ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c154210 con 0x7f628c15dd20 2024-01-11T08:10:38.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f62917ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c0b2cf0 con 0x7f628c15d950 2024-01-11T08:10:38.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f62917ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f628c153840 con 0x7f628c1616d0 2024-01-11T08:10:38.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f45aa19b700 1 --2- 172.21.15.27:0/414922584 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4588067080 0x7f4588069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.960+0000 7f55625ff700 1 -- 172.21.15.27:0/2370316653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55400052b0 con 0x7f555c0cc960 2024-01-11T08:10:38.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.960+0000 7f5558ff9700 1 -- 172.21.15.27:0/2370316653 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5554003020 con 0x7f555c0cc960 2024-01-11T08:10:38.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f628b7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f628c1616d0 0x7f628c152070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f628b7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f628c1616d0 0x7f628c152070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:59718/0 (socket says 172.21.15.27:59718) 2024-01-11T08:10:38.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f628b7fe700 1 -- 172.21.15.27:0/1282187772 learned_addr learned my addr 172.21.15.27:0/1282187772 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:38.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.960+0000 7f628affd700 1 --2- 172.21.15.27:0/1282187772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f628c15d950 0x7f628c153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.960+0000 7f628a7fc700 1 --2- 172.21.15.27:0/1282187772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f628c15dd20 0x7f628c13f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.960+0000 7f628b7fe700 1 -- 172.21.15.27:0/1282187772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f628c15d950 msgr2=0x7f628c153cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.960+0000 7f628b7fe700 1 --2- 172.21.15.27:0/1282187772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f628c15d950 0x7f628c153cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.960+0000 7f628b7fe700 1 -- 172.21.15.27:0/1282187772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f628c15dd20 msgr2=0x7f628c13f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.960+0000 7f628b7fe700 1 --2- 172.21.15.27:0/1282187772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f628c15dd20 0x7f628c13f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.960+0000 7f628b7fe700 1 -- 172.21.15.27:0/1282187772 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6280009420 con 0x7f628c1616d0 2024-01-11T08:10:38.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f18b159a700 1 -- 172.21.15.27:0/511246927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18ac0af9a0 msgr2=0x7f18ac157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f18b159a700 1 --2- 172.21.15.27:0/511246927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18ac0af9a0 0x7f18ac157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.959+0000 7f18b159a700 1 -- 172.21.15.27:0/511246927 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a8009420 con 0x7f18ac0ccb70 2024-01-11T08:10:38.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.960+0000 7f18b259c700 1 --2- 172.21.15.27:0/511246927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18ac15d9c0 0x7f18ac15b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:38.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.960+0000 7f628b7fe700 1 --2- 172.21.15.27:0/1282187772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f628c1616d0 0x7f628c152070 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f627c00b5e0 tx=0x7f627c009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:38.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.961+0000 7f45aa19b700 1 --2- 172.21.15.27:0/414922584 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4588067080 0x7f4588069530 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4598000f80 tx=0x7f4598016000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:38.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.961+0000 7f5ddd39b700 1 --2- 172.21.15.27:0/2650736049 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5dbc067080 0x7f5dbc069530 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0001f40 tx=0x7f5dd0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:38.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.960+0000 7f18b1d9b700 1 --2- 172.21.15.27:0/511246927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18ac0af9a0 0x7f18ac157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:38.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.961+0000 7f18b159a700 1 --2- 172.21.15.27:0/511246927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18ac0ccb70 0x7f18ac1584a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f189c00c7e0 tx=0x7f189c0011e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:38.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.961+0000 7f654cff9700 1 -- 172.21.15.27:0/3719447329 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6544052c80 con 0x7f65480e0bc0 2024-01-11T08:10:38.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.961+0000 7f63f37fe700 1 -- 172.21.15.27:0/964937385 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f63fc16a920 con 0x7f63fc0cd310 2024-01-11T08:10:38.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.962+0000 7f5558ff9700 1 --2- 172.21.15.27:0/2370316653 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f553c066fb0 0x7f553c069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.962+0000 7f5558ff9700 1 -- 172.21.15.27:0/2370316653 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f5554084640 con 0x7f555c0cc960 2024-01-11T08:10:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.962+0000 7f555bfff700 1 --2- 172.21.15.27:0/2370316653 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f553c066fb0 0x7f553c069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.963+0000 7f555bfff700 1 --2- 172.21.15.27:0/2370316653 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f553c066fb0 0x7f553c069460 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5544009290 tx=0x7f5544017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.965+0000 7f18a1ffb700 1 -- 172.21.15.27:0/511246927 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189c005b30 con 0x7f18ac0ccb70 2024-01-11T08:10:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.965+0000 7f18a1ffb700 1 -- 172.21.15.27:0/511246927 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f189c005c90 con 0x7f18ac0ccb70 2024-01-11T08:10:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.965+0000 7f18a1ffb700 1 -- 172.21.15.27:0/511246927 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189c007300 con 0x7f18ac0ccb70 2024-01-11T08:10:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.965+0000 7f5558ff9700 1 -- 172.21.15.27:0/2370316653 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5554051810 con 0x7f555c0cc960 2024-01-11T08:10:38.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.965+0000 7f18b3fff700 1 -- 172.21.15.27:0/511246927 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18ac158cb0 con 0x7f18ac0ccb70 2024-01-11T08:10:38.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.966+0000 7f18b3fff700 1 -- 172.21.15.27:0/511246927 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18ac159270 con 0x7f18ac0ccb70 2024-01-11T08:10:38.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.966+0000 7f45a27fc700 1 -- 172.21.15.27:0/414922584 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4590051690 con 0x7f45a40a8f20 2024-01-11T08:10:38.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.968+0000 7f18b3fff700 1 -- 172.21.15.27:0/511246927 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18ac04c3c0 con 0x7f18ac0ccb70 2024-01-11T08:10:38.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.968+0000 7f6272ffd700 1 -- 172.21.15.27:0/1282187772 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c00f510 con 0x7f628c1616d0 2024-01-11T08:10:38.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.968+0000 7f6272ffd700 1 -- 172.21.15.27:0/1282187772 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627c00f690 con 0x7f628c1616d0 2024-01-11T08:10:38.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.968+0000 7f6272ffd700 1 -- 172.21.15.27:0/1282187772 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c00fa50 con 0x7f628c1616d0 2024-01-11T08:10:38.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.968+0000 7f62917ff700 1 -- 172.21.15.27:0/1282187772 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628c13fdc0 con 0x7f628c1616d0 2024-01-11T08:10:38.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.969+0000 7f1876ffd700 1 -- 172.21.15.27:0/2716276892 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1878026400 con 0x7f188015eda0 2024-01-11T08:10:38.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.969+0000 7f62917ff700 1 -- 172.21.15.27:0/1282187772 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628c152f10 con 0x7f628c1616d0 2024-01-11T08:10:38.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.970+0000 7f18a1ffb700 1 -- 172.21.15.27:0/511246927 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f189c01b400 con 0x7f18ac0ccb70 2024-01-11T08:10:38.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.970+0000 7f62917ff700 1 -- 172.21.15.27:0/1282187772 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f628c04c3c0 con 0x7f628c1616d0 2024-01-11T08:10:38.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.971+0000 7f18a1ffb700 1 --2- 172.21.15.27:0/511246927 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1890067080 0x7f1890069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:38.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.971+0000 7f18a1ffb700 1 -- 172.21.15.27:0/511246927 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f189c0841b0 con 0x7f18ac0ccb70 2024-01-11T08:10:38.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.972+0000 7f18b1d9b700 1 --2- 172.21.15.27:0/511246927 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1890067080 0x7f1890069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.972+0000 7f1876ffd700 1 --2- 172.21.15.27:0/2716276892 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1860067080 0x7f1860069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:38.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.972+0000 7f1876ffd700 1 -- 172.21.15.27:0/2716276892 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f1878086f70 con 0x7f188015eda0 2024-01-11T08:10:38.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.972+0000 7f187e59c700 1 --2- 172.21.15.27:0/2716276892 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1860067080 0x7f1860069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.972+0000 7f1876ffd700 1 -- 172.21.15.27:0/2716276892 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1878087360 con 0x7f188015eda0 2024-01-11T08:10:38.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.973+0000 7f187e59c700 1 --2- 172.21.15.27:0/2716276892 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1860067080 0x7f1860069530 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1870000c00 tx=0x7f1870009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:38.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.974+0000 7f18b1d9b700 1 --2- 172.21.15.27:0/511246927 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1890067080 0x7f1890069530 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f18a80030c0 tx=0x7f18a8017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:38.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.974+0000 7f6272ffd700 1 -- 172.21.15.27:0/1282187772 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f627c024430 con 0x7f628c1616d0 2024-01-11T08:10:38.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.975+0000 7f18a1ffb700 1 -- 172.21.15.27:0/511246927 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f189c0514b0 con 0x7f18ac0ccb70 2024-01-11T08:10:38.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.975+0000 7f6272ffd700 1 --2- 172.21.15.27:0/1282187772 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6278067000 0x7f62780694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:38.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.975+0000 7f6272ffd700 1 -- 172.21.15.27:0/1282187772 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f627c084b10 con 0x7f628c1616d0 2024-01-11T08:10:38.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.976+0000 7f628affd700 1 --2- 172.21.15.27:0/1282187772 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6278067000 0x7f62780694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.976+0000 7f6272ffd700 1 -- 172.21.15.27:0/1282187772 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f627c051de0 con 0x7f628c1616d0 2024-01-11T08:10:38.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:37.977+0000 7f628affd700 1 --2- 172.21.15.27:0/1282187772 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6278067000 0x7f62780694b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6284001ce0 tx=0x7f6284001770 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:38.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.416+0000 7f5ddf5ff700 1 -- 172.21.15.27:0/2650736049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f5dd80b4580 con 0x7f5dd815ca10 2024-01-11T08:10:38.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.417+0000 7f5dc97fa700 1 -- 172.21.15.27:0/2650736049 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5dd4054b90 con 0x7f5dd815ca10 2024-01-11T08:10:38.421 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2024-01-11T08:10:38.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.419+0000 7f64017ff700 1 -- 172.21.15.27:0/964937385 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f63fc16a920 con 0x7f63fc0cd310 2024-01-11T08:10:38.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.420+0000 7f63f37fe700 1 -- 172.21.15.27:0/964937385 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f63fc16a920 con 0x7f63fc0cd310 2024-01-11T08:10:38.422 INFO:teuthology.orchestra.run.smithi027.stdout:55834574856 2024-01-11T08:10:38.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.422+0000 7f5ddf5ff700 1 -- 172.21.15.27:0/2650736049 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5dbc067080 msgr2=0x7f5dbc069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.422+0000 7f5ddf5ff700 1 --2- 172.21.15.27:0/2650736049 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5dbc067080 0x7f5dbc069530 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0001f40 tx=0x7f5dd0009040 comp rx=0 tx=0).stop 2024-01-11T08:10:38.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.422+0000 7f5ddf5ff700 1 -- 172.21.15.27:0/2650736049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dd815ca10 msgr2=0x7f5dd80c6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.422+0000 7f5ddf5ff700 1 --2- 172.21.15.27:0/2650736049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dd815ca10 0x7f5dd80c6980 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5dd400c810 tx=0x7f5dd4007d90 comp rx=0 tx=0).stop 2024-01-11T08:10:38.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.422+0000 7f5ddf5ff700 1 -- 172.21.15.27:0/2650736049 shutdown_connections 2024-01-11T08:10:38.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.422+0000 7f5ddf5ff700 1 --2- 172.21.15.27:0/2650736049 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5dbc067080 0x7f5dbc069530 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.422+0000 7f5ddf5ff700 1 --2- 172.21.15.27:0/2650736049 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5dd815cde0 0x7f5dd80c6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.422+0000 7f5ddf5ff700 1 --2- 172.21.15.27:0/2650736049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5dd8068cd0 0x7f5dd8148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.422+0000 7f5ddf5ff700 1 --2- 172.21.15.27:0/2650736049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dd815ca10 0x7f5dd80c6980 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.422+0000 7f5ddf5ff700 1 -- 172.21.15.27:0/2650736049 >> 172.21.15.27:0/2650736049 conn(0x7f5dd806ed40 msgr2=0x7f5dd80b5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:38.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.423+0000 7f5ddf5ff700 1 -- 172.21.15.27:0/2650736049 shutdown_connections 2024-01-11T08:10:38.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.423+0000 7f5ddf5ff700 1 -- 172.21.15.27:0/2650736049 wait complete. 2024-01-11T08:10:38.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.423+0000 7f6555b9d700 1 -- 172.21.15.27:0/3719447329 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f65480053d0 con 0x7f65480e0bc0 2024-01-11T08:10:38.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.423+0000 7f654cff9700 1 -- 172.21.15.27:0/3719447329 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f65440562a0 con 0x7f65480e0bc0 2024-01-11T08:10:38.425 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2024-01-11T08:10:38.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.424+0000 7f64017ff700 1 -- 172.21.15.27:0/964937385 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f63dc067080 msgr2=0x7f63dc069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.424+0000 7f64017ff700 1 --2- 172.21.15.27:0/964937385 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f63dc067080 0x7f63dc069530 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f63fc0d16b0 tx=0x7f63ec017000 comp rx=0 tx=0).stop 2024-01-11T08:10:38.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.424+0000 7f64017ff700 1 -- 172.21.15.27:0/964937385 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63fc0cd310 msgr2=0x7f63fc143f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.424+0000 7f64017ff700 1 --2- 172.21.15.27:0/964937385 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63fc0cd310 0x7f63fc143f40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f63e400d890 tx=0x7f63e4008040 comp rx=0 tx=0).stop 2024-01-11T08:10:38.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.425+0000 7f1884dff700 1 -- 172.21.15.27:0/2716276892 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f1864005830 con 0x7f188015eda0 2024-01-11T08:10:38.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.425+0000 7f1876ffd700 1 -- 172.21.15.27:0/2716276892 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1878054240 con 0x7f188015eda0 2024-01-11T08:10:38.426 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2024-01-11T08:10:38.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.425+0000 7f64017ff700 1 -- 172.21.15.27:0/964937385 shutdown_connections 2024-01-11T08:10:38.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.425+0000 7f64017ff700 1 --2- 172.21.15.27:0/964937385 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f63dc067080 0x7f63dc069530 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.425+0000 7f64017ff700 1 --2- 172.21.15.27:0/964937385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63fc15d300 0x7f63fc0c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.425+0000 7f64017ff700 1 --2- 172.21.15.27:0/964937385 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63fc0cd310 0x7f63fc143f40 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f63e400d890 tx=0x7f63e4008040 comp rx=0 tx=0).stop 2024-01-11T08:10:38.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.425+0000 7f64017ff700 1 --2- 172.21.15.27:0/964937385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc153cd0 0x7f63fc16ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.425+0000 7f64017ff700 1 -- 172.21.15.27:0/964937385 >> 172.21.15.27:0/964937385 conn(0x7f63fc06ed40 msgr2=0x7f63fc159ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:38.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.425+0000 7f64017ff700 1 -- 172.21.15.27:0/964937385 shutdown_connections 2024-01-11T08:10:38.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.425+0000 7f64017ff700 1 -- 172.21.15.27:0/964937385 wait complete. 2024-01-11T08:10:38.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.427+0000 7f6555b9d700 1 -- 172.21.15.27:0/3719447329 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6534067000 msgr2=0x7f65340694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.427+0000 7f6555b9d700 1 --2- 172.21.15.27:0/3719447329 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6534067000 0x7f65340694b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f65500614f0 tx=0x7f655006a040 comp rx=0 tx=0).stop 2024-01-11T08:10:38.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.427+0000 7f6555b9d700 1 -- 172.21.15.27:0/3719447329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65480e0bc0 msgr2=0x7f65480e4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.427+0000 7f6555b9d700 1 --2- 172.21.15.27:0/3719447329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65480e0bc0 0x7f65480e4e20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f654800dae0 tx=0x7f6544017000 comp rx=0 tx=0).stop 2024-01-11T08:10:38.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.427+0000 7f55625ff700 1 -- 172.21.15.27:0/2370316653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f5540005560 con 0x7f555c0cc960 2024-01-11T08:10:38.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.428+0000 7f6555b9d700 1 -- 172.21.15.27:0/3719447329 shutdown_connections 2024-01-11T08:10:38.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.428+0000 7f6555b9d700 1 --2- 172.21.15.27:0/3719447329 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6534067000 0x7f65340694b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.428+0000 7f6555b9d700 1 --2- 172.21.15.27:0/3719447329 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65480cec80 0x7f65480cf100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.428+0000 7f6555b9d700 1 --2- 172.21.15.27:0/3719447329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65480068c0 0x7f65480e48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.428+0000 7f6555b9d700 1 --2- 172.21.15.27:0/3719447329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65480e0bc0 0x7f65480e4e20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.428+0000 7f6555b9d700 1 -- 172.21.15.27:0/3719447329 >> 172.21.15.27:0/3719447329 conn(0x7f6548054d40 msgr2=0x7f65480d3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:38.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.428+0000 7f5558ff9700 1 -- 172.21.15.27:0/2370316653 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5554009990 con 0x7f555c0cc960 2024-01-11T08:10:38.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.428+0000 7f45ac3ff700 1 -- 172.21.15.27:0/414922584 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f45a4165cb0 con 0x7f45a40a8f20 2024-01-11T08:10:38.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.429+0000 7f6555b9d700 1 -- 172.21.15.27:0/3719447329 shutdown_connections 2024-01-11T08:10:38.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.429+0000 7f6555b9d700 1 -- 172.21.15.27:0/3719447329 wait complete. 2024-01-11T08:10:38.431 INFO:teuthology.orchestra.run.smithi027.stdout:55834574854 2024-01-11T08:10:38.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.429+0000 7f45a27fc700 1 -- 172.21.15.27:0/414922584 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4590054cb0 con 0x7f45a40a8f20 2024-01-11T08:10:38.431 INFO:teuthology.orchestra.run.smithi027.stdout:55834574854 2024-01-11T08:10:38.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.429+0000 7f1884dff700 1 -- 172.21.15.27:0/2716276892 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1860067080 msgr2=0x7f1860069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.429+0000 7f1884dff700 1 --2- 172.21.15.27:0/2716276892 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1860067080 0x7f1860069530 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1870000c00 tx=0x7f1870009040 comp rx=0 tx=0).stop 2024-01-11T08:10:38.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.430+0000 7f1884dff700 1 -- 172.21.15.27:0/2716276892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f188015eda0 msgr2=0x7f188015c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.430+0000 7f1884dff700 1 --2- 172.21.15.27:0/2716276892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f188015eda0 0x7f188015c4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1878003000 tx=0x7f1878018000 comp rx=0 tx=0).stop 2024-01-11T08:10:38.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.432+0000 7f45ac3ff700 1 -- 172.21.15.27:0/414922584 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4588067080 msgr2=0x7f4588069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.432+0000 7f45ac3ff700 1 --2- 172.21.15.27:0/414922584 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4588067080 0x7f4588069530 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4598000f80 tx=0x7f4598016000 comp rx=0 tx=0).stop 2024-01-11T08:10:38.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.432+0000 7f45ac3ff700 1 -- 172.21.15.27:0/414922584 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45a40a8f20 msgr2=0x7f45a413fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.432+0000 7f45ac3ff700 1 --2- 172.21.15.27:0/414922584 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45a40a8f20 0x7f45a413fbb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f459000d830 tx=0x7f4590009040 comp rx=0 tx=0).stop 2024-01-11T08:10:38.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.433+0000 7f1884dff700 1 -- 172.21.15.27:0/2716276892 shutdown_connections 2024-01-11T08:10:38.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.433+0000 7f1884dff700 1 --2- 172.21.15.27:0/2716276892 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1860067080 0x7f1860069530 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.433+0000 7f1884dff700 1 --2- 172.21.15.27:0/2716276892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f188015eda0 0x7f188015c4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.433+0000 7f1884dff700 1 --2- 172.21.15.27:0/2716276892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18800af9a0 0x7f188016b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.433+0000 7f1884dff700 1 --2- 172.21.15.27:0/2716276892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f188015d9c0 0x7f18800caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.433+0000 7f1884dff700 1 -- 172.21.15.27:0/2716276892 >> 172.21.15.27:0/2716276892 conn(0x7f18800680b0 msgr2=0x7f188015e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:38.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.433+0000 7f45ac3ff700 1 -- 172.21.15.27:0/414922584 shutdown_connections 2024-01-11T08:10:38.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.433+0000 7f45ac3ff700 1 --2- 172.21.15.27:0/414922584 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4588067080 0x7f4588069530 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.433+0000 7f45ac3ff700 1 --2- 172.21.15.27:0/414922584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45a40c3050 0x7f45a40c34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.433+0000 7f45ac3ff700 1 --2- 172.21.15.27:0/414922584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4594003bf0 0x7f45a40a6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.433+0000 7f45ac3ff700 1 --2- 172.21.15.27:0/414922584 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45a40a8f20 0x7f45a413fbb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.433+0000 7f45ac3ff700 1 -- 172.21.15.27:0/414922584 >> 172.21.15.27:0/414922584 conn(0x7f45a40680a0 msgr2=0x7f45a4140480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:38.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.433+0000 7f45ac3ff700 1 -- 172.21.15.27:0/414922584 shutdown_connections 2024-01-11T08:10:38.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.433+0000 7f45ac3ff700 1 -- 172.21.15.27:0/414922584 wait complete. 2024-01-11T08:10:38.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.433+0000 7f55625ff700 1 -- 172.21.15.27:0/2370316653 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f553c066fb0 msgr2=0x7f553c069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.433+0000 7f55625ff700 1 --2- 172.21.15.27:0/2370316653 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f553c066fb0 0x7f553c069460 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5544009290 tx=0x7f5544017000 comp rx=0 tx=0).stop 2024-01-11T08:10:38.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.434+0000 7f55625ff700 1 -- 172.21.15.27:0/2370316653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f555c0cc960 msgr2=0x7f555c13eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.434+0000 7f55625ff700 1 --2- 172.21.15.27:0/2370316653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f555c0cc960 0x7f555c13eb20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5554007ed0 tx=0x7f5554009850 comp rx=0 tx=0).stop 2024-01-11T08:10:38.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.433+0000 7f1884dff700 1 -- 172.21.15.27:0/2716276892 shutdown_connections 2024-01-11T08:10:38.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.434+0000 7f55625ff700 1 -- 172.21.15.27:0/2370316653 shutdown_connections 2024-01-11T08:10:38.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.434+0000 7f55625ff700 1 --2- 172.21.15.27:0/2370316653 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f553c066fb0 0x7f553c069460 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.434+0000 7f55625ff700 1 --2- 172.21.15.27:0/2370316653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f555c0cc960 0x7f555c13eb20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.434+0000 7f55625ff700 1 --2- 172.21.15.27:0/2370316653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f555c06f570 0x7f555c16dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.434+0000 7f55625ff700 1 --2- 172.21.15.27:0/2370316653 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f555c0ae070 0x7f555c16c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.434+0000 7f55625ff700 1 -- 172.21.15.27:0/2370316653 >> 172.21.15.27:0/2370316653 conn(0x7f555c0680a0 msgr2=0x7f555c0b1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:38.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.433+0000 7f1884dff700 1 -- 172.21.15.27:0/2716276892 wait complete. 2024-01-11T08:10:38.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.434+0000 7f55625ff700 1 -- 172.21.15.27:0/2370316653 shutdown_connections 2024-01-11T08:10:38.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.434+0000 7f55625ff700 1 -- 172.21.15.27:0/2370316653 wait complete. 2024-01-11T08:10:38.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.434+0000 7f18b3fff700 1 -- 172.21.15.27:0/511246927 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f18ac159a00 con 0x7f18ac0ccb70 2024-01-11T08:10:38.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.436+0000 7f18a1ffb700 1 -- 172.21.15.27:0/511246927 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f189c0076f0 con 0x7f18ac0ccb70 2024-01-11T08:10:38.440 INFO:teuthology.orchestra.run.smithi027.stdout:55834574854 2024-01-11T08:10:38.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.442+0000 7f18b3fff700 1 -- 172.21.15.27:0/511246927 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1890067080 msgr2=0x7f1890069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.442+0000 7f18b3fff700 1 --2- 172.21.15.27:0/511246927 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1890067080 0x7f1890069530 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f18a80030c0 tx=0x7f18a8017000 comp rx=0 tx=0).stop 2024-01-11T08:10:38.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.442+0000 7f18b3fff700 1 -- 172.21.15.27:0/511246927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18ac0ccb70 msgr2=0x7f18ac1584a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.442+0000 7f18b3fff700 1 --2- 172.21.15.27:0/511246927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18ac0ccb70 0x7f18ac1584a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f189c00c7e0 tx=0x7f189c0011e0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.442+0000 7f18b3fff700 1 -- 172.21.15.27:0/511246927 shutdown_connections 2024-01-11T08:10:38.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.442+0000 7f18b3fff700 1 --2- 172.21.15.27:0/511246927 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1890067080 0x7f1890069530 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.442+0000 7f18b3fff700 1 --2- 172.21.15.27:0/511246927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18ac15d9c0 0x7f18ac15b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.442+0000 7f18b3fff700 1 --2- 172.21.15.27:0/511246927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18ac0af9a0 0x7f18ac157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.442+0000 7f18b3fff700 1 --2- 172.21.15.27:0/511246927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18ac0ccb70 0x7f18ac1584a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.442+0000 7f18b3fff700 1 -- 172.21.15.27:0/511246927 >> 172.21.15.27:0/511246927 conn(0x7f18ac0680b0 msgr2=0x7f18ac06d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:38.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.443+0000 7f62917ff700 1 -- 172.21.15.27:0/1282187772 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f628c153980 con 0x7f628c1616d0 2024-01-11T08:10:38.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.443+0000 7f18b3fff700 1 -- 172.21.15.27:0/511246927 shutdown_connections 2024-01-11T08:10:38.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.443+0000 7f18b3fff700 1 -- 172.21.15.27:0/511246927 wait complete. 2024-01-11T08:10:38.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.444+0000 7f6272ffd700 1 -- 172.21.15.27:0/1282187772 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f627c055400 con 0x7f628c1616d0 2024-01-11T08:10:38.445 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2024-01-11T08:10:38.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.449+0000 7f62917ff700 1 -- 172.21.15.27:0/1282187772 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6278067000 msgr2=0x7f62780694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.449+0000 7f62917ff700 1 --2- 172.21.15.27:0/1282187772 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6278067000 0x7f62780694b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6284001ce0 tx=0x7f6284001770 comp rx=0 tx=0).stop 2024-01-11T08:10:38.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.449+0000 7f62917ff700 1 -- 172.21.15.27:0/1282187772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f628c1616d0 msgr2=0x7f628c152070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.449+0000 7f62917ff700 1 --2- 172.21.15.27:0/1282187772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f628c1616d0 0x7f628c152070 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f627c00b5e0 tx=0x7f627c009120 comp rx=0 tx=0).stop 2024-01-11T08:10:38.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.449+0000 7f62917ff700 1 -- 172.21.15.27:0/1282187772 shutdown_connections 2024-01-11T08:10:38.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.449+0000 7f62917ff700 1 --2- 172.21.15.27:0/1282187772 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6278067000 0x7f62780694b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.449+0000 7f62917ff700 1 --2- 172.21.15.27:0/1282187772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f628c1616d0 0x7f628c152070 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.449+0000 7f62917ff700 1 --2- 172.21.15.27:0/1282187772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f628c15d950 0x7f628c153cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.449+0000 7f62917ff700 1 --2- 172.21.15.27:0/1282187772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f628c15dd20 0x7f628c13f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.449+0000 7f62917ff700 1 -- 172.21.15.27:0/1282187772 >> 172.21.15.27:0/1282187772 conn(0x7f628c06ed40 msgr2=0x7f628c15c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:38.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.451+0000 7f62917ff700 1 -- 172.21.15.27:0/1282187772 shutdown_connections 2024-01-11T08:10:38.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.451+0000 7f62917ff700 1 -- 172.21.15.27:0/1282187772 wait complete. 2024-01-11T08:10:38.459 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2024-01-11T08:10:38.459 DEBUG:teuthology.parallel:result is None 2024-01-11T08:10:38.463 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.1 2024-01-11T08:10:38.463 DEBUG:teuthology.parallel:result is None 2024-01-11T08:10:38.471 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.4 2024-01-11T08:10:38.471 DEBUG:teuthology.parallel:result is None 2024-01-11T08:10:38.471 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2024-01-11T08:10:38.471 DEBUG:teuthology.parallel:result is None 2024-01-11T08:10:38.473 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2024-01-11T08:10:38.473 DEBUG:teuthology.parallel:result is None 2024-01-11T08:10:38.475 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.6 2024-01-11T08:10:38.475 DEBUG:teuthology.parallel:result is None 2024-01-11T08:10:38.476 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2024-01-11T08:10:38.477 DEBUG:teuthology.parallel:result is None 2024-01-11T08:10:38.481 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2024-01-11T08:10:38.481 DEBUG:teuthology.parallel:result is None 2024-01-11T08:10:38.481 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-11T08:10:38.481 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:10:38.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.720+0000 7f0cd5dff700 1 Processor -- start 2024-01-11T08:10:38.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.720+0000 7f0cd5dff700 1 -- start start 2024-01-11T08:10:38.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.721+0000 7f0cd5dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0cd00b03e0 0x7f0cd00b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:38.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.721+0000 7f0cd5dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd00cd730 con 0x7f0cd00b29c0 2024-01-11T08:10:38.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.721+0000 7f0cd5dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd00b2de0 con 0x7f0cd00b07b0 2024-01-11T08:10:38.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.721+0000 7f0cd5dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd00ccb20 con 0x7f0cd00b03e0 2024-01-11T08:10:38.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.721+0000 7f0ccf7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0cd00b03e0 0x7f0cd00b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.721+0000 7f0ccffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0cd00b29c0 0x7f0cd00aefc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37230/0 (socket says 172.21.15.27:37230) 2024-01-11T08:10:38.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.721+0000 7f0ccffff700 1 -- 172.21.15.27:0/1656069969 learned_addr learned my addr 172.21.15.27:0/1656069969 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:38.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.721+0000 7f0ccf7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0cd00b03e0 0x7f0cd00b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53398/0 (socket says 172.21.15.27:53398) 2024-01-11T08:10:38.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.722+0000 7f0ccf7fe700 1 -- 172.21.15.27:0/1656069969 >> v1:172.21.15.77:6789/0 conn(0x7f0cd00b07b0 legacy=0x7f0cd0159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.722+0000 7f0ccf7fe700 1 -- 172.21.15.27:0/1656069969 >> v1:172.21.15.27:6789/0 conn(0x7f0cd00b29c0 legacy=0x7f0cd00aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.722+0000 7f0ccf7fe700 1 -- 172.21.15.27:0/1656069969 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd00cce00 con 0x7f0cd00b03e0 2024-01-11T08:10:38.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.722+0000 7f0cce7fc700 1 -- 172.21.15.27:0/1656069969 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296639524 0 0) 0x7f0cd00cd730 con 0x7f0cd00b29c0 2024-01-11T08:10:38.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.722+0000 7f0ccf7fe700 1 --2- 172.21.15.27:0/1656069969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00b03e0 0x7f0cd00b2480 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0cc00098e0 tx=0x7f0cc00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=450d5d74ccb0ee9a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:38.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.722+0000 7f0cce7fc700 1 -- 172.21.15.27:0/1656069969 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0009e80 con 0x7f0cd00b03e0 2024-01-11T08:10:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.722+0000 7f0cce7fc700 1 -- 172.21.15.27:0/1656069969 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc000b860 con 0x7f0cd00b03e0 2024-01-11T08:10:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.722+0000 7f0cce7fc700 1 -- 172.21.15.27:0/1656069969 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc000bbe0 con 0x7f0cd00b03e0 2024-01-11T08:10:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.723+0000 7f0cd5dff700 1 -- 172.21.15.27:0/1656069969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00b03e0 msgr2=0x7f0cd00b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.723+0000 7f0cd5dff700 1 --2- 172.21.15.27:0/1656069969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00b03e0 0x7f0cd00b2480 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0cc00098e0 tx=0x7f0cc00091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.723+0000 7f0cd5dff700 1 -- 172.21.15.27:0/1656069969 shutdown_connections 2024-01-11T08:10:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.723+0000 7f0cd5dff700 1 --2- 172.21.15.27:0/1656069969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00b03e0 0x7f0cd00b2480 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.723+0000 7f0cd5dff700 1 -- 172.21.15.27:0/1656069969 >> 172.21.15.27:0/1656069969 conn(0x7f0cd006ed40 msgr2=0x7f0cd006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.723+0000 7f0cd5dff700 1 -- 172.21.15.27:0/1656069969 shutdown_connections 2024-01-11T08:10:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.723+0000 7f0cd5dff700 1 -- 172.21.15.27:0/1656069969 wait complete. 2024-01-11T08:10:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.725+0000 7f0cd5dff700 1 Processor -- start 2024-01-11T08:10:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.725+0000 7f0cd5dff700 1 -- start start 2024-01-11T08:10:38.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.726+0000 7f0cd5dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00b03e0 0x7f0cd00a90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:38.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.726+0000 7f0cd5dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0cd00b07b0 0x7f0cd015ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:38.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.726+0000 7f0cd5dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cd00b29c0 0x7f0cd00a99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:38.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.726+0000 7f0cd5dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd00abe90 con 0x7f0cd00b03e0 2024-01-11T08:10:38.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.726+0000 7f0cd5dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd00ac170 con 0x7f0cd00b29c0 2024-01-11T08:10:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.726+0000 7f0cd5dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd00ac450 con 0x7f0cd00b07b0 2024-01-11T08:10:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.726+0000 7f0ccf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00b03e0 0x7f0cd00a90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.726+0000 7f0ccf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00b03e0 0x7f0cd00a90f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53406/0 (socket says 172.21.15.27:53406) 2024-01-11T08:10:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.726+0000 7f0ccf7fe700 1 -- 172.21.15.27:0/1182150204 learned_addr learned my addr 172.21.15.27:0/1182150204 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.726+0000 7f0cceffd700 1 --2- 172.21.15.27:0/1182150204 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0cd00b07b0 0x7f0cd015ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.726+0000 7f0ccffff700 1 --2- 172.21.15.27:0/1182150204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cd00b29c0 0x7f0cd00a99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.726+0000 7f0ccf7fe700 1 -- 172.21.15.27:0/1182150204 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0cd00b07b0 msgr2=0x7f0cd015ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.726+0000 7f0ccf7fe700 1 --2- 172.21.15.27:0/1182150204 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0cd00b07b0 0x7f0cd015ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.726+0000 7f0ccf7fe700 1 -- 172.21.15.27:0/1182150204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cd00b29c0 msgr2=0x7f0cd00a99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.726+0000 7f0ccf7fe700 1 --2- 172.21.15.27:0/1182150204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cd00b29c0 0x7f0cd00a99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.726+0000 7f0ccf7fe700 1 -- 172.21.15.27:0/1182150204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc0009420 con 0x7f0cd00b03e0 2024-01-11T08:10:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.726+0000 7f0cceffd700 1 --2- 172.21.15.27:0/1182150204 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0cd00b07b0 0x7f0cd015ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.727+0000 7f0ccf7fe700 1 --2- 172.21.15.27:0/1182150204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00b03e0 0x7f0cd00a90f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0069540 tx=0x7f0cc0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:38.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.727+0000 7f0cb77fe700 1 -- 172.21.15.27:0/1182150204 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc001f070 con 0x7f0cd00b03e0 2024-01-11T08:10:38.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.727+0000 7f0cb77fe700 1 -- 172.21.15.27:0/1182150204 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc0011420 con 0x7f0cd00b03e0 2024-01-11T08:10:38.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.727+0000 7f0cd5dff700 1 -- 172.21.15.27:0/1182150204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd00ade30 con 0x7f0cd00b03e0 2024-01-11T08:10:38.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.727+0000 7f0cb77fe700 1 -- 172.21.15.27:0/1182150204 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc000bbe0 con 0x7f0cd00b03e0 2024-01-11T08:10:38.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.727+0000 7f0cd5dff700 1 -- 172.21.15.27:0/1182150204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd00a6f40 con 0x7f0cd00b03e0 2024-01-11T08:10:38.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.728+0000 7f0cd5dff700 1 -- 172.21.15.27:0/1182150204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd00718d0 con 0x7f0cd00b03e0 2024-01-11T08:10:38.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.729+0000 7f0cb77fe700 1 -- 172.21.15.27:0/1182150204 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0cc0011a50 con 0x7f0cd00b03e0 2024-01-11T08:10:38.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.730+0000 7f0cb77fe700 1 --2- 172.21.15.27:0/1182150204 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0cbc066f30 0x7f0cbc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:38.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.730+0000 7f0cb77fe700 1 -- 172.21.15.27:0/1182150204 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f0cc0085b60 con 0x7f0cd00b03e0 2024-01-11T08:10:38.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.730+0000 7f0cceffd700 1 --2- 172.21.15.27:0/1182150204 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0cbc066f30 0x7f0cbc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:38.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.731+0000 7f0cceffd700 1 --2- 172.21.15.27:0/1182150204 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0cbc066f30 0x7f0cbc0693e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4007fd0 tx=0x7f0cc4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:38.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.734+0000 7f0cb77fe700 1 -- 172.21.15.27:0/1182150204 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0cc0052dd0 con 0x7f0cd00b03e0 2024-01-11T08:10:38.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.932+0000 7f0cd5dff700 1 -- 172.21.15.27:0/1182150204 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0cd00cd0e0 con 0x7f0cbc066f30 2024-01-11T08:10:38.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.941+0000 7f0cb77fe700 1 -- 172.21.15.27:0/1182150204 <== mgr.4099 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264566 (secure 0 0 0) 0x7f0cd00cd0e0 con 0x7f0cbc066f30 2024-01-11T08:10:38.945 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.945+0000 7f0cd5dff700 1 -- 172.21.15.27:0/1182150204 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0cbc066f30 msgr2=0x7f0cbc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.945+0000 7f0cd5dff700 1 --2- 172.21.15.27:0/1182150204 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0cbc066f30 0x7f0cbc0693e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4007fd0 tx=0x7f0cc4009040 comp rx=0 tx=0).stop 2024-01-11T08:10:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.945+0000 7f0cd5dff700 1 -- 172.21.15.27:0/1182150204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00b03e0 msgr2=0x7f0cd00a90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.945+0000 7f0cd5dff700 1 --2- 172.21.15.27:0/1182150204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00b03e0 0x7f0cd00a90f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0069540 tx=0x7f0cc0017000 comp rx=0 tx=0).stop 2024-01-11T08:10:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.945+0000 7f0cd5dff700 1 -- 172.21.15.27:0/1182150204 shutdown_connections 2024-01-11T08:10:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.945+0000 7f0cd5dff700 1 --2- 172.21.15.27:0/1182150204 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0cbc066f30 0x7f0cbc0693e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.945+0000 7f0cd5dff700 1 --2- 172.21.15.27:0/1182150204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cd00b29c0 0x7f0cd00a99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.945+0000 7f0cd5dff700 1 --2- 172.21.15.27:0/1182150204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00b03e0 0x7f0cd00a90f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.945+0000 7f0cd5dff700 1 --2- 172.21.15.27:0/1182150204 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0cd00b07b0 0x7f0cd015ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.945+0000 7f0cd5dff700 1 -- 172.21.15.27:0/1182150204 >> 172.21.15.27:0/1182150204 conn(0x7f0cd006ed40 msgr2=0x7f0cd00cb880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.945+0000 7f0cd5dff700 1 -- 172.21.15.27:0/1182150204 shutdown_connections 2024-01-11T08:10:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:38.945+0000 7f0cd5dff700 1 -- 172.21.15.27:0/1182150204 wait complete. 2024-01-11T08:10:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:10:38.964 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-01-11T08:10:37.510267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381224,"kb_used_data":2920,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752593496,"statfs":{"total":773094113280,"available":770655739904,"internally_reserved":0,"allocated":2990080,"data_stored":1787852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.863498"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.498429+0000","last_change":"2024-01-11T08:10:30.495770+0000","last_active":"2024-01-11T08:10:30.498429+0000","last_peered":"2024-01-11T08:10:30.498429+0000","last_clean":"2024-01-11T08:10:30.498429+0000","last_became_active":"2024-01-11T08:10:30.495355+0000","last_became_peered":"2024-01-11T08:10:30.495355+0000","last_unstale":"2024-01-11T08:10:30.498429+0000","last_undegraded":"2024-01-11T08:10:30.498429+0000","last_fullsized":"2024-01-11T08:10:30.498429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:02.887025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.493541+0000","last_change":"2024-01-11T08:10:30.492484+0000","last_active":"2024-01-11T08:10:30.493541+0000","last_peered":"2024-01-11T08:10:30.493541+0000","last_clean":"2024-01-11T08:10:30.493541+0000","last_became_active":"2024-01-11T08:10:30.491796+0000","last_became_peered":"2024-01-11T08:10:30.491796+0000","last_unstale":"2024-01-11T08:10:30.493541+0000","last_undegraded":"2024-01-11T08:10:30.493541+0000","last_fullsized":"2024-01-11T08:10:30.493541+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:21.127779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.493213+0000","last_change":"2024-01-11T08:10:30.489955+0000","last_active":"2024-01-11T08:10:30.493213+0000","last_peered":"2024-01-11T08:10:30.493213+0000","last_clean":"2024-01-11T08:10:30.493213+0000","last_became_active":"2024-01-11T08:10:30.489649+0000","last_became_peered":"2024-01-11T08:10:30.489649+0000","last_unstale":"2024-01-11T08:10:30.493213+0000","last_undegraded":"2024-01-11T08:10:30.493213+0000","last_fullsized":"2024-01-11T08:10:30.493213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:08:10.140431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.495948+0000","last_change":"2024-01-11T08:10:30.489768+0000","last_active":"2024-01-11T08:10:30.495948+0000","last_peered":"2024-01-11T08:10:30.495948+0000","last_clean":"2024-01-11T08:10:30.495948+0000","last_became_active":"2024-01-11T08:10:30.489392+0000","last_became_peered":"2024-01-11T08:10:30.489392+0000","last_unstale":"2024-01-11T08:10:30.495948+0000","last_undegraded":"2024-01-11T08:10:30.495948+0000","last_fullsized":"2024-01-11T08:10:30.495948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:08:10.140431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.494482+0000","last_change":"2024-01-11T08:10:30.490346+0000","last_active":"2024-01-11T08:10:30.494482+0000","last_peered":"2024-01-11T08:10:30.494482+0000","last_clean":"2024-01-11T08:10:30.494482+0000","last_became_active":"2024-01-11T08:10:30.490042+0000","last_became_peered":"2024-01-11T08:10:30.490042+0000","last_unstale":"2024-01-11T08:10:30.494482+0000","last_undegraded":"2024-01-11T08:10:30.494482+0000","last_fullsized":"2024-01-11T08:10:30.494482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:57:19.989587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.492082+0000","last_change":"2024-01-11T08:10:30.487385+0000","last_active":"2024-01-11T08:10:30.492082+0000","last_peered":"2024-01-11T08:10:30.492082+0000","last_clean":"2024-01-11T08:10:30.492082+0000","last_became_active":"2024-01-11T08:10:30.487027+0000","last_became_peered":"2024-01-11T08:10:30.487027+0000","last_unstale":"2024-01-11T08:10:30.492082+0000","last_undegraded":"2024-01-11T08:10:30.492082+0000","last_fullsized":"2024-01-11T08:10:30.492082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:12.649092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.498136+0000","last_change":"2024-01-11T08:10:30.494485+0000","last_active":"2024-01-11T08:10:30.498136+0000","last_peered":"2024-01-11T08:10:30.498136+0000","last_clean":"2024-01-11T08:10:30.498136+0000","last_became_active":"2024-01-11T08:10:30.494227+0000","last_became_peered":"2024-01-11T08:10:30.494227+0000","last_unstale":"2024-01-11T08:10:30.498136+0000","last_undegraded":"2024-01-11T08:10:30.498136+0000","last_fullsized":"2024-01-11T08:10:30.498136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:31:36.455594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.484735+0000","last_change":"2024-01-11T08:10:30.477637+0000","last_active":"2024-01-11T08:10:30.484735+0000","last_peered":"2024-01-11T08:10:30.484735+0000","last_clean":"2024-01-11T08:10:30.484735+0000","last_became_active":"2024-01-11T08:10:30.477267+0000","last_became_peered":"2024-01-11T08:10:30.477267+0000","last_unstale":"2024-01-11T08:10:30.484735+0000","last_undegraded":"2024-01-11T08:10:30.484735+0000","last_fullsized":"2024-01-11T08:10:30.484735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:47.753678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.496708+0000","last_change":"2024-01-11T08:10:30.494031+0000","last_active":"2024-01-11T08:10:30.496708+0000","last_peered":"2024-01-11T08:10:30.496708+0000","last_clean":"2024-01-11T08:10:30.496708+0000","last_became_active":"2024-01-11T08:10:30.493769+0000","last_became_peered":"2024-01-11T08:10:30.493769+0000","last_unstale":"2024-01-11T08:10:30.496708+0000","last_undegraded":"2024-01-11T08:10:30.496708+0000","last_fullsized":"2024-01-11T08:10:30.496708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:14:36.595264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.502209+0000","last_change":"2024-01-11T08:10:30.498360+0000","last_active":"2024-01-11T08:10:30.502209+0000","last_peered":"2024-01-11T08:10:30.502209+0000","last_clean":"2024-01-11T08:10:30.502209+0000","last_became_active":"2024-01-11T08:10:30.498064+0000","last_became_peered":"2024-01-11T08:10:30.498064+0000","last_unstale":"2024-01-11T08:10:30.502209+0000","last_undegraded":"2024-01-11T08:10:30.502209+0000","last_fullsized":"2024-01-11T08:10:30.502209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:08:10.140431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.501156+0000","last_change":"2024-01-11T08:10:30.493099+0000","last_active":"2024-01-11T08:10:30.501156+0000","last_peered":"2024-01-11T08:10:30.501156+0000","last_clean":"2024-01-11T08:10:30.501156+0000","last_became_active":"2024-01-11T08:10:30.492806+0000","last_became_peered":"2024-01-11T08:10:30.492806+0000","last_unstale":"2024-01-11T08:10:30.501156+0000","last_undegraded":"2024-01-11T08:10:30.501156+0000","last_fullsized":"2024-01-11T08:10:30.501156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:21.127779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.503398+0000","last_change":"2024-01-11T08:10:30.499317+0000","last_active":"2024-01-11T08:10:30.503398+0000","last_peered":"2024-01-11T08:10:30.503398+0000","last_clean":"2024-01-11T08:10:30.503398+0000","last_became_active":"2024-01-11T08:10:30.497953+0000","last_became_peered":"2024-01-11T08:10:30.497953+0000","last_unstale":"2024-01-11T08:10:30.503398+0000","last_undegraded":"2024-01-11T08:10:30.503398+0000","last_fullsized":"2024-01-11T08:10:30.503398+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:27:24.758750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.490479+0000","last_change":"2024-01-11T08:10:30.488701+0000","last_active":"2024-01-11T08:10:30.490479+0000","last_peered":"2024-01-11T08:10:30.490479+0000","last_clean":"2024-01-11T08:10:30.490479+0000","last_became_active":"2024-01-11T08:10:30.488244+0000","last_became_peered":"2024-01-11T08:10:30.488244+0000","last_unstale":"2024-01-11T08:10:30.490479+0000","last_undegraded":"2024-01-11T08:10:30.490479+0000","last_fullsized":"2024-01-11T08:10:30.490479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:21.127779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.491292+0000","last_change":"2024-01-11T08:10:30.486156+0000","last_active":"2024-01-11T08:10:30.491292+0000","last_peered":"2024-01-11T08:10:30.491292+0000","last_clean":"2024-01-11T08:10:30.491292+0000","last_became_active":"2024-01-11T08:10:30.485871+0000","last_became_peered":"2024-01-11T08:10:30.485871+0000","last_unstale":"2024-01-11T08:10:30.491292+0000","last_undegraded":"2024-01-11T08:10:30.491292+0000","last_fullsized":"2024-01-11T08:10:30.491292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:22:43.961595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.487083+0000","last_change":"2024-01-11T08:10:30.484060+0000","last_active":"2024-01-11T08:10:30.487083+0000","last_peered":"2024-01-11T08:10:30.487083+0000","last_clean":"2024-01-11T08:10:30.487083+0000","last_became_active":"2024-01-11T08:10:30.483788+0000","last_became_peered":"2024-01-11T08:10:30.483788+0000","last_unstale":"2024-01-11T08:10:30.487083+0000","last_undegraded":"2024-01-11T08:10:30.487083+0000","last_fullsized":"2024-01-11T08:10:30.487083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:08:10.140431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.494360+0000","last_change":"2024-01-11T08:10:30.487817+0000","last_active":"2024-01-11T08:10:30.494360+0000","last_peered":"2024-01-11T08:10:30.494360+0000","last_clean":"2024-01-11T08:10:30.494360+0000","last_became_active":"2024-01-11T08:10:30.486728+0000","last_became_peered":"2024-01-11T08:10:30.486728+0000","last_unstale":"2024-01-11T08:10:30.494360+0000","last_undegraded":"2024-01-11T08:10:30.494360+0000","last_fullsized":"2024-01-11T08:10:30.494360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:33:25.110688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.494728+0000","last_change":"2024-01-11T08:10:30.490019+0000","last_active":"2024-01-11T08:10:30.494728+0000","last_peered":"2024-01-11T08:10:30.494728+0000","last_clean":"2024-01-11T08:10:30.494728+0000","last_became_active":"2024-01-11T08:10:30.489527+0000","last_became_peered":"2024-01-11T08:10:30.489527+0000","last_unstale":"2024-01-11T08:10:30.494728+0000","last_undegraded":"2024-01-11T08:10:30.494728+0000","last_fullsized":"2024-01-11T08:10:30.494728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:36:17.244182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.489961+0000","last_change":"2024-01-11T08:10:30.488678+0000","last_active":"2024-01-11T08:10:30.489961+0000","last_peered":"2024-01-11T08:10:30.489961+0000","last_clean":"2024-01-11T08:10:30.489961+0000","last_became_active":"2024-01-11T08:10:30.487511+0000","last_became_peered":"2024-01-11T08:10:30.487511+0000","last_unstale":"2024-01-11T08:10:30.489961+0000","last_undegraded":"2024-01-11T08:10:30.489961+0000","last_fullsized":"2024-01-11T08:10:30.489961+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:56:13.553335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.494305+0000","last_change":"2024-01-11T08:10:30.489544+0000","last_active":"2024-01-11T08:10:30.494305+0000","last_peered":"2024-01-11T08:10:30.494305+0000","last_clean":"2024-01-11T08:10:30.494305+0000","last_became_active":"2024-01-11T08:10:30.489020+0000","last_became_peered":"2024-01-11T08:10:30.489020+0000","last_unstale":"2024-01-11T08:10:30.494305+0000","last_undegraded":"2024-01-11T08:10:30.494305+0000","last_fullsized":"2024-01-11T08:10:30.494305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:33:25.110688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.680471+0000","last_change":"2024-01-11T08:10:29.680721+0000","last_active":"2024-01-11T08:10:30.680471+0000","last_peered":"2024-01-11T08:10:30.680471+0000","last_clean":"2024-01-11T08:10:30.680471+0000","last_became_active":"2024-01-11T08:10:29.680457+0000","last_became_peered":"2024-01-11T08:10:29.680457+0000","last_unstale":"2024-01-11T08:10:30.680471+0000","last_undegraded":"2024-01-11T08:10:30.680471+0000","last_fullsized":"2024-01-11T08:10:30.680471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:54.150785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.496156+0000","last_change":"2024-01-11T08:10:30.494749+0000","last_active":"2024-01-11T08:10:30.496156+0000","last_peered":"2024-01-11T08:10:30.496156+0000","last_clean":"2024-01-11T08:10:30.496156+0000","last_became_active":"2024-01-11T08:10:30.494471+0000","last_became_peered":"2024-01-11T08:10:30.494471+0000","last_unstale":"2024-01-11T08:10:30.496156+0000","last_undegraded":"2024-01-11T08:10:30.496156+0000","last_fullsized":"2024-01-11T08:10:30.496156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:50:04.791490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.684107+0000","last_change":"2024-01-11T08:10:29.665411+0000","last_active":"2024-01-11T08:10:30.684107+0000","last_peered":"2024-01-11T08:10:30.684107+0000","last_clean":"2024-01-11T08:10:30.684107+0000","last_became_active":"2024-01-11T08:10:29.665100+0000","last_became_peered":"2024-01-11T08:10:29.665100+0000","last_unstale":"2024-01-11T08:10:30.684107+0000","last_undegraded":"2024-01-11T08:10:30.684107+0000","last_fullsized":"2024-01-11T08:10:30.684107+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:33.696922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.493355+0000","last_change":"2024-01-11T08:10:30.490972+0000","last_active":"2024-01-11T08:10:30.493355+0000","last_peered":"2024-01-11T08:10:30.493355+0000","last_clean":"2024-01-11T08:10:30.493355+0000","last_became_active":"2024-01-11T08:10:30.490679+0000","last_became_peered":"2024-01-11T08:10:30.490679+0000","last_unstale":"2024-01-11T08:10:30.493355+0000","last_undegraded":"2024-01-11T08:10:30.493355+0000","last_fullsized":"2024-01-11T08:10:30.493355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:31:43.759712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.683204+0000","last_change":"2024-01-11T08:10:29.682048+0000","last_active":"2024-01-11T08:10:30.683204+0000","last_peered":"2024-01-11T08:10:30.683204+0000","last_clean":"2024-01-11T08:10:30.683204+0000","last_became_active":"2024-01-11T08:10:29.678360+0000","last_became_peered":"2024-01-11T08:10:29.678360+0000","last_unstale":"2024-01-11T08:10:30.683204+0000","last_undegraded":"2024-01-11T08:10:30.683204+0000","last_fullsized":"2024-01-11T08:10:30.683204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:56:12.550627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.481573+0000","last_change":"2024-01-11T08:10:30.477253+0000","last_active":"2024-01-11T08:10:30.481573+0000","last_peered":"2024-01-11T08:10:30.481573+0000","last_clean":"2024-01-11T08:10:30.481573+0000","last_became_active":"2024-01-11T08:10:30.476942+0000","last_became_peered":"2024-01-11T08:10:30.476942+0000","last_unstale":"2024-01-11T08:10:30.481573+0000","last_undegraded":"2024-01-11T08:10:30.481573+0000","last_fullsized":"2024-01-11T08:10:30.481573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:47:17.705767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.680175+0000","last_change":"2024-01-11T08:10:29.682657+0000","last_active":"2024-01-11T08:10:30.680175+0000","last_peered":"2024-01-11T08:10:30.680175+0000","last_clean":"2024-01-11T08:10:30.680175+0000","last_became_active":"2024-01-11T08:10:29.680246+0000","last_became_peered":"2024-01-11T08:10:29.680246+0000","last_unstale":"2024-01-11T08:10:30.680175+0000","last_undegraded":"2024-01-11T08:10:30.680175+0000","last_fullsized":"2024-01-11T08:10:30.680175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:48.964847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.480068+0000","last_change":"2024-01-11T08:10:30.476899+0000","last_active":"2024-01-11T08:10:30.480068+0000","last_peered":"2024-01-11T08:10:30.480068+0000","last_clean":"2024-01-11T08:10:30.480068+0000","last_became_active":"2024-01-11T08:10:30.475555+0000","last_became_peered":"2024-01-11T08:10:30.475555+0000","last_unstale":"2024-01-11T08:10:30.480068+0000","last_undegraded":"2024-01-11T08:10:30.480068+0000","last_fullsized":"2024-01-11T08:10:30.480068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:51:50.621267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.683162+0000","last_change":"2024-01-11T08:10:29.670617+0000","last_active":"2024-01-11T08:10:30.683162+0000","last_peered":"2024-01-11T08:10:30.683162+0000","last_clean":"2024-01-11T08:10:30.683162+0000","last_became_active":"2024-01-11T08:10:29.670316+0000","last_became_peered":"2024-01-11T08:10:29.670316+0000","last_unstale":"2024-01-11T08:10:30.683162+0000","last_undegraded":"2024-01-11T08:10:30.683162+0000","last_fullsized":"2024-01-11T08:10:30.683162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:50:14.474575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.486483+0000","last_change":"2024-01-11T08:10:30.477966+0000","last_active":"2024-01-11T08:10:30.486483+0000","last_peered":"2024-01-11T08:10:30.486483+0000","last_clean":"2024-01-11T08:10:30.486483+0000","last_became_active":"2024-01-11T08:10:30.477699+0000","last_became_peered":"2024-01-11T08:10:30.477699+0000","last_unstale":"2024-01-11T08:10:30.486483+0000","last_undegraded":"2024-01-11T08:10:30.486483+0000","last_fullsized":"2024-01-11T08:10:30.486483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:06.679100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.680666+0000","last_change":"2024-01-11T08:10:29.669295+0000","last_active":"2024-01-11T08:10:30.680666+0000","last_peered":"2024-01-11T08:10:30.680666+0000","last_clean":"2024-01-11T08:10:30.680666+0000","last_became_active":"2024-01-11T08:10:29.668947+0000","last_became_peered":"2024-01-11T08:10:29.668947+0000","last_unstale":"2024-01-11T08:10:30.680666+0000","last_undegraded":"2024-01-11T08:10:30.680666+0000","last_fullsized":"2024-01-11T08:10:30.680666+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:50:14.474575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.496590+0000","last_change":"2024-01-11T08:10:30.494173+0000","last_active":"2024-01-11T08:10:30.496590+0000","last_peered":"2024-01-11T08:10:30.496590+0000","last_clean":"2024-01-11T08:10:30.496590+0000","last_became_active":"2024-01-11T08:10:30.493875+0000","last_became_peered":"2024-01-11T08:10:30.493875+0000","last_unstale":"2024-01-11T08:10:30.496590+0000","last_undegraded":"2024-01-11T08:10:30.496590+0000","last_fullsized":"2024-01-11T08:10:30.496590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:47.753678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.466964+0000","last_change":"2024-01-11T08:10:29.673258+0000","last_active":"2024-01-11T08:10:30.466964+0000","last_peered":"2024-01-11T08:10:30.466964+0000","last_clean":"2024-01-11T08:10:30.466964+0000","last_became_active":"2024-01-11T08:10:29.672939+0000","last_became_peered":"2024-01-11T08:10:29.672939+0000","last_unstale":"2024-01-11T08:10:30.466964+0000","last_undegraded":"2024-01-11T08:10:30.466964+0000","last_fullsized":"2024-01-11T08:10:30.466964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:21.132838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.487736+0000","last_change":"2024-01-11T08:10:30.486579+0000","last_active":"2024-01-11T08:10:30.487736+0000","last_peered":"2024-01-11T08:10:30.487736+0000","last_clean":"2024-01-11T08:10:30.487736+0000","last_became_active":"2024-01-11T08:10:30.486332+0000","last_became_peered":"2024-01-11T08:10:30.486332+0000","last_unstale":"2024-01-11T08:10:30.487736+0000","last_undegraded":"2024-01-11T08:10:30.487736+0000","last_fullsized":"2024-01-11T08:10:30.487736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:03:43.208019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.471082+0000","last_change":"2024-01-11T08:10:29.676952+0000","last_active":"2024-01-11T08:10:30.471082+0000","last_peered":"2024-01-11T08:10:30.471082+0000","last_clean":"2024-01-11T08:10:30.471082+0000","last_became_active":"2024-01-11T08:10:29.676624+0000","last_became_peered":"2024-01-11T08:10:29.676624+0000","last_unstale":"2024-01-11T08:10:30.471082+0000","last_undegraded":"2024-01-11T08:10:30.471082+0000","last_fullsized":"2024-01-11T08:10:30.471082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:05.676391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.487063+0000","last_change":"2024-01-11T08:10:30.481939+0000","last_active":"2024-01-11T08:10:30.487063+0000","last_peered":"2024-01-11T08:10:30.487063+0000","last_clean":"2024-01-11T08:10:30.487063+0000","last_became_active":"2024-01-11T08:10:30.481619+0000","last_became_peered":"2024-01-11T08:10:30.481619+0000","last_unstale":"2024-01-11T08:10:30.487063+0000","last_undegraded":"2024-01-11T08:10:30.487063+0000","last_fullsized":"2024-01-11T08:10:30.487063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:03:43.208019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.485859+0000","last_change":"2024-01-11T08:10:29.679415+0000","last_active":"2024-01-11T08:10:30.485859+0000","last_peered":"2024-01-11T08:10:30.485859+0000","last_clean":"2024-01-11T08:10:30.485859+0000","last_became_active":"2024-01-11T08:10:29.679087+0000","last_became_peered":"2024-01-11T08:10:29.679087+0000","last_unstale":"2024-01-11T08:10:30.485859+0000","last_undegraded":"2024-01-11T08:10:30.485859+0000","last_fullsized":"2024-01-11T08:10:30.485859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:05.676391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.497485+0000","last_change":"2024-01-11T08:10:30.495369+0000","last_active":"2024-01-11T08:10:30.497485+0000","last_peered":"2024-01-11T08:10:30.497485+0000","last_clean":"2024-01-11T08:10:30.497485+0000","last_became_active":"2024-01-11T08:10:30.495037+0000","last_became_peered":"2024-01-11T08:10:30.495037+0000","last_unstale":"2024-01-11T08:10:30.497485+0000","last_undegraded":"2024-01-11T08:10:30.497485+0000","last_fullsized":"2024-01-11T08:10:30.497485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:21:29.220934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.472907+0000","last_change":"2024-01-11T08:10:29.681233+0000","last_active":"2024-01-11T08:10:30.472907+0000","last_peered":"2024-01-11T08:10:30.472907+0000","last_clean":"2024-01-11T08:10:30.472907+0000","last_became_active":"2024-01-11T08:10:29.680606+0000","last_became_peered":"2024-01-11T08:10:29.680606+0000","last_unstale":"2024-01-11T08:10:30.472907+0000","last_undegraded":"2024-01-11T08:10:30.472907+0000","last_fullsized":"2024-01-11T08:10:30.472907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:12:48.941888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.491516+0000","last_change":"2024-01-11T08:10:30.487651+0000","last_active":"2024-01-11T08:10:30.491516+0000","last_peered":"2024-01-11T08:10:30.491516+0000","last_clean":"2024-01-11T08:10:30.491516+0000","last_became_active":"2024-01-11T08:10:30.486641+0000","last_became_peered":"2024-01-11T08:10:30.486641+0000","last_unstale":"2024-01-11T08:10:30.491516+0000","last_undegraded":"2024-01-11T08:10:30.491516+0000","last_fullsized":"2024-01-11T08:10:30.491516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:01:02.902411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.678617+0000","last_change":"2024-01-11T08:10:29.666353+0000","last_active":"2024-01-11T08:10:30.678617+0000","last_peered":"2024-01-11T08:10:30.678617+0000","last_clean":"2024-01-11T08:10:30.678617+0000","last_became_active":"2024-01-11T08:10:29.665995+0000","last_became_peered":"2024-01-11T08:10:29.665995+0000","last_unstale":"2024-01-11T08:10:30.678617+0000","last_undegraded":"2024-01-11T08:10:30.678617+0000","last_fullsized":"2024-01-11T08:10:30.678617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:12:48.941888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.503379+0000","last_change":"2024-01-11T08:10:30.499297+0000","last_active":"2024-01-11T08:10:30.503379+0000","last_peered":"2024-01-11T08:10:30.503379+0000","last_clean":"2024-01-11T08:10:30.503379+0000","last_became_active":"2024-01-11T08:10:30.497567+0000","last_became_peered":"2024-01-11T08:10:30.497567+0000","last_unstale":"2024-01-11T08:10:30.503379+0000","last_undegraded":"2024-01-11T08:10:30.503379+0000","last_fullsized":"2024-01-11T08:10:30.503379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:49.967555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.685960+0000","last_change":"2024-01-11T08:10:29.671697+0000","last_active":"2024-01-11T08:10:30.685960+0000","last_peered":"2024-01-11T08:10:30.685960+0000","last_clean":"2024-01-11T08:10:30.685960+0000","last_became_active":"2024-01-11T08:10:29.671395+0000","last_became_peered":"2024-01-11T08:10:29.671395+0000","last_unstale":"2024-01-11T08:10:30.685960+0000","last_undegraded":"2024-01-11T08:10:30.685960+0000","last_fullsized":"2024-01-11T08:10:30.685960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:12:48.941888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.494324+0000","last_change":"2024-01-11T08:10:30.489779+0000","last_active":"2024-01-11T08:10:30.494324+0000","last_peered":"2024-01-11T08:10:30.494324+0000","last_clean":"2024-01-11T08:10:30.494324+0000","last_became_active":"2024-01-11T08:10:30.489290+0000","last_became_peered":"2024-01-11T08:10:30.489290+0000","last_unstale":"2024-01-11T08:10:30.494324+0000","last_undegraded":"2024-01-11T08:10:30.494324+0000","last_fullsized":"2024-01-11T08:10:30.494324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:50:04.791490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.678500+0000","last_change":"2024-01-11T08:10:29.682961+0000","last_active":"2024-01-11T08:10:30.678500+0000","last_peered":"2024-01-11T08:10:30.678500+0000","last_clean":"2024-01-11T08:10:30.678500+0000","last_became_active":"2024-01-11T08:10:29.682646+0000","last_became_peered":"2024-01-11T08:10:29.682646+0000","last_unstale":"2024-01-11T08:10:30.678500+0000","last_undegraded":"2024-01-11T08:10:30.678500+0000","last_fullsized":"2024-01-11T08:10:30.678500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:11:42.997404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.496569+0000","last_change":"2024-01-11T08:10:30.493989+0000","last_active":"2024-01-11T08:10:30.496569+0000","last_peered":"2024-01-11T08:10:30.496569+0000","last_clean":"2024-01-11T08:10:30.496569+0000","last_became_active":"2024-01-11T08:10:30.493678+0000","last_became_peered":"2024-01-11T08:10:30.493678+0000","last_unstale":"2024-01-11T08:10:30.496569+0000","last_undegraded":"2024-01-11T08:10:30.496569+0000","last_fullsized":"2024-01-11T08:10:30.496569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:03:43.208019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.471218+0000","last_change":"2024-01-11T08:10:29.670911+0000","last_active":"2024-01-11T08:10:30.471218+0000","last_peered":"2024-01-11T08:10:30.471218+0000","last_clean":"2024-01-11T08:10:30.471218+0000","last_became_active":"2024-01-11T08:10:29.670615+0000","last_became_peered":"2024-01-11T08:10:29.670615+0000","last_unstale":"2024-01-11T08:10:30.471218+0000","last_undegraded":"2024-01-11T08:10:30.471218+0000","last_fullsized":"2024-01-11T08:10:30.471218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:46:57.523656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.500438+0000","last_change":"2024-01-11T08:10:30.497306+0000","last_active":"2024-01-11T08:10:30.500438+0000","last_peered":"2024-01-11T08:10:30.500438+0000","last_clean":"2024-01-11T08:10:30.500438+0000","last_became_active":"2024-01-11T08:10:30.496966+0000","last_became_peered":"2024-01-11T08:10:30.496966+0000","last_unstale":"2024-01-11T08:10:30.500438+0000","last_undegraded":"2024-01-11T08:10:30.500438+0000","last_fullsized":"2024-01-11T08:10:30.500438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:47:17.705767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.483926+0000","last_change":"2024-01-11T08:10:29.680426+0000","last_active":"2024-01-11T08:10:30.483926+0000","last_peered":"2024-01-11T08:10:30.483926+0000","last_clean":"2024-01-11T08:10:30.483926+0000","last_became_active":"2024-01-11T08:10:29.680133+0000","last_became_peered":"2024-01-11T08:10:29.680133+0000","last_unstale":"2024-01-11T08:10:30.483926+0000","last_undegraded":"2024-01-11T08:10:30.483926+0000","last_fullsized":"2024-01-11T08:10:30.483926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:21.132838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.500317+0000","last_change":"2024-01-11T08:10:30.494824+0000","last_active":"2024-01-11T08:10:30.500317+0000","last_peered":"2024-01-11T08:10:30.500317+0000","last_clean":"2024-01-11T08:10:30.500317+0000","last_became_active":"2024-01-11T08:10:30.494564+0000","last_became_peered":"2024-01-11T08:10:30.494564+0000","last_unstale":"2024-01-11T08:10:30.500317+0000","last_undegraded":"2024-01-11T08:10:30.500317+0000","last_fullsized":"2024-01-11T08:10:30.500317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:37:25.109490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.687461+0000","last_change":"2024-01-11T08:10:29.671294+0000","last_active":"2024-01-11T08:10:30.687461+0000","last_peered":"2024-01-11T08:10:30.687461+0000","last_clean":"2024-01-11T08:10:30.687461+0000","last_became_active":"2024-01-11T08:10:29.670997+0000","last_became_peered":"2024-01-11T08:10:29.670997+0000","last_unstale":"2024-01-11T08:10:30.687461+0000","last_undegraded":"2024-01-11T08:10:30.687461+0000","last_fullsized":"2024-01-11T08:10:30.687461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:46:57.523656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.502943+0000","last_change":"2024-01-11T08:10:30.498224+0000","last_active":"2024-01-11T08:10:30.502943+0000","last_peered":"2024-01-11T08:10:30.502943+0000","last_clean":"2024-01-11T08:10:30.502943+0000","last_became_active":"2024-01-11T08:10:30.497993+0000","last_became_peered":"2024-01-11T08:10:30.497993+0000","last_unstale":"2024-01-11T08:10:30.502943+0000","last_undegraded":"2024-01-11T08:10:30.502943+0000","last_fullsized":"2024-01-11T08:10:30.502943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:51:50.621267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.676301+0000","last_change":"2024-01-11T08:10:29.678760+0000","last_active":"2024-01-11T08:10:30.676301+0000","last_peered":"2024-01-11T08:10:30.676301+0000","last_clean":"2024-01-11T08:10:30.676301+0000","last_became_active":"2024-01-11T08:10:29.678519+0000","last_became_peered":"2024-01-11T08:10:29.678519+0000","last_unstale":"2024-01-11T08:10:30.676301+0000","last_undegraded":"2024-01-11T08:10:30.676301+0000","last_fullsized":"2024-01-11T08:10:30.676301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:11:42.997404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.505207+0000","last_change":"2024-01-11T08:10:30.504188+0000","last_active":"2024-01-11T08:10:30.505207+0000","last_peered":"2024-01-11T08:10:30.505207+0000","last_clean":"2024-01-11T08:10:30.505207+0000","last_became_active":"2024-01-11T08:10:30.503805+0000","last_became_peered":"2024-01-11T08:10:30.503805+0000","last_unstale":"2024-01-11T08:10:30.505207+0000","last_undegraded":"2024-01-11T08:10:30.505207+0000","last_fullsized":"2024-01-11T08:10:30.505207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:36:17.244182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.473300+0000","last_change":"2024-01-11T08:10:29.671224+0000","last_active":"2024-01-11T08:10:30.473300+0000","last_peered":"2024-01-11T08:10:30.473300+0000","last_clean":"2024-01-11T08:10:30.473300+0000","last_became_active":"2024-01-11T08:10:29.670078+0000","last_became_peered":"2024-01-11T08:10:29.670078+0000","last_unstale":"2024-01-11T08:10:30.473300+0000","last_undegraded":"2024-01-11T08:10:30.473300+0000","last_fullsized":"2024-01-11T08:10:30.473300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:50:45.947278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.486167+0000","last_change":"2024-01-11T08:10:30.479486+0000","last_active":"2024-01-11T08:10:30.486167+0000","last_peered":"2024-01-11T08:10:30.486167+0000","last_clean":"2024-01-11T08:10:30.486167+0000","last_became_active":"2024-01-11T08:10:30.477788+0000","last_became_peered":"2024-01-11T08:10:30.477788+0000","last_unstale":"2024-01-11T08:10:30.486167+0000","last_undegraded":"2024-01-11T08:10:30.486167+0000","last_fullsized":"2024-01-11T08:10:30.486167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:50:04.791490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.471183+0000","last_change":"2024-01-11T08:10:29.670241+0000","last_active":"2024-01-11T08:10:30.471183+0000","last_peered":"2024-01-11T08:10:30.471183+0000","last_clean":"2024-01-11T08:10:30.471183+0000","last_became_active":"2024-01-11T08:10:29.669919+0000","last_became_peered":"2024-01-11T08:10:29.669919+0000","last_unstale":"2024-01-11T08:10:30.471183+0000","last_undegraded":"2024-01-11T08:10:30.471183+0000","last_fullsized":"2024-01-11T08:10:30.471183+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:50:14.474575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.498632+0000","last_change":"2024-01-11T08:10:30.490994+0000","last_active":"2024-01-11T08:10:30.498632+0000","last_peered":"2024-01-11T08:10:30.498632+0000","last_clean":"2024-01-11T08:10:30.498632+0000","last_became_active":"2024-01-11T08:10:30.490710+0000","last_became_peered":"2024-01-11T08:10:30.490710+0000","last_unstale":"2024-01-11T08:10:30.498632+0000","last_undegraded":"2024-01-11T08:10:30.498632+0000","last_fullsized":"2024-01-11T08:10:30.498632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:56:13.553335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.486574+0000","last_change":"2024-01-11T08:10:30.483950+0000","last_active":"2024-01-11T08:10:30.486574+0000","last_peered":"2024-01-11T08:10:30.486574+0000","last_clean":"2024-01-11T08:10:30.486574+0000","last_became_active":"2024-01-11T08:10:30.483643+0000","last_became_peered":"2024-01-11T08:10:30.483643+0000","last_unstale":"2024-01-11T08:10:30.486574+0000","last_undegraded":"2024-01-11T08:10:30.486574+0000","last_fullsized":"2024-01-11T08:10:30.486574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:47.753678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.472149+0000","last_change":"2024-01-11T08:10:29.667798+0000","last_active":"2024-01-11T08:10:30.472149+0000","last_peered":"2024-01-11T08:10:30.472149+0000","last_clean":"2024-01-11T08:10:30.472149+0000","last_became_active":"2024-01-11T08:10:29.667455+0000","last_became_peered":"2024-01-11T08:10:29.667455+0000","last_unstale":"2024-01-11T08:10:30.472149+0000","last_undegraded":"2024-01-11T08:10:30.472149+0000","last_fullsized":"2024-01-11T08:10:30.472149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:33.696922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.480195+0000","last_change":"2024-01-11T08:10:30.477204+0000","last_active":"2024-01-11T08:10:30.480195+0000","last_peered":"2024-01-11T08:10:30.480195+0000","last_clean":"2024-01-11T08:10:30.480195+0000","last_became_active":"2024-01-11T08:10:30.476896+0000","last_became_peered":"2024-01-11T08:10:30.476896+0000","last_unstale":"2024-01-11T08:10:30.480195+0000","last_undegraded":"2024-01-11T08:10:30.480195+0000","last_fullsized":"2024-01-11T08:10:30.480195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:22:43.961595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.472646+0000","last_change":"2024-01-11T08:10:29.666032+0000","last_active":"2024-01-11T08:10:30.472646+0000","last_peered":"2024-01-11T08:10:30.472646+0000","last_clean":"2024-01-11T08:10:30.472646+0000","last_became_active":"2024-01-11T08:10:29.665708+0000","last_became_peered":"2024-01-11T08:10:29.665708+0000","last_unstale":"2024-01-11T08:10:30.472646+0000","last_undegraded":"2024-01-11T08:10:30.472646+0000","last_fullsized":"2024-01-11T08:10:30.472646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:50:45.947278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.476809+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:10:30.476809+0000","last_peered":"2024-01-11T08:10:30.476809+0000","last_clean":"2024-01-11T08:10:30.476809+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:10:30.476809+0000","last_undegraded":"2024-01-11T08:10:30.476809+0000","last_fullsized":"2024-01-11T08:10:30.476809+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.495364+0000","last_change":"2024-01-11T08:10:30.491665+0000","last_active":"2024-01-11T08:10:30.495364+0000","last_peered":"2024-01-11T08:10:30.495364+0000","last_clean":"2024-01-11T08:10:30.495364+0000","last_became_active":"2024-01-11T08:10:30.491329+0000","last_became_peered":"2024-01-11T08:10:30.491329+0000","last_unstale":"2024-01-11T08:10:30.495364+0000","last_undegraded":"2024-01-11T08:10:30.495364+0000","last_fullsized":"2024-01-11T08:10:30.495364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:26:51.888257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.476968+0000","last_change":"2024-01-11T08:10:29.676382+0000","last_active":"2024-01-11T08:10:30.476968+0000","last_peered":"2024-01-11T08:10:30.476968+0000","last_clean":"2024-01-11T08:10:30.476968+0000","last_became_active":"2024-01-11T08:10:29.674690+0000","last_became_peered":"2024-01-11T08:10:29.674690+0000","last_unstale":"2024-01-11T08:10:30.476968+0000","last_undegraded":"2024-01-11T08:10:30.476968+0000","last_fullsized":"2024-01-11T08:10:30.476968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:12:48.941888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.479210+0000","last_change":"2024-01-11T08:10:29.680063+0000","last_active":"2024-01-11T08:10:30.479210+0000","last_peered":"2024-01-11T08:10:30.479210+0000","last_clean":"2024-01-11T08:10:30.479210+0000","last_became_active":"2024-01-11T08:10:29.677976+0000","last_became_peered":"2024-01-11T08:10:29.677976+0000","last_unstale":"2024-01-11T08:10:30.479210+0000","last_undegraded":"2024-01-11T08:10:30.479210+0000","last_fullsized":"2024-01-11T08:10:30.479210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:50:45.947278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.486597+0000","last_change":"2024-01-11T08:10:30.483048+0000","last_active":"2024-01-11T08:10:30.486597+0000","last_peered":"2024-01-11T08:10:30.486597+0000","last_clean":"2024-01-11T08:10:30.486597+0000","last_became_active":"2024-01-11T08:10:30.482693+0000","last_became_peered":"2024-01-11T08:10:30.482693+0000","last_unstale":"2024-01-11T08:10:30.486597+0000","last_undegraded":"2024-01-11T08:10:30.486597+0000","last_fullsized":"2024-01-11T08:10:30.486597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:22:43.961595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T08:10:31.469542+0000","last_change":"2024-01-11T08:10:29.673099+0000","last_active":"2024-01-11T08:10:31.469542+0000","last_peered":"2024-01-11T08:10:31.469542+0000","last_clean":"2024-01-11T08:10:31.469542+0000","last_became_active":"2024-01-11T08:10:29.672796+0000","last_became_peered":"2024-01-11T08:10:29.672796+0000","last_unstale":"2024-01-11T08:10:31.469542+0000","last_undegraded":"2024-01-11T08:10:31.469542+0000","last_fullsized":"2024-01-11T08:10:31.469542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:05.676391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.490638+0000","last_change":"2024-01-11T08:10:30.489252+0000","last_active":"2024-01-11T08:10:30.490638+0000","last_peered":"2024-01-11T08:10:30.490638+0000","last_clean":"2024-01-11T08:10:30.490638+0000","last_became_active":"2024-01-11T08:10:30.488728+0000","last_became_peered":"2024-01-11T08:10:30.488728+0000","last_unstale":"2024-01-11T08:10:30.490638+0000","last_undegraded":"2024-01-11T08:10:30.490638+0000","last_fullsized":"2024-01-11T08:10:30.490638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:01:02.902411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.680479+0000","last_change":"2024-01-11T08:10:29.678413+0000","last_active":"2024-01-11T08:10:30.680479+0000","last_peered":"2024-01-11T08:10:30.680479+0000","last_clean":"2024-01-11T08:10:30.680479+0000","last_became_active":"2024-01-11T08:10:29.678023+0000","last_became_peered":"2024-01-11T08:10:29.678023+0000","last_unstale":"2024-01-11T08:10:30.680479+0000","last_undegraded":"2024-01-11T08:10:30.680479+0000","last_fullsized":"2024-01-11T08:10:30.680479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:50:45.947278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.491408+0000","last_change":"2024-01-11T08:10:30.486306+0000","last_active":"2024-01-11T08:10:30.491408+0000","last_peered":"2024-01-11T08:10:30.491408+0000","last_clean":"2024-01-11T08:10:30.491408+0000","last_became_active":"2024-01-11T08:10:30.486002+0000","last_became_peered":"2024-01-11T08:10:30.486002+0000","last_unstale":"2024-01-11T08:10:30.491408+0000","last_undegraded":"2024-01-11T08:10:30.491408+0000","last_fullsized":"2024-01-11T08:10:30.491408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:21.127779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.476646+0000","last_change":"2024-01-11T08:10:29.674369+0000","last_active":"2024-01-11T08:10:30.476646+0000","last_peered":"2024-01-11T08:10:30.476646+0000","last_clean":"2024-01-11T08:10:30.476646+0000","last_became_active":"2024-01-11T08:10:29.673110+0000","last_became_peered":"2024-01-11T08:10:29.673110+0000","last_unstale":"2024-01-11T08:10:30.476646+0000","last_undegraded":"2024-01-11T08:10:30.476646+0000","last_fullsized":"2024-01-11T08:10:30.476646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:27:23.756042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.483307+0000","last_change":"2024-01-11T08:10:30.476023+0000","last_active":"2024-01-11T08:10:30.483307+0000","last_peered":"2024-01-11T08:10:30.483307+0000","last_clean":"2024-01-11T08:10:30.483307+0000","last_became_active":"2024-01-11T08:10:30.474988+0000","last_became_peered":"2024-01-11T08:10:30.474988+0000","last_unstale":"2024-01-11T08:10:30.483307+0000","last_undegraded":"2024-01-11T08:10:30.483307+0000","last_fullsized":"2024-01-11T08:10:30.483307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:12:49.944597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.472984+0000","last_change":"2024-01-11T08:10:29.675819+0000","last_active":"2024-01-11T08:10:30.472984+0000","last_peered":"2024-01-11T08:10:30.472984+0000","last_clean":"2024-01-11T08:10:30.472984+0000","last_became_active":"2024-01-11T08:10:29.675005+0000","last_became_peered":"2024-01-11T08:10:29.675005+0000","last_unstale":"2024-01-11T08:10:30.472984+0000","last_undegraded":"2024-01-11T08:10:30.472984+0000","last_fullsized":"2024-01-11T08:10:30.472984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:27:23.756042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.479995+0000","last_change":"2024-01-11T08:10:30.475525+0000","last_active":"2024-01-11T08:10:30.479995+0000","last_peered":"2024-01-11T08:10:30.479995+0000","last_clean":"2024-01-11T08:10:30.479995+0000","last_became_active":"2024-01-11T08:10:30.475243+0000","last_became_peered":"2024-01-11T08:10:30.475243+0000","last_unstale":"2024-01-11T08:10:30.479995+0000","last_undegraded":"2024-01-11T08:10:30.479995+0000","last_fullsized":"2024-01-11T08:10:30.479995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:37:39.503131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.473867+0000","last_change":"2024-01-11T08:10:29.672091+0000","last_active":"2024-01-11T08:10:30.473867+0000","last_peered":"2024-01-11T08:10:30.473867+0000","last_clean":"2024-01-11T08:10:30.473867+0000","last_became_active":"2024-01-11T08:10:29.671820+0000","last_became_peered":"2024-01-11T08:10:29.671820+0000","last_unstale":"2024-01-11T08:10:30.473867+0000","last_undegraded":"2024-01-11T08:10:30.473867+0000","last_fullsized":"2024-01-11T08:10:30.473867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:46:57.523656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.490433+0000","last_change":"2024-01-11T08:10:30.488516+0000","last_active":"2024-01-11T08:10:30.490433+0000","last_peered":"2024-01-11T08:10:30.490433+0000","last_clean":"2024-01-11T08:10:30.490433+0000","last_became_active":"2024-01-11T08:10:30.487592+0000","last_became_peered":"2024-01-11T08:10:30.487592+0000","last_unstale":"2024-01-11T08:10:30.490433+0000","last_undegraded":"2024-01-11T08:10:30.490433+0000","last_fullsized":"2024-01-11T08:10:30.490433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:57:19.989587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.678978+0000","last_change":"2024-01-11T08:10:29.683377+0000","last_active":"2024-01-11T08:10:30.678978+0000","last_peered":"2024-01-11T08:10:30.678978+0000","last_clean":"2024-01-11T08:10:30.678978+0000","last_became_active":"2024-01-11T08:10:29.680358+0000","last_became_peered":"2024-01-11T08:10:29.680358+0000","last_unstale":"2024-01-11T08:10:30.678978+0000","last_undegraded":"2024-01-11T08:10:30.678978+0000","last_fullsized":"2024-01-11T08:10:30.678978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:05.676391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.502306+0000","last_change":"2024-01-11T08:10:30.499934+0000","last_active":"2024-01-11T08:10:30.502306+0000","last_peered":"2024-01-11T08:10:30.502306+0000","last_clean":"2024-01-11T08:10:30.502306+0000","last_became_active":"2024-01-11T08:10:30.499676+0000","last_became_peered":"2024-01-11T08:10:30.499676+0000","last_unstale":"2024-01-11T08:10:30.502306+0000","last_undegraded":"2024-01-11T08:10:30.502306+0000","last_fullsized":"2024-01-11T08:10:30.502306+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:47.753678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.471402+0000","last_change":"2024-01-11T08:10:29.665360+0000","last_active":"2024-01-11T08:10:30.471402+0000","last_peered":"2024-01-11T08:10:30.471402+0000","last_clean":"2024-01-11T08:10:30.471402+0000","last_became_active":"2024-01-11T08:10:29.664493+0000","last_became_peered":"2024-01-11T08:10:29.664493+0000","last_unstale":"2024-01-11T08:10:30.471402+0000","last_undegraded":"2024-01-11T08:10:30.471402+0000","last_fullsized":"2024-01-11T08:10:30.471402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:12:48.941888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.495532+0000","last_change":"2024-01-11T08:10:30.491426+0000","last_active":"2024-01-11T08:10:30.495532+0000","last_peered":"2024-01-11T08:10:30.495532+0000","last_clean":"2024-01-11T08:10:30.495532+0000","last_became_active":"2024-01-11T08:10:30.491167+0000","last_became_peered":"2024-01-11T08:10:30.491167+0000","last_unstale":"2024-01-11T08:10:30.495532+0000","last_undegraded":"2024-01-11T08:10:30.495532+0000","last_fullsized":"2024-01-11T08:10:30.495532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:12.649092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.678720+0000","last_change":"2024-01-11T08:10:29.669311+0000","last_active":"2024-01-11T08:10:30.678720+0000","last_peered":"2024-01-11T08:10:30.678720+0000","last_clean":"2024-01-11T08:10:30.678720+0000","last_became_active":"2024-01-11T08:10:29.669042+0000","last_became_peered":"2024-01-11T08:10:29.669042+0000","last_unstale":"2024-01-11T08:10:30.678720+0000","last_undegraded":"2024-01-11T08:10:30.678720+0000","last_fullsized":"2024-01-11T08:10:30.678720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:48.964847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.494014+0000","last_change":"2024-01-11T08:10:30.486285+0000","last_active":"2024-01-11T08:10:30.494014+0000","last_peered":"2024-01-11T08:10:30.494014+0000","last_clean":"2024-01-11T08:10:30.494014+0000","last_became_active":"2024-01-11T08:10:30.482784+0000","last_became_peered":"2024-01-11T08:10:30.482784+0000","last_unstale":"2024-01-11T08:10:30.494014+0000","last_undegraded":"2024-01-11T08:10:30.494014+0000","last_fullsized":"2024-01-11T08:10:30.494014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:37:39.503131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.687135+0000","last_change":"2024-01-11T08:10:29.668553+0000","last_active":"2024-01-11T08:10:30.687135+0000","last_peered":"2024-01-11T08:10:30.687135+0000","last_clean":"2024-01-11T08:10:30.687135+0000","last_became_active":"2024-01-11T08:10:29.666406+0000","last_became_peered":"2024-01-11T08:10:29.666406+0000","last_unstale":"2024-01-11T08:10:30.687135+0000","last_undegraded":"2024-01-11T08:10:30.687135+0000","last_fullsized":"2024-01-11T08:10:30.687135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:50:14.474575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.503635+0000","last_change":"2024-01-11T08:10:30.501614+0000","last_active":"2024-01-11T08:10:30.503635+0000","last_peered":"2024-01-11T08:10:30.503635+0000","last_clean":"2024-01-11T08:10:30.503635+0000","last_became_active":"2024-01-11T08:10:30.501255+0000","last_became_peered":"2024-01-11T08:10:30.501255+0000","last_unstale":"2024-01-11T08:10:30.503635+0000","last_undegraded":"2024-01-11T08:10:30.503635+0000","last_fullsized":"2024-01-11T08:10:30.503635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:18:09.736814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.491229+0000","last_change":"2024-01-11T08:10:30.488971+0000","last_active":"2024-01-11T08:10:30.491229+0000","last_peered":"2024-01-11T08:10:30.491229+0000","last_clean":"2024-01-11T08:10:30.491229+0000","last_became_active":"2024-01-11T08:10:30.488681+0000","last_became_peered":"2024-01-11T08:10:30.488681+0000","last_unstale":"2024-01-11T08:10:30.491229+0000","last_undegraded":"2024-01-11T08:10:30.491229+0000","last_fullsized":"2024-01-11T08:10:30.491229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:26:51.888257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.481588+0000","last_change":"2024-01-11T08:10:30.477289+0000","last_active":"2024-01-11T08:10:30.481588+0000","last_peered":"2024-01-11T08:10:30.481588+0000","last_clean":"2024-01-11T08:10:30.481588+0000","last_became_active":"2024-01-11T08:10:30.477029+0000","last_became_peered":"2024-01-11T08:10:30.477029+0000","last_unstale":"2024-01-11T08:10:30.481588+0000","last_undegraded":"2024-01-11T08:10:30.481588+0000","last_fullsized":"2024-01-11T08:10:30.481588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:49.967555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.483210+0000","last_change":"2024-01-11T08:10:30.478289+0000","last_active":"2024-01-11T08:10:30.483210+0000","last_peered":"2024-01-11T08:10:30.483210+0000","last_clean":"2024-01-11T08:10:30.483210+0000","last_became_active":"2024-01-11T08:10:30.477815+0000","last_became_peered":"2024-01-11T08:10:30.477815+0000","last_unstale":"2024-01-11T08:10:30.483210+0000","last_undegraded":"2024-01-11T08:10:30.483210+0000","last_fullsized":"2024-01-11T08:10:30.483210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:14:36.595264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.491328+0000","last_change":"2024-01-11T08:10:30.486406+0000","last_active":"2024-01-11T08:10:30.491328+0000","last_peered":"2024-01-11T08:10:30.491328+0000","last_clean":"2024-01-11T08:10:30.491328+0000","last_became_active":"2024-01-11T08:10:30.485979+0000","last_became_peered":"2024-01-11T08:10:30.485979+0000","last_unstale":"2024-01-11T08:10:30.491328+0000","last_undegraded":"2024-01-11T08:10:30.491328+0000","last_fullsized":"2024-01-11T08:10:30.491328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:37:25.109490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.485482+0000","last_change":"2024-01-11T08:10:30.480947+0000","last_active":"2024-01-11T08:10:30.485482+0000","last_peered":"2024-01-11T08:10:30.485482+0000","last_clean":"2024-01-11T08:10:30.485482+0000","last_became_active":"2024-01-11T08:10:30.478542+0000","last_became_peered":"2024-01-11T08:10:30.478542+0000","last_unstale":"2024-01-11T08:10:30.485482+0000","last_undegraded":"2024-01-11T08:10:30.485482+0000","last_fullsized":"2024-01-11T08:10:30.485482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:21:29.220934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.487827+0000","last_change":"2024-01-11T08:10:30.486206+0000","last_active":"2024-01-11T08:10:30.487827+0000","last_peered":"2024-01-11T08:10:30.487827+0000","last_clean":"2024-01-11T08:10:30.487827+0000","last_became_active":"2024-01-11T08:10:30.482118+0000","last_became_peered":"2024-01-11T08:10:30.482118+0000","last_unstale":"2024-01-11T08:10:30.487827+0000","last_undegraded":"2024-01-11T08:10:30.487827+0000","last_fullsized":"2024-01-11T08:10:30.487827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:37:39.503131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.502193+0000","last_change":"2024-01-11T08:10:30.498251+0000","last_active":"2024-01-11T08:10:30.502193+0000","last_peered":"2024-01-11T08:10:30.502193+0000","last_clean":"2024-01-11T08:10:30.502193+0000","last_became_active":"2024-01-11T08:10:30.498032+0000","last_became_peered":"2024-01-11T08:10:30.498032+0000","last_unstale":"2024-01-11T08:10:30.502193+0000","last_undegraded":"2024-01-11T08:10:30.502193+0000","last_fullsized":"2024-01-11T08:10:30.502193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:22:43.961595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.480236+0000","last_change":"2024-01-11T08:10:30.477044+0000","last_active":"2024-01-11T08:10:30.480236+0000","last_peered":"2024-01-11T08:10:30.480236+0000","last_clean":"2024-01-11T08:10:30.480236+0000","last_became_active":"2024-01-11T08:10:30.476728+0000","last_became_peered":"2024-01-11T08:10:30.476728+0000","last_unstale":"2024-01-11T08:10:30.480236+0000","last_undegraded":"2024-01-11T08:10:30.480236+0000","last_fullsized":"2024-01-11T08:10:30.480236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:08:10.140431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.485414+0000","last_change":"2024-01-11T08:10:30.481145+0000","last_active":"2024-01-11T08:10:30.485414+0000","last_peered":"2024-01-11T08:10:30.485414+0000","last_clean":"2024-01-11T08:10:30.485414+0000","last_became_active":"2024-01-11T08:10:30.477464+0000","last_became_peered":"2024-01-11T08:10:30.477464+0000","last_unstale":"2024-01-11T08:10:30.485414+0000","last_undegraded":"2024-01-11T08:10:30.485414+0000","last_fullsized":"2024-01-11T08:10:30.485414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:02.887025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.485358+0000","last_change":"2024-01-11T08:10:30.480047+0000","last_active":"2024-01-11T08:10:30.485358+0000","last_peered":"2024-01-11T08:10:30.485358+0000","last_clean":"2024-01-11T08:10:30.485358+0000","last_became_active":"2024-01-11T08:10:30.479594+0000","last_became_peered":"2024-01-11T08:10:30.479594+0000","last_unstale":"2024-01-11T08:10:30.485358+0000","last_undegraded":"2024-01-11T08:10:30.485358+0000","last_fullsized":"2024-01-11T08:10:30.485358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:14:36.595264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.490024+0000","last_change":"2024-01-11T08:10:30.486964+0000","last_active":"2024-01-11T08:10:30.490024+0000","last_peered":"2024-01-11T08:10:30.490024+0000","last_clean":"2024-01-11T08:10:30.490024+0000","last_became_active":"2024-01-11T08:10:30.486679+0000","last_became_peered":"2024-01-11T08:10:30.486679+0000","last_unstale":"2024-01-11T08:10:30.490024+0000","last_undegraded":"2024-01-11T08:10:30.490024+0000","last_fullsized":"2024-01-11T08:10:30.490024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:57:19.989587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.497053+0000","last_change":"2024-01-11T08:10:30.495904+0000","last_active":"2024-01-11T08:10:30.497053+0000","last_peered":"2024-01-11T08:10:30.497053+0000","last_clean":"2024-01-11T08:10:30.497053+0000","last_became_active":"2024-01-11T08:10:30.495590+0000","last_became_peered":"2024-01-11T08:10:30.495590+0000","last_unstale":"2024-01-11T08:10:30.497053+0000","last_undegraded":"2024-01-11T08:10:30.497053+0000","last_fullsized":"2024-01-11T08:10:30.497053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:37:39.503131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.502287+0000","last_change":"2024-01-11T08:10:30.499796+0000","last_active":"2024-01-11T08:10:30.502287+0000","last_peered":"2024-01-11T08:10:30.502287+0000","last_clean":"2024-01-11T08:10:30.502287+0000","last_became_active":"2024-01-11T08:10:30.499391+0000","last_became_peered":"2024-01-11T08:10:30.499391+0000","last_unstale":"2024-01-11T08:10:30.502287+0000","last_undegraded":"2024-01-11T08:10:30.502287+0000","last_fullsized":"2024-01-11T08:10:30.502287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:37:39.503131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298084,"kb_used_data":796,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073756,"statfs":{"total":96636764160,"available":96331526144,"internally_reserved":0,"allocated":815104,"data_stored":666235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9119999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":76849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":225280,"data_stored":75473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298092,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073748,"statfs":{"total":96636764160,"available":96331517952,"internally_reserved":0,"allocated":823296,"data_stored":666291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":217088,"data_stored":75313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81699999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":76849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":75439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":221184,"data_stored":75403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:10:38.968 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:10:39.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.202+0000 7f6402fff700 1 Processor -- start 2024-01-11T08:10:39.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.202+0000 7f6402fff700 1 -- start start 2024-01-11T08:10:39.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.203+0000 7f6402fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f63fc0b2b90 0x7f63fc159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:39.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.204+0000 7f6402fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63fc0cd730 con 0x7f63fc0b07b0 2024-01-11T08:10:39.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.204+0000 7f6402fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63fc0ccb20 con 0x7f63fc0b03e0 2024-01-11T08:10:39.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.204+0000 7f6402fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63fc0cce00 con 0x7f63fc0b2b90 2024-01-11T08:10:39.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.204+0000 7f640159c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f63fc0b07b0 0x7f63fc159740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37240/0 (socket says 172.21.15.27:37240) 2024-01-11T08:10:39.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.204+0000 7f640159c700 1 -- 172.21.15.27:0/2327549766 learned_addr learned my addr 172.21.15.27:0/2327549766 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:39.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.204+0000 7f63fbfff700 1 --2- 172.21.15.27:0/2327549766 >> v2:172.21.15.77:3300/0 conn(0x7f63fc0b2b90 0x7f63fc159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:39.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.204+0000 7f63fb7fe700 1 -- 172.21.15.27:0/2327549766 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505982863 0 0) 0x7f63fc0cd730 con 0x7f63fc0b07b0 2024-01-11T08:10:39.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.204+0000 7f63fb7fe700 1 -- 172.21.15.27:0/2327549766 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63ec003570 con 0x7f63fc0b07b0 2024-01-11T08:10:39.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.204+0000 7f63fb7fe700 1 -- 172.21.15.27:0/2327549766 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 168649785 0 0) 0x7f63fc0ccb20 con 0x7f63fc0b03e0 2024-01-11T08:10:39.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.204+0000 7f63fb7fe700 1 -- 172.21.15.27:0/2327549766 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63fc0cd730 con 0x7f63fc0b03e0 2024-01-11T08:10:39.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.204+0000 7f63fbfff700 1 -- 172.21.15.27:0/2327549766 >> v1:172.21.15.77:6789/0 conn(0x7f63fc0b03e0 legacy=0x7f63fc0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:39.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.204+0000 7f63fbfff700 1 -- 172.21.15.27:0/2327549766 >> v1:172.21.15.27:6789/0 conn(0x7f63fc0b07b0 legacy=0x7f63fc159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:39.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.204+0000 7f63fbfff700 1 -- 172.21.15.27:0/2327549766 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63fc0cd0e0 con 0x7f63fc0b2b90 2024-01-11T08:10:39.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.205+0000 7f63fbfff700 1 --2- 172.21.15.27:0/2327549766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc0b2b90 0x7f63fc159200 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f63e80098e0 tx=0x7f63e80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aea9bf5259fb45b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:39.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.205+0000 7f63fb7fe700 1 -- 172.21.15.27:0/2327549766 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8011630 con 0x7f63fc0b2b90 2024-01-11T08:10:39.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.205+0000 7f63fb7fe700 1 -- 172.21.15.27:0/2327549766 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e8011790 con 0x7f63fc0b2b90 2024-01-11T08:10:39.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.205+0000 7f63fb7fe700 1 -- 172.21.15.27:0/2327549766 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8011b30 con 0x7f63fc0b2b90 2024-01-11T08:10:39.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.206+0000 7f6402fff700 1 -- 172.21.15.27:0/2327549766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc0b2b90 msgr2=0x7f63fc159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:39.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.206+0000 7f6402fff700 1 --2- 172.21.15.27:0/2327549766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc0b2b90 0x7f63fc159200 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f63e80098e0 tx=0x7f63e80091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:39.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.206+0000 7f6402fff700 1 -- 172.21.15.27:0/2327549766 shutdown_connections 2024-01-11T08:10:39.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.206+0000 7f6402fff700 1 --2- 172.21.15.27:0/2327549766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc0b2b90 0x7f63fc159200 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:39.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.206+0000 7f6402fff700 1 -- 172.21.15.27:0/2327549766 >> 172.21.15.27:0/2327549766 conn(0x7f63fc06ed40 msgr2=0x7f63fc06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:39.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.206+0000 7f6402fff700 1 -- 172.21.15.27:0/2327549766 shutdown_connections 2024-01-11T08:10:39.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.206+0000 7f6402fff700 1 -- 172.21.15.27:0/2327549766 wait complete. 2024-01-11T08:10:39.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.207+0000 7f6402fff700 1 Processor -- start 2024-01-11T08:10:39.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.207+0000 7f6402fff700 1 -- start start 2024-01-11T08:10:39.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.208+0000 7f6402fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc0b03e0 0x7f63fc0c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:39.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.208+0000 7f6402fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63fc0b07b0 0x7f63fc169b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:39.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.208+0000 7f6402fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63fc0b2b90 0x7f63fc0af3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:39.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.208+0000 7f6402fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc16d4a0 con 0x7f63fc0b2b90 2024-01-11T08:10:39.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.208+0000 7f6402fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc16a3a0 con 0x7f63fc0b03e0 2024-01-11T08:10:39.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.208+0000 7f6402fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc16a680 con 0x7f63fc0b07b0 2024-01-11T08:10:39.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.209+0000 7f640159c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63fc0b2b90 0x7f63fc0af3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:39.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.209+0000 7f6400d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc0b03e0 0x7f63fc0c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:39.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.209+0000 7f6400d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc0b03e0 0x7f63fc0c9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:58670/0 (socket says 172.21.15.27:58670) 2024-01-11T08:10:39.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.209+0000 7f6400d9b700 1 -- 172.21.15.27:0/2759365498 learned_addr learned my addr 172.21.15.27:0/2759365498 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:39.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.209+0000 7f63fbfff700 1 --2- 172.21.15.27:0/2759365498 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63fc0b07b0 0x7f63fc169b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:39.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.209+0000 7f640159c700 1 -- 172.21.15.27:0/2759365498 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63fc0b07b0 msgr2=0x7f63fc169b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:39.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.209+0000 7f640159c700 1 --2- 172.21.15.27:0/2759365498 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63fc0b07b0 0x7f63fc169b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:39.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.209+0000 7f640159c700 1 -- 172.21.15.27:0/2759365498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc0b03e0 msgr2=0x7f63fc0c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:39.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.209+0000 7f640159c700 1 --2- 172.21.15.27:0/2759365498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc0b03e0 0x7f63fc0c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:39.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.209+0000 7f640159c700 1 -- 172.21.15.27:0/2759365498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e8009420 con 0x7f63fc0b2b90 2024-01-11T08:10:39.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.209+0000 7f6400d9b700 1 --2- 172.21.15.27:0/2759365498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc0b03e0 0x7f63fc0c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:39.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.209+0000 7f640159c700 1 --2- 172.21.15.27:0/2759365498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63fc0b2b90 0x7f63fc0af3e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f63f4001460 tx=0x7f63f4001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:39.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.210+0000 7f63f8ff9700 1 -- 172.21.15.27:0/2759365498 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400a840 con 0x7f63fc0b2b90 2024-01-11T08:10:39.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.210+0000 7f63f8ff9700 1 -- 172.21.15.27:0/2759365498 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f400a9a0 con 0x7f63fc0b2b90 2024-01-11T08:10:39.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.210+0000 7f6402fff700 1 -- 172.21.15.27:0/2759365498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63fc16a960 con 0x7f63fc0b2b90 2024-01-11T08:10:39.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.210+0000 7f63f8ff9700 1 -- 172.21.15.27:0/2759365498 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400ad40 con 0x7f63fc0b2b90 2024-01-11T08:10:39.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.210+0000 7f6402fff700 1 -- 172.21.15.27:0/2759365498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63fc16a060 con 0x7f63fc0b2b90 2024-01-11T08:10:39.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.211+0000 7f6402fff700 1 -- 172.21.15.27:0/2759365498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63fc16d610 con 0x7f63fc0b2b90 2024-01-11T08:10:39.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.212+0000 7f63f8ff9700 1 -- 172.21.15.27:0/2759365498 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f63f4007300 con 0x7f63fc0b2b90 2024-01-11T08:10:39.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.213+0000 7f63f8ff9700 1 --2- 172.21.15.27:0/2759365498 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f63e0066f30 0x7f63e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:39.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.213+0000 7f63f8ff9700 1 -- 172.21.15.27:0/2759365498 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f63f4084850 con 0x7f63fc0b2b90 2024-01-11T08:10:39.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.213+0000 7f6400d9b700 1 --2- 172.21.15.27:0/2759365498 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f63e0066f30 0x7f63e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:39.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.214+0000 7f6400d9b700 1 --2- 172.21.15.27:0/2759365498 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f63e0066f30 0x7f63e00693e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f63f0007fd0 tx=0x7f63f0007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:39.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.217+0000 7f63f8ff9700 1 -- 172.21.15.27:0/2759365498 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f63f4051aa0 con 0x7f63fc0b2b90 2024-01-11T08:10:39.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.413+0000 7f6402fff700 1 -- 172.21.15.27:0/2759365498 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f63fc0718d0 con 0x7f63e0066f30 2024-01-11T08:10:39.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.421+0000 7f63f8ff9700 1 -- 172.21.15.27:0/2759365498 <== mgr.4099 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264566 (secure 0 0 0) 0x7f63fc0718d0 con 0x7f63e0066f30 2024-01-11T08:10:39.426 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:39.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.425+0000 7f6402fff700 1 -- 172.21.15.27:0/2759365498 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f63e0066f30 msgr2=0x7f63e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:39.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.425+0000 7f6402fff700 1 --2- 172.21.15.27:0/2759365498 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f63e0066f30 0x7f63e00693e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f63f0007fd0 tx=0x7f63f0007e10 comp rx=0 tx=0).stop 2024-01-11T08:10:39.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.425+0000 7f6402fff700 1 -- 172.21.15.27:0/2759365498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63fc0b2b90 msgr2=0x7f63fc0af3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:39.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.425+0000 7f6402fff700 1 --2- 172.21.15.27:0/2759365498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63fc0b2b90 0x7f63fc0af3e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f63f4001460 tx=0x7f63f4001a40 comp rx=0 tx=0).stop 2024-01-11T08:10:39.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.425+0000 7f6402fff700 1 -- 172.21.15.27:0/2759365498 shutdown_connections 2024-01-11T08:10:39.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.425+0000 7f6402fff700 1 --2- 172.21.15.27:0/2759365498 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f63e0066f30 0x7f63e00693e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:39.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.425+0000 7f6402fff700 1 --2- 172.21.15.27:0/2759365498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63fc0b2b90 0x7f63fc0af3e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:39.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.425+0000 7f6402fff700 1 --2- 172.21.15.27:0/2759365498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc0b03e0 0x7f63fc0c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:39.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.425+0000 7f6402fff700 1 --2- 172.21.15.27:0/2759365498 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63fc0b07b0 0x7f63fc169b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:39.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.425+0000 7f6402fff700 1 -- 172.21.15.27:0/2759365498 >> 172.21.15.27:0/2759365498 conn(0x7f63fc06ed40 msgr2=0x7f63fc0b47c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:39.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.426+0000 7f6402fff700 1 -- 172.21.15.27:0/2759365498 shutdown_connections 2024-01-11T08:10:39.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.426+0000 7f6402fff700 1 -- 172.21.15.27:0/2759365498 wait complete. 2024-01-11T08:10:39.429 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:10:39.444 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-01-11T08:10:37.510267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381224,"kb_used_data":2920,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752593496,"statfs":{"total":773094113280,"available":770655739904,"internally_reserved":0,"allocated":2990080,"data_stored":1787852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.863498"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.498429+0000","last_change":"2024-01-11T08:10:30.495770+0000","last_active":"2024-01-11T08:10:30.498429+0000","last_peered":"2024-01-11T08:10:30.498429+0000","last_clean":"2024-01-11T08:10:30.498429+0000","last_became_active":"2024-01-11T08:10:30.495355+0000","last_became_peered":"2024-01-11T08:10:30.495355+0000","last_unstale":"2024-01-11T08:10:30.498429+0000","last_undegraded":"2024-01-11T08:10:30.498429+0000","last_fullsized":"2024-01-11T08:10:30.498429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:02.887025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.493541+0000","last_change":"2024-01-11T08:10:30.492484+0000","last_active":"2024-01-11T08:10:30.493541+0000","last_peered":"2024-01-11T08:10:30.493541+0000","last_clean":"2024-01-11T08:10:30.493541+0000","last_became_active":"2024-01-11T08:10:30.491796+0000","last_became_peered":"2024-01-11T08:10:30.491796+0000","last_unstale":"2024-01-11T08:10:30.493541+0000","last_undegraded":"2024-01-11T08:10:30.493541+0000","last_fullsized":"2024-01-11T08:10:30.493541+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:21.127779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.493213+0000","last_change":"2024-01-11T08:10:30.489955+0000","last_active":"2024-01-11T08:10:30.493213+0000","last_peered":"2024-01-11T08:10:30.493213+0000","last_clean":"2024-01-11T08:10:30.493213+0000","last_became_active":"2024-01-11T08:10:30.489649+0000","last_became_peered":"2024-01-11T08:10:30.489649+0000","last_unstale":"2024-01-11T08:10:30.493213+0000","last_undegraded":"2024-01-11T08:10:30.493213+0000","last_fullsized":"2024-01-11T08:10:30.493213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:08:10.140431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.495948+0000","last_change":"2024-01-11T08:10:30.489768+0000","last_active":"2024-01-11T08:10:30.495948+0000","last_peered":"2024-01-11T08:10:30.495948+0000","last_clean":"2024-01-11T08:10:30.495948+0000","last_became_active":"2024-01-11T08:10:30.489392+0000","last_became_peered":"2024-01-11T08:10:30.489392+0000","last_unstale":"2024-01-11T08:10:30.495948+0000","last_undegraded":"2024-01-11T08:10:30.495948+0000","last_fullsized":"2024-01-11T08:10:30.495948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:08:10.140431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.494482+0000","last_change":"2024-01-11T08:10:30.490346+0000","last_active":"2024-01-11T08:10:30.494482+0000","last_peered":"2024-01-11T08:10:30.494482+0000","last_clean":"2024-01-11T08:10:30.494482+0000","last_became_active":"2024-01-11T08:10:30.490042+0000","last_became_peered":"2024-01-11T08:10:30.490042+0000","last_unstale":"2024-01-11T08:10:30.494482+0000","last_undegraded":"2024-01-11T08:10:30.494482+0000","last_fullsized":"2024-01-11T08:10:30.494482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:57:19.989587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.492082+0000","last_change":"2024-01-11T08:10:30.487385+0000","last_active":"2024-01-11T08:10:30.492082+0000","last_peered":"2024-01-11T08:10:30.492082+0000","last_clean":"2024-01-11T08:10:30.492082+0000","last_became_active":"2024-01-11T08:10:30.487027+0000","last_became_peered":"2024-01-11T08:10:30.487027+0000","last_unstale":"2024-01-11T08:10:30.492082+0000","last_undegraded":"2024-01-11T08:10:30.492082+0000","last_fullsized":"2024-01-11T08:10:30.492082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:12.649092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.498136+0000","last_change":"2024-01-11T08:10:30.494485+0000","last_active":"2024-01-11T08:10:30.498136+0000","last_peered":"2024-01-11T08:10:30.498136+0000","last_clean":"2024-01-11T08:10:30.498136+0000","last_became_active":"2024-01-11T08:10:30.494227+0000","last_became_peered":"2024-01-11T08:10:30.494227+0000","last_unstale":"2024-01-11T08:10:30.498136+0000","last_undegraded":"2024-01-11T08:10:30.498136+0000","last_fullsized":"2024-01-11T08:10:30.498136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:31:36.455594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.484735+0000","last_change":"2024-01-11T08:10:30.477637+0000","last_active":"2024-01-11T08:10:30.484735+0000","last_peered":"2024-01-11T08:10:30.484735+0000","last_clean":"2024-01-11T08:10:30.484735+0000","last_became_active":"2024-01-11T08:10:30.477267+0000","last_became_peered":"2024-01-11T08:10:30.477267+0000","last_unstale":"2024-01-11T08:10:30.484735+0000","last_undegraded":"2024-01-11T08:10:30.484735+0000","last_fullsized":"2024-01-11T08:10:30.484735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:47.753678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.496708+0000","last_change":"2024-01-11T08:10:30.494031+0000","last_active":"2024-01-11T08:10:30.496708+0000","last_peered":"2024-01-11T08:10:30.496708+0000","last_clean":"2024-01-11T08:10:30.496708+0000","last_became_active":"2024-01-11T08:10:30.493769+0000","last_became_peered":"2024-01-11T08:10:30.493769+0000","last_unstale":"2024-01-11T08:10:30.496708+0000","last_undegraded":"2024-01-11T08:10:30.496708+0000","last_fullsized":"2024-01-11T08:10:30.496708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:14:36.595264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.502209+0000","last_change":"2024-01-11T08:10:30.498360+0000","last_active":"2024-01-11T08:10:30.502209+0000","last_peered":"2024-01-11T08:10:30.502209+0000","last_clean":"2024-01-11T08:10:30.502209+0000","last_became_active":"2024-01-11T08:10:30.498064+0000","last_became_peered":"2024-01-11T08:10:30.498064+0000","last_unstale":"2024-01-11T08:10:30.502209+0000","last_undegraded":"2024-01-11T08:10:30.502209+0000","last_fullsized":"2024-01-11T08:10:30.502209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:08:10.140431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.501156+0000","last_change":"2024-01-11T08:10:30.493099+0000","last_active":"2024-01-11T08:10:30.501156+0000","last_peered":"2024-01-11T08:10:30.501156+0000","last_clean":"2024-01-11T08:10:30.501156+0000","last_became_active":"2024-01-11T08:10:30.492806+0000","last_became_peered":"2024-01-11T08:10:30.492806+0000","last_unstale":"2024-01-11T08:10:30.501156+0000","last_undegraded":"2024-01-11T08:10:30.501156+0000","last_fullsized":"2024-01-11T08:10:30.501156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:21.127779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.503398+0000","last_change":"2024-01-11T08:10:30.499317+0000","last_active":"2024-01-11T08:10:30.503398+0000","last_peered":"2024-01-11T08:10:30.503398+0000","last_clean":"2024-01-11T08:10:30.503398+0000","last_became_active":"2024-01-11T08:10:30.497953+0000","last_became_peered":"2024-01-11T08:10:30.497953+0000","last_unstale":"2024-01-11T08:10:30.503398+0000","last_undegraded":"2024-01-11T08:10:30.503398+0000","last_fullsized":"2024-01-11T08:10:30.503398+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:27:24.758750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.490479+0000","last_change":"2024-01-11T08:10:30.488701+0000","last_active":"2024-01-11T08:10:30.490479+0000","last_peered":"2024-01-11T08:10:30.490479+0000","last_clean":"2024-01-11T08:10:30.490479+0000","last_became_active":"2024-01-11T08:10:30.488244+0000","last_became_peered":"2024-01-11T08:10:30.488244+0000","last_unstale":"2024-01-11T08:10:30.490479+0000","last_undegraded":"2024-01-11T08:10:30.490479+0000","last_fullsized":"2024-01-11T08:10:30.490479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:21.127779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.491292+0000","last_change":"2024-01-11T08:10:30.486156+0000","last_active":"2024-01-11T08:10:30.491292+0000","last_peered":"2024-01-11T08:10:30.491292+0000","last_clean":"2024-01-11T08:10:30.491292+0000","last_became_active":"2024-01-11T08:10:30.485871+0000","last_became_peered":"2024-01-11T08:10:30.485871+0000","last_unstale":"2024-01-11T08:10:30.491292+0000","last_undegraded":"2024-01-11T08:10:30.491292+0000","last_fullsized":"2024-01-11T08:10:30.491292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:22:43.961595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.487083+0000","last_change":"2024-01-11T08:10:30.484060+0000","last_active":"2024-01-11T08:10:30.487083+0000","last_peered":"2024-01-11T08:10:30.487083+0000","last_clean":"2024-01-11T08:10:30.487083+0000","last_became_active":"2024-01-11T08:10:30.483788+0000","last_became_peered":"2024-01-11T08:10:30.483788+0000","last_unstale":"2024-01-11T08:10:30.487083+0000","last_undegraded":"2024-01-11T08:10:30.487083+0000","last_fullsized":"2024-01-11T08:10:30.487083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:08:10.140431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.494360+0000","last_change":"2024-01-11T08:10:30.487817+0000","last_active":"2024-01-11T08:10:30.494360+0000","last_peered":"2024-01-11T08:10:30.494360+0000","last_clean":"2024-01-11T08:10:30.494360+0000","last_became_active":"2024-01-11T08:10:30.486728+0000","last_became_peered":"2024-01-11T08:10:30.486728+0000","last_unstale":"2024-01-11T08:10:30.494360+0000","last_undegraded":"2024-01-11T08:10:30.494360+0000","last_fullsized":"2024-01-11T08:10:30.494360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:33:25.110688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.494728+0000","last_change":"2024-01-11T08:10:30.490019+0000","last_active":"2024-01-11T08:10:30.494728+0000","last_peered":"2024-01-11T08:10:30.494728+0000","last_clean":"2024-01-11T08:10:30.494728+0000","last_became_active":"2024-01-11T08:10:30.489527+0000","last_became_peered":"2024-01-11T08:10:30.489527+0000","last_unstale":"2024-01-11T08:10:30.494728+0000","last_undegraded":"2024-01-11T08:10:30.494728+0000","last_fullsized":"2024-01-11T08:10:30.494728+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:36:17.244182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.489961+0000","last_change":"2024-01-11T08:10:30.488678+0000","last_active":"2024-01-11T08:10:30.489961+0000","last_peered":"2024-01-11T08:10:30.489961+0000","last_clean":"2024-01-11T08:10:30.489961+0000","last_became_active":"2024-01-11T08:10:30.487511+0000","last_became_peered":"2024-01-11T08:10:30.487511+0000","last_unstale":"2024-01-11T08:10:30.489961+0000","last_undegraded":"2024-01-11T08:10:30.489961+0000","last_fullsized":"2024-01-11T08:10:30.489961+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:56:13.553335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.494305+0000","last_change":"2024-01-11T08:10:30.489544+0000","last_active":"2024-01-11T08:10:30.494305+0000","last_peered":"2024-01-11T08:10:30.494305+0000","last_clean":"2024-01-11T08:10:30.494305+0000","last_became_active":"2024-01-11T08:10:30.489020+0000","last_became_peered":"2024-01-11T08:10:30.489020+0000","last_unstale":"2024-01-11T08:10:30.494305+0000","last_undegraded":"2024-01-11T08:10:30.494305+0000","last_fullsized":"2024-01-11T08:10:30.494305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:33:25.110688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.680471+0000","last_change":"2024-01-11T08:10:29.680721+0000","last_active":"2024-01-11T08:10:30.680471+0000","last_peered":"2024-01-11T08:10:30.680471+0000","last_clean":"2024-01-11T08:10:30.680471+0000","last_became_active":"2024-01-11T08:10:29.680457+0000","last_became_peered":"2024-01-11T08:10:29.680457+0000","last_unstale":"2024-01-11T08:10:30.680471+0000","last_undegraded":"2024-01-11T08:10:30.680471+0000","last_fullsized":"2024-01-11T08:10:30.680471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:54.150785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.496156+0000","last_change":"2024-01-11T08:10:30.494749+0000","last_active":"2024-01-11T08:10:30.496156+0000","last_peered":"2024-01-11T08:10:30.496156+0000","last_clean":"2024-01-11T08:10:30.496156+0000","last_became_active":"2024-01-11T08:10:30.494471+0000","last_became_peered":"2024-01-11T08:10:30.494471+0000","last_unstale":"2024-01-11T08:10:30.496156+0000","last_undegraded":"2024-01-11T08:10:30.496156+0000","last_fullsized":"2024-01-11T08:10:30.496156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:50:04.791490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.684107+0000","last_change":"2024-01-11T08:10:29.665411+0000","last_active":"2024-01-11T08:10:30.684107+0000","last_peered":"2024-01-11T08:10:30.684107+0000","last_clean":"2024-01-11T08:10:30.684107+0000","last_became_active":"2024-01-11T08:10:29.665100+0000","last_became_peered":"2024-01-11T08:10:29.665100+0000","last_unstale":"2024-01-11T08:10:30.684107+0000","last_undegraded":"2024-01-11T08:10:30.684107+0000","last_fullsized":"2024-01-11T08:10:30.684107+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:33.696922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.493355+0000","last_change":"2024-01-11T08:10:30.490972+0000","last_active":"2024-01-11T08:10:30.493355+0000","last_peered":"2024-01-11T08:10:30.493355+0000","last_clean":"2024-01-11T08:10:30.493355+0000","last_became_active":"2024-01-11T08:10:30.490679+0000","last_became_peered":"2024-01-11T08:10:30.490679+0000","last_unstale":"2024-01-11T08:10:30.493355+0000","last_undegraded":"2024-01-11T08:10:30.493355+0000","last_fullsized":"2024-01-11T08:10:30.493355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:31:43.759712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.683204+0000","last_change":"2024-01-11T08:10:29.682048+0000","last_active":"2024-01-11T08:10:30.683204+0000","last_peered":"2024-01-11T08:10:30.683204+0000","last_clean":"2024-01-11T08:10:30.683204+0000","last_became_active":"2024-01-11T08:10:29.678360+0000","last_became_peered":"2024-01-11T08:10:29.678360+0000","last_unstale":"2024-01-11T08:10:30.683204+0000","last_undegraded":"2024-01-11T08:10:30.683204+0000","last_fullsized":"2024-01-11T08:10:30.683204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:56:12.550627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.481573+0000","last_change":"2024-01-11T08:10:30.477253+0000","last_active":"2024-01-11T08:10:30.481573+0000","last_peered":"2024-01-11T08:10:30.481573+0000","last_clean":"2024-01-11T08:10:30.481573+0000","last_became_active":"2024-01-11T08:10:30.476942+0000","last_became_peered":"2024-01-11T08:10:30.476942+0000","last_unstale":"2024-01-11T08:10:30.481573+0000","last_undegraded":"2024-01-11T08:10:30.481573+0000","last_fullsized":"2024-01-11T08:10:30.481573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:47:17.705767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.680175+0000","last_change":"2024-01-11T08:10:29.682657+0000","last_active":"2024-01-11T08:10:30.680175+0000","last_peered":"2024-01-11T08:10:30.680175+0000","last_clean":"2024-01-11T08:10:30.680175+0000","last_became_active":"2024-01-11T08:10:29.680246+0000","last_became_peered":"2024-01-11T08:10:29.680246+0000","last_unstale":"2024-01-11T08:10:30.680175+0000","last_undegraded":"2024-01-11T08:10:30.680175+0000","last_fullsized":"2024-01-11T08:10:30.680175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:48.964847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.480068+0000","last_change":"2024-01-11T08:10:30.476899+0000","last_active":"2024-01-11T08:10:30.480068+0000","last_peered":"2024-01-11T08:10:30.480068+0000","last_clean":"2024-01-11T08:10:30.480068+0000","last_became_active":"2024-01-11T08:10:30.475555+0000","last_became_peered":"2024-01-11T08:10:30.475555+0000","last_unstale":"2024-01-11T08:10:30.480068+0000","last_undegraded":"2024-01-11T08:10:30.480068+0000","last_fullsized":"2024-01-11T08:10:30.480068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:51:50.621267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.683162+0000","last_change":"2024-01-11T08:10:29.670617+0000","last_active":"2024-01-11T08:10:30.683162+0000","last_peered":"2024-01-11T08:10:30.683162+0000","last_clean":"2024-01-11T08:10:30.683162+0000","last_became_active":"2024-01-11T08:10:29.670316+0000","last_became_peered":"2024-01-11T08:10:29.670316+0000","last_unstale":"2024-01-11T08:10:30.683162+0000","last_undegraded":"2024-01-11T08:10:30.683162+0000","last_fullsized":"2024-01-11T08:10:30.683162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:50:14.474575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.486483+0000","last_change":"2024-01-11T08:10:30.477966+0000","last_active":"2024-01-11T08:10:30.486483+0000","last_peered":"2024-01-11T08:10:30.486483+0000","last_clean":"2024-01-11T08:10:30.486483+0000","last_became_active":"2024-01-11T08:10:30.477699+0000","last_became_peered":"2024-01-11T08:10:30.477699+0000","last_unstale":"2024-01-11T08:10:30.486483+0000","last_undegraded":"2024-01-11T08:10:30.486483+0000","last_fullsized":"2024-01-11T08:10:30.486483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:06.679100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.680666+0000","last_change":"2024-01-11T08:10:29.669295+0000","last_active":"2024-01-11T08:10:30.680666+0000","last_peered":"2024-01-11T08:10:30.680666+0000","last_clean":"2024-01-11T08:10:30.680666+0000","last_became_active":"2024-01-11T08:10:29.668947+0000","last_became_peered":"2024-01-11T08:10:29.668947+0000","last_unstale":"2024-01-11T08:10:30.680666+0000","last_undegraded":"2024-01-11T08:10:30.680666+0000","last_fullsized":"2024-01-11T08:10:30.680666+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:50:14.474575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.496590+0000","last_change":"2024-01-11T08:10:30.494173+0000","last_active":"2024-01-11T08:10:30.496590+0000","last_peered":"2024-01-11T08:10:30.496590+0000","last_clean":"2024-01-11T08:10:30.496590+0000","last_became_active":"2024-01-11T08:10:30.493875+0000","last_became_peered":"2024-01-11T08:10:30.493875+0000","last_unstale":"2024-01-11T08:10:30.496590+0000","last_undegraded":"2024-01-11T08:10:30.496590+0000","last_fullsized":"2024-01-11T08:10:30.496590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:47.753678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.466964+0000","last_change":"2024-01-11T08:10:29.673258+0000","last_active":"2024-01-11T08:10:30.466964+0000","last_peered":"2024-01-11T08:10:30.466964+0000","last_clean":"2024-01-11T08:10:30.466964+0000","last_became_active":"2024-01-11T08:10:29.672939+0000","last_became_peered":"2024-01-11T08:10:29.672939+0000","last_unstale":"2024-01-11T08:10:30.466964+0000","last_undegraded":"2024-01-11T08:10:30.466964+0000","last_fullsized":"2024-01-11T08:10:30.466964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:21.132838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.487736+0000","last_change":"2024-01-11T08:10:30.486579+0000","last_active":"2024-01-11T08:10:30.487736+0000","last_peered":"2024-01-11T08:10:30.487736+0000","last_clean":"2024-01-11T08:10:30.487736+0000","last_became_active":"2024-01-11T08:10:30.486332+0000","last_became_peered":"2024-01-11T08:10:30.486332+0000","last_unstale":"2024-01-11T08:10:30.487736+0000","last_undegraded":"2024-01-11T08:10:30.487736+0000","last_fullsized":"2024-01-11T08:10:30.487736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:03:43.208019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.471082+0000","last_change":"2024-01-11T08:10:29.676952+0000","last_active":"2024-01-11T08:10:30.471082+0000","last_peered":"2024-01-11T08:10:30.471082+0000","last_clean":"2024-01-11T08:10:30.471082+0000","last_became_active":"2024-01-11T08:10:29.676624+0000","last_became_peered":"2024-01-11T08:10:29.676624+0000","last_unstale":"2024-01-11T08:10:30.471082+0000","last_undegraded":"2024-01-11T08:10:30.471082+0000","last_fullsized":"2024-01-11T08:10:30.471082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:05.676391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.487063+0000","last_change":"2024-01-11T08:10:30.481939+0000","last_active":"2024-01-11T08:10:30.487063+0000","last_peered":"2024-01-11T08:10:30.487063+0000","last_clean":"2024-01-11T08:10:30.487063+0000","last_became_active":"2024-01-11T08:10:30.481619+0000","last_became_peered":"2024-01-11T08:10:30.481619+0000","last_unstale":"2024-01-11T08:10:30.487063+0000","last_undegraded":"2024-01-11T08:10:30.487063+0000","last_fullsized":"2024-01-11T08:10:30.487063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:03:43.208019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.485859+0000","last_change":"2024-01-11T08:10:29.679415+0000","last_active":"2024-01-11T08:10:30.485859+0000","last_peered":"2024-01-11T08:10:30.485859+0000","last_clean":"2024-01-11T08:10:30.485859+0000","last_became_active":"2024-01-11T08:10:29.679087+0000","last_became_peered":"2024-01-11T08:10:29.679087+0000","last_unstale":"2024-01-11T08:10:30.485859+0000","last_undegraded":"2024-01-11T08:10:30.485859+0000","last_fullsized":"2024-01-11T08:10:30.485859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:05.676391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.497485+0000","last_change":"2024-01-11T08:10:30.495369+0000","last_active":"2024-01-11T08:10:30.497485+0000","last_peered":"2024-01-11T08:10:30.497485+0000","last_clean":"2024-01-11T08:10:30.497485+0000","last_became_active":"2024-01-11T08:10:30.495037+0000","last_became_peered":"2024-01-11T08:10:30.495037+0000","last_unstale":"2024-01-11T08:10:30.497485+0000","last_undegraded":"2024-01-11T08:10:30.497485+0000","last_fullsized":"2024-01-11T08:10:30.497485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:21:29.220934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.472907+0000","last_change":"2024-01-11T08:10:29.681233+0000","last_active":"2024-01-11T08:10:30.472907+0000","last_peered":"2024-01-11T08:10:30.472907+0000","last_clean":"2024-01-11T08:10:30.472907+0000","last_became_active":"2024-01-11T08:10:29.680606+0000","last_became_peered":"2024-01-11T08:10:29.680606+0000","last_unstale":"2024-01-11T08:10:30.472907+0000","last_undegraded":"2024-01-11T08:10:30.472907+0000","last_fullsized":"2024-01-11T08:10:30.472907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:12:48.941888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.491516+0000","last_change":"2024-01-11T08:10:30.487651+0000","last_active":"2024-01-11T08:10:30.491516+0000","last_peered":"2024-01-11T08:10:30.491516+0000","last_clean":"2024-01-11T08:10:30.491516+0000","last_became_active":"2024-01-11T08:10:30.486641+0000","last_became_peered":"2024-01-11T08:10:30.486641+0000","last_unstale":"2024-01-11T08:10:30.491516+0000","last_undegraded":"2024-01-11T08:10:30.491516+0000","last_fullsized":"2024-01-11T08:10:30.491516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:01:02.902411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.678617+0000","last_change":"2024-01-11T08:10:29.666353+0000","last_active":"2024-01-11T08:10:30.678617+0000","last_peered":"2024-01-11T08:10:30.678617+0000","last_clean":"2024-01-11T08:10:30.678617+0000","last_became_active":"2024-01-11T08:10:29.665995+0000","last_became_peered":"2024-01-11T08:10:29.665995+0000","last_unstale":"2024-01-11T08:10:30.678617+0000","last_undegraded":"2024-01-11T08:10:30.678617+0000","last_fullsized":"2024-01-11T08:10:30.678617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:12:48.941888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.503379+0000","last_change":"2024-01-11T08:10:30.499297+0000","last_active":"2024-01-11T08:10:30.503379+0000","last_peered":"2024-01-11T08:10:30.503379+0000","last_clean":"2024-01-11T08:10:30.503379+0000","last_became_active":"2024-01-11T08:10:30.497567+0000","last_became_peered":"2024-01-11T08:10:30.497567+0000","last_unstale":"2024-01-11T08:10:30.503379+0000","last_undegraded":"2024-01-11T08:10:30.503379+0000","last_fullsized":"2024-01-11T08:10:30.503379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:49.967555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.685960+0000","last_change":"2024-01-11T08:10:29.671697+0000","last_active":"2024-01-11T08:10:30.685960+0000","last_peered":"2024-01-11T08:10:30.685960+0000","last_clean":"2024-01-11T08:10:30.685960+0000","last_became_active":"2024-01-11T08:10:29.671395+0000","last_became_peered":"2024-01-11T08:10:29.671395+0000","last_unstale":"2024-01-11T08:10:30.685960+0000","last_undegraded":"2024-01-11T08:10:30.685960+0000","last_fullsized":"2024-01-11T08:10:30.685960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:12:48.941888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.494324+0000","last_change":"2024-01-11T08:10:30.489779+0000","last_active":"2024-01-11T08:10:30.494324+0000","last_peered":"2024-01-11T08:10:30.494324+0000","last_clean":"2024-01-11T08:10:30.494324+0000","last_became_active":"2024-01-11T08:10:30.489290+0000","last_became_peered":"2024-01-11T08:10:30.489290+0000","last_unstale":"2024-01-11T08:10:30.494324+0000","last_undegraded":"2024-01-11T08:10:30.494324+0000","last_fullsized":"2024-01-11T08:10:30.494324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:50:04.791490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.678500+0000","last_change":"2024-01-11T08:10:29.682961+0000","last_active":"2024-01-11T08:10:30.678500+0000","last_peered":"2024-01-11T08:10:30.678500+0000","last_clean":"2024-01-11T08:10:30.678500+0000","last_became_active":"2024-01-11T08:10:29.682646+0000","last_became_peered":"2024-01-11T08:10:29.682646+0000","last_unstale":"2024-01-11T08:10:30.678500+0000","last_undegraded":"2024-01-11T08:10:30.678500+0000","last_fullsized":"2024-01-11T08:10:30.678500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:11:42.997404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.496569+0000","last_change":"2024-01-11T08:10:30.493989+0000","last_active":"2024-01-11T08:10:30.496569+0000","last_peered":"2024-01-11T08:10:30.496569+0000","last_clean":"2024-01-11T08:10:30.496569+0000","last_became_active":"2024-01-11T08:10:30.493678+0000","last_became_peered":"2024-01-11T08:10:30.493678+0000","last_unstale":"2024-01-11T08:10:30.496569+0000","last_undegraded":"2024-01-11T08:10:30.496569+0000","last_fullsized":"2024-01-11T08:10:30.496569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:03:43.208019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.471218+0000","last_change":"2024-01-11T08:10:29.670911+0000","last_active":"2024-01-11T08:10:30.471218+0000","last_peered":"2024-01-11T08:10:30.471218+0000","last_clean":"2024-01-11T08:10:30.471218+0000","last_became_active":"2024-01-11T08:10:29.670615+0000","last_became_peered":"2024-01-11T08:10:29.670615+0000","last_unstale":"2024-01-11T08:10:30.471218+0000","last_undegraded":"2024-01-11T08:10:30.471218+0000","last_fullsized":"2024-01-11T08:10:30.471218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:46:57.523656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.500438+0000","last_change":"2024-01-11T08:10:30.497306+0000","last_active":"2024-01-11T08:10:30.500438+0000","last_peered":"2024-01-11T08:10:30.500438+0000","last_clean":"2024-01-11T08:10:30.500438+0000","last_became_active":"2024-01-11T08:10:30.496966+0000","last_became_peered":"2024-01-11T08:10:30.496966+0000","last_unstale":"2024-01-11T08:10:30.500438+0000","last_undegraded":"2024-01-11T08:10:30.500438+0000","last_fullsized":"2024-01-11T08:10:30.500438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:47:17.705767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.483926+0000","last_change":"2024-01-11T08:10:29.680426+0000","last_active":"2024-01-11T08:10:30.483926+0000","last_peered":"2024-01-11T08:10:30.483926+0000","last_clean":"2024-01-11T08:10:30.483926+0000","last_became_active":"2024-01-11T08:10:29.680133+0000","last_became_peered":"2024-01-11T08:10:29.680133+0000","last_unstale":"2024-01-11T08:10:30.483926+0000","last_undegraded":"2024-01-11T08:10:30.483926+0000","last_fullsized":"2024-01-11T08:10:30.483926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:21.132838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.500317+0000","last_change":"2024-01-11T08:10:30.494824+0000","last_active":"2024-01-11T08:10:30.500317+0000","last_peered":"2024-01-11T08:10:30.500317+0000","last_clean":"2024-01-11T08:10:30.500317+0000","last_became_active":"2024-01-11T08:10:30.494564+0000","last_became_peered":"2024-01-11T08:10:30.494564+0000","last_unstale":"2024-01-11T08:10:30.500317+0000","last_undegraded":"2024-01-11T08:10:30.500317+0000","last_fullsized":"2024-01-11T08:10:30.500317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:37:25.109490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.687461+0000","last_change":"2024-01-11T08:10:29.671294+0000","last_active":"2024-01-11T08:10:30.687461+0000","last_peered":"2024-01-11T08:10:30.687461+0000","last_clean":"2024-01-11T08:10:30.687461+0000","last_became_active":"2024-01-11T08:10:29.670997+0000","last_became_peered":"2024-01-11T08:10:29.670997+0000","last_unstale":"2024-01-11T08:10:30.687461+0000","last_undegraded":"2024-01-11T08:10:30.687461+0000","last_fullsized":"2024-01-11T08:10:30.687461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:46:57.523656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.502943+0000","last_change":"2024-01-11T08:10:30.498224+0000","last_active":"2024-01-11T08:10:30.502943+0000","last_peered":"2024-01-11T08:10:30.502943+0000","last_clean":"2024-01-11T08:10:30.502943+0000","last_became_active":"2024-01-11T08:10:30.497993+0000","last_became_peered":"2024-01-11T08:10:30.497993+0000","last_unstale":"2024-01-11T08:10:30.502943+0000","last_undegraded":"2024-01-11T08:10:30.502943+0000","last_fullsized":"2024-01-11T08:10:30.502943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:51:50.621267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.676301+0000","last_change":"2024-01-11T08:10:29.678760+0000","last_active":"2024-01-11T08:10:30.676301+0000","last_peered":"2024-01-11T08:10:30.676301+0000","last_clean":"2024-01-11T08:10:30.676301+0000","last_became_active":"2024-01-11T08:10:29.678519+0000","last_became_peered":"2024-01-11T08:10:29.678519+0000","last_unstale":"2024-01-11T08:10:30.676301+0000","last_undegraded":"2024-01-11T08:10:30.676301+0000","last_fullsized":"2024-01-11T08:10:30.676301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:11:42.997404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.505207+0000","last_change":"2024-01-11T08:10:30.504188+0000","last_active":"2024-01-11T08:10:30.505207+0000","last_peered":"2024-01-11T08:10:30.505207+0000","last_clean":"2024-01-11T08:10:30.505207+0000","last_became_active":"2024-01-11T08:10:30.503805+0000","last_became_peered":"2024-01-11T08:10:30.503805+0000","last_unstale":"2024-01-11T08:10:30.505207+0000","last_undegraded":"2024-01-11T08:10:30.505207+0000","last_fullsized":"2024-01-11T08:10:30.505207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:36:17.244182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.473300+0000","last_change":"2024-01-11T08:10:29.671224+0000","last_active":"2024-01-11T08:10:30.473300+0000","last_peered":"2024-01-11T08:10:30.473300+0000","last_clean":"2024-01-11T08:10:30.473300+0000","last_became_active":"2024-01-11T08:10:29.670078+0000","last_became_peered":"2024-01-11T08:10:29.670078+0000","last_unstale":"2024-01-11T08:10:30.473300+0000","last_undegraded":"2024-01-11T08:10:30.473300+0000","last_fullsized":"2024-01-11T08:10:30.473300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:50:45.947278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.486167+0000","last_change":"2024-01-11T08:10:30.479486+0000","last_active":"2024-01-11T08:10:30.486167+0000","last_peered":"2024-01-11T08:10:30.486167+0000","last_clean":"2024-01-11T08:10:30.486167+0000","last_became_active":"2024-01-11T08:10:30.477788+0000","last_became_peered":"2024-01-11T08:10:30.477788+0000","last_unstale":"2024-01-11T08:10:30.486167+0000","last_undegraded":"2024-01-11T08:10:30.486167+0000","last_fullsized":"2024-01-11T08:10:30.486167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:50:04.791490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.471183+0000","last_change":"2024-01-11T08:10:29.670241+0000","last_active":"2024-01-11T08:10:30.471183+0000","last_peered":"2024-01-11T08:10:30.471183+0000","last_clean":"2024-01-11T08:10:30.471183+0000","last_became_active":"2024-01-11T08:10:29.669919+0000","last_became_peered":"2024-01-11T08:10:29.669919+0000","last_unstale":"2024-01-11T08:10:30.471183+0000","last_undegraded":"2024-01-11T08:10:30.471183+0000","last_fullsized":"2024-01-11T08:10:30.471183+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:50:14.474575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.498632+0000","last_change":"2024-01-11T08:10:30.490994+0000","last_active":"2024-01-11T08:10:30.498632+0000","last_peered":"2024-01-11T08:10:30.498632+0000","last_clean":"2024-01-11T08:10:30.498632+0000","last_became_active":"2024-01-11T08:10:30.490710+0000","last_became_peered":"2024-01-11T08:10:30.490710+0000","last_unstale":"2024-01-11T08:10:30.498632+0000","last_undegraded":"2024-01-11T08:10:30.498632+0000","last_fullsized":"2024-01-11T08:10:30.498632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:56:13.553335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.486574+0000","last_change":"2024-01-11T08:10:30.483950+0000","last_active":"2024-01-11T08:10:30.486574+0000","last_peered":"2024-01-11T08:10:30.486574+0000","last_clean":"2024-01-11T08:10:30.486574+0000","last_became_active":"2024-01-11T08:10:30.483643+0000","last_became_peered":"2024-01-11T08:10:30.483643+0000","last_unstale":"2024-01-11T08:10:30.486574+0000","last_undegraded":"2024-01-11T08:10:30.486574+0000","last_fullsized":"2024-01-11T08:10:30.486574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:47.753678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.472149+0000","last_change":"2024-01-11T08:10:29.667798+0000","last_active":"2024-01-11T08:10:30.472149+0000","last_peered":"2024-01-11T08:10:30.472149+0000","last_clean":"2024-01-11T08:10:30.472149+0000","last_became_active":"2024-01-11T08:10:29.667455+0000","last_became_peered":"2024-01-11T08:10:29.667455+0000","last_unstale":"2024-01-11T08:10:30.472149+0000","last_undegraded":"2024-01-11T08:10:30.472149+0000","last_fullsized":"2024-01-11T08:10:30.472149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:33.696922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.480195+0000","last_change":"2024-01-11T08:10:30.477204+0000","last_active":"2024-01-11T08:10:30.480195+0000","last_peered":"2024-01-11T08:10:30.480195+0000","last_clean":"2024-01-11T08:10:30.480195+0000","last_became_active":"2024-01-11T08:10:30.476896+0000","last_became_peered":"2024-01-11T08:10:30.476896+0000","last_unstale":"2024-01-11T08:10:30.480195+0000","last_undegraded":"2024-01-11T08:10:30.480195+0000","last_fullsized":"2024-01-11T08:10:30.480195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:22:43.961595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.472646+0000","last_change":"2024-01-11T08:10:29.666032+0000","last_active":"2024-01-11T08:10:30.472646+0000","last_peered":"2024-01-11T08:10:30.472646+0000","last_clean":"2024-01-11T08:10:30.472646+0000","last_became_active":"2024-01-11T08:10:29.665708+0000","last_became_peered":"2024-01-11T08:10:29.665708+0000","last_unstale":"2024-01-11T08:10:30.472646+0000","last_undegraded":"2024-01-11T08:10:30.472646+0000","last_fullsized":"2024-01-11T08:10:30.472646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:50:45.947278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":163,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.476809+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:10:30.476809+0000","last_peered":"2024-01-11T08:10:30.476809+0000","last_clean":"2024-01-11T08:10:30.476809+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:10:30.476809+0000","last_undegraded":"2024-01-11T08:10:30.476809+0000","last_fullsized":"2024-01-11T08:10:30.476809+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.495364+0000","last_change":"2024-01-11T08:10:30.491665+0000","last_active":"2024-01-11T08:10:30.495364+0000","last_peered":"2024-01-11T08:10:30.495364+0000","last_clean":"2024-01-11T08:10:30.495364+0000","last_became_active":"2024-01-11T08:10:30.491329+0000","last_became_peered":"2024-01-11T08:10:30.491329+0000","last_unstale":"2024-01-11T08:10:30.495364+0000","last_undegraded":"2024-01-11T08:10:30.495364+0000","last_fullsized":"2024-01-11T08:10:30.495364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:26:51.888257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.476968+0000","last_change":"2024-01-11T08:10:29.676382+0000","last_active":"2024-01-11T08:10:30.476968+0000","last_peered":"2024-01-11T08:10:30.476968+0000","last_clean":"2024-01-11T08:10:30.476968+0000","last_became_active":"2024-01-11T08:10:29.674690+0000","last_became_peered":"2024-01-11T08:10:29.674690+0000","last_unstale":"2024-01-11T08:10:30.476968+0000","last_undegraded":"2024-01-11T08:10:30.476968+0000","last_fullsized":"2024-01-11T08:10:30.476968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:12:48.941888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.479210+0000","last_change":"2024-01-11T08:10:29.680063+0000","last_active":"2024-01-11T08:10:30.479210+0000","last_peered":"2024-01-11T08:10:30.479210+0000","last_clean":"2024-01-11T08:10:30.479210+0000","last_became_active":"2024-01-11T08:10:29.677976+0000","last_became_peered":"2024-01-11T08:10:29.677976+0000","last_unstale":"2024-01-11T08:10:30.479210+0000","last_undegraded":"2024-01-11T08:10:30.479210+0000","last_fullsized":"2024-01-11T08:10:30.479210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:50:45.947278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.486597+0000","last_change":"2024-01-11T08:10:30.483048+0000","last_active":"2024-01-11T08:10:30.486597+0000","last_peered":"2024-01-11T08:10:30.486597+0000","last_clean":"2024-01-11T08:10:30.486597+0000","last_became_active":"2024-01-11T08:10:30.482693+0000","last_became_peered":"2024-01-11T08:10:30.482693+0000","last_unstale":"2024-01-11T08:10:30.486597+0000","last_undegraded":"2024-01-11T08:10:30.486597+0000","last_fullsized":"2024-01-11T08:10:30.486597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:22:43.961595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T08:10:31.469542+0000","last_change":"2024-01-11T08:10:29.673099+0000","last_active":"2024-01-11T08:10:31.469542+0000","last_peered":"2024-01-11T08:10:31.469542+0000","last_clean":"2024-01-11T08:10:31.469542+0000","last_became_active":"2024-01-11T08:10:29.672796+0000","last_became_peered":"2024-01-11T08:10:29.672796+0000","last_unstale":"2024-01-11T08:10:31.469542+0000","last_undegraded":"2024-01-11T08:10:31.469542+0000","last_fullsized":"2024-01-11T08:10:31.469542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:05.676391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.490638+0000","last_change":"2024-01-11T08:10:30.489252+0000","last_active":"2024-01-11T08:10:30.490638+0000","last_peered":"2024-01-11T08:10:30.490638+0000","last_clean":"2024-01-11T08:10:30.490638+0000","last_became_active":"2024-01-11T08:10:30.488728+0000","last_became_peered":"2024-01-11T08:10:30.488728+0000","last_unstale":"2024-01-11T08:10:30.490638+0000","last_undegraded":"2024-01-11T08:10:30.490638+0000","last_fullsized":"2024-01-11T08:10:30.490638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:01:02.902411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.680479+0000","last_change":"2024-01-11T08:10:29.678413+0000","last_active":"2024-01-11T08:10:30.680479+0000","last_peered":"2024-01-11T08:10:30.680479+0000","last_clean":"2024-01-11T08:10:30.680479+0000","last_became_active":"2024-01-11T08:10:29.678023+0000","last_became_peered":"2024-01-11T08:10:29.678023+0000","last_unstale":"2024-01-11T08:10:30.680479+0000","last_undegraded":"2024-01-11T08:10:30.680479+0000","last_fullsized":"2024-01-11T08:10:30.680479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:50:45.947278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.491408+0000","last_change":"2024-01-11T08:10:30.486306+0000","last_active":"2024-01-11T08:10:30.491408+0000","last_peered":"2024-01-11T08:10:30.491408+0000","last_clean":"2024-01-11T08:10:30.491408+0000","last_became_active":"2024-01-11T08:10:30.486002+0000","last_became_peered":"2024-01-11T08:10:30.486002+0000","last_unstale":"2024-01-11T08:10:30.491408+0000","last_undegraded":"2024-01-11T08:10:30.491408+0000","last_fullsized":"2024-01-11T08:10:30.491408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:21.127779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.476646+0000","last_change":"2024-01-11T08:10:29.674369+0000","last_active":"2024-01-11T08:10:30.476646+0000","last_peered":"2024-01-11T08:10:30.476646+0000","last_clean":"2024-01-11T08:10:30.476646+0000","last_became_active":"2024-01-11T08:10:29.673110+0000","last_became_peered":"2024-01-11T08:10:29.673110+0000","last_unstale":"2024-01-11T08:10:30.476646+0000","last_undegraded":"2024-01-11T08:10:30.476646+0000","last_fullsized":"2024-01-11T08:10:30.476646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:27:23.756042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.483307+0000","last_change":"2024-01-11T08:10:30.476023+0000","last_active":"2024-01-11T08:10:30.483307+0000","last_peered":"2024-01-11T08:10:30.483307+0000","last_clean":"2024-01-11T08:10:30.483307+0000","last_became_active":"2024-01-11T08:10:30.474988+0000","last_became_peered":"2024-01-11T08:10:30.474988+0000","last_unstale":"2024-01-11T08:10:30.483307+0000","last_undegraded":"2024-01-11T08:10:30.483307+0000","last_fullsized":"2024-01-11T08:10:30.483307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:12:49.944597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.472984+0000","last_change":"2024-01-11T08:10:29.675819+0000","last_active":"2024-01-11T08:10:30.472984+0000","last_peered":"2024-01-11T08:10:30.472984+0000","last_clean":"2024-01-11T08:10:30.472984+0000","last_became_active":"2024-01-11T08:10:29.675005+0000","last_became_peered":"2024-01-11T08:10:29.675005+0000","last_unstale":"2024-01-11T08:10:30.472984+0000","last_undegraded":"2024-01-11T08:10:30.472984+0000","last_fullsized":"2024-01-11T08:10:30.472984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:27:23.756042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.479995+0000","last_change":"2024-01-11T08:10:30.475525+0000","last_active":"2024-01-11T08:10:30.479995+0000","last_peered":"2024-01-11T08:10:30.479995+0000","last_clean":"2024-01-11T08:10:30.479995+0000","last_became_active":"2024-01-11T08:10:30.475243+0000","last_became_peered":"2024-01-11T08:10:30.475243+0000","last_unstale":"2024-01-11T08:10:30.479995+0000","last_undegraded":"2024-01-11T08:10:30.479995+0000","last_fullsized":"2024-01-11T08:10:30.479995+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:37:39.503131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.473867+0000","last_change":"2024-01-11T08:10:29.672091+0000","last_active":"2024-01-11T08:10:30.473867+0000","last_peered":"2024-01-11T08:10:30.473867+0000","last_clean":"2024-01-11T08:10:30.473867+0000","last_became_active":"2024-01-11T08:10:29.671820+0000","last_became_peered":"2024-01-11T08:10:29.671820+0000","last_unstale":"2024-01-11T08:10:30.473867+0000","last_undegraded":"2024-01-11T08:10:30.473867+0000","last_fullsized":"2024-01-11T08:10:30.473867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:46:57.523656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.490433+0000","last_change":"2024-01-11T08:10:30.488516+0000","last_active":"2024-01-11T08:10:30.490433+0000","last_peered":"2024-01-11T08:10:30.490433+0000","last_clean":"2024-01-11T08:10:30.490433+0000","last_became_active":"2024-01-11T08:10:30.487592+0000","last_became_peered":"2024-01-11T08:10:30.487592+0000","last_unstale":"2024-01-11T08:10:30.490433+0000","last_undegraded":"2024-01-11T08:10:30.490433+0000","last_fullsized":"2024-01-11T08:10:30.490433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:57:19.989587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.678978+0000","last_change":"2024-01-11T08:10:29.683377+0000","last_active":"2024-01-11T08:10:30.678978+0000","last_peered":"2024-01-11T08:10:30.678978+0000","last_clean":"2024-01-11T08:10:30.678978+0000","last_became_active":"2024-01-11T08:10:29.680358+0000","last_became_peered":"2024-01-11T08:10:29.680358+0000","last_unstale":"2024-01-11T08:10:30.678978+0000","last_undegraded":"2024-01-11T08:10:30.678978+0000","last_fullsized":"2024-01-11T08:10:30.678978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:05.676391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.502306+0000","last_change":"2024-01-11T08:10:30.499934+0000","last_active":"2024-01-11T08:10:30.502306+0000","last_peered":"2024-01-11T08:10:30.502306+0000","last_clean":"2024-01-11T08:10:30.502306+0000","last_became_active":"2024-01-11T08:10:30.499676+0000","last_became_peered":"2024-01-11T08:10:30.499676+0000","last_unstale":"2024-01-11T08:10:30.502306+0000","last_undegraded":"2024-01-11T08:10:30.502306+0000","last_fullsized":"2024-01-11T08:10:30.502306+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:47.753678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.471402+0000","last_change":"2024-01-11T08:10:29.665360+0000","last_active":"2024-01-11T08:10:30.471402+0000","last_peered":"2024-01-11T08:10:30.471402+0000","last_clean":"2024-01-11T08:10:30.471402+0000","last_became_active":"2024-01-11T08:10:29.664493+0000","last_became_peered":"2024-01-11T08:10:29.664493+0000","last_unstale":"2024-01-11T08:10:30.471402+0000","last_undegraded":"2024-01-11T08:10:30.471402+0000","last_fullsized":"2024-01-11T08:10:30.471402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:12:48.941888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.495532+0000","last_change":"2024-01-11T08:10:30.491426+0000","last_active":"2024-01-11T08:10:30.495532+0000","last_peered":"2024-01-11T08:10:30.495532+0000","last_clean":"2024-01-11T08:10:30.495532+0000","last_became_active":"2024-01-11T08:10:30.491167+0000","last_became_peered":"2024-01-11T08:10:30.491167+0000","last_unstale":"2024-01-11T08:10:30.495532+0000","last_undegraded":"2024-01-11T08:10:30.495532+0000","last_fullsized":"2024-01-11T08:10:30.495532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:12.649092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.678720+0000","last_change":"2024-01-11T08:10:29.669311+0000","last_active":"2024-01-11T08:10:30.678720+0000","last_peered":"2024-01-11T08:10:30.678720+0000","last_clean":"2024-01-11T08:10:30.678720+0000","last_became_active":"2024-01-11T08:10:29.669042+0000","last_became_peered":"2024-01-11T08:10:29.669042+0000","last_unstale":"2024-01-11T08:10:30.678720+0000","last_undegraded":"2024-01-11T08:10:30.678720+0000","last_fullsized":"2024-01-11T08:10:30.678720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:48.964847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.494014+0000","last_change":"2024-01-11T08:10:30.486285+0000","last_active":"2024-01-11T08:10:30.494014+0000","last_peered":"2024-01-11T08:10:30.494014+0000","last_clean":"2024-01-11T08:10:30.494014+0000","last_became_active":"2024-01-11T08:10:30.482784+0000","last_became_peered":"2024-01-11T08:10:30.482784+0000","last_unstale":"2024-01-11T08:10:30.494014+0000","last_undegraded":"2024-01-11T08:10:30.494014+0000","last_fullsized":"2024-01-11T08:10:30.494014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:37:39.503131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.687135+0000","last_change":"2024-01-11T08:10:29.668553+0000","last_active":"2024-01-11T08:10:30.687135+0000","last_peered":"2024-01-11T08:10:30.687135+0000","last_clean":"2024-01-11T08:10:30.687135+0000","last_became_active":"2024-01-11T08:10:29.666406+0000","last_became_peered":"2024-01-11T08:10:29.666406+0000","last_unstale":"2024-01-11T08:10:30.687135+0000","last_undegraded":"2024-01-11T08:10:30.687135+0000","last_fullsized":"2024-01-11T08:10:30.687135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:28.640227+0000","last_clean_scrub_stamp":"2024-01-11T08:10:28.640227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:50:14.474575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.503635+0000","last_change":"2024-01-11T08:10:30.501614+0000","last_active":"2024-01-11T08:10:30.503635+0000","last_peered":"2024-01-11T08:10:30.503635+0000","last_clean":"2024-01-11T08:10:30.503635+0000","last_became_active":"2024-01-11T08:10:30.501255+0000","last_became_peered":"2024-01-11T08:10:30.501255+0000","last_unstale":"2024-01-11T08:10:30.503635+0000","last_undegraded":"2024-01-11T08:10:30.503635+0000","last_fullsized":"2024-01-11T08:10:30.503635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:18:09.736814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.491229+0000","last_change":"2024-01-11T08:10:30.488971+0000","last_active":"2024-01-11T08:10:30.491229+0000","last_peered":"2024-01-11T08:10:30.491229+0000","last_clean":"2024-01-11T08:10:30.491229+0000","last_became_active":"2024-01-11T08:10:30.488681+0000","last_became_peered":"2024-01-11T08:10:30.488681+0000","last_unstale":"2024-01-11T08:10:30.491229+0000","last_undegraded":"2024-01-11T08:10:30.491229+0000","last_fullsized":"2024-01-11T08:10:30.491229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:26:51.888257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.481588+0000","last_change":"2024-01-11T08:10:30.477289+0000","last_active":"2024-01-11T08:10:30.481588+0000","last_peered":"2024-01-11T08:10:30.481588+0000","last_clean":"2024-01-11T08:10:30.481588+0000","last_became_active":"2024-01-11T08:10:30.477029+0000","last_became_peered":"2024-01-11T08:10:30.477029+0000","last_unstale":"2024-01-11T08:10:30.481588+0000","last_undegraded":"2024-01-11T08:10:30.481588+0000","last_fullsized":"2024-01-11T08:10:30.481588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:49.967555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.483210+0000","last_change":"2024-01-11T08:10:30.478289+0000","last_active":"2024-01-11T08:10:30.483210+0000","last_peered":"2024-01-11T08:10:30.483210+0000","last_clean":"2024-01-11T08:10:30.483210+0000","last_became_active":"2024-01-11T08:10:30.477815+0000","last_became_peered":"2024-01-11T08:10:30.477815+0000","last_unstale":"2024-01-11T08:10:30.483210+0000","last_undegraded":"2024-01-11T08:10:30.483210+0000","last_fullsized":"2024-01-11T08:10:30.483210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:14:36.595264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.491328+0000","last_change":"2024-01-11T08:10:30.486406+0000","last_active":"2024-01-11T08:10:30.491328+0000","last_peered":"2024-01-11T08:10:30.491328+0000","last_clean":"2024-01-11T08:10:30.491328+0000","last_became_active":"2024-01-11T08:10:30.485979+0000","last_became_peered":"2024-01-11T08:10:30.485979+0000","last_unstale":"2024-01-11T08:10:30.491328+0000","last_undegraded":"2024-01-11T08:10:30.491328+0000","last_fullsized":"2024-01-11T08:10:30.491328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:37:25.109490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.485482+0000","last_change":"2024-01-11T08:10:30.480947+0000","last_active":"2024-01-11T08:10:30.485482+0000","last_peered":"2024-01-11T08:10:30.485482+0000","last_clean":"2024-01-11T08:10:30.485482+0000","last_became_active":"2024-01-11T08:10:30.478542+0000","last_became_peered":"2024-01-11T08:10:30.478542+0000","last_unstale":"2024-01-11T08:10:30.485482+0000","last_undegraded":"2024-01-11T08:10:30.485482+0000","last_fullsized":"2024-01-11T08:10:30.485482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:21:29.220934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.487827+0000","last_change":"2024-01-11T08:10:30.486206+0000","last_active":"2024-01-11T08:10:30.487827+0000","last_peered":"2024-01-11T08:10:30.487827+0000","last_clean":"2024-01-11T08:10:30.487827+0000","last_became_active":"2024-01-11T08:10:30.482118+0000","last_became_peered":"2024-01-11T08:10:30.482118+0000","last_unstale":"2024-01-11T08:10:30.487827+0000","last_undegraded":"2024-01-11T08:10:30.487827+0000","last_fullsized":"2024-01-11T08:10:30.487827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:37:39.503131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.502193+0000","last_change":"2024-01-11T08:10:30.498251+0000","last_active":"2024-01-11T08:10:30.502193+0000","last_peered":"2024-01-11T08:10:30.502193+0000","last_clean":"2024-01-11T08:10:30.502193+0000","last_became_active":"2024-01-11T08:10:30.498032+0000","last_became_peered":"2024-01-11T08:10:30.498032+0000","last_unstale":"2024-01-11T08:10:30.502193+0000","last_undegraded":"2024-01-11T08:10:30.502193+0000","last_fullsized":"2024-01-11T08:10:30.502193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:22:43.961595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.480236+0000","last_change":"2024-01-11T08:10:30.477044+0000","last_active":"2024-01-11T08:10:30.480236+0000","last_peered":"2024-01-11T08:10:30.480236+0000","last_clean":"2024-01-11T08:10:30.480236+0000","last_became_active":"2024-01-11T08:10:30.476728+0000","last_became_peered":"2024-01-11T08:10:30.476728+0000","last_unstale":"2024-01-11T08:10:30.480236+0000","last_undegraded":"2024-01-11T08:10:30.480236+0000","last_fullsized":"2024-01-11T08:10:30.480236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:08:10.140431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.485414+0000","last_change":"2024-01-11T08:10:30.481145+0000","last_active":"2024-01-11T08:10:30.485414+0000","last_peered":"2024-01-11T08:10:30.485414+0000","last_clean":"2024-01-11T08:10:30.485414+0000","last_became_active":"2024-01-11T08:10:30.477464+0000","last_became_peered":"2024-01-11T08:10:30.477464+0000","last_unstale":"2024-01-11T08:10:30.485414+0000","last_undegraded":"2024-01-11T08:10:30.485414+0000","last_fullsized":"2024-01-11T08:10:30.485414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:02.887025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.485358+0000","last_change":"2024-01-11T08:10:30.480047+0000","last_active":"2024-01-11T08:10:30.485358+0000","last_peered":"2024-01-11T08:10:30.485358+0000","last_clean":"2024-01-11T08:10:30.485358+0000","last_became_active":"2024-01-11T08:10:30.479594+0000","last_became_peered":"2024-01-11T08:10:30.479594+0000","last_unstale":"2024-01-11T08:10:30.485358+0000","last_undegraded":"2024-01-11T08:10:30.485358+0000","last_fullsized":"2024-01-11T08:10:30.485358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:14:36.595264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.490024+0000","last_change":"2024-01-11T08:10:30.486964+0000","last_active":"2024-01-11T08:10:30.490024+0000","last_peered":"2024-01-11T08:10:30.490024+0000","last_clean":"2024-01-11T08:10:30.490024+0000","last_became_active":"2024-01-11T08:10:30.486679+0000","last_became_peered":"2024-01-11T08:10:30.486679+0000","last_unstale":"2024-01-11T08:10:30.490024+0000","last_undegraded":"2024-01-11T08:10:30.490024+0000","last_fullsized":"2024-01-11T08:10:30.490024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:57:19.989587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.497053+0000","last_change":"2024-01-11T08:10:30.495904+0000","last_active":"2024-01-11T08:10:30.497053+0000","last_peered":"2024-01-11T08:10:30.497053+0000","last_clean":"2024-01-11T08:10:30.497053+0000","last_became_active":"2024-01-11T08:10:30.495590+0000","last_became_peered":"2024-01-11T08:10:30.495590+0000","last_unstale":"2024-01-11T08:10:30.497053+0000","last_undegraded":"2024-01-11T08:10:30.497053+0000","last_fullsized":"2024-01-11T08:10:30.497053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:37:39.503131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T08:10:30.502287+0000","last_change":"2024-01-11T08:10:30.499796+0000","last_active":"2024-01-11T08:10:30.502287+0000","last_peered":"2024-01-11T08:10:30.502287+0000","last_clean":"2024-01-11T08:10:30.502287+0000","last_became_active":"2024-01-11T08:10:30.499391+0000","last_became_peered":"2024-01-11T08:10:30.499391+0000","last_unstale":"2024-01-11T08:10:30.502287+0000","last_undegraded":"2024-01-11T08:10:30.502287+0000","last_fullsized":"2024-01-11T08:10:30.502287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:29.642935+0000","last_clean_scrub_stamp":"2024-01-11T08:10:29.642935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:37:39.503131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298084,"kb_used_data":796,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073756,"statfs":{"total":96636764160,"available":96331526144,"internally_reserved":0,"allocated":815104,"data_stored":666235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9119999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":76849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":225280,"data_stored":75473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298092,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073748,"statfs":{"total":96636764160,"available":96331517952,"internally_reserved":0,"allocated":823296,"data_stored":666291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":217088,"data_stored":75313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81699999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":76849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":229376,"data_stored":75439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":221184,"data_stored":75403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:10:39.448 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-11T08:10:39.448 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-11T08:10:39.448 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-11T08:10:39.448 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-11T08:10:39.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.681+0000 7f4351dff700 1 Processor -- start 2024-01-11T08:10:39.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.681+0000 7f4351dff700 1 -- start start 2024-01-11T08:10:39.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.683+0000 7f4351dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f434c1596c0 0x7f434c15d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:39.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.683+0000 7f4351dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f434c068d30 0x7f434c0aa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:39.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.683+0000 7f4351dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f434c0a9cb0 con 0x7f434c0ca670 2024-01-11T08:10:39.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.683+0000 7f4351dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f434c15e210 con 0x7f434c1596c0 2024-01-11T08:10:39.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.683+0000 7f4351dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f434c15dc50 con 0x7f434c068d30 2024-01-11T08:10:39.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.683+0000 7f434b7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f434c1596c0 0x7f434c15d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:39.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.683+0000 7f434b7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f434c1596c0 0x7f434c15d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53430/0 (socket says 172.21.15.27:53430) 2024-01-11T08:10:39.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.683+0000 7f434b7fe700 1 -- 172.21.15.27:0/1338392070 learned_addr learned my addr 172.21.15.27:0/1338392070 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:39.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.683+0000 7f434bfff700 1 --2- 172.21.15.27:0/1338392070 >> v2:172.21.15.77:3300/0 conn(0x7f434c068d30 0x7f434c0aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:39.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.683+0000 7f434b7fe700 1 -- 172.21.15.27:0/1338392070 >> v2:172.21.15.77:3300/0 conn(0x7f434c068d30 msgr2=0x7f434c0aa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:39.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.683+0000 7f434b7fe700 1 --2- 172.21.15.27:0/1338392070 >> v2:172.21.15.77:3300/0 conn(0x7f434c068d30 0x7f434c0aa140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:39.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.683+0000 7f434b7fe700 1 -- 172.21.15.27:0/1338392070 >> v1:172.21.15.77:6789/0 conn(0x7f434c0ca670 legacy=0x7f434c068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:39.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.683+0000 7f434b7fe700 1 -- 172.21.15.27:0/1338392070 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434c15df30 con 0x7f434c1596c0 2024-01-11T08:10:39.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.684+0000 7f434b7fe700 1 --2- 172.21.15.27:0/1338392070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f434c1596c0 0x7f434c15d300 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f434c069a50 tx=0x7f433c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4b93ec208ac30a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:39.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.684+0000 7f434a7fc700 1 -- 172.21.15.27:0/1338392070 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c00b860 con 0x7f434c1596c0 2024-01-11T08:10:39.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.684+0000 7f434a7fc700 1 -- 172.21.15.27:0/1338392070 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433c00b9c0 con 0x7f434c1596c0 2024-01-11T08:10:39.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.684+0000 7f434a7fc700 1 -- 172.21.15.27:0/1338392070 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c00bd60 con 0x7f434c1596c0 2024-01-11T08:10:39.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.684+0000 7f4351dff700 1 -- 172.21.15.27:0/1338392070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f434c1596c0 msgr2=0x7f434c15d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:39.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.684+0000 7f4351dff700 1 --2- 172.21.15.27:0/1338392070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f434c1596c0 0x7f434c15d300 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f434c069a50 tx=0x7f433c009250 comp rx=0 tx=0).stop 2024-01-11T08:10:39.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.685+0000 7f4351dff700 1 -- 172.21.15.27:0/1338392070 shutdown_connections 2024-01-11T08:10:39.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.685+0000 7f4351dff700 1 --2- 172.21.15.27:0/1338392070 >> v2:172.21.15.77:3300/0 conn(0x7f434c068d30 0x7f434c0aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:39.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.685+0000 7f4351dff700 1 --2- 172.21.15.27:0/1338392070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f434c1596c0 0x7f434c15d300 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:39.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.685+0000 7f4351dff700 1 -- 172.21.15.27:0/1338392070 >> 172.21.15.27:0/1338392070 conn(0x7f434c0c9f70 msgr2=0x7f434c06deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:39.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.685+0000 7f4351dff700 1 -- 172.21.15.27:0/1338392070 shutdown_connections 2024-01-11T08:10:39.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.685+0000 7f4351dff700 1 -- 172.21.15.27:0/1338392070 wait complete. 2024-01-11T08:10:39.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.686+0000 7f4351dff700 1 Processor -- start 2024-01-11T08:10:39.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.686+0000 7f4351dff700 1 -- start start 2024-01-11T08:10:39.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.687+0000 7f4351dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f434c068d30 0x7f434c0aa900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:39.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.687+0000 7f4351dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f434c0ca670 0x7f434c148050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:39.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.687+0000 7f4351dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f434c1596c0 0x7f434c148590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:39.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.687+0000 7f4351dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c148ad0 con 0x7f434c068d30 2024-01-11T08:10:39.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.687+0000 7f4351dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c16e080 con 0x7f434c0ca670 2024-01-11T08:10:39.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.687+0000 7f4351dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434c16d470 con 0x7f434c1596c0 2024-01-11T08:10:39.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.687+0000 7f434b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f434c068d30 0x7f434c0aa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:39.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.687+0000 7f434b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f434c068d30 0x7f434c0aa900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53442/0 (socket says 172.21.15.27:53442) 2024-01-11T08:10:39.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.687+0000 7f434b7fe700 1 -- 172.21.15.27:0/1641572636 learned_addr learned my addr 172.21.15.27:0/1641572636 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:39.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.688+0000 7f434affd700 1 --2- 172.21.15.27:0/1641572636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f434c0ca670 0x7f434c148050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:39.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.688+0000 7f434bfff700 1 --2- 172.21.15.27:0/1641572636 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f434c1596c0 0x7f434c148590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:39.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.688+0000 7f434b7fe700 1 -- 172.21.15.27:0/1641572636 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f434c1596c0 msgr2=0x7f434c148590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:39.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.688+0000 7f434b7fe700 1 --2- 172.21.15.27:0/1641572636 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f434c1596c0 0x7f434c148590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:39.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.688+0000 7f434b7fe700 1 -- 172.21.15.27:0/1641572636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f434c0ca670 msgr2=0x7f434c148050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:39.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.688+0000 7f434b7fe700 1 --2- 172.21.15.27:0/1641572636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f434c0ca670 0x7f434c148050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:39.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.688+0000 7f434b7fe700 1 -- 172.21.15.27:0/1641572636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c0094f0 con 0x7f434c068d30 2024-01-11T08:10:39.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.688+0000 7f434b7fe700 1 --2- 172.21.15.27:0/1641572636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f434c068d30 0x7f434c0aa900 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f434c06d490 tx=0x7f433c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:39.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.688+0000 7f43337fe700 1 -- 172.21.15.27:0/1641572636 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c011e60 con 0x7f434c068d30 2024-01-11T08:10:39.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.688+0000 7f43337fe700 1 -- 172.21.15.27:0/1641572636 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433c01a630 con 0x7f434c068d30 2024-01-11T08:10:39.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.688+0000 7f43337fe700 1 -- 172.21.15.27:0/1641572636 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c01a9f0 con 0x7f434c068d30 2024-01-11T08:10:39.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.688+0000 7f4351dff700 1 -- 172.21.15.27:0/1641572636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434c16d750 con 0x7f434c068d30 2024-01-11T08:10:39.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.689+0000 7f4351dff700 1 -- 172.21.15.27:0/1641572636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434c16dd10 con 0x7f434c068d30 2024-01-11T08:10:39.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.690+0000 7f4351dff700 1 -- 172.21.15.27:0/1641572636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f434c16e410 con 0x7f434c068d30 2024-01-11T08:10:39.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.690+0000 7f43337fe700 1 -- 172.21.15.27:0/1641572636 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f433c02d440 con 0x7f434c068d30 2024-01-11T08:10:39.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.691+0000 7f43337fe700 1 --2- 172.21.15.27:0/1641572636 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4338067000 0x7f43380694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:39.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.691+0000 7f43337fe700 1 -- 172.21.15.27:0/1641572636 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f433c0860e0 con 0x7f434c068d30 2024-01-11T08:10:39.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.692+0000 7f434affd700 1 --2- 172.21.15.27:0/1641572636 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4338067000 0x7f43380694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:39.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.692+0000 7f434affd700 1 --2- 172.21.15.27:0/1641572636 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4338067000 0x7f43380694b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f434c0abe50 tx=0x7f4340009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:39.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.696+0000 7f43337fe700 1 -- 172.21.15.27:0/1641572636 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f434c16e410 con 0x7f434c068d30 2024-01-11T08:10:39.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.943+0000 7f4351dff700 1 -- 172.21.15.27:0/1641572636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f434c16e410 con 0x7f434c068d30 2024-01-11T08:10:39.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.943+0000 7f43337fe700 1 -- 172.21.15.27:0/1641572636 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f434c16e410 con 0x7f434c068d30 2024-01-11T08:10:39.944 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:39.944 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-11T08:10:39.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.946+0000 7f4351dff700 1 -- 172.21.15.27:0/1641572636 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4338067000 msgr2=0x7f43380694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:39.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.946+0000 7f4351dff700 1 --2- 172.21.15.27:0/1641572636 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4338067000 0x7f43380694b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f434c0abe50 tx=0x7f4340009040 comp rx=0 tx=0).stop 2024-01-11T08:10:39.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.946+0000 7f4351dff700 1 -- 172.21.15.27:0/1641572636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f434c068d30 msgr2=0x7f434c0aa900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:39.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.946+0000 7f4351dff700 1 --2- 172.21.15.27:0/1641572636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f434c068d30 0x7f434c0aa900 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f434c06d490 tx=0x7f433c017000 comp rx=0 tx=0).stop 2024-01-11T08:10:39.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.946+0000 7f4351dff700 1 -- 172.21.15.27:0/1641572636 shutdown_connections 2024-01-11T08:10:39.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.946+0000 7f4351dff700 1 --2- 172.21.15.27:0/1641572636 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4338067000 0x7f43380694b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:39.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.946+0000 7f4351dff700 1 --2- 172.21.15.27:0/1641572636 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f434c1596c0 0x7f434c148590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:39.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.946+0000 7f4351dff700 1 --2- 172.21.15.27:0/1641572636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f434c068d30 0x7f434c0aa900 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:39.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.946+0000 7f4351dff700 1 --2- 172.21.15.27:0/1641572636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f434c0ca670 0x7f434c148050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:39.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.946+0000 7f4351dff700 1 -- 172.21.15.27:0/1641572636 >> 172.21.15.27:0/1641572636 conn(0x7f434c0c9f70 msgr2=0x7f434c0aef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:39.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.946+0000 7f4351dff700 1 -- 172.21.15.27:0/1641572636 shutdown_connections 2024-01-11T08:10:39.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:39.946+0000 7f4351dff700 1 -- 172.21.15.27:0/1641572636 wait complete. 2024-01-11T08:10:39.964 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-11T08:10:39.965 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:10:40.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.164+0000 7f91453ff700 1 Processor -- start 2024-01-11T08:10:40.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.164+0000 7f91453ff700 1 -- start start 2024-01-11T08:10:40.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.165+0000 7f91453ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f91400b2b90 0x7f9140159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:40.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.165+0000 7f91453ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91400cd730 con 0x7f91400b03e0 2024-01-11T08:10:40.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.165+0000 7f91453ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91400ccb20 con 0x7f91400b07b0 2024-01-11T08:10:40.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.165+0000 7f91453ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91400cce00 con 0x7f91400b2b90 2024-01-11T08:10:40.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.165+0000 7f913effd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f91400b03e0 0x7f91400b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37250/0 (socket says 172.21.15.27:37250) 2024-01-11T08:10:40.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.165+0000 7f913effd700 1 -- 172.21.15.27:0/1247252318 learned_addr learned my addr 172.21.15.27:0/1247252318 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:40.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.165+0000 7f913e7fc700 1 --2- 172.21.15.27:0/1247252318 >> v2:172.21.15.77:3300/0 conn(0x7f91400b2b90 0x7f9140159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:40.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913dffb700 1 -- 172.21.15.27:0/1247252318 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239947676 0 0) 0x7f91400cd730 con 0x7f91400b03e0 2024-01-11T08:10:40.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913dffb700 1 -- 172.21.15.27:0/1247252318 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9128003570 con 0x7f91400b03e0 2024-01-11T08:10:40.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913dffb700 1 -- 172.21.15.27:0/1247252318 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492385140 0 0) 0x7f91400ccb20 con 0x7f91400b07b0 2024-01-11T08:10:40.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913dffb700 1 -- 172.21.15.27:0/1247252318 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91400cd730 con 0x7f91400b07b0 2024-01-11T08:10:40.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913dffb700 1 -- 172.21.15.27:0/1247252318 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 702613285 0 0) 0x7f9128003570 con 0x7f91400b03e0 2024-01-11T08:10:40.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913dffb700 1 -- 172.21.15.27:0/1247252318 >> v2:172.21.15.77:3300/0 conn(0x7f91400b2b90 msgr2=0x7f9140159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:40.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913dffb700 1 --2- 172.21.15.27:0/1247252318 >> v2:172.21.15.77:3300/0 conn(0x7f91400b2b90 0x7f9140159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913dffb700 1 -- 172.21.15.27:0/1247252318 >> v1:172.21.15.77:6789/0 conn(0x7f91400b07b0 legacy=0x7f9140159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:40.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913dffb700 1 -- 172.21.15.27:0/1247252318 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91400cd0e0 con 0x7f91400b03e0 2024-01-11T08:10:40.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913dffb700 1 -- 172.21.15.27:0/1247252318 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f9130001470 con 0x7f91400b03e0 2024-01-11T08:10:40.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913dffb700 1 -- 172.21.15.27:0/1247252318 >> v1:172.21.15.27:6789/0 conn(0x7f91400b03e0 legacy=0x7f91400b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:40.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913dffb700 1 --2- 172.21.15.27:0/1247252318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9128003c40 0x7f9128004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:40.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913dffb700 1 --2- 172.21.15.27:0/1247252318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9128004810 0x7f9128004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:40.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913dffb700 1 --2- 172.21.15.27:0/1247252318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9128005210 0x7f912800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:40.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913dffb700 1 -- 172.21.15.27:0/1247252318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91400cce00 con 0x7f9128004810 2024-01-11T08:10:40.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913dffb700 1 -- 172.21.15.27:0/1247252318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912800c130 con 0x7f9128005210 2024-01-11T08:10:40.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913dffb700 1 -- 172.21.15.27:0/1247252318 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f912800c2c0 con 0x7f9128003c40 2024-01-11T08:10:40.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913e7fc700 1 --2- 172.21.15.27:0/1247252318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9128004810 0x7f9128004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:40.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913effd700 1 --2- 172.21.15.27:0/1247252318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9128003c40 0x7f9128004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:40.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.166+0000 7f913f7fe700 1 --2- 172.21.15.27:0/1247252318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9128005210 0x7f912800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:40.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.167+0000 7f913e7fc700 1 -- 172.21.15.27:0/1247252318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9128003c40 msgr2=0x7f9128004030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:40.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.167+0000 7f913e7fc700 1 --2- 172.21.15.27:0/1247252318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9128003c40 0x7f9128004030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.167+0000 7f913e7fc700 1 -- 172.21.15.27:0/1247252318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9128005210 msgr2=0x7f912800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:40.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.167+0000 7f913e7fc700 1 --2- 172.21.15.27:0/1247252318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9128005210 0x7f912800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.167+0000 7f913e7fc700 1 -- 172.21.15.27:0/1247252318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9128003570 con 0x7f9128004810 2024-01-11T08:10:40.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.167+0000 7f913f7fe700 1 --2- 172.21.15.27:0/1247252318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9128005210 0x7f912800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:40.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.167+0000 7f913f7fe700 1 -- 172.21.15.27:0/1247252318 reap_dead start 2024-01-11T08:10:40.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.167+0000 7f913e7fc700 1 --2- 172.21.15.27:0/1247252318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9128004810 0x7f9128004c80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9134009c30 tx=0x7f91340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3819b8b7e0f64a08 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:40.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.167+0000 7f913dffb700 1 -- 172.21.15.27:0/1247252318 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91340095b0 con 0x7f9128004810 2024-01-11T08:10:40.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.167+0000 7f913dffb700 1 -- 172.21.15.27:0/1247252318 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913400e710 con 0x7f9128004810 2024-01-11T08:10:40.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.167+0000 7f913dffb700 1 -- 172.21.15.27:0/1247252318 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913400ead0 con 0x7f9128004810 2024-01-11T08:10:40.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.168+0000 7f91453ff700 1 -- 172.21.15.27:0/1247252318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9128004810 msgr2=0x7f9128004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:40.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.168+0000 7f91453ff700 1 --2- 172.21.15.27:0/1247252318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9128004810 0x7f9128004c80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9134009c30 tx=0x7f91340091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.168+0000 7f91453ff700 1 -- 172.21.15.27:0/1247252318 shutdown_connections 2024-01-11T08:10:40.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.168+0000 7f91453ff700 1 --2- 172.21.15.27:0/1247252318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9128004810 0x7f9128004c80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.168+0000 7f91453ff700 1 -- 172.21.15.27:0/1247252318 >> 172.21.15.27:0/1247252318 conn(0x7f914006ed40 msgr2=0x7f914006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:40.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.168+0000 7f91453ff700 1 -- 172.21.15.27:0/1247252318 shutdown_connections 2024-01-11T08:10:40.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.168+0000 7f91453ff700 1 -- 172.21.15.27:0/1247252318 wait complete. 2024-01-11T08:10:40.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.169+0000 7f91453ff700 1 Processor -- start 2024-01-11T08:10:40.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.170+0000 7f91453ff700 1 -- start start 2024-01-11T08:10:40.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.170+0000 7f91453ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9128004810 0x7f9140169a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:40.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.170+0000 7f91453ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9140169fc0 0x7f914016ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:40.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.170+0000 7f91453ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f914016b190 0x7f91400a78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:40.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.170+0000 7f91453ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91400a7450 con 0x7f9140169fc0 2024-01-11T08:10:40.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.170+0000 7f91453ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914016a540 con 0x7f914016b190 2024-01-11T08:10:40.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.170+0000 7f91453ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914016a820 con 0x7f9128004810 2024-01-11T08:10:40.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.171+0000 7f913e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9140169fc0 0x7f914016ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:40.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.171+0000 7f913e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9140169fc0 0x7f914016ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53470/0 (socket says 172.21.15.27:53470) 2024-01-11T08:10:40.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.171+0000 7f913e7fc700 1 -- 172.21.15.27:0/96290640 learned_addr learned my addr 172.21.15.27:0/96290640 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:40.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.171+0000 7f913effd700 1 --2- 172.21.15.27:0/96290640 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9128004810 0x7f9140169a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:40.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.171+0000 7f913f7fe700 1 --2- 172.21.15.27:0/96290640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f914016b190 0x7f91400a78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:40.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.171+0000 7f913e7fc700 1 -- 172.21.15.27:0/96290640 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9128004810 msgr2=0x7f9140169a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:40.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.171+0000 7f913e7fc700 1 --2- 172.21.15.27:0/96290640 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9128004810 0x7f9140169a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.171+0000 7f913e7fc700 1 -- 172.21.15.27:0/96290640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f914016b190 msgr2=0x7f91400a78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:40.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.171+0000 7f913e7fc700 1 --2- 172.21.15.27:0/96290640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f914016b190 0x7f91400a78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.171+0000 7f913e7fc700 1 -- 172.21.15.27:0/96290640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91400a70e0 con 0x7f9140169fc0 2024-01-11T08:10:40.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.171+0000 7f913effd700 1 --2- 172.21.15.27:0/96290640 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9128004810 0x7f9140169a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:40.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.171+0000 7f913e7fc700 1 --2- 172.21.15.27:0/96290640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9140169fc0 0x7f914016ac50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f913400e4b0 tx=0x7f913401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:40.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.172+0000 7f9126ffd700 1 -- 172.21.15.27:0/96290640 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9134022070 con 0x7f9140169fc0 2024-01-11T08:10:40.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.172+0000 7f9126ffd700 1 -- 172.21.15.27:0/96290640 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913400ea90 con 0x7f9140169fc0 2024-01-11T08:10:40.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.172+0000 7f9126ffd700 1 -- 172.21.15.27:0/96290640 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913401c680 con 0x7f9140169fc0 2024-01-11T08:10:40.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.172+0000 7f91453ff700 1 -- 172.21.15.27:0/96290640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914015bf00 con 0x7f9140169fc0 2024-01-11T08:10:40.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.172+0000 7f91453ff700 1 -- 172.21.15.27:0/96290640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91400a7fe0 con 0x7f9140169fc0 2024-01-11T08:10:40.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.173+0000 7f91453ff700 1 -- 172.21.15.27:0/96290640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91400a81d0 con 0x7f9140169fc0 2024-01-11T08:10:40.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.174+0000 7f9126ffd700 1 -- 172.21.15.27:0/96290640 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f913401ca70 con 0x7f9140169fc0 2024-01-11T08:10:40.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.175+0000 7f9126ffd700 1 --2- 172.21.15.27:0/96290640 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f912c066f30 0x7f912c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:40.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.175+0000 7f9126ffd700 1 -- 172.21.15.27:0/96290640 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f9134085c90 con 0x7f9140169fc0 2024-01-11T08:10:40.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.176+0000 7f913effd700 1 --2- 172.21.15.27:0/96290640 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f912c066f30 0x7f912c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:40.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.176+0000 7f913effd700 1 --2- 172.21.15.27:0/96290640 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f912c066f30 0x7f912c0693e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9130001bd0 tx=0x7f9130009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:40.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.179+0000 7f9126ffd700 1 -- 172.21.15.27:0/96290640 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f91340b75a0 con 0x7f9140169fc0 2024-01-11T08:10:40.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.382+0000 7f91453ff700 1 -- 172.21.15.27:0/96290640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91400a6f40 con 0x7f9140169fc0 2024-01-11T08:10:40.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.383+0000 7f9126ffd700 1 -- 172.21.15.27:0/96290640 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15302 (secure 0 0 0) 0x7f91400a6f40 con 0x7f9140169fc0 2024-01-11T08:10:40.384 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:40.385 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:31.464323+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:10:27.993251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-11T08:10:29.153959+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:10:40.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.387+0000 7f91453ff700 1 -- 172.21.15.27:0/96290640 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f912c066f30 msgr2=0x7f912c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:40.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.387+0000 7f91453ff700 1 --2- 172.21.15.27:0/96290640 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f912c066f30 0x7f912c0693e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9130001bd0 tx=0x7f9130009040 comp rx=0 tx=0).stop 2024-01-11T08:10:40.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.387+0000 7f91453ff700 1 -- 172.21.15.27:0/96290640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9140169fc0 msgr2=0x7f914016ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:40.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.387+0000 7f91453ff700 1 --2- 172.21.15.27:0/96290640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9140169fc0 0x7f914016ac50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f913400e4b0 tx=0x7f913401a000 comp rx=0 tx=0).stop 2024-01-11T08:10:40.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.387+0000 7f91453ff700 1 -- 172.21.15.27:0/96290640 shutdown_connections 2024-01-11T08:10:40.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.387+0000 7f91453ff700 1 --2- 172.21.15.27:0/96290640 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f912c066f30 0x7f912c0693e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.387+0000 7f91453ff700 1 --2- 172.21.15.27:0/96290640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f914016b190 0x7f91400a78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.387+0000 7f91453ff700 1 --2- 172.21.15.27:0/96290640 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9128004810 0x7f9140169a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.387+0000 7f91453ff700 1 --2- 172.21.15.27:0/96290640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9140169fc0 0x7f914016ac50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.387+0000 7f91453ff700 1 -- 172.21.15.27:0/96290640 >> 172.21.15.27:0/96290640 conn(0x7f914006ed40 msgr2=0x7f91400d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:40.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.387+0000 7f91453ff700 1 -- 172.21.15.27:0/96290640 shutdown_connections 2024-01-11T08:10:40.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.387+0000 7f91453ff700 1 -- 172.21.15.27:0/96290640 wait complete. 2024-01-11T08:10:40.405 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T08:10:27.993251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T08:10:29.153959+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T08:10:40.405 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:10:40.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.606+0000 7f9c1bdff700 1 Processor -- start 2024-01-11T08:10:40.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.607+0000 7f9c1bdff700 1 -- start start 2024-01-11T08:10:40.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.608+0000 7f9c1bdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c141596f0 0x7f9c140a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:40.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.608+0000 7f9c1bdff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9c14068d30 0x7f9c140c8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:40.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.608+0000 7f9c1bdff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c140c80a0 con 0x7f9c140a9de0 2024-01-11T08:10:40.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.608+0000 7f9c1bdff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c140cbac0 con 0x7f9c14068d30 2024-01-11T08:10:40.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.608+0000 7f9c1bdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140c7490 con 0x7f9c141596f0 2024-01-11T08:10:40.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.608+0000 7f9c1a39c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9c14068d30 0x7f9c140c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:40.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.608+0000 7f9c1a39c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9c14068d30 0x7f9c140c8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53474/0 (socket says 172.21.15.27:53474) 2024-01-11T08:10:40.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.608+0000 7f9c1a39c700 1 -- 172.21.15.27:0/3967277808 learned_addr learned my addr 172.21.15.27:0/3967277808 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:40.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.609+0000 7f9c19b9b700 1 --2- 172.21.15.27:0/3967277808 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c141596f0 0x7f9c140a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:40.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.609+0000 7f9c1a39c700 1 -- 172.21.15.27:0/3967277808 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c141596f0 msgr2=0x7f9c140a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:40.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.609+0000 7f9c1a39c700 1 --2- 172.21.15.27:0/3967277808 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c141596f0 0x7f9c140a98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.609+0000 7f9c1a39c700 1 -- 172.21.15.27:0/3967277808 >> v1:172.21.15.77:6789/0 conn(0x7f9c140a9de0 legacy=0x7f9c14068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:40.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.609+0000 7f9c1a39c700 1 -- 172.21.15.27:0/3967277808 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c140c7770 con 0x7f9c14068d30 2024-01-11T08:10:40.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.609+0000 7f9c1a39c700 1 --2- 172.21.15.27:0/3967277808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c14068d30 0x7f9c140c8530 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c009f20 tx=0x7f9c0c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c33a2b71b30efc0d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:40.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.609+0000 7f9c18b99700 1 -- 172.21.15.27:0/3967277808 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c00e070 con 0x7f9c14068d30 2024-01-11T08:10:40.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.609+0000 7f9c18b99700 1 -- 172.21.15.27:0/3967277808 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0c00ba80 con 0x7f9c14068d30 2024-01-11T08:10:40.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.609+0000 7f9c18b99700 1 -- 172.21.15.27:0/3967277808 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c0092e0 con 0x7f9c14068d30 2024-01-11T08:10:40.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.610+0000 7f9c1bdff700 1 -- 172.21.15.27:0/3967277808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c14068d30 msgr2=0x7f9c140c8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:40.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.610+0000 7f9c1bdff700 1 --2- 172.21.15.27:0/3967277808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c14068d30 0x7f9c140c8530 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c009f20 tx=0x7f9c0c009840 comp rx=0 tx=0).stop 2024-01-11T08:10:40.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.610+0000 7f9c1bdff700 1 -- 172.21.15.27:0/3967277808 shutdown_connections 2024-01-11T08:10:40.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.610+0000 7f9c1bdff700 1 --2- 172.21.15.27:0/3967277808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c14068d30 0x7f9c140c8530 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.610+0000 7f9c1bdff700 1 --2- 172.21.15.27:0/3967277808 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c141596f0 0x7f9c140a98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.610+0000 7f9c1bdff700 1 -- 172.21.15.27:0/3967277808 >> 172.21.15.27:0/3967277808 conn(0x7f9c140c9fb0 msgr2=0x7f9c1406ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:40.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.610+0000 7f9c1bdff700 1 -- 172.21.15.27:0/3967277808 shutdown_connections 2024-01-11T08:10:40.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.610+0000 7f9c1bdff700 1 -- 172.21.15.27:0/3967277808 wait complete. 2024-01-11T08:10:40.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.611+0000 7f9c1bdff700 1 Processor -- start 2024-01-11T08:10:40.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.612+0000 7f9c1bdff700 1 -- start start 2024-01-11T08:10:40.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.613+0000 7f9c1bdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c14068d30 0x7f9c1414b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:40.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.613+0000 7f9c1bdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c140a9de0 0x7f9c140ab960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:40.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.613+0000 7f9c1bdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c141596f0 0x7f9c140b1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:40.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.613+0000 7f9c1bdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140c4860 con 0x7f9c14068d30 2024-01-11T08:10:40.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.613+0000 7f9c1bdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140c3c50 con 0x7f9c141596f0 2024-01-11T08:10:40.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.613+0000 7f9c1bdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140c3f30 con 0x7f9c140a9de0 2024-01-11T08:10:40.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.613+0000 7f9c19b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c14068d30 0x7f9c1414b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:40.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.613+0000 7f9c19b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c14068d30 0x7f9c1414b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53478/0 (socket says 172.21.15.27:53478) 2024-01-11T08:10:40.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.613+0000 7f9c19b9b700 1 -- 172.21.15.27:0/561266368 learned_addr learned my addr 172.21.15.27:0/561266368 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:40.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.614+0000 7f9c1939a700 1 --2- 172.21.15.27:0/561266368 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c140a9de0 0x7f9c140ab960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:40.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.614+0000 7f9c1a39c700 1 --2- 172.21.15.27:0/561266368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c141596f0 0x7f9c140b1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:40.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.614+0000 7f9c19b9b700 1 -- 172.21.15.27:0/561266368 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c140a9de0 msgr2=0x7f9c140ab960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:40.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.614+0000 7f9c19b9b700 1 --2- 172.21.15.27:0/561266368 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c140a9de0 0x7f9c140ab960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.614+0000 7f9c19b9b700 1 -- 172.21.15.27:0/561266368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c141596f0 msgr2=0x7f9c140b1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:40.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.614+0000 7f9c19b9b700 1 --2- 172.21.15.27:0/561266368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c141596f0 0x7f9c140b1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.614+0000 7f9c19b9b700 1 -- 172.21.15.27:0/561266368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0c009ae0 con 0x7f9c14068d30 2024-01-11T08:10:40.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.614+0000 7f9c1939a700 1 --2- 172.21.15.27:0/561266368 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c140a9de0 0x7f9c140ab960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:40.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.614+0000 7f9c19b9b700 1 --2- 172.21.15.27:0/561266368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c14068d30 0x7f9c1414b3e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9c1000d860 tx=0x7f9c10008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:40.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.614+0000 7f9c09ffb700 1 -- 172.21.15.27:0/561266368 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c10009610 con 0x7f9c14068d30 2024-01-11T08:10:40.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.614+0000 7f9c09ffb700 1 -- 172.21.15.27:0/561266368 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c10009770 con 0x7f9c14068d30 2024-01-11T08:10:40.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.614+0000 7f9c1bdff700 1 -- 172.21.15.27:0/561266368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c140c4210 con 0x7f9c14068d30 2024-01-11T08:10:40.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.614+0000 7f9c09ffb700 1 -- 172.21.15.27:0/561266368 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c10009b30 con 0x7f9c14068d30 2024-01-11T08:10:40.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.614+0000 7f9c1bdff700 1 -- 172.21.15.27:0/561266368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c140b2370 con 0x7f9c14068d30 2024-01-11T08:10:40.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.615+0000 7f9c1bdff700 1 -- 172.21.15.27:0/561266368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c1404c3c0 con 0x7f9c14068d30 2024-01-11T08:10:40.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.616+0000 7f9c09ffb700 1 -- 172.21.15.27:0/561266368 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9c1001e460 con 0x7f9c14068d30 2024-01-11T08:10:40.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.617+0000 7f9c09ffb700 1 --2- 172.21.15.27:0/561266368 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9bf8066f30 0x7f9bf80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:40.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.617+0000 7f9c09ffb700 1 -- 172.21.15.27:0/561266368 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f9c10084b50 con 0x7f9c14068d30 2024-01-11T08:10:40.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.618+0000 7f9c1939a700 1 --2- 172.21.15.27:0/561266368 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9bf8066f30 0x7f9bf80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:40.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.618+0000 7f9c1939a700 1 --2- 172.21.15.27:0/561266368 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9bf8066f30 0x7f9bf80693e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9c04007fd0 tx=0x7f9c04009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:40.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.621+0000 7f9c09ffb700 1 -- 172.21.15.27:0/561266368 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9c10051dc0 con 0x7f9c14068d30 2024-01-11T08:10:40.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.825+0000 7f9c1bdff700 1 -- 172.21.15.27:0/561266368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c140c7a50 con 0x7f9c14068d30 2024-01-11T08:10:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.825+0000 7f9c09ffb700 1 -- 172.21.15.27:0/561266368 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9c100553e0 con 0x7f9c14068d30 2024-01-11T08:10:40.825 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:10:40.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.827+0000 7f9c1bdff700 1 -- 172.21.15.27:0/561266368 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9bf8066f30 msgr2=0x7f9bf80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:40.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.827+0000 7f9c1bdff700 1 --2- 172.21.15.27:0/561266368 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9bf8066f30 0x7f9bf80693e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9c04007fd0 tx=0x7f9c04009040 comp rx=0 tx=0).stop 2024-01-11T08:10:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.827+0000 7f9c1bdff700 1 -- 172.21.15.27:0/561266368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c14068d30 msgr2=0x7f9c1414b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.827+0000 7f9c1bdff700 1 --2- 172.21.15.27:0/561266368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c14068d30 0x7f9c1414b3e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9c1000d860 tx=0x7f9c10008040 comp rx=0 tx=0).stop 2024-01-11T08:10:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.827+0000 7f9c1bdff700 1 -- 172.21.15.27:0/561266368 shutdown_connections 2024-01-11T08:10:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.827+0000 7f9c1bdff700 1 --2- 172.21.15.27:0/561266368 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9bf8066f30 0x7f9bf80693e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.827+0000 7f9c1bdff700 1 --2- 172.21.15.27:0/561266368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c141596f0 0x7f9c140b1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.827+0000 7f9c1bdff700 1 --2- 172.21.15.27:0/561266368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c14068d30 0x7f9c1414b3e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.827+0000 7f9c1bdff700 1 --2- 172.21.15.27:0/561266368 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c140a9de0 0x7f9c140ab960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.827+0000 7f9c1bdff700 1 -- 172.21.15.27:0/561266368 >> 172.21.15.27:0/561266368 conn(0x7f9c140c9fb0 msgr2=0x7f9c140cac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.828+0000 7f9c1bdff700 1 -- 172.21.15.27:0/561266368 shutdown_connections 2024-01-11T08:10:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:40.828+0000 7f9c1bdff700 1 -- 172.21.15.27:0/561266368 wait complete. 2024-01-11T08:10:40.845 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T08:10:41.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.047+0000 7f12583ff700 1 Processor -- start 2024-01-11T08:10:41.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.047+0000 7f12583ff700 1 -- start start 2024-01-11T08:10:41.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.048+0000 7f12583ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1250142dc0 0x7f12500cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:41.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.048+0000 7f12583ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f125015d320 con 0x7f12500cb2e0 2024-01-11T08:10:41.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.048+0000 7f12583ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12500cb700 con 0x7f1250143190 2024-01-11T08:10:41.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.048+0000 7f12583ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f125015df30 con 0x7f1250142dc0 2024-01-11T08:10:41.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.048+0000 7f125619b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1250142dc0 0x7f12500cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:41.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.048+0000 7f125699c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f12500cb2e0 0x7f1250068720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37254/0 (socket says 172.21.15.27:37254) 2024-01-11T08:10:41.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.048+0000 7f125619b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1250142dc0 0x7f12500cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53490/0 (socket says 172.21.15.27:53490) 2024-01-11T08:10:41.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.048+0000 7f125699c700 1 -- 172.21.15.27:0/2418248884 learned_addr learned my addr 172.21.15.27:0/2418248884 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:41.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.049+0000 7f125619b700 1 -- 172.21.15.27:0/2418248884 >> v1:172.21.15.77:6789/0 conn(0x7f1250143190 legacy=0x7f12500cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:41.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.049+0000 7f125619b700 1 -- 172.21.15.27:0/2418248884 >> v1:172.21.15.27:6789/0 conn(0x7f12500cb2e0 legacy=0x7f1250068720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:41.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.049+0000 7f125619b700 1 -- 172.21.15.27:0/2418248884 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f125015d590 con 0x7f1250142dc0 2024-01-11T08:10:41.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.049+0000 7f1255199700 1 -- 172.21.15.27:0/2418248884 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1030243609 0 0) 0x7f125015d320 con 0x7f12500cb2e0 2024-01-11T08:10:41.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.049+0000 7f125619b700 1 --2- 172.21.15.27:0/2418248884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1250142dc0 0x7f12500cada0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f124c0098e0 tx=0x7f124c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7266447d669dc25e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:41.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.049+0000 7f1255199700 1 -- 172.21.15.27:0/2418248884 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124c009e80 con 0x7f1250142dc0 2024-01-11T08:10:41.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.049+0000 7f1255199700 1 -- 172.21.15.27:0/2418248884 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f124c00b860 con 0x7f1250142dc0 2024-01-11T08:10:41.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.049+0000 7f1255199700 1 -- 172.21.15.27:0/2418248884 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124c00bbe0 con 0x7f1250142dc0 2024-01-11T08:10:41.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.050+0000 7f12583ff700 1 -- 172.21.15.27:0/2418248884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1250142dc0 msgr2=0x7f12500cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:41.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.050+0000 7f12583ff700 1 --2- 172.21.15.27:0/2418248884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1250142dc0 0x7f12500cada0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f124c0098e0 tx=0x7f124c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.050+0000 7f12583ff700 1 -- 172.21.15.27:0/2418248884 shutdown_connections 2024-01-11T08:10:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.050+0000 7f12583ff700 1 --2- 172.21.15.27:0/2418248884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1250142dc0 0x7f12500cada0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.050+0000 7f12583ff700 1 -- 172.21.15.27:0/2418248884 >> 172.21.15.27:0/2418248884 conn(0x7f1250063ca0 msgr2=0x7f125006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.050+0000 7f12583ff700 1 -- 172.21.15.27:0/2418248884 shutdown_connections 2024-01-11T08:10:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.050+0000 7f12583ff700 1 -- 172.21.15.27:0/2418248884 wait complete. 2024-01-11T08:10:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.052+0000 7f12583ff700 1 Processor -- start 2024-01-11T08:10:41.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.053+0000 7f12583ff700 1 -- start start 2024-01-11T08:10:41.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.054+0000 7f12583ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12500cb2e0 0x7f12500c4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:41.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.054+0000 7f12583ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1250142dc0 0x7f12500b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:41.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.054+0000 7f12583ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1250143190 0x7f12500b3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:41.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.054+0000 7f12583ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125013db60 con 0x7f1250142dc0 2024-01-11T08:10:41.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.054+0000 7f12583ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125013cf50 con 0x7f1250143190 2024-01-11T08:10:41.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.054+0000 7f12583ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f125013d230 con 0x7f12500cb2e0 2024-01-11T08:10:41.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.054+0000 7f125599a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1250142dc0 0x7f12500b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:41.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.054+0000 7f125599a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1250142dc0 0x7f12500b2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53500/0 (socket says 172.21.15.27:53500) 2024-01-11T08:10:41.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.054+0000 7f125599a700 1 -- 172.21.15.27:0/3931915834 learned_addr learned my addr 172.21.15.27:0/3931915834 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:41.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.054+0000 7f125619b700 1 --2- 172.21.15.27:0/3931915834 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12500cb2e0 0x7f12500c4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:41.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.054+0000 7f125699c700 1 --2- 172.21.15.27:0/3931915834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1250143190 0x7f12500b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:41.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.054+0000 7f125599a700 1 -- 172.21.15.27:0/3931915834 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12500cb2e0 msgr2=0x7f12500c4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:41.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.054+0000 7f125599a700 1 --2- 172.21.15.27:0/3931915834 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12500cb2e0 0x7f12500c4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.054+0000 7f125599a700 1 -- 172.21.15.27:0/3931915834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1250143190 msgr2=0x7f12500b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:41.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.054+0000 7f125599a700 1 --2- 172.21.15.27:0/3931915834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1250143190 0x7f12500b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.054+0000 7f125599a700 1 -- 172.21.15.27:0/3931915834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124c009420 con 0x7f1250142dc0 2024-01-11T08:10:41.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.055+0000 7f125599a700 1 --2- 172.21.15.27:0/3931915834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1250142dc0 0x7f12500b2cc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f124000c510 tx=0x7f124000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:41.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.055+0000 7f12467fc700 1 -- 172.21.15.27:0/3931915834 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1240011590 con 0x7f1250142dc0 2024-01-11T08:10:41.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.055+0000 7f12467fc700 1 -- 172.21.15.27:0/3931915834 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1240011710 con 0x7f1250142dc0 2024-01-11T08:10:41.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.055+0000 7f12467fc700 1 -- 172.21.15.27:0/3931915834 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1240011ad0 con 0x7f1250142dc0 2024-01-11T08:10:41.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.055+0000 7f12583ff700 1 -- 172.21.15.27:0/3931915834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f125013d510 con 0x7f1250142dc0 2024-01-11T08:10:41.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.055+0000 7f12583ff700 1 -- 172.21.15.27:0/3931915834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12500c4dd0 con 0x7f1250142dc0 2024-01-11T08:10:41.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.056+0000 7f12583ff700 1 -- 172.21.15.27:0/3931915834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12500c4ff0 con 0x7f1250142dc0 2024-01-11T08:10:41.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.057+0000 7f12467fc700 1 -- 172.21.15.27:0/3931915834 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f124001b3b0 con 0x7f1250142dc0 2024-01-11T08:10:41.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.058+0000 7f12467fc700 1 --2- 172.21.15.27:0/3931915834 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1234066f30 0x7f12340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:41.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.058+0000 7f12467fc700 1 -- 172.21.15.27:0/3931915834 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f1240083930 con 0x7f1250142dc0 2024-01-11T08:10:41.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.058+0000 7f125619b700 1 --2- 172.21.15.27:0/3931915834 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1234066f30 0x7f12340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:41.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.059+0000 7f125619b700 1 --2- 172.21.15.27:0/3931915834 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1234066f30 0x7f12340693e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f124c0093f0 tx=0x7f124c017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:41.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.062+0000 7f12467fc700 1 -- 172.21.15.27:0/3931915834 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1240050ba0 con 0x7f1250142dc0 2024-01-11T08:10:41.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.265+0000 7f12583ff700 1 -- 172.21.15.27:0/3931915834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f125004c3c0 con 0x7f1250142dc0 2024-01-11T08:10:41.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.265+0000 7f12467fc700 1 -- 172.21.15.27:0/3931915834 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1240017020 con 0x7f1250142dc0 2024-01-11T08:10:41.266 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T08:10:41.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.268+0000 7f12583ff700 1 -- 172.21.15.27:0/3931915834 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1234066f30 msgr2=0x7f12340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:41.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.268+0000 7f12583ff700 1 --2- 172.21.15.27:0/3931915834 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1234066f30 0x7f12340693e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f124c0093f0 tx=0x7f124c017000 comp rx=0 tx=0).stop 2024-01-11T08:10:41.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.268+0000 7f12583ff700 1 -- 172.21.15.27:0/3931915834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1250142dc0 msgr2=0x7f12500b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:41.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.268+0000 7f12583ff700 1 --2- 172.21.15.27:0/3931915834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1250142dc0 0x7f12500b2cc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f124000c510 tx=0x7f124000c820 comp rx=0 tx=0).stop 2024-01-11T08:10:41.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.268+0000 7f12583ff700 1 -- 172.21.15.27:0/3931915834 shutdown_connections 2024-01-11T08:10:41.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.268+0000 7f12583ff700 1 --2- 172.21.15.27:0/3931915834 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1234066f30 0x7f12340693e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.268+0000 7f12583ff700 1 --2- 172.21.15.27:0/3931915834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1250143190 0x7f12500b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.268+0000 7f12583ff700 1 --2- 172.21.15.27:0/3931915834 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12500cb2e0 0x7f12500c4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.268+0000 7f12583ff700 1 --2- 172.21.15.27:0/3931915834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1250142dc0 0x7f12500b2cc0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.268+0000 7f12583ff700 1 -- 172.21.15.27:0/3931915834 >> 172.21.15.27:0/3931915834 conn(0x7f1250063ca0 msgr2=0x7f125015ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:41.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.268+0000 7f12583ff700 1 -- 172.21.15.27:0/3931915834 shutdown_connections 2024-01-11T08:10:41.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.268+0000 7f12583ff700 1 -- 172.21.15.27:0/3931915834 wait complete. 2024-01-11T08:10:41.286 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T08:10:41.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.489+0000 7f3431dff700 1 Processor -- start 2024-01-11T08:10:41.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.489+0000 7f3431dff700 1 -- start start 2024-01-11T08:10:41.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.490+0000 7f3431dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f342c15a1b0 0x7f342c15a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:41.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.490+0000 7f3431dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f342c0cf850 con 0x7f342c0ccd70 2024-01-11T08:10:41.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.490+0000 7f3431dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f342c0cec40 con 0x7f342c15ae70 2024-01-11T08:10:41.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.490+0000 7f3431dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f342c0cef20 con 0x7f342c15a1b0 2024-01-11T08:10:41.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.491+0000 7f342affd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f342c0ccd70 0x7f342c0cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37256/0 (socket says 172.21.15.27:37256) 2024-01-11T08:10:41.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.491+0000 7f342affd700 1 -- 172.21.15.27:0/2389918295 learned_addr learned my addr 172.21.15.27:0/2389918295 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:41.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.491+0000 7f342b7fe700 1 --2- 172.21.15.27:0/2389918295 >> v2:172.21.15.77:3300/0 conn(0x7f342c15a1b0 0x7f342c15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:41.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.491+0000 7f342a7fc700 1 -- 172.21.15.27:0/2389918295 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380275106 0 0) 0x7f342c0cf850 con 0x7f342c0ccd70 2024-01-11T08:10:41.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.491+0000 7f342a7fc700 1 -- 172.21.15.27:0/2389918295 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3414003570 con 0x7f342c0ccd70 2024-01-11T08:10:41.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.491+0000 7f342a7fc700 1 -- 172.21.15.27:0/2389918295 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433018794 0 0) 0x7f342c0cec40 con 0x7f342c15ae70 2024-01-11T08:10:41.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.491+0000 7f342a7fc700 1 -- 172.21.15.27:0/2389918295 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f342c0cf850 con 0x7f342c15ae70 2024-01-11T08:10:41.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.491+0000 7f342b7fe700 1 -- 172.21.15.27:0/2389918295 >> v1:172.21.15.77:6789/0 conn(0x7f342c15ae70 legacy=0x7f342c15b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:41.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.491+0000 7f342b7fe700 1 -- 172.21.15.27:0/2389918295 >> v1:172.21.15.27:6789/0 conn(0x7f342c0ccd70 legacy=0x7f342c0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:41.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.491+0000 7f342a7fc700 1 -- 172.21.15.27:0/2389918295 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2976166513 0 0) 0x7f3414003570 con 0x7f342c0ccd70 2024-01-11T08:10:41.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.491+0000 7f342b7fe700 1 -- 172.21.15.27:0/2389918295 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342c0cf200 con 0x7f342c15a1b0 2024-01-11T08:10:41.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.492+0000 7f342b7fe700 1 --2- 172.21.15.27:0/2389918295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f342c15a1b0 0x7f342c15a580 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f341c0098e0 tx=0x7f341c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d72f7f7381f61d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:41.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.492+0000 7f342a7fc700 1 -- 172.21.15.27:0/2389918295 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3420001470 con 0x7f342c15a1b0 2024-01-11T08:10:41.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.492+0000 7f342a7fc700 1 -- 172.21.15.27:0/2389918295 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341c011420 con 0x7f342c15a1b0 2024-01-11T08:10:41.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.492+0000 7f342a7fc700 1 -- 172.21.15.27:0/2389918295 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c0117c0 con 0x7f342c15a1b0 2024-01-11T08:10:41.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.492+0000 7f3431dff700 1 -- 172.21.15.27:0/2389918295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f342c15a1b0 msgr2=0x7f342c15a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:41.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.492+0000 7f3431dff700 1 --2- 172.21.15.27:0/2389918295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f342c15a1b0 0x7f342c15a580 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f341c0098e0 tx=0x7f341c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.492+0000 7f3431dff700 1 -- 172.21.15.27:0/2389918295 shutdown_connections 2024-01-11T08:10:41.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.493+0000 7f3431dff700 1 --2- 172.21.15.27:0/2389918295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f342c15a1b0 0x7f342c15a580 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.493+0000 7f3431dff700 1 -- 172.21.15.27:0/2389918295 >> 172.21.15.27:0/2389918295 conn(0x7f342c06ed40 msgr2=0x7f342c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:41.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.493+0000 7f3431dff700 1 -- 172.21.15.27:0/2389918295 shutdown_connections 2024-01-11T08:10:41.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.493+0000 7f3431dff700 1 -- 172.21.15.27:0/2389918295 wait complete. 2024-01-11T08:10:41.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.494+0000 7f3431dff700 1 Processor -- start 2024-01-11T08:10:41.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.494+0000 7f3431dff700 1 -- start start 2024-01-11T08:10:41.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.495+0000 7f3431dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f342c0ccd70 0x7f342c0ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:41.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.495+0000 7f3431dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f342c15a1b0 0x7f342c158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:41.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.495+0000 7f3431dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f342c15ae70 0x7f342c14e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.495+0000 7f3431dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342c0c9950 con 0x7f342c15a1b0 2024-01-11T08:10:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.495+0000 7f3431dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342c0a7920 con 0x7f342c15ae70 2024-01-11T08:10:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.495+0000 7f3431dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f342c0a7ea0 con 0x7f342c0ccd70 2024-01-11T08:10:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.495+0000 7f342affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f342c15a1b0 0x7f342c158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.496+0000 7f342affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f342c15a1b0 0x7f342c158140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53504/0 (socket says 172.21.15.27:53504) 2024-01-11T08:10:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.496+0000 7f342affd700 1 -- 172.21.15.27:0/2247263262 learned_addr learned my addr 172.21.15.27:0/2247263262 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.496+0000 7f342b7fe700 1 --2- 172.21.15.27:0/2247263262 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f342c0ccd70 0x7f342c0ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.496+0000 7f342bfff700 1 --2- 172.21.15.27:0/2247263262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f342c15ae70 0x7f342c14e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.496+0000 7f342affd700 1 -- 172.21.15.27:0/2247263262 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f342c0ccd70 msgr2=0x7f342c0ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.496+0000 7f342affd700 1 --2- 172.21.15.27:0/2247263262 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f342c0ccd70 0x7f342c0ca1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.496+0000 7f342affd700 1 -- 172.21.15.27:0/2247263262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f342c15ae70 msgr2=0x7f342c14e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.496+0000 7f342affd700 1 --2- 172.21.15.27:0/2247263262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f342c15ae70 0x7f342c14e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.496+0000 7f342affd700 1 -- 172.21.15.27:0/2247263262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341c009420 con 0x7f342c15a1b0 2024-01-11T08:10:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.496+0000 7f342affd700 1 --2- 172.21.15.27:0/2247263262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f342c15a1b0 0x7f342c158140 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f342000c7f0 tx=0x7f3420001a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:41.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.496+0000 7f34137fe700 1 -- 172.21.15.27:0/2247263262 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3420007360 con 0x7f342c15a1b0 2024-01-11T08:10:41.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.497+0000 7f34137fe700 1 -- 172.21.15.27:0/2247263262 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34200074c0 con 0x7f342c15a1b0 2024-01-11T08:10:41.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.497+0000 7f3431dff700 1 -- 172.21.15.27:0/2247263262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f342c0b60f0 con 0x7f342c15a1b0 2024-01-11T08:10:41.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.497+0000 7f34137fe700 1 -- 172.21.15.27:0/2247263262 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3420004830 con 0x7f342c15a1b0 2024-01-11T08:10:41.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.497+0000 7f3431dff700 1 -- 172.21.15.27:0/2247263262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f342c0c95e0 con 0x7f342c15a1b0 2024-01-11T08:10:41.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.498+0000 7f3431dff700 1 -- 172.21.15.27:0/2247263262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f342c04c3c0 con 0x7f342c15a1b0 2024-01-11T08:10:41.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.499+0000 7f34137fe700 1 -- 172.21.15.27:0/2247263262 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3420004c20 con 0x7f342c15a1b0 2024-01-11T08:10:41.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.499+0000 7f34137fe700 1 --2- 172.21.15.27:0/2247263262 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3418066f30 0x7f34180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:41.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.499+0000 7f34137fe700 1 -- 172.21.15.27:0/2247263262 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f3420083de0 con 0x7f342c15a1b0 2024-01-11T08:10:41.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.500+0000 7f342b7fe700 1 --2- 172.21.15.27:0/2247263262 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3418066f30 0x7f34180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:41.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.500+0000 7f342b7fe700 1 --2- 172.21.15.27:0/2247263262 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3418066f30 0x7f34180693e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f341c0093f0 tx=0x7f341c016000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:41.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.504+0000 7f34137fe700 1 -- 172.21.15.27:0/2247263262 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3420051030 con 0x7f342c15a1b0 2024-01-11T08:10:41.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.706+0000 7f3431dff700 1 -- 172.21.15.27:0/2247263262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f342c0caf90 con 0x7f342c15a1b0 2024-01-11T08:10:41.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.707+0000 7f34137fe700 1 -- 172.21.15.27:0/2247263262 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3420018020 con 0x7f342c15a1b0 2024-01-11T08:10:41.707 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T08:10:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.709+0000 7f3431dff700 1 -- 172.21.15.27:0/2247263262 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3418066f30 msgr2=0x7f34180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:41.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.709+0000 7f3431dff700 1 --2- 172.21.15.27:0/2247263262 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3418066f30 0x7f34180693e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f341c0093f0 tx=0x7f341c016000 comp rx=0 tx=0).stop 2024-01-11T08:10:41.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.709+0000 7f3431dff700 1 -- 172.21.15.27:0/2247263262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f342c15a1b0 msgr2=0x7f342c158140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:41.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.709+0000 7f3431dff700 1 --2- 172.21.15.27:0/2247263262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f342c15a1b0 0x7f342c158140 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f342000c7f0 tx=0x7f3420001a10 comp rx=0 tx=0).stop 2024-01-11T08:10:41.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.709+0000 7f3431dff700 1 -- 172.21.15.27:0/2247263262 shutdown_connections 2024-01-11T08:10:41.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.709+0000 7f3431dff700 1 --2- 172.21.15.27:0/2247263262 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3418066f30 0x7f34180693e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.709+0000 7f3431dff700 1 --2- 172.21.15.27:0/2247263262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f342c15ae70 0x7f342c14e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.709+0000 7f3431dff700 1 --2- 172.21.15.27:0/2247263262 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f342c0ccd70 0x7f342c0ca1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.709+0000 7f3431dff700 1 --2- 172.21.15.27:0/2247263262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f342c15a1b0 0x7f342c158140 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.709+0000 7f3431dff700 1 -- 172.21.15.27:0/2247263262 >> 172.21.15.27:0/2247263262 conn(0x7f342c06ed40 msgr2=0x7f342c0c5480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:41.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.709+0000 7f3431dff700 1 -- 172.21.15.27:0/2247263262 shutdown_connections 2024-01-11T08:10:41.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.709+0000 7f3431dff700 1 -- 172.21.15.27:0/2247263262 wait complete. 2024-01-11T08:10:41.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:10:41.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.928+0000 7f6b925ff700 1 Processor -- start 2024-01-11T08:10:41.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.928+0000 7f6b925ff700 1 -- start start 2024-01-11T08:10:41.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.929+0000 7f6b925ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6b8c1596c0 0x7f6b8c15d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:41.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.929+0000 7f6b925ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6b8c0ca670 0x7f6b8c068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:41.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.929+0000 7f6b925ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b8c068fc0 con 0x7f6b8c068b60 2024-01-11T08:10:41.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.929+0000 7f6b925ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b8c0a9cb0 con 0x7f6b8c1596c0 2024-01-11T08:10:41.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.929+0000 7f6b925ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b8c15e210 con 0x7f6b8c0ca670 2024-01-11T08:10:41.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.930+0000 7f6b8bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6b8c1596c0 0x7f6b8c15d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:41.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.930+0000 7f6b8bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6b8c1596c0 0x7f6b8c15d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53516/0 (socket says 172.21.15.27:53516) 2024-01-11T08:10:41.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.930+0000 7f6b8bfff700 1 -- 172.21.15.27:0/317271141 learned_addr learned my addr 172.21.15.27:0/317271141 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:41.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.930+0000 7f6b8b7fe700 1 --2- 172.21.15.27:0/317271141 >> v2:172.21.15.77:3300/0 conn(0x7f6b8c0ca670 0x7f6b8c068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:41.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.930+0000 7f6b8bfff700 1 -- 172.21.15.27:0/317271141 >> v2:172.21.15.77:3300/0 conn(0x7f6b8c0ca670 msgr2=0x7f6b8c068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:41.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.930+0000 7f6b8bfff700 1 --2- 172.21.15.27:0/317271141 >> v2:172.21.15.77:3300/0 conn(0x7f6b8c0ca670 0x7f6b8c068620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.930+0000 7f6b8bfff700 1 -- 172.21.15.27:0/317271141 >> v1:172.21.15.77:6789/0 conn(0x7f6b8c068b60 legacy=0x7f6b8c159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:41.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.930+0000 7f6b8bfff700 1 -- 172.21.15.27:0/317271141 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8c15dc50 con 0x7f6b8c1596c0 2024-01-11T08:10:41.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.930+0000 7f6b8bfff700 1 --2- 172.21.15.27:0/317271141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b8c1596c0 0x7f6b8c15d300 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c0b2c00 tx=0x7f6b7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c73de39e46fec2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.931+0000 7f6b8affd700 1 -- 172.21.15.27:0/317271141 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c009e70 con 0x7f6b8c1596c0 2024-01-11T08:10:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.931+0000 7f6b8affd700 1 -- 172.21.15.27:0/317271141 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7c00b860 con 0x7f6b8c1596c0 2024-01-11T08:10:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.931+0000 7f6b8affd700 1 -- 172.21.15.27:0/317271141 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c00bc00 con 0x7f6b8c1596c0 2024-01-11T08:10:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.931+0000 7f6b925ff700 1 -- 172.21.15.27:0/317271141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b8c1596c0 msgr2=0x7f6b8c15d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.931+0000 7f6b925ff700 1 --2- 172.21.15.27:0/317271141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b8c1596c0 0x7f6b8c15d300 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c0b2c00 tx=0x7f6b7c009250 comp rx=0 tx=0).stop 2024-01-11T08:10:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.931+0000 7f6b925ff700 1 -- 172.21.15.27:0/317271141 shutdown_connections 2024-01-11T08:10:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.931+0000 7f6b925ff700 1 --2- 172.21.15.27:0/317271141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b8c1596c0 0x7f6b8c15d300 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.931+0000 7f6b925ff700 1 --2- 172.21.15.27:0/317271141 >> v2:172.21.15.77:3300/0 conn(0x7f6b8c0ca670 0x7f6b8c068620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.931+0000 7f6b925ff700 1 -- 172.21.15.27:0/317271141 >> 172.21.15.27:0/317271141 conn(0x7f6b8c0c9f70 msgr2=0x7f6b8c06deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.931+0000 7f6b925ff700 1 -- 172.21.15.27:0/317271141 shutdown_connections 2024-01-11T08:10:41.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.931+0000 7f6b925ff700 1 -- 172.21.15.27:0/317271141 wait complete. 2024-01-11T08:10:41.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.933+0000 7f6b925ff700 1 Processor -- start 2024-01-11T08:10:41.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.933+0000 7f6b925ff700 1 -- start start 2024-01-11T08:10:41.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.934+0000 7f6b925ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b8c068b60 0x7f6b8c0c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:41.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.934+0000 7f6b925ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6b8c0ca670 0x7f6b8c156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:41.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.934+0000 7f6b925ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b8c1596c0 0x7f6b8c0c23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:41.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.934+0000 7f6b925ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c0c8d70 con 0x7f6b8c068b60 2024-01-11T08:10:41.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.934+0000 7f6b925ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c0b0d80 con 0x7f6b8c1596c0 2024-01-11T08:10:41.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.934+0000 7f6b925ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c0c7ee0 con 0x7f6b8c0ca670 2024-01-11T08:10:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.934+0000 7f6b8bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b8c068b60 0x7f6b8c0c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.934+0000 7f6b8bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b8c068b60 0x7f6b8c0c79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53530/0 (socket says 172.21.15.27:53530) 2024-01-11T08:10:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.934+0000 7f6b8bfff700 1 -- 172.21.15.27:0/4167115152 learned_addr learned my addr 172.21.15.27:0/4167115152 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.935+0000 7f6b8bfff700 1 -- 172.21.15.27:0/4167115152 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6b8c0ca670 msgr2=0x7f6b8c156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.935+0000 7f6b8bfff700 1 --2- 172.21.15.27:0/4167115152 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6b8c0ca670 0x7f6b8c156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.935+0000 7f6b90b9c700 1 --2- 172.21.15.27:0/4167115152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b8c1596c0 0x7f6b8c0c23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.935+0000 7f6b8bfff700 1 -- 172.21.15.27:0/4167115152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b8c1596c0 msgr2=0x7f6b8c0c23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.935+0000 7f6b8bfff700 1 --2- 172.21.15.27:0/4167115152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b8c1596c0 0x7f6b8c0c23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.935+0000 7f6b8bfff700 1 -- 172.21.15.27:0/4167115152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7c0094f0 con 0x7f6b8c068b60 2024-01-11T08:10:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.935+0000 7f6b90b9c700 1 --2- 172.21.15.27:0/4167115152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b8c1596c0 0x7f6b8c0c23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.935+0000 7f6b8bfff700 1 --2- 172.21.15.27:0/4167115152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b8c068b60 0x7f6b8c0c79a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c006fd0 tx=0x7f6b7c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.935+0000 7f6b73fff700 1 -- 172.21.15.27:0/4167115152 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c011420 con 0x7f6b8c068b60 2024-01-11T08:10:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.935+0000 7f6b73fff700 1 -- 172.21.15.27:0/4167115152 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7c0115a0 con 0x7f6b8c068b60 2024-01-11T08:10:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.935+0000 7f6b73fff700 1 -- 172.21.15.27:0/4167115152 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c011960 con 0x7f6b8c068b60 2024-01-11T08:10:41.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.935+0000 7f6b925ff700 1 -- 172.21.15.27:0/4167115152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b8c0afea0 con 0x7f6b8c068b60 2024-01-11T08:10:41.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.935+0000 7f6b925ff700 1 -- 172.21.15.27:0/4167115152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b8c0b04e0 con 0x7f6b8c068b60 2024-01-11T08:10:41.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.936+0000 7f6b925ff700 1 -- 172.21.15.27:0/4167115152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b8c0718d0 con 0x7f6b8c068b60 2024-01-11T08:10:41.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.937+0000 7f6b73fff700 1 -- 172.21.15.27:0/4167115152 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6b7c0253f0 con 0x7f6b8c068b60 2024-01-11T08:10:41.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.938+0000 7f6b73fff700 1 --2- 172.21.15.27:0/4167115152 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6b78066f30 0x7f6b780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:41.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.938+0000 7f6b73fff700 1 -- 172.21.15.27:0/4167115152 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f6b7c085ef0 con 0x7f6b8c068b60 2024-01-11T08:10:41.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.939+0000 7f6b8b7fe700 1 --2- 172.21.15.27:0/4167115152 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6b78066f30 0x7f6b780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:41.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.939+0000 7f6b8b7fe700 1 --2- 172.21.15.27:0/4167115152 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6b78066f30 0x7f6b780693e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c067ba0 tx=0x7f6b800091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:41.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:41.942+0000 7f6b73fff700 1 -- 172.21.15.27:0/4167115152 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6b7c0530c0 con 0x7f6b8c068b60 2024-01-11T08:10:42.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.187+0000 7f6b925ff700 1 -- 172.21.15.27:0/4167115152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b8c15df30 con 0x7f6b8c068b60 2024-01-11T08:10:42.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.188+0000 7f6b73fff700 1 -- 172.21.15.27:0/4167115152 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4588 (secure 0 0 0) 0x7f6b8c15df30 con 0x7f6b8c068b60 2024-01-11T08:10:42.188 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:42.188 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3168764915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3168764915},{"type":"v1","addr":"172.21.15.27:6834","nonce":3168764915}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2607167395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2607167395},{"type":"v1","addr":"172.21.15.27:6835","nonce":2607167395}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2123423970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2123423970},{"type":"v1","addr":"172.21.15.77:6837","nonce":2123423970}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:10:30.160827+0000","modified":"2024-01-11T08:10:33.681210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6836/269464429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":269464429},{"type":"v1","addr":"172.21.15.77:6836","nonce":269464429}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2024-01-11T08:10:42.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.190+0000 7f6b925ff700 1 -- 172.21.15.27:0/4167115152 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6b78066f30 msgr2=0x7f6b780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:42.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.190+0000 7f6b925ff700 1 --2- 172.21.15.27:0/4167115152 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6b78066f30 0x7f6b780693e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c067ba0 tx=0x7f6b800091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:42.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.190+0000 7f6b925ff700 1 -- 172.21.15.27:0/4167115152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b8c068b60 msgr2=0x7f6b8c0c79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.190+0000 7f6b925ff700 1 --2- 172.21.15.27:0/4167115152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b8c068b60 0x7f6b8c0c79a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c006fd0 tx=0x7f6b7c017000 comp rx=0 tx=0).stop 2024-01-11T08:10:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.190+0000 7f6b925ff700 1 -- 172.21.15.27:0/4167115152 shutdown_connections 2024-01-11T08:10:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.190+0000 7f6b925ff700 1 --2- 172.21.15.27:0/4167115152 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6b78066f30 0x7f6b780693e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.190+0000 7f6b925ff700 1 --2- 172.21.15.27:0/4167115152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b8c1596c0 0x7f6b8c0c23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.190+0000 7f6b925ff700 1 --2- 172.21.15.27:0/4167115152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b8c068b60 0x7f6b8c0c79a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.190+0000 7f6b925ff700 1 --2- 172.21.15.27:0/4167115152 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6b8c0ca670 0x7f6b8c156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.190+0000 7f6b925ff700 1 -- 172.21.15.27:0/4167115152 >> 172.21.15.27:0/4167115152 conn(0x7f6b8c0c9f70 msgr2=0x7f6b8c0680f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.191+0000 7f6b925ff700 1 -- 172.21.15.27:0/4167115152 shutdown_connections 2024-01-11T08:10:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.191+0000 7f6b925ff700 1 -- 172.21.15.27:0/4167115152 wait complete. 2024-01-11T08:10:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 8 2024-01-11T08:10:42.208 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:10:42.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.409+0000 7fa6f3dff700 1 Processor -- start 2024-01-11T08:10:42.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.409+0000 7fa6f3dff700 1 -- start start 2024-01-11T08:10:42.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.410+0000 7fa6f3dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6ec0b2b90 0x7fa6ec159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:42.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.410+0000 7fa6f3dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ec0cd730 con 0x7fa6ec0b03e0 2024-01-11T08:10:42.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.410+0000 7fa6f3dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ec0ccb20 con 0x7fa6ec0b07b0 2024-01-11T08:10:42.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.410+0000 7fa6f3dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec0cce00 con 0x7fa6ec0b2b90 2024-01-11T08:10:42.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.410+0000 7fa6f1b9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa6ec0b03e0 0x7fa6ec0b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37262/0 (socket says 172.21.15.27:37262) 2024-01-11T08:10:42.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.410+0000 7fa6f1b9b700 1 -- 172.21.15.27:0/3585564192 learned_addr learned my addr 172.21.15.27:0/3585564192 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:42.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.410+0000 7fa6f139a700 1 --2- 172.21.15.27:0/3585564192 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6ec0b2b90 0x7fa6ec159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:42.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.411+0000 7fa6f0b99700 1 -- 172.21.15.27:0/3585564192 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140829765 0 0) 0x7fa6ec0cd730 con 0x7fa6ec0b03e0 2024-01-11T08:10:42.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.411+0000 7fa6f0b99700 1 -- 172.21.15.27:0/3585564192 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d8003570 con 0x7fa6ec0b03e0 2024-01-11T08:10:42.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.411+0000 7fa6f0b99700 1 -- 172.21.15.27:0/3585564192 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518223932 0 0) 0x7fa6ec0ccb20 con 0x7fa6ec0b07b0 2024-01-11T08:10:42.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.411+0000 7fa6f0b99700 1 -- 172.21.15.27:0/3585564192 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6ec0cd730 con 0x7fa6ec0b07b0 2024-01-11T08:10:42.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.411+0000 7fa6f139a700 1 -- 172.21.15.27:0/3585564192 >> v1:172.21.15.77:6789/0 conn(0x7fa6ec0b07b0 legacy=0x7fa6ec159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:42.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.411+0000 7fa6f139a700 1 -- 172.21.15.27:0/3585564192 >> v1:172.21.15.27:6789/0 conn(0x7fa6ec0b03e0 legacy=0x7fa6ec0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:42.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.411+0000 7fa6f139a700 1 -- 172.21.15.27:0/3585564192 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ec0cd0e0 con 0x7fa6ec0b2b90 2024-01-11T08:10:42.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.411+0000 7fa6f139a700 1 --2- 172.21.15.27:0/3585564192 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6ec0b2b90 0x7fa6ec159200 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc009910 tx=0x7fa6dc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3358d5034cdcfc9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:42.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.411+0000 7fa6f0b99700 1 -- 172.21.15.27:0/3585564192 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc011630 con 0x7fa6ec0b2b90 2024-01-11T08:10:42.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.411+0000 7fa6f0b99700 1 -- 172.21.15.27:0/3585564192 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6dc011790 con 0x7fa6ec0b2b90 2024-01-11T08:10:42.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.412+0000 7fa6f0b99700 1 -- 172.21.15.27:0/3585564192 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc011b10 con 0x7fa6ec0b2b90 2024-01-11T08:10:42.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.412+0000 7fa6f3dff700 1 -- 172.21.15.27:0/3585564192 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6ec0b2b90 msgr2=0x7fa6ec159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:42.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.412+0000 7fa6f3dff700 1 --2- 172.21.15.27:0/3585564192 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6ec0b2b90 0x7fa6ec159200 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc009910 tx=0x7fa6dc0091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:42.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.412+0000 7fa6f3dff700 1 -- 172.21.15.27:0/3585564192 shutdown_connections 2024-01-11T08:10:42.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.412+0000 7fa6f3dff700 1 --2- 172.21.15.27:0/3585564192 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6ec0b2b90 0x7fa6ec159200 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:42.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.412+0000 7fa6f3dff700 1 -- 172.21.15.27:0/3585564192 >> 172.21.15.27:0/3585564192 conn(0x7fa6ec06ed40 msgr2=0x7fa6ec06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:42.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.412+0000 7fa6f3dff700 1 -- 172.21.15.27:0/3585564192 shutdown_connections 2024-01-11T08:10:42.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.412+0000 7fa6f3dff700 1 -- 172.21.15.27:0/3585564192 wait complete. 2024-01-11T08:10:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.414+0000 7fa6f3dff700 1 Processor -- start 2024-01-11T08:10:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.414+0000 7fa6f3dff700 1 -- start start 2024-01-11T08:10:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.415+0000 7fa6f3dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6ec0b03e0 0x7fa6ec0c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.415+0000 7fa6f3dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6ec0b07b0 0x7fa6ec0cf860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.415+0000 7fa6f3dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6ec0b2b90 0x7fa6ec16a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.415+0000 7fa6f3dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec0c87b0 con 0x7fa6ec0b07b0 2024-01-11T08:10:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.415+0000 7fa6f3dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec0cfda0 con 0x7fa6ec0b2b90 2024-01-11T08:10:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.415+0000 7fa6f3dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec0c7ba0 con 0x7fa6ec0b03e0 2024-01-11T08:10:42.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.415+0000 7fa6f139a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6ec0b07b0 0x7fa6ec0cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:42.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.415+0000 7fa6f1b9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6ec0b03e0 0x7fa6ec0c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:42.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.415+0000 7fa6f139a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6ec0b07b0 0x7fa6ec0cf860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53538/0 (socket says 172.21.15.27:53538) 2024-01-11T08:10:42.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.415+0000 7fa6f139a700 1 -- 172.21.15.27:0/77144780 learned_addr learned my addr 172.21.15.27:0/77144780 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:42.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.415+0000 7fa6f239c700 1 --2- 172.21.15.27:0/77144780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6ec0b2b90 0x7fa6ec16a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:42.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.415+0000 7fa6f139a700 1 -- 172.21.15.27:0/77144780 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6ec0b03e0 msgr2=0x7fa6ec0c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:42.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.415+0000 7fa6f139a700 1 --2- 172.21.15.27:0/77144780 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6ec0b03e0 0x7fa6ec0c75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:42.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.415+0000 7fa6f139a700 1 -- 172.21.15.27:0/77144780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6ec0b2b90 msgr2=0x7fa6ec16a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:42.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.415+0000 7fa6f139a700 1 --2- 172.21.15.27:0/77144780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6ec0b2b90 0x7fa6ec16a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:42.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.415+0000 7fa6f139a700 1 -- 172.21.15.27:0/77144780 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6dc009420 con 0x7fa6ec0b07b0 2024-01-11T08:10:42.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.416+0000 7fa6f139a700 1 --2- 172.21.15.27:0/77144780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6ec0b07b0 0x7fa6ec0cf860 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec069540 tx=0x7fa6dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:42.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.416+0000 7fa6e1ffb700 1 -- 172.21.15.27:0/77144780 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc000c90 con 0x7fa6ec0b07b0 2024-01-11T08:10:42.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.416+0000 7fa6e1ffb700 1 -- 172.21.15.27:0/77144780 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6dc011900 con 0x7fa6ec0b07b0 2024-01-11T08:10:42.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.416+0000 7fa6e1ffb700 1 -- 172.21.15.27:0/77144780 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc026660 con 0x7fa6ec0b07b0 2024-01-11T08:10:42.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.416+0000 7fa6f3dff700 1 -- 172.21.15.27:0/77144780 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ec0c7e80 con 0x7fa6ec0b07b0 2024-01-11T08:10:42.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.416+0000 7fa6f3dff700 1 -- 172.21.15.27:0/77144780 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ec0c8440 con 0x7fa6ec0b07b0 2024-01-11T08:10:42.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.417+0000 7fa6f3dff700 1 -- 172.21.15.27:0/77144780 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ec0c8960 con 0x7fa6ec0b07b0 2024-01-11T08:10:42.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.418+0000 7fa6e1ffb700 1 -- 172.21.15.27:0/77144780 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa6dc023030 con 0x7fa6ec0b07b0 2024-01-11T08:10:42.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.419+0000 7fa6e1ffb700 1 --2- 172.21.15.27:0/77144780 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa6d0066f30 0x7fa6d00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:42.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.419+0000 7fa6e1ffb700 1 -- 172.21.15.27:0/77144780 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa6dc085d00 con 0x7fa6ec0b07b0 2024-01-11T08:10:42.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.419+0000 7fa6f1b9b700 1 --2- 172.21.15.27:0/77144780 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa6d0066f30 0x7fa6d00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:42.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.420+0000 7fa6f1b9b700 1 --2- 172.21.15.27:0/77144780 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa6d0066f30 0x7fa6d00693e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8007fd0 tx=0x7fa6e8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:42.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.423+0000 7fa6e1ffb700 1 -- 172.21.15.27:0/77144780 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa6ec0c8960 con 0x7fa6ec0b07b0 2024-01-11T08:10:42.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.629+0000 7fa6f3dff700 1 -- 172.21.15.27:0/77144780 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6ec0c8960 con 0x7fa6ec0b07b0 2024-01-11T08:10:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.629+0000 7fa6e1ffb700 1 -- 172.21.15.27:0/77144780 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15302 (secure 0 0 0) 0x7fa6ec0c8960 con 0x7fa6ec0b07b0 2024-01-11T08:10:42.629 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:42.630 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:31.464323+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:10:27.993251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-11T08:10:29.153959+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:10:42.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.632+0000 7fa6f3dff700 1 -- 172.21.15.27:0/77144780 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa6d0066f30 msgr2=0x7fa6d00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:42.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.632+0000 7fa6f3dff700 1 --2- 172.21.15.27:0/77144780 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa6d0066f30 0x7fa6d00693e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8007fd0 tx=0x7fa6e8009040 comp rx=0 tx=0).stop 2024-01-11T08:10:42.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.632+0000 7fa6f3dff700 1 -- 172.21.15.27:0/77144780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6ec0b07b0 msgr2=0x7fa6ec0cf860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:42.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.632+0000 7fa6f3dff700 1 --2- 172.21.15.27:0/77144780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6ec0b07b0 0x7fa6ec0cf860 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec069540 tx=0x7fa6dc017000 comp rx=0 tx=0).stop 2024-01-11T08:10:42.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.632+0000 7fa6f3dff700 1 -- 172.21.15.27:0/77144780 shutdown_connections 2024-01-11T08:10:42.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.632+0000 7fa6f3dff700 1 --2- 172.21.15.27:0/77144780 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa6d0066f30 0x7fa6d00693e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:42.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.632+0000 7fa6f3dff700 1 --2- 172.21.15.27:0/77144780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6ec0b2b90 0x7fa6ec16a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:42.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.632+0000 7fa6f3dff700 1 --2- 172.21.15.27:0/77144780 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6ec0b03e0 0x7fa6ec0c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:42.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.632+0000 7fa6f3dff700 1 --2- 172.21.15.27:0/77144780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6ec0b07b0 0x7fa6ec0cf860 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:42.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.632+0000 7fa6f3dff700 1 -- 172.21.15.27:0/77144780 >> 172.21.15.27:0/77144780 conn(0x7fa6ec06ed40 msgr2=0x7fa6ec1596d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:42.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.632+0000 7fa6f3dff700 1 -- 172.21.15.27:0/77144780 shutdown_connections 2024-01-11T08:10:42.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.632+0000 7fa6f3dff700 1 -- 172.21.15.27:0/77144780 wait complete. 2024-01-11T08:10:42.650 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T08:10:27.993251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T08:10:29.153959+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T08:10:42.650 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:10:42.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.852+0000 7faaa53ff700 1 Processor -- start 2024-01-11T08:10:42.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.852+0000 7faaa53ff700 1 -- start start 2024-01-11T08:10:42.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.853+0000 7faaa53ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faaa015a1b0 0x7faaa015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:42.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.853+0000 7faaa53ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa00cf850 con 0x7faaa015ae70 2024-01-11T08:10:42.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.853+0000 7faaa53ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa00cec40 con 0x7faaa00ccd70 2024-01-11T08:10:42.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.853+0000 7faaa53ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaa00cef20 con 0x7faaa015a1b0 2024-01-11T08:10:42.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.853+0000 7faa9effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faaa015a1b0 0x7faaa015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:42.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.854+0000 7faa9effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faaa015a1b0 0x7faaa015a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53548/0 (socket says 172.21.15.27:53548) 2024-01-11T08:10:42.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.854+0000 7faa9effd700 1 -- 172.21.15.27:0/4120926203 learned_addr learned my addr 172.21.15.27:0/4120926203 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:42.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.854+0000 7faa9f7fe700 1 --1- 172.21.15.27:0/4120926203 >> v1:172.21.15.27:6789/0 conn(0x7faaa015ae70 0x7faaa015b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37264/0 (socket says 172.21.15.27:37264) 2024-01-11T08:10:42.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.854+0000 7faa9effd700 1 -- 172.21.15.27:0/4120926203 >> v1:172.21.15.77:6789/0 conn(0x7faaa00ccd70 legacy=0x7faaa00cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:42.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.854+0000 7faa9effd700 1 -- 172.21.15.27:0/4120926203 >> v1:172.21.15.27:6789/0 conn(0x7faaa015ae70 legacy=0x7faaa015b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:42.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.854+0000 7faa9effd700 1 -- 172.21.15.27:0/4120926203 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa00cf200 con 0x7faaa015a1b0 2024-01-11T08:10:42.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.854+0000 7faa9effd700 1 --2- 172.21.15.27:0/4120926203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faaa015a1b0 0x7faaa015a580 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faa900099b0 tx=0x7faa90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b29e65a4b05fb4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:42.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.854+0000 7faa9dffb700 1 -- 172.21.15.27:0/4120926203 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9000b860 con 0x7faaa015a1b0 2024-01-11T08:10:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.855+0000 7faa9dffb700 1 -- 172.21.15.27:0/4120926203 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa9000b9c0 con 0x7faaa015a1b0 2024-01-11T08:10:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.855+0000 7faa9dffb700 1 -- 172.21.15.27:0/4120926203 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9000bd60 con 0x7faaa015a1b0 2024-01-11T08:10:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.855+0000 7faaa53ff700 1 -- 172.21.15.27:0/4120926203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faaa015a1b0 msgr2=0x7faaa015a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.855+0000 7faaa53ff700 1 --2- 172.21.15.27:0/4120926203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faaa015a1b0 0x7faaa015a580 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faa900099b0 tx=0x7faa90009250 comp rx=0 tx=0).stop 2024-01-11T08:10:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.855+0000 7faaa53ff700 1 -- 172.21.15.27:0/4120926203 shutdown_connections 2024-01-11T08:10:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.855+0000 7faaa53ff700 1 --2- 172.21.15.27:0/4120926203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faaa015a1b0 0x7faaa015a580 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.855+0000 7faaa53ff700 1 -- 172.21.15.27:0/4120926203 >> 172.21.15.27:0/4120926203 conn(0x7faaa006ed40 msgr2=0x7faaa006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.855+0000 7faaa53ff700 1 -- 172.21.15.27:0/4120926203 shutdown_connections 2024-01-11T08:10:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.855+0000 7faaa53ff700 1 -- 172.21.15.27:0/4120926203 wait complete. 2024-01-11T08:10:42.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.857+0000 7faaa53ff700 1 Processor -- start 2024-01-11T08:10:42.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.858+0000 7faaa53ff700 1 -- start start 2024-01-11T08:10:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.859+0000 7faaa53ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faaa00ccd70 0x7faaa00c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.859+0000 7faaa53ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaa015a1b0 0x7faaa00c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.859+0000 7faaa53ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faaa015ae70 0x7faaa013cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.859+0000 7faaa53ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa00ca8c0 con 0x7faaa015ae70 2024-01-11T08:10:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.859+0000 7faaa53ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa00b4510 con 0x7faaa015a1b0 2024-01-11T08:10:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.859+0000 7faaa53ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa00ca020 con 0x7faaa00ccd70 2024-01-11T08:10:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.859+0000 7faa9f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faaa015ae70 0x7faaa013cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.859+0000 7faa9f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faaa015ae70 0x7faaa013cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53564/0 (socket says 172.21.15.27:53564) 2024-01-11T08:10:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.859+0000 7faa9f7fe700 1 -- 172.21.15.27:0/910656931 learned_addr learned my addr 172.21.15.27:0/910656931 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.859+0000 7faa9effd700 1 --2- 172.21.15.27:0/910656931 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faaa00ccd70 0x7faaa00c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:42.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.859+0000 7faa9e7fc700 1 --2- 172.21.15.27:0/910656931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaa015a1b0 0x7faaa00c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:42.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.860+0000 7faa9f7fe700 1 -- 172.21.15.27:0/910656931 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faaa00ccd70 msgr2=0x7faaa00c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:42.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.860+0000 7faa9f7fe700 1 --2- 172.21.15.27:0/910656931 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faaa00ccd70 0x7faaa00c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:42.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.860+0000 7faa9f7fe700 1 -- 172.21.15.27:0/910656931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaa015a1b0 msgr2=0x7faaa00c95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:42.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.860+0000 7faa9f7fe700 1 --2- 172.21.15.27:0/910656931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaa015a1b0 0x7faaa00c95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:42.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.860+0000 7faa9f7fe700 1 -- 172.21.15.27:0/910656931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa900094f0 con 0x7faaa015ae70 2024-01-11T08:10:42.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.860+0000 7faa9f7fe700 1 --2- 172.21.15.27:0/910656931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faaa015ae70 0x7faaa013cf20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7faa9800cd00 tx=0x7faa98001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:42.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.860+0000 7faa86ffd700 1 -- 172.21.15.27:0/910656931 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9800ce60 con 0x7faaa015ae70 2024-01-11T08:10:42.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.860+0000 7faa86ffd700 1 -- 172.21.15.27:0/910656931 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa98005df0 con 0x7faaa015ae70 2024-01-11T08:10:42.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.860+0000 7faaa53ff700 1 -- 172.21.15.27:0/910656931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa00ca300 con 0x7faaa015ae70 2024-01-11T08:10:42.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.860+0000 7faa86ffd700 1 -- 172.21.15.27:0/910656931 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9800ab00 con 0x7faaa015ae70 2024-01-11T08:10:42.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.860+0000 7faaa53ff700 1 -- 172.21.15.27:0/910656931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa00c7ea0 con 0x7faaa015ae70 2024-01-11T08:10:42.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.861+0000 7faaa53ff700 1 -- 172.21.15.27:0/910656931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa004c3c0 con 0x7faaa015ae70 2024-01-11T08:10:42.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.862+0000 7faa86ffd700 1 -- 172.21.15.27:0/910656931 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7faa980070f0 con 0x7faaa015ae70 2024-01-11T08:10:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.863+0000 7faa86ffd700 1 --2- 172.21.15.27:0/910656931 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7faa8c066f30 0x7faa8c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.863+0000 7faa86ffd700 1 -- 172.21.15.27:0/910656931 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7faa980845c0 con 0x7faaa015ae70 2024-01-11T08:10:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.864+0000 7faa9effd700 1 --2- 172.21.15.27:0/910656931 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7faa8c066f30 0x7faa8c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:42.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.864+0000 7faa9effd700 1 --2- 172.21.15.27:0/910656931 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7faa8c066f30 0x7faa8c0693e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7faa900099b0 tx=0x7faa90017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:42.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:42.867+0000 7faa86ffd700 1 -- 172.21.15.27:0/910656931 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faa98051810 con 0x7faaa015ae70 2024-01-11T08:10:43.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.069+0000 7faaa53ff700 1 -- 172.21.15.27:0/910656931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faaa006f1a0 con 0x7faaa015ae70 2024-01-11T08:10:43.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.069+0000 7faa86ffd700 1 -- 172.21.15.27:0/910656931 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7faa98082e60 con 0x7faaa015ae70 2024-01-11T08:10:43.069 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:10:43.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.072+0000 7faaa53ff700 1 -- 172.21.15.27:0/910656931 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7faa8c066f30 msgr2=0x7faa8c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:43.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.072+0000 7faaa53ff700 1 --2- 172.21.15.27:0/910656931 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7faa8c066f30 0x7faa8c0693e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7faa900099b0 tx=0x7faa90017000 comp rx=0 tx=0).stop 2024-01-11T08:10:43.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.072+0000 7faaa53ff700 1 -- 172.21.15.27:0/910656931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faaa015ae70 msgr2=0x7faaa013cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:43.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.072+0000 7faaa53ff700 1 --2- 172.21.15.27:0/910656931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faaa015ae70 0x7faaa013cf20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7faa9800cd00 tx=0x7faa98001b10 comp rx=0 tx=0).stop 2024-01-11T08:10:43.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.072+0000 7faaa53ff700 1 -- 172.21.15.27:0/910656931 shutdown_connections 2024-01-11T08:10:43.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.072+0000 7faaa53ff700 1 --2- 172.21.15.27:0/910656931 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7faa8c066f30 0x7faa8c0693e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.072+0000 7faaa53ff700 1 --2- 172.21.15.27:0/910656931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faaa015ae70 0x7faaa013cf20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.072+0000 7faaa53ff700 1 --2- 172.21.15.27:0/910656931 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faaa00ccd70 0x7faaa00c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.072+0000 7faaa53ff700 1 --2- 172.21.15.27:0/910656931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaa015a1b0 0x7faaa00c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.072+0000 7faaa53ff700 1 -- 172.21.15.27:0/910656931 >> 172.21.15.27:0/910656931 conn(0x7faaa006ed40 msgr2=0x7faaa00d0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:43.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.072+0000 7faaa53ff700 1 -- 172.21.15.27:0/910656931 shutdown_connections 2024-01-11T08:10:43.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.072+0000 7faaa53ff700 1 -- 172.21.15.27:0/910656931 wait complete. 2024-01-11T08:10:43.089 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T08:10:43.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.291+0000 7fa3d35ff700 1 Processor -- start 2024-01-11T08:10:43.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.291+0000 7fa3d35ff700 1 -- start start 2024-01-11T08:10:43.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.292+0000 7fa3d35ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa3cc1596f0 0x7fa3cc0a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:43.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.292+0000 7fa3d35ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3cc0a9de0 0x7fa3cc068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:43.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.292+0000 7fa3d35ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa3cc159ac0 0x7fa3cc068b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:43.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.292+0000 7fa3d35ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3cc0c80a0 con 0x7fa3cc1596f0 2024-01-11T08:10:43.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.292+0000 7fa3d35ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3cc0cbac0 con 0x7fa3cc159ac0 2024-01-11T08:10:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.292+0000 7fa3d35ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc0c7490 con 0x7fa3cc0a9de0 2024-01-11T08:10:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.293+0000 7fa3d139b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa3cc1596f0 0x7fa3cc0a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.293+0000 7fa3d139b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa3cc1596f0 0x7fa3cc0a98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53580/0 (socket says 172.21.15.27:53580) 2024-01-11T08:10:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.293+0000 7fa3d139b700 1 -- 172.21.15.27:0/122625813 learned_addr learned my addr 172.21.15.27:0/122625813 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.293+0000 7fa3d0b9a700 1 --2- 172.21.15.27:0/122625813 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3cc0a9de0 0x7fa3cc068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.293+0000 7fa3d1b9c700 1 --2- 172.21.15.27:0/122625813 >> v2:172.21.15.77:3300/0 conn(0x7fa3cc159ac0 0x7fa3cc068b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.293+0000 7fa3d139b700 1 -- 172.21.15.27:0/122625813 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3cc0a9de0 msgr2=0x7fa3cc068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.293+0000 7fa3d139b700 1 --2- 172.21.15.27:0/122625813 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3cc0a9de0 0x7fa3cc068620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.293+0000 7fa3d139b700 1 -- 172.21.15.27:0/122625813 >> v2:172.21.15.77:3300/0 conn(0x7fa3cc159ac0 msgr2=0x7fa3cc068b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.293+0000 7fa3d139b700 1 --2- 172.21.15.27:0/122625813 >> v2:172.21.15.77:3300/0 conn(0x7fa3cc159ac0 0x7fa3cc068b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.293+0000 7fa3d139b700 1 -- 172.21.15.27:0/122625813 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc0c7770 con 0x7fa3cc1596f0 2024-01-11T08:10:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.293+0000 7fa3d139b700 1 --2- 172.21.15.27:0/122625813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3cc1596f0 0x7fa3cc0a98a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc06e590 tx=0x7fa3c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b770928c218ac9d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.293+0000 7fa3c3fff700 1 -- 172.21.15.27:0/122625813 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c800b860 con 0x7fa3cc1596f0 2024-01-11T08:10:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.294+0000 7fa3c3fff700 1 -- 172.21.15.27:0/122625813 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c800b9c0 con 0x7fa3cc1596f0 2024-01-11T08:10:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.294+0000 7fa3c3fff700 1 -- 172.21.15.27:0/122625813 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c800bd60 con 0x7fa3cc1596f0 2024-01-11T08:10:43.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.294+0000 7fa3d35ff700 1 -- 172.21.15.27:0/122625813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3cc1596f0 msgr2=0x7fa3cc0a98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:43.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.294+0000 7fa3d35ff700 1 --2- 172.21.15.27:0/122625813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3cc1596f0 0x7fa3cc0a98a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc06e590 tx=0x7fa3c8009250 comp rx=0 tx=0).stop 2024-01-11T08:10:43.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.294+0000 7fa3d35ff700 1 -- 172.21.15.27:0/122625813 shutdown_connections 2024-01-11T08:10:43.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.294+0000 7fa3d35ff700 1 --2- 172.21.15.27:0/122625813 >> v2:172.21.15.77:3300/0 conn(0x7fa3cc159ac0 0x7fa3cc068b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.294+0000 7fa3d35ff700 1 --2- 172.21.15.27:0/122625813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3cc1596f0 0x7fa3cc0a98a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.294+0000 7fa3d35ff700 1 --2- 172.21.15.27:0/122625813 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3cc0a9de0 0x7fa3cc068620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.294+0000 7fa3d35ff700 1 -- 172.21.15.27:0/122625813 >> 172.21.15.27:0/122625813 conn(0x7fa3cc0c9fb0 msgr2=0x7fa3cc06ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:43.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.294+0000 7fa3d35ff700 1 -- 172.21.15.27:0/122625813 shutdown_connections 2024-01-11T08:10:43.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.294+0000 7fa3d35ff700 1 -- 172.21.15.27:0/122625813 wait complete. 2024-01-11T08:10:43.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.296+0000 7fa3d35ff700 1 Processor -- start 2024-01-11T08:10:43.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.296+0000 7fa3d35ff700 1 -- start start 2024-01-11T08:10:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.297+0000 7fa3d35ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3cc0a9de0 0x7fa3cc144c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:43.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.297+0000 7fa3d35ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3cc1596f0 0x7fa3cc0abe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:43.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.297+0000 7fa3d35ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3cc159ac0 0x7fa3cc16cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:43.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.297+0000 7fa3d35ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc0ae5c0 con 0x7fa3cc159ac0 2024-01-11T08:10:43.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.297+0000 7fa3d35ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc0ac700 con 0x7fa3cc1596f0 2024-01-11T08:10:43.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.297+0000 7fa3d35ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc0af0e0 con 0x7fa3cc0a9de0 2024-01-11T08:10:43.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.297+0000 7fa3d1b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3cc159ac0 0x7fa3cc16cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:43.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.297+0000 7fa3d1b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3cc159ac0 0x7fa3cc16cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53596/0 (socket says 172.21.15.27:53596) 2024-01-11T08:10:43.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.297+0000 7fa3d1b9c700 1 -- 172.21.15.27:0/2449945976 learned_addr learned my addr 172.21.15.27:0/2449945976 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:43.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.297+0000 7fa3d139b700 1 --2- 172.21.15.27:0/2449945976 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3cc0a9de0 0x7fa3cc144c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:43.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.297+0000 7fa3d0b9a700 1 --2- 172.21.15.27:0/2449945976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3cc1596f0 0x7fa3cc0abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:43.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.297+0000 7fa3d1b9c700 1 -- 172.21.15.27:0/2449945976 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3cc0a9de0 msgr2=0x7fa3cc144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:43.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.297+0000 7fa3d1b9c700 1 --2- 172.21.15.27:0/2449945976 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3cc0a9de0 0x7fa3cc144c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.297+0000 7fa3d1b9c700 1 -- 172.21.15.27:0/2449945976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3cc1596f0 msgr2=0x7fa3cc0abe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:43.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.297+0000 7fa3d1b9c700 1 --2- 172.21.15.27:0/2449945976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3cc1596f0 0x7fa3cc0abe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.297+0000 7fa3d1b9c700 1 -- 172.21.15.27:0/2449945976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c80094f0 con 0x7fa3cc159ac0 2024-01-11T08:10:43.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.298+0000 7fa3d1b9c700 1 --2- 172.21.15.27:0/2449945976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3cc159ac0 0x7fa3cc16cf80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4007ef0 tx=0x7fa3c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:43.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.298+0000 7fa3c17fa700 1 -- 172.21.15.27:0/2449945976 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c400ea60 con 0x7fa3cc159ac0 2024-01-11T08:10:43.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.298+0000 7fa3c17fa700 1 -- 172.21.15.27:0/2449945976 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c400ebc0 con 0x7fa3cc159ac0 2024-01-11T08:10:43.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.298+0000 7fa3c17fa700 1 -- 172.21.15.27:0/2449945976 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4014420 con 0x7fa3cc159ac0 2024-01-11T08:10:43.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.298+0000 7fa3d35ff700 1 -- 172.21.15.27:0/2449945976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3cc145180 con 0x7fa3cc159ac0 2024-01-11T08:10:43.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.298+0000 7fa3d35ff700 1 -- 172.21.15.27:0/2449945976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3cc0aea90 con 0x7fa3cc159ac0 2024-01-11T08:10:43.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.299+0000 7fa3d35ff700 1 -- 172.21.15.27:0/2449945976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3cc04c3c0 con 0x7fa3cc159ac0 2024-01-11T08:10:43.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.300+0000 7fa3c17fa700 1 -- 172.21.15.27:0/2449945976 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa3c4014810 con 0x7fa3cc159ac0 2024-01-11T08:10:43.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.301+0000 7fa3c17fa700 1 --2- 172.21.15.27:0/2449945976 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa3b0066f30 0x7fa3b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:43.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.301+0000 7fa3c17fa700 1 -- 172.21.15.27:0/2449945976 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fa3c4084c50 con 0x7fa3cc159ac0 2024-01-11T08:10:43.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.301+0000 7fa3d139b700 1 --2- 172.21.15.27:0/2449945976 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa3b0066f30 0x7fa3b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:43.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.302+0000 7fa3d139b700 1 --2- 172.21.15.27:0/2449945976 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa3b0066f30 0x7fa3b00693e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc06e590 tx=0x7fa3c8017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:43.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.305+0000 7fa3c17fa700 1 -- 172.21.15.27:0/2449945976 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa3c4051e20 con 0x7fa3cc159ac0 2024-01-11T08:10:43.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.509+0000 7fa3d35ff700 1 -- 172.21.15.27:0/2449945976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa3cc0c7a50 con 0x7fa3cc159ac0 2024-01-11T08:10:43.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.510+0000 7fa3c17fa700 1 -- 172.21.15.27:0/2449945976 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa3c4055440 con 0x7fa3cc159ac0 2024-01-11T08:10:43.510 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T08:10:43.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.512+0000 7fa3d35ff700 1 -- 172.21.15.27:0/2449945976 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa3b0066f30 msgr2=0x7fa3b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:43.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.512+0000 7fa3d35ff700 1 --2- 172.21.15.27:0/2449945976 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa3b0066f30 0x7fa3b00693e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc06e590 tx=0x7fa3c8017000 comp rx=0 tx=0).stop 2024-01-11T08:10:43.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.512+0000 7fa3d35ff700 1 -- 172.21.15.27:0/2449945976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3cc159ac0 msgr2=0x7fa3cc16cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:43.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.512+0000 7fa3d35ff700 1 --2- 172.21.15.27:0/2449945976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3cc159ac0 0x7fa3cc16cf80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4007ef0 tx=0x7fa3c4009840 comp rx=0 tx=0).stop 2024-01-11T08:10:43.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.512+0000 7fa3d35ff700 1 -- 172.21.15.27:0/2449945976 shutdown_connections 2024-01-11T08:10:43.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.512+0000 7fa3d35ff700 1 --2- 172.21.15.27:0/2449945976 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa3b0066f30 0x7fa3b00693e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.512+0000 7fa3d35ff700 1 --2- 172.21.15.27:0/2449945976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3cc159ac0 0x7fa3cc16cf80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.512+0000 7fa3d35ff700 1 --2- 172.21.15.27:0/2449945976 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3cc0a9de0 0x7fa3cc144c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.512+0000 7fa3d35ff700 1 --2- 172.21.15.27:0/2449945976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3cc1596f0 0x7fa3cc0abe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.513+0000 7fa3d35ff700 1 -- 172.21.15.27:0/2449945976 >> 172.21.15.27:0/2449945976 conn(0x7fa3cc0c9fb0 msgr2=0x7fa3cc0c8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:43.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.513+0000 7fa3d35ff700 1 -- 172.21.15.27:0/2449945976 shutdown_connections 2024-01-11T08:10:43.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.513+0000 7fa3d35ff700 1 -- 172.21.15.27:0/2449945976 wait complete. 2024-01-11T08:10:43.530 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T08:10:43.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.731+0000 7f17449ff700 1 Processor -- start 2024-01-11T08:10:43.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.731+0000 7f17449ff700 1 -- start start 2024-01-11T08:10:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.733+0000 7f17449ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f17401596f0 0x7f17400a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.733+0000 7f17449ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17400c80a0 con 0x7f1740068d30 2024-01-11T08:10:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.733+0000 7f17449ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17400cbac0 con 0x7f17400a9de0 2024-01-11T08:10:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.733+0000 7f17449ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17400c7490 con 0x7f17401596f0 2024-01-11T08:10:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.733+0000 7f173ed9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1740068d30 0x7f17400c8530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37278/0 (socket says 172.21.15.27:37278) 2024-01-11T08:10:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.733+0000 7f173ed9d700 1 -- 172.21.15.27:0/1441386669 learned_addr learned my addr 172.21.15.27:0/1441386669 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.733+0000 7f173e59c700 1 --2- 172.21.15.27:0/1441386669 >> v2:172.21.15.77:3300/0 conn(0x7f17401596f0 0x7f17400a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.734+0000 7f173d59a700 1 -- 172.21.15.27:0/1441386669 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444308703 0 0) 0x7f17400c80a0 con 0x7f1740068d30 2024-01-11T08:10:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.734+0000 7f173d59a700 1 -- 172.21.15.27:0/1441386669 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1728003570 con 0x7f1740068d30 2024-01-11T08:10:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.734+0000 7f173d59a700 1 -- 172.21.15.27:0/1441386669 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281826756 0 0) 0x7f17400cbac0 con 0x7f17400a9de0 2024-01-11T08:10:43.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.734+0000 7f173d59a700 1 -- 172.21.15.27:0/1441386669 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17400c80a0 con 0x7f17400a9de0 2024-01-11T08:10:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.734+0000 7f173e59c700 1 -- 172.21.15.27:0/1441386669 >> v1:172.21.15.77:6789/0 conn(0x7f17400a9de0 legacy=0x7f1740068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.734+0000 7f173e59c700 1 -- 172.21.15.27:0/1441386669 >> v1:172.21.15.27:6789/0 conn(0x7f1740068d30 legacy=0x7f17400c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.734+0000 7f173e59c700 1 -- 172.21.15.27:0/1441386669 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17400c7770 con 0x7f17401596f0 2024-01-11T08:10:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.734+0000 7f173e59c700 1 --2- 172.21.15.27:0/1441386669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401596f0 0x7f17400a98a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f17300098e0 tx=0x7f17300091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b01fed6b3fcc10a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.734+0000 7f173d59a700 1 -- 172.21.15.27:0/1441386669 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1730011630 con 0x7f17401596f0 2024-01-11T08:10:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.734+0000 7f173d59a700 1 -- 172.21.15.27:0/1441386669 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1730011790 con 0x7f17401596f0 2024-01-11T08:10:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.734+0000 7f173d59a700 1 -- 172.21.15.27:0/1441386669 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1730011b30 con 0x7f17401596f0 2024-01-11T08:10:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.735+0000 7f17449ff700 1 -- 172.21.15.27:0/1441386669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401596f0 msgr2=0x7f17400a98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.735+0000 7f17449ff700 1 --2- 172.21.15.27:0/1441386669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401596f0 0x7f17400a98a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f17300098e0 tx=0x7f17300091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.735+0000 7f17449ff700 1 -- 172.21.15.27:0/1441386669 shutdown_connections 2024-01-11T08:10:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.735+0000 7f17449ff700 1 --2- 172.21.15.27:0/1441386669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401596f0 0x7f17400a98a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.735+0000 7f17449ff700 1 -- 172.21.15.27:0/1441386669 >> 172.21.15.27:0/1441386669 conn(0x7f17400c9fb0 msgr2=0x7f174006ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.735+0000 7f17449ff700 1 -- 172.21.15.27:0/1441386669 shutdown_connections 2024-01-11T08:10:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.735+0000 7f17449ff700 1 -- 172.21.15.27:0/1441386669 wait complete. 2024-01-11T08:10:43.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.737+0000 7f17449ff700 1 Processor -- start 2024-01-11T08:10:43.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.737+0000 7f17449ff700 1 -- start start 2024-01-11T08:10:43.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.738+0000 7f17449ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1740068d30 0x7f17400c3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:43.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.738+0000 7f17449ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17400a9de0 0x7f17400c5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:43.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.738+0000 7f17449ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401596f0 0x7f174014a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:43.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.738+0000 7f17449ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17400c4100 con 0x7f1740068d30 2024-01-11T08:10:43.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.738+0000 7f17449ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17400c43e0 con 0x7f17401596f0 2024-01-11T08:10:43.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.738+0000 7f17449ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17400c46c0 con 0x7f17400a9de0 2024-01-11T08:10:43.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.738+0000 7f173e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1740068d30 0x7f17400c3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:43.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.738+0000 7f173e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1740068d30 0x7f17400c3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53602/0 (socket says 172.21.15.27:53602) 2024-01-11T08:10:43.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.738+0000 7f173e59c700 1 -- 172.21.15.27:0/2465597825 learned_addr learned my addr 172.21.15.27:0/2465597825 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:43.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.738+0000 7f173dd9b700 1 --2- 172.21.15.27:0/2465597825 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17400a9de0 0x7f17400c5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:43.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.738+0000 7f173ed9d700 1 --2- 172.21.15.27:0/2465597825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401596f0 0x7f174014a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:43.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.739+0000 7f173e59c700 1 -- 172.21.15.27:0/2465597825 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17400a9de0 msgr2=0x7f17400c5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:43.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.739+0000 7f173e59c700 1 --2- 172.21.15.27:0/2465597825 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17400a9de0 0x7f17400c5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.739+0000 7f173e59c700 1 -- 172.21.15.27:0/2465597825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401596f0 msgr2=0x7f174014a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:43.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.739+0000 7f173e59c700 1 --2- 172.21.15.27:0/2465597825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401596f0 0x7f174014a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.739+0000 7f173e59c700 1 -- 172.21.15.27:0/2465597825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1730009420 con 0x7f1740068d30 2024-01-11T08:10:43.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.739+0000 7f173ed9d700 1 --2- 172.21.15.27:0/2465597825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401596f0 0x7f174014a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:43.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.739+0000 7f173e59c700 1 --2- 172.21.15.27:0/2465597825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1740068d30 0x7f17400c3b20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f174006d730 tx=0x7f1730017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:43.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.739+0000 7f172e7fc700 1 -- 172.21.15.27:0/2465597825 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173001f070 con 0x7f1740068d30 2024-01-11T08:10:43.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.739+0000 7f172e7fc700 1 -- 172.21.15.27:0/2465597825 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1730011b10 con 0x7f1740068d30 2024-01-11T08:10:43.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.739+0000 7f172e7fc700 1 -- 172.21.15.27:0/2465597825 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1730023640 con 0x7f1740068d30 2024-01-11T08:10:43.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.739+0000 7f17449ff700 1 -- 172.21.15.27:0/2465597825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17400c49a0 con 0x7f1740068d30 2024-01-11T08:10:43.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.739+0000 7f17449ff700 1 -- 172.21.15.27:0/2465597825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f174014a8f0 con 0x7f1740068d30 2024-01-11T08:10:43.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.740+0000 7f17449ff700 1 -- 172.21.15.27:0/2465597825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17400ab960 con 0x7f1740068d30 2024-01-11T08:10:43.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.741+0000 7f172e7fc700 1 -- 172.21.15.27:0/2465597825 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f173002c3b0 con 0x7f1740068d30 2024-01-11T08:10:43.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.742+0000 7f172e7fc700 1 --2- 172.21.15.27:0/2465597825 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1720066f30 0x7f17200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:43.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.742+0000 7f172e7fc700 1 -- 172.21.15.27:0/2465597825 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f1730085840 con 0x7f1740068d30 2024-01-11T08:10:43.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.743+0000 7f173dd9b700 1 --2- 172.21.15.27:0/2465597825 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1720066f30 0x7f17200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:43.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.743+0000 7f173dd9b700 1 --2- 172.21.15.27:0/2465597825 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1720066f30 0x7f17200693e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1740157b10 tx=0x7f1734009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:43.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.746+0000 7f172e7fc700 1 -- 172.21.15.27:0/2465597825 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1730052a10 con 0x7f1740068d30 2024-01-11T08:10:43.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.948+0000 7f17449ff700 1 -- 172.21.15.27:0/2465597825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f174004c3c0 con 0x7f1740068d30 2024-01-11T08:10:43.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.948+0000 7f172e7fc700 1 -- 172.21.15.27:0/2465597825 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1730015020 con 0x7f1740068d30 2024-01-11T08:10:43.948 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T08:10:43.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.951+0000 7f17449ff700 1 -- 172.21.15.27:0/2465597825 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1720066f30 msgr2=0x7f17200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:43.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.951+0000 7f17449ff700 1 --2- 172.21.15.27:0/2465597825 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1720066f30 0x7f17200693e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1740157b10 tx=0x7f1734009040 comp rx=0 tx=0).stop 2024-01-11T08:10:43.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.951+0000 7f17449ff700 1 -- 172.21.15.27:0/2465597825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1740068d30 msgr2=0x7f17400c3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:43.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.951+0000 7f17449ff700 1 --2- 172.21.15.27:0/2465597825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1740068d30 0x7f17400c3b20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f174006d730 tx=0x7f1730017000 comp rx=0 tx=0).stop 2024-01-11T08:10:43.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.951+0000 7f17449ff700 1 -- 172.21.15.27:0/2465597825 shutdown_connections 2024-01-11T08:10:43.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.951+0000 7f17449ff700 1 --2- 172.21.15.27:0/2465597825 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1720066f30 0x7f17200693e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.951+0000 7f17449ff700 1 --2- 172.21.15.27:0/2465597825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17401596f0 0x7f174014a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.951+0000 7f17449ff700 1 --2- 172.21.15.27:0/2465597825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1740068d30 0x7f17400c3b20 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.951+0000 7f17449ff700 1 --2- 172.21.15.27:0/2465597825 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17400a9de0 0x7f17400c5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:43.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.951+0000 7f17449ff700 1 -- 172.21.15.27:0/2465597825 >> 172.21.15.27:0/2465597825 conn(0x7f17400c9fb0 msgr2=0x7f17400c8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:43.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.951+0000 7f17449ff700 1 -- 172.21.15.27:0/2465597825 shutdown_connections 2024-01-11T08:10:43.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:43.951+0000 7f17449ff700 1 -- 172.21.15.27:0/2465597825 wait complete. 2024-01-11T08:10:43.968 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:10:44.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.168+0000 7f7296fff700 1 Processor -- start 2024-01-11T08:10:44.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.168+0000 7f7296fff700 1 -- start start 2024-01-11T08:10:44.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.170+0000 7f7296fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7290069d40 0x7f72900b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:44.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.170+0000 7f7296fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7290157390 con 0x7f72900c8620 2024-01-11T08:10:44.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.170+0000 7f7296fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729006a200 con 0x7f72900c89f0 2024-01-11T08:10:44.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.170+0000 7f7296fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7290156d40 con 0x7f7290069d40 2024-01-11T08:10:44.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.170+0000 7f7294d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f72900c8620 0x7f72900af970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51378/0 (socket says 172.21.15.27:51378) 2024-01-11T08:10:44.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.170+0000 7f7294d9b700 1 -- 172.21.15.27:0/1688955046 learned_addr learned my addr 172.21.15.27:0/1688955046 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:44.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.170+0000 7f728ffff700 1 --2- 172.21.15.27:0/1688955046 >> v2:172.21.15.77:3300/0 conn(0x7f7290069d40 0x7f72900b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.170+0000 7f728f7fe700 1 -- 172.21.15.27:0/1688955046 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101041272 0 0) 0x7f7290157390 con 0x7f72900c8620 2024-01-11T08:10:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.170+0000 7f728f7fe700 1 -- 172.21.15.27:0/1688955046 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7280003570 con 0x7f72900c8620 2024-01-11T08:10:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.171+0000 7f728f7fe700 1 -- 172.21.15.27:0/1688955046 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014371257 0 0) 0x7f729006a200 con 0x7f72900c89f0 2024-01-11T08:10:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.171+0000 7f728f7fe700 1 -- 172.21.15.27:0/1688955046 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7290157390 con 0x7f72900c89f0 2024-01-11T08:10:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.171+0000 7f728ffff700 1 -- 172.21.15.27:0/1688955046 >> v1:172.21.15.77:6789/0 conn(0x7f72900c89f0 legacy=0x7f72900accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.171+0000 7f728f7fe700 1 -- 172.21.15.27:0/1688955046 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3499660133 0 0) 0x7f7280003570 con 0x7f72900c8620 2024-01-11T08:10:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.171+0000 7f728ffff700 1 -- 172.21.15.27:0/1688955046 >> v1:172.21.15.27:6789/0 conn(0x7f72900c8620 legacy=0x7f72900af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.171+0000 7f728ffff700 1 -- 172.21.15.27:0/1688955046 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7290157020 con 0x7f7290069d40 2024-01-11T08:10:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.171+0000 7f728ffff700 1 --2- 172.21.15.27:0/1688955046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7290069d40 0x7f72900b02d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f727c0098e0 tx=0x7f727c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4743e7715911095c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.171+0000 7f728f7fe700 1 -- 172.21.15.27:0/1688955046 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7284001470 con 0x7f7290069d40 2024-01-11T08:10:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.171+0000 7f728f7fe700 1 -- 172.21.15.27:0/1688955046 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727c011420 con 0x7f7290069d40 2024-01-11T08:10:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.171+0000 7f728f7fe700 1 -- 172.21.15.27:0/1688955046 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727c0117c0 con 0x7f7290069d40 2024-01-11T08:10:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.172+0000 7f7296fff700 1 -- 172.21.15.27:0/1688955046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7290069d40 msgr2=0x7f72900b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.172+0000 7f7296fff700 1 --2- 172.21.15.27:0/1688955046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7290069d40 0x7f72900b02d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f727c0098e0 tx=0x7f727c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.172+0000 7f7296fff700 1 -- 172.21.15.27:0/1688955046 shutdown_connections 2024-01-11T08:10:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.172+0000 7f7296fff700 1 --2- 172.21.15.27:0/1688955046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7290069d40 0x7f72900b02d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.172+0000 7f7296fff700 1 -- 172.21.15.27:0/1688955046 >> 172.21.15.27:0/1688955046 conn(0x7f72900680e0 msgr2=0x7f7290069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.172+0000 7f7296fff700 1 -- 172.21.15.27:0/1688955046 shutdown_connections 2024-01-11T08:10:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.172+0000 7f7296fff700 1 -- 172.21.15.27:0/1688955046 wait complete. 2024-01-11T08:10:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.174+0000 7f7296fff700 1 Processor -- start 2024-01-11T08:10:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.174+0000 7f7296fff700 1 -- start start 2024-01-11T08:10:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.175+0000 7f7296fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7290069d40 0x7f72900ae030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.175+0000 7f7296fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f72900c8620 0x7f72900c6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.175+0000 7f7296fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72900c89f0 0x7f729006d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.175+0000 7f7296fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7290143570 con 0x7f72900c89f0 2024-01-11T08:10:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.175+0000 7f7296fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72900c7820 con 0x7f7290069d40 2024-01-11T08:10:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.175+0000 7f7296fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72900c6c10 con 0x7f72900c8620 2024-01-11T08:10:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.175+0000 7f729559c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72900c89f0 0x7f729006d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.175+0000 7f729559c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72900c89f0 0x7f729006d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54660/0 (socket says 172.21.15.27:54660) 2024-01-11T08:10:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.175+0000 7f729559c700 1 -- 172.21.15.27:0/2420821073 learned_addr learned my addr 172.21.15.27:0/2420821073 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.175+0000 7f728ffff700 1 --2- 172.21.15.27:0/2420821073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f72900c8620 0x7f72900c6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.175+0000 7f7294d9b700 1 --2- 172.21.15.27:0/2420821073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7290069d40 0x7f72900ae030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.176+0000 7f729559c700 1 -- 172.21.15.27:0/2420821073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f72900c8620 msgr2=0x7f72900c6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.176+0000 7f729559c700 1 --2- 172.21.15.27:0/2420821073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f72900c8620 0x7f72900c6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.176+0000 7f729559c700 1 -- 172.21.15.27:0/2420821073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7290069d40 msgr2=0x7f72900ae030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.176+0000 7f729559c700 1 --2- 172.21.15.27:0/2420821073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7290069d40 0x7f72900ae030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.176+0000 7f729559c700 1 -- 172.21.15.27:0/2420821073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f727c009420 con 0x7f72900c89f0 2024-01-11T08:10:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.176+0000 7f729559c700 1 --2- 172.21.15.27:0/2420821073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72900c89f0 0x7f729006d5f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f728800c910 tx=0x7f7288007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.176+0000 7f728cff9700 1 -- 172.21.15.27:0/2420821073 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7288012630 con 0x7f72900c89f0 2024-01-11T08:10:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.176+0000 7f728cff9700 1 -- 172.21.15.27:0/2420821073 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72880127b0 con 0x7f72900c89f0 2024-01-11T08:10:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.176+0000 7f728cff9700 1 -- 172.21.15.27:0/2420821073 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7288012b70 con 0x7f72900c89f0 2024-01-11T08:10:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.176+0000 7f7296fff700 1 -- 172.21.15.27:0/2420821073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72900c6ef0 con 0x7f72900c89f0 2024-01-11T08:10:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.176+0000 7f7296fff700 1 -- 172.21.15.27:0/2420821073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72900c74b0 con 0x7f72900c89f0 2024-01-11T08:10:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.177+0000 7f7296fff700 1 -- 172.21.15.27:0/2420821073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72900c79b0 con 0x7f72900c89f0 2024-01-11T08:10:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.178+0000 7f728cff9700 1 -- 172.21.15.27:0/2420821073 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f728801b400 con 0x7f72900c89f0 2024-01-11T08:10:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.179+0000 7f728cff9700 1 --2- 172.21.15.27:0/2420821073 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7274066f30 0x7f72740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.179+0000 7f728cff9700 1 -- 172.21.15.27:0/2420821073 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f72880848f0 con 0x7f72900c89f0 2024-01-11T08:10:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.179+0000 7f7294d9b700 1 --2- 172.21.15.27:0/2420821073 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7274066f30 0x7f72740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.180+0000 7f7294d9b700 1 --2- 172.21.15.27:0/2420821073 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7274066f30 0x7f72740693e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f72840011e0 tx=0x7f7284000bc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.183+0000 7f728cff9700 1 -- 172.21.15.27:0/2420821073 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7288051ac0 con 0x7f72900c89f0 2024-01-11T08:10:44.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.428+0000 7f7296fff700 1 -- 172.21.15.27:0/2420821073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72900718d0 con 0x7f72900c89f0 2024-01-11T08:10:44.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.428+0000 7f728cff9700 1 -- 172.21.15.27:0/2420821073 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4588 (secure 0 0 0) 0x7f7288016020 con 0x7f72900c89f0 2024-01-11T08:10:44.428 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:44.429 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3168764915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3168764915},{"type":"v1","addr":"172.21.15.27:6834","nonce":3168764915}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2607167395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2607167395},{"type":"v1","addr":"172.21.15.27:6835","nonce":2607167395}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2123423970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2123423970},{"type":"v1","addr":"172.21.15.77:6837","nonce":2123423970}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:10:30.160827+0000","modified":"2024-01-11T08:10:33.681210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6836/269464429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":269464429},{"type":"v1","addr":"172.21.15.77:6836","nonce":269464429}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2024-01-11T08:10:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.431+0000 7f7296fff700 1 -- 172.21.15.27:0/2420821073 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7274066f30 msgr2=0x7f72740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.431+0000 7f7296fff700 1 --2- 172.21.15.27:0/2420821073 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7274066f30 0x7f72740693e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f72840011e0 tx=0x7f7284000bc0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.431+0000 7f7296fff700 1 -- 172.21.15.27:0/2420821073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72900c89f0 msgr2=0x7f729006d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.431+0000 7f7296fff700 1 --2- 172.21.15.27:0/2420821073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72900c89f0 0x7f729006d5f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f728800c910 tx=0x7f7288007f90 comp rx=0 tx=0).stop 2024-01-11T08:10:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.431+0000 7f7296fff700 1 -- 172.21.15.27:0/2420821073 shutdown_connections 2024-01-11T08:10:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.431+0000 7f7296fff700 1 --2- 172.21.15.27:0/2420821073 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7274066f30 0x7f72740693e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.431+0000 7f7296fff700 1 --2- 172.21.15.27:0/2420821073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72900c89f0 0x7f729006d5f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.431+0000 7f7296fff700 1 --2- 172.21.15.27:0/2420821073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7290069d40 0x7f72900ae030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.431+0000 7f7296fff700 1 --2- 172.21.15.27:0/2420821073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f72900c8620 0x7f72900c6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.431+0000 7f7296fff700 1 -- 172.21.15.27:0/2420821073 >> 172.21.15.27:0/2420821073 conn(0x7f72900680e0 msgr2=0x7f72901517d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:44.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.431+0000 7f7296fff700 1 -- 172.21.15.27:0/2420821073 shutdown_connections 2024-01-11T08:10:44.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.431+0000 7f7296fff700 1 -- 172.21.15.27:0/2420821073 wait complete. 2024-01-11T08:10:44.432 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 8 2024-01-11T08:10:44.448 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:10:44.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.649+0000 7f7724bff700 1 Processor -- start 2024-01-11T08:10:44.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.649+0000 7f7724bff700 1 -- start start 2024-01-11T08:10:44.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.650+0000 7f7724bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77201596c0 0x7f772015d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:44.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.650+0000 7f7724bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77200ca670 0x7f7720068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:44.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.650+0000 7f7724bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7720068b60 0x7f7720159a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:44.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.650+0000 7f7724bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7720068fe0 con 0x7f77201596c0 2024-01-11T08:10:44.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.650+0000 7f7724bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77200a9cb0 con 0x7f7720068b60 2024-01-11T08:10:44.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.650+0000 7f7724bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f772015e210 con 0x7f77200ca670 2024-01-11T08:10:44.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.651+0000 7f771e59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77201596c0 0x7f772015d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:44.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.651+0000 7f771e59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77201596c0 0x7f772015d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54668/0 (socket says 172.21.15.27:54668) 2024-01-11T08:10:44.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.651+0000 7f771e59c700 1 -- 172.21.15.27:0/2615779502 learned_addr learned my addr 172.21.15.27:0/2615779502 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:44.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.651+0000 7f771dd9b700 1 --2- 172.21.15.27:0/2615779502 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77200ca670 0x7f7720068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:44.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.651+0000 7f771ed9d700 1 --2- 172.21.15.27:0/2615779502 >> v2:172.21.15.77:3300/0 conn(0x7f7720068b60 0x7f7720159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:44.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.651+0000 7f771e59c700 1 -- 172.21.15.27:0/2615779502 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77200ca670 msgr2=0x7f7720068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:44.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.651+0000 7f771e59c700 1 --2- 172.21.15.27:0/2615779502 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77200ca670 0x7f7720068620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.651+0000 7f771e59c700 1 -- 172.21.15.27:0/2615779502 >> v2:172.21.15.77:3300/0 conn(0x7f7720068b60 msgr2=0x7f7720159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:44.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.651+0000 7f771e59c700 1 --2- 172.21.15.27:0/2615779502 >> v2:172.21.15.77:3300/0 conn(0x7f7720068b60 0x7f7720159a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.651+0000 7f771e59c700 1 -- 172.21.15.27:0/2615779502 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772015dc50 con 0x7f77201596c0 2024-01-11T08:10:44.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.651+0000 7f771e59c700 1 --2- 172.21.15.27:0/2615779502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77201596c0 0x7f772015d300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f772006ca60 tx=0x7f7710009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a13cc364b7038e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:44.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.652+0000 7f771d59a700 1 -- 172.21.15.27:0/2615779502 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771000b640 con 0x7f77201596c0 2024-01-11T08:10:44.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.652+0000 7f771d59a700 1 -- 172.21.15.27:0/2615779502 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771000b7a0 con 0x7f77201596c0 2024-01-11T08:10:44.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.652+0000 7f771d59a700 1 -- 172.21.15.27:0/2615779502 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771000bb60 con 0x7f77201596c0 2024-01-11T08:10:44.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.652+0000 7f7724bff700 1 -- 172.21.15.27:0/2615779502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77201596c0 msgr2=0x7f772015d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:44.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.652+0000 7f7724bff700 1 --2- 172.21.15.27:0/2615779502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77201596c0 0x7f772015d300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f772006ca60 tx=0x7f7710009250 comp rx=0 tx=0).stop 2024-01-11T08:10:44.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.652+0000 7f7724bff700 1 -- 172.21.15.27:0/2615779502 shutdown_connections 2024-01-11T08:10:44.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.652+0000 7f7724bff700 1 --2- 172.21.15.27:0/2615779502 >> v2:172.21.15.77:3300/0 conn(0x7f7720068b60 0x7f7720159a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.652+0000 7f7724bff700 1 --2- 172.21.15.27:0/2615779502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77201596c0 0x7f772015d300 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.652+0000 7f7724bff700 1 --2- 172.21.15.27:0/2615779502 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77200ca670 0x7f7720068620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.652+0000 7f7724bff700 1 -- 172.21.15.27:0/2615779502 >> 172.21.15.27:0/2615779502 conn(0x7f77200c9f70 msgr2=0x7f772006deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:44.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.653+0000 7f7724bff700 1 -- 172.21.15.27:0/2615779502 shutdown_connections 2024-01-11T08:10:44.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.653+0000 7f7724bff700 1 -- 172.21.15.27:0/2615779502 wait complete. 2024-01-11T08:10:44.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.654+0000 7f7724bff700 1 Processor -- start 2024-01-11T08:10:44.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.654+0000 7f7724bff700 1 -- start start 2024-01-11T08:10:44.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.655+0000 7f7724bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7720068b60 0x7f7720147400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:44.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.655+0000 7f7724bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77200ca670 0x7f7720156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:44.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.655+0000 7f7724bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77201596c0 0x7f7720157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:44.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.655+0000 7f7724bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77200ae060 con 0x7f77200ca670 2024-01-11T08:10:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.655+0000 7f7724bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77200ac240 con 0x7f77201596c0 2024-01-11T08:10:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.655+0000 7f7724bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77200afe40 con 0x7f7720068b60 2024-01-11T08:10:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.655+0000 7f771dd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77200ca670 0x7f7720156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.655+0000 7f771dd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77200ca670 0x7f7720156ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54678/0 (socket says 172.21.15.27:54678) 2024-01-11T08:10:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.655+0000 7f771dd9b700 1 -- 172.21.15.27:0/1639209386 learned_addr learned my addr 172.21.15.27:0/1639209386 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.656+0000 7f771e59c700 1 --2- 172.21.15.27:0/1639209386 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7720068b60 0x7f7720147400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.656+0000 7f771ed9d700 1 --2- 172.21.15.27:0/1639209386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77201596c0 0x7f7720157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.656+0000 7f771dd9b700 1 -- 172.21.15.27:0/1639209386 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7720068b60 msgr2=0x7f7720147400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.656+0000 7f771dd9b700 1 --2- 172.21.15.27:0/1639209386 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7720068b60 0x7f7720147400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.656+0000 7f771dd9b700 1 -- 172.21.15.27:0/1639209386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77201596c0 msgr2=0x7f7720157400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.656+0000 7f771dd9b700 1 --2- 172.21.15.27:0/1639209386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77201596c0 0x7f7720157400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.656+0000 7f771dd9b700 1 -- 172.21.15.27:0/1639209386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77100094f0 con 0x7f77200ca670 2024-01-11T08:10:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.656+0000 7f771ed9d700 1 --2- 172.21.15.27:0/1639209386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77201596c0 0x7f7720157400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.656+0000 7f771dd9b700 1 --2- 172.21.15.27:0/1639209386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77200ca670 0x7f7720156ec0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f771400eaa0 tx=0x7f77140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:44.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.656+0000 7f770e7fc700 1 -- 172.21.15.27:0/1639209386 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771400c850 con 0x7f77200ca670 2024-01-11T08:10:44.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.656+0000 7f770e7fc700 1 -- 172.21.15.27:0/1639209386 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771400c9d0 con 0x7f77200ca670 2024-01-11T08:10:44.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.656+0000 7f770e7fc700 1 -- 172.21.15.27:0/1639209386 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771400cd90 con 0x7f77200ca670 2024-01-11T08:10:44.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.656+0000 7f7724bff700 1 -- 172.21.15.27:0/1639209386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77200adaa0 con 0x7f77200ca670 2024-01-11T08:10:44.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.657+0000 7f7724bff700 1 -- 172.21.15.27:0/1639209386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77200afad0 con 0x7f77200ca670 2024-01-11T08:10:44.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.658+0000 7f7724bff700 1 -- 172.21.15.27:0/1639209386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77200af930 con 0x7f77200ca670 2024-01-11T08:10:44.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.659+0000 7f770e7fc700 1 -- 172.21.15.27:0/1639209386 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7714013680 con 0x7f77200ca670 2024-01-11T08:10:44.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.660+0000 7f770e7fc700 1 --2- 172.21.15.27:0/1639209386 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7700066f30 0x7f77000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:44.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.660+0000 7f770e7fc700 1 -- 172.21.15.27:0/1639209386 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f7714084830 con 0x7f77200ca670 2024-01-11T08:10:44.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.660+0000 7f771e59c700 1 --2- 172.21.15.27:0/1639209386 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7700066f30 0x7f77000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:44.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.661+0000 7f771e59c700 1 --2- 172.21.15.27:0/1639209386 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7700066f30 0x7f77000693e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f772006ca60 tx=0x7f7710017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:44.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.664+0000 7f770e7fc700 1 -- 172.21.15.27:0/1639209386 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f77200af930 con 0x7f77200ca670 2024-01-11T08:10:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.866+0000 7f7724bff700 1 -- 172.21.15.27:0/1639209386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77200af930 con 0x7f77200ca670 2024-01-11T08:10:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.867+0000 7f770e7fc700 1 -- 172.21.15.27:0/1639209386 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15302 (secure 0 0 0) 0x7f77200af930 con 0x7f77200ca670 2024-01-11T08:10:44.867 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:44.867 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:31.464323+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:10:27.993251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-11T08:10:29.153959+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:10:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.869+0000 7f7724bff700 1 -- 172.21.15.27:0/1639209386 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7700066f30 msgr2=0x7f77000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:44.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.869+0000 7f7724bff700 1 --2- 172.21.15.27:0/1639209386 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7700066f30 0x7f77000693e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f772006ca60 tx=0x7f7710017000 comp rx=0 tx=0).stop 2024-01-11T08:10:44.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.869+0000 7f7724bff700 1 -- 172.21.15.27:0/1639209386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77200ca670 msgr2=0x7f7720156ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:44.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.869+0000 7f7724bff700 1 --2- 172.21.15.27:0/1639209386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77200ca670 0x7f7720156ec0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f771400eaa0 tx=0x7f77140091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.870+0000 7f7724bff700 1 -- 172.21.15.27:0/1639209386 shutdown_connections 2024-01-11T08:10:44.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.870+0000 7f7724bff700 1 --2- 172.21.15.27:0/1639209386 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7700066f30 0x7f77000693e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.870+0000 7f7724bff700 1 --2- 172.21.15.27:0/1639209386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77201596c0 0x7f7720157400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.870+0000 7f7724bff700 1 --2- 172.21.15.27:0/1639209386 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7720068b60 0x7f7720147400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.870+0000 7f7724bff700 1 --2- 172.21.15.27:0/1639209386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77200ca670 0x7f7720156ec0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:44.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.870+0000 7f7724bff700 1 -- 172.21.15.27:0/1639209386 >> 172.21.15.27:0/1639209386 conn(0x7f77200c9f70 msgr2=0x7f7720156700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:44.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.870+0000 7f7724bff700 1 -- 172.21.15.27:0/1639209386 shutdown_connections 2024-01-11T08:10:44.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:44.870+0000 7f7724bff700 1 -- 172.21.15.27:0/1639209386 wait complete. 2024-01-11T08:10:44.888 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:10:45.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.087+0000 7efc843ff700 1 Processor -- start 2024-01-11T08:10:45.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.087+0000 7efc843ff700 1 -- start start 2024-01-11T08:10:45.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.088+0000 7efc843ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc7c0ca3e0 0x7efc7c0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:45.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.088+0000 7efc843ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efc7c0ca7b0 0x7efc7c159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:45.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.088+0000 7efc843ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efc7c0c8a20 0x7efc7c0c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:45.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.088+0000 7efc843ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc7c15ae70 con 0x7efc7c0ca7b0 2024-01-11T08:10:45.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.088+0000 7efc843ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc7c15a3c0 con 0x7efc7c0c8a20 2024-01-11T08:10:45.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.088+0000 7efc843ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c0cc4f0 con 0x7efc7c0ca3e0 2024-01-11T08:10:45.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.089+0000 7efc8199a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efc7c0ca7b0 0x7efc7c159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:45.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.089+0000 7efc8199a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efc7c0ca7b0 0x7efc7c159e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54684/0 (socket says 172.21.15.27:54684) 2024-01-11T08:10:45.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.089+0000 7efc8199a700 1 -- 172.21.15.27:0/2872227820 learned_addr learned my addr 172.21.15.27:0/2872227820 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:45.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.089+0000 7efc8299c700 1 --2- 172.21.15.27:0/2872227820 >> v2:172.21.15.77:3300/0 conn(0x7efc7c0c8a20 0x7efc7c0c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:45.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.089+0000 7efc8219b700 1 --2- 172.21.15.27:0/2872227820 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc7c0ca3e0 0x7efc7c0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:45.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.089+0000 7efc8199a700 1 -- 172.21.15.27:0/2872227820 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc7c0ca3e0 msgr2=0x7efc7c0cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:45.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.089+0000 7efc8199a700 1 --2- 172.21.15.27:0/2872227820 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc7c0ca3e0 0x7efc7c0cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.089+0000 7efc8199a700 1 -- 172.21.15.27:0/2872227820 >> v2:172.21.15.77:3300/0 conn(0x7efc7c0c8a20 msgr2=0x7efc7c0c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:45.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.089+0000 7efc8199a700 1 --2- 172.21.15.27:0/2872227820 >> v2:172.21.15.77:3300/0 conn(0x7efc7c0c8a20 0x7efc7c0c8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.089+0000 7efc8199a700 1 -- 172.21.15.27:0/2872227820 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c15d640 con 0x7efc7c0ca7b0 2024-01-11T08:10:45.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.089+0000 7efc8299c700 1 --2- 172.21.15.27:0/2872227820 >> v2:172.21.15.77:3300/0 conn(0x7efc7c0c8a20 0x7efc7c0c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:45.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.089+0000 7efc8199a700 1 --2- 172.21.15.27:0/2872227820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc7c0ca7b0 0x7efc7c159e80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efc7c0ce2b0 tx=0x7efc6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b21a86f98ebf7480 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:45.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.090+0000 7efc81199700 1 -- 172.21.15.27:0/2872227820 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c00b860 con 0x7efc7c0ca7b0 2024-01-11T08:10:45.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.090+0000 7efc81199700 1 -- 172.21.15.27:0/2872227820 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6c00b9c0 con 0x7efc7c0ca7b0 2024-01-11T08:10:45.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.090+0000 7efc81199700 1 -- 172.21.15.27:0/2872227820 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c00bd60 con 0x7efc7c0ca7b0 2024-01-11T08:10:45.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.090+0000 7efc843ff700 1 -- 172.21.15.27:0/2872227820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc7c0ca7b0 msgr2=0x7efc7c159e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:45.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.090+0000 7efc843ff700 1 --2- 172.21.15.27:0/2872227820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc7c0ca7b0 0x7efc7c159e80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efc7c0ce2b0 tx=0x7efc6c009250 comp rx=0 tx=0).stop 2024-01-11T08:10:45.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.090+0000 7efc843ff700 1 -- 172.21.15.27:0/2872227820 shutdown_connections 2024-01-11T08:10:45.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.090+0000 7efc843ff700 1 --2- 172.21.15.27:0/2872227820 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc7c0ca3e0 0x7efc7c0cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.090+0000 7efc843ff700 1 --2- 172.21.15.27:0/2872227820 >> v2:172.21.15.77:3300/0 conn(0x7efc7c0c8a20 0x7efc7c0c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.090+0000 7efc843ff700 1 --2- 172.21.15.27:0/2872227820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc7c0ca7b0 0x7efc7c159e80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.090+0000 7efc843ff700 1 -- 172.21.15.27:0/2872227820 >> 172.21.15.27:0/2872227820 conn(0x7efc7c0ce3e0 msgr2=0x7efc7c0ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:45.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.090+0000 7efc843ff700 1 -- 172.21.15.27:0/2872227820 shutdown_connections 2024-01-11T08:10:45.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.091+0000 7efc843ff700 1 -- 172.21.15.27:0/2872227820 wait complete. 2024-01-11T08:10:45.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.092+0000 7efc843ff700 1 Processor -- start 2024-01-11T08:10:45.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.092+0000 7efc843ff700 1 -- start start 2024-01-11T08:10:45.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.093+0000 7efc843ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc7c0c8a20 0x7efc7c140f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:45.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.093+0000 7efc843ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc7c0ca3e0 0x7efc7c15d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.093+0000 7efc843ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc7c0ca7b0 0x7efc7c15e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.093+0000 7efc843ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c16e480 con 0x7efc7c0ca3e0 2024-01-11T08:10:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.093+0000 7efc843ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c1514d0 con 0x7efc7c0ca7b0 2024-01-11T08:10:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.093+0000 7efc843ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c16d870 con 0x7efc7c0c8a20 2024-01-11T08:10:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.093+0000 7efc8199a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc7c0ca3e0 0x7efc7c15d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.093+0000 7efc8199a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc7c0ca3e0 0x7efc7c15d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54694/0 (socket says 172.21.15.27:54694) 2024-01-11T08:10:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.093+0000 7efc8199a700 1 -- 172.21.15.27:0/2275045577 learned_addr learned my addr 172.21.15.27:0/2275045577 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.094+0000 7efc8299c700 1 --2- 172.21.15.27:0/2275045577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc7c0ca7b0 0x7efc7c15e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.094+0000 7efc8219b700 1 --2- 172.21.15.27:0/2275045577 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc7c0c8a20 0x7efc7c140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.094+0000 7efc8199a700 1 -- 172.21.15.27:0/2275045577 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc7c0c8a20 msgr2=0x7efc7c140f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.094+0000 7efc8199a700 1 --2- 172.21.15.27:0/2275045577 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc7c0c8a20 0x7efc7c140f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.094+0000 7efc8199a700 1 -- 172.21.15.27:0/2275045577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc7c0ca7b0 msgr2=0x7efc7c15e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.094+0000 7efc8199a700 1 --2- 172.21.15.27:0/2275045577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc7c0ca7b0 0x7efc7c15e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.094+0000 7efc8199a700 1 -- 172.21.15.27:0/2275045577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6c0094f0 con 0x7efc7c0ca3e0 2024-01-11T08:10:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.094+0000 7efc8219b700 1 --2- 172.21.15.27:0/2275045577 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc7c0c8a20 0x7efc7c140f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.094+0000 7efc8199a700 1 --2- 172.21.15.27:0/2275045577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc7c0ca3e0 0x7efc7c15d9d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efc6c0090b0 tx=0x7efc6c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.094+0000 7efc727fc700 1 -- 172.21.15.27:0/2275045577 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c024440 con 0x7efc7c0ca3e0 2024-01-11T08:10:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.094+0000 7efc727fc700 1 -- 172.21.15.27:0/2275045577 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6c0245e0 con 0x7efc7c0ca3e0 2024-01-11T08:10:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.094+0000 7efc727fc700 1 -- 172.21.15.27:0/2275045577 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c0249a0 con 0x7efc7c0ca3e0 2024-01-11T08:10:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.094+0000 7efc843ff700 1 -- 172.21.15.27:0/2275045577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7c16db50 con 0x7efc7c0ca3e0 2024-01-11T08:10:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.094+0000 7efc843ff700 1 -- 172.21.15.27:0/2275045577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7c16e110 con 0x7efc7c0ca3e0 2024-01-11T08:10:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.096+0000 7efc843ff700 1 -- 172.21.15.27:0/2275045577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc7c0718d0 con 0x7efc7c0ca3e0 2024-01-11T08:10:45.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.096+0000 7efc727fc700 1 -- 172.21.15.27:0/2275045577 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7efc6c01a420 con 0x7efc7c0ca3e0 2024-01-11T08:10:45.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.097+0000 7efc727fc700 1 --2- 172.21.15.27:0/2275045577 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efc60066f30 0x7efc600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:45.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.097+0000 7efc727fc700 1 -- 172.21.15.27:0/2275045577 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7efc6c0864c0 con 0x7efc7c0ca3e0 2024-01-11T08:10:45.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.098+0000 7efc8219b700 1 --2- 172.21.15.27:0/2275045577 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efc60066f30 0x7efc600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:45.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.098+0000 7efc8219b700 1 --2- 172.21.15.27:0/2275045577 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efc60066f30 0x7efc600693e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efc78006fd0 tx=0x7efc78008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:45.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.101+0000 7efc727fc700 1 -- 172.21.15.27:0/2275045577 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efc6c053690 con 0x7efc7c0ca3e0 2024-01-11T08:10:45.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.348+0000 7efc843ff700 1 -- 172.21.15.27:0/2275045577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc7c16e5c0 con 0x7efc7c0ca3e0 2024-01-11T08:10:45.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.348+0000 7efc727fc700 1 -- 172.21.15.27:0/2275045577 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4588 (secure 0 0 0) 0x7efc6c056cb0 con 0x7efc7c0ca3e0 2024-01-11T08:10:45.348 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:45.349 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3168764915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3168764915},{"type":"v1","addr":"172.21.15.27:6834","nonce":3168764915}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2607167395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2607167395},{"type":"v1","addr":"172.21.15.27:6835","nonce":2607167395}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2123423970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2123423970},{"type":"v1","addr":"172.21.15.77:6837","nonce":2123423970}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:10:30.160827+0000","modified":"2024-01-11T08:10:33.681210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6836/269464429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":269464429},{"type":"v1","addr":"172.21.15.77:6836","nonce":269464429}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2024-01-11T08:10:45.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.351+0000 7efc843ff700 1 -- 172.21.15.27:0/2275045577 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efc60066f30 msgr2=0x7efc600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:45.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.351+0000 7efc843ff700 1 --2- 172.21.15.27:0/2275045577 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efc60066f30 0x7efc600693e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efc78006fd0 tx=0x7efc78008040 comp rx=0 tx=0).stop 2024-01-11T08:10:45.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.351+0000 7efc843ff700 1 -- 172.21.15.27:0/2275045577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc7c0ca3e0 msgr2=0x7efc7c15d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:45.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.351+0000 7efc843ff700 1 --2- 172.21.15.27:0/2275045577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc7c0ca3e0 0x7efc7c15d9d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efc6c0090b0 tx=0x7efc6c017000 comp rx=0 tx=0).stop 2024-01-11T08:10:45.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.351+0000 7efc843ff700 1 -- 172.21.15.27:0/2275045577 shutdown_connections 2024-01-11T08:10:45.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.351+0000 7efc843ff700 1 --2- 172.21.15.27:0/2275045577 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efc60066f30 0x7efc600693e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.351+0000 7efc843ff700 1 --2- 172.21.15.27:0/2275045577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc7c0ca7b0 0x7efc7c15e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.351+0000 7efc843ff700 1 --2- 172.21.15.27:0/2275045577 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc7c0c8a20 0x7efc7c140f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.351+0000 7efc843ff700 1 --2- 172.21.15.27:0/2275045577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc7c0ca3e0 0x7efc7c15d9d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.351+0000 7efc843ff700 1 -- 172.21.15.27:0/2275045577 >> 172.21.15.27:0/2275045577 conn(0x7efc7c0ce3e0 msgr2=0x7efc7c06d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:45.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.351+0000 7efc843ff700 1 -- 172.21.15.27:0/2275045577 shutdown_connections 2024-01-11T08:10:45.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.351+0000 7efc843ff700 1 -- 172.21.15.27:0/2275045577 wait complete. 2024-01-11T08:10:45.352 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 8 2024-01-11T08:10:45.771 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-11T08:10:30.160827+0000', 'modified': '2024-01-11T08:10:33.681210+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4291}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4291': {'gid': 4291, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6836/269464429', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 269464429}, {'type': 'v1', 'addr': '172.21.15.77:6836', 'nonce': 269464429}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-11T08:10:45.771 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-11T08:10:45.771 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-01-11T08:10:45.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.973+0000 7fdc0c5ff700 1 Processor -- start 2024-01-11T08:10:45.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.973+0000 7fdc0c5ff700 1 -- start start 2024-01-11T08:10:45.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.974+0000 7fdc0c5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdc04068cd0 0x7fdc0415a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:45.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.975+0000 7fdc0c5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc040b4bd0 con 0x7fdc0415cde0 2024-01-11T08:10:45.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.975+0000 7fdc0c5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc0415ac70 con 0x7fdc0415ca10 2024-01-11T08:10:45.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.975+0000 7fdc0c5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc040b3fc0 con 0x7fdc04068cd0 2024-01-11T08:10:45.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.975+0000 7fdc0ab9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdc0415cde0 0x7fdc040b5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51390/0 (socket says 172.21.15.27:51390) 2024-01-11T08:10:45.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.975+0000 7fdc0ab9c700 1 -- 172.21.15.27:0/448063054 learned_addr learned my addr 172.21.15.27:0/448063054 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:45.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.975+0000 7fdc09b9a700 1 --2- 172.21.15.27:0/448063054 >> v2:172.21.15.77:3300/0 conn(0x7fdc04068cd0 0x7fdc0415a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:45.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.975+0000 7fdc09399700 1 -- 172.21.15.27:0/448063054 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126519114 0 0) 0x7fdc040b4bd0 con 0x7fdc0415cde0 2024-01-11T08:10:45.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.975+0000 7fdc09399700 1 -- 172.21.15.27:0/448063054 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf0003570 con 0x7fdc0415cde0 2024-01-11T08:10:45.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.975+0000 7fdc09b9a700 1 -- 172.21.15.27:0/448063054 >> v1:172.21.15.77:6789/0 conn(0x7fdc0415ca10 legacy=0x7fdc040685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:45.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.975+0000 7fdc09b9a700 1 -- 172.21.15.27:0/448063054 >> v1:172.21.15.27:6789/0 conn(0x7fdc0415cde0 legacy=0x7fdc040b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:45.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.975+0000 7fdc09b9a700 1 -- 172.21.15.27:0/448063054 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc040b42a0 con 0x7fdc04068cd0 2024-01-11T08:10:45.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.976+0000 7fdc09b9a700 1 --2- 172.21.15.27:0/448063054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc04068cd0 0x7fdc0415a730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4009910 tx=0x7fdbf40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98139791fc3bedbd server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:45.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.976+0000 7fdc09399700 1 -- 172.21.15.27:0/448063054 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf400b640 con 0x7fdc04068cd0 2024-01-11T08:10:45.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.976+0000 7fdc09399700 1 -- 172.21.15.27:0/448063054 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf400b7a0 con 0x7fdc04068cd0 2024-01-11T08:10:45.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.976+0000 7fdc09399700 1 -- 172.21.15.27:0/448063054 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf400bb60 con 0x7fdc04068cd0 2024-01-11T08:10:45.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.977+0000 7fdc0c5ff700 1 -- 172.21.15.27:0/448063054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc04068cd0 msgr2=0x7fdc0415a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:45.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.977+0000 7fdc0c5ff700 1 --2- 172.21.15.27:0/448063054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc04068cd0 0x7fdc0415a730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4009910 tx=0x7fdbf40091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.977+0000 7fdc0c5ff700 1 -- 172.21.15.27:0/448063054 shutdown_connections 2024-01-11T08:10:45.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.977+0000 7fdc0c5ff700 1 --2- 172.21.15.27:0/448063054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc04068cd0 0x7fdc0415a730 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.977+0000 7fdc0c5ff700 1 -- 172.21.15.27:0/448063054 >> 172.21.15.27:0/448063054 conn(0x7fdc0406ed40 msgr2=0x7fdc0406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:45.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.977+0000 7fdc0c5ff700 1 -- 172.21.15.27:0/448063054 shutdown_connections 2024-01-11T08:10:45.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.977+0000 7fdc0c5ff700 1 -- 172.21.15.27:0/448063054 wait complete. 2024-01-11T08:10:45.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.978+0000 7fdc0c5ff700 1 Processor -- start 2024-01-11T08:10:45.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.979+0000 7fdc0c5ff700 1 -- start start 2024-01-11T08:10:45.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.980+0000 7fdc0c5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc04068cd0 0x7fdc041527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.980+0000 7fdc0c5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc0415ca10 0x7fdc0414e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.980+0000 7fdc0c5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc0415cde0 0x7fdc04154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.980+0000 7fdc0c5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0416da40 con 0x7fdc0415ca10 2024-01-11T08:10:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.980+0000 7fdc0c5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0416dd20 con 0x7fdc04068cd0 2024-01-11T08:10:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.980+0000 7fdc0c5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0416e000 con 0x7fdc0415cde0 2024-01-11T08:10:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.980+0000 7fdc09b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc0415ca10 0x7fdc0414e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.980+0000 7fdc09b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc0415ca10 0x7fdc0414e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54710/0 (socket says 172.21.15.27:54710) 2024-01-11T08:10:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.980+0000 7fdc09b9a700 1 -- 172.21.15.27:0/1557585685 learned_addr learned my addr 172.21.15.27:0/1557585685 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.980+0000 7fdc0a39b700 1 --2- 172.21.15.27:0/1557585685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc04068cd0 0x7fdc041527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.980+0000 7fdc0ab9c700 1 --2- 172.21.15.27:0/1557585685 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc0415cde0 0x7fdc04154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.980+0000 7fdc09b9a700 1 -- 172.21.15.27:0/1557585685 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc0415cde0 msgr2=0x7fdc04154b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.980+0000 7fdc09b9a700 1 --2- 172.21.15.27:0/1557585685 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc0415cde0 0x7fdc04154b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.980+0000 7fdc09b9a700 1 -- 172.21.15.27:0/1557585685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc04068cd0 msgr2=0x7fdc041527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:45.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.980+0000 7fdc09b9a700 1 --2- 172.21.15.27:0/1557585685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc04068cd0 0x7fdc041527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.980+0000 7fdc09b9a700 1 -- 172.21.15.27:0/1557585685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf4009420 con 0x7fdc0415ca10 2024-01-11T08:10:45.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.981+0000 7fdc09b9a700 1 --2- 172.21.15.27:0/1557585685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc0415ca10 0x7fdc0414e280 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdbf40093f0 tx=0x7fdbf4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:45.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.981+0000 7fdbfa7fc700 1 -- 172.21.15.27:0/1557585685 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf401f070 con 0x7fdc0415ca10 2024-01-11T08:10:45.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.981+0000 7fdbfa7fc700 1 -- 172.21.15.27:0/1557585685 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf4000bc0 con 0x7fdc0415ca10 2024-01-11T08:10:45.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.981+0000 7fdbfa7fc700 1 -- 172.21.15.27:0/1557585685 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf400bb60 con 0x7fdc0415ca10 2024-01-11T08:10:45.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.981+0000 7fdc0c5ff700 1 -- 172.21.15.27:0/1557585685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc0416e2e0 con 0x7fdc0415ca10 2024-01-11T08:10:45.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.981+0000 7fdc0c5ff700 1 -- 172.21.15.27:0/1557585685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc04152d00 con 0x7fdc0415ca10 2024-01-11T08:10:45.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.983+0000 7fdbfa7fc700 1 -- 172.21.15.27:0/1557585685 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fdbf4023070 con 0x7fdc0415ca10 2024-01-11T08:10:45.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.984+0000 7fdbfa7fc700 1 --2- 172.21.15.27:0/1557585685 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdbe8066f30 0x7fdbe80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:45.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.984+0000 7fdbfa7fc700 1 -- 172.21.15.27:0/1557585685 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fdbf4085710 con 0x7fdc0415ca10 2024-01-11T08:10:45.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.984+0000 7fdc0a39b700 1 --2- 172.21.15.27:0/1557585685 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdbe8066f30 0x7fdbe80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:45.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.985+0000 7fdc0a39b700 1 --2- 172.21.15.27:0/1557585685 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdbe8066f30 0x7fdbe80693e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdc00003a80 tx=0x7fdc00007ec0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:45.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.986+0000 7fdc1ace4b80 1 Processor -- start 2024-01-11T08:10:45.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.986+0000 7fdc1ace4b80 1 -- start start 2024-01-11T08:10:45.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.987+0000 7fdc1ace4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57e6b20 0x55b2d57c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:45.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.987+0000 7fdc1ace4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57c23a0 0x55b2d57ee860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:45.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.987+0000 7fdc1ace4b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57e6ef0 0x55b2d57eeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:45.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.987+0000 7fdc1ace4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2d56df310 con 0x55b2d57c23a0 2024-01-11T08:10:45.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.987+0000 7fdc1ace4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2d571c130 con 0x55b2d57e6b20 2024-01-11T08:10:45.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.987+0000 7fdc1ace4b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b2d5719af0 con 0x55b2d57e6ef0 2024-01-11T08:10:45.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.987+0000 7fdc09b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57c23a0 0x55b2d57ee860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:45.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.987+0000 7fdc09b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57c23a0 0x55b2d57ee860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54718/0 (socket says 172.21.15.27:54718) 2024-01-11T08:10:45.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.987+0000 7fdc09b9a700 1 -- 172.21.15.27:0/646928414 learned_addr learned my addr 172.21.15.27:0/646928414 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:45.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.987+0000 7fdc0a39b700 1 --2- 172.21.15.27:0/646928414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57e6b20 0x55b2d57c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:45.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.987+0000 7fdc0ab9c700 1 --2- 172.21.15.27:0/646928414 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57e6ef0 0x55b2d57eeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:45.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.988+0000 7fdc09b9a700 1 -- 172.21.15.27:0/646928414 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57e6ef0 msgr2=0x55b2d57eeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:45.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.988+0000 7fdc09b9a700 1 --2- 172.21.15.27:0/646928414 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57e6ef0 0x55b2d57eeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.988+0000 7fdc09b9a700 1 -- 172.21.15.27:0/646928414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57e6b20 msgr2=0x55b2d57c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:45.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.988+0000 7fdc09b9a700 1 --2- 172.21.15.27:0/646928414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57e6b20 0x55b2d57c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.988+0000 7fdc09b9a700 1 -- 172.21.15.27:0/646928414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b2d56080a0 con 0x55b2d57c23a0 2024-01-11T08:10:45.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.988+0000 7fdc0a39b700 1 --2- 172.21.15.27:0/646928414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57e6b20 0x55b2d57c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:45.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.988+0000 7fdc0ab9c700 1 --2- 172.21.15.27:0/646928414 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57e6ef0 0x55b2d57eeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:45.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.988+0000 7fdc09b9a700 1 --2- 172.21.15.27:0/646928414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57c23a0 0x55b2d57ee860 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4006c60 tx=0x7fdbf4083af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=820db68b9c6277c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:45.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.988+0000 7fdbe77fe700 1 -- 172.21.15.27:0/646928414 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4056d20 con 0x55b2d57c23a0 2024-01-11T08:10:45.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.988+0000 7fdbe77fe700 1 -- 172.21.15.27:0/646928414 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf40565a0 con 0x55b2d57c23a0 2024-01-11T08:10:45.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.988+0000 7fdbe77fe700 1 -- 172.21.15.27:0/646928414 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40597f0 con 0x55b2d57c23a0 2024-01-11T08:10:45.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.989+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/646928414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57c23a0 msgr2=0x55b2d57ee860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:45.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.989+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/646928414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57c23a0 0x55b2d57ee860 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4006c60 tx=0x7fdbf4083af0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.989+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/646928414 shutdown_connections 2024-01-11T08:10:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.989+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/646928414 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57e6ef0 0x55b2d57eeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.989+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/646928414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57e6b20 0x55b2d57c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.989+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/646928414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57c23a0 0x55b2d57ee860 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.989+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/646928414 >> 172.21.15.27:0/646928414 conn(0x55b2d57c4300 msgr2=0x55b2d57c6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.989+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/646928414 shutdown_connections 2024-01-11T08:10:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.989+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/646928414 wait complete. 2024-01-11T08:10:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.990+0000 7fdc1ace4b80 1 -- start start 2024-01-11T08:10:45.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.990+0000 7fdc1ace4b80 1 Processor -- start 2024-01-11T08:10:45.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.992+0000 7fdc1ace4b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57c23a0 0x55b2d57d5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:45.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.992+0000 7fdc1ace4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57e6b20 0x55b2d57f2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:45.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.992+0000 7fdc1ace4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57e6ef0 0x55b2d57e73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:45.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.992+0000 7fdc1ace4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2d57158e0 con 0x55b2d57e6b20 2024-01-11T08:10:45.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.992+0000 7fdc1ace4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2d56c1980 con 0x55b2d57e6ef0 2024-01-11T08:10:45.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.992+0000 7fdc1ace4b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b2d56c1620 con 0x55b2d57c23a0 2024-01-11T08:10:45.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.992+0000 7fdc1ace4b80 10 client.0 _ll_register_callbacks cb 0x55b2d5524a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T08:10:45.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.992+0000 7fdc09b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57e6b20 0x55b2d57f2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:45.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.992+0000 7fdc09b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57e6b20 0x55b2d57f2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54722/0 (socket says 172.21.15.27:54722) 2024-01-11T08:10:45.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.992+0000 7fdc09b9a700 1 -- 172.21.15.27:0/2451435534 learned_addr learned my addr 172.21.15.27:0/2451435534 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:45.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.992+0000 7fdc0a39b700 1 --2- 172.21.15.27:0/2451435534 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57c23a0 0x55b2d57d5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:45.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.992+0000 7fdc0ab9c700 1 --2- 172.21.15.27:0/2451435534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57e6ef0 0x55b2d57e73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:45.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.992+0000 7fdc09b9a700 1 -- 172.21.15.27:0/2451435534 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57c23a0 msgr2=0x55b2d57d5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:45.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.992+0000 7fdc09b9a700 1 --2- 172.21.15.27:0/2451435534 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57c23a0 0x55b2d57d5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.992+0000 7fdc09b9a700 1 -- 172.21.15.27:0/2451435534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57e6ef0 msgr2=0x55b2d57e73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:45.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.992+0000 7fdc09b9a700 1 --2- 172.21.15.27:0/2451435534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57e6ef0 0x55b2d57e73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.992+0000 7fdc09b9a700 1 -- 172.21.15.27:0/2451435534 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc04152d00 con 0x55b2d57e6b20 2024-01-11T08:10:45.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.992+0000 7fdc0a39b700 1 --2- 172.21.15.27:0/2451435534 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57c23a0 0x55b2d57d5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:45.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.992+0000 7fdc0ab9c700 1 --2- 172.21.15.27:0/2451435534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57e6ef0 0x55b2d57e73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:45.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.993+0000 7fdc09b9a700 1 --2- 172.21.15.27:0/2451435534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57e6b20 0x55b2d57f2710 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4008010 tx=0x7fdbf406a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10005c95cf7bd76d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:45.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.993+0000 7fdbe4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T08:10:45.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.993+0000 7fdbe4ff9700 1 -- 172.21.15.27:0/2451435534 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4071070 con 0x55b2d57e6b20 2024-01-11T08:10:45.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.993+0000 7fdbe4ff9700 1 -- 172.21.15.27:0/2451435534 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fdbf40597f0 con 0x55b2d57e6b20 2024-01-11T08:10:45.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.993+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/2451435534 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b2d57ba560 con 0x55b2d57e6b20 2024-01-11T08:10:45.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.993+0000 7fdbe4ff9700 1 -- 172.21.15.27:0/2451435534 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdbf4072220 con 0x55b2d57e6b20 2024-01-11T08:10:45.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.993+0000 7fdc1ace4b80 10 client.4611 fetch_fsmap learned FSMap version 8 2024-01-11T08:10:45.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.993+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/2451435534 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55b2d56e43c0 con 0x55b2d57e6b20 2024-01-11T08:10:45.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.993+0000 7fdbe4ff9700 1 -- 172.21.15.27:0/2451435534 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf4061080 con 0x55b2d57e6b20 2024-01-11T08:10:45.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.993+0000 7fdbe4ff9700 1 -- 172.21.15.27:0/2451435534 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 8) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fdbf405d0b0 con 0x55b2d57e6b20 2024-01-11T08:10:45.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.993+0000 7fdc1ace4b80 10 client.4611 fetch_fsmap finished waiting for FSMap version 8 2024-01-11T08:10:45.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.993+0000 7fdc1ace4b80 10 client.4611 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-11T08:10:45.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.993+0000 7fdc1ace4b80 20 client.4611 populate_metadata read hostname 'smithi027' 2024-01-11T08:10:45.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.993+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/2451435534 >> [v2:172.21.15.77:6834/269464429,v1:172.21.15.77:6836/269464429] conn(0x55b2d57f4e40 0x55b2d57f5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:45.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.993+0000 7fdc1ace4b80 4 client.4611 mds_command: new command op to 4291 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-11T08:10:45.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.993+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/2451435534 --> [v2:172.21.15.77:6834/269464429,v1:172.21.15.77:6836/269464429] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b2d57970e0 con 0x55b2d57f4e40 2024-01-11T08:10:45.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.994+0000 7fdc0a39b700 1 --2- 172.21.15.27:0/2451435534 >> [v2:172.21.15.77:6834/269464429,v1:172.21.15.77:6836/269464429] conn(0x55b2d57f4e40 0x55b2d57f5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:45.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.994+0000 7fdc0a39b700 1 --2- 172.21.15.27:0/2451435534 >> [v2:172.21.15.77:6834/269464429,v1:172.21.15.77:6836/269464429] conn(0x55b2d57f4e40 0x55b2d57f5210 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a8ae52aff6bfec7 server_cookie=dd1bbde41c3444d8 in_seq=0 out_seq=0 2024-01-11T08:10:45.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.994+0000 7fdbe4ff9700 10 client.4611 ms_handle_connect on v2:172.21.15.77:6834/269464429 2024-01-11T08:10:45.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.995+0000 7fdbe4ff9700 1 -- 172.21.15.27:0/2451435534 <== mds.0 v2:172.21.15.77:6834/269464429 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x55b2d57970e0 con 0x55b2d57f4e40 2024-01-11T08:10:45.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.995+0000 7fdbe4ff9700 10 client.4611 handle_command_reply: tid=0 2024-01-11T08:10:45.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.995+0000 7fdc1ace4b80 1 client.4611 shutdown 2024-01-11T08:10:45.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.996+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/2451435534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57e6b20 msgr2=0x55b2d57f2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:45.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.996+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/2451435534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57e6b20 0x55b2d57f2710 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4008010 tx=0x7fdbf406a000 comp rx=0 tx=0).stop 2024-01-11T08:10:45.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.996+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/2451435534 shutdown_connections 2024-01-11T08:10:45.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.996+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/2451435534 >> [v2:172.21.15.77:6834/269464429,v1:172.21.15.77:6836/269464429] conn(0x55b2d57f4e40 0x55b2d57f5210 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:45.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.996+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/2451435534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57e6ef0 0x55b2d57e73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.996+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/2451435534 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57c23a0 0x55b2d57d5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.996+0000 7fdbe4ff9700 0 client.4611 ms_handle_reset on v2:172.21.15.77:6834/269464429 2024-01-11T08:10:46.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.996+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/2451435534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57e6b20 0x55b2d57f2710 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.996+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/2451435534 >> 172.21.15.27:0/2451435534 conn(0x55b2d57c4300 msgr2=0x55b2d57c1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:46.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.996+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/2451435534 shutdown_connections 2024-01-11T08:10:46.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.996+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/2451435534 wait complete. 2024-01-11T08:10:46.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:45.996+0000 7fdc1ace4b80 20 client.4611 trim_cache size 0 max 16384 2024-01-11T08:10:46.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.014+0000 7fdc1ace4b80 1 Processor -- start 2024-01-11T08:10:46.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.014+0000 7fdc1ace4b80 1 -- start start 2024-01-11T08:10:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.015+0000 7fdc1ace4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57c23a0 0x55b2d581b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.015+0000 7fdc1ace4b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57e6b20 0x55b2d57c16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.015+0000 7fdc1ace4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57e6ef0 0x55b2d57d5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.015+0000 7fdc1ace4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2d55ad1e0 con 0x55b2d57e6ef0 2024-01-11T08:10:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.015+0000 7fdc1ace4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2d57f4640 con 0x55b2d57c23a0 2024-01-11T08:10:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.015+0000 7fdc1ace4b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b2d57c4ca0 con 0x55b2d57e6b20 2024-01-11T08:10:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.015+0000 7fdc0ab9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57e6ef0 0x55b2d57d5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.015+0000 7fdc0ab9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57e6ef0 0x55b2d57d5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54732/0 (socket says 172.21.15.27:54732) 2024-01-11T08:10:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.015+0000 7fdc0ab9c700 1 -- 172.21.15.27:0/2045199247 learned_addr learned my addr 172.21.15.27:0/2045199247 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.015+0000 7fdc09b9a700 1 --2- 172.21.15.27:0/2045199247 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57e6b20 0x55b2d57c16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.015+0000 7fdc0a39b700 1 --2- 172.21.15.27:0/2045199247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57c23a0 0x55b2d581b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.016+0000 7fdc0ab9c700 1 -- 172.21.15.27:0/2045199247 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57e6b20 msgr2=0x55b2d57c16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.016+0000 7fdc0ab9c700 1 --2- 172.21.15.27:0/2045199247 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57e6b20 0x55b2d57c16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.016+0000 7fdc0ab9c700 1 -- 172.21.15.27:0/2045199247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57c23a0 msgr2=0x55b2d581b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:46.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.016+0000 7fdc0ab9c700 1 --2- 172.21.15.27:0/2045199247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57c23a0 0x55b2d581b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.016+0000 7fdc0ab9c700 1 -- 172.21.15.27:0/2045199247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b2d557c1d0 con 0x55b2d57e6ef0 2024-01-11T08:10:46.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.016+0000 7fdc0ab9c700 1 --2- 172.21.15.27:0/2045199247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57e6ef0 0x55b2d57d5780 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0021f0 tx=0x7fdbfc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbdc6bdb58ebf1f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:46.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.016+0000 7fdbe7fff700 1 -- 172.21.15.27:0/2045199247 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc007aa0 con 0x55b2d57e6ef0 2024-01-11T08:10:46.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.016+0000 7fdbe7fff700 1 -- 172.21.15.27:0/2045199247 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbfc016450 con 0x55b2d57e6ef0 2024-01-11T08:10:46.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.016+0000 7fdbe7fff700 1 -- 172.21.15.27:0/2045199247 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc0167d0 con 0x55b2d57e6ef0 2024-01-11T08:10:46.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.017+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/2045199247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57e6ef0 msgr2=0x55b2d57d5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:46.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.017+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/2045199247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57e6ef0 0x55b2d57d5780 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0021f0 tx=0x7fdbfc009040 comp rx=0 tx=0).stop 2024-01-11T08:10:46.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.017+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/2045199247 shutdown_connections 2024-01-11T08:10:46.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.017+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/2045199247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57e6ef0 0x55b2d57d5780 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.017+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/2045199247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57c23a0 0x55b2d581b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.017+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/2045199247 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57e6b20 0x55b2d57c16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.017+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/2045199247 >> 172.21.15.27:0/2045199247 conn(0x55b2d57c4300 msgr2=0x55b2d57c35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:46.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.017+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/2045199247 shutdown_connections 2024-01-11T08:10:46.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.017+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/2045199247 wait complete. 2024-01-11T08:10:46.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.017+0000 7fdc1ace4b80 1 -- start start 2024-01-11T08:10:46.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.018+0000 7fdc1ace4b80 1 Processor -- start 2024-01-11T08:10:46.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.019+0000 7fdc1ace4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57c23a0 0x55b2d58556e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:46.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.019+0000 7fdc1ace4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57e6b20 0x55b2d5810160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:46.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.019+0000 7fdc1ace4b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57e6ef0 0x55b2d580b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:46.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.019+0000 7fdc1ace4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2d5871ef0 con 0x55b2d57c23a0 2024-01-11T08:10:46.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.019+0000 7fdc1ace4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2d58106a0 con 0x55b2d57e6b20 2024-01-11T08:10:46.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.019+0000 7fdc1ace4b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b2d57f2710 con 0x55b2d57e6ef0 2024-01-11T08:10:46.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.019+0000 7fdc1ace4b80 10 client.0 _ll_register_callbacks cb 0x55b2d5524a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T08:10:46.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.019+0000 7fdc0a39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57c23a0 0x55b2d58556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:46.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.019+0000 7fdc0a39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57c23a0 0x55b2d58556e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54748/0 (socket says 172.21.15.27:54748) 2024-01-11T08:10:46.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.019+0000 7fdc0a39b700 1 -- 172.21.15.27:0/3588049618 learned_addr learned my addr 172.21.15.27:0/3588049618 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:46.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.019+0000 7fdc09b9a700 1 --2- 172.21.15.27:0/3588049618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57e6b20 0x55b2d5810160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:46.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.019+0000 7fdc0ab9c700 1 --2- 172.21.15.27:0/3588049618 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57e6ef0 0x55b2d580b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:46.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.019+0000 7fdc0a39b700 1 -- 172.21.15.27:0/3588049618 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57e6ef0 msgr2=0x55b2d580b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:46.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.019+0000 7fdc0a39b700 1 --2- 172.21.15.27:0/3588049618 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57e6ef0 0x55b2d580b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.019+0000 7fdc0a39b700 1 -- 172.21.15.27:0/3588049618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57e6b20 msgr2=0x55b2d5810160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:46.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.019+0000 7fdc0a39b700 1 --2- 172.21.15.27:0/3588049618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57e6b20 0x55b2d5810160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.019+0000 7fdc0a39b700 1 -- 172.21.15.27:0/3588049618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf0003570 con 0x55b2d57c23a0 2024-01-11T08:10:46.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.020+0000 7fdc0a39b700 1 --2- 172.21.15.27:0/3588049618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57c23a0 0x55b2d58556e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x55b2d56c1710 tx=0x7fdc00016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf599c3d8b11f55 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:46.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.020+0000 7fdbe4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T08:10:46.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.020+0000 7fdbe4ff9700 1 -- 172.21.15.27:0/3588049618 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0002c070 con 0x55b2d57c23a0 2024-01-11T08:10:46.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.020+0000 7fdbe4ff9700 1 -- 172.21.15.27:0/3588049618 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fdc0001c040 con 0x55b2d57c23a0 2024-01-11T08:10:46.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.020+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/3588049618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdbf40565a0 con 0x55b2d57c23a0 2024-01-11T08:10:46.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.020+0000 7fdbe4ff9700 1 -- 172.21.15.27:0/3588049618 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc00027bd0 con 0x55b2d57c23a0 2024-01-11T08:10:46.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.020+0000 7fdc1ace4b80 10 client.4617 fetch_fsmap learned FSMap version 8 2024-01-11T08:10:46.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.020+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/3588049618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55b2d5555d60 con 0x55b2d57c23a0 2024-01-11T08:10:46.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.021+0000 7fdbe4ff9700 1 -- 172.21.15.27:0/3588049618 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf40565a0 con 0x55b2d57c23a0 2024-01-11T08:10:46.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.021+0000 7fdbe4ff9700 1 -- 172.21.15.27:0/3588049618 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 8) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fdc00002050 con 0x55b2d57c23a0 2024-01-11T08:10:46.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.021+0000 7fdc1ace4b80 10 client.4617 fetch_fsmap finished waiting for FSMap version 8 2024-01-11T08:10:46.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.021+0000 7fdc1ace4b80 10 client.4617 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-11T08:10:46.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.021+0000 7fdc1ace4b80 20 client.4617 populate_metadata read hostname 'smithi027' 2024-01-11T08:10:46.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.021+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/3588049618 >> [v2:172.21.15.77:6834/269464429,v1:172.21.15.77:6836/269464429] conn(0x55b2d580a710 0x55b2d580aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:46.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.021+0000 7fdc1ace4b80 4 client.4617 mds_command: new command op to 4291 tid=0[{"prefix": "status"}] 2024-01-11T08:10:46.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.021+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/3588049618 --> [v2:172.21.15.77:6834/269464429,v1:172.21.15.77:6836/269464429] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b2d56b88d0 con 0x55b2d580a710 2024-01-11T08:10:46.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.021+0000 7fdc09b9a700 1 --2- 172.21.15.27:0/3588049618 >> [v2:172.21.15.77:6834/269464429,v1:172.21.15.77:6836/269464429] conn(0x55b2d580a710 0x55b2d580aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:46.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.022+0000 7fdc09b9a700 1 --2- 172.21.15.27:0/3588049618 >> [v2:172.21.15.77:6834/269464429,v1:172.21.15.77:6836/269464429] conn(0x55b2d580a710 0x55b2d580aae0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=472ddec28c413b55 server_cookie=c9197395278406d6 in_seq=0 out_seq=0 2024-01-11T08:10:46.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.022+0000 7fdbe4ff9700 10 client.4617 ms_handle_connect on v2:172.21.15.77:6834/269464429 2024-01-11T08:10:46.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.022+0000 7fdbe4ff9700 1 -- 172.21.15.27:0/3588049618 <== mds.0 v2:172.21.15.77:6834/269464429 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x55b2d56b88d0 con 0x55b2d580a710 2024-01-11T08:10:46.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.022+0000 7fdbe4ff9700 10 client.4617 handle_command_reply: tid=0 2024-01-11T08:10:46.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.022+0000 7fdc1ace4b80 1 client.4617 shutdown 2024-01-11T08:10:46.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.022+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/3588049618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57c23a0 msgr2=0x55b2d58556e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:46.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.022+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/3588049618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57c23a0 0x55b2d58556e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x55b2d56c1710 tx=0x7fdc00016040 comp rx=0 tx=0).stop 2024-01-11T08:10:46.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.022+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/3588049618 shutdown_connections 2024-01-11T08:10:46.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.022+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/3588049618 >> [v2:172.21.15.77:6834/269464429,v1:172.21.15.77:6836/269464429] conn(0x55b2d580a710 0x55b2d580aae0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.022+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/3588049618 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55b2d57e6ef0 0x55b2d580b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.022+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/3588049618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b2d57c23a0 0x55b2d58556e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.022+0000 7fdc1ace4b80 1 --2- 172.21.15.27:0/3588049618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b2d57e6b20 0x55b2d5810160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.022+0000 7fdbe4ff9700 0 client.4617 ms_handle_reset on v2:172.21.15.77:6834/269464429 2024-01-11T08:10:46.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.022+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/3588049618 >> 172.21.15.27:0/3588049618 conn(0x55b2d57c4300 msgr2=0x55b2d57f1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:46.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.022+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/3588049618 shutdown_connections 2024-01-11T08:10:46.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.022+0000 7fdc1ace4b80 1 -- 172.21.15.27:0/3588049618 wait complete. 2024-01-11T08:10:46.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.022+0000 7fdc1ace4b80 20 client.4617 trim_cache size 0 max 16384 2024-01-11T08:10:46.027 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T08:10:46.028 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "8fddb4a7-9d66-40f1-892d-6fe86c728ee4", 2024-01-11T08:10:46.028 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-11T08:10:46.028 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4291, 2024-01-11T08:10:46.028 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-11T08:10:46.028 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-11T08:10:46.028 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-11T08:10:46.028 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 15.356918953999999, 2024-01-11T08:10:46.028 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 8, 2024-01-11T08:10:46.028 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 21, 2024-01-11T08:10:46.028 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 21, 2024-01-11T08:10:46.028 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 21.056255158999999 2024-01-11T08:10:46.029 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T08:10:46.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.023+0000 7fdc0c5ff700 1 -- 172.21.15.27:0/1557585685 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdbe8066f30 msgr2=0x7fdbe80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:46.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.023+0000 7fdc0c5ff700 1 --2- 172.21.15.27:0/1557585685 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdbe8066f30 0x7fdbe80693e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdc00003a80 tx=0x7fdc00007ec0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.023+0000 7fdc0c5ff700 1 -- 172.21.15.27:0/1557585685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc0415ca10 msgr2=0x7fdc0414e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:46.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.023+0000 7fdc0c5ff700 1 --2- 172.21.15.27:0/1557585685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc0415ca10 0x7fdc0414e280 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdbf40093f0 tx=0x7fdbf4017000 comp rx=0 tx=0).stop 2024-01-11T08:10:46.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.023+0000 7fdc0c5ff700 1 -- 172.21.15.27:0/1557585685 shutdown_connections 2024-01-11T08:10:46.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.023+0000 7fdc0c5ff700 1 --2- 172.21.15.27:0/1557585685 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdbe8066f30 0x7fdbe80693e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.023+0000 7fdc0c5ff700 1 --2- 172.21.15.27:0/1557585685 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc0415cde0 0x7fdc04154b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.023+0000 7fdc0c5ff700 1 --2- 172.21.15.27:0/1557585685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc04068cd0 0x7fdc041527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.023+0000 7fdc0c5ff700 1 --2- 172.21.15.27:0/1557585685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc0415ca10 0x7fdc0414e280 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.023+0000 7fdc0c5ff700 1 -- 172.21.15.27:0/1557585685 >> 172.21.15.27:0/1557585685 conn(0x7fdc0406ed40 msgr2=0x7fdc0414f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:46.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.023+0000 7fdc0c5ff700 1 -- 172.21.15.27:0/1557585685 shutdown_connections 2024-01-11T08:10:46.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.023+0000 7fdc0c5ff700 1 -- 172.21.15.27:0/1557585685 wait complete. 2024-01-11T08:10:46.040 INFO:teuthology.run_tasks:Running task kclient... 2024-01-11T08:10:46.063 INFO:tasks.kclient:Mounting kernel clients... 2024-01-11T08:10:46.063 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-01-11T08:10:46.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:46.064 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T08:10:46.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:46.113 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2024-01-11T08:10:46.189 INFO:teuthology.orchestra.run.smithi027.stderr:Cannot find device "ceph-brx" 2024-01-11T08:10:46.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T08:10:46.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:46.193 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T08:10:46.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:46.241 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2024-01-11T08:10:46.315 INFO:teuthology.orchestra.run.smithi027.stderr:Cannot find device "ceph-brx" 2024-01-11T08:10:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T08:10:46.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:46.318 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-01-11T08:10:46.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:46.372 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2024-01-11T08:10:46.448 INFO:teuthology.orchestra.run.smithi077.stderr:Cannot find device "ceph-brx" 2024-01-11T08:10:46.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T08:10:46.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:46.451 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-01-11T08:10:46.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:46.502 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2024-01-11T08:10:46.573 INFO:teuthology.orchestra.run.smithi077.stderr:Cannot find device "ceph-brx" 2024-01-11T08:10:46.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T08:10:46.576 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-01-11T08:10:46.577 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-11T08:10:46.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.776+0000 7efc9c7ff700 1 Processor -- start 2024-01-11T08:10:46.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.777+0000 7efc9c7ff700 1 -- start start 2024-01-11T08:10:46.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.778+0000 7efc9c7ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efc940b03e0 0x7efc940b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:46.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.778+0000 7efc9c7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc940b07b0 0x7efc940b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:46.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.778+0000 7efc9c7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efc94159200 0x7efc94159640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:46.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.778+0000 7efc9c7ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc940cd730 con 0x7efc940b03e0 2024-01-11T08:10:46.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.778+0000 7efc9c7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc940ccb20 con 0x7efc94159200 2024-01-11T08:10:46.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.778+0000 7efc9c7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc940cce00 con 0x7efc940b07b0 2024-01-11T08:10:46.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.778+0000 7efc9a59b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efc940b03e0 0x7efc940b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:46.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.778+0000 7efc99d9a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc940b07b0 0x7efc940b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:46.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.778+0000 7efc9a59b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efc940b03e0 0x7efc940b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54752/0 (socket says 172.21.15.27:54752) 2024-01-11T08:10:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.778+0000 7efc9ad9c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efc94159200 0x7efc94159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.778+0000 7efc9a59b700 1 -- 172.21.15.27:0/4170614257 learned_addr learned my addr 172.21.15.27:0/4170614257 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.779+0000 7efc9a59b700 1 -- 172.21.15.27:0/4170614257 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc940b07b0 msgr2=0x7efc940b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.779+0000 7efc9a59b700 1 --2- 172.21.15.27:0/4170614257 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc940b07b0 0x7efc940b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.779+0000 7efc9a59b700 1 -- 172.21.15.27:0/4170614257 >> v2:172.21.15.77:3300/0 conn(0x7efc94159200 msgr2=0x7efc94159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.779+0000 7efc9a59b700 1 --2- 172.21.15.27:0/4170614257 >> v2:172.21.15.77:3300/0 conn(0x7efc94159200 0x7efc94159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.779+0000 7efc9a59b700 1 -- 172.21.15.27:0/4170614257 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc940cd0e0 con 0x7efc940b03e0 2024-01-11T08:10:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.779+0000 7efc9ad9c700 1 --2- 172.21.15.27:0/4170614257 >> v2:172.21.15.77:3300/0 conn(0x7efc94159200 0x7efc94159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.779+0000 7efc9a59b700 1 --2- 172.21.15.27:0/4170614257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc940b03e0 0x7efc940b2480 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7efc940b5f00 tx=0x7efc90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98364f20ca332a37 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.779+0000 7efc99599700 1 -- 172.21.15.27:0/4170614257 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9000b640 con 0x7efc940b03e0 2024-01-11T08:10:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.779+0000 7efc99599700 1 -- 172.21.15.27:0/4170614257 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9000b7a0 con 0x7efc940b03e0 2024-01-11T08:10:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.779+0000 7efc99599700 1 -- 172.21.15.27:0/4170614257 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9000bb60 con 0x7efc940b03e0 2024-01-11T08:10:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.780+0000 7efc9c7ff700 1 -- 172.21.15.27:0/4170614257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc940b03e0 msgr2=0x7efc940b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.780+0000 7efc9c7ff700 1 --2- 172.21.15.27:0/4170614257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc940b03e0 0x7efc940b2480 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7efc940b5f00 tx=0x7efc90009250 comp rx=0 tx=0).stop 2024-01-11T08:10:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.780+0000 7efc9c7ff700 1 -- 172.21.15.27:0/4170614257 shutdown_connections 2024-01-11T08:10:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.780+0000 7efc9c7ff700 1 --2- 172.21.15.27:0/4170614257 >> v2:172.21.15.77:3300/0 conn(0x7efc94159200 0x7efc94159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.780+0000 7efc9c7ff700 1 --2- 172.21.15.27:0/4170614257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc940b03e0 0x7efc940b2480 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.780+0000 7efc9c7ff700 1 --2- 172.21.15.27:0/4170614257 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc940b07b0 0x7efc940b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.780+0000 7efc9c7ff700 1 -- 172.21.15.27:0/4170614257 >> 172.21.15.27:0/4170614257 conn(0x7efc9406ed40 msgr2=0x7efc9406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.780+0000 7efc9c7ff700 1 -- 172.21.15.27:0/4170614257 shutdown_connections 2024-01-11T08:10:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.780+0000 7efc9c7ff700 1 -- 172.21.15.27:0/4170614257 wait complete. 2024-01-11T08:10:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.782+0000 7efc9c7ff700 1 Processor -- start 2024-01-11T08:10:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.782+0000 7efc9c7ff700 1 -- start start 2024-01-11T08:10:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.783+0000 7efc9c7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc940b03e0 0x7efc94157e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.784+0000 7efc9c7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc940b07b0 0x7efc940c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.784+0000 7efc9c7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc94159200 0x7efc940af2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.784+0000 7efc9c7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9415ae00 con 0x7efc94159200 2024-01-11T08:10:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.784+0000 7efc9c7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9415a820 con 0x7efc940b07b0 2024-01-11T08:10:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.784+0000 7efc9c7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc940c8150 con 0x7efc940b03e0 2024-01-11T08:10:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.784+0000 7efc9a59b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc940b03e0 0x7efc94157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.784+0000 7efc9ad9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc94159200 0x7efc940af2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.784+0000 7efc9a59b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc940b03e0 0x7efc94157e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:50190/0 (socket says 172.21.15.27:50190) 2024-01-11T08:10:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.784+0000 7efc9a59b700 1 -- 172.21.15.27:0/311101979 learned_addr learned my addr 172.21.15.27:0/311101979 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.784+0000 7efc99d9a700 1 --2- 172.21.15.27:0/311101979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc940b07b0 0x7efc940c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.784+0000 7efc9a59b700 1 -- 172.21.15.27:0/311101979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc940b07b0 msgr2=0x7efc940c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.784+0000 7efc9a59b700 1 --2- 172.21.15.27:0/311101979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc940b07b0 0x7efc940c75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.784+0000 7efc9a59b700 1 -- 172.21.15.27:0/311101979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc94159200 msgr2=0x7efc940af2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.784+0000 7efc9a59b700 1 --2- 172.21.15.27:0/311101979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc94159200 0x7efc940af2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.784+0000 7efc9a59b700 1 -- 172.21.15.27:0/311101979 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c009a20 con 0x7efc940b03e0 2024-01-11T08:10:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.784+0000 7efc9ad9c700 1 --2- 172.21.15.27:0/311101979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc94159200 0x7efc940af2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.785+0000 7efc9a59b700 1 --2- 172.21.15.27:0/311101979 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc940b03e0 0x7efc94157e80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efc94069540 tx=0x7efc90017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.785+0000 7efc8a7fc700 1 -- 172.21.15.27:0/311101979 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc90000d10 con 0x7efc940b03e0 2024-01-11T08:10:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.785+0000 7efc8a7fc700 1 -- 172.21.15.27:0/311101979 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9000bb20 con 0x7efc940b03e0 2024-01-11T08:10:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.785+0000 7efc8a7fc700 1 -- 172.21.15.27:0/311101979 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc90011630 con 0x7efc940b03e0 2024-01-11T08:10:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.785+0000 7efc9c7ff700 1 -- 172.21.15.27:0/311101979 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc900094f0 con 0x7efc940b03e0 2024-01-11T08:10:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.785+0000 7efc9c7ff700 1 -- 172.21.15.27:0/311101979 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc9415aa90 con 0x7efc940b03e0 2024-01-11T08:10:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.786+0000 7efc9c7ff700 1 -- 172.21.15.27:0/311101979 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc940718d0 con 0x7efc940b03e0 2024-01-11T08:10:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.787+0000 7efc8a7fc700 1 -- 172.21.15.27:0/311101979 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7efc900119a0 con 0x7efc940b03e0 2024-01-11T08:10:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.788+0000 7efc8a7fc700 1 --2- 172.21.15.27:0/311101979 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efc78066fb0 0x7efc78069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.788+0000 7efc8a7fc700 1 -- 172.21.15.27:0/311101979 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7efc90085750 con 0x7efc940b03e0 2024-01-11T08:10:46.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.788+0000 7efc99d9a700 1 --2- 172.21.15.27:0/311101979 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efc78066fb0 0x7efc78069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:46.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.789+0000 7efc99d9a700 1 --2- 172.21.15.27:0/311101979 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efc78066fb0 0x7efc78069460 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efc941583c0 tx=0x7efc840091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:46.793+0000 7efc8a7fc700 1 -- 172.21.15.27:0/311101979 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efc90052b10 con 0x7efc940b03e0 2024-01-11T08:10:47.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:47.043+0000 7efc9c7ff700 1 -- 172.21.15.27:0/311101979 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7efc940cd3c0 con 0x7efc940b03e0 2024-01-11T08:10:47.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:47.043+0000 7efc8a7fc700 1 -- 172.21.15.27:0/311101979 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7efc90056130 con 0x7efc940b03e0 2024-01-11T08:10:47.043 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-11T08:10:47.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:47.046+0000 7efc9c7ff700 1 -- 172.21.15.27:0/311101979 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efc78066fb0 msgr2=0x7efc78069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:47.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:47.046+0000 7efc9c7ff700 1 --2- 172.21.15.27:0/311101979 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efc78066fb0 0x7efc78069460 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efc941583c0 tx=0x7efc840091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:47.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:47.046+0000 7efc9c7ff700 1 -- 172.21.15.27:0/311101979 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc940b03e0 msgr2=0x7efc94157e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:47.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:47.046+0000 7efc9c7ff700 1 --2- 172.21.15.27:0/311101979 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc940b03e0 0x7efc94157e80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efc94069540 tx=0x7efc90017000 comp rx=0 tx=0).stop 2024-01-11T08:10:47.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:47.046+0000 7efc9c7ff700 1 -- 172.21.15.27:0/311101979 shutdown_connections 2024-01-11T08:10:47.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:47.046+0000 7efc9c7ff700 1 --2- 172.21.15.27:0/311101979 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efc78066fb0 0x7efc78069460 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:47.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:47.046+0000 7efc9c7ff700 1 --2- 172.21.15.27:0/311101979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc94159200 0x7efc940af2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:47.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:47.046+0000 7efc9c7ff700 1 --2- 172.21.15.27:0/311101979 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc940b03e0 0x7efc94157e80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:47.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:47.046+0000 7efc9c7ff700 1 --2- 172.21.15.27:0/311101979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc940b07b0 0x7efc940c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:47.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:47.046+0000 7efc9c7ff700 1 -- 172.21.15.27:0/311101979 >> 172.21.15.27:0/311101979 conn(0x7efc9406ed40 msgr2=0x7efc940d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:47.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:47.046+0000 7efc9c7ff700 1 -- 172.21.15.27:0/311101979 shutdown_connections 2024-01-11T08:10:47.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:47.046+0000 7efc9c7ff700 1 -- 172.21.15.27:0/311101979 wait complete. 2024-01-11T08:10:47.063 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-11T08:10:47.063 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-11T08:10:47.064 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-11T08:10:47.064 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-11T08:10:47.064 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-11T08:10:47.064 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-11T08:10:47.064 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-11T08:10:47.064 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-11T08:10:47.064 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:10:47.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:47.064 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-11T08:10:47.113 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T08:10:47.113 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T08:10:47.113 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T08:10:47.113 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:47.113 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-11T08:10:47.113 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:47.113 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:10:47.113 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:10:47.113 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-11T08:10:47.113 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:10:47.114 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:10:47.114 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-11T08:10:47.114 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-11T08:10:47.114 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-11T08:10:47.114 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-11T08:10:47.114 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-11T08:10:47.114 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42475sec preferred_lft 42475sec 2024-01-11T08:10:47.114 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-11T08:10:47.114 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:47.114 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:10:47.114 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-11T08:10:47.114 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-11T08:10:47.115 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-11T08:10:47.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:47.115 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T08:10:47.115 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T08:10:47.115 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add name ceph-brx type bridge 2024-01-11T08:10:47.115 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr flush dev ceph-brx 2024-01-11T08:10:47.115 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set ceph-brx up 2024-01-11T08:10:47.116 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-11T08:10:47.116 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T08:10:47.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:47.576 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-11T08:10:47.653 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-11T08:10:47.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:47.657 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-11T08:10:47.731 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-11T08:10:47.731 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-11T08:10:47.731 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-11T08:10:47.731 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-11T08:10:47.731 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-11T08:10:47.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:47.732 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T08:10:47.733 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T08:10:47.733 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-11T08:10:47.733 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-11T08:10:47.733 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-11T08:10:47.733 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T08:10:48.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:48.461 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T08:10:48.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:48.511 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list-id 2024-01-11T08:10:48.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:48.559 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T08:10:48.560 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T08:10:48.560 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-11T08:10:48.560 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-01-11T08:10:48.560 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T08:10:48.708 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-01-11T08:10:48.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:48.709 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T08:10:48.709 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T08:10:48.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-01-11T08:10:48.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-11T08:10:48.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-01-11T08:10:48.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-01-11T08:10:48.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-01-11T08:10:48.709 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T08:10:48.710 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:10:48.717 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:10:48.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:48.991 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T08:10:48.992 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T08:10:48.992 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-11T08:10:48.992 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set dev brx.0 master ceph-brx 2024-01-11T08:10:48.992 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T08:10:49.168 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:10:49.168 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-11T08:10:49.216 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:10:49.217 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:10:49.217 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-11T08:10:49.263 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-11T08:10:49.264 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-11T08:10:49.264 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:10:49.407 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:10:49.407 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:10:49.407 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:10:49.842 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:10:49.842 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:10:49.842 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:10:49.842 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:10:49.842 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:10:49.866 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:10:49.866 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:10:49.866 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:10:49.866 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:10:49.866 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:10:49.866 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790" 2024-01-11T08:10:49.866 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790:/ 2024-01-11T08:10:49.867 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=8fddb4a7-9d66-40f1-892d-6fe86c728ee4" will pass to kernel 2024-01-11T08:10:49.871 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-11T08:10:49.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:49.871 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-11T08:10:49.944 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2024-01-11T08:10:49.945 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-11T08:10:50.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.144+0000 7ff1fcdff700 1 Processor -- start 2024-01-11T08:10:50.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.145+0000 7ff1fcdff700 1 -- start start 2024-01-11T08:10:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.146+0000 7ff1fcdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1f80ca670 0x7ff1f8068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.146+0000 7ff1fcdff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff1f8068b60 0x7ff1f80aa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.146+0000 7ff1fcdff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f815da10 con 0x7ff1f81596c0 2024-01-11T08:10:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.146+0000 7ff1fcdff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1f8068fe0 con 0x7ff1f8068b60 2024-01-11T08:10:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.146+0000 7ff1fcdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f80a9cb0 con 0x7ff1f80ca670 2024-01-11T08:10:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.146+0000 7ff1f6d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff1f8068b60 0x7ff1f80aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.146+0000 7ff1f659c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff1f81596c0 0x7ff1f815d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:45572/0 (socket says 172.21.15.27:45572) 2024-01-11T08:10:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.146+0000 7ff1f659c700 1 -- 172.21.15.27:0/3451215884 learned_addr learned my addr 172.21.15.27:0/3451215884 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:50.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.146+0000 7ff1f5d9b700 1 --2- 172.21.15.27:0/3451215884 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1f80ca670 0x7ff1f8068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:50.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.146+0000 7ff1f6d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff1f8068b60 0x7ff1f80aa140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54756/0 (socket says 172.21.15.27:54756) 2024-01-11T08:10:50.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.147+0000 7ff1f6d9d700 1 -- 172.21.15.27:0/3451215884 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1f80ca670 msgr2=0x7ff1f8068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:50.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.147+0000 7ff1f6d9d700 1 --2- 172.21.15.27:0/3451215884 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1f80ca670 0x7ff1f8068620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:50.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.147+0000 7ff1f559a700 1 -- 172.21.15.27:0/3451215884 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257155269 0 0) 0x7ff1e8001110 con 0x7ff1f81596c0 2024-01-11T08:10:50.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.147+0000 7ff1f6d9d700 1 -- 172.21.15.27:0/3451215884 >> v1:172.21.15.77:6789/0 conn(0x7ff1f81596c0 legacy=0x7ff1f815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:50.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.147+0000 7ff1f6d9d700 1 -- 172.21.15.27:0/3451215884 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f815e210 con 0x7ff1f8068b60 2024-01-11T08:10:50.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.147+0000 7ff1f6d9d700 1 --2- 172.21.15.27:0/3451215884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1f8068b60 0x7ff1f80aa140 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff1f815e400 tx=0x7ff1f0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b8d2d8bb52039a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:50.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.147+0000 7ff1f559a700 1 -- 172.21.15.27:0/3451215884 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0016070 con 0x7ff1f8068b60 2024-01-11T08:10:50.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.147+0000 7ff1f559a700 1 -- 172.21.15.27:0/3451215884 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f000bb60 con 0x7ff1f8068b60 2024-01-11T08:10:50.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.147+0000 7ff1f559a700 1 -- 172.21.15.27:0/3451215884 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f000e070 con 0x7ff1f8068b60 2024-01-11T08:10:50.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.148+0000 7ff1fcdff700 1 -- 172.21.15.27:0/3451215884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1f8068b60 msgr2=0x7ff1f80aa140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:50.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.148+0000 7ff1fcdff700 1 --2- 172.21.15.27:0/3451215884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1f8068b60 0x7ff1f80aa140 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff1f815e400 tx=0x7ff1f0009780 comp rx=0 tx=0).stop 2024-01-11T08:10:50.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.148+0000 7ff1fcdff700 1 -- 172.21.15.27:0/3451215884 shutdown_connections 2024-01-11T08:10:50.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.148+0000 7ff1fcdff700 1 --2- 172.21.15.27:0/3451215884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1f8068b60 0x7ff1f80aa140 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:50.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.148+0000 7ff1fcdff700 1 --2- 172.21.15.27:0/3451215884 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1f80ca670 0x7ff1f8068620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:50.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.148+0000 7ff1fcdff700 1 -- 172.21.15.27:0/3451215884 >> 172.21.15.27:0/3451215884 conn(0x7ff1f80c9f70 msgr2=0x7ff1f806deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:50.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.148+0000 7ff1fcdff700 1 -- 172.21.15.27:0/3451215884 shutdown_connections 2024-01-11T08:10:50.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.148+0000 7ff1fcdff700 1 -- 172.21.15.27:0/3451215884 wait complete. 2024-01-11T08:10:50.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.149+0000 7ff1fcdff700 1 Processor -- start 2024-01-11T08:10:50.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.150+0000 7ff1fcdff700 1 -- start start 2024-01-11T08:10:50.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.151+0000 7ff1fcdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1f8068b60 0x7ff1f80afae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:50.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.151+0000 7ff1fcdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1f80ca670 0x7ff1f80ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:50.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.151+0000 7ff1fcdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1f81596c0 0x7ff1f8145c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:50.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.151+0000 7ff1fcdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f80b00c0 con 0x7ff1f80ca670 2024-01-11T08:10:50.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.151+0000 7ff1fcdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f80b03a0 con 0x7ff1f81596c0 2024-01-11T08:10:50.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.151+0000 7ff1fcdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f80b0680 con 0x7ff1f8068b60 2024-01-11T08:10:50.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.151+0000 7ff1f5d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1f80ca670 0x7ff1f80ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:50.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.151+0000 7ff1f5d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1f80ca670 0x7ff1f80ae240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54762/0 (socket says 172.21.15.27:54762) 2024-01-11T08:10:50.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.151+0000 7ff1f5d9b700 1 -- 172.21.15.27:0/216055152 learned_addr learned my addr 172.21.15.27:0/216055152 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:50.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.151+0000 7ff1f659c700 1 --2- 172.21.15.27:0/216055152 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1f8068b60 0x7ff1f80afae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:50.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.151+0000 7ff1f6d9d700 1 --2- 172.21.15.27:0/216055152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1f81596c0 0x7ff1f8145c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:50.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.151+0000 7ff1f5d9b700 1 -- 172.21.15.27:0/216055152 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1f8068b60 msgr2=0x7ff1f80afae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:50.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.151+0000 7ff1f5d9b700 1 --2- 172.21.15.27:0/216055152 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1f8068b60 0x7ff1f80afae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:50.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.151+0000 7ff1f5d9b700 1 -- 172.21.15.27:0/216055152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1f81596c0 msgr2=0x7ff1f8145c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:50.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.151+0000 7ff1f5d9b700 1 --2- 172.21.15.27:0/216055152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1f81596c0 0x7ff1f8145c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:50.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.151+0000 7ff1f5d9b700 1 -- 172.21.15.27:0/216055152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f0009a20 con 0x7ff1f80ca670 2024-01-11T08:10:50.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.152+0000 7ff1f5d9b700 1 --2- 172.21.15.27:0/216055152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1f80ca670 0x7ff1f80ae240 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec00ec50 tx=0x7ff1ec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:50.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.152+0000 7ff1e67fc700 1 -- 172.21.15.27:0/216055152 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec00c850 con 0x7ff1f80ca670 2024-01-11T08:10:50.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.152+0000 7ff1e67fc700 1 -- 172.21.15.27:0/216055152 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1ec00c9b0 con 0x7ff1f80ca670 2024-01-11T08:10:50.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.152+0000 7ff1e67fc700 1 -- 172.21.15.27:0/216055152 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec00cd70 con 0x7ff1f80ca670 2024-01-11T08:10:50.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.152+0000 7ff1fcdff700 1 -- 172.21.15.27:0/216055152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f80b0960 con 0x7ff1f80ca670 2024-01-11T08:10:50.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.152+0000 7ff1fcdff700 1 -- 172.21.15.27:0/216055152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f80ae780 con 0x7ff1f80ca670 2024-01-11T08:10:50.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.153+0000 7ff1fcdff700 1 -- 172.21.15.27:0/216055152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f80718d0 con 0x7ff1f80ca670 2024-01-11T08:10:50.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.154+0000 7ff1e67fc700 1 -- 172.21.15.27:0/216055152 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff1ec013680 con 0x7ff1f80ca670 2024-01-11T08:10:50.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.155+0000 7ff1e67fc700 1 --2- 172.21.15.27:0/216055152 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff1d8066f30 0x7ff1d80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:50.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.155+0000 7ff1e67fc700 1 -- 172.21.15.27:0/216055152 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7ff1ec0847e0 con 0x7ff1f80ca670 2024-01-11T08:10:50.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.156+0000 7ff1f659c700 1 --2- 172.21.15.27:0/216055152 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff1d8066f30 0x7ff1d80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:50.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.156+0000 7ff1f659c700 1 --2- 172.21.15.27:0/216055152 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff1d8066f30 0x7ff1d80693e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff1f80c9080 tx=0x7ff1e8007b90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:50.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.160+0000 7ff1e67fc700 1 -- 172.21.15.27:0/216055152 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff1ec0519b0 con 0x7ff1f80ca670 2024-01-11T08:10:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.412+0000 7ff1fcdff700 1 -- 172.21.15.27:0/216055152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff1f806ea80 con 0x7ff1f80ca670 2024-01-11T08:10:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.413+0000 7ff1e67fc700 1 -- 172.21.15.27:0/216055152 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff1ec054fd0 con 0x7ff1f80ca670 2024-01-11T08:10:50.413 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-11T08:10:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.415+0000 7ff1fcdff700 1 -- 172.21.15.27:0/216055152 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff1d8066f30 msgr2=0x7ff1d80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.415+0000 7ff1fcdff700 1 --2- 172.21.15.27:0/216055152 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff1d8066f30 0x7ff1d80693e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff1f80c9080 tx=0x7ff1e8007b90 comp rx=0 tx=0).stop 2024-01-11T08:10:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.415+0000 7ff1fcdff700 1 -- 172.21.15.27:0/216055152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1f80ca670 msgr2=0x7ff1f80ae240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.415+0000 7ff1fcdff700 1 --2- 172.21.15.27:0/216055152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1f80ca670 0x7ff1f80ae240 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec00ec50 tx=0x7ff1ec0091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.415+0000 7ff1fcdff700 1 -- 172.21.15.27:0/216055152 shutdown_connections 2024-01-11T08:10:50.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.415+0000 7ff1fcdff700 1 --2- 172.21.15.27:0/216055152 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff1d8066f30 0x7ff1d80693e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:50.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.415+0000 7ff1fcdff700 1 --2- 172.21.15.27:0/216055152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1f81596c0 0x7ff1f8145c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:50.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.415+0000 7ff1fcdff700 1 --2- 172.21.15.27:0/216055152 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1f8068b60 0x7ff1f80afae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:50.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.415+0000 7ff1fcdff700 1 --2- 172.21.15.27:0/216055152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1f80ca670 0x7ff1f80ae240 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:50.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.415+0000 7ff1fcdff700 1 -- 172.21.15.27:0/216055152 >> 172.21.15.27:0/216055152 conn(0x7ff1f80c9f70 msgr2=0x7ff1f80aadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:50.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.415+0000 7ff1fcdff700 1 -- 172.21.15.27:0/216055152 shutdown_connections 2024-01-11T08:10:50.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:50.415+0000 7ff1fcdff700 1 -- 172.21.15.27:0/216055152 wait complete. 2024-01-11T08:10:50.433 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-11T08:10:50.433 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-11T08:10:50.434 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-11T08:10:50.434 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-01-11T08:10:50.434 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-11T08:10:50.434 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-11T08:10:50.434 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-11T08:10:50.434 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-11T08:10:50.434 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-01-11T08:10:50.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:50.435 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-11T08:10:50.483 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T08:10:50.483 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T08:10:50.483 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T08:10:50.484 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:50.484 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-11T08:10:50.484 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:50.484 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:10:50.484 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:10:50.484 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-11T08:10:50.484 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:10:50.484 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:10:50.484 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-11T08:10:50.484 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-11T08:10:50.484 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-11T08:10:50.484 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-11T08:10:50.484 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-11T08:10:50.485 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42471sec preferred_lft 42471sec 2024-01-11T08:10:50.485 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-11T08:10:50.485 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:50.485 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:10:50.485 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-11T08:10:50.485 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-11T08:10:50.485 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T08:10:50.485 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:fa:85:1b:3a:bc brd ff:ff:ff:ff:ff:ff 2024-01-11T08:10:50.485 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-11T08:10:50.485 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:50.485 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::acd2:b3ff:fe6a:db4b/64 scope link 2024-01-11T08:10:50.485 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:50.485 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T08:10:50.485 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:fa:85:1b:3a:bc brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-11T08:10:50.486 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ecfa:85ff:fe1b:3abc/64 scope link 2024-01-11T08:10:50.486 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:50.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:50.487 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-11T08:10:50.556 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-11T08:10:50.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:50.559 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-11T08:10:50.609 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-11T08:10:50.610 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-11T08:10:50.610 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-11T08:10:50.610 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-11T08:10:50.610 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-11T08:10:50.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:50.611 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T08:10:50.611 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T08:10:50.611 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-11T08:10:50.611 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-11T08:10:50.611 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-11T08:10:50.611 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T08:10:50.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:50.787 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T08:10:50.836 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-11T08:10:50.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:50.836 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list-id 2024-01-11T08:10:50.884 INFO:teuthology.orchestra.run.smithi027.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-01-11T08:10:50.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:50.885 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T08:10:50.885 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T08:10:50.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-11T08:10:50.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-01-11T08:10:50.885 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T08:10:51.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:51.032 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-11T08:10:51.105 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T08:10:51.105 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T08:10:51.105 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T08:10:51.105 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:51.105 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-11T08:10:51.105 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:51.105 INFO:teuthology.orchestra.run.smithi027.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T08:10:51.105 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 12:4c:68:b1:dc:7a brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-11T08:10:51.106 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-11T08:10:51.106 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:51.106 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::104c:68ff:feb1:dc7a/64 scope link 2024-01-11T08:10:51.106 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:51.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:51.108 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-11T08:10:51.182 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T08:10:51.182 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T08:10:51.182 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T08:10:51.182 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:51.182 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-11T08:10:51.182 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:51.183 INFO:teuthology.orchestra.run.smithi027.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T08:10:51.183 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 12:4c:68:b1:dc:7a brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-11T08:10:51.183 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-11T08:10:51.183 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:51.183 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::104c:68ff:feb1:dc7a/64 scope link 2024-01-11T08:10:51.183 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:51.185 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-01-11T08:10:51.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:51.186 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T08:10:51.186 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T08:10:51.186 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-01-11T08:10:51.186 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-11T08:10:51.186 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-01-11T08:10:51.186 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-01-11T08:10:51.186 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-01-11T08:10:51.186 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T08:10:51.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:51.443 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T08:10:51.443 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T08:10:51.443 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.1 up 2024-01-11T08:10:51.443 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set dev brx.1 master ceph-brx 2024-01-11T08:10:51.443 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T08:10:51.621 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:10:51.622 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-01-11T08:10:51.669 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-01-11T08:10:51.669 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:10:51.669 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-01-11T08:10:51.716 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-11T08:10:51.716 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-11T08:10:51.717 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-01-11T08:10:51.819 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-01-11T08:10:51.819 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-01-11T08:10:51.819 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:10:51.834 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-01-11T08:10:51.834 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-01-11T08:10:51.834 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:10:51.834 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:10:51.834 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:10:51.856 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-01-11T08:10:51.856 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-01-11T08:10:51.856 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:10:51.857 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:10:51.857 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:10:51.857 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790" 2024-01-11T08:10:51.857 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790:/ 2024-01-11T08:10:51.857 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=8fddb4a7-9d66-40f1-892d-6fe86c728ee4" will pass to kernel 2024-01-11T08:10:51.861 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-11T08:10:51.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:51.861 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-01-11T08:10:51.933 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-01-11T08:10:51.933 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-11T08:10:52.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.131+0000 7f09775ff700 1 Processor -- start 2024-01-11T08:10:52.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.131+0000 7f09775ff700 1 -- start start 2024-01-11T08:10:52.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.132+0000 7f09775ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f09701599e0 0x7f09700b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:52.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.133+0000 7f09775ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0970150330 0x7f097015a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:52.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.133+0000 7f09775ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0970157e30 con 0x7f0970159db0 2024-01-11T08:10:52.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.133+0000 7f09775ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0970157500 con 0x7f0970150330 2024-01-11T08:10:52.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.133+0000 7f09775ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09701577e0 con 0x7f09701599e0 2024-01-11T08:10:52.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.133+0000 7f0975b9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0970150330 0x7f097015a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:52.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.133+0000 7f0975b9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0970150330 0x7f097015a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54766/0 (socket says 172.21.15.27:54766) 2024-01-11T08:10:52.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.133+0000 7f0975b9c700 1 -- 172.21.15.27:0/3615859077 learned_addr learned my addr 172.21.15.27:0/3615859077 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:52.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.133+0000 7f097539b700 1 --2- 172.21.15.27:0/3615859077 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f09701599e0 0x7f09700b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:52.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.133+0000 7f0975b9c700 1 -- 172.21.15.27:0/3615859077 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f09701599e0 msgr2=0x7f09700b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:52.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.133+0000 7f0975b9c700 1 --2- 172.21.15.27:0/3615859077 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f09701599e0 0x7f09700b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:52.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.133+0000 7f0975b9c700 1 -- 172.21.15.27:0/3615859077 >> v1:172.21.15.77:6789/0 conn(0x7f0970159db0 legacy=0x7f097014fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:52.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.133+0000 7f0975b9c700 1 -- 172.21.15.27:0/3615859077 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0970157ac0 con 0x7f0970150330 2024-01-11T08:10:52.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.134+0000 7f0975b9c700 1 --2- 172.21.15.27:0/3615859077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0970150330 0x7f097015a220 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0968009f20 tx=0x7f0968009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1ba6e36464c82c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:52.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.134+0000 7f0967fff700 1 -- 172.21.15.27:0/3615859077 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096800e070 con 0x7f0970150330 2024-01-11T08:10:52.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.134+0000 7f0967fff700 1 -- 172.21.15.27:0/3615859077 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f096800b9d0 con 0x7f0970150330 2024-01-11T08:10:52.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.134+0000 7f0967fff700 1 -- 172.21.15.27:0/3615859077 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09680092e0 con 0x7f0970150330 2024-01-11T08:10:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.134+0000 7f09775ff700 1 -- 172.21.15.27:0/3615859077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0970150330 msgr2=0x7f097015a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.134+0000 7f09775ff700 1 --2- 172.21.15.27:0/3615859077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0970150330 0x7f097015a220 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0968009f20 tx=0x7f0968009840 comp rx=0 tx=0).stop 2024-01-11T08:10:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.135+0000 7f09775ff700 1 -- 172.21.15.27:0/3615859077 shutdown_connections 2024-01-11T08:10:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.135+0000 7f09775ff700 1 --2- 172.21.15.27:0/3615859077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0970150330 0x7f097015a220 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.135+0000 7f09775ff700 1 --2- 172.21.15.27:0/3615859077 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f09701599e0 0x7f09700b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.135+0000 7f09775ff700 1 -- 172.21.15.27:0/3615859077 >> 172.21.15.27:0/3615859077 conn(0x7f09700680e0 msgr2=0x7f0970069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.135+0000 7f09775ff700 1 -- 172.21.15.27:0/3615859077 shutdown_connections 2024-01-11T08:10:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.135+0000 7f09775ff700 1 -- 172.21.15.27:0/3615859077 wait complete. 2024-01-11T08:10:52.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.136+0000 7f09775ff700 1 Processor -- start 2024-01-11T08:10:52.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.136+0000 7f09775ff700 1 -- start start 2024-01-11T08:10:52.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.137+0000 7f09775ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0970150330 0x7f097014c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:52.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.137+0000 7f09775ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09701599e0 0x7f09700c9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:52.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.137+0000 7f09775ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0970159db0 0x7f09700c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:52.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.137+0000 7f09775ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0970155780 con 0x7f09701599e0 2024-01-11T08:10:52.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.137+0000 7f09775ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0970154e90 con 0x7f0970159db0 2024-01-11T08:10:52.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.137+0000 7f09775ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0970153550 con 0x7f0970150330 2024-01-11T08:10:52.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.137+0000 7f0974b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09701599e0 0x7f09700c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:52.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.138+0000 7f0974b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09701599e0 0x7f09700c9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54768/0 (socket says 172.21.15.27:54768) 2024-01-11T08:10:52.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.138+0000 7f0974b9a700 1 -- 172.21.15.27:0/1334885493 learned_addr learned my addr 172.21.15.27:0/1334885493 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:52.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.138+0000 7f097539b700 1 --2- 172.21.15.27:0/1334885493 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0970150330 0x7f097014c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:52.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.138+0000 7f0975b9c700 1 --2- 172.21.15.27:0/1334885493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0970159db0 0x7f09700c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:52.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.138+0000 7f0974b9a700 1 -- 172.21.15.27:0/1334885493 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0970150330 msgr2=0x7f097014c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:52.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.138+0000 7f0974b9a700 1 --2- 172.21.15.27:0/1334885493 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0970150330 0x7f097014c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:52.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.138+0000 7f0974b9a700 1 -- 172.21.15.27:0/1334885493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0970159db0 msgr2=0x7f09700c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:52.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.138+0000 7f0974b9a700 1 --2- 172.21.15.27:0/1334885493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0970159db0 0x7f09700c9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:52.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.138+0000 7f0974b9a700 1 -- 172.21.15.27:0/1334885493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0968009ae0 con 0x7f09701599e0 2024-01-11T08:10:52.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.138+0000 7f097539b700 1 --2- 172.21.15.27:0/1334885493 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0970150330 0x7f097014c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:52.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.138+0000 7f0975b9c700 1 --2- 172.21.15.27:0/1334885493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0970159db0 0x7f09700c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:52.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.138+0000 7f0974b9a700 1 --2- 172.21.15.27:0/1334885493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09701599e0 0x7f09700c9310 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f096000c4e0 tx=0x7f096000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:52.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.138+0000 7f09657fa700 1 -- 172.21.15.27:0/1334885493 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0960011590 con 0x7f09701599e0 2024-01-11T08:10:52.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.138+0000 7f09657fa700 1 -- 172.21.15.27:0/1334885493 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0960011710 con 0x7f09701599e0 2024-01-11T08:10:52.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.138+0000 7f09657fa700 1 -- 172.21.15.27:0/1334885493 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0960011ad0 con 0x7f09701599e0 2024-01-11T08:10:52.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.138+0000 7f09775ff700 1 -- 172.21.15.27:0/1334885493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0970155130 con 0x7f09701599e0 2024-01-11T08:10:52.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.139+0000 7f09775ff700 1 -- 172.21.15.27:0/1334885493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f097014c7b0 con 0x7f09701599e0 2024-01-11T08:10:52.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.140+0000 7f09775ff700 1 -- 172.21.15.27:0/1334885493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09700718d0 con 0x7f09701599e0 2024-01-11T08:10:52.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.141+0000 7f09657fa700 1 -- 172.21.15.27:0/1334885493 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f096001b3b0 con 0x7f09701599e0 2024-01-11T08:10:52.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.142+0000 7f09657fa700 1 --2- 172.21.15.27:0/1334885493 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0954066f30 0x7f09540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:52.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.142+0000 7f09657fa700 1 -- 172.21.15.27:0/1334885493 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f09600839f0 con 0x7f09701599e0 2024-01-11T08:10:52.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.143+0000 7f097539b700 1 --2- 172.21.15.27:0/1334885493 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0954066f30 0x7f09540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:52.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.143+0000 7f097539b700 1 --2- 172.21.15.27:0/1334885493 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0954066f30 0x7f09540693e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f096c006f00 tx=0x7f096c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:52.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.146+0000 7f09657fa700 1 -- 172.21.15.27:0/1334885493 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0960050c60 con 0x7f09701599e0 2024-01-11T08:10:52.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.397+0000 7f09775ff700 1 -- 172.21.15.27:0/1334885493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0970156e70 con 0x7f09701599e0 2024-01-11T08:10:52.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.397+0000 7f09657fa700 1 -- 172.21.15.27:0/1334885493 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0960017070 con 0x7f09701599e0 2024-01-11T08:10:52.397 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-11T08:10:52.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.399+0000 7f09775ff700 1 -- 172.21.15.27:0/1334885493 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0954066f30 msgr2=0x7f09540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.399+0000 7f09775ff700 1 --2- 172.21.15.27:0/1334885493 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0954066f30 0x7f09540693e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f096c006f00 tx=0x7f096c008040 comp rx=0 tx=0).stop 2024-01-11T08:10:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.399+0000 7f09775ff700 1 -- 172.21.15.27:0/1334885493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09701599e0 msgr2=0x7f09700c9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.399+0000 7f09775ff700 1 --2- 172.21.15.27:0/1334885493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09701599e0 0x7f09700c9310 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f096000c4e0 tx=0x7f096000c8a0 comp rx=0 tx=0).stop 2024-01-11T08:10:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.399+0000 7f09775ff700 1 -- 172.21.15.27:0/1334885493 shutdown_connections 2024-01-11T08:10:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.399+0000 7f09775ff700 1 --2- 172.21.15.27:0/1334885493 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0954066f30 0x7f09540693e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.400+0000 7f09775ff700 1 --2- 172.21.15.27:0/1334885493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0970159db0 0x7f09700c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.400+0000 7f09775ff700 1 --2- 172.21.15.27:0/1334885493 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0970150330 0x7f097014c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.400+0000 7f09775ff700 1 --2- 172.21.15.27:0/1334885493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09701599e0 0x7f09700c9310 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.400+0000 7f09775ff700 1 -- 172.21.15.27:0/1334885493 >> 172.21.15.27:0/1334885493 conn(0x7f09700680e0 msgr2=0x7f097014e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.400+0000 7f09775ff700 1 -- 172.21.15.27:0/1334885493 shutdown_connections 2024-01-11T08:10:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:52.400+0000 7f09775ff700 1 -- 172.21.15.27:0/1334885493 wait complete. 2024-01-11T08:10:52.417 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-11T08:10:52.417 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-11T08:10:52.417 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-01-11T08:10:52.417 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-11T08:10:52.418 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-11T08:10:52.418 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-11T08:10:52.418 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-11T08:10:52.418 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-11T08:10:52.418 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-11T08:10:52.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:52.418 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-01-11T08:10:52.470 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T08:10:52.470 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T08:10:52.471 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T08:10:52.471 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:52.471 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-01-11T08:10:52.471 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:52.471 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:10:52.471 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:10:52.471 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-01-11T08:10:52.472 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:10:52.472 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:10:52.472 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-01-11T08:10:52.472 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:10:52.472 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:10:52.472 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-11T08:10:52.472 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:10:52.472 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-11T08:10:52.472 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42434sec preferred_lft 42434sec 2024-01-11T08:10:52.472 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-01-11T08:10:52.472 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:52.474 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-11T08:10:52.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:52.474 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-01-11T08:10:52.474 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-01-11T08:10:52.474 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add name ceph-brx type bridge 2024-01-11T08:10:52.474 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr flush dev ceph-brx 2024-01-11T08:10:52.474 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set ceph-brx up 2024-01-11T08:10:52.474 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-11T08:10:52.475 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-01-11T08:10:52.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:52.904 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-11T08:10:52.978 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-01-11T08:10:53.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:53.692 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-01-11T08:10:53.765 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-01-11T08:10:53.765 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-11T08:10:53.765 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-11T08:10:53.765 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-11T08:10:53.765 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-11T08:10:53.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:53.766 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-01-11T08:10:53.766 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-01-11T08:10:53.766 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-11T08:10:53.766 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-11T08:10:53.766 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-11T08:10:53.766 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-01-11T08:10:54.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:54.474 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-01-11T08:10:54.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:54.524 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list-id 2024-01-11T08:10:54.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:54.572 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-01-11T08:10:54.572 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-01-11T08:10:54.572 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-11T08:10:54.572 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-01-11T08:10:54.572 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-01-11T08:10:54.716 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-01-11T08:10:54.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:54.716 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-01-11T08:10:54.716 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-01-11T08:10:54.716 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-01-11T08:10:54.716 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-11T08:10:54.716 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-01-11T08:10:54.716 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-01-11T08:10:54.716 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-01-11T08:10:54.717 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-01-11T08:10:54.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:54.983 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-01-11T08:10:54.983 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-01-11T08:10:54.984 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-01-11T08:10:54.984 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set dev brx.0 master ceph-brx 2024-01-11T08:10:54.984 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-01-11T08:10:55.156 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:10:55.156 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-11T08:10:55.202 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-11T08:10:55.203 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:10:55.203 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-11T08:10:55.289 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-11T08:10:55.289 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-11T08:10:55.289 DEBUG:teuthology.orchestra.run.smithi077:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-11T08:10:55.429 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-11T08:10:55.429 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-11T08:10:55.429 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-01-11T08:10:55.858 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-11T08:10:55.858 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-11T08:10:55.858 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-01-11T08:10:55.858 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:10:55.859 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:10:55.882 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-11T08:10:55.882 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-11T08:10:55.882 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-01-11T08:10:55.882 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:10:55.882 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:10:55.882 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790" 2024-01-11T08:10:55.882 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790:/ 2024-01-11T08:10:55.882 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=8fddb4a7-9d66-40f1-892d-6fe86c728ee4" will pass to kernel 2024-01-11T08:10:55.888 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-11T08:10:55.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:55.888 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-11T08:10:55.958 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2024-01-11T08:10:55.958 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-11T08:10:56.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.168+0000 7f1b161ff700 1 Processor -- start 2024-01-11T08:10:56.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.168+0000 7f1b161ff700 1 -- start start 2024-01-11T08:10:56.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.169+0000 7f1b161ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1b100b2b90 0x7f1b10159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:56.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.169+0000 7f1b161ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1b100b07b0 0x7f1b10159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:56.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.170+0000 7f1b161ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b100cd730 con 0x7f1b100b03e0 2024-01-11T08:10:56.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.170+0000 7f1b161ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b100ccb20 con 0x7f1b100b07b0 2024-01-11T08:10:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.170+0000 7f1b161ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b100cce00 con 0x7f1b100b2b90 2024-01-11T08:10:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.170+0000 7f1b0ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1b100b07b0 0x7f1b10159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.170+0000 7f1b0effd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1b100b2b90 0x7f1b10159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.170+0000 7f1b0effd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1b100b2b90 0x7f1b10159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:60328/0 (socket says 172.21.15.27:60328) 2024-01-11T08:10:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.170+0000 7f1b0effd700 1 -- 172.21.15.27:0/3382028201 learned_addr learned my addr 172.21.15.27:0/3382028201 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.170+0000 7f1b0f7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1b100b03e0 0x7f1b100b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:51690/0 (socket says 172.21.15.27:51690) 2024-01-11T08:10:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.170+0000 7f1b0ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1b100b07b0 0x7f1b10159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:54766/0 (socket says 172.21.15.27:54766) 2024-01-11T08:10:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.170+0000 7f1b0ffff700 1 -- 172.21.15.27:0/3382028201 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1b100b2b90 msgr2=0x7f1b10159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.170+0000 7f1b0ffff700 1 --2- 172.21.15.27:0/3382028201 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1b100b2b90 0x7f1b10159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.170+0000 7f1b0ffff700 1 -- 172.21.15.27:0/3382028201 >> v1:172.21.15.77:6789/0 conn(0x7f1b100b03e0 legacy=0x7f1b100b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.170+0000 7f1b0e7fc700 1 -- 172.21.15.27:0/3382028201 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191891899 0 0) 0x7f1b100cd730 con 0x7f1b100b03e0 2024-01-11T08:10:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.170+0000 7f1b0ffff700 1 -- 172.21.15.27:0/3382028201 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b100cd0e0 con 0x7f1b100b07b0 2024-01-11T08:10:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.170+0000 7f1b0effd700 1 --2- 172.21.15.27:0/3382028201 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1b100b2b90 0x7f1b10159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.171+0000 7f1b0ffff700 1 --2- 172.21.15.27:0/3382028201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b100b07b0 0x7f1b10159740 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1b08009e60 tx=0x7f1b08009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cc7ec211f9fc55c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.171+0000 7f1b0e7fc700 1 -- 172.21.15.27:0/3382028201 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0800e070 con 0x7f1b100b07b0 2024-01-11T08:10:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.171+0000 7f1b0e7fc700 1 -- 172.21.15.27:0/3382028201 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b0800bb60 con 0x7f1b100b07b0 2024-01-11T08:10:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.171+0000 7f1b0e7fc700 1 -- 172.21.15.27:0/3382028201 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b08012430 con 0x7f1b100b07b0 2024-01-11T08:10:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.171+0000 7f1b161ff700 1 -- 172.21.15.27:0/3382028201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b100b07b0 msgr2=0x7f1b10159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.171+0000 7f1b161ff700 1 --2- 172.21.15.27:0/3382028201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b100b07b0 0x7f1b10159740 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1b08009e60 tx=0x7f1b08009780 comp rx=0 tx=0).stop 2024-01-11T08:10:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.172+0000 7f1b161ff700 1 -- 172.21.15.27:0/3382028201 shutdown_connections 2024-01-11T08:10:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.172+0000 7f1b161ff700 1 --2- 172.21.15.27:0/3382028201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b100b07b0 0x7f1b10159740 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.172+0000 7f1b161ff700 1 --2- 172.21.15.27:0/3382028201 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1b100b2b90 0x7f1b10159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.172+0000 7f1b161ff700 1 -- 172.21.15.27:0/3382028201 >> 172.21.15.27:0/3382028201 conn(0x7f1b1006ed40 msgr2=0x7f1b1006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.172+0000 7f1b161ff700 1 -- 172.21.15.27:0/3382028201 shutdown_connections 2024-01-11T08:10:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.172+0000 7f1b161ff700 1 -- 172.21.15.27:0/3382028201 wait complete. 2024-01-11T08:10:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.173+0000 7f1b161ff700 1 Processor -- start 2024-01-11T08:10:56.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.173+0000 7f1b161ff700 1 -- start start 2024-01-11T08:10:56.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.174+0000 7f1b161ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1b100b03e0 0x7f1b101588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:56.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.174+0000 7f1b161ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b100b07b0 0x7f1b100c7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:56.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.174+0000 7f1b161ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b100b2b90 0x7f1b100c9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:56.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.175+0000 7f1b161ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b100c8a50 con 0x7f1b100b07b0 2024-01-11T08:10:56.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.175+0000 7f1b161ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b100c8d30 con 0x7f1b100b2b90 2024-01-11T08:10:56.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.175+0000 7f1b161ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1016a9e0 con 0x7f1b100b03e0 2024-01-11T08:10:56.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.175+0000 7f1b0effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b100b07b0 0x7f1b100c7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:56.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.175+0000 7f1b0effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b100b07b0 0x7f1b100c7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53304/0 (socket says 172.21.15.27:53304) 2024-01-11T08:10:56.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.175+0000 7f1b0effd700 1 -- 172.21.15.27:0/906301946 learned_addr learned my addr 172.21.15.27:0/906301946 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:56.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.175+0000 7f1b0ffff700 1 --2- 172.21.15.27:0/906301946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b100b2b90 0x7f1b100c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:56.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.175+0000 7f1b0f7fe700 1 --2- 172.21.15.27:0/906301946 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1b100b03e0 0x7f1b101588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:56.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.175+0000 7f1b0effd700 1 -- 172.21.15.27:0/906301946 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1b100b03e0 msgr2=0x7f1b101588f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:56.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.175+0000 7f1b0effd700 1 --2- 172.21.15.27:0/906301946 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1b100b03e0 0x7f1b101588f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:56.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.175+0000 7f1b0effd700 1 -- 172.21.15.27:0/906301946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b100b2b90 msgr2=0x7f1b100c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:56.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.175+0000 7f1b0effd700 1 --2- 172.21.15.27:0/906301946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b100b2b90 0x7f1b100c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:56.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.175+0000 7f1b0effd700 1 -- 172.21.15.27:0/906301946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b040094f0 con 0x7f1b100b07b0 2024-01-11T08:10:56.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.175+0000 7f1b0ffff700 1 --2- 172.21.15.27:0/906301946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b100b2b90 0x7f1b100c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:56.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.176+0000 7f1b0effd700 1 --2- 172.21.15.27:0/906301946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b100b07b0 0x7f1b100c7fd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1b0400ec50 tx=0x7f1b04009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:56.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.176+0000 7f1af77fe700 1 -- 172.21.15.27:0/906301946 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0400c9d0 con 0x7f1b100b07b0 2024-01-11T08:10:56.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.176+0000 7f1af77fe700 1 -- 172.21.15.27:0/906301946 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b0400cb30 con 0x7f1b100b07b0 2024-01-11T08:10:56.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.176+0000 7f1b161ff700 1 -- 172.21.15.27:0/906301946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b08009a20 con 0x7f1b100b07b0 2024-01-11T08:10:56.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.176+0000 7f1af77fe700 1 -- 172.21.15.27:0/906301946 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b04013470 con 0x7f1b100b07b0 2024-01-11T08:10:56.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.176+0000 7f1b161ff700 1 -- 172.21.15.27:0/906301946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b100aeb40 con 0x7f1b100b07b0 2024-01-11T08:10:56.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.177+0000 7f1b161ff700 1 -- 172.21.15.27:0/906301946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b1016a840 con 0x7f1b100b07b0 2024-01-11T08:10:56.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.178+0000 7f1af77fe700 1 -- 172.21.15.27:0/906301946 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1b04013860 con 0x7f1b100b07b0 2024-01-11T08:10:56.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.179+0000 7f1af77fe700 1 --2- 172.21.15.27:0/906301946 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1afc066f30 0x7f1afc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:56.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.179+0000 7f1af77fe700 1 -- 172.21.15.27:0/906301946 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f1b040849b0 con 0x7f1b100b07b0 2024-01-11T08:10:56.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.180+0000 7f1b0f7fe700 1 --2- 172.21.15.27:0/906301946 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1afc066f30 0x7f1afc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:56.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.181+0000 7f1b0f7fe700 1 --2- 172.21.15.27:0/906301946 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1afc066f30 0x7f1afc0693e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1b000011e0 tx=0x7f1b00007bf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:56.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.184+0000 7f1af77fe700 1 -- 172.21.15.27:0/906301946 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1b1016a840 con 0x7f1b100b07b0 2024-01-11T08:10:56.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.436+0000 7f1b161ff700 1 -- 172.21.15.27:0/906301946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1b1016af00 con 0x7f1b100b07b0 2024-01-11T08:10:56.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.436+0000 7f1af77fe700 1 -- 172.21.15.27:0/906301946 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1b1016af00 con 0x7f1b100b07b0 2024-01-11T08:10:56.436 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-11T08:10:56.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.439+0000 7f1b161ff700 1 -- 172.21.15.27:0/906301946 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1afc066f30 msgr2=0x7f1afc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:56.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.439+0000 7f1b161ff700 1 --2- 172.21.15.27:0/906301946 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1afc066f30 0x7f1afc0693e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1b000011e0 tx=0x7f1b00007bf0 comp rx=0 tx=0).stop 2024-01-11T08:10:56.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.439+0000 7f1b161ff700 1 -- 172.21.15.27:0/906301946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b100b07b0 msgr2=0x7f1b100c7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:56.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.439+0000 7f1b161ff700 1 --2- 172.21.15.27:0/906301946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b100b07b0 0x7f1b100c7fd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1b0400ec50 tx=0x7f1b04009250 comp rx=0 tx=0).stop 2024-01-11T08:10:56.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.439+0000 7f1b161ff700 1 -- 172.21.15.27:0/906301946 shutdown_connections 2024-01-11T08:10:56.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.439+0000 7f1b161ff700 1 --2- 172.21.15.27:0/906301946 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1afc066f30 0x7f1afc0693e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:56.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.439+0000 7f1b161ff700 1 --2- 172.21.15.27:0/906301946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b100b2b90 0x7f1b100c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:56.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.439+0000 7f1b161ff700 1 --2- 172.21.15.27:0/906301946 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1b100b03e0 0x7f1b101588f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:56.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.439+0000 7f1b161ff700 1 --2- 172.21.15.27:0/906301946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b100b07b0 0x7f1b100c7fd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:56.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.439+0000 7f1b161ff700 1 -- 172.21.15.27:0/906301946 >> 172.21.15.27:0/906301946 conn(0x7f1b1006ed40 msgr2=0x7f1b100d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:56.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.439+0000 7f1b161ff700 1 -- 172.21.15.27:0/906301946 shutdown_connections 2024-01-11T08:10:56.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:56.439+0000 7f1b161ff700 1 -- 172.21.15.27:0/906301946 wait complete. 2024-01-11T08:10:56.456 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-11T08:10:56.456 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-11T08:10:56.456 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-01-11T08:10:56.456 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-01-11T08:10:56.456 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-11T08:10:56.456 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-11T08:10:56.456 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-11T08:10:56.457 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-11T08:10:56.457 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-01-11T08:10:56.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:56.457 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-01-11T08:10:56.504 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T08:10:56.504 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T08:10:56.504 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T08:10:56.504 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:56.504 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-01-11T08:10:56.504 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:56.505 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:10:56.505 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:10:56.505 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-01-11T08:10:56.505 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:10:56.505 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:10:56.505 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-01-11T08:10:56.505 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:10:56.505 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:10:56.505 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-11T08:10:56.505 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:10:56.505 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-11T08:10:56.505 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42430sec preferred_lft 42430sec 2024-01-11T08:10:56.505 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-01-11T08:10:56.505 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:56.506 INFO:teuthology.orchestra.run.smithi077.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T08:10:56.506 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 8e:f7:7d:ac:2a:db brd ff:ff:ff:ff:ff:ff 2024-01-11T08:10:56.506 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-11T08:10:56.506 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:56.506 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::d3:a6ff:fe67:ef3a/64 scope link 2024-01-11T08:10:56.506 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:56.506 INFO:teuthology.orchestra.run.smithi077.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T08:10:56.506 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 8e:f7:7d:ac:2a:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-11T08:10:56.506 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::8cf7:7dff:feac:2adb/64 scope link tentative 2024-01-11T08:10:56.506 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:56.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:56.507 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-11T08:10:56.574 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-01-11T08:10:56.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:56.577 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-01-11T08:10:56.625 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-01-11T08:10:56.626 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-11T08:10:56.626 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-11T08:10:56.626 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-11T08:10:56.626 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-11T08:10:56.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:56.627 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-01-11T08:10:56.627 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-01-11T08:10:56.627 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-11T08:10:56.627 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-11T08:10:56.627 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-11T08:10:56.627 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-01-11T08:10:56.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:56.796 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-01-11T08:10:56.843 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-11T08:10:56.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:56.844 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list-id 2024-01-11T08:10:56.929 INFO:teuthology.orchestra.run.smithi077.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-01-11T08:10:56.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:56.931 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-01-11T08:10:56.931 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-01-11T08:10:56.931 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-11T08:10:56.931 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-01-11T08:10:56.931 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-01-11T08:10:57.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:57.073 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-11T08:10:57.143 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T08:10:57.143 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T08:10:57.144 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T08:10:57.144 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:57.144 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-01-11T08:10:57.144 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:57.144 INFO:teuthology.orchestra.run.smithi077.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T08:10:57.144 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0a:a6:7c:24:79:ad brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-11T08:10:57.144 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-11T08:10:57.144 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:57.144 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::8a6:7cff:fe24:79ad/64 scope link 2024-01-11T08:10:57.144 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:57.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:57.146 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-11T08:10:57.217 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T08:10:57.217 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T08:10:57.217 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T08:10:57.217 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:57.217 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-01-11T08:10:57.217 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:57.217 INFO:teuthology.orchestra.run.smithi077.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T08:10:57.217 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0a:a6:7c:24:79:ad brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-11T08:10:57.218 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-11T08:10:57.218 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:57.218 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::8a6:7cff:fe24:79ad/64 scope link 2024-01-11T08:10:57.218 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:10:57.220 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-01-11T08:10:57.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:57.220 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-01-11T08:10:57.220 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-01-11T08:10:57.220 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-01-11T08:10:57.220 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-11T08:10:57.220 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-01-11T08:10:57.221 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-01-11T08:10:57.221 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-01-11T08:10:57.221 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-01-11T08:10:57.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:57.466 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-01-11T08:10:57.466 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-01-11T08:10:57.466 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.1 up 2024-01-11T08:10:57.466 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set dev brx.1 master ceph-brx 2024-01-11T08:10:57.466 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-01-11T08:10:57.639 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:10:57.639 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-01-11T08:10:57.688 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-01-11T08:10:57.689 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:10:57.689 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-01-11T08:10:57.774 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-11T08:10:57.774 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-11T08:10:57.774 DEBUG:teuthology.orchestra.run.smithi077:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-01-11T08:10:57.913 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-01-11T08:10:57.914 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-01-11T08:10:57.914 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-01-11T08:10:57.928 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-01-11T08:10:57.928 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-01-11T08:10:57.928 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-01-11T08:10:57.928 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:10:57.928 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:10:57.949 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-01-11T08:10:57.949 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-01-11T08:10:57.950 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-01-11T08:10:57.950 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:10:57.950 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:10:57.950 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790" 2024-01-11T08:10:57.950 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790:/ 2024-01-11T08:10:57.950 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=8fddb4a7-9d66-40f1-892d-6fe86c728ee4" will pass to kernel 2024-01-11T08:10:57.954 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-11T08:10:57.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:10:57.954 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-01-11T08:10:58.023 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-01-11T08:10:58.056 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:10:58.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.266+0000 7f47a0bff700 1 Processor -- start 2024-01-11T08:10:58.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.267+0000 7f47a0bff700 1 -- start start 2024-01-11T08:10:58.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.268+0000 7f47a0bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f479c0c8620 0x7f479c0af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:58.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.268+0000 7f47a0bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f479c0b09e0 0x7f479c0accc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:58.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.268+0000 7f47a0bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f479c157390 con 0x7f479c069d40 2024-01-11T08:10:58.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.268+0000 7f47a0bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f479c0afeb0 con 0x7f479c0c8620 2024-01-11T08:10:58.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.268+0000 7f47a0bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f479c06a200 con 0x7f479c0b09e0 2024-01-11T08:10:58.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.268+0000 7f479a59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f479c0c8620 0x7f479c0af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:58.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.268+0000 7f479a59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f479c0c8620 0x7f479c0af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53318/0 (socket says 172.21.15.27:53318) 2024-01-11T08:10:58.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.268+0000 7f479a59c700 1 -- 172.21.15.27:0/797925818 learned_addr learned my addr 172.21.15.27:0/797925818 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:58.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.268+0000 7f479ad9d700 1 --2- 172.21.15.27:0/797925818 >> v2:172.21.15.77:3300/0 conn(0x7f479c0b09e0 0x7f479c0accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:58.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.269+0000 7f479a59c700 1 -- 172.21.15.27:0/797925818 >> v2:172.21.15.77:3300/0 conn(0x7f479c0b09e0 msgr2=0x7f479c0accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:58.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.269+0000 7f479a59c700 1 --2- 172.21.15.27:0/797925818 >> v2:172.21.15.77:3300/0 conn(0x7f479c0b09e0 0x7f479c0accc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.269+0000 7f479a59c700 1 -- 172.21.15.27:0/797925818 >> v1:172.21.15.77:6789/0 conn(0x7f479c069d40 legacy=0x7f479c0b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:58.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.269+0000 7f479a59c700 1 -- 172.21.15.27:0/797925818 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479c156d40 con 0x7f479c0c8620 2024-01-11T08:10:58.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.269+0000 7f479a59c700 1 --2- 172.21.15.27:0/797925818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f479c0c8620 0x7f479c0af970 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f478c0099b0 tx=0x7f478c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1863c37999de084a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:58.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.269+0000 7f479959a700 1 -- 172.21.15.27:0/797925818 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478c009ea0 con 0x7f479c0c8620 2024-01-11T08:10:58.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.269+0000 7f479959a700 1 -- 172.21.15.27:0/797925818 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f478c00b820 con 0x7f479c0c8620 2024-01-11T08:10:58.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.269+0000 7f479959a700 1 -- 172.21.15.27:0/797925818 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478c00bbc0 con 0x7f479c0c8620 2024-01-11T08:10:58.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.270+0000 7f47a0bff700 1 -- 172.21.15.27:0/797925818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f479c0c8620 msgr2=0x7f479c0af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:58.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.270+0000 7f47a0bff700 1 --2- 172.21.15.27:0/797925818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f479c0c8620 0x7f479c0af970 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f478c0099b0 tx=0x7f478c009250 comp rx=0 tx=0).stop 2024-01-11T08:10:58.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.270+0000 7f47a0bff700 1 -- 172.21.15.27:0/797925818 shutdown_connections 2024-01-11T08:10:58.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.270+0000 7f47a0bff700 1 --2- 172.21.15.27:0/797925818 >> v2:172.21.15.77:3300/0 conn(0x7f479c0b09e0 0x7f479c0accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.270+0000 7f47a0bff700 1 --2- 172.21.15.27:0/797925818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f479c0c8620 0x7f479c0af970 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.270+0000 7f47a0bff700 1 -- 172.21.15.27:0/797925818 >> 172.21.15.27:0/797925818 conn(0x7f479c0680e0 msgr2=0x7f479c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:58.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.270+0000 7f47a0bff700 1 -- 172.21.15.27:0/797925818 shutdown_connections 2024-01-11T08:10:58.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.270+0000 7f47a0bff700 1 -- 172.21.15.27:0/797925818 wait complete. 2024-01-11T08:10:58.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.271+0000 7f47a0bff700 1 Processor -- start 2024-01-11T08:10:58.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.272+0000 7f47a0bff700 1 -- start start 2024-01-11T08:10:58.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.273+0000 7f47a0bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f479c069d40 0x7f479c14d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:58.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.273+0000 7f47a0bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f479c0b09e0 0x7f479c14e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:58.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.273+0000 7f47a0bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f479c0c8620 0x7f479c14ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:58.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.273+0000 7f47a0bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479c14ec70 con 0x7f479c069d40 2024-01-11T08:10:58.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.273+0000 7f47a0bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479c14fa70 con 0x7f479c0c8620 2024-01-11T08:10:58.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.273+0000 7f47a0bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f479c150440 con 0x7f479c0b09e0 2024-01-11T08:10:58.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.273+0000 7f479a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f479c069d40 0x7f479c14d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:58.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.273+0000 7f479a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f479c069d40 0x7f479c14d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53332/0 (socket says 172.21.15.27:53332) 2024-01-11T08:10:58.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.273+0000 7f479a59c700 1 -- 172.21.15.27:0/2762554754 learned_addr learned my addr 172.21.15.27:0/2762554754 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:58.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.273+0000 7f479ad9d700 1 --2- 172.21.15.27:0/2762554754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f479c0c8620 0x7f479c14ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:58.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.273+0000 7f4799d9b700 1 --2- 172.21.15.27:0/2762554754 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f479c0b09e0 0x7f479c14e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:58.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.273+0000 7f479a59c700 1 -- 172.21.15.27:0/2762554754 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f479c0b09e0 msgr2=0x7f479c14e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:58.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.273+0000 7f479a59c700 1 --2- 172.21.15.27:0/2762554754 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f479c0b09e0 0x7f479c14e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.273+0000 7f479a59c700 1 -- 172.21.15.27:0/2762554754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f479c0c8620 msgr2=0x7f479c14ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:58.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.273+0000 7f479a59c700 1 --2- 172.21.15.27:0/2762554754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f479c0c8620 0x7f479c14ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.273+0000 7f479a59c700 1 -- 172.21.15.27:0/2762554754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478c0094f0 con 0x7f479c069d40 2024-01-11T08:10:58.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.273+0000 7f479ad9d700 1 --2- 172.21.15.27:0/2762554754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f479c0c8620 0x7f479c14ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:58.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.273+0000 7f4799d9b700 1 --2- 172.21.15.27:0/2762554754 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f479c0b09e0 0x7f479c14e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:10:58.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.274+0000 7f479a59c700 1 --2- 172.21.15.27:0/2762554754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f479c069d40 0x7f479c14d340 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f478c0094c0 tx=0x7f478c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:58.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.274+0000 7f478a7fc700 1 -- 172.21.15.27:0/2762554754 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478c00bba0 con 0x7f479c069d40 2024-01-11T08:10:58.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.274+0000 7f478a7fc700 1 -- 172.21.15.27:0/2762554754 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f478c011420 con 0x7f479c069d40 2024-01-11T08:10:58.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.274+0000 7f478a7fc700 1 -- 172.21.15.27:0/2762554754 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478c0117e0 con 0x7f479c069d40 2024-01-11T08:10:58.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.274+0000 7f47a0bff700 1 -- 172.21.15.27:0/2762554754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f479c14ee60 con 0x7f479c069d40 2024-01-11T08:10:58.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.274+0000 7f47a0bff700 1 -- 172.21.15.27:0/2762554754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f479c14f420 con 0x7f479c069d40 2024-01-11T08:10:58.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.275+0000 7f47a0bff700 1 -- 172.21.15.27:0/2762554754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f479c14fd50 con 0x7f479c069d40 2024-01-11T08:10:58.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.276+0000 7f478a7fc700 1 -- 172.21.15.27:0/2762554754 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f478c011b90 con 0x7f479c069d40 2024-01-11T08:10:58.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.277+0000 7f478a7fc700 1 --2- 172.21.15.27:0/2762554754 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f477c066f30 0x7f477c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:58.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.277+0000 7f478a7fc700 1 -- 172.21.15.27:0/2762554754 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f478c085eb0 con 0x7f479c069d40 2024-01-11T08:10:58.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.278+0000 7f4799d9b700 1 --2- 172.21.15.27:0/2762554754 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f477c066f30 0x7f477c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:58.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.278+0000 7f4799d9b700 1 --2- 172.21.15.27:0/2762554754 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f477c066f30 0x7f477c0693e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f479c0cc870 tx=0x7f4790009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:58.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.281+0000 7f478a7fc700 1 -- 172.21.15.27:0/2762554754 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f478c053080 con 0x7f479c069d40 2024-01-11T08:10:58.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.482+0000 7f47a0bff700 1 -- 172.21.15.27:0/2762554754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f479c0718d0 con 0x7f479c069d40 2024-01-11T08:10:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.483+0000 7f478a7fc700 1 -- 172.21.15.27:0/2762554754 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15302 (secure 0 0 0) 0x7f478c0566a0 con 0x7f479c069d40 2024-01-11T08:10:58.483 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:10:58.483 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:31.464323+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:10:27.993251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-11T08:10:29.153959+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:10:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.485+0000 7f47a0bff700 1 -- 172.21.15.27:0/2762554754 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f477c066f30 msgr2=0x7f477c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.485+0000 7f47a0bff700 1 --2- 172.21.15.27:0/2762554754 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f477c066f30 0x7f477c0693e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f479c0cc870 tx=0x7f4790009040 comp rx=0 tx=0).stop 2024-01-11T08:10:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.485+0000 7f47a0bff700 1 -- 172.21.15.27:0/2762554754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f479c069d40 msgr2=0x7f479c14d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.485+0000 7f47a0bff700 1 --2- 172.21.15.27:0/2762554754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f479c069d40 0x7f479c14d340 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f478c0094c0 tx=0x7f478c017000 comp rx=0 tx=0).stop 2024-01-11T08:10:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.486+0000 7f47a0bff700 1 -- 172.21.15.27:0/2762554754 shutdown_connections 2024-01-11T08:10:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.486+0000 7f47a0bff700 1 --2- 172.21.15.27:0/2762554754 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f477c066f30 0x7f477c0693e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.486+0000 7f47a0bff700 1 --2- 172.21.15.27:0/2762554754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f479c0c8620 0x7f479c14ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.486+0000 7f47a0bff700 1 --2- 172.21.15.27:0/2762554754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f479c069d40 0x7f479c14d340 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.486+0000 7f47a0bff700 1 --2- 172.21.15.27:0/2762554754 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f479c0b09e0 0x7f479c14e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.486+0000 7f47a0bff700 1 -- 172.21.15.27:0/2762554754 >> 172.21.15.27:0/2762554754 conn(0x7f479c0680e0 msgr2=0x7f479c14a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.486+0000 7f47a0bff700 1 -- 172.21.15.27:0/2762554754 shutdown_connections 2024-01-11T08:10:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.486+0000 7f47a0bff700 1 -- 172.21.15.27:0/2762554754 wait complete. 2024-01-11T08:10:58.504 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T08:10:27.993251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T08:10:29.153959+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T08:10:58.505 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:10:58.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.712+0000 7f7ddcfff700 1 Processor -- start 2024-01-11T08:10:58.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.712+0000 7f7ddcfff700 1 -- start start 2024-01-11T08:10:58.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.713+0000 7f7ddcfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dd8142dc0 0x7f7dd80cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:58.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.713+0000 7f7ddcfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7dd8143190 0x7f7dd80cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:58.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.713+0000 7f7ddcfff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd815d320 con 0x7f7dd80cc020 2024-01-11T08:10:58.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.713+0000 7f7ddcfff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dd815df30 con 0x7f7dd8143190 2024-01-11T08:10:58.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.713+0000 7f7ddcfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd815d590 con 0x7f7dd8142dc0 2024-01-11T08:10:58.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.713+0000 7f7dd6d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7dd80cc020 0x7f7dd80cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38986/0 (socket says 172.21.15.27:38986) 2024-01-11T08:10:58.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.713+0000 7f7dd6d9d700 1 -- 172.21.15.27:0/839692375 learned_addr learned my addr 172.21.15.27:0/839692375 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:58.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.714+0000 7f7dd5d9b700 1 --2- 172.21.15.27:0/839692375 >> v2:172.21.15.77:3300/0 conn(0x7f7dd8143190 0x7f7dd80cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:58.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.714+0000 7f7dd659c700 1 --2- 172.21.15.27:0/839692375 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dd8142dc0 0x7f7dd80cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:58.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.714+0000 7f7dd559a700 1 -- 172.21.15.27:0/839692375 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814180512 0 0) 0x7f7dd815d320 con 0x7f7dd80cc020 2024-01-11T08:10:58.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.714+0000 7f7dd559a700 1 -- 172.21.15.27:0/839692375 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dc0003570 con 0x7f7dd80cc020 2024-01-11T08:10:58.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.714+0000 7f7dd659c700 1 -- 172.21.15.27:0/839692375 >> v2:172.21.15.77:3300/0 conn(0x7f7dd8143190 msgr2=0x7f7dd80cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:58.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.714+0000 7f7dd659c700 1 --2- 172.21.15.27:0/839692375 >> v2:172.21.15.77:3300/0 conn(0x7f7dd8143190 0x7f7dd80cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.714+0000 7f7dd659c700 1 -- 172.21.15.27:0/839692375 >> v1:172.21.15.27:6789/0 conn(0x7f7dd80cc020 legacy=0x7f7dd80cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:58.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.714+0000 7f7dd659c700 1 -- 172.21.15.27:0/839692375 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd815d870 con 0x7f7dd8142dc0 2024-01-11T08:10:58.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.714+0000 7f7dd5d9b700 1 --2- 172.21.15.27:0/839692375 >> v2:172.21.15.77:3300/0 conn(0x7f7dd8143190 0x7f7dd80cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:10:58.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.714+0000 7f7dd659c700 1 --2- 172.21.15.27:0/839692375 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dd8142dc0 0x7f7dd80cada0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7dc80098e0 tx=0x7f7dc80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e41a0099bdd674fb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:58.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.715+0000 7f7dd559a700 1 -- 172.21.15.27:0/839692375 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8011630 con 0x7f7dd8142dc0 2024-01-11T08:10:58.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.715+0000 7f7dd559a700 1 -- 172.21.15.27:0/839692375 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc8011790 con 0x7f7dd8142dc0 2024-01-11T08:10:58.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.715+0000 7f7dd559a700 1 -- 172.21.15.27:0/839692375 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8011b30 con 0x7f7dd8142dc0 2024-01-11T08:10:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.715+0000 7f7ddcfff700 1 -- 172.21.15.27:0/839692375 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dd8142dc0 msgr2=0x7f7dd80cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.715+0000 7f7ddcfff700 1 --2- 172.21.15.27:0/839692375 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dd8142dc0 0x7f7dd80cada0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7dc80098e0 tx=0x7f7dc80091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.715+0000 7f7ddcfff700 1 -- 172.21.15.27:0/839692375 shutdown_connections 2024-01-11T08:10:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.715+0000 7f7ddcfff700 1 --2- 172.21.15.27:0/839692375 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dd8142dc0 0x7f7dd80cada0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.715+0000 7f7ddcfff700 1 --2- 172.21.15.27:0/839692375 >> v2:172.21.15.77:3300/0 conn(0x7f7dd8143190 0x7f7dd80cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.715+0000 7f7ddcfff700 1 -- 172.21.15.27:0/839692375 >> 172.21.15.27:0/839692375 conn(0x7f7dd8063ca0 msgr2=0x7f7dd806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:58.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.715+0000 7f7ddcfff700 1 -- 172.21.15.27:0/839692375 shutdown_connections 2024-01-11T08:10:58.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.715+0000 7f7ddcfff700 1 -- 172.21.15.27:0/839692375 wait complete. 2024-01-11T08:10:58.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.717+0000 7f7ddcfff700 1 Processor -- start 2024-01-11T08:10:58.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.717+0000 7f7ddcfff700 1 -- start start 2024-01-11T08:10:58.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.718+0000 7f7ddcfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dd80cc020 0x7f7dd813ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:58.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.718+0000 7f7ddcfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dd8142dc0 0x7f7dd80c7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:58.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.718+0000 7f7ddcfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dd8143190 0x7f7dd80c3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.718+0000 7f7ddcfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd813f1b0 con 0x7f7dd80cc020 2024-01-11T08:10:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.718+0000 7f7ddcfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd80c82d0 con 0x7f7dd8142dc0 2024-01-11T08:10:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.718+0000 7f7ddcfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd80c9730 con 0x7f7dd8143190 2024-01-11T08:10:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.718+0000 7f7dd659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dd80cc020 0x7f7dd813ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.718+0000 7f7dd659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dd80cc020 0x7f7dd813ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53348/0 (socket says 172.21.15.27:53348) 2024-01-11T08:10:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.718+0000 7f7dd659c700 1 -- 172.21.15.27:0/3105415772 learned_addr learned my addr 172.21.15.27:0/3105415772 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.718+0000 7f7dd5d9b700 1 --2- 172.21.15.27:0/3105415772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dd8142dc0 0x7f7dd80c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.718+0000 7f7dd6d9d700 1 --2- 172.21.15.27:0/3105415772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dd8143190 0x7f7dd80c3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.719+0000 7f7dd659c700 1 -- 172.21.15.27:0/3105415772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dd8143190 msgr2=0x7f7dd80c3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.719+0000 7f7dd659c700 1 --2- 172.21.15.27:0/3105415772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dd8143190 0x7f7dd80c3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.719+0000 7f7dd659c700 1 -- 172.21.15.27:0/3105415772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dd8142dc0 msgr2=0x7f7dd80c7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.719+0000 7f7dd659c700 1 --2- 172.21.15.27:0/3105415772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dd8142dc0 0x7f7dd80c7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.719+0000 7f7dd659c700 1 -- 172.21.15.27:0/3105415772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dcc009420 con 0x7f7dd80cc020 2024-01-11T08:10:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.719+0000 7f7dd6d9d700 1 --2- 172.21.15.27:0/3105415772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dd8143190 0x7f7dd80c3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:58.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.719+0000 7f7dd659c700 1 --2- 172.21.15.27:0/3105415772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dd80cc020 0x7f7dd813ec70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7dd815d6d0 tx=0x7f7dc8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:58.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.719+0000 7f7dc67fc700 1 -- 172.21.15.27:0/3105415772 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8011b10 con 0x7f7dd80cc020 2024-01-11T08:10:58.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.719+0000 7f7dc67fc700 1 -- 172.21.15.27:0/3105415772 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc80253f0 con 0x7f7dd80cc020 2024-01-11T08:10:58.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.719+0000 7f7ddcfff700 1 -- 172.21.15.27:0/3105415772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc8009420 con 0x7f7dd80cc020 2024-01-11T08:10:58.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.719+0000 7f7dc67fc700 1 -- 172.21.15.27:0/3105415772 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc80257b0 con 0x7f7dd80cc020 2024-01-11T08:10:58.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.720+0000 7f7ddcfff700 1 -- 172.21.15.27:0/3105415772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd80c99d0 con 0x7f7dd80cc020 2024-01-11T08:10:58.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.721+0000 7f7ddcfff700 1 -- 172.21.15.27:0/3105415772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd804c3c0 con 0x7f7dd80cc020 2024-01-11T08:10:58.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.722+0000 7f7dc67fc700 1 -- 172.21.15.27:0/3105415772 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7dc8025ba0 con 0x7f7dd80cc020 2024-01-11T08:10:58.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.723+0000 7f7dc67fc700 1 --2- 172.21.15.27:0/3105415772 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7db8066f30 0x7f7db80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:58.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.723+0000 7f7dc67fc700 1 -- 172.21.15.27:0/3105415772 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f7dc8085ed0 con 0x7f7dd80cc020 2024-01-11T08:10:58.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.723+0000 7f7dd5d9b700 1 --2- 172.21.15.27:0/3105415772 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7db8066f30 0x7f7db80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:58.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.724+0000 7f7dd5d9b700 1 --2- 172.21.15.27:0/3105415772 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7db8066f30 0x7f7db80693e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7dd80b3530 tx=0x7f7dcc0091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:58.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.727+0000 7f7dc67fc700 1 -- 172.21.15.27:0/3105415772 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7dc80530a0 con 0x7f7dd80cc020 2024-01-11T08:10:58.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.928+0000 7f7ddcfff700 1 -- 172.21.15.27:0/3105415772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7dd80cbd80 con 0x7f7dd80cc020 2024-01-11T08:10:58.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.929+0000 7f7dc67fc700 1 -- 172.21.15.27:0/3105415772 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7dc80566c0 con 0x7f7dd80cc020 2024-01-11T08:10:58.929 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:10:58.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.931+0000 7f7ddcfff700 1 -- 172.21.15.27:0/3105415772 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7db8066f30 msgr2=0x7f7db80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:58.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.931+0000 7f7ddcfff700 1 --2- 172.21.15.27:0/3105415772 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7db8066f30 0x7f7db80693e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7dd80b3530 tx=0x7f7dcc0091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.931+0000 7f7ddcfff700 1 -- 172.21.15.27:0/3105415772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dd80cc020 msgr2=0x7f7dd813ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:58.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.931+0000 7f7ddcfff700 1 --2- 172.21.15.27:0/3105415772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dd80cc020 0x7f7dd813ec70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7dd815d6d0 tx=0x7f7dc8017000 comp rx=0 tx=0).stop 2024-01-11T08:10:58.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.931+0000 7f7ddcfff700 1 -- 172.21.15.27:0/3105415772 shutdown_connections 2024-01-11T08:10:58.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.931+0000 7f7ddcfff700 1 --2- 172.21.15.27:0/3105415772 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7db8066f30 0x7f7db80693e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.931+0000 7f7ddcfff700 1 --2- 172.21.15.27:0/3105415772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dd8143190 0x7f7dd80c3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.931+0000 7f7ddcfff700 1 --2- 172.21.15.27:0/3105415772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dd80cc020 0x7f7dd813ec70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.931+0000 7f7ddcfff700 1 --2- 172.21.15.27:0/3105415772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dd8142dc0 0x7f7dd80c7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:58.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.931+0000 7f7ddcfff700 1 -- 172.21.15.27:0/3105415772 >> 172.21.15.27:0/3105415772 conn(0x7f7dd8063ca0 msgr2=0x7f7dd80c4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:58.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.931+0000 7f7ddcfff700 1 -- 172.21.15.27:0/3105415772 shutdown_connections 2024-01-11T08:10:58.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:58.931+0000 7f7ddcfff700 1 -- 172.21.15.27:0/3105415772 wait complete. 2024-01-11T08:10:58.949 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T08:10:59.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.152+0000 7f87c63ff700 1 Processor -- start 2024-01-11T08:10:59.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.152+0000 7f87c63ff700 1 -- start start 2024-01-11T08:10:59.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.153+0000 7f87c63ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f87c00ccd70 0x7f87c00cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:59.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.153+0000 7f87c63ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87c015ae70 0x7f87c015b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:59.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.153+0000 7f87c63ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c00cf850 con 0x7f87c015a1b0 2024-01-11T08:10:59.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.154+0000 7f87c63ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87c015ac90 con 0x7f87c00ccd70 2024-01-11T08:10:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.154+0000 7f87c63ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c00cec40 con 0x7f87c015ae70 2024-01-11T08:10:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.154+0000 7f87bffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f87c015a1b0 0x7f87c015a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39000/0 (socket says 172.21.15.27:39000) 2024-01-11T08:10:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.154+0000 7f87bffff700 1 -- 172.21.15.27:0/575334408 learned_addr learned my addr 172.21.15.27:0/575334408 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:10:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.154+0000 7f87bf7fe700 1 --2- 172.21.15.27:0/575334408 >> v2:172.21.15.77:3300/0 conn(0x7f87c00ccd70 0x7f87c00cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.154+0000 7f87c499c700 1 --2- 172.21.15.27:0/575334408 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87c015ae70 0x7f87c015b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.154+0000 7f87beffd700 1 -- 172.21.15.27:0/575334408 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210262381 0 0) 0x7f87c00cf850 con 0x7f87c015a1b0 2024-01-11T08:10:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.154+0000 7f87beffd700 1 -- 172.21.15.27:0/575334408 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87a8003570 con 0x7f87c015a1b0 2024-01-11T08:10:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.154+0000 7f87beffd700 1 -- 172.21.15.27:0/575334408 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3308598466 0 0) 0x7f87a8003570 con 0x7f87c015a1b0 2024-01-11T08:10:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87beffd700 1 -- 172.21.15.27:0/575334408 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87c015ae70 msgr2=0x7f87c015b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87beffd700 1 --2- 172.21.15.27:0/575334408 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87c015ae70 0x7f87c015b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87beffd700 1 -- 172.21.15.27:0/575334408 >> v2:172.21.15.77:3300/0 conn(0x7f87c00ccd70 msgr2=0x7f87c00cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87beffd700 1 --2- 172.21.15.27:0/575334408 >> v2:172.21.15.77:3300/0 conn(0x7f87c00ccd70 0x7f87c00cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87beffd700 1 -- 172.21.15.27:0/575334408 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c00cef20 con 0x7f87c015a1b0 2024-01-11T08:10:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87beffd700 1 -- 172.21.15.27:0/575334408 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f87b0001470 con 0x7f87c015a1b0 2024-01-11T08:10:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87beffd700 1 -- 172.21.15.27:0/575334408 >> v1:172.21.15.27:6789/0 conn(0x7f87c015a1b0 legacy=0x7f87c015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87beffd700 1 --2- 172.21.15.27:0/575334408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87a8003ae0 0x7f87a8003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87beffd700 1 --2- 172.21.15.27:0/575334408 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87c015ae70 0x7f87a8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87beffd700 1 --2- 172.21.15.27:0/575334408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87a8004d50 0x7f87a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87beffd700 1 -- 172.21.15.27:0/575334408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c015ac90 con 0x7f87a8004d50 2024-01-11T08:10:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87beffd700 1 -- 172.21.15.27:0/575334408 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c00cec40 con 0x7f87a8003ae0 2024-01-11T08:10:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87beffd700 1 -- 172.21.15.27:0/575334408 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a800bb10 con 0x7f87c015ae70 2024-01-11T08:10:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87c499c700 1 --2- 172.21.15.27:0/575334408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87a8004d50 0x7f87a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87bffff700 1 --2- 172.21.15.27:0/575334408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87a8003ae0 0x7f87a8003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87bf7fe700 1 --2- 172.21.15.27:0/575334408 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87c015ae70 0x7f87a8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87c499c700 1 -- 172.21.15.27:0/575334408 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87c015ae70 msgr2=0x7f87a8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87c499c700 1 --2- 172.21.15.27:0/575334408 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87c015ae70 0x7f87a8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87c499c700 1 -- 172.21.15.27:0/575334408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87a8003ae0 msgr2=0x7f87a8003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87c499c700 1 --2- 172.21.15.27:0/575334408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87a8003ae0 0x7f87a8003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.155+0000 7f87c499c700 1 -- 172.21.15.27:0/575334408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a8003570 con 0x7f87a8004d50 2024-01-11T08:10:59.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.156+0000 7f87c499c700 1 --2- 172.21.15.27:0/575334408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87a8004d50 0x7f87a80051c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f87b800e240 tx=0x7f87b800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18940de0dde680e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:59.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.156+0000 7f87beffd700 1 -- 172.21.15.27:0/575334408 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b800eaf0 con 0x7f87a8004d50 2024-01-11T08:10:59.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.156+0000 7f87beffd700 1 -- 172.21.15.27:0/575334408 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b800f040 con 0x7f87a8004d50 2024-01-11T08:10:59.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.156+0000 7f87beffd700 1 -- 172.21.15.27:0/575334408 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b800eea0 con 0x7f87a8004d50 2024-01-11T08:10:59.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.156+0000 7f87c63ff700 1 -- 172.21.15.27:0/575334408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87a8004d50 msgr2=0x7f87a80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:59.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.156+0000 7f87c63ff700 1 --2- 172.21.15.27:0/575334408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87a8004d50 0x7f87a80051c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f87b800e240 tx=0x7f87b800b040 comp rx=0 tx=0).stop 2024-01-11T08:10:59.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.156+0000 7f87c499c700 1 -- 172.21.15.27:0/575334408 reap_dead start 2024-01-11T08:10:59.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.157+0000 7f87c63ff700 1 -- 172.21.15.27:0/575334408 shutdown_connections 2024-01-11T08:10:59.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.157+0000 7f87c63ff700 1 -- 172.21.15.27:0/575334408 >> 172.21.15.27:0/575334408 conn(0x7f87c006ed40 msgr2=0x7f87c006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:59.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.157+0000 7f87c63ff700 1 -- 172.21.15.27:0/575334408 shutdown_connections 2024-01-11T08:10:59.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.157+0000 7f87c63ff700 1 -- 172.21.15.27:0/575334408 wait complete. 2024-01-11T08:10:59.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.158+0000 7f87c63ff700 1 Processor -- start 2024-01-11T08:10:59.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.158+0000 7f87c63ff700 1 -- start start 2024-01-11T08:10:59.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.159+0000 7f87c63ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87c014b0b0 0x7f87c00b2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:59.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.159+0000 7f87c63ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c014b480 0x7f87c01582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:59.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.159+0000 7f87c63ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87c0158800 0x7f87c00b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:59.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.159+0000 7f87c63ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c00b6480 con 0x7f87c014b480 2024-01-11T08:10:59.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.159+0000 7f87c63ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c00b2170 con 0x7f87c014b0b0 2024-01-11T08:10:59.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.159+0000 7f87c63ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c00b5ec0 con 0x7f87c0158800 2024-01-11T08:10:59.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.160+0000 7f87bf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c014b480 0x7f87c01582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:59.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.160+0000 7f87bf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c014b480 0x7f87c01582c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53362/0 (socket says 172.21.15.27:53362) 2024-01-11T08:10:59.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.160+0000 7f87bf7fe700 1 -- 172.21.15.27:0/366522812 learned_addr learned my addr 172.21.15.27:0/366522812 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:59.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.160+0000 7f87c499c700 1 --2- 172.21.15.27:0/366522812 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87c0158800 0x7f87c00b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:59.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.160+0000 7f87bffff700 1 --2- 172.21.15.27:0/366522812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87c014b0b0 0x7f87c00b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:59.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.160+0000 7f87bf7fe700 1 -- 172.21.15.27:0/366522812 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87c0158800 msgr2=0x7f87c00b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:59.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.160+0000 7f87bf7fe700 1 --2- 172.21.15.27:0/366522812 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87c0158800 0x7f87c00b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.160+0000 7f87bf7fe700 1 -- 172.21.15.27:0/366522812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87c014b0b0 msgr2=0x7f87c00b2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:59.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.160+0000 7f87bf7fe700 1 --2- 172.21.15.27:0/366522812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87c014b0b0 0x7f87c00b2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.160+0000 7f87bf7fe700 1 -- 172.21.15.27:0/366522812 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c00b61a0 con 0x7f87c014b480 2024-01-11T08:10:59.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.160+0000 7f87bf7fe700 1 --2- 172.21.15.27:0/366522812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c014b480 0x7f87c01582c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f87b4006ac0 tx=0x7f87b40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:59.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.161+0000 7f87a7fff700 1 -- 172.21.15.27:0/366522812 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b4011ec0 con 0x7f87c014b480 2024-01-11T08:10:59.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.161+0000 7f87a7fff700 1 -- 172.21.15.27:0/366522812 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b400f940 con 0x7f87c014b480 2024-01-11T08:10:59.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.161+0000 7f87a7fff700 1 -- 172.21.15.27:0/366522812 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b400fcc0 con 0x7f87c014b480 2024-01-11T08:10:59.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.161+0000 7f87c63ff700 1 -- 172.21.15.27:0/366522812 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c00b1e00 con 0x7f87c014b480 2024-01-11T08:10:59.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.161+0000 7f87c63ff700 1 -- 172.21.15.27:0/366522812 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c0158f40 con 0x7f87c014b480 2024-01-11T08:10:59.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.162+0000 7f87c63ff700 1 -- 172.21.15.27:0/366522812 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c00b1c60 con 0x7f87c014b480 2024-01-11T08:10:59.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.163+0000 7f87a7fff700 1 -- 172.21.15.27:0/366522812 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f87b4014650 con 0x7f87c014b480 2024-01-11T08:10:59.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.164+0000 7f87a7fff700 1 --2- 172.21.15.27:0/366522812 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f87ac066f30 0x7f87ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:59.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.164+0000 7f87a7fff700 1 -- 172.21.15.27:0/366522812 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f87b4018030 con 0x7f87c014b480 2024-01-11T08:10:59.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.164+0000 7f87bffff700 1 --2- 172.21.15.27:0/366522812 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f87ac066f30 0x7f87ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:59.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.165+0000 7f87bffff700 1 --2- 172.21.15.27:0/366522812 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f87ac066f30 0x7f87ac0693e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f87b0001ed0 tx=0x7f87b0001dc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:59.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.168+0000 7f87a7fff700 1 -- 172.21.15.27:0/366522812 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f87c00b1c60 con 0x7f87c014b480 2024-01-11T08:10:59.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.370+0000 7f87c63ff700 1 -- 172.21.15.27:0/366522812 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f87c00b1c60 con 0x7f87c014b480 2024-01-11T08:10:59.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.371+0000 7f87a7fff700 1 -- 172.21.15.27:0/366522812 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f87b4022020 con 0x7f87c014b480 2024-01-11T08:10:59.370 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T08:10:59.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.373+0000 7f87c63ff700 1 -- 172.21.15.27:0/366522812 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f87ac066f30 msgr2=0x7f87ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:59.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.373+0000 7f87c63ff700 1 --2- 172.21.15.27:0/366522812 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f87ac066f30 0x7f87ac0693e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f87b0001ed0 tx=0x7f87b0001dc0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.373+0000 7f87c63ff700 1 -- 172.21.15.27:0/366522812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c014b480 msgr2=0x7f87c01582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:59.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.373+0000 7f87c63ff700 1 --2- 172.21.15.27:0/366522812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c014b480 0x7f87c01582c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f87b4006ac0 tx=0x7f87b40091a0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.373+0000 7f87c63ff700 1 -- 172.21.15.27:0/366522812 shutdown_connections 2024-01-11T08:10:59.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.373+0000 7f87c63ff700 1 --2- 172.21.15.27:0/366522812 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f87ac066f30 0x7f87ac0693e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.373+0000 7f87c63ff700 1 --2- 172.21.15.27:0/366522812 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f87c0158800 0x7f87c00b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.373+0000 7f87c63ff700 1 --2- 172.21.15.27:0/366522812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87c014b0b0 0x7f87c00b2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.373+0000 7f87c63ff700 1 --2- 172.21.15.27:0/366522812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c014b480 0x7f87c01582c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.373+0000 7f87c63ff700 1 -- 172.21.15.27:0/366522812 >> 172.21.15.27:0/366522812 conn(0x7f87c006ed40 msgr2=0x7f87c015c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:59.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.373+0000 7f87c63ff700 1 -- 172.21.15.27:0/366522812 shutdown_connections 2024-01-11T08:10:59.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.373+0000 7f87c63ff700 1 -- 172.21.15.27:0/366522812 wait complete. 2024-01-11T08:10:59.391 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T08:10:59.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.597+0000 7f9ccf7ff700 1 Processor -- start 2024-01-11T08:10:59.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.597+0000 7f9ccf7ff700 1 -- start start 2024-01-11T08:10:59.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.598+0000 7f9ccf7ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9cc80ccd70 0x7f9cc80cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:59.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.598+0000 7f9ccf7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9cc815ae70 0x7f9cc815b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:59.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.598+0000 7f9ccf7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc80cf850 con 0x7f9cc815a1b0 2024-01-11T08:10:59.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.598+0000 7f9ccf7ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc815ac90 con 0x7f9cc80ccd70 2024-01-11T08:10:59.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.598+0000 7f9ccf7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc80cec40 con 0x7f9cc815ae70 2024-01-11T08:10:59.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.599+0000 7f9cccd9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9cc80ccd70 0x7f9cc80cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:59.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.599+0000 7f9cccd9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9cc80ccd70 0x7f9cc80cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53372/0 (socket says 172.21.15.27:53372) 2024-01-11T08:10:59.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.599+0000 7f9ccd59b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9cc815a1b0 0x7f9cc815a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39006/0 (socket says 172.21.15.27:39006) 2024-01-11T08:10:59.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.599+0000 7f9cccd9a700 1 -- 172.21.15.27:0/303369383 learned_addr learned my addr 172.21.15.27:0/303369383 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:59.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.599+0000 7f9ccdd9c700 1 --2- 172.21.15.27:0/303369383 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9cc815ae70 0x7f9cc815b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:59.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.599+0000 7f9cccd9a700 1 -- 172.21.15.27:0/303369383 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9cc815ae70 msgr2=0x7f9cc815b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:10:59.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.599+0000 7f9cccd9a700 1 --2- 172.21.15.27:0/303369383 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9cc815ae70 0x7f9cc815b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.599+0000 7f9cccd9a700 1 -- 172.21.15.27:0/303369383 >> v1:172.21.15.27:6789/0 conn(0x7f9cc815a1b0 legacy=0x7f9cc815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:59.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.599+0000 7f9cccd9a700 1 -- 172.21.15.27:0/303369383 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc80cef20 con 0x7f9cc80ccd70 2024-01-11T08:10:59.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.599+0000 7f9ccdd9c700 1 --2- 172.21.15.27:0/303369383 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9cc815ae70 0x7f9cc815b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:59.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.599+0000 7f9cbffff700 1 -- 172.21.15.27:0/303369383 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093316444 0 0) 0x7f9cc80cf850 con 0x7f9cc815a1b0 2024-01-11T08:10:59.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.599+0000 7f9cccd9a700 1 --2- 172.21.15.27:0/303369383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cc80ccd70 0x7f9cc80cd210 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9cb80099b0 tx=0x7f9cb8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c317dd9d9b643c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:59.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.600+0000 7f9cbffff700 1 -- 172.21.15.27:0/303369383 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb800b860 con 0x7f9cc80ccd70 2024-01-11T08:10:59.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.600+0000 7f9cbffff700 1 -- 172.21.15.27:0/303369383 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb800b9c0 con 0x7f9cc80ccd70 2024-01-11T08:10:59.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.600+0000 7f9cbffff700 1 -- 172.21.15.27:0/303369383 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb800bd60 con 0x7f9cc80ccd70 2024-01-11T08:10:59.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.600+0000 7f9ccf7ff700 1 -- 172.21.15.27:0/303369383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cc80ccd70 msgr2=0x7f9cc80cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.600+0000 7f9ccf7ff700 1 --2- 172.21.15.27:0/303369383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cc80ccd70 0x7f9cc80cd210 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9cb80099b0 tx=0x7f9cb8009250 comp rx=0 tx=0).stop 2024-01-11T08:10:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.600+0000 7f9ccf7ff700 1 -- 172.21.15.27:0/303369383 shutdown_connections 2024-01-11T08:10:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.600+0000 7f9ccf7ff700 1 --2- 172.21.15.27:0/303369383 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9cc815ae70 0x7f9cc815b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.600+0000 7f9ccf7ff700 1 --2- 172.21.15.27:0/303369383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cc80ccd70 0x7f9cc80cd210 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.600+0000 7f9ccf7ff700 1 -- 172.21.15.27:0/303369383 >> 172.21.15.27:0/303369383 conn(0x7f9cc806ed40 msgr2=0x7f9cc806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.600+0000 7f9ccf7ff700 1 -- 172.21.15.27:0/303369383 shutdown_connections 2024-01-11T08:10:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.600+0000 7f9ccf7ff700 1 -- 172.21.15.27:0/303369383 wait complete. 2024-01-11T08:10:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.602+0000 7f9ccf7ff700 1 Processor -- start 2024-01-11T08:10:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.602+0000 7f9ccf7ff700 1 -- start start 2024-01-11T08:10:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.603+0000 7f9ccf7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cc80ccd70 0x7f9cc80c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.603+0000 7f9ccf7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9cc815a1b0 0x7f9cc814c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.603+0000 7f9ccf7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cc815ae70 0x7f9cc80c56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.603+0000 7f9ccf7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc80c9cb0 con 0x7f9cc815ae70 2024-01-11T08:10:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.603+0000 7f9ccf7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc814c700 con 0x7f9cc80ccd70 2024-01-11T08:10:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.603+0000 7f9ccf7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc80b0780 con 0x7f9cc815a1b0 2024-01-11T08:10:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.603+0000 7f9ccdd9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cc815ae70 0x7f9cc80c56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.603+0000 7f9ccdd9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cc815ae70 0x7f9cc80c56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53388/0 (socket says 172.21.15.27:53388) 2024-01-11T08:10:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.603+0000 7f9ccdd9c700 1 -- 172.21.15.27:0/3798194030 learned_addr learned my addr 172.21.15.27:0/3798194030 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:10:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.603+0000 7f9cccd9a700 1 --2- 172.21.15.27:0/3798194030 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9cc815a1b0 0x7f9cc814c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.603+0000 7f9ccd59b700 1 --2- 172.21.15.27:0/3798194030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cc80ccd70 0x7f9cc80c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.604+0000 7f9ccdd9c700 1 -- 172.21.15.27:0/3798194030 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9cc815a1b0 msgr2=0x7f9cc814c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.604+0000 7f9ccdd9c700 1 --2- 172.21.15.27:0/3798194030 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9cc815a1b0 0x7f9cc814c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.604+0000 7f9ccdd9c700 1 -- 172.21.15.27:0/3798194030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cc80ccd70 msgr2=0x7f9cc80c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.604+0000 7f9ccdd9c700 1 --2- 172.21.15.27:0/3798194030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cc80ccd70 0x7f9cc80c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.604+0000 7f9ccdd9c700 1 -- 172.21.15.27:0/3798194030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb80094f0 con 0x7f9cc815ae70 2024-01-11T08:10:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.604+0000 7f9ccd59b700 1 --2- 172.21.15.27:0/3798194030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cc80ccd70 0x7f9cc80c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:10:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.604+0000 7f9ccdd9c700 1 --2- 172.21.15.27:0/3798194030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cc815ae70 0x7f9cc80c56b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9cc000d1b0 tx=0x7f9cc0009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.604+0000 7f9cbd7fa700 1 -- 172.21.15.27:0/3798194030 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc000dda0 con 0x7f9cc815ae70 2024-01-11T08:10:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.604+0000 7f9cbd7fa700 1 -- 172.21.15.27:0/3798194030 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc000f040 con 0x7f9cc815ae70 2024-01-11T08:10:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.604+0000 7f9ccf7ff700 1 -- 172.21.15.27:0/3798194030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc80b0a60 con 0x7f9cc815ae70 2024-01-11T08:10:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.604+0000 7f9cbd7fa700 1 -- 172.21.15.27:0/3798194030 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc0015690 con 0x7f9cc815ae70 2024-01-11T08:10:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.604+0000 7f9ccf7ff700 1 -- 172.21.15.27:0/3798194030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc80b1020 con 0x7f9cc815ae70 2024-01-11T08:10:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.605+0000 7f9ccf7ff700 1 -- 172.21.15.27:0/3798194030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc804c3c0 con 0x7f9cc815ae70 2024-01-11T08:10:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.606+0000 7f9cbd7fa700 1 -- 172.21.15.27:0/3798194030 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9cc001e3b0 con 0x7f9cc815ae70 2024-01-11T08:10:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.607+0000 7f9cbd7fa700 1 --2- 172.21.15.27:0/3798194030 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9cac066f30 0x7f9cac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:10:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.607+0000 7f9cbd7fa700 1 -- 172.21.15.27:0/3798194030 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f9cc0084b00 con 0x7f9cc815ae70 2024-01-11T08:10:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.608+0000 7f9ccd59b700 1 --2- 172.21.15.27:0/3798194030 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9cac066f30 0x7f9cac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:10:59.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.608+0000 7f9ccd59b700 1 --2- 172.21.15.27:0/3798194030 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9cac066f30 0x7f9cac0693e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8156b00 tx=0x7f9cc4007cb0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:10:59.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.612+0000 7f9cbd7fa700 1 -- 172.21.15.27:0/3798194030 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9cc0086e60 con 0x7f9cc815ae70 2024-01-11T08:10:59.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.820+0000 7f9ccf7ff700 1 -- 172.21.15.27:0/3798194030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9cc80b2d60 con 0x7f9cc815ae70 2024-01-11T08:10:59.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.820+0000 7f9cbd7fa700 1 -- 172.21.15.27:0/3798194030 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9cc001a0a0 con 0x7f9cc815ae70 2024-01-11T08:10:59.820 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T08:10:59.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.822+0000 7f9ccf7ff700 1 -- 172.21.15.27:0/3798194030 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9cac066f30 msgr2=0x7f9cac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:59.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.822+0000 7f9ccf7ff700 1 --2- 172.21.15.27:0/3798194030 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9cac066f30 0x7f9cac0693e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8156b00 tx=0x7f9cc4007cb0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.822+0000 7f9ccf7ff700 1 -- 172.21.15.27:0/3798194030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cc815ae70 msgr2=0x7f9cc80c56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:10:59.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.822+0000 7f9ccf7ff700 1 --2- 172.21.15.27:0/3798194030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cc815ae70 0x7f9cc80c56b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9cc000d1b0 tx=0x7f9cc0009a50 comp rx=0 tx=0).stop 2024-01-11T08:10:59.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.823+0000 7f9ccf7ff700 1 -- 172.21.15.27:0/3798194030 shutdown_connections 2024-01-11T08:10:59.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.823+0000 7f9ccf7ff700 1 --2- 172.21.15.27:0/3798194030 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9cac066f30 0x7f9cac0693e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.823+0000 7f9ccf7ff700 1 --2- 172.21.15.27:0/3798194030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cc815ae70 0x7f9cc80c56b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.823+0000 7f9ccf7ff700 1 --2- 172.21.15.27:0/3798194030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9cc80ccd70 0x7f9cc80c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.823+0000 7f9ccf7ff700 1 --2- 172.21.15.27:0/3798194030 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9cc815a1b0 0x7f9cc814c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:10:59.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.823+0000 7f9ccf7ff700 1 -- 172.21.15.27:0/3798194030 >> 172.21.15.27:0/3798194030 conn(0x7f9cc806ed40 msgr2=0x7f9cc8158620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:10:59.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.823+0000 7f9ccf7ff700 1 -- 172.21.15.27:0/3798194030 shutdown_connections 2024-01-11T08:10:59.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:10:59.823+0000 7f9ccf7ff700 1 -- 172.21.15.27:0/3798194030 wait complete. 2024-01-11T08:10:59.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:11:00.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.040+0000 7f39967ff700 1 Processor -- start 2024-01-11T08:11:00.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.041+0000 7f39967ff700 1 -- start start 2024-01-11T08:11:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.042+0000 7f39967ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39900b03e0 0x7f39900b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.042+0000 7f39967ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39900b29c0 0x7f3990159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.042+0000 7f39967ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39900cd730 con 0x7f39900b07b0 2024-01-11T08:11:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.042+0000 7f39967ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39900b2de0 con 0x7f39900b03e0 2024-01-11T08:11:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.042+0000 7f39967ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39900ccb20 con 0x7f39900b29c0 2024-01-11T08:11:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.042+0000 7f398ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39900b03e0 0x7f39900b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.042+0000 7f398ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39900b03e0 0x7f39900b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53394/0 (socket says 172.21.15.27:53394) 2024-01-11T08:11:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.042+0000 7f398ffff700 1 -- 172.21.15.27:0/2555594531 learned_addr learned my addr 172.21.15.27:0/2555594531 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.042+0000 7f3994d9c700 1 --2- 172.21.15.27:0/2555594531 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39900b29c0 0x7f3990159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:00.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.043+0000 7f398ffff700 1 -- 172.21.15.27:0/2555594531 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39900b29c0 msgr2=0x7f3990159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:00.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.043+0000 7f398ffff700 1 --2- 172.21.15.27:0/2555594531 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39900b29c0 0x7f3990159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.043+0000 7f398ffff700 1 -- 172.21.15.27:0/2555594531 >> v1:172.21.15.77:6789/0 conn(0x7f39900b07b0 legacy=0x7f3990159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:00.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.043+0000 7f398ffff700 1 -- 172.21.15.27:0/2555594531 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39900cce00 con 0x7f39900b03e0 2024-01-11T08:11:00.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.043+0000 7f398ffff700 1 --2- 172.21.15.27:0/2555594531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39900b03e0 0x7f39900b2480 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f39800099b0 tx=0x7f3980009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cf05af57c069cdc server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:00.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.043+0000 7f398effd700 1 -- 172.21.15.27:0/2555594531 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398000b860 con 0x7f39900b03e0 2024-01-11T08:11:00.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.043+0000 7f398effd700 1 -- 172.21.15.27:0/2555594531 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398000b9c0 con 0x7f39900b03e0 2024-01-11T08:11:00.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.043+0000 7f398effd700 1 -- 172.21.15.27:0/2555594531 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398000bd60 con 0x7f39900b03e0 2024-01-11T08:11:00.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.044+0000 7f39967ff700 1 -- 172.21.15.27:0/2555594531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39900b03e0 msgr2=0x7f39900b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:00.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.044+0000 7f39967ff700 1 --2- 172.21.15.27:0/2555594531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39900b03e0 0x7f39900b2480 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f39800099b0 tx=0x7f3980009250 comp rx=0 tx=0).stop 2024-01-11T08:11:00.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.044+0000 7f39967ff700 1 -- 172.21.15.27:0/2555594531 shutdown_connections 2024-01-11T08:11:00.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.044+0000 7f39967ff700 1 --2- 172.21.15.27:0/2555594531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39900b03e0 0x7f39900b2480 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.044+0000 7f39967ff700 1 --2- 172.21.15.27:0/2555594531 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39900b29c0 0x7f3990159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.044+0000 7f39967ff700 1 -- 172.21.15.27:0/2555594531 >> 172.21.15.27:0/2555594531 conn(0x7f399006ed40 msgr2=0x7f399006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:00.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.044+0000 7f39967ff700 1 -- 172.21.15.27:0/2555594531 shutdown_connections 2024-01-11T08:11:00.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.044+0000 7f39967ff700 1 -- 172.21.15.27:0/2555594531 wait complete. 2024-01-11T08:11:00.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.045+0000 7f39967ff700 1 Processor -- start 2024-01-11T08:11:00.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.046+0000 7f39967ff700 1 -- start start 2024-01-11T08:11:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.046+0000 7f39967ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39900b03e0 0x7f39900d0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.046+0000 7f39967ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39900b07b0 0x7f39900c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.046+0000 7f39967ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39900b29c0 0x7f39900c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.046+0000 7f39967ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39900ca490 con 0x7f39900b29c0 2024-01-11T08:11:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.046+0000 7f39967ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39900d0570 con 0x7f39900b07b0 2024-01-11T08:11:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.046+0000 7f39967ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39900d0810 con 0x7f39900b03e0 2024-01-11T08:11:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.047+0000 7f3994d9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39900b29c0 0x7f39900c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.047+0000 7f3994d9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39900b29c0 0x7f39900c8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53400/0 (socket says 172.21.15.27:53400) 2024-01-11T08:11:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.047+0000 7f3994d9c700 1 -- 172.21.15.27:0/2042852351 learned_addr learned my addr 172.21.15.27:0/2042852351 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.047+0000 7f398ffff700 1 --2- 172.21.15.27:0/2042852351 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39900b03e0 0x7f39900d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.047+0000 7f398f7fe700 1 --2- 172.21.15.27:0/2042852351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39900b07b0 0x7f39900c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:00.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.047+0000 7f3994d9c700 1 -- 172.21.15.27:0/2042852351 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39900b03e0 msgr2=0x7f39900d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:00.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.047+0000 7f3994d9c700 1 --2- 172.21.15.27:0/2042852351 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39900b03e0 0x7f39900d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.047+0000 7f3994d9c700 1 -- 172.21.15.27:0/2042852351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39900b07b0 msgr2=0x7f39900c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:00.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.047+0000 7f3994d9c700 1 --2- 172.21.15.27:0/2042852351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39900b07b0 0x7f39900c8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.047+0000 7f3994d9c700 1 -- 172.21.15.27:0/2042852351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39800094f0 con 0x7f39900b29c0 2024-01-11T08:11:00.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.047+0000 7f398f7fe700 1 --2- 172.21.15.27:0/2042852351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39900b07b0 0x7f39900c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:00.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.047+0000 7f3994d9c700 1 --2- 172.21.15.27:0/2042852351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39900b29c0 0x7f39900c8e90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3988007f20 tx=0x7f3988009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:00.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.048+0000 7f3977fff700 1 -- 172.21.15.27:0/2042852351 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398800ebf0 con 0x7f39900b29c0 2024-01-11T08:11:00.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.048+0000 7f3977fff700 1 -- 172.21.15.27:0/2042852351 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398800ed70 con 0x7f39900b29c0 2024-01-11T08:11:00.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.048+0000 7f3977fff700 1 -- 172.21.15.27:0/2042852351 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3988014670 con 0x7f39900b29c0 2024-01-11T08:11:00.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.048+0000 7f39967ff700 1 -- 172.21.15.27:0/2042852351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39900d0af0 con 0x7f39900b29c0 2024-01-11T08:11:00.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.048+0000 7f39967ff700 1 -- 172.21.15.27:0/2042852351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39900d10b0 con 0x7f39900b29c0 2024-01-11T08:11:00.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.049+0000 7f39967ff700 1 -- 172.21.15.27:0/2042852351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39900718d0 con 0x7f39900b29c0 2024-01-11T08:11:00.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.050+0000 7f3977fff700 1 -- 172.21.15.27:0/2042852351 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f39880149e0 con 0x7f39900b29c0 2024-01-11T08:11:00.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.051+0000 7f3977fff700 1 --2- 172.21.15.27:0/2042852351 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f397c066f30 0x7f397c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:00.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.051+0000 7f3977fff700 1 -- 172.21.15.27:0/2042852351 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f3988084c10 con 0x7f39900b29c0 2024-01-11T08:11:00.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.052+0000 7f398ffff700 1 --2- 172.21.15.27:0/2042852351 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f397c066f30 0x7f397c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:00.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.052+0000 7f398ffff700 1 --2- 172.21.15.27:0/2042852351 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f397c066f30 0x7f397c0693e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f39800099b0 tx=0x7f3980017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:00.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.056+0000 7f3977fff700 1 -- 172.21.15.27:0/2042852351 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3988051de0 con 0x7f39900b29c0 2024-01-11T08:11:00.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.260+0000 7f39967ff700 1 -- 172.21.15.27:0/2042852351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39900cd0e0 con 0x7f39900b29c0 2024-01-11T08:11:00.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.261+0000 7f3977fff700 1 -- 172.21.15.27:0/2042852351 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15302 (secure 0 0 0) 0x7f3988055400 con 0x7f39900b29c0 2024-01-11T08:11:00.261 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:00.261 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:31.464323+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:10:27.993251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-11T08:10:29.153959+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:11:00.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.264+0000 7f39967ff700 1 -- 172.21.15.27:0/2042852351 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f397c066f30 msgr2=0x7f397c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:00.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.264+0000 7f39967ff700 1 --2- 172.21.15.27:0/2042852351 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f397c066f30 0x7f397c0693e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f39800099b0 tx=0x7f3980017000 comp rx=0 tx=0).stop 2024-01-11T08:11:00.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.264+0000 7f39967ff700 1 -- 172.21.15.27:0/2042852351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39900b29c0 msgr2=0x7f39900c8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:00.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.264+0000 7f39967ff700 1 --2- 172.21.15.27:0/2042852351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39900b29c0 0x7f39900c8e90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3988007f20 tx=0x7f3988009860 comp rx=0 tx=0).stop 2024-01-11T08:11:00.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.264+0000 7f39967ff700 1 -- 172.21.15.27:0/2042852351 shutdown_connections 2024-01-11T08:11:00.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.264+0000 7f39967ff700 1 --2- 172.21.15.27:0/2042852351 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f397c066f30 0x7f397c0693e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.264+0000 7f39967ff700 1 --2- 172.21.15.27:0/2042852351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39900b29c0 0x7f39900c8e90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.264+0000 7f39967ff700 1 --2- 172.21.15.27:0/2042852351 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39900b03e0 0x7f39900d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.264+0000 7f39967ff700 1 --2- 172.21.15.27:0/2042852351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39900b07b0 0x7f39900c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.264+0000 7f39967ff700 1 -- 172.21.15.27:0/2042852351 >> 172.21.15.27:0/2042852351 conn(0x7f399006ed40 msgr2=0x7f39900b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:00.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.264+0000 7f39967ff700 1 -- 172.21.15.27:0/2042852351 shutdown_connections 2024-01-11T08:11:00.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.264+0000 7f39967ff700 1 -- 172.21.15.27:0/2042852351 wait complete. 2024-01-11T08:11:00.281 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T08:10:27.993251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T08:10:29.153959+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T08:11:00.282 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:11:00.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.485+0000 7f24a7bff700 1 Processor -- start 2024-01-11T08:11:00.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.485+0000 7f24a7bff700 1 -- start start 2024-01-11T08:11:00.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.486+0000 7f24a7bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f24a015a1b0 0x7f24a015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:00.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.486+0000 7f24a7bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24a015ae70 0x7f24a015b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:00.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.486+0000 7f24a7bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a00cf850 con 0x7f24a00ccd70 2024-01-11T08:11:00.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.486+0000 7f24a7bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a00cec40 con 0x7f24a015a1b0 2024-01-11T08:11:00.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.486+0000 7f24a7bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a00cef20 con 0x7f24a015ae70 2024-01-11T08:11:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.487+0000 7f24a619c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24a015ae70 0x7f24a015b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.487+0000 7f24a599b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f24a015a1b0 0x7f24a015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.487+0000 7f24a519a700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f24a00ccd70 0x7f24a00cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:51720/0 (socket says 172.21.15.27:51720) 2024-01-11T08:11:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.487+0000 7f24a599b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f24a015a1b0 0x7f24a015a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:54866/0 (socket says 172.21.15.27:54866) 2024-01-11T08:11:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.487+0000 7f24a619c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24a015ae70 0x7f24a015b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:60410/0 (socket says 172.21.15.27:60410) 2024-01-11T08:11:00.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.487+0000 7f24a519a700 1 -- 172.21.15.27:0/1091862628 learned_addr learned my addr 172.21.15.27:0/1091862628 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:00.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.487+0000 7f24a599b700 1 -- 172.21.15.27:0/1091862628 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24a015ae70 msgr2=0x7f24a015b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:00.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.487+0000 7f24a599b700 1 --2- 172.21.15.27:0/1091862628 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24a015ae70 0x7f24a015b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.487+0000 7f24a599b700 1 -- 172.21.15.27:0/1091862628 >> v1:172.21.15.77:6789/0 conn(0x7f24a00ccd70 legacy=0x7f24a00cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:00.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.487+0000 7f24a599b700 1 -- 172.21.15.27:0/1091862628 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a00cf200 con 0x7f24a015a1b0 2024-01-11T08:11:00.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.487+0000 7f24a619c700 1 --2- 172.21.15.27:0/1091862628 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24a015ae70 0x7f24a015b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:00.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.487+0000 7f24a4999700 1 -- 172.21.15.27:0/1091862628 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650933003 0 0) 0x7f24a00cf850 con 0x7f24a00ccd70 2024-01-11T08:11:00.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.488+0000 7f24a599b700 1 --2- 172.21.15.27:0/1091862628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24a015a1b0 0x7f24a015a580 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f249c0098e0 tx=0x7f249c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b97ed70f06b9a2cc server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:00.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.488+0000 7f24a4999700 1 -- 172.21.15.27:0/1091862628 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c00b640 con 0x7f24a015a1b0 2024-01-11T08:11:00.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.488+0000 7f24a4999700 1 -- 172.21.15.27:0/1091862628 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249c00b7a0 con 0x7f24a015a1b0 2024-01-11T08:11:00.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.488+0000 7f24a4999700 1 -- 172.21.15.27:0/1091862628 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c00bb40 con 0x7f24a015a1b0 2024-01-11T08:11:00.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.488+0000 7f24a7bff700 1 -- 172.21.15.27:0/1091862628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24a015a1b0 msgr2=0x7f24a015a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:00.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.488+0000 7f24a7bff700 1 --2- 172.21.15.27:0/1091862628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24a015a1b0 0x7f24a015a580 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f249c0098e0 tx=0x7f249c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.488+0000 7f24a7bff700 1 -- 172.21.15.27:0/1091862628 shutdown_connections 2024-01-11T08:11:00.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.488+0000 7f24a7bff700 1 --2- 172.21.15.27:0/1091862628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24a015a1b0 0x7f24a015a580 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.488+0000 7f24a7bff700 1 --2- 172.21.15.27:0/1091862628 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24a015ae70 0x7f24a015b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.488+0000 7f24a7bff700 1 -- 172.21.15.27:0/1091862628 >> 172.21.15.27:0/1091862628 conn(0x7f24a006ed40 msgr2=0x7f24a006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:00.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.489+0000 7f24a7bff700 1 -- 172.21.15.27:0/1091862628 shutdown_connections 2024-01-11T08:11:00.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.489+0000 7f24a7bff700 1 -- 172.21.15.27:0/1091862628 wait complete. 2024-01-11T08:11:00.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.490+0000 7f24a7bff700 1 Processor -- start 2024-01-11T08:11:00.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.490+0000 7f24a7bff700 1 -- start start 2024-01-11T08:11:00.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.491+0000 7f24a7bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24a00ccd70 0x7f24a00b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:00.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.491+0000 7f24a7bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24a015a1b0 0x7f24a00c9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:00.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.491+0000 7f24a7bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a015ae70 0x7f24a0157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:00.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.491+0000 7f24a7bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a0169ae0 con 0x7f24a015ae70 2024-01-11T08:11:00.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.491+0000 7f24a7bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a00c95b0 con 0x7f24a00ccd70 2024-01-11T08:11:00.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.491+0000 7f24a7bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a0169d80 con 0x7f24a015a1b0 2024-01-11T08:11:00.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.491+0000 7f24a619c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a015ae70 0x7f24a0157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:00.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.492+0000 7f24a599b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24a00ccd70 0x7f24a00b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:00.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.492+0000 7f24a619c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a015ae70 0x7f24a0157dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53404/0 (socket says 172.21.15.27:53404) 2024-01-11T08:11:00.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.492+0000 7f24a619c700 1 -- 172.21.15.27:0/572524492 learned_addr learned my addr 172.21.15.27:0/572524492 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:00.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.492+0000 7f24a519a700 1 --2- 172.21.15.27:0/572524492 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24a015a1b0 0x7f24a00c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:00.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.492+0000 7f24a599b700 1 -- 172.21.15.27:0/572524492 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24a015a1b0 msgr2=0x7f24a00c9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:00.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.492+0000 7f24a599b700 1 --2- 172.21.15.27:0/572524492 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24a015a1b0 0x7f24a00c9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.492+0000 7f24a599b700 1 -- 172.21.15.27:0/572524492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a015ae70 msgr2=0x7f24a0157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:00.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.492+0000 7f24a599b700 1 --2- 172.21.15.27:0/572524492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a015ae70 0x7f24a0157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.492+0000 7f24a599b700 1 -- 172.21.15.27:0/572524492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2498009a20 con 0x7f24a00ccd70 2024-01-11T08:11:00.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.492+0000 7f24a619c700 1 --2- 172.21.15.27:0/572524492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a015ae70 0x7f24a0157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:00.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.492+0000 7f24a599b700 1 --2- 172.21.15.27:0/572524492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24a00ccd70 0x7f24a00b3e30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f249c016000 tx=0x7f249c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:00.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.493+0000 7f2495ffb700 1 -- 172.21.15.27:0/572524492 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c00bb20 con 0x7f24a00ccd70 2024-01-11T08:11:00.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.493+0000 7f2495ffb700 1 -- 172.21.15.27:0/572524492 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249c000c90 con 0x7f24a00ccd70 2024-01-11T08:11:00.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.493+0000 7f24a7bff700 1 -- 172.21.15.27:0/572524492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f249c009420 con 0x7f24a00ccd70 2024-01-11T08:11:00.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.493+0000 7f2495ffb700 1 -- 172.21.15.27:0/572524492 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c011650 con 0x7f24a00ccd70 2024-01-11T08:11:00.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.493+0000 7f24a7bff700 1 -- 172.21.15.27:0/572524492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a00c9aa0 con 0x7f24a00ccd70 2024-01-11T08:11:00.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.494+0000 7f24a7bff700 1 -- 172.21.15.27:0/572524492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a004c3c0 con 0x7f24a00ccd70 2024-01-11T08:11:00.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.495+0000 7f2495ffb700 1 -- 172.21.15.27:0/572524492 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f249c043bd0 con 0x7f24a00ccd70 2024-01-11T08:11:00.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.496+0000 7f2495ffb700 1 --2- 172.21.15.27:0/572524492 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2484066f30 0x7f24840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:00.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.496+0000 7f2495ffb700 1 -- 172.21.15.27:0/572524492 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f249c085a40 con 0x7f24a00ccd70 2024-01-11T08:11:00.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.496+0000 7f24a519a700 1 --2- 172.21.15.27:0/572524492 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2484066f30 0x7f24840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:00.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.497+0000 7f24a519a700 1 --2- 172.21.15.27:0/572524492 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2484066f30 0x7f24840693e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f24a00ce3a0 tx=0x7f2490007d30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:00.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.501+0000 7f2495ffb700 1 -- 172.21.15.27:0/572524492 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f249c052c10 con 0x7f24a00ccd70 2024-01-11T08:11:00.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.707+0000 7f24a7bff700 1 -- 172.21.15.27:0/572524492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f24a00686f0 con 0x7f24a00ccd70 2024-01-11T08:11:00.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.708+0000 7f2495ffb700 1 -- 172.21.15.27:0/572524492 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f249c056230 con 0x7f24a00ccd70 2024-01-11T08:11:00.708 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:11:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.710+0000 7f24a7bff700 1 -- 172.21.15.27:0/572524492 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2484066f30 msgr2=0x7f24840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.710+0000 7f24a7bff700 1 --2- 172.21.15.27:0/572524492 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2484066f30 0x7f24840693e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f24a00ce3a0 tx=0x7f2490007d30 comp rx=0 tx=0).stop 2024-01-11T08:11:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.710+0000 7f24a7bff700 1 -- 172.21.15.27:0/572524492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24a00ccd70 msgr2=0x7f24a00b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.711+0000 7f24a7bff700 1 --2- 172.21.15.27:0/572524492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24a00ccd70 0x7f24a00b3e30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f249c016000 tx=0x7f249c017000 comp rx=0 tx=0).stop 2024-01-11T08:11:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.711+0000 7f24a7bff700 1 -- 172.21.15.27:0/572524492 shutdown_connections 2024-01-11T08:11:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.711+0000 7f24a7bff700 1 --2- 172.21.15.27:0/572524492 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2484066f30 0x7f24840693e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.711+0000 7f24a7bff700 1 --2- 172.21.15.27:0/572524492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a015ae70 0x7f24a0157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.711+0000 7f24a7bff700 1 --2- 172.21.15.27:0/572524492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24a00ccd70 0x7f24a00b3e30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.711+0000 7f24a7bff700 1 --2- 172.21.15.27:0/572524492 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24a015a1b0 0x7f24a00c9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.711+0000 7f24a7bff700 1 -- 172.21.15.27:0/572524492 >> 172.21.15.27:0/572524492 conn(0x7f24a006ed40 msgr2=0x7f24a0068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.711+0000 7f24a7bff700 1 -- 172.21.15.27:0/572524492 shutdown_connections 2024-01-11T08:11:00.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.711+0000 7f24a7bff700 1 -- 172.21.15.27:0/572524492 wait complete. 2024-01-11T08:11:00.728 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T08:11:00.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.933+0000 7feeb53ff700 1 Processor -- start 2024-01-11T08:11:00.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.933+0000 7feeb53ff700 1 -- start start 2024-01-11T08:11:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.934+0000 7feeb53ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb0159ac0 0x7feeb0068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.934+0000 7feeb53ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb00c80a0 con 0x7feeb01596f0 2024-01-11T08:11:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.934+0000 7feeb53ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb00cbac0 con 0x7feeb00ca640 2024-01-11T08:11:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.934+0000 7feeb53ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb00a9fb0 con 0x7feeb0159ac0 2024-01-11T08:11:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.934+0000 7feeaeffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7feeb01596f0 0x7feeb00a98a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39022/0 (socket says 172.21.15.27:39022) 2024-01-11T08:11:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.934+0000 7feeaeffd700 1 -- 172.21.15.27:0/2440625174 learned_addr learned my addr 172.21.15.27:0/2440625174 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.935+0000 7feeae7fc700 1 --2- 172.21.15.27:0/2440625174 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb0159ac0 0x7feeb0068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.935+0000 7feeadffb700 1 -- 172.21.15.27:0/2440625174 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837380157 0 0) 0x7feeb00c80a0 con 0x7feeb01596f0 2024-01-11T08:11:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.935+0000 7feeadffb700 1 -- 172.21.15.27:0/2440625174 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee98003570 con 0x7feeb01596f0 2024-01-11T08:11:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.935+0000 7feeadffb700 1 -- 172.21.15.27:0/2440625174 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783568747 0 0) 0x7feeb00cbac0 con 0x7feeb00ca640 2024-01-11T08:11:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.935+0000 7feeadffb700 1 -- 172.21.15.27:0/2440625174 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeb00c80a0 con 0x7feeb00ca640 2024-01-11T08:11:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.935+0000 7feeae7fc700 1 -- 172.21.15.27:0/2440625174 >> v1:172.21.15.77:6789/0 conn(0x7feeb00ca640 legacy=0x7feeb00c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:00.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.935+0000 7feeae7fc700 1 -- 172.21.15.27:0/2440625174 >> v1:172.21.15.27:6789/0 conn(0x7feeb01596f0 legacy=0x7feeb00a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:00.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.935+0000 7feeae7fc700 1 -- 172.21.15.27:0/2440625174 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb00c7490 con 0x7feeb0159ac0 2024-01-11T08:11:00.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.935+0000 7feeae7fc700 1 --2- 172.21.15.27:0/2440625174 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb0159ac0 0x7feeb0068620 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7feea4009910 tx=0x7feea40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=442640cb4637e82e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:00.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.936+0000 7feeadffb700 1 -- 172.21.15.27:0/2440625174 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4011630 con 0x7feeb0159ac0 2024-01-11T08:11:00.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.936+0000 7feeadffb700 1 -- 172.21.15.27:0/2440625174 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea4011790 con 0x7feeb0159ac0 2024-01-11T08:11:00.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.936+0000 7feeadffb700 1 -- 172.21.15.27:0/2440625174 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4011b30 con 0x7feeb0159ac0 2024-01-11T08:11:00.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.936+0000 7feeb53ff700 1 -- 172.21.15.27:0/2440625174 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb0159ac0 msgr2=0x7feeb0068620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:00.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.936+0000 7feeb53ff700 1 --2- 172.21.15.27:0/2440625174 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb0159ac0 0x7feeb0068620 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7feea4009910 tx=0x7feea40091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.937+0000 7feeb53ff700 1 -- 172.21.15.27:0/2440625174 shutdown_connections 2024-01-11T08:11:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.937+0000 7feeb53ff700 1 --2- 172.21.15.27:0/2440625174 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb0159ac0 0x7feeb0068620 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.937+0000 7feeb53ff700 1 -- 172.21.15.27:0/2440625174 >> 172.21.15.27:0/2440625174 conn(0x7feeb00c9fb0 msgr2=0x7feeb006ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.937+0000 7feeb53ff700 1 -- 172.21.15.27:0/2440625174 shutdown_connections 2024-01-11T08:11:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.937+0000 7feeb53ff700 1 -- 172.21.15.27:0/2440625174 wait complete. 2024-01-11T08:11:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.938+0000 7feeb53ff700 1 Processor -- start 2024-01-11T08:11:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.938+0000 7feeb53ff700 1 -- start start 2024-01-11T08:11:00.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.939+0000 7feeb53ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb00ca640 0x7feeb00c4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:00.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.939+0000 7feeb53ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb01596f0 0x7feeb00ab960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:00.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.939+0000 7feeb53ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feeb0159ac0 0x7feeb00c8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:00.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.939+0000 7feeb53ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb00af1d0 con 0x7feeb00ca640 2024-01-11T08:11:00.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.939+0000 7feeb53ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb00c3ff0 con 0x7feeb0159ac0 2024-01-11T08:11:00.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.939+0000 7feeb53ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb00c3cc0 con 0x7feeb01596f0 2024-01-11T08:11:00.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.940+0000 7feeaeffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb00ca640 0x7feeb00c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:00.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.940+0000 7feeaeffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb00ca640 0x7feeb00c4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53422/0 (socket says 172.21.15.27:53422) 2024-01-11T08:11:00.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.940+0000 7feeaeffd700 1 -- 172.21.15.27:0/4155719615 learned_addr learned my addr 172.21.15.27:0/4155719615 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:00.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.940+0000 7feeae7fc700 1 --2- 172.21.15.27:0/4155719615 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb01596f0 0x7feeb00ab960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:00.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.940+0000 7feeaf7fe700 1 --2- 172.21.15.27:0/4155719615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feeb0159ac0 0x7feeb00c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:00.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.940+0000 7feeaeffd700 1 -- 172.21.15.27:0/4155719615 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb01596f0 msgr2=0x7feeb00ab960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:00.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.940+0000 7feeaeffd700 1 --2- 172.21.15.27:0/4155719615 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb01596f0 0x7feeb00ab960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.940+0000 7feeaeffd700 1 -- 172.21.15.27:0/4155719615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feeb0159ac0 msgr2=0x7feeb00c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:00.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.940+0000 7feeaeffd700 1 --2- 172.21.15.27:0/4155719615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feeb0159ac0 0x7feeb00c8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:00.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.940+0000 7feeaeffd700 1 -- 172.21.15.27:0/4155719615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea4009420 con 0x7feeb00ca640 2024-01-11T08:11:00.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.940+0000 7feeaeffd700 1 --2- 172.21.15.27:0/4155719615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb00ca640 0x7feeb00c4480 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7feea000c790 tx=0x7feea0007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.941+0000 7fee96ffd700 1 -- 172.21.15.27:0/4155719615 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea0005b30 con 0x7feeb00ca640 2024-01-11T08:11:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.941+0000 7fee96ffd700 1 -- 172.21.15.27:0/4155719615 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea0005cd0 con 0x7feeb00ca640 2024-01-11T08:11:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.941+0000 7fee96ffd700 1 -- 172.21.15.27:0/4155719615 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea0007320 con 0x7feeb00ca640 2024-01-11T08:11:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.941+0000 7feeb53ff700 1 -- 172.21.15.27:0/4155719615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeb00aec10 con 0x7feeb00ca640 2024-01-11T08:11:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.941+0000 7feeb53ff700 1 -- 172.21.15.27:0/4155719615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb00c49c0 con 0x7feeb00ca640 2024-01-11T08:11:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.942+0000 7feeb53ff700 1 -- 172.21.15.27:0/4155719615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeb00c42d0 con 0x7feeb00ca640 2024-01-11T08:11:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.943+0000 7fee96ffd700 1 -- 172.21.15.27:0/4155719615 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7feea0012400 con 0x7feeb00ca640 2024-01-11T08:11:00.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.944+0000 7fee96ffd700 1 --2- 172.21.15.27:0/4155719615 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fee9c066f30 0x7fee9c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:00.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.944+0000 7fee96ffd700 1 -- 172.21.15.27:0/4155719615 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7feea00838f0 con 0x7feeb00ca640 2024-01-11T08:11:00.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.945+0000 7feeae7fc700 1 --2- 172.21.15.27:0/4155719615 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fee9c066f30 0x7fee9c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:00.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.945+0000 7feeae7fc700 1 --2- 172.21.15.27:0/4155719615 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fee9c066f30 0x7fee9c0693e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7feea40093f0 tx=0x7feea4017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:00.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:00.948+0000 7fee96ffd700 1 -- 172.21.15.27:0/4155719615 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feea004b5a0 con 0x7feeb00ca640 2024-01-11T08:11:01.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.150+0000 7feeb53ff700 1 -- 172.21.15.27:0/4155719615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feeb004c3c0 con 0x7feeb00ca640 2024-01-11T08:11:01.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.151+0000 7fee96ffd700 1 -- 172.21.15.27:0/4155719615 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7feea0017020 con 0x7feeb00ca640 2024-01-11T08:11:01.151 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T08:11:01.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.153+0000 7feeb53ff700 1 -- 172.21.15.27:0/4155719615 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fee9c066f30 msgr2=0x7fee9c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:01.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.153+0000 7feeb53ff700 1 --2- 172.21.15.27:0/4155719615 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fee9c066f30 0x7fee9c0693e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7feea40093f0 tx=0x7feea4017000 comp rx=0 tx=0).stop 2024-01-11T08:11:01.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.153+0000 7feeb53ff700 1 -- 172.21.15.27:0/4155719615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb00ca640 msgr2=0x7feeb00c4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:01.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.153+0000 7feeb53ff700 1 --2- 172.21.15.27:0/4155719615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb00ca640 0x7feeb00c4480 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7feea000c790 tx=0x7feea0007e80 comp rx=0 tx=0).stop 2024-01-11T08:11:01.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.153+0000 7feeb53ff700 1 -- 172.21.15.27:0/4155719615 shutdown_connections 2024-01-11T08:11:01.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.153+0000 7feeb53ff700 1 --2- 172.21.15.27:0/4155719615 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fee9c066f30 0x7fee9c0693e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:01.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.153+0000 7feeb53ff700 1 --2- 172.21.15.27:0/4155719615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feeb0159ac0 0x7feeb00c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:01.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.153+0000 7feeb53ff700 1 --2- 172.21.15.27:0/4155719615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb00ca640 0x7feeb00c4480 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:01.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.153+0000 7feeb53ff700 1 --2- 172.21.15.27:0/4155719615 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb01596f0 0x7feeb00ab960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:01.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.153+0000 7feeb53ff700 1 -- 172.21.15.27:0/4155719615 >> 172.21.15.27:0/4155719615 conn(0x7feeb00c9fb0 msgr2=0x7feeb01587c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:01.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.153+0000 7feeb53ff700 1 -- 172.21.15.27:0/4155719615 shutdown_connections 2024-01-11T08:11:01.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.153+0000 7feeb53ff700 1 -- 172.21.15.27:0/4155719615 wait complete. 2024-01-11T08:11:01.173 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T08:11:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.375+0000 7f339e7ff700 1 Processor -- start 2024-01-11T08:11:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.376+0000 7f339e7ff700 1 -- start start 2024-01-11T08:11:01.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.377+0000 7f339e7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33981593b0 0x7f33980b0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:01.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.377+0000 7f339e7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33980b0620 con 0x7f3398068490 2024-01-11T08:11:01.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.377+0000 7f339e7ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3398159bf0 con 0x7f3398159780 2024-01-11T08:11:01.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.377+0000 7f339e7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33980b2b10 con 0x7f33981593b0 2024-01-11T08:11:01.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.377+0000 7f339cd9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3398068490 0x7f33980cb350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39030/0 (socket says 172.21.15.27:39030) 2024-01-11T08:11:01.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.377+0000 7f339cd9c700 1 -- 172.21.15.27:0/2318160543 learned_addr learned my addr 172.21.15.27:0/2318160543 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:01.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.377+0000 7f3397fff700 1 --2- 172.21.15.27:0/2318160543 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33981593b0 0x7f33980b0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:01.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.377+0000 7f3396ffd700 1 -- 172.21.15.27:0/2318160543 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890512867 0 0) 0x7f33980b0620 con 0x7f3398068490 2024-01-11T08:11:01.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.378+0000 7f3396ffd700 1 -- 172.21.15.27:0/2318160543 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3380003570 con 0x7f3398068490 2024-01-11T08:11:01.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.378+0000 7f3396ffd700 1 -- 172.21.15.27:0/2318160543 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394633256 0 0) 0x7f3398159bf0 con 0x7f3398159780 2024-01-11T08:11:01.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.378+0000 7f3397fff700 1 -- 172.21.15.27:0/2318160543 >> v1:172.21.15.77:6789/0 conn(0x7f3398159780 legacy=0x7f33980b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:01.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.378+0000 7f3397fff700 1 -- 172.21.15.27:0/2318160543 >> v1:172.21.15.27:6789/0 conn(0x7f3398068490 legacy=0x7f33980cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:01.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.378+0000 7f3397fff700 1 -- 172.21.15.27:0/2318160543 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33980b2cf0 con 0x7f33981593b0 2024-01-11T08:11:01.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.378+0000 7f3397fff700 1 --2- 172.21.15.27:0/2318160543 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33981593b0 0x7f33980b0ab0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3388009910 tx=0x7f33880091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c8a01fcdee9690f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:01.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.378+0000 7f3396ffd700 1 -- 172.21.15.27:0/2318160543 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338800b680 con 0x7f33981593b0 2024-01-11T08:11:01.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.378+0000 7f3396ffd700 1 -- 172.21.15.27:0/2318160543 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338800b7e0 con 0x7f33981593b0 2024-01-11T08:11:01.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.378+0000 7f3396ffd700 1 -- 172.21.15.27:0/2318160543 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338800bba0 con 0x7f33981593b0 2024-01-11T08:11:01.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.379+0000 7f339e7ff700 1 -- 172.21.15.27:0/2318160543 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33981593b0 msgr2=0x7f33980b0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:01.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.379+0000 7f339e7ff700 1 --2- 172.21.15.27:0/2318160543 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33981593b0 0x7f33980b0ab0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3388009910 tx=0x7f33880091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.379+0000 7f339e7ff700 1 -- 172.21.15.27:0/2318160543 shutdown_connections 2024-01-11T08:11:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.379+0000 7f339e7ff700 1 --2- 172.21.15.27:0/2318160543 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33981593b0 0x7f33980b0ab0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.379+0000 7f339e7ff700 1 -- 172.21.15.27:0/2318160543 >> 172.21.15.27:0/2318160543 conn(0x7f339806ed40 msgr2=0x7f339806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.379+0000 7f339e7ff700 1 -- 172.21.15.27:0/2318160543 shutdown_connections 2024-01-11T08:11:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.379+0000 7f339e7ff700 1 -- 172.21.15.27:0/2318160543 wait complete. 2024-01-11T08:11:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.381+0000 7f339e7ff700 1 Processor -- start 2024-01-11T08:11:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.381+0000 7f339e7ff700 1 -- start start 2024-01-11T08:11:01.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.383+0000 7f339e7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3398068490 0x7f33980b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:01.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.383+0000 7f339e7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33981593b0 0x7f33981427d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:01.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.383+0000 7f339e7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3398159780 0x7f339816a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:01.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.383+0000 7f339e7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33980b5920 con 0x7f3398159780 2024-01-11T08:11:01.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.383+0000 7f339e7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339816aed0 con 0x7f3398068490 2024-01-11T08:11:01.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.383+0000 7f339e7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33980b4d10 con 0x7f33981593b0 2024-01-11T08:11:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.383+0000 7f339cd9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3398159780 0x7f339816a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.383+0000 7f339cd9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3398159780 0x7f339816a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53436/0 (socket says 172.21.15.27:53436) 2024-01-11T08:11:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.383+0000 7f339cd9c700 1 -- 172.21.15.27:0/3611358048 learned_addr learned my addr 172.21.15.27:0/3611358048 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.383+0000 7f3397fff700 1 --2- 172.21.15.27:0/3611358048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3398068490 0x7f33980b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.383+0000 7f33977fe700 1 --2- 172.21.15.27:0/3611358048 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33981593b0 0x7f33981427d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.383+0000 7f339cd9c700 1 -- 172.21.15.27:0/3611358048 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33981593b0 msgr2=0x7f33981427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.383+0000 7f339cd9c700 1 --2- 172.21.15.27:0/3611358048 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33981593b0 0x7f33981427d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.383+0000 7f339cd9c700 1 -- 172.21.15.27:0/3611358048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3398068490 msgr2=0x7f33980b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.383+0000 7f339cd9c700 1 --2- 172.21.15.27:0/3611358048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3398068490 0x7f33980b5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.383+0000 7f339cd9c700 1 -- 172.21.15.27:0/3611358048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3388009420 con 0x7f3398159780 2024-01-11T08:11:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.384+0000 7f339cd9c700 1 --2- 172.21.15.27:0/3611358048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3398159780 0x7f339816a990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3390001460 tx=0x7f3390001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.384+0000 7f337ffff700 1 -- 172.21.15.27:0/3611358048 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339000a870 con 0x7f3398159780 2024-01-11T08:11:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.384+0000 7f337ffff700 1 -- 172.21.15.27:0/3611358048 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f339000a9d0 con 0x7f3398159780 2024-01-11T08:11:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.384+0000 7f339e7ff700 1 -- 172.21.15.27:0/3611358048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33980b4ff0 con 0x7f3398159780 2024-01-11T08:11:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.384+0000 7f337ffff700 1 -- 172.21.15.27:0/3611358048 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339000ad50 con 0x7f3398159780 2024-01-11T08:11:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.384+0000 7f339e7ff700 1 -- 172.21.15.27:0/3611358048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33980b55b0 con 0x7f3398159780 2024-01-11T08:11:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.385+0000 7f339e7ff700 1 -- 172.21.15.27:0/3611358048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f339804c3c0 con 0x7f3398159780 2024-01-11T08:11:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.386+0000 7f337ffff700 1 -- 172.21.15.27:0/3611358048 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3390007300 con 0x7f3398159780 2024-01-11T08:11:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.387+0000 7f337ffff700 1 --2- 172.21.15.27:0/3611358048 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3384066f30 0x7f33840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.387+0000 7f337ffff700 1 -- 172.21.15.27:0/3611358048 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f3390084910 con 0x7f3398159780 2024-01-11T08:11:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.387+0000 7f3397fff700 1 --2- 172.21.15.27:0/3611358048 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3384066f30 0x7f33840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.388+0000 7f3397fff700 1 --2- 172.21.15.27:0/3611358048 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3384066f30 0x7f33840693e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f33880093f0 tx=0x7f3388017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:01.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.391+0000 7f337ffff700 1 -- 172.21.15.27:0/3611358048 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3390051c10 con 0x7f3398159780 2024-01-11T08:11:01.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.595+0000 7f339e7ff700 1 -- 172.21.15.27:0/3611358048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f33980b5a60 con 0x7f3398159780 2024-01-11T08:11:01.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.595+0000 7f337ffff700 1 -- 172.21.15.27:0/3611358048 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3390055230 con 0x7f3398159780 2024-01-11T08:11:01.595 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T08:11:01.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.597+0000 7f339e7ff700 1 -- 172.21.15.27:0/3611358048 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3384066f30 msgr2=0x7f33840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:01.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.598+0000 7f339e7ff700 1 --2- 172.21.15.27:0/3611358048 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3384066f30 0x7f33840693e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f33880093f0 tx=0x7f3388017000 comp rx=0 tx=0).stop 2024-01-11T08:11:01.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.598+0000 7f339e7ff700 1 -- 172.21.15.27:0/3611358048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3398159780 msgr2=0x7f339816a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:01.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.598+0000 7f339e7ff700 1 --2- 172.21.15.27:0/3611358048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3398159780 0x7f339816a990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3390001460 tx=0x7f3390001a40 comp rx=0 tx=0).stop 2024-01-11T08:11:01.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.598+0000 7f339e7ff700 1 -- 172.21.15.27:0/3611358048 shutdown_connections 2024-01-11T08:11:01.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.598+0000 7f339e7ff700 1 --2- 172.21.15.27:0/3611358048 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3384066f30 0x7f33840693e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:01.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.598+0000 7f339e7ff700 1 --2- 172.21.15.27:0/3611358048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3398159780 0x7f339816a990 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:01.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.598+0000 7f339e7ff700 1 --2- 172.21.15.27:0/3611358048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3398068490 0x7f33980b5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:01.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.598+0000 7f339e7ff700 1 --2- 172.21.15.27:0/3611358048 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33981593b0 0x7f33981427d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:01.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.598+0000 7f339e7ff700 1 -- 172.21.15.27:0/3611358048 >> 172.21.15.27:0/3611358048 conn(0x7f339806ed40 msgr2=0x7f33980ccef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:01.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.598+0000 7f339e7ff700 1 -- 172.21.15.27:0/3611358048 shutdown_connections 2024-01-11T08:11:01.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.598+0000 7f339e7ff700 1 -- 172.21.15.27:0/3611358048 wait complete. 2024-01-11T08:11:01.616 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:11:01.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.817+0000 7f5faa5ff700 1 Processor -- start 2024-01-11T08:11:01.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.817+0000 7f5faa5ff700 1 -- start start 2024-01-11T08:11:01.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.818+0000 7f5faa5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5fa40ca3e0 0x7f5fa40cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:01.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.818+0000 7f5faa5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5fa4159e80 0x7f5fa415dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:01.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.818+0000 7f5faa5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa415d640 con 0x7f5fa40ca7b0 2024-01-11T08:11:01.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.818+0000 7f5faa5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa415ae70 con 0x7f5fa4159e80 2024-01-11T08:11:01.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.818+0000 7f5faa5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa40cc4f0 con 0x7f5fa40ca3e0 2024-01-11T08:11:01.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.818+0000 7f5fa8b9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5fa4159e80 0x7f5fa415dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:01.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.819+0000 7f5fa8b9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5fa4159e80 0x7f5fa415dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53452/0 (socket says 172.21.15.27:53452) 2024-01-11T08:11:01.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.819+0000 7f5fa8b9c700 1 -- 172.21.15.27:0/1603453783 learned_addr learned my addr 172.21.15.27:0/1603453783 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:01.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.819+0000 7f5fa3fff700 1 --2- 172.21.15.27:0/1603453783 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5fa40ca3e0 0x7f5fa40cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:01.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.819+0000 7f5fa8b9c700 1 -- 172.21.15.27:0/1603453783 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5fa40ca3e0 msgr2=0x7f5fa40cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:01.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.819+0000 7f5fa8b9c700 1 --2- 172.21.15.27:0/1603453783 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5fa40ca3e0 0x7f5fa40cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:01.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.819+0000 7f5fa8b9c700 1 -- 172.21.15.27:0/1603453783 >> v1:172.21.15.77:6789/0 conn(0x7f5fa40ca7b0 legacy=0x7f5fa40c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:01.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.819+0000 7f5fa8b9c700 1 -- 172.21.15.27:0/1603453783 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa415a5d0 con 0x7f5fa4159e80 2024-01-11T08:11:01.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.819+0000 7f5fa3fff700 1 --2- 172.21.15.27:0/1603453783 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5fa40ca3e0 0x7f5fa40cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:01.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.819+0000 7f5fa8b9c700 1 --2- 172.21.15.27:0/1603453783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa4159e80 0x7f5fa415dad0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009f20 tx=0x7f5f9c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3697db2ffd0fa72c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:01.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.820+0000 7f5fa2ffd700 1 -- 172.21.15.27:0/1603453783 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c00e070 con 0x7f5fa4159e80 2024-01-11T08:11:01.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.820+0000 7f5fa2ffd700 1 -- 172.21.15.27:0/1603453783 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9c00ba80 con 0x7f5fa4159e80 2024-01-11T08:11:01.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.820+0000 7f5fa2ffd700 1 -- 172.21.15.27:0/1603453783 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c0092e0 con 0x7f5fa4159e80 2024-01-11T08:11:01.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.820+0000 7f5faa5ff700 1 -- 172.21.15.27:0/1603453783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa4159e80 msgr2=0x7f5fa415dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:01.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.820+0000 7f5faa5ff700 1 --2- 172.21.15.27:0/1603453783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa4159e80 0x7f5fa415dad0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009f20 tx=0x7f5f9c009840 comp rx=0 tx=0).stop 2024-01-11T08:11:01.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.820+0000 7f5faa5ff700 1 -- 172.21.15.27:0/1603453783 shutdown_connections 2024-01-11T08:11:01.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.820+0000 7f5faa5ff700 1 --2- 172.21.15.27:0/1603453783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa4159e80 0x7f5fa415dad0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:01.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.820+0000 7f5faa5ff700 1 --2- 172.21.15.27:0/1603453783 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5fa40ca3e0 0x7f5fa40cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:01.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.820+0000 7f5faa5ff700 1 -- 172.21.15.27:0/1603453783 >> 172.21.15.27:0/1603453783 conn(0x7f5fa40ce3e0 msgr2=0x7f5fa40ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:01.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.820+0000 7f5faa5ff700 1 -- 172.21.15.27:0/1603453783 shutdown_connections 2024-01-11T08:11:01.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.820+0000 7f5faa5ff700 1 -- 172.21.15.27:0/1603453783 wait complete. 2024-01-11T08:11:01.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.822+0000 7f5faa5ff700 1 Processor -- start 2024-01-11T08:11:01.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.822+0000 7f5faa5ff700 1 -- start start 2024-01-11T08:11:01.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.823+0000 7f5faa5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa40ca3e0 0x7f5fa4151330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:01.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.823+0000 7f5faa5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5fa40ca7b0 0x7f5fa416a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:01.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.823+0000 7f5faa5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fa4159e80 0x7f5fa4145390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:01.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.823+0000 7f5faa5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa4151870 con 0x7f5fa40ca3e0 2024-01-11T08:11:01.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.823+0000 7f5faa5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa4145f30 con 0x7f5fa4159e80 2024-01-11T08:11:01.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.823+0000 7f5faa5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa4146210 con 0x7f5fa40ca7b0 2024-01-11T08:11:01.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.823+0000 7f5fa3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa40ca3e0 0x7f5fa4151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:01.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.823+0000 7f5fa3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa40ca3e0 0x7f5fa4151330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53456/0 (socket says 172.21.15.27:53456) 2024-01-11T08:11:01.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.823+0000 7f5fa3fff700 1 -- 172.21.15.27:0/1610796097 learned_addr learned my addr 172.21.15.27:0/1610796097 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:01.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.823+0000 7f5fa37fe700 1 --2- 172.21.15.27:0/1610796097 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5fa40ca7b0 0x7f5fa416a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:01.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.823+0000 7f5fa8b9c700 1 --2- 172.21.15.27:0/1610796097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fa4159e80 0x7f5fa4145390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:01.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.824+0000 7f5fa3fff700 1 -- 172.21.15.27:0/1610796097 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5fa40ca7b0 msgr2=0x7f5fa416a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:01.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.824+0000 7f5fa3fff700 1 --2- 172.21.15.27:0/1610796097 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5fa40ca7b0 0x7f5fa416a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:01.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.824+0000 7f5fa3fff700 1 -- 172.21.15.27:0/1610796097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fa4159e80 msgr2=0x7f5fa4145390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:01.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.824+0000 7f5fa3fff700 1 --2- 172.21.15.27:0/1610796097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fa4159e80 0x7f5fa4145390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:01.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.824+0000 7f5fa3fff700 1 -- 172.21.15.27:0/1610796097 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c009ae0 con 0x7f5fa40ca3e0 2024-01-11T08:11:01.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.824+0000 7f5fa8b9c700 1 --2- 172.21.15.27:0/1610796097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fa4159e80 0x7f5fa4145390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:01.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.824+0000 7f5fa3fff700 1 --2- 172.21.15.27:0/1610796097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa40ca3e0 0x7f5fa4151330 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5f9400e8d0 tx=0x7f5f94008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:01.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.824+0000 7f5f8bfff700 1 -- 172.21.15.27:0/1610796097 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f94013410 con 0x7f5fa40ca3e0 2024-01-11T08:11:01.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.824+0000 7f5f8bfff700 1 -- 172.21.15.27:0/1610796097 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f94013570 con 0x7f5fa40ca3e0 2024-01-11T08:11:01.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.824+0000 7f5faa5ff700 1 -- 172.21.15.27:0/1610796097 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa41464f0 con 0x7f5fa40ca3e0 2024-01-11T08:11:01.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.824+0000 7f5f8bfff700 1 -- 172.21.15.27:0/1610796097 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f94013910 con 0x7f5fa40ca3e0 2024-01-11T08:11:01.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.824+0000 7f5faa5ff700 1 -- 172.21.15.27:0/1610796097 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa4146ab0 con 0x7f5fa40ca3e0 2024-01-11T08:11:01.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.825+0000 7f5faa5ff700 1 -- 172.21.15.27:0/1610796097 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa4146e20 con 0x7f5fa40ca3e0 2024-01-11T08:11:01.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.826+0000 7f5f8bfff700 1 -- 172.21.15.27:0/1610796097 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5f94013d00 con 0x7f5fa40ca3e0 2024-01-11T08:11:01.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.827+0000 7f5f8bfff700 1 --2- 172.21.15.27:0/1610796097 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5f90066f30 0x7f5f900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:01.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.827+0000 7f5f8bfff700 1 -- 172.21.15.27:0/1610796097 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f5f94084270 con 0x7f5fa40ca3e0 2024-01-11T08:11:01.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.827+0000 7f5fa37fe700 1 --2- 172.21.15.27:0/1610796097 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5f90066f30 0x7f5f900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:01.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.828+0000 7f5fa37fe700 1 --2- 172.21.15.27:0/1610796097 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5f90066f30 0x7f5f900693e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5f98003a80 tx=0x7f5f98007ec0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:01.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:01.831+0000 7f5f8bfff700 1 -- 172.21.15.27:0/1610796097 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5f940514e0 con 0x7f5fa40ca3e0 2024-01-11T08:11:02.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.037+0000 7f5faa5ff700 1 -- 172.21.15.27:0/1610796097 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fa40718d0 con 0x7f5fa40ca3e0 2024-01-11T08:11:02.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.038+0000 7f5f8bfff700 1 -- 172.21.15.27:0/1610796097 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15302 (secure 0 0 0) 0x7f5f94054b00 con 0x7f5fa40ca3e0 2024-01-11T08:11:02.038 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:02.038 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:31.464323+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:10:27.993251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-11T08:10:29.153959+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:11:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.040+0000 7f5faa5ff700 1 -- 172.21.15.27:0/1610796097 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5f90066f30 msgr2=0x7f5f900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.040+0000 7f5faa5ff700 1 --2- 172.21.15.27:0/1610796097 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5f90066f30 0x7f5f900693e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5f98003a80 tx=0x7f5f98007ec0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.040+0000 7f5faa5ff700 1 -- 172.21.15.27:0/1610796097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa40ca3e0 msgr2=0x7f5fa4151330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.040+0000 7f5faa5ff700 1 --2- 172.21.15.27:0/1610796097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa40ca3e0 0x7f5fa4151330 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5f9400e8d0 tx=0x7f5f94008040 comp rx=0 tx=0).stop 2024-01-11T08:11:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.041+0000 7f5faa5ff700 1 -- 172.21.15.27:0/1610796097 shutdown_connections 2024-01-11T08:11:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.041+0000 7f5faa5ff700 1 --2- 172.21.15.27:0/1610796097 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5f90066f30 0x7f5f900693e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.041+0000 7f5faa5ff700 1 --2- 172.21.15.27:0/1610796097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fa4159e80 0x7f5fa4145390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.041+0000 7f5faa5ff700 1 --2- 172.21.15.27:0/1610796097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fa40ca3e0 0x7f5fa4151330 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.041+0000 7f5faa5ff700 1 --2- 172.21.15.27:0/1610796097 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5fa40ca7b0 0x7f5fa416a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.041+0000 7f5faa5ff700 1 -- 172.21.15.27:0/1610796097 >> 172.21.15.27:0/1610796097 conn(0x7f5fa40ce3e0 msgr2=0x7f5fa40ce7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.041+0000 7f5faa5ff700 1 -- 172.21.15.27:0/1610796097 shutdown_connections 2024-01-11T08:11:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.041+0000 7f5faa5ff700 1 -- 172.21.15.27:0/1610796097 wait complete. 2024-01-11T08:11:02.058 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T08:10:27.993251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T08:10:29.153959+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T08:11:02.059 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:11:02.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.260+0000 7ff9a73ff700 1 Processor -- start 2024-01-11T08:11:02.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.260+0000 7ff9a73ff700 1 -- start start 2024-01-11T08:11:02.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.262+0000 7ff9a73ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9a0159ac0 0x7ff9a0068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:02.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.262+0000 7ff9a73ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9a00ca640 0x7ff9a0068b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:02.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.262+0000 7ff9a73ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a00c80a0 con 0x7ff9a01596f0 2024-01-11T08:11:02.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.262+0000 7ff9a73ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9a00cbac0 con 0x7ff9a00ca640 2024-01-11T08:11:02.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.262+0000 7ff9a73ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a00a9fb0 con 0x7ff9a0159ac0 2024-01-11T08:11:02.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.262+0000 7ff9a599c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9a00ca640 0x7ff9a0068b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:02.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.262+0000 7ff9a599c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9a00ca640 0x7ff9a0068b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53466/0 (socket says 172.21.15.27:53466) 2024-01-11T08:11:02.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.262+0000 7ff9a599c700 1 -- 172.21.15.27:0/2939504181 learned_addr learned my addr 172.21.15.27:0/2939504181 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:02.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.262+0000 7ff9a499a700 1 --2- 172.21.15.27:0/2939504181 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9a0159ac0 0x7ff9a0068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:02.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.263+0000 7ff9a599c700 1 -- 172.21.15.27:0/2939504181 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9a0159ac0 msgr2=0x7ff9a0068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:02.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.263+0000 7ff9a599c700 1 --2- 172.21.15.27:0/2939504181 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9a0159ac0 0x7ff9a0068620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.263+0000 7ff9a599c700 1 -- 172.21.15.27:0/2939504181 >> v1:172.21.15.77:6789/0 conn(0x7ff9a01596f0 legacy=0x7ff9a00a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:02.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.263+0000 7ff9a599c700 1 -- 172.21.15.27:0/2939504181 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a00c7490 con 0x7ff9a00ca640 2024-01-11T08:11:02.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.263+0000 7ff997fff700 1 -- 172.21.15.27:0/2939504181 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274716881 0 0) 0x7ff9a00c80a0 con 0x7ff9a01596f0 2024-01-11T08:11:02.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.263+0000 7ff9a599c700 1 --2- 172.21.15.27:0/2939504181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00ca640 0x7ff9a0068b60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff998009f20 tx=0x7ff998009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b8be20e0b1cdff4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:02.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.263+0000 7ff997fff700 1 -- 172.21.15.27:0/2939504181 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99800e070 con 0x7ff9a00ca640 2024-01-11T08:11:02.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.263+0000 7ff997fff700 1 -- 172.21.15.27:0/2939504181 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99800b9d0 con 0x7ff9a00ca640 2024-01-11T08:11:02.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.263+0000 7ff997fff700 1 -- 172.21.15.27:0/2939504181 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9980092e0 con 0x7ff9a00ca640 2024-01-11T08:11:02.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.264+0000 7ff9a73ff700 1 -- 172.21.15.27:0/2939504181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00ca640 msgr2=0x7ff9a0068b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:02.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.264+0000 7ff9a73ff700 1 --2- 172.21.15.27:0/2939504181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00ca640 0x7ff9a0068b60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff998009f20 tx=0x7ff998009840 comp rx=0 tx=0).stop 2024-01-11T08:11:02.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.264+0000 7ff9a73ff700 1 -- 172.21.15.27:0/2939504181 shutdown_connections 2024-01-11T08:11:02.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.264+0000 7ff9a73ff700 1 --2- 172.21.15.27:0/2939504181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00ca640 0x7ff9a0068b60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.264+0000 7ff9a73ff700 1 --2- 172.21.15.27:0/2939504181 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9a0159ac0 0x7ff9a0068620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.264+0000 7ff9a73ff700 1 -- 172.21.15.27:0/2939504181 >> 172.21.15.27:0/2939504181 conn(0x7ff9a00c9fb0 msgr2=0x7ff9a006ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:02.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.264+0000 7ff9a73ff700 1 -- 172.21.15.27:0/2939504181 shutdown_connections 2024-01-11T08:11:02.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.264+0000 7ff9a73ff700 1 -- 172.21.15.27:0/2939504181 wait complete. 2024-01-11T08:11:02.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.265+0000 7ff9a73ff700 1 Processor -- start 2024-01-11T08:11:02.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.265+0000 7ff9a73ff700 1 -- start start 2024-01-11T08:11:02.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.266+0000 7ff9a73ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00ca640 0x7ff9a0169380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:02.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.266+0000 7ff9a73ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a01596f0 0x7ff9a016d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:02.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.266+0000 7ff9a73ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9a0159ac0 0x7ff9a00c3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:02.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.266+0000 7ff9a73ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a016a300 con 0x7ff9a00ca640 2024-01-11T08:11:02.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.266+0000 7ff9a73ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a016d8f0 con 0x7ff9a01596f0 2024-01-11T08:11:02.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.266+0000 7ff9a73ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a0169a60 con 0x7ff9a0159ac0 2024-01-11T08:11:02.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.267+0000 7ff9a519b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00ca640 0x7ff9a0169380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:02.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.267+0000 7ff9a519b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00ca640 0x7ff9a0169380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53468/0 (socket says 172.21.15.27:53468) 2024-01-11T08:11:02.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.267+0000 7ff9a519b700 1 -- 172.21.15.27:0/3625955518 learned_addr learned my addr 172.21.15.27:0/3625955518 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:02.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.267+0000 7ff9a599c700 1 --2- 172.21.15.27:0/3625955518 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9a0159ac0 0x7ff9a00c3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:02.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.267+0000 7ff9a499a700 1 --2- 172.21.15.27:0/3625955518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a01596f0 0x7ff9a016d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:02.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.267+0000 7ff9a519b700 1 -- 172.21.15.27:0/3625955518 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9a0159ac0 msgr2=0x7ff9a00c3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:02.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.267+0000 7ff9a519b700 1 --2- 172.21.15.27:0/3625955518 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9a0159ac0 0x7ff9a00c3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.267+0000 7ff9a519b700 1 -- 172.21.15.27:0/3625955518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a01596f0 msgr2=0x7ff9a016d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:02.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.267+0000 7ff9a519b700 1 --2- 172.21.15.27:0/3625955518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a01596f0 0x7ff9a016d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.267+0000 7ff9a519b700 1 -- 172.21.15.27:0/3625955518 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff998009ae0 con 0x7ff9a00ca640 2024-01-11T08:11:02.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.267+0000 7ff9a519b700 1 --2- 172.21.15.27:0/3625955518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00ca640 0x7ff9a0169380 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff99c00c7e0 tx=0x7ff99c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:02.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.268+0000 7ff9957fa700 1 -- 172.21.15.27:0/3625955518 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c0105c0 con 0x7ff9a00ca640 2024-01-11T08:11:02.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.268+0000 7ff9957fa700 1 -- 172.21.15.27:0/3625955518 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99c010740 con 0x7ff9a00ca640 2024-01-11T08:11:02.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.268+0000 7ff9957fa700 1 -- 172.21.15.27:0/3625955518 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c010b00 con 0x7ff9a00ca640 2024-01-11T08:11:02.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.268+0000 7ff9a73ff700 1 -- 172.21.15.27:0/3625955518 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a0169d40 con 0x7ff9a00ca640 2024-01-11T08:11:02.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.268+0000 7ff9a73ff700 1 -- 172.21.15.27:0/3625955518 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a00af950 con 0x7ff9a00ca640 2024-01-11T08:11:02.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.269+0000 7ff9a73ff700 1 -- 172.21.15.27:0/3625955518 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a004c3c0 con 0x7ff9a00ca640 2024-01-11T08:11:02.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.270+0000 7ff9957fa700 1 -- 172.21.15.27:0/3625955518 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff99c01b3b0 con 0x7ff9a00ca640 2024-01-11T08:11:02.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.271+0000 7ff9957fa700 1 --2- 172.21.15.27:0/3625955518 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff984066f30 0x7ff9840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:02.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.271+0000 7ff9957fa700 1 -- 172.21.15.27:0/3625955518 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7ff99c083b20 con 0x7ff9a00ca640 2024-01-11T08:11:02.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.271+0000 7ff9a499a700 1 --2- 172.21.15.27:0/3625955518 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff984066f30 0x7ff9840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:02.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.272+0000 7ff9a499a700 1 --2- 172.21.15.27:0/3625955518 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff984066f30 0x7ff9840693e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff9a00af670 tx=0x7ff9900091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:02.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.275+0000 7ff9957fa700 1 -- 172.21.15.27:0/3625955518 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff99c050d90 con 0x7ff9a00ca640 2024-01-11T08:11:02.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.478+0000 7ff9a73ff700 1 -- 172.21.15.27:0/3625955518 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff9a00c7770 con 0x7ff9a00ca640 2024-01-11T08:11:02.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.479+0000 7ff9957fa700 1 -- 172.21.15.27:0/3625955518 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff99c017070 con 0x7ff9a00ca640 2024-01-11T08:11:02.479 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:11:02.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.481+0000 7ff9a73ff700 1 -- 172.21.15.27:0/3625955518 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff984066f30 msgr2=0x7ff9840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:02.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.481+0000 7ff9a73ff700 1 --2- 172.21.15.27:0/3625955518 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff984066f30 0x7ff9840693e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff9a00af670 tx=0x7ff9900091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.481+0000 7ff9a73ff700 1 -- 172.21.15.27:0/3625955518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00ca640 msgr2=0x7ff9a0169380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.481+0000 7ff9a73ff700 1 --2- 172.21.15.27:0/3625955518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00ca640 0x7ff9a0169380 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff99c00c7e0 tx=0x7ff99c007df0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.481+0000 7ff9a73ff700 1 -- 172.21.15.27:0/3625955518 shutdown_connections 2024-01-11T08:11:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.481+0000 7ff9a73ff700 1 --2- 172.21.15.27:0/3625955518 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff984066f30 0x7ff9840693e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.481+0000 7ff9a73ff700 1 --2- 172.21.15.27:0/3625955518 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9a0159ac0 0x7ff9a00c3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.481+0000 7ff9a73ff700 1 --2- 172.21.15.27:0/3625955518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00ca640 0x7ff9a0169380 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.481+0000 7ff9a73ff700 1 --2- 172.21.15.27:0/3625955518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a01596f0 0x7ff9a016d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.481+0000 7ff9a73ff700 1 -- 172.21.15.27:0/3625955518 >> 172.21.15.27:0/3625955518 conn(0x7ff9a00c9fb0 msgr2=0x7ff9a00c8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.482+0000 7ff9a73ff700 1 -- 172.21.15.27:0/3625955518 shutdown_connections 2024-01-11T08:11:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.482+0000 7ff9a73ff700 1 -- 172.21.15.27:0/3625955518 wait complete. 2024-01-11T08:11:02.501 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T08:11:02.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.703+0000 7fb0de9ff700 1 Processor -- start 2024-01-11T08:11:02.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.703+0000 7fb0de9ff700 1 -- start start 2024-01-11T08:11:02.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.704+0000 7fb0de9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0d80cd5f0 0x7fb0d81591e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:02.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.704+0000 7fb0de9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d815aef0 con 0x7fb0d8159720 2024-01-11T08:11:02.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.704+0000 7fb0de9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d80cee00 con 0x7fb0d80cd220 2024-01-11T08:11:02.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.704+0000 7fb0de9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d8159b60 con 0x7fb0d80cd5f0 2024-01-11T08:11:02.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.705+0000 7fb0dcf9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb0d8159720 0x7fb0d80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39038/0 (socket says 172.21.15.27:39038) 2024-01-11T08:11:02.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.705+0000 7fb0dcf9c700 1 -- 172.21.15.27:0/2498492400 learned_addr learned my addr 172.21.15.27:0/2498492400 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:02.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.705+0000 7fb0d77fe700 1 --2- 172.21.15.27:0/2498492400 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0d80cd5f0 0x7fb0d81591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:02.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.705+0000 7fb0d6ffd700 1 -- 172.21.15.27:0/2498492400 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052670407 0 0) 0x7fb0d815aef0 con 0x7fb0d8159720 2024-01-11T08:11:02.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.705+0000 7fb0d6ffd700 1 -- 172.21.15.27:0/2498492400 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0c0003570 con 0x7fb0d8159720 2024-01-11T08:11:02.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.705+0000 7fb0d77fe700 1 -- 172.21.15.27:0/2498492400 >> v1:172.21.15.77:6789/0 conn(0x7fb0d80cd220 legacy=0x7fb0d815d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:02.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.705+0000 7fb0d6ffd700 1 -- 172.21.15.27:0/2498492400 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953087657 0 0) 0x7fb0d80cee00 con 0x7fb0d80cd220 2024-01-11T08:11:02.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.705+0000 7fb0d77fe700 1 -- 172.21.15.27:0/2498492400 >> v1:172.21.15.27:6789/0 conn(0x7fb0d8159720 legacy=0x7fb0d80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:02.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.705+0000 7fb0d77fe700 1 -- 172.21.15.27:0/2498492400 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d815dd70 con 0x7fb0d80cd5f0 2024-01-11T08:11:02.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.706+0000 7fb0d77fe700 1 --2- 172.21.15.27:0/2498492400 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0d80cd5f0 0x7fb0d81591e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009910 tx=0x7fb0cc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a99d15bf698c0307 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:02.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.706+0000 7fb0d6ffd700 1 -- 172.21.15.27:0/2498492400 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc00b680 con 0x7fb0d80cd5f0 2024-01-11T08:11:02.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.706+0000 7fb0d6ffd700 1 -- 172.21.15.27:0/2498492400 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0cc00b7e0 con 0x7fb0d80cd5f0 2024-01-11T08:11:02.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.706+0000 7fb0d6ffd700 1 -- 172.21.15.27:0/2498492400 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc00bba0 con 0x7fb0d80cd5f0 2024-01-11T08:11:02.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.706+0000 7fb0de9ff700 1 -- 172.21.15.27:0/2498492400 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0d80cd5f0 msgr2=0x7fb0d81591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:02.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.706+0000 7fb0de9ff700 1 --2- 172.21.15.27:0/2498492400 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0d80cd5f0 0x7fb0d81591e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009910 tx=0x7fb0cc0091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.706+0000 7fb0de9ff700 1 -- 172.21.15.27:0/2498492400 shutdown_connections 2024-01-11T08:11:02.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.706+0000 7fb0de9ff700 1 --2- 172.21.15.27:0/2498492400 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0d80cd5f0 0x7fb0d81591e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.706+0000 7fb0de9ff700 1 -- 172.21.15.27:0/2498492400 >> 172.21.15.27:0/2498492400 conn(0x7fb0d806ed40 msgr2=0x7fb0d806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:02.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.707+0000 7fb0de9ff700 1 -- 172.21.15.27:0/2498492400 shutdown_connections 2024-01-11T08:11:02.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.707+0000 7fb0de9ff700 1 -- 172.21.15.27:0/2498492400 wait complete. 2024-01-11T08:11:02.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.708+0000 7fb0de9ff700 1 Processor -- start 2024-01-11T08:11:02.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.708+0000 7fb0de9ff700 1 -- start start 2024-01-11T08:11:02.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.709+0000 7fb0de9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0d80cd220 0x7fb0d814f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:02.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.709+0000 7fb0de9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0d80cd5f0 0x7fb0d80b3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:02.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.709+0000 7fb0de9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0d8159720 0x7fb0d80b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:02.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.709+0000 7fb0de9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d80b40b0 con 0x7fb0d8159720 2024-01-11T08:11:02.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.709+0000 7fb0de9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d80b0c80 con 0x7fb0d80cd220 2024-01-11T08:11:02.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.709+0000 7fb0de9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d80b0f20 con 0x7fb0d80cd5f0 2024-01-11T08:11:02.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.709+0000 7fb0dcf9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0d8159720 0x7fb0d80b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:02.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.710+0000 7fb0dcf9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0d8159720 0x7fb0d80b3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53480/0 (socket says 172.21.15.27:53480) 2024-01-11T08:11:02.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.710+0000 7fb0d7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0d80cd220 0x7fb0d814f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:02.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.710+0000 7fb0dcf9c700 1 -- 172.21.15.27:0/1796537514 learned_addr learned my addr 172.21.15.27:0/1796537514 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:02.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.710+0000 7fb0d77fe700 1 --2- 172.21.15.27:0/1796537514 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0d80cd5f0 0x7fb0d80b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:02.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.710+0000 7fb0dcf9c700 1 -- 172.21.15.27:0/1796537514 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0d80cd5f0 msgr2=0x7fb0d80b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:02.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.710+0000 7fb0dcf9c700 1 --2- 172.21.15.27:0/1796537514 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0d80cd5f0 0x7fb0d80b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.710+0000 7fb0dcf9c700 1 -- 172.21.15.27:0/1796537514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0d80cd220 msgr2=0x7fb0d814f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:02.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.710+0000 7fb0dcf9c700 1 --2- 172.21.15.27:0/1796537514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0d80cd220 0x7fb0d814f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.710+0000 7fb0dcf9c700 1 -- 172.21.15.27:0/1796537514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0cc009420 con 0x7fb0d8159720 2024-01-11T08:11:02.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.710+0000 7fb0d77fe700 1 --2- 172.21.15.27:0/1796537514 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0d80cd5f0 0x7fb0d80b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:02.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.710+0000 7fb0dcf9c700 1 --2- 172.21.15.27:0/1796537514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0d8159720 0x7fb0d80b3b70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0001460 tx=0x7fb0d0001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:02.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.710+0000 7fb0bffff700 1 -- 172.21.15.27:0/1796537514 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d000a870 con 0x7fb0d8159720 2024-01-11T08:11:02.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.711+0000 7fb0bffff700 1 -- 172.21.15.27:0/1796537514 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d000a9d0 con 0x7fb0d8159720 2024-01-11T08:11:02.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.711+0000 7fb0de9ff700 1 -- 172.21.15.27:0/1796537514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d80b1200 con 0x7fb0d8159720 2024-01-11T08:11:02.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.711+0000 7fb0bffff700 1 -- 172.21.15.27:0/1796537514 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d000ad50 con 0x7fb0d8159720 2024-01-11T08:11:02.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.711+0000 7fb0de9ff700 1 -- 172.21.15.27:0/1796537514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d80b17c0 con 0x7fb0d8159720 2024-01-11T08:11:02.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.712+0000 7fb0de9ff700 1 -- 172.21.15.27:0/1796537514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d815dab0 con 0x7fb0d8159720 2024-01-11T08:11:02.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.713+0000 7fb0bffff700 1 -- 172.21.15.27:0/1796537514 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb0d0007300 con 0x7fb0d8159720 2024-01-11T08:11:02.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.713+0000 7fb0bffff700 1 --2- 172.21.15.27:0/1796537514 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb0c4066f30 0x7fb0c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:02.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.714+0000 7fb0bffff700 1 -- 172.21.15.27:0/1796537514 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fb0d0084830 con 0x7fb0d8159720 2024-01-11T08:11:02.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.714+0000 7fb0d7fff700 1 --2- 172.21.15.27:0/1796537514 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb0c4066f30 0x7fb0c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:02.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.715+0000 7fb0d7fff700 1 --2- 172.21.15.27:0/1796537514 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb0c4066f30 0x7fb0c40693e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8007fd0 tx=0x7fb0c8007d30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:02.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.718+0000 7fb0bffff700 1 -- 172.21.15.27:0/1796537514 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb0d0051a80 con 0x7fb0d8159720 2024-01-11T08:11:02.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.918+0000 7fb0de9ff700 1 -- 172.21.15.27:0/1796537514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb0d804c3c0 con 0x7fb0d8159720 2024-01-11T08:11:02.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.919+0000 7fb0bffff700 1 -- 172.21.15.27:0/1796537514 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb0d00550a0 con 0x7fb0d8159720 2024-01-11T08:11:02.919 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T08:11:02.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.921+0000 7fb0de9ff700 1 -- 172.21.15.27:0/1796537514 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb0c4066f30 msgr2=0x7fb0c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:02.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.921+0000 7fb0de9ff700 1 --2- 172.21.15.27:0/1796537514 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb0c4066f30 0x7fb0c40693e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8007fd0 tx=0x7fb0c8007d30 comp rx=0 tx=0).stop 2024-01-11T08:11:02.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.921+0000 7fb0de9ff700 1 -- 172.21.15.27:0/1796537514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0d8159720 msgr2=0x7fb0d80b3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:02.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.921+0000 7fb0de9ff700 1 --2- 172.21.15.27:0/1796537514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0d8159720 0x7fb0d80b3b70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0001460 tx=0x7fb0d0001a40 comp rx=0 tx=0).stop 2024-01-11T08:11:02.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.921+0000 7fb0de9ff700 1 -- 172.21.15.27:0/1796537514 shutdown_connections 2024-01-11T08:11:02.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.921+0000 7fb0de9ff700 1 --2- 172.21.15.27:0/1796537514 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb0c4066f30 0x7fb0c40693e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.921+0000 7fb0de9ff700 1 --2- 172.21.15.27:0/1796537514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0d8159720 0x7fb0d80b3b70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.921+0000 7fb0de9ff700 1 --2- 172.21.15.27:0/1796537514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0d80cd220 0x7fb0d814f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.921+0000 7fb0de9ff700 1 --2- 172.21.15.27:0/1796537514 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0d80cd5f0 0x7fb0d80b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:02.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.921+0000 7fb0de9ff700 1 -- 172.21.15.27:0/1796537514 >> 172.21.15.27:0/1796537514 conn(0x7fb0d806ed40 msgr2=0x7fb0d8152850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:02.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.921+0000 7fb0de9ff700 1 -- 172.21.15.27:0/1796537514 shutdown_connections 2024-01-11T08:11:02.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:02.921+0000 7fb0de9ff700 1 -- 172.21.15.27:0/1796537514 wait complete. 2024-01-11T08:11:02.939 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T08:11:03.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.142+0000 7f0e265ff700 1 Processor -- start 2024-01-11T08:11:03.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.142+0000 7f0e265ff700 1 -- start start 2024-01-11T08:11:03.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.143+0000 7f0e265ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0e20143190 0x7f0e200cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:03.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.143+0000 7f0e265ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0e200cb4b0 0x7f0e200cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:03.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.143+0000 7f0e265ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2015d320 con 0x7f0e20142dc0 2024-01-11T08:11:03.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.143+0000 7f0e265ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e2015df30 con 0x7f0e20143190 2024-01-11T08:11:03.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.143+0000 7f0e265ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e2015d590 con 0x7f0e200cb4b0 2024-01-11T08:11:03.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.143+0000 7f0e1f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0e20143190 0x7f0e200cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:03.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.144+0000 7f0e1f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0e20143190 0x7f0e200cc020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53492/0 (socket says 172.21.15.27:53492) 2024-01-11T08:11:03.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.144+0000 7f0e1f7fe700 1 -- 172.21.15.27:0/2929759538 learned_addr learned my addr 172.21.15.27:0/2929759538 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:03.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.144+0000 7f0e1ffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0e20142dc0 0x7f0e200cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39044/0 (socket says 172.21.15.27:39044) 2024-01-11T08:11:03.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.144+0000 7f0e24b9c700 1 --2- 172.21.15.27:0/2929759538 >> v2:172.21.15.77:3300/0 conn(0x7f0e200cb4b0 0x7f0e200cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:03.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.144+0000 7f0e1f7fe700 1 -- 172.21.15.27:0/2929759538 >> v2:172.21.15.77:3300/0 conn(0x7f0e200cb4b0 msgr2=0x7f0e200cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:03.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.144+0000 7f0e1f7fe700 1 --2- 172.21.15.27:0/2929759538 >> v2:172.21.15.77:3300/0 conn(0x7f0e200cb4b0 0x7f0e200cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.144+0000 7f0e1f7fe700 1 -- 172.21.15.27:0/2929759538 >> v1:172.21.15.27:6789/0 conn(0x7f0e20142dc0 legacy=0x7f0e200cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:03.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.144+0000 7f0e1f7fe700 1 -- 172.21.15.27:0/2929759538 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2015d870 con 0x7f0e20143190 2024-01-11T08:11:03.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.144+0000 7f0e1effd700 1 -- 172.21.15.27:0/2929759538 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841402897 0 0) 0x7f0e2015d320 con 0x7f0e20142dc0 2024-01-11T08:11:03.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.144+0000 7f0e1f7fe700 1 --2- 172.21.15.27:0/2929759538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e20143190 0x7f0e200cc020 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0e201577b0 tx=0x7f0e14009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d3064e413dcb75 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:03.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.145+0000 7f0e1effd700 1 -- 172.21.15.27:0/2929759538 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1400b860 con 0x7f0e20143190 2024-01-11T08:11:03.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.145+0000 7f0e1effd700 1 -- 172.21.15.27:0/2929759538 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1400b9c0 con 0x7f0e20143190 2024-01-11T08:11:03.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.145+0000 7f0e1effd700 1 -- 172.21.15.27:0/2929759538 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1400bd60 con 0x7f0e20143190 2024-01-11T08:11:03.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.145+0000 7f0e265ff700 1 -- 172.21.15.27:0/2929759538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e20143190 msgr2=0x7f0e200cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:03.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.145+0000 7f0e265ff700 1 --2- 172.21.15.27:0/2929759538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e20143190 0x7f0e200cc020 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0e201577b0 tx=0x7f0e14009250 comp rx=0 tx=0).stop 2024-01-11T08:11:03.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.145+0000 7f0e265ff700 1 -- 172.21.15.27:0/2929759538 shutdown_connections 2024-01-11T08:11:03.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.145+0000 7f0e265ff700 1 --2- 172.21.15.27:0/2929759538 >> v2:172.21.15.77:3300/0 conn(0x7f0e200cb4b0 0x7f0e200cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.145+0000 7f0e265ff700 1 --2- 172.21.15.27:0/2929759538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e20143190 0x7f0e200cc020 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.145+0000 7f0e265ff700 1 -- 172.21.15.27:0/2929759538 >> 172.21.15.27:0/2929759538 conn(0x7f0e20063ca0 msgr2=0x7f0e2006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:03.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.145+0000 7f0e265ff700 1 -- 172.21.15.27:0/2929759538 shutdown_connections 2024-01-11T08:11:03.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.145+0000 7f0e265ff700 1 -- 172.21.15.27:0/2929759538 wait complete. 2024-01-11T08:11:03.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.147+0000 7f0e265ff700 1 Processor -- start 2024-01-11T08:11:03.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.147+0000 7f0e265ff700 1 -- start start 2024-01-11T08:11:03.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.148+0000 7f0e265ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e200cb4b0 0x7f0e20169940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:03.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.148+0000 7f0e265ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e20142dc0 0x7f0e200b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:03.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.148+0000 7f0e265ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e20143190 0x7f0e20153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:03.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.148+0000 7f0e265ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e201551d0 con 0x7f0e20143190 2024-01-11T08:11:03.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.148+0000 7f0e265ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e201545c0 con 0x7f0e200cb4b0 2024-01-11T08:11:03.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.148+0000 7f0e265ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e201548a0 con 0x7f0e20142dc0 2024-01-11T08:11:03.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.148+0000 7f0e24b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e20143190 0x7f0e20153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:03.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.148+0000 7f0e24b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e20143190 0x7f0e20153ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53496/0 (socket says 172.21.15.27:53496) 2024-01-11T08:11:03.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.148+0000 7f0e24b9c700 1 -- 172.21.15.27:0/709489652 learned_addr learned my addr 172.21.15.27:0/709489652 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:03.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.148+0000 7f0e1ffff700 1 --2- 172.21.15.27:0/709489652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e200cb4b0 0x7f0e20169940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:03.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.148+0000 7f0e1f7fe700 1 --2- 172.21.15.27:0/709489652 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e20142dc0 0x7f0e200b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:03.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.148+0000 7f0e24b9c700 1 -- 172.21.15.27:0/709489652 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e20142dc0 msgr2=0x7f0e200b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:03.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.148+0000 7f0e24b9c700 1 --2- 172.21.15.27:0/709489652 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e20142dc0 0x7f0e200b35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.149+0000 7f0e24b9c700 1 -- 172.21.15.27:0/709489652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e200cb4b0 msgr2=0x7f0e20169940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:03.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.149+0000 7f0e24b9c700 1 --2- 172.21.15.27:0/709489652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e200cb4b0 0x7f0e20169940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.149+0000 7f0e24b9c700 1 -- 172.21.15.27:0/709489652 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e140094f0 con 0x7f0e20143190 2024-01-11T08:11:03.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.149+0000 7f0e24b9c700 1 --2- 172.21.15.27:0/709489652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e20143190 0x7f0e20153ec0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0e18007f30 tx=0x7f0e180098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:03.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.149+0000 7f0e07fff700 1 -- 172.21.15.27:0/709489652 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1800ecf0 con 0x7f0e20143190 2024-01-11T08:11:03.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.149+0000 7f0e07fff700 1 -- 172.21.15.27:0/709489652 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1800ee50 con 0x7f0e20143190 2024-01-11T08:11:03.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.149+0000 7f0e265ff700 1 -- 172.21.15.27:0/709489652 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e20154b80 con 0x7f0e20143190 2024-01-11T08:11:03.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.149+0000 7f0e07fff700 1 -- 172.21.15.27:0/709489652 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18014670 con 0x7f0e20143190 2024-01-11T08:11:03.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.149+0000 7f0e265ff700 1 -- 172.21.15.27:0/709489652 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e20155660 con 0x7f0e20143190 2024-01-11T08:11:03.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.150+0000 7f0e265ff700 1 -- 172.21.15.27:0/709489652 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e20155340 con 0x7f0e20143190 2024-01-11T08:11:03.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.151+0000 7f0e07fff700 1 -- 172.21.15.27:0/709489652 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0e180149e0 con 0x7f0e20143190 2024-01-11T08:11:03.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.152+0000 7f0e07fff700 1 --2- 172.21.15.27:0/709489652 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0e0c067000 0x7f0e0c0694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:03.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.152+0000 7f0e07fff700 1 -- 172.21.15.27:0/709489652 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f0e18084c80 con 0x7f0e20143190 2024-01-11T08:11:03.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.153+0000 7f0e1ffff700 1 --2- 172.21.15.27:0/709489652 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0e0c067000 0x7f0e0c0694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:03.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.153+0000 7f0e1ffff700 1 --2- 172.21.15.27:0/709489652 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0e0c067000 0x7f0e0c0694b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0e100011e0 tx=0x7f0e10007bf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:03.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.156+0000 7f0e07fff700 1 -- 172.21.15.27:0/709489652 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0e18055a00 con 0x7f0e20143190 2024-01-11T08:11:03.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.363+0000 7f0e265ff700 1 -- 172.21.15.27:0/709489652 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0e2004c3c0 con 0x7f0e20143190 2024-01-11T08:11:03.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.363+0000 7f0e07fff700 1 -- 172.21.15.27:0/709489652 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0e18009a20 con 0x7f0e20143190 2024-01-11T08:11:03.363 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T08:11:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.365+0000 7f0e265ff700 1 -- 172.21.15.27:0/709489652 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0e0c067000 msgr2=0x7f0e0c0694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.365+0000 7f0e265ff700 1 --2- 172.21.15.27:0/709489652 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0e0c067000 0x7f0e0c0694b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0e100011e0 tx=0x7f0e10007bf0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.365+0000 7f0e265ff700 1 -- 172.21.15.27:0/709489652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e20143190 msgr2=0x7f0e20153ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.365+0000 7f0e265ff700 1 --2- 172.21.15.27:0/709489652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e20143190 0x7f0e20153ec0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0e18007f30 tx=0x7f0e180098e0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.366+0000 7f0e265ff700 1 -- 172.21.15.27:0/709489652 shutdown_connections 2024-01-11T08:11:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.366+0000 7f0e265ff700 1 --2- 172.21.15.27:0/709489652 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0e0c067000 0x7f0e0c0694b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.366+0000 7f0e265ff700 1 --2- 172.21.15.27:0/709489652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e20143190 0x7f0e20153ec0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.366+0000 7f0e265ff700 1 --2- 172.21.15.27:0/709489652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e200cb4b0 0x7f0e20169940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.366+0000 7f0e265ff700 1 --2- 172.21.15.27:0/709489652 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e20142dc0 0x7f0e200b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.366+0000 7f0e265ff700 1 -- 172.21.15.27:0/709489652 >> 172.21.15.27:0/709489652 conn(0x7f0e20063ca0 msgr2=0x7f0e20141cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:03.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.366+0000 7f0e265ff700 1 -- 172.21.15.27:0/709489652 shutdown_connections 2024-01-11T08:11:03.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.366+0000 7f0e265ff700 1 -- 172.21.15.27:0/709489652 wait complete. 2024-01-11T08:11:03.384 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:11:03.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.583+0000 7fe34bfff700 1 Processor -- start 2024-01-11T08:11:03.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.584+0000 7fe34bfff700 1 -- start start 2024-01-11T08:11:03.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.585+0000 7fe34bfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe3440b03e0 0x7fe3440b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:03.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.585+0000 7fe34bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe3440b29c0 0x7fe344159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:03.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.585+0000 7fe34bfff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3440cd730 con 0x7fe3440b07b0 2024-01-11T08:11:03.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.585+0000 7fe34bfff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3440b2de0 con 0x7fe3440b29c0 2024-01-11T08:11:03.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.585+0000 7fe34bfff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3440ccb20 con 0x7fe3440b03e0 2024-01-11T08:11:03.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.585+0000 7fe34a59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe3440b29c0 0x7fe344159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:03.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.585+0000 7fe34959a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe3440b07b0 0x7fe344159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39054/0 (socket says 172.21.15.27:39054) 2024-01-11T08:11:03.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.585+0000 7fe34959a700 1 -- 172.21.15.27:0/5069990 learned_addr learned my addr 172.21.15.27:0/5069990 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:03.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.585+0000 7fe34a59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe3440b29c0 0x7fe344159910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53510/0 (socket says 172.21.15.27:53510) 2024-01-11T08:11:03.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.586+0000 7fe349d9b700 1 --2- 172.21.15.27:0/5069990 >> v2:172.21.15.77:3300/0 conn(0x7fe3440b03e0 0x7fe3440b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:03.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.586+0000 7fe34a59c700 1 -- 172.21.15.27:0/5069990 >> v2:172.21.15.77:3300/0 conn(0x7fe3440b03e0 msgr2=0x7fe3440b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:03.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.586+0000 7fe34a59c700 1 --2- 172.21.15.27:0/5069990 >> v2:172.21.15.77:3300/0 conn(0x7fe3440b03e0 0x7fe3440b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.586+0000 7fe34a59c700 1 -- 172.21.15.27:0/5069990 >> v1:172.21.15.27:6789/0 conn(0x7fe3440b07b0 legacy=0x7fe344159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:03.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.586+0000 7fe34a59c700 1 -- 172.21.15.27:0/5069990 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3440cce00 con 0x7fe3440b29c0 2024-01-11T08:11:03.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.586+0000 7fe34a59c700 1 --2- 172.21.15.27:0/5069990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3440b29c0 0x7fe344159910 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe344089470 tx=0x7fe33c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe728f7d2c8e89ea server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:03.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.586+0000 7fe348d99700 1 -- 172.21.15.27:0/5069990 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c00e070 con 0x7fe3440b29c0 2024-01-11T08:11:03.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.586+0000 7fe348d99700 1 -- 172.21.15.27:0/5069990 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33c00bc30 con 0x7fe3440b29c0 2024-01-11T08:11:03.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.586+0000 7fe348d99700 1 -- 172.21.15.27:0/5069990 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c012430 con 0x7fe3440b29c0 2024-01-11T08:11:03.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.587+0000 7fe34bfff700 1 -- 172.21.15.27:0/5069990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3440b29c0 msgr2=0x7fe344159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:03.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.587+0000 7fe34bfff700 1 --2- 172.21.15.27:0/5069990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3440b29c0 0x7fe344159910 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe344089470 tx=0x7fe33c009780 comp rx=0 tx=0).stop 2024-01-11T08:11:03.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.587+0000 7fe34bfff700 1 -- 172.21.15.27:0/5069990 shutdown_connections 2024-01-11T08:11:03.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.587+0000 7fe34bfff700 1 --2- 172.21.15.27:0/5069990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3440b29c0 0x7fe344159910 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.587+0000 7fe34bfff700 1 --2- 172.21.15.27:0/5069990 >> v2:172.21.15.77:3300/0 conn(0x7fe3440b03e0 0x7fe3440b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.587+0000 7fe34bfff700 1 -- 172.21.15.27:0/5069990 >> 172.21.15.27:0/5069990 conn(0x7fe34406ed40 msgr2=0x7fe34406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:03.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.587+0000 7fe34bfff700 1 -- 172.21.15.27:0/5069990 shutdown_connections 2024-01-11T08:11:03.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.587+0000 7fe34bfff700 1 -- 172.21.15.27:0/5069990 wait complete. 2024-01-11T08:11:03.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.589+0000 7fe34bfff700 1 Processor -- start 2024-01-11T08:11:03.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.589+0000 7fe34bfff700 1 -- start start 2024-01-11T08:11:03.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.591+0000 7fe34bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3440b03e0 0x7fe344149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:03.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.591+0000 7fe34bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3440b07b0 0x7fe3440c9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:03.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.591+0000 7fe34bfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe3440b29c0 0x7fe3440c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:03.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.591+0000 7fe34bfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3440c6e50 con 0x7fe3440b07b0 2024-01-11T08:11:03.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.591+0000 7fe34bfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3440d0c10 con 0x7fe3440b03e0 2024-01-11T08:11:03.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.591+0000 7fe34bfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3440d0000 con 0x7fe3440b29c0 2024-01-11T08:11:03.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.591+0000 7fe34959a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3440b07b0 0x7fe3440c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:03.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.591+0000 7fe349d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3440b03e0 0x7fe344149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:03.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.591+0000 7fe34959a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3440b07b0 0x7fe3440c9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53522/0 (socket says 172.21.15.27:53522) 2024-01-11T08:11:03.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.591+0000 7fe34959a700 1 -- 172.21.15.27:0/1937397773 learned_addr learned my addr 172.21.15.27:0/1937397773 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:03.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.591+0000 7fe34a59c700 1 --2- 172.21.15.27:0/1937397773 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe3440b29c0 0x7fe3440c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:03.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.591+0000 7fe34959a700 1 -- 172.21.15.27:0/1937397773 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe3440b29c0 msgr2=0x7fe3440c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:03.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.591+0000 7fe34959a700 1 --2- 172.21.15.27:0/1937397773 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe3440b29c0 0x7fe3440c9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.591+0000 7fe34959a700 1 -- 172.21.15.27:0/1937397773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3440b03e0 msgr2=0x7fe344149470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:03.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.591+0000 7fe34959a700 1 --2- 172.21.15.27:0/1937397773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3440b03e0 0x7fe344149470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.592+0000 7fe34959a700 1 -- 172.21.15.27:0/1937397773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33c009a20 con 0x7fe3440b07b0 2024-01-11T08:11:03.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.592+0000 7fe34959a700 1 --2- 172.21.15.27:0/1937397773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3440b07b0 0x7fe3440c9710 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe33400c7c0 tx=0x7fe334007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:03.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.593+0000 7fe339ffb700 1 -- 172.21.15.27:0/1937397773 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe334005b30 con 0x7fe3440b07b0 2024-01-11T08:11:03.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.593+0000 7fe339ffb700 1 -- 172.21.15.27:0/1937397773 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe334005c90 con 0x7fe3440b07b0 2024-01-11T08:11:03.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.593+0000 7fe339ffb700 1 -- 172.21.15.27:0/1937397773 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe334004a80 con 0x7fe3440b07b0 2024-01-11T08:11:03.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.593+0000 7fe34bfff700 1 -- 172.21.15.27:0/1937397773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3440d02e0 con 0x7fe3440b07b0 2024-01-11T08:11:03.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.594+0000 7fe34bfff700 1 -- 172.21.15.27:0/1937397773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3440d08a0 con 0x7fe3440b07b0 2024-01-11T08:11:03.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.595+0000 7fe34bfff700 1 -- 172.21.15.27:0/1937397773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3440718d0 con 0x7fe3440b07b0 2024-01-11T08:11:03.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.595+0000 7fe339ffb700 1 -- 172.21.15.27:0/1937397773 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe3340070d0 con 0x7fe3440b07b0 2024-01-11T08:11:03.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.597+0000 7fe339ffb700 1 --2- 172.21.15.27:0/1937397773 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe328067000 0x7fe3280694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:03.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.597+0000 7fe339ffb700 1 -- 172.21.15.27:0/1937397773 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fe3340837b0 con 0x7fe3440b07b0 2024-01-11T08:11:03.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.597+0000 7fe349d9b700 1 --2- 172.21.15.27:0/1937397773 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe328067000 0x7fe3280694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:03.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.598+0000 7fe349d9b700 1 --2- 172.21.15.27:0/1937397773 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe328067000 0x7fe3280694b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe340006fd0 tx=0x7fe340008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:03.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.601+0000 7fe339ffb700 1 -- 172.21.15.27:0/1937397773 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe33404b5a0 con 0x7fe3440b07b0 2024-01-11T08:11:03.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.802+0000 7fe34bfff700 1 -- 172.21.15.27:0/1937397773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3440cd0e0 con 0x7fe3440b07b0 2024-01-11T08:11:03.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.803+0000 7fe339ffb700 1 -- 172.21.15.27:0/1937397773 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15302 (secure 0 0 0) 0x7fe334050aa0 con 0x7fe3440b07b0 2024-01-11T08:11:03.803 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:03.803 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:31.464323+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:10:27.993251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-11T08:10:29.153959+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:11:03.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.805+0000 7fe34bfff700 1 -- 172.21.15.27:0/1937397773 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe328067000 msgr2=0x7fe3280694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:03.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.805+0000 7fe34bfff700 1 --2- 172.21.15.27:0/1937397773 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe328067000 0x7fe3280694b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe340006fd0 tx=0x7fe340008040 comp rx=0 tx=0).stop 2024-01-11T08:11:03.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.805+0000 7fe34bfff700 1 -- 172.21.15.27:0/1937397773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3440b07b0 msgr2=0x7fe3440c9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:03.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.805+0000 7fe34bfff700 1 --2- 172.21.15.27:0/1937397773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3440b07b0 0x7fe3440c9710 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe33400c7c0 tx=0x7fe334007f70 comp rx=0 tx=0).stop 2024-01-11T08:11:03.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.806+0000 7fe34bfff700 1 -- 172.21.15.27:0/1937397773 shutdown_connections 2024-01-11T08:11:03.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.806+0000 7fe34bfff700 1 --2- 172.21.15.27:0/1937397773 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe328067000 0x7fe3280694b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.806+0000 7fe34bfff700 1 --2- 172.21.15.27:0/1937397773 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe3440b29c0 0x7fe3440c9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.806+0000 7fe34bfff700 1 --2- 172.21.15.27:0/1937397773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3440b03e0 0x7fe344149470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.806+0000 7fe34bfff700 1 --2- 172.21.15.27:0/1937397773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3440b07b0 0x7fe3440c9710 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:03.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.806+0000 7fe34bfff700 1 -- 172.21.15.27:0/1937397773 >> 172.21.15.27:0/1937397773 conn(0x7fe34406ed40 msgr2=0x7fe3441592a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:03.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.806+0000 7fe34bfff700 1 -- 172.21.15.27:0/1937397773 shutdown_connections 2024-01-11T08:11:03.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:03.806+0000 7fe34bfff700 1 -- 172.21.15.27:0/1937397773 wait complete. 2024-01-11T08:11:03.823 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T08:10:27.993251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T08:10:29.153959+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T08:11:03.824 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:11:04.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.027+0000 7f786d7ff700 1 Processor -- start 2024-01-11T08:11:04.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.027+0000 7f786d7ff700 1 -- start start 2024-01-11T08:11:04.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.028+0000 7f786d7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78680ccd70 0x7f78680cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:04.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.028+0000 7f786d7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78680cf850 con 0x7f786815ae70 2024-01-11T08:11:04.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.028+0000 7f786d7ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78680cec40 con 0x7f786815a1b0 2024-01-11T08:11:04.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.028+0000 7f786d7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786815ac90 con 0x7f78680ccd70 2024-01-11T08:11:04.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.029+0000 7f78677fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f786815ae70 0x7f786815b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50548/0 (socket says 172.21.15.27:50548) 2024-01-11T08:11:04.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.029+0000 7f78677fe700 1 -- 172.21.15.27:0/1621451918 learned_addr learned my addr 172.21.15.27:0/1621451918 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:04.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.029+0000 7f78667fc700 1 --2- 172.21.15.27:0/1621451918 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78680ccd70 0x7f78680cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:04.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.029+0000 7f7865ffb700 1 -- 172.21.15.27:0/1621451918 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316497905 0 0) 0x7f78680cf850 con 0x7f786815ae70 2024-01-11T08:11:04.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.029+0000 7f7865ffb700 1 -- 172.21.15.27:0/1621451918 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7850003570 con 0x7f786815ae70 2024-01-11T08:11:04.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.029+0000 7f78667fc700 1 -- 172.21.15.27:0/1621451918 >> v1:172.21.15.77:6789/0 conn(0x7f786815a1b0 legacy=0x7f786815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:04.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.029+0000 7f7865ffb700 1 -- 172.21.15.27:0/1621451918 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021514947 0 0) 0x7f78680cec40 con 0x7f786815a1b0 2024-01-11T08:11:04.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.029+0000 7f78667fc700 1 -- 172.21.15.27:0/1621451918 >> v1:172.21.15.27:6789/0 conn(0x7f786815ae70 legacy=0x7f786815b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:04.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.029+0000 7f78667fc700 1 -- 172.21.15.27:0/1621451918 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78680cef20 con 0x7f78680ccd70 2024-01-11T08:11:04.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.030+0000 7f78667fc700 1 --2- 172.21.15.27:0/1621451918 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78680ccd70 0x7f78680cd1e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f785c009910 tx=0x7f785c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c0069e24e4de159 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:04.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.030+0000 7f7865ffb700 1 -- 172.21.15.27:0/1621451918 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c00b680 con 0x7f78680ccd70 2024-01-11T08:11:04.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.030+0000 7f7865ffb700 1 -- 172.21.15.27:0/1621451918 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785c00b7e0 con 0x7f78680ccd70 2024-01-11T08:11:04.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.030+0000 7f7865ffb700 1 -- 172.21.15.27:0/1621451918 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c00bba0 con 0x7f78680ccd70 2024-01-11T08:11:04.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.030+0000 7f786d7ff700 1 -- 172.21.15.27:0/1621451918 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78680ccd70 msgr2=0x7f78680cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:04.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.030+0000 7f786d7ff700 1 --2- 172.21.15.27:0/1621451918 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78680ccd70 0x7f78680cd1e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f785c009910 tx=0x7f785c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.031+0000 7f786d7ff700 1 -- 172.21.15.27:0/1621451918 shutdown_connections 2024-01-11T08:11:04.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.031+0000 7f786d7ff700 1 --2- 172.21.15.27:0/1621451918 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78680ccd70 0x7f78680cd1e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.031+0000 7f786d7ff700 1 -- 172.21.15.27:0/1621451918 >> 172.21.15.27:0/1621451918 conn(0x7f786806ed40 msgr2=0x7f786806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:04.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.031+0000 7f786d7ff700 1 -- 172.21.15.27:0/1621451918 shutdown_connections 2024-01-11T08:11:04.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.031+0000 7f786d7ff700 1 -- 172.21.15.27:0/1621451918 wait complete. 2024-01-11T08:11:04.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.032+0000 7f786d7ff700 1 Processor -- start 2024-01-11T08:11:04.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.032+0000 7f786d7ff700 1 -- start start 2024-01-11T08:11:04.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.033+0000 7f786d7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78680ccd70 0x7f7868156860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:04.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.033+0000 7f786d7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f786815a1b0 0x7f78680c8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:04.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.033+0000 7f786d7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f786815ae70 0x7f78680c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:04.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.033+0000 7f786d7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78680c9080 con 0x7f78680ccd70 2024-01-11T08:11:04.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.033+0000 7f786d7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78680ca8f0 con 0x7f786815a1b0 2024-01-11T08:11:04.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.033+0000 7f786d7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78680b4e20 con 0x7f786815ae70 2024-01-11T08:11:04.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.033+0000 7f7866ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78680ccd70 0x7f7868156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:04.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.033+0000 7f7866ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78680ccd70 0x7f7868156860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44098/0 (socket says 172.21.15.27:44098) 2024-01-11T08:11:04.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.033+0000 7f7866ffd700 1 -- 172.21.15.27:0/1315303783 learned_addr learned my addr 172.21.15.27:0/1315303783 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:04.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.034+0000 7f78667fc700 1 --2- 172.21.15.27:0/1315303783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f786815a1b0 0x7f78680c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:04.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.034+0000 7f78677fe700 1 --2- 172.21.15.27:0/1315303783 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f786815ae70 0x7f78680c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:04.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.034+0000 7f7866ffd700 1 -- 172.21.15.27:0/1315303783 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f786815ae70 msgr2=0x7f78680c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:04.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.034+0000 7f7866ffd700 1 --2- 172.21.15.27:0/1315303783 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f786815ae70 0x7f78680c88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.034+0000 7f7866ffd700 1 -- 172.21.15.27:0/1315303783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f786815a1b0 msgr2=0x7f78680c8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:04.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.034+0000 7f7866ffd700 1 --2- 172.21.15.27:0/1315303783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f786815a1b0 0x7f78680c8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.034+0000 7f7866ffd700 1 -- 172.21.15.27:0/1315303783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785c009420 con 0x7f78680ccd70 2024-01-11T08:11:04.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.034+0000 7f78667fc700 1 --2- 172.21.15.27:0/1315303783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f786815a1b0 0x7f78680c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.034+0000 7f7866ffd700 1 --2- 172.21.15.27:0/1315303783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78680ccd70 0x7f7868156860 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f785800c7e0 tx=0x7f7858007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.034+0000 7f784effd700 1 -- 172.21.15.27:0/1315303783 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7858012630 con 0x7f78680ccd70 2024-01-11T08:11:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.034+0000 7f784effd700 1 -- 172.21.15.27:0/1315303783 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78580127b0 con 0x7f78680ccd70 2024-01-11T08:11:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.034+0000 7f784effd700 1 -- 172.21.15.27:0/1315303783 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7858012b70 con 0x7f78680ccd70 2024-01-11T08:11:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.034+0000 7f786d7ff700 1 -- 172.21.15.27:0/1315303783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78680b4530 con 0x7f78680ccd70 2024-01-11T08:11:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.034+0000 7f786d7ff700 1 -- 172.21.15.27:0/1315303783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78680b4ab0 con 0x7f78680ccd70 2024-01-11T08:11:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.036+0000 7f786d7ff700 1 -- 172.21.15.27:0/1315303783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f786804c3c0 con 0x7f78680ccd70 2024-01-11T08:11:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.037+0000 7f784effd700 1 -- 172.21.15.27:0/1315303783 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f785801b400 con 0x7f78680ccd70 2024-01-11T08:11:04.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.038+0000 7f784effd700 1 --2- 172.21.15.27:0/1315303783 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7854066f30 0x7f78540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:04.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.039+0000 7f784effd700 1 -- 172.21.15.27:0/1315303783 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f78580840b0 con 0x7f78680ccd70 2024-01-11T08:11:04.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.039+0000 7f78667fc700 1 --2- 172.21.15.27:0/1315303783 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7854066f30 0x7f78540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:04.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.040+0000 7f78667fc700 1 --2- 172.21.15.27:0/1315303783 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7854066f30 0x7f78540693e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f785c0093f0 tx=0x7f785c017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:04.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.042+0000 7f784effd700 1 -- 172.21.15.27:0/1315303783 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7858051320 con 0x7f78680ccd70 2024-01-11T08:11:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.244+0000 7f786d7ff700 1 -- 172.21.15.27:0/1315303783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f78680cf4d0 con 0x7f78680ccd70 2024-01-11T08:11:04.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.244+0000 7f784effd700 1 -- 172.21.15.27:0/1315303783 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7858054940 con 0x7f78680ccd70 2024-01-11T08:11:04.244 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:11:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.246+0000 7f786d7ff700 1 -- 172.21.15.27:0/1315303783 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7854066f30 msgr2=0x7f78540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.247+0000 7f786d7ff700 1 --2- 172.21.15.27:0/1315303783 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7854066f30 0x7f78540693e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f785c0093f0 tx=0x7f785c017000 comp rx=0 tx=0).stop 2024-01-11T08:11:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.247+0000 7f786d7ff700 1 -- 172.21.15.27:0/1315303783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78680ccd70 msgr2=0x7f7868156860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.247+0000 7f786d7ff700 1 --2- 172.21.15.27:0/1315303783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78680ccd70 0x7f7868156860 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f785800c7e0 tx=0x7f7858007df0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.247+0000 7f786d7ff700 1 -- 172.21.15.27:0/1315303783 shutdown_connections 2024-01-11T08:11:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.247+0000 7f786d7ff700 1 --2- 172.21.15.27:0/1315303783 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7854066f30 0x7f78540693e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.247+0000 7f786d7ff700 1 --2- 172.21.15.27:0/1315303783 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f786815ae70 0x7f78680c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.247+0000 7f786d7ff700 1 --2- 172.21.15.27:0/1315303783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78680ccd70 0x7f7868156860 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.247+0000 7f786d7ff700 1 --2- 172.21.15.27:0/1315303783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f786815a1b0 0x7f78680c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.247+0000 7f786d7ff700 1 -- 172.21.15.27:0/1315303783 >> 172.21.15.27:0/1315303783 conn(0x7f786806ed40 msgr2=0x7f78680d05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.247+0000 7f786d7ff700 1 -- 172.21.15.27:0/1315303783 shutdown_connections 2024-01-11T08:11:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.247+0000 7f786d7ff700 1 -- 172.21.15.27:0/1315303783 wait complete. 2024-01-11T08:11:04.264 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T08:11:04.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.466+0000 7f510bbff700 1 Processor -- start 2024-01-11T08:11:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.467+0000 7f510bbff700 1 -- start start 2024-01-11T08:11:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.468+0000 7f510bbff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f51040ccd70 0x7f51040cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.468+0000 7f510bbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f510415ae70 0x7f510415b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.468+0000 7f510bbff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51040cf850 con 0x7f510415a1b0 2024-01-11T08:11:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.468+0000 7f510bbff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f510415ac90 con 0x7f51040ccd70 2024-01-11T08:11:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.468+0000 7f510bbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51040cec40 con 0x7f510415ae70 2024-01-11T08:11:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.468+0000 7f510919a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f51040ccd70 0x7f51040cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.469+0000 7f510919a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f51040ccd70 0x7f51040cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44114/0 (socket says 172.21.15.27:44114) 2024-01-11T08:11:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.469+0000 7f510919a700 1 -- 172.21.15.27:0/2194582990 learned_addr learned my addr 172.21.15.27:0/2194582990 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.468+0000 7f510999b700 1 --1- 172.21.15.27:0/2194582990 >> v1:172.21.15.27:6789/0 conn(0x7f510415a1b0 0x7f510415a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50562/0 (socket says 172.21.15.27:50562) 2024-01-11T08:11:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.469+0000 7f510a19c700 1 --2- 172.21.15.27:0/2194582990 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f510415ae70 0x7f510415b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.469+0000 7f510919a700 1 -- 172.21.15.27:0/2194582990 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f510415ae70 msgr2=0x7f510415b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.469+0000 7f510919a700 1 --2- 172.21.15.27:0/2194582990 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f510415ae70 0x7f510415b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.469+0000 7f510919a700 1 -- 172.21.15.27:0/2194582990 >> v1:172.21.15.27:6789/0 conn(0x7f510415a1b0 legacy=0x7f510415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.469+0000 7f510919a700 1 -- 172.21.15.27:0/2194582990 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51040cef20 con 0x7f51040ccd70 2024-01-11T08:11:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.469+0000 7f510919a700 1 --2- 172.21.15.27:0/2194582990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51040ccd70 0x7f51040cd210 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f50f40099b0 tx=0x7f50f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4320022cc85eb1d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.471+0000 7f5108999700 1 -- 172.21.15.27:0/2194582990 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f4009ea0 con 0x7f51040ccd70 2024-01-11T08:11:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.471+0000 7f5108999700 1 -- 172.21.15.27:0/2194582990 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f400b860 con 0x7f51040ccd70 2024-01-11T08:11:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.471+0000 7f5108999700 1 -- 172.21.15.27:0/2194582990 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f400bc00 con 0x7f51040ccd70 2024-01-11T08:11:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.471+0000 7f510bbff700 1 -- 172.21.15.27:0/2194582990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51040ccd70 msgr2=0x7f51040cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.471+0000 7f510bbff700 1 --2- 172.21.15.27:0/2194582990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51040ccd70 0x7f51040cd210 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f50f40099b0 tx=0x7f50f4009250 comp rx=0 tx=0).stop 2024-01-11T08:11:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.471+0000 7f510bbff700 1 -- 172.21.15.27:0/2194582990 shutdown_connections 2024-01-11T08:11:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.471+0000 7f510bbff700 1 --2- 172.21.15.27:0/2194582990 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f510415ae70 0x7f510415b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.471+0000 7f510bbff700 1 --2- 172.21.15.27:0/2194582990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51040ccd70 0x7f51040cd210 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.471+0000 7f510bbff700 1 -- 172.21.15.27:0/2194582990 >> 172.21.15.27:0/2194582990 conn(0x7f510406ed40 msgr2=0x7f510406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.472+0000 7f510bbff700 1 -- 172.21.15.27:0/2194582990 shutdown_connections 2024-01-11T08:11:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.472+0000 7f510bbff700 1 -- 172.21.15.27:0/2194582990 wait complete. 2024-01-11T08:11:04.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.473+0000 7f510bbff700 1 Processor -- start 2024-01-11T08:11:04.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.473+0000 7f510bbff700 1 -- start start 2024-01-11T08:11:04.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.474+0000 7f510bbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f51040ccd70 0x7f51040af7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:04.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.474+0000 7f510bbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f510415a1b0 0x7f51040cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:04.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.474+0000 7f510bbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f510415ae70 0x7f51040cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:04.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.474+0000 7f510bbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510416ad50 con 0x7f510415a1b0 2024-01-11T08:11:04.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.474+0000 7f510bbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510416b6d0 con 0x7f510415ae70 2024-01-11T08:11:04.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.474+0000 7f510bbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f510416aa70 con 0x7f51040ccd70 2024-01-11T08:11:04.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.474+0000 7f510919a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f510415a1b0 0x7f51040cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:04.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.475+0000 7f510919a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f510415a1b0 0x7f51040cd1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44130/0 (socket says 172.21.15.27:44130) 2024-01-11T08:11:04.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.475+0000 7f510919a700 1 -- 172.21.15.27:0/1474539934 learned_addr learned my addr 172.21.15.27:0/1474539934 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:04.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.475+0000 7f510a19c700 1 --2- 172.21.15.27:0/1474539934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f510415ae70 0x7f51040cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:04.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.475+0000 7f510999b700 1 --2- 172.21.15.27:0/1474539934 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f51040ccd70 0x7f51040af7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:04.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.475+0000 7f510919a700 1 -- 172.21.15.27:0/1474539934 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f51040ccd70 msgr2=0x7f51040af7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:04.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.475+0000 7f510919a700 1 --2- 172.21.15.27:0/1474539934 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f51040ccd70 0x7f51040af7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.475+0000 7f510919a700 1 -- 172.21.15.27:0/1474539934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f510415ae70 msgr2=0x7f51040cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:04.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.475+0000 7f510919a700 1 --2- 172.21.15.27:0/1474539934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f510415ae70 0x7f51040cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.475+0000 7f510919a700 1 -- 172.21.15.27:0/1474539934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f40094f0 con 0x7f510415a1b0 2024-01-11T08:11:04.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.475+0000 7f510999b700 1 --2- 172.21.15.27:0/1474539934 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f51040ccd70 0x7f51040af7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:04.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.475+0000 7f510919a700 1 --2- 172.21.15.27:0/1474539934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f510415a1b0 0x7f51040cd1b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5104069770 tx=0x7f50f4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:04.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.475+0000 7f50f9ffb700 1 -- 172.21.15.27:0/1474539934 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f400bbe0 con 0x7f510415a1b0 2024-01-11T08:11:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.476+0000 7f50f9ffb700 1 -- 172.21.15.27:0/1474539934 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f4011420 con 0x7f510415a1b0 2024-01-11T08:11:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.476+0000 7f50f9ffb700 1 -- 172.21.15.27:0/1474539934 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f40117c0 con 0x7f510415a1b0 2024-01-11T08:11:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.476+0000 7f510bbff700 1 -- 172.21.15.27:0/1474539934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f510416b080 con 0x7f510415a1b0 2024-01-11T08:11:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.476+0000 7f510bbff700 1 -- 172.21.15.27:0/1474539934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51040afd00 con 0x7f510415a1b0 2024-01-11T08:11:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.477+0000 7f510bbff700 1 -- 172.21.15.27:0/1474539934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f510416aee0 con 0x7f510415a1b0 2024-01-11T08:11:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.478+0000 7f50f9ffb700 1 -- 172.21.15.27:0/1474539934 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f50f4011bf0 con 0x7f510415a1b0 2024-01-11T08:11:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.479+0000 7f50f9ffb700 1 --2- 172.21.15.27:0/1474539934 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f50e8066f30 0x7f50e80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.479+0000 7f50f9ffb700 1 -- 172.21.15.27:0/1474539934 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f50f4085d80 con 0x7f510415a1b0 2024-01-11T08:11:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.479+0000 7f510999b700 1 --2- 172.21.15.27:0/1474539934 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f50e8066f30 0x7f50e80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.480+0000 7f510999b700 1 --2- 172.21.15.27:0/1474539934 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f50e8066f30 0x7f50e80693e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5100007fd0 tx=0x7f5100007e30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:04.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.483+0000 7f50f9ffb700 1 -- 172.21.15.27:0/1474539934 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f510416aee0 con 0x7f510415a1b0 2024-01-11T08:11:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.686+0000 7f510bbff700 1 -- 172.21.15.27:0/1474539934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f510416aee0 con 0x7f510415a1b0 2024-01-11T08:11:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.686+0000 7f50f9ffb700 1 -- 172.21.15.27:0/1474539934 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f510416aee0 con 0x7f510415a1b0 2024-01-11T08:11:04.687 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T08:11:04.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.689+0000 7f510bbff700 1 -- 172.21.15.27:0/1474539934 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f50e8066f30 msgr2=0x7f50e80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:04.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.689+0000 7f510bbff700 1 --2- 172.21.15.27:0/1474539934 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f50e8066f30 0x7f50e80693e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5100007fd0 tx=0x7f5100007e30 comp rx=0 tx=0).stop 2024-01-11T08:11:04.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.689+0000 7f510bbff700 1 -- 172.21.15.27:0/1474539934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f510415a1b0 msgr2=0x7f51040cd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:04.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.689+0000 7f510bbff700 1 --2- 172.21.15.27:0/1474539934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f510415a1b0 0x7f51040cd1b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5104069770 tx=0x7f50f4017000 comp rx=0 tx=0).stop 2024-01-11T08:11:04.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.689+0000 7f510bbff700 1 -- 172.21.15.27:0/1474539934 shutdown_connections 2024-01-11T08:11:04.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.689+0000 7f510bbff700 1 --2- 172.21.15.27:0/1474539934 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f50e8066f30 0x7f50e80693e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.689+0000 7f510bbff700 1 --2- 172.21.15.27:0/1474539934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f510415ae70 0x7f51040cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.689+0000 7f510bbff700 1 --2- 172.21.15.27:0/1474539934 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f51040ccd70 0x7f51040af7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.689+0000 7f510bbff700 1 --2- 172.21.15.27:0/1474539934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f510415a1b0 0x7f51040cd1b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.689+0000 7f510bbff700 1 -- 172.21.15.27:0/1474539934 >> 172.21.15.27:0/1474539934 conn(0x7f510406ed40 msgr2=0x7f5104158750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:04.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.689+0000 7f510bbff700 1 -- 172.21.15.27:0/1474539934 shutdown_connections 2024-01-11T08:11:04.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.689+0000 7f510bbff700 1 -- 172.21.15.27:0/1474539934 wait complete. 2024-01-11T08:11:04.707 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T08:11:04.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.910+0000 7f0a877ff700 1 Processor -- start 2024-01-11T08:11:04.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.911+0000 7f0a877ff700 1 -- start start 2024-01-11T08:11:04.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.912+0000 7f0a877ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0a801577e0 0x7f0a800ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:04.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.912+0000 7f0a877ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a80069540 0x7f0a800ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:04.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.912+0000 7f0a877ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a800699d0 con 0x7f0a80157410 2024-01-11T08:11:04.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.912+0000 7f0a877ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a80069cb0 con 0x7f0a801577e0 2024-01-11T08:11:04.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.912+0000 7f0a877ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a80069f90 con 0x7f0a80069540 2024-01-11T08:11:04.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.912+0000 7f0a8559b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0a80157410 0x7f0a800ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50566/0 (socket says 172.21.15.27:50566) 2024-01-11T08:11:04.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.912+0000 7f0a8559b700 1 -- 172.21.15.27:0/1063562845 learned_addr learned my addr 172.21.15.27:0/1063562845 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.912+0000 7f0a85d9c700 1 --2- 172.21.15.27:0/1063562845 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a80069540 0x7f0a800ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.912+0000 7f0a84d9a700 1 --2- 172.21.15.27:0/1063562845 >> v2:172.21.15.77:3300/0 conn(0x7f0a801577e0 0x7f0a800ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.913+0000 7f0a77fff700 1 -- 172.21.15.27:0/1063562845 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415531953 0 0) 0x7f0a800699d0 con 0x7f0a80157410 2024-01-11T08:11:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.913+0000 7f0a77fff700 1 -- 172.21.15.27:0/1063562845 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a6c003570 con 0x7f0a80157410 2024-01-11T08:11:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.913+0000 7f0a77fff700 1 -- 172.21.15.27:0/1063562845 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 822628559 0 0) 0x7f0a6c003570 con 0x7f0a80157410 2024-01-11T08:11:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.913+0000 7f0a85d9c700 1 -- 172.21.15.27:0/1063562845 >> v2:172.21.15.77:3300/0 conn(0x7f0a801577e0 msgr2=0x7f0a800ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.913+0000 7f0a85d9c700 1 --2- 172.21.15.27:0/1063562845 >> v2:172.21.15.77:3300/0 conn(0x7f0a801577e0 0x7f0a800ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.913+0000 7f0a85d9c700 1 -- 172.21.15.27:0/1063562845 >> v1:172.21.15.27:6789/0 conn(0x7f0a80157410 legacy=0x7f0a800ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.913+0000 7f0a85d9c700 1 -- 172.21.15.27:0/1063562845 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8006a270 con 0x7f0a80069540 2024-01-11T08:11:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.913+0000 7f0a84d9a700 1 --2- 172.21.15.27:0/1063562845 >> v2:172.21.15.77:3300/0 conn(0x7f0a801577e0 0x7f0a800ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.913+0000 7f0a85d9c700 1 --2- 172.21.15.27:0/1063562845 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a80069540 0x7f0a800ae540 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0a78009e60 tx=0x7f0a78009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e07ee43c537c65c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.914+0000 7f0a77fff700 1 -- 172.21.15.27:0/1063562845 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7800e070 con 0x7f0a80069540 2024-01-11T08:11:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.914+0000 7f0a77fff700 1 -- 172.21.15.27:0/1063562845 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a7800b7a0 con 0x7f0a80069540 2024-01-11T08:11:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.914+0000 7f0a77fff700 1 -- 172.21.15.27:0/1063562845 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c0015f0 con 0x7f0a80069540 2024-01-11T08:11:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.914+0000 7f0a877ff700 1 -- 172.21.15.27:0/1063562845 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a80069540 msgr2=0x7f0a800ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.914+0000 7f0a877ff700 1 --2- 172.21.15.27:0/1063562845 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a80069540 0x7f0a800ae540 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0a78009e60 tx=0x7f0a78009780 comp rx=0 tx=0).stop 2024-01-11T08:11:04.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.914+0000 7f0a877ff700 1 -- 172.21.15.27:0/1063562845 shutdown_connections 2024-01-11T08:11:04.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.914+0000 7f0a877ff700 1 --2- 172.21.15.27:0/1063562845 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a80069540 0x7f0a800ae540 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.914+0000 7f0a877ff700 1 --2- 172.21.15.27:0/1063562845 >> v2:172.21.15.77:3300/0 conn(0x7f0a801577e0 0x7f0a800ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.914+0000 7f0a877ff700 1 -- 172.21.15.27:0/1063562845 >> 172.21.15.27:0/1063562845 conn(0x7f0a800680a0 msgr2=0x7f0a800cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:04.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.914+0000 7f0a877ff700 1 -- 172.21.15.27:0/1063562845 shutdown_connections 2024-01-11T08:11:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.914+0000 7f0a877ff700 1 -- 172.21.15.27:0/1063562845 wait complete. 2024-01-11T08:11:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.916+0000 7f0a877ff700 1 Processor -- start 2024-01-11T08:11:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.916+0000 7f0a877ff700 1 -- start start 2024-01-11T08:11:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.917+0000 7f0a877ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a80069540 0x7f0a80168360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.917+0000 7f0a877ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a80157410 0x7f0a8013c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.917+0000 7f0a877ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a801577e0 0x7f0a800b0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.917+0000 7f0a877ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a80155120 con 0x7f0a80069540 2024-01-11T08:11:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.917+0000 7f0a877ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a800c37f0 con 0x7f0a80157410 2024-01-11T08:11:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.917+0000 7f0a877ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a800c3ad0 con 0x7f0a801577e0 2024-01-11T08:11:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.917+0000 7f0a8559b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a80069540 0x7f0a80168360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.917+0000 7f0a8559b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a80069540 0x7f0a80168360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44136/0 (socket says 172.21.15.27:44136) 2024-01-11T08:11:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.917+0000 7f0a8559b700 1 -- 172.21.15.27:0/1272205128 learned_addr learned my addr 172.21.15.27:0/1272205128 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.918+0000 7f0a85d9c700 1 --2- 172.21.15.27:0/1272205128 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a801577e0 0x7f0a800b0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.918+0000 7f0a84d9a700 1 --2- 172.21.15.27:0/1272205128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a80157410 0x7f0a8013c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.918+0000 7f0a8559b700 1 -- 172.21.15.27:0/1272205128 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a801577e0 msgr2=0x7f0a800b0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.918+0000 7f0a8559b700 1 --2- 172.21.15.27:0/1272205128 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a801577e0 0x7f0a800b0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.918+0000 7f0a8559b700 1 -- 172.21.15.27:0/1272205128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a80157410 msgr2=0x7f0a8013c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.918+0000 7f0a8559b700 1 --2- 172.21.15.27:0/1272205128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a80157410 0x7f0a8013c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.918+0000 7f0a8559b700 1 -- 172.21.15.27:0/1272205128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a70009420 con 0x7f0a80069540 2024-01-11T08:11:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.918+0000 7f0a8559b700 1 --2- 172.21.15.27:0/1272205128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a80069540 0x7f0a80168360 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c0010e0 tx=0x7f0a7c001440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.918+0000 7f0a757fa700 1 -- 172.21.15.27:0/1272205128 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c00ce40 con 0x7f0a80069540 2024-01-11T08:11:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.918+0000 7f0a757fa700 1 -- 172.21.15.27:0/1272205128 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a7c007300 con 0x7f0a80069540 2024-01-11T08:11:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.918+0000 7f0a757fa700 1 -- 172.21.15.27:0/1272205128 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c0057f0 con 0x7f0a80069540 2024-01-11T08:11:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.918+0000 7f0a877ff700 1 -- 172.21.15.27:0/1272205128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a78009a20 con 0x7f0a80069540 2024-01-11T08:11:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.919+0000 7f0a877ff700 1 -- 172.21.15.27:0/1272205128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a800c4090 con 0x7f0a80069540 2024-01-11T08:11:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.920+0000 7f0a877ff700 1 -- 172.21.15.27:0/1272205128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a8004c3c0 con 0x7f0a80069540 2024-01-11T08:11:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.921+0000 7f0a757fa700 1 -- 172.21.15.27:0/1272205128 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0a7c005c20 con 0x7f0a80069540 2024-01-11T08:11:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.921+0000 7f0a757fa700 1 --2- 172.21.15.27:0/1272205128 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0a64066f30 0x7f0a640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.922+0000 7f0a757fa700 1 -- 172.21.15.27:0/1272205128 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f0a7c0841e0 con 0x7f0a80069540 2024-01-11T08:11:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.922+0000 7f0a84d9a700 1 --2- 172.21.15.27:0/1272205128 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0a64066f30 0x7f0a640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:04.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.922+0000 7f0a84d9a700 1 --2- 172.21.15.27:0/1272205128 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0a64066f30 0x7f0a640693e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0a8006d560 tx=0x7f0a700091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:04.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:04.926+0000 7f0a757fa700 1 -- 172.21.15.27:0/1272205128 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0a7c051430 con 0x7f0a80069540 2024-01-11T08:11:05.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.128+0000 7f0a877ff700 1 -- 172.21.15.27:0/1272205128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0a800c7f80 con 0x7f0a80069540 2024-01-11T08:11:05.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.128+0000 7f0a757fa700 1 -- 172.21.15.27:0/1272205128 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0a7c054a50 con 0x7f0a80069540 2024-01-11T08:11:05.128 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T08:11:05.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.131+0000 7f0a877ff700 1 -- 172.21.15.27:0/1272205128 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0a64066f30 msgr2=0x7f0a640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:05.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.131+0000 7f0a877ff700 1 --2- 172.21.15.27:0/1272205128 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0a64066f30 0x7f0a640693e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0a8006d560 tx=0x7f0a700091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:05.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.131+0000 7f0a877ff700 1 -- 172.21.15.27:0/1272205128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a80069540 msgr2=0x7f0a80168360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:05.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.131+0000 7f0a877ff700 1 --2- 172.21.15.27:0/1272205128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a80069540 0x7f0a80168360 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c0010e0 tx=0x7f0a7c001440 comp rx=0 tx=0).stop 2024-01-11T08:11:05.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.131+0000 7f0a877ff700 1 -- 172.21.15.27:0/1272205128 shutdown_connections 2024-01-11T08:11:05.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.131+0000 7f0a877ff700 1 --2- 172.21.15.27:0/1272205128 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0a64066f30 0x7f0a640693e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:05.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.131+0000 7f0a877ff700 1 --2- 172.21.15.27:0/1272205128 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a801577e0 0x7f0a800b0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:05.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.131+0000 7f0a877ff700 1 --2- 172.21.15.27:0/1272205128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a80069540 0x7f0a80168360 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:05.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.131+0000 7f0a877ff700 1 --2- 172.21.15.27:0/1272205128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a80157410 0x7f0a8013c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:05.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.131+0000 7f0a877ff700 1 -- 172.21.15.27:0/1272205128 >> 172.21.15.27:0/1272205128 conn(0x7f0a800680a0 msgr2=0x7f0a80152a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:05.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.131+0000 7f0a877ff700 1 -- 172.21.15.27:0/1272205128 shutdown_connections 2024-01-11T08:11:05.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.131+0000 7f0a877ff700 1 -- 172.21.15.27:0/1272205128 wait complete. 2024-01-11T08:11:05.161 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2024-01-11T08:11:05.161 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-11T08:11:05.161 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2024-01-11T08:11:05.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.360+0000 7f66ef9ff700 1 Processor -- start 2024-01-11T08:11:05.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.360+0000 7f66ef9ff700 1 -- start start 2024-01-11T08:11:05.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.361+0000 7f66ef9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f66e80b03e0 0x7f66e80b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:05.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.362+0000 7f66ef9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f66e80b07b0 0x7f66e80b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:05.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.362+0000 7f66ef9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66e80cd730 con 0x7f66e8159200 2024-01-11T08:11:05.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.362+0000 7f66ef9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e8159d30 con 0x7f66e80b07b0 2024-01-11T08:11:05.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.362+0000 7f66ef9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e80ccb20 con 0x7f66e80b03e0 2024-01-11T08:11:05.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.362+0000 7f66ecf9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f66e80b07b0 0x7f66e80b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:05.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.362+0000 7f66ecf9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f66e80b07b0 0x7f66e80b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44148/0 (socket says 172.21.15.27:44148) 2024-01-11T08:11:05.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.362+0000 7f66ecf9a700 1 -- 172.21.15.27:0/3623440366 learned_addr learned my addr 172.21.15.27:0/3623440366 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:05.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.362+0000 7f66ed79b700 1 --2- 172.21.15.27:0/3623440366 >> v2:172.21.15.77:3300/0 conn(0x7f66e80b03e0 0x7f66e80b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.362+0000 7f66ecf9a700 1 -- 172.21.15.27:0/3623440366 >> v2:172.21.15.77:3300/0 conn(0x7f66e80b03e0 msgr2=0x7f66e80b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.362+0000 7f66ecf9a700 1 --2- 172.21.15.27:0/3623440366 >> v2:172.21.15.77:3300/0 conn(0x7f66e80b03e0 0x7f66e80b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.362+0000 7f66ecf9a700 1 -- 172.21.15.27:0/3623440366 >> v1:172.21.15.77:6789/0 conn(0x7f66e8159200 legacy=0x7f66e8159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.362+0000 7f66ecf9a700 1 -- 172.21.15.27:0/3623440366 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e80cce00 con 0x7f66e80b07b0 2024-01-11T08:11:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.362+0000 7f66dffff700 1 -- 172.21.15.27:0/3623440366 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084249161 0 0) 0x7f66e80cd730 con 0x7f66e8159200 2024-01-11T08:11:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.363+0000 7f66ecf9a700 1 --2- 172.21.15.27:0/3623440366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66e80b07b0 0x7f66e80b29c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f66e8089470 tx=0x7f66d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64cf8e3b59c8fdb4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.363+0000 7f66dffff700 1 -- 172.21.15.27:0/3623440366 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8009e70 con 0x7f66e80b07b0 2024-01-11T08:11:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.363+0000 7f66dffff700 1 -- 172.21.15.27:0/3623440366 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d800b860 con 0x7f66e80b07b0 2024-01-11T08:11:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.363+0000 7f66dffff700 1 -- 172.21.15.27:0/3623440366 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d800bc00 con 0x7f66e80b07b0 2024-01-11T08:11:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.363+0000 7f66ef9ff700 1 -- 172.21.15.27:0/3623440366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66e80b07b0 msgr2=0x7f66e80b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.363+0000 7f66ef9ff700 1 --2- 172.21.15.27:0/3623440366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66e80b07b0 0x7f66e80b29c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f66e8089470 tx=0x7f66d8009250 comp rx=0 tx=0).stop 2024-01-11T08:11:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.364+0000 7f66ef9ff700 1 -- 172.21.15.27:0/3623440366 shutdown_connections 2024-01-11T08:11:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.364+0000 7f66ef9ff700 1 --2- 172.21.15.27:0/3623440366 >> v2:172.21.15.77:3300/0 conn(0x7f66e80b03e0 0x7f66e80b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.364+0000 7f66ef9ff700 1 --2- 172.21.15.27:0/3623440366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66e80b07b0 0x7f66e80b29c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.364+0000 7f66ef9ff700 1 -- 172.21.15.27:0/3623440366 >> 172.21.15.27:0/3623440366 conn(0x7f66e806ed40 msgr2=0x7f66e806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.364+0000 7f66ef9ff700 1 -- 172.21.15.27:0/3623440366 shutdown_connections 2024-01-11T08:11:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.364+0000 7f66ef9ff700 1 -- 172.21.15.27:0/3623440366 wait complete. 2024-01-11T08:11:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.365+0000 7f66ef9ff700 1 Processor -- start 2024-01-11T08:11:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.365+0000 7f66ef9ff700 1 -- start start 2024-01-11T08:11:05.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.366+0000 7f66ef9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66e80b03e0 0x7f66e80ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:05.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.366+0000 7f66ef9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66e80b07b0 0x7f66e80d0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.366+0000 7f66ef9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66e8159200 0x7f66e81699f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.366+0000 7f66ef9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e80c9f40 con 0x7f66e80b03e0 2024-01-11T08:11:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.366+0000 7f66ef9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e816a8b0 con 0x7f66e80b07b0 2024-01-11T08:11:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.366+0000 7f66ef9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e816a5d0 con 0x7f66e8159200 2024-01-11T08:11:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.366+0000 7f66ed79b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66e80b03e0 0x7f66e80ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.367+0000 7f66ed79b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66e80b03e0 0x7f66e80ca3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44162/0 (socket says 172.21.15.27:44162) 2024-01-11T08:11:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.367+0000 7f66ed79b700 1 -- 172.21.15.27:0/4164192327 learned_addr learned my addr 172.21.15.27:0/4164192327 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.367+0000 7f66ecf9a700 1 --2- 172.21.15.27:0/4164192327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66e80b07b0 0x7f66e80d0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.367+0000 7f66edf9c700 1 --2- 172.21.15.27:0/4164192327 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66e8159200 0x7f66e81699f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.367+0000 7f66ed79b700 1 -- 172.21.15.27:0/4164192327 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66e8159200 msgr2=0x7f66e81699f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.367+0000 7f66ed79b700 1 --2- 172.21.15.27:0/4164192327 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66e8159200 0x7f66e81699f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.367+0000 7f66ed79b700 1 -- 172.21.15.27:0/4164192327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66e80b07b0 msgr2=0x7f66e80d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.367+0000 7f66ed79b700 1 --2- 172.21.15.27:0/4164192327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66e80b07b0 0x7f66e80d0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.367+0000 7f66ed79b700 1 -- 172.21.15.27:0/4164192327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d80094f0 con 0x7f66e80b03e0 2024-01-11T08:11:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.367+0000 7f66ed79b700 1 --2- 172.21.15.27:0/4164192327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66e80b03e0 0x7f66e80ca3d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f66e400ec80 tx=0x7f66e40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.367+0000 7f66dd7fa700 1 -- 172.21.15.27:0/4164192327 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e400c850 con 0x7f66e80b03e0 2024-01-11T08:11:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.367+0000 7f66dd7fa700 1 -- 172.21.15.27:0/4164192327 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e400c9b0 con 0x7f66e80b03e0 2024-01-11T08:11:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.367+0000 7f66dd7fa700 1 -- 172.21.15.27:0/4164192327 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e400cd50 con 0x7f66e80b03e0 2024-01-11T08:11:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.367+0000 7f66ef9ff700 1 -- 172.21.15.27:0/4164192327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e80c98f0 con 0x7f66e80b03e0 2024-01-11T08:11:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.368+0000 7f66ef9ff700 1 -- 172.21.15.27:0/4164192327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e81581d0 con 0x7f66e80b03e0 2024-01-11T08:11:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.369+0000 7f66ef9ff700 1 -- 172.21.15.27:0/4164192327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66e80718d0 con 0x7f66e80b03e0 2024-01-11T08:11:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.370+0000 7f66dd7fa700 1 -- 172.21.15.27:0/4164192327 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f66e4013680 con 0x7f66e80b03e0 2024-01-11T08:11:05.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.371+0000 7f66dd7fa700 1 --2- 172.21.15.27:0/4164192327 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f66cc066f30 0x7f66cc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:05.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.371+0000 7f66dd7fa700 1 -- 172.21.15.27:0/4164192327 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f66e4084780 con 0x7f66e80b03e0 2024-01-11T08:11:05.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.371+0000 7f66ecf9a700 1 --2- 172.21.15.27:0/4164192327 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f66cc066f30 0x7f66cc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:05.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.372+0000 7f66ecf9a700 1 --2- 172.21.15.27:0/4164192327 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f66cc066f30 0x7f66cc0693e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f66e8089470 tx=0x7f66d8017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:05.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.375+0000 7f66dd7fa700 1 -- 172.21.15.27:0/4164192327 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f66e40519d0 con 0x7f66e80b03e0 2024-01-11T08:11:05.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:05.658+0000 7f66ef9ff700 1 -- 172.21.15.27:0/4164192327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f66e80cd0e0 con 0x7f66e80b03e0 2024-01-11T08:11:06.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.615+0000 7f66dd7fa700 1 -- 172.21.15.27:0/4164192327 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v56) v1 ==== 161+0+0 (secure 0 0 0) 0x7f66e4054ff0 con 0x7f66e80b03e0 2024-01-11T08:11:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.618+0000 7f66ef9ff700 1 -- 172.21.15.27:0/4164192327 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f66cc066f30 msgr2=0x7f66cc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.618+0000 7f66ef9ff700 1 --2- 172.21.15.27:0/4164192327 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f66cc066f30 0x7f66cc0693e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f66e8089470 tx=0x7f66d8017000 comp rx=0 tx=0).stop 2024-01-11T08:11:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.618+0000 7f66ef9ff700 1 -- 172.21.15.27:0/4164192327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66e80b03e0 msgr2=0x7f66e80ca3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:06.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.618+0000 7f66ef9ff700 1 --2- 172.21.15.27:0/4164192327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66e80b03e0 0x7f66e80ca3d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f66e400ec80 tx=0x7f66e40091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.618+0000 7f66ef9ff700 1 -- 172.21.15.27:0/4164192327 shutdown_connections 2024-01-11T08:11:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.618+0000 7f66ef9ff700 1 --2- 172.21.15.27:0/4164192327 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f66cc066f30 0x7f66cc0693e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.618+0000 7f66ef9ff700 1 --2- 172.21.15.27:0/4164192327 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66e8159200 0x7f66e81699f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.618+0000 7f66ef9ff700 1 --2- 172.21.15.27:0/4164192327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66e80b03e0 0x7f66e80ca3d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.618+0000 7f66ef9ff700 1 --2- 172.21.15.27:0/4164192327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66e80b07b0 0x7f66e80d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.618+0000 7f66ef9ff700 1 -- 172.21.15.27:0/4164192327 >> 172.21.15.27:0/4164192327 conn(0x7f66e806ed40 msgr2=0x7f66e80cb400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.618+0000 7f66ef9ff700 1 -- 172.21.15.27:0/4164192327 shutdown_connections 2024-01-11T08:11:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.618+0000 7f66ef9ff700 1 -- 172.21.15.27:0/4164192327 wait complete. 2024-01-11T08:11:06.639 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-11T08:11:06.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.846+0000 7fc2eb9ff700 1 Processor -- start 2024-01-11T08:11:06.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.846+0000 7fc2eb9ff700 1 -- start start 2024-01-11T08:11:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.847+0000 7fc2eb9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc2e4142dc0 0x7fc2e40cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.847+0000 7fc2eb9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e415d320 con 0x7fc2e40cb2e0 2024-01-11T08:11:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.847+0000 7fc2eb9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e40cb700 con 0x7fc2e4143190 2024-01-11T08:11:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.847+0000 7fc2eb9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e415df30 con 0x7fc2e4142dc0 2024-01-11T08:11:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.847+0000 7fc2e9f9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc2e40cb2e0 0x7fc2e4068720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50580/0 (socket says 172.21.15.27:50580) 2024-01-11T08:11:06.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.847+0000 7fc2e9f9c700 1 -- 172.21.15.27:0/429791563 learned_addr learned my addr 172.21.15.27:0/429791563 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:06.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.848+0000 7fc2e979b700 1 --2- 172.21.15.27:0/429791563 >> v2:172.21.15.77:3300/0 conn(0x7fc2e4142dc0 0x7fc2e40cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.848+0000 7fc2dbfff700 1 -- 172.21.15.27:0/429791563 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752224559 0 0) 0x7fc2e415d320 con 0x7fc2e40cb2e0 2024-01-11T08:11:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.848+0000 7fc2dbfff700 1 -- 172.21.15.27:0/429791563 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2d0003570 con 0x7fc2e40cb2e0 2024-01-11T08:11:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.848+0000 7fc2dbfff700 1 -- 172.21.15.27:0/429791563 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2055059658 0 0) 0x7fc2d0003570 con 0x7fc2e40cb2e0 2024-01-11T08:11:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.848+0000 7fc2dbfff700 1 -- 172.21.15.27:0/429791563 >> v2:172.21.15.77:3300/0 conn(0x7fc2e4142dc0 msgr2=0x7fc2e40cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.848+0000 7fc2dbfff700 1 --2- 172.21.15.27:0/429791563 >> v2:172.21.15.77:3300/0 conn(0x7fc2e4142dc0 0x7fc2e40cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.848+0000 7fc2dbfff700 1 -- 172.21.15.27:0/429791563 >> v1:172.21.15.77:6789/0 conn(0x7fc2e4143190 legacy=0x7fc2e40cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.848+0000 7fc2dbfff700 1 -- 172.21.15.27:0/429791563 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e415d590 con 0x7fc2e40cb2e0 2024-01-11T08:11:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.848+0000 7fc2dbfff700 1 -- 172.21.15.27:0/429791563 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fc2dc0017f0 con 0x7fc2e40cb2e0 2024-01-11T08:11:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.848+0000 7fc2dbfff700 1 -- 172.21.15.27:0/429791563 >> v1:172.21.15.27:6789/0 conn(0x7fc2e40cb2e0 legacy=0x7fc2e4068720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:06.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.848+0000 7fc2dbfff700 1 --2- 172.21.15.27:0/429791563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2d0003a60 0x7fc2d0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:06.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.848+0000 7fc2dbfff700 1 --2- 172.21.15.27:0/429791563 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2d0004810 0x7fc2d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:06.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.848+0000 7fc2dbfff700 1 --2- 172.21.15.27:0/429791563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d00051c0 0x7fc2d000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:06.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.848+0000 7fc2dbfff700 1 -- 172.21.15.27:0/429791563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e415df30 con 0x7fc2d00051c0 2024-01-11T08:11:06.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.848+0000 7fc2dbfff700 1 -- 172.21.15.27:0/429791563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d000c050 con 0x7fc2d0003a60 2024-01-11T08:11:06.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.848+0000 7fc2dbfff700 1 -- 172.21.15.27:0/429791563 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d000c210 con 0x7fc2d0004810 2024-01-11T08:11:06.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.849+0000 7fc2e9f9c700 1 --2- 172.21.15.27:0/429791563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d00051c0 0x7fc2d000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:06.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.849+0000 7fc2e979b700 1 --2- 172.21.15.27:0/429791563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2d0003a60 0x7fc2d0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:06.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.849+0000 7fc2e8f9a700 1 --2- 172.21.15.27:0/429791563 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2d0004810 0x7fc2d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:06.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.849+0000 7fc2e9f9c700 1 -- 172.21.15.27:0/429791563 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2d0004810 msgr2=0x7fc2d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.849+0000 7fc2e9f9c700 1 --2- 172.21.15.27:0/429791563 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2d0004810 0x7fc2d0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.849+0000 7fc2e9f9c700 1 -- 172.21.15.27:0/429791563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2d0003a60 msgr2=0x7fc2d0003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.849+0000 7fc2e9f9c700 1 --2- 172.21.15.27:0/429791563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2d0003a60 0x7fc2d0003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.849+0000 7fc2e9f9c700 1 -- 172.21.15.27:0/429791563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d0003570 con 0x7fc2d00051c0 2024-01-11T08:11:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.849+0000 7fc2e9f9c700 1 -- 172.21.15.27:0/429791563 reap_dead start 2024-01-11T08:11:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.849+0000 7fc2e9f9c700 1 --2- 172.21.15.27:0/429791563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d00051c0 0x7fc2d000ba60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00a8c0 tx=0x7fc2e4154220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e3599c7893b22f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.850+0000 7fc2dbfff700 1 -- 172.21.15.27:0/429791563 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc0070f0 con 0x7fc2d00051c0 2024-01-11T08:11:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.850+0000 7fc2dbfff700 1 -- 172.21.15.27:0/429791563 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2dc007270 con 0x7fc2d00051c0 2024-01-11T08:11:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.850+0000 7fc2dbfff700 1 -- 172.21.15.27:0/429791563 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc007630 con 0x7fc2d00051c0 2024-01-11T08:11:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.850+0000 7fc2eb9ff700 1 -- 172.21.15.27:0/429791563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d00051c0 msgr2=0x7fc2d000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.850+0000 7fc2eb9ff700 1 --2- 172.21.15.27:0/429791563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d00051c0 0x7fc2d000ba60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00a8c0 tx=0x7fc2e4154220 comp rx=0 tx=0).stop 2024-01-11T08:11:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.850+0000 7fc2eb9ff700 1 -- 172.21.15.27:0/429791563 shutdown_connections 2024-01-11T08:11:06.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.850+0000 7fc2eb9ff700 1 --2- 172.21.15.27:0/429791563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d00051c0 0x7fc2d000ba60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:06.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.850+0000 7fc2eb9ff700 1 -- 172.21.15.27:0/429791563 >> 172.21.15.27:0/429791563 conn(0x7fc2e4063ca0 msgr2=0x7fc2e406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:06.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.850+0000 7fc2eb9ff700 1 -- 172.21.15.27:0/429791563 shutdown_connections 2024-01-11T08:11:06.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.850+0000 7fc2eb9ff700 1 -- 172.21.15.27:0/429791563 wait complete. 2024-01-11T08:11:06.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.852+0000 7fc2eb9ff700 1 Processor -- start 2024-01-11T08:11:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.852+0000 7fc2eb9ff700 1 -- start start 2024-01-11T08:11:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.853+0000 7fc2eb9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d00051c0 0x7fc2e40afcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.853+0000 7fc2eb9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2e40b0200 0x7fc2e416c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.853+0000 7fc2eb9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2e416ce30 0x7fc2e40c5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.853+0000 7fc2eb9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e416d760 con 0x7fc2d00051c0 2024-01-11T08:11:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.853+0000 7fc2eb9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e416d2a0 con 0x7fc2e40b0200 2024-01-11T08:11:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.853+0000 7fc2eb9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e416a400 con 0x7fc2e416ce30 2024-01-11T08:11:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.853+0000 7fc2e979b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d00051c0 0x7fc2e40afcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.853+0000 7fc2e979b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d00051c0 0x7fc2e40afcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44182/0 (socket says 172.21.15.27:44182) 2024-01-11T08:11:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.853+0000 7fc2e979b700 1 -- 172.21.15.27:0/1463975256 learned_addr learned my addr 172.21.15.27:0/1463975256 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.853+0000 7fc2e8f9a700 1 --2- 172.21.15.27:0/1463975256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2e40b0200 0x7fc2e416c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.853+0000 7fc2e9f9c700 1 --2- 172.21.15.27:0/1463975256 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2e416ce30 0x7fc2e40c5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:06.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.854+0000 7fc2e979b700 1 -- 172.21.15.27:0/1463975256 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2e416ce30 msgr2=0x7fc2e40c5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:06.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.854+0000 7fc2e979b700 1 --2- 172.21.15.27:0/1463975256 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2e416ce30 0x7fc2e40c5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:06.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.854+0000 7fc2e979b700 1 -- 172.21.15.27:0/1463975256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2e40b0200 msgr2=0x7fc2e416c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:06.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.854+0000 7fc2e979b700 1 --2- 172.21.15.27:0/1463975256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2e40b0200 0x7fc2e416c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:06.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.854+0000 7fc2e979b700 1 -- 172.21.15.27:0/1463975256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e40c6a80 con 0x7fc2d00051c0 2024-01-11T08:11:06.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.854+0000 7fc2e979b700 1 --2- 172.21.15.27:0/1463975256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d00051c0 0x7fc2e40afcc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0011b10 tx=0x7fc2e00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:06.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.854+0000 7fc2d97fa700 1 -- 172.21.15.27:0/1463975256 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e000f940 con 0x7fc2d00051c0 2024-01-11T08:11:06.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.854+0000 7fc2d97fa700 1 -- 172.21.15.27:0/1463975256 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e000faa0 con 0x7fc2d00051c0 2024-01-11T08:11:06.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.854+0000 7fc2d97fa700 1 -- 172.21.15.27:0/1463975256 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e000fe20 con 0x7fc2d00051c0 2024-01-11T08:11:06.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.854+0000 7fc2eb9ff700 1 -- 172.21.15.27:0/1463975256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e40c6d60 con 0x7fc2d00051c0 2024-01-11T08:11:06.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.854+0000 7fc2eb9ff700 1 -- 172.21.15.27:0/1463975256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e416a950 con 0x7fc2d00051c0 2024-01-11T08:11:06.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.855+0000 7fc2eb9ff700 1 -- 172.21.15.27:0/1463975256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e404c3c0 con 0x7fc2d00051c0 2024-01-11T08:11:06.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.856+0000 7fc2d97fa700 1 -- 172.21.15.27:0/1463975256 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc2e0014650 con 0x7fc2d00051c0 2024-01-11T08:11:06.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.857+0000 7fc2d97fa700 1 --2- 172.21.15.27:0/1463975256 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc2c8066f30 0x7fc2c80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:06.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.857+0000 7fc2d97fa700 1 -- 172.21.15.27:0/1463975256 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fc2e0018030 con 0x7fc2d00051c0 2024-01-11T08:11:06.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.858+0000 7fc2e8f9a700 1 --2- 172.21.15.27:0/1463975256 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc2c8066f30 0x7fc2c80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:06.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.858+0000 7fc2e8f9a700 1 --2- 172.21.15.27:0/1463975256 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc2c8066f30 0x7fc2c80693e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40aa880 tx=0x7fc2d4007d30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:06.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:06.861+0000 7fc2d97fa700 1 -- 172.21.15.27:0/1463975256 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc2e00520e0 con 0x7fc2d00051c0 2024-01-11T08:11:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:07.066+0000 7fc2eb9ff700 1 -- 172.21.15.27:0/1463975256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc2e415d870 con 0x7fc2d00051c0 2024-01-11T08:11:07.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:07.072+0000 7fc2d97fa700 1 -- 172.21.15.27:0/1463975256 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc2e00220c0 con 0x7fc2d00051c0 2024-01-11T08:11:07.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:07.075+0000 7fc2eb9ff700 1 -- 172.21.15.27:0/1463975256 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc2c8066f30 msgr2=0x7fc2c80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:07.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:07.075+0000 7fc2eb9ff700 1 --2- 172.21.15.27:0/1463975256 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc2c8066f30 0x7fc2c80693e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40aa880 tx=0x7fc2d4007d30 comp rx=0 tx=0).stop 2024-01-11T08:11:07.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:07.075+0000 7fc2eb9ff700 1 -- 172.21.15.27:0/1463975256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d00051c0 msgr2=0x7fc2e40afcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:07.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:07.075+0000 7fc2eb9ff700 1 --2- 172.21.15.27:0/1463975256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d00051c0 0x7fc2e40afcc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0011b10 tx=0x7fc2e00091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:07.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:07.075+0000 7fc2eb9ff700 1 -- 172.21.15.27:0/1463975256 shutdown_connections 2024-01-11T08:11:07.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:07.075+0000 7fc2eb9ff700 1 --2- 172.21.15.27:0/1463975256 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc2c8066f30 0x7fc2c80693e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:07.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:07.075+0000 7fc2eb9ff700 1 --2- 172.21.15.27:0/1463975256 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2e416ce30 0x7fc2e40c5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:07.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:07.075+0000 7fc2eb9ff700 1 --2- 172.21.15.27:0/1463975256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d00051c0 0x7fc2e40afcc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:07.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:07.075+0000 7fc2eb9ff700 1 --2- 172.21.15.27:0/1463975256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2e40b0200 0x7fc2e416c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:07.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:07.075+0000 7fc2eb9ff700 1 -- 172.21.15.27:0/1463975256 >> 172.21.15.27:0/1463975256 conn(0x7fc2e4063ca0 msgr2=0x7fc2e40cbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:07.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:07.075+0000 7fc2eb9ff700 1 -- 172.21.15.27:0/1463975256 shutdown_connections 2024-01-11T08:11:07.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:07.075+0000 7fc2eb9ff700 1 -- 172.21.15.27:0/1463975256 wait complete. 2024-01-11T08:11:07.092 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T08:11:07.093 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:11:07.181 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:11:07.274 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T08:11:07.274 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-11T08:11:07.274 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:11:07.275 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-01-11T08:11:07.352 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T08:11:07.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:11:07.352 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-11T08:11:07.412 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-01-11T08:11:07.412 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:11:07.412 DEBUG:teuthology.orchestra.run.smithi077:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-01-11T08:11:07.490 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-01-11T08:11:07.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:11:07.491 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-11T08:11:07.546 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-01-11T08:11:07.546 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:11:07.546 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-01-11T08:11:07.631 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T08:11:07.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:11:07.632 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-11T08:11:07.684 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-01-11T08:11:07.684 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:11:07.684 DEBUG:teuthology.orchestra.run.smithi077:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-01-11T08:11:07.758 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-01-11T08:11:07.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:11:07.759 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-11T08:11:07.806 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-11T08:11:07.806 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:11:08.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.005+0000 7ff5843ff700 1 Processor -- start 2024-01-11T08:11:08.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.006+0000 7ff5843ff700 1 -- start start 2024-01-11T08:11:08.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.007+0000 7ff5843ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff57c0c8620 0x7ff57c0af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:08.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.007+0000 7ff5843ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff57c069d40 0x7ff57c0b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:08.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.007+0000 7ff5843ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff57c0c89f0 0x7ff57c0b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:08.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.007+0000 7ff5843ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff57c157390 con 0x7ff57c0c89f0 2024-01-11T08:11:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.007+0000 7ff5843ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff57c0afeb0 con 0x7ff57c069d40 2024-01-11T08:11:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.007+0000 7ff5843ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c06a200 con 0x7ff57c0c8620 2024-01-11T08:11:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.007+0000 7ff58299c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff57c0c89f0 0x7ff57c0b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.007+0000 7ff58299c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff57c0c89f0 0x7ff57c0b0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44184/0 (socket says 172.21.15.27:44184) 2024-01-11T08:11:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.007+0000 7ff58299c700 1 -- 172.21.15.27:0/4048997069 learned_addr learned my addr 172.21.15.27:0/4048997069 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.007+0000 7ff58219b700 1 --2- 172.21.15.27:0/4048997069 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff57c0c8620 0x7ff57c0af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.007+0000 7ff58199a700 1 --2- 172.21.15.27:0/4048997069 >> v2:172.21.15.77:3300/0 conn(0x7ff57c069d40 0x7ff57c0b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.008+0000 7ff58299c700 1 -- 172.21.15.27:0/4048997069 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff57c0c8620 msgr2=0x7ff57c0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.008+0000 7ff58299c700 1 --2- 172.21.15.27:0/4048997069 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff57c0c8620 0x7ff57c0af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.008+0000 7ff58299c700 1 -- 172.21.15.27:0/4048997069 >> v2:172.21.15.77:3300/0 conn(0x7ff57c069d40 msgr2=0x7ff57c0b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.008+0000 7ff58299c700 1 --2- 172.21.15.27:0/4048997069 >> v2:172.21.15.77:3300/0 conn(0x7ff57c069d40 0x7ff57c0b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.008+0000 7ff58299c700 1 -- 172.21.15.27:0/4048997069 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57c156d40 con 0x7ff57c0c89f0 2024-01-11T08:11:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.008+0000 7ff58219b700 1 --2- 172.21.15.27:0/4048997069 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff57c0c8620 0x7ff57c0af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.008+0000 7ff58199a700 1 --2- 172.21.15.27:0/4048997069 >> v2:172.21.15.77:3300/0 conn(0x7ff57c069d40 0x7ff57c0b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.008+0000 7ff58299c700 1 --2- 172.21.15.27:0/4048997069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff57c0c89f0 0x7ff57c0b0810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff57c069180 tx=0x7ff574009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7840ba12e1ef4616 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.008+0000 7ff581199700 1 -- 172.21.15.27:0/4048997069 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57400e070 con 0x7ff57c0c89f0 2024-01-11T08:11:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.008+0000 7ff581199700 1 -- 172.21.15.27:0/4048997069 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57400ba50 con 0x7ff57c0c89f0 2024-01-11T08:11:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.008+0000 7ff581199700 1 -- 172.21.15.27:0/4048997069 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5740092e0 con 0x7ff57c0c89f0 2024-01-11T08:11:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.009+0000 7ff5843ff700 1 -- 172.21.15.27:0/4048997069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff57c0c89f0 msgr2=0x7ff57c0b0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.009+0000 7ff5843ff700 1 --2- 172.21.15.27:0/4048997069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff57c0c89f0 0x7ff57c0b0810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff57c069180 tx=0x7ff574009840 comp rx=0 tx=0).stop 2024-01-11T08:11:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.009+0000 7ff5843ff700 1 -- 172.21.15.27:0/4048997069 shutdown_connections 2024-01-11T08:11:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.009+0000 7ff5843ff700 1 --2- 172.21.15.27:0/4048997069 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff57c0c8620 0x7ff57c0af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.009+0000 7ff5843ff700 1 --2- 172.21.15.27:0/4048997069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff57c0c89f0 0x7ff57c0b0810 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.009+0000 7ff5843ff700 1 --2- 172.21.15.27:0/4048997069 >> v2:172.21.15.77:3300/0 conn(0x7ff57c069d40 0x7ff57c0b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.009+0000 7ff5843ff700 1 -- 172.21.15.27:0/4048997069 >> 172.21.15.27:0/4048997069 conn(0x7ff57c0680e0 msgr2=0x7ff57c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.009+0000 7ff5843ff700 1 -- 172.21.15.27:0/4048997069 shutdown_connections 2024-01-11T08:11:08.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.009+0000 7ff5843ff700 1 -- 172.21.15.27:0/4048997069 wait complete. 2024-01-11T08:11:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.010+0000 7ff5843ff700 1 Processor -- start 2024-01-11T08:11:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.011+0000 7ff5843ff700 1 -- start start 2024-01-11T08:11:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.011+0000 7ff5843ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff57c069d40 0x7ff57c14b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.011+0000 7ff5843ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff57c0c8620 0x7ff57c0aa7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.012+0000 7ff5843ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff57c0c89f0 0x7ff57c14c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.012+0000 7ff5843ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c0ab240 con 0x7ff57c0c89f0 2024-01-11T08:11:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.012+0000 7ff5843ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c0ac3b0 con 0x7ff57c069d40 2024-01-11T08:11:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.012+0000 7ff5843ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c0ac690 con 0x7ff57c0c8620 2024-01-11T08:11:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.012+0000 7ff58299c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff57c0c89f0 0x7ff57c14c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:08.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.012+0000 7ff58299c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff57c0c89f0 0x7ff57c14c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44198/0 (socket says 172.21.15.27:44198) 2024-01-11T08:11:08.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.012+0000 7ff58299c700 1 -- 172.21.15.27:0/306657675 learned_addr learned my addr 172.21.15.27:0/306657675 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:08.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.012+0000 7ff58199a700 1 --2- 172.21.15.27:0/306657675 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff57c0c8620 0x7ff57c0aa7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:08.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.012+0000 7ff58219b700 1 --2- 172.21.15.27:0/306657675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff57c069d40 0x7ff57c14b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:08.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.012+0000 7ff58299c700 1 -- 172.21.15.27:0/306657675 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff57c0c8620 msgr2=0x7ff57c0aa7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:08.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.012+0000 7ff58299c700 1 --2- 172.21.15.27:0/306657675 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff57c0c8620 0x7ff57c0aa7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.012+0000 7ff58299c700 1 -- 172.21.15.27:0/306657675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff57c069d40 msgr2=0x7ff57c14b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:08.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.012+0000 7ff58299c700 1 --2- 172.21.15.27:0/306657675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff57c069d40 0x7ff57c14b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.012+0000 7ff58299c700 1 -- 172.21.15.27:0/306657675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff574009ae0 con 0x7ff57c0c89f0 2024-01-11T08:11:08.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.012+0000 7ff58219b700 1 --2- 172.21.15.27:0/306657675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff57c069d40 0x7ff57c14b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:11:08.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.013+0000 7ff58299c700 1 --2- 172.21.15.27:0/306657675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff57c0c89f0 0x7ff57c14c2d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff574012000 tx=0x7ff574018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:08.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.013+0000 7ff5727fc700 1 -- 172.21.15.27:0/306657675 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5740032f0 con 0x7ff57c0c89f0 2024-01-11T08:11:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.013+0000 7ff5727fc700 1 -- 172.21.15.27:0/306657675 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff574008e40 con 0x7ff57c0c89f0 2024-01-11T08:11:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.013+0000 7ff5727fc700 1 -- 172.21.15.27:0/306657675 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5740266b0 con 0x7ff57c0c89f0 2024-01-11T08:11:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.013+0000 7ff5843ff700 1 -- 172.21.15.27:0/306657675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57c0ac970 con 0x7ff57c0c89f0 2024-01-11T08:11:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.013+0000 7ff5843ff700 1 -- 172.21.15.27:0/306657675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff57c0a9ac0 con 0x7ff57c0c89f0 2024-01-11T08:11:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.014+0000 7ff5843ff700 1 -- 172.21.15.27:0/306657675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff57c0ace90 con 0x7ff57c0c89f0 2024-01-11T08:11:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.015+0000 7ff5727fc700 1 -- 172.21.15.27:0/306657675 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff574003040 con 0x7ff57c0c89f0 2024-01-11T08:11:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.016+0000 7ff5727fc700 1 --2- 172.21.15.27:0/306657675 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff560066f30 0x7ff5600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.016+0000 7ff5727fc700 1 -- 172.21.15.27:0/306657675 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7ff574086450 con 0x7ff57c0c89f0 2024-01-11T08:11:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.016+0000 7ff58219b700 1 --2- 172.21.15.27:0/306657675 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff560066f30 0x7ff5600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.017+0000 7ff58219b700 1 --2- 172.21.15.27:0/306657675 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff560066f30 0x7ff5600693e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff578009b70 tx=0x7ff578008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:08.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.020+0000 7ff5727fc700 1 -- 172.21.15.27:0/306657675 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff57c0ace90 con 0x7ff57c0c89f0 2024-01-11T08:11:08.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.265+0000 7ff5843ff700 1 -- 172.21.15.27:0/306657675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff57c0ace90 con 0x7ff57c0c89f0 2024-01-11T08:11:08.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.266+0000 7ff5727fc700 1 -- 172.21.15.27:0/306657675 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4588 (secure 0 0 0) 0x7ff57c0ace90 con 0x7ff57c0c89f0 2024-01-11T08:11:08.266 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:08.266 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3168764915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3168764915},{"type":"v1","addr":"172.21.15.27:6834","nonce":3168764915}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2607167395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2607167395},{"type":"v1","addr":"172.21.15.27:6835","nonce":2607167395}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2123423970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2123423970},{"type":"v1","addr":"172.21.15.77:6837","nonce":2123423970}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:10:30.160827+0000","modified":"2024-01-11T08:10:33.681210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6836/269464429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":269464429},{"type":"v1","addr":"172.21.15.77:6836","nonce":269464429}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2024-01-11T08:11:08.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.268+0000 7ff5843ff700 1 -- 172.21.15.27:0/306657675 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff560066f30 msgr2=0x7ff5600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:08.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.268+0000 7ff5843ff700 1 --2- 172.21.15.27:0/306657675 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff560066f30 0x7ff5600693e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff578009b70 tx=0x7ff578008040 comp rx=0 tx=0).stop 2024-01-11T08:11:08.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.268+0000 7ff5843ff700 1 -- 172.21.15.27:0/306657675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff57c0c89f0 msgr2=0x7ff57c14c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:08.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.268+0000 7ff5843ff700 1 --2- 172.21.15.27:0/306657675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff57c0c89f0 0x7ff57c14c2d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff574012000 tx=0x7ff574018000 comp rx=0 tx=0).stop 2024-01-11T08:11:08.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.268+0000 7ff5843ff700 1 -- 172.21.15.27:0/306657675 shutdown_connections 2024-01-11T08:11:08.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.268+0000 7ff5843ff700 1 --2- 172.21.15.27:0/306657675 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff560066f30 0x7ff5600693e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.268+0000 7ff5843ff700 1 --2- 172.21.15.27:0/306657675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff57c0c89f0 0x7ff57c14c2d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.268+0000 7ff5843ff700 1 --2- 172.21.15.27:0/306657675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff57c069d40 0x7ff57c14b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.268+0000 7ff5843ff700 1 --2- 172.21.15.27:0/306657675 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff57c0c8620 0x7ff57c0aa7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.268+0000 7ff5843ff700 1 -- 172.21.15.27:0/306657675 >> 172.21.15.27:0/306657675 conn(0x7ff57c0680e0 msgr2=0x7ff57c0cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:08.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.269+0000 7ff5843ff700 1 -- 172.21.15.27:0/306657675 shutdown_connections 2024-01-11T08:11:08.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.269+0000 7ff5843ff700 1 -- 172.21.15.27:0/306657675 wait complete. 2024-01-11T08:11:08.270 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 8 2024-01-11T08:11:08.286 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:11:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.487+0000 7f4db9fff700 1 Processor -- start 2024-01-11T08:11:08.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.487+0000 7f4db9fff700 1 -- start start 2024-01-11T08:11:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.488+0000 7f4db9fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4db40ca7b0 0x7f4db40cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.488+0000 7f4db9fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db415d640 con 0x7f4db4159e80 2024-01-11T08:11:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.489+0000 7f4db9fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db415a5d0 con 0x7f4db40ca3e0 2024-01-11T08:11:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.489+0000 7f4db9fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db415ae70 con 0x7f4db40ca7b0 2024-01-11T08:11:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.489+0000 7f4db2ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4db40ca7b0 0x7f4db40cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.489+0000 7f4db2ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4db40ca7b0 0x7f4db40cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44200/0 (socket says 172.21.15.27:44200) 2024-01-11T08:11:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.489+0000 7f4db2ffd700 1 -- 172.21.15.27:0/3115497892 learned_addr learned my addr 172.21.15.27:0/3115497892 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.489+0000 7f4db3fff700 1 --1- 172.21.15.27:0/3115497892 >> v1:172.21.15.27:6789/0 conn(0x7f4db4159e80 0x7f4db415dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50584/0 (socket says 172.21.15.27:50584) 2024-01-11T08:11:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.489+0000 7f4db2ffd700 1 -- 172.21.15.27:0/3115497892 >> v1:172.21.15.77:6789/0 conn(0x7f4db40ca3e0 legacy=0x7f4db40c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.489+0000 7f4db2ffd700 1 -- 172.21.15.27:0/3115497892 >> v1:172.21.15.27:6789/0 conn(0x7f4db4159e80 legacy=0x7f4db415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.489+0000 7f4db2ffd700 1 -- 172.21.15.27:0/3115497892 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db415a8b0 con 0x7f4db40ca7b0 2024-01-11T08:11:08.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.490+0000 7f4db2ffd700 1 --2- 172.21.15.27:0/3115497892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4db40ca7b0 0x7f4db40cca20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4da80099e0 tx=0x7f4da8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3246631e9066a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.490+0000 7f4db27fc700 1 -- 172.21.15.27:0/3115497892 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da800b820 con 0x7f4db40ca7b0 2024-01-11T08:11:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.490+0000 7f4db27fc700 1 -- 172.21.15.27:0/3115497892 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da800b980 con 0x7f4db40ca7b0 2024-01-11T08:11:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.490+0000 7f4db27fc700 1 -- 172.21.15.27:0/3115497892 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da800bd20 con 0x7f4db40ca7b0 2024-01-11T08:11:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.490+0000 7f4db9fff700 1 -- 172.21.15.27:0/3115497892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4db40ca7b0 msgr2=0x7f4db40cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.490+0000 7f4db9fff700 1 --2- 172.21.15.27:0/3115497892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4db40ca7b0 0x7f4db40cca20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4da80099e0 tx=0x7f4da8009250 comp rx=0 tx=0).stop 2024-01-11T08:11:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.490+0000 7f4db9fff700 1 -- 172.21.15.27:0/3115497892 shutdown_connections 2024-01-11T08:11:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.490+0000 7f4db9fff700 1 --2- 172.21.15.27:0/3115497892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4db40ca7b0 0x7f4db40cca20 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.490+0000 7f4db9fff700 1 -- 172.21.15.27:0/3115497892 >> 172.21.15.27:0/3115497892 conn(0x7f4db40ce3e0 msgr2=0x7f4db40ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.490+0000 7f4db9fff700 1 -- 172.21.15.27:0/3115497892 shutdown_connections 2024-01-11T08:11:08.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.491+0000 7f4db9fff700 1 -- 172.21.15.27:0/3115497892 wait complete. 2024-01-11T08:11:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.492+0000 7f4db9fff700 1 Processor -- start 2024-01-11T08:11:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.492+0000 7f4db9fff700 1 -- start start 2024-01-11T08:11:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.493+0000 7f4db9fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4db40ca3e0 0x7f4db41584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.493+0000 7f4db9fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4db40ca7b0 0x7f4db415bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.493+0000 7f4db9fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4db4159e80 0x7f4db414fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.493+0000 7f4db9fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db4158d90 con 0x7f4db40ca7b0 2024-01-11T08:11:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.493+0000 7f4db9fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db40b25e0 con 0x7f4db40ca3e0 2024-01-11T08:11:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.493+0000 7f4db9fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db40b2880 con 0x7f4db4159e80 2024-01-11T08:11:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.493+0000 7f4db2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4db40ca7b0 0x7f4db415bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.493+0000 7f4db2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4db40ca7b0 0x7f4db415bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44212/0 (socket says 172.21.15.27:44212) 2024-01-11T08:11:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.493+0000 7f4db2ffd700 1 -- 172.21.15.27:0/2466442960 learned_addr learned my addr 172.21.15.27:0/2466442960 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.494+0000 7f4db3fff700 1 --2- 172.21.15.27:0/2466442960 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4db4159e80 0x7f4db414fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.494+0000 7f4db37fe700 1 --2- 172.21.15.27:0/2466442960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4db40ca3e0 0x7f4db41584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.494+0000 7f4db2ffd700 1 -- 172.21.15.27:0/2466442960 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4db4159e80 msgr2=0x7f4db414fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.494+0000 7f4db2ffd700 1 --2- 172.21.15.27:0/2466442960 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4db4159e80 0x7f4db414fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.494+0000 7f4db2ffd700 1 -- 172.21.15.27:0/2466442960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4db40ca3e0 msgr2=0x7f4db41584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.494+0000 7f4db2ffd700 1 --2- 172.21.15.27:0/2466442960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4db40ca3e0 0x7f4db41584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.494+0000 7f4db2ffd700 1 -- 172.21.15.27:0/2466442960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da80094f0 con 0x7f4db40ca7b0 2024-01-11T08:11:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.494+0000 7f4db3fff700 1 --2- 172.21.15.27:0/2466442960 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4db4159e80 0x7f4db414fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:11:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.494+0000 7f4db37fe700 1 --2- 172.21.15.27:0/2466442960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4db40ca3e0 0x7f4db41584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:11:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.494+0000 7f4db2ffd700 1 --2- 172.21.15.27:0/2466442960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4db40ca7b0 0x7f4db415bcb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4da80094c0 tx=0x7f4da8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.494+0000 7f4d9b7fe700 1 -- 172.21.15.27:0/2466442960 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da800bcc0 con 0x7f4db40ca7b0 2024-01-11T08:11:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.494+0000 7f4d9b7fe700 1 -- 172.21.15.27:0/2466442960 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da801a630 con 0x7f4db40ca7b0 2024-01-11T08:11:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.494+0000 7f4d9b7fe700 1 -- 172.21.15.27:0/2466442960 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da801a9f0 con 0x7f4db40ca7b0 2024-01-11T08:11:08.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.494+0000 7f4db9fff700 1 -- 172.21.15.27:0/2466442960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db40b2b60 con 0x7f4db40ca7b0 2024-01-11T08:11:08.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.494+0000 7f4db9fff700 1 -- 172.21.15.27:0/2466442960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db40b3120 con 0x7f4db40ca7b0 2024-01-11T08:11:08.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.496+0000 7f4db9fff700 1 -- 172.21.15.27:0/2466442960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db40b3310 con 0x7f4db40ca7b0 2024-01-11T08:11:08.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.497+0000 7f4d9b7fe700 1 -- 172.21.15.27:0/2466442960 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4da802e450 con 0x7f4db40ca7b0 2024-01-11T08:11:08.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.497+0000 7f4d9b7fe700 1 --2- 172.21.15.27:0/2466442960 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4da0066f30 0x7f4da00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:08.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.498+0000 7f4d9b7fe700 1 -- 172.21.15.27:0/2466442960 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f4da80860a0 con 0x7f4db40ca7b0 2024-01-11T08:11:08.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.498+0000 7f4db37fe700 1 --2- 172.21.15.27:0/2466442960 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4da0066f30 0x7f4da00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.499+0000 7f4db37fe700 1 --2- 172.21.15.27:0/2466442960 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4da0066f30 0x7f4da00693e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4da40016b0 tx=0x7f4da4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:08.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.502+0000 7f4d9b7fe700 1 -- 172.21.15.27:0/2466442960 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4da8053270 con 0x7f4db40ca7b0 2024-01-11T08:11:08.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.704+0000 7f4db9fff700 1 -- 172.21.15.27:0/2466442960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4db40718d0 con 0x7f4db40ca7b0 2024-01-11T08:11:08.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.705+0000 7f4d9b7fe700 1 -- 172.21.15.27:0/2466442960 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15302 (secure 0 0 0) 0x7f4da8056890 con 0x7f4db40ca7b0 2024-01-11T08:11:08.705 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:08.705 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:31.464323+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:10:27.993251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-11T08:10:29.153959+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:11:08.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.707+0000 7f4db9fff700 1 -- 172.21.15.27:0/2466442960 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4da0066f30 msgr2=0x7f4da00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:08.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.707+0000 7f4db9fff700 1 --2- 172.21.15.27:0/2466442960 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4da0066f30 0x7f4da00693e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4da40016b0 tx=0x7f4da4009040 comp rx=0 tx=0).stop 2024-01-11T08:11:08.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.707+0000 7f4db9fff700 1 -- 172.21.15.27:0/2466442960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4db40ca7b0 msgr2=0x7f4db415bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:08.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.707+0000 7f4db9fff700 1 --2- 172.21.15.27:0/2466442960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4db40ca7b0 0x7f4db415bcb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4da80094c0 tx=0x7f4da8017000 comp rx=0 tx=0).stop 2024-01-11T08:11:08.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.707+0000 7f4db9fff700 1 -- 172.21.15.27:0/2466442960 shutdown_connections 2024-01-11T08:11:08.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.707+0000 7f4db9fff700 1 --2- 172.21.15.27:0/2466442960 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4da0066f30 0x7f4da00693e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.707+0000 7f4db9fff700 1 --2- 172.21.15.27:0/2466442960 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4db4159e80 0x7f4db414fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.707+0000 7f4db9fff700 1 --2- 172.21.15.27:0/2466442960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4db40ca3e0 0x7f4db41584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.707+0000 7f4db9fff700 1 --2- 172.21.15.27:0/2466442960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4db40ca7b0 0x7f4db415bcb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.707+0000 7f4db9fff700 1 -- 172.21.15.27:0/2466442960 >> 172.21.15.27:0/2466442960 conn(0x7f4db40ce3e0 msgr2=0x7f4db4069790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:08.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.708+0000 7f4db9fff700 1 -- 172.21.15.27:0/2466442960 shutdown_connections 2024-01-11T08:11:08.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.708+0000 7f4db9fff700 1 -- 172.21.15.27:0/2466442960 wait complete. 2024-01-11T08:11:08.725 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T08:10:27.993251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T08:10:29.153959+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T08:11:08.726 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:11:08.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.930+0000 7f8811fff700 1 Processor -- start 2024-01-11T08:11:08.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.930+0000 7f8811fff700 1 -- start start 2024-01-11T08:11:08.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.931+0000 7f8811fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f880c0ccd70 0x7f880c0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:08.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.931+0000 7f8811fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f880c15ae70 0x7f880c15b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:08.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.931+0000 7f8811fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f880c0cf850 con 0x7f880c15a1b0 2024-01-11T08:11:08.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.932+0000 7f8811fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f880c15ac90 con 0x7f880c0ccd70 2024-01-11T08:11:08.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.932+0000 7f8811fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f880c0cec40 con 0x7f880c15ae70 2024-01-11T08:11:08.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.932+0000 7f880affd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f880c0ccd70 0x7f880c0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.932+0000 7f880affd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f880c0ccd70 0x7f880c0cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44226/0 (socket says 172.21.15.27:44226) 2024-01-11T08:11:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.932+0000 7f880affd700 1 -- 172.21.15.27:0/1172449143 learned_addr learned my addr 172.21.15.27:0/1172449143 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.932+0000 7f880bfff700 1 --2- 172.21.15.27:0/1172449143 >> v2:172.21.15.77:3300/0 conn(0x7f880c15ae70 0x7f880c15b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.932+0000 7f880affd700 1 -- 172.21.15.27:0/1172449143 >> v2:172.21.15.77:3300/0 conn(0x7f880c15ae70 msgr2=0x7f880c15b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.932+0000 7f880affd700 1 --2- 172.21.15.27:0/1172449143 >> v2:172.21.15.77:3300/0 conn(0x7f880c15ae70 0x7f880c15b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.932+0000 7f880affd700 1 -- 172.21.15.27:0/1172449143 >> v1:172.21.15.77:6789/0 conn(0x7f880c15a1b0 legacy=0x7f880c15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.932+0000 7f880affd700 1 -- 172.21.15.27:0/1172449143 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f880c0cef20 con 0x7f880c0ccd70 2024-01-11T08:11:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.933+0000 7f880affd700 1 --2- 172.21.15.27:0/1172449143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f880c0ccd70 0x7f880c0cd210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f880c089810 tx=0x7f8800009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4996336f21bdfd2a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:08.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.933+0000 7f880a7fc700 1 -- 172.21.15.27:0/1172449143 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880000b860 con 0x7f880c0ccd70 2024-01-11T08:11:08.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.933+0000 7f880a7fc700 1 -- 172.21.15.27:0/1172449143 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880000b9c0 con 0x7f880c0ccd70 2024-01-11T08:11:08.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.933+0000 7f880a7fc700 1 -- 172.21.15.27:0/1172449143 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880000bd60 con 0x7f880c0ccd70 2024-01-11T08:11:08.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.933+0000 7f8811fff700 1 -- 172.21.15.27:0/1172449143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f880c0ccd70 msgr2=0x7f880c0cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:08.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.933+0000 7f8811fff700 1 --2- 172.21.15.27:0/1172449143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f880c0ccd70 0x7f880c0cd210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f880c089810 tx=0x7f8800009250 comp rx=0 tx=0).stop 2024-01-11T08:11:08.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.933+0000 7f8811fff700 1 -- 172.21.15.27:0/1172449143 shutdown_connections 2024-01-11T08:11:08.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.933+0000 7f8811fff700 1 --2- 172.21.15.27:0/1172449143 >> v2:172.21.15.77:3300/0 conn(0x7f880c15ae70 0x7f880c15b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.934+0000 7f8811fff700 1 --2- 172.21.15.27:0/1172449143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f880c0ccd70 0x7f880c0cd210 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.934+0000 7f8811fff700 1 -- 172.21.15.27:0/1172449143 >> 172.21.15.27:0/1172449143 conn(0x7f880c06ed40 msgr2=0x7f880c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:08.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.934+0000 7f8811fff700 1 -- 172.21.15.27:0/1172449143 shutdown_connections 2024-01-11T08:11:08.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.934+0000 7f8811fff700 1 -- 172.21.15.27:0/1172449143 wait complete. 2024-01-11T08:11:08.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.935+0000 7f8811fff700 1 Processor -- start 2024-01-11T08:11:08.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.935+0000 7f8811fff700 1 -- start start 2024-01-11T08:11:08.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.936+0000 7f8811fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f880c0ccd70 0x7f880c0af910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:08.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.936+0000 7f8811fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f880c15a1b0 0x7f880c0b2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:08.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.936+0000 7f8811fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f880c15ae70 0x7f880c0b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:08.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.936+0000 7f8811fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880c16ab40 con 0x7f880c15ae70 2024-01-11T08:11:08.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.936+0000 7f8811fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880c16b4e0 con 0x7f880c0ccd70 2024-01-11T08:11:08.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.936+0000 7f8811fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f880c0afe50 con 0x7f880c15a1b0 2024-01-11T08:11:08.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.936+0000 7f880bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f880c15ae70 0x7f880c0b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:08.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.936+0000 7f880bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f880c15ae70 0x7f880c0b27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44228/0 (socket says 172.21.15.27:44228) 2024-01-11T08:11:08.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.936+0000 7f880affd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f880c15a1b0 0x7f880c0b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:08.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.936+0000 7f880bfff700 1 -- 172.21.15.27:0/1976057879 learned_addr learned my addr 172.21.15.27:0/1976057879 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:08.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.936+0000 7f880b7fe700 1 --2- 172.21.15.27:0/1976057879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f880c0ccd70 0x7f880c0af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:08.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.937+0000 7f880bfff700 1 -- 172.21.15.27:0/1976057879 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f880c15a1b0 msgr2=0x7f880c0b2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:08.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.937+0000 7f880bfff700 1 --2- 172.21.15.27:0/1976057879 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f880c15a1b0 0x7f880c0b2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.937+0000 7f880bfff700 1 -- 172.21.15.27:0/1976057879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f880c0ccd70 msgr2=0x7f880c0af910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:08.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.937+0000 7f880bfff700 1 --2- 172.21.15.27:0/1976057879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f880c0ccd70 0x7f880c0af910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:08.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.937+0000 7f880bfff700 1 -- 172.21.15.27:0/1976057879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88000094f0 con 0x7f880c15ae70 2024-01-11T08:11:08.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.937+0000 7f880bfff700 1 --2- 172.21.15.27:0/1976057879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f880c15ae70 0x7f880c0b27d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8804009fa0 tx=0x7f8804009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:08.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.937+0000 7f87f37fe700 1 -- 172.21.15.27:0/1976057879 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880400ebc0 con 0x7f880c15ae70 2024-01-11T08:11:08.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.937+0000 7f87f37fe700 1 -- 172.21.15.27:0/1976057879 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880400ed20 con 0x7f880c15ae70 2024-01-11T08:11:08.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.937+0000 7f8811fff700 1 -- 172.21.15.27:0/1976057879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f880c16a860 con 0x7f880c15ae70 2024-01-11T08:11:08.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.937+0000 7f87f37fe700 1 -- 172.21.15.27:0/1976057879 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8804014670 con 0x7f880c15ae70 2024-01-11T08:11:08.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.938+0000 7f8811fff700 1 -- 172.21.15.27:0/1976057879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f880c16b170 con 0x7f880c15ae70 2024-01-11T08:11:08.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.939+0000 7f8811fff700 1 -- 172.21.15.27:0/1976057879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f880c16acf0 con 0x7f880c15ae70 2024-01-11T08:11:08.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.939+0000 7f87f37fe700 1 -- 172.21.15.27:0/1976057879 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f88040149e0 con 0x7f880c15ae70 2024-01-11T08:11:08.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.940+0000 7f87f37fe700 1 --2- 172.21.15.27:0/1976057879 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f87f8066f30 0x7f87f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:08.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.940+0000 7f87f37fe700 1 -- 172.21.15.27:0/1976057879 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f8804084c60 con 0x7f880c15ae70 2024-01-11T08:11:08.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.941+0000 7f880b7fe700 1 --2- 172.21.15.27:0/1976057879 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f87f8066f30 0x7f87f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:08.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.941+0000 7f880b7fe700 1 --2- 172.21.15.27:0/1976057879 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f87f8066f30 0x7f87f80693e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f87fc003a80 tx=0x7f87fc007ec0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:08.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:08.944+0000 7f87f37fe700 1 -- 172.21.15.27:0/1976057879 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f880c16acf0 con 0x7f880c15ae70 2024-01-11T08:11:09.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.151+0000 7f8811fff700 1 -- 172.21.15.27:0/1976057879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f880c16acf0 con 0x7f880c15ae70 2024-01-11T08:11:09.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.151+0000 7f87f37fe700 1 -- 172.21.15.27:0/1976057879 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f880c16acf0 con 0x7f880c15ae70 2024-01-11T08:11:09.151 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:11:09.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.154+0000 7f8811fff700 1 -- 172.21.15.27:0/1976057879 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f87f8066f30 msgr2=0x7f87f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:09.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.154+0000 7f8811fff700 1 --2- 172.21.15.27:0/1976057879 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f87f8066f30 0x7f87f80693e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f87fc003a80 tx=0x7f87fc007ec0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.154+0000 7f8811fff700 1 -- 172.21.15.27:0/1976057879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f880c15ae70 msgr2=0x7f880c0b27d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:09.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.154+0000 7f8811fff700 1 --2- 172.21.15.27:0/1976057879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f880c15ae70 0x7f880c0b27d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8804009fa0 tx=0x7f8804009900 comp rx=0 tx=0).stop 2024-01-11T08:11:09.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.154+0000 7f8811fff700 1 -- 172.21.15.27:0/1976057879 shutdown_connections 2024-01-11T08:11:09.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.154+0000 7f8811fff700 1 --2- 172.21.15.27:0/1976057879 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f87f8066f30 0x7f87f80693e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.154+0000 7f8811fff700 1 --2- 172.21.15.27:0/1976057879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f880c15ae70 0x7f880c0b27d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.154+0000 7f8811fff700 1 --2- 172.21.15.27:0/1976057879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f880c0ccd70 0x7f880c0af910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.154+0000 7f8811fff700 1 --2- 172.21.15.27:0/1976057879 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f880c15a1b0 0x7f880c0b2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.154+0000 7f8811fff700 1 -- 172.21.15.27:0/1976057879 >> 172.21.15.27:0/1976057879 conn(0x7f880c06ed40 msgr2=0x7f880c158a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:09.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.154+0000 7f8811fff700 1 -- 172.21.15.27:0/1976057879 shutdown_connections 2024-01-11T08:11:09.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.154+0000 7f8811fff700 1 -- 172.21.15.27:0/1976057879 wait complete. 2024-01-11T08:11:09.172 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T08:11:09.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.374+0000 7f4db17ff700 1 Processor -- start 2024-01-11T08:11:09.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.375+0000 7f4db17ff700 1 -- start start 2024-01-11T08:11:09.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.376+0000 7f4db17ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4dac143190 0x7f4dac0cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:09.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.376+0000 7f4db17ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4dac0cb4b0 0x7f4dac0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:09.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.376+0000 7f4db17ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dac15d320 con 0x7f4dac142dc0 2024-01-11T08:11:09.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.376+0000 7f4db17ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dac15df30 con 0x7f4dac0cb4b0 2024-01-11T08:11:09.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.376+0000 7f4db17ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dac15d590 con 0x7f4dac143190 2024-01-11T08:11:09.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.376+0000 7f4dab7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4dac0cb4b0 0x7f4dac0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:09.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.376+0000 7f4daaffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4dac142dc0 0x7f4dac0cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50588/0 (socket says 172.21.15.27:50588) 2024-01-11T08:11:09.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.376+0000 7f4dab7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4dac0cb4b0 0x7f4dac0cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44242/0 (socket says 172.21.15.27:44242) 2024-01-11T08:11:09.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.376+0000 7f4daaffd700 1 -- 172.21.15.27:0/2167364712 learned_addr learned my addr 172.21.15.27:0/2167364712 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:09.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.376+0000 7f4daa7fc700 1 --2- 172.21.15.27:0/2167364712 >> v2:172.21.15.77:3300/0 conn(0x7f4dac143190 0x7f4dac0cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:09.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.377+0000 7f4dab7fe700 1 -- 172.21.15.27:0/2167364712 >> v2:172.21.15.77:3300/0 conn(0x7f4dac143190 msgr2=0x7f4dac0cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:09.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.377+0000 7f4dab7fe700 1 --2- 172.21.15.27:0/2167364712 >> v2:172.21.15.77:3300/0 conn(0x7f4dac143190 0x7f4dac0cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.377+0000 7f4dab7fe700 1 -- 172.21.15.27:0/2167364712 >> v1:172.21.15.27:6789/0 conn(0x7f4dac142dc0 legacy=0x7f4dac0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:09.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.377+0000 7f4dab7fe700 1 -- 172.21.15.27:0/2167364712 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac15d870 con 0x7f4dac0cb4b0 2024-01-11T08:11:09.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.377+0000 7f4da9ffb700 1 -- 172.21.15.27:0/2167364712 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268023351 0 0) 0x7f4dac15d320 con 0x7f4dac142dc0 2024-01-11T08:11:09.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.377+0000 7f4daa7fc700 1 --2- 172.21.15.27:0/2167364712 >> v2:172.21.15.77:3300/0 conn(0x7f4dac143190 0x7f4dac0cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:09.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.377+0000 7f4dab7fe700 1 --2- 172.21.15.27:0/2167364712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac0cb4b0 0x7f4dac0cc560 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4dac06c480 tx=0x7f4da4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb0843954afcdd59 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:09.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.377+0000 7f4da9ffb700 1 -- 172.21.15.27:0/2167364712 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da400e070 con 0x7f4dac0cb4b0 2024-01-11T08:11:09.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.377+0000 7f4da9ffb700 1 -- 172.21.15.27:0/2167364712 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da400bb60 con 0x7f4dac0cb4b0 2024-01-11T08:11:09.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.377+0000 7f4da9ffb700 1 -- 172.21.15.27:0/2167364712 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da4012430 con 0x7f4dac0cb4b0 2024-01-11T08:11:09.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.378+0000 7f4db17ff700 1 -- 172.21.15.27:0/2167364712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac0cb4b0 msgr2=0x7f4dac0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:09.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.378+0000 7f4db17ff700 1 --2- 172.21.15.27:0/2167364712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac0cb4b0 0x7f4dac0cc560 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4dac06c480 tx=0x7f4da4009780 comp rx=0 tx=0).stop 2024-01-11T08:11:09.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.378+0000 7f4db17ff700 1 -- 172.21.15.27:0/2167364712 shutdown_connections 2024-01-11T08:11:09.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.378+0000 7f4db17ff700 1 --2- 172.21.15.27:0/2167364712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac0cb4b0 0x7f4dac0cc560 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.378+0000 7f4db17ff700 1 --2- 172.21.15.27:0/2167364712 >> v2:172.21.15.77:3300/0 conn(0x7f4dac143190 0x7f4dac0cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.378+0000 7f4db17ff700 1 -- 172.21.15.27:0/2167364712 >> 172.21.15.27:0/2167364712 conn(0x7f4dac063ca0 msgr2=0x7f4dac06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:09.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.378+0000 7f4db17ff700 1 -- 172.21.15.27:0/2167364712 shutdown_connections 2024-01-11T08:11:09.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.378+0000 7f4db17ff700 1 -- 172.21.15.27:0/2167364712 wait complete. 2024-01-11T08:11:09.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.380+0000 7f4db17ff700 1 Processor -- start 2024-01-11T08:11:09.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.380+0000 7f4db17ff700 1 -- start start 2024-01-11T08:11:09.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.382+0000 7f4db17ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4dac0cb4b0 0x7f4dac134ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.382+0000 7f4db17ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac142dc0 0x7f4dac0c83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.382+0000 7f4db17ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4dac143190 0x7f4dac0c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.382+0000 7f4db17ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac135220 con 0x7f4dac142dc0 2024-01-11T08:11:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.382+0000 7f4db17ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac169b10 con 0x7f4dac143190 2024-01-11T08:11:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.382+0000 7f4db17ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac0b2150 con 0x7f4dac0cb4b0 2024-01-11T08:11:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.382+0000 7f4daa7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac142dc0 0x7f4dac0c83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.382+0000 7f4daa7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac142dc0 0x7f4dac0c83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44244/0 (socket says 172.21.15.27:44244) 2024-01-11T08:11:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.382+0000 7f4daa7fc700 1 -- 172.21.15.27:0/464763048 learned_addr learned my addr 172.21.15.27:0/464763048 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.382+0000 7f4daaffd700 1 --2- 172.21.15.27:0/464763048 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4dac0cb4b0 0x7f4dac134ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.382+0000 7f4dab7fe700 1 --2- 172.21.15.27:0/464763048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4dac143190 0x7f4dac0c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.382+0000 7f4daa7fc700 1 -- 172.21.15.27:0/464763048 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4dac0cb4b0 msgr2=0x7f4dac134ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.382+0000 7f4daa7fc700 1 --2- 172.21.15.27:0/464763048 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4dac0cb4b0 0x7f4dac134ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.382+0000 7f4daa7fc700 1 -- 172.21.15.27:0/464763048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4dac143190 msgr2=0x7f4dac0c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.382+0000 7f4daa7fc700 1 --2- 172.21.15.27:0/464763048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4dac143190 0x7f4dac0c8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.382+0000 7f4daa7fc700 1 -- 172.21.15.27:0/464763048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da0009420 con 0x7f4dac142dc0 2024-01-11T08:11:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.383+0000 7f4daa7fc700 1 --2- 172.21.15.27:0/464763048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac142dc0 0x7f4dac0c83e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4da000ec80 tx=0x7f4da00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.383+0000 7f4d92ffd700 1 -- 172.21.15.27:0/464763048 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da000c9d0 con 0x7f4dac142dc0 2024-01-11T08:11:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.383+0000 7f4d92ffd700 1 -- 172.21.15.27:0/464763048 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da000cb30 con 0x7f4dac142dc0 2024-01-11T08:11:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.383+0000 7f4d92ffd700 1 -- 172.21.15.27:0/464763048 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da000ceb0 con 0x7f4dac142dc0 2024-01-11T08:11:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.383+0000 7f4db17ff700 1 -- 172.21.15.27:0/464763048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da4009a20 con 0x7f4dac142dc0 2024-01-11T08:11:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.383+0000 7f4db17ff700 1 -- 172.21.15.27:0/464763048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dac0b2710 con 0x7f4dac142dc0 2024-01-11T08:11:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.384+0000 7f4db17ff700 1 -- 172.21.15.27:0/464763048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dac0b28a0 con 0x7f4dac142dc0 2024-01-11T08:11:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.385+0000 7f4d92ffd700 1 -- 172.21.15.27:0/464763048 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4da0013680 con 0x7f4dac142dc0 2024-01-11T08:11:09.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.386+0000 7f4d92ffd700 1 --2- 172.21.15.27:0/464763048 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4d98066f30 0x7f4d980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:09.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.386+0000 7f4d92ffd700 1 -- 172.21.15.27:0/464763048 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f4da0084920 con 0x7f4dac142dc0 2024-01-11T08:11:09.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.387+0000 7f4daaffd700 1 --2- 172.21.15.27:0/464763048 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4d98066f30 0x7f4d980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.387+0000 7f4daaffd700 1 --2- 172.21.15.27:0/464763048 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4d98066f30 0x7f4d980693e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c0011e0 tx=0x7f4d9c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.390+0000 7f4d92ffd700 1 -- 172.21.15.27:0/464763048 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4da0051af0 con 0x7f4dac142dc0 2024-01-11T08:11:09.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.594+0000 7f4db17ff700 1 -- 172.21.15.27:0/464763048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4dac04c3f0 con 0x7f4dac142dc0 2024-01-11T08:11:09.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.595+0000 7f4d92ffd700 1 -- 172.21.15.27:0/464763048 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4da0055110 con 0x7f4dac142dc0 2024-01-11T08:11:09.595 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T08:11:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.597+0000 7f4db17ff700 1 -- 172.21.15.27:0/464763048 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4d98066f30 msgr2=0x7f4d980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.597+0000 7f4db17ff700 1 --2- 172.21.15.27:0/464763048 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4d98066f30 0x7f4d980693e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c0011e0 tx=0x7f4d9c009040 comp rx=0 tx=0).stop 2024-01-11T08:11:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.597+0000 7f4db17ff700 1 -- 172.21.15.27:0/464763048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac142dc0 msgr2=0x7f4dac0c83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.597+0000 7f4db17ff700 1 --2- 172.21.15.27:0/464763048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac142dc0 0x7f4dac0c83e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4da000ec80 tx=0x7f4da00091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.597+0000 7f4db17ff700 1 -- 172.21.15.27:0/464763048 shutdown_connections 2024-01-11T08:11:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.597+0000 7f4db17ff700 1 --2- 172.21.15.27:0/464763048 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4d98066f30 0x7f4d980693e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.597+0000 7f4db17ff700 1 --2- 172.21.15.27:0/464763048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4dac143190 0x7f4dac0c8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.597+0000 7f4db17ff700 1 --2- 172.21.15.27:0/464763048 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4dac0cb4b0 0x7f4dac134ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.597+0000 7f4db17ff700 1 --2- 172.21.15.27:0/464763048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dac142dc0 0x7f4dac0c83e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.597+0000 7f4db17ff700 1 -- 172.21.15.27:0/464763048 >> 172.21.15.27:0/464763048 conn(0x7f4dac063ca0 msgr2=0x7f4dac0c3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.597+0000 7f4db17ff700 1 -- 172.21.15.27:0/464763048 shutdown_connections 2024-01-11T08:11:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.597+0000 7f4db17ff700 1 -- 172.21.15.27:0/464763048 wait complete. 2024-01-11T08:11:09.615 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T08:11:09.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.819+0000 7f5f857ff700 1 Processor -- start 2024-01-11T08:11:09.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.819+0000 7f5f857ff700 1 -- start start 2024-01-11T08:11:09.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.821+0000 7f5f857ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f80068d30 0x7f5f800c8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:09.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.821+0000 7f5f857ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f800c80a0 con 0x7f5f801596f0 2024-01-11T08:11:09.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.821+0000 7f5f857ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f800cbac0 con 0x7f5f80159ac0 2024-01-11T08:11:09.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.821+0000 7f5f857ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f800a9fb0 con 0x7f5f80068d30 2024-01-11T08:11:09.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.821+0000 7f5f7effd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5f801596f0 0x7f5f800a98a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50590/0 (socket says 172.21.15.27:50590) 2024-01-11T08:11:09.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.821+0000 7f5f7effd700 1 -- 172.21.15.27:0/1706468504 learned_addr learned my addr 172.21.15.27:0/1706468504 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:09.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.821+0000 7f5f7f7fe700 1 --2- 172.21.15.27:0/1706468504 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f80068d30 0x7f5f800c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:09.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.821+0000 7f5f7dffb700 1 -- 172.21.15.27:0/1706468504 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037506574 0 0) 0x7f5f800c80a0 con 0x7f5f801596f0 2024-01-11T08:11:09.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.821+0000 7f5f7dffb700 1 -- 172.21.15.27:0/1706468504 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f68003570 con 0x7f5f801596f0 2024-01-11T08:11:09.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.822+0000 7f5f7dffb700 1 -- 172.21.15.27:0/1706468504 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552983536 0 0) 0x7f5f800cbac0 con 0x7f5f80159ac0 2024-01-11T08:11:09.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.822+0000 7f5f7dffb700 1 -- 172.21.15.27:0/1706468504 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f800c80a0 con 0x7f5f80159ac0 2024-01-11T08:11:09.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.822+0000 7f5f7dffb700 1 -- 172.21.15.27:0/1706468504 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1160889006 0 0) 0x7f5f68003570 con 0x7f5f801596f0 2024-01-11T08:11:09.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.822+0000 7f5f7dffb700 1 -- 172.21.15.27:0/1706468504 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f80068d30 msgr2=0x7f5f800c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:09.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.822+0000 7f5f7dffb700 1 --2- 172.21.15.27:0/1706468504 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f80068d30 0x7f5f800c8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.822+0000 7f5f7dffb700 1 -- 172.21.15.27:0/1706468504 >> v1:172.21.15.77:6789/0 conn(0x7f5f80159ac0 legacy=0x7f5f80068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:09.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.822+0000 7f5f7dffb700 1 -- 172.21.15.27:0/1706468504 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f800c7490 con 0x7f5f801596f0 2024-01-11T08:11:09.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.822+0000 7f5f7dffb700 1 -- 172.21.15.27:0/1706468504 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f5f70001470 con 0x7f5f801596f0 2024-01-11T08:11:09.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.822+0000 7f5f7dffb700 1 -- 172.21.15.27:0/1706468504 >> v1:172.21.15.27:6789/0 conn(0x7f5f801596f0 legacy=0x7f5f800a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:09.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.822+0000 7f5f7dffb700 1 --2- 172.21.15.27:0/1706468504 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f68003c20 0x7f5f68004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:09.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.822+0000 7f5f7dffb700 1 --2- 172.21.15.27:0/1706468504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f68004810 0x7f5f68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:09.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.822+0000 7f5f7dffb700 1 --2- 172.21.15.27:0/1706468504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f680051c0 0x7f5f6800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:09.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.822+0000 7f5f7dffb700 1 -- 172.21.15.27:0/1706468504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f800a9fb0 con 0x7f5f68004810 2024-01-11T08:11:09.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.822+0000 7f5f7dffb700 1 -- 172.21.15.27:0/1706468504 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6800c080 con 0x7f5f680051c0 2024-01-11T08:11:09.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.822+0000 7f5f7dffb700 1 -- 172.21.15.27:0/1706468504 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6800c240 con 0x7f5f68003c20 2024-01-11T08:11:09.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.822+0000 7f5f7e7fc700 1 --2- 172.21.15.27:0/1706468504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f68004810 0x7f5f68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:09.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.822+0000 7f5f7effd700 1 --2- 172.21.15.27:0/1706468504 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f68003c20 0x7f5f68004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:09.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.822+0000 7f5f7f7fe700 1 --2- 172.21.15.27:0/1706468504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f680051c0 0x7f5f6800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:09.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.823+0000 7f5f7e7fc700 1 -- 172.21.15.27:0/1706468504 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f68003c20 msgr2=0x7f5f68004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:09.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.823+0000 7f5f7e7fc700 1 --2- 172.21.15.27:0/1706468504 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f68003c20 0x7f5f68004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.823+0000 7f5f7e7fc700 1 -- 172.21.15.27:0/1706468504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f680051c0 msgr2=0x7f5f6800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:09.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.823+0000 7f5f7e7fc700 1 --2- 172.21.15.27:0/1706468504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f680051c0 0x7f5f6800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.823+0000 7f5f7e7fc700 1 -- 172.21.15.27:0/1706468504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f68003570 con 0x7f5f68004810 2024-01-11T08:11:09.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.823+0000 7f5f7e7fc700 1 --2- 172.21.15.27:0/1706468504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f68004810 0x7f5f68004c80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5f74001a50 tx=0x7f5f74007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b7b06c7a6130e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:09.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.823+0000 7f5f7dffb700 1 -- 172.21.15.27:0/1706468504 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7400c5d0 con 0x7f5f68004810 2024-01-11T08:11:09.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.823+0000 7f5f7dffb700 1 -- 172.21.15.27:0/1706468504 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7400c730 con 0x7f5f68004810 2024-01-11T08:11:09.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.823+0000 7f5f7dffb700 1 -- 172.21.15.27:0/1706468504 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7400cad0 con 0x7f5f68004810 2024-01-11T08:11:09.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.823+0000 7f5f857ff700 1 -- 172.21.15.27:0/1706468504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f68004810 msgr2=0x7f5f68004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:09.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.823+0000 7f5f857ff700 1 --2- 172.21.15.27:0/1706468504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f68004810 0x7f5f68004c80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5f74001a50 tx=0x7f5f74007d30 comp rx=0 tx=0).stop 2024-01-11T08:11:09.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.823+0000 7f5f7f7fe700 1 -- 172.21.15.27:0/1706468504 reap_dead start 2024-01-11T08:11:09.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.824+0000 7f5f857ff700 1 -- 172.21.15.27:0/1706468504 shutdown_connections 2024-01-11T08:11:09.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.824+0000 7f5f857ff700 1 -- 172.21.15.27:0/1706468504 >> 172.21.15.27:0/1706468504 conn(0x7f5f800c9fb0 msgr2=0x7f5f8006ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:09.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.824+0000 7f5f857ff700 1 -- 172.21.15.27:0/1706468504 shutdown_connections 2024-01-11T08:11:09.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.824+0000 7f5f857ff700 1 -- 172.21.15.27:0/1706468504 wait complete. 2024-01-11T08:11:09.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.825+0000 7f5f857ff700 1 Processor -- start 2024-01-11T08:11:09.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.825+0000 7f5f857ff700 1 -- start start 2024-01-11T08:11:09.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.826+0000 7f5f857ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f68004810 0x7f5f801551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:09.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.826+0000 7f5f857ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f800b0040 0x7f5f800c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:09.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.826+0000 7f5f857ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f800c8530 0x7f5f800c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:09.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.826+0000 7f5f857ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f800c8ef0 con 0x7f5f68004810 2024-01-11T08:11:09.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.826+0000 7f5f857ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f800b0770 con 0x7f5f800b0040 2024-01-11T08:11:09.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.826+0000 7f5f857ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f800b0a50 con 0x7f5f800c8530 2024-01-11T08:11:09.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.826+0000 7f5f7effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f68004810 0x7f5f801551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:09.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.826+0000 7f5f7effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f68004810 0x7f5f801551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44258/0 (socket says 172.21.15.27:44258) 2024-01-11T08:11:09.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.826+0000 7f5f7effd700 1 -- 172.21.15.27:0/881228124 learned_addr learned my addr 172.21.15.27:0/881228124 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:09.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.827+0000 7f5f7e7fc700 1 --2- 172.21.15.27:0/881228124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f800b0040 0x7f5f800c5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:09.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.827+0000 7f5f7f7fe700 1 --2- 172.21.15.27:0/881228124 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f800c8530 0x7f5f800c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:09.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.827+0000 7f5f7effd700 1 -- 172.21.15.27:0/881228124 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f800c8530 msgr2=0x7f5f800c89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:09.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.827+0000 7f5f7effd700 1 --2- 172.21.15.27:0/881228124 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f800c8530 0x7f5f800c89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.827+0000 7f5f7effd700 1 -- 172.21.15.27:0/881228124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f800b0040 msgr2=0x7f5f800c5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:09.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.827+0000 7f5f7effd700 1 --2- 172.21.15.27:0/881228124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f800b0040 0x7f5f800c5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:09.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.827+0000 7f5f7effd700 1 -- 172.21.15.27:0/881228124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f800c6340 con 0x7f5f68004810 2024-01-11T08:11:09.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.827+0000 7f5f7effd700 1 --2- 172.21.15.27:0/881228124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f68004810 0x7f5f801551e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5f700073f0 tx=0x7f5f70001d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:09.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.827+0000 7f5f66ffd700 1 -- 172.21.15.27:0/881228124 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f70001060 con 0x7f5f68004810 2024-01-11T08:11:09.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.827+0000 7f5f66ffd700 1 -- 172.21.15.27:0/881228124 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7000f640 con 0x7f5f68004810 2024-01-11T08:11:09.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.827+0000 7f5f66ffd700 1 -- 172.21.15.27:0/881228124 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7000fa00 con 0x7f5f68004810 2024-01-11T08:11:09.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.827+0000 7f5f857ff700 1 -- 172.21.15.27:0/881228124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f800c6620 con 0x7f5f68004810 2024-01-11T08:11:09.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.828+0000 7f5f857ff700 1 -- 172.21.15.27:0/881228124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f80155720 con 0x7f5f68004810 2024-01-11T08:11:09.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.829+0000 7f5f857ff700 1 -- 172.21.15.27:0/881228124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f8004c3c0 con 0x7f5f68004810 2024-01-11T08:11:09.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.830+0000 7f5f66ffd700 1 -- 172.21.15.27:0/881228124 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5f70014450 con 0x7f5f68004810 2024-01-11T08:11:09.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.831+0000 7f5f66ffd700 1 --2- 172.21.15.27:0/881228124 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5f6c066f30 0x7f5f6c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:09.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.831+0000 7f5f66ffd700 1 -- 172.21.15.27:0/881228124 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f5f70083c50 con 0x7f5f68004810 2024-01-11T08:11:09.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.831+0000 7f5f7e7fc700 1 --2- 172.21.15.27:0/881228124 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5f6c066f30 0x7f5f6c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:09.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.832+0000 7f5f7e7fc700 1 --2- 172.21.15.27:0/881228124 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5f6c066f30 0x7f5f6c0693e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5f74001ed0 tx=0x7f5f74015000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:09.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:09.835+0000 7f5f66ffd700 1 -- 172.21.15.27:0/881228124 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5f7004d5a0 con 0x7f5f68004810 2024-01-11T08:11:10.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.037+0000 7f5f857ff700 1 -- 172.21.15.27:0/881228124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5f800c7770 con 0x7f5f68004810 2024-01-11T08:11:10.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.037+0000 7f5f66ffd700 1 -- 172.21.15.27:0/881228124 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5f7001a070 con 0x7f5f68004810 2024-01-11T08:11:10.037 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T08:11:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.040+0000 7f5f857ff700 1 -- 172.21.15.27:0/881228124 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5f6c066f30 msgr2=0x7f5f6c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.040+0000 7f5f857ff700 1 --2- 172.21.15.27:0/881228124 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5f6c066f30 0x7f5f6c0693e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5f74001ed0 tx=0x7f5f74015000 comp rx=0 tx=0).stop 2024-01-11T08:11:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.040+0000 7f5f857ff700 1 -- 172.21.15.27:0/881228124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f68004810 msgr2=0x7f5f801551e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.040+0000 7f5f857ff700 1 --2- 172.21.15.27:0/881228124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f68004810 0x7f5f801551e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5f700073f0 tx=0x7f5f70001d20 comp rx=0 tx=0).stop 2024-01-11T08:11:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.040+0000 7f5f857ff700 1 -- 172.21.15.27:0/881228124 shutdown_connections 2024-01-11T08:11:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.040+0000 7f5f857ff700 1 --2- 172.21.15.27:0/881228124 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5f6c066f30 0x7f5f6c0693e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.040+0000 7f5f857ff700 1 --2- 172.21.15.27:0/881228124 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f800c8530 0x7f5f800c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.040+0000 7f5f857ff700 1 --2- 172.21.15.27:0/881228124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f68004810 0x7f5f801551e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.040+0000 7f5f857ff700 1 --2- 172.21.15.27:0/881228124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f800b0040 0x7f5f800c5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.040+0000 7f5f857ff700 1 -- 172.21.15.27:0/881228124 >> 172.21.15.27:0/881228124 conn(0x7f5f800c9fb0 msgr2=0x7f5f8006dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:10.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.040+0000 7f5f857ff700 1 -- 172.21.15.27:0/881228124 shutdown_connections 2024-01-11T08:11:10.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.040+0000 7f5f857ff700 1 -- 172.21.15.27:0/881228124 wait complete. 2024-01-11T08:11:10.058 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:11:10.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.256+0000 7f7afedff700 1 Processor -- start 2024-01-11T08:11:10.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.256+0000 7f7afedff700 1 -- start start 2024-01-11T08:11:10.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.258+0000 7f7afedff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7af80c8620 0x7f7af80af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:10.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.258+0000 7f7afedff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7af8069d40 0x7f7af80b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:10.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.258+0000 7f7afedff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af8157390 con 0x7f7af80c89f0 2024-01-11T08:11:10.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.258+0000 7f7afedff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af80adcd0 con 0x7f7af8069d40 2024-01-11T08:11:10.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.258+0000 7f7afedff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af80afeb0 con 0x7f7af80c8620 2024-01-11T08:11:10.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.258+0000 7f7af7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7af8069d40 0x7f7af80b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:10.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.258+0000 7f7afd39c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7af80c89f0 0x7f7af8159ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50596/0 (socket says 172.21.15.27:50596) 2024-01-11T08:11:10.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.258+0000 7f7af7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7af8069d40 0x7f7af80b02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44268/0 (socket says 172.21.15.27:44268) 2024-01-11T08:11:10.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.258+0000 7f7afd39c700 1 -- 172.21.15.27:0/733063271 learned_addr learned my addr 172.21.15.27:0/733063271 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:10.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.258+0000 7f7afcb9b700 1 --2- 172.21.15.27:0/733063271 >> v2:172.21.15.77:3300/0 conn(0x7f7af80c8620 0x7f7af80af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:10.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.259+0000 7f7af7fff700 1 -- 172.21.15.27:0/733063271 >> v2:172.21.15.77:3300/0 conn(0x7f7af80c8620 msgr2=0x7f7af80af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:10.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.259+0000 7f7af7fff700 1 --2- 172.21.15.27:0/733063271 >> v2:172.21.15.77:3300/0 conn(0x7f7af80c8620 0x7f7af80af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.259+0000 7f7af7fff700 1 -- 172.21.15.27:0/733063271 >> v1:172.21.15.27:6789/0 conn(0x7f7af80c89f0 legacy=0x7f7af8159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:10.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.259+0000 7f7af7fff700 1 -- 172.21.15.27:0/733063271 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af806a200 con 0x7f7af8069d40 2024-01-11T08:11:10.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.259+0000 7f7af77fe700 1 -- 172.21.15.27:0/733063271 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151402088 0 0) 0x7f7af8157390 con 0x7f7af80c89f0 2024-01-11T08:11:10.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.259+0000 7f7af7fff700 1 --2- 172.21.15.27:0/733063271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af8069d40 0x7f7af80b02d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7ae40098e0 tx=0x7f7ae40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7da6595471cdd90e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:10.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.259+0000 7f7af77fe700 1 -- 172.21.15.27:0/733063271 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4009e80 con 0x7f7af8069d40 2024-01-11T08:11:10.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.259+0000 7f7af77fe700 1 -- 172.21.15.27:0/733063271 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae400b820 con 0x7f7af8069d40 2024-01-11T08:11:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.259+0000 7f7af77fe700 1 -- 172.21.15.27:0/733063271 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae400bba0 con 0x7f7af8069d40 2024-01-11T08:11:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.260+0000 7f7afedff700 1 -- 172.21.15.27:0/733063271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af8069d40 msgr2=0x7f7af80b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.260+0000 7f7afedff700 1 --2- 172.21.15.27:0/733063271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af8069d40 0x7f7af80b02d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7ae40098e0 tx=0x7f7ae40091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.260+0000 7f7afedff700 1 -- 172.21.15.27:0/733063271 shutdown_connections 2024-01-11T08:11:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.260+0000 7f7afedff700 1 --2- 172.21.15.27:0/733063271 >> v2:172.21.15.77:3300/0 conn(0x7f7af80c8620 0x7f7af80af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.260+0000 7f7afedff700 1 --2- 172.21.15.27:0/733063271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af8069d40 0x7f7af80b02d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.260+0000 7f7afedff700 1 -- 172.21.15.27:0/733063271 >> 172.21.15.27:0/733063271 conn(0x7f7af80680e0 msgr2=0x7f7af8069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.260+0000 7f7afedff700 1 -- 172.21.15.27:0/733063271 shutdown_connections 2024-01-11T08:11:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.260+0000 7f7afedff700 1 -- 172.21.15.27:0/733063271 wait complete. 2024-01-11T08:11:10.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.261+0000 7f7afedff700 1 Processor -- start 2024-01-11T08:11:10.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.261+0000 7f7afedff700 1 -- start start 2024-01-11T08:11:10.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.262+0000 7f7afedff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7af8069d40 0x7f7af8153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:10.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.262+0000 7f7afedff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af80c8620 0x7f7af80a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:10.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.262+0000 7f7afedff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af80c89f0 0x7f7af8155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:10.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.262+0000 7f7afedff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af80c6130 con 0x7f7af80c89f0 2024-01-11T08:11:10.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.262+0000 7f7afedff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af80c5520 con 0x7f7af80c8620 2024-01-11T08:11:10.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.262+0000 7f7afedff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af80c5800 con 0x7f7af8069d40 2024-01-11T08:11:10.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.263+0000 7f7afd39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af80c89f0 0x7f7af8155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:10.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.263+0000 7f7afd39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af80c89f0 0x7f7af8155f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44272/0 (socket says 172.21.15.27:44272) 2024-01-11T08:11:10.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.263+0000 7f7afd39c700 1 -- 172.21.15.27:0/2829727684 learned_addr learned my addr 172.21.15.27:0/2829727684 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:10.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.263+0000 7f7af7fff700 1 --2- 172.21.15.27:0/2829727684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af80c8620 0x7f7af80a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:10.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.263+0000 7f7afcb9b700 1 --2- 172.21.15.27:0/2829727684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7af8069d40 0x7f7af8153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:10.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.263+0000 7f7afd39c700 1 -- 172.21.15.27:0/2829727684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7af8069d40 msgr2=0x7f7af8153140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:10.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.263+0000 7f7afd39c700 1 --2- 172.21.15.27:0/2829727684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7af8069d40 0x7f7af8153140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.263+0000 7f7afd39c700 1 -- 172.21.15.27:0/2829727684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af80c8620 msgr2=0x7f7af80a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:10.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.263+0000 7f7afd39c700 1 --2- 172.21.15.27:0/2829727684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af80c8620 0x7f7af80a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.263+0000 7f7afd39c700 1 -- 172.21.15.27:0/2829727684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae4009420 con 0x7f7af80c89f0 2024-01-11T08:11:10.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.263+0000 7f7afcb9b700 1 --2- 172.21.15.27:0/2829727684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7af8069d40 0x7f7af8153140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:10.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.263+0000 7f7afd39c700 1 --2- 172.21.15.27:0/2829727684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af80c89f0 0x7f7af8155f60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7af000cbc0 tx=0x7f7af0001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:10.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.264+0000 7f7af4ff9700 1 -- 172.21.15.27:0/2829727684 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af000a890 con 0x7f7af80c89f0 2024-01-11T08:11:10.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.264+0000 7f7af4ff9700 1 -- 172.21.15.27:0/2829727684 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af000a9f0 con 0x7f7af80c89f0 2024-01-11T08:11:10.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.264+0000 7f7afedff700 1 -- 172.21.15.27:0/2829727684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af80c5ae0 con 0x7f7af80c89f0 2024-01-11T08:11:10.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.264+0000 7f7af4ff9700 1 -- 172.21.15.27:0/2829727684 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af000adb0 con 0x7f7af80c89f0 2024-01-11T08:11:10.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.264+0000 7f7afedff700 1 -- 172.21.15.27:0/2829727684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af8153680 con 0x7f7af80c89f0 2024-01-11T08:11:10.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.265+0000 7f7afedff700 1 -- 172.21.15.27:0/2829727684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af80c62a0 con 0x7f7af80c89f0 2024-01-11T08:11:10.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.266+0000 7f7af4ff9700 1 -- 172.21.15.27:0/2829727684 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7af0007300 con 0x7f7af80c89f0 2024-01-11T08:11:10.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.267+0000 7f7af4ff9700 1 --2- 172.21.15.27:0/2829727684 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7adc066f30 0x7f7adc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:10.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.267+0000 7f7af4ff9700 1 -- 172.21.15.27:0/2829727684 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f7af0084820 con 0x7f7af80c89f0 2024-01-11T08:11:10.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.267+0000 7f7afcb9b700 1 --2- 172.21.15.27:0/2829727684 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7adc066f30 0x7f7adc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:10.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.268+0000 7f7afcb9b700 1 --2- 172.21.15.27:0/2829727684 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7adc066f30 0x7f7adc0693e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7aec006fd0 tx=0x7f7aec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:10.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.271+0000 7f7af4ff9700 1 -- 172.21.15.27:0/2829727684 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7af0051b10 con 0x7f7af80c89f0 2024-01-11T08:11:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.518+0000 7f7afedff700 1 -- 172.21.15.27:0/2829727684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7af80718d0 con 0x7f7af80c89f0 2024-01-11T08:11:10.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.518+0000 7f7af4ff9700 1 -- 172.21.15.27:0/2829727684 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4588 (secure 0 0 0) 0x7f7af0055130 con 0x7f7af80c89f0 2024-01-11T08:11:10.518 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:10.518 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3168764915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3168764915},{"type":"v1","addr":"172.21.15.27:6834","nonce":3168764915}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2607167395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2607167395},{"type":"v1","addr":"172.21.15.27:6835","nonce":2607167395}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2123423970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2123423970},{"type":"v1","addr":"172.21.15.77:6837","nonce":2123423970}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:10:30.160827+0000","modified":"2024-01-11T08:10:33.681210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6836/269464429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":269464429},{"type":"v1","addr":"172.21.15.77:6836","nonce":269464429}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2024-01-11T08:11:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.521+0000 7f7afedff700 1 -- 172.21.15.27:0/2829727684 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7adc066f30 msgr2=0x7f7adc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.521+0000 7f7afedff700 1 --2- 172.21.15.27:0/2829727684 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7adc066f30 0x7f7adc0693e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7aec006fd0 tx=0x7f7aec008040 comp rx=0 tx=0).stop 2024-01-11T08:11:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.521+0000 7f7afedff700 1 -- 172.21.15.27:0/2829727684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af80c89f0 msgr2=0x7f7af8155f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.521+0000 7f7afedff700 1 --2- 172.21.15.27:0/2829727684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af80c89f0 0x7f7af8155f60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7af000cbc0 tx=0x7f7af0001980 comp rx=0 tx=0).stop 2024-01-11T08:11:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.521+0000 7f7afedff700 1 -- 172.21.15.27:0/2829727684 shutdown_connections 2024-01-11T08:11:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.521+0000 7f7afedff700 1 --2- 172.21.15.27:0/2829727684 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7adc066f30 0x7f7adc0693e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.521+0000 7f7afedff700 1 --2- 172.21.15.27:0/2829727684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af80c89f0 0x7f7af8155f60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.521+0000 7f7afedff700 1 --2- 172.21.15.27:0/2829727684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7af8069d40 0x7f7af8153140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.521+0000 7f7afedff700 1 --2- 172.21.15.27:0/2829727684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af80c8620 0x7f7af80a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.521+0000 7f7afedff700 1 -- 172.21.15.27:0/2829727684 >> 172.21.15.27:0/2829727684 conn(0x7f7af80680e0 msgr2=0x7f7af80cb110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.521+0000 7f7afedff700 1 -- 172.21.15.27:0/2829727684 shutdown_connections 2024-01-11T08:11:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.521+0000 7f7afedff700 1 -- 172.21.15.27:0/2829727684 wait complete. 2024-01-11T08:11:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 8 2024-01-11T08:11:10.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:11:10.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.737+0000 7f99c25ff700 1 Processor -- start 2024-01-11T08:11:10.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.738+0000 7f99c25ff700 1 -- start start 2024-01-11T08:11:10.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.739+0000 7f99c25ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f99bc1599e0 0x7f99bc0b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:10.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.739+0000 7f99c25ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f99bc150330 0x7f99bc15a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:10.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.739+0000 7f99c25ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99bc157e30 con 0x7f99bc159db0 2024-01-11T08:11:10.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.739+0000 7f99c25ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99bc157500 con 0x7f99bc150330 2024-01-11T08:11:10.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.739+0000 7f99c25ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99bc1577e0 con 0x7f99bc1599e0 2024-01-11T08:11:10.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.739+0000 7f99c0b9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f99bc150330 0x7f99bc15a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:10.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.739+0000 7f99c0b9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f99bc150330 0x7f99bc15a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44286/0 (socket says 172.21.15.27:44286) 2024-01-11T08:11:10.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.739+0000 7f99c0b9c700 1 -- 172.21.15.27:0/1008632634 learned_addr learned my addr 172.21.15.27:0/1008632634 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:10.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.740+0000 7f99bbfff700 1 --2- 172.21.15.27:0/1008632634 >> v2:172.21.15.77:3300/0 conn(0x7f99bc1599e0 0x7f99bc0b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:10.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.740+0000 7f99c0b9c700 1 -- 172.21.15.27:0/1008632634 >> v2:172.21.15.77:3300/0 conn(0x7f99bc1599e0 msgr2=0x7f99bc0b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:10.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.740+0000 7f99c0b9c700 1 --2- 172.21.15.27:0/1008632634 >> v2:172.21.15.77:3300/0 conn(0x7f99bc1599e0 0x7f99bc0b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.740+0000 7f99c0b9c700 1 -- 172.21.15.27:0/1008632634 >> v1:172.21.15.77:6789/0 conn(0x7f99bc159db0 legacy=0x7f99bc14fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:10.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.740+0000 7f99c0b9c700 1 -- 172.21.15.27:0/1008632634 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99bc157ac0 con 0x7f99bc150330 2024-01-11T08:11:10.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.740+0000 7f99c0b9c700 1 --2- 172.21.15.27:0/1008632634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99bc150330 0x7f99bc15a230 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f99b400d1a0 tx=0x7f99b400ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcdc13dcf5cb47eb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:10.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.740+0000 7f99baffd700 1 -- 172.21.15.27:0/1008632634 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b4010070 con 0x7f99bc150330 2024-01-11T08:11:10.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.740+0000 7f99baffd700 1 -- 172.21.15.27:0/1008632634 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b400de40 con 0x7f99bc150330 2024-01-11T08:11:10.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.740+0000 7f99baffd700 1 -- 172.21.15.27:0/1008632634 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b400b320 con 0x7f99bc150330 2024-01-11T08:11:10.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.741+0000 7f99c25ff700 1 -- 172.21.15.27:0/1008632634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99bc150330 msgr2=0x7f99bc15a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:10.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.741+0000 7f99c25ff700 1 --2- 172.21.15.27:0/1008632634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99bc150330 0x7f99bc15a230 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f99b400d1a0 tx=0x7f99b400ba50 comp rx=0 tx=0).stop 2024-01-11T08:11:10.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.741+0000 7f99c25ff700 1 -- 172.21.15.27:0/1008632634 shutdown_connections 2024-01-11T08:11:10.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.741+0000 7f99c25ff700 1 --2- 172.21.15.27:0/1008632634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99bc150330 0x7f99bc15a230 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.741+0000 7f99c25ff700 1 --2- 172.21.15.27:0/1008632634 >> v2:172.21.15.77:3300/0 conn(0x7f99bc1599e0 0x7f99bc0b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.741+0000 7f99c25ff700 1 -- 172.21.15.27:0/1008632634 >> 172.21.15.27:0/1008632634 conn(0x7f99bc0680e0 msgr2=0x7f99bc069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:10.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.741+0000 7f99c25ff700 1 -- 172.21.15.27:0/1008632634 shutdown_connections 2024-01-11T08:11:10.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.741+0000 7f99c25ff700 1 -- 172.21.15.27:0/1008632634 wait complete. 2024-01-11T08:11:10.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.743+0000 7f99c25ff700 1 Processor -- start 2024-01-11T08:11:10.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.743+0000 7f99c25ff700 1 -- start start 2024-01-11T08:11:10.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.744+0000 7f99c25ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99bc150330 0x7f99bc0ae130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:10.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.744+0000 7f99c25ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99bc1599e0 0x7f99bc0a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:10.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.744+0000 7f99c25ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99bc159db0 0x7f99bc149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.744+0000 7f99c25ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc0ad0f0 con 0x7f99bc1599e0 2024-01-11T08:11:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.744+0000 7f99c25ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc0aab20 con 0x7f99bc150330 2024-01-11T08:11:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.744+0000 7f99c25ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc0a9f10 con 0x7f99bc159db0 2024-01-11T08:11:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.745+0000 7f99bb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99bc1599e0 0x7f99bc0a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.745+0000 7f99bbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99bc150330 0x7f99bc0ae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.745+0000 7f99bbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99bc150330 0x7f99bc0ae130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:48660/0 (socket says 172.21.15.27:48660) 2024-01-11T08:11:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.745+0000 7f99bbfff700 1 -- 172.21.15.27:0/3509825172 learned_addr learned my addr 172.21.15.27:0/3509825172 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.745+0000 7f99c0b9c700 1 --2- 172.21.15.27:0/3509825172 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99bc159db0 0x7f99bc149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.745+0000 7f99bb7fe700 1 -- 172.21.15.27:0/3509825172 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99bc159db0 msgr2=0x7f99bc149df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.745+0000 7f99bb7fe700 1 --2- 172.21.15.27:0/3509825172 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99bc159db0 0x7f99bc149df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.745+0000 7f99bb7fe700 1 -- 172.21.15.27:0/3509825172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99bc150330 msgr2=0x7f99bc0ae130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.745+0000 7f99bb7fe700 1 --2- 172.21.15.27:0/3509825172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99bc150330 0x7f99bc0ae130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.745+0000 7f99bb7fe700 1 -- 172.21.15.27:0/3509825172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b400bcf0 con 0x7f99bc1599e0 2024-01-11T08:11:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.745+0000 7f99bbfff700 1 --2- 172.21.15.27:0/3509825172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99bc150330 0x7f99bc0ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:10.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.745+0000 7f99bb7fe700 1 --2- 172.21.15.27:0/3509825172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99bc1599e0 0x7f99bc0a98a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f99b000b4c0 tx=0x7f99b0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:10.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.746+0000 7f99a3fff700 1 -- 172.21.15.27:0/3509825172 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b000be70 con 0x7f99bc1599e0 2024-01-11T08:11:10.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.746+0000 7f99a3fff700 1 -- 172.21.15.27:0/3509825172 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b000f510 con 0x7f99bc1599e0 2024-01-11T08:11:10.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.746+0000 7f99c25ff700 1 -- 172.21.15.27:0/3509825172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99bc0aa1f0 con 0x7f99bc1599e0 2024-01-11T08:11:10.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.746+0000 7f99a3fff700 1 -- 172.21.15.27:0/3509825172 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b000f8d0 con 0x7f99bc1599e0 2024-01-11T08:11:10.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.746+0000 7f99c25ff700 1 -- 172.21.15.27:0/3509825172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99bc0aa7b0 con 0x7f99bc1599e0 2024-01-11T08:11:10.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.747+0000 7f99c25ff700 1 -- 172.21.15.27:0/3509825172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99bc0b0ee0 con 0x7f99bc1599e0 2024-01-11T08:11:10.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.748+0000 7f99a3fff700 1 -- 172.21.15.27:0/3509825172 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f99b000fcc0 con 0x7f99bc1599e0 2024-01-11T08:11:10.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.749+0000 7f99a3fff700 1 --2- 172.21.15.27:0/3509825172 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f99a8066f30 0x7f99a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:10.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.749+0000 7f99a3fff700 1 -- 172.21.15.27:0/3509825172 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f99b0083e60 con 0x7f99bc1599e0 2024-01-11T08:11:10.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.749+0000 7f99bbfff700 1 --2- 172.21.15.27:0/3509825172 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f99a8066f30 0x7f99a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:10.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.750+0000 7f99bbfff700 1 --2- 172.21.15.27:0/3509825172 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f99a8066f30 0x7f99a80693e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f99ac006fd0 tx=0x7f99ac008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:10.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.753+0000 7f99a3fff700 1 -- 172.21.15.27:0/3509825172 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f99b0051150 con 0x7f99bc1599e0 2024-01-11T08:11:10.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.959+0000 7f99c25ff700 1 -- 172.21.15.27:0/3509825172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99bc0aac90 con 0x7f99bc1599e0 2024-01-11T08:11:10.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.960+0000 7f99a3fff700 1 -- 172.21.15.27:0/3509825172 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+15302 (secure 0 0 0) 0x7f99b0054770 con 0x7f99bc1599e0 2024-01-11T08:11:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:10.961 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:10:31.464323+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:10:27.993251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-11T08:10:29.153959+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:11:10.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.963+0000 7f99c25ff700 1 -- 172.21.15.27:0/3509825172 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f99a8066f30 msgr2=0x7f99a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:10.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.963+0000 7f99c25ff700 1 --2- 172.21.15.27:0/3509825172 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f99a8066f30 0x7f99a80693e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f99ac006fd0 tx=0x7f99ac008040 comp rx=0 tx=0).stop 2024-01-11T08:11:10.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.963+0000 7f99c25ff700 1 -- 172.21.15.27:0/3509825172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99bc1599e0 msgr2=0x7f99bc0a98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:10.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.963+0000 7f99c25ff700 1 --2- 172.21.15.27:0/3509825172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99bc1599e0 0x7f99bc0a98a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f99b000b4c0 tx=0x7f99b0009060 comp rx=0 tx=0).stop 2024-01-11T08:11:10.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.963+0000 7f99c25ff700 1 -- 172.21.15.27:0/3509825172 shutdown_connections 2024-01-11T08:11:10.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.963+0000 7f99c25ff700 1 --2- 172.21.15.27:0/3509825172 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f99a8066f30 0x7f99a80693e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.963+0000 7f99c25ff700 1 --2- 172.21.15.27:0/3509825172 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99bc159db0 0x7f99bc149df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.963+0000 7f99c25ff700 1 --2- 172.21.15.27:0/3509825172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99bc150330 0x7f99bc0ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.963+0000 7f99c25ff700 1 --2- 172.21.15.27:0/3509825172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99bc1599e0 0x7f99bc0a98a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:10.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.963+0000 7f99c25ff700 1 -- 172.21.15.27:0/3509825172 >> 172.21.15.27:0/3509825172 conn(0x7f99bc0680e0 msgr2=0x7f99bc0cbe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:10.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.963+0000 7f99c25ff700 1 -- 172.21.15.27:0/3509825172 shutdown_connections 2024-01-11T08:11:10.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:10.963+0000 7f99c25ff700 1 -- 172.21.15.27:0/3509825172 wait complete. 2024-01-11T08:11:10.982 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-11T08:11:10.982 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:11:11.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.181+0000 7faf67bff700 1 Processor -- start 2024-01-11T08:11:11.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.181+0000 7faf67bff700 1 -- start start 2024-01-11T08:11:11.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.182+0000 7faf67bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faf600b03e0 0x7faf600b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:11.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.182+0000 7faf67bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faf600b07b0 0x7faf600b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:11.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.182+0000 7faf67bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faf60159200 0x7faf60159640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:11.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.182+0000 7faf67bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf600cd730 con 0x7faf600b03e0 2024-01-11T08:11:11.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.182+0000 7faf67bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf600ccb20 con 0x7faf60159200 2024-01-11T08:11:11.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.182+0000 7faf67bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf600cce00 con 0x7faf600b07b0 2024-01-11T08:11:11.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.183+0000 7faf6599b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faf600b03e0 0x7faf600b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:11.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.183+0000 7faf6599b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faf600b03e0 0x7faf600b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44312/0 (socket says 172.21.15.27:44312) 2024-01-11T08:11:11.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.183+0000 7faf6599b700 1 -- 172.21.15.27:0/3059450445 learned_addr learned my addr 172.21.15.27:0/3059450445 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:11.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.183+0000 7faf6619c700 1 --2- 172.21.15.27:0/3059450445 >> v2:172.21.15.77:3300/0 conn(0x7faf60159200 0x7faf60159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:11.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.183+0000 7faf6519a700 1 --2- 172.21.15.27:0/3059450445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faf600b07b0 0x7faf600b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:11.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.183+0000 7faf6599b700 1 -- 172.21.15.27:0/3059450445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faf600b07b0 msgr2=0x7faf600b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:11.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.183+0000 7faf6599b700 1 --2- 172.21.15.27:0/3059450445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faf600b07b0 0x7faf600b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.183+0000 7faf6599b700 1 -- 172.21.15.27:0/3059450445 >> v2:172.21.15.77:3300/0 conn(0x7faf60159200 msgr2=0x7faf60159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:11.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.183+0000 7faf6599b700 1 --2- 172.21.15.27:0/3059450445 >> v2:172.21.15.77:3300/0 conn(0x7faf60159200 0x7faf60159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.183+0000 7faf6599b700 1 -- 172.21.15.27:0/3059450445 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf600cd0e0 con 0x7faf600b03e0 2024-01-11T08:11:11.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.183+0000 7faf6599b700 1 --2- 172.21.15.27:0/3059450445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf600b03e0 0x7faf600b2480 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7faf600b5f00 tx=0x7faf5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63386f5c8f4e3442 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:11.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.184+0000 7faf64999700 1 -- 172.21.15.27:0/3059450445 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c00b640 con 0x7faf600b03e0 2024-01-11T08:11:11.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.184+0000 7faf64999700 1 -- 172.21.15.27:0/3059450445 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5c00b7a0 con 0x7faf600b03e0 2024-01-11T08:11:11.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.184+0000 7faf64999700 1 -- 172.21.15.27:0/3059450445 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c00bb60 con 0x7faf600b03e0 2024-01-11T08:11:11.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.184+0000 7faf67bff700 1 -- 172.21.15.27:0/3059450445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf600b03e0 msgr2=0x7faf600b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:11.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.184+0000 7faf67bff700 1 --2- 172.21.15.27:0/3059450445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf600b03e0 0x7faf600b2480 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7faf600b5f00 tx=0x7faf5c009250 comp rx=0 tx=0).stop 2024-01-11T08:11:11.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.184+0000 7faf67bff700 1 -- 172.21.15.27:0/3059450445 shutdown_connections 2024-01-11T08:11:11.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.184+0000 7faf67bff700 1 --2- 172.21.15.27:0/3059450445 >> v2:172.21.15.77:3300/0 conn(0x7faf60159200 0x7faf60159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.184+0000 7faf67bff700 1 --2- 172.21.15.27:0/3059450445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf600b03e0 0x7faf600b2480 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.184+0000 7faf67bff700 1 --2- 172.21.15.27:0/3059450445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faf600b07b0 0x7faf600b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.184+0000 7faf67bff700 1 -- 172.21.15.27:0/3059450445 >> 172.21.15.27:0/3059450445 conn(0x7faf6006ed40 msgr2=0x7faf6006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:11.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.184+0000 7faf67bff700 1 -- 172.21.15.27:0/3059450445 shutdown_connections 2024-01-11T08:11:11.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.184+0000 7faf67bff700 1 -- 172.21.15.27:0/3059450445 wait complete. 2024-01-11T08:11:11.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.186+0000 7faf67bff700 1 Processor -- start 2024-01-11T08:11:11.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.186+0000 7faf67bff700 1 -- start start 2024-01-11T08:11:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.187+0000 7faf67bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf600b03e0 0x7faf60157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.187+0000 7faf67bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faf600b07b0 0x7faf60159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.187+0000 7faf67bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf60159200 0x7faf600af2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.187+0000 7faf67bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6015c7f0 con 0x7faf600b03e0 2024-01-11T08:11:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.187+0000 7faf67bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf60159b80 con 0x7faf60159200 2024-01-11T08:11:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.187+0000 7faf67bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6015a6a0 con 0x7faf600b07b0 2024-01-11T08:11:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.187+0000 7faf6599b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf600b03e0 0x7faf60157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.187+0000 7faf6599b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf600b03e0 0x7faf60157bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44320/0 (socket says 172.21.15.27:44320) 2024-01-11T08:11:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.187+0000 7faf6599b700 1 -- 172.21.15.27:0/2457317505 learned_addr learned my addr 172.21.15.27:0/2457317505 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.187+0000 7faf6619c700 1 --2- 172.21.15.27:0/2457317505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf60159200 0x7faf600af2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.187+0000 7faf6519a700 1 --2- 172.21.15.27:0/2457317505 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faf600b07b0 0x7faf60159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.188+0000 7faf6599b700 1 -- 172.21.15.27:0/2457317505 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faf600b07b0 msgr2=0x7faf60159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.188+0000 7faf6599b700 1 --2- 172.21.15.27:0/2457317505 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faf600b07b0 0x7faf60159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.188+0000 7faf6599b700 1 -- 172.21.15.27:0/2457317505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf60159200 msgr2=0x7faf600af2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:11.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.188+0000 7faf6599b700 1 --2- 172.21.15.27:0/2457317505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf60159200 0x7faf600af2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.188+0000 7faf6599b700 1 -- 172.21.15.27:0/2457317505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf5c0094f0 con 0x7faf600b03e0 2024-01-11T08:11:11.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.188+0000 7faf6599b700 1 --2- 172.21.15.27:0/2457317505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf600b03e0 0x7faf60157bc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7faf5c006fd0 tx=0x7faf5c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.188+0000 7faf55ffb700 1 -- 172.21.15.27:0/2457317505 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c000bc0 con 0x7faf600b03e0 2024-01-11T08:11:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.188+0000 7faf55ffb700 1 -- 172.21.15.27:0/2457317505 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5c00b910 con 0x7faf600b03e0 2024-01-11T08:11:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.188+0000 7faf67bff700 1 -- 172.21.15.27:0/2457317505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6015a3c0 con 0x7faf600b03e0 2024-01-11T08:11:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.188+0000 7faf55ffb700 1 -- 172.21.15.27:0/2457317505 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c000d20 con 0x7faf600b03e0 2024-01-11T08:11:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.188+0000 7faf67bff700 1 -- 172.21.15.27:0/2457317505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf600aeb50 con 0x7faf600b03e0 2024-01-11T08:11:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.189+0000 7faf67bff700 1 -- 172.21.15.27:0/2457317505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf6015c620 con 0x7faf600b03e0 2024-01-11T08:11:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.190+0000 7faf55ffb700 1 -- 172.21.15.27:0/2457317505 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7faf5c023030 con 0x7faf600b03e0 2024-01-11T08:11:11.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.191+0000 7faf55ffb700 1 --2- 172.21.15.27:0/2457317505 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7faf44066f30 0x7faf440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:11.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.191+0000 7faf55ffb700 1 -- 172.21.15.27:0/2457317505 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5705+0+0 (secure 0 0 0) 0x7faf5c085bc0 con 0x7faf600b03e0 2024-01-11T08:11:11.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.192+0000 7faf6519a700 1 --2- 172.21.15.27:0/2457317505 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7faf44066f30 0x7faf440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:11.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.192+0000 7faf6519a700 1 --2- 172.21.15.27:0/2457317505 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7faf44066f30 0x7faf440693e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faf600af000 tx=0x7faf500091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:11.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.195+0000 7faf55ffb700 1 -- 172.21.15.27:0/2457317505 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faf5c052e10 con 0x7faf600b03e0 2024-01-11T08:11:11.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.439+0000 7faf67bff700 1 -- 172.21.15.27:0/2457317505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf600718d0 con 0x7faf600b03e0 2024-01-11T08:11:11.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.439+0000 7faf55ffb700 1 -- 172.21.15.27:0/2457317505 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4588 (secure 0 0 0) 0x7faf5c02e3a0 con 0x7faf600b03e0 2024-01-11T08:11:11.440 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:11.441 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3168764915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3168764915},{"type":"v1","addr":"172.21.15.27:6834","nonce":3168764915}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2607167395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2607167395},{"type":"v1","addr":"172.21.15.27:6835","nonce":2607167395}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2123423970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2123423970},{"type":"v1","addr":"172.21.15.77:6837","nonce":2123423970}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:10:30.160827+0000","modified":"2024-01-11T08:10:33.681210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6836/269464429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":269464429},{"type":"v1","addr":"172.21.15.77:6836","nonce":269464429}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2024-01-11T08:11:11.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.442+0000 7faf67bff700 1 -- 172.21.15.27:0/2457317505 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7faf44066f30 msgr2=0x7faf440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:11.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.442+0000 7faf67bff700 1 --2- 172.21.15.27:0/2457317505 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7faf44066f30 0x7faf440693e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faf600af000 tx=0x7faf500091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.442+0000 7faf67bff700 1 -- 172.21.15.27:0/2457317505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf600b03e0 msgr2=0x7faf60157bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:11.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.442+0000 7faf67bff700 1 --2- 172.21.15.27:0/2457317505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf600b03e0 0x7faf60157bc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7faf5c006fd0 tx=0x7faf5c017000 comp rx=0 tx=0).stop 2024-01-11T08:11:11.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.442+0000 7faf67bff700 1 -- 172.21.15.27:0/2457317505 shutdown_connections 2024-01-11T08:11:11.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.442+0000 7faf67bff700 1 --2- 172.21.15.27:0/2457317505 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7faf44066f30 0x7faf440693e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.442+0000 7faf67bff700 1 --2- 172.21.15.27:0/2457317505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf60159200 0x7faf600af2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.442+0000 7faf67bff700 1 --2- 172.21.15.27:0/2457317505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf600b03e0 0x7faf60157bc0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.442+0000 7faf67bff700 1 --2- 172.21.15.27:0/2457317505 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faf600b07b0 0x7faf60159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.442+0000 7faf67bff700 1 -- 172.21.15.27:0/2457317505 >> 172.21.15.27:0/2457317505 conn(0x7faf6006ed40 msgr2=0x7faf600d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.442+0000 7faf67bff700 1 -- 172.21.15.27:0/2457317505 shutdown_connections 2024-01-11T08:11:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.442+0000 7faf67bff700 1 -- 172.21.15.27:0/2457317505 wait complete. 2024-01-11T08:11:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 8 2024-01-11T08:11:11.461 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:11:11.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.665+0000 7fe8b4bff700 1 Processor -- start 2024-01-11T08:11:11.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.666+0000 7fe8b4bff700 1 -- start start 2024-01-11T08:11:11.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.667+0000 7fe8b4bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe8b01596c0 0x7fe8b015d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:11.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.667+0000 7fe8b4bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b00a9cb0 con 0x7fe8b0068d30 2024-01-11T08:11:11.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.667+0000 7fe8b4bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b015e210 con 0x7fe8b00ca670 2024-01-11T08:11:11.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.667+0000 7fe8b4bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8b015dc50 con 0x7fe8b01596c0 2024-01-11T08:11:11.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.667+0000 7fe8aed9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe8b0068d30 0x7fe8b00aa140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50604/0 (socket says 172.21.15.27:50604) 2024-01-11T08:11:11.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.667+0000 7fe8aed9d700 1 -- 172.21.15.27:0/1718500424 learned_addr learned my addr 172.21.15.27:0/1718500424 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:11.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.668+0000 7fe8ae59c700 1 --2- 172.21.15.27:0/1718500424 >> v2:172.21.15.77:3300/0 conn(0x7fe8b01596c0 0x7fe8b015d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:11.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.668+0000 7fe8ad59a700 1 -- 172.21.15.27:0/1718500424 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113335223 0 0) 0x7fe8b00a9cb0 con 0x7fe8b0068d30 2024-01-11T08:11:11.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.668+0000 7fe8ad59a700 1 -- 172.21.15.27:0/1718500424 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe898003570 con 0x7fe8b0068d30 2024-01-11T08:11:11.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.668+0000 7fe8ae59c700 1 -- 172.21.15.27:0/1718500424 >> v1:172.21.15.77:6789/0 conn(0x7fe8b00ca670 legacy=0x7fe8b0068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:11.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.668+0000 7fe8ad59a700 1 -- 172.21.15.27:0/1718500424 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256113016 0 0) 0x7fe8b015e210 con 0x7fe8b00ca670 2024-01-11T08:11:11.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.668+0000 7fe8ae59c700 1 -- 172.21.15.27:0/1718500424 >> v1:172.21.15.27:6789/0 conn(0x7fe8b0068d30 legacy=0x7fe8b00aa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:11.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.668+0000 7fe8ae59c700 1 -- 172.21.15.27:0/1718500424 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b015df30 con 0x7fe8b01596c0 2024-01-11T08:11:11.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.669+0000 7fe8ae59c700 1 --2- 172.21.15.27:0/1718500424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b01596c0 0x7fe8b015d300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe8a00098e0 tx=0x7fe8a00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37b2336049d14d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:11.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.669+0000 7fe8ad59a700 1 -- 172.21.15.27:0/1718500424 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a80017f0 con 0x7fe8b01596c0 2024-01-11T08:11:11.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.669+0000 7fe8ad59a700 1 -- 172.21.15.27:0/1718500424 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8a0011420 con 0x7fe8b01596c0 2024-01-11T08:11:11.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.669+0000 7fe8ad59a700 1 -- 172.21.15.27:0/1718500424 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a00117c0 con 0x7fe8b01596c0 2024-01-11T08:11:11.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.669+0000 7fe8b4bff700 1 -- 172.21.15.27:0/1718500424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b01596c0 msgr2=0x7fe8b015d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:11.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.669+0000 7fe8b4bff700 1 --2- 172.21.15.27:0/1718500424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b01596c0 0x7fe8b015d300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe8a00098e0 tx=0x7fe8a00091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.669+0000 7fe8b4bff700 1 -- 172.21.15.27:0/1718500424 shutdown_connections 2024-01-11T08:11:11.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.669+0000 7fe8b4bff700 1 --2- 172.21.15.27:0/1718500424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b01596c0 0x7fe8b015d300 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.669+0000 7fe8b4bff700 1 -- 172.21.15.27:0/1718500424 >> 172.21.15.27:0/1718500424 conn(0x7fe8b00c9f70 msgr2=0x7fe8b006deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:11.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.670+0000 7fe8b4bff700 1 -- 172.21.15.27:0/1718500424 shutdown_connections 2024-01-11T08:11:11.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.670+0000 7fe8b4bff700 1 -- 172.21.15.27:0/1718500424 wait complete. 2024-01-11T08:11:11.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.671+0000 7fe8b4bff700 1 Processor -- start 2024-01-11T08:11:11.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.671+0000 7fe8b4bff700 1 -- start start 2024-01-11T08:11:11.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.672+0000 7fe8b4bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8b0068d30 0x7fe8b0155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:11.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.672+0000 7fe8b4bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8b00ca670 0x7fe8b00b0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:11.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.672+0000 7fe8b4bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b01596c0 0x7fe8b006a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:11.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.672+0000 7fe8b4bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b01560b0 con 0x7fe8b0068d30 2024-01-11T08:11:11.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.672+0000 7fe8b4bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b00af930 con 0x7fe8b01596c0 2024-01-11T08:11:11.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.672+0000 7fe8b4bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b00b01d0 con 0x7fe8b00ca670 2024-01-11T08:11:11.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.672+0000 7fe8ae59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8b0068d30 0x7fe8b0155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:11.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.672+0000 7fe8ae59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8b0068d30 0x7fe8b0155b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44334/0 (socket says 172.21.15.27:44334) 2024-01-11T08:11:11.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.672+0000 7fe8ae59c700 1 -- 172.21.15.27:0/406986914 learned_addr learned my addr 172.21.15.27:0/406986914 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:11.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.672+0000 7fe8aed9d700 1 --2- 172.21.15.27:0/406986914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b01596c0 0x7fe8b006a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:11.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.672+0000 7fe8add9b700 1 --2- 172.21.15.27:0/406986914 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8b00ca670 0x7fe8b00b0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:11.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.673+0000 7fe8ae59c700 1 -- 172.21.15.27:0/406986914 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8b00ca670 msgr2=0x7fe8b00b0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:11.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.673+0000 7fe8ae59c700 1 --2- 172.21.15.27:0/406986914 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8b00ca670 0x7fe8b00b0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.673+0000 7fe8ae59c700 1 -- 172.21.15.27:0/406986914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b01596c0 msgr2=0x7fe8b006a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:11.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.673+0000 7fe8ae59c700 1 --2- 172.21.15.27:0/406986914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b01596c0 0x7fe8b006a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.673+0000 7fe8ae59c700 1 -- 172.21.15.27:0/406986914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a0009420 con 0x7fe8b0068d30 2024-01-11T08:11:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.673+0000 7fe8aed9d700 1 --2- 172.21.15.27:0/406986914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b01596c0 0x7fe8b006a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.673+0000 7fe8ae59c700 1 --2- 172.21.15.27:0/406986914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8b0068d30 0x7fe8b0155b70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe8b006d490 tx=0x7fe8a0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.673+0000 7fe89e7fc700 1 -- 172.21.15.27:0/406986914 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a0010b30 con 0x7fe8b0068d30 2024-01-11T08:11:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.673+0000 7fe89e7fc700 1 -- 172.21.15.27:0/406986914 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8a0010c90 con 0x7fe8b0068d30 2024-01-11T08:11:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.673+0000 7fe89e7fc700 1 -- 172.21.15.27:0/406986914 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a001d660 con 0x7fe8b0068d30 2024-01-11T08:11:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.673+0000 7fe8b4bff700 1 -- 172.21.15.27:0/406986914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b00c7e60 con 0x7fe8b0068d30 2024-01-11T08:11:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.674+0000 7fe8b4bff700 1 -- 172.21.15.27:0/406986914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b00afe60 con 0x7fe8b0068d30 2024-01-11T08:11:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.675+0000 7fe8b4bff700 1 -- 172.21.15.27:0/406986914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b00b04b0 con 0x7fe8b0068d30 2024-01-11T08:11:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.675+0000 7fe89e7fc700 1 -- 172.21.15.27:0/406986914 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe8a001d9d0 con 0x7fe8b0068d30 2024-01-11T08:11:11.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.677+0000 7fe89e7fc700 1 --2- 172.21.15.27:0/406986914 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe890066f30 0x7fe8900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:11.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.678+0000 7fe89e7fc700 1 -- 172.21.15.27:0/406986914 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fe8a0086480 con 0x7fe8b0068d30 2024-01-11T08:11:11.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.678+0000 7fe8add9b700 1 --2- 172.21.15.27:0/406986914 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe890066f30 0x7fe8900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:11.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.679+0000 7fe8add9b700 1 --2- 172.21.15.27:0/406986914 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe890066f30 0x7fe8900693e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe8b00aebb0 tx=0x7fe8a4007ce0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:11.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.681+0000 7fe89e7fc700 1 -- 172.21.15.27:0/406986914 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe8a00535e0 con 0x7fe8b0068d30 2024-01-11T08:11:11.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.930+0000 7fe8b4bff700 1 -- 172.21.15.27:0/406986914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8b00718d0 con 0x7fe8b0068d30 2024-01-11T08:11:11.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.930+0000 7fe89e7fc700 1 -- 172.21.15.27:0/406986914 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4588 (secure 0 0 0) 0x7fe8a0056c00 con 0x7fe8b0068d30 2024-01-11T08:11:11.931 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:11.931 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3168764915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3168764915},{"type":"v1","addr":"172.21.15.27:6834","nonce":3168764915}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2607167395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2607167395},{"type":"v1","addr":"172.21.15.27:6835","nonce":2607167395}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2123423970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2123423970},{"type":"v1","addr":"172.21.15.77:6837","nonce":2123423970}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:10:30.160827+0000","modified":"2024-01-11T08:10:33.681210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6836/269464429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":269464429},{"type":"v1","addr":"172.21.15.77:6836","nonce":269464429}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2024-01-11T08:11:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.933+0000 7fe8b4bff700 1 -- 172.21.15.27:0/406986914 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe890066f30 msgr2=0x7fe8900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.933+0000 7fe8b4bff700 1 --2- 172.21.15.27:0/406986914 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe890066f30 0x7fe8900693e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe8b00aebb0 tx=0x7fe8a4007ce0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.933+0000 7fe8b4bff700 1 -- 172.21.15.27:0/406986914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8b0068d30 msgr2=0x7fe8b0155b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.933+0000 7fe8b4bff700 1 --2- 172.21.15.27:0/406986914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8b0068d30 0x7fe8b0155b70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe8b006d490 tx=0x7fe8a0017000 comp rx=0 tx=0).stop 2024-01-11T08:11:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.933+0000 7fe8b4bff700 1 -- 172.21.15.27:0/406986914 shutdown_connections 2024-01-11T08:11:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.933+0000 7fe8b4bff700 1 --2- 172.21.15.27:0/406986914 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe890066f30 0x7fe8900693e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.933+0000 7fe8b4bff700 1 --2- 172.21.15.27:0/406986914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8b01596c0 0x7fe8b006a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.933+0000 7fe8b4bff700 1 --2- 172.21.15.27:0/406986914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8b0068d30 0x7fe8b0155b70 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.933+0000 7fe8b4bff700 1 --2- 172.21.15.27:0/406986914 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8b00ca670 0x7fe8b00b0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.933+0000 7fe8b4bff700 1 -- 172.21.15.27:0/406986914 >> 172.21.15.27:0/406986914 conn(0x7fe8b00c9f70 msgr2=0x7fe8b00c27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.933+0000 7fe8b4bff700 1 -- 172.21.15.27:0/406986914 shutdown_connections 2024-01-11T08:11:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:11.933+0000 7fe8b4bff700 1 -- 172.21.15.27:0/406986914 wait complete. 2024-01-11T08:11:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 8 2024-01-11T08:11:11.952 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:11:12.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.151+0000 7fb5457ff700 1 Processor -- start 2024-01-11T08:11:12.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.151+0000 7fb5457ff700 1 -- start start 2024-01-11T08:11:12.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.152+0000 7fb5457ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5400ca7b0 0x7fb5400cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:12.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.153+0000 7fb5457ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54015d640 con 0x7fb540159e80 2024-01-11T08:11:12.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.153+0000 7fb5457ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54015a5d0 con 0x7fb5400ca3e0 2024-01-11T08:11:12.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.153+0000 7fb5457ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb54015ae70 con 0x7fb5400ca7b0 2024-01-11T08:11:12.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.153+0000 7fb53e7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5400ca7b0 0x7fb5400cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:12.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.153+0000 7fb53e7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5400ca7b0 0x7fb5400cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44338/0 (socket says 172.21.15.27:44338) 2024-01-11T08:11:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.153+0000 7fb53e7fc700 1 -- 172.21.15.27:0/4112761839 learned_addr learned my addr 172.21.15.27:0/4112761839 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.153+0000 7fb53f7fe700 1 --1- 172.21.15.27:0/4112761839 >> v1:172.21.15.27:6789/0 conn(0x7fb540159e80 0x7fb54015dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50612/0 (socket says 172.21.15.27:50612) 2024-01-11T08:11:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.153+0000 7fb53e7fc700 1 -- 172.21.15.27:0/4112761839 >> v1:172.21.15.77:6789/0 conn(0x7fb5400ca3e0 legacy=0x7fb5400c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.153+0000 7fb53e7fc700 1 -- 172.21.15.27:0/4112761839 >> v1:172.21.15.27:6789/0 conn(0x7fb540159e80 legacy=0x7fb54015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.153+0000 7fb53e7fc700 1 -- 172.21.15.27:0/4112761839 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54015a8b0 con 0x7fb5400ca7b0 2024-01-11T08:11:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.153+0000 7fb53dffb700 1 -- 172.21.15.27:0/4112761839 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796553332 0 0) 0x7fb54015d640 con 0x7fb540159e80 2024-01-11T08:11:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.154+0000 7fb53e7fc700 1 --2- 172.21.15.27:0/4112761839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5400ca7b0 0x7fb5400cca20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb5340099e0 tx=0x7fb534009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac885245a18872d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.154+0000 7fb53dffb700 1 -- 172.21.15.27:0/4112761839 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53400b820 con 0x7fb5400ca7b0 2024-01-11T08:11:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.154+0000 7fb53dffb700 1 -- 172.21.15.27:0/4112761839 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53400b980 con 0x7fb5400ca7b0 2024-01-11T08:11:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.154+0000 7fb53dffb700 1 -- 172.21.15.27:0/4112761839 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53400bd20 con 0x7fb5400ca7b0 2024-01-11T08:11:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.154+0000 7fb5457ff700 1 -- 172.21.15.27:0/4112761839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5400ca7b0 msgr2=0x7fb5400cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.154+0000 7fb5457ff700 1 --2- 172.21.15.27:0/4112761839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5400ca7b0 0x7fb5400cca20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb5340099e0 tx=0x7fb534009250 comp rx=0 tx=0).stop 2024-01-11T08:11:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.155+0000 7fb5457ff700 1 -- 172.21.15.27:0/4112761839 shutdown_connections 2024-01-11T08:11:12.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.155+0000 7fb5457ff700 1 --2- 172.21.15.27:0/4112761839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5400ca7b0 0x7fb5400cca20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:12.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.155+0000 7fb5457ff700 1 -- 172.21.15.27:0/4112761839 >> 172.21.15.27:0/4112761839 conn(0x7fb5400ce3e0 msgr2=0x7fb5400ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:12.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.155+0000 7fb5457ff700 1 -- 172.21.15.27:0/4112761839 shutdown_connections 2024-01-11T08:11:12.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.155+0000 7fb5457ff700 1 -- 172.21.15.27:0/4112761839 wait complete. 2024-01-11T08:11:12.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.156+0000 7fb5457ff700 1 Processor -- start 2024-01-11T08:11:12.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.156+0000 7fb5457ff700 1 -- start start 2024-01-11T08:11:12.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.157+0000 7fb5457ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5400ca3e0 0x7fb5400c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:12.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.157+0000 7fb5457ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5400ca7b0 0x7fb540140da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:12.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.157+0000 7fb5457ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb540159e80 0x7fb5400c5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:12.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.157+0000 7fb5457ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5400c8730 con 0x7fb5400ca3e0 2024-01-11T08:11:12.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.157+0000 7fb5457ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54015db70 con 0x7fb5400ca7b0 2024-01-11T08:11:12.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.157+0000 7fb5457ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb54015de50 con 0x7fb540159e80 2024-01-11T08:11:12.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.158+0000 7fb53effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5400ca3e0 0x7fb5400c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:12.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.158+0000 7fb53effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5400ca3e0 0x7fb5400c6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44350/0 (socket says 172.21.15.27:44350) 2024-01-11T08:11:12.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.158+0000 7fb53effd700 1 -- 172.21.15.27:0/3010274950 learned_addr learned my addr 172.21.15.27:0/3010274950 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:12.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.158+0000 7fb53e7fc700 1 --2- 172.21.15.27:0/3010274950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5400ca7b0 0x7fb540140da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:12.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.158+0000 7fb53f7fe700 1 --2- 172.21.15.27:0/3010274950 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb540159e80 0x7fb5400c5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:12.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.158+0000 7fb53effd700 1 -- 172.21.15.27:0/3010274950 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb540159e80 msgr2=0x7fb5400c5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:12.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.158+0000 7fb53effd700 1 --2- 172.21.15.27:0/3010274950 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb540159e80 0x7fb5400c5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:12.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.158+0000 7fb53effd700 1 -- 172.21.15.27:0/3010274950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5400ca7b0 msgr2=0x7fb540140da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:12.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.158+0000 7fb53effd700 1 --2- 172.21.15.27:0/3010274950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5400ca7b0 0x7fb540140da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:12.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.158+0000 7fb53effd700 1 -- 172.21.15.27:0/3010274950 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5340094f0 con 0x7fb5400ca3e0 2024-01-11T08:11:12.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.158+0000 7fb53effd700 1 --2- 172.21.15.27:0/3010274950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5400ca3e0 0x7fb5400c6f60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb53000c4e0 tx=0x7fb53000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:12.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.159+0000 7fb526ffd700 1 -- 172.21.15.27:0/3010274950 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb530011590 con 0x7fb5400ca3e0 2024-01-11T08:11:12.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.159+0000 7fb526ffd700 1 -- 172.21.15.27:0/3010274950 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb530011710 con 0x7fb5400ca3e0 2024-01-11T08:11:12.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.159+0000 7fb526ffd700 1 -- 172.21.15.27:0/3010274950 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb530011ad0 con 0x7fb5400ca3e0 2024-01-11T08:11:12.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.159+0000 7fb5457ff700 1 -- 172.21.15.27:0/3010274950 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb54015e130 con 0x7fb5400ca3e0 2024-01-11T08:11:12.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.159+0000 7fb5457ff700 1 -- 172.21.15.27:0/3010274950 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54015e6f0 con 0x7fb5400ca3e0 2024-01-11T08:11:12.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.160+0000 7fb5457ff700 1 -- 172.21.15.27:0/3010274950 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5400cbcf0 con 0x7fb5400ca3e0 2024-01-11T08:11:12.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.161+0000 7fb526ffd700 1 -- 172.21.15.27:0/3010274950 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb53001b3b0 con 0x7fb5400ca3e0 2024-01-11T08:11:12.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.162+0000 7fb526ffd700 1 --2- 172.21.15.27:0/3010274950 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb52c066f30 0x7fb52c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:12.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.162+0000 7fb526ffd700 1 -- 172.21.15.27:0/3010274950 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fb530083a00 con 0x7fb5400ca3e0 2024-01-11T08:11:12.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.163+0000 7fb53e7fc700 1 --2- 172.21.15.27:0/3010274950 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb52c066f30 0x7fb52c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:12.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.163+0000 7fb53e7fc700 1 --2- 172.21.15.27:0/3010274950 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb52c066f30 0x7fb52c0693e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb5340099e0 tx=0x7fb534017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:12.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.166+0000 7fb526ffd700 1 -- 172.21.15.27:0/3010274950 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb530050b60 con 0x7fb5400ca3e0 2024-01-11T08:11:12.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.370+0000 7fb5457ff700 1 -- 172.21.15.27:0/3010274950 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5400718d0 con 0x7fb5400ca3e0 2024-01-11T08:11:12.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.371+0000 7fb526ffd700 1 -- 172.21.15.27:0/3010274950 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+15492 (secure 0 0 0) 0x7fb530017070 con 0x7fb5400ca3e0 2024-01-11T08:11:12.372 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:12.372 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":22,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:11:11.620745+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:10:27.993251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-11T08:10:29.153959+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.11","mappings":[{"from":7,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.33","mappings":[{"from":3,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:11:12.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.373+0000 7fb5457ff700 1 -- 172.21.15.27:0/3010274950 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb52c066f30 msgr2=0x7fb52c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.373+0000 7fb5457ff700 1 --2- 172.21.15.27:0/3010274950 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb52c066f30 0x7fb52c0693e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb5340099e0 tx=0x7fb534017000 comp rx=0 tx=0).stop 2024-01-11T08:11:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.373+0000 7fb5457ff700 1 -- 172.21.15.27:0/3010274950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5400ca3e0 msgr2=0x7fb5400c6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.373+0000 7fb5457ff700 1 --2- 172.21.15.27:0/3010274950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5400ca3e0 0x7fb5400c6f60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb53000c4e0 tx=0x7fb53000c7f0 comp rx=0 tx=0).stop 2024-01-11T08:11:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.374+0000 7fb5457ff700 1 -- 172.21.15.27:0/3010274950 shutdown_connections 2024-01-11T08:11:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.374+0000 7fb5457ff700 1 --2- 172.21.15.27:0/3010274950 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb52c066f30 0x7fb52c0693e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.374+0000 7fb5457ff700 1 --2- 172.21.15.27:0/3010274950 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb540159e80 0x7fb5400c5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.374+0000 7fb5457ff700 1 --2- 172.21.15.27:0/3010274950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5400ca3e0 0x7fb5400c6f60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.374+0000 7fb5457ff700 1 --2- 172.21.15.27:0/3010274950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5400ca7b0 0x7fb540140da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.374+0000 7fb5457ff700 1 -- 172.21.15.27:0/3010274950 >> 172.21.15.27:0/3010274950 conn(0x7fb5400ce3e0 msgr2=0x7fb5400cacd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.374+0000 7fb5457ff700 1 -- 172.21.15.27:0/3010274950 shutdown_connections 2024-01-11T08:11:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.374+0000 7fb5457ff700 1 -- 172.21.15.27:0/3010274950 wait complete. 2024-01-11T08:11:12.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-11T08:11:12.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.591+0000 7fd2675ff700 1 Processor -- start 2024-01-11T08:11:12.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.592+0000 7fd2675ff700 1 -- start start 2024-01-11T08:11:12.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.593+0000 7fd2675ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd26006bf10 0x7fd260159c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:12.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.593+0000 7fd2675ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26015d290 con 0x7fd260068720 2024-01-11T08:11:12.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.593+0000 7fd2675ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26015d570 con 0x7fd2600b0a40 2024-01-11T08:11:12.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.593+0000 7fd2675ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd260154180 con 0x7fd26006bf10 2024-01-11T08:11:12.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.593+0000 7fd265b9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd260068720 0x7fd2600c8960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50618/0 (socket says 172.21.15.27:50618) 2024-01-11T08:11:12.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.593+0000 7fd265b9c700 1 -- 172.21.15.27:0/2521147351 learned_addr learned my addr 172.21.15.27:0/2521147351 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.594+0000 7fd26539b700 1 --2- 172.21.15.27:0/2521147351 >> v2:172.21.15.77:3300/0 conn(0x7fd26006bf10 0x7fd260159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.594+0000 7fd257fff700 1 -- 172.21.15.27:0/2521147351 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456157910 0 0) 0x7fd26015d290 con 0x7fd260068720 2024-01-11T08:11:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.594+0000 7fd257fff700 1 -- 172.21.15.27:0/2521147351 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd24c003570 con 0x7fd260068720 2024-01-11T08:11:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.594+0000 7fd257fff700 1 -- 172.21.15.27:0/2521147351 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 567289582 0 0) 0x7fd24c003570 con 0x7fd260068720 2024-01-11T08:11:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.594+0000 7fd26539b700 1 -- 172.21.15.27:0/2521147351 >> v1:172.21.15.77:6789/0 conn(0x7fd2600b0a40 legacy=0x7fd26015da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.594+0000 7fd26539b700 1 -- 172.21.15.27:0/2521147351 >> v1:172.21.15.27:6789/0 conn(0x7fd260068720 legacy=0x7fd2600c8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.594+0000 7fd26539b700 1 -- 172.21.15.27:0/2521147351 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26015a510 con 0x7fd26006bf10 2024-01-11T08:11:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.594+0000 7fd26539b700 1 --2- 172.21.15.27:0/2521147351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd26006bf10 0x7fd260159c50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd25c0098e0 tx=0x7fd25c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6dcc2991368fec server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.595+0000 7fd257fff700 1 -- 172.21.15.27:0/2521147351 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2580017f0 con 0x7fd26006bf10 2024-01-11T08:11:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.595+0000 7fd257fff700 1 -- 172.21.15.27:0/2521147351 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25c011420 con 0x7fd26006bf10 2024-01-11T08:11:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.595+0000 7fd257fff700 1 -- 172.21.15.27:0/2521147351 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25c0117c0 con 0x7fd26006bf10 2024-01-11T08:11:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.595+0000 7fd2675ff700 1 -- 172.21.15.27:0/2521147351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd26006bf10 msgr2=0x7fd260159c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.595+0000 7fd2675ff700 1 --2- 172.21.15.27:0/2521147351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd26006bf10 0x7fd260159c50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd25c0098e0 tx=0x7fd25c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.595+0000 7fd2675ff700 1 -- 172.21.15.27:0/2521147351 shutdown_connections 2024-01-11T08:11:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.595+0000 7fd2675ff700 1 --2- 172.21.15.27:0/2521147351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd26006bf10 0x7fd260159c50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.595+0000 7fd2675ff700 1 -- 172.21.15.27:0/2521147351 >> 172.21.15.27:0/2521147351 conn(0x7fd2600ce3c0 msgr2=0x7fd2600ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.595+0000 7fd2675ff700 1 -- 172.21.15.27:0/2521147351 shutdown_connections 2024-01-11T08:11:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.595+0000 7fd2675ff700 1 -- 172.21.15.27:0/2521147351 wait complete. 2024-01-11T08:11:12.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.597+0000 7fd2675ff700 1 Processor -- start 2024-01-11T08:11:12.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.597+0000 7fd2675ff700 1 -- start start 2024-01-11T08:11:12.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.598+0000 7fd2675ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd260068720 0x7fd2601533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:12.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.598+0000 7fd2675ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd26006bf10 0x7fd260156470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.598+0000 7fd2675ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2600b0a40 0x7fd2601569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.598+0000 7fd2675ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2600b3980 con 0x7fd26006bf10 2024-01-11T08:11:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.598+0000 7fd2675ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2600b3c60 con 0x7fd2600b0a40 2024-01-11T08:11:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.598+0000 7fd2675ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2600b3f40 con 0x7fd260068720 2024-01-11T08:11:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.598+0000 7fd264b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd26006bf10 0x7fd260156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.598+0000 7fd264b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd26006bf10 0x7fd260156470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44364/0 (socket says 172.21.15.27:44364) 2024-01-11T08:11:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.598+0000 7fd264b9a700 1 -- 172.21.15.27:0/2500255672 learned_addr learned my addr 172.21.15.27:0/2500255672 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.598+0000 7fd26539b700 1 --2- 172.21.15.27:0/2500255672 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd260068720 0x7fd2601533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.599+0000 7fd264b9a700 1 -- 172.21.15.27:0/2500255672 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd260068720 msgr2=0x7fd2601533a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.599+0000 7fd265b9c700 1 --2- 172.21.15.27:0/2500255672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2600b0a40 0x7fd2601569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.599+0000 7fd264b9a700 1 --2- 172.21.15.27:0/2500255672 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd260068720 0x7fd2601533a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.599+0000 7fd264b9a700 1 -- 172.21.15.27:0/2500255672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2600b0a40 msgr2=0x7fd2601569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.599+0000 7fd264b9a700 1 --2- 172.21.15.27:0/2500255672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2600b0a40 0x7fd2601569b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.599+0000 7fd264b9a700 1 -- 172.21.15.27:0/2500255672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25c009420 con 0x7fd26006bf10 2024-01-11T08:11:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.599+0000 7fd265b9c700 1 --2- 172.21.15.27:0/2500255672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2600b0a40 0x7fd2601569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:11:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.599+0000 7fd264b9a700 1 --2- 172.21.15.27:0/2500255672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd26006bf10 0x7fd260156470 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd25000c590 tx=0x7fd25000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.599+0000 7fd2557fa700 1 -- 172.21.15.27:0/2500255672 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd250011590 con 0x7fd26006bf10 2024-01-11T08:11:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.599+0000 7fd2557fa700 1 -- 172.21.15.27:0/2500255672 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2500116f0 con 0x7fd26006bf10 2024-01-11T08:11:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.599+0000 7fd2675ff700 1 -- 172.21.15.27:0/2500255672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd26015db80 con 0x7fd26006bf10 2024-01-11T08:11:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.599+0000 7fd2557fa700 1 -- 172.21.15.27:0/2500255672 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd250011ab0 con 0x7fd26006bf10 2024-01-11T08:11:12.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.599+0000 7fd2675ff700 1 -- 172.21.15.27:0/2500255672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2600b2c50 con 0x7fd26006bf10 2024-01-11T08:11:12.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.600+0000 7fd2675ff700 1 -- 172.21.15.27:0/2500255672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd26015d9e0 con 0x7fd26006bf10 2024-01-11T08:11:12.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.601+0000 7fd2557fa700 1 -- 172.21.15.27:0/2500255672 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd25001b3b0 con 0x7fd26006bf10 2024-01-11T08:11:12.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.602+0000 7fd2557fa700 1 --2- 172.21.15.27:0/2500255672 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd244066f60 0x7fd244069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:12.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.602+0000 7fd2557fa700 1 -- 172.21.15.27:0/2500255672 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fd250083af0 con 0x7fd26006bf10 2024-01-11T08:11:12.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.603+0000 7fd26539b700 1 --2- 172.21.15.27:0/2500255672 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd244066f60 0x7fd244069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:12.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.604+0000 7fd26539b700 1 --2- 172.21.15.27:0/2500255672 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd244066f60 0x7fd244069410 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd25c0091e0 tx=0x7fd25c016000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:12.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.607+0000 7fd2557fa700 1 -- 172.21.15.27:0/2500255672 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd26015d9e0 con 0x7fd26006bf10 2024-01-11T08:11:12.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.859+0000 7fd2675ff700 1 -- 172.21.15.27:0/2500255672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd26015d9e0 con 0x7fd26006bf10 2024-01-11T08:11:12.860 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-11T08:11:12.859+0000 7f27065f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-11T08:11:12.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.864+0000 7fd2557fa700 1 -- 172.21.15.27:0/2500255672 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd26015d9e0 con 0x7fd26006bf10 2024-01-11T08:11:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.866+0000 7fd2675ff700 1 -- 172.21.15.27:0/2500255672 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd244066f60 msgr2=0x7fd244069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.866+0000 7fd2675ff700 1 --2- 172.21.15.27:0/2500255672 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd244066f60 0x7fd244069410 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd25c0091e0 tx=0x7fd25c016000 comp rx=0 tx=0).stop 2024-01-11T08:11:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.866+0000 7fd2675ff700 1 -- 172.21.15.27:0/2500255672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd26006bf10 msgr2=0x7fd260156470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.866+0000 7fd2675ff700 1 --2- 172.21.15.27:0/2500255672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd26006bf10 0x7fd260156470 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd25000c590 tx=0x7fd25000c950 comp rx=0 tx=0).stop 2024-01-11T08:11:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.867+0000 7fd2675ff700 1 -- 172.21.15.27:0/2500255672 shutdown_connections 2024-01-11T08:11:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.867+0000 7fd2675ff700 1 --2- 172.21.15.27:0/2500255672 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd244066f60 0x7fd244069410 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.867+0000 7fd2675ff700 1 --2- 172.21.15.27:0/2500255672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2600b0a40 0x7fd2601569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.867+0000 7fd2675ff700 1 --2- 172.21.15.27:0/2500255672 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd260068720 0x7fd2601533a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.867+0000 7fd2675ff700 1 --2- 172.21.15.27:0/2500255672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd26006bf10 0x7fd260156470 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.867+0000 7fd2675ff700 1 -- 172.21.15.27:0/2500255672 >> 172.21.15.27:0/2500255672 conn(0x7fd2600ce3c0 msgr2=0x7fd2600cbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.867+0000 7fd2675ff700 1 -- 172.21.15.27:0/2500255672 shutdown_connections 2024-01-11T08:11:12.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:12.867+0000 7fd2675ff700 1 -- 172.21.15.27:0/2500255672 wait complete. 2024-01-11T08:11:12.869 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-11T08:11:12.872 INFO:tasks.ceph.mds.d.smithi077.stderr: -2112> 2024-01-11T08:10:35.981+0000 7fda615f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T08:11:12.886 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-11T08:11:12.915 INFO:tasks.ceph.mds.d.smithi077.stdout:starting mds.d at 2024-01-11T08:11:13.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.087+0000 7f0b349ff700 1 Processor -- start 2024-01-11T08:11:13.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.087+0000 7f0b349ff700 1 -- start start 2024-01-11T08:11:13.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.089+0000 7f0b349ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b30142dc0 0x7f0b300cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:13.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.089+0000 7f0b349ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0b300cb2e0 0x7f0b300cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:13.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.089+0000 7f0b349ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b300ccc70 con 0x7f0b30143190 2024-01-11T08:11:13.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.089+0000 7f0b349ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b3015df30 con 0x7f0b300cb2e0 2024-01-11T08:11:13.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.089+0000 7f0b349ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3015d320 con 0x7f0b30142dc0 2024-01-11T08:11:13.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.089+0000 7f0b2e59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b30142dc0 0x7f0b300cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:13.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.089+0000 7f0b2e59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b30142dc0 0x7f0b300cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:36142/0 (socket says 172.21.15.27:36142) 2024-01-11T08:11:13.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.089+0000 7f0b2e59c700 1 -- 172.21.15.27:0/839409922 learned_addr learned my addr 172.21.15.27:0/839409922 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:13.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.089+0000 7f0b2ed9d700 1 --2- 172.21.15.27:0/839409922 >> v2:172.21.15.77:3300/0 conn(0x7f0b300cb2e0 0x7f0b300cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:13.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.090+0000 7f0b2e59c700 1 -- 172.21.15.27:0/839409922 >> v2:172.21.15.77:3300/0 conn(0x7f0b300cb2e0 msgr2=0x7f0b300cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:13.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.090+0000 7f0b2e59c700 1 --2- 172.21.15.27:0/839409922 >> v2:172.21.15.77:3300/0 conn(0x7f0b300cb2e0 0x7f0b300cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:13.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.090+0000 7f0b2e59c700 1 -- 172.21.15.27:0/839409922 >> v1:172.21.15.77:6789/0 conn(0x7f0b30143190 legacy=0x7f0b300cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:13.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.090+0000 7f0b2e59c700 1 -- 172.21.15.27:0/839409922 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b300cb700 con 0x7f0b30142dc0 2024-01-11T08:11:13.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.090+0000 7f0b2d59a700 1 -- 172.21.15.27:0/839409922 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3631352 0 0) 0x7f0b300ccc70 con 0x7f0b30143190 2024-01-11T08:11:13.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.090+0000 7f0b2e59c700 1 --2- 172.21.15.27:0/839409922 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b30142dc0 0x7f0b300cada0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0b200099b0 tx=0x7f0b20009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f154aa3c1598ee2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:13.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.090+0000 7f0b2d59a700 1 -- 172.21.15.27:0/839409922 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20009ea0 con 0x7f0b30142dc0 2024-01-11T08:11:13.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.090+0000 7f0b2d59a700 1 -- 172.21.15.27:0/839409922 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2000b860 con 0x7f0b30142dc0 2024-01-11T08:11:13.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.091+0000 7f0b2d59a700 1 -- 172.21.15.27:0/839409922 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2000bc00 con 0x7f0b30142dc0 2024-01-11T08:11:13.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.091+0000 7f0b349ff700 1 -- 172.21.15.27:0/839409922 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b30142dc0 msgr2=0x7f0b300cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:13.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.091+0000 7f0b349ff700 1 --2- 172.21.15.27:0/839409922 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b30142dc0 0x7f0b300cada0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0b200099b0 tx=0x7f0b20009250 comp rx=0 tx=0).stop 2024-01-11T08:11:13.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.091+0000 7f0b349ff700 1 -- 172.21.15.27:0/839409922 shutdown_connections 2024-01-11T08:11:13.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.091+0000 7f0b349ff700 1 --2- 172.21.15.27:0/839409922 >> v2:172.21.15.77:3300/0 conn(0x7f0b300cb2e0 0x7f0b300cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:13.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.091+0000 7f0b349ff700 1 --2- 172.21.15.27:0/839409922 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b30142dc0 0x7f0b300cada0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:13.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.091+0000 7f0b349ff700 1 -- 172.21.15.27:0/839409922 >> 172.21.15.27:0/839409922 conn(0x7f0b30063ca0 msgr2=0x7f0b3006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:13.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.091+0000 7f0b349ff700 1 -- 172.21.15.27:0/839409922 shutdown_connections 2024-01-11T08:11:13.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.091+0000 7f0b349ff700 1 -- 172.21.15.27:0/839409922 wait complete. 2024-01-11T08:11:13.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.093+0000 7f0b349ff700 1 Processor -- start 2024-01-11T08:11:13.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.093+0000 7f0b349ff700 1 -- start start 2024-01-11T08:11:13.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.094+0000 7f0b349ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b300cb2e0 0x7f0b3016a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:13.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.094+0000 7f0b349ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b30142dc0 0x7f0b30152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:13.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.094+0000 7f0b349ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b30143190 0x7f0b300b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:13.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.094+0000 7f0b349ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300b2540 con 0x7f0b300cb2e0 2024-01-11T08:11:13.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.094+0000 7f0b349ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300b1930 con 0x7f0b30142dc0 2024-01-11T08:11:13.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.094+0000 7f0b349ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300b1c10 con 0x7f0b30143190 2024-01-11T08:11:13.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.094+0000 7f0b2e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b300cb2e0 0x7f0b3016a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:13.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.094+0000 7f0b2e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b300cb2e0 0x7f0b3016a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44370/0 (socket says 172.21.15.27:44370) 2024-01-11T08:11:13.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.094+0000 7f0b2e59c700 1 -- 172.21.15.27:0/1592494695 learned_addr learned my addr 172.21.15.27:0/1592494695 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:13.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.094+0000 7f0b2dd9b700 1 --2- 172.21.15.27:0/1592494695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b30142dc0 0x7f0b30152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:13.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.094+0000 7f0b2ed9d700 1 --2- 172.21.15.27:0/1592494695 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b30143190 0x7f0b300b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:13.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.094+0000 7f0b2e59c700 1 -- 172.21.15.27:0/1592494695 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b30143190 msgr2=0x7f0b300b29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:13.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.094+0000 7f0b2e59c700 1 --2- 172.21.15.27:0/1592494695 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b30143190 0x7f0b300b29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:13.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.094+0000 7f0b2e59c700 1 -- 172.21.15.27:0/1592494695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b30142dc0 msgr2=0x7f0b30152ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:13.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.094+0000 7f0b2e59c700 1 --2- 172.21.15.27:0/1592494695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b30142dc0 0x7f0b30152ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:13.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.094+0000 7f0b2e59c700 1 -- 172.21.15.27:0/1592494695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b200094f0 con 0x7f0b300cb2e0 2024-01-11T08:11:13.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.095+0000 7f0b2e59c700 1 --2- 172.21.15.27:0/1592494695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b300cb2e0 0x7f0b3016a4b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0b3015d460 tx=0x7f0b20017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:13.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.095+0000 7f0b1e7fc700 1 -- 172.21.15.27:0/1592494695 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20000d10 con 0x7f0b300cb2e0 2024-01-11T08:11:13.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.095+0000 7f0b1e7fc700 1 -- 172.21.15.27:0/1592494695 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2000bbe0 con 0x7f0b300cb2e0 2024-01-11T08:11:13.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.095+0000 7f0b1e7fc700 1 -- 172.21.15.27:0/1592494695 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20011690 con 0x7f0b300cb2e0 2024-01-11T08:11:13.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.095+0000 7f0b349ff700 1 -- 172.21.15.27:0/1592494695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b300b1ef0 con 0x7f0b300cb2e0 2024-01-11T08:11:13.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.095+0000 7f0b349ff700 1 -- 172.21.15.27:0/1592494695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3016a9f0 con 0x7f0b300cb2e0 2024-01-11T08:11:13.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.096+0000 7f0b349ff700 1 -- 172.21.15.27:0/1592494695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b300b2820 con 0x7f0b300cb2e0 2024-01-11T08:11:13.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.097+0000 7f0b1e7fc700 1 -- 172.21.15.27:0/1592494695 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0b20011ac0 con 0x7f0b300cb2e0 2024-01-11T08:11:13.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.098+0000 7f0b1e7fc700 1 --2- 172.21.15.27:0/1592494695 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0b10066f30 0x7f0b100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:13.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.098+0000 7f0b1e7fc700 1 -- 172.21.15.27:0/1592494695 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f0b20085f90 con 0x7f0b300cb2e0 2024-01-11T08:11:13.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.099+0000 7f0b2dd9b700 1 --2- 172.21.15.27:0/1592494695 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0b10066f30 0x7f0b100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:13.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.099+0000 7f0b2dd9b700 1 --2- 172.21.15.27:0/1592494695 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0b10066f30 0x7f0b100693e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0b301407d0 tx=0x7f0b24007ec0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:13.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.102+0000 7f0b1e7fc700 1 -- 172.21.15.27:0/1592494695 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0b20053090 con 0x7f0b300cb2e0 2024-01-11T08:11:13.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.350+0000 7f0b349ff700 1 -- 172.21.15.27:0/1592494695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0b3004c3c0 con 0x7f0b300cb2e0 2024-01-11T08:11:13.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.869+0000 7f0b1e7fc700 1 -- 172.21.15.27:0/1592494695 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0b200566b0 con 0x7f0b300cb2e0 2024-01-11T08:11:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.872+0000 7f0b349ff700 1 -- 172.21.15.27:0/1592494695 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0b10066f30 msgr2=0x7f0b100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.872+0000 7f0b349ff700 1 --2- 172.21.15.27:0/1592494695 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0b10066f30 0x7f0b100693e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0b301407d0 tx=0x7f0b24007ec0 comp rx=0 tx=0).stop 2024-01-11T08:11:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.872+0000 7f0b349ff700 1 -- 172.21.15.27:0/1592494695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b300cb2e0 msgr2=0x7f0b3016a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.872+0000 7f0b349ff700 1 --2- 172.21.15.27:0/1592494695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b300cb2e0 0x7f0b3016a4b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0b3015d460 tx=0x7f0b20017000 comp rx=0 tx=0).stop 2024-01-11T08:11:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.872+0000 7f0b349ff700 1 -- 172.21.15.27:0/1592494695 shutdown_connections 2024-01-11T08:11:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.872+0000 7f0b349ff700 1 --2- 172.21.15.27:0/1592494695 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0b10066f30 0x7f0b100693e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.872+0000 7f0b349ff700 1 --2- 172.21.15.27:0/1592494695 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b30143190 0x7f0b300b29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.872+0000 7f0b349ff700 1 --2- 172.21.15.27:0/1592494695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b300cb2e0 0x7f0b3016a4b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.872+0000 7f0b349ff700 1 --2- 172.21.15.27:0/1592494695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b30142dc0 0x7f0b30152ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.872+0000 7f0b349ff700 1 -- 172.21.15.27:0/1592494695 >> 172.21.15.27:0/1592494695 conn(0x7f0b30063ca0 msgr2=0x7f0b30141ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.872+0000 7f0b349ff700 1 -- 172.21.15.27:0/1592494695 shutdown_connections 2024-01-11T08:11:13.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:13.872+0000 7f0b349ff700 1 -- 172.21.15.27:0/1592494695 wait complete. 2024-01-11T08:11:13.891 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-11T08:11:14.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.095+0000 7f70ad9ff700 1 Processor -- start 2024-01-11T08:11:14.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.095+0000 7f70ad9ff700 1 -- start start 2024-01-11T08:11:14.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.096+0000 7f70ad9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f70a815a7a0 0x7f70a815c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:14.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.096+0000 7f70ad9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f70a8159e60 0x7f70a815aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:14.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.096+0000 7f70ad9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a815c4b0 con 0x7f70a815b400 2024-01-11T08:11:14.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.096+0000 7f70ad9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a815b8a0 con 0x7f70a8159e60 2024-01-11T08:11:14.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.096+0000 7f70ad9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a815bb80 con 0x7f70a815a7a0 2024-01-11T08:11:14.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.096+0000 7f70a67fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f70a8159e60 0x7f70a815aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:14.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.096+0000 7f70a77fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f70a815b400 0x7f70a80685e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57676/0 (socket says 172.21.15.27:57676) 2024-01-11T08:11:14.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.096+0000 7f70a77fe700 1 -- 172.21.15.27:0/3763788550 learned_addr learned my addr 172.21.15.27:0/3763788550 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:14.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.097+0000 7f70a6ffd700 1 --2- 172.21.15.27:0/3763788550 >> v2:172.21.15.77:3300/0 conn(0x7f70a815a7a0 0x7f70a815c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:14.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.097+0000 7f70a5ffb700 1 -- 172.21.15.27:0/3763788550 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 541680765 0 0) 0x7f70a815c4b0 con 0x7f70a815b400 2024-01-11T08:11:14.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.097+0000 7f70a5ffb700 1 -- 172.21.15.27:0/3763788550 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7090003570 con 0x7f70a815b400 2024-01-11T08:11:14.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.097+0000 7f70a67fc700 1 -- 172.21.15.27:0/3763788550 >> v2:172.21.15.77:3300/0 conn(0x7f70a815a7a0 msgr2=0x7f70a815c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:14.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.097+0000 7f70a67fc700 1 --2- 172.21.15.27:0/3763788550 >> v2:172.21.15.77:3300/0 conn(0x7f70a815a7a0 0x7f70a815c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:14.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.097+0000 7f70a67fc700 1 -- 172.21.15.27:0/3763788550 >> v1:172.21.15.27:6789/0 conn(0x7f70a815b400 legacy=0x7f70a80685e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:14.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.097+0000 7f70a67fc700 1 -- 172.21.15.27:0/3763788550 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a815be60 con 0x7f70a8159e60 2024-01-11T08:11:14.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.097+0000 7f70a67fc700 1 --2- 172.21.15.27:0/3763788550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70a8159e60 0x7f70a815aec0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f70a8069650 tx=0x7f709c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe2e7c214335e56 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:14.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.098+0000 7f70a5ffb700 1 -- 172.21.15.27:0/3763788550 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c011630 con 0x7f70a8159e60 2024-01-11T08:11:14.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.098+0000 7f70a5ffb700 1 -- 172.21.15.27:0/3763788550 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709c0117b0 con 0x7f70a8159e60 2024-01-11T08:11:14.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.098+0000 7f70a5ffb700 1 -- 172.21.15.27:0/3763788550 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c011b70 con 0x7f70a8159e60 2024-01-11T08:11:14.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.098+0000 7f70ad9ff700 1 -- 172.21.15.27:0/3763788550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70a8159e60 msgr2=0x7f70a815aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:14.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.098+0000 7f70ad9ff700 1 --2- 172.21.15.27:0/3763788550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70a8159e60 0x7f70a815aec0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f70a8069650 tx=0x7f709c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:14.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.098+0000 7f70ad9ff700 1 -- 172.21.15.27:0/3763788550 shutdown_connections 2024-01-11T08:11:14.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.098+0000 7f70ad9ff700 1 --2- 172.21.15.27:0/3763788550 >> v2:172.21.15.77:3300/0 conn(0x7f70a815a7a0 0x7f70a815c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:14.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.098+0000 7f70ad9ff700 1 --2- 172.21.15.27:0/3763788550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70a8159e60 0x7f70a815aec0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:14.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.098+0000 7f70ad9ff700 1 -- 172.21.15.27:0/3763788550 >> 172.21.15.27:0/3763788550 conn(0x7f70a806ed40 msgr2=0x7f70a806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:14.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.098+0000 7f70ad9ff700 1 -- 172.21.15.27:0/3763788550 shutdown_connections 2024-01-11T08:11:14.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.098+0000 7f70ad9ff700 1 -- 172.21.15.27:0/3763788550 wait complete. 2024-01-11T08:11:14.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.100+0000 7f70ad9ff700 1 Processor -- start 2024-01-11T08:11:14.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.101+0000 7f70ad9ff700 1 -- start start 2024-01-11T08:11:14.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.102+0000 7f70ad9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70a815a7a0 0x7f70a81443f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:14.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.102+0000 7f70ad9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70a815b400 0x7f70a80aff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:14.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.102+0000 7f70ad9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70a80b5070 0x7f70a8158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:14.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.102+0000 7f70ad9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a80b16a0 con 0x7f70a815b400 2024-01-11T08:11:14.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.102+0000 7f70ad9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a80b0870 con 0x7f70a815a7a0 2024-01-11T08:11:14.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.102+0000 7f70ad9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a80b0b50 con 0x7f70a80b5070 2024-01-11T08:11:14.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.102+0000 7f70a67fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70a815b400 0x7f70a80aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:14.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.102+0000 7f70a67fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70a815b400 0x7f70a80aff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51954/0 (socket says 172.21.15.27:51954) 2024-01-11T08:11:14.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.102+0000 7f70a67fc700 1 -- 172.21.15.27:0/1996006727 learned_addr learned my addr 172.21.15.27:0/1996006727 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:14.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.103+0000 7f70a6ffd700 1 --2- 172.21.15.27:0/1996006727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70a815a7a0 0x7f70a81443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:14.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.103+0000 7f70a67fc700 1 -- 172.21.15.27:0/1996006727 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70a80b5070 msgr2=0x7f70a8158b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:14.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.103+0000 7f70a67fc700 1 --2- 172.21.15.27:0/1996006727 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70a80b5070 0x7f70a8158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:14.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.103+0000 7f70a67fc700 1 -- 172.21.15.27:0/1996006727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70a815a7a0 msgr2=0x7f70a81443f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:14.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.103+0000 7f70a67fc700 1 --2- 172.21.15.27:0/1996006727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70a815a7a0 0x7f70a81443f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:14.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.103+0000 7f70a67fc700 1 -- 172.21.15.27:0/1996006727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f709c009420 con 0x7f70a815b400 2024-01-11T08:11:14.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.103+0000 7f70a6ffd700 1 --2- 172.21.15.27:0/1996006727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70a815a7a0 0x7f70a81443f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:11:14.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.103+0000 7f70a67fc700 1 --2- 172.21.15.27:0/1996006727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70a815b400 0x7f70a80aff70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f709c013000 tx=0x7f709c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:14.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.103+0000 7f708effd700 1 -- 172.21.15.27:0/1996006727 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c01f070 con 0x7f70a815b400 2024-01-11T08:11:14.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.103+0000 7f708effd700 1 -- 172.21.15.27:0/1996006727 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709c000c90 con 0x7f70a815b400 2024-01-11T08:11:14.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.103+0000 7f70ad9ff700 1 -- 172.21.15.27:0/1996006727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a80b0e30 con 0x7f70a815b400 2024-01-11T08:11:14.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.103+0000 7f708effd700 1 -- 172.21.15.27:0/1996006727 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c011b70 con 0x7f70a815b400 2024-01-11T08:11:14.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.103+0000 7f70ad9ff700 1 -- 172.21.15.27:0/1996006727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a8144930 con 0x7f70a815b400 2024-01-11T08:11:14.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.104+0000 7f70ad9ff700 1 -- 172.21.15.27:0/1996006727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a804c3c0 con 0x7f70a815b400 2024-01-11T08:11:14.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.105+0000 7f708effd700 1 -- 172.21.15.27:0/1996006727 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f709c023070 con 0x7f70a815b400 2024-01-11T08:11:14.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.106+0000 7f708effd700 1 --2- 172.21.15.27:0/1996006727 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7094066f30 0x7f70940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:14.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.106+0000 7f708effd700 1 -- 172.21.15.27:0/1996006727 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f709c0859c0 con 0x7f70a815b400 2024-01-11T08:11:14.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.107+0000 7f70a6ffd700 1 --2- 172.21.15.27:0/1996006727 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7094066f30 0x7f70940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:14.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.107+0000 7f70a6ffd700 1 --2- 172.21.15.27:0/1996006727 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7094066f30 0x7f70940693e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7098006f00 tx=0x7f7098008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:14.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.111+0000 7f708effd700 1 -- 172.21.15.27:0/1996006727 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f709c052ac0 con 0x7f70a815b400 2024-01-11T08:11:14.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.317+0000 7f70ad9ff700 1 -- 172.21.15.27:0/1996006727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f70a815c140 con 0x7f70a815b400 2024-01-11T08:11:14.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.873+0000 7f708effd700 1 -- 172.21.15.27:0/1996006727 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) v1 ==== 181+0+0 (secure 0 0 0) 0x7f709c02a020 con 0x7f70a815b400 2024-01-11T08:11:14.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.877+0000 7f70ad9ff700 1 -- 172.21.15.27:0/1996006727 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7094066f30 msgr2=0x7f70940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:14.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.877+0000 7f70ad9ff700 1 --2- 172.21.15.27:0/1996006727 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7094066f30 0x7f70940693e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7098006f00 tx=0x7f7098008040 comp rx=0 tx=0).stop 2024-01-11T08:11:14.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.877+0000 7f70ad9ff700 1 -- 172.21.15.27:0/1996006727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70a815b400 msgr2=0x7f70a80aff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:14.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.877+0000 7f70ad9ff700 1 --2- 172.21.15.27:0/1996006727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70a815b400 0x7f70a80aff70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f709c013000 tx=0x7f709c017000 comp rx=0 tx=0).stop 2024-01-11T08:11:14.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.877+0000 7f70ad9ff700 1 -- 172.21.15.27:0/1996006727 shutdown_connections 2024-01-11T08:11:14.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.877+0000 7f70ad9ff700 1 --2- 172.21.15.27:0/1996006727 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7094066f30 0x7f70940693e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:14.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.877+0000 7f70ad9ff700 1 --2- 172.21.15.27:0/1996006727 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70a80b5070 0x7f70a8158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:14.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.877+0000 7f70ad9ff700 1 --2- 172.21.15.27:0/1996006727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70a815a7a0 0x7f70a81443f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:14.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.877+0000 7f70ad9ff700 1 --2- 172.21.15.27:0/1996006727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70a815b400 0x7f70a80aff70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:14.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.877+0000 7f70ad9ff700 1 -- 172.21.15.27:0/1996006727 >> 172.21.15.27:0/1996006727 conn(0x7f70a806ed40 msgr2=0x7f70a80cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:14.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.877+0000 7f70ad9ff700 1 -- 172.21.15.27:0/1996006727 shutdown_connections 2024-01-11T08:11:14.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:14.877+0000 7f70ad9ff700 1 -- 172.21.15.27:0/1996006727 wait complete. 2024-01-11T08:11:14.879 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-01-11T08:11:14.900 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-11T08:11:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.105+0000 7f88d6bff700 1 Processor -- start 2024-01-11T08:11:15.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.105+0000 7f88d6bff700 1 -- start start 2024-01-11T08:11:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.106+0000 7f88d6bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f88d0068be0 0x7f88d00ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.106+0000 7f88d6bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d00b0d20 con 0x7f88d0159250 2024-01-11T08:11:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.106+0000 7f88d6bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d015a440 con 0x7f88d00b08a0 2024-01-11T08:11:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.106+0000 7f88d6bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d015ace0 con 0x7f88d0068be0 2024-01-11T08:11:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.106+0000 7f88d499b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f88d0159250 0x7f88d00cb410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57682/0 (socket says 172.21.15.27:57682) 2024-01-11T08:11:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.106+0000 7f88d499b700 1 -- 172.21.15.27:0/1801784581 learned_addr learned my addr 172.21.15.27:0/1801784581 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.107+0000 7f88d519c700 1 --2- 172.21.15.27:0/1801784581 >> v2:172.21.15.77:3300/0 conn(0x7f88d0068be0 0x7f88d00ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.107+0000 7f88cf7fe700 1 -- 172.21.15.27:0/1801784581 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1209122737 0 0) 0x7f88d00b0d20 con 0x7f88d0159250 2024-01-11T08:11:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.107+0000 7f88cf7fe700 1 -- 172.21.15.27:0/1801784581 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88c0003570 con 0x7f88d0159250 2024-01-11T08:11:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.107+0000 7f88cf7fe700 1 -- 172.21.15.27:0/1801784581 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1315212551 0 0) 0x7f88c0003570 con 0x7f88d0159250 2024-01-11T08:11:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.107+0000 7f88cf7fe700 1 -- 172.21.15.27:0/1801784581 >> v2:172.21.15.77:3300/0 conn(0x7f88d0068be0 msgr2=0x7f88d00ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.107+0000 7f88cf7fe700 1 --2- 172.21.15.27:0/1801784581 >> v2:172.21.15.77:3300/0 conn(0x7f88d0068be0 0x7f88d00ce2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.107+0000 7f88cf7fe700 1 -- 172.21.15.27:0/1801784581 >> v1:172.21.15.77:6789/0 conn(0x7f88d00b08a0 legacy=0x7f88d00684d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.107+0000 7f88cf7fe700 1 -- 172.21.15.27:0/1801784581 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d015a720 con 0x7f88d0159250 2024-01-11T08:11:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.107+0000 7f88cf7fe700 1 -- 172.21.15.27:0/1801784581 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f88c4001470 con 0x7f88d0159250 2024-01-11T08:11:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.107+0000 7f88cf7fe700 1 -- 172.21.15.27:0/1801784581 >> v1:172.21.15.27:6789/0 conn(0x7f88d0159250 legacy=0x7f88d00cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.107+0000 7f88cf7fe700 1 --2- 172.21.15.27:0/1801784581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0003ab0 0x7f88c0003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.107+0000 7f88cf7fe700 1 --2- 172.21.15.27:0/1801784581 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88c0004810 0x7f88c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.107+0000 7f88cf7fe700 1 --2- 172.21.15.27:0/1801784581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88c0005210 0x7f88c000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.107+0000 7f88cf7fe700 1 -- 172.21.15.27:0/1801784581 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d015ace0 con 0x7f88c0005210 2024-01-11T08:11:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.107+0000 7f88cf7fe700 1 -- 172.21.15.27:0/1801784581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c000c130 con 0x7f88c0003ab0 2024-01-11T08:11:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.107+0000 7f88cf7fe700 1 -- 172.21.15.27:0/1801784581 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c000c2f0 con 0x7f88c0004810 2024-01-11T08:11:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.108+0000 7f88d519c700 1 --2- 172.21.15.27:0/1801784581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88c0005210 0x7f88c000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.108+0000 7f88d499b700 1 --2- 172.21.15.27:0/1801784581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0003ab0 0x7f88c0003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.108+0000 7f88cffff700 1 --2- 172.21.15.27:0/1801784581 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88c0004810 0x7f88c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.108+0000 7f88d499b700 1 -- 172.21.15.27:0/1801784581 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88c0004810 msgr2=0x7f88c0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.108+0000 7f88d499b700 1 --2- 172.21.15.27:0/1801784581 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88c0004810 0x7f88c0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:15.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.108+0000 7f88d499b700 1 -- 172.21.15.27:0/1801784581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88c0005210 msgr2=0x7f88c000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:15.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.108+0000 7f88d499b700 1 --2- 172.21.15.27:0/1801784581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88c0005210 0x7f88c000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:15.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.108+0000 7f88d499b700 1 -- 172.21.15.27:0/1801784581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c0003570 con 0x7f88c0003ab0 2024-01-11T08:11:15.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.108+0000 7f88d519c700 1 --2- 172.21.15.27:0/1801784581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88c0005210 0x7f88c000ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:15.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.108+0000 7f88d519c700 1 -- 172.21.15.27:0/1801784581 reap_dead start 2024-01-11T08:11:15.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.109+0000 7f88d499b700 1 --2- 172.21.15.27:0/1801784581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0003ab0 0x7f88c0003ec0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f88c400c2d0 tx=0x7f88c4001d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40bb095c6bdc1196 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:15.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.109+0000 7f88cf7fe700 1 -- 172.21.15.27:0/1801784581 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c400cba0 con 0x7f88c0003ab0 2024-01-11T08:11:15.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.109+0000 7f88cf7fe700 1 -- 172.21.15.27:0/1801784581 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c400cd20 con 0x7f88c0003ab0 2024-01-11T08:11:15.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.109+0000 7f88cf7fe700 1 -- 172.21.15.27:0/1801784581 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c4005b70 con 0x7f88c0003ab0 2024-01-11T08:11:15.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.109+0000 7f88d6bff700 1 -- 172.21.15.27:0/1801784581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0003ab0 msgr2=0x7f88c0003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:15.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.109+0000 7f88d6bff700 1 --2- 172.21.15.27:0/1801784581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0003ab0 0x7f88c0003ec0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f88c400c2d0 tx=0x7f88c4001d80 comp rx=0 tx=0).stop 2024-01-11T08:11:15.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.109+0000 7f88d6bff700 1 -- 172.21.15.27:0/1801784581 shutdown_connections 2024-01-11T08:11:15.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.109+0000 7f88d6bff700 1 --2- 172.21.15.27:0/1801784581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0003ab0 0x7f88c0003ec0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:15.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.109+0000 7f88d6bff700 1 -- 172.21.15.27:0/1801784581 >> 172.21.15.27:0/1801784581 conn(0x7f88d0067610 msgr2=0x7f88d0067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:15.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.109+0000 7f88d6bff700 1 -- 172.21.15.27:0/1801784581 shutdown_connections 2024-01-11T08:11:15.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.110+0000 7f88d6bff700 1 -- 172.21.15.27:0/1801784581 wait complete. 2024-01-11T08:11:15.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.112+0000 7f88d6bff700 1 Processor -- start 2024-01-11T08:11:15.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.112+0000 7f88d6bff700 1 -- start start 2024-01-11T08:11:15.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.113+0000 7f88d6bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0003ab0 0x7f88d0156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:15.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.113+0000 7f88d6bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88d00ad040 0x7f88d00af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:15.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.113+0000 7f88d6bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88d00ad4b0 0x7f88d00b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:15.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.113+0000 7f88d6bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d01573e0 con 0x7f88d00ad4b0 2024-01-11T08:11:15.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.113+0000 7f88d6bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d00ae520 con 0x7f88c0003ab0 2024-01-11T08:11:15.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.113+0000 7f88d6bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d00ae800 con 0x7f88d00ad040 2024-01-11T08:11:15.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.113+0000 7f88d519c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88d00ad4b0 0x7f88d00b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:15.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.113+0000 7f88d519c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88d00ad4b0 0x7f88d00b1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51970/0 (socket says 172.21.15.27:51970) 2024-01-11T08:11:15.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.113+0000 7f88d519c700 1 -- 172.21.15.27:0/1214785047 learned_addr learned my addr 172.21.15.27:0/1214785047 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:15.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.113+0000 7f88d499b700 1 --2- 172.21.15.27:0/1214785047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0003ab0 0x7f88d0156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:15.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.113+0000 7f88cffff700 1 --2- 172.21.15.27:0/1214785047 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88d00ad040 0x7f88d00af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:15.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.114+0000 7f88d519c700 1 -- 172.21.15.27:0/1214785047 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88d00ad040 msgr2=0x7f88d00af5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:15.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.114+0000 7f88d519c700 1 --2- 172.21.15.27:0/1214785047 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88d00ad040 0x7f88d00af5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.114+0000 7f88d519c700 1 -- 172.21.15.27:0/1214785047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0003ab0 msgr2=0x7f88d0156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.114+0000 7f88d519c700 1 --2- 172.21.15.27:0/1214785047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0003ab0 0x7f88d0156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.114+0000 7f88d519c700 1 -- 172.21.15.27:0/1214785047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d00aeae0 con 0x7f88d00ad4b0 2024-01-11T08:11:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.114+0000 7f88d519c700 1 --2- 172.21.15.27:0/1214785047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88d00ad4b0 0x7f88d00b1f30 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f88d00c9be0 tx=0x7f88c800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.114+0000 7f88ccff9700 1 -- 172.21.15.27:0/1214785047 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c800e040 con 0x7f88d00ad4b0 2024-01-11T08:11:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.114+0000 7f88ccff9700 1 -- 172.21.15.27:0/1214785047 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c800f040 con 0x7f88d00ad4b0 2024-01-11T08:11:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.114+0000 7f88d6bff700 1 -- 172.21.15.27:0/1214785047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d00aedc0 con 0x7f88d00ad4b0 2024-01-11T08:11:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.114+0000 7f88ccff9700 1 -- 172.21.15.27:0/1214785047 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c800fe60 con 0x7f88d00ad4b0 2024-01-11T08:11:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.115+0000 7f88d6bff700 1 -- 172.21.15.27:0/1214785047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d00afb00 con 0x7f88d00ad4b0 2024-01-11T08:11:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.116+0000 7f88d6bff700 1 -- 172.21.15.27:0/1214785047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d00af2e0 con 0x7f88d00ad4b0 2024-01-11T08:11:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.118+0000 7f88ccff9700 1 -- 172.21.15.27:0/1214785047 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f88c8015640 con 0x7f88d00ad4b0 2024-01-11T08:11:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.119+0000 7f88ccff9700 1 --2- 172.21.15.27:0/1214785047 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f88b4066f30 0x7f88b40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.119+0000 7f88ccff9700 1 -- 172.21.15.27:0/1214785047 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5469+0+0 (secure 0 0 0) 0x7f88c80847e0 con 0x7f88d00ad4b0 2024-01-11T08:11:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.119+0000 7f88d499b700 1 --2- 172.21.15.27:0/1214785047 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f88b4066f30 0x7f88b40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.120+0000 7f88d499b700 1 --2- 172.21.15.27:0/1214785047 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f88b4066f30 0x7f88b40693e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f88c4007c00 tx=0x7f88c4015000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:15.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.122+0000 7f88ccff9700 1 -- 172.21.15.27:0/1214785047 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f88c8051b20 con 0x7f88d00ad4b0 2024-01-11T08:11:15.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.333+0000 7f88d6bff700 1 -- 172.21.15.27:0/1214785047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f88d00af2e0 con 0x7f88d00ad4b0 2024-01-11T08:11:15.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.881+0000 7f88ccff9700 1 -- 172.21.15.27:0/1214785047 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) v1 ==== 169+0+0 (secure 0 0 0) 0x7f88d00af2e0 con 0x7f88d00ad4b0 2024-01-11T08:11:15.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.885+0000 7f88d6bff700 1 -- 172.21.15.27:0/1214785047 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f88b4066f30 msgr2=0x7f88b40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:15.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.885+0000 7f88d6bff700 1 --2- 172.21.15.27:0/1214785047 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f88b4066f30 0x7f88b40693e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f88c4007c00 tx=0x7f88c4015000 comp rx=0 tx=0).stop 2024-01-11T08:11:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.885+0000 7f88d6bff700 1 -- 172.21.15.27:0/1214785047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88d00ad4b0 msgr2=0x7f88d00b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.885+0000 7f88d6bff700 1 --2- 172.21.15.27:0/1214785047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88d00ad4b0 0x7f88d00b1f30 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f88d00c9be0 tx=0x7f88c800b040 comp rx=0 tx=0).stop 2024-01-11T08:11:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.885+0000 7f88d6bff700 1 -- 172.21.15.27:0/1214785047 shutdown_connections 2024-01-11T08:11:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.885+0000 7f88d6bff700 1 --2- 172.21.15.27:0/1214785047 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f88b4066f30 0x7f88b40693e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.885+0000 7f88d6bff700 1 --2- 172.21.15.27:0/1214785047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88d00ad4b0 0x7f88d00b1f30 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.885+0000 7f88d6bff700 1 --2- 172.21.15.27:0/1214785047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0003ab0 0x7f88d0156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.885+0000 7f88d6bff700 1 --2- 172.21.15.27:0/1214785047 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88d00ad040 0x7f88d00af5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.885+0000 7f88d6bff700 1 -- 172.21.15.27:0/1214785047 >> 172.21.15.27:0/1214785047 conn(0x7f88d0067610 msgr2=0x7f88d00cb310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.886+0000 7f88d6bff700 1 -- 172.21.15.27:0/1214785047 shutdown_connections 2024-01-11T08:11:15.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:15.886+0000 7f88d6bff700 1 -- 172.21.15.27:0/1214785047 wait complete. 2024-01-11T08:11:15.888 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-01-11T08:11:15.907 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-11T08:11:15.907 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-11T08:11:15.907 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-11T08:11:15.908 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:11:15.908 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-11T08:11:15.908 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-11T08:11:15.908 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-11T08:11:15.908 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:11:15.908 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-11T08:11:15.908 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-11T08:11:15.909 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-11T08:11:15.909 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:11:15.909 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-11T08:11:15.909 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-11T08:11:15.909 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-11T08:11:15.909 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:11:15.922 INFO:tasks.ceph.mds.a:Stopped 2024-01-11T08:11:15.923 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-11T08:11:15.924 INFO:tasks.ceph.mds.c:Stopped 2024-01-11T08:11:15.924 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-11T08:11:15.925 INFO:tasks.ceph.mds.d:Stopped 2024-01-11T08:11:15.926 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-11T08:11:15.926 INFO:tasks.ceph.mds.b:Stopped 2024-01-11T08:11:15.926 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-11T08:11:15.968 INFO:tasks.ceph.mds.a:Started 2024-01-11T08:11:15.968 DEBUG:teuthology.parallel:result is None 2024-01-11T08:11:15.971 INFO:tasks.ceph.mds.d:Started 2024-01-11T08:11:15.971 DEBUG:teuthology.parallel:result is None 2024-01-11T08:11:15.971 INFO:tasks.ceph.mds.c:Started 2024-01-11T08:11:15.972 DEBUG:teuthology.parallel:result is None 2024-01-11T08:11:15.974 INFO:tasks.ceph.mds.b:Started 2024-01-11T08:11:15.974 DEBUG:teuthology.parallel:result is None 2024-01-11T08:11:15.974 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-11T08:11:16.117 INFO:tasks.ceph.mds.d.smithi077.stdout:starting mds.d at 2024-01-11T08:11:16.117 INFO:tasks.ceph.mds.b.smithi077.stdout:starting mds.b at 2024-01-11T08:11:16.125 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T08:11:16.126 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T08:11:16.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.193+0000 7fc87ddff700 1 Processor -- start 2024-01-11T08:11:16.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.193+0000 7fc87ddff700 1 -- start start 2024-01-11T08:11:16.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.194+0000 7fc87ddff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc878157680 0x7fc878063230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:16.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.194+0000 7fc87ddff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc878157a50 0x7fc8780b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:16.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.194+0000 7fc87ddff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc878157ec0 0x7fc8780b2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:16.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.194+0000 7fc87ddff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc878063770 con 0x7fc878157ec0 2024-01-11T08:11:16.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.194+0000 7fc87ddff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc878067980 con 0x7fc878157680 2024-01-11T08:11:16.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.194+0000 7fc87ddff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc878067c60 con 0x7fc878157a50 2024-01-11T08:11:16.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.195+0000 7fc877fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc878157ec0 0x7fc8780b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:16.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.195+0000 7fc8777fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc878157680 0x7fc878063230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:16.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.195+0000 7fc876ffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc878157a50 0x7fc8780b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:16.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.195+0000 7fc8777fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc878157680 0x7fc878063230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:49080/0 (socket says 172.21.15.27:49080) 2024-01-11T08:11:16.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.195+0000 7fc877fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc878157ec0 0x7fc8780b2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52012/0 (socket says 172.21.15.27:52012) 2024-01-11T08:11:16.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.195+0000 7fc8777fe700 1 -- 172.21.15.27:0/3441579031 learned_addr learned my addr 172.21.15.27:0/3441579031 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:16.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.195+0000 7fc8777fe700 1 -- 172.21.15.27:0/3441579031 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc878157a50 msgr2=0x7fc8780b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:16.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.195+0000 7fc8777fe700 1 --2- 172.21.15.27:0/3441579031 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc878157a50 0x7fc8780b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:16.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.195+0000 7fc8777fe700 1 -- 172.21.15.27:0/3441579031 >> v2:172.21.15.27:3300/0 conn(0x7fc878157ec0 msgr2=0x7fc8780b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:16.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.195+0000 7fc8777fe700 1 --2- 172.21.15.27:0/3441579031 >> v2:172.21.15.27:3300/0 conn(0x7fc878157ec0 0x7fc8780b2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:16.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.195+0000 7fc8777fe700 1 -- 172.21.15.27:0/3441579031 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc878067f40 con 0x7fc878157680 2024-01-11T08:11:16.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.195+0000 7fc877fff700 1 --2- 172.21.15.27:0/3441579031 >> v2:172.21.15.27:3300/0 conn(0x7fc878157ec0 0x7fc8780b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:16.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.196+0000 7fc8777fe700 1 --2- 172.21.15.27:0/3441579031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc878157680 0x7fc878063230 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc87806d3b0 tx=0x7fc868009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f172fc6ba26bc360 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:16.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.196+0000 7fc8767fc700 1 -- 172.21.15.27:0/3441579031 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86800b640 con 0x7fc878157680 2024-01-11T08:11:16.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.196+0000 7fc8767fc700 1 -- 172.21.15.27:0/3441579031 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86800b7a0 con 0x7fc878157680 2024-01-11T08:11:16.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.196+0000 7fc8767fc700 1 -- 172.21.15.27:0/3441579031 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86800bb40 con 0x7fc878157680 2024-01-11T08:11:16.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.196+0000 7fc87ddff700 1 -- 172.21.15.27:0/3441579031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc878157680 msgr2=0x7fc878063230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:16.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.196+0000 7fc87ddff700 1 --2- 172.21.15.27:0/3441579031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc878157680 0x7fc878063230 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc87806d3b0 tx=0x7fc868009250 comp rx=0 tx=0).stop 2024-01-11T08:11:16.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.197+0000 7fc87ddff700 1 -- 172.21.15.27:0/3441579031 shutdown_connections 2024-01-11T08:11:16.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.197+0000 7fc87ddff700 1 --2- 172.21.15.27:0/3441579031 >> v2:172.21.15.27:3300/0 conn(0x7fc878157ec0 0x7fc8780b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:16.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.197+0000 7fc87ddff700 1 --2- 172.21.15.27:0/3441579031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc878157680 0x7fc878063230 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:16.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.197+0000 7fc87ddff700 1 --2- 172.21.15.27:0/3441579031 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc878157a50 0x7fc8780b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:16.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.197+0000 7fc87ddff700 1 -- 172.21.15.27:0/3441579031 >> 172.21.15.27:0/3441579031 conn(0x7fc87806a8b0 msgr2=0x7fc8780ae520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.197+0000 7fc87ddff700 1 -- 172.21.15.27:0/3441579031 shutdown_connections 2024-01-11T08:11:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.197+0000 7fc87ddff700 1 -- 172.21.15.27:0/3441579031 wait complete. 2024-01-11T08:11:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.198+0000 7fc87ddff700 1 Processor -- start 2024-01-11T08:11:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.198+0000 7fc87ddff700 1 -- start start 2024-01-11T08:11:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.199+0000 7fc87ddff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc878157680 0x7fc878145340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.199+0000 7fc87ddff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc878157a50 0x7fc8780c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.199+0000 7fc87ddff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc878157ec0 0x7fc87815e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.199+0000 7fc87ddff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8780c81e0 con 0x7fc878157680 2024-01-11T08:11:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.199+0000 7fc87ddff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8780c75d0 con 0x7fc878157a50 2024-01-11T08:11:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.199+0000 7fc87ddff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8780c78b0 con 0x7fc878157ec0 2024-01-11T08:11:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.200+0000 7fc8777fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc878157680 0x7fc878145340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.200+0000 7fc8777fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc878157680 0x7fc878145340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52018/0 (socket says 172.21.15.27:52018) 2024-01-11T08:11:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.200+0000 7fc8777fe700 1 -- 172.21.15.27:0/572877138 learned_addr learned my addr 172.21.15.27:0/572877138 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.200+0000 7fc876ffd700 1 --2- 172.21.15.27:0/572877138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc878157a50 0x7fc8780c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.200+0000 7fc877fff700 1 --2- 172.21.15.27:0/572877138 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc878157ec0 0x7fc87815e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.200+0000 7fc8777fe700 1 -- 172.21.15.27:0/572877138 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc878157ec0 msgr2=0x7fc87815e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.200+0000 7fc8777fe700 1 --2- 172.21.15.27:0/572877138 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc878157ec0 0x7fc87815e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.200+0000 7fc8777fe700 1 -- 172.21.15.27:0/572877138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc878157a50 msgr2=0x7fc8780c6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.200+0000 7fc8777fe700 1 --2- 172.21.15.27:0/572877138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc878157a50 0x7fc8780c6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.200+0000 7fc8777fe700 1 -- 172.21.15.27:0/572877138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc870009a20 con 0x7fc878157680 2024-01-11T08:11:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.200+0000 7fc876ffd700 1 --2- 172.21.15.27:0/572877138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc878157a50 0x7fc8780c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.200+0000 7fc877fff700 1 --2- 172.21.15.27:0/572877138 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc878157ec0 0x7fc87815e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.200+0000 7fc8777fe700 1 --2- 172.21.15.27:0/572877138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc878157680 0x7fc878145340 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc868006fd0 tx=0x7fc868017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.201+0000 7fc85f7fe700 1 -- 172.21.15.27:0/572877138 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86800b910 con 0x7fc878157680 2024-01-11T08:11:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.201+0000 7fc85f7fe700 1 -- 172.21.15.27:0/572877138 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86800ba90 con 0x7fc878157680 2024-01-11T08:11:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.201+0000 7fc85f7fe700 1 -- 172.21.15.27:0/572877138 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc868011670 con 0x7fc878157680 2024-01-11T08:11:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.201+0000 7fc87ddff700 1 -- 172.21.15.27:0/572877138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8680094f0 con 0x7fc878157680 2024-01-11T08:11:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.201+0000 7fc87ddff700 1 -- 172.21.15.27:0/572877138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8780c7e70 con 0x7fc878157680 2024-01-11T08:11:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.202+0000 7fc87ddff700 1 -- 172.21.15.27:0/572877138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8780c8570 con 0x7fc878157680 2024-01-11T08:11:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.203+0000 7fc85f7fe700 1 -- 172.21.15.27:0/572877138 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc868011a60 con 0x7fc878157680 2024-01-11T08:11:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.204+0000 7fc85f7fe700 1 --2- 172.21.15.27:0/572877138 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc864066f30 0x7fc8640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.204+0000 7fc85f7fe700 1 -- 172.21.15.27:0/572877138 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4939+0+0 (secure 0 0 0) 0x7fc868085a60 con 0x7fc878157680 2024-01-11T08:11:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.204+0000 7fc876ffd700 1 --2- 172.21.15.27:0/572877138 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc864066f30 0x7fc8640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.205+0000 7fc876ffd700 1 --2- 172.21.15.27:0/572877138 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc864066f30 0x7fc8640693e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc8780cb300 tx=0x7fc86c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:16.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.208+0000 7fc85f7fe700 1 -- 172.21.15.27:0/572877138 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc8780c8570 con 0x7fc878157680 2024-01-11T08:11:16.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.429+0000 7fc87ddff700 1 -- 172.21.15.27:0/572877138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc8780c8570 con 0x7fc878157680 2024-01-11T08:11:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.896+0000 7fc85f7fe700 1 -- 172.21.15.27:0/572877138 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v28) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc8780c8570 con 0x7fc878157680 2024-01-11T08:11:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.899+0000 7fc87ddff700 1 -- 172.21.15.27:0/572877138 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc864066f30 msgr2=0x7fc8640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.899+0000 7fc87ddff700 1 --2- 172.21.15.27:0/572877138 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc864066f30 0x7fc8640693e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc8780cb300 tx=0x7fc86c008040 comp rx=0 tx=0).stop 2024-01-11T08:11:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.899+0000 7fc87ddff700 1 -- 172.21.15.27:0/572877138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc878157680 msgr2=0x7fc878145340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.899+0000 7fc87ddff700 1 --2- 172.21.15.27:0/572877138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc878157680 0x7fc878145340 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc868006fd0 tx=0x7fc868017000 comp rx=0 tx=0).stop 2024-01-11T08:11:16.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.899+0000 7fc87ddff700 1 -- 172.21.15.27:0/572877138 shutdown_connections 2024-01-11T08:11:16.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.899+0000 7fc87ddff700 1 --2- 172.21.15.27:0/572877138 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc864066f30 0x7fc8640693e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:16.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.899+0000 7fc87ddff700 1 --2- 172.21.15.27:0/572877138 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc878157ec0 0x7fc87815e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:16.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.899+0000 7fc87ddff700 1 --2- 172.21.15.27:0/572877138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc878157680 0x7fc878145340 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:16.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.899+0000 7fc87ddff700 1 --2- 172.21.15.27:0/572877138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc878157a50 0x7fc8780c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:16.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.899+0000 7fc87ddff700 1 -- 172.21.15.27:0/572877138 >> 172.21.15.27:0/572877138 conn(0x7fc87806a8b0 msgr2=0x7fc8780b19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:16.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.899+0000 7fc87ddff700 1 -- 172.21.15.27:0/572877138 shutdown_connections 2024-01-11T08:11:16.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:16.899+0000 7fc87ddff700 1 -- 172.21.15.27:0/572877138 wait complete. 2024-01-11T08:11:16.902 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-11T08:11:16.924 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-11T08:11:17.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.123+0000 7f5cd0bff700 1 Processor -- start 2024-01-11T08:11:17.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.123+0000 7f5cd0bff700 1 -- start start 2024-01-11T08:11:17.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.125+0000 7f5cd0bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ccc0b03e0 0x7f5ccc0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:17.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.125+0000 7f5cd0bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ccc0b07b0 0x7f5ccc0b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:17.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.125+0000 7f5cd0bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc0cd730 con 0x7f5ccc159200 2024-01-11T08:11:17.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.125+0000 7f5cd0bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ccc159d50 con 0x7f5ccc0b07b0 2024-01-11T08:11:17.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.125+0000 7f5cd0bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc0ccb20 con 0x7f5ccc0b03e0 2024-01-11T08:11:17.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.125+0000 7f5cc9d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ccc0b07b0 0x7f5ccc0b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:17.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.125+0000 7f5cc9d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ccc0b07b0 0x7f5ccc0b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52020/0 (socket says 172.21.15.27:52020) 2024-01-11T08:11:17.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.125+0000 7f5cc9d9b700 1 -- 172.21.15.27:0/766073886 learned_addr learned my addr 172.21.15.27:0/766073886 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:17.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.125+0000 7f5cca59c700 1 --2- 172.21.15.27:0/766073886 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ccc0b03e0 0x7f5ccc0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.126+0000 7f5cc9d9b700 1 -- 172.21.15.27:0/766073886 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ccc0b03e0 msgr2=0x7f5ccc0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.126+0000 7f5cc9d9b700 1 --2- 172.21.15.27:0/766073886 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ccc0b03e0 0x7f5ccc0b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.126+0000 7f5cc9d9b700 1 -- 172.21.15.27:0/766073886 >> v1:172.21.15.77:6789/0 conn(0x7f5ccc159200 legacy=0x7f5ccc159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.126+0000 7f5cc9d9b700 1 -- 172.21.15.27:0/766073886 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc0cce00 con 0x7f5ccc0b07b0 2024-01-11T08:11:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.126+0000 7f5cc959a700 1 -- 172.21.15.27:0/766073886 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292922528 0 0) 0x7f5ccc0cd730 con 0x7f5ccc159200 2024-01-11T08:11:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.126+0000 7f5cc9d9b700 1 --2- 172.21.15.27:0/766073886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ccc0b07b0 0x7f5ccc0b29c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5cc00099b0 tx=0x7f5cc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b87669856a3c9b7e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.126+0000 7f5cc959a700 1 -- 172.21.15.27:0/766073886 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0009ea0 con 0x7f5ccc0b07b0 2024-01-11T08:11:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.126+0000 7f5cc959a700 1 -- 172.21.15.27:0/766073886 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc000b860 con 0x7f5ccc0b07b0 2024-01-11T08:11:17.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.126+0000 7f5cc959a700 1 -- 172.21.15.27:0/766073886 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc000bc00 con 0x7f5ccc0b07b0 2024-01-11T08:11:17.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.127+0000 7f5cd0bff700 1 -- 172.21.15.27:0/766073886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ccc0b07b0 msgr2=0x7f5ccc0b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:17.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.127+0000 7f5cd0bff700 1 --2- 172.21.15.27:0/766073886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ccc0b07b0 0x7f5ccc0b29c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5cc00099b0 tx=0x7f5cc0009250 comp rx=0 tx=0).stop 2024-01-11T08:11:17.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.127+0000 7f5cd0bff700 1 -- 172.21.15.27:0/766073886 shutdown_connections 2024-01-11T08:11:17.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.127+0000 7f5cd0bff700 1 --2- 172.21.15.27:0/766073886 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ccc0b03e0 0x7f5ccc0b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:17.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.127+0000 7f5cd0bff700 1 --2- 172.21.15.27:0/766073886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ccc0b07b0 0x7f5ccc0b29c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:17.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.127+0000 7f5cd0bff700 1 -- 172.21.15.27:0/766073886 >> 172.21.15.27:0/766073886 conn(0x7f5ccc06ed40 msgr2=0x7f5ccc06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:17.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.127+0000 7f5cd0bff700 1 -- 172.21.15.27:0/766073886 shutdown_connections 2024-01-11T08:11:17.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.127+0000 7f5cd0bff700 1 -- 172.21.15.27:0/766073886 wait complete. 2024-01-11T08:11:17.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.128+0000 7f5cd0bff700 1 Processor -- start 2024-01-11T08:11:17.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.128+0000 7f5cd0bff700 1 -- start start 2024-01-11T08:11:17.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.129+0000 7f5cd0bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ccc0b03e0 0x7f5ccc0cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:17.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.129+0000 7f5cd0bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ccc0b07b0 0x7f5ccc0a7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:17.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.129+0000 7f5cd0bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ccc159200 0x7f5ccc157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:17.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.129+0000 7f5cd0bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc0c75c0 con 0x7f5ccc0b03e0 2024-01-11T08:11:17.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.129+0000 7f5cd0bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc0c7810 con 0x7f5ccc0b07b0 2024-01-11T08:11:17.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.129+0000 7f5cd0bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc0c7af0 con 0x7f5ccc159200 2024-01-11T08:11:17.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.130+0000 7f5cca59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ccc0b03e0 0x7f5ccc0cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:17.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.130+0000 7f5cca59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ccc0b03e0 0x7f5ccc0cbf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52028/0 (socket says 172.21.15.27:52028) 2024-01-11T08:11:17.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.130+0000 7f5cca59c700 1 -- 172.21.15.27:0/1884968052 learned_addr learned my addr 172.21.15.27:0/1884968052 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:17.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.130+0000 7f5cc9d9b700 1 --2- 172.21.15.27:0/1884968052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ccc0b07b0 0x7f5ccc0a7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:17.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.130+0000 7f5ccad9d700 1 --2- 172.21.15.27:0/1884968052 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ccc159200 0x7f5ccc157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:17.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.130+0000 7f5cca59c700 1 -- 172.21.15.27:0/1884968052 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ccc159200 msgr2=0x7f5ccc157740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:17.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.130+0000 7f5cca59c700 1 --2- 172.21.15.27:0/1884968052 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ccc159200 0x7f5ccc157740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:17.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.130+0000 7f5cca59c700 1 -- 172.21.15.27:0/1884968052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ccc0b07b0 msgr2=0x7f5ccc0a7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:17.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.130+0000 7f5cca59c700 1 --2- 172.21.15.27:0/1884968052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ccc0b07b0 0x7f5ccc0a7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:17.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.130+0000 7f5cca59c700 1 -- 172.21.15.27:0/1884968052 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc00094f0 con 0x7f5ccc0b03e0 2024-01-11T08:11:17.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.130+0000 7f5cc9d9b700 1 --2- 172.21.15.27:0/1884968052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ccc0b07b0 0x7f5ccc0a7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:17.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.130+0000 7f5cca59c700 1 --2- 172.21.15.27:0/1884968052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ccc0b03e0 0x7f5ccc0cbf40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00ec80 tx=0x7f5cbc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:17.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.131+0000 7f5cba7fc700 1 -- 172.21.15.27:0/1884968052 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc00c850 con 0x7f5ccc0b03e0 2024-01-11T08:11:17.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.131+0000 7f5cba7fc700 1 -- 172.21.15.27:0/1884968052 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cbc00c9b0 con 0x7f5ccc0b03e0 2024-01-11T08:11:17.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.131+0000 7f5cd0bff700 1 -- 172.21.15.27:0/1884968052 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ccc0c7dd0 con 0x7f5ccc0b03e0 2024-01-11T08:11:17.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.131+0000 7f5cba7fc700 1 -- 172.21.15.27:0/1884968052 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc00cd50 con 0x7f5ccc0b03e0 2024-01-11T08:11:17.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.131+0000 7f5cd0bff700 1 -- 172.21.15.27:0/1884968052 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ccc0c8390 con 0x7f5ccc0b03e0 2024-01-11T08:11:17.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.132+0000 7f5cd0bff700 1 -- 172.21.15.27:0/1884968052 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ccc169bc0 con 0x7f5ccc0b03e0 2024-01-11T08:11:17.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.133+0000 7f5cba7fc700 1 -- 172.21.15.27:0/1884968052 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5cbc013680 con 0x7f5ccc0b03e0 2024-01-11T08:11:17.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.134+0000 7f5cba7fc700 1 --2- 172.21.15.27:0/1884968052 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5cac066f30 0x7f5cac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:17.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.134+0000 7f5cba7fc700 1 -- 172.21.15.27:0/1884968052 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f5cbc084500 con 0x7f5ccc0b03e0 2024-01-11T08:11:17.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.135+0000 7f5cc9d9b700 1 --2- 172.21.15.27:0/1884968052 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5cac066f30 0x7f5cac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:17.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.135+0000 7f5cc9d9b700 1 --2- 172.21.15.27:0/1884968052 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5cac066f30 0x7f5cac0693e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5cc00099b0 tx=0x7f5cc0017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:17.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.138+0000 7f5cba7fc700 1 -- 172.21.15.27:0/1884968052 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5cbc051a20 con 0x7f5ccc0b03e0 2024-01-11T08:11:17.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.419+0000 7f5cd0bff700 1 -- 172.21.15.27:0/1884968052 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5ccc0718d0 con 0x7f5ccc0b03e0 2024-01-11T08:11:17.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.420+0000 7f5cba7fc700 1 -- 172.21.15.27:0/1884968052 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v2) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f5cbc055040 con 0x7f5ccc0b03e0 2024-01-11T08:11:17.421 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:17.421 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T08:11:17.422 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-11T08:11:17.422 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.422 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-11T08:11:17.422 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lQm3yLhAAv6erz5hMfat8EUzRA5luMA==", 2024-01-11T08:11:17.422 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.422 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:11:17.423 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:11:17.423 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:11:17.423 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:11:17.423 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.423 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.423 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.423 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-11T08:11:17.423 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9l+FG8ChAACHndtVLx7qS3sidPniSycA==", 2024-01-11T08:11:17.423 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.423 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:11:17.423 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:11:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:11:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:11:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-11T08:11:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l1NnEOhAALHXsc5es7hufCthGdRzkyw==", 2024-01-11T08:11:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:11:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:11:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:11:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:11:17.424 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-11T08:11:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lruoWFhAAm8AmrZtTJhYI1+19F79RPQ==", 2024-01-11T08:11:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:11:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:11:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:11:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:11:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-11T08:11:17.425 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA2op9ltZygORAAZr1X1nCN0ce8rIVaGDpFyQ==", 2024-01-11T08:11:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:11:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:11:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:11:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-11T08:11:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA6op9lqSRPAhAAmtyCsyK5ZVGX2wfhOCMuFQ==", 2024-01-11T08:11:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:11:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:11:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:11:17.426 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-11T08:11:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA9op9lZcs8BhAA8j+U+jPOoy+fliVWZAPFYg==", 2024-01-11T08:11:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:11:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:11:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:11:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.427 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-11T08:11:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBAop9lFuCUChAAP6si/2W1mxH4ecQJvuumkw==", 2024-01-11T08:11:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:11:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:11:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:11:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-11T08:11:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBGop9lOpwBIRAAcpcsjwmX2IldcKHJWAgHuA==", 2024-01-11T08:11:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:11:17.428 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:11:17.429 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:11:17.429 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.429 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.429 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.429 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-11T08:11:17.429 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBJop9l4q8hJRAAK8mAYVRQMoWS6wBOU69HmA==", 2024-01-11T08:11:17.429 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.429 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:11:17.429 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:11:17.429 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:11:17.430 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.430 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.430 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.430 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-11T08:11:17.430 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBMop9lqMF9KBAAFgFFI2GWt9AGif4AoScThw==", 2024-01-11T08:11:17.430 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.431 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:11:17.431 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:11:17.431 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:11:17.431 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.431 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.431 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.432 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-11T08:11:17.432 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBPop9lN7eDLBAAKswPZFfs0g2AaSe6V+pbbQ==", 2024-01-11T08:11:17.432 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.432 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:11:17.432 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:11:17.432 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:11:17.432 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.432 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.432 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.432 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-11T08:11:17.432 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lT5okIBAABy8PbJJbA062VNmb1Y8qhg==", 2024-01-11T08:11:17.432 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.432 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:11:17.433 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:11:17.433 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:11:17.433 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:11:17.433 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.433 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.433 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-11T08:11:17.433 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lDf37LhAAqV+EabKXJEW1sS5xtu1tXw==", 2024-01-11T08:11:17.434 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.434 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:11:17.434 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:11:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:11:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:11:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-11T08:11:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lWayxJxAAcalM8QSOGtbCXxjYZGaNQA==", 2024-01-11T08:11:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:11:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:11:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:11:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:11:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.436 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.436 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-11T08:11:17.436 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lHD4tNhAAdmnFBmMrK+QhQYdpbzW0mg==", 2024-01-11T08:11:17.436 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:11:17.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:11:17.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:11:17.436 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:11:17.437 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.437 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.437 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-11T08:11:17.437 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAwop9lGPtDMxAAJ8FjrSEdCtQL8AhtAyqnTw==", 2024-01-11T08:11:17.437 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:11:17.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:11:17.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:11:17.437 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:11:17.437 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.437 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-mds", 2024-01-11T08:11:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBcop9lFnbCKBAAXXMxeWS1kvU54G/WttPUww==", 2024-01-11T08:11:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-mds" 2024-01-11T08:11:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-mgr", 2024-01-11T08:11:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBcop9lI4rCKBAAH4Byl7JamKpaSx4idNwB5g==", 2024-01-11T08:11:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-mgr" 2024-01-11T08:11:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.438 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-osd", 2024-01-11T08:11:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBcop9lLZ3CKBAAcm8fiqCtUjvA0zdTX3onoQ==", 2024-01-11T08:11:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-osd" 2024-01-11T08:11:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-rbd", 2024-01-11T08:11:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBcop9luq3CKBAAE/9TdO7N2SuYgAEZ7yrvug==", 2024-01-11T08:11:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-rbd" 2024-01-11T08:11:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.439 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.440 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-01-11T08:11:17.440 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBcop9luL7CKBAAobK+KLDeibdHIWRPMp9V/Q==", 2024-01-11T08:11:17.440 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.440 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-01-11T08:11:17.440 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.440 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.440 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.440 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-rgw", 2024-01-11T08:11:17.440 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBcop9l/M/CKBAAToEhf+S5GJPTdzM/3bvPbg==", 2024-01-11T08:11:17.440 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.440 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-rgw" 2024-01-11T08:11:17.440 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.440 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-11T08:11:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lXBZLJxAAXa1S8Vkm3MiV4igtNbg/1Q==", 2024-01-11T08:11:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:11:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T08:11:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:11:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:11:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-11T08:11:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l3ZGoHxAANqzytTTvWYSomG+RFsFDyg==", 2024-01-11T08:11:17.441 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:11:17.442 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:11:17.442 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T08:11:17.442 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:11:17.442 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.442 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:17.442 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-11T08:11:17.442 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T08:11:17.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.422+0000 7f5cd0bff700 1 -- 172.21.15.27:0/1884968052 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5cac066f30 msgr2=0x7f5cac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:17.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.422+0000 7f5cd0bff700 1 --2- 172.21.15.27:0/1884968052 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5cac066f30 0x7f5cac0693e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5cc00099b0 tx=0x7f5cc0017000 comp rx=0 tx=0).stop 2024-01-11T08:11:17.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.422+0000 7f5cd0bff700 1 -- 172.21.15.27:0/1884968052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ccc0b03e0 msgr2=0x7f5ccc0cbf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:17.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.422+0000 7f5cd0bff700 1 --2- 172.21.15.27:0/1884968052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ccc0b03e0 0x7f5ccc0cbf40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00ec80 tx=0x7f5cbc0091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:17.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.422+0000 7f5cd0bff700 1 -- 172.21.15.27:0/1884968052 shutdown_connections 2024-01-11T08:11:17.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.422+0000 7f5cd0bff700 1 --2- 172.21.15.27:0/1884968052 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5cac066f30 0x7f5cac0693e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:17.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.422+0000 7f5cd0bff700 1 --2- 172.21.15.27:0/1884968052 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ccc159200 0x7f5ccc157740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:17.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.422+0000 7f5cd0bff700 1 --2- 172.21.15.27:0/1884968052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ccc0b03e0 0x7f5ccc0cbf40 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:17.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.422+0000 7f5cd0bff700 1 --2- 172.21.15.27:0/1884968052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ccc0b07b0 0x7f5ccc0a7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:17.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.422+0000 7f5cd0bff700 1 -- 172.21.15.27:0/1884968052 >> 172.21.15.27:0/1884968052 conn(0x7f5ccc06ed40 msgr2=0x7f5ccc159640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:17.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.423+0000 7f5cd0bff700 1 -- 172.21.15.27:0/1884968052 shutdown_connections 2024-01-11T08:11:17.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.423+0000 7f5cd0bff700 1 -- 172.21.15.27:0/1884968052 wait complete. 2024-01-11T08:11:17.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-01-11T08:11:17.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.644+0000 7f04511ff700 1 Processor -- start 2024-01-11T08:11:17.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.645+0000 7f04511ff700 1 -- start start 2024-01-11T08:11:17.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.646+0000 7f04511ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f044c15ca10 0x7f044c15a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:17.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.646+0000 7f04511ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f044c068cd0 0x7f044c0b5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:17.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.646+0000 7f04511ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f044c0b4bd0 con 0x7f044c15cde0 2024-01-11T08:11:17.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.646+0000 7f04511ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044c15ac70 con 0x7f044c068cd0 2024-01-11T08:11:17.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.646+0000 7f04511ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044c0b3fc0 con 0x7f044c15ca10 2024-01-11T08:11:17.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.646+0000 7f044b59e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f044c068cd0 0x7f044c0b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:17.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.646+0000 7f044a59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f044c15cde0 0x7f044c0685c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57700/0 (socket says 172.21.15.27:57700) 2024-01-11T08:11:17.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.646+0000 7f044b59e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f044c068cd0 0x7f044c0b5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52034/0 (socket says 172.21.15.27:52034) 2024-01-11T08:11:17.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.646+0000 7f044a59c700 1 -- 172.21.15.27:0/4099782303 learned_addr learned my addr 172.21.15.27:0/4099782303 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:17.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.647+0000 7f044ad9d700 1 --2- 172.21.15.27:0/4099782303 >> v2:172.21.15.77:3300/0 conn(0x7f044c15ca10 0x7f044c15a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:17.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.647+0000 7f044b59e700 1 -- 172.21.15.27:0/4099782303 >> v2:172.21.15.77:3300/0 conn(0x7f044c15ca10 msgr2=0x7f044c15a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:17.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.647+0000 7f044b59e700 1 --2- 172.21.15.27:0/4099782303 >> v2:172.21.15.77:3300/0 conn(0x7f044c15ca10 0x7f044c15a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:17.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.647+0000 7f044b59e700 1 -- 172.21.15.27:0/4099782303 >> v1:172.21.15.27:6789/0 conn(0x7f044c15cde0 legacy=0x7f044c0685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:17.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.647+0000 7f044b59e700 1 -- 172.21.15.27:0/4099782303 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044c0b42a0 con 0x7f044c068cd0 2024-01-11T08:11:17.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.647+0000 7f0449d9b700 1 -- 172.21.15.27:0/4099782303 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316128681 0 0) 0x7f044c0b4bd0 con 0x7f044c15cde0 2024-01-11T08:11:17.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.647+0000 7f044b59e700 1 --2- 172.21.15.27:0/4099782303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f044c068cd0 0x7f044c0b5060 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f044c0b6630 tx=0x7f0444009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba240bd0c07a298a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:17.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.647+0000 7f0449d9b700 1 -- 172.21.15.27:0/4099782303 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044400e070 con 0x7f044c068cd0 2024-01-11T08:11:17.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.647+0000 7f0449d9b700 1 -- 172.21.15.27:0/4099782303 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044400bb60 con 0x7f044c068cd0 2024-01-11T08:11:17.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.647+0000 7f0449d9b700 1 -- 172.21.15.27:0/4099782303 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0444012430 con 0x7f044c068cd0 2024-01-11T08:11:17.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.648+0000 7f04511ff700 1 -- 172.21.15.27:0/4099782303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f044c068cd0 msgr2=0x7f044c0b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:17.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.648+0000 7f04511ff700 1 --2- 172.21.15.27:0/4099782303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f044c068cd0 0x7f044c0b5060 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f044c0b6630 tx=0x7f0444009780 comp rx=0 tx=0).stop 2024-01-11T08:11:17.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.648+0000 7f04511ff700 1 -- 172.21.15.27:0/4099782303 shutdown_connections 2024-01-11T08:11:17.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.648+0000 7f04511ff700 1 --2- 172.21.15.27:0/4099782303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f044c068cd0 0x7f044c0b5060 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:17.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.648+0000 7f04511ff700 1 --2- 172.21.15.27:0/4099782303 >> v2:172.21.15.77:3300/0 conn(0x7f044c15ca10 0x7f044c15a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:17.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.648+0000 7f04511ff700 1 -- 172.21.15.27:0/4099782303 >> 172.21.15.27:0/4099782303 conn(0x7f044c06ed40 msgr2=0x7f044c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:17.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.648+0000 7f04511ff700 1 -- 172.21.15.27:0/4099782303 shutdown_connections 2024-01-11T08:11:17.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.648+0000 7f04511ff700 1 -- 172.21.15.27:0/4099782303 wait complete. 2024-01-11T08:11:17.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.650+0000 7f04511ff700 1 Processor -- start 2024-01-11T08:11:17.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.650+0000 7f04511ff700 1 -- start start 2024-01-11T08:11:17.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.651+0000 7f04511ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f044c068cd0 0x7f044c158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:17.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.651+0000 7f04511ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f044c15ca10 0x7f044c13f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:17.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.651+0000 7f04511ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f044c15cde0 0x7f044c155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:17.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.651+0000 7f04511ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044c13f5a0 con 0x7f044c15cde0 2024-01-11T08:11:17.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.651+0000 7f04511ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044c158930 con 0x7f044c068cd0 2024-01-11T08:11:17.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.651+0000 7f04511ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044c157d20 con 0x7f044c15ca10 2024-01-11T08:11:17.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.651+0000 7f044b59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f044c15cde0 0x7f044c155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:17.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.651+0000 7f044b59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f044c15cde0 0x7f044c155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52040/0 (socket says 172.21.15.27:52040) 2024-01-11T08:11:17.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.651+0000 7f044b59e700 1 -- 172.21.15.27:0/3695807278 learned_addr learned my addr 172.21.15.27:0/3695807278 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.651+0000 7f044ad9d700 1 --2- 172.21.15.27:0/3695807278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f044c068cd0 0x7f044c158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.651+0000 7f044a59c700 1 --2- 172.21.15.27:0/3695807278 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f044c15ca10 0x7f044c13f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.651+0000 7f044b59e700 1 -- 172.21.15.27:0/3695807278 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f044c15ca10 msgr2=0x7f044c13f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.651+0000 7f044b59e700 1 --2- 172.21.15.27:0/3695807278 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f044c15ca10 0x7f044c13f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.652+0000 7f044b59e700 1 -- 172.21.15.27:0/3695807278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f044c068cd0 msgr2=0x7f044c158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.652+0000 7f044b59e700 1 --2- 172.21.15.27:0/3695807278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f044c068cd0 0x7f044c158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.652+0000 7f044b59e700 1 -- 172.21.15.27:0/3695807278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0444009a20 con 0x7f044c15cde0 2024-01-11T08:11:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.652+0000 7f044ad9d700 1 --2- 172.21.15.27:0/3695807278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f044c068cd0 0x7f044c158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.652+0000 7f044b59e700 1 --2- 172.21.15.27:0/3695807278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f044c15cde0 0x7f044c155180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0444013000 tx=0x7f0444019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.652+0000 7f0432ffd700 1 -- 172.21.15.27:0/3695807278 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0444012e60 con 0x7f044c15cde0 2024-01-11T08:11:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.652+0000 7f0432ffd700 1 -- 172.21.15.27:0/3695807278 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044401d630 con 0x7f044c15cde0 2024-01-11T08:11:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.652+0000 7f04511ff700 1 -- 172.21.15.27:0/3695807278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f044c158000 con 0x7f044c15cde0 2024-01-11T08:11:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.652+0000 7f0432ffd700 1 -- 172.21.15.27:0/3695807278 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044401d9f0 con 0x7f044c15cde0 2024-01-11T08:11:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.652+0000 7f04511ff700 1 -- 172.21.15.27:0/3695807278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f044c1585c0 con 0x7f044c15cde0 2024-01-11T08:11:17.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.653+0000 7f04511ff700 1 -- 172.21.15.27:0/3695807278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044c158aa0 con 0x7f044c15cde0 2024-01-11T08:11:17.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.654+0000 7f0432ffd700 1 -- 172.21.15.27:0/3695807278 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0444024070 con 0x7f044c15cde0 2024-01-11T08:11:17.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.655+0000 7f0432ffd700 1 --2- 172.21.15.27:0/3695807278 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0438066f30 0x7f04380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:17.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.655+0000 7f0432ffd700 1 -- 172.21.15.27:0/3695807278 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f04440869f0 con 0x7f044c15cde0 2024-01-11T08:11:17.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.656+0000 7f044ad9d700 1 --2- 172.21.15.27:0/3695807278 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0438066f30 0x7f04380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:17.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.656+0000 7f044ad9d700 1 --2- 172.21.15.27:0/3695807278 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0438066f30 0x7f04380693e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f043c009d80 tx=0x7f043c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:17.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.660+0000 7f0432ffd700 1 -- 172.21.15.27:0/3695807278 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0444053fb0 con 0x7f044c15cde0 2024-01-11T08:11:17.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.944+0000 7f04511ff700 1 -- 172.21.15.27:0/3695807278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f044c04c3c0 con 0x7f044c15cde0 2024-01-11T08:11:17.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.948+0000 7f0432ffd700 1 -- 172.21.15.27:0/3695807278 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v3) v1 ==== 90+0+0 (secure 0 0 0) 0x7f0444009880 con 0x7f044c15cde0 2024-01-11T08:11:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.950+0000 7f04511ff700 1 -- 172.21.15.27:0/3695807278 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0438066f30 msgr2=0x7f04380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.950+0000 7f04511ff700 1 --2- 172.21.15.27:0/3695807278 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0438066f30 0x7f04380693e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f043c009d80 tx=0x7f043c008040 comp rx=0 tx=0).stop 2024-01-11T08:11:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.950+0000 7f04511ff700 1 -- 172.21.15.27:0/3695807278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f044c15cde0 msgr2=0x7f044c155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.950+0000 7f04511ff700 1 --2- 172.21.15.27:0/3695807278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f044c15cde0 0x7f044c155180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0444013000 tx=0x7f0444019000 comp rx=0 tx=0).stop 2024-01-11T08:11:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.950+0000 7f04511ff700 1 -- 172.21.15.27:0/3695807278 shutdown_connections 2024-01-11T08:11:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.950+0000 7f04511ff700 1 --2- 172.21.15.27:0/3695807278 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0438066f30 0x7f04380693e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.950+0000 7f04511ff700 1 --2- 172.21.15.27:0/3695807278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f044c15cde0 0x7f044c155180 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.950+0000 7f04511ff700 1 --2- 172.21.15.27:0/3695807278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f044c068cd0 0x7f044c158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.950+0000 7f04511ff700 1 --2- 172.21.15.27:0/3695807278 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f044c15ca10 0x7f044c13f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.950+0000 7f04511ff700 1 -- 172.21.15.27:0/3695807278 >> 172.21.15.27:0/3695807278 conn(0x7f044c06ed40 msgr2=0x7f044c0cfcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.950+0000 7f04511ff700 1 -- 172.21.15.27:0/3695807278 shutdown_connections 2024-01-11T08:11:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:17.950+0000 7f04511ff700 1 -- 172.21.15.27:0/3695807278 wait complete. 2024-01-11T08:11:17.968 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-01-11T08:11:18.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.167+0000 7f61861ff700 1 Processor -- start 2024-01-11T08:11:18.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.167+0000 7f61861ff700 1 -- start start 2024-01-11T08:11:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.169+0000 7f61861ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6180068cd0 0x7f618015a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.169+0000 7f61861ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f618015cde0 0x7f61800b5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.169+0000 7f61861ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61800b4bd0 con 0x7f618015ca10 2024-01-11T08:11:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.169+0000 7f61861ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f618015ac70 con 0x7f618015cde0 2024-01-11T08:11:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.169+0000 7f61861ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61800b3fc0 con 0x7f6180068cd0 2024-01-11T08:11:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.169+0000 7f617ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f618015cde0 0x7f61800b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.169+0000 7f617ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f618015cde0 0x7f61800b5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52042/0 (socket says 172.21.15.27:52042) 2024-01-11T08:11:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.169+0000 7f617ffff700 1 -- 172.21.15.27:0/1758563473 learned_addr learned my addr 172.21.15.27:0/1758563473 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:18.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.169+0000 7f617effd700 1 --2- 172.21.15.27:0/1758563473 >> v2:172.21.15.77:3300/0 conn(0x7f6180068cd0 0x7f618015a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:18.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.169+0000 7f617ffff700 1 -- 172.21.15.27:0/1758563473 >> v2:172.21.15.77:3300/0 conn(0x7f6180068cd0 msgr2=0x7f618015a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:18.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.169+0000 7f617ffff700 1 --2- 172.21.15.27:0/1758563473 >> v2:172.21.15.77:3300/0 conn(0x7f6180068cd0 0x7f618015a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.169+0000 7f617ffff700 1 -- 172.21.15.27:0/1758563473 >> v1:172.21.15.77:6789/0 conn(0x7f618015ca10 legacy=0x7f61800685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:18.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.169+0000 7f617ffff700 1 -- 172.21.15.27:0/1758563473 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61800b42a0 con 0x7f618015cde0 2024-01-11T08:11:18.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.170+0000 7f617ffff700 1 --2- 172.21.15.27:0/1758563473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f618015cde0 0x7f61800b5060 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6178009f20 tx=0x7f6178009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=423c2564ec868716 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:18.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.170+0000 7f617e7fc700 1 -- 172.21.15.27:0/1758563473 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617800e070 con 0x7f618015cde0 2024-01-11T08:11:18.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.170+0000 7f617e7fc700 1 -- 172.21.15.27:0/1758563473 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f617800b990 con 0x7f618015cde0 2024-01-11T08:11:18.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.170+0000 7f617e7fc700 1 -- 172.21.15.27:0/1758563473 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61780092e0 con 0x7f618015cde0 2024-01-11T08:11:18.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.171+0000 7f61861ff700 1 -- 172.21.15.27:0/1758563473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f618015cde0 msgr2=0x7f61800b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:18.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.171+0000 7f61861ff700 1 --2- 172.21.15.27:0/1758563473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f618015cde0 0x7f61800b5060 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6178009f20 tx=0x7f6178009840 comp rx=0 tx=0).stop 2024-01-11T08:11:18.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.171+0000 7f61861ff700 1 -- 172.21.15.27:0/1758563473 shutdown_connections 2024-01-11T08:11:18.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.171+0000 7f61861ff700 1 --2- 172.21.15.27:0/1758563473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f618015cde0 0x7f61800b5060 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.171+0000 7f61861ff700 1 --2- 172.21.15.27:0/1758563473 >> v2:172.21.15.77:3300/0 conn(0x7f6180068cd0 0x7f618015a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.171+0000 7f61861ff700 1 -- 172.21.15.27:0/1758563473 >> 172.21.15.27:0/1758563473 conn(0x7f618006ed40 msgr2=0x7f618006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:18.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.171+0000 7f61861ff700 1 -- 172.21.15.27:0/1758563473 shutdown_connections 2024-01-11T08:11:18.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.171+0000 7f61861ff700 1 -- 172.21.15.27:0/1758563473 wait complete. 2024-01-11T08:11:18.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.172+0000 7f61861ff700 1 Processor -- start 2024-01-11T08:11:18.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.172+0000 7f61861ff700 1 -- start start 2024-01-11T08:11:18.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.173+0000 7f61861ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6180068cd0 0x7f6180157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:18.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.173+0000 7f61861ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f618015ca10 0x7f61800a8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:18.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.173+0000 7f61861ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f618015cde0 0x7f618015f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:18.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.173+0000 7f61861ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61800a8d80 con 0x7f6180068cd0 2024-01-11T08:11:18.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.173+0000 7f61861ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61800a9060 con 0x7f618015cde0 2024-01-11T08:11:18.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.173+0000 7f61861ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61800a9340 con 0x7f618015ca10 2024-01-11T08:11:18.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.173+0000 7f617f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6180068cd0 0x7f6180157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:18.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.174+0000 7f617effd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f618015ca10 0x7f61800a8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:18.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.174+0000 7f617ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f618015cde0 0x7f618015f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:18.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.174+0000 7f617f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6180068cd0 0x7f6180157b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52052/0 (socket says 172.21.15.27:52052) 2024-01-11T08:11:18.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.174+0000 7f617effd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f618015ca10 0x7f61800a8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:46066/0 (socket says 172.21.15.27:46066) 2024-01-11T08:11:18.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.174+0000 7f617f7fe700 1 -- 172.21.15.27:0/2277467389 learned_addr learned my addr 172.21.15.27:0/2277467389 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:18.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.174+0000 7f617f7fe700 1 -- 172.21.15.27:0/2277467389 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f618015ca10 msgr2=0x7f61800a8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:18.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.174+0000 7f617f7fe700 1 --2- 172.21.15.27:0/2277467389 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f618015ca10 0x7f61800a8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.174+0000 7f617f7fe700 1 -- 172.21.15.27:0/2277467389 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f618015cde0 msgr2=0x7f618015f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.174+0000 7f617f7fe700 1 --2- 172.21.15.27:0/2277467389 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f618015cde0 0x7f618015f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.174+0000 7f617f7fe700 1 -- 172.21.15.27:0/2277467389 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6178009ae0 con 0x7f6180068cd0 2024-01-11T08:11:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.174+0000 7f617f7fe700 1 --2- 172.21.15.27:0/2277467389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6180068cd0 0x7f6180157b10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f617000c590 tx=0x7f617000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.175+0000 7f61677fe700 1 -- 172.21.15.27:0/2277467389 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6170011590 con 0x7f6180068cd0 2024-01-11T08:11:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.175+0000 7f61677fe700 1 -- 172.21.15.27:0/2277467389 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6170011710 con 0x7f6180068cd0 2024-01-11T08:11:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.175+0000 7f61861ff700 1 -- 172.21.15.27:0/2277467389 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61800a9620 con 0x7f6180068cd0 2024-01-11T08:11:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.175+0000 7f61677fe700 1 -- 172.21.15.27:0/2277467389 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6170011ad0 con 0x7f6180068cd0 2024-01-11T08:11:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.175+0000 7f61861ff700 1 -- 172.21.15.27:0/2277467389 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61800aa420 con 0x7f6180068cd0 2024-01-11T08:11:18.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.176+0000 7f61861ff700 1 -- 172.21.15.27:0/2277467389 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61800a9970 con 0x7f6180068cd0 2024-01-11T08:11:18.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.177+0000 7f61677fe700 1 -- 172.21.15.27:0/2277467389 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f617001b3b0 con 0x7f6180068cd0 2024-01-11T08:11:18.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.178+0000 7f61677fe700 1 --2- 172.21.15.27:0/2277467389 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f616c066f60 0x7f616c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:18.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.178+0000 7f61677fe700 1 -- 172.21.15.27:0/2277467389 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f6170083670 con 0x7f6180068cd0 2024-01-11T08:11:18.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.178+0000 7f617effd700 1 --2- 172.21.15.27:0/2277467389 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f616c066f60 0x7f616c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:18.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.179+0000 7f617effd700 1 --2- 172.21.15.27:0/2277467389 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f616c066f60 0x7f616c069410 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f61800aef20 tx=0x7f61740091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:18.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.182+0000 7f61677fe700 1 -- 172.21.15.27:0/2277467389 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6170050c30 con 0x7f6180068cd0 2024-01-11T08:11:18.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.461+0000 7f61861ff700 1 -- 172.21.15.27:0/2277467389 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f6180158210 con 0x7f6180068cd0 2024-01-11T08:11:18.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.465+0000 7f61677fe700 1 -- 172.21.15.27:0/2277467389 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v4) v1 ==== 90+0+0 (secure 0 0 0) 0x7f6170017020 con 0x7f6180068cd0 2024-01-11T08:11:18.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.467+0000 7f61861ff700 1 -- 172.21.15.27:0/2277467389 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f616c066f60 msgr2=0x7f616c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:18.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.467+0000 7f61861ff700 1 --2- 172.21.15.27:0/2277467389 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f616c066f60 0x7f616c069410 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f61800aef20 tx=0x7f61740091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.467+0000 7f61861ff700 1 -- 172.21.15.27:0/2277467389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6180068cd0 msgr2=0x7f6180157b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:18.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.467+0000 7f61861ff700 1 --2- 172.21.15.27:0/2277467389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6180068cd0 0x7f6180157b10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f617000c590 tx=0x7f617000c8a0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.467+0000 7f61861ff700 1 -- 172.21.15.27:0/2277467389 shutdown_connections 2024-01-11T08:11:18.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.467+0000 7f61861ff700 1 --2- 172.21.15.27:0/2277467389 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f616c066f60 0x7f616c069410 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.467+0000 7f61861ff700 1 --2- 172.21.15.27:0/2277467389 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f618015cde0 0x7f618015f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.467+0000 7f61861ff700 1 --2- 172.21.15.27:0/2277467389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6180068cd0 0x7f6180157b10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.467+0000 7f61861ff700 1 --2- 172.21.15.27:0/2277467389 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f618015ca10 0x7f61800a8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.467+0000 7f61861ff700 1 -- 172.21.15.27:0/2277467389 >> 172.21.15.27:0/2277467389 conn(0x7f618006ed40 msgr2=0x7f61800b5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:18.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.468+0000 7f61861ff700 1 -- 172.21.15.27:0/2277467389 shutdown_connections 2024-01-11T08:11:18.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.468+0000 7f61861ff700 1 -- 172.21.15.27:0/2277467389 wait complete. 2024-01-11T08:11:18.486 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-01-11T08:11:18.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.686+0000 7f23009ff700 1 Processor -- start 2024-01-11T08:11:18.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.686+0000 7f23009ff700 1 -- start start 2024-01-11T08:11:18.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.688+0000 7f23009ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f22fc1596a0 0x7f22fc0abd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:18.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.688+0000 7f23009ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22fc068ba0 0x7f22fc0af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:18.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.688+0000 7f23009ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22fc0ac250 con 0x7f22fc0afdf0 2024-01-11T08:11:18.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.688+0000 7f23009ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22fc067b00 con 0x7f22fc1596a0 2024-01-11T08:11:18.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.688+0000 7f23009ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc06bf60 con 0x7f22fc068ba0 2024-01-11T08:11:18.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.688+0000 7f22fa59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f22fc1596a0 0x7f22fc0abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:18.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.688+0000 7f22fa59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f22fc1596a0 0x7f22fc0abd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52066/0 (socket says 172.21.15.27:52066) 2024-01-11T08:11:18.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.688+0000 7f22fa59c700 1 -- 172.21.15.27:0/621382289 learned_addr learned my addr 172.21.15.27:0/621382289 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:18.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.688+0000 7f22f9d9b700 1 --2- 172.21.15.27:0/621382289 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22fc068ba0 0x7f22fc0af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:18.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.688+0000 7f22fa59c700 1 -- 172.21.15.27:0/621382289 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22fc068ba0 msgr2=0x7f22fc0af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:18.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.688+0000 7f22fa59c700 1 --2- 172.21.15.27:0/621382289 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22fc068ba0 0x7f22fc0af8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.689+0000 7f22fa59c700 1 -- 172.21.15.27:0/621382289 >> v1:172.21.15.77:6789/0 conn(0x7f22fc0afdf0 legacy=0x7f22fc067f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:18.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.689+0000 7f22fa59c700 1 -- 172.21.15.27:0/621382289 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22fc06c240 con 0x7f22fc1596a0 2024-01-11T08:11:18.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.689+0000 7f22fa59c700 1 --2- 172.21.15.27:0/621382289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc1596a0 0x7f22fc0abd10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0099b0 tx=0x7f22ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f8ecca69217c1a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:18.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.689+0000 7f22f959a700 1 -- 172.21.15.27:0/621382289 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec009ea0 con 0x7f22fc1596a0 2024-01-11T08:11:18.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.689+0000 7f22f959a700 1 -- 172.21.15.27:0/621382289 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22ec00b860 con 0x7f22fc1596a0 2024-01-11T08:11:18.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.689+0000 7f22f959a700 1 -- 172.21.15.27:0/621382289 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec00bc00 con 0x7f22fc1596a0 2024-01-11T08:11:18.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.690+0000 7f23009ff700 1 -- 172.21.15.27:0/621382289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc1596a0 msgr2=0x7f22fc0abd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:18.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.690+0000 7f23009ff700 1 --2- 172.21.15.27:0/621382289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc1596a0 0x7f22fc0abd10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0099b0 tx=0x7f22ec009250 comp rx=0 tx=0).stop 2024-01-11T08:11:18.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.690+0000 7f23009ff700 1 -- 172.21.15.27:0/621382289 shutdown_connections 2024-01-11T08:11:18.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.690+0000 7f23009ff700 1 --2- 172.21.15.27:0/621382289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc1596a0 0x7f22fc0abd10 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.690+0000 7f23009ff700 1 --2- 172.21.15.27:0/621382289 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22fc068ba0 0x7f22fc0af8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.690+0000 7f23009ff700 1 -- 172.21.15.27:0/621382289 >> 172.21.15.27:0/621382289 conn(0x7f22fc06a8b0 msgr2=0x7f22fc0ae520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:18.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.690+0000 7f23009ff700 1 -- 172.21.15.27:0/621382289 shutdown_connections 2024-01-11T08:11:18.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.690+0000 7f23009ff700 1 -- 172.21.15.27:0/621382289 wait complete. 2024-01-11T08:11:18.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.691+0000 7f23009ff700 1 Processor -- start 2024-01-11T08:11:18.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.691+0000 7f23009ff700 1 -- start start 2024-01-11T08:11:18.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.692+0000 7f23009ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22fc068ba0 0x7f22fc169ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:18.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.692+0000 7f23009ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22fc0afdf0 0x7f22fc1494c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:18.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.692+0000 7f23009ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc1596a0 0x7f22fc0b1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:18.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.692+0000 7f23009ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc156ec0 con 0x7f22fc1596a0 2024-01-11T08:11:18.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.692+0000 7f23009ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc0acaa0 con 0x7f22fc068ba0 2024-01-11T08:11:18.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.692+0000 7f23009ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc16a2e0 con 0x7f22fc0afdf0 2024-01-11T08:11:18.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.693+0000 7f22fa59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22fc068ba0 0x7f22fc169ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:18.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.693+0000 7f22fad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc1596a0 0x7f22fc0b1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:18.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.693+0000 7f22fad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc1596a0 0x7f22fc0b1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52072/0 (socket says 172.21.15.27:52072) 2024-01-11T08:11:18.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.693+0000 7f22fad9d700 1 -- 172.21.15.27:0/2523124951 learned_addr learned my addr 172.21.15.27:0/2523124951 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:18.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.693+0000 7f22f9d9b700 1 --2- 172.21.15.27:0/2523124951 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22fc0afdf0 0x7f22fc1494c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:18.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.693+0000 7f22fad9d700 1 -- 172.21.15.27:0/2523124951 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22fc0afdf0 msgr2=0x7f22fc1494c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:18.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.693+0000 7f22fad9d700 1 --2- 172.21.15.27:0/2523124951 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22fc0afdf0 0x7f22fc1494c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.693+0000 7f22fad9d700 1 -- 172.21.15.27:0/2523124951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22fc068ba0 msgr2=0x7f22fc169ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:18.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.693+0000 7f22fad9d700 1 --2- 172.21.15.27:0/2523124951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22fc068ba0 0x7f22fc169ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.693+0000 7f22fad9d700 1 -- 172.21.15.27:0/2523124951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ec0094f0 con 0x7f22fc1596a0 2024-01-11T08:11:18.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.693+0000 7f22fad9d700 1 --2- 172.21.15.27:0/2523124951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc1596a0 0x7f22fc0b1990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f22f400c8e0 tx=0x7f22f4001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:18.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.693+0000 7f22ea7fc700 1 -- 172.21.15.27:0/2523124951 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4012630 con 0x7f22fc1596a0 2024-01-11T08:11:18.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.694+0000 7f22ea7fc700 1 -- 172.21.15.27:0/2523124951 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f4012790 con 0x7f22fc1596a0 2024-01-11T08:11:18.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.694+0000 7f22ea7fc700 1 -- 172.21.15.27:0/2523124951 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4012b50 con 0x7f22fc1596a0 2024-01-11T08:11:18.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.694+0000 7f23009ff700 1 -- 172.21.15.27:0/2523124951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22fc16a5c0 con 0x7f22fc1596a0 2024-01-11T08:11:18.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.694+0000 7f23009ff700 1 -- 172.21.15.27:0/2523124951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22fc16a000 con 0x7f22fc1596a0 2024-01-11T08:11:18.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.695+0000 7f23009ff700 1 -- 172.21.15.27:0/2523124951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22fc0ac870 con 0x7f22fc1596a0 2024-01-11T08:11:18.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.696+0000 7f22ea7fc700 1 -- 172.21.15.27:0/2523124951 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f22f401a3a0 con 0x7f22fc1596a0 2024-01-11T08:11:18.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.698+0000 7f22ea7fc700 1 --2- 172.21.15.27:0/2523124951 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f22dc066f30 0x7f22dc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:18.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.698+0000 7f22ea7fc700 1 -- 172.21.15.27:0/2523124951 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f22f4084350 con 0x7f22fc1596a0 2024-01-11T08:11:18.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.698+0000 7f22fa59c700 1 --2- 172.21.15.27:0/2523124951 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f22dc066f30 0x7f22dc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:18.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.699+0000 7f22fa59c700 1 --2- 172.21.15.27:0/2523124951 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f22dc066f30 0x7f22dc0693e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0099b0 tx=0x7f22ec017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:18.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.702+0000 7f22ea7fc700 1 -- 172.21.15.27:0/2523124951 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f22fc0ac870 con 0x7f22fc1596a0 2024-01-11T08:11:18.831 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:11:18.836 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:11:18.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.986+0000 7f23009ff700 1 -- 172.21.15.27:0/2523124951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f22fc0ac870 con 0x7f22fc1596a0 2024-01-11T08:11:18.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.990+0000 7f22ea7fc700 1 -- 172.21.15.27:0/2523124951 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f22fc0ac870 con 0x7f22fc1596a0 2024-01-11T08:11:18.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.993+0000 7f23009ff700 1 -- 172.21.15.27:0/2523124951 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f22dc066f30 msgr2=0x7f22dc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:18.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.993+0000 7f23009ff700 1 --2- 172.21.15.27:0/2523124951 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f22dc066f30 0x7f22dc0693e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0099b0 tx=0x7f22ec017000 comp rx=0 tx=0).stop 2024-01-11T08:11:18.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.993+0000 7f23009ff700 1 -- 172.21.15.27:0/2523124951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc1596a0 msgr2=0x7f22fc0b1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:18.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.993+0000 7f23009ff700 1 --2- 172.21.15.27:0/2523124951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc1596a0 0x7f22fc0b1990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f22f400c8e0 tx=0x7f22f4001710 comp rx=0 tx=0).stop 2024-01-11T08:11:18.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.993+0000 7f23009ff700 1 -- 172.21.15.27:0/2523124951 shutdown_connections 2024-01-11T08:11:18.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.993+0000 7f23009ff700 1 --2- 172.21.15.27:0/2523124951 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f22dc066f30 0x7f22dc0693e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.993+0000 7f23009ff700 1 --2- 172.21.15.27:0/2523124951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc1596a0 0x7f22fc0b1990 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.993+0000 7f23009ff700 1 --2- 172.21.15.27:0/2523124951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22fc068ba0 0x7f22fc169ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.993+0000 7f23009ff700 1 --2- 172.21.15.27:0/2523124951 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22fc0afdf0 0x7f22fc1494c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:18.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.993+0000 7f23009ff700 1 -- 172.21.15.27:0/2523124951 >> 172.21.15.27:0/2523124951 conn(0x7f22fc06a8b0 msgr2=0x7f22fc0b0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:18.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.993+0000 7f23009ff700 1 -- 172.21.15.27:0/2523124951 shutdown_connections 2024-01-11T08:11:18.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:18.993+0000 7f23009ff700 1 -- 172.21.15.27:0/2523124951 wait complete. 2024-01-11T08:11:19.011 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-01-11T08:11:19.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.211+0000 7f0a387ff700 1 Processor -- start 2024-01-11T08:11:19.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.211+0000 7f0a387ff700 1 -- start start 2024-01-11T08:11:19.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.212+0000 7f0a387ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a3015ca10 0x7f0a3015a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:19.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.212+0000 7f0a387ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0a30068cd0 0x7f0a300b5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:19.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.212+0000 7f0a387ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a300b4bd0 con 0x7f0a3015cde0 2024-01-11T08:11:19.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.212+0000 7f0a387ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a3015ac70 con 0x7f0a30068cd0 2024-01-11T08:11:19.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.212+0000 7f0a387ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a300b3fc0 con 0x7f0a3015ca10 2024-01-11T08:11:19.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.213+0000 7f0a3659b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a3015ca10 0x7f0a3015a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:19.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.213+0000 7f0a36d9c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0a30068cd0 0x7f0a300b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:19.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.213+0000 7f0a35d9a700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0a3015cde0 0x7f0a300685c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:50320/0 (socket says 172.21.15.27:50320) 2024-01-11T08:11:19.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.213+0000 7f0a3659b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a3015ca10 0x7f0a3015a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:46098/0 (socket says 172.21.15.27:46098) 2024-01-11T08:11:19.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.213+0000 7f0a36d9c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0a30068cd0 0x7f0a300b5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:49156/0 (socket says 172.21.15.27:49156) 2024-01-11T08:11:19.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.213+0000 7f0a35d9a700 1 -- 172.21.15.27:0/4128348084 learned_addr learned my addr 172.21.15.27:0/4128348084 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:19.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.213+0000 7f0a36d9c700 1 -- 172.21.15.27:0/4128348084 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a3015ca10 msgr2=0x7f0a3015a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:19.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.213+0000 7f0a36d9c700 1 --2- 172.21.15.27:0/4128348084 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a3015ca10 0x7f0a3015a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:19.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.213+0000 7f0a36d9c700 1 -- 172.21.15.27:0/4128348084 >> v1:172.21.15.77:6789/0 conn(0x7f0a3015cde0 legacy=0x7f0a300685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:19.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.213+0000 7f0a36d9c700 1 -- 172.21.15.27:0/4128348084 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a300b42a0 con 0x7f0a30068cd0 2024-01-11T08:11:19.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.213+0000 7f0a3659b700 1 --2- 172.21.15.27:0/4128348084 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a3015ca10 0x7f0a3015a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:19.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.213+0000 7f0a35599700 1 -- 172.21.15.27:0/4128348084 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356099698 0 0) 0x7f0a300b4bd0 con 0x7f0a3015cde0 2024-01-11T08:11:19.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.214+0000 7f0a36d9c700 1 --2- 172.21.15.27:0/4128348084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a30068cd0 0x7f0a300b5060 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0a28009e60 tx=0x7f0a28009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=899743c02d697d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:19.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.214+0000 7f0a35599700 1 -- 172.21.15.27:0/4128348084 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2800e070 con 0x7f0a30068cd0 2024-01-11T08:11:19.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.214+0000 7f0a35599700 1 -- 172.21.15.27:0/4128348084 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a2800bb60 con 0x7f0a30068cd0 2024-01-11T08:11:19.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.214+0000 7f0a35599700 1 -- 172.21.15.27:0/4128348084 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a28012430 con 0x7f0a30068cd0 2024-01-11T08:11:19.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.215+0000 7f0a387ff700 1 -- 172.21.15.27:0/4128348084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a30068cd0 msgr2=0x7f0a300b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:19.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.215+0000 7f0a387ff700 1 --2- 172.21.15.27:0/4128348084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a30068cd0 0x7f0a300b5060 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0a28009e60 tx=0x7f0a28009780 comp rx=0 tx=0).stop 2024-01-11T08:11:19.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.215+0000 7f0a387ff700 1 -- 172.21.15.27:0/4128348084 shutdown_connections 2024-01-11T08:11:19.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.215+0000 7f0a387ff700 1 --2- 172.21.15.27:0/4128348084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a30068cd0 0x7f0a300b5060 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:19.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.215+0000 7f0a387ff700 1 --2- 172.21.15.27:0/4128348084 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a3015ca10 0x7f0a3015a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:19.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.215+0000 7f0a387ff700 1 -- 172.21.15.27:0/4128348084 >> 172.21.15.27:0/4128348084 conn(0x7f0a3006ed40 msgr2=0x7f0a3006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:19.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.215+0000 7f0a387ff700 1 -- 172.21.15.27:0/4128348084 shutdown_connections 2024-01-11T08:11:19.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.215+0000 7f0a387ff700 1 -- 172.21.15.27:0/4128348084 wait complete. 2024-01-11T08:11:19.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.216+0000 7f0a387ff700 1 Processor -- start 2024-01-11T08:11:19.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.216+0000 7f0a387ff700 1 -- start start 2024-01-11T08:11:19.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.217+0000 7f0a387ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a30068cd0 0x7f0a300b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:19.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.217+0000 7f0a387ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a3015ca10 0x7f0a301573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:19.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.217+0000 7f0a387ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a3015cde0 0x7f0a30157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:19.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.217+0000 7f0a387ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a30158160 con 0x7f0a30068cd0 2024-01-11T08:11:19.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.217+0000 7f0a387ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a30158440 con 0x7f0a3015cde0 2024-01-11T08:11:19.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.217+0000 7f0a387ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a30158720 con 0x7f0a3015ca10 2024-01-11T08:11:19.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.218+0000 7f0a3659b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a30068cd0 0x7f0a300b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:19.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.218+0000 7f0a3659b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a30068cd0 0x7f0a300b5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52076/0 (socket says 172.21.15.27:52076) 2024-01-11T08:11:19.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.218+0000 7f0a3659b700 1 -- 172.21.15.27:0/164935844 learned_addr learned my addr 172.21.15.27:0/164935844 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:19.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.218+0000 7f0a35d9a700 1 --2- 172.21.15.27:0/164935844 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a3015ca10 0x7f0a301573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:19.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.218+0000 7f0a36d9c700 1 --2- 172.21.15.27:0/164935844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a3015cde0 0x7f0a30157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:19.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.218+0000 7f0a3659b700 1 -- 172.21.15.27:0/164935844 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a3015ca10 msgr2=0x7f0a301573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:19.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.218+0000 7f0a3659b700 1 --2- 172.21.15.27:0/164935844 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a3015ca10 0x7f0a301573e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.218+0000 7f0a3659b700 1 -- 172.21.15.27:0/164935844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a3015cde0 msgr2=0x7f0a30157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.218+0000 7f0a3659b700 1 --2- 172.21.15.27:0/164935844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a3015cde0 0x7f0a30157920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.218+0000 7f0a3659b700 1 -- 172.21.15.27:0/164935844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2c009420 con 0x7f0a30068cd0 2024-01-11T08:11:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.218+0000 7f0a35d9a700 1 --2- 172.21.15.27:0/164935844 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a3015ca10 0x7f0a301573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.218+0000 7f0a3659b700 1 --2- 172.21.15.27:0/164935844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a30068cd0 0x7f0a300b5cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c00ec80 tx=0x7f0a2c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.219+0000 7f0a267fc700 1 -- 172.21.15.27:0/164935844 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c00c8f0 con 0x7f0a30068cd0 2024-01-11T08:11:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.219+0000 7f0a267fc700 1 -- 172.21.15.27:0/164935844 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a2c00ca50 con 0x7f0a30068cd0 2024-01-11T08:11:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.219+0000 7f0a267fc700 1 -- 172.21.15.27:0/164935844 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c00cdd0 con 0x7f0a30068cd0 2024-01-11T08:11:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.219+0000 7f0a387ff700 1 -- 172.21.15.27:0/164935844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a28009a20 con 0x7f0a30068cd0 2024-01-11T08:11:19.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.219+0000 7f0a387ff700 1 -- 172.21.15.27:0/164935844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a30158ce0 con 0x7f0a30068cd0 2024-01-11T08:11:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.220+0000 7f0a387ff700 1 -- 172.21.15.27:0/164935844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a30159050 con 0x7f0a30068cd0 2024-01-11T08:11:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.221+0000 7f0a267fc700 1 -- 172.21.15.27:0/164935844 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0a2c013680 con 0x7f0a30068cd0 2024-01-11T08:11:19.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.222+0000 7f0a267fc700 1 --2- 172.21.15.27:0/164935844 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0a14066f30 0x7f0a140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:19.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.222+0000 7f0a267fc700 1 -- 172.21.15.27:0/164935844 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f0a2c0848b0 con 0x7f0a30068cd0 2024-01-11T08:11:19.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.223+0000 7f0a35d9a700 1 --2- 172.21.15.27:0/164935844 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0a14066f30 0x7f0a140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.223+0000 7f0a35d9a700 1 --2- 172.21.15.27:0/164935844 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0a14066f30 0x7f0a140693e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0a30150650 tx=0x7f0a20009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.226+0000 7f0a267fc700 1 -- 172.21.15.27:0/164935844 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0a2c051e70 con 0x7f0a30068cd0 2024-01-11T08:11:19.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.511+0000 7f0a387ff700 1 -- 172.21.15.27:0/164935844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f0a30157ec0 con 0x7f0a30068cd0 2024-01-11T08:11:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.515+0000 7f0a267fc700 1 -- 172.21.15.27:0/164935844 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f0a2c0170a0 con 0x7f0a30068cd0 2024-01-11T08:11:19.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.517+0000 7f0a387ff700 1 -- 172.21.15.27:0/164935844 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0a14066f30 msgr2=0x7f0a140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:19.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.517+0000 7f0a387ff700 1 --2- 172.21.15.27:0/164935844 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0a14066f30 0x7f0a140693e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0a30150650 tx=0x7f0a20009040 comp rx=0 tx=0).stop 2024-01-11T08:11:19.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.517+0000 7f0a387ff700 1 -- 172.21.15.27:0/164935844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a30068cd0 msgr2=0x7f0a300b5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.517+0000 7f0a387ff700 1 --2- 172.21.15.27:0/164935844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a30068cd0 0x7f0a300b5cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c00ec80 tx=0x7f0a2c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.517+0000 7f0a387ff700 1 -- 172.21.15.27:0/164935844 shutdown_connections 2024-01-11T08:11:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.517+0000 7f0a387ff700 1 --2- 172.21.15.27:0/164935844 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0a14066f30 0x7f0a140693e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.517+0000 7f0a387ff700 1 --2- 172.21.15.27:0/164935844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a3015cde0 0x7f0a30157920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.517+0000 7f0a387ff700 1 --2- 172.21.15.27:0/164935844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a30068cd0 0x7f0a300b5cc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.517+0000 7f0a387ff700 1 --2- 172.21.15.27:0/164935844 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a3015ca10 0x7f0a301573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.517+0000 7f0a387ff700 1 -- 172.21.15.27:0/164935844 >> 172.21.15.27:0/164935844 conn(0x7f0a3006ed40 msgr2=0x7f0a30152db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.517+0000 7f0a387ff700 1 -- 172.21.15.27:0/164935844 shutdown_connections 2024-01-11T08:11:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.518+0000 7f0a387ff700 1 -- 172.21.15.27:0/164935844 wait complete. 2024-01-11T08:11:19.536 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-01-11T08:11:19.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.735+0000 7f59719ff700 1 Processor -- start 2024-01-11T08:11:19.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.735+0000 7f59719ff700 1 -- start start 2024-01-11T08:11:19.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.736+0000 7f59719ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f596c0b0360 0x7f596c160950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:19.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.736+0000 7f59719ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596c0aeb50 con 0x7f596c155e30 2024-01-11T08:11:19.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.736+0000 7f59719ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596c0abf20 con 0x7f596c155a60 2024-01-11T08:11:19.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.736+0000 7f59719ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596c0ae590 con 0x7f596c0b0360 2024-01-11T08:11:19.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.737+0000 7f596a7fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f596c155e30 0x7f596c0afc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57708/0 (socket says 172.21.15.27:57708) 2024-01-11T08:11:19.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.737+0000 7f596a7fc700 1 -- 172.21.15.27:0/3094676196 learned_addr learned my addr 172.21.15.27:0/3094676196 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:19.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.737+0000 7f596b7fe700 1 --2- 172.21.15.27:0/3094676196 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f596c0b0360 0x7f596c160950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:19.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.737+0000 7f5969ffb700 1 -- 172.21.15.27:0/3094676196 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709147439 0 0) 0x7f596c0aeb50 con 0x7f596c155e30 2024-01-11T08:11:19.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.737+0000 7f5969ffb700 1 -- 172.21.15.27:0/3094676196 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5954004680 con 0x7f596c155e30 2024-01-11T08:11:19.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.737+0000 7f5969ffb700 1 -- 172.21.15.27:0/3094676196 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123530420 0 0) 0x7f596c0abf20 con 0x7f596c155a60 2024-01-11T08:11:19.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.737+0000 7f5969ffb700 1 -- 172.21.15.27:0/3094676196 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f596c0aeb50 con 0x7f596c155a60 2024-01-11T08:11:19.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.737+0000 7f596b7fe700 1 -- 172.21.15.27:0/3094676196 >> v1:172.21.15.77:6789/0 conn(0x7f596c155a60 legacy=0x7f596c0aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:19.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.737+0000 7f596b7fe700 1 -- 172.21.15.27:0/3094676196 >> v1:172.21.15.27:6789/0 conn(0x7f596c155e30 legacy=0x7f596c0afc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:19.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.737+0000 7f596b7fe700 1 -- 172.21.15.27:0/3094676196 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c0ae870 con 0x7f596c0b0360 2024-01-11T08:11:19.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.738+0000 7f596b7fe700 1 --2- 172.21.15.27:0/3094676196 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f596c0b0360 0x7f596c160950 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5964009e90 tx=0x7f5964009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fac245de65ece12b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:19.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.738+0000 7f5969ffb700 1 -- 172.21.15.27:0/3094676196 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596400e070 con 0x7f596c0b0360 2024-01-11T08:11:19.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.738+0000 7f5969ffb700 1 -- 172.21.15.27:0/3094676196 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596400bb80 con 0x7f596c0b0360 2024-01-11T08:11:19.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.738+0000 7f5969ffb700 1 -- 172.21.15.27:0/3094676196 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5964012430 con 0x7f596c0b0360 2024-01-11T08:11:19.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.739+0000 7f59719ff700 1 -- 172.21.15.27:0/3094676196 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f596c0b0360 msgr2=0x7f596c160950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:19.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.739+0000 7f59719ff700 1 --2- 172.21.15.27:0/3094676196 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f596c0b0360 0x7f596c160950 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5964009e90 tx=0x7f5964009780 comp rx=0 tx=0).stop 2024-01-11T08:11:19.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.739+0000 7f59719ff700 1 -- 172.21.15.27:0/3094676196 shutdown_connections 2024-01-11T08:11:19.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.739+0000 7f59719ff700 1 --2- 172.21.15.27:0/3094676196 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f596c0b0360 0x7f596c160950 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:19.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.739+0000 7f59719ff700 1 -- 172.21.15.27:0/3094676196 >> 172.21.15.27:0/3094676196 conn(0x7f596c0680b0 msgr2=0x7f596c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:19.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.739+0000 7f59719ff700 1 -- 172.21.15.27:0/3094676196 shutdown_connections 2024-01-11T08:11:19.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.739+0000 7f59719ff700 1 -- 172.21.15.27:0/3094676196 wait complete. 2024-01-11T08:11:19.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.740+0000 7f59719ff700 1 Processor -- start 2024-01-11T08:11:19.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.740+0000 7f59719ff700 1 -- start start 2024-01-11T08:11:19.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.741+0000 7f59719ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f596c0b0360 0x7f596c149220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:19.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.741+0000 7f59719ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f596c155a60 0x7f596c146c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:19.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.741+0000 7f59719ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c155e30 0x7f596c157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:19.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.741+0000 7f59719ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c14ef80 con 0x7f596c155e30 2024-01-11T08:11:19.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.741+0000 7f59719ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c14f260 con 0x7f596c0b0360 2024-01-11T08:11:19.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.741+0000 7f59719ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596c14f540 con 0x7f596c155a60 2024-01-11T08:11:19.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.742+0000 7f596b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c155e30 0x7f596c157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:19.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.742+0000 7f596b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c155e30 0x7f596c157040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52084/0 (socket says 172.21.15.27:52084) 2024-01-11T08:11:19.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.742+0000 7f596b7fe700 1 -- 172.21.15.27:0/3938055440 learned_addr learned my addr 172.21.15.27:0/3938055440 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:19.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.742+0000 7f596a7fc700 1 --2- 172.21.15.27:0/3938055440 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f596c155a60 0x7f596c146c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:19.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.742+0000 7f596affd700 1 --2- 172.21.15.27:0/3938055440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f596c0b0360 0x7f596c149220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:19.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.742+0000 7f596b7fe700 1 -- 172.21.15.27:0/3938055440 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f596c155a60 msgr2=0x7f596c146c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:19.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.742+0000 7f596b7fe700 1 --2- 172.21.15.27:0/3938055440 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f596c155a60 0x7f596c146c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:19.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.742+0000 7f596b7fe700 1 -- 172.21.15.27:0/3938055440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f596c0b0360 msgr2=0x7f596c149220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:19.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.742+0000 7f596b7fe700 1 --2- 172.21.15.27:0/3938055440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f596c0b0360 0x7f596c149220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:19.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.742+0000 7f596b7fe700 1 -- 172.21.15.27:0/3938055440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5964009a20 con 0x7f596c155e30 2024-01-11T08:11:19.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.742+0000 7f596a7fc700 1 --2- 172.21.15.27:0/3938055440 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f596c155a60 0x7f596c146c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:19.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.742+0000 7f596b7fe700 1 --2- 172.21.15.27:0/3938055440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c155e30 0x7f596c157040 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5964013000 tx=0x7f5964019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:19.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.743+0000 7f5952ffd700 1 -- 172.21.15.27:0/3938055440 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5964012e00 con 0x7f596c155e30 2024-01-11T08:11:19.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.743+0000 7f5952ffd700 1 -- 172.21.15.27:0/3938055440 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596401b630 con 0x7f596c155e30 2024-01-11T08:11:19.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.743+0000 7f5952ffd700 1 -- 172.21.15.27:0/3938055440 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596401b9f0 con 0x7f596c155e30 2024-01-11T08:11:19.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.743+0000 7f59719ff700 1 -- 172.21.15.27:0/3938055440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596c14f820 con 0x7f596c155e30 2024-01-11T08:11:19.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.743+0000 7f59719ff700 1 -- 172.21.15.27:0/3938055440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f596c149760 con 0x7f596c155e30 2024-01-11T08:11:19.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.744+0000 7f59719ff700 1 -- 172.21.15.27:0/3938055440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596c04c3c0 con 0x7f596c155e30 2024-01-11T08:11:19.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.745+0000 7f5952ffd700 1 -- 172.21.15.27:0/3938055440 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f596401c400 con 0x7f596c155e30 2024-01-11T08:11:19.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.746+0000 7f5952ffd700 1 --2- 172.21.15.27:0/3938055440 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5958066f30 0x7f59580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:19.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.746+0000 7f5952ffd700 1 -- 172.21.15.27:0/3938055440 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f5964085c10 con 0x7f596c155e30 2024-01-11T08:11:19.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.747+0000 7f596affd700 1 --2- 172.21.15.27:0/3938055440 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5958066f30 0x7f59580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.747+0000 7f596affd700 1 --2- 172.21.15.27:0/3938055440 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5958066f30 0x7f59580693e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f595c003870 tx=0x7f595c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:19.751+0000 7f5952ffd700 1 -- 172.21.15.27:0/3938055440 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f59640531d0 con 0x7f596c155e30 2024-01-11T08:11:20.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.034+0000 7f59719ff700 1 -- 172.21.15.27:0/3938055440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f596c0af430 con 0x7f596c155e30 2024-01-11T08:11:20.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.038+0000 7f5952ffd700 1 -- 172.21.15.27:0/3938055440 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7f59640567f0 con 0x7f596c155e30 2024-01-11T08:11:20.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.040+0000 7f59719ff700 1 -- 172.21.15.27:0/3938055440 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5958066f30 msgr2=0x7f59580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:20.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.040+0000 7f59719ff700 1 --2- 172.21.15.27:0/3938055440 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5958066f30 0x7f59580693e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f595c003870 tx=0x7f595c009040 comp rx=0 tx=0).stop 2024-01-11T08:11:20.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.040+0000 7f59719ff700 1 -- 172.21.15.27:0/3938055440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c155e30 msgr2=0x7f596c157040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:20.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.040+0000 7f59719ff700 1 --2- 172.21.15.27:0/3938055440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c155e30 0x7f596c157040 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5964013000 tx=0x7f5964019000 comp rx=0 tx=0).stop 2024-01-11T08:11:20.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.040+0000 7f59719ff700 1 -- 172.21.15.27:0/3938055440 shutdown_connections 2024-01-11T08:11:20.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.040+0000 7f59719ff700 1 --2- 172.21.15.27:0/3938055440 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5958066f30 0x7f59580693e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.040+0000 7f59719ff700 1 --2- 172.21.15.27:0/3938055440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c155e30 0x7f596c157040 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.040+0000 7f59719ff700 1 --2- 172.21.15.27:0/3938055440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f596c0b0360 0x7f596c149220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.040+0000 7f59719ff700 1 --2- 172.21.15.27:0/3938055440 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f596c155a60 0x7f596c146c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.040+0000 7f59719ff700 1 -- 172.21.15.27:0/3938055440 >> 172.21.15.27:0/3938055440 conn(0x7f596c0680b0 msgr2=0x7f596c152460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:20.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.040+0000 7f59719ff700 1 -- 172.21.15.27:0/3938055440 shutdown_connections 2024-01-11T08:11:20.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.040+0000 7f59719ff700 1 -- 172.21.15.27:0/3938055440 wait complete. 2024-01-11T08:11:20.058 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-01-11T08:11:20.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.259+0000 7fb8d8bff700 1 Processor -- start 2024-01-11T08:11:20.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.259+0000 7fb8d8bff700 1 -- start start 2024-01-11T08:11:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.260+0000 7fb8d8bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb8d4156ab0 0x7fb8d40ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.260+0000 7fb8d8bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8d40cc9b0 0x7fb8d4156e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.260+0000 7fb8d8bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8d40adc80 con 0x7fb8d41573c0 2024-01-11T08:11:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.260+0000 7fb8d8bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8d40adf60 con 0x7fb8d4156ab0 2024-01-11T08:11:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.260+0000 7fb8d8bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d40ae240 con 0x7fb8d40cc9b0 2024-01-11T08:11:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.261+0000 7fb8d259c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb8d4156ab0 0x7fb8d40ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.261+0000 7fb8d1d9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8d40cc9b0 0x7fb8d4156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.261+0000 7fb8d1d9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8d40cc9b0 0x7fb8d4156e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:46148/0 (socket says 172.21.15.27:46148) 2024-01-11T08:11:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.261+0000 7fb8d259c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb8d4156ab0 0x7fb8d40ad650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:49180/0 (socket says 172.21.15.27:49180) 2024-01-11T08:11:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.261+0000 7fb8d1d9b700 1 -- 172.21.15.27:0/143306608 learned_addr learned my addr 172.21.15.27:0/143306608 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.261+0000 7fb8d2d9d700 1 --1- 172.21.15.27:0/143306608 >> v1:172.21.15.77:6789/0 conn(0x7fb8d41573c0 0x7fb8d40af540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:50338/0 (socket says 172.21.15.27:50338) 2024-01-11T08:11:20.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.261+0000 7fb8d159a700 1 -- 172.21.15.27:0/143306608 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040105971 0 0) 0x7fb8d40adc80 con 0x7fb8d41573c0 2024-01-11T08:11:20.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.261+0000 7fb8d159a700 1 -- 172.21.15.27:0/143306608 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8bc003570 con 0x7fb8d41573c0 2024-01-11T08:11:20.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.261+0000 7fb8d1d9b700 1 -- 172.21.15.27:0/143306608 >> v2:172.21.15.77:3300/0 conn(0x7fb8d4156ab0 msgr2=0x7fb8d40ad650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:20.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.262+0000 7fb8d1d9b700 1 --2- 172.21.15.27:0/143306608 >> v2:172.21.15.77:3300/0 conn(0x7fb8d4156ab0 0x7fb8d40ad650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.262+0000 7fb8d1d9b700 1 -- 172.21.15.27:0/143306608 >> v1:172.21.15.77:6789/0 conn(0x7fb8d41573c0 legacy=0x7fb8d40af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:20.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.262+0000 7fb8d1d9b700 1 -- 172.21.15.27:0/143306608 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d40ae520 con 0x7fb8d40cc9b0 2024-01-11T08:11:20.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.262+0000 7fb8d259c700 1 --2- 172.21.15.27:0/143306608 >> v2:172.21.15.77:3300/0 conn(0x7fb8d4156ab0 0x7fb8d40ad650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:20.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.262+0000 7fb8d1d9b700 1 --2- 172.21.15.27:0/143306608 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8d40cc9b0 0x7fb8d4156e80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb8c800bc50 tx=0x7fb8c800b4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f959ea4916e4e602 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:20.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.262+0000 7fb8d159a700 1 -- 172.21.15.27:0/143306608 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c800db40 con 0x7fb8d40cc9b0 2024-01-11T08:11:20.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.262+0000 7fb8d159a700 1 -- 172.21.15.27:0/143306608 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c800dca0 con 0x7fb8d40cc9b0 2024-01-11T08:11:20.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.262+0000 7fb8d159a700 1 -- 172.21.15.27:0/143306608 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c80196b0 con 0x7fb8d40cc9b0 2024-01-11T08:11:20.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.263+0000 7fb8d8bff700 1 -- 172.21.15.27:0/143306608 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8d40cc9b0 msgr2=0x7fb8d4156e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:20.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.263+0000 7fb8d8bff700 1 --2- 172.21.15.27:0/143306608 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8d40cc9b0 0x7fb8d4156e80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb8c800bc50 tx=0x7fb8c800b4f0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.263+0000 7fb8d8bff700 1 -- 172.21.15.27:0/143306608 shutdown_connections 2024-01-11T08:11:20.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.263+0000 7fb8d8bff700 1 --2- 172.21.15.27:0/143306608 >> v2:172.21.15.77:3300/0 conn(0x7fb8d4156ab0 0x7fb8d40ad650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.263+0000 7fb8d8bff700 1 --2- 172.21.15.27:0/143306608 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8d40cc9b0 0x7fb8d4156e80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.263+0000 7fb8d8bff700 1 -- 172.21.15.27:0/143306608 >> 172.21.15.27:0/143306608 conn(0x7fb8d40680b0 msgr2=0x7fb8d40cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:20.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.263+0000 7fb8d8bff700 1 -- 172.21.15.27:0/143306608 shutdown_connections 2024-01-11T08:11:20.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.263+0000 7fb8d8bff700 1 -- 172.21.15.27:0/143306608 wait complete. 2024-01-11T08:11:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.264+0000 7fb8d8bff700 1 Processor -- start 2024-01-11T08:11:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.264+0000 7fb8d8bff700 1 -- start start 2024-01-11T08:11:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.265+0000 7fb8d8bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8d40cc9b0 0x7fb8d414c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.265+0000 7fb8d8bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8d4156ab0 0x7fb8d41586d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.265+0000 7fb8d8bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8d41573c0 0x7fb8d4069730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.265+0000 7fb8d8bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d414c650 con 0x7fb8d41573c0 2024-01-11T08:11:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.266+0000 7fb8d8bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d4158240 con 0x7fb8d40cc9b0 2024-01-11T08:11:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.266+0000 7fb8d8bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d4157910 con 0x7fb8d4156ab0 2024-01-11T08:11:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.266+0000 7fb8d2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8d41573c0 0x7fb8d4069730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.266+0000 7fb8d2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8d41573c0 0x7fb8d4069730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52090/0 (socket says 172.21.15.27:52090) 2024-01-11T08:11:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.266+0000 7fb8d2d9d700 1 -- 172.21.15.27:0/4149992213 learned_addr learned my addr 172.21.15.27:0/4149992213 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.266+0000 7fb8d259c700 1 --2- 172.21.15.27:0/4149992213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8d40cc9b0 0x7fb8d414c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.266+0000 7fb8d1d9b700 1 --2- 172.21.15.27:0/4149992213 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8d4156ab0 0x7fb8d41586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.266+0000 7fb8d2d9d700 1 -- 172.21.15.27:0/4149992213 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8d4156ab0 msgr2=0x7fb8d41586d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.266+0000 7fb8d2d9d700 1 --2- 172.21.15.27:0/4149992213 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8d4156ab0 0x7fb8d41586d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.266+0000 7fb8d2d9d700 1 -- 172.21.15.27:0/4149992213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8d40cc9b0 msgr2=0x7fb8d414c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.266+0000 7fb8d2d9d700 1 --2- 172.21.15.27:0/4149992213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8d40cc9b0 0x7fb8d414c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.266+0000 7fb8d2d9d700 1 -- 172.21.15.27:0/4149992213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c4009420 con 0x7fb8d41573c0 2024-01-11T08:11:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.267+0000 7fb8d2d9d700 1 --2- 172.21.15.27:0/4149992213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8d41573c0 0x7fb8d4069730 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc00cd30 tx=0x7fb8cc001ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.267+0000 7fb8c27fc700 1 -- 172.21.15.27:0/4149992213 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc00ce30 con 0x7fb8d41573c0 2024-01-11T08:11:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.267+0000 7fb8c27fc700 1 -- 172.21.15.27:0/4149992213 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8cc005df0 con 0x7fb8d41573c0 2024-01-11T08:11:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.267+0000 7fb8c27fc700 1 -- 172.21.15.27:0/4149992213 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc00ab70 con 0x7fb8d41573c0 2024-01-11T08:11:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.267+0000 7fb8d8bff700 1 -- 172.21.15.27:0/4149992213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c800b790 con 0x7fb8d41573c0 2024-01-11T08:11:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.267+0000 7fb8d8bff700 1 -- 172.21.15.27:0/4149992213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8d4157ed0 con 0x7fb8d41573c0 2024-01-11T08:11:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.268+0000 7fb8d8bff700 1 -- 172.21.15.27:0/4149992213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8d404c3c0 con 0x7fb8d41573c0 2024-01-11T08:11:20.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.269+0000 7fb8c27fc700 1 -- 172.21.15.27:0/4149992213 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb8cc0070f0 con 0x7fb8d41573c0 2024-01-11T08:11:20.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.270+0000 7fb8c27fc700 1 --2- 172.21.15.27:0/4149992213 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb8b4066f30 0x7fb8b40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:20.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.270+0000 7fb8c27fc700 1 -- 172.21.15.27:0/4149992213 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fb8cc084410 con 0x7fb8d41573c0 2024-01-11T08:11:20.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.270+0000 7fb8d259c700 1 --2- 172.21.15.27:0/4149992213 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb8b4066f30 0x7fb8b40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:20.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.271+0000 7fb8d259c700 1 --2- 172.21.15.27:0/4149992213 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb8b4066f30 0x7fb8b40693e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4009f40 tx=0x7fb8c40091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.274+0000 7fb8c27fc700 1 -- 172.21.15.27:0/4149992213 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb8cc051920 con 0x7fb8d41573c0 2024-01-11T08:11:20.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.553+0000 7fb8d8bff700 1 -- 172.21.15.27:0/4149992213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7fb8d4158380 con 0x7fb8d41573c0 2024-01-11T08:11:20.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.556+0000 7fb8c27fc700 1 -- 172.21.15.27:0/4149992213 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7fb8cc054f40 con 0x7fb8d41573c0 2024-01-11T08:11:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.559+0000 7fb8d8bff700 1 -- 172.21.15.27:0/4149992213 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb8b4066f30 msgr2=0x7fb8b40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.559+0000 7fb8d8bff700 1 --2- 172.21.15.27:0/4149992213 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb8b4066f30 0x7fb8b40693e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4009f40 tx=0x7fb8c40091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.559+0000 7fb8d8bff700 1 -- 172.21.15.27:0/4149992213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8d41573c0 msgr2=0x7fb8d4069730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.559+0000 7fb8d8bff700 1 --2- 172.21.15.27:0/4149992213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8d41573c0 0x7fb8d4069730 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc00cd30 tx=0x7fb8cc001ac0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.559+0000 7fb8d8bff700 1 -- 172.21.15.27:0/4149992213 shutdown_connections 2024-01-11T08:11:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.559+0000 7fb8d8bff700 1 --2- 172.21.15.27:0/4149992213 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb8b4066f30 0x7fb8b40693e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.559+0000 7fb8d8bff700 1 --2- 172.21.15.27:0/4149992213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8d41573c0 0x7fb8d4069730 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.559+0000 7fb8d8bff700 1 --2- 172.21.15.27:0/4149992213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8d40cc9b0 0x7fb8d414c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.559+0000 7fb8d8bff700 1 --2- 172.21.15.27:0/4149992213 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8d4156ab0 0x7fb8d41586d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.559+0000 7fb8d8bff700 1 -- 172.21.15.27:0/4149992213 >> 172.21.15.27:0/4149992213 conn(0x7fb8d40680b0 msgr2=0x7fb8d40af520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:20.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.559+0000 7fb8d8bff700 1 -- 172.21.15.27:0/4149992213 shutdown_connections 2024-01-11T08:11:20.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.559+0000 7fb8d8bff700 1 -- 172.21.15.27:0/4149992213 wait complete. 2024-01-11T08:11:20.576 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:11:20.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.775+0000 7fe95c5ff700 1 Processor -- start 2024-01-11T08:11:20.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.776+0000 7fe95c5ff700 1 -- start start 2024-01-11T08:11:20.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.777+0000 7fe95c5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe9540ca7b0 0x7fe9540cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:20.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.777+0000 7fe95c5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe95415d640 con 0x7fe954159e80 2024-01-11T08:11:20.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.777+0000 7fe95c5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe95415a5d0 con 0x7fe9540ca3e0 2024-01-11T08:11:20.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.777+0000 7fe95c5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe95415ae70 con 0x7fe9540ca7b0 2024-01-11T08:11:20.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.777+0000 7fe95ab9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe954159e80 0x7fe95415dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57716/0 (socket says 172.21.15.27:57716) 2024-01-11T08:11:20.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.777+0000 7fe95ab9c700 1 -- 172.21.15.27:0/2575918257 learned_addr learned my addr 172.21.15.27:0/2575918257 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:20.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.777+0000 7fe959b9a700 1 --2- 172.21.15.27:0/2575918257 >> v2:172.21.15.77:3300/0 conn(0x7fe9540ca7b0 0x7fe9540cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:20.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.778+0000 7fe959399700 1 -- 172.21.15.27:0/2575918257 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1495447508 0 0) 0x7fe95415d640 con 0x7fe954159e80 2024-01-11T08:11:20.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.778+0000 7fe959399700 1 -- 172.21.15.27:0/2575918257 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe940003570 con 0x7fe954159e80 2024-01-11T08:11:20.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.778+0000 7fe959b9a700 1 -- 172.21.15.27:0/2575918257 >> v1:172.21.15.77:6789/0 conn(0x7fe9540ca3e0 legacy=0x7fe9540c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:20.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.778+0000 7fe959b9a700 1 -- 172.21.15.27:0/2575918257 >> v1:172.21.15.27:6789/0 conn(0x7fe954159e80 legacy=0x7fe95415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:20.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.778+0000 7fe959b9a700 1 -- 172.21.15.27:0/2575918257 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95415a8b0 con 0x7fe9540ca7b0 2024-01-11T08:11:20.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.778+0000 7fe959b9a700 1 --2- 172.21.15.27:0/2575918257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9540ca7b0 0x7fe9540cca20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe944009910 tx=0x7fe9440091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebbfd88051421a9a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:20.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.779+0000 7fe959399700 1 -- 172.21.15.27:0/2575918257 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94400b640 con 0x7fe9540ca7b0 2024-01-11T08:11:20.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.779+0000 7fe959399700 1 -- 172.21.15.27:0/2575918257 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94400b7a0 con 0x7fe9540ca7b0 2024-01-11T08:11:20.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.779+0000 7fe959399700 1 -- 172.21.15.27:0/2575918257 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94400bb60 con 0x7fe9540ca7b0 2024-01-11T08:11:20.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.779+0000 7fe95c5ff700 1 -- 172.21.15.27:0/2575918257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9540ca7b0 msgr2=0x7fe9540cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:20.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.779+0000 7fe95c5ff700 1 --2- 172.21.15.27:0/2575918257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9540ca7b0 0x7fe9540cca20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe944009910 tx=0x7fe9440091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.779+0000 7fe95c5ff700 1 -- 172.21.15.27:0/2575918257 shutdown_connections 2024-01-11T08:11:20.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.779+0000 7fe95c5ff700 1 --2- 172.21.15.27:0/2575918257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9540ca7b0 0x7fe9540cca20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.779+0000 7fe95c5ff700 1 -- 172.21.15.27:0/2575918257 >> 172.21.15.27:0/2575918257 conn(0x7fe9540ce3e0 msgr2=0x7fe9540ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:20.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.779+0000 7fe95c5ff700 1 -- 172.21.15.27:0/2575918257 shutdown_connections 2024-01-11T08:11:20.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.779+0000 7fe95c5ff700 1 -- 172.21.15.27:0/2575918257 wait complete. 2024-01-11T08:11:20.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.781+0000 7fe95c5ff700 1 Processor -- start 2024-01-11T08:11:20.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.781+0000 7fe95c5ff700 1 -- start start 2024-01-11T08:11:20.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.782+0000 7fe95c5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9540ca3e0 0x7fe954151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:20.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.782+0000 7fe95c5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9540ca7b0 0x7fe95415de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:20.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.782+0000 7fe95c5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe954159e80 0x7fe95415e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:20.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.782+0000 7fe95c5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9540b27a0 con 0x7fe9540ca3e0 2024-01-11T08:11:20.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.782+0000 7fe95c5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe954151c70 con 0x7fe9540ca7b0 2024-01-11T08:11:20.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.782+0000 7fe95c5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9540b1e70 con 0x7fe954159e80 2024-01-11T08:11:20.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.782+0000 7fe95a39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9540ca3e0 0x7fe954151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:20.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.782+0000 7fe95a39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9540ca3e0 0x7fe954151730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52106/0 (socket says 172.21.15.27:52106) 2024-01-11T08:11:20.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.782+0000 7fe95a39b700 1 -- 172.21.15.27:0/2565112464 learned_addr learned my addr 172.21.15.27:0/2565112464 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:20.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.782+0000 7fe959b9a700 1 --2- 172.21.15.27:0/2565112464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9540ca7b0 0x7fe95415de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:20.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.782+0000 7fe95ab9c700 1 --2- 172.21.15.27:0/2565112464 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe954159e80 0x7fe95415e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:20.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.783+0000 7fe95a39b700 1 -- 172.21.15.27:0/2565112464 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe954159e80 msgr2=0x7fe95415e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:20.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.783+0000 7fe95a39b700 1 --2- 172.21.15.27:0/2565112464 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe954159e80 0x7fe95415e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.783+0000 7fe95a39b700 1 -- 172.21.15.27:0/2565112464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9540ca7b0 msgr2=0x7fe95415de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:20.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.783+0000 7fe95a39b700 1 --2- 172.21.15.27:0/2565112464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9540ca7b0 0x7fe95415de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:20.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.783+0000 7fe95a39b700 1 -- 172.21.15.27:0/2565112464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe944009420 con 0x7fe9540ca3e0 2024-01-11T08:11:20.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.783+0000 7fe95a39b700 1 --2- 172.21.15.27:0/2565112464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9540ca3e0 0x7fe954151730 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe95000c4e0 tx=0x7fe95000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:20.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.783+0000 7fe94a7fc700 1 -- 172.21.15.27:0/2565112464 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe950011590 con 0x7fe9540ca3e0 2024-01-11T08:11:20.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.783+0000 7fe94a7fc700 1 -- 172.21.15.27:0/2565112464 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe950011710 con 0x7fe9540ca3e0 2024-01-11T08:11:20.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.783+0000 7fe94a7fc700 1 -- 172.21.15.27:0/2565112464 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe950011ad0 con 0x7fe9540ca3e0 2024-01-11T08:11:20.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.783+0000 7fe95c5ff700 1 -- 172.21.15.27:0/2565112464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9540b2150 con 0x7fe9540ca3e0 2024-01-11T08:11:20.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.784+0000 7fe95c5ff700 1 -- 172.21.15.27:0/2565112464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9540b1b40 con 0x7fe9540ca3e0 2024-01-11T08:11:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.785+0000 7fe95c5ff700 1 -- 172.21.15.27:0/2565112464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9540718d0 con 0x7fe9540ca3e0 2024-01-11T08:11:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.785+0000 7fe94a7fc700 1 -- 172.21.15.27:0/2565112464 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe95001b3b0 con 0x7fe9540ca3e0 2024-01-11T08:11:20.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.786+0000 7fe94a7fc700 1 --2- 172.21.15.27:0/2565112464 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe938066f60 0x7fe938069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:20.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.786+0000 7fe94a7fc700 1 -- 172.21.15.27:0/2565112464 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fe9500835a0 con 0x7fe9540ca3e0 2024-01-11T08:11:20.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.787+0000 7fe959b9a700 1 --2- 172.21.15.27:0/2565112464 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe938066f60 0x7fe938069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.787+0000 7fe959b9a700 1 --2- 172.21.15.27:0/2565112464 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe938066f60 0x7fe938069410 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe9440093f0 tx=0x7fe944017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:20.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:20.790+0000 7fe94a7fc700 1 -- 172.21.15.27:0/2565112464 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe950050b60 con 0x7fe9540ca3e0 2024-01-11T08:11:21.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.001+0000 7fe95c5ff700 1 -- 172.21.15.27:0/2565112464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe95415cee0 con 0x7fe9540ca3e0 2024-01-11T08:11:21.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.002+0000 7fe94a7fc700 1 -- 172.21.15.27:0/2565112464 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+12086 (secure 0 0 0) 0x7fe950017020 con 0x7fe9540ca3e0 2024-01-11T08:11:21.002 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:21.002 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":30,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:11:16.907250+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:11:21.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.004+0000 7fe95c5ff700 1 -- 172.21.15.27:0/2565112464 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe938066f60 msgr2=0x7fe938069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:21.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.004+0000 7fe95c5ff700 1 --2- 172.21.15.27:0/2565112464 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe938066f60 0x7fe938069410 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe9440093f0 tx=0x7fe944017000 comp rx=0 tx=0).stop 2024-01-11T08:11:21.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.004+0000 7fe95c5ff700 1 -- 172.21.15.27:0/2565112464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9540ca3e0 msgr2=0x7fe954151730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:21.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.004+0000 7fe95c5ff700 1 --2- 172.21.15.27:0/2565112464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9540ca3e0 0x7fe954151730 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe95000c4e0 tx=0x7fe95000c7f0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.005+0000 7fe95c5ff700 1 -- 172.21.15.27:0/2565112464 shutdown_connections 2024-01-11T08:11:21.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.005+0000 7fe95c5ff700 1 --2- 172.21.15.27:0/2565112464 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe938066f60 0x7fe938069410 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.005+0000 7fe95c5ff700 1 --2- 172.21.15.27:0/2565112464 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe954159e80 0x7fe95415e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.005+0000 7fe95c5ff700 1 --2- 172.21.15.27:0/2565112464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9540ca3e0 0x7fe954151730 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.005+0000 7fe95c5ff700 1 --2- 172.21.15.27:0/2565112464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9540ca7b0 0x7fe95415de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.005+0000 7fe95c5ff700 1 -- 172.21.15.27:0/2565112464 >> 172.21.15.27:0/2565112464 conn(0x7fe9540ce3e0 msgr2=0x7fe954068700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:21.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.005+0000 7fe95c5ff700 1 -- 172.21.15.27:0/2565112464 shutdown_connections 2024-01-11T08:11:21.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.005+0000 7fe95c5ff700 1 -- 172.21.15.27:0/2565112464 wait complete. 2024-01-11T08:11:21.022 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-11T08:11:21.023 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:11:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.225+0000 7f70701ff700 1 Processor -- start 2024-01-11T08:11:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.225+0000 7f70701ff700 1 -- start start 2024-01-11T08:11:21.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.226+0000 7f70701ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f70680ccd70 0x7f70680cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:21.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.226+0000 7f70701ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706815ae70 0x7f706815b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:21.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.226+0000 7f70701ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70680cf850 con 0x7f706815a1b0 2024-01-11T08:11:21.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.227+0000 7f70701ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f706815ac90 con 0x7f70680ccd70 2024-01-11T08:11:21.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.227+0000 7f70701ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70680cec40 con 0x7f706815ae70 2024-01-11T08:11:21.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.227+0000 7f706d79a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f70680ccd70 0x7f70680cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:21.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.227+0000 7f706e79c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706815ae70 0x7f706815b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:21.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.227+0000 7f706df9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f706815a1b0 0x7f706815a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:50360/0 (socket says 172.21.15.27:50360) 2024-01-11T08:11:21.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.227+0000 7f706df9b700 1 -- 172.21.15.27:0/3206624125 learned_addr learned my addr 172.21.15.27:0/3206624125 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:21.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.228+0000 7f706e79c700 1 -- 172.21.15.27:0/3206624125 >> v2:172.21.15.77:3300/0 conn(0x7f70680ccd70 msgr2=0x7f70680cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:21.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.228+0000 7f706e79c700 1 --2- 172.21.15.27:0/3206624125 >> v2:172.21.15.77:3300/0 conn(0x7f70680ccd70 0x7f70680cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.228+0000 7f706cf99700 1 -- 172.21.15.27:0/3206624125 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718394726 0 0) 0x7f70680cf850 con 0x7f706815a1b0 2024-01-11T08:11:21.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.228+0000 7f706e79c700 1 -- 172.21.15.27:0/3206624125 >> v1:172.21.15.77:6789/0 conn(0x7f706815a1b0 legacy=0x7f706815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:21.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.228+0000 7f706e79c700 1 -- 172.21.15.27:0/3206624125 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70680cef20 con 0x7f706815ae70 2024-01-11T08:11:21.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.228+0000 7f706e79c700 1 --2- 172.21.15.27:0/3206624125 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706815ae70 0x7f706815b2f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7060009e60 tx=0x7f7060009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d89a7dcdab1f304 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.228+0000 7f706cf99700 1 -- 172.21.15.27:0/3206624125 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706000e070 con 0x7f706815ae70 2024-01-11T08:11:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.229+0000 7f706cf99700 1 -- 172.21.15.27:0/3206624125 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706000bb60 con 0x7f706815ae70 2024-01-11T08:11:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.229+0000 7f706cf99700 1 -- 172.21.15.27:0/3206624125 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7060012430 con 0x7f706815ae70 2024-01-11T08:11:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.229+0000 7f70701ff700 1 -- 172.21.15.27:0/3206624125 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706815ae70 msgr2=0x7f706815b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.229+0000 7f70701ff700 1 --2- 172.21.15.27:0/3206624125 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706815ae70 0x7f706815b2f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7060009e60 tx=0x7f7060009780 comp rx=0 tx=0).stop 2024-01-11T08:11:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.229+0000 7f70701ff700 1 -- 172.21.15.27:0/3206624125 shutdown_connections 2024-01-11T08:11:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.229+0000 7f70701ff700 1 --2- 172.21.15.27:0/3206624125 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706815ae70 0x7f706815b2f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.229+0000 7f70701ff700 1 --2- 172.21.15.27:0/3206624125 >> v2:172.21.15.77:3300/0 conn(0x7f70680ccd70 0x7f70680cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.229+0000 7f70701ff700 1 -- 172.21.15.27:0/3206624125 >> 172.21.15.27:0/3206624125 conn(0x7f706806ed40 msgr2=0x7f706806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:21.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.229+0000 7f70701ff700 1 -- 172.21.15.27:0/3206624125 shutdown_connections 2024-01-11T08:11:21.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.229+0000 7f70701ff700 1 -- 172.21.15.27:0/3206624125 wait complete. 2024-01-11T08:11:21.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.231+0000 7f70701ff700 1 Processor -- start 2024-01-11T08:11:21.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.231+0000 7f70701ff700 1 -- start start 2024-01-11T08:11:21.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.232+0000 7f70701ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70680ccd70 0x7f70681502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:21.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.232+0000 7f70701ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706815a1b0 0x7f706815f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:21.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.232+0000 7f70701ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706815ae70 0x7f706815b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:21.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.232+0000 7f70701ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7068157610 con 0x7f70680ccd70 2024-01-11T08:11:21.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.232+0000 7f70701ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70681578f0 con 0x7f706815a1b0 2024-01-11T08:11:21.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.232+0000 7f70701ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7068151fb0 con 0x7f706815ae70 2024-01-11T08:11:21.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.232+0000 7f706df9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70680ccd70 0x7f70681502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:21.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.232+0000 7f706df9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70680ccd70 0x7f70681502a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52120/0 (socket says 172.21.15.27:52120) 2024-01-11T08:11:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.232+0000 7f706df9b700 1 -- 172.21.15.27:0/84409749 learned_addr learned my addr 172.21.15.27:0/84409749 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.232+0000 7f706e79c700 1 --2- 172.21.15.27:0/84409749 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706815ae70 0x7f706815b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.232+0000 7f706d79a700 1 --2- 172.21.15.27:0/84409749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706815a1b0 0x7f706815f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.233+0000 7f706df9b700 1 -- 172.21.15.27:0/84409749 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706815ae70 msgr2=0x7f706815b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.233+0000 7f706df9b700 1 --2- 172.21.15.27:0/84409749 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706815ae70 0x7f706815b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.233+0000 7f706df9b700 1 -- 172.21.15.27:0/84409749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706815a1b0 msgr2=0x7f706815f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.233+0000 7f706df9b700 1 --2- 172.21.15.27:0/84409749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706815a1b0 0x7f706815f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.233+0000 7f706df9b700 1 -- 172.21.15.27:0/84409749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7060009a20 con 0x7f70680ccd70 2024-01-11T08:11:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.233+0000 7f706df9b700 1 --2- 172.21.15.27:0/84409749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70680ccd70 0x7f70681502a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f706400c810 tx=0x7f7064007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.233+0000 7f705dffb700 1 -- 172.21.15.27:0/84409749 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7064005b30 con 0x7f70680ccd70 2024-01-11T08:11:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.233+0000 7f705dffb700 1 -- 172.21.15.27:0/84409749 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7064005c90 con 0x7f70680ccd70 2024-01-11T08:11:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.233+0000 7f70701ff700 1 -- 172.21.15.27:0/84409749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70681507e0 con 0x7f70680ccd70 2024-01-11T08:11:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.233+0000 7f705dffb700 1 -- 172.21.15.27:0/84409749 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7064007300 con 0x7f70680ccd70 2024-01-11T08:11:21.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.233+0000 7f70701ff700 1 -- 172.21.15.27:0/84409749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f706815f000 con 0x7f70680ccd70 2024-01-11T08:11:21.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.234+0000 7f70701ff700 1 -- 172.21.15.27:0/84409749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70680ce290 con 0x7f70680ccd70 2024-01-11T08:11:21.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.235+0000 7f705dffb700 1 -- 172.21.15.27:0/84409749 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f706401b3b0 con 0x7f70680ccd70 2024-01-11T08:11:21.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.236+0000 7f705dffb700 1 --2- 172.21.15.27:0/84409749 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f704c066f30 0x7f704c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:21.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.236+0000 7f705dffb700 1 -- 172.21.15.27:0/84409749 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f7064083730 con 0x7f70680ccd70 2024-01-11T08:11:21.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.237+0000 7f706d79a700 1 --2- 172.21.15.27:0/84409749 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f704c066f30 0x7f704c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:21.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.237+0000 7f706d79a700 1 --2- 172.21.15.27:0/84409749 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f704c066f30 0x7f704c0693e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f706815b2e0 tx=0x7f70580091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:21.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.240+0000 7f705dffb700 1 -- 172.21.15.27:0/84409749 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7064050c50 con 0x7f70680ccd70 2024-01-11T08:11:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.444+0000 7f70701ff700 1 -- 172.21.15.27:0/84409749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f706804c3c0 con 0x7f70680ccd70 2024-01-11T08:11:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.444+0000 7f705dffb700 1 -- 172.21.15.27:0/84409749 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7064017020 con 0x7f70680ccd70 2024-01-11T08:11:21.444 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:11:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.447+0000 7f70701ff700 1 -- 172.21.15.27:0/84409749 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f704c066f30 msgr2=0x7f704c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.447+0000 7f70701ff700 1 --2- 172.21.15.27:0/84409749 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f704c066f30 0x7f704c0693e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f706815b2e0 tx=0x7f70580091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.447+0000 7f70701ff700 1 -- 172.21.15.27:0/84409749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70680ccd70 msgr2=0x7f70681502a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.447+0000 7f70701ff700 1 --2- 172.21.15.27:0/84409749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70680ccd70 0x7f70681502a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f706400c810 tx=0x7f7064007df0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.447+0000 7f70701ff700 1 -- 172.21.15.27:0/84409749 shutdown_connections 2024-01-11T08:11:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.447+0000 7f70701ff700 1 --2- 172.21.15.27:0/84409749 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f704c066f30 0x7f704c0693e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.447+0000 7f70701ff700 1 --2- 172.21.15.27:0/84409749 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706815ae70 0x7f706815b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.447+0000 7f70701ff700 1 --2- 172.21.15.27:0/84409749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70680ccd70 0x7f70681502a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.447+0000 7f70701ff700 1 --2- 172.21.15.27:0/84409749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706815a1b0 0x7f706815f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.447+0000 7f70701ff700 1 -- 172.21.15.27:0/84409749 >> 172.21.15.27:0/84409749 conn(0x7f706806ed40 msgr2=0x7f7068155f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.447+0000 7f70701ff700 1 -- 172.21.15.27:0/84409749 shutdown_connections 2024-01-11T08:11:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.447+0000 7f70701ff700 1 -- 172.21.15.27:0/84409749 wait complete. 2024-01-11T08:11:21.464 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-11T08:11:21.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.663+0000 7f7253bff700 1 Processor -- start 2024-01-11T08:11:21.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.663+0000 7f7253bff700 1 -- start start 2024-01-11T08:11:21.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.664+0000 7f7253bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f724c0c8620 0x7f724c0af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:21.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.664+0000 7f7253bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724c157390 con 0x7f724c069d40 2024-01-11T08:11:21.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.664+0000 7f7253bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724c0afeb0 con 0x7f724c0b09e0 2024-01-11T08:11:21.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.664+0000 7f7253bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f724c06a200 con 0x7f724c0c8620 2024-01-11T08:11:21.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.664+0000 7f725119a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f724c069d40 0x7f724c0b02d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57728/0 (socket says 172.21.15.27:57728) 2024-01-11T08:11:21.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.664+0000 7f725119a700 1 -- 172.21.15.27:0/2949714376 learned_addr learned my addr 172.21.15.27:0/2949714376 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:21.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.665+0000 7f725199b700 1 --2- 172.21.15.27:0/2949714376 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f724c0c8620 0x7f724c0af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:21.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.665+0000 7f7250999700 1 -- 172.21.15.27:0/2949714376 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794849762 0 0) 0x7f724c157390 con 0x7f724c069d40 2024-01-11T08:11:21.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.665+0000 7f7250999700 1 -- 172.21.15.27:0/2949714376 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7238003570 con 0x7f724c069d40 2024-01-11T08:11:21.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.665+0000 7f7250999700 1 -- 172.21.15.27:0/2949714376 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703491317 0 0) 0x7f724c0afeb0 con 0x7f724c0b09e0 2024-01-11T08:11:21.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.665+0000 7f7250999700 1 -- 172.21.15.27:0/2949714376 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f724c157390 con 0x7f724c0b09e0 2024-01-11T08:11:21.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.665+0000 7f725199b700 1 -- 172.21.15.27:0/2949714376 >> v1:172.21.15.77:6789/0 conn(0x7f724c0b09e0 legacy=0x7f724c0accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:21.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.665+0000 7f725199b700 1 -- 172.21.15.27:0/2949714376 >> v1:172.21.15.27:6789/0 conn(0x7f724c069d40 legacy=0x7f724c0b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:21.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.665+0000 7f725199b700 1 -- 172.21.15.27:0/2949714376 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724c156d40 con 0x7f724c0c8620 2024-01-11T08:11:21.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.665+0000 7f725199b700 1 --2- 172.21.15.27:0/2949714376 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f724c0c8620 0x7f724c0af970 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7248009910 tx=0x7f72480091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=404566a897992ebb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:21.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.666+0000 7f7250999700 1 -- 172.21.15.27:0/2949714376 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7248011630 con 0x7f724c0c8620 2024-01-11T08:11:21.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.666+0000 7f7250999700 1 -- 172.21.15.27:0/2949714376 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7248011790 con 0x7f724c0c8620 2024-01-11T08:11:21.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.666+0000 7f7250999700 1 -- 172.21.15.27:0/2949714376 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7248011b30 con 0x7f724c0c8620 2024-01-11T08:11:21.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.666+0000 7f7253bff700 1 -- 172.21.15.27:0/2949714376 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f724c0c8620 msgr2=0x7f724c0af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:21.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.666+0000 7f7253bff700 1 --2- 172.21.15.27:0/2949714376 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f724c0c8620 0x7f724c0af970 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7248009910 tx=0x7f72480091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.666+0000 7f7253bff700 1 -- 172.21.15.27:0/2949714376 shutdown_connections 2024-01-11T08:11:21.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.666+0000 7f7253bff700 1 --2- 172.21.15.27:0/2949714376 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f724c0c8620 0x7f724c0af970 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.666+0000 7f7253bff700 1 -- 172.21.15.27:0/2949714376 >> 172.21.15.27:0/2949714376 conn(0x7f724c0680e0 msgr2=0x7f724c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.667+0000 7f7253bff700 1 -- 172.21.15.27:0/2949714376 shutdown_connections 2024-01-11T08:11:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.667+0000 7f7253bff700 1 -- 172.21.15.27:0/2949714376 wait complete. 2024-01-11T08:11:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.668+0000 7f7253bff700 1 Processor -- start 2024-01-11T08:11:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.668+0000 7f7253bff700 1 -- start start 2024-01-11T08:11:21.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.669+0000 7f7253bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f724c069d40 0x7f724c0ae110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:21.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.669+0000 7f7253bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724c0b09e0 0x7f724c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:21.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.669+0000 7f7253bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f724c0c8620 0x7f724c169b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:21.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.669+0000 7f7253bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724c0ae650 con 0x7f724c0b09e0 2024-01-11T08:11:21.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.669+0000 7f7253bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724c0ad070 con 0x7f724c069d40 2024-01-11T08:11:21.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.669+0000 7f7253bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f724c0ad350 con 0x7f724c0c8620 2024-01-11T08:11:21.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.669+0000 7f725119a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724c0b09e0 0x7f724c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:21.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.669+0000 7f725119a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724c0b09e0 0x7f724c0cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52126/0 (socket says 172.21.15.27:52126) 2024-01-11T08:11:21.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.669+0000 7f725119a700 1 -- 172.21.15.27:0/2515914823 learned_addr learned my addr 172.21.15.27:0/2515914823 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:21.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.670+0000 7f725199b700 1 --2- 172.21.15.27:0/2515914823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f724c069d40 0x7f724c0ae110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:21.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.670+0000 7f725219c700 1 --2- 172.21.15.27:0/2515914823 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f724c0c8620 0x7f724c169b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:21.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.670+0000 7f725119a700 1 -- 172.21.15.27:0/2515914823 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f724c0c8620 msgr2=0x7f724c169b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:21.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.670+0000 7f725119a700 1 --2- 172.21.15.27:0/2515914823 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f724c0c8620 0x7f724c169b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.670+0000 7f725119a700 1 -- 172.21.15.27:0/2515914823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f724c069d40 msgr2=0x7f724c0ae110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:21.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.670+0000 7f725119a700 1 --2- 172.21.15.27:0/2515914823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f724c069d40 0x7f724c0ae110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.670+0000 7f725119a700 1 -- 172.21.15.27:0/2515914823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7248009420 con 0x7f724c0b09e0 2024-01-11T08:11:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.670+0000 7f725119a700 1 --2- 172.21.15.27:0/2515914823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724c0b09e0 0x7f724c0cc870 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f723c00c790 tx=0x7f723c007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.670+0000 7f7241ffb700 1 -- 172.21.15.27:0/2515914823 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f723c005b30 con 0x7f724c0b09e0 2024-01-11T08:11:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.670+0000 7f7241ffb700 1 -- 172.21.15.27:0/2515914823 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f723c005c90 con 0x7f724c0b09e0 2024-01-11T08:11:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.671+0000 7f7253bff700 1 -- 172.21.15.27:0/2515914823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f724c0ad630 con 0x7f724c0b09e0 2024-01-11T08:11:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.671+0000 7f7241ffb700 1 -- 172.21.15.27:0/2515914823 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f723c007300 con 0x7f724c0b09e0 2024-01-11T08:11:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.671+0000 7f7253bff700 1 -- 172.21.15.27:0/2515914823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f724c0adbf0 con 0x7f724c0b09e0 2024-01-11T08:11:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.672+0000 7f7253bff700 1 -- 172.21.15.27:0/2515914823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f724c0718d0 con 0x7f724c0b09e0 2024-01-11T08:11:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.673+0000 7f7241ffb700 1 -- 172.21.15.27:0/2515914823 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f723c01b3b0 con 0x7f724c0b09e0 2024-01-11T08:11:21.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.673+0000 7f7241ffb700 1 --2- 172.21.15.27:0/2515914823 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7230066f30 0x7f72300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:21.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.674+0000 7f7241ffb700 1 -- 172.21.15.27:0/2515914823 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f723c083820 con 0x7f724c0b09e0 2024-01-11T08:11:21.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.674+0000 7f725199b700 1 --2- 172.21.15.27:0/2515914823 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7230066f30 0x7f72300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:21.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.675+0000 7f725199b700 1 --2- 172.21.15.27:0/2515914823 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7230066f30 0x7f72300693e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f72480091e0 tx=0x7f7248017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:21.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.678+0000 7f7241ffb700 1 -- 172.21.15.27:0/2515914823 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f723c050d40 con 0x7f724c0b09e0 2024-01-11T08:11:21.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.881+0000 7f7253bff700 1 -- 172.21.15.27:0/2515914823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f724c157020 con 0x7f724c0b09e0 2024-01-11T08:11:21.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.882+0000 7f7241ffb700 1 -- 172.21.15.27:0/2515914823 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) v1 ==== 84+0+65 (secure 0 0 0) 0x7f723c017090 con 0x7f724c0b09e0 2024-01-11T08:11:21.882 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:21.882 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T08:11:21.882 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:21.882 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-11T08:11:21.882 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-11T08:11:21.882 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:21.882 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T08:11:21.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.884+0000 7f7253bff700 1 -- 172.21.15.27:0/2515914823 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7230066f30 msgr2=0x7f72300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:21.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.884+0000 7f7253bff700 1 --2- 172.21.15.27:0/2515914823 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7230066f30 0x7f72300693e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f72480091e0 tx=0x7f7248017000 comp rx=0 tx=0).stop 2024-01-11T08:11:21.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.884+0000 7f7253bff700 1 -- 172.21.15.27:0/2515914823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724c0b09e0 msgr2=0x7f724c0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:21.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.884+0000 7f7253bff700 1 --2- 172.21.15.27:0/2515914823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724c0b09e0 0x7f724c0cc870 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f723c00c790 tx=0x7f723c007e40 comp rx=0 tx=0).stop 2024-01-11T08:11:21.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.884+0000 7f7253bff700 1 -- 172.21.15.27:0/2515914823 shutdown_connections 2024-01-11T08:11:21.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.884+0000 7f7253bff700 1 --2- 172.21.15.27:0/2515914823 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7230066f30 0x7f72300693e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.884+0000 7f7253bff700 1 --2- 172.21.15.27:0/2515914823 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f724c0c8620 0x7f724c169b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.884+0000 7f7253bff700 1 --2- 172.21.15.27:0/2515914823 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f724c069d40 0x7f724c0ae110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.884+0000 7f7253bff700 1 --2- 172.21.15.27:0/2515914823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f724c0b09e0 0x7f724c0cc870 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:21.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.884+0000 7f7253bff700 1 -- 172.21.15.27:0/2515914823 >> 172.21.15.27:0/2515914823 conn(0x7f724c0680e0 msgr2=0x7f724c153a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:21.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.884+0000 7f7253bff700 1 -- 172.21.15.27:0/2515914823 shutdown_connections 2024-01-11T08:11:21.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:21.884+0000 7f7253bff700 1 -- 172.21.15.27:0/2515914823 wait complete. 2024-01-11T08:11:21.902 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-11T08:11:21.902 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-11T08:11:22.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.106+0000 7f3c6fbff700 1 Processor -- start 2024-01-11T08:11:22.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.106+0000 7f3c6fbff700 1 -- start start 2024-01-11T08:11:22.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.107+0000 7f3c6fbff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3c68159250 0x7f3c680b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:22.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.107+0000 7f3c6fbff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6815a440 con 0x7f3c68068be0 2024-01-11T08:11:22.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.107+0000 7f3c6fbff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6815a720 con 0x7f3c680cb410 2024-01-11T08:11:22.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.107+0000 7f3c6fbff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c6815ace0 con 0x7f3c68159250 2024-01-11T08:11:22.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.107+0000 7f3c6d99b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3c68159250 0x7f3c680b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:22.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.107+0000 7f3c6e19c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3c68068be0 0x7f3c680ce2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57734/0 (socket says 172.21.15.27:57734) 2024-01-11T08:11:22.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.107+0000 7f3c6e19c700 1 -- 172.21.15.27:0/883121654 learned_addr learned my addr 172.21.15.27:0/883121654 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:22.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.108+0000 7f3c6c999700 1 -- 172.21.15.27:0/883121654 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492647735 0 0) 0x7f3c6815a440 con 0x7f3c68068be0 2024-01-11T08:11:22.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.108+0000 7f3c6c999700 1 -- 172.21.15.27:0/883121654 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c54003570 con 0x7f3c68068be0 2024-01-11T08:11:22.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.108+0000 7f3c6d99b700 1 -- 172.21.15.27:0/883121654 >> v1:172.21.15.77:6789/0 conn(0x7f3c680cb410 legacy=0x7f3c680684d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:22.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.108+0000 7f3c6d99b700 1 -- 172.21.15.27:0/883121654 >> v1:172.21.15.27:6789/0 conn(0x7f3c68068be0 legacy=0x7f3c680ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:22.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.108+0000 7f3c6d99b700 1 -- 172.21.15.27:0/883121654 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6815aa00 con 0x7f3c68159250 2024-01-11T08:11:22.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.108+0000 7f3c6d99b700 1 --2- 172.21.15.27:0/883121654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c68159250 0x7f3c680b08a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3c640098e0 tx=0x7f3c640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f52307e2c92d3e34 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:22.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.108+0000 7f3c6c999700 1 -- 172.21.15.27:0/883121654 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6400b640 con 0x7f3c68159250 2024-01-11T08:11:22.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.108+0000 7f3c6c999700 1 -- 172.21.15.27:0/883121654 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6400b7a0 con 0x7f3c68159250 2024-01-11T08:11:22.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.108+0000 7f3c6c999700 1 -- 172.21.15.27:0/883121654 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6400bb40 con 0x7f3c68159250 2024-01-11T08:11:22.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.109+0000 7f3c6fbff700 1 -- 172.21.15.27:0/883121654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c68159250 msgr2=0x7f3c680b08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:22.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.109+0000 7f3c6fbff700 1 --2- 172.21.15.27:0/883121654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c68159250 0x7f3c680b08a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3c640098e0 tx=0x7f3c640091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:22.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.109+0000 7f3c6fbff700 1 -- 172.21.15.27:0/883121654 shutdown_connections 2024-01-11T08:11:22.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.109+0000 7f3c6fbff700 1 --2- 172.21.15.27:0/883121654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c68159250 0x7f3c680b08a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:22.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.109+0000 7f3c6fbff700 1 -- 172.21.15.27:0/883121654 >> 172.21.15.27:0/883121654 conn(0x7f3c68067610 msgr2=0x7f3c68067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:22.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.109+0000 7f3c6fbff700 1 -- 172.21.15.27:0/883121654 shutdown_connections 2024-01-11T08:11:22.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.109+0000 7f3c6fbff700 1 -- 172.21.15.27:0/883121654 wait complete. 2024-01-11T08:11:22.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.111+0000 7f3c6fbff700 1 Processor -- start 2024-01-11T08:11:22.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.111+0000 7f3c6fbff700 1 -- start start 2024-01-11T08:11:22.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.111+0000 7f3c6fbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c68068be0 0x7f3c6816a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:22.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.112+0000 7f3c6fbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c680cb410 0x7f3c680c2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:22.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.112+0000 7f3c6fbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c68159250 0x7f3c680c3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:22.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.112+0000 7f3c6fbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6816ad50 con 0x7f3c68068be0 2024-01-11T08:11:22.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.112+0000 7f3c6fbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c680c8d20 con 0x7f3c680cb410 2024-01-11T08:11:22.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.112+0000 7f3c6fbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c680c1fd0 con 0x7f3c68159250 2024-01-11T08:11:22.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.112+0000 7f3c6d99b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c68068be0 0x7f3c6816a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:22.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.112+0000 7f3c6d99b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c68068be0 0x7f3c6816a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52156/0 (socket says 172.21.15.27:52156) 2024-01-11T08:11:22.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.112+0000 7f3c6d99b700 1 -- 172.21.15.27:0/182990231 learned_addr learned my addr 172.21.15.27:0/182990231 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:22.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.112+0000 7f3c6d19a700 1 --2- 172.21.15.27:0/182990231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c680cb410 0x7f3c680c2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:22.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.112+0000 7f3c6e19c700 1 --2- 172.21.15.27:0/182990231 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c68159250 0x7f3c680c3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:22.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.112+0000 7f3c6d99b700 1 -- 172.21.15.27:0/182990231 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c68159250 msgr2=0x7f3c680c3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:22.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.112+0000 7f3c6d99b700 1 --2- 172.21.15.27:0/182990231 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c68159250 0x7f3c680c3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:22.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.112+0000 7f3c6d99b700 1 -- 172.21.15.27:0/182990231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c680cb410 msgr2=0x7f3c680c2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:22.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.112+0000 7f3c6d99b700 1 --2- 172.21.15.27:0/182990231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c680cb410 0x7f3c680c2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:22.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.112+0000 7f3c6d99b700 1 -- 172.21.15.27:0/182990231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c64009420 con 0x7f3c68068be0 2024-01-11T08:11:22.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.113+0000 7f3c6d99b700 1 --2- 172.21.15.27:0/182990231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c68068be0 0x7f3c6816a810 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3c640097b0 tx=0x7f3c64017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:22.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.113+0000 7f3c5dffb700 1 -- 172.21.15.27:0/182990231 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6400bb20 con 0x7f3c68068be0 2024-01-11T08:11:22.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.113+0000 7f3c5dffb700 1 -- 172.21.15.27:0/182990231 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c64011420 con 0x7f3c68068be0 2024-01-11T08:11:22.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.113+0000 7f3c6fbff700 1 -- 172.21.15.27:0/182990231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c680c22b0 con 0x7f3c68068be0 2024-01-11T08:11:22.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.113+0000 7f3c5dffb700 1 -- 172.21.15.27:0/182990231 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c640117e0 con 0x7f3c68068be0 2024-01-11T08:11:22.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.113+0000 7f3c6fbff700 1 -- 172.21.15.27:0/182990231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c680c2870 con 0x7f3c68068be0 2024-01-11T08:11:22.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.114+0000 7f3c6fbff700 1 -- 172.21.15.27:0/182990231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c680c2be0 con 0x7f3c68068be0 2024-01-11T08:11:22.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.115+0000 7f3c5dffb700 1 -- 172.21.15.27:0/182990231 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3c64011b90 con 0x7f3c68068be0 2024-01-11T08:11:22.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.116+0000 7f3c5dffb700 1 --2- 172.21.15.27:0/182990231 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3c4c066fb0 0x7f3c4c069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:22.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.116+0000 7f3c5dffb700 1 -- 172.21.15.27:0/182990231 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f3c64085b70 con 0x7f3c68068be0 2024-01-11T08:11:22.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.116+0000 7f3c6d19a700 1 --2- 172.21.15.27:0/182990231 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3c4c066fb0 0x7f3c4c069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.117+0000 7f3c6d19a700 1 --2- 172.21.15.27:0/182990231 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3c4c066fb0 0x7f3c4c069460 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3c6816d2f0 tx=0x7f3c58009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:22.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.120+0000 7f3c5dffb700 1 -- 172.21.15.27:0/182990231 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3c64053090 con 0x7f3c68068be0 2024-01-11T08:11:22.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.324+0000 7f3c6fbff700 1 -- 172.21.15.27:0/182990231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3c6804c3c0 con 0x7f3c68068be0 2024-01-11T08:11:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.915+0000 7f3c5dffb700 1 -- 172.21.15.27:0/182990231 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3c640566b0 con 0x7f3c68068be0 2024-01-11T08:11:22.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.919+0000 7f3c6fbff700 1 -- 172.21.15.27:0/182990231 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3c4c066fb0 msgr2=0x7f3c4c069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:22.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.919+0000 7f3c6fbff700 1 --2- 172.21.15.27:0/182990231 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3c4c066fb0 0x7f3c4c069460 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3c6816d2f0 tx=0x7f3c58009040 comp rx=0 tx=0).stop 2024-01-11T08:11:22.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.919+0000 7f3c6fbff700 1 -- 172.21.15.27:0/182990231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c68068be0 msgr2=0x7f3c6816a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:22.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.919+0000 7f3c6fbff700 1 --2- 172.21.15.27:0/182990231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c68068be0 0x7f3c6816a810 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3c640097b0 tx=0x7f3c64017000 comp rx=0 tx=0).stop 2024-01-11T08:11:22.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.920+0000 7f3c6fbff700 1 -- 172.21.15.27:0/182990231 shutdown_connections 2024-01-11T08:11:22.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.920+0000 7f3c6fbff700 1 --2- 172.21.15.27:0/182990231 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3c4c066fb0 0x7f3c4c069460 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:22.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.920+0000 7f3c6fbff700 1 --2- 172.21.15.27:0/182990231 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c68159250 0x7f3c680c3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:22.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.920+0000 7f3c6fbff700 1 --2- 172.21.15.27:0/182990231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c68068be0 0x7f3c6816a810 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:22.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.920+0000 7f3c6fbff700 1 --2- 172.21.15.27:0/182990231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c680cb410 0x7f3c680c2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:22.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.920+0000 7f3c6fbff700 1 -- 172.21.15.27:0/182990231 >> 172.21.15.27:0/182990231 conn(0x7f3c68067610 msgr2=0x7f3c68159620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:22.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.920+0000 7f3c6fbff700 1 -- 172.21.15.27:0/182990231 shutdown_connections 2024-01-11T08:11:22.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:22.920+0000 7f3c6fbff700 1 -- 172.21.15.27:0/182990231 wait complete. 2024-01-11T08:11:22.921 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-11T08:11:22.940 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-11T08:11:23.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.148+0000 7f882adff700 1 Processor -- start 2024-01-11T08:11:23.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.148+0000 7f882adff700 1 -- start start 2024-01-11T08:11:23.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.150+0000 7f882adff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f88240b19d0 0x7f8824158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:23.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.150+0000 7f882adff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88240cabe0 con 0x7f88240c8860 2024-01-11T08:11:23.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.150+0000 7f882adff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8824158a80 con 0x7f8824158600 2024-01-11T08:11:23.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.150+0000 7f882adff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88240ae960 con 0x7f88240b19d0 2024-01-11T08:11:23.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.150+0000 7f8823fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f88240b19d0 0x7f8824158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:23.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.150+0000 7f8823fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f88240b19d0 0x7f8824158ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52160/0 (socket says 172.21.15.27:52160) 2024-01-11T08:11:23.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.150+0000 7f8823fff700 1 -- 172.21.15.27:0/3485032468 learned_addr learned my addr 172.21.15.27:0/3485032468 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:23.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.150+0000 7f8828b9b700 1 --1- 172.21.15.27:0/3485032468 >> v1:172.21.15.27:6789/0 conn(0x7f88240c8860 0x7f88240b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57736/0 (socket says 172.21.15.27:57736) 2024-01-11T08:11:23.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.150+0000 7f8823fff700 1 -- 172.21.15.27:0/3485032468 >> v1:172.21.15.77:6789/0 conn(0x7f8824158600 legacy=0x7f88240cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:23.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.150+0000 7f8823fff700 1 -- 172.21.15.27:0/3485032468 >> v1:172.21.15.27:6789/0 conn(0x7f88240c8860 legacy=0x7f88240b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:23.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.150+0000 7f8823fff700 1 -- 172.21.15.27:0/3485032468 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88240ca590 con 0x7f88240b19d0 2024-01-11T08:11:23.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.150+0000 7f88237fe700 1 -- 172.21.15.27:0/3485032468 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654016816 0 0) 0x7f88240cabe0 con 0x7f88240c8860 2024-01-11T08:11:23.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.151+0000 7f8823fff700 1 --2- 172.21.15.27:0/3485032468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88240b19d0 0x7f8824158ca0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f88100099b0 tx=0x7f8810009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a785384918525920 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:23.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.151+0000 7f88237fe700 1 -- 172.21.15.27:0/3485032468 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881000b860 con 0x7f88240b19d0 2024-01-11T08:11:23.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.151+0000 7f88237fe700 1 -- 172.21.15.27:0/3485032468 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f881000b9c0 con 0x7f88240b19d0 2024-01-11T08:11:23.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.151+0000 7f88237fe700 1 -- 172.21.15.27:0/3485032468 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881000bd60 con 0x7f88240b19d0 2024-01-11T08:11:23.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.151+0000 7f882adff700 1 -- 172.21.15.27:0/3485032468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88240b19d0 msgr2=0x7f8824158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:23.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.151+0000 7f882adff700 1 --2- 172.21.15.27:0/3485032468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88240b19d0 0x7f8824158ca0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f88100099b0 tx=0x7f8810009250 comp rx=0 tx=0).stop 2024-01-11T08:11:23.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.152+0000 7f882adff700 1 -- 172.21.15.27:0/3485032468 shutdown_connections 2024-01-11T08:11:23.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.152+0000 7f882adff700 1 --2- 172.21.15.27:0/3485032468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88240b19d0 0x7f8824158ca0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:23.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.152+0000 7f882adff700 1 -- 172.21.15.27:0/3485032468 >> 172.21.15.27:0/3485032468 conn(0x7f88240680a0 msgr2=0x7f8824069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:23.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.152+0000 7f882adff700 1 -- 172.21.15.27:0/3485032468 shutdown_connections 2024-01-11T08:11:23.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.152+0000 7f882adff700 1 -- 172.21.15.27:0/3485032468 wait complete. 2024-01-11T08:11:23.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.153+0000 7f882adff700 1 Processor -- start 2024-01-11T08:11:23.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.153+0000 7f882adff700 1 -- start start 2024-01-11T08:11:23.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.154+0000 7f882adff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88240b19d0 0x7f882414ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:23.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.154+0000 7f882adff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88240c8860 0x7f88240ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:23.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.154+0000 7f882adff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8824158600 0x7f88240ad700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:23.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.154+0000 7f882adff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882415d670 con 0x7f88240b19d0 2024-01-11T08:11:23.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.154+0000 7f882adff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88241523a0 con 0x7f8824158600 2024-01-11T08:11:23.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.154+0000 7f882adff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882415d0b0 con 0x7f88240c8860 2024-01-11T08:11:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.155+0000 7f8828b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88240b19d0 0x7f882414ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.155+0000 7f8828b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88240b19d0 0x7f882414ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52174/0 (socket says 172.21.15.27:52174) 2024-01-11T08:11:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.155+0000 7f8828b9b700 1 -- 172.21.15.27:0/2582434036 learned_addr learned my addr 172.21.15.27:0/2582434036 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.155+0000 7f8823fff700 1 --2- 172.21.15.27:0/2582434036 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88240c8860 0x7f88240ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.155+0000 7f882939c700 1 --2- 172.21.15.27:0/2582434036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8824158600 0x7f88240ad700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.155+0000 7f8828b9b700 1 -- 172.21.15.27:0/2582434036 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88240c8860 msgr2=0x7f88240ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.155+0000 7f8828b9b700 1 --2- 172.21.15.27:0/2582434036 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88240c8860 0x7f88240ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.155+0000 7f8828b9b700 1 -- 172.21.15.27:0/2582434036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8824158600 msgr2=0x7f88240ad700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.155+0000 7f8828b9b700 1 --2- 172.21.15.27:0/2582434036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8824158600 0x7f88240ad700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.155+0000 7f8828b9b700 1 -- 172.21.15.27:0/2582434036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88100094f0 con 0x7f88240b19d0 2024-01-11T08:11:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.155+0000 7f8828b9b700 1 --2- 172.21.15.27:0/2582434036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88240b19d0 0x7f882414ede0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f881800c7e0 tx=0x7f8818007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.156+0000 7f8820ff9700 1 -- 172.21.15.27:0/2582434036 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88180073d0 con 0x7f88240b19d0 2024-01-11T08:11:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.156+0000 7f8820ff9700 1 -- 172.21.15.27:0/2582434036 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8818007530 con 0x7f88240b19d0 2024-01-11T08:11:23.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.156+0000 7f8820ff9700 1 -- 172.21.15.27:0/2582434036 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8818004a80 con 0x7f88240b19d0 2024-01-11T08:11:23.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.156+0000 7f882adff700 1 -- 172.21.15.27:0/2582434036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f882415d390 con 0x7f88240b19d0 2024-01-11T08:11:23.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.156+0000 7f882adff700 1 -- 172.21.15.27:0/2582434036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8824152920 con 0x7f88240b19d0 2024-01-11T08:11:23.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.157+0000 7f882adff700 1 -- 172.21.15.27:0/2582434036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8824155220 con 0x7f88240b19d0 2024-01-11T08:11:23.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.158+0000 7f8820ff9700 1 -- 172.21.15.27:0/2582434036 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f881801c3b0 con 0x7f88240b19d0 2024-01-11T08:11:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.159+0000 7f8820ff9700 1 --2- 172.21.15.27:0/2582434036 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8808066f30 0x7f88080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.159+0000 7f8820ff9700 1 -- 172.21.15.27:0/2582434036 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5223+0+0 (secure 0 0 0) 0x7f8818083ba0 con 0x7f88240b19d0 2024-01-11T08:11:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.159+0000 7f8823fff700 1 --2- 172.21.15.27:0/2582434036 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8808066f30 0x7f88080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.160+0000 7f8823fff700 1 --2- 172.21.15.27:0/2582434036 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8808066f30 0x7f88080693e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f88100099b0 tx=0x7f8810017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:23.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.163+0000 7f8820ff9700 1 -- 172.21.15.27:0/2582434036 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8818051150 con 0x7f88240b19d0 2024-01-11T08:11:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.368+0000 7f882adff700 1 -- 172.21.15.27:0/2582434036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f88240ca870 con 0x7f88240b19d0 2024-01-11T08:11:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.926+0000 7f8820ff9700 1 -- 172.21.15.27:0/2582434036 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8818018070 con 0x7f88240b19d0 2024-01-11T08:11:23.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.933+0000 7f882adff700 1 -- 172.21.15.27:0/2582434036 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8808066f30 msgr2=0x7f88080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:23.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.933+0000 7f882adff700 1 --2- 172.21.15.27:0/2582434036 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8808066f30 0x7f88080693e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f88100099b0 tx=0x7f8810017000 comp rx=0 tx=0).stop 2024-01-11T08:11:23.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.933+0000 7f882adff700 1 -- 172.21.15.27:0/2582434036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88240b19d0 msgr2=0x7f882414ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:23.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.933+0000 7f882adff700 1 --2- 172.21.15.27:0/2582434036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88240b19d0 0x7f882414ede0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f881800c7e0 tx=0x7f8818007de0 comp rx=0 tx=0).stop 2024-01-11T08:11:23.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.934+0000 7f882adff700 1 -- 172.21.15.27:0/2582434036 shutdown_connections 2024-01-11T08:11:23.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.934+0000 7f882adff700 1 --2- 172.21.15.27:0/2582434036 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8808066f30 0x7f88080693e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:23.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.934+0000 7f882adff700 1 --2- 172.21.15.27:0/2582434036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8824158600 0x7f88240ad700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:23.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.934+0000 7f882adff700 1 --2- 172.21.15.27:0/2582434036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88240b19d0 0x7f882414ede0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:23.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.934+0000 7f882adff700 1 --2- 172.21.15.27:0/2582434036 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88240c8860 0x7f88240ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:23.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.934+0000 7f882adff700 1 -- 172.21.15.27:0/2582434036 >> 172.21.15.27:0/2582434036 conn(0x7f88240680a0 msgr2=0x7f88240cba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:23.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.934+0000 7f882adff700 1 -- 172.21.15.27:0/2582434036 shutdown_connections 2024-01-11T08:11:23.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:23.935+0000 7f882adff700 1 -- 172.21.15.27:0/2582434036 wait complete. 2024-01-11T08:11:23.942 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-11T08:11:23.965 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-11T08:11:24.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.169+0000 7fd1aadff700 1 Processor -- start 2024-01-11T08:11:24.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.170+0000 7fd1aadff700 1 -- start start 2024-01-11T08:11:24.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.171+0000 7fd1aadff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd1a4142dc0 0x7fd1a40cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:24.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.171+0000 7fd1aadff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd1a40cb2e0 0x7fd1a40cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:24.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.171+0000 7fd1aadff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a40ccc70 con 0x7fd1a4143190 2024-01-11T08:11:24.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.171+0000 7fd1aadff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a415df30 con 0x7fd1a40cb2e0 2024-01-11T08:11:24.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.171+0000 7fd1aadff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a415d320 con 0x7fd1a4142dc0 2024-01-11T08:11:24.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.171+0000 7fd1a939c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd1a40cb2e0 0x7fd1a40cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:24.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.171+0000 7fd1a939c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd1a40cb2e0 0x7fd1a40cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42328/0 (socket says 172.21.15.27:42328) 2024-01-11T08:11:24.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.171+0000 7fd1a939c700 1 -- 172.21.15.27:0/3955147807 learned_addr learned my addr 172.21.15.27:0/3955147807 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:24.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.171+0000 7fd1a8b9b700 1 --2- 172.21.15.27:0/3955147807 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd1a4142dc0 0x7fd1a40cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:24.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.172+0000 7fd1a939c700 1 -- 172.21.15.27:0/3955147807 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd1a4142dc0 msgr2=0x7fd1a40cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:24.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.172+0000 7fd1a939c700 1 --2- 172.21.15.27:0/3955147807 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd1a4142dc0 0x7fd1a40cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:24.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.172+0000 7fd1a939c700 1 -- 172.21.15.27:0/3955147807 >> v1:172.21.15.77:6789/0 conn(0x7fd1a4143190 legacy=0x7fd1a40cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:24.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.172+0000 7fd1a939c700 1 -- 172.21.15.27:0/3955147807 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a40cb700 con 0x7fd1a40cb2e0 2024-01-11T08:11:24.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.172+0000 7fd1a939c700 1 --2- 172.21.15.27:0/3955147807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1a40cb2e0 0x7fd1a40cc730 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd19c009f20 tx=0x7fd19c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937aa6416489f641 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:24.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.172+0000 7fd1a37fe700 1 -- 172.21.15.27:0/3955147807 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c00e070 con 0x7fd1a40cb2e0 2024-01-11T08:11:24.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.172+0000 7fd1a37fe700 1 -- 172.21.15.27:0/3955147807 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c00b9d0 con 0x7fd1a40cb2e0 2024-01-11T08:11:24.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.172+0000 7fd1a37fe700 1 -- 172.21.15.27:0/3955147807 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c0092e0 con 0x7fd1a40cb2e0 2024-01-11T08:11:24.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.173+0000 7fd1aadff700 1 -- 172.21.15.27:0/3955147807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1a40cb2e0 msgr2=0x7fd1a40cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:24.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.173+0000 7fd1aadff700 1 --2- 172.21.15.27:0/3955147807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1a40cb2e0 0x7fd1a40cc730 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd19c009f20 tx=0x7fd19c009840 comp rx=0 tx=0).stop 2024-01-11T08:11:24.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.173+0000 7fd1aadff700 1 -- 172.21.15.27:0/3955147807 shutdown_connections 2024-01-11T08:11:24.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.173+0000 7fd1aadff700 1 --2- 172.21.15.27:0/3955147807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1a40cb2e0 0x7fd1a40cc730 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:24.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.173+0000 7fd1aadff700 1 --2- 172.21.15.27:0/3955147807 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd1a4142dc0 0x7fd1a40cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:24.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.173+0000 7fd1aadff700 1 -- 172.21.15.27:0/3955147807 >> 172.21.15.27:0/3955147807 conn(0x7fd1a4063ca0 msgr2=0x7fd1a406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:24.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.173+0000 7fd1aadff700 1 -- 172.21.15.27:0/3955147807 shutdown_connections 2024-01-11T08:11:24.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.173+0000 7fd1aadff700 1 -- 172.21.15.27:0/3955147807 wait complete. 2024-01-11T08:11:24.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.175+0000 7fd1aadff700 1 Processor -- start 2024-01-11T08:11:24.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.175+0000 7fd1aadff700 1 -- start start 2024-01-11T08:11:24.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.176+0000 7fd1aadff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd1a40cb2e0 0x7fd1a413fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:24.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.176+0000 7fd1aadff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1a4142dc0 0x7fd1a415bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:24.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.176+0000 7fd1aadff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1a4143190 0x7fd1a4140920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:24.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.176+0000 7fd1aadff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a412e9f0 con 0x7fd1a4142dc0 2024-01-11T08:11:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.176+0000 7fd1aadff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a412dde0 con 0x7fd1a4143190 2024-01-11T08:11:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.176+0000 7fd1aadff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a412e0c0 con 0x7fd1a40cb2e0 2024-01-11T08:11:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.176+0000 7fd1a3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1a4142dc0 0x7fd1a415bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.176+0000 7fd1a8b9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd1a40cb2e0 0x7fd1a413fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.176+0000 7fd1a8b9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd1a40cb2e0 0x7fd1a413fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:42350/0 (socket says 172.21.15.27:42350) 2024-01-11T08:11:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.176+0000 7fd1a8b9b700 1 -- 172.21.15.27:0/3705602334 learned_addr learned my addr 172.21.15.27:0/3705602334 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.176+0000 7fd1a939c700 1 --2- 172.21.15.27:0/3705602334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1a4143190 0x7fd1a4140920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.176+0000 7fd1a3fff700 1 -- 172.21.15.27:0/3705602334 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd1a40cb2e0 msgr2=0x7fd1a413fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.177+0000 7fd1a3fff700 1 --2- 172.21.15.27:0/3705602334 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd1a40cb2e0 0x7fd1a413fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.177+0000 7fd1a3fff700 1 -- 172.21.15.27:0/3705602334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1a4143190 msgr2=0x7fd1a4140920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.177+0000 7fd1a3fff700 1 --2- 172.21.15.27:0/3705602334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1a4143190 0x7fd1a4140920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.177+0000 7fd1a3fff700 1 -- 172.21.15.27:0/3705602334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19c009ae0 con 0x7fd1a4142dc0 2024-01-11T08:11:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.177+0000 7fd1a8b9b700 1 --2- 172.21.15.27:0/3705602334 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd1a40cb2e0 0x7fd1a413fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.177+0000 7fd1a3fff700 1 --2- 172.21.15.27:0/3705602334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1a4142dc0 0x7fd1a415bc70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd19000c490 tx=0x7fd190007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:24.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.177+0000 7fd1a0ff9700 1 -- 172.21.15.27:0/3705602334 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19000ce40 con 0x7fd1a4142dc0 2024-01-11T08:11:24.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.177+0000 7fd1a0ff9700 1 -- 172.21.15.27:0/3705602334 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd190011590 con 0x7fd1a4142dc0 2024-01-11T08:11:24.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.177+0000 7fd1a0ff9700 1 -- 172.21.15.27:0/3705602334 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd190011950 con 0x7fd1a4142dc0 2024-01-11T08:11:24.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.177+0000 7fd1aadff700 1 -- 172.21.15.27:0/3705602334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a412e3a0 con 0x7fd1a4142dc0 2024-01-11T08:11:24.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.177+0000 7fd1aadff700 1 -- 172.21.15.27:0/3705602334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a415c1b0 con 0x7fd1a4142dc0 2024-01-11T08:11:24.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.178+0000 7fd1aadff700 1 -- 172.21.15.27:0/3705602334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a404c3c0 con 0x7fd1a4142dc0 2024-01-11T08:11:24.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.180+0000 7fd1a0ff9700 1 -- 172.21.15.27:0/3705602334 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd190011cc0 con 0x7fd1a4142dc0 2024-01-11T08:11:24.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.180+0000 7fd1a0ff9700 1 --2- 172.21.15.27:0/3705602334 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd188066f30 0x7fd1880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:24.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.181+0000 7fd1a0ff9700 1 -- 172.21.15.27:0/3705602334 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5605+0+0 (secure 0 0 0) 0x7fd190083860 con 0x7fd1a4142dc0 2024-01-11T08:11:24.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.181+0000 7fd1a8b9b700 1 --2- 172.21.15.27:0/3705602334 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd188066f30 0x7fd1880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:24.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.182+0000 7fd1a8b9b700 1 --2- 172.21.15.27:0/3705602334 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd188066f30 0x7fd1880693e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd198009fd0 tx=0x7fd1980091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:24.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.185+0000 7fd1a0ff9700 1 -- 172.21.15.27:0/3705602334 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd190050a90 con 0x7fd1a4142dc0 2024-01-11T08:11:24.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.433+0000 7fd1aadff700 1 -- 172.21.15.27:0/3705602334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd1a412eb30 con 0x7fd1a4142dc0 2024-01-11T08:11:24.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.922+0000 7fd1a0ff9700 1 -- 172.21.15.27:0/3705602334 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7fd1900170c0 con 0x7fd1a4142dc0 2024-01-11T08:11:24.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.925+0000 7fd1aadff700 1 -- 172.21.15.27:0/3705602334 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd188066f30 msgr2=0x7fd1880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:24.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.925+0000 7fd1aadff700 1 --2- 172.21.15.27:0/3705602334 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd188066f30 0x7fd1880693e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd198009fd0 tx=0x7fd1980091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:24.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.925+0000 7fd1aadff700 1 -- 172.21.15.27:0/3705602334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1a4142dc0 msgr2=0x7fd1a415bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:24.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.925+0000 7fd1aadff700 1 --2- 172.21.15.27:0/3705602334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1a4142dc0 0x7fd1a415bc70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd19000c490 tx=0x7fd190007ec0 comp rx=0 tx=0).stop 2024-01-11T08:11:24.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.926+0000 7fd1aadff700 1 -- 172.21.15.27:0/3705602334 shutdown_connections 2024-01-11T08:11:24.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.926+0000 7fd1aadff700 1 --2- 172.21.15.27:0/3705602334 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd188066f30 0x7fd1880693e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:24.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.926+0000 7fd1aadff700 1 --2- 172.21.15.27:0/3705602334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1a4143190 0x7fd1a4140920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:24.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.926+0000 7fd1aadff700 1 --2- 172.21.15.27:0/3705602334 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd1a40cb2e0 0x7fd1a413fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:24.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.926+0000 7fd1aadff700 1 --2- 172.21.15.27:0/3705602334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1a4142dc0 0x7fd1a415bc70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:24.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.926+0000 7fd1aadff700 1 -- 172.21.15.27:0/3705602334 >> 172.21.15.27:0/3705602334 conn(0x7fd1a4063ca0 msgr2=0x7fd1a4068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:24.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.926+0000 7fd1aadff700 1 -- 172.21.15.27:0/3705602334 shutdown_connections 2024-01-11T08:11:24.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:24.926+0000 7fd1aadff700 1 -- 172.21.15.27:0/3705602334 wait complete. 2024-01-11T08:11:24.929 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 4 and data pool 5 2024-01-11T08:11:24.951 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:11:25.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.152+0000 7f46117ff700 1 Processor -- start 2024-01-11T08:11:25.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.153+0000 7f46117ff700 1 -- start start 2024-01-11T08:11:25.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.154+0000 7f46117ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f460c159db0 0x7f460c15a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:25.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.154+0000 7f46117ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f460c14fc20 0x7f460c1500a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:25.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.154+0000 7f46117ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460c157e30 con 0x7f460c1599e0 2024-01-11T08:11:25.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.154+0000 7f46117ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f460c157500 con 0x7f460c159db0 2024-01-11T08:11:25.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.154+0000 7f46117ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f460c1577e0 con 0x7f460c14fc20 2024-01-11T08:11:25.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.154+0000 7f460affd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f460c1599e0 0x7f460c0b0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50650/0 (socket says 172.21.15.27:50650) 2024-01-11T08:11:25.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.154+0000 7f460affd700 1 -- 172.21.15.27:0/3847578498 learned_addr learned my addr 172.21.15.27:0/3847578498 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:25.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.154+0000 7f460a7fc700 1 --2- 172.21.15.27:0/3847578498 >> v2:172.21.15.27:3300/0 conn(0x7f460c159db0 0x7f460c15a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:25.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.154+0000 7f460b7fe700 1 --2- 172.21.15.27:0/3847578498 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f460c14fc20 0x7f460c1500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:25.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.155+0000 7f4609ffb700 1 -- 172.21.15.27:0/3847578498 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275041583 0 0) 0x7f460c157e30 con 0x7f460c1599e0 2024-01-11T08:11:25.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.155+0000 7f4609ffb700 1 -- 172.21.15.27:0/3847578498 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45f4003570 con 0x7f460c1599e0 2024-01-11T08:11:25.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.155+0000 7f460a7fc700 1 -- 172.21.15.27:0/3847578498 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f460c14fc20 msgr2=0x7f460c1500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:25.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.155+0000 7f460a7fc700 1 --2- 172.21.15.27:0/3847578498 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f460c14fc20 0x7f460c1500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:25.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.155+0000 7f460a7fc700 1 -- 172.21.15.27:0/3847578498 >> v1:172.21.15.27:6789/0 conn(0x7f460c1599e0 legacy=0x7f460c0b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:25.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.155+0000 7f460a7fc700 1 -- 172.21.15.27:0/3847578498 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460c157ac0 con 0x7f460c159db0 2024-01-11T08:11:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.155+0000 7f460b7fe700 1 --2- 172.21.15.27:0/3847578498 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f460c14fc20 0x7f460c1500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.155+0000 7f460a7fc700 1 --2- 172.21.15.27:0/3847578498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c159db0 0x7f460c15a230 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4600009910 tx=0x7f46000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc440ee8df50ec5e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.155+0000 7f4609ffb700 1 -- 172.21.15.27:0/3847578498 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460000b640 con 0x7f460c159db0 2024-01-11T08:11:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.155+0000 7f4609ffb700 1 -- 172.21.15.27:0/3847578498 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460000b7a0 con 0x7f460c159db0 2024-01-11T08:11:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.155+0000 7f4609ffb700 1 -- 172.21.15.27:0/3847578498 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460000bb60 con 0x7f460c159db0 2024-01-11T08:11:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.156+0000 7f46117ff700 1 -- 172.21.15.27:0/3847578498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c159db0 msgr2=0x7f460c15a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.156+0000 7f46117ff700 1 --2- 172.21.15.27:0/3847578498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c159db0 0x7f460c15a230 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4600009910 tx=0x7f46000091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.156+0000 7f46117ff700 1 -- 172.21.15.27:0/3847578498 shutdown_connections 2024-01-11T08:11:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.156+0000 7f46117ff700 1 --2- 172.21.15.27:0/3847578498 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f460c14fc20 0x7f460c1500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.156+0000 7f46117ff700 1 --2- 172.21.15.27:0/3847578498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c159db0 0x7f460c15a230 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.156+0000 7f46117ff700 1 -- 172.21.15.27:0/3847578498 >> 172.21.15.27:0/3847578498 conn(0x7f460c0680e0 msgr2=0x7f460c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.156+0000 7f46117ff700 1 -- 172.21.15.27:0/3847578498 shutdown_connections 2024-01-11T08:11:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.156+0000 7f46117ff700 1 -- 172.21.15.27:0/3847578498 wait complete. 2024-01-11T08:11:25.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.158+0000 7f46117ff700 1 Processor -- start 2024-01-11T08:11:25.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.158+0000 7f46117ff700 1 -- start start 2024-01-11T08:11:25.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.159+0000 7f46117ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f460c14fc20 0x7f460c1517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:25.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.159+0000 7f46117ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f460c1599e0 0x7f460c146d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:25.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.159+0000 7f46117ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c159db0 0x7f460c147260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:25.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.159+0000 7f46117ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460c16dca0 con 0x7f460c159db0 2024-01-11T08:11:25.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.159+0000 7f46117ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460c16d090 con 0x7f460c1599e0 2024-01-11T08:11:25.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.159+0000 7f46117ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f460c16d370 con 0x7f460c14fc20 2024-01-11T08:11:25.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.159+0000 7f460b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c159db0 0x7f460c147260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:25.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.159+0000 7f460b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c159db0 0x7f460c147260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42354/0 (socket says 172.21.15.27:42354) 2024-01-11T08:11:25.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.159+0000 7f460b7fe700 1 -- 172.21.15.27:0/3354985470 learned_addr learned my addr 172.21.15.27:0/3354985470 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:25.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.159+0000 7f460a7fc700 1 --2- 172.21.15.27:0/3354985470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f460c1599e0 0x7f460c146d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:25.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.159+0000 7f460affd700 1 --2- 172.21.15.27:0/3354985470 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f460c14fc20 0x7f460c1517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:25.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.160+0000 7f460b7fe700 1 -- 172.21.15.27:0/3354985470 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f460c14fc20 msgr2=0x7f460c1517a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:25.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.160+0000 7f460b7fe700 1 --2- 172.21.15.27:0/3354985470 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f460c14fc20 0x7f460c1517a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:25.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.160+0000 7f460b7fe700 1 -- 172.21.15.27:0/3354985470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f460c1599e0 msgr2=0x7f460c146d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:25.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.160+0000 7f460b7fe700 1 --2- 172.21.15.27:0/3354985470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f460c1599e0 0x7f460c146d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:25.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.160+0000 7f460b7fe700 1 -- 172.21.15.27:0/3354985470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4600009420 con 0x7f460c159db0 2024-01-11T08:11:25.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.160+0000 7f460b7fe700 1 --2- 172.21.15.27:0/3354985470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c159db0 0x7f460c147260 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f460400d180 tx=0x7f4604009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:25.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.160+0000 7f45f2ffd700 1 -- 172.21.15.27:0/3354985470 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460400de20 con 0x7f460c159db0 2024-01-11T08:11:25.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.160+0000 7f45f2ffd700 1 -- 172.21.15.27:0/3354985470 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460400f040 con 0x7f460c159db0 2024-01-11T08:11:25.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.160+0000 7f45f2ffd700 1 -- 172.21.15.27:0/3354985470 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4604015690 con 0x7f460c159db0 2024-01-11T08:11:25.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.160+0000 7f46117ff700 1 -- 172.21.15.27:0/3354985470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f460c16d650 con 0x7f460c159db0 2024-01-11T08:11:25.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.160+0000 7f46117ff700 1 -- 172.21.15.27:0/3354985470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f460c151ce0 con 0x7f460c159db0 2024-01-11T08:11:25.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.161+0000 7f46117ff700 1 -- 172.21.15.27:0/3354985470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f460c16de10 con 0x7f460c159db0 2024-01-11T08:11:25.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.162+0000 7f45f2ffd700 1 -- 172.21.15.27:0/3354985470 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f460401e3b0 con 0x7f460c159db0 2024-01-11T08:11:25.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.164+0000 7f45f2ffd700 1 --2- 172.21.15.27:0/3354985470 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f45f8066f30 0x7f45f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:25.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.164+0000 7f45f2ffd700 1 -- 172.21.15.27:0/3354985470 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f4604084be0 con 0x7f460c159db0 2024-01-11T08:11:25.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.164+0000 7f460affd700 1 --2- 172.21.15.27:0/3354985470 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f45f8066f30 0x7f45f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:25.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.165+0000 7f460affd700 1 --2- 172.21.15.27:0/3354985470 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f45f8066f30 0x7f45f80693e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f460c0cbaf0 tx=0x7f45fc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:25.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.168+0000 7f45f2ffd700 1 -- 172.21.15.27:0/3354985470 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4604051db0 con 0x7f460c159db0 2024-01-11T08:11:25.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.372+0000 7f46117ff700 1 -- 172.21.15.27:0/3354985470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f460c0718d0 con 0x7f460c159db0 2024-01-11T08:11:25.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.373+0000 7f45f2ffd700 1 -- 172.21.15.27:0/3354985470 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+15315 (secure 0 0 0) 0x7f460401a090 con 0x7f460c159db0 2024-01-11T08:11:25.373 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:25.373 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":33,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:11:24.434697+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:11:22.376067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-11T08:11:23.419993+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:11:25.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.375+0000 7f46117ff700 1 -- 172.21.15.27:0/3354985470 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f45f8066f30 msgr2=0x7f45f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:25.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.375+0000 7f46117ff700 1 --2- 172.21.15.27:0/3354985470 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f45f8066f30 0x7f45f80693e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f460c0cbaf0 tx=0x7f45fc009040 comp rx=0 tx=0).stop 2024-01-11T08:11:25.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.375+0000 7f46117ff700 1 -- 172.21.15.27:0/3354985470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c159db0 msgr2=0x7f460c147260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:25.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.375+0000 7f46117ff700 1 --2- 172.21.15.27:0/3354985470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c159db0 0x7f460c147260 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f460400d180 tx=0x7f4604009a50 comp rx=0 tx=0).stop 2024-01-11T08:11:25.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.375+0000 7f46117ff700 1 -- 172.21.15.27:0/3354985470 shutdown_connections 2024-01-11T08:11:25.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.375+0000 7f46117ff700 1 --2- 172.21.15.27:0/3354985470 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f45f8066f30 0x7f45f80693e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:25.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.375+0000 7f46117ff700 1 --2- 172.21.15.27:0/3354985470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c159db0 0x7f460c147260 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:25.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.375+0000 7f46117ff700 1 --2- 172.21.15.27:0/3354985470 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f460c14fc20 0x7f460c1517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:25.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.375+0000 7f46117ff700 1 --2- 172.21.15.27:0/3354985470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f460c1599e0 0x7f460c146d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:25.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.375+0000 7f46117ff700 1 -- 172.21.15.27:0/3354985470 >> 172.21.15.27:0/3354985470 conn(0x7f460c0680e0 msgr2=0x7f460c06f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:25.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.375+0000 7f46117ff700 1 -- 172.21.15.27:0/3354985470 shutdown_connections 2024-01-11T08:11:25.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.376+0000 7f46117ff700 1 -- 172.21.15.27:0/3354985470 wait complete. 2024-01-11T08:11:25.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:11:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.596+0000 7fb739fff700 1 Processor -- start 2024-01-11T08:11:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.596+0000 7fb739fff700 1 -- start start 2024-01-11T08:11:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.597+0000 7fb739fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb734068d30 0x7fb7341646a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.597+0000 7fb739fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73415da10 con 0x7fb7340ca670 2024-01-11T08:11:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.597+0000 7fb739fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7340a9cb0 con 0x7fb7341596c0 2024-01-11T08:11:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.597+0000 7fb739fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb73415e210 con 0x7fb734068d30 2024-01-11T08:11:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.598+0000 7fb733fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb734068d30 0x7fb7341646a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.598+0000 7fb733fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb734068d30 0x7fb7341646a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42358/0 (socket says 172.21.15.27:42358) 2024-01-11T08:11:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.598+0000 7fb733fff700 1 -- 172.21.15.27:0/2370856911 learned_addr learned my addr 172.21.15.27:0/2370856911 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.598+0000 7fb732ffd700 1 --1- 172.21.15.27:0/2370856911 >> v1:172.21.15.27:6789/0 conn(0x7fb7340ca670 0x7fb734068620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50662/0 (socket says 172.21.15.27:50662) 2024-01-11T08:11:25.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.598+0000 7fb733fff700 1 -- 172.21.15.27:0/2370856911 >> v1:172.21.15.77:6789/0 conn(0x7fb7341596c0 legacy=0x7fb73415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.598+0000 7fb733fff700 1 -- 172.21.15.27:0/2370856911 >> v1:172.21.15.27:6789/0 conn(0x7fb7340ca670 legacy=0x7fb734068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.598+0000 7fb733fff700 1 -- 172.21.15.27:0/2370856911 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73415dc50 con 0x7fb734068d30 2024-01-11T08:11:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.598+0000 7fb7327fc700 1 -- 172.21.15.27:0/2370856911 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175962251 0 0) 0x7fb728001110 con 0x7fb7340ca670 2024-01-11T08:11:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.599+0000 7fb733fff700 1 --2- 172.21.15.27:0/2370856911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb734068d30 0x7fb7341646a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009f20 tx=0x7fb72c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8420899dd7d5b319 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.599+0000 7fb7327fc700 1 -- 172.21.15.27:0/2370856911 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c00e070 con 0x7fb734068d30 2024-01-11T08:11:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.599+0000 7fb7327fc700 1 -- 172.21.15.27:0/2370856911 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72c00ba80 con 0x7fb734068d30 2024-01-11T08:11:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.599+0000 7fb7327fc700 1 -- 172.21.15.27:0/2370856911 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c0092e0 con 0x7fb734068d30 2024-01-11T08:11:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.599+0000 7fb739fff700 1 -- 172.21.15.27:0/2370856911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb734068d30 msgr2=0x7fb7341646a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.599+0000 7fb739fff700 1 --2- 172.21.15.27:0/2370856911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb734068d30 0x7fb7341646a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009f20 tx=0x7fb72c009840 comp rx=0 tx=0).stop 2024-01-11T08:11:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.599+0000 7fb739fff700 1 -- 172.21.15.27:0/2370856911 shutdown_connections 2024-01-11T08:11:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.599+0000 7fb739fff700 1 --2- 172.21.15.27:0/2370856911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb734068d30 0x7fb7341646a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.599+0000 7fb739fff700 1 -- 172.21.15.27:0/2370856911 >> 172.21.15.27:0/2370856911 conn(0x7fb7340c9f70 msgr2=0x7fb73406deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:25.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.599+0000 7fb739fff700 1 -- 172.21.15.27:0/2370856911 shutdown_connections 2024-01-11T08:11:25.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.599+0000 7fb739fff700 1 -- 172.21.15.27:0/2370856911 wait complete. 2024-01-11T08:11:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.601+0000 7fb739fff700 1 Processor -- start 2024-01-11T08:11:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.601+0000 7fb739fff700 1 -- start start 2024-01-11T08:11:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.602+0000 7fb739fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb734068d30 0x7fb73415d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.602+0000 7fb739fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7340ca670 0x7fb734156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.602+0000 7fb739fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7341596c0 0x7fb7340c2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.602+0000 7fb739fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7340c4380 con 0x7fb7340ca670 2024-01-11T08:11:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.602+0000 7fb739fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7340b1370 con 0x7fb734068d30 2024-01-11T08:11:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.602+0000 7fb739fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7340afed0 con 0x7fb7341596c0 2024-01-11T08:11:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.602+0000 7fb732ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7340ca670 0x7fb734156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.602+0000 7fb732ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7340ca670 0x7fb734156500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42366/0 (socket says 172.21.15.27:42366) 2024-01-11T08:11:25.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.602+0000 7fb732ffd700 1 -- 172.21.15.27:0/1295502962 learned_addr learned my addr 172.21.15.27:0/1295502962 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:25.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.603+0000 7fb7337fe700 1 --2- 172.21.15.27:0/1295502962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb734068d30 0x7fb73415d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:25.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.603+0000 7fb732ffd700 1 -- 172.21.15.27:0/1295502962 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7341596c0 msgr2=0x7fb7340c2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:25.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.603+0000 7fb733fff700 1 --2- 172.21.15.27:0/1295502962 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7341596c0 0x7fb7340c2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:25.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.603+0000 7fb732ffd700 1 --2- 172.21.15.27:0/1295502962 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7341596c0 0x7fb7340c2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:25.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.603+0000 7fb732ffd700 1 -- 172.21.15.27:0/1295502962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb734068d30 msgr2=0x7fb73415d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:25.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.603+0000 7fb732ffd700 1 --2- 172.21.15.27:0/1295502962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb734068d30 0x7fb73415d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:25.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.603+0000 7fb732ffd700 1 -- 172.21.15.27:0/1295502962 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c009ae0 con 0x7fb7340ca670 2024-01-11T08:11:25.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.603+0000 7fb7337fe700 1 --2- 172.21.15.27:0/1295502962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb734068d30 0x7fb73415d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:11:25.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.603+0000 7fb733fff700 1 --2- 172.21.15.27:0/1295502962 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7341596c0 0x7fb7340c2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:11:25.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.603+0000 7fb732ffd700 1 --2- 172.21.15.27:0/1295502962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7340ca670 0x7fb734156500 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb72800c810 tx=0x7fb728007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:25.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.603+0000 7fb71b7fe700 1 -- 172.21.15.27:0/1295502962 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb728005a70 con 0x7fb7340ca670 2024-01-11T08:11:25.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.603+0000 7fb71b7fe700 1 -- 172.21.15.27:0/1295502962 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb728005bf0 con 0x7fb7340ca670 2024-01-11T08:11:25.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.603+0000 7fb739fff700 1 -- 172.21.15.27:0/1295502962 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7340b01b0 con 0x7fb7340ca670 2024-01-11T08:11:25.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.603+0000 7fb71b7fe700 1 -- 172.21.15.27:0/1295502962 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7280070f0 con 0x7fb7340ca670 2024-01-11T08:11:25.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.603+0000 7fb739fff700 1 -- 172.21.15.27:0/1295502962 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7340b0770 con 0x7fb7340ca670 2024-01-11T08:11:25.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.604+0000 7fb739fff700 1 -- 172.21.15.27:0/1295502962 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7340b0960 con 0x7fb7340ca670 2024-01-11T08:11:25.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.605+0000 7fb71b7fe700 1 -- 172.21.15.27:0/1295502962 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb7280074a0 con 0x7fb7340ca670 2024-01-11T08:11:25.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.606+0000 7fb71b7fe700 1 --2- 172.21.15.27:0/1295502962 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb720066f30 0x7fb7200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:25.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.606+0000 7fb71b7fe700 1 -- 172.21.15.27:0/1295502962 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fb728083eb0 con 0x7fb7340ca670 2024-01-11T08:11:25.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.607+0000 7fb7337fe700 1 --2- 172.21.15.27:0/1295502962 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb720066f30 0x7fb7200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:25.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.608+0000 7fb7337fe700 1 --2- 172.21.15.27:0/1295502962 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb720066f30 0x7fb7200693e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb7240017d0 tx=0x7fb724009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:25.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.611+0000 7fb71b7fe700 1 -- 172.21.15.27:0/1295502962 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb728051100 con 0x7fb7340ca670 2024-01-11T08:11:25.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.815+0000 7fb739fff700 1 -- 172.21.15.27:0/1295502962 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7340718d0 con 0x7fb7340ca670 2024-01-11T08:11:25.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.816+0000 7fb71b7fe700 1 -- 172.21.15.27:0/1295502962 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+15302 (secure 0 0 0) 0x7fb728054720 con 0x7fb7340ca670 2024-01-11T08:11:25.816 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:25.816 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:11:25.474635+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:11:22.376067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-11T08:11:23.419993+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:11:25.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.818+0000 7fb739fff700 1 -- 172.21.15.27:0/1295502962 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb720066f30 msgr2=0x7fb7200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:25.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.818+0000 7fb739fff700 1 --2- 172.21.15.27:0/1295502962 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb720066f30 0x7fb7200693e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb7240017d0 tx=0x7fb724009040 comp rx=0 tx=0).stop 2024-01-11T08:11:25.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.818+0000 7fb739fff700 1 -- 172.21.15.27:0/1295502962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7340ca670 msgr2=0x7fb734156500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:25.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.818+0000 7fb739fff700 1 --2- 172.21.15.27:0/1295502962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7340ca670 0x7fb734156500 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb72800c810 tx=0x7fb728007bd0 comp rx=0 tx=0).stop 2024-01-11T08:11:25.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.818+0000 7fb739fff700 1 -- 172.21.15.27:0/1295502962 shutdown_connections 2024-01-11T08:11:25.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.818+0000 7fb739fff700 1 --2- 172.21.15.27:0/1295502962 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb720066f30 0x7fb7200693e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:25.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.818+0000 7fb739fff700 1 --2- 172.21.15.27:0/1295502962 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7341596c0 0x7fb7340c2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:25.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.818+0000 7fb739fff700 1 --2- 172.21.15.27:0/1295502962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb734068d30 0x7fb73415d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:25.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.818+0000 7fb739fff700 1 --2- 172.21.15.27:0/1295502962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7340ca670 0x7fb734156500 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:25.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.818+0000 7fb739fff700 1 -- 172.21.15.27:0/1295502962 >> 172.21.15.27:0/1295502962 conn(0x7fb7340c9f70 msgr2=0x7fb7340aef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:25.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.818+0000 7fb739fff700 1 -- 172.21.15.27:0/1295502962 shutdown_connections 2024-01-11T08:11:25.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:25.818+0000 7fb739fff700 1 -- 172.21.15.27:0/1295502962 wait complete. 2024-01-11T08:11:25.837 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-11T08:11:26.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.041+0000 7f62503ff700 1 Processor -- start 2024-01-11T08:11:26.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.042+0000 7f62503ff700 1 -- start start 2024-01-11T08:11:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.043+0000 7f62503ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6248069bd0 0x7f62480aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.043+0000 7f62503ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f624806e480 con 0x7f6248155c50 2024-01-11T08:11:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.043+0000 7f62503ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62480c7a60 con 0x7f62480c6de0 2024-01-11T08:11:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.043+0000 7f62503ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62480cb5a0 con 0x7f6248069bd0 2024-01-11T08:11:26.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.043+0000 7f624e19b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6248155c50 0x7f62480694c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50676/0 (socket says 172.21.15.27:50676) 2024-01-11T08:11:26.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.043+0000 7f624e19b700 1 -- 172.21.15.27:0/3531563926 learned_addr learned my addr 172.21.15.27:0/3531563926 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:26.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.043+0000 7f624d99a700 1 --2- 172.21.15.27:0/3531563926 >> v2:172.21.15.77:3300/0 conn(0x7f6248069bd0 0x7f62480aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:26.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.044+0000 7f624d199700 1 -- 172.21.15.27:0/3531563926 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721718541 0 0) 0x7f624806e480 con 0x7f6248155c50 2024-01-11T08:11:26.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.044+0000 7f624d199700 1 -- 172.21.15.27:0/3531563926 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6234004680 con 0x7f6248155c50 2024-01-11T08:11:26.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.044+0000 7f624d199700 1 -- 172.21.15.27:0/3531563926 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485597087 0 0) 0x7f62480c7a60 con 0x7f62480c6de0 2024-01-11T08:11:26.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.044+0000 7f624d199700 1 -- 172.21.15.27:0/3531563926 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f624806e480 con 0x7f62480c6de0 2024-01-11T08:11:26.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.044+0000 7f624d99a700 1 -- 172.21.15.27:0/3531563926 >> v1:172.21.15.77:6789/0 conn(0x7f62480c6de0 legacy=0x7f62480c7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:26.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.044+0000 7f624d199700 1 -- 172.21.15.27:0/3531563926 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2007365284 0 0) 0x7f6234004680 con 0x7f6248155c50 2024-01-11T08:11:26.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.044+0000 7f624d99a700 1 -- 172.21.15.27:0/3531563926 >> v1:172.21.15.27:6789/0 conn(0x7f6248155c50 legacy=0x7f62480694c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:26.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.044+0000 7f624d99a700 1 -- 172.21.15.27:0/3531563926 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62480c7d40 con 0x7f6248069bd0 2024-01-11T08:11:26.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.044+0000 7f624d99a700 1 --2- 172.21.15.27:0/3531563926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6248069bd0 0x7f62480aef80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f62380098e0 tx=0x7f62380091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76409d2b3f1c729e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:26.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.045+0000 7f624d199700 1 -- 172.21.15.27:0/3531563926 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62440015f0 con 0x7f6248069bd0 2024-01-11T08:11:26.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.045+0000 7f624d199700 1 -- 172.21.15.27:0/3531563926 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6238011420 con 0x7f6248069bd0 2024-01-11T08:11:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.045+0000 7f624d199700 1 -- 172.21.15.27:0/3531563926 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62380117c0 con 0x7f6248069bd0 2024-01-11T08:11:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.045+0000 7f62503ff700 1 -- 172.21.15.27:0/3531563926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6248069bd0 msgr2=0x7f62480aef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.045+0000 7f62503ff700 1 --2- 172.21.15.27:0/3531563926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6248069bd0 0x7f62480aef80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f62380098e0 tx=0x7f62380091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.045+0000 7f62503ff700 1 -- 172.21.15.27:0/3531563926 shutdown_connections 2024-01-11T08:11:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.045+0000 7f62503ff700 1 --2- 172.21.15.27:0/3531563926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6248069bd0 0x7f62480aef80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.045+0000 7f62503ff700 1 -- 172.21.15.27:0/3531563926 >> 172.21.15.27:0/3531563926 conn(0x7f62480680a0 msgr2=0x7f62480cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.045+0000 7f62503ff700 1 -- 172.21.15.27:0/3531563926 shutdown_connections 2024-01-11T08:11:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.045+0000 7f62503ff700 1 -- 172.21.15.27:0/3531563926 wait complete. 2024-01-11T08:11:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.047+0000 7f62503ff700 1 Processor -- start 2024-01-11T08:11:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.047+0000 7f62503ff700 1 -- start start 2024-01-11T08:11:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.048+0000 7f62503ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6248069bd0 0x7f6248156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.048+0000 7f62503ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62480c6de0 0x7f6248152c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.048+0000 7f62503ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6248155c50 0x7f6248153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.048+0000 7f62503ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6248156ae0 con 0x7f62480c6de0 2024-01-11T08:11:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.048+0000 7f62503ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6248156dc0 con 0x7f6248155c50 2024-01-11T08:11:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.048+0000 7f62503ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62481570a0 con 0x7f6248069bd0 2024-01-11T08:11:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.048+0000 7f624d99a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62480c6de0 0x7f6248152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.048+0000 7f624d99a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62480c6de0 0x7f6248152c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42376/0 (socket says 172.21.15.27:42376) 2024-01-11T08:11:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.048+0000 7f624d99a700 1 -- 172.21.15.27:0/1305639684 learned_addr learned my addr 172.21.15.27:0/1305639684 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.048+0000 7f624e19b700 1 --2- 172.21.15.27:0/1305639684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6248069bd0 0x7f6248156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.048+0000 7f624e99c700 1 --2- 172.21.15.27:0/1305639684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6248155c50 0x7f6248153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.049+0000 7f624d99a700 1 -- 172.21.15.27:0/1305639684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6248069bd0 msgr2=0x7f6248156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.049+0000 7f624d99a700 1 --2- 172.21.15.27:0/1305639684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6248069bd0 0x7f6248156020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.049+0000 7f624d99a700 1 -- 172.21.15.27:0/1305639684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6248155c50 msgr2=0x7f6248153170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.049+0000 7f624d99a700 1 --2- 172.21.15.27:0/1305639684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6248155c50 0x7f6248153170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.049+0000 7f624d99a700 1 -- 172.21.15.27:0/1305639684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6238009420 con 0x7f62480c6de0 2024-01-11T08:11:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.049+0000 7f624d99a700 1 --2- 172.21.15.27:0/1305639684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62480c6de0 0x7f6248152c30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6248069490 tx=0x7f6238016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.049+0000 7f623e7fc700 1 -- 172.21.15.27:0/1305639684 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623801cb30 con 0x7f62480c6de0 2024-01-11T08:11:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.050+0000 7f623e7fc700 1 -- 172.21.15.27:0/1305639684 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f623801cc90 con 0x7f62480c6de0 2024-01-11T08:11:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.050+0000 7f62503ff700 1 -- 172.21.15.27:0/1305639684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6248157380 con 0x7f62480c6de0 2024-01-11T08:11:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.050+0000 7f623e7fc700 1 -- 172.21.15.27:0/1305639684 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623801a6a0 con 0x7f62480c6de0 2024-01-11T08:11:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.050+0000 7f62503ff700 1 -- 172.21.15.27:0/1305639684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62480c4df0 con 0x7f62480c6de0 2024-01-11T08:11:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.051+0000 7f62503ff700 1 -- 172.21.15.27:0/1305639684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62481578a0 con 0x7f62480c6de0 2024-01-11T08:11:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.051+0000 7f623e7fc700 1 -- 172.21.15.27:0/1305639684 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f623801aa50 con 0x7f62480c6de0 2024-01-11T08:11:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.052+0000 7f623e7fc700 1 --2- 172.21.15.27:0/1305639684 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f622c066f30 0x7f622c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.052+0000 7f623e7fc700 1 -- 172.21.15.27:0/1305639684 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f6238085ee0 con 0x7f62480c6de0 2024-01-11T08:11:26.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.053+0000 7f624e19b700 1 --2- 172.21.15.27:0/1305639684 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f622c066f30 0x7f622c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:26.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.053+0000 7f624e19b700 1 --2- 172.21.15.27:0/1305639684 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f622c066f30 0x7f622c0693e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6244000c00 tx=0x7f62440016e0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:26.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.057+0000 7f623e7fc700 1 -- 172.21.15.27:0/1305639684 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f62481578a0 con 0x7f62480c6de0 2024-01-11T08:11:26.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.308+0000 7f62503ff700 1 -- 172.21.15.27:0/1305639684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f62481578a0 con 0x7f62480c6de0 2024-01-11T08:11:26.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.941+0000 7f623e7fc700 1 -- 172.21.15.27:0/1305639684 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7f62481578a0 con 0x7f62480c6de0 2024-01-11T08:11:26.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.943+0000 7f62503ff700 1 -- 172.21.15.27:0/1305639684 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f622c066f30 msgr2=0x7f622c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:26.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.943+0000 7f62503ff700 1 --2- 172.21.15.27:0/1305639684 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f622c066f30 0x7f622c0693e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6244000c00 tx=0x7f62440016e0 comp rx=0 tx=0).stop 2024-01-11T08:11:26.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.943+0000 7f62503ff700 1 -- 172.21.15.27:0/1305639684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62480c6de0 msgr2=0x7f6248152c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:26.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.943+0000 7f62503ff700 1 --2- 172.21.15.27:0/1305639684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62480c6de0 0x7f6248152c30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6248069490 tx=0x7f6238016000 comp rx=0 tx=0).stop 2024-01-11T08:11:26.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.943+0000 7f62503ff700 1 -- 172.21.15.27:0/1305639684 shutdown_connections 2024-01-11T08:11:26.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.943+0000 7f62503ff700 1 --2- 172.21.15.27:0/1305639684 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f622c066f30 0x7f622c0693e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:26.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.943+0000 7f62503ff700 1 --2- 172.21.15.27:0/1305639684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6248155c50 0x7f6248153170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:26.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.943+0000 7f62503ff700 1 --2- 172.21.15.27:0/1305639684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6248069bd0 0x7f6248156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:26.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.943+0000 7f62503ff700 1 --2- 172.21.15.27:0/1305639684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62480c6de0 0x7f6248152c30 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:26.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.943+0000 7f62503ff700 1 -- 172.21.15.27:0/1305639684 >> 172.21.15.27:0/1305639684 conn(0x7f62480680a0 msgr2=0x7f62480c9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:26.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.943+0000 7f62503ff700 1 -- 172.21.15.27:0/1305639684 shutdown_connections 2024-01-11T08:11:26.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:26.944+0000 7f62503ff700 1 -- 172.21.15.27:0/1305639684 wait complete. 2024-01-11T08:11:26.963 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-11T08:11:27.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.168+0000 7f1fe17ff700 1 Processor -- start 2024-01-11T08:11:27.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.168+0000 7f1fe17ff700 1 -- start start 2024-01-11T08:11:27.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.170+0000 7f1fe17ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1fdc15a1b0 0x7f1fdc15a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:27.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.170+0000 7f1fe17ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1fdc0ccd70 0x7f1fdc0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:27.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.170+0000 7f1fe17ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1fdc0685c0 0x7f1fdc068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:27.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.170+0000 7f1fe17ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fdc0cf850 con 0x7f1fdc0685c0 2024-01-11T08:11:27.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.170+0000 7f1fe17ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fdc0cec40 con 0x7f1fdc15a1b0 2024-01-11T08:11:27.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.170+0000 7f1fe17ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fdc0cef20 con 0x7f1fdc0ccd70 2024-01-11T08:11:27.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.170+0000 7f1fdb7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1fdc0685c0 0x7f1fdc068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:27.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.170+0000 7f1fdb7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1fdc0685c0 0x7f1fdc068a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42380/0 (socket says 172.21.15.27:42380) 2024-01-11T08:11:27.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.170+0000 7f1fdb7fe700 1 -- 172.21.15.27:0/1398287059 learned_addr learned my addr 172.21.15.27:0/1398287059 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:27.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.170+0000 7f1fdaffd700 1 --2- 172.21.15.27:0/1398287059 >> v2:172.21.15.77:3300/0 conn(0x7f1fdc15a1b0 0x7f1fdc15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.170+0000 7f1fda7fc700 1 --2- 172.21.15.27:0/1398287059 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1fdc0ccd70 0x7f1fdc0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.171+0000 7f1fdb7fe700 1 -- 172.21.15.27:0/1398287059 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1fdc0ccd70 msgr2=0x7f1fdc0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.171+0000 7f1fdb7fe700 1 --2- 172.21.15.27:0/1398287059 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1fdc0ccd70 0x7f1fdc0cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.171+0000 7f1fdb7fe700 1 -- 172.21.15.27:0/1398287059 >> v2:172.21.15.77:3300/0 conn(0x7f1fdc15a1b0 msgr2=0x7f1fdc15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.171+0000 7f1fdb7fe700 1 --2- 172.21.15.27:0/1398287059 >> v2:172.21.15.77:3300/0 conn(0x7f1fdc15a1b0 0x7f1fdc15a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.171+0000 7f1fdb7fe700 1 -- 172.21.15.27:0/1398287059 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fdc0cf200 con 0x7f1fdc0685c0 2024-01-11T08:11:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.171+0000 7f1fdb7fe700 1 --2- 172.21.15.27:0/1398287059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fdc0685c0 0x7f1fdc068a40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0d16f0 tx=0x7f1fd4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc974c782fe13718 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.171+0000 7f1fd9ffb700 1 -- 172.21.15.27:0/1398287059 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd400e070 con 0x7f1fdc0685c0 2024-01-11T08:11:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.171+0000 7f1fd9ffb700 1 -- 172.21.15.27:0/1398287059 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd400ba20 con 0x7f1fdc0685c0 2024-01-11T08:11:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.171+0000 7f1fd9ffb700 1 -- 172.21.15.27:0/1398287059 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd40092e0 con 0x7f1fdc0685c0 2024-01-11T08:11:27.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.172+0000 7f1fe17ff700 1 -- 172.21.15.27:0/1398287059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fdc0685c0 msgr2=0x7f1fdc068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:27.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.172+0000 7f1fe17ff700 1 --2- 172.21.15.27:0/1398287059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fdc0685c0 0x7f1fdc068a40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0d16f0 tx=0x7f1fd4009840 comp rx=0 tx=0).stop 2024-01-11T08:11:27.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.172+0000 7f1fe17ff700 1 -- 172.21.15.27:0/1398287059 shutdown_connections 2024-01-11T08:11:27.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.172+0000 7f1fe17ff700 1 --2- 172.21.15.27:0/1398287059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fdc0685c0 0x7f1fdc068a40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:27.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.172+0000 7f1fe17ff700 1 --2- 172.21.15.27:0/1398287059 >> v2:172.21.15.77:3300/0 conn(0x7f1fdc15a1b0 0x7f1fdc15a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:27.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.172+0000 7f1fe17ff700 1 --2- 172.21.15.27:0/1398287059 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1fdc0ccd70 0x7f1fdc0cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:27.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.172+0000 7f1fe17ff700 1 -- 172.21.15.27:0/1398287059 >> 172.21.15.27:0/1398287059 conn(0x7f1fdc06ed40 msgr2=0x7f1fdc06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:27.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.172+0000 7f1fe17ff700 1 -- 172.21.15.27:0/1398287059 shutdown_connections 2024-01-11T08:11:27.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.172+0000 7f1fe17ff700 1 -- 172.21.15.27:0/1398287059 wait complete. 2024-01-11T08:11:27.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.174+0000 7f1fe17ff700 1 Processor -- start 2024-01-11T08:11:27.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.174+0000 7f1fe17ff700 1 -- start start 2024-01-11T08:11:27.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.176+0000 7f1fe17ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1fdc0685c0 0x7f1fdc0b60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:27.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.176+0000 7f1fe17ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fdc0ccd70 0x7f1fdc0b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:27.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.176+0000 7f1fe17ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fdc15a1b0 0x7f1fdc1592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:27.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.176+0000 7f1fe17ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fdc0c9130 con 0x7f1fdc15a1b0 2024-01-11T08:11:27.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.176+0000 7f1fe17ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fdc0b42d0 con 0x7f1fdc0ccd70 2024-01-11T08:11:27.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.176+0000 7f1fe17ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fdc0b45a0 con 0x7f1fdc0685c0 2024-01-11T08:11:27.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.176+0000 7f1fdb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fdc15a1b0 0x7f1fdc1592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:27.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.176+0000 7f1fdb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fdc15a1b0 0x7f1fdc1592e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42382/0 (socket says 172.21.15.27:42382) 2024-01-11T08:11:27.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.176+0000 7f1fdb7fe700 1 -- 172.21.15.27:0/2010887661 learned_addr learned my addr 172.21.15.27:0/2010887661 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:27.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.176+0000 7f1fdaffd700 1 --2- 172.21.15.27:0/2010887661 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1fdc0685c0 0x7f1fdc0b60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:27.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.176+0000 7f1fda7fc700 1 --2- 172.21.15.27:0/2010887661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fdc0ccd70 0x7f1fdc0b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:27.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.176+0000 7f1fdb7fe700 1 -- 172.21.15.27:0/2010887661 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1fdc0685c0 msgr2=0x7f1fdc0b60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:27.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.176+0000 7f1fdb7fe700 1 --2- 172.21.15.27:0/2010887661 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1fdc0685c0 0x7f1fdc0b60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:27.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.176+0000 7f1fdb7fe700 1 -- 172.21.15.27:0/2010887661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fdc0ccd70 msgr2=0x7f1fdc0b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:27.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.176+0000 7f1fdb7fe700 1 --2- 172.21.15.27:0/2010887661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fdc0ccd70 0x7f1fdc0b3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:27.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.176+0000 7f1fdb7fe700 1 -- 172.21.15.27:0/2010887661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd4009ae0 con 0x7f1fdc15a1b0 2024-01-11T08:11:27.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.176+0000 7f1fda7fc700 1 --2- 172.21.15.27:0/2010887661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fdc0ccd70 0x7f1fdc0b3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:11:27.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.177+0000 7f1fdb7fe700 1 --2- 172.21.15.27:0/2010887661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fdc15a1b0 0x7f1fdc1592e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc069770 tx=0x7f1fd4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:27.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.177+0000 7f1fc2ffd700 1 -- 172.21.15.27:0/2010887661 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd40032f0 con 0x7f1fdc15a1b0 2024-01-11T08:11:27.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.177+0000 7f1fc2ffd700 1 -- 172.21.15.27:0/2010887661 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd401c410 con 0x7f1fdc15a1b0 2024-01-11T08:11:27.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.177+0000 7f1fc2ffd700 1 -- 172.21.15.27:0/2010887661 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd401c7d0 con 0x7f1fdc15a1b0 2024-01-11T08:11:27.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.177+0000 7f1fe17ff700 1 -- 172.21.15.27:0/2010887661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fdc0b4880 con 0x7f1fdc15a1b0 2024-01-11T08:11:27.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.177+0000 7f1fe17ff700 1 -- 172.21.15.27:0/2010887661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fdc0c8dc0 con 0x7f1fdc15a1b0 2024-01-11T08:11:27.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.178+0000 7f1fe17ff700 1 -- 172.21.15.27:0/2010887661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fdc0c8940 con 0x7f1fdc15a1b0 2024-01-11T08:11:27.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.179+0000 7f1fc2ffd700 1 -- 172.21.15.27:0/2010887661 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1fd4003040 con 0x7f1fdc15a1b0 2024-01-11T08:11:27.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.180+0000 7f1fc2ffd700 1 --2- 172.21.15.27:0/2010887661 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1fc8066f30 0x7f1fc80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:27.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.180+0000 7f1fc2ffd700 1 -- 172.21.15.27:0/2010887661 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f1fd4085fe0 con 0x7f1fdc15a1b0 2024-01-11T08:11:27.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.181+0000 7f1fdaffd700 1 --2- 172.21.15.27:0/2010887661 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1fc8066f30 0x7f1fc80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:27.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.182+0000 7f1fdaffd700 1 --2- 172.21.15.27:0/2010887661 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1fc8066f30 0x7f1fc80693e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc009d40 tx=0x7f1fcc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:27.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.185+0000 7f1fc2ffd700 1 -- 172.21.15.27:0/2010887661 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1fdc0c8940 con 0x7f1fdc15a1b0 2024-01-11T08:11:27.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.435+0000 7f1fe17ff700 1 -- 172.21.15.27:0/2010887661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1fdc0c8940 con 0x7f1fdc15a1b0 2024-01-11T08:11:27.900 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-01-11T08:11:27.900+0000 7f3752df1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T08:11:27.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.949+0000 7f1fc2ffd700 1 -- 172.21.15.27:0/2010887661 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1fdc0c8940 con 0x7f1fdc15a1b0 2024-01-11T08:11:27.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.952+0000 7f1fe17ff700 1 -- 172.21.15.27:0/2010887661 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1fc8066f30 msgr2=0x7f1fc80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:27.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.952+0000 7f1fe17ff700 1 --2- 172.21.15.27:0/2010887661 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1fc8066f30 0x7f1fc80693e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc009d40 tx=0x7f1fcc008040 comp rx=0 tx=0).stop 2024-01-11T08:11:27.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.952+0000 7f1fe17ff700 1 -- 172.21.15.27:0/2010887661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fdc15a1b0 msgr2=0x7f1fdc1592e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:27.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.952+0000 7f1fe17ff700 1 --2- 172.21.15.27:0/2010887661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fdc15a1b0 0x7f1fdc1592e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc069770 tx=0x7f1fd4018000 comp rx=0 tx=0).stop 2024-01-11T08:11:27.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.952+0000 7f1fe17ff700 1 -- 172.21.15.27:0/2010887661 shutdown_connections 2024-01-11T08:11:27.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.952+0000 7f1fe17ff700 1 --2- 172.21.15.27:0/2010887661 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1fc8066f30 0x7f1fc80693e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:27.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.952+0000 7f1fe17ff700 1 --2- 172.21.15.27:0/2010887661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fdc15a1b0 0x7f1fdc1592e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:27.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.952+0000 7f1fe17ff700 1 --2- 172.21.15.27:0/2010887661 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1fdc0685c0 0x7f1fdc0b60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:27.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.952+0000 7f1fe17ff700 1 --2- 172.21.15.27:0/2010887661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fdc0ccd70 0x7f1fdc0b3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:27.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.952+0000 7f1fe17ff700 1 -- 172.21.15.27:0/2010887661 >> 172.21.15.27:0/2010887661 conn(0x7f1fdc06ed40 msgr2=0x7f1fdc158660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:27.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.952+0000 7f1fe17ff700 1 -- 172.21.15.27:0/2010887661 shutdown_connections 2024-01-11T08:11:27.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:27.952+0000 7f1fe17ff700 1 -- 172.21.15.27:0/2010887661 wait complete. 2024-01-11T08:11:27.971 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:11:28.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.174+0000 7fe3a5fff700 1 Processor -- start 2024-01-11T08:11:28.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.175+0000 7fe3a5fff700 1 -- start start 2024-01-11T08:11:28.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.176+0000 7fe3a5fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe3a00cc410 0x7fe3a0159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:28.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.176+0000 7fe3a5fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a00cd730 con 0x7fe3a00b2b90 2024-01-11T08:11:28.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.176+0000 7fe3a5fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a00ccb20 con 0x7fe3a00b03e0 2024-01-11T08:11:28.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.176+0000 7fe3a5fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3a00cce00 con 0x7fe3a00cc410 2024-01-11T08:11:28.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.176+0000 7fe39ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe3a00cc410 0x7fe3a0159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:28.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.176+0000 7fe39effd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe3a00b2b90 0x7fe3a0159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50680/0 (socket says 172.21.15.27:50680) 2024-01-11T08:11:28.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.176+0000 7fe39effd700 1 -- 172.21.15.27:0/3335866567 learned_addr learned my addr 172.21.15.27:0/3335866567 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:28.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.177+0000 7fe39ffff700 1 -- 172.21.15.27:0/3335866567 >> v1:172.21.15.77:6789/0 conn(0x7fe3a00b03e0 legacy=0x7fe3a00b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:28.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.177+0000 7fe39ffff700 1 -- 172.21.15.27:0/3335866567 >> v1:172.21.15.27:6789/0 conn(0x7fe3a00b2b90 legacy=0x7fe3a0159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:28.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.177+0000 7fe39ffff700 1 -- 172.21.15.27:0/3335866567 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a00cd0e0 con 0x7fe3a00cc410 2024-01-11T08:11:28.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.177+0000 7fe39ffff700 1 --2- 172.21.15.27:0/3335866567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a00cc410 0x7fe3a0159910 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe398009e90 tx=0x7fe398009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81e4657febf7e622 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:28.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.177+0000 7fe39e7fc700 1 -- 172.21.15.27:0/3335866567 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39800e070 con 0x7fe3a00cc410 2024-01-11T08:11:28.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.177+0000 7fe39e7fc700 1 -- 172.21.15.27:0/3335866567 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39800bbf0 con 0x7fe3a00cc410 2024-01-11T08:11:28.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.177+0000 7fe39e7fc700 1 -- 172.21.15.27:0/3335866567 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe398012430 con 0x7fe3a00cc410 2024-01-11T08:11:28.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.178+0000 7fe3a5fff700 1 -- 172.21.15.27:0/3335866567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a00cc410 msgr2=0x7fe3a0159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:28.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.178+0000 7fe3a5fff700 1 --2- 172.21.15.27:0/3335866567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a00cc410 0x7fe3a0159910 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe398009e90 tx=0x7fe398009780 comp rx=0 tx=0).stop 2024-01-11T08:11:28.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.178+0000 7fe3a5fff700 1 -- 172.21.15.27:0/3335866567 shutdown_connections 2024-01-11T08:11:28.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.178+0000 7fe3a5fff700 1 --2- 172.21.15.27:0/3335866567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a00cc410 0x7fe3a0159910 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:28.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.178+0000 7fe3a5fff700 1 -- 172.21.15.27:0/3335866567 >> 172.21.15.27:0/3335866567 conn(0x7fe3a006ed40 msgr2=0x7fe3a006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:28.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.178+0000 7fe3a5fff700 1 -- 172.21.15.27:0/3335866567 shutdown_connections 2024-01-11T08:11:28.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.178+0000 7fe3a5fff700 1 -- 172.21.15.27:0/3335866567 wait complete. 2024-01-11T08:11:28.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.179+0000 7fe3a5fff700 1 Processor -- start 2024-01-11T08:11:28.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.180+0000 7fe3a5fff700 1 -- start start 2024-01-11T08:11:28.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.180+0000 7fe3a5fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3a00b03e0 0x7fe3a00c4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:28.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.180+0000 7fe3a5fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a00b2b90 0x7fe3a016cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:28.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.180+0000 7fe3a5fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe3a00cc410 0x7fe3a016d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:28.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.180+0000 7fe3a5fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a0147a90 con 0x7fe3a00b2b90 2024-01-11T08:11:28.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.180+0000 7fe3a5fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a00c5880 con 0x7fe3a00b03e0 2024-01-11T08:11:28.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.180+0000 7fe3a5fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a00c7660 con 0x7fe3a00cc410 2024-01-11T08:11:28.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.181+0000 7fe39effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a00b2b90 0x7fe3a016cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:28.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.181+0000 7fe39effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a00b2b90 0x7fe3a016cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42394/0 (socket says 172.21.15.27:42394) 2024-01-11T08:11:28.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.181+0000 7fe39effd700 1 -- 172.21.15.27:0/4060587572 learned_addr learned my addr 172.21.15.27:0/4060587572 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:28.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.181+0000 7fe39f7fe700 1 --2- 172.21.15.27:0/4060587572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3a00b03e0 0x7fe3a00c4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:28.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.181+0000 7fe39ffff700 1 --2- 172.21.15.27:0/4060587572 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe3a00cc410 0x7fe3a016d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:28.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.181+0000 7fe39effd700 1 -- 172.21.15.27:0/4060587572 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe3a00cc410 msgr2=0x7fe3a016d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:28.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.181+0000 7fe39effd700 1 --2- 172.21.15.27:0/4060587572 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe3a00cc410 0x7fe3a016d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:28.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.181+0000 7fe39effd700 1 -- 172.21.15.27:0/4060587572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3a00b03e0 msgr2=0x7fe3a00c4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:28.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.181+0000 7fe39effd700 1 --2- 172.21.15.27:0/4060587572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3a00b03e0 0x7fe3a00c4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:28.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.181+0000 7fe39effd700 1 -- 172.21.15.27:0/4060587572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe398009a20 con 0x7fe3a00b2b90 2024-01-11T08:11:28.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.182+0000 7fe39effd700 1 --2- 172.21.15.27:0/4060587572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a00b2b90 0x7fe3a016cd60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe39400c7c0 tx=0x7fe394007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:28.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.182+0000 7fe3877fe700 1 -- 172.21.15.27:0/4060587572 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe394005b30 con 0x7fe3a00b2b90 2024-01-11T08:11:28.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.182+0000 7fe3877fe700 1 -- 172.21.15.27:0/4060587572 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe394005c90 con 0x7fe3a00b2b90 2024-01-11T08:11:28.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.182+0000 7fe3a5fff700 1 -- 172.21.15.27:0/4060587572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a00c7940 con 0x7fe3a00b2b90 2024-01-11T08:11:28.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.182+0000 7fe3877fe700 1 -- 172.21.15.27:0/4060587572 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe394004a80 con 0x7fe3a00b2b90 2024-01-11T08:11:28.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.182+0000 7fe3a5fff700 1 -- 172.21.15.27:0/4060587572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a00c7f00 con 0x7fe3a00b2b90 2024-01-11T08:11:28.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.183+0000 7fe3a5fff700 1 -- 172.21.15.27:0/4060587572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a00718d0 con 0x7fe3a00b2b90 2024-01-11T08:11:28.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.184+0000 7fe3877fe700 1 -- 172.21.15.27:0/4060587572 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe3940070d0 con 0x7fe3a00b2b90 2024-01-11T08:11:28.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.185+0000 7fe3877fe700 1 --2- 172.21.15.27:0/4060587572 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe38c066f30 0x7fe38c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:28.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.185+0000 7fe3877fe700 1 -- 172.21.15.27:0/4060587572 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fe3940838c0 con 0x7fe3a00b2b90 2024-01-11T08:11:28.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.186+0000 7fe39f7fe700 1 --2- 172.21.15.27:0/4060587572 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe38c066f30 0x7fe38c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:28.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.186+0000 7fe39f7fe700 1 --2- 172.21.15.27:0/4060587572 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe38c066f30 0x7fe38c0693e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe390003a80 tx=0x7fe390007ec0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:28.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.190+0000 7fe3877fe700 1 -- 172.21.15.27:0/4060587572 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe39404b5a0 con 0x7fe3a00b2b90 2024-01-11T08:11:28.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.436+0000 7fe3a5fff700 1 -- 172.21.15.27:0/4060587572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3a00cd3c0 con 0x7fe3a00b2b90 2024-01-11T08:11:28.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.437+0000 7fe3877fe700 1 -- 172.21.15.27:0/4060587572 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4591 (secure 0 0 0) 0x7fe394050a90 con 0x7fe3a00b2b90 2024-01-11T08:11:28.437 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:28.437 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/2930156910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2930156910},{"type":"v1","addr":"172.21.15.77:6836","nonce":2930156910}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/685019444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":685019444},{"type":"v1","addr":"172.21.15.27:6834","nonce":685019444}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4861,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/144853545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":144853545},{"type":"v1","addr":"172.21.15.77:6837","nonce":144853545}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:11:24.435181+0000","modified":"2024-01-11T08:11:27.940837+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/4056690675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4056690675},{"type":"v1","addr":"172.21.15.27:6835","nonce":4056690675}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2024-01-11T08:11:28.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.439+0000 7fe3a5fff700 1 -- 172.21.15.27:0/4060587572 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe38c066f30 msgr2=0x7fe38c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:28.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.439+0000 7fe3a5fff700 1 --2- 172.21.15.27:0/4060587572 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe38c066f30 0x7fe38c0693e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe390003a80 tx=0x7fe390007ec0 comp rx=0 tx=0).stop 2024-01-11T08:11:28.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.439+0000 7fe3a5fff700 1 -- 172.21.15.27:0/4060587572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a00b2b90 msgr2=0x7fe3a016cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:28.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.439+0000 7fe3a5fff700 1 --2- 172.21.15.27:0/4060587572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a00b2b90 0x7fe3a016cd60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe39400c7c0 tx=0x7fe394007f70 comp rx=0 tx=0).stop 2024-01-11T08:11:28.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.439+0000 7fe3a5fff700 1 -- 172.21.15.27:0/4060587572 shutdown_connections 2024-01-11T08:11:28.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.439+0000 7fe3a5fff700 1 --2- 172.21.15.27:0/4060587572 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe38c066f30 0x7fe38c0693e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:28.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.439+0000 7fe3a5fff700 1 --2- 172.21.15.27:0/4060587572 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe3a00cc410 0x7fe3a016d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:28.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.439+0000 7fe3a5fff700 1 --2- 172.21.15.27:0/4060587572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3a00b03e0 0x7fe3a00c4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:28.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.439+0000 7fe3a5fff700 1 --2- 172.21.15.27:0/4060587572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a00b2b90 0x7fe3a016cd60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:28.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.439+0000 7fe3a5fff700 1 -- 172.21.15.27:0/4060587572 >> 172.21.15.27:0/4060587572 conn(0x7fe3a006ed40 msgr2=0x7fe3a01596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:28.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.439+0000 7fe3a5fff700 1 -- 172.21.15.27:0/4060587572 shutdown_connections 2024-01-11T08:11:28.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.439+0000 7fe3a5fff700 1 -- 172.21.15.27:0/4060587572 wait complete. 2024-01-11T08:11:28.440 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-01-11T08:11:28.457 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:11:28.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.657+0000 7f4f617ff700 1 Processor -- start 2024-01-11T08:11:28.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.657+0000 7f4f617ff700 1 -- start start 2024-01-11T08:11:28.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.658+0000 7f4f617ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4f5c0b03e0 0x7f4f5c0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:28.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.658+0000 7f4f617ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4f5c0b07b0 0x7f4f5c0b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:28.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.658+0000 7f4f617ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f5c0cd730 con 0x7f4f5c159200 2024-01-11T08:11:28.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.658+0000 7f4f617ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f5c159d30 con 0x7f4f5c0b07b0 2024-01-11T08:11:28.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.658+0000 7f4f617ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f5c0ccb20 con 0x7f4f5c0b03e0 2024-01-11T08:11:28.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.658+0000 7f4f53fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4f5c0b07b0 0x7f4f5c0b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:28.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.658+0000 7f4f5b7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4f5c159200 0x7f4f5c159620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50696/0 (socket says 172.21.15.27:50696) 2024-01-11T08:11:28.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.658+0000 7f4f5b7fe700 1 -- 172.21.15.27:0/3375951533 learned_addr learned my addr 172.21.15.27:0/3375951533 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:28.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.659+0000 7f4f5affd700 1 --2- 172.21.15.27:0/3375951533 >> v2:172.21.15.77:3300/0 conn(0x7f4f5c0b03e0 0x7f4f5c0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:28.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.659+0000 7f4f53fff700 1 -- 172.21.15.27:0/3375951533 >> v2:172.21.15.77:3300/0 conn(0x7f4f5c0b03e0 msgr2=0x7f4f5c0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:28.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.659+0000 7f4f53fff700 1 --2- 172.21.15.27:0/3375951533 >> v2:172.21.15.77:3300/0 conn(0x7f4f5c0b03e0 0x7f4f5c0b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:28.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.659+0000 7f4f53fff700 1 -- 172.21.15.27:0/3375951533 >> v1:172.21.15.27:6789/0 conn(0x7f4f5c159200 legacy=0x7f4f5c159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:28.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.659+0000 7f4f53fff700 1 -- 172.21.15.27:0/3375951533 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5c0cce00 con 0x7f4f5c0b07b0 2024-01-11T08:11:28.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.659+0000 7f4f5affd700 1 --2- 172.21.15.27:0/3375951533 >> v2:172.21.15.77:3300/0 conn(0x7f4f5c0b03e0 0x7f4f5c0b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:28.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.659+0000 7f4f5a7fc700 1 -- 172.21.15.27:0/3375951533 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926835084 0 0) 0x7f4f5c0cd730 con 0x7f4f5c159200 2024-01-11T08:11:28.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.659+0000 7f4f53fff700 1 --2- 172.21.15.27:0/3375951533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f5c0b07b0 0x7f4f5c0b29c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c089470 tx=0x7f4f440091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11be9510275e9a7a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:28.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.659+0000 7f4f5a7fc700 1 -- 172.21.15.27:0/3375951533 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4400b6b0 con 0x7f4f5c0b07b0 2024-01-11T08:11:28.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.659+0000 7f4f5a7fc700 1 -- 172.21.15.27:0/3375951533 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4400b810 con 0x7f4f5c0b07b0 2024-01-11T08:11:28.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.659+0000 7f4f5a7fc700 1 -- 172.21.15.27:0/3375951533 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4400bbb0 con 0x7f4f5c0b07b0 2024-01-11T08:11:28.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.660+0000 7f4f617ff700 1 -- 172.21.15.27:0/3375951533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f5c0b07b0 msgr2=0x7f4f5c0b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:28.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.660+0000 7f4f617ff700 1 --2- 172.21.15.27:0/3375951533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f5c0b07b0 0x7f4f5c0b29c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c089470 tx=0x7f4f440091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:28.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.660+0000 7f4f617ff700 1 -- 172.21.15.27:0/3375951533 shutdown_connections 2024-01-11T08:11:28.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.660+0000 7f4f617ff700 1 --2- 172.21.15.27:0/3375951533 >> v2:172.21.15.77:3300/0 conn(0x7f4f5c0b03e0 0x7f4f5c0b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:28.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.660+0000 7f4f617ff700 1 --2- 172.21.15.27:0/3375951533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f5c0b07b0 0x7f4f5c0b29c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:28.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.660+0000 7f4f617ff700 1 -- 172.21.15.27:0/3375951533 >> 172.21.15.27:0/3375951533 conn(0x7f4f5c06ed40 msgr2=0x7f4f5c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:28.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.660+0000 7f4f617ff700 1 -- 172.21.15.27:0/3375951533 shutdown_connections 2024-01-11T08:11:28.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.660+0000 7f4f617ff700 1 -- 172.21.15.27:0/3375951533 wait complete. 2024-01-11T08:11:28.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.662+0000 7f4f617ff700 1 Processor -- start 2024-01-11T08:11:28.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.662+0000 7f4f617ff700 1 -- start start 2024-01-11T08:11:28.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.663+0000 7f4f617ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f5c0b03e0 0x7f4f5c16b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:28.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.663+0000 7f4f617ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4f5c0b07b0 0x7f4f5c0a7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:28.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.663+0000 7f4f617ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f5c159200 0x7f4f5c157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:28.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.663+0000 7f4f617ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c0a76b0 con 0x7f4f5c0b03e0 2024-01-11T08:11:28.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.663+0000 7f4f617ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c16c640 con 0x7f4f5c159200 2024-01-11T08:11:28.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.663+0000 7f4f617ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c16ba30 con 0x7f4f5c0b07b0 2024-01-11T08:11:28.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.663+0000 7f4f5affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f5c0b03e0 0x7f4f5c16b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:28.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.663+0000 7f4f5affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f5c0b03e0 0x7f4f5c16b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42408/0 (socket says 172.21.15.27:42408) 2024-01-11T08:11:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.663+0000 7f4f5affd700 1 -- 172.21.15.27:0/28664659 learned_addr learned my addr 172.21.15.27:0/28664659 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.663+0000 7f4f53fff700 1 --2- 172.21.15.27:0/28664659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4f5c0b07b0 0x7f4f5c0a7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.663+0000 7f4f5b7fe700 1 --2- 172.21.15.27:0/28664659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f5c159200 0x7f4f5c157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.663+0000 7f4f5affd700 1 -- 172.21.15.27:0/28664659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4f5c0b07b0 msgr2=0x7f4f5c0a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.663+0000 7f4f5affd700 1 --2- 172.21.15.27:0/28664659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4f5c0b07b0 0x7f4f5c0a7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.663+0000 7f4f5affd700 1 -- 172.21.15.27:0/28664659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f5c159200 msgr2=0x7f4f5c157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.663+0000 7f4f5affd700 1 --2- 172.21.15.27:0/28664659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f5c159200 0x7f4f5c157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.664+0000 7f4f5affd700 1 -- 172.21.15.27:0/28664659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f44009420 con 0x7f4f5c0b03e0 2024-01-11T08:11:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.664+0000 7f4f5b7fe700 1 --2- 172.21.15.27:0/28664659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f5c159200 0x7f4f5c157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:11:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.664+0000 7f4f5affd700 1 --2- 172.21.15.27:0/28664659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f5c0b03e0 0x7f4f5c16b420 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c00e8d0 tx=0x7f4f4c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.664+0000 7f4f52ffd700 1 -- 172.21.15.27:0/28664659 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c00ee70 con 0x7f4f5c0b03e0 2024-01-11T08:11:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.664+0000 7f4f52ffd700 1 -- 172.21.15.27:0/28664659 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4c00c5e0 con 0x7f4f5c0b03e0 2024-01-11T08:11:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.664+0000 7f4f52ffd700 1 -- 172.21.15.27:0/28664659 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c00c980 con 0x7f4f5c0b03e0 2024-01-11T08:11:28.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.664+0000 7f4f617ff700 1 -- 172.21.15.27:0/28664659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f5c16bd10 con 0x7f4f5c0b03e0 2024-01-11T08:11:28.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.664+0000 7f4f617ff700 1 -- 172.21.15.27:0/28664659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5c16c2d0 con 0x7f4f5c0b03e0 2024-01-11T08:11:28.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.665+0000 7f4f617ff700 1 -- 172.21.15.27:0/28664659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f5c16c920 con 0x7f4f5c0b03e0 2024-01-11T08:11:28.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.666+0000 7f4f52ffd700 1 -- 172.21.15.27:0/28664659 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4f4c013410 con 0x7f4f5c0b03e0 2024-01-11T08:11:28.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.667+0000 7f4f52ffd700 1 --2- 172.21.15.27:0/28664659 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4f3c066f30 0x7f4f3c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:28.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.667+0000 7f4f52ffd700 1 -- 172.21.15.27:0/28664659 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f4f4c084430 con 0x7f4f5c0b03e0 2024-01-11T08:11:28.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.667+0000 7f4f53fff700 1 --2- 172.21.15.27:0/28664659 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4f3c066f30 0x7f4f3c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:28.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.668+0000 7f4f53fff700 1 --2- 172.21.15.27:0/28664659 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4f3c066f30 0x7f4f3c0693e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4f440093f0 tx=0x7f4f44017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:28.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.671+0000 7f4f52ffd700 1 -- 172.21.15.27:0/28664659 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4f4c051600 con 0x7f4f5c0b03e0 2024-01-11T08:11:28.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.875+0000 7f4f617ff700 1 -- 172.21.15.27:0/28664659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f5c0718d0 con 0x7f4f5c0b03e0 2024-01-11T08:11:28.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.876+0000 7f4f52ffd700 1 -- 172.21.15.27:0/28664659 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+15302 (secure 0 0 0) 0x7f4f4c054c20 con 0x7f4f5c0b03e0 2024-01-11T08:11:28.877 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:28.877 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:11:25.474635+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:11:22.376067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-11T08:11:23.419993+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:11:28.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.879+0000 7f4f617ff700 1 -- 172.21.15.27:0/28664659 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4f3c066f30 msgr2=0x7f4f3c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:28.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.879+0000 7f4f617ff700 1 --2- 172.21.15.27:0/28664659 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4f3c066f30 0x7f4f3c0693e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4f440093f0 tx=0x7f4f44017000 comp rx=0 tx=0).stop 2024-01-11T08:11:28.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.879+0000 7f4f617ff700 1 -- 172.21.15.27:0/28664659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f5c0b03e0 msgr2=0x7f4f5c16b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:28.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.879+0000 7f4f617ff700 1 --2- 172.21.15.27:0/28664659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f5c0b03e0 0x7f4f5c16b420 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c00e8d0 tx=0x7f4f4c008040 comp rx=0 tx=0).stop 2024-01-11T08:11:28.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.879+0000 7f4f617ff700 1 -- 172.21.15.27:0/28664659 shutdown_connections 2024-01-11T08:11:28.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.879+0000 7f4f617ff700 1 --2- 172.21.15.27:0/28664659 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4f3c066f30 0x7f4f3c0693e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:28.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.879+0000 7f4f617ff700 1 --2- 172.21.15.27:0/28664659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f5c159200 0x7f4f5c157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:28.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.879+0000 7f4f617ff700 1 --2- 172.21.15.27:0/28664659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f5c0b03e0 0x7f4f5c16b420 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:28.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.879+0000 7f4f617ff700 1 --2- 172.21.15.27:0/28664659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4f5c0b07b0 0x7f4f5c0a7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:28.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.879+0000 7f4f617ff700 1 -- 172.21.15.27:0/28664659 >> 172.21.15.27:0/28664659 conn(0x7f4f5c06ed40 msgr2=0x7f4f5c158a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:28.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.879+0000 7f4f617ff700 1 -- 172.21.15.27:0/28664659 shutdown_connections 2024-01-11T08:11:28.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:28.879+0000 7f4f617ff700 1 -- 172.21.15.27:0/28664659 wait complete. 2024-01-11T08:11:28.897 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-11T08:11:28.897 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:11:29.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.096+0000 7fc46d5ff700 1 Processor -- start 2024-01-11T08:11:29.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.096+0000 7fc46d5ff700 1 -- start start 2024-01-11T08:11:29.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.098+0000 7fc46d5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc4680c80d0 0x7fc4680c3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:29.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.098+0000 7fc46d5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4680cad00 con 0x7fc46813fac0 2024-01-11T08:11:29.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.098+0000 7fc46d5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4680ca0f0 con 0x7fc46806e740 2024-01-11T08:11:29.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.098+0000 7fc46d5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4680ca3d0 con 0x7fc4680c80d0 2024-01-11T08:11:29.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.098+0000 7fc466ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc46813fac0 0x7fc4680c79c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50704/0 (socket says 172.21.15.27:50704) 2024-01-11T08:11:29.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.098+0000 7fc466ffd700 1 -- 172.21.15.27:0/1528611454 learned_addr learned my addr 172.21.15.27:0/1528611454 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:29.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.098+0000 7fc4677fe700 1 --2- 172.21.15.27:0/1528611454 >> v2:172.21.15.77:3300/0 conn(0x7fc4680c80d0 0x7fc4680c3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:29.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.098+0000 7fc465ffb700 1 -- 172.21.15.27:0/1528611454 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992254398 0 0) 0x7fc4680cad00 con 0x7fc46813fac0 2024-01-11T08:11:29.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.098+0000 7fc465ffb700 1 -- 172.21.15.27:0/1528611454 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc450003570 con 0x7fc46813fac0 2024-01-11T08:11:29.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.099+0000 7fc4677fe700 1 -- 172.21.15.27:0/1528611454 >> v1:172.21.15.77:6789/0 conn(0x7fc46806e740 legacy=0x7fc4680cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:29.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.099+0000 7fc4677fe700 1 -- 172.21.15.27:0/1528611454 >> v1:172.21.15.27:6789/0 conn(0x7fc46813fac0 legacy=0x7fc4680c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:29.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.099+0000 7fc4677fe700 1 -- 172.21.15.27:0/1528611454 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4680ca6b0 con 0x7fc4680c80d0 2024-01-11T08:11:29.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.099+0000 7fc4677fe700 1 --2- 172.21.15.27:0/1528611454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4680c80d0 0x7fc4680c3210 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc460009e60 tx=0x7fc460009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b24d1c2effc2b55 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:29.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.099+0000 7fc465ffb700 1 -- 172.21.15.27:0/1528611454 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46000e070 con 0x7fc4680c80d0 2024-01-11T08:11:29.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.099+0000 7fc465ffb700 1 -- 172.21.15.27:0/1528611454 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46000bb60 con 0x7fc4680c80d0 2024-01-11T08:11:29.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.099+0000 7fc465ffb700 1 -- 172.21.15.27:0/1528611454 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc460012430 con 0x7fc4680c80d0 2024-01-11T08:11:29.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.100+0000 7fc46d5ff700 1 -- 172.21.15.27:0/1528611454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4680c80d0 msgr2=0x7fc4680c3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:29.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.100+0000 7fc46d5ff700 1 --2- 172.21.15.27:0/1528611454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4680c80d0 0x7fc4680c3210 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc460009e60 tx=0x7fc460009780 comp rx=0 tx=0).stop 2024-01-11T08:11:29.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.100+0000 7fc46d5ff700 1 -- 172.21.15.27:0/1528611454 shutdown_connections 2024-01-11T08:11:29.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.100+0000 7fc46d5ff700 1 --2- 172.21.15.27:0/1528611454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4680c80d0 0x7fc4680c3210 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:29.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.100+0000 7fc46d5ff700 1 -- 172.21.15.27:0/1528611454 >> 172.21.15.27:0/1528611454 conn(0x7fc4680680a0 msgr2=0x7fc4680cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:29.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.100+0000 7fc46d5ff700 1 -- 172.21.15.27:0/1528611454 shutdown_connections 2024-01-11T08:11:29.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.100+0000 7fc46d5ff700 1 -- 172.21.15.27:0/1528611454 wait complete. 2024-01-11T08:11:29.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.101+0000 7fc46d5ff700 1 Processor -- start 2024-01-11T08:11:29.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.102+0000 7fc46d5ff700 1 -- start start 2024-01-11T08:11:29.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.102+0000 7fc46d5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc46806e740 0x7fc4680b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:29.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.103+0000 7fc46d5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4680c80d0 0x7fc46815c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:29.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.103+0000 7fc46d5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc46813fac0 0x7fc4680c3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:29.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.103+0000 7fc46d5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4680c28e0 con 0x7fc46813fac0 2024-01-11T08:11:29.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.103+0000 7fc46d5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4680b1a90 con 0x7fc4680c80d0 2024-01-11T08:11:29.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.103+0000 7fc46d5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4680c18a0 con 0x7fc46806e740 2024-01-11T08:11:29.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.103+0000 7fc4677fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc46813fac0 0x7fc4680c3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:29.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.103+0000 7fc4677fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc46813fac0 0x7fc4680c3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42424/0 (socket says 172.21.15.27:42424) 2024-01-11T08:11:29.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.103+0000 7fc4677fe700 1 -- 172.21.15.27:0/4126284526 learned_addr learned my addr 172.21.15.27:0/4126284526 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:29.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.103+0000 7fc466ffd700 1 --2- 172.21.15.27:0/4126284526 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc46806e740 0x7fc4680b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:29.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.103+0000 7fc4667fc700 1 --2- 172.21.15.27:0/4126284526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4680c80d0 0x7fc46815c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:29.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.103+0000 7fc4677fe700 1 -- 172.21.15.27:0/4126284526 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc46806e740 msgr2=0x7fc4680b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:29.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.103+0000 7fc4677fe700 1 --2- 172.21.15.27:0/4126284526 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc46806e740 0x7fc4680b1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:29.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.103+0000 7fc4677fe700 1 -- 172.21.15.27:0/4126284526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4680c80d0 msgr2=0x7fc46815c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:29.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.103+0000 7fc4677fe700 1 --2- 172.21.15.27:0/4126284526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4680c80d0 0x7fc46815c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:29.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.103+0000 7fc4677fe700 1 -- 172.21.15.27:0/4126284526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc460009a20 con 0x7fc46813fac0 2024-01-11T08:11:29.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.104+0000 7fc4677fe700 1 --2- 172.21.15.27:0/4126284526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc46813fac0 0x7fc4680c3790 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc460013000 tx=0x7fc460019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:29.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.104+0000 7fc44effd700 1 -- 172.21.15.27:0/4126284526 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc460027470 con 0x7fc46813fac0 2024-01-11T08:11:29.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.104+0000 7fc44effd700 1 -- 172.21.15.27:0/4126284526 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4600275f0 con 0x7fc46813fac0 2024-01-11T08:11:29.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.104+0000 7fc44effd700 1 -- 172.21.15.27:0/4126284526 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4600279b0 con 0x7fc46813fac0 2024-01-11T08:11:29.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.104+0000 7fc46d5ff700 1 -- 172.21.15.27:0/4126284526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4680c1b80 con 0x7fc46813fac0 2024-01-11T08:11:29.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.104+0000 7fc46d5ff700 1 -- 172.21.15.27:0/4126284526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4680c2140 con 0x7fc46813fac0 2024-01-11T08:11:29.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.105+0000 7fc46d5ff700 1 -- 172.21.15.27:0/4126284526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc46815af00 con 0x7fc46813fac0 2024-01-11T08:11:29.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.106+0000 7fc44effd700 1 -- 172.21.15.27:0/4126284526 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc460024070 con 0x7fc46813fac0 2024-01-11T08:11:29.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.107+0000 7fc44effd700 1 --2- 172.21.15.27:0/4126284526 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc454066f30 0x7fc4540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:29.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.107+0000 7fc44effd700 1 -- 172.21.15.27:0/4126284526 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fc460086e40 con 0x7fc46813fac0 2024-01-11T08:11:29.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.107+0000 7fc466ffd700 1 --2- 172.21.15.27:0/4126284526 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc454066f30 0x7fc4540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:29.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.108+0000 7fc466ffd700 1 --2- 172.21.15.27:0/4126284526 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc454066f30 0x7fc4540693e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc458001360 tx=0x7fc458009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:29.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.111+0000 7fc44effd700 1 -- 172.21.15.27:0/4126284526 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc46815af00 con 0x7fc46813fac0 2024-01-11T08:11:29.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.308+0000 7fc46d5ff700 1 -- 172.21.15.27:0/4126284526 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc46815af00 con 0x7fc454066f30 2024-01-11T08:11:29.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.319+0000 7fc44effd700 1 -- 172.21.15.27:0/4126284526 <== mgr.4099 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+277804 (secure 0 0 0) 0x7fc46815af00 con 0x7fc454066f30 2024-01-11T08:11:29.319 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:29.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.323+0000 7fc46d5ff700 1 -- 172.21.15.27:0/4126284526 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc454066f30 msgr2=0x7fc4540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:29.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.323+0000 7fc46d5ff700 1 --2- 172.21.15.27:0/4126284526 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc454066f30 0x7fc4540693e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc458001360 tx=0x7fc458009040 comp rx=0 tx=0).stop 2024-01-11T08:11:29.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.323+0000 7fc46d5ff700 1 -- 172.21.15.27:0/4126284526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc46813fac0 msgr2=0x7fc4680c3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.323+0000 7fc46d5ff700 1 --2- 172.21.15.27:0/4126284526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc46813fac0 0x7fc4680c3790 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc460013000 tx=0x7fc460019000 comp rx=0 tx=0).stop 2024-01-11T08:11:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.323+0000 7fc46d5ff700 1 -- 172.21.15.27:0/4126284526 shutdown_connections 2024-01-11T08:11:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.323+0000 7fc46d5ff700 1 --2- 172.21.15.27:0/4126284526 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fc454066f30 0x7fc4540693e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.323+0000 7fc46d5ff700 1 --2- 172.21.15.27:0/4126284526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc46813fac0 0x7fc4680c3790 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.323+0000 7fc46d5ff700 1 --2- 172.21.15.27:0/4126284526 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc46806e740 0x7fc4680b1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.323+0000 7fc46d5ff700 1 --2- 172.21.15.27:0/4126284526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4680c80d0 0x7fc46815c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.323+0000 7fc46d5ff700 1 -- 172.21.15.27:0/4126284526 >> 172.21.15.27:0/4126284526 conn(0x7fc4680680a0 msgr2=0x7fc46815cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.323+0000 7fc46d5ff700 1 -- 172.21.15.27:0/4126284526 shutdown_connections 2024-01-11T08:11:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.323+0000 7fc46d5ff700 1 -- 172.21.15.27:0/4126284526 wait complete. 2024-01-11T08:11:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:11:29.340 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-01-11T08:11:27.522360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2382148,"kb_used_data":3844,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752592572,"statfs":{"total":773094113280,"available":770654793728,"internally_reserved":0,"allocated":3936256,"data_stored":2215585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.608088"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.922436+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:09:52.585500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:23.120268+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:33:11.028049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.919266+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:10:41.966291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.925500+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:59:33.762129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.925992+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:28:10.562456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.925053+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:40:51.401692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.919380+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:55:31.548418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.925186+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:43:13.359811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.917262+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:40:50.968620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:23.121115+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:09:52.585500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.917539+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:12:41.969655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.917155+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:09:52.585500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.925264+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:59:33.762129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.917474+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:33:11.028049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.917378+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:09:20.134819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.918997+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:12:41.969655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.918375+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:09:52.585500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.925245+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:20:07.483752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.925154+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:25:39.967919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.925217+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:54:47.128267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:23.120784+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:53:33.242677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.922408+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:17.235033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.925890+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:31:38.322574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.917539+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:10:41.966291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.917768+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:17.235033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:23.120662+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:17.235033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.917608+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:29:09.480491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:23.120269+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:29:09.480491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.918067+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:53:33.242677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.925108+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:30:52.479899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.917199+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:10:53.026857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T08:11:23.914281+0000","last_change":"2024-01-11T08:11:23.914281+0000","last_active":"2024-01-11T08:11:23.914281+0000","last_peered":"2024-01-11T08:11:23.914281+0000","last_clean":"2024-01-11T08:11:23.914281+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:23.914281+0000","last_undegraded":"2024-01-11T08:11:23.914281+0000","last_fullsized":"2024-01-11T08:11:23.914281+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-01-11T08:11:22.904830+0000","last_change":"2024-01-11T08:11:22.922420+0000","last_active":"2024-01-11T08:11:22.904830+0000","last_peered":"2024-01-11T08:11:22.904830+0000","last_clean":"2024-01-11T08:11:22.904830+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T08:11:22.904830+0000","last_undegraded":"2024-01-11T08:11:22.904830+0000","last_fullsized":"2024-01-11T08:11:22.904830+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:53:33.242677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":168,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-11T08:11:17.169781+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:11:17.169781+0000","last_peered":"2024-01-11T08:11:17.169781+0000","last_clean":"2024-01-11T08:11:17.169781+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:11:17.169781+0000","last_undegraded":"2024-01-11T08:11:17.169781+0000","last_fullsized":"2024-01-11T08:11:17.169781+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298196,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073644,"statfs":{"total":96636764160,"available":96331411456,"internally_reserved":0,"allocated":929792,"data_stored":716096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":2.8300000000000001,"5min":2.8300000000000001,"15min":2.8300000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.65100000000000002}]},{"osd":1,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.57999999999999996}]},{"osd":2,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.78800000000000003}]},{"osd":3,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":2.9119999999999999,"5min":2.9119999999999999,"15min":2.9119999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.629}]},{"osd":4,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.70499999999999996}]},{"osd":5,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.77100000000000002}]},{"osd":6,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.67100000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297628,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074212,"statfs":{"total":96636764160,"available":96331993088,"internally_reserved":0,"allocated":348160,"data_stored":131533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.55100000000000005}]},{"osd":1,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.97299999999999998}]},{"osd":2,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.79400000000000004}]},{"osd":3,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.70399999999999996}]},{"osd":4,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.83799999999999997}]},{"osd":5,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":1.0009999999999999}]},{"osd":7,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.88}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297628,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074212,"statfs":{"total":96636764160,"available":96331993088,"internally_reserved":0,"allocated":348160,"data_stored":131533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.66600000000000004}]},{"osd":2,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.59899999999999998}]},{"osd":3,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.57099999999999995}]},{"osd":4,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.82199999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.77500000000000002}]},{"osd":6,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.871},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.84299999999999997}]},{"osd":7,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.85699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298196,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073644,"statfs":{"total":96636764160,"available":96331411456,"internally_reserved":0,"allocated":929792,"data_stored":716096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.40899999999999997}]},{"osd":2,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.68999999999999995}]},{"osd":3,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.48499999999999999}]},{"osd":4,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.60199999999999998}]},{"osd":5,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.61299999999999999}]},{"osd":6,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.66700000000000004}]},{"osd":7,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.70999999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297628,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074212,"statfs":{"total":96636764160,"available":96331993088,"internally_reserved":0,"allocated":348160,"data_stored":131533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.55100000000000005}]},{"osd":1,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":0.72299999999999998}]},{"osd":3,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.73699999999999999}]},{"osd":4,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.64100000000000001}]},{"osd":5,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":2.9209999999999998,"5min":2.9209999999999998,"15min":2.9209999999999998},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":0.77600000000000002}]},{"osd":6,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.754},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":2.968,"5min":2.968,"15min":2.968},"last":0.68400000000000005}]},{"osd":7,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":0.70699999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297628,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074212,"statfs":{"total":96636764160,"available":96331993088,"internally_reserved":0,"allocated":348160,"data_stored":131533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.505},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.55800000000000005}]},{"osd":1,"last update":"Thu Jan 11 08:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.54100000000000004}]},{"osd":2,"last update":"Thu Jan 11 08:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.57399999999999995}]},{"osd":4,"last update":"Thu Jan 11 08:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.88900000000000001}]},{"osd":5,"last update":"Thu Jan 11 08:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.69399999999999995}]},{"osd":6,"last update":"Thu Jan 11 08:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":3.5579999999999998,"5min":3.5579999999999998,"15min":3.5579999999999998},"last":0.73799999999999999}]},{"osd":7,"last update":"Thu Jan 11 08:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":3.5089999999999999,"5min":3.5089999999999999,"15min":3.5089999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.85999999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297616,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074224,"statfs":{"total":96636764160,"available":96332005376,"internally_reserved":0,"allocated":335872,"data_stored":125728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.65700000000000003}]},{"osd":1,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.4740000000000002,"5min":2.4740000000000002,"15min":2.4740000000000002},"last":0.69199999999999995}]},{"osd":2,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.60999999999999999}]},{"osd":3,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.57299999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.47699999999999998}]},{"osd":6,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.78400000000000003}]},{"osd":7,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.75700000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297628,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074212,"statfs":{"total":96636764160,"available":96331993088,"internally_reserved":0,"allocated":348160,"data_stored":131533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.69699999999999995}]},{"osd":1,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.79800000000000004}]},{"osd":2,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.76100000000000001}]},{"osd":3,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.78400000000000003}]},{"osd":4,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.621},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.55400000000000005}]},{"osd":6,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.57699999999999996}]},{"osd":7,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.65500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:11:29.345 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:11:29.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.580+0000 7f65aa1ff700 1 Processor -- start 2024-01-11T08:11:29.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.581+0000 7f65aa1ff700 1 -- start start 2024-01-11T08:11:29.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.582+0000 7f65aa1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f65a40b03e0 0x7f65a40b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:29.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.582+0000 7f65aa1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65a40cd730 con 0x7f65a40b29c0 2024-01-11T08:11:29.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.582+0000 7f65aa1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65a40b2de0 con 0x7f65a40b07b0 2024-01-11T08:11:29.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.582+0000 7f65aa1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65a40ccb20 con 0x7f65a40b03e0 2024-01-11T08:11:29.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.582+0000 7f65a37fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f65a40b03e0 0x7f65a40b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:29.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.582+0000 7f65a3fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f65a40b29c0 0x7f65a40aefc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50720/0 (socket says 172.21.15.27:50720) 2024-01-11T08:11:29.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.582+0000 7f65a37fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f65a40b03e0 0x7f65a40b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42438/0 (socket says 172.21.15.27:42438) 2024-01-11T08:11:29.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.582+0000 7f65a3fff700 1 -- 172.21.15.27:0/2455584844 learned_addr learned my addr 172.21.15.27:0/2455584844 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.583+0000 7f65a37fe700 1 -- 172.21.15.27:0/2455584844 >> v1:172.21.15.77:6789/0 conn(0x7f65a40b07b0 legacy=0x7f65a4159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.583+0000 7f65a37fe700 1 -- 172.21.15.27:0/2455584844 >> v1:172.21.15.27:6789/0 conn(0x7f65a40b29c0 legacy=0x7f65a40aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.583+0000 7f65a37fe700 1 -- 172.21.15.27:0/2455584844 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a40cce00 con 0x7f65a40b03e0 2024-01-11T08:11:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.583+0000 7f65a27fc700 1 -- 172.21.15.27:0/2455584844 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385960483 0 0) 0x7f65a40cd730 con 0x7f65a40b29c0 2024-01-11T08:11:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.583+0000 7f65a37fe700 1 --2- 172.21.15.27:0/2455584844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65a40b03e0 0x7f65a40b2480 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f65940098e0 tx=0x7f65940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b54f2d7ced2ce44d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.583+0000 7f65a27fc700 1 -- 172.21.15.27:0/2455584844 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594009e80 con 0x7f65a40b03e0 2024-01-11T08:11:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.583+0000 7f65a27fc700 1 -- 172.21.15.27:0/2455584844 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659400b860 con 0x7f65a40b03e0 2024-01-11T08:11:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.583+0000 7f65a27fc700 1 -- 172.21.15.27:0/2455584844 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659400bbe0 con 0x7f65a40b03e0 2024-01-11T08:11:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.584+0000 7f65aa1ff700 1 -- 172.21.15.27:0/2455584844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65a40b03e0 msgr2=0x7f65a40b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.584+0000 7f65aa1ff700 1 --2- 172.21.15.27:0/2455584844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65a40b03e0 0x7f65a40b2480 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f65940098e0 tx=0x7f65940091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.584+0000 7f65aa1ff700 1 -- 172.21.15.27:0/2455584844 shutdown_connections 2024-01-11T08:11:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.584+0000 7f65aa1ff700 1 --2- 172.21.15.27:0/2455584844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65a40b03e0 0x7f65a40b2480 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.584+0000 7f65aa1ff700 1 -- 172.21.15.27:0/2455584844 >> 172.21.15.27:0/2455584844 conn(0x7f65a406ed40 msgr2=0x7f65a406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.584+0000 7f65aa1ff700 1 -- 172.21.15.27:0/2455584844 shutdown_connections 2024-01-11T08:11:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.584+0000 7f65aa1ff700 1 -- 172.21.15.27:0/2455584844 wait complete. 2024-01-11T08:11:29.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.586+0000 7f65aa1ff700 1 Processor -- start 2024-01-11T08:11:29.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.586+0000 7f65aa1ff700 1 -- start start 2024-01-11T08:11:29.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.587+0000 7f65aa1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65a40b03e0 0x7f65a4159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:29.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.587+0000 7f65aa1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65a40b07b0 0x7f65a4157730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:29.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.587+0000 7f65aa1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65a40b29c0 0x7f65a40c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:29.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.587+0000 7f65aa1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a40c49a0 con 0x7f65a40b07b0 2024-01-11T08:11:29.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.587+0000 7f65aa1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a40c4000 con 0x7f65a40b29c0 2024-01-11T08:11:29.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.587+0000 7f65aa1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a40c42e0 con 0x7f65a40b03e0 2024-01-11T08:11:29.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.587+0000 7f65a2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65a40b07b0 0x7f65a4157730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:29.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.587+0000 7f65a2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65a40b07b0 0x7f65a4157730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42440/0 (socket says 172.21.15.27:42440) 2024-01-11T08:11:29.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.587+0000 7f65a2ffd700 1 -- 172.21.15.27:0/3950729670 learned_addr learned my addr 172.21.15.27:0/3950729670 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:29.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.587+0000 7f65a37fe700 1 --2- 172.21.15.27:0/3950729670 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65a40b03e0 0x7f65a4159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:29.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.587+0000 7f65a3fff700 1 --2- 172.21.15.27:0/3950729670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65a40b29c0 0x7f65a40c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:29.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.587+0000 7f65a2ffd700 1 -- 172.21.15.27:0/3950729670 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65a40b03e0 msgr2=0x7f65a4159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:29.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.587+0000 7f65a2ffd700 1 --2- 172.21.15.27:0/3950729670 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65a40b03e0 0x7f65a4159770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:29.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.587+0000 7f65a2ffd700 1 -- 172.21.15.27:0/3950729670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65a40b29c0 msgr2=0x7f65a40c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:29.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.587+0000 7f65a2ffd700 1 --2- 172.21.15.27:0/3950729670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65a40b29c0 0x7f65a40c95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:29.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.587+0000 7f65a2ffd700 1 -- 172.21.15.27:0/3950729670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6594009420 con 0x7f65a40b07b0 2024-01-11T08:11:29.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.587+0000 7f65a37fe700 1 --2- 172.21.15.27:0/3950729670 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65a40b03e0 0x7f65a4159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:11:29.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.588+0000 7f65a2ffd700 1 --2- 172.21.15.27:0/3950729670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65a40b07b0 0x7f65a4157730 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f659800c590 tx=0x7f659800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:29.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.588+0000 7f658b7fe700 1 -- 172.21.15.27:0/3950729670 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6598011590 con 0x7f65a40b07b0 2024-01-11T08:11:29.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.588+0000 7f658b7fe700 1 -- 172.21.15.27:0/3950729670 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65980116f0 con 0x7f65a40b07b0 2024-01-11T08:11:29.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.588+0000 7f658b7fe700 1 -- 172.21.15.27:0/3950729670 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6598011ab0 con 0x7f65a40b07b0 2024-01-11T08:11:29.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.588+0000 7f65aa1ff700 1 -- 172.21.15.27:0/3950729670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a40c45c0 con 0x7f65a40b07b0 2024-01-11T08:11:29.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.588+0000 7f65aa1ff700 1 -- 172.21.15.27:0/3950729670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65a40c6a00 con 0x7f65a40b07b0 2024-01-11T08:11:29.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.589+0000 7f65aa1ff700 1 -- 172.21.15.27:0/3950729670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a40c4ae0 con 0x7f65a40b07b0 2024-01-11T08:11:29.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.590+0000 7f658b7fe700 1 -- 172.21.15.27:0/3950729670 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f659801b3b0 con 0x7f65a40b07b0 2024-01-11T08:11:29.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.591+0000 7f658b7fe700 1 --2- 172.21.15.27:0/3950729670 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6590066f30 0x7f65900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:29.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.591+0000 7f658b7fe700 1 -- 172.21.15.27:0/3950729670 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f6598083b00 con 0x7f65a40b07b0 2024-01-11T08:11:29.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.592+0000 7f65a37fe700 1 --2- 172.21.15.27:0/3950729670 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6590066f30 0x7f65900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:29.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.593+0000 7f65a37fe700 1 --2- 172.21.15.27:0/3950729670 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6590066f30 0x7f65900693e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f65940098e0 tx=0x7f6594017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:29.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.596+0000 7f658b7fe700 1 -- 172.21.15.27:0/3950729670 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f65a40c4ae0 con 0x7f65a40b07b0 2024-01-11T08:11:29.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.796+0000 7f65aa1ff700 1 -- 172.21.15.27:0/3950729670 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f65a40c4ae0 con 0x7f6590066f30 2024-01-11T08:11:29.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.805+0000 7f658b7fe700 1 -- 172.21.15.27:0/3950729670 <== mgr.4099 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288611 (secure 0 0 0) 0x7f65a40c4ae0 con 0x7f6590066f30 2024-01-11T08:11:29.809 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:29.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.809+0000 7f65aa1ff700 1 -- 172.21.15.27:0/3950729670 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6590066f30 msgr2=0x7f65900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:29.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.809+0000 7f65aa1ff700 1 --2- 172.21.15.27:0/3950729670 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6590066f30 0x7f65900693e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f65940098e0 tx=0x7f6594017000 comp rx=0 tx=0).stop 2024-01-11T08:11:29.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.809+0000 7f65aa1ff700 1 -- 172.21.15.27:0/3950729670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65a40b07b0 msgr2=0x7f65a4157730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.809+0000 7f65aa1ff700 1 --2- 172.21.15.27:0/3950729670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65a40b07b0 0x7f65a4157730 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f659800c590 tx=0x7f659800c950 comp rx=0 tx=0).stop 2024-01-11T08:11:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.809+0000 7f65aa1ff700 1 -- 172.21.15.27:0/3950729670 shutdown_connections 2024-01-11T08:11:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.809+0000 7f65aa1ff700 1 --2- 172.21.15.27:0/3950729670 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6590066f30 0x7f65900693e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.809+0000 7f65aa1ff700 1 --2- 172.21.15.27:0/3950729670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65a40b29c0 0x7f65a40c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.809+0000 7f65aa1ff700 1 --2- 172.21.15.27:0/3950729670 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65a40b03e0 0x7f65a4159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.809+0000 7f65aa1ff700 1 --2- 172.21.15.27:0/3950729670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65a40b07b0 0x7f65a4157730 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.809+0000 7f65aa1ff700 1 -- 172.21.15.27:0/3950729670 >> 172.21.15.27:0/3950729670 conn(0x7f65a406ed40 msgr2=0x7f65a415c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.810+0000 7f65aa1ff700 1 -- 172.21.15.27:0/3950729670 shutdown_connections 2024-01-11T08:11:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:29.810+0000 7f65aa1ff700 1 -- 172.21.15.27:0/3950729670 wait complete. 2024-01-11T08:11:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:11:29.827 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-01-11T08:11:29.522782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2386632,"kb_used_data":4232,"kb_used_omap":0,"kb_used_meta":2382336,"kb_avail":752588088,"statfs":{"total":773094113280,"available":770650202112,"internally_reserved":0,"allocated":4333568,"data_stored":2392060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2439512064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001257"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.499757+0000","last_change":"2024-01-11T08:11:24.492551+0000","last_active":"2024-01-11T08:11:24.499757+0000","last_peered":"2024-01-11T08:11:24.499757+0000","last_clean":"2024-01-11T08:11:24.499757+0000","last_became_active":"2024-01-11T08:11:24.491993+0000","last_became_peered":"2024-01-11T08:11:24.491993+0000","last_unstale":"2024-01-11T08:11:24.499757+0000","last_undegraded":"2024-01-11T08:11:24.499757+0000","last_fullsized":"2024-01-11T08:11:24.499757+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:27:00.721768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.500951+0000","last_change":"2024-01-11T08:11:24.493586+0000","last_active":"2024-01-11T08:11:24.500951+0000","last_peered":"2024-01-11T08:11:24.500951+0000","last_clean":"2024-01-11T08:11:24.500951+0000","last_became_active":"2024-01-11T08:11:24.493298+0000","last_became_peered":"2024-01-11T08:11:24.493298+0000","last_unstale":"2024-01-11T08:11:24.500951+0000","last_undegraded":"2024-01-11T08:11:24.500951+0000","last_fullsized":"2024-01-11T08:11:24.500951+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:09:21.568163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.506575+0000","last_change":"2024-01-11T08:11:24.502688+0000","last_active":"2024-01-11T08:11:24.506575+0000","last_peered":"2024-01-11T08:11:24.506575+0000","last_clean":"2024-01-11T08:11:24.506575+0000","last_became_active":"2024-01-11T08:11:24.502016+0000","last_became_peered":"2024-01-11T08:11:24.502016+0000","last_unstale":"2024-01-11T08:11:24.506575+0000","last_undegraded":"2024-01-11T08:11:24.506575+0000","last_fullsized":"2024-01-11T08:11:24.506575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:03:41.116550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.505605+0000","last_change":"2024-01-11T08:11:24.501947+0000","last_active":"2024-01-11T08:11:24.505605+0000","last_peered":"2024-01-11T08:11:24.505605+0000","last_clean":"2024-01-11T08:11:24.505605+0000","last_became_active":"2024-01-11T08:11:24.501621+0000","last_became_peered":"2024-01-11T08:11:24.501621+0000","last_unstale":"2024-01-11T08:11:24.505605+0000","last_undegraded":"2024-01-11T08:11:24.505605+0000","last_fullsized":"2024-01-11T08:11:24.505605+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:52:14.812398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.502880+0000","last_change":"2024-01-11T08:11:24.497279+0000","last_active":"2024-01-11T08:11:24.502880+0000","last_peered":"2024-01-11T08:11:24.502880+0000","last_clean":"2024-01-11T08:11:24.502880+0000","last_became_active":"2024-01-11T08:11:24.496999+0000","last_became_peered":"2024-01-11T08:11:24.496999+0000","last_unstale":"2024-01-11T08:11:24.502880+0000","last_undegraded":"2024-01-11T08:11:24.502880+0000","last_fullsized":"2024-01-11T08:11:24.502880+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:40:51.978071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.495227+0000","last_change":"2024-01-11T08:11:24.489253+0000","last_active":"2024-01-11T08:11:24.495227+0000","last_peered":"2024-01-11T08:11:24.495227+0000","last_clean":"2024-01-11T08:11:24.495227+0000","last_became_active":"2024-01-11T08:11:24.488999+0000","last_became_peered":"2024-01-11T08:11:24.488999+0000","last_unstale":"2024-01-11T08:11:24.495227+0000","last_undegraded":"2024-01-11T08:11:24.495227+0000","last_fullsized":"2024-01-11T08:11:24.495227+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:42:29.637708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.502294+0000","last_change":"2024-01-11T08:11:24.495178+0000","last_active":"2024-01-11T08:11:24.502294+0000","last_peered":"2024-01-11T08:11:24.502294+0000","last_clean":"2024-01-11T08:11:24.502294+0000","last_became_active":"2024-01-11T08:11:24.494914+0000","last_became_peered":"2024-01-11T08:11:24.494914+0000","last_unstale":"2024-01-11T08:11:24.502294+0000","last_undegraded":"2024-01-11T08:11:24.502294+0000","last_fullsized":"2024-01-11T08:11:24.502294+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:06:14.717197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.506138+0000","last_change":"2024-01-11T08:11:24.497350+0000","last_active":"2024-01-11T08:11:24.506138+0000","last_peered":"2024-01-11T08:11:24.506138+0000","last_clean":"2024-01-11T08:11:24.506138+0000","last_became_active":"2024-01-11T08:11:24.496889+0000","last_became_peered":"2024-01-11T08:11:24.496889+0000","last_unstale":"2024-01-11T08:11:24.506138+0000","last_undegraded":"2024-01-11T08:11:24.506138+0000","last_fullsized":"2024-01-11T08:11:24.506138+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:08:51.690170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.502775+0000","last_change":"2024-01-11T08:11:24.499065+0000","last_active":"2024-01-11T08:11:24.502775+0000","last_peered":"2024-01-11T08:11:24.502775+0000","last_clean":"2024-01-11T08:11:24.502775+0000","last_became_active":"2024-01-11T08:11:24.498803+0000","last_became_peered":"2024-01-11T08:11:24.498803+0000","last_unstale":"2024-01-11T08:11:24.502775+0000","last_undegraded":"2024-01-11T08:11:24.502775+0000","last_fullsized":"2024-01-11T08:11:24.502775+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:49:27.059571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.503373+0000","last_change":"2024-01-11T08:11:24.496116+0000","last_active":"2024-01-11T08:11:24.503373+0000","last_peered":"2024-01-11T08:11:24.503373+0000","last_clean":"2024-01-11T08:11:24.503373+0000","last_became_active":"2024-01-11T08:11:24.495656+0000","last_became_peered":"2024-01-11T08:11:24.495656+0000","last_unstale":"2024-01-11T08:11:24.503373+0000","last_undegraded":"2024-01-11T08:11:24.503373+0000","last_fullsized":"2024-01-11T08:11:24.503373+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:11:39.778515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.506392+0000","last_change":"2024-01-11T08:11:24.500279+0000","last_active":"2024-01-11T08:11:24.506392+0000","last_peered":"2024-01-11T08:11:24.506392+0000","last_clean":"2024-01-11T08:11:24.506392+0000","last_became_active":"2024-01-11T08:11:24.499961+0000","last_became_peered":"2024-01-11T08:11:24.499961+0000","last_unstale":"2024-01-11T08:11:24.506392+0000","last_undegraded":"2024-01-11T08:11:24.506392+0000","last_fullsized":"2024-01-11T08:11:24.506392+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:52:14.812398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.489999+0000","last_change":"2024-01-11T08:11:24.487280+0000","last_active":"2024-01-11T08:11:24.489999+0000","last_peered":"2024-01-11T08:11:24.489999+0000","last_clean":"2024-01-11T08:11:24.489999+0000","last_became_active":"2024-01-11T08:11:24.487066+0000","last_became_peered":"2024-01-11T08:11:24.487066+0000","last_unstale":"2024-01-11T08:11:24.489999+0000","last_undegraded":"2024-01-11T08:11:24.489999+0000","last_fullsized":"2024-01-11T08:11:24.489999+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:04:02.047325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.502908+0000","last_change":"2024-01-11T08:11:24.498637+0000","last_active":"2024-01-11T08:11:24.502908+0000","last_peered":"2024-01-11T08:11:24.502908+0000","last_clean":"2024-01-11T08:11:24.502908+0000","last_became_active":"2024-01-11T08:11:24.498356+0000","last_became_peered":"2024-01-11T08:11:24.498356+0000","last_unstale":"2024-01-11T08:11:24.502908+0000","last_undegraded":"2024-01-11T08:11:24.502908+0000","last_fullsized":"2024-01-11T08:11:24.502908+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:10:42.975741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.495247+0000","last_change":"2024-01-11T08:11:24.489891+0000","last_active":"2024-01-11T08:11:24.495247+0000","last_peered":"2024-01-11T08:11:24.495247+0000","last_clean":"2024-01-11T08:11:24.495247+0000","last_became_active":"2024-01-11T08:11:24.489645+0000","last_became_peered":"2024-01-11T08:11:24.489645+0000","last_unstale":"2024-01-11T08:11:24.495247+0000","last_undegraded":"2024-01-11T08:11:24.495247+0000","last_fullsized":"2024-01-11T08:11:24.495247+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:37:07.641386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.500470+0000","last_change":"2024-01-11T08:11:24.495074+0000","last_active":"2024-01-11T08:11:24.500470+0000","last_peered":"2024-01-11T08:11:24.500470+0000","last_clean":"2024-01-11T08:11:24.500470+0000","last_became_active":"2024-01-11T08:11:24.494793+0000","last_became_peered":"2024-01-11T08:11:24.494793+0000","last_unstale":"2024-01-11T08:11:24.500470+0000","last_undegraded":"2024-01-11T08:11:24.500470+0000","last_fullsized":"2024-01-11T08:11:24.500470+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:09:21.568163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.506160+0000","last_change":"2024-01-11T08:11:24.498053+0000","last_active":"2024-01-11T08:11:24.506160+0000","last_peered":"2024-01-11T08:11:24.506160+0000","last_clean":"2024-01-11T08:11:24.506160+0000","last_became_active":"2024-01-11T08:11:24.497320+0000","last_became_peered":"2024-01-11T08:11:24.497320+0000","last_unstale":"2024-01-11T08:11:24.506160+0000","last_undegraded":"2024-01-11T08:11:24.506160+0000","last_fullsized":"2024-01-11T08:11:24.506160+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:07:39.577576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.502366+0000","last_change":"2024-01-11T08:11:24.494756+0000","last_active":"2024-01-11T08:11:24.502366+0000","last_peered":"2024-01-11T08:11:24.502366+0000","last_clean":"2024-01-11T08:11:24.502366+0000","last_became_active":"2024-01-11T08:11:24.494487+0000","last_became_peered":"2024-01-11T08:11:24.494487+0000","last_unstale":"2024-01-11T08:11:24.502366+0000","last_undegraded":"2024-01-11T08:11:24.502366+0000","last_fullsized":"2024-01-11T08:11:24.502366+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:43:30.233475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.501818+0000","last_change":"2024-01-11T08:11:24.496856+0000","last_active":"2024-01-11T08:11:24.501818+0000","last_peered":"2024-01-11T08:11:24.501818+0000","last_clean":"2024-01-11T08:11:24.501818+0000","last_became_active":"2024-01-11T08:11:24.496370+0000","last_became_peered":"2024-01-11T08:11:24.496370+0000","last_unstale":"2024-01-11T08:11:24.501818+0000","last_undegraded":"2024-01-11T08:11:24.501818+0000","last_fullsized":"2024-01-11T08:11:24.501818+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:25:29.239749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.501896+0000","last_change":"2024-01-11T08:11:24.498839+0000","last_active":"2024-01-11T08:11:24.501896+0000","last_peered":"2024-01-11T08:11:24.501896+0000","last_clean":"2024-01-11T08:11:24.501896+0000","last_became_active":"2024-01-11T08:11:24.498533+0000","last_became_peered":"2024-01-11T08:11:24.498533+0000","last_unstale":"2024-01-11T08:11:24.501896+0000","last_undegraded":"2024-01-11T08:11:24.501896+0000","last_fullsized":"2024-01-11T08:11:24.501896+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:07:39.577576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.502813+0000","last_change":"2024-01-11T08:11:24.492280+0000","last_active":"2024-01-11T08:11:24.502813+0000","last_peered":"2024-01-11T08:11:24.502813+0000","last_clean":"2024-01-11T08:11:24.502813+0000","last_became_active":"2024-01-11T08:11:24.492011+0000","last_became_peered":"2024-01-11T08:11:24.492011+0000","last_unstale":"2024-01-11T08:11:24.502813+0000","last_undegraded":"2024-01-11T08:11:24.502813+0000","last_fullsized":"2024-01-11T08:11:24.502813+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:15:07.720958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.502640+0000","last_change":"2024-01-11T08:11:24.496893+0000","last_active":"2024-01-11T08:11:24.502640+0000","last_peered":"2024-01-11T08:11:24.502640+0000","last_clean":"2024-01-11T08:11:24.502640+0000","last_became_active":"2024-01-11T08:11:24.495876+0000","last_became_peered":"2024-01-11T08:11:24.495876+0000","last_unstale":"2024-01-11T08:11:24.502640+0000","last_undegraded":"2024-01-11T08:11:24.502640+0000","last_fullsized":"2024-01-11T08:11:24.502640+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:11:39.778515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.501428+0000","last_change":"2024-01-11T08:11:24.494238+0000","last_active":"2024-01-11T08:11:24.501428+0000","last_peered":"2024-01-11T08:11:24.501428+0000","last_clean":"2024-01-11T08:11:24.501428+0000","last_became_active":"2024-01-11T08:11:24.492821+0000","last_became_peered":"2024-01-11T08:11:24.492821+0000","last_unstale":"2024-01-11T08:11:24.501428+0000","last_undegraded":"2024-01-11T08:11:24.501428+0000","last_fullsized":"2024-01-11T08:11:24.501428+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:43:30.233475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.494527+0000","last_change":"2024-01-11T08:11:24.492044+0000","last_active":"2024-01-11T08:11:24.494527+0000","last_peered":"2024-01-11T08:11:24.494527+0000","last_clean":"2024-01-11T08:11:24.494527+0000","last_became_active":"2024-01-11T08:11:24.491790+0000","last_became_peered":"2024-01-11T08:11:24.491790+0000","last_unstale":"2024-01-11T08:11:24.494527+0000","last_undegraded":"2024-01-11T08:11:24.494527+0000","last_fullsized":"2024-01-11T08:11:24.494527+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:30:53.909322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.500911+0000","last_change":"2024-01-11T08:11:24.496337+0000","last_active":"2024-01-11T08:11:24.500911+0000","last_peered":"2024-01-11T08:11:24.500911+0000","last_clean":"2024-01-11T08:11:24.500911+0000","last_became_active":"2024-01-11T08:11:24.495790+0000","last_became_peered":"2024-01-11T08:11:24.495790+0000","last_unstale":"2024-01-11T08:11:24.500911+0000","last_undegraded":"2024-01-11T08:11:24.500911+0000","last_fullsized":"2024-01-11T08:11:24.500911+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:07:39.577576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.503505+0000","last_change":"2024-01-11T08:11:24.497653+0000","last_active":"2024-01-11T08:11:24.503505+0000","last_peered":"2024-01-11T08:11:24.503505+0000","last_clean":"2024-01-11T08:11:24.503505+0000","last_became_active":"2024-01-11T08:11:24.497345+0000","last_became_peered":"2024-01-11T08:11:24.497345+0000","last_unstale":"2024-01-11T08:11:24.503505+0000","last_undegraded":"2024-01-11T08:11:24.503505+0000","last_fullsized":"2024-01-11T08:11:24.503505+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:04:02.047325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.496470+0000","last_change":"2024-01-11T08:11:24.490630+0000","last_active":"2024-01-11T08:11:24.496470+0000","last_peered":"2024-01-11T08:11:24.496470+0000","last_clean":"2024-01-11T08:11:24.496470+0000","last_became_active":"2024-01-11T08:11:24.490189+0000","last_became_peered":"2024-01-11T08:11:24.490189+0000","last_unstale":"2024-01-11T08:11:24.496470+0000","last_undegraded":"2024-01-11T08:11:24.496470+0000","last_fullsized":"2024-01-11T08:11:24.496470+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:00:16.213369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.506166+0000","last_change":"2024-01-11T08:11:24.504739+0000","last_active":"2024-01-11T08:11:24.506166+0000","last_peered":"2024-01-11T08:11:24.506166+0000","last_clean":"2024-01-11T08:11:24.506166+0000","last_became_active":"2024-01-11T08:11:24.503916+0000","last_became_peered":"2024-01-11T08:11:24.503916+0000","last_unstale":"2024-01-11T08:11:24.506166+0000","last_undegraded":"2024-01-11T08:11:24.506166+0000","last_fullsized":"2024-01-11T08:11:24.506166+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:43:30.233475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.499955+0000","last_change":"2024-01-11T08:11:24.492699+0000","last_active":"2024-01-11T08:11:24.499955+0000","last_peered":"2024-01-11T08:11:24.499955+0000","last_clean":"2024-01-11T08:11:24.499955+0000","last_became_active":"2024-01-11T08:11:24.492443+0000","last_became_peered":"2024-01-11T08:11:24.492443+0000","last_unstale":"2024-01-11T08:11:24.499955+0000","last_undegraded":"2024-01-11T08:11:24.499955+0000","last_fullsized":"2024-01-11T08:11:24.499955+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:25:29.239749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.502618+0000","last_change":"2024-01-11T08:11:24.495229+0000","last_active":"2024-01-11T08:11:24.502618+0000","last_peered":"2024-01-11T08:11:24.502618+0000","last_clean":"2024-01-11T08:11:24.502618+0000","last_became_active":"2024-01-11T08:11:24.491426+0000","last_became_peered":"2024-01-11T08:11:24.491426+0000","last_unstale":"2024-01-11T08:11:24.502618+0000","last_undegraded":"2024-01-11T08:11:24.502618+0000","last_fullsized":"2024-01-11T08:11:24.502618+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:00:22.406184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.506108+0000","last_change":"2024-01-11T08:11:24.496651+0000","last_active":"2024-01-11T08:11:24.506108+0000","last_peered":"2024-01-11T08:11:24.506108+0000","last_clean":"2024-01-11T08:11:24.506108+0000","last_became_active":"2024-01-11T08:11:24.495761+0000","last_became_peered":"2024-01-11T08:11:24.495761+0000","last_unstale":"2024-01-11T08:11:24.506108+0000","last_undegraded":"2024-01-11T08:11:24.506108+0000","last_fullsized":"2024-01-11T08:11:24.506108+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:28:27.237960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.505725+0000","last_change":"2024-01-11T08:11:24.496526+0000","last_active":"2024-01-11T08:11:24.505725+0000","last_peered":"2024-01-11T08:11:24.505725+0000","last_clean":"2024-01-11T08:11:24.505725+0000","last_became_active":"2024-01-11T08:11:24.495927+0000","last_became_peered":"2024-01-11T08:11:24.495927+0000","last_unstale":"2024-01-11T08:11:24.505725+0000","last_undegraded":"2024-01-11T08:11:24.505725+0000","last_fullsized":"2024-01-11T08:11:24.505725+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:46:59.139202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.507640+0000","last_change":"2024-01-11T08:11:24.504256+0000","last_active":"2024-01-11T08:11:24.507640+0000","last_peered":"2024-01-11T08:11:24.507640+0000","last_clean":"2024-01-11T08:11:24.507640+0000","last_became_active":"2024-01-11T08:11:24.503892+0000","last_became_peered":"2024-01-11T08:11:24.503892+0000","last_unstale":"2024-01-11T08:11:24.507640+0000","last_undegraded":"2024-01-11T08:11:24.507640+0000","last_fullsized":"2024-01-11T08:11:24.507640+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:09:21.568163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.492599+0000","last_change":"2024-01-11T08:11:24.489768+0000","last_active":"2024-01-11T08:11:24.492599+0000","last_peered":"2024-01-11T08:11:24.492599+0000","last_clean":"2024-01-11T08:11:24.492599+0000","last_became_active":"2024-01-11T08:11:24.489111+0000","last_became_peered":"2024-01-11T08:11:24.489111+0000","last_unstale":"2024-01-11T08:11:24.492599+0000","last_undegraded":"2024-01-11T08:11:24.492599+0000","last_fullsized":"2024-01-11T08:11:24.492599+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:52:10.203426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.933233+0000","last_change":"2024-01-11T08:11:23.943112+0000","last_active":"2024-01-11T08:11:24.933233+0000","last_peered":"2024-01-11T08:11:24.933233+0000","last_clean":"2024-01-11T08:11:24.933233+0000","last_became_active":"2024-01-11T08:11:23.941605+0000","last_became_peered":"2024-01-11T08:11:23.941605+0000","last_unstale":"2024-01-11T08:11:24.933233+0000","last_undegraded":"2024-01-11T08:11:24.933233+0000","last_fullsized":"2024-01-11T08:11:24.933233+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:40:50.968620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.506238+0000","last_change":"2024-01-11T08:11:24.503057+0000","last_active":"2024-01-11T08:11:24.506238+0000","last_peered":"2024-01-11T08:11:24.506238+0000","last_clean":"2024-01-11T08:11:24.506238+0000","last_became_active":"2024-01-11T08:11:24.502472+0000","last_became_peered":"2024-01-11T08:11:24.502472+0000","last_unstale":"2024-01-11T08:11:24.506238+0000","last_undegraded":"2024-01-11T08:11:24.506238+0000","last_fullsized":"2024-01-11T08:11:24.506238+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:52:10.203426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.935709+0000","last_change":"2024-01-11T08:11:23.947408+0000","last_active":"2024-01-11T08:11:24.935709+0000","last_peered":"2024-01-11T08:11:24.935709+0000","last_clean":"2024-01-11T08:11:24.935709+0000","last_became_active":"2024-01-11T08:11:23.946995+0000","last_became_peered":"2024-01-11T08:11:23.946995+0000","last_unstale":"2024-01-11T08:11:24.935709+0000","last_undegraded":"2024-01-11T08:11:24.935709+0000","last_fullsized":"2024-01-11T08:11:24.935709+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:40:50.968620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.501284+0000","last_change":"2024-01-11T08:11:24.491643+0000","last_active":"2024-01-11T08:11:24.501284+0000","last_peered":"2024-01-11T08:11:24.501284+0000","last_clean":"2024-01-11T08:11:24.501284+0000","last_became_active":"2024-01-11T08:11:24.491283+0000","last_became_peered":"2024-01-11T08:11:24.491283+0000","last_unstale":"2024-01-11T08:11:24.501284+0000","last_undegraded":"2024-01-11T08:11:24.501284+0000","last_fullsized":"2024-01-11T08:11:24.501284+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:01:16.202292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.936380+0000","last_change":"2024-01-11T08:11:23.943473+0000","last_active":"2024-01-11T08:11:24.936380+0000","last_peered":"2024-01-11T08:11:24.936380+0000","last_clean":"2024-01-11T08:11:24.936380+0000","last_became_active":"2024-01-11T08:11:23.943162+0000","last_became_peered":"2024-01-11T08:11:23.943162+0000","last_unstale":"2024-01-11T08:11:24.936380+0000","last_undegraded":"2024-01-11T08:11:24.936380+0000","last_fullsized":"2024-01-11T08:11:24.936380+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:40:53.546449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.507857+0000","last_change":"2024-01-11T08:11:24.506459+0000","last_active":"2024-01-11T08:11:24.507857+0000","last_peered":"2024-01-11T08:11:24.507857+0000","last_clean":"2024-01-11T08:11:24.507857+0000","last_became_active":"2024-01-11T08:11:24.506211+0000","last_became_peered":"2024-01-11T08:11:24.506211+0000","last_unstale":"2024-01-11T08:11:24.507857+0000","last_undegraded":"2024-01-11T08:11:24.507857+0000","last_fullsized":"2024-01-11T08:11:24.507857+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:49:27.059571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.938895+0000","last_change":"2024-01-11T08:11:23.951138+0000","last_active":"2024-01-11T08:11:24.938895+0000","last_peered":"2024-01-11T08:11:24.938895+0000","last_clean":"2024-01-11T08:11:24.938895+0000","last_became_active":"2024-01-11T08:11:23.950807+0000","last_became_peered":"2024-01-11T08:11:23.950807+0000","last_unstale":"2024-01-11T08:11:24.938895+0000","last_undegraded":"2024-01-11T08:11:24.938895+0000","last_fullsized":"2024-01-11T08:11:24.938895+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:20:07.483752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.507595+0000","last_change":"2024-01-11T08:11:24.502127+0000","last_active":"2024-01-11T08:11:24.507595+0000","last_peered":"2024-01-11T08:11:24.507595+0000","last_clean":"2024-01-11T08:11:24.507595+0000","last_became_active":"2024-01-11T08:11:24.501836+0000","last_became_peered":"2024-01-11T08:11:24.501836+0000","last_unstale":"2024-01-11T08:11:24.507595+0000","last_undegraded":"2024-01-11T08:11:24.507595+0000","last_fullsized":"2024-01-11T08:11:24.507595+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:42:29.637708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.940651+0000","last_change":"2024-01-11T08:11:23.939004+0000","last_active":"2024-01-11T08:11:24.940651+0000","last_peered":"2024-01-11T08:11:24.940651+0000","last_clean":"2024-01-11T08:11:24.940651+0000","last_became_active":"2024-01-11T08:11:23.938109+0000","last_became_peered":"2024-01-11T08:11:23.938109+0000","last_unstale":"2024-01-11T08:11:24.940651+0000","last_undegraded":"2024-01-11T08:11:24.940651+0000","last_fullsized":"2024-01-11T08:11:24.940651+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:10:53.026857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.500277+0000","last_change":"2024-01-11T08:11:24.494245+0000","last_active":"2024-01-11T08:11:24.500277+0000","last_peered":"2024-01-11T08:11:24.500277+0000","last_clean":"2024-01-11T08:11:24.500277+0000","last_became_active":"2024-01-11T08:11:24.493941+0000","last_became_peered":"2024-01-11T08:11:24.493941+0000","last_unstale":"2024-01-11T08:11:24.500277+0000","last_undegraded":"2024-01-11T08:11:24.500277+0000","last_fullsized":"2024-01-11T08:11:24.500277+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:00:16.213369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.938288+0000","last_change":"2024-01-11T08:11:23.942772+0000","last_active":"2024-01-11T08:11:24.938288+0000","last_peered":"2024-01-11T08:11:24.938288+0000","last_clean":"2024-01-11T08:11:24.938288+0000","last_became_active":"2024-01-11T08:11:23.942436+0000","last_became_peered":"2024-01-11T08:11:23.942436+0000","last_unstale":"2024-01-11T08:11:24.938288+0000","last_undegraded":"2024-01-11T08:11:24.938288+0000","last_fullsized":"2024-01-11T08:11:24.938288+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:33:11.028049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.502805+0000","last_change":"2024-01-11T08:11:24.498406+0000","last_active":"2024-01-11T08:11:24.502805+0000","last_peered":"2024-01-11T08:11:24.502805+0000","last_clean":"2024-01-11T08:11:24.502805+0000","last_became_active":"2024-01-11T08:11:24.498120+0000","last_became_peered":"2024-01-11T08:11:24.498120+0000","last_unstale":"2024-01-11T08:11:24.502805+0000","last_undegraded":"2024-01-11T08:11:24.502805+0000","last_fullsized":"2024-01-11T08:11:24.502805+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:46:59.139202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.478296+0000","last_change":"2024-01-11T08:11:23.939190+0000","last_active":"2024-01-11T08:11:24.478296+0000","last_peered":"2024-01-11T08:11:24.478296+0000","last_clean":"2024-01-11T08:11:24.478296+0000","last_became_active":"2024-01-11T08:11:23.938313+0000","last_became_peered":"2024-01-11T08:11:23.938313+0000","last_unstale":"2024-01-11T08:11:24.478296+0000","last_undegraded":"2024-01-11T08:11:24.478296+0000","last_fullsized":"2024-01-11T08:11:24.478296+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:30:52.899871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.505545+0000","last_change":"2024-01-11T08:11:24.501530+0000","last_active":"2024-01-11T08:11:24.505545+0000","last_peered":"2024-01-11T08:11:24.505545+0000","last_clean":"2024-01-11T08:11:24.505545+0000","last_became_active":"2024-01-11T08:11:24.501322+0000","last_became_peered":"2024-01-11T08:11:24.501322+0000","last_unstale":"2024-01-11T08:11:24.505545+0000","last_undegraded":"2024-01-11T08:11:24.505545+0000","last_fullsized":"2024-01-11T08:11:24.505545+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:01:16.202292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.479497+0000","last_change":"2024-01-11T08:11:23.938478+0000","last_active":"2024-01-11T08:11:24.479497+0000","last_peered":"2024-01-11T08:11:24.479497+0000","last_clean":"2024-01-11T08:11:24.479497+0000","last_became_active":"2024-01-11T08:11:23.938117+0000","last_became_peered":"2024-01-11T08:11:23.938117+0000","last_unstale":"2024-01-11T08:11:24.479497+0000","last_undegraded":"2024-01-11T08:11:24.479497+0000","last_fullsized":"2024-01-11T08:11:24.479497+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:09:52.585500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.499265+0000","last_change":"2024-01-11T08:11:24.489927+0000","last_active":"2024-01-11T08:11:24.499265+0000","last_peered":"2024-01-11T08:11:24.499265+0000","last_clean":"2024-01-11T08:11:24.499265+0000","last_became_active":"2024-01-11T08:11:24.486819+0000","last_became_peered":"2024-01-11T08:11:24.486819+0000","last_unstale":"2024-01-11T08:11:24.499265+0000","last_undegraded":"2024-01-11T08:11:24.499265+0000","last_fullsized":"2024-01-11T08:11:24.499265+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:52:14.812398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.478449+0000","last_change":"2024-01-11T08:11:23.950109+0000","last_active":"2024-01-11T08:11:24.478449+0000","last_peered":"2024-01-11T08:11:24.478449+0000","last_clean":"2024-01-11T08:11:24.478449+0000","last_became_active":"2024-01-11T08:11:23.947241+0000","last_became_peered":"2024-01-11T08:11:23.947241+0000","last_unstale":"2024-01-11T08:11:24.478449+0000","last_undegraded":"2024-01-11T08:11:24.478449+0000","last_fullsized":"2024-01-11T08:11:24.478449+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:26:59.712317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.501943+0000","last_change":"2024-01-11T08:11:24.500129+0000","last_active":"2024-01-11T08:11:24.501943+0000","last_peered":"2024-01-11T08:11:24.501943+0000","last_clean":"2024-01-11T08:11:24.501943+0000","last_became_active":"2024-01-11T08:11:24.499815+0000","last_became_peered":"2024-01-11T08:11:24.499815+0000","last_unstale":"2024-01-11T08:11:24.501943+0000","last_undegraded":"2024-01-11T08:11:24.501943+0000","last_fullsized":"2024-01-11T08:11:24.501943+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:56:43.124008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.489945+0000","last_change":"2024-01-11T08:11:23.953673+0000","last_active":"2024-01-11T08:11:24.489945+0000","last_peered":"2024-01-11T08:11:24.489945+0000","last_clean":"2024-01-11T08:11:24.489945+0000","last_became_active":"2024-01-11T08:11:23.953330+0000","last_became_peered":"2024-01-11T08:11:23.953330+0000","last_unstale":"2024-01-11T08:11:24.489945+0000","last_undegraded":"2024-01-11T08:11:24.489945+0000","last_fullsized":"2024-01-11T08:11:24.489945+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:15:06.711507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.495425+0000","last_change":"2024-01-11T08:11:24.490492+0000","last_active":"2024-01-11T08:11:24.495425+0000","last_peered":"2024-01-11T08:11:24.495425+0000","last_clean":"2024-01-11T08:11:24.495425+0000","last_became_active":"2024-01-11T08:11:24.490201+0000","last_became_peered":"2024-01-11T08:11:24.490201+0000","last_unstale":"2024-01-11T08:11:24.495425+0000","last_undegraded":"2024-01-11T08:11:24.495425+0000","last_fullsized":"2024-01-11T08:11:24.495425+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:28:27.237960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.937659+0000","last_change":"2024-01-11T08:11:23.948669+0000","last_active":"2024-01-11T08:11:24.937659+0000","last_peered":"2024-01-11T08:11:24.937659+0000","last_clean":"2024-01-11T08:11:24.937659+0000","last_became_active":"2024-01-11T08:11:23.948361+0000","last_became_peered":"2024-01-11T08:11:23.948361+0000","last_unstale":"2024-01-11T08:11:24.937659+0000","last_undegraded":"2024-01-11T08:11:24.937659+0000","last_fullsized":"2024-01-11T08:11:24.937659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:55:31.548418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.506593+0000","last_change":"2024-01-11T08:11:24.500951+0000","last_active":"2024-01-11T08:11:24.506593+0000","last_peered":"2024-01-11T08:11:24.506593+0000","last_clean":"2024-01-11T08:11:24.506593+0000","last_became_active":"2024-01-11T08:11:24.500685+0000","last_became_peered":"2024-01-11T08:11:24.500685+0000","last_unstale":"2024-01-11T08:11:24.506593+0000","last_undegraded":"2024-01-11T08:11:24.506593+0000","last_fullsized":"2024-01-11T08:11:24.506593+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:06:14.717197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.946350+0000","last_change":"2024-01-11T08:11:23.945990+0000","last_active":"2024-01-11T08:11:24.946350+0000","last_peered":"2024-01-11T08:11:24.946350+0000","last_clean":"2024-01-11T08:11:24.946350+0000","last_became_active":"2024-01-11T08:11:23.945645+0000","last_became_peered":"2024-01-11T08:11:23.945645+0000","last_unstale":"2024-01-11T08:11:24.946350+0000","last_undegraded":"2024-01-11T08:11:24.946350+0000","last_fullsized":"2024-01-11T08:11:24.946350+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:15:06.711507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.503474+0000","last_change":"2024-01-11T08:11:24.496482+0000","last_active":"2024-01-11T08:11:24.503474+0000","last_peered":"2024-01-11T08:11:24.503474+0000","last_clean":"2024-01-11T08:11:24.503474+0000","last_became_active":"2024-01-11T08:11:24.495898+0000","last_became_peered":"2024-01-11T08:11:24.495898+0000","last_unstale":"2024-01-11T08:11:24.503474+0000","last_undegraded":"2024-01-11T08:11:24.503474+0000","last_fullsized":"2024-01-11T08:11:24.503474+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:00:22.406184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.940509+0000","last_change":"2024-01-11T08:11:23.943985+0000","last_active":"2024-01-11T08:11:24.940509+0000","last_peered":"2024-01-11T08:11:24.940509+0000","last_clean":"2024-01-11T08:11:24.940509+0000","last_became_active":"2024-01-11T08:11:23.943428+0000","last_became_peered":"2024-01-11T08:11:23.943428+0000","last_unstale":"2024-01-11T08:11:24.940509+0000","last_undegraded":"2024-01-11T08:11:24.940509+0000","last_fullsized":"2024-01-11T08:11:24.940509+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:33:35.425614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.500891+0000","last_change":"2024-01-11T08:11:24.493534+0000","last_active":"2024-01-11T08:11:24.500891+0000","last_peered":"2024-01-11T08:11:24.500891+0000","last_clean":"2024-01-11T08:11:24.500891+0000","last_became_active":"2024-01-11T08:11:24.493261+0000","last_became_peered":"2024-01-11T08:11:24.493261+0000","last_unstale":"2024-01-11T08:11:24.500891+0000","last_undegraded":"2024-01-11T08:11:24.500891+0000","last_fullsized":"2024-01-11T08:11:24.500891+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:08:51.690170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.483955+0000","last_change":"2024-01-11T08:11:23.941252+0000","last_active":"2024-01-11T08:11:24.483955+0000","last_peered":"2024-01-11T08:11:24.483955+0000","last_clean":"2024-01-11T08:11:24.483955+0000","last_became_active":"2024-01-11T08:11:23.939437+0000","last_became_peered":"2024-01-11T08:11:23.939437+0000","last_unstale":"2024-01-11T08:11:24.483955+0000","last_undegraded":"2024-01-11T08:11:24.483955+0000","last_fullsized":"2024-01-11T08:11:24.483955+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:15:06.711507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.507607+0000","last_change":"2024-01-11T08:11:24.503922+0000","last_active":"2024-01-11T08:11:24.507607+0000","last_peered":"2024-01-11T08:11:24.507607+0000","last_clean":"2024-01-11T08:11:24.507607+0000","last_became_active":"2024-01-11T08:11:24.503448+0000","last_became_peered":"2024-01-11T08:11:24.503448+0000","last_unstale":"2024-01-11T08:11:24.507607+0000","last_undegraded":"2024-01-11T08:11:24.507607+0000","last_fullsized":"2024-01-11T08:11:24.507607+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:58:45.894019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.478384+0000","last_change":"2024-01-11T08:11:23.948996+0000","last_active":"2024-01-11T08:11:24.478384+0000","last_peered":"2024-01-11T08:11:24.478384+0000","last_clean":"2024-01-11T08:11:24.478384+0000","last_became_active":"2024-01-11T08:11:23.948714+0000","last_became_peered":"2024-01-11T08:11:23.948714+0000","last_unstale":"2024-01-11T08:11:24.478384+0000","last_undegraded":"2024-01-11T08:11:24.478384+0000","last_fullsized":"2024-01-11T08:11:24.478384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:28:29.782931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.505743+0000","last_change":"2024-01-11T08:11:24.501548+0000","last_active":"2024-01-11T08:11:24.505743+0000","last_peered":"2024-01-11T08:11:24.505743+0000","last_clean":"2024-01-11T08:11:24.505743+0000","last_became_active":"2024-01-11T08:11:24.501265+0000","last_became_peered":"2024-01-11T08:11:24.501265+0000","last_unstale":"2024-01-11T08:11:24.505743+0000","last_undegraded":"2024-01-11T08:11:24.505743+0000","last_fullsized":"2024-01-11T08:11:24.505743+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:30:53.909322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.940874+0000","last_change":"2024-01-11T08:11:23.945536+0000","last_active":"2024-01-11T08:11:24.940874+0000","last_peered":"2024-01-11T08:11:24.940874+0000","last_clean":"2024-01-11T08:11:24.940874+0000","last_became_active":"2024-01-11T08:11:23.945303+0000","last_became_peered":"2024-01-11T08:11:23.945303+0000","last_unstale":"2024-01-11T08:11:24.940874+0000","last_undegraded":"2024-01-11T08:11:24.940874+0000","last_fullsized":"2024-01-11T08:11:24.940874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:50:20.362519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.491822+0000","last_change":"2024-01-11T08:11:24.485777+0000","last_active":"2024-01-11T08:11:24.491822+0000","last_peered":"2024-01-11T08:11:24.491822+0000","last_clean":"2024-01-11T08:11:24.491822+0000","last_became_active":"2024-01-11T08:11:24.485460+0000","last_became_peered":"2024-01-11T08:11:24.485460+0000","last_unstale":"2024-01-11T08:11:24.491822+0000","last_undegraded":"2024-01-11T08:11:24.491822+0000","last_fullsized":"2024-01-11T08:11:24.491822+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:52:10.203426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.938508+0000","last_change":"2024-01-11T08:11:23.946612+0000","last_active":"2024-01-11T08:11:24.938508+0000","last_peered":"2024-01-11T08:11:24.938508+0000","last_clean":"2024-01-11T08:11:24.938508+0000","last_became_active":"2024-01-11T08:11:23.946302+0000","last_became_peered":"2024-01-11T08:11:23.946302+0000","last_unstale":"2024-01-11T08:11:24.938508+0000","last_undegraded":"2024-01-11T08:11:24.938508+0000","last_fullsized":"2024-01-11T08:11:24.938508+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:40:50.968620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.495584+0000","last_change":"2024-01-11T08:11:24.487755+0000","last_active":"2024-01-11T08:11:24.495584+0000","last_peered":"2024-01-11T08:11:24.495584+0000","last_clean":"2024-01-11T08:11:24.495584+0000","last_became_active":"2024-01-11T08:11:24.486857+0000","last_became_peered":"2024-01-11T08:11:24.486857+0000","last_unstale":"2024-01-11T08:11:24.495584+0000","last_undegraded":"2024-01-11T08:11:24.495584+0000","last_fullsized":"2024-01-11T08:11:24.495584+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:28:27.237960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.937972+0000","last_change":"2024-01-11T08:11:23.940732+0000","last_active":"2024-01-11T08:11:24.937972+0000","last_peered":"2024-01-11T08:11:24.937972+0000","last_clean":"2024-01-11T08:11:24.937972+0000","last_became_active":"2024-01-11T08:11:23.940429+0000","last_became_peered":"2024-01-11T08:11:23.940429+0000","last_unstale":"2024-01-11T08:11:24.937972+0000","last_undegraded":"2024-01-11T08:11:24.937972+0000","last_fullsized":"2024-01-11T08:11:24.937972+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:17.235033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.505476+0000","last_change":"2024-01-11T08:11:24.501338+0000","last_active":"2024-01-11T08:11:24.505476+0000","last_peered":"2024-01-11T08:11:24.505476+0000","last_clean":"2024-01-11T08:11:24.505476+0000","last_became_active":"2024-01-11T08:11:24.501132+0000","last_became_peered":"2024-01-11T08:11:24.501132+0000","last_unstale":"2024-01-11T08:11:24.505476+0000","last_undegraded":"2024-01-11T08:11:24.505476+0000","last_fullsized":"2024-01-11T08:11:24.505476+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:43:46.629601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.938268+0000","last_change":"2024-01-11T08:11:23.943261+0000","last_active":"2024-01-11T08:11:24.938268+0000","last_peered":"2024-01-11T08:11:24.938268+0000","last_clean":"2024-01-11T08:11:24.938268+0000","last_became_active":"2024-01-11T08:11:23.942934+0000","last_became_peered":"2024-01-11T08:11:23.942934+0000","last_unstale":"2024-01-11T08:11:24.938268+0000","last_undegraded":"2024-01-11T08:11:24.938268+0000","last_fullsized":"2024-01-11T08:11:24.938268+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:49:50.495213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.506046+0000","last_change":"2024-01-11T08:11:24.503916+0000","last_active":"2024-01-11T08:11:24.506046+0000","last_peered":"2024-01-11T08:11:24.506046+0000","last_clean":"2024-01-11T08:11:24.506046+0000","last_became_active":"2024-01-11T08:11:24.503613+0000","last_became_peered":"2024-01-11T08:11:24.503613+0000","last_unstale":"2024-01-11T08:11:24.506046+0000","last_undegraded":"2024-01-11T08:11:24.506046+0000","last_fullsized":"2024-01-11T08:11:24.506046+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:55:32.557869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.480550+0000","last_change":"2024-01-11T08:11:23.940794+0000","last_active":"2024-01-11T08:11:24.480550+0000","last_peered":"2024-01-11T08:11:24.480550+0000","last_clean":"2024-01-11T08:11:24.480550+0000","last_became_active":"2024-01-11T08:11:23.940412+0000","last_became_peered":"2024-01-11T08:11:23.940412+0000","last_unstale":"2024-01-11T08:11:24.480550+0000","last_undegraded":"2024-01-11T08:11:24.480550+0000","last_fullsized":"2024-01-11T08:11:24.480550+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:22:49.208962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.497136+0000","last_change":"2024-01-11T08:11:24.493812+0000","last_active":"2024-01-11T08:11:24.497136+0000","last_peered":"2024-01-11T08:11:24.497136+0000","last_clean":"2024-01-11T08:11:24.497136+0000","last_became_active":"2024-01-11T08:11:24.492943+0000","last_became_peered":"2024-01-11T08:11:24.492943+0000","last_unstale":"2024-01-11T08:11:24.497136+0000","last_undegraded":"2024-01-11T08:11:24.497136+0000","last_fullsized":"2024-01-11T08:11:24.497136+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:18.244484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.936318+0000","last_change":"2024-01-11T08:11:23.942355+0000","last_active":"2024-01-11T08:11:24.936318+0000","last_peered":"2024-01-11T08:11:24.936318+0000","last_clean":"2024-01-11T08:11:24.936318+0000","last_became_active":"2024-01-11T08:11:23.942053+0000","last_became_peered":"2024-01-11T08:11:23.942053+0000","last_unstale":"2024-01-11T08:11:24.936318+0000","last_undegraded":"2024-01-11T08:11:24.936318+0000","last_fullsized":"2024-01-11T08:11:24.936318+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:09:20.134819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.496875+0000","last_change":"2024-01-11T08:11:24.494115+0000","last_active":"2024-01-11T08:11:24.496875+0000","last_peered":"2024-01-11T08:11:24.496875+0000","last_clean":"2024-01-11T08:11:24.496875+0000","last_became_active":"2024-01-11T08:11:24.493549+0000","last_became_peered":"2024-01-11T08:11:24.493549+0000","last_unstale":"2024-01-11T08:11:24.496875+0000","last_undegraded":"2024-01-11T08:11:24.496875+0000","last_fullsized":"2024-01-11T08:11:24.496875+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:09:21.144270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.485581+0000","last_change":"2024-01-11T08:11:23.950092+0000","last_active":"2024-01-11T08:11:24.485581+0000","last_peered":"2024-01-11T08:11:24.485581+0000","last_clean":"2024-01-11T08:11:24.485581+0000","last_became_active":"2024-01-11T08:11:23.949716+0000","last_became_peered":"2024-01-11T08:11:23.949716+0000","last_unstale":"2024-01-11T08:11:24.485581+0000","last_undegraded":"2024-01-11T08:11:24.485581+0000","last_fullsized":"2024-01-11T08:11:24.485581+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:10:53.026857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.502863+0000","last_change":"2024-01-11T08:11:24.495593+0000","last_active":"2024-01-11T08:11:24.502863+0000","last_peered":"2024-01-11T08:11:24.502863+0000","last_clean":"2024-01-11T08:11:24.502863+0000","last_became_active":"2024-01-11T08:11:24.495334+0000","last_became_peered":"2024-01-11T08:11:24.495334+0000","last_unstale":"2024-01-11T08:11:24.502863+0000","last_undegraded":"2024-01-11T08:11:24.502863+0000","last_fullsized":"2024-01-11T08:11:24.502863+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:10:54.036308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.480623+0000","last_change":"2024-01-11T08:11:23.948216+0000","last_active":"2024-01-11T08:11:24.480623+0000","last_peered":"2024-01-11T08:11:24.480623+0000","last_clean":"2024-01-11T08:11:24.480623+0000","last_became_active":"2024-01-11T08:11:23.947925+0000","last_became_peered":"2024-01-11T08:11:23.947925+0000","last_unstale":"2024-01-11T08:11:24.480623+0000","last_undegraded":"2024-01-11T08:11:24.480623+0000","last_fullsized":"2024-01-11T08:11:24.480623+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:53:33.242677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.506090+0000","last_change":"2024-01-11T08:11:24.497065+0000","last_active":"2024-01-11T08:11:24.506090+0000","last_peered":"2024-01-11T08:11:24.506090+0000","last_clean":"2024-01-11T08:11:24.506090+0000","last_became_active":"2024-01-11T08:11:24.496780+0000","last_became_peered":"2024-01-11T08:11:24.496780+0000","last_unstale":"2024-01-11T08:11:24.506090+0000","last_undegraded":"2024-01-11T08:11:24.506090+0000","last_fullsized":"2024-01-11T08:11:24.506090+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:04:02.047325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.479809+0000","last_change":"2024-01-11T08:11:23.949068+0000","last_active":"2024-01-11T08:11:24.479809+0000","last_peered":"2024-01-11T08:11:24.479809+0000","last_clean":"2024-01-11T08:11:24.479809+0000","last_became_active":"2024-01-11T08:11:23.948491+0000","last_became_peered":"2024-01-11T08:11:23.948491+0000","last_unstale":"2024-01-11T08:11:24.479809+0000","last_undegraded":"2024-01-11T08:11:24.479809+0000","last_fullsized":"2024-01-11T08:11:24.479809+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:50:20.362519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.505998+0000","last_change":"2024-01-11T08:11:24.503170+0000","last_active":"2024-01-11T08:11:24.505998+0000","last_peered":"2024-01-11T08:11:24.505998+0000","last_clean":"2024-01-11T08:11:24.505998+0000","last_became_active":"2024-01-11T08:11:24.502892+0000","last_became_peered":"2024-01-11T08:11:24.502892+0000","last_unstale":"2024-01-11T08:11:24.505998+0000","last_undegraded":"2024-01-11T08:11:24.505998+0000","last_fullsized":"2024-01-11T08:11:24.505998+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:12:42.979106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.934051+0000","last_change":"2024-01-11T08:11:23.940407+0000","last_active":"2024-01-11T08:11:24.934051+0000","last_peered":"2024-01-11T08:11:24.934051+0000","last_clean":"2024-01-11T08:11:24.934051+0000","last_became_active":"2024-01-11T08:11:23.940061+0000","last_became_peered":"2024-01-11T08:11:23.940061+0000","last_unstale":"2024-01-11T08:11:24.934051+0000","last_undegraded":"2024-01-11T08:11:24.934051+0000","last_fullsized":"2024-01-11T08:11:24.934051+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:09:20.134819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.506434+0000","last_change":"2024-01-11T08:11:24.500597+0000","last_active":"2024-01-11T08:11:24.506434+0000","last_peered":"2024-01-11T08:11:24.506434+0000","last_clean":"2024-01-11T08:11:24.506434+0000","last_became_active":"2024-01-11T08:11:24.500311+0000","last_became_peered":"2024-01-11T08:11:24.500311+0000","last_unstale":"2024-01-11T08:11:24.506434+0000","last_undegraded":"2024-01-11T08:11:24.506434+0000","last_fullsized":"2024-01-11T08:11:24.506434+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:42:29.637708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"34'1","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-11T08:11:25.984725+0000","last_change":"2024-01-11T08:11:23.949154+0000","last_active":"2024-01-11T08:11:25.984725+0000","last_peered":"2024-01-11T08:11:25.984725+0000","last_clean":"2024-01-11T08:11:25.984725+0000","last_became_active":"2024-01-11T08:11:23.948780+0000","last_became_peered":"2024-01-11T08:11:23.948780+0000","last_unstale":"2024-01-11T08:11:25.984725+0000","last_undegraded":"2024-01-11T08:11:25.984725+0000","last_fullsized":"2024-01-11T08:11:25.984725+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:12:41.969655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.501788+0000","last_change":"2024-01-11T08:11:24.499073+0000","last_active":"2024-01-11T08:11:24.501788+0000","last_peered":"2024-01-11T08:11:24.501788+0000","last_clean":"2024-01-11T08:11:24.501788+0000","last_became_active":"2024-01-11T08:11:24.498800+0000","last_became_peered":"2024-01-11T08:11:24.498800+0000","last_unstale":"2024-01-11T08:11:24.501788+0000","last_undegraded":"2024-01-11T08:11:24.501788+0000","last_fullsized":"2024-01-11T08:11:24.501788+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:27:00.721768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.486043+0000","last_change":"2024-01-11T08:11:23.942071+0000","last_active":"2024-01-11T08:11:24.486043+0000","last_peered":"2024-01-11T08:11:24.486043+0000","last_clean":"2024-01-11T08:11:24.486043+0000","last_became_active":"2024-01-11T08:11:23.941698+0000","last_became_peered":"2024-01-11T08:11:23.941698+0000","last_unstale":"2024-01-11T08:11:24.486043+0000","last_undegraded":"2024-01-11T08:11:24.486043+0000","last_fullsized":"2024-01-11T08:11:24.486043+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:12:41.969655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.502211+0000","last_change":"2024-01-11T08:11:24.493924+0000","last_active":"2024-01-11T08:11:24.502211+0000","last_peered":"2024-01-11T08:11:24.502211+0000","last_clean":"2024-01-11T08:11:24.502211+0000","last_became_active":"2024-01-11T08:11:24.493411+0000","last_became_peered":"2024-01-11T08:11:24.493411+0000","last_unstale":"2024-01-11T08:11:24.502211+0000","last_undegraded":"2024-01-11T08:11:24.502211+0000","last_fullsized":"2024-01-11T08:11:24.502211+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:40:54.555900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.489610+0000","last_change":"2024-01-11T08:11:23.940144+0000","last_active":"2024-01-11T08:11:24.489610+0000","last_peered":"2024-01-11T08:11:24.489610+0000","last_clean":"2024-01-11T08:11:24.489610+0000","last_became_active":"2024-01-11T08:11:23.939701+0000","last_became_peered":"2024-01-11T08:11:23.939701+0000","last_unstale":"2024-01-11T08:11:24.489610+0000","last_undegraded":"2024-01-11T08:11:24.489610+0000","last_fullsized":"2024-01-11T08:11:24.489610+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:10:53.026857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.497569+0000","last_change":"2024-01-11T08:11:24.493706+0000","last_active":"2024-01-11T08:11:24.497569+0000","last_peered":"2024-01-11T08:11:24.497569+0000","last_clean":"2024-01-11T08:11:24.497569+0000","last_became_active":"2024-01-11T08:11:24.492676+0000","last_became_peered":"2024-01-11T08:11:24.492676+0000","last_unstale":"2024-01-11T08:11:24.497569+0000","last_undegraded":"2024-01-11T08:11:24.497569+0000","last_fullsized":"2024-01-11T08:11:24.497569+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:28:30.792382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.483853+0000","last_change":"2024-01-11T08:11:23.936600+0000","last_active":"2024-01-11T08:11:24.483853+0000","last_peered":"2024-01-11T08:11:24.483853+0000","last_clean":"2024-01-11T08:11:24.483853+0000","last_became_active":"2024-01-11T08:11:23.936226+0000","last_became_peered":"2024-01-11T08:11:23.936226+0000","last_unstale":"2024-01-11T08:11:24.483853+0000","last_undegraded":"2024-01-11T08:11:24.483853+0000","last_fullsized":"2024-01-11T08:11:24.483853+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:10:53.026857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.501840+0000","last_change":"2024-01-11T08:11:24.496865+0000","last_active":"2024-01-11T08:11:24.501840+0000","last_peered":"2024-01-11T08:11:24.501840+0000","last_clean":"2024-01-11T08:11:24.501840+0000","last_became_active":"2024-01-11T08:11:24.496580+0000","last_became_peered":"2024-01-11T08:11:24.496580+0000","last_unstale":"2024-01-11T08:11:24.501840+0000","last_undegraded":"2024-01-11T08:11:24.501840+0000","last_fullsized":"2024-01-11T08:11:24.501840+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:58:45.894019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.480727+0000","last_change":"2024-01-11T08:11:23.936870+0000","last_active":"2024-01-11T08:11:24.480727+0000","last_peered":"2024-01-11T08:11:24.480727+0000","last_clean":"2024-01-11T08:11:24.480727+0000","last_became_active":"2024-01-11T08:11:23.935874+0000","last_became_peered":"2024-01-11T08:11:23.935874+0000","last_unstale":"2024-01-11T08:11:24.480727+0000","last_undegraded":"2024-01-11T08:11:24.480727+0000","last_fullsized":"2024-01-11T08:11:24.480727+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:29:09.480491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.501839+0000","last_change":"2024-01-11T08:11:24.492858+0000","last_active":"2024-01-11T08:11:24.501839+0000","last_peered":"2024-01-11T08:11:24.501839+0000","last_clean":"2024-01-11T08:11:24.501839+0000","last_became_active":"2024-01-11T08:11:24.491538+0000","last_became_peered":"2024-01-11T08:11:24.491538+0000","last_unstale":"2024-01-11T08:11:24.501839+0000","last_undegraded":"2024-01-11T08:11:24.501839+0000","last_fullsized":"2024-01-11T08:11:24.501839+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:43:46.629601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.478800+0000","last_change":"2024-01-11T08:11:23.949235+0000","last_active":"2024-01-11T08:11:24.478800+0000","last_peered":"2024-01-11T08:11:24.478800+0000","last_clean":"2024-01-11T08:11:24.478800+0000","last_became_active":"2024-01-11T08:11:23.948857+0000","last_became_peered":"2024-01-11T08:11:23.948857+0000","last_unstale":"2024-01-11T08:11:24.478800+0000","last_undegraded":"2024-01-11T08:11:24.478800+0000","last_fullsized":"2024-01-11T08:11:24.478800+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:52:09.193975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.507666+0000","last_change":"2024-01-11T08:11:24.504765+0000","last_active":"2024-01-11T08:11:24.507666+0000","last_peered":"2024-01-11T08:11:24.507666+0000","last_clean":"2024-01-11T08:11:24.507666+0000","last_became_active":"2024-01-11T08:11:24.504476+0000","last_became_peered":"2024-01-11T08:11:24.504476+0000","last_unstale":"2024-01-11T08:11:24.507666+0000","last_undegraded":"2024-01-11T08:11:24.507666+0000","last_fullsized":"2024-01-11T08:11:24.507666+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:23.914281+0000","last_clean_scrub_stamp":"2024-01-11T08:11:23.914281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:43:46.629601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.486652+0000","last_change":"2024-01-11T08:11:23.943065+0000","last_active":"2024-01-11T08:11:24.486652+0000","last_peered":"2024-01-11T08:11:24.486652+0000","last_clean":"2024-01-11T08:11:24.486652+0000","last_became_active":"2024-01-11T08:11:23.942262+0000","last_became_peered":"2024-01-11T08:11:23.942262+0000","last_unstale":"2024-01-11T08:11:24.486652+0000","last_undegraded":"2024-01-11T08:11:24.486652+0000","last_fullsized":"2024-01-11T08:11:24.486652+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:11:22.904830+0000","last_clean_scrub_stamp":"2024-01-11T08:11:22.904830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:09:20.134819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":171,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T08:11:24.483181+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:11:24.483181+0000","last_peered":"2024-01-11T08:11:24.483181+0000","last_clean":"2024-01-11T08:11:24.483181+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:11:24.483181+0000","last_undegraded":"2024-01-11T08:11:24.483181+0000","last_fullsized":"2024-01-11T08:11:24.483181+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298248,"kb_used_data":960,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073592,"statfs":{"total":96636764160,"available":96331358208,"internally_reserved":0,"allocated":983040,"data_stored":741297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":2.8300000000000001,"5min":2.8300000000000001,"15min":2.8300000000000001},"last":0.502},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.57199999999999995}]},{"osd":1,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.63800000000000001}]},{"osd":2,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.69699999999999995}]},{"osd":3,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":2.9119999999999999,"5min":2.9119999999999999,"15min":2.9119999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.72699999999999998}]},{"osd":4,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.59199999999999997}]},{"osd":5,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.68700000000000006}]},{"osd":6,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.67500000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574865,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297672,"kb_used_data":384,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074168,"statfs":{"total":96636764160,"available":96331948032,"internally_reserved":0,"allocated":393216,"data_stored":151427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.81699999999999995}]},{"osd":1,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.53100000000000003}]},{"osd":2,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.621},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.65300000000000002}]},{"osd":3,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67600000000000005}]},{"osd":4,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.76900000000000002}]},{"osd":5,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.72599999999999998}]},{"osd":7,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.80900000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297680,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074160,"statfs":{"total":96636764160,"available":96331939840,"internally_reserved":0,"allocated":401408,"data_stored":152397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.48399999999999999}]},{"osd":2,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.44900000000000001}]},{"osd":3,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.57399999999999995}]},{"osd":4,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.77500000000000002}]},{"osd":5,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.68799999999999994}]},{"osd":6,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.76000000000000001}]},{"osd":7,"last update":"Thu Jan 11 08:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.65400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302340,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069500,"statfs":{"total":96636764160,"available":96327168000,"internally_reserved":0,"allocated":978944,"data_stored":741207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.433}]},{"osd":2,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.48599999999999999}]},{"osd":3,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.61899999999999999}]},{"osd":4,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.67100000000000004}]},{"osd":5,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.69399999999999995}]},{"osd":6,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.79800000000000004}]},{"osd":7,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.79100000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297668,"kb_used_data":380,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":389120,"data_stored":150909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.52700000000000002}]},{"osd":1,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.59499999999999997}]},{"osd":4,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.77300000000000002}]},{"osd":5,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":2.9209999999999998,"5min":2.9209999999999998,"15min":2.9209999999999998},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":0.95199999999999996}]},{"osd":6,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":2.968,"5min":2.968,"15min":2.968},"last":0.82199999999999995}]},{"osd":7,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":0.69099999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297680,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074160,"statfs":{"total":96636764160,"available":96331939840,"internally_reserved":0,"allocated":401408,"data_stored":152397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.439}]},{"osd":1,"last update":"Thu Jan 11 08:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.66200000000000003}]},{"osd":2,"last update":"Thu Jan 11 08:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.36599999999999999}]},{"osd":4,"last update":"Thu Jan 11 08:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Jan 11 08:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.622}]},{"osd":6,"last update":"Thu Jan 11 08:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":3.5579999999999998,"5min":3.5579999999999998,"15min":3.5579999999999998},"last":0.755}]},{"osd":7,"last update":"Thu Jan 11 08:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":3.5089999999999999,"5min":3.5089999999999999,"15min":3.5089999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.59299999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297672,"kb_used_data":384,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074168,"statfs":{"total":96636764160,"available":96331948032,"internally_reserved":0,"allocated":393216,"data_stored":150943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.69099999999999995}]},{"osd":1,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.4740000000000002,"5min":2.4740000000000002,"15min":2.4740000000000002},"last":0.71499999999999997}]},{"osd":2,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.77200000000000002}]},{"osd":3,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.626},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.96599999999999997}]},{"osd":5,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.79100000000000004}]},{"osd":6,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.86599999999999999}]},{"osd":7,"last update":"Thu Jan 11 08:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.747},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.98999999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297672,"kb_used_data":384,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074168,"statfs":{"total":96636764160,"available":96331948032,"internally_reserved":0,"allocated":393216,"data_stored":151483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.69699999999999995}]},{"osd":1,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.79800000000000004}]},{"osd":2,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.76100000000000001}]},{"osd":3,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.78400000000000003}]},{"osd":4,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.621},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.55400000000000005}]},{"osd":6,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.57699999999999996}]},{"osd":7,"last update":"Thu Jan 11 08:11:21 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.65500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:11:29.832 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-11T08:11:29.832 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-11T08:11:30.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.074+0000 7f3f549ff700 1 Processor -- start 2024-01-11T08:11:30.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.075+0000 7f3f549ff700 1 -- start start 2024-01-11T08:11:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.076+0000 7f3f549ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f500b2b90 0x7f3f50159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.076+0000 7f3f549ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f500b07b0 0x7f3f50159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.076+0000 7f3f549ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f500cd730 con 0x7f3f500b03e0 2024-01-11T08:11:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.076+0000 7f3f549ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f500ccb20 con 0x7f3f500b2b90 2024-01-11T08:11:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.076+0000 7f3f549ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f500cce00 con 0x7f3f500b07b0 2024-01-11T08:11:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.076+0000 7f3f4dd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f500b2b90 0x7f3f50159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.076+0000 7f3f4dd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f500b2b90 0x7f3f50159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42456/0 (socket says 172.21.15.27:42456) 2024-01-11T08:11:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.076+0000 7f3f4dd9b700 1 -- 172.21.15.27:0/4004883947 learned_addr learned my addr 172.21.15.27:0/4004883947 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.076+0000 7f3f4ed9d700 1 --2- 172.21.15.27:0/4004883947 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f500b07b0 0x7f3f50159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.077+0000 7f3f4dd9b700 1 -- 172.21.15.27:0/4004883947 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f500b07b0 msgr2=0x7f3f50159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.077+0000 7f3f4dd9b700 1 --2- 172.21.15.27:0/4004883947 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f500b07b0 0x7f3f50159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.077+0000 7f3f4dd9b700 1 -- 172.21.15.27:0/4004883947 >> v1:172.21.15.77:6789/0 conn(0x7f3f500b03e0 legacy=0x7f3f500b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.077+0000 7f3f4dd9b700 1 -- 172.21.15.27:0/4004883947 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f500cd0e0 con 0x7f3f500b2b90 2024-01-11T08:11:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.077+0000 7f3f4dd9b700 1 --2- 172.21.15.27:0/4004883947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f500b2b90 0x7f3f50159200 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3f440099b0 tx=0x7f3f44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c4e27514ce1d7c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.077+0000 7f3f4d59a700 1 -- 172.21.15.27:0/4004883947 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44009ea0 con 0x7f3f500b2b90 2024-01-11T08:11:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.077+0000 7f3f4d59a700 1 -- 172.21.15.27:0/4004883947 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4400b860 con 0x7f3f500b2b90 2024-01-11T08:11:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.077+0000 7f3f4d59a700 1 -- 172.21.15.27:0/4004883947 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4400bc00 con 0x7f3f500b2b90 2024-01-11T08:11:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.078+0000 7f3f549ff700 1 -- 172.21.15.27:0/4004883947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f500b2b90 msgr2=0x7f3f50159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.078+0000 7f3f549ff700 1 --2- 172.21.15.27:0/4004883947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f500b2b90 0x7f3f50159200 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3f440099b0 tx=0x7f3f44009250 comp rx=0 tx=0).stop 2024-01-11T08:11:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.078+0000 7f3f549ff700 1 -- 172.21.15.27:0/4004883947 shutdown_connections 2024-01-11T08:11:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.078+0000 7f3f549ff700 1 --2- 172.21.15.27:0/4004883947 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f500b07b0 0x7f3f50159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.078+0000 7f3f549ff700 1 --2- 172.21.15.27:0/4004883947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f500b2b90 0x7f3f50159200 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.078+0000 7f3f549ff700 1 -- 172.21.15.27:0/4004883947 >> 172.21.15.27:0/4004883947 conn(0x7f3f5006ed40 msgr2=0x7f3f5006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.078+0000 7f3f549ff700 1 -- 172.21.15.27:0/4004883947 shutdown_connections 2024-01-11T08:11:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.078+0000 7f3f549ff700 1 -- 172.21.15.27:0/4004883947 wait complete. 2024-01-11T08:11:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.079+0000 7f3f549ff700 1 Processor -- start 2024-01-11T08:11:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.080+0000 7f3f549ff700 1 -- start start 2024-01-11T08:11:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.080+0000 7f3f549ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f500b03e0 0x7f3f5014da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.081+0000 7f3f549ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f500b07b0 0x7f3f500cf780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.081+0000 7f3f549ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f500b2b90 0x7f3f500cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.081+0000 7f3f549ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5016dd60 con 0x7f3f500b07b0 2024-01-11T08:11:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.081+0000 7f3f549ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f50169a30 con 0x7f3f500b2b90 2024-01-11T08:11:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.081+0000 7f3f549ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5014dfd0 con 0x7f3f500b03e0 2024-01-11T08:11:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.081+0000 7f3f4dd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f500b07b0 0x7f3f500cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.081+0000 7f3f4dd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f500b07b0 0x7f3f500cf780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42468/0 (socket says 172.21.15.27:42468) 2024-01-11T08:11:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.081+0000 7f3f4dd9b700 1 -- 172.21.15.27:0/3849518083 learned_addr learned my addr 172.21.15.27:0/3849518083 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.081+0000 7f3f4e59c700 1 --2- 172.21.15.27:0/3849518083 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f500b03e0 0x7f3f5014da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.081+0000 7f3f4ed9d700 1 --2- 172.21.15.27:0/3849518083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f500b2b90 0x7f3f500cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.081+0000 7f3f4dd9b700 1 -- 172.21.15.27:0/3849518083 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f500b03e0 msgr2=0x7f3f5014da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.081+0000 7f3f4dd9b700 1 --2- 172.21.15.27:0/3849518083 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f500b03e0 0x7f3f5014da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.081+0000 7f3f4dd9b700 1 -- 172.21.15.27:0/3849518083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f500b2b90 msgr2=0x7f3f500cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.081+0000 7f3f4dd9b700 1 --2- 172.21.15.27:0/3849518083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f500b2b90 0x7f3f500cfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.081+0000 7f3f4dd9b700 1 -- 172.21.15.27:0/3849518083 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f440094f0 con 0x7f3f500b07b0 2024-01-11T08:11:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.081+0000 7f3f4ed9d700 1 --2- 172.21.15.27:0/3849518083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f500b2b90 0x7f3f500cfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.082+0000 7f3f4dd9b700 1 --2- 172.21.15.27:0/3849518083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f500b07b0 0x7f3f500cf780 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3f50069540 tx=0x7f3f44017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.082+0000 7f3f3e7fc700 1 -- 172.21.15.27:0/3849518083 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4400bbe0 con 0x7f3f500b07b0 2024-01-11T08:11:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.082+0000 7f3f3e7fc700 1 -- 172.21.15.27:0/3849518083 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f44011420 con 0x7f3f500b07b0 2024-01-11T08:11:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.082+0000 7f3f549ff700 1 -- 172.21.15.27:0/3849518083 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f5016d0e0 con 0x7f3f500b07b0 2024-01-11T08:11:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.082+0000 7f3f3e7fc700 1 -- 172.21.15.27:0/3849518083 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f440117c0 con 0x7f3f500b07b0 2024-01-11T08:11:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.082+0000 7f3f549ff700 1 -- 172.21.15.27:0/3849518083 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f5016d6a0 con 0x7f3f500b07b0 2024-01-11T08:11:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.083+0000 7f3f549ff700 1 -- 172.21.15.27:0/3849518083 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f5004c3c0 con 0x7f3f500b07b0 2024-01-11T08:11:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.084+0000 7f3f3e7fc700 1 -- 172.21.15.27:0/3849518083 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3f44011bb0 con 0x7f3f500b07b0 2024-01-11T08:11:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.085+0000 7f3f3e7fc700 1 --2- 172.21.15.27:0/3849518083 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3f30066f30 0x7f3f300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.085+0000 7f3f3e7fc700 1 -- 172.21.15.27:0/3849518083 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f3f44085e60 con 0x7f3f500b07b0 2024-01-11T08:11:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.086+0000 7f3f4e59c700 1 --2- 172.21.15.27:0/3849518083 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3f30066f30 0x7f3f300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.086+0000 7f3f4e59c700 1 --2- 172.21.15.27:0/3849518083 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3f30066f30 0x7f3f300693e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3f500afe60 tx=0x7f3f40009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.089+0000 7f3f3e7fc700 1 -- 172.21.15.27:0/3849518083 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3f44053030 con 0x7f3f500b07b0 2024-01-11T08:11:30.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.369+0000 7f3f549ff700 1 -- 172.21.15.27:0/3849518083 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f3f500cd3c0 con 0x7f3f500b07b0 2024-01-11T08:11:30.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.373+0000 7f3f3e7fc700 1 -- 172.21.15.27:0/3849518083 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v9) v1 ==== 190+0+0 (secure 0 0 0) 0x7f3f44011e60 con 0x7f3f500b07b0 2024-01-11T08:11:30.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.375+0000 7f3f549ff700 1 -- 172.21.15.27:0/3849518083 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3f30066f30 msgr2=0x7f3f300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:30.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.375+0000 7f3f549ff700 1 --2- 172.21.15.27:0/3849518083 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3f30066f30 0x7f3f300693e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3f500afe60 tx=0x7f3f40009040 comp rx=0 tx=0).stop 2024-01-11T08:11:30.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.375+0000 7f3f549ff700 1 -- 172.21.15.27:0/3849518083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f500b07b0 msgr2=0x7f3f500cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:30.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.375+0000 7f3f549ff700 1 --2- 172.21.15.27:0/3849518083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f500b07b0 0x7f3f500cf780 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3f50069540 tx=0x7f3f44017000 comp rx=0 tx=0).stop 2024-01-11T08:11:30.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.375+0000 7f3f549ff700 1 -- 172.21.15.27:0/3849518083 shutdown_connections 2024-01-11T08:11:30.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.375+0000 7f3f549ff700 1 --2- 172.21.15.27:0/3849518083 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3f30066f30 0x7f3f300693e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:30.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.375+0000 7f3f549ff700 1 --2- 172.21.15.27:0/3849518083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f500b2b90 0x7f3f500cfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:30.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.375+0000 7f3f549ff700 1 --2- 172.21.15.27:0/3849518083 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f500b03e0 0x7f3f5014da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:30.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.375+0000 7f3f549ff700 1 --2- 172.21.15.27:0/3849518083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f500b07b0 0x7f3f500cf780 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:30.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.375+0000 7f3f549ff700 1 -- 172.21.15.27:0/3849518083 >> 172.21.15.27:0/3849518083 conn(0x7f3f5006ed40 msgr2=0x7f3f500d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:30.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.376+0000 7f3f549ff700 1 -- 172.21.15.27:0/3849518083 shutdown_connections 2024-01-11T08:11:30.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.376+0000 7f3f549ff700 1 -- 172.21.15.27:0/3849518083 wait complete. 2024-01-11T08:11:30.379 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-11T08:11:30.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:11:30.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.596+0000 7f07077ff700 1 Processor -- start 2024-01-11T08:11:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.596+0000 7f07077ff700 1 -- start start 2024-01-11T08:11:30.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.598+0000 7f07077ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f07000cc410 0x7f0700159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:30.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.598+0000 7f07077ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07000cd730 con 0x7f07000b03e0 2024-01-11T08:11:30.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.598+0000 7f07077ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07000ccb20 con 0x7f07000b2b90 2024-01-11T08:11:30.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.598+0000 7f07077ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07000cce00 con 0x7f07000cc410 2024-01-11T08:11:30.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.598+0000 7f0705d9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f07000cc410 0x7f0700159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:30.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.598+0000 7f070559b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f07000b03e0 0x7f07000b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50732/0 (socket says 172.21.15.27:50732) 2024-01-11T08:11:30.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.598+0000 7f070559b700 1 -- 172.21.15.27:0/1527941194 learned_addr learned my addr 172.21.15.27:0/1527941194 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:30.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.598+0000 7f0705d9c700 1 -- 172.21.15.27:0/1527941194 >> v1:172.21.15.77:6789/0 conn(0x7f07000b2b90 legacy=0x7f0700159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:30.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.598+0000 7f0705d9c700 1 -- 172.21.15.27:0/1527941194 >> v1:172.21.15.27:6789/0 conn(0x7f07000b03e0 legacy=0x7f07000b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:30.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.599+0000 7f0705d9c700 1 -- 172.21.15.27:0/1527941194 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07000cd0e0 con 0x7f07000cc410 2024-01-11T08:11:30.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.598+0000 7f06f7fff700 1 -- 172.21.15.27:0/1527941194 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703225939 0 0) 0x7f07000cd730 con 0x7f07000b03e0 2024-01-11T08:11:30.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.599+0000 7f0705d9c700 1 --2- 172.21.15.27:0/1527941194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07000cc410 0x7f0700159910 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f06f8009e90 tx=0x7f06f8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef7d27484567f4e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:30.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.599+0000 7f06f7fff700 1 -- 172.21.15.27:0/1527941194 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f800e070 con 0x7f07000cc410 2024-01-11T08:11:30.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.599+0000 7f06f7fff700 1 -- 172.21.15.27:0/1527941194 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f800bbf0 con 0x7f07000cc410 2024-01-11T08:11:30.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.599+0000 7f06f7fff700 1 -- 172.21.15.27:0/1527941194 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f8012430 con 0x7f07000cc410 2024-01-11T08:11:30.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.600+0000 7f07077ff700 1 -- 172.21.15.27:0/1527941194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07000cc410 msgr2=0x7f0700159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:30.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.600+0000 7f07077ff700 1 --2- 172.21.15.27:0/1527941194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07000cc410 0x7f0700159910 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f06f8009e90 tx=0x7f06f8009780 comp rx=0 tx=0).stop 2024-01-11T08:11:30.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.600+0000 7f07077ff700 1 -- 172.21.15.27:0/1527941194 shutdown_connections 2024-01-11T08:11:30.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.600+0000 7f07077ff700 1 --2- 172.21.15.27:0/1527941194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07000cc410 0x7f0700159910 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:30.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.600+0000 7f07077ff700 1 -- 172.21.15.27:0/1527941194 >> 172.21.15.27:0/1527941194 conn(0x7f070006ed40 msgr2=0x7f070006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:30.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.600+0000 7f07077ff700 1 -- 172.21.15.27:0/1527941194 shutdown_connections 2024-01-11T08:11:30.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.600+0000 7f07077ff700 1 -- 172.21.15.27:0/1527941194 wait complete. 2024-01-11T08:11:30.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.601+0000 7f07077ff700 1 Processor -- start 2024-01-11T08:11:30.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.601+0000 7f07077ff700 1 -- start start 2024-01-11T08:11:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.602+0000 7f07077ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07000b03e0 0x7f07000c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.602+0000 7f07077ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07000b2b90 0x7f0700158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.602+0000 7f07077ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07000cc410 0x7f07000c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.602+0000 7f07077ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07000ca5a0 con 0x7f07000b03e0 2024-01-11T08:11:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.602+0000 7f07077ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070015cc40 con 0x7f07000b2b90 2024-01-11T08:11:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.602+0000 7f07077ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07000cecb0 con 0x7f07000cc410 2024-01-11T08:11:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.603+0000 7f070559b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07000b03e0 0x7f07000c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.603+0000 7f070559b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07000b03e0 0x7f07000c9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42500/0 (socket says 172.21.15.27:42500) 2024-01-11T08:11:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.603+0000 7f070559b700 1 -- 172.21.15.27:0/3455213714 learned_addr learned my addr 172.21.15.27:0/3455213714 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:30.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.603+0000 7f0704d9a700 1 --2- 172.21.15.27:0/3455213714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07000b2b90 0x7f0700158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:30.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.603+0000 7f0705d9c700 1 --2- 172.21.15.27:0/3455213714 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07000cc410 0x7f07000c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:30.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.603+0000 7f070559b700 1 -- 172.21.15.27:0/3455213714 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07000cc410 msgr2=0x7f07000c8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:30.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.603+0000 7f070559b700 1 --2- 172.21.15.27:0/3455213714 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07000cc410 0x7f07000c8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:30.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.603+0000 7f070559b700 1 -- 172.21.15.27:0/3455213714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07000b2b90 msgr2=0x7f0700158aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:30.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.603+0000 7f070559b700 1 --2- 172.21.15.27:0/3455213714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07000b2b90 0x7f0700158aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:30.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.603+0000 7f070559b700 1 -- 172.21.15.27:0/3455213714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f8009a20 con 0x7f07000b03e0 2024-01-11T08:11:30.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.603+0000 7f070559b700 1 --2- 172.21.15.27:0/3455213714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07000b03e0 0x7f07000c9ee0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f06fc00c870 tx=0x7f06fc007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:30.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.604+0000 7f06f57fa700 1 -- 172.21.15.27:0/3455213714 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc005c00 con 0x7f07000b03e0 2024-01-11T08:11:30.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.604+0000 7f06f57fa700 1 -- 172.21.15.27:0/3455213714 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06fc005d60 con 0x7f07000b03e0 2024-01-11T08:11:30.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.604+0000 7f06f57fa700 1 -- 172.21.15.27:0/3455213714 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc007320 con 0x7f07000b03e0 2024-01-11T08:11:30.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.604+0000 7f07077ff700 1 -- 172.21.15.27:0/3455213714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f070015c680 con 0x7f07000b03e0 2024-01-11T08:11:30.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.604+0000 7f07077ff700 1 -- 172.21.15.27:0/3455213714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0700158fe0 con 0x7f07000b03e0 2024-01-11T08:11:30.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.605+0000 7f07077ff700 1 -- 172.21.15.27:0/3455213714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07000718d0 con 0x7f07000b03e0 2024-01-11T08:11:30.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.606+0000 7f06f57fa700 1 -- 172.21.15.27:0/3455213714 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f06fc01b3b0 con 0x7f07000b03e0 2024-01-11T08:11:30.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.607+0000 7f06f57fa700 1 --2- 172.21.15.27:0/3455213714 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f06e4066f30 0x7f06e40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:30.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.607+0000 7f06f57fa700 1 -- 172.21.15.27:0/3455213714 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f06fc083950 con 0x7f07000b03e0 2024-01-11T08:11:30.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.608+0000 7f0704d9a700 1 --2- 172.21.15.27:0/3455213714 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f06e4066f30 0x7f06e40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:30.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.608+0000 7f0704d9a700 1 --2- 172.21.15.27:0/3455213714 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f06e4066f30 0x7f06e40693e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f07000c9cd0 tx=0x7f06f0007ec0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:30.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.612+0000 7f06f57fa700 1 -- 172.21.15.27:0/3455213714 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f06fc050bc0 con 0x7f07000b03e0 2024-01-11T08:11:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.858+0000 7f07077ff700 1 -- 172.21.15.27:0/3455213714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07000ceb00 con 0x7f07000b03e0 2024-01-11T08:11:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.859+0000 7f06f57fa700 1 -- 172.21.15.27:0/3455213714 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4591 (secure 0 0 0) 0x7f06fc017090 con 0x7f07000b03e0 2024-01-11T08:11:30.859 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:11:30.859 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/2930156910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2930156910},{"type":"v1","addr":"172.21.15.77:6836","nonce":2930156910}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/685019444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":685019444},{"type":"v1","addr":"172.21.15.27:6834","nonce":685019444}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4861,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/144853545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":144853545},{"type":"v1","addr":"172.21.15.77:6837","nonce":144853545}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:11:24.435181+0000","modified":"2024-01-11T08:11:27.940837+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/4056690675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4056690675},{"type":"v1","addr":"172.21.15.27:6835","nonce":4056690675}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2024-01-11T08:11:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.861+0000 7f07077ff700 1 -- 172.21.15.27:0/3455213714 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f06e4066f30 msgr2=0x7f06e40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.861+0000 7f07077ff700 1 --2- 172.21.15.27:0/3455213714 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f06e4066f30 0x7f06e40693e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f07000c9cd0 tx=0x7f06f0007ec0 comp rx=0 tx=0).stop 2024-01-11T08:11:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.861+0000 7f07077ff700 1 -- 172.21.15.27:0/3455213714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07000b03e0 msgr2=0x7f07000c9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.861+0000 7f07077ff700 1 --2- 172.21.15.27:0/3455213714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07000b03e0 0x7f07000c9ee0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f06fc00c870 tx=0x7f06fc007d90 comp rx=0 tx=0).stop 2024-01-11T08:11:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.861+0000 7f07077ff700 1 -- 172.21.15.27:0/3455213714 shutdown_connections 2024-01-11T08:11:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.861+0000 7f07077ff700 1 --2- 172.21.15.27:0/3455213714 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f06e4066f30 0x7f06e40693e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.861+0000 7f07077ff700 1 --2- 172.21.15.27:0/3455213714 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07000cc410 0x7f07000c8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.861+0000 7f07077ff700 1 --2- 172.21.15.27:0/3455213714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07000b03e0 0x7f07000c9ee0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.861+0000 7f07077ff700 1 --2- 172.21.15.27:0/3455213714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07000b2b90 0x7f0700158aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.861+0000 7f07077ff700 1 -- 172.21.15.27:0/3455213714 >> 172.21.15.27:0/3455213714 conn(0x7f070006ed40 msgr2=0x7f07000b4c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.862+0000 7f07077ff700 1 -- 172.21.15.27:0/3455213714 shutdown_connections 2024-01-11T08:11:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:30.862+0000 7f07077ff700 1 -- 172.21.15.27:0/3455213714 wait complete. 2024-01-11T08:11:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-01-11T08:11:31.282 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-11T08:11:24.435181+0000', 'modified': '2024-01-11T08:11:27.940837+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4870}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4870': {'gid': 4870, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.27:6835/4056690675', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 4056690675}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 4056690675}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-11T08:11:31.282 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-11T08:11:31.283 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-11T08:11:31.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.485+0000 7f64949ff700 1 Processor -- start 2024-01-11T08:11:31.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.485+0000 7f64949ff700 1 -- start start 2024-01-11T08:11:31.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.487+0000 7f64949ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f64900b0a40 0x7f649015da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.487+0000 7f64949ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f649015df40 0x7f6490068720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.487+0000 7f64949ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649015d290 con 0x7f649006bf10 2024-01-11T08:11:31.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.487+0000 7f64949ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f649015d570 con 0x7f64900b0a40 2024-01-11T08:11:31.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.487+0000 7f64949ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6490154180 con 0x7f649015df40 2024-01-11T08:11:31.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.487+0000 7f648dd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f64900b0a40 0x7f649015da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.487+0000 7f648e59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f649006bf10 0x7f6490159c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50734/0 (socket says 172.21.15.27:50734) 2024-01-11T08:11:31.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.487+0000 7f648e59c700 1 -- 172.21.15.27:0/2002657501 learned_addr learned my addr 172.21.15.27:0/2002657501 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:31.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.487+0000 7f648ed9d700 1 --2- 172.21.15.27:0/2002657501 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f649015df40 0x7f6490068720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.487+0000 7f648d59a700 1 -- 172.21.15.27:0/2002657501 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188832713 0 0) 0x7f649015d290 con 0x7f649006bf10 2024-01-11T08:11:31.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.487+0000 7f648d59a700 1 -- 172.21.15.27:0/2002657501 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6478003570 con 0x7f649006bf10 2024-01-11T08:11:31.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.487+0000 7f648dd9b700 1 -- 172.21.15.27:0/2002657501 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f649015df40 msgr2=0x7f6490068720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:31.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.487+0000 7f648dd9b700 1 --2- 172.21.15.27:0/2002657501 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f649015df40 0x7f6490068720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.488+0000 7f648dd9b700 1 -- 172.21.15.27:0/2002657501 >> v1:172.21.15.27:6789/0 conn(0x7f649006bf10 legacy=0x7f6490159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:31.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.488+0000 7f648dd9b700 1 -- 172.21.15.27:0/2002657501 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649015a510 con 0x7f64900b0a40 2024-01-11T08:11:31.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.488+0000 7f648dd9b700 1 --2- 172.21.15.27:0/2002657501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64900b0a40 0x7f649015da00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f64840098e0 tx=0x7f64840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e55a94e9a983cd server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:31.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.488+0000 7f648d59a700 1 -- 172.21.15.27:0/2002657501 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6484011630 con 0x7f64900b0a40 2024-01-11T08:11:31.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.488+0000 7f648d59a700 1 -- 172.21.15.27:0/2002657501 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6484011790 con 0x7f64900b0a40 2024-01-11T08:11:31.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.488+0000 7f648d59a700 1 -- 172.21.15.27:0/2002657501 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6484011b10 con 0x7f64900b0a40 2024-01-11T08:11:31.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.489+0000 7f64949ff700 1 -- 172.21.15.27:0/2002657501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64900b0a40 msgr2=0x7f649015da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:31.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.489+0000 7f64949ff700 1 --2- 172.21.15.27:0/2002657501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64900b0a40 0x7f649015da00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f64840098e0 tx=0x7f64840091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.489+0000 7f64949ff700 1 -- 172.21.15.27:0/2002657501 shutdown_connections 2024-01-11T08:11:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.489+0000 7f64949ff700 1 --2- 172.21.15.27:0/2002657501 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f649015df40 0x7f6490068720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.489+0000 7f64949ff700 1 --2- 172.21.15.27:0/2002657501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64900b0a40 0x7f649015da00 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.489+0000 7f64949ff700 1 -- 172.21.15.27:0/2002657501 >> 172.21.15.27:0/2002657501 conn(0x7f64900ce3c0 msgr2=0x7f64900ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.489+0000 7f64949ff700 1 -- 172.21.15.27:0/2002657501 shutdown_connections 2024-01-11T08:11:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.489+0000 7f64949ff700 1 -- 172.21.15.27:0/2002657501 wait complete. 2024-01-11T08:11:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.491+0000 7f64949ff700 1 Processor -- start 2024-01-11T08:11:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.491+0000 7f64949ff700 1 -- start start 2024-01-11T08:11:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.492+0000 7f64949ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f649006bf10 0x7f64900698b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.492+0000 7f64949ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64900b0a40 0x7f64901523b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.493+0000 7f64949ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f649015df40 0x7f64901528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.493+0000 7f64949ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64901565c0 con 0x7f649015df40 2024-01-11T08:11:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.493+0000 7f64949ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6490152e30 con 0x7f64900b0a40 2024-01-11T08:11:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.493+0000 7f64949ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64900c5ab0 con 0x7f649006bf10 2024-01-11T08:11:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.493+0000 7f648ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f649015df40 0x7f64901528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.493+0000 7f648ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f649015df40 0x7f64901528f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42524/0 (socket says 172.21.15.27:42524) 2024-01-11T08:11:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.493+0000 7f648ed9d700 1 -- 172.21.15.27:0/2099694705 learned_addr learned my addr 172.21.15.27:0/2099694705 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.493+0000 7f648dd9b700 1 --2- 172.21.15.27:0/2099694705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64900b0a40 0x7f64901523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.493+0000 7f648e59c700 1 --2- 172.21.15.27:0/2099694705 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f649006bf10 0x7f64900698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.493+0000 7f648ed9d700 1 -- 172.21.15.27:0/2099694705 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f649006bf10 msgr2=0x7f64900698b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.493+0000 7f648ed9d700 1 --2- 172.21.15.27:0/2099694705 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f649006bf10 0x7f64900698b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.493+0000 7f648ed9d700 1 -- 172.21.15.27:0/2099694705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64900b0a40 msgr2=0x7f64901523b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.493+0000 7f648ed9d700 1 --2- 172.21.15.27:0/2099694705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64900b0a40 0x7f64901523b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.493+0000 7f648ed9d700 1 -- 172.21.15.27:0/2099694705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6484009420 con 0x7f649015df40 2024-01-11T08:11:31.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.494+0000 7f648ed9d700 1 --2- 172.21.15.27:0/2099694705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f649015df40 0x7f64901528f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6488009df0 tx=0x7f648800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.494+0000 7f647e7fc700 1 -- 172.21.15.27:0/2099694705 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648800caa0 con 0x7f649015df40 2024-01-11T08:11:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.494+0000 7f647e7fc700 1 -- 172.21.15.27:0/2099694705 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648800f040 con 0x7f649015df40 2024-01-11T08:11:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.494+0000 7f647e7fc700 1 -- 172.21.15.27:0/2099694705 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648800ce70 con 0x7f649015df40 2024-01-11T08:11:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.494+0000 7f64949ff700 1 -- 172.21.15.27:0/2099694705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6490155d20 con 0x7f649015df40 2024-01-11T08:11:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.494+0000 7f64949ff700 1 -- 172.21.15.27:0/2099694705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64901562e0 con 0x7f649015df40 2024-01-11T08:11:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.496+0000 7f647e7fc700 1 -- 172.21.15.27:0/2099694705 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6488014690 con 0x7f649015df40 2024-01-11T08:11:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.497+0000 7f647e7fc700 1 --2- 172.21.15.27:0/2099694705 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6470066f30 0x7f64700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.497+0000 7f647e7fc700 1 -- 172.21.15.27:0/2099694705 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f6488084de0 con 0x7f649015df40 2024-01-11T08:11:31.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.498+0000 7f648e59c700 1 --2- 172.21.15.27:0/2099694705 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6470066f30 0x7f64700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.498+0000 7f648e59c700 1 --2- 172.21.15.27:0/2099694705 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6470066f30 0x7f64700693e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6480007fd0 tx=0x7f6480009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:31.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.499+0000 7f64a3213b80 1 Processor -- start 2024-01-11T08:11:31.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.499+0000 7f64a3213b80 1 -- start start 2024-01-11T08:11:31.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.500+0000 7f64a3213b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a4058b20 0x55a6a4039df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.500+0000 7f64a3213b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40343a0 0x55a6a4060860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.500+0000 7f64a3213b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a4058ef0 0x55a6a4060da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.500+0000 7f64a3213b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6a3f51310 con 0x55a6a40343a0 2024-01-11T08:11:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.500+0000 7f64a3213b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6a3f8e130 con 0x55a6a4058b20 2024-01-11T08:11:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.500+0000 7f64a3213b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a6a3f8baf0 con 0x55a6a4058ef0 2024-01-11T08:11:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.500+0000 7f648dd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40343a0 0x55a6a4060860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.500+0000 7f648dd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40343a0 0x55a6a4060860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42526/0 (socket says 172.21.15.27:42526) 2024-01-11T08:11:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.500+0000 7f648dd9b700 1 -- 172.21.15.27:0/3142057317 learned_addr learned my addr 172.21.15.27:0/3142057317 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.500+0000 7f648e59c700 1 --2- 172.21.15.27:0/3142057317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a4058b20 0x55a6a4039df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.501+0000 7f648ed9d700 1 --2- 172.21.15.27:0/3142057317 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a4058ef0 0x55a6a4060da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.501+0000 7f648dd9b700 1 -- 172.21.15.27:0/3142057317 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a4058ef0 msgr2=0x55a6a4060da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.501+0000 7f648dd9b700 1 --2- 172.21.15.27:0/3142057317 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a4058ef0 0x55a6a4060da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.501+0000 7f648dd9b700 1 -- 172.21.15.27:0/3142057317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a4058b20 msgr2=0x55a6a4039df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:31.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.501+0000 7f648dd9b700 1 --2- 172.21.15.27:0/3142057317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a4058b20 0x55a6a4039df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.501+0000 7f648dd9b700 1 -- 172.21.15.27:0/3142057317 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6a3e7a0a0 con 0x55a6a40343a0 2024-01-11T08:11:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.501+0000 7f648e59c700 1 --2- 172.21.15.27:0/3142057317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a4058b20 0x55a6a4039df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.501+0000 7f648dd9b700 1 --2- 172.21.15.27:0/3142057317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40343a0 0x55a6a4060860 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f649006b3c0 tx=0x7f6484017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f13de8ac6a12f5d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.501+0000 7f646f7fe700 1 -- 172.21.15.27:0/3142057317 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6484021070 con 0x55a6a40343a0 2024-01-11T08:11:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.501+0000 7f646f7fe700 1 -- 172.21.15.27:0/3142057317 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6484000bc0 con 0x55a6a40343a0 2024-01-11T08:11:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.501+0000 7f646f7fe700 1 -- 172.21.15.27:0/3142057317 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648401b430 con 0x55a6a40343a0 2024-01-11T08:11:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.502+0000 7f64a3213b80 1 -- 172.21.15.27:0/3142057317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40343a0 msgr2=0x55a6a4060860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.502+0000 7f64a3213b80 1 --2- 172.21.15.27:0/3142057317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40343a0 0x55a6a4060860 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f649006b3c0 tx=0x7f6484017000 comp rx=0 tx=0).stop 2024-01-11T08:11:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.502+0000 7f64a3213b80 1 -- 172.21.15.27:0/3142057317 shutdown_connections 2024-01-11T08:11:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.502+0000 7f64a3213b80 1 --2- 172.21.15.27:0/3142057317 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a4058ef0 0x55a6a4060da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.502+0000 7f64a3213b80 1 --2- 172.21.15.27:0/3142057317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a4058b20 0x55a6a4039df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.502+0000 7f64a3213b80 1 --2- 172.21.15.27:0/3142057317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40343a0 0x55a6a4060860 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.502+0000 7f64a3213b80 1 -- 172.21.15.27:0/3142057317 >> 172.21.15.27:0/3142057317 conn(0x55a6a4036300 msgr2=0x55a6a4038bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:31.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.502+0000 7f64a3213b80 1 -- 172.21.15.27:0/3142057317 shutdown_connections 2024-01-11T08:11:31.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.502+0000 7f64a3213b80 1 -- 172.21.15.27:0/3142057317 wait complete. 2024-01-11T08:11:31.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.503+0000 7f64a3213b80 1 -- start start 2024-01-11T08:11:31.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.503+0000 7f64a3213b80 1 Processor -- start 2024-01-11T08:11:31.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.504+0000 7f64a3213b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40343a0 0x55a6a4059db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.504+0000 7f64a3213b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a4058b20 0x55a6a408dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.504+0000 7f64a3213b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a4058ef0 0x55a6a4056d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.504+0000 7f64a3213b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6a3f878e0 con 0x55a6a40343a0 2024-01-11T08:11:31.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.504+0000 7f64a3213b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6a3f33980 con 0x55a6a4058ef0 2024-01-11T08:11:31.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.504+0000 7f64a3213b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a6a3f33620 con 0x55a6a4058b20 2024-01-11T08:11:31.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.504+0000 7f64a3213b80 10 client.0 _ll_register_callbacks cb 0x55a6a3d96a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T08:11:31.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.504+0000 7f648e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40343a0 0x55a6a4059db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.504+0000 7f648e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40343a0 0x55a6a4059db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42538/0 (socket says 172.21.15.27:42538) 2024-01-11T08:11:31.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.504+0000 7f648e59c700 1 -- 172.21.15.27:0/939982586 learned_addr learned my addr 172.21.15.27:0/939982586 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:31.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.505+0000 7f648ed9d700 1 --2- 172.21.15.27:0/939982586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a4058ef0 0x55a6a4056d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.505+0000 7f648dd9b700 1 --2- 172.21.15.27:0/939982586 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a4058b20 0x55a6a408dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.505+0000 7f648e59c700 1 -- 172.21.15.27:0/939982586 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a4058b20 msgr2=0x55a6a408dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:31.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.505+0000 7f648e59c700 1 --2- 172.21.15.27:0/939982586 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a4058b20 0x55a6a408dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.505+0000 7f648e59c700 1 -- 172.21.15.27:0/939982586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a4058ef0 msgr2=0x55a6a4056d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:31.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.505+0000 7f648e59c700 1 --2- 172.21.15.27:0/939982586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a4058ef0 0x55a6a4056d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.505+0000 7f648e59c700 1 -- 172.21.15.27:0/939982586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f649015a510 con 0x55a6a40343a0 2024-01-11T08:11:31.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.505+0000 7f648ed9d700 1 --2- 172.21.15.27:0/939982586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a4058ef0 0x55a6a4056d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:11:31.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.505+0000 7f648e59c700 1 --2- 172.21.15.27:0/939982586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40343a0 0x55a6a4059db0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6480014800 tx=0x7f6480015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=361172f28ede0c5b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:31.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.505+0000 7f646cff9700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T08:11:31.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.505+0000 7f646cff9700 1 -- 172.21.15.27:0/939982586 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64800011f0 con 0x55a6a40343a0 2024-01-11T08:11:31.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.505+0000 7f646cff9700 1 -- 172.21.15.27:0/939982586 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f648001c040 con 0x55a6a40343a0 2024-01-11T08:11:31.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.505+0000 7f64a3213b80 1 -- 172.21.15.27:0/939982586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a6a402c560 con 0x55a6a40343a0 2024-01-11T08:11:31.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.505+0000 7f646cff9700 1 -- 172.21.15.27:0/939982586 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6480014e20 con 0x55a6a40343a0 2024-01-11T08:11:31.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.505+0000 7f64a3213b80 10 client.5058 fetch_fsmap learned FSMap version 17 2024-01-11T08:11:31.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.505+0000 7f64a3213b80 1 -- 172.21.15.27:0/939982586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55a6a3f563c0 con 0x55a6a40343a0 2024-01-11T08:11:31.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.506+0000 7f646cff9700 1 -- 172.21.15.27:0/939982586 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a6a402c560 con 0x55a6a40343a0 2024-01-11T08:11:31.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.506+0000 7f646cff9700 1 -- 172.21.15.27:0/939982586 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 17) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f64800137d0 con 0x55a6a40343a0 2024-01-11T08:11:31.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.506+0000 7f64a3213b80 10 client.5058 fetch_fsmap finished waiting for FSMap version 17 2024-01-11T08:11:31.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.506+0000 7f64a3213b80 10 client.5058 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-11T08:11:31.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.506+0000 7f64a3213b80 20 client.5058 populate_metadata read hostname 'smithi027' 2024-01-11T08:11:31.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.506+0000 7f64a3213b80 1 --2- 172.21.15.27:0/939982586 >> [v2:172.21.15.27:6833/4056690675,v1:172.21.15.27:6835/4056690675] conn(0x55a6a4066130 0x55a6a4066500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.506+0000 7f64a3213b80 4 client.5058 mds_command: new command op to 4870 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-11T08:11:31.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.506+0000 7f64a3213b80 1 -- 172.21.15.27:0/939982586 --> [v2:172.21.15.27:6833/4056690675,v1:172.21.15.27:6835/4056690675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a6a40090e0 con 0x55a6a4066130 2024-01-11T08:11:31.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.506+0000 7f648dd9b700 1 --2- 172.21.15.27:0/939982586 >> [v2:172.21.15.27:6833/4056690675,v1:172.21.15.27:6835/4056690675] conn(0x55a6a4066130 0x55a6a4066500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.507+0000 7f648dd9b700 1 --2- 172.21.15.27:0/939982586 >> [v2:172.21.15.27:6833/4056690675,v1:172.21.15.27:6835/4056690675] conn(0x55a6a4066130 0x55a6a4066500 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c738341958f7863 server_cookie=c64ed969ac0f30e6 in_seq=0 out_seq=0 2024-01-11T08:11:31.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.507+0000 7f646cff9700 10 client.5058 ms_handle_connect on v2:172.21.15.27:6833/4056690675 2024-01-11T08:11:31.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.507+0000 7f646cff9700 1 -- 172.21.15.27:0/939982586 <== mds.0 v2:172.21.15.27:6833/4056690675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x55a6a40090e0 con 0x55a6a4066130 2024-01-11T08:11:31.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.507+0000 7f646cff9700 10 client.5058 handle_command_reply: tid=0 2024-01-11T08:11:31.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.507+0000 7f64a3213b80 1 client.5058 shutdown 2024-01-11T08:11:31.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.508+0000 7f64a3213b80 1 -- 172.21.15.27:0/939982586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40343a0 msgr2=0x55a6a4059db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:31.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.508+0000 7f64a3213b80 1 --2- 172.21.15.27:0/939982586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40343a0 0x55a6a4059db0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6480014800 tx=0x7f6480015000 comp rx=0 tx=0).stop 2024-01-11T08:11:31.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.508+0000 7f64a3213b80 1 -- 172.21.15.27:0/939982586 shutdown_connections 2024-01-11T08:11:31.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.508+0000 7f64a3213b80 1 --2- 172.21.15.27:0/939982586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a4058ef0 0x55a6a4056d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.508+0000 7f64a3213b80 1 --2- 172.21.15.27:0/939982586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40343a0 0x55a6a4059db0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.508+0000 7f64a3213b80 1 --2- 172.21.15.27:0/939982586 >> [v2:172.21.15.27:6833/4056690675,v1:172.21.15.27:6835/4056690675] conn(0x55a6a4066130 0x55a6a4066500 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.508+0000 7f64a3213b80 1 --2- 172.21.15.27:0/939982586 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a4058b20 0x55a6a408dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.508+0000 7f646cff9700 0 client.5058 ms_handle_reset on v2:172.21.15.27:6833/4056690675 2024-01-11T08:11:31.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.508+0000 7f64a3213b80 1 -- 172.21.15.27:0/939982586 >> 172.21.15.27:0/939982586 conn(0x55a6a4036300 msgr2=0x55a6a4034b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:31.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.508+0000 7f64a3213b80 1 -- 172.21.15.27:0/939982586 shutdown_connections 2024-01-11T08:11:31.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.508+0000 7f64a3213b80 1 -- 172.21.15.27:0/939982586 wait complete. 2024-01-11T08:11:31.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.508+0000 7f64a3213b80 20 client.5058 trim_cache size 0 max 16384 2024-01-11T08:11:31.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.526+0000 7f64a3213b80 1 Processor -- start 2024-01-11T08:11:31.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.526+0000 7f64a3213b80 1 -- start start 2024-01-11T08:11:31.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.527+0000 7f64a3213b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40678c0 0x55a6a4097830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.527+0000 7f64a3213b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a4036300 0x55a6a4039e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.527+0000 7f64a3213b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a40343a0 0x55a6a40338c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.527+0000 7f64a3213b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6a3e1f1e0 con 0x55a6a40678c0 2024-01-11T08:11:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.527+0000 7f64a3213b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6a4098230 con 0x55a6a40343a0 2024-01-11T08:11:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.527+0000 7f64a3213b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a6a4036ca0 con 0x55a6a4036300 2024-01-11T08:11:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.527+0000 7f648e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40678c0 0x55a6a4097830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.527+0000 7f648e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40678c0 0x55a6a4097830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42546/0 (socket says 172.21.15.27:42546) 2024-01-11T08:11:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.527+0000 7f648e59c700 1 -- 172.21.15.27:0/373181160 learned_addr learned my addr 172.21.15.27:0/373181160 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.527+0000 7f648ed9d700 1 --2- 172.21.15.27:0/373181160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a40343a0 0x55a6a40338c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.527+0000 7f648dd9b700 1 --2- 172.21.15.27:0/373181160 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a4036300 0x55a6a4039e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.527+0000 7f648e59c700 1 -- 172.21.15.27:0/373181160 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a4036300 msgr2=0x55a6a4039e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.527+0000 7f648e59c700 1 --2- 172.21.15.27:0/373181160 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a4036300 0x55a6a4039e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.527+0000 7f648e59c700 1 -- 172.21.15.27:0/373181160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a40343a0 msgr2=0x55a6a40338c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.527+0000 7f648e59c700 1 --2- 172.21.15.27:0/373181160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a40343a0 0x55a6a40338c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.527+0000 7f648e59c700 1 -- 172.21.15.27:0/373181160 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6a3dee1d0 con 0x55a6a40678c0 2024-01-11T08:11:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.527+0000 7f648ed9d700 1 --2- 172.21.15.27:0/373181160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a40343a0 0x55a6a40338c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:31.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.527+0000 7f648dd9b700 1 --2- 172.21.15.27:0/373181160 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a4036300 0x55a6a4039e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:31.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.528+0000 7f648e59c700 1 --2- 172.21.15.27:0/373181160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40678c0 0x55a6a4097830 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f64800068e0 tx=0x7f6480019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e095c4b38f2208 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:31.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.528+0000 7f646ffff700 1 -- 172.21.15.27:0/373181160 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64800011f0 con 0x55a6a40678c0 2024-01-11T08:11:31.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.528+0000 7f646ffff700 1 -- 172.21.15.27:0/373181160 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6480013d30 con 0x55a6a40678c0 2024-01-11T08:11:31.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.528+0000 7f646ffff700 1 -- 172.21.15.27:0/373181160 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6480000c30 con 0x55a6a40678c0 2024-01-11T08:11:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.528+0000 7f64a3213b80 1 -- 172.21.15.27:0/373181160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40678c0 msgr2=0x55a6a4097830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.528+0000 7f64a3213b80 1 --2- 172.21.15.27:0/373181160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40678c0 0x55a6a4097830 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f64800068e0 tx=0x7f6480019000 comp rx=0 tx=0).stop 2024-01-11T08:11:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.528+0000 7f64a3213b80 1 -- 172.21.15.27:0/373181160 shutdown_connections 2024-01-11T08:11:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.528+0000 7f64a3213b80 1 --2- 172.21.15.27:0/373181160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a40343a0 0x55a6a40338c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.528+0000 7f64a3213b80 1 --2- 172.21.15.27:0/373181160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40678c0 0x55a6a4097830 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.528+0000 7f64a3213b80 1 --2- 172.21.15.27:0/373181160 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a4036300 0x55a6a4039e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.528+0000 7f64a3213b80 1 -- 172.21.15.27:0/373181160 >> 172.21.15.27:0/373181160 conn(0x55a6a405ffc0 msgr2=0x55a6a4034a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.529+0000 7f64a3213b80 1 -- 172.21.15.27:0/373181160 shutdown_connections 2024-01-11T08:11:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.529+0000 7f64a3213b80 1 -- 172.21.15.27:0/373181160 wait complete. 2024-01-11T08:11:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.529+0000 7f64a3213b80 1 -- start start 2024-01-11T08:11:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.529+0000 7f64a3213b80 1 Processor -- start 2024-01-11T08:11:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.530+0000 7f64a3213b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a40343a0 0x55a6a402dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.530+0000 7f64a3213b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a4036300 0x55a6a4062fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.530+0000 7f64a3213b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40678c0 0x55a6a40e2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.530+0000 7f64a3213b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6a4063700 con 0x55a6a40678c0 2024-01-11T08:11:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.530+0000 7f64a3213b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6a40e4a50 con 0x55a6a4036300 2024-01-11T08:11:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.530+0000 7f64a3213b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a6a402e170 con 0x55a6a40343a0 2024-01-11T08:11:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.531+0000 7f64a3213b80 10 client.0 _ll_register_callbacks cb 0x55a6a3d96a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T08:11:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.531+0000 7f648ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40678c0 0x55a6a40e2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.531+0000 7f648ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40678c0 0x55a6a40e2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42548/0 (socket says 172.21.15.27:42548) 2024-01-11T08:11:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.531+0000 7f648ed9d700 1 -- 172.21.15.27:0/2421729353 learned_addr learned my addr 172.21.15.27:0/2421729353 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.531+0000 7f648e59c700 1 --2- 172.21.15.27:0/2421729353 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a40343a0 0x55a6a402dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.531+0000 7f648dd9b700 1 --2- 172.21.15.27:0/2421729353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a4036300 0x55a6a4062fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.531+0000 7f648ed9d700 1 -- 172.21.15.27:0/2421729353 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a40343a0 msgr2=0x55a6a402dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.531+0000 7f648ed9d700 1 --2- 172.21.15.27:0/2421729353 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a40343a0 0x55a6a402dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.531+0000 7f648ed9d700 1 -- 172.21.15.27:0/2421729353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a4036300 msgr2=0x55a6a4062fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.531+0000 7f648ed9d700 1 --2- 172.21.15.27:0/2421729353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a4036300 0x55a6a4062fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.531+0000 7f648ed9d700 1 -- 172.21.15.27:0/2421729353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6a3f563c0 con 0x55a6a40678c0 2024-01-11T08:11:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.532+0000 7f648ed9d700 1 --2- 172.21.15.27:0/2421729353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40678c0 0x55a6a40e2860 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f64880094d0 tx=0x7f6488011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60aab9484e2b1236 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.532+0000 7f646cff9700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T08:11:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.532+0000 7f646cff9700 1 -- 172.21.15.27:0/2421729353 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6488055160 con 0x55a6a40678c0 2024-01-11T08:11:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.532+0000 7f646cff9700 1 -- 172.21.15.27:0/2421729353 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f64880552c0 con 0x55a6a40678c0 2024-01-11T08:11:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.532+0000 7f64a3213b80 1 -- 172.21.15.27:0/2421729353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a6a4061750 con 0x55a6a40678c0 2024-01-11T08:11:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.532+0000 7f646cff9700 1 -- 172.21.15.27:0/2421729353 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64880555d0 con 0x55a6a40678c0 2024-01-11T08:11:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.532+0000 7f64a3213b80 10 client.5064 fetch_fsmap learned FSMap version 17 2024-01-11T08:11:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.532+0000 7f64a3213b80 1 -- 172.21.15.27:0/2421729353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55a6a3dc7d60 con 0x55a6a40678c0 2024-01-11T08:11:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.532+0000 7f646cff9700 1 -- 172.21.15.27:0/2421729353 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6488060080 con 0x55a6a40678c0 2024-01-11T08:11:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.532+0000 7f646cff9700 1 -- 172.21.15.27:0/2421729353 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 17) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f6488051a60 con 0x55a6a40678c0 2024-01-11T08:11:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.532+0000 7f64a3213b80 10 client.5064 fetch_fsmap finished waiting for FSMap version 17 2024-01-11T08:11:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.532+0000 7f64a3213b80 10 client.5064 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-11T08:11:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.532+0000 7f64a3213b80 20 client.5064 populate_metadata read hostname 'smithi027' 2024-01-11T08:11:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.532+0000 7f64a3213b80 1 --2- 172.21.15.27:0/2421729353 >> [v2:172.21.15.27:6833/4056690675,v1:172.21.15.27:6835/4056690675] conn(0x55a6a4058b20 0x55a6a40970b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.532+0000 7f64a3213b80 4 client.5064 mds_command: new command op to 4870 tid=0[{"prefix": "status"}] 2024-01-11T08:11:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.532+0000 7f64a3213b80 1 -- 172.21.15.27:0/2421729353 --> [v2:172.21.15.27:6833/4056690675,v1:172.21.15.27:6835/4056690675] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f6484000bc0 con 0x55a6a4058b20 2024-01-11T08:11:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.533+0000 7f648e59c700 1 --2- 172.21.15.27:0/2421729353 >> [v2:172.21.15.27:6833/4056690675,v1:172.21.15.27:6835/4056690675] conn(0x55a6a4058b20 0x55a6a40970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.533+0000 7f646cff9700 10 client.5064 ms_handle_connect on v2:172.21.15.27:6833/4056690675 2024-01-11T08:11:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.533+0000 7f648e59c700 1 --2- 172.21.15.27:0/2421729353 >> [v2:172.21.15.27:6833/4056690675,v1:172.21.15.27:6835/4056690675] conn(0x55a6a4058b20 0x55a6a40970b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=775650a52a39c2ff server_cookie=46c1c041196424f7 in_seq=0 out_seq=0 2024-01-11T08:11:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.533+0000 7f646cff9700 1 -- 172.21.15.27:0/2421729353 <== mds.0 v2:172.21.15.27:6833/4056690675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x7f6484000bc0 con 0x55a6a4058b20 2024-01-11T08:11:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.533+0000 7f646cff9700 10 client.5064 handle_command_reply: tid=0 2024-01-11T08:11:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.533+0000 7f64a3213b80 1 client.5064 shutdown 2024-01-11T08:11:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.533+0000 7f64a3213b80 1 -- 172.21.15.27:0/2421729353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40678c0 msgr2=0x55a6a40e2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.533+0000 7f64a3213b80 1 --2- 172.21.15.27:0/2421729353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40678c0 0x55a6a40e2860 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f64880094d0 tx=0x7f6488011000 comp rx=0 tx=0).stop 2024-01-11T08:11:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.534+0000 7f64a3213b80 1 -- 172.21.15.27:0/2421729353 shutdown_connections 2024-01-11T08:11:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.534+0000 7f64a3213b80 1 --2- 172.21.15.27:0/2421729353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6a40678c0 0x55a6a40e2860 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.534+0000 7f64a3213b80 1 --2- 172.21.15.27:0/2421729353 >> [v2:172.21.15.27:6833/4056690675,v1:172.21.15.27:6835/4056690675] conn(0x55a6a4058b20 0x55a6a40970b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.534+0000 7f64a3213b80 1 --2- 172.21.15.27:0/2421729353 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55a6a40343a0 0x55a6a402dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.534+0000 7f64a3213b80 1 --2- 172.21.15.27:0/2421729353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6a4036300 0x55a6a4062fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.534+0000 7f646cff9700 0 client.5064 ms_handle_reset on v2:172.21.15.27:6833/4056690675 2024-01-11T08:11:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.534+0000 7f64a3213b80 1 -- 172.21.15.27:0/2421729353 >> 172.21.15.27:0/2421729353 conn(0x55a6a405ffc0 msgr2=0x55a6a4061010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.534+0000 7f64a3213b80 1 -- 172.21.15.27:0/2421729353 shutdown_connections 2024-01-11T08:11:31.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.534+0000 7f64a3213b80 1 -- 172.21.15.27:0/2421729353 wait complete. 2024-01-11T08:11:31.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.534+0000 7f64a3213b80 20 client.5064 trim_cache size 0 max 16384 2024-01-11T08:11:31.538 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T08:11:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "8fddb4a7-9d66-40f1-892d-6fe86c728ee4", 2024-01-11T08:11:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-11T08:11:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4870, 2024-01-11T08:11:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-11T08:11:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-11T08:11:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-11T08:11:31.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 6.6116588370000002, 2024-01-11T08:11:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 17, 2024-01-11T08:11:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 34, 2024-01-11T08:11:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 34, 2024-01-11T08:11:31.539 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 15.402198341 2024-01-11T08:11:31.539 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T08:11:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.534+0000 7f64949ff700 1 -- 172.21.15.27:0/2099694705 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6470066f30 msgr2=0x7f64700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.534+0000 7f64949ff700 1 --2- 172.21.15.27:0/2099694705 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6470066f30 0x7f64700693e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6480007fd0 tx=0x7f6480009040 comp rx=0 tx=0).stop 2024-01-11T08:11:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.535+0000 7f64949ff700 1 -- 172.21.15.27:0/2099694705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f649015df40 msgr2=0x7f64901528f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.535+0000 7f64949ff700 1 --2- 172.21.15.27:0/2099694705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f649015df40 0x7f64901528f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6488009df0 tx=0x7f648800b040 comp rx=0 tx=0).stop 2024-01-11T08:11:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.535+0000 7f64949ff700 1 -- 172.21.15.27:0/2099694705 shutdown_connections 2024-01-11T08:11:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.535+0000 7f64949ff700 1 --2- 172.21.15.27:0/2099694705 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6470066f30 0x7f64700693e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.535+0000 7f64949ff700 1 --2- 172.21.15.27:0/2099694705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f649015df40 0x7f64901528f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.535+0000 7f64949ff700 1 --2- 172.21.15.27:0/2099694705 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f649006bf10 0x7f64900698b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.535+0000 7f64949ff700 1 --2- 172.21.15.27:0/2099694705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64900b0a40 0x7f64901523b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.535+0000 7f64949ff700 1 -- 172.21.15.27:0/2099694705 >> 172.21.15.27:0/2099694705 conn(0x7f64900ce3c0 msgr2=0x7f649015c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:31.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.535+0000 7f64949ff700 1 -- 172.21.15.27:0/2099694705 shutdown_connections 2024-01-11T08:11:31.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.535+0000 7f64949ff700 1 -- 172.21.15.27:0/2099694705 wait complete. 2024-01-11T08:11:31.552 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-11T08:11:31.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.751+0000 7f9ed9dff700 1 Processor -- start 2024-01-11T08:11:31.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.751+0000 7f9ed9dff700 1 -- start start 2024-01-11T08:11:31.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.752+0000 7f9ed9dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ed40c8620 0x7f9ed40af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.752+0000 7f9ed9dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9ed4069d40 0x7f9ed40b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.753+0000 7f9ed9dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9ed40c89f0 0x7f9ed40b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.753+0000 7f9ed9dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed4157390 con 0x7f9ed40c89f0 2024-01-11T08:11:31.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.753+0000 7f9ed9dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed40afeb0 con 0x7f9ed4069d40 2024-01-11T08:11:31.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.753+0000 7f9ed9dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed406a200 con 0x7f9ed40c8620 2024-01-11T08:11:31.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.753+0000 7f9ed3fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9ed40c89f0 0x7f9ed40b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.753+0000 7f9ed2ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9ed4069d40 0x7f9ed40b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.753+0000 7f9ed37fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ed40c8620 0x7f9ed40af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.753+0000 7f9ed2ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9ed4069d40 0x7f9ed40b02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:51860/0 (socket says 172.21.15.27:51860) 2024-01-11T08:11:31.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.753+0000 7f9ed2ffd700 1 -- 172.21.15.27:0/3296841696 learned_addr learned my addr 172.21.15.27:0/3296841696 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:31.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.753+0000 7f9ed3fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9ed40c89f0 0x7f9ed40b0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42558/0 (socket says 172.21.15.27:42558) 2024-01-11T08:11:31.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.753+0000 7f9ed3fff700 1 -- 172.21.15.27:0/3296841696 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ed40c8620 msgr2=0x7f9ed40af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:31.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.753+0000 7f9ed3fff700 1 --2- 172.21.15.27:0/3296841696 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ed40c8620 0x7f9ed40af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.753+0000 7f9ed3fff700 1 -- 172.21.15.27:0/3296841696 >> v2:172.21.15.77:3300/0 conn(0x7f9ed4069d40 msgr2=0x7f9ed40b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:31.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.753+0000 7f9ed3fff700 1 --2- 172.21.15.27:0/3296841696 >> v2:172.21.15.77:3300/0 conn(0x7f9ed4069d40 0x7f9ed40b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.753+0000 7f9ed3fff700 1 -- 172.21.15.27:0/3296841696 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed4156d40 con 0x7f9ed40c89f0 2024-01-11T08:11:31.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.753+0000 7f9ed2ffd700 1 --2- 172.21.15.27:0/3296841696 >> v2:172.21.15.77:3300/0 conn(0x7f9ed4069d40 0x7f9ed40b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:31.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.754+0000 7f9ed3fff700 1 --2- 172.21.15.27:0/3296841696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ed40c89f0 0x7f9ed40b0810 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4069180 tx=0x7f9ecc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7433ca1a9765ba99 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:31.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.754+0000 7f9ed27fc700 1 -- 172.21.15.27:0/3296841696 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc00e070 con 0x7f9ed40c89f0 2024-01-11T08:11:31.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.754+0000 7f9ed27fc700 1 -- 172.21.15.27:0/3296841696 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ecc00bab0 con 0x7f9ed40c89f0 2024-01-11T08:11:31.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.754+0000 7f9ed27fc700 1 -- 172.21.15.27:0/3296841696 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc00be70 con 0x7f9ed40c89f0 2024-01-11T08:11:31.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.754+0000 7f9ed9dff700 1 -- 172.21.15.27:0/3296841696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ed40c89f0 msgr2=0x7f9ed40b0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:31.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.754+0000 7f9ed9dff700 1 --2- 172.21.15.27:0/3296841696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ed40c89f0 0x7f9ed40b0810 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4069180 tx=0x7f9ecc009780 comp rx=0 tx=0).stop 2024-01-11T08:11:31.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.755+0000 7f9ed9dff700 1 -- 172.21.15.27:0/3296841696 shutdown_connections 2024-01-11T08:11:31.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.755+0000 7f9ed9dff700 1 --2- 172.21.15.27:0/3296841696 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ed40c8620 0x7f9ed40af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.755+0000 7f9ed9dff700 1 --2- 172.21.15.27:0/3296841696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ed40c89f0 0x7f9ed40b0810 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.755+0000 7f9ed9dff700 1 --2- 172.21.15.27:0/3296841696 >> v2:172.21.15.77:3300/0 conn(0x7f9ed4069d40 0x7f9ed40b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.755+0000 7f9ed9dff700 1 -- 172.21.15.27:0/3296841696 >> 172.21.15.27:0/3296841696 conn(0x7f9ed40680e0 msgr2=0x7f9ed4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:31.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.755+0000 7f9ed9dff700 1 -- 172.21.15.27:0/3296841696 shutdown_connections 2024-01-11T08:11:31.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.755+0000 7f9ed9dff700 1 -- 172.21.15.27:0/3296841696 wait complete. 2024-01-11T08:11:31.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.756+0000 7f9ed9dff700 1 Processor -- start 2024-01-11T08:11:31.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.756+0000 7f9ed9dff700 1 -- start start 2024-01-11T08:11:31.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.757+0000 7f9ed9dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ed4069d40 0x7f9ed4143830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.757+0000 7f9ed9dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ed40c8620 0x7f9ed40cbb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.757+0000 7f9ed9dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ed40c89f0 0x7f9ed40cc090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.757+0000 7f9ed9dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed40c9f10 con 0x7f9ed40c8620 2024-01-11T08:11:31.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.757+0000 7f9ed9dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed4143d70 con 0x7f9ed40c89f0 2024-01-11T08:11:31.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.757+0000 7f9ed9dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed40c9300 con 0x7f9ed4069d40 2024-01-11T08:11:31.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.758+0000 7f9ed2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ed40c8620 0x7f9ed40cbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.758+0000 7f9ed37fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ed4069d40 0x7f9ed4143830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.758+0000 7f9ed2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ed40c8620 0x7f9ed40cbb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42566/0 (socket says 172.21.15.27:42566) 2024-01-11T08:11:31.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.758+0000 7f9ed2ffd700 1 -- 172.21.15.27:0/1905370355 learned_addr learned my addr 172.21.15.27:0/1905370355 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:31.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.758+0000 7f9ed3fff700 1 --2- 172.21.15.27:0/1905370355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ed40c89f0 0x7f9ed40cc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.758+0000 7f9ed37fe700 1 -- 172.21.15.27:0/1905370355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ed40c89f0 msgr2=0x7f9ed40cc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:31.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.758+0000 7f9ed37fe700 1 --2- 172.21.15.27:0/1905370355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ed40c89f0 0x7f9ed40cc090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.758+0000 7f9ed37fe700 1 -- 172.21.15.27:0/1905370355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ed40c8620 msgr2=0x7f9ed40cbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:31.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.758+0000 7f9ed37fe700 1 --2- 172.21.15.27:0/1905370355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ed40c8620 0x7f9ed40cbb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:31.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.758+0000 7f9ed37fe700 1 -- 172.21.15.27:0/1905370355 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec80094f0 con 0x7f9ed4069d40 2024-01-11T08:11:31.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.758+0000 7f9ed2ffd700 1 --2- 172.21.15.27:0/1905370355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ed40c8620 0x7f9ed40cbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:31.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.759+0000 7f9ed37fe700 1 --2- 172.21.15.27:0/1905370355 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ed4069d40 0x7f9ed4143830 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9ec400ede0 tx=0x7f9ec400b430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:31.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.759+0000 7f9ebb7fe700 1 -- 172.21.15.27:0/1905370355 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec400dbd0 con 0x7f9ed4069d40 2024-01-11T08:11:31.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.759+0000 7f9ebb7fe700 1 -- 172.21.15.27:0/1905370355 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec400dd30 con 0x7f9ed4069d40 2024-01-11T08:11:31.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.759+0000 7f9ed9dff700 1 -- 172.21.15.27:0/1905370355 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ecc009a20 con 0x7f9ed4069d40 2024-01-11T08:11:31.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.759+0000 7f9ebb7fe700 1 -- 172.21.15.27:0/1905370355 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec400dea0 con 0x7f9ed4069d40 2024-01-11T08:11:31.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.759+0000 7f9ed9dff700 1 -- 172.21.15.27:0/1905370355 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed40c98c0 con 0x7f9ed4069d40 2024-01-11T08:11:31.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.760+0000 7f9ed9dff700 1 -- 172.21.15.27:0/1905370355 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed40718d0 con 0x7f9ed4069d40 2024-01-11T08:11:31.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.761+0000 7f9ebb7fe700 1 -- 172.21.15.27:0/1905370355 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9ec40108a0 con 0x7f9ed4069d40 2024-01-11T08:11:31.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.762+0000 7f9ebb7fe700 1 --2- 172.21.15.27:0/1905370355 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9ec0066f30 0x7f9ec00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:31.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.762+0000 7f9ebb7fe700 1 -- 172.21.15.27:0/1905370355 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f9ec4084800 con 0x7f9ed4069d40 2024-01-11T08:11:31.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.763+0000 7f9ed2ffd700 1 --2- 172.21.15.27:0/1905370355 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9ec0066f30 0x7f9ec00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:31.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.763+0000 7f9ed2ffd700 1 --2- 172.21.15.27:0/1905370355 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9ec0066f30 0x7f9ec00693e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9ed40cd150 tx=0x7f9ec8009250 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:31.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:31.767+0000 7f9ebb7fe700 1 -- 172.21.15.27:0/1905370355 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9ec4051a90 con 0x7f9ed4069d40 2024-01-11T08:11:32.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:32.019+0000 7f9ed9dff700 1 -- 172.21.15.27:0/1905370355 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9ed4157020 con 0x7f9ed4069d40 2024-01-11T08:11:32.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:32.020+0000 7f9ebb7fe700 1 -- 172.21.15.27:0/1905370355 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9ec40550b0 con 0x7f9ed4069d40 2024-01-11T08:11:32.019 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-11T08:11:32.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:32.022+0000 7f9ed9dff700 1 -- 172.21.15.27:0/1905370355 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9ec0066f30 msgr2=0x7f9ec00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:32.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:32.022+0000 7f9ed9dff700 1 --2- 172.21.15.27:0/1905370355 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9ec0066f30 0x7f9ec00693e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9ed40cd150 tx=0x7f9ec8009250 comp rx=0 tx=0).stop 2024-01-11T08:11:32.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:32.022+0000 7f9ed9dff700 1 -- 172.21.15.27:0/1905370355 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ed4069d40 msgr2=0x7f9ed4143830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:32.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:32.022+0000 7f9ed9dff700 1 --2- 172.21.15.27:0/1905370355 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ed4069d40 0x7f9ed4143830 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9ec400ede0 tx=0x7f9ec400b430 comp rx=0 tx=0).stop 2024-01-11T08:11:32.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:32.022+0000 7f9ed9dff700 1 -- 172.21.15.27:0/1905370355 shutdown_connections 2024-01-11T08:11:32.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:32.022+0000 7f9ed9dff700 1 --2- 172.21.15.27:0/1905370355 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9ec0066f30 0x7f9ec00693e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:32.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:32.022+0000 7f9ed9dff700 1 --2- 172.21.15.27:0/1905370355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ed40c89f0 0x7f9ed40cc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:32.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:32.022+0000 7f9ed9dff700 1 --2- 172.21.15.27:0/1905370355 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ed4069d40 0x7f9ed4143830 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:32.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:32.022+0000 7f9ed9dff700 1 --2- 172.21.15.27:0/1905370355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ed40c8620 0x7f9ed40cbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:32.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:32.022+0000 7f9ed9dff700 1 -- 172.21.15.27:0/1905370355 >> 172.21.15.27:0/1905370355 conn(0x7f9ed40680e0 msgr2=0x7f9ed4156600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:32.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:32.022+0000 7f9ed9dff700 1 -- 172.21.15.27:0/1905370355 shutdown_connections 2024-01-11T08:11:32.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:32.022+0000 7f9ed9dff700 1 -- 172.21.15.27:0/1905370355 wait complete. 2024-01-11T08:11:32.040 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-11T08:11:32.040 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-11T08:11:32.040 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-11T08:11:32.040 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-11T08:11:32.040 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-11T08:11:32.040 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-11T08:11:32.040 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-11T08:11:32.040 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-11T08:11:32.040 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:11:32.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:11:32.041 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-11T08:11:32.089 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T08:11:32.089 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T08:11:32.089 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T08:11:32.089 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:11:32.090 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-11T08:11:32.090 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:11:32.090 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:11:32.090 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:11:32.090 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-11T08:11:32.090 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:11:32.090 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:11:32.090 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-11T08:11:32.090 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-11T08:11:32.090 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-11T08:11:32.090 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-11T08:11:32.090 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-11T08:11:32.090 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42430sec preferred_lft 42430sec 2024-01-11T08:11:32.090 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-11T08:11:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:11:32.091 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:11:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-11T08:11:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-11T08:11:32.091 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T08:11:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff 2024-01-11T08:11:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-11T08:11:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:11:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::acd2:b3ff:fe6a:db4b/64 scope link 2024-01-11T08:11:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:11:32.091 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T08:11:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:fa:85:1b:3a:bc brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-11T08:11:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ecfa:85ff:fe1b:3abc/64 scope link 2024-01-11T08:11:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:11:32.091 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T08:11:32.091 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-11T08:11:32.092 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ec88:faff:fe17:41aa/64 scope link 2024-01-11T08:11:32.092 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:11:32.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:11:32.092 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-11T08:11:32.162 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-11T08:11:32.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:11:32.166 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-11T08:11:32.217 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-11T08:11:32.217 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-11T08:11:32.217 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-11T08:11:32.217 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-11T08:11:32.217 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-11T08:11:32.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:11:32.219 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T08:11:32.219 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T08:11:32.219 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-11T08:11:32.219 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-11T08:11:32.219 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-11T08:11:32.219 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T08:11:32.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:11:32.395 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T08:11:32.442 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-11T08:11:32.442 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-11T08:11:32.443 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:11:32.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:11:32.443 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-11T08:11:32.520 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:11:32.520 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-11T08:11:32.567 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:11:32.568 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:11:32.568 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-11T08:11:32.615 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-11T08:11:32.615 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-11T08:11:32.615 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:11:32.758 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:11:32.758 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:11:32.758 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:11:32.773 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:11:32.774 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:11:32.774 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:11:32.774 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:11:32.774 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:11:32.796 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:11:32.796 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:11:32.796 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:11:32.796 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:11:32.797 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:11:32.797 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790" 2024-01-11T08:11:32.797 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790:/ 2024-01-11T08:11:32.797 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=8fddb4a7-9d66-40f1-892d-6fe86c728ee4" will pass to kernel 2024-01-11T08:11:32.801 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-11T08:11:32.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:11:32.801 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-11T08:11:32.874 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-11T08:11:33.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.075+0000 7fe1c6bff700 1 Processor -- start 2024-01-11T08:11:33.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.075+0000 7fe1c6bff700 1 -- start start 2024-01-11T08:11:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.076+0000 7fe1c6bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1c01596a0 0x7fe1c00abd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.076+0000 7fe1c6bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe1c0067f90 0x7fe1c0068410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.076+0000 7fe1c6bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c00ac250 con 0x7fe1c0068ba0 2024-01-11T08:11:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.076+0000 7fe1c6bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c0068950 con 0x7fe1c0067f90 2024-01-11T08:11:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.076+0000 7fe1c6bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c0067b00 con 0x7fe1c01596a0 2024-01-11T08:11:33.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.077+0000 7fe1c519c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe1c0067f90 0x7fe1c0068410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:33.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.077+0000 7fe1c499b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1c01596a0 0x7fe1c00abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:33.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.077+0000 7fe1c499b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1c01596a0 0x7fe1c00abd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:42532/0 (socket says 172.21.15.27:42532) 2024-01-11T08:11:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.077+0000 7fe1c499b700 1 -- 172.21.15.27:0/4199062009 learned_addr learned my addr 172.21.15.27:0/4199062009 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.077+0000 7fe1c519c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe1c0067f90 0x7fe1c0068410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:51880/0 (socket says 172.21.15.27:51880) 2024-01-11T08:11:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.077+0000 7fe1c499b700 1 -- 172.21.15.27:0/4199062009 >> v2:172.21.15.77:3300/0 conn(0x7fe1c0067f90 msgr2=0x7fe1c0068410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.077+0000 7fe1c499b700 1 --2- 172.21.15.27:0/4199062009 >> v2:172.21.15.77:3300/0 conn(0x7fe1c0067f90 0x7fe1c0068410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.077+0000 7fe1c499b700 1 -- 172.21.15.27:0/4199062009 >> v1:172.21.15.77:6789/0 conn(0x7fe1c0068ba0 legacy=0x7fe1c00af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.077+0000 7fe1c499b700 1 -- 172.21.15.27:0/4199062009 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c006bf60 con 0x7fe1c01596a0 2024-01-11T08:11:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.078+0000 7fe1c499b700 1 --2- 172.21.15.27:0/4199062009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1c01596a0 0x7fe1c00abd10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe1b40099b0 tx=0x7fe1b4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=422dbe5b09a2e1c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.078+0000 7fe1bf7fe700 1 -- 172.21.15.27:0/4199062009 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b4009ea0 con 0x7fe1c01596a0 2024-01-11T08:11:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.078+0000 7fe1bf7fe700 1 -- 172.21.15.27:0/4199062009 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b400b860 con 0x7fe1c01596a0 2024-01-11T08:11:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.078+0000 7fe1bf7fe700 1 -- 172.21.15.27:0/4199062009 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b400bc00 con 0x7fe1c01596a0 2024-01-11T08:11:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.078+0000 7fe1c6bff700 1 -- 172.21.15.27:0/4199062009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1c01596a0 msgr2=0x7fe1c00abd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.078+0000 7fe1c6bff700 1 --2- 172.21.15.27:0/4199062009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1c01596a0 0x7fe1c00abd10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe1b40099b0 tx=0x7fe1b4009250 comp rx=0 tx=0).stop 2024-01-11T08:11:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.078+0000 7fe1c6bff700 1 -- 172.21.15.27:0/4199062009 shutdown_connections 2024-01-11T08:11:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.078+0000 7fe1c6bff700 1 --2- 172.21.15.27:0/4199062009 >> v2:172.21.15.77:3300/0 conn(0x7fe1c0067f90 0x7fe1c0068410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.078+0000 7fe1c6bff700 1 --2- 172.21.15.27:0/4199062009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1c01596a0 0x7fe1c00abd10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.079+0000 7fe1c6bff700 1 -- 172.21.15.27:0/4199062009 >> 172.21.15.27:0/4199062009 conn(0x7fe1c006a8b0 msgr2=0x7fe1c00ae520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.079+0000 7fe1c6bff700 1 -- 172.21.15.27:0/4199062009 shutdown_connections 2024-01-11T08:11:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.079+0000 7fe1c6bff700 1 -- 172.21.15.27:0/4199062009 wait complete. 2024-01-11T08:11:33.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.080+0000 7fe1c6bff700 1 Processor -- start 2024-01-11T08:11:33.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.080+0000 7fe1c6bff700 1 -- start start 2024-01-11T08:11:33.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.081+0000 7fe1c6bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1c0067f90 0x7fe1c016cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:33.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.081+0000 7fe1c6bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1c0068ba0 0x7fe1c0151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:33.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.081+0000 7fe1c6bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1c01596a0 0x7fe1c0151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:33.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.081+0000 7fe1c6bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c00b0e20 con 0x7fe1c0067f90 2024-01-11T08:11:33.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.081+0000 7fe1c6bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c00b1100 con 0x7fe1c0068ba0 2024-01-11T08:11:33.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.081+0000 7fe1c6bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c00b13e0 con 0x7fe1c01596a0 2024-01-11T08:11:33.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.082+0000 7fe1c499b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1c0067f90 0x7fe1c016cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:33.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.082+0000 7fe1c499b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1c0067f90 0x7fe1c016cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42580/0 (socket says 172.21.15.27:42580) 2024-01-11T08:11:33.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.082+0000 7fe1c499b700 1 -- 172.21.15.27:0/295672772 learned_addr learned my addr 172.21.15.27:0/295672772 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:33.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.082+0000 7fe1c519c700 1 --2- 172.21.15.27:0/295672772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1c01596a0 0x7fe1c0151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:33.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.082+0000 7fe1bffff700 1 --2- 172.21.15.27:0/295672772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1c0068ba0 0x7fe1c0151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:33.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.082+0000 7fe1c499b700 1 -- 172.21.15.27:0/295672772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1c01596a0 msgr2=0x7fe1c0151670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:33.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.082+0000 7fe1c499b700 1 --2- 172.21.15.27:0/295672772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1c01596a0 0x7fe1c0151670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:33.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.082+0000 7fe1c499b700 1 -- 172.21.15.27:0/295672772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1c0068ba0 msgr2=0x7fe1c0151130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.082+0000 7fe1c499b700 1 --2- 172.21.15.27:0/295672772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1c0068ba0 0x7fe1c0151130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.082+0000 7fe1c499b700 1 -- 172.21.15.27:0/295672772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b40094f0 con 0x7fe1c0067f90 2024-01-11T08:11:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.082+0000 7fe1bffff700 1 --2- 172.21.15.27:0/295672772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1c0068ba0 0x7fe1c0151130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.082+0000 7fe1c499b700 1 --2- 172.21.15.27:0/295672772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1c0067f90 0x7fe1c016cb90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0067c40 tx=0x7fe1b4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.083+0000 7fe1bcff9700 1 -- 172.21.15.27:0/295672772 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b400bbe0 con 0x7fe1c0067f90 2024-01-11T08:11:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.083+0000 7fe1bcff9700 1 -- 172.21.15.27:0/295672772 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b4011420 con 0x7fe1c0067f90 2024-01-11T08:11:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.083+0000 7fe1bcff9700 1 -- 172.21.15.27:0/295672772 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b40117e0 con 0x7fe1c0067f90 2024-01-11T08:11:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.083+0000 7fe1c6bff700 1 -- 172.21.15.27:0/295672772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c00b16c0 con 0x7fe1c0067f90 2024-01-11T08:11:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.083+0000 7fe1c6bff700 1 -- 172.21.15.27:0/295672772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c016d0d0 con 0x7fe1c0067f90 2024-01-11T08:11:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.084+0000 7fe1c6bff700 1 -- 172.21.15.27:0/295672772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c004c3c0 con 0x7fe1c0067f90 2024-01-11T08:11:33.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.085+0000 7fe1bcff9700 1 -- 172.21.15.27:0/295672772 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe1b4011b50 con 0x7fe1c0067f90 2024-01-11T08:11:33.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.085+0000 7fe1bcff9700 1 --2- 172.21.15.27:0/295672772 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe1a4066f30 0x7fe1a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:33.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.086+0000 7fe1bcff9700 1 -- 172.21.15.27:0/295672772 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fe1b4085d00 con 0x7fe1c0067f90 2024-01-11T08:11:33.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.086+0000 7fe1bffff700 1 --2- 172.21.15.27:0/295672772 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe1a4066f30 0x7fe1a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:33.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.087+0000 7fe1bffff700 1 --2- 172.21.15.27:0/295672772 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe1a4066f30 0x7fe1a40693e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac007fd0 tx=0x7fe1ac009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:33.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.090+0000 7fe1bcff9700 1 -- 172.21.15.27:0/295672772 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe1b4052ed0 con 0x7fe1c0067f90 2024-01-11T08:11:33.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.320+0000 7fe1c6bff700 1 -- 172.21.15.27:0/295672772 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1c00b1270 con 0x7fe1a4066f30 2024-01-11T08:11:33.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.322+0000 7fe1bcff9700 1 -- 172.21.15.27:0/295672772 <== mgr.4099 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe1c00b1270 con 0x7fe1a4066f30 2024-01-11T08:11:33.322 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T08:11:33.322 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:11:33.322 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-11T08:11:33.322 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:11:33.322 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T08:11:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.324+0000 7fe1c6bff700 1 -- 172.21.15.27:0/295672772 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe1a4066f30 msgr2=0x7fe1a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.324+0000 7fe1c6bff700 1 --2- 172.21.15.27:0/295672772 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe1a4066f30 0x7fe1a40693e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac007fd0 tx=0x7fe1ac009040 comp rx=0 tx=0).stop 2024-01-11T08:11:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.324+0000 7fe1c6bff700 1 -- 172.21.15.27:0/295672772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1c0067f90 msgr2=0x7fe1c016cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.324+0000 7fe1c6bff700 1 --2- 172.21.15.27:0/295672772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1c0067f90 0x7fe1c016cb90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0067c40 tx=0x7fe1b4017000 comp rx=0 tx=0).stop 2024-01-11T08:11:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.325+0000 7fe1c6bff700 1 -- 172.21.15.27:0/295672772 shutdown_connections 2024-01-11T08:11:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.325+0000 7fe1c6bff700 1 --2- 172.21.15.27:0/295672772 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe1a4066f30 0x7fe1a40693e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.325+0000 7fe1c6bff700 1 --2- 172.21.15.27:0/295672772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1c01596a0 0x7fe1c0151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.325+0000 7fe1c6bff700 1 --2- 172.21.15.27:0/295672772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1c0067f90 0x7fe1c016cb90 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.325+0000 7fe1c6bff700 1 --2- 172.21.15.27:0/295672772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1c0068ba0 0x7fe1c0151130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.325+0000 7fe1c6bff700 1 -- 172.21.15.27:0/295672772 >> 172.21.15.27:0/295672772 conn(0x7fe1c006a8b0 msgr2=0x7fe1c00ac4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.325+0000 7fe1c6bff700 1 -- 172.21.15.27:0/295672772 shutdown_connections 2024-01-11T08:11:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.325+0000 7fe1c6bff700 1 -- 172.21.15.27:0/295672772 wait complete. 2024-01-11T08:11:33.342 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-11T08:11:33.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.543+0000 7f3a3ebff700 1 Processor -- start 2024-01-11T08:11:33.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.543+0000 7f3a3ebff700 1 -- start start 2024-01-11T08:11:33.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.544+0000 7f3a3ebff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3a3813ff80 0x7f3a3815a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:33.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.544+0000 7f3a3ebff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a380c6b30 con 0x7f3a3813fbb0 2024-01-11T08:11:33.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.545+0000 7f3a3ebff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a380c6800 con 0x7f3a3815a560 2024-01-11T08:11:33.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.545+0000 7f3a3ebff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a3815a960 con 0x7f3a3813ff80 2024-01-11T08:11:33.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.545+0000 7f3a3c99b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3a3813fbb0 0x7f3a380ca7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50746/0 (socket says 172.21.15.27:50746) 2024-01-11T08:11:33.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.545+0000 7f3a3c99b700 1 -- 172.21.15.27:0/96102100 learned_addr learned my addr 172.21.15.27:0/96102100 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:33.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.545+0000 7f3a37fff700 1 --2- 172.21.15.27:0/96102100 >> v2:172.21.15.77:3300/0 conn(0x7f3a3813ff80 0x7f3a3815a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:33.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.545+0000 7f3a377fe700 1 -- 172.21.15.27:0/96102100 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1829066465 0 0) 0x7f3a380c6b30 con 0x7f3a3813fbb0 2024-01-11T08:11:33.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.545+0000 7f3a377fe700 1 -- 172.21.15.27:0/96102100 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a28003570 con 0x7f3a3813fbb0 2024-01-11T08:11:33.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.545+0000 7f3a37fff700 1 -- 172.21.15.27:0/96102100 >> v1:172.21.15.77:6789/0 conn(0x7f3a3815a560 legacy=0x7f3a380c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:33.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.545+0000 7f3a37fff700 1 -- 172.21.15.27:0/96102100 >> v1:172.21.15.27:6789/0 conn(0x7f3a3813fbb0 legacy=0x7f3a380ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:33.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.545+0000 7f3a37fff700 1 -- 172.21.15.27:0/96102100 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a380cb240 con 0x7f3a3813ff80 2024-01-11T08:11:33.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.546+0000 7f3a37fff700 1 --2- 172.21.15.27:0/96102100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a3813ff80 0x7f3a3815a020 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3a240098e0 tx=0x7f3a240091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82b98e183d5bac92 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:33.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.546+0000 7f3a377fe700 1 -- 172.21.15.27:0/96102100 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2400b640 con 0x7f3a3813ff80 2024-01-11T08:11:33.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.546+0000 7f3a377fe700 1 -- 172.21.15.27:0/96102100 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a2400b7a0 con 0x7f3a3813ff80 2024-01-11T08:11:33.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.546+0000 7f3a377fe700 1 -- 172.21.15.27:0/96102100 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2400bb20 con 0x7f3a3813ff80 2024-01-11T08:11:33.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.546+0000 7f3a3ebff700 1 -- 172.21.15.27:0/96102100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a3813ff80 msgr2=0x7f3a3815a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:33.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.547+0000 7f3a3ebff700 1 --2- 172.21.15.27:0/96102100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a3813ff80 0x7f3a3815a020 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3a240098e0 tx=0x7f3a240091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:33.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.547+0000 7f3a3ebff700 1 -- 172.21.15.27:0/96102100 shutdown_connections 2024-01-11T08:11:33.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.547+0000 7f3a3ebff700 1 --2- 172.21.15.27:0/96102100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a3813ff80 0x7f3a3815a020 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:33.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.547+0000 7f3a3ebff700 1 -- 172.21.15.27:0/96102100 >> 172.21.15.27:0/96102100 conn(0x7f3a380680a0 msgr2=0x7f3a380cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:33.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.547+0000 7f3a3ebff700 1 -- 172.21.15.27:0/96102100 shutdown_connections 2024-01-11T08:11:33.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.547+0000 7f3a3ebff700 1 -- 172.21.15.27:0/96102100 wait complete. 2024-01-11T08:11:33.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.548+0000 7f3a3ebff700 1 Processor -- start 2024-01-11T08:11:33.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.548+0000 7f3a3ebff700 1 -- start start 2024-01-11T08:11:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.549+0000 7f3a3ebff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3813fbb0 0x7f3a38155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:33.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.549+0000 7f3a3ebff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3a3813ff80 0x7f3a380c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:33.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.549+0000 7f3a3ebff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a3815a560 0x7f3a380c7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:33.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.549+0000 7f3a3ebff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a381598e0 con 0x7f3a3813fbb0 2024-01-11T08:11:33.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.549+0000 7f3a3ebff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a38159bc0 con 0x7f3a3815a560 2024-01-11T08:11:33.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.549+0000 7f3a3ebff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a38159ea0 con 0x7f3a3813ff80 2024-01-11T08:11:33.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.550+0000 7f3a3c99b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3813fbb0 0x7f3a38155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:33.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.550+0000 7f3a3c99b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3813fbb0 0x7f3a38155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42592/0 (socket says 172.21.15.27:42592) 2024-01-11T08:11:33.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.550+0000 7f3a3c99b700 1 -- 172.21.15.27:0/3078785899 learned_addr learned my addr 172.21.15.27:0/3078785899 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:33.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.550+0000 7f3a37fff700 1 --2- 172.21.15.27:0/3078785899 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3a3813ff80 0x7f3a380c6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:33.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.550+0000 7f3a3d19c700 1 --2- 172.21.15.27:0/3078785899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a3815a560 0x7f3a380c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:33.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.550+0000 7f3a3c99b700 1 -- 172.21.15.27:0/3078785899 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3a3813ff80 msgr2=0x7f3a380c6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:33.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.550+0000 7f3a3c99b700 1 --2- 172.21.15.27:0/3078785899 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3a3813ff80 0x7f3a380c6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:33.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.550+0000 7f3a3c99b700 1 -- 172.21.15.27:0/3078785899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a3815a560 msgr2=0x7f3a380c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:33.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.550+0000 7f3a3c99b700 1 --2- 172.21.15.27:0/3078785899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a3815a560 0x7f3a380c7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:33.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.550+0000 7f3a3c99b700 1 -- 172.21.15.27:0/3078785899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a24009420 con 0x7f3a3813fbb0 2024-01-11T08:11:33.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.550+0000 7f3a3d19c700 1 --2- 172.21.15.27:0/3078785899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a3815a560 0x7f3a380c7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:33.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.550+0000 7f3a3c99b700 1 --2- 172.21.15.27:0/3078785899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3813fbb0 0x7f3a38155c60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c00c790 tx=0x7f3a2c007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:33.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.550+0000 7f3a34ff9700 1 -- 172.21.15.27:0/3078785899 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c0072e0 con 0x7f3a3813fbb0 2024-01-11T08:11:33.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.551+0000 7f3a34ff9700 1 -- 172.21.15.27:0/3078785899 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a2c007440 con 0x7f3a3813fbb0 2024-01-11T08:11:33.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.551+0000 7f3a34ff9700 1 -- 172.21.15.27:0/3078785899 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c0057f0 con 0x7f3a3813fbb0 2024-01-11T08:11:33.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.551+0000 7f3a3ebff700 1 -- 172.21.15.27:0/3078785899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a3815a180 con 0x7f3a3813fbb0 2024-01-11T08:11:33.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.551+0000 7f3a3ebff700 1 -- 172.21.15.27:0/3078785899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a38158bd0 con 0x7f3a3813fbb0 2024-01-11T08:11:33.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.552+0000 7f3a3ebff700 1 -- 172.21.15.27:0/3078785899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a380c38b0 con 0x7f3a3813fbb0 2024-01-11T08:11:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.553+0000 7f3a34ff9700 1 -- 172.21.15.27:0/3078785899 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3a2c005c20 con 0x7f3a3813fbb0 2024-01-11T08:11:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.554+0000 7f3a34ff9700 1 --2- 172.21.15.27:0/3078785899 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a1c066f30 0x7f3a1c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.554+0000 7f3a34ff9700 1 -- 172.21.15.27:0/3078785899 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f3a2c083990 con 0x7f3a3813fbb0 2024-01-11T08:11:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.554+0000 7f3a37fff700 1 --2- 172.21.15.27:0/3078785899 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a1c066f30 0x7f3a1c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.555+0000 7f3a37fff700 1 --2- 172.21.15.27:0/3078785899 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a1c066f30 0x7f3a1c0693e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3a240091e0 tx=0x7f3a24017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:33.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.558+0000 7f3a34ff9700 1 -- 172.21.15.27:0/3078785899 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3a2c04b5a0 con 0x7f3a3813fbb0 2024-01-11T08:11:33.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:33.821+0000 7f3a3ebff700 1 -- 172.21.15.27:0/3078785899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f3a3804c3c0 con 0x7f3a3813fbb0 2024-01-11T08:11:34.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.582+0000 7f3a34ff9700 1 -- 172.21.15.27:0/3078785899 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 6) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f3a2c051080 con 0x7f3a3813fbb0 2024-01-11T08:11:34.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.585+0000 7f3a34ff9700 1 -- 172.21.15.27:0/3078785899 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) v1 ==== 92+0+0 (secure 0 0 0) 0x7f3a2c054740 con 0x7f3a3813fbb0 2024-01-11T08:11:34.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.588+0000 7f3a3ebff700 1 -- 172.21.15.27:0/3078785899 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a1c066f30 msgr2=0x7f3a1c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:34.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.588+0000 7f3a3ebff700 1 --2- 172.21.15.27:0/3078785899 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a1c066f30 0x7f3a1c0693e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3a240091e0 tx=0x7f3a24017000 comp rx=0 tx=0).stop 2024-01-11T08:11:34.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.588+0000 7f3a3ebff700 1 -- 172.21.15.27:0/3078785899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3813fbb0 msgr2=0x7f3a38155c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:34.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.588+0000 7f3a3ebff700 1 --2- 172.21.15.27:0/3078785899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3813fbb0 0x7f3a38155c60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c00c790 tx=0x7f3a2c007c80 comp rx=0 tx=0).stop 2024-01-11T08:11:34.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.589+0000 7f3a3ebff700 1 -- 172.21.15.27:0/3078785899 shutdown_connections 2024-01-11T08:11:34.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.589+0000 7f3a3ebff700 1 --2- 172.21.15.27:0/3078785899 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a1c066f30 0x7f3a1c0693e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:34.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.589+0000 7f3a3ebff700 1 --2- 172.21.15.27:0/3078785899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a3815a560 0x7f3a380c7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:34.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.589+0000 7f3a3ebff700 1 --2- 172.21.15.27:0/3078785899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3813fbb0 0x7f3a38155c60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:34.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.589+0000 7f3a3ebff700 1 --2- 172.21.15.27:0/3078785899 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3a3813ff80 0x7f3a380c6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:34.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.589+0000 7f3a3ebff700 1 -- 172.21.15.27:0/3078785899 >> 172.21.15.27:0/3078785899 conn(0x7f3a380680a0 msgr2=0x7f3a38069db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:34.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.589+0000 7f3a3ebff700 1 -- 172.21.15.27:0/3078785899 shutdown_connections 2024-01-11T08:11:34.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.589+0000 7f3a3ebff700 1 -- 172.21.15.27:0/3078785899 wait complete. 2024-01-11T08:11:34.611 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-11T08:11:34.792 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:34.793+0000 7f9f52e981c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:11:34.801 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:34.802+0000 7f1dce0521c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:11:34.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.823+0000 7f56b61ff700 1 Processor -- start 2024-01-11T08:11:34.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.823+0000 7f56b61ff700 1 -- start start 2024-01-11T08:11:34.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.825+0000 7f56b61ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f56b0069790 0x7f56b00c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:34.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.825+0000 7f56b61ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56b0069b60 0x7f56b01598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:34.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.825+0000 7f56b61ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b00cd0a0 con 0x7f56b00c8720 2024-01-11T08:11:34.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.825+0000 7f56b61ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b00c7680 con 0x7f56b0069790 2024-01-11T08:11:34.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.825+0000 7f56b61ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b00c7960 con 0x7f56b0069b60 2024-01-11T08:11:34.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.825+0000 7f56aeffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f56b00c8720 0x7f56b01591a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57708/0 (socket says 172.21.15.27:57708) 2024-01-11T08:11:34.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.825+0000 7f56aeffd700 1 -- 172.21.15.27:0/1574853139 learned_addr learned my addr 172.21.15.27:0/1574853139 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:34.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.825+0000 7f56af7fe700 1 --2- 172.21.15.27:0/1574853139 >> v2:172.21.15.77:3300/0 conn(0x7f56b0069790 0x7f56b00c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:34.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.825+0000 7f56affff700 1 --2- 172.21.15.27:0/1574853139 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56b0069b60 0x7f56b01598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:34.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.826+0000 7f56ae7fc700 1 -- 172.21.15.27:0/1574853139 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163677089 0 0) 0x7f56b00cd0a0 con 0x7f56b00c8720 2024-01-11T08:11:34.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.826+0000 7f56ae7fc700 1 -- 172.21.15.27:0/1574853139 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5698003570 con 0x7f56b00c8720 2024-01-11T08:11:34.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.826+0000 7f56af7fe700 1 -- 172.21.15.27:0/1574853139 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56b0069b60 msgr2=0x7f56b01598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:34.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.826+0000 7f56af7fe700 1 --2- 172.21.15.27:0/1574853139 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56b0069b60 0x7f56b01598b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:34.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.826+0000 7f56af7fe700 1 -- 172.21.15.27:0/1574853139 >> v1:172.21.15.27:6789/0 conn(0x7f56b00c8720 legacy=0x7f56b01591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:34.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.826+0000 7f56af7fe700 1 -- 172.21.15.27:0/1574853139 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b00c7c40 con 0x7f56b0069790 2024-01-11T08:11:34.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.826+0000 7f56af7fe700 1 --2- 172.21.15.27:0/1574853139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56b0069790 0x7f56b00c69a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f56a00098d0 tx=0x7f56a0009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9b2c99d2adcabb3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:34.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.826+0000 7f56ae7fc700 1 -- 172.21.15.27:0/1574853139 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a0005c10 con 0x7f56b0069790 2024-01-11T08:11:34.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.826+0000 7f56ae7fc700 1 -- 172.21.15.27:0/1574853139 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a0005d70 con 0x7f56b0069790 2024-01-11T08:11:34.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.826+0000 7f56ae7fc700 1 -- 172.21.15.27:0/1574853139 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a0006260 con 0x7f56b0069790 2024-01-11T08:11:34.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.827+0000 7f56b61ff700 1 -- 172.21.15.27:0/1574853139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56b0069790 msgr2=0x7f56b00c69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:34.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.827+0000 7f56b61ff700 1 --2- 172.21.15.27:0/1574853139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56b0069790 0x7f56b00c69a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f56a00098d0 tx=0x7f56a0009120 comp rx=0 tx=0).stop 2024-01-11T08:11:34.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.827+0000 7f56b61ff700 1 -- 172.21.15.27:0/1574853139 shutdown_connections 2024-01-11T08:11:34.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.827+0000 7f56b61ff700 1 --2- 172.21.15.27:0/1574853139 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56b0069b60 0x7f56b01598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:34.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.827+0000 7f56b61ff700 1 --2- 172.21.15.27:0/1574853139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56b0069790 0x7f56b00c69a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:34.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.827+0000 7f56b61ff700 1 -- 172.21.15.27:0/1574853139 >> 172.21.15.27:0/1574853139 conn(0x7f56b00680a0 msgr2=0x7f56b00cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:34.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.827+0000 7f56b61ff700 1 -- 172.21.15.27:0/1574853139 shutdown_connections 2024-01-11T08:11:34.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.827+0000 7f56b61ff700 1 -- 172.21.15.27:0/1574853139 wait complete. 2024-01-11T08:11:34.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.829+0000 7f56b61ff700 1 Processor -- start 2024-01-11T08:11:34.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.829+0000 7f56b61ff700 1 -- start start 2024-01-11T08:11:34.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.830+0000 7f56b61ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56b0069790 0x7f56b00aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:34.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.830+0000 7f56b61ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56b0069b60 0x7f56b00c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:34.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.830+0000 7f56b61ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56b00c8720 0x7f56b0144ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:34.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.830+0000 7f56b61ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b0145220 con 0x7f56b00c8720 2024-01-11T08:11:34.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.830+0000 7f56b61ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b00c4d70 con 0x7f56b0069790 2024-01-11T08:11:34.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.830+0000 7f56b61ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b00c4a90 con 0x7f56b0069b60 2024-01-11T08:11:34.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.830+0000 7f56af7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56b0069790 0x7f56b00aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:34.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.830+0000 7f56affff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56b00c8720 0x7f56b0144ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:34.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.830+0000 7f56af7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56b0069790 0x7f56b00aef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:50542/0 (socket says 172.21.15.27:50542) 2024-01-11T08:11:34.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.830+0000 7f56af7fe700 1 -- 172.21.15.27:0/1987453034 learned_addr learned my addr 172.21.15.27:0/1987453034 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:34.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.830+0000 7f56aeffd700 1 --2- 172.21.15.27:0/1987453034 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56b0069b60 0x7f56b00c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:34.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.831+0000 7f56affff700 1 -- 172.21.15.27:0/1987453034 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56b0069b60 msgr2=0x7f56b00c5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:34.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.831+0000 7f56affff700 1 --2- 172.21.15.27:0/1987453034 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56b0069b60 0x7f56b00c5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:34.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.831+0000 7f56affff700 1 -- 172.21.15.27:0/1987453034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56b0069790 msgr2=0x7f56b00aef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:34.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.831+0000 7f56affff700 1 --2- 172.21.15.27:0/1987453034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56b0069790 0x7f56b00aef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:34.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.831+0000 7f56affff700 1 -- 172.21.15.27:0/1987453034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a00093c0 con 0x7f56b00c8720 2024-01-11T08:11:34.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.831+0000 7f56affff700 1 --2- 172.21.15.27:0/1987453034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56b00c8720 0x7f56b0144ce0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f56a8009e50 tx=0x7f56a800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:34.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.831+0000 7f56977fe700 1 -- 172.21.15.27:0/1987453034 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a800caa0 con 0x7f56b00c8720 2024-01-11T08:11:34.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.831+0000 7f56977fe700 1 -- 172.21.15.27:0/1987453034 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a800f040 con 0x7f56b00c8720 2024-01-11T08:11:34.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.831+0000 7f56977fe700 1 -- 172.21.15.27:0/1987453034 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a800ce50 con 0x7f56b00c8720 2024-01-11T08:11:34.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.831+0000 7f56b61ff700 1 -- 172.21.15.27:0/1987453034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b00c5150 con 0x7f56b00c8720 2024-01-11T08:11:34.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.831+0000 7f56b61ff700 1 -- 172.21.15.27:0/1987453034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b00c5710 con 0x7f56b00c8720 2024-01-11T08:11:34.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.833+0000 7f56b61ff700 1 -- 172.21.15.27:0/1987453034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b004c3c0 con 0x7f56b00c8720 2024-01-11T08:11:34.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.833+0000 7f56977fe700 1 -- 172.21.15.27:0/1987453034 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 6) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f56a8014690 con 0x7f56b00c8720 2024-01-11T08:11:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.834+0000 7f56977fe700 1 --2- 172.21.15.27:0/1987453034 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f569c066f80 0x7f569c069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.834+0000 7f56977fe700 1 -- 172.21.15.27:0/1987453034 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f56a8085160 con 0x7f56b00c8720 2024-01-11T08:11:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.834+0000 7f56af7fe700 1 -- 172.21.15.27:0/1987453034 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f569c066f80 msgr2=0x7f569c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:11:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.834+0000 7f56af7fe700 1 --2- 172.21.15.27:0/1987453034 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f569c066f80 0x7f569c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:11:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:34.839+0000 7f56977fe700 1 -- 172.21.15.27:0/1987453034 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f56a8055e60 con 0x7f56b00c8720 2024-01-11T08:11:34.970 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:34.971+0000 7f9f52e981c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:11:35.000 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:35.000+0000 7f1dce0521c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:11:35.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.036+0000 7f56af7fe700 1 -- 172.21.15.27:0/1987453034 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f569c066f80 msgr2=0x7f569c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:11:35.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.036+0000 7f56af7fe700 1 --2- 172.21.15.27:0/1987453034 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f569c066f80 0x7f569c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:11:35.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.087+0000 7f56b61ff700 1 -- 172.21.15.27:0/1987453034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f56b00c6670 con 0x7f56b00c8720 2024-01-11T08:11:35.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.091+0000 7f56977fe700 1 -- 172.21.15.27:0/1987453034 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v11)=0 v11) v1 ==== 138+0+0 (secure 0 0 0) 0x7f56a80559f0 con 0x7f56b00c8720 2024-01-11T08:11:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.093+0000 7f56b61ff700 1 -- 172.21.15.27:0/1987453034 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f569c066f80 msgr2=0x7f569c069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:11:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.093+0000 7f56b61ff700 1 --2- 172.21.15.27:0/1987453034 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f569c066f80 0x7f569c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.094+0000 7f56b61ff700 1 -- 172.21.15.27:0/1987453034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56b00c8720 msgr2=0x7f56b0144ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.094+0000 7f56b61ff700 1 --2- 172.21.15.27:0/1987453034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56b00c8720 0x7f56b0144ce0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f56a8009e50 tx=0x7f56a800b040 comp rx=0 tx=0).stop 2024-01-11T08:11:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.094+0000 7f56b61ff700 1 -- 172.21.15.27:0/1987453034 shutdown_connections 2024-01-11T08:11:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.094+0000 7f56b61ff700 1 --2- 172.21.15.27:0/1987453034 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f569c066f80 0x7f569c069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.094+0000 7f56b61ff700 1 --2- 172.21.15.27:0/1987453034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56b00c8720 0x7f56b0144ce0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.094+0000 7f56b61ff700 1 --2- 172.21.15.27:0/1987453034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56b0069790 0x7f56b00aef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.094+0000 7f56b61ff700 1 --2- 172.21.15.27:0/1987453034 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56b0069b60 0x7f56b00c5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.094+0000 7f56b61ff700 1 -- 172.21.15.27:0/1987453034 >> 172.21.15.27:0/1987453034 conn(0x7f56b00680a0 msgr2=0x7f56b00c9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:35.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.094+0000 7f56b61ff700 1 -- 172.21.15.27:0/1987453034 shutdown_connections 2024-01-11T08:11:35.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.094+0000 7f56b61ff700 1 -- 172.21.15.27:0/1987453034 wait complete. 2024-01-11T08:11:35.112 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-11T08:11:35.284 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:35.284+0000 7f9f52e981c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:11:35.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.321+0000 7fddc1bff700 1 Processor -- start 2024-01-11T08:11:35.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.322+0000 7fddc1bff700 1 -- start start 2024-01-11T08:11:35.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.323+0000 7fddc1bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fddbc1596f0 0x7fddbc0a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:35.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.323+0000 7fddc1bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fddbc0a9de0 0x7fddbc068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:35.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.323+0000 7fddc1bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddbc159ac0 0x7fddbc068b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:35.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.323+0000 7fddc1bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddbc0c80a0 con 0x7fddbc1596f0 2024-01-11T08:11:35.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.323+0000 7fddc1bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddbc0cbac0 con 0x7fddbc0a9de0 2024-01-11T08:11:35.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.323+0000 7fddc1bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc0c7490 con 0x7fddbc159ac0 2024-01-11T08:11:35.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.323+0000 7fddbb7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fddbc1596f0 0x7fddbc0a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:35.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.323+0000 7fddbb7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fddbc1596f0 0x7fddbc0a98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55064/0 (socket says 172.21.15.27:55064) 2024-01-11T08:11:35.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.323+0000 7fddbb7fe700 1 -- 172.21.15.27:0/3260189024 learned_addr learned my addr 172.21.15.27:0/3260189024 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:35.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.323+0000 7fddbaffd700 1 --2- 172.21.15.27:0/3260189024 >> v2:172.21.15.77:3300/0 conn(0x7fddbc0a9de0 0x7fddbc068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:35.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.323+0000 7fddbbfff700 1 --2- 172.21.15.27:0/3260189024 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddbc159ac0 0x7fddbc068b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:35.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.324+0000 7fddbb7fe700 1 -- 172.21.15.27:0/3260189024 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddbc159ac0 msgr2=0x7fddbc068b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:35.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.324+0000 7fddbb7fe700 1 --2- 172.21.15.27:0/3260189024 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddbc159ac0 0x7fddbc068b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.324+0000 7fddbb7fe700 1 -- 172.21.15.27:0/3260189024 >> v2:172.21.15.77:3300/0 conn(0x7fddbc0a9de0 msgr2=0x7fddbc068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:35.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.324+0000 7fddbb7fe700 1 --2- 172.21.15.27:0/3260189024 >> v2:172.21.15.77:3300/0 conn(0x7fddbc0a9de0 0x7fddbc068620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.324+0000 7fddbb7fe700 1 -- 172.21.15.27:0/3260189024 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddbc0c7770 con 0x7fddbc1596f0 2024-01-11T08:11:35.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.324+0000 7fddbbfff700 1 --2- 172.21.15.27:0/3260189024 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddbc159ac0 0x7fddbc068b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:35.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.324+0000 7fddbb7fe700 1 --2- 172.21.15.27:0/3260189024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddbc1596f0 0x7fddbc0a98a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fddbc06eff0 tx=0x7fddac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4491991853aa545 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:35.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.325+0000 7fddba7fc700 1 -- 172.21.15.27:0/3260189024 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac009e70 con 0x7fddbc1596f0 2024-01-11T08:11:35.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.325+0000 7fddba7fc700 1 -- 172.21.15.27:0/3260189024 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddac00b860 con 0x7fddbc1596f0 2024-01-11T08:11:35.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.325+0000 7fddba7fc700 1 -- 172.21.15.27:0/3260189024 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac00bc00 con 0x7fddbc1596f0 2024-01-11T08:11:35.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.325+0000 7fddc1bff700 1 -- 172.21.15.27:0/3260189024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddbc1596f0 msgr2=0x7fddbc0a98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:35.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.325+0000 7fddc1bff700 1 --2- 172.21.15.27:0/3260189024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddbc1596f0 0x7fddbc0a98a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fddbc06eff0 tx=0x7fddac009250 comp rx=0 tx=0).stop 2024-01-11T08:11:35.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.325+0000 7fddc1bff700 1 -- 172.21.15.27:0/3260189024 shutdown_connections 2024-01-11T08:11:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.325+0000 7fddc1bff700 1 --2- 172.21.15.27:0/3260189024 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddbc159ac0 0x7fddbc068b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.325+0000 7fddc1bff700 1 --2- 172.21.15.27:0/3260189024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddbc1596f0 0x7fddbc0a98a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.325+0000 7fddc1bff700 1 --2- 172.21.15.27:0/3260189024 >> v2:172.21.15.77:3300/0 conn(0x7fddbc0a9de0 0x7fddbc068620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.325+0000 7fddc1bff700 1 -- 172.21.15.27:0/3260189024 >> 172.21.15.27:0/3260189024 conn(0x7fddbc0c9fb0 msgr2=0x7fddbc06ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.325+0000 7fddc1bff700 1 -- 172.21.15.27:0/3260189024 shutdown_connections 2024-01-11T08:11:35.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.325+0000 7fddc1bff700 1 -- 172.21.15.27:0/3260189024 wait complete. 2024-01-11T08:11:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.327+0000 7fddc1bff700 1 Processor -- start 2024-01-11T08:11:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.327+0000 7fddc1bff700 1 -- start start 2024-01-11T08:11:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.328+0000 7fddc1bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddbc0a9de0 0x7fddbc16dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.328+0000 7fddc1bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddbc1596f0 0x7fddbc0abe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.328+0000 7fddc1bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddbc159ac0 0x7fddbc14c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:35.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.328+0000 7fddc1bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc0ac340 con 0x7fddbc159ac0 2024-01-11T08:11:35.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.328+0000 7fddc1bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc16d170 con 0x7fddbc1596f0 2024-01-11T08:11:35.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.328+0000 7fddc1bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc0af160 con 0x7fddbc0a9de0 2024-01-11T08:11:35.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.328+0000 7fddbb7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddbc0a9de0 0x7fddbc16dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:35.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.328+0000 7fddbbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddbc159ac0 0x7fddbc14c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:35.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.328+0000 7fddbb7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddbc0a9de0 0x7fddbc16dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:43904/0 (socket says 172.21.15.27:43904) 2024-01-11T08:11:35.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.328+0000 7fddbb7fe700 1 -- 172.21.15.27:0/1732043843 learned_addr learned my addr 172.21.15.27:0/1732043843 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:35.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.328+0000 7fddbaffd700 1 --2- 172.21.15.27:0/1732043843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddbc1596f0 0x7fddbc0abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:35.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.329+0000 7fddbb7fe700 1 -- 172.21.15.27:0/1732043843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddbc1596f0 msgr2=0x7fddbc0abe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:35.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.329+0000 7fddbb7fe700 1 --2- 172.21.15.27:0/1732043843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddbc1596f0 0x7fddbc0abe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.329+0000 7fddbb7fe700 1 -- 172.21.15.27:0/1732043843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddbc159ac0 msgr2=0x7fddbc14c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:35.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.329+0000 7fddbb7fe700 1 --2- 172.21.15.27:0/1732043843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddbc159ac0 0x7fddbc14c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.329+0000 7fddbb7fe700 1 -- 172.21.15.27:0/1732043843 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddac0094f0 con 0x7fddbc0a9de0 2024-01-11T08:11:35.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.329+0000 7fddbbfff700 1 --2- 172.21.15.27:0/1732043843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddbc159ac0 0x7fddbc14c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:35.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.329+0000 7fddbb7fe700 1 --2- 172.21.15.27:0/1732043843 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddbc0a9de0 0x7fddbc16dc10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fddac009290 tx=0x7fddac017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:35.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.329+0000 7fdda37fe700 1 -- 172.21.15.27:0/1732043843 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac00bbe0 con 0x7fddbc0a9de0 2024-01-11T08:11:35.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.329+0000 7fdda37fe700 1 -- 172.21.15.27:0/1732043843 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddac000c90 con 0x7fddbc0a9de0 2024-01-11T08:11:35.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.329+0000 7fddc1bff700 1 -- 172.21.15.27:0/1732043843 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddbc16d410 con 0x7fddbc0a9de0 2024-01-11T08:11:35.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.330+0000 7fdda37fe700 1 -- 172.21.15.27:0/1732043843 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac01a680 con 0x7fddbc0a9de0 2024-01-11T08:11:35.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.330+0000 7fddc1bff700 1 -- 172.21.15.27:0/1732043843 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddbc0af4c0 con 0x7fddbc0a9de0 2024-01-11T08:11:35.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.331+0000 7fddc1bff700 1 -- 172.21.15.27:0/1732043843 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddbc04c3c0 con 0x7fddbc0a9de0 2024-01-11T08:11:35.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.332+0000 7fdda37fe700 1 -- 172.21.15.27:0/1732043843 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 6) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fddac01aa30 con 0x7fddbc0a9de0 2024-01-11T08:11:35.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.332+0000 7fdda37fe700 1 --2- 172.21.15.27:0/1732043843 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdda8066f80 0x7fdda8069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:35.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.333+0000 7fdda37fe700 1 -- 172.21.15.27:0/1732043843 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fddac085b30 con 0x7fddbc0a9de0 2024-01-11T08:11:35.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.333+0000 7fddbaffd700 1 -- 172.21.15.27:0/1732043843 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdda8066f80 msgr2=0x7fdda8069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:11:35.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.333+0000 7fddbaffd700 1 --2- 172.21.15.27:0/1732043843 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdda8066f80 0x7fdda8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:11:35.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.343+0000 7fdda37fe700 1 -- 172.21.15.27:0/1732043843 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fddac052d00 con 0x7fddbc0a9de0 2024-01-11T08:11:35.347 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:35.347+0000 7f1dce0521c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:11:35.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.534+0000 7fddbaffd700 1 -- 172.21.15.27:0/1732043843 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdda8066f80 msgr2=0x7fdda8069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:11:35.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.534+0000 7fddbaffd700 1 --2- 172.21.15.27:0/1732043843 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdda8066f80 0x7fdda8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:11:35.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.546+0000 7fddc1bff700 1 -- 172.21.15.27:0/1732043843 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fddbc1579c0 con 0x7fddbc0a9de0 2024-01-11T08:11:35.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.553+0000 7fdda37fe700 1 -- 172.21.15.27:0/1732043843 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v12)=0 v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7fddac056320 con 0x7fddbc0a9de0 2024-01-11T08:11:35.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.555+0000 7fddc1bff700 1 -- 172.21.15.27:0/1732043843 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdda8066f80 msgr2=0x7fdda8069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:11:35.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.555+0000 7fddc1bff700 1 --2- 172.21.15.27:0/1732043843 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdda8066f80 0x7fdda8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.555+0000 7fddc1bff700 1 -- 172.21.15.27:0/1732043843 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddbc0a9de0 msgr2=0x7fddbc16dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:35.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.555+0000 7fddc1bff700 1 --2- 172.21.15.27:0/1732043843 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddbc0a9de0 0x7fddbc16dc10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fddac009290 tx=0x7fddac017000 comp rx=0 tx=0).stop 2024-01-11T08:11:35.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.555+0000 7fddc1bff700 1 -- 172.21.15.27:0/1732043843 shutdown_connections 2024-01-11T08:11:35.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.555+0000 7fddc1bff700 1 --2- 172.21.15.27:0/1732043843 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdda8066f80 0x7fdda8069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.555+0000 7fddc1bff700 1 --2- 172.21.15.27:0/1732043843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddbc159ac0 0x7fddbc14c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.555+0000 7fddc1bff700 1 --2- 172.21.15.27:0/1732043843 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddbc0a9de0 0x7fddbc16dc10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.555+0000 7fddc1bff700 1 --2- 172.21.15.27:0/1732043843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddbc1596f0 0x7fddbc0abe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.555+0000 7fddc1bff700 1 -- 172.21.15.27:0/1732043843 >> 172.21.15.27:0/1732043843 conn(0x7fddbc0c9fb0 msgr2=0x7fddbc1586c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:35.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.555+0000 7fddc1bff700 1 -- 172.21.15.27:0/1732043843 shutdown_connections 2024-01-11T08:11:35.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.555+0000 7fddc1bff700 1 -- 172.21.15.27:0/1732043843 wait complete. 2024-01-11T08:11:35.573 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:11:35.574 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-01-11T08:11:35.630 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:11:35.631 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2024-01-11T08:11:35.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir0 1m 2024-01-11T08:11:35.815 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:35.815+0000 7f9f52e981c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:11:35.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.888+0000 7f3a411ff700 1 Processor -- start 2024-01-11T08:11:35.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.888+0000 7f3a411ff700 1 -- start start 2024-01-11T08:11:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.889+0000 7f3a411ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3a3c13fbb0 0x7f3a3c0ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.889+0000 7f3a411ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3a3c15a730 0x7f3a3c0c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.889+0000 7f3a411ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a3c0c6b30 con 0x7f3a3c13ff80 2024-01-11T08:11:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.889+0000 7f3a411ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a3c0c6800 con 0x7f3a3c13fbb0 2024-01-11T08:11:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.889+0000 7f3a411ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a3c0cb240 con 0x7f3a3c15a730 2024-01-11T08:11:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.890+0000 7f3a3ad9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3a3c13fbb0 0x7f3a3c0ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.890+0000 7f3a3ad9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3a3c13fbb0 0x7f3a3c0ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55088/0 (socket says 172.21.15.27:55088) 2024-01-11T08:11:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.890+0000 7f3a3ad9d700 1 -- 172.21.15.27:0/2017894422 learned_addr learned my addr 172.21.15.27:0/2017894422 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.890+0000 7f3a3a59c700 1 --1- 172.21.15.27:0/2017894422 >> v1:172.21.15.77:6789/0 conn(0x7f3a3c13ff80 0x7f3a3c15a020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:34892/0 (socket says 172.21.15.27:34892) 2024-01-11T08:11:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.890+0000 7f3a3b59e700 1 --2- 172.21.15.27:0/2017894422 >> v2:172.21.15.77:3300/0 conn(0x7f3a3c15a730 0x7f3a3c0c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.890+0000 7f3a39d9b700 1 -- 172.21.15.27:0/2017894422 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857097885 0 0) 0x7f3a3c0c6b30 con 0x7f3a3c13ff80 2024-01-11T08:11:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.890+0000 7f3a3ad9d700 1 -- 172.21.15.27:0/2017894422 >> v2:172.21.15.77:3300/0 conn(0x7f3a3c15a730 msgr2=0x7f3a3c0c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.890+0000 7f3a3ad9d700 1 --2- 172.21.15.27:0/2017894422 >> v2:172.21.15.77:3300/0 conn(0x7f3a3c15a730 0x7f3a3c0c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.890+0000 7f3a3ad9d700 1 -- 172.21.15.27:0/2017894422 >> v1:172.21.15.77:6789/0 conn(0x7f3a3c13ff80 legacy=0x7f3a3c15a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.890+0000 7f3a3ad9d700 1 -- 172.21.15.27:0/2017894422 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3c0cb520 con 0x7f3a3c13fbb0 2024-01-11T08:11:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.891+0000 7f3a3ad9d700 1 --2- 172.21.15.27:0/2017894422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3c13fbb0 0x7f3a3c0ca7a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c06fe10 tx=0x7f3a2c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d50b80be78bfc6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.891+0000 7f3a39d9b700 1 -- 172.21.15.27:0/2017894422 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c005c10 con 0x7f3a3c13fbb0 2024-01-11T08:11:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.891+0000 7f3a39d9b700 1 -- 172.21.15.27:0/2017894422 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a2c005d70 con 0x7f3a3c13fbb0 2024-01-11T08:11:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.891+0000 7f3a39d9b700 1 -- 172.21.15.27:0/2017894422 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c006260 con 0x7f3a3c13fbb0 2024-01-11T08:11:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.892+0000 7f3a411ff700 1 -- 172.21.15.27:0/2017894422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3c13fbb0 msgr2=0x7f3a3c0ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.892+0000 7f3a411ff700 1 --2- 172.21.15.27:0/2017894422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3c13fbb0 0x7f3a3c0ca7a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c06fe10 tx=0x7f3a2c009220 comp rx=0 tx=0).stop 2024-01-11T08:11:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.892+0000 7f3a411ff700 1 -- 172.21.15.27:0/2017894422 shutdown_connections 2024-01-11T08:11:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.892+0000 7f3a411ff700 1 --2- 172.21.15.27:0/2017894422 >> v2:172.21.15.77:3300/0 conn(0x7f3a3c15a730 0x7f3a3c0c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.892+0000 7f3a411ff700 1 --2- 172.21.15.27:0/2017894422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3c13fbb0 0x7f3a3c0ca7a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.892+0000 7f3a411ff700 1 -- 172.21.15.27:0/2017894422 >> 172.21.15.27:0/2017894422 conn(0x7f3a3c0680a0 msgr2=0x7f3a3c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.892+0000 7f3a411ff700 1 -- 172.21.15.27:0/2017894422 shutdown_connections 2024-01-11T08:11:35.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.892+0000 7f3a411ff700 1 -- 172.21.15.27:0/2017894422 wait complete. 2024-01-11T08:11:35.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.894+0000 7f3a411ff700 1 Processor -- start 2024-01-11T08:11:35.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.895+0000 7f3a411ff700 1 -- start start 2024-01-11T08:11:35.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.896+0000 7f3a411ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3c13fbb0 0x7f3a3c156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:35.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.896+0000 7f3a411ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3a3c13ff80 0x7f3a3c138a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:35.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.896+0000 7f3a411ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a3c15a730 0x7f3a3c155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:35.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.896+0000 7f3a411ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c16d970 con 0x7f3a3c13fbb0 2024-01-11T08:11:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.896+0000 7f3a411ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c155e60 con 0x7f3a3c15a730 2024-01-11T08:11:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.896+0000 7f3a411ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c16a1a0 con 0x7f3a3c13ff80 2024-01-11T08:11:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.897+0000 7f3a3ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3c13fbb0 0x7f3a3c156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.897+0000 7f3a3ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3c13fbb0 0x7f3a3c156000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55102/0 (socket says 172.21.15.27:55102) 2024-01-11T08:11:35.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.897+0000 7f3a3ad9d700 1 -- 172.21.15.27:0/3391331529 learned_addr learned my addr 172.21.15.27:0/3391331529 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.897+0000 7f3a3a59c700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3a3c13ff80 0x7f3a3c138a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.897+0000 7f3a3b59e700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a3c15a730 0x7f3a3c155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.897+0000 7f3a3ad9d700 1 -- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3a3c13ff80 msgr2=0x7f3a3c138a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.897+0000 7f3a3ad9d700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3a3c13ff80 0x7f3a3c138a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.897+0000 7f3a3ad9d700 1 -- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a3c15a730 msgr2=0x7f3a3c155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:35.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.897+0000 7f3a3ad9d700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a3c15a730 0x7f3a3c155920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.897+0000 7f3a3ad9d700 1 -- 172.21.15.27:0/3391331529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2c009480 con 0x7f3a3c13fbb0 2024-01-11T08:11:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.897+0000 7f3a3a59c700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3a3c13ff80 0x7f3a3c138a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.897+0000 7f3a3ad9d700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3c13fbb0 0x7f3a3c156000 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c00b0e0 tx=0x7f3a2c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.898+0000 7f3a22ffd700 1 -- 172.21.15.27:0/3391331529 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c00b780 con 0x7f3a3c13fbb0 2024-01-11T08:11:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.898+0000 7f3a22ffd700 1 -- 172.21.15.27:0/3391331529 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a2c00b900 con 0x7f3a3c13fbb0 2024-01-11T08:11:35.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.898+0000 7f3a22ffd700 1 -- 172.21.15.27:0/3391331529 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c00bcc0 con 0x7f3a3c13fbb0 2024-01-11T08:11:35.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.898+0000 7f3a411ff700 1 -- 172.21.15.27:0/3391331529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a3c16de40 con 0x7f3a3c13fbb0 2024-01-11T08:11:35.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.898+0000 7f3a411ff700 1 -- 172.21.15.27:0/3391331529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a3c169b50 con 0x7f3a3c13fbb0 2024-01-11T08:11:35.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.899+0000 7f3a411ff700 1 -- 172.21.15.27:0/3391331529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a3c1699b0 con 0x7f3a3c13fbb0 2024-01-11T08:11:35.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.900+0000 7f3a22ffd700 1 -- 172.21.15.27:0/3391331529 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 6) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f3a2c01b480 con 0x7f3a3c13fbb0 2024-01-11T08:11:35.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.902+0000 7f3a22ffd700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a28066fb0 0x7f3a28069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:35.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.902+0000 7f3a22ffd700 1 -- 172.21.15.27:0/3391331529 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f3a2c0865d0 con 0x7f3a3c13fbb0 2024-01-11T08:11:35.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.902+0000 7f3a3a59c700 1 -- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a28066fb0 msgr2=0x7f3a28069460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:11:35.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.902+0000 7f3a3a59c700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a28066fb0 0x7f3a28069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:11:35.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:35.905+0000 7f3a22ffd700 1 -- 172.21.15.27:0/3391331529 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3a3c1699b0 con 0x7f3a3c13fbb0 2024-01-11T08:11:35.934 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:35.934+0000 7f1dce0521c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:11:35.984 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:35.984+0000 7f9f52e981c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:11:36.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:36.103+0000 7f3a3a59c700 1 -- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a28066fb0 msgr2=0x7f3a28069460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:11:36.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:36.103+0000 7f3a3a59c700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a28066fb0 0x7f3a28069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:11:36.114 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:36.115+0000 7f1dce0521c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:11:36.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:36.135+0000 7f3a411ff700 1 -- 172.21.15.27:0/3391331529 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a3c1699b0 con 0x7f3a28066fb0 2024-01-11T08:11:36.395 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:36.395+0000 7f9f52e981c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:11:36.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:36.504+0000 7f3a3a59c700 1 -- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a28066fb0 msgr2=0x7f3a28069460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:11:36.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:36.504+0000 7f3a3a59c700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a28066fb0 0x7f3a28069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-11T08:11:36.550 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:36.550+0000 7f1dce0521c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:11:37.167 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:37.167+0000 7f9f52e981c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:11:37.254 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:37.254+0000 7f9f52e981c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:11:37.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:37.305+0000 7f3a3a59c700 1 -- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a28066fb0 msgr2=0x7f3a28069460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:11:37.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:37.305+0000 7f3a3a59c700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a28066fb0 0x7f3a28069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-11T08:11:37.368 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:37.369+0000 7f1dce0521c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:11:37.406 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:37.407+0000 7f9f52e981c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:11:37.460 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:37.461+0000 7f1dce0521c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:11:37.492 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:37.492+0000 7f9f52e981c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:11:37.624 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:37.624+0000 7f1dce0521c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:11:37.706 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:37.707+0000 7f9f52e981c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:11:37.716 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:37.717+0000 7f1dce0521c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:11:37.801 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:37.801+0000 7f9f52e981c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:11:37.895 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:37.895+0000 7f9f52e981c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:11:37.941 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:37.941+0000 7f1dce0521c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:11:37.987 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:37.988+0000 7f9f52e981c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:11:38.041 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:38.041+0000 7f1dce0521c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:11:38.140 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:38.141+0000 7f1dce0521c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:11:38.173 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:38.173+0000 7f9f52e981c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:11:38.240 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:38.240+0000 7f1dce0521c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:11:38.269 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:38.269+0000 7f9f52e981c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:11:38.436 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:38.437+0000 7f1dce0521c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:11:38.538 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:38.538+0000 7f1dce0521c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:11:38.605 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:38.606+0000 7f9f52e981c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:11:38.693 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:38.694+0000 7f9f52e981c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:11:38.893 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:38.893+0000 7f1dce0521c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:11:38.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:38.906+0000 7f3a3a59c700 1 -- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a28066fb0 msgr2=0x7f3a28069460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:11:38.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:38.906+0000 7f3a3a59c700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a28066fb0 0x7f3a28069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-11T08:11:38.986 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:38.986+0000 7f1dce0521c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:11:39.231 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:39.231+0000 7f9f52e981c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:11:39.412 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:39.413+0000 7f9f52e981c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:11:39.556 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:39.556+0000 7f1dce0521c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:11:39.748 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:39.749+0000 7f1dce0521c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:11:39.787 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:39.787+0000 7f9f52e981c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:11:40.141 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:40.141+0000 7f1dce0521c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:11:40.203 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:40.202+0000 7f9f52e981c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:11:40.380 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:40.380+0000 7f9f52e981c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:11:40.566 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:40.566+0000 7f9f52e981c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:11:40.582 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:40.582+0000 7f1dce0521c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:11:40.672 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:40.672+0000 7f9f52e981c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:11:40.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:40.684+0000 7f3a22ffd700 1 -- 172.21.15.27:0/3391331529 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 7) v1 ==== 84470+0+0 (secure 0 0 0) 0x7f3a2c053c10 con 0x7f3a3c13fbb0 2024-01-11T08:11:40.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:40.684+0000 7f3a22ffd700 1 -- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a28066fb0 msgr2=0x7f3a28069460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:11:40.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:40.684+0000 7f3a22ffd700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a28066fb0 0x7f3a28069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:40.782 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:40.783+0000 7f1dce0521c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:11:40.824 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:40.824+0000 7f9ecb4bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:11:40.824 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:40.824+0000 7f9ecb4bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:11:40.824 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:40.824+0000 7f9ecb4bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:11:40.824 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:40.824+0000 7f9ecb4bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:11:40.824 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:40.824+0000 7f9ecb4bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:11:40.825 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:40.824+0000 7f9ec7cb8700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:11:40.825 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:40.824+0000 7f9ec7cb8700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:11:40.825 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:40.824+0000 7f9ec7cb8700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:11:40.825 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:40.824+0000 7f9ec7cb8700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:11:40.825 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:11:40.824+0000 7f9ec7cb8700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:11:40.983 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:40.984+0000 7f1dce0521c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:11:41.096 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:11:41.096+0000 7f1dce0521c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:11:41.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:41.686+0000 7f3a22ffd700 1 -- 172.21.15.27:0/3391331529 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 8) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f3a2c05e010 con 0x7f3a3c13fbb0 2024-01-11T08:11:41.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:41.686+0000 7f3a22ffd700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a28066fb0 0x7f3a28069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:41.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:41.687+0000 7f3a22ffd700 1 -- 172.21.15.27:0/3391331529 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a3c1699b0 con 0x7f3a28066fb0 2024-01-11T08:11:41.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:41.687+0000 7f3a3a59c700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a28066fb0 0x7f3a28069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:41.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:41.688+0000 7f3a3a59c700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a28066fb0 0x7f3a28069460 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c1387f0 tx=0x7f3a30007d50 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:41.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:41.783+0000 7f3a22ffd700 1 -- 172.21.15.27:0/3391331529 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f3a3c1699b0 con 0x7f3a28066fb0 2024-01-11T08:11:41.784 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path /snap_test_dir1/dir0 2024-01-11T08:11:41.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:41.786+0000 7f3a411ff700 1 -- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a28066fb0 msgr2=0x7f3a28069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:41.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:41.786+0000 7f3a411ff700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a28066fb0 0x7f3a28069460 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c1387f0 tx=0x7f3a30007d50 comp rx=0 tx=0).stop 2024-01-11T08:11:41.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:41.786+0000 7f3a411ff700 1 -- 172.21.15.27:0/3391331529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3c13fbb0 msgr2=0x7f3a3c156000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:41.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:41.786+0000 7f3a411ff700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3c13fbb0 0x7f3a3c156000 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c00b0e0 tx=0x7f3a2c016000 comp rx=0 tx=0).stop 2024-01-11T08:11:41.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:41.786+0000 7f3a411ff700 1 -- 172.21.15.27:0/3391331529 shutdown_connections 2024-01-11T08:11:41.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:41.786+0000 7f3a411ff700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3a28066fb0 0x7f3a28069460 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:41.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:41.786+0000 7f3a411ff700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a3c15a730 0x7f3a3c155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:41.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:41.786+0000 7f3a411ff700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3c13fbb0 0x7f3a3c156000 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:41.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:41.786+0000 7f3a411ff700 1 --2- 172.21.15.27:0/3391331529 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3a3c13ff80 0x7f3a3c138a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:41.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:41.786+0000 7f3a411ff700 1 -- 172.21.15.27:0/3391331529 >> 172.21.15.27:0/3391331529 conn(0x7f3a3c0680a0 msgr2=0x7f3a3c0704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:41.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:41.786+0000 7f3a411ff700 1 -- 172.21.15.27:0/3391331529 shutdown_connections 2024-01-11T08:11:41.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:41.786+0000 7f3a411ff700 1 -- 172.21.15.27:0/3391331529 wait complete. 2024-01-11T08:11:41.805 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2024-01-11T08:11:41.805 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:11:41.805 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2024-01-11T08:11:41.856 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir1 1m 2024-01-11T08:11:42.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.058+0000 7fdad5fff700 1 Processor -- start 2024-01-11T08:11:42.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.059+0000 7fdad5fff700 1 -- start start 2024-01-11T08:11:42.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.060+0000 7fdad5fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdad0142dc0 0x7fdad00cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:42.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.060+0000 7fdad5fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdad0143190 0x7fdad00cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:42.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.060+0000 7fdad5fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad015d320 con 0x7fdad00cc020 2024-01-11T08:11:42.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.060+0000 7fdad5fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdad015df30 con 0x7fdad0142dc0 2024-01-11T08:11:42.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.060+0000 7fdad5fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad015d590 con 0x7fdad0143190 2024-01-11T08:11:42.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.060+0000 7fdacffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdad00cc020 0x7fdad00cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57712/0 (socket says 172.21.15.27:57712) 2024-01-11T08:11:42.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.060+0000 7fdacffff700 1 -- 172.21.15.27:0/187709223 learned_addr learned my addr 172.21.15.27:0/187709223 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.060+0000 7fdacf7fe700 1 --2- 172.21.15.27:0/187709223 >> v2:172.21.15.77:3300/0 conn(0x7fdad0142dc0 0x7fdad00cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.060+0000 7fdaceffd700 1 --2- 172.21.15.27:0/187709223 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdad0143190 0x7fdad00cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.061+0000 7fdace7fc700 1 -- 172.21.15.27:0/187709223 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650863167 0 0) 0x7fdad015d320 con 0x7fdad00cc020 2024-01-11T08:11:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.061+0000 7fdace7fc700 1 -- 172.21.15.27:0/187709223 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab8003570 con 0x7fdad00cc020 2024-01-11T08:11:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.061+0000 7fdace7fc700 1 -- 172.21.15.27:0/187709223 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3645818052 0 0) 0x7fdab8003570 con 0x7fdad00cc020 2024-01-11T08:11:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.061+0000 7fdace7fc700 1 -- 172.21.15.27:0/187709223 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdad0143190 msgr2=0x7fdad00cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.061+0000 7fdace7fc700 1 --2- 172.21.15.27:0/187709223 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdad0143190 0x7fdad00cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.061+0000 7fdace7fc700 1 -- 172.21.15.27:0/187709223 >> v2:172.21.15.77:3300/0 conn(0x7fdad0142dc0 msgr2=0x7fdad00cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.061+0000 7fdace7fc700 1 --2- 172.21.15.27:0/187709223 >> v2:172.21.15.77:3300/0 conn(0x7fdad0142dc0 0x7fdad00cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.061+0000 7fdace7fc700 1 -- 172.21.15.27:0/187709223 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad015d870 con 0x7fdad00cc020 2024-01-11T08:11:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.061+0000 7fdace7fc700 1 -- 172.21.15.27:0/187709223 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fdac80017f0 con 0x7fdad00cc020 2024-01-11T08:11:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.061+0000 7fdace7fc700 1 -- 172.21.15.27:0/187709223 >> v1:172.21.15.27:6789/0 conn(0x7fdad00cc020 legacy=0x7fdad00cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.061+0000 7fdace7fc700 1 --2- 172.21.15.27:0/187709223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab8003a60 0x7fdab8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.061+0000 7fdace7fc700 1 --2- 172.21.15.27:0/187709223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab8004810 0x7fdab8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.061+0000 7fdace7fc700 1 --2- 172.21.15.27:0/187709223 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdab80051c0 0x7fdab800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.061+0000 7fdace7fc700 1 -- 172.21.15.27:0/187709223 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad015df30 con 0x7fdab8003a60 2024-01-11T08:11:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.061+0000 7fdace7fc700 1 -- 172.21.15.27:0/187709223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad015d590 con 0x7fdab8004810 2024-01-11T08:11:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.061+0000 7fdace7fc700 1 -- 172.21.15.27:0/187709223 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab800bfa0 con 0x7fdab80051c0 2024-01-11T08:11:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.061+0000 7fdacf7fe700 1 --2- 172.21.15.27:0/187709223 >> v2:172.21.15.77:3300/0 conn(0x7fdad0142dc0 0x7fdad00cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.061+0000 7fdaceffd700 1 --2- 172.21.15.27:0/187709223 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdad0143190 0x7fdad00cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.062+0000 7fdacffff700 1 --2- 172.21.15.27:0/187709223 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdab80051c0 0x7fdab800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.062+0000 7fdacf7fe700 1 --2- 172.21.15.27:0/187709223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab8003a60 0x7fdab8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.062+0000 7fdaceffd700 1 --2- 172.21.15.27:0/187709223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab8004810 0x7fdab8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.062+0000 7fdacffff700 1 -- 172.21.15.27:0/187709223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab8004810 msgr2=0x7fdab8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.062+0000 7fdacffff700 1 --2- 172.21.15.27:0/187709223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab8004810 0x7fdab8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.062+0000 7fdacffff700 1 -- 172.21.15.27:0/187709223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab8003a60 msgr2=0x7fdab8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.062+0000 7fdacffff700 1 --2- 172.21.15.27:0/187709223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab8003a60 0x7fdab8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.062+0000 7fdacffff700 1 -- 172.21.15.27:0/187709223 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab8003570 con 0x7fdab80051c0 2024-01-11T08:11:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.062+0000 7fdacffff700 1 --2- 172.21.15.27:0/187709223 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdab80051c0 0x7fdab800ba60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdac800a970 tx=0x7fdac8002210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a4d2a0c96d431fc server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.062+0000 7fdace7fc700 1 -- 172.21.15.27:0/187709223 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac80070d0 con 0x7fdab80051c0 2024-01-11T08:11:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.062+0000 7fdace7fc700 1 -- 172.21.15.27:0/187709223 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac8007270 con 0x7fdab80051c0 2024-01-11T08:11:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.062+0000 7fdace7fc700 1 -- 172.21.15.27:0/187709223 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac8007630 con 0x7fdab80051c0 2024-01-11T08:11:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.063+0000 7fdad5fff700 1 -- 172.21.15.27:0/187709223 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdab80051c0 msgr2=0x7fdab800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.063+0000 7fdad5fff700 1 --2- 172.21.15.27:0/187709223 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdab80051c0 0x7fdab800ba60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdac800a970 tx=0x7fdac8002210 comp rx=0 tx=0).stop 2024-01-11T08:11:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.063+0000 7fdacffff700 1 -- 172.21.15.27:0/187709223 reap_dead start 2024-01-11T08:11:42.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.063+0000 7fdad5fff700 1 -- 172.21.15.27:0/187709223 shutdown_connections 2024-01-11T08:11:42.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.063+0000 7fdad5fff700 1 -- 172.21.15.27:0/187709223 >> 172.21.15.27:0/187709223 conn(0x7fdad0063ca0 msgr2=0x7fdad006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:42.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.063+0000 7fdad5fff700 1 -- 172.21.15.27:0/187709223 shutdown_connections 2024-01-11T08:11:42.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.063+0000 7fdad5fff700 1 -- 172.21.15.27:0/187709223 wait complete. 2024-01-11T08:11:42.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.065+0000 7fdad5fff700 1 Processor -- start 2024-01-11T08:11:42.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.065+0000 7fdad5fff700 1 -- start start 2024-01-11T08:11:42.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.066+0000 7fdad5fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab80051c0 0x7fdad013f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:42.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.066+0000 7fdad5fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdad0156d40 0x7fdad00b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:42.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.066+0000 7fdad5fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdad013b140 0x7fdad00b16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:42.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.066+0000 7fdad5fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad0158a10 con 0x7fdad0156d40 2024-01-11T08:11:42.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.066+0000 7fdad5fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad01400e0 con 0x7fdab80051c0 2024-01-11T08:11:42.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.066+0000 7fdad5fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad01403c0 con 0x7fdad013b140 2024-01-11T08:11:42.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.066+0000 7fdaceffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdad0156d40 0x7fdad00b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:42.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.066+0000 7fdaceffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdad0156d40 0x7fdad00b1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55120/0 (socket says 172.21.15.27:55120) 2024-01-11T08:11:42.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.066+0000 7fdaceffd700 1 -- 172.21.15.27:0/2412582873 learned_addr learned my addr 172.21.15.27:0/2412582873 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:42.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.066+0000 7fdacf7fe700 1 --2- 172.21.15.27:0/2412582873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab80051c0 0x7fdad013f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:42.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.066+0000 7fdacffff700 1 --2- 172.21.15.27:0/2412582873 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdad013b140 0x7fdad00b16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:42.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.066+0000 7fdaceffd700 1 -- 172.21.15.27:0/2412582873 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdad013b140 msgr2=0x7fdad00b16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:42.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.066+0000 7fdaceffd700 1 --2- 172.21.15.27:0/2412582873 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdad013b140 0x7fdad00b16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.066+0000 7fdaceffd700 1 -- 172.21.15.27:0/2412582873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab80051c0 msgr2=0x7fdad013f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:42.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.066+0000 7fdaceffd700 1 --2- 172.21.15.27:0/2412582873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab80051c0 0x7fdad013f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.066+0000 7fdaceffd700 1 -- 172.21.15.27:0/2412582873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad01406a0 con 0x7fdad0156d40 2024-01-11T08:11:42.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.067+0000 7fdaceffd700 1 --2- 172.21.15.27:0/2412582873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdad0156d40 0x7fdad00b1180 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fdac4011d20 tx=0x7fdac40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:42.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.067+0000 7fdab77fe700 1 -- 172.21.15.27:0/2412582873 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac400fa10 con 0x7fdad0156d40 2024-01-11T08:11:42.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.067+0000 7fdab77fe700 1 -- 172.21.15.27:0/2412582873 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac400fb70 con 0x7fdad0156d40 2024-01-11T08:11:42.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.067+0000 7fdad5fff700 1 -- 172.21.15.27:0/2412582873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdad01586a0 con 0x7fdad0156d40 2024-01-11T08:11:42.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.067+0000 7fdab77fe700 1 -- 172.21.15.27:0/2412582873 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac401d480 con 0x7fdad0156d40 2024-01-11T08:11:42.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.067+0000 7fdad5fff700 1 -- 172.21.15.27:0/2412582873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad00b3090 con 0x7fdad0156d40 2024-01-11T08:11:42.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.068+0000 7fdad5fff700 1 -- 172.21.15.27:0/2412582873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdad0158b80 con 0x7fdad0156d40 2024-01-11T08:11:42.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.069+0000 7fdab77fe700 1 -- 172.21.15.27:0/2412582873 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fdac401d8b0 con 0x7fdad0156d40 2024-01-11T08:11:42.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.070+0000 7fdab77fe700 1 --2- 172.21.15.27:0/2412582873 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdabc066e00 0x7fdabc0692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:42.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.070+0000 7fdab77fe700 1 -- 172.21.15.27:0/2412582873 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fdac4018070 con 0x7fdad0156d40 2024-01-11T08:11:42.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.070+0000 7fdacf7fe700 1 --2- 172.21.15.27:0/2412582873 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdabc066e00 0x7fdabc0692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:42.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.071+0000 7fdacf7fe700 1 --2- 172.21.15.27:0/2412582873 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdabc066e00 0x7fdabc0692b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdac0006f00 tx=0x7fdac00091a0 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:42.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.074+0000 7fdab77fe700 1 -- 172.21.15.27:0/2412582873 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdac4052460 con 0x7fdad0156d40 2024-01-11T08:11:42.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.303+0000 7fdad5fff700 1 -- 172.21.15.27:0/2412582873 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fdad004c3c0 con 0x7fdabc066e00 2024-01-11T08:11:42.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.372+0000 7fdab77fe700 1 -- 172.21.15.27:0/2412582873 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fdad004c3c0 con 0x7fdabc066e00 2024-01-11T08:11:42.372 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path /snap_test_dir1/dir1 2024-01-11T08:11:42.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.374+0000 7fdad5fff700 1 -- 172.21.15.27:0/2412582873 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdabc066e00 msgr2=0x7fdabc0692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.374+0000 7fdad5fff700 1 --2- 172.21.15.27:0/2412582873 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdabc066e00 0x7fdabc0692b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdac0006f00 tx=0x7fdac00091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.374+0000 7fdad5fff700 1 -- 172.21.15.27:0/2412582873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdad0156d40 msgr2=0x7fdad00b1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.374+0000 7fdad5fff700 1 --2- 172.21.15.27:0/2412582873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdad0156d40 0x7fdad00b1180 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fdac4011d20 tx=0x7fdac40091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.374+0000 7fdad5fff700 1 -- 172.21.15.27:0/2412582873 shutdown_connections 2024-01-11T08:11:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.374+0000 7fdad5fff700 1 --2- 172.21.15.27:0/2412582873 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdabc066e00 0x7fdabc0692b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.374+0000 7fdad5fff700 1 --2- 172.21.15.27:0/2412582873 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdad013b140 0x7fdad00b16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.374+0000 7fdad5fff700 1 --2- 172.21.15.27:0/2412582873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab80051c0 0x7fdad013f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.374+0000 7fdad5fff700 1 --2- 172.21.15.27:0/2412582873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdad0156d40 0x7fdad00b1180 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.374+0000 7fdad5fff700 1 -- 172.21.15.27:0/2412582873 >> 172.21.15.27:0/2412582873 conn(0x7fdad0063ca0 msgr2=0x7fdad00cac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.374+0000 7fdad5fff700 1 -- 172.21.15.27:0/2412582873 shutdown_connections 2024-01-11T08:11:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.375+0000 7fdad5fff700 1 -- 172.21.15.27:0/2412582873 wait complete. 2024-01-11T08:11:42.392 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2024-01-11T08:11:42.392 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:11:42.392 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2024-01-11T08:11:42.444 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir2 1m 2024-01-11T08:11:42.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.646+0000 7f8ebbdff700 1 Processor -- start 2024-01-11T08:11:42.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.646+0000 7f8ebbdff700 1 -- start start 2024-01-11T08:11:42.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.647+0000 7f8ebbdff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8eb41596f0 0x7f8eb40a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:42.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.647+0000 7f8ebbdff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8eb4068d30 0x7f8eb40c8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:42.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.647+0000 7f8ebbdff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb40c80a0 con 0x7f8eb40a9de0 2024-01-11T08:11:42.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.647+0000 7f8ebbdff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eb40cbac0 con 0x7f8eb4068d30 2024-01-11T08:11:42.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.647+0000 7f8ebbdff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eb40c7490 con 0x7f8eb41596f0 2024-01-11T08:11:42.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.648+0000 7f8eba39c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8eb4068d30 0x7f8eb40c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:42.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.648+0000 7f8eba39c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8eb4068d30 0x7f8eb40c8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55126/0 (socket says 172.21.15.27:55126) 2024-01-11T08:11:42.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.648+0000 7f8eba39c700 1 -- 172.21.15.27:0/1581844478 learned_addr learned my addr 172.21.15.27:0/1581844478 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:42.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.648+0000 7f8eb9b9b700 1 --2- 172.21.15.27:0/1581844478 >> v2:172.21.15.77:3300/0 conn(0x7f8eb41596f0 0x7f8eb40a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:42.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.648+0000 7f8eba39c700 1 -- 172.21.15.27:0/1581844478 >> v2:172.21.15.77:3300/0 conn(0x7f8eb41596f0 msgr2=0x7f8eb40a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:42.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.648+0000 7f8eba39c700 1 --2- 172.21.15.27:0/1581844478 >> v2:172.21.15.77:3300/0 conn(0x7f8eb41596f0 0x7f8eb40a98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.648+0000 7f8eba39c700 1 -- 172.21.15.27:0/1581844478 >> v1:172.21.15.77:6789/0 conn(0x7f8eb40a9de0 legacy=0x7f8eb4068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:42.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.648+0000 7f8eba39c700 1 -- 172.21.15.27:0/1581844478 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb40c7770 con 0x7f8eb4068d30 2024-01-11T08:11:42.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.649+0000 7f8eba39c700 1 --2- 172.21.15.27:0/1581844478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eb4068d30 0x7f8eb40c8530 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8eb40cc9b0 tx=0x7f8eac009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d487599731fbd6c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:42.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.649+0000 7f8eb8b99700 1 -- 172.21.15.27:0/1581844478 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eac00e070 con 0x7f8eb4068d30 2024-01-11T08:11:42.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.649+0000 7f8eb8b99700 1 -- 172.21.15.27:0/1581844478 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eac00ba10 con 0x7f8eb4068d30 2024-01-11T08:11:42.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.649+0000 7f8eb8b99700 1 -- 172.21.15.27:0/1581844478 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eac0092e0 con 0x7f8eb4068d30 2024-01-11T08:11:42.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.649+0000 7f8ebbdff700 1 -- 172.21.15.27:0/1581844478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eb4068d30 msgr2=0x7f8eb40c8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:42.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.649+0000 7f8ebbdff700 1 --2- 172.21.15.27:0/1581844478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eb4068d30 0x7f8eb40c8530 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8eb40cc9b0 tx=0x7f8eac009840 comp rx=0 tx=0).stop 2024-01-11T08:11:42.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.649+0000 7f8ebbdff700 1 -- 172.21.15.27:0/1581844478 shutdown_connections 2024-01-11T08:11:42.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.649+0000 7f8ebbdff700 1 --2- 172.21.15.27:0/1581844478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eb4068d30 0x7f8eb40c8530 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.649+0000 7f8ebbdff700 1 --2- 172.21.15.27:0/1581844478 >> v2:172.21.15.77:3300/0 conn(0x7f8eb41596f0 0x7f8eb40a98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.649+0000 7f8ebbdff700 1 -- 172.21.15.27:0/1581844478 >> 172.21.15.27:0/1581844478 conn(0x7f8eb40c9fb0 msgr2=0x7f8eb406ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:42.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.650+0000 7f8ebbdff700 1 -- 172.21.15.27:0/1581844478 shutdown_connections 2024-01-11T08:11:42.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.650+0000 7f8ebbdff700 1 -- 172.21.15.27:0/1581844478 wait complete. 2024-01-11T08:11:42.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.651+0000 7f8ebbdff700 1 Processor -- start 2024-01-11T08:11:42.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.651+0000 7f8ebbdff700 1 -- start start 2024-01-11T08:11:42.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.652+0000 7f8ebbdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8eb4068d30 0x7f8eb40af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:42.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.652+0000 7f8ebbdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eb40a9de0 0x7f8eb40aa620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:42.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.652+0000 7f8ebbdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8eb41596f0 0x7f8eb40aab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:42.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.652+0000 7f8ebbdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb40b25d0 con 0x7f8eb40a9de0 2024-01-11T08:11:42.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.652+0000 7f8ebbdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb40afdf0 con 0x7f8eb4068d30 2024-01-11T08:11:42.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.652+0000 7f8ebbdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb40c6cf0 con 0x7f8eb41596f0 2024-01-11T08:11:42.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.652+0000 7f8eb939a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eb40a9de0 0x7f8eb40aa620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:42.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.652+0000 7f8eb9b9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8eb4068d30 0x7f8eb40af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:42.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.652+0000 7f8eb9b9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8eb4068d30 0x7f8eb40af8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:50622/0 (socket says 172.21.15.27:50622) 2024-01-11T08:11:42.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.652+0000 7f8eb9b9b700 1 -- 172.21.15.27:0/1923970776 learned_addr learned my addr 172.21.15.27:0/1923970776 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:42.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.653+0000 7f8eba39c700 1 --2- 172.21.15.27:0/1923970776 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8eb41596f0 0x7f8eb40aab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:42.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.653+0000 7f8eb939a700 1 -- 172.21.15.27:0/1923970776 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8eb41596f0 msgr2=0x7f8eb40aab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:42.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.653+0000 7f8eb939a700 1 --2- 172.21.15.27:0/1923970776 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8eb41596f0 0x7f8eb40aab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.653+0000 7f8eb939a700 1 -- 172.21.15.27:0/1923970776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8eb4068d30 msgr2=0x7f8eb40af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:42.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.653+0000 7f8eb939a700 1 --2- 172.21.15.27:0/1923970776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8eb4068d30 0x7f8eb40af8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.653+0000 7f8eb939a700 1 -- 172.21.15.27:0/1923970776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eac009ae0 con 0x7f8eb40a9de0 2024-01-11T08:11:42.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.653+0000 7f8eb939a700 1 --2- 172.21.15.27:0/1923970776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eb40a9de0 0x7f8eb40aa620 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400c510 tx=0x7f8ea4007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.653+0000 7f8ea9ffb700 1 -- 172.21.15.27:0/1923970776 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea400ce80 con 0x7f8eb40a9de0 2024-01-11T08:11:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.653+0000 7f8ea9ffb700 1 -- 172.21.15.27:0/1923970776 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea4011590 con 0x7f8eb40a9de0 2024-01-11T08:11:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.653+0000 7f8ea9ffb700 1 -- 172.21.15.27:0/1923970776 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4011950 con 0x7f8eb40a9de0 2024-01-11T08:11:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.653+0000 7f8ebbdff700 1 -- 172.21.15.27:0/1923970776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb40b0000 con 0x7f8eb40a9de0 2024-01-11T08:11:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.654+0000 7f8ebbdff700 1 -- 172.21.15.27:0/1923970776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb40b0640 con 0x7f8eb40a9de0 2024-01-11T08:11:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.655+0000 7f8ebbdff700 1 -- 172.21.15.27:0/1923970776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb40c4ac0 con 0x7f8eb40a9de0 2024-01-11T08:11:42.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.655+0000 7f8ea9ffb700 1 -- 172.21.15.27:0/1923970776 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f8ea4011d40 con 0x7f8eb40a9de0 2024-01-11T08:11:42.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.657+0000 7f8ea9ffb700 1 --2- 172.21.15.27:0/1923970776 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8e98066dd0 0x7f8e98069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:42.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.657+0000 7f8ea9ffb700 1 -- 172.21.15.27:0/1923970776 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f8ea4083fe0 con 0x7f8eb40a9de0 2024-01-11T08:11:42.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.657+0000 7f8eb9b9b700 1 --2- 172.21.15.27:0/1923970776 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8e98066dd0 0x7f8e98069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:42.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.658+0000 7f8eb9b9b700 1 --2- 172.21.15.27:0/1923970776 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8e98066dd0 0x7f8e98069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0006fd0 tx=0x7f8eb0008040 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:42.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.661+0000 7f8ea9ffb700 1 -- 172.21.15.27:0/1923970776 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8ea4051330 con 0x7f8eb40a9de0 2024-01-11T08:11:42.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.740+0000 7f8ea9ffb700 1 -- 172.21.15.27:0/1923970776 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 9) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f8ea4051700 con 0x7f8eb40a9de0 2024-01-11T08:11:42.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.898+0000 7f8ebbdff700 1 -- 172.21.15.27:0/1923970776 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir2", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f8eb404c420 con 0x7f8e98066dd0 2024-01-11T08:11:42.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.964+0000 7f8ea9ffb700 1 -- 172.21.15.27:0/1923970776 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f8eb404c420 con 0x7f8e98066dd0 2024-01-11T08:11:42.964 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path /snap_test_dir1/dir2 2024-01-11T08:11:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.966+0000 7f8ebbdff700 1 -- 172.21.15.27:0/1923970776 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8e98066dd0 msgr2=0x7f8e98069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.966+0000 7f8ebbdff700 1 --2- 172.21.15.27:0/1923970776 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8e98066dd0 0x7f8e98069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0006fd0 tx=0x7f8eb0008040 comp rx=0 tx=0).stop 2024-01-11T08:11:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.966+0000 7f8ebbdff700 1 -- 172.21.15.27:0/1923970776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eb40a9de0 msgr2=0x7f8eb40aa620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.966+0000 7f8ebbdff700 1 --2- 172.21.15.27:0/1923970776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eb40a9de0 0x7f8eb40aa620 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400c510 tx=0x7f8ea4007ea0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.967+0000 7f8ebbdff700 1 -- 172.21.15.27:0/1923970776 shutdown_connections 2024-01-11T08:11:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.967+0000 7f8ebbdff700 1 --2- 172.21.15.27:0/1923970776 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8e98066dd0 0x7f8e98069280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.967+0000 7f8ebbdff700 1 --2- 172.21.15.27:0/1923970776 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8eb41596f0 0x7f8eb40aab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.967+0000 7f8ebbdff700 1 --2- 172.21.15.27:0/1923970776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8eb4068d30 0x7f8eb40af8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.967+0000 7f8ebbdff700 1 --2- 172.21.15.27:0/1923970776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eb40a9de0 0x7f8eb40aa620 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.967+0000 7f8ebbdff700 1 -- 172.21.15.27:0/1923970776 >> 172.21.15.27:0/1923970776 conn(0x7f8eb40c9fb0 msgr2=0x7f8eb40cac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.967+0000 7f8ebbdff700 1 -- 172.21.15.27:0/1923970776 shutdown_connections 2024-01-11T08:11:42.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:42.967+0000 7f8ebbdff700 1 -- 172.21.15.27:0/1923970776 wait complete. 2024-01-11T08:11:42.984 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2024-01-11T08:11:42.984 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:11:42.984 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2024-01-11T08:11:43.035 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir3 1m 2024-01-11T08:11:43.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.237+0000 7f0fdc1ff700 1 Processor -- start 2024-01-11T08:11:43.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.237+0000 7f0fdc1ff700 1 -- start start 2024-01-11T08:11:43.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.239+0000 7f0fdc1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0fd41596f0 0x7f0fd40a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:43.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.239+0000 7f0fdc1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0fd40a9de0 0x7f0fd4068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:43.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.239+0000 7f0fdc1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd40c80a0 con 0x7f0fd4159ac0 2024-01-11T08:11:43.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.239+0000 7f0fdc1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd40cbac0 con 0x7f0fd40a9de0 2024-01-11T08:11:43.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.239+0000 7f0fdc1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd40c7490 con 0x7f0fd41596f0 2024-01-11T08:11:43.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.239+0000 7f0fd9f9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0fd41596f0 0x7f0fd40a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:43.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.239+0000 7f0fd979a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0fd40a9de0 0x7f0fd4068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:43.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.239+0000 7f0fd9f9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0fd41596f0 0x7f0fd40a98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:43956/0 (socket says 172.21.15.27:43956) 2024-01-11T08:11:43.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.239+0000 7f0fd9f9b700 1 -- 172.21.15.27:0/2443850037 learned_addr learned my addr 172.21.15.27:0/2443850037 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:43.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.239+0000 7f0fd979a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0fd40a9de0 0x7f0fd4068620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:50632/0 (socket says 172.21.15.27:50632) 2024-01-11T08:11:43.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.240+0000 7f0fd9f9b700 1 -- 172.21.15.27:0/2443850037 >> v2:172.21.15.77:3300/0 conn(0x7f0fd40a9de0 msgr2=0x7f0fd4068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:43.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.240+0000 7f0fd9f9b700 1 --2- 172.21.15.27:0/2443850037 >> v2:172.21.15.77:3300/0 conn(0x7f0fd40a9de0 0x7f0fd4068620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:43.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.240+0000 7f0fd9f9b700 1 -- 172.21.15.27:0/2443850037 >> v1:172.21.15.77:6789/0 conn(0x7f0fd4159ac0 legacy=0x7f0fd40c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:43.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.240+0000 7f0fd8f99700 1 -- 172.21.15.27:0/2443850037 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963938643 0 0) 0x7f0fd40c80a0 con 0x7f0fd4159ac0 2024-01-11T08:11:43.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.240+0000 7f0fd9f9b700 1 -- 172.21.15.27:0/2443850037 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd40c7770 con 0x7f0fd41596f0 2024-01-11T08:11:43.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.240+0000 7f0fd979a700 1 --2- 172.21.15.27:0/2443850037 >> v2:172.21.15.77:3300/0 conn(0x7f0fd40a9de0 0x7f0fd4068620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:43.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.240+0000 7f0fd9f9b700 1 --2- 172.21.15.27:0/2443850037 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0fd41596f0 0x7f0fd40a98a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00099b0 tx=0x7f0fd0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93f4b86cff1f79d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:43.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.240+0000 7f0fd8f99700 1 -- 172.21.15.27:0/2443850037 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0009ea0 con 0x7f0fd41596f0 2024-01-11T08:11:43.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.240+0000 7f0fd8f99700 1 -- 172.21.15.27:0/2443850037 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd000b860 con 0x7f0fd41596f0 2024-01-11T08:11:43.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.240+0000 7f0fd8f99700 1 -- 172.21.15.27:0/2443850037 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd000bc00 con 0x7f0fd41596f0 2024-01-11T08:11:43.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.241+0000 7f0fdc1ff700 1 -- 172.21.15.27:0/2443850037 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0fd41596f0 msgr2=0x7f0fd40a98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:43.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.241+0000 7f0fdc1ff700 1 --2- 172.21.15.27:0/2443850037 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0fd41596f0 0x7f0fd40a98a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00099b0 tx=0x7f0fd0009250 comp rx=0 tx=0).stop 2024-01-11T08:11:43.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.241+0000 7f0fdc1ff700 1 -- 172.21.15.27:0/2443850037 shutdown_connections 2024-01-11T08:11:43.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.241+0000 7f0fdc1ff700 1 --2- 172.21.15.27:0/2443850037 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0fd41596f0 0x7f0fd40a98a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:43.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.241+0000 7f0fdc1ff700 1 --2- 172.21.15.27:0/2443850037 >> v2:172.21.15.77:3300/0 conn(0x7f0fd40a9de0 0x7f0fd4068620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:43.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.241+0000 7f0fdc1ff700 1 -- 172.21.15.27:0/2443850037 >> 172.21.15.27:0/2443850037 conn(0x7f0fd40c9fb0 msgr2=0x7f0fd406ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:43.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.241+0000 7f0fdc1ff700 1 -- 172.21.15.27:0/2443850037 shutdown_connections 2024-01-11T08:11:43.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.241+0000 7f0fdc1ff700 1 -- 172.21.15.27:0/2443850037 wait complete. 2024-01-11T08:11:43.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.243+0000 7f0fdc1ff700 1 Processor -- start 2024-01-11T08:11:43.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.243+0000 7f0fdc1ff700 1 -- start start 2024-01-11T08:11:43.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.245+0000 7f0fdc1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0fd40a9de0 0x7f0fd40c20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:43.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.245+0000 7f0fdc1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fd41596f0 0x7f0fd40c3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:43.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.245+0000 7f0fdc1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4159ac0 0x7f0fd40c4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:43.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.245+0000 7f0fdc1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd40c4a30 con 0x7f0fd4159ac0 2024-01-11T08:11:43.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.245+0000 7f0fdc1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd4157d20 con 0x7f0fd41596f0 2024-01-11T08:11:43.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.245+0000 7f0fdc1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd4169d60 con 0x7f0fd40a9de0 2024-01-11T08:11:43.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.245+0000 7f0fda79c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4159ac0 0x7f0fd40c4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:43.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.245+0000 7f0fd979a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fd41596f0 0x7f0fd40c3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:43.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.245+0000 7f0fd9f9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0fd40a9de0 0x7f0fd40c20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:43.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.245+0000 7f0fda79c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4159ac0 0x7f0fd40c4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55144/0 (socket says 172.21.15.27:55144) 2024-01-11T08:11:43.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.245+0000 7f0fd979a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fd41596f0 0x7f0fd40c3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:50644/0 (socket says 172.21.15.27:50644) 2024-01-11T08:11:43.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.245+0000 7f0fd9f9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0fd40a9de0 0x7f0fd40c20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:43970/0 (socket says 172.21.15.27:43970) 2024-01-11T08:11:43.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.245+0000 7f0fda79c700 1 -- 172.21.15.27:0/2870998607 learned_addr learned my addr 172.21.15.27:0/2870998607 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:43.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.246+0000 7f0fda79c700 1 -- 172.21.15.27:0/2870998607 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0fd40a9de0 msgr2=0x7f0fd40c20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:43.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.246+0000 7f0fda79c700 1 --2- 172.21.15.27:0/2870998607 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0fd40a9de0 0x7f0fd40c20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:43.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.246+0000 7f0fda79c700 1 -- 172.21.15.27:0/2870998607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fd41596f0 msgr2=0x7f0fd40c3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:43.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.246+0000 7f0fda79c700 1 --2- 172.21.15.27:0/2870998607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fd41596f0 0x7f0fd40c3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:43.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.246+0000 7f0fda79c700 1 -- 172.21.15.27:0/2870998607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc4009420 con 0x7f0fd4159ac0 2024-01-11T08:11:43.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.246+0000 7f0fd979a700 1 --2- 172.21.15.27:0/2870998607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fd41596f0 0x7f0fd40c3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:43.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.246+0000 7f0fda79c700 1 --2- 172.21.15.27:0/2870998607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4159ac0 0x7f0fd40c4060 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00c840 tx=0x7f0fcc0015f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:43.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.246+0000 7f0fc9ffb700 1 -- 172.21.15.27:0/2870998607 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc012630 con 0x7f0fd4159ac0 2024-01-11T08:11:43.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.246+0000 7f0fc9ffb700 1 -- 172.21.15.27:0/2870998607 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fcc012790 con 0x7f0fd4159ac0 2024-01-11T08:11:43.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.246+0000 7f0fdc1ff700 1 -- 172.21.15.27:0/2870998607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd00094f0 con 0x7f0fd4159ac0 2024-01-11T08:11:43.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.246+0000 7f0fc9ffb700 1 -- 172.21.15.27:0/2870998607 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc012b50 con 0x7f0fd4159ac0 2024-01-11T08:11:43.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.246+0000 7f0fdc1ff700 1 -- 172.21.15.27:0/2870998607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd40c4800 con 0x7f0fd4159ac0 2024-01-11T08:11:43.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.247+0000 7f0fdc1ff700 1 -- 172.21.15.27:0/2870998607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd404c3c0 con 0x7f0fd4159ac0 2024-01-11T08:11:43.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.248+0000 7f0fc9ffb700 1 -- 172.21.15.27:0/2870998607 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f0fcc003030 con 0x7f0fd4159ac0 2024-01-11T08:11:43.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.249+0000 7f0fc9ffb700 1 --2- 172.21.15.27:0/2870998607 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0fb80670a0 0x7f0fb8069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:43.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.249+0000 7f0fc9ffb700 1 -- 172.21.15.27:0/2870998607 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f0fcc084720 con 0x7f0fd4159ac0 2024-01-11T08:11:43.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.250+0000 7f0fd9f9b700 1 --2- 172.21.15.27:0/2870998607 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0fb80670a0 0x7f0fb8069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:43.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.250+0000 7f0fd9f9b700 1 --2- 172.21.15.27:0/2870998607 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0fb80670a0 0x7f0fb8069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00099b0 tx=0x7f0fd0017000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:43.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.253+0000 7f0fc9ffb700 1 -- 172.21.15.27:0/2870998607 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0fcc0516e0 con 0x7f0fd4159ac0 2024-01-11T08:11:43.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.485+0000 7f0fdc1ff700 1 -- 172.21.15.27:0/2870998607 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir3", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fd40c7a50 con 0x7f0fb80670a0 2024-01-11T08:11:43.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.553+0000 7f0fc9ffb700 1 -- 172.21.15.27:0/2870998607 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f0fd40c7a50 con 0x7f0fb80670a0 2024-01-11T08:11:43.553 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path /snap_test_dir1/dir3 2024-01-11T08:11:43.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.555+0000 7f0fdc1ff700 1 -- 172.21.15.27:0/2870998607 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0fb80670a0 msgr2=0x7f0fb8069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:43.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.555+0000 7f0fdc1ff700 1 --2- 172.21.15.27:0/2870998607 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0fb80670a0 0x7f0fb8069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00099b0 tx=0x7f0fd0017000 comp rx=0 tx=0).stop 2024-01-11T08:11:43.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.555+0000 7f0fdc1ff700 1 -- 172.21.15.27:0/2870998607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4159ac0 msgr2=0x7f0fd40c4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:43.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.555+0000 7f0fdc1ff700 1 --2- 172.21.15.27:0/2870998607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4159ac0 0x7f0fd40c4060 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00c840 tx=0x7f0fcc0015f0 comp rx=0 tx=0).stop 2024-01-11T08:11:43.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.555+0000 7f0fdc1ff700 1 -- 172.21.15.27:0/2870998607 shutdown_connections 2024-01-11T08:11:43.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.555+0000 7f0fdc1ff700 1 --2- 172.21.15.27:0/2870998607 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0fb80670a0 0x7f0fb8069550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:43.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.555+0000 7f0fdc1ff700 1 --2- 172.21.15.27:0/2870998607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4159ac0 0x7f0fd40c4060 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:43.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.555+0000 7f0fdc1ff700 1 --2- 172.21.15.27:0/2870998607 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0fd40a9de0 0x7f0fd40c20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:43.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.555+0000 7f0fdc1ff700 1 --2- 172.21.15.27:0/2870998607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fd41596f0 0x7f0fd40c3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:43.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.555+0000 7f0fdc1ff700 1 -- 172.21.15.27:0/2870998607 >> 172.21.15.27:0/2870998607 conn(0x7f0fd40c9fb0 msgr2=0x7f0fd40ab060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:43.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.555+0000 7f0fdc1ff700 1 -- 172.21.15.27:0/2870998607 shutdown_connections 2024-01-11T08:11:43.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.556+0000 7f0fdc1ff700 1 -- 172.21.15.27:0/2870998607 wait complete. 2024-01-11T08:11:43.573 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2024-01-11T08:11:43.573 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:11:43.573 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2024-01-11T08:11:43.623 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir4 1m 2024-01-11T08:11:43.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.827+0000 7f56713ff700 1 Processor -- start 2024-01-11T08:11:43.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.827+0000 7f56713ff700 1 -- start start 2024-01-11T08:11:43.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.828+0000 7f56713ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f566c15a730 0x7f566c1613a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:43.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.828+0000 7f56713ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566c0c6b30 con 0x7f566c13fbb0 2024-01-11T08:11:43.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.828+0000 7f56713ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566c0c6800 con 0x7f566c13ff80 2024-01-11T08:11:43.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.828+0000 7f56713ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f566c0cb240 con 0x7f566c15a730 2024-01-11T08:11:43.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.829+0000 7f566affd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f566c13fbb0 0x7f566c0ca7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45448/0 (socket says 172.21.15.27:45448) 2024-01-11T08:11:43.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.829+0000 7f566affd700 1 -- 172.21.15.27:0/1900358315 learned_addr learned my addr 172.21.15.27:0/1900358315 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:43.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.829+0000 7f566b7fe700 1 --2- 172.21.15.27:0/1900358315 >> v2:172.21.15.77:3300/0 conn(0x7f566c15a730 0x7f566c1613a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:43.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.829+0000 7f5669ffb700 1 -- 172.21.15.27:0/1900358315 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173256490 0 0) 0x7f566c0c6b30 con 0x7f566c13fbb0 2024-01-11T08:11:43.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.829+0000 7f5669ffb700 1 -- 172.21.15.27:0/1900358315 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5654003570 con 0x7f566c13fbb0 2024-01-11T08:11:43.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.829+0000 7f5669ffb700 1 -- 172.21.15.27:0/1900358315 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781406590 0 0) 0x7f566c0c6800 con 0x7f566c13ff80 2024-01-11T08:11:43.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.829+0000 7f5669ffb700 1 -- 172.21.15.27:0/1900358315 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f566c0c6b30 con 0x7f566c13ff80 2024-01-11T08:11:43.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.829+0000 7f566b7fe700 1 -- 172.21.15.27:0/1900358315 >> v1:172.21.15.77:6789/0 conn(0x7f566c13ff80 legacy=0x7f566c15a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:43.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.829+0000 7f566b7fe700 1 -- 172.21.15.27:0/1900358315 >> v1:172.21.15.27:6789/0 conn(0x7f566c13fbb0 legacy=0x7f566c0ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:43.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.829+0000 7f566b7fe700 1 -- 172.21.15.27:0/1900358315 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566c0cb520 con 0x7f566c15a730 2024-01-11T08:11:43.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.830+0000 7f566b7fe700 1 --2- 172.21.15.27:0/1900358315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c15a730 0x7f566c1613a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5664009e60 tx=0x7f5664009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32259fa23f910336 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:43.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.830+0000 7f5669ffb700 1 -- 172.21.15.27:0/1900358315 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566400e070 con 0x7f566c15a730 2024-01-11T08:11:43.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.830+0000 7f5669ffb700 1 -- 172.21.15.27:0/1900358315 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566400bab0 con 0x7f566c15a730 2024-01-11T08:11:43.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.830+0000 7f5669ffb700 1 -- 172.21.15.27:0/1900358315 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566400be70 con 0x7f566c15a730 2024-01-11T08:11:43.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.830+0000 7f56713ff700 1 -- 172.21.15.27:0/1900358315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c15a730 msgr2=0x7f566c1613a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:43.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.830+0000 7f56713ff700 1 --2- 172.21.15.27:0/1900358315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c15a730 0x7f566c1613a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5664009e60 tx=0x7f5664009780 comp rx=0 tx=0).stop 2024-01-11T08:11:43.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.831+0000 7f56713ff700 1 -- 172.21.15.27:0/1900358315 shutdown_connections 2024-01-11T08:11:43.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.831+0000 7f56713ff700 1 --2- 172.21.15.27:0/1900358315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c15a730 0x7f566c1613a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:43.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.831+0000 7f56713ff700 1 -- 172.21.15.27:0/1900358315 >> 172.21.15.27:0/1900358315 conn(0x7f566c0680a0 msgr2=0x7f566c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:43.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.831+0000 7f56713ff700 1 -- 172.21.15.27:0/1900358315 shutdown_connections 2024-01-11T08:11:43.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.831+0000 7f56713ff700 1 -- 172.21.15.27:0/1900358315 wait complete. 2024-01-11T08:11:43.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.832+0000 7f56713ff700 1 Processor -- start 2024-01-11T08:11:43.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.832+0000 7f56713ff700 1 -- start start 2024-01-11T08:11:43.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.833+0000 7f56713ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c13fbb0 0x7f566c156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:43.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.833+0000 7f56713ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f566c13ff80 0x7f566c0c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:43.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.833+0000 7f56713ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f566c15a730 0x7f566c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:43.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.833+0000 7f56713ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c0c4b50 con 0x7f566c15a730 2024-01-11T08:11:43.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.833+0000 7f56713ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c157070 con 0x7f566c13fbb0 2024-01-11T08:11:43.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.833+0000 7f56713ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f566c157350 con 0x7f566c13ff80 2024-01-11T08:11:43.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.834+0000 7f566b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f566c15a730 0x7f566c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:43.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.834+0000 7f566b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f566c15a730 0x7f566c155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55066/0 (socket says 172.21.15.27:55066) 2024-01-11T08:11:43.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.834+0000 7f566b7fe700 1 -- 172.21.15.27:0/3489165888 learned_addr learned my addr 172.21.15.27:0/3489165888 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:43.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.834+0000 7f566affd700 1 --2- 172.21.15.27:0/3489165888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c13fbb0 0x7f566c156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:43.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.834+0000 7f566a7fc700 1 --2- 172.21.15.27:0/3489165888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f566c13ff80 0x7f566c0c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:43.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.834+0000 7f566b7fe700 1 -- 172.21.15.27:0/3489165888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f566c13ff80 msgr2=0x7f566c0c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:43.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.834+0000 7f566b7fe700 1 --2- 172.21.15.27:0/3489165888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f566c13ff80 0x7f566c0c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:43.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.834+0000 7f566b7fe700 1 -- 172.21.15.27:0/3489165888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c13fbb0 msgr2=0x7f566c156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:43.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.834+0000 7f566b7fe700 1 --2- 172.21.15.27:0/3489165888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c13fbb0 0x7f566c156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:43.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.834+0000 7f566b7fe700 1 -- 172.21.15.27:0/3489165888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5664009a20 con 0x7f566c15a730 2024-01-11T08:11:43.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.834+0000 7f566b7fe700 1 --2- 172.21.15.27:0/3489165888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f566c15a730 0x7f566c155fb0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5664009460 tx=0x7f5664019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:43.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.835+0000 7f5652ffd700 1 -- 172.21.15.27:0/3489165888 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5664012dd0 con 0x7f566c15a730 2024-01-11T08:11:43.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.835+0000 7f5652ffd700 1 -- 172.21.15.27:0/3489165888 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5664026480 con 0x7f566c15a730 2024-01-11T08:11:43.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.835+0000 7f5652ffd700 1 -- 172.21.15.27:0/3489165888 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5664026840 con 0x7f566c15a730 2024-01-11T08:11:43.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.835+0000 7f56713ff700 1 -- 172.21.15.27:0/3489165888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f566c157630 con 0x7f566c15a730 2024-01-11T08:11:43.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.835+0000 7f56713ff700 1 -- 172.21.15.27:0/3489165888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f566c157bf0 con 0x7f566c15a730 2024-01-11T08:11:43.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.836+0000 7f56713ff700 1 -- 172.21.15.27:0/3489165888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f566c157e30 con 0x7f566c15a730 2024-01-11T08:11:43.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.837+0000 7f5652ffd700 1 -- 172.21.15.27:0/3489165888 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f566401c400 con 0x7f566c15a730 2024-01-11T08:11:43.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.838+0000 7f5652ffd700 1 --2- 172.21.15.27:0/3489165888 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f56580670a0 0x7f5658069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:43.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.838+0000 7f5652ffd700 1 -- 172.21.15.27:0/3489165888 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f5664086840 con 0x7f566c15a730 2024-01-11T08:11:43.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.839+0000 7f566affd700 1 --2- 172.21.15.27:0/3489165888 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f56580670a0 0x7f5658069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:43.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.839+0000 7f566affd700 1 --2- 172.21.15.27:0/3489165888 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f56580670a0 0x7f5658069550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f565c001360 tx=0x7f565c009040 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:43.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:43.842+0000 7f5652ffd700 1 -- 172.21.15.27:0/3489165888 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f566c157e30 con 0x7f566c15a730 2024-01-11T08:11:44.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.075+0000 7f56713ff700 1 -- 172.21.15.27:0/3489165888 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir4", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f566c157e30 con 0x7f56580670a0 2024-01-11T08:11:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.149+0000 7f5652ffd700 1 -- 172.21.15.27:0/3489165888 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f566c157e30 con 0x7f56580670a0 2024-01-11T08:11:44.149 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path /snap_test_dir1/dir4 2024-01-11T08:11:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.152+0000 7f56713ff700 1 -- 172.21.15.27:0/3489165888 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f56580670a0 msgr2=0x7f5658069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.152+0000 7f56713ff700 1 --2- 172.21.15.27:0/3489165888 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f56580670a0 0x7f5658069550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f565c001360 tx=0x7f565c009040 comp rx=0 tx=0).stop 2024-01-11T08:11:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.152+0000 7f56713ff700 1 -- 172.21.15.27:0/3489165888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f566c15a730 msgr2=0x7f566c155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.152+0000 7f56713ff700 1 --2- 172.21.15.27:0/3489165888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f566c15a730 0x7f566c155fb0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5664009460 tx=0x7f5664019000 comp rx=0 tx=0).stop 2024-01-11T08:11:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.152+0000 7f56713ff700 1 -- 172.21.15.27:0/3489165888 shutdown_connections 2024-01-11T08:11:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.152+0000 7f56713ff700 1 --2- 172.21.15.27:0/3489165888 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f56580670a0 0x7f5658069550 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.152+0000 7f56713ff700 1 --2- 172.21.15.27:0/3489165888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f566c15a730 0x7f566c155fb0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.152+0000 7f56713ff700 1 --2- 172.21.15.27:0/3489165888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f566c13fbb0 0x7f566c156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.152+0000 7f56713ff700 1 --2- 172.21.15.27:0/3489165888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f566c13ff80 0x7f566c0c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.152+0000 7f56713ff700 1 -- 172.21.15.27:0/3489165888 >> 172.21.15.27:0/3489165888 conn(0x7f566c0680a0 msgr2=0x7f566c0cbf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.152+0000 7f56713ff700 1 -- 172.21.15.27:0/3489165888 shutdown_connections 2024-01-11T08:11:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.152+0000 7f56713ff700 1 -- 172.21.15.27:0/3489165888 wait complete. 2024-01-11T08:11:44.169 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2024-01-11T08:11:44.169 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:11:44.169 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2024-01-11T08:11:44.219 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir5 1m 2024-01-11T08:11:44.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.421+0000 7f7704bff700 1 Processor -- start 2024-01-11T08:11:44.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.421+0000 7f7704bff700 1 -- start start 2024-01-11T08:11:44.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.423+0000 7f7704bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f770015a1b0 0x7f770015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:44.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.423+0000 7f7704bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f770015ae70 0x7f770015b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:44.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.423+0000 7f7704bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77000cf850 con 0x7f77000ccd70 2024-01-11T08:11:44.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.423+0000 7f7704bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77000cec40 con 0x7f770015a1b0 2024-01-11T08:11:44.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.423+0000 7f7704bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77000cef20 con 0x7f770015ae70 2024-01-11T08:11:44.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.423+0000 7f76fe59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f770015a1b0 0x7f770015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:44.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.423+0000 7f76fe59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f770015a1b0 0x7f770015a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55072/0 (socket says 172.21.15.27:55072) 2024-01-11T08:11:44.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.423+0000 7f76fe59c700 1 -- 172.21.15.27:0/2631447941 learned_addr learned my addr 172.21.15.27:0/2631447941 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:44.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.423+0000 7f76fed9d700 1 --2- 172.21.15.27:0/2631447941 >> v2:172.21.15.77:3300/0 conn(0x7f770015ae70 0x7f770015b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:44.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.423+0000 7f76fe59c700 1 -- 172.21.15.27:0/2631447941 >> v2:172.21.15.77:3300/0 conn(0x7f770015ae70 msgr2=0x7f770015b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:44.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.423+0000 7f76fe59c700 1 --2- 172.21.15.27:0/2631447941 >> v2:172.21.15.77:3300/0 conn(0x7f770015ae70 0x7f770015b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:44.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.423+0000 7f76fe59c700 1 -- 172.21.15.27:0/2631447941 >> v1:172.21.15.77:6789/0 conn(0x7f77000ccd70 legacy=0x7f77000cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:44.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.423+0000 7f76fe59c700 1 -- 172.21.15.27:0/2631447941 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77000cf200 con 0x7f770015a1b0 2024-01-11T08:11:44.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.424+0000 7f76fe59c700 1 --2- 172.21.15.27:0/2631447941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f770015a1b0 0x7f770015a580 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7700089810 tx=0x7f76f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac21e193983c6ce4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:44.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.424+0000 7f76fd59a700 1 -- 172.21.15.27:0/2631447941 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f000b820 con 0x7f770015a1b0 2024-01-11T08:11:44.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.424+0000 7f76fd59a700 1 -- 172.21.15.27:0/2631447941 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f000b980 con 0x7f770015a1b0 2024-01-11T08:11:44.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.424+0000 7f76fd59a700 1 -- 172.21.15.27:0/2631447941 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f000bd20 con 0x7f770015a1b0 2024-01-11T08:11:44.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.425+0000 7f7704bff700 1 -- 172.21.15.27:0/2631447941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f770015a1b0 msgr2=0x7f770015a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:44.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.425+0000 7f7704bff700 1 --2- 172.21.15.27:0/2631447941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f770015a1b0 0x7f770015a580 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7700089810 tx=0x7f76f0009250 comp rx=0 tx=0).stop 2024-01-11T08:11:44.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.425+0000 7f7704bff700 1 -- 172.21.15.27:0/2631447941 shutdown_connections 2024-01-11T08:11:44.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.425+0000 7f7704bff700 1 --2- 172.21.15.27:0/2631447941 >> v2:172.21.15.77:3300/0 conn(0x7f770015ae70 0x7f770015b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:44.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.425+0000 7f7704bff700 1 --2- 172.21.15.27:0/2631447941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f770015a1b0 0x7f770015a580 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:44.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.425+0000 7f7704bff700 1 -- 172.21.15.27:0/2631447941 >> 172.21.15.27:0/2631447941 conn(0x7f770006ed40 msgr2=0x7f770006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:44.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.425+0000 7f7704bff700 1 -- 172.21.15.27:0/2631447941 shutdown_connections 2024-01-11T08:11:44.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.425+0000 7f7704bff700 1 -- 172.21.15.27:0/2631447941 wait complete. 2024-01-11T08:11:44.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.426+0000 7f7704bff700 1 Processor -- start 2024-01-11T08:11:44.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.426+0000 7f7704bff700 1 -- start start 2024-01-11T08:11:44.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.427+0000 7f7704bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77000ccd70 0x7f77001582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:44.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.427+0000 7f7704bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f770015a1b0 0x7f77001556b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:44.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.427+0000 7f7704bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f770015ae70 0x7f77000b17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:44.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.427+0000 7f7704bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77000b4f10 con 0x7f77000ccd70 2024-01-11T08:11:44.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.427+0000 7f7704bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77000b4300 con 0x7f770015a1b0 2024-01-11T08:11:44.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.427+0000 7f7704bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77000b45e0 con 0x7f770015ae70 2024-01-11T08:11:44.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.428+0000 7f76fe59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77000ccd70 0x7f77001582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:44.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.428+0000 7f76fe59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77000ccd70 0x7f77001582d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55090/0 (socket says 172.21.15.27:55090) 2024-01-11T08:11:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.428+0000 7f76fe59c700 1 -- 172.21.15.27:0/20453776 learned_addr learned my addr 172.21.15.27:0/20453776 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.428+0000 7f76fed9d700 1 --2- 172.21.15.27:0/20453776 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f770015ae70 0x7f77000b17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.428+0000 7f76fdd9b700 1 --2- 172.21.15.27:0/20453776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f770015a1b0 0x7f77001556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.428+0000 7f76fe59c700 1 -- 172.21.15.27:0/20453776 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f770015ae70 msgr2=0x7f77000b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.428+0000 7f76fe59c700 1 --2- 172.21.15.27:0/20453776 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f770015ae70 0x7f77000b17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.428+0000 7f76fe59c700 1 -- 172.21.15.27:0/20453776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f770015a1b0 msgr2=0x7f77001556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.428+0000 7f76fe59c700 1 --2- 172.21.15.27:0/20453776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f770015a1b0 0x7f77001556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.428+0000 7f76fe59c700 1 -- 172.21.15.27:0/20453776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f00094f0 con 0x7f77000ccd70 2024-01-11T08:11:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.428+0000 7f76fdd9b700 1 --2- 172.21.15.27:0/20453776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f770015a1b0 0x7f77001556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.428+0000 7f76fe59c700 1 --2- 172.21.15.27:0/20453776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77000ccd70 0x7f77001582d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f76f0009290 tx=0x7f76f0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:44.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.429+0000 7f76ee7fc700 1 -- 172.21.15.27:0/20453776 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f0011d70 con 0x7f77000ccd70 2024-01-11T08:11:44.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.429+0000 7f76ee7fc700 1 -- 172.21.15.27:0/20453776 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f0020070 con 0x7f77000ccd70 2024-01-11T08:11:44.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.429+0000 7f7704bff700 1 -- 172.21.15.27:0/20453776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77000b48c0 con 0x7f77000ccd70 2024-01-11T08:11:44.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.429+0000 7f76ee7fc700 1 -- 172.21.15.27:0/20453776 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f001b6a0 con 0x7f77000ccd70 2024-01-11T08:11:44.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.429+0000 7f7704bff700 1 -- 172.21.15.27:0/20453776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7700158810 con 0x7f77000ccd70 2024-01-11T08:11:44.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.430+0000 7f7704bff700 1 -- 172.21.15.27:0/20453776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f770004c3c0 con 0x7f77000ccd70 2024-01-11T08:11:44.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.431+0000 7f76ee7fc700 1 -- 172.21.15.27:0/20453776 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f76f0022030 con 0x7f77000ccd70 2024-01-11T08:11:44.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.432+0000 7f76ee7fc700 1 --2- 172.21.15.27:0/20453776 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f76e00670a0 0x7f76e0069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:44.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.432+0000 7f76ee7fc700 1 -- 172.21.15.27:0/20453776 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f76f00867d0 con 0x7f77000ccd70 2024-01-11T08:11:44.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.433+0000 7f76fdd9b700 1 --2- 172.21.15.27:0/20453776 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f76e00670a0 0x7f76e0069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:44.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.433+0000 7f76fdd9b700 1 --2- 172.21.15.27:0/20453776 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f76e00670a0 0x7f76e0069550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f76f4006fd0 tx=0x7f76f4008040 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:44.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.436+0000 7f76ee7fc700 1 -- 172.21.15.27:0/20453776 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f76f0053790 con 0x7f77000ccd70 2024-01-11T08:11:44.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.667+0000 7f7704bff700 1 -- 172.21.15.27:0/20453776 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir5", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f77000b5050 con 0x7f76e00670a0 2024-01-11T08:11:44.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.742+0000 7f76ee7fc700 1 -- 172.21.15.27:0/20453776 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f76f400d630 con 0x7f76e00670a0 2024-01-11T08:11:44.742 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path /snap_test_dir1/dir5 2024-01-11T08:11:44.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.744+0000 7f7704bff700 1 -- 172.21.15.27:0/20453776 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f76e00670a0 msgr2=0x7f76e0069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:44.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.744+0000 7f7704bff700 1 --2- 172.21.15.27:0/20453776 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f76e00670a0 0x7f76e0069550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f76f4006fd0 tx=0x7f76f4008040 comp rx=0 tx=0).stop 2024-01-11T08:11:44.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.744+0000 7f7704bff700 1 -- 172.21.15.27:0/20453776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77000ccd70 msgr2=0x7f77001582d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:44.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.744+0000 7f7704bff700 1 --2- 172.21.15.27:0/20453776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77000ccd70 0x7f77001582d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f76f0009290 tx=0x7f76f0016000 comp rx=0 tx=0).stop 2024-01-11T08:11:44.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.744+0000 7f7704bff700 1 -- 172.21.15.27:0/20453776 shutdown_connections 2024-01-11T08:11:44.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.744+0000 7f7704bff700 1 --2- 172.21.15.27:0/20453776 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f76e00670a0 0x7f76e0069550 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:44.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.744+0000 7f7704bff700 1 --2- 172.21.15.27:0/20453776 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f770015ae70 0x7f77000b17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:44.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.744+0000 7f7704bff700 1 --2- 172.21.15.27:0/20453776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77000ccd70 0x7f77001582d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:44.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.744+0000 7f7704bff700 1 --2- 172.21.15.27:0/20453776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f770015a1b0 0x7f77001556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:44.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.744+0000 7f7704bff700 1 -- 172.21.15.27:0/20453776 >> 172.21.15.27:0/20453776 conn(0x7f770006ed40 msgr2=0x7f77000d0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:44.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.744+0000 7f7704bff700 1 -- 172.21.15.27:0/20453776 shutdown_connections 2024-01-11T08:11:44.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:44.745+0000 7f7704bff700 1 -- 172.21.15.27:0/20453776 wait complete. 2024-01-11T08:11:44.762 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2024-01-11T08:11:44.762 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:11:44.762 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2024-01-11T08:11:44.813 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir6 1m 2024-01-11T08:11:45.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.015+0000 7f151edff700 1 Processor -- start 2024-01-11T08:11:45.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.015+0000 7f151edff700 1 -- start start 2024-01-11T08:11:45.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.016+0000 7f151edff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f151813fbb0 0x7f15180ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:45.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.016+0000 7f151edff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f151815a730 0x7f15180c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:45.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.016+0000 7f151edff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15180c6b30 con 0x7f151813ff80 2024-01-11T08:11:45.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.017+0000 7f151edff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15180c6800 con 0x7f151815a730 2024-01-11T08:11:45.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.017+0000 7f151edff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15180cb240 con 0x7f151813fbb0 2024-01-11T08:11:45.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.017+0000 7f151d39c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f151815a730 0x7f15180c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:45.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.017+0000 7f151d39c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f151815a730 0x7f15180c6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55094/0 (socket says 172.21.15.27:55094) 2024-01-11T08:11:45.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.017+0000 7f151d39c700 1 -- 172.21.15.27:0/3446814519 learned_addr learned my addr 172.21.15.27:0/3446814519 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:45.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.017+0000 7f151cb9b700 1 --2- 172.21.15.27:0/3446814519 >> v2:172.21.15.77:3300/0 conn(0x7f151813fbb0 0x7f15180ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:45.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.017+0000 7f151d39c700 1 -- 172.21.15.27:0/3446814519 >> v2:172.21.15.77:3300/0 conn(0x7f151813fbb0 msgr2=0x7f15180ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:45.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.017+0000 7f151d39c700 1 --2- 172.21.15.27:0/3446814519 >> v2:172.21.15.77:3300/0 conn(0x7f151813fbb0 0x7f15180ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.017+0000 7f151d39c700 1 -- 172.21.15.27:0/3446814519 >> v1:172.21.15.77:6789/0 conn(0x7f151813ff80 legacy=0x7f151815a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:45.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.017+0000 7f151d39c700 1 -- 172.21.15.27:0/3446814519 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15180cb520 con 0x7f151815a730 2024-01-11T08:11:45.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.018+0000 7f151d39c700 1 --2- 172.21.15.27:0/3446814519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f151815a730 0x7f15180c6fc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1518064df0 tx=0x7f151000ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c931a39690b363 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:45.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.018+0000 7f15177fe700 1 -- 172.21.15.27:0/3446814519 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1510010070 con 0x7f151815a730 2024-01-11T08:11:45.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.018+0000 7f15177fe700 1 -- 172.21.15.27:0/3446814519 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151000dde0 con 0x7f151815a730 2024-01-11T08:11:45.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.018+0000 7f15177fe700 1 -- 172.21.15.27:0/3446814519 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151000b320 con 0x7f151815a730 2024-01-11T08:11:45.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.018+0000 7f151edff700 1 -- 172.21.15.27:0/3446814519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f151815a730 msgr2=0x7f15180c6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:45.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.018+0000 7f151edff700 1 --2- 172.21.15.27:0/3446814519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f151815a730 0x7f15180c6fc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1518064df0 tx=0x7f151000ba50 comp rx=0 tx=0).stop 2024-01-11T08:11:45.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.019+0000 7f151edff700 1 -- 172.21.15.27:0/3446814519 shutdown_connections 2024-01-11T08:11:45.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.019+0000 7f151edff700 1 --2- 172.21.15.27:0/3446814519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f151815a730 0x7f15180c6fc0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.019+0000 7f151edff700 1 --2- 172.21.15.27:0/3446814519 >> v2:172.21.15.77:3300/0 conn(0x7f151813fbb0 0x7f15180ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.019+0000 7f151edff700 1 -- 172.21.15.27:0/3446814519 >> 172.21.15.27:0/3446814519 conn(0x7f15180680a0 msgr2=0x7f15180cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:45.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.019+0000 7f151edff700 1 -- 172.21.15.27:0/3446814519 shutdown_connections 2024-01-11T08:11:45.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.019+0000 7f151edff700 1 -- 172.21.15.27:0/3446814519 wait complete. 2024-01-11T08:11:45.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.020+0000 7f151edff700 1 Processor -- start 2024-01-11T08:11:45.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.020+0000 7f151edff700 1 -- start start 2024-01-11T08:11:45.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.021+0000 7f151edff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f151813fbb0 0x7f15181663b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:45.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.021+0000 7f151edff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f151813ff80 0x7f15180c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:45.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.021+0000 7f151edff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f151815a730 0x7f1518156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:45.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.021+0000 7f151edff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15181668f0 con 0x7f151815a730 2024-01-11T08:11:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.021+0000 7f151edff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1518165d00 con 0x7f151813ff80 2024-01-11T08:11:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.021+0000 7f151edff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15180c4240 con 0x7f151813fbb0 2024-01-11T08:11:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.022+0000 7f151d39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f151815a730 0x7f1518156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.022+0000 7f151d39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f151815a730 0x7f1518156a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55106/0 (socket says 172.21.15.27:55106) 2024-01-11T08:11:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.022+0000 7f151d39c700 1 -- 172.21.15.27:0/184314923 learned_addr learned my addr 172.21.15.27:0/184314923 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.022+0000 7f151cb9b700 1 --2- 172.21.15.27:0/184314923 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f151813fbb0 0x7f15181663b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.022+0000 7f1517fff700 1 --2- 172.21.15.27:0/184314923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f151813ff80 0x7f15180c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.022+0000 7f151d39c700 1 -- 172.21.15.27:0/184314923 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f151813fbb0 msgr2=0x7f15181663b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.022+0000 7f151d39c700 1 --2- 172.21.15.27:0/184314923 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f151813fbb0 0x7f15181663b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.022+0000 7f151d39c700 1 -- 172.21.15.27:0/184314923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f151813ff80 msgr2=0x7f15180c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.022+0000 7f151d39c700 1 --2- 172.21.15.27:0/184314923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f151813ff80 0x7f15180c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.022+0000 7f151d39c700 1 -- 172.21.15.27:0/184314923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151000bcf0 con 0x7f151815a730 2024-01-11T08:11:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.022+0000 7f151d39c700 1 --2- 172.21.15.27:0/184314923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f151815a730 0x7f1518156a90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1518069490 tx=0x7f151001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.023+0000 7f1514ff9700 1 -- 172.21.15.27:0/184314923 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15100032f0 con 0x7f151815a730 2024-01-11T08:11:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.023+0000 7f1514ff9700 1 -- 172.21.15.27:0/184314923 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151000b4d0 con 0x7f151815a730 2024-01-11T08:11:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.023+0000 7f151edff700 1 -- 172.21.15.27:0/184314923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15180c4800 con 0x7f151815a730 2024-01-11T08:11:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.023+0000 7f1514ff9700 1 -- 172.21.15.27:0/184314923 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15100276b0 con 0x7f151815a730 2024-01-11T08:11:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.023+0000 7f151edff700 1 -- 172.21.15.27:0/184314923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15181597c0 con 0x7f151815a730 2024-01-11T08:11:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.024+0000 7f151edff700 1 -- 172.21.15.27:0/184314923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f151804c3c0 con 0x7f151815a730 2024-01-11T08:11:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.025+0000 7f1514ff9700 1 -- 172.21.15.27:0/184314923 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f1510003040 con 0x7f151815a730 2024-01-11T08:11:45.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.026+0000 7f1514ff9700 1 --2- 172.21.15.27:0/184314923 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f14fc0670a0 0x7f14fc069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:45.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.026+0000 7f1514ff9700 1 -- 172.21.15.27:0/184314923 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f1510088550 con 0x7f151815a730 2024-01-11T08:11:45.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.026+0000 7f151cb9b700 1 --2- 172.21.15.27:0/184314923 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f14fc0670a0 0x7f14fc069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:45.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.027+0000 7f151cb9b700 1 --2- 172.21.15.27:0/184314923 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f14fc0670a0 0x7f14fc069550 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f15180c5a10 tx=0x7f150c009120 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:45.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.030+0000 7f1514ff9700 1 -- 172.21.15.27:0/184314923 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f15100554a0 con 0x7f151815a730 2024-01-11T08:11:45.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.258+0000 7f151edff700 1 -- 172.21.15.27:0/184314923 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir6", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f15180cb800 con 0x7f14fc0670a0 2024-01-11T08:11:45.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.330+0000 7f1514ff9700 1 -- 172.21.15.27:0/184314923 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f15180cb800 con 0x7f14fc0670a0 2024-01-11T08:11:45.330 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path /snap_test_dir1/dir6 2024-01-11T08:11:45.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.332+0000 7f151edff700 1 -- 172.21.15.27:0/184314923 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f14fc0670a0 msgr2=0x7f14fc069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:45.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.332+0000 7f151edff700 1 --2- 172.21.15.27:0/184314923 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f14fc0670a0 0x7f14fc069550 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f15180c5a10 tx=0x7f150c009120 comp rx=0 tx=0).stop 2024-01-11T08:11:45.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.332+0000 7f151edff700 1 -- 172.21.15.27:0/184314923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f151815a730 msgr2=0x7f1518156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:45.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.332+0000 7f151edff700 1 --2- 172.21.15.27:0/184314923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f151815a730 0x7f1518156a90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1518069490 tx=0x7f151001a000 comp rx=0 tx=0).stop 2024-01-11T08:11:45.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.332+0000 7f151edff700 1 -- 172.21.15.27:0/184314923 shutdown_connections 2024-01-11T08:11:45.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.332+0000 7f151edff700 1 --2- 172.21.15.27:0/184314923 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f14fc0670a0 0x7f14fc069550 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.332+0000 7f151edff700 1 --2- 172.21.15.27:0/184314923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f151815a730 0x7f1518156a90 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.332+0000 7f151edff700 1 --2- 172.21.15.27:0/184314923 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f151813fbb0 0x7f15181663b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.332+0000 7f151edff700 1 --2- 172.21.15.27:0/184314923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f151813ff80 0x7f15180c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.332+0000 7f151edff700 1 -- 172.21.15.27:0/184314923 >> 172.21.15.27:0/184314923 conn(0x7f15180680a0 msgr2=0x7f15180c0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:45.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.333+0000 7f151edff700 1 -- 172.21.15.27:0/184314923 shutdown_connections 2024-01-11T08:11:45.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.333+0000 7f151edff700 1 -- 172.21.15.27:0/184314923 wait complete. 2024-01-11T08:11:45.350 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2024-01-11T08:11:45.350 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:11:45.350 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2024-01-11T08:11:45.400 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir7 1m 2024-01-11T08:11:45.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.604+0000 7f21b97ff700 1 Processor -- start 2024-01-11T08:11:45.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.605+0000 7f21b97ff700 1 -- start start 2024-01-11T08:11:45.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.606+0000 7f21b97ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f21b4157410 0x7f21b40ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:45.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.606+0000 7f21b97ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f21b41577e0 0x7f21b40ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:45.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.606+0000 7f21b97ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f21b4069540 0x7f21b40ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:45.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.606+0000 7f21b97ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b40699d0 con 0x7f21b4069540 2024-01-11T08:11:45.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.606+0000 7f21b97ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b4069cb0 con 0x7f21b41577e0 2024-01-11T08:11:45.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.606+0000 7f21b97ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b4069f90 con 0x7f21b4157410 2024-01-11T08:11:45.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.606+0000 7f21b37fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f21b4069540 0x7f21b40ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:45.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.606+0000 7f21b37fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f21b4069540 0x7f21b40ae540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55122/0 (socket says 172.21.15.27:55122) 2024-01-11T08:11:45.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.606+0000 7f21b37fe700 1 -- 172.21.15.27:0/2175692971 learned_addr learned my addr 172.21.15.27:0/2175692971 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:45.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.607+0000 7f21b27fc700 1 --2- 172.21.15.27:0/2175692971 >> v2:172.21.15.77:3300/0 conn(0x7f21b41577e0 0x7f21b40ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:45.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.607+0000 7f21b2ffd700 1 --2- 172.21.15.27:0/2175692971 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f21b4157410 0x7f21b40ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.607+0000 7f21b37fe700 1 -- 172.21.15.27:0/2175692971 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f21b4157410 msgr2=0x7f21b40ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.607+0000 7f21b37fe700 1 --2- 172.21.15.27:0/2175692971 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f21b4157410 0x7f21b40ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.607+0000 7f21b37fe700 1 -- 172.21.15.27:0/2175692971 >> v2:172.21.15.77:3300/0 conn(0x7f21b41577e0 msgr2=0x7f21b40ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:45.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.607+0000 7f21b37fe700 1 --2- 172.21.15.27:0/2175692971 >> v2:172.21.15.77:3300/0 conn(0x7f21b41577e0 0x7f21b40ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.607+0000 7f21b37fe700 1 -- 172.21.15.27:0/2175692971 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b406a270 con 0x7f21b4069540 2024-01-11T08:11:45.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.607+0000 7f21b37fe700 1 --2- 172.21.15.27:0/2175692971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b4069540 0x7f21b40ae540 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f21b4070cc0 tx=0x7f21ac009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b121fe679917fbb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:45.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.608+0000 7f21b1ffb700 1 -- 172.21.15.27:0/2175692971 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ac00e070 con 0x7f21b4069540 2024-01-11T08:11:45.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.608+0000 7f21b1ffb700 1 -- 172.21.15.27:0/2175692971 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ac00b970 con 0x7f21b4069540 2024-01-11T08:11:45.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.608+0000 7f21b1ffb700 1 -- 172.21.15.27:0/2175692971 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ac0092e0 con 0x7f21b4069540 2024-01-11T08:11:45.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.608+0000 7f21b97ff700 1 -- 172.21.15.27:0/2175692971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b4069540 msgr2=0x7f21b40ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:45.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.608+0000 7f21b97ff700 1 --2- 172.21.15.27:0/2175692971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b4069540 0x7f21b40ae540 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f21b4070cc0 tx=0x7f21ac009840 comp rx=0 tx=0).stop 2024-01-11T08:11:45.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.608+0000 7f21b97ff700 1 -- 172.21.15.27:0/2175692971 shutdown_connections 2024-01-11T08:11:45.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.608+0000 7f21b97ff700 1 --2- 172.21.15.27:0/2175692971 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f21b4157410 0x7f21b40ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.608+0000 7f21b97ff700 1 --2- 172.21.15.27:0/2175692971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b4069540 0x7f21b40ae540 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.608+0000 7f21b97ff700 1 --2- 172.21.15.27:0/2175692971 >> v2:172.21.15.77:3300/0 conn(0x7f21b41577e0 0x7f21b40ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.608+0000 7f21b97ff700 1 -- 172.21.15.27:0/2175692971 >> 172.21.15.27:0/2175692971 conn(0x7f21b40680a0 msgr2=0x7f21b40cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:45.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.608+0000 7f21b97ff700 1 -- 172.21.15.27:0/2175692971 shutdown_connections 2024-01-11T08:11:45.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.608+0000 7f21b97ff700 1 -- 172.21.15.27:0/2175692971 wait complete. 2024-01-11T08:11:45.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.610+0000 7f21b97ff700 1 Processor -- start 2024-01-11T08:11:45.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.610+0000 7f21b97ff700 1 -- start start 2024-01-11T08:11:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.611+0000 7f21b97ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21b4069540 0x7f21b414be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.611+0000 7f21b97ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f21b4157410 0x7f21b414e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.611+0000 7f21b97ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b41577e0 0x7f21b414fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.611+0000 7f21b97ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b414e0f0 con 0x7f21b41577e0 2024-01-11T08:11:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.611+0000 7f21b97ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b416ca30 con 0x7f21b4069540 2024-01-11T08:11:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.611+0000 7f21b97ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b40ca150 con 0x7f21b4157410 2024-01-11T08:11:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.611+0000 7f21b37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b41577e0 0x7f21b414fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.611+0000 7f21b37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b41577e0 0x7f21b414fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55124/0 (socket says 172.21.15.27:55124) 2024-01-11T08:11:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.611+0000 7f21b37fe700 1 -- 172.21.15.27:0/3948607044 learned_addr learned my addr 172.21.15.27:0/3948607044 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.611+0000 7f21b27fc700 1 --2- 172.21.15.27:0/3948607044 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f21b4157410 0x7f21b414e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.611+0000 7f21b2ffd700 1 --2- 172.21.15.27:0/3948607044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21b4069540 0x7f21b414be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.612+0000 7f21b37fe700 1 -- 172.21.15.27:0/3948607044 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f21b4157410 msgr2=0x7f21b414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.612+0000 7f21b37fe700 1 --2- 172.21.15.27:0/3948607044 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f21b4157410 0x7f21b414e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.612+0000 7f21b37fe700 1 -- 172.21.15.27:0/3948607044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21b4069540 msgr2=0x7f21b414be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.612+0000 7f21b37fe700 1 --2- 172.21.15.27:0/3948607044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21b4069540 0x7f21b414be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.612+0000 7f21b37fe700 1 -- 172.21.15.27:0/3948607044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ac009ae0 con 0x7f21b41577e0 2024-01-11T08:11:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.612+0000 7f21b37fe700 1 --2- 172.21.15.27:0/3948607044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b41577e0 0x7f21b414fe90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f21ac012000 tx=0x7f21ac018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.612+0000 7f219affd700 1 -- 172.21.15.27:0/3948607044 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ac008b80 con 0x7f21b41577e0 2024-01-11T08:11:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.612+0000 7f219affd700 1 -- 172.21.15.27:0/3948607044 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ac008ce0 con 0x7f21b41577e0 2024-01-11T08:11:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.612+0000 7f219affd700 1 -- 172.21.15.27:0/3948607044 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ac0266b0 con 0x7f21b41577e0 2024-01-11T08:11:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.612+0000 7f21b97ff700 1 -- 172.21.15.27:0/3948607044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b40ca430 con 0x7f21b41577e0 2024-01-11T08:11:45.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.612+0000 7f21b97ff700 1 -- 172.21.15.27:0/3948607044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b40ca9f0 con 0x7f21b41577e0 2024-01-11T08:11:45.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.613+0000 7f21b97ff700 1 -- 172.21.15.27:0/3948607044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b416cba0 con 0x7f21b41577e0 2024-01-11T08:11:45.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.614+0000 7f219affd700 1 -- 172.21.15.27:0/3948607044 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f21ac003040 con 0x7f21b41577e0 2024-01-11T08:11:45.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.616+0000 7f219affd700 1 --2- 172.21.15.27:0/3948607044 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f21a0067130 0x7f21a00695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:45.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.617+0000 7f219affd700 1 -- 172.21.15.27:0/3948607044 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f21ac086430 con 0x7f21b41577e0 2024-01-11T08:11:45.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.617+0000 7f21b2ffd700 1 --2- 172.21.15.27:0/3948607044 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f21a0067130 0x7f21a00695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:45.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.617+0000 7f21b2ffd700 1 --2- 172.21.15.27:0/3948607044 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f21a0067130 0x7f21a00695e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009d30 tx=0x7f21a4008040 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.621+0000 7f219affd700 1 -- 172.21.15.27:0/3948607044 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f21b416cba0 con 0x7f21b41577e0 2024-01-11T08:11:45.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.850+0000 7f21b97ff700 1 -- 172.21.15.27:0/3948607044 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir7", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f21b416cba0 con 0x7f21a0067130 2024-01-11T08:11:45.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.926+0000 7f219affd700 1 -- 172.21.15.27:0/3948607044 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f21b416cba0 con 0x7f21a0067130 2024-01-11T08:11:45.926 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path /snap_test_dir1/dir7 2024-01-11T08:11:45.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.928+0000 7f21b97ff700 1 -- 172.21.15.27:0/3948607044 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f21a0067130 msgr2=0x7f21a00695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:45.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.928+0000 7f21b97ff700 1 --2- 172.21.15.27:0/3948607044 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f21a0067130 0x7f21a00695e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009d30 tx=0x7f21a4008040 comp rx=0 tx=0).stop 2024-01-11T08:11:45.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.928+0000 7f21b97ff700 1 -- 172.21.15.27:0/3948607044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b41577e0 msgr2=0x7f21b414fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:45.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.928+0000 7f21b97ff700 1 --2- 172.21.15.27:0/3948607044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b41577e0 0x7f21b414fe90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f21ac012000 tx=0x7f21ac018000 comp rx=0 tx=0).stop 2024-01-11T08:11:45.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.928+0000 7f21b97ff700 1 -- 172.21.15.27:0/3948607044 shutdown_connections 2024-01-11T08:11:45.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.928+0000 7f21b97ff700 1 --2- 172.21.15.27:0/3948607044 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f21a0067130 0x7f21a00695e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.928+0000 7f21b97ff700 1 --2- 172.21.15.27:0/3948607044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b41577e0 0x7f21b414fe90 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.928+0000 7f21b97ff700 1 --2- 172.21.15.27:0/3948607044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21b4069540 0x7f21b414be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.928+0000 7f21b97ff700 1 --2- 172.21.15.27:0/3948607044 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f21b4157410 0x7f21b414e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:45.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.928+0000 7f21b97ff700 1 -- 172.21.15.27:0/3948607044 >> 172.21.15.27:0/3948607044 conn(0x7f21b40680a0 msgr2=0x7f21b40c8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:45.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.928+0000 7f21b97ff700 1 -- 172.21.15.27:0/3948607044 shutdown_connections 2024-01-11T08:11:45.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:45.928+0000 7f21b97ff700 1 -- 172.21.15.27:0/3948607044 wait complete. 2024-01-11T08:11:45.946 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2024-01-11T08:11:45.946 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:11:45.946 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2024-01-11T08:11:45.995 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir8 1m 2024-01-11T08:11:46.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.200+0000 7fcc07fff700 1 Processor -- start 2024-01-11T08:11:46.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.200+0000 7fcc07fff700 1 -- start start 2024-01-11T08:11:46.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.202+0000 7fcc07fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc000ccd70 0x7fcc000cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:46.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.202+0000 7fcc07fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcc0015ae70 0x7fcc0015b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:46.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.202+0000 7fcc07fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc000cf850 con 0x7fcc0015a1b0 2024-01-11T08:11:46.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.202+0000 7fcc07fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc0015ac90 con 0x7fcc000ccd70 2024-01-11T08:11:46.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.202+0000 7fcc07fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc000cec40 con 0x7fcc0015ae70 2024-01-11T08:11:46.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.202+0000 7fcc0559a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc000ccd70 0x7fcc000cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:46.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.202+0000 7fcc0559a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc000ccd70 0x7fcc000cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55132/0 (socket says 172.21.15.27:55132) 2024-01-11T08:11:46.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.202+0000 7fcc0559a700 1 -- 172.21.15.27:0/2559883277 learned_addr learned my addr 172.21.15.27:0/2559883277 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:46.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.202+0000 7fcc05d9b700 1 --1- 172.21.15.27:0/2559883277 >> v1:172.21.15.27:6789/0 conn(0x7fcc0015a1b0 0x7fcc0015a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45456/0 (socket says 172.21.15.27:45456) 2024-01-11T08:11:46.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.202+0000 7fcc0559a700 1 -- 172.21.15.27:0/2559883277 >> v2:172.21.15.77:3300/0 conn(0x7fcc0015ae70 msgr2=0x7fcc0015b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:46.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.202+0000 7fcc0559a700 1 --2- 172.21.15.27:0/2559883277 >> v2:172.21.15.77:3300/0 conn(0x7fcc0015ae70 0x7fcc0015b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:46.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.202+0000 7fcc0559a700 1 -- 172.21.15.27:0/2559883277 >> v1:172.21.15.27:6789/0 conn(0x7fcc0015a1b0 legacy=0x7fcc0015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:46.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.202+0000 7fcc0559a700 1 -- 172.21.15.27:0/2559883277 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc000cef20 con 0x7fcc000ccd70 2024-01-11T08:11:46.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.203+0000 7fcc0559a700 1 --2- 172.21.15.27:0/2559883277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc000ccd70 0x7fcc000cd210 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fcc00089810 tx=0x7fcbf0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b03ee7cd22f51af server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:46.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.203+0000 7fcc04d99700 1 -- 172.21.15.27:0/2559883277 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf000b860 con 0x7fcc000ccd70 2024-01-11T08:11:46.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.203+0000 7fcc04d99700 1 -- 172.21.15.27:0/2559883277 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf000b9c0 con 0x7fcc000ccd70 2024-01-11T08:11:46.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.203+0000 7fcc04d99700 1 -- 172.21.15.27:0/2559883277 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf000bd60 con 0x7fcc000ccd70 2024-01-11T08:11:46.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.203+0000 7fcc07fff700 1 -- 172.21.15.27:0/2559883277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc000ccd70 msgr2=0x7fcc000cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:46.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.203+0000 7fcc07fff700 1 --2- 172.21.15.27:0/2559883277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc000ccd70 0x7fcc000cd210 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fcc00089810 tx=0x7fcbf0009250 comp rx=0 tx=0).stop 2024-01-11T08:11:46.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.204+0000 7fcc07fff700 1 -- 172.21.15.27:0/2559883277 shutdown_connections 2024-01-11T08:11:46.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.204+0000 7fcc07fff700 1 --2- 172.21.15.27:0/2559883277 >> v2:172.21.15.77:3300/0 conn(0x7fcc0015ae70 0x7fcc0015b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:46.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.204+0000 7fcc07fff700 1 --2- 172.21.15.27:0/2559883277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc000ccd70 0x7fcc000cd210 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:46.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.204+0000 7fcc07fff700 1 -- 172.21.15.27:0/2559883277 >> 172.21.15.27:0/2559883277 conn(0x7fcc0006ed40 msgr2=0x7fcc0006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:46.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.204+0000 7fcc07fff700 1 -- 172.21.15.27:0/2559883277 shutdown_connections 2024-01-11T08:11:46.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.204+0000 7fcc07fff700 1 -- 172.21.15.27:0/2559883277 wait complete. 2024-01-11T08:11:46.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.205+0000 7fcc07fff700 1 Processor -- start 2024-01-11T08:11:46.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.205+0000 7fcc07fff700 1 -- start start 2024-01-11T08:11:46.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.206+0000 7fcc07fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc000ccd70 0x7fcc000c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:46.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.206+0000 7fcc07fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc0015a1b0 0x7fcc000c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:46.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.206+0000 7fcc07fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc0015ae70 0x7fcc000ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:46.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.206+0000 7fcc07fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc00157af0 con 0x7fcc0015ae70 2024-01-11T08:11:46.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.206+0000 7fcc07fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc000c79a0 con 0x7fcc0015a1b0 2024-01-11T08:11:46.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.206+0000 7fcc07fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc00156f20 con 0x7fcc000ccd70 2024-01-11T08:11:46.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.207+0000 7fcc0659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc0015ae70 0x7fcc000ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:46.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.207+0000 7fcc05d9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc000ccd70 0x7fcc000c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:46.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.207+0000 7fcc0559a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc0015a1b0 0x7fcc000c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:46.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.207+0000 7fcc0659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc0015ae70 0x7fcc000ca0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55142/0 (socket says 172.21.15.27:55142) 2024-01-11T08:11:46.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.207+0000 7fcc05d9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc000ccd70 0x7fcc000c9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:45638/0 (socket says 172.21.15.27:45638) 2024-01-11T08:11:46.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.207+0000 7fcc0559a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc0015a1b0 0x7fcc000c9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:44924/0 (socket says 172.21.15.27:44924) 2024-01-11T08:11:46.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.207+0000 7fcc0659c700 1 -- 172.21.15.27:0/1644179117 learned_addr learned my addr 172.21.15.27:0/1644179117 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:46.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.207+0000 7fcc0659c700 1 -- 172.21.15.27:0/1644179117 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc000ccd70 msgr2=0x7fcc000c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:46.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.207+0000 7fcc0659c700 1 --2- 172.21.15.27:0/1644179117 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc000ccd70 0x7fcc000c9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:46.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.207+0000 7fcc0659c700 1 -- 172.21.15.27:0/1644179117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc0015a1b0 msgr2=0x7fcc000c9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:46.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.207+0000 7fcc0659c700 1 --2- 172.21.15.27:0/1644179117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc0015a1b0 0x7fcc000c9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:46.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.207+0000 7fcc0659c700 1 -- 172.21.15.27:0/1644179117 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf00094f0 con 0x7fcc0015ae70 2024-01-11T08:11:46.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.207+0000 7fcc0659c700 1 --2- 172.21.15.27:0/1644179117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc0015ae70 0x7fcc000ca0d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800bc30 tx=0x7fcbf8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:46.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.207+0000 7fcbf5ffb700 1 -- 172.21.15.27:0/1644179117 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf80092e0 con 0x7fcc0015ae70 2024-01-11T08:11:46.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.208+0000 7fcbf5ffb700 1 -- 172.21.15.27:0/1644179117 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf800c630 con 0x7fcc0015ae70 2024-01-11T08:11:46.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.208+0000 7fcc07fff700 1 -- 172.21.15.27:0/1644179117 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc001571c0 con 0x7fcc0015ae70 2024-01-11T08:11:46.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.208+0000 7fcbf5ffb700 1 -- 172.21.15.27:0/1644179117 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf800ca10 con 0x7fcc0015ae70 2024-01-11T08:11:46.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.208+0000 7fcc07fff700 1 -- 172.21.15.27:0/1644179117 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc00157780 con 0x7fcc0015ae70 2024-01-11T08:11:46.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.209+0000 7fcc07fff700 1 -- 172.21.15.27:0/1644179117 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc00157c60 con 0x7fcc0015ae70 2024-01-11T08:11:46.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.210+0000 7fcbf5ffb700 1 -- 172.21.15.27:0/1644179117 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7fcbf8012070 con 0x7fcc0015ae70 2024-01-11T08:11:46.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.211+0000 7fcbf5ffb700 1 --2- 172.21.15.27:0/1644179117 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcbe4067130 0x7fcbe40695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:46.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.211+0000 7fcbf5ffb700 1 -- 172.21.15.27:0/1644179117 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fcbf8084610 con 0x7fcc0015ae70 2024-01-11T08:11:46.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.211+0000 7fcc05d9b700 1 --2- 172.21.15.27:0/1644179117 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcbe4067130 0x7fcbe40695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:46.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.212+0000 7fcc05d9b700 1 --2- 172.21.15.27:0/1644179117 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcbe4067130 0x7fcbe40695e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc004fd0 tx=0x7fcbfc007eb0 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:46.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.215+0000 7fcbf5ffb700 1 -- 172.21.15.27:0/1644179117 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcbf8051460 con 0x7fcc0015ae70 2024-01-11T08:11:46.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.446+0000 7fcc07fff700 1 -- 172.21.15.27:0/1644179117 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir8", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc0004c3c0 con 0x7fcbe4067130 2024-01-11T08:11:46.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.519+0000 7fcbf5ffb700 1 -- 172.21.15.27:0/1644179117 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fcc0004c3c0 con 0x7fcbe4067130 2024-01-11T08:11:46.519 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path /snap_test_dir1/dir8 2024-01-11T08:11:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.521+0000 7fcc07fff700 1 -- 172.21.15.27:0/1644179117 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcbe4067130 msgr2=0x7fcbe40695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.521+0000 7fcc07fff700 1 --2- 172.21.15.27:0/1644179117 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcbe4067130 0x7fcbe40695e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc004fd0 tx=0x7fcbfc007eb0 comp rx=0 tx=0).stop 2024-01-11T08:11:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.521+0000 7fcc07fff700 1 -- 172.21.15.27:0/1644179117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc0015ae70 msgr2=0x7fcc000ca0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.521+0000 7fcc07fff700 1 --2- 172.21.15.27:0/1644179117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc0015ae70 0x7fcc000ca0d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800bc30 tx=0x7fcbf8009840 comp rx=0 tx=0).stop 2024-01-11T08:11:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.522+0000 7fcc07fff700 1 -- 172.21.15.27:0/1644179117 shutdown_connections 2024-01-11T08:11:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.522+0000 7fcc07fff700 1 --2- 172.21.15.27:0/1644179117 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcbe4067130 0x7fcbe40695e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.522+0000 7fcc07fff700 1 --2- 172.21.15.27:0/1644179117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc0015ae70 0x7fcc000ca0d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.522+0000 7fcc07fff700 1 --2- 172.21.15.27:0/1644179117 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc000ccd70 0x7fcc000c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.522+0000 7fcc07fff700 1 --2- 172.21.15.27:0/1644179117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc0015a1b0 0x7fcc000c9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.522+0000 7fcc07fff700 1 -- 172.21.15.27:0/1644179117 >> 172.21.15.27:0/1644179117 conn(0x7fcc0006ed40 msgr2=0x7fcc000cd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.522+0000 7fcc07fff700 1 -- 172.21.15.27:0/1644179117 shutdown_connections 2024-01-11T08:11:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.522+0000 7fcc07fff700 1 -- 172.21.15.27:0/1644179117 wait complete. 2024-01-11T08:11:46.539 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2024-01-11T08:11:46.539 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:11:46.539 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2024-01-11T08:11:46.589 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir9 1m 2024-01-11T08:11:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.791+0000 7fd06dfff700 1 Processor -- start 2024-01-11T08:11:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.792+0000 7fd06dfff700 1 -- start start 2024-01-11T08:11:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.793+0000 7fd06dfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd068157410 0x7fd0680ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.793+0000 7fd06dfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0681577e0 0x7fd0680ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.793+0000 7fd06dfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd068069540 0x7fd0680ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.793+0000 7fd06dfff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0680699d0 con 0x7fd068069540 2024-01-11T08:11:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.793+0000 7fd06dfff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd068069cb0 con 0x7fd068157410 2024-01-11T08:11:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.793+0000 7fd06dfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd068069f90 con 0x7fd0681577e0 2024-01-11T08:11:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.793+0000 7fd067fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd068069540 0x7fd0680ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.793+0000 7fd067fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd068069540 0x7fd0680ae540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55154/0 (socket says 172.21.15.27:55154) 2024-01-11T08:11:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.793+0000 7fd067fff700 1 -- 172.21.15.27:0/4197478535 learned_addr learned my addr 172.21.15.27:0/4197478535 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.793+0000 7fd0677fe700 1 --2- 172.21.15.27:0/4197478535 >> v2:172.21.15.77:3300/0 conn(0x7fd068157410 0x7fd0680ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.794+0000 7fd066ffd700 1 --2- 172.21.15.27:0/4197478535 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0681577e0 0x7fd0680ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.794+0000 7fd067fff700 1 -- 172.21.15.27:0/4197478535 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0681577e0 msgr2=0x7fd0680ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.794+0000 7fd067fff700 1 --2- 172.21.15.27:0/4197478535 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0681577e0 0x7fd0680ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.794+0000 7fd067fff700 1 -- 172.21.15.27:0/4197478535 >> v2:172.21.15.77:3300/0 conn(0x7fd068157410 msgr2=0x7fd0680ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.794+0000 7fd067fff700 1 --2- 172.21.15.27:0/4197478535 >> v2:172.21.15.77:3300/0 conn(0x7fd068157410 0x7fd0680ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.794+0000 7fd067fff700 1 -- 172.21.15.27:0/4197478535 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd06806a270 con 0x7fd068069540 2024-01-11T08:11:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.794+0000 7fd0677fe700 1 --2- 172.21.15.27:0/4197478535 >> v2:172.21.15.77:3300/0 conn(0x7fd068157410 0x7fd0680ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.794+0000 7fd067fff700 1 --2- 172.21.15.27:0/4197478535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd068069540 0x7fd0680ae540 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd068070c90 tx=0x7fd060009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1a38da08ac2d54c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.794+0000 7fd0667fc700 1 -- 172.21.15.27:0/4197478535 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06000e070 con 0x7fd068069540 2024-01-11T08:11:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.794+0000 7fd0667fc700 1 -- 172.21.15.27:0/4197478535 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06000ba50 con 0x7fd068069540 2024-01-11T08:11:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.794+0000 7fd0667fc700 1 -- 172.21.15.27:0/4197478535 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0600092e0 con 0x7fd068069540 2024-01-11T08:11:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.795+0000 7fd06dfff700 1 -- 172.21.15.27:0/4197478535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd068069540 msgr2=0x7fd0680ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.795+0000 7fd06dfff700 1 --2- 172.21.15.27:0/4197478535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd068069540 0x7fd0680ae540 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd068070c90 tx=0x7fd060009840 comp rx=0 tx=0).stop 2024-01-11T08:11:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.795+0000 7fd06dfff700 1 -- 172.21.15.27:0/4197478535 shutdown_connections 2024-01-11T08:11:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.795+0000 7fd06dfff700 1 --2- 172.21.15.27:0/4197478535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd068069540 0x7fd0680ae540 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.795+0000 7fd06dfff700 1 --2- 172.21.15.27:0/4197478535 >> v2:172.21.15.77:3300/0 conn(0x7fd068157410 0x7fd0680ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.795+0000 7fd06dfff700 1 --2- 172.21.15.27:0/4197478535 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0681577e0 0x7fd0680ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.795+0000 7fd06dfff700 1 -- 172.21.15.27:0/4197478535 >> 172.21.15.27:0/4197478535 conn(0x7fd0680680a0 msgr2=0x7fd0680cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.795+0000 7fd06dfff700 1 -- 172.21.15.27:0/4197478535 shutdown_connections 2024-01-11T08:11:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.795+0000 7fd06dfff700 1 -- 172.21.15.27:0/4197478535 wait complete. 2024-01-11T08:11:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.797+0000 7fd06dfff700 1 Processor -- start 2024-01-11T08:11:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.797+0000 7fd06dfff700 1 -- start start 2024-01-11T08:11:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.798+0000 7fd06dfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd068069540 0x7fd068154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.798+0000 7fd06dfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd068157410 0x7fd0680c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.798+0000 7fd06dfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0681577e0 0x7fd068151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.798+0000 7fd06dfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0681555a0 con 0x7fd0681577e0 2024-01-11T08:11:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.798+0000 7fd06dfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd068155880 con 0x7fd068157410 2024-01-11T08:11:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.798+0000 7fd06dfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd068155b60 con 0x7fd068069540 2024-01-11T08:11:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.798+0000 7fd067fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0681577e0 0x7fd068151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.798+0000 7fd067fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0681577e0 0x7fd068151830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55164/0 (socket says 172.21.15.27:55164) 2024-01-11T08:11:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.798+0000 7fd067fff700 1 -- 172.21.15.27:0/423742969 learned_addr learned my addr 172.21.15.27:0/423742969 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.798+0000 7fd066ffd700 1 --2- 172.21.15.27:0/423742969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd068157410 0x7fd0680c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.798+0000 7fd067fff700 1 -- 172.21.15.27:0/423742969 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd068069540 msgr2=0x7fd068154f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.798+0000 7fd067fff700 1 --2- 172.21.15.27:0/423742969 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd068069540 0x7fd068154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.798+0000 7fd067fff700 1 -- 172.21.15.27:0/423742969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd068157410 msgr2=0x7fd0680c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.798+0000 7fd067fff700 1 --2- 172.21.15.27:0/423742969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd068157410 0x7fd0680c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.798+0000 7fd067fff700 1 -- 172.21.15.27:0/423742969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd060009ae0 con 0x7fd0681577e0 2024-01-11T08:11:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.798+0000 7fd066ffd700 1 --2- 172.21.15.27:0/423742969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd068157410 0x7fd0680c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:11:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.799+0000 7fd067fff700 1 --2- 172.21.15.27:0/423742969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0681577e0 0x7fd068151830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd068069490 tx=0x7fd060018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.799+0000 7fd04f7fe700 1 -- 172.21.15.27:0/423742969 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0600032f0 con 0x7fd0681577e0 2024-01-11T08:11:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.799+0000 7fd04f7fe700 1 -- 172.21.15.27:0/423742969 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd060008e80 con 0x7fd0681577e0 2024-01-11T08:11:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.799+0000 7fd06dfff700 1 -- 172.21.15.27:0/423742969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd068155e40 con 0x7fd0681577e0 2024-01-11T08:11:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.799+0000 7fd04f7fe700 1 -- 172.21.15.27:0/423742969 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd060026690 con 0x7fd0681577e0 2024-01-11T08:11:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.799+0000 7fd06dfff700 1 -- 172.21.15.27:0/423742969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd068152ba0 con 0x7fd0681577e0 2024-01-11T08:11:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.800+0000 7fd06dfff700 1 -- 172.21.15.27:0/423742969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd06804c3c0 con 0x7fd0681577e0 2024-01-11T08:11:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.801+0000 7fd04f7fe700 1 -- 172.21.15.27:0/423742969 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7fd060003040 con 0x7fd0681577e0 2024-01-11T08:11:46.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.802+0000 7fd04f7fe700 1 --2- 172.21.15.27:0/423742969 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd054067130 0x7fd0540695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:46.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.802+0000 7fd04f7fe700 1 -- 172.21.15.27:0/423742969 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fd060086aa0 con 0x7fd0681577e0 2024-01-11T08:11:46.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.803+0000 7fd0677fe700 1 --2- 172.21.15.27:0/423742969 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd054067130 0x7fd0540695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:46.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.803+0000 7fd0677fe700 1 --2- 172.21.15.27:0/423742969 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd054067130 0x7fd0540695e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd058006ea0 tx=0x7fd058008040 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:46.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:46.806+0000 7fd04f7fe700 1 -- 172.21.15.27:0/423742969 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd0600537e0 con 0x7fd0681577e0 2024-01-11T08:11:47.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.036+0000 7fd06dfff700 1 -- 172.21.15.27:0/423742969 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir9", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0681559f0 con 0x7fd054067130 2024-01-11T08:11:47.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.104+0000 7fd04f7fe700 1 -- 172.21.15.27:0/423742969 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fd0681559f0 con 0x7fd054067130 2024-01-11T08:11:47.104 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path /snap_test_dir1/dir9 2024-01-11T08:11:47.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.107+0000 7fd06dfff700 1 -- 172.21.15.27:0/423742969 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd054067130 msgr2=0x7fd0540695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:47.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.107+0000 7fd06dfff700 1 --2- 172.21.15.27:0/423742969 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd054067130 0x7fd0540695e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd058006ea0 tx=0x7fd058008040 comp rx=0 tx=0).stop 2024-01-11T08:11:47.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.107+0000 7fd06dfff700 1 -- 172.21.15.27:0/423742969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0681577e0 msgr2=0x7fd068151830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:47.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.107+0000 7fd06dfff700 1 --2- 172.21.15.27:0/423742969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0681577e0 0x7fd068151830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd068069490 tx=0x7fd060018000 comp rx=0 tx=0).stop 2024-01-11T08:11:47.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.107+0000 7fd06dfff700 1 -- 172.21.15.27:0/423742969 shutdown_connections 2024-01-11T08:11:47.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.107+0000 7fd06dfff700 1 --2- 172.21.15.27:0/423742969 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd054067130 0x7fd0540695e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:47.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.107+0000 7fd06dfff700 1 --2- 172.21.15.27:0/423742969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0681577e0 0x7fd068151830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:47.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.107+0000 7fd06dfff700 1 --2- 172.21.15.27:0/423742969 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd068069540 0x7fd068154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:47.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.107+0000 7fd06dfff700 1 --2- 172.21.15.27:0/423742969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd068157410 0x7fd0680c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:47.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.107+0000 7fd06dfff700 1 -- 172.21.15.27:0/423742969 >> 172.21.15.27:0/423742969 conn(0x7fd0680680a0 msgr2=0x7fd0680c6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:47.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.107+0000 7fd06dfff700 1 -- 172.21.15.27:0/423742969 shutdown_connections 2024-01-11T08:11:47.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.107+0000 7fd06dfff700 1 -- 172.21.15.27:0/423742969 wait complete. 2024-01-11T08:11:47.124 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2024-01-11T08:11:47.124 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir0 1m 2024-01-11T08:11:47.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.326+0000 7fd0df9ff700 1 Processor -- start 2024-01-11T08:11:47.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.326+0000 7fd0df9ff700 1 -- start start 2024-01-11T08:11:47.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.328+0000 7fd0df9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd0d81577e0 0x7fd0d80ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:47.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.328+0000 7fd0df9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0d8069540 0x7fd0d80ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:47.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.328+0000 7fd0df9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d80699d0 con 0x7fd0d8157410 2024-01-11T08:11:47.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.328+0000 7fd0df9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d8069cb0 con 0x7fd0d81577e0 2024-01-11T08:11:47.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.328+0000 7fd0df9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d8069f90 con 0x7fd0d8069540 2024-01-11T08:11:47.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.328+0000 7fd0ddf9c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0d8069540 0x7fd0d80ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:47.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.328+0000 7fd0dcf9a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd0d81577e0 0x7fd0d80ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:47.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.329+0000 7fd0ddf9c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0d8069540 0x7fd0d80ae540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:45664/0 (socket says 172.21.15.27:45664) 2024-01-11T08:11:47.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.329+0000 7fd0dcf9a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd0d81577e0 0x7fd0d80ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:44942/0 (socket says 172.21.15.27:44942) 2024-01-11T08:11:47.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.329+0000 7fd0ddf9c700 1 -- 172.21.15.27:0/1975149108 learned_addr learned my addr 172.21.15.27:0/1975149108 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:47.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.329+0000 7fd0dcf9a700 1 -- 172.21.15.27:0/1975149108 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0d8069540 msgr2=0x7fd0d80ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:47.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.329+0000 7fd0dcf9a700 1 --2- 172.21.15.27:0/1975149108 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0d8069540 0x7fd0d80ae540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:47.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.329+0000 7fd0dcf9a700 1 -- 172.21.15.27:0/1975149108 >> v1:172.21.15.77:6789/0 conn(0x7fd0d8157410 legacy=0x7fd0d80ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:47.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.329+0000 7fd0dcf9a700 1 -- 172.21.15.27:0/1975149108 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d806a270 con 0x7fd0d81577e0 2024-01-11T08:11:47.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.329+0000 7fd0ddf9c700 1 --2- 172.21.15.27:0/1975149108 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0d8069540 0x7fd0d80ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:47.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.329+0000 7fd0dcf9a700 1 --2- 172.21.15.27:0/1975149108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0d81577e0 0x7fd0d80ae000 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd0c80098e0 tx=0x7fd0c80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3495260105903886 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:47.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.330+0000 7fd0cffff700 1 -- 172.21.15.27:0/1975149108 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c800b640 con 0x7fd0d81577e0 2024-01-11T08:11:47.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.330+0000 7fd0cffff700 1 -- 172.21.15.27:0/1975149108 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c800b7a0 con 0x7fd0d81577e0 2024-01-11T08:11:47.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.330+0000 7fd0cffff700 1 -- 172.21.15.27:0/1975149108 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c800bb40 con 0x7fd0d81577e0 2024-01-11T08:11:47.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.330+0000 7fd0df9ff700 1 -- 172.21.15.27:0/1975149108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0d81577e0 msgr2=0x7fd0d80ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:47.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.330+0000 7fd0df9ff700 1 --2- 172.21.15.27:0/1975149108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0d81577e0 0x7fd0d80ae000 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd0c80098e0 tx=0x7fd0c80091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:47.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.330+0000 7fd0df9ff700 1 -- 172.21.15.27:0/1975149108 shutdown_connections 2024-01-11T08:11:47.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.330+0000 7fd0df9ff700 1 --2- 172.21.15.27:0/1975149108 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0d8069540 0x7fd0d80ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:47.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.330+0000 7fd0df9ff700 1 --2- 172.21.15.27:0/1975149108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0d81577e0 0x7fd0d80ae000 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:47.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.330+0000 7fd0df9ff700 1 -- 172.21.15.27:0/1975149108 >> 172.21.15.27:0/1975149108 conn(0x7fd0d80680a0 msgr2=0x7fd0d80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:47.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.330+0000 7fd0df9ff700 1 -- 172.21.15.27:0/1975149108 shutdown_connections 2024-01-11T08:11:47.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.330+0000 7fd0df9ff700 1 -- 172.21.15.27:0/1975149108 wait complete. 2024-01-11T08:11:47.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.332+0000 7fd0df9ff700 1 Processor -- start 2024-01-11T08:11:47.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.332+0000 7fd0df9ff700 1 -- start start 2024-01-11T08:11:47.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.333+0000 7fd0df9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d8069540 0x7fd0d814e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:47.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.333+0000 7fd0df9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0d8157410 0x7fd0d80b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:47.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.333+0000 7fd0df9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0d81577e0 0x7fd0d80ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:47.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.333+0000 7fd0df9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d815bdd0 con 0x7fd0d8069540 2024-01-11T08:11:47.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.333+0000 7fd0df9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d814e940 con 0x7fd0d8157410 2024-01-11T08:11:47.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.333+0000 7fd0df9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d815bfe0 con 0x7fd0d81577e0 2024-01-11T08:11:47.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.333+0000 7fd0dd79b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d8069540 0x7fd0d814e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:47.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.333+0000 7fd0dd79b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d8069540 0x7fd0d814e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55170/0 (socket says 172.21.15.27:55170) 2024-01-11T08:11:47.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.333+0000 7fd0dd79b700 1 -- 172.21.15.27:0/1240213167 learned_addr learned my addr 172.21.15.27:0/1240213167 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:47.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.333+0000 7fd0dcf9a700 1 --2- 172.21.15.27:0/1240213167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0d8157410 0x7fd0d80b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:47.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.333+0000 7fd0ddf9c700 1 --2- 172.21.15.27:0/1240213167 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0d81577e0 0x7fd0d80ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:47.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.334+0000 7fd0dd79b700 1 -- 172.21.15.27:0/1240213167 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0d81577e0 msgr2=0x7fd0d80ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:47.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.334+0000 7fd0dd79b700 1 --2- 172.21.15.27:0/1240213167 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0d81577e0 0x7fd0d80ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:47.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.334+0000 7fd0dd79b700 1 -- 172.21.15.27:0/1240213167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0d8157410 msgr2=0x7fd0d80b0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:47.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.334+0000 7fd0dd79b700 1 --2- 172.21.15.27:0/1240213167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0d8157410 0x7fd0d80b0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:47.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.334+0000 7fd0dd79b700 1 -- 172.21.15.27:0/1240213167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d0009ae0 con 0x7fd0d8069540 2024-01-11T08:11:47.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.334+0000 7fd0dd79b700 1 --2- 172.21.15.27:0/1240213167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d8069540 0x7fd0d814e400 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400c7c0 tx=0x7fd0d4007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:47.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.334+0000 7fd0cd7fa700 1 -- 172.21.15.27:0/1240213167 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d4011590 con 0x7fd0d8069540 2024-01-11T08:11:47.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.334+0000 7fd0cd7fa700 1 -- 172.21.15.27:0/1240213167 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d40116f0 con 0x7fd0d8069540 2024-01-11T08:11:47.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.334+0000 7fd0df9ff700 1 -- 172.21.15.27:0/1240213167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c8009420 con 0x7fd0d8069540 2024-01-11T08:11:47.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.334+0000 7fd0cd7fa700 1 -- 172.21.15.27:0/1240213167 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d4011ab0 con 0x7fd0d8069540 2024-01-11T08:11:47.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.334+0000 7fd0df9ff700 1 -- 172.21.15.27:0/1240213167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d815c5a0 con 0x7fd0d8069540 2024-01-11T08:11:47.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.335+0000 7fd0df9ff700 1 -- 172.21.15.27:0/1240213167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d815c120 con 0x7fd0d8069540 2024-01-11T08:11:47.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.336+0000 7fd0cd7fa700 1 -- 172.21.15.27:0/1240213167 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7fd0d401b3b0 con 0x7fd0d8069540 2024-01-11T08:11:47.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.337+0000 7fd0cd7fa700 1 --2- 172.21.15.27:0/1240213167 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd0bc067130 0x7fd0bc0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:47.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.337+0000 7fd0cd7fa700 1 -- 172.21.15.27:0/1240213167 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fd0d4083d70 con 0x7fd0d8069540 2024-01-11T08:11:47.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.338+0000 7fd0dcf9a700 1 --2- 172.21.15.27:0/1240213167 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd0bc067130 0x7fd0bc0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:47.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.338+0000 7fd0dcf9a700 1 --2- 172.21.15.27:0/1240213167 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd0bc067130 0x7fd0bc0695e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd0c80098e0 tx=0x7fd0c8017000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:47.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.341+0000 7fd0cd7fa700 1 -- 172.21.15.27:0/1240213167 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd0d815c120 con 0x7fd0d8069540 2024-01-11T08:11:47.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.573+0000 7fd0df9ff700 1 -- 172.21.15.27:0/1240213167 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0d815c120 con 0x7fd0bc067130 2024-01-11T08:11:47.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.631+0000 7fd0cd7fa700 1 -- 172.21.15.27:0/1240213167 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fd0d815c120 con 0x7fd0bc067130 2024-01-11T08:11:47.631 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule activated for path /snap_test_dir1/dir0 2024-01-11T08:11:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.633+0000 7fd0df9ff700 1 -- 172.21.15.27:0/1240213167 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd0bc067130 msgr2=0x7fd0bc0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.633+0000 7fd0df9ff700 1 --2- 172.21.15.27:0/1240213167 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd0bc067130 0x7fd0bc0695e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd0c80098e0 tx=0x7fd0c8017000 comp rx=0 tx=0).stop 2024-01-11T08:11:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.634+0000 7fd0df9ff700 1 -- 172.21.15.27:0/1240213167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d8069540 msgr2=0x7fd0d814e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.634+0000 7fd0df9ff700 1 --2- 172.21.15.27:0/1240213167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d8069540 0x7fd0d814e400 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400c7c0 tx=0x7fd0d4007f70 comp rx=0 tx=0).stop 2024-01-11T08:11:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.634+0000 7fd0df9ff700 1 -- 172.21.15.27:0/1240213167 shutdown_connections 2024-01-11T08:11:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.634+0000 7fd0df9ff700 1 --2- 172.21.15.27:0/1240213167 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd0bc067130 0x7fd0bc0695e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.634+0000 7fd0df9ff700 1 --2- 172.21.15.27:0/1240213167 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0d81577e0 0x7fd0d80ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.634+0000 7fd0df9ff700 1 --2- 172.21.15.27:0/1240213167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d8069540 0x7fd0d814e400 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.634+0000 7fd0df9ff700 1 --2- 172.21.15.27:0/1240213167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0d8157410 0x7fd0d80b0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.634+0000 7fd0df9ff700 1 -- 172.21.15.27:0/1240213167 >> 172.21.15.27:0/1240213167 conn(0x7fd0d80680a0 msgr2=0x7fd0d815f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.634+0000 7fd0df9ff700 1 -- 172.21.15.27:0/1240213167 shutdown_connections 2024-01-11T08:11:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.634+0000 7fd0df9ff700 1 -- 172.21.15.27:0/1240213167 wait complete. 2024-01-11T08:11:47.651 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2024-01-11T08:11:47.651 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir1 1m 2024-01-11T08:11:47.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.853+0000 7f15511ff700 1 Processor -- start 2024-01-11T08:11:47.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.853+0000 7f15511ff700 1 -- start start 2024-01-11T08:11:47.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.855+0000 7f15511ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f154c0ccd70 0x7f154c0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:47.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.855+0000 7f15511ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f154c15ae70 0x7f154c15b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:47.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.855+0000 7f15511ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f154c0cf850 con 0x7f154c15a1b0 2024-01-11T08:11:47.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.855+0000 7f15511ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f154c15ac90 con 0x7f154c0ccd70 2024-01-11T08:11:47.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.855+0000 7f15511ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f154c0cec40 con 0x7f154c15ae70 2024-01-11T08:11:47.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.855+0000 7f154a59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f154c0ccd70 0x7f154c0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:47.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.855+0000 7f154a59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f154c0ccd70 0x7f154c0cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55184/0 (socket says 172.21.15.27:55184) 2024-01-11T08:11:47.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.855+0000 7f154a59c700 1 -- 172.21.15.27:0/2033576183 learned_addr learned my addr 172.21.15.27:0/2033576183 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:47.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.855+0000 7f154ad9d700 1 --1- 172.21.15.27:0/2033576183 >> v1:172.21.15.27:6789/0 conn(0x7f154c15a1b0 0x7f154c15a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45466/0 (socket says 172.21.15.27:45466) 2024-01-11T08:11:47.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.855+0000 7f154b59e700 1 --2- 172.21.15.27:0/2033576183 >> v2:172.21.15.77:3300/0 conn(0x7f154c15ae70 0x7f154c15b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:47.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.855+0000 7f154a59c700 1 -- 172.21.15.27:0/2033576183 >> v2:172.21.15.77:3300/0 conn(0x7f154c15ae70 msgr2=0x7f154c15b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:47.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.855+0000 7f154a59c700 1 --2- 172.21.15.27:0/2033576183 >> v2:172.21.15.77:3300/0 conn(0x7f154c15ae70 0x7f154c15b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:47.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.855+0000 7f154a59c700 1 -- 172.21.15.27:0/2033576183 >> v1:172.21.15.27:6789/0 conn(0x7f154c15a1b0 legacy=0x7f154c15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:47.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.855+0000 7f154a59c700 1 -- 172.21.15.27:0/2033576183 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c0cef20 con 0x7f154c0ccd70 2024-01-11T08:11:47.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.855+0000 7f1549d9b700 1 -- 172.21.15.27:0/2033576183 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166598278 0 0) 0x7f154c0cf850 con 0x7f154c15a1b0 2024-01-11T08:11:47.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.856+0000 7f154a59c700 1 --2- 172.21.15.27:0/2033576183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c0ccd70 0x7f154c0cd210 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f154c089810 tx=0x7f1540009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a656847d5ad0faf1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:47.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.856+0000 7f1549d9b700 1 -- 172.21.15.27:0/2033576183 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1540009e70 con 0x7f154c0ccd70 2024-01-11T08:11:47.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.856+0000 7f1549d9b700 1 -- 172.21.15.27:0/2033576183 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154000b860 con 0x7f154c0ccd70 2024-01-11T08:11:47.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.856+0000 7f1549d9b700 1 -- 172.21.15.27:0/2033576183 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154000bc00 con 0x7f154c0ccd70 2024-01-11T08:11:47.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.856+0000 7f15511ff700 1 -- 172.21.15.27:0/2033576183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c0ccd70 msgr2=0x7f154c0cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:47.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.856+0000 7f15511ff700 1 --2- 172.21.15.27:0/2033576183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c0ccd70 0x7f154c0cd210 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f154c089810 tx=0x7f1540009250 comp rx=0 tx=0).stop 2024-01-11T08:11:47.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.857+0000 7f15511ff700 1 -- 172.21.15.27:0/2033576183 shutdown_connections 2024-01-11T08:11:47.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.857+0000 7f15511ff700 1 --2- 172.21.15.27:0/2033576183 >> v2:172.21.15.77:3300/0 conn(0x7f154c15ae70 0x7f154c15b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:47.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.857+0000 7f15511ff700 1 --2- 172.21.15.27:0/2033576183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c0ccd70 0x7f154c0cd210 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:47.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.857+0000 7f15511ff700 1 -- 172.21.15.27:0/2033576183 >> 172.21.15.27:0/2033576183 conn(0x7f154c06ed40 msgr2=0x7f154c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:47.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.857+0000 7f15511ff700 1 -- 172.21.15.27:0/2033576183 shutdown_connections 2024-01-11T08:11:47.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.857+0000 7f15511ff700 1 -- 172.21.15.27:0/2033576183 wait complete. 2024-01-11T08:11:47.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.858+0000 7f15511ff700 1 Processor -- start 2024-01-11T08:11:47.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.858+0000 7f15511ff700 1 -- start start 2024-01-11T08:11:47.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.859+0000 7f15511ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c0ccd70 0x7f154c0a6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:47.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.859+0000 7f15511ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f154c15a1b0 0x7f154c15c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.859+0000 7f15511ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f154c15ae70 0x7f154c0c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.859+0000 7f15511ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c0b50e0 con 0x7f154c0ccd70 2024-01-11T08:11:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.859+0000 7f15511ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c0b44d0 con 0x7f154c15ae70 2024-01-11T08:11:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.859+0000 7f15511ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154c0b47b0 con 0x7f154c15a1b0 2024-01-11T08:11:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.860+0000 7f154ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c0ccd70 0x7f154c0a6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.860+0000 7f154ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c0ccd70 0x7f154c0a6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55190/0 (socket says 172.21.15.27:55190) 2024-01-11T08:11:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.860+0000 7f154ad9d700 1 -- 172.21.15.27:0/3843752872 learned_addr learned my addr 172.21.15.27:0/3843752872 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.860+0000 7f154a59c700 1 --2- 172.21.15.27:0/3843752872 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f154c15a1b0 0x7f154c15c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.860+0000 7f154b59e700 1 --2- 172.21.15.27:0/3843752872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f154c15ae70 0x7f154c0c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.860+0000 7f154ad9d700 1 -- 172.21.15.27:0/3843752872 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f154c15a1b0 msgr2=0x7f154c15c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.860+0000 7f154ad9d700 1 --2- 172.21.15.27:0/3843752872 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f154c15a1b0 0x7f154c15c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.860+0000 7f154ad9d700 1 -- 172.21.15.27:0/3843752872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f154c15ae70 msgr2=0x7f154c0c8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.860+0000 7f154ad9d700 1 --2- 172.21.15.27:0/3843752872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f154c15ae70 0x7f154c0c8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:47.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.860+0000 7f154ad9d700 1 -- 172.21.15.27:0/3843752872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15400094f0 con 0x7f154c0ccd70 2024-01-11T08:11:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.860+0000 7f154b59e700 1 --2- 172.21.15.27:0/3843752872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f154c15ae70 0x7f154c0c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.860+0000 7f154ad9d700 1 --2- 172.21.15.27:0/3843752872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c0ccd70 0x7f154c0a6f10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f153c0011e0 tx=0x7f153c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.861+0000 7f1532ffd700 1 -- 172.21.15.27:0/3843752872 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c005a70 con 0x7f154c0ccd70 2024-01-11T08:11:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.861+0000 7f1532ffd700 1 -- 172.21.15.27:0/3843752872 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153c005bd0 con 0x7f154c0ccd70 2024-01-11T08:11:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.861+0000 7f15511ff700 1 -- 172.21.15.27:0/3843752872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f154c0b4a90 con 0x7f154c0ccd70 2024-01-11T08:11:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.861+0000 7f1532ffd700 1 -- 172.21.15.27:0/3843752872 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c0070d0 con 0x7f154c0ccd70 2024-01-11T08:11:47.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.861+0000 7f15511ff700 1 -- 172.21.15.27:0/3843752872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f154c0b4130 con 0x7f154c0ccd70 2024-01-11T08:11:47.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.862+0000 7f15511ff700 1 -- 172.21.15.27:0/3843752872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f154c04c3c0 con 0x7f154c0ccd70 2024-01-11T08:11:47.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.863+0000 7f1532ffd700 1 -- 172.21.15.27:0/3843752872 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f153c007480 con 0x7f154c0ccd70 2024-01-11T08:11:47.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.864+0000 7f1532ffd700 1 --2- 172.21.15.27:0/3843752872 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1538067130 0x7f15380695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:47.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.864+0000 7f1532ffd700 1 -- 172.21.15.27:0/3843752872 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f153c083e00 con 0x7f154c0ccd70 2024-01-11T08:11:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.864+0000 7f154a59c700 1 --2- 172.21.15.27:0/3843752872 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1538067130 0x7f15380695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.865+0000 7f154a59c700 1 --2- 172.21.15.27:0/3843752872 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1538067130 0x7f15380695e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f154c089810 tx=0x7f1540017000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:47.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:47.868+0000 7f1532ffd700 1 -- 172.21.15.27:0/3843752872 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f153c04b5a0 con 0x7f154c0ccd70 2024-01-11T08:11:48.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.100+0000 7f15511ff700 1 -- 172.21.15.27:0/3843752872 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f154c0b5220 con 0x7f1538067130 2024-01-11T08:11:48.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.155+0000 7f1532ffd700 1 -- 172.21.15.27:0/3843752872 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f154000b9d0 con 0x7f1538067130 2024-01-11T08:11:48.157 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule activated for path /snap_test_dir1/dir1 2024-01-11T08:11:48.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.158+0000 7f15511ff700 1 -- 172.21.15.27:0/3843752872 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1538067130 msgr2=0x7f15380695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:48.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.158+0000 7f15511ff700 1 --2- 172.21.15.27:0/3843752872 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1538067130 0x7f15380695e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f154c089810 tx=0x7f1540017000 comp rx=0 tx=0).stop 2024-01-11T08:11:48.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.158+0000 7f15511ff700 1 -- 172.21.15.27:0/3843752872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c0ccd70 msgr2=0x7f154c0a6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:48.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.158+0000 7f15511ff700 1 --2- 172.21.15.27:0/3843752872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c0ccd70 0x7f154c0a6f10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f153c0011e0 tx=0x7f153c007e10 comp rx=0 tx=0).stop 2024-01-11T08:11:48.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.158+0000 7f15511ff700 1 -- 172.21.15.27:0/3843752872 shutdown_connections 2024-01-11T08:11:48.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.158+0000 7f15511ff700 1 --2- 172.21.15.27:0/3843752872 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1538067130 0x7f15380695e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:48.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.158+0000 7f15511ff700 1 --2- 172.21.15.27:0/3843752872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f154c15ae70 0x7f154c0c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:48.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.158+0000 7f15511ff700 1 --2- 172.21.15.27:0/3843752872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c0ccd70 0x7f154c0a6f10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:48.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.158+0000 7f15511ff700 1 --2- 172.21.15.27:0/3843752872 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f154c15a1b0 0x7f154c15c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:48.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.158+0000 7f15511ff700 1 -- 172.21.15.27:0/3843752872 >> 172.21.15.27:0/3843752872 conn(0x7f154c06ed40 msgr2=0x7f154c0c5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:48.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.158+0000 7f15511ff700 1 -- 172.21.15.27:0/3843752872 shutdown_connections 2024-01-11T08:11:48.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.158+0000 7f15511ff700 1 -- 172.21.15.27:0/3843752872 wait complete. 2024-01-11T08:11:48.176 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2024-01-11T08:11:48.176 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir2 1m 2024-01-11T08:11:48.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.379+0000 7f4be63ff700 1 Processor -- start 2024-01-11T08:11:48.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.379+0000 7f4be63ff700 1 -- start start 2024-01-11T08:11:48.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.380+0000 7f4be63ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4be01596f0 0x7f4be00a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.380+0000 7f4be63ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be00c80a0 con 0x7f4be0068d30 2024-01-11T08:11:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.381+0000 7f4be63ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be00cbac0 con 0x7f4be00a9de0 2024-01-11T08:11:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.381+0000 7f4be63ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be00c7490 con 0x7f4be01596f0 2024-01-11T08:11:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.381+0000 7f4bdffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4be01596f0 0x7f4be00a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.381+0000 7f4bdffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4be01596f0 0x7f4be00a98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55202/0 (socket says 172.21.15.27:55202) 2024-01-11T08:11:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.381+0000 7f4bdffff700 1 -- 172.21.15.27:0/4016700779 learned_addr learned my addr 172.21.15.27:0/4016700779 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.381+0000 7f4be499c700 1 --1- 172.21.15.27:0/4016700779 >> v1:172.21.15.27:6789/0 conn(0x7f4be0068d30 0x7f4be00c8530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45468/0 (socket says 172.21.15.27:45468) 2024-01-11T08:11:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.381+0000 7f4bdffff700 1 -- 172.21.15.27:0/4016700779 >> v1:172.21.15.77:6789/0 conn(0x7f4be00a9de0 legacy=0x7f4be0068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.381+0000 7f4bdffff700 1 -- 172.21.15.27:0/4016700779 >> v1:172.21.15.27:6789/0 conn(0x7f4be0068d30 legacy=0x7f4be00c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.381+0000 7f4bdffff700 1 -- 172.21.15.27:0/4016700779 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be00c7770 con 0x7f4be01596f0 2024-01-11T08:11:48.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.381+0000 7f4bdeffd700 1 -- 172.21.15.27:0/4016700779 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867673331 0 0) 0x7f4be00c80a0 con 0x7f4be0068d30 2024-01-11T08:11:48.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.382+0000 7f4bdffff700 1 --2- 172.21.15.27:0/4016700779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4be01596f0 0x7f4be00a98a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4bd00099b0 tx=0x7f4bd0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b57c554015932bd9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:48.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.382+0000 7f4bdeffd700 1 -- 172.21.15.27:0/4016700779 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd000b860 con 0x7f4be01596f0 2024-01-11T08:11:48.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.382+0000 7f4bdeffd700 1 -- 172.21.15.27:0/4016700779 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd000b9c0 con 0x7f4be01596f0 2024-01-11T08:11:48.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.382+0000 7f4bdeffd700 1 -- 172.21.15.27:0/4016700779 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd000bd60 con 0x7f4be01596f0 2024-01-11T08:11:48.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.382+0000 7f4be63ff700 1 -- 172.21.15.27:0/4016700779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4be01596f0 msgr2=0x7f4be00a98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:48.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.382+0000 7f4be63ff700 1 --2- 172.21.15.27:0/4016700779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4be01596f0 0x7f4be00a98a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4bd00099b0 tx=0x7f4bd0009250 comp rx=0 tx=0).stop 2024-01-11T08:11:48.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.382+0000 7f4be63ff700 1 -- 172.21.15.27:0/4016700779 shutdown_connections 2024-01-11T08:11:48.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.382+0000 7f4be63ff700 1 --2- 172.21.15.27:0/4016700779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4be01596f0 0x7f4be00a98a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:48.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.382+0000 7f4be63ff700 1 -- 172.21.15.27:0/4016700779 >> 172.21.15.27:0/4016700779 conn(0x7f4be00c9fb0 msgr2=0x7f4be006ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:48.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.383+0000 7f4be63ff700 1 -- 172.21.15.27:0/4016700779 shutdown_connections 2024-01-11T08:11:48.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.383+0000 7f4be63ff700 1 -- 172.21.15.27:0/4016700779 wait complete. 2024-01-11T08:11:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.384+0000 7f4be63ff700 1 Processor -- start 2024-01-11T08:11:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.384+0000 7f4be63ff700 1 -- start start 2024-01-11T08:11:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.385+0000 7f4be63ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4be0068d30 0x7f4be00ae230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.385+0000 7f4be63ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4be00a9de0 0x7f4be00c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.385+0000 7f4be63ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4be01596f0 0x7f4be01581c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.385+0000 7f4be63ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be00c6080 con 0x7f4be0068d30 2024-01-11T08:11:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.385+0000 7f4be63ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be00c6360 con 0x7f4be00a9de0 2024-01-11T08:11:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.385+0000 7f4be63ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be00c6640 con 0x7f4be01596f0 2024-01-11T08:11:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.385+0000 7f4bdffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4be0068d30 0x7f4be00ae230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.385+0000 7f4bdffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4be0068d30 0x7f4be00ae230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55204/0 (socket says 172.21.15.27:55204) 2024-01-11T08:11:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.385+0000 7f4bdffff700 1 -- 172.21.15.27:0/2170075153 learned_addr learned my addr 172.21.15.27:0/2170075153 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.385+0000 7f4be499c700 1 --2- 172.21.15.27:0/2170075153 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4be01596f0 0x7f4be01581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.385+0000 7f4bdf7fe700 1 --2- 172.21.15.27:0/2170075153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4be00a9de0 0x7f4be00c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.386+0000 7f4bdffff700 1 -- 172.21.15.27:0/2170075153 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4be01596f0 msgr2=0x7f4be01581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.386+0000 7f4bdffff700 1 --2- 172.21.15.27:0/2170075153 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4be01596f0 0x7f4be01581c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.386+0000 7f4bdffff700 1 -- 172.21.15.27:0/2170075153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4be00a9de0 msgr2=0x7f4be00c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.386+0000 7f4bdffff700 1 --2- 172.21.15.27:0/2170075153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4be00a9de0 0x7f4be00c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:48.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.386+0000 7f4bdffff700 1 -- 172.21.15.27:0/2170075153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd00094f0 con 0x7f4be0068d30 2024-01-11T08:11:48.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.386+0000 7f4bdffff700 1 --2- 172.21.15.27:0/2170075153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4be0068d30 0x7f4be00ae230 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0006f50 tx=0x7f4bd0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:48.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.386+0000 7f4bc7fff700 1 -- 172.21.15.27:0/2170075153 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd001f070 con 0x7f4be0068d30 2024-01-11T08:11:48.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.386+0000 7f4bc7fff700 1 -- 172.21.15.27:0/2170075153 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd0011d10 con 0x7f4be0068d30 2024-01-11T08:11:48.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.386+0000 7f4bc7fff700 1 -- 172.21.15.27:0/2170075153 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0025670 con 0x7f4be0068d30 2024-01-11T08:11:48.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.386+0000 7f4be63ff700 1 -- 172.21.15.27:0/2170075153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be00c6920 con 0x7f4be0068d30 2024-01-11T08:11:48.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.387+0000 7f4be63ff700 1 -- 172.21.15.27:0/2170075153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be00c3c70 con 0x7f4be0068d30 2024-01-11T08:11:48.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.387+0000 7f4be63ff700 1 -- 172.21.15.27:0/2170075153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be00c3e60 con 0x7f4be0068d30 2024-01-11T08:11:48.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.388+0000 7f4bc7fff700 1 -- 172.21.15.27:0/2170075153 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f4bd0023070 con 0x7f4be0068d30 2024-01-11T08:11:48.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.389+0000 7f4bc7fff700 1 --2- 172.21.15.27:0/2170075153 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4bcc067130 0x7f4bcc0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:48.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.389+0000 7f4bc7fff700 1 -- 172.21.15.27:0/2170075153 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f4bd0085c60 con 0x7f4be0068d30 2024-01-11T08:11:48.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.390+0000 7f4bdf7fe700 1 --2- 172.21.15.27:0/2170075153 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4bcc067130 0x7f4bcc0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:48.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.390+0000 7f4bdf7fe700 1 --2- 172.21.15.27:0/2170075153 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4bcc067130 0x7f4bcc0695e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4be00b2ad0 tx=0x7f4bd4009040 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:48.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.394+0000 7f4bc7fff700 1 -- 172.21.15.27:0/2170075153 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4be00c3e60 con 0x7f4be0068d30 2024-01-11T08:11:48.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.626+0000 7f4be63ff700 1 -- 172.21.15.27:0/2170075153 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f4be00c3e60 con 0x7f4bcc067130 2024-01-11T08:11:48.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.686+0000 7f4bc7fff700 1 -- 172.21.15.27:0/2170075153 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f4be00c3e60 con 0x7f4bcc067130 2024-01-11T08:11:48.686 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule activated for path /snap_test_dir1/dir2 2024-01-11T08:11:48.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.688+0000 7f4be63ff700 1 -- 172.21.15.27:0/2170075153 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4bcc067130 msgr2=0x7f4bcc0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:48.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.688+0000 7f4be63ff700 1 --2- 172.21.15.27:0/2170075153 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4bcc067130 0x7f4bcc0695e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4be00b2ad0 tx=0x7f4bd4009040 comp rx=0 tx=0).stop 2024-01-11T08:11:48.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.688+0000 7f4be63ff700 1 -- 172.21.15.27:0/2170075153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4be0068d30 msgr2=0x7f4be00ae230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:48.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.688+0000 7f4be63ff700 1 --2- 172.21.15.27:0/2170075153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4be0068d30 0x7f4be00ae230 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0006f50 tx=0x7f4bd0017000 comp rx=0 tx=0).stop 2024-01-11T08:11:48.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.688+0000 7f4be63ff700 1 -- 172.21.15.27:0/2170075153 shutdown_connections 2024-01-11T08:11:48.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.688+0000 7f4be63ff700 1 --2- 172.21.15.27:0/2170075153 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4bcc067130 0x7f4bcc0695e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:48.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.688+0000 7f4be63ff700 1 --2- 172.21.15.27:0/2170075153 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4be01596f0 0x7f4be01581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:48.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.688+0000 7f4be63ff700 1 --2- 172.21.15.27:0/2170075153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4be0068d30 0x7f4be00ae230 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:48.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.688+0000 7f4be63ff700 1 --2- 172.21.15.27:0/2170075153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4be00a9de0 0x7f4be00c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:48.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.688+0000 7f4be63ff700 1 -- 172.21.15.27:0/2170075153 >> 172.21.15.27:0/2170075153 conn(0x7f4be00c9fb0 msgr2=0x7f4be00c22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:48.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.688+0000 7f4be63ff700 1 -- 172.21.15.27:0/2170075153 shutdown_connections 2024-01-11T08:11:48.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.688+0000 7f4be63ff700 1 -- 172.21.15.27:0/2170075153 wait complete. 2024-01-11T08:11:48.706 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2024-01-11T08:11:48.706 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir3 1m 2024-01-11T08:11:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.910+0000 7f5d809ff700 1 Processor -- start 2024-01-11T08:11:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.910+0000 7f5d809ff700 1 -- start start 2024-01-11T08:11:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.911+0000 7f5d809ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5d7c155c50 0x7f5d7c0aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.911+0000 7f5d809ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5d7c0c74f0 0x7f5d7c069910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.911+0000 7f5d809ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c06e480 con 0x7f5d7c0694c0 2024-01-11T08:11:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.911+0000 7f5d809ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d7c0cb5a0 con 0x7f5d7c0c74f0 2024-01-11T08:11:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.911+0000 7f5d809ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d7c06df00 con 0x7f5d7c155c50 2024-01-11T08:11:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.911+0000 7f5d79d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5d7c0694c0 0x7f5d7c0c6de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45472/0 (socket says 172.21.15.27:45472) 2024-01-11T08:11:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.911+0000 7f5d79d9b700 1 -- 172.21.15.27:0/866523654 learned_addr learned my addr 172.21.15.27:0/866523654 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.912+0000 7f5d7ad9d700 1 --2- 172.21.15.27:0/866523654 >> v2:172.21.15.27:3300/0 conn(0x7f5d7c0c74f0 0x7f5d7c069910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.912+0000 7f5d7a59c700 1 --2- 172.21.15.27:0/866523654 >> v2:172.21.15.77:3300/0 conn(0x7f5d7c155c50 0x7f5d7c0aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.912+0000 7f5d7959a700 1 -- 172.21.15.27:0/866523654 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881098001 0 0) 0x7f5d7c06e480 con 0x7f5d7c0694c0 2024-01-11T08:11:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.912+0000 7f5d7959a700 1 -- 172.21.15.27:0/866523654 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d64003570 con 0x7f5d7c0694c0 2024-01-11T08:11:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.912+0000 7f5d7ad9d700 1 -- 172.21.15.27:0/866523654 >> v2:172.21.15.77:3300/0 conn(0x7f5d7c155c50 msgr2=0x7f5d7c0aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.912+0000 7f5d7ad9d700 1 --2- 172.21.15.27:0/866523654 >> v2:172.21.15.77:3300/0 conn(0x7f5d7c155c50 0x7f5d7c0aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.912+0000 7f5d7ad9d700 1 -- 172.21.15.27:0/866523654 >> v1:172.21.15.27:6789/0 conn(0x7f5d7c0694c0 legacy=0x7f5d7c0c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.912+0000 7f5d7ad9d700 1 -- 172.21.15.27:0/866523654 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c069e50 con 0x7f5d7c0c74f0 2024-01-11T08:11:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.912+0000 7f5d7ad9d700 1 --2- 172.21.15.27:0/866523654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0c74f0 0x7f5d7c069910 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c06dcd0 tx=0x7f5d74009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b73fd56922ab3482 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.913+0000 7f5d7959a700 1 -- 172.21.15.27:0/866523654 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7400e070 con 0x7f5d7c0c74f0 2024-01-11T08:11:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.913+0000 7f5d7959a700 1 -- 172.21.15.27:0/866523654 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d7400bb40 con 0x7f5d7c0c74f0 2024-01-11T08:11:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.913+0000 7f5d7959a700 1 -- 172.21.15.27:0/866523654 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d74012430 con 0x7f5d7c0c74f0 2024-01-11T08:11:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.913+0000 7f5d809ff700 1 -- 172.21.15.27:0/866523654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0c74f0 msgr2=0x7f5d7c069910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.913+0000 7f5d809ff700 1 --2- 172.21.15.27:0/866523654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0c74f0 0x7f5d7c069910 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c06dcd0 tx=0x7f5d74009780 comp rx=0 tx=0).stop 2024-01-11T08:11:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.913+0000 7f5d809ff700 1 -- 172.21.15.27:0/866523654 shutdown_connections 2024-01-11T08:11:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.913+0000 7f5d809ff700 1 --2- 172.21.15.27:0/866523654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0c74f0 0x7f5d7c069910 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.913+0000 7f5d809ff700 1 --2- 172.21.15.27:0/866523654 >> v2:172.21.15.77:3300/0 conn(0x7f5d7c155c50 0x7f5d7c0aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.913+0000 7f5d809ff700 1 -- 172.21.15.27:0/866523654 >> 172.21.15.27:0/866523654 conn(0x7f5d7c0680a0 msgr2=0x7f5d7c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.913+0000 7f5d809ff700 1 -- 172.21.15.27:0/866523654 shutdown_connections 2024-01-11T08:11:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.913+0000 7f5d809ff700 1 -- 172.21.15.27:0/866523654 wait complete. 2024-01-11T08:11:48.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.915+0000 7f5d809ff700 1 Processor -- start 2024-01-11T08:11:48.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.915+0000 7f5d809ff700 1 -- start start 2024-01-11T08:11:48.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.916+0000 7f5d809ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0694c0 0x7f5d7c169ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.916+0000 7f5d809ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d7c0c74f0 0x7f5d7c0ae020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.916+0000 7f5d809ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d7c155c50 0x7f5d7c16c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.916+0000 7f5d809ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c0ae560 con 0x7f5d7c0694c0 2024-01-11T08:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.916+0000 7f5d809ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c16d270 con 0x7f5d7c0c74f0 2024-01-11T08:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.916+0000 7f5d809ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c16d550 con 0x7f5d7c155c50 2024-01-11T08:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.916+0000 7f5d7a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0694c0 0x7f5d7c169ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.916+0000 7f5d7a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0694c0 0x7f5d7c169ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55224/0 (socket says 172.21.15.27:55224) 2024-01-11T08:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.916+0000 7f5d7a59c700 1 -- 172.21.15.27:0/484715811 learned_addr learned my addr 172.21.15.27:0/484715811 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.917+0000 7f5d79d9b700 1 --2- 172.21.15.27:0/484715811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d7c0c74f0 0x7f5d7c0ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.917+0000 7f5d7ad9d700 1 --2- 172.21.15.27:0/484715811 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d7c155c50 0x7f5d7c16c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.917+0000 7f5d7a59c700 1 -- 172.21.15.27:0/484715811 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d7c155c50 msgr2=0x7f5d7c16c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.917+0000 7f5d7a59c700 1 --2- 172.21.15.27:0/484715811 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d7c155c50 0x7f5d7c16c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.917+0000 7f5d7a59c700 1 -- 172.21.15.27:0/484715811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d7c0c74f0 msgr2=0x7f5d7c0ae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.917+0000 7f5d7a59c700 1 --2- 172.21.15.27:0/484715811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d7c0c74f0 0x7f5d7c0ae020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.917+0000 7f5d7a59c700 1 -- 172.21.15.27:0/484715811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d74009a20 con 0x7f5d7c0694c0 2024-01-11T08:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.917+0000 7f5d7a59c700 1 --2- 172.21.15.27:0/484715811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0694c0 0x7f5d7c169ce0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00ea00 tx=0x7f5d6c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.917+0000 7f5d6a7fc700 1 -- 172.21.15.27:0/484715811 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c00c850 con 0x7f5d7c0694c0 2024-01-11T08:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.917+0000 7f5d6a7fc700 1 -- 172.21.15.27:0/484715811 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d6c00c9b0 con 0x7f5d7c0694c0 2024-01-11T08:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.917+0000 7f5d809ff700 1 -- 172.21.15.27:0/484715811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7c1694e0 con 0x7f5d7c0694c0 2024-01-11T08:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.917+0000 7f5d6a7fc700 1 -- 172.21.15.27:0/484715811 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c00cd30 con 0x7f5d7c0694c0 2024-01-11T08:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.918+0000 7f5d809ff700 1 -- 172.21.15.27:0/484715811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7c1691b0 con 0x7f5d7c0694c0 2024-01-11T08:11:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.918+0000 7f5d809ff700 1 -- 172.21.15.27:0/484715811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7c169a00 con 0x7f5d7c0694c0 2024-01-11T08:11:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.919+0000 7f5d6a7fc700 1 -- 172.21.15.27:0/484715811 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f5d6c013680 con 0x7f5d7c0694c0 2024-01-11T08:11:48.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.921+0000 7f5d6a7fc700 1 --2- 172.21.15.27:0/484715811 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5d5c067130 0x7f5d5c0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:48.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.922+0000 7f5d6a7fc700 1 -- 172.21.15.27:0/484715811 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f5d6c0850b0 con 0x7f5d7c0694c0 2024-01-11T08:11:48.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.922+0000 7f5d79d9b700 1 --2- 172.21.15.27:0/484715811 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5d5c067130 0x7f5d5c0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:48.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.923+0000 7f5d79d9b700 1 --2- 172.21.15.27:0/484715811 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5d5c067130 0x7f5d5c0695e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c06f720 tx=0x7f5d70009040 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:48.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:48.925+0000 7f5d6a7fc700 1 -- 172.21.15.27:0/484715811 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5d7c169a00 con 0x7f5d7c0694c0 2024-01-11T08:11:48.941 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:11:48.946 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:11:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.159+0000 7f5d809ff700 1 -- 172.21.15.27:0/484715811 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d7c169a00 con 0x7f5d5c067130 2024-01-11T08:11:49.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.213+0000 7f5d6a7fc700 1 -- 172.21.15.27:0/484715811 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f5d7c169a00 con 0x7f5d5c067130 2024-01-11T08:11:49.213 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule activated for path /snap_test_dir1/dir3 2024-01-11T08:11:49.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.216+0000 7f5d809ff700 1 -- 172.21.15.27:0/484715811 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5d5c067130 msgr2=0x7f5d5c0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:49.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.216+0000 7f5d809ff700 1 --2- 172.21.15.27:0/484715811 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5d5c067130 0x7f5d5c0695e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c06f720 tx=0x7f5d70009040 comp rx=0 tx=0).stop 2024-01-11T08:11:49.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.216+0000 7f5d809ff700 1 -- 172.21.15.27:0/484715811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0694c0 msgr2=0x7f5d7c169ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:49.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.216+0000 7f5d809ff700 1 --2- 172.21.15.27:0/484715811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0694c0 0x7f5d7c169ce0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00ea00 tx=0x7f5d6c009120 comp rx=0 tx=0).stop 2024-01-11T08:11:49.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.216+0000 7f5d809ff700 1 -- 172.21.15.27:0/484715811 shutdown_connections 2024-01-11T08:11:49.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.216+0000 7f5d809ff700 1 --2- 172.21.15.27:0/484715811 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5d5c067130 0x7f5d5c0695e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:49.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.216+0000 7f5d809ff700 1 --2- 172.21.15.27:0/484715811 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d7c155c50 0x7f5d7c16c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:49.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.216+0000 7f5d809ff700 1 --2- 172.21.15.27:0/484715811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0694c0 0x7f5d7c169ce0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:49.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.216+0000 7f5d809ff700 1 --2- 172.21.15.27:0/484715811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d7c0c74f0 0x7f5d7c0ae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:49.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.216+0000 7f5d809ff700 1 -- 172.21.15.27:0/484715811 >> 172.21.15.27:0/484715811 conn(0x7f5d7c0680a0 msgr2=0x7f5d7c0c9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:49.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.216+0000 7f5d809ff700 1 -- 172.21.15.27:0/484715811 shutdown_connections 2024-01-11T08:11:49.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.216+0000 7f5d809ff700 1 -- 172.21.15.27:0/484715811 wait complete. 2024-01-11T08:11:49.233 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2024-01-11T08:11:49.233 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir4 1m 2024-01-11T08:11:49.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.435+0000 7f5ac01ff700 1 Processor -- start 2024-01-11T08:11:49.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.436+0000 7f5ac01ff700 1 -- start start 2024-01-11T08:11:49.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.437+0000 7f5ac01ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ab8143190 0x7f5ab80cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:49.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.437+0000 7f5ac01ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab80ccc70 con 0x7f5ab8142dc0 2024-01-11T08:11:49.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.437+0000 7f5ac01ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab815d320 con 0x7f5ab80cb4b0 2024-01-11T08:11:49.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.437+0000 7f5ac01ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ab815df30 con 0x7f5ab8143190 2024-01-11T08:11:49.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.437+0000 7f5abd79a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ab8143190 0x7f5ab80cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:49.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.437+0000 7f5abdf9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5ab8142dc0 0x7f5ab80cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45482/0 (socket says 172.21.15.27:45482) 2024-01-11T08:11:49.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.437+0000 7f5abdf9b700 1 -- 172.21.15.27:0/205512683 learned_addr learned my addr 172.21.15.27:0/205512683 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:49.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.438+0000 7f5abcf99700 1 -- 172.21.15.27:0/205512683 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347631397 0 0) 0x7f5ab80ccc70 con 0x7f5ab8142dc0 2024-01-11T08:11:49.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.438+0000 7f5abd79a700 1 -- 172.21.15.27:0/205512683 >> v1:172.21.15.77:6789/0 conn(0x7f5ab80cb4b0 legacy=0x7f5ab80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:49.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.438+0000 7f5abd79a700 1 -- 172.21.15.27:0/205512683 >> v1:172.21.15.27:6789/0 conn(0x7f5ab8142dc0 legacy=0x7f5ab80cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:49.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.438+0000 7f5abd79a700 1 -- 172.21.15.27:0/205512683 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab815d590 con 0x7f5ab8143190 2024-01-11T08:11:49.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.438+0000 7f5abd79a700 1 --2- 172.21.15.27:0/205512683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ab8143190 0x7f5ab80cc020 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80098e0 tx=0x7f5aa80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b687989ca64d3cce server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:49.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.438+0000 7f5abcf99700 1 -- 172.21.15.27:0/205512683 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8009e80 con 0x7f5ab8143190 2024-01-11T08:11:49.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.438+0000 7f5abcf99700 1 -- 172.21.15.27:0/205512683 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa800b860 con 0x7f5ab8143190 2024-01-11T08:11:49.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.438+0000 7f5abcf99700 1 -- 172.21.15.27:0/205512683 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa800bbe0 con 0x7f5ab8143190 2024-01-11T08:11:49.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.439+0000 7f5ac01ff700 1 -- 172.21.15.27:0/205512683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ab8143190 msgr2=0x7f5ab80cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:49.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.439+0000 7f5ac01ff700 1 --2- 172.21.15.27:0/205512683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ab8143190 0x7f5ab80cc020 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80098e0 tx=0x7f5aa80091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:49.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.439+0000 7f5ac01ff700 1 -- 172.21.15.27:0/205512683 shutdown_connections 2024-01-11T08:11:49.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.439+0000 7f5ac01ff700 1 --2- 172.21.15.27:0/205512683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ab8143190 0x7f5ab80cc020 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:49.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.439+0000 7f5ac01ff700 1 -- 172.21.15.27:0/205512683 >> 172.21.15.27:0/205512683 conn(0x7f5ab8063ca0 msgr2=0x7f5ab806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:49.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.439+0000 7f5ac01ff700 1 -- 172.21.15.27:0/205512683 shutdown_connections 2024-01-11T08:11:49.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.439+0000 7f5ac01ff700 1 -- 172.21.15.27:0/205512683 wait complete. 2024-01-11T08:11:49.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.440+0000 7f5ac01ff700 1 Processor -- start 2024-01-11T08:11:49.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.441+0000 7f5ac01ff700 1 -- start start 2024-01-11T08:11:49.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.441+0000 7f5ac01ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ab80cb4b0 0x7f5ab8131b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:49.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.441+0000 7f5ac01ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ab8142dc0 0x7f5ab816e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:49.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.441+0000 7f5ac01ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab8143190 0x7f5ab80cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:49.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.441+0000 7f5ac01ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab8132de0 con 0x7f5ab80cb4b0 2024-01-11T08:11:49.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.441+0000 7f5ac01ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab816e6f0 con 0x7f5ab8143190 2024-01-11T08:11:49.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.442+0000 7f5ac01ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab8132160 con 0x7f5ab8142dc0 2024-01-11T08:11:49.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.442+0000 7f5abdf9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ab80cb4b0 0x7f5ab8131b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:49.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.442+0000 7f5abdf9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ab80cb4b0 0x7f5ab8131b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55244/0 (socket says 172.21.15.27:55244) 2024-01-11T08:11:49.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.442+0000 7f5abdf9b700 1 -- 172.21.15.27:0/1019853336 learned_addr learned my addr 172.21.15.27:0/1019853336 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:49.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.442+0000 7f5abe79c700 1 --2- 172.21.15.27:0/1019853336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab8143190 0x7f5ab80cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:49.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.442+0000 7f5abd79a700 1 --2- 172.21.15.27:0/1019853336 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ab8142dc0 0x7f5ab816e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:49.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.442+0000 7f5abdf9b700 1 -- 172.21.15.27:0/1019853336 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ab8142dc0 msgr2=0x7f5ab816e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:49.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.442+0000 7f5abdf9b700 1 --2- 172.21.15.27:0/1019853336 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ab8142dc0 0x7f5ab816e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:49.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.442+0000 7f5abdf9b700 1 -- 172.21.15.27:0/1019853336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab8143190 msgr2=0x7f5ab80cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:49.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.442+0000 7f5abdf9b700 1 --2- 172.21.15.27:0/1019853336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab8143190 0x7f5ab80cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:49.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.442+0000 7f5abdf9b700 1 -- 172.21.15.27:0/1019853336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa8009420 con 0x7f5ab80cb4b0 2024-01-11T08:11:49.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.442+0000 7f5abd79a700 1 --2- 172.21.15.27:0/1019853336 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ab8142dc0 0x7f5ab816e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:49.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.442+0000 7f5abe79c700 1 --2- 172.21.15.27:0/1019853336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab8143190 0x7f5ab80cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:49.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.442+0000 7f5abdf9b700 1 --2- 172.21.15.27:0/1019853336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ab80cb4b0 0x7f5ab8131b80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5ab400c810 tx=0x7f5ab4007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:49.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.443+0000 7f5aadffb700 1 -- 172.21.15.27:0/1019853336 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab4005c00 con 0x7f5ab80cb4b0 2024-01-11T08:11:49.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.443+0000 7f5aadffb700 1 -- 172.21.15.27:0/1019853336 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab4005d60 con 0x7f5ab80cb4b0 2024-01-11T08:11:49.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.443+0000 7f5aadffb700 1 -- 172.21.15.27:0/1019853336 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab4007340 con 0x7f5ab80cb4b0 2024-01-11T08:11:49.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.443+0000 7f5ac01ff700 1 -- 172.21.15.27:0/1019853336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab8132440 con 0x7f5ab80cb4b0 2024-01-11T08:11:49.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.443+0000 7f5ac01ff700 1 -- 172.21.15.27:0/1019853336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab8132a00 con 0x7f5ab80cb4b0 2024-01-11T08:11:49.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.444+0000 7f5ac01ff700 1 -- 172.21.15.27:0/1019853336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab804c3c0 con 0x7f5ab80cb4b0 2024-01-11T08:11:49.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.445+0000 7f5aadffb700 1 -- 172.21.15.27:0/1019853336 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f5ab401b400 con 0x7f5ab80cb4b0 2024-01-11T08:11:49.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.446+0000 7f5aadffb700 1 --2- 172.21.15.27:0/1019853336 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5a9c067200 0x7f5a9c0696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:49.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.446+0000 7f5aadffb700 1 -- 172.21.15.27:0/1019853336 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f5ab4084470 con 0x7f5ab80cb4b0 2024-01-11T08:11:49.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.446+0000 7f5abd79a700 1 --2- 172.21.15.27:0/1019853336 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5a9c067200 0x7f5a9c0696b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:49.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.447+0000 7f5abd79a700 1 --2- 172.21.15.27:0/1019853336 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5a9c067200 0x7f5a9c0696b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80093f0 tx=0x7f5aa8017000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:49.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.450+0000 7f5aadffb700 1 -- 172.21.15.27:0/1019853336 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5ab40513c0 con 0x7f5ab80cb4b0 2024-01-11T08:11:49.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.684+0000 7f5ac01ff700 1 -- 172.21.15.27:0/1019853336 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ab815d870 con 0x7f5a9c067200 2024-01-11T08:11:49.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.738+0000 7f5aadffb700 1 -- 172.21.15.27:0/1019853336 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f5ab815d870 con 0x7f5a9c067200 2024-01-11T08:11:49.738 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule activated for path /snap_test_dir1/dir4 2024-01-11T08:11:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.740+0000 7f5ac01ff700 1 -- 172.21.15.27:0/1019853336 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5a9c067200 msgr2=0x7f5a9c0696b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.740+0000 7f5ac01ff700 1 --2- 172.21.15.27:0/1019853336 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5a9c067200 0x7f5a9c0696b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80093f0 tx=0x7f5aa8017000 comp rx=0 tx=0).stop 2024-01-11T08:11:49.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.740+0000 7f5ac01ff700 1 -- 172.21.15.27:0/1019853336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ab80cb4b0 msgr2=0x7f5ab8131b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:49.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.740+0000 7f5ac01ff700 1 --2- 172.21.15.27:0/1019853336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ab80cb4b0 0x7f5ab8131b80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5ab400c810 tx=0x7f5ab4007d90 comp rx=0 tx=0).stop 2024-01-11T08:11:49.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.740+0000 7f5ac01ff700 1 -- 172.21.15.27:0/1019853336 shutdown_connections 2024-01-11T08:11:49.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.740+0000 7f5ac01ff700 1 --2- 172.21.15.27:0/1019853336 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5a9c067200 0x7f5a9c0696b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:49.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.741+0000 7f5ac01ff700 1 --2- 172.21.15.27:0/1019853336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab8143190 0x7f5ab80cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:49.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.741+0000 7f5ac01ff700 1 --2- 172.21.15.27:0/1019853336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ab80cb4b0 0x7f5ab8131b80 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:49.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.741+0000 7f5ac01ff700 1 --2- 172.21.15.27:0/1019853336 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ab8142dc0 0x7f5ab816e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:49.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.741+0000 7f5ac01ff700 1 -- 172.21.15.27:0/1019853336 >> 172.21.15.27:0/1019853336 conn(0x7f5ab8063ca0 msgr2=0x7f5ab80c36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:49.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.741+0000 7f5ac01ff700 1 -- 172.21.15.27:0/1019853336 shutdown_connections 2024-01-11T08:11:49.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.741+0000 7f5ac01ff700 1 -- 172.21.15.27:0/1019853336 wait complete. 2024-01-11T08:11:49.758 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2024-01-11T08:11:49.758 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir5 1m 2024-01-11T08:11:49.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.961+0000 7fa9b9fff700 1 Processor -- start 2024-01-11T08:11:49.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.961+0000 7fa9b9fff700 1 -- start start 2024-01-11T08:11:49.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.962+0000 7fa9b9fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa9b413ff80 0x7fa9b415a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:49.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.962+0000 7fa9b9fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa9b415a560 0x7fa9b40c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:49.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.962+0000 7fa9b9fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b415a9c0 con 0x7fa9b413fbb0 2024-01-11T08:11:49.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.962+0000 7fa9b9fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b40c6b30 con 0x7fa9b415a560 2024-01-11T08:11:49.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.962+0000 7fa9b9fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b40c6800 con 0x7fa9b413ff80 2024-01-11T08:11:49.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.962+0000 7fa9b3fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa9b415a560 0x7fa9b40c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:49.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.962+0000 7fa9b3fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa9b415a560 0x7fa9b40c6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55256/0 (socket says 172.21.15.27:55256) 2024-01-11T08:11:49.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.962+0000 7fa9b3fff700 1 -- 172.21.15.27:0/3711787530 learned_addr learned my addr 172.21.15.27:0/3711787530 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:49.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.963+0000 7fa9b2ffd700 1 --2- 172.21.15.27:0/3711787530 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa9b413ff80 0x7fa9b415a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:49.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.963+0000 7fa9b3fff700 1 -- 172.21.15.27:0/3711787530 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa9b413ff80 msgr2=0x7fa9b415a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:49.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.963+0000 7fa9b3fff700 1 --2- 172.21.15.27:0/3711787530 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa9b413ff80 0x7fa9b415a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:49.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.963+0000 7fa9b3fff700 1 -- 172.21.15.27:0/3711787530 >> v1:172.21.15.27:6789/0 conn(0x7fa9b413fbb0 legacy=0x7fa9b40ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:49.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.963+0000 7fa9b3fff700 1 -- 172.21.15.27:0/3711787530 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b40cb240 con 0x7fa9b415a560 2024-01-11T08:11:49.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.963+0000 7fa9b27fc700 1 -- 172.21.15.27:0/3711787530 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530642196 0 0) 0x7fa9a4001180 con 0x7fa9b413fbb0 2024-01-11T08:11:49.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.963+0000 7fa9b3fff700 1 --2- 172.21.15.27:0/3711787530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b415a560 0x7fa9b40c6fc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009f20 tx=0x7fa9ac009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=702a0fa7722cf837 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:49.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.963+0000 7fa9b27fc700 1 -- 172.21.15.27:0/3711787530 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac00e070 con 0x7fa9b415a560 2024-01-11T08:11:49.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.964+0000 7fa9b27fc700 1 -- 172.21.15.27:0/3711787530 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9ac00b9d0 con 0x7fa9b415a560 2024-01-11T08:11:49.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.964+0000 7fa9b27fc700 1 -- 172.21.15.27:0/3711787530 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac0092e0 con 0x7fa9b415a560 2024-01-11T08:11:49.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.964+0000 7fa9b9fff700 1 -- 172.21.15.27:0/3711787530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b415a560 msgr2=0x7fa9b40c6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:49.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.964+0000 7fa9b9fff700 1 --2- 172.21.15.27:0/3711787530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b415a560 0x7fa9b40c6fc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009f20 tx=0x7fa9ac009840 comp rx=0 tx=0).stop 2024-01-11T08:11:49.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.964+0000 7fa9b9fff700 1 -- 172.21.15.27:0/3711787530 shutdown_connections 2024-01-11T08:11:49.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.964+0000 7fa9b9fff700 1 --2- 172.21.15.27:0/3711787530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b415a560 0x7fa9b40c6fc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:49.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.964+0000 7fa9b9fff700 1 --2- 172.21.15.27:0/3711787530 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa9b413ff80 0x7fa9b415a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:49.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.964+0000 7fa9b9fff700 1 -- 172.21.15.27:0/3711787530 >> 172.21.15.27:0/3711787530 conn(0x7fa9b40680a0 msgr2=0x7fa9b40cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:49.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.964+0000 7fa9b9fff700 1 -- 172.21.15.27:0/3711787530 shutdown_connections 2024-01-11T08:11:49.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.964+0000 7fa9b9fff700 1 -- 172.21.15.27:0/3711787530 wait complete. 2024-01-11T08:11:49.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.966+0000 7fa9b9fff700 1 Processor -- start 2024-01-11T08:11:49.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.966+0000 7fa9b9fff700 1 -- start start 2024-01-11T08:11:49.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.967+0000 7fa9b9fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b413fbb0 0x7fa9b413d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:49.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.967+0000 7fa9b9fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa9b413ff80 0x7fa9b40b1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:49.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.967+0000 7fa9b9fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9b415a560 0x7fa9b40b1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:49.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.967+0000 7fa9b9fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b40c4af0 con 0x7fa9b413fbb0 2024-01-11T08:11:49.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.967+0000 7fa9b9fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b4157ff0 con 0x7fa9b415a560 2024-01-11T08:11:49.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.967+0000 7fa9b9fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b4157750 con 0x7fa9b413ff80 2024-01-11T08:11:49.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.967+0000 7fa9b37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b413fbb0 0x7fa9b413d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:49.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.967+0000 7fa9b37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b413fbb0 0x7fa9b413d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55270/0 (socket says 172.21.15.27:55270) 2024-01-11T08:11:49.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.967+0000 7fa9b37fe700 1 -- 172.21.15.27:0/3130800731 learned_addr learned my addr 172.21.15.27:0/3130800731 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.967+0000 7fa9b2ffd700 1 --2- 172.21.15.27:0/3130800731 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa9b413ff80 0x7fa9b40b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.967+0000 7fa9b3fff700 1 --2- 172.21.15.27:0/3130800731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9b415a560 0x7fa9b40b1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.968+0000 7fa9b37fe700 1 -- 172.21.15.27:0/3130800731 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa9b413ff80 msgr2=0x7fa9b40b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.968+0000 7fa9b37fe700 1 --2- 172.21.15.27:0/3130800731 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa9b413ff80 0x7fa9b40b1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.968+0000 7fa9b37fe700 1 -- 172.21.15.27:0/3130800731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9b415a560 msgr2=0x7fa9b40b1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.968+0000 7fa9b37fe700 1 --2- 172.21.15.27:0/3130800731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9b415a560 0x7fa9b40b1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.968+0000 7fa9b37fe700 1 -- 172.21.15.27:0/3130800731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac009ae0 con 0x7fa9b413fbb0 2024-01-11T08:11:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.968+0000 7fa9b37fe700 1 --2- 172.21.15.27:0/3130800731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b413fbb0 0x7fa9b413d070 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400c740 tx=0x7fa9a4007be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.968+0000 7fa99b7fe700 1 -- 172.21.15.27:0/3130800731 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a400f470 con 0x7fa9b413fbb0 2024-01-11T08:11:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.968+0000 7fa99b7fe700 1 -- 172.21.15.27:0/3130800731 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a400f5f0 con 0x7fa9b413fbb0 2024-01-11T08:11:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.968+0000 7fa99b7fe700 1 -- 172.21.15.27:0/3130800731 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a400f9b0 con 0x7fa9b413fbb0 2024-01-11T08:11:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.968+0000 7fa9b9fff700 1 -- 172.21.15.27:0/3130800731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b4157a30 con 0x7fa9b413fbb0 2024-01-11T08:11:49.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.968+0000 7fa9b9fff700 1 -- 172.21.15.27:0/3130800731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b413d5b0 con 0x7fa9b413fbb0 2024-01-11T08:11:49.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.969+0000 7fa9b9fff700 1 -- 172.21.15.27:0/3130800731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b40c48c0 con 0x7fa9b413fbb0 2024-01-11T08:11:49.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.970+0000 7fa99b7fe700 1 -- 172.21.15.27:0/3130800731 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7fa9a400fd20 con 0x7fa9b413fbb0 2024-01-11T08:11:49.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.971+0000 7fa99b7fe700 1 --2- 172.21.15.27:0/3130800731 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa9a0067130 0x7fa9a00695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:49.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.971+0000 7fa99b7fe700 1 -- 172.21.15.27:0/3130800731 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fa9a4083e00 con 0x7fa9b413fbb0 2024-01-11T08:11:49.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.971+0000 7fa9b2ffd700 1 --2- 172.21.15.27:0/3130800731 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa9a0067130 0x7fa9a00695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:49.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.972+0000 7fa9b2ffd700 1 --2- 172.21.15.27:0/3130800731 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa9a0067130 0x7fa9a00695e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40c77a0 tx=0x7fa9a80091a0 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:49.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:49.975+0000 7fa99b7fe700 1 -- 172.21.15.27:0/3130800731 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa9b40c48c0 con 0x7fa9b413fbb0 2024-01-11T08:11:50.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.206+0000 7fa9b9fff700 1 -- 172.21.15.27:0/3130800731 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9b40c4c30 con 0x7fa9a0067130 2024-01-11T08:11:50.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.259+0000 7fa99b7fe700 1 -- 172.21.15.27:0/3130800731 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fa9a8011a20 con 0x7fa9a0067130 2024-01-11T08:11:50.260 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule activated for path /snap_test_dir1/dir5 2024-01-11T08:11:50.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.262+0000 7fa9b9fff700 1 -- 172.21.15.27:0/3130800731 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa9a0067130 msgr2=0x7fa9a00695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:50.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.262+0000 7fa9b9fff700 1 --2- 172.21.15.27:0/3130800731 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa9a0067130 0x7fa9a00695e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40c77a0 tx=0x7fa9a80091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:50.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.262+0000 7fa9b9fff700 1 -- 172.21.15.27:0/3130800731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b413fbb0 msgr2=0x7fa9b413d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:50.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.262+0000 7fa9b9fff700 1 --2- 172.21.15.27:0/3130800731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b413fbb0 0x7fa9b413d070 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400c740 tx=0x7fa9a4007be0 comp rx=0 tx=0).stop 2024-01-11T08:11:50.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.262+0000 7fa9b9fff700 1 -- 172.21.15.27:0/3130800731 shutdown_connections 2024-01-11T08:11:50.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.262+0000 7fa9b9fff700 1 --2- 172.21.15.27:0/3130800731 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa9a0067130 0x7fa9a00695e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:50.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.262+0000 7fa9b9fff700 1 --2- 172.21.15.27:0/3130800731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9b415a560 0x7fa9b40b1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:50.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.262+0000 7fa9b9fff700 1 --2- 172.21.15.27:0/3130800731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b413fbb0 0x7fa9b413d070 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:50.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.262+0000 7fa9b9fff700 1 --2- 172.21.15.27:0/3130800731 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa9b413ff80 0x7fa9b40b1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:50.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.262+0000 7fa9b9fff700 1 -- 172.21.15.27:0/3130800731 >> 172.21.15.27:0/3130800731 conn(0x7fa9b40680a0 msgr2=0x7fa9b413d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:50.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.262+0000 7fa9b9fff700 1 -- 172.21.15.27:0/3130800731 shutdown_connections 2024-01-11T08:11:50.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.262+0000 7fa9b9fff700 1 -- 172.21.15.27:0/3130800731 wait complete. 2024-01-11T08:11:50.280 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2024-01-11T08:11:50.280 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir6 1m 2024-01-11T08:11:50.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.485+0000 7f2cc5fff700 1 Processor -- start 2024-01-11T08:11:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.485+0000 7f2cc5fff700 1 -- start start 2024-01-11T08:11:50.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.486+0000 7f2cc5fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2cc015a1b0 0x7f2cc015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:50.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.486+0000 7f2cc5fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2cc00ccd70 0x7f2cc00cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:50.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.486+0000 7f2cc5fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2cc00685c0 0x7f2cc0068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.486+0000 7f2cc5fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc00cf850 con 0x7f2cc00ccd70 2024-01-11T08:11:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.486+0000 7f2cc5fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc00cec40 con 0x7f2cc00685c0 2024-01-11T08:11:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.486+0000 7f2cc5fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc00cef20 con 0x7f2cc015a1b0 2024-01-11T08:11:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.486+0000 7f2cbeffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2cc00ccd70 0x7f2cc00cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.487+0000 7f2cbeffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2cc00ccd70 0x7f2cc00cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55274/0 (socket says 172.21.15.27:55274) 2024-01-11T08:11:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.487+0000 7f2cbeffd700 1 -- 172.21.15.27:0/4081601533 learned_addr learned my addr 172.21.15.27:0/4081601533 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.487+0000 7f2cbf7fe700 1 --2- 172.21.15.27:0/4081601533 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2cc015a1b0 0x7f2cc015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.487+0000 7f2cbffff700 1 --2- 172.21.15.27:0/4081601533 >> v2:172.21.15.77:3300/0 conn(0x7f2cc00685c0 0x7f2cc0068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.487+0000 7f2cbeffd700 1 -- 172.21.15.27:0/4081601533 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2cc015a1b0 msgr2=0x7f2cc015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.487+0000 7f2cbeffd700 1 --2- 172.21.15.27:0/4081601533 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2cc015a1b0 0x7f2cc015a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.487+0000 7f2cbeffd700 1 -- 172.21.15.27:0/4081601533 >> v2:172.21.15.77:3300/0 conn(0x7f2cc00685c0 msgr2=0x7f2cc0068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.487+0000 7f2cbeffd700 1 --2- 172.21.15.27:0/4081601533 >> v2:172.21.15.77:3300/0 conn(0x7f2cc00685c0 0x7f2cc0068a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.487+0000 7f2cbeffd700 1 -- 172.21.15.27:0/4081601533 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc00cf200 con 0x7f2cc00ccd70 2024-01-11T08:11:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.487+0000 7f2cbeffd700 1 --2- 172.21.15.27:0/4081601533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc00ccd70 0x7f2cc00cd1e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2cc00d0580 tx=0x7f2cb4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d51f43d31ac1ff17 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.487+0000 7f2cbe7fc700 1 -- 172.21.15.27:0/4081601533 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb400b640 con 0x7f2cc00ccd70 2024-01-11T08:11:50.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.488+0000 7f2cbe7fc700 1 -- 172.21.15.27:0/4081601533 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb400b7a0 con 0x7f2cc00ccd70 2024-01-11T08:11:50.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.488+0000 7f2cbe7fc700 1 -- 172.21.15.27:0/4081601533 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb400bb60 con 0x7f2cc00ccd70 2024-01-11T08:11:50.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.488+0000 7f2cc5fff700 1 -- 172.21.15.27:0/4081601533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc00ccd70 msgr2=0x7f2cc00cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:50.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.488+0000 7f2cc5fff700 1 --2- 172.21.15.27:0/4081601533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc00ccd70 0x7f2cc00cd1e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2cc00d0580 tx=0x7f2cb4009250 comp rx=0 tx=0).stop 2024-01-11T08:11:50.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.488+0000 7f2cc5fff700 1 -- 172.21.15.27:0/4081601533 shutdown_connections 2024-01-11T08:11:50.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.488+0000 7f2cc5fff700 1 --2- 172.21.15.27:0/4081601533 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2cc015a1b0 0x7f2cc015a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:50.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.488+0000 7f2cc5fff700 1 --2- 172.21.15.27:0/4081601533 >> v2:172.21.15.77:3300/0 conn(0x7f2cc00685c0 0x7f2cc0068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:50.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.488+0000 7f2cc5fff700 1 --2- 172.21.15.27:0/4081601533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc00ccd70 0x7f2cc00cd1e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:50.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.488+0000 7f2cc5fff700 1 -- 172.21.15.27:0/4081601533 >> 172.21.15.27:0/4081601533 conn(0x7f2cc006ed40 msgr2=0x7f2cc006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:50.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.488+0000 7f2cc5fff700 1 -- 172.21.15.27:0/4081601533 shutdown_connections 2024-01-11T08:11:50.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.488+0000 7f2cc5fff700 1 -- 172.21.15.27:0/4081601533 wait complete. 2024-01-11T08:11:50.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.490+0000 7f2cc5fff700 1 Processor -- start 2024-01-11T08:11:50.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.490+0000 7f2cc5fff700 1 -- start start 2024-01-11T08:11:50.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.491+0000 7f2cc5fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc00685c0 0x7f2cc0150a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:50.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.491+0000 7f2cc5fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cc00ccd70 0x7f2cc0169a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:50.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.491+0000 7f2cc5fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2cc015a1b0 0x7f2cc0169f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:50.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.491+0000 7f2cc5fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc01543b0 con 0x7f2cc00685c0 2024-01-11T08:11:50.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.491+0000 7f2cc5fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc016a5f0 con 0x7f2cc00ccd70 2024-01-11T08:11:50.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.491+0000 7f2cc5fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc016a8d0 con 0x7f2cc015a1b0 2024-01-11T08:11:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.491+0000 7f2cbf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc00685c0 0x7f2cc0150a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.491+0000 7f2cbf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc00685c0 0x7f2cc0150a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55276/0 (socket says 172.21.15.27:55276) 2024-01-11T08:11:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.491+0000 7f2cbf7fe700 1 -- 172.21.15.27:0/2319554457 learned_addr learned my addr 172.21.15.27:0/2319554457 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.491+0000 7f2cbeffd700 1 --2- 172.21.15.27:0/2319554457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cc00ccd70 0x7f2cc0169a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.491+0000 7f2cbffff700 1 --2- 172.21.15.27:0/2319554457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2cc015a1b0 0x7f2cc0169f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.491+0000 7f2cbf7fe700 1 -- 172.21.15.27:0/2319554457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2cc015a1b0 msgr2=0x7f2cc0169f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.491+0000 7f2cbf7fe700 1 --2- 172.21.15.27:0/2319554457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2cc015a1b0 0x7f2cc0169f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.491+0000 7f2cbf7fe700 1 -- 172.21.15.27:0/2319554457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cc00ccd70 msgr2=0x7f2cc0169a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.491+0000 7f2cbf7fe700 1 --2- 172.21.15.27:0/2319554457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cc00ccd70 0x7f2cc0169a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.492+0000 7f2cbf7fe700 1 -- 172.21.15.27:0/2319554457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb40094f0 con 0x7f2cc00685c0 2024-01-11T08:11:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.492+0000 7f2cbffff700 1 --2- 172.21.15.27:0/2319554457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2cc015a1b0 0x7f2cc0169f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:11:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.492+0000 7f2cbf7fe700 1 --2- 172.21.15.27:0/2319554457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc00685c0 0x7f2cc0150a60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2cb000ec80 tx=0x7f2cb00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.492+0000 7f2ca77fe700 1 -- 172.21.15.27:0/2319554457 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb000c850 con 0x7f2cc00685c0 2024-01-11T08:11:50.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.492+0000 7f2ca77fe700 1 -- 172.21.15.27:0/2319554457 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb000c9d0 con 0x7f2cc00685c0 2024-01-11T08:11:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.492+0000 7f2ca77fe700 1 -- 172.21.15.27:0/2319554457 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb000cd90 con 0x7f2cc00685c0 2024-01-11T08:11:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.492+0000 7f2cc5fff700 1 -- 172.21.15.27:0/2319554457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc0153d60 con 0x7f2cc00685c0 2024-01-11T08:11:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.492+0000 7f2cc5fff700 1 -- 172.21.15.27:0/2319554457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc01515d0 con 0x7f2cc00685c0 2024-01-11T08:11:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.493+0000 7f2cc5fff700 1 -- 172.21.15.27:0/2319554457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc004c3c0 con 0x7f2cc00685c0 2024-01-11T08:11:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.494+0000 7f2ca77fe700 1 -- 172.21.15.27:0/2319554457 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f2cb0013680 con 0x7f2cc00685c0 2024-01-11T08:11:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.495+0000 7f2ca77fe700 1 --2- 172.21.15.27:0/2319554457 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2cac067130 0x7f2cac0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.495+0000 7f2ca77fe700 1 -- 172.21.15.27:0/2319554457 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f2cb0084cb0 con 0x7f2cc00685c0 2024-01-11T08:11:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.496+0000 7f2cbeffd700 1 --2- 172.21.15.27:0/2319554457 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2cac067130 0x7f2cac0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.497+0000 7f2cbeffd700 1 --2- 172.21.15.27:0/2319554457 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2cac067130 0x7f2cac0695e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2cc00d0580 tx=0x7f2cb4017000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:50.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.500+0000 7f2ca77fe700 1 -- 172.21.15.27:0/2319554457 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2cb0051ba0 con 0x7f2cc00685c0 2024-01-11T08:11:50.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.732+0000 7f2cc5fff700 1 -- 172.21.15.27:0/2319554457 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cc01544f0 con 0x7f2cac067130 2024-01-11T08:11:50.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.790+0000 7f2ca77fe700 1 -- 172.21.15.27:0/2319554457 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f2cb4000c50 con 0x7f2cac067130 2024-01-11T08:11:50.791 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule activated for path /snap_test_dir1/dir6 2024-01-11T08:11:50.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.792+0000 7f2cc5fff700 1 -- 172.21.15.27:0/2319554457 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2cac067130 msgr2=0x7f2cac0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:50.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.792+0000 7f2cc5fff700 1 --2- 172.21.15.27:0/2319554457 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2cac067130 0x7f2cac0695e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2cc00d0580 tx=0x7f2cb4017000 comp rx=0 tx=0).stop 2024-01-11T08:11:50.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.792+0000 7f2cc5fff700 1 -- 172.21.15.27:0/2319554457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc00685c0 msgr2=0x7f2cc0150a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:50.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.792+0000 7f2cc5fff700 1 --2- 172.21.15.27:0/2319554457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc00685c0 0x7f2cc0150a60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2cb000ec80 tx=0x7f2cb00091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:50.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.792+0000 7f2cc5fff700 1 -- 172.21.15.27:0/2319554457 shutdown_connections 2024-01-11T08:11:50.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.792+0000 7f2cc5fff700 1 --2- 172.21.15.27:0/2319554457 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2cac067130 0x7f2cac0695e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:50.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.792+0000 7f2cc5fff700 1 --2- 172.21.15.27:0/2319554457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2cc015a1b0 0x7f2cc0169f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:50.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.792+0000 7f2cc5fff700 1 --2- 172.21.15.27:0/2319554457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc00685c0 0x7f2cc0150a60 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:50.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.792+0000 7f2cc5fff700 1 --2- 172.21.15.27:0/2319554457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cc00ccd70 0x7f2cc0169a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:50.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.793+0000 7f2cc5fff700 1 -- 172.21.15.27:0/2319554457 >> 172.21.15.27:0/2319554457 conn(0x7f2cc006ed40 msgr2=0x7f2cc0158260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:50.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.793+0000 7f2cc5fff700 1 -- 172.21.15.27:0/2319554457 shutdown_connections 2024-01-11T08:11:50.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:50.793+0000 7f2cc5fff700 1 -- 172.21.15.27:0/2319554457 wait complete. 2024-01-11T08:11:50.811 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2024-01-11T08:11:50.812 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir7 1m 2024-01-11T08:11:51.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.013+0000 7f16f41ff700 1 Processor -- start 2024-01-11T08:11:51.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.013+0000 7f16f41ff700 1 -- start start 2024-01-11T08:11:51.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.015+0000 7f16f41ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f16ec1577e0 0x7f16ec0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.015+0000 7f16f41ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f16ec069540 0x7f16ec0ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.015+0000 7f16f41ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ec0699d0 con 0x7f16ec157410 2024-01-11T08:11:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.015+0000 7f16f41ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16ec069cb0 con 0x7f16ec1577e0 2024-01-11T08:11:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.015+0000 7f16f41ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16ec069f90 con 0x7f16ec069540 2024-01-11T08:11:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.015+0000 7f16f179a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f16ec1577e0 0x7f16ec0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.015+0000 7f16f179a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f16ec1577e0 0x7f16ec0ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55278/0 (socket says 172.21.15.27:55278) 2024-01-11T08:11:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.015+0000 7f16f1f9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f16ec157410 0x7f16ec0ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45496/0 (socket says 172.21.15.27:45496) 2024-01-11T08:11:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.015+0000 7f16f179a700 1 -- 172.21.15.27:0/1590915983 learned_addr learned my addr 172.21.15.27:0/1590915983 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.015+0000 7f16f279c700 1 --2- 172.21.15.27:0/1590915983 >> v2:172.21.15.77:3300/0 conn(0x7f16ec069540 0x7f16ec0ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.015+0000 7f16f179a700 1 -- 172.21.15.27:0/1590915983 >> v2:172.21.15.77:3300/0 conn(0x7f16ec069540 msgr2=0x7f16ec0ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.015+0000 7f16f179a700 1 --2- 172.21.15.27:0/1590915983 >> v2:172.21.15.77:3300/0 conn(0x7f16ec069540 0x7f16ec0ae540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.015+0000 7f16f179a700 1 -- 172.21.15.27:0/1590915983 >> v1:172.21.15.27:6789/0 conn(0x7f16ec157410 legacy=0x7f16ec0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.015+0000 7f16f179a700 1 -- 172.21.15.27:0/1590915983 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ec06a270 con 0x7f16ec1577e0 2024-01-11T08:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.015+0000 7f16f0f99700 1 -- 172.21.15.27:0/1590915983 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530134293 0 0) 0x7f16ec0699d0 con 0x7f16ec157410 2024-01-11T08:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.016+0000 7f16f179a700 1 --2- 172.21.15.27:0/1590915983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16ec1577e0 0x7f16ec0ae000 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f16ec06d030 tx=0x7f16dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b40b5c23db7d1cd server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.016+0000 7f16f0f99700 1 -- 172.21.15.27:0/1590915983 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16dc00b860 con 0x7f16ec1577e0 2024-01-11T08:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.016+0000 7f16f0f99700 1 -- 172.21.15.27:0/1590915983 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16dc00b9c0 con 0x7f16ec1577e0 2024-01-11T08:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.016+0000 7f16f0f99700 1 -- 172.21.15.27:0/1590915983 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16dc00bd60 con 0x7f16ec1577e0 2024-01-11T08:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.016+0000 7f16f41ff700 1 -- 172.21.15.27:0/1590915983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16ec1577e0 msgr2=0x7f16ec0ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.016+0000 7f16f41ff700 1 --2- 172.21.15.27:0/1590915983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16ec1577e0 0x7f16ec0ae000 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f16ec06d030 tx=0x7f16dc009250 comp rx=0 tx=0).stop 2024-01-11T08:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.017+0000 7f16f41ff700 1 -- 172.21.15.27:0/1590915983 shutdown_connections 2024-01-11T08:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.017+0000 7f16f41ff700 1 --2- 172.21.15.27:0/1590915983 >> v2:172.21.15.77:3300/0 conn(0x7f16ec069540 0x7f16ec0ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.017+0000 7f16f41ff700 1 --2- 172.21.15.27:0/1590915983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16ec1577e0 0x7f16ec0ae000 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.017+0000 7f16f41ff700 1 -- 172.21.15.27:0/1590915983 >> 172.21.15.27:0/1590915983 conn(0x7f16ec0680a0 msgr2=0x7f16ec0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.017+0000 7f16f41ff700 1 -- 172.21.15.27:0/1590915983 shutdown_connections 2024-01-11T08:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.017+0000 7f16f41ff700 1 -- 172.21.15.27:0/1590915983 wait complete. 2024-01-11T08:11:51.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.018+0000 7f16f41ff700 1 Processor -- start 2024-01-11T08:11:51.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.018+0000 7f16f41ff700 1 -- start start 2024-01-11T08:11:51.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.019+0000 7f16f41ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16ec069540 0x7f16ec0c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:51.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.019+0000 7f16f41ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16ec157410 0x7f16ec157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:51.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.019+0000 7f16f41ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16ec1577e0 0x7f16ec1580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:51.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.019+0000 7f16f41ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec151eb0 con 0x7f16ec1577e0 2024-01-11T08:11:51.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.019+0000 7f16f41ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec159060 con 0x7f16ec157410 2024-01-11T08:11:51.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.019+0000 7f16f41ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec1587c0 con 0x7f16ec069540 2024-01-11T08:11:51.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.020+0000 7f16f1f9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16ec069540 0x7f16ec0c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:51.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.020+0000 7f16f279c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16ec1577e0 0x7f16ec1580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.020+0000 7f16f179a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16ec157410 0x7f16ec157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.020+0000 7f16f1f9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16ec069540 0x7f16ec0c8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:45772/0 (socket says 172.21.15.27:45772) 2024-01-11T08:11:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.020+0000 7f16f1f9b700 1 -- 172.21.15.27:0/877746068 learned_addr learned my addr 172.21.15.27:0/877746068 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.020+0000 7f16f279c700 1 -- 172.21.15.27:0/877746068 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16ec069540 msgr2=0x7f16ec0c8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.020+0000 7f16f279c700 1 --2- 172.21.15.27:0/877746068 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16ec069540 0x7f16ec0c8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.020+0000 7f16f279c700 1 -- 172.21.15.27:0/877746068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16ec157410 msgr2=0x7f16ec157bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.020+0000 7f16f279c700 1 --2- 172.21.15.27:0/877746068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16ec157410 0x7f16ec157bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.020+0000 7f16f279c700 1 -- 172.21.15.27:0/877746068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16dc0094f0 con 0x7f16ec1577e0 2024-01-11T08:11:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.020+0000 7f16f279c700 1 --2- 172.21.15.27:0/877746068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16ec1577e0 0x7f16ec1580f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f16e4007ed0 tx=0x7f16e4009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.021+0000 7f16e1ffb700 1 -- 172.21.15.27:0/877746068 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4018070 con 0x7f16ec1577e0 2024-01-11T08:11:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.021+0000 7f16e1ffb700 1 -- 172.21.15.27:0/877746068 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e400eb90 con 0x7f16ec1577e0 2024-01-11T08:11:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.021+0000 7f16f41ff700 1 -- 172.21.15.27:0/877746068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ec158aa0 con 0x7f16ec1577e0 2024-01-11T08:11:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.021+0000 7f16e1ffb700 1 -- 172.21.15.27:0/877746068 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4014420 con 0x7f16ec1577e0 2024-01-11T08:11:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.021+0000 7f16f41ff700 1 -- 172.21.15.27:0/877746068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ec1596c0 con 0x7f16ec1577e0 2024-01-11T08:11:51.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.022+0000 7f16f41ff700 1 -- 172.21.15.27:0/877746068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ec04c3c0 con 0x7f16ec1577e0 2024-01-11T08:11:51.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.023+0000 7f16e1ffb700 1 -- 172.21.15.27:0/877746068 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f16e4003020 con 0x7f16ec1577e0 2024-01-11T08:11:51.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.024+0000 7f16e1ffb700 1 --2- 172.21.15.27:0/877746068 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f16d0067130 0x7f16d00695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:51.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.024+0000 7f16e1ffb700 1 -- 172.21.15.27:0/877746068 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f16e4084a90 con 0x7f16ec1577e0 2024-01-11T08:11:51.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.025+0000 7f16f1f9b700 1 --2- 172.21.15.27:0/877746068 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f16d0067130 0x7f16d00695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:51.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.025+0000 7f16f1f9b700 1 --2- 172.21.15.27:0/877746068 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f16d0067130 0x7f16d00695e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f16e8002b30 tx=0x7f16e8009040 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:51.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.028+0000 7f16e1ffb700 1 -- 172.21.15.27:0/877746068 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f16e4051980 con 0x7f16ec1577e0 2024-01-11T08:11:51.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.261+0000 7f16f41ff700 1 -- 172.21.15.27:0/877746068 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f16ec0c7800 con 0x7f16d0067130 2024-01-11T08:11:51.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.318+0000 7f16e1ffb700 1 -- 172.21.15.27:0/877746068 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f16ec0c7800 con 0x7f16d0067130 2024-01-11T08:11:51.319 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule activated for path /snap_test_dir1/dir7 2024-01-11T08:11:51.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.321+0000 7f16f41ff700 1 -- 172.21.15.27:0/877746068 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f16d0067130 msgr2=0x7f16d00695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:51.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.321+0000 7f16f41ff700 1 --2- 172.21.15.27:0/877746068 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f16d0067130 0x7f16d00695e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f16e8002b30 tx=0x7f16e8009040 comp rx=0 tx=0).stop 2024-01-11T08:11:51.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.321+0000 7f16f41ff700 1 -- 172.21.15.27:0/877746068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16ec1577e0 msgr2=0x7f16ec1580f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:51.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.321+0000 7f16f41ff700 1 --2- 172.21.15.27:0/877746068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16ec1577e0 0x7f16ec1580f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f16e4007ed0 tx=0x7f16e4009850 comp rx=0 tx=0).stop 2024-01-11T08:11:51.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.321+0000 7f16f41ff700 1 -- 172.21.15.27:0/877746068 shutdown_connections 2024-01-11T08:11:51.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.321+0000 7f16f41ff700 1 --2- 172.21.15.27:0/877746068 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f16d0067130 0x7f16d00695e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.321+0000 7f16f41ff700 1 --2- 172.21.15.27:0/877746068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16ec1577e0 0x7f16ec1580f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.321+0000 7f16f41ff700 1 --2- 172.21.15.27:0/877746068 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16ec069540 0x7f16ec0c8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.321+0000 7f16f41ff700 1 --2- 172.21.15.27:0/877746068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16ec157410 0x7f16ec157bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.321+0000 7f16f41ff700 1 -- 172.21.15.27:0/877746068 >> 172.21.15.27:0/877746068 conn(0x7f16ec0680a0 msgr2=0x7f16ec0ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:51.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.321+0000 7f16f41ff700 1 -- 172.21.15.27:0/877746068 shutdown_connections 2024-01-11T08:11:51.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.321+0000 7f16f41ff700 1 -- 172.21.15.27:0/877746068 wait complete. 2024-01-11T08:11:51.340 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2024-01-11T08:11:51.340 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir8 1m 2024-01-11T08:11:51.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.542+0000 7f69e21ff700 1 Processor -- start 2024-01-11T08:11:51.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.543+0000 7f69e21ff700 1 -- start start 2024-01-11T08:11:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.544+0000 7f69e21ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f69dc15a1b0 0x7f69dc15a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.544+0000 7f69e21ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f69dc0ccd70 0x7f69dc0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.544+0000 7f69e21ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc0cf850 con 0x7f69dc0685c0 2024-01-11T08:11:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.544+0000 7f69e21ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69dc0cec40 con 0x7f69dc0ccd70 2024-01-11T08:11:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.544+0000 7f69e21ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc0cef20 con 0x7f69dc15a1b0 2024-01-11T08:11:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.544+0000 7f69dbfff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f69dc0685c0 0x7f69dc068a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45512/0 (socket says 172.21.15.27:45512) 2024-01-11T08:11:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.544+0000 7f69dbfff700 1 -- 172.21.15.27:0/4272111027 learned_addr learned my addr 172.21.15.27:0/4272111027 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.545+0000 7f69db7fe700 1 --2- 172.21.15.27:0/4272111027 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f69dc15a1b0 0x7f69dc15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.545+0000 7f69daffd700 1 --2- 172.21.15.27:0/4272111027 >> v2:172.21.15.77:3300/0 conn(0x7f69dc0ccd70 0x7f69dc0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.545+0000 7f69da7fc700 1 -- 172.21.15.27:0/4272111027 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919615166 0 0) 0x7f69dc0cf850 con 0x7f69dc0685c0 2024-01-11T08:11:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.545+0000 7f69da7fc700 1 -- 172.21.15.27:0/4272111027 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69c4003570 con 0x7f69dc0685c0 2024-01-11T08:11:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.545+0000 7f69db7fe700 1 -- 172.21.15.27:0/4272111027 >> v2:172.21.15.77:3300/0 conn(0x7f69dc0ccd70 msgr2=0x7f69dc0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.545+0000 7f69db7fe700 1 --2- 172.21.15.27:0/4272111027 >> v2:172.21.15.77:3300/0 conn(0x7f69dc0ccd70 0x7f69dc0cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.545+0000 7f69db7fe700 1 -- 172.21.15.27:0/4272111027 >> v1:172.21.15.27:6789/0 conn(0x7f69dc0685c0 legacy=0x7f69dc068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.545+0000 7f69da7fc700 1 -- 172.21.15.27:0/4272111027 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2191223845 0 0) 0x7f69c4003570 con 0x7f69dc0685c0 2024-01-11T08:11:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.545+0000 7f69db7fe700 1 -- 172.21.15.27:0/4272111027 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69dc0cf200 con 0x7f69dc15a1b0 2024-01-11T08:11:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.545+0000 7f69db7fe700 1 --2- 172.21.15.27:0/4272111027 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f69dc15a1b0 0x7f69dc15a580 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f69cc0098e0 tx=0x7f69cc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1407685f948d604d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:51.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.546+0000 7f69da7fc700 1 -- 172.21.15.27:0/4272111027 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d40017f0 con 0x7f69dc15a1b0 2024-01-11T08:11:51.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.546+0000 7f69da7fc700 1 -- 172.21.15.27:0/4272111027 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69cc011420 con 0x7f69dc15a1b0 2024-01-11T08:11:51.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.546+0000 7f69da7fc700 1 -- 172.21.15.27:0/4272111027 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69cc0117c0 con 0x7f69dc15a1b0 2024-01-11T08:11:51.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.546+0000 7f69e21ff700 1 -- 172.21.15.27:0/4272111027 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f69dc15a1b0 msgr2=0x7f69dc15a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:51.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.546+0000 7f69e21ff700 1 --2- 172.21.15.27:0/4272111027 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f69dc15a1b0 0x7f69dc15a580 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f69cc0098e0 tx=0x7f69cc0091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.546+0000 7f69e21ff700 1 -- 172.21.15.27:0/4272111027 shutdown_connections 2024-01-11T08:11:51.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.546+0000 7f69e21ff700 1 --2- 172.21.15.27:0/4272111027 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f69dc15a1b0 0x7f69dc15a580 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.546+0000 7f69e21ff700 1 --2- 172.21.15.27:0/4272111027 >> v2:172.21.15.77:3300/0 conn(0x7f69dc0ccd70 0x7f69dc0cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.546+0000 7f69e21ff700 1 -- 172.21.15.27:0/4272111027 >> 172.21.15.27:0/4272111027 conn(0x7f69dc06ed40 msgr2=0x7f69dc06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:51.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.546+0000 7f69e21ff700 1 -- 172.21.15.27:0/4272111027 shutdown_connections 2024-01-11T08:11:51.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.546+0000 7f69e21ff700 1 -- 172.21.15.27:0/4272111027 wait complete. 2024-01-11T08:11:51.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.548+0000 7f69e21ff700 1 Processor -- start 2024-01-11T08:11:51.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.548+0000 7f69e21ff700 1 -- start start 2024-01-11T08:11:51.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.549+0000 7f69e21ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f69dc0685c0 0x7f69dc14aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:51.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.549+0000 7f69e21ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0ccd70 0x7f69dc15b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:51.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.549+0000 7f69e21ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69dc15a1b0 0x7f69dc0a6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:51.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.549+0000 7f69e21ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc0c87d0 con 0x7f69dc15a1b0 2024-01-11T08:11:51.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.549+0000 7f69e21ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc0c8ab0 con 0x7f69dc0ccd70 2024-01-11T08:11:51.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.549+0000 7f69e21ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc0c8d90 con 0x7f69dc0685c0 2024-01-11T08:11:51.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.549+0000 7f69dbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69dc15a1b0 0x7f69dc0a6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:51.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.549+0000 7f69dbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69dc15a1b0 0x7f69dc0a6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55296/0 (socket says 172.21.15.27:55296) 2024-01-11T08:11:51.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.549+0000 7f69dbfff700 1 -- 172.21.15.27:0/2119282910 learned_addr learned my addr 172.21.15.27:0/2119282910 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:51.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.550+0000 7f69db7fe700 1 --2- 172.21.15.27:0/2119282910 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f69dc0685c0 0x7f69dc14aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:51.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.550+0000 7f69daffd700 1 --2- 172.21.15.27:0/2119282910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0ccd70 0x7f69dc15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:51.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.550+0000 7f69dbfff700 1 -- 172.21.15.27:0/2119282910 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f69dc0685c0 msgr2=0x7f69dc14aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:51.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.550+0000 7f69dbfff700 1 --2- 172.21.15.27:0/2119282910 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f69dc0685c0 0x7f69dc14aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.550+0000 7f69dbfff700 1 -- 172.21.15.27:0/2119282910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0ccd70 msgr2=0x7f69dc15b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:51.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.550+0000 7f69dbfff700 1 --2- 172.21.15.27:0/2119282910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0ccd70 0x7f69dc15b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.550+0000 7f69dbfff700 1 -- 172.21.15.27:0/2119282910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc009420 con 0x7f69dc15a1b0 2024-01-11T08:11:51.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.550+0000 7f69dbfff700 1 --2- 172.21.15.27:0/2119282910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69dc15a1b0 0x7f69dc0a6f10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f69d400cb90 tx=0x7f69d4007ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:51.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.550+0000 7f69c37fe700 1 -- 172.21.15.27:0/2119282910 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4004dc0 con 0x7f69dc15a1b0 2024-01-11T08:11:51.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.550+0000 7f69c37fe700 1 -- 172.21.15.27:0/2119282910 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d400a700 con 0x7f69dc15a1b0 2024-01-11T08:11:51.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.550+0000 7f69c37fe700 1 -- 172.21.15.27:0/2119282910 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d400aac0 con 0x7f69dc15a1b0 2024-01-11T08:11:51.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.550+0000 7f69e21ff700 1 -- 172.21.15.27:0/2119282910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69dc0c9070 con 0x7f69dc15a1b0 2024-01-11T08:11:51.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.551+0000 7f69e21ff700 1 -- 172.21.15.27:0/2119282910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69dc157b20 con 0x7f69dc15a1b0 2024-01-11T08:11:51.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.552+0000 7f69e21ff700 1 -- 172.21.15.27:0/2119282910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69dc04c3c0 con 0x7f69dc15a1b0 2024-01-11T08:11:51.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.552+0000 7f69c37fe700 1 -- 172.21.15.27:0/2119282910 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f69d4007150 con 0x7f69dc15a1b0 2024-01-11T08:11:51.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.554+0000 7f69c37fe700 1 --2- 172.21.15.27:0/2119282910 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f69c8067130 0x7f69c80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:51.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.554+0000 7f69c37fe700 1 -- 172.21.15.27:0/2119282910 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f69d40845b0 con 0x7f69dc15a1b0 2024-01-11T08:11:51.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.554+0000 7f69db7fe700 1 --2- 172.21.15.27:0/2119282910 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f69c8067130 0x7f69c80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:51.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.555+0000 7f69db7fe700 1 --2- 172.21.15.27:0/2119282910 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f69c8067130 0x7f69c80695e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f69cc0030c0 tx=0x7f69cc016000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:51.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.558+0000 7f69c37fe700 1 -- 172.21.15.27:0/2119282910 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f69d4051480 con 0x7f69dc15a1b0 2024-01-11T08:11:51.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.787+0000 7f69e21ff700 1 -- 172.21.15.27:0/2119282910 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f69dc0cd750 con 0x7f69c8067130 2024-01-11T08:11:51.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.842+0000 7f69c37fe700 1 -- 172.21.15.27:0/2119282910 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f69dc0cd750 con 0x7f69c8067130 2024-01-11T08:11:51.843 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule activated for path /snap_test_dir1/dir8 2024-01-11T08:11:51.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.844+0000 7f69e21ff700 1 -- 172.21.15.27:0/2119282910 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f69c8067130 msgr2=0x7f69c80695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.844+0000 7f69e21ff700 1 --2- 172.21.15.27:0/2119282910 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f69c8067130 0x7f69c80695e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f69cc0030c0 tx=0x7f69cc016000 comp rx=0 tx=0).stop 2024-01-11T08:11:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.844+0000 7f69e21ff700 1 -- 172.21.15.27:0/2119282910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69dc15a1b0 msgr2=0x7f69dc0a6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.844+0000 7f69e21ff700 1 --2- 172.21.15.27:0/2119282910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69dc15a1b0 0x7f69dc0a6f10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f69d400cb90 tx=0x7f69d4007ca0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.844+0000 7f69e21ff700 1 -- 172.21.15.27:0/2119282910 shutdown_connections 2024-01-11T08:11:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.844+0000 7f69e21ff700 1 --2- 172.21.15.27:0/2119282910 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f69c8067130 0x7f69c80695e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.844+0000 7f69e21ff700 1 --2- 172.21.15.27:0/2119282910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69dc15a1b0 0x7f69dc0a6f10 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.844+0000 7f69e21ff700 1 --2- 172.21.15.27:0/2119282910 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f69dc0685c0 0x7f69dc14aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.844+0000 7f69e21ff700 1 --2- 172.21.15.27:0/2119282910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69dc0ccd70 0x7f69dc15b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.844+0000 7f69e21ff700 1 -- 172.21.15.27:0/2119282910 >> 172.21.15.27:0/2119282910 conn(0x7f69dc06ed40 msgr2=0x7f69dc0c5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.844+0000 7f69e21ff700 1 -- 172.21.15.27:0/2119282910 shutdown_connections 2024-01-11T08:11:51.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:51.844+0000 7f69e21ff700 1 -- 172.21.15.27:0/2119282910 wait complete. 2024-01-11T08:11:51.863 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2024-01-11T08:11:51.863 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir9 1m 2024-01-11T08:11:52.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.063+0000 7f2a58bff700 1 Processor -- start 2024-01-11T08:11:52.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.063+0000 7f2a58bff700 1 -- start start 2024-01-11T08:11:52.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.065+0000 7f2a58bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2a54143190 0x7f2a540cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:52.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.065+0000 7f2a58bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a540cb4b0 0x7f2a540cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:52.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.065+0000 7f2a58bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a5415d320 con 0x7f2a54142dc0 2024-01-11T08:11:52.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.065+0000 7f2a58bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a5415df30 con 0x7f2a54143190 2024-01-11T08:11:52.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.065+0000 7f2a58bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5415d590 con 0x7f2a540cb4b0 2024-01-11T08:11:52.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.065+0000 7f2a51d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2a54143190 0x7f2a540cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:52.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.065+0000 7f2a5259c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2a54142dc0 0x7f2a540cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:47642/0 (socket says 172.21.15.27:47642) 2024-01-11T08:11:52.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.065+0000 7f2a5259c700 1 -- 172.21.15.27:0/590191942 learned_addr learned my addr 172.21.15.27:0/590191942 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:11:52.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.065+0000 7f2a52d9d700 1 --2- 172.21.15.27:0/590191942 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a540cb4b0 0x7f2a540cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:52.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.066+0000 7f2a5159a700 1 -- 172.21.15.27:0/590191942 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281335067 0 0) 0x7f2a5415d320 con 0x7f2a54142dc0 2024-01-11T08:11:52.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.066+0000 7f2a5159a700 1 -- 172.21.15.27:0/590191942 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a3c003570 con 0x7f2a54142dc0 2024-01-11T08:11:52.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.066+0000 7f2a51d9b700 1 -- 172.21.15.27:0/590191942 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a540cb4b0 msgr2=0x7f2a540cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:11:52.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.066+0000 7f2a51d9b700 1 --2- 172.21.15.27:0/590191942 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a540cb4b0 0x7f2a540cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:52.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.066+0000 7f2a51d9b700 1 -- 172.21.15.27:0/590191942 >> v1:172.21.15.77:6789/0 conn(0x7f2a54142dc0 legacy=0x7f2a540cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:52.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.066+0000 7f2a51d9b700 1 -- 172.21.15.27:0/590191942 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5415d870 con 0x7f2a54143190 2024-01-11T08:11:52.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.066+0000 7f2a52d9d700 1 --2- 172.21.15.27:0/590191942 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a540cb4b0 0x7f2a540cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:11:52.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.066+0000 7f2a51d9b700 1 --2- 172.21.15.27:0/590191942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a54143190 0x7f2a540cc020 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2a480098e0 tx=0x7f2a480091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b82797ce7e235747 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:52.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.066+0000 7f2a5159a700 1 -- 172.21.15.27:0/590191942 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a48011630 con 0x7f2a54143190 2024-01-11T08:11:52.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.066+0000 7f2a5159a700 1 -- 172.21.15.27:0/590191942 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a48011790 con 0x7f2a54143190 2024-01-11T08:11:52.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.066+0000 7f2a5159a700 1 -- 172.21.15.27:0/590191942 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a48011b30 con 0x7f2a54143190 2024-01-11T08:11:52.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.067+0000 7f2a58bff700 1 -- 172.21.15.27:0/590191942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a54143190 msgr2=0x7f2a540cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:52.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.067+0000 7f2a58bff700 1 --2- 172.21.15.27:0/590191942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a54143190 0x7f2a540cc020 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2a480098e0 tx=0x7f2a480091a0 comp rx=0 tx=0).stop 2024-01-11T08:11:52.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.067+0000 7f2a58bff700 1 -- 172.21.15.27:0/590191942 shutdown_connections 2024-01-11T08:11:52.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.067+0000 7f2a58bff700 1 --2- 172.21.15.27:0/590191942 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a540cb4b0 0x7f2a540cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:52.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.067+0000 7f2a58bff700 1 --2- 172.21.15.27:0/590191942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a54143190 0x7f2a540cc020 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:52.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.067+0000 7f2a58bff700 1 -- 172.21.15.27:0/590191942 >> 172.21.15.27:0/590191942 conn(0x7f2a54063ca0 msgr2=0x7f2a5406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:52.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.067+0000 7f2a58bff700 1 -- 172.21.15.27:0/590191942 shutdown_connections 2024-01-11T08:11:52.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.067+0000 7f2a58bff700 1 -- 172.21.15.27:0/590191942 wait complete. 2024-01-11T08:11:52.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.068+0000 7f2a58bff700 1 Processor -- start 2024-01-11T08:11:52.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.069+0000 7f2a58bff700 1 -- start start 2024-01-11T08:11:52.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.070+0000 7f2a58bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a540cb4b0 0x7f2a54140050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:52.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.070+0000 7f2a58bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a54142dc0 0x7f2a54155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:52.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.070+0000 7f2a58bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a54143190 0x7f2a5415c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:52.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.070+0000 7f2a58bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a54140590 con 0x7f2a54142dc0 2024-01-11T08:11:52.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.070+0000 7f2a58bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a54156360 con 0x7f2a540cb4b0 2024-01-11T08:11:52.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.070+0000 7f2a58bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a54156640 con 0x7f2a54143190 2024-01-11T08:11:52.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.070+0000 7f2a51d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a54142dc0 0x7f2a54155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:52.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.070+0000 7f2a51d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a54142dc0 0x7f2a54155910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55306/0 (socket says 172.21.15.27:55306) 2024-01-11T08:11:52.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.070+0000 7f2a51d9b700 1 -- 172.21.15.27:0/1581014069 learned_addr learned my addr 172.21.15.27:0/1581014069 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:11:52.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.070+0000 7f2a52d9d700 1 --2- 172.21.15.27:0/1581014069 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a54143190 0x7f2a5415c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:52.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.070+0000 7f2a5259c700 1 --2- 172.21.15.27:0/1581014069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a540cb4b0 0x7f2a54140050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:52.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.070+0000 7f2a51d9b700 1 -- 172.21.15.27:0/1581014069 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a54143190 msgr2=0x7f2a5415c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:52.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.070+0000 7f2a51d9b700 1 --2- 172.21.15.27:0/1581014069 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a54143190 0x7f2a5415c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:52.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.070+0000 7f2a51d9b700 1 -- 172.21.15.27:0/1581014069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a540cb4b0 msgr2=0x7f2a54140050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:52.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.070+0000 7f2a51d9b700 1 --2- 172.21.15.27:0/1581014069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a540cb4b0 0x7f2a54140050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:52.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.070+0000 7f2a51d9b700 1 -- 172.21.15.27:0/1581014069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4c009a20 con 0x7f2a54142dc0 2024-01-11T08:11:52.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.071+0000 7f2a51d9b700 1 --2- 172.21.15.27:0/1581014069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a54142dc0 0x7f2a54155910 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2a54064a70 tx=0x7f2a48017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:52.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.071+0000 7f2a427fc700 1 -- 172.21.15.27:0/1581014069 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a48011b10 con 0x7f2a54142dc0 2024-01-11T08:11:52.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.071+0000 7f2a58bff700 1 -- 172.21.15.27:0/1581014069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a48009420 con 0x7f2a54142dc0 2024-01-11T08:11:52.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.071+0000 7f2a427fc700 1 -- 172.21.15.27:0/1581014069 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a480253f0 con 0x7f2a54142dc0 2024-01-11T08:11:52.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.071+0000 7f2a427fc700 1 -- 172.21.15.27:0/1581014069 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a48025790 con 0x7f2a54142dc0 2024-01-11T08:11:52.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.071+0000 7f2a58bff700 1 -- 172.21.15.27:0/1581014069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a54156c00 con 0x7f2a54142dc0 2024-01-11T08:11:52.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.072+0000 7f2a58bff700 1 -- 172.21.15.27:0/1581014069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a5404c3c0 con 0x7f2a54142dc0 2024-01-11T08:11:52.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.073+0000 7f2a427fc700 1 -- 172.21.15.27:0/1581014069 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f2a48025b80 con 0x7f2a54142dc0 2024-01-11T08:11:52.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.075+0000 7f2a427fc700 1 --2- 172.21.15.27:0/1581014069 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2a34067130 0x7f2a340695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:11:52.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.075+0000 7f2a427fc700 1 -- 172.21.15.27:0/1581014069 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f2a480862c0 con 0x7f2a54142dc0 2024-01-11T08:11:52.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.075+0000 7f2a5259c700 1 --2- 172.21.15.27:0/1581014069 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2a34067130 0x7f2a340695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:11:52.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.076+0000 7f2a5259c700 1 --2- 172.21.15.27:0/1581014069 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2a34067130 0x7f2a340695e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2a440058d0 tx=0x7f2a44009040 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:11:52.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.079+0000 7f2a427fc700 1 -- 172.21.15.27:0/1581014069 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2a48053110 con 0x7f2a54142dc0 2024-01-11T08:11:52.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.309+0000 7f2a58bff700 1 -- 172.21.15.27:0/1581014069 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a5415db50 con 0x7f2a34067130 2024-01-11T08:11:52.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.361+0000 7f2a427fc700 1 -- 172.21.15.27:0/1581014069 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f2a5415db50 con 0x7f2a34067130 2024-01-11T08:11:52.362 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule activated for path /snap_test_dir1/dir9 2024-01-11T08:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.363+0000 7f2a58bff700 1 -- 172.21.15.27:0/1581014069 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2a34067130 msgr2=0x7f2a340695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.363+0000 7f2a58bff700 1 --2- 172.21.15.27:0/1581014069 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2a34067130 0x7f2a340695e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2a440058d0 tx=0x7f2a44009040 comp rx=0 tx=0).stop 2024-01-11T08:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.363+0000 7f2a58bff700 1 -- 172.21.15.27:0/1581014069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a54142dc0 msgr2=0x7f2a54155910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.363+0000 7f2a58bff700 1 --2- 172.21.15.27:0/1581014069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a54142dc0 0x7f2a54155910 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2a54064a70 tx=0x7f2a48017000 comp rx=0 tx=0).stop 2024-01-11T08:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.363+0000 7f2a58bff700 1 -- 172.21.15.27:0/1581014069 shutdown_connections 2024-01-11T08:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.363+0000 7f2a58bff700 1 --2- 172.21.15.27:0/1581014069 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2a34067130 0x7f2a340695e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.363+0000 7f2a58bff700 1 --2- 172.21.15.27:0/1581014069 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a54143190 0x7f2a5415c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.363+0000 7f2a58bff700 1 --2- 172.21.15.27:0/1581014069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a540cb4b0 0x7f2a54140050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:52.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.363+0000 7f2a58bff700 1 --2- 172.21.15.27:0/1581014069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a54142dc0 0x7f2a54155910 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:11:52.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.363+0000 7f2a58bff700 1 -- 172.21.15.27:0/1581014069 >> 172.21.15.27:0/1581014069 conn(0x7f2a54063ca0 msgr2=0x7f2a54141df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:11:52.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.363+0000 7f2a58bff700 1 -- 172.21.15.27:0/1581014069 shutdown_connections 2024-01-11T08:11:52.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:11:52.363+0000 7f2a58bff700 1 -- 172.21.15.27:0/1581014069 wait complete. 2024-01-11T08:11:52.381 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2024-01-11T08:12:00.170 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:12:00.169+0000 7f9ebac9e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:12:00.171 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:12:00.169+0000 7f9ebac9e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:12:00.171 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:12:00.169+0000 7f9ebac9e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:12:00.171 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:12:00.169+0000 7f9ebac9e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:12:00.171 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:12:00.169+0000 7f9ebac9e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:12:19.048 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:12:19.053 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:12:49.162 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:12:49.168 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:13:19.296 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:13:19.301 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:13:49.417 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:13:49.423 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:14:19.531 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:14:19.536 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:14:49.670 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:14:49.676 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:15:19.775 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:15:19.781 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:15:49.890 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:15:49.896 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:16:00.641 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:16:00.641+0000 7f9eb340f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:16:00.642 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:16:00.641+0000 7f9eb340f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:16:00.642 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:16:00.641+0000 7f9eb340f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:16:00.642 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:16:00.641+0000 7f9eb340f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:16:00.642 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:16:00.641+0000 7f9eb340f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:16:20.006 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:16:20.012 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:16:50.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:16:50.126 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:17:20.259 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:17:20.265 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:17:50.373 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:17:50.379 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:18:20.488 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:18:20.494 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:18:50.603 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:18:50.609 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:19:20.733 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:19:20.740 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:19:50.849 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:19:50.854 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:20:20.963 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:20:20.969 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:20:51.077 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:20:51.084 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:21:21.217 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:21:21.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:21:51.339 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:21:51.345 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:22:20.382 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir0 1m 2024-01-11T08:22:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.596+0000 7f103a1ff700 1 Processor -- start 2024-01-11T08:22:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.597+0000 7f103a1ff700 1 -- start start 2024-01-11T08:22:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.598+0000 7f103a1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f10341577e0 0x7f10340ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.598+0000 7f103a1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10340699d0 con 0x7f1034069540 2024-01-11T08:22:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.598+0000 7f103a1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1034069cb0 con 0x7f1034157410 2024-01-11T08:22:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.598+0000 7f103a1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1034069f90 con 0x7f10341577e0 2024-01-11T08:22:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.598+0000 7f1033fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1034069540 0x7f10340c6ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41570/0 (socket says 172.21.15.27:41570) 2024-01-11T08:22:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.598+0000 7f1033fff700 1 -- 172.21.15.27:0/1999692808 learned_addr learned my addr 172.21.15.27:0/1999692808 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.599+0000 7f1032ffd700 1 --2- 172.21.15.27:0/1999692808 >> v2:172.21.15.77:3300/0 conn(0x7f10341577e0 0x7f10340ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.599+0000 7f10327fc700 1 -- 172.21.15.27:0/1999692808 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609606979 0 0) 0x7f10340699d0 con 0x7f1034069540 2024-01-11T08:22:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.599+0000 7f10327fc700 1 -- 172.21.15.27:0/1999692808 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f101c003570 con 0x7f1034069540 2024-01-11T08:22:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.599+0000 7f10327fc700 1 -- 172.21.15.27:0/1999692808 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222135570 0 0) 0x7f1034069cb0 con 0x7f1034157410 2024-01-11T08:22:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.599+0000 7f10327fc700 1 -- 172.21.15.27:0/1999692808 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10340699d0 con 0x7f1034157410 2024-01-11T08:22:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.599+0000 7f1032ffd700 1 -- 172.21.15.27:0/1999692808 >> v1:172.21.15.77:6789/0 conn(0x7f1034157410 legacy=0x7f10340ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.599+0000 7f1032ffd700 1 -- 172.21.15.27:0/1999692808 >> v1:172.21.15.27:6789/0 conn(0x7f1034069540 legacy=0x7f10340c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.599+0000 7f1032ffd700 1 -- 172.21.15.27:0/1999692808 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103406a270 con 0x7f10341577e0 2024-01-11T08:22:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.599+0000 7f1032ffd700 1 --2- 172.21.15.27:0/1999692808 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10341577e0 0x7f10340ae000 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f10280098e0 tx=0x7f10280091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9115f40b2c3c8c8b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.600+0000 7f10327fc700 1 -- 172.21.15.27:0/1999692808 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1028011630 con 0x7f10341577e0 2024-01-11T08:22:20.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.600+0000 7f10327fc700 1 -- 172.21.15.27:0/1999692808 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1028011790 con 0x7f10341577e0 2024-01-11T08:22:20.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.600+0000 7f10327fc700 1 -- 172.21.15.27:0/1999692808 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1028011b30 con 0x7f10341577e0 2024-01-11T08:22:20.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.600+0000 7f103a1ff700 1 -- 172.21.15.27:0/1999692808 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10341577e0 msgr2=0x7f10340ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:20.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.600+0000 7f103a1ff700 1 --2- 172.21.15.27:0/1999692808 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10341577e0 0x7f10340ae000 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f10280098e0 tx=0x7f10280091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:20.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.600+0000 7f103a1ff700 1 -- 172.21.15.27:0/1999692808 shutdown_connections 2024-01-11T08:22:20.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.600+0000 7f103a1ff700 1 --2- 172.21.15.27:0/1999692808 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10341577e0 0x7f10340ae000 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:20.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.600+0000 7f103a1ff700 1 -- 172.21.15.27:0/1999692808 >> 172.21.15.27:0/1999692808 conn(0x7f10340680a0 msgr2=0x7f10340cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:20.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.600+0000 7f103a1ff700 1 -- 172.21.15.27:0/1999692808 shutdown_connections 2024-01-11T08:22:20.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.600+0000 7f103a1ff700 1 -- 172.21.15.27:0/1999692808 wait complete. 2024-01-11T08:22:20.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.602+0000 7f103a1ff700 1 Processor -- start 2024-01-11T08:22:20.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.602+0000 7f103a1ff700 1 -- start start 2024-01-11T08:22:20.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.603+0000 7f103a1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1034069540 0x7f10340cb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:20.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.603+0000 7f103a1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1034157410 0x7f10340caad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:20.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.603+0000 7f103a1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10341577e0 0x7f10340ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:20.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.603+0000 7f103a1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10340cbe60 con 0x7f10341577e0 2024-01-11T08:22:20.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.603+0000 7f103a1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1034156810 con 0x7f1034157410 2024-01-11T08:22:20.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.603+0000 7f103a1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10340cb010 con 0x7f1034069540 2024-01-11T08:22:20.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.603+0000 7f1033fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10341577e0 0x7f10340ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:20.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.604+0000 7f1033fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10341577e0 0x7f10340ca140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32966/0 (socket says 172.21.15.27:32966) 2024-01-11T08:22:20.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.604+0000 7f1033fff700 1 -- 172.21.15.27:0/4131495477 learned_addr learned my addr 172.21.15.27:0/4131495477 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:20.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.604+0000 7f1032ffd700 1 --2- 172.21.15.27:0/4131495477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1034157410 0x7f10340caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:20.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.604+0000 7f10337fe700 1 --2- 172.21.15.27:0/4131495477 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1034069540 0x7f10340cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:20.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.604+0000 7f1033fff700 1 -- 172.21.15.27:0/4131495477 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1034069540 msgr2=0x7f10340cb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:20.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.604+0000 7f1033fff700 1 --2- 172.21.15.27:0/4131495477 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1034069540 0x7f10340cb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:20.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.604+0000 7f1033fff700 1 -- 172.21.15.27:0/4131495477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1034157410 msgr2=0x7f10340caad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:20.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.604+0000 7f1033fff700 1 --2- 172.21.15.27:0/4131495477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1034157410 0x7f10340caad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:20.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.604+0000 7f1033fff700 1 -- 172.21.15.27:0/4131495477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1028009420 con 0x7f10341577e0 2024-01-11T08:22:20.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.604+0000 7f1032ffd700 1 --2- 172.21.15.27:0/4131495477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1034157410 0x7f10340caad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:22:20.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.604+0000 7f1033fff700 1 --2- 172.21.15.27:0/4131495477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10341577e0 0x7f10340ca140 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f102c00cc90 tx=0x7f102c001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:20.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.604+0000 7f101b7fe700 1 -- 172.21.15.27:0/4131495477 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c00cea0 con 0x7f10341577e0 2024-01-11T08:22:20.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.605+0000 7f101b7fe700 1 -- 172.21.15.27:0/4131495477 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102c00ab30 con 0x7f10341577e0 2024-01-11T08:22:20.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.605+0000 7f103a1ff700 1 -- 172.21.15.27:0/4131495477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1034169e80 con 0x7f10341577e0 2024-01-11T08:22:20.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.605+0000 7f101b7fe700 1 -- 172.21.15.27:0/4131495477 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c0070f0 con 0x7f10341577e0 2024-01-11T08:22:20.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.605+0000 7f103a1ff700 1 -- 172.21.15.27:0/4131495477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f103416a440 con 0x7f10341577e0 2024-01-11T08:22:20.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.606+0000 7f103a1ff700 1 -- 172.21.15.27:0/4131495477 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f103404c3c0 con 0x7f10341577e0 2024-01-11T08:22:20.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.607+0000 7f101b7fe700 1 -- 172.21.15.27:0/4131495477 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f102c0074e0 con 0x7f10341577e0 2024-01-11T08:22:20.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.608+0000 7f101b7fe700 1 --2- 172.21.15.27:0/4131495477 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1020067130 0x7f10200695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:20.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.608+0000 7f101b7fe700 1 -- 172.21.15.27:0/4131495477 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f102c084f30 con 0x7f10341577e0 2024-01-11T08:22:20.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.609+0000 7f10337fe700 1 --2- 172.21.15.27:0/4131495477 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1020067130 0x7f10200695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:20.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.609+0000 7f10337fe700 1 --2- 172.21.15.27:0/4131495477 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1020067130 0x7f10200695e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f10340c91d0 tx=0x7f1024009040 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:20.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.613+0000 7f101b7fe700 1 -- 172.21.15.27:0/4131495477 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f102c051e00 con 0x7f10341577e0 2024-01-11T08:22:20.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.845+0000 7f103a1ff700 1 -- 172.21.15.27:0/4131495477 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1034154100 con 0x7f1020067130 2024-01-11T08:22:20.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.915+0000 7f101b7fe700 1 -- 172.21.15.27:0/4131495477 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f1034154100 con 0x7f1020067130 2024-01-11T08:22:20.916 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2024-01-11T08:22:20.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.918+0000 7f103a1ff700 1 -- 172.21.15.27:0/4131495477 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1020067130 msgr2=0x7f10200695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:20.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.918+0000 7f103a1ff700 1 --2- 172.21.15.27:0/4131495477 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1020067130 0x7f10200695e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f10340c91d0 tx=0x7f1024009040 comp rx=0 tx=0).stop 2024-01-11T08:22:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.918+0000 7f103a1ff700 1 -- 172.21.15.27:0/4131495477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10341577e0 msgr2=0x7f10340ca140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.918+0000 7f103a1ff700 1 --2- 172.21.15.27:0/4131495477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10341577e0 0x7f10340ca140 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f102c00cc90 tx=0x7f102c001ae0 comp rx=0 tx=0).stop 2024-01-11T08:22:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.918+0000 7f103a1ff700 1 -- 172.21.15.27:0/4131495477 shutdown_connections 2024-01-11T08:22:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.918+0000 7f103a1ff700 1 --2- 172.21.15.27:0/4131495477 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1020067130 0x7f10200695e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.918+0000 7f103a1ff700 1 --2- 172.21.15.27:0/4131495477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10341577e0 0x7f10340ca140 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.918+0000 7f103a1ff700 1 --2- 172.21.15.27:0/4131495477 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1034069540 0x7f10340cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.918+0000 7f103a1ff700 1 --2- 172.21.15.27:0/4131495477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1034157410 0x7f10340caad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.918+0000 7f103a1ff700 1 -- 172.21.15.27:0/4131495477 >> 172.21.15.27:0/4131495477 conn(0x7f10340680a0 msgr2=0x7f1034152d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.918+0000 7f103a1ff700 1 -- 172.21.15.27:0/4131495477 shutdown_connections 2024-01-11T08:22:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:20.918+0000 7f103a1ff700 1 -- 172.21.15.27:0/4131495477 wait complete. 2024-01-11T08:22:20.936 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2024-01-11T08:22:20.936 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir1 1m 2024-01-11T08:22:21.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.138+0000 7fd4ea9ff700 1 Processor -- start 2024-01-11T08:22:21.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.138+0000 7fd4ea9ff700 1 -- start start 2024-01-11T08:22:21.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.139+0000 7fd4ea9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4e415a1b0 0x7fd4e415a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:21.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.139+0000 7fd4ea9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd4e40ccd70 0x7fd4e40cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:21.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.139+0000 7fd4ea9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd4e40685c0 0x7fd4e4068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:21.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.139+0000 7fd4ea9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e40cf850 con 0x7fd4e40ccd70 2024-01-11T08:22:21.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.139+0000 7fd4ea9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e40cec40 con 0x7fd4e40685c0 2024-01-11T08:22:21.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.139+0000 7fd4ea9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e40cef20 con 0x7fd4e415a1b0 2024-01-11T08:22:21.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.139+0000 7fd4e37fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd4e40ccd70 0x7fd4e40cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:21.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.139+0000 7fd4e37fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd4e40ccd70 0x7fd4e40cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32968/0 (socket says 172.21.15.27:32968) 2024-01-11T08:22:21.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.139+0000 7fd4e37fe700 1 -- 172.21.15.27:0/90965192 learned_addr learned my addr 172.21.15.27:0/90965192 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:21.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.139+0000 7fd4e8f9c700 1 --2- 172.21.15.27:0/90965192 >> v2:172.21.15.77:3300/0 conn(0x7fd4e40685c0 0x7fd4e4068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:21.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.139+0000 7fd4e3fff700 1 --2- 172.21.15.27:0/90965192 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4e415a1b0 0x7fd4e415a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:21.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.140+0000 7fd4e37fe700 1 -- 172.21.15.27:0/90965192 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4e415a1b0 msgr2=0x7fd4e415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:21.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.140+0000 7fd4e37fe700 1 --2- 172.21.15.27:0/90965192 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4e415a1b0 0x7fd4e415a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:21.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.140+0000 7fd4e37fe700 1 -- 172.21.15.27:0/90965192 >> v2:172.21.15.77:3300/0 conn(0x7fd4e40685c0 msgr2=0x7fd4e4068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:21.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.140+0000 7fd4e37fe700 1 --2- 172.21.15.27:0/90965192 >> v2:172.21.15.77:3300/0 conn(0x7fd4e40685c0 0x7fd4e4068a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:21.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.140+0000 7fd4e37fe700 1 -- 172.21.15.27:0/90965192 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e40cf200 con 0x7fd4e40ccd70 2024-01-11T08:22:21.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.140+0000 7fd4e37fe700 1 --2- 172.21.15.27:0/90965192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4e40ccd70 0x7fd4e40cd1e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40d0580 tx=0x7fd4d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e276828e1b3849f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:21.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.140+0000 7fd4e2ffd700 1 -- 172.21.15.27:0/90965192 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d800b640 con 0x7fd4e40ccd70 2024-01-11T08:22:21.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.141+0000 7fd4e2ffd700 1 -- 172.21.15.27:0/90965192 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d800b7a0 con 0x7fd4e40ccd70 2024-01-11T08:22:21.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.141+0000 7fd4e2ffd700 1 -- 172.21.15.27:0/90965192 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d800bb60 con 0x7fd4e40ccd70 2024-01-11T08:22:21.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.141+0000 7fd4ea9ff700 1 -- 172.21.15.27:0/90965192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4e40ccd70 msgr2=0x7fd4e40cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:21.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.141+0000 7fd4ea9ff700 1 --2- 172.21.15.27:0/90965192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4e40ccd70 0x7fd4e40cd1e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40d0580 tx=0x7fd4d8009250 comp rx=0 tx=0).stop 2024-01-11T08:22:21.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.141+0000 7fd4ea9ff700 1 -- 172.21.15.27:0/90965192 shutdown_connections 2024-01-11T08:22:21.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.141+0000 7fd4ea9ff700 1 --2- 172.21.15.27:0/90965192 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4e415a1b0 0x7fd4e415a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:21.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.141+0000 7fd4ea9ff700 1 --2- 172.21.15.27:0/90965192 >> v2:172.21.15.77:3300/0 conn(0x7fd4e40685c0 0x7fd4e4068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:21.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.141+0000 7fd4ea9ff700 1 --2- 172.21.15.27:0/90965192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4e40ccd70 0x7fd4e40cd1e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:21.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.141+0000 7fd4ea9ff700 1 -- 172.21.15.27:0/90965192 >> 172.21.15.27:0/90965192 conn(0x7fd4e406ed40 msgr2=0x7fd4e406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:21.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.141+0000 7fd4ea9ff700 1 -- 172.21.15.27:0/90965192 shutdown_connections 2024-01-11T08:22:21.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.141+0000 7fd4ea9ff700 1 -- 172.21.15.27:0/90965192 wait complete. 2024-01-11T08:22:21.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.143+0000 7fd4ea9ff700 1 Processor -- start 2024-01-11T08:22:21.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.143+0000 7fd4ea9ff700 1 -- start start 2024-01-11T08:22:21.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.144+0000 7fd4ea9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4e40685c0 0x7fd4e41553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:21.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.144+0000 7fd4ea9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4e40ccd70 0x7fd4e415b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:21.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.144+0000 7fd4ea9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4e415a1b0 0x7fd4e4150280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:21.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.144+0000 7fd4ea9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e40b2a80 con 0x7fd4e415a1b0 2024-01-11T08:22:21.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.144+0000 7fd4ea9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e40b4570 con 0x7fd4e40685c0 2024-01-11T08:22:21.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.144+0000 7fd4ea9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e40b1f00 con 0x7fd4e40ccd70 2024-01-11T08:22:21.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.144+0000 7fd4e8f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4e415a1b0 0x7fd4e4150280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:21.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.144+0000 7fd4e8f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4e415a1b0 0x7fd4e4150280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32970/0 (socket says 172.21.15.27:32970) 2024-01-11T08:22:21.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.144+0000 7fd4e8f9c700 1 -- 172.21.15.27:0/3122629639 learned_addr learned my addr 172.21.15.27:0/3122629639 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:21.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.144+0000 7fd4e3fff700 1 --2- 172.21.15.27:0/3122629639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4e40685c0 0x7fd4e41553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:21.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.144+0000 7fd4e37fe700 1 --2- 172.21.15.27:0/3122629639 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4e40ccd70 0x7fd4e415b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:21.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.144+0000 7fd4e8f9c700 1 -- 172.21.15.27:0/3122629639 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4e40ccd70 msgr2=0x7fd4e415b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:21.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.144+0000 7fd4e8f9c700 1 --2- 172.21.15.27:0/3122629639 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4e40ccd70 0x7fd4e415b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:21.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.144+0000 7fd4e8f9c700 1 -- 172.21.15.27:0/3122629639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4e40685c0 msgr2=0x7fd4e41553c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:21.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.144+0000 7fd4e8f9c700 1 --2- 172.21.15.27:0/3122629639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4e40685c0 0x7fd4e41553c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:21.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.144+0000 7fd4e8f9c700 1 -- 172.21.15.27:0/3122629639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d80094f0 con 0x7fd4e415a1b0 2024-01-11T08:22:21.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.145+0000 7fd4e37fe700 1 --2- 172.21.15.27:0/3122629639 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4e40ccd70 0x7fd4e415b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:22:21.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.145+0000 7fd4e8f9c700 1 --2- 172.21.15.27:0/3122629639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4e415a1b0 0x7fd4e4150280 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009e20 tx=0x7fd4dc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:21.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.145+0000 7fd4cbfff700 1 -- 172.21.15.27:0/3122629639 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc00cc20 con 0x7fd4e415a1b0 2024-01-11T08:22:21.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.145+0000 7fd4cbfff700 1 -- 172.21.15.27:0/3122629639 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4dc00f040 con 0x7fd4e415a1b0 2024-01-11T08:22:21.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.145+0000 7fd4ea9ff700 1 -- 172.21.15.27:0/3122629639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e40b2240 con 0x7fd4e415a1b0 2024-01-11T08:22:21.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.145+0000 7fd4cbfff700 1 -- 172.21.15.27:0/3122629639 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc014480 con 0x7fd4e415a1b0 2024-01-11T08:22:21.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.145+0000 7fd4ea9ff700 1 -- 172.21.15.27:0/3122629639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e4155900 con 0x7fd4e415a1b0 2024-01-11T08:22:21.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.146+0000 7fd4ea9ff700 1 -- 172.21.15.27:0/3122629639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e404c3c0 con 0x7fd4e415a1b0 2024-01-11T08:22:21.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.147+0000 7fd4cbfff700 1 -- 172.21.15.27:0/3122629639 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fd4dc014830 con 0x7fd4e415a1b0 2024-01-11T08:22:21.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.148+0000 7fd4cbfff700 1 --2- 172.21.15.27:0/3122629639 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd4d0067130 0x7fd4d00695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:21.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.148+0000 7fd4cbfff700 1 -- 172.21.15.27:0/3122629639 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fd4dc085570 con 0x7fd4e415a1b0 2024-01-11T08:22:21.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.149+0000 7fd4e3fff700 1 --2- 172.21.15.27:0/3122629639 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd4d0067130 0x7fd4d00695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:21.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.149+0000 7fd4e3fff700 1 --2- 172.21.15.27:0/3122629639 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd4d0067130 0x7fd4d00695e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009d40 tx=0x7fd4d4008040 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:21.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.152+0000 7fd4cbfff700 1 -- 172.21.15.27:0/3122629639 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd4dc0522b0 con 0x7fd4e415a1b0 2024-01-11T08:22:21.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.388+0000 7fd4ea9ff700 1 -- 172.21.15.27:0/3122629639 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4e40b2d60 con 0x7fd4d0067130 2024-01-11T08:22:21.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.464+0000 7fd4cbfff700 1 -- 172.21.15.27:0/3122629639 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fd4e40b2d60 con 0x7fd4d0067130 2024-01-11T08:22:21.465 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2024-01-11T08:22:21.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.467+0000 7fd4ea9ff700 1 -- 172.21.15.27:0/3122629639 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd4d0067130 msgr2=0x7fd4d00695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:21.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.467+0000 7fd4ea9ff700 1 --2- 172.21.15.27:0/3122629639 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd4d0067130 0x7fd4d00695e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009d40 tx=0x7fd4d4008040 comp rx=0 tx=0).stop 2024-01-11T08:22:21.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.467+0000 7fd4ea9ff700 1 -- 172.21.15.27:0/3122629639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4e415a1b0 msgr2=0x7fd4e4150280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:21.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.467+0000 7fd4ea9ff700 1 --2- 172.21.15.27:0/3122629639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4e415a1b0 0x7fd4e4150280 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009e20 tx=0x7fd4dc00b040 comp rx=0 tx=0).stop 2024-01-11T08:22:21.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.467+0000 7fd4ea9ff700 1 -- 172.21.15.27:0/3122629639 shutdown_connections 2024-01-11T08:22:21.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.467+0000 7fd4ea9ff700 1 --2- 172.21.15.27:0/3122629639 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd4d0067130 0x7fd4d00695e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:21.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.467+0000 7fd4ea9ff700 1 --2- 172.21.15.27:0/3122629639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4e415a1b0 0x7fd4e4150280 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:21.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.467+0000 7fd4ea9ff700 1 --2- 172.21.15.27:0/3122629639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4e40685c0 0x7fd4e41553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:21.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.467+0000 7fd4ea9ff700 1 --2- 172.21.15.27:0/3122629639 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4e40ccd70 0x7fd4e415b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:21.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.467+0000 7fd4ea9ff700 1 -- 172.21.15.27:0/3122629639 >> 172.21.15.27:0/3122629639 conn(0x7fd4e406ed40 msgr2=0x7fd4e415a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:21.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.467+0000 7fd4ea9ff700 1 -- 172.21.15.27:0/3122629639 shutdown_connections 2024-01-11T08:22:21.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.467+0000 7fd4ea9ff700 1 -- 172.21.15.27:0/3122629639 wait complete. 2024-01-11T08:22:21.488 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2024-01-11T08:22:21.488 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir2 1m 2024-01-11T08:22:21.512 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:22:21.543 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:22:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.695+0000 7f20133ff700 1 Processor -- start 2024-01-11T08:22:21.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.695+0000 7f20133ff700 1 -- start start 2024-01-11T08:22:21.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.697+0000 7f20133ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f200c069540 0x7f200c162270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:21.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.697+0000 7f20133ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f200c0699d0 con 0x7f200c1577e0 2024-01-11T08:22:21.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.697+0000 7f20133ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f200c069cb0 con 0x7f200c157410 2024-01-11T08:22:21.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.697+0000 7f20133ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f200c069f90 con 0x7f200c069540 2024-01-11T08:22:21.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.697+0000 7f201099a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f200c1577e0 0x7f200c0ae000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41584/0 (socket says 172.21.15.27:41584) 2024-01-11T08:22:21.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.697+0000 7f201099a700 1 -- 172.21.15.27:0/3331598191 learned_addr learned my addr 172.21.15.27:0/3331598191 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:21.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.697+0000 7f201199c700 1 --2- 172.21.15.27:0/3331598191 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f200c069540 0x7f200c162270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:21.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.697+0000 7f2003fff700 1 -- 172.21.15.27:0/3331598191 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219605117 0 0) 0x7f200c0699d0 con 0x7f200c1577e0 2024-01-11T08:22:21.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.697+0000 7f2003fff700 1 -- 172.21.15.27:0/3331598191 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ff8003570 con 0x7f200c1577e0 2024-01-11T08:22:21.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.698+0000 7f2003fff700 1 -- 172.21.15.27:0/3331598191 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610530252 0 0) 0x7f200c069cb0 con 0x7f200c157410 2024-01-11T08:22:21.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.698+0000 7f2003fff700 1 -- 172.21.15.27:0/3331598191 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f200c0699d0 con 0x7f200c157410 2024-01-11T08:22:21.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.698+0000 7f201199c700 1 -- 172.21.15.27:0/3331598191 >> v1:172.21.15.77:6789/0 conn(0x7f200c157410 legacy=0x7f200c0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:21.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.698+0000 7f201199c700 1 -- 172.21.15.27:0/3331598191 >> v1:172.21.15.27:6789/0 conn(0x7f200c1577e0 legacy=0x7f200c0ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:21.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.698+0000 7f201199c700 1 -- 172.21.15.27:0/3331598191 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f200c06a270 con 0x7f200c069540 2024-01-11T08:22:21.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.698+0000 7f201199c700 1 --2- 172.21.15.27:0/3331598191 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f200c069540 0x7f200c162270 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2004009e90 tx=0x7f2004009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=580f130dab1d0cf9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:21.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.698+0000 7f2003fff700 1 -- 172.21.15.27:0/3331598191 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200400e070 con 0x7f200c069540 2024-01-11T08:22:21.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.698+0000 7f2003fff700 1 -- 172.21.15.27:0/3331598191 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200400bb80 con 0x7f200c069540 2024-01-11T08:22:21.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.698+0000 7f2003fff700 1 -- 172.21.15.27:0/3331598191 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2004012430 con 0x7f200c069540 2024-01-11T08:22:21.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.699+0000 7f20133ff700 1 -- 172.21.15.27:0/3331598191 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f200c069540 msgr2=0x7f200c162270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:21.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.699+0000 7f20133ff700 1 --2- 172.21.15.27:0/3331598191 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f200c069540 0x7f200c162270 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2004009e90 tx=0x7f2004009780 comp rx=0 tx=0).stop 2024-01-11T08:22:21.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.699+0000 7f20133ff700 1 -- 172.21.15.27:0/3331598191 shutdown_connections 2024-01-11T08:22:21.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.699+0000 7f20133ff700 1 --2- 172.21.15.27:0/3331598191 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f200c069540 0x7f200c162270 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:21.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.699+0000 7f20133ff700 1 -- 172.21.15.27:0/3331598191 >> 172.21.15.27:0/3331598191 conn(0x7f200c0680a0 msgr2=0x7f200c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:21.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.699+0000 7f20133ff700 1 -- 172.21.15.27:0/3331598191 shutdown_connections 2024-01-11T08:22:21.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.699+0000 7f20133ff700 1 -- 172.21.15.27:0/3331598191 wait complete. 2024-01-11T08:22:21.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.701+0000 7f20133ff700 1 Processor -- start 2024-01-11T08:22:21.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.701+0000 7f20133ff700 1 -- start start 2024-01-11T08:22:21.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.702+0000 7f20133ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f200c069540 0x7f200c0c1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:21.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.702+0000 7f20133ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f200c157410 0x7f200c1508d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:21.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.702+0000 7f20133ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f200c1577e0 0x7f200c167500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:21.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.702+0000 7f20133ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200c0c21c0 con 0x7f200c157410 2024-01-11T08:22:21.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.702+0000 7f20133ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200c0c24a0 con 0x7f200c069540 2024-01-11T08:22:21.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.702+0000 7f20133ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f200c0c2780 con 0x7f200c1577e0 2024-01-11T08:22:21.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.702+0000 7f201099a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f200c157410 0x7f200c1508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:21.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.702+0000 7f201099a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f200c157410 0x7f200c1508d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32972/0 (socket says 172.21.15.27:32972) 2024-01-11T08:22:21.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.702+0000 7f201099a700 1 -- 172.21.15.27:0/1004337264 learned_addr learned my addr 172.21.15.27:0/1004337264 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:21.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.702+0000 7f201199c700 1 --2- 172.21.15.27:0/1004337264 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f200c1577e0 0x7f200c167500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:21.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.702+0000 7f201119b700 1 --2- 172.21.15.27:0/1004337264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f200c069540 0x7f200c0c1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:21.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.702+0000 7f201099a700 1 -- 172.21.15.27:0/1004337264 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f200c1577e0 msgr2=0x7f200c167500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:21.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.702+0000 7f201099a700 1 --2- 172.21.15.27:0/1004337264 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f200c1577e0 0x7f200c167500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:21.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.702+0000 7f201099a700 1 -- 172.21.15.27:0/1004337264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f200c069540 msgr2=0x7f200c0c1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:21.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.702+0000 7f201099a700 1 --2- 172.21.15.27:0/1004337264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f200c069540 0x7f200c0c1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:21.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.702+0000 7f201099a700 1 -- 172.21.15.27:0/1004337264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2004009a20 con 0x7f200c157410 2024-01-11T08:22:21.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.703+0000 7f201099a700 1 --2- 172.21.15.27:0/1004337264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f200c157410 0x7f200c1508d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00c580 tx=0x7f1ffc007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:21.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.703+0000 7f20017fa700 1 -- 172.21.15.27:0/1004337264 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc005b30 con 0x7f200c157410 2024-01-11T08:22:21.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.703+0000 7f20017fa700 1 -- 172.21.15.27:0/1004337264 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ffc005c90 con 0x7f200c157410 2024-01-11T08:22:21.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.703+0000 7f20133ff700 1 -- 172.21.15.27:0/1004337264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f200c168690 con 0x7f200c157410 2024-01-11T08:22:21.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.703+0000 7f20017fa700 1 -- 172.21.15.27:0/1004337264 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc007320 con 0x7f200c157410 2024-01-11T08:22:21.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.703+0000 7f20133ff700 1 -- 172.21.15.27:0/1004337264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f200c0c1eb0 con 0x7f200c157410 2024-01-11T08:22:21.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.704+0000 7f20133ff700 1 -- 172.21.15.27:0/1004337264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f200c168bb0 con 0x7f200c157410 2024-01-11T08:22:21.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.705+0000 7f20017fa700 1 -- 172.21.15.27:0/1004337264 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f1ffc01b3b0 con 0x7f200c157410 2024-01-11T08:22:21.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.706+0000 7f20017fa700 1 --2- 172.21.15.27:0/1004337264 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1ff0067130 0x7f1ff00695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:21.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.706+0000 7f20017fa700 1 -- 172.21.15.27:0/1004337264 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f1ffc083ec0 con 0x7f200c157410 2024-01-11T08:22:21.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.707+0000 7f201119b700 1 --2- 172.21.15.27:0/1004337264 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1ff0067130 0x7f1ff00695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:21.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.707+0000 7f201119b700 1 --2- 172.21.15.27:0/1004337264 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1ff0067130 0x7f1ff00695e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f200c153580 tx=0x7f2008007d40 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:21.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.710+0000 7f20017fa700 1 -- 172.21.15.27:0/1004337264 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f200c168bb0 con 0x7f200c157410 2024-01-11T08:22:21.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:21.945+0000 7f20133ff700 1 -- 172.21.15.27:0/1004337264 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f200c0c2610 con 0x7f1ff0067130 2024-01-11T08:22:22.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.015+0000 7f20017fa700 1 -- 172.21.15.27:0/1004337264 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f200c0c2610 con 0x7f1ff0067130 2024-01-11T08:22:22.016 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2024-01-11T08:22:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.018+0000 7f20133ff700 1 -- 172.21.15.27:0/1004337264 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1ff0067130 msgr2=0x7f1ff00695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.018+0000 7f20133ff700 1 --2- 172.21.15.27:0/1004337264 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1ff0067130 0x7f1ff00695e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f200c153580 tx=0x7f2008007d40 comp rx=0 tx=0).stop 2024-01-11T08:22:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.018+0000 7f20133ff700 1 -- 172.21.15.27:0/1004337264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f200c157410 msgr2=0x7f200c1508d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.018+0000 7f20133ff700 1 --2- 172.21.15.27:0/1004337264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f200c157410 0x7f200c1508d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00c580 tx=0x7f1ffc007e40 comp rx=0 tx=0).stop 2024-01-11T08:22:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.018+0000 7f20133ff700 1 -- 172.21.15.27:0/1004337264 shutdown_connections 2024-01-11T08:22:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.018+0000 7f20133ff700 1 --2- 172.21.15.27:0/1004337264 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1ff0067130 0x7f1ff00695e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.018+0000 7f20133ff700 1 --2- 172.21.15.27:0/1004337264 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f200c1577e0 0x7f200c167500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.018+0000 7f20133ff700 1 --2- 172.21.15.27:0/1004337264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f200c069540 0x7f200c0c1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.018+0000 7f20133ff700 1 --2- 172.21.15.27:0/1004337264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f200c157410 0x7f200c1508d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.018+0000 7f20133ff700 1 -- 172.21.15.27:0/1004337264 >> 172.21.15.27:0/1004337264 conn(0x7f200c0680a0 msgr2=0x7f200c0b1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.018+0000 7f20133ff700 1 -- 172.21.15.27:0/1004337264 shutdown_connections 2024-01-11T08:22:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.018+0000 7f20133ff700 1 -- 172.21.15.27:0/1004337264 wait complete. 2024-01-11T08:22:22.036 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2024-01-11T08:22:22.036 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir3 1m 2024-01-11T08:22:22.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.238+0000 7f06647ff700 1 Processor -- start 2024-01-11T08:22:22.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.239+0000 7f06647ff700 1 -- start start 2024-01-11T08:22:22.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.240+0000 7f06647ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f065c157410 0x7f065c0ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:22.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.240+0000 7f06647ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c069540 0x7f065c0c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:22.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.240+0000 7f06647ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065c0699d0 con 0x7f065c1577e0 2024-01-11T08:22:22.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.240+0000 7f06647ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f065c069cb0 con 0x7f065c157410 2024-01-11T08:22:22.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.240+0000 7f06647ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065c069f90 con 0x7f065c069540 2024-01-11T08:22:22.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.240+0000 7f0661d9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f065c1577e0 0x7f065c0ae000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41596/0 (socket says 172.21.15.27:41596) 2024-01-11T08:22:22.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.240+0000 7f0661d9a700 1 -- 172.21.15.27:0/3038020392 learned_addr learned my addr 172.21.15.27:0/3038020392 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:22.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.240+0000 7f066259b700 1 --2- 172.21.15.27:0/3038020392 >> v2:172.21.15.77:3300/0 conn(0x7f065c157410 0x7f065c0ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:22.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.240+0000 7f0662d9c700 1 --2- 172.21.15.27:0/3038020392 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c069540 0x7f065c0c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.241+0000 7f0661599700 1 -- 172.21.15.27:0/3038020392 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181514549 0 0) 0x7f065c0699d0 con 0x7f065c1577e0 2024-01-11T08:22:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.241+0000 7f0661599700 1 -- 172.21.15.27:0/3038020392 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0648003570 con 0x7f065c1577e0 2024-01-11T08:22:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.241+0000 7f066259b700 1 -- 172.21.15.27:0/3038020392 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c069540 msgr2=0x7f065c0c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.241+0000 7f066259b700 1 --2- 172.21.15.27:0/3038020392 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c069540 0x7f065c0c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.241+0000 7f066259b700 1 -- 172.21.15.27:0/3038020392 >> v1:172.21.15.27:6789/0 conn(0x7f065c1577e0 legacy=0x7f065c0ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.241+0000 7f066259b700 1 -- 172.21.15.27:0/3038020392 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065c06a270 con 0x7f065c157410 2024-01-11T08:22:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.241+0000 7f066259b700 1 --2- 172.21.15.27:0/3038020392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f065c157410 0x7f065c0ccae0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f06580098d0 tx=0x7f0658009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=300b0c8b0a9d75c2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.241+0000 7f0661599700 1 -- 172.21.15.27:0/3038020392 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0658005c10 con 0x7f065c157410 2024-01-11T08:22:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.241+0000 7f0661599700 1 -- 172.21.15.27:0/3038020392 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0658005d70 con 0x7f065c157410 2024-01-11T08:22:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.241+0000 7f0661599700 1 -- 172.21.15.27:0/3038020392 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0658006260 con 0x7f065c157410 2024-01-11T08:22:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.242+0000 7f06647ff700 1 -- 172.21.15.27:0/3038020392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f065c157410 msgr2=0x7f065c0ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.242+0000 7f06647ff700 1 --2- 172.21.15.27:0/3038020392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f065c157410 0x7f065c0ccae0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f06580098d0 tx=0x7f0658009120 comp rx=0 tx=0).stop 2024-01-11T08:22:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.242+0000 7f06647ff700 1 -- 172.21.15.27:0/3038020392 shutdown_connections 2024-01-11T08:22:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.242+0000 7f06647ff700 1 --2- 172.21.15.27:0/3038020392 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c069540 0x7f065c0c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.242+0000 7f06647ff700 1 --2- 172.21.15.27:0/3038020392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f065c157410 0x7f065c0ccae0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.242+0000 7f06647ff700 1 -- 172.21.15.27:0/3038020392 >> 172.21.15.27:0/3038020392 conn(0x7f065c0680a0 msgr2=0x7f065c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.242+0000 7f06647ff700 1 -- 172.21.15.27:0/3038020392 shutdown_connections 2024-01-11T08:22:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.242+0000 7f06647ff700 1 -- 172.21.15.27:0/3038020392 wait complete. 2024-01-11T08:22:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.244+0000 7f06647ff700 1 Processor -- start 2024-01-11T08:22:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.244+0000 7f06647ff700 1 -- start start 2024-01-11T08:22:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.244+0000 7f06647ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f065c069540 0x7f065c151e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.245+0000 7f06647ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f065c157410 0x7f065c15d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.245+0000 7f06647ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c1577e0 0x7f065c153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.245+0000 7f06647ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c154bc0 con 0x7f065c157410 2024-01-11T08:22:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.245+0000 7f06647ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c1519c0 con 0x7f065c069540 2024-01-11T08:22:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.245+0000 7f06647ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065c1548e0 con 0x7f065c1577e0 2024-01-11T08:22:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.245+0000 7f0661d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f065c157410 0x7f065c15d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.245+0000 7f0661d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f065c157410 0x7f065c15d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32978/0 (socket says 172.21.15.27:32978) 2024-01-11T08:22:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.245+0000 7f0661d9a700 1 -- 172.21.15.27:0/3957820719 learned_addr learned my addr 172.21.15.27:0/3957820719 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.245+0000 7f066259b700 1 --2- 172.21.15.27:0/3957820719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f065c069540 0x7f065c151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.245+0000 7f0662d9c700 1 --2- 172.21.15.27:0/3957820719 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c1577e0 0x7f065c153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.245+0000 7f0661d9a700 1 -- 172.21.15.27:0/3957820719 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c1577e0 msgr2=0x7f065c153f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.245+0000 7f0661d9a700 1 --2- 172.21.15.27:0/3957820719 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c1577e0 0x7f065c153f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.245+0000 7f0661d9a700 1 -- 172.21.15.27:0/3957820719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f065c069540 msgr2=0x7f065c151e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.245+0000 7f0661d9a700 1 --2- 172.21.15.27:0/3957820719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f065c069540 0x7f065c151e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.245+0000 7f0661d9a700 1 -- 172.21.15.27:0/3957820719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06580093c0 con 0x7f065c157410 2024-01-11T08:22:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.246+0000 7f0661d9a700 1 --2- 172.21.15.27:0/3957820719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f065c157410 0x7f065c15d5e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f064c00c560 tx=0x7f064c007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.246+0000 7f06527fc700 1 -- 172.21.15.27:0/3957820719 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c005b30 con 0x7f065c157410 2024-01-11T08:22:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.246+0000 7f06527fc700 1 -- 172.21.15.27:0/3957820719 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064c005c90 con 0x7f065c157410 2024-01-11T08:22:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.246+0000 7f06527fc700 1 -- 172.21.15.27:0/3957820719 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c0072e0 con 0x7f065c157410 2024-01-11T08:22:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.246+0000 7f06647ff700 1 -- 172.21.15.27:0/3957820719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065c151370 con 0x7f065c157410 2024-01-11T08:22:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.246+0000 7f06647ff700 1 -- 172.21.15.27:0/3957820719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f065c152390 con 0x7f065c157410 2024-01-11T08:22:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.247+0000 7f06647ff700 1 -- 172.21.15.27:0/3957820719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065c151b30 con 0x7f065c157410 2024-01-11T08:22:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.248+0000 7f06527fc700 1 -- 172.21.15.27:0/3957820719 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f064c01b3b0 con 0x7f065c157410 2024-01-11T08:22:22.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.249+0000 7f06527fc700 1 --2- 172.21.15.27:0/3957820719 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0640067130 0x7f06400695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:22.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.249+0000 7f06527fc700 1 -- 172.21.15.27:0/3957820719 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f064c083ce0 con 0x7f065c157410 2024-01-11T08:22:22.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.249+0000 7f066259b700 1 --2- 172.21.15.27:0/3957820719 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0640067130 0x7f06400695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:22.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.250+0000 7f066259b700 1 --2- 172.21.15.27:0/3957820719 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0640067130 0x7f06400695e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0658009160 tx=0x7f0658016000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:22.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.253+0000 7f06527fc700 1 -- 172.21.15.27:0/3957820719 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f064c04b5a0 con 0x7f065c157410 2024-01-11T08:22:22.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.486+0000 7f06647ff700 1 -- 172.21.15.27:0/3957820719 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f065c04c3c0 con 0x7f0640067130 2024-01-11T08:22:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.563+0000 7f06527fc700 1 -- 172.21.15.27:0/3957820719 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f065c04c3c0 con 0x7f0640067130 2024-01-11T08:22:22.563 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2024-01-11T08:22:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.566+0000 7f06647ff700 1 -- 172.21.15.27:0/3957820719 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0640067130 msgr2=0x7f06400695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.566+0000 7f06647ff700 1 --2- 172.21.15.27:0/3957820719 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0640067130 0x7f06400695e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0658009160 tx=0x7f0658016000 comp rx=0 tx=0).stop 2024-01-11T08:22:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.566+0000 7f06647ff700 1 -- 172.21.15.27:0/3957820719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f065c157410 msgr2=0x7f065c15d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.566+0000 7f06647ff700 1 --2- 172.21.15.27:0/3957820719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f065c157410 0x7f065c15d5e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f064c00c560 tx=0x7f064c007dd0 comp rx=0 tx=0).stop 2024-01-11T08:22:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.566+0000 7f06647ff700 1 -- 172.21.15.27:0/3957820719 shutdown_connections 2024-01-11T08:22:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.566+0000 7f06647ff700 1 --2- 172.21.15.27:0/3957820719 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0640067130 0x7f06400695e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.566+0000 7f06647ff700 1 --2- 172.21.15.27:0/3957820719 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c1577e0 0x7f065c153f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.566+0000 7f06647ff700 1 --2- 172.21.15.27:0/3957820719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f065c069540 0x7f065c151e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.566+0000 7f06647ff700 1 --2- 172.21.15.27:0/3957820719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f065c157410 0x7f065c15d5e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.566+0000 7f06647ff700 1 -- 172.21.15.27:0/3957820719 >> 172.21.15.27:0/3957820719 conn(0x7f065c0680a0 msgr2=0x7f065c0c25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.566+0000 7f06647ff700 1 -- 172.21.15.27:0/3957820719 shutdown_connections 2024-01-11T08:22:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.566+0000 7f06647ff700 1 -- 172.21.15.27:0/3957820719 wait complete. 2024-01-11T08:22:22.583 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2024-01-11T08:22:22.583 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir4 1m 2024-01-11T08:22:22.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.786+0000 7fcdfe9ff700 1 Processor -- start 2024-01-11T08:22:22.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.786+0000 7fcdfe9ff700 1 -- start start 2024-01-11T08:22:22.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.788+0000 7fcdfe9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcdf813ff80 0x7fcdf815a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:22.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.788+0000 7fcdfe9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf815a9c0 con 0x7fcdf813fbb0 2024-01-11T08:22:22.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.788+0000 7fcdfe9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf80c6b30 con 0x7fcdf815a560 2024-01-11T08:22:22.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.788+0000 7fcdfe9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf80c6800 con 0x7fcdf813ff80 2024-01-11T08:22:22.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.788+0000 7fcdf7fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcdf813fbb0 0x7fcdf80ca7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41608/0 (socket says 172.21.15.27:41608) 2024-01-11T08:22:22.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.788+0000 7fcdf7fff700 1 -- 172.21.15.27:0/2171721772 learned_addr learned my addr 172.21.15.27:0/2171721772 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:22.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.788+0000 7fcdf77fe700 1 --2- 172.21.15.27:0/2171721772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcdf813ff80 0x7fcdf815a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:22.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.788+0000 7fcdf6ffd700 1 -- 172.21.15.27:0/2171721772 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077416160 0 0) 0x7fcde8001290 con 0x7fcdf813fbb0 2024-01-11T08:22:22.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdf6ffd700 1 -- 172.21.15.27:0/2171721772 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcde0003570 con 0x7fcdf813fbb0 2024-01-11T08:22:22.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdf6ffd700 1 -- 172.21.15.27:0/2171721772 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991818618 0 0) 0x7fcdf80c6b30 con 0x7fcdf815a560 2024-01-11T08:22:22.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdf6ffd700 1 -- 172.21.15.27:0/2171721772 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcde8001290 con 0x7fcdf815a560 2024-01-11T08:22:22.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdf6ffd700 1 -- 172.21.15.27:0/2171721772 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3272267109 0 0) 0x7fcde0003570 con 0x7fcdf813fbb0 2024-01-11T08:22:22.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdf6ffd700 1 -- 172.21.15.27:0/2171721772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcdf813ff80 msgr2=0x7fcdf815a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:22.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdf6ffd700 1 --2- 172.21.15.27:0/2171721772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcdf813ff80 0x7fcdf815a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:22.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdf6ffd700 1 -- 172.21.15.27:0/2171721772 >> v1:172.21.15.77:6789/0 conn(0x7fcdf815a560 legacy=0x7fcdf80c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:22.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdf6ffd700 1 -- 172.21.15.27:0/2171721772 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf80cb240 con 0x7fcdf813fbb0 2024-01-11T08:22:22.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdf6ffd700 1 -- 172.21.15.27:0/2171721772 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fcde8001780 con 0x7fcdf813fbb0 2024-01-11T08:22:22.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdf77fe700 1 --2- 172.21.15.27:0/2171721772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcdf813ff80 0x7fcdf815a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:22:22.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdf6ffd700 1 -- 172.21.15.27:0/2171721772 >> v1:172.21.15.27:6789/0 conn(0x7fcdf813fbb0 legacy=0x7fcdf80ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:22.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdf6ffd700 1 --2- 172.21.15.27:0/2171721772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde0003c20 0x7fcde0004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:22.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdf6ffd700 1 --2- 172.21.15.27:0/2171721772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcde0004810 0x7fcde0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:22.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdf6ffd700 1 --2- 172.21.15.27:0/2171721772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcdf813ff80 0x7fcde00051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:22.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdf6ffd700 1 -- 172.21.15.27:0/2171721772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf80c6800 con 0x7fcde0003c20 2024-01-11T08:22:22.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdf6ffd700 1 -- 172.21.15.27:0/2171721772 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde000bbb0 con 0x7fcde0004810 2024-01-11T08:22:22.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdf6ffd700 1 -- 172.21.15.27:0/2171721772 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde000bd70 con 0x7fcdf813ff80 2024-01-11T08:22:22.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdf7fff700 1 --2- 172.21.15.27:0/2171721772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde0003c20 0x7fcde0004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:22.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdf77fe700 1 --2- 172.21.15.27:0/2171721772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcde0004810 0x7fcde0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:22.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.789+0000 7fcdfcf9c700 1 --2- 172.21.15.27:0/2171721772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcdf813ff80 0x7fcde00051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:22.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.790+0000 7fcdf7fff700 1 -- 172.21.15.27:0/2171721772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcdf813ff80 msgr2=0x7fcde00051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:22.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.790+0000 7fcdf7fff700 1 --2- 172.21.15.27:0/2171721772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcdf813ff80 0x7fcde00051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:22.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.790+0000 7fcdf7fff700 1 -- 172.21.15.27:0/2171721772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcde0004810 msgr2=0x7fcde0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:22.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.790+0000 7fcdf7fff700 1 --2- 172.21.15.27:0/2171721772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcde0004810 0x7fcde0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:22.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.790+0000 7fcdf7fff700 1 -- 172.21.15.27:0/2171721772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec009420 con 0x7fcde0003c20 2024-01-11T08:22:22.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.790+0000 7fcdfcf9c700 1 --2- 172.21.15.27:0/2171721772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcdf813ff80 0x7fcde00051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:22:22.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.790+0000 7fcdf7fff700 1 --2- 172.21.15.27:0/2171721772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde0003c20 0x7fcde0004010 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fcde800c3c0 tx=0x7fcde8007970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df4e3eb896ba4206 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:22.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.790+0000 7fcdf6ffd700 1 -- 172.21.15.27:0/2171721772 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde800cc90 con 0x7fcde0003c20 2024-01-11T08:22:22.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.790+0000 7fcdf6ffd700 1 -- 172.21.15.27:0/2171721772 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde800ce10 con 0x7fcde0003c20 2024-01-11T08:22:22.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.790+0000 7fcdf6ffd700 1 -- 172.21.15.27:0/2171721772 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde8007320 con 0x7fcde0003c20 2024-01-11T08:22:22.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.791+0000 7fcdfe9ff700 1 -- 172.21.15.27:0/2171721772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde0003c20 msgr2=0x7fcde0004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:22.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.791+0000 7fcdfe9ff700 1 --2- 172.21.15.27:0/2171721772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde0003c20 0x7fcde0004010 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fcde800c3c0 tx=0x7fcde8007970 comp rx=0 tx=0).stop 2024-01-11T08:22:22.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.791+0000 7fcdfcf9c700 1 -- 172.21.15.27:0/2171721772 reap_dead start 2024-01-11T08:22:22.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.791+0000 7fcdfe9ff700 1 -- 172.21.15.27:0/2171721772 shutdown_connections 2024-01-11T08:22:22.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.791+0000 7fcdfe9ff700 1 -- 172.21.15.27:0/2171721772 >> 172.21.15.27:0/2171721772 conn(0x7fcdf80680a0 msgr2=0x7fcdf80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:22.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.791+0000 7fcdfe9ff700 1 -- 172.21.15.27:0/2171721772 shutdown_connections 2024-01-11T08:22:22.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.791+0000 7fcdfe9ff700 1 -- 172.21.15.27:0/2171721772 wait complete. 2024-01-11T08:22:22.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.792+0000 7fcdfe9ff700 1 Processor -- start 2024-01-11T08:22:22.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.792+0000 7fcdfe9ff700 1 -- start start 2024-01-11T08:22:22.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.793+0000 7fcdfe9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde0003c20 0x7fcdf8155f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:22.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.793+0000 7fcdfe9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcdf8158e20 0x7fcdf80c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:22.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.793+0000 7fcdfe9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdf8159290 0x7fcdf80c7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:22.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.793+0000 7fcdfe9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf8156460 con 0x7fcde0003c20 2024-01-11T08:22:22.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.793+0000 7fcdfe9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf8156e50 con 0x7fcdf8159290 2024-01-11T08:22:22.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.793+0000 7fcdfe9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf8157130 con 0x7fcdf8158e20 2024-01-11T08:22:22.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.794+0000 7fcdf7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde0003c20 0x7fcdf8155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:22.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.794+0000 7fcdf7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde0003c20 0x7fcdf8155f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33000/0 (socket says 172.21.15.27:33000) 2024-01-11T08:22:22.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.794+0000 7fcdf7fff700 1 -- 172.21.15.27:0/4210767115 learned_addr learned my addr 172.21.15.27:0/4210767115 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:22.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.794+0000 7fcdfcf9c700 1 --2- 172.21.15.27:0/4210767115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdf8159290 0x7fcdf80c7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:22.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.794+0000 7fcdf77fe700 1 --2- 172.21.15.27:0/4210767115 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcdf8158e20 0x7fcdf80c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:22.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.794+0000 7fcdf7fff700 1 -- 172.21.15.27:0/4210767115 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcdf8158e20 msgr2=0x7fcdf80c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:22.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.794+0000 7fcdf7fff700 1 --2- 172.21.15.27:0/4210767115 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcdf8158e20 0x7fcdf80c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:22.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.794+0000 7fcdf7fff700 1 -- 172.21.15.27:0/4210767115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdf8159290 msgr2=0x7fcdf80c7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:22.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.794+0000 7fcdf7fff700 1 --2- 172.21.15.27:0/4210767115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdf8159290 0x7fcdf80c7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:22.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.794+0000 7fcdf7fff700 1 -- 172.21.15.27:0/4210767115 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf8157410 con 0x7fcde0003c20 2024-01-11T08:22:22.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.794+0000 7fcdf7fff700 1 --2- 172.21.15.27:0/4210767115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde0003c20 0x7fcdf8155f20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fcde000bb80 tx=0x7fcde8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:22.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.794+0000 7fcddffff700 1 -- 172.21.15.27:0/4210767115 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde8007690 con 0x7fcde0003c20 2024-01-11T08:22:22.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.795+0000 7fcddffff700 1 -- 172.21.15.27:0/4210767115 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde8001680 con 0x7fcde0003c20 2024-01-11T08:22:22.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.795+0000 7fcddffff700 1 -- 172.21.15.27:0/4210767115 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde8007320 con 0x7fcde0003c20 2024-01-11T08:22:22.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.795+0000 7fcdfe9ff700 1 -- 172.21.15.27:0/4210767115 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf81576f0 con 0x7fcde0003c20 2024-01-11T08:22:22.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.795+0000 7fcdfe9ff700 1 -- 172.21.15.27:0/4210767115 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf8156a90 con 0x7fcde0003c20 2024-01-11T08:22:22.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.796+0000 7fcdfe9ff700 1 -- 172.21.15.27:0/4210767115 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf804c3c0 con 0x7fcde0003c20 2024-01-11T08:22:22.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.797+0000 7fcddffff700 1 -- 172.21.15.27:0/4210767115 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fcde8019030 con 0x7fcde0003c20 2024-01-11T08:22:22.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.798+0000 7fcddffff700 1 --2- 172.21.15.27:0/4210767115 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcde4067130 0x7fcde40695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:22.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.798+0000 7fcddffff700 1 -- 172.21.15.27:0/4210767115 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fcde8085840 con 0x7fcde0003c20 2024-01-11T08:22:22.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.798+0000 7fcdf77fe700 1 --2- 172.21.15.27:0/4210767115 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcde4067130 0x7fcde40695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:22.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.799+0000 7fcdf77fe700 1 --2- 172.21.15.27:0/4210767115 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcde4067130 0x7fcde40695e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcdf813ee90 tx=0x7fcdec0091a0 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:22.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:22.802+0000 7fcddffff700 1 -- 172.21.15.27:0/4210767115 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcde80527f0 con 0x7fcde0003c20 2024-01-11T08:22:23.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.036+0000 7fcdfe9ff700 1 -- 172.21.15.27:0/4210767115 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdf8156460 con 0x7fcde4067130 2024-01-11T08:22:23.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.113+0000 7fcddffff700 1 -- 172.21.15.27:0/4210767115 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fcdf8156460 con 0x7fcde4067130 2024-01-11T08:22:23.112 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2024-01-11T08:22:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.115+0000 7fcdfe9ff700 1 -- 172.21.15.27:0/4210767115 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcde4067130 msgr2=0x7fcde40695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.115+0000 7fcdfe9ff700 1 --2- 172.21.15.27:0/4210767115 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcde4067130 0x7fcde40695e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcdf813ee90 tx=0x7fcdec0091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.115+0000 7fcdfe9ff700 1 -- 172.21.15.27:0/4210767115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde0003c20 msgr2=0x7fcdf8155f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.115+0000 7fcdfe9ff700 1 --2- 172.21.15.27:0/4210767115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde0003c20 0x7fcdf8155f20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fcde000bb80 tx=0x7fcde8015000 comp rx=0 tx=0).stop 2024-01-11T08:22:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.115+0000 7fcdfe9ff700 1 -- 172.21.15.27:0/4210767115 shutdown_connections 2024-01-11T08:22:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.115+0000 7fcdfe9ff700 1 --2- 172.21.15.27:0/4210767115 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcde4067130 0x7fcde40695e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.115+0000 7fcdfe9ff700 1 --2- 172.21.15.27:0/4210767115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdf8159290 0x7fcdf80c7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.115+0000 7fcdfe9ff700 1 --2- 172.21.15.27:0/4210767115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde0003c20 0x7fcdf8155f20 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.115+0000 7fcdfe9ff700 1 --2- 172.21.15.27:0/4210767115 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcdf8158e20 0x7fcdf80c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.115+0000 7fcdfe9ff700 1 -- 172.21.15.27:0/4210767115 >> 172.21.15.27:0/4210767115 conn(0x7fcdf80680a0 msgr2=0x7fcdf80c8110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.116+0000 7fcdfe9ff700 1 -- 172.21.15.27:0/4210767115 shutdown_connections 2024-01-11T08:22:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.116+0000 7fcdfe9ff700 1 -- 172.21.15.27:0/4210767115 wait complete. 2024-01-11T08:22:23.133 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2024-01-11T08:22:23.133 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir5 1m 2024-01-11T08:22:23.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.336+0000 7fef45dff700 1 Processor -- start 2024-01-11T08:22:23.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.336+0000 7fef45dff700 1 -- start start 2024-01-11T08:22:23.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.337+0000 7fef45dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fef4015a1b0 0x7fef4015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:23.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.338+0000 7fef45dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef400ccd70 0x7fef400cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:23.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.338+0000 7fef45dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef400cf850 con 0x7fef400685c0 2024-01-11T08:22:23.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.338+0000 7fef45dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef400cec40 con 0x7fef4015a1b0 2024-01-11T08:22:23.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.338+0000 7fef45dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef400cef20 con 0x7fef400ccd70 2024-01-11T08:22:23.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.338+0000 7fef3f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fef4015a1b0 0x7fef4015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:23.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.338+0000 7fef3f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fef4015a1b0 0x7fef4015a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33006/0 (socket says 172.21.15.27:33006) 2024-01-11T08:22:23.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.338+0000 7fef3f7fe700 1 -- 172.21.15.27:0/2546542064 learned_addr learned my addr 172.21.15.27:0/2546542064 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:23.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.338+0000 7fef3effd700 1 --2- 172.21.15.27:0/2546542064 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef400ccd70 0x7fef400cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:23.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.338+0000 7fef3f7fe700 1 -- 172.21.15.27:0/2546542064 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef400ccd70 msgr2=0x7fef400cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:23.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.338+0000 7fef3f7fe700 1 --2- 172.21.15.27:0/2546542064 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef400ccd70 0x7fef400cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.338+0000 7fef3f7fe700 1 -- 172.21.15.27:0/2546542064 >> v1:172.21.15.27:6789/0 conn(0x7fef400685c0 legacy=0x7fef40068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:23.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.338+0000 7fef3f7fe700 1 -- 172.21.15.27:0/2546542064 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef400cf200 con 0x7fef4015a1b0 2024-01-11T08:22:23.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.338+0000 7fef3e7fc700 1 -- 172.21.15.27:0/2546542064 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149928246 0 0) 0x7fef400cf850 con 0x7fef400685c0 2024-01-11T08:22:23.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.339+0000 7fef3f7fe700 1 --2- 172.21.15.27:0/2546542064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef4015a1b0 0x7fef4015a580 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fef300099b0 tx=0x7fef30009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35b588a76b92c4a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:23.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.339+0000 7fef3e7fc700 1 -- 172.21.15.27:0/2546542064 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef30009ea0 con 0x7fef4015a1b0 2024-01-11T08:22:23.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.339+0000 7fef3e7fc700 1 -- 172.21.15.27:0/2546542064 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3000b860 con 0x7fef4015a1b0 2024-01-11T08:22:23.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.339+0000 7fef3e7fc700 1 -- 172.21.15.27:0/2546542064 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3000bc00 con 0x7fef4015a1b0 2024-01-11T08:22:23.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.339+0000 7fef45dff700 1 -- 172.21.15.27:0/2546542064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef4015a1b0 msgr2=0x7fef4015a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:23.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.339+0000 7fef45dff700 1 --2- 172.21.15.27:0/2546542064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef4015a1b0 0x7fef4015a580 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fef300099b0 tx=0x7fef30009250 comp rx=0 tx=0).stop 2024-01-11T08:22:23.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.340+0000 7fef45dff700 1 -- 172.21.15.27:0/2546542064 shutdown_connections 2024-01-11T08:22:23.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.340+0000 7fef45dff700 1 --2- 172.21.15.27:0/2546542064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef4015a1b0 0x7fef4015a580 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.340+0000 7fef45dff700 1 --2- 172.21.15.27:0/2546542064 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef400ccd70 0x7fef400cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.340+0000 7fef45dff700 1 -- 172.21.15.27:0/2546542064 >> 172.21.15.27:0/2546542064 conn(0x7fef4006ed40 msgr2=0x7fef4006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:23.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.340+0000 7fef45dff700 1 -- 172.21.15.27:0/2546542064 shutdown_connections 2024-01-11T08:22:23.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.340+0000 7fef45dff700 1 -- 172.21.15.27:0/2546542064 wait complete. 2024-01-11T08:22:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.341+0000 7fef45dff700 1 Processor -- start 2024-01-11T08:22:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.341+0000 7fef45dff700 1 -- start start 2024-01-11T08:22:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.342+0000 7fef45dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef400685c0 0x7fef40159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.342+0000 7fef45dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef400ccd70 0x7fef400b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.342+0000 7fef45dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef4015a1b0 0x7fef400b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.342+0000 7fef45dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef400a7390 con 0x7fef400ccd70 2024-01-11T08:22:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.342+0000 7fef45dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef40159770 con 0x7fef400685c0 2024-01-11T08:22:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.342+0000 7fef45dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef400c8dd0 con 0x7fef4015a1b0 2024-01-11T08:22:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.342+0000 7fef3effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef400ccd70 0x7fef400b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.342+0000 7fef3effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef400ccd70 0x7fef400b54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33008/0 (socket says 172.21.15.27:33008) 2024-01-11T08:22:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.342+0000 7fef3effd700 1 -- 172.21.15.27:0/516929990 learned_addr learned my addr 172.21.15.27:0/516929990 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.343+0000 7fef3f7fe700 1 --2- 172.21.15.27:0/516929990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef400685c0 0x7fef40159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.343+0000 7fef3ffff700 1 --2- 172.21.15.27:0/516929990 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef4015a1b0 0x7fef400b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.343+0000 7fef3effd700 1 -- 172.21.15.27:0/516929990 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef4015a1b0 msgr2=0x7fef400b5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.343+0000 7fef3effd700 1 --2- 172.21.15.27:0/516929990 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef4015a1b0 0x7fef400b5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.343+0000 7fef3effd700 1 -- 172.21.15.27:0/516929990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef400685c0 msgr2=0x7fef40159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.343+0000 7fef3effd700 1 --2- 172.21.15.27:0/516929990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef400685c0 0x7fef40159230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.343+0000 7fef3effd700 1 -- 172.21.15.27:0/516929990 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef300094f0 con 0x7fef400ccd70 2024-01-11T08:22:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.343+0000 7fef3ffff700 1 --2- 172.21.15.27:0/516929990 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef4015a1b0 0x7fef400b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:22:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.343+0000 7fef3effd700 1 --2- 172.21.15.27:0/516929990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef400ccd70 0x7fef400b54d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fef3400ec80 tx=0x7fef340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.343+0000 7fef277fe700 1 -- 172.21.15.27:0/516929990 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3400c850 con 0x7fef400ccd70 2024-01-11T08:22:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.343+0000 7fef277fe700 1 -- 172.21.15.27:0/516929990 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3400c9b0 con 0x7fef400ccd70 2024-01-11T08:22:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.343+0000 7fef277fe700 1 -- 172.21.15.27:0/516929990 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3400cd50 con 0x7fef400ccd70 2024-01-11T08:22:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.344+0000 7fef45dff700 1 -- 172.21.15.27:0/516929990 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef400c90b0 con 0x7fef400ccd70 2024-01-11T08:22:23.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.344+0000 7fef45dff700 1 -- 172.21.15.27:0/516929990 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef400c9390 con 0x7fef400ccd70 2024-01-11T08:22:23.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.345+0000 7fef45dff700 1 -- 172.21.15.27:0/516929990 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef4004c3c0 con 0x7fef400ccd70 2024-01-11T08:22:23.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.345+0000 7fef277fe700 1 -- 172.21.15.27:0/516929990 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fef34013680 con 0x7fef400ccd70 2024-01-11T08:22:23.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.346+0000 7fef277fe700 1 --2- 172.21.15.27:0/516929990 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fef2c067130 0x7fef2c0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.346+0000 7fef277fe700 1 -- 172.21.15.27:0/516929990 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fef34084fe0 con 0x7fef400ccd70 2024-01-11T08:22:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.347+0000 7fef3f7fe700 1 --2- 172.21.15.27:0/516929990 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fef2c067130 0x7fef2c0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.348+0000 7fef3f7fe700 1 --2- 172.21.15.27:0/516929990 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fef2c067130 0x7fef2c0695e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fef300099b0 tx=0x7fef30017000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:23.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.351+0000 7fef277fe700 1 -- 172.21.15.27:0/516929990 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fef34051e90 con 0x7fef400ccd70 2024-01-11T08:22:23.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.583+0000 7fef45dff700 1 -- 172.21.15.27:0/516929990 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fef400a74d0 con 0x7fef2c067130 2024-01-11T08:22:23.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.658+0000 7fef277fe700 1 -- 172.21.15.27:0/516929990 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fef400a74d0 con 0x7fef2c067130 2024-01-11T08:22:23.657 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2024-01-11T08:22:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.660+0000 7fef45dff700 1 -- 172.21.15.27:0/516929990 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fef2c067130 msgr2=0x7fef2c0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.660+0000 7fef45dff700 1 --2- 172.21.15.27:0/516929990 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fef2c067130 0x7fef2c0695e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fef300099b0 tx=0x7fef30017000 comp rx=0 tx=0).stop 2024-01-11T08:22:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.660+0000 7fef45dff700 1 -- 172.21.15.27:0/516929990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef400ccd70 msgr2=0x7fef400b54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.660+0000 7fef45dff700 1 --2- 172.21.15.27:0/516929990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef400ccd70 0x7fef400b54d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fef3400ec80 tx=0x7fef340091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.660+0000 7fef45dff700 1 -- 172.21.15.27:0/516929990 shutdown_connections 2024-01-11T08:22:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.660+0000 7fef45dff700 1 --2- 172.21.15.27:0/516929990 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fef2c067130 0x7fef2c0695e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.660+0000 7fef45dff700 1 --2- 172.21.15.27:0/516929990 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef4015a1b0 0x7fef400b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.660+0000 7fef45dff700 1 --2- 172.21.15.27:0/516929990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef400685c0 0x7fef40159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.660+0000 7fef45dff700 1 --2- 172.21.15.27:0/516929990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef400ccd70 0x7fef400b54d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.660+0000 7fef45dff700 1 -- 172.21.15.27:0/516929990 >> 172.21.15.27:0/516929990 conn(0x7fef4006ed40 msgr2=0x7fef400b24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.660+0000 7fef45dff700 1 -- 172.21.15.27:0/516929990 shutdown_connections 2024-01-11T08:22:23.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.660+0000 7fef45dff700 1 -- 172.21.15.27:0/516929990 wait complete. 2024-01-11T08:22:23.677 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2024-01-11T08:22:23.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir6 1m 2024-01-11T08:22:23.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.881+0000 7ff0a39ff700 1 Processor -- start 2024-01-11T08:22:23.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.881+0000 7ff0a39ff700 1 -- start start 2024-01-11T08:22:23.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.882+0000 7ff0a39ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff09c15a1b0 0x7ff09c15a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:23.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.882+0000 7ff0a39ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff09c0ccd70 0x7ff09c0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:23.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.882+0000 7ff0a39ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff09c0cf850 con 0x7ff09c0685c0 2024-01-11T08:22:23.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.882+0000 7ff0a39ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff09c0cec40 con 0x7ff09c0ccd70 2024-01-11T08:22:23.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.882+0000 7ff0a39ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c0cef20 con 0x7ff09c15a1b0 2024-01-11T08:22:23.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.882+0000 7ff0a1f9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff09c0685c0 0x7ff09c068a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54380/0 (socket says 172.21.15.27:54380) 2024-01-11T08:22:23.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.882+0000 7ff0a1f9c700 1 -- 172.21.15.27:0/2813908812 learned_addr learned my addr 172.21.15.27:0/2813908812 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:23.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.883+0000 7ff0a179b700 1 --2- 172.21.15.27:0/2813908812 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff09c15a1b0 0x7ff09c15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:23.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.883+0000 7ff0a0f9a700 1 --2- 172.21.15.27:0/2813908812 >> v2:172.21.15.77:3300/0 conn(0x7ff09c0ccd70 0x7ff09c0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:23.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.883+0000 7ff093fff700 1 -- 172.21.15.27:0/2813908812 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286836176 0 0) 0x7ff09c0cf850 con 0x7ff09c0685c0 2024-01-11T08:22:23.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.883+0000 7ff093fff700 1 -- 172.21.15.27:0/2813908812 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff088003570 con 0x7ff09c0685c0 2024-01-11T08:22:23.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.883+0000 7ff0a179b700 1 -- 172.21.15.27:0/2813908812 >> v2:172.21.15.77:3300/0 conn(0x7ff09c0ccd70 msgr2=0x7ff09c0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:23.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.883+0000 7ff0a179b700 1 --2- 172.21.15.27:0/2813908812 >> v2:172.21.15.77:3300/0 conn(0x7ff09c0ccd70 0x7ff09c0cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.883+0000 7ff0a179b700 1 -- 172.21.15.27:0/2813908812 >> v1:172.21.15.27:6789/0 conn(0x7ff09c0685c0 legacy=0x7ff09c068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:23.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.883+0000 7ff093fff700 1 -- 172.21.15.27:0/2813908812 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3166985571 0 0) 0x7ff088003570 con 0x7ff09c0685c0 2024-01-11T08:22:23.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.883+0000 7ff0a179b700 1 -- 172.21.15.27:0/2813908812 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09c0cf200 con 0x7ff09c15a1b0 2024-01-11T08:22:23.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.883+0000 7ff0a179b700 1 --2- 172.21.15.27:0/2813908812 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff09c15a1b0 0x7ff09c15a580 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff0980098e0 tx=0x7ff0980091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=103486f2903d6bdf server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:23.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.884+0000 7ff093fff700 1 -- 172.21.15.27:0/2813908812 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0940017f0 con 0x7ff09c15a1b0 2024-01-11T08:22:23.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.884+0000 7ff093fff700 1 -- 172.21.15.27:0/2813908812 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff098011420 con 0x7ff09c15a1b0 2024-01-11T08:22:23.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.884+0000 7ff093fff700 1 -- 172.21.15.27:0/2813908812 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0980117c0 con 0x7ff09c15a1b0 2024-01-11T08:22:23.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.884+0000 7ff0a39ff700 1 -- 172.21.15.27:0/2813908812 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff09c15a1b0 msgr2=0x7ff09c15a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:23.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.884+0000 7ff0a39ff700 1 --2- 172.21.15.27:0/2813908812 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff09c15a1b0 0x7ff09c15a580 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff0980098e0 tx=0x7ff0980091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.884+0000 7ff0a39ff700 1 -- 172.21.15.27:0/2813908812 shutdown_connections 2024-01-11T08:22:23.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.884+0000 7ff0a39ff700 1 --2- 172.21.15.27:0/2813908812 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff09c15a1b0 0x7ff09c15a580 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.884+0000 7ff0a39ff700 1 --2- 172.21.15.27:0/2813908812 >> v2:172.21.15.77:3300/0 conn(0x7ff09c0ccd70 0x7ff09c0cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.884+0000 7ff0a39ff700 1 -- 172.21.15.27:0/2813908812 >> 172.21.15.27:0/2813908812 conn(0x7ff09c06ed40 msgr2=0x7ff09c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:23.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.885+0000 7ff0a39ff700 1 -- 172.21.15.27:0/2813908812 shutdown_connections 2024-01-11T08:22:23.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.885+0000 7ff0a39ff700 1 -- 172.21.15.27:0/2813908812 wait complete. 2024-01-11T08:22:23.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.886+0000 7ff0a39ff700 1 Processor -- start 2024-01-11T08:22:23.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.886+0000 7ff0a39ff700 1 -- start start 2024-01-11T08:22:23.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.887+0000 7ff0a39ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff09c0685c0 0x7ff09c14aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:23.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.887+0000 7ff0a39ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff09c0ccd70 0x7ff09c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.887+0000 7ff0a39ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff09c15a1b0 0x7ff09c0a6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.887+0000 7ff0a39ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c0c87d0 con 0x7ff09c0ccd70 2024-01-11T08:22:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.887+0000 7ff0a39ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c0c8ab0 con 0x7ff09c0685c0 2024-01-11T08:22:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.887+0000 7ff0a39ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c0c8d90 con 0x7ff09c15a1b0 2024-01-11T08:22:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.887+0000 7ff0a0f9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff09c0ccd70 0x7ff09c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.887+0000 7ff0a0f9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff09c0ccd70 0x7ff09c15b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51736/0 (socket says 172.21.15.27:51736) 2024-01-11T08:22:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.887+0000 7ff0a0f9a700 1 -- 172.21.15.27:0/2163573760 learned_addr learned my addr 172.21.15.27:0/2163573760 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.887+0000 7ff0a1f9c700 1 --2- 172.21.15.27:0/2163573760 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff09c15a1b0 0x7ff09c0a6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.887+0000 7ff0a179b700 1 --2- 172.21.15.27:0/2163573760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff09c0685c0 0x7ff09c14aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.888+0000 7ff0a0f9a700 1 -- 172.21.15.27:0/2163573760 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff09c15a1b0 msgr2=0x7ff09c0a6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.888+0000 7ff0a0f9a700 1 --2- 172.21.15.27:0/2163573760 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff09c15a1b0 0x7ff09c0a6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.888+0000 7ff0a0f9a700 1 -- 172.21.15.27:0/2163573760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff09c0685c0 msgr2=0x7ff09c14aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.888+0000 7ff0a0f9a700 1 --2- 172.21.15.27:0/2163573760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff09c0685c0 0x7ff09c14aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.888+0000 7ff0a0f9a700 1 -- 172.21.15.27:0/2163573760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff098009420 con 0x7ff09c0ccd70 2024-01-11T08:22:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.888+0000 7ff0a179b700 1 --2- 172.21.15.27:0/2163573760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff09c0685c0 0x7ff09c14aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:22:23.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.888+0000 7ff0a0f9a700 1 --2- 172.21.15.27:0/2163573760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff09c0ccd70 0x7ff09c15b430 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff08c00ec80 tx=0x7ff08c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:23.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.888+0000 7ff0917fa700 1 -- 172.21.15.27:0/2163573760 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c00c850 con 0x7ff09c0ccd70 2024-01-11T08:22:23.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.888+0000 7ff0917fa700 1 -- 172.21.15.27:0/2163573760 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08c00c9b0 con 0x7ff09c0ccd70 2024-01-11T08:22:23.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.888+0000 7ff0917fa700 1 -- 172.21.15.27:0/2163573760 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c00cd50 con 0x7ff09c0ccd70 2024-01-11T08:22:23.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.888+0000 7ff0a39ff700 1 -- 172.21.15.27:0/2163573760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff09c0c9070 con 0x7ff09c0ccd70 2024-01-11T08:22:23.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.889+0000 7ff0a39ff700 1 -- 172.21.15.27:0/2163573760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff09c157b20 con 0x7ff09c0ccd70 2024-01-11T08:22:23.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.890+0000 7ff0a39ff700 1 -- 172.21.15.27:0/2163573760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff09c04c3c0 con 0x7ff09c0ccd70 2024-01-11T08:22:23.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.891+0000 7ff0917fa700 1 -- 172.21.15.27:0/2163573760 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7ff08c013680 con 0x7ff09c0ccd70 2024-01-11T08:22:23.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.892+0000 7ff0917fa700 1 --2- 172.21.15.27:0/2163573760 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff080067130 0x7ff0800695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:23.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.892+0000 7ff0917fa700 1 -- 172.21.15.27:0/2163573760 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7ff08c084d20 con 0x7ff09c0ccd70 2024-01-11T08:22:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.893+0000 7ff0a179b700 1 --2- 172.21.15.27:0/2163573760 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff080067130 0x7ff0800695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.893+0000 7ff0a179b700 1 --2- 172.21.15.27:0/2163573760 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff080067130 0x7ff0800695e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff0980030c0 tx=0x7ff098016000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:23.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:23.896+0000 7ff0917fa700 1 -- 172.21.15.27:0/2163573760 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff08c051b70 con 0x7ff09c0ccd70 2024-01-11T08:22:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.124+0000 7ff0a39ff700 1 -- 172.21.15.27:0/2163573760 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff09c0cd750 con 0x7ff080067130 2024-01-11T08:22:24.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.197+0000 7ff0917fa700 1 -- 172.21.15.27:0/2163573760 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7ff09c0cd750 con 0x7ff080067130 2024-01-11T08:22:24.196 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2024-01-11T08:22:24.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.199+0000 7ff0a39ff700 1 -- 172.21.15.27:0/2163573760 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff080067130 msgr2=0x7ff0800695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:24.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.199+0000 7ff0a39ff700 1 --2- 172.21.15.27:0/2163573760 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff080067130 0x7ff0800695e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff0980030c0 tx=0x7ff098016000 comp rx=0 tx=0).stop 2024-01-11T08:22:24.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.199+0000 7ff0a39ff700 1 -- 172.21.15.27:0/2163573760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff09c0ccd70 msgr2=0x7ff09c15b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:24.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.199+0000 7ff0a39ff700 1 --2- 172.21.15.27:0/2163573760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff09c0ccd70 0x7ff09c15b430 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff08c00ec80 tx=0x7ff08c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:24.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.199+0000 7ff0a39ff700 1 -- 172.21.15.27:0/2163573760 shutdown_connections 2024-01-11T08:22:24.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.199+0000 7ff0a39ff700 1 --2- 172.21.15.27:0/2163573760 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ff080067130 0x7ff0800695e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:24.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.199+0000 7ff0a39ff700 1 --2- 172.21.15.27:0/2163573760 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff09c15a1b0 0x7ff09c0a6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:24.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.199+0000 7ff0a39ff700 1 --2- 172.21.15.27:0/2163573760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff09c0685c0 0x7ff09c14aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:24.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.199+0000 7ff0a39ff700 1 --2- 172.21.15.27:0/2163573760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff09c0ccd70 0x7ff09c15b430 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:24.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.199+0000 7ff0a39ff700 1 -- 172.21.15.27:0/2163573760 >> 172.21.15.27:0/2163573760 conn(0x7ff09c06ed40 msgr2=0x7ff09c0c5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:24.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.200+0000 7ff0a39ff700 1 -- 172.21.15.27:0/2163573760 shutdown_connections 2024-01-11T08:22:24.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.200+0000 7ff0a39ff700 1 -- 172.21.15.27:0/2163573760 wait complete. 2024-01-11T08:22:24.216 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2024-01-11T08:22:24.217 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir7 1m 2024-01-11T08:22:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.420+0000 7f714f3ff700 1 Processor -- start 2024-01-11T08:22:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.420+0000 7f714f3ff700 1 -- start start 2024-01-11T08:22:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.421+0000 7f714f3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71481577e0 0x7f71480ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.421+0000 7f714f3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71480699d0 con 0x7f7148069540 2024-01-11T08:22:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.421+0000 7f714f3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7148069cb0 con 0x7f7148157410 2024-01-11T08:22:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.421+0000 7f714f3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7148069f90 con 0x7f71481577e0 2024-01-11T08:22:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.421+0000 7f714c99a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71481577e0 0x7f71480ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.422+0000 7f714d99c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7148069540 0x7f71480c6ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54388/0 (socket says 172.21.15.27:54388) 2024-01-11T08:22:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.422+0000 7f714d99c700 1 -- 172.21.15.27:0/528480068 learned_addr learned my addr 172.21.15.27:0/528480068 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:24.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.422+0000 7f714c99a700 1 -- 172.21.15.27:0/528480068 >> v1:172.21.15.77:6789/0 conn(0x7f7148157410 legacy=0x7f71480ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:24.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.422+0000 7f714c99a700 1 -- 172.21.15.27:0/528480068 >> v1:172.21.15.27:6789/0 conn(0x7f7148069540 legacy=0x7f71480c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:24.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.422+0000 7f714c99a700 1 -- 172.21.15.27:0/528480068 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714806a270 con 0x7f71481577e0 2024-01-11T08:22:24.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.422+0000 7f713ffff700 1 -- 172.21.15.27:0/528480068 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765343185 0 0) 0x7f71480699d0 con 0x7f7148069540 2024-01-11T08:22:24.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.422+0000 7f714c99a700 1 --2- 172.21.15.27:0/528480068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71481577e0 0x7f71480ae000 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f71380098e0 tx=0x7f71380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d546fc66fa042d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.423+0000 7f713ffff700 1 -- 172.21.15.27:0/528480068 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7138009e80 con 0x7f71481577e0 2024-01-11T08:22:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.423+0000 7f713ffff700 1 -- 172.21.15.27:0/528480068 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713800b860 con 0x7f71481577e0 2024-01-11T08:22:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.423+0000 7f713ffff700 1 -- 172.21.15.27:0/528480068 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713800bbe0 con 0x7f71481577e0 2024-01-11T08:22:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.423+0000 7f714f3ff700 1 -- 172.21.15.27:0/528480068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71481577e0 msgr2=0x7f71480ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.423+0000 7f714f3ff700 1 --2- 172.21.15.27:0/528480068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71481577e0 0x7f71480ae000 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f71380098e0 tx=0x7f71380091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.423+0000 7f714f3ff700 1 -- 172.21.15.27:0/528480068 shutdown_connections 2024-01-11T08:22:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.423+0000 7f714f3ff700 1 --2- 172.21.15.27:0/528480068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71481577e0 0x7f71480ae000 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.423+0000 7f714f3ff700 1 -- 172.21.15.27:0/528480068 >> 172.21.15.27:0/528480068 conn(0x7f71480680a0 msgr2=0x7f71480cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.423+0000 7f714f3ff700 1 -- 172.21.15.27:0/528480068 shutdown_connections 2024-01-11T08:22:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.423+0000 7f714f3ff700 1 -- 172.21.15.27:0/528480068 wait complete. 2024-01-11T08:22:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.425+0000 7f714f3ff700 1 Processor -- start 2024-01-11T08:22:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.425+0000 7f714f3ff700 1 -- start start 2024-01-11T08:22:24.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.426+0000 7f714f3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7148069540 0x7f71480c4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:24.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.426+0000 7f714f3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7148157410 0x7f71480ae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:24.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.426+0000 7f714f3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f71481577e0 0x7f7148154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:24.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.426+0000 7f714f3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71480ae6b0 con 0x7f7148157410 2024-01-11T08:22:24.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.426+0000 7f714f3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71480c4870 con 0x7f7148069540 2024-01-11T08:22:24.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.426+0000 7f714f3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71480c4b50 con 0x7f71481577e0 2024-01-11T08:22:24.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.426+0000 7f714c99a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7148157410 0x7f71480ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:24.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.426+0000 7f714c99a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7148157410 0x7f71480ae170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51748/0 (socket says 172.21.15.27:51748) 2024-01-11T08:22:24.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.426+0000 7f714c99a700 1 -- 172.21.15.27:0/3218519315 learned_addr learned my addr 172.21.15.27:0/3218519315 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:24.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.426+0000 7f714d99c700 1 --2- 172.21.15.27:0/3218519315 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f71481577e0 0x7f7148154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:24.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.426+0000 7f714d19b700 1 --2- 172.21.15.27:0/3218519315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7148069540 0x7f71480c4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:24.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.426+0000 7f714c99a700 1 -- 172.21.15.27:0/3218519315 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f71481577e0 msgr2=0x7f7148154350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:24.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.427+0000 7f714c99a700 1 --2- 172.21.15.27:0/3218519315 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f71481577e0 0x7f7148154350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:24.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.427+0000 7f714c99a700 1 -- 172.21.15.27:0/3218519315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7148069540 msgr2=0x7f71480c4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:24.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.427+0000 7f714c99a700 1 --2- 172.21.15.27:0/3218519315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7148069540 0x7f71480c4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:24.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.427+0000 7f714c99a700 1 -- 172.21.15.27:0/3218519315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7138009420 con 0x7f7148157410 2024-01-11T08:22:24.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.427+0000 7f714c99a700 1 --2- 172.21.15.27:0/3218519315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7148157410 0x7f71480ae170 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7138013000 tx=0x7f7138017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:24.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.427+0000 7f713d7fa700 1 -- 172.21.15.27:0/3218519315 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713800bbe0 con 0x7f7148157410 2024-01-11T08:22:24.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.427+0000 7f713d7fa700 1 -- 172.21.15.27:0/3218519315 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7138011420 con 0x7f7148157410 2024-01-11T08:22:24.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.427+0000 7f713d7fa700 1 -- 172.21.15.27:0/3218519315 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71380117c0 con 0x7f7148157410 2024-01-11T08:22:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.427+0000 7f714f3ff700 1 -- 172.21.15.27:0/3218519315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71480c4e30 con 0x7f7148157410 2024-01-11T08:22:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.427+0000 7f714f3ff700 1 -- 172.21.15.27:0/3218519315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7148159e70 con 0x7f7148157410 2024-01-11T08:22:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.429+0000 7f714f3ff700 1 -- 172.21.15.27:0/3218519315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714815a060 con 0x7f7148157410 2024-01-11T08:22:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.429+0000 7f713d7fa700 1 -- 172.21.15.27:0/3218519315 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f7138011b30 con 0x7f7148157410 2024-01-11T08:22:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.430+0000 7f713d7fa700 1 --2- 172.21.15.27:0/3218519315 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f712c067130 0x7f712c0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.430+0000 7f713d7fa700 1 -- 172.21.15.27:0/3218519315 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f7138086250 con 0x7f7148157410 2024-01-11T08:22:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.431+0000 7f714d19b700 1 --2- 172.21.15.27:0/3218519315 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f712c067130 0x7f712c0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.431+0000 7f714d19b700 1 --2- 172.21.15.27:0/3218519315 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f712c067130 0x7f712c0695e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7144007fd0 tx=0x7f7144007eb0 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:24.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.434+0000 7f713d7fa700 1 -- 172.21.15.27:0/3218519315 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f71380531c0 con 0x7f7148157410 2024-01-11T08:22:24.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.668+0000 7f714f3ff700 1 -- 172.21.15.27:0/3218519315 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f714804c3c0 con 0x7f712c067130 2024-01-11T08:22:24.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.741+0000 7f713d7fa700 1 -- 172.21.15.27:0/3218519315 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f714804c3c0 con 0x7f712c067130 2024-01-11T08:22:24.740 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2024-01-11T08:22:24.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.743+0000 7f714f3ff700 1 -- 172.21.15.27:0/3218519315 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f712c067130 msgr2=0x7f712c0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:24.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.743+0000 7f714f3ff700 1 --2- 172.21.15.27:0/3218519315 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f712c067130 0x7f712c0695e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7144007fd0 tx=0x7f7144007eb0 comp rx=0 tx=0).stop 2024-01-11T08:22:24.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.743+0000 7f714f3ff700 1 -- 172.21.15.27:0/3218519315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7148157410 msgr2=0x7f71480ae170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:24.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.743+0000 7f714f3ff700 1 --2- 172.21.15.27:0/3218519315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7148157410 0x7f71480ae170 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7138013000 tx=0x7f7138017000 comp rx=0 tx=0).stop 2024-01-11T08:22:24.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.743+0000 7f714f3ff700 1 -- 172.21.15.27:0/3218519315 shutdown_connections 2024-01-11T08:22:24.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.743+0000 7f714f3ff700 1 --2- 172.21.15.27:0/3218519315 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f712c067130 0x7f712c0695e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:24.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.743+0000 7f714f3ff700 1 --2- 172.21.15.27:0/3218519315 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f71481577e0 0x7f7148154350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:24.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.743+0000 7f714f3ff700 1 --2- 172.21.15.27:0/3218519315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7148069540 0x7f71480c4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:24.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.743+0000 7f714f3ff700 1 --2- 172.21.15.27:0/3218519315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7148157410 0x7f71480ae170 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:24.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.743+0000 7f714f3ff700 1 -- 172.21.15.27:0/3218519315 >> 172.21.15.27:0/3218519315 conn(0x7f71480680a0 msgr2=0x7f71480c8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.744+0000 7f714f3ff700 1 -- 172.21.15.27:0/3218519315 shutdown_connections 2024-01-11T08:22:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.744+0000 7f714f3ff700 1 -- 172.21.15.27:0/3218519315 wait complete. 2024-01-11T08:22:24.761 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2024-01-11T08:22:24.761 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir8 1m 2024-01-11T08:22:24.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.965+0000 7fcb3b3ff700 1 Processor -- start 2024-01-11T08:22:24.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.965+0000 7fcb3b3ff700 1 -- start start 2024-01-11T08:22:24.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.967+0000 7fcb3b3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb340ccd70 0x7fcb340cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:24.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.967+0000 7fcb3b3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcb3415ae70 0x7fcb3415b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:24.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.967+0000 7fcb3b3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb340cf850 con 0x7fcb3415a1b0 2024-01-11T08:22:24.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.967+0000 7fcb3b3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb3415ac90 con 0x7fcb3415ae70 2024-01-11T08:22:24.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.967+0000 7fcb3b3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb340cec40 con 0x7fcb340ccd70 2024-01-11T08:22:24.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.967+0000 7fcb3919b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcb3415a1b0 0x7fcb3415a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54402/0 (socket says 172.21.15.27:54402) 2024-01-11T08:22:24.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.967+0000 7fcb3919b700 1 -- 172.21.15.27:0/3870560285 learned_addr learned my addr 172.21.15.27:0/3870560285 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:24.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.967+0000 7fcb3999c700 1 --2- 172.21.15.27:0/3870560285 >> v2:172.21.15.27:3300/0 conn(0x7fcb3415ae70 0x7fcb3415b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:24.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.967+0000 7fcb3899a700 1 --2- 172.21.15.27:0/3870560285 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb340ccd70 0x7fcb340cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:24.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.967+0000 7fcb2bfff700 1 -- 172.21.15.27:0/3870560285 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196361890 0 0) 0x7fcb340cf850 con 0x7fcb3415a1b0 2024-01-11T08:22:24.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.968+0000 7fcb2bfff700 1 -- 172.21.15.27:0/3870560285 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb20003570 con 0x7fcb3415a1b0 2024-01-11T08:22:24.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.968+0000 7fcb3999c700 1 -- 172.21.15.27:0/3870560285 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb340ccd70 msgr2=0x7fcb340cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:24.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.968+0000 7fcb3999c700 1 --2- 172.21.15.27:0/3870560285 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb340ccd70 0x7fcb340cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:24.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.968+0000 7fcb3999c700 1 -- 172.21.15.27:0/3870560285 >> v1:172.21.15.27:6789/0 conn(0x7fcb3415a1b0 legacy=0x7fcb3415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:24.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.968+0000 7fcb3999c700 1 -- 172.21.15.27:0/3870560285 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb340cef20 con 0x7fcb3415ae70 2024-01-11T08:22:24.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.968+0000 7fcb3999c700 1 --2- 172.21.15.27:0/3870560285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb3415ae70 0x7fcb3415b2f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c009e60 tx=0x7fcb2c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678c2e0f2a18afcb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:24.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.968+0000 7fcb2bfff700 1 -- 172.21.15.27:0/3870560285 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c00e070 con 0x7fcb3415ae70 2024-01-11T08:22:24.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.968+0000 7fcb2bfff700 1 -- 172.21.15.27:0/3870560285 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2c00bab0 con 0x7fcb3415ae70 2024-01-11T08:22:24.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.968+0000 7fcb2bfff700 1 -- 172.21.15.27:0/3870560285 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c00be70 con 0x7fcb3415ae70 2024-01-11T08:22:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.969+0000 7fcb3b3ff700 1 -- 172.21.15.27:0/3870560285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb3415ae70 msgr2=0x7fcb3415b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.969+0000 7fcb3b3ff700 1 --2- 172.21.15.27:0/3870560285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb3415ae70 0x7fcb3415b2f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c009e60 tx=0x7fcb2c009780 comp rx=0 tx=0).stop 2024-01-11T08:22:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.969+0000 7fcb3b3ff700 1 -- 172.21.15.27:0/3870560285 shutdown_connections 2024-01-11T08:22:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.969+0000 7fcb3b3ff700 1 --2- 172.21.15.27:0/3870560285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb3415ae70 0x7fcb3415b2f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.969+0000 7fcb3b3ff700 1 --2- 172.21.15.27:0/3870560285 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb340ccd70 0x7fcb340cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.969+0000 7fcb3b3ff700 1 -- 172.21.15.27:0/3870560285 >> 172.21.15.27:0/3870560285 conn(0x7fcb3406ed40 msgr2=0x7fcb3406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.969+0000 7fcb3b3ff700 1 -- 172.21.15.27:0/3870560285 shutdown_connections 2024-01-11T08:22:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.969+0000 7fcb3b3ff700 1 -- 172.21.15.27:0/3870560285 wait complete. 2024-01-11T08:22:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.970+0000 7fcb3b3ff700 1 Processor -- start 2024-01-11T08:22:24.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.971+0000 7fcb3b3ff700 1 -- start start 2024-01-11T08:22:24.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.971+0000 7fcb3b3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb340ccd70 0x7fcb34148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:24.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.972+0000 7fcb3b3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb3415a1b0 0x7fcb3415da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:24.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.972+0000 7fcb3b3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb3415ae70 0x7fcb3415df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:24.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.972+0000 7fcb3b3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb341485c0 con 0x7fcb3415a1b0 2024-01-11T08:22:24.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.972+0000 7fcb3b3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb340b4330 con 0x7fcb340ccd70 2024-01-11T08:22:24.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.972+0000 7fcb3b3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb340b2670 con 0x7fcb3415ae70 2024-01-11T08:22:24.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.972+0000 7fcb3899a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb3415a1b0 0x7fcb3415da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:24.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.972+0000 7fcb3899a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb3415a1b0 0x7fcb3415da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51778/0 (socket says 172.21.15.27:51778) 2024-01-11T08:22:24.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.972+0000 7fcb3899a700 1 -- 172.21.15.27:0/2577631814 learned_addr learned my addr 172.21.15.27:0/2577631814 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:24.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.972+0000 7fcb3919b700 1 --2- 172.21.15.27:0/2577631814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb340ccd70 0x7fcb34148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:24.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.972+0000 7fcb3999c700 1 --2- 172.21.15.27:0/2577631814 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb3415ae70 0x7fcb3415df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:24.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.972+0000 7fcb3899a700 1 -- 172.21.15.27:0/2577631814 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb3415ae70 msgr2=0x7fcb3415df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:24.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.972+0000 7fcb3899a700 1 --2- 172.21.15.27:0/2577631814 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb3415ae70 0x7fcb3415df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:24.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.972+0000 7fcb3899a700 1 -- 172.21.15.27:0/2577631814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb340ccd70 msgr2=0x7fcb34148080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:24.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.972+0000 7fcb3899a700 1 --2- 172.21.15.27:0/2577631814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb340ccd70 0x7fcb34148080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:24.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.972+0000 7fcb3899a700 1 -- 172.21.15.27:0/2577631814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2c009a20 con 0x7fcb3415a1b0 2024-01-11T08:22:24.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.972+0000 7fcb3899a700 1 --2- 172.21.15.27:0/2577631814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb3415a1b0 0x7fcb3415da20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fcb2400ec80 tx=0x7fcb240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:24.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.973+0000 7fcb297fa700 1 -- 172.21.15.27:0/2577631814 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2400c9d0 con 0x7fcb3415a1b0 2024-01-11T08:22:24.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.973+0000 7fcb297fa700 1 -- 172.21.15.27:0/2577631814 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2400cb30 con 0x7fcb3415a1b0 2024-01-11T08:22:24.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.973+0000 7fcb3b3ff700 1 -- 172.21.15.27:0/2577631814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb340b45d0 con 0x7fcb3415a1b0 2024-01-11T08:22:24.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.973+0000 7fcb297fa700 1 -- 172.21.15.27:0/2577631814 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb24013470 con 0x7fcb3415a1b0 2024-01-11T08:22:24.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.973+0000 7fcb3b3ff700 1 -- 172.21.15.27:0/2577631814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb340b2950 con 0x7fcb3415a1b0 2024-01-11T08:22:24.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.974+0000 7fcb3b3ff700 1 -- 172.21.15.27:0/2577631814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb340cfe50 con 0x7fcb3415a1b0 2024-01-11T08:22:24.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.975+0000 7fcb297fa700 1 -- 172.21.15.27:0/2577631814 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fcb24013820 con 0x7fcb3415a1b0 2024-01-11T08:22:24.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.976+0000 7fcb297fa700 1 --2- 172.21.15.27:0/2577631814 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcb18067130 0x7fcb180695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:24.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.976+0000 7fcb297fa700 1 -- 172.21.15.27:0/2577631814 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fcb24084dd0 con 0x7fcb3415a1b0 2024-01-11T08:22:24.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.976+0000 7fcb3919b700 1 --2- 172.21.15.27:0/2577631814 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcb18067130 0x7fcb180695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:24.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.977+0000 7fcb3919b700 1 --2- 172.21.15.27:0/2577631814 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcb18067130 0x7fcb180695e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcb300058d0 tx=0x7fcb30009040 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:24.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:24.980+0000 7fcb297fa700 1 -- 172.21.15.27:0/2577631814 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcb24051c20 con 0x7fcb3415a1b0 2024-01-11T08:22:25.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.211+0000 7fcb3b3ff700 1 -- 172.21.15.27:0/2577631814 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb3404c3c0 con 0x7fcb18067130 2024-01-11T08:22:25.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.288+0000 7fcb297fa700 1 -- 172.21.15.27:0/2577631814 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fcb3404c3c0 con 0x7fcb18067130 2024-01-11T08:22:25.288 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2024-01-11T08:22:25.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.290+0000 7fcb3b3ff700 1 -- 172.21.15.27:0/2577631814 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcb18067130 msgr2=0x7fcb180695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:25.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.290+0000 7fcb3b3ff700 1 --2- 172.21.15.27:0/2577631814 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcb18067130 0x7fcb180695e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcb300058d0 tx=0x7fcb30009040 comp rx=0 tx=0).stop 2024-01-11T08:22:25.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.290+0000 7fcb3b3ff700 1 -- 172.21.15.27:0/2577631814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb3415a1b0 msgr2=0x7fcb3415da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:25.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.290+0000 7fcb3b3ff700 1 --2- 172.21.15.27:0/2577631814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb3415a1b0 0x7fcb3415da20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fcb2400ec80 tx=0x7fcb240091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:25.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.291+0000 7fcb3b3ff700 1 -- 172.21.15.27:0/2577631814 shutdown_connections 2024-01-11T08:22:25.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.291+0000 7fcb3b3ff700 1 --2- 172.21.15.27:0/2577631814 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcb18067130 0x7fcb180695e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:25.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.291+0000 7fcb3b3ff700 1 --2- 172.21.15.27:0/2577631814 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb3415ae70 0x7fcb3415df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:25.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.291+0000 7fcb3b3ff700 1 --2- 172.21.15.27:0/2577631814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb340ccd70 0x7fcb34148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:25.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.291+0000 7fcb3b3ff700 1 --2- 172.21.15.27:0/2577631814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb3415a1b0 0x7fcb3415da20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:25.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.291+0000 7fcb3b3ff700 1 -- 172.21.15.27:0/2577631814 >> 172.21.15.27:0/2577631814 conn(0x7fcb3406ed40 msgr2=0x7fcb34155e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:25.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.291+0000 7fcb3b3ff700 1 -- 172.21.15.27:0/2577631814 shutdown_connections 2024-01-11T08:22:25.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.291+0000 7fcb3b3ff700 1 -- 172.21.15.27:0/2577631814 wait complete. 2024-01-11T08:22:25.308 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2024-01-11T08:22:25.308 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir9 1m 2024-01-11T08:22:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.512+0000 7fba6a7ff700 1 Processor -- start 2024-01-11T08:22:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.513+0000 7fba6a7ff700 1 -- start start 2024-01-11T08:22:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.514+0000 7fba6a7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fba64069bd0 0x7fba640aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.514+0000 7fba6a7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fba640c6de0 0x7fba640c7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.514+0000 7fba6a7ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6406e480 con 0x7fba64155c50 2024-01-11T08:22:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.514+0000 7fba6a7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba640cb5a0 con 0x7fba640c6de0 2024-01-11T08:22:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.514+0000 7fba6a7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6406df00 con 0x7fba64069bd0 2024-01-11T08:22:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.515+0000 7fba637fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fba64069bd0 0x7fba640aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.515+0000 7fba68d9c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fba640c6de0 0x7fba640c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.515+0000 7fba63fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fba64155c50 0x7fba640694c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:34032/0 (socket says 172.21.15.27:34032) 2024-01-11T08:22:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.515+0000 7fba63fff700 1 -- 172.21.15.27:0/3677352770 learned_addr learned my addr 172.21.15.27:0/3677352770 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.515+0000 7fba637fe700 1 -- 172.21.15.27:0/3677352770 >> v2:172.21.15.77:3300/0 conn(0x7fba640c6de0 msgr2=0x7fba640c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.515+0000 7fba637fe700 1 --2- 172.21.15.27:0/3677352770 >> v2:172.21.15.77:3300/0 conn(0x7fba640c6de0 0x7fba640c7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.515+0000 7fba62ffd700 1 -- 172.21.15.27:0/3677352770 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103577278 0 0) 0x7fba6406e480 con 0x7fba64155c50 2024-01-11T08:22:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.515+0000 7fba637fe700 1 -- 172.21.15.27:0/3677352770 >> v1:172.21.15.77:6789/0 conn(0x7fba64155c50 legacy=0x7fba640694c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.515+0000 7fba637fe700 1 -- 172.21.15.27:0/3677352770 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba640c77c0 con 0x7fba64069bd0 2024-01-11T08:22:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.515+0000 7fba637fe700 1 --2- 172.21.15.27:0/3677352770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fba64069bd0 0x7fba640aef80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fba5800bb90 tx=0x7fba5800b430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2cbadfce8ec8d51 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.516+0000 7fba62ffd700 1 -- 172.21.15.27:0/3677352770 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5800dbd0 con 0x7fba64069bd0 2024-01-11T08:22:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.516+0000 7fba62ffd700 1 -- 172.21.15.27:0/3677352770 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5800dd30 con 0x7fba64069bd0 2024-01-11T08:22:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.516+0000 7fba62ffd700 1 -- 172.21.15.27:0/3677352770 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5800dea0 con 0x7fba64069bd0 2024-01-11T08:22:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.516+0000 7fba6a7ff700 1 -- 172.21.15.27:0/3677352770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fba64069bd0 msgr2=0x7fba640aef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.516+0000 7fba6a7ff700 1 --2- 172.21.15.27:0/3677352770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fba64069bd0 0x7fba640aef80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fba5800bb90 tx=0x7fba5800b430 comp rx=0 tx=0).stop 2024-01-11T08:22:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.516+0000 7fba6a7ff700 1 -- 172.21.15.27:0/3677352770 shutdown_connections 2024-01-11T08:22:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.516+0000 7fba6a7ff700 1 --2- 172.21.15.27:0/3677352770 >> v2:172.21.15.77:3300/0 conn(0x7fba640c6de0 0x7fba640c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.516+0000 7fba6a7ff700 1 --2- 172.21.15.27:0/3677352770 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fba64069bd0 0x7fba640aef80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.516+0000 7fba6a7ff700 1 -- 172.21.15.27:0/3677352770 >> 172.21.15.27:0/3677352770 conn(0x7fba640680a0 msgr2=0x7fba640cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.516+0000 7fba6a7ff700 1 -- 172.21.15.27:0/3677352770 shutdown_connections 2024-01-11T08:22:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.516+0000 7fba6a7ff700 1 -- 172.21.15.27:0/3677352770 wait complete. 2024-01-11T08:22:25.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.519+0000 7fba6a7ff700 1 Processor -- start 2024-01-11T08:22:25.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.519+0000 7fba6a7ff700 1 -- start start 2024-01-11T08:22:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.520+0000 7fba6a7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba64069bd0 0x7fba64158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.520+0000 7fba6a7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba640c6de0 0x7fba6416d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.520+0000 7fba6a7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fba64155c50 0x7fba6416d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.520+0000 7fba6a7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6416ace0 con 0x7fba640c6de0 2024-01-11T08:22:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.520+0000 7fba6a7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba640c5150 con 0x7fba64069bd0 2024-01-11T08:22:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.520+0000 7fba6a7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba640c5430 con 0x7fba64155c50 2024-01-11T08:22:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.520+0000 7fba637fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba640c6de0 0x7fba6416d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.520+0000 7fba637fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba640c6de0 0x7fba6416d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51794/0 (socket says 172.21.15.27:51794) 2024-01-11T08:22:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.520+0000 7fba637fe700 1 -- 172.21.15.27:0/1241732238 learned_addr learned my addr 172.21.15.27:0/1241732238 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.521+0000 7fba68d9c700 1 --2- 172.21.15.27:0/1241732238 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fba64155c50 0x7fba6416d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.521+0000 7fba63fff700 1 --2- 172.21.15.27:0/1241732238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba64069bd0 0x7fba64158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.521+0000 7fba637fe700 1 -- 172.21.15.27:0/1241732238 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fba64155c50 msgr2=0x7fba6416d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.521+0000 7fba637fe700 1 --2- 172.21.15.27:0/1241732238 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fba64155c50 0x7fba6416d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:25.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.521+0000 7fba637fe700 1 -- 172.21.15.27:0/1241732238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba64069bd0 msgr2=0x7fba64158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.521+0000 7fba637fe700 1 --2- 172.21.15.27:0/1241732238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba64069bd0 0x7fba64158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.521+0000 7fba637fe700 1 -- 172.21.15.27:0/1241732238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5800b6d0 con 0x7fba640c6de0 2024-01-11T08:22:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.521+0000 7fba637fe700 1 --2- 172.21.15.27:0/1241732238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba640c6de0 0x7fba6416d110 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fba5800b2c0 tx=0x7fba5801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.521+0000 7fba4bfff700 1 -- 172.21.15.27:0/1241732238 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba580106c0 con 0x7fba640c6de0 2024-01-11T08:22:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.521+0000 7fba4bfff700 1 -- 172.21.15.27:0/1241732238 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba58010820 con 0x7fba640c6de0 2024-01-11T08:22:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.521+0000 7fba6a7ff700 1 -- 172.21.15.27:0/1241732238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba640c5710 con 0x7fba640c6de0 2024-01-11T08:22:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.521+0000 7fba4bfff700 1 -- 172.21.15.27:0/1241732238 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58010be0 con 0x7fba640c6de0 2024-01-11T08:22:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.522+0000 7fba6a7ff700 1 -- 172.21.15.27:0/1241732238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba64159500 con 0x7fba640c6de0 2024-01-11T08:22:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.523+0000 7fba6a7ff700 1 -- 172.21.15.27:0/1241732238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba6416ae50 con 0x7fba640c6de0 2024-01-11T08:22:25.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.524+0000 7fba4bfff700 1 -- 172.21.15.27:0/1241732238 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fba5802d430 con 0x7fba640c6de0 2024-01-11T08:22:25.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.524+0000 7fba4bfff700 1 --2- 172.21.15.27:0/1241732238 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fba50067130 0x7fba500695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:25.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.524+0000 7fba4bfff700 1 -- 172.21.15.27:0/1241732238 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fba58020070 con 0x7fba640c6de0 2024-01-11T08:22:25.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.525+0000 7fba63fff700 1 --2- 172.21.15.27:0/1241732238 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fba50067130 0x7fba500695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:25.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.526+0000 7fba63fff700 1 --2- 172.21.15.27:0/1241732238 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fba50067130 0x7fba500695e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fba6416ac90 tx=0x7fba54009040 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:25.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.529+0000 7fba4bfff700 1 -- 172.21.15.27:0/1241732238 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fba580537d0 con 0x7fba640c6de0 2024-01-11T08:22:25.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.761+0000 7fba6a7ff700 1 -- 172.21.15.27:0/1241732238 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fba6404c3c0 con 0x7fba50067130 2024-01-11T08:22:25.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.834+0000 7fba4bfff700 1 -- 172.21.15.27:0/1241732238 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fba6404c3c0 con 0x7fba50067130 2024-01-11T08:22:25.833 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2024-01-11T08:22:25.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.836+0000 7fba6a7ff700 1 -- 172.21.15.27:0/1241732238 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fba50067130 msgr2=0x7fba500695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:25.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.836+0000 7fba6a7ff700 1 --2- 172.21.15.27:0/1241732238 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fba50067130 0x7fba500695e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fba6416ac90 tx=0x7fba54009040 comp rx=0 tx=0).stop 2024-01-11T08:22:25.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.836+0000 7fba6a7ff700 1 -- 172.21.15.27:0/1241732238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba640c6de0 msgr2=0x7fba6416d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:25.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.836+0000 7fba6a7ff700 1 --2- 172.21.15.27:0/1241732238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba640c6de0 0x7fba6416d110 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fba5800b2c0 tx=0x7fba5801b000 comp rx=0 tx=0).stop 2024-01-11T08:22:25.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.836+0000 7fba6a7ff700 1 -- 172.21.15.27:0/1241732238 shutdown_connections 2024-01-11T08:22:25.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.836+0000 7fba6a7ff700 1 --2- 172.21.15.27:0/1241732238 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fba50067130 0x7fba500695e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:25.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.836+0000 7fba6a7ff700 1 --2- 172.21.15.27:0/1241732238 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fba64155c50 0x7fba6416d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:25.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.837+0000 7fba6a7ff700 1 --2- 172.21.15.27:0/1241732238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba64069bd0 0x7fba64158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:25.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.837+0000 7fba6a7ff700 1 --2- 172.21.15.27:0/1241732238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba640c6de0 0x7fba6416d110 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:25.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.837+0000 7fba6a7ff700 1 -- 172.21.15.27:0/1241732238 >> 172.21.15.27:0/1241732238 conn(0x7fba640680a0 msgr2=0x7fba64154de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:25.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.837+0000 7fba6a7ff700 1 -- 172.21.15.27:0/1241732238 shutdown_connections 2024-01-11T08:22:25.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:25.837+0000 7fba6a7ff700 1 -- 172.21.15.27:0/1241732238 wait complete. 2024-01-11T08:22:25.853 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2024-01-11T08:22:25.854 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:25.854 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2024-01-11T08:22:25.905 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:25.906 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:25.906 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:25.906 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:25.906 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:25.906 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:25.906 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:25.906 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:25.906 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:25.906 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:25.906 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:25.907 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-11-08_12_00_UTC', 'scheduled-2024-01-11-08_13_00_UTC', 'scheduled-2024-01-11-08_14_00_UTC', 'scheduled-2024-01-11-08_15_00_UTC', 'scheduled-2024-01-11-08_16_00_UTC', 'scheduled-2024-01-11-08_17_00_UTC', 'scheduled-2024-01-11-08_18_00_UTC', 'scheduled-2024-01-11-08_19_00_UTC', 'scheduled-2024-01-11-08_20_00_UTC', 'scheduled-2024-01-11-08_21_00_UTC', 'scheduled-2024-01-11-08_22_00_UTC'] 2024-01-11T08:22:25.907 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir0 2024-01-11T08:22:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.109+0000 7fcd871ff700 1 Processor -- start 2024-01-11T08:22:26.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.109+0000 7fcd871ff700 1 -- start start 2024-01-11T08:22:26.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.111+0000 7fcd871ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcd80143190 0x7fcd800cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:26.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.111+0000 7fcd871ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd800cb4b0 0x7fcd800cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:26.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.111+0000 7fcd871ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8015d320 con 0x7fcd80142dc0 2024-01-11T08:22:26.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.111+0000 7fcd871ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd8015df30 con 0x7fcd80143190 2024-01-11T08:22:26.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.111+0000 7fcd871ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8015d590 con 0x7fcd800cb4b0 2024-01-11T08:22:26.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.111+0000 7fcd7ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcd80143190 0x7fcd800cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:26.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.111+0000 7fcd84f9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcd80142dc0 0x7fcd800cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54412/0 (socket says 172.21.15.27:54412) 2024-01-11T08:22:26.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.111+0000 7fcd84f9b700 1 -- 172.21.15.27:0/3920286781 learned_addr learned my addr 172.21.15.27:0/3920286781 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:26.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.111+0000 7fcd7ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcd80143190 0x7fcd800cc020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51802/0 (socket says 172.21.15.27:51802) 2024-01-11T08:22:26.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.111+0000 7fcd7ffff700 1 -- 172.21.15.27:0/3920286781 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd800cb4b0 msgr2=0x7fcd800cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.111+0000 7fcd7ffff700 1 --2- 172.21.15.27:0/3920286781 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd800cb4b0 0x7fcd800cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.111+0000 7fcd7ffff700 1 -- 172.21.15.27:0/3920286781 >> v1:172.21.15.27:6789/0 conn(0x7fcd80142dc0 legacy=0x7fcd800cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.111+0000 7fcd7ffff700 1 -- 172.21.15.27:0/3920286781 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8015d870 con 0x7fcd80143190 2024-01-11T08:22:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.111+0000 7fcd7f7fe700 1 -- 172.21.15.27:0/3920286781 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390757221 0 0) 0x7fcd8015d320 con 0x7fcd80142dc0 2024-01-11T08:22:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.112+0000 7fcd7ffff700 1 --2- 172.21.15.27:0/3920286781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd80143190 0x7fcd800cc020 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c0098e0 tx=0x7fcd6c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4df1901d24cbbd5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.112+0000 7fcd7f7fe700 1 -- 172.21.15.27:0/3920286781 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c009e80 con 0x7fcd80143190 2024-01-11T08:22:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.112+0000 7fcd7f7fe700 1 -- 172.21.15.27:0/3920286781 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd6c00b860 con 0x7fcd80143190 2024-01-11T08:22:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.112+0000 7fcd7f7fe700 1 -- 172.21.15.27:0/3920286781 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c00bc20 con 0x7fcd80143190 2024-01-11T08:22:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.112+0000 7fcd871ff700 1 -- 172.21.15.27:0/3920286781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd80143190 msgr2=0x7fcd800cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.112+0000 7fcd871ff700 1 --2- 172.21.15.27:0/3920286781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd80143190 0x7fcd800cc020 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c0098e0 tx=0x7fcd6c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.113+0000 7fcd871ff700 1 -- 172.21.15.27:0/3920286781 shutdown_connections 2024-01-11T08:22:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.113+0000 7fcd871ff700 1 --2- 172.21.15.27:0/3920286781 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd800cb4b0 0x7fcd800cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.113+0000 7fcd871ff700 1 --2- 172.21.15.27:0/3920286781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd80143190 0x7fcd800cc020 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.113+0000 7fcd871ff700 1 -- 172.21.15.27:0/3920286781 >> 172.21.15.27:0/3920286781 conn(0x7fcd80063ca0 msgr2=0x7fcd8006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.113+0000 7fcd871ff700 1 -- 172.21.15.27:0/3920286781 shutdown_connections 2024-01-11T08:22:26.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.113+0000 7fcd871ff700 1 -- 172.21.15.27:0/3920286781 wait complete. 2024-01-11T08:22:26.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.114+0000 7fcd871ff700 1 Processor -- start 2024-01-11T08:22:26.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.114+0000 7fcd871ff700 1 -- start start 2024-01-11T08:22:26.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.115+0000 7fcd871ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd800cb4b0 0x7fcd801411d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.115+0000 7fcd871ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd80142dc0 0x7fcd800c66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.115+0000 7fcd871ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd80143190 0x7fcd800b3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.115+0000 7fcd871ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd800b1180 con 0x7fcd80142dc0 2024-01-11T08:22:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.115+0000 7fcd871ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd800c5490 con 0x7fcd800cb4b0 2024-01-11T08:22:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.115+0000 7fcd871ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd800c62f0 con 0x7fcd80143190 2024-01-11T08:22:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.116+0000 7fcd7ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd80142dc0 0x7fcd800c66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.116+0000 7fcd7ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd80142dc0 0x7fcd800c66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51818/0 (socket says 172.21.15.27:51818) 2024-01-11T08:22:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.116+0000 7fcd7ffff700 1 -- 172.21.15.27:0/3621755432 learned_addr learned my addr 172.21.15.27:0/3621755432 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.116+0000 7fcd84f9b700 1 --2- 172.21.15.27:0/3621755432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd800cb4b0 0x7fcd801411d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.116+0000 7fcd8579c700 1 --2- 172.21.15.27:0/3621755432 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd80143190 0x7fcd800b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.116+0000 7fcd7ffff700 1 -- 172.21.15.27:0/3621755432 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd80143190 msgr2=0x7fcd800b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.116+0000 7fcd7ffff700 1 --2- 172.21.15.27:0/3621755432 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd80143190 0x7fcd800b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.116+0000 7fcd7ffff700 1 -- 172.21.15.27:0/3621755432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd800cb4b0 msgr2=0x7fcd801411d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.116+0000 7fcd7ffff700 1 --2- 172.21.15.27:0/3621755432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd800cb4b0 0x7fcd801411d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.116+0000 7fcd7ffff700 1 -- 172.21.15.27:0/3621755432 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6c009420 con 0x7fcd80142dc0 2024-01-11T08:22:26.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.116+0000 7fcd84f9b700 1 --2- 172.21.15.27:0/3621755432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd800cb4b0 0x7fcd801411d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:22:26.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.116+0000 7fcd7ffff700 1 --2- 172.21.15.27:0/3621755432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd80142dc0 0x7fcd800c66c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcd80064a70 tx=0x7fcd6c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:26.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.117+0000 7fcd7cff9700 1 -- 172.21.15.27:0/3621755432 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c01f070 con 0x7fcd80142dc0 2024-01-11T08:22:26.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.117+0000 7fcd7cff9700 1 -- 172.21.15.27:0/3621755432 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd6c00b9d0 con 0x7fcd80142dc0 2024-01-11T08:22:26.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.117+0000 7fcd871ff700 1 -- 172.21.15.27:0/3621755432 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd800b0380 con 0x7fcd80142dc0 2024-01-11T08:22:26.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.117+0000 7fcd7cff9700 1 -- 172.21.15.27:0/3621755432 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c025420 con 0x7fcd80142dc0 2024-01-11T08:22:26.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.117+0000 7fcd871ff700 1 -- 172.21.15.27:0/3621755432 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd800c59c0 con 0x7fcd80142dc0 2024-01-11T08:22:26.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.118+0000 7fcd871ff700 1 -- 172.21.15.27:0/3621755432 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd800c6480 con 0x7fcd80142dc0 2024-01-11T08:22:26.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.119+0000 7fcd7cff9700 1 -- 172.21.15.27:0/3621755432 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fcd6c023070 con 0x7fcd80142dc0 2024-01-11T08:22:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.120+0000 7fcd7cff9700 1 --2- 172.21.15.27:0/3621755432 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcd64067130 0x7fcd640695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.120+0000 7fcd7cff9700 1 -- 172.21.15.27:0/3621755432 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fcd6c085fc0 con 0x7fcd80142dc0 2024-01-11T08:22:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.120+0000 7fcd84f9b700 1 --2- 172.21.15.27:0/3621755432 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcd64067130 0x7fcd640695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.121+0000 7fcd84f9b700 1 --2- 172.21.15.27:0/3621755432 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcd64067130 0x7fcd640695e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcd8006d750 tx=0x7fcd74009040 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:26.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.124+0000 7fcd7cff9700 1 -- 172.21.15.27:0/3621755432 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcd6c052e10 con 0x7fcd80142dc0 2024-01-11T08:22:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.354+0000 7fcd871ff700 1 -- 172.21.15.27:0/3621755432 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcd8006c4b0 con 0x7fcd64067130 2024-01-11T08:22:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.363+0000 7fcd7cff9700 1 -- 172.21.15.27:0/3621755432 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fcd8006c4b0 con 0x7fcd64067130 2024-01-11T08:22:26.362 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:22:26.363 INFO:teuthology.orchestra.run.smithi027.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:41", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.365+0000 7fcd871ff700 1 -- 172.21.15.27:0/3621755432 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcd64067130 msgr2=0x7fcd640695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.365+0000 7fcd871ff700 1 --2- 172.21.15.27:0/3621755432 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcd64067130 0x7fcd640695e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcd8006d750 tx=0x7fcd74009040 comp rx=0 tx=0).stop 2024-01-11T08:22:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.365+0000 7fcd871ff700 1 -- 172.21.15.27:0/3621755432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd80142dc0 msgr2=0x7fcd800c66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.365+0000 7fcd871ff700 1 --2- 172.21.15.27:0/3621755432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd80142dc0 0x7fcd800c66c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcd80064a70 tx=0x7fcd6c017000 comp rx=0 tx=0).stop 2024-01-11T08:22:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.365+0000 7fcd871ff700 1 -- 172.21.15.27:0/3621755432 shutdown_connections 2024-01-11T08:22:26.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.365+0000 7fcd871ff700 1 --2- 172.21.15.27:0/3621755432 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcd64067130 0x7fcd640695e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.365+0000 7fcd871ff700 1 --2- 172.21.15.27:0/3621755432 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd80143190 0x7fcd800b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.365+0000 7fcd871ff700 1 --2- 172.21.15.27:0/3621755432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd800cb4b0 0x7fcd801411d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.365+0000 7fcd871ff700 1 --2- 172.21.15.27:0/3621755432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd80142dc0 0x7fcd800c66c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.365+0000 7fcd871ff700 1 -- 172.21.15.27:0/3621755432 >> 172.21.15.27:0/3621755432 conn(0x7fcd80063ca0 msgr2=0x7fcd8015b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:26.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.366+0000 7fcd871ff700 1 -- 172.21.15.27:0/3621755432 shutdown_connections 2024-01-11T08:22:26.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.366+0000 7fcd871ff700 1 -- 172.21.15.27:0/3621755432 wait complete. 2024-01-11T08:22:26.383 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:41", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:26.383 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1m', 'retention': {}, 'start': '2024-01-11T00:00:00', 'created': '2024-01-11T08:11:41', 'first': '2024-01-11T08:12:00', 'last': '2024-01-11T08:22:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-11T08:22:26.383 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-11T08:22:26.383 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-11T08:22:26.383 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:26.383 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2024-01-11T08:22:26.432 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:26.432 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:26.433 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:26.433 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:26.433 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:26.433 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:26.433 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:26.433 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:26.433 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:26.433 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:26.433 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:26.434 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-11-08_12_00_UTC', 'scheduled-2024-01-11-08_13_00_UTC', 'scheduled-2024-01-11-08_14_00_UTC', 'scheduled-2024-01-11-08_15_00_UTC', 'scheduled-2024-01-11-08_16_00_UTC', 'scheduled-2024-01-11-08_17_00_UTC', 'scheduled-2024-01-11-08_18_00_UTC', 'scheduled-2024-01-11-08_19_00_UTC', 'scheduled-2024-01-11-08_20_00_UTC', 'scheduled-2024-01-11-08_21_00_UTC', 'scheduled-2024-01-11-08_22_00_UTC'] 2024-01-11T08:22:26.434 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir1 2024-01-11T08:22:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.635+0000 7f68acbff700 1 Processor -- start 2024-01-11T08:22:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.635+0000 7f68acbff700 1 -- start start 2024-01-11T08:22:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.637+0000 7f68acbff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f68a813ff80 0x7f68a815a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.637+0000 7f68acbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f68a815a560 0x7f68a80c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.637+0000 7f68acbff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a80c6b30 con 0x7f68a813fbb0 2024-01-11T08:22:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.637+0000 7f68acbff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68a80c6800 con 0x7f68a813ff80 2024-01-11T08:22:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.637+0000 7f68acbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68a815a960 con 0x7f68a815a560 2024-01-11T08:22:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.637+0000 7f68a659c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f68a813fbb0 0x7f68a80ca7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54414/0 (socket says 172.21.15.27:54414) 2024-01-11T08:22:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.637+0000 7f68a659c700 1 -- 172.21.15.27:0/3418033673 learned_addr learned my addr 172.21.15.27:0/3418033673 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.637+0000 7f68a5d9b700 1 --2- 172.21.15.27:0/3418033673 >> v2:172.21.15.27:3300/0 conn(0x7f68a813ff80 0x7f68a815a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.637+0000 7f68a6d9d700 1 --2- 172.21.15.27:0/3418033673 >> v2:172.21.15.77:3300/0 conn(0x7f68a815a560 0x7f68a80c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.638+0000 7f68a559a700 1 -- 172.21.15.27:0/3418033673 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235376081 0 0) 0x7f68a80c6b30 con 0x7f68a813fbb0 2024-01-11T08:22:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.638+0000 7f68a559a700 1 -- 172.21.15.27:0/3418033673 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6890003570 con 0x7f68a813fbb0 2024-01-11T08:22:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.638+0000 7f68a5d9b700 1 -- 172.21.15.27:0/3418033673 >> v2:172.21.15.77:3300/0 conn(0x7f68a815a560 msgr2=0x7f68a80c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.638+0000 7f68a5d9b700 1 --2- 172.21.15.27:0/3418033673 >> v2:172.21.15.77:3300/0 conn(0x7f68a815a560 0x7f68a80c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.638+0000 7f68a5d9b700 1 -- 172.21.15.27:0/3418033673 >> v1:172.21.15.27:6789/0 conn(0x7f68a813fbb0 legacy=0x7f68a80ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.638+0000 7f68a5d9b700 1 -- 172.21.15.27:0/3418033673 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a80cb240 con 0x7f68a813ff80 2024-01-11T08:22:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.638+0000 7f68a5d9b700 1 --2- 172.21.15.27:0/3418033673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a813ff80 0x7f68a815a020 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f68a806fe10 tx=0x7f689c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed3f60c803045a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.638+0000 7f68a559a700 1 -- 172.21.15.27:0/3418033673 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c00f6b0 con 0x7f68a813ff80 2024-01-11T08:22:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.638+0000 7f68a559a700 1 -- 172.21.15.27:0/3418033673 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689c00f810 con 0x7f68a813ff80 2024-01-11T08:22:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.638+0000 7f68a559a700 1 -- 172.21.15.27:0/3418033673 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c00fbb0 con 0x7f68a813ff80 2024-01-11T08:22:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.639+0000 7f68acbff700 1 -- 172.21.15.27:0/3418033673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a813ff80 msgr2=0x7f68a815a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.639+0000 7f68acbff700 1 --2- 172.21.15.27:0/3418033673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a813ff80 0x7f68a815a020 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f68a806fe10 tx=0x7f689c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.639+0000 7f68acbff700 1 -- 172.21.15.27:0/3418033673 shutdown_connections 2024-01-11T08:22:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.639+0000 7f68acbff700 1 --2- 172.21.15.27:0/3418033673 >> v2:172.21.15.77:3300/0 conn(0x7f68a815a560 0x7f68a80c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.639+0000 7f68acbff700 1 --2- 172.21.15.27:0/3418033673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a813ff80 0x7f68a815a020 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.639+0000 7f68acbff700 1 -- 172.21.15.27:0/3418033673 >> 172.21.15.27:0/3418033673 conn(0x7f68a80680a0 msgr2=0x7f68a80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.639+0000 7f68acbff700 1 -- 172.21.15.27:0/3418033673 shutdown_connections 2024-01-11T08:22:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.639+0000 7f68acbff700 1 -- 172.21.15.27:0/3418033673 wait complete. 2024-01-11T08:22:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.640+0000 7f68acbff700 1 Processor -- start 2024-01-11T08:22:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.641+0000 7f68acbff700 1 -- start start 2024-01-11T08:22:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.642+0000 7f68acbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a813fbb0 0x7f68a8157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.642+0000 7f68acbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f68a813ff80 0x7f68a8155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.642+0000 7f68acbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68a815a560 0x7f68a8158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.642+0000 7f68acbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a81576a0 con 0x7f68a813fbb0 2024-01-11T08:22:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.642+0000 7f68acbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a8158070 con 0x7f68a815a560 2024-01-11T08:22:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.642+0000 7f68acbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a8156a90 con 0x7f68a813ff80 2024-01-11T08:22:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.642+0000 7f68a659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a813fbb0 0x7f68a8157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.642+0000 7f68a659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a813fbb0 0x7f68a8157b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51846/0 (socket says 172.21.15.27:51846) 2024-01-11T08:22:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.642+0000 7f68a659c700 1 -- 172.21.15.27:0/3207958060 learned_addr learned my addr 172.21.15.27:0/3207958060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.642+0000 7f68a5d9b700 1 --2- 172.21.15.27:0/3207958060 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f68a813ff80 0x7f68a8155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.642+0000 7f68a6d9d700 1 --2- 172.21.15.27:0/3207958060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68a815a560 0x7f68a8158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:26.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.642+0000 7f68a659c700 1 -- 172.21.15.27:0/3207958060 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f68a813ff80 msgr2=0x7f68a8155bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:26.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.642+0000 7f68a659c700 1 --2- 172.21.15.27:0/3207958060 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f68a813ff80 0x7f68a8155bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.642+0000 7f68a659c700 1 -- 172.21.15.27:0/3207958060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68a815a560 msgr2=0x7f68a8158540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:26.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.642+0000 7f68a659c700 1 --2- 172.21.15.27:0/3207958060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68a815a560 0x7f68a8158540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.642+0000 7f68a659c700 1 -- 172.21.15.27:0/3207958060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689c009420 con 0x7f68a813fbb0 2024-01-11T08:22:26.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.643+0000 7f68a659c700 1 --2- 172.21.15.27:0/3207958060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a813fbb0 0x7f68a8157b30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f689800c790 tx=0x7f6898007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:26.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.647+0000 7f68967fc700 1 -- 172.21.15.27:0/3207958060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68980072e0 con 0x7f68a813fbb0 2024-01-11T08:22:26.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.647+0000 7f68967fc700 1 -- 172.21.15.27:0/3207958060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6898007440 con 0x7f68a813fbb0 2024-01-11T08:22:26.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.647+0000 7f68acbff700 1 -- 172.21.15.27:0/3207958060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a8156d70 con 0x7f68a813fbb0 2024-01-11T08:22:26.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.647+0000 7f68967fc700 1 -- 172.21.15.27:0/3207958060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68980057f0 con 0x7f68a813fbb0 2024-01-11T08:22:26.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.647+0000 7f68acbff700 1 -- 172.21.15.27:0/3207958060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a8157330 con 0x7f68a813fbb0 2024-01-11T08:22:26.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.648+0000 7f68acbff700 1 -- 172.21.15.27:0/3207958060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a804c3c0 con 0x7f68a813fbb0 2024-01-11T08:22:26.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.649+0000 7f68967fc700 1 -- 172.21.15.27:0/3207958060 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f6898005be0 con 0x7f68a813fbb0 2024-01-11T08:22:26.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.650+0000 7f68967fc700 1 --2- 172.21.15.27:0/3207958060 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6888067200 0x7f68880696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:26.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.650+0000 7f68967fc700 1 -- 172.21.15.27:0/3207958060 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f6898083ce0 con 0x7f68a813fbb0 2024-01-11T08:22:26.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.651+0000 7f68a5d9b700 1 --2- 172.21.15.27:0/3207958060 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6888067200 0x7f68880696b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:26.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.652+0000 7f68a5d9b700 1 --2- 172.21.15.27:0/3207958060 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6888067200 0x7f68880696b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f689c0093f0 tx=0x7f689c019000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:26.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.654+0000 7f68967fc700 1 -- 172.21.15.27:0/3207958060 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f689804b5a0 con 0x7f68a813fbb0 2024-01-11T08:22:26.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.883+0000 7f68acbff700 1 -- 172.21.15.27:0/3207958060 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f68a81577e0 con 0x7f6888067200 2024-01-11T08:22:26.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.893+0000 7f68967fc700 1 -- 172.21.15.27:0/3207958060 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f689c000bc0 con 0x7f6888067200 2024-01-11T08:22:26.892 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:22:26.892 INFO:teuthology.orchestra.run.smithi027.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:42", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.895+0000 7f68acbff700 1 -- 172.21.15.27:0/3207958060 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6888067200 msgr2=0x7f68880696b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.895+0000 7f68acbff700 1 --2- 172.21.15.27:0/3207958060 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6888067200 0x7f68880696b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f689c0093f0 tx=0x7f689c019000 comp rx=0 tx=0).stop 2024-01-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.895+0000 7f68acbff700 1 -- 172.21.15.27:0/3207958060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a813fbb0 msgr2=0x7f68a8157b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.895+0000 7f68acbff700 1 --2- 172.21.15.27:0/3207958060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a813fbb0 0x7f68a8157b30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f689800c790 tx=0x7f6898007c80 comp rx=0 tx=0).stop 2024-01-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.895+0000 7f68acbff700 1 -- 172.21.15.27:0/3207958060 shutdown_connections 2024-01-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.895+0000 7f68acbff700 1 --2- 172.21.15.27:0/3207958060 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6888067200 0x7f68880696b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.895+0000 7f68acbff700 1 --2- 172.21.15.27:0/3207958060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68a815a560 0x7f68a8158540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.895+0000 7f68acbff700 1 --2- 172.21.15.27:0/3207958060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68a813fbb0 0x7f68a8157b30 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.895+0000 7f68acbff700 1 --2- 172.21.15.27:0/3207958060 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f68a813ff80 0x7f68a8155bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:26.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.895+0000 7f68acbff700 1 -- 172.21.15.27:0/3207958060 >> 172.21.15.27:0/3207958060 conn(0x7f68a80680a0 msgr2=0x7f68a80ca810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:26.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.896+0000 7f68acbff700 1 -- 172.21.15.27:0/3207958060 shutdown_connections 2024-01-11T08:22:26.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:26.896+0000 7f68acbff700 1 -- 172.21.15.27:0/3207958060 wait complete. 2024-01-11T08:22:26.913 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:42", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:26.913 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1m', 'retention': {}, 'start': '2024-01-11T00:00:00', 'created': '2024-01-11T08:11:42', 'first': '2024-01-11T08:12:00', 'last': '2024-01-11T08:22:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-11T08:22:26.913 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-11T08:22:26.913 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-11T08:22:26.913 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:26.914 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2024-01-11T08:22:26.962 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:26.963 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:26.964 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-11-08_12_00_UTC', 'scheduled-2024-01-11-08_13_00_UTC', 'scheduled-2024-01-11-08_14_00_UTC', 'scheduled-2024-01-11-08_15_00_UTC', 'scheduled-2024-01-11-08_16_00_UTC', 'scheduled-2024-01-11-08_17_00_UTC', 'scheduled-2024-01-11-08_18_00_UTC', 'scheduled-2024-01-11-08_19_00_UTC', 'scheduled-2024-01-11-08_20_00_UTC', 'scheduled-2024-01-11-08_21_00_UTC', 'scheduled-2024-01-11-08_22_00_UTC'] 2024-01-11T08:22:26.964 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir2 2024-01-11T08:22:27.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.166+0000 7fd988fff700 1 Processor -- start 2024-01-11T08:22:27.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.166+0000 7fd988fff700 1 -- start start 2024-01-11T08:22:27.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.167+0000 7fd988fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd984142dc0 0x7fd9840cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:27.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.167+0000 7fd988fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd984143190 0x7fd9840cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:27.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.167+0000 7fd988fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd98415d320 con 0x7fd9840cc020 2024-01-11T08:22:27.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.167+0000 7fd988fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd98415df30 con 0x7fd984142dc0 2024-01-11T08:22:27.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.167+0000 7fd988fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd98415d590 con 0x7fd984143190 2024-01-11T08:22:27.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.168+0000 7fd98259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd984142dc0 0x7fd9840cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:27.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.168+0000 7fd98259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd984142dc0 0x7fd9840cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51850/0 (socket says 172.21.15.27:51850) 2024-01-11T08:22:27.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.168+0000 7fd98259c700 1 -- 172.21.15.27:0/145289630 learned_addr learned my addr 172.21.15.27:0/145289630 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:27.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.168+0000 7fd982d9d700 1 --1- 172.21.15.27:0/145289630 >> v1:172.21.15.27:6789/0 conn(0x7fd9840cc020 0x7fd9840cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54428/0 (socket says 172.21.15.27:54428) 2024-01-11T08:22:27.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.168+0000 7fd981d9b700 1 --2- 172.21.15.27:0/145289630 >> v2:172.21.15.77:3300/0 conn(0x7fd984143190 0x7fd9840cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:27.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.168+0000 7fd98259c700 1 -- 172.21.15.27:0/145289630 >> v2:172.21.15.77:3300/0 conn(0x7fd984143190 msgr2=0x7fd9840cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:27.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.168+0000 7fd98259c700 1 --2- 172.21.15.27:0/145289630 >> v2:172.21.15.77:3300/0 conn(0x7fd984143190 0x7fd9840cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:27.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.168+0000 7fd98259c700 1 -- 172.21.15.27:0/145289630 >> v1:172.21.15.27:6789/0 conn(0x7fd9840cc020 legacy=0x7fd9840cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:27.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.168+0000 7fd98259c700 1 -- 172.21.15.27:0/145289630 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd98415d870 con 0x7fd984142dc0 2024-01-11T08:22:27.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.168+0000 7fd98259c700 1 --2- 172.21.15.27:0/145289630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd984142dc0 0x7fd9840cada0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd98406c480 tx=0x7fd974009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71e681abe1355f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:27.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.169+0000 7fd98159a700 1 -- 172.21.15.27:0/145289630 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974009e70 con 0x7fd984142dc0 2024-01-11T08:22:27.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.169+0000 7fd98159a700 1 -- 172.21.15.27:0/145289630 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97400b860 con 0x7fd984142dc0 2024-01-11T08:22:27.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.169+0000 7fd98159a700 1 -- 172.21.15.27:0/145289630 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97400bc00 con 0x7fd984142dc0 2024-01-11T08:22:27.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.169+0000 7fd988fff700 1 -- 172.21.15.27:0/145289630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd984142dc0 msgr2=0x7fd9840cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:27.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.169+0000 7fd988fff700 1 --2- 172.21.15.27:0/145289630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd984142dc0 0x7fd9840cada0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd98406c480 tx=0x7fd974009250 comp rx=0 tx=0).stop 2024-01-11T08:22:27.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.169+0000 7fd988fff700 1 -- 172.21.15.27:0/145289630 shutdown_connections 2024-01-11T08:22:27.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.169+0000 7fd988fff700 1 --2- 172.21.15.27:0/145289630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd984142dc0 0x7fd9840cada0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:27.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.169+0000 7fd988fff700 1 --2- 172.21.15.27:0/145289630 >> v2:172.21.15.77:3300/0 conn(0x7fd984143190 0x7fd9840cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:27.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.169+0000 7fd988fff700 1 -- 172.21.15.27:0/145289630 >> 172.21.15.27:0/145289630 conn(0x7fd984063ca0 msgr2=0x7fd98406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:27.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.170+0000 7fd988fff700 1 -- 172.21.15.27:0/145289630 shutdown_connections 2024-01-11T08:22:27.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.170+0000 7fd988fff700 1 -- 172.21.15.27:0/145289630 wait complete. 2024-01-11T08:22:27.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.171+0000 7fd988fff700 1 Processor -- start 2024-01-11T08:22:27.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.171+0000 7fd988fff700 1 -- start start 2024-01-11T08:22:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.172+0000 7fd988fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9840cc020 0x7fd984158ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.172+0000 7fd988fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd984142dc0 0x7fd984159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.172+0000 7fd988fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd984143190 0x7fd9840c6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.172+0000 7fd988fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9840c6190 con 0x7fd984143190 2024-01-11T08:22:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.172+0000 7fd988fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9840c55c0 con 0x7fd9840cc020 2024-01-11T08:22:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.172+0000 7fd988fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9840c5860 con 0x7fd984142dc0 2024-01-11T08:22:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.172+0000 7fd982d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd984143190 0x7fd9840c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.172+0000 7fd982d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd984143190 0x7fd9840c6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51852/0 (socket says 172.21.15.27:51852) 2024-01-11T08:22:27.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.172+0000 7fd982d9d700 1 -- 172.21.15.27:0/1953610021 learned_addr learned my addr 172.21.15.27:0/1953610021 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:27.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.172+0000 7fd981d9b700 1 --2- 172.21.15.27:0/1953610021 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd984142dc0 0x7fd984159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:27.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.172+0000 7fd98259c700 1 --2- 172.21.15.27:0/1953610021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9840cc020 0x7fd984158ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:27.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.173+0000 7fd982d9d700 1 -- 172.21.15.27:0/1953610021 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd984142dc0 msgr2=0x7fd984159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:27.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.173+0000 7fd982d9d700 1 --2- 172.21.15.27:0/1953610021 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd984142dc0 0x7fd984159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:27.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.173+0000 7fd982d9d700 1 -- 172.21.15.27:0/1953610021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9840cc020 msgr2=0x7fd984158ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:27.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.173+0000 7fd982d9d700 1 --2- 172.21.15.27:0/1953610021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9840cc020 0x7fd984158ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:27.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.173+0000 7fd982d9d700 1 -- 172.21.15.27:0/1953610021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9740094f0 con 0x7fd984143190 2024-01-11T08:22:27.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.173+0000 7fd982d9d700 1 --2- 172.21.15.27:0/1953610021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd984143190 0x7fd9840c6620 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00cd00 tx=0x7fd97c001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:27.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.173+0000 7fd9727fc700 1 -- 172.21.15.27:0/1953610021 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c00ce60 con 0x7fd984143190 2024-01-11T08:22:27.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.173+0000 7fd9727fc700 1 -- 172.21.15.27:0/1953610021 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97c005df0 con 0x7fd984143190 2024-01-11T08:22:27.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.173+0000 7fd988fff700 1 -- 172.21.15.27:0/1953610021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9840c5b40 con 0x7fd984143190 2024-01-11T08:22:27.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.173+0000 7fd9727fc700 1 -- 172.21.15.27:0/1953610021 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c00ab00 con 0x7fd984143190 2024-01-11T08:22:27.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.174+0000 7fd988fff700 1 -- 172.21.15.27:0/1953610021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd98416af60 con 0x7fd984143190 2024-01-11T08:22:27.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.174+0000 7fd988fff700 1 -- 172.21.15.27:0/1953610021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9840c6300 con 0x7fd984143190 2024-01-11T08:22:27.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.175+0000 7fd9727fc700 1 -- 172.21.15.27:0/1953610021 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fd97c0070f0 con 0x7fd984143190 2024-01-11T08:22:27.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.177+0000 7fd9727fc700 1 --2- 172.21.15.27:0/1953610021 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd964067130 0x7fd9640695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:27.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.177+0000 7fd9727fc700 1 -- 172.21.15.27:0/1953610021 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fd97c084a50 con 0x7fd984143190 2024-01-11T08:22:27.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.177+0000 7fd98259c700 1 --2- 172.21.15.27:0/1953610021 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd964067130 0x7fd9640695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:27.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.178+0000 7fd98259c700 1 --2- 172.21.15.27:0/1953610021 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd964067130 0x7fd9640695e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd98406c480 tx=0x7fd974017000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:27.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.180+0000 7fd9727fc700 1 -- 172.21.15.27:0/1953610021 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd97c051920 con 0x7fd984143190 2024-01-11T08:22:27.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.408+0000 7fd988fff700 1 -- 172.21.15.27:0/1953610021 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd98404c3c0 con 0x7fd964067130 2024-01-11T08:22:27.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.417+0000 7fd9727fc700 1 -- 172.21.15.27:0/1953610021 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fd98404c3c0 con 0x7fd964067130 2024-01-11T08:22:27.417 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:22:27.417 INFO:teuthology.orchestra.run.smithi027.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:42", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:27.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.420+0000 7fd988fff700 1 -- 172.21.15.27:0/1953610021 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd964067130 msgr2=0x7fd9640695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:27.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.420+0000 7fd988fff700 1 --2- 172.21.15.27:0/1953610021 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd964067130 0x7fd9640695e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd98406c480 tx=0x7fd974017000 comp rx=0 tx=0).stop 2024-01-11T08:22:27.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.420+0000 7fd988fff700 1 -- 172.21.15.27:0/1953610021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd984143190 msgr2=0x7fd9840c6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:27.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.420+0000 7fd988fff700 1 --2- 172.21.15.27:0/1953610021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd984143190 0x7fd9840c6620 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00cd00 tx=0x7fd97c001b10 comp rx=0 tx=0).stop 2024-01-11T08:22:27.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.420+0000 7fd988fff700 1 -- 172.21.15.27:0/1953610021 shutdown_connections 2024-01-11T08:22:27.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.420+0000 7fd988fff700 1 --2- 172.21.15.27:0/1953610021 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd964067130 0x7fd9640695e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:27.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.420+0000 7fd988fff700 1 --2- 172.21.15.27:0/1953610021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd984143190 0x7fd9840c6620 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:27.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.420+0000 7fd988fff700 1 --2- 172.21.15.27:0/1953610021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9840cc020 0x7fd984158ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:27.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.420+0000 7fd988fff700 1 --2- 172.21.15.27:0/1953610021 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd984142dc0 0x7fd984159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:27.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.420+0000 7fd988fff700 1 -- 172.21.15.27:0/1953610021 >> 172.21.15.27:0/1953610021 conn(0x7fd984063ca0 msgr2=0x7fd98415b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:27.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.420+0000 7fd988fff700 1 -- 172.21.15.27:0/1953610021 shutdown_connections 2024-01-11T08:22:27.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.420+0000 7fd988fff700 1 -- 172.21.15.27:0/1953610021 wait complete. 2024-01-11T08:22:27.437 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:42", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:27.437 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1m', 'retention': {}, 'start': '2024-01-11T00:00:00', 'created': '2024-01-11T08:11:42', 'first': '2024-01-11T08:12:00', 'last': '2024-01-11T08:22:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-11T08:22:27.437 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-11T08:22:27.437 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-11T08:22:27.437 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:27.437 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2024-01-11T08:22:27.486 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:27.486 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:27.486 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:27.486 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:27.486 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:27.486 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:27.486 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:27.487 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:27.487 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:27.487 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:27.487 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_01_UTC 2024-01-11T08:22:27.488 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-11-08_12_00_UTC', 'scheduled-2024-01-11-08_13_00_UTC', 'scheduled-2024-01-11-08_14_00_UTC', 'scheduled-2024-01-11-08_15_00_UTC', 'scheduled-2024-01-11-08_16_00_UTC', 'scheduled-2024-01-11-08_17_00_UTC', 'scheduled-2024-01-11-08_18_00_UTC', 'scheduled-2024-01-11-08_19_00_UTC', 'scheduled-2024-01-11-08_20_00_UTC', 'scheduled-2024-01-11-08_21_00_UTC', 'scheduled-2024-01-11-08_22_01_UTC'] 2024-01-11T08:22:27.488 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir3 2024-01-11T08:22:27.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.690+0000 7fdbcb9ff700 1 Processor -- start 2024-01-11T08:22:27.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.691+0000 7fdbcb9ff700 1 -- start start 2024-01-11T08:22:27.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.692+0000 7fdbcb9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbc413fbb0 0x7fdbc40ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:27.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.692+0000 7fdbcb9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdbc413ff80 0x7fdbc415a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:27.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.692+0000 7fdbcb9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc40c6b30 con 0x7fdbc40cace0 2024-01-11T08:22:27.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.692+0000 7fdbcb9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc40c6800 con 0x7fdbc413ff80 2024-01-11T08:22:27.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.692+0000 7fdbcb9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc40cb240 con 0x7fdbc413fbb0 2024-01-11T08:22:27.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.692+0000 7fdbc8f9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdbc413ff80 0x7fdbc415a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:27.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.692+0000 7fdbc8f9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdbc413ff80 0x7fdbc415a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51864/0 (socket says 172.21.15.27:51864) 2024-01-11T08:22:27.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.692+0000 7fdbc8f9a700 1 -- 172.21.15.27:0/582887495 learned_addr learned my addr 172.21.15.27:0/582887495 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:27.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.692+0000 7fdbc979b700 1 --2- 172.21.15.27:0/582887495 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbc413fbb0 0x7fdbc40ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:27.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.693+0000 7fdbc8f9a700 1 -- 172.21.15.27:0/582887495 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbc413fbb0 msgr2=0x7fdbc40ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:27.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.693+0000 7fdbc8f9a700 1 --2- 172.21.15.27:0/582887495 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbc413fbb0 0x7fdbc40ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:27.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.693+0000 7fdbc8f9a700 1 -- 172.21.15.27:0/582887495 >> v1:172.21.15.77:6789/0 conn(0x7fdbc40cace0 legacy=0x7fdbc406e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:27.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.693+0000 7fdbc8f9a700 1 -- 172.21.15.27:0/582887495 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc40cb520 con 0x7fdbc413ff80 2024-01-11T08:22:27.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.693+0000 7fdbc979b700 1 --2- 172.21.15.27:0/582887495 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbc413fbb0 0x7fdbc40ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:22:27.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.693+0000 7fdbc8f9a700 1 --2- 172.21.15.27:0/582887495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc413ff80 0x7fdbc415a020 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400bc50 tx=0x7fdbb400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d457cc2c4edf34aa server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:27.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.693+0000 7fdbbbfff700 1 -- 172.21.15.27:0/582887495 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb400be60 con 0x7fdbc413ff80 2024-01-11T08:22:27.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.693+0000 7fdbbbfff700 1 -- 172.21.15.27:0/582887495 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb400de00 con 0x7fdbc413ff80 2024-01-11T08:22:27.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.693+0000 7fdbbbfff700 1 -- 172.21.15.27:0/582887495 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb40196b0 con 0x7fdbc413ff80 2024-01-11T08:22:27.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.694+0000 7fdbcb9ff700 1 -- 172.21.15.27:0/582887495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc413ff80 msgr2=0x7fdbc415a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:27.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.694+0000 7fdbcb9ff700 1 --2- 172.21.15.27:0/582887495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc413ff80 0x7fdbc415a020 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400bc50 tx=0x7fdbb400b4f0 comp rx=0 tx=0).stop 2024-01-11T08:22:27.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.694+0000 7fdbcb9ff700 1 -- 172.21.15.27:0/582887495 shutdown_connections 2024-01-11T08:22:27.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.694+0000 7fdbcb9ff700 1 --2- 172.21.15.27:0/582887495 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbc413fbb0 0x7fdbc40ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:27.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.694+0000 7fdbcb9ff700 1 --2- 172.21.15.27:0/582887495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc413ff80 0x7fdbc415a020 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:27.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.694+0000 7fdbcb9ff700 1 -- 172.21.15.27:0/582887495 >> 172.21.15.27:0/582887495 conn(0x7fdbc40680a0 msgr2=0x7fdbc40cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:27.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.694+0000 7fdbcb9ff700 1 -- 172.21.15.27:0/582887495 shutdown_connections 2024-01-11T08:22:27.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.694+0000 7fdbcb9ff700 1 -- 172.21.15.27:0/582887495 wait complete. 2024-01-11T08:22:27.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.695+0000 7fdbcb9ff700 1 Processor -- start 2024-01-11T08:22:27.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.696+0000 7fdbcb9ff700 1 -- start start 2024-01-11T08:22:27.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.697+0000 7fdbcb9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc40cace0 0x7fdbc40c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:27.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.697+0000 7fdbcb9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbc413fbb0 0x7fdbc413d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:27.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.697+0000 7fdbcb9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbc413ff80 0x7fdbc413d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:27.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.697+0000 7fdbcb9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc40c3270 con 0x7fdbc40cace0 2024-01-11T08:22:27.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.697+0000 7fdbcb9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc40affc0 con 0x7fdbc413fbb0 2024-01-11T08:22:27.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.697+0000 7fdbcb9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc40af3b0 con 0x7fdbc413ff80 2024-01-11T08:22:27.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.697+0000 7fdbc979b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc40cace0 0x7fdbc40c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:27.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.697+0000 7fdbc979b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc40cace0 0x7fdbc40c5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51872/0 (socket says 172.21.15.27:51872) 2024-01-11T08:22:27.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.697+0000 7fdbc979b700 1 -- 172.21.15.27:0/1947504817 learned_addr learned my addr 172.21.15.27:0/1947504817 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:27.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.697+0000 7fdbc8f9a700 1 --2- 172.21.15.27:0/1947504817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbc413fbb0 0x7fdbc413d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:27.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.697+0000 7fdbc9f9c700 1 --2- 172.21.15.27:0/1947504817 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbc413ff80 0x7fdbc413d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:27.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.697+0000 7fdbc979b700 1 -- 172.21.15.27:0/1947504817 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbc413ff80 msgr2=0x7fdbc413d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:27.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.697+0000 7fdbc979b700 1 --2- 172.21.15.27:0/1947504817 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbc413ff80 0x7fdbc413d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:27.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.697+0000 7fdbc979b700 1 -- 172.21.15.27:0/1947504817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbc413fbb0 msgr2=0x7fdbc413d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:27.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.697+0000 7fdbc979b700 1 --2- 172.21.15.27:0/1947504817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbc413fbb0 0x7fdbc413d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:27.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.697+0000 7fdbc979b700 1 -- 172.21.15.27:0/1947504817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb400b790 con 0x7fdbc40cace0 2024-01-11T08:22:27.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.697+0000 7fdbc8f9a700 1 --2- 172.21.15.27:0/1947504817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbc413fbb0 0x7fdbc413d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:22:27.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.697+0000 7fdbc979b700 1 --2- 172.21.15.27:0/1947504817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc40cace0 0x7fdbc40c5a60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fdbc000e920 tx=0x7fdbc0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:27.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.698+0000 7fdbb97fa700 1 -- 172.21.15.27:0/1947504817 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0005a00 con 0x7fdbc40cace0 2024-01-11T08:22:27.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.698+0000 7fdbb97fa700 1 -- 172.21.15.27:0/1947504817 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc0005b60 con 0x7fdbc40cace0 2024-01-11T08:22:27.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.698+0000 7fdbb97fa700 1 -- 172.21.15.27:0/1947504817 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0006020 con 0x7fdbc40cace0 2024-01-11T08:22:27.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.698+0000 7fdbcb9ff700 1 -- 172.21.15.27:0/1947504817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc40af690 con 0x7fdbc40cace0 2024-01-11T08:22:27.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.698+0000 7fdbcb9ff700 1 -- 172.21.15.27:0/1947504817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc40afc50 con 0x7fdbc40cace0 2024-01-11T08:22:27.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.699+0000 7fdbcb9ff700 1 -- 172.21.15.27:0/1947504817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc40b02a0 con 0x7fdbc40cace0 2024-01-11T08:22:27.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.700+0000 7fdbb97fa700 1 -- 172.21.15.27:0/1947504817 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fdbc0006410 con 0x7fdbc40cace0 2024-01-11T08:22:27.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.701+0000 7fdbb97fa700 1 --2- 172.21.15.27:0/1947504817 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdba8067130 0x7fdba80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:27.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.701+0000 7fdbb97fa700 1 -- 172.21.15.27:0/1947504817 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fdbc0084ca0 con 0x7fdbc40cace0 2024-01-11T08:22:27.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.701+0000 7fdbc8f9a700 1 --2- 172.21.15.27:0/1947504817 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdba8067130 0x7fdba80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:27.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.702+0000 7fdbc8f9a700 1 --2- 172.21.15.27:0/1947504817 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdba8067130 0x7fdba80695e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400bc50 tx=0x7fdbb401b000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:27.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.705+0000 7fdbb97fa700 1 -- 172.21.15.27:0/1947504817 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdbc00519e0 con 0x7fdbc40cace0 2024-01-11T08:22:27.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.934+0000 7fdbcb9ff700 1 -- 172.21.15.27:0/1947504817 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdbc404c3c0 con 0x7fdba8067130 2024-01-11T08:22:27.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.944+0000 7fdbb97fa700 1 -- 172.21.15.27:0/1947504817 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fdbc404c3c0 con 0x7fdba8067130 2024-01-11T08:22:27.943 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:22:27.943 INFO:teuthology.orchestra.run.smithi027.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:43", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:27.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.946+0000 7fdbcb9ff700 1 -- 172.21.15.27:0/1947504817 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdba8067130 msgr2=0x7fdba80695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:27.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.946+0000 7fdbcb9ff700 1 --2- 172.21.15.27:0/1947504817 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdba8067130 0x7fdba80695e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400bc50 tx=0x7fdbb401b000 comp rx=0 tx=0).stop 2024-01-11T08:22:27.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.947+0000 7fdbcb9ff700 1 -- 172.21.15.27:0/1947504817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc40cace0 msgr2=0x7fdbc40c5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:27.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.947+0000 7fdbcb9ff700 1 --2- 172.21.15.27:0/1947504817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc40cace0 0x7fdbc40c5a60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fdbc000e920 tx=0x7fdbc0008040 comp rx=0 tx=0).stop 2024-01-11T08:22:27.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.947+0000 7fdbcb9ff700 1 -- 172.21.15.27:0/1947504817 shutdown_connections 2024-01-11T08:22:27.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.947+0000 7fdbcb9ff700 1 --2- 172.21.15.27:0/1947504817 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdba8067130 0x7fdba80695e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:27.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.947+0000 7fdbcb9ff700 1 --2- 172.21.15.27:0/1947504817 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbc413ff80 0x7fdbc413d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:27.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.947+0000 7fdbcb9ff700 1 --2- 172.21.15.27:0/1947504817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc40cace0 0x7fdbc40c5a60 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:27.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.947+0000 7fdbcb9ff700 1 --2- 172.21.15.27:0/1947504817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbc413fbb0 0x7fdbc413d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:27.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.947+0000 7fdbcb9ff700 1 -- 172.21.15.27:0/1947504817 >> 172.21.15.27:0/1947504817 conn(0x7fdbc40680a0 msgr2=0x7fdbc40c80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:27.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.947+0000 7fdbcb9ff700 1 -- 172.21.15.27:0/1947504817 shutdown_connections 2024-01-11T08:22:27.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:27.947+0000 7fdbcb9ff700 1 -- 172.21.15.27:0/1947504817 wait complete. 2024-01-11T08:22:27.964 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:43", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:27.964 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1m', 'retention': {}, 'start': '2024-01-11T00:00:00', 'created': '2024-01-11T08:11:43', 'first': '2024-01-11T08:12:00', 'last': '2024-01-11T08:22:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-11T08:22:27.964 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-11T08:22:27.964 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-11T08:22:27.965 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:27.965 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2024-01-11T08:22:28.014 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:28.014 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:28.014 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:28.014 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:28.014 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:28.014 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:28.015 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:28.015 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:28.015 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:28.015 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:28.015 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:28.016 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-11-08_12_00_UTC', 'scheduled-2024-01-11-08_13_00_UTC', 'scheduled-2024-01-11-08_14_00_UTC', 'scheduled-2024-01-11-08_15_00_UTC', 'scheduled-2024-01-11-08_16_00_UTC', 'scheduled-2024-01-11-08_17_00_UTC', 'scheduled-2024-01-11-08_18_00_UTC', 'scheduled-2024-01-11-08_19_00_UTC', 'scheduled-2024-01-11-08_20_00_UTC', 'scheduled-2024-01-11-08_21_00_UTC', 'scheduled-2024-01-11-08_22_00_UTC'] 2024-01-11T08:22:28.016 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir4 2024-01-11T08:22:28.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.217+0000 7f35571ff700 1 Processor -- start 2024-01-11T08:22:28.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.218+0000 7f35571ff700 1 -- start start 2024-01-11T08:22:28.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.219+0000 7f35571ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35500ccd70 0x7f35500cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:28.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.219+0000 7f35571ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35500cf850 con 0x7f355015a1b0 2024-01-11T08:22:28.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.219+0000 7f35571ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35500cec40 con 0x7f355015ae70 2024-01-11T08:22:28.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.219+0000 7f35571ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355015ac90 con 0x7f35500ccd70 2024-01-11T08:22:28.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.219+0000 7f3554f9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f355015a1b0 0x7f355015a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54438/0 (socket says 172.21.15.27:54438) 2024-01-11T08:22:28.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.219+0000 7f3554f9b700 1 -- 172.21.15.27:0/1924320102 learned_addr learned my addr 172.21.15.27:0/1924320102 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:28.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.219+0000 7f354ffff700 1 --2- 172.21.15.27:0/1924320102 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35500ccd70 0x7f35500cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.220+0000 7f354f7fe700 1 -- 172.21.15.27:0/1924320102 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3203701049 0 0) 0x7f35500cf850 con 0x7f355015a1b0 2024-01-11T08:22:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.220+0000 7f354f7fe700 1 -- 172.21.15.27:0/1924320102 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3540003570 con 0x7f355015a1b0 2024-01-11T08:22:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.220+0000 7f354f7fe700 1 -- 172.21.15.27:0/1924320102 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006999195 0 0) 0x7f35500cec40 con 0x7f355015ae70 2024-01-11T08:22:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.220+0000 7f354f7fe700 1 -- 172.21.15.27:0/1924320102 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35500cf850 con 0x7f355015ae70 2024-01-11T08:22:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.220+0000 7f354ffff700 1 -- 172.21.15.27:0/1924320102 >> v1:172.21.15.77:6789/0 conn(0x7f355015ae70 legacy=0x7f355015b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.220+0000 7f354ffff700 1 -- 172.21.15.27:0/1924320102 >> v1:172.21.15.27:6789/0 conn(0x7f355015a1b0 legacy=0x7f355015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.220+0000 7f354ffff700 1 -- 172.21.15.27:0/1924320102 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35500cef20 con 0x7f35500ccd70 2024-01-11T08:22:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.220+0000 7f354ffff700 1 --2- 172.21.15.27:0/1924320102 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35500ccd70 0x7f35500cd1e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f353c009910 tx=0x7f353c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d67de4336c2dbde4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.221+0000 7f354f7fe700 1 -- 172.21.15.27:0/1924320102 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c011630 con 0x7f35500ccd70 2024-01-11T08:22:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.221+0000 7f354f7fe700 1 -- 172.21.15.27:0/1924320102 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353c011790 con 0x7f35500ccd70 2024-01-11T08:22:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.221+0000 7f354f7fe700 1 -- 172.21.15.27:0/1924320102 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c011b30 con 0x7f35500ccd70 2024-01-11T08:22:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.221+0000 7f35571ff700 1 -- 172.21.15.27:0/1924320102 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35500ccd70 msgr2=0x7f35500cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:28.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.221+0000 7f35571ff700 1 --2- 172.21.15.27:0/1924320102 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35500ccd70 0x7f35500cd1e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f353c009910 tx=0x7f353c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:28.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.221+0000 7f35571ff700 1 -- 172.21.15.27:0/1924320102 shutdown_connections 2024-01-11T08:22:28.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.221+0000 7f35571ff700 1 --2- 172.21.15.27:0/1924320102 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35500ccd70 0x7f35500cd1e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:28.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.221+0000 7f35571ff700 1 -- 172.21.15.27:0/1924320102 >> 172.21.15.27:0/1924320102 conn(0x7f355006ed40 msgr2=0x7f355006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:28.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.221+0000 7f35571ff700 1 -- 172.21.15.27:0/1924320102 shutdown_connections 2024-01-11T08:22:28.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.221+0000 7f35571ff700 1 -- 172.21.15.27:0/1924320102 wait complete. 2024-01-11T08:22:28.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.223+0000 7f35571ff700 1 Processor -- start 2024-01-11T08:22:28.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.223+0000 7f35571ff700 1 -- start start 2024-01-11T08:22:28.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.224+0000 7f35571ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35500ccd70 0x7f35500c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:28.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.224+0000 7f35571ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f355015a1b0 0x7f3550147ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:28.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.224+0000 7f35571ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f355015ae70 0x7f35500b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:28.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.224+0000 7f35571ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35500b3380 con 0x7f355015ae70 2024-01-11T08:22:28.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.224+0000 7f35571ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355016a110 con 0x7f355015a1b0 2024-01-11T08:22:28.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.224+0000 7f35571ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355016a3f0 con 0x7f35500ccd70 2024-01-11T08:22:28.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.224+0000 7f355579c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f355015ae70 0x7f35500b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:28.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.224+0000 7f355579c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f355015ae70 0x7f35500b44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51876/0 (socket says 172.21.15.27:51876) 2024-01-11T08:22:28.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.224+0000 7f355579c700 1 -- 172.21.15.27:0/2505852630 learned_addr learned my addr 172.21.15.27:0/2505852630 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:28.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.224+0000 7f3554f9b700 1 --2- 172.21.15.27:0/2505852630 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35500ccd70 0x7f35500c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:28.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.224+0000 7f354ffff700 1 --2- 172.21.15.27:0/2505852630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f355015a1b0 0x7f3550147ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:28.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.225+0000 7f355579c700 1 -- 172.21.15.27:0/2505852630 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35500ccd70 msgr2=0x7f35500c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:28.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.225+0000 7f355579c700 1 --2- 172.21.15.27:0/2505852630 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35500ccd70 0x7f35500c7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:28.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.225+0000 7f355579c700 1 -- 172.21.15.27:0/2505852630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f355015a1b0 msgr2=0x7f3550147ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:28.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.225+0000 7f355579c700 1 --2- 172.21.15.27:0/2505852630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f355015a1b0 0x7f3550147ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:28.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.225+0000 7f355579c700 1 -- 172.21.15.27:0/2505852630 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353c009420 con 0x7f355015ae70 2024-01-11T08:22:28.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.225+0000 7f355579c700 1 --2- 172.21.15.27:0/2505852630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f355015ae70 0x7f35500b44b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f354800c910 tx=0x7f35480027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:28.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.225+0000 7f354cff9700 1 -- 172.21.15.27:0/2505852630 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3548012630 con 0x7f355015ae70 2024-01-11T08:22:28.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.225+0000 7f354cff9700 1 -- 172.21.15.27:0/2505852630 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35480127b0 con 0x7f355015ae70 2024-01-11T08:22:28.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.225+0000 7f35571ff700 1 -- 172.21.15.27:0/2505852630 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355016a6d0 con 0x7f355015ae70 2024-01-11T08:22:28.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.225+0000 7f354cff9700 1 -- 172.21.15.27:0/2505852630 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3548012b70 con 0x7f355015ae70 2024-01-11T08:22:28.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.225+0000 7f35571ff700 1 -- 172.21.15.27:0/2505852630 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35500ca450 con 0x7f355015ae70 2024-01-11T08:22:28.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.226+0000 7f35571ff700 1 -- 172.21.15.27:0/2505852630 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355016abd0 con 0x7f355015ae70 2024-01-11T08:22:28.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.227+0000 7f354cff9700 1 -- 172.21.15.27:0/2505852630 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f354801b400 con 0x7f355015ae70 2024-01-11T08:22:28.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.228+0000 7f354cff9700 1 --2- 172.21.15.27:0/2505852630 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3534067130 0x7f35340695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:28.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.228+0000 7f354cff9700 1 -- 172.21.15.27:0/2505852630 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f3548084d20 con 0x7f355015ae70 2024-01-11T08:22:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.228+0000 7f3554f9b700 1 --2- 172.21.15.27:0/2505852630 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3534067130 0x7f35340695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.229+0000 7f3554f9b700 1 --2- 172.21.15.27:0/2505852630 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3534067130 0x7f35340695e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3544007fd0 tx=0x7f3544009040 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:28.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.232+0000 7f354cff9700 1 -- 172.21.15.27:0/2505852630 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3548051c10 con 0x7f355015ae70 2024-01-11T08:22:28.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.460+0000 7f35571ff700 1 -- 172.21.15.27:0/2505852630 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f35500b34f0 con 0x7f3534067130 2024-01-11T08:22:28.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.470+0000 7f354cff9700 1 -- 172.21.15.27:0/2505852630 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f35500b34f0 con 0x7f3534067130 2024-01-11T08:22:28.469 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:22:28.469 INFO:teuthology.orchestra.run.smithi027.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:44", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:28.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.472+0000 7f35571ff700 1 -- 172.21.15.27:0/2505852630 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3534067130 msgr2=0x7f35340695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:28.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.472+0000 7f35571ff700 1 --2- 172.21.15.27:0/2505852630 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3534067130 0x7f35340695e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3544007fd0 tx=0x7f3544009040 comp rx=0 tx=0).stop 2024-01-11T08:22:28.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.472+0000 7f35571ff700 1 -- 172.21.15.27:0/2505852630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f355015ae70 msgr2=0x7f35500b44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:28.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.472+0000 7f35571ff700 1 --2- 172.21.15.27:0/2505852630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f355015ae70 0x7f35500b44b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f354800c910 tx=0x7f35480027a0 comp rx=0 tx=0).stop 2024-01-11T08:22:28.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.472+0000 7f35571ff700 1 -- 172.21.15.27:0/2505852630 shutdown_connections 2024-01-11T08:22:28.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.472+0000 7f35571ff700 1 --2- 172.21.15.27:0/2505852630 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3534067130 0x7f35340695e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:28.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.472+0000 7f35571ff700 1 --2- 172.21.15.27:0/2505852630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f355015ae70 0x7f35500b44b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:28.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.472+0000 7f35571ff700 1 --2- 172.21.15.27:0/2505852630 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35500ccd70 0x7f35500c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:28.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.472+0000 7f35571ff700 1 --2- 172.21.15.27:0/2505852630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f355015a1b0 0x7f3550147ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:28.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.472+0000 7f35571ff700 1 -- 172.21.15.27:0/2505852630 >> 172.21.15.27:0/2505852630 conn(0x7f355006ed40 msgr2=0x7f35501561b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:28.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.473+0000 7f35571ff700 1 -- 172.21.15.27:0/2505852630 shutdown_connections 2024-01-11T08:22:28.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.473+0000 7f35571ff700 1 -- 172.21.15.27:0/2505852630 wait complete. 2024-01-11T08:22:28.489 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:44", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:28.489 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1m', 'retention': {}, 'start': '2024-01-11T00:00:00', 'created': '2024-01-11T08:11:44', 'first': '2024-01-11T08:12:00', 'last': '2024-01-11T08:22:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-11T08:22:28.489 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-11T08:22:28.489 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-11T08:22:28.490 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:28.490 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2024-01-11T08:22:28.541 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:28.541 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:28.541 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:28.541 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:28.541 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:28.541 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:28.541 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:28.541 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:28.541 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:28.541 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_01_UTC 2024-01-11T08:22:28.542 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:28.542 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-11-08_12_00_UTC', 'scheduled-2024-01-11-08_13_00_UTC', 'scheduled-2024-01-11-08_14_00_UTC', 'scheduled-2024-01-11-08_15_00_UTC', 'scheduled-2024-01-11-08_16_00_UTC', 'scheduled-2024-01-11-08_17_00_UTC', 'scheduled-2024-01-11-08_18_00_UTC', 'scheduled-2024-01-11-08_19_00_UTC', 'scheduled-2024-01-11-08_20_00_UTC', 'scheduled-2024-01-11-08_21_01_UTC', 'scheduled-2024-01-11-08_22_00_UTC'] 2024-01-11T08:22:28.543 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir5 2024-01-11T08:22:28.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.744+0000 7f7c70dff700 1 Processor -- start 2024-01-11T08:22:28.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.744+0000 7f7c70dff700 1 -- start start 2024-01-11T08:22:28.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.746+0000 7f7c70dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c6c1577e0 0x7f7c6c0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:28.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.746+0000 7f7c70dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c6c0699d0 con 0x7f7c6c157410 2024-01-11T08:22:28.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.746+0000 7f7c70dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c6c069cb0 con 0x7f7c6c069540 2024-01-11T08:22:28.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.746+0000 7f7c70dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c6c069f90 con 0x7f7c6c1577e0 2024-01-11T08:22:28.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.746+0000 7f7c69d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c6c1577e0 0x7f7c6c0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:28.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.746+0000 7f7c6a59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7c6c157410 0x7f7c6c0ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54454/0 (socket says 172.21.15.27:54454) 2024-01-11T08:22:28.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.746+0000 7f7c6a59c700 1 -- 172.21.15.27:0/2267734657 learned_addr learned my addr 172.21.15.27:0/2267734657 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:28.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.746+0000 7f7c69d9b700 1 -- 172.21.15.27:0/2267734657 >> v1:172.21.15.77:6789/0 conn(0x7f7c6c069540 legacy=0x7f7c6c0c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:28.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.746+0000 7f7c69d9b700 1 -- 172.21.15.27:0/2267734657 >> v1:172.21.15.27:6789/0 conn(0x7f7c6c157410 legacy=0x7f7c6c0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:28.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.746+0000 7f7c6959a700 1 -- 172.21.15.27:0/2267734657 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141019845 0 0) 0x7f7c6c0699d0 con 0x7f7c6c157410 2024-01-11T08:22:28.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.746+0000 7f7c69d9b700 1 -- 172.21.15.27:0/2267734657 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c6c06a270 con 0x7f7c6c1577e0 2024-01-11T08:22:28.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.747+0000 7f7c69d9b700 1 --2- 172.21.15.27:0/2267734657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c6c1577e0 0x7f7c6c0ae000 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7c600098e0 tx=0x7f7c600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45a74fe94042a265 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:28.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.747+0000 7f7c6959a700 1 -- 172.21.15.27:0/2267734657 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c60009e80 con 0x7f7c6c1577e0 2024-01-11T08:22:28.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.747+0000 7f7c6959a700 1 -- 172.21.15.27:0/2267734657 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6000b860 con 0x7f7c6c1577e0 2024-01-11T08:22:28.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.747+0000 7f7c6959a700 1 -- 172.21.15.27:0/2267734657 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6000bbe0 con 0x7f7c6c1577e0 2024-01-11T08:22:28.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.747+0000 7f7c70dff700 1 -- 172.21.15.27:0/2267734657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c6c1577e0 msgr2=0x7f7c6c0ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:28.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.747+0000 7f7c70dff700 1 --2- 172.21.15.27:0/2267734657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c6c1577e0 0x7f7c6c0ae000 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7c600098e0 tx=0x7f7c600091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:28.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.747+0000 7f7c70dff700 1 -- 172.21.15.27:0/2267734657 shutdown_connections 2024-01-11T08:22:28.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.748+0000 7f7c70dff700 1 --2- 172.21.15.27:0/2267734657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c6c1577e0 0x7f7c6c0ae000 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:28.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.748+0000 7f7c70dff700 1 -- 172.21.15.27:0/2267734657 >> 172.21.15.27:0/2267734657 conn(0x7f7c6c0680a0 msgr2=0x7f7c6c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:28.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.748+0000 7f7c70dff700 1 -- 172.21.15.27:0/2267734657 shutdown_connections 2024-01-11T08:22:28.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.748+0000 7f7c70dff700 1 -- 172.21.15.27:0/2267734657 wait complete. 2024-01-11T08:22:28.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.750+0000 7f7c70dff700 1 Processor -- start 2024-01-11T08:22:28.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.750+0000 7f7c70dff700 1 -- start start 2024-01-11T08:22:28.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.751+0000 7f7c70dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c6c069540 0x7f7c6c16a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:28.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.751+0000 7f7c70dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c6c157410 0x7f7c6c158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:28.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.751+0000 7f7c70dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c6c1577e0 0x7f7c6c154000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:28.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.751+0000 7f7c70dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6c16a8c0 con 0x7f7c6c069540 2024-01-11T08:22:28.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.751+0000 7f7c70dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6c15b580 con 0x7f7c6c157410 2024-01-11T08:22:28.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.751+0000 7f7c70dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6c15ba10 con 0x7f7c6c1577e0 2024-01-11T08:22:28.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.751+0000 7f7c6a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c6c069540 0x7f7c6c16a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:28.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.752+0000 7f7c6a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c6c069540 0x7f7c6c16a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51894/0 (socket says 172.21.15.27:51894) 2024-01-11T08:22:28.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.752+0000 7f7c6a59c700 1 -- 172.21.15.27:0/1859916066 learned_addr learned my addr 172.21.15.27:0/1859916066 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.752+0000 7f7c69d9b700 1 --2- 172.21.15.27:0/1859916066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c6c157410 0x7f7c6c158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.752+0000 7f7c6ad9d700 1 --2- 172.21.15.27:0/1859916066 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c6c1577e0 0x7f7c6c154000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.752+0000 7f7c6a59c700 1 -- 172.21.15.27:0/1859916066 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c6c1577e0 msgr2=0x7f7c6c154000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.752+0000 7f7c6a59c700 1 --2- 172.21.15.27:0/1859916066 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c6c1577e0 0x7f7c6c154000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.752+0000 7f7c6a59c700 1 -- 172.21.15.27:0/1859916066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c6c157410 msgr2=0x7f7c6c158810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.752+0000 7f7c6a59c700 1 --2- 172.21.15.27:0/1859916066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c6c157410 0x7f7c6c158810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.752+0000 7f7c6a59c700 1 -- 172.21.15.27:0/1859916066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c60009420 con 0x7f7c6c069540 2024-01-11T08:22:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.752+0000 7f7c6a59c700 1 --2- 172.21.15.27:0/1859916066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c6c069540 0x7f7c6c16a380 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00c810 tx=0x7f7c5c007c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.753+0000 7f7c5a7fc700 1 -- 172.21.15.27:0/1859916066 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0072e0 con 0x7f7c6c069540 2024-01-11T08:22:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.753+0000 7f7c5a7fc700 1 -- 172.21.15.27:0/1859916066 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5c007440 con 0x7f7c6c069540 2024-01-11T08:22:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.753+0000 7f7c70dff700 1 -- 172.21.15.27:0/1859916066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c6c15a970 con 0x7f7c6c069540 2024-01-11T08:22:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.753+0000 7f7c5a7fc700 1 -- 172.21.15.27:0/1859916066 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0057f0 con 0x7f7c6c069540 2024-01-11T08:22:28.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.753+0000 7f7c70dff700 1 -- 172.21.15.27:0/1859916066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c6c15af30 con 0x7f7c6c069540 2024-01-11T08:22:28.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.754+0000 7f7c70dff700 1 -- 172.21.15.27:0/1859916066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c6c15b860 con 0x7f7c6c069540 2024-01-11T08:22:28.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.755+0000 7f7c5a7fc700 1 -- 172.21.15.27:0/1859916066 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f7c5c005c40 con 0x7f7c6c069540 2024-01-11T08:22:28.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.756+0000 7f7c5a7fc700 1 --2- 172.21.15.27:0/1859916066 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7c4c067130 0x7f7c4c0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:28.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.756+0000 7f7c5a7fc700 1 -- 172.21.15.27:0/1859916066 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f7c5c083fa0 con 0x7f7c6c069540 2024-01-11T08:22:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.757+0000 7f7c69d9b700 1 --2- 172.21.15.27:0/1859916066 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7c4c067130 0x7f7c4c0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:28.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.757+0000 7f7c69d9b700 1 --2- 172.21.15.27:0/1859916066 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7c4c067130 0x7f7c4c0695e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7c600098e0 tx=0x7f7c60017000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:28.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.760+0000 7f7c5a7fc700 1 -- 172.21.15.27:0/1859916066 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c5c050df0 con 0x7f7c6c069540 2024-01-11T08:22:28.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:28.990+0000 7f7c70dff700 1 -- 172.21.15.27:0/1859916066 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7c6c15b6c0 con 0x7f7c4c067130 2024-01-11T08:22:28.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.000+0000 7f7c5a7fc700 1 -- 172.21.15.27:0/1859916066 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f7c60020040 con 0x7f7c4c067130 2024-01-11T08:22:28.999 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:22:28.999 INFO:teuthology.orchestra.run.smithi027.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:44", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:29.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.002+0000 7f7c70dff700 1 -- 172.21.15.27:0/1859916066 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7c4c067130 msgr2=0x7f7c4c0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:29.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.002+0000 7f7c70dff700 1 --2- 172.21.15.27:0/1859916066 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7c4c067130 0x7f7c4c0695e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7c600098e0 tx=0x7f7c60017000 comp rx=0 tx=0).stop 2024-01-11T08:22:29.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.002+0000 7f7c70dff700 1 -- 172.21.15.27:0/1859916066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c6c069540 msgr2=0x7f7c6c16a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:29.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.002+0000 7f7c70dff700 1 --2- 172.21.15.27:0/1859916066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c6c069540 0x7f7c6c16a380 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00c810 tx=0x7f7c5c007c00 comp rx=0 tx=0).stop 2024-01-11T08:22:29.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.002+0000 7f7c70dff700 1 -- 172.21.15.27:0/1859916066 shutdown_connections 2024-01-11T08:22:29.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.002+0000 7f7c70dff700 1 --2- 172.21.15.27:0/1859916066 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7c4c067130 0x7f7c4c0695e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:29.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.002+0000 7f7c70dff700 1 --2- 172.21.15.27:0/1859916066 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c6c1577e0 0x7f7c6c154000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:29.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.002+0000 7f7c70dff700 1 --2- 172.21.15.27:0/1859916066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c6c069540 0x7f7c6c16a380 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:29.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.002+0000 7f7c70dff700 1 --2- 172.21.15.27:0/1859916066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c6c157410 0x7f7c6c158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:29.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.002+0000 7f7c70dff700 1 -- 172.21.15.27:0/1859916066 >> 172.21.15.27:0/1859916066 conn(0x7f7c6c0680a0 msgr2=0x7f7c6c0af6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:29.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.002+0000 7f7c70dff700 1 -- 172.21.15.27:0/1859916066 shutdown_connections 2024-01-11T08:22:29.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.002+0000 7f7c70dff700 1 -- 172.21.15.27:0/1859916066 wait complete. 2024-01-11T08:22:29.019 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:44", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:29.019 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1m', 'retention': {}, 'start': '2024-01-11T00:00:00', 'created': '2024-01-11T08:11:44', 'first': '2024-01-11T08:12:00', 'last': '2024-01-11T08:22:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-11T08:22:29.019 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-11T08:22:29.019 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-11T08:22:29.019 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:29.020 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2024-01-11T08:22:29.068 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:29.068 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:29.068 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:29.069 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:29.069 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:29.069 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:29.069 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:29.069 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:29.069 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:29.069 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:29.069 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:29.070 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-11-08_12_00_UTC', 'scheduled-2024-01-11-08_13_00_UTC', 'scheduled-2024-01-11-08_14_00_UTC', 'scheduled-2024-01-11-08_15_00_UTC', 'scheduled-2024-01-11-08_16_00_UTC', 'scheduled-2024-01-11-08_17_00_UTC', 'scheduled-2024-01-11-08_18_00_UTC', 'scheduled-2024-01-11-08_19_00_UTC', 'scheduled-2024-01-11-08_20_00_UTC', 'scheduled-2024-01-11-08_21_00_UTC', 'scheduled-2024-01-11-08_22_00_UTC'] 2024-01-11T08:22:29.070 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir6 2024-01-11T08:22:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.271+0000 7fe8153ff700 1 Processor -- start 2024-01-11T08:22:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.272+0000 7fe8153ff700 1 -- start start 2024-01-11T08:22:29.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.273+0000 7fe8153ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe810068d30 0x7fe8100c8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:29.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.273+0000 7fe8153ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8100c80a0 con 0x7fe8101596f0 2024-01-11T08:22:29.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.273+0000 7fe8153ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8100cbac0 con 0x7fe810159ac0 2024-01-11T08:22:29.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.273+0000 7fe8153ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8100a9fb0 con 0x7fe810068d30 2024-01-11T08:22:29.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.273+0000 7fe80effd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe8101596f0 0x7fe8100a98a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54460/0 (socket says 172.21.15.27:54460) 2024-01-11T08:22:29.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.273+0000 7fe80effd700 1 -- 172.21.15.27:0/2879336383 learned_addr learned my addr 172.21.15.27:0/2879336383 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:29.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.274+0000 7fe80f7fe700 1 --2- 172.21.15.27:0/2879336383 >> v2:172.21.15.77:3300/0 conn(0x7fe810068d30 0x7fe8100c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:29.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.274+0000 7fe80dffb700 1 -- 172.21.15.27:0/2879336383 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814553729 0 0) 0x7fe8100cbac0 con 0x7fe810159ac0 2024-01-11T08:22:29.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.274+0000 7fe80dffb700 1 -- 172.21.15.27:0/2879336383 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7f8003570 con 0x7fe810159ac0 2024-01-11T08:22:29.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.274+0000 7fe80dffb700 1 -- 172.21.15.27:0/2879336383 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892110388 0 0) 0x7fe8100c80a0 con 0x7fe8101596f0 2024-01-11T08:22:29.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.274+0000 7fe80dffb700 1 -- 172.21.15.27:0/2879336383 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8100cbac0 con 0x7fe8101596f0 2024-01-11T08:22:29.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.274+0000 7fe80f7fe700 1 -- 172.21.15.27:0/2879336383 >> v1:172.21.15.77:6789/0 conn(0x7fe810159ac0 legacy=0x7fe810068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:29.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.274+0000 7fe80f7fe700 1 -- 172.21.15.27:0/2879336383 >> v1:172.21.15.27:6789/0 conn(0x7fe8101596f0 legacy=0x7fe8100a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:29.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.274+0000 7fe80f7fe700 1 -- 172.21.15.27:0/2879336383 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8100c7490 con 0x7fe810068d30 2024-01-11T08:22:29.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.274+0000 7fe80f7fe700 1 --2- 172.21.15.27:0/2879336383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe810068d30 0x7fe8100c8530 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe808009e60 tx=0x7fe808009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95042ec7b7284236 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:29.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.275+0000 7fe80dffb700 1 -- 172.21.15.27:0/2879336383 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80800e070 con 0x7fe810068d30 2024-01-11T08:22:29.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.275+0000 7fe80dffb700 1 -- 172.21.15.27:0/2879336383 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80800bab0 con 0x7fe810068d30 2024-01-11T08:22:29.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.275+0000 7fe80dffb700 1 -- 172.21.15.27:0/2879336383 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80800be70 con 0x7fe810068d30 2024-01-11T08:22:29.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.275+0000 7fe8153ff700 1 -- 172.21.15.27:0/2879336383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe810068d30 msgr2=0x7fe8100c8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:29.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.275+0000 7fe8153ff700 1 --2- 172.21.15.27:0/2879336383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe810068d30 0x7fe8100c8530 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe808009e60 tx=0x7fe808009780 comp rx=0 tx=0).stop 2024-01-11T08:22:29.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.275+0000 7fe8153ff700 1 -- 172.21.15.27:0/2879336383 shutdown_connections 2024-01-11T08:22:29.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.275+0000 7fe8153ff700 1 --2- 172.21.15.27:0/2879336383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe810068d30 0x7fe8100c8530 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:29.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.275+0000 7fe8153ff700 1 -- 172.21.15.27:0/2879336383 >> 172.21.15.27:0/2879336383 conn(0x7fe8100c9fb0 msgr2=0x7fe81006ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:29.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.275+0000 7fe8153ff700 1 -- 172.21.15.27:0/2879336383 shutdown_connections 2024-01-11T08:22:29.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.275+0000 7fe8153ff700 1 -- 172.21.15.27:0/2879336383 wait complete. 2024-01-11T08:22:29.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.277+0000 7fe8153ff700 1 Processor -- start 2024-01-11T08:22:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.278+0000 7fe8153ff700 1 -- start start 2024-01-11T08:22:29.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.279+0000 7fe8153ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe810068d30 0x7fe8101454b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:29.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.279+0000 7fe8153ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8101596f0 0x7fe81015ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:29.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.279+0000 7fe8153ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe810159ac0 0x7fe81015d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:29.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.279+0000 7fe8153ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8100ac0f0 con 0x7fe810159ac0 2024-01-11T08:22:29.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.279+0000 7fe8153ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81015cb10 con 0x7fe810068d30 2024-01-11T08:22:29.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.279+0000 7fe8153ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe81015c270 con 0x7fe8101596f0 2024-01-11T08:22:29.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.279+0000 7fe80f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe810159ac0 0x7fe81015d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:29.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.279+0000 7fe80f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe810159ac0 0x7fe81015d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51902/0 (socket says 172.21.15.27:51902) 2024-01-11T08:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.279+0000 7fe80f7fe700 1 -- 172.21.15.27:0/2352266267 learned_addr learned my addr 172.21.15.27:0/2352266267 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.279+0000 7fe80effd700 1 --2- 172.21.15.27:0/2352266267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe810068d30 0x7fe8101454b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.279+0000 7fe80e7fc700 1 --2- 172.21.15.27:0/2352266267 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8101596f0 0x7fe81015ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.280+0000 7fe80f7fe700 1 -- 172.21.15.27:0/2352266267 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8101596f0 msgr2=0x7fe81015ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.280+0000 7fe80f7fe700 1 --2- 172.21.15.27:0/2352266267 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8101596f0 0x7fe81015ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.280+0000 7fe80f7fe700 1 -- 172.21.15.27:0/2352266267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe810068d30 msgr2=0x7fe8101454b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.280+0000 7fe80f7fe700 1 --2- 172.21.15.27:0/2352266267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe810068d30 0x7fe8101454b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.280+0000 7fe80f7fe700 1 -- 172.21.15.27:0/2352266267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe808009a20 con 0x7fe810159ac0 2024-01-11T08:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.280+0000 7fe80f7fe700 1 --2- 172.21.15.27:0/2352266267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe810159ac0 0x7fe81015d9e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe81006d730 tx=0x7fe808019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.280+0000 7fe7f6ffd700 1 -- 172.21.15.27:0/2352266267 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe808012db0 con 0x7fe810159ac0 2024-01-11T08:22:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.280+0000 7fe7f6ffd700 1 -- 172.21.15.27:0/2352266267 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe808026480 con 0x7fe810159ac0 2024-01-11T08:22:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.280+0000 7fe8153ff700 1 -- 172.21.15.27:0/2352266267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe81015c550 con 0x7fe810159ac0 2024-01-11T08:22:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.280+0000 7fe7f6ffd700 1 -- 172.21.15.27:0/2352266267 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe808026840 con 0x7fe810159ac0 2024-01-11T08:22:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.281+0000 7fe8153ff700 1 -- 172.21.15.27:0/2352266267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8101459f0 con 0x7fe810159ac0 2024-01-11T08:22:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.281+0000 7fe8153ff700 1 -- 172.21.15.27:0/2352266267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8100abec0 con 0x7fe810159ac0 2024-01-11T08:22:29.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.282+0000 7fe7f6ffd700 1 -- 172.21.15.27:0/2352266267 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fe80801c400 con 0x7fe810159ac0 2024-01-11T08:22:29.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.283+0000 7fe7f6ffd700 1 --2- 172.21.15.27:0/2352266267 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe7fc067130 0x7fe7fc0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:29.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.283+0000 7fe7f6ffd700 1 -- 172.21.15.27:0/2352266267 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fe808086a00 con 0x7fe810159ac0 2024-01-11T08:22:29.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.284+0000 7fe80effd700 1 --2- 172.21.15.27:0/2352266267 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe7fc067130 0x7fe7fc0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:29.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.284+0000 7fe80effd700 1 --2- 172.21.15.27:0/2352266267 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe7fc067130 0x7fe7fc0695e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe800007fd0 tx=0x7fe800009040 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.288+0000 7fe7f6ffd700 1 -- 172.21.15.27:0/2352266267 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe8100abec0 con 0x7fe810159ac0 2024-01-11T08:22:29.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.520+0000 7fe8153ff700 1 -- 172.21.15.27:0/2352266267 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe8100abec0 con 0x7fe7fc067130 2024-01-11T08:22:29.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.529+0000 7fe7f6ffd700 1 -- 172.21.15.27:0/2352266267 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fe8100abec0 con 0x7fe7fc067130 2024-01-11T08:22:29.529 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:22:29.529 INFO:teuthology.orchestra.run.smithi027.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:45", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:29.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.532+0000 7fe8153ff700 1 -- 172.21.15.27:0/2352266267 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe7fc067130 msgr2=0x7fe7fc0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:29.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.532+0000 7fe8153ff700 1 --2- 172.21.15.27:0/2352266267 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe7fc067130 0x7fe7fc0695e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe800007fd0 tx=0x7fe800009040 comp rx=0 tx=0).stop 2024-01-11T08:22:29.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.532+0000 7fe8153ff700 1 -- 172.21.15.27:0/2352266267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe810159ac0 msgr2=0x7fe81015d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:29.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.532+0000 7fe8153ff700 1 --2- 172.21.15.27:0/2352266267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe810159ac0 0x7fe81015d9e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe81006d730 tx=0x7fe808019000 comp rx=0 tx=0).stop 2024-01-11T08:22:29.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.532+0000 7fe8153ff700 1 -- 172.21.15.27:0/2352266267 shutdown_connections 2024-01-11T08:22:29.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.532+0000 7fe8153ff700 1 --2- 172.21.15.27:0/2352266267 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe7fc067130 0x7fe7fc0695e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:29.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.532+0000 7fe8153ff700 1 --2- 172.21.15.27:0/2352266267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe810159ac0 0x7fe81015d9e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:29.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.532+0000 7fe8153ff700 1 --2- 172.21.15.27:0/2352266267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe810068d30 0x7fe8101454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:29.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.532+0000 7fe8153ff700 1 --2- 172.21.15.27:0/2352266267 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8101596f0 0x7fe81015ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:29.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.532+0000 7fe8153ff700 1 -- 172.21.15.27:0/2352266267 >> 172.21.15.27:0/2352266267 conn(0x7fe8100c9fb0 msgr2=0x7fe8100cac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:29.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.532+0000 7fe8153ff700 1 -- 172.21.15.27:0/2352266267 shutdown_connections 2024-01-11T08:22:29.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.532+0000 7fe8153ff700 1 -- 172.21.15.27:0/2352266267 wait complete. 2024-01-11T08:22:29.549 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:45", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:29.549 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1m', 'retention': {}, 'start': '2024-01-11T00:00:00', 'created': '2024-01-11T08:11:45', 'first': '2024-01-11T08:12:00', 'last': '2024-01-11T08:22:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-11T08:22:29.549 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-11T08:22:29.549 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-11T08:22:29.549 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:29.549 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2024-01-11T08:22:29.598 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:29.598 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:29.598 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:29.598 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:29.598 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:29.599 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:29.599 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:29.599 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:29.599 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:29.599 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:29.599 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:29.600 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-11-08_12_00_UTC', 'scheduled-2024-01-11-08_13_00_UTC', 'scheduled-2024-01-11-08_14_00_UTC', 'scheduled-2024-01-11-08_15_00_UTC', 'scheduled-2024-01-11-08_16_00_UTC', 'scheduled-2024-01-11-08_17_00_UTC', 'scheduled-2024-01-11-08_18_00_UTC', 'scheduled-2024-01-11-08_19_00_UTC', 'scheduled-2024-01-11-08_20_00_UTC', 'scheduled-2024-01-11-08_21_00_UTC', 'scheduled-2024-01-11-08_22_00_UTC'] 2024-01-11T08:22:29.600 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir7 2024-01-11T08:22:29.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.803+0000 7f98761ff700 1 Processor -- start 2024-01-11T08:22:29.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.803+0000 7f98761ff700 1 -- start start 2024-01-11T08:22:29.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.804+0000 7f98761ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9870143190 0x7f98700cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:29.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.805+0000 7f98761ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98700cb4b0 0x7f98700cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:29.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.805+0000 7f98761ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f987015d320 con 0x7f9870142dc0 2024-01-11T08:22:29.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.805+0000 7f98761ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f987015df30 con 0x7f9870143190 2024-01-11T08:22:29.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.805+0000 7f98761ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987015d590 con 0x7f98700cb4b0 2024-01-11T08:22:29.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.805+0000 7f986f7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9870142dc0 0x7f98700cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54470/0 (socket says 172.21.15.27:54470) 2024-01-11T08:22:29.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.805+0000 7f986f7fe700 1 -- 172.21.15.27:0/668621917 learned_addr learned my addr 172.21.15.27:0/668621917 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:29.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.805+0000 7f986effd700 1 --2- 172.21.15.27:0/668621917 >> v2:172.21.15.77:3300/0 conn(0x7f9870143190 0x7f98700cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:29.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.805+0000 7f986ffff700 1 --2- 172.21.15.27:0/668621917 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98700cb4b0 0x7f98700cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:29.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.805+0000 7f986e7fc700 1 -- 172.21.15.27:0/668621917 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965900625 0 0) 0x7f987015d320 con 0x7f9870142dc0 2024-01-11T08:22:29.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.805+0000 7f986e7fc700 1 -- 172.21.15.27:0/668621917 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9858003570 con 0x7f9870142dc0 2024-01-11T08:22:29.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.806+0000 7f986effd700 1 -- 172.21.15.27:0/668621917 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98700cb4b0 msgr2=0x7f98700cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:29.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.806+0000 7f986effd700 1 --2- 172.21.15.27:0/668621917 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98700cb4b0 0x7f98700cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:29.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.806+0000 7f986effd700 1 -- 172.21.15.27:0/668621917 >> v1:172.21.15.27:6789/0 conn(0x7f9870142dc0 legacy=0x7f98700cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:29.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.806+0000 7f986effd700 1 -- 172.21.15.27:0/668621917 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987015d870 con 0x7f9870143190 2024-01-11T08:22:29.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.806+0000 7f986effd700 1 --2- 172.21.15.27:0/668621917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9870143190 0x7f98700cc020 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f98640098e0 tx=0x7f98640091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25e684cb6a972f80 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:29.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.806+0000 7f986e7fc700 1 -- 172.21.15.27:0/668621917 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9864011630 con 0x7f9870143190 2024-01-11T08:22:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.806+0000 7f986e7fc700 1 -- 172.21.15.27:0/668621917 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9864011790 con 0x7f9870143190 2024-01-11T08:22:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.806+0000 7f986e7fc700 1 -- 172.21.15.27:0/668621917 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9864011b30 con 0x7f9870143190 2024-01-11T08:22:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.807+0000 7f98761ff700 1 -- 172.21.15.27:0/668621917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9870143190 msgr2=0x7f98700cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.807+0000 7f98761ff700 1 --2- 172.21.15.27:0/668621917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9870143190 0x7f98700cc020 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f98640098e0 tx=0x7f98640091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.807+0000 7f98761ff700 1 -- 172.21.15.27:0/668621917 shutdown_connections 2024-01-11T08:22:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.807+0000 7f98761ff700 1 --2- 172.21.15.27:0/668621917 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98700cb4b0 0x7f98700cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.807+0000 7f98761ff700 1 --2- 172.21.15.27:0/668621917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9870143190 0x7f98700cc020 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.807+0000 7f98761ff700 1 -- 172.21.15.27:0/668621917 >> 172.21.15.27:0/668621917 conn(0x7f9870063ca0 msgr2=0x7f987006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.807+0000 7f98761ff700 1 -- 172.21.15.27:0/668621917 shutdown_connections 2024-01-11T08:22:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.807+0000 7f98761ff700 1 -- 172.21.15.27:0/668621917 wait complete. 2024-01-11T08:22:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.808+0000 7f98761ff700 1 Processor -- start 2024-01-11T08:22:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.808+0000 7f98761ff700 1 -- start start 2024-01-11T08:22:29.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.809+0000 7f98761ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98700cb4b0 0x7f987006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:29.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.809+0000 7f98761ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9870142dc0 0x7f9870133870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:29.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.809+0000 7f98761ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9870143190 0x7f9870133db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:29.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.809+0000 7f98761ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98701342f0 con 0x7f98700cb4b0 2024-01-11T08:22:29.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.809+0000 7f98761ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987016db10 con 0x7f9870143190 2024-01-11T08:22:29.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.809+0000 7f98761ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987016cf00 con 0x7f9870142dc0 2024-01-11T08:22:29.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.810+0000 7f986f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98700cb4b0 0x7f987006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:29.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.810+0000 7f986f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98700cb4b0 0x7f987006d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51914/0 (socket says 172.21.15.27:51914) 2024-01-11T08:22:29.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.810+0000 7f986f7fe700 1 -- 172.21.15.27:0/999422877 learned_addr learned my addr 172.21.15.27:0/999422877 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:29.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.810+0000 7f986effd700 1 --2- 172.21.15.27:0/999422877 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9870142dc0 0x7f9870133870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:29.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.810+0000 7f986ffff700 1 --2- 172.21.15.27:0/999422877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9870143190 0x7f9870133db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:29.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.810+0000 7f986f7fe700 1 -- 172.21.15.27:0/999422877 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9870142dc0 msgr2=0x7f9870133870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.810+0000 7f986f7fe700 1 --2- 172.21.15.27:0/999422877 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9870142dc0 0x7f9870133870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.810+0000 7f986f7fe700 1 -- 172.21.15.27:0/999422877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9870143190 msgr2=0x7f9870133db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.810+0000 7f986f7fe700 1 --2- 172.21.15.27:0/999422877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9870143190 0x7f9870133db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.810+0000 7f986f7fe700 1 -- 172.21.15.27:0/999422877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9864009420 con 0x7f98700cb4b0 2024-01-11T08:22:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.810+0000 7f986ffff700 1 --2- 172.21.15.27:0/999422877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9870143190 0x7f9870133db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:22:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.810+0000 7f986f7fe700 1 --2- 172.21.15.27:0/999422877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98700cb4b0 0x7f987006d680 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f98600011e0 tx=0x7f9860007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.811+0000 7f98577fe700 1 -- 172.21.15.27:0/999422877 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9860005b30 con 0x7f98700cb4b0 2024-01-11T08:22:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.811+0000 7f98577fe700 1 -- 172.21.15.27:0/999422877 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9860005c90 con 0x7f98700cb4b0 2024-01-11T08:22:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.811+0000 7f98577fe700 1 -- 172.21.15.27:0/999422877 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98600072e0 con 0x7f98700cb4b0 2024-01-11T08:22:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.811+0000 7f98761ff700 1 -- 172.21.15.27:0/999422877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f987016d1e0 con 0x7f98700cb4b0 2024-01-11T08:22:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.811+0000 7f98761ff700 1 -- 172.21.15.27:0/999422877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f987016d7a0 con 0x7f98700cb4b0 2024-01-11T08:22:29.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.812+0000 7f98761ff700 1 -- 172.21.15.27:0/999422877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f987016ddf0 con 0x7f98700cb4b0 2024-01-11T08:22:29.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.813+0000 7f98577fe700 1 -- 172.21.15.27:0/999422877 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f9860007650 con 0x7f98700cb4b0 2024-01-11T08:22:29.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.814+0000 7f98577fe700 1 --2- 172.21.15.27:0/999422877 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f985c067130 0x7f985c0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:29.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.814+0000 7f98577fe700 1 -- 172.21.15.27:0/999422877 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f9860083f00 con 0x7f98700cb4b0 2024-01-11T08:22:29.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.815+0000 7f986effd700 1 --2- 172.21.15.27:0/999422877 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f985c067130 0x7f985c0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:29.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.815+0000 7f986effd700 1 --2- 172.21.15.27:0/999422877 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f985c067130 0x7f985c0695e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f98640091e0 tx=0x7f9864017000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:29.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:29.819+0000 7f98577fe700 1 -- 172.21.15.27:0/999422877 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9860050e00 con 0x7f98700cb4b0 2024-01-11T08:22:30.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.044+0000 7f98761ff700 1 -- 172.21.15.27:0/999422877 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f987004c3c0 con 0x7f985c067130 2024-01-11T08:22:30.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.054+0000 7f98577fe700 1 -- 172.21.15.27:0/999422877 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f987004c3c0 con 0x7f985c067130 2024-01-11T08:22:30.053 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:22:30.053 INFO:teuthology.orchestra.run.smithi027.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:45", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.056+0000 7f98761ff700 1 -- 172.21.15.27:0/999422877 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f985c067130 msgr2=0x7f985c0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.056+0000 7f98761ff700 1 --2- 172.21.15.27:0/999422877 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f985c067130 0x7f985c0695e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f98640091e0 tx=0x7f9864017000 comp rx=0 tx=0).stop 2024-01-11T08:22:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.056+0000 7f98761ff700 1 -- 172.21.15.27:0/999422877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98700cb4b0 msgr2=0x7f987006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.056+0000 7f98761ff700 1 --2- 172.21.15.27:0/999422877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98700cb4b0 0x7f987006d680 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f98600011e0 tx=0x7f9860007df0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.056+0000 7f98761ff700 1 -- 172.21.15.27:0/999422877 shutdown_connections 2024-01-11T08:22:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.056+0000 7f98761ff700 1 --2- 172.21.15.27:0/999422877 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f985c067130 0x7f985c0695e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.056+0000 7f98761ff700 1 --2- 172.21.15.27:0/999422877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9870143190 0x7f9870133db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.056+0000 7f98761ff700 1 --2- 172.21.15.27:0/999422877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98700cb4b0 0x7f987006d680 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.056+0000 7f98761ff700 1 --2- 172.21.15.27:0/999422877 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9870142dc0 0x7f9870133870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.056+0000 7f98761ff700 1 -- 172.21.15.27:0/999422877 >> 172.21.15.27:0/999422877 conn(0x7f9870063ca0 msgr2=0x7f98700c45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.056+0000 7f98761ff700 1 -- 172.21.15.27:0/999422877 shutdown_connections 2024-01-11T08:22:30.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.056+0000 7f98761ff700 1 -- 172.21.15.27:0/999422877 wait complete. 2024-01-11T08:22:30.073 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:45", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:30.073 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1m', 'retention': {}, 'start': '2024-01-11T00:00:00', 'created': '2024-01-11T08:11:45', 'first': '2024-01-11T08:12:00', 'last': '2024-01-11T08:22:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-11T08:22:30.074 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-11T08:22:30.074 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-11T08:22:30.074 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:30.074 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2024-01-11T08:22:30.122 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:30.122 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:30.123 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:30.123 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:30.123 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:30.123 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:30.123 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:30.123 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:30.123 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:30.123 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:30.123 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:30.124 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-11-08_12_00_UTC', 'scheduled-2024-01-11-08_13_00_UTC', 'scheduled-2024-01-11-08_14_00_UTC', 'scheduled-2024-01-11-08_15_00_UTC', 'scheduled-2024-01-11-08_16_00_UTC', 'scheduled-2024-01-11-08_17_00_UTC', 'scheduled-2024-01-11-08_18_00_UTC', 'scheduled-2024-01-11-08_19_00_UTC', 'scheduled-2024-01-11-08_20_00_UTC', 'scheduled-2024-01-11-08_21_00_UTC', 'scheduled-2024-01-11-08_22_00_UTC'] 2024-01-11T08:22:30.124 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir8 2024-01-11T08:22:30.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.326+0000 7ffb427ff700 1 Processor -- start 2024-01-11T08:22:30.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.327+0000 7ffb427ff700 1 -- start start 2024-01-11T08:22:30.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.328+0000 7ffb427ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ffb3c159ac0 0x7ffb3c068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:30.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.328+0000 7ffb427ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffb3c0ca640 0x7ffb3c068b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:30.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.328+0000 7ffb427ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb3c0c80a0 con 0x7ffb3c1596f0 2024-01-11T08:22:30.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.328+0000 7ffb427ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb3c0cbac0 con 0x7ffb3c159ac0 2024-01-11T08:22:30.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.328+0000 7ffb427ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb3c0a9fb0 con 0x7ffb3c0ca640 2024-01-11T08:22:30.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.329+0000 7ffb3b7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ffb3c159ac0 0x7ffb3c068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:30.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.329+0000 7ffb3b7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ffb3c159ac0 0x7ffb3c068620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51922/0 (socket says 172.21.15.27:51922) 2024-01-11T08:22:30.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.329+0000 7ffb3b7fe700 1 -- 172.21.15.27:0/1353580097 learned_addr learned my addr 172.21.15.27:0/1353580097 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:30.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.329+0000 7ffb3bfff700 1 --1- 172.21.15.27:0/1353580097 >> v1:172.21.15.27:6789/0 conn(0x7ffb3c1596f0 0x7ffb3c0a98a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54478/0 (socket says 172.21.15.27:54478) 2024-01-11T08:22:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.329+0000 7ffb40d9c700 1 --2- 172.21.15.27:0/1353580097 >> v2:172.21.15.77:3300/0 conn(0x7ffb3c0ca640 0x7ffb3c068b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.329+0000 7ffb3b7fe700 1 -- 172.21.15.27:0/1353580097 >> v2:172.21.15.77:3300/0 conn(0x7ffb3c0ca640 msgr2=0x7ffb3c068b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.329+0000 7ffb3b7fe700 1 --2- 172.21.15.27:0/1353580097 >> v2:172.21.15.77:3300/0 conn(0x7ffb3c0ca640 0x7ffb3c068b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.329+0000 7ffb3b7fe700 1 -- 172.21.15.27:0/1353580097 >> v1:172.21.15.27:6789/0 conn(0x7ffb3c1596f0 legacy=0x7ffb3c0a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.329+0000 7ffb3b7fe700 1 -- 172.21.15.27:0/1353580097 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb3c0c7490 con 0x7ffb3c159ac0 2024-01-11T08:22:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.329+0000 7ffb3affd700 1 -- 172.21.15.27:0/1353580097 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101596947 0 0) 0x7ffb3c0c80a0 con 0x7ffb3c1596f0 2024-01-11T08:22:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.330+0000 7ffb3b7fe700 1 --2- 172.21.15.27:0/1353580097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb3c159ac0 0x7ffb3c068620 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c0cc960 tx=0x7ffb30009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a506e4633adf6415 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.330+0000 7ffb3affd700 1 -- 172.21.15.27:0/1353580097 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30009e70 con 0x7ffb3c159ac0 2024-01-11T08:22:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.330+0000 7ffb3affd700 1 -- 172.21.15.27:0/1353580097 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3000b860 con 0x7ffb3c159ac0 2024-01-11T08:22:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.330+0000 7ffb3affd700 1 -- 172.21.15.27:0/1353580097 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3000bc00 con 0x7ffb3c159ac0 2024-01-11T08:22:30.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.330+0000 7ffb427ff700 1 -- 172.21.15.27:0/1353580097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb3c159ac0 msgr2=0x7ffb3c068620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:30.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.330+0000 7ffb427ff700 1 --2- 172.21.15.27:0/1353580097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb3c159ac0 0x7ffb3c068620 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c0cc960 tx=0x7ffb30009250 comp rx=0 tx=0).stop 2024-01-11T08:22:30.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.330+0000 7ffb427ff700 1 -- 172.21.15.27:0/1353580097 shutdown_connections 2024-01-11T08:22:30.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.330+0000 7ffb427ff700 1 --2- 172.21.15.27:0/1353580097 >> v2:172.21.15.77:3300/0 conn(0x7ffb3c0ca640 0x7ffb3c068b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.330+0000 7ffb427ff700 1 --2- 172.21.15.27:0/1353580097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb3c159ac0 0x7ffb3c068620 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.330+0000 7ffb427ff700 1 -- 172.21.15.27:0/1353580097 >> 172.21.15.27:0/1353580097 conn(0x7ffb3c0c9fb0 msgr2=0x7ffb3c06ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:30.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.331+0000 7ffb427ff700 1 -- 172.21.15.27:0/1353580097 shutdown_connections 2024-01-11T08:22:30.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.331+0000 7ffb427ff700 1 -- 172.21.15.27:0/1353580097 wait complete. 2024-01-11T08:22:30.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.332+0000 7ffb427ff700 1 Processor -- start 2024-01-11T08:22:30.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.332+0000 7ffb427ff700 1 -- start start 2024-01-11T08:22:30.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.333+0000 7ffb427ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb3c0ca640 0x7ffb3c0af0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:30.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.333+0000 7ffb427ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb3c1596f0 0x7ffb3c14c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:30.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.333+0000 7ffb427ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ffb3c159ac0 0x7ffb3c06bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:30.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.333+0000 7ffb427ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c0c4af0 con 0x7ffb3c0ca640 2024-01-11T08:22:30.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.333+0000 7ffb427ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c0c5950 con 0x7ffb3c1596f0 2024-01-11T08:22:30.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.333+0000 7ffb427ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c0adc30 con 0x7ffb3c159ac0 2024-01-11T08:22:30.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.334+0000 7ffb3bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb3c0ca640 0x7ffb3c0af0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:30.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.334+0000 7ffb3bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb3c0ca640 0x7ffb3c0af0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51936/0 (socket says 172.21.15.27:51936) 2024-01-11T08:22:30.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.334+0000 7ffb3bfff700 1 -- 172.21.15.27:0/3290895682 learned_addr learned my addr 172.21.15.27:0/3290895682 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:30.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.334+0000 7ffb3b7fe700 1 --2- 172.21.15.27:0/3290895682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb3c1596f0 0x7ffb3c14c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:30.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.334+0000 7ffb40d9c700 1 --2- 172.21.15.27:0/3290895682 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ffb3c159ac0 0x7ffb3c06bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.334+0000 7ffb3bfff700 1 -- 172.21.15.27:0/3290895682 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ffb3c159ac0 msgr2=0x7ffb3c06bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.334+0000 7ffb3bfff700 1 --2- 172.21.15.27:0/3290895682 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ffb3c159ac0 0x7ffb3c06bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.334+0000 7ffb3bfff700 1 -- 172.21.15.27:0/3290895682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb3c1596f0 msgr2=0x7ffb3c14c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.334+0000 7ffb3bfff700 1 --2- 172.21.15.27:0/3290895682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb3c1596f0 0x7ffb3c14c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.334+0000 7ffb3bfff700 1 -- 172.21.15.27:0/3290895682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb300094f0 con 0x7ffb3c0ca640 2024-01-11T08:22:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.334+0000 7ffb3bfff700 1 --2- 172.21.15.27:0/3290895682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb3c0ca640 0x7ffb3c0af0b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c0011e0 tx=0x7ffb2c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.335+0000 7ffb23fff700 1 -- 172.21.15.27:0/3290895682 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2c005a70 con 0x7ffb3c0ca640 2024-01-11T08:22:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.335+0000 7ffb23fff700 1 -- 172.21.15.27:0/3290895682 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb2c005bd0 con 0x7ffb3c0ca640 2024-01-11T08:22:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.335+0000 7ffb427ff700 1 -- 172.21.15.27:0/3290895682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb3c0c5de0 con 0x7ffb3c0ca640 2024-01-11T08:22:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.335+0000 7ffb23fff700 1 -- 172.21.15.27:0/3290895682 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2c0070d0 con 0x7ffb3c0ca640 2024-01-11T08:22:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.335+0000 7ffb427ff700 1 -- 172.21.15.27:0/3290895682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb3c0c5020 con 0x7ffb3c0ca640 2024-01-11T08:22:30.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.336+0000 7ffb427ff700 1 -- 172.21.15.27:0/3290895682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb3c0c5c30 con 0x7ffb3c0ca640 2024-01-11T08:22:30.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.337+0000 7ffb23fff700 1 -- 172.21.15.27:0/3290895682 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7ffb2c007480 con 0x7ffb3c0ca640 2024-01-11T08:22:30.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.337+0000 7ffb23fff700 1 --2- 172.21.15.27:0/3290895682 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ffb28067130 0x7ffb280695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:30.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.338+0000 7ffb23fff700 1 -- 172.21.15.27:0/3290895682 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7ffb2c083f10 con 0x7ffb3c0ca640 2024-01-11T08:22:30.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.338+0000 7ffb3b7fe700 1 --2- 172.21.15.27:0/3290895682 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ffb28067130 0x7ffb280695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:30.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.339+0000 7ffb3b7fe700 1 --2- 172.21.15.27:0/3290895682 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ffb28067130 0x7ffb280695e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c0cc960 tx=0x7ffb30017000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:30.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.341+0000 7ffb23fff700 1 -- 172.21.15.27:0/3290895682 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffb2c050d60 con 0x7ffb3c0ca640 2024-01-11T08:22:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.569+0000 7ffb427ff700 1 -- 172.21.15.27:0/3290895682 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffb3c04c3c0 con 0x7ffb28067130 2024-01-11T08:22:30.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.578+0000 7ffb23fff700 1 -- 172.21.15.27:0/3290895682 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7ffb3c04c3c0 con 0x7ffb28067130 2024-01-11T08:22:30.578 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:22:30.578 INFO:teuthology.orchestra.run.smithi027.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:46", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.580+0000 7ffb427ff700 1 -- 172.21.15.27:0/3290895682 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ffb28067130 msgr2=0x7ffb280695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.580+0000 7ffb427ff700 1 --2- 172.21.15.27:0/3290895682 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ffb28067130 0x7ffb280695e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c0cc960 tx=0x7ffb30017000 comp rx=0 tx=0).stop 2024-01-11T08:22:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.580+0000 7ffb427ff700 1 -- 172.21.15.27:0/3290895682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb3c0ca640 msgr2=0x7ffb3c0af0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.580+0000 7ffb427ff700 1 --2- 172.21.15.27:0/3290895682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb3c0ca640 0x7ffb3c0af0b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c0011e0 tx=0x7ffb2c007e10 comp rx=0 tx=0).stop 2024-01-11T08:22:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.581+0000 7ffb427ff700 1 -- 172.21.15.27:0/3290895682 shutdown_connections 2024-01-11T08:22:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.581+0000 7ffb427ff700 1 --2- 172.21.15.27:0/3290895682 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7ffb28067130 0x7ffb280695e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.581+0000 7ffb427ff700 1 --2- 172.21.15.27:0/3290895682 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ffb3c159ac0 0x7ffb3c06bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.581+0000 7ffb427ff700 1 --2- 172.21.15.27:0/3290895682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb3c0ca640 0x7ffb3c0af0b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.581+0000 7ffb427ff700 1 --2- 172.21.15.27:0/3290895682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb3c1596f0 0x7ffb3c14c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.581+0000 7ffb427ff700 1 -- 172.21.15.27:0/3290895682 >> 172.21.15.27:0/3290895682 conn(0x7ffb3c0c9fb0 msgr2=0x7ffb3c0c3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:30.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.581+0000 7ffb427ff700 1 -- 172.21.15.27:0/3290895682 shutdown_connections 2024-01-11T08:22:30.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.581+0000 7ffb427ff700 1 -- 172.21.15.27:0/3290895682 wait complete. 2024-01-11T08:22:30.597 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:46", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:30.598 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1m', 'retention': {}, 'start': '2024-01-11T00:00:00', 'created': '2024-01-11T08:11:46', 'first': '2024-01-11T08:12:00', 'last': '2024-01-11T08:22:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-11T08:22:30.598 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-11T08:22:30.598 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-11T08:22:30.598 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:30.598 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2024-01-11T08:22:30.646 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:30.646 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:30.646 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:30.646 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:30.647 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:30.647 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:30.647 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:30.647 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:30.647 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:30.647 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:30.647 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:30.648 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-11-08_12_00_UTC', 'scheduled-2024-01-11-08_13_00_UTC', 'scheduled-2024-01-11-08_14_00_UTC', 'scheduled-2024-01-11-08_15_00_UTC', 'scheduled-2024-01-11-08_16_00_UTC', 'scheduled-2024-01-11-08_17_00_UTC', 'scheduled-2024-01-11-08_18_00_UTC', 'scheduled-2024-01-11-08_19_00_UTC', 'scheduled-2024-01-11-08_20_00_UTC', 'scheduled-2024-01-11-08_21_00_UTC', 'scheduled-2024-01-11-08_22_00_UTC'] 2024-01-11T08:22:30.648 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir9 2024-01-11T08:22:30.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.852+0000 7f76d7fff700 1 Processor -- start 2024-01-11T08:22:30.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.852+0000 7f76d7fff700 1 -- start start 2024-01-11T08:22:30.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.854+0000 7f76d7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76d00687c0 0x7f76d00b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:30.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.854+0000 7f76d7fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76d015dff0 0x7f76d01591d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:30.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.854+0000 7f76d7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f76d00b2a00 0x7f76d0159710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:30.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.854+0000 7f76d7fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d00cbfe0 con 0x7f76d00687c0 2024-01-11T08:22:30.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.854+0000 7f76d7fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d0159c50 con 0x7f76d00b2a00 2024-01-11T08:22:30.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.854+0000 7f76d7fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d00cb3d0 con 0x7f76d015dff0 2024-01-11T08:22:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.854+0000 7f76d5d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76d00687c0 0x7f76d00b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.854+0000 7f76d5d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76d00687c0 0x7f76d00b24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51948/0 (socket says 172.21.15.27:51948) 2024-01-11T08:22:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.854+0000 7f76d5d9b700 1 -- 172.21.15.27:0/421045587 learned_addr learned my addr 172.21.15.27:0/421045587 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.854+0000 7f76d559a700 1 --2- 172.21.15.27:0/421045587 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76d015dff0 0x7f76d01591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.854+0000 7f76d659c700 1 --2- 172.21.15.27:0/421045587 >> v2:172.21.15.77:3300/0 conn(0x7f76d00b2a00 0x7f76d0159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.855+0000 7f76d5d9b700 1 -- 172.21.15.27:0/421045587 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76d015dff0 msgr2=0x7f76d01591d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.855+0000 7f76d5d9b700 1 --2- 172.21.15.27:0/421045587 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76d015dff0 0x7f76d01591d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.855+0000 7f76d5d9b700 1 -- 172.21.15.27:0/421045587 >> v2:172.21.15.77:3300/0 conn(0x7f76d00b2a00 msgr2=0x7f76d0159710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.855+0000 7f76d5d9b700 1 --2- 172.21.15.27:0/421045587 >> v2:172.21.15.77:3300/0 conn(0x7f76d00b2a00 0x7f76d0159710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.855+0000 7f76d5d9b700 1 -- 172.21.15.27:0/421045587 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d00cb6b0 con 0x7f76d00687c0 2024-01-11T08:22:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.855+0000 7f76d559a700 1 --2- 172.21.15.27:0/421045587 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76d015dff0 0x7f76d01591d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:22:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.855+0000 7f76d659c700 1 --2- 172.21.15.27:0/421045587 >> v2:172.21.15.77:3300/0 conn(0x7f76d00b2a00 0x7f76d0159710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:22:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.855+0000 7f76d5d9b700 1 --2- 172.21.15.27:0/421045587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76d00687c0 0x7f76d00b24c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f76d00cbf50 tx=0x7f76cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc538293ac0f1be server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:30.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.855+0000 7f76d4d99700 1 -- 172.21.15.27:0/421045587 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc00b860 con 0x7f76d00687c0 2024-01-11T08:22:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.855+0000 7f76d4d99700 1 -- 172.21.15.27:0/421045587 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76cc00b9c0 con 0x7f76d00687c0 2024-01-11T08:22:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.855+0000 7f76d4d99700 1 -- 172.21.15.27:0/421045587 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc00bd60 con 0x7f76d00687c0 2024-01-11T08:22:30.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.856+0000 7f76d7fff700 1 -- 172.21.15.27:0/421045587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76d00687c0 msgr2=0x7f76d00b24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.856+0000 7f76d7fff700 1 --2- 172.21.15.27:0/421045587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76d00687c0 0x7f76d00b24c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f76d00cbf50 tx=0x7f76cc009250 comp rx=0 tx=0).stop 2024-01-11T08:22:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.856+0000 7f76d7fff700 1 -- 172.21.15.27:0/421045587 shutdown_connections 2024-01-11T08:22:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.856+0000 7f76d7fff700 1 --2- 172.21.15.27:0/421045587 >> v2:172.21.15.77:3300/0 conn(0x7f76d00b2a00 0x7f76d0159710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.856+0000 7f76d7fff700 1 --2- 172.21.15.27:0/421045587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76d00687c0 0x7f76d00b24c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.856+0000 7f76d7fff700 1 --2- 172.21.15.27:0/421045587 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76d015dff0 0x7f76d01591d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.856+0000 7f76d7fff700 1 -- 172.21.15.27:0/421045587 >> 172.21.15.27:0/421045587 conn(0x7f76d006ed40 msgr2=0x7f76d006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.856+0000 7f76d7fff700 1 -- 172.21.15.27:0/421045587 shutdown_connections 2024-01-11T08:22:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.856+0000 7f76d7fff700 1 -- 172.21.15.27:0/421045587 wait complete. 2024-01-11T08:22:30.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.858+0000 7f76d7fff700 1 Processor -- start 2024-01-11T08:22:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.858+0000 7f76d7fff700 1 -- start start 2024-01-11T08:22:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.859+0000 7f76d7fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76d00687c0 0x7f76d00cc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.859+0000 7f76d7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76d00b2a00 0x7f76d00cc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.859+0000 7f76d7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76d015dff0 0x7f76d016ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.859+0000 7f76d7fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d00c94c0 con 0x7f76d00b2a00 2024-01-11T08:22:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.860+0000 7f76d7fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d016d780 con 0x7f76d015dff0 2024-01-11T08:22:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.860+0000 7f76d7fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d00c96d0 con 0x7f76d00687c0 2024-01-11T08:22:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.860+0000 7f76d559a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76d00b2a00 0x7f76d00cc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.860+0000 7f76d559a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76d00b2a00 0x7f76d00cc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51958/0 (socket says 172.21.15.27:51958) 2024-01-11T08:22:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.860+0000 7f76d559a700 1 -- 172.21.15.27:0/241105195 learned_addr learned my addr 172.21.15.27:0/241105195 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.860+0000 7f76d5d9b700 1 --2- 172.21.15.27:0/241105195 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76d00687c0 0x7f76d00cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.860+0000 7f76d659c700 1 --2- 172.21.15.27:0/241105195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76d015dff0 0x7f76d016ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.860+0000 7f76d559a700 1 -- 172.21.15.27:0/241105195 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76d00687c0 msgr2=0x7f76d00cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.860+0000 7f76d559a700 1 --2- 172.21.15.27:0/241105195 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76d00687c0 0x7f76d00cc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.860+0000 7f76d559a700 1 -- 172.21.15.27:0/241105195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76d015dff0 msgr2=0x7f76d016ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.860+0000 7f76d559a700 1 --2- 172.21.15.27:0/241105195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76d015dff0 0x7f76d016ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.860+0000 7f76d559a700 1 -- 172.21.15.27:0/241105195 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc0094f0 con 0x7f76d00b2a00 2024-01-11T08:22:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.860+0000 7f76d559a700 1 --2- 172.21.15.27:0/241105195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76d00b2a00 0x7f76d00cc9b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f76c000c800 tx=0x7f76c0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.861+0000 7f76c5ffb700 1 -- 172.21.15.27:0/241105195 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c0011600 con 0x7f76d00b2a00 2024-01-11T08:22:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.861+0000 7f76c5ffb700 1 -- 172.21.15.27:0/241105195 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c0011760 con 0x7f76d00b2a00 2024-01-11T08:22:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.861+0000 7f76c5ffb700 1 -- 172.21.15.27:0/241105195 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c0011ae0 con 0x7f76d00b2a00 2024-01-11T08:22:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.861+0000 7f76d7fff700 1 -- 172.21.15.27:0/241105195 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d00c99b0 con 0x7f76d00b2a00 2024-01-11T08:22:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.861+0000 7f76d7fff700 1 -- 172.21.15.27:0/241105195 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d00ca1b0 con 0x7f76d00b2a00 2024-01-11T08:22:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.862+0000 7f76d7fff700 1 -- 172.21.15.27:0/241105195 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d00c9ed0 con 0x7f76d00b2a00 2024-01-11T08:22:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.863+0000 7f76c5ffb700 1 -- 172.21.15.27:0/241105195 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f76c0010350 con 0x7f76d00b2a00 2024-01-11T08:22:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.864+0000 7f76c5ffb700 1 --2- 172.21.15.27:0/241105195 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f76b4067130 0x7f76b40695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.864+0000 7f76c5ffb700 1 -- 172.21.15.27:0/241105195 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f76c0084a00 con 0x7f76d00b2a00 2024-01-11T08:22:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.864+0000 7f76d5d9b700 1 --2- 172.21.15.27:0/241105195 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f76b4067130 0x7f76b40695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:30.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.865+0000 7f76d5d9b700 1 --2- 172.21.15.27:0/241105195 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f76b4067130 0x7f76b40695e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f76d00cbf50 tx=0x7f76cc017000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:30.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:30.868+0000 7f76c5ffb700 1 -- 172.21.15.27:0/241105195 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f76d00c9ed0 con 0x7f76d00b2a00 2024-01-11T08:22:31.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.095+0000 7f76d7fff700 1 -- 172.21.15.27:0/241105195 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f76d00c9ed0 con 0x7f76b4067130 2024-01-11T08:22:31.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.104+0000 7f76c5ffb700 1 -- 172.21.15.27:0/241105195 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f76d00c9ed0 con 0x7f76b4067130 2024-01-11T08:22:31.104 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:22:31.104 INFO:teuthology.orchestra.run.smithi027.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:47", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:31.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.107+0000 7f76d7fff700 1 -- 172.21.15.27:0/241105195 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f76b4067130 msgr2=0x7f76b40695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:31.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.107+0000 7f76d7fff700 1 --2- 172.21.15.27:0/241105195 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f76b4067130 0x7f76b40695e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f76d00cbf50 tx=0x7f76cc017000 comp rx=0 tx=0).stop 2024-01-11T08:22:31.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.107+0000 7f76d7fff700 1 -- 172.21.15.27:0/241105195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76d00b2a00 msgr2=0x7f76d00cc9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:31.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.107+0000 7f76d7fff700 1 --2- 172.21.15.27:0/241105195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76d00b2a00 0x7f76d00cc9b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f76c000c800 tx=0x7f76c0008040 comp rx=0 tx=0).stop 2024-01-11T08:22:31.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.107+0000 7f76d7fff700 1 -- 172.21.15.27:0/241105195 shutdown_connections 2024-01-11T08:22:31.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.107+0000 7f76d7fff700 1 --2- 172.21.15.27:0/241105195 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f76b4067130 0x7f76b40695e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:31.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.107+0000 7f76d7fff700 1 --2- 172.21.15.27:0/241105195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76d015dff0 0x7f76d016ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:31.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.107+0000 7f76d7fff700 1 --2- 172.21.15.27:0/241105195 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76d00687c0 0x7f76d00cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:31.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.107+0000 7f76d7fff700 1 --2- 172.21.15.27:0/241105195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76d00b2a00 0x7f76d00cc9b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:31.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.107+0000 7f76d7fff700 1 -- 172.21.15.27:0/241105195 >> 172.21.15.27:0/241105195 conn(0x7f76d006ed40 msgr2=0x7f76d00cfd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:31.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.107+0000 7f76d7fff700 1 -- 172.21.15.27:0/241105195 shutdown_connections 2024-01-11T08:22:31.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.107+0000 7f76d7fff700 1 -- 172.21.15.27:0/241105195 wait complete. 2024-01-11T08:22:31.124 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:11:47", "first": "2024-01-11T08:12:00", "last": "2024-01-11T08:22:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-01-11T08:22:31.124 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1m', 'retention': {}, 'start': '2024-01-11T00:00:00', 'created': '2024-01-11T08:11:47', 'first': '2024-01-11T08:12:00', 'last': '2024-01-11T08:22:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-01-11T08:22:31.124 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-11T08:22:31.124 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-11T08:22:31.124 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir0 1m 2024-01-11T08:22:31.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.326+0000 7fd7559ff700 1 Processor -- start 2024-01-11T08:22:31.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.327+0000 7fd7559ff700 1 -- start start 2024-01-11T08:22:31.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.328+0000 7fd7559ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd75013fbb0 0x7fd7500ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:31.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.328+0000 7fd7559ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd75015a730 0x7fd7500c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:31.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.328+0000 7fd7559ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7500c6b30 con 0x7fd75013ff80 2024-01-11T08:22:31.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.328+0000 7fd7559ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7500c6800 con 0x7fd75015a730 2024-01-11T08:22:31.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.328+0000 7fd7559ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7500cb240 con 0x7fd75013fbb0 2024-01-11T08:22:31.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.328+0000 7fd74f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd75015a730 0x7fd7500c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:31.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.328+0000 7fd74f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd75015a730 0x7fd7500c6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51972/0 (socket says 172.21.15.27:51972) 2024-01-11T08:22:31.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.328+0000 7fd74f7fe700 1 -- 172.21.15.27:0/253255589 learned_addr learned my addr 172.21.15.27:0/253255589 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:31.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.329+0000 7fd74effd700 1 --2- 172.21.15.27:0/253255589 >> v2:172.21.15.77:3300/0 conn(0x7fd75013fbb0 0x7fd7500ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:31.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.329+0000 7fd74f7fe700 1 -- 172.21.15.27:0/253255589 >> v2:172.21.15.77:3300/0 conn(0x7fd75013fbb0 msgr2=0x7fd7500ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:31.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.329+0000 7fd74f7fe700 1 --2- 172.21.15.27:0/253255589 >> v2:172.21.15.77:3300/0 conn(0x7fd75013fbb0 0x7fd7500ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:31.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.329+0000 7fd74f7fe700 1 -- 172.21.15.27:0/253255589 >> v1:172.21.15.77:6789/0 conn(0x7fd75013ff80 legacy=0x7fd75015a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:31.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.329+0000 7fd74f7fe700 1 -- 172.21.15.27:0/253255589 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7500cb520 con 0x7fd75015a730 2024-01-11T08:22:31.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.329+0000 7fd74effd700 1 --2- 172.21.15.27:0/253255589 >> v2:172.21.15.77:3300/0 conn(0x7fd75013fbb0 0x7fd7500ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:22:31.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.329+0000 7fd74f7fe700 1 --2- 172.21.15.27:0/253255589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd75015a730 0x7fd7500c6fc0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd750064df0 tx=0x7fd748009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3aee09f0d9a427 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:31.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.329+0000 7fd74dffb700 1 -- 172.21.15.27:0/253255589 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74800e070 con 0x7fd75015a730 2024-01-11T08:22:31.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.329+0000 7fd74dffb700 1 -- 172.21.15.27:0/253255589 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74800ba50 con 0x7fd75015a730 2024-01-11T08:22:31.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.329+0000 7fd74dffb700 1 -- 172.21.15.27:0/253255589 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7480092e0 con 0x7fd75015a730 2024-01-11T08:22:31.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.330+0000 7fd7559ff700 1 -- 172.21.15.27:0/253255589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd75015a730 msgr2=0x7fd7500c6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:31.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.330+0000 7fd7559ff700 1 --2- 172.21.15.27:0/253255589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd75015a730 0x7fd7500c6fc0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd750064df0 tx=0x7fd748009840 comp rx=0 tx=0).stop 2024-01-11T08:22:31.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.330+0000 7fd7559ff700 1 -- 172.21.15.27:0/253255589 shutdown_connections 2024-01-11T08:22:31.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.330+0000 7fd7559ff700 1 --2- 172.21.15.27:0/253255589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd75015a730 0x7fd7500c6fc0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:31.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.330+0000 7fd7559ff700 1 --2- 172.21.15.27:0/253255589 >> v2:172.21.15.77:3300/0 conn(0x7fd75013fbb0 0x7fd7500ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:31.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.330+0000 7fd7559ff700 1 -- 172.21.15.27:0/253255589 >> 172.21.15.27:0/253255589 conn(0x7fd7500680a0 msgr2=0x7fd7500cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:31.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.330+0000 7fd7559ff700 1 -- 172.21.15.27:0/253255589 shutdown_connections 2024-01-11T08:22:31.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.330+0000 7fd7559ff700 1 -- 172.21.15.27:0/253255589 wait complete. 2024-01-11T08:22:31.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.332+0000 7fd7559ff700 1 Processor -- start 2024-01-11T08:22:31.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.332+0000 7fd7559ff700 1 -- start start 2024-01-11T08:22:31.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.333+0000 7fd7559ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd75013fbb0 0x7fd7500c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:31.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.333+0000 7fd7559ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd75013ff80 0x7fd7500c8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:31.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.333+0000 7fd7559ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd75015a730 0x7fd750158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:31.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.333+0000 7fd7559ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75016a8c0 con 0x7fd75013fbb0 2024-01-11T08:22:31.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.333+0000 7fd7559ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd750169cf0 con 0x7fd75015a730 2024-01-11T08:22:31.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.333+0000 7fd7559ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd750169f90 con 0x7fd75013ff80 2024-01-11T08:22:31.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.333+0000 7fd74effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd75013fbb0 0x7fd7500c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:31.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.333+0000 7fd74effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd75013fbb0 0x7fd7500c7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51982/0 (socket says 172.21.15.27:51982) 2024-01-11T08:22:31.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.333+0000 7fd74effd700 1 -- 172.21.15.27:0/1363224038 learned_addr learned my addr 172.21.15.27:0/1363224038 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:31.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.334+0000 7fd74f7fe700 1 --2- 172.21.15.27:0/1363224038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd75015a730 0x7fd750158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:31.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.334+0000 7fd74effd700 1 -- 172.21.15.27:0/1363224038 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd75013ff80 msgr2=0x7fd7500c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:31.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.334+0000 7fd74effd700 1 --2- 172.21.15.27:0/1363224038 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd75013ff80 0x7fd7500c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:31.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.334+0000 7fd74effd700 1 -- 172.21.15.27:0/1363224038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd75015a730 msgr2=0x7fd750158f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:31.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.334+0000 7fd74effd700 1 --2- 172.21.15.27:0/1363224038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd75015a730 0x7fd750158f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:31.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.334+0000 7fd74effd700 1 -- 172.21.15.27:0/1363224038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd748009ae0 con 0x7fd75013fbb0 2024-01-11T08:22:31.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.334+0000 7fd74effd700 1 --2- 172.21.15.27:0/1363224038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd75013fbb0 0x7fd7500c7dd0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd74000e920 tx=0x7fd740008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:31.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.334+0000 7fd736ffd700 1 -- 172.21.15.27:0/1363224038 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd740005a00 con 0x7fd75013fbb0 2024-01-11T08:22:31.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.334+0000 7fd736ffd700 1 -- 172.21.15.27:0/1363224038 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd740005b60 con 0x7fd75013fbb0 2024-01-11T08:22:31.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.334+0000 7fd736ffd700 1 -- 172.21.15.27:0/1363224038 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd740006020 con 0x7fd75013fbb0 2024-01-11T08:22:31.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.334+0000 7fd7559ff700 1 -- 172.21.15.27:0/1363224038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd75016a270 con 0x7fd75013fbb0 2024-01-11T08:22:31.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.334+0000 7fd7559ff700 1 -- 172.21.15.27:0/1363224038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd750159480 con 0x7fd75013fbb0 2024-01-11T08:22:31.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.335+0000 7fd7559ff700 1 -- 172.21.15.27:0/1363224038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd75004c3c0 con 0x7fd75013fbb0 2024-01-11T08:22:31.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.336+0000 7fd736ffd700 1 -- 172.21.15.27:0/1363224038 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fd740006410 con 0x7fd75013fbb0 2024-01-11T08:22:31.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.337+0000 7fd736ffd700 1 --2- 172.21.15.27:0/1363224038 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd73c067130 0x7fd73c0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:31.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.338+0000 7fd736ffd700 1 -- 172.21.15.27:0/1363224038 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fd740084ca0 con 0x7fd75013fbb0 2024-01-11T08:22:31.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.338+0000 7fd74e7fc700 1 --2- 172.21.15.27:0/1363224038 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd73c067130 0x7fd73c0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:31.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.339+0000 7fd74e7fc700 1 --2- 172.21.15.27:0/1363224038 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd73c067130 0x7fd73c0695e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd75016d0b0 tx=0x7fd744007b40 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:31.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.341+0000 7fd736ffd700 1 -- 172.21.15.27:0/1363224038 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd7400519e0 con 0x7fd75013fbb0 2024-01-11T08:22:31.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.571+0000 7fd7559ff700 1 -- 172.21.15.27:0/1363224038 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7500c8850 con 0x7fd73c067130 2024-01-11T08:22:31.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.663+0000 7fd736ffd700 1 -- 172.21.15.27:0/1363224038 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fd7500c8850 con 0x7fd73c067130 2024-01-11T08:22:31.663 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule removed for path /snap_test_dir1/dir0 2024-01-11T08:22:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.666+0000 7fd7559ff700 1 -- 172.21.15.27:0/1363224038 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd73c067130 msgr2=0x7fd73c0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.666+0000 7fd7559ff700 1 --2- 172.21.15.27:0/1363224038 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd73c067130 0x7fd73c0695e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd75016d0b0 tx=0x7fd744007b40 comp rx=0 tx=0).stop 2024-01-11T08:22:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.666+0000 7fd7559ff700 1 -- 172.21.15.27:0/1363224038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd75013fbb0 msgr2=0x7fd7500c7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.666+0000 7fd7559ff700 1 --2- 172.21.15.27:0/1363224038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd75013fbb0 0x7fd7500c7dd0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd74000e920 tx=0x7fd740008040 comp rx=0 tx=0).stop 2024-01-11T08:22:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.666+0000 7fd7559ff700 1 -- 172.21.15.27:0/1363224038 shutdown_connections 2024-01-11T08:22:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.666+0000 7fd7559ff700 1 --2- 172.21.15.27:0/1363224038 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd73c067130 0x7fd73c0695e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.666+0000 7fd7559ff700 1 --2- 172.21.15.27:0/1363224038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd75015a730 0x7fd750158f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.666+0000 7fd7559ff700 1 --2- 172.21.15.27:0/1363224038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd75013fbb0 0x7fd7500c7dd0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.666+0000 7fd7559ff700 1 --2- 172.21.15.27:0/1363224038 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd75013ff80 0x7fd7500c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.666+0000 7fd7559ff700 1 -- 172.21.15.27:0/1363224038 >> 172.21.15.27:0/1363224038 conn(0x7fd7500680a0 msgr2=0x7fd7500c6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.666+0000 7fd7559ff700 1 -- 172.21.15.27:0/1363224038 shutdown_connections 2024-01-11T08:22:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:31.666+0000 7fd7559ff700 1 -- 172.21.15.27:0/1363224038 wait complete. 2024-01-11T08:22:31.683 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2024-01-11T08:22:31.684 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:31.684 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2024-01-11T08:22:31.732 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:31.732 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:31.732 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:31.732 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:31.732 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:31.732 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:31.732 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:31.732 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:31.733 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:31.733 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:31.733 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:31.734 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:31.734 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:31.734 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_12_00_UTC) 2024-01-11T08:22:31.790 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:31.790 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:31.790 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_13_00_UTC) 2024-01-11T08:22:31.845 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:31.845 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:31.845 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_14_00_UTC) 2024-01-11T08:22:31.899 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:31.899 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:31.899 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_15_00_UTC) 2024-01-11T08:22:31.953 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:31.954 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:31.954 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_16_00_UTC) 2024-01-11T08:22:32.009 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:32.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:32.010 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_17_00_UTC) 2024-01-11T08:22:32.067 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:32.067 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:32.067 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_18_00_UTC) 2024-01-11T08:22:32.128 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:32.128 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:32.128 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_19_00_UTC) 2024-01-11T08:22:32.183 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:32.183 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:32.183 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_20_00_UTC) 2024-01-11T08:22:32.238 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:32.238 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:32.238 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_21_00_UTC) 2024-01-11T08:22:32.293 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:32.293 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:32.293 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-01-11-08_22_00_UTC) 2024-01-11T08:22:32.348 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:32.349 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2024-01-11T08:22:32.400 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir1 1m 2024-01-11T08:22:32.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.603+0000 7f9ca01ff700 1 Processor -- start 2024-01-11T08:22:32.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.603+0000 7f9ca01ff700 1 -- start start 2024-01-11T08:22:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.604+0000 7f9ca01ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c9813fbb0 0x7f9c980ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.604+0000 7f9ca01ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c980c6b30 con 0x7f9c9813ff80 2024-01-11T08:22:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.604+0000 7f9ca01ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c980c6800 con 0x7f9c9815a730 2024-01-11T08:22:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.604+0000 7f9ca01ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c980cb240 con 0x7f9c9813fbb0 2024-01-11T08:22:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.605+0000 7f9c9d79a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9c9813ff80 0x7f9c9815a020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54484/0 (socket says 172.21.15.27:54484) 2024-01-11T08:22:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.605+0000 7f9c9d79a700 1 -- 172.21.15.27:0/1907949968 learned_addr learned my addr 172.21.15.27:0/1907949968 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.605+0000 7f9c9df9b700 1 --2- 172.21.15.27:0/1907949968 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c9813fbb0 0x7f9c980ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.605+0000 7f9c9cf99700 1 -- 172.21.15.27:0/1907949968 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727541087 0 0) 0x7f9c980c6b30 con 0x7f9c9813ff80 2024-01-11T08:22:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.605+0000 7f9c9cf99700 1 -- 172.21.15.27:0/1907949968 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c84003570 con 0x7f9c9813ff80 2024-01-11T08:22:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.605+0000 7f9c9cf99700 1 -- 172.21.15.27:0/1907949968 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627607161 0 0) 0x7f9c980c6800 con 0x7f9c9815a730 2024-01-11T08:22:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.605+0000 7f9c9cf99700 1 -- 172.21.15.27:0/1907949968 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c980c6b30 con 0x7f9c9815a730 2024-01-11T08:22:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.605+0000 7f9c9df9b700 1 -- 172.21.15.27:0/1907949968 >> v1:172.21.15.77:6789/0 conn(0x7f9c9815a730 legacy=0x7f9c980c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.605+0000 7f9c9df9b700 1 -- 172.21.15.27:0/1907949968 >> v1:172.21.15.27:6789/0 conn(0x7f9c9813ff80 legacy=0x7f9c9815a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.606+0000 7f9c9df9b700 1 -- 172.21.15.27:0/1907949968 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c980cb520 con 0x7f9c9813fbb0 2024-01-11T08:22:32.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.606+0000 7f9c9df9b700 1 --2- 172.21.15.27:0/1907949968 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c9813fbb0 0x7f9c980ca7a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9c94009900 tx=0x7f9c94009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc49133791f8a8a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.606+0000 7f9c9cf99700 1 -- 172.21.15.27:0/1907949968 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c94005c10 con 0x7f9c9813fbb0 2024-01-11T08:22:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.606+0000 7f9c9cf99700 1 -- 172.21.15.27:0/1907949968 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c94005d70 con 0x7f9c9813fbb0 2024-01-11T08:22:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.606+0000 7f9c9cf99700 1 -- 172.21.15.27:0/1907949968 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c94006260 con 0x7f9c9813fbb0 2024-01-11T08:22:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.607+0000 7f9ca01ff700 1 -- 172.21.15.27:0/1907949968 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c9813fbb0 msgr2=0x7f9c980ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.607+0000 7f9ca01ff700 1 --2- 172.21.15.27:0/1907949968 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c9813fbb0 0x7f9c980ca7a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9c94009900 tx=0x7f9c94009120 comp rx=0 tx=0).stop 2024-01-11T08:22:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.607+0000 7f9ca01ff700 1 -- 172.21.15.27:0/1907949968 shutdown_connections 2024-01-11T08:22:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.607+0000 7f9ca01ff700 1 --2- 172.21.15.27:0/1907949968 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c9813fbb0 0x7f9c980ca7a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.607+0000 7f9ca01ff700 1 -- 172.21.15.27:0/1907949968 >> 172.21.15.27:0/1907949968 conn(0x7f9c980680a0 msgr2=0x7f9c980cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.607+0000 7f9ca01ff700 1 -- 172.21.15.27:0/1907949968 shutdown_connections 2024-01-11T08:22:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.607+0000 7f9ca01ff700 1 -- 172.21.15.27:0/1907949968 wait complete. 2024-01-11T08:22:32.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.608+0000 7f9ca01ff700 1 Processor -- start 2024-01-11T08:22:32.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.609+0000 7f9ca01ff700 1 -- start start 2024-01-11T08:22:32.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.609+0000 7f9ca01ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c9813fbb0 0x7f9c98155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.609+0000 7f9ca01ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c9813ff80 0x7f9c980c22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.609+0000 7f9ca01ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c9815a730 0x7f9c980c2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.609+0000 7f9ca01ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9813d150 con 0x7f9c9813ff80 2024-01-11T08:22:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.609+0000 7f9ca01ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c980b16d0 con 0x7f9c9813fbb0 2024-01-11T08:22:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.609+0000 7f9ca01ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c98156a90 con 0x7f9c9815a730 2024-01-11T08:22:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.610+0000 7f9c9d79a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c9813ff80 0x7f9c980c22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.610+0000 7f9c9d79a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c9813ff80 0x7f9c980c22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51988/0 (socket says 172.21.15.27:51988) 2024-01-11T08:22:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.610+0000 7f9c9d79a700 1 -- 172.21.15.27:0/806150856 learned_addr learned my addr 172.21.15.27:0/806150856 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.610+0000 7f9c9d79a700 1 -- 172.21.15.27:0/806150856 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c9815a730 msgr2=0x7f9c980c2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.610+0000 7f9c9df9b700 1 --2- 172.21.15.27:0/806150856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c9813fbb0 0x7f9c98155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.610+0000 7f9c9e79c700 1 --2- 172.21.15.27:0/806150856 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c9815a730 0x7f9c980c2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.610+0000 7f9c9d79a700 1 --2- 172.21.15.27:0/806150856 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c9815a730 0x7f9c980c2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.610+0000 7f9c9d79a700 1 -- 172.21.15.27:0/806150856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c9813fbb0 msgr2=0x7f9c98155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.610+0000 7f9c9d79a700 1 --2- 172.21.15.27:0/806150856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c9813fbb0 0x7f9c98155c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.610+0000 7f9c9d79a700 1 -- 172.21.15.27:0/806150856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c940093c0 con 0x7f9c9813ff80 2024-01-11T08:22:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.610+0000 7f9c9df9b700 1 --2- 172.21.15.27:0/806150856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c9813fbb0 0x7f9c98155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:22:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.610+0000 7f9c9e79c700 1 --2- 172.21.15.27:0/806150856 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c9815a730 0x7f9c980c2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:22:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.611+0000 7f9c9d79a700 1 --2- 172.21.15.27:0/806150856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c9813ff80 0x7f9c980c22c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9c8800c790 tx=0x7f9c88007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.611+0000 7f9c8dffb700 1 -- 172.21.15.27:0/806150856 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c88005b30 con 0x7f9c9813ff80 2024-01-11T08:22:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.611+0000 7f9c8dffb700 1 -- 172.21.15.27:0/806150856 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c88005c90 con 0x7f9c9813ff80 2024-01-11T08:22:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.611+0000 7f9ca01ff700 1 -- 172.21.15.27:0/806150856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c98156d70 con 0x7f9c9813ff80 2024-01-11T08:22:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.611+0000 7f9c8dffb700 1 -- 172.21.15.27:0/806150856 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c88007300 con 0x7f9c9813ff80 2024-01-11T08:22:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.611+0000 7f9ca01ff700 1 -- 172.21.15.27:0/806150856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c98157330 con 0x7f9c9813ff80 2024-01-11T08:22:32.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.612+0000 7f9ca01ff700 1 -- 172.21.15.27:0/806150856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c980c91b0 con 0x7f9c9813ff80 2024-01-11T08:22:32.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.613+0000 7f9c8dffb700 1 -- 172.21.15.27:0/806150856 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f9c8801b400 con 0x7f9c9813ff80 2024-01-11T08:22:32.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.614+0000 7f9c8dffb700 1 --2- 172.21.15.27:0/806150856 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9c7c067130 0x7f9c7c0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:32.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.614+0000 7f9c8dffb700 1 -- 172.21.15.27:0/806150856 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f9c88084500 con 0x7f9c9813ff80 2024-01-11T08:22:32.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.614+0000 7f9c9df9b700 1 --2- 172.21.15.27:0/806150856 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9c7c067130 0x7f9c7c0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:32.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.615+0000 7f9c9df9b700 1 --2- 172.21.15.27:0/806150856 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9c7c067130 0x7f9c7c0695e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9c980c2f00 tx=0x7f9c94016000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:32.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.618+0000 7f9c8dffb700 1 -- 172.21.15.27:0/806150856 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9c88051520 con 0x7f9c9813ff80 2024-01-11T08:22:32.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.849+0000 7f9ca01ff700 1 -- 172.21.15.27:0/806150856 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c9804c3c0 con 0x7f9c7c067130 2024-01-11T08:22:32.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.937+0000 7f9c8dffb700 1 -- 172.21.15.27:0/806150856 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f9c9804c3c0 con 0x7f9c7c067130 2024-01-11T08:22:32.937 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule removed for path /snap_test_dir1/dir1 2024-01-11T08:22:32.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.940+0000 7f9ca01ff700 1 -- 172.21.15.27:0/806150856 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9c7c067130 msgr2=0x7f9c7c0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:32.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.940+0000 7f9ca01ff700 1 --2- 172.21.15.27:0/806150856 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9c7c067130 0x7f9c7c0695e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9c980c2f00 tx=0x7f9c94016000 comp rx=0 tx=0).stop 2024-01-11T08:22:32.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.940+0000 7f9ca01ff700 1 -- 172.21.15.27:0/806150856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c9813ff80 msgr2=0x7f9c980c22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:32.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.940+0000 7f9ca01ff700 1 --2- 172.21.15.27:0/806150856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c9813ff80 0x7f9c980c22c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9c8800c790 tx=0x7f9c88007ea0 comp rx=0 tx=0).stop 2024-01-11T08:22:32.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.940+0000 7f9ca01ff700 1 -- 172.21.15.27:0/806150856 shutdown_connections 2024-01-11T08:22:32.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.940+0000 7f9ca01ff700 1 --2- 172.21.15.27:0/806150856 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9c7c067130 0x7f9c7c0695e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:32.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.940+0000 7f9ca01ff700 1 --2- 172.21.15.27:0/806150856 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c9815a730 0x7f9c980c2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:32.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.940+0000 7f9ca01ff700 1 --2- 172.21.15.27:0/806150856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c9813fbb0 0x7f9c98155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:32.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.940+0000 7f9ca01ff700 1 --2- 172.21.15.27:0/806150856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c9813ff80 0x7f9c980c22c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:32.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.940+0000 7f9ca01ff700 1 -- 172.21.15.27:0/806150856 >> 172.21.15.27:0/806150856 conn(0x7f9c980680a0 msgr2=0x7f9c980c8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:32.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.940+0000 7f9ca01ff700 1 -- 172.21.15.27:0/806150856 shutdown_connections 2024-01-11T08:22:32.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:32.940+0000 7f9ca01ff700 1 -- 172.21.15.27:0/806150856 wait complete. 2024-01-11T08:22:32.958 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2024-01-11T08:22:32.958 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:32.958 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2024-01-11T08:22:33.007 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:33.007 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:33.008 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:33.008 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:33.008 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:33.008 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:33.008 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:33.008 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:33.008 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:33.008 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:33.008 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:33.009 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:33.009 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:33.009 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_12_00_UTC) 2024-01-11T08:22:33.066 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:33.066 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:33.067 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_13_00_UTC) 2024-01-11T08:22:33.121 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:33.122 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:33.122 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_14_00_UTC) 2024-01-11T08:22:33.177 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:33.177 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:33.177 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_15_00_UTC) 2024-01-11T08:22:33.232 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:33.232 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:33.232 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_16_00_UTC) 2024-01-11T08:22:33.287 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:33.288 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:33.288 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_17_00_UTC) 2024-01-11T08:22:33.344 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:33.344 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:33.344 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_18_00_UTC) 2024-01-11T08:22:33.399 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:33.399 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:33.399 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_19_00_UTC) 2024-01-11T08:22:33.454 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:33.455 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:33.455 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_20_00_UTC) 2024-01-11T08:22:33.509 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:33.509 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:33.510 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_21_00_UTC) 2024-01-11T08:22:33.568 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:33.568 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:33.568 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-01-11-08_22_00_UTC) 2024-01-11T08:22:33.624 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:33.624 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2024-01-11T08:22:33.675 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir2 1m 2024-01-11T08:22:33.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.878+0000 7f7a1b1ff700 1 Processor -- start 2024-01-11T08:22:33.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.879+0000 7f7a1b1ff700 1 -- start start 2024-01-11T08:22:33.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.880+0000 7f7a1b1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a14157410 0x7f7a140ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:33.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.880+0000 7f7a1b1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7a141577e0 0x7f7a140ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:33.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.880+0000 7f7a1b1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a140699d0 con 0x7f7a14069540 2024-01-11T08:22:33.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.880+0000 7f7a1b1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a14069cb0 con 0x7f7a141577e0 2024-01-11T08:22:33.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.880+0000 7f7a1b1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a14069f90 con 0x7f7a14157410 2024-01-11T08:22:33.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.880+0000 7f7a1979c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7a14069540 0x7f7a1415b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53834/0 (socket says 172.21.15.27:53834) 2024-01-11T08:22:33.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.880+0000 7f7a1979c700 1 -- 172.21.15.27:0/3645182888 learned_addr learned my addr 172.21.15.27:0/3645182888 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:33.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.881+0000 7f7a13fff700 1 --2- 172.21.15.27:0/3645182888 >> v2:172.21.15.77:3300/0 conn(0x7f7a141577e0 0x7f7a140ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:33.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.881+0000 7f7a18f9b700 1 --2- 172.21.15.27:0/3645182888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a14157410 0x7f7a140ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:33.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.881+0000 7f7a137fe700 1 -- 172.21.15.27:0/3645182888 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276972674 0 0) 0x7f7a140699d0 con 0x7f7a14069540 2024-01-11T08:22:33.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.881+0000 7f7a137fe700 1 -- 172.21.15.27:0/3645182888 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a04003570 con 0x7f7a14069540 2024-01-11T08:22:33.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.881+0000 7f7a137fe700 1 -- 172.21.15.27:0/3645182888 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 346126108 0 0) 0x7f7a04003570 con 0x7f7a14069540 2024-01-11T08:22:33.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.881+0000 7f7a137fe700 1 -- 172.21.15.27:0/3645182888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a14157410 msgr2=0x7f7a140ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:33.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.881+0000 7f7a137fe700 1 --2- 172.21.15.27:0/3645182888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a14157410 0x7f7a140ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:33.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.881+0000 7f7a137fe700 1 -- 172.21.15.27:0/3645182888 >> v2:172.21.15.77:3300/0 conn(0x7f7a141577e0 msgr2=0x7f7a140ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:33.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.881+0000 7f7a137fe700 1 --2- 172.21.15.27:0/3645182888 >> v2:172.21.15.77:3300/0 conn(0x7f7a141577e0 0x7f7a140ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:33.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.881+0000 7f7a137fe700 1 -- 172.21.15.27:0/3645182888 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1406a270 con 0x7f7a14069540 2024-01-11T08:22:33.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.881+0000 7f7a137fe700 1 -- 172.21.15.27:0/3645182888 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f7a0c0017f0 con 0x7f7a14069540 2024-01-11T08:22:33.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.881+0000 7f7a137fe700 1 -- 172.21.15.27:0/3645182888 >> v1:172.21.15.27:6789/0 conn(0x7f7a14069540 legacy=0x7f7a1415b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:33.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.881+0000 7f7a137fe700 1 --2- 172.21.15.27:0/3645182888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a14157410 0x7f7a04003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:33.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.881+0000 7f7a137fe700 1 --2- 172.21.15.27:0/3645182888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a040040b0 0x7f7a04004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.881+0000 7f7a137fe700 1 --2- 172.21.15.27:0/3645182888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a04004d50 0x7f7a040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.881+0000 7f7a137fe700 1 -- 172.21.15.27:0/3645182888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a14069cb0 con 0x7f7a04004d50 2024-01-11T08:22:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.881+0000 7f7a137fe700 1 -- 172.21.15.27:0/3645182888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a14069f90 con 0x7f7a040040b0 2024-01-11T08:22:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.881+0000 7f7a137fe700 1 -- 172.21.15.27:0/3645182888 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0400bb60 con 0x7f7a14157410 2024-01-11T08:22:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.882+0000 7f7a1979c700 1 --2- 172.21.15.27:0/3645182888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a04004d50 0x7f7a040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.882+0000 7f7a13fff700 1 --2- 172.21.15.27:0/3645182888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a040040b0 0x7f7a04004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.882+0000 7f7a18f9b700 1 --2- 172.21.15.27:0/3645182888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a14157410 0x7f7a04003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.882+0000 7f7a1979c700 1 -- 172.21.15.27:0/3645182888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a14157410 msgr2=0x7f7a04003ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.882+0000 7f7a1979c700 1 --2- 172.21.15.27:0/3645182888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a14157410 0x7f7a04003ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.882+0000 7f7a1979c700 1 -- 172.21.15.27:0/3645182888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a040040b0 msgr2=0x7f7a04004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.882+0000 7f7a1979c700 1 --2- 172.21.15.27:0/3645182888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a040040b0 0x7f7a04004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.882+0000 7f7a1979c700 1 -- 172.21.15.27:0/3645182888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a04003570 con 0x7f7a04004d50 2024-01-11T08:22:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.882+0000 7f7a1979c700 1 --2- 172.21.15.27:0/3645182888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a04004d50 0x7f7a040051c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c002250 tx=0x7f7a1406a550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13909f00a92b8d5d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:33.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.882+0000 7f7a137fe700 1 -- 172.21.15.27:0/3645182888 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c0070f0 con 0x7f7a04004d50 2024-01-11T08:22:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.882+0000 7f7a137fe700 1 -- 172.21.15.27:0/3645182888 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0c007290 con 0x7f7a04004d50 2024-01-11T08:22:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.882+0000 7f7a137fe700 1 -- 172.21.15.27:0/3645182888 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c007650 con 0x7f7a04004d50 2024-01-11T08:22:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.883+0000 7f7a1b1ff700 1 -- 172.21.15.27:0/3645182888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a04004d50 msgr2=0x7f7a040051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.883+0000 7f7a1b1ff700 1 --2- 172.21.15.27:0/3645182888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a04004d50 0x7f7a040051c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c002250 tx=0x7f7a1406a550 comp rx=0 tx=0).stop 2024-01-11T08:22:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.883+0000 7f7a1979c700 1 -- 172.21.15.27:0/3645182888 reap_dead start 2024-01-11T08:22:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.883+0000 7f7a1b1ff700 1 -- 172.21.15.27:0/3645182888 shutdown_connections 2024-01-11T08:22:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.883+0000 7f7a1b1ff700 1 -- 172.21.15.27:0/3645182888 >> 172.21.15.27:0/3645182888 conn(0x7f7a140680a0 msgr2=0x7f7a140cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.883+0000 7f7a1b1ff700 1 -- 172.21.15.27:0/3645182888 shutdown_connections 2024-01-11T08:22:33.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.883+0000 7f7a1b1ff700 1 -- 172.21.15.27:0/3645182888 wait complete. 2024-01-11T08:22:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.885+0000 7f7a1b1ff700 1 Processor -- start 2024-01-11T08:22:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.885+0000 7f7a1b1ff700 1 -- start start 2024-01-11T08:22:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.886+0000 7f7a1b1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a04004d50 0x7f7a14149db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.886+0000 7f7a1b1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a1415bfc0 0x7f7a1415c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.886+0000 7f7a1b1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a140c59b0 0x7f7a140c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.886+0000 7f7a1b1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1415c970 con 0x7f7a04004d50 2024-01-11T08:22:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.886+0000 7f7a1b1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a140c9e30 con 0x7f7a140c59b0 2024-01-11T08:22:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.886+0000 7f7a1b1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a140ca110 con 0x7f7a1415bfc0 2024-01-11T08:22:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.886+0000 7f7a18f9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a04004d50 0x7f7a14149db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.886+0000 7f7a18f9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a04004d50 0x7f7a14149db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43936/0 (socket says 172.21.15.27:43936) 2024-01-11T08:22:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.886+0000 7f7a18f9b700 1 -- 172.21.15.27:0/2701157473 learned_addr learned my addr 172.21.15.27:0/2701157473 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.886+0000 7f7a1979c700 1 --2- 172.21.15.27:0/2701157473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a140c59b0 0x7f7a140c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.886+0000 7f7a13fff700 1 --2- 172.21.15.27:0/2701157473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a1415bfc0 0x7f7a1415c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.887+0000 7f7a18f9b700 1 -- 172.21.15.27:0/2701157473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a1415bfc0 msgr2=0x7f7a1415c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.887+0000 7f7a18f9b700 1 --2- 172.21.15.27:0/2701157473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a1415bfc0 0x7f7a1415c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:33.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.887+0000 7f7a18f9b700 1 -- 172.21.15.27:0/2701157473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a140c59b0 msgr2=0x7f7a140c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.887+0000 7f7a18f9b700 1 --2- 172.21.15.27:0/2701157473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a140c59b0 0x7f7a140c5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.887+0000 7f7a18f9b700 1 -- 172.21.15.27:0/2701157473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a140ca3f0 con 0x7f7a04004d50 2024-01-11T08:22:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.887+0000 7f7a13fff700 1 --2- 172.21.15.27:0/2701157473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a1415bfc0 0x7f7a1415c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:22:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.887+0000 7f7a18f9b700 1 --2- 172.21.15.27:0/2701157473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a04004d50 0x7f7a14149db0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7a08011b10 tx=0x7f7a08009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.887+0000 7f7a10ff9700 1 -- 172.21.15.27:0/2701157473 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a08005c10 con 0x7f7a04004d50 2024-01-11T08:22:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.887+0000 7f7a10ff9700 1 -- 172.21.15.27:0/2701157473 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a08005d70 con 0x7f7a04004d50 2024-01-11T08:22:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.887+0000 7f7a10ff9700 1 -- 172.21.15.27:0/2701157473 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0800f960 con 0x7f7a04004d50 2024-01-11T08:22:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.887+0000 7f7a1b1ff700 1 -- 172.21.15.27:0/2701157473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a140ca6d0 con 0x7f7a04004d50 2024-01-11T08:22:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.887+0000 7f7a1b1ff700 1 -- 172.21.15.27:0/2701157473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a140c6340 con 0x7f7a04004d50 2024-01-11T08:22:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.888+0000 7f7a1b1ff700 1 -- 172.21.15.27:0/2701157473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a14155170 con 0x7f7a04004d50 2024-01-11T08:22:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.889+0000 7f7a10ff9700 1 -- 172.21.15.27:0/2701157473 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f7a0800fd50 con 0x7f7a04004d50 2024-01-11T08:22:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.890+0000 7f7a10ff9700 1 --2- 172.21.15.27:0/2701157473 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f79f8067130 0x7f79f80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.890+0000 7f7a10ff9700 1 -- 172.21.15.27:0/2701157473 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f7a080853e0 con 0x7f7a04004d50 2024-01-11T08:22:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.891+0000 7f7a13fff700 1 --2- 172.21.15.27:0/2701157473 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f79f8067130 0x7f79f80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.891+0000 7f7a13fff700 1 --2- 172.21.15.27:0/2701157473 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f79f8067130 0x7f79f80695e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7a00006eb0 tx=0x7f7a00008040 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:33.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:33.894+0000 7f7a10ff9700 1 -- 172.21.15.27:0/2701157473 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7a08052230 con 0x7f7a04004d50 2024-01-11T08:22:34.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:34.122+0000 7f7a1b1ff700 1 -- 172.21.15.27:0/2701157473 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a1404c3c0 con 0x7f79f8067130 2024-01-11T08:22:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:34.210+0000 7f7a10ff9700 1 -- 172.21.15.27:0/2701157473 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f7a1404c3c0 con 0x7f79f8067130 2024-01-11T08:22:34.209 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule removed for path /snap_test_dir1/dir2 2024-01-11T08:22:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:34.212+0000 7f7a1b1ff700 1 -- 172.21.15.27:0/2701157473 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f79f8067130 msgr2=0x7f79f80695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:34.212+0000 7f7a1b1ff700 1 --2- 172.21.15.27:0/2701157473 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f79f8067130 0x7f79f80695e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7a00006eb0 tx=0x7f7a00008040 comp rx=0 tx=0).stop 2024-01-11T08:22:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:34.212+0000 7f7a1b1ff700 1 -- 172.21.15.27:0/2701157473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a04004d50 msgr2=0x7f7a14149db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:34.212+0000 7f7a1b1ff700 1 --2- 172.21.15.27:0/2701157473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a04004d50 0x7f7a14149db0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7a08011b10 tx=0x7f7a08009120 comp rx=0 tx=0).stop 2024-01-11T08:22:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:34.212+0000 7f7a1b1ff700 1 -- 172.21.15.27:0/2701157473 shutdown_connections 2024-01-11T08:22:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:34.212+0000 7f7a1b1ff700 1 --2- 172.21.15.27:0/2701157473 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f79f8067130 0x7f79f80695e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:34.212+0000 7f7a1b1ff700 1 --2- 172.21.15.27:0/2701157473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a140c59b0 0x7f7a140c5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:34.212+0000 7f7a1b1ff700 1 --2- 172.21.15.27:0/2701157473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a04004d50 0x7f7a14149db0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:34.212+0000 7f7a1b1ff700 1 --2- 172.21.15.27:0/2701157473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a1415bfc0 0x7f7a1415c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:34.212+0000 7f7a1b1ff700 1 -- 172.21.15.27:0/2701157473 >> 172.21.15.27:0/2701157473 conn(0x7f7a140680a0 msgr2=0x7f7a1414bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:34.212+0000 7f7a1b1ff700 1 -- 172.21.15.27:0/2701157473 shutdown_connections 2024-01-11T08:22:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:34.212+0000 7f7a1b1ff700 1 -- 172.21.15.27:0/2701157473 wait complete. 2024-01-11T08:22:34.229 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2024-01-11T08:22:34.229 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:34.230 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2024-01-11T08:22:34.278 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:34.278 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:34.278 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:34.278 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:34.278 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:34.278 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:34.278 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:34.278 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:34.278 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:34.279 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:34.279 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:34.279 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:34.280 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:34.280 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_12_00_UTC) 2024-01-11T08:22:34.336 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:34.336 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:34.337 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_13_00_UTC) 2024-01-11T08:22:34.393 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:34.393 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:34.393 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_14_00_UTC) 2024-01-11T08:22:34.448 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:34.449 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:34.449 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_15_00_UTC) 2024-01-11T08:22:34.504 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:34.504 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:34.504 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_16_00_UTC) 2024-01-11T08:22:34.560 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:34.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:34.560 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_17_00_UTC) 2024-01-11T08:22:34.616 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:34.617 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:34.617 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_18_00_UTC) 2024-01-11T08:22:34.673 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:34.673 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:34.673 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_19_00_UTC) 2024-01-11T08:22:34.729 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:34.729 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:34.729 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_20_00_UTC) 2024-01-11T08:22:34.784 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:34.784 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:34.785 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_21_00_UTC) 2024-01-11T08:22:34.840 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:34.840 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:34.840 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-01-11-08_22_00_UTC) 2024-01-11T08:22:34.895 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:34.895 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2024-01-11T08:22:34.946 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir3 1m 2024-01-11T08:22:35.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.148+0000 7fb8ad9ff700 1 Processor -- start 2024-01-11T08:22:35.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.148+0000 7fb8ad9ff700 1 -- start start 2024-01-11T08:22:35.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.150+0000 7fb8ad9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb8a8142dc0 0x7fb8a80cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:35.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.150+0000 7fb8ad9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8a80cb2e0 0x7fb8a80cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:35.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.150+0000 7fb8ad9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a80ccc70 con 0x7fb8a8143190 2024-01-11T08:22:35.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.150+0000 7fb8ad9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a815df30 con 0x7fb8a8142dc0 2024-01-11T08:22:35.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.150+0000 7fb8ad9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a80cb740 con 0x7fb8a80cb2e0 2024-01-11T08:22:35.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.150+0000 7fb8a6ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb8a8142dc0 0x7fb8a80cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:35.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.150+0000 7fb8a6ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb8a8142dc0 0x7fb8a80cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43938/0 (socket says 172.21.15.27:43938) 2024-01-11T08:22:35.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.150+0000 7fb8a6ffd700 1 -- 172.21.15.27:0/1429795484 learned_addr learned my addr 172.21.15.27:0/1429795484 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:35.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.150+0000 7fb8a77fe700 1 --2- 172.21.15.27:0/1429795484 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8a80cb2e0 0x7fb8a80cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:35.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.151+0000 7fb8a6ffd700 1 -- 172.21.15.27:0/1429795484 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8a80cb2e0 msgr2=0x7fb8a80cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:35.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.151+0000 7fb8a6ffd700 1 --2- 172.21.15.27:0/1429795484 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8a80cb2e0 0x7fb8a80cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:35.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.151+0000 7fb8a6ffd700 1 -- 172.21.15.27:0/1429795484 >> v1:172.21.15.77:6789/0 conn(0x7fb8a8143190 legacy=0x7fb8a80cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:35.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.151+0000 7fb8a6ffd700 1 -- 172.21.15.27:0/1429795484 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a815d320 con 0x7fb8a8142dc0 2024-01-11T08:22:35.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.151+0000 7fb8a77fe700 1 --2- 172.21.15.27:0/1429795484 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8a80cb2e0 0x7fb8a80cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:22:35.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.151+0000 7fb8a6ffd700 1 --2- 172.21.15.27:0/1429795484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8a8142dc0 0x7fb8a80cada0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb8980099b0 tx=0x7fb898009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6d1f6ea4a40298 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:35.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.151+0000 7fb8a5ffb700 1 -- 172.21.15.27:0/1429795484 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89800b860 con 0x7fb8a8142dc0 2024-01-11T08:22:35.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.151+0000 7fb8a5ffb700 1 -- 172.21.15.27:0/1429795484 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89800b9c0 con 0x7fb8a8142dc0 2024-01-11T08:22:35.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.151+0000 7fb8a5ffb700 1 -- 172.21.15.27:0/1429795484 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89800bd60 con 0x7fb8a8142dc0 2024-01-11T08:22:35.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.152+0000 7fb8ad9ff700 1 -- 172.21.15.27:0/1429795484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8a8142dc0 msgr2=0x7fb8a80cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:35.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.152+0000 7fb8ad9ff700 1 --2- 172.21.15.27:0/1429795484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8a8142dc0 0x7fb8a80cada0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb8980099b0 tx=0x7fb898009250 comp rx=0 tx=0).stop 2024-01-11T08:22:35.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.152+0000 7fb8ad9ff700 1 -- 172.21.15.27:0/1429795484 shutdown_connections 2024-01-11T08:22:35.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.152+0000 7fb8ad9ff700 1 --2- 172.21.15.27:0/1429795484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8a8142dc0 0x7fb8a80cada0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:35.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.152+0000 7fb8ad9ff700 1 --2- 172.21.15.27:0/1429795484 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8a80cb2e0 0x7fb8a80cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:35.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.152+0000 7fb8ad9ff700 1 -- 172.21.15.27:0/1429795484 >> 172.21.15.27:0/1429795484 conn(0x7fb8a8063ca0 msgr2=0x7fb8a806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:35.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.152+0000 7fb8ad9ff700 1 -- 172.21.15.27:0/1429795484 shutdown_connections 2024-01-11T08:22:35.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.152+0000 7fb8ad9ff700 1 -- 172.21.15.27:0/1429795484 wait complete. 2024-01-11T08:22:35.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.153+0000 7fb8ad9ff700 1 Processor -- start 2024-01-11T08:22:35.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.154+0000 7fb8ad9ff700 1 -- start start 2024-01-11T08:22:35.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.155+0000 7fb8ad9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8a80cb2e0 0x7fb8a80b1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:35.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.155+0000 7fb8ad9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8a8142dc0 0x7fb8a80b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:35.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.155+0000 7fb8ad9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8a8143190 0x7fb8a8157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:35.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.155+0000 7fb8ad9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a80b1f10 con 0x7fb8a8142dc0 2024-01-11T08:22:35.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.155+0000 7fb8ad9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a80b2180 con 0x7fb8a8143190 2024-01-11T08:22:35.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.155+0000 7fb8ad9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a80b2460 con 0x7fb8a80cb2e0 2024-01-11T08:22:35.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.155+0000 7fb8a67fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8a8142dc0 0x7fb8a80b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:35.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.155+0000 7fb8a67fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8a8142dc0 0x7fb8a80b19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43950/0 (socket says 172.21.15.27:43950) 2024-01-11T08:22:35.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.155+0000 7fb8a67fc700 1 -- 172.21.15.27:0/1728230974 learned_addr learned my addr 172.21.15.27:0/1728230974 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:35.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.155+0000 7fb8a6ffd700 1 --2- 172.21.15.27:0/1728230974 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8a80cb2e0 0x7fb8a80b1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:35.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.155+0000 7fb8a77fe700 1 --2- 172.21.15.27:0/1728230974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8a8143190 0x7fb8a8157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:35.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.155+0000 7fb8a67fc700 1 -- 172.21.15.27:0/1728230974 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8a80cb2e0 msgr2=0x7fb8a80b1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:35.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.155+0000 7fb8a67fc700 1 --2- 172.21.15.27:0/1728230974 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8a80cb2e0 0x7fb8a80b1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:35.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.155+0000 7fb8a67fc700 1 -- 172.21.15.27:0/1728230974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8a8143190 msgr2=0x7fb8a8157c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:35.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.155+0000 7fb8a67fc700 1 --2- 172.21.15.27:0/1728230974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8a8143190 0x7fb8a8157c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:35.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.155+0000 7fb8a67fc700 1 -- 172.21.15.27:0/1728230974 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8980094f0 con 0x7fb8a8142dc0 2024-01-11T08:22:35.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.155+0000 7fb8a77fe700 1 --2- 172.21.15.27:0/1728230974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8a8143190 0x7fb8a8157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:22:35.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.156+0000 7fb8a67fc700 1 --2- 172.21.15.27:0/1728230974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8a8142dc0 0x7fb8a80b19d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb89c00c4e0 tx=0x7fb89c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:35.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.156+0000 7fb88effd700 1 -- 172.21.15.27:0/1728230974 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c011590 con 0x7fb8a8142dc0 2024-01-11T08:22:35.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.156+0000 7fb88effd700 1 -- 172.21.15.27:0/1728230974 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89c011710 con 0x7fb8a8142dc0 2024-01-11T08:22:35.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.156+0000 7fb88effd700 1 -- 172.21.15.27:0/1728230974 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c011ad0 con 0x7fb8a8142dc0 2024-01-11T08:22:35.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.156+0000 7fb8ad9ff700 1 -- 172.21.15.27:0/1728230974 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a80b2740 con 0x7fb8a8142dc0 2024-01-11T08:22:35.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.156+0000 7fb8ad9ff700 1 -- 172.21.15.27:0/1728230974 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a80b2d00 con 0x7fb8a8142dc0 2024-01-11T08:22:35.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.157+0000 7fb8ad9ff700 1 -- 172.21.15.27:0/1728230974 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a80b2f20 con 0x7fb8a8142dc0 2024-01-11T08:22:35.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.158+0000 7fb88effd700 1 -- 172.21.15.27:0/1728230974 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fb89c01b3b0 con 0x7fb8a8142dc0 2024-01-11T08:22:35.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.159+0000 7fb88effd700 1 --2- 172.21.15.27:0/1728230974 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb894067130 0x7fb8940695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:35.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.159+0000 7fb88effd700 1 -- 172.21.15.27:0/1728230974 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fb89c083ed0 con 0x7fb8a8142dc0 2024-01-11T08:22:35.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.160+0000 7fb8a6ffd700 1 --2- 172.21.15.27:0/1728230974 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb894067130 0x7fb8940695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:35.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.160+0000 7fb8a6ffd700 1 --2- 172.21.15.27:0/1728230974 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb894067130 0x7fb8940695e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb8980099b0 tx=0x7fb898017000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:35.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.163+0000 7fb88effd700 1 -- 172.21.15.27:0/1728230974 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb89c050d20 con 0x7fb8a8142dc0 2024-01-11T08:22:35.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.392+0000 7fb8ad9ff700 1 -- 172.21.15.27:0/1728230974 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8a804c3c0 con 0x7fb894067130 2024-01-11T08:22:35.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.474+0000 7fb88effd700 1 -- 172.21.15.27:0/1728230974 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fb8a804c3c0 con 0x7fb894067130 2024-01-11T08:22:35.473 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule removed for path /snap_test_dir1/dir3 2024-01-11T08:22:35.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.476+0000 7fb8ad9ff700 1 -- 172.21.15.27:0/1728230974 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb894067130 msgr2=0x7fb8940695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:35.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.476+0000 7fb8ad9ff700 1 --2- 172.21.15.27:0/1728230974 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb894067130 0x7fb8940695e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb8980099b0 tx=0x7fb898017000 comp rx=0 tx=0).stop 2024-01-11T08:22:35.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.476+0000 7fb8ad9ff700 1 -- 172.21.15.27:0/1728230974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8a8142dc0 msgr2=0x7fb8a80b19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:35.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.476+0000 7fb8ad9ff700 1 --2- 172.21.15.27:0/1728230974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8a8142dc0 0x7fb8a80b19d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb89c00c4e0 tx=0x7fb89c00c7f0 comp rx=0 tx=0).stop 2024-01-11T08:22:35.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.476+0000 7fb8ad9ff700 1 -- 172.21.15.27:0/1728230974 shutdown_connections 2024-01-11T08:22:35.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.476+0000 7fb8ad9ff700 1 --2- 172.21.15.27:0/1728230974 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb894067130 0x7fb8940695e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:35.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.476+0000 7fb8ad9ff700 1 --2- 172.21.15.27:0/1728230974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8a8143190 0x7fb8a8157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:35.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.476+0000 7fb8ad9ff700 1 --2- 172.21.15.27:0/1728230974 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8a80cb2e0 0x7fb8a80b1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:35.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.476+0000 7fb8ad9ff700 1 --2- 172.21.15.27:0/1728230974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8a8142dc0 0x7fb8a80b19d0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:35.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.476+0000 7fb8ad9ff700 1 -- 172.21.15.27:0/1728230974 >> 172.21.15.27:0/1728230974 conn(0x7fb8a8063ca0 msgr2=0x7fb8a813fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:35.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.476+0000 7fb8ad9ff700 1 -- 172.21.15.27:0/1728230974 shutdown_connections 2024-01-11T08:22:35.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:35.476+0000 7fb8ad9ff700 1 -- 172.21.15.27:0/1728230974 wait complete. 2024-01-11T08:22:35.494 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2024-01-11T08:22:35.494 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:35.494 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2024-01-11T08:22:35.544 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:35.544 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:35.544 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:35.544 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:35.544 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:35.544 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:35.544 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:35.544 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:35.544 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:35.545 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:35.545 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_01_UTC 2024-01-11T08:22:35.545 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:35.546 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:35.546 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_12_00_UTC) 2024-01-11T08:22:35.603 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:35.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:35.603 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_13_00_UTC) 2024-01-11T08:22:35.658 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:35.658 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:35.659 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_14_00_UTC) 2024-01-11T08:22:35.715 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:35.715 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:35.715 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_15_00_UTC) 2024-01-11T08:22:35.771 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:35.771 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:35.772 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_16_00_UTC) 2024-01-11T08:22:35.830 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:35.830 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:35.830 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_17_00_UTC) 2024-01-11T08:22:35.885 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:35.885 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:35.886 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_18_00_UTC) 2024-01-11T08:22:35.941 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:35.941 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:35.941 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_19_00_UTC) 2024-01-11T08:22:35.997 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:35.997 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:35.998 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_20_00_UTC) 2024-01-11T08:22:36.054 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:36.055 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:36.055 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_21_00_UTC) 2024-01-11T08:22:36.109 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_22_01_UTC 2024-01-11T08:22:36.109 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:36.110 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-01-11-08_22_01_UTC) 2024-01-11T08:22:36.165 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:36.165 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2024-01-11T08:22:36.217 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir4 1m 2024-01-11T08:22:36.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.420+0000 7fcda33ff700 1 Processor -- start 2024-01-11T08:22:36.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.420+0000 7fcda33ff700 1 -- start start 2024-01-11T08:22:36.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.421+0000 7fcda33ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcd9c15a1b0 0x7fcd9c15a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:36.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.421+0000 7fcda33ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd9c0ccd70 0x7fcd9c0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:36.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.421+0000 7fcda33ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcd9c0685c0 0x7fcd9c068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:36.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.421+0000 7fcda33ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd9c0cf850 con 0x7fcd9c0685c0 2024-01-11T08:22:36.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.421+0000 7fcda33ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd9c0cec40 con 0x7fcd9c15a1b0 2024-01-11T08:22:36.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.421+0000 7fcda33ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0cef20 con 0x7fcd9c0ccd70 2024-01-11T08:22:36.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.421+0000 7fcda199c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcd9c0685c0 0x7fcd9c068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:36.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.421+0000 7fcda119b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcd9c15a1b0 0x7fcd9c15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:36.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.421+0000 7fcda199c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcd9c0685c0 0x7fcd9c068a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43956/0 (socket says 172.21.15.27:43956) 2024-01-11T08:22:36.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.421+0000 7fcda199c700 1 -- 172.21.15.27:0/1686173205 learned_addr learned my addr 172.21.15.27:0/1686173205 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:36.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.421+0000 7fcda099a700 1 --2- 172.21.15.27:0/1686173205 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd9c0ccd70 0x7fcd9c0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:36.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.422+0000 7fcda199c700 1 -- 172.21.15.27:0/1686173205 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd9c0ccd70 msgr2=0x7fcd9c0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:36.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.422+0000 7fcda199c700 1 --2- 172.21.15.27:0/1686173205 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd9c0ccd70 0x7fcd9c0cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:36.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.422+0000 7fcda199c700 1 -- 172.21.15.27:0/1686173205 >> v2:172.21.15.77:3300/0 conn(0x7fcd9c15a1b0 msgr2=0x7fcd9c15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:36.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.422+0000 7fcda199c700 1 --2- 172.21.15.27:0/1686173205 >> v2:172.21.15.77:3300/0 conn(0x7fcd9c15a1b0 0x7fcd9c15a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:36.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.422+0000 7fcda199c700 1 -- 172.21.15.27:0/1686173205 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c0cf200 con 0x7fcd9c0685c0 2024-01-11T08:22:36.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.422+0000 7fcda199c700 1 --2- 172.21.15.27:0/1686173205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd9c0685c0 0x7fcd9c068a40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0d16f0 tx=0x7fcd94009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f537941c530420d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:36.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.422+0000 7fcd93fff700 1 -- 172.21.15.27:0/1686173205 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9400e070 con 0x7fcd9c0685c0 2024-01-11T08:22:36.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.423+0000 7fcd93fff700 1 -- 172.21.15.27:0/1686173205 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9400b970 con 0x7fcd9c0685c0 2024-01-11T08:22:36.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.423+0000 7fcd93fff700 1 -- 172.21.15.27:0/1686173205 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd940092e0 con 0x7fcd9c0685c0 2024-01-11T08:22:36.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.423+0000 7fcda33ff700 1 -- 172.21.15.27:0/1686173205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd9c0685c0 msgr2=0x7fcd9c068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:36.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.423+0000 7fcda33ff700 1 --2- 172.21.15.27:0/1686173205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd9c0685c0 0x7fcd9c068a40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0d16f0 tx=0x7fcd94009840 comp rx=0 tx=0).stop 2024-01-11T08:22:36.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.423+0000 7fcda33ff700 1 -- 172.21.15.27:0/1686173205 shutdown_connections 2024-01-11T08:22:36.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.423+0000 7fcda33ff700 1 --2- 172.21.15.27:0/1686173205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd9c0685c0 0x7fcd9c068a40 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:36.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.423+0000 7fcda33ff700 1 --2- 172.21.15.27:0/1686173205 >> v2:172.21.15.77:3300/0 conn(0x7fcd9c15a1b0 0x7fcd9c15a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:36.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.423+0000 7fcda33ff700 1 --2- 172.21.15.27:0/1686173205 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd9c0ccd70 0x7fcd9c0cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:36.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.423+0000 7fcda33ff700 1 -- 172.21.15.27:0/1686173205 >> 172.21.15.27:0/1686173205 conn(0x7fcd9c06ed40 msgr2=0x7fcd9c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:36.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.423+0000 7fcda33ff700 1 -- 172.21.15.27:0/1686173205 shutdown_connections 2024-01-11T08:22:36.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.423+0000 7fcda33ff700 1 -- 172.21.15.27:0/1686173205 wait complete. 2024-01-11T08:22:36.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.425+0000 7fcda33ff700 1 Processor -- start 2024-01-11T08:22:36.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.425+0000 7fcda33ff700 1 -- start start 2024-01-11T08:22:36.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.426+0000 7fcda33ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd9c0685c0 0x7fcd9c0b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:36.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.426+0000 7fcda33ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd9c0ccd70 0x7fcd9c14af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:36.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.426+0000 7fcda33ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd9c15a1b0 0x7fcd9c0b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:36.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.426+0000 7fcda33ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0c7b00 con 0x7fcd9c15a1b0 2024-01-11T08:22:36.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.426+0000 7fcda33ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0c7de0 con 0x7fcd9c0685c0 2024-01-11T08:22:36.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.426+0000 7fcda33ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0c80c0 con 0x7fcd9c0ccd70 2024-01-11T08:22:36.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.426+0000 7fcda199c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd9c15a1b0 0x7fcd9c0b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:36.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.426+0000 7fcda199c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd9c15a1b0 0x7fcd9c0b58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43972/0 (socket says 172.21.15.27:43972) 2024-01-11T08:22:36.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.426+0000 7fcda199c700 1 -- 172.21.15.27:0/2806814568 learned_addr learned my addr 172.21.15.27:0/2806814568 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:36.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.426+0000 7fcda199c700 1 -- 172.21.15.27:0/2806814568 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd9c0ccd70 msgr2=0x7fcd9c14af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:36.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.426+0000 7fcda099a700 1 --2- 172.21.15.27:0/2806814568 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd9c0ccd70 0x7fcd9c14af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:36.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.426+0000 7fcda119b700 1 --2- 172.21.15.27:0/2806814568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd9c0685c0 0x7fcd9c0b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:36.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.426+0000 7fcda199c700 1 --2- 172.21.15.27:0/2806814568 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd9c0ccd70 0x7fcd9c14af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:36.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.426+0000 7fcda199c700 1 -- 172.21.15.27:0/2806814568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd9c0685c0 msgr2=0x7fcd9c0b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:36.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.426+0000 7fcda199c700 1 --2- 172.21.15.27:0/2806814568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd9c0685c0 0x7fcd9c0b3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:36.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.426+0000 7fcda199c700 1 -- 172.21.15.27:0/2806814568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd94009ae0 con 0x7fcd9c15a1b0 2024-01-11T08:22:36.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.426+0000 7fcda099a700 1 --2- 172.21.15.27:0/2806814568 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd9c0ccd70 0x7fcd9c14af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:22:36.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.426+0000 7fcda119b700 1 --2- 172.21.15.27:0/2806814568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd9c0685c0 0x7fcd9c0b3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:22:36.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.427+0000 7fcda199c700 1 --2- 172.21.15.27:0/2806814568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd9c15a1b0 0x7fcd9c0b58f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c069770 tx=0x7fcd94018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:36.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.427+0000 7fcd917fa700 1 -- 172.21.15.27:0/2806814568 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94008b80 con 0x7fcd9c15a1b0 2024-01-11T08:22:36.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.427+0000 7fcd917fa700 1 -- 172.21.15.27:0/2806814568 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd94008ce0 con 0x7fcd9c15a1b0 2024-01-11T08:22:36.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.427+0000 7fcda33ff700 1 -- 172.21.15.27:0/2806814568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9c0c83a0 con 0x7fcd9c15a1b0 2024-01-11T08:22:36.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.427+0000 7fcd917fa700 1 -- 172.21.15.27:0/2806814568 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd940256b0 con 0x7fcd9c15a1b0 2024-01-11T08:22:36.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.427+0000 7fcda33ff700 1 -- 172.21.15.27:0/2806814568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9c0b4260 con 0x7fcd9c15a1b0 2024-01-11T08:22:36.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.428+0000 7fcda33ff700 1 -- 172.21.15.27:0/2806814568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9c04c3c0 con 0x7fcd9c15a1b0 2024-01-11T08:22:36.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.429+0000 7fcd917fa700 1 -- 172.21.15.27:0/2806814568 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fcd94003040 con 0x7fcd9c15a1b0 2024-01-11T08:22:36.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.430+0000 7fcd917fa700 1 --2- 172.21.15.27:0/2806814568 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcd80067160 0x7fcd80069610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:36.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.430+0000 7fcd917fa700 1 -- 172.21.15.27:0/2806814568 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5963+0+0 (secure 0 0 0) 0x7fcd94085fe0 con 0x7fcd9c15a1b0 2024-01-11T08:22:36.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.431+0000 7fcda119b700 1 --2- 172.21.15.27:0/2806814568 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcd80067160 0x7fcd80069610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:36.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.431+0000 7fcda119b700 1 --2- 172.21.15.27:0/2806814568 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcd80067160 0x7fcd80069610 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcd98006f00 tx=0x7fcd98008040 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:36.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.434+0000 7fcd917fa700 1 -- 172.21.15.27:0/2806814568 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcd94052ee0 con 0x7fcd9c15a1b0 2024-01-11T08:22:36.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.665+0000 7fcda33ff700 1 -- 172.21.15.27:0/2806814568 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd9c0c7f50 con 0x7fcd80067160 2024-01-11T08:22:36.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:36.995+0000 7fcd917fa700 1 -- 172.21.15.27:0/2806814568 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fcd9c0c7f50 con 0x7fcd80067160 2024-01-11T08:22:36.995 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule removed for path /snap_test_dir1/dir4 2024-01-11T08:22:37.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:37.000+0000 7fcda33ff700 1 -- 172.21.15.27:0/2806814568 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcd80067160 msgr2=0x7fcd80069610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:37.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:37.000+0000 7fcda33ff700 1 --2- 172.21.15.27:0/2806814568 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcd80067160 0x7fcd80069610 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcd98006f00 tx=0x7fcd98008040 comp rx=0 tx=0).stop 2024-01-11T08:22:37.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:37.000+0000 7fcda33ff700 1 -- 172.21.15.27:0/2806814568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd9c15a1b0 msgr2=0x7fcd9c0b58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:37.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:37.001+0000 7fcda33ff700 1 --2- 172.21.15.27:0/2806814568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd9c15a1b0 0x7fcd9c0b58f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c069770 tx=0x7fcd94018000 comp rx=0 tx=0).stop 2024-01-11T08:22:37.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:37.001+0000 7fcda33ff700 1 -- 172.21.15.27:0/2806814568 shutdown_connections 2024-01-11T08:22:37.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:37.001+0000 7fcda33ff700 1 --2- 172.21.15.27:0/2806814568 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcd80067160 0x7fcd80069610 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:37.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:37.001+0000 7fcda33ff700 1 --2- 172.21.15.27:0/2806814568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd9c15a1b0 0x7fcd9c0b58f0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:37.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:37.001+0000 7fcda33ff700 1 --2- 172.21.15.27:0/2806814568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd9c0685c0 0x7fcd9c0b3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:37.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:37.001+0000 7fcda33ff700 1 --2- 172.21.15.27:0/2806814568 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd9c0ccd70 0x7fcd9c14af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:37.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:37.001+0000 7fcda33ff700 1 -- 172.21.15.27:0/2806814568 >> 172.21.15.27:0/2806814568 conn(0x7fcd9c06ed40 msgr2=0x7fcd9c0c5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:37.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:37.001+0000 7fcda33ff700 1 -- 172.21.15.27:0/2806814568 shutdown_connections 2024-01-11T08:22:37.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:37.001+0000 7fcda33ff700 1 -- 172.21.15.27:0/2806814568 wait complete. 2024-01-11T08:22:37.033 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2024-01-11T08:22:37.033 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:37.034 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2024-01-11T08:22:37.092 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:37.093 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:37.093 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:37.093 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:37.093 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:37.093 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:37.093 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:37.093 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:37.093 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:37.093 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:37.093 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:37.094 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:37.094 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:37.095 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_12_00_UTC) 2024-01-11T08:22:37.151 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:37.151 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:37.151 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_13_00_UTC) 2024-01-11T08:22:37.206 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:37.206 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:37.207 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_14_00_UTC) 2024-01-11T08:22:37.262 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:37.262 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:37.262 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_15_00_UTC) 2024-01-11T08:22:37.323 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:37.324 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:37.324 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_16_00_UTC) 2024-01-11T08:22:37.401 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:37.401 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:37.401 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_17_00_UTC) 2024-01-11T08:22:37.485 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:37.485 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:37.485 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_18_00_UTC) 2024-01-11T08:22:37.562 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:37.562 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:37.563 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_19_00_UTC) 2024-01-11T08:22:37.618 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:37.618 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:37.618 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_20_00_UTC) 2024-01-11T08:22:37.673 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:37.673 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:37.673 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_21_00_UTC) 2024-01-11T08:22:37.737 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:37.738 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:37.738 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-01-11-08_22_00_UTC) 2024-01-11T08:22:37.793 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:37.794 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2024-01-11T08:22:37.844 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir5 1m 2024-01-11T08:22:38.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.047+0000 7fb6653ff700 1 Processor -- start 2024-01-11T08:22:38.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.047+0000 7fb6653ff700 1 -- start start 2024-01-11T08:22:38.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.049+0000 7fb6653ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb660159ac0 0x7fb660068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:38.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.049+0000 7fb6653ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6600c80a0 con 0x7fb6600ca640 2024-01-11T08:22:38.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.049+0000 7fb6653ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6600cbac0 con 0x7fb6601596f0 2024-01-11T08:22:38.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.049+0000 7fb6653ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6600a9fb0 con 0x7fb660159ac0 2024-01-11T08:22:38.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.049+0000 7fb65f7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb6600ca640 0x7fb6600c8530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53850/0 (socket says 172.21.15.27:53850) 2024-01-11T08:22:38.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.049+0000 7fb65f7fe700 1 -- 172.21.15.27:0/1969998239 learned_addr learned my addr 172.21.15.27:0/1969998239 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:38.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.049+0000 7fb65e7fc700 1 --2- 172.21.15.27:0/1969998239 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb660159ac0 0x7fb660068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:38.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.050+0000 7fb65dffb700 1 -- 172.21.15.27:0/1969998239 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722512139 0 0) 0x7fb6600c80a0 con 0x7fb6600ca640 2024-01-11T08:22:38.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.050+0000 7fb65dffb700 1 -- 172.21.15.27:0/1969998239 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb648003570 con 0x7fb6600ca640 2024-01-11T08:22:38.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.050+0000 7fb65dffb700 1 -- 172.21.15.27:0/1969998239 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007986391 0 0) 0x7fb6600cbac0 con 0x7fb6601596f0 2024-01-11T08:22:38.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.050+0000 7fb65dffb700 1 -- 172.21.15.27:0/1969998239 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6600c80a0 con 0x7fb6601596f0 2024-01-11T08:22:38.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.050+0000 7fb65e7fc700 1 -- 172.21.15.27:0/1969998239 >> v1:172.21.15.77:6789/0 conn(0x7fb6601596f0 legacy=0x7fb6600a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:38.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.050+0000 7fb65e7fc700 1 -- 172.21.15.27:0/1969998239 >> v1:172.21.15.27:6789/0 conn(0x7fb6600ca640 legacy=0x7fb6600c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:38.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.050+0000 7fb65e7fc700 1 -- 172.21.15.27:0/1969998239 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6600c7490 con 0x7fb660159ac0 2024-01-11T08:22:38.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.050+0000 7fb65e7fc700 1 --2- 172.21.15.27:0/1969998239 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb660159ac0 0x7fb660068620 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb654009910 tx=0x7fb6540091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed583b95c2027f4e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:38.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.050+0000 7fb65dffb700 1 -- 172.21.15.27:0/1969998239 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb654011630 con 0x7fb660159ac0 2024-01-11T08:22:38.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.050+0000 7fb65dffb700 1 -- 172.21.15.27:0/1969998239 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb654011790 con 0x7fb660159ac0 2024-01-11T08:22:38.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.050+0000 7fb65dffb700 1 -- 172.21.15.27:0/1969998239 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb654011b30 con 0x7fb660159ac0 2024-01-11T08:22:38.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.051+0000 7fb6653ff700 1 -- 172.21.15.27:0/1969998239 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb660159ac0 msgr2=0x7fb660068620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:38.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.051+0000 7fb6653ff700 1 --2- 172.21.15.27:0/1969998239 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb660159ac0 0x7fb660068620 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb654009910 tx=0x7fb6540091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:38.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.051+0000 7fb6653ff700 1 -- 172.21.15.27:0/1969998239 shutdown_connections 2024-01-11T08:22:38.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.051+0000 7fb6653ff700 1 --2- 172.21.15.27:0/1969998239 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb660159ac0 0x7fb660068620 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:38.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.051+0000 7fb6653ff700 1 -- 172.21.15.27:0/1969998239 >> 172.21.15.27:0/1969998239 conn(0x7fb6600c9fb0 msgr2=0x7fb66006ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:38.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.051+0000 7fb6653ff700 1 -- 172.21.15.27:0/1969998239 shutdown_connections 2024-01-11T08:22:38.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.051+0000 7fb6653ff700 1 -- 172.21.15.27:0/1969998239 wait complete. 2024-01-11T08:22:38.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.053+0000 7fb6653ff700 1 Processor -- start 2024-01-11T08:22:38.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.053+0000 7fb6653ff700 1 -- start start 2024-01-11T08:22:38.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.054+0000 7fb6653ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6600ca640 0x7fb660156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:38.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.054+0000 7fb6653ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6601596f0 0x7fb6600ae750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:38.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.054+0000 7fb6653ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb660159ac0 0x7fb66006b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:38.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.054+0000 7fb6653ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6600af4a0 con 0x7fb6601596f0 2024-01-11T08:22:38.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.054+0000 7fb6653ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6600af1c0 con 0x7fb6600ca640 2024-01-11T08:22:38.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.054+0000 7fb6653ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6600c50c0 con 0x7fb660159ac0 2024-01-11T08:22:38.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.054+0000 7fb65e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6601596f0 0x7fb6600ae750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:38.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.054+0000 7fb65e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6601596f0 0x7fb6600ae750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43982/0 (socket says 172.21.15.27:43982) 2024-01-11T08:22:38.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.054+0000 7fb65e7fc700 1 -- 172.21.15.27:0/1175081266 learned_addr learned my addr 172.21.15.27:0/1175081266 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:38.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.054+0000 7fb65effd700 1 --2- 172.21.15.27:0/1175081266 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6600ca640 0x7fb660156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:38.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.054+0000 7fb65f7fe700 1 --2- 172.21.15.27:0/1175081266 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb660159ac0 0x7fb66006b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:38.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.055+0000 7fb65e7fc700 1 -- 172.21.15.27:0/1175081266 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb660159ac0 msgr2=0x7fb66006b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:38.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.055+0000 7fb65e7fc700 1 --2- 172.21.15.27:0/1175081266 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb660159ac0 0x7fb66006b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:38.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.055+0000 7fb65e7fc700 1 -- 172.21.15.27:0/1175081266 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6600ca640 msgr2=0x7fb660156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:38.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.055+0000 7fb65e7fc700 1 --2- 172.21.15.27:0/1175081266 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6600ca640 0x7fb660156b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:38.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.055+0000 7fb65e7fc700 1 -- 172.21.15.27:0/1175081266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb654009420 con 0x7fb6601596f0 2024-01-11T08:22:38.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.055+0000 7fb65effd700 1 --2- 172.21.15.27:0/1175081266 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6600ca640 0x7fb660156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:22:38.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.055+0000 7fb65e7fc700 1 --2- 172.21.15.27:0/1175081266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6601596f0 0x7fb6600ae750 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb6540091e0 tx=0x7fb654017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:38.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.055+0000 7fb646ffd700 1 -- 172.21.15.27:0/1175081266 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65401f070 con 0x7fb6601596f0 2024-01-11T08:22:38.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.055+0000 7fb646ffd700 1 -- 172.21.15.27:0/1175081266 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb654002f70 con 0x7fb6601596f0 2024-01-11T08:22:38.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.055+0000 7fb6653ff700 1 -- 172.21.15.27:0/1175081266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6600c53a0 con 0x7fb6601596f0 2024-01-11T08:22:38.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.055+0000 7fb646ffd700 1 -- 172.21.15.27:0/1175081266 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb654011b30 con 0x7fb6601596f0 2024-01-11T08:22:38.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.055+0000 7fb6653ff700 1 -- 172.21.15.27:0/1175081266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6601570d0 con 0x7fb6601596f0 2024-01-11T08:22:38.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.056+0000 7fb6653ff700 1 -- 172.21.15.27:0/1175081266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb66004c3c0 con 0x7fb6601596f0 2024-01-11T08:22:38.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.057+0000 7fb646ffd700 1 -- 172.21.15.27:0/1175081266 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fb6540193b0 con 0x7fb6601596f0 2024-01-11T08:22:38.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.058+0000 7fb646ffd700 1 --2- 172.21.15.27:0/1175081266 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb64c067130 0x7fb64c0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:38.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.058+0000 7fb646ffd700 1 -- 172.21.15.27:0/1175081266 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6371+0+0 (secure 0 0 0) 0x7fb654085dc0 con 0x7fb6601596f0 2024-01-11T08:22:38.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.059+0000 7fb65effd700 1 --2- 172.21.15.27:0/1175081266 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb64c067130 0x7fb64c0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:38.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.059+0000 7fb65effd700 1 --2- 172.21.15.27:0/1175081266 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb64c067130 0x7fb64c0695e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb650007fd0 tx=0x7fb650007d50 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:38.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.062+0000 7fb646ffd700 1 -- 172.21.15.27:0/1175081266 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb654052a70 con 0x7fb6601596f0 2024-01-11T08:22:38.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.293+0000 7fb6653ff700 1 -- 172.21.15.27:0/1175081266 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6600c7770 con 0x7fb64c067130 2024-01-11T08:22:38.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.379+0000 7fb646ffd700 1 -- 172.21.15.27:0/1175081266 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fb6600c7770 con 0x7fb64c067130 2024-01-11T08:22:38.379 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule removed for path /snap_test_dir1/dir5 2024-01-11T08:22:38.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.382+0000 7fb6653ff700 1 -- 172.21.15.27:0/1175081266 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb64c067130 msgr2=0x7fb64c0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:38.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.382+0000 7fb6653ff700 1 --2- 172.21.15.27:0/1175081266 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb64c067130 0x7fb64c0695e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb650007fd0 tx=0x7fb650007d50 comp rx=0 tx=0).stop 2024-01-11T08:22:38.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.382+0000 7fb6653ff700 1 -- 172.21.15.27:0/1175081266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6601596f0 msgr2=0x7fb6600ae750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:38.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.382+0000 7fb6653ff700 1 --2- 172.21.15.27:0/1175081266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6601596f0 0x7fb6600ae750 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb6540091e0 tx=0x7fb654017000 comp rx=0 tx=0).stop 2024-01-11T08:22:38.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.382+0000 7fb6653ff700 1 -- 172.21.15.27:0/1175081266 shutdown_connections 2024-01-11T08:22:38.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.382+0000 7fb6653ff700 1 --2- 172.21.15.27:0/1175081266 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb64c067130 0x7fb64c0695e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:38.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.382+0000 7fb6653ff700 1 --2- 172.21.15.27:0/1175081266 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb660159ac0 0x7fb66006b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:38.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.382+0000 7fb6653ff700 1 --2- 172.21.15.27:0/1175081266 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6600ca640 0x7fb660156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:38.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.382+0000 7fb6653ff700 1 --2- 172.21.15.27:0/1175081266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6601596f0 0x7fb6600ae750 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:38.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.382+0000 7fb6653ff700 1 -- 172.21.15.27:0/1175081266 >> 172.21.15.27:0/1175081266 conn(0x7fb6600c9fb0 msgr2=0x7fb6600c1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:38.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.382+0000 7fb6653ff700 1 -- 172.21.15.27:0/1175081266 shutdown_connections 2024-01-11T08:22:38.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:38.382+0000 7fb6653ff700 1 -- 172.21.15.27:0/1175081266 wait complete. 2024-01-11T08:22:38.398 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2024-01-11T08:22:38.398 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:38.399 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2024-01-11T08:22:38.448 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:38.448 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:38.448 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:38.448 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:38.448 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:38.448 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:38.448 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:38.448 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:38.448 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:38.449 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_01_UTC 2024-01-11T08:22:38.449 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:38.449 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:38.450 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:38.450 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_12_00_UTC) 2024-01-11T08:22:38.506 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:38.507 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:38.507 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_13_00_UTC) 2024-01-11T08:22:38.561 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:38.561 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:38.561 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_14_00_UTC) 2024-01-11T08:22:38.615 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:38.615 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:38.615 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_15_00_UTC) 2024-01-11T08:22:38.670 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:38.670 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:38.670 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_16_00_UTC) 2024-01-11T08:22:38.773 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:38.774 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:38.774 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_17_00_UTC) 2024-01-11T08:22:38.844 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:38.845 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:38.845 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_18_00_UTC) 2024-01-11T08:22:38.908 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:38.908 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:38.909 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_19_00_UTC) 2024-01-11T08:22:38.964 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:38.964 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:38.964 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_20_00_UTC) 2024-01-11T08:22:39.019 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_21_01_UTC 2024-01-11T08:22:39.020 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:39.020 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_21_01_UTC) 2024-01-11T08:22:39.075 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:39.075 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:39.075 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-01-11-08_22_00_UTC) 2024-01-11T08:22:39.130 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:39.130 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2024-01-11T08:22:39.181 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir6 1m 2024-01-11T08:22:39.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.384+0000 7f9b60bff700 1 Processor -- start 2024-01-11T08:22:39.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.384+0000 7f9b60bff700 1 -- start start 2024-01-11T08:22:39.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.385+0000 7f9b60bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9b5c13fbb0 0x7f9b5c0ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:39.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.386+0000 7f9b60bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b5c15a730 0x7f9b5c0c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:39.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.386+0000 7f9b60bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b5c0c6b30 con 0x7f9b5c13ff80 2024-01-11T08:22:39.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.386+0000 7f9b60bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b5c0c6800 con 0x7f9b5c15a730 2024-01-11T08:22:39.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.386+0000 7f9b60bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c0cb240 con 0x7f9b5c13fbb0 2024-01-11T08:22:39.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.386+0000 7f9b5ad9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b5c15a730 0x7f9b5c0c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:39.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.386+0000 7f9b5ad9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b5c15a730 0x7f9b5c0c6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43984/0 (socket says 172.21.15.27:43984) 2024-01-11T08:22:39.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.386+0000 7f9b5ad9d700 1 -- 172.21.15.27:0/749827135 learned_addr learned my addr 172.21.15.27:0/749827135 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:39.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.386+0000 7f9b5a59c700 1 --2- 172.21.15.27:0/749827135 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9b5c13fbb0 0x7f9b5c0ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:39.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.386+0000 7f9b5ad9d700 1 -- 172.21.15.27:0/749827135 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9b5c13fbb0 msgr2=0x7f9b5c0ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:39.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.386+0000 7f9b5ad9d700 1 --2- 172.21.15.27:0/749827135 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9b5c13fbb0 0x7f9b5c0ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:39.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.386+0000 7f9b5ad9d700 1 -- 172.21.15.27:0/749827135 >> v1:172.21.15.77:6789/0 conn(0x7f9b5c13ff80 legacy=0x7f9b5c15a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:39.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.386+0000 7f9b5ad9d700 1 -- 172.21.15.27:0/749827135 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b5c0cb520 con 0x7f9b5c15a730 2024-01-11T08:22:39.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.386+0000 7f9b5959a700 1 -- 172.21.15.27:0/749827135 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814436130 0 0) 0x7f9b5c0c6b30 con 0x7f9b5c13ff80 2024-01-11T08:22:39.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.387+0000 7f9b5ad9d700 1 --2- 172.21.15.27:0/749827135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b5c15a730 0x7f9b5c0c6fc0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9b54009f20 tx=0x7f9b54009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c9d5d250c28a3c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:39.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.387+0000 7f9b5959a700 1 -- 172.21.15.27:0/749827135 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5400e070 con 0x7f9b5c15a730 2024-01-11T08:22:39.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.387+0000 7f9b5959a700 1 -- 172.21.15.27:0/749827135 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b5400b9d0 con 0x7f9b5c15a730 2024-01-11T08:22:39.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.387+0000 7f9b5959a700 1 -- 172.21.15.27:0/749827135 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b540092e0 con 0x7f9b5c15a730 2024-01-11T08:22:39.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.388+0000 7f9b60bff700 1 -- 172.21.15.27:0/749827135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b5c15a730 msgr2=0x7f9b5c0c6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:39.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.388+0000 7f9b60bff700 1 --2- 172.21.15.27:0/749827135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b5c15a730 0x7f9b5c0c6fc0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9b54009f20 tx=0x7f9b54009840 comp rx=0 tx=0).stop 2024-01-11T08:22:39.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.388+0000 7f9b60bff700 1 -- 172.21.15.27:0/749827135 shutdown_connections 2024-01-11T08:22:39.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.388+0000 7f9b60bff700 1 --2- 172.21.15.27:0/749827135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b5c15a730 0x7f9b5c0c6fc0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:39.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.388+0000 7f9b60bff700 1 --2- 172.21.15.27:0/749827135 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9b5c13fbb0 0x7f9b5c0ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:39.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.388+0000 7f9b60bff700 1 -- 172.21.15.27:0/749827135 >> 172.21.15.27:0/749827135 conn(0x7f9b5c0680a0 msgr2=0x7f9b5c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:39.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.388+0000 7f9b60bff700 1 -- 172.21.15.27:0/749827135 shutdown_connections 2024-01-11T08:22:39.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.388+0000 7f9b60bff700 1 -- 172.21.15.27:0/749827135 wait complete. 2024-01-11T08:22:39.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.389+0000 7f9b60bff700 1 Processor -- start 2024-01-11T08:22:39.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.389+0000 7f9b60bff700 1 -- start start 2024-01-11T08:22:39.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.390+0000 7f9b60bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b5c13fbb0 0x7f9b5c0b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:39.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.390+0000 7f9b60bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b5c13ff80 0x7f9b5c13b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:39.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.390+0000 7f9b60bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9b5c15a730 0x7f9b5c157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:39.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.390+0000 7f9b60bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c159b80 con 0x7f9b5c13fbb0 2024-01-11T08:22:39.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.390+0000 7f9b60bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c13bcc0 con 0x7f9b5c13ff80 2024-01-11T08:22:39.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.390+0000 7f9b60bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c157e60 con 0x7f9b5c15a730 2024-01-11T08:22:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.391+0000 7f9b5a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b5c13fbb0 0x7f9b5c0b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.391+0000 7f9b5a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b5c13fbb0 0x7f9b5c0b0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43998/0 (socket says 172.21.15.27:43998) 2024-01-11T08:22:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.391+0000 7f9b5a59c700 1 -- 172.21.15.27:0/2263508404 learned_addr learned my addr 172.21.15.27:0/2263508404 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.391+0000 7f9b59d9b700 1 --2- 172.21.15.27:0/2263508404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b5c13ff80 0x7f9b5c13b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.391+0000 7f9b5ad9d700 1 --2- 172.21.15.27:0/2263508404 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9b5c15a730 0x7f9b5c157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.391+0000 7f9b5a59c700 1 -- 172.21.15.27:0/2263508404 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9b5c15a730 msgr2=0x7f9b5c157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.391+0000 7f9b5a59c700 1 --2- 172.21.15.27:0/2263508404 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9b5c15a730 0x7f9b5c157570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.391+0000 7f9b5a59c700 1 -- 172.21.15.27:0/2263508404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b5c13ff80 msgr2=0x7f9b5c13b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.391+0000 7f9b5a59c700 1 --2- 172.21.15.27:0/2263508404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b5c13ff80 0x7f9b5c13b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.391+0000 7f9b5a59c700 1 -- 172.21.15.27:0/2263508404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b54009ae0 con 0x7f9b5c13fbb0 2024-01-11T08:22:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.391+0000 7f9b5a59c700 1 --2- 172.21.15.27:0/2263508404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b5c13fbb0 0x7f9b5c0b0240 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c00ec80 tx=0x7f9b4c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.392+0000 7f9b4a7fc700 1 -- 172.21.15.27:0/2263508404 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c00c850 con 0x7f9b5c13fbb0 2024-01-11T08:22:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.392+0000 7f9b4a7fc700 1 -- 172.21.15.27:0/2263508404 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b4c00c9b0 con 0x7f9b5c13fbb0 2024-01-11T08:22:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.392+0000 7f9b4a7fc700 1 -- 172.21.15.27:0/2263508404 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c00cd30 con 0x7f9b5c13fbb0 2024-01-11T08:22:39.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.392+0000 7f9b60bff700 1 -- 172.21.15.27:0/2263508404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b5c158140 con 0x7f9b5c13fbb0 2024-01-11T08:22:39.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.392+0000 7f9b60bff700 1 -- 172.21.15.27:0/2263508404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b5c0c12e0 con 0x7f9b5c13fbb0 2024-01-11T08:22:39.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.393+0000 7f9b60bff700 1 -- 172.21.15.27:0/2263508404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b5c04c3c0 con 0x7f9b5c13fbb0 2024-01-11T08:22:39.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.394+0000 7f9b4a7fc700 1 -- 172.21.15.27:0/2263508404 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f9b4c013680 con 0x7f9b5c13fbb0 2024-01-11T08:22:39.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.395+0000 7f9b4a7fc700 1 --2- 172.21.15.27:0/2263508404 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9b3c067130 0x7f9b3c0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:39.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.395+0000 7f9b4a7fc700 1 -- 172.21.15.27:0/2263508404 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6467+0+0 (secure 0 0 0) 0x7f9b4c0852b0 con 0x7f9b5c13fbb0 2024-01-11T08:22:39.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.395+0000 7f9b59d9b700 1 --2- 172.21.15.27:0/2263508404 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9b3c067130 0x7f9b3c0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:39.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.396+0000 7f9b59d9b700 1 --2- 172.21.15.27:0/2263508404 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9b3c067130 0x7f9b3c0695e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c167830 tx=0x7f9b50007d30 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:39.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.399+0000 7f9b4a7fc700 1 -- 172.21.15.27:0/2263508404 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9b4c051f00 con 0x7f9b5c13fbb0 2024-01-11T08:22:39.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.625+0000 7f9b60bff700 1 -- 172.21.15.27:0/2263508404 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b5c0cb800 con 0x7f9b3c067130 2024-01-11T08:22:39.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.713+0000 7f9b4a7fc700 1 -- 172.21.15.27:0/2263508404 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f9b5c0cb800 con 0x7f9b3c067130 2024-01-11T08:22:39.712 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule removed for path /snap_test_dir1/dir6 2024-01-11T08:22:39.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.717+0000 7f9b60bff700 1 -- 172.21.15.27:0/2263508404 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9b3c067130 msgr2=0x7f9b3c0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:39.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.717+0000 7f9b60bff700 1 --2- 172.21.15.27:0/2263508404 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9b3c067130 0x7f9b3c0695e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c167830 tx=0x7f9b50007d30 comp rx=0 tx=0).stop 2024-01-11T08:22:39.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.717+0000 7f9b60bff700 1 -- 172.21.15.27:0/2263508404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b5c13fbb0 msgr2=0x7f9b5c0b0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:39.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.717+0000 7f9b60bff700 1 --2- 172.21.15.27:0/2263508404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b5c13fbb0 0x7f9b5c0b0240 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c00ec80 tx=0x7f9b4c009120 comp rx=0 tx=0).stop 2024-01-11T08:22:39.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.718+0000 7f9b60bff700 1 -- 172.21.15.27:0/2263508404 shutdown_connections 2024-01-11T08:22:39.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.718+0000 7f9b60bff700 1 --2- 172.21.15.27:0/2263508404 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9b3c067130 0x7f9b3c0695e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:39.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.718+0000 7f9b60bff700 1 --2- 172.21.15.27:0/2263508404 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9b5c15a730 0x7f9b5c157570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:39.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.718+0000 7f9b60bff700 1 --2- 172.21.15.27:0/2263508404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b5c13fbb0 0x7f9b5c0b0240 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:39.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.718+0000 7f9b60bff700 1 --2- 172.21.15.27:0/2263508404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b5c13ff80 0x7f9b5c13b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:39.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.718+0000 7f9b60bff700 1 -- 172.21.15.27:0/2263508404 >> 172.21.15.27:0/2263508404 conn(0x7f9b5c0680a0 msgr2=0x7f9b5c13d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:39.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.718+0000 7f9b60bff700 1 -- 172.21.15.27:0/2263508404 shutdown_connections 2024-01-11T08:22:39.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:39.718+0000 7f9b60bff700 1 -- 172.21.15.27:0/2263508404 wait complete. 2024-01-11T08:22:39.744 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2024-01-11T08:22:39.745 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:39.745 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2024-01-11T08:22:39.820 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:39.821 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:39.821 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:39.821 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:39.821 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:39.821 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:39.821 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:39.821 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:39.821 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:39.821 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:39.821 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:39.822 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:39.822 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:39.822 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_12_00_UTC) 2024-01-11T08:22:39.879 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:39.879 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:39.879 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_13_00_UTC) 2024-01-11T08:22:39.934 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:39.934 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:39.934 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_14_00_UTC) 2024-01-11T08:22:39.989 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:39.990 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:39.990 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_15_00_UTC) 2024-01-11T08:22:40.045 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:40.045 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:40.046 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_16_00_UTC) 2024-01-11T08:22:40.105 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:40.105 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:40.105 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_17_00_UTC) 2024-01-11T08:22:40.160 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:40.161 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:40.161 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_18_00_UTC) 2024-01-11T08:22:40.216 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:40.216 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:40.216 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_19_00_UTC) 2024-01-11T08:22:40.270 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:40.271 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:40.271 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_20_00_UTC) 2024-01-11T08:22:40.325 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:40.325 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:40.325 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_21_00_UTC) 2024-01-11T08:22:40.379 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:40.379 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:40.380 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-01-11-08_22_00_UTC) 2024-01-11T08:22:40.434 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:40.434 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2024-01-11T08:22:40.486 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir7 1m 2024-01-11T08:22:40.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.689+0000 7f0b97bff700 1 Processor -- start 2024-01-11T08:22:40.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.690+0000 7f0b97bff700 1 -- start start 2024-01-11T08:22:40.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.691+0000 7f0b97bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0b90067c30 0x7f0b900cae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:40.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.691+0000 7f0b97bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b900cb3d0 0x7f0b900c7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:40.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.691+0000 7f0b97bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9006c3c0 con 0x7f0b90067860 2024-01-11T08:22:40.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.691+0000 7f0b97bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b900cb9c0 con 0x7f0b90067c30 2024-01-11T08:22:40.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.691+0000 7f0b97bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9006f650 con 0x7f0b900cb3d0 2024-01-11T08:22:40.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.691+0000 7f0b9599b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0b90067860 0x7f0b9006c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53852/0 (socket says 172.21.15.27:53852) 2024-01-11T08:22:40.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.691+0000 7f0b9599b700 1 -- 172.21.15.27:0/2865051242 learned_addr learned my addr 172.21.15.27:0/2865051242 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:40.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.692+0000 7f0b9619c700 1 --2- 172.21.15.27:0/2865051242 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b900cb3d0 0x7f0b900c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:40.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.692+0000 7f0b9519a700 1 --2- 172.21.15.27:0/2865051242 >> v2:172.21.15.77:3300/0 conn(0x7f0b90067c30 0x7f0b900cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:40.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.692+0000 7f0b94999700 1 -- 172.21.15.27:0/2865051242 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124921725 0 0) 0x7f0b9006c3c0 con 0x7f0b90067860 2024-01-11T08:22:40.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.692+0000 7f0b94999700 1 -- 172.21.15.27:0/2865051242 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b7c003570 con 0x7f0b90067860 2024-01-11T08:22:40.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.692+0000 7f0b9519a700 1 -- 172.21.15.27:0/2865051242 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b900cb3d0 msgr2=0x7f0b900c7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:40.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.692+0000 7f0b9519a700 1 --2- 172.21.15.27:0/2865051242 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b900cb3d0 0x7f0b900c7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:40.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.692+0000 7f0b94999700 1 -- 172.21.15.27:0/2865051242 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3907380 0 0) 0x7f0b7c003570 con 0x7f0b90067860 2024-01-11T08:22:40.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.692+0000 7f0b9519a700 1 -- 172.21.15.27:0/2865051242 >> v1:172.21.15.27:6789/0 conn(0x7f0b90067860 legacy=0x7f0b9006c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:40.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.692+0000 7f0b9519a700 1 -- 172.21.15.27:0/2865051242 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9006ba90 con 0x7f0b90067c30 2024-01-11T08:22:40.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.692+0000 7f0b9519a700 1 --2- 172.21.15.27:0/2865051242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b90067c30 0x7f0b900cae90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0b800098e0 tx=0x7f0b800091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec230302acacea93 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:40.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.693+0000 7f0b94999700 1 -- 172.21.15.27:0/2865051242 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c001470 con 0x7f0b90067c30 2024-01-11T08:22:40.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.693+0000 7f0b94999700 1 -- 172.21.15.27:0/2865051242 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b80011420 con 0x7f0b90067c30 2024-01-11T08:22:40.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.693+0000 7f0b94999700 1 -- 172.21.15.27:0/2865051242 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b800117c0 con 0x7f0b90067c30 2024-01-11T08:22:40.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.693+0000 7f0b97bff700 1 -- 172.21.15.27:0/2865051242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b90067c30 msgr2=0x7f0b900cae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:40.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.693+0000 7f0b97bff700 1 --2- 172.21.15.27:0/2865051242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b90067c30 0x7f0b900cae90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0b800098e0 tx=0x7f0b800091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:40.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.693+0000 7f0b97bff700 1 -- 172.21.15.27:0/2865051242 shutdown_connections 2024-01-11T08:22:40.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.693+0000 7f0b97bff700 1 --2- 172.21.15.27:0/2865051242 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b900cb3d0 0x7f0b900c7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:40.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.693+0000 7f0b97bff700 1 --2- 172.21.15.27:0/2865051242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b90067c30 0x7f0b900cae90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:40.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.693+0000 7f0b97bff700 1 -- 172.21.15.27:0/2865051242 >> 172.21.15.27:0/2865051242 conn(0x7f0b9006b1f0 msgr2=0x7f0b900cd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:40.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.694+0000 7f0b97bff700 1 -- 172.21.15.27:0/2865051242 shutdown_connections 2024-01-11T08:22:40.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.694+0000 7f0b97bff700 1 -- 172.21.15.27:0/2865051242 wait complete. 2024-01-11T08:22:40.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.695+0000 7f0b97bff700 1 Processor -- start 2024-01-11T08:22:40.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.695+0000 7f0b97bff700 1 -- start start 2024-01-11T08:22:40.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.696+0000 7f0b97bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b90067860 0x7f0b900af290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:40.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.696+0000 7f0b97bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b90067c30 0x7f0b9015d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:40.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.696+0000 7f0b97bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b900cb3d0 0x7f0b9016d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:40.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.696+0000 7f0b97bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b90159d10 con 0x7f0b900cb3d0 2024-01-11T08:22:40.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.696+0000 7f0b97bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b90159f60 con 0x7f0b90067860 2024-01-11T08:22:40.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.696+0000 7f0b97bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9015a240 con 0x7f0b90067c30 2024-01-11T08:22:40.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.696+0000 7f0b9619c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b900cb3d0 0x7f0b9016d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:40.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.697+0000 7f0b9619c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b900cb3d0 0x7f0b9016d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44000/0 (socket says 172.21.15.27:44000) 2024-01-11T08:22:40.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.697+0000 7f0b9619c700 1 -- 172.21.15.27:0/3254025344 learned_addr learned my addr 172.21.15.27:0/3254025344 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:40.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.697+0000 7f0b9599b700 1 --2- 172.21.15.27:0/3254025344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b90067860 0x7f0b900af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:40.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.697+0000 7f0b9519a700 1 --2- 172.21.15.27:0/3254025344 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b90067c30 0x7f0b9015d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:40.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.697+0000 7f0b9619c700 1 -- 172.21.15.27:0/3254025344 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b90067c30 msgr2=0x7f0b9015d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:40.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.697+0000 7f0b9619c700 1 --2- 172.21.15.27:0/3254025344 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b90067c30 0x7f0b9015d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:40.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.697+0000 7f0b9619c700 1 -- 172.21.15.27:0/3254025344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b90067860 msgr2=0x7f0b900af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:40.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.697+0000 7f0b9619c700 1 --2- 172.21.15.27:0/3254025344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b90067860 0x7f0b900af290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:40.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.697+0000 7f0b9619c700 1 -- 172.21.15.27:0/3254025344 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b80009420 con 0x7f0b900cb3d0 2024-01-11T08:22:40.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.697+0000 7f0b9619c700 1 --2- 172.21.15.27:0/3254025344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b900cb3d0 0x7f0b9016d360 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0b88009df0 tx=0x7f0b8800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:40.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.697+0000 7f0b85ffb700 1 -- 172.21.15.27:0/3254025344 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8800caa0 con 0x7f0b900cb3d0 2024-01-11T08:22:40.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.697+0000 7f0b85ffb700 1 -- 172.21.15.27:0/3254025344 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8800f040 con 0x7f0b900cb3d0 2024-01-11T08:22:40.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.698+0000 7f0b97bff700 1 -- 172.21.15.27:0/3254025344 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b9015a520 con 0x7f0b900cb3d0 2024-01-11T08:22:40.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.698+0000 7f0b85ffb700 1 -- 172.21.15.27:0/3254025344 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8800ce70 con 0x7f0b900cb3d0 2024-01-11T08:22:40.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.698+0000 7f0b97bff700 1 -- 172.21.15.27:0/3254025344 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9015ad20 con 0x7f0b900cb3d0 2024-01-11T08:22:40.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.699+0000 7f0b97bff700 1 -- 172.21.15.27:0/3254025344 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b9015aa40 con 0x7f0b900cb3d0 2024-01-11T08:22:40.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.700+0000 7f0b85ffb700 1 -- 172.21.15.27:0/3254025344 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f0b88014690 con 0x7f0b900cb3d0 2024-01-11T08:22:40.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.701+0000 7f0b85ffb700 1 --2- 172.21.15.27:0/3254025344 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0b74067130 0x7f0b740695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:40.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.701+0000 7f0b85ffb700 1 -- 172.21.15.27:0/3254025344 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6435+0+0 (secure 0 0 0) 0x7f0b880854c0 con 0x7f0b900cb3d0 2024-01-11T08:22:40.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.701+0000 7f0b9599b700 1 --2- 172.21.15.27:0/3254025344 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0b74067130 0x7f0b740695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:40.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.702+0000 7f0b9599b700 1 --2- 172.21.15.27:0/3254025344 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0b74067130 0x7f0b740695e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c001ed0 tx=0x7f0b8c0019d0 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:40.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.705+0000 7f0b85ffb700 1 -- 172.21.15.27:0/3254025344 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0b9015aa40 con 0x7f0b900cb3d0 2024-01-11T08:22:40.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:40.961+0000 7f0b97bff700 1 -- 172.21.15.27:0/3254025344 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b9015a0d0 con 0x7f0b74067130 2024-01-11T08:22:41.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:41.077+0000 7f0b85ffb700 1 -- 172.21.15.27:0/3254025344 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f0b9015a0d0 con 0x7f0b74067130 2024-01-11T08:22:41.077 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule removed for path /snap_test_dir1/dir7 2024-01-11T08:22:41.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:41.079+0000 7f0b97bff700 1 -- 172.21.15.27:0/3254025344 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0b74067130 msgr2=0x7f0b740695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:41.079+0000 7f0b97bff700 1 --2- 172.21.15.27:0/3254025344 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0b74067130 0x7f0b740695e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c001ed0 tx=0x7f0b8c0019d0 comp rx=0 tx=0).stop 2024-01-11T08:22:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:41.079+0000 7f0b97bff700 1 -- 172.21.15.27:0/3254025344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b900cb3d0 msgr2=0x7f0b9016d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:41.079+0000 7f0b97bff700 1 --2- 172.21.15.27:0/3254025344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b900cb3d0 0x7f0b9016d360 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0b88009df0 tx=0x7f0b8800b040 comp rx=0 tx=0).stop 2024-01-11T08:22:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:41.079+0000 7f0b97bff700 1 -- 172.21.15.27:0/3254025344 shutdown_connections 2024-01-11T08:22:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:41.079+0000 7f0b97bff700 1 --2- 172.21.15.27:0/3254025344 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0b74067130 0x7f0b740695e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:41.079+0000 7f0b97bff700 1 --2- 172.21.15.27:0/3254025344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b900cb3d0 0x7f0b9016d360 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:41.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:41.079+0000 7f0b97bff700 1 --2- 172.21.15.27:0/3254025344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b90067860 0x7f0b900af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:41.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:41.079+0000 7f0b97bff700 1 --2- 172.21.15.27:0/3254025344 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b90067c30 0x7f0b9015d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:41.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:41.079+0000 7f0b97bff700 1 -- 172.21.15.27:0/3254025344 >> 172.21.15.27:0/3254025344 conn(0x7f0b9006b1f0 msgr2=0x7f0b9006e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:41.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:41.079+0000 7f0b97bff700 1 -- 172.21.15.27:0/3254025344 shutdown_connections 2024-01-11T08:22:41.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:41.080+0000 7f0b97bff700 1 -- 172.21.15.27:0/3254025344 wait complete. 2024-01-11T08:22:41.097 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2024-01-11T08:22:41.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:41.097 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2024-01-11T08:22:41.146 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:41.146 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:41.146 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:41.146 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:41.146 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:41.147 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:41.148 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:41.148 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:41.148 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_12_00_UTC) 2024-01-11T08:22:41.206 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:41.206 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:41.207 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_13_00_UTC) 2024-01-11T08:22:41.262 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:41.262 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:41.262 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_14_00_UTC) 2024-01-11T08:22:41.316 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:41.317 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:41.317 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_15_00_UTC) 2024-01-11T08:22:41.371 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:41.371 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:41.371 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_16_00_UTC) 2024-01-11T08:22:41.428 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:41.429 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:41.429 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_17_00_UTC) 2024-01-11T08:22:41.485 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:41.486 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:41.486 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_18_00_UTC) 2024-01-11T08:22:41.541 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:41.541 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:41.541 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_19_00_UTC) 2024-01-11T08:22:41.596 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:41.596 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:41.596 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_20_00_UTC) 2024-01-11T08:22:41.651 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:41.651 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:41.651 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_21_00_UTC) 2024-01-11T08:22:41.706 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:41.706 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:41.707 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-01-11-08_22_00_UTC) 2024-01-11T08:22:41.795 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:41.795 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2024-01-11T08:22:41.853 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir8 1m 2024-01-11T08:22:42.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.056+0000 7fe208dff700 1 Processor -- start 2024-01-11T08:22:42.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.056+0000 7fe208dff700 1 -- start start 2024-01-11T08:22:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.057+0000 7fe208dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2041577e0 0x7fe2040ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.057+0000 7fe208dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe204069540 0x7fe2040ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.057+0000 7fe208dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2040699d0 con 0x7fe204157410 2024-01-11T08:22:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.057+0000 7fe208dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe204069cb0 con 0x7fe2041577e0 2024-01-11T08:22:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.057+0000 7fe208dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe204069f90 con 0x7fe204069540 2024-01-11T08:22:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.057+0000 7fe201d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2041577e0 0x7fe2040ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.058+0000 7fe201d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2041577e0 0x7fe2040ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44010/0 (socket says 172.21.15.27:44010) 2024-01-11T08:22:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.058+0000 7fe201d9b700 1 -- 172.21.15.27:0/2453649680 learned_addr learned my addr 172.21.15.27:0/2453649680 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.058+0000 7fe202d9d700 1 --2- 172.21.15.27:0/2453649680 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe204069540 0x7fe2040ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:42.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.058+0000 7fe201d9b700 1 -- 172.21.15.27:0/2453649680 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe204069540 msgr2=0x7fe2040ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:42.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.058+0000 7fe201d9b700 1 --2- 172.21.15.27:0/2453649680 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe204069540 0x7fe2040ae540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:42.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.058+0000 7fe201d9b700 1 -- 172.21.15.27:0/2453649680 >> v1:172.21.15.77:6789/0 conn(0x7fe204157410 legacy=0x7fe2040ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:42.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.058+0000 7fe201d9b700 1 -- 172.21.15.27:0/2453649680 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20406a270 con 0x7fe2041577e0 2024-01-11T08:22:42.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.058+0000 7fe20159a700 1 -- 172.21.15.27:0/2453649680 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605216857 0 0) 0x7fe2040699d0 con 0x7fe204157410 2024-01-11T08:22:42.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.058+0000 7fe201d9b700 1 --2- 172.21.15.27:0/2453649680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2041577e0 0x7fe2040ae000 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe1f80099b0 tx=0x7fe1f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c099bd36d846eb4d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:42.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.059+0000 7fe20159a700 1 -- 172.21.15.27:0/2453649680 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8009ea0 con 0x7fe2041577e0 2024-01-11T08:22:42.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.059+0000 7fe20159a700 1 -- 172.21.15.27:0/2453649680 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f800b860 con 0x7fe2041577e0 2024-01-11T08:22:42.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.059+0000 7fe20159a700 1 -- 172.21.15.27:0/2453649680 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f800bc00 con 0x7fe2041577e0 2024-01-11T08:22:42.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.059+0000 7fe208dff700 1 -- 172.21.15.27:0/2453649680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2041577e0 msgr2=0x7fe2040ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:42.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.059+0000 7fe208dff700 1 --2- 172.21.15.27:0/2453649680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2041577e0 0x7fe2040ae000 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe1f80099b0 tx=0x7fe1f8009250 comp rx=0 tx=0).stop 2024-01-11T08:22:42.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.059+0000 7fe208dff700 1 -- 172.21.15.27:0/2453649680 shutdown_connections 2024-01-11T08:22:42.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.059+0000 7fe208dff700 1 --2- 172.21.15.27:0/2453649680 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe204069540 0x7fe2040ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:42.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.059+0000 7fe208dff700 1 --2- 172.21.15.27:0/2453649680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2041577e0 0x7fe2040ae000 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:42.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.059+0000 7fe208dff700 1 -- 172.21.15.27:0/2453649680 >> 172.21.15.27:0/2453649680 conn(0x7fe2040680a0 msgr2=0x7fe2040cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:42.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.059+0000 7fe208dff700 1 -- 172.21.15.27:0/2453649680 shutdown_connections 2024-01-11T08:22:42.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.059+0000 7fe208dff700 1 -- 172.21.15.27:0/2453649680 wait complete. 2024-01-11T08:22:42.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.061+0000 7fe208dff700 1 Processor -- start 2024-01-11T08:22:42.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.061+0000 7fe208dff700 1 -- start start 2024-01-11T08:22:42.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.062+0000 7fe208dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe204069540 0x7fe204167fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:42.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.062+0000 7fe208dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe204157410 0x7fe2040b1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:42.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.062+0000 7fe208dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2041577e0 0x7fe2041538e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:42.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.062+0000 7fe208dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2040b0680 con 0x7fe2041577e0 2024-01-11T08:22:42.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.062+0000 7fe208dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2040b0960 con 0x7fe204069540 2024-01-11T08:22:42.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.062+0000 7fe208dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2040b0c40 con 0x7fe204157410 2024-01-11T08:22:42.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.062+0000 7fe202d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2041577e0 0x7fe2041538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:42.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.062+0000 7fe202d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2041577e0 0x7fe2041538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44012/0 (socket says 172.21.15.27:44012) 2024-01-11T08:22:42.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.062+0000 7fe202d9d700 1 -- 172.21.15.27:0/1451828541 learned_addr learned my addr 172.21.15.27:0/1451828541 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:42.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.062+0000 7fe201d9b700 1 --2- 172.21.15.27:0/1451828541 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe204157410 0x7fe2040b1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:42.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.062+0000 7fe20259c700 1 --2- 172.21.15.27:0/1451828541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe204069540 0x7fe204167fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.063+0000 7fe202d9d700 1 -- 172.21.15.27:0/1451828541 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe204157410 msgr2=0x7fe2040b1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.063+0000 7fe202d9d700 1 --2- 172.21.15.27:0/1451828541 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe204157410 0x7fe2040b1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.063+0000 7fe202d9d700 1 -- 172.21.15.27:0/1451828541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe204069540 msgr2=0x7fe204167fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.063+0000 7fe202d9d700 1 --2- 172.21.15.27:0/1451828541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe204069540 0x7fe204167fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.063+0000 7fe202d9d700 1 -- 172.21.15.27:0/1451828541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f80094f0 con 0x7fe2041577e0 2024-01-11T08:22:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.063+0000 7fe202d9d700 1 --2- 172.21.15.27:0/1451828541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2041577e0 0x7fe2041538e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009e90 tx=0x7fe1fc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.063+0000 7fe1f27fc700 1 -- 172.21.15.27:0/1451828541 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc00cc20 con 0x7fe2041577e0 2024-01-11T08:22:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.063+0000 7fe1f27fc700 1 -- 172.21.15.27:0/1451828541 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1fc00f040 con 0x7fe2041577e0 2024-01-11T08:22:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.063+0000 7fe208dff700 1 -- 172.21.15.27:0/1451828541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2040b0f20 con 0x7fe2041577e0 2024-01-11T08:22:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.063+0000 7fe1f27fc700 1 -- 172.21.15.27:0/1451828541 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc014480 con 0x7fe2041577e0 2024-01-11T08:22:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.063+0000 7fe208dff700 1 -- 172.21.15.27:0/1451828541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe204150de0 con 0x7fe2041577e0 2024-01-11T08:22:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.064+0000 7fe208dff700 1 -- 172.21.15.27:0/1451828541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe20404c3c0 con 0x7fe2041577e0 2024-01-11T08:22:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.065+0000 7fe1f27fc700 1 -- 172.21.15.27:0/1451828541 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7fe1fc014830 con 0x7fe2041577e0 2024-01-11T08:22:42.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.067+0000 7fe1f27fc700 1 --2- 172.21.15.27:0/1451828541 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe1e4067130 0x7fe1e40695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:42.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.067+0000 7fe1f27fc700 1 -- 172.21.15.27:0/1451828541 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6499+0+0 (secure 0 0 0) 0x7fe1fc0857b0 con 0x7fe2041577e0 2024-01-11T08:22:42.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.068+0000 7fe20259c700 1 --2- 172.21.15.27:0/1451828541 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe1e4067130 0x7fe1e40695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:42.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.068+0000 7fe20259c700 1 --2- 172.21.15.27:0/1451828541 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe1e4067130 0x7fe1e40695e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe204152640 tx=0x7fe1f4009040 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:42.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.071+0000 7fe1f27fc700 1 -- 172.21.15.27:0/1451828541 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe1fc055d60 con 0x7fe2041577e0 2024-01-11T08:22:42.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.332+0000 7fe208dff700 1 -- 172.21.15.27:0/1451828541 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2040b0ad0 con 0x7fe1e4067130 2024-01-11T08:22:42.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.429+0000 7fe1f27fc700 1 -- 172.21.15.27:0/1451828541 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fe2040b0ad0 con 0x7fe1e4067130 2024-01-11T08:22:42.428 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule removed for path /snap_test_dir1/dir8 2024-01-11T08:22:42.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.431+0000 7fe208dff700 1 -- 172.21.15.27:0/1451828541 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe1e4067130 msgr2=0x7fe1e40695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:42.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.431+0000 7fe208dff700 1 --2- 172.21.15.27:0/1451828541 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe1e4067130 0x7fe1e40695e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe204152640 tx=0x7fe1f4009040 comp rx=0 tx=0).stop 2024-01-11T08:22:42.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.431+0000 7fe208dff700 1 -- 172.21.15.27:0/1451828541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2041577e0 msgr2=0x7fe2041538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:42.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.431+0000 7fe208dff700 1 --2- 172.21.15.27:0/1451828541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2041577e0 0x7fe2041538e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009e90 tx=0x7fe1fc00b040 comp rx=0 tx=0).stop 2024-01-11T08:22:42.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.431+0000 7fe208dff700 1 -- 172.21.15.27:0/1451828541 shutdown_connections 2024-01-11T08:22:42.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.431+0000 7fe208dff700 1 --2- 172.21.15.27:0/1451828541 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe1e4067130 0x7fe1e40695e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:42.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.431+0000 7fe208dff700 1 --2- 172.21.15.27:0/1451828541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2041577e0 0x7fe2041538e0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:42.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.431+0000 7fe208dff700 1 --2- 172.21.15.27:0/1451828541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe204069540 0x7fe204167fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:42.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.431+0000 7fe208dff700 1 --2- 172.21.15.27:0/1451828541 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe204157410 0x7fe2040b1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:42.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.431+0000 7fe208dff700 1 -- 172.21.15.27:0/1451828541 >> 172.21.15.27:0/1451828541 conn(0x7fe2040680a0 msgr2=0x7fe204152c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:42.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.431+0000 7fe208dff700 1 -- 172.21.15.27:0/1451828541 shutdown_connections 2024-01-11T08:22:42.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:42.431+0000 7fe208dff700 1 -- 172.21.15.27:0/1451828541 wait complete. 2024-01-11T08:22:42.448 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2024-01-11T08:22:42.448 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:42.448 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2024-01-11T08:22:42.497 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:42.497 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:42.497 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:42.497 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:42.497 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:42.497 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:42.497 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:42.497 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:42.497 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:42.497 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:42.497 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:42.498 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:42.498 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:42.499 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_12_00_UTC) 2024-01-11T08:22:42.554 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:42.555 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:42.555 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_13_00_UTC) 2024-01-11T08:22:42.609 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:42.609 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:42.609 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_14_00_UTC) 2024-01-11T08:22:42.664 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:42.664 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:42.664 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_15_00_UTC) 2024-01-11T08:22:42.769 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:42.769 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:42.769 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_16_00_UTC) 2024-01-11T08:22:42.837 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:42.837 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:42.837 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_17_00_UTC) 2024-01-11T08:22:42.892 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:42.892 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:42.893 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_18_00_UTC) 2024-01-11T08:22:42.963 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:42.963 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:42.963 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_19_00_UTC) 2024-01-11T08:22:43.022 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:43.022 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:43.022 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_20_00_UTC) 2024-01-11T08:22:43.077 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:43.077 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:43.077 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_21_00_UTC) 2024-01-11T08:22:43.132 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:43.132 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:43.132 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-01-11-08_22_00_UTC) 2024-01-11T08:22:43.187 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:43.187 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2024-01-11T08:22:43.238 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir9 1m 2024-01-11T08:22:43.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.440+0000 7f22c81ff700 1 Processor -- start 2024-01-11T08:22:43.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.440+0000 7f22c81ff700 1 -- start start 2024-01-11T08:22:43.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.441+0000 7f22c81ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f22c013fbb0 0x7f22c00ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:43.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.441+0000 7f22c81ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22c013ff80 0x7f22c015a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:43.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.442+0000 7f22c81ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c00c6b30 con 0x7f22c00cace0 2024-01-11T08:22:43.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.442+0000 7f22c81ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22c00c6800 con 0x7f22c013fbb0 2024-01-11T08:22:43.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.442+0000 7f22c81ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c00cb240 con 0x7f22c013ff80 2024-01-11T08:22:43.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.442+0000 7f22c5f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f22c013fbb0 0x7f22c00ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:43.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.442+0000 7f22c5f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f22c013fbb0 0x7f22c00ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44022/0 (socket says 172.21.15.27:44022) 2024-01-11T08:22:43.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.442+0000 7f22c579a700 1 --2- 172.21.15.27:0/422713103 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22c013ff80 0x7f22c015a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:43.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.442+0000 7f22c5f9b700 1 -- 172.21.15.27:0/422713103 learned_addr learned my addr 172.21.15.27:0/422713103 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:43.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.442+0000 7f22c5f9b700 1 -- 172.21.15.27:0/422713103 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22c013ff80 msgr2=0x7f22c015a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:43.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.442+0000 7f22c5f9b700 1 --2- 172.21.15.27:0/422713103 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22c013ff80 0x7f22c015a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:43.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.442+0000 7f22c5f9b700 1 -- 172.21.15.27:0/422713103 >> v1:172.21.15.77:6789/0 conn(0x7f22c00cace0 legacy=0x7f22c006e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:43.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.442+0000 7f22c5f9b700 1 -- 172.21.15.27:0/422713103 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c00cb520 con 0x7f22c013fbb0 2024-01-11T08:22:43.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.442+0000 7f22c4f99700 1 -- 172.21.15.27:0/422713103 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067442120 0 0) 0x7f22c00c6b30 con 0x7f22c00cace0 2024-01-11T08:22:43.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.443+0000 7f22c5f9b700 1 --2- 172.21.15.27:0/422713103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22c013fbb0 0x7f22c00ca7a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f22bc009990 tx=0x7f22bc009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc6c797f998abf88 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:43.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.443+0000 7f22c4f99700 1 -- 172.21.15.27:0/422713103 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22bc009e80 con 0x7f22c013fbb0 2024-01-11T08:22:43.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.443+0000 7f22c4f99700 1 -- 172.21.15.27:0/422713103 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22bc005c10 con 0x7f22c013fbb0 2024-01-11T08:22:43.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.443+0000 7f22c4f99700 1 -- 172.21.15.27:0/422713103 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22bc00b650 con 0x7f22c013fbb0 2024-01-11T08:22:43.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.443+0000 7f22c81ff700 1 -- 172.21.15.27:0/422713103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22c013fbb0 msgr2=0x7f22c00ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:43.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.443+0000 7f22c81ff700 1 --2- 172.21.15.27:0/422713103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22c013fbb0 0x7f22c00ca7a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f22bc009990 tx=0x7f22bc009220 comp rx=0 tx=0).stop 2024-01-11T08:22:43.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.444+0000 7f22c81ff700 1 -- 172.21.15.27:0/422713103 shutdown_connections 2024-01-11T08:22:43.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.444+0000 7f22c81ff700 1 --2- 172.21.15.27:0/422713103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22c013fbb0 0x7f22c00ca7a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:43.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.444+0000 7f22c81ff700 1 --2- 172.21.15.27:0/422713103 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22c013ff80 0x7f22c015a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:43.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.444+0000 7f22c81ff700 1 -- 172.21.15.27:0/422713103 >> 172.21.15.27:0/422713103 conn(0x7f22c00680a0 msgr2=0x7f22c00cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:43.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.444+0000 7f22c81ff700 1 -- 172.21.15.27:0/422713103 shutdown_connections 2024-01-11T08:22:43.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.444+0000 7f22c81ff700 1 -- 172.21.15.27:0/422713103 wait complete. 2024-01-11T08:22:43.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.445+0000 7f22c81ff700 1 Processor -- start 2024-01-11T08:22:43.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.445+0000 7f22c81ff700 1 -- start start 2024-01-11T08:22:43.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.446+0000 7f22c81ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22c00cace0 0x7f22c0158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:43.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.446+0000 7f22c81ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22c013fbb0 0x7f22c0155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:43.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.446+0000 7f22c81ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22c013ff80 0x7f22c0155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:43.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.446+0000 7f22c81ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c00c40a0 con 0x7f22c013fbb0 2024-01-11T08:22:43.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.446+0000 7f22c81ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c00c4380 con 0x7f22c00cace0 2024-01-11T08:22:43.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.446+0000 7f22c81ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c00c4660 con 0x7f22c013ff80 2024-01-11T08:22:43.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.446+0000 7f22c5f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22c00cace0 0x7f22c0158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:43.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.446+0000 7f22c579a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22c013fbb0 0x7f22c0155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:43.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.447+0000 7f22c5f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22c00cace0 0x7f22c0158cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:42230/0 (socket says 172.21.15.27:42230) 2024-01-11T08:22:43.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.447+0000 7f22c5f9b700 1 -- 172.21.15.27:0/380666507 learned_addr learned my addr 172.21.15.27:0/380666507 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:43.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.446+0000 7f22c679c700 1 --2- 172.21.15.27:0/380666507 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22c013ff80 0x7f22c0155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:43.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.447+0000 7f22c579a700 1 -- 172.21.15.27:0/380666507 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22c013ff80 msgr2=0x7f22c0155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:43.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.447+0000 7f22c579a700 1 --2- 172.21.15.27:0/380666507 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22c013ff80 0x7f22c0155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:43.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.447+0000 7f22c579a700 1 -- 172.21.15.27:0/380666507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22c00cace0 msgr2=0x7f22c0158cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:43.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.447+0000 7f22c579a700 1 --2- 172.21.15.27:0/380666507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22c00cace0 0x7f22c0158cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:43.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.447+0000 7f22c579a700 1 -- 172.21.15.27:0/380666507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22bc009480 con 0x7f22c013fbb0 2024-01-11T08:22:43.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.447+0000 7f22c5f9b700 1 --2- 172.21.15.27:0/380666507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22c00cace0 0x7f22c0158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:22:43.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.447+0000 7f22c579a700 1 --2- 172.21.15.27:0/380666507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22c013fbb0 0x7f22c0155a00 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f22b000e900 tx=0x7f22b00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:43.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.447+0000 7f22b5ffb700 1 -- 172.21.15.27:0/380666507 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b0013680 con 0x7f22c013fbb0 2024-01-11T08:22:43.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.447+0000 7f22b5ffb700 1 -- 172.21.15.27:0/380666507 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b00137e0 con 0x7f22c013fbb0 2024-01-11T08:22:43.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.447+0000 7f22b5ffb700 1 -- 172.21.15.27:0/380666507 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b0013ba0 con 0x7f22c013fbb0 2024-01-11T08:22:43.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.448+0000 7f22c81ff700 1 -- 172.21.15.27:0/380666507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c00c4940 con 0x7f22c013fbb0 2024-01-11T08:22:43.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.448+0000 7f22c81ff700 1 -- 172.21.15.27:0/380666507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c0159230 con 0x7f22c013fbb0 2024-01-11T08:22:43.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.449+0000 7f22c81ff700 1 -- 172.21.15.27:0/380666507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c004c3c0 con 0x7f22c013fbb0 2024-01-11T08:22:43.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.450+0000 7f22b5ffb700 1 -- 172.21.15.27:0/380666507 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7f22b00123d0 con 0x7f22c013fbb0 2024-01-11T08:22:43.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.450+0000 7f22b5ffb700 1 --2- 172.21.15.27:0/380666507 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f22a4067130 0x7f22a40695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:43.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.451+0000 7f22b5ffb700 1 -- 172.21.15.27:0/380666507 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6499+0+0 (secure 0 0 0) 0x7f22b0084ce0 con 0x7f22c013fbb0 2024-01-11T08:22:43.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.451+0000 7f22c5f9b700 1 --2- 172.21.15.27:0/380666507 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f22a4067130 0x7f22a40695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:43.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.451+0000 7f22c5f9b700 1 --2- 172.21.15.27:0/380666507 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f22a4067130 0x7f22a40695e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f22bc0030c0 tx=0x7f22bc016000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:43.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.455+0000 7f22b5ffb700 1 -- 172.21.15.27:0/380666507 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f22b0051910 con 0x7f22c013fbb0 2024-01-11T08:22:43.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.686+0000 7f22c81ff700 1 -- 172.21.15.27:0/380666507 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f22c00c44f0 con 0x7f22a4067130 2024-01-11T08:22:43.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.782+0000 7f22b5ffb700 1 -- 172.21.15.27:0/380666507 <== mgr.4793 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f22c00c44f0 con 0x7f22a4067130 2024-01-11T08:22:43.782 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule removed for path /snap_test_dir1/dir9 2024-01-11T08:22:43.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.785+0000 7f22c81ff700 1 -- 172.21.15.27:0/380666507 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f22a4067130 msgr2=0x7f22a40695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:43.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.785+0000 7f22c81ff700 1 --2- 172.21.15.27:0/380666507 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f22a4067130 0x7f22a40695e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f22bc0030c0 tx=0x7f22bc016000 comp rx=0 tx=0).stop 2024-01-11T08:22:43.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.785+0000 7f22c81ff700 1 -- 172.21.15.27:0/380666507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22c013fbb0 msgr2=0x7f22c0155a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:43.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.785+0000 7f22c81ff700 1 --2- 172.21.15.27:0/380666507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22c013fbb0 0x7f22c0155a00 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f22b000e900 tx=0x7f22b00091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:43.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.785+0000 7f22c81ff700 1 -- 172.21.15.27:0/380666507 shutdown_connections 2024-01-11T08:22:43.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.785+0000 7f22c81ff700 1 --2- 172.21.15.27:0/380666507 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f22a4067130 0x7f22a40695e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:43.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.785+0000 7f22c81ff700 1 --2- 172.21.15.27:0/380666507 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22c013ff80 0x7f22c0155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:43.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.785+0000 7f22c81ff700 1 --2- 172.21.15.27:0/380666507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22c00cace0 0x7f22c0158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:43.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.785+0000 7f22c81ff700 1 --2- 172.21.15.27:0/380666507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22c013fbb0 0x7f22c0155a00 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:43.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.785+0000 7f22c81ff700 1 -- 172.21.15.27:0/380666507 >> 172.21.15.27:0/380666507 conn(0x7f22c00680a0 msgr2=0x7f22c00c7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:43.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.785+0000 7f22c81ff700 1 -- 172.21.15.27:0/380666507 shutdown_connections 2024-01-11T08:22:43.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:43.785+0000 7f22c81ff700 1 -- 172.21.15.27:0/380666507 wait complete. 2024-01-11T08:22:43.801 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2024-01-11T08:22:43.802 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:43.802 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2024-01-11T08:22:43.850 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:43.850 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:43.850 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:43.851 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:43.851 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:43.851 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:43.851 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:43.851 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:43.851 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:43.851 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:43.851 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:43.852 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_12_00_UTC 2024-01-11T08:22:43.852 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:43.852 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_12_00_UTC) 2024-01-11T08:22:43.917 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_13_00_UTC 2024-01-11T08:22:43.917 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:43.917 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_13_00_UTC) 2024-01-11T08:22:43.975 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_14_00_UTC 2024-01-11T08:22:43.976 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:43.976 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_14_00_UTC) 2024-01-11T08:22:44.031 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_15_00_UTC 2024-01-11T08:22:44.031 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:44.031 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_15_00_UTC) 2024-01-11T08:22:44.085 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_16_00_UTC 2024-01-11T08:22:44.085 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:44.085 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_16_00_UTC) 2024-01-11T08:22:44.140 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_17_00_UTC 2024-01-11T08:22:44.140 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:44.140 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_17_00_UTC) 2024-01-11T08:22:44.195 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_18_00_UTC 2024-01-11T08:22:44.195 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:44.195 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_18_00_UTC) 2024-01-11T08:22:44.250 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_19_00_UTC 2024-01-11T08:22:44.250 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:44.250 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_19_00_UTC) 2024-01-11T08:22:44.308 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_20_00_UTC 2024-01-11T08:22:44.308 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:44.308 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_20_00_UTC) 2024-01-11T08:22:44.364 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_21_00_UTC 2024-01-11T08:22:44.364 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:44.364 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_21_00_UTC) 2024-01-11T08:22:44.418 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_22_00_UTC 2024-01-11T08:22:44.418 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:44.418 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-01-11-08_22_00_UTC) 2024-01-11T08:22:44.473 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:44.474 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2024-01-11T08:22:44.527 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-11T08:22:44.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.763+0000 7efc5a5ff700 1 Processor -- start 2024-01-11T08:22:44.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.763+0000 7efc5a5ff700 1 -- start start 2024-01-11T08:22:44.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.764+0000 7efc5a5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc54143190 0x7efc540cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:44.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.764+0000 7efc5a5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efc540cb4b0 0x7efc540cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:44.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.764+0000 7efc5a5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc5415d320 con 0x7efc54142dc0 2024-01-11T08:22:44.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.764+0000 7efc5a5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc5415df30 con 0x7efc540cb4b0 2024-01-11T08:22:44.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.764+0000 7efc5a5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5415d590 con 0x7efc54143190 2024-01-11T08:22:44.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.764+0000 7efc53fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efc54142dc0 0x7efc540cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34250/0 (socket says 172.21.15.27:34250) 2024-01-11T08:22:44.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.764+0000 7efc53fff700 1 -- 172.21.15.27:0/1606010738 learned_addr learned my addr 172.21.15.27:0/1606010738 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:44.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.764+0000 7efc537fe700 1 --2- 172.21.15.27:0/1606010738 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc54143190 0x7efc540cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:44.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.765+0000 7efc58b9c700 1 --2- 172.21.15.27:0/1606010738 >> v2:172.21.15.77:3300/0 conn(0x7efc540cb4b0 0x7efc540cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:44.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.765+0000 7efc52ffd700 1 -- 172.21.15.27:0/1606010738 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815725473 0 0) 0x7efc5415d320 con 0x7efc54142dc0 2024-01-11T08:22:44.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.765+0000 7efc52ffd700 1 -- 172.21.15.27:0/1606010738 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc3c003570 con 0x7efc54142dc0 2024-01-11T08:22:44.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.765+0000 7efc537fe700 1 -- 172.21.15.27:0/1606010738 >> v2:172.21.15.77:3300/0 conn(0x7efc540cb4b0 msgr2=0x7efc540cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:44.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.765+0000 7efc537fe700 1 --2- 172.21.15.27:0/1606010738 >> v2:172.21.15.77:3300/0 conn(0x7efc540cb4b0 0x7efc540cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:44.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.765+0000 7efc537fe700 1 -- 172.21.15.27:0/1606010738 >> v1:172.21.15.27:6789/0 conn(0x7efc54142dc0 legacy=0x7efc540cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:44.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.765+0000 7efc537fe700 1 -- 172.21.15.27:0/1606010738 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5415d870 con 0x7efc54143190 2024-01-11T08:22:44.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.765+0000 7efc58b9c700 1 --2- 172.21.15.27:0/1606010738 >> v2:172.21.15.77:3300/0 conn(0x7efc540cb4b0 0x7efc540cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:22:44.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.765+0000 7efc537fe700 1 --2- 172.21.15.27:0/1606010738 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc54143190 0x7efc540cc020 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efc480098e0 tx=0x7efc480091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eab6227a0c4e2828 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:44.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.766+0000 7efc52ffd700 1 -- 172.21.15.27:0/1606010738 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc48011630 con 0x7efc54143190 2024-01-11T08:22:44.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.766+0000 7efc52ffd700 1 -- 172.21.15.27:0/1606010738 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc48011790 con 0x7efc54143190 2024-01-11T08:22:44.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.766+0000 7efc52ffd700 1 -- 172.21.15.27:0/1606010738 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc48011b30 con 0x7efc54143190 2024-01-11T08:22:44.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.766+0000 7efc5a5ff700 1 -- 172.21.15.27:0/1606010738 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc54143190 msgr2=0x7efc540cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:44.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.766+0000 7efc5a5ff700 1 --2- 172.21.15.27:0/1606010738 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc54143190 0x7efc540cc020 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efc480098e0 tx=0x7efc480091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:44.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.766+0000 7efc5a5ff700 1 -- 172.21.15.27:0/1606010738 shutdown_connections 2024-01-11T08:22:44.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.766+0000 7efc5a5ff700 1 --2- 172.21.15.27:0/1606010738 >> v2:172.21.15.77:3300/0 conn(0x7efc540cb4b0 0x7efc540cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:44.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.766+0000 7efc5a5ff700 1 --2- 172.21.15.27:0/1606010738 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc54143190 0x7efc540cc020 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:44.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.766+0000 7efc5a5ff700 1 -- 172.21.15.27:0/1606010738 >> 172.21.15.27:0/1606010738 conn(0x7efc54063ca0 msgr2=0x7efc5406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:44.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.766+0000 7efc5a5ff700 1 -- 172.21.15.27:0/1606010738 shutdown_connections 2024-01-11T08:22:44.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.766+0000 7efc5a5ff700 1 -- 172.21.15.27:0/1606010738 wait complete. 2024-01-11T08:22:44.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.768+0000 7efc5a5ff700 1 Processor -- start 2024-01-11T08:22:44.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.768+0000 7efc5a5ff700 1 -- start start 2024-01-11T08:22:44.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.769+0000 7efc5a5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc540cb4b0 0x7efc54140110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:44.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.769+0000 7efc5a5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc54142dc0 0x7efc54169880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:44.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.769+0000 7efc5a5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc54143190 0x7efc540c93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:44.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.769+0000 7efc5a5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc540c58a0 con 0x7efc540cb4b0 2024-01-11T08:22:44.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.769+0000 7efc5a5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc540c9fb0 con 0x7efc54143190 2024-01-11T08:22:44.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.769+0000 7efc5a5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc540ca290 con 0x7efc54142dc0 2024-01-11T08:22:44.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.769+0000 7efc53fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc540cb4b0 0x7efc54140110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:44.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.769+0000 7efc53fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc540cb4b0 0x7efc54140110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54258/0 (socket says 172.21.15.27:54258) 2024-01-11T08:22:44.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.769+0000 7efc53fff700 1 -- 172.21.15.27:0/1339949157 learned_addr learned my addr 172.21.15.27:0/1339949157 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:44.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.769+0000 7efc53fff700 1 -- 172.21.15.27:0/1339949157 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc54142dc0 msgr2=0x7efc54169880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:44.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.769+0000 7efc53fff700 1 --2- 172.21.15.27:0/1339949157 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc54142dc0 0x7efc54169880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:44.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.769+0000 7efc53fff700 1 -- 172.21.15.27:0/1339949157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc54143190 msgr2=0x7efc540c93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:44.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.769+0000 7efc53fff700 1 --2- 172.21.15.27:0/1339949157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc54143190 0x7efc540c93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:44.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.769+0000 7efc53fff700 1 -- 172.21.15.27:0/1339949157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4c009a20 con 0x7efc540cb4b0 2024-01-11T08:22:44.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.770+0000 7efc53fff700 1 --2- 172.21.15.27:0/1339949157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc540cb4b0 0x7efc54140110 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7efc4400c7c0 tx=0x7efc44007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:44.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.770+0000 7efc3bfff700 1 -- 172.21.15.27:0/1339949157 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc44005b30 con 0x7efc540cb4b0 2024-01-11T08:22:44.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.770+0000 7efc3bfff700 1 -- 172.21.15.27:0/1339949157 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc44005c90 con 0x7efc540cb4b0 2024-01-11T08:22:44.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.770+0000 7efc5a5ff700 1 -- 172.21.15.27:0/1339949157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc48009420 con 0x7efc540cb4b0 2024-01-11T08:22:44.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.770+0000 7efc3bfff700 1 -- 172.21.15.27:0/1339949157 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc44007340 con 0x7efc540cb4b0 2024-01-11T08:22:44.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.770+0000 7efc5a5ff700 1 -- 172.21.15.27:0/1339949157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc540c5530 con 0x7efc540cb4b0 2024-01-11T08:22:44.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.771+0000 7efc5a5ff700 1 -- 172.21.15.27:0/1339949157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5404c3c0 con 0x7efc540cb4b0 2024-01-11T08:22:44.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.772+0000 7efc3bfff700 1 -- 172.21.15.27:0/1339949157 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 84930+0+0 (secure 0 0 0) 0x7efc4401b400 con 0x7efc540cb4b0 2024-01-11T08:22:44.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.773+0000 7efc3bfff700 1 --2- 172.21.15.27:0/1339949157 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efc40067130 0x7efc400695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:44.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.773+0000 7efc3bfff700 1 -- 172.21.15.27:0/1339949157 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6815+0+0 (secure 0 0 0) 0x7efc44084820 con 0x7efc540cb4b0 2024-01-11T08:22:44.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.773+0000 7efc537fe700 1 --2- 172.21.15.27:0/1339949157 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efc40067130 0x7efc400695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:44.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.774+0000 7efc537fe700 1 --2- 172.21.15.27:0/1339949157 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efc40067130 0x7efc400695e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efc54141c70 tx=0x7efc48017000 comp rx=0 tx=0).ready entity=mgr.4793 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:44.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:44.777+0000 7efc3bfff700 1 -- 172.21.15.27:0/1339949157 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efc44051440 con 0x7efc540cb4b0 2024-01-11T08:22:45.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:45.055+0000 7efc5a5ff700 1 -- 172.21.15.27:0/1339949157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7efc5415db50 con 0x7efc540cb4b0 2024-01-11T08:22:45.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:45.733+0000 7efc3bfff700 1 -- 172.21.15.27:0/1339949157 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7efc44051810 con 0x7efc540cb4b0 2024-01-11T08:22:45.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:45.734+0000 7efc3bfff700 1 -- 172.21.15.27:0/1339949157 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v14) v1 ==== 93+0+0 (secure 0 0 0) 0x7efc44054ed0 con 0x7efc540cb4b0 2024-01-11T08:22:45.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:45.737+0000 7efc5a5ff700 1 -- 172.21.15.27:0/1339949157 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efc40067130 msgr2=0x7efc400695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:45.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:45.737+0000 7efc5a5ff700 1 --2- 172.21.15.27:0/1339949157 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efc40067130 0x7efc400695e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efc54141c70 tx=0x7efc48017000 comp rx=0 tx=0).stop 2024-01-11T08:22:45.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:45.737+0000 7efc5a5ff700 1 -- 172.21.15.27:0/1339949157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc540cb4b0 msgr2=0x7efc54140110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:45.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:45.737+0000 7efc5a5ff700 1 --2- 172.21.15.27:0/1339949157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc540cb4b0 0x7efc54140110 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7efc4400c7c0 tx=0x7efc44007e00 comp rx=0 tx=0).stop 2024-01-11T08:22:45.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:45.737+0000 7efc5a5ff700 1 -- 172.21.15.27:0/1339949157 shutdown_connections 2024-01-11T08:22:45.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:45.737+0000 7efc5a5ff700 1 --2- 172.21.15.27:0/1339949157 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efc40067130 0x7efc400695e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:45.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:45.737+0000 7efc5a5ff700 1 --2- 172.21.15.27:0/1339949157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc54143190 0x7efc540c93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:45.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:45.737+0000 7efc5a5ff700 1 --2- 172.21.15.27:0/1339949157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc540cb4b0 0x7efc54140110 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:45.737+0000 7efc5a5ff700 1 --2- 172.21.15.27:0/1339949157 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efc54142dc0 0x7efc54169880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:45.737+0000 7efc5a5ff700 1 -- 172.21.15.27:0/1339949157 >> 172.21.15.27:0/1339949157 conn(0x7efc54063ca0 msgr2=0x7efc540c3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:45.737+0000 7efc5a5ff700 1 -- 172.21.15.27:0/1339949157 shutdown_connections 2024-01-11T08:22:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:45.737+0000 7efc5a5ff700 1 -- 172.21.15.27:0/1339949157 wait complete. 2024-01-11T08:22:45.755 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T08:22:45.755 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:22:45.849 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:22:45.955 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T08:22:45.956 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-11T08:22:45.956 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:22:45.956 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-01-11T08:22:45.960 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:45.961+0000 7f909e5aa1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:22:45.965 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:45.966+0000 7f4a678c91c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:22:46.052 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T08:22:46.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:22:46.053 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-11T08:22:46.151 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:46.152+0000 7f909e5aa1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:22:46.191 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:22:46.204 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:46.204+0000 7f4a678c91c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:22:46.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.400+0000 7f5c2e1ff700 1 Processor -- start 2024-01-11T08:22:46.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.400+0000 7f5c2e1ff700 1 -- start start 2024-01-11T08:22:46.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.401+0000 7f5c2e1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c280b03e0 0x7f5c280b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:46.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.401+0000 7f5c2e1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c280b2e00 con 0x7f5c280b07b0 2024-01-11T08:22:46.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.401+0000 7f5c2e1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c280cd730 con 0x7f5c280b29c0 2024-01-11T08:22:46.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.401+0000 7f5c2e1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c280ccb20 con 0x7f5c280b03e0 2024-01-11T08:22:46.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.402+0000 7f5c277fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c280b03e0 0x7f5c280b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:46.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.402+0000 7f5c26ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5c280b07b0 0x7f5c28159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34266/0 (socket says 172.21.15.27:34266) 2024-01-11T08:22:46.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.402+0000 7f5c26ffd700 1 -- 172.21.15.27:0/4254034930 learned_addr learned my addr 172.21.15.27:0/4254034930 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:46.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.402+0000 7f5c27fff700 1 --1- 172.21.15.27:0/4254034930 >> v1:172.21.15.77:6789/0 conn(0x7f5c280b29c0 0x7f5c280aefc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:58656/0 (socket says 172.21.15.27:58656) 2024-01-11T08:22:46.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.402+0000 7f5c267fc700 1 -- 172.21.15.27:0/4254034930 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795318648 0 0) 0x7f5c280cd730 con 0x7f5c280b29c0 2024-01-11T08:22:46.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.402+0000 7f5c277fe700 1 -- 172.21.15.27:0/4254034930 >> v1:172.21.15.77:6789/0 conn(0x7f5c280b29c0 legacy=0x7f5c280aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:46.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.402+0000 7f5c277fe700 1 -- 172.21.15.27:0/4254034930 >> v1:172.21.15.27:6789/0 conn(0x7f5c280b07b0 legacy=0x7f5c28159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:46.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.402+0000 7f5c277fe700 1 -- 172.21.15.27:0/4254034930 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c280cce00 con 0x7f5c280b03e0 2024-01-11T08:22:46.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.402+0000 7f5c277fe700 1 --2- 172.21.15.27:0/4254034930 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c280b03e0 0x7f5c280b2480 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5c18009910 tx=0x7f5c180091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74608f260d2b929a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:46.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.403+0000 7f5c267fc700 1 -- 172.21.15.27:0/4254034930 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1800b760 con 0x7f5c280b03e0 2024-01-11T08:22:46.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.403+0000 7f5c267fc700 1 -- 172.21.15.27:0/4254034930 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c1800b8c0 con 0x7f5c280b03e0 2024-01-11T08:22:46.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.403+0000 7f5c267fc700 1 -- 172.21.15.27:0/4254034930 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1800bc60 con 0x7f5c280b03e0 2024-01-11T08:22:46.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.403+0000 7f5c2e1ff700 1 -- 172.21.15.27:0/4254034930 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c280b03e0 msgr2=0x7f5c280b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:46.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.403+0000 7f5c2e1ff700 1 --2- 172.21.15.27:0/4254034930 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c280b03e0 0x7f5c280b2480 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5c18009910 tx=0x7f5c180091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:46.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.403+0000 7f5c2e1ff700 1 -- 172.21.15.27:0/4254034930 shutdown_connections 2024-01-11T08:22:46.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.403+0000 7f5c2e1ff700 1 --2- 172.21.15.27:0/4254034930 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c280b03e0 0x7f5c280b2480 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:46.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.403+0000 7f5c2e1ff700 1 -- 172.21.15.27:0/4254034930 >> 172.21.15.27:0/4254034930 conn(0x7f5c2806ed40 msgr2=0x7f5c2806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:46.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.403+0000 7f5c2e1ff700 1 -- 172.21.15.27:0/4254034930 shutdown_connections 2024-01-11T08:22:46.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.403+0000 7f5c2e1ff700 1 -- 172.21.15.27:0/4254034930 wait complete. 2024-01-11T08:22:46.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.405+0000 7f5c2e1ff700 1 Processor -- start 2024-01-11T08:22:46.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.405+0000 7f5c2e1ff700 1 -- start start 2024-01-11T08:22:46.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.406+0000 7f5c2e1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c280b03e0 0x7f5c280c6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:46.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.406+0000 7f5c2e1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c280b07b0 0x7f5c280ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.406+0000 7f5c2e1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c280b29c0 0x7f5c280ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.406+0000 7f5c2e1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c280aee20 con 0x7f5c280b29c0 2024-01-11T08:22:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.406+0000 7f5c2e1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c280c8270 con 0x7f5c280b03e0 2024-01-11T08:22:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.406+0000 7f5c2e1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c280c7660 con 0x7f5c280b07b0 2024-01-11T08:22:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.406+0000 7f5c277fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c280b03e0 0x7f5c280c6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.406+0000 7f5c27fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c280b29c0 0x7f5c280ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.406+0000 7f5c277fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c280b03e0 0x7f5c280c6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:57754/0 (socket says 172.21.15.27:57754) 2024-01-11T08:22:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.406+0000 7f5c277fe700 1 -- 172.21.15.27:0/3435795769 learned_addr learned my addr 172.21.15.27:0/3435795769 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.407+0000 7f5c26ffd700 1 --2- 172.21.15.27:0/3435795769 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c280b07b0 0x7f5c280ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.407+0000 7f5c277fe700 1 -- 172.21.15.27:0/3435795769 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c280b07b0 msgr2=0x7f5c280ae3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.407+0000 7f5c277fe700 1 --2- 172.21.15.27:0/3435795769 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c280b07b0 0x7f5c280ae3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.407+0000 7f5c277fe700 1 -- 172.21.15.27:0/3435795769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c280b29c0 msgr2=0x7f5c280ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.407+0000 7f5c277fe700 1 --2- 172.21.15.27:0/3435795769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c280b29c0 0x7f5c280ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:46.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.407+0000 7f5c277fe700 1 -- 172.21.15.27:0/3435795769 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c18009420 con 0x7f5c280b03e0 2024-01-11T08:22:46.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.407+0000 7f5c277fe700 1 --2- 172.21.15.27:0/3435795769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c280b03e0 0x7f5c280c6250 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5c180093f0 tx=0x7f5c18016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:46.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.408+0000 7f5c0f7fe700 1 -- 172.21.15.27:0/3435795769 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c18011da0 con 0x7f5c280b03e0 2024-01-11T08:22:46.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.408+0000 7f5c0f7fe700 1 -- 172.21.15.27:0/3435795769 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c1801c460 con 0x7f5c280b03e0 2024-01-11T08:22:46.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.408+0000 7f5c0f7fe700 1 -- 172.21.15.27:0/3435795769 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1801c820 con 0x7f5c280b03e0 2024-01-11T08:22:46.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.408+0000 7f5c2e1ff700 1 -- 172.21.15.27:0/3435795769 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c280c7940 con 0x7f5c280b03e0 2024-01-11T08:22:46.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.408+0000 7f5c2e1ff700 1 -- 172.21.15.27:0/3435795769 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c280c7f00 con 0x7f5c280b03e0 2024-01-11T08:22:46.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.409+0000 7f5c2e1ff700 1 -- 172.21.15.27:0/3435795769 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c280718d0 con 0x7f5c280b03e0 2024-01-11T08:22:46.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.410+0000 7f5c0f7fe700 1 -- 172.21.15.27:0/3435795769 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7f5c1801cc50 con 0x7f5c280b03e0 2024-01-11T08:22:46.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.411+0000 7f5c0f7fe700 1 --2- 172.21.15.27:0/3435795769 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5c140670e0 0x7f5c14069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:46.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.411+0000 7f5c0f7fe700 1 -- 172.21.15.27:0/3435795769 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6499+0+0 (secure 0 0 0) 0x7f5c180870c0 con 0x7f5c280b03e0 2024-01-11T08:22:46.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.411+0000 7f5c26ffd700 1 -- 172.21.15.27:0/3435795769 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5c140670e0 msgr2=0x7f5c14069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:46.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.411+0000 7f5c26ffd700 1 --2- 172.21.15.27:0/3435795769 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5c140670e0 0x7f5c14069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:22:46.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.416+0000 7f5c0f7fe700 1 -- 172.21.15.27:0/3435795769 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5c18053cf0 con 0x7f5c280b03e0 2024-01-11T08:22:46.490 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:46.491+0000 7f909e5aa1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:22:46.551 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:46.552+0000 7f4a678c91c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:22:46.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.612+0000 7f5c26ffd700 1 -- 172.21.15.27:0/3435795769 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5c140670e0 msgr2=0x7f5c14069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:46.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.612+0000 7f5c26ffd700 1 --2- 172.21.15.27:0/3435795769 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5c140670e0 0x7f5c14069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:22:46.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.675+0000 7f5c2e1ff700 1 -- 172.21.15.27:0/3435795769 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c280c83b0 con 0x7f5c280b03e0 2024-01-11T08:22:46.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.676+0000 7f5c0f7fe700 1 -- 172.21.15.27:0/3435795769 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4591 (secure 0 0 0) 0x7f5c18057310 con 0x7f5c280b03e0 2024-01-11T08:22:46.675 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:22:46.675 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/2930156910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2930156910},{"type":"v1","addr":"172.21.15.77:6836","nonce":2930156910}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/685019444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":685019444},{"type":"v1","addr":"172.21.15.27:6834","nonce":685019444}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4861,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/144853545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":144853545},{"type":"v1","addr":"172.21.15.77:6837","nonce":144853545}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:11:24.435181+0000","modified":"2024-01-11T08:11:27.940837+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/4056690675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4056690675},{"type":"v1","addr":"172.21.15.27:6835","nonce":4056690675}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2024-01-11T08:22:46.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.678+0000 7f5c2e1ff700 1 -- 172.21.15.27:0/3435795769 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5c140670e0 msgr2=0x7f5c14069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:22:46.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.678+0000 7f5c2e1ff700 1 --2- 172.21.15.27:0/3435795769 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5c140670e0 0x7f5c14069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:46.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.678+0000 7f5c2e1ff700 1 -- 172.21.15.27:0/3435795769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c280b03e0 msgr2=0x7f5c280c6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:46.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.678+0000 7f5c2e1ff700 1 --2- 172.21.15.27:0/3435795769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c280b03e0 0x7f5c280c6250 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5c180093f0 tx=0x7f5c18016000 comp rx=0 tx=0).stop 2024-01-11T08:22:46.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.679+0000 7f5c2e1ff700 1 -- 172.21.15.27:0/3435795769 shutdown_connections 2024-01-11T08:22:46.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.679+0000 7f5c2e1ff700 1 --2- 172.21.15.27:0/3435795769 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5c140670e0 0x7f5c14069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:46.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.679+0000 7f5c2e1ff700 1 --2- 172.21.15.27:0/3435795769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c280b29c0 0x7f5c280ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:46.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.679+0000 7f5c2e1ff700 1 --2- 172.21.15.27:0/3435795769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c280b03e0 0x7f5c280c6250 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:46.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.679+0000 7f5c2e1ff700 1 --2- 172.21.15.27:0/3435795769 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c280b07b0 0x7f5c280ae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:46.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.679+0000 7f5c2e1ff700 1 -- 172.21.15.27:0/3435795769 >> 172.21.15.27:0/3435795769 conn(0x7f5c2806ed40 msgr2=0x7f5c280cb400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:46.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.679+0000 7f5c2e1ff700 1 -- 172.21.15.27:0/3435795769 shutdown_connections 2024-01-11T08:22:46.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.679+0000 7f5c2e1ff700 1 -- 172.21.15.27:0/3435795769 wait complete. 2024-01-11T08:22:46.679 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-01-11T08:22:46.697 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:22:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.950+0000 7f6fb21ff700 1 Processor -- start 2024-01-11T08:22:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.951+0000 7f6fb21ff700 1 -- start start 2024-01-11T08:22:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.952+0000 7f6fb21ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6fac1575e0 0x7f6fac158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.952+0000 7f6fb21ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fac158b60 con 0x7f6fac0b19d0 2024-01-11T08:22:46.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.952+0000 7f6fb21ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fac1599b0 con 0x7f6fac06b610 2024-01-11T08:22:46.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.952+0000 7f6fb21ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fac0cb5e0 con 0x7f6fac1575e0 2024-01-11T08:22:46.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.952+0000 7f6fabfff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6fac0b19d0 0x7f6fac159e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34280/0 (socket says 172.21.15.27:34280) 2024-01-11T08:22:46.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.952+0000 7f6fabfff700 1 -- 172.21.15.27:0/1159845743 learned_addr learned my addr 172.21.15.27:0/1159845743 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:46.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.952+0000 7f6fab7fe700 1 --2- 172.21.15.27:0/1159845743 >> v2:172.21.15.77:3300/0 conn(0x7f6fac1575e0 0x7f6fac158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:46.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.953+0000 7f6faa7fc700 1 -- 172.21.15.27:0/1159845743 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216600637 0 0) 0x7f6fa4001490 con 0x7f6fac0b19d0 2024-01-11T08:22:46.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.953+0000 7f6faa7fc700 1 -- 172.21.15.27:0/1159845743 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f94003570 con 0x7f6fac0b19d0 2024-01-11T08:22:46.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.953+0000 7f6faa7fc700 1 -- 172.21.15.27:0/1159845743 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333297333 0 0) 0x7f6fac1599b0 con 0x7f6fac06b610 2024-01-11T08:22:46.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.953+0000 7f6faa7fc700 1 -- 172.21.15.27:0/1159845743 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fa4001490 con 0x7f6fac06b610 2024-01-11T08:22:46.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.953+0000 7f6fab7fe700 1 -- 172.21.15.27:0/1159845743 >> v1:172.21.15.77:6789/0 conn(0x7f6fac06b610 legacy=0x7f6fac0b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:46.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.953+0000 7f6fab7fe700 1 -- 172.21.15.27:0/1159845743 >> v1:172.21.15.27:6789/0 conn(0x7f6fac0b19d0 legacy=0x7f6fac159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:46.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.953+0000 7f6fab7fe700 1 -- 172.21.15.27:0/1159845743 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac0c0f90 con 0x7f6fac1575e0 2024-01-11T08:22:46.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.953+0000 7f6fab7fe700 1 --2- 172.21.15.27:0/1159845743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fac1575e0 0x7f6fac158620 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0098e0 tx=0x7f6f9c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55e27fa1baca2986 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:46.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.954+0000 7f6faa7fc700 1 -- 172.21.15.27:0/1159845743 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c011630 con 0x7f6fac1575e0 2024-01-11T08:22:46.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.954+0000 7f6faa7fc700 1 -- 172.21.15.27:0/1159845743 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9c011790 con 0x7f6fac1575e0 2024-01-11T08:22:46.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.954+0000 7f6faa7fc700 1 -- 172.21.15.27:0/1159845743 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c011b30 con 0x7f6fac1575e0 2024-01-11T08:22:46.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.954+0000 7f6fb21ff700 1 -- 172.21.15.27:0/1159845743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fac1575e0 msgr2=0x7f6fac158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:46.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.954+0000 7f6fb21ff700 1 --2- 172.21.15.27:0/1159845743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fac1575e0 0x7f6fac158620 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0098e0 tx=0x7f6f9c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:46.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.954+0000 7f6fb21ff700 1 -- 172.21.15.27:0/1159845743 shutdown_connections 2024-01-11T08:22:46.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.954+0000 7f6fb21ff700 1 --2- 172.21.15.27:0/1159845743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fac1575e0 0x7f6fac158620 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:46.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.954+0000 7f6fb21ff700 1 -- 172.21.15.27:0/1159845743 >> 172.21.15.27:0/1159845743 conn(0x7f6fac0680e0 msgr2=0x7f6fac069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:46.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.954+0000 7f6fb21ff700 1 -- 172.21.15.27:0/1159845743 shutdown_connections 2024-01-11T08:22:46.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.954+0000 7f6fb21ff700 1 -- 172.21.15.27:0/1159845743 wait complete. 2024-01-11T08:22:46.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.956+0000 7f6fb21ff700 1 Processor -- start 2024-01-11T08:22:46.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.956+0000 7f6fb21ff700 1 -- start start 2024-01-11T08:22:46.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.957+0000 7f6fb21ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fac06b610 0x7f6fac1521d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.957+0000 7f6fb21ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fac0b19d0 0x7f6fac0c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.957+0000 7f6fb21ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fac1575e0 0x7f6fac0c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.957+0000 7f6fb21ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac0c6160 con 0x7f6fac0b19d0 2024-01-11T08:22:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.957+0000 7f6fb21ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac152710 con 0x7f6fac1575e0 2024-01-11T08:22:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.957+0000 7f6fb21ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac0affc0 con 0x7f6fac06b610 2024-01-11T08:22:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.957+0000 7f6faaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fac0b19d0 0x7f6fac0c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.957+0000 7f6faaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fac0b19d0 0x7f6fac0c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54300/0 (socket says 172.21.15.27:54300) 2024-01-11T08:22:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.957+0000 7f6fab7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fac06b610 0x7f6fac1521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.957+0000 7f6faaffd700 1 -- 172.21.15.27:0/270091992 learned_addr learned my addr 172.21.15.27:0/270091992 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.957+0000 7f6fabfff700 1 --2- 172.21.15.27:0/270091992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fac1575e0 0x7f6fac0c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.958+0000 7f6faaffd700 1 -- 172.21.15.27:0/270091992 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fac06b610 msgr2=0x7f6fac1521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.958+0000 7f6faaffd700 1 --2- 172.21.15.27:0/270091992 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fac06b610 0x7f6fac1521d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.958+0000 7f6faaffd700 1 -- 172.21.15.27:0/270091992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fac1575e0 msgr2=0x7f6fac0c7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.958+0000 7f6faaffd700 1 --2- 172.21.15.27:0/270091992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fac1575e0 0x7f6fac0c7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.958+0000 7f6faaffd700 1 -- 172.21.15.27:0/270091992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9c009420 con 0x7f6fac0b19d0 2024-01-11T08:22:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.958+0000 7f6fab7fe700 1 --2- 172.21.15.27:0/270091992 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fac06b610 0x7f6fac1521d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:22:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.958+0000 7f6faaffd700 1 --2- 172.21.15.27:0/270091992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fac0b19d0 0x7f6fac0c57d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000c790 tx=0x7f6fa0007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.958+0000 7f6f937fe700 1 -- 172.21.15.27:0/270091992 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0010510 con 0x7f6fac0b19d0 2024-01-11T08:22:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.958+0000 7f6f937fe700 1 -- 172.21.15.27:0/270091992 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa0010670 con 0x7f6fac0b19d0 2024-01-11T08:22:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.958+0000 7f6f937fe700 1 -- 172.21.15.27:0/270091992 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0010a10 con 0x7f6fac0b19d0 2024-01-11T08:22:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.958+0000 7f6fb21ff700 1 -- 172.21.15.27:0/270091992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fac0b02a0 con 0x7f6fac0b19d0 2024-01-11T08:22:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.958+0000 7f6fb21ff700 1 -- 172.21.15.27:0/270091992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fac0b0860 con 0x7f6fac0b19d0 2024-01-11T08:22:46.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.960+0000 7f6fb21ff700 1 -- 172.21.15.27:0/270091992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fac0718d0 con 0x7f6fac0b19d0 2024-01-11T08:22:46.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.960+0000 7f6f937fe700 1 -- 172.21.15.27:0/270091992 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7f6fa001b3b0 con 0x7f6fac0b19d0 2024-01-11T08:22:46.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.961+0000 7f6f937fe700 1 --2- 172.21.15.27:0/270091992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6f980670b0 0x7f6f98069560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:46.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.961+0000 7f6f937fe700 1 -- 172.21.15.27:0/270091992 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f6fa00843f0 con 0x7f6fac0b19d0 2024-01-11T08:22:46.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.962+0000 7f6fab7fe700 1 -- 172.21.15.27:0/270091992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6f980670b0 msgr2=0x7f6f98069560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:46.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.962+0000 7f6fab7fe700 1 --2- 172.21.15.27:0/270091992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6f980670b0 0x7f6f98069560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:22:46.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:46.966+0000 7f6f937fe700 1 -- 172.21.15.27:0/270091992 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6fa00510b0 con 0x7f6fac0b19d0 2024-01-11T08:22:47.026 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:47.027+0000 7f909e5aa1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:22:47.126 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:47.127+0000 7f4a678c91c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:22:47.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.163+0000 7f6fab7fe700 1 -- 172.21.15.27:0/270091992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6f980670b0 msgr2=0x7f6f98069560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:47.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.163+0000 7f6fab7fe700 1 --2- 172.21.15.27:0/270091992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6f980670b0 0x7f6f98069560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:22:47.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.171+0000 7f6fb21ff700 1 -- 172.21.15.27:0/270091992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fac0c62a0 con 0x7f6fac0b19d0 2024-01-11T08:22:47.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.172+0000 7f6f937fe700 1 -- 172.21.15.27:0/270091992 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16289 (secure 0 0 0) 0x7f6fa00546d0 con 0x7f6fac0b19d0 2024-01-11T08:22:47.172 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:22:47.172 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":46,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:22:46.732163+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:11:22.376067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-11T08:11:23.419993+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":46,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6833/41381":"2024-01-12T08:11:40.675932+0000","172.21.15.77:6832/41381":"2024-01-12T08:11:40.675932+0000","172.21.15.77:0/969431132":"2024-01-12T08:11:40.675932+0000","172.21.15.77:0/4222476855":"2024-01-12T08:11:40.675932+0000","172.21.15.77:0/3211563793":"2024-01-12T08:11:40.675932+0000","172.21.15.77:0/2257956400":"2024-01-12T08:11:40.675932+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":10,"length":2},{"begin":15,"length":2},{"begin":20,"length":3},{"begin":26,"length":2},{"begin":31,"length":3},{"begin":37,"length":2},{"begin":43,"length":2},{"begin":48,"length":2},{"begin":54,"length":2},{"begin":59,"length":2},{"begin":65,"length":2},{"begin":70,"length":2},{"begin":75,"length":2},{"begin":80,"length":3},{"begin":86,"length":2},{"begin":91,"length":3},{"begin":97,"length":2},{"begin":102,"length":3},{"begin":108,"length":2},{"begin":177,"length":45}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":104,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:22:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.175+0000 7f6fb21ff700 1 -- 172.21.15.27:0/270091992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6f980670b0 msgr2=0x7f6f98069560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:22:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.175+0000 7f6fb21ff700 1 --2- 172.21.15.27:0/270091992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6f980670b0 0x7f6f98069560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.175+0000 7f6fb21ff700 1 -- 172.21.15.27:0/270091992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fac0b19d0 msgr2=0x7f6fac0c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.175+0000 7f6fb21ff700 1 --2- 172.21.15.27:0/270091992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fac0b19d0 0x7f6fac0c57d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000c790 tx=0x7f6fa0007e10 comp rx=0 tx=0).stop 2024-01-11T08:22:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.175+0000 7f6fb21ff700 1 -- 172.21.15.27:0/270091992 shutdown_connections 2024-01-11T08:22:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.175+0000 7f6fb21ff700 1 --2- 172.21.15.27:0/270091992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6f980670b0 0x7f6f98069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.175+0000 7f6fb21ff700 1 --2- 172.21.15.27:0/270091992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fac1575e0 0x7f6fac0c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.175+0000 7f6fb21ff700 1 --2- 172.21.15.27:0/270091992 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fac06b610 0x7f6fac1521d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.175+0000 7f6fb21ff700 1 --2- 172.21.15.27:0/270091992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fac0b19d0 0x7f6fac0c57d0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.175+0000 7f6fb21ff700 1 -- 172.21.15.27:0/270091992 >> 172.21.15.27:0/270091992 conn(0x7f6fac0680e0 msgr2=0x7f6fac152b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.175+0000 7f6fb21ff700 1 -- 172.21.15.27:0/270091992 shutdown_connections 2024-01-11T08:22:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.175+0000 7f6fb21ff700 1 -- 172.21.15.27:0/270091992 wait complete. 2024-01-11T08:22:47.193 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T08:11:22.376067+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T08:11:23.419993+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 221, 'snap_epoch': 46, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T08:22:47.193 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:22:47.199 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:47.200+0000 7f909e5aa1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:22:47.316 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:47.317+0000 7f4a678c91c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:22:47.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.409+0000 7f23fddff700 1 Processor -- start 2024-01-11T08:22:47.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.409+0000 7f23fddff700 1 -- start start 2024-01-11T08:22:47.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.410+0000 7f23fddff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f23f8142dc0 0x7f23f80cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:47.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.411+0000 7f23fddff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f23f80cb2e0 0x7f23f80cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:47.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.411+0000 7f23fddff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23f80ccc70 con 0x7f23f8143190 2024-01-11T08:22:47.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.411+0000 7f23fddff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23f815df30 con 0x7f23f8142dc0 2024-01-11T08:22:47.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.411+0000 7f23fddff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23f815d320 con 0x7f23f80cb2e0 2024-01-11T08:22:47.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.411+0000 7f23f77fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f23f8142dc0 0x7f23f80cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:47.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.411+0000 7f23f77fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f23f8142dc0 0x7f23f80cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54310/0 (socket says 172.21.15.27:54310) 2024-01-11T08:22:47.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.411+0000 7f23f77fe700 1 -- 172.21.15.27:0/4278735142 learned_addr learned my addr 172.21.15.27:0/4278735142 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:47.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.411+0000 7f23f7fff700 1 --2- 172.21.15.27:0/4278735142 >> v2:172.21.15.77:3300/0 conn(0x7f23f80cb2e0 0x7f23f80cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:47.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.411+0000 7f23f77fe700 1 -- 172.21.15.27:0/4278735142 >> v2:172.21.15.77:3300/0 conn(0x7f23f80cb2e0 msgr2=0x7f23f80cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:47.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.411+0000 7f23f77fe700 1 --2- 172.21.15.27:0/4278735142 >> v2:172.21.15.77:3300/0 conn(0x7f23f80cb2e0 0x7f23f80cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:47.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.411+0000 7f23f77fe700 1 -- 172.21.15.27:0/4278735142 >> v1:172.21.15.77:6789/0 conn(0x7f23f8143190 legacy=0x7f23f80cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:47.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.411+0000 7f23f77fe700 1 -- 172.21.15.27:0/4278735142 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f80cb700 con 0x7f23f8142dc0 2024-01-11T08:22:47.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.411+0000 7f23f67fc700 1 -- 172.21.15.27:0/4278735142 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060907081 0 0) 0x7f23f80ccc70 con 0x7f23f8143190 2024-01-11T08:22:47.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.412+0000 7f23f77fe700 1 --2- 172.21.15.27:0/4278735142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f8142dc0 0x7f23f80cada0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f23f81577b0 tx=0x7f23e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4bc6d058a440a42 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:47.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.412+0000 7f23f67fc700 1 -- 172.21.15.27:0/4278735142 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e8009e70 con 0x7f23f8142dc0 2024-01-11T08:22:47.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.412+0000 7f23f67fc700 1 -- 172.21.15.27:0/4278735142 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e800b860 con 0x7f23f8142dc0 2024-01-11T08:22:47.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.412+0000 7f23f67fc700 1 -- 172.21.15.27:0/4278735142 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e800bc00 con 0x7f23f8142dc0 2024-01-11T08:22:47.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.413+0000 7f23fddff700 1 -- 172.21.15.27:0/4278735142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f8142dc0 msgr2=0x7f23f80cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:47.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.413+0000 7f23fddff700 1 --2- 172.21.15.27:0/4278735142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f8142dc0 0x7f23f80cada0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f23f81577b0 tx=0x7f23e8009250 comp rx=0 tx=0).stop 2024-01-11T08:22:47.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.413+0000 7f23fddff700 1 -- 172.21.15.27:0/4278735142 shutdown_connections 2024-01-11T08:22:47.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.413+0000 7f23fddff700 1 --2- 172.21.15.27:0/4278735142 >> v2:172.21.15.77:3300/0 conn(0x7f23f80cb2e0 0x7f23f80cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:47.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.413+0000 7f23fddff700 1 --2- 172.21.15.27:0/4278735142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f8142dc0 0x7f23f80cada0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:47.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.413+0000 7f23fddff700 1 -- 172.21.15.27:0/4278735142 >> 172.21.15.27:0/4278735142 conn(0x7f23f8063ca0 msgr2=0x7f23f806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:47.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.413+0000 7f23fddff700 1 -- 172.21.15.27:0/4278735142 shutdown_connections 2024-01-11T08:22:47.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.413+0000 7f23fddff700 1 -- 172.21.15.27:0/4278735142 wait complete. 2024-01-11T08:22:47.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.414+0000 7f23fddff700 1 Processor -- start 2024-01-11T08:22:47.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.414+0000 7f23fddff700 1 -- start start 2024-01-11T08:22:47.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.415+0000 7f23fddff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23f80cb2e0 0x7f23f81556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:47.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.415+0000 7f23fddff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f8142dc0 0x7f23f81410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:47.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.415+0000 7f23fddff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f8143190 0x7f23f80b2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:47.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.415+0000 7f23fddff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f8155240 con 0x7f23f8143190 2024-01-11T08:22:47.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.415+0000 7f23fddff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f8155c10 con 0x7f23f8142dc0 2024-01-11T08:22:47.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.415+0000 7f23fddff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f8154630 con 0x7f23f80cb2e0 2024-01-11T08:22:47.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.416+0000 7f23f77fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23f80cb2e0 0x7f23f81556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:47.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.416+0000 7f23f7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f8143190 0x7f23f80b2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:47.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.416+0000 7f23f6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f8142dc0 0x7f23f81410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:47.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.416+0000 7f23f77fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23f80cb2e0 0x7f23f81556d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:38932/0 (socket says 172.21.15.27:38932) 2024-01-11T08:22:47.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.416+0000 7f23f77fe700 1 -- 172.21.15.27:0/625573340 learned_addr learned my addr 172.21.15.27:0/625573340 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:47.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.416+0000 7f23f77fe700 1 -- 172.21.15.27:0/625573340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f8142dc0 msgr2=0x7f23f81410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:47.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.416+0000 7f23f77fe700 1 --2- 172.21.15.27:0/625573340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f8142dc0 0x7f23f81410f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:47.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.416+0000 7f23f77fe700 1 -- 172.21.15.27:0/625573340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f8143190 msgr2=0x7f23f80b2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:47.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.416+0000 7f23f77fe700 1 --2- 172.21.15.27:0/625573340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f8143190 0x7f23f80b2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:47.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.416+0000 7f23f77fe700 1 -- 172.21.15.27:0/625573340 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e80094f0 con 0x7f23f80cb2e0 2024-01-11T08:22:47.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.416+0000 7f23f7fff700 1 --2- 172.21.15.27:0/625573340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f8143190 0x7f23f80b2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:22:47.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.416+0000 7f23f77fe700 1 --2- 172.21.15.27:0/625573340 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23f80cb2e0 0x7f23f81556d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f23f8064a70 tx=0x7f23e8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:47.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.417+0000 7f23df7fe700 1 -- 172.21.15.27:0/625573340 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e800bbe0 con 0x7f23f80cb2e0 2024-01-11T08:22:47.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.417+0000 7f23df7fe700 1 -- 172.21.15.27:0/625573340 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e8000c90 con 0x7f23f80cb2e0 2024-01-11T08:22:47.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.417+0000 7f23df7fe700 1 -- 172.21.15.27:0/625573340 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e8011650 con 0x7f23f80cb2e0 2024-01-11T08:22:47.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.417+0000 7f23fddff700 1 -- 172.21.15.27:0/625573340 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f8154910 con 0x7f23f80cb2e0 2024-01-11T08:22:47.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.417+0000 7f23fddff700 1 -- 172.21.15.27:0/625573340 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f8154ed0 con 0x7f23f80cb2e0 2024-01-11T08:22:47.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.418+0000 7f23fddff700 1 -- 172.21.15.27:0/625573340 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f8155520 con 0x7f23f80cb2e0 2024-01-11T08:22:47.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.419+0000 7f23df7fe700 1 -- 172.21.15.27:0/625573340 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7f23e8011a40 con 0x7f23f80cb2e0 2024-01-11T08:22:47.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.420+0000 7f23df7fe700 1 --2- 172.21.15.27:0/625573340 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f23e40670e0 0x7f23e4069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:47.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.420+0000 7f23df7fe700 1 -- 172.21.15.27:0/625573340 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f23e8085fe0 con 0x7f23f80cb2e0 2024-01-11T08:22:47.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.420+0000 7f23f6ffd700 1 -- 172.21.15.27:0/625573340 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f23e40670e0 msgr2=0x7f23e4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:47.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.420+0000 7f23f6ffd700 1 --2- 172.21.15.27:0/625573340 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f23e40670e0 0x7f23e4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:22:47.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.425+0000 7f23df7fe700 1 -- 172.21.15.27:0/625573340 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f23e8052d00 con 0x7f23f80cb2e0 2024-01-11T08:22:47.614 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:47.615+0000 7f909e5aa1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:22:47.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.621+0000 7f23f6ffd700 1 -- 172.21.15.27:0/625573340 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f23e40670e0 msgr2=0x7f23e4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:47.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.621+0000 7f23f6ffd700 1 --2- 172.21.15.27:0/625573340 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f23e40670e0 0x7f23e4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:22:47.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.627+0000 7f23fddff700 1 -- 172.21.15.27:0/625573340 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f23f804c3c0 con 0x7f23f80cb2e0 2024-01-11T08:22:47.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.628+0000 7f23df7fe700 1 -- 172.21.15.27:0/625573340 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f23e8056320 con 0x7f23f80cb2e0 2024-01-11T08:22:47.627 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:22:47.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.631+0000 7f23fddff700 1 -- 172.21.15.27:0/625573340 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f23e40670e0 msgr2=0x7f23e4069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:22:47.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.631+0000 7f23fddff700 1 --2- 172.21.15.27:0/625573340 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f23e40670e0 0x7f23e4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:47.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.631+0000 7f23fddff700 1 -- 172.21.15.27:0/625573340 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23f80cb2e0 msgr2=0x7f23f81556d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:47.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.631+0000 7f23fddff700 1 --2- 172.21.15.27:0/625573340 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23f80cb2e0 0x7f23f81556d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f23f8064a70 tx=0x7f23e8017000 comp rx=0 tx=0).stop 2024-01-11T08:22:47.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.631+0000 7f23fddff700 1 -- 172.21.15.27:0/625573340 shutdown_connections 2024-01-11T08:22:47.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.631+0000 7f23fddff700 1 --2- 172.21.15.27:0/625573340 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f23e40670e0 0x7f23e4069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:47.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.631+0000 7f23fddff700 1 --2- 172.21.15.27:0/625573340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f8143190 0x7f23f80b2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:47.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.631+0000 7f23fddff700 1 --2- 172.21.15.27:0/625573340 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23f80cb2e0 0x7f23f81556d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:47.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.631+0000 7f23fddff700 1 --2- 172.21.15.27:0/625573340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f8142dc0 0x7f23f81410f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:47.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.631+0000 7f23fddff700 1 -- 172.21.15.27:0/625573340 >> 172.21.15.27:0/625573340 conn(0x7f23f8063ca0 msgr2=0x7f23f8142040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:47.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.631+0000 7f23fddff700 1 -- 172.21.15.27:0/625573340 shutdown_connections 2024-01-11T08:22:47.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.631+0000 7f23fddff700 1 -- 172.21.15.27:0/625573340 wait complete. 2024-01-11T08:22:47.656 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T08:22:47.772 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:47.773+0000 7f4a678c91c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:22:47.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.871+0000 7f26663ff700 1 Processor -- start 2024-01-11T08:22:47.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.871+0000 7f26663ff700 1 -- start start 2024-01-11T08:22:47.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.872+0000 7f26663ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f26601577e0 0x7f26600ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:47.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.872+0000 7f26663ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26600699d0 con 0x7f2660069540 2024-01-11T08:22:47.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.872+0000 7f26663ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2660069cb0 con 0x7f2660157410 2024-01-11T08:22:47.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.872+0000 7f26663ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2660069f90 con 0x7f26601577e0 2024-01-11T08:22:47.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.872+0000 7f266499c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2660069540 0x7f26600c6ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34284/0 (socket says 172.21.15.27:34284) 2024-01-11T08:22:47.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.872+0000 7f266499c700 1 -- 172.21.15.27:0/1878359236 learned_addr learned my addr 172.21.15.27:0/1878359236 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:47.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.872+0000 7f265f7fe700 1 --2- 172.21.15.27:0/1878359236 >> v2:172.21.15.77:3300/0 conn(0x7f26601577e0 0x7f26600ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.873+0000 7f265f7fe700 1 -- 172.21.15.27:0/1878359236 >> v1:172.21.15.77:6789/0 conn(0x7f2660157410 legacy=0x7f26600ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.873+0000 7f265f7fe700 1 -- 172.21.15.27:0/1878359236 >> v1:172.21.15.27:6789/0 conn(0x7f2660069540 legacy=0x7f26600c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.873+0000 7f265f7fe700 1 -- 172.21.15.27:0/1878359236 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266006a270 con 0x7f26601577e0 2024-01-11T08:22:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.873+0000 7f265effd700 1 -- 172.21.15.27:0/1878359236 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260784440 0 0) 0x7f26600699d0 con 0x7f2660069540 2024-01-11T08:22:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.873+0000 7f265f7fe700 1 --2- 172.21.15.27:0/1878359236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26601577e0 0x7f26600ae000 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f26540098e0 tx=0x7f26540091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a64fe116f3488d6f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.873+0000 7f265effd700 1 -- 172.21.15.27:0/1878359236 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2654009e80 con 0x7f26601577e0 2024-01-11T08:22:47.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.874+0000 7f265effd700 1 -- 172.21.15.27:0/1878359236 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f265400b860 con 0x7f26601577e0 2024-01-11T08:22:47.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.874+0000 7f265effd700 1 -- 172.21.15.27:0/1878359236 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265400bbe0 con 0x7f26601577e0 2024-01-11T08:22:47.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.874+0000 7f26663ff700 1 -- 172.21.15.27:0/1878359236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26601577e0 msgr2=0x7f26600ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:47.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.874+0000 7f26663ff700 1 --2- 172.21.15.27:0/1878359236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26601577e0 0x7f26600ae000 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f26540098e0 tx=0x7f26540091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:47.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.874+0000 7f26663ff700 1 -- 172.21.15.27:0/1878359236 shutdown_connections 2024-01-11T08:22:47.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.874+0000 7f26663ff700 1 --2- 172.21.15.27:0/1878359236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26601577e0 0x7f26600ae000 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:47.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.874+0000 7f26663ff700 1 -- 172.21.15.27:0/1878359236 >> 172.21.15.27:0/1878359236 conn(0x7f26600680a0 msgr2=0x7f26600cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.874+0000 7f26663ff700 1 -- 172.21.15.27:0/1878359236 shutdown_connections 2024-01-11T08:22:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.874+0000 7f26663ff700 1 -- 172.21.15.27:0/1878359236 wait complete. 2024-01-11T08:22:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.876+0000 7f26663ff700 1 Processor -- start 2024-01-11T08:22:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.876+0000 7f26663ff700 1 -- start start 2024-01-11T08:22:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.877+0000 7f26663ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2660069540 0x7f26601537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.877+0000 7f26663ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2660157410 0x7f2660154ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.877+0000 7f26663ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26601577e0 0x7f2660143b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.877+0000 7f26663ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2660154a40 con 0x7f2660157410 2024-01-11T08:22:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.877+0000 7f26663ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2660153e30 con 0x7f26601577e0 2024-01-11T08:22:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.877+0000 7f26663ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2660154110 con 0x7f2660069540 2024-01-11T08:22:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.877+0000 7f265f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2660157410 0x7f2660154ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.877+0000 7f266499c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26601577e0 0x7f2660143b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.877+0000 7f265f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2660157410 0x7f2660154ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54336/0 (socket says 172.21.15.27:54336) 2024-01-11T08:22:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.877+0000 7f265f7fe700 1 -- 172.21.15.27:0/2967213867 learned_addr learned my addr 172.21.15.27:0/2967213867 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.877+0000 7f265ffff700 1 --2- 172.21.15.27:0/2967213867 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2660069540 0x7f26601537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.878+0000 7f265f7fe700 1 -- 172.21.15.27:0/2967213867 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2660069540 msgr2=0x7f26601537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.878+0000 7f265f7fe700 1 --2- 172.21.15.27:0/2967213867 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2660069540 0x7f26601537a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:47.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.878+0000 7f265f7fe700 1 -- 172.21.15.27:0/2967213867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26601577e0 msgr2=0x7f2660143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:47.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.878+0000 7f265f7fe700 1 --2- 172.21.15.27:0/2967213867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26601577e0 0x7f2660143b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:47.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.878+0000 7f265f7fe700 1 -- 172.21.15.27:0/2967213867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2654009420 con 0x7f2660157410 2024-01-11T08:22:47.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.878+0000 7f266499c700 1 --2- 172.21.15.27:0/2967213867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26601577e0 0x7f2660143b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:22:47.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.878+0000 7f265f7fe700 1 --2- 172.21.15.27:0/2967213867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2660157410 0x7f2660154ed0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f2654013000 tx=0x7f2654017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:47.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.879+0000 7f2647fff700 1 -- 172.21.15.27:0/2967213867 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265401f070 con 0x7f2660157410 2024-01-11T08:22:47.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.879+0000 7f2647fff700 1 -- 172.21.15.27:0/2967213867 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2654000c90 con 0x7f2660157410 2024-01-11T08:22:47.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.879+0000 7f26663ff700 1 -- 172.21.15.27:0/2967213867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26601543f0 con 0x7f2660157410 2024-01-11T08:22:47.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.879+0000 7f2647fff700 1 -- 172.21.15.27:0/2967213867 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2654025420 con 0x7f2660157410 2024-01-11T08:22:47.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.879+0000 7f26663ff700 1 -- 172.21.15.27:0/2967213867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2660156ba0 con 0x7f2660157410 2024-01-11T08:22:47.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.880+0000 7f26663ff700 1 -- 172.21.15.27:0/2967213867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2660154bb0 con 0x7f2660157410 2024-01-11T08:22:47.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.882+0000 7f2647fff700 1 -- 172.21.15.27:0/2967213867 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7f2654023070 con 0x7f2660157410 2024-01-11T08:22:47.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.883+0000 7f2647fff700 1 --2- 172.21.15.27:0/2967213867 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f264c0670e0 0x7f264c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:47.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.883+0000 7f2647fff700 1 -- 172.21.15.27:0/2967213867 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f2654085ed0 con 0x7f2660157410 2024-01-11T08:22:47.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.883+0000 7f265ffff700 1 -- 172.21.15.27:0/2967213867 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f264c0670e0 msgr2=0x7f264c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:47.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.883+0000 7f265ffff700 1 --2- 172.21.15.27:0/2967213867 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f264c0670e0 0x7f264c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:22:47.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:47.888+0000 7f2647fff700 1 -- 172.21.15.27:0/2967213867 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2654052c30 con 0x7f2660157410 2024-01-11T08:22:48.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.084+0000 7f265ffff700 1 -- 172.21.15.27:0/2967213867 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f264c0670e0 msgr2=0x7f264c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:48.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.084+0000 7f265ffff700 1 --2- 172.21.15.27:0/2967213867 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f264c0670e0 0x7f264c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:22:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.092+0000 7f26663ff700 1 -- 172.21.15.27:0/2967213867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f266004c3c0 con 0x7f2660157410 2024-01-11T08:22:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.092+0000 7f2647fff700 1 -- 172.21.15.27:0/2967213867 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v46) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2654015020 con 0x7f2660157410 2024-01-11T08:22:48.092 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T08:22:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.095+0000 7f26663ff700 1 -- 172.21.15.27:0/2967213867 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f264c0670e0 msgr2=0x7f264c069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:22:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.095+0000 7f26663ff700 1 --2- 172.21.15.27:0/2967213867 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f264c0670e0 0x7f264c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.095+0000 7f26663ff700 1 -- 172.21.15.27:0/2967213867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2660157410 msgr2=0x7f2660154ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.095+0000 7f26663ff700 1 --2- 172.21.15.27:0/2967213867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2660157410 0x7f2660154ed0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f2654013000 tx=0x7f2654017000 comp rx=0 tx=0).stop 2024-01-11T08:22:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.095+0000 7f26663ff700 1 -- 172.21.15.27:0/2967213867 shutdown_connections 2024-01-11T08:22:48.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.095+0000 7f26663ff700 1 --2- 172.21.15.27:0/2967213867 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f264c0670e0 0x7f264c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:48.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.095+0000 7f26663ff700 1 --2- 172.21.15.27:0/2967213867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26601577e0 0x7f2660143b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:48.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.095+0000 7f26663ff700 1 --2- 172.21.15.27:0/2967213867 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2660069540 0x7f26601537a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:48.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.095+0000 7f26663ff700 1 --2- 172.21.15.27:0/2967213867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2660157410 0x7f2660154ed0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:48.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.095+0000 7f26663ff700 1 -- 172.21.15.27:0/2967213867 >> 172.21.15.27:0/2967213867 conn(0x7f26600680a0 msgr2=0x7f26600af120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:48.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.095+0000 7f26663ff700 1 -- 172.21.15.27:0/2967213867 shutdown_connections 2024-01-11T08:22:48.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.095+0000 7f26663ff700 1 -- 172.21.15.27:0/2967213867 wait complete. 2024-01-11T08:22:48.112 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T08:22:48.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.317+0000 7f59905ff700 1 Processor -- start 2024-01-11T08:22:48.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.317+0000 7f59905ff700 1 -- start start 2024-01-11T08:22:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.318+0000 7f59905ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5988068d30 0x7f59880c8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.318+0000 7f59905ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59880c80a0 con 0x7f5988159ac0 2024-01-11T08:22:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.318+0000 7f59905ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59880cbac0 con 0x7f59881596f0 2024-01-11T08:22:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.318+0000 7f59905ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59880a9fb0 con 0x7f5988068d30 2024-01-11T08:22:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.319+0000 7f598db9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5988159ac0 0x7f5988068620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34298/0 (socket says 172.21.15.27:34298) 2024-01-11T08:22:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.319+0000 7f598e39b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f59881596f0 0x7f59880a98a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:58700/0 (socket says 172.21.15.27:58700) 2024-01-11T08:22:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.319+0000 7f598db9a700 1 -- 172.21.15.27:0/2682757270 learned_addr learned my addr 172.21.15.27:0/2682757270 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.319+0000 7f598eb9c700 1 --2- 172.21.15.27:0/2682757270 >> v2:172.21.15.77:3300/0 conn(0x7f5988068d30 0x7f59880c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.319+0000 7f598d399700 1 -- 172.21.15.27:0/2682757270 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733435912 0 0) 0x7f59880cbac0 con 0x7f59881596f0 2024-01-11T08:22:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.319+0000 7f598d399700 1 -- 172.21.15.27:0/2682757270 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5974003570 con 0x7f59881596f0 2024-01-11T08:22:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.319+0000 7f598d399700 1 -- 172.21.15.27:0/2682757270 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482155731 0 0) 0x7f59880c80a0 con 0x7f5988159ac0 2024-01-11T08:22:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.319+0000 7f598d399700 1 -- 172.21.15.27:0/2682757270 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59880cbac0 con 0x7f5988159ac0 2024-01-11T08:22:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.319+0000 7f598eb9c700 1 -- 172.21.15.27:0/2682757270 >> v1:172.21.15.77:6789/0 conn(0x7f59881596f0 legacy=0x7f59880a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.319+0000 7f598eb9c700 1 -- 172.21.15.27:0/2682757270 >> v1:172.21.15.27:6789/0 conn(0x7f5988159ac0 legacy=0x7f5988068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.319+0000 7f598eb9c700 1 -- 172.21.15.27:0/2682757270 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59880c7490 con 0x7f5988068d30 2024-01-11T08:22:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.320+0000 7f598eb9c700 1 --2- 172.21.15.27:0/2682757270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5988068d30 0x7f59880c8530 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5980009e60 tx=0x7f5980009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1faa07cf295c302 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.320+0000 7f598d399700 1 -- 172.21.15.27:0/2682757270 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598000e070 con 0x7f5988068d30 2024-01-11T08:22:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.320+0000 7f598d399700 1 -- 172.21.15.27:0/2682757270 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598000bab0 con 0x7f5988068d30 2024-01-11T08:22:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.320+0000 7f598d399700 1 -- 172.21.15.27:0/2682757270 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598000be70 con 0x7f5988068d30 2024-01-11T08:22:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.321+0000 7f59905ff700 1 -- 172.21.15.27:0/2682757270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5988068d30 msgr2=0x7f59880c8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.321+0000 7f59905ff700 1 --2- 172.21.15.27:0/2682757270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5988068d30 0x7f59880c8530 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5980009e60 tx=0x7f5980009780 comp rx=0 tx=0).stop 2024-01-11T08:22:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.321+0000 7f59905ff700 1 -- 172.21.15.27:0/2682757270 shutdown_connections 2024-01-11T08:22:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.321+0000 7f59905ff700 1 --2- 172.21.15.27:0/2682757270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5988068d30 0x7f59880c8530 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.321+0000 7f59905ff700 1 -- 172.21.15.27:0/2682757270 >> 172.21.15.27:0/2682757270 conn(0x7f59880c9fb0 msgr2=0x7f598806ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.321+0000 7f59905ff700 1 -- 172.21.15.27:0/2682757270 shutdown_connections 2024-01-11T08:22:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.321+0000 7f59905ff700 1 -- 172.21.15.27:0/2682757270 wait complete. 2024-01-11T08:22:48.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.322+0000 7f59905ff700 1 Processor -- start 2024-01-11T08:22:48.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.323+0000 7f59905ff700 1 -- start start 2024-01-11T08:22:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.324+0000 7f59905ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5988068d30 0x7f59880b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.324+0000 7f59905ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59881596f0 0x7f59880b1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.324+0000 7f59905ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5988159ac0 0x7f59880c5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.324+0000 7f59905ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59880ac490 con 0x7f5988068d30 2024-01-11T08:22:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.324+0000 7f59905ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59880b1900 con 0x7f5988159ac0 2024-01-11T08:22:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.324+0000 7f59905ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59880b1be0 con 0x7f59881596f0 2024-01-11T08:22:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.324+0000 7f598e39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5988068d30 0x7f59880b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.324+0000 7f598e39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5988068d30 0x7f59880b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54342/0 (socket says 172.21.15.27:54342) 2024-01-11T08:22:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.324+0000 7f598e39b700 1 -- 172.21.15.27:0/1698537674 learned_addr learned my addr 172.21.15.27:0/1698537674 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.324+0000 7f598db9a700 1 --2- 172.21.15.27:0/1698537674 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59881596f0 0x7f59880b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.324+0000 7f598eb9c700 1 --2- 172.21.15.27:0/1698537674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5988159ac0 0x7f59880c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.324+0000 7f598e39b700 1 -- 172.21.15.27:0/1698537674 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59881596f0 msgr2=0x7f59880b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.324+0000 7f598e39b700 1 --2- 172.21.15.27:0/1698537674 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59881596f0 0x7f59880b1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.324+0000 7f598e39b700 1 -- 172.21.15.27:0/1698537674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5988159ac0 msgr2=0x7f59880c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.324+0000 7f598e39b700 1 --2- 172.21.15.27:0/1698537674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5988159ac0 0x7f59880c5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.324+0000 7f598e39b700 1 -- 172.21.15.27:0/1698537674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5980009a20 con 0x7f5988068d30 2024-01-11T08:22:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.325+0000 7f598e39b700 1 --2- 172.21.15.27:0/1698537674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5988068d30 0x7f59880b0b40 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f598400c7c0 tx=0x7f5984007e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.325+0000 7f597e7fc700 1 -- 172.21.15.27:0/1698537674 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984005c10 con 0x7f5988068d30 2024-01-11T08:22:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.325+0000 7f597e7fc700 1 -- 172.21.15.27:0/1698537674 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5984005d70 con 0x7f5988068d30 2024-01-11T08:22:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.325+0000 7f59905ff700 1 -- 172.21.15.27:0/1698537674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59880b1ec0 con 0x7f5988068d30 2024-01-11T08:22:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.325+0000 7f597e7fc700 1 -- 172.21.15.27:0/1698537674 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984007320 con 0x7f5988068d30 2024-01-11T08:22:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.325+0000 7f59905ff700 1 -- 172.21.15.27:0/1698537674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59880b15c0 con 0x7f5988068d30 2024-01-11T08:22:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.327+0000 7f59905ff700 1 -- 172.21.15.27:0/1698537674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f598804c3c0 con 0x7f5988068d30 2024-01-11T08:22:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.327+0000 7f597e7fc700 1 -- 172.21.15.27:0/1698537674 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7f598401b3b0 con 0x7f5988068d30 2024-01-11T08:22:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.328+0000 7f597e7fc700 1 --2- 172.21.15.27:0/1698537674 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f596c0670e0 0x7f596c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.328+0000 7f597e7fc700 1 -- 172.21.15.27:0/1698537674 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f59840841a0 con 0x7f5988068d30 2024-01-11T08:22:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.328+0000 7f598db9a700 1 -- 172.21.15.27:0/1698537674 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f596c0670e0 msgr2=0x7f596c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.328+0000 7f598db9a700 1 --2- 172.21.15.27:0/1698537674 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f596c0670e0 0x7f596c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:22:48.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.336+0000 7f597e7fc700 1 -- 172.21.15.27:0/1698537674 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5984050f00 con 0x7f5988068d30 2024-01-11T08:22:48.392 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:48.393+0000 7f909e5aa1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:22:48.479 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:48.481+0000 7f909e5aa1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:22:48.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.529+0000 7f598db9a700 1 -- 172.21.15.27:0/1698537674 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f596c0670e0 msgr2=0x7f596c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:48.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.529+0000 7f598db9a700 1 --2- 172.21.15.27:0/1698537674 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f596c0670e0 0x7f596c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:22:48.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.543+0000 7f59905ff700 1 -- 172.21.15.27:0/1698537674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f59880c7770 con 0x7f5988068d30 2024-01-11T08:22:48.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.543+0000 7f597e7fc700 1 -- 172.21.15.27:0/1698537674 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v46) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5984054520 con 0x7f5988068d30 2024-01-11T08:22:48.543 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T08:22:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.547+0000 7f59905ff700 1 -- 172.21.15.27:0/1698537674 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f596c0670e0 msgr2=0x7f596c069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:22:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.547+0000 7f59905ff700 1 --2- 172.21.15.27:0/1698537674 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f596c0670e0 0x7f596c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.547+0000 7f59905ff700 1 -- 172.21.15.27:0/1698537674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5988068d30 msgr2=0x7f59880b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.547+0000 7f59905ff700 1 --2- 172.21.15.27:0/1698537674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5988068d30 0x7f59880b0b40 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f598400c7c0 tx=0x7f5984007e20 comp rx=0 tx=0).stop 2024-01-11T08:22:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.547+0000 7f59905ff700 1 -- 172.21.15.27:0/1698537674 shutdown_connections 2024-01-11T08:22:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.547+0000 7f59905ff700 1 --2- 172.21.15.27:0/1698537674 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f596c0670e0 0x7f596c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.547+0000 7f59905ff700 1 --2- 172.21.15.27:0/1698537674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5988159ac0 0x7f59880c5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.547+0000 7f59905ff700 1 --2- 172.21.15.27:0/1698537674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5988068d30 0x7f59880b0b40 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.547+0000 7f59905ff700 1 --2- 172.21.15.27:0/1698537674 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59881596f0 0x7f59880b1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.547+0000 7f59905ff700 1 -- 172.21.15.27:0/1698537674 >> 172.21.15.27:0/1698537674 conn(0x7f59880c9fb0 msgr2=0x7f59880cac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.548+0000 7f59905ff700 1 -- 172.21.15.27:0/1698537674 shutdown_connections 2024-01-11T08:22:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.548+0000 7f59905ff700 1 -- 172.21.15.27:0/1698537674 wait complete. 2024-01-11T08:22:48.574 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:22:48.630 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:48.631+0000 7f4a678c91c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:22:48.634 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:48.636+0000 7f909e5aa1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:22:48.721 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:48.722+0000 7f909e5aa1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:22:48.725 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:48.726+0000 7f4a678c91c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:22:48.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.781+0000 7f6d667ff700 1 Processor -- start 2024-01-11T08:22:48.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.782+0000 7f6d667ff700 1 -- start start 2024-01-11T08:22:48.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.783+0000 7f6d667ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6d6006c060 0x7f6d600cd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:48.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.783+0000 7f6d667ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d600c9040 0x7f6d600cd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:48.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.783+0000 7f6d667ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d6015bc30 con 0x7f6d600c8c70 2024-01-11T08:22:48.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.783+0000 7f6d667ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d600cde90 con 0x7f6d6006c060 2024-01-11T08:22:48.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.783+0000 7f6d667ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d60158c40 con 0x7f6d600c9040 2024-01-11T08:22:48.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.783+0000 7f6d5ffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6d600c8c70 0x7f6d6006b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34308/0 (socket says 172.21.15.27:34308) 2024-01-11T08:22:48.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.783+0000 7f6d5ffff700 1 -- 172.21.15.27:0/1304004592 learned_addr learned my addr 172.21.15.27:0/1304004592 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:48.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.783+0000 7f6d5f7fe700 1 --2- 172.21.15.27:0/1304004592 >> v2:172.21.15.27:3300/0 conn(0x7f6d6006c060 0x7f6d600cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:48.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.783+0000 7f6d64d9c700 1 --2- 172.21.15.27:0/1304004592 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d600c9040 0x7f6d600cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:48.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.784+0000 7f6d5effd700 1 -- 172.21.15.27:0/1304004592 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772057807 0 0) 0x7f6d6015bc30 con 0x7f6d600c8c70 2024-01-11T08:22:48.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.784+0000 7f6d5effd700 1 -- 172.21.15.27:0/1304004592 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d48003570 con 0x7f6d600c8c70 2024-01-11T08:22:48.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.784+0000 7f6d5f7fe700 1 -- 172.21.15.27:0/1304004592 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d600c9040 msgr2=0x7f6d600cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.784+0000 7f6d5f7fe700 1 --2- 172.21.15.27:0/1304004592 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d600c9040 0x7f6d600cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.784+0000 7f6d5f7fe700 1 -- 172.21.15.27:0/1304004592 >> v1:172.21.15.27:6789/0 conn(0x7f6d600c8c70 legacy=0x7f6d6006b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.784+0000 7f6d5f7fe700 1 -- 172.21.15.27:0/1304004592 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6006c5a0 con 0x7f6d6006c060 2024-01-11T08:22:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.784+0000 7f6d5f7fe700 1 --2- 172.21.15.27:0/1304004592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d6006c060 0x7f6d600cd410 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6d540098e0 tx=0x7f6d540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c14e625c7d011b03 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.785+0000 7f6d5effd700 1 -- 172.21.15.27:0/1304004592 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d54011630 con 0x7f6d6006c060 2024-01-11T08:22:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.785+0000 7f6d5effd700 1 -- 172.21.15.27:0/1304004592 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d54011790 con 0x7f6d6006c060 2024-01-11T08:22:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.785+0000 7f6d5effd700 1 -- 172.21.15.27:0/1304004592 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d54011b30 con 0x7f6d6006c060 2024-01-11T08:22:48.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.785+0000 7f6d667ff700 1 -- 172.21.15.27:0/1304004592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d6006c060 msgr2=0x7f6d600cd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:48.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.785+0000 7f6d667ff700 1 --2- 172.21.15.27:0/1304004592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d6006c060 0x7f6d600cd410 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6d540098e0 tx=0x7f6d540091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:48.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.785+0000 7f6d667ff700 1 -- 172.21.15.27:0/1304004592 shutdown_connections 2024-01-11T08:22:48.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.785+0000 7f6d667ff700 1 --2- 172.21.15.27:0/1304004592 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d600c9040 0x7f6d600cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:48.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.785+0000 7f6d667ff700 1 --2- 172.21.15.27:0/1304004592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d6006c060 0x7f6d600cd410 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.785+0000 7f6d667ff700 1 -- 172.21.15.27:0/1304004592 >> 172.21.15.27:0/1304004592 conn(0x7f6d600ce3e0 msgr2=0x7f6d600ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.785+0000 7f6d667ff700 1 -- 172.21.15.27:0/1304004592 shutdown_connections 2024-01-11T08:22:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.785+0000 7f6d667ff700 1 -- 172.21.15.27:0/1304004592 wait complete. 2024-01-11T08:22:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.787+0000 7f6d667ff700 1 Processor -- start 2024-01-11T08:22:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.787+0000 7f6d667ff700 1 -- start start 2024-01-11T08:22:48.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.788+0000 7f6d667ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d6006c060 0x7f6d6016cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:48.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.788+0000 7f6d667ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d600c8c70 0x7f6d6014cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:48.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.788+0000 7f6d667ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d600c9040 0x7f6d60157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:48.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.788+0000 7f6d667ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6014ff70 con 0x7f6d600c8c70 2024-01-11T08:22:48.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.788+0000 7f6d667ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6016de90 con 0x7f6d600c9040 2024-01-11T08:22:48.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.788+0000 7f6d667ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6014fc40 con 0x7f6d6006c060 2024-01-11T08:22:48.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.788+0000 7f6d5f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d600c8c70 0x7f6d6014cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:48.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.788+0000 7f6d5ffff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d6006c060 0x7f6d6016cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:48.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.788+0000 7f6d64d9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d600c9040 0x7f6d60157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:48.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.788+0000 7f6d5ffff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d6006c060 0x7f6d6016cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:38968/0 (socket says 172.21.15.27:38968) 2024-01-11T08:22:48.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.788+0000 7f6d5ffff700 1 -- 172.21.15.27:0/1176569829 learned_addr learned my addr 172.21.15.27:0/1176569829 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:48.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.789+0000 7f6d5ffff700 1 -- 172.21.15.27:0/1176569829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d600c9040 msgr2=0x7f6d60157c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:48.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.789+0000 7f6d5ffff700 1 --2- 172.21.15.27:0/1176569829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d600c9040 0x7f6d60157c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:48.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.789+0000 7f6d5ffff700 1 -- 172.21.15.27:0/1176569829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d600c8c70 msgr2=0x7f6d6014cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:48.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.789+0000 7f6d5ffff700 1 --2- 172.21.15.27:0/1176569829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d600c8c70 0x7f6d6014cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:48.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.789+0000 7f6d5ffff700 1 -- 172.21.15.27:0/1176569829 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d54009420 con 0x7f6d6006c060 2024-01-11T08:22:48.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.789+0000 7f6d5ffff700 1 --2- 172.21.15.27:0/1176569829 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d6006c060 0x7f6d6016cfe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6d5000c7c0 tx=0x7f6d50007e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:48.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.789+0000 7f6d47fff700 1 -- 172.21.15.27:0/1176569829 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d50005cd0 con 0x7f6d6006c060 2024-01-11T08:22:48.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.789+0000 7f6d47fff700 1 -- 172.21.15.27:0/1176569829 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d50005e30 con 0x7f6d6006c060 2024-01-11T08:22:48.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.790+0000 7f6d667ff700 1 -- 172.21.15.27:0/1176569829 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d6016d8d0 con 0x7f6d6006c060 2024-01-11T08:22:48.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.790+0000 7f6d47fff700 1 -- 172.21.15.27:0/1176569829 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d50007320 con 0x7f6d6006c060 2024-01-11T08:22:48.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.790+0000 7f6d667ff700 1 -- 172.21.15.27:0/1176569829 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d6016d520 con 0x7f6d6006c060 2024-01-11T08:22:48.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.791+0000 7f6d667ff700 1 -- 172.21.15.27:0/1176569829 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d600718d0 con 0x7f6d6006c060 2024-01-11T08:22:48.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.792+0000 7f6d47fff700 1 -- 172.21.15.27:0/1176569829 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7f6d5001b3b0 con 0x7f6d6006c060 2024-01-11T08:22:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.792+0000 7f6d47fff700 1 --2- 172.21.15.27:0/1176569829 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6d4c0670e0 0x7f6d4c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.793+0000 7f6d47fff700 1 -- 172.21.15.27:0/1176569829 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f6d50084230 con 0x7f6d6006c060 2024-01-11T08:22:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.793+0000 7f6d5f7fe700 1 -- 172.21.15.27:0/1176569829 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6d4c0670e0 msgr2=0x7f6d4c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.793+0000 7f6d5f7fe700 1 --2- 172.21.15.27:0/1176569829 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6d4c0670e0 0x7f6d4c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:22:48.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.797+0000 7f6d47fff700 1 -- 172.21.15.27:0/1176569829 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6d50050ef0 con 0x7f6d6006c060 2024-01-11T08:22:48.897 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:48.897+0000 7f4a678c91c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:22:48.938 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:48.939+0000 7f909e5aa1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:22:48.988 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:48.988+0000 7f4a678c91c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:22:48.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.994+0000 7f6d5f7fe700 1 -- 172.21.15.27:0/1176569829 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6d4c0670e0 msgr2=0x7f6d4c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:48.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:48.994+0000 7f6d5f7fe700 1 --2- 172.21.15.27:0/1176569829 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6d4c0670e0 0x7f6d4c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:22:49.034 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:49.035+0000 7f909e5aa1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:22:49.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.049+0000 7f6d667ff700 1 -- 172.21.15.27:0/1176569829 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d6015b300 con 0x7f6d6006c060 2024-01-11T08:22:49.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.050+0000 7f6d47fff700 1 -- 172.21.15.27:0/1176569829 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4591 (secure 0 0 0) 0x7f6d50054510 con 0x7f6d6006c060 2024-01-11T08:22:49.049 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:22:49.049 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/2930156910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2930156910},{"type":"v1","addr":"172.21.15.77:6836","nonce":2930156910}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/685019444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":685019444},{"type":"v1","addr":"172.21.15.27:6834","nonce":685019444}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4861,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/144853545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":144853545},{"type":"v1","addr":"172.21.15.77:6837","nonce":144853545}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:11:24.435181+0000","modified":"2024-01-11T08:11:27.940837+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/4056690675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4056690675},{"type":"v1","addr":"172.21.15.27:6835","nonce":4056690675}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2024-01-11T08:22:49.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.052+0000 7f6d667ff700 1 -- 172.21.15.27:0/1176569829 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6d4c0670e0 msgr2=0x7f6d4c069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:22:49.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.052+0000 7f6d667ff700 1 --2- 172.21.15.27:0/1176569829 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6d4c0670e0 0x7f6d4c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.052+0000 7f6d667ff700 1 -- 172.21.15.27:0/1176569829 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d6006c060 msgr2=0x7f6d6016cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:49.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.052+0000 7f6d667ff700 1 --2- 172.21.15.27:0/1176569829 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d6006c060 0x7f6d6016cfe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6d5000c7c0 tx=0x7f6d50007e00 comp rx=0 tx=0).stop 2024-01-11T08:22:49.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.052+0000 7f6d667ff700 1 -- 172.21.15.27:0/1176569829 shutdown_connections 2024-01-11T08:22:49.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.052+0000 7f6d667ff700 1 --2- 172.21.15.27:0/1176569829 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6d4c0670e0 0x7f6d4c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.052+0000 7f6d667ff700 1 --2- 172.21.15.27:0/1176569829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d600c9040 0x7f6d60157c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.052+0000 7f6d667ff700 1 --2- 172.21.15.27:0/1176569829 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d6006c060 0x7f6d6016cfe0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.053+0000 7f6d667ff700 1 --2- 172.21.15.27:0/1176569829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d600c8c70 0x7f6d6014cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.053+0000 7f6d667ff700 1 -- 172.21.15.27:0/1176569829 >> 172.21.15.27:0/1176569829 conn(0x7f6d600ce3e0 msgr2=0x7f6d6014eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:49.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.053+0000 7f6d667ff700 1 -- 172.21.15.27:0/1176569829 shutdown_connections 2024-01-11T08:22:49.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.053+0000 7f6d667ff700 1 -- 172.21.15.27:0/1176569829 wait complete. 2024-01-11T08:22:49.053 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-01-11T08:22:49.071 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:22:49.129 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:49.130+0000 7f909e5aa1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:22:49.217 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:49.217+0000 7f4a678c91c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:22:49.224 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:49.225+0000 7f909e5aa1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:22:49.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.278+0000 7f071ddff700 1 Processor -- start 2024-01-11T08:22:49.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.278+0000 7f071ddff700 1 -- start start 2024-01-11T08:22:49.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.280+0000 7f071ddff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f07180b2b90 0x7f0718159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:49.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.280+0000 7f071ddff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07180b07b0 0x7f0718159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:49.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.280+0000 7f071ddff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07180cd730 con 0x7f07180b03e0 2024-01-11T08:22:49.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.280+0000 7f071ddff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07180ccb20 con 0x7f07180b2b90 2024-01-11T08:22:49.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.280+0000 7f071ddff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07180cce00 con 0x7f07180b07b0 2024-01-11T08:22:49.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.280+0000 7f0716ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f07180b2b90 0x7f0718159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:49.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.280+0000 7f0716ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f07180b2b90 0x7f0718159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54378/0 (socket says 172.21.15.27:54378) 2024-01-11T08:22:49.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.280+0000 7f0716ffd700 1 -- 172.21.15.27:0/2841212750 learned_addr learned my addr 172.21.15.27:0/2841212750 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:49.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.280+0000 7f0717fff700 1 --2- 172.21.15.27:0/2841212750 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07180b07b0 0x7f0718159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:49.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.280+0000 7f07167fc700 1 -- 172.21.15.27:0/2841212750 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025959197 0 0) 0x7f07180cd730 con 0x7f07180b03e0 2024-01-11T08:22:49.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.281+0000 7f07167fc700 1 -- 172.21.15.27:0/2841212750 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0700003570 con 0x7f07180b03e0 2024-01-11T08:22:49.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.281+0000 7f0716ffd700 1 -- 172.21.15.27:0/2841212750 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07180b07b0 msgr2=0x7f0718159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:49.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.281+0000 7f0716ffd700 1 --2- 172.21.15.27:0/2841212750 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07180b07b0 0x7f0718159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.281+0000 7f0716ffd700 1 -- 172.21.15.27:0/2841212750 >> v1:172.21.15.77:6789/0 conn(0x7f07180b03e0 legacy=0x7f07180b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:49.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.281+0000 7f0716ffd700 1 -- 172.21.15.27:0/2841212750 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07180cd0e0 con 0x7f07180b2b90 2024-01-11T08:22:49.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.281+0000 7f0717fff700 1 --2- 172.21.15.27:0/2841212750 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07180b07b0 0x7f0718159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:22:49.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.281+0000 7f0716ffd700 1 --2- 172.21.15.27:0/2841212750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07180b2b90 0x7f0718159200 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f070c0099b0 tx=0x7f070c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bae9f0cdbddbd277 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:49.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.281+0000 7f07167fc700 1 -- 172.21.15.27:0/2841212750 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070c00b680 con 0x7f07180b2b90 2024-01-11T08:22:49.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.281+0000 7f07167fc700 1 -- 172.21.15.27:0/2841212750 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f070c00b7e0 con 0x7f07180b2b90 2024-01-11T08:22:49.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.281+0000 7f07167fc700 1 -- 172.21.15.27:0/2841212750 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070c00bb80 con 0x7f07180b2b90 2024-01-11T08:22:49.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.282+0000 7f071ddff700 1 -- 172.21.15.27:0/2841212750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07180b2b90 msgr2=0x7f0718159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:49.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.282+0000 7f071ddff700 1 --2- 172.21.15.27:0/2841212750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07180b2b90 0x7f0718159200 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f070c0099b0 tx=0x7f070c009250 comp rx=0 tx=0).stop 2024-01-11T08:22:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.282+0000 7f071ddff700 1 -- 172.21.15.27:0/2841212750 shutdown_connections 2024-01-11T08:22:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.282+0000 7f071ddff700 1 --2- 172.21.15.27:0/2841212750 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07180b07b0 0x7f0718159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.282+0000 7f071ddff700 1 --2- 172.21.15.27:0/2841212750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07180b2b90 0x7f0718159200 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.282+0000 7f071ddff700 1 -- 172.21.15.27:0/2841212750 >> 172.21.15.27:0/2841212750 conn(0x7f071806ed40 msgr2=0x7f071806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.282+0000 7f071ddff700 1 -- 172.21.15.27:0/2841212750 shutdown_connections 2024-01-11T08:22:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.282+0000 7f071ddff700 1 -- 172.21.15.27:0/2841212750 wait complete. 2024-01-11T08:22:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.284+0000 7f071ddff700 1 Processor -- start 2024-01-11T08:22:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.284+0000 7f071ddff700 1 -- start start 2024-01-11T08:22:49.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.285+0000 7f071ddff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07180b03e0 0x7f07180c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.285+0000 7f071ddff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07180b07b0 0x7f07180c5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.285+0000 7f071ddff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07180b2b90 0x7f07180c5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.285+0000 7f071ddff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07180c93b0 con 0x7f07180b07b0 2024-01-11T08:22:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.285+0000 7f071ddff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07180c8fd0 con 0x7f07180b2b90 2024-01-11T08:22:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.285+0000 7f071ddff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0718158d20 con 0x7f07180b03e0 2024-01-11T08:22:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.285+0000 7f0716ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07180b07b0 0x7f07180c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.285+0000 7f0716ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07180b07b0 0x7f07180c5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54384/0 (socket says 172.21.15.27:54384) 2024-01-11T08:22:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.285+0000 7f0716ffd700 1 -- 172.21.15.27:0/212921150 learned_addr learned my addr 172.21.15.27:0/212921150 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.285+0000 7f07177fe700 1 --2- 172.21.15.27:0/212921150 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07180b03e0 0x7f07180c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.285+0000 7f0717fff700 1 --2- 172.21.15.27:0/212921150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07180b2b90 0x7f07180c5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.285+0000 7f0716ffd700 1 -- 172.21.15.27:0/212921150 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07180b03e0 msgr2=0x7f07180c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.285+0000 7f0716ffd700 1 --2- 172.21.15.27:0/212921150 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07180b03e0 0x7f07180c8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.285+0000 7f0716ffd700 1 -- 172.21.15.27:0/212921150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07180b2b90 msgr2=0x7f07180c5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:49.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.285+0000 7f0716ffd700 1 --2- 172.21.15.27:0/212921150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07180b2b90 0x7f07180c5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.286+0000 7f0716ffd700 1 -- 172.21.15.27:0/212921150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0710009a20 con 0x7f07180b07b0 2024-01-11T08:22:49.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.286+0000 7f0716ffd700 1 --2- 172.21.15.27:0/212921150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07180b07b0 0x7f07180c5030 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f070c009290 tx=0x7f070c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:49.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.286+0000 7f06ff7fe700 1 -- 172.21.15.27:0/212921150 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070c00b950 con 0x7f07180b07b0 2024-01-11T08:22:49.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.286+0000 7f06ff7fe700 1 -- 172.21.15.27:0/212921150 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f070c00bab0 con 0x7f07180b07b0 2024-01-11T08:22:49.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.286+0000 7f071ddff700 1 -- 172.21.15.27:0/212921150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f070c0094f0 con 0x7f07180b07b0 2024-01-11T08:22:49.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.286+0000 7f06ff7fe700 1 -- 172.21.15.27:0/212921150 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070c01a680 con 0x7f07180b07b0 2024-01-11T08:22:49.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.286+0000 7f071ddff700 1 -- 172.21.15.27:0/212921150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07180c4690 con 0x7f07180b07b0 2024-01-11T08:22:49.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.288+0000 7f071ddff700 1 -- 172.21.15.27:0/212921150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07180c9560 con 0x7f07180b07b0 2024-01-11T08:22:49.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.288+0000 7f06ff7fe700 1 -- 172.21.15.27:0/212921150 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7f070c01aa70 con 0x7f07180b07b0 2024-01-11T08:22:49.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.290+0000 7f06ff7fe700 1 --2- 172.21.15.27:0/212921150 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f07040670e0 0x7f0704069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:49.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.290+0000 7f06ff7fe700 1 -- 172.21.15.27:0/212921150 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f070c086630 con 0x7f07180b07b0 2024-01-11T08:22:49.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.290+0000 7f07177fe700 1 -- 172.21.15.27:0/212921150 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f07040670e0 msgr2=0x7f0704069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:49.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.290+0000 7f07177fe700 1 --2- 172.21.15.27:0/212921150 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f07040670e0 0x7f0704069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:22:49.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.294+0000 7f06ff7fe700 1 -- 172.21.15.27:0/212921150 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f07180c9560 con 0x7f07180b07b0 2024-01-11T08:22:49.321 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:49.322+0000 7f4a678c91c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:22:49.412 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:49.413+0000 7f909e5aa1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:22:49.424 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:49.425+0000 7f4a678c91c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:22:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.491+0000 7f07177fe700 1 -- 172.21.15.27:0/212921150 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f07040670e0 msgr2=0x7f0704069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.491+0000 7f07177fe700 1 --2- 172.21.15.27:0/212921150 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f07040670e0 0x7f0704069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:22:49.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.501+0000 7f071ddff700 1 -- 172.21.15.27:0/212921150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07180c9560 con 0x7f07180b07b0 2024-01-11T08:22:49.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.502+0000 7f06ff7fe700 1 -- 172.21.15.27:0/212921150 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16289 (secure 0 0 0) 0x7f07180c9560 con 0x7f07180b07b0 2024-01-11T08:22:49.502 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:22:49.502 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":46,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:22:46.732163+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:11:22.376067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-11T08:11:23.419993+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":46,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6833/41381":"2024-01-12T08:11:40.675932+0000","172.21.15.77:6832/41381":"2024-01-12T08:11:40.675932+0000","172.21.15.77:0/969431132":"2024-01-12T08:11:40.675932+0000","172.21.15.77:0/4222476855":"2024-01-12T08:11:40.675932+0000","172.21.15.77:0/3211563793":"2024-01-12T08:11:40.675932+0000","172.21.15.77:0/2257956400":"2024-01-12T08:11:40.675932+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":10,"length":2},{"begin":15,"length":2},{"begin":20,"length":3},{"begin":26,"length":2},{"begin":31,"length":3},{"begin":37,"length":2},{"begin":43,"length":2},{"begin":48,"length":2},{"begin":54,"length":2},{"begin":59,"length":2},{"begin":65,"length":2},{"begin":70,"length":2},{"begin":75,"length":2},{"begin":80,"length":3},{"begin":86,"length":2},{"begin":91,"length":3},{"begin":97,"length":2},{"begin":102,"length":3},{"begin":108,"length":2},{"begin":177,"length":45}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":104,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:22:49.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.506+0000 7f071ddff700 1 -- 172.21.15.27:0/212921150 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f07040670e0 msgr2=0x7f0704069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:22:49.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.506+0000 7f071ddff700 1 --2- 172.21.15.27:0/212921150 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f07040670e0 0x7f0704069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.506+0000 7f071ddff700 1 -- 172.21.15.27:0/212921150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07180b07b0 msgr2=0x7f07180c5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:49.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.506+0000 7f071ddff700 1 --2- 172.21.15.27:0/212921150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07180b07b0 0x7f07180c5030 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f070c009290 tx=0x7f070c017000 comp rx=0 tx=0).stop 2024-01-11T08:22:49.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.506+0000 7f071ddff700 1 -- 172.21.15.27:0/212921150 shutdown_connections 2024-01-11T08:22:49.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.506+0000 7f071ddff700 1 --2- 172.21.15.27:0/212921150 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f07040670e0 0x7f0704069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.506+0000 7f071ddff700 1 --2- 172.21.15.27:0/212921150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07180b2b90 0x7f07180c5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.506+0000 7f071ddff700 1 --2- 172.21.15.27:0/212921150 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07180b03e0 0x7f07180c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.506+0000 7f071ddff700 1 --2- 172.21.15.27:0/212921150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07180b07b0 0x7f07180c5030 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.506+0000 7f071ddff700 1 -- 172.21.15.27:0/212921150 >> 172.21.15.27:0/212921150 conn(0x7f071806ed40 msgr2=0x7f07180d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:49.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.507+0000 7f071ddff700 1 -- 172.21.15.27:0/212921150 shutdown_connections 2024-01-11T08:22:49.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.507+0000 7f071ddff700 1 -- 172.21.15.27:0/212921150 wait complete. 2024-01-11T08:22:49.508 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:49.509+0000 7f909e5aa1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:22:49.526 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-11T08:22:49.527 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:22:49.528 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:49.529+0000 7f4a678c91c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:22:49.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.730+0000 7fcf751ff700 1 Processor -- start 2024-01-11T08:22:49.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.730+0000 7fcf751ff700 1 -- start start 2024-01-11T08:22:49.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.731+0000 7fcf751ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcf700b2b90 0x7fcf70159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:49.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.731+0000 7fcf751ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf700b07b0 0x7fcf70159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:49.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.731+0000 7fcf751ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf700cd730 con 0x7fcf700b03e0 2024-01-11T08:22:49.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.731+0000 7fcf751ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf700ccb20 con 0x7fcf700b2b90 2024-01-11T08:22:49.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.731+0000 7fcf751ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf700cce00 con 0x7fcf700b07b0 2024-01-11T08:22:49.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.731+0000 7fcf6ed9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcf700b03e0 0x7fcf700b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34312/0 (socket says 172.21.15.27:34312) 2024-01-11T08:22:49.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.731+0000 7fcf6ed9d700 1 -- 172.21.15.27:0/3405372006 learned_addr learned my addr 172.21.15.27:0/3405372006 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:49.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.732+0000 7fcf6e59c700 1 --2- 172.21.15.27:0/3405372006 >> v2:172.21.15.77:3300/0 conn(0x7fcf700b2b90 0x7fcf70159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:49.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.732+0000 7fcf6f59e700 1 --2- 172.21.15.27:0/3405372006 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf700b07b0 0x7fcf70159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:49.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.732+0000 7fcf6dd9b700 1 -- 172.21.15.27:0/3405372006 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210423583 0 0) 0x7fcf700cd730 con 0x7fcf700b03e0 2024-01-11T08:22:49.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.732+0000 7fcf6dd9b700 1 -- 172.21.15.27:0/3405372006 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf58003570 con 0x7fcf700b03e0 2024-01-11T08:22:49.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.732+0000 7fcf6e59c700 1 -- 172.21.15.27:0/3405372006 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf700b07b0 msgr2=0x7fcf70159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:49.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.732+0000 7fcf6e59c700 1 --2- 172.21.15.27:0/3405372006 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf700b07b0 0x7fcf70159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.732+0000 7fcf6e59c700 1 -- 172.21.15.27:0/3405372006 >> v1:172.21.15.27:6789/0 conn(0x7fcf700b03e0 legacy=0x7fcf700b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:49.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.732+0000 7fcf6e59c700 1 -- 172.21.15.27:0/3405372006 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf700cd0e0 con 0x7fcf700b2b90 2024-01-11T08:22:49.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.732+0000 7fcf6f59e700 1 --2- 172.21.15.27:0/3405372006 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf700b07b0 0x7fcf70159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:22:49.735 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:49.732+0000 7f4a678c91c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:22:49.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.732+0000 7fcf6e59c700 1 --2- 172.21.15.27:0/3405372006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf700b2b90 0x7fcf70159200 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcf640098e0 tx=0x7fcf640091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=861eed25984d1275 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:49.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.733+0000 7fcf6dd9b700 1 -- 172.21.15.27:0/3405372006 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf64011630 con 0x7fcf700b2b90 2024-01-11T08:22:49.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.733+0000 7fcf6dd9b700 1 -- 172.21.15.27:0/3405372006 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf64011790 con 0x7fcf700b2b90 2024-01-11T08:22:49.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.733+0000 7fcf6dd9b700 1 -- 172.21.15.27:0/3405372006 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf64011b30 con 0x7fcf700b2b90 2024-01-11T08:22:49.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.733+0000 7fcf751ff700 1 -- 172.21.15.27:0/3405372006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf700b2b90 msgr2=0x7fcf70159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:49.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.733+0000 7fcf751ff700 1 --2- 172.21.15.27:0/3405372006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf700b2b90 0x7fcf70159200 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcf640098e0 tx=0x7fcf640091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.733+0000 7fcf751ff700 1 -- 172.21.15.27:0/3405372006 shutdown_connections 2024-01-11T08:22:49.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.733+0000 7fcf751ff700 1 --2- 172.21.15.27:0/3405372006 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf700b07b0 0x7fcf70159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.733+0000 7fcf751ff700 1 --2- 172.21.15.27:0/3405372006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf700b2b90 0x7fcf70159200 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.733+0000 7fcf751ff700 1 -- 172.21.15.27:0/3405372006 >> 172.21.15.27:0/3405372006 conn(0x7fcf7006ed40 msgr2=0x7fcf7006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:49.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.734+0000 7fcf751ff700 1 -- 172.21.15.27:0/3405372006 shutdown_connections 2024-01-11T08:22:49.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.734+0000 7fcf751ff700 1 -- 172.21.15.27:0/3405372006 wait complete. 2024-01-11T08:22:49.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.735+0000 7fcf751ff700 1 Processor -- start 2024-01-11T08:22:49.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.735+0000 7fcf751ff700 1 -- start start 2024-01-11T08:22:49.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.736+0000 7fcf751ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf700b03e0 0x7fcf701554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:49.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.736+0000 7fcf751ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf700b07b0 0x7fcf700c5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:49.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.736+0000 7fcf751ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf700b2b90 0x7fcf700c5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:49.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.736+0000 7fcf751ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf70169930 con 0x7fcf700b07b0 2024-01-11T08:22:49.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.736+0000 7fcf751ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7016a8f0 con 0x7fcf700b03e0 2024-01-11T08:22:49.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.736+0000 7fcf751ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7016a460 con 0x7fcf700b2b90 2024-01-11T08:22:49.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.737+0000 7fcf6e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf700b07b0 0x7fcf700c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:49.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.737+0000 7fcf6e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf700b07b0 0x7fcf700c5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54394/0 (socket says 172.21.15.27:54394) 2024-01-11T08:22:49.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.737+0000 7fcf6e59c700 1 -- 172.21.15.27:0/12559315 learned_addr learned my addr 172.21.15.27:0/12559315 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.737+0000 7fcf6ed9d700 1 --2- 172.21.15.27:0/12559315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf700b03e0 0x7fcf701554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.737+0000 7fcf6f59e700 1 --2- 172.21.15.27:0/12559315 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf700b2b90 0x7fcf700c5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.737+0000 7fcf6e59c700 1 -- 172.21.15.27:0/12559315 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf700b2b90 msgr2=0x7fcf700c5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.737+0000 7fcf6e59c700 1 --2- 172.21.15.27:0/12559315 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf700b2b90 0x7fcf700c5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.737+0000 7fcf6e59c700 1 -- 172.21.15.27:0/12559315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf700b03e0 msgr2=0x7fcf701554e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.737+0000 7fcf6e59c700 1 --2- 172.21.15.27:0/12559315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf700b03e0 0x7fcf701554e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.737+0000 7fcf6e59c700 1 -- 172.21.15.27:0/12559315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf68009a20 con 0x7fcf700b07b0 2024-01-11T08:22:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.737+0000 7fcf6e59c700 1 --2- 172.21.15.27:0/12559315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf700b07b0 0x7fcf700c5890 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fcf64009ed0 tx=0x7fcf64017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.738+0000 7fcf56ffd700 1 -- 172.21.15.27:0/12559315 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf64000d10 con 0x7fcf700b07b0 2024-01-11T08:22:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.738+0000 7fcf56ffd700 1 -- 172.21.15.27:0/12559315 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf64011b10 con 0x7fcf700b07b0 2024-01-11T08:22:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.738+0000 7fcf56ffd700 1 -- 172.21.15.27:0/12559315 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf64025640 con 0x7fcf700b07b0 2024-01-11T08:22:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.738+0000 7fcf751ff700 1 -- 172.21.15.27:0/12559315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf64009420 con 0x7fcf700b07b0 2024-01-11T08:22:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.738+0000 7fcf751ff700 1 -- 172.21.15.27:0/12559315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf70169b30 con 0x7fcf700b07b0 2024-01-11T08:22:49.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.739+0000 7fcf751ff700 1 -- 172.21.15.27:0/12559315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf700718d0 con 0x7fcf700b07b0 2024-01-11T08:22:49.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.740+0000 7fcf56ffd700 1 -- 172.21.15.27:0/12559315 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7fcf64025a30 con 0x7fcf700b07b0 2024-01-11T08:22:49.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.741+0000 7fcf56ffd700 1 --2- 172.21.15.27:0/12559315 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcf5c0671b0 0x7fcf5c069660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:49.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.741+0000 7fcf56ffd700 1 -- 172.21.15.27:0/12559315 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6355+0+0 (secure 0 0 0) 0x7fcf64086200 con 0x7fcf700b07b0 2024-01-11T08:22:49.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.741+0000 7fcf6ed9d700 1 -- 172.21.15.27:0/12559315 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcf5c0671b0 msgr2=0x7fcf5c069660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:49.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.741+0000 7fcf6ed9d700 1 --2- 172.21.15.27:0/12559315 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcf5c0671b0 0x7fcf5c069660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:22:49.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.746+0000 7fcf56ffd700 1 -- 172.21.15.27:0/12559315 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcf64053000 con 0x7fcf700b07b0 2024-01-11T08:22:49.838 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:49.839+0000 7f4a678c91c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:22:49.850 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:49.852+0000 7f909e5aa1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:22:49.940 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:49.941+0000 7f909e5aa1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:22:49.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.942+0000 7fcf6ed9d700 1 -- 172.21.15.27:0/12559315 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcf5c0671b0 msgr2=0x7fcf5c069660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:49.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:49.942+0000 7fcf6ed9d700 1 --2- 172.21.15.27:0/12559315 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcf5c0671b0 0x7fcf5c069660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:22:50.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.001+0000 7fcf751ff700 1 -- 172.21.15.27:0/12559315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf700cc860 con 0x7fcf700b07b0 2024-01-11T08:22:50.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.002+0000 7fcf56ffd700 1 -- 172.21.15.27:0/12559315 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4591 (secure 0 0 0) 0x7fcf64056620 con 0x7fcf700b07b0 2024-01-11T08:22:50.001 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:22:50.002 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/2930156910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2930156910},{"type":"v1","addr":"172.21.15.77:6836","nonce":2930156910}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/685019444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":685019444},{"type":"v1","addr":"172.21.15.27:6834","nonce":685019444}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4861,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/144853545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":144853545},{"type":"v1","addr":"172.21.15.77:6837","nonce":144853545}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:11:24.435181+0000","modified":"2024-01-11T08:11:27.940837+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/4056690675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4056690675},{"type":"v1","addr":"172.21.15.27:6835","nonce":4056690675}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2024-01-11T08:22:50.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.004+0000 7fcf751ff700 1 -- 172.21.15.27:0/12559315 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcf5c0671b0 msgr2=0x7fcf5c069660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:22:50.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.004+0000 7fcf751ff700 1 --2- 172.21.15.27:0/12559315 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcf5c0671b0 0x7fcf5c069660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.004+0000 7fcf751ff700 1 -- 172.21.15.27:0/12559315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf700b07b0 msgr2=0x7fcf700c5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:50.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.004+0000 7fcf751ff700 1 --2- 172.21.15.27:0/12559315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf700b07b0 0x7fcf700c5890 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fcf64009ed0 tx=0x7fcf64017000 comp rx=0 tx=0).stop 2024-01-11T08:22:50.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.005+0000 7fcf751ff700 1 -- 172.21.15.27:0/12559315 shutdown_connections 2024-01-11T08:22:50.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.005+0000 7fcf751ff700 1 --2- 172.21.15.27:0/12559315 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcf5c0671b0 0x7fcf5c069660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.005+0000 7fcf751ff700 1 --2- 172.21.15.27:0/12559315 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf700b2b90 0x7fcf700c5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.005+0000 7fcf751ff700 1 --2- 172.21.15.27:0/12559315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf700b03e0 0x7fcf701554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.005+0000 7fcf751ff700 1 --2- 172.21.15.27:0/12559315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf700b07b0 0x7fcf700c5890 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.005+0000 7fcf751ff700 1 -- 172.21.15.27:0/12559315 >> 172.21.15.27:0/12559315 conn(0x7fcf7006ed40 msgr2=0x7fcf700b3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:50.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.005+0000 7fcf751ff700 1 -- 172.21.15.27:0/12559315 shutdown_connections 2024-01-11T08:22:50.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.005+0000 7fcf751ff700 1 -- 172.21.15.27:0/12559315 wait complete. 2024-01-11T08:22:50.005 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-01-11T08:22:50.024 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:22:50.204 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:50.205+0000 7f4a678c91c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:22:50.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.228+0000 7fd2099ff700 1 Processor -- start 2024-01-11T08:22:50.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.228+0000 7fd2099ff700 1 -- start start 2024-01-11T08:22:50.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.229+0000 7fd2099ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2041596c0 0x7fd20415d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:50.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.229+0000 7fd2099ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd2040ca670 0x7fd204068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:50.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.229+0000 7fd2099ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2040a9cb0 con 0x7fd204068b60 2024-01-11T08:22:50.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.230+0000 7fd2099ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd204068fe0 con 0x7fd2040ca670 2024-01-11T08:22:50.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.230+0000 7fd2099ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd20415e210 con 0x7fd2041596c0 2024-01-11T08:22:50.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.230+0000 7fd2027fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd2040ca670 0x7fd204068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:50.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.230+0000 7fd2027fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd2040ca670 0x7fd204068620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54398/0 (socket says 172.21.15.27:54398) 2024-01-11T08:22:50.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.230+0000 7fd2027fc700 1 -- 172.21.15.27:0/2078293512 learned_addr learned my addr 172.21.15.27:0/2078293512 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.230+0000 7fd202ffd700 1 --2- 172.21.15.27:0/2078293512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2041596c0 0x7fd20415d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.230+0000 7fd2027fc700 1 -- 172.21.15.27:0/2078293512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2041596c0 msgr2=0x7fd20415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.230+0000 7fd2027fc700 1 --2- 172.21.15.27:0/2078293512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2041596c0 0x7fd20415d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.230+0000 7fd2027fc700 1 -- 172.21.15.27:0/2078293512 >> v1:172.21.15.77:6789/0 conn(0x7fd204068b60 legacy=0x7fd204159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.230+0000 7fd2027fc700 1 -- 172.21.15.27:0/2078293512 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd20415dc50 con 0x7fd2040ca670 2024-01-11T08:22:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.230+0000 7fd201ffb700 1 -- 172.21.15.27:0/2078293512 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540445930 0 0) 0x7fd2040a9cb0 con 0x7fd204068b60 2024-01-11T08:22:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.231+0000 7fd2027fc700 1 --2- 172.21.15.27:0/2078293512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2040ca670 0x7fd204068620 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd1f80099b0 tx=0x7fd1f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d94b22404254095 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.231+0000 7fd201ffb700 1 -- 172.21.15.27:0/2078293512 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f8009ea0 con 0x7fd2040ca670 2024-01-11T08:22:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.231+0000 7fd201ffb700 1 -- 172.21.15.27:0/2078293512 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f800b860 con 0x7fd2040ca670 2024-01-11T08:22:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.231+0000 7fd201ffb700 1 -- 172.21.15.27:0/2078293512 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f800bc00 con 0x7fd2040ca670 2024-01-11T08:22:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.231+0000 7fd2099ff700 1 -- 172.21.15.27:0/2078293512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2040ca670 msgr2=0x7fd204068620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.231+0000 7fd2099ff700 1 --2- 172.21.15.27:0/2078293512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2040ca670 0x7fd204068620 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd1f80099b0 tx=0x7fd1f8009250 comp rx=0 tx=0).stop 2024-01-11T08:22:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.232+0000 7fd2099ff700 1 -- 172.21.15.27:0/2078293512 shutdown_connections 2024-01-11T08:22:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.232+0000 7fd2099ff700 1 --2- 172.21.15.27:0/2078293512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2041596c0 0x7fd20415d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.232+0000 7fd2099ff700 1 --2- 172.21.15.27:0/2078293512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2040ca670 0x7fd204068620 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.232+0000 7fd2099ff700 1 -- 172.21.15.27:0/2078293512 >> 172.21.15.27:0/2078293512 conn(0x7fd2040c9f70 msgr2=0x7fd20406deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:50.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.232+0000 7fd2099ff700 1 -- 172.21.15.27:0/2078293512 shutdown_connections 2024-01-11T08:22:50.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.232+0000 7fd2099ff700 1 -- 172.21.15.27:0/2078293512 wait complete. 2024-01-11T08:22:50.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.234+0000 7fd2099ff700 1 Processor -- start 2024-01-11T08:22:50.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.234+0000 7fd2099ff700 1 -- start start 2024-01-11T08:22:50.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.235+0000 7fd2099ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd204068b60 0x7fd2041485b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:50.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.235+0000 7fd2099ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2040ca670 0x7fd20415b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:50.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.235+0000 7fd2099ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2041596c0 0x7fd20415bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:50.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.235+0000 7fd2099ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd204148af0 con 0x7fd2040ca670 2024-01-11T08:22:50.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.235+0000 7fd2099ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20406bfb0 con 0x7fd2041596c0 2024-01-11T08:22:50.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.235+0000 7fd2099ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2040c7730 con 0x7fd204068b60 2024-01-11T08:22:50.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.236+0000 7fd202ffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd204068b60 0x7fd2041485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:50.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.236+0000 7fd2027fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2040ca670 0x7fd20415b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:50.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.236+0000 7fd202ffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd204068b60 0x7fd2041485b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:39044/0 (socket says 172.21.15.27:39044) 2024-01-11T08:22:50.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.236+0000 7fd2027fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2040ca670 0x7fd20415b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54400/0 (socket says 172.21.15.27:54400) 2024-01-11T08:22:50.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.236+0000 7fd202ffd700 1 -- 172.21.15.27:0/514100105 learned_addr learned my addr 172.21.15.27:0/514100105 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.236+0000 7fd2037fe700 1 --2- 172.21.15.27:0/514100105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2041596c0 0x7fd20415bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.236+0000 7fd2027fc700 1 -- 172.21.15.27:0/514100105 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd204068b60 msgr2=0x7fd2041485b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.236+0000 7fd2027fc700 1 --2- 172.21.15.27:0/514100105 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd204068b60 0x7fd2041485b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.236+0000 7fd2027fc700 1 -- 172.21.15.27:0/514100105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2041596c0 msgr2=0x7fd20415bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.236+0000 7fd2027fc700 1 --2- 172.21.15.27:0/514100105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2041596c0 0x7fd20415bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.236+0000 7fd2027fc700 1 -- 172.21.15.27:0/514100105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f80094f0 con 0x7fd2040ca670 2024-01-11T08:22:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.236+0000 7fd2027fc700 1 --2- 172.21.15.27:0/514100105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2040ca670 0x7fd20415b790 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd20406ca60 tx=0x7fd1f8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.237+0000 7fd1eaffd700 1 -- 172.21.15.27:0/514100105 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f800bbe0 con 0x7fd2040ca670 2024-01-11T08:22:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.237+0000 7fd1eaffd700 1 -- 172.21.15.27:0/514100105 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f8011420 con 0x7fd2040ca670 2024-01-11T08:22:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.237+0000 7fd1eaffd700 1 -- 172.21.15.27:0/514100105 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f80117c0 con 0x7fd2040ca670 2024-01-11T08:22:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.237+0000 7fd2099ff700 1 -- 172.21.15.27:0/514100105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd20406bcd0 con 0x7fd2040ca670 2024-01-11T08:22:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.237+0000 7fd2099ff700 1 -- 172.21.15.27:0/514100105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2040c73c0 con 0x7fd2040ca670 2024-01-11T08:22:50.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.238+0000 7fd2099ff700 1 -- 172.21.15.27:0/514100105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2040c78a0 con 0x7fd2040ca670 2024-01-11T08:22:50.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.239+0000 7fd1eaffd700 1 -- 172.21.15.27:0/514100105 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7fd1f8011bf0 con 0x7fd2040ca670 2024-01-11T08:22:50.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.240+0000 7fd1eaffd700 1 --2- 172.21.15.27:0/514100105 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd1f00670e0 0x7fd1f0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:50.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.240+0000 7fd1eaffd700 1 -- 172.21.15.27:0/514100105 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6355+0+0 (secure 0 0 0) 0x7fd1f8086380 con 0x7fd2040ca670 2024-01-11T08:22:50.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.240+0000 7fd202ffd700 1 -- 172.21.15.27:0/514100105 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd1f00670e0 msgr2=0x7fd1f0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:50.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.240+0000 7fd202ffd700 1 --2- 172.21.15.27:0/514100105 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd1f00670e0 0x7fd1f0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:22:50.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.244+0000 7fd1eaffd700 1 -- 172.21.15.27:0/514100105 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd1f80530c0 con 0x7fd2040ca670 2024-01-11T08:22:50.302 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:50.303+0000 7f4a678c91c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:22:50.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.441+0000 7fd202ffd700 1 -- 172.21.15.27:0/514100105 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd1f00670e0 msgr2=0x7fd1f0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.441+0000 7fd202ffd700 1 --2- 172.21.15.27:0/514100105 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd1f00670e0 0x7fd1f0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:22:50.499 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:50.486+0000 7f909e5aa1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:22:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.488+0000 7fd2099ff700 1 -- 172.21.15.27:0/514100105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2040718d0 con 0x7fd2040ca670 2024-01-11T08:22:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.489+0000 7fd1eaffd700 1 -- 172.21.15.27:0/514100105 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4591 (secure 0 0 0) 0x7fd1f80566e0 con 0x7fd2040ca670 2024-01-11T08:22:50.499 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:22:50.499 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/2930156910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2930156910},{"type":"v1","addr":"172.21.15.77:6836","nonce":2930156910}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/685019444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":685019444},{"type":"v1","addr":"172.21.15.27:6834","nonce":685019444}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4861,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/144853545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":144853545},{"type":"v1","addr":"172.21.15.77:6837","nonce":144853545}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:11:24.435181+0000","modified":"2024-01-11T08:11:27.940837+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6835/4056690675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4056690675},{"type":"v1","addr":"172.21.15.27:6835","nonce":4056690675}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2024-01-11T08:22:50.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.491+0000 7fd2099ff700 1 -- 172.21.15.27:0/514100105 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd1f00670e0 msgr2=0x7fd1f0069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:22:50.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.491+0000 7fd2099ff700 1 --2- 172.21.15.27:0/514100105 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd1f00670e0 0x7fd1f0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.491+0000 7fd2099ff700 1 -- 172.21.15.27:0/514100105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2040ca670 msgr2=0x7fd20415b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.491+0000 7fd2099ff700 1 --2- 172.21.15.27:0/514100105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2040ca670 0x7fd20415b790 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd20406ca60 tx=0x7fd1f8017000 comp rx=0 tx=0).stop 2024-01-11T08:22:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.491+0000 7fd2099ff700 1 -- 172.21.15.27:0/514100105 shutdown_connections 2024-01-11T08:22:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.491+0000 7fd2099ff700 1 --2- 172.21.15.27:0/514100105 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd1f00670e0 0x7fd1f0069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.491+0000 7fd2099ff700 1 --2- 172.21.15.27:0/514100105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2041596c0 0x7fd20415bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.491+0000 7fd2099ff700 1 --2- 172.21.15.27:0/514100105 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd204068b60 0x7fd2041485b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.491+0000 7fd2099ff700 1 --2- 172.21.15.27:0/514100105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2040ca670 0x7fd20415b790 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.491+0000 7fd2099ff700 1 -- 172.21.15.27:0/514100105 >> 172.21.15.27:0/514100105 conn(0x7fd2040c9f70 msgr2=0x7fd204158560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.491+0000 7fd2099ff700 1 -- 172.21.15.27:0/514100105 shutdown_connections 2024-01-11T08:22:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.491+0000 7fd2099ff700 1 -- 172.21.15.27:0/514100105 wait complete. 2024-01-11T08:22:50.501 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-01-11T08:22:50.508 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:22:50.672 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:50.673+0000 7f909e5aa1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:22:50.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.711+0000 7f2f783ff700 1 Processor -- start 2024-01-11T08:22:50.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.711+0000 7f2f783ff700 1 -- start start 2024-01-11T08:22:50.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.712+0000 7f2f783ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2f701596c0 0x7f2f7015d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:50.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.712+0000 7f2f783ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2f700ca670 0x7f2f70068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:50.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.712+0000 7f2f783ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2f70068b60 0x7f2f70159a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:50.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.712+0000 7f2f783ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f70068fc0 con 0x7f2f700ca670 2024-01-11T08:22:50.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.712+0000 7f2f783ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f700a9cb0 con 0x7f2f701596c0 2024-01-11T08:22:50.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.712+0000 7f2f783ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7015e210 con 0x7f2f70068b60 2024-01-11T08:22:50.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.713+0000 7f2f7599a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2f700ca670 0x7f2f70068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:50.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.713+0000 7f2f7599a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2f700ca670 0x7f2f70068620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54408/0 (socket says 172.21.15.27:54408) 2024-01-11T08:22:50.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.713+0000 7f2f7599a700 1 -- 172.21.15.27:0/3723460564 learned_addr learned my addr 172.21.15.27:0/3723460564 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:50.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.713+0000 7f2f7699c700 1 --2- 172.21.15.27:0/3723460564 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2f70068b60 0x7f2f70159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:50.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.713+0000 7f2f7619b700 1 --2- 172.21.15.27:0/3723460564 >> v2:172.21.15.77:3300/0 conn(0x7f2f701596c0 0x7f2f7015d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:50.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.713+0000 7f2f7599a700 1 -- 172.21.15.27:0/3723460564 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2f70068b60 msgr2=0x7f2f70159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:50.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.713+0000 7f2f7599a700 1 --2- 172.21.15.27:0/3723460564 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2f70068b60 0x7f2f70159a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.713+0000 7f2f7599a700 1 -- 172.21.15.27:0/3723460564 >> v2:172.21.15.77:3300/0 conn(0x7f2f701596c0 msgr2=0x7f2f7015d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:50.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.713+0000 7f2f7599a700 1 --2- 172.21.15.27:0/3723460564 >> v2:172.21.15.77:3300/0 conn(0x7f2f701596c0 0x7f2f7015d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.713+0000 7f2f7599a700 1 -- 172.21.15.27:0/3723460564 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7015dc50 con 0x7f2f700ca670 2024-01-11T08:22:50.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.714+0000 7f2f7599a700 1 --2- 172.21.15.27:0/3723460564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f700ca670 0x7f2f70068620 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2f700c3a20 tx=0x7f2f60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b474f92005565e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:50.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.714+0000 7f2f75199700 1 -- 172.21.15.27:0/3723460564 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6000b640 con 0x7f2f700ca670 2024-01-11T08:22:50.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.714+0000 7f2f75199700 1 -- 172.21.15.27:0/3723460564 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6000b7a0 con 0x7f2f700ca670 2024-01-11T08:22:50.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.714+0000 7f2f75199700 1 -- 172.21.15.27:0/3723460564 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6000bb60 con 0x7f2f700ca670 2024-01-11T08:22:50.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.714+0000 7f2f783ff700 1 -- 172.21.15.27:0/3723460564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f700ca670 msgr2=0x7f2f70068620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:50.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.714+0000 7f2f783ff700 1 --2- 172.21.15.27:0/3723460564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f700ca670 0x7f2f70068620 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2f700c3a20 tx=0x7f2f60009250 comp rx=0 tx=0).stop 2024-01-11T08:22:50.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.715+0000 7f2f783ff700 1 -- 172.21.15.27:0/3723460564 shutdown_connections 2024-01-11T08:22:50.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.715+0000 7f2f783ff700 1 --2- 172.21.15.27:0/3723460564 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2f70068b60 0x7f2f70159a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.715+0000 7f2f783ff700 1 --2- 172.21.15.27:0/3723460564 >> v2:172.21.15.77:3300/0 conn(0x7f2f701596c0 0x7f2f7015d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.715+0000 7f2f783ff700 1 --2- 172.21.15.27:0/3723460564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f700ca670 0x7f2f70068620 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.715+0000 7f2f783ff700 1 -- 172.21.15.27:0/3723460564 >> 172.21.15.27:0/3723460564 conn(0x7f2f700c9f70 msgr2=0x7f2f7006deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:50.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.715+0000 7f2f783ff700 1 -- 172.21.15.27:0/3723460564 shutdown_connections 2024-01-11T08:22:50.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.715+0000 7f2f783ff700 1 -- 172.21.15.27:0/3723460564 wait complete. 2024-01-11T08:22:50.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.716+0000 7f2f783ff700 1 Processor -- start 2024-01-11T08:22:50.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.716+0000 7f2f783ff700 1 -- start start 2024-01-11T08:22:50.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.717+0000 7f2f783ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f70068b60 0x7f2f7006aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:50.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.717+0000 7f2f783ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f700ca670 0x7f2f701555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:50.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.717+0000 7f2f783ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2f701596c0 0x7f2f700c8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:50.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.717+0000 7f2f783ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7006af70 con 0x7f2f70068b60 2024-01-11T08:22:50.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.717+0000 7f2f783ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f700c8cf0 con 0x7f2f700ca670 2024-01-11T08:22:50.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.717+0000 7f2f783ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7016d6e0 con 0x7f2f701596c0 2024-01-11T08:22:50.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.717+0000 7f2f7619b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f70068b60 0x7f2f7006aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:50.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.718+0000 7f2f7599a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f700ca670 0x7f2f701555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:50.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.718+0000 7f2f7699c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2f701596c0 0x7f2f700c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:50.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.718+0000 7f2f7599a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f700ca670 0x7f2f701555a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:57906/0 (socket says 172.21.15.27:57906) 2024-01-11T08:22:50.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.718+0000 7f2f7599a700 1 -- 172.21.15.27:0/2307377280 learned_addr learned my addr 172.21.15.27:0/2307377280 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:50.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.718+0000 7f2f7619b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f70068b60 0x7f2f7006aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54416/0 (socket says 172.21.15.27:54416) 2024-01-11T08:22:50.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.718+0000 7f2f7599a700 1 -- 172.21.15.27:0/2307377280 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2f701596c0 msgr2=0x7f2f700c8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:50.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.718+0000 7f2f7599a700 1 --2- 172.21.15.27:0/2307377280 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2f701596c0 0x7f2f700c8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.718+0000 7f2f7599a700 1 -- 172.21.15.27:0/2307377280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f70068b60 msgr2=0x7f2f7006aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:50.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.718+0000 7f2f7599a700 1 --2- 172.21.15.27:0/2307377280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f70068b60 0x7f2f7006aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.718+0000 7f2f7599a700 1 -- 172.21.15.27:0/2307377280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f600094f0 con 0x7f2f700ca670 2024-01-11T08:22:50.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.718+0000 7f2f7599a700 1 --2- 172.21.15.27:0/2307377280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f700ca670 0x7f2f701555a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2f7006d490 tx=0x7f2f60017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:50.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.719+0000 7f2f667fc700 1 -- 172.21.15.27:0/2307377280 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6000b910 con 0x7f2f700ca670 2024-01-11T08:22:50.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.719+0000 7f2f667fc700 1 -- 172.21.15.27:0/2307377280 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6000ba70 con 0x7f2f700ca670 2024-01-11T08:22:50.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.719+0000 7f2f783ff700 1 -- 172.21.15.27:0/2307377280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f7016d9c0 con 0x7f2f700ca670 2024-01-11T08:22:50.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.719+0000 7f2f667fc700 1 -- 172.21.15.27:0/2307377280 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f60011630 con 0x7f2f700ca670 2024-01-11T08:22:50.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.719+0000 7f2f783ff700 1 -- 172.21.15.27:0/2307377280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f7016df80 con 0x7f2f700ca670 2024-01-11T08:22:50.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.720+0000 7f2f783ff700 1 -- 172.21.15.27:0/2307377280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f700af9c0 con 0x7f2f700ca670 2024-01-11T08:22:50.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.721+0000 7f2f667fc700 1 -- 172.21.15.27:0/2307377280 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7f2f600119a0 con 0x7f2f700ca670 2024-01-11T08:22:50.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.722+0000 7f2f667fc700 1 --2- 172.21.15.27:0/2307377280 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2f540670e0 0x7f2f54069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:50.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.722+0000 7f2f667fc700 1 -- 172.21.15.27:0/2307377280 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f2f60086240 con 0x7f2f700ca670 2024-01-11T08:22:50.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.722+0000 7f2f7619b700 1 -- 172.21.15.27:0/2307377280 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2f540670e0 msgr2=0x7f2f54069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:50.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.722+0000 7f2f7619b700 1 --2- 172.21.15.27:0/2307377280 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2f540670e0 0x7f2f54069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:22:50.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.727+0000 7f2f667fc700 1 -- 172.21.15.27:0/2307377280 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2f700af9c0 con 0x7f2f700ca670 2024-01-11T08:22:50.891 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:50.892+0000 7f4a678c91c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:22:50.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.923+0000 7f2f7619b700 1 -- 172.21.15.27:0/2307377280 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2f540670e0 msgr2=0x7f2f54069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:50.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.923+0000 7f2f7619b700 1 --2- 172.21.15.27:0/2307377280 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2f540670e0 0x7f2f54069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:22:50.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.932+0000 7f2f783ff700 1 -- 172.21.15.27:0/2307377280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f700af9c0 con 0x7f2f700ca670 2024-01-11T08:22:50.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.934+0000 7f2f667fc700 1 -- 172.21.15.27:0/2307377280 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16289 (secure 0 0 0) 0x7f2f700af9c0 con 0x7f2f700ca670 2024-01-11T08:22:50.933 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:22:50.933 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":46,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:22:46.732163+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:11:22.376067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-11T08:11:23.419993+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":46,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6833/41381":"2024-01-12T08:11:40.675932+0000","172.21.15.77:6832/41381":"2024-01-12T08:11:40.675932+0000","172.21.15.77:0/969431132":"2024-01-12T08:11:40.675932+0000","172.21.15.77:0/4222476855":"2024-01-12T08:11:40.675932+0000","172.21.15.77:0/3211563793":"2024-01-12T08:11:40.675932+0000","172.21.15.77:0/2257956400":"2024-01-12T08:11:40.675932+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":5,"length":2},{"begin":10,"length":2},{"begin":15,"length":2},{"begin":20,"length":3},{"begin":26,"length":2},{"begin":31,"length":3},{"begin":37,"length":2},{"begin":43,"length":2},{"begin":48,"length":2},{"begin":54,"length":2},{"begin":59,"length":2},{"begin":65,"length":2},{"begin":70,"length":2},{"begin":75,"length":2},{"begin":80,"length":3},{"begin":86,"length":2},{"begin":91,"length":3},{"begin":97,"length":2},{"begin":102,"length":3},{"begin":108,"length":2},{"begin":177,"length":45}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":104,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:22:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.936+0000 7f2f783ff700 1 -- 172.21.15.27:0/2307377280 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2f540670e0 msgr2=0x7f2f54069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:22:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.936+0000 7f2f783ff700 1 --2- 172.21.15.27:0/2307377280 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2f540670e0 0x7f2f54069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.936+0000 7f2f783ff700 1 -- 172.21.15.27:0/2307377280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f700ca670 msgr2=0x7f2f701555a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.936+0000 7f2f783ff700 1 --2- 172.21.15.27:0/2307377280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f700ca670 0x7f2f701555a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2f7006d490 tx=0x7f2f60017000 comp rx=0 tx=0).stop 2024-01-11T08:22:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.936+0000 7f2f783ff700 1 -- 172.21.15.27:0/2307377280 shutdown_connections 2024-01-11T08:22:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.936+0000 7f2f783ff700 1 --2- 172.21.15.27:0/2307377280 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2f540670e0 0x7f2f54069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.936+0000 7f2f783ff700 1 --2- 172.21.15.27:0/2307377280 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2f701596c0 0x7f2f700c8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.936+0000 7f2f783ff700 1 --2- 172.21.15.27:0/2307377280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f70068b60 0x7f2f7006aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.936+0000 7f2f783ff700 1 --2- 172.21.15.27:0/2307377280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f700ca670 0x7f2f701555a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:50.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.936+0000 7f2f783ff700 1 -- 172.21.15.27:0/2307377280 >> 172.21.15.27:0/2307377280 conn(0x7f2f700c9f70 msgr2=0x7f2f701564d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:50.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.936+0000 7f2f783ff700 1 -- 172.21.15.27:0/2307377280 shutdown_connections 2024-01-11T08:22:50.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:50.936+0000 7f2f783ff700 1 -- 172.21.15.27:0/2307377280 wait complete. 2024-01-11T08:22:50.954 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-11T08:22:51.051 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:51.052+0000 7f909e5aa1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:22:51.093 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:51.094+0000 7f4a678c91c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:22:51.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.160+0000 7f5c889ff700 1 Processor -- start 2024-01-11T08:22:51.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.160+0000 7f5c889ff700 1 -- start start 2024-01-11T08:22:51.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.161+0000 7f5c889ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c840b19a0 0x7f5c8406f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:51.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.161+0000 7f5c889ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c840cb540 con 0x7f5c84157360 2024-01-11T08:22:51.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.161+0000 7f5c889ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c840ca960 con 0x7f5c84157730 2024-01-11T08:22:51.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.161+0000 7f5c889ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c840cac10 con 0x7f5c840b19a0 2024-01-11T08:22:51.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.162+0000 7f5c8259c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5c84157360 0x7f5c840b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34320/0 (socket says 172.21.15.27:34320) 2024-01-11T08:22:51.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.162+0000 7f5c8259c700 1 -- 172.21.15.27:0/599052098 learned_addr learned my addr 172.21.15.27:0/599052098 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:51.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.162+0000 7f5c81d9b700 1 --2- 172.21.15.27:0/599052098 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c840b19a0 0x7f5c8406f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:51.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.162+0000 7f5c8159a700 1 -- 172.21.15.27:0/599052098 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380761615 0 0) 0x7f5c840cb540 con 0x7f5c84157360 2024-01-11T08:22:51.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.162+0000 7f5c8159a700 1 -- 172.21.15.27:0/599052098 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c6c003570 con 0x7f5c84157360 2024-01-11T08:22:51.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.162+0000 7f5c8159a700 1 -- 172.21.15.27:0/599052098 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741880155 0 0) 0x7f5c840ca960 con 0x7f5c84157730 2024-01-11T08:22:51.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.162+0000 7f5c8159a700 1 -- 172.21.15.27:0/599052098 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c840cb540 con 0x7f5c84157730 2024-01-11T08:22:51.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.162+0000 7f5c81d9b700 1 -- 172.21.15.27:0/599052098 >> v1:172.21.15.77:6789/0 conn(0x7f5c84157730 legacy=0x7f5c840abb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:51.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.162+0000 7f5c81d9b700 1 -- 172.21.15.27:0/599052098 >> v1:172.21.15.27:6789/0 conn(0x7f5c84157360 legacy=0x7f5c840b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:51.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.162+0000 7f5c81d9b700 1 -- 172.21.15.27:0/599052098 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c840caef0 con 0x7f5c840b19a0 2024-01-11T08:22:51.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.163+0000 7f5c81d9b700 1 --2- 172.21.15.27:0/599052098 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c840b19a0 0x7f5c8406f570 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009910 tx=0x7f5c780091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2ac76bac69e7576 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:51.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.163+0000 7f5c8159a700 1 -- 172.21.15.27:0/599052098 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c78011630 con 0x7f5c840b19a0 2024-01-11T08:22:51.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.163+0000 7f5c8159a700 1 -- 172.21.15.27:0/599052098 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c78011790 con 0x7f5c840b19a0 2024-01-11T08:22:51.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.163+0000 7f5c8159a700 1 -- 172.21.15.27:0/599052098 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c78011b30 con 0x7f5c840b19a0 2024-01-11T08:22:51.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.164+0000 7f5c889ff700 1 -- 172.21.15.27:0/599052098 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c840b19a0 msgr2=0x7f5c8406f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:51.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.164+0000 7f5c889ff700 1 --2- 172.21.15.27:0/599052098 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c840b19a0 0x7f5c8406f570 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009910 tx=0x7f5c780091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:51.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.164+0000 7f5c889ff700 1 -- 172.21.15.27:0/599052098 shutdown_connections 2024-01-11T08:22:51.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.164+0000 7f5c889ff700 1 --2- 172.21.15.27:0/599052098 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c840b19a0 0x7f5c8406f570 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:51.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.164+0000 7f5c889ff700 1 -- 172.21.15.27:0/599052098 >> 172.21.15.27:0/599052098 conn(0x7f5c840680b0 msgr2=0x7f5c840cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:51.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.164+0000 7f5c889ff700 1 -- 172.21.15.27:0/599052098 shutdown_connections 2024-01-11T08:22:51.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.164+0000 7f5c889ff700 1 -- 172.21.15.27:0/599052098 wait complete. 2024-01-11T08:22:51.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.165+0000 7f5c889ff700 1 Processor -- start 2024-01-11T08:22:51.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.165+0000 7f5c889ff700 1 -- start start 2024-01-11T08:22:51.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.166+0000 7f5c889ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c840b19a0 0x7f5c84151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:51.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.166+0000 7f5c889ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c84157360 0x7f5c840c5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:51.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.166+0000 7f5c889ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c84157730 0x7f5c840af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:51.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.166+0000 7f5c889ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c840c5860 con 0x7f5c84157360 2024-01-11T08:22:51.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.166+0000 7f5c889ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c840afb40 con 0x7f5c84157730 2024-01-11T08:22:51.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.166+0000 7f5c889ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c840afe20 con 0x7f5c840b19a0 2024-01-11T08:22:51.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.167+0000 7f5c81d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c84157360 0x7f5c840c5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:51.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.167+0000 7f5c8259c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c840b19a0 0x7f5c84151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:51.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.167+0000 7f5c81d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c84157360 0x7f5c840c5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54422/0 (socket says 172.21.15.27:54422) 2024-01-11T08:22:51.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.167+0000 7f5c81d9b700 1 -- 172.21.15.27:0/1547160441 learned_addr learned my addr 172.21.15.27:0/1547160441 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:51.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.167+0000 7f5c82d9d700 1 --2- 172.21.15.27:0/1547160441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c84157730 0x7f5c840af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:51.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.167+0000 7f5c81d9b700 1 -- 172.21.15.27:0/1547160441 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c840b19a0 msgr2=0x7f5c84151f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:51.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.167+0000 7f5c81d9b700 1 --2- 172.21.15.27:0/1547160441 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c840b19a0 0x7f5c84151f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:51.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.167+0000 7f5c81d9b700 1 -- 172.21.15.27:0/1547160441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c84157730 msgr2=0x7f5c840af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:51.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.167+0000 7f5c81d9b700 1 --2- 172.21.15.27:0/1547160441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c84157730 0x7f5c840af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:51.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.167+0000 7f5c81d9b700 1 -- 172.21.15.27:0/1547160441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c78009420 con 0x7f5c84157360 2024-01-11T08:22:51.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.167+0000 7f5c8259c700 1 --2- 172.21.15.27:0/1547160441 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c840b19a0 0x7f5c84151f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:22:51.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.167+0000 7f5c81d9b700 1 --2- 172.21.15.27:0/1547160441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c84157360 0x7f5c840c5cf0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009f00 tx=0x7f5c78017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:51.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.168+0000 7f5c727fc700 1 -- 172.21.15.27:0/1547160441 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7801f070 con 0x7f5c84157360 2024-01-11T08:22:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.168+0000 7f5c727fc700 1 -- 172.21.15.27:0/1547160441 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c78002f70 con 0x7f5c84157360 2024-01-11T08:22:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.168+0000 7f5c727fc700 1 -- 172.21.15.27:0/1547160441 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c78023410 con 0x7f5c84157360 2024-01-11T08:22:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.168+0000 7f5c889ff700 1 -- 172.21.15.27:0/1547160441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c840c54f0 con 0x7f5c84157360 2024-01-11T08:22:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.168+0000 7f5c889ff700 1 -- 172.21.15.27:0/1547160441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c841524c0 con 0x7f5c84157360 2024-01-11T08:22:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.169+0000 7f5c889ff700 1 -- 172.21.15.27:0/1547160441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c8404c3c0 con 0x7f5c84157360 2024-01-11T08:22:51.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.170+0000 7f5c727fc700 1 -- 172.21.15.27:0/1547160441 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7f5c78023840 con 0x7f5c84157360 2024-01-11T08:22:51.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.171+0000 7f5c727fc700 1 --2- 172.21.15.27:0/1547160441 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5c640670e0 0x7f5c64069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:51.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.171+0000 7f5c727fc700 1 -- 172.21.15.27:0/1547160441 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6355+0+0 (secure 0 0 0) 0x7f5c78085b30 con 0x7f5c84157360 2024-01-11T08:22:51.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.172+0000 7f5c8259c700 1 -- 172.21.15.27:0/1547160441 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5c640670e0 msgr2=0x7f5c64069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:51.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.172+0000 7f5c8259c700 1 --2- 172.21.15.27:0/1547160441 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5c640670e0 0x7f5c64069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:22:51.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.175+0000 7f5c727fc700 1 -- 172.21.15.27:0/1547160441 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5c78052890 con 0x7f5c84157360 2024-01-11T08:22:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.373+0000 7f5c8259c700 1 -- 172.21.15.27:0/1547160441 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5c640670e0 msgr2=0x7f5c64069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.373+0000 7f5c8259c700 1 --2- 172.21.15.27:0/1547160441 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5c640670e0 0x7f5c64069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:22:51.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.427+0000 7f5c889ff700 1 -- 172.21.15.27:0/1547160441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5c840c59a0 con 0x7f5c84157360 2024-01-11T08:22:51.427 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-11T08:22:51.428+0000 7f27065f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-11T08:22:51.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.432+0000 7f5c727fc700 1 -- 172.21.15.27:0/1547160441 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5c78015020 con 0x7f5c84157360 2024-01-11T08:22:51.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.435+0000 7f5c889ff700 1 -- 172.21.15.27:0/1547160441 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5c640670e0 msgr2=0x7f5c64069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:22:51.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.435+0000 7f5c889ff700 1 --2- 172.21.15.27:0/1547160441 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5c640670e0 0x7f5c64069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:51.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.435+0000 7f5c889ff700 1 -- 172.21.15.27:0/1547160441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c84157360 msgr2=0x7f5c840c5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:51.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.435+0000 7f5c889ff700 1 --2- 172.21.15.27:0/1547160441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c84157360 0x7f5c840c5cf0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009f00 tx=0x7f5c78017000 comp rx=0 tx=0).stop 2024-01-11T08:22:51.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.436+0000 7f5c889ff700 1 -- 172.21.15.27:0/1547160441 shutdown_connections 2024-01-11T08:22:51.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.436+0000 7f5c889ff700 1 --2- 172.21.15.27:0/1547160441 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5c640670e0 0x7f5c64069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:51.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.436+0000 7f5c889ff700 1 --2- 172.21.15.27:0/1547160441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c84157730 0x7f5c840af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:51.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.436+0000 7f5c889ff700 1 --2- 172.21.15.27:0/1547160441 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c840b19a0 0x7f5c84151f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:51.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.436+0000 7f5c889ff700 1 --2- 172.21.15.27:0/1547160441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c84157360 0x7f5c840c5cf0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:51.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.436+0000 7f5c889ff700 1 -- 172.21.15.27:0/1547160441 >> 172.21.15.27:0/1547160441 conn(0x7f5c840680b0 msgr2=0x7f5c841565e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:51.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.436+0000 7f5c889ff700 1 -- 172.21.15.27:0/1547160441 shutdown_connections 2024-01-11T08:22:51.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.436+0000 7f5c889ff700 1 -- 172.21.15.27:0/1547160441 wait complete. 2024-01-11T08:22:51.436 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-11T08:22:51.453 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-11T08:22:51.471 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:51.472+0000 7f909e5aa1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:22:51.516 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T08:22:51.517 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:51.517+0000 7f4a678c91c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:22:51.651 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:22:51.652 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:51.653+0000 7f909e5aa1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:22:51.658 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:22:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.666+0000 7f7caf9ff700 1 Processor -- start 2024-01-11T08:22:51.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.666+0000 7f7caf9ff700 1 -- start start 2024-01-11T08:22:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.668+0000 7f7caf9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7ca8142dc0 0x7f7ca80cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.668+0000 7f7caf9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7ca8143190 0x7f7ca80cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:51.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.668+0000 7f7caf9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ca815d320 con 0x7f7ca80cc020 2024-01-11T08:22:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.668+0000 7f7caf9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca815df30 con 0x7f7ca8142dc0 2024-01-11T08:22:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.668+0000 7f7caf9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca815d590 con 0x7f7ca8143190 2024-01-11T08:22:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.668+0000 7f7cad79b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7ca8142dc0 0x7f7ca80cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.668+0000 7f7cadf9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7ca80cc020 0x7f7ca80cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34342/0 (socket says 172.21.15.27:34342) 2024-01-11T08:22:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.668+0000 7f7cad79b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7ca8142dc0 0x7f7ca80cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54456/0 (socket says 172.21.15.27:54456) 2024-01-11T08:22:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.668+0000 7f7cacf9a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7ca8143190 0x7f7ca80cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:51.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.668+0000 7f7cadf9c700 1 -- 172.21.15.27:0/2768035423 learned_addr learned my addr 172.21.15.27:0/2768035423 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.669+0000 7f7cacf9a700 1 -- 172.21.15.27:0/2768035423 >> v2:172.21.15.27:3300/0 conn(0x7f7ca8142dc0 msgr2=0x7f7ca80cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.669+0000 7f7cacf9a700 1 --2- 172.21.15.27:0/2768035423 >> v2:172.21.15.27:3300/0 conn(0x7f7ca8142dc0 0x7f7ca80cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.669+0000 7f7cacf9a700 1 -- 172.21.15.27:0/2768035423 >> v1:172.21.15.27:6789/0 conn(0x7f7ca80cc020 legacy=0x7f7ca80cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.669+0000 7f7cacf9a700 1 -- 172.21.15.27:0/2768035423 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca815d870 con 0x7f7ca8143190 2024-01-11T08:22:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.669+0000 7f7c9ffff700 1 -- 172.21.15.27:0/2768035423 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554467527 0 0) 0x7f7ca815d320 con 0x7f7ca80cc020 2024-01-11T08:22:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.669+0000 7f7cacf9a700 1 --2- 172.21.15.27:0/2768035423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ca8143190 0x7f7ca80cb2e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7ca806c480 tx=0x7f7c980091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3554c324ca5d11b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.669+0000 7f7c9ffff700 1 -- 172.21.15.27:0/2768035423 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98011630 con 0x7f7ca8143190 2024-01-11T08:22:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.669+0000 7f7c9ffff700 1 -- 172.21.15.27:0/2768035423 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c98011790 con 0x7f7ca8143190 2024-01-11T08:22:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.669+0000 7f7c9ffff700 1 -- 172.21.15.27:0/2768035423 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98011b30 con 0x7f7ca8143190 2024-01-11T08:22:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.670+0000 7f7caf9ff700 1 -- 172.21.15.27:0/2768035423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ca8143190 msgr2=0x7f7ca80cb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.670+0000 7f7caf9ff700 1 --2- 172.21.15.27:0/2768035423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ca8143190 0x7f7ca80cb2e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7ca806c480 tx=0x7f7c980091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.670+0000 7f7caf9ff700 1 -- 172.21.15.27:0/2768035423 shutdown_connections 2024-01-11T08:22:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.670+0000 7f7caf9ff700 1 --2- 172.21.15.27:0/2768035423 >> v2:172.21.15.27:3300/0 conn(0x7f7ca8142dc0 0x7f7ca80cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.670+0000 7f7caf9ff700 1 --2- 172.21.15.27:0/2768035423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ca8143190 0x7f7ca80cb2e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.670+0000 7f7caf9ff700 1 -- 172.21.15.27:0/2768035423 >> 172.21.15.27:0/2768035423 conn(0x7f7ca8063ca0 msgr2=0x7f7ca806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.670+0000 7f7caf9ff700 1 -- 172.21.15.27:0/2768035423 shutdown_connections 2024-01-11T08:22:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.670+0000 7f7caf9ff700 1 -- 172.21.15.27:0/2768035423 wait complete. 2024-01-11T08:22:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.672+0000 7f7caf9ff700 1 Processor -- start 2024-01-11T08:22:51.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.672+0000 7f7caf9ff700 1 -- start start 2024-01-11T08:22:51.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.673+0000 7f7caf9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ca80cc020 0x7f7ca8153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:51.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.673+0000 7f7caf9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ca8142dc0 0x7f7ca8152f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:51.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.673+0000 7f7caf9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ca8143190 0x7f7ca80b2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:51.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.673+0000 7f7caf9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca8142140 con 0x7f7ca8142dc0 2024-01-11T08:22:51.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.673+0000 7f7caf9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca816e160 con 0x7f7ca8143190 2024-01-11T08:22:51.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.673+0000 7f7caf9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca8169d30 con 0x7f7ca80cc020 2024-01-11T08:22:51.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.673+0000 7f7cacf9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ca8142dc0 0x7f7ca8152f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:51.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.673+0000 7f7cad79b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ca80cc020 0x7f7ca8153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:51.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.673+0000 7f7cadf9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ca8143190 0x7f7ca80b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:51.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.673+0000 7f7cacf9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ca8142dc0 0x7f7ca8152f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54466/0 (socket says 172.21.15.27:54466) 2024-01-11T08:22:51.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.673+0000 7f7cacf9a700 1 -- 172.21.15.27:0/2334587180 learned_addr learned my addr 172.21.15.27:0/2334587180 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.674+0000 7f7cacf9a700 1 -- 172.21.15.27:0/2334587180 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ca80cc020 msgr2=0x7f7ca8153a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.674+0000 7f7cacf9a700 1 --2- 172.21.15.27:0/2334587180 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ca80cc020 0x7f7ca8153a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.674+0000 7f7cacf9a700 1 -- 172.21.15.27:0/2334587180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ca8143190 msgr2=0x7f7ca80b2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:51.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.674+0000 7f7cacf9a700 1 --2- 172.21.15.27:0/2334587180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ca8143190 0x7f7ca80b2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:51.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.674+0000 7f7cacf9a700 1 -- 172.21.15.27:0/2334587180 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c98009420 con 0x7f7ca8142dc0 2024-01-11T08:22:51.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.674+0000 7f7cadf9c700 1 --2- 172.21.15.27:0/2334587180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ca8143190 0x7f7ca80b2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:22:51.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.674+0000 7f7cad79b700 1 --2- 172.21.15.27:0/2334587180 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ca80cc020 0x7f7ca8153a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:22:51.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.674+0000 7f7cacf9a700 1 --2- 172.21.15.27:0/2334587180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ca8142dc0 0x7f7ca8152f50 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7c98009f50 tx=0x7f7c98017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:51.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.675+0000 7f7c9d7fa700 1 -- 172.21.15.27:0/2334587180 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98000d10 con 0x7f7ca8142dc0 2024-01-11T08:22:51.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.675+0000 7f7c9d7fa700 1 -- 172.21.15.27:0/2334587180 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c98011b10 con 0x7f7ca8142dc0 2024-01-11T08:22:51.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.675+0000 7f7caf9ff700 1 -- 172.21.15.27:0/2334587180 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca8156380 con 0x7f7ca8142dc0 2024-01-11T08:22:51.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.675+0000 7f7c9d7fa700 1 -- 172.21.15.27:0/2334587180 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98025640 con 0x7f7ca8142dc0 2024-01-11T08:22:51.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.675+0000 7f7caf9ff700 1 -- 172.21.15.27:0/2334587180 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca8156940 con 0x7f7ca8142dc0 2024-01-11T08:22:51.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.676+0000 7f7caf9ff700 1 -- 172.21.15.27:0/2334587180 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca8169ec0 con 0x7f7ca8142dc0 2024-01-11T08:22:51.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.678+0000 7f7c9d7fa700 1 -- 172.21.15.27:0/2334587180 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 84913+0+0 (secure 0 0 0) 0x7f7c98025a30 con 0x7f7ca8142dc0 2024-01-11T08:22:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.679+0000 7f7c9d7fa700 1 --2- 172.21.15.27:0/2334587180 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7c8c0670e0 0x7f7c8c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.679+0000 7f7c9d7fa700 1 -- 172.21.15.27:0/2334587180 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f7c98086310 con 0x7f7ca8142dc0 2024-01-11T08:22:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.679+0000 7f7cad79b700 1 -- 172.21.15.27:0/2334587180 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7c8c0670e0 msgr2=0x7f7c8c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.679+0000 7f7cad79b700 1 --2- 172.21.15.27:0/2334587180 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7c8c0670e0 0x7f7c8c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:22:51.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.683+0000 7f7c9d7fa700 1 -- 172.21.15.27:0/2334587180 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c98052fb0 con 0x7f7ca8142dc0 2024-01-11T08:22:51.849 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:51.850+0000 7f909e5aa1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:22:51.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.880+0000 7f7cad79b700 1 -- 172.21.15.27:0/2334587180 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7c8c0670e0 msgr2=0x7f7c8c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:22:51.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.880+0000 7f7cad79b700 1 --2- 172.21.15.27:0/2334587180 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7c8c0670e0 0x7f7c8c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:22:51.957 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:51.958+0000 7f909e5aa1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:22:51.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.969+0000 7f7c9d7fa700 1 -- 172.21.15.27:0/2334587180 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 15) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f7c98053380 con 0x7f7ca8142dc0 2024-01-11T08:22:51.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.970+0000 7f7c9d7fa700 1 -- 172.21.15.27:0/2334587180 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7c8c0670e0 msgr2=0x7f7c8c069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:22:51.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.970+0000 7f7c9d7fa700 1 --2- 172.21.15.27:0/2334587180 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7c8c0670e0 0x7f7c8c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:51.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:51.981+0000 7f7caf9ff700 1 -- 172.21.15.27:0/2334587180 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7ca804c3c0 con 0x7f7ca8142dc0 2024-01-11T08:22:52.004 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:52.003+0000 7f4a678c91c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:22:52.038 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:52.039+0000 7f9019444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:22:52.038 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:52.039+0000 7f9019444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:22:52.038 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:52.039+0000 7f9019444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:22:52.038 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:52.039+0000 7f9019444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:22:52.038 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:52.039+0000 7f9019444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:22:52.038 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:52.039+0000 7f901bc49700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:22:52.038 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:52.039+0000 7f901bc49700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:22:52.039 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:52.039+0000 7f901bc49700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:22:52.039 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:52.039+0000 7f901bc49700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:22:52.039 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:22:52.039+0000 7f901bc49700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:22:52.197 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:52.197+0000 7f4a678c91c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:22:52.393 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:52.392+0000 7f4a678c91c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:22:52.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.432+0000 7f7c9d7fa700 1 -- 172.21.15.27:0/2334587180 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7c98026e30 con 0x7f7ca8142dc0 2024-01-11T08:22:52.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.434+0000 7f7caf9ff700 1 -- 172.21.15.27:0/2334587180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ca8142dc0 msgr2=0x7f7ca8152f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:52.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.434+0000 7f7caf9ff700 1 --2- 172.21.15.27:0/2334587180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ca8142dc0 0x7f7ca8152f50 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7c98009f50 tx=0x7f7c98017000 comp rx=0 tx=0).stop 2024-01-11T08:22:52.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.435+0000 7f7caf9ff700 1 -- 172.21.15.27:0/2334587180 shutdown_connections 2024-01-11T08:22:52.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.435+0000 7f7caf9ff700 1 --2- 172.21.15.27:0/2334587180 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7c8c0670e0 0x7f7c8c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:52.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.435+0000 7f7caf9ff700 1 --2- 172.21.15.27:0/2334587180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ca8143190 0x7f7ca80b2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:52.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.435+0000 7f7caf9ff700 1 --2- 172.21.15.27:0/2334587180 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ca80cc020 0x7f7ca8153a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:52.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.435+0000 7f7caf9ff700 1 --2- 172.21.15.27:0/2334587180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ca8142dc0 0x7f7ca8152f50 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:52.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.435+0000 7f7caf9ff700 1 -- 172.21.15.27:0/2334587180 >> 172.21.15.27:0/2334587180 conn(0x7f7ca8063ca0 msgr2=0x7f7ca80c45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:52.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.435+0000 7f7caf9ff700 1 -- 172.21.15.27:0/2334587180 shutdown_connections 2024-01-11T08:22:52.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.435+0000 7f7caf9ff700 1 -- 172.21.15.27:0/2334587180 wait complete. 2024-01-11T08:22:52.454 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-11T08:22:52.515 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:22:52.515+0000 7f4a678c91c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:22:52.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.663+0000 7f87159ff700 1 Processor -- start 2024-01-11T08:22:52.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.663+0000 7f87159ff700 1 -- start start 2024-01-11T08:22:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.664+0000 7f87159ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8710157440 0x7f87100ae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.664+0000 7f87159ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8710157810 0x7f8710069660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.664+0000 7f87159ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f871006a570 0x7f8710069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.664+0000 7f87159ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f871006d420 con 0x7f8710157440 2024-01-11T08:22:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.664+0000 7f87159ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f871006d700 con 0x7f871006a570 2024-01-11T08:22:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.664+0000 7f87159ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871006d9e0 con 0x7f8710157810 2024-01-11T08:22:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.664+0000 7f870effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8710157440 0x7f87100ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.665+0000 7f870effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8710157440 0x7f87100ae430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54468/0 (socket says 172.21.15.27:54468) 2024-01-11T08:22:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.665+0000 7f870e7fc700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8710157810 0x7f8710069660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.665+0000 7f870effd700 1 -- 172.21.15.27:0/3003964767 learned_addr learned my addr 172.21.15.27:0/3003964767 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:52.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.665+0000 7f870f7fe700 1 --2- 172.21.15.27:0/3003964767 >> v2:172.21.15.77:3300/0 conn(0x7f871006a570 0x7f8710069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:52.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.665+0000 7f870e7fc700 1 -- 172.21.15.27:0/3003964767 >> v2:172.21.15.77:3300/0 conn(0x7f871006a570 msgr2=0x7f8710069ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:52.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.665+0000 7f870e7fc700 1 --2- 172.21.15.27:0/3003964767 >> v2:172.21.15.77:3300/0 conn(0x7f871006a570 0x7f8710069ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:52.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.665+0000 7f870e7fc700 1 -- 172.21.15.27:0/3003964767 >> v2:172.21.15.27:3300/0 conn(0x7f8710157440 msgr2=0x7f87100ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:52.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.665+0000 7f870e7fc700 1 --2- 172.21.15.27:0/3003964767 >> v2:172.21.15.27:3300/0 conn(0x7f8710157440 0x7f87100ae430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:52.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.665+0000 7f870e7fc700 1 -- 172.21.15.27:0/3003964767 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871006dcc0 con 0x7f8710157810 2024-01-11T08:22:52.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.665+0000 7f870effd700 1 --2- 172.21.15.27:0/3003964767 >> v2:172.21.15.27:3300/0 conn(0x7f8710157440 0x7f87100ae430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:22:52.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.665+0000 7f870e7fc700 1 --2- 172.21.15.27:0/3003964767 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8710157810 0x7f8710069660 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8710069140 tx=0x7f87040091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adc460c3384a80cc server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:52.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.666+0000 7f870dffb700 1 -- 172.21.15.27:0/3003964767 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704011630 con 0x7f8710157810 2024-01-11T08:22:52.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.666+0000 7f870dffb700 1 -- 172.21.15.27:0/3003964767 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87040117b0 con 0x7f8710157810 2024-01-11T08:22:52.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.666+0000 7f870dffb700 1 -- 172.21.15.27:0/3003964767 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704011b70 con 0x7f8710157810 2024-01-11T08:22:52.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.666+0000 7f87159ff700 1 -- 172.21.15.27:0/3003964767 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8710157810 msgr2=0x7f8710069660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:52.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.666+0000 7f87159ff700 1 --2- 172.21.15.27:0/3003964767 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8710157810 0x7f8710069660 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8710069140 tx=0x7f87040091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:52.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.666+0000 7f87159ff700 1 -- 172.21.15.27:0/3003964767 shutdown_connections 2024-01-11T08:22:52.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.666+0000 7f87159ff700 1 --2- 172.21.15.27:0/3003964767 >> v2:172.21.15.77:3300/0 conn(0x7f871006a570 0x7f8710069ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:52.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.666+0000 7f87159ff700 1 --2- 172.21.15.27:0/3003964767 >> v2:172.21.15.27:3300/0 conn(0x7f8710157440 0x7f87100ae430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:52.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.666+0000 7f87159ff700 1 --2- 172.21.15.27:0/3003964767 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8710157810 0x7f8710069660 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:52.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.666+0000 7f87159ff700 1 -- 172.21.15.27:0/3003964767 >> 172.21.15.27:0/3003964767 conn(0x7f87100680e0 msgr2=0x7f87100cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:52.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.666+0000 7f87159ff700 1 -- 172.21.15.27:0/3003964767 shutdown_connections 2024-01-11T08:22:52.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.666+0000 7f87159ff700 1 -- 172.21.15.27:0/3003964767 wait complete. 2024-01-11T08:22:52.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.668+0000 7f87159ff700 1 Processor -- start 2024-01-11T08:22:52.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.668+0000 7f87159ff700 1 -- start start 2024-01-11T08:22:52.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.669+0000 7f87159ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f871006a570 0x7f871014d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:52.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.669+0000 7f87159ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8710157440 0x7f871014f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:52.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.669+0000 7f87159ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8710157810 0x7f87100c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:52.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.669+0000 7f87159ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87100af0c0 con 0x7f8710157440 2024-01-11T08:22:52.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.669+0000 7f87159ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87100a98b0 con 0x7f8710157810 2024-01-11T08:22:52.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.669+0000 7f87159ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87100a9b90 con 0x7f871006a570 2024-01-11T08:22:52.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.669+0000 7f870e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8710157440 0x7f871014f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:52.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.669+0000 7f870e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8710157440 0x7f871014f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54482/0 (socket says 172.21.15.27:54482) 2024-01-11T08:22:52.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.669+0000 7f870e7fc700 1 -- 172.21.15.27:0/2786323708 learned_addr learned my addr 172.21.15.27:0/2786323708 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:52.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.670+0000 7f870f7fe700 1 --2- 172.21.15.27:0/2786323708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8710157810 0x7f87100c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:52.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.670+0000 7f870effd700 1 --2- 172.21.15.27:0/2786323708 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f871006a570 0x7f871014d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:52.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.670+0000 7f870e7fc700 1 -- 172.21.15.27:0/2786323708 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f871006a570 msgr2=0x7f871014d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:52.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.670+0000 7f870e7fc700 1 --2- 172.21.15.27:0/2786323708 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f871006a570 0x7f871014d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:52.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.670+0000 7f870e7fc700 1 -- 172.21.15.27:0/2786323708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8710157810 msgr2=0x7f87100c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:52.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.670+0000 7f870e7fc700 1 --2- 172.21.15.27:0/2786323708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8710157810 0x7f87100c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:52.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.670+0000 7f870e7fc700 1 -- 172.21.15.27:0/2786323708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8700009480 con 0x7f8710157440 2024-01-11T08:22:52.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.670+0000 7f870effd700 1 --2- 172.21.15.27:0/2786323708 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f871006a570 0x7f871014d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:22:52.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.670+0000 7f870e7fc700 1 --2- 172.21.15.27:0/2786323708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8710157440 0x7f871014f8b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8704013000 tx=0x7f8704017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:52.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.670+0000 7f86f6ffd700 1 -- 172.21.15.27:0/2786323708 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870401f070 con 0x7f8710157440 2024-01-11T08:22:52.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.670+0000 7f86f6ffd700 1 -- 172.21.15.27:0/2786323708 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8704000c90 con 0x7f8710157440 2024-01-11T08:22:52.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.671+0000 7f87159ff700 1 -- 172.21.15.27:0/2786323708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8704009420 con 0x7f8710157440 2024-01-11T08:22:52.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.671+0000 7f86f6ffd700 1 -- 172.21.15.27:0/2786323708 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704011b50 con 0x7f8710157440 2024-01-11T08:22:52.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.671+0000 7f87159ff700 1 -- 172.21.15.27:0/2786323708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8710150410 con 0x7f8710157440 2024-01-11T08:22:52.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.672+0000 7f87159ff700 1 -- 172.21.15.27:0/2786323708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87100a9d40 con 0x7f8710157440 2024-01-11T08:22:52.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.673+0000 7f86f6ffd700 1 -- 172.21.15.27:0/2786323708 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 15) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f8704023070 con 0x7f8710157440 2024-01-11T08:22:52.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.674+0000 7f86f6ffd700 1 -- 172.21.15.27:0/2786323708 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6698+0+0 (secure 0 0 0) 0x7f8704085760 con 0x7f8710157440 2024-01-11T08:22:52.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.678+0000 7f86f6ffd700 1 -- 172.21.15.27:0/2786323708 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f87100a9d40 con 0x7f8710157440 2024-01-11T08:22:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.883+0000 7f87159ff700 1 -- 172.21.15.27:0/2786323708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f87100a9d00 con 0x7f8710157440 2024-01-11T08:22:52.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.969+0000 7f86f6ffd700 1 -- 172.21.15.27:0/2786323708 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 16) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f8704052370 con 0x7f8710157440 2024-01-11T08:22:52.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.969+0000 7f86f6ffd700 1 --2- 172.21.15.27:0/2786323708 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f86fc06c980 0x7f86fc06ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:52.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.970+0000 7f870effd700 1 --2- 172.21.15.27:0/2786323708 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f86fc06c980 0x7f86fc06ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:52.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:52.971+0000 7f870effd700 1 --2- 172.21.15.27:0/2786323708 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f86fc06c980 0x7f86fc06ed60 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8700009f60 tx=0x7f8700008040 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:53.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:53.760+0000 7f86f6ffd700 1 -- 172.21.15.27:0/2786323708 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v49) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8704027110 con 0x7f8710157440 2024-01-11T08:22:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:53.765+0000 7f87159ff700 1 -- 172.21.15.27:0/2786323708 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f86fc06c980 msgr2=0x7f86fc06ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:53.765+0000 7f87159ff700 1 --2- 172.21.15.27:0/2786323708 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f86fc06c980 0x7f86fc06ed60 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8700009f60 tx=0x7f8700008040 comp rx=0 tx=0).stop 2024-01-11T08:22:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:53.765+0000 7f87159ff700 1 -- 172.21.15.27:0/2786323708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8710157440 msgr2=0x7f871014f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:53.765+0000 7f87159ff700 1 --2- 172.21.15.27:0/2786323708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8710157440 0x7f871014f8b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8704013000 tx=0x7f8704017000 comp rx=0 tx=0).stop 2024-01-11T08:22:53.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:53.765+0000 7f87159ff700 1 -- 172.21.15.27:0/2786323708 shutdown_connections 2024-01-11T08:22:53.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:53.765+0000 7f87159ff700 1 --2- 172.21.15.27:0/2786323708 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f86fc06c980 0x7f86fc06ed60 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:53.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:53.765+0000 7f87159ff700 1 --2- 172.21.15.27:0/2786323708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8710157810 0x7f87100c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:53.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:53.765+0000 7f87159ff700 1 --2- 172.21.15.27:0/2786323708 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f871006a570 0x7f871014d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:53.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:53.765+0000 7f87159ff700 1 --2- 172.21.15.27:0/2786323708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8710157440 0x7f871014f8b0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:53.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:53.765+0000 7f87159ff700 1 -- 172.21.15.27:0/2786323708 >> 172.21.15.27:0/2786323708 conn(0x7f87100680e0 msgr2=0x7f8710151b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:53.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:53.766+0000 7f87159ff700 1 -- 172.21.15.27:0/2786323708 shutdown_connections 2024-01-11T08:22:53.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:53.766+0000 7f87159ff700 1 -- 172.21.15.27:0/2786323708 wait complete. 2024-01-11T08:22:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-01-11T08:22:53.794 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-11T08:22:54.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.003+0000 7efcdc5ff700 1 Processor -- start 2024-01-11T08:22:54.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.003+0000 7efcdc5ff700 1 -- start start 2024-01-11T08:22:54.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.004+0000 7efcdc5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efcd4156440 0x7efcd40c7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:54.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.004+0000 7efcdc5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efcd4156810 0x7efcd406e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:54.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.004+0000 7efcdc5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efcd40ccac0 0x7efcd406ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:54.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.004+0000 7efcdc5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd40c8590 con 0x7efcd40ccac0 2024-01-11T08:22:54.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.004+0000 7efcdc5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd40c7980 con 0x7efcd4156810 2024-01-11T08:22:54.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.004+0000 7efcdc5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd40c7c60 con 0x7efcd4156440 2024-01-11T08:22:54.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.005+0000 7efcdab9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efcd40ccac0 0x7efcd406ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:54.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.005+0000 7efcd9b9a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efcd4156810 0x7efcd406e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:54.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.005+0000 7efcda39b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efcd4156440 0x7efcd40c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:54.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.005+0000 7efcd9b9a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efcd4156810 0x7efcd406e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:42334/0 (socket says 172.21.15.27:42334) 2024-01-11T08:22:54.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.005+0000 7efcd9b9a700 1 -- 172.21.15.27:0/4073549336 learned_addr learned my addr 172.21.15.27:0/4073549336 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:54.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.005+0000 7efcd9b9a700 1 -- 172.21.15.27:0/4073549336 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efcd4156440 msgr2=0x7efcd40c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:54.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.005+0000 7efcd9b9a700 1 --2- 172.21.15.27:0/4073549336 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efcd4156440 0x7efcd40c7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:54.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.005+0000 7efcd9b9a700 1 -- 172.21.15.27:0/4073549336 >> v2:172.21.15.27:3300/0 conn(0x7efcd40ccac0 msgr2=0x7efcd406ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:54.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.005+0000 7efcd9b9a700 1 --2- 172.21.15.27:0/4073549336 >> v2:172.21.15.27:3300/0 conn(0x7efcd40ccac0 0x7efcd406ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:54.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.005+0000 7efcd9b9a700 1 -- 172.21.15.27:0/4073549336 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd40c7f40 con 0x7efcd4156810 2024-01-11T08:22:54.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.005+0000 7efcdab9c700 1 --2- 172.21.15.27:0/4073549336 >> v2:172.21.15.27:3300/0 conn(0x7efcd40ccac0 0x7efcd406ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:22:54.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.006+0000 7efcd9b9a700 1 --2- 172.21.15.27:0/4073549336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcd4156810 0x7efcd406e720 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7efcd406d210 tx=0x7efcc4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1b8f10fe2d3500e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.006+0000 7efcd9399700 1 -- 172.21.15.27:0/4073549336 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc400b640 con 0x7efcd4156810 2024-01-11T08:22:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.006+0000 7efcd9399700 1 -- 172.21.15.27:0/4073549336 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc400b7a0 con 0x7efcd4156810 2024-01-11T08:22:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.006+0000 7efcd9399700 1 -- 172.21.15.27:0/4073549336 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc400bb60 con 0x7efcd4156810 2024-01-11T08:22:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.007+0000 7efcdc5ff700 1 -- 172.21.15.27:0/4073549336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcd4156810 msgr2=0x7efcd406e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.007+0000 7efcdc5ff700 1 --2- 172.21.15.27:0/4073549336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcd4156810 0x7efcd406e720 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7efcd406d210 tx=0x7efcc4009250 comp rx=0 tx=0).stop 2024-01-11T08:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.007+0000 7efcdc5ff700 1 -- 172.21.15.27:0/4073549336 shutdown_connections 2024-01-11T08:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.007+0000 7efcdc5ff700 1 --2- 172.21.15.27:0/4073549336 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efcd4156440 0x7efcd40c7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.007+0000 7efcdc5ff700 1 --2- 172.21.15.27:0/4073549336 >> v2:172.21.15.27:3300/0 conn(0x7efcd40ccac0 0x7efcd406ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.007+0000 7efcdc5ff700 1 --2- 172.21.15.27:0/4073549336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcd4156810 0x7efcd406e720 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.007+0000 7efcdc5ff700 1 -- 172.21.15.27:0/4073549336 >> 172.21.15.27:0/4073549336 conn(0x7efcd40680e0 msgr2=0x7efcd40cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.007+0000 7efcdc5ff700 1 -- 172.21.15.27:0/4073549336 shutdown_connections 2024-01-11T08:22:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.007+0000 7efcdc5ff700 1 -- 172.21.15.27:0/4073549336 wait complete. 2024-01-11T08:22:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.009+0000 7efcdc5ff700 1 Processor -- start 2024-01-11T08:22:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.009+0000 7efcdc5ff700 1 -- start start 2024-01-11T08:22:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.010+0000 7efcdc5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd40ccac0 0x7efcd40c6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.010+0000 7efcdc5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efcd4156440 0x7efcd40c6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.010+0000 7efcdc5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcd4156810 0x7efcd40af2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.010+0000 7efcdc5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd416a690 con 0x7efcd40ccac0 2024-01-11T08:22:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.010+0000 7efcdc5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd416a820 con 0x7efcd4156810 2024-01-11T08:22:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.010+0000 7efcdc5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd40c48b0 con 0x7efcd4156440 2024-01-11T08:22:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.010+0000 7efcda39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd40ccac0 0x7efcd40c6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.010+0000 7efcda39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd40ccac0 0x7efcd40c6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49502/0 (socket says 172.21.15.27:49502) 2024-01-11T08:22:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.010+0000 7efcda39b700 1 -- 172.21.15.27:0/1037712162 learned_addr learned my addr 172.21.15.27:0/1037712162 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.010+0000 7efcd9b9a700 1 --2- 172.21.15.27:0/1037712162 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efcd4156440 0x7efcd40c6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.010+0000 7efcdab9c700 1 --2- 172.21.15.27:0/1037712162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcd4156810 0x7efcd40af2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.011+0000 7efcda39b700 1 -- 172.21.15.27:0/1037712162 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efcd4156440 msgr2=0x7efcd40c6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.011+0000 7efcda39b700 1 --2- 172.21.15.27:0/1037712162 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efcd4156440 0x7efcd40c6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.011+0000 7efcda39b700 1 -- 172.21.15.27:0/1037712162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcd4156810 msgr2=0x7efcd40af2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.011+0000 7efcda39b700 1 --2- 172.21.15.27:0/1037712162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcd4156810 0x7efcd40af2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.011+0000 7efcda39b700 1 -- 172.21.15.27:0/1037712162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efccc009a20 con 0x7efcd40ccac0 2024-01-11T08:22:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.011+0000 7efcdab9c700 1 --2- 172.21.15.27:0/1037712162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcd4156810 0x7efcd40af2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:22:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.011+0000 7efcda39b700 1 --2- 172.21.15.27:0/1037712162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd40ccac0 0x7efcd40c6700 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7efcd000ecb0 tx=0x7efcd0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.011+0000 7efcca7fc700 1 -- 172.21.15.27:0/1037712162 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd000c920 con 0x7efcd40ccac0 2024-01-11T08:22:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.011+0000 7efcca7fc700 1 -- 172.21.15.27:0/1037712162 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd000ca80 con 0x7efcd40ccac0 2024-01-11T08:22:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.011+0000 7efcdc5ff700 1 -- 172.21.15.27:0/1037712162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc40094f0 con 0x7efcd40ccac0 2024-01-11T08:22:54.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.011+0000 7efcca7fc700 1 -- 172.21.15.27:0/1037712162 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd000ce40 con 0x7efcd40ccac0 2024-01-11T08:22:54.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.011+0000 7efcdc5ff700 1 -- 172.21.15.27:0/1037712162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd40c4260 con 0x7efcd40ccac0 2024-01-11T08:22:54.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.012+0000 7efcdc5ff700 1 -- 172.21.15.27:0/1037712162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcd404c3c0 con 0x7efcd40ccac0 2024-01-11T08:22:54.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.013+0000 7efcca7fc700 1 -- 172.21.15.27:0/1037712162 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 84580+0+0 (secure 0 0 0) 0x7efcd0013680 con 0x7efcd40ccac0 2024-01-11T08:22:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.015+0000 7efcca7fc700 1 --2- 172.21.15.27:0/1037712162 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efcb8066d80 0x7efcb8069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.015+0000 7efcca7fc700 1 -- 172.21.15.27:0/1037712162 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6260+0+0 (secure 0 0 0) 0x7efcd0084b40 con 0x7efcd40ccac0 2024-01-11T08:22:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.015+0000 7efcd9b9a700 1 --2- 172.21.15.27:0/1037712162 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efcb8066d80 0x7efcb8069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:54.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.016+0000 7efcd9b9a700 1 --2- 172.21.15.27:0/1037712162 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efcb8066d80 0x7efcb8069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7efcd406d210 tx=0x7efcc4017000 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:54.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.020+0000 7efcca7fc700 1 -- 172.21.15.27:0/1037712162 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efcd0051d60 con 0x7efcd40ccac0 2024-01-11T08:22:54.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.225+0000 7efcdc5ff700 1 -- 172.21.15.27:0/1037712162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efcd40c8220 con 0x7efcd40ccac0 2024-01-11T08:22:54.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.643+0000 7efcca7fc700 1 -- 172.21.15.27:0/1037712162 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v50) v1 ==== 169+0+0 (secure 0 0 0) 0x7efcd00210d0 con 0x7efcd40ccac0 2024-01-11T08:22:54.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.650+0000 7efcdc5ff700 1 -- 172.21.15.27:0/1037712162 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efcb8066d80 msgr2=0x7efcb8069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:54.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.650+0000 7efcdc5ff700 1 --2- 172.21.15.27:0/1037712162 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efcb8066d80 0x7efcb8069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7efcd406d210 tx=0x7efcc4017000 comp rx=0 tx=0).stop 2024-01-11T08:22:54.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.650+0000 7efcdc5ff700 1 -- 172.21.15.27:0/1037712162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd40ccac0 msgr2=0x7efcd40c6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:54.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.650+0000 7efcdc5ff700 1 --2- 172.21.15.27:0/1037712162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd40ccac0 0x7efcd40c6700 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7efcd000ecb0 tx=0x7efcd0009120 comp rx=0 tx=0).stop 2024-01-11T08:22:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.652+0000 7efcdc5ff700 1 -- 172.21.15.27:0/1037712162 shutdown_connections 2024-01-11T08:22:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.652+0000 7efcdc5ff700 1 --2- 172.21.15.27:0/1037712162 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7efcb8066d80 0x7efcb8069230 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.652+0000 7efcdc5ff700 1 --2- 172.21.15.27:0/1037712162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcd4156810 0x7efcd40af2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.652+0000 7efcdc5ff700 1 --2- 172.21.15.27:0/1037712162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd40ccac0 0x7efcd40c6700 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.652+0000 7efcdc5ff700 1 --2- 172.21.15.27:0/1037712162 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efcd4156440 0x7efcd40c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.652+0000 7efcdc5ff700 1 -- 172.21.15.27:0/1037712162 >> 172.21.15.27:0/1037712162 conn(0x7efcd40680e0 msgr2=0x7efcd40acb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.652+0000 7efcdc5ff700 1 -- 172.21.15.27:0/1037712162 shutdown_connections 2024-01-11T08:22:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.652+0000 7efcdc5ff700 1 -- 172.21.15.27:0/1037712162 wait complete. 2024-01-11T08:22:54.655 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-01-11T08:22:54.676 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-11T08:22:54.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.881+0000 7f9a121ff700 1 Processor -- start 2024-01-11T08:22:54.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.881+0000 7f9a121ff700 1 -- start start 2024-01-11T08:22:54.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.882+0000 7f9a121ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9a0c15a1b0 0x7f9a0c15a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:54.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.882+0000 7f9a121ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a0c0ccd70 0x7f9a0c0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:54.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.882+0000 7f9a121ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0c0cf850 con 0x7f9a0c0685c0 2024-01-11T08:22:54.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.882+0000 7f9a121ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a0c0cec40 con 0x7f9a0c15a1b0 2024-01-11T08:22:54.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.882+0000 7f9a121ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c0cef20 con 0x7f9a0c0ccd70 2024-01-11T08:22:54.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.883+0000 7f9a0bfff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9a0c0685c0 0x7f9a0c068a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51134/0 (socket says 172.21.15.27:51134) 2024-01-11T08:22:54.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.883+0000 7f9a0bfff700 1 -- 172.21.15.27:0/3601149400 learned_addr learned my addr 172.21.15.27:0/3601149400 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.883+0000 7f9a0affd700 1 --2- 172.21.15.27:0/3601149400 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a0c0ccd70 0x7f9a0c0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.883+0000 7f9a0b7fe700 1 --2- 172.21.15.27:0/3601149400 >> v2:172.21.15.77:3300/0 conn(0x7f9a0c15a1b0 0x7f9a0c15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.883+0000 7f9a0a7fc700 1 -- 172.21.15.27:0/3601149400 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996137850 0 0) 0x7f9a0c0cf850 con 0x7f9a0c0685c0 2024-01-11T08:22:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.883+0000 7f9a0a7fc700 1 -- 172.21.15.27:0/3601149400 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f4003570 con 0x7f9a0c0685c0 2024-01-11T08:22:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.883+0000 7f9a0a7fc700 1 -- 172.21.15.27:0/3601149400 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2449197787 0 0) 0x7f99f4003570 con 0x7f9a0c0685c0 2024-01-11T08:22:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.883+0000 7f9a0a7fc700 1 -- 172.21.15.27:0/3601149400 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a0c0ccd70 msgr2=0x7f9a0c0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.883+0000 7f9a0a7fc700 1 --2- 172.21.15.27:0/3601149400 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a0c0ccd70 0x7f9a0c0cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.883+0000 7f9a0a7fc700 1 -- 172.21.15.27:0/3601149400 >> v2:172.21.15.77:3300/0 conn(0x7f9a0c15a1b0 msgr2=0x7f9a0c15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.883+0000 7f9a0a7fc700 1 --2- 172.21.15.27:0/3601149400 >> v2:172.21.15.77:3300/0 conn(0x7f9a0c15a1b0 0x7f9a0c15a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.883+0000 7f9a0a7fc700 1 -- 172.21.15.27:0/3601149400 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a0c0cf200 con 0x7f9a0c0685c0 2024-01-11T08:22:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.883+0000 7f9a0a7fc700 1 -- 172.21.15.27:0/3601149400 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f9a040017f0 con 0x7f9a0c0685c0 2024-01-11T08:22:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.884+0000 7f9a0a7fc700 1 -- 172.21.15.27:0/3601149400 >> v1:172.21.15.27:6789/0 conn(0x7f9a0c0685c0 legacy=0x7f9a0c068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.884+0000 7f9a0a7fc700 1 --2- 172.21.15.27:0/3601149400 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a0c0ccd70 0x7f99f4003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.884+0000 7f9a0a7fc700 1 --2- 172.21.15.27:0/3601149400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99f40040b0 0x7f99f4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.884+0000 7f9a0a7fc700 1 --2- 172.21.15.27:0/3601149400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99f4004d50 0x7f99f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.884+0000 7f9a0a7fc700 1 -- 172.21.15.27:0/3601149400 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c0cec40 con 0x7f99f40040b0 2024-01-11T08:22:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.884+0000 7f9a0a7fc700 1 -- 172.21.15.27:0/3601149400 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c0cef20 con 0x7f99f4004d50 2024-01-11T08:22:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.884+0000 7f9a0a7fc700 1 -- 172.21.15.27:0/3601149400 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f400bb60 con 0x7f9a0c0ccd70 2024-01-11T08:22:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.884+0000 7f9a0affd700 1 --2- 172.21.15.27:0/3601149400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99f40040b0 0x7f99f4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.884+0000 7f9a0b7fe700 1 --2- 172.21.15.27:0/3601149400 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a0c0ccd70 0x7f99f4003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.884+0000 7f9a0bfff700 1 --2- 172.21.15.27:0/3601149400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99f4004d50 0x7f99f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.884+0000 7f9a0b7fe700 1 -- 172.21.15.27:0/3601149400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99f4004d50 msgr2=0x7f99f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.884+0000 7f9a0b7fe700 1 --2- 172.21.15.27:0/3601149400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99f4004d50 0x7f99f40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.884+0000 7f9a0b7fe700 1 -- 172.21.15.27:0/3601149400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99f40040b0 msgr2=0x7f99f4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.884+0000 7f9a0b7fe700 1 --2- 172.21.15.27:0/3601149400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99f40040b0 0x7f99f4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.884+0000 7f9a0b7fe700 1 -- 172.21.15.27:0/3601149400 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f4003570 con 0x7f9a0c0ccd70 2024-01-11T08:22:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.885+0000 7f9a0b7fe700 1 --2- 172.21.15.27:0/3601149400 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a0c0ccd70 0x7f99f4003ae0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f99fc006f00 tx=0x7f99fc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f58d77055ed0061a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.885+0000 7f9a0a7fc700 1 -- 172.21.15.27:0/3601149400 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99fc00e710 con 0x7f9a0c0ccd70 2024-01-11T08:22:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.885+0000 7f9a0a7fc700 1 -- 172.21.15.27:0/3601149400 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99fc00e870 con 0x7f9a0c0ccd70 2024-01-11T08:22:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.885+0000 7f9a0a7fc700 1 -- 172.21.15.27:0/3601149400 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99fc00ec30 con 0x7f9a0c0ccd70 2024-01-11T08:22:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.885+0000 7f9a121ff700 1 -- 172.21.15.27:0/3601149400 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a0c0ccd70 msgr2=0x7f99f4003ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.885+0000 7f9a121ff700 1 --2- 172.21.15.27:0/3601149400 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a0c0ccd70 0x7f99f4003ae0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f99fc006f00 tx=0x7f99fc0091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.885+0000 7f9a0bfff700 1 -- 172.21.15.27:0/3601149400 reap_dead start 2024-01-11T08:22:54.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.886+0000 7f9a121ff700 1 -- 172.21.15.27:0/3601149400 shutdown_connections 2024-01-11T08:22:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.886+0000 7f9a121ff700 1 -- 172.21.15.27:0/3601149400 >> 172.21.15.27:0/3601149400 conn(0x7f9a0c06ed40 msgr2=0x7f9a0c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.886+0000 7f9a121ff700 1 -- 172.21.15.27:0/3601149400 shutdown_connections 2024-01-11T08:22:54.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.886+0000 7f9a121ff700 1 -- 172.21.15.27:0/3601149400 wait complete. 2024-01-11T08:22:54.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.888+0000 7f9a121ff700 1 Processor -- start 2024-01-11T08:22:54.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.888+0000 7f9a121ff700 1 -- start start 2024-01-11T08:22:54.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.889+0000 7f9a121ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a0c0ccd70 0x7f9a0c158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:54.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.889+0000 7f9a121ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a0c0c8630 0x7f9a0c1692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:54.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.889+0000 7f9a121ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a0c1697f0 0x7f9a0c0c68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:54.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.889+0000 7f9a121ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c16ce00 con 0x7f9a0c0c8630 2024-01-11T08:22:54.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.889+0000 7f9a121ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c169d40 con 0x7f9a0c0ccd70 2024-01-11T08:22:54.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.889+0000 7f9a121ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c16a020 con 0x7f9a0c1697f0 2024-01-11T08:22:54.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.890+0000 7f9a0b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a0c0ccd70 0x7f9a0c158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:54.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.890+0000 7f9a0affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a0c0c8630 0x7f9a0c1692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:54.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.890+0000 7f9a0b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a0c0ccd70 0x7f9a0c158670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:42378/0 (socket says 172.21.15.27:42378) 2024-01-11T08:22:54.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.890+0000 7f9a0bfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a0c1697f0 0x7f9a0c0c68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:54.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.890+0000 7f9a0b7fe700 1 -- 172.21.15.27:0/2894361387 learned_addr learned my addr 172.21.15.27:0/2894361387 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:54.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.890+0000 7f9a0b7fe700 1 -- 172.21.15.27:0/2894361387 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a0c1697f0 msgr2=0x7f9a0c0c68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:54.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.890+0000 7f9a0b7fe700 1 --2- 172.21.15.27:0/2894361387 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a0c1697f0 0x7f9a0c0c68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:54.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.890+0000 7f9a0b7fe700 1 -- 172.21.15.27:0/2894361387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a0c0c8630 msgr2=0x7f9a0c1692b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:54.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.890+0000 7f9a0b7fe700 1 --2- 172.21.15.27:0/2894361387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a0c0c8630 0x7f9a0c1692b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:54.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.890+0000 7f9a0b7fe700 1 -- 172.21.15.27:0/2894361387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a0c16ca90 con 0x7f9a0c0ccd70 2024-01-11T08:22:54.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.890+0000 7f9a0affd700 1 --2- 172.21.15.27:0/2894361387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a0c0c8630 0x7f9a0c1692b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:22:54.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.890+0000 7f9a0b7fe700 1 --2- 172.21.15.27:0/2894361387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a0c0ccd70 0x7f9a0c158670 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f99f400bcf0 tx=0x7f99fc01a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:54.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.891+0000 7f99f37fe700 1 -- 172.21.15.27:0/2894361387 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99fc022070 con 0x7f9a0c0ccd70 2024-01-11T08:22:54.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.891+0000 7f99f37fe700 1 -- 172.21.15.27:0/2894361387 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99fc00e9e0 con 0x7f9a0c0ccd70 2024-01-11T08:22:54.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.891+0000 7f99f37fe700 1 -- 172.21.15.27:0/2894361387 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99fc00e710 con 0x7f9a0c0ccd70 2024-01-11T08:22:54.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.891+0000 7f9a121ff700 1 -- 172.21.15.27:0/2894361387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a0c158bb0 con 0x7f9a0c0ccd70 2024-01-11T08:22:54.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.891+0000 7f9a121ff700 1 -- 172.21.15.27:0/2894361387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a0c0c6fa0 con 0x7f9a0c0ccd70 2024-01-11T08:22:54.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.892+0000 7f9a121ff700 1 -- 172.21.15.27:0/2894361387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a0c169d40 con 0x7f9a0c0ccd70 2024-01-11T08:22:54.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.893+0000 7f99f37fe700 1 -- 172.21.15.27:0/2894361387 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f99fc027680 con 0x7f9a0c0ccd70 2024-01-11T08:22:54.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.893+0000 7f99f37fe700 1 --2- 172.21.15.27:0/2894361387 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f99f8066ea0 0x7f99f8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:54.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.894+0000 7f99f37fe700 1 -- 172.21.15.27:0/2894361387 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5846+0+0 (secure 0 0 0) 0x7f99fc085f50 con 0x7f9a0c0ccd70 2024-01-11T08:22:54.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.894+0000 7f9a0affd700 1 --2- 172.21.15.27:0/2894361387 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f99f8066ea0 0x7f99f8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:54.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.895+0000 7f9a0affd700 1 --2- 172.21.15.27:0/2894361387 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f99f8066ea0 0x7f99f8069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9a00009fd0 tx=0x7f9a000091a0 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:54.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:54.899+0000 7f99f37fe700 1 -- 172.21.15.27:0/2894361387 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f99fc053260 con 0x7f9a0c0ccd70 2024-01-11T08:22:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.101+0000 7f9a121ff700 1 -- 172.21.15.27:0/2894361387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f9a0c16d0e0 con 0x7f9a0c0ccd70 2024-01-11T08:22:55.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.107+0000 7f99f37fe700 1 -- 172.21.15.27:0/2894361387 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v13) v1 ==== 120+0+0 (secure 0 0 0) 0x7f99fc056840 con 0x7f9a0c0ccd70 2024-01-11T08:22:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.110+0000 7f9a121ff700 1 -- 172.21.15.27:0/2894361387 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f99f8066ea0 msgr2=0x7f99f8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.110+0000 7f9a121ff700 1 --2- 172.21.15.27:0/2894361387 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f99f8066ea0 0x7f99f8069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9a00009fd0 tx=0x7f9a000091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.110+0000 7f9a121ff700 1 -- 172.21.15.27:0/2894361387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a0c0ccd70 msgr2=0x7f9a0c158670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.110+0000 7f9a121ff700 1 --2- 172.21.15.27:0/2894361387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a0c0ccd70 0x7f9a0c158670 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f99f400bcf0 tx=0x7f99fc01a000 comp rx=0 tx=0).stop 2024-01-11T08:22:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.110+0000 7f9a121ff700 1 -- 172.21.15.27:0/2894361387 shutdown_connections 2024-01-11T08:22:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.110+0000 7f9a121ff700 1 --2- 172.21.15.27:0/2894361387 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f99f8066ea0 0x7f99f8069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.110+0000 7f9a121ff700 1 --2- 172.21.15.27:0/2894361387 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a0c1697f0 0x7f9a0c0c68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.110+0000 7f9a121ff700 1 --2- 172.21.15.27:0/2894361387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a0c0ccd70 0x7f9a0c158670 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.110+0000 7f9a121ff700 1 --2- 172.21.15.27:0/2894361387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a0c0c8630 0x7f9a0c1692b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.110+0000 7f9a121ff700 1 -- 172.21.15.27:0/2894361387 >> 172.21.15.27:0/2894361387 conn(0x7f9a0c06ed40 msgr2=0x7f9a0c0b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.110+0000 7f9a121ff700 1 -- 172.21.15.27:0/2894361387 shutdown_connections 2024-01-11T08:22:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.110+0000 7f9a121ff700 1 -- 172.21.15.27:0/2894361387 wait complete. 2024-01-11T08:22:55.128 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-11T08:22:55.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.332+0000 7f6761dff700 1 Processor -- start 2024-01-11T08:22:55.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.332+0000 7f6761dff700 1 -- start start 2024-01-11T08:22:55.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.334+0000 7f6761dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f675c143190 0x7f675c0cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:55.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.334+0000 7f6761dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f675c0ccc70 con 0x7f675c142dc0 2024-01-11T08:22:55.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.334+0000 7f6761dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f675c15d320 con 0x7f675c0cb4b0 2024-01-11T08:22:55.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.334+0000 7f6761dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f675c15df30 con 0x7f675c143190 2024-01-11T08:22:55.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.334+0000 7f675b7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f675c142dc0 0x7f675c0cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51138/0 (socket says 172.21.15.27:51138) 2024-01-11T08:22:55.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.334+0000 7f675b7fe700 1 -- 172.21.15.27:0/2505509567 learned_addr learned my addr 172.21.15.27:0/2505509567 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:55.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.334+0000 7f675affd700 1 --2- 172.21.15.27:0/2505509567 >> v2:172.21.15.27:3300/0 conn(0x7f675c143190 0x7f675c0cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:55.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.334+0000 7f675a7fc700 1 -- 172.21.15.27:0/2505509567 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1309500820 0 0) 0x7f675c0ccc70 con 0x7f675c142dc0 2024-01-11T08:22:55.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.334+0000 7f675a7fc700 1 -- 172.21.15.27:0/2505509567 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6744003570 con 0x7f675c142dc0 2024-01-11T08:22:55.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.335+0000 7f675affd700 1 -- 172.21.15.27:0/2505509567 >> v1:172.21.15.77:6789/0 conn(0x7f675c0cb4b0 legacy=0x7f675c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:55.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.335+0000 7f675affd700 1 -- 172.21.15.27:0/2505509567 >> v1:172.21.15.27:6789/0 conn(0x7f675c142dc0 legacy=0x7f675c0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:55.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.335+0000 7f675affd700 1 -- 172.21.15.27:0/2505509567 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675c15d590 con 0x7f675c143190 2024-01-11T08:22:55.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.335+0000 7f675affd700 1 --2- 172.21.15.27:0/2505509567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f675c143190 0x7f675c0cc020 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f67500098e0 tx=0x7f67500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2417e161273c715c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:55.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.335+0000 7f675a7fc700 1 -- 172.21.15.27:0/2505509567 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675000b640 con 0x7f675c143190 2024-01-11T08:22:55.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.335+0000 7f675a7fc700 1 -- 172.21.15.27:0/2505509567 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675000b7a0 con 0x7f675c143190 2024-01-11T08:22:55.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.335+0000 7f675a7fc700 1 -- 172.21.15.27:0/2505509567 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675000bb40 con 0x7f675c143190 2024-01-11T08:22:55.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.336+0000 7f6761dff700 1 -- 172.21.15.27:0/2505509567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f675c143190 msgr2=0x7f675c0cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:55.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.336+0000 7f6761dff700 1 --2- 172.21.15.27:0/2505509567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f675c143190 0x7f675c0cc020 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f67500098e0 tx=0x7f67500091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:55.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.336+0000 7f6761dff700 1 -- 172.21.15.27:0/2505509567 shutdown_connections 2024-01-11T08:22:55.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.336+0000 7f6761dff700 1 --2- 172.21.15.27:0/2505509567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f675c143190 0x7f675c0cc020 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:55.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.336+0000 7f6761dff700 1 -- 172.21.15.27:0/2505509567 >> 172.21.15.27:0/2505509567 conn(0x7f675c063ca0 msgr2=0x7f675c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:55.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.336+0000 7f6761dff700 1 -- 172.21.15.27:0/2505509567 shutdown_connections 2024-01-11T08:22:55.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.336+0000 7f6761dff700 1 -- 172.21.15.27:0/2505509567 wait complete. 2024-01-11T08:22:55.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.338+0000 7f6761dff700 1 Processor -- start 2024-01-11T08:22:55.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.338+0000 7f6761dff700 1 -- start start 2024-01-11T08:22:55.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.339+0000 7f6761dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f675c0cb4b0 0x7f675c16b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:55.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.339+0000 7f6761dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f675c142dc0 0x7f675c16abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:55.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.339+0000 7f6761dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f675c143190 0x7f675c140d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:55.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.339+0000 7f6761dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675c154430 con 0x7f675c143190 2024-01-11T08:22:55.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.339+0000 7f6761dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675c0b1f10 con 0x7f675c142dc0 2024-01-11T08:22:55.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.339+0000 7f6761dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f675c16b110 con 0x7f675c0cb4b0 2024-01-11T08:22:55.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.340+0000 7f675bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f675c143190 0x7f675c140d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:55.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.340+0000 7f675affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f675c142dc0 0x7f675c16abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:55.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.340+0000 7f675b7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f675c0cb4b0 0x7f675c16b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:55.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.340+0000 7f675bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f675c143190 0x7f675c140d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49528/0 (socket says 172.21.15.27:49528) 2024-01-11T08:22:55.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.340+0000 7f675bfff700 1 -- 172.21.15.27:0/2131759636 learned_addr learned my addr 172.21.15.27:0/2131759636 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:55.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.340+0000 7f675bfff700 1 -- 172.21.15.27:0/2131759636 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f675c0cb4b0 msgr2=0x7f675c16b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:55.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.340+0000 7f675bfff700 1 --2- 172.21.15.27:0/2131759636 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f675c0cb4b0 0x7f675c16b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:55.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.340+0000 7f675bfff700 1 -- 172.21.15.27:0/2131759636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f675c142dc0 msgr2=0x7f675c16abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:55.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.340+0000 7f675bfff700 1 --2- 172.21.15.27:0/2131759636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f675c142dc0 0x7f675c16abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:55.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.340+0000 7f675bfff700 1 -- 172.21.15.27:0/2131759636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6750009420 con 0x7f675c143190 2024-01-11T08:22:55.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.341+0000 7f675bfff700 1 --2- 172.21.15.27:0/2131759636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f675c143190 0x7f675c140d00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f675400c900 tx=0x7f6754001730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:55.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.341+0000 7f67437fe700 1 -- 172.21.15.27:0/2131759636 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6754012630 con 0x7f675c143190 2024-01-11T08:22:55.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.341+0000 7f67437fe700 1 -- 172.21.15.27:0/2131759636 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6754012790 con 0x7f675c143190 2024-01-11T08:22:55.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.341+0000 7f6761dff700 1 -- 172.21.15.27:0/2131759636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f675c141240 con 0x7f675c143190 2024-01-11T08:22:55.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.341+0000 7f67437fe700 1 -- 172.21.15.27:0/2131759636 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6754012b50 con 0x7f675c143190 2024-01-11T08:22:55.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.341+0000 7f6761dff700 1 -- 172.21.15.27:0/2131759636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675c0b15e0 con 0x7f675c143190 2024-01-11T08:22:55.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.342+0000 7f6761dff700 1 -- 172.21.15.27:0/2131759636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f675c0b2080 con 0x7f675c143190 2024-01-11T08:22:55.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.343+0000 7f67437fe700 1 -- 172.21.15.27:0/2131759636 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f6754003030 con 0x7f675c143190 2024-01-11T08:22:55.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.344+0000 7f67437fe700 1 --2- 172.21.15.27:0/2131759636 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6748066ea0 0x7f6748069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:55.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.344+0000 7f67437fe700 1 -- 172.21.15.27:0/2131759636 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5846+0+0 (secure 0 0 0) 0x7f67540844c0 con 0x7f675c143190 2024-01-11T08:22:55.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.344+0000 7f675b7fe700 1 --2- 172.21.15.27:0/2131759636 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6748066ea0 0x7f6748069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:55.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.345+0000 7f675b7fe700 1 --2- 172.21.15.27:0/2131759636 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6748066ea0 0x7f6748069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f674c007fd0 tx=0x7f674c009040 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:55.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.348+0000 7f67437fe700 1 -- 172.21.15.27:0/2131759636 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f67540516e0 con 0x7f675c143190 2024-01-11T08:22:55.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.551+0000 7f6761dff700 1 -- 172.21.15.27:0/2131759636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f675c15d870 con 0x7f675c143190 2024-01-11T08:22:55.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.556+0000 7f67437fe700 1 -- 172.21.15.27:0/2131759636 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v14) v1 ==== 115+0+0 (secure 0 0 0) 0x7f6754054d00 con 0x7f675c143190 2024-01-11T08:22:55.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.558+0000 7f6761dff700 1 -- 172.21.15.27:0/2131759636 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6748066ea0 msgr2=0x7f6748069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:55.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.558+0000 7f6761dff700 1 --2- 172.21.15.27:0/2131759636 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6748066ea0 0x7f6748069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f674c007fd0 tx=0x7f674c009040 comp rx=0 tx=0).stop 2024-01-11T08:22:55.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.558+0000 7f6761dff700 1 -- 172.21.15.27:0/2131759636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f675c143190 msgr2=0x7f675c140d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:55.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.558+0000 7f6761dff700 1 --2- 172.21.15.27:0/2131759636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f675c143190 0x7f675c140d00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f675400c900 tx=0x7f6754001730 comp rx=0 tx=0).stop 2024-01-11T08:22:55.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.558+0000 7f6761dff700 1 -- 172.21.15.27:0/2131759636 shutdown_connections 2024-01-11T08:22:55.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.558+0000 7f6761dff700 1 --2- 172.21.15.27:0/2131759636 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6748066ea0 0x7f6748069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:55.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.558+0000 7f6761dff700 1 --2- 172.21.15.27:0/2131759636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f675c143190 0x7f675c140d00 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:55.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.558+0000 7f6761dff700 1 --2- 172.21.15.27:0/2131759636 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f675c0cb4b0 0x7f675c16b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:55.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.558+0000 7f6761dff700 1 --2- 172.21.15.27:0/2131759636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f675c142dc0 0x7f675c16abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:55.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.558+0000 7f6761dff700 1 -- 172.21.15.27:0/2131759636 >> 172.21.15.27:0/2131759636 conn(0x7f675c063ca0 msgr2=0x7f675c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:55.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.558+0000 7f6761dff700 1 -- 172.21.15.27:0/2131759636 shutdown_connections 2024-01-11T08:22:55.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.558+0000 7f6761dff700 1 -- 172.21.15.27:0/2131759636 wait complete. 2024-01-11T08:22:55.577 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-11T08:22:55.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.781+0000 7fb5607ff700 1 Processor -- start 2024-01-11T08:22:55.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.781+0000 7fb5607ff700 1 -- start start 2024-01-11T08:22:55.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.782+0000 7fb5607ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb558067860 0x7fb55806c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:55.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.782+0000 7fb5607ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb558067c30 0x7fb5580c7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:55.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.783+0000 7fb5607ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55806c3c0 con 0x7fb55806cd90 2024-01-11T08:22:55.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.783+0000 7fb5607ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5580cb9c0 con 0x7fb558067860 2024-01-11T08:22:55.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.783+0000 7fb5607ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb55806f650 con 0x7fb558067c30 2024-01-11T08:22:55.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.783+0000 7fb55e59b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb558067860 0x7fb55806c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:55.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.783+0000 7fb55dd9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb55806cd90 0x7fb5580cae90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51150/0 (socket says 172.21.15.27:51150) 2024-01-11T08:22:55.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.783+0000 7fb55dd9a700 1 -- 172.21.15.27:0/3964340514 learned_addr learned my addr 172.21.15.27:0/3964340514 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:55.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.783+0000 7fb55ed9c700 1 --2- 172.21.15.27:0/3964340514 >> v2:172.21.15.77:3300/0 conn(0x7fb558067c30 0x7fb5580c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:55.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.783+0000 7fb55e59b700 1 -- 172.21.15.27:0/3964340514 >> v2:172.21.15.77:3300/0 conn(0x7fb558067c30 msgr2=0x7fb5580c7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:55.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.783+0000 7fb55e59b700 1 --2- 172.21.15.27:0/3964340514 >> v2:172.21.15.77:3300/0 conn(0x7fb558067c30 0x7fb5580c7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:55.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.783+0000 7fb55e59b700 1 -- 172.21.15.27:0/3964340514 >> v1:172.21.15.27:6789/0 conn(0x7fb55806cd90 legacy=0x7fb5580cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:55.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.783+0000 7fb55e59b700 1 -- 172.21.15.27:0/3964340514 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55806ba90 con 0x7fb558067860 2024-01-11T08:22:55.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.783+0000 7fb55d599700 1 -- 172.21.15.27:0/3964340514 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062276556 0 0) 0x7fb55806c3c0 con 0x7fb55806cd90 2024-01-11T08:22:55.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.784+0000 7fb55e59b700 1 --2- 172.21.15.27:0/3964340514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb558067860 0x7fb55806c850 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb5580cd260 tx=0x7fb5540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbfed9e3812dd869 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:55.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.784+0000 7fb55d599700 1 -- 172.21.15.27:0/3964340514 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb554011630 con 0x7fb558067860 2024-01-11T08:22:55.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.784+0000 7fb55d599700 1 -- 172.21.15.27:0/3964340514 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb554011790 con 0x7fb558067860 2024-01-11T08:22:55.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.784+0000 7fb55d599700 1 -- 172.21.15.27:0/3964340514 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb554011b30 con 0x7fb558067860 2024-01-11T08:22:55.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.784+0000 7fb5607ff700 1 -- 172.21.15.27:0/3964340514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb558067860 msgr2=0x7fb55806c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:55.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.784+0000 7fb5607ff700 1 --2- 172.21.15.27:0/3964340514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb558067860 0x7fb55806c850 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb5580cd260 tx=0x7fb5540091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:55.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.785+0000 7fb5607ff700 1 -- 172.21.15.27:0/3964340514 shutdown_connections 2024-01-11T08:22:55.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.785+0000 7fb5607ff700 1 --2- 172.21.15.27:0/3964340514 >> v2:172.21.15.77:3300/0 conn(0x7fb558067c30 0x7fb5580c7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:55.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.785+0000 7fb5607ff700 1 --2- 172.21.15.27:0/3964340514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb558067860 0x7fb55806c850 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:55.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.785+0000 7fb5607ff700 1 -- 172.21.15.27:0/3964340514 >> 172.21.15.27:0/3964340514 conn(0x7fb55806b1f0 msgr2=0x7fb5580cd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:55.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.785+0000 7fb5607ff700 1 -- 172.21.15.27:0/3964340514 shutdown_connections 2024-01-11T08:22:55.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.785+0000 7fb5607ff700 1 -- 172.21.15.27:0/3964340514 wait complete. 2024-01-11T08:22:55.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.786+0000 7fb5607ff700 1 Processor -- start 2024-01-11T08:22:55.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.786+0000 7fb5607ff700 1 -- start start 2024-01-11T08:22:55.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.787+0000 7fb5607ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb558067860 0x7fb5580c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:55.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.787+0000 7fb5607ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb558067c30 0x7fb5580adc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:55.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.787+0000 7fb5607ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb55806cd90 0x7fb5580ae1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:55.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.787+0000 7fb5607ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55815d8f0 con 0x7fb558067c30 2024-01-11T08:22:55.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.787+0000 7fb5607ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55815d070 con 0x7fb55806cd90 2024-01-11T08:22:55.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.787+0000 7fb5607ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5580ae7d0 con 0x7fb558067860 2024-01-11T08:22:55.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.787+0000 7fb55dd9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb558067c30 0x7fb5580adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:55.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.788+0000 7fb55dd9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb558067c30 0x7fb5580adc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49546/0 (socket says 172.21.15.27:49546) 2024-01-11T08:22:55.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.788+0000 7fb55dd9a700 1 -- 172.21.15.27:0/162238732 learned_addr learned my addr 172.21.15.27:0/162238732 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:55.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.788+0000 7fb55e59b700 1 --2- 172.21.15.27:0/162238732 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb558067860 0x7fb5580c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:55.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.788+0000 7fb55ed9c700 1 --2- 172.21.15.27:0/162238732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb55806cd90 0x7fb5580ae1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:55.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.788+0000 7fb55dd9a700 1 -- 172.21.15.27:0/162238732 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb558067860 msgr2=0x7fb5580c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:55.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.788+0000 7fb55dd9a700 1 --2- 172.21.15.27:0/162238732 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb558067860 0x7fb5580c8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:55.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.788+0000 7fb55dd9a700 1 -- 172.21.15.27:0/162238732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb55806cd90 msgr2=0x7fb5580ae1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:55.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.788+0000 7fb55dd9a700 1 --2- 172.21.15.27:0/162238732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb55806cd90 0x7fb5580ae1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:55.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.788+0000 7fb55dd9a700 1 -- 172.21.15.27:0/162238732 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb554009420 con 0x7fb558067c30 2024-01-11T08:22:55.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.788+0000 7fb55dd9a700 1 --2- 172.21.15.27:0/162238732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb558067c30 0x7fb5580adc60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb54800c5b0 tx=0x7fb548001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:55.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.788+0000 7fb54e7fc700 1 -- 172.21.15.27:0/162238732 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb548005b30 con 0x7fb558067c30 2024-01-11T08:22:55.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.789+0000 7fb54e7fc700 1 -- 172.21.15.27:0/162238732 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb548005c90 con 0x7fb558067c30 2024-01-11T08:22:55.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.789+0000 7fb54e7fc700 1 -- 172.21.15.27:0/162238732 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb548007300 con 0x7fb558067c30 2024-01-11T08:22:55.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.789+0000 7fb5607ff700 1 -- 172.21.15.27:0/162238732 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5580aeab0 con 0x7fb558067c30 2024-01-11T08:22:55.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.789+0000 7fb5607ff700 1 -- 172.21.15.27:0/162238732 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5580c9140 con 0x7fb558067c30 2024-01-11T08:22:55.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.790+0000 7fb5607ff700 1 -- 172.21.15.27:0/162238732 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb55815d3e0 con 0x7fb558067c30 2024-01-11T08:22:55.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.791+0000 7fb54e7fc700 1 -- 172.21.15.27:0/162238732 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fb54801b3b0 con 0x7fb558067c30 2024-01-11T08:22:55.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.792+0000 7fb54e7fc700 1 --2- 172.21.15.27:0/162238732 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb53c066ea0 0x7fb53c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:55.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.792+0000 7fb54e7fc700 1 -- 172.21.15.27:0/162238732 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5846+0+0 (secure 0 0 0) 0x7fb548083940 con 0x7fb558067c30 2024-01-11T08:22:55.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.792+0000 7fb55e59b700 1 --2- 172.21.15.27:0/162238732 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb53c066ea0 0x7fb53c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:55.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.793+0000 7fb55e59b700 1 --2- 172.21.15.27:0/162238732 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb53c066ea0 0x7fb53c069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb5540097b0 tx=0x7fb554017000 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:55.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:55.796+0000 7fb54e7fc700 1 -- 172.21.15.27:0/162238732 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb55815d3e0 con 0x7fb558067c30 2024-01-11T08:22:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.001+0000 7fb5607ff700 1 -- 172.21.15.27:0/162238732 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb55815d3e0 con 0x7fb558067c30 2024-01-11T08:22:56.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.005+0000 7fb54e7fc700 1 -- 172.21.15.27:0/162238732 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb55815d3e0 con 0x7fb558067c30 2024-01-11T08:22:56.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.008+0000 7fb5607ff700 1 -- 172.21.15.27:0/162238732 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb53c066ea0 msgr2=0x7fb53c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:56.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.008+0000 7fb5607ff700 1 --2- 172.21.15.27:0/162238732 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb53c066ea0 0x7fb53c069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb5540097b0 tx=0x7fb554017000 comp rx=0 tx=0).stop 2024-01-11T08:22:56.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.008+0000 7fb5607ff700 1 -- 172.21.15.27:0/162238732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb558067c30 msgr2=0x7fb5580adc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:56.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.008+0000 7fb5607ff700 1 --2- 172.21.15.27:0/162238732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb558067c30 0x7fb5580adc60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb54800c5b0 tx=0x7fb548001070 comp rx=0 tx=0).stop 2024-01-11T08:22:56.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.008+0000 7fb5607ff700 1 -- 172.21.15.27:0/162238732 shutdown_connections 2024-01-11T08:22:56.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.008+0000 7fb5607ff700 1 --2- 172.21.15.27:0/162238732 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb53c066ea0 0x7fb53c069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:56.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.008+0000 7fb5607ff700 1 --2- 172.21.15.27:0/162238732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb55806cd90 0x7fb5580ae1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:56.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.008+0000 7fb5607ff700 1 --2- 172.21.15.27:0/162238732 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb558067860 0x7fb5580c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:56.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.008+0000 7fb5607ff700 1 --2- 172.21.15.27:0/162238732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb558067c30 0x7fb5580adc60 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:56.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.008+0000 7fb5607ff700 1 -- 172.21.15.27:0/162238732 >> 172.21.15.27:0/162238732 conn(0x7fb55806b1f0 msgr2=0x7fb5580c2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:56.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.008+0000 7fb5607ff700 1 -- 172.21.15.27:0/162238732 shutdown_connections 2024-01-11T08:22:56.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.008+0000 7fb5607ff700 1 -- 172.21.15.27:0/162238732 wait complete. 2024-01-11T08:22:56.026 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2024-01-11T08:22:56.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.227+0000 7f09273ff700 1 Processor -- start 2024-01-11T08:22:56.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.227+0000 7f09273ff700 1 -- start start 2024-01-11T08:22:56.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.228+0000 7f09273ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f09200b19d0 0x7f0920158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:56.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.228+0000 7f09273ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f092006b610 0x7f0920159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:56.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.228+0000 7f09273ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0920158b60 con 0x7f09201575e0 2024-01-11T08:22:56.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.228+0000 7f09273ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09200cb5e0 con 0x7f092006b610 2024-01-11T08:22:56.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.228+0000 7f09273ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09201599b0 con 0x7f09200b19d0 2024-01-11T08:22:56.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.228+0000 7f092599c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f092006b610 0x7f0920159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:56.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.228+0000 7f092599c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f092006b610 0x7f0920159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49560/0 (socket says 172.21.15.27:49560) 2024-01-11T08:22:56.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.228+0000 7f092599c700 1 -- 172.21.15.27:0/3733514903 learned_addr learned my addr 172.21.15.27:0/3733514903 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:56.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.228+0000 7f092499a700 1 --2- 172.21.15.27:0/3733514903 >> v2:172.21.15.77:3300/0 conn(0x7f09200b19d0 0x7f0920158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:56.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.229+0000 7f092599c700 1 -- 172.21.15.27:0/3733514903 >> v2:172.21.15.77:3300/0 conn(0x7f09200b19d0 msgr2=0x7f0920158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:56.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.229+0000 7f092599c700 1 --2- 172.21.15.27:0/3733514903 >> v2:172.21.15.77:3300/0 conn(0x7f09200b19d0 0x7f0920158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:56.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.229+0000 7f092599c700 1 -- 172.21.15.27:0/3733514903 >> v1:172.21.15.77:6789/0 conn(0x7f09201575e0 legacy=0x7f09200b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:56.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.229+0000 7f092599c700 1 -- 172.21.15.27:0/3733514903 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09200c0f90 con 0x7f092006b610 2024-01-11T08:22:56.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.229+0000 7f092599c700 1 --2- 172.21.15.27:0/3733514903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f092006b610 0x7f0920159e40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0920064fe0 tx=0x7f0918009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aadfaf525b62a1d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:56.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.229+0000 7f0917fff700 1 -- 172.21.15.27:0/3733514903 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091800e070 con 0x7f092006b610 2024-01-11T08:22:56.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.229+0000 7f0917fff700 1 -- 172.21.15.27:0/3733514903 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f091800ba20 con 0x7f092006b610 2024-01-11T08:22:56.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.229+0000 7f0917fff700 1 -- 172.21.15.27:0/3733514903 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09180092e0 con 0x7f092006b610 2024-01-11T08:22:56.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.230+0000 7f09273ff700 1 -- 172.21.15.27:0/3733514903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f092006b610 msgr2=0x7f0920159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:56.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.230+0000 7f09273ff700 1 --2- 172.21.15.27:0/3733514903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f092006b610 0x7f0920159e40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0920064fe0 tx=0x7f0918009840 comp rx=0 tx=0).stop 2024-01-11T08:22:56.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.230+0000 7f09273ff700 1 -- 172.21.15.27:0/3733514903 shutdown_connections 2024-01-11T08:22:56.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.230+0000 7f09273ff700 1 --2- 172.21.15.27:0/3733514903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f092006b610 0x7f0920159e40 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:56.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.230+0000 7f09273ff700 1 --2- 172.21.15.27:0/3733514903 >> v2:172.21.15.77:3300/0 conn(0x7f09200b19d0 0x7f0920158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:56.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.230+0000 7f09273ff700 1 -- 172.21.15.27:0/3733514903 >> 172.21.15.27:0/3733514903 conn(0x7f09200680e0 msgr2=0x7f0920069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:56.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.230+0000 7f09273ff700 1 -- 172.21.15.27:0/3733514903 shutdown_connections 2024-01-11T08:22:56.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.230+0000 7f09273ff700 1 -- 172.21.15.27:0/3733514903 wait complete. 2024-01-11T08:22:56.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.232+0000 7f09273ff700 1 Processor -- start 2024-01-11T08:22:56.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.232+0000 7f09273ff700 1 -- start start 2024-01-11T08:22:56.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.233+0000 7f09273ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f092006b610 0x7f0920153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:56.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.233+0000 7f09273ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09200b19d0 0x7f09200c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:56.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.233+0000 7f09273ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09201575e0 0x7f09200c4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:56.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.233+0000 7f09273ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09200c3df0 con 0x7f09201575e0 2024-01-11T08:22:56.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.233+0000 7f09273ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09200c3130 con 0x7f09200b19d0 2024-01-11T08:22:56.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.233+0000 7f09273ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09200c3ac0 con 0x7f092006b610 2024-01-11T08:22:56.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.233+0000 7f092599c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09201575e0 0x7f09200c4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:56.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.233+0000 7f092519b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f092006b610 0x7f0920153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:56.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.233+0000 7f092499a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09200b19d0 0x7f09200c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:56.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.233+0000 7f092599c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09201575e0 0x7f09200c4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49564/0 (socket says 172.21.15.27:49564) 2024-01-11T08:22:56.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.233+0000 7f092599c700 1 -- 172.21.15.27:0/1862283633 learned_addr learned my addr 172.21.15.27:0/1862283633 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:56.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.233+0000 7f092599c700 1 -- 172.21.15.27:0/1862283633 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f092006b610 msgr2=0x7f0920153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:56.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.233+0000 7f092599c700 1 --2- 172.21.15.27:0/1862283633 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f092006b610 0x7f0920153660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:56.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.233+0000 7f092599c700 1 -- 172.21.15.27:0/1862283633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09200b19d0 msgr2=0x7f09200c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:56.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.233+0000 7f092599c700 1 --2- 172.21.15.27:0/1862283633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09200b19d0 0x7f09200c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:56.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.234+0000 7f092599c700 1 -- 172.21.15.27:0/1862283633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0918009ae0 con 0x7f09201575e0 2024-01-11T08:22:56.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.234+0000 7f092599c700 1 --2- 172.21.15.27:0/1862283633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09201575e0 0x7f09200c4280 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0918009880 tx=0x7f0918018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:56.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.234+0000 7f09157fa700 1 -- 172.21.15.27:0/1862283633 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0918008a80 con 0x7f09201575e0 2024-01-11T08:22:56.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.234+0000 7f09157fa700 1 -- 172.21.15.27:0/1862283633 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0918008be0 con 0x7f09201575e0 2024-01-11T08:22:56.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.234+0000 7f09157fa700 1 -- 172.21.15.27:0/1862283633 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0918026460 con 0x7f09201575e0 2024-01-11T08:22:56.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.234+0000 7f09273ff700 1 -- 172.21.15.27:0/1862283633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09200c33d0 con 0x7f09201575e0 2024-01-11T08:22:56.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.234+0000 7f09273ff700 1 -- 172.21.15.27:0/1862283633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0920153e30 con 0x7f09201575e0 2024-01-11T08:22:56.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.235+0000 7f09273ff700 1 -- 172.21.15.27:0/1862283633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09200c38f0 con 0x7f09201575e0 2024-01-11T08:22:56.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.236+0000 7f09157fa700 1 -- 172.21.15.27:0/1862283633 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f0918003040 con 0x7f09201575e0 2024-01-11T08:22:56.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.237+0000 7f09157fa700 1 --2- 172.21.15.27:0/1862283633 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0904066ea0 0x7f0904069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:56.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.237+0000 7f09157fa700 1 -- 172.21.15.27:0/1862283633 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5846+0+0 (secure 0 0 0) 0x7f0918086370 con 0x7f09201575e0 2024-01-11T08:22:56.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.238+0000 7f092519b700 1 --2- 172.21.15.27:0/1862283633 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0904066ea0 0x7f0904069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:56.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.238+0000 7f092519b700 1 --2- 172.21.15.27:0/1862283633 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0904066ea0 0x7f0904069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f091c0039e0 tx=0x7f091c007ec0 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:56.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.241+0000 7f09157fa700 1 -- 172.21.15.27:0/1862283633 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0918053590 con 0x7f09201575e0 2024-01-11T08:22:56.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.522+0000 7f09273ff700 1 -- 172.21.15.27:0/1862283633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f09200718d0 con 0x7f09201575e0 2024-01-11T08:22:56.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.782+0000 7f09157fa700 1 -- 172.21.15.27:0/1862283633 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0918086020 con 0x7f09201575e0 2024-01-11T08:22:56.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.783+0000 7f09157fa700 1 -- 172.21.15.27:0/1862283633 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v443) v1 ==== 158+0+0 (secure 0 0 0) 0x7f0918069aa0 con 0x7f09201575e0 2024-01-11T08:22:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.786+0000 7f09273ff700 1 -- 172.21.15.27:0/1862283633 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0904066ea0 msgr2=0x7f0904069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.786+0000 7f09273ff700 1 --2- 172.21.15.27:0/1862283633 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0904066ea0 0x7f0904069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f091c0039e0 tx=0x7f091c007ec0 comp rx=0 tx=0).stop 2024-01-11T08:22:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.786+0000 7f09273ff700 1 -- 172.21.15.27:0/1862283633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09201575e0 msgr2=0x7f09200c4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.786+0000 7f09273ff700 1 --2- 172.21.15.27:0/1862283633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09201575e0 0x7f09200c4280 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0918009880 tx=0x7f0918018000 comp rx=0 tx=0).stop 2024-01-11T08:22:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.786+0000 7f09273ff700 1 -- 172.21.15.27:0/1862283633 shutdown_connections 2024-01-11T08:22:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.786+0000 7f09273ff700 1 --2- 172.21.15.27:0/1862283633 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0904066ea0 0x7f0904069350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.786+0000 7f09273ff700 1 --2- 172.21.15.27:0/1862283633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09201575e0 0x7f09200c4280 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.786+0000 7f09273ff700 1 --2- 172.21.15.27:0/1862283633 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f092006b610 0x7f0920153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.786+0000 7f09273ff700 1 --2- 172.21.15.27:0/1862283633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09200b19d0 0x7f09200c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.786+0000 7f09273ff700 1 -- 172.21.15.27:0/1862283633 >> 172.21.15.27:0/1862283633 conn(0x7f09200680e0 msgr2=0x7f092006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.787+0000 7f09273ff700 1 -- 172.21.15.27:0/1862283633 shutdown_connections 2024-01-11T08:22:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:56.787+0000 7f09273ff700 1 -- 172.21.15.27:0/1862283633 wait complete. 2024-01-11T08:22:56.806 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2024-01-11T08:22:56.806 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2024-01-11T08:22:56.806 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-11T08:22:56.806 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2024-01-11T08:22:57.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.007+0000 7f6f9fdff700 1 Processor -- start 2024-01-11T08:22:57.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.007+0000 7f6f9fdff700 1 -- start start 2024-01-11T08:22:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.009+0000 7f6f9fdff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6f980b03e0 0x7f6f980b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.009+0000 7f6f9fdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f980b29c0 0x7f6f98159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.009+0000 7f6f9fdff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f980cd730 con 0x7f6f980b07b0 2024-01-11T08:22:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.009+0000 7f6f9fdff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f980b2de0 con 0x7f6f980b03e0 2024-01-11T08:22:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.009+0000 7f6f9fdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980ccb20 con 0x7f6f980b29c0 2024-01-11T08:22:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.009+0000 7f6f9e39c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f980b29c0 0x7f6f98159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.009+0000 7f6f9d39a700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6f980b07b0 0x7f6f98159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:36732/0 (socket says 172.21.15.27:36732) 2024-01-11T08:22:57.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.009+0000 7f6f9db9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6f980b03e0 0x7f6f980b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.009+0000 7f6f9d39a700 1 -- 172.21.15.27:0/2773145736 learned_addr learned my addr 172.21.15.27:0/2773145736 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.010+0000 7f6f9cb99700 1 -- 172.21.15.27:0/2773145736 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732718706 0 0) 0x7f6f980cd730 con 0x7f6f980b07b0 2024-01-11T08:22:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.010+0000 7f6f9cb99700 1 -- 172.21.15.27:0/2773145736 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f84003570 con 0x7f6f980b07b0 2024-01-11T08:22:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.010+0000 7f6f9e39c700 1 -- 172.21.15.27:0/2773145736 >> v2:172.21.15.77:3300/0 conn(0x7f6f980b03e0 msgr2=0x7f6f980b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:57.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.010+0000 7f6f9e39c700 1 --2- 172.21.15.27:0/2773145736 >> v2:172.21.15.77:3300/0 conn(0x7f6f980b03e0 0x7f6f980b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.010+0000 7f6f9e39c700 1 -- 172.21.15.27:0/2773145736 >> v1:172.21.15.77:6789/0 conn(0x7f6f980b07b0 legacy=0x7f6f98159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.010+0000 7f6f9e39c700 1 -- 172.21.15.27:0/2773145736 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f980cce00 con 0x7f6f980b29c0 2024-01-11T08:22:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.010+0000 7f6f9db9b700 1 --2- 172.21.15.27:0/2773145736 >> v2:172.21.15.77:3300/0 conn(0x7f6f980b03e0 0x7f6f980b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:22:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.010+0000 7f6f9e39c700 1 --2- 172.21.15.27:0/2773145736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f980b29c0 0x7f6f98159910 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6f90009e60 tx=0x7f6f90009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2995e00a36f88ed8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.010+0000 7f6f9cb99700 1 -- 172.21.15.27:0/2773145736 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9000e070 con 0x7f6f980b29c0 2024-01-11T08:22:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.010+0000 7f6f9cb99700 1 -- 172.21.15.27:0/2773145736 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9000bab0 con 0x7f6f980b29c0 2024-01-11T08:22:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.010+0000 7f6f9cb99700 1 -- 172.21.15.27:0/2773145736 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9000be70 con 0x7f6f980b29c0 2024-01-11T08:22:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.011+0000 7f6f9fdff700 1 -- 172.21.15.27:0/2773145736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f980b29c0 msgr2=0x7f6f98159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.011+0000 7f6f9fdff700 1 --2- 172.21.15.27:0/2773145736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f980b29c0 0x7f6f98159910 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6f90009e60 tx=0x7f6f90009780 comp rx=0 tx=0).stop 2024-01-11T08:22:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.011+0000 7f6f9fdff700 1 -- 172.21.15.27:0/2773145736 shutdown_connections 2024-01-11T08:22:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.011+0000 7f6f9fdff700 1 --2- 172.21.15.27:0/2773145736 >> v2:172.21.15.77:3300/0 conn(0x7f6f980b03e0 0x7f6f980b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.011+0000 7f6f9fdff700 1 --2- 172.21.15.27:0/2773145736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f980b29c0 0x7f6f98159910 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.011+0000 7f6f9fdff700 1 -- 172.21.15.27:0/2773145736 >> 172.21.15.27:0/2773145736 conn(0x7f6f9806ed40 msgr2=0x7f6f9806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:57.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.011+0000 7f6f9fdff700 1 -- 172.21.15.27:0/2773145736 shutdown_connections 2024-01-11T08:22:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.011+0000 7f6f9fdff700 1 -- 172.21.15.27:0/2773145736 wait complete. 2024-01-11T08:22:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.013+0000 7f6f9fdff700 1 Processor -- start 2024-01-11T08:22:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.013+0000 7f6f9fdff700 1 -- start start 2024-01-11T08:22:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.014+0000 7f6f9fdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f980b03e0 0x7f6f980ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.014+0000 7f6f9fdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f980b07b0 0x7f6f98154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.014+0000 7f6f9fdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f980b29c0 0x7f6f980cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.014+0000 7f6f9fdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980cf5b0 con 0x7f6f980b29c0 2024-01-11T08:22:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.014+0000 7f6f9fdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980ce9a0 con 0x7f6f980b07b0 2024-01-11T08:22:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.014+0000 7f6f9fdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980cec80 con 0x7f6f980b03e0 2024-01-11T08:22:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.014+0000 7f6f9e39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f980b29c0 0x7f6f980cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.014+0000 7f6f9e39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f980b29c0 0x7f6f980cfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49576/0 (socket says 172.21.15.27:49576) 2024-01-11T08:22:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.014+0000 7f6f9e39c700 1 -- 172.21.15.27:0/239992122 learned_addr learned my addr 172.21.15.27:0/239992122 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.014+0000 7f6f9db9b700 1 --2- 172.21.15.27:0/239992122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f980b03e0 0x7f6f980ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.014+0000 7f6f9d39a700 1 --2- 172.21.15.27:0/239992122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f980b07b0 0x7f6f98154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.015+0000 7f6f9e39c700 1 -- 172.21.15.27:0/239992122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f980b03e0 msgr2=0x7f6f980ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.015+0000 7f6f9e39c700 1 --2- 172.21.15.27:0/239992122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f980b03e0 0x7f6f980ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.015+0000 7f6f9e39c700 1 -- 172.21.15.27:0/239992122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f980b07b0 msgr2=0x7f6f98154140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.015+0000 7f6f9e39c700 1 --2- 172.21.15.27:0/239992122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f980b07b0 0x7f6f98154140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.015+0000 7f6f9e39c700 1 -- 172.21.15.27:0/239992122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f94009420 con 0x7f6f980b29c0 2024-01-11T08:22:57.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.015+0000 7f6f9e39c700 1 --2- 172.21.15.27:0/239992122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f980b29c0 0x7f6f980cfa40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6f980ccc60 tx=0x7f6f90019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:57.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.015+0000 7f6f8dffb700 1 -- 172.21.15.27:0/239992122 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f90012e10 con 0x7f6f980b29c0 2024-01-11T08:22:57.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.015+0000 7f6f8dffb700 1 -- 172.21.15.27:0/239992122 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f90027470 con 0x7f6f980b29c0 2024-01-11T08:22:57.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.015+0000 7f6f8dffb700 1 -- 172.21.15.27:0/239992122 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f90027830 con 0x7f6f980b29c0 2024-01-11T08:22:57.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.015+0000 7f6f9fdff700 1 -- 172.21.15.27:0/239992122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f90009a20 con 0x7f6f980b29c0 2024-01-11T08:22:57.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.015+0000 7f6f9fdff700 1 -- 172.21.15.27:0/239992122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f980cf240 con 0x7f6f980b29c0 2024-01-11T08:22:57.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.017+0000 7f6f9fdff700 1 -- 172.21.15.27:0/239992122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f980cf760 con 0x7f6f980b29c0 2024-01-11T08:22:57.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.017+0000 7f6f8dffb700 1 -- 172.21.15.27:0/239992122 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6f90024070 con 0x7f6f980b29c0 2024-01-11T08:22:57.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.018+0000 7f6f8dffb700 1 --2- 172.21.15.27:0/239992122 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6f7c066f30 0x7f6f7c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:57.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.018+0000 7f6f8dffb700 1 -- 172.21.15.27:0/239992122 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5846+0+0 (secure 0 0 0) 0x7f6f90086ff0 con 0x7f6f980b29c0 2024-01-11T08:22:57.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.019+0000 7f6f9db9b700 1 --2- 172.21.15.27:0/239992122 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6f7c066f30 0x7f6f7c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:57.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.019+0000 7f6f9db9b700 1 --2- 172.21.15.27:0/239992122 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6f7c066f30 0x7f6f7c0693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6f94009f40 tx=0x7f6f940091a0 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:57.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.022+0000 7f6f8dffb700 1 -- 172.21.15.27:0/239992122 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6f90054190 con 0x7f6f980b29c0 2024-01-11T08:22:57.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.304+0000 7f6f9fdff700 1 -- 172.21.15.27:0/239992122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7f6f980cd0e0 con 0x7f6f980b29c0 2024-01-11T08:22:57.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.784+0000 7f6f8dffb700 1 -- 172.21.15.27:0/239992122 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v444) v1 ==== 157+0+0 (secure 0 0 0) 0x7f6f900577b0 con 0x7f6f980b29c0 2024-01-11T08:22:57.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.786+0000 7f6f9fdff700 1 -- 172.21.15.27:0/239992122 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6f7c066f30 msgr2=0x7f6f7c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:57.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.786+0000 7f6f9fdff700 1 --2- 172.21.15.27:0/239992122 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6f7c066f30 0x7f6f7c0693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6f94009f40 tx=0x7f6f940091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:57.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.786+0000 7f6f9fdff700 1 -- 172.21.15.27:0/239992122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f980b29c0 msgr2=0x7f6f980cfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:57.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.786+0000 7f6f9fdff700 1 --2- 172.21.15.27:0/239992122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f980b29c0 0x7f6f980cfa40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6f980ccc60 tx=0x7f6f90019000 comp rx=0 tx=0).stop 2024-01-11T08:22:57.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.787+0000 7f6f9fdff700 1 -- 172.21.15.27:0/239992122 shutdown_connections 2024-01-11T08:22:57.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.787+0000 7f6f9fdff700 1 --2- 172.21.15.27:0/239992122 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6f7c066f30 0x7f6f7c0693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:57.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.787+0000 7f6f9fdff700 1 --2- 172.21.15.27:0/239992122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f980b29c0 0x7f6f980cfa40 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:57.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.787+0000 7f6f9fdff700 1 --2- 172.21.15.27:0/239992122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f980b03e0 0x7f6f980ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:57.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.787+0000 7f6f9fdff700 1 --2- 172.21.15.27:0/239992122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f980b07b0 0x7f6f98154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:57.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.787+0000 7f6f9fdff700 1 -- 172.21.15.27:0/239992122 >> 172.21.15.27:0/239992122 conn(0x7f6f9806ed40 msgr2=0x7f6f980cc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:57.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.787+0000 7f6f9fdff700 1 -- 172.21.15.27:0/239992122 shutdown_connections 2024-01-11T08:22:57.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:57.787+0000 7f6f9fdff700 1 -- 172.21.15.27:0/239992122 wait complete. 2024-01-11T08:22:57.807 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-11T08:22:58.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.011+0000 7fa6dc7ff700 1 Processor -- start 2024-01-11T08:22:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.011+0000 7fa6dc7ff700 1 -- start start 2024-01-11T08:22:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.013+0000 7fa6dc7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6d41622d0 0x7fa6d41646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.013+0000 7fa6dc7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d40cf850 con 0x7fa6d415a1b0 2024-01-11T08:22:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.013+0000 7fa6dc7ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d40cec40 con 0x7fa6d40ccd70 2024-01-11T08:22:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.013+0000 7fa6dc7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d415ac90 con 0x7fa6d41622d0 2024-01-11T08:22:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.013+0000 7fa6da59b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa6d415a1b0 0x7fa6d415a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51162/0 (socket says 172.21.15.27:51162) 2024-01-11T08:22:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.013+0000 7fa6da59b700 1 -- 172.21.15.27:0/3394019736 learned_addr learned my addr 172.21.15.27:0/3394019736 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.013+0000 7fa6dad9c700 1 --2- 172.21.15.27:0/3394019736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6d41622d0 0x7fa6d41646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.013+0000 7fa6d9599700 1 -- 172.21.15.27:0/3394019736 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945721829 0 0) 0x7fa6d40cf850 con 0x7fa6d415a1b0 2024-01-11T08:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.013+0000 7fa6d9599700 1 -- 172.21.15.27:0/3394019736 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c0003570 con 0x7fa6d415a1b0 2024-01-11T08:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.013+0000 7fa6d9599700 1 -- 172.21.15.27:0/3394019736 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2494078744 0 0) 0x7fa6d40cec40 con 0x7fa6d40ccd70 2024-01-11T08:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.014+0000 7fa6d9599700 1 -- 172.21.15.27:0/3394019736 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d40cf850 con 0x7fa6d40ccd70 2024-01-11T08:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.014+0000 7fa6dad9c700 1 -- 172.21.15.27:0/3394019736 >> v1:172.21.15.77:6789/0 conn(0x7fa6d40ccd70 legacy=0x7fa6d40cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.014+0000 7fa6d9599700 1 -- 172.21.15.27:0/3394019736 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2061065247 0 0) 0x7fa6c0003570 con 0x7fa6d415a1b0 2024-01-11T08:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.014+0000 7fa6dad9c700 1 -- 172.21.15.27:0/3394019736 >> v1:172.21.15.27:6789/0 conn(0x7fa6d415a1b0 legacy=0x7fa6d415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.014+0000 7fa6dad9c700 1 -- 172.21.15.27:0/3394019736 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d40cef20 con 0x7fa6d41622d0 2024-01-11T08:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.014+0000 7fa6dad9c700 1 --2- 172.21.15.27:0/3394019736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6d41622d0 0x7fa6d41646b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc009e90 tx=0x7fa6cc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=553be6daca742a55 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.014+0000 7fa6d9599700 1 -- 172.21.15.27:0/3394019736 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc00e070 con 0x7fa6d41622d0 2024-01-11T08:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.014+0000 7fa6d9599700 1 -- 172.21.15.27:0/3394019736 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6cc00b870 con 0x7fa6d41622d0 2024-01-11T08:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.014+0000 7fa6d9599700 1 -- 172.21.15.27:0/3394019736 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d0001470 con 0x7fa6d41622d0 2024-01-11T08:22:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.015+0000 7fa6dc7ff700 1 -- 172.21.15.27:0/3394019736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6d41622d0 msgr2=0x7fa6d41646b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.015+0000 7fa6dc7ff700 1 --2- 172.21.15.27:0/3394019736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6d41622d0 0x7fa6d41646b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc009e90 tx=0x7fa6cc009780 comp rx=0 tx=0).stop 2024-01-11T08:22:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.015+0000 7fa6dc7ff700 1 -- 172.21.15.27:0/3394019736 shutdown_connections 2024-01-11T08:22:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.015+0000 7fa6dc7ff700 1 --2- 172.21.15.27:0/3394019736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6d41622d0 0x7fa6d41646b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.015+0000 7fa6dc7ff700 1 -- 172.21.15.27:0/3394019736 >> 172.21.15.27:0/3394019736 conn(0x7fa6d406ed40 msgr2=0x7fa6d406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:58.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.015+0000 7fa6dc7ff700 1 -- 172.21.15.27:0/3394019736 shutdown_connections 2024-01-11T08:22:58.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.015+0000 7fa6dc7ff700 1 -- 172.21.15.27:0/3394019736 wait complete. 2024-01-11T08:22:58.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.017+0000 7fa6dc7ff700 1 Processor -- start 2024-01-11T08:22:58.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.018+0000 7fa6dc7ff700 1 -- start start 2024-01-11T08:22:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.019+0000 7fa6dc7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6d40ccd70 0x7fa6d414bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.019+0000 7fa6dc7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6d415a1b0 0x7fa6d40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.019+0000 7fa6dc7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6d41622d0 0x7fa6d40b5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.019+0000 7fa6dc7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d40b59b0 con 0x7fa6d40ccd70 2024-01-11T08:22:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.019+0000 7fa6dc7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d414bd60 con 0x7fa6d415a1b0 2024-01-11T08:22:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.019+0000 7fa6dc7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d40b4870 con 0x7fa6d41622d0 2024-01-11T08:22:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.019+0000 7fa6da59b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6d40ccd70 0x7fa6d414bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.019+0000 7fa6da59b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6d40ccd70 0x7fa6d414bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49586/0 (socket says 172.21.15.27:49586) 2024-01-11T08:22:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.019+0000 7fa6da59b700 1 -- 172.21.15.27:0/2435590767 learned_addr learned my addr 172.21.15.27:0/2435590767 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.019+0000 7fa6da59b700 1 -- 172.21.15.27:0/2435590767 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6d41622d0 msgr2=0x7fa6d40b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.019+0000 7fa6da59b700 1 --2- 172.21.15.27:0/2435590767 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6d41622d0 0x7fa6d40b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.019+0000 7fa6da59b700 1 -- 172.21.15.27:0/2435590767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6d415a1b0 msgr2=0x7fa6d40b4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.019+0000 7fa6d9d9a700 1 --2- 172.21.15.27:0/2435590767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6d415a1b0 0x7fa6d40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.020+0000 7fa6da59b700 1 --2- 172.21.15.27:0/2435590767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6d415a1b0 0x7fa6d40b4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.020+0000 7fa6da59b700 1 -- 172.21.15.27:0/2435590767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6cc009a20 con 0x7fa6d40ccd70 2024-01-11T08:22:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.020+0000 7fa6d9d9a700 1 --2- 172.21.15.27:0/2435590767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6d415a1b0 0x7fa6d40b4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:22:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.020+0000 7fa6da59b700 1 --2- 172.21.15.27:0/2435590767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6d40ccd70 0x7fa6d414bf70 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa6d000cb90 tx=0x7fa6d0001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.020+0000 7fa6ca7fc700 1 -- 172.21.15.27:0/2435590767 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d0001d00 con 0x7fa6d40ccd70 2024-01-11T08:22:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.020+0000 7fa6ca7fc700 1 -- 172.21.15.27:0/2435590767 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d0007360 con 0x7fa6d40ccd70 2024-01-11T08:22:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.020+0000 7fa6ca7fc700 1 -- 172.21.15.27:0/2435590767 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d0004830 con 0x7fa6d40ccd70 2024-01-11T08:22:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.020+0000 7fa6dc7ff700 1 -- 172.21.15.27:0/2435590767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d40b5b90 con 0x7fa6d40ccd70 2024-01-11T08:22:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.020+0000 7fa6dc7ff700 1 -- 172.21.15.27:0/2435590767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d40b4350 con 0x7fa6d40ccd70 2024-01-11T08:22:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.021+0000 7fa6dc7ff700 1 -- 172.21.15.27:0/2435590767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d40b4570 con 0x7fa6d40ccd70 2024-01-11T08:22:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.022+0000 7fa6ca7fc700 1 -- 172.21.15.27:0/2435590767 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa6d0004be0 con 0x7fa6d40ccd70 2024-01-11T08:22:58.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.023+0000 7fa6ca7fc700 1 --2- 172.21.15.27:0/2435590767 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa6b8066f30 0x7fa6b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:58.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.023+0000 7fa6ca7fc700 1 -- 172.21.15.27:0/2435590767 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5846+0+0 (secure 0 0 0) 0x7fa6d0083f50 con 0x7fa6d40ccd70 2024-01-11T08:22:58.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.024+0000 7fa6d9d9a700 1 --2- 172.21.15.27:0/2435590767 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa6b8066f30 0x7fa6b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:58.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.024+0000 7fa6d9d9a700 1 --2- 172.21.15.27:0/2435590767 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa6b8066f30 0x7fa6b80693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4007fd0 tx=0x7fa6c4009040 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.027+0000 7fa6ca7fc700 1 -- 172.21.15.27:0/2435590767 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa6d0051110 con 0x7fa6d40ccd70 2024-01-11T08:22:58.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.227+0000 7fa6dc7ff700 1 -- 172.21.15.27:0/2435590767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa6d404c3c0 con 0x7fa6d40ccd70 2024-01-11T08:22:58.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.231+0000 7fa6ca7fc700 1 -- 172.21.15.27:0/2435590767 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa6d0018020 con 0x7fa6d40ccd70 2024-01-11T08:22:58.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.233+0000 7fa6dc7ff700 1 -- 172.21.15.27:0/2435590767 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa6b8066f30 msgr2=0x7fa6b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:58.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.233+0000 7fa6dc7ff700 1 --2- 172.21.15.27:0/2435590767 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa6b8066f30 0x7fa6b80693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4007fd0 tx=0x7fa6c4009040 comp rx=0 tx=0).stop 2024-01-11T08:22:58.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.233+0000 7fa6dc7ff700 1 -- 172.21.15.27:0/2435590767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6d40ccd70 msgr2=0x7fa6d414bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:58.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.233+0000 7fa6dc7ff700 1 --2- 172.21.15.27:0/2435590767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6d40ccd70 0x7fa6d414bf70 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa6d000cb90 tx=0x7fa6d0001ab0 comp rx=0 tx=0).stop 2024-01-11T08:22:58.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.234+0000 7fa6dc7ff700 1 -- 172.21.15.27:0/2435590767 shutdown_connections 2024-01-11T08:22:58.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.234+0000 7fa6dc7ff700 1 --2- 172.21.15.27:0/2435590767 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa6b8066f30 0x7fa6b80693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:58.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.234+0000 7fa6dc7ff700 1 --2- 172.21.15.27:0/2435590767 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa6d41622d0 0x7fa6d40b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:58.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.234+0000 7fa6dc7ff700 1 --2- 172.21.15.27:0/2435590767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6d40ccd70 0x7fa6d414bf70 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:58.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.234+0000 7fa6dc7ff700 1 --2- 172.21.15.27:0/2435590767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6d415a1b0 0x7fa6d40b4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:58.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.234+0000 7fa6dc7ff700 1 -- 172.21.15.27:0/2435590767 >> 172.21.15.27:0/2435590767 conn(0x7fa6d406ed40 msgr2=0x7fa6d4068ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:58.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.234+0000 7fa6dc7ff700 1 -- 172.21.15.27:0/2435590767 shutdown_connections 2024-01-11T08:22:58.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.234+0000 7fa6dc7ff700 1 -- 172.21.15.27:0/2435590767 wait complete. 2024-01-11T08:22:58.252 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T08:22:58.252 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:22:58.340 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:22:58.441 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T08:22:58.441 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-11T08:22:58.441 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:22:58.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.640+0000 7eff3fbff700 1 Processor -- start 2024-01-11T08:22:58.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.640+0000 7eff3fbff700 1 -- start start 2024-01-11T08:22:58.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.641+0000 7eff3fbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7eff380b03e0 0x7eff380b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:58.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.641+0000 7eff3fbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7eff380b07b0 0x7eff380b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:58.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.641+0000 7eff3fbff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff380cd730 con 0x7eff38159200 2024-01-11T08:22:58.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.641+0000 7eff3fbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff38159d50 con 0x7eff380b07b0 2024-01-11T08:22:58.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.642+0000 7eff3fbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff380ccb20 con 0x7eff380b03e0 2024-01-11T08:22:58.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.642+0000 7eff3e19c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7eff38159200 0x7eff38159640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51176/0 (socket says 172.21.15.27:51176) 2024-01-11T08:22:58.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.642+0000 7eff3e19c700 1 -- 172.21.15.27:0/1369899564 learned_addr learned my addr 172.21.15.27:0/1369899564 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:58.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.642+0000 7eff3d19a700 1 --2- 172.21.15.27:0/1369899564 >> v2:172.21.15.77:3300/0 conn(0x7eff380b07b0 0x7eff380b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:58.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.642+0000 7eff3d99b700 1 --2- 172.21.15.27:0/1369899564 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7eff380b03e0 0x7eff380b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:58.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.642+0000 7eff3c999700 1 -- 172.21.15.27:0/1369899564 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484826372 0 0) 0x7eff380cd730 con 0x7eff38159200 2024-01-11T08:22:58.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.642+0000 7eff3c999700 1 -- 172.21.15.27:0/1369899564 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff24003570 con 0x7eff38159200 2024-01-11T08:22:58.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.643+0000 7eff3d19a700 1 -- 172.21.15.27:0/1369899564 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7eff380b03e0 msgr2=0x7eff380b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:58.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.643+0000 7eff3c999700 1 -- 172.21.15.27:0/1369899564 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 522521914 0 0) 0x7eff24003570 con 0x7eff38159200 2024-01-11T08:22:58.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.643+0000 7eff3d19a700 1 --2- 172.21.15.27:0/1369899564 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7eff380b03e0 0x7eff380b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:58.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.643+0000 7eff3d19a700 1 -- 172.21.15.27:0/1369899564 >> v1:172.21.15.27:6789/0 conn(0x7eff38159200 legacy=0x7eff38159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:58.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.643+0000 7eff3d19a700 1 -- 172.21.15.27:0/1369899564 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff380cce00 con 0x7eff380b07b0 2024-01-11T08:22:58.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.643+0000 7eff3d19a700 1 --2- 172.21.15.27:0/1369899564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff380b07b0 0x7eff380b29c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7eff280098e0 tx=0x7eff280091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8a1cd01aab01328 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:58.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.643+0000 7eff3c999700 1 -- 172.21.15.27:0/1369899564 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff300017f0 con 0x7eff380b07b0 2024-01-11T08:22:58.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.643+0000 7eff3c999700 1 -- 172.21.15.27:0/1369899564 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff28011420 con 0x7eff380b07b0 2024-01-11T08:22:58.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.643+0000 7eff3c999700 1 -- 172.21.15.27:0/1369899564 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff280117c0 con 0x7eff380b07b0 2024-01-11T08:22:58.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.644+0000 7eff3fbff700 1 -- 172.21.15.27:0/1369899564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff380b07b0 msgr2=0x7eff380b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:58.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.644+0000 7eff3fbff700 1 --2- 172.21.15.27:0/1369899564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff380b07b0 0x7eff380b29c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7eff280098e0 tx=0x7eff280091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:58.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.644+0000 7eff3fbff700 1 -- 172.21.15.27:0/1369899564 shutdown_connections 2024-01-11T08:22:58.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.644+0000 7eff3fbff700 1 --2- 172.21.15.27:0/1369899564 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7eff380b03e0 0x7eff380b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:58.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.644+0000 7eff3fbff700 1 --2- 172.21.15.27:0/1369899564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff380b07b0 0x7eff380b29c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:58.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.644+0000 7eff3fbff700 1 -- 172.21.15.27:0/1369899564 >> 172.21.15.27:0/1369899564 conn(0x7eff3806ed40 msgr2=0x7eff3806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:58.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.644+0000 7eff3fbff700 1 -- 172.21.15.27:0/1369899564 shutdown_connections 2024-01-11T08:22:58.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.644+0000 7eff3fbff700 1 -- 172.21.15.27:0/1369899564 wait complete. 2024-01-11T08:22:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.645+0000 7eff3fbff700 1 Processor -- start 2024-01-11T08:22:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.646+0000 7eff3fbff700 1 -- start start 2024-01-11T08:22:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.646+0000 7eff3fbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff380b03e0 0x7eff380c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.646+0000 7eff3fbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7eff380b07b0 0x7eff380c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.647+0000 7eff3fbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff38159200 0x7eff3816a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.647+0000 7eff3fbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff380c7b00 con 0x7eff38159200 2024-01-11T08:22:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.647+0000 7eff3fbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff380c7f60 con 0x7eff380b03e0 2024-01-11T08:22:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.647+0000 7eff3fbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff380c9fe0 con 0x7eff380b07b0 2024-01-11T08:22:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.647+0000 7eff3e19c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff38159200 0x7eff3816a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.647+0000 7eff3d99b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff380b03e0 0x7eff380c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.647+0000 7eff3d99b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff380b03e0 0x7eff380c75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:42470/0 (socket says 172.21.15.27:42470) 2024-01-11T08:22:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.647+0000 7eff3d19a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7eff380b07b0 0x7eff380c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.647+0000 7eff3d99b700 1 -- 172.21.15.27:0/2500185457 learned_addr learned my addr 172.21.15.27:0/2500185457 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.647+0000 7eff3d99b700 1 -- 172.21.15.27:0/2500185457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7eff380b07b0 msgr2=0x7eff380c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.647+0000 7eff3d99b700 1 --2- 172.21.15.27:0/2500185457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7eff380b07b0 0x7eff380c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.647+0000 7eff3d99b700 1 -- 172.21.15.27:0/2500185457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff38159200 msgr2=0x7eff3816a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.647+0000 7eff3d99b700 1 --2- 172.21.15.27:0/2500185457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff38159200 0x7eff3816a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.647+0000 7eff3d99b700 1 -- 172.21.15.27:0/2500185457 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff28009420 con 0x7eff380b03e0 2024-01-11T08:22:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.647+0000 7eff3e19c700 1 --2- 172.21.15.27:0/2500185457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff38159200 0x7eff3816a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:22:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.648+0000 7eff3d99b700 1 --2- 172.21.15.27:0/2500185457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff380b03e0 0x7eff380c75c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7eff3400ec80 tx=0x7eff340091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.648+0000 7eff2dffb700 1 -- 172.21.15.27:0/2500185457 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3400c850 con 0x7eff380b03e0 2024-01-11T08:22:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.648+0000 7eff2dffb700 1 -- 172.21.15.27:0/2500185457 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3400c9b0 con 0x7eff380b03e0 2024-01-11T08:22:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.648+0000 7eff2dffb700 1 -- 172.21.15.27:0/2500185457 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3400cd50 con 0x7eff380b03e0 2024-01-11T08:22:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.648+0000 7eff3fbff700 1 -- 172.21.15.27:0/2500185457 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff380c56d0 con 0x7eff380b03e0 2024-01-11T08:22:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.648+0000 7eff3fbff700 1 -- 172.21.15.27:0/2500185457 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff380c84e0 con 0x7eff380b03e0 2024-01-11T08:22:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.649+0000 7eff3fbff700 1 -- 172.21.15.27:0/2500185457 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff380c3ed0 con 0x7eff380b03e0 2024-01-11T08:22:58.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.650+0000 7eff2dffb700 1 -- 172.21.15.27:0/2500185457 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7eff34013680 con 0x7eff380b03e0 2024-01-11T08:22:58.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.651+0000 7eff2dffb700 1 --2- 172.21.15.27:0/2500185457 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7eff1c066f30 0x7eff1c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:58.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.651+0000 7eff2dffb700 1 -- 172.21.15.27:0/2500185457 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5846+0+0 (secure 0 0 0) 0x7eff34084890 con 0x7eff380b03e0 2024-01-11T08:22:58.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.652+0000 7eff3d19a700 1 --2- 172.21.15.27:0/2500185457 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7eff1c066f30 0x7eff1c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:58.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.652+0000 7eff3d19a700 1 --2- 172.21.15.27:0/2500185457 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7eff1c066f30 0x7eff1c0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7eff280030c0 tx=0x7eff28016000 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:58.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.656+0000 7eff2dffb700 1 -- 172.21.15.27:0/2500185457 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7eff380c3ed0 con 0x7eff380b03e0 2024-01-11T08:22:58.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.902+0000 7eff3fbff700 1 -- 172.21.15.27:0/2500185457 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff380c3ed0 con 0x7eff380b03e0 2024-01-11T08:22:58.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.904+0000 7eff2dffb700 1 -- 172.21.15.27:0/2500185457 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3523 (secure 0 0 0) 0x7eff34017de0 con 0x7eff380b03e0 2024-01-11T08:22:58.904 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:22:58.904 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/2930156910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2930156910},{"type":"v1","addr":"172.21.15.77:6836","nonce":2930156910}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/685019444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":685019444},{"type":"v1","addr":"172.21.15.27:6834","nonce":685019444}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4861,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/144853545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":144853545},{"type":"v1","addr":"172.21.15.77:6837","nonce":144853545}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5586,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3665960176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":3665960176},{"type":"v1","addr":"172.21.15.27:6835","nonce":3665960176}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[]} 2024-01-11T08:22:58.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.906+0000 7eff3fbff700 1 -- 172.21.15.27:0/2500185457 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7eff1c066f30 msgr2=0x7eff1c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:58.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.906+0000 7eff3fbff700 1 --2- 172.21.15.27:0/2500185457 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7eff1c066f30 0x7eff1c0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7eff280030c0 tx=0x7eff28016000 comp rx=0 tx=0).stop 2024-01-11T08:22:58.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.906+0000 7eff3fbff700 1 -- 172.21.15.27:0/2500185457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff380b03e0 msgr2=0x7eff380c75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:58.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.906+0000 7eff3fbff700 1 --2- 172.21.15.27:0/2500185457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff380b03e0 0x7eff380c75c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7eff3400ec80 tx=0x7eff340091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:58.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.906+0000 7eff3fbff700 1 -- 172.21.15.27:0/2500185457 shutdown_connections 2024-01-11T08:22:58.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.906+0000 7eff3fbff700 1 --2- 172.21.15.27:0/2500185457 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7eff1c066f30 0x7eff1c0693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:58.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.906+0000 7eff3fbff700 1 --2- 172.21.15.27:0/2500185457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff38159200 0x7eff3816a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:58.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.906+0000 7eff3fbff700 1 --2- 172.21.15.27:0/2500185457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff380b03e0 0x7eff380c75c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:58.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.906+0000 7eff3fbff700 1 --2- 172.21.15.27:0/2500185457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7eff380b07b0 0x7eff380c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:58.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.906+0000 7eff3fbff700 1 -- 172.21.15.27:0/2500185457 >> 172.21.15.27:0/2500185457 conn(0x7eff3806ed40 msgr2=0x7eff3815c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:58.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.906+0000 7eff3fbff700 1 -- 172.21.15.27:0/2500185457 shutdown_connections 2024-01-11T08:22:58.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:58.906+0000 7eff3fbff700 1 -- 172.21.15.27:0/2500185457 wait complete. 2024-01-11T08:22:58.907 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-01-11T08:22:58.924 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-11T08:22:58.924 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-11T08:22:58.925 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-11T08:22:58.925 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:22:58.925 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-11T08:22:58.925 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-11T08:22:58.925 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-11T08:22:58.925 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:22:58.926 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-11T08:22:58.926 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-11T08:22:58.926 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-11T08:22:58.926 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:22:58.926 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-11T08:22:58.926 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-11T08:22:58.926 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-11T08:22:58.927 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:22:58.939 INFO:tasks.ceph.mds.c:Stopped 2024-01-11T08:22:58.939 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-11T08:22:58.940 INFO:tasks.ceph.mds.a:Stopped 2024-01-11T08:22:58.940 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-11T08:22:58.943 INFO:tasks.ceph.mds.d:Stopped 2024-01-11T08:22:58.944 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-11T08:22:58.944 INFO:tasks.ceph.mds.b:Stopped 2024-01-11T08:22:58.944 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-11T08:22:58.984 INFO:tasks.ceph.mds.c:Started 2024-01-11T08:22:58.984 DEBUG:teuthology.parallel:result is None 2024-01-11T08:22:58.986 INFO:tasks.ceph.mds.a:Started 2024-01-11T08:22:58.986 DEBUG:teuthology.parallel:result is None 2024-01-11T08:22:58.988 INFO:tasks.ceph.mds.d:Started 2024-01-11T08:22:58.988 DEBUG:teuthology.parallel:result is None 2024-01-11T08:22:58.991 INFO:tasks.ceph.mds.b:Started 2024-01-11T08:22:58.991 DEBUG:teuthology.parallel:result is None 2024-01-11T08:22:58.991 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-11T08:22:59.134 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T08:22:59.134 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T08:22:59.138 INFO:tasks.ceph.mds.b.smithi077.stdout:starting mds.b at 2024-01-11T08:22:59.138 INFO:tasks.ceph.mds.d.smithi077.stdout:starting mds.d at 2024-01-11T08:22:59.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.202+0000 7f062afff700 1 Processor -- start 2024-01-11T08:22:59.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.202+0000 7f062afff700 1 -- start start 2024-01-11T08:22:59.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.203+0000 7f062afff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0624157360 0x7f062406f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:59.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.203+0000 7f062afff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0624157730 0x7f06240cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:59.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.203+0000 7f062afff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06240cce10 con 0x7f06240b1290 2024-01-11T08:22:59.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.203+0000 7f062afff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06240cb540 con 0x7f0624157730 2024-01-11T08:22:59.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.203+0000 7f062afff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06240ca960 con 0x7f0624157360 2024-01-11T08:22:59.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.203+0000 7f0623fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0624157730 0x7f06240cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:59.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.204+0000 7f062959c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f06240b1290 0x7f062415b140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51190/0 (socket says 172.21.15.27:51190) 2024-01-11T08:22:59.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.204+0000 7f062959c700 1 -- 172.21.15.27:0/431632499 learned_addr learned my addr 172.21.15.27:0/431632499 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:22:59.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.204+0000 7f0628d9b700 1 --2- 172.21.15.27:0/431632499 >> v2:172.21.15.77:3300/0 conn(0x7f0624157360 0x7f062406f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:59.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.204+0000 7f0623fff700 1 -- 172.21.15.27:0/431632499 >> v2:172.21.15.77:3300/0 conn(0x7f0624157360 msgr2=0x7f062406f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:22:59.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.204+0000 7f0623fff700 1 --2- 172.21.15.27:0/431632499 >> v2:172.21.15.77:3300/0 conn(0x7f0624157360 0x7f062406f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:59.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.204+0000 7f0623fff700 1 -- 172.21.15.27:0/431632499 >> v1:172.21.15.27:6789/0 conn(0x7f06240b1290 legacy=0x7f062415b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:59.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.204+0000 7f0623fff700 1 -- 172.21.15.27:0/431632499 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06240cac10 con 0x7f0624157730 2024-01-11T08:22:59.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.204+0000 7f0623fff700 1 --2- 172.21.15.27:0/431632499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0624157730 0x7f06240cc8d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0624076d60 tx=0x7f06100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=105edbc8c5e3b58c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:59.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.204+0000 7f06237fe700 1 -- 172.21.15.27:0/431632499 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061000b6b0 con 0x7f0624157730 2024-01-11T08:22:59.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.205+0000 7f06237fe700 1 -- 172.21.15.27:0/431632499 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061000b810 con 0x7f0624157730 2024-01-11T08:22:59.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.205+0000 7f06237fe700 1 -- 172.21.15.27:0/431632499 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061000bbb0 con 0x7f0624157730 2024-01-11T08:22:59.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.205+0000 7f062afff700 1 -- 172.21.15.27:0/431632499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0624157730 msgr2=0x7f06240cc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:59.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.205+0000 7f062afff700 1 --2- 172.21.15.27:0/431632499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0624157730 0x7f06240cc8d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0624076d60 tx=0x7f06100091a0 comp rx=0 tx=0).stop 2024-01-11T08:22:59.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.205+0000 7f062afff700 1 -- 172.21.15.27:0/431632499 shutdown_connections 2024-01-11T08:22:59.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.205+0000 7f062afff700 1 --2- 172.21.15.27:0/431632499 >> v2:172.21.15.77:3300/0 conn(0x7f0624157360 0x7f062406f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:59.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.205+0000 7f062afff700 1 --2- 172.21.15.27:0/431632499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0624157730 0x7f06240cc8d0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:59.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.205+0000 7f062afff700 1 -- 172.21.15.27:0/431632499 >> 172.21.15.27:0/431632499 conn(0x7f06240680b0 msgr2=0x7f06240cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:22:59.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.205+0000 7f062afff700 1 -- 172.21.15.27:0/431632499 shutdown_connections 2024-01-11T08:22:59.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.205+0000 7f062afff700 1 -- 172.21.15.27:0/431632499 wait complete. 2024-01-11T08:22:59.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.207+0000 7f062afff700 1 Processor -- start 2024-01-11T08:22:59.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.207+0000 7f062afff700 1 -- start start 2024-01-11T08:22:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.208+0000 7f062afff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06240b1290 0x7f06241474c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.208+0000 7f062afff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0624157360 0x7f0624154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.208+0000 7f062afff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0624157730 0x7f06241697d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.208+0000 7f062afff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06240aba80 con 0x7f06240b1290 2024-01-11T08:22:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.208+0000 7f062afff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0624147a00 con 0x7f0624157730 2024-01-11T08:22:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.208+0000 7f062afff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0624154cd0 con 0x7f0624157360 2024-01-11T08:22:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.208+0000 7f0628d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06240b1290 0x7f06241474c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.208+0000 7f0628d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06240b1290 0x7f06241474c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49646/0 (socket says 172.21.15.27:49646) 2024-01-11T08:22:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.208+0000 7f0628d9b700 1 -- 172.21.15.27:0/4076499174 learned_addr learned my addr 172.21.15.27:0/4076499174 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:22:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.208+0000 7f062959c700 1 --2- 172.21.15.27:0/4076499174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0624157730 0x7f06241697d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.208+0000 7f0623fff700 1 --2- 172.21.15.27:0/4076499174 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0624157360 0x7f0624154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:59.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.209+0000 7f0628d9b700 1 -- 172.21.15.27:0/4076499174 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0624157360 msgr2=0x7f0624154440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:59.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.209+0000 7f0628d9b700 1 --2- 172.21.15.27:0/4076499174 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0624157360 0x7f0624154440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:59.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.209+0000 7f0628d9b700 1 -- 172.21.15.27:0/4076499174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0624157730 msgr2=0x7f06241697d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:22:59.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.209+0000 7f0628d9b700 1 --2- 172.21.15.27:0/4076499174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0624157730 0x7f06241697d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:22:59.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.209+0000 7f0628d9b700 1 -- 172.21.15.27:0/4076499174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0610009420 con 0x7f06240b1290 2024-01-11T08:22:59.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.209+0000 7f0628d9b700 1 --2- 172.21.15.27:0/4076499174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06240b1290 0x7f06241474c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f061800d890 tx=0x7f0618008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:59.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.209+0000 7f0620ff9700 1 -- 172.21.15.27:0/4076499174 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061800b600 con 0x7f06240b1290 2024-01-11T08:22:59.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.209+0000 7f0620ff9700 1 -- 172.21.15.27:0/4076499174 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061800b760 con 0x7f06240b1290 2024-01-11T08:22:59.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.209+0000 7f062afff700 1 -- 172.21.15.27:0/4076499174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06240abcd0 con 0x7f06240b1290 2024-01-11T08:22:59.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.209+0000 7f0620ff9700 1 -- 172.21.15.27:0/4076499174 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061800bae0 con 0x7f06240b1290 2024-01-11T08:22:59.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.209+0000 7f062afff700 1 -- 172.21.15.27:0/4076499174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06240ac290 con 0x7f06240b1290 2024-01-11T08:22:59.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.210+0000 7f062afff700 1 -- 172.21.15.27:0/4076499174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f062404c3c0 con 0x7f06240b1290 2024-01-11T08:22:59.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.211+0000 7f0620ff9700 1 -- 172.21.15.27:0/4076499174 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f061801e460 con 0x7f06240b1290 2024-01-11T08:22:59.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.212+0000 7f0620ff9700 1 --2- 172.21.15.27:0/4076499174 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0608066f30 0x7f06080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:22:59.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.212+0000 7f0620ff9700 1 -- 172.21.15.27:0/4076499174 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5846+0+0 (secure 0 0 0) 0x7f0618084d10 con 0x7f06240b1290 2024-01-11T08:22:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.213+0000 7f0623fff700 1 --2- 172.21.15.27:0/4076499174 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0608066f30 0x7f06080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:22:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.214+0000 7f0623fff700 1 --2- 172.21.15.27:0/4076499174 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0608066f30 0x7f06080693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f06100093f0 tx=0x7f0610017000 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:22:59.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.216+0000 7f0620ff9700 1 -- 172.21.15.27:0/4076499174 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f061808a020 con 0x7f06240b1290 2024-01-11T08:22:59.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:22:59.433+0000 7f062afff700 1 -- 172.21.15.27:0/4076499174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f06240c9770 con 0x7f06240b1290 2024-01-11T08:23:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.663+0000 7f0620ff9700 1 -- 172.21.15.27:0/4076499174 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v55) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0618051e50 con 0x7f06240b1290 2024-01-11T08:23:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.667+0000 7f062afff700 1 -- 172.21.15.27:0/4076499174 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0608066f30 msgr2=0x7f06080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.667+0000 7f062afff700 1 --2- 172.21.15.27:0/4076499174 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0608066f30 0x7f06080693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f06100093f0 tx=0x7f0610017000 comp rx=0 tx=0).stop 2024-01-11T08:23:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.667+0000 7f062afff700 1 -- 172.21.15.27:0/4076499174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06240b1290 msgr2=0x7f06241474c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.667+0000 7f062afff700 1 --2- 172.21.15.27:0/4076499174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06240b1290 0x7f06241474c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f061800d890 tx=0x7f0618008040 comp rx=0 tx=0).stop 2024-01-11T08:23:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.667+0000 7f062afff700 1 -- 172.21.15.27:0/4076499174 shutdown_connections 2024-01-11T08:23:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.667+0000 7f062afff700 1 --2- 172.21.15.27:0/4076499174 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0608066f30 0x7f06080693e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.667+0000 7f062afff700 1 --2- 172.21.15.27:0/4076499174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0624157730 0x7f06241697d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.667+0000 7f062afff700 1 --2- 172.21.15.27:0/4076499174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06240b1290 0x7f06241474c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.667+0000 7f062afff700 1 --2- 172.21.15.27:0/4076499174 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0624157360 0x7f0624154440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.667+0000 7f062afff700 1 -- 172.21.15.27:0/4076499174 >> 172.21.15.27:0/4076499174 conn(0x7f06240680b0 msgr2=0x7f06240cb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.667+0000 7f062afff700 1 -- 172.21.15.27:0/4076499174 shutdown_connections 2024-01-11T08:23:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.667+0000 7f062afff700 1 -- 172.21.15.27:0/4076499174 wait complete. 2024-01-11T08:23:00.669 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-11T08:23:00.688 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-11T08:23:00.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.897+0000 7fad057ff700 1 Processor -- start 2024-01-11T08:23:00.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.897+0000 7fad057ff700 1 -- start start 2024-01-11T08:23:00.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.898+0000 7fad057ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fad000ca3e0 0x7fad000cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:00.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.898+0000 7fad057ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fad00159e80 0x7fad0015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:00.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.898+0000 7fad057ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad0015d640 con 0x7fad000ca7b0 2024-01-11T08:23:00.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.898+0000 7fad057ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad0015ae70 con 0x7fad00159e80 2024-01-11T08:23:00.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.898+0000 7fad057ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad000cc4f0 con 0x7fad000ca3e0 2024-01-11T08:23:00.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.899+0000 7facff7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fad00159e80 0x7fad0015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:00.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.899+0000 7facff7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fad00159e80 0x7fad0015dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49656/0 (socket says 172.21.15.27:49656) 2024-01-11T08:23:00.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.899+0000 7facff7fe700 1 -- 172.21.15.27:0/3685734025 learned_addr learned my addr 172.21.15.27:0/3685734025 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:00.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.899+0000 7facfeffd700 1 --2- 172.21.15.27:0/3685734025 >> v2:172.21.15.77:3300/0 conn(0x7fad000ca3e0 0x7fad000cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:00.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.899+0000 7facff7fe700 1 -- 172.21.15.27:0/3685734025 >> v2:172.21.15.77:3300/0 conn(0x7fad000ca3e0 msgr2=0x7fad000cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:00.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.899+0000 7facff7fe700 1 --2- 172.21.15.27:0/3685734025 >> v2:172.21.15.77:3300/0 conn(0x7fad000ca3e0 0x7fad000cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:00.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.899+0000 7facff7fe700 1 -- 172.21.15.27:0/3685734025 >> v1:172.21.15.77:6789/0 conn(0x7fad000ca7b0 legacy=0x7fad000c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:00.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.899+0000 7facff7fe700 1 -- 172.21.15.27:0/3685734025 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0015a5d0 con 0x7fad00159e80 2024-01-11T08:23:00.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.899+0000 7facfeffd700 1 --2- 172.21.15.27:0/3685734025 >> v2:172.21.15.77:3300/0 conn(0x7fad000ca3e0 0x7fad000cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:23:00.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.899+0000 7facff7fe700 1 --2- 172.21.15.27:0/3685734025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad00159e80 0x7fad0015dad0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7facf8009f50 tx=0x7facf8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4fdbea989ec8fed server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:00.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.900+0000 7facfdffb700 1 -- 172.21.15.27:0/3685734025 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf800e070 con 0x7fad00159e80 2024-01-11T08:23:00.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.900+0000 7facfdffb700 1 -- 172.21.15.27:0/3685734025 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf800ba30 con 0x7fad00159e80 2024-01-11T08:23:00.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.900+0000 7facfdffb700 1 -- 172.21.15.27:0/3685734025 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf80092e0 con 0x7fad00159e80 2024-01-11T08:23:00.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.900+0000 7fad057ff700 1 -- 172.21.15.27:0/3685734025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad00159e80 msgr2=0x7fad0015dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:00.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.900+0000 7fad057ff700 1 --2- 172.21.15.27:0/3685734025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad00159e80 0x7fad0015dad0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7facf8009f50 tx=0x7facf8009840 comp rx=0 tx=0).stop 2024-01-11T08:23:00.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.900+0000 7fad057ff700 1 -- 172.21.15.27:0/3685734025 shutdown_connections 2024-01-11T08:23:00.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.900+0000 7fad057ff700 1 --2- 172.21.15.27:0/3685734025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad00159e80 0x7fad0015dad0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:00.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.900+0000 7fad057ff700 1 --2- 172.21.15.27:0/3685734025 >> v2:172.21.15.77:3300/0 conn(0x7fad000ca3e0 0x7fad000cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:00.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.900+0000 7fad057ff700 1 -- 172.21.15.27:0/3685734025 >> 172.21.15.27:0/3685734025 conn(0x7fad000ce3e0 msgr2=0x7fad000ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:00.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.900+0000 7fad057ff700 1 -- 172.21.15.27:0/3685734025 shutdown_connections 2024-01-11T08:23:00.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.900+0000 7fad057ff700 1 -- 172.21.15.27:0/3685734025 wait complete. 2024-01-11T08:23:00.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.902+0000 7fad057ff700 1 Processor -- start 2024-01-11T08:23:00.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.902+0000 7fad057ff700 1 -- start start 2024-01-11T08:23:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.903+0000 7fad057ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fad000ca3e0 0x7fad00150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.903+0000 7fad057ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad000ca7b0 0x7fad000c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.903+0000 7fad057ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad00159e80 0x7fad0016d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.903+0000 7fad057ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad000c7ab0 con 0x7fad00159e80 2024-01-11T08:23:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.903+0000 7fad057ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad000b2c20 con 0x7fad000ca7b0 2024-01-11T08:23:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.903+0000 7fad057ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad000b2280 con 0x7fad000ca3e0 2024-01-11T08:23:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.903+0000 7facff7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad00159e80 0x7fad0016d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.903+0000 7facff7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad00159e80 0x7fad0016d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49662/0 (socket says 172.21.15.27:49662) 2024-01-11T08:23:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.903+0000 7facff7fe700 1 -- 172.21.15.27:0/2616721043 learned_addr learned my addr 172.21.15.27:0/2616721043 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.903+0000 7facfe7fc700 1 --2- 172.21.15.27:0/2616721043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad000ca7b0 0x7fad000c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.903+0000 7facfeffd700 1 --2- 172.21.15.27:0/2616721043 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fad000ca3e0 0x7fad00150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:00.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.903+0000 7facff7fe700 1 -- 172.21.15.27:0/2616721043 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fad000ca3e0 msgr2=0x7fad00150c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:00.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.903+0000 7facff7fe700 1 --2- 172.21.15.27:0/2616721043 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fad000ca3e0 0x7fad00150c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:00.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.903+0000 7facff7fe700 1 -- 172.21.15.27:0/2616721043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad000ca7b0 msgr2=0x7fad000c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:00.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.903+0000 7facff7fe700 1 --2- 172.21.15.27:0/2616721043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad000ca7b0 0x7fad000c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:00.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.904+0000 7facff7fe700 1 -- 172.21.15.27:0/2616721043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf8009ae0 con 0x7fad00159e80 2024-01-11T08:23:00.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.904+0000 7facfe7fc700 1 --2- 172.21.15.27:0/2616721043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad000ca7b0 0x7fad000c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:23:00.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.904+0000 7facfeffd700 1 --2- 172.21.15.27:0/2616721043 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fad000ca3e0 0x7fad00150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:23:00.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.904+0000 7facff7fe700 1 --2- 172.21.15.27:0/2616721043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad00159e80 0x7fad0016d110 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7facf8009880 tx=0x7facf8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:00.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.904+0000 7face6ffd700 1 -- 172.21.15.27:0/2616721043 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf8008e80 con 0x7fad00159e80 2024-01-11T08:23:00.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.904+0000 7face6ffd700 1 -- 172.21.15.27:0/2616721043 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf80094e0 con 0x7fad00159e80 2024-01-11T08:23:00.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.904+0000 7fad057ff700 1 -- 172.21.15.27:0/2616721043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad000b2560 con 0x7fad00159e80 2024-01-11T08:23:00.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.904+0000 7face6ffd700 1 -- 172.21.15.27:0/2616721043 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf80256b0 con 0x7fad00159e80 2024-01-11T08:23:00.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.904+0000 7fad057ff700 1 -- 172.21.15.27:0/2616721043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad0016de80 con 0x7fad00159e80 2024-01-11T08:23:00.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.905+0000 7fad057ff700 1 -- 172.21.15.27:0/2616721043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad000b2d60 con 0x7fad00159e80 2024-01-11T08:23:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.906+0000 7face6ffd700 1 -- 172.21.15.27:0/2616721043 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7facf8003040 con 0x7fad00159e80 2024-01-11T08:23:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.907+0000 7face6ffd700 1 --2- 172.21.15.27:0/2616721043 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7facec066f30 0x7facec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.907+0000 7face6ffd700 1 -- 172.21.15.27:0/2616721043 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4853+0+0 (secure 0 0 0) 0x7facf8085ed0 con 0x7fad00159e80 2024-01-11T08:23:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.908+0000 7facfeffd700 1 --2- 172.21.15.27:0/2616721043 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7facec066f30 0x7facec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.908+0000 7facfeffd700 1 --2- 172.21.15.27:0/2616721043 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7facec066f30 0x7facec0693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7facf0006f00 tx=0x7facf0008040 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:00.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:00.911+0000 7face6ffd700 1 -- 172.21.15.27:0/2616721043 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7facf80533f0 con 0x7fad00159e80 2024-01-11T08:23:01.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.191+0000 7fad057ff700 1 -- 172.21.15.27:0/2616721043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fad000718d0 con 0x7fad00159e80 2024-01-11T08:23:01.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.191+0000 7face6ffd700 1 -- 172.21.15.27:0/2616721043 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7facf8056a10 con 0x7fad00159e80 2024-01-11T08:23:01.190 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:23:01.191 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T08:23:01.191 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-11T08:23:01.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:01.191 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-11T08:23:01.191 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lQm3yLhAAv6erz5hMfat8EUzRA5luMA==", 2024-01-11T08:23:01.191 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:23:01.191 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:23:01.191 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:23:01.191 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:23:01.191 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:23:01.191 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:23:01.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:01.191 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-11T08:23:01.192 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9l+FG8ChAACHndtVLx7qS3sidPniSycA==", 2024-01-11T08:23:01.192 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:23:01.192 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:23:01.192 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:23:01.192 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:23:01.192 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:23:01.192 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:23:01.192 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:01.192 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-11T08:23:01.192 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l1NnEOhAALHXsc5es7hufCthGdRzkyw==", 2024-01-11T08:23:01.192 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:23:01.192 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:23:01.192 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:23:01.193 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:23:01.193 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:23:01.193 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:23:01.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:01.193 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-11T08:23:01.193 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lruoWFhAAm8AmrZtTJhYI1+19F79RPQ==", 2024-01-11T08:23:01.193 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:23:01.193 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:23:01.193 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:23:01.193 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:23:01.193 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:23:01.193 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:23:01.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:01.194 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-11T08:23:01.194 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA2op9ltZygORAAZr1X1nCN0ce8rIVaGDpFyQ==", 2024-01-11T08:23:01.194 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:23:01.194 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:23:01.194 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:23:01.194 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:23:01.194 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:23:01.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:01.194 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-11T08:23:01.194 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA6op9lqSRPAhAAmtyCsyK5ZVGX2wfhOCMuFQ==", 2024-01-11T08:23:01.194 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:23:01.194 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:23:01.194 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:23:01.194 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:23:01.195 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:23:01.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:01.195 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-11T08:23:01.195 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA9op9lZcs8BhAA8j+U+jPOoy+fliVWZAPFYg==", 2024-01-11T08:23:01.195 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:23:01.195 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:23:01.195 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:23:01.195 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:23:01.195 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:23:01.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:01.195 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-11T08:23:01.195 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBAop9lFuCUChAAP6si/2W1mxH4ecQJvuumkw==", 2024-01-11T08:23:01.196 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:23:01.196 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:23:01.196 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:23:01.196 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:23:01.196 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:23:01.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:01.196 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-11T08:23:01.196 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBGop9lOpwBIRAAcpcsjwmX2IldcKHJWAgHuA==", 2024-01-11T08:23:01.196 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:23:01.196 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:23:01.196 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:23:01.196 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:23:01.196 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:23:01.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:01.197 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-11T08:23:01.197 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBJop9l4q8hJRAAK8mAYVRQMoWS6wBOU69HmA==", 2024-01-11T08:23:01.197 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:23:01.197 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:23:01.197 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:23:01.197 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:23:01.197 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:23:01.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:01.197 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-11T08:23:01.197 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBMop9lqMF9KBAAFgFFI2GWt9AGif4AoScThw==", 2024-01-11T08:23:01.197 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:23:01.197 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:23:01.198 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:23:01.198 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:23:01.198 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:23:01.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:01.198 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-11T08:23:01.198 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBPop9lN7eDLBAAKswPZFfs0g2AaSe6V+pbbQ==", 2024-01-11T08:23:01.198 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:23:01.198 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:23:01.198 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:23:01.198 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:23:01.198 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:23:01.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:01.198 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-11T08:23:01.199 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lT5okIBAABy8PbJJbA062VNmb1Y8qhg==", 2024-01-11T08:23:01.199 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:23:01.199 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:23:01.199 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-01-11T08:23:01.199 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-11T08:23:01.199 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:23:01.199 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:01.199 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-11T08:23:01.199 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lDf37LhAAqV+EabKXJEW1sS5xtu1tXw==", 2024-01-11T08:23:01.199 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:23:01.199 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:23:01.199 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:23:01.199 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:23:01.200 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:23:01.200 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:23:01.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:01.200 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-11T08:23:01.200 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lWayxJxAAcalM8QSOGtbCXxjYZGaNQA==", 2024-01-11T08:23:01.200 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:23:01.200 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:23:01.200 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:23:01.200 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:23:01.200 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:23:01.200 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:23:01.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:01.200 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-11T08:23:01.201 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lHD4tNhAAdmnFBmMrK+QhQYdpbzW0mg==", 2024-01-11T08:23:01.201 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:23:01.201 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:23:01.201 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:23:01.201 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:23:01.201 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:23:01.201 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:23:01.201 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:01.201 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-11T08:23:01.201 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAwop9lGPtDMxAAJ8FjrSEdCtQL8AhtAyqnTw==", 2024-01-11T08:23:01.201 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:23:01.201 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:23:01.201 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:23:01.201 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:23:01.202 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:23:01.202 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:23:01.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:01.202 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-11T08:23:01.202 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lXBZLJxAAXa1S8Vkm3MiV4igtNbg/1Q==", 2024-01-11T08:23:01.202 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:23:01.202 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:23:01.202 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T08:23:01.202 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:23:01.202 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:23:01.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:01.202 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-11T08:23:01.202 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l3ZGoHxAANqzytTTvWYSomG+RFsFDyg==", 2024-01-11T08:23:01.202 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:23:01.203 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:23:01.203 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T08:23:01.203 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:23:01.203 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.203 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:01.203 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-11T08:23:01.203 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T08:23:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.194+0000 7fad057ff700 1 -- 172.21.15.27:0/2616721043 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7facec066f30 msgr2=0x7facec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.194+0000 7fad057ff700 1 --2- 172.21.15.27:0/2616721043 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7facec066f30 0x7facec0693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7facf0006f00 tx=0x7facf0008040 comp rx=0 tx=0).stop 2024-01-11T08:23:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.194+0000 7fad057ff700 1 -- 172.21.15.27:0/2616721043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad00159e80 msgr2=0x7fad0016d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.194+0000 7fad057ff700 1 --2- 172.21.15.27:0/2616721043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad00159e80 0x7fad0016d110 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7facf8009880 tx=0x7facf8018000 comp rx=0 tx=0).stop 2024-01-11T08:23:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.194+0000 7fad057ff700 1 -- 172.21.15.27:0/2616721043 shutdown_connections 2024-01-11T08:23:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.194+0000 7fad057ff700 1 --2- 172.21.15.27:0/2616721043 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7facec066f30 0x7facec0693e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.194+0000 7fad057ff700 1 --2- 172.21.15.27:0/2616721043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad00159e80 0x7fad0016d110 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:01.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.194+0000 7fad057ff700 1 --2- 172.21.15.27:0/2616721043 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fad000ca3e0 0x7fad00150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:01.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.194+0000 7fad057ff700 1 --2- 172.21.15.27:0/2616721043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad000ca7b0 0x7fad000c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:01.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.194+0000 7fad057ff700 1 -- 172.21.15.27:0/2616721043 >> 172.21.15.27:0/2616721043 conn(0x7fad000ce3e0 msgr2=0x7fad000694e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:01.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.194+0000 7fad057ff700 1 -- 172.21.15.27:0/2616721043 shutdown_connections 2024-01-11T08:23:01.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.194+0000 7fad057ff700 1 -- 172.21.15.27:0/2616721043 wait complete. 2024-01-11T08:23:01.211 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:23:01.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.410+0000 7fdb629ff700 1 Processor -- start 2024-01-11T08:23:01.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.411+0000 7fdb629ff700 1 -- start start 2024-01-11T08:23:01.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.412+0000 7fdb629ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb5c0b03e0 0x7fdb5c0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:01.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.412+0000 7fdb629ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c0b2e00 con 0x7fdb5c0b29c0 2024-01-11T08:23:01.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.412+0000 7fdb629ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c0cd730 con 0x7fdb5c0b07b0 2024-01-11T08:23:01.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.412+0000 7fdb629ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0ccb20 con 0x7fdb5c0b03e0 2024-01-11T08:23:01.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.412+0000 7fdb60f9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdb5c0b29c0 0x7fdb5c0aefc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51194/0 (socket says 172.21.15.27:51194) 2024-01-11T08:23:01.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.412+0000 7fdb60f9c700 1 -- 172.21.15.27:0/3906139428 learned_addr learned my addr 172.21.15.27:0/3906139428 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:23:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.412+0000 7fdb5bfff700 1 --2- 172.21.15.27:0/3906139428 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb5c0b03e0 0x7fdb5c0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.413+0000 7fdb5affd700 1 -- 172.21.15.27:0/3906139428 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486813606 0 0) 0x7fdb54001490 con 0x7fdb5c0b29c0 2024-01-11T08:23:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.413+0000 7fdb5affd700 1 -- 172.21.15.27:0/3906139428 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb44003570 con 0x7fdb5c0b29c0 2024-01-11T08:23:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.413+0000 7fdb5affd700 1 -- 172.21.15.27:0/3906139428 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766829743 0 0) 0x7fdb5c0cd730 con 0x7fdb5c0b07b0 2024-01-11T08:23:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.413+0000 7fdb5affd700 1 -- 172.21.15.27:0/3906139428 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb54001490 con 0x7fdb5c0b07b0 2024-01-11T08:23:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.413+0000 7fdb5affd700 1 -- 172.21.15.27:0/3906139428 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2619442543 0 0) 0x7fdb44003570 con 0x7fdb5c0b29c0 2024-01-11T08:23:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.413+0000 7fdb5bfff700 1 -- 172.21.15.27:0/3906139428 >> v1:172.21.15.77:6789/0 conn(0x7fdb5c0b07b0 legacy=0x7fdb5c159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.413+0000 7fdb5bfff700 1 -- 172.21.15.27:0/3906139428 >> v1:172.21.15.27:6789/0 conn(0x7fdb5c0b29c0 legacy=0x7fdb5c0aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.413+0000 7fdb5bfff700 1 -- 172.21.15.27:0/3906139428 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c0cce00 con 0x7fdb5c0b03e0 2024-01-11T08:23:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.413+0000 7fdb5bfff700 1 --2- 172.21.15.27:0/3906139428 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb5c0b03e0 0x7fdb5c0b2480 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009910 tx=0x7fdb4c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a16b3fe18ab38e73 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.414+0000 7fdb5affd700 1 -- 172.21.15.27:0/3906139428 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb54001980 con 0x7fdb5c0b03e0 2024-01-11T08:23:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.414+0000 7fdb5affd700 1 -- 172.21.15.27:0/3906139428 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4c011420 con 0x7fdb5c0b03e0 2024-01-11T08:23:01.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.414+0000 7fdb5affd700 1 -- 172.21.15.27:0/3906139428 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c0117c0 con 0x7fdb5c0b03e0 2024-01-11T08:23:01.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.414+0000 7fdb629ff700 1 -- 172.21.15.27:0/3906139428 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb5c0b03e0 msgr2=0x7fdb5c0b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:01.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.414+0000 7fdb629ff700 1 --2- 172.21.15.27:0/3906139428 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb5c0b03e0 0x7fdb5c0b2480 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009910 tx=0x7fdb4c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:23:01.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.414+0000 7fdb629ff700 1 -- 172.21.15.27:0/3906139428 shutdown_connections 2024-01-11T08:23:01.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.414+0000 7fdb629ff700 1 --2- 172.21.15.27:0/3906139428 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb5c0b03e0 0x7fdb5c0b2480 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:01.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.414+0000 7fdb629ff700 1 -- 172.21.15.27:0/3906139428 >> 172.21.15.27:0/3906139428 conn(0x7fdb5c06ed40 msgr2=0x7fdb5c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:01.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.414+0000 7fdb629ff700 1 -- 172.21.15.27:0/3906139428 shutdown_connections 2024-01-11T08:23:01.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.414+0000 7fdb629ff700 1 -- 172.21.15.27:0/3906139428 wait complete. 2024-01-11T08:23:01.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.416+0000 7fdb629ff700 1 Processor -- start 2024-01-11T08:23:01.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.416+0000 7fdb629ff700 1 -- start start 2024-01-11T08:23:01.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.417+0000 7fdb629ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb5c0b03e0 0x7fdb5c15bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:01.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.417+0000 7fdb629ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb5c0b07b0 0x7fdb5c0a8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:01.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.417+0000 7fdb629ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb5c0b29c0 0x7fdb5c0a9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:01.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.417+0000 7fdb629ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0c75c0 con 0x7fdb5c0b03e0 2024-01-11T08:23:01.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.417+0000 7fdb629ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0c7b40 con 0x7fdb5c0b29c0 2024-01-11T08:23:01.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.417+0000 7fdb629ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0a7df0 con 0x7fdb5c0b07b0 2024-01-11T08:23:01.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.417+0000 7fdb5bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb5c0b03e0 0x7fdb5c15bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:01.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.417+0000 7fdb5bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb5c0b03e0 0x7fdb5c15bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49678/0 (socket says 172.21.15.27:49678) 2024-01-11T08:23:01.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.417+0000 7fdb5bfff700 1 -- 172.21.15.27:0/2121327768 learned_addr learned my addr 172.21.15.27:0/2121327768 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:01.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.417+0000 7fdb5b7fe700 1 --2- 172.21.15.27:0/2121327768 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb5c0b07b0 0x7fdb5c0a8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:01.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.417+0000 7fdb60f9c700 1 --2- 172.21.15.27:0/2121327768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb5c0b29c0 0x7fdb5c0a9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:01.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.418+0000 7fdb5bfff700 1 -- 172.21.15.27:0/2121327768 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb5c0b07b0 msgr2=0x7fdb5c0a8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:01.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.418+0000 7fdb5bfff700 1 --2- 172.21.15.27:0/2121327768 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb5c0b07b0 0x7fdb5c0a8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:01.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.418+0000 7fdb5bfff700 1 -- 172.21.15.27:0/2121327768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb5c0b29c0 msgr2=0x7fdb5c0a9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:01.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.418+0000 7fdb5bfff700 1 --2- 172.21.15.27:0/2121327768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb5c0b29c0 0x7fdb5c0a9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:01.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.418+0000 7fdb5bfff700 1 -- 172.21.15.27:0/2121327768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4c009420 con 0x7fdb5c0b03e0 2024-01-11T08:23:01.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.418+0000 7fdb5bfff700 1 --2- 172.21.15.27:0/2121327768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb5c0b03e0 0x7fdb5c15bea0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c0091e0 tx=0x7fdb4c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:01.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.418+0000 7fdb43fff700 1 -- 172.21.15.27:0/2121327768 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c01cc90 con 0x7fdb5c0b03e0 2024-01-11T08:23:01.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.418+0000 7fdb43fff700 1 -- 172.21.15.27:0/2121327768 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4c01cdf0 con 0x7fdb5c0b03e0 2024-01-11T08:23:01.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.418+0000 7fdb43fff700 1 -- 172.21.15.27:0/2121327768 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c01a6a0 con 0x7fdb5c0b03e0 2024-01-11T08:23:01.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.418+0000 7fdb629ff700 1 -- 172.21.15.27:0/2121327768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5c0a80d0 con 0x7fdb5c0b03e0 2024-01-11T08:23:01.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.418+0000 7fdb629ff700 1 -- 172.21.15.27:0/2121327768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c0ca400 con 0x7fdb5c0b03e0 2024-01-11T08:23:01.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.419+0000 7fdb629ff700 1 -- 172.21.15.27:0/2121327768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5c0718d0 con 0x7fdb5c0b03e0 2024-01-11T08:23:01.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.420+0000 7fdb43fff700 1 -- 172.21.15.27:0/2121327768 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fdb4c01aad0 con 0x7fdb5c0b03e0 2024-01-11T08:23:01.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.422+0000 7fdb43fff700 1 --2- 172.21.15.27:0/2121327768 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdb48066f30 0x7fdb480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:01.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.422+0000 7fdb43fff700 1 -- 172.21.15.27:0/2121327768 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fdb4c085ff0 con 0x7fdb5c0b03e0 2024-01-11T08:23:01.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.422+0000 7fdb5b7fe700 1 --2- 172.21.15.27:0/2121327768 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdb48066f30 0x7fdb480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:01.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.423+0000 7fdb5b7fe700 1 --2- 172.21.15.27:0/2121327768 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdb48066f30 0x7fdb480693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c0b3610 tx=0x7fdb50009040 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:01.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.426+0000 7fdb43fff700 1 -- 172.21.15.27:0/2121327768 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdb4c053570 con 0x7fdb5c0b03e0 2024-01-11T08:23:01.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.630+0000 7fdb629ff700 1 -- 172.21.15.27:0/2121327768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb5c0cd0e0 con 0x7fdb5c0b03e0 2024-01-11T08:23:01.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.631+0000 7fdb43fff700 1 -- 172.21.15.27:0/2121327768 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+12086 (secure 0 0 0) 0x7fdb4c056b90 con 0x7fdb5c0b03e0 2024-01-11T08:23:01.630 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:23:01.630 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":55,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:23:00.660958+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:23:01.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.633+0000 7fdb629ff700 1 -- 172.21.15.27:0/2121327768 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdb48066f30 msgr2=0x7fdb480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:01.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.633+0000 7fdb629ff700 1 --2- 172.21.15.27:0/2121327768 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdb48066f30 0x7fdb480693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c0b3610 tx=0x7fdb50009040 comp rx=0 tx=0).stop 2024-01-11T08:23:01.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.634+0000 7fdb629ff700 1 -- 172.21.15.27:0/2121327768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb5c0b03e0 msgr2=0x7fdb5c15bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:01.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.634+0000 7fdb629ff700 1 --2- 172.21.15.27:0/2121327768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb5c0b03e0 0x7fdb5c15bea0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c0091e0 tx=0x7fdb4c016000 comp rx=0 tx=0).stop 2024-01-11T08:23:01.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.634+0000 7fdb629ff700 1 -- 172.21.15.27:0/2121327768 shutdown_connections 2024-01-11T08:23:01.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.634+0000 7fdb629ff700 1 --2- 172.21.15.27:0/2121327768 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdb48066f30 0x7fdb480693e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:01.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.634+0000 7fdb629ff700 1 --2- 172.21.15.27:0/2121327768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb5c0b29c0 0x7fdb5c0a9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:01.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.634+0000 7fdb629ff700 1 --2- 172.21.15.27:0/2121327768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb5c0b03e0 0x7fdb5c15bea0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:01.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.634+0000 7fdb629ff700 1 --2- 172.21.15.27:0/2121327768 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb5c0b07b0 0x7fdb5c0a8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:01.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.634+0000 7fdb629ff700 1 -- 172.21.15.27:0/2121327768 >> 172.21.15.27:0/2121327768 conn(0x7fdb5c06ed40 msgr2=0x7fdb5c15c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:01.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.634+0000 7fdb629ff700 1 -- 172.21.15.27:0/2121327768 shutdown_connections 2024-01-11T08:23:01.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.634+0000 7fdb629ff700 1 -- 172.21.15.27:0/2121327768 wait complete. 2024-01-11T08:23:01.651 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-11T08:23:01.651 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:23:01.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.855+0000 7f98ad3ff700 1 Processor -- start 2024-01-11T08:23:01.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.856+0000 7f98ad3ff700 1 -- start start 2024-01-11T08:23:01.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.857+0000 7f98ad3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f98a8157410 0x7f98a80ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:01.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.857+0000 7f98ad3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98a81577e0 0x7f98a80ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:01.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.857+0000 7f98ad3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a80699d0 con 0x7f98a8069540 2024-01-11T08:23:01.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.857+0000 7f98ad3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98a8069cb0 con 0x7f98a8157410 2024-01-11T08:23:01.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.857+0000 7f98ad3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a8069f90 con 0x7f98a81577e0 2024-01-11T08:23:01.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.857+0000 7f98a77fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f98a8069540 0x7f98a815b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51210/0 (socket says 172.21.15.27:51210) 2024-01-11T08:23:01.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.857+0000 7f98a77fe700 1 -- 172.21.15.27:0/545814842 learned_addr learned my addr 172.21.15.27:0/545814842 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:23:01.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.857+0000 7f98a67fc700 1 --2- 172.21.15.27:0/545814842 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98a81577e0 0x7f98a80ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:01.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.857+0000 7f98a6ffd700 1 --2- 172.21.15.27:0/545814842 >> v2:172.21.15.77:3300/0 conn(0x7f98a8157410 0x7f98a80ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:01.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.858+0000 7f98a5ffb700 1 -- 172.21.15.27:0/545814842 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972762964 0 0) 0x7f98a80699d0 con 0x7f98a8069540 2024-01-11T08:23:01.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.858+0000 7f98a5ffb700 1 -- 172.21.15.27:0/545814842 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9890003570 con 0x7f98a8069540 2024-01-11T08:23:01.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.858+0000 7f98a67fc700 1 -- 172.21.15.27:0/545814842 >> v2:172.21.15.77:3300/0 conn(0x7f98a8157410 msgr2=0x7f98a80ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:01.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.858+0000 7f98a67fc700 1 --2- 172.21.15.27:0/545814842 >> v2:172.21.15.77:3300/0 conn(0x7f98a8157410 0x7f98a80ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:01.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.858+0000 7f98a67fc700 1 -- 172.21.15.27:0/545814842 >> v1:172.21.15.27:6789/0 conn(0x7f98a8069540 legacy=0x7f98a815b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:01.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.858+0000 7f98a67fc700 1 -- 172.21.15.27:0/545814842 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a806a270 con 0x7f98a81577e0 2024-01-11T08:23:01.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.858+0000 7f98a67fc700 1 --2- 172.21.15.27:0/545814842 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98a81577e0 0x7f98a80ae000 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f989c0098e0 tx=0x7f989c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=662f2da92e025ee9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:01.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.859+0000 7f98a5ffb700 1 -- 172.21.15.27:0/545814842 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c011630 con 0x7f98a81577e0 2024-01-11T08:23:01.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.859+0000 7f98a5ffb700 1 -- 172.21.15.27:0/545814842 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989c011790 con 0x7f98a81577e0 2024-01-11T08:23:01.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.859+0000 7f98a5ffb700 1 -- 172.21.15.27:0/545814842 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c011b30 con 0x7f98a81577e0 2024-01-11T08:23:01.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.859+0000 7f98ad3ff700 1 -- 172.21.15.27:0/545814842 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98a81577e0 msgr2=0x7f98a80ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:01.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.859+0000 7f98ad3ff700 1 --2- 172.21.15.27:0/545814842 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98a81577e0 0x7f98a80ae000 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f989c0098e0 tx=0x7f989c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:23:01.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.859+0000 7f98ad3ff700 1 -- 172.21.15.27:0/545814842 shutdown_connections 2024-01-11T08:23:01.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.859+0000 7f98ad3ff700 1 --2- 172.21.15.27:0/545814842 >> v2:172.21.15.77:3300/0 conn(0x7f98a8157410 0x7f98a80ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:01.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.859+0000 7f98ad3ff700 1 --2- 172.21.15.27:0/545814842 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98a81577e0 0x7f98a80ae000 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:01.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.859+0000 7f98ad3ff700 1 -- 172.21.15.27:0/545814842 >> 172.21.15.27:0/545814842 conn(0x7f98a80680a0 msgr2=0x7f98a80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:01.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.859+0000 7f98ad3ff700 1 -- 172.21.15.27:0/545814842 shutdown_connections 2024-01-11T08:23:01.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.859+0000 7f98ad3ff700 1 -- 172.21.15.27:0/545814842 wait complete. 2024-01-11T08:23:01.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.861+0000 7f98ad3ff700 1 Processor -- start 2024-01-11T08:23:01.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.862+0000 7f98ad3ff700 1 -- start start 2024-01-11T08:23:01.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.863+0000 7f98ad3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98a8069540 0x7f98a8150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:01.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.863+0000 7f98ad3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98a8157410 0x7f98a815a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:01.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.863+0000 7f98ad3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98a81577e0 0x7f98a815c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:01.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.863+0000 7f98ad3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a80c6210 con 0x7f98a8069540 2024-01-11T08:23:01.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.863+0000 7f98ad3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a80c5970 con 0x7f98a81577e0 2024-01-11T08:23:01.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.863+0000 7f98ad3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a80c5c50 con 0x7f98a8157410 2024-01-11T08:23:01.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.863+0000 7f98a6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98a8069540 0x7f98a8150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:01.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.863+0000 7f98a6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98a8069540 0x7f98a8150d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49686/0 (socket says 172.21.15.27:49686) 2024-01-11T08:23:01.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.863+0000 7f98a6ffd700 1 -- 172.21.15.27:0/2082558015 learned_addr learned my addr 172.21.15.27:0/2082558015 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:01.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.863+0000 7f98a67fc700 1 --2- 172.21.15.27:0/2082558015 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98a8157410 0x7f98a815a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:01.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.863+0000 7f98a77fe700 1 --2- 172.21.15.27:0/2082558015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98a81577e0 0x7f98a815c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:01.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.864+0000 7f98a6ffd700 1 -- 172.21.15.27:0/2082558015 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98a8157410 msgr2=0x7f98a815a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:01.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.864+0000 7f98a6ffd700 1 --2- 172.21.15.27:0/2082558015 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98a8157410 0x7f98a815a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:01.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.864+0000 7f98a6ffd700 1 -- 172.21.15.27:0/2082558015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98a81577e0 msgr2=0x7f98a815c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:01.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.864+0000 7f98a6ffd700 1 --2- 172.21.15.27:0/2082558015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98a81577e0 0x7f98a815c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:01.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.864+0000 7f98a6ffd700 1 -- 172.21.15.27:0/2082558015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f989c009420 con 0x7f98a8069540 2024-01-11T08:23:01.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.864+0000 7f98a6ffd700 1 --2- 172.21.15.27:0/2082558015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98a8069540 0x7f98a8150d80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f989800ec80 tx=0x7f9898009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:01.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.864+0000 7f988effd700 1 -- 172.21.15.27:0/2082558015 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989800c9d0 con 0x7f98a8069540 2024-01-11T08:23:01.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.864+0000 7f988effd700 1 -- 172.21.15.27:0/2082558015 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989800cb30 con 0x7f98a8069540 2024-01-11T08:23:01.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.864+0000 7f988effd700 1 -- 172.21.15.27:0/2082558015 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9898013470 con 0x7f98a8069540 2024-01-11T08:23:01.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.864+0000 7f98ad3ff700 1 -- 172.21.15.27:0/2082558015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a80c5f30 con 0x7f98a8069540 2024-01-11T08:23:01.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.864+0000 7f98ad3ff700 1 -- 172.21.15.27:0/2082558015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a81512c0 con 0x7f98a8069540 2024-01-11T08:23:01.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.865+0000 7f98ad3ff700 1 -- 172.21.15.27:0/2082558015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a80ca2a0 con 0x7f98a8069540 2024-01-11T08:23:01.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.866+0000 7f988effd700 1 -- 172.21.15.27:0/2082558015 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9898013860 con 0x7f98a8069540 2024-01-11T08:23:01.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.867+0000 7f988effd700 1 --2- 172.21.15.27:0/2082558015 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9894066f30 0x7f98940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:01.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.868+0000 7f988effd700 1 -- 172.21.15.27:0/2082558015 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f98980849c0 con 0x7f98a8069540 2024-01-11T08:23:01.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.868+0000 7f98a67fc700 1 --2- 172.21.15.27:0/2082558015 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9894066f30 0x7f98940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:01.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.869+0000 7f98a67fc700 1 --2- 172.21.15.27:0/2082558015 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9894066f30 0x7f98940693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f989c0098e0 tx=0x7f989c017000 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:01.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:01.871+0000 7f988effd700 1 -- 172.21.15.27:0/2082558015 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9898051ee0 con 0x7f98a8069540 2024-01-11T08:23:02.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.073+0000 7f98ad3ff700 1 -- 172.21.15.27:0/2082558015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f98a80c9d90 con 0x7f98a8069540 2024-01-11T08:23:02.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.074+0000 7f988effd700 1 -- 172.21.15.27:0/2082558015 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) v1 ==== 93+0+10 (secure 0 0 0) 0x7f98a80c9d90 con 0x7f98a8069540 2024-01-11T08:23:02.073 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:23:02.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.076+0000 7f98ad3ff700 1 -- 172.21.15.27:0/2082558015 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9894066f30 msgr2=0x7f98940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.076+0000 7f98ad3ff700 1 --2- 172.21.15.27:0/2082558015 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9894066f30 0x7f98940693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f989c0098e0 tx=0x7f989c017000 comp rx=0 tx=0).stop 2024-01-11T08:23:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.076+0000 7f98ad3ff700 1 -- 172.21.15.27:0/2082558015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98a8069540 msgr2=0x7f98a8150d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.076+0000 7f98ad3ff700 1 --2- 172.21.15.27:0/2082558015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98a8069540 0x7f98a8150d80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f989800ec80 tx=0x7f9898009120 comp rx=0 tx=0).stop 2024-01-11T08:23:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.076+0000 7f98ad3ff700 1 -- 172.21.15.27:0/2082558015 shutdown_connections 2024-01-11T08:23:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.076+0000 7f98ad3ff700 1 --2- 172.21.15.27:0/2082558015 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9894066f30 0x7f98940693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.076+0000 7f98ad3ff700 1 --2- 172.21.15.27:0/2082558015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98a81577e0 0x7f98a815c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.076+0000 7f98ad3ff700 1 --2- 172.21.15.27:0/2082558015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98a8069540 0x7f98a8150d80 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.076+0000 7f98ad3ff700 1 --2- 172.21.15.27:0/2082558015 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98a8157410 0x7f98a815a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.076+0000 7f98ad3ff700 1 -- 172.21.15.27:0/2082558015 >> 172.21.15.27:0/2082558015 conn(0x7f98a80680a0 msgr2=0x7f98a80c2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.076+0000 7f98ad3ff700 1 -- 172.21.15.27:0/2082558015 shutdown_connections 2024-01-11T08:23:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.076+0000 7f98ad3ff700 1 -- 172.21.15.27:0/2082558015 wait complete. 2024-01-11T08:23:02.093 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-11T08:23:02.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.294+0000 7f81bafff700 1 Processor -- start 2024-01-11T08:23:02.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.294+0000 7f81bafff700 1 -- start start 2024-01-11T08:23:02.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.295+0000 7f81bafff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f81b40ca7b0 0x7f81b40cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:02.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.295+0000 7f81bafff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b415d640 con 0x7f81b4159e80 2024-01-11T08:23:02.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.295+0000 7f81bafff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b415a5d0 con 0x7f81b40ca3e0 2024-01-11T08:23:02.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.295+0000 7f81bafff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b415ae70 con 0x7f81b40ca7b0 2024-01-11T08:23:02.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.296+0000 7f81b959c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f81b4159e80 0x7f81b415dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51218/0 (socket says 172.21.15.27:51218) 2024-01-11T08:23:02.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.296+0000 7f81b959c700 1 -- 172.21.15.27:0/3940784875 learned_addr learned my addr 172.21.15.27:0/3940784875 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:23:02.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.296+0000 7f81b3fff700 1 --2- 172.21.15.27:0/3940784875 >> v2:172.21.15.77:3300/0 conn(0x7f81b40ca7b0 0x7f81b40cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:02.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.296+0000 7f81b37fe700 1 -- 172.21.15.27:0/3940784875 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421703436 0 0) 0x7f81b415d640 con 0x7f81b4159e80 2024-01-11T08:23:02.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.296+0000 7f81b37fe700 1 -- 172.21.15.27:0/3940784875 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a4003570 con 0x7f81b4159e80 2024-01-11T08:23:02.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.296+0000 7f81b37fe700 1 -- 172.21.15.27:0/3940784875 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1630639985 0 0) 0x7f81a4003570 con 0x7f81b4159e80 2024-01-11T08:23:02.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.296+0000 7f81b3fff700 1 -- 172.21.15.27:0/3940784875 >> v1:172.21.15.77:6789/0 conn(0x7f81b40ca3e0 legacy=0x7f81b40c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:02.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.296+0000 7f81b3fff700 1 -- 172.21.15.27:0/3940784875 >> v1:172.21.15.27:6789/0 conn(0x7f81b4159e80 legacy=0x7f81b415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:02.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.297+0000 7f81b3fff700 1 -- 172.21.15.27:0/3940784875 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b415a8b0 con 0x7f81b40ca7b0 2024-01-11T08:23:02.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.297+0000 7f81b3fff700 1 --2- 172.21.15.27:0/3940784875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81b40ca7b0 0x7f81b40cca20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f81a0009910 tx=0x7f81a00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87ebb51967d0bcd6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:02.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.297+0000 7f81b37fe700 1 -- 172.21.15.27:0/3940784875 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac0017f0 con 0x7f81b40ca7b0 2024-01-11T08:23:02.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.297+0000 7f81b37fe700 1 -- 172.21.15.27:0/3940784875 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a0011420 con 0x7f81b40ca7b0 2024-01-11T08:23:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.297+0000 7f81b37fe700 1 -- 172.21.15.27:0/3940784875 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a00117c0 con 0x7f81b40ca7b0 2024-01-11T08:23:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.298+0000 7f81bafff700 1 -- 172.21.15.27:0/3940784875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81b40ca7b0 msgr2=0x7f81b40cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.298+0000 7f81bafff700 1 --2- 172.21.15.27:0/3940784875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81b40ca7b0 0x7f81b40cca20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f81a0009910 tx=0x7f81a00091a0 comp rx=0 tx=0).stop 2024-01-11T08:23:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.298+0000 7f81bafff700 1 -- 172.21.15.27:0/3940784875 shutdown_connections 2024-01-11T08:23:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.298+0000 7f81bafff700 1 --2- 172.21.15.27:0/3940784875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81b40ca7b0 0x7f81b40cca20 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.298+0000 7f81bafff700 1 -- 172.21.15.27:0/3940784875 >> 172.21.15.27:0/3940784875 conn(0x7f81b40ce3e0 msgr2=0x7f81b40ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.298+0000 7f81bafff700 1 -- 172.21.15.27:0/3940784875 shutdown_connections 2024-01-11T08:23:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.298+0000 7f81bafff700 1 -- 172.21.15.27:0/3940784875 wait complete. 2024-01-11T08:23:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.299+0000 7f81bafff700 1 Processor -- start 2024-01-11T08:23:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.299+0000 7f81bafff700 1 -- start start 2024-01-11T08:23:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.300+0000 7f81bafff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81b40ca3e0 0x7f81b40b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.300+0000 7f81bafff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81b40ca7b0 0x7f81b415fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.300+0000 7f81bafff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f81b4159e80 0x7f81b4160430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.300+0000 7f81bafff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b4151ae0 con 0x7f81b40ca7b0 2024-01-11T08:23:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.300+0000 7f81bafff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b40b1180 con 0x7f81b40ca3e0 2024-01-11T08:23:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.300+0000 7f81bafff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b4160970 con 0x7f81b4159e80 2024-01-11T08:23:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.301+0000 7f81b3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81b40ca7b0 0x7f81b415fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.301+0000 7f81b3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81b40ca7b0 0x7f81b415fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49702/0 (socket says 172.21.15.27:49702) 2024-01-11T08:23:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.301+0000 7f81b3fff700 1 -- 172.21.15.27:0/1282507796 learned_addr learned my addr 172.21.15.27:0/1282507796 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:02.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.301+0000 7f81b959c700 1 --2- 172.21.15.27:0/1282507796 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f81b4159e80 0x7f81b4160430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:02.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.301+0000 7f81b8d9b700 1 --2- 172.21.15.27:0/1282507796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81b40ca3e0 0x7f81b40b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.301+0000 7f81b3fff700 1 -- 172.21.15.27:0/1282507796 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f81b4159e80 msgr2=0x7f81b4160430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.301+0000 7f81b3fff700 1 --2- 172.21.15.27:0/1282507796 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f81b4159e80 0x7f81b4160430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.301+0000 7f81b3fff700 1 -- 172.21.15.27:0/1282507796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81b40ca3e0 msgr2=0x7f81b40b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.301+0000 7f81b3fff700 1 --2- 172.21.15.27:0/1282507796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81b40ca3e0 0x7f81b40b2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.301+0000 7f81b3fff700 1 -- 172.21.15.27:0/1282507796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a0009420 con 0x7f81b40ca7b0 2024-01-11T08:23:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.301+0000 7f81b8d9b700 1 --2- 172.21.15.27:0/1282507796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81b40ca3e0 0x7f81b40b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:23:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.301+0000 7f81b3fff700 1 --2- 172.21.15.27:0/1282507796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81b40ca7b0 0x7f81b415fef0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f81a00093f0 tx=0x7f81a0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.302+0000 7f81b0ff9700 1 -- 172.21.15.27:0/1282507796 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a001bc80 con 0x7f81b40ca7b0 2024-01-11T08:23:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.302+0000 7f81bafff700 1 -- 172.21.15.27:0/1282507796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b40b1fa0 con 0x7f81b40ca7b0 2024-01-11T08:23:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.302+0000 7f81b0ff9700 1 -- 172.21.15.27:0/1282507796 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a0020070 con 0x7f81b40ca7b0 2024-01-11T08:23:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.302+0000 7f81b0ff9700 1 -- 172.21.15.27:0/1282507796 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a0027480 con 0x7f81b40ca7b0 2024-01-11T08:23:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.302+0000 7f81bafff700 1 -- 172.21.15.27:0/1282507796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b40b1670 con 0x7f81b40ca7b0 2024-01-11T08:23:02.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.303+0000 7f81bafff700 1 -- 172.21.15.27:0/1282507796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b40b2130 con 0x7f81b40ca7b0 2024-01-11T08:23:02.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.304+0000 7f81b0ff9700 1 -- 172.21.15.27:0/1282507796 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f81a0027830 con 0x7f81b40ca7b0 2024-01-11T08:23:02.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.305+0000 7f81b0ff9700 1 --2- 172.21.15.27:0/1282507796 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8198066f30 0x7f81980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:02.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.305+0000 7f81b0ff9700 1 -- 172.21.15.27:0/1282507796 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f81a0022070 con 0x7f81b40ca7b0 2024-01-11T08:23:02.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.305+0000 7f81b8d9b700 1 --2- 172.21.15.27:0/1282507796 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8198066f30 0x7f81980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.306+0000 7f81b8d9b700 1 --2- 172.21.15.27:0/1282507796 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8198066f30 0x7f81980693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f81a80017a0 tx=0x7f81a8009040 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:02.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.309+0000 7f81b0ff9700 1 -- 172.21.15.27:0/1282507796 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f81a0053540 con 0x7f81b40ca7b0 2024-01-11T08:23:02.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.516+0000 7f81bafff700 1 -- 172.21.15.27:0/1282507796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f81b40718d0 con 0x7f81b40ca7b0 2024-01-11T08:23:02.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.517+0000 7f81b0ff9700 1 -- 172.21.15.27:0/1282507796 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v55) v1 ==== 84+0+65 (secure 0 0 0) 0x7f81a002c090 con 0x7f81b40ca7b0 2024-01-11T08:23:02.516 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:23:02.516 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T08:23:02.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:02.516 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-11T08:23:02.516 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-11T08:23:02.516 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:02.517 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T08:23:02.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.519+0000 7f81bafff700 1 -- 172.21.15.27:0/1282507796 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8198066f30 msgr2=0x7f81980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:02.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.519+0000 7f81bafff700 1 --2- 172.21.15.27:0/1282507796 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8198066f30 0x7f81980693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f81a80017a0 tx=0x7f81a8009040 comp rx=0 tx=0).stop 2024-01-11T08:23:02.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.519+0000 7f81bafff700 1 -- 172.21.15.27:0/1282507796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81b40ca7b0 msgr2=0x7f81b415fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:02.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.519+0000 7f81bafff700 1 --2- 172.21.15.27:0/1282507796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81b40ca7b0 0x7f81b415fef0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f81a00093f0 tx=0x7f81a0016000 comp rx=0 tx=0).stop 2024-01-11T08:23:02.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.519+0000 7f81bafff700 1 -- 172.21.15.27:0/1282507796 shutdown_connections 2024-01-11T08:23:02.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.519+0000 7f81bafff700 1 --2- 172.21.15.27:0/1282507796 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8198066f30 0x7f81980693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:02.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.519+0000 7f81bafff700 1 --2- 172.21.15.27:0/1282507796 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f81b4159e80 0x7f81b4160430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:02.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.519+0000 7f81bafff700 1 --2- 172.21.15.27:0/1282507796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81b40ca3e0 0x7f81b40b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:02.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.519+0000 7f81bafff700 1 --2- 172.21.15.27:0/1282507796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81b40ca7b0 0x7f81b415fef0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:02.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.519+0000 7f81bafff700 1 -- 172.21.15.27:0/1282507796 >> 172.21.15.27:0/1282507796 conn(0x7f81b40ce3e0 msgr2=0x7f81b40693b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:02.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.519+0000 7f81bafff700 1 -- 172.21.15.27:0/1282507796 shutdown_connections 2024-01-11T08:23:02.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.519+0000 7f81bafff700 1 -- 172.21.15.27:0/1282507796 wait complete. 2024-01-11T08:23:02.536 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-11T08:23:02.536 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-11T08:23:02.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.739+0000 7f54063ff700 1 Processor -- start 2024-01-11T08:23:02.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.739+0000 7f54063ff700 1 -- start start 2024-01-11T08:23:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.741+0000 7f54063ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f540015a7a0 0x7f540015c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.741+0000 7f54063ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f54000685e0 0x7f5400068a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.741+0000 7f54063ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540015c4b0 con 0x7f5400159e60 2024-01-11T08:23:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.741+0000 7f54063ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5400068fc0 con 0x7f54000685e0 2024-01-11T08:23:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.741+0000 7f54063ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540015b5d0 con 0x7f540015a7a0 2024-01-11T08:23:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.741+0000 7f53fffff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f540015a7a0 0x7f540015c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.742+0000 7f53fffff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f540015a7a0 0x7f540015c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:33170/0 (socket says 172.21.15.27:33170) 2024-01-11T08:23:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.742+0000 7f53fffff700 1 -- 172.21.15.27:0/3555938552 learned_addr learned my addr 172.21.15.27:0/3555938552 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.742+0000 7f540499c700 1 --2- 172.21.15.27:0/3555938552 >> v2:172.21.15.77:3300/0 conn(0x7f54000685e0 0x7f5400068a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:02.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.742+0000 7f53fffff700 1 -- 172.21.15.27:0/3555938552 >> v2:172.21.15.77:3300/0 conn(0x7f54000685e0 msgr2=0x7f5400068a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:02.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.742+0000 7f53fffff700 1 --2- 172.21.15.27:0/3555938552 >> v2:172.21.15.77:3300/0 conn(0x7f54000685e0 0x7f5400068a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:02.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.742+0000 7f53fffff700 1 -- 172.21.15.27:0/3555938552 >> v1:172.21.15.77:6789/0 conn(0x7f5400159e60 legacy=0x7f540015aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:02.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.742+0000 7f53fffff700 1 -- 172.21.15.27:0/3555938552 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540015b8a0 con 0x7f540015a7a0 2024-01-11T08:23:02.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.742+0000 7f53fffff700 1 --2- 172.21.15.27:0/3555938552 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f540015a7a0 0x7f540015c940 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f53f00099b0 tx=0x7f53f0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bb2895b3de6998c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:02.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.743+0000 7f53feffd700 1 -- 172.21.15.27:0/3555938552 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f0009ea0 con 0x7f540015a7a0 2024-01-11T08:23:02.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.743+0000 7f53feffd700 1 -- 172.21.15.27:0/3555938552 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f000b860 con 0x7f540015a7a0 2024-01-11T08:23:02.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.743+0000 7f53feffd700 1 -- 172.21.15.27:0/3555938552 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f000bc00 con 0x7f540015a7a0 2024-01-11T08:23:02.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.743+0000 7f54063ff700 1 -- 172.21.15.27:0/3555938552 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f540015a7a0 msgr2=0x7f540015c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:02.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.743+0000 7f54063ff700 1 --2- 172.21.15.27:0/3555938552 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f540015a7a0 0x7f540015c940 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f53f00099b0 tx=0x7f53f0009250 comp rx=0 tx=0).stop 2024-01-11T08:23:02.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.743+0000 7f54063ff700 1 -- 172.21.15.27:0/3555938552 shutdown_connections 2024-01-11T08:23:02.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.743+0000 7f54063ff700 1 --2- 172.21.15.27:0/3555938552 >> v2:172.21.15.77:3300/0 conn(0x7f54000685e0 0x7f5400068a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:02.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.743+0000 7f54063ff700 1 --2- 172.21.15.27:0/3555938552 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f540015a7a0 0x7f540015c940 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:02.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.743+0000 7f54063ff700 1 -- 172.21.15.27:0/3555938552 >> 172.21.15.27:0/3555938552 conn(0x7f540006ed40 msgr2=0x7f540006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:02.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.743+0000 7f54063ff700 1 -- 172.21.15.27:0/3555938552 shutdown_connections 2024-01-11T08:23:02.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.743+0000 7f54063ff700 1 -- 172.21.15.27:0/3555938552 wait complete. 2024-01-11T08:23:02.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.745+0000 7f54063ff700 1 Processor -- start 2024-01-11T08:23:02.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.745+0000 7f54063ff700 1 -- start start 2024-01-11T08:23:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.746+0000 7f54063ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54000685e0 0x7f540014e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.746+0000 7f54063ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f540015a7a0 0x7f5400154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.746+0000 7f54063ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5400154e70 0x7f54001525b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.746+0000 7f54063ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540014e9e0 con 0x7f54000685e0 2024-01-11T08:23:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.746+0000 7f54063ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54001553d0 con 0x7f540015a7a0 2024-01-11T08:23:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.746+0000 7f54063ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54001556b0 con 0x7f5400154e70 2024-01-11T08:23:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.746+0000 7f53fffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54000685e0 0x7f540014e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.746+0000 7f53fffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54000685e0 0x7f540014e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49718/0 (socket says 172.21.15.27:49718) 2024-01-11T08:23:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.746+0000 7f53fffff700 1 -- 172.21.15.27:0/2577012076 learned_addr learned my addr 172.21.15.27:0/2577012076 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.747+0000 7f540499c700 1 --2- 172.21.15.27:0/2577012076 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5400154e70 0x7f54001525b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.747+0000 7f53ff7fe700 1 --2- 172.21.15.27:0/2577012076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f540015a7a0 0x7f5400154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.747+0000 7f53fffff700 1 -- 172.21.15.27:0/2577012076 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5400154e70 msgr2=0x7f54001525b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.747+0000 7f53fffff700 1 --2- 172.21.15.27:0/2577012076 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5400154e70 0x7f54001525b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.747+0000 7f53fffff700 1 -- 172.21.15.27:0/2577012076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f540015a7a0 msgr2=0x7f5400154930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.747+0000 7f53fffff700 1 --2- 172.21.15.27:0/2577012076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f540015a7a0 0x7f5400154930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.747+0000 7f53fffff700 1 -- 172.21.15.27:0/2577012076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f00094f0 con 0x7f54000685e0 2024-01-11T08:23:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.747+0000 7f53fffff700 1 --2- 172.21.15.27:0/2577012076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54000685e0 0x7f540014e4a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f540015b710 tx=0x7f53f0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.747+0000 7f53e7fff700 1 -- 172.21.15.27:0/2577012076 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f0000d10 con 0x7f54000685e0 2024-01-11T08:23:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.748+0000 7f53e7fff700 1 -- 172.21.15.27:0/2577012076 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f000bbe0 con 0x7f54000685e0 2024-01-11T08:23:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.748+0000 7f53e7fff700 1 -- 172.21.15.27:0/2577012076 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f0011670 con 0x7f54000685e0 2024-01-11T08:23:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.748+0000 7f54063ff700 1 -- 172.21.15.27:0/2577012076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5400150e90 con 0x7f54000685e0 2024-01-11T08:23:02.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.748+0000 7f54063ff700 1 -- 172.21.15.27:0/2577012076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5400157280 con 0x7f54000685e0 2024-01-11T08:23:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.749+0000 7f54063ff700 1 -- 172.21.15.27:0/2577012076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54001513b0 con 0x7f54000685e0 2024-01-11T08:23:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.750+0000 7f53e7fff700 1 -- 172.21.15.27:0/2577012076 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f53f0011a60 con 0x7f54000685e0 2024-01-11T08:23:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.751+0000 7f53e7fff700 1 --2- 172.21.15.27:0/2577012076 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f53ec066f30 0x7f53ec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.751+0000 7f53e7fff700 1 -- 172.21.15.27:0/2577012076 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f53f0085a50 con 0x7f54000685e0 2024-01-11T08:23:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.751+0000 7f53ff7fe700 1 --2- 172.21.15.27:0/2577012076 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f53ec066f30 0x7f53ec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.752+0000 7f53ff7fe700 1 --2- 172.21.15.27:0/2577012076 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f53ec066f30 0x7f53ec0693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5400152320 tx=0x7f53f4007ec0 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:02.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.755+0000 7f53e7fff700 1 -- 172.21.15.27:0/2577012076 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f54001513b0 con 0x7f54000685e0 2024-01-11T08:23:02.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:02.959+0000 7f54063ff700 1 -- 172.21.15.27:0/2577012076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f54001513b0 con 0x7f54000685e0 2024-01-11T08:23:03.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:03.829+0000 7f53e7fff700 1 -- 172.21.15.27:0/2577012076 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v56) v1 ==== 138+0+0 (secure 0 0 0) 0x7f54001513b0 con 0x7f54000685e0 2024-01-11T08:23:03.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:03.833+0000 7f54063ff700 1 -- 172.21.15.27:0/2577012076 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f53ec066f30 msgr2=0x7f53ec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:03.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:03.833+0000 7f54063ff700 1 --2- 172.21.15.27:0/2577012076 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f53ec066f30 0x7f53ec0693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5400152320 tx=0x7f53f4007ec0 comp rx=0 tx=0).stop 2024-01-11T08:23:03.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:03.833+0000 7f54063ff700 1 -- 172.21.15.27:0/2577012076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54000685e0 msgr2=0x7f540014e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:03.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:03.833+0000 7f54063ff700 1 --2- 172.21.15.27:0/2577012076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54000685e0 0x7f540014e4a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f540015b710 tx=0x7f53f0017000 comp rx=0 tx=0).stop 2024-01-11T08:23:03.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:03.833+0000 7f54063ff700 1 -- 172.21.15.27:0/2577012076 shutdown_connections 2024-01-11T08:23:03.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:03.833+0000 7f54063ff700 1 --2- 172.21.15.27:0/2577012076 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f53ec066f30 0x7f53ec0693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:03.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:03.833+0000 7f54063ff700 1 --2- 172.21.15.27:0/2577012076 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5400154e70 0x7f54001525b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:03.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:03.833+0000 7f54063ff700 1 --2- 172.21.15.27:0/2577012076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54000685e0 0x7f540014e4a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:03.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:03.833+0000 7f54063ff700 1 --2- 172.21.15.27:0/2577012076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f540015a7a0 0x7f5400154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:03.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:03.833+0000 7f54063ff700 1 -- 172.21.15.27:0/2577012076 >> 172.21.15.27:0/2577012076 conn(0x7f540006ed40 msgr2=0x7f54000d0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:03.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:03.833+0000 7f54063ff700 1 -- 172.21.15.27:0/2577012076 shutdown_connections 2024-01-11T08:23:03.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:03.833+0000 7f54063ff700 1 -- 172.21.15.27:0/2577012076 wait complete. 2024-01-11T08:23:03.836 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-11T08:23:03.857 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-11T08:23:04.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.068+0000 7f062f3ff700 1 Processor -- start 2024-01-11T08:23:04.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.068+0000 7f062f3ff700 1 -- start start 2024-01-11T08:23:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.069+0000 7f062f3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f06280ca670 0x7f0628068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.069+0000 7f062f3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0628068b60 0x7f06280aa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.069+0000 7f062f3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f062815da10 con 0x7f06281596c0 2024-01-11T08:23:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.069+0000 7f062f3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0628068fc0 con 0x7f06280ca670 2024-01-11T08:23:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.069+0000 7f062f3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06280a9cb0 con 0x7f0628068b60 2024-01-11T08:23:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.070+0000 7f062c99a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f06280ca670 0x7f0628068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.070+0000 7f062c99a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f06280ca670 0x7f0628068620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33572/0 (socket says 172.21.15.27:33572) 2024-01-11T08:23:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.070+0000 7f062c99a700 1 -- 172.21.15.27:0/2077869697 learned_addr learned my addr 172.21.15.27:0/2077869697 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.070+0000 7f062d19b700 1 --1- 172.21.15.27:0/2077869697 >> v1:172.21.15.27:6789/0 conn(0x7f06281596c0 0x7f062815d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37766/0 (socket says 172.21.15.27:37766) 2024-01-11T08:23:04.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.070+0000 7f062d99c700 1 --2- 172.21.15.27:0/2077869697 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0628068b60 0x7f06280aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:04.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.070+0000 7f062c99a700 1 -- 172.21.15.27:0/2077869697 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0628068b60 msgr2=0x7f06280aa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:04.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.070+0000 7f062c99a700 1 --2- 172.21.15.27:0/2077869697 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0628068b60 0x7f06280aa140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:04.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.070+0000 7f062c99a700 1 -- 172.21.15.27:0/2077869697 >> v1:172.21.15.27:6789/0 conn(0x7f06281596c0 legacy=0x7f062815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:04.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.070+0000 7f062c99a700 1 -- 172.21.15.27:0/2077869697 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062815e210 con 0x7f06280ca670 2024-01-11T08:23:04.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.070+0000 7f062d99c700 1 --2- 172.21.15.27:0/2077869697 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0628068b60 0x7f06280aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:23:04.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.071+0000 7f062c99a700 1 --2- 172.21.15.27:0/2077869697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06280ca670 0x7f0628068620 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f062815e400 tx=0x7f0618009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4814aa8ee941cadf server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:04.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.071+0000 7f061ffff700 1 -- 172.21.15.27:0/2077869697 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061800b860 con 0x7f06280ca670 2024-01-11T08:23:04.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.071+0000 7f061ffff700 1 -- 172.21.15.27:0/2077869697 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061800b9c0 con 0x7f06280ca670 2024-01-11T08:23:04.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.071+0000 7f061ffff700 1 -- 172.21.15.27:0/2077869697 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061800bd80 con 0x7f06280ca670 2024-01-11T08:23:04.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.071+0000 7f062f3ff700 1 -- 172.21.15.27:0/2077869697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06280ca670 msgr2=0x7f0628068620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:04.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.071+0000 7f062f3ff700 1 --2- 172.21.15.27:0/2077869697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06280ca670 0x7f0628068620 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f062815e400 tx=0x7f0618009250 comp rx=0 tx=0).stop 2024-01-11T08:23:04.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.071+0000 7f062f3ff700 1 -- 172.21.15.27:0/2077869697 shutdown_connections 2024-01-11T08:23:04.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.071+0000 7f062f3ff700 1 --2- 172.21.15.27:0/2077869697 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0628068b60 0x7f06280aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:04.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.071+0000 7f062f3ff700 1 --2- 172.21.15.27:0/2077869697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06280ca670 0x7f0628068620 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:04.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.071+0000 7f062f3ff700 1 -- 172.21.15.27:0/2077869697 >> 172.21.15.27:0/2077869697 conn(0x7f06280c9f70 msgr2=0x7f062806deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.072+0000 7f062f3ff700 1 -- 172.21.15.27:0/2077869697 shutdown_connections 2024-01-11T08:23:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.072+0000 7f062f3ff700 1 -- 172.21.15.27:0/2077869697 wait complete. 2024-01-11T08:23:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.073+0000 7f062f3ff700 1 Processor -- start 2024-01-11T08:23:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.073+0000 7f062f3ff700 1 -- start start 2024-01-11T08:23:04.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.074+0000 7f062f3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0628068b60 0x7f062816af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:04.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.074+0000 7f062f3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06280ca670 0x7f0628169d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:04.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.074+0000 7f062f3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f06281596c0 0x7f06280b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:04.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.074+0000 7f062f3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06280b12e0 con 0x7f0628068b60 2024-01-11T08:23:04.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.074+0000 7f062f3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06280b0240 con 0x7f06280ca670 2024-01-11T08:23:04.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.074+0000 7f062f3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06280b0520 con 0x7f06281596c0 2024-01-11T08:23:04.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.074+0000 7f062d19b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0628068b60 0x7f062816af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:04.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.074+0000 7f062d19b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0628068b60 0x7f062816af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33574/0 (socket says 172.21.15.27:33574) 2024-01-11T08:23:04.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.074+0000 7f062d19b700 1 -- 172.21.15.27:0/3321945111 learned_addr learned my addr 172.21.15.27:0/3321945111 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:04.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.074+0000 7f062d99c700 1 --2- 172.21.15.27:0/3321945111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f06281596c0 0x7f06280b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:04.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.074+0000 7f062c99a700 1 --2- 172.21.15.27:0/3321945111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06280ca670 0x7f0628169d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:04.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.075+0000 7f062d19b700 1 -- 172.21.15.27:0/3321945111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f06281596c0 msgr2=0x7f06280b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:04.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.075+0000 7f062d19b700 1 --2- 172.21.15.27:0/3321945111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f06281596c0 0x7f06280b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:04.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.075+0000 7f062d19b700 1 -- 172.21.15.27:0/3321945111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06280ca670 msgr2=0x7f0628169d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:04.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.075+0000 7f062d19b700 1 --2- 172.21.15.27:0/3321945111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06280ca670 0x7f0628169d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:04.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.075+0000 7f062d19b700 1 -- 172.21.15.27:0/3321945111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06180094f0 con 0x7f0628068b60 2024-01-11T08:23:04.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.075+0000 7f062d19b700 1 --2- 172.21.15.27:0/3321945111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0628068b60 0x7f062816af00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f062400c560 tx=0x7f0624007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:04.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.075+0000 7f061d7fa700 1 -- 172.21.15.27:0/3321945111 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0624005a70 con 0x7f0628068b60 2024-01-11T08:23:04.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.075+0000 7f061d7fa700 1 -- 172.21.15.27:0/3321945111 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0624005bd0 con 0x7f0628068b60 2024-01-11T08:23:04.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.075+0000 7f062f3ff700 1 -- 172.21.15.27:0/3321945111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06280b0800 con 0x7f0628068b60 2024-01-11T08:23:04.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.075+0000 7f061d7fa700 1 -- 172.21.15.27:0/3321945111 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06240070d0 con 0x7f0628068b60 2024-01-11T08:23:04.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.076+0000 7f062f3ff700 1 -- 172.21.15.27:0/3321945111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06280b0dc0 con 0x7f0628068b60 2024-01-11T08:23:04.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.076+0000 7f062f3ff700 1 -- 172.21.15.27:0/3321945111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f062815aa70 con 0x7f0628068b60 2024-01-11T08:23:04.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.078+0000 7f061d7fa700 1 -- 172.21.15.27:0/3321945111 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0624007480 con 0x7f0628068b60 2024-01-11T08:23:04.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.079+0000 7f061d7fa700 1 --2- 172.21.15.27:0/3321945111 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f060c066f30 0x7f060c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:04.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.079+0000 7f061d7fa700 1 -- 172.21.15.27:0/3321945111 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5223+0+0 (secure 0 0 0) 0x7f06240835f0 con 0x7f0628068b60 2024-01-11T08:23:04.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.079+0000 7f062c99a700 1 --2- 172.21.15.27:0/3321945111 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f060c066f30 0x7f060c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:04.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.080+0000 7f062c99a700 1 --2- 172.21.15.27:0/3321945111 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f060c066f30 0x7f060c0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f062815e400 tx=0x7f0618016000 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.084+0000 7f061d7fa700 1 -- 172.21.15.27:0/3321945111 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f06240509a0 con 0x7f0628068b60 2024-01-11T08:23:04.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.285+0000 7f062f3ff700 1 -- 172.21.15.27:0/3321945111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f06280b0fe0 con 0x7f0628068b60 2024-01-11T08:23:04.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.837+0000 7f061d7fa700 1 -- 172.21.15.27:0/3321945111 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v57) v1 ==== 170+0+0 (secure 0 0 0) 0x7f06240170d0 con 0x7f0628068b60 2024-01-11T08:23:04.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.843+0000 7f062f3ff700 1 -- 172.21.15.27:0/3321945111 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f060c066f30 msgr2=0x7f060c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:04.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.843+0000 7f062f3ff700 1 --2- 172.21.15.27:0/3321945111 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f060c066f30 0x7f060c0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f062815e400 tx=0x7f0618016000 comp rx=0 tx=0).stop 2024-01-11T08:23:04.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.843+0000 7f062f3ff700 1 -- 172.21.15.27:0/3321945111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0628068b60 msgr2=0x7f062816af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:04.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.843+0000 7f062f3ff700 1 --2- 172.21.15.27:0/3321945111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0628068b60 0x7f062816af00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f062400c560 tx=0x7f0624007f70 comp rx=0 tx=0).stop 2024-01-11T08:23:04.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.843+0000 7f062f3ff700 1 -- 172.21.15.27:0/3321945111 shutdown_connections 2024-01-11T08:23:04.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.843+0000 7f062f3ff700 1 --2- 172.21.15.27:0/3321945111 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f060c066f30 0x7f060c0693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:04.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.843+0000 7f062f3ff700 1 --2- 172.21.15.27:0/3321945111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f06281596c0 0x7f06280b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:04.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.843+0000 7f062f3ff700 1 --2- 172.21.15.27:0/3321945111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0628068b60 0x7f062816af00 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:04.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.843+0000 7f062f3ff700 1 --2- 172.21.15.27:0/3321945111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06280ca670 0x7f0628169d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:04.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.843+0000 7f062f3ff700 1 -- 172.21.15.27:0/3321945111 >> 172.21.15.27:0/3321945111 conn(0x7f06280c9f70 msgr2=0x7f06280c2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:04.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.844+0000 7f062f3ff700 1 -- 172.21.15.27:0/3321945111 shutdown_connections 2024-01-11T08:23:04.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:04.844+0000 7f062f3ff700 1 -- 172.21.15.27:0/3321945111 wait complete. 2024-01-11T08:23:04.847 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-11T08:23:04.876 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-11T08:23:05.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.081+0000 7f316e7ff700 1 Processor -- start 2024-01-11T08:23:05.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.081+0000 7f316e7ff700 1 -- start start 2024-01-11T08:23:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.083+0000 7f316e7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3168068d30 0x7f31680c8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.083+0000 7f316e7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31680c80a0 con 0x7f31681596f0 2024-01-11T08:23:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.083+0000 7f316e7ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31680cbac0 con 0x7f3168159ac0 2024-01-11T08:23:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.083+0000 7f316e7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31680a9fb0 con 0x7f3168068d30 2024-01-11T08:23:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.083+0000 7f3167fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f31681596f0 0x7f31680a98a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37778/0 (socket says 172.21.15.27:37778) 2024-01-11T08:23:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.083+0000 7f3167fff700 1 -- 172.21.15.27:0/1946402069 learned_addr learned my addr 172.21.15.27:0/1946402069 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:23:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.083+0000 7f316cd9c700 1 --2- 172.21.15.27:0/1946402069 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3168068d30 0x7f31680c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:05.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.083+0000 7f3166ffd700 1 -- 172.21.15.27:0/1946402069 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930993936 0 0) 0x7f31680c80a0 con 0x7f31681596f0 2024-01-11T08:23:05.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.084+0000 7f3166ffd700 1 -- 172.21.15.27:0/1946402069 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3150003570 con 0x7f31681596f0 2024-01-11T08:23:05.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.084+0000 7f3166ffd700 1 -- 172.21.15.27:0/1946402069 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222747758 0 0) 0x7f31680cbac0 con 0x7f3168159ac0 2024-01-11T08:23:05.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.084+0000 7f3166ffd700 1 -- 172.21.15.27:0/1946402069 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31680c80a0 con 0x7f3168159ac0 2024-01-11T08:23:05.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.084+0000 7f316cd9c700 1 -- 172.21.15.27:0/1946402069 >> v1:172.21.15.77:6789/0 conn(0x7f3168159ac0 legacy=0x7f3168068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.084+0000 7f316cd9c700 1 -- 172.21.15.27:0/1946402069 >> v1:172.21.15.27:6789/0 conn(0x7f31681596f0 legacy=0x7f31680a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.084+0000 7f3166ffd700 1 -- 172.21.15.27:0/1946402069 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3487620741 0 0) 0x7f3150003570 con 0x7f31681596f0 2024-01-11T08:23:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.084+0000 7f316cd9c700 1 -- 172.21.15.27:0/1946402069 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31680c7490 con 0x7f3168068d30 2024-01-11T08:23:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.084+0000 7f316cd9c700 1 --2- 172.21.15.27:0/1946402069 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3168068d30 0x7f31680c8530 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3160009e90 tx=0x7f3160009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff3245d67e48cb68 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.084+0000 7f3166ffd700 1 -- 172.21.15.27:0/1946402069 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316000e070 con 0x7f3168068d30 2024-01-11T08:23:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.085+0000 7f3166ffd700 1 -- 172.21.15.27:0/1946402069 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f316000b870 con 0x7f3168068d30 2024-01-11T08:23:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.085+0000 7f3166ffd700 1 -- 172.21.15.27:0/1946402069 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3158001470 con 0x7f3168068d30 2024-01-11T08:23:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.085+0000 7f316e7ff700 1 -- 172.21.15.27:0/1946402069 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3168068d30 msgr2=0x7f31680c8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.085+0000 7f316e7ff700 1 --2- 172.21.15.27:0/1946402069 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3168068d30 0x7f31680c8530 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3160009e90 tx=0x7f3160009780 comp rx=0 tx=0).stop 2024-01-11T08:23:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.085+0000 7f316e7ff700 1 -- 172.21.15.27:0/1946402069 shutdown_connections 2024-01-11T08:23:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.085+0000 7f316e7ff700 1 --2- 172.21.15.27:0/1946402069 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3168068d30 0x7f31680c8530 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.085+0000 7f316e7ff700 1 -- 172.21.15.27:0/1946402069 >> 172.21.15.27:0/1946402069 conn(0x7f31680c9fb0 msgr2=0x7f316806ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.085+0000 7f316e7ff700 1 -- 172.21.15.27:0/1946402069 shutdown_connections 2024-01-11T08:23:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.085+0000 7f316e7ff700 1 -- 172.21.15.27:0/1946402069 wait complete. 2024-01-11T08:23:05.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.087+0000 7f316e7ff700 1 Processor -- start 2024-01-11T08:23:05.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.087+0000 7f316e7ff700 1 -- start start 2024-01-11T08:23:05.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.088+0000 7f316e7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3168068d30 0x7f316816da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:05.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.088+0000 7f316e7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31681596f0 0x7f316806ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.088+0000 7f316e7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3168159ac0 0x7f31680c3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.088+0000 7f316e7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316816dfb0 con 0x7f31681596f0 2024-01-11T08:23:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.088+0000 7f316e7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31680af8b0 con 0x7f3168159ac0 2024-01-11T08:23:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.088+0000 7f316e7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31680afb00 con 0x7f3168068d30 2024-01-11T08:23:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.088+0000 7f31677fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31681596f0 0x7f316806ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.088+0000 7f31677fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31681596f0 0x7f316806ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33576/0 (socket says 172.21.15.27:33576) 2024-01-11T08:23:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.088+0000 7f31677fe700 1 -- 172.21.15.27:0/2530996396 learned_addr learned my addr 172.21.15.27:0/2530996396 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.088+0000 7f3167fff700 1 --2- 172.21.15.27:0/2530996396 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3168068d30 0x7f316816da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.088+0000 7f316cd9c700 1 --2- 172.21.15.27:0/2530996396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3168159ac0 0x7f31680c3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.089+0000 7f31677fe700 1 -- 172.21.15.27:0/2530996396 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3168068d30 msgr2=0x7f316816da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.089+0000 7f31677fe700 1 --2- 172.21.15.27:0/2530996396 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3168068d30 0x7f316816da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.089+0000 7f31677fe700 1 -- 172.21.15.27:0/2530996396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3168159ac0 msgr2=0x7f31680c3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.089+0000 7f31677fe700 1 --2- 172.21.15.27:0/2530996396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3168159ac0 0x7f31680c3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.089+0000 7f31677fe700 1 -- 172.21.15.27:0/2530996396 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3160009a20 con 0x7f31681596f0 2024-01-11T08:23:05.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.089+0000 7f31677fe700 1 --2- 172.21.15.27:0/2530996396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31681596f0 0x7f316806ae90 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f315c00c590 tx=0x7f315c007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:05.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.089+0000 7f314ffff700 1 -- 172.21.15.27:0/2530996396 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c010510 con 0x7f31681596f0 2024-01-11T08:23:05.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.089+0000 7f314ffff700 1 -- 172.21.15.27:0/2530996396 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315c010690 con 0x7f31681596f0 2024-01-11T08:23:05.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.089+0000 7f314ffff700 1 -- 172.21.15.27:0/2530996396 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c010a50 con 0x7f31681596f0 2024-01-11T08:23:05.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.089+0000 7f316e7ff700 1 -- 172.21.15.27:0/2530996396 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31680afde0 con 0x7f31681596f0 2024-01-11T08:23:05.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.089+0000 7f316e7ff700 1 -- 172.21.15.27:0/2530996396 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31680b0680 con 0x7f31681596f0 2024-01-11T08:23:05.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.090+0000 7f316e7ff700 1 -- 172.21.15.27:0/2530996396 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31680b01e0 con 0x7f31681596f0 2024-01-11T08:23:05.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.092+0000 7f314ffff700 1 -- 172.21.15.27:0/2530996396 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f315c01b400 con 0x7f31681596f0 2024-01-11T08:23:05.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.093+0000 7f314ffff700 1 --2- 172.21.15.27:0/2530996396 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3154066f30 0x7f31540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:05.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.093+0000 7f314ffff700 1 -- 172.21.15.27:0/2530996396 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5605+0+0 (secure 0 0 0) 0x7f315c0841e0 con 0x7f31681596f0 2024-01-11T08:23:05.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.093+0000 7f3167fff700 1 --2- 172.21.15.27:0/2530996396 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3154066f30 0x7f31540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:05.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.094+0000 7f3167fff700 1 --2- 172.21.15.27:0/2530996396 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3154066f30 0x7f31540693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f31580015a0 tx=0x7f3158000bc0 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:05.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.098+0000 7f314ffff700 1 -- 172.21.15.27:0/2530996396 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f315c051410 con 0x7f31681596f0 2024-01-11T08:23:05.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.343+0000 7f316e7ff700 1 -- 172.21.15.27:0/2530996396 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f31680b0870 con 0x7f31681596f0 2024-01-11T08:23:05.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.841+0000 7f314ffff700 1 -- 172.21.15.27:0/2530996396 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v21) v1 ==== 172+0+0 (secure 0 0 0) 0x7f315c054a30 con 0x7f31681596f0 2024-01-11T08:23:05.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.848+0000 7f316e7ff700 1 -- 172.21.15.27:0/2530996396 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3154066f30 msgr2=0x7f31540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:05.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.848+0000 7f316e7ff700 1 --2- 172.21.15.27:0/2530996396 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3154066f30 0x7f31540693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f31580015a0 tx=0x7f3158000bc0 comp rx=0 tx=0).stop 2024-01-11T08:23:05.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.848+0000 7f316e7ff700 1 -- 172.21.15.27:0/2530996396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31681596f0 msgr2=0x7f316806ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:05.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.848+0000 7f316e7ff700 1 --2- 172.21.15.27:0/2530996396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31681596f0 0x7f316806ae90 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f315c00c590 tx=0x7f315c007dd0 comp rx=0 tx=0).stop 2024-01-11T08:23:05.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.849+0000 7f316e7ff700 1 -- 172.21.15.27:0/2530996396 shutdown_connections 2024-01-11T08:23:05.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.849+0000 7f316e7ff700 1 --2- 172.21.15.27:0/2530996396 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3154066f30 0x7f31540693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:05.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.849+0000 7f316e7ff700 1 --2- 172.21.15.27:0/2530996396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3168159ac0 0x7f31680c3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:05.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.849+0000 7f316e7ff700 1 --2- 172.21.15.27:0/2530996396 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3168068d30 0x7f316816da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:05.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.849+0000 7f316e7ff700 1 --2- 172.21.15.27:0/2530996396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31681596f0 0x7f316806ae90 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:05.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.849+0000 7f316e7ff700 1 -- 172.21.15.27:0/2530996396 >> 172.21.15.27:0/2530996396 conn(0x7f31680c9fb0 msgr2=0x7f31680c31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:05.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.850+0000 7f316e7ff700 1 -- 172.21.15.27:0/2530996396 shutdown_connections 2024-01-11T08:23:05.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:05.851+0000 7f316e7ff700 1 -- 172.21.15.27:0/2530996396 wait complete. 2024-01-11T08:23:05.853 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 6 and data pool 7 2024-01-11T08:23:05.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:23:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.090+0000 7fdc123ff700 1 Processor -- start 2024-01-11T08:23:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.090+0000 7fdc123ff700 1 -- start start 2024-01-11T08:23:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.091+0000 7fdc123ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdc0c0b03e0 0x7fdc0c0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.091+0000 7fdc123ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc0c0b07b0 0x7fdc0c0b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.091+0000 7fdc123ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc0c0cd730 con 0x7fdc0c159200 2024-01-11T08:23:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.091+0000 7fdc123ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc0c159d30 con 0x7fdc0c0b07b0 2024-01-11T08:23:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.091+0000 7fdc123ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc0c0ccb20 con 0x7fdc0c0b03e0 2024-01-11T08:23:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.091+0000 7fdc0b7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc0c0b07b0 0x7fdc0c0b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:06.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.091+0000 7fdc0b7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc0c0b07b0 0x7fdc0c0b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33582/0 (socket says 172.21.15.27:33582) 2024-01-11T08:23:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.091+0000 7fdc0b7fe700 1 -- 172.21.15.27:0/228670913 learned_addr learned my addr 172.21.15.27:0/228670913 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.091+0000 7fdc1099c700 1 --1- 172.21.15.27:0/228670913 >> v1:172.21.15.27:6789/0 conn(0x7fdc0c159200 0x7fdc0c159620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37780/0 (socket says 172.21.15.27:37780) 2024-01-11T08:23:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.091+0000 7fdc0bfff700 1 --2- 172.21.15.27:0/228670913 >> v2:172.21.15.77:3300/0 conn(0x7fdc0c0b03e0 0x7fdc0c0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.092+0000 7fdc0b7fe700 1 -- 172.21.15.27:0/228670913 >> v2:172.21.15.77:3300/0 conn(0x7fdc0c0b03e0 msgr2=0x7fdc0c0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.092+0000 7fdc0b7fe700 1 --2- 172.21.15.27:0/228670913 >> v2:172.21.15.77:3300/0 conn(0x7fdc0c0b03e0 0x7fdc0c0b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.092+0000 7fdc0b7fe700 1 -- 172.21.15.27:0/228670913 >> v1:172.21.15.27:6789/0 conn(0x7fdc0c159200 legacy=0x7fdc0c159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.092+0000 7fdc0b7fe700 1 -- 172.21.15.27:0/228670913 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0c0cce00 con 0x7fdc0c0b07b0 2024-01-11T08:23:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.092+0000 7fdc0affd700 1 -- 172.21.15.27:0/228670913 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342383478 0 0) 0x7fdc0c0cd730 con 0x7fdc0c159200 2024-01-11T08:23:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.092+0000 7fdc0b7fe700 1 --2- 172.21.15.27:0/228670913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc0c0b07b0 0x7fdc0c0b29c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c089470 tx=0x7fdc00009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b37a820e91f9c8fb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.092+0000 7fdc0affd700 1 -- 172.21.15.27:0/228670913 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc00009e70 con 0x7fdc0c0b07b0 2024-01-11T08:23:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.093+0000 7fdc0affd700 1 -- 172.21.15.27:0/228670913 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0000b860 con 0x7fdc0c0b07b0 2024-01-11T08:23:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.093+0000 7fdc0affd700 1 -- 172.21.15.27:0/228670913 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0000bc00 con 0x7fdc0c0b07b0 2024-01-11T08:23:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.093+0000 7fdc123ff700 1 -- 172.21.15.27:0/228670913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc0c0b07b0 msgr2=0x7fdc0c0b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.093+0000 7fdc123ff700 1 --2- 172.21.15.27:0/228670913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc0c0b07b0 0x7fdc0c0b29c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c089470 tx=0x7fdc00009250 comp rx=0 tx=0).stop 2024-01-11T08:23:06.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.093+0000 7fdc123ff700 1 -- 172.21.15.27:0/228670913 shutdown_connections 2024-01-11T08:23:06.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.093+0000 7fdc123ff700 1 --2- 172.21.15.27:0/228670913 >> v2:172.21.15.77:3300/0 conn(0x7fdc0c0b03e0 0x7fdc0c0b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.093+0000 7fdc123ff700 1 --2- 172.21.15.27:0/228670913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc0c0b07b0 0x7fdc0c0b29c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.093+0000 7fdc123ff700 1 -- 172.21.15.27:0/228670913 >> 172.21.15.27:0/228670913 conn(0x7fdc0c06ed40 msgr2=0x7fdc0c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:06.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.093+0000 7fdc123ff700 1 -- 172.21.15.27:0/228670913 shutdown_connections 2024-01-11T08:23:06.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.093+0000 7fdc123ff700 1 -- 172.21.15.27:0/228670913 wait complete. 2024-01-11T08:23:06.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.095+0000 7fdc123ff700 1 Processor -- start 2024-01-11T08:23:06.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.095+0000 7fdc123ff700 1 -- start start 2024-01-11T08:23:06.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.096+0000 7fdc123ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc0c0b03e0 0x7fdc0c146710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:06.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.096+0000 7fdc123ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc0c0b07b0 0x7fdc0c0c6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:06.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.096+0000 7fdc123ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc0c159200 0x7fdc0c169960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:06.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.096+0000 7fdc123ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c0c98c0 con 0x7fdc0c0b07b0 2024-01-11T08:23:06.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.096+0000 7fdc123ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c169ea0 con 0x7fdc0c159200 2024-01-11T08:23:06.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.096+0000 7fdc123ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c16a140 con 0x7fdc0c0b03e0 2024-01-11T08:23:06.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.096+0000 7fdc0b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc0c0b07b0 0x7fdc0c0c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.096+0000 7fdc0b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc0c0b07b0 0x7fdc0c0c6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33586/0 (socket says 172.21.15.27:33586) 2024-01-11T08:23:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.096+0000 7fdc0b7fe700 1 -- 172.21.15.27:0/2969511982 learned_addr learned my addr 172.21.15.27:0/2969511982 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.096+0000 7fdc0bfff700 1 --2- 172.21.15.27:0/2969511982 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc0c0b03e0 0x7fdc0c146710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.096+0000 7fdc1099c700 1 --2- 172.21.15.27:0/2969511982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc0c159200 0x7fdc0c169960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.097+0000 7fdc0b7fe700 1 -- 172.21.15.27:0/2969511982 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc0c0b03e0 msgr2=0x7fdc0c146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.097+0000 7fdc0b7fe700 1 --2- 172.21.15.27:0/2969511982 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc0c0b03e0 0x7fdc0c146710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.097+0000 7fdc0b7fe700 1 -- 172.21.15.27:0/2969511982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc0c159200 msgr2=0x7fdc0c169960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.097+0000 7fdc0b7fe700 1 --2- 172.21.15.27:0/2969511982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc0c159200 0x7fdc0c169960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.097+0000 7fdc0b7fe700 1 -- 172.21.15.27:0/2969511982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc000094f0 con 0x7fdc0c0b07b0 2024-01-11T08:23:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.097+0000 7fdc0b7fe700 1 --2- 172.21.15.27:0/2969511982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc0c0b07b0 0x7fdc0c0c6890 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fdc000094c0 tx=0x7fdc00017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.097+0000 7fdbf3fff700 1 -- 172.21.15.27:0/2969511982 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc00000c90 con 0x7fdc0c0b07b0 2024-01-11T08:23:06.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.097+0000 7fdbf3fff700 1 -- 172.21.15.27:0/2969511982 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0000b9d0 con 0x7fdc0c0b07b0 2024-01-11T08:23:06.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.097+0000 7fdc123ff700 1 -- 172.21.15.27:0/2969511982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc0c16a420 con 0x7fdc0c0b07b0 2024-01-11T08:23:06.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.097+0000 7fdbf3fff700 1 -- 172.21.15.27:0/2969511982 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc00025420 con 0x7fdc0c0b07b0 2024-01-11T08:23:06.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.097+0000 7fdc123ff700 1 -- 172.21.15.27:0/2969511982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc0c0c9500 con 0x7fdc0c0b07b0 2024-01-11T08:23:06.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.098+0000 7fdc123ff700 1 -- 172.21.15.27:0/2969511982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc0c0c9690 con 0x7fdc0c0b07b0 2024-01-11T08:23:06.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.099+0000 7fdbf3fff700 1 -- 172.21.15.27:0/2969511982 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fdc00023070 con 0x7fdc0c0b07b0 2024-01-11T08:23:06.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.100+0000 7fdbf3fff700 1 --2- 172.21.15.27:0/2969511982 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdbf8066fb0 0x7fdbf8069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:06.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.100+0000 7fdbf3fff700 1 -- 172.21.15.27:0/2969511982 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fdc00085ac0 con 0x7fdc0c0b07b0 2024-01-11T08:23:06.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.101+0000 7fdc0bfff700 1 --2- 172.21.15.27:0/2969511982 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdbf8066fb0 0x7fdbf8069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:06.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.101+0000 7fdc0bfff700 1 --2- 172.21.15.27:0/2969511982 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdbf8066fb0 0x7fdbf8069460 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009db0 tx=0x7fdbfc0091a0 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:06.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.105+0000 7fdbf3fff700 1 -- 172.21.15.27:0/2969511982 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdc0c0c9690 con 0x7fdc0c0b07b0 2024-01-11T08:23:06.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.308+0000 7fdc123ff700 1 -- 172.21.15.27:0/2969511982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc0c0c9690 con 0x7fdc0c0b07b0 2024-01-11T08:23:06.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.309+0000 7fdbf3fff700 1 -- 172.21.15.27:0/2969511982 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+15315 (secure 0 0 0) 0x7fdc0c0c9690 con 0x7fdc0c0b07b0 2024-01-11T08:23:06.308 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:23:06.309 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":58,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:23:05.344378+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:23:03.012171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-11T08:23:04.337714+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:23:06.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.311+0000 7fdc123ff700 1 -- 172.21.15.27:0/2969511982 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdbf8066fb0 msgr2=0x7fdbf8069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:06.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.311+0000 7fdc123ff700 1 --2- 172.21.15.27:0/2969511982 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdbf8066fb0 0x7fdbf8069460 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009db0 tx=0x7fdbfc0091a0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.311+0000 7fdc123ff700 1 -- 172.21.15.27:0/2969511982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc0c0b07b0 msgr2=0x7fdc0c0c6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:06.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.311+0000 7fdc123ff700 1 --2- 172.21.15.27:0/2969511982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc0c0b07b0 0x7fdc0c0c6890 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fdc000094c0 tx=0x7fdc00017000 comp rx=0 tx=0).stop 2024-01-11T08:23:06.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.311+0000 7fdc123ff700 1 -- 172.21.15.27:0/2969511982 shutdown_connections 2024-01-11T08:23:06.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.311+0000 7fdc123ff700 1 --2- 172.21.15.27:0/2969511982 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdbf8066fb0 0x7fdbf8069460 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.311+0000 7fdc123ff700 1 --2- 172.21.15.27:0/2969511982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc0c159200 0x7fdc0c169960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.311+0000 7fdc123ff700 1 --2- 172.21.15.27:0/2969511982 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc0c0b03e0 0x7fdc0c146710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.311+0000 7fdc123ff700 1 --2- 172.21.15.27:0/2969511982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc0c0b07b0 0x7fdc0c0c6890 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.311+0000 7fdc123ff700 1 -- 172.21.15.27:0/2969511982 >> 172.21.15.27:0/2969511982 conn(0x7fdc0c06ed40 msgr2=0x7fdc0c15c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:06.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.311+0000 7fdc123ff700 1 -- 172.21.15.27:0/2969511982 shutdown_connections 2024-01-11T08:23:06.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.311+0000 7fdc123ff700 1 -- 172.21.15.27:0/2969511982 wait complete. 2024-01-11T08:23:06.328 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:23:06.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.529+0000 7f5a937ff700 1 Processor -- start 2024-01-11T08:23:06.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.530+0000 7f5a937ff700 1 -- start start 2024-01-11T08:23:06.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.531+0000 7f5a937ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a8c069d40 0x7f5a8c0b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:06.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.531+0000 7f5a937ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8c157390 con 0x7f5a8c0c89f0 2024-01-11T08:23:06.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.531+0000 7f5a937ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8c06a200 con 0x7f5a8c0c8620 2024-01-11T08:23:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.531+0000 7f5a937ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c156d40 con 0x7f5a8c069d40 2024-01-11T08:23:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.531+0000 7f5a91d9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5a8c0c89f0 0x7f5a8c0accc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37784/0 (socket says 172.21.15.27:37784) 2024-01-11T08:23:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.531+0000 7f5a91d9c700 1 -- 172.21.15.27:0/1892292215 learned_addr learned my addr 172.21.15.27:0/1892292215 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:23:06.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.531+0000 7f5a90d9a700 1 --2- 172.21.15.27:0/1892292215 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a8c069d40 0x7f5a8c0b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:06.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.532+0000 7f5a83fff700 1 -- 172.21.15.27:0/1892292215 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464794082 0 0) 0x7f5a8c157390 con 0x7f5a8c0c89f0 2024-01-11T08:23:06.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.532+0000 7f5a83fff700 1 -- 172.21.15.27:0/1892292215 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a78003570 con 0x7f5a8c0c89f0 2024-01-11T08:23:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.532+0000 7f5a83fff700 1 -- 172.21.15.27:0/1892292215 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1613415787 0 0) 0x7f5a8c06a200 con 0x7f5a8c0c8620 2024-01-11T08:23:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.532+0000 7f5a83fff700 1 -- 172.21.15.27:0/1892292215 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a8c157390 con 0x7f5a8c0c8620 2024-01-11T08:23:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.532+0000 7f5a83fff700 1 -- 172.21.15.27:0/1892292215 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 791988750 0 0) 0x7f5a78003570 con 0x7f5a8c0c89f0 2024-01-11T08:23:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.532+0000 7f5a83fff700 1 -- 172.21.15.27:0/1892292215 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a8c069d40 msgr2=0x7f5a8c0b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.532+0000 7f5a83fff700 1 --2- 172.21.15.27:0/1892292215 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a8c069d40 0x7f5a8c0b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.532+0000 7f5a83fff700 1 -- 172.21.15.27:0/1892292215 >> v1:172.21.15.77:6789/0 conn(0x7f5a8c0c8620 legacy=0x7f5a8c0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:06.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.532+0000 7f5a83fff700 1 -- 172.21.15.27:0/1892292215 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8c157020 con 0x7f5a8c0c89f0 2024-01-11T08:23:06.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.532+0000 7f5a83fff700 1 -- 172.21.15.27:0/1892292215 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f5a840017f0 con 0x7f5a8c0c89f0 2024-01-11T08:23:06.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.532+0000 7f5a83fff700 1 -- 172.21.15.27:0/1892292215 >> v1:172.21.15.27:6789/0 conn(0x7f5a8c0c89f0 legacy=0x7f5a8c0accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:06.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.532+0000 7f5a83fff700 1 --2- 172.21.15.27:0/1892292215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a78003c20 0x7f5a78004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:06.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.532+0000 7f5a83fff700 1 --2- 172.21.15.27:0/1892292215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a78004810 0x7f5a78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:06.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.532+0000 7f5a83fff700 1 --2- 172.21.15.27:0/1892292215 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a8c069d40 0x7f5a78005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:06.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.532+0000 7f5a83fff700 1 -- 172.21.15.27:0/1892292215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c156d40 con 0x7f5a78004810 2024-01-11T08:23:06.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.532+0000 7f5a83fff700 1 -- 172.21.15.27:0/1892292215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7800bbc0 con 0x7f5a78003c20 2024-01-11T08:23:06.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.532+0000 7f5a83fff700 1 -- 172.21.15.27:0/1892292215 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7800bd80 con 0x7f5a8c069d40 2024-01-11T08:23:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.532+0000 7f5a90d9a700 1 --2- 172.21.15.27:0/1892292215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a78004810 0x7f5a78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.532+0000 7f5a9159b700 1 --2- 172.21.15.27:0/1892292215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a78003c20 0x7f5a78004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.533+0000 7f5a91d9c700 1 --2- 172.21.15.27:0/1892292215 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a8c069d40 0x7f5a78005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.533+0000 7f5a90d9a700 1 -- 172.21.15.27:0/1892292215 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a8c069d40 msgr2=0x7f5a78005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.533+0000 7f5a90d9a700 1 --2- 172.21.15.27:0/1892292215 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a8c069d40 0x7f5a78005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.533+0000 7f5a90d9a700 1 -- 172.21.15.27:0/1892292215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a78003c20 msgr2=0x7f5a78004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.533+0000 7f5a90d9a700 1 --2- 172.21.15.27:0/1892292215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a78003c20 0x7f5a78004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.533+0000 7f5a90d9a700 1 -- 172.21.15.27:0/1892292215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a78003570 con 0x7f5a78004810 2024-01-11T08:23:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.533+0000 7f5a90d9a700 1 --2- 172.21.15.27:0/1892292215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a78004810 0x7f5a78004c80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c06a130 tx=0x7f5a7c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7fed31495af4b59 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.533+0000 7f5a83fff700 1 -- 172.21.15.27:0/1892292215 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c00e710 con 0x7f5a78004810 2024-01-11T08:23:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.533+0000 7f5a83fff700 1 -- 172.21.15.27:0/1892292215 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7c00e870 con 0x7f5a78004810 2024-01-11T08:23:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.533+0000 7f5a83fff700 1 -- 172.21.15.27:0/1892292215 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c00ec30 con 0x7f5a78004810 2024-01-11T08:23:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.534+0000 7f5a937ff700 1 -- 172.21.15.27:0/1892292215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a78004810 msgr2=0x7f5a78004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.534+0000 7f5a937ff700 1 --2- 172.21.15.27:0/1892292215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a78004810 0x7f5a78004c80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c06a130 tx=0x7f5a7c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.534+0000 7f5a91d9c700 1 -- 172.21.15.27:0/1892292215 reap_dead start 2024-01-11T08:23:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.534+0000 7f5a937ff700 1 -- 172.21.15.27:0/1892292215 shutdown_connections 2024-01-11T08:23:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.534+0000 7f5a937ff700 1 -- 172.21.15.27:0/1892292215 >> 172.21.15.27:0/1892292215 conn(0x7f5a8c0680e0 msgr2=0x7f5a8c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.534+0000 7f5a937ff700 1 -- 172.21.15.27:0/1892292215 shutdown_connections 2024-01-11T08:23:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.534+0000 7f5a937ff700 1 -- 172.21.15.27:0/1892292215 wait complete. 2024-01-11T08:23:06.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.536+0000 7f5a937ff700 1 Processor -- start 2024-01-11T08:23:06.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.536+0000 7f5a937ff700 1 -- start start 2024-01-11T08:23:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.537+0000 7f5a937ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a78004810 0x7f5a8c0ac300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.537+0000 7f5a937ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a8c0ac840 0x7f5a8c14c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.537+0000 7f5a937ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a8c14cae0 0x7f5a8c136ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.537+0000 7f5a937ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c0ad8f0 con 0x7f5a8c0ac840 2024-01-11T08:23:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.537+0000 7f5a937ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c0add80 con 0x7f5a8c14cae0 2024-01-11T08:23:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.537+0000 7f5a937ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c0acce0 con 0x7f5a78004810 2024-01-11T08:23:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.537+0000 7f5a90d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a8c0ac840 0x7f5a8c14c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.537+0000 7f5a90d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a8c0ac840 0x7f5a8c14c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33604/0 (socket says 172.21.15.27:33604) 2024-01-11T08:23:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.537+0000 7f5a90d9a700 1 -- 172.21.15.27:0/3200973726 learned_addr learned my addr 172.21.15.27:0/3200973726 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.537+0000 7f5a9159b700 1 --2- 172.21.15.27:0/3200973726 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a78004810 0x7f5a8c0ac300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.537+0000 7f5a91d9c700 1 --2- 172.21.15.27:0/3200973726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a8c14cae0 0x7f5a8c136ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.537+0000 7f5a90d9a700 1 -- 172.21.15.27:0/3200973726 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a78004810 msgr2=0x7f5a8c0ac300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.537+0000 7f5a90d9a700 1 --2- 172.21.15.27:0/3200973726 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a78004810 0x7f5a8c0ac300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.537+0000 7f5a90d9a700 1 -- 172.21.15.27:0/3200973726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a8c14cae0 msgr2=0x7f5a8c136ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.537+0000 7f5a90d9a700 1 --2- 172.21.15.27:0/3200973726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a8c14cae0 0x7f5a8c136ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.537+0000 7f5a90d9a700 1 -- 172.21.15.27:0/3200973726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8c0acfc0 con 0x7f5a8c0ac840 2024-01-11T08:23:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.537+0000 7f5a9159b700 1 --2- 172.21.15.27:0/3200973726 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a78004810 0x7f5a8c0ac300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:23:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.537+0000 7f5a91d9c700 1 --2- 172.21.15.27:0/3200973726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a8c14cae0 0x7f5a8c136ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:23:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.538+0000 7f5a90d9a700 1 --2- 172.21.15.27:0/3200973726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a8c0ac840 0x7f5a8c14c5a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5a7800bb90 tx=0x7f5a7c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.538+0000 7f5a817fa700 1 -- 172.21.15.27:0/3200973726 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c022070 con 0x7f5a8c0ac840 2024-01-11T08:23:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.538+0000 7f5a817fa700 1 -- 172.21.15.27:0/3200973726 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7c00ebf0 con 0x7f5a8c0ac840 2024-01-11T08:23:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.538+0000 7f5a817fa700 1 -- 172.21.15.27:0/3200973726 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c01c6a0 con 0x7f5a8c0ac840 2024-01-11T08:23:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.538+0000 7f5a937ff700 1 -- 172.21.15.27:0/3200973726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a8c0ad2a0 con 0x7f5a8c0ac840 2024-01-11T08:23:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.538+0000 7f5a937ff700 1 -- 172.21.15.27:0/3200973726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a8c14e3e0 con 0x7f5a8c0ac840 2024-01-11T08:23:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.539+0000 7f5a937ff700 1 -- 172.21.15.27:0/3200973726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a8c0adaa0 con 0x7f5a8c0ac840 2024-01-11T08:23:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.540+0000 7f5a817fa700 1 -- 172.21.15.27:0/3200973726 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5a7c01cad0 con 0x7f5a8c0ac840 2024-01-11T08:23:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.541+0000 7f5a817fa700 1 --2- 172.21.15.27:0/3200973726 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5a70066f60 0x7f5a70069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.541+0000 7f5a817fa700 1 -- 172.21.15.27:0/3200973726 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f5a7c085d00 con 0x7f5a8c0ac840 2024-01-11T08:23:06.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.541+0000 7f5a9159b700 1 --2- 172.21.15.27:0/3200973726 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5a70066f60 0x7f5a70069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:06.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.542+0000 7f5a9159b700 1 --2- 172.21.15.27:0/3200973726 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5a70066f60 0x7f5a70069410 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c0cd1d0 tx=0x7f5a88009040 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:06.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.545+0000 7f5a817fa700 1 -- 172.21.15.27:0/3200973726 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a8c0adaa0 con 0x7f5a8c0ac840 2024-01-11T08:23:06.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.748+0000 7f5a937ff700 1 -- 172.21.15.27:0/3200973726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a8c06d1d0 con 0x7f5a8c0ac840 2024-01-11T08:23:06.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.749+0000 7f5a817fa700 1 -- 172.21.15.27:0/3200973726 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+15315 (secure 0 0 0) 0x7f5a7c017090 con 0x7f5a8c0ac840 2024-01-11T08:23:06.748 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:23:06.748 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":58,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:23:05.344378+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:23:03.012171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-11T08:23:04.337714+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:23:06.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.751+0000 7f5a937ff700 1 -- 172.21.15.27:0/3200973726 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5a70066f60 msgr2=0x7f5a70069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:06.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.751+0000 7f5a937ff700 1 --2- 172.21.15.27:0/3200973726 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5a70066f60 0x7f5a70069410 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c0cd1d0 tx=0x7f5a88009040 comp rx=0 tx=0).stop 2024-01-11T08:23:06.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.752+0000 7f5a937ff700 1 -- 172.21.15.27:0/3200973726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a8c0ac840 msgr2=0x7f5a8c14c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:06.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.752+0000 7f5a937ff700 1 --2- 172.21.15.27:0/3200973726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a8c0ac840 0x7f5a8c14c5a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5a7800bb90 tx=0x7f5a7c01a000 comp rx=0 tx=0).stop 2024-01-11T08:23:06.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.752+0000 7f5a937ff700 1 -- 172.21.15.27:0/3200973726 shutdown_connections 2024-01-11T08:23:06.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.752+0000 7f5a937ff700 1 --2- 172.21.15.27:0/3200973726 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5a70066f60 0x7f5a70069410 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.752+0000 7f5a937ff700 1 --2- 172.21.15.27:0/3200973726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a8c14cae0 0x7f5a8c136ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.752+0000 7f5a937ff700 1 --2- 172.21.15.27:0/3200973726 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a78004810 0x7f5a8c0ac300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.752+0000 7f5a937ff700 1 --2- 172.21.15.27:0/3200973726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a8c0ac840 0x7f5a8c14c5a0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.752+0000 7f5a937ff700 1 -- 172.21.15.27:0/3200973726 >> 172.21.15.27:0/3200973726 conn(0x7f5a8c0680e0 msgr2=0x7f5a8c157f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:06.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.752+0000 7f5a937ff700 1 -- 172.21.15.27:0/3200973726 shutdown_connections 2024-01-11T08:23:06.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.752+0000 7f5a937ff700 1 -- 172.21.15.27:0/3200973726 wait complete. 2024-01-11T08:23:06.769 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-11T08:23:06.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.974+0000 7fb4e3fff700 1 Processor -- start 2024-01-11T08:23:06.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.974+0000 7fb4e3fff700 1 -- start start 2024-01-11T08:23:06.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.975+0000 7fb4e3fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4dc1577e0 0x7fb4dc0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:06.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.975+0000 7fb4e3fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4dc0699d0 con 0x7fb4dc069540 2024-01-11T08:23:06.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.975+0000 7fb4e3fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4dc069cb0 con 0x7fb4dc157410 2024-01-11T08:23:06.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.976+0000 7fb4e3fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc069f90 con 0x7fb4dc1577e0 2024-01-11T08:23:06.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.976+0000 7fb4e259c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb4dc069540 0x7fb4dc0c6ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37786/0 (socket says 172.21.15.27:37786) 2024-01-11T08:23:06.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.976+0000 7fb4e259c700 1 -- 172.21.15.27:0/2693001439 learned_addr learned my addr 172.21.15.27:0/2693001439 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:23:06.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.976+0000 7fb4e159a700 1 --2- 172.21.15.27:0/2693001439 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4dc1577e0 0x7fb4dc0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:06.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.976+0000 7fb4e0d99700 1 -- 172.21.15.27:0/2693001439 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557127400 0 0) 0x7fb4dc0699d0 con 0x7fb4dc069540 2024-01-11T08:23:06.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.976+0000 7fb4e0d99700 1 -- 172.21.15.27:0/2693001439 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4c8003570 con 0x7fb4dc069540 2024-01-11T08:23:06.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.976+0000 7fb4e0d99700 1 -- 172.21.15.27:0/2693001439 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612226178 0 0) 0x7fb4dc069cb0 con 0x7fb4dc157410 2024-01-11T08:23:06.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.976+0000 7fb4e0d99700 1 -- 172.21.15.27:0/2693001439 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4dc0699d0 con 0x7fb4dc157410 2024-01-11T08:23:06.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.976+0000 7fb4e159a700 1 -- 172.21.15.27:0/2693001439 >> v1:172.21.15.77:6789/0 conn(0x7fb4dc157410 legacy=0x7fb4dc0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:06.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.976+0000 7fb4e159a700 1 -- 172.21.15.27:0/2693001439 >> v1:172.21.15.27:6789/0 conn(0x7fb4dc069540 legacy=0x7fb4dc0c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:06.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.976+0000 7fb4e159a700 1 -- 172.21.15.27:0/2693001439 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc06a270 con 0x7fb4dc1577e0 2024-01-11T08:23:06.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.977+0000 7fb4e159a700 1 --2- 172.21.15.27:0/2693001439 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4dc1577e0 0x7fb4dc0ae000 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc009910 tx=0x7fb4cc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b1e766b0d68b20d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:06.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.977+0000 7fb4e0d99700 1 -- 172.21.15.27:0/2693001439 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc011630 con 0x7fb4dc1577e0 2024-01-11T08:23:06.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.977+0000 7fb4e0d99700 1 -- 172.21.15.27:0/2693001439 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4cc011790 con 0x7fb4dc1577e0 2024-01-11T08:23:06.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.977+0000 7fb4e0d99700 1 -- 172.21.15.27:0/2693001439 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc011b30 con 0x7fb4dc1577e0 2024-01-11T08:23:06.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.978+0000 7fb4e3fff700 1 -- 172.21.15.27:0/2693001439 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4dc1577e0 msgr2=0x7fb4dc0ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:06.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.978+0000 7fb4e3fff700 1 --2- 172.21.15.27:0/2693001439 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4dc1577e0 0x7fb4dc0ae000 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc009910 tx=0x7fb4cc0091a0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.978+0000 7fb4e3fff700 1 -- 172.21.15.27:0/2693001439 shutdown_connections 2024-01-11T08:23:06.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.978+0000 7fb4e3fff700 1 --2- 172.21.15.27:0/2693001439 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4dc1577e0 0x7fb4dc0ae000 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.978+0000 7fb4e3fff700 1 -- 172.21.15.27:0/2693001439 >> 172.21.15.27:0/2693001439 conn(0x7fb4dc0680a0 msgr2=0x7fb4dc0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:06.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.978+0000 7fb4e3fff700 1 -- 172.21.15.27:0/2693001439 shutdown_connections 2024-01-11T08:23:06.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.978+0000 7fb4e3fff700 1 -- 172.21.15.27:0/2693001439 wait complete. 2024-01-11T08:23:06.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.980+0000 7fb4e3fff700 1 Processor -- start 2024-01-11T08:23:06.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.980+0000 7fb4e3fff700 1 -- start start 2024-01-11T08:23:06.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.981+0000 7fb4e3fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4dc069540 0x7fb4dc150e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:06.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.981+0000 7fb4e3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4dc157410 0x7fb4dc0c3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:06.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.981+0000 7fb4e3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4dc1577e0 0x7fb4dc16adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:06.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.981+0000 7fb4e3fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc158240 con 0x7fb4dc1577e0 2024-01-11T08:23:06.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.981+0000 7fb4e3fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc153980 con 0x7fb4dc157410 2024-01-11T08:23:06.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.981+0000 7fb4e3fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc0c37b0 con 0x7fb4dc069540 2024-01-11T08:23:06.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.982+0000 7fb4e259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4dc1577e0 0x7fb4dc16adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:06.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.982+0000 7fb4e259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4dc1577e0 0x7fb4dc16adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33610/0 (socket says 172.21.15.27:33610) 2024-01-11T08:23:06.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.982+0000 7fb4e159a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4dc157410 0x7fb4dc0c3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:06.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.982+0000 7fb4e259c700 1 -- 172.21.15.27:0/3550227030 learned_addr learned my addr 172.21.15.27:0/3550227030 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:06.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.982+0000 7fb4e1d9b700 1 --2- 172.21.15.27:0/3550227030 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4dc069540 0x7fb4dc150e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:06.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.982+0000 7fb4e259c700 1 -- 172.21.15.27:0/3550227030 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4dc069540 msgr2=0x7fb4dc150e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:06.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.982+0000 7fb4e259c700 1 --2- 172.21.15.27:0/3550227030 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4dc069540 0x7fb4dc150e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.982+0000 7fb4e259c700 1 -- 172.21.15.27:0/3550227030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4dc157410 msgr2=0x7fb4dc0c3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:06.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.982+0000 7fb4e259c700 1 --2- 172.21.15.27:0/3550227030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4dc157410 0x7fb4dc0c3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:06.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.982+0000 7fb4e259c700 1 -- 172.21.15.27:0/3550227030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc009420 con 0x7fb4dc1577e0 2024-01-11T08:23:06.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.982+0000 7fb4e259c700 1 --2- 172.21.15.27:0/3550227030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4dc1577e0 0x7fb4dc16adf0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb4d400cd00 tx=0x7fb4d4001ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:06.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.983+0000 7fb4d1ffb700 1 -- 172.21.15.27:0/3550227030 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d400ce00 con 0x7fb4dc1577e0 2024-01-11T08:23:06.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.983+0000 7fb4d1ffb700 1 -- 172.21.15.27:0/3550227030 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d400aa80 con 0x7fb4dc1577e0 2024-01-11T08:23:06.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.983+0000 7fb4d1ffb700 1 -- 172.21.15.27:0/3550227030 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d400ae20 con 0x7fb4dc1577e0 2024-01-11T08:23:06.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.983+0000 7fb4e3fff700 1 -- 172.21.15.27:0/3550227030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4dc0c5c20 con 0x7fb4dc1577e0 2024-01-11T08:23:06.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.983+0000 7fb4e3fff700 1 -- 172.21.15.27:0/3550227030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4dc0c61e0 con 0x7fb4dc1577e0 2024-01-11T08:23:06.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.984+0000 7fb4e3fff700 1 -- 172.21.15.27:0/3550227030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4dc1583f0 con 0x7fb4dc1577e0 2024-01-11T08:23:06.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.985+0000 7fb4d1ffb700 1 -- 172.21.15.27:0/3550227030 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb4d4007300 con 0x7fb4dc1577e0 2024-01-11T08:23:06.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.986+0000 7fb4d1ffb700 1 --2- 172.21.15.27:0/3550227030 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb4c0066f30 0x7fb4c00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:06.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.986+0000 7fb4d1ffb700 1 -- 172.21.15.27:0/3550227030 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fb4d40848f0 con 0x7fb4dc1577e0 2024-01-11T08:23:06.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.986+0000 7fb4e1d9b700 1 --2- 172.21.15.27:0/3550227030 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb4c0066f30 0x7fb4c00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:06.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.987+0000 7fb4e1d9b700 1 --2- 172.21.15.27:0/3550227030 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb4c0066f30 0x7fb4c00693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8001130 tx=0x7fb4d8007d00 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:06.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:06.990+0000 7fb4d1ffb700 1 -- 172.21.15.27:0/3550227030 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb4d4051b40 con 0x7fb4dc1577e0 2024-01-11T08:23:07.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:07.242+0000 7fb4e3fff700 1 -- 172.21.15.27:0/3550227030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb4dc04c3c0 con 0x7fb4dc1577e0 2024-01-11T08:23:07.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:07.864+0000 7fb4d1ffb700 1 -- 172.21.15.27:0/3550227030 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb4d4055160 con 0x7fb4dc1577e0 2024-01-11T08:23:07.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:07.867+0000 7fb4e3fff700 1 -- 172.21.15.27:0/3550227030 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb4c0066f30 msgr2=0x7fb4c00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:07.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:07.867+0000 7fb4e3fff700 1 --2- 172.21.15.27:0/3550227030 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb4c0066f30 0x7fb4c00693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8001130 tx=0x7fb4d8007d00 comp rx=0 tx=0).stop 2024-01-11T08:23:07.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:07.867+0000 7fb4e3fff700 1 -- 172.21.15.27:0/3550227030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4dc1577e0 msgr2=0x7fb4dc16adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:07.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:07.867+0000 7fb4e3fff700 1 --2- 172.21.15.27:0/3550227030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4dc1577e0 0x7fb4dc16adf0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb4d400cd00 tx=0x7fb4d4001ac0 comp rx=0 tx=0).stop 2024-01-11T08:23:07.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:07.868+0000 7fb4e3fff700 1 -- 172.21.15.27:0/3550227030 shutdown_connections 2024-01-11T08:23:07.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:07.868+0000 7fb4e3fff700 1 --2- 172.21.15.27:0/3550227030 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb4c0066f30 0x7fb4c00693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:07.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:07.868+0000 7fb4e3fff700 1 --2- 172.21.15.27:0/3550227030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4dc1577e0 0x7fb4dc16adf0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:07.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:07.868+0000 7fb4e3fff700 1 --2- 172.21.15.27:0/3550227030 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4dc069540 0x7fb4dc150e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:07.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:07.868+0000 7fb4e3fff700 1 --2- 172.21.15.27:0/3550227030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4dc157410 0x7fb4dc0c3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:07.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:07.868+0000 7fb4e3fff700 1 -- 172.21.15.27:0/3550227030 >> 172.21.15.27:0/3550227030 conn(0x7fb4dc0680a0 msgr2=0x7fb4dc151850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:07.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:07.868+0000 7fb4e3fff700 1 -- 172.21.15.27:0/3550227030 shutdown_connections 2024-01-11T08:23:07.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:07.868+0000 7fb4e3fff700 1 -- 172.21.15.27:0/3550227030 wait complete. 2024-01-11T08:23:07.888 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-11T08:23:08.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.098+0000 7f6073dff700 1 Processor -- start 2024-01-11T08:23:08.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.098+0000 7f6073dff700 1 -- start start 2024-01-11T08:23:08.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.099+0000 7f6073dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f606c0cd220 0x7f606c15d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:08.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.099+0000 7f6073dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f606c15d860 0x7f606c1591e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:08.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.099+0000 7f6073dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c15aef0 con 0x7f606c0cd5f0 2024-01-11T08:23:08.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.099+0000 7f6073dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f606c0cee00 con 0x7f606c0cd220 2024-01-11T08:23:08.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.099+0000 7f6073dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f606c15dd70 con 0x7f606c15d860 2024-01-11T08:23:08.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.099+0000 7f6071b9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f606c0cd220 0x7f606c15d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:08.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.100+0000 7f6071b9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f606c0cd220 0x7f606c15d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33626/0 (socket says 172.21.15.27:33626) 2024-01-11T08:23:08.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.100+0000 7f6071b9b700 1 -- 172.21.15.27:0/2587221979 learned_addr learned my addr 172.21.15.27:0/2587221979 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:08.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.100+0000 7f6071b9b700 1 -- 172.21.15.27:0/2587221979 >> v2:172.21.15.77:3300/0 conn(0x7f606c15d860 msgr2=0x7f606c1591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:08.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.100+0000 7f607139a700 1 --2- 172.21.15.27:0/2587221979 >> v2:172.21.15.77:3300/0 conn(0x7f606c15d860 0x7f606c1591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:08.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.100+0000 7f6071b9b700 1 --2- 172.21.15.27:0/2587221979 >> v2:172.21.15.77:3300/0 conn(0x7f606c15d860 0x7f606c1591e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:08.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.100+0000 7f6071b9b700 1 -- 172.21.15.27:0/2587221979 >> v1:172.21.15.77:6789/0 conn(0x7f606c0cd5f0 legacy=0x7f606c068570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:08.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.100+0000 7f6071b9b700 1 -- 172.21.15.27:0/2587221979 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606c15e050 con 0x7f606c0cd220 2024-01-11T08:23:08.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.100+0000 7f607139a700 1 --2- 172.21.15.27:0/2587221979 >> v2:172.21.15.77:3300/0 conn(0x7f606c15d860 0x7f606c1591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:23:08.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.100+0000 7f6071b9b700 1 --2- 172.21.15.27:0/2587221979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f606c0cd220 0x7f606c15d320 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f606c089300 tx=0x7f6068009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a6d8360ede462dd server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:08.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.100+0000 7f6070b99700 1 -- 172.21.15.27:0/2587221979 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606800b820 con 0x7f606c0cd220 2024-01-11T08:23:08.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.101+0000 7f6070b99700 1 -- 172.21.15.27:0/2587221979 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606800b980 con 0x7f606c0cd220 2024-01-11T08:23:08.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.101+0000 7f6070b99700 1 -- 172.21.15.27:0/2587221979 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606800bd20 con 0x7f606c0cd220 2024-01-11T08:23:08.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.101+0000 7f6073dff700 1 -- 172.21.15.27:0/2587221979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f606c0cd220 msgr2=0x7f606c15d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:08.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.101+0000 7f6073dff700 1 --2- 172.21.15.27:0/2587221979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f606c0cd220 0x7f606c15d320 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f606c089300 tx=0x7f6068009250 comp rx=0 tx=0).stop 2024-01-11T08:23:08.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.101+0000 7f6073dff700 1 -- 172.21.15.27:0/2587221979 shutdown_connections 2024-01-11T08:23:08.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.101+0000 7f6073dff700 1 --2- 172.21.15.27:0/2587221979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f606c0cd220 0x7f606c15d320 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:08.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.101+0000 7f6073dff700 1 --2- 172.21.15.27:0/2587221979 >> v2:172.21.15.77:3300/0 conn(0x7f606c15d860 0x7f606c1591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:08.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.101+0000 7f6073dff700 1 -- 172.21.15.27:0/2587221979 >> 172.21.15.27:0/2587221979 conn(0x7f606c06ed40 msgr2=0x7f606c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:08.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.101+0000 7f6073dff700 1 -- 172.21.15.27:0/2587221979 shutdown_connections 2024-01-11T08:23:08.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.101+0000 7f6073dff700 1 -- 172.21.15.27:0/2587221979 wait complete. 2024-01-11T08:23:08.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.103+0000 7f6073dff700 1 Processor -- start 2024-01-11T08:23:08.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.103+0000 7f6073dff700 1 -- start start 2024-01-11T08:23:08.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.104+0000 7f6073dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f606c0cd220 0x7f606c15b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:08.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.104+0000 7f6073dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f606c0cd5f0 0x7f606c0b5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:08.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.104+0000 7f6073dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f606c15d860 0x7f606c169b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:08.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.104+0000 7f6073dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c15b710 con 0x7f606c0cd5f0 2024-01-11T08:23:08.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.104+0000 7f6073dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c15b9f0 con 0x7f606c0cd220 2024-01-11T08:23:08.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.104+0000 7f6073dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606c15bcd0 con 0x7f606c15d860 2024-01-11T08:23:08.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.104+0000 7f607139a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f606c0cd5f0 0x7f606c0b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:08.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.104+0000 7f607139a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f606c0cd5f0 0x7f606c0b5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33628/0 (socket says 172.21.15.27:33628) 2024-01-11T08:23:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.104+0000 7f607139a700 1 -- 172.21.15.27:0/2168840814 learned_addr learned my addr 172.21.15.27:0/2168840814 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.104+0000 7f6071b9b700 1 --2- 172.21.15.27:0/2168840814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f606c0cd220 0x7f606c15b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.104+0000 7f607239c700 1 --2- 172.21.15.27:0/2168840814 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f606c15d860 0x7f606c169b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.105+0000 7f607139a700 1 -- 172.21.15.27:0/2168840814 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f606c15d860 msgr2=0x7f606c169b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.105+0000 7f607139a700 1 --2- 172.21.15.27:0/2168840814 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f606c15d860 0x7f606c169b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.105+0000 7f607139a700 1 -- 172.21.15.27:0/2168840814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f606c0cd220 msgr2=0x7f606c15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.105+0000 7f607139a700 1 --2- 172.21.15.27:0/2168840814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f606c0cd220 0x7f606c15b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.105+0000 7f607139a700 1 -- 172.21.15.27:0/2168840814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60680094f0 con 0x7f606c0cd5f0 2024-01-11T08:23:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.105+0000 7f607139a700 1 --2- 172.21.15.27:0/2168840814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f606c0cd5f0 0x7f606c0b5540 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f605c00c850 tx=0x7f605c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.105+0000 7f6061ffb700 1 -- 172.21.15.27:0/2168840814 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c011600 con 0x7f606c0cd5f0 2024-01-11T08:23:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.105+0000 7f6061ffb700 1 -- 172.21.15.27:0/2168840814 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f605c011760 con 0x7f606c0cd5f0 2024-01-11T08:23:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.105+0000 7f6073dff700 1 -- 172.21.15.27:0/2168840814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f606c15bfb0 con 0x7f606c0cd5f0 2024-01-11T08:23:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.105+0000 7f6061ffb700 1 -- 172.21.15.27:0/2168840814 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c011b00 con 0x7f606c0cd5f0 2024-01-11T08:23:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.105+0000 7f6073dff700 1 -- 172.21.15.27:0/2168840814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f606c0b5a80 con 0x7f606c0cd5f0 2024-01-11T08:23:08.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.106+0000 7f6073dff700 1 -- 172.21.15.27:0/2168840814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f606c0c8750 con 0x7f606c0cd5f0 2024-01-11T08:23:08.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.107+0000 7f6061ffb700 1 -- 172.21.15.27:0/2168840814 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f605c0073b0 con 0x7f606c0cd5f0 2024-01-11T08:23:08.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.108+0000 7f6061ffb700 1 --2- 172.21.15.27:0/2168840814 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6050066f30 0x7f60500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:08.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.108+0000 7f6061ffb700 1 -- 172.21.15.27:0/2168840814 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f605c084530 con 0x7f606c0cd5f0 2024-01-11T08:23:08.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.109+0000 7f6071b9b700 1 --2- 172.21.15.27:0/2168840814 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6050066f30 0x7f60500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:08.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.109+0000 7f6071b9b700 1 --2- 172.21.15.27:0/2168840814 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6050066f30 0x7f60500693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f606c089300 tx=0x7f6068016000 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:08.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.112+0000 7f6061ffb700 1 -- 172.21.15.27:0/2168840814 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f605c055230 con 0x7f606c0cd5f0 2024-01-11T08:23:08.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.369+0000 7f6073dff700 1 -- 172.21.15.27:0/2168840814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f606c0c8240 con 0x7f606c0cd5f0 2024-01-11T08:23:08.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.873+0000 7f6061ffb700 1 -- 172.21.15.27:0/2168840814 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) v1 ==== 122+0+0 (secure 0 0 0) 0x7f606c0c8240 con 0x7f606c0cd5f0 2024-01-11T08:23:08.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.876+0000 7f6073dff700 1 -- 172.21.15.27:0/2168840814 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6050066f30 msgr2=0x7f60500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:08.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.876+0000 7f6073dff700 1 --2- 172.21.15.27:0/2168840814 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6050066f30 0x7f60500693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f606c089300 tx=0x7f6068016000 comp rx=0 tx=0).stop 2024-01-11T08:23:08.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.876+0000 7f6073dff700 1 -- 172.21.15.27:0/2168840814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f606c0cd5f0 msgr2=0x7f606c0b5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:08.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.876+0000 7f6073dff700 1 --2- 172.21.15.27:0/2168840814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f606c0cd5f0 0x7f606c0b5540 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f605c00c850 tx=0x7f605c008040 comp rx=0 tx=0).stop 2024-01-11T08:23:08.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.876+0000 7f6073dff700 1 -- 172.21.15.27:0/2168840814 shutdown_connections 2024-01-11T08:23:08.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.876+0000 7f6073dff700 1 --2- 172.21.15.27:0/2168840814 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6050066f30 0x7f60500693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:08.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.876+0000 7f6073dff700 1 --2- 172.21.15.27:0/2168840814 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f606c15d860 0x7f606c169b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:08.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.876+0000 7f6073dff700 1 --2- 172.21.15.27:0/2168840814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f606c0cd220 0x7f606c15b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:08.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.876+0000 7f6073dff700 1 --2- 172.21.15.27:0/2168840814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f606c0cd5f0 0x7f606c0b5540 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:08.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.876+0000 7f6073dff700 1 -- 172.21.15.27:0/2168840814 >> 172.21.15.27:0/2168840814 conn(0x7f606c06ed40 msgr2=0x7f606c149cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:08.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.876+0000 7f6073dff700 1 -- 172.21.15.27:0/2168840814 shutdown_connections 2024-01-11T08:23:08.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:08.876+0000 7f6073dff700 1 -- 172.21.15.27:0/2168840814 wait complete. 2024-01-11T08:23:08.895 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:23:09.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.096+0000 7fe2a7bff700 1 Processor -- start 2024-01-11T08:23:09.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.097+0000 7fe2a7bff700 1 -- start start 2024-01-11T08:23:09.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.098+0000 7fe2a7bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe2a00cc410 0x7fe2a0159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:09.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.098+0000 7fe2a7bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a00cd730 con 0x7fe2a00b03e0 2024-01-11T08:23:09.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.098+0000 7fe2a7bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a00ccb20 con 0x7fe2a00b2b90 2024-01-11T08:23:09.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.098+0000 7fe2a7bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2a00cce00 con 0x7fe2a00cc410 2024-01-11T08:23:09.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.098+0000 7fe2a599b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe2a00b03e0 0x7fe2a00b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37800/0 (socket says 172.21.15.27:37800) 2024-01-11T08:23:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.098+0000 7fe2a599b700 1 -- 172.21.15.27:0/3770161843 learned_addr learned my addr 172.21.15.27:0/3770161843 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:23:09.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.098+0000 7fe2a619c700 1 --2- 172.21.15.27:0/3770161843 >> v2:172.21.15.77:3300/0 conn(0x7fe2a00cc410 0x7fe2a0159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.099+0000 7fe2a4999700 1 -- 172.21.15.27:0/3770161843 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013720782 0 0) 0x7fe2a00cd730 con 0x7fe2a00b03e0 2024-01-11T08:23:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.099+0000 7fe2a4999700 1 -- 172.21.15.27:0/3770161843 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe28c003570 con 0x7fe2a00b03e0 2024-01-11T08:23:09.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.099+0000 7fe2a619c700 1 -- 172.21.15.27:0/3770161843 >> v1:172.21.15.77:6789/0 conn(0x7fe2a00b2b90 legacy=0x7fe2a0159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:09.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.099+0000 7fe2a619c700 1 -- 172.21.15.27:0/3770161843 >> v1:172.21.15.27:6789/0 conn(0x7fe2a00b03e0 legacy=0x7fe2a00b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:09.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.099+0000 7fe2a619c700 1 -- 172.21.15.27:0/3770161843 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a00cd0e0 con 0x7fe2a00cc410 2024-01-11T08:23:09.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.099+0000 7fe2a619c700 1 --2- 172.21.15.27:0/3770161843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2a00cc410 0x7fe2a0159910 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe298009e90 tx=0x7fe298009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8f0c8102bfd112a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:09.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.099+0000 7fe2a4999700 1 -- 172.21.15.27:0/3770161843 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29800e070 con 0x7fe2a00cc410 2024-01-11T08:23:09.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.100+0000 7fe2a4999700 1 -- 172.21.15.27:0/3770161843 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe29800bc40 con 0x7fe2a00cc410 2024-01-11T08:23:09.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.100+0000 7fe2a4999700 1 -- 172.21.15.27:0/3770161843 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe298012430 con 0x7fe2a00cc410 2024-01-11T08:23:09.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.100+0000 7fe2a7bff700 1 -- 172.21.15.27:0/3770161843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2a00cc410 msgr2=0x7fe2a0159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:09.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.100+0000 7fe2a7bff700 1 --2- 172.21.15.27:0/3770161843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2a00cc410 0x7fe2a0159910 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe298009e90 tx=0x7fe298009780 comp rx=0 tx=0).stop 2024-01-11T08:23:09.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.100+0000 7fe2a7bff700 1 -- 172.21.15.27:0/3770161843 shutdown_connections 2024-01-11T08:23:09.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.100+0000 7fe2a7bff700 1 --2- 172.21.15.27:0/3770161843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2a00cc410 0x7fe2a0159910 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:09.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.100+0000 7fe2a7bff700 1 -- 172.21.15.27:0/3770161843 >> 172.21.15.27:0/3770161843 conn(0x7fe2a006ed40 msgr2=0x7fe2a006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:09.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.100+0000 7fe2a7bff700 1 -- 172.21.15.27:0/3770161843 shutdown_connections 2024-01-11T08:23:09.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.100+0000 7fe2a7bff700 1 -- 172.21.15.27:0/3770161843 wait complete. 2024-01-11T08:23:09.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.102+0000 7fe2a7bff700 1 Processor -- start 2024-01-11T08:23:09.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.102+0000 7fe2a7bff700 1 -- start start 2024-01-11T08:23:09.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.103+0000 7fe2a7bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2a00b03e0 0x7fe2a00c8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:09.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.103+0000 7fe2a7bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2a00b2b90 0x7fe2a00c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:09.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.103+0000 7fe2a7bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2a00cc410 0x7fe2a00d0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:09.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.103+0000 7fe2a7bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a00c77f0 con 0x7fe2a00b2b90 2024-01-11T08:23:09.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.103+0000 7fe2a7bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a00cf000 con 0x7fe2a00cc410 2024-01-11T08:23:09.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.103+0000 7fe2a7bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a00cfb80 con 0x7fe2a00b03e0 2024-01-11T08:23:09.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.103+0000 7fe2a519a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2a00b2b90 0x7fe2a00c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:09.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.103+0000 7fe2a519a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2a00b2b90 0x7fe2a00c8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33634/0 (socket says 172.21.15.27:33634) 2024-01-11T08:23:09.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.103+0000 7fe2a519a700 1 -- 172.21.15.27:0/3932993572 learned_addr learned my addr 172.21.15.27:0/3932993572 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:09.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.103+0000 7fe2a619c700 1 --2- 172.21.15.27:0/3932993572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2a00cc410 0x7fe2a00d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:09.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.103+0000 7fe2a599b700 1 --2- 172.21.15.27:0/3932993572 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2a00b03e0 0x7fe2a00c8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:09.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.103+0000 7fe2a519a700 1 -- 172.21.15.27:0/3932993572 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2a00b03e0 msgr2=0x7fe2a00c8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:09.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.103+0000 7fe2a519a700 1 --2- 172.21.15.27:0/3932993572 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2a00b03e0 0x7fe2a00c8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:09.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.103+0000 7fe2a519a700 1 -- 172.21.15.27:0/3932993572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2a00cc410 msgr2=0x7fe2a00d0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:09.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.103+0000 7fe2a519a700 1 --2- 172.21.15.27:0/3932993572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2a00cc410 0x7fe2a00d0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:09.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.103+0000 7fe2a519a700 1 -- 172.21.15.27:0/3932993572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe298009a20 con 0x7fe2a00b2b90 2024-01-11T08:23:09.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.104+0000 7fe2a519a700 1 --2- 172.21.15.27:0/3932993572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2a00b2b90 0x7fe2a00c8790 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe29000c5c0 tx=0x7fe29000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:09.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.104+0000 7fe295ffb700 1 -- 172.21.15.27:0/3932993572 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe290011590 con 0x7fe2a00b2b90 2024-01-11T08:23:09.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.104+0000 7fe295ffb700 1 -- 172.21.15.27:0/3932993572 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe290011710 con 0x7fe2a00b2b90 2024-01-11T08:23:09.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.104+0000 7fe2a7bff700 1 -- 172.21.15.27:0/3932993572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a00cf250 con 0x7fe2a00b2b90 2024-01-11T08:23:09.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.104+0000 7fe295ffb700 1 -- 172.21.15.27:0/3932993572 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe290011ad0 con 0x7fe2a00b2b90 2024-01-11T08:23:09.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.104+0000 7fe2a7bff700 1 -- 172.21.15.27:0/3932993572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a00cf810 con 0x7fe2a00b2b90 2024-01-11T08:23:09.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.105+0000 7fe2a7bff700 1 -- 172.21.15.27:0/3932993572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a015d590 con 0x7fe2a00b2b90 2024-01-11T08:23:09.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.106+0000 7fe295ffb700 1 -- 172.21.15.27:0/3932993572 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe29001b3b0 con 0x7fe2a00b2b90 2024-01-11T08:23:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.108+0000 7fe295ffb700 1 --2- 172.21.15.27:0/3932993572 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe284066f30 0x7fe2840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.108+0000 7fe295ffb700 1 -- 172.21.15.27:0/3932993572 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fe2900839e0 con 0x7fe2a00b2b90 2024-01-11T08:23:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.108+0000 7fe2a599b700 1 --2- 172.21.15.27:0/3932993572 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe284066f30 0x7fe2840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:09.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.109+0000 7fe2a599b700 1 --2- 172.21.15.27:0/3932993572 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe284066f30 0x7fe2840693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe29c007fd0 tx=0x7fe29c009040 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:09.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.112+0000 7fe295ffb700 1 -- 172.21.15.27:0/3932993572 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe290050c50 con 0x7fe2a00b2b90 2024-01-11T08:23:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.360+0000 7fe2a7bff700 1 -- 172.21.15.27:0/3932993572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2a00cfd10 con 0x7fe2a00b2b90 2024-01-11T08:23:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.361+0000 7fe295ffb700 1 -- 172.21.15.27:0/3932993572 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4594 (secure 0 0 0) 0x7fe290017070 con 0x7fe2a00b2b90 2024-01-11T08:23:09.360 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:23:09.360 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5243,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2418922463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2418922463},{"type":"v1","addr":"172.21.15.27:6835","nonce":2418922463}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5246,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3919152683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3919152683},{"type":"v1","addr":"172.21.15.77:6836","nonce":3919152683}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5641,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/329572459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":329572459},{"type":"v1","addr":"172.21.15.77:6837","nonce":329572459}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:23:05.344753+0000","modified":"2024-01-11T08:23:08.864031+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/4173068012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":4173068012},{"type":"v1","addr":"172.21.15.27:6834","nonce":4173068012}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2024-01-11T08:23:09.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.363+0000 7fe2a7bff700 1 -- 172.21.15.27:0/3932993572 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe284066f30 msgr2=0x7fe2840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:09.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.363+0000 7fe2a7bff700 1 --2- 172.21.15.27:0/3932993572 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe284066f30 0x7fe2840693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe29c007fd0 tx=0x7fe29c009040 comp rx=0 tx=0).stop 2024-01-11T08:23:09.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.363+0000 7fe2a7bff700 1 -- 172.21.15.27:0/3932993572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2a00b2b90 msgr2=0x7fe2a00c8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:09.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.363+0000 7fe2a7bff700 1 --2- 172.21.15.27:0/3932993572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2a00b2b90 0x7fe2a00c8790 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe29000c5c0 tx=0x7fe29000c8d0 comp rx=0 tx=0).stop 2024-01-11T08:23:09.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.364+0000 7fe2a7bff700 1 -- 172.21.15.27:0/3932993572 shutdown_connections 2024-01-11T08:23:09.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.364+0000 7fe2a7bff700 1 --2- 172.21.15.27:0/3932993572 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fe284066f30 0x7fe2840693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:09.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.364+0000 7fe2a7bff700 1 --2- 172.21.15.27:0/3932993572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2a00cc410 0x7fe2a00d0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:09.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.364+0000 7fe2a7bff700 1 --2- 172.21.15.27:0/3932993572 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2a00b03e0 0x7fe2a00c8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:09.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.364+0000 7fe2a7bff700 1 --2- 172.21.15.27:0/3932993572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2a00b2b90 0x7fe2a00c8790 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:09.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.364+0000 7fe2a7bff700 1 -- 172.21.15.27:0/3932993572 >> 172.21.15.27:0/3932993572 conn(0x7fe2a006ed40 msgr2=0x7fe2a00b4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:09.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.364+0000 7fe2a7bff700 1 -- 172.21.15.27:0/3932993572 shutdown_connections 2024-01-11T08:23:09.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.364+0000 7fe2a7bff700 1 -- 172.21.15.27:0/3932993572 wait complete. 2024-01-11T08:23:09.364 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-01-11T08:23:09.381 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:23:09.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.581+0000 7fbfcb7ff700 1 Processor -- start 2024-01-11T08:23:09.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.581+0000 7fbfcb7ff700 1 -- start start 2024-01-11T08:23:09.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.582+0000 7fbfcb7ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbfc41596c0 0x7fbfc415d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:09.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.582+0000 7fbfcb7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbfc40ca670 0x7fbfc4068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:09.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.582+0000 7fbfcb7ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc4068fc0 con 0x7fbfc4068b60 2024-01-11T08:23:09.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.583+0000 7fbfcb7ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfc40a9cb0 con 0x7fbfc41596c0 2024-01-11T08:23:09.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.583+0000 7fbfcb7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfc415e210 con 0x7fbfc40ca670 2024-01-11T08:23:09.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.583+0000 7fbfc959b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbfc41596c0 0x7fbfc415d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:09.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.583+0000 7fbfc959b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbfc41596c0 0x7fbfc415d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33636/0 (socket says 172.21.15.27:33636) 2024-01-11T08:23:09.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.583+0000 7fbfc959b700 1 -- 172.21.15.27:0/3810003012 learned_addr learned my addr 172.21.15.27:0/3810003012 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:09.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.583+0000 7fbfc8d9a700 1 --2- 172.21.15.27:0/3810003012 >> v2:172.21.15.77:3300/0 conn(0x7fbfc40ca670 0x7fbfc4068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:09.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.583+0000 7fbfc959b700 1 -- 172.21.15.27:0/3810003012 >> v2:172.21.15.77:3300/0 conn(0x7fbfc40ca670 msgr2=0x7fbfc4068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:09.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.583+0000 7fbfc959b700 1 --2- 172.21.15.27:0/3810003012 >> v2:172.21.15.77:3300/0 conn(0x7fbfc40ca670 0x7fbfc4068620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:09.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.583+0000 7fbfc959b700 1 -- 172.21.15.27:0/3810003012 >> v1:172.21.15.77:6789/0 conn(0x7fbfc4068b60 legacy=0x7fbfc4159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:09.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.583+0000 7fbfc959b700 1 -- 172.21.15.27:0/3810003012 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc415dc50 con 0x7fbfc41596c0 2024-01-11T08:23:09.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.583+0000 7fbfc8d9a700 1 --2- 172.21.15.27:0/3810003012 >> v2:172.21.15.77:3300/0 conn(0x7fbfc40ca670 0x7fbfc4068620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:23:09.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.584+0000 7fbfc959b700 1 --2- 172.21.15.27:0/3810003012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfc41596c0 0x7fbfc415d300 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40b2c00 tx=0x7fbfc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b987eb3e2ed16c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:09.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.584+0000 7fbfbbfff700 1 -- 172.21.15.27:0/3810003012 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc000b820 con 0x7fbfc41596c0 2024-01-11T08:23:09.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.584+0000 7fbfbbfff700 1 -- 172.21.15.27:0/3810003012 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc000b980 con 0x7fbfc41596c0 2024-01-11T08:23:09.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.584+0000 7fbfbbfff700 1 -- 172.21.15.27:0/3810003012 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc000bd20 con 0x7fbfc41596c0 2024-01-11T08:23:09.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.584+0000 7fbfcb7ff700 1 -- 172.21.15.27:0/3810003012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfc41596c0 msgr2=0x7fbfc415d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:09.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.584+0000 7fbfcb7ff700 1 --2- 172.21.15.27:0/3810003012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfc41596c0 0x7fbfc415d300 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40b2c00 tx=0x7fbfc0009250 comp rx=0 tx=0).stop 2024-01-11T08:23:09.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.584+0000 7fbfcb7ff700 1 -- 172.21.15.27:0/3810003012 shutdown_connections 2024-01-11T08:23:09.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.584+0000 7fbfcb7ff700 1 --2- 172.21.15.27:0/3810003012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfc41596c0 0x7fbfc415d300 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:09.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.584+0000 7fbfcb7ff700 1 --2- 172.21.15.27:0/3810003012 >> v2:172.21.15.77:3300/0 conn(0x7fbfc40ca670 0x7fbfc4068620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:09.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.584+0000 7fbfcb7ff700 1 -- 172.21.15.27:0/3810003012 >> 172.21.15.27:0/3810003012 conn(0x7fbfc40c9f70 msgr2=0x7fbfc406deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:09.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.584+0000 7fbfcb7ff700 1 -- 172.21.15.27:0/3810003012 shutdown_connections 2024-01-11T08:23:09.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.585+0000 7fbfcb7ff700 1 -- 172.21.15.27:0/3810003012 wait complete. 2024-01-11T08:23:09.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.586+0000 7fbfcb7ff700 1 Processor -- start 2024-01-11T08:23:09.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.586+0000 7fbfcb7ff700 1 -- start start 2024-01-11T08:23:09.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.587+0000 7fbfcb7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbfc4068b60 0x7fbfc414b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:09.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.587+0000 7fbfcb7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfc40ca670 0x7fbfc40c2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:09.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.587+0000 7fbfcb7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfc41596c0 0x7fbfc40c2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.587+0000 7fbfcb7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc40ac4d0 con 0x7fbfc40ca670 2024-01-11T08:23:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.587+0000 7fbfcb7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc40ab8c0 con 0x7fbfc41596c0 2024-01-11T08:23:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.587+0000 7fbfcb7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc40abba0 con 0x7fbfc4068b60 2024-01-11T08:23:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.587+0000 7fbfc8d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfc40ca670 0x7fbfc40c2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.587+0000 7fbfc8d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfc40ca670 0x7fbfc40c2880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33640/0 (socket says 172.21.15.27:33640) 2024-01-11T08:23:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.587+0000 7fbfc8d9a700 1 -- 172.21.15.27:0/3057771392 learned_addr learned my addr 172.21.15.27:0/3057771392 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.587+0000 7fbfc9d9c700 1 --2- 172.21.15.27:0/3057771392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfc41596c0 0x7fbfc40c2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.587+0000 7fbfc959b700 1 --2- 172.21.15.27:0/3057771392 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbfc4068b60 0x7fbfc414b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.588+0000 7fbfc8d9a700 1 -- 172.21.15.27:0/3057771392 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbfc4068b60 msgr2=0x7fbfc414b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.588+0000 7fbfc8d9a700 1 --2- 172.21.15.27:0/3057771392 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbfc4068b60 0x7fbfc414b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.588+0000 7fbfc8d9a700 1 -- 172.21.15.27:0/3057771392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfc41596c0 msgr2=0x7fbfc40c2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.588+0000 7fbfc8d9a700 1 --2- 172.21.15.27:0/3057771392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfc41596c0 0x7fbfc40c2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:09.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.588+0000 7fbfc8d9a700 1 -- 172.21.15.27:0/3057771392 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc00094f0 con 0x7fbfc40ca670 2024-01-11T08:23:09.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.588+0000 7fbfc8d9a700 1 --2- 172.21.15.27:0/3057771392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfc40ca670 0x7fbfc40c2880 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fbfb400c880 tx=0x7fbfb4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:09.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.588+0000 7fbfb97fa700 1 -- 172.21.15.27:0/3057771392 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb4011600 con 0x7fbfc40ca670 2024-01-11T08:23:09.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.588+0000 7fbfb97fa700 1 -- 172.21.15.27:0/3057771392 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb4011760 con 0x7fbfc40ca670 2024-01-11T08:23:09.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.588+0000 7fbfb97fa700 1 -- 172.21.15.27:0/3057771392 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb4011ae0 con 0x7fbfc40ca670 2024-01-11T08:23:09.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.588+0000 7fbfcb7ff700 1 -- 172.21.15.27:0/3057771392 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc40abe80 con 0x7fbfc40ca670 2024-01-11T08:23:09.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.589+0000 7fbfcb7ff700 1 -- 172.21.15.27:0/3057771392 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc40b0030 con 0x7fbfc40ca670 2024-01-11T08:23:09.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.589+0000 7fbfcb7ff700 1 -- 172.21.15.27:0/3057771392 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc40718d0 con 0x7fbfc40ca670 2024-01-11T08:23:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.590+0000 7fbfb97fa700 1 -- 172.21.15.27:0/3057771392 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fbfb40193b0 con 0x7fbfc40ca670 2024-01-11T08:23:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.591+0000 7fbfb97fa700 1 --2- 172.21.15.27:0/3057771392 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fbfa8066f30 0x7fbfa80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.591+0000 7fbfb97fa700 1 -- 172.21.15.27:0/3057771392 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fbfb40844b0 con 0x7fbfc40ca670 2024-01-11T08:23:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.592+0000 7fbfc959b700 1 --2- 172.21.15.27:0/3057771392 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fbfa8066f30 0x7fbfa80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.593+0000 7fbfc959b700 1 --2- 172.21.15.27:0/3057771392 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fbfa8066f30 0x7fbfa80693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40b2c00 tx=0x7fbfc0016000 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:09.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.595+0000 7fbfb97fa700 1 -- 172.21.15.27:0/3057771392 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbfb4051680 con 0x7fbfc40ca670 2024-01-11T08:23:09.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.800+0000 7fbfcb7ff700 1 -- 172.21.15.27:0/3057771392 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfc415df30 con 0x7fbfc40ca670 2024-01-11T08:23:09.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.801+0000 7fbfb97fa700 1 -- 172.21.15.27:0/3057771392 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+15302 (secure 0 0 0) 0x7fbfb4054ca0 con 0x7fbfc40ca670 2024-01-11T08:23:09.800 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:23:09.801 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":59,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:23:06.849296+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:23:03.012171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-11T08:23:04.337714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:23:09.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.803+0000 7fbfcb7ff700 1 -- 172.21.15.27:0/3057771392 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fbfa8066f30 msgr2=0x7fbfa80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:09.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.803+0000 7fbfcb7ff700 1 --2- 172.21.15.27:0/3057771392 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fbfa8066f30 0x7fbfa80693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40b2c00 tx=0x7fbfc0016000 comp rx=0 tx=0).stop 2024-01-11T08:23:09.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.804+0000 7fbfcb7ff700 1 -- 172.21.15.27:0/3057771392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfc40ca670 msgr2=0x7fbfc40c2880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:09.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.804+0000 7fbfcb7ff700 1 --2- 172.21.15.27:0/3057771392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfc40ca670 0x7fbfc40c2880 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fbfb400c880 tx=0x7fbfb4008040 comp rx=0 tx=0).stop 2024-01-11T08:23:09.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.804+0000 7fbfcb7ff700 1 -- 172.21.15.27:0/3057771392 shutdown_connections 2024-01-11T08:23:09.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.804+0000 7fbfcb7ff700 1 --2- 172.21.15.27:0/3057771392 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fbfa8066f30 0x7fbfa80693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:09.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.804+0000 7fbfcb7ff700 1 --2- 172.21.15.27:0/3057771392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfc41596c0 0x7fbfc40c2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:09.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.804+0000 7fbfcb7ff700 1 --2- 172.21.15.27:0/3057771392 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbfc4068b60 0x7fbfc414b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:09.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.804+0000 7fbfcb7ff700 1 --2- 172.21.15.27:0/3057771392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfc40ca670 0x7fbfc40c2880 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:09.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.804+0000 7fbfcb7ff700 1 -- 172.21.15.27:0/3057771392 >> 172.21.15.27:0/3057771392 conn(0x7fbfc40c9f70 msgr2=0x7fbfc415a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:09.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.804+0000 7fbfcb7ff700 1 -- 172.21.15.27:0/3057771392 shutdown_connections 2024-01-11T08:23:09.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:09.804+0000 7fbfcb7ff700 1 -- 172.21.15.27:0/3057771392 wait complete. 2024-01-11T08:23:09.804 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-01-11T08:23:09.805+0000 7f7b21df1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T08:23:09.821 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-11T08:23:09.821 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:23:10.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.023+0000 7f5bdb5ff700 1 Processor -- start 2024-01-11T08:23:10.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.023+0000 7f5bdb5ff700 1 -- start start 2024-01-11T08:23:10.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.024+0000 7f5bdb5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5bd40b2b90 0x7f5bd4159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:10.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.024+0000 7f5bdb5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5bd40b07b0 0x7f5bd4159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.024+0000 7f5bdb5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40cd730 con 0x7f5bd40b03e0 2024-01-11T08:23:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.024+0000 7f5bdb5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bd40ccb20 con 0x7f5bd40b07b0 2024-01-11T08:23:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.024+0000 7f5bdb5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd40cce00 con 0x7f5bd40b2b90 2024-01-11T08:23:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.024+0000 7f5bd939b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5bd40b03e0 0x7f5bd40b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37804/0 (socket says 172.21.15.27:37804) 2024-01-11T08:23:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.024+0000 7f5bd939b700 1 -- 172.21.15.27:0/463557927 learned_addr learned my addr 172.21.15.27:0/463557927 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:23:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.024+0000 7f5bd9b9c700 1 --2- 172.21.15.27:0/463557927 >> v2:172.21.15.27:3300/0 conn(0x7f5bd40b07b0 0x7f5bd4159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.025+0000 7f5bd8b9a700 1 --2- 172.21.15.27:0/463557927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5bd40b2b90 0x7f5bd4159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.025+0000 7f5bcbfff700 1 -- 172.21.15.27:0/463557927 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673740791 0 0) 0x7f5bd40cd730 con 0x7f5bd40b03e0 2024-01-11T08:23:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.025+0000 7f5bd9b9c700 1 -- 172.21.15.27:0/463557927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5bd40b2b90 msgr2=0x7f5bd4159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.025+0000 7f5bd9b9c700 1 --2- 172.21.15.27:0/463557927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5bd40b2b90 0x7f5bd4159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.025+0000 7f5bd9b9c700 1 -- 172.21.15.27:0/463557927 >> v1:172.21.15.27:6789/0 conn(0x7f5bd40b03e0 legacy=0x7f5bd40b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.025+0000 7f5bd9b9c700 1 -- 172.21.15.27:0/463557927 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd40cd0e0 con 0x7f5bd40b07b0 2024-01-11T08:23:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.025+0000 7f5bd8b9a700 1 --2- 172.21.15.27:0/463557927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5bd40b2b90 0x7f5bd4159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:23:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.025+0000 7f5bd9b9c700 1 --2- 172.21.15.27:0/463557927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bd40b07b0 0x7f5bd4159740 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009e60 tx=0x7f5bcc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b5d0f22e7a7a39d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.025+0000 7f5bcbfff700 1 -- 172.21.15.27:0/463557927 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc00e070 con 0x7f5bd40b07b0 2024-01-11T08:23:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.025+0000 7f5bcbfff700 1 -- 172.21.15.27:0/463557927 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bcc00bc30 con 0x7f5bd40b07b0 2024-01-11T08:23:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.025+0000 7f5bcbfff700 1 -- 172.21.15.27:0/463557927 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc012430 con 0x7f5bd40b07b0 2024-01-11T08:23:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.026+0000 7f5bdb5ff700 1 -- 172.21.15.27:0/463557927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bd40b07b0 msgr2=0x7f5bd4159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.026+0000 7f5bdb5ff700 1 --2- 172.21.15.27:0/463557927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bd40b07b0 0x7f5bd4159740 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009e60 tx=0x7f5bcc009780 comp rx=0 tx=0).stop 2024-01-11T08:23:10.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.026+0000 7f5bdb5ff700 1 -- 172.21.15.27:0/463557927 shutdown_connections 2024-01-11T08:23:10.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.026+0000 7f5bdb5ff700 1 --2- 172.21.15.27:0/463557927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bd40b07b0 0x7f5bd4159740 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.026+0000 7f5bdb5ff700 1 --2- 172.21.15.27:0/463557927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5bd40b2b90 0x7f5bd4159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.026+0000 7f5bdb5ff700 1 -- 172.21.15.27:0/463557927 >> 172.21.15.27:0/463557927 conn(0x7f5bd406ed40 msgr2=0x7f5bd406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:10.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.026+0000 7f5bdb5ff700 1 -- 172.21.15.27:0/463557927 shutdown_connections 2024-01-11T08:23:10.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.026+0000 7f5bdb5ff700 1 -- 172.21.15.27:0/463557927 wait complete. 2024-01-11T08:23:10.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.028+0000 7f5bdb5ff700 1 Processor -- start 2024-01-11T08:23:10.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.028+0000 7f5bdb5ff700 1 -- start start 2024-01-11T08:23:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.030+0000 7f5bdb5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5bd40b03e0 0x7f5bd414da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.030+0000 7f5bdb5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bd40b07b0 0x7f5bd415c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.030+0000 7f5bdb5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bd40b2b90 0x7f5bd40c86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.030+0000 7f5bdb5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd40c6870 con 0x7f5bd40b07b0 2024-01-11T08:23:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.030+0000 7f5bdb5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd414df50 con 0x7f5bd40b2b90 2024-01-11T08:23:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.030+0000 7f5bdb5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd40c5c60 con 0x7f5bd40b03e0 2024-01-11T08:23:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.030+0000 7f5bd8b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bd40b07b0 0x7f5bd415c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.030+0000 7f5bd8b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bd40b07b0 0x7f5bd415c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33664/0 (socket says 172.21.15.27:33664) 2024-01-11T08:23:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.030+0000 7f5bd8b9a700 1 -- 172.21.15.27:0/337837334 learned_addr learned my addr 172.21.15.27:0/337837334 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.030+0000 7f5bd939b700 1 --2- 172.21.15.27:0/337837334 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5bd40b03e0 0x7f5bd414da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.030+0000 7f5bd9b9c700 1 --2- 172.21.15.27:0/337837334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bd40b2b90 0x7f5bd40c86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.030+0000 7f5bd8b9a700 1 -- 172.21.15.27:0/337837334 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5bd40b03e0 msgr2=0x7f5bd414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.030+0000 7f5bd8b9a700 1 --2- 172.21.15.27:0/337837334 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5bd40b03e0 0x7f5bd414da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.030+0000 7f5bd8b9a700 1 -- 172.21.15.27:0/337837334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bd40b2b90 msgr2=0x7f5bd40c86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:10.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.030+0000 7f5bd8b9a700 1 --2- 172.21.15.27:0/337837334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bd40b2b90 0x7f5bd40c86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.030+0000 7f5bd8b9a700 1 -- 172.21.15.27:0/337837334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bcc009a20 con 0x7f5bd40b07b0 2024-01-11T08:23:10.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.031+0000 7f5bd8b9a700 1 --2- 172.21.15.27:0/337837334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bd40b07b0 0x7f5bd415c5d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5bc400d860 tx=0x7f5bc4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:10.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.031+0000 7f5bc97fa700 1 -- 172.21.15.27:0/337837334 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc40125f0 con 0x7f5bd40b07b0 2024-01-11T08:23:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.031+0000 7f5bc97fa700 1 -- 172.21.15.27:0/337837334 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc4012750 con 0x7f5bd40b07b0 2024-01-11T08:23:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.031+0000 7f5bc97fa700 1 -- 172.21.15.27:0/337837334 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc4012af0 con 0x7f5bd40b07b0 2024-01-11T08:23:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.031+0000 7f5bdb5ff700 1 -- 172.21.15.27:0/337837334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd40c5f40 con 0x7f5bd40b07b0 2024-01-11T08:23:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.031+0000 7f5bdb5ff700 1 -- 172.21.15.27:0/337837334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd40c6500 con 0x7f5bd40b07b0 2024-01-11T08:23:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.032+0000 7f5bdb5ff700 1 -- 172.21.15.27:0/337837334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd40718d0 con 0x7f5bd40b07b0 2024-01-11T08:23:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.033+0000 7f5bc97fa700 1 -- 172.21.15.27:0/337837334 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5bc4013480 con 0x7f5bd40b07b0 2024-01-11T08:23:10.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.034+0000 7f5bc97fa700 1 --2- 172.21.15.27:0/337837334 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5bb8066f30 0x7f5bb80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:10.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.034+0000 7f5bc97fa700 1 -- 172.21.15.27:0/337837334 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f5bc4084710 con 0x7f5bd40b07b0 2024-01-11T08:23:10.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.034+0000 7f5bd939b700 1 --2- 172.21.15.27:0/337837334 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5bb8066f30 0x7f5bb80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.035+0000 7f5bd939b700 1 --2- 172.21.15.27:0/337837334 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5bb8066f30 0x7f5bb80693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0007fd0 tx=0x7f5bd0007d30 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:10.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.038+0000 7f5bc97fa700 1 -- 172.21.15.27:0/337837334 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5bc40518e0 con 0x7f5bd40b07b0 2024-01-11T08:23:10.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.243+0000 7f5bdb5ff700 1 -- 172.21.15.27:0/337837334 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5bd40c69b0 con 0x7f5bb8066f30 2024-01-11T08:23:10.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.255+0000 7f5bc97fa700 1 -- 172.21.15.27:0/337837334 <== mgr.5434 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289106 (secure 0 0 0) 0x7f5bd0001420 con 0x7f5bb8066f30 2024-01-11T08:23:10.258 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:23:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.259+0000 7f5bdb5ff700 1 -- 172.21.15.27:0/337837334 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5bb8066f30 msgr2=0x7f5bb80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.259+0000 7f5bdb5ff700 1 --2- 172.21.15.27:0/337837334 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5bb8066f30 0x7f5bb80693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0007fd0 tx=0x7f5bd0007d30 comp rx=0 tx=0).stop 2024-01-11T08:23:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.259+0000 7f5bdb5ff700 1 -- 172.21.15.27:0/337837334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bd40b07b0 msgr2=0x7f5bd415c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.259+0000 7f5bdb5ff700 1 --2- 172.21.15.27:0/337837334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bd40b07b0 0x7f5bd415c5d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5bc400d860 tx=0x7f5bc4008040 comp rx=0 tx=0).stop 2024-01-11T08:23:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.259+0000 7f5bdb5ff700 1 -- 172.21.15.27:0/337837334 shutdown_connections 2024-01-11T08:23:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.259+0000 7f5bdb5ff700 1 --2- 172.21.15.27:0/337837334 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5bb8066f30 0x7f5bb80693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.259+0000 7f5bdb5ff700 1 --2- 172.21.15.27:0/337837334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bd40b2b90 0x7f5bd40c86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.259+0000 7f5bdb5ff700 1 --2- 172.21.15.27:0/337837334 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5bd40b03e0 0x7f5bd414da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.259+0000 7f5bdb5ff700 1 --2- 172.21.15.27:0/337837334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bd40b07b0 0x7f5bd415c5d0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.259+0000 7f5bdb5ff700 1 -- 172.21.15.27:0/337837334 >> 172.21.15.27:0/337837334 conn(0x7f5bd406ed40 msgr2=0x7f5bd40b33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.259+0000 7f5bdb5ff700 1 -- 172.21.15.27:0/337837334 shutdown_connections 2024-01-11T08:23:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.259+0000 7f5bdb5ff700 1 -- 172.21.15.27:0/337837334 wait complete. 2024-01-11T08:23:10.262 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:23:10.276 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-11T08:23:09.987994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2425924,"kb_used_data":6660,"kb_used_omap":0,"kb_used_meta":2419200,"kb_avail":752548796,"statfs":{"total":773094113280,"available":770609967104,"internally_reserved":0,"allocated":6819840,"data_stored":3712604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2477260800},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001408"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.865255+0000","last_change":"2024-01-11T08:23:05.862186+0000","last_active":"2024-01-11T08:23:05.865255+0000","last_peered":"2024-01-11T08:23:05.865255+0000","last_clean":"2024-01-11T08:23:05.865255+0000","last_became_active":"2024-01-11T08:23:05.861864+0000","last_became_peered":"2024-01-11T08:23:05.861864+0000","last_unstale":"2024-01-11T08:23:05.865255+0000","last_undegraded":"2024-01-11T08:23:05.865255+0000","last_fullsized":"2024-01-11T08:23:05.865255+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:02:56.833975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.886017+0000","last_change":"2024-01-11T08:23:05.885287+0000","last_active":"2024-01-11T08:23:05.886017+0000","last_peered":"2024-01-11T08:23:05.886017+0000","last_clean":"2024-01-11T08:23:05.886017+0000","last_became_active":"2024-01-11T08:23:05.884984+0000","last_became_peered":"2024-01-11T08:23:05.884984+0000","last_unstale":"2024-01-11T08:23:05.886017+0000","last_undegraded":"2024-01-11T08:23:05.886017+0000","last_fullsized":"2024-01-11T08:23:05.886017+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:47:05.958625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.883537+0000","last_change":"2024-01-11T08:23:05.882966+0000","last_active":"2024-01-11T08:23:05.883537+0000","last_peered":"2024-01-11T08:23:05.883537+0000","last_clean":"2024-01-11T08:23:05.883537+0000","last_became_active":"2024-01-11T08:23:05.882672+0000","last_became_peered":"2024-01-11T08:23:05.882672+0000","last_unstale":"2024-01-11T08:23:05.883537+0000","last_undegraded":"2024-01-11T08:23:05.883537+0000","last_fullsized":"2024-01-11T08:23:05.883537+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:49:01.889692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.874604+0000","last_change":"2024-01-11T08:23:05.874604+0000","last_active":"2024-01-11T08:23:05.874604+0000","last_peered":"2024-01-11T08:23:05.874604+0000","last_clean":"2024-01-11T08:23:05.874604+0000","last_became_active":"2024-01-11T08:23:05.874352+0000","last_became_peered":"2024-01-11T08:23:05.874352+0000","last_unstale":"2024-01-11T08:23:05.874604+0000","last_undegraded":"2024-01-11T08:23:05.874604+0000","last_fullsized":"2024-01-11T08:23:05.874604+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:18:28.158621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.872003+0000","last_change":"2024-01-11T08:23:05.869002+0000","last_active":"2024-01-11T08:23:05.872003+0000","last_peered":"2024-01-11T08:23:05.872003+0000","last_clean":"2024-01-11T08:23:05.872003+0000","last_became_active":"2024-01-11T08:23:05.868168+0000","last_became_peered":"2024-01-11T08:23:05.868168+0000","last_unstale":"2024-01-11T08:23:05.872003+0000","last_undegraded":"2024-01-11T08:23:05.872003+0000","last_fullsized":"2024-01-11T08:23:05.872003+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:02:56.833975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.869217+0000","last_change":"2024-01-11T08:23:05.865110+0000","last_active":"2024-01-11T08:23:05.869217+0000","last_peered":"2024-01-11T08:23:05.869217+0000","last_clean":"2024-01-11T08:23:05.869217+0000","last_became_active":"2024-01-11T08:23:05.863707+0000","last_became_peered":"2024-01-11T08:23:05.863707+0000","last_unstale":"2024-01-11T08:23:05.869217+0000","last_undegraded":"2024-01-11T08:23:05.869217+0000","last_fullsized":"2024-01-11T08:23:05.869217+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:57:29.862218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.872112+0000","last_change":"2024-01-11T08:23:05.869703+0000","last_active":"2024-01-11T08:23:05.872112+0000","last_peered":"2024-01-11T08:23:05.872112+0000","last_clean":"2024-01-11T08:23:05.872112+0000","last_became_active":"2024-01-11T08:23:05.868773+0000","last_became_peered":"2024-01-11T08:23:05.868773+0000","last_unstale":"2024-01-11T08:23:05.872112+0000","last_undegraded":"2024-01-11T08:23:05.872112+0000","last_fullsized":"2024-01-11T08:23:05.872112+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:36:07.478679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.882012+0000","last_change":"2024-01-11T08:23:05.882012+0000","last_active":"2024-01-11T08:23:05.882012+0000","last_peered":"2024-01-11T08:23:05.882012+0000","last_clean":"2024-01-11T08:23:05.882012+0000","last_became_active":"2024-01-11T08:23:05.877500+0000","last_became_peered":"2024-01-11T08:23:05.877500+0000","last_unstale":"2024-01-11T08:23:05.882012+0000","last_undegraded":"2024-01-11T08:23:05.882012+0000","last_fullsized":"2024-01-11T08:23:05.882012+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:34:55.229035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.883363+0000","last_change":"2024-01-11T08:23:05.883363+0000","last_active":"2024-01-11T08:23:05.883363+0000","last_peered":"2024-01-11T08:23:05.883363+0000","last_clean":"2024-01-11T08:23:05.883363+0000","last_became_active":"2024-01-11T08:23:05.882795+0000","last_became_peered":"2024-01-11T08:23:05.882795+0000","last_unstale":"2024-01-11T08:23:05.883363+0000","last_undegraded":"2024-01-11T08:23:05.883363+0000","last_fullsized":"2024-01-11T08:23:05.883363+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:36:07.478679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.874722+0000","last_change":"2024-01-11T08:23:05.874722+0000","last_active":"2024-01-11T08:23:05.874722+0000","last_peered":"2024-01-11T08:23:05.874722+0000","last_clean":"2024-01-11T08:23:05.874722+0000","last_became_active":"2024-01-11T08:23:05.874450+0000","last_became_peered":"2024-01-11T08:23:05.874450+0000","last_unstale":"2024-01-11T08:23:05.874722+0000","last_undegraded":"2024-01-11T08:23:05.874722+0000","last_fullsized":"2024-01-11T08:23:05.874722+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:41:27.131872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.886264+0000","last_change":"2024-01-11T08:23:05.886264+0000","last_active":"2024-01-11T08:23:05.886264+0000","last_peered":"2024-01-11T08:23:05.886264+0000","last_clean":"2024-01-11T08:23:05.886264+0000","last_became_active":"2024-01-11T08:23:05.883711+0000","last_became_peered":"2024-01-11T08:23:05.883711+0000","last_unstale":"2024-01-11T08:23:05.886264+0000","last_undegraded":"2024-01-11T08:23:05.886264+0000","last_fullsized":"2024-01-11T08:23:05.886264+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:04:56.439046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.890255+0000","last_change":"2024-01-11T08:23:05.889528+0000","last_active":"2024-01-11T08:23:05.890255+0000","last_peered":"2024-01-11T08:23:05.890255+0000","last_clean":"2024-01-11T08:23:05.890255+0000","last_became_active":"2024-01-11T08:23:05.889215+0000","last_became_peered":"2024-01-11T08:23:05.889215+0000","last_unstale":"2024-01-11T08:23:05.890255+0000","last_undegraded":"2024-01-11T08:23:05.890255+0000","last_fullsized":"2024-01-11T08:23:05.890255+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:49:01.889692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.886049+0000","last_change":"2024-01-11T08:23:05.881048+0000","last_active":"2024-01-11T08:23:05.886049+0000","last_peered":"2024-01-11T08:23:05.886049+0000","last_clean":"2024-01-11T08:23:05.886049+0000","last_became_active":"2024-01-11T08:23:05.878079+0000","last_became_peered":"2024-01-11T08:23:05.878079+0000","last_unstale":"2024-01-11T08:23:05.886049+0000","last_undegraded":"2024-01-11T08:23:05.886049+0000","last_fullsized":"2024-01-11T08:23:05.886049+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:56:08.275331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.877379+0000","last_change":"2024-01-11T08:23:05.877379+0000","last_active":"2024-01-11T08:23:05.877379+0000","last_peered":"2024-01-11T08:23:05.877379+0000","last_clean":"2024-01-11T08:23:05.877379+0000","last_became_active":"2024-01-11T08:23:05.877091+0000","last_became_peered":"2024-01-11T08:23:05.877091+0000","last_unstale":"2024-01-11T08:23:05.877379+0000","last_undegraded":"2024-01-11T08:23:05.877379+0000","last_fullsized":"2024-01-11T08:23:05.877379+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:02:56.833975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.853788+0000","last_change":"2024-01-11T08:23:05.851687+0000","last_active":"2024-01-11T08:23:05.853788+0000","last_peered":"2024-01-11T08:23:05.853788+0000","last_clean":"2024-01-11T08:23:05.853788+0000","last_became_active":"2024-01-11T08:23:05.851416+0000","last_became_peered":"2024-01-11T08:23:05.851416+0000","last_unstale":"2024-01-11T08:23:05.853788+0000","last_undegraded":"2024-01-11T08:23:05.853788+0000","last_fullsized":"2024-01-11T08:23:05.853788+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:59:45.344026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.886738+0000","last_change":"2024-01-11T08:23:05.885754+0000","last_active":"2024-01-11T08:23:05.886738+0000","last_peered":"2024-01-11T08:23:05.886738+0000","last_clean":"2024-01-11T08:23:05.886738+0000","last_became_active":"2024-01-11T08:23:05.885415+0000","last_became_peered":"2024-01-11T08:23:05.885415+0000","last_unstale":"2024-01-11T08:23:05.886738+0000","last_undegraded":"2024-01-11T08:23:05.886738+0000","last_fullsized":"2024-01-11T08:23:05.886738+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:53:15.590430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.864547+0000","last_change":"2024-01-11T08:23:05.863275+0000","last_active":"2024-01-11T08:23:05.864547+0000","last_peered":"2024-01-11T08:23:05.864547+0000","last_clean":"2024-01-11T08:23:05.864547+0000","last_became_active":"2024-01-11T08:23:05.863004+0000","last_became_peered":"2024-01-11T08:23:05.863004+0000","last_unstale":"2024-01-11T08:23:05.864547+0000","last_undegraded":"2024-01-11T08:23:05.864547+0000","last_fullsized":"2024-01-11T08:23:05.864547+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:52:16.057511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.876616+0000","last_change":"2024-01-11T08:23:05.875741+0000","last_active":"2024-01-11T08:23:05.876616+0000","last_peered":"2024-01-11T08:23:05.876616+0000","last_clean":"2024-01-11T08:23:05.876616+0000","last_became_active":"2024-01-11T08:23:05.875464+0000","last_became_peered":"2024-01-11T08:23:05.875464+0000","last_unstale":"2024-01-11T08:23:05.876616+0000","last_undegraded":"2024-01-11T08:23:05.876616+0000","last_fullsized":"2024-01-11T08:23:05.876616+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:22:05.338870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.870257+0000","last_change":"2024-01-11T08:23:05.868727+0000","last_active":"2024-01-11T08:23:05.870257+0000","last_peered":"2024-01-11T08:23:05.870257+0000","last_clean":"2024-01-11T08:23:05.870257+0000","last_became_active":"2024-01-11T08:23:05.868429+0000","last_became_peered":"2024-01-11T08:23:05.868429+0000","last_unstale":"2024-01-11T08:23:05.870257+0000","last_undegraded":"2024-01-11T08:23:05.870257+0000","last_fullsized":"2024-01-11T08:23:05.870257+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:01:39.987317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.892263+0000","last_change":"2024-01-11T08:23:05.891223+0000","last_active":"2024-01-11T08:23:05.892263+0000","last_peered":"2024-01-11T08:23:05.892263+0000","last_clean":"2024-01-11T08:23:05.892263+0000","last_became_active":"2024-01-11T08:23:05.890964+0000","last_became_peered":"2024-01-11T08:23:05.890964+0000","last_unstale":"2024-01-11T08:23:05.892263+0000","last_undegraded":"2024-01-11T08:23:05.892263+0000","last_fullsized":"2024-01-11T08:23:05.892263+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:34:03.306099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.853675+0000","last_change":"2024-01-11T08:23:05.850649+0000","last_active":"2024-01-11T08:23:05.853675+0000","last_peered":"2024-01-11T08:23:05.853675+0000","last_clean":"2024-01-11T08:23:05.853675+0000","last_became_active":"2024-01-11T08:23:05.848941+0000","last_became_peered":"2024-01-11T08:23:05.848941+0000","last_unstale":"2024-01-11T08:23:05.853675+0000","last_undegraded":"2024-01-11T08:23:05.853675+0000","last_fullsized":"2024-01-11T08:23:05.853675+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:54:18.633954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.873704+0000","last_change":"2024-01-11T08:23:05.873704+0000","last_active":"2024-01-11T08:23:05.873704+0000","last_peered":"2024-01-11T08:23:05.873704+0000","last_clean":"2024-01-11T08:23:05.873704+0000","last_became_active":"2024-01-11T08:23:05.871602+0000","last_became_peered":"2024-01-11T08:23:05.871602+0000","last_unstale":"2024-01-11T08:23:05.873704+0000","last_undegraded":"2024-01-11T08:23:05.873704+0000","last_fullsized":"2024-01-11T08:23:05.873704+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:14:57.878995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.861548+0000","last_change":"2024-01-11T08:23:05.858982+0000","last_active":"2024-01-11T08:23:05.861548+0000","last_peered":"2024-01-11T08:23:05.861548+0000","last_clean":"2024-01-11T08:23:05.861548+0000","last_became_active":"2024-01-11T08:23:05.858302+0000","last_became_peered":"2024-01-11T08:23:05.858302+0000","last_unstale":"2024-01-11T08:23:05.861548+0000","last_undegraded":"2024-01-11T08:23:05.861548+0000","last_fullsized":"2024-01-11T08:23:05.861548+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:19:21.740106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.861156+0000","last_change":"2024-01-11T08:23:05.857315+0000","last_active":"2024-01-11T08:23:05.861156+0000","last_peered":"2024-01-11T08:23:05.861156+0000","last_clean":"2024-01-11T08:23:05.861156+0000","last_became_active":"2024-01-11T08:23:05.857016+0000","last_became_peered":"2024-01-11T08:23:05.857016+0000","last_unstale":"2024-01-11T08:23:05.861156+0000","last_undegraded":"2024-01-11T08:23:05.861156+0000","last_fullsized":"2024-01-11T08:23:05.861156+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:48:30.019533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.867553+0000","last_change":"2024-01-11T08:23:05.864675+0000","last_active":"2024-01-11T08:23:05.867553+0000","last_peered":"2024-01-11T08:23:05.867553+0000","last_clean":"2024-01-11T08:23:05.867553+0000","last_became_active":"2024-01-11T08:23:05.864394+0000","last_became_peered":"2024-01-11T08:23:05.864394+0000","last_unstale":"2024-01-11T08:23:05.867553+0000","last_undegraded":"2024-01-11T08:23:05.867553+0000","last_fullsized":"2024-01-11T08:23:05.867553+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:20:13.817226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.885850+0000","last_change":"2024-01-11T08:23:05.878287+0000","last_active":"2024-01-11T08:23:05.885850+0000","last_peered":"2024-01-11T08:23:05.885850+0000","last_clean":"2024-01-11T08:23:05.885850+0000","last_became_active":"2024-01-11T08:23:05.878031+0000","last_became_peered":"2024-01-11T08:23:05.878031+0000","last_unstale":"2024-01-11T08:23:05.885850+0000","last_undegraded":"2024-01-11T08:23:05.885850+0000","last_fullsized":"2024-01-11T08:23:05.885850+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:42:48.781007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.865022+0000","last_change":"2024-01-11T08:23:05.863410+0000","last_active":"2024-01-11T08:23:05.865022+0000","last_peered":"2024-01-11T08:23:05.865022+0000","last_clean":"2024-01-11T08:23:05.865022+0000","last_became_active":"2024-01-11T08:23:05.861968+0000","last_became_peered":"2024-01-11T08:23:05.861968+0000","last_unstale":"2024-01-11T08:23:05.865022+0000","last_undegraded":"2024-01-11T08:23:05.865022+0000","last_fullsized":"2024-01-11T08:23:05.865022+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:16:23.523824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.890528+0000","last_change":"2024-01-11T08:23:05.889772+0000","last_active":"2024-01-11T08:23:05.890528+0000","last_peered":"2024-01-11T08:23:05.890528+0000","last_clean":"2024-01-11T08:23:05.890528+0000","last_became_active":"2024-01-11T08:23:05.889401+0000","last_became_peered":"2024-01-11T08:23:05.889401+0000","last_unstale":"2024-01-11T08:23:05.890528+0000","last_undegraded":"2024-01-11T08:23:05.890528+0000","last_fullsized":"2024-01-11T08:23:05.890528+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:52:16.057511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.869844+0000","last_change":"2024-01-11T08:23:05.866735+0000","last_active":"2024-01-11T08:23:05.869844+0000","last_peered":"2024-01-11T08:23:05.869844+0000","last_clean":"2024-01-11T08:23:05.869844+0000","last_became_active":"2024-01-11T08:23:05.865276+0000","last_became_peered":"2024-01-11T08:23:05.865276+0000","last_unstale":"2024-01-11T08:23:05.869844+0000","last_undegraded":"2024-01-11T08:23:05.869844+0000","last_fullsized":"2024-01-11T08:23:05.869844+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:26:59.623293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.861220+0000","last_change":"2024-01-11T08:23:05.856965+0000","last_active":"2024-01-11T08:23:05.861220+0000","last_peered":"2024-01-11T08:23:05.861220+0000","last_clean":"2024-01-11T08:23:05.861220+0000","last_became_active":"2024-01-11T08:23:05.856672+0000","last_became_peered":"2024-01-11T08:23:05.856672+0000","last_unstale":"2024-01-11T08:23:05.861220+0000","last_undegraded":"2024-01-11T08:23:05.861220+0000","last_fullsized":"2024-01-11T08:23:05.861220+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:40:30.954875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.869528+0000","last_change":"2024-01-11T08:23:05.867498+0000","last_active":"2024-01-11T08:23:05.869528+0000","last_peered":"2024-01-11T08:23:05.869528+0000","last_clean":"2024-01-11T08:23:05.869528+0000","last_became_active":"2024-01-11T08:23:05.866152+0000","last_became_peered":"2024-01-11T08:23:05.866152+0000","last_unstale":"2024-01-11T08:23:05.869528+0000","last_undegraded":"2024-01-11T08:23:05.869528+0000","last_fullsized":"2024-01-11T08:23:05.869528+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:54:38.853272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.885513+0000","last_change":"2024-01-11T08:23:05.877836+0000","last_active":"2024-01-11T08:23:05.885513+0000","last_peered":"2024-01-11T08:23:05.885513+0000","last_clean":"2024-01-11T08:23:05.885513+0000","last_became_active":"2024-01-11T08:23:05.877536+0000","last_became_peered":"2024-01-11T08:23:05.877536+0000","last_unstale":"2024-01-11T08:23:05.885513+0000","last_undegraded":"2024-01-11T08:23:05.885513+0000","last_fullsized":"2024-01-11T08:23:05.885513+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:39:11.620775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.885245+0000","last_change":"2024-01-11T08:23:05.878365+0000","last_active":"2024-01-11T08:23:05.885245+0000","last_peered":"2024-01-11T08:23:05.885245+0000","last_clean":"2024-01-11T08:23:05.885245+0000","last_became_active":"2024-01-11T08:23:05.878064+0000","last_became_peered":"2024-01-11T08:23:05.878064+0000","last_unstale":"2024-01-11T08:23:05.885245+0000","last_undegraded":"2024-01-11T08:23:05.885245+0000","last_fullsized":"2024-01-11T08:23:05.885245+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:01:39.987317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"58'4","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.861945+0000","last_change":"2024-01-11T08:23:04.860362+0000","last_active":"2024-01-11T08:23:05.861945+0000","last_peered":"2024-01-11T08:23:05.861945+0000","last_clean":"2024-01-11T08:23:05.861945+0000","last_became_active":"2024-01-11T08:23:04.859926+0000","last_became_peered":"2024-01-11T08:23:04.859926+0000","last_unstale":"2024-01-11T08:23:05.861945+0000","last_undegraded":"2024-01-11T08:23:05.861945+0000","last_fullsized":"2024-01-11T08:23:05.861945+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:58:55.564598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.861197+0000","last_change":"2024-01-11T08:23:05.857393+0000","last_active":"2024-01-11T08:23:05.861197+0000","last_peered":"2024-01-11T08:23:05.861197+0000","last_clean":"2024-01-11T08:23:05.861197+0000","last_became_active":"2024-01-11T08:23:05.856528+0000","last_became_peered":"2024-01-11T08:23:05.856528+0000","last_unstale":"2024-01-11T08:23:05.861197+0000","last_undegraded":"2024-01-11T08:23:05.861197+0000","last_fullsized":"2024-01-11T08:23:05.861197+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:44:44.140597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"58'1","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.860534+0000","last_change":"2024-01-11T08:23:04.863311+0000","last_active":"2024-01-11T08:23:05.860534+0000","last_peered":"2024-01-11T08:23:05.860534+0000","last_clean":"2024-01-11T08:23:05.860534+0000","last_became_active":"2024-01-11T08:23:04.862049+0000","last_became_peered":"2024-01-11T08:23:04.862049+0000","last_unstale":"2024-01-11T08:23:05.860534+0000","last_undegraded":"2024-01-11T08:23:05.860534+0000","last_fullsized":"2024-01-11T08:23:05.860534+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:48:20.943166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.867645+0000","last_change":"2024-01-11T08:23:05.863342+0000","last_active":"2024-01-11T08:23:05.867645+0000","last_peered":"2024-01-11T08:23:05.867645+0000","last_clean":"2024-01-11T08:23:05.867645+0000","last_became_active":"2024-01-11T08:23:05.862886+0000","last_became_peered":"2024-01-11T08:23:05.862886+0000","last_unstale":"2024-01-11T08:23:05.867645+0000","last_undegraded":"2024-01-11T08:23:05.867645+0000","last_fullsized":"2024-01-11T08:23:05.867645+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:22:05.338870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"58'2","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.885836+0000","last_change":"2024-01-11T08:23:04.857468+0000","last_active":"2024-01-11T08:23:05.885836+0000","last_peered":"2024-01-11T08:23:05.885836+0000","last_clean":"2024-01-11T08:23:05.885836+0000","last_became_active":"2024-01-11T08:23:04.857201+0000","last_became_peered":"2024-01-11T08:23:04.857201+0000","last_unstale":"2024-01-11T08:23:05.885836+0000","last_undegraded":"2024-01-11T08:23:05.885836+0000","last_fullsized":"2024-01-11T08:23:05.885836+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:19:20.730546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.885888+0000","last_change":"2024-01-11T08:23:05.879255+0000","last_active":"2024-01-11T08:23:05.885888+0000","last_peered":"2024-01-11T08:23:05.885888+0000","last_clean":"2024-01-11T08:23:05.885888+0000","last_became_active":"2024-01-11T08:23:05.878961+0000","last_became_peered":"2024-01-11T08:23:05.878961+0000","last_unstale":"2024-01-11T08:23:05.885888+0000","last_undegraded":"2024-01-11T08:23:05.885888+0000","last_fullsized":"2024-01-11T08:23:05.885888+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:40:30.954875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"58'2","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.864716+0000","last_change":"2024-01-11T08:23:04.853541+0000","last_active":"2024-01-11T08:23:05.864716+0000","last_peered":"2024-01-11T08:23:05.864716+0000","last_clean":"2024-01-11T08:23:05.864716+0000","last_became_active":"2024-01-11T08:23:04.853182+0000","last_became_peered":"2024-01-11T08:23:04.853182+0000","last_unstale":"2024-01-11T08:23:05.864716+0000","last_undegraded":"2024-01-11T08:23:05.864716+0000","last_fullsized":"2024-01-11T08:23:05.864716+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:20:12.807666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.860851+0000","last_change":"2024-01-11T08:23:05.855157+0000","last_active":"2024-01-11T08:23:05.860851+0000","last_peered":"2024-01-11T08:23:05.860851+0000","last_clean":"2024-01-11T08:23:05.860851+0000","last_became_active":"2024-01-11T08:23:05.854437+0000","last_became_peered":"2024-01-11T08:23:05.854437+0000","last_unstale":"2024-01-11T08:23:05.860851+0000","last_undegraded":"2024-01-11T08:23:05.860851+0000","last_fullsized":"2024-01-11T08:23:05.860851+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:36:09.262909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"58'1","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.871192+0000","last_change":"2024-01-11T08:23:04.851430+0000","last_active":"2024-01-11T08:23:05.871192+0000","last_peered":"2024-01-11T08:23:05.871192+0000","last_clean":"2024-01-11T08:23:05.871192+0000","last_became_active":"2024-01-11T08:23:04.851027+0000","last_became_peered":"2024-01-11T08:23:04.851027+0000","last_unstale":"2024-01-11T08:23:05.871192+0000","last_undegraded":"2024-01-11T08:23:05.871192+0000","last_fullsized":"2024-01-11T08:23:05.871192+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:36:08.253349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.887418+0000","last_change":"2024-01-11T08:23:05.887418+0000","last_active":"2024-01-11T08:23:05.887418+0000","last_peered":"2024-01-11T08:23:05.887418+0000","last_clean":"2024-01-11T08:23:05.887418+0000","last_became_active":"2024-01-11T08:23:05.887168+0000","last_became_peered":"2024-01-11T08:23:05.887168+0000","last_unstale":"2024-01-11T08:23:05.887418+0000","last_undegraded":"2024-01-11T08:23:05.887418+0000","last_fullsized":"2024-01-11T08:23:05.887418+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:58:56.574158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"58'1","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.861062+0000","last_change":"2024-01-11T08:23:04.859924+0000","last_active":"2024-01-11T08:23:05.861062+0000","last_peered":"2024-01-11T08:23:05.861062+0000","last_clean":"2024-01-11T08:23:05.861062+0000","last_became_active":"2024-01-11T08:23:04.859604+0000","last_became_peered":"2024-01-11T08:23:04.859604+0000","last_unstale":"2024-01-11T08:23:05.861062+0000","last_undegraded":"2024-01-11T08:23:05.861062+0000","last_fullsized":"2024-01-11T08:23:05.861062+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:18:53.731073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.882017+0000","last_change":"2024-01-11T08:23:05.882017+0000","last_active":"2024-01-11T08:23:05.882017+0000","last_peered":"2024-01-11T08:23:05.882017+0000","last_clean":"2024-01-11T08:23:05.882017+0000","last_became_active":"2024-01-11T08:23:05.877798+0000","last_became_peered":"2024-01-11T08:23:05.877798+0000","last_unstale":"2024-01-11T08:23:05.882017+0000","last_undegraded":"2024-01-11T08:23:05.882017+0000","last_fullsized":"2024-01-11T08:23:05.882017+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:47:05.958625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.852185+0000","last_change":"2024-01-11T08:23:04.852427+0000","last_active":"2024-01-11T08:23:05.852185+0000","last_peered":"2024-01-11T08:23:05.852185+0000","last_clean":"2024-01-11T08:23:05.852185+0000","last_became_active":"2024-01-11T08:23:04.852073+0000","last_became_peered":"2024-01-11T08:23:04.852073+0000","last_unstale":"2024-01-11T08:23:05.852185+0000","last_undegraded":"2024-01-11T08:23:05.852185+0000","last_fullsized":"2024-01-11T08:23:05.852185+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:26:58.613733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.861529+0000","last_change":"2024-01-11T08:23:05.858598+0000","last_active":"2024-01-11T08:23:05.861529+0000","last_peered":"2024-01-11T08:23:05.861529+0000","last_clean":"2024-01-11T08:23:05.861529+0000","last_became_active":"2024-01-11T08:23:05.858141+0000","last_became_peered":"2024-01-11T08:23:05.858141+0000","last_unstale":"2024-01-11T08:23:05.861529+0000","last_undegraded":"2024-01-11T08:23:05.861529+0000","last_fullsized":"2024-01-11T08:23:05.861529+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:17:25.876941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-11T08:23:04.870102+0000","last_change":"2024-01-11T08:23:04.867054+0000","last_active":"2024-01-11T08:23:04.870102+0000","last_peered":"2024-01-11T08:23:04.870102+0000","last_clean":"2024-01-11T08:23:04.870102+0000","last_became_active":"2024-01-11T08:23:04.866431+0000","last_became_peered":"2024-01-11T08:23:04.866431+0000","last_unstale":"2024-01-11T08:23:04.870102+0000","last_undegraded":"2024-01-11T08:23:04.870102+0000","last_fullsized":"2024-01-11T08:23:04.870102+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:53:02.013837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.889838+0000","last_change":"2024-01-11T08:23:05.889137+0000","last_active":"2024-01-11T08:23:05.889838+0000","last_peered":"2024-01-11T08:23:05.889838+0000","last_clean":"2024-01-11T08:23:05.889838+0000","last_became_active":"2024-01-11T08:23:05.888815+0000","last_became_peered":"2024-01-11T08:23:05.888815+0000","last_unstale":"2024-01-11T08:23:05.889838+0000","last_undegraded":"2024-01-11T08:23:05.889838+0000","last_fullsized":"2024-01-11T08:23:05.889838+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:36:07.478679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.845776+0000","last_change":"2024-01-11T08:23:04.860508+0000","last_active":"2024-01-11T08:23:05.845776+0000","last_peered":"2024-01-11T08:23:05.845776+0000","last_clean":"2024-01-11T08:23:05.845776+0000","last_became_active":"2024-01-11T08:23:04.859843+0000","last_became_peered":"2024-01-11T08:23:04.859843+0000","last_unstale":"2024-01-11T08:23:05.845776+0000","last_undegraded":"2024-01-11T08:23:05.845776+0000","last_fullsized":"2024-01-11T08:23:05.845776+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:22:32.656105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.872081+0000","last_change":"2024-01-11T08:23:05.867536+0000","last_active":"2024-01-11T08:23:05.872081+0000","last_peered":"2024-01-11T08:23:05.872081+0000","last_clean":"2024-01-11T08:23:05.872081+0000","last_became_active":"2024-01-11T08:23:05.867252+0000","last_became_peered":"2024-01-11T08:23:05.867252+0000","last_unstale":"2024-01-11T08:23:05.872081+0000","last_undegraded":"2024-01-11T08:23:05.872081+0000","last_fullsized":"2024-01-11T08:23:05.872081+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:49:01.889692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-11T08:23:04.872822+0000","last_change":"2024-01-11T08:23:04.867792+0000","last_active":"2024-01-11T08:23:04.872822+0000","last_peered":"2024-01-11T08:23:04.872822+0000","last_clean":"2024-01-11T08:23:04.872822+0000","last_became_active":"2024-01-11T08:23:04.866448+0000","last_became_peered":"2024-01-11T08:23:04.866448+0000","last_unstale":"2024-01-11T08:23:04.872822+0000","last_undegraded":"2024-01-11T08:23:04.872822+0000","last_fullsized":"2024-01-11T08:23:04.872822+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:19:33.536966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.891518+0000","last_change":"2024-01-11T08:23:05.889421+0000","last_active":"2024-01-11T08:23:05.891518+0000","last_peered":"2024-01-11T08:23:05.891518+0000","last_clean":"2024-01-11T08:23:05.891518+0000","last_became_active":"2024-01-11T08:23:05.889152+0000","last_became_peered":"2024-01-11T08:23:05.889152+0000","last_unstale":"2024-01-11T08:23:05.891518+0000","last_undegraded":"2024-01-11T08:23:05.891518+0000","last_fullsized":"2024-01-11T08:23:05.891518+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:38:03.236711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"58'3","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.862285+0000","last_change":"2024-01-11T08:23:04.860830+0000","last_active":"2024-01-11T08:23:05.862285+0000","last_peered":"2024-01-11T08:23:05.862285+0000","last_clean":"2024-01-11T08:23:05.862285+0000","last_became_active":"2024-01-11T08:23:04.860528+0000","last_became_peered":"2024-01-11T08:23:04.860528+0000","last_unstale":"2024-01-11T08:23:05.862285+0000","last_undegraded":"2024-01-11T08:23:05.862285+0000","last_fullsized":"2024-01-11T08:23:05.862285+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:19:20.730546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.880178+0000","last_change":"2024-01-11T08:23:05.877105+0000","last_active":"2024-01-11T08:23:05.880178+0000","last_peered":"2024-01-11T08:23:05.880178+0000","last_clean":"2024-01-11T08:23:05.880178+0000","last_became_active":"2024-01-11T08:23:05.876688+0000","last_became_peered":"2024-01-11T08:23:05.876688+0000","last_unstale":"2024-01-11T08:23:05.880178+0000","last_undegraded":"2024-01-11T08:23:05.880178+0000","last_fullsized":"2024-01-11T08:23:05.880178+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:52:16.057511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"58'6","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.893405+0000","last_change":"2024-01-11T08:23:04.859827+0000","last_active":"2024-01-11T08:23:05.893405+0000","last_peered":"2024-01-11T08:23:05.893405+0000","last_clean":"2024-01-11T08:23:05.893405+0000","last_became_active":"2024-01-11T08:23:04.859463+0000","last_became_peered":"2024-01-11T08:23:04.859463+0000","last_unstale":"2024-01-11T08:23:05.893405+0000","last_undegraded":"2024-01-11T08:23:05.893405+0000","last_fullsized":"2024-01-11T08:23:05.893405+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:30:41.699759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.877791+0000","last_change":"2024-01-11T08:23:05.876914+0000","last_active":"2024-01-11T08:23:05.877791+0000","last_peered":"2024-01-11T08:23:05.877791+0000","last_clean":"2024-01-11T08:23:05.877791+0000","last_became_active":"2024-01-11T08:23:05.876179+0000","last_became_peered":"2024-01-11T08:23:05.876179+0000","last_unstale":"2024-01-11T08:23:05.877791+0000","last_undegraded":"2024-01-11T08:23:05.877791+0000","last_fullsized":"2024-01-11T08:23:05.877791+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:58:56.574158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"58'2","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.888560+0000","last_change":"2024-01-11T08:23:04.864260+0000","last_active":"2024-01-11T08:23:05.888560+0000","last_peered":"2024-01-11T08:23:05.888560+0000","last_clean":"2024-01-11T08:23:05.888560+0000","last_became_active":"2024-01-11T08:23:04.863911+0000","last_became_peered":"2024-01-11T08:23:04.863911+0000","last_unstale":"2024-01-11T08:23:05.888560+0000","last_undegraded":"2024-01-11T08:23:05.888560+0000","last_fullsized":"2024-01-11T08:23:05.888560+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:52:15.047951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.877300+0000","last_change":"2024-01-11T08:23:05.877300+0000","last_active":"2024-01-11T08:23:05.877300+0000","last_peered":"2024-01-11T08:23:05.877300+0000","last_clean":"2024-01-11T08:23:05.877300+0000","last_became_active":"2024-01-11T08:23:05.876871+0000","last_became_peered":"2024-01-11T08:23:05.876871+0000","last_unstale":"2024-01-11T08:23:05.877300+0000","last_undegraded":"2024-01-11T08:23:05.877300+0000","last_fullsized":"2024-01-11T08:23:05.877300+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:40:30.954875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.853938+0000","last_change":"2024-01-11T08:23:04.855272+0000","last_active":"2024-01-11T08:23:05.853938+0000","last_peered":"2024-01-11T08:23:05.853938+0000","last_clean":"2024-01-11T08:23:05.853938+0000","last_became_active":"2024-01-11T08:23:04.854948+0000","last_became_peered":"2024-01-11T08:23:04.854948+0000","last_unstale":"2024-01-11T08:23:05.853938+0000","last_undegraded":"2024-01-11T08:23:05.853938+0000","last_fullsized":"2024-01-11T08:23:05.853938+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:46:38.189844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.889091+0000","last_change":"2024-01-11T08:23:05.886119+0000","last_active":"2024-01-11T08:23:05.889091+0000","last_peered":"2024-01-11T08:23:05.889091+0000","last_clean":"2024-01-11T08:23:05.889091+0000","last_became_active":"2024-01-11T08:23:05.884572+0000","last_became_peered":"2024-01-11T08:23:05.884572+0000","last_unstale":"2024-01-11T08:23:05.889091+0000","last_undegraded":"2024-01-11T08:23:05.889091+0000","last_fullsized":"2024-01-11T08:23:05.889091+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:59:57.583185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.841119+0000","last_change":"2024-01-11T08:23:04.866816+0000","last_active":"2024-01-11T08:23:05.841119+0000","last_peered":"2024-01-11T08:23:05.841119+0000","last_clean":"2024-01-11T08:23:05.841119+0000","last_became_active":"2024-01-11T08:23:04.866516+0000","last_became_peered":"2024-01-11T08:23:04.866516+0000","last_unstale":"2024-01-11T08:23:05.841119+0000","last_undegraded":"2024-01-11T08:23:05.841119+0000","last_fullsized":"2024-01-11T08:23:05.841119+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:49:00.880132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.865632+0000","last_change":"2024-01-11T08:23:05.863068+0000","last_active":"2024-01-11T08:23:05.865632+0000","last_peered":"2024-01-11T08:23:05.865632+0000","last_clean":"2024-01-11T08:23:05.865632+0000","last_became_active":"2024-01-11T08:23:05.862177+0000","last_became_peered":"2024-01-11T08:23:05.862177+0000","last_unstale":"2024-01-11T08:23:05.865632+0000","last_undegraded":"2024-01-11T08:23:05.865632+0000","last_fullsized":"2024-01-11T08:23:05.865632+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:22:05.338870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"58'8","reported_seq":17,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.877334+0000","last_change":"2024-01-11T08:23:04.867225+0000","last_active":"2024-01-11T08:23:05.877334+0000","last_peered":"2024-01-11T08:23:05.877334+0000","last_clean":"2024-01-11T08:23:05.877334+0000","last_became_active":"2024-01-11T08:23:04.866784+0000","last_became_peered":"2024-01-11T08:23:04.866784+0000","last_unstale":"2024-01-11T08:23:05.877334+0000","last_undegraded":"2024-01-11T08:23:05.877334+0000","last_fullsized":"2024-01-11T08:23:05.877334+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:55:02.211673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.889488+0000","last_change":"2024-01-11T08:23:05.882693+0000","last_active":"2024-01-11T08:23:05.889488+0000","last_peered":"2024-01-11T08:23:05.889488+0000","last_clean":"2024-01-11T08:23:05.889488+0000","last_became_active":"2024-01-11T08:23:05.882457+0000","last_became_peered":"2024-01-11T08:23:05.882457+0000","last_unstale":"2024-01-11T08:23:05.889488+0000","last_undegraded":"2024-01-11T08:23:05.889488+0000","last_fullsized":"2024-01-11T08:23:05.889488+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:10.276798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"58'1","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.888885+0000","last_change":"2024-01-11T08:23:04.849494+0000","last_active":"2024-01-11T08:23:05.888885+0000","last_peered":"2024-01-11T08:23:05.888885+0000","last_clean":"2024-01-11T08:23:05.888885+0000","last_became_active":"2024-01-11T08:23:04.849251+0000","last_became_peered":"2024-01-11T08:23:04.849251+0000","last_unstale":"2024-01-11T08:23:05.888885+0000","last_undegraded":"2024-01-11T08:23:05.888885+0000","last_fullsized":"2024-01-11T08:23:05.888885+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:47:34.139983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.874269+0000","last_change":"2024-01-11T08:23:05.874269+0000","last_active":"2024-01-11T08:23:05.874269+0000","last_peered":"2024-01-11T08:23:05.874269+0000","last_clean":"2024-01-11T08:23:05.874269+0000","last_became_active":"2024-01-11T08:23:05.872970+0000","last_became_peered":"2024-01-11T08:23:05.872970+0000","last_unstale":"2024-01-11T08:23:05.874269+0000","last_undegraded":"2024-01-11T08:23:05.874269+0000","last_fullsized":"2024-01-11T08:23:05.874269+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:22:05.338870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"58'2","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.868865+0000","last_change":"2024-01-11T08:23:04.857254+0000","last_active":"2024-01-11T08:23:05.868865+0000","last_peered":"2024-01-11T08:23:05.868865+0000","last_clean":"2024-01-11T08:23:05.868865+0000","last_became_active":"2024-01-11T08:23:04.856997+0000","last_became_peered":"2024-01-11T08:23:04.856997+0000","last_unstale":"2024-01-11T08:23:05.868865+0000","last_undegraded":"2024-01-11T08:23:05.868865+0000","last_fullsized":"2024-01-11T08:23:05.868865+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:48:29.009973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.859977+0000","last_change":"2024-01-11T08:23:05.856341+0000","last_active":"2024-01-11T08:23:05.859977+0000","last_peered":"2024-01-11T08:23:05.859977+0000","last_clean":"2024-01-11T08:23:05.859977+0000","last_became_active":"2024-01-11T08:23:05.855566+0000","last_became_peered":"2024-01-11T08:23:05.855566+0000","last_unstale":"2024-01-11T08:23:05.859977+0000","last_undegraded":"2024-01-11T08:23:05.859977+0000","last_fullsized":"2024-01-11T08:23:05.859977+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:46:39.199404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"58'1","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.885389+0000","last_change":"2024-01-11T08:23:04.856455+0000","last_active":"2024-01-11T08:23:05.885389+0000","last_peered":"2024-01-11T08:23:05.885389+0000","last_clean":"2024-01-11T08:23:05.885389+0000","last_became_active":"2024-01-11T08:23:04.856005+0000","last_became_peered":"2024-01-11T08:23:04.856005+0000","last_unstale":"2024-01-11T08:23:05.885389+0000","last_undegraded":"2024-01-11T08:23:05.885389+0000","last_fullsized":"2024-01-11T08:23:05.885389+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:44:43.131037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.867634+0000","last_change":"2024-01-11T08:23:05.864974+0000","last_active":"2024-01-11T08:23:05.867634+0000","last_peered":"2024-01-11T08:23:05.867634+0000","last_clean":"2024-01-11T08:23:05.867634+0000","last_became_active":"2024-01-11T08:23:05.864724+0000","last_became_peered":"2024-01-11T08:23:05.864724+0000","last_unstale":"2024-01-11T08:23:05.867634+0000","last_undegraded":"2024-01-11T08:23:05.867634+0000","last_fullsized":"2024-01-11T08:23:05.867634+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:18:54.740633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.842262+0000","last_change":"2024-01-11T08:23:04.852928+0000","last_active":"2024-01-11T08:23:05.842262+0000","last_peered":"2024-01-11T08:23:05.842262+0000","last_clean":"2024-01-11T08:23:05.842262+0000","last_became_active":"2024-01-11T08:23:04.852633+0000","last_became_peered":"2024-01-11T08:23:04.852633+0000","last_unstale":"2024-01-11T08:23:05.842262+0000","last_undegraded":"2024-01-11T08:23:05.842262+0000","last_fullsized":"2024-01-11T08:23:05.842262+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:01:38.977757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.891399+0000","last_change":"2024-01-11T08:23:05.886811+0000","last_active":"2024-01-11T08:23:05.891399+0000","last_peered":"2024-01-11T08:23:05.891399+0000","last_clean":"2024-01-11T08:23:05.891399+0000","last_became_active":"2024-01-11T08:23:05.886505+0000","last_became_peered":"2024-01-11T08:23:05.886505+0000","last_unstale":"2024-01-11T08:23:05.891399+0000","last_undegraded":"2024-01-11T08:23:05.891399+0000","last_fullsized":"2024-01-11T08:23:05.891399+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:58:01.022706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"58'2","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.871539+0000","last_change":"2024-01-11T08:23:04.851210+0000","last_active":"2024-01-11T08:23:05.871539+0000","last_peered":"2024-01-11T08:23:05.871539+0000","last_clean":"2024-01-11T08:23:05.871539+0000","last_became_active":"2024-01-11T08:23:04.850926+0000","last_became_peered":"2024-01-11T08:23:04.850926+0000","last_unstale":"2024-01-11T08:23:05.871539+0000","last_undegraded":"2024-01-11T08:23:05.871539+0000","last_fullsized":"2024-01-11T08:23:05.871539+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:04:55.429486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.885272+0000","last_change":"2024-01-11T08:23:05.878300+0000","last_active":"2024-01-11T08:23:05.885272+0000","last_peered":"2024-01-11T08:23:05.885272+0000","last_clean":"2024-01-11T08:23:05.885272+0000","last_became_active":"2024-01-11T08:23:05.875460+0000","last_became_peered":"2024-01-11T08:23:05.875460+0000","last_unstale":"2024-01-11T08:23:05.885272+0000","last_undegraded":"2024-01-11T08:23:05.885272+0000","last_fullsized":"2024-01-11T08:23:05.885272+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:53:15.590430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.855049+0000","last_change":"2024-01-11T08:23:04.853953+0000","last_active":"2024-01-11T08:23:05.855049+0000","last_peered":"2024-01-11T08:23:05.855049+0000","last_clean":"2024-01-11T08:23:05.855049+0000","last_became_active":"2024-01-11T08:23:04.852271+0000","last_became_peered":"2024-01-11T08:23:04.852271+0000","last_unstale":"2024-01-11T08:23:05.855049+0000","last_undegraded":"2024-01-11T08:23:05.855049+0000","last_fullsized":"2024-01-11T08:23:05.855049+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:46:38.189844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.885492+0000","last_change":"2024-01-11T08:23:05.877143+0000","last_active":"2024-01-11T08:23:05.885492+0000","last_peered":"2024-01-11T08:23:05.885492+0000","last_clean":"2024-01-11T08:23:05.885492+0000","last_became_active":"2024-01-11T08:23:05.872690+0000","last_became_peered":"2024-01-11T08:23:05.872690+0000","last_unstale":"2024-01-11T08:23:05.885492+0000","last_undegraded":"2024-01-11T08:23:05.885492+0000","last_fullsized":"2024-01-11T08:23:05.885492+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:33:45.602596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-11T08:23:04.867138+0000","last_change":"2024-01-11T08:23:04.864969+0000","last_active":"2024-01-11T08:23:04.867138+0000","last_peered":"2024-01-11T08:23:04.867138+0000","last_clean":"2024-01-11T08:23:04.867138+0000","last_became_active":"2024-01-11T08:23:04.864594+0000","last_became_peered":"2024-01-11T08:23:04.864594+0000","last_unstale":"2024-01-11T08:23:04.867138+0000","last_undegraded":"2024-01-11T08:23:04.867138+0000","last_fullsized":"2024-01-11T08:23:04.867138+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:17:24.989859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.874220+0000","last_change":"2024-01-11T08:23:05.874220+0000","last_active":"2024-01-11T08:23:05.874220+0000","last_peered":"2024-01-11T08:23:05.874220+0000","last_clean":"2024-01-11T08:23:05.874220+0000","last_became_active":"2024-01-11T08:23:05.873797+0000","last_became_peered":"2024-01-11T08:23:05.873797+0000","last_unstale":"2024-01-11T08:23:05.874220+0000","last_undegraded":"2024-01-11T08:23:05.874220+0000","last_fullsized":"2024-01-11T08:23:05.874220+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:46:39.199404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.852635+0000","last_change":"2024-01-11T08:23:04.855002+0000","last_active":"2024-01-11T08:23:05.852635+0000","last_peered":"2024-01-11T08:23:05.852635+0000","last_clean":"2024-01-11T08:23:05.852635+0000","last_became_active":"2024-01-11T08:23:04.854716+0000","last_became_peered":"2024-01-11T08:23:04.854716+0000","last_unstale":"2024-01-11T08:23:05.852635+0000","last_undegraded":"2024-01-11T08:23:05.852635+0000","last_fullsized":"2024-01-11T08:23:05.852635+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:26:58.613733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.870106+0000","last_change":"2024-01-11T08:23:05.868723+0000","last_active":"2024-01-11T08:23:05.870106+0000","last_peered":"2024-01-11T08:23:05.870106+0000","last_clean":"2024-01-11T08:23:05.870106+0000","last_became_active":"2024-01-11T08:23:05.868410+0000","last_became_peered":"2024-01-11T08:23:05.868410+0000","last_unstale":"2024-01-11T08:23:05.870106+0000","last_undegraded":"2024-01-11T08:23:05.870106+0000","last_fullsized":"2024-01-11T08:23:05.870106+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:40:30.954875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.842431+0000","last_change":"2024-01-11T08:23:04.861846+0000","last_active":"2024-01-11T08:23:05.842431+0000","last_peered":"2024-01-11T08:23:05.842431+0000","last_clean":"2024-01-11T08:23:05.842431+0000","last_became_active":"2024-01-11T08:23:04.861524+0000","last_became_peered":"2024-01-11T08:23:04.861524+0000","last_unstale":"2024-01-11T08:23:05.842431+0000","last_undegraded":"2024-01-11T08:23:05.842431+0000","last_fullsized":"2024-01-11T08:23:05.842431+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:16:22.514264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.867509+0000","last_change":"2024-01-11T08:23:05.867509+0000","last_active":"2024-01-11T08:23:05.867509+0000","last_peered":"2024-01-11T08:23:05.867509+0000","last_clean":"2024-01-11T08:23:05.867509+0000","last_became_active":"2024-01-11T08:23:05.867227+0000","last_became_peered":"2024-01-11T08:23:05.867227+0000","last_unstale":"2024-01-11T08:23:05.867509+0000","last_undegraded":"2024-01-11T08:23:05.867509+0000","last_fullsized":"2024-01-11T08:23:05.867509+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:04:53.426280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.857085+0000","last_change":"2024-01-11T08:23:04.864163+0000","last_active":"2024-01-11T08:23:05.857085+0000","last_peered":"2024-01-11T08:23:05.857085+0000","last_clean":"2024-01-11T08:23:05.857085+0000","last_became_active":"2024-01-11T08:23:04.863911+0000","last_became_peered":"2024-01-11T08:23:04.863911+0000","last_unstale":"2024-01-11T08:23:05.857085+0000","last_undegraded":"2024-01-11T08:23:05.857085+0000","last_fullsized":"2024-01-11T08:23:05.857085+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:22:32.656105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.887275+0000","last_change":"2024-01-11T08:23:05.886605+0000","last_active":"2024-01-11T08:23:05.887275+0000","last_peered":"2024-01-11T08:23:05.887275+0000","last_clean":"2024-01-11T08:23:05.887275+0000","last_became_active":"2024-01-11T08:23:05.886284+0000","last_became_peered":"2024-01-11T08:23:05.886284+0000","last_unstale":"2024-01-11T08:23:05.887275+0000","last_undegraded":"2024-01-11T08:23:05.887275+0000","last_fullsized":"2024-01-11T08:23:05.887275+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:16:23.523824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.862833+0000","last_change":"2024-01-11T08:23:04.857360+0000","last_active":"2024-01-11T08:23:05.862833+0000","last_peered":"2024-01-11T08:23:05.862833+0000","last_clean":"2024-01-11T08:23:05.862833+0000","last_became_active":"2024-01-11T08:23:04.857051+0000","last_became_peered":"2024-01-11T08:23:04.857051+0000","last_unstale":"2024-01-11T08:23:05.862833+0000","last_undegraded":"2024-01-11T08:23:05.862833+0000","last_fullsized":"2024-01-11T08:23:05.862833+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:22:32.656105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.856039+0000","last_change":"2024-01-11T08:23:05.852346+0000","last_active":"2024-01-11T08:23:05.856039+0000","last_peered":"2024-01-11T08:23:05.856039+0000","last_clean":"2024-01-11T08:23:05.856039+0000","last_became_active":"2024-01-11T08:23:05.852074+0000","last_became_peered":"2024-01-11T08:23:05.852074+0000","last_unstale":"2024-01-11T08:23:05.856039+0000","last_undegraded":"2024-01-11T08:23:05.856039+0000","last_fullsized":"2024-01-11T08:23:05.856039+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:03:28.430072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.854174+0000","last_change":"2024-01-11T08:23:04.862981+0000","last_active":"2024-01-11T08:23:05.854174+0000","last_peered":"2024-01-11T08:23:05.854174+0000","last_clean":"2024-01-11T08:23:05.854174+0000","last_became_active":"2024-01-11T08:23:04.862681+0000","last_became_peered":"2024-01-11T08:23:04.862681+0000","last_unstale":"2024-01-11T08:23:05.854174+0000","last_undegraded":"2024-01-11T08:23:05.854174+0000","last_fullsized":"2024-01-11T08:23:05.854174+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:17:24.867381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.885572+0000","last_change":"2024-01-11T08:23:05.884668+0000","last_active":"2024-01-11T08:23:05.885572+0000","last_peered":"2024-01-11T08:23:05.885572+0000","last_clean":"2024-01-11T08:23:05.885572+0000","last_became_active":"2024-01-11T08:23:05.882603+0000","last_became_peered":"2024-01-11T08:23:05.882603+0000","last_unstale":"2024-01-11T08:23:05.885572+0000","last_undegraded":"2024-01-11T08:23:05.885572+0000","last_fullsized":"2024-01-11T08:23:05.885572+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:04:07.668684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.846954+0000","last_change":"2024-01-11T08:23:04.859381+0000","last_active":"2024-01-11T08:23:05.846954+0000","last_peered":"2024-01-11T08:23:05.846954+0000","last_clean":"2024-01-11T08:23:05.846954+0000","last_became_active":"2024-01-11T08:23:04.859101+0000","last_became_peered":"2024-01-11T08:23:04.859101+0000","last_unstale":"2024-01-11T08:23:05.846954+0000","last_undegraded":"2024-01-11T08:23:05.846954+0000","last_fullsized":"2024-01-11T08:23:05.846954+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:56:13.627003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.872166+0000","last_change":"2024-01-11T08:23:05.869943+0000","last_active":"2024-01-11T08:23:05.872166+0000","last_peered":"2024-01-11T08:23:05.872166+0000","last_clean":"2024-01-11T08:23:05.872166+0000","last_became_active":"2024-01-11T08:23:05.868835+0000","last_became_peered":"2024-01-11T08:23:05.868835+0000","last_unstale":"2024-01-11T08:23:05.872166+0000","last_undegraded":"2024-01-11T08:23:05.872166+0000","last_fullsized":"2024-01-11T08:23:05.872166+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:36:07.478679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.856492+0000","last_change":"2024-01-11T08:23:04.860552+0000","last_active":"2024-01-11T08:23:05.856492+0000","last_peered":"2024-01-11T08:23:05.856492+0000","last_clean":"2024-01-11T08:23:05.856492+0000","last_became_active":"2024-01-11T08:23:04.860118+0000","last_became_peered":"2024-01-11T08:23:04.860118+0000","last_unstale":"2024-01-11T08:23:05.856492+0000","last_undegraded":"2024-01-11T08:23:05.856492+0000","last_fullsized":"2024-01-11T08:23:05.856492+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:17:24.867381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.862432+0000","last_change":"2024-01-11T08:23:05.860271+0000","last_active":"2024-01-11T08:23:05.862432+0000","last_peered":"2024-01-11T08:23:05.862432+0000","last_clean":"2024-01-11T08:23:05.862432+0000","last_became_active":"2024-01-11T08:23:05.859968+0000","last_became_peered":"2024-01-11T08:23:05.859968+0000","last_unstale":"2024-01-11T08:23:05.862432+0000","last_undegraded":"2024-01-11T08:23:05.862432+0000","last_fullsized":"2024-01-11T08:23:05.862432+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:04:56.439046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-11T08:23:04.868484+0000","last_change":"2024-01-11T08:23:04.866026+0000","last_active":"2024-01-11T08:23:04.868484+0000","last_peered":"2024-01-11T08:23:04.868484+0000","last_clean":"2024-01-11T08:23:04.868484+0000","last_became_active":"2024-01-11T08:23:04.864799+0000","last_became_peered":"2024-01-11T08:23:04.864799+0000","last_unstale":"2024-01-11T08:23:04.868484+0000","last_undegraded":"2024-01-11T08:23:04.868484+0000","last_fullsized":"2024-01-11T08:23:04.868484+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:14:20.580853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.876603+0000","last_change":"2024-01-11T08:23:05.876603+0000","last_active":"2024-01-11T08:23:05.876603+0000","last_peered":"2024-01-11T08:23:05.876603+0000","last_clean":"2024-01-11T08:23:05.876603+0000","last_became_active":"2024-01-11T08:23:05.876336+0000","last_became_peered":"2024-01-11T08:23:05.876336+0000","last_unstale":"2024-01-11T08:23:05.876603+0000","last_undegraded":"2024-01-11T08:23:05.876603+0000","last_fullsized":"2024-01-11T08:23:05.876603+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:52:16.057511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"58'2","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.870384+0000","last_change":"2024-01-11T08:23:04.855083+0000","last_active":"2024-01-11T08:23:05.870384+0000","last_peered":"2024-01-11T08:23:05.870384+0000","last_clean":"2024-01-11T08:23:05.870384+0000","last_became_active":"2024-01-11T08:23:04.854808+0000","last_became_peered":"2024-01-11T08:23:04.854808+0000","last_unstale":"2024-01-11T08:23:05.870384+0000","last_undegraded":"2024-01-11T08:23:05.870384+0000","last_fullsized":"2024-01-11T08:23:05.870384+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:20:12.807666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"49'139","reported_seq":271,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.852819+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:23:05.852819+0000","last_peered":"2024-01-11T08:23:05.852819+0000","last_clean":"2024-01-11T08:23:05.852819+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:23:05.852819+0000","last_undegraded":"2024-01-11T08:23:05.852819+0000","last_fullsized":"2024-01-11T08:23:05.852819+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575004,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302652,"kb_used_data":1268,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069188,"statfs":{"total":96636764160,"available":96326848512,"internally_reserved":0,"allocated":1298432,"data_stored":908723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.70699999999999996,"15min":0.70499999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.996,"5min":0.996,"15min":2.8300000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.72299999999999998,"15min":0.70299999999999996},"min":{"1min":0.503,"5min":0.46200000000000002,"15min":0.372},"max":{"1min":0.91500000000000004,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.752}]},{"osd":1,"last update":"Thu Jan 11 08:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.71099999999999997,"15min":0.69299999999999995},"min":{"1min":0.45600000000000002,"5min":0.41899999999999998,"15min":0.35799999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.998},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.73099999999999998,"15min":0.71599999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.95899999999999996,"5min":1.014,"15min":1.153},"last":0.56599999999999995}]},{"osd":2,"last update":"Thu Jan 11 08:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.748,"15min":0.71799999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.92900000000000005,"5min":1.03,"15min":1.034},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.73499999999999999,"15min":0.71799999999999997},"min":{"1min":0.46999999999999997,"5min":0.46400000000000002,"15min":0.41099999999999998},"max":{"1min":0.99399999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.70099999999999996}]},{"osd":3,"last update":"Thu Jan 11 08:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.75600000000000001,"15min":0.76800000000000002},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.40300000000000002},"max":{"1min":1.032,"5min":1.032,"15min":2.9119999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.748,"15min":0.73999999999999999},"min":{"1min":0.50800000000000001,"5min":0.502,"15min":0.48999999999999999},"max":{"1min":0.93300000000000005,"5min":0.97599999999999998,"15min":0.97799999999999998},"last":0.73899999999999999}]},{"osd":4,"last update":"Thu Jan 11 08:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59399999999999997,"15min":0.61799999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":1.089},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.625,"5min":0.627,"15min":0.64500000000000002},"min":{"1min":0.442,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.872,"5min":0.93400000000000005,"15min":1.1020000000000001},"last":0.82299999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.63700000000000001,"15min":0.65800000000000003},"min":{"1min":0.42199999999999999,"5min":0.41699999999999998,"15min":0.34599999999999997},"max":{"1min":0.92200000000000004,"5min":0.92800000000000005,"15min":1.0589999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66800000000000004,"15min":0.68899999999999995},"min":{"1min":0.47199999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.032},"last":0.89000000000000001}]},{"osd":6,"last update":"Thu Jan 11 08:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.76000000000000001,"15min":0.77000000000000002},"min":{"1min":0.55800000000000005,"5min":0.501,"15min":0.46999999999999997},"max":{"1min":0.94099999999999995,"5min":1.0009999999999999,"15min":1.1359999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.73299999999999998,"15min":0.754},"min":{"1min":0.57299999999999995,"5min":0.45200000000000001,"15min":0.42099999999999999},"max":{"1min":0.99099999999999999,"5min":1.018,"15min":1.1299999999999999},"last":0.76500000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575003,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302064,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069776,"statfs":{"total":96636764160,"available":96327450624,"internally_reserved":0,"allocated":696320,"data_stored":312621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.66700000000000004,"15min":0.64800000000000002},"min":{"1min":0.38700000000000001,"5min":0.379,"15min":0.29999999999999999},"max":{"1min":0.85399999999999998,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.70199999999999996,"15min":0.67300000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":0.91200000000000003,"5min":0.96999999999999997,"15min":1},"last":0.60499999999999998}]},{"osd":1,"last update":"Thu Jan 11 08:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.70799999999999996,"15min":0.66900000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.35899999999999999},"max":{"1min":0.94699999999999995,"5min":1.163,"15min":1.163},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.70499999999999996,"15min":0.69399999999999995},"min":{"1min":0.52300000000000002,"5min":0.47599999999999998,"15min":0.39900000000000002},"max":{"1min":0.92900000000000005,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.55400000000000005}]},{"osd":2,"last update":"Thu Jan 11 08:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.71799999999999997,"15min":0.69699999999999995},"min":{"1min":0.50600000000000001,"5min":0.48399999999999999,"15min":0.38600000000000001},"max":{"1min":0.97199999999999998,"5min":1.016,"15min":1.016},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.73199999999999998,"15min":0.71599999999999997},"min":{"1min":0.53600000000000003,"5min":0.46200000000000002,"15min":0.373},"max":{"1min":0.88900000000000001,"5min":0.96899999999999997,"15min":0.995},"last":0.72699999999999998}]},{"osd":3,"last update":"Thu Jan 11 08:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.73199999999999998,"15min":0.73599999999999999},"min":{"1min":0.47699999999999998,"5min":0.46800000000000003,"15min":0.42799999999999999},"max":{"1min":0.95799999999999996,"5min":1.0149999999999999,"15min":1.3260000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.75800000000000001,"15min":0.73099999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.44800000000000001},"max":{"1min":0.95899999999999996,"5min":0.97999999999999998,"15min":1.0049999999999999},"last":0.75900000000000001}]},{"osd":4,"last update":"Thu Jan 11 08:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54800000000000004,"15min":0.58899999999999997},"min":{"1min":0.39700000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.874,"5min":0.874,"15min":1.212},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.58599999999999997,"15min":0.63700000000000001},"min":{"1min":0.41499999999999998,"5min":0.38900000000000001,"15min":0.374},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":1.8320000000000001},"last":0.66800000000000004}]},{"osd":5,"last update":"Thu Jan 11 08:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64700000000000002,"15min":0.66600000000000004},"min":{"1min":0.443,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.85699999999999998,"5min":0.92800000000000005,"15min":1.0209999999999999},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.69999999999999996,"15min":0.72399999999999998},"min":{"1min":0.55300000000000005,"5min":0.47399999999999998,"15min":0.41899999999999998},"max":{"1min":0.88700000000000001,"5min":0.93700000000000006,"15min":1.7929999999999999},"last":0.87}]},{"osd":7,"last update":"Thu Jan 11 08:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.69999999999999996,"15min":0.73299999999999998},"min":{"1min":0.56799999999999995,"5min":0.42399999999999999,"15min":0.41099999999999998},"max":{"1min":0.84399999999999997,"5min":0.90900000000000003,"15min":1.278},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.75900000000000001,"15min":0.75700000000000001},"min":{"1min":0.623,"5min":0.48299999999999998,"15min":0.44700000000000001},"max":{"1min":0.94799999999999995,"5min":0.99399999999999999,"15min":1.1599999999999999},"last":0.98399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575005,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302060,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069780,"statfs":{"total":96636764160,"available":96327454720,"internally_reserved":0,"allocated":692224,"data_stored":312047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.32100000000000001,"5min":0.317,"15min":0.26900000000000002},"max":{"1min":0.624,"5min":0.74399999999999999,"15min":0.97699999999999998},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59899999999999998,"15min":0.57299999999999995},"min":{"1min":0.437,"5min":0.436,"15min":0.373},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.90700000000000003},"last":0.51400000000000001}]},{"osd":2,"last update":"Thu Jan 11 08:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.5,"15min":0.48899999999999999},"min":{"1min":0.35699999999999998,"5min":0.33400000000000002,"15min":0.32400000000000001},"max":{"1min":0.66000000000000003,"5min":1.113,"15min":1.113},"last":0.432},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56399999999999995,"15min":0.56899999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":1.25},"last":0.48999999999999999}]},{"osd":3,"last update":"Thu Jan 11 08:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.621,"15min":0.61599999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73699999999999999,"5min":0.86599999999999999,"15min":0.92600000000000005},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57899999999999996,"15min":0.56699999999999995},"min":{"1min":0.374,"5min":0.372,"15min":0.34899999999999998},"max":{"1min":0.93400000000000005,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.46000000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.81299999999999994,"15min":0.79200000000000004},"min":{"1min":0.55600000000000005,"5min":0.45700000000000002,"15min":0.44600000000000001},"max":{"1min":0.98999999999999999,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.84499999999999997,"15min":0.83899999999999997},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56000000000000005},"max":{"1min":1.0740000000000001,"5min":1.246,"15min":1.3680000000000001},"last":0.73999999999999999}]},{"osd":5,"last update":"Thu Jan 11 08:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.82299999999999995,"15min":0.81399999999999995},"min":{"1min":0.503,"5min":0.503,"15min":0.46999999999999997},"max":{"1min":1.052,"5min":1.1759999999999999,"15min":1.7849999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.80600000000000005,"15min":0.80900000000000005},"min":{"1min":0.52700000000000002,"5min":0.51200000000000001,"15min":0.50700000000000001},"max":{"1min":1.0129999999999999,"5min":1.1579999999999999,"15min":1.371},"last":0.77000000000000002}]},{"osd":6,"last update":"Thu Jan 11 08:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.83199999999999996,"15min":0.82499999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0600000000000001,"5min":1.0840000000000001,"15min":1.413},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.84999999999999998,"15min":0.82199999999999995},"min":{"1min":0.629,"5min":0.56999999999999995,"15min":0.55500000000000005},"max":{"1min":1.087,"5min":1.228,"15min":1.228},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Jan 11 08:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.875,"15min":0.84499999999999997},"min":{"1min":0.65900000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1180000000000001,"5min":1.204,"15min":1.226},"last":0.629},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.85299999999999998,"15min":0.84099999999999997},"min":{"1min":0.58599999999999997,"5min":0.57999999999999996,"15min":0.51400000000000001},"max":{"1min":1.0369999999999999,"5min":1.258,"15min":1.321},"last":0.73099999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575004,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298544,"kb_used_data":1256,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073296,"statfs":{"total":96636764160,"available":96331055104,"internally_reserved":0,"allocated":1286144,"data_stored":902379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 08:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.49099999999999999,"15min":0.495},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.24299999999999999},"max":{"1min":0.625,"5min":0.89700000000000002,"15min":0.92700000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.35299999999999998},"max":{"1min":0.68799999999999994,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.63}]},{"osd":2,"last update":"Thu Jan 11 08:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.53400000000000003,"15min":0.54400000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.29099999999999998},"max":{"1min":0.60799999999999998,"5min":0.77400000000000002,"15min":0.86799999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77500000000000002,"5min":0.81299999999999994,"15min":0.83999999999999997},"last":0.56000000000000005}]},{"osd":3,"last update":"Thu Jan 11 08:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52300000000000002,"15min":0.51500000000000001},"min":{"1min":0.39900000000000002,"5min":0.36199999999999999,"15min":0.33600000000000002},"max":{"1min":0.71899999999999997,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.437},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61899999999999999,"15min":0.61799999999999999},"min":{"1min":0.46400000000000002,"5min":0.40400000000000003,"15min":0.39600000000000002},"max":{"1min":0.69999999999999996,"5min":0.877,"15min":0.91000000000000003},"last":0.59999999999999998}]},{"osd":4,"last update":"Thu Jan 11 08:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.78400000000000003,"15min":0.77900000000000003},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.437},"max":{"1min":0.91100000000000003,"5min":1.165,"15min":1.165},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.78300000000000003,"15min":0.78200000000000003},"min":{"1min":0.53100000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.038,"5min":1.038,"15min":1.2350000000000001},"last":0.58199999999999996}]},{"osd":5,"last update":"Thu Jan 11 08:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.80200000000000005,"15min":0.80600000000000005},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.505},"max":{"1min":0.98899999999999999,"5min":1.048,"15min":1.2929999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.80300000000000005,"15min":0.81200000000000006},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.47799999999999998},"max":{"1min":0.96099999999999997,"5min":1.0800000000000001,"15min":1.2130000000000001},"last":0.71899999999999997}]},{"osd":6,"last update":"Thu Jan 11 08:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.79600000000000004,"15min":0.79500000000000004},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.497},"max":{"1min":0.97999999999999998,"5min":1.1419999999999999,"15min":1.2689999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.81699999999999995,"15min":0.83199999999999996},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.44900000000000001},"max":{"1min":0.93999999999999995,"5min":1.1339999999999999,"15min":1.2749999999999999},"last":0.73599999999999999}]},{"osd":7,"last update":"Thu Jan 11 08:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.84799999999999998,"15min":0.84199999999999997},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.51400000000000001},"max":{"1min":1.0680000000000001,"5min":1.21,"15min":1.446},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.83899999999999997,"15min":0.83999999999999997},"min":{"1min":0.622,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.0620000000000001,"5min":1.1930000000000001,"15min":1.298},"last":0.69499999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575003,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306180,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065660,"statfs":{"total":96636764160,"available":96323235840,"internally_reserved":0,"allocated":716800,"data_stored":319823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46800000000000003,"15min":0.46500000000000002},"min":{"1min":0.307,"5min":0.30299999999999999,"15min":0.26000000000000001},"max":{"1min":0.73499999999999999,"5min":0.82799999999999996,"15min":0.86099999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.55500000000000005,"15min":0.55900000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.28899999999999998},"max":{"1min":0.624,"5min":0.91400000000000003,"15min":1.4119999999999999},"last":0.39700000000000002}]},{"osd":1,"last update":"Thu Jan 11 08:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.30399999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.90000000000000002},"last":0.372},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59199999999999997,"15min":0.59699999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.38100000000000001},"max":{"1min":0.92900000000000005,"5min":0.94499999999999995,"15min":1.5640000000000001},"last":0.442}]},{"osd":3,"last update":"Thu Jan 11 08:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.54300000000000004,"15min":0.55000000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.32900000000000001},"max":{"1min":0.67700000000000005,"5min":0.84999999999999998,"15min":1.1220000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63600000000000001,"15min":0.65000000000000002},"min":{"1min":0.45300000000000001,"5min":0.439,"15min":0.41799999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.22},"last":0.50700000000000001}]},{"osd":4,"last update":"Thu Jan 11 08:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.81100000000000005,"15min":0.79800000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.97999999999999998,"5min":1.2889999999999999,"15min":1.488},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.81200000000000006,"15min":0.79100000000000004},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.75900000000000001}]},{"osd":5,"last update":"Thu Jan 11 08:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.85499999999999998,"15min":0.86599999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.51600000000000001},"max":{"1min":1.214,"5min":1.3540000000000001,"15min":2.9209999999999998},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.85599999999999998,"15min":0.86099999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55400000000000005},"max":{"1min":1.1399999999999999,"5min":1.23,"15min":1.845},"last":0.98299999999999998}]},{"osd":6,"last update":"Thu Jan 11 08:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82799999999999996,"15min":0.80600000000000005},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.51700000000000002},"max":{"1min":1.1919999999999999,"5min":1.2669999999999999,"15min":1.3560000000000001},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.84399999999999997,"15min":0.85599999999999998},"min":{"1min":0.66800000000000004,"5min":0.61399999999999999,"15min":0.56299999999999994},"max":{"1min":1.123,"5min":1.47,"15min":2.968},"last":0.81899999999999995}]},{"osd":7,"last update":"Thu Jan 11 08:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.879,"15min":0.88300000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.2330000000000001,"5min":1.4099999999999999,"15min":1.6359999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.88600000000000001,"15min":0.878},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.53800000000000003},"max":{"1min":1.0900000000000001,"5min":1.327,"15min":1.391},"last":0.90800000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306176,"kb_used_data":696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065664,"statfs":{"total":96636764160,"available":96323239936,"internally_reserved":0,"allocated":712704,"data_stored":318875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.49299999999999999,"15min":0.48399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.27600000000000002},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.55700000000000005,"15min":0.54500000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.64600000000000002,"5min":0.75600000000000001,"15min":1.0109999999999999},"last":0.46300000000000002}]},{"osd":1,"last update":"Thu Jan 11 08:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.502,"15min":0.48699999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.25900000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59899999999999998,"15min":0.56999999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.32300000000000001},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Jan 11 08:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.59899999999999998,"15min":0.58599999999999997},"min":{"1min":0.39900000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.90900000000000003,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.44700000000000001,"5min":0.439,"15min":0.32000000000000001},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.45200000000000001}]},{"osd":4,"last update":"Thu Jan 11 08:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.73499999999999999,"15min":0.76100000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96099999999999997,"5min":0.997,"15min":1.351},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.78400000000000003,"15min":0.78600000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.51300000000000001},"max":{"1min":1.0569999999999999,"5min":1.1819999999999999,"15min":1.198},"last":0.71799999999999997}]},{"osd":5,"last update":"Thu Jan 11 08:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.79500000000000004,"15min":0.80300000000000005},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.51900000000000002},"max":{"1min":1.139,"5min":1.139,"15min":1.3779999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.499,"5min":0.499,"15min":0.44900000000000001},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.4239999999999999},"last":0.67800000000000005}]},{"osd":6,"last update":"Thu Jan 11 08:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.86199999999999999,"15min":0.85599999999999998},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.52600000000000002},"max":{"1min":1.335,"5min":1.335,"15min":1.927},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.80100000000000005,"15min":0.84799999999999998},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.54400000000000004},"max":{"1min":1.0569999999999999,"5min":1.2230000000000001,"15min":3.5579999999999998},"last":0.73499999999999999}]},{"osd":7,"last update":"Thu Jan 11 08:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.84299999999999997,"15min":0.88100000000000001},"min":{"1min":0.58899999999999997,"5min":0.52800000000000002,"15min":0.46600000000000003},"max":{"1min":1.1220000000000001,"5min":1.1579999999999999,"15min":3.5089999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.872,"15min":0.86399999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.59199999999999997},"max":{"1min":1.1879999999999999,"5min":1.204,"15min":1.21},"last":0.89100000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306176,"kb_used_data":696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065664,"statfs":{"total":96636764160,"available":96323239936,"internally_reserved":0,"allocated":712704,"data_stored":319801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.75600000000000001,"15min":0.73699999999999999},"min":{"1min":0.45600000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.95999999999999996,"5min":1.117,"15min":1.524},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.748,"15min":0.749},"min":{"1min":0.51600000000000001,"5min":0.47399999999999998,"15min":0.38600000000000001},"max":{"1min":0.94799999999999995,"5min":0.96999999999999997,"15min":1.177},"last":0.86399999999999999}]},{"osd":1,"last update":"Thu Jan 11 08:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.46300000000000002,"5min":0.42199999999999999,"15min":0.33300000000000002},"max":{"1min":0.88700000000000001,"5min":0.99099999999999999,"15min":1.1499999999999999},"last":0.872},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.71899999999999997,"15min":0.753},"min":{"1min":0.379,"5min":0.379,"15min":0.36299999999999999},"max":{"1min":0.874,"5min":1.046,"15min":2.4740000000000002},"last":0.85399999999999998}]},{"osd":2,"last update":"Thu Jan 11 08:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.72499999999999998,"15min":0.70899999999999996},"min":{"1min":0.53900000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94299999999999995,"5min":0.98599999999999999,"15min":1.022},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.75,"5min":0.748,"15min":0.73199999999999998},"min":{"1min":0.56699999999999995,"5min":0.48399999999999999,"15min":0.47199999999999998},"max":{"1min":0.92200000000000004,"5min":0.95699999999999996,"15min":1.212},"last":0.91500000000000004}]},{"osd":3,"last update":"Thu Jan 11 08:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.71199999999999997,"15min":0.71799999999999997},"min":{"1min":0.52700000000000002,"5min":0.433,"15min":0.38100000000000001},"max":{"1min":0.88800000000000001,"5min":1.0189999999999999,"15min":1.2649999999999999},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.747,"15min":0.75700000000000001},"min":{"1min":0.61199999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.89900000000000002,"5min":1.02,"15min":1.069},"last":0.82299999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60499999999999998,"15min":0.64200000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.35899999999999999},"max":{"1min":0.83599999999999997,"5min":0.93200000000000005,"15min":1.0700000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.623,"15min":0.65700000000000003},"min":{"1min":0.442,"5min":0.40400000000000003,"15min":0.32600000000000001},"max":{"1min":0.85899999999999999,"5min":0.94199999999999995,"15min":0.98199999999999998},"last":0.73899999999999999}]},{"osd":6,"last update":"Thu Jan 11 08:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.63700000000000001,"15min":0.67700000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85799999999999998,"5min":0.86599999999999999,"15min":1.0289999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.67800000000000005,"15min":0.72199999999999998},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.40400000000000003},"max":{"1min":0.878,"5min":0.91600000000000004,"15min":1.119},"last":0.79900000000000004}]},{"osd":7,"last update":"Thu Jan 11 08:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.66600000000000004,"15min":0.71499999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.90600000000000003,"5min":0.96999999999999997,"15min":0.98999999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.72299999999999998,"15min":0.76500000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.42299999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.4279999999999999},"last":0.90700000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575004,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302072,"kb_used_data":688,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069768,"statfs":{"total":96636764160,"available":96327442432,"internally_reserved":0,"allocated":704512,"data_stored":318335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.72999999999999998,"15min":0.70199999999999996},"min":{"1min":0.45900000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89700000000000002,"5min":1.032,"15min":1.032},"last":0.628},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.71899999999999997,"15min":0.69699999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44400000000000001},"max":{"1min":0.91400000000000003,"5min":1.0609999999999999,"15min":1.696},"last":0.64900000000000002}]},{"osd":1,"last update":"Thu Jan 11 08:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70999999999999996,"15min":0.69699999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.36399999999999999},"max":{"1min":0.84299999999999997,"5min":1.0089999999999999,"15min":1.4359999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.74299999999999999,"15min":0.73199999999999998},"min":{"1min":0.59599999999999997,"5min":0.51400000000000001,"15min":0.47299999999999998},"max":{"1min":0.88400000000000001,"5min":0.96799999999999997,"15min":4.4969999999999999},"last":0.752}]},{"osd":2,"last update":"Thu Jan 11 08:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.73399999999999999,"15min":0.70199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.92600000000000005,"5min":1.028,"15min":1.238},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.73799999999999999,"15min":0.751},"min":{"1min":0.49399999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.84199999999999997,"5min":1.0049999999999999,"15min":8.0429999999999993},"last":0.59999999999999998}]},{"osd":3,"last update":"Thu Jan 11 08:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.745,"15min":0.76500000000000001},"min":{"1min":0.50700000000000001,"5min":0.46999999999999997,"15min":0.442},"max":{"1min":0.93400000000000005,"5min":1.0449999999999999,"15min":8.0839999999999996},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.75600000000000001,"15min":0.749},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.872,"5min":0.93899999999999995,"15min":0.96999999999999997},"last":0.56599999999999995}]},{"osd":4,"last update":"Thu Jan 11 08:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61299999999999999,"15min":0.63600000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83699999999999997,"5min":0.96599999999999997,"15min":1.113},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.63100000000000001,"15min":0.66800000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.38200000000000001},"max":{"1min":0.78800000000000003,"5min":0.94399999999999995,"15min":1.002},"last":0.81799999999999995}]},{"osd":6,"last update":"Thu Jan 11 08:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65800000000000003,"15min":0.66700000000000004},"min":{"1min":0.46300000000000002,"5min":0.443,"15min":0.42699999999999999},"max":{"1min":0.86899999999999999,"5min":0.92100000000000004,"15min":1.034},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.70099999999999996,"15min":0.71099999999999997},"min":{"1min":0.53400000000000003,"5min":0.48999999999999999,"15min":0.45400000000000001},"max":{"1min":0.90100000000000002,"5min":1.004,"15min":1.143},"last":0.72399999999999998}]},{"osd":7,"last update":"Thu Jan 11 08:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.749,"15min":0.76600000000000001},"min":{"1min":0.53200000000000003,"5min":0.501,"15min":0.437},"max":{"1min":0.90800000000000003,"5min":1.028,"15min":1.1399999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.78600000000000003,"15min":0.78500000000000003},"min":{"1min":0.56799999999999995,"5min":0.53500000000000003,"15min":0.50600000000000001},"max":{"1min":0.96499999999999997,"5min":1.0489999999999999,"15min":1.079},"last":0.76700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:23:10.282 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:23:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.517+0000 7f322afff700 1 Processor -- start 2024-01-11T08:23:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.518+0000 7f322afff700 1 -- start start 2024-01-11T08:23:10.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.519+0000 7f322afff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f32240ca670 0x7f3224068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:10.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.519+0000 7f322afff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3224068b60 0x7f32240aa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.519+0000 7f322afff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322415da10 con 0x7f32241596c0 2024-01-11T08:23:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.519+0000 7f322afff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3224068fc0 con 0x7f32240ca670 2024-01-11T08:23:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.519+0000 7f322afff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32240a9cb0 con 0x7f3224068b60 2024-01-11T08:23:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.519+0000 7f3223fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f32240ca670 0x7f3224068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.519+0000 7f3228d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f32241596c0 0x7f322415d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37818/0 (socket says 172.21.15.27:37818) 2024-01-11T08:23:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.519+0000 7f3228d9b700 1 -- 172.21.15.27:0/2017319447 learned_addr learned my addr 172.21.15.27:0/2017319447 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:23:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.520+0000 7f322959c700 1 --2- 172.21.15.27:0/2017319447 >> v2:172.21.15.77:3300/0 conn(0x7f3224068b60 0x7f32240aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.520+0000 7f32237fe700 1 -- 172.21.15.27:0/2017319447 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354348680 0 0) 0x7f3218001110 con 0x7f32241596c0 2024-01-11T08:23:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.520+0000 7f32237fe700 1 -- 172.21.15.27:0/2017319447 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3214003570 con 0x7f32241596c0 2024-01-11T08:23:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.520+0000 7f3223fff700 1 -- 172.21.15.27:0/2017319447 >> v2:172.21.15.77:3300/0 conn(0x7f3224068b60 msgr2=0x7f32240aa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.520+0000 7f3223fff700 1 --2- 172.21.15.27:0/2017319447 >> v2:172.21.15.77:3300/0 conn(0x7f3224068b60 0x7f32240aa140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.520+0000 7f3223fff700 1 -- 172.21.15.27:0/2017319447 >> v1:172.21.15.27:6789/0 conn(0x7f32241596c0 legacy=0x7f322415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.520+0000 7f3223fff700 1 -- 172.21.15.27:0/2017319447 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322415e210 con 0x7f32240ca670 2024-01-11T08:23:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.520+0000 7f3223fff700 1 --2- 172.21.15.27:0/2017319447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32240ca670 0x7f3224068620 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3224069a50 tx=0x7f32100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=140909a26a8b46ad server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.520+0000 7f32237fe700 1 -- 172.21.15.27:0/2017319447 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3210011630 con 0x7f32240ca670 2024-01-11T08:23:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.520+0000 7f32237fe700 1 -- 172.21.15.27:0/2017319447 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3210011790 con 0x7f32240ca670 2024-01-11T08:23:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.520+0000 7f32237fe700 1 -- 172.21.15.27:0/2017319447 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3210011b30 con 0x7f32240ca670 2024-01-11T08:23:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.521+0000 7f322afff700 1 -- 172.21.15.27:0/2017319447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32240ca670 msgr2=0x7f3224068620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.521+0000 7f322afff700 1 --2- 172.21.15.27:0/2017319447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32240ca670 0x7f3224068620 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3224069a50 tx=0x7f32100091a0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.521+0000 7f322afff700 1 -- 172.21.15.27:0/2017319447 shutdown_connections 2024-01-11T08:23:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.521+0000 7f322afff700 1 --2- 172.21.15.27:0/2017319447 >> v2:172.21.15.77:3300/0 conn(0x7f3224068b60 0x7f32240aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.521+0000 7f322afff700 1 --2- 172.21.15.27:0/2017319447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32240ca670 0x7f3224068620 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.521+0000 7f322afff700 1 -- 172.21.15.27:0/2017319447 >> 172.21.15.27:0/2017319447 conn(0x7f32240c9f70 msgr2=0x7f322406deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.521+0000 7f322afff700 1 -- 172.21.15.27:0/2017319447 shutdown_connections 2024-01-11T08:23:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.521+0000 7f322afff700 1 -- 172.21.15.27:0/2017319447 wait complete. 2024-01-11T08:23:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.523+0000 7f322afff700 1 Processor -- start 2024-01-11T08:23:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.523+0000 7f322afff700 1 -- start start 2024-01-11T08:23:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.524+0000 7f322afff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3224068b60 0x7f322416a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.524+0000 7f322afff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32240ca670 0x7f3224155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.524+0000 7f322afff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32241596c0 0x7f3224156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.524+0000 7f322afff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32240b1480 con 0x7f32241596c0 2024-01-11T08:23:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.524+0000 7f322afff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32240b1a00 con 0x7f3224068b60 2024-01-11T08:23:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.524+0000 7f322afff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322416af50 con 0x7f32240ca670 2024-01-11T08:23:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.524+0000 7f322959c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32241596c0 0x7f3224156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.524+0000 7f322959c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32241596c0 0x7f3224156280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33692/0 (socket says 172.21.15.27:33692) 2024-01-11T08:23:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.524+0000 7f322959c700 1 -- 172.21.15.27:0/57261072 learned_addr learned my addr 172.21.15.27:0/57261072 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.524+0000 7f3223fff700 1 --2- 172.21.15.27:0/57261072 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32240ca670 0x7f3224155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.524+0000 7f3228d9b700 1 --2- 172.21.15.27:0/57261072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3224068b60 0x7f322416a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.525+0000 7f322959c700 1 -- 172.21.15.27:0/57261072 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32240ca670 msgr2=0x7f3224155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:10.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.525+0000 7f322959c700 1 --2- 172.21.15.27:0/57261072 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32240ca670 0x7f3224155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.525+0000 7f322959c700 1 -- 172.21.15.27:0/57261072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3224068b60 msgr2=0x7f322416a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.525+0000 7f322959c700 1 --2- 172.21.15.27:0/57261072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3224068b60 0x7f322416a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.525+0000 7f322959c700 1 -- 172.21.15.27:0/57261072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3210009420 con 0x7f32241596c0 2024-01-11T08:23:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.525+0000 7f322959c700 1 --2- 172.21.15.27:0/57261072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32241596c0 0x7f3224156280 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f321c007f20 tx=0x7f321c009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.525+0000 7f3220ff9700 1 -- 172.21.15.27:0/57261072 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c00eb90 con 0x7f32241596c0 2024-01-11T08:23:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.525+0000 7f3220ff9700 1 -- 172.21.15.27:0/57261072 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321c00ed10 con 0x7f32241596c0 2024-01-11T08:23:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.525+0000 7f322afff700 1 -- 172.21.15.27:0/57261072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f322416b230 con 0x7f32241596c0 2024-01-11T08:23:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.525+0000 7f3220ff9700 1 -- 172.21.15.27:0/57261072 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c014670 con 0x7f32241596c0 2024-01-11T08:23:10.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.526+0000 7f322afff700 1 -- 172.21.15.27:0/57261072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322416e900 con 0x7f32241596c0 2024-01-11T08:23:10.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.527+0000 7f322afff700 1 -- 172.21.15.27:0/57261072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3224068620 con 0x7f32241596c0 2024-01-11T08:23:10.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.527+0000 7f3220ff9700 1 -- 172.21.15.27:0/57261072 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f321c0149e0 con 0x7f32241596c0 2024-01-11T08:23:10.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.528+0000 7f3220ff9700 1 --2- 172.21.15.27:0/57261072 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3208066f30 0x7f32080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:10.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.529+0000 7f3220ff9700 1 -- 172.21.15.27:0/57261072 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f321c084bd0 con 0x7f32241596c0 2024-01-11T08:23:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.529+0000 7f3228d9b700 1 --2- 172.21.15.27:0/57261072 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3208066f30 0x7f32080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:10.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.529+0000 7f3228d9b700 1 --2- 172.21.15.27:0/57261072 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3208066f30 0x7f32080693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3218001270 tx=0x7f3218007bf0 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:10.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.532+0000 7f3220ff9700 1 -- 172.21.15.27:0/57261072 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f321c051da0 con 0x7f32241596c0 2024-01-11T08:23:10.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.734+0000 7f322afff700 1 -- 172.21.15.27:0/57261072 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f32240b1b70 con 0x7f3208066f30 2024-01-11T08:23:10.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.743+0000 7f3220ff9700 1 -- 172.21.15.27:0/57261072 <== mgr.5434 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289106 (secure 0 0 0) 0x7f32240b1b70 con 0x7f3208066f30 2024-01-11T08:23:10.746 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:23:10.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.747+0000 7f322afff700 1 -- 172.21.15.27:0/57261072 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3208066f30 msgr2=0x7f32080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:10.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.747+0000 7f322afff700 1 --2- 172.21.15.27:0/57261072 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3208066f30 0x7f32080693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3218001270 tx=0x7f3218007bf0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.747+0000 7f322afff700 1 -- 172.21.15.27:0/57261072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32241596c0 msgr2=0x7f3224156280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:10.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.747+0000 7f322afff700 1 --2- 172.21.15.27:0/57261072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32241596c0 0x7f3224156280 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f321c007f20 tx=0x7f321c009860 comp rx=0 tx=0).stop 2024-01-11T08:23:10.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.747+0000 7f322afff700 1 -- 172.21.15.27:0/57261072 shutdown_connections 2024-01-11T08:23:10.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.747+0000 7f322afff700 1 --2- 172.21.15.27:0/57261072 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3208066f30 0x7f32080693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.747+0000 7f322afff700 1 --2- 172.21.15.27:0/57261072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32241596c0 0x7f3224156280 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.747+0000 7f322afff700 1 --2- 172.21.15.27:0/57261072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3224068b60 0x7f322416a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.747+0000 7f322afff700 1 --2- 172.21.15.27:0/57261072 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32240ca670 0x7f3224155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:10.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.747+0000 7f322afff700 1 -- 172.21.15.27:0/57261072 >> 172.21.15.27:0/57261072 conn(0x7f32240c9f70 msgr2=0x7f32240703a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:10.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.748+0000 7f322afff700 1 -- 172.21.15.27:0/57261072 shutdown_connections 2024-01-11T08:23:10.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:10.748+0000 7f322afff700 1 -- 172.21.15.27:0/57261072 wait complete. 2024-01-11T08:23:10.752 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:23:10.765 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-11T08:23:09.987994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2425924,"kb_used_data":6660,"kb_used_omap":0,"kb_used_meta":2419200,"kb_avail":752548796,"statfs":{"total":773094113280,"available":770609967104,"internally_reserved":0,"allocated":6819840,"data_stored":3712604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2477260800},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001408"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.865255+0000","last_change":"2024-01-11T08:23:05.862186+0000","last_active":"2024-01-11T08:23:05.865255+0000","last_peered":"2024-01-11T08:23:05.865255+0000","last_clean":"2024-01-11T08:23:05.865255+0000","last_became_active":"2024-01-11T08:23:05.861864+0000","last_became_peered":"2024-01-11T08:23:05.861864+0000","last_unstale":"2024-01-11T08:23:05.865255+0000","last_undegraded":"2024-01-11T08:23:05.865255+0000","last_fullsized":"2024-01-11T08:23:05.865255+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:02:56.833975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.886017+0000","last_change":"2024-01-11T08:23:05.885287+0000","last_active":"2024-01-11T08:23:05.886017+0000","last_peered":"2024-01-11T08:23:05.886017+0000","last_clean":"2024-01-11T08:23:05.886017+0000","last_became_active":"2024-01-11T08:23:05.884984+0000","last_became_peered":"2024-01-11T08:23:05.884984+0000","last_unstale":"2024-01-11T08:23:05.886017+0000","last_undegraded":"2024-01-11T08:23:05.886017+0000","last_fullsized":"2024-01-11T08:23:05.886017+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:47:05.958625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.883537+0000","last_change":"2024-01-11T08:23:05.882966+0000","last_active":"2024-01-11T08:23:05.883537+0000","last_peered":"2024-01-11T08:23:05.883537+0000","last_clean":"2024-01-11T08:23:05.883537+0000","last_became_active":"2024-01-11T08:23:05.882672+0000","last_became_peered":"2024-01-11T08:23:05.882672+0000","last_unstale":"2024-01-11T08:23:05.883537+0000","last_undegraded":"2024-01-11T08:23:05.883537+0000","last_fullsized":"2024-01-11T08:23:05.883537+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:49:01.889692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.874604+0000","last_change":"2024-01-11T08:23:05.874604+0000","last_active":"2024-01-11T08:23:05.874604+0000","last_peered":"2024-01-11T08:23:05.874604+0000","last_clean":"2024-01-11T08:23:05.874604+0000","last_became_active":"2024-01-11T08:23:05.874352+0000","last_became_peered":"2024-01-11T08:23:05.874352+0000","last_unstale":"2024-01-11T08:23:05.874604+0000","last_undegraded":"2024-01-11T08:23:05.874604+0000","last_fullsized":"2024-01-11T08:23:05.874604+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:18:28.158621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.872003+0000","last_change":"2024-01-11T08:23:05.869002+0000","last_active":"2024-01-11T08:23:05.872003+0000","last_peered":"2024-01-11T08:23:05.872003+0000","last_clean":"2024-01-11T08:23:05.872003+0000","last_became_active":"2024-01-11T08:23:05.868168+0000","last_became_peered":"2024-01-11T08:23:05.868168+0000","last_unstale":"2024-01-11T08:23:05.872003+0000","last_undegraded":"2024-01-11T08:23:05.872003+0000","last_fullsized":"2024-01-11T08:23:05.872003+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:02:56.833975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.869217+0000","last_change":"2024-01-11T08:23:05.865110+0000","last_active":"2024-01-11T08:23:05.869217+0000","last_peered":"2024-01-11T08:23:05.869217+0000","last_clean":"2024-01-11T08:23:05.869217+0000","last_became_active":"2024-01-11T08:23:05.863707+0000","last_became_peered":"2024-01-11T08:23:05.863707+0000","last_unstale":"2024-01-11T08:23:05.869217+0000","last_undegraded":"2024-01-11T08:23:05.869217+0000","last_fullsized":"2024-01-11T08:23:05.869217+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:57:29.862218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.872112+0000","last_change":"2024-01-11T08:23:05.869703+0000","last_active":"2024-01-11T08:23:05.872112+0000","last_peered":"2024-01-11T08:23:05.872112+0000","last_clean":"2024-01-11T08:23:05.872112+0000","last_became_active":"2024-01-11T08:23:05.868773+0000","last_became_peered":"2024-01-11T08:23:05.868773+0000","last_unstale":"2024-01-11T08:23:05.872112+0000","last_undegraded":"2024-01-11T08:23:05.872112+0000","last_fullsized":"2024-01-11T08:23:05.872112+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:36:07.478679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.882012+0000","last_change":"2024-01-11T08:23:05.882012+0000","last_active":"2024-01-11T08:23:05.882012+0000","last_peered":"2024-01-11T08:23:05.882012+0000","last_clean":"2024-01-11T08:23:05.882012+0000","last_became_active":"2024-01-11T08:23:05.877500+0000","last_became_peered":"2024-01-11T08:23:05.877500+0000","last_unstale":"2024-01-11T08:23:05.882012+0000","last_undegraded":"2024-01-11T08:23:05.882012+0000","last_fullsized":"2024-01-11T08:23:05.882012+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:34:55.229035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.883363+0000","last_change":"2024-01-11T08:23:05.883363+0000","last_active":"2024-01-11T08:23:05.883363+0000","last_peered":"2024-01-11T08:23:05.883363+0000","last_clean":"2024-01-11T08:23:05.883363+0000","last_became_active":"2024-01-11T08:23:05.882795+0000","last_became_peered":"2024-01-11T08:23:05.882795+0000","last_unstale":"2024-01-11T08:23:05.883363+0000","last_undegraded":"2024-01-11T08:23:05.883363+0000","last_fullsized":"2024-01-11T08:23:05.883363+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:36:07.478679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.874722+0000","last_change":"2024-01-11T08:23:05.874722+0000","last_active":"2024-01-11T08:23:05.874722+0000","last_peered":"2024-01-11T08:23:05.874722+0000","last_clean":"2024-01-11T08:23:05.874722+0000","last_became_active":"2024-01-11T08:23:05.874450+0000","last_became_peered":"2024-01-11T08:23:05.874450+0000","last_unstale":"2024-01-11T08:23:05.874722+0000","last_undegraded":"2024-01-11T08:23:05.874722+0000","last_fullsized":"2024-01-11T08:23:05.874722+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:41:27.131872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.886264+0000","last_change":"2024-01-11T08:23:05.886264+0000","last_active":"2024-01-11T08:23:05.886264+0000","last_peered":"2024-01-11T08:23:05.886264+0000","last_clean":"2024-01-11T08:23:05.886264+0000","last_became_active":"2024-01-11T08:23:05.883711+0000","last_became_peered":"2024-01-11T08:23:05.883711+0000","last_unstale":"2024-01-11T08:23:05.886264+0000","last_undegraded":"2024-01-11T08:23:05.886264+0000","last_fullsized":"2024-01-11T08:23:05.886264+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:04:56.439046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.890255+0000","last_change":"2024-01-11T08:23:05.889528+0000","last_active":"2024-01-11T08:23:05.890255+0000","last_peered":"2024-01-11T08:23:05.890255+0000","last_clean":"2024-01-11T08:23:05.890255+0000","last_became_active":"2024-01-11T08:23:05.889215+0000","last_became_peered":"2024-01-11T08:23:05.889215+0000","last_unstale":"2024-01-11T08:23:05.890255+0000","last_undegraded":"2024-01-11T08:23:05.890255+0000","last_fullsized":"2024-01-11T08:23:05.890255+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:49:01.889692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.886049+0000","last_change":"2024-01-11T08:23:05.881048+0000","last_active":"2024-01-11T08:23:05.886049+0000","last_peered":"2024-01-11T08:23:05.886049+0000","last_clean":"2024-01-11T08:23:05.886049+0000","last_became_active":"2024-01-11T08:23:05.878079+0000","last_became_peered":"2024-01-11T08:23:05.878079+0000","last_unstale":"2024-01-11T08:23:05.886049+0000","last_undegraded":"2024-01-11T08:23:05.886049+0000","last_fullsized":"2024-01-11T08:23:05.886049+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:56:08.275331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.877379+0000","last_change":"2024-01-11T08:23:05.877379+0000","last_active":"2024-01-11T08:23:05.877379+0000","last_peered":"2024-01-11T08:23:05.877379+0000","last_clean":"2024-01-11T08:23:05.877379+0000","last_became_active":"2024-01-11T08:23:05.877091+0000","last_became_peered":"2024-01-11T08:23:05.877091+0000","last_unstale":"2024-01-11T08:23:05.877379+0000","last_undegraded":"2024-01-11T08:23:05.877379+0000","last_fullsized":"2024-01-11T08:23:05.877379+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:02:56.833975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.853788+0000","last_change":"2024-01-11T08:23:05.851687+0000","last_active":"2024-01-11T08:23:05.853788+0000","last_peered":"2024-01-11T08:23:05.853788+0000","last_clean":"2024-01-11T08:23:05.853788+0000","last_became_active":"2024-01-11T08:23:05.851416+0000","last_became_peered":"2024-01-11T08:23:05.851416+0000","last_unstale":"2024-01-11T08:23:05.853788+0000","last_undegraded":"2024-01-11T08:23:05.853788+0000","last_fullsized":"2024-01-11T08:23:05.853788+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:59:45.344026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.886738+0000","last_change":"2024-01-11T08:23:05.885754+0000","last_active":"2024-01-11T08:23:05.886738+0000","last_peered":"2024-01-11T08:23:05.886738+0000","last_clean":"2024-01-11T08:23:05.886738+0000","last_became_active":"2024-01-11T08:23:05.885415+0000","last_became_peered":"2024-01-11T08:23:05.885415+0000","last_unstale":"2024-01-11T08:23:05.886738+0000","last_undegraded":"2024-01-11T08:23:05.886738+0000","last_fullsized":"2024-01-11T08:23:05.886738+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:53:15.590430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.864547+0000","last_change":"2024-01-11T08:23:05.863275+0000","last_active":"2024-01-11T08:23:05.864547+0000","last_peered":"2024-01-11T08:23:05.864547+0000","last_clean":"2024-01-11T08:23:05.864547+0000","last_became_active":"2024-01-11T08:23:05.863004+0000","last_became_peered":"2024-01-11T08:23:05.863004+0000","last_unstale":"2024-01-11T08:23:05.864547+0000","last_undegraded":"2024-01-11T08:23:05.864547+0000","last_fullsized":"2024-01-11T08:23:05.864547+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:52:16.057511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.876616+0000","last_change":"2024-01-11T08:23:05.875741+0000","last_active":"2024-01-11T08:23:05.876616+0000","last_peered":"2024-01-11T08:23:05.876616+0000","last_clean":"2024-01-11T08:23:05.876616+0000","last_became_active":"2024-01-11T08:23:05.875464+0000","last_became_peered":"2024-01-11T08:23:05.875464+0000","last_unstale":"2024-01-11T08:23:05.876616+0000","last_undegraded":"2024-01-11T08:23:05.876616+0000","last_fullsized":"2024-01-11T08:23:05.876616+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:22:05.338870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.870257+0000","last_change":"2024-01-11T08:23:05.868727+0000","last_active":"2024-01-11T08:23:05.870257+0000","last_peered":"2024-01-11T08:23:05.870257+0000","last_clean":"2024-01-11T08:23:05.870257+0000","last_became_active":"2024-01-11T08:23:05.868429+0000","last_became_peered":"2024-01-11T08:23:05.868429+0000","last_unstale":"2024-01-11T08:23:05.870257+0000","last_undegraded":"2024-01-11T08:23:05.870257+0000","last_fullsized":"2024-01-11T08:23:05.870257+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:01:39.987317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.892263+0000","last_change":"2024-01-11T08:23:05.891223+0000","last_active":"2024-01-11T08:23:05.892263+0000","last_peered":"2024-01-11T08:23:05.892263+0000","last_clean":"2024-01-11T08:23:05.892263+0000","last_became_active":"2024-01-11T08:23:05.890964+0000","last_became_peered":"2024-01-11T08:23:05.890964+0000","last_unstale":"2024-01-11T08:23:05.892263+0000","last_undegraded":"2024-01-11T08:23:05.892263+0000","last_fullsized":"2024-01-11T08:23:05.892263+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:34:03.306099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.853675+0000","last_change":"2024-01-11T08:23:05.850649+0000","last_active":"2024-01-11T08:23:05.853675+0000","last_peered":"2024-01-11T08:23:05.853675+0000","last_clean":"2024-01-11T08:23:05.853675+0000","last_became_active":"2024-01-11T08:23:05.848941+0000","last_became_peered":"2024-01-11T08:23:05.848941+0000","last_unstale":"2024-01-11T08:23:05.853675+0000","last_undegraded":"2024-01-11T08:23:05.853675+0000","last_fullsized":"2024-01-11T08:23:05.853675+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:54:18.633954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.873704+0000","last_change":"2024-01-11T08:23:05.873704+0000","last_active":"2024-01-11T08:23:05.873704+0000","last_peered":"2024-01-11T08:23:05.873704+0000","last_clean":"2024-01-11T08:23:05.873704+0000","last_became_active":"2024-01-11T08:23:05.871602+0000","last_became_peered":"2024-01-11T08:23:05.871602+0000","last_unstale":"2024-01-11T08:23:05.873704+0000","last_undegraded":"2024-01-11T08:23:05.873704+0000","last_fullsized":"2024-01-11T08:23:05.873704+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:14:57.878995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.861548+0000","last_change":"2024-01-11T08:23:05.858982+0000","last_active":"2024-01-11T08:23:05.861548+0000","last_peered":"2024-01-11T08:23:05.861548+0000","last_clean":"2024-01-11T08:23:05.861548+0000","last_became_active":"2024-01-11T08:23:05.858302+0000","last_became_peered":"2024-01-11T08:23:05.858302+0000","last_unstale":"2024-01-11T08:23:05.861548+0000","last_undegraded":"2024-01-11T08:23:05.861548+0000","last_fullsized":"2024-01-11T08:23:05.861548+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:19:21.740106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.861156+0000","last_change":"2024-01-11T08:23:05.857315+0000","last_active":"2024-01-11T08:23:05.861156+0000","last_peered":"2024-01-11T08:23:05.861156+0000","last_clean":"2024-01-11T08:23:05.861156+0000","last_became_active":"2024-01-11T08:23:05.857016+0000","last_became_peered":"2024-01-11T08:23:05.857016+0000","last_unstale":"2024-01-11T08:23:05.861156+0000","last_undegraded":"2024-01-11T08:23:05.861156+0000","last_fullsized":"2024-01-11T08:23:05.861156+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:48:30.019533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.867553+0000","last_change":"2024-01-11T08:23:05.864675+0000","last_active":"2024-01-11T08:23:05.867553+0000","last_peered":"2024-01-11T08:23:05.867553+0000","last_clean":"2024-01-11T08:23:05.867553+0000","last_became_active":"2024-01-11T08:23:05.864394+0000","last_became_peered":"2024-01-11T08:23:05.864394+0000","last_unstale":"2024-01-11T08:23:05.867553+0000","last_undegraded":"2024-01-11T08:23:05.867553+0000","last_fullsized":"2024-01-11T08:23:05.867553+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:20:13.817226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.885850+0000","last_change":"2024-01-11T08:23:05.878287+0000","last_active":"2024-01-11T08:23:05.885850+0000","last_peered":"2024-01-11T08:23:05.885850+0000","last_clean":"2024-01-11T08:23:05.885850+0000","last_became_active":"2024-01-11T08:23:05.878031+0000","last_became_peered":"2024-01-11T08:23:05.878031+0000","last_unstale":"2024-01-11T08:23:05.885850+0000","last_undegraded":"2024-01-11T08:23:05.885850+0000","last_fullsized":"2024-01-11T08:23:05.885850+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:42:48.781007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.865022+0000","last_change":"2024-01-11T08:23:05.863410+0000","last_active":"2024-01-11T08:23:05.865022+0000","last_peered":"2024-01-11T08:23:05.865022+0000","last_clean":"2024-01-11T08:23:05.865022+0000","last_became_active":"2024-01-11T08:23:05.861968+0000","last_became_peered":"2024-01-11T08:23:05.861968+0000","last_unstale":"2024-01-11T08:23:05.865022+0000","last_undegraded":"2024-01-11T08:23:05.865022+0000","last_fullsized":"2024-01-11T08:23:05.865022+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:16:23.523824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.890528+0000","last_change":"2024-01-11T08:23:05.889772+0000","last_active":"2024-01-11T08:23:05.890528+0000","last_peered":"2024-01-11T08:23:05.890528+0000","last_clean":"2024-01-11T08:23:05.890528+0000","last_became_active":"2024-01-11T08:23:05.889401+0000","last_became_peered":"2024-01-11T08:23:05.889401+0000","last_unstale":"2024-01-11T08:23:05.890528+0000","last_undegraded":"2024-01-11T08:23:05.890528+0000","last_fullsized":"2024-01-11T08:23:05.890528+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:52:16.057511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.869844+0000","last_change":"2024-01-11T08:23:05.866735+0000","last_active":"2024-01-11T08:23:05.869844+0000","last_peered":"2024-01-11T08:23:05.869844+0000","last_clean":"2024-01-11T08:23:05.869844+0000","last_became_active":"2024-01-11T08:23:05.865276+0000","last_became_peered":"2024-01-11T08:23:05.865276+0000","last_unstale":"2024-01-11T08:23:05.869844+0000","last_undegraded":"2024-01-11T08:23:05.869844+0000","last_fullsized":"2024-01-11T08:23:05.869844+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:26:59.623293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.861220+0000","last_change":"2024-01-11T08:23:05.856965+0000","last_active":"2024-01-11T08:23:05.861220+0000","last_peered":"2024-01-11T08:23:05.861220+0000","last_clean":"2024-01-11T08:23:05.861220+0000","last_became_active":"2024-01-11T08:23:05.856672+0000","last_became_peered":"2024-01-11T08:23:05.856672+0000","last_unstale":"2024-01-11T08:23:05.861220+0000","last_undegraded":"2024-01-11T08:23:05.861220+0000","last_fullsized":"2024-01-11T08:23:05.861220+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:40:30.954875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.869528+0000","last_change":"2024-01-11T08:23:05.867498+0000","last_active":"2024-01-11T08:23:05.869528+0000","last_peered":"2024-01-11T08:23:05.869528+0000","last_clean":"2024-01-11T08:23:05.869528+0000","last_became_active":"2024-01-11T08:23:05.866152+0000","last_became_peered":"2024-01-11T08:23:05.866152+0000","last_unstale":"2024-01-11T08:23:05.869528+0000","last_undegraded":"2024-01-11T08:23:05.869528+0000","last_fullsized":"2024-01-11T08:23:05.869528+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:54:38.853272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.885513+0000","last_change":"2024-01-11T08:23:05.877836+0000","last_active":"2024-01-11T08:23:05.885513+0000","last_peered":"2024-01-11T08:23:05.885513+0000","last_clean":"2024-01-11T08:23:05.885513+0000","last_became_active":"2024-01-11T08:23:05.877536+0000","last_became_peered":"2024-01-11T08:23:05.877536+0000","last_unstale":"2024-01-11T08:23:05.885513+0000","last_undegraded":"2024-01-11T08:23:05.885513+0000","last_fullsized":"2024-01-11T08:23:05.885513+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:39:11.620775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.885245+0000","last_change":"2024-01-11T08:23:05.878365+0000","last_active":"2024-01-11T08:23:05.885245+0000","last_peered":"2024-01-11T08:23:05.885245+0000","last_clean":"2024-01-11T08:23:05.885245+0000","last_became_active":"2024-01-11T08:23:05.878064+0000","last_became_peered":"2024-01-11T08:23:05.878064+0000","last_unstale":"2024-01-11T08:23:05.885245+0000","last_undegraded":"2024-01-11T08:23:05.885245+0000","last_fullsized":"2024-01-11T08:23:05.885245+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:01:39.987317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"58'4","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.861945+0000","last_change":"2024-01-11T08:23:04.860362+0000","last_active":"2024-01-11T08:23:05.861945+0000","last_peered":"2024-01-11T08:23:05.861945+0000","last_clean":"2024-01-11T08:23:05.861945+0000","last_became_active":"2024-01-11T08:23:04.859926+0000","last_became_peered":"2024-01-11T08:23:04.859926+0000","last_unstale":"2024-01-11T08:23:05.861945+0000","last_undegraded":"2024-01-11T08:23:05.861945+0000","last_fullsized":"2024-01-11T08:23:05.861945+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:58:55.564598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.861197+0000","last_change":"2024-01-11T08:23:05.857393+0000","last_active":"2024-01-11T08:23:05.861197+0000","last_peered":"2024-01-11T08:23:05.861197+0000","last_clean":"2024-01-11T08:23:05.861197+0000","last_became_active":"2024-01-11T08:23:05.856528+0000","last_became_peered":"2024-01-11T08:23:05.856528+0000","last_unstale":"2024-01-11T08:23:05.861197+0000","last_undegraded":"2024-01-11T08:23:05.861197+0000","last_fullsized":"2024-01-11T08:23:05.861197+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:44:44.140597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"58'1","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.860534+0000","last_change":"2024-01-11T08:23:04.863311+0000","last_active":"2024-01-11T08:23:05.860534+0000","last_peered":"2024-01-11T08:23:05.860534+0000","last_clean":"2024-01-11T08:23:05.860534+0000","last_became_active":"2024-01-11T08:23:04.862049+0000","last_became_peered":"2024-01-11T08:23:04.862049+0000","last_unstale":"2024-01-11T08:23:05.860534+0000","last_undegraded":"2024-01-11T08:23:05.860534+0000","last_fullsized":"2024-01-11T08:23:05.860534+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:48:20.943166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.867645+0000","last_change":"2024-01-11T08:23:05.863342+0000","last_active":"2024-01-11T08:23:05.867645+0000","last_peered":"2024-01-11T08:23:05.867645+0000","last_clean":"2024-01-11T08:23:05.867645+0000","last_became_active":"2024-01-11T08:23:05.862886+0000","last_became_peered":"2024-01-11T08:23:05.862886+0000","last_unstale":"2024-01-11T08:23:05.867645+0000","last_undegraded":"2024-01-11T08:23:05.867645+0000","last_fullsized":"2024-01-11T08:23:05.867645+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:22:05.338870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"58'2","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.885836+0000","last_change":"2024-01-11T08:23:04.857468+0000","last_active":"2024-01-11T08:23:05.885836+0000","last_peered":"2024-01-11T08:23:05.885836+0000","last_clean":"2024-01-11T08:23:05.885836+0000","last_became_active":"2024-01-11T08:23:04.857201+0000","last_became_peered":"2024-01-11T08:23:04.857201+0000","last_unstale":"2024-01-11T08:23:05.885836+0000","last_undegraded":"2024-01-11T08:23:05.885836+0000","last_fullsized":"2024-01-11T08:23:05.885836+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:19:20.730546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.885888+0000","last_change":"2024-01-11T08:23:05.879255+0000","last_active":"2024-01-11T08:23:05.885888+0000","last_peered":"2024-01-11T08:23:05.885888+0000","last_clean":"2024-01-11T08:23:05.885888+0000","last_became_active":"2024-01-11T08:23:05.878961+0000","last_became_peered":"2024-01-11T08:23:05.878961+0000","last_unstale":"2024-01-11T08:23:05.885888+0000","last_undegraded":"2024-01-11T08:23:05.885888+0000","last_fullsized":"2024-01-11T08:23:05.885888+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:40:30.954875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"58'2","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.864716+0000","last_change":"2024-01-11T08:23:04.853541+0000","last_active":"2024-01-11T08:23:05.864716+0000","last_peered":"2024-01-11T08:23:05.864716+0000","last_clean":"2024-01-11T08:23:05.864716+0000","last_became_active":"2024-01-11T08:23:04.853182+0000","last_became_peered":"2024-01-11T08:23:04.853182+0000","last_unstale":"2024-01-11T08:23:05.864716+0000","last_undegraded":"2024-01-11T08:23:05.864716+0000","last_fullsized":"2024-01-11T08:23:05.864716+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:20:12.807666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.860851+0000","last_change":"2024-01-11T08:23:05.855157+0000","last_active":"2024-01-11T08:23:05.860851+0000","last_peered":"2024-01-11T08:23:05.860851+0000","last_clean":"2024-01-11T08:23:05.860851+0000","last_became_active":"2024-01-11T08:23:05.854437+0000","last_became_peered":"2024-01-11T08:23:05.854437+0000","last_unstale":"2024-01-11T08:23:05.860851+0000","last_undegraded":"2024-01-11T08:23:05.860851+0000","last_fullsized":"2024-01-11T08:23:05.860851+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:36:09.262909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"58'1","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.871192+0000","last_change":"2024-01-11T08:23:04.851430+0000","last_active":"2024-01-11T08:23:05.871192+0000","last_peered":"2024-01-11T08:23:05.871192+0000","last_clean":"2024-01-11T08:23:05.871192+0000","last_became_active":"2024-01-11T08:23:04.851027+0000","last_became_peered":"2024-01-11T08:23:04.851027+0000","last_unstale":"2024-01-11T08:23:05.871192+0000","last_undegraded":"2024-01-11T08:23:05.871192+0000","last_fullsized":"2024-01-11T08:23:05.871192+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:36:08.253349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.887418+0000","last_change":"2024-01-11T08:23:05.887418+0000","last_active":"2024-01-11T08:23:05.887418+0000","last_peered":"2024-01-11T08:23:05.887418+0000","last_clean":"2024-01-11T08:23:05.887418+0000","last_became_active":"2024-01-11T08:23:05.887168+0000","last_became_peered":"2024-01-11T08:23:05.887168+0000","last_unstale":"2024-01-11T08:23:05.887418+0000","last_undegraded":"2024-01-11T08:23:05.887418+0000","last_fullsized":"2024-01-11T08:23:05.887418+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:58:56.574158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"58'1","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.861062+0000","last_change":"2024-01-11T08:23:04.859924+0000","last_active":"2024-01-11T08:23:05.861062+0000","last_peered":"2024-01-11T08:23:05.861062+0000","last_clean":"2024-01-11T08:23:05.861062+0000","last_became_active":"2024-01-11T08:23:04.859604+0000","last_became_peered":"2024-01-11T08:23:04.859604+0000","last_unstale":"2024-01-11T08:23:05.861062+0000","last_undegraded":"2024-01-11T08:23:05.861062+0000","last_fullsized":"2024-01-11T08:23:05.861062+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:18:53.731073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.882017+0000","last_change":"2024-01-11T08:23:05.882017+0000","last_active":"2024-01-11T08:23:05.882017+0000","last_peered":"2024-01-11T08:23:05.882017+0000","last_clean":"2024-01-11T08:23:05.882017+0000","last_became_active":"2024-01-11T08:23:05.877798+0000","last_became_peered":"2024-01-11T08:23:05.877798+0000","last_unstale":"2024-01-11T08:23:05.882017+0000","last_undegraded":"2024-01-11T08:23:05.882017+0000","last_fullsized":"2024-01-11T08:23:05.882017+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:47:05.958625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.852185+0000","last_change":"2024-01-11T08:23:04.852427+0000","last_active":"2024-01-11T08:23:05.852185+0000","last_peered":"2024-01-11T08:23:05.852185+0000","last_clean":"2024-01-11T08:23:05.852185+0000","last_became_active":"2024-01-11T08:23:04.852073+0000","last_became_peered":"2024-01-11T08:23:04.852073+0000","last_unstale":"2024-01-11T08:23:05.852185+0000","last_undegraded":"2024-01-11T08:23:05.852185+0000","last_fullsized":"2024-01-11T08:23:05.852185+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:26:58.613733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.861529+0000","last_change":"2024-01-11T08:23:05.858598+0000","last_active":"2024-01-11T08:23:05.861529+0000","last_peered":"2024-01-11T08:23:05.861529+0000","last_clean":"2024-01-11T08:23:05.861529+0000","last_became_active":"2024-01-11T08:23:05.858141+0000","last_became_peered":"2024-01-11T08:23:05.858141+0000","last_unstale":"2024-01-11T08:23:05.861529+0000","last_undegraded":"2024-01-11T08:23:05.861529+0000","last_fullsized":"2024-01-11T08:23:05.861529+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:17:25.876941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-11T08:23:04.870102+0000","last_change":"2024-01-11T08:23:04.867054+0000","last_active":"2024-01-11T08:23:04.870102+0000","last_peered":"2024-01-11T08:23:04.870102+0000","last_clean":"2024-01-11T08:23:04.870102+0000","last_became_active":"2024-01-11T08:23:04.866431+0000","last_became_peered":"2024-01-11T08:23:04.866431+0000","last_unstale":"2024-01-11T08:23:04.870102+0000","last_undegraded":"2024-01-11T08:23:04.870102+0000","last_fullsized":"2024-01-11T08:23:04.870102+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:53:02.013837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.889838+0000","last_change":"2024-01-11T08:23:05.889137+0000","last_active":"2024-01-11T08:23:05.889838+0000","last_peered":"2024-01-11T08:23:05.889838+0000","last_clean":"2024-01-11T08:23:05.889838+0000","last_became_active":"2024-01-11T08:23:05.888815+0000","last_became_peered":"2024-01-11T08:23:05.888815+0000","last_unstale":"2024-01-11T08:23:05.889838+0000","last_undegraded":"2024-01-11T08:23:05.889838+0000","last_fullsized":"2024-01-11T08:23:05.889838+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:36:07.478679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.845776+0000","last_change":"2024-01-11T08:23:04.860508+0000","last_active":"2024-01-11T08:23:05.845776+0000","last_peered":"2024-01-11T08:23:05.845776+0000","last_clean":"2024-01-11T08:23:05.845776+0000","last_became_active":"2024-01-11T08:23:04.859843+0000","last_became_peered":"2024-01-11T08:23:04.859843+0000","last_unstale":"2024-01-11T08:23:05.845776+0000","last_undegraded":"2024-01-11T08:23:05.845776+0000","last_fullsized":"2024-01-11T08:23:05.845776+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:22:32.656105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.872081+0000","last_change":"2024-01-11T08:23:05.867536+0000","last_active":"2024-01-11T08:23:05.872081+0000","last_peered":"2024-01-11T08:23:05.872081+0000","last_clean":"2024-01-11T08:23:05.872081+0000","last_became_active":"2024-01-11T08:23:05.867252+0000","last_became_peered":"2024-01-11T08:23:05.867252+0000","last_unstale":"2024-01-11T08:23:05.872081+0000","last_undegraded":"2024-01-11T08:23:05.872081+0000","last_fullsized":"2024-01-11T08:23:05.872081+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:49:01.889692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-11T08:23:04.872822+0000","last_change":"2024-01-11T08:23:04.867792+0000","last_active":"2024-01-11T08:23:04.872822+0000","last_peered":"2024-01-11T08:23:04.872822+0000","last_clean":"2024-01-11T08:23:04.872822+0000","last_became_active":"2024-01-11T08:23:04.866448+0000","last_became_peered":"2024-01-11T08:23:04.866448+0000","last_unstale":"2024-01-11T08:23:04.872822+0000","last_undegraded":"2024-01-11T08:23:04.872822+0000","last_fullsized":"2024-01-11T08:23:04.872822+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:19:33.536966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.891518+0000","last_change":"2024-01-11T08:23:05.889421+0000","last_active":"2024-01-11T08:23:05.891518+0000","last_peered":"2024-01-11T08:23:05.891518+0000","last_clean":"2024-01-11T08:23:05.891518+0000","last_became_active":"2024-01-11T08:23:05.889152+0000","last_became_peered":"2024-01-11T08:23:05.889152+0000","last_unstale":"2024-01-11T08:23:05.891518+0000","last_undegraded":"2024-01-11T08:23:05.891518+0000","last_fullsized":"2024-01-11T08:23:05.891518+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:38:03.236711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"58'3","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.862285+0000","last_change":"2024-01-11T08:23:04.860830+0000","last_active":"2024-01-11T08:23:05.862285+0000","last_peered":"2024-01-11T08:23:05.862285+0000","last_clean":"2024-01-11T08:23:05.862285+0000","last_became_active":"2024-01-11T08:23:04.860528+0000","last_became_peered":"2024-01-11T08:23:04.860528+0000","last_unstale":"2024-01-11T08:23:05.862285+0000","last_undegraded":"2024-01-11T08:23:05.862285+0000","last_fullsized":"2024-01-11T08:23:05.862285+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:19:20.730546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.880178+0000","last_change":"2024-01-11T08:23:05.877105+0000","last_active":"2024-01-11T08:23:05.880178+0000","last_peered":"2024-01-11T08:23:05.880178+0000","last_clean":"2024-01-11T08:23:05.880178+0000","last_became_active":"2024-01-11T08:23:05.876688+0000","last_became_peered":"2024-01-11T08:23:05.876688+0000","last_unstale":"2024-01-11T08:23:05.880178+0000","last_undegraded":"2024-01-11T08:23:05.880178+0000","last_fullsized":"2024-01-11T08:23:05.880178+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:52:16.057511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"58'6","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.893405+0000","last_change":"2024-01-11T08:23:04.859827+0000","last_active":"2024-01-11T08:23:05.893405+0000","last_peered":"2024-01-11T08:23:05.893405+0000","last_clean":"2024-01-11T08:23:05.893405+0000","last_became_active":"2024-01-11T08:23:04.859463+0000","last_became_peered":"2024-01-11T08:23:04.859463+0000","last_unstale":"2024-01-11T08:23:05.893405+0000","last_undegraded":"2024-01-11T08:23:05.893405+0000","last_fullsized":"2024-01-11T08:23:05.893405+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:30:41.699759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.877791+0000","last_change":"2024-01-11T08:23:05.876914+0000","last_active":"2024-01-11T08:23:05.877791+0000","last_peered":"2024-01-11T08:23:05.877791+0000","last_clean":"2024-01-11T08:23:05.877791+0000","last_became_active":"2024-01-11T08:23:05.876179+0000","last_became_peered":"2024-01-11T08:23:05.876179+0000","last_unstale":"2024-01-11T08:23:05.877791+0000","last_undegraded":"2024-01-11T08:23:05.877791+0000","last_fullsized":"2024-01-11T08:23:05.877791+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:58:56.574158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"58'2","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.888560+0000","last_change":"2024-01-11T08:23:04.864260+0000","last_active":"2024-01-11T08:23:05.888560+0000","last_peered":"2024-01-11T08:23:05.888560+0000","last_clean":"2024-01-11T08:23:05.888560+0000","last_became_active":"2024-01-11T08:23:04.863911+0000","last_became_peered":"2024-01-11T08:23:04.863911+0000","last_unstale":"2024-01-11T08:23:05.888560+0000","last_undegraded":"2024-01-11T08:23:05.888560+0000","last_fullsized":"2024-01-11T08:23:05.888560+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:52:15.047951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.877300+0000","last_change":"2024-01-11T08:23:05.877300+0000","last_active":"2024-01-11T08:23:05.877300+0000","last_peered":"2024-01-11T08:23:05.877300+0000","last_clean":"2024-01-11T08:23:05.877300+0000","last_became_active":"2024-01-11T08:23:05.876871+0000","last_became_peered":"2024-01-11T08:23:05.876871+0000","last_unstale":"2024-01-11T08:23:05.877300+0000","last_undegraded":"2024-01-11T08:23:05.877300+0000","last_fullsized":"2024-01-11T08:23:05.877300+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:40:30.954875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.853938+0000","last_change":"2024-01-11T08:23:04.855272+0000","last_active":"2024-01-11T08:23:05.853938+0000","last_peered":"2024-01-11T08:23:05.853938+0000","last_clean":"2024-01-11T08:23:05.853938+0000","last_became_active":"2024-01-11T08:23:04.854948+0000","last_became_peered":"2024-01-11T08:23:04.854948+0000","last_unstale":"2024-01-11T08:23:05.853938+0000","last_undegraded":"2024-01-11T08:23:05.853938+0000","last_fullsized":"2024-01-11T08:23:05.853938+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:46:38.189844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.889091+0000","last_change":"2024-01-11T08:23:05.886119+0000","last_active":"2024-01-11T08:23:05.889091+0000","last_peered":"2024-01-11T08:23:05.889091+0000","last_clean":"2024-01-11T08:23:05.889091+0000","last_became_active":"2024-01-11T08:23:05.884572+0000","last_became_peered":"2024-01-11T08:23:05.884572+0000","last_unstale":"2024-01-11T08:23:05.889091+0000","last_undegraded":"2024-01-11T08:23:05.889091+0000","last_fullsized":"2024-01-11T08:23:05.889091+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:59:57.583185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.841119+0000","last_change":"2024-01-11T08:23:04.866816+0000","last_active":"2024-01-11T08:23:05.841119+0000","last_peered":"2024-01-11T08:23:05.841119+0000","last_clean":"2024-01-11T08:23:05.841119+0000","last_became_active":"2024-01-11T08:23:04.866516+0000","last_became_peered":"2024-01-11T08:23:04.866516+0000","last_unstale":"2024-01-11T08:23:05.841119+0000","last_undegraded":"2024-01-11T08:23:05.841119+0000","last_fullsized":"2024-01-11T08:23:05.841119+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:49:00.880132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.865632+0000","last_change":"2024-01-11T08:23:05.863068+0000","last_active":"2024-01-11T08:23:05.865632+0000","last_peered":"2024-01-11T08:23:05.865632+0000","last_clean":"2024-01-11T08:23:05.865632+0000","last_became_active":"2024-01-11T08:23:05.862177+0000","last_became_peered":"2024-01-11T08:23:05.862177+0000","last_unstale":"2024-01-11T08:23:05.865632+0000","last_undegraded":"2024-01-11T08:23:05.865632+0000","last_fullsized":"2024-01-11T08:23:05.865632+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:22:05.338870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"58'8","reported_seq":17,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.877334+0000","last_change":"2024-01-11T08:23:04.867225+0000","last_active":"2024-01-11T08:23:05.877334+0000","last_peered":"2024-01-11T08:23:05.877334+0000","last_clean":"2024-01-11T08:23:05.877334+0000","last_became_active":"2024-01-11T08:23:04.866784+0000","last_became_peered":"2024-01-11T08:23:04.866784+0000","last_unstale":"2024-01-11T08:23:05.877334+0000","last_undegraded":"2024-01-11T08:23:05.877334+0000","last_fullsized":"2024-01-11T08:23:05.877334+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:55:02.211673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.889488+0000","last_change":"2024-01-11T08:23:05.882693+0000","last_active":"2024-01-11T08:23:05.889488+0000","last_peered":"2024-01-11T08:23:05.889488+0000","last_clean":"2024-01-11T08:23:05.889488+0000","last_became_active":"2024-01-11T08:23:05.882457+0000","last_became_peered":"2024-01-11T08:23:05.882457+0000","last_unstale":"2024-01-11T08:23:05.889488+0000","last_undegraded":"2024-01-11T08:23:05.889488+0000","last_fullsized":"2024-01-11T08:23:05.889488+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:10.276798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"58'1","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.888885+0000","last_change":"2024-01-11T08:23:04.849494+0000","last_active":"2024-01-11T08:23:05.888885+0000","last_peered":"2024-01-11T08:23:05.888885+0000","last_clean":"2024-01-11T08:23:05.888885+0000","last_became_active":"2024-01-11T08:23:04.849251+0000","last_became_peered":"2024-01-11T08:23:04.849251+0000","last_unstale":"2024-01-11T08:23:05.888885+0000","last_undegraded":"2024-01-11T08:23:05.888885+0000","last_fullsized":"2024-01-11T08:23:05.888885+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:47:34.139983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.874269+0000","last_change":"2024-01-11T08:23:05.874269+0000","last_active":"2024-01-11T08:23:05.874269+0000","last_peered":"2024-01-11T08:23:05.874269+0000","last_clean":"2024-01-11T08:23:05.874269+0000","last_became_active":"2024-01-11T08:23:05.872970+0000","last_became_peered":"2024-01-11T08:23:05.872970+0000","last_unstale":"2024-01-11T08:23:05.874269+0000","last_undegraded":"2024-01-11T08:23:05.874269+0000","last_fullsized":"2024-01-11T08:23:05.874269+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:22:05.338870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"58'2","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.868865+0000","last_change":"2024-01-11T08:23:04.857254+0000","last_active":"2024-01-11T08:23:05.868865+0000","last_peered":"2024-01-11T08:23:05.868865+0000","last_clean":"2024-01-11T08:23:05.868865+0000","last_became_active":"2024-01-11T08:23:04.856997+0000","last_became_peered":"2024-01-11T08:23:04.856997+0000","last_unstale":"2024-01-11T08:23:05.868865+0000","last_undegraded":"2024-01-11T08:23:05.868865+0000","last_fullsized":"2024-01-11T08:23:05.868865+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:48:29.009973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.859977+0000","last_change":"2024-01-11T08:23:05.856341+0000","last_active":"2024-01-11T08:23:05.859977+0000","last_peered":"2024-01-11T08:23:05.859977+0000","last_clean":"2024-01-11T08:23:05.859977+0000","last_became_active":"2024-01-11T08:23:05.855566+0000","last_became_peered":"2024-01-11T08:23:05.855566+0000","last_unstale":"2024-01-11T08:23:05.859977+0000","last_undegraded":"2024-01-11T08:23:05.859977+0000","last_fullsized":"2024-01-11T08:23:05.859977+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:46:39.199404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"58'1","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.885389+0000","last_change":"2024-01-11T08:23:04.856455+0000","last_active":"2024-01-11T08:23:05.885389+0000","last_peered":"2024-01-11T08:23:05.885389+0000","last_clean":"2024-01-11T08:23:05.885389+0000","last_became_active":"2024-01-11T08:23:04.856005+0000","last_became_peered":"2024-01-11T08:23:04.856005+0000","last_unstale":"2024-01-11T08:23:05.885389+0000","last_undegraded":"2024-01-11T08:23:05.885389+0000","last_fullsized":"2024-01-11T08:23:05.885389+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:44:43.131037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.867634+0000","last_change":"2024-01-11T08:23:05.864974+0000","last_active":"2024-01-11T08:23:05.867634+0000","last_peered":"2024-01-11T08:23:05.867634+0000","last_clean":"2024-01-11T08:23:05.867634+0000","last_became_active":"2024-01-11T08:23:05.864724+0000","last_became_peered":"2024-01-11T08:23:05.864724+0000","last_unstale":"2024-01-11T08:23:05.867634+0000","last_undegraded":"2024-01-11T08:23:05.867634+0000","last_fullsized":"2024-01-11T08:23:05.867634+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:18:54.740633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.842262+0000","last_change":"2024-01-11T08:23:04.852928+0000","last_active":"2024-01-11T08:23:05.842262+0000","last_peered":"2024-01-11T08:23:05.842262+0000","last_clean":"2024-01-11T08:23:05.842262+0000","last_became_active":"2024-01-11T08:23:04.852633+0000","last_became_peered":"2024-01-11T08:23:04.852633+0000","last_unstale":"2024-01-11T08:23:05.842262+0000","last_undegraded":"2024-01-11T08:23:05.842262+0000","last_fullsized":"2024-01-11T08:23:05.842262+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:01:38.977757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.891399+0000","last_change":"2024-01-11T08:23:05.886811+0000","last_active":"2024-01-11T08:23:05.891399+0000","last_peered":"2024-01-11T08:23:05.891399+0000","last_clean":"2024-01-11T08:23:05.891399+0000","last_became_active":"2024-01-11T08:23:05.886505+0000","last_became_peered":"2024-01-11T08:23:05.886505+0000","last_unstale":"2024-01-11T08:23:05.891399+0000","last_undegraded":"2024-01-11T08:23:05.891399+0000","last_fullsized":"2024-01-11T08:23:05.891399+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:58:01.022706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"58'2","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.871539+0000","last_change":"2024-01-11T08:23:04.851210+0000","last_active":"2024-01-11T08:23:05.871539+0000","last_peered":"2024-01-11T08:23:05.871539+0000","last_clean":"2024-01-11T08:23:05.871539+0000","last_became_active":"2024-01-11T08:23:04.850926+0000","last_became_peered":"2024-01-11T08:23:04.850926+0000","last_unstale":"2024-01-11T08:23:05.871539+0000","last_undegraded":"2024-01-11T08:23:05.871539+0000","last_fullsized":"2024-01-11T08:23:05.871539+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:04:55.429486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.885272+0000","last_change":"2024-01-11T08:23:05.878300+0000","last_active":"2024-01-11T08:23:05.885272+0000","last_peered":"2024-01-11T08:23:05.885272+0000","last_clean":"2024-01-11T08:23:05.885272+0000","last_became_active":"2024-01-11T08:23:05.875460+0000","last_became_peered":"2024-01-11T08:23:05.875460+0000","last_unstale":"2024-01-11T08:23:05.885272+0000","last_undegraded":"2024-01-11T08:23:05.885272+0000","last_fullsized":"2024-01-11T08:23:05.885272+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:53:15.590430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.855049+0000","last_change":"2024-01-11T08:23:04.853953+0000","last_active":"2024-01-11T08:23:05.855049+0000","last_peered":"2024-01-11T08:23:05.855049+0000","last_clean":"2024-01-11T08:23:05.855049+0000","last_became_active":"2024-01-11T08:23:04.852271+0000","last_became_peered":"2024-01-11T08:23:04.852271+0000","last_unstale":"2024-01-11T08:23:05.855049+0000","last_undegraded":"2024-01-11T08:23:05.855049+0000","last_fullsized":"2024-01-11T08:23:05.855049+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:46:38.189844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.885492+0000","last_change":"2024-01-11T08:23:05.877143+0000","last_active":"2024-01-11T08:23:05.885492+0000","last_peered":"2024-01-11T08:23:05.885492+0000","last_clean":"2024-01-11T08:23:05.885492+0000","last_became_active":"2024-01-11T08:23:05.872690+0000","last_became_peered":"2024-01-11T08:23:05.872690+0000","last_unstale":"2024-01-11T08:23:05.885492+0000","last_undegraded":"2024-01-11T08:23:05.885492+0000","last_fullsized":"2024-01-11T08:23:05.885492+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:33:45.602596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-11T08:23:04.867138+0000","last_change":"2024-01-11T08:23:04.864969+0000","last_active":"2024-01-11T08:23:04.867138+0000","last_peered":"2024-01-11T08:23:04.867138+0000","last_clean":"2024-01-11T08:23:04.867138+0000","last_became_active":"2024-01-11T08:23:04.864594+0000","last_became_peered":"2024-01-11T08:23:04.864594+0000","last_unstale":"2024-01-11T08:23:04.867138+0000","last_undegraded":"2024-01-11T08:23:04.867138+0000","last_fullsized":"2024-01-11T08:23:04.867138+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:17:24.989859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.874220+0000","last_change":"2024-01-11T08:23:05.874220+0000","last_active":"2024-01-11T08:23:05.874220+0000","last_peered":"2024-01-11T08:23:05.874220+0000","last_clean":"2024-01-11T08:23:05.874220+0000","last_became_active":"2024-01-11T08:23:05.873797+0000","last_became_peered":"2024-01-11T08:23:05.873797+0000","last_unstale":"2024-01-11T08:23:05.874220+0000","last_undegraded":"2024-01-11T08:23:05.874220+0000","last_fullsized":"2024-01-11T08:23:05.874220+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:46:39.199404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.852635+0000","last_change":"2024-01-11T08:23:04.855002+0000","last_active":"2024-01-11T08:23:05.852635+0000","last_peered":"2024-01-11T08:23:05.852635+0000","last_clean":"2024-01-11T08:23:05.852635+0000","last_became_active":"2024-01-11T08:23:04.854716+0000","last_became_peered":"2024-01-11T08:23:04.854716+0000","last_unstale":"2024-01-11T08:23:05.852635+0000","last_undegraded":"2024-01-11T08:23:05.852635+0000","last_fullsized":"2024-01-11T08:23:05.852635+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:26:58.613733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.870106+0000","last_change":"2024-01-11T08:23:05.868723+0000","last_active":"2024-01-11T08:23:05.870106+0000","last_peered":"2024-01-11T08:23:05.870106+0000","last_clean":"2024-01-11T08:23:05.870106+0000","last_became_active":"2024-01-11T08:23:05.868410+0000","last_became_peered":"2024-01-11T08:23:05.868410+0000","last_unstale":"2024-01-11T08:23:05.870106+0000","last_undegraded":"2024-01-11T08:23:05.870106+0000","last_fullsized":"2024-01-11T08:23:05.870106+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:40:30.954875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.842431+0000","last_change":"2024-01-11T08:23:04.861846+0000","last_active":"2024-01-11T08:23:05.842431+0000","last_peered":"2024-01-11T08:23:05.842431+0000","last_clean":"2024-01-11T08:23:05.842431+0000","last_became_active":"2024-01-11T08:23:04.861524+0000","last_became_peered":"2024-01-11T08:23:04.861524+0000","last_unstale":"2024-01-11T08:23:05.842431+0000","last_undegraded":"2024-01-11T08:23:05.842431+0000","last_fullsized":"2024-01-11T08:23:05.842431+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:16:22.514264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.867509+0000","last_change":"2024-01-11T08:23:05.867509+0000","last_active":"2024-01-11T08:23:05.867509+0000","last_peered":"2024-01-11T08:23:05.867509+0000","last_clean":"2024-01-11T08:23:05.867509+0000","last_became_active":"2024-01-11T08:23:05.867227+0000","last_became_peered":"2024-01-11T08:23:05.867227+0000","last_unstale":"2024-01-11T08:23:05.867509+0000","last_undegraded":"2024-01-11T08:23:05.867509+0000","last_fullsized":"2024-01-11T08:23:05.867509+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:04:53.426280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.857085+0000","last_change":"2024-01-11T08:23:04.864163+0000","last_active":"2024-01-11T08:23:05.857085+0000","last_peered":"2024-01-11T08:23:05.857085+0000","last_clean":"2024-01-11T08:23:05.857085+0000","last_became_active":"2024-01-11T08:23:04.863911+0000","last_became_peered":"2024-01-11T08:23:04.863911+0000","last_unstale":"2024-01-11T08:23:05.857085+0000","last_undegraded":"2024-01-11T08:23:05.857085+0000","last_fullsized":"2024-01-11T08:23:05.857085+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:22:32.656105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.887275+0000","last_change":"2024-01-11T08:23:05.886605+0000","last_active":"2024-01-11T08:23:05.887275+0000","last_peered":"2024-01-11T08:23:05.887275+0000","last_clean":"2024-01-11T08:23:05.887275+0000","last_became_active":"2024-01-11T08:23:05.886284+0000","last_became_peered":"2024-01-11T08:23:05.886284+0000","last_unstale":"2024-01-11T08:23:05.887275+0000","last_undegraded":"2024-01-11T08:23:05.887275+0000","last_fullsized":"2024-01-11T08:23:05.887275+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:16:23.523824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.862833+0000","last_change":"2024-01-11T08:23:04.857360+0000","last_active":"2024-01-11T08:23:05.862833+0000","last_peered":"2024-01-11T08:23:05.862833+0000","last_clean":"2024-01-11T08:23:05.862833+0000","last_became_active":"2024-01-11T08:23:04.857051+0000","last_became_peered":"2024-01-11T08:23:04.857051+0000","last_unstale":"2024-01-11T08:23:05.862833+0000","last_undegraded":"2024-01-11T08:23:05.862833+0000","last_fullsized":"2024-01-11T08:23:05.862833+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:22:32.656105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.856039+0000","last_change":"2024-01-11T08:23:05.852346+0000","last_active":"2024-01-11T08:23:05.856039+0000","last_peered":"2024-01-11T08:23:05.856039+0000","last_clean":"2024-01-11T08:23:05.856039+0000","last_became_active":"2024-01-11T08:23:05.852074+0000","last_became_peered":"2024-01-11T08:23:05.852074+0000","last_unstale":"2024-01-11T08:23:05.856039+0000","last_undegraded":"2024-01-11T08:23:05.856039+0000","last_fullsized":"2024-01-11T08:23:05.856039+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:03:28.430072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.854174+0000","last_change":"2024-01-11T08:23:04.862981+0000","last_active":"2024-01-11T08:23:05.854174+0000","last_peered":"2024-01-11T08:23:05.854174+0000","last_clean":"2024-01-11T08:23:05.854174+0000","last_became_active":"2024-01-11T08:23:04.862681+0000","last_became_peered":"2024-01-11T08:23:04.862681+0000","last_unstale":"2024-01-11T08:23:05.854174+0000","last_undegraded":"2024-01-11T08:23:05.854174+0000","last_fullsized":"2024-01-11T08:23:05.854174+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:17:24.867381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.885572+0000","last_change":"2024-01-11T08:23:05.884668+0000","last_active":"2024-01-11T08:23:05.885572+0000","last_peered":"2024-01-11T08:23:05.885572+0000","last_clean":"2024-01-11T08:23:05.885572+0000","last_became_active":"2024-01-11T08:23:05.882603+0000","last_became_peered":"2024-01-11T08:23:05.882603+0000","last_unstale":"2024-01-11T08:23:05.885572+0000","last_undegraded":"2024-01-11T08:23:05.885572+0000","last_fullsized":"2024-01-11T08:23:05.885572+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:04:07.668684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.846954+0000","last_change":"2024-01-11T08:23:04.859381+0000","last_active":"2024-01-11T08:23:05.846954+0000","last_peered":"2024-01-11T08:23:05.846954+0000","last_clean":"2024-01-11T08:23:05.846954+0000","last_became_active":"2024-01-11T08:23:04.859101+0000","last_became_peered":"2024-01-11T08:23:04.859101+0000","last_unstale":"2024-01-11T08:23:05.846954+0000","last_undegraded":"2024-01-11T08:23:05.846954+0000","last_fullsized":"2024-01-11T08:23:05.846954+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:56:13.627003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.872166+0000","last_change":"2024-01-11T08:23:05.869943+0000","last_active":"2024-01-11T08:23:05.872166+0000","last_peered":"2024-01-11T08:23:05.872166+0000","last_clean":"2024-01-11T08:23:05.872166+0000","last_became_active":"2024-01-11T08:23:05.868835+0000","last_became_peered":"2024-01-11T08:23:05.868835+0000","last_unstale":"2024-01-11T08:23:05.872166+0000","last_undegraded":"2024-01-11T08:23:05.872166+0000","last_fullsized":"2024-01-11T08:23:05.872166+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:36:07.478679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.856492+0000","last_change":"2024-01-11T08:23:04.860552+0000","last_active":"2024-01-11T08:23:05.856492+0000","last_peered":"2024-01-11T08:23:05.856492+0000","last_clean":"2024-01-11T08:23:05.856492+0000","last_became_active":"2024-01-11T08:23:04.860118+0000","last_became_peered":"2024-01-11T08:23:04.860118+0000","last_unstale":"2024-01-11T08:23:05.856492+0000","last_undegraded":"2024-01-11T08:23:05.856492+0000","last_fullsized":"2024-01-11T08:23:05.856492+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:17:24.867381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.862432+0000","last_change":"2024-01-11T08:23:05.860271+0000","last_active":"2024-01-11T08:23:05.862432+0000","last_peered":"2024-01-11T08:23:05.862432+0000","last_clean":"2024-01-11T08:23:05.862432+0000","last_became_active":"2024-01-11T08:23:05.859968+0000","last_became_peered":"2024-01-11T08:23:05.859968+0000","last_unstale":"2024-01-11T08:23:05.862432+0000","last_undegraded":"2024-01-11T08:23:05.862432+0000","last_fullsized":"2024-01-11T08:23:05.862432+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:04:56.439046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-11T08:23:04.868484+0000","last_change":"2024-01-11T08:23:04.866026+0000","last_active":"2024-01-11T08:23:04.868484+0000","last_peered":"2024-01-11T08:23:04.868484+0000","last_clean":"2024-01-11T08:23:04.868484+0000","last_became_active":"2024-01-11T08:23:04.864799+0000","last_became_peered":"2024-01-11T08:23:04.864799+0000","last_unstale":"2024-01-11T08:23:04.868484+0000","last_undegraded":"2024-01-11T08:23:04.868484+0000","last_fullsized":"2024-01-11T08:23:04.868484+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:14:20.580853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":8,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.876603+0000","last_change":"2024-01-11T08:23:05.876603+0000","last_active":"2024-01-11T08:23:05.876603+0000","last_peered":"2024-01-11T08:23:05.876603+0000","last_clean":"2024-01-11T08:23:05.876603+0000","last_became_active":"2024-01-11T08:23:05.876336+0000","last_became_peered":"2024-01-11T08:23:05.876336+0000","last_unstale":"2024-01-11T08:23:05.876603+0000","last_undegraded":"2024-01-11T08:23:05.876603+0000","last_fullsized":"2024-01-11T08:23:05.876603+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:04.826152+0000","last_clean_scrub_stamp":"2024-01-11T08:23:04.826152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:52:16.057511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"58'2","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.870384+0000","last_change":"2024-01-11T08:23:04.855083+0000","last_active":"2024-01-11T08:23:05.870384+0000","last_peered":"2024-01-11T08:23:05.870384+0000","last_clean":"2024-01-11T08:23:05.870384+0000","last_became_active":"2024-01-11T08:23:04.854808+0000","last_became_peered":"2024-01-11T08:23:04.854808+0000","last_unstale":"2024-01-11T08:23:05.870384+0000","last_undegraded":"2024-01-11T08:23:05.870384+0000","last_fullsized":"2024-01-11T08:23:05.870384+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:23:03.816592+0000","last_clean_scrub_stamp":"2024-01-11T08:23:03.816592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:20:12.807666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"49'139","reported_seq":271,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-11T08:23:05.852819+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:23:05.852819+0000","last_peered":"2024-01-11T08:23:05.852819+0000","last_clean":"2024-01-11T08:23:05.852819+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:23:05.852819+0000","last_undegraded":"2024-01-11T08:23:05.852819+0000","last_fullsized":"2024-01-11T08:23:05.852819+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575004,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302652,"kb_used_data":1268,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069188,"statfs":{"total":96636764160,"available":96326848512,"internally_reserved":0,"allocated":1298432,"data_stored":908723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.70699999999999996,"15min":0.70499999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.996,"5min":0.996,"15min":2.8300000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.72299999999999998,"15min":0.70299999999999996},"min":{"1min":0.503,"5min":0.46200000000000002,"15min":0.372},"max":{"1min":0.91500000000000004,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.752}]},{"osd":1,"last update":"Thu Jan 11 08:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.71099999999999997,"15min":0.69299999999999995},"min":{"1min":0.45600000000000002,"5min":0.41899999999999998,"15min":0.35799999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.998},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.73099999999999998,"15min":0.71599999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.95899999999999996,"5min":1.014,"15min":1.153},"last":0.56599999999999995}]},{"osd":2,"last update":"Thu Jan 11 08:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.748,"15min":0.71799999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.92900000000000005,"5min":1.03,"15min":1.034},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.73499999999999999,"15min":0.71799999999999997},"min":{"1min":0.46999999999999997,"5min":0.46400000000000002,"15min":0.41099999999999998},"max":{"1min":0.99399999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.70099999999999996}]},{"osd":3,"last update":"Thu Jan 11 08:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.75600000000000001,"15min":0.76800000000000002},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.40300000000000002},"max":{"1min":1.032,"5min":1.032,"15min":2.9119999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.748,"15min":0.73999999999999999},"min":{"1min":0.50800000000000001,"5min":0.502,"15min":0.48999999999999999},"max":{"1min":0.93300000000000005,"5min":0.97599999999999998,"15min":0.97799999999999998},"last":0.73899999999999999}]},{"osd":4,"last update":"Thu Jan 11 08:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59399999999999997,"15min":0.61799999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":1.089},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.625,"5min":0.627,"15min":0.64500000000000002},"min":{"1min":0.442,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.872,"5min":0.93400000000000005,"15min":1.1020000000000001},"last":0.82299999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.63700000000000001,"15min":0.65800000000000003},"min":{"1min":0.42199999999999999,"5min":0.41699999999999998,"15min":0.34599999999999997},"max":{"1min":0.92200000000000004,"5min":0.92800000000000005,"15min":1.0589999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66800000000000004,"15min":0.68899999999999995},"min":{"1min":0.47199999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.032},"last":0.89000000000000001}]},{"osd":6,"last update":"Thu Jan 11 08:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.76000000000000001,"15min":0.77000000000000002},"min":{"1min":0.55800000000000005,"5min":0.501,"15min":0.46999999999999997},"max":{"1min":0.94099999999999995,"5min":1.0009999999999999,"15min":1.1359999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.73299999999999998,"15min":0.754},"min":{"1min":0.57299999999999995,"5min":0.45200000000000001,"15min":0.42099999999999999},"max":{"1min":0.99099999999999999,"5min":1.018,"15min":1.1299999999999999},"last":0.76500000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575003,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302064,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069776,"statfs":{"total":96636764160,"available":96327450624,"internally_reserved":0,"allocated":696320,"data_stored":312621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.66700000000000004,"15min":0.64800000000000002},"min":{"1min":0.38700000000000001,"5min":0.379,"15min":0.29999999999999999},"max":{"1min":0.85399999999999998,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.70199999999999996,"15min":0.67300000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":0.91200000000000003,"5min":0.96999999999999997,"15min":1},"last":0.60499999999999998}]},{"osd":1,"last update":"Thu Jan 11 08:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.70799999999999996,"15min":0.66900000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.35899999999999999},"max":{"1min":0.94699999999999995,"5min":1.163,"15min":1.163},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.70499999999999996,"15min":0.69399999999999995},"min":{"1min":0.52300000000000002,"5min":0.47599999999999998,"15min":0.39900000000000002},"max":{"1min":0.92900000000000005,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.55400000000000005}]},{"osd":2,"last update":"Thu Jan 11 08:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.71799999999999997,"15min":0.69699999999999995},"min":{"1min":0.50600000000000001,"5min":0.48399999999999999,"15min":0.38600000000000001},"max":{"1min":0.97199999999999998,"5min":1.016,"15min":1.016},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.73199999999999998,"15min":0.71599999999999997},"min":{"1min":0.53600000000000003,"5min":0.46200000000000002,"15min":0.373},"max":{"1min":0.88900000000000001,"5min":0.96899999999999997,"15min":0.995},"last":0.72699999999999998}]},{"osd":3,"last update":"Thu Jan 11 08:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.73199999999999998,"15min":0.73599999999999999},"min":{"1min":0.47699999999999998,"5min":0.46800000000000003,"15min":0.42799999999999999},"max":{"1min":0.95799999999999996,"5min":1.0149999999999999,"15min":1.3260000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.75800000000000001,"15min":0.73099999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.44800000000000001},"max":{"1min":0.95899999999999996,"5min":0.97999999999999998,"15min":1.0049999999999999},"last":0.75900000000000001}]},{"osd":4,"last update":"Thu Jan 11 08:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54800000000000004,"15min":0.58899999999999997},"min":{"1min":0.39700000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.874,"5min":0.874,"15min":1.212},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.58599999999999997,"15min":0.63700000000000001},"min":{"1min":0.41499999999999998,"5min":0.38900000000000001,"15min":0.374},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":1.8320000000000001},"last":0.66800000000000004}]},{"osd":5,"last update":"Thu Jan 11 08:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64700000000000002,"15min":0.66600000000000004},"min":{"1min":0.443,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.85699999999999998,"5min":0.92800000000000005,"15min":1.0209999999999999},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.69999999999999996,"15min":0.72399999999999998},"min":{"1min":0.55300000000000005,"5min":0.47399999999999998,"15min":0.41899999999999998},"max":{"1min":0.88700000000000001,"5min":0.93700000000000006,"15min":1.7929999999999999},"last":0.87}]},{"osd":7,"last update":"Thu Jan 11 08:22:34 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.69999999999999996,"15min":0.73299999999999998},"min":{"1min":0.56799999999999995,"5min":0.42399999999999999,"15min":0.41099999999999998},"max":{"1min":0.84399999999999997,"5min":0.90900000000000003,"15min":1.278},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.75900000000000001,"15min":0.75700000000000001},"min":{"1min":0.623,"5min":0.48299999999999998,"15min":0.44700000000000001},"max":{"1min":0.94799999999999995,"5min":0.99399999999999999,"15min":1.1599999999999999},"last":0.98399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575005,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302060,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069780,"statfs":{"total":96636764160,"available":96327454720,"internally_reserved":0,"allocated":692224,"data_stored":312047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.32100000000000001,"5min":0.317,"15min":0.26900000000000002},"max":{"1min":0.624,"5min":0.74399999999999999,"15min":0.97699999999999998},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59899999999999998,"15min":0.57299999999999995},"min":{"1min":0.437,"5min":0.436,"15min":0.373},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.90700000000000003},"last":0.51400000000000001}]},{"osd":2,"last update":"Thu Jan 11 08:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.5,"15min":0.48899999999999999},"min":{"1min":0.35699999999999998,"5min":0.33400000000000002,"15min":0.32400000000000001},"max":{"1min":0.66000000000000003,"5min":1.113,"15min":1.113},"last":0.432},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56399999999999995,"15min":0.56899999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":1.25},"last":0.48999999999999999}]},{"osd":3,"last update":"Thu Jan 11 08:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.621,"15min":0.61599999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73699999999999999,"5min":0.86599999999999999,"15min":0.92600000000000005},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57899999999999996,"15min":0.56699999999999995},"min":{"1min":0.374,"5min":0.372,"15min":0.34899999999999998},"max":{"1min":0.93400000000000005,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.46000000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.81299999999999994,"15min":0.79200000000000004},"min":{"1min":0.55600000000000005,"5min":0.45700000000000002,"15min":0.44600000000000001},"max":{"1min":0.98999999999999999,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.84499999999999997,"15min":0.83899999999999997},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56000000000000005},"max":{"1min":1.0740000000000001,"5min":1.246,"15min":1.3680000000000001},"last":0.73999999999999999}]},{"osd":5,"last update":"Thu Jan 11 08:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.82299999999999995,"15min":0.81399999999999995},"min":{"1min":0.503,"5min":0.503,"15min":0.46999999999999997},"max":{"1min":1.052,"5min":1.1759999999999999,"15min":1.7849999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.80600000000000005,"15min":0.80900000000000005},"min":{"1min":0.52700000000000002,"5min":0.51200000000000001,"15min":0.50700000000000001},"max":{"1min":1.0129999999999999,"5min":1.1579999999999999,"15min":1.371},"last":0.77000000000000002}]},{"osd":6,"last update":"Thu Jan 11 08:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.83199999999999996,"15min":0.82499999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0600000000000001,"5min":1.0840000000000001,"15min":1.413},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.84999999999999998,"15min":0.82199999999999995},"min":{"1min":0.629,"5min":0.56999999999999995,"15min":0.55500000000000005},"max":{"1min":1.087,"5min":1.228,"15min":1.228},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Jan 11 08:22:35 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.875,"15min":0.84499999999999997},"min":{"1min":0.65900000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1180000000000001,"5min":1.204,"15min":1.226},"last":0.629},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.85299999999999998,"15min":0.84099999999999997},"min":{"1min":0.58599999999999997,"5min":0.57999999999999996,"15min":0.51400000000000001},"max":{"1min":1.0369999999999999,"5min":1.258,"15min":1.321},"last":0.73099999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575004,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298544,"kb_used_data":1256,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073296,"statfs":{"total":96636764160,"available":96331055104,"internally_reserved":0,"allocated":1286144,"data_stored":902379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 08:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.49099999999999999,"15min":0.495},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.24299999999999999},"max":{"1min":0.625,"5min":0.89700000000000002,"15min":0.92700000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.35299999999999998},"max":{"1min":0.68799999999999994,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.63}]},{"osd":2,"last update":"Thu Jan 11 08:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.53400000000000003,"15min":0.54400000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.29099999999999998},"max":{"1min":0.60799999999999998,"5min":0.77400000000000002,"15min":0.86799999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77500000000000002,"5min":0.81299999999999994,"15min":0.83999999999999997},"last":0.56000000000000005}]},{"osd":3,"last update":"Thu Jan 11 08:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52300000000000002,"15min":0.51500000000000001},"min":{"1min":0.39900000000000002,"5min":0.36199999999999999,"15min":0.33600000000000002},"max":{"1min":0.71899999999999997,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.437},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61899999999999999,"15min":0.61799999999999999},"min":{"1min":0.46400000000000002,"5min":0.40400000000000003,"15min":0.39600000000000002},"max":{"1min":0.69999999999999996,"5min":0.877,"15min":0.91000000000000003},"last":0.59999999999999998}]},{"osd":4,"last update":"Thu Jan 11 08:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.78400000000000003,"15min":0.77900000000000003},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.437},"max":{"1min":0.91100000000000003,"5min":1.165,"15min":1.165},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.78300000000000003,"15min":0.78200000000000003},"min":{"1min":0.53100000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.038,"5min":1.038,"15min":1.2350000000000001},"last":0.58199999999999996}]},{"osd":5,"last update":"Thu Jan 11 08:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.80200000000000005,"15min":0.80600000000000005},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.505},"max":{"1min":0.98899999999999999,"5min":1.048,"15min":1.2929999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.80300000000000005,"15min":0.81200000000000006},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.47799999999999998},"max":{"1min":0.96099999999999997,"5min":1.0800000000000001,"15min":1.2130000000000001},"last":0.71899999999999997}]},{"osd":6,"last update":"Thu Jan 11 08:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.79600000000000004,"15min":0.79500000000000004},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.497},"max":{"1min":0.97999999999999998,"5min":1.1419999999999999,"15min":1.2689999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.81699999999999995,"15min":0.83199999999999996},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.44900000000000001},"max":{"1min":0.93999999999999995,"5min":1.1339999999999999,"15min":1.2749999999999999},"last":0.73599999999999999}]},{"osd":7,"last update":"Thu Jan 11 08:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.84799999999999998,"15min":0.84199999999999997},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.51400000000000001},"max":{"1min":1.0680000000000001,"5min":1.21,"15min":1.446},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.83899999999999997,"15min":0.83999999999999997},"min":{"1min":0.622,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.0620000000000001,"5min":1.1930000000000001,"15min":1.298},"last":0.69499999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575003,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306180,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065660,"statfs":{"total":96636764160,"available":96323235840,"internally_reserved":0,"allocated":716800,"data_stored":319823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46800000000000003,"15min":0.46500000000000002},"min":{"1min":0.307,"5min":0.30299999999999999,"15min":0.26000000000000001},"max":{"1min":0.73499999999999999,"5min":0.82799999999999996,"15min":0.86099999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.55500000000000005,"15min":0.55900000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.28899999999999998},"max":{"1min":0.624,"5min":0.91400000000000003,"15min":1.4119999999999999},"last":0.39700000000000002}]},{"osd":1,"last update":"Thu Jan 11 08:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.30399999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.90000000000000002},"last":0.372},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59199999999999997,"15min":0.59699999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.38100000000000001},"max":{"1min":0.92900000000000005,"5min":0.94499999999999995,"15min":1.5640000000000001},"last":0.442}]},{"osd":3,"last update":"Thu Jan 11 08:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.54300000000000004,"15min":0.55000000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.32900000000000001},"max":{"1min":0.67700000000000005,"5min":0.84999999999999998,"15min":1.1220000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63600000000000001,"15min":0.65000000000000002},"min":{"1min":0.45300000000000001,"5min":0.439,"15min":0.41799999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.22},"last":0.50700000000000001}]},{"osd":4,"last update":"Thu Jan 11 08:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.81100000000000005,"15min":0.79800000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.97999999999999998,"5min":1.2889999999999999,"15min":1.488},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.81200000000000006,"15min":0.79100000000000004},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.75900000000000001}]},{"osd":5,"last update":"Thu Jan 11 08:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.85499999999999998,"15min":0.86599999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.51600000000000001},"max":{"1min":1.214,"5min":1.3540000000000001,"15min":2.9209999999999998},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.85599999999999998,"15min":0.86099999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55400000000000005},"max":{"1min":1.1399999999999999,"5min":1.23,"15min":1.845},"last":0.98299999999999998}]},{"osd":6,"last update":"Thu Jan 11 08:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82799999999999996,"15min":0.80600000000000005},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.51700000000000002},"max":{"1min":1.1919999999999999,"5min":1.2669999999999999,"15min":1.3560000000000001},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.84399999999999997,"15min":0.85599999999999998},"min":{"1min":0.66800000000000004,"5min":0.61399999999999999,"15min":0.56299999999999994},"max":{"1min":1.123,"5min":1.47,"15min":2.968},"last":0.81899999999999995}]},{"osd":7,"last update":"Thu Jan 11 08:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.879,"15min":0.88300000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.2330000000000001,"5min":1.4099999999999999,"15min":1.6359999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.88600000000000001,"15min":0.878},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.53800000000000003},"max":{"1min":1.0900000000000001,"5min":1.327,"15min":1.391},"last":0.90800000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306176,"kb_used_data":696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065664,"statfs":{"total":96636764160,"available":96323239936,"internally_reserved":0,"allocated":712704,"data_stored":318875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.49299999999999999,"15min":0.48399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.27600000000000002},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.55700000000000005,"15min":0.54500000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.64600000000000002,"5min":0.75600000000000001,"15min":1.0109999999999999},"last":0.46300000000000002}]},{"osd":1,"last update":"Thu Jan 11 08:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.502,"15min":0.48699999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.25900000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59899999999999998,"15min":0.56999999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.32300000000000001},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Jan 11 08:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.59899999999999998,"15min":0.58599999999999997},"min":{"1min":0.39900000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.90900000000000003,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.44700000000000001,"5min":0.439,"15min":0.32000000000000001},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.45200000000000001}]},{"osd":4,"last update":"Thu Jan 11 08:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.73499999999999999,"15min":0.76100000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96099999999999997,"5min":0.997,"15min":1.351},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.78400000000000003,"15min":0.78600000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.51300000000000001},"max":{"1min":1.0569999999999999,"5min":1.1819999999999999,"15min":1.198},"last":0.71799999999999997}]},{"osd":5,"last update":"Thu Jan 11 08:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.79500000000000004,"15min":0.80300000000000005},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.51900000000000002},"max":{"1min":1.139,"5min":1.139,"15min":1.3779999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.499,"5min":0.499,"15min":0.44900000000000001},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.4239999999999999},"last":0.67800000000000005}]},{"osd":6,"last update":"Thu Jan 11 08:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.86199999999999999,"15min":0.85599999999999998},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.52600000000000002},"max":{"1min":1.335,"5min":1.335,"15min":1.927},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.80100000000000005,"15min":0.84799999999999998},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.54400000000000004},"max":{"1min":1.0569999999999999,"5min":1.2230000000000001,"15min":3.5579999999999998},"last":0.73499999999999999}]},{"osd":7,"last update":"Thu Jan 11 08:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.84299999999999997,"15min":0.88100000000000001},"min":{"1min":0.58899999999999997,"5min":0.52800000000000002,"15min":0.46600000000000003},"max":{"1min":1.1220000000000001,"5min":1.1579999999999999,"15min":3.5089999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.872,"15min":0.86399999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.59199999999999997},"max":{"1min":1.1879999999999999,"5min":1.204,"15min":1.21},"last":0.89100000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306176,"kb_used_data":696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065664,"statfs":{"total":96636764160,"available":96323239936,"internally_reserved":0,"allocated":712704,"data_stored":319801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.75600000000000001,"15min":0.73699999999999999},"min":{"1min":0.45600000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.95999999999999996,"5min":1.117,"15min":1.524},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.748,"15min":0.749},"min":{"1min":0.51600000000000001,"5min":0.47399999999999998,"15min":0.38600000000000001},"max":{"1min":0.94799999999999995,"5min":0.96999999999999997,"15min":1.177},"last":0.86399999999999999}]},{"osd":1,"last update":"Thu Jan 11 08:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.46300000000000002,"5min":0.42199999999999999,"15min":0.33300000000000002},"max":{"1min":0.88700000000000001,"5min":0.99099999999999999,"15min":1.1499999999999999},"last":0.872},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.71899999999999997,"15min":0.753},"min":{"1min":0.379,"5min":0.379,"15min":0.36299999999999999},"max":{"1min":0.874,"5min":1.046,"15min":2.4740000000000002},"last":0.85399999999999998}]},{"osd":2,"last update":"Thu Jan 11 08:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.72499999999999998,"15min":0.70899999999999996},"min":{"1min":0.53900000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94299999999999995,"5min":0.98599999999999999,"15min":1.022},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.75,"5min":0.748,"15min":0.73199999999999998},"min":{"1min":0.56699999999999995,"5min":0.48399999999999999,"15min":0.47199999999999998},"max":{"1min":0.92200000000000004,"5min":0.95699999999999996,"15min":1.212},"last":0.91500000000000004}]},{"osd":3,"last update":"Thu Jan 11 08:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.71199999999999997,"15min":0.71799999999999997},"min":{"1min":0.52700000000000002,"5min":0.433,"15min":0.38100000000000001},"max":{"1min":0.88800000000000001,"5min":1.0189999999999999,"15min":1.2649999999999999},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.747,"15min":0.75700000000000001},"min":{"1min":0.61199999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.89900000000000002,"5min":1.02,"15min":1.069},"last":0.82299999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60499999999999998,"15min":0.64200000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.35899999999999999},"max":{"1min":0.83599999999999997,"5min":0.93200000000000005,"15min":1.0700000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.623,"15min":0.65700000000000003},"min":{"1min":0.442,"5min":0.40400000000000003,"15min":0.32600000000000001},"max":{"1min":0.85899999999999999,"5min":0.94199999999999995,"15min":0.98199999999999998},"last":0.73899999999999999}]},{"osd":6,"last update":"Thu Jan 11 08:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.63700000000000001,"15min":0.67700000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85799999999999998,"5min":0.86599999999999999,"15min":1.0289999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.67800000000000005,"15min":0.72199999999999998},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.40400000000000003},"max":{"1min":0.878,"5min":0.91600000000000004,"15min":1.119},"last":0.79900000000000004}]},{"osd":7,"last update":"Thu Jan 11 08:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.66600000000000004,"15min":0.71499999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.90600000000000003,"5min":0.96999999999999997,"15min":0.98999999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.72299999999999998,"15min":0.76500000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.42299999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.4279999999999999},"last":0.90700000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575004,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302072,"kb_used_data":688,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069768,"statfs":{"total":96636764160,"available":96327442432,"internally_reserved":0,"allocated":704512,"data_stored":318335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.72999999999999998,"15min":0.70199999999999996},"min":{"1min":0.45900000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89700000000000002,"5min":1.032,"15min":1.032},"last":0.628},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.71899999999999997,"15min":0.69699999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44400000000000001},"max":{"1min":0.91400000000000003,"5min":1.0609999999999999,"15min":1.696},"last":0.64900000000000002}]},{"osd":1,"last update":"Thu Jan 11 08:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70999999999999996,"15min":0.69699999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.36399999999999999},"max":{"1min":0.84299999999999997,"5min":1.0089999999999999,"15min":1.4359999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.74299999999999999,"15min":0.73199999999999998},"min":{"1min":0.59599999999999997,"5min":0.51400000000000001,"15min":0.47299999999999998},"max":{"1min":0.88400000000000001,"5min":0.96799999999999997,"15min":4.4969999999999999},"last":0.752}]},{"osd":2,"last update":"Thu Jan 11 08:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.73399999999999999,"15min":0.70199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.92600000000000005,"5min":1.028,"15min":1.238},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.73799999999999999,"15min":0.751},"min":{"1min":0.49399999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.84199999999999997,"5min":1.0049999999999999,"15min":8.0429999999999993},"last":0.59999999999999998}]},{"osd":3,"last update":"Thu Jan 11 08:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.745,"15min":0.76500000000000001},"min":{"1min":0.50700000000000001,"5min":0.46999999999999997,"15min":0.442},"max":{"1min":0.93400000000000005,"5min":1.0449999999999999,"15min":8.0839999999999996},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.75600000000000001,"15min":0.749},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.872,"5min":0.93899999999999995,"15min":0.96999999999999997},"last":0.56599999999999995}]},{"osd":4,"last update":"Thu Jan 11 08:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61299999999999999,"15min":0.63600000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83699999999999997,"5min":0.96599999999999997,"15min":1.113},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.63100000000000001,"15min":0.66800000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.38200000000000001},"max":{"1min":0.78800000000000003,"5min":0.94399999999999995,"15min":1.002},"last":0.81799999999999995}]},{"osd":6,"last update":"Thu Jan 11 08:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65800000000000003,"15min":0.66700000000000004},"min":{"1min":0.46300000000000002,"5min":0.443,"15min":0.42699999999999999},"max":{"1min":0.86899999999999999,"5min":0.92100000000000004,"15min":1.034},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.70099999999999996,"15min":0.71099999999999997},"min":{"1min":0.53400000000000003,"5min":0.48999999999999999,"15min":0.45400000000000001},"max":{"1min":0.90100000000000002,"5min":1.004,"15min":1.143},"last":0.72399999999999998}]},{"osd":7,"last update":"Thu Jan 11 08:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.749,"15min":0.76600000000000001},"min":{"1min":0.53200000000000003,"5min":0.501,"15min":0.437},"max":{"1min":0.90800000000000003,"5min":1.028,"15min":1.1399999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.78600000000000003,"15min":0.78500000000000003},"min":{"1min":0.56799999999999995,"5min":0.53500000000000003,"15min":0.50600000000000001},"max":{"1min":0.96499999999999997,"5min":1.0489999999999999,"15min":1.079},"last":0.76700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:23:10.770 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-11T08:23:10.770 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-11T08:23:11.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.013+0000 7f8b1b5ff700 1 Processor -- start 2024-01-11T08:23:11.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.013+0000 7f8b1b5ff700 1 -- start start 2024-01-11T08:23:11.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.014+0000 7f8b1b5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8b1406bba0 0x7f8b140cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:11.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.014+0000 7f8b1b5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b1406bf70 0x7f8b14159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:11.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.014+0000 7f8b1b5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1415d670 con 0x7f8b1415db00 2024-01-11T08:23:11.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.014+0000 7f8b1b5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b14069fc0 con 0x7f8b1406bf70 2024-01-11T08:23:11.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.014+0000 7f8b1b5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b140cce10 con 0x7f8b1406bba0 2024-01-11T08:23:11.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.015+0000 7f8b18b9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b1406bf70 0x7f8b14159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:11.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.015+0000 7f8b18b9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b1406bf70 0x7f8b14159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33698/0 (socket says 172.21.15.27:33698) 2024-01-11T08:23:11.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.015+0000 7f8b18b9a700 1 -- 172.21.15.27:0/1751845130 learned_addr learned my addr 172.21.15.27:0/1751845130 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:11.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.015+0000 7f8b1939b700 1 --2- 172.21.15.27:0/1751845130 >> v2:172.21.15.77:3300/0 conn(0x7f8b1406bba0 0x7f8b140cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:11.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.015+0000 7f8b18b9a700 1 -- 172.21.15.27:0/1751845130 >> v2:172.21.15.77:3300/0 conn(0x7f8b1406bba0 msgr2=0x7f8b140cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:11.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.015+0000 7f8b18b9a700 1 --2- 172.21.15.27:0/1751845130 >> v2:172.21.15.77:3300/0 conn(0x7f8b1406bba0 0x7f8b140cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.015+0000 7f8b18b9a700 1 -- 172.21.15.27:0/1751845130 >> v1:172.21.15.77:6789/0 conn(0x7f8b1415db00 legacy=0x7f8b140c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:11.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.015+0000 7f8b18b9a700 1 -- 172.21.15.27:0/1751845130 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1415a380 con 0x7f8b1406bf70 2024-01-11T08:23:11.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.016+0000 7f8b18b9a700 1 --2- 172.21.15.27:0/1751845130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b1406bf70 0x7f8b14159e40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8b140adc70 tx=0x7f8b04009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b904eb06ffb34fff server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:11.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.016+0000 7f8b0bfff700 1 -- 172.21.15.27:0/1751845130 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0400b820 con 0x7f8b1406bf70 2024-01-11T08:23:11.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.016+0000 7f8b0bfff700 1 -- 172.21.15.27:0/1751845130 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0400b980 con 0x7f8b1406bf70 2024-01-11T08:23:11.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.016+0000 7f8b0bfff700 1 -- 172.21.15.27:0/1751845130 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0400bd20 con 0x7f8b1406bf70 2024-01-11T08:23:11.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.016+0000 7f8b1b5ff700 1 -- 172.21.15.27:0/1751845130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b1406bf70 msgr2=0x7f8b14159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:11.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.016+0000 7f8b1b5ff700 1 --2- 172.21.15.27:0/1751845130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b1406bf70 0x7f8b14159e40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8b140adc70 tx=0x7f8b04009250 comp rx=0 tx=0).stop 2024-01-11T08:23:11.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.017+0000 7f8b1b5ff700 1 -- 172.21.15.27:0/1751845130 shutdown_connections 2024-01-11T08:23:11.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.017+0000 7f8b1b5ff700 1 --2- 172.21.15.27:0/1751845130 >> v2:172.21.15.77:3300/0 conn(0x7f8b1406bba0 0x7f8b140cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.017+0000 7f8b1b5ff700 1 --2- 172.21.15.27:0/1751845130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b1406bf70 0x7f8b14159e40 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.017+0000 7f8b1b5ff700 1 -- 172.21.15.27:0/1751845130 >> 172.21.15.27:0/1751845130 conn(0x7f8b140ce3e0 msgr2=0x7f8b140ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:11.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.017+0000 7f8b1b5ff700 1 -- 172.21.15.27:0/1751845130 shutdown_connections 2024-01-11T08:23:11.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.017+0000 7f8b1b5ff700 1 -- 172.21.15.27:0/1751845130 wait complete. 2024-01-11T08:23:11.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.018+0000 7f8b1b5ff700 1 Processor -- start 2024-01-11T08:23:11.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.018+0000 7f8b1b5ff700 1 -- start start 2024-01-11T08:23:11.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.019+0000 7f8b1b5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b1406bba0 0x7f8b140caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:11.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.019+0000 7f8b1b5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b1406bf70 0x7f8b140cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:11.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.019+0000 7f8b1b5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b1415db00 0x7f8b14169e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:11.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.019+0000 7f8b1b5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b140afa20 con 0x7f8b1415db00 2024-01-11T08:23:11.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.019+0000 7f8b1b5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b140aee10 con 0x7f8b1406bf70 2024-01-11T08:23:11.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.019+0000 7f8b1b5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b140af0f0 con 0x7f8b1406bba0 2024-01-11T08:23:11.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.020+0000 7f8b19b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b1415db00 0x7f8b14169e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:11.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.020+0000 7f8b19b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b1415db00 0x7f8b14169e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33704/0 (socket says 172.21.15.27:33704) 2024-01-11T08:23:11.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.020+0000 7f8b19b9c700 1 -- 172.21.15.27:0/1337096126 learned_addr learned my addr 172.21.15.27:0/1337096126 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:11.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.020+0000 7f8b1939b700 1 --2- 172.21.15.27:0/1337096126 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b1406bba0 0x7f8b140caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:11.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.020+0000 7f8b18b9a700 1 --2- 172.21.15.27:0/1337096126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b1406bf70 0x7f8b140cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:11.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.020+0000 7f8b19b9c700 1 -- 172.21.15.27:0/1337096126 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b1406bba0 msgr2=0x7f8b140caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:11.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.020+0000 7f8b19b9c700 1 --2- 172.21.15.27:0/1337096126 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b1406bba0 0x7f8b140caf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.020+0000 7f8b19b9c700 1 -- 172.21.15.27:0/1337096126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b1406bf70 msgr2=0x7f8b140cb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:11.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.020+0000 7f8b19b9c700 1 --2- 172.21.15.27:0/1337096126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b1406bf70 0x7f8b140cb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.020+0000 7f8b19b9c700 1 -- 172.21.15.27:0/1337096126 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b040094f0 con 0x7f8b1415db00 2024-01-11T08:23:11.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.020+0000 7f8b19b9c700 1 --2- 172.21.15.27:0/1337096126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b1415db00 0x7f8b14169e80 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c00c930 tx=0x7f8b0c0026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:11.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.021+0000 7f8b097fa700 1 -- 172.21.15.27:0/1337096126 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c012630 con 0x7f8b1415db00 2024-01-11T08:23:11.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.021+0000 7f8b097fa700 1 -- 172.21.15.27:0/1337096126 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0c0127d0 con 0x7f8b1415db00 2024-01-11T08:23:11.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.021+0000 7f8b097fa700 1 -- 172.21.15.27:0/1337096126 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c012b90 con 0x7f8b1415db00 2024-01-11T08:23:11.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.021+0000 7f8b1b5ff700 1 -- 172.21.15.27:0/1337096126 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b140af3d0 con 0x7f8b1415db00 2024-01-11T08:23:11.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.021+0000 7f8b1b5ff700 1 -- 172.21.15.27:0/1337096126 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b140b3f10 con 0x7f8b1415db00 2024-01-11T08:23:11.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.022+0000 7f8b1b5ff700 1 -- 172.21.15.27:0/1337096126 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b140afb90 con 0x7f8b1415db00 2024-01-11T08:23:11.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.023+0000 7f8b097fa700 1 -- 172.21.15.27:0/1337096126 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8b0c01a3b0 con 0x7f8b1415db00 2024-01-11T08:23:11.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.024+0000 7f8b097fa700 1 --2- 172.21.15.27:0/1337096126 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8af8066f30 0x7f8af80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:11.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.024+0000 7f8b097fa700 1 -- 172.21.15.27:0/1337096126 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f8b0c084180 con 0x7f8b1415db00 2024-01-11T08:23:11.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.025+0000 7f8b1939b700 1 --2- 172.21.15.27:0/1337096126 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8af8066f30 0x7f8af80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:11.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.025+0000 7f8b1939b700 1 --2- 172.21.15.27:0/1337096126 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8af8066f30 0x7f8af80693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8b10006fd0 tx=0x7f8b10008040 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:11.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.028+0000 7f8b097fa700 1 -- 172.21.15.27:0/1337096126 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8b0c054e80 con 0x7f8b1415db00 2024-01-11T08:23:11.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.313+0000 7f8b1b5ff700 1 -- 172.21.15.27:0/1337096126 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f8b140afdb0 con 0x7f8b1415db00 2024-01-11T08:23:11.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.318+0000 7f8b097fa700 1 -- 172.21.15.27:0/1337096126 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+0 (secure 0 0 0) 0x7f8b0c054a10 con 0x7f8b1415db00 2024-01-11T08:23:11.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.320+0000 7f8b1b5ff700 1 -- 172.21.15.27:0/1337096126 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8af8066f30 msgr2=0x7f8af80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:11.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.320+0000 7f8b1b5ff700 1 --2- 172.21.15.27:0/1337096126 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8af8066f30 0x7f8af80693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8b10006fd0 tx=0x7f8b10008040 comp rx=0 tx=0).stop 2024-01-11T08:23:11.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.320+0000 7f8b1b5ff700 1 -- 172.21.15.27:0/1337096126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b1415db00 msgr2=0x7f8b14169e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:11.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.320+0000 7f8b1b5ff700 1 --2- 172.21.15.27:0/1337096126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b1415db00 0x7f8b14169e80 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c00c930 tx=0x7f8b0c0026e0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.320+0000 7f8b1b5ff700 1 -- 172.21.15.27:0/1337096126 shutdown_connections 2024-01-11T08:23:11.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.320+0000 7f8b1b5ff700 1 --2- 172.21.15.27:0/1337096126 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8af8066f30 0x7f8af80693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.320+0000 7f8b1b5ff700 1 --2- 172.21.15.27:0/1337096126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b1415db00 0x7f8b14169e80 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.320+0000 7f8b1b5ff700 1 --2- 172.21.15.27:0/1337096126 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b1406bba0 0x7f8b140caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.320+0000 7f8b1b5ff700 1 --2- 172.21.15.27:0/1337096126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b1406bf70 0x7f8b140cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.320+0000 7f8b1b5ff700 1 -- 172.21.15.27:0/1337096126 >> 172.21.15.27:0/1337096126 conn(0x7f8b140ce3e0 msgr2=0x7f8b140c3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:11.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.320+0000 7f8b1b5ff700 1 -- 172.21.15.27:0/1337096126 shutdown_connections 2024-01-11T08:23:11.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.320+0000 7f8b1b5ff700 1 -- 172.21.15.27:0/1337096126 wait complete. 2024-01-11T08:23:11.321 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-11T08:23:11.338 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:23:11.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.537+0000 7f73513ff700 1 Processor -- start 2024-01-11T08:23:11.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.537+0000 7f73513ff700 1 -- start start 2024-01-11T08:23:11.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.538+0000 7f73513ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f734c1575e0 0x7f734c158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:11.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.539+0000 7f73513ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f734c0b19d0 0x7f734c159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:11.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.539+0000 7f73513ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c158b60 con 0x7f734c06b610 2024-01-11T08:23:11.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.539+0000 7f73513ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f734c0cb5e0 con 0x7f734c0b19d0 2024-01-11T08:23:11.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.539+0000 7f73513ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734c1599b0 con 0x7f734c1575e0 2024-01-11T08:23:11.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.539+0000 7f734b7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f734c0b19d0 0x7f734c159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:11.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.539+0000 7f734b7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f734c0b19d0 0x7f734c159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33706/0 (socket says 172.21.15.27:33706) 2024-01-11T08:23:11.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.539+0000 7f734b7fe700 1 -- 172.21.15.27:0/2769015889 learned_addr learned my addr 172.21.15.27:0/2769015889 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:11.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.539+0000 7f734affd700 1 --2- 172.21.15.27:0/2769015889 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f734c1575e0 0x7f734c158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:11.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.539+0000 7f734b7fe700 1 -- 172.21.15.27:0/2769015889 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f734c1575e0 msgr2=0x7f734c158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:11.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.539+0000 7f734b7fe700 1 --2- 172.21.15.27:0/2769015889 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f734c1575e0 0x7f734c158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.539+0000 7f734b7fe700 1 -- 172.21.15.27:0/2769015889 >> v1:172.21.15.77:6789/0 conn(0x7f734c06b610 legacy=0x7f734c0b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:11.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.539+0000 7f734b7fe700 1 -- 172.21.15.27:0/2769015889 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c0c0f90 con 0x7f734c0b19d0 2024-01-11T08:23:11.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.540+0000 7f734b7fe700 1 --2- 172.21.15.27:0/2769015889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f734c0b19d0 0x7f734c159e40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7344009f20 tx=0x7f7344009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f87a2eca5bc4fd server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:11.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.540+0000 7f7349ffb700 1 -- 172.21.15.27:0/2769015889 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734400e070 con 0x7f734c0b19d0 2024-01-11T08:23:11.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.540+0000 7f7349ffb700 1 -- 172.21.15.27:0/2769015889 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734400ba80 con 0x7f734c0b19d0 2024-01-11T08:23:11.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.540+0000 7f7349ffb700 1 -- 172.21.15.27:0/2769015889 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73440092e0 con 0x7f734c0b19d0 2024-01-11T08:23:11.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.540+0000 7f73513ff700 1 -- 172.21.15.27:0/2769015889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f734c0b19d0 msgr2=0x7f734c159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:11.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.540+0000 7f73513ff700 1 --2- 172.21.15.27:0/2769015889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f734c0b19d0 0x7f734c159e40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7344009f20 tx=0x7f7344009840 comp rx=0 tx=0).stop 2024-01-11T08:23:11.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.540+0000 7f73513ff700 1 -- 172.21.15.27:0/2769015889 shutdown_connections 2024-01-11T08:23:11.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.540+0000 7f73513ff700 1 --2- 172.21.15.27:0/2769015889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f734c0b19d0 0x7f734c159e40 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.540+0000 7f73513ff700 1 --2- 172.21.15.27:0/2769015889 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f734c1575e0 0x7f734c158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.540+0000 7f73513ff700 1 -- 172.21.15.27:0/2769015889 >> 172.21.15.27:0/2769015889 conn(0x7f734c0680e0 msgr2=0x7f734c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:11.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.541+0000 7f73513ff700 1 -- 172.21.15.27:0/2769015889 shutdown_connections 2024-01-11T08:23:11.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.541+0000 7f73513ff700 1 -- 172.21.15.27:0/2769015889 wait complete. 2024-01-11T08:23:11.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.542+0000 7f73513ff700 1 Processor -- start 2024-01-11T08:23:11.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.542+0000 7f73513ff700 1 -- start start 2024-01-11T08:23:11.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.543+0000 7f73513ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f734c06b610 0x7f734c146cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:11.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.543+0000 7f73513ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f734c0b19d0 0x7f734c0cbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:11.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.543+0000 7f73513ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f734c1575e0 0x7f734c0cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:11.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.543+0000 7f73513ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734c1698a0 con 0x7f734c0b19d0 2024-01-11T08:23:11.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.543+0000 7f73513ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734c16a420 con 0x7f734c06b610 2024-01-11T08:23:11.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.543+0000 7f73513ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734c169af0 con 0x7f734c1575e0 2024-01-11T08:23:11.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.543+0000 7f734a7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f734c0b19d0 0x7f734c0cbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:11.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.543+0000 7f734a7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f734c0b19d0 0x7f734c0cbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33722/0 (socket says 172.21.15.27:33722) 2024-01-11T08:23:11.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.544+0000 7f734a7fc700 1 -- 172.21.15.27:0/440791296 learned_addr learned my addr 172.21.15.27:0/440791296 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:11.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.544+0000 7f734b7fe700 1 --2- 172.21.15.27:0/440791296 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f734c1575e0 0x7f734c0cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:11.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.544+0000 7f734affd700 1 --2- 172.21.15.27:0/440791296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f734c06b610 0x7f734c146cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:11.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.544+0000 7f734a7fc700 1 -- 172.21.15.27:0/440791296 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f734c1575e0 msgr2=0x7f734c0cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:11.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.544+0000 7f734a7fc700 1 --2- 172.21.15.27:0/440791296 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f734c1575e0 0x7f734c0cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.544+0000 7f734a7fc700 1 -- 172.21.15.27:0/440791296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f734c06b610 msgr2=0x7f734c146cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:11.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.544+0000 7f734a7fc700 1 --2- 172.21.15.27:0/440791296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f734c06b610 0x7f734c146cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.544+0000 7f734a7fc700 1 -- 172.21.15.27:0/440791296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7344009ae0 con 0x7f734c0b19d0 2024-01-11T08:23:11.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.544+0000 7f734a7fc700 1 --2- 172.21.15.27:0/440791296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f734c0b19d0 0x7f734c0cbd90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f734000c590 tx=0x7f734000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:11.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.544+0000 7f7332ffd700 1 -- 172.21.15.27:0/440791296 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7340011590 con 0x7f734c0b19d0 2024-01-11T08:23:11.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.544+0000 7f7332ffd700 1 -- 172.21.15.27:0/440791296 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7340011710 con 0x7f734c0b19d0 2024-01-11T08:23:11.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.544+0000 7f7332ffd700 1 -- 172.21.15.27:0/440791296 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7340011ad0 con 0x7f734c0b19d0 2024-01-11T08:23:11.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.544+0000 7f73513ff700 1 -- 172.21.15.27:0/440791296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f734c169dd0 con 0x7f734c0b19d0 2024-01-11T08:23:11.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.545+0000 7f73513ff700 1 -- 172.21.15.27:0/440791296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f734c147200 con 0x7f734c0b19d0 2024-01-11T08:23:11.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.546+0000 7f73513ff700 1 -- 172.21.15.27:0/440791296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f734c16a590 con 0x7f734c0b19d0 2024-01-11T08:23:11.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.547+0000 7f7332ffd700 1 -- 172.21.15.27:0/440791296 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f734001b3b0 con 0x7f734c0b19d0 2024-01-11T08:23:11.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.548+0000 7f7332ffd700 1 --2- 172.21.15.27:0/440791296 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7338066f30 0x7f73380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:11.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.548+0000 7f7332ffd700 1 -- 172.21.15.27:0/440791296 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f7340083890 con 0x7f734c0b19d0 2024-01-11T08:23:11.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.548+0000 7f734affd700 1 --2- 172.21.15.27:0/440791296 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7338066f30 0x7f73380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:11.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.549+0000 7f734affd700 1 --2- 172.21.15.27:0/440791296 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7338066f30 0x7f73380693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f733c006fd0 tx=0x7f733c008040 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:11.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.552+0000 7f7332ffd700 1 -- 172.21.15.27:0/440791296 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7340050a60 con 0x7f734c0b19d0 2024-01-11T08:23:11.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.797+0000 7f73513ff700 1 -- 172.21.15.27:0/440791296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f734c158b60 con 0x7f734c0b19d0 2024-01-11T08:23:11.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.797+0000 7f7332ffd700 1 -- 172.21.15.27:0/440791296 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4594 (secure 0 0 0) 0x7f734c158b60 con 0x7f734c0b19d0 2024-01-11T08:23:11.797 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:23:11.797 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5243,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2418922463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2418922463},{"type":"v1","addr":"172.21.15.27:6835","nonce":2418922463}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5246,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3919152683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3919152683},{"type":"v1","addr":"172.21.15.77:6836","nonce":3919152683}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5641,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/329572459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":329572459},{"type":"v1","addr":"172.21.15.77:6837","nonce":329572459}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:23:05.344753+0000","modified":"2024-01-11T08:23:08.864031+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/4173068012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":4173068012},{"type":"v1","addr":"172.21.15.27:6834","nonce":4173068012}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2024-01-11T08:23:11.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.800+0000 7f73513ff700 1 -- 172.21.15.27:0/440791296 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7338066f30 msgr2=0x7f73380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:11.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.800+0000 7f73513ff700 1 --2- 172.21.15.27:0/440791296 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7338066f30 0x7f73380693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f733c006fd0 tx=0x7f733c008040 comp rx=0 tx=0).stop 2024-01-11T08:23:11.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.800+0000 7f73513ff700 1 -- 172.21.15.27:0/440791296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f734c0b19d0 msgr2=0x7f734c0cbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:11.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.800+0000 7f73513ff700 1 --2- 172.21.15.27:0/440791296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f734c0b19d0 0x7f734c0cbd90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f734000c590 tx=0x7f734000c8a0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.800+0000 7f73513ff700 1 -- 172.21.15.27:0/440791296 shutdown_connections 2024-01-11T08:23:11.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.800+0000 7f73513ff700 1 --2- 172.21.15.27:0/440791296 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7338066f30 0x7f73380693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.800+0000 7f73513ff700 1 --2- 172.21.15.27:0/440791296 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f734c1575e0 0x7f734c0cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.800+0000 7f73513ff700 1 --2- 172.21.15.27:0/440791296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f734c06b610 0x7f734c146cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.800+0000 7f73513ff700 1 --2- 172.21.15.27:0/440791296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f734c0b19d0 0x7f734c0cbd90 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:11.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.800+0000 7f73513ff700 1 -- 172.21.15.27:0/440791296 >> 172.21.15.27:0/440791296 conn(0x7f734c0680e0 msgr2=0x7f734c06e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:11.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.800+0000 7f73513ff700 1 -- 172.21.15.27:0/440791296 shutdown_connections 2024-01-11T08:23:11.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:11.800+0000 7f73513ff700 1 -- 172.21.15.27:0/440791296 wait complete. 2024-01-11T08:23:11.800 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-01-11T08:23:12.220 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-11T08:23:05.344753+0000', 'modified': '2024-01-11T08:23:08.864031+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5697}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5697': {'gid': 5697, 'name': 'c', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/4173068012', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 4173068012}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 4173068012}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-11T08:23:12.220 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-11T08:23:12.220 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-11T08:23:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.425+0000 7f07e57ff700 1 Processor -- start 2024-01-11T08:23:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.425+0000 7f07e57ff700 1 -- start start 2024-01-11T08:23:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.426+0000 7f07e57ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07e006bf10 0x7f07e0159c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.426+0000 7f07e57ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f07e00b0a40 0x7f07e015da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.426+0000 7f07e57ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f07e015df40 0x7f07e0068720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.426+0000 7f07e57ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07e015d290 con 0x7f07e00b0a40 2024-01-11T08:23:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.426+0000 7f07e57ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07e015d570 con 0x7f07e015df40 2024-01-11T08:23:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.426+0000 7f07e57ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e0154180 con 0x7f07e006bf10 2024-01-11T08:23:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.427+0000 7f07de7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f07e00b0a40 0x7f07e015da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.427+0000 7f07de7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f07e00b0a40 0x7f07e015da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33734/0 (socket says 172.21.15.27:33734) 2024-01-11T08:23:12.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.427+0000 7f07de7fc700 1 -- 172.21.15.27:0/722552030 learned_addr learned my addr 172.21.15.27:0/722552030 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:12.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.427+0000 7f07deffd700 1 --2- 172.21.15.27:0/722552030 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07e006bf10 0x7f07e0159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.427+0000 7f07df7fe700 1 --2- 172.21.15.27:0/722552030 >> v2:172.21.15.77:3300/0 conn(0x7f07e015df40 0x7f07e0068720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.427+0000 7f07de7fc700 1 -- 172.21.15.27:0/722552030 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07e006bf10 msgr2=0x7f07e0159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:12.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.427+0000 7f07de7fc700 1 --2- 172.21.15.27:0/722552030 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07e006bf10 0x7f07e0159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.427+0000 7f07de7fc700 1 -- 172.21.15.27:0/722552030 >> v2:172.21.15.77:3300/0 conn(0x7f07e015df40 msgr2=0x7f07e0068720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:12.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.427+0000 7f07de7fc700 1 --2- 172.21.15.27:0/722552030 >> v2:172.21.15.77:3300/0 conn(0x7f07e015df40 0x7f07e0068720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.427+0000 7f07de7fc700 1 -- 172.21.15.27:0/722552030 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e015a510 con 0x7f07e00b0a40 2024-01-11T08:23:12.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.427+0000 7f07de7fc700 1 --2- 172.21.15.27:0/722552030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07e00b0a40 0x7f07e015da00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f07e0157860 tx=0x7f07d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4f1750e5cb60f14 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:12.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.428+0000 7f07ddffb700 1 -- 172.21.15.27:0/722552030 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d400b640 con 0x7f07e00b0a40 2024-01-11T08:23:12.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.428+0000 7f07ddffb700 1 -- 172.21.15.27:0/722552030 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d400b7a0 con 0x7f07e00b0a40 2024-01-11T08:23:12.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.428+0000 7f07ddffb700 1 -- 172.21.15.27:0/722552030 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d400bb40 con 0x7f07e00b0a40 2024-01-11T08:23:12.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.428+0000 7f07e57ff700 1 -- 172.21.15.27:0/722552030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07e00b0a40 msgr2=0x7f07e015da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:12.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.428+0000 7f07e57ff700 1 --2- 172.21.15.27:0/722552030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07e00b0a40 0x7f07e015da00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f07e0157860 tx=0x7f07d4009250 comp rx=0 tx=0).stop 2024-01-11T08:23:12.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.428+0000 7f07e57ff700 1 -- 172.21.15.27:0/722552030 shutdown_connections 2024-01-11T08:23:12.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.428+0000 7f07e57ff700 1 --2- 172.21.15.27:0/722552030 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07e006bf10 0x7f07e0159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.428+0000 7f07e57ff700 1 --2- 172.21.15.27:0/722552030 >> v2:172.21.15.77:3300/0 conn(0x7f07e015df40 0x7f07e0068720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.428+0000 7f07e57ff700 1 --2- 172.21.15.27:0/722552030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07e00b0a40 0x7f07e015da00 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.428+0000 7f07e57ff700 1 -- 172.21.15.27:0/722552030 >> 172.21.15.27:0/722552030 conn(0x7f07e00ce3c0 msgr2=0x7f07e00ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:12.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.428+0000 7f07e57ff700 1 -- 172.21.15.27:0/722552030 shutdown_connections 2024-01-11T08:23:12.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.428+0000 7f07e57ff700 1 -- 172.21.15.27:0/722552030 wait complete. 2024-01-11T08:23:12.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.431+0000 7f07e57ff700 1 Processor -- start 2024-01-11T08:23:12.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.431+0000 7f07e57ff700 1 -- start start 2024-01-11T08:23:12.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.432+0000 7f07e57ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07e006bf10 0x7f07e0148900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.432+0000 7f07e57ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07e00b0a40 0x7f07e0068720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.432+0000 7f07e57ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07e015df40 0x7f07e0068c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.432+0000 7f07e57ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e0148e40 con 0x7f07e00b0a40 2024-01-11T08:23:12.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.432+0000 7f07e57ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e00c8090 con 0x7f07e015df40 2024-01-11T08:23:12.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.432+0000 7f07e57ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e00c77f0 con 0x7f07e006bf10 2024-01-11T08:23:12.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.432+0000 7f07de7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07e00b0a40 0x7f07e0068720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.432+0000 7f07de7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07e00b0a40 0x7f07e0068720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33750/0 (socket says 172.21.15.27:33750) 2024-01-11T08:23:12.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.432+0000 7f07de7fc700 1 -- 172.21.15.27:0/1926708124 learned_addr learned my addr 172.21.15.27:0/1926708124 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:12.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.433+0000 7f07deffd700 1 --2- 172.21.15.27:0/1926708124 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07e006bf10 0x7f07e0148900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.433+0000 7f07df7fe700 1 --2- 172.21.15.27:0/1926708124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07e015df40 0x7f07e0068c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.433+0000 7f07de7fc700 1 -- 172.21.15.27:0/1926708124 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07e006bf10 msgr2=0x7f07e0148900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:12.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.433+0000 7f07de7fc700 1 --2- 172.21.15.27:0/1926708124 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07e006bf10 0x7f07e0148900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.433+0000 7f07de7fc700 1 -- 172.21.15.27:0/1926708124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07e015df40 msgr2=0x7f07e0068c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:12.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.433+0000 7f07de7fc700 1 --2- 172.21.15.27:0/1926708124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07e015df40 0x7f07e0068c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.433+0000 7f07de7fc700 1 -- 172.21.15.27:0/1926708124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d40094f0 con 0x7f07e00b0a40 2024-01-11T08:23:12.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.433+0000 7f07de7fc700 1 --2- 172.21.15.27:0/1926708124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07e00b0a40 0x7f07e0068720 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f07d4006fd0 tx=0x7f07d4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:12.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.433+0000 7f07c6ffd700 1 -- 172.21.15.27:0/1926708124 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d400b910 con 0x7f07e00b0a40 2024-01-11T08:23:12.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.433+0000 7f07c6ffd700 1 -- 172.21.15.27:0/1926708124 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d400ba90 con 0x7f07e00b0a40 2024-01-11T08:23:12.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.433+0000 7f07c6ffd700 1 -- 172.21.15.27:0/1926708124 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d4011670 con 0x7f07e00b0a40 2024-01-11T08:23:12.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.434+0000 7f07e57ff700 1 -- 172.21.15.27:0/1926708124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e00c7ad0 con 0x7f07e00b0a40 2024-01-11T08:23:12.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.434+0000 7f07e57ff700 1 -- 172.21.15.27:0/1926708124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e016af50 con 0x7f07e00b0a40 2024-01-11T08:23:12.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.436+0000 7f07c6ffd700 1 -- 172.21.15.27:0/1926708124 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f07d4011a60 con 0x7f07e00b0a40 2024-01-11T08:23:12.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.437+0000 7f07c6ffd700 1 --2- 172.21.15.27:0/1926708124 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f07cc066fb0 0x7f07cc069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.437+0000 7f07c6ffd700 1 -- 172.21.15.27:0/1926708124 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f07d4085b80 con 0x7f07e00b0a40 2024-01-11T08:23:12.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.437+0000 7f07deffd700 1 --2- 172.21.15.27:0/1926708124 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f07cc066fb0 0x7f07cc069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.438+0000 7f07deffd700 1 --2- 172.21.15.27:0/1926708124 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f07cc066fb0 0x7f07cc069460 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f07e00b3170 tx=0x7f07d00091a0 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:12.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.439+0000 7f07f3f1eb80 1 Processor -- start 2024-01-11T08:23:12.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.439+0000 7f07f3f1eb80 1 -- start start 2024-01-11T08:23:12.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.440+0000 7f07f3f1eb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561697655b20 0x561697636df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.440+0000 7f07f3f1eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616976313a0 0x56169765d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.440+0000 7f07f3f1eb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x561697655ef0 0x56169765dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.440+0000 7f07f3f1eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x56169754e310 con 0x5616976313a0 2024-01-11T08:23:12.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.440+0000 7f07f3f1eb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56169758b130 con 0x561697655b20 2024-01-11T08:23:12.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.440+0000 7f07f3f1eb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x561697588af0 con 0x561697655ef0 2024-01-11T08:23:12.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.440+0000 7f07de7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616976313a0 0x56169765d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.440+0000 7f07de7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616976313a0 0x56169765d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33756/0 (socket says 172.21.15.27:33756) 2024-01-11T08:23:12.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.440+0000 7f07de7fc700 1 -- 172.21.15.27:0/4169766524 learned_addr learned my addr 172.21.15.27:0/4169766524 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:12.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.440+0000 7f07df7fe700 1 --2- 172.21.15.27:0/4169766524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x561697655ef0 0x56169765dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.440+0000 7f07deffd700 1 --2- 172.21.15.27:0/4169766524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561697655b20 0x561697636df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.441+0000 7f07de7fc700 1 -- 172.21.15.27:0/4169766524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x561697655ef0 msgr2=0x56169765dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:12.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.441+0000 7f07de7fc700 1 --2- 172.21.15.27:0/4169766524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x561697655ef0 0x56169765dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.441+0000 7f07de7fc700 1 -- 172.21.15.27:0/4169766524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561697655b20 msgr2=0x561697636df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:12.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.441+0000 7f07de7fc700 1 --2- 172.21.15.27:0/4169766524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561697655b20 0x561697636df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.441+0000 7f07de7fc700 1 -- 172.21.15.27:0/4169766524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616974770a0 con 0x5616976313a0 2024-01-11T08:23:12.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.441+0000 7f07de7fc700 1 --2- 172.21.15.27:0/4169766524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616976313a0 0x56169765d860 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f07d4006e20 tx=0x7f07d4083f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e89102f56e959fb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:12.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.441+0000 7f07bffff700 1 -- 172.21.15.27:0/4169766524 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d4056370 con 0x5616976313a0 2024-01-11T08:23:12.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.441+0000 7f07bffff700 1 -- 172.21.15.27:0/4169766524 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d4057110 con 0x5616976313a0 2024-01-11T08:23:12.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.441+0000 7f07bffff700 1 -- 172.21.15.27:0/4169766524 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d4056a10 con 0x5616976313a0 2024-01-11T08:23:12.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.442+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/4169766524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616976313a0 msgr2=0x56169765d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:12.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.442+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/4169766524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616976313a0 0x56169765d860 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f07d4006e20 tx=0x7f07d4083f60 comp rx=0 tx=0).stop 2024-01-11T08:23:12.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.442+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/4169766524 shutdown_connections 2024-01-11T08:23:12.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.442+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/4169766524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x561697655ef0 0x56169765dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.442+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/4169766524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561697655b20 0x561697636df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.442+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/4169766524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5616976313a0 0x56169765d860 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.442+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/4169766524 >> 172.21.15.27:0/4169766524 conn(0x561697633300 msgr2=0x561697635bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:12.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.442+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/4169766524 shutdown_connections 2024-01-11T08:23:12.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.442+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/4169766524 wait complete. 2024-01-11T08:23:12.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.443+0000 7f07f3f1eb80 1 -- start start 2024-01-11T08:23:12.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.443+0000 7f07f3f1eb80 1 Processor -- start 2024-01-11T08:23:12.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.444+0000 7f07f3f1eb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5616976313a0 0x56169765c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.444+0000 7f07f3f1eb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x561697655b20 0x56169768a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.444+0000 7f07f3f1eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655ef0 0x5616976363d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.444+0000 7f07f3f1eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616975848e0 con 0x561697655ef0 2024-01-11T08:23:12.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.444+0000 7f07f3f1eb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561697530980 con 0x5616976313a0 2024-01-11T08:23:12.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.444+0000 7f07f3f1eb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x561697530620 con 0x561697655b20 2024-01-11T08:23:12.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.444+0000 7f07f3f1eb80 10 client.0 _ll_register_callbacks cb 0x561697393a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T08:23:12.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.444+0000 7f07df7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655ef0 0x5616976363d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.444+0000 7f07df7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655ef0 0x5616976363d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33764/0 (socket says 172.21.15.27:33764) 2024-01-11T08:23:12.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.444+0000 7f07df7fe700 1 -- 172.21.15.27:0/4114262884 learned_addr learned my addr 172.21.15.27:0/4114262884 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:12.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.444+0000 7f07de7fc700 1 --2- 172.21.15.27:0/4114262884 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x561697655b20 0x56169768a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.444+0000 7f07deffd700 1 --2- 172.21.15.27:0/4114262884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5616976313a0 0x56169765c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.445+0000 7f07df7fe700 1 -- 172.21.15.27:0/4114262884 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x561697655b20 msgr2=0x56169768a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:12.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.445+0000 7f07df7fe700 1 --2- 172.21.15.27:0/4114262884 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x561697655b20 0x56169768a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.445+0000 7f07df7fe700 1 -- 172.21.15.27:0/4114262884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5616976313a0 msgr2=0x56169765c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:12.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.445+0000 7f07df7fe700 1 --2- 172.21.15.27:0/4114262884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5616976313a0 0x56169765c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.445+0000 7f07df7fe700 1 -- 172.21.15.27:0/4114262884 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e016af50 con 0x561697655ef0 2024-01-11T08:23:12.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.445+0000 7f07deffd700 1 --2- 172.21.15.27:0/4114262884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5616976313a0 0x56169765c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:23:12.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.445+0000 7f07df7fe700 1 --2- 172.21.15.27:0/4114262884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655ef0 0x5616976363d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f07d8007e50 tx=0x7f07d800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9895c5d23f884e68 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:12.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.445+0000 7f07bd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T08:23:12.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.445+0000 7f07bd7fa700 1 -- 172.21.15.27:0/4114262884 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d800e050 con 0x561697655ef0 2024-01-11T08:23:12.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.445+0000 7f07bd7fa700 1 -- 172.21.15.27:0/4114262884 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(59..59 src has 1..59) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f07d8013ce0 con 0x561697655ef0 2024-01-11T08:23:12.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.445+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/4114262884 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561697629560 con 0x561697655ef0 2024-01-11T08:23:12.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.445+0000 7f07bd7fa700 1 -- 172.21.15.27:0/4114262884 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f07d8013ea0 con 0x561697655ef0 2024-01-11T08:23:12.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.446+0000 7f07f3f1eb80 10 client.5838 fetch_fsmap learned FSMap version 25 2024-01-11T08:23:12.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.446+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/4114262884 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x5616975533c0 con 0x561697655ef0 2024-01-11T08:23:12.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.446+0000 7f07bd7fa700 1 -- 172.21.15.27:0/4114262884 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561697629560 con 0x561697655ef0 2024-01-11T08:23:12.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.446+0000 7f07bd7fa700 1 -- 172.21.15.27:0/4114262884 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f07d8019690 con 0x561697655ef0 2024-01-11T08:23:12.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.446+0000 7f07f3f1eb80 10 client.5838 fetch_fsmap finished waiting for FSMap version 25 2024-01-11T08:23:12.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.446+0000 7f07f3f1eb80 10 client.5838 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-11T08:23:12.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.446+0000 7f07f3f1eb80 20 client.5838 populate_metadata read hostname 'smithi027' 2024-01-11T08:23:12.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.446+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/4114262884 >> [v2:172.21.15.27:6832/4173068012,v1:172.21.15.27:6834/4173068012] conn(0x56169767f380 0x56169767f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.446+0000 7f07f3f1eb80 4 client.5838 mds_command: new command op to 5697 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-11T08:23:12.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.446+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/4114262884 --> [v2:172.21.15.27:6832/4173068012,v1:172.21.15.27:6834/4173068012] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5616976060e0 con 0x56169767f380 2024-01-11T08:23:12.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.446+0000 7f07deffd700 1 --2- 172.21.15.27:0/4114262884 >> [v2:172.21.15.27:6832/4173068012,v1:172.21.15.27:6834/4173068012] conn(0x56169767f380 0x56169767f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.447+0000 7f07deffd700 1 --2- 172.21.15.27:0/4114262884 >> [v2:172.21.15.27:6832/4173068012,v1:172.21.15.27:6834/4173068012] conn(0x56169767f380 0x56169767f750 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26f39b6deaf4e30d server_cookie=f7e6caeac29077c in_seq=0 out_seq=0 2024-01-11T08:23:12.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.447+0000 7f07bd7fa700 10 client.5838 ms_handle_connect on v2:172.21.15.27:6832/4173068012 2024-01-11T08:23:12.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.447+0000 7f07bd7fa700 1 -- 172.21.15.27:0/4114262884 <== mds.0 v2:172.21.15.27:6832/4173068012 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x5616976060e0 con 0x56169767f380 2024-01-11T08:23:12.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.447+0000 7f07bd7fa700 10 client.5838 handle_command_reply: tid=0 2024-01-11T08:23:12.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.447+0000 7f07f3f1eb80 1 client.5838 shutdown 2024-01-11T08:23:12.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.448+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/4114262884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655ef0 msgr2=0x5616976363d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:12.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.448+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/4114262884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655ef0 0x5616976363d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f07d8007e50 tx=0x7f07d800b040 comp rx=0 tx=0).stop 2024-01-11T08:23:12.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.448+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/4114262884 shutdown_connections 2024-01-11T08:23:12.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.448+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/4114262884 >> [v2:172.21.15.27:6832/4173068012,v1:172.21.15.27:6834/4173068012] conn(0x56169767f380 0x56169767f750 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.448+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/4114262884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655ef0 0x5616976363d0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.448+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/4114262884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5616976313a0 0x56169765c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.448+0000 7f07bd7fa700 0 client.5838 ms_handle_reset on v2:172.21.15.27:6832/4173068012 2024-01-11T08:23:12.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.448+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/4114262884 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x561697655b20 0x56169768a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.448+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/4114262884 >> 172.21.15.27:0/4114262884 conn(0x561697633300 msgr2=0x561697637be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:12.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.448+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/4114262884 shutdown_connections 2024-01-11T08:23:12.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.448+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/4114262884 wait complete. 2024-01-11T08:23:12.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.448+0000 7f07f3f1eb80 20 client.5838 trim_cache size 0 max 16384 2024-01-11T08:23:12.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.466+0000 7f07f3f1eb80 1 Processor -- start 2024-01-11T08:23:12.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.466+0000 7f07f3f1eb80 1 -- start start 2024-01-11T08:23:12.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.467+0000 7f07f3f1eb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5616976313a0 0x561697663ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.467+0000 7f07f3f1eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655b20 0x5616976620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.467+0000 7f07f3f1eb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561697655ef0 0x5616976306f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.467+0000 7f07f3f1eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x56169741c1e0 con 0x561697655b20 2024-01-11T08:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.467+0000 7f07f3f1eb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561697657410 con 0x561697655ef0 2024-01-11T08:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.467+0000 7f07f3f1eb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x561697632190 con 0x5616976313a0 2024-01-11T08:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.467+0000 7f07de7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655b20 0x5616976620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.467+0000 7f07de7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655b20 0x5616976620a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33776/0 (socket says 172.21.15.27:33776) 2024-01-11T08:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.467+0000 7f07de7fc700 1 -- 172.21.15.27:0/368998340 learned_addr learned my addr 172.21.15.27:0/368998340 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.467+0000 7f07df7fe700 1 --2- 172.21.15.27:0/368998340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561697655ef0 0x5616976306f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.467+0000 7f07deffd700 1 --2- 172.21.15.27:0/368998340 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5616976313a0 0x561697663ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.467+0000 7f07de7fc700 1 -- 172.21.15.27:0/368998340 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5616976313a0 msgr2=0x561697663ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.467+0000 7f07de7fc700 1 --2- 172.21.15.27:0/368998340 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5616976313a0 0x561697663ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.467+0000 7f07de7fc700 1 -- 172.21.15.27:0/368998340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561697655ef0 msgr2=0x5616976306f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.467+0000 7f07de7fc700 1 --2- 172.21.15.27:0/368998340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561697655ef0 0x5616976306f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.467+0000 7f07de7fc700 1 -- 172.21.15.27:0/368998340 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616973eb1d0 con 0x561697655b20 2024-01-11T08:23:12.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.468+0000 7f07de7fc700 1 --2- 172.21.15.27:0/368998340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655b20 0x5616976620a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f07d4014040 tx=0x7f07d406f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10b87f1f3e892b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:12.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.468+0000 7f07c4ff9700 1 -- 172.21.15.27:0/368998340 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d4056370 con 0x561697655b20 2024-01-11T08:23:12.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.468+0000 7f07c4ff9700 1 -- 172.21.15.27:0/368998340 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d4059c60 con 0x561697655b20 2024-01-11T08:23:12.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.468+0000 7f07c4ff9700 1 -- 172.21.15.27:0/368998340 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d405d9b0 con 0x561697655b20 2024-01-11T08:23:12.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.468+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/368998340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655b20 msgr2=0x5616976620a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:12.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.468+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/368998340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655b20 0x5616976620a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f07d4014040 tx=0x7f07d406f000 comp rx=0 tx=0).stop 2024-01-11T08:23:12.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.468+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/368998340 shutdown_connections 2024-01-11T08:23:12.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.468+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/368998340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561697655ef0 0x5616976306f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.468+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/368998340 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5616976313a0 0x561697663ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.468+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/368998340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655b20 0x5616976620a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.468+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/368998340 >> 172.21.15.27:0/368998340 conn(0x561697633300 msgr2=0x561697637b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:12.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.468+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/368998340 shutdown_connections 2024-01-11T08:23:12.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.469+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/368998340 wait complete. 2024-01-11T08:23:12.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.469+0000 7f07f3f1eb80 1 -- start start 2024-01-11T08:23:12.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.470+0000 7f07f3f1eb80 1 Processor -- start 2024-01-11T08:23:12.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.471+0000 7f07f3f1eb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5616976313a0 0x5616976c2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.471+0000 7f07f3f1eb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x561697655b20 0x5616976543b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.471+0000 7f07f3f1eb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655ef0 0x5616976306f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.471+0000 7f07f3f1eb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616976943b0 con 0x561697655ef0 2024-01-11T08:23:12.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.471+0000 7f07f3f1eb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x561697654de0 con 0x5616976313a0 2024-01-11T08:23:12.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.471+0000 7f07f3f1eb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x561697655030 con 0x561697655b20 2024-01-11T08:23:12.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.471+0000 7f07f3f1eb80 10 client.0 _ll_register_callbacks cb 0x561697393a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T08:23:12.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.471+0000 7f07df7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655ef0 0x5616976306f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.472+0000 7f07df7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655ef0 0x5616976306f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33792/0 (socket says 172.21.15.27:33792) 2024-01-11T08:23:12.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.472+0000 7f07df7fe700 1 -- 172.21.15.27:0/2161781398 learned_addr learned my addr 172.21.15.27:0/2161781398 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.472+0000 7f07df7fe700 1 -- 172.21.15.27:0/2161781398 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x561697655b20 msgr2=0x5616976543b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.472+0000 7f07de7fc700 1 --2- 172.21.15.27:0/2161781398 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x561697655b20 0x5616976543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.472+0000 7f07df7fe700 1 --2- 172.21.15.27:0/2161781398 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x561697655b20 0x5616976543b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.472+0000 7f07df7fe700 1 -- 172.21.15.27:0/2161781398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5616976313a0 msgr2=0x5616976c2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.472+0000 7f07deffd700 1 --2- 172.21.15.27:0/2161781398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5616976313a0 0x5616976c2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.472+0000 7f07df7fe700 1 --2- 172.21.15.27:0/2161781398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5616976313a0 0x5616976c2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.472+0000 7f07df7fe700 1 -- 172.21.15.27:0/2161781398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5616974770a0 con 0x561697655ef0 2024-01-11T08:23:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.472+0000 7f07deffd700 1 --2- 172.21.15.27:0/2161781398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5616976313a0 0x5616976c2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:23:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.472+0000 7f07df7fe700 1 --2- 172.21.15.27:0/2161781398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655ef0 0x5616976306f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f07d8009870 tx=0x7f07d801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6be7ba46c54fa2ce server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.472+0000 7f07bd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T08:23:12.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.472+0000 7f07bd7fa700 1 -- 172.21.15.27:0/2161781398 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d800e040 con 0x561697655ef0 2024-01-11T08:23:12.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.472+0000 7f07bd7fa700 1 -- 172.21.15.27:0/2161781398 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(59..59 src has 1..59) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f07d8007b00 con 0x561697655ef0 2024-01-11T08:23:12.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.472+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/2161781398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5616976557e0 con 0x561697655ef0 2024-01-11T08:23:12.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.473+0000 7f07bd7fa700 1 -- 172.21.15.27:0/2161781398 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f07d8017430 con 0x561697655ef0 2024-01-11T08:23:12.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.473+0000 7f07f3f1eb80 10 client.5844 fetch_fsmap learned FSMap version 25 2024-01-11T08:23:12.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.473+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/2161781398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x5616973c4d60 con 0x561697655ef0 2024-01-11T08:23:12.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.473+0000 7f07bd7fa700 1 -- 172.21.15.27:0/2161781398 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5616976557e0 con 0x561697655ef0 2024-01-11T08:23:12.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.473+0000 7f07bd7fa700 1 -- 172.21.15.27:0/2161781398 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 25) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f07d8017590 con 0x561697655ef0 2024-01-11T08:23:12.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.473+0000 7f07f3f1eb80 10 client.5844 fetch_fsmap finished waiting for FSMap version 25 2024-01-11T08:23:12.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.473+0000 7f07f3f1eb80 10 client.5844 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-11T08:23:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.473+0000 7f07f3f1eb80 20 client.5844 populate_metadata read hostname 'smithi027' 2024-01-11T08:23:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.473+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/2161781398 >> [v2:172.21.15.27:6832/4173068012,v1:172.21.15.27:6834/4173068012] conn(0x561697646b20 0x561697646ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.473+0000 7f07f3f1eb80 4 client.5844 mds_command: new command op to 5697 tid=0[{"prefix": "status"}] 2024-01-11T08:23:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.473+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/2161781398 --> [v2:172.21.15.27:6832/4173068012,v1:172.21.15.27:6834/4173068012] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f07d4057110 con 0x561697646b20 2024-01-11T08:23:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.473+0000 7f07deffd700 1 --2- 172.21.15.27:0/2161781398 >> [v2:172.21.15.27:6832/4173068012,v1:172.21.15.27:6834/4173068012] conn(0x561697646b20 0x561697646ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.474+0000 7f07deffd700 1 --2- 172.21.15.27:0/2161781398 >> [v2:172.21.15.27:6832/4173068012,v1:172.21.15.27:6834/4173068012] conn(0x561697646b20 0x561697646ef0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab4ee8bae5ea375 server_cookie=2ffdc184b9617253 in_seq=0 out_seq=0 2024-01-11T08:23:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.474+0000 7f07bd7fa700 10 client.5844 ms_handle_connect on v2:172.21.15.27:6832/4173068012 2024-01-11T08:23:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.474+0000 7f07bd7fa700 1 -- 172.21.15.27:0/2161781398 <== mds.0 v2:172.21.15.27:6832/4173068012 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+323 (crc 0 0 0) 0x7f07d4057110 con 0x561697646b20 2024-01-11T08:23:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.474+0000 7f07bd7fa700 10 client.5844 handle_command_reply: tid=0 2024-01-11T08:23:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.474+0000 7f07f3f1eb80 1 client.5844 shutdown 2024-01-11T08:23:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.474+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/2161781398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655ef0 msgr2=0x5616976306f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.474+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/2161781398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655ef0 0x5616976306f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f07d8009870 tx=0x7f07d801f000 comp rx=0 tx=0).stop 2024-01-11T08:23:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.474+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/2161781398 shutdown_connections 2024-01-11T08:23:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.474+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/2161781398 >> [v2:172.21.15.27:6832/4173068012,v1:172.21.15.27:6834/4173068012] conn(0x561697646b20 0x561697646ef0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.474+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/2161781398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561697655ef0 0x5616976306f0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.474+0000 7f07bd7fa700 0 client.5844 ms_handle_reset on v2:172.21.15.27:6832/4173068012 2024-01-11T08:23:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.474+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/2161781398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5616976313a0 0x5616976c2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.474+0000 7f07f3f1eb80 1 --2- 172.21.15.27:0/2161781398 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x561697655b20 0x5616976543b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.474+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/2161781398 >> 172.21.15.27:0/2161781398 conn(0x561697633300 msgr2=0x5616976325e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.474+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/2161781398 shutdown_connections 2024-01-11T08:23:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.474+0000 7f07f3f1eb80 1 -- 172.21.15.27:0/2161781398 wait complete. 2024-01-11T08:23:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.475+0000 7f07f3f1eb80 20 client.5844 trim_cache size 0 max 16384 2024-01-11T08:23:12.482 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T08:23:12.482 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "8fddb4a7-9d66-40f1-892d-6fe86c728ee4", 2024-01-11T08:23:12.482 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-11T08:23:12.482 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5697, 2024-01-11T08:23:12.482 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-11T08:23:12.482 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-11T08:23:12.482 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-11T08:23:12.482 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 6.6368801319999999, 2024-01-11T08:23:12.482 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 25, 2024-01-11T08:23:12.482 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 59, 2024-01-11T08:23:12.482 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 59, 2024-01-11T08:23:12.482 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 13.3348146 2024-01-11T08:23:12.482 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T08:23:12.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.475+0000 7f07e57ff700 1 -- 172.21.15.27:0/1926708124 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f07cc066fb0 msgr2=0x7f07cc069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:12.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.475+0000 7f07e57ff700 1 --2- 172.21.15.27:0/1926708124 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f07cc066fb0 0x7f07cc069460 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f07e00b3170 tx=0x7f07d00091a0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.475+0000 7f07e57ff700 1 -- 172.21.15.27:0/1926708124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07e00b0a40 msgr2=0x7f07e0068720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.475+0000 7f07e57ff700 1 --2- 172.21.15.27:0/1926708124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07e00b0a40 0x7f07e0068720 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f07d4006fd0 tx=0x7f07d4017000 comp rx=0 tx=0).stop 2024-01-11T08:23:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.475+0000 7f07e57ff700 1 -- 172.21.15.27:0/1926708124 shutdown_connections 2024-01-11T08:23:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.475+0000 7f07e57ff700 1 --2- 172.21.15.27:0/1926708124 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f07cc066fb0 0x7f07cc069460 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.475+0000 7f07e57ff700 1 --2- 172.21.15.27:0/1926708124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07e015df40 0x7f07e0068c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.475+0000 7f07e57ff700 1 --2- 172.21.15.27:0/1926708124 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07e006bf10 0x7f07e0148900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.475+0000 7f07e57ff700 1 --2- 172.21.15.27:0/1926708124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07e00b0a40 0x7f07e0068720 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.475+0000 7f07e57ff700 1 -- 172.21.15.27:0/1926708124 >> 172.21.15.27:0/1926708124 conn(0x7f07e00ce3c0 msgr2=0x7f07e015cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.475+0000 7f07e57ff700 1 -- 172.21.15.27:0/1926708124 shutdown_connections 2024-01-11T08:23:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.475+0000 7f07e57ff700 1 -- 172.21.15.27:0/1926708124 wait complete. 2024-01-11T08:23:12.492 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-11T08:23:12.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.692+0000 7f78e01ff700 1 Processor -- start 2024-01-11T08:23:12.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.693+0000 7f78e01ff700 1 -- start start 2024-01-11T08:23:12.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.694+0000 7f78e01ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78d80b19d0 0x7f78d8158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.694+0000 7f78e01ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f78d806b610 0x7f78d8159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.694+0000 7f78e01ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d8158b60 con 0x7f78d81575e0 2024-01-11T08:23:12.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.694+0000 7f78e01ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d80cb5e0 con 0x7f78d806b610 2024-01-11T08:23:12.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.694+0000 7f78e01ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d81599b0 con 0x7f78d80b19d0 2024-01-11T08:23:12.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.694+0000 7f78de79c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f78d806b610 0x7f78d8159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.694+0000 7f78de79c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f78d806b610 0x7f78d8159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33800/0 (socket says 172.21.15.27:33800) 2024-01-11T08:23:12.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.694+0000 7f78de79c700 1 -- 172.21.15.27:0/754540713 learned_addr learned my addr 172.21.15.27:0/754540713 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:12.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.694+0000 7f78dd79a700 1 --2- 172.21.15.27:0/754540713 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78d80b19d0 0x7f78d8158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.695+0000 7f78de79c700 1 -- 172.21.15.27:0/754540713 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78d80b19d0 msgr2=0x7f78d8158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:12.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.695+0000 7f78de79c700 1 --2- 172.21.15.27:0/754540713 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78d80b19d0 0x7f78d8158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.695+0000 7f78de79c700 1 -- 172.21.15.27:0/754540713 >> v1:172.21.15.77:6789/0 conn(0x7f78d81575e0 legacy=0x7f78d80b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:12.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.695+0000 7f78de79c700 1 -- 172.21.15.27:0/754540713 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d80c0f90 con 0x7f78d806b610 2024-01-11T08:23:12.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.695+0000 7f78de79c700 1 --2- 172.21.15.27:0/754540713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d806b610 0x7f78d8159e40 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f78d0009f20 tx=0x7f78d0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f798506083f57a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:12.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.695+0000 7f78dcf99700 1 -- 172.21.15.27:0/754540713 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d000e070 con 0x7f78d806b610 2024-01-11T08:23:12.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.695+0000 7f78dcf99700 1 -- 172.21.15.27:0/754540713 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d000b9d0 con 0x7f78d806b610 2024-01-11T08:23:12.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.695+0000 7f78dcf99700 1 -- 172.21.15.27:0/754540713 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d00092e0 con 0x7f78d806b610 2024-01-11T08:23:12.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.696+0000 7f78e01ff700 1 -- 172.21.15.27:0/754540713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d806b610 msgr2=0x7f78d8159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:12.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.696+0000 7f78e01ff700 1 --2- 172.21.15.27:0/754540713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d806b610 0x7f78d8159e40 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f78d0009f20 tx=0x7f78d0009840 comp rx=0 tx=0).stop 2024-01-11T08:23:12.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.696+0000 7f78e01ff700 1 -- 172.21.15.27:0/754540713 shutdown_connections 2024-01-11T08:23:12.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.696+0000 7f78e01ff700 1 --2- 172.21.15.27:0/754540713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d806b610 0x7f78d8159e40 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.696+0000 7f78e01ff700 1 --2- 172.21.15.27:0/754540713 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78d80b19d0 0x7f78d8158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.696+0000 7f78e01ff700 1 -- 172.21.15.27:0/754540713 >> 172.21.15.27:0/754540713 conn(0x7f78d80680e0 msgr2=0x7f78d8069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:12.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.696+0000 7f78e01ff700 1 -- 172.21.15.27:0/754540713 shutdown_connections 2024-01-11T08:23:12.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.696+0000 7f78e01ff700 1 -- 172.21.15.27:0/754540713 wait complete. 2024-01-11T08:23:12.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.698+0000 7f78e01ff700 1 Processor -- start 2024-01-11T08:23:12.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.698+0000 7f78e01ff700 1 -- start start 2024-01-11T08:23:12.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.700+0000 7f78e01ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d806b610 0x7f78d815db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.700+0000 7f78e01ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78d80b19d0 0x7f78d80c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.700+0000 7f78e01ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78d81575e0 0x7f78d813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.700+0000 7f78e01ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d80afd60 con 0x7f78d806b610 2024-01-11T08:23:12.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.700+0000 7f78e01ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d80c5b00 con 0x7f78d81575e0 2024-01-11T08:23:12.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.700+0000 7f78e01ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d80af150 con 0x7f78d80b19d0 2024-01-11T08:23:12.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.700+0000 7f78ddf9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d806b610 0x7f78d815db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.700+0000 7f78ddf9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d806b610 0x7f78d815db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33804/0 (socket says 172.21.15.27:33804) 2024-01-11T08:23:12.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.700+0000 7f78ddf9b700 1 -- 172.21.15.27:0/3396710819 learned_addr learned my addr 172.21.15.27:0/3396710819 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:12.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.700+0000 7f78dd79a700 1 --2- 172.21.15.27:0/3396710819 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78d80b19d0 0x7f78d80c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.700+0000 7f78ddf9b700 1 -- 172.21.15.27:0/3396710819 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78d80b19d0 msgr2=0x7f78d80c55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:12.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.700+0000 7f78ddf9b700 1 --2- 172.21.15.27:0/3396710819 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78d80b19d0 0x7f78d80c55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.700+0000 7f78de79c700 1 --2- 172.21.15.27:0/3396710819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78d81575e0 0x7f78d813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.700+0000 7f78ddf9b700 1 -- 172.21.15.27:0/3396710819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78d81575e0 msgr2=0x7f78d813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:12.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.701+0000 7f78ddf9b700 1 --2- 172.21.15.27:0/3396710819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78d81575e0 0x7f78d813f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.701+0000 7f78ddf9b700 1 -- 172.21.15.27:0/3396710819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d0009ae0 con 0x7f78d806b610 2024-01-11T08:23:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.701+0000 7f78dd79a700 1 --2- 172.21.15.27:0/3396710819 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78d80b19d0 0x7f78d80c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:23:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.701+0000 7f78de79c700 1 --2- 172.21.15.27:0/3396710819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78d81575e0 0x7f78d813f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:23:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.701+0000 7f78ddf9b700 1 --2- 172.21.15.27:0/3396710819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d806b610 0x7f78d815db30 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f78d400c4e0 tx=0x7f78d400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.701+0000 7f78cdffb700 1 -- 172.21.15.27:0/3396710819 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d4011590 con 0x7f78d806b610 2024-01-11T08:23:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.701+0000 7f78cdffb700 1 -- 172.21.15.27:0/3396710819 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d4011710 con 0x7f78d806b610 2024-01-11T08:23:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.701+0000 7f78e01ff700 1 -- 172.21.15.27:0/3396710819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d80af430 con 0x7f78d806b610 2024-01-11T08:23:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.701+0000 7f78cdffb700 1 -- 172.21.15.27:0/3396710819 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d4011ad0 con 0x7f78d806b610 2024-01-11T08:23:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.701+0000 7f78e01ff700 1 -- 172.21.15.27:0/3396710819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d80af9f0 con 0x7f78d806b610 2024-01-11T08:23:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.703+0000 7f78e01ff700 1 -- 172.21.15.27:0/3396710819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d80718d0 con 0x7f78d806b610 2024-01-11T08:23:12.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.703+0000 7f78cdffb700 1 -- 172.21.15.27:0/3396710819 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f78d401b3b0 con 0x7f78d806b610 2024-01-11T08:23:12.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.704+0000 7f78cdffb700 1 --2- 172.21.15.27:0/3396710819 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f78bc066f30 0x7f78bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:12.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.704+0000 7f78cdffb700 1 -- 172.21.15.27:0/3396710819 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f78d4083860 con 0x7f78d806b610 2024-01-11T08:23:12.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.705+0000 7f78dd79a700 1 --2- 172.21.15.27:0/3396710819 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f78bc066f30 0x7f78bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:12.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.705+0000 7f78dd79a700 1 --2- 172.21.15.27:0/3396710819 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f78bc066f30 0x7f78bc0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f78c8006f00 tx=0x7f78c8008040 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:12.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.708+0000 7f78cdffb700 1 -- 172.21.15.27:0/3396710819 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f78d4050ad0 con 0x7f78d806b610 2024-01-11T08:23:12.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.960+0000 7f78e01ff700 1 -- 172.21.15.27:0/3396710819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f78d80afbb0 con 0x7f78d806b610 2024-01-11T08:23:12.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.960+0000 7f78cdffb700 1 -- 172.21.15.27:0/3396710819 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7f78d4017070 con 0x7f78d806b610 2024-01-11T08:23:12.959 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-11T08:23:12.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.962+0000 7f78e01ff700 1 -- 172.21.15.27:0/3396710819 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f78bc066f30 msgr2=0x7f78bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:12.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.962+0000 7f78e01ff700 1 --2- 172.21.15.27:0/3396710819 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f78bc066f30 0x7f78bc0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f78c8006f00 tx=0x7f78c8008040 comp rx=0 tx=0).stop 2024-01-11T08:23:12.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.962+0000 7f78e01ff700 1 -- 172.21.15.27:0/3396710819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d806b610 msgr2=0x7f78d815db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:12.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.962+0000 7f78e01ff700 1 --2- 172.21.15.27:0/3396710819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d806b610 0x7f78d815db30 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f78d400c4e0 tx=0x7f78d400c8a0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.962+0000 7f78e01ff700 1 -- 172.21.15.27:0/3396710819 shutdown_connections 2024-01-11T08:23:12.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.962+0000 7f78e01ff700 1 --2- 172.21.15.27:0/3396710819 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f78bc066f30 0x7f78bc0693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.962+0000 7f78e01ff700 1 --2- 172.21.15.27:0/3396710819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78d81575e0 0x7f78d813f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.962+0000 7f78e01ff700 1 --2- 172.21.15.27:0/3396710819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78d806b610 0x7f78d815db30 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.962+0000 7f78e01ff700 1 --2- 172.21.15.27:0/3396710819 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78d80b19d0 0x7f78d80c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:12.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.962+0000 7f78e01ff700 1 -- 172.21.15.27:0/3396710819 >> 172.21.15.27:0/3396710819 conn(0x7f78d80680e0 msgr2=0x7f78d8159d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:12.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.962+0000 7f78e01ff700 1 -- 172.21.15.27:0/3396710819 shutdown_connections 2024-01-11T08:23:12.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:12.962+0000 7f78e01ff700 1 -- 172.21.15.27:0/3396710819 wait complete. 2024-01-11T08:23:12.979 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-11T08:23:12.980 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-11T08:23:12.980 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-11T08:23:12.980 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-11T08:23:12.980 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-11T08:23:12.980 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-11T08:23:12.980 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-11T08:23:12.980 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-11T08:23:12.980 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:23:12.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:23:12.980 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-11T08:23:13.029 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T08:23:13.029 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T08:23:13.029 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T08:23:13.029 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:23:13.029 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-11T08:23:13.029 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:23:13.029 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:23:13.030 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:23:13.030 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-11T08:23:13.030 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:23:13.030 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:23:13.030 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-11T08:23:13.030 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-11T08:23:13.030 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-11T08:23:13.030 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-11T08:23:13.030 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-11T08:23:13.030 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 41729sec preferred_lft 41729sec 2024-01-11T08:23:13.030 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-11T08:23:13.030 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:23:13.030 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:23:13.030 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-11T08:23:13.030 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-11T08:23:13.031 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T08:23:13.031 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff 2024-01-11T08:23:13.031 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-11T08:23:13.031 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:23:13.031 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::acd2:b3ff:fe6a:db4b/64 scope link 2024-01-11T08:23:13.031 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:23:13.031 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T08:23:13.031 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:fa:85:1b:3a:bc brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-11T08:23:13.031 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ecfa:85ff:fe1b:3abc/64 scope link 2024-01-11T08:23:13.031 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:23:13.031 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T08:23:13.031 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-11T08:23:13.031 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ec88:faff:fe17:41aa/64 scope link 2024-01-11T08:23:13.031 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:23:13.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:23:13.032 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-11T08:23:13.102 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-11T08:23:13.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:23:13.105 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-11T08:23:13.157 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-11T08:23:13.157 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-11T08:23:13.157 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-11T08:23:13.157 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-11T08:23:13.158 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-11T08:23:13.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:23:13.158 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T08:23:13.158 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T08:23:13.158 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-11T08:23:13.158 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-11T08:23:13.158 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-11T08:23:13.158 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T08:23:13.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:23:13.334 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T08:23:13.383 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-11T08:23:13.383 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-11T08:23:13.384 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:23:13.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:23:13.385 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-11T08:23:13.458 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:23:13.458 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-11T08:23:13.505 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:23:13.506 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:23:13.506 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-11T08:23:13.552 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-11T08:23:13.552 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-11T08:23:13.553 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:23:13.655 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:23:13.655 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:23:13.655 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:23:13.670 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:23:13.670 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:23:13.671 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:23:13.671 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:23:13.671 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:23:13.694 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:23:13.695 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:23:13.695 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:23:13.695 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:23:13.695 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:23:13.695 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790" 2024-01-11T08:23:13.695 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790:/ 2024-01-11T08:23:13.695 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=8fddb4a7-9d66-40f1-892d-6fe86c728ee4" will pass to kernel 2024-01-11T08:23:13.699 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-11T08:23:13.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:23:13.700 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-11T08:23:13.773 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-11T08:23:13.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.977+0000 7f12a4fff700 1 Processor -- start 2024-01-11T08:23:13.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.977+0000 7f12a4fff700 1 -- start start 2024-01-11T08:23:13.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.978+0000 7f12a4fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f12a00b0a40 0x7f12a015da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:13.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.978+0000 7f12a4fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12a015df40 0x7f12a0068720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:13.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.978+0000 7f12a4fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a015d290 con 0x7f12a006bf10 2024-01-11T08:23:13.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.978+0000 7f12a4fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a015d570 con 0x7f12a00b0a40 2024-01-11T08:23:13.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.978+0000 7f12a4fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a0154180 con 0x7f12a015df40 2024-01-11T08:23:13.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.979+0000 7f129e59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f12a006bf10 0x7f12a0159c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58706/0 (socket says 172.21.15.27:58706) 2024-01-11T08:23:13.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.979+0000 7f129e59c700 1 -- 172.21.15.27:0/3462587021 learned_addr learned my addr 172.21.15.27:0/3462587021 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:23:13.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.979+0000 7f129ed9d700 1 --2- 172.21.15.27:0/3462587021 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12a015df40 0x7f12a0068720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:13.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.979+0000 7f129d59a700 1 -- 172.21.15.27:0/3462587021 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551760228 0 0) 0x7f12a015d290 con 0x7f12a006bf10 2024-01-11T08:23:13.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.979+0000 7f129dd9b700 1 --2- 172.21.15.27:0/3462587021 >> v2:172.21.15.77:3300/0 conn(0x7f12a00b0a40 0x7f12a015da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:13.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.979+0000 7f129d59a700 1 -- 172.21.15.27:0/3462587021 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1288003570 con 0x7f12a006bf10 2024-01-11T08:23:13.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.979+0000 7f129d59a700 1 -- 172.21.15.27:0/3462587021 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1893080529 0 0) 0x7f1288003570 con 0x7f12a006bf10 2024-01-11T08:23:13.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.979+0000 7f129d59a700 1 -- 172.21.15.27:0/3462587021 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12a015df40 msgr2=0x7f12a0068720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:13.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.979+0000 7f129d59a700 1 --2- 172.21.15.27:0/3462587021 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12a015df40 0x7f12a0068720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:13.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.979+0000 7f129d59a700 1 -- 172.21.15.27:0/3462587021 >> v2:172.21.15.77:3300/0 conn(0x7f12a00b0a40 msgr2=0x7f12a015da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:13.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.979+0000 7f129d59a700 1 --2- 172.21.15.27:0/3462587021 >> v2:172.21.15.77:3300/0 conn(0x7f12a00b0a40 0x7f12a015da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:13.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.979+0000 7f129d59a700 1 -- 172.21.15.27:0/3462587021 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a015a510 con 0x7f12a006bf10 2024-01-11T08:23:13.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.979+0000 7f129d59a700 1 -- 172.21.15.27:0/3462587021 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f1290001470 con 0x7f12a006bf10 2024-01-11T08:23:13.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.979+0000 7f129d59a700 1 -- 172.21.15.27:0/3462587021 >> v1:172.21.15.27:6789/0 conn(0x7f12a006bf10 legacy=0x7f12a0159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:13.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.979+0000 7f129d59a700 1 --2- 172.21.15.27:0/3462587021 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12a015df40 0x7f1288003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:13.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.979+0000 7f129d59a700 1 --2- 172.21.15.27:0/3462587021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12880040b0 0x7f1288004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:13.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.980+0000 7f129d59a700 1 --2- 172.21.15.27:0/3462587021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1288004d50 0x7f12880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:13.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.980+0000 7f129d59a700 1 -- 172.21.15.27:0/3462587021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a015d570 con 0x7f12880040b0 2024-01-11T08:23:13.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.980+0000 7f129d59a700 1 -- 172.21.15.27:0/3462587021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a0154180 con 0x7f1288004d50 2024-01-11T08:23:13.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.980+0000 7f129d59a700 1 -- 172.21.15.27:0/3462587021 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f128800bbf0 con 0x7f12a015df40 2024-01-11T08:23:13.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.980+0000 7f129dd9b700 1 --2- 172.21.15.27:0/3462587021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12880040b0 0x7f1288004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:13.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.980+0000 7f129ed9d700 1 --2- 172.21.15.27:0/3462587021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1288004d50 0x7f12880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:13.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.980+0000 7f129e59c700 1 --2- 172.21.15.27:0/3462587021 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12a015df40 0x7f1288003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:13.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.980+0000 7f129dd9b700 1 -- 172.21.15.27:0/3462587021 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12a015df40 msgr2=0x7f1288003ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:13.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.980+0000 7f129dd9b700 1 --2- 172.21.15.27:0/3462587021 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12a015df40 0x7f1288003ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:13.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.980+0000 7f129dd9b700 1 -- 172.21.15.27:0/3462587021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1288004d50 msgr2=0x7f12880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:13.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.980+0000 7f129dd9b700 1 --2- 172.21.15.27:0/3462587021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1288004d50 0x7f12880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:13.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.980+0000 7f129dd9b700 1 -- 172.21.15.27:0/3462587021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1288003570 con 0x7f12880040b0 2024-01-11T08:23:13.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.980+0000 7f129ed9d700 1 --2- 172.21.15.27:0/3462587021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1288004d50 0x7f12880051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:23:13.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.981+0000 7f129dd9b700 1 --2- 172.21.15.27:0/3462587021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12880040b0 0x7f1288004810 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f1294009c00 tx=0x7f12940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85af7405ed18f068 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:13.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.981+0000 7f129d59a700 1 -- 172.21.15.27:0/3462587021 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129400e710 con 0x7f12880040b0 2024-01-11T08:23:13.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.981+0000 7f129d59a700 1 -- 172.21.15.27:0/3462587021 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129400e870 con 0x7f12880040b0 2024-01-11T08:23:13.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.981+0000 7f129d59a700 1 -- 172.21.15.27:0/3462587021 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129400ec30 con 0x7f12880040b0 2024-01-11T08:23:13.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.981+0000 7f12a4fff700 1 -- 172.21.15.27:0/3462587021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12880040b0 msgr2=0x7f1288004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:13.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.981+0000 7f12a4fff700 1 --2- 172.21.15.27:0/3462587021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12880040b0 0x7f1288004810 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f1294009c00 tx=0x7f12940091a0 comp rx=0 tx=0).stop 2024-01-11T08:23:13.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.981+0000 7f129ed9d700 1 -- 172.21.15.27:0/3462587021 reap_dead start 2024-01-11T08:23:13.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.981+0000 7f12a4fff700 1 -- 172.21.15.27:0/3462587021 shutdown_connections 2024-01-11T08:23:13.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.981+0000 7f12a4fff700 1 -- 172.21.15.27:0/3462587021 >> 172.21.15.27:0/3462587021 conn(0x7f12a00ce3c0 msgr2=0x7f12a00ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:13.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.982+0000 7f12a4fff700 1 -- 172.21.15.27:0/3462587021 shutdown_connections 2024-01-11T08:23:13.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.982+0000 7f12a4fff700 1 -- 172.21.15.27:0/3462587021 wait complete. 2024-01-11T08:23:13.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.983+0000 7f12a4fff700 1 Processor -- start 2024-01-11T08:23:13.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.983+0000 7f12a4fff700 1 -- start start 2024-01-11T08:23:13.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.984+0000 7f12a4fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12880040b0 0x7f12a0155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:13.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.984+0000 7f12a4fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12a0155200 0x7f12a0152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:13.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.984+0000 7f12a4fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12a00cd130 0x7f12a00cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:13.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.984+0000 7f12a4fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a00cdae0 con 0x7f12880040b0 2024-01-11T08:23:13.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.984+0000 7f12a4fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a0157000 con 0x7f12a00cd130 2024-01-11T08:23:13.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.984+0000 7f12a4fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a0155670 con 0x7f12a0155200 2024-01-11T08:23:13.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.984+0000 7f129e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12880040b0 0x7f12a0155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:13.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.985+0000 7f129e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12880040b0 0x7f12a0155d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38214/0 (socket says 172.21.15.27:38214) 2024-01-11T08:23:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.985+0000 7f129e59c700 1 -- 172.21.15.27:0/734027151 learned_addr learned my addr 172.21.15.27:0/734027151 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.985+0000 7f129dd9b700 1 --2- 172.21.15.27:0/734027151 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12a0155200 0x7f12a0152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.985+0000 7f129ed9d700 1 --2- 172.21.15.27:0/734027151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12a00cd130 0x7f12a00cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.985+0000 7f129e59c700 1 -- 172.21.15.27:0/734027151 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12a0155200 msgr2=0x7f12a0152bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.985+0000 7f129e59c700 1 --2- 172.21.15.27:0/734027151 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12a0155200 0x7f12a0152bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.985+0000 7f129e59c700 1 -- 172.21.15.27:0/734027151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12a00cd130 msgr2=0x7f12a00cd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.985+0000 7f129e59c700 1 --2- 172.21.15.27:0/734027151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12a00cd130 0x7f12a00cd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.985+0000 7f129e59c700 1 -- 172.21.15.27:0/734027151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a01563f0 con 0x7f12880040b0 2024-01-11T08:23:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.985+0000 7f129e59c700 1 --2- 172.21.15.27:0/734027151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12880040b0 0x7f12a0155d30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f12900073c0 tx=0x7f1290001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.985+0000 7f128e7fc700 1 -- 172.21.15.27:0/734027151 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1290018030 con 0x7f12880040b0 2024-01-11T08:23:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.985+0000 7f128e7fc700 1 -- 172.21.15.27:0/734027151 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129000f850 con 0x7f12880040b0 2024-01-11T08:23:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.985+0000 7f128e7fc700 1 -- 172.21.15.27:0/734027151 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129000fc30 con 0x7f12880040b0 2024-01-11T08:23:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.985+0000 7f12a4fff700 1 -- 172.21.15.27:0/734027151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a01566d0 con 0x7f12880040b0 2024-01-11T08:23:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.986+0000 7f12a4fff700 1 -- 172.21.15.27:0/734027151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a0156c90 con 0x7f12880040b0 2024-01-11T08:23:13.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.987+0000 7f12a4fff700 1 -- 172.21.15.27:0/734027151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a004c3c0 con 0x7f12880040b0 2024-01-11T08:23:13.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.987+0000 7f128e7fc700 1 -- 172.21.15.27:0/734027151 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f129001b070 con 0x7f12880040b0 2024-01-11T08:23:13.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.989+0000 7f128e7fc700 1 --2- 172.21.15.27:0/734027151 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1280066f30 0x7f12800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:13.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.989+0000 7f128e7fc700 1 -- 172.21.15.27:0/734027151 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f1290019070 con 0x7f12880040b0 2024-01-11T08:23:13.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.989+0000 7f129dd9b700 1 --2- 172.21.15.27:0/734027151 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1280066f30 0x7f12800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:13.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.990+0000 7f129dd9b700 1 --2- 172.21.15.27:0/734027151 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1280066f30 0x7f12800693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f12940091e0 tx=0x7f129401a000 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:13.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:13.993+0000 7f128e7fc700 1 -- 172.21.15.27:0/734027151 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1290051700 con 0x7f12880040b0 2024-01-11T08:23:14.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.223+0000 7f12a4fff700 1 -- 172.21.15.27:0/734027151 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f12a006c680 con 0x7f1280066f30 2024-01-11T08:23:14.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.225+0000 7f128e7fc700 1 -- 172.21.15.27:0/734027151 <== mgr.5434 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f12a006c680 con 0x7f1280066f30 2024-01-11T08:23:14.224 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T08:23:14.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:23:14.224 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-11T08:23:14.224 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:23:14.224 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T08:23:14.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.227+0000 7f12a4fff700 1 -- 172.21.15.27:0/734027151 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1280066f30 msgr2=0x7f12800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:14.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.227+0000 7f12a4fff700 1 --2- 172.21.15.27:0/734027151 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1280066f30 0x7f12800693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f12940091e0 tx=0x7f129401a000 comp rx=0 tx=0).stop 2024-01-11T08:23:14.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.227+0000 7f12a4fff700 1 -- 172.21.15.27:0/734027151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12880040b0 msgr2=0x7f12a0155d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:14.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.227+0000 7f12a4fff700 1 --2- 172.21.15.27:0/734027151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12880040b0 0x7f12a0155d30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f12900073c0 tx=0x7f1290001dc0 comp rx=0 tx=0).stop 2024-01-11T08:23:14.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.227+0000 7f12a4fff700 1 -- 172.21.15.27:0/734027151 shutdown_connections 2024-01-11T08:23:14.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.227+0000 7f12a4fff700 1 --2- 172.21.15.27:0/734027151 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1280066f30 0x7f12800693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:14.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.227+0000 7f12a4fff700 1 --2- 172.21.15.27:0/734027151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12a00cd130 0x7f12a00cd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:14.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.227+0000 7f12a4fff700 1 --2- 172.21.15.27:0/734027151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12880040b0 0x7f12a0155d30 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:14.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.227+0000 7f12a4fff700 1 --2- 172.21.15.27:0/734027151 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12a0155200 0x7f12a0152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:14.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.227+0000 7f12a4fff700 1 -- 172.21.15.27:0/734027151 >> 172.21.15.27:0/734027151 conn(0x7f12a00ce3c0 msgr2=0x7f12a015b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:14.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.227+0000 7f12a4fff700 1 -- 172.21.15.27:0/734027151 shutdown_connections 2024-01-11T08:23:14.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.227+0000 7f12a4fff700 1 -- 172.21.15.27:0/734027151 wait complete. 2024-01-11T08:23:14.244 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-11T08:23:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.446+0000 7f0107dff700 1 Processor -- start 2024-01-11T08:23:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.446+0000 7f0107dff700 1 -- start start 2024-01-11T08:23:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.448+0000 7f0107dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0100069bd0 0x7f01000aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.448+0000 7f0107dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f01000c6de0 0x7f01000c7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.448+0000 7f0107dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f010006e480 con 0x7f0100155c50 2024-01-11T08:23:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.448+0000 7f0107dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01000cb5a0 con 0x7f0100069bd0 2024-01-11T08:23:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.448+0000 7f0107dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f010006df00 con 0x7f01000c6de0 2024-01-11T08:23:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.448+0000 7f010539a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0100069bd0 0x7f01000aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.448+0000 7f010539a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0100069bd0 0x7f01000aef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38222/0 (socket says 172.21.15.27:38222) 2024-01-11T08:23:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.448+0000 7f010539a700 1 -- 172.21.15.27:0/2956159959 learned_addr learned my addr 172.21.15.27:0/2956159959 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.448+0000 7f010639c700 1 --2- 172.21.15.27:0/2956159959 >> v2:172.21.15.77:3300/0 conn(0x7f01000c6de0 0x7f01000c7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.449+0000 7f010539a700 1 -- 172.21.15.27:0/2956159959 >> v2:172.21.15.77:3300/0 conn(0x7f01000c6de0 msgr2=0x7f01000c7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.449+0000 7f010539a700 1 --2- 172.21.15.27:0/2956159959 >> v2:172.21.15.77:3300/0 conn(0x7f01000c6de0 0x7f01000c7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:14.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.449+0000 7f010539a700 1 -- 172.21.15.27:0/2956159959 >> v1:172.21.15.27:6789/0 conn(0x7f0100155c50 legacy=0x7f01000694c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:14.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.449+0000 7f010539a700 1 -- 172.21.15.27:0/2956159959 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01000c7a60 con 0x7f0100069bd0 2024-01-11T08:23:14.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.449+0000 7f010539a700 1 --2- 172.21.15.27:0/2956159959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0100069bd0 0x7f01000aef80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f010006d540 tx=0x7f00f000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d398b8b454a1d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:14.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.449+0000 7f0104b99700 1 -- 172.21.15.27:0/2956159959 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f000dce0 con 0x7f0100069bd0 2024-01-11T08:23:14.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.449+0000 7f0104b99700 1 -- 172.21.15.27:0/2956159959 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f000de40 con 0x7f0100069bd0 2024-01-11T08:23:14.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.449+0000 7f0104b99700 1 -- 172.21.15.27:0/2956159959 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f00196d0 con 0x7f0100069bd0 2024-01-11T08:23:14.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.450+0000 7f0107dff700 1 -- 172.21.15.27:0/2956159959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0100069bd0 msgr2=0x7f01000aef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:14.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.450+0000 7f0107dff700 1 --2- 172.21.15.27:0/2956159959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0100069bd0 0x7f01000aef80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f010006d540 tx=0x7f00f000b4f0 comp rx=0 tx=0).stop 2024-01-11T08:23:14.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.450+0000 7f0107dff700 1 -- 172.21.15.27:0/2956159959 shutdown_connections 2024-01-11T08:23:14.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.450+0000 7f0107dff700 1 --2- 172.21.15.27:0/2956159959 >> v2:172.21.15.77:3300/0 conn(0x7f01000c6de0 0x7f01000c7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:14.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.450+0000 7f0107dff700 1 --2- 172.21.15.27:0/2956159959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0100069bd0 0x7f01000aef80 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:14.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.450+0000 7f0107dff700 1 -- 172.21.15.27:0/2956159959 >> 172.21.15.27:0/2956159959 conn(0x7f01000680a0 msgr2=0x7f01000cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:14.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.450+0000 7f0107dff700 1 -- 172.21.15.27:0/2956159959 shutdown_connections 2024-01-11T08:23:14.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.450+0000 7f0107dff700 1 -- 172.21.15.27:0/2956159959 wait complete. 2024-01-11T08:23:14.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.451+0000 7f0107dff700 1 Processor -- start 2024-01-11T08:23:14.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.452+0000 7f0107dff700 1 -- start start 2024-01-11T08:23:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.452+0000 7f0107dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0100069bd0 0x7f010013f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.452+0000 7f0107dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01000c6de0 0x7f01000ca420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.453+0000 7f0107dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0100155c50 0x7f01000ca960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.453+0000 7f0107dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0100140d30 con 0x7f0100069bd0 2024-01-11T08:23:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.453+0000 7f0107dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0100140160 con 0x7f0100155c50 2024-01-11T08:23:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.453+0000 7f0107dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010013fd00 con 0x7f01000c6de0 2024-01-11T08:23:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.453+0000 7f0105b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0100069bd0 0x7f010013f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.453+0000 7f0105b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0100069bd0 0x7f010013f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38226/0 (socket says 172.21.15.27:38226) 2024-01-11T08:23:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.453+0000 7f0105b9b700 1 -- 172.21.15.27:0/1555203353 learned_addr learned my addr 172.21.15.27:0/1555203353 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.453+0000 7f010639c700 1 --2- 172.21.15.27:0/1555203353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0100155c50 0x7f01000ca960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.453+0000 7f010539a700 1 --2- 172.21.15.27:0/1555203353 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01000c6de0 0x7f01000ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.453+0000 7f0105b9b700 1 -- 172.21.15.27:0/1555203353 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01000c6de0 msgr2=0x7f01000ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.453+0000 7f0105b9b700 1 --2- 172.21.15.27:0/1555203353 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01000c6de0 0x7f01000ca420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.453+0000 7f0105b9b700 1 -- 172.21.15.27:0/1555203353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0100155c50 msgr2=0x7f01000ca960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.453+0000 7f0105b9b700 1 --2- 172.21.15.27:0/1555203353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0100155c50 0x7f01000ca960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.453+0000 7f0105b9b700 1 -- 172.21.15.27:0/1555203353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f000b790 con 0x7f0100069bd0 2024-01-11T08:23:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.453+0000 7f010539a700 1 --2- 172.21.15.27:0/1555203353 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01000c6de0 0x7f01000ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:23:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.454+0000 7f0105b9b700 1 --2- 172.21.15.27:0/1555203353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0100069bd0 0x7f010013f7c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f00fc00c7c0 tx=0x7f00fc007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.454+0000 7f00f5ffb700 1 -- 172.21.15.27:0/1555203353 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00fc010510 con 0x7f0100069bd0 2024-01-11T08:23:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.454+0000 7f00f5ffb700 1 -- 172.21.15.27:0/1555203353 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00fc010690 con 0x7f0100069bd0 2024-01-11T08:23:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.454+0000 7f00f5ffb700 1 -- 172.21.15.27:0/1555203353 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00fc010a50 con 0x7f0100069bd0 2024-01-11T08:23:14.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.454+0000 7f0107dff700 1 -- 172.21.15.27:0/1555203353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0100140400 con 0x7f0100069bd0 2024-01-11T08:23:14.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.454+0000 7f0107dff700 1 -- 172.21.15.27:0/1555203353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01001409c0 con 0x7f0100069bd0 2024-01-11T08:23:14.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.455+0000 7f0107dff700 1 -- 172.21.15.27:0/1555203353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0100141010 con 0x7f0100069bd0 2024-01-11T08:23:14.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.456+0000 7f00f5ffb700 1 -- 172.21.15.27:0/1555203353 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f00fc01b400 con 0x7f0100069bd0 2024-01-11T08:23:14.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.457+0000 7f00f5ffb700 1 --2- 172.21.15.27:0/1555203353 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f00e4067000 0x7f00e40694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:14.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.457+0000 7f00f5ffb700 1 -- 172.21.15.27:0/1555203353 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f00fc083f70 con 0x7f0100069bd0 2024-01-11T08:23:14.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.457+0000 7f010539a700 1 --2- 172.21.15.27:0/1555203353 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f00e4067000 0x7f00e40694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:14.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.458+0000 7f010539a700 1 --2- 172.21.15.27:0/1555203353 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f00e4067000 0x7f00e40694b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f010006d540 tx=0x7f00f001b000 comp rx=0 tx=0).ready entity=mgr.5434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:14.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.461+0000 7f00f5ffb700 1 -- 172.21.15.27:0/1555203353 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f00fc051340 con 0x7f0100069bd0 2024-01-11T08:23:14.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:14.718+0000 7f0107dff700 1 -- 172.21.15.27:0/1555203353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f010004c3c0 con 0x7f0100069bd0 2024-01-11T08:23:15.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.658+0000 7f00f5ffb700 1 -- 172.21.15.27:0/1555203353 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 19) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f00fc051710 con 0x7f0100069bd0 2024-01-11T08:23:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.659+0000 7f00f5ffb700 1 -- 172.21.15.27:0/1555203353 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7f00fc054dd0 con 0x7f0100069bd0 2024-01-11T08:23:15.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.663+0000 7f0107dff700 1 -- 172.21.15.27:0/1555203353 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f00e4067000 msgr2=0x7f00e40694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:15.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.663+0000 7f0107dff700 1 --2- 172.21.15.27:0/1555203353 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f00e4067000 0x7f00e40694b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f010006d540 tx=0x7f00f001b000 comp rx=0 tx=0).stop 2024-01-11T08:23:15.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.663+0000 7f0107dff700 1 -- 172.21.15.27:0/1555203353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0100069bd0 msgr2=0x7f010013f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:15.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.663+0000 7f0107dff700 1 --2- 172.21.15.27:0/1555203353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0100069bd0 0x7f010013f7c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f00fc00c7c0 tx=0x7f00fc007f00 comp rx=0 tx=0).stop 2024-01-11T08:23:15.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.664+0000 7f0107dff700 1 -- 172.21.15.27:0/1555203353 shutdown_connections 2024-01-11T08:23:15.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.664+0000 7f0107dff700 1 --2- 172.21.15.27:0/1555203353 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f00e4067000 0x7f00e40694b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:15.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.664+0000 7f0107dff700 1 --2- 172.21.15.27:0/1555203353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0100155c50 0x7f01000ca960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:15.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.664+0000 7f0107dff700 1 --2- 172.21.15.27:0/1555203353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0100069bd0 0x7f010013f7c0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:15.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.664+0000 7f0107dff700 1 --2- 172.21.15.27:0/1555203353 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01000c6de0 0x7f01000ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:15.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.664+0000 7f0107dff700 1 -- 172.21.15.27:0/1555203353 >> 172.21.15.27:0/1555203353 conn(0x7f01000680a0 msgr2=0x7f01000c2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:15.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.664+0000 7f0107dff700 1 -- 172.21.15.27:0/1555203353 shutdown_connections 2024-01-11T08:23:15.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.664+0000 7f0107dff700 1 -- 172.21.15.27:0/1555203353 wait complete. 2024-01-11T08:23:15.684 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-11T08:23:15.870 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:15.870+0000 7f1fe125a1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:23:15.880 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:15.881+0000 7f90db4a01c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:23:15.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.898+0000 7fcb48fff700 1 Processor -- start 2024-01-11T08:23:15.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.898+0000 7fcb48fff700 1 -- start start 2024-01-11T08:23:15.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.899+0000 7fcb48fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcb44141190 0x7fcb440c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:15.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.899+0000 7fcb48fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcb440c8f60 0x7fcb4415b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:15.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.899+0000 7fcb48fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4415c060 con 0x7fcb44140dc0 2024-01-11T08:23:15.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.899+0000 7fcb48fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb440cc9b0 con 0x7fcb440c8f60 2024-01-11T08:23:15.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.899+0000 7fcb48fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb440ae390 con 0x7fcb44141190 2024-01-11T08:23:15.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.900+0000 7fcb4259c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcb44140dc0 0x7fcb440adc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58730/0 (socket says 172.21.15.27:58730) 2024-01-11T08:23:15.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.900+0000 7fcb4259c700 1 -- 172.21.15.27:0/3387560591 learned_addr learned my addr 172.21.15.27:0/3387560591 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:23:15.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.900+0000 7fcb42d9d700 1 --2- 172.21.15.27:0/3387560591 >> v2:172.21.15.27:3300/0 conn(0x7fcb440c8f60 0x7fcb4415b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:15.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.900+0000 7fcb41d9b700 1 --2- 172.21.15.27:0/3387560591 >> v2:172.21.15.77:3300/0 conn(0x7fcb44141190 0x7fcb440c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:15.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.900+0000 7fcb42d9d700 1 -- 172.21.15.27:0/3387560591 >> v2:172.21.15.77:3300/0 conn(0x7fcb44141190 msgr2=0x7fcb440c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:15.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.900+0000 7fcb42d9d700 1 --2- 172.21.15.27:0/3387560591 >> v2:172.21.15.77:3300/0 conn(0x7fcb44141190 0x7fcb440c8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:15.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.900+0000 7fcb42d9d700 1 -- 172.21.15.27:0/3387560591 >> v1:172.21.15.27:6789/0 conn(0x7fcb44140dc0 legacy=0x7fcb440adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:15.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.900+0000 7fcb42d9d700 1 -- 172.21.15.27:0/3387560591 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb440c9370 con 0x7fcb440c8f60 2024-01-11T08:23:15.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.900+0000 7fcb41d9b700 1 --2- 172.21.15.27:0/3387560591 >> v2:172.21.15.77:3300/0 conn(0x7fcb44141190 0x7fcb440c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:23:15.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.900+0000 7fcb4159a700 1 -- 172.21.15.27:0/3387560591 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1902173407 0 0) 0x7fcb34001110 con 0x7fcb44140dc0 2024-01-11T08:23:15.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.900+0000 7fcb42d9d700 1 --2- 172.21.15.27:0/3387560591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb440c8f60 0x7fcb4415b100 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fcb440c5410 tx=0x7fcb3c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5d328701cbeceaa server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.901+0000 7fcb4159a700 1 -- 172.21.15.27:0/3387560591 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c00e070 con 0x7fcb440c8f60 2024-01-11T08:23:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.901+0000 7fcb4159a700 1 -- 172.21.15.27:0/3387560591 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3c00bb60 con 0x7fcb440c8f60 2024-01-11T08:23:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.901+0000 7fcb4159a700 1 -- 172.21.15.27:0/3387560591 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c012430 con 0x7fcb440c8f60 2024-01-11T08:23:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.901+0000 7fcb48fff700 1 -- 172.21.15.27:0/3387560591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb440c8f60 msgr2=0x7fcb4415b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.901+0000 7fcb48fff700 1 --2- 172.21.15.27:0/3387560591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb440c8f60 0x7fcb4415b100 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fcb440c5410 tx=0x7fcb3c009780 comp rx=0 tx=0).stop 2024-01-11T08:23:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.901+0000 7fcb48fff700 1 -- 172.21.15.27:0/3387560591 shutdown_connections 2024-01-11T08:23:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.901+0000 7fcb48fff700 1 --2- 172.21.15.27:0/3387560591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb440c8f60 0x7fcb4415b100 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.901+0000 7fcb48fff700 1 --2- 172.21.15.27:0/3387560591 >> v2:172.21.15.77:3300/0 conn(0x7fcb44141190 0x7fcb440c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.901+0000 7fcb48fff700 1 -- 172.21.15.27:0/3387560591 >> 172.21.15.27:0/3387560591 conn(0x7fcb44063ca0 msgr2=0x7fcb4406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.902+0000 7fcb48fff700 1 -- 172.21.15.27:0/3387560591 shutdown_connections 2024-01-11T08:23:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.902+0000 7fcb48fff700 1 -- 172.21.15.27:0/3387560591 wait complete. 2024-01-11T08:23:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.903+0000 7fcb48fff700 1 Processor -- start 2024-01-11T08:23:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.903+0000 7fcb48fff700 1 -- start start 2024-01-11T08:23:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.904+0000 7fcb48fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb440c8f60 0x7fcb44156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.904+0000 7fcb48fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb44140dc0 0x7fcb4415e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.904+0000 7fcb48fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb44141190 0x7fcb440af4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.904+0000 7fcb48fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb440b1390 con 0x7fcb440c8f60 2024-01-11T08:23:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.904+0000 7fcb48fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb440b1f10 con 0x7fcb44141190 2024-01-11T08:23:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.904+0000 7fcb48fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb440b15e0 con 0x7fcb44140dc0 2024-01-11T08:23:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.904+0000 7fcb4259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb440c8f60 0x7fcb44156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.904+0000 7fcb4259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb440c8f60 0x7fcb44156a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38248/0 (socket says 172.21.15.27:38248) 2024-01-11T08:23:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.904+0000 7fcb4259c700 1 -- 172.21.15.27:0/797290634 learned_addr learned my addr 172.21.15.27:0/797290634 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.904+0000 7fcb42d9d700 1 --2- 172.21.15.27:0/797290634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb44141190 0x7fcb440af4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.904+0000 7fcb41d9b700 1 --2- 172.21.15.27:0/797290634 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb44140dc0 0x7fcb4415e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.905+0000 7fcb4259c700 1 -- 172.21.15.27:0/797290634 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb44140dc0 msgr2=0x7fcb4415e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.905+0000 7fcb4259c700 1 --2- 172.21.15.27:0/797290634 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb44140dc0 0x7fcb4415e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.905+0000 7fcb4259c700 1 -- 172.21.15.27:0/797290634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb44141190 msgr2=0x7fcb440af4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.905+0000 7fcb4259c700 1 --2- 172.21.15.27:0/797290634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb44141190 0x7fcb440af4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.905+0000 7fcb4259c700 1 -- 172.21.15.27:0/797290634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb38009420 con 0x7fcb440c8f60 2024-01-11T08:23:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.905+0000 7fcb42d9d700 1 --2- 172.21.15.27:0/797290634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb44141190 0x7fcb440af4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:23:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.905+0000 7fcb4259c700 1 --2- 172.21.15.27:0/797290634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb440c8f60 0x7fcb44156a80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fcb3400c840 tx=0x7fcb34007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:15.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.905+0000 7fcb327fc700 1 -- 172.21.15.27:0/797290634 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34005b30 con 0x7fcb440c8f60 2024-01-11T08:23:15.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.905+0000 7fcb327fc700 1 -- 172.21.15.27:0/797290634 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb34005c90 con 0x7fcb440c8f60 2024-01-11T08:23:15.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.906+0000 7fcb327fc700 1 -- 172.21.15.27:0/797290634 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34007300 con 0x7fcb440c8f60 2024-01-11T08:23:15.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.906+0000 7fcb48fff700 1 -- 172.21.15.27:0/797290634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3c009a20 con 0x7fcb440c8f60 2024-01-11T08:23:15.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.906+0000 7fcb48fff700 1 -- 172.21.15.27:0/797290634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb440b1ba0 con 0x7fcb440c8f60 2024-01-11T08:23:15.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.907+0000 7fcb48fff700 1 -- 172.21.15.27:0/797290634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4404c3c0 con 0x7fcb440c8f60 2024-01-11T08:23:15.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.908+0000 7fcb327fc700 1 -- 172.21.15.27:0/797290634 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fcb3401b3b0 con 0x7fcb440c8f60 2024-01-11T08:23:15.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.909+0000 7fcb327fc700 1 --2- 172.21.15.27:0/797290634 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcb24066f80 0x7fcb24069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:15.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.909+0000 7fcb327fc700 1 -- 172.21.15.27:0/797290634 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fcb340839c0 con 0x7fcb440c8f60 2024-01-11T08:23:15.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.909+0000 7fcb41d9b700 1 -- 172.21.15.27:0/797290634 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcb24066f80 msgr2=0x7fcb24069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:23:15.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.909+0000 7fcb41d9b700 1 --2- 172.21.15.27:0/797290634 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcb24066f80 0x7fcb24069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:23:15.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:15.913+0000 7fcb327fc700 1 -- 172.21.15.27:0/797290634 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcb34050b90 con 0x7fcb440c8f60 2024-01-11T08:23:16.049 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:16.049+0000 7f1fe125a1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:23:16.090 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:16.087+0000 7f90db4a01c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:23:16.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.110+0000 7fcb41d9b700 1 -- 172.21.15.27:0/797290634 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcb24066f80 msgr2=0x7fcb24069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:23:16.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.110+0000 7fcb41d9b700 1 --2- 172.21.15.27:0/797290634 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcb24066f80 0x7fcb24069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:23:16.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.162+0000 7fcb48fff700 1 -- 172.21.15.27:0/797290634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fcb4415c060 con 0x7fcb440c8f60 2024-01-11T08:23:16.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.168+0000 7fcb327fc700 1 -- 172.21.15.27:0/797290634 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v17)=0 v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7fcb4415c060 con 0x7fcb440c8f60 2024-01-11T08:23:16.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.170+0000 7fcb48fff700 1 -- 172.21.15.27:0/797290634 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcb24066f80 msgr2=0x7fcb24069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:23:16.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.170+0000 7fcb48fff700 1 --2- 172.21.15.27:0/797290634 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcb24066f80 0x7fcb24069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.170+0000 7fcb48fff700 1 -- 172.21.15.27:0/797290634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb440c8f60 msgr2=0x7fcb44156a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:16.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.170+0000 7fcb48fff700 1 --2- 172.21.15.27:0/797290634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb440c8f60 0x7fcb44156a80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fcb3400c840 tx=0x7fcb34007c50 comp rx=0 tx=0).stop 2024-01-11T08:23:16.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.170+0000 7fcb48fff700 1 -- 172.21.15.27:0/797290634 shutdown_connections 2024-01-11T08:23:16.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.170+0000 7fcb48fff700 1 --2- 172.21.15.27:0/797290634 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcb24066f80 0x7fcb24069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.170+0000 7fcb48fff700 1 --2- 172.21.15.27:0/797290634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb44141190 0x7fcb440af4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.170+0000 7fcb48fff700 1 --2- 172.21.15.27:0/797290634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb440c8f60 0x7fcb44156a80 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.170+0000 7fcb48fff700 1 --2- 172.21.15.27:0/797290634 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb44140dc0 0x7fcb4415e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.170+0000 7fcb48fff700 1 -- 172.21.15.27:0/797290634 >> 172.21.15.27:0/797290634 conn(0x7fcb44063ca0 msgr2=0x7fcb4406d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:16.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.170+0000 7fcb48fff700 1 -- 172.21.15.27:0/797290634 shutdown_connections 2024-01-11T08:23:16.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.170+0000 7fcb48fff700 1 -- 172.21.15.27:0/797290634 wait complete. 2024-01-11T08:23:16.188 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-11T08:23:16.366 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:16.366+0000 7f1fe125a1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:23:16.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.395+0000 7fafd3fff700 1 Processor -- start 2024-01-11T08:23:16.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.395+0000 7fafd3fff700 1 -- start start 2024-01-11T08:23:16.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.397+0000 7fafd3fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fafcc157410 0x7fafcc0ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:16.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.397+0000 7fafd3fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafcc1577e0 0x7fafcc0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:16.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.397+0000 7fafd3fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fafcc069540 0x7fafcc0ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:16.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.397+0000 7fafd3fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafcc0699d0 con 0x7fafcc069540 2024-01-11T08:23:16.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.397+0000 7fafd3fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafcc069cb0 con 0x7fafcc157410 2024-01-11T08:23:16.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.397+0000 7fafd3fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc069f90 con 0x7fafcc1577e0 2024-01-11T08:23:16.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.398+0000 7fafd259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fafcc069540 0x7fafcc0ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:16.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.398+0000 7fafd159a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafcc1577e0 0x7fafcc0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:16.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.398+0000 7fafd1d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fafcc157410 0x7fafcc0ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:16.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.398+0000 7fafd259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fafcc069540 0x7fafcc0ae540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38264/0 (socket says 172.21.15.27:38264) 2024-01-11T08:23:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.398+0000 7fafd1d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fafcc157410 0x7fafcc0ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:34410/0 (socket says 172.21.15.27:34410) 2024-01-11T08:23:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.398+0000 7fafd259c700 1 -- 172.21.15.27:0/4104388075 learned_addr learned my addr 172.21.15.27:0/4104388075 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.398+0000 7fafd259c700 1 -- 172.21.15.27:0/4104388075 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafcc1577e0 msgr2=0x7fafcc0ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.398+0000 7fafd259c700 1 --2- 172.21.15.27:0/4104388075 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafcc1577e0 0x7fafcc0ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.398+0000 7fafd259c700 1 -- 172.21.15.27:0/4104388075 >> v2:172.21.15.77:3300/0 conn(0x7fafcc157410 msgr2=0x7fafcc0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.398+0000 7fafd259c700 1 --2- 172.21.15.27:0/4104388075 >> v2:172.21.15.77:3300/0 conn(0x7fafcc157410 0x7fafcc0ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.398+0000 7fafd259c700 1 -- 172.21.15.27:0/4104388075 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafcc06a270 con 0x7fafcc069540 2024-01-11T08:23:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.398+0000 7fafd1d9b700 1 --2- 172.21.15.27:0/4104388075 >> v2:172.21.15.77:3300/0 conn(0x7fafcc157410 0x7fafcc0ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:23:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.398+0000 7fafd259c700 1 --2- 172.21.15.27:0/4104388075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafcc069540 0x7fafcc0ae540 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fafcc070c90 tx=0x7fafc4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a9acfb9890dadb7 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.399+0000 7fafd0d99700 1 -- 172.21.15.27:0/4104388075 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc400e070 con 0x7fafcc069540 2024-01-11T08:23:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.399+0000 7fafd0d99700 1 -- 172.21.15.27:0/4104388075 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc400b970 con 0x7fafcc069540 2024-01-11T08:23:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.399+0000 7fafd0d99700 1 -- 172.21.15.27:0/4104388075 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc40092e0 con 0x7fafcc069540 2024-01-11T08:23:16.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.399+0000 7fafd3fff700 1 -- 172.21.15.27:0/4104388075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafcc069540 msgr2=0x7fafcc0ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:16.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.399+0000 7fafd3fff700 1 --2- 172.21.15.27:0/4104388075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafcc069540 0x7fafcc0ae540 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fafcc070c90 tx=0x7fafc4009840 comp rx=0 tx=0).stop 2024-01-11T08:23:16.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.400+0000 7fafd3fff700 1 -- 172.21.15.27:0/4104388075 shutdown_connections 2024-01-11T08:23:16.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.400+0000 7fafd3fff700 1 --2- 172.21.15.27:0/4104388075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafcc069540 0x7fafcc0ae540 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.400+0000 7fafd3fff700 1 --2- 172.21.15.27:0/4104388075 >> v2:172.21.15.77:3300/0 conn(0x7fafcc157410 0x7fafcc0ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.400+0000 7fafd3fff700 1 --2- 172.21.15.27:0/4104388075 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafcc1577e0 0x7fafcc0ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.400+0000 7fafd3fff700 1 -- 172.21.15.27:0/4104388075 >> 172.21.15.27:0/4104388075 conn(0x7fafcc0680a0 msgr2=0x7fafcc0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:16.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.400+0000 7fafd3fff700 1 -- 172.21.15.27:0/4104388075 shutdown_connections 2024-01-11T08:23:16.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.400+0000 7fafd3fff700 1 -- 172.21.15.27:0/4104388075 wait complete. 2024-01-11T08:23:16.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.402+0000 7fafd3fff700 1 Processor -- start 2024-01-11T08:23:16.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.402+0000 7fafd3fff700 1 -- start start 2024-01-11T08:23:16.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.404+0000 7fafd3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafcc069540 0x7fafcc0b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.404+0000 7fafd3fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafcc157410 0x7fafcc16c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.404+0000 7fafd3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafcc1577e0 0x7fafcc0c5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.404+0000 7fafd3fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc16d380 con 0x7fafcc069540 2024-01-11T08:23:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.404+0000 7fafd3fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc16d660 con 0x7fafcc1577e0 2024-01-11T08:23:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.404+0000 7fafd3fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc0c4980 con 0x7fafcc157410 2024-01-11T08:23:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.404+0000 7fafd1d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafcc069540 0x7fafcc0b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.404+0000 7fafd1d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafcc069540 0x7fafcc0b0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38278/0 (socket says 172.21.15.27:38278) 2024-01-11T08:23:16.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.404+0000 7fafd1d9b700 1 -- 172.21.15.27:0/1721006096 learned_addr learned my addr 172.21.15.27:0/1721006096 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:16.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.404+0000 7fafd159a700 1 --2- 172.21.15.27:0/1721006096 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafcc157410 0x7fafcc16c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:16.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.404+0000 7fafd259c700 1 --2- 172.21.15.27:0/1721006096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafcc1577e0 0x7fafcc0c5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:16.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.405+0000 7fafd1d9b700 1 -- 172.21.15.27:0/1721006096 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafcc157410 msgr2=0x7fafcc16c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:16.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.405+0000 7fafd1d9b700 1 --2- 172.21.15.27:0/1721006096 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafcc157410 0x7fafcc16c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.405+0000 7fafd1d9b700 1 -- 172.21.15.27:0/1721006096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafcc1577e0 msgr2=0x7fafcc0c5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:16.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.405+0000 7fafd1d9b700 1 --2- 172.21.15.27:0/1721006096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafcc1577e0 0x7fafcc0c5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.405+0000 7fafd1d9b700 1 -- 172.21.15.27:0/1721006096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc80093c0 con 0x7fafcc069540 2024-01-11T08:23:16.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.405+0000 7fafd259c700 1 --2- 172.21.15.27:0/1721006096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafcc1577e0 0x7fafcc0c5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:23:16.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.405+0000 7fafd1d9b700 1 --2- 172.21.15.27:0/1721006096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafcc069540 0x7fafcc0b0a30 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fafc800ec80 tx=0x7fafc8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:16.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.405+0000 7fafc1ffb700 1 -- 172.21.15.27:0/1721006096 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc8005a00 con 0x7fafcc069540 2024-01-11T08:23:16.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.405+0000 7fafc1ffb700 1 -- 172.21.15.27:0/1721006096 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc8005b60 con 0x7fafcc069540 2024-01-11T08:23:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.405+0000 7fafc1ffb700 1 -- 172.21.15.27:0/1721006096 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc800c850 con 0x7fafcc069540 2024-01-11T08:23:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.406+0000 7fafd3fff700 1 -- 172.21.15.27:0/1721006096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc4009ae0 con 0x7fafcc069540 2024-01-11T08:23:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.406+0000 7fafd3fff700 1 -- 172.21.15.27:0/1721006096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafcc0c4690 con 0x7fafcc069540 2024-01-11T08:23:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.407+0000 7fafd3fff700 1 -- 172.21.15.27:0/1721006096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafcc04c3c0 con 0x7fafcc069540 2024-01-11T08:23:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.408+0000 7fafc1ffb700 1 -- 172.21.15.27:0/1721006096 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fafc800cbc0 con 0x7fafcc069540 2024-01-11T08:23:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.409+0000 7fafc1ffb700 1 --2- 172.21.15.27:0/1721006096 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fafb0066f80 0x7fafb0069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.409+0000 7fafc1ffb700 1 -- 172.21.15.27:0/1721006096 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5705+0+0 (secure 0 0 0) 0x7fafc8084a00 con 0x7fafcc069540 2024-01-11T08:23:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.409+0000 7fafd159a700 1 -- 172.21.15.27:0/1721006096 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fafb0066f80 msgr2=0x7fafb0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:23:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.409+0000 7fafd159a700 1 --2- 172.21.15.27:0/1721006096 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fafb0066f80 0x7fafb0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:23:16.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.413+0000 7fafc1ffb700 1 -- 172.21.15.27:0/1721006096 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fafc8055700 con 0x7fafcc069540 2024-01-11T08:23:16.432 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:16.432+0000 7f90db4a01c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:23:16.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.610+0000 7fafd159a700 1 -- 172.21.15.27:0/1721006096 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fafb0066f80 msgr2=0x7fafb0069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:23:16.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.610+0000 7fafd159a700 1 --2- 172.21.15.27:0/1721006096 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fafb0066f80 0x7fafb0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:23:16.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.618+0000 7fafd3fff700 1 -- 172.21.15.27:0/1721006096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fafcc153100 con 0x7fafcc069540 2024-01-11T08:23:16.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.624+0000 7fafc1ffb700 1 -- 172.21.15.27:0/1721006096 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v18)=0 v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7fafc8055290 con 0x7fafcc069540 2024-01-11T08:23:16.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.626+0000 7fafd3fff700 1 -- 172.21.15.27:0/1721006096 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fafb0066f80 msgr2=0x7fafb0069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:23:16.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.626+0000 7fafd3fff700 1 --2- 172.21.15.27:0/1721006096 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fafb0066f80 0x7fafb0069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.626+0000 7fafd3fff700 1 -- 172.21.15.27:0/1721006096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafcc069540 msgr2=0x7fafcc0b0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:16.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.626+0000 7fafd3fff700 1 --2- 172.21.15.27:0/1721006096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafcc069540 0x7fafcc0b0a30 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fafc800ec80 tx=0x7fafc8009120 comp rx=0 tx=0).stop 2024-01-11T08:23:16.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.627+0000 7fafd3fff700 1 -- 172.21.15.27:0/1721006096 shutdown_connections 2024-01-11T08:23:16.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.627+0000 7fafd3fff700 1 --2- 172.21.15.27:0/1721006096 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fafb0066f80 0x7fafb0069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.627+0000 7fafd3fff700 1 --2- 172.21.15.27:0/1721006096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafcc1577e0 0x7fafcc0c5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.627+0000 7fafd3fff700 1 --2- 172.21.15.27:0/1721006096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafcc069540 0x7fafcc0b0a30 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.627+0000 7fafd3fff700 1 --2- 172.21.15.27:0/1721006096 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafcc157410 0x7fafcc16c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.627+0000 7fafd3fff700 1 -- 172.21.15.27:0/1721006096 >> 172.21.15.27:0/1721006096 conn(0x7fafcc0680a0 msgr2=0x7fafcc0c2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:16.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.627+0000 7fafd3fff700 1 -- 172.21.15.27:0/1721006096 shutdown_connections 2024-01-11T08:23:16.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.627+0000 7fafd3fff700 1 -- 172.21.15.27:0/1721006096 wait complete. 2024-01-11T08:23:16.645 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:16.645 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-01-11T08:23:16.699 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-01-11T08:23:16.893 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:16.893+0000 7f1fe125a1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:23:16.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.907+0000 7f2db7bff700 1 Processor -- start 2024-01-11T08:23:16.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.907+0000 7f2db7bff700 1 -- start start 2024-01-11T08:23:16.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.908+0000 7f2db7bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2db0142dc0 0x7f2db00cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:16.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.909+0000 7f2db7bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2db0143190 0x7f2db00cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:16.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.909+0000 7f2db7bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2db00cc020 0x7f2db00cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:16.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.909+0000 7f2db7bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db015df30 con 0x7f2db00cc020 2024-01-11T08:23:16.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.909+0000 7f2db7bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db015d320 con 0x7f2db0142dc0 2024-01-11T08:23:16.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.909+0000 7f2db7bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db015d590 con 0x7f2db0143190 2024-01-11T08:23:16.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.909+0000 7f2db619c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2db00cc020 0x7f2db00cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:16.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.909+0000 7f2db599b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2db0142dc0 0x7f2db00cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.909+0000 7f2db519a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2db0143190 0x7f2db00cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.909+0000 7f2db599b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2db0142dc0 0x7f2db00cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:34432/0 (socket says 172.21.15.27:34432) 2024-01-11T08:23:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.909+0000 7f2db519a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2db0143190 0x7f2db00cb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:45892/0 (socket says 172.21.15.27:45892) 2024-01-11T08:23:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.909+0000 7f2db599b700 1 -- 172.21.15.27:0/2711132412 learned_addr learned my addr 172.21.15.27:0/2711132412 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.910+0000 7f2db619c700 1 -- 172.21.15.27:0/2711132412 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2db0143190 msgr2=0x7f2db00cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.910+0000 7f2db619c700 1 --2- 172.21.15.27:0/2711132412 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2db0143190 0x7f2db00cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.910+0000 7f2db619c700 1 -- 172.21.15.27:0/2711132412 >> v2:172.21.15.77:3300/0 conn(0x7f2db0142dc0 msgr2=0x7f2db00cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.910+0000 7f2db619c700 1 --2- 172.21.15.27:0/2711132412 >> v2:172.21.15.77:3300/0 conn(0x7f2db0142dc0 0x7f2db00cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.910+0000 7f2db619c700 1 -- 172.21.15.27:0/2711132412 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db015d870 con 0x7f2db00cc020 2024-01-11T08:23:16.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.910+0000 7f2db519a700 1 --2- 172.21.15.27:0/2711132412 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2db0143190 0x7f2db00cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:23:16.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.910+0000 7f2db619c700 1 --2- 172.21.15.27:0/2711132412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db00cc020 0x7f2db00cc440 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f2db006f4d0 tx=0x7f2da8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4208b05e88c4a126 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:16.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.910+0000 7f2db4999700 1 -- 172.21.15.27:0/2711132412 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da800e070 con 0x7f2db00cc020 2024-01-11T08:23:16.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.910+0000 7f2db4999700 1 -- 172.21.15.27:0/2711132412 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da800bab0 con 0x7f2db00cc020 2024-01-11T08:23:16.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.910+0000 7f2db4999700 1 -- 172.21.15.27:0/2711132412 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da800be70 con 0x7f2db00cc020 2024-01-11T08:23:16.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.911+0000 7f2db7bff700 1 -- 172.21.15.27:0/2711132412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db00cc020 msgr2=0x7f2db00cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:16.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.911+0000 7f2db7bff700 1 --2- 172.21.15.27:0/2711132412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db00cc020 0x7f2db00cc440 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f2db006f4d0 tx=0x7f2da8009780 comp rx=0 tx=0).stop 2024-01-11T08:23:16.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.911+0000 7f2db7bff700 1 -- 172.21.15.27:0/2711132412 shutdown_connections 2024-01-11T08:23:16.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.911+0000 7f2db7bff700 1 --2- 172.21.15.27:0/2711132412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db00cc020 0x7f2db00cc440 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.911+0000 7f2db7bff700 1 --2- 172.21.15.27:0/2711132412 >> v2:172.21.15.77:3300/0 conn(0x7f2db0142dc0 0x7f2db00cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.911+0000 7f2db7bff700 1 --2- 172.21.15.27:0/2711132412 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2db0143190 0x7f2db00cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.911+0000 7f2db7bff700 1 -- 172.21.15.27:0/2711132412 >> 172.21.15.27:0/2711132412 conn(0x7f2db0063ca0 msgr2=0x7f2db006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:16.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.911+0000 7f2db7bff700 1 -- 172.21.15.27:0/2711132412 shutdown_connections 2024-01-11T08:23:16.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.911+0000 7f2db7bff700 1 -- 172.21.15.27:0/2711132412 wait complete. 2024-01-11T08:23:16.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.913+0000 7f2db7bff700 1 Processor -- start 2024-01-11T08:23:16.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.913+0000 7f2db7bff700 1 -- start start 2024-01-11T08:23:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.914+0000 7f2db7bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2db00cc020 0x7f2db0141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.914+0000 7f2db7bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2db0142dc0 0x7f2db00c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.914+0000 7f2db7bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db0143190 0x7f2db0169a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.914+0000 7f2db7bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db00c6460 con 0x7f2db0143190 2024-01-11T08:23:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.914+0000 7f2db7bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db00c72c0 con 0x7f2db0142dc0 2024-01-11T08:23:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.914+0000 7f2db7bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db00c66b0 con 0x7f2db00cc020 2024-01-11T08:23:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.914+0000 7f2db619c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db0143190 0x7f2db0169a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.914+0000 7f2db599b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2db00cc020 0x7f2db0141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.914+0000 7f2db619c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db0143190 0x7f2db0169a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38296/0 (socket says 172.21.15.27:38296) 2024-01-11T08:23:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.914+0000 7f2db619c700 1 -- 172.21.15.27:0/1578625644 learned_addr learned my addr 172.21.15.27:0/1578625644 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.914+0000 7f2db519a700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2db0142dc0 0x7f2db00c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.915+0000 7f2db599b700 1 -- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2db0142dc0 msgr2=0x7f2db00c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.915+0000 7f2db599b700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2db0142dc0 0x7f2db00c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.915+0000 7f2db599b700 1 -- 172.21.15.27:0/1578625644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db0143190 msgr2=0x7f2db0169a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.915+0000 7f2db599b700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db0143190 0x7f2db0169a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.915+0000 7f2db599b700 1 -- 172.21.15.27:0/1578625644 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da0009420 con 0x7f2db00cc020 2024-01-11T08:23:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.915+0000 7f2db619c700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db0143190 0x7f2db0169a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:23:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.915+0000 7f2db599b700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2db00cc020 0x7f2db0141ca0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2dac006d40 tx=0x7f2dac009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.915+0000 7f2da5ffb700 1 -- 172.21.15.27:0/1578625644 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac013680 con 0x7f2db00cc020 2024-01-11T08:23:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.915+0000 7f2da5ffb700 1 -- 172.21.15.27:0/1578625644 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dac0137e0 con 0x7f2db00cc020 2024-01-11T08:23:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.915+0000 7f2db7bff700 1 -- 172.21.15.27:0/1578625644 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da8009a20 con 0x7f2db00cc020 2024-01-11T08:23:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.915+0000 7f2da5ffb700 1 -- 172.21.15.27:0/1578625644 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac013b60 con 0x7f2db00cc020 2024-01-11T08:23:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.915+0000 7f2db7bff700 1 -- 172.21.15.27:0/1578625644 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db00c6cf0 con 0x7f2db00cc020 2024-01-11T08:23:16.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.916+0000 7f2db7bff700 1 -- 172.21.15.27:0/1578625644 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db00c7070 con 0x7f2db00cc020 2024-01-11T08:23:16.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.918+0000 7f2da5ffb700 1 -- 172.21.15.27:0/1578625644 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 19) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f2dac00c3e0 con 0x7f2db00cc020 2024-01-11T08:23:16.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.919+0000 7f2da5ffb700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d94066f80 0x7f2d94069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:16.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.919+0000 7f2da5ffb700 1 -- 172.21.15.27:0/1578625644 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f2dac0846e0 con 0x7f2db00cc020 2024-01-11T08:23:16.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.919+0000 7f2db519a700 1 -- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d94066f80 msgr2=0x7f2d94069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:23:16.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.919+0000 7f2db519a700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d94066f80 0x7f2d94069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:23:16.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:16.928+0000 7f2da5ffb700 1 -- 172.21.15.27:0/1578625644 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2dac0518b0 con 0x7f2db00cc020 2024-01-11T08:23:17.002 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:17.003+0000 7f90db4a01c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:23:17.068 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:17.069+0000 7f1fe125a1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:23:17.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:17.120+0000 7f2db519a700 1 -- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d94066f80 msgr2=0x7f2d94069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:23:17.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:17.120+0000 7f2db519a700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d94066f80 0x7f2d94069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:23:17.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:17.162+0000 7f2db7bff700 1 -- 172.21.15.27:0/1578625644 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2db004c3c0 con 0x7f2d94066f80 2024-01-11T08:23:17.182 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:17.183+0000 7f90db4a01c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:23:17.478 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:17.478+0000 7f1fe125a1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:23:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:17.521+0000 7f2db519a700 1 -- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d94066f80 msgr2=0x7f2d94069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:23:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:17.521+0000 7f2db519a700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d94066f80 0x7f2d94069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-11T08:23:17.619 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:17.620+0000 7f90db4a01c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:23:18.251 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:18.251+0000 7f1fe125a1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:23:18.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:18.322+0000 7f2db519a700 1 -- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d94066f80 msgr2=0x7f2d94069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:23:18.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:18.322+0000 7f2db519a700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d94066f80 0x7f2d94069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-11T08:23:18.338 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:18.338+0000 7f1fe125a1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:23:18.444 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:18.445+0000 7f90db4a01c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:23:18.492 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:18.492+0000 7f1fe125a1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:23:18.537 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:18.538+0000 7f90db4a01c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:23:18.577 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:18.577+0000 7f1fe125a1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:23:18.702 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:18.703+0000 7f90db4a01c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:23:18.790 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:18.791+0000 7f1fe125a1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:23:18.794 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:18.795+0000 7f90db4a01c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:23:18.885 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:18.885+0000 7f1fe125a1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:23:18.979 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:18.980+0000 7f1fe125a1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:23:19.021 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:19.022+0000 7f90db4a01c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:23:19.073 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:19.073+0000 7f1fe125a1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:23:19.122 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:19.123+0000 7f90db4a01c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:23:19.224 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:19.225+0000 7f90db4a01c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:23:19.261 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:19.261+0000 7f1fe125a1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:23:19.324 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:19.325+0000 7f90db4a01c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:23:19.356 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:19.356+0000 7f1fe125a1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:23:19.524 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:19.525+0000 7f90db4a01c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:23:19.627 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:19.628+0000 7f90db4a01c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:23:19.697 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:19.697+0000 7f1fe125a1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:23:19.784 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:19.785+0000 7f1fe125a1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:23:19.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:19.924+0000 7f2db519a700 1 -- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d94066f80 msgr2=0x7f2d94069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:23:19.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:19.924+0000 7f2db519a700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d94066f80 0x7f2d94069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-11T08:23:19.987 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:19.988+0000 7f90db4a01c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:23:20.081 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:20.082+0000 7f90db4a01c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:23:20.322 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:20.322+0000 7f1fe125a1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:23:20.506 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:20.506+0000 7f1fe125a1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:23:20.660 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:20.660+0000 7f90db4a01c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:23:20.857 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:20.858+0000 7f90db4a01c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:23:20.879 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:20.879+0000 7f1fe125a1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:23:21.251 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:21.252+0000 7f90db4a01c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:23:21.297 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:21.297+0000 7f1fe125a1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:23:21.477 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:21.477+0000 7f1fe125a1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:23:21.661 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:21.661+0000 7f1fe125a1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:23:21.697 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:21.698+0000 7f90db4a01c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:23:21.767 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:21.767+0000 7f1fe125a1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:23:21.773 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:23:21.781 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:23:21.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:21.781+0000 7f2da5ffb700 1 -- 172.21.15.27:0/1578625644 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mgrmap(e 20) v1 ==== 84470+0+0 (secure 0 0 0) 0x7f2dac051c80 con 0x7f2db00cc020 2024-01-11T08:23:21.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:21.781+0000 7f2da5ffb700 1 -- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d94066f80 msgr2=0x7f2d94069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:23:21.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:21.781+0000 7f2da5ffb700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d94066f80 0x7f2d94069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:21.907 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:21.908+0000 7f90db4a01c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:23:21.909 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:21.910+0000 7f1f56839700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:23:21.910 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:21.910+0000 7f1f56839700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:23:21.910 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:21.910+0000 7f1f56839700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:23:21.910 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:21.910+0000 7f1f56839700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:23:21.910 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:21.910+0000 7f1f56839700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:23:21.910 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:21.910+0000 7f1f5983f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:23:21.910 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:21.910+0000 7f1f5983f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:23:21.911 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:21.910+0000 7f1f5983f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:23:21.911 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:21.910+0000 7f1f5983f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:23:21.911 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:23:21.910+0000 7f1f5983f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:23:22.105 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:22.106+0000 7f90db4a01c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:23:22.221 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:23:22.222+0000 7f90db4a01c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:23:22.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:22.786+0000 7f2da5ffb700 1 -- 172.21.15.27:0/1578625644 <== mon.2 v2:172.21.15.77:3301/0 8 ==== mgrmap(e 21) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f2dac05c080 con 0x7f2db00cc020 2024-01-11T08:23:22.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:22.786+0000 7f2da5ffb700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d94066f80 0x7f2d940693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:22.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:22.786+0000 7f2da5ffb700 1 -- 172.21.15.27:0/1578625644 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2db004c3c0 con 0x7f2d94066f80 2024-01-11T08:23:22.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:22.790+0000 7f2db519a700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d94066f80 0x7f2d940693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:22.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:22.791+0000 7f2db519a700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d94066f80 0x7f2d940693b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2db00caa10 tx=0x7f2da00091a0 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:22.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:22.861+0000 7f2da5ffb700 1 -- 172.21.15.27:0/1578625644 <== mgr.5791 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2db004c3c0 con 0x7f2d94066f80 2024-01-11T08:23:22.860 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path snap_test_dir1 2024-01-11T08:23:22.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:22.863+0000 7f2db7bff700 1 -- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d94066f80 msgr2=0x7f2d940693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:22.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:22.863+0000 7f2db7bff700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d94066f80 0x7f2d940693b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2db00caa10 tx=0x7f2da00091a0 comp rx=0 tx=0).stop 2024-01-11T08:23:22.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:22.863+0000 7f2db7bff700 1 -- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2db00cc020 msgr2=0x7f2db0141ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:22.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:22.863+0000 7f2db7bff700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2db00cc020 0x7f2db0141ca0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2dac006d40 tx=0x7f2dac009250 comp rx=0 tx=0).stop 2024-01-11T08:23:22.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:22.864+0000 7f2db7bff700 1 -- 172.21.15.27:0/1578625644 shutdown_connections 2024-01-11T08:23:22.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:22.864+0000 7f2db7bff700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2d94066f80 0x7f2d940693b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:22.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:22.864+0000 7f2db7bff700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db0143190 0x7f2db0169a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:22.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:22.864+0000 7f2db7bff700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2db00cc020 0x7f2db0141ca0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:22.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:22.864+0000 7f2db7bff700 1 --2- 172.21.15.27:0/1578625644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2db0142dc0 0x7f2db00c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:22.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:22.864+0000 7f2db7bff700 1 -- 172.21.15.27:0/1578625644 >> 172.21.15.27:0/1578625644 conn(0x7f2db0063ca0 msgr2=0x7f2db00c4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:22.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:22.864+0000 7f2db7bff700 1 -- 172.21.15.27:0/1578625644 shutdown_connections 2024-01-11T08:23:22.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:22.864+0000 7f2db7bff700 1 -- 172.21.15.27:0/1578625644 wait complete. 2024-01-11T08:23:22.882 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-01-11T08:23:22.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 2m 2024-01-11T08:23:23.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.126+0000 7f3ed23ff700 1 Processor -- start 2024-01-11T08:23:23.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.126+0000 7f3ed23ff700 1 -- start start 2024-01-11T08:23:23.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.127+0000 7f3ed23ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3ecc069540 0x7f3ecc162270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:23.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.127+0000 7f3ed23ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ecc0699d0 con 0x7f3ecc1577e0 2024-01-11T08:23:23.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.127+0000 7f3ed23ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ecc069cb0 con 0x7f3ecc157410 2024-01-11T08:23:23.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.127+0000 7f3ed23ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ecc069f90 con 0x7f3ecc069540 2024-01-11T08:23:23.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.128+0000 7f3ecb7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3ecc1577e0 0x7f3ecc0ae000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58738/0 (socket says 172.21.15.27:58738) 2024-01-11T08:23:23.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.128+0000 7f3ecb7fe700 1 -- 172.21.15.27:0/1286619087 learned_addr learned my addr 172.21.15.27:0/1286619087 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:23:23.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.128+0000 7f3ed099c700 1 --2- 172.21.15.27:0/1286619087 >> v2:172.21.15.77:3300/0 conn(0x7f3ecc069540 0x7f3ecc162270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:23.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.128+0000 7f3ecaffd700 1 -- 172.21.15.27:0/1286619087 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771452097 0 0) 0x7f3ecc0699d0 con 0x7f3ecc1577e0 2024-01-11T08:23:23.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.128+0000 7f3ecaffd700 1 -- 172.21.15.27:0/1286619087 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eb4003570 con 0x7f3ecc1577e0 2024-01-11T08:23:23.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.128+0000 7f3ecaffd700 1 -- 172.21.15.27:0/1286619087 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064832854 0 0) 0x7f3ecc069cb0 con 0x7f3ecc157410 2024-01-11T08:23:23.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.128+0000 7f3ecaffd700 1 -- 172.21.15.27:0/1286619087 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ecc0699d0 con 0x7f3ecc157410 2024-01-11T08:23:23.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.128+0000 7f3ed099c700 1 -- 172.21.15.27:0/1286619087 >> v1:172.21.15.77:6789/0 conn(0x7f3ecc157410 legacy=0x7f3ecc0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:23.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.128+0000 7f3ed099c700 1 -- 172.21.15.27:0/1286619087 >> v1:172.21.15.27:6789/0 conn(0x7f3ecc1577e0 legacy=0x7f3ecc0ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:23.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.128+0000 7f3ed099c700 1 -- 172.21.15.27:0/1286619087 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ecc06a270 con 0x7f3ecc069540 2024-01-11T08:23:23.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.129+0000 7f3ed099c700 1 --2- 172.21.15.27:0/1286619087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ecc069540 0x7f3ecc162270 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009e60 tx=0x7f3ec4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3e0f13dfa7c31ce server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:23.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.129+0000 7f3ecaffd700 1 -- 172.21.15.27:0/1286619087 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec400e070 con 0x7f3ecc069540 2024-01-11T08:23:23.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.129+0000 7f3ecaffd700 1 -- 172.21.15.27:0/1286619087 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec400bab0 con 0x7f3ecc069540 2024-01-11T08:23:23.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.129+0000 7f3ecaffd700 1 -- 172.21.15.27:0/1286619087 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec400be70 con 0x7f3ecc069540 2024-01-11T08:23:23.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.129+0000 7f3ed23ff700 1 -- 172.21.15.27:0/1286619087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ecc069540 msgr2=0x7f3ecc162270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:23.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.129+0000 7f3ed23ff700 1 --2- 172.21.15.27:0/1286619087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ecc069540 0x7f3ecc162270 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009e60 tx=0x7f3ec4009780 comp rx=0 tx=0).stop 2024-01-11T08:23:23.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.129+0000 7f3ed23ff700 1 -- 172.21.15.27:0/1286619087 shutdown_connections 2024-01-11T08:23:23.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.129+0000 7f3ed23ff700 1 --2- 172.21.15.27:0/1286619087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ecc069540 0x7f3ecc162270 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:23.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.129+0000 7f3ed23ff700 1 -- 172.21.15.27:0/1286619087 >> 172.21.15.27:0/1286619087 conn(0x7f3ecc0680a0 msgr2=0x7f3ecc0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:23.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.129+0000 7f3ed23ff700 1 -- 172.21.15.27:0/1286619087 shutdown_connections 2024-01-11T08:23:23.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.130+0000 7f3ed23ff700 1 -- 172.21.15.27:0/1286619087 wait complete. 2024-01-11T08:23:23.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.131+0000 7f3ed23ff700 1 Processor -- start 2024-01-11T08:23:23.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.131+0000 7f3ed23ff700 1 -- start start 2024-01-11T08:23:23.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.132+0000 7f3ed23ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3ecc069540 0x7f3ecc146d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:23.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.132+0000 7f3ed23ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ecc157410 0x7f3ecc15bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:23.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.132+0000 7f3ed23ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ecc1577e0 0x7f3ecc15c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:23.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.132+0000 7f3ed23ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc16d8d0 con 0x7f3ecc157410 2024-01-11T08:23:23.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.132+0000 7f3ed23ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc0ca3b0 con 0x7f3ecc1577e0 2024-01-11T08:23:23.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.132+0000 7f3ed23ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc147240 con 0x7f3ecc069540 2024-01-11T08:23:23.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.132+0000 7f3ecb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ecc157410 0x7f3ecc15bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:23.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.132+0000 7f3ecb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ecc157410 0x7f3ecc15bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38300/0 (socket says 172.21.15.27:38300) 2024-01-11T08:23:23.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.132+0000 7f3ecb7fe700 1 -- 172.21.15.27:0/470000992 learned_addr learned my addr 172.21.15.27:0/470000992 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:23.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.132+0000 7f3ecbfff700 1 --2- 172.21.15.27:0/470000992 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3ecc069540 0x7f3ecc146d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:23.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.133+0000 7f3ed099c700 1 --2- 172.21.15.27:0/470000992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ecc1577e0 0x7f3ecc15c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:23.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.133+0000 7f3ecb7fe700 1 -- 172.21.15.27:0/470000992 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3ecc069540 msgr2=0x7f3ecc146d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:23.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.133+0000 7f3ecb7fe700 1 --2- 172.21.15.27:0/470000992 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3ecc069540 0x7f3ecc146d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:23.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.133+0000 7f3ecb7fe700 1 -- 172.21.15.27:0/470000992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ecc1577e0 msgr2=0x7f3ecc15c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.133+0000 7f3ecb7fe700 1 --2- 172.21.15.27:0/470000992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ecc1577e0 0x7f3ecc15c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.133+0000 7f3ecb7fe700 1 -- 172.21.15.27:0/470000992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec4009a20 con 0x7f3ecc157410 2024-01-11T08:23:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.133+0000 7f3ecbfff700 1 --2- 172.21.15.27:0/470000992 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3ecc069540 0x7f3ecc146d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:23:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.133+0000 7f3ed099c700 1 --2- 172.21.15.27:0/470000992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ecc1577e0 0x7f3ecc15c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:23:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.133+0000 7f3ecb7fe700 1 --2- 172.21.15.27:0/470000992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ecc157410 0x7f3ecc15bfc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000c5e0 tx=0x7f3ec0007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.133+0000 7f3eb3fff700 1 -- 172.21.15.27:0/470000992 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0005b30 con 0x7f3ecc157410 2024-01-11T08:23:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.133+0000 7f3eb3fff700 1 -- 172.21.15.27:0/470000992 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec0005c90 con 0x7f3ecc157410 2024-01-11T08:23:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.133+0000 7f3eb3fff700 1 -- 172.21.15.27:0/470000992 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0007300 con 0x7f3ecc157410 2024-01-11T08:23:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.133+0000 7f3ed23ff700 1 -- 172.21.15.27:0/470000992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ecc16d440 con 0x7f3ecc157410 2024-01-11T08:23:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.133+0000 7f3ed23ff700 1 -- 172.21.15.27:0/470000992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ecc16cdf0 con 0x7f3ecc157410 2024-01-11T08:23:23.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.135+0000 7f3ed23ff700 1 -- 172.21.15.27:0/470000992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ecc04c3c0 con 0x7f3ecc157410 2024-01-11T08:23:23.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.136+0000 7f3eb3fff700 1 -- 172.21.15.27:0/470000992 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f3ec001b3b0 con 0x7f3ecc157410 2024-01-11T08:23:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.137+0000 7f3eb3fff700 1 --2- 172.21.15.27:0/470000992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3eb8066dd0 0x7f3eb8069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.137+0000 7f3eb3fff700 1 -- 172.21.15.27:0/470000992 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f3ec00838a0 con 0x7f3ecc157410 2024-01-11T08:23:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.137+0000 7f3ecbfff700 1 --2- 172.21.15.27:0/470000992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3eb8066dd0 0x7f3eb8069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:23.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.138+0000 7f3ecbfff700 1 --2- 172.21.15.27:0/470000992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3eb8066dd0 0x7f3eb8069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc003980 tx=0x7f3ebc009040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:23.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.141+0000 7f3eb3fff700 1 -- 172.21.15.27:0/470000992 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3ec0050cc0 con 0x7f3ecc157410 2024-01-11T08:23:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.369+0000 7f3ed23ff700 1 -- 172.21.15.27:0/470000992 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "2m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ecc0c2640 con 0x7f3eb8066dd0 2024-01-11T08:23:23.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.428+0000 7f3eb3fff700 1 -- 172.21.15.27:0/470000992 <== mgr.5791 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3ecc0c2640 con 0x7f3eb8066dd0 2024-01-11T08:23:23.428 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path snap_test_dir1 2024-01-11T08:23:23.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.430+0000 7f3ed23ff700 1 -- 172.21.15.27:0/470000992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3eb8066dd0 msgr2=0x7f3eb8069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:23.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.430+0000 7f3ed23ff700 1 --2- 172.21.15.27:0/470000992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3eb8066dd0 0x7f3eb8069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc003980 tx=0x7f3ebc009040 comp rx=0 tx=0).stop 2024-01-11T08:23:23.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.430+0000 7f3ed23ff700 1 -- 172.21.15.27:0/470000992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ecc157410 msgr2=0x7f3ecc15bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:23.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.430+0000 7f3ed23ff700 1 --2- 172.21.15.27:0/470000992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ecc157410 0x7f3ecc15bfc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000c5e0 tx=0x7f3ec0007e40 comp rx=0 tx=0).stop 2024-01-11T08:23:23.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.431+0000 7f3ed23ff700 1 -- 172.21.15.27:0/470000992 shutdown_connections 2024-01-11T08:23:23.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.431+0000 7f3ed23ff700 1 --2- 172.21.15.27:0/470000992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3eb8066dd0 0x7f3eb8069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:23.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.431+0000 7f3ed23ff700 1 --2- 172.21.15.27:0/470000992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ecc1577e0 0x7f3ecc15c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:23.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.431+0000 7f3ed23ff700 1 --2- 172.21.15.27:0/470000992 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3ecc069540 0x7f3ecc146d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:23.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.431+0000 7f3ed23ff700 1 --2- 172.21.15.27:0/470000992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ecc157410 0x7f3ecc15bfc0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:23.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.431+0000 7f3ed23ff700 1 -- 172.21.15.27:0/470000992 >> 172.21.15.27:0/470000992 conn(0x7f3ecc0680a0 msgr2=0x7f3ecc152b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:23.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.431+0000 7f3ed23ff700 1 -- 172.21.15.27:0/470000992 shutdown_connections 2024-01-11T08:23:23.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.431+0000 7f3ed23ff700 1 -- 172.21.15.27:0/470000992 wait complete. 2024-01-11T08:23:23.448 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-01-11T08:23:23.448 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-01-11-08_24_00 in ~37s... 2024-01-11T08:23:23.448 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-01-11-08_25_00 in ~97s... 2024-01-11T08:23:23.448 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m', '2m'], expected_retention: [] 2024-01-11T08:23:23.448 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-01-11T08:23:23.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.651+0000 7f9824fff700 1 Processor -- start 2024-01-11T08:23:23.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.651+0000 7f9824fff700 1 -- start start 2024-01-11T08:23:23.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.652+0000 7f9824fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f982013fbb0 0x7f98200ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:23.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.652+0000 7f9824fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f982013ff80 0x7f982015a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:23.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.652+0000 7f9824fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98200c6b30 con 0x7f98200cace0 2024-01-11T08:23:23.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.652+0000 7f9824fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98200c6800 con 0x7f982013ff80 2024-01-11T08:23:23.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.652+0000 7f9824fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98200cb240 con 0x7f982013fbb0 2024-01-11T08:23:23.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.653+0000 7f981ed9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f98200cace0 0x7f982006e6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58752/0 (socket says 172.21.15.27:58752) 2024-01-11T08:23:23.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.653+0000 7f981ed9d700 1 -- 172.21.15.27:0/2145657687 learned_addr learned my addr 172.21.15.27:0/2145657687 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:23:23.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.653+0000 7f981dd9b700 1 --2- 172.21.15.27:0/2145657687 >> v2:172.21.15.77:3300/0 conn(0x7f982013ff80 0x7f982015a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:23.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.653+0000 7f981e59c700 1 --2- 172.21.15.27:0/2145657687 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f982013fbb0 0x7f98200ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:23.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.653+0000 7f981d59a700 1 -- 172.21.15.27:0/2145657687 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2961318975 0 0) 0x7f98200c6b30 con 0x7f98200cace0 2024-01-11T08:23:23.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.653+0000 7f981d59a700 1 -- 172.21.15.27:0/2145657687 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9808003570 con 0x7f98200cace0 2024-01-11T08:23:23.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.653+0000 7f981dd9b700 1 -- 172.21.15.27:0/2145657687 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f982013fbb0 msgr2=0x7f98200ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:23:23.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.653+0000 7f981dd9b700 1 --2- 172.21.15.27:0/2145657687 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f982013fbb0 0x7f98200ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:23.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.653+0000 7f981dd9b700 1 -- 172.21.15.27:0/2145657687 >> v1:172.21.15.27:6789/0 conn(0x7f98200cace0 legacy=0x7f982006e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:23.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.653+0000 7f981dd9b700 1 -- 172.21.15.27:0/2145657687 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98200cb520 con 0x7f982013ff80 2024-01-11T08:23:23.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.653+0000 7f981e59c700 1 --2- 172.21.15.27:0/2145657687 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f982013fbb0 0x7f98200ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:23:23.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.654+0000 7f981dd9b700 1 --2- 172.21.15.27:0/2145657687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f982013ff80 0x7f982015a020 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f98140098e0 tx=0x7f98140091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6831de9fa35abf9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:23.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.654+0000 7f981d59a700 1 -- 172.21.15.27:0/2145657687 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9814011630 con 0x7f982013ff80 2024-01-11T08:23:23.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.654+0000 7f981d59a700 1 -- 172.21.15.27:0/2145657687 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9814011790 con 0x7f982013ff80 2024-01-11T08:23:23.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.654+0000 7f981d59a700 1 -- 172.21.15.27:0/2145657687 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9814011b10 con 0x7f982013ff80 2024-01-11T08:23:23.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.655+0000 7f9824fff700 1 -- 172.21.15.27:0/2145657687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f982013ff80 msgr2=0x7f982015a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:23.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.655+0000 7f9824fff700 1 --2- 172.21.15.27:0/2145657687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f982013ff80 0x7f982015a020 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f98140098e0 tx=0x7f98140091a0 comp rx=0 tx=0).stop 2024-01-11T08:23:23.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.655+0000 7f9824fff700 1 -- 172.21.15.27:0/2145657687 shutdown_connections 2024-01-11T08:23:23.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.655+0000 7f9824fff700 1 --2- 172.21.15.27:0/2145657687 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f982013fbb0 0x7f98200ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:23.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.655+0000 7f9824fff700 1 --2- 172.21.15.27:0/2145657687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f982013ff80 0x7f982015a020 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:23.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.655+0000 7f9824fff700 1 -- 172.21.15.27:0/2145657687 >> 172.21.15.27:0/2145657687 conn(0x7f98200680a0 msgr2=0x7f98200cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:23.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.655+0000 7f9824fff700 1 -- 172.21.15.27:0/2145657687 shutdown_connections 2024-01-11T08:23:23.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.655+0000 7f9824fff700 1 -- 172.21.15.27:0/2145657687 wait complete. 2024-01-11T08:23:23.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.656+0000 7f9824fff700 1 Processor -- start 2024-01-11T08:23:23.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.656+0000 7f9824fff700 1 -- start start 2024-01-11T08:23:23.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.657+0000 7f9824fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98200cace0 0x7f98200c27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:23.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.657+0000 7f9824fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f982013fbb0 0x7f98200c5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:23.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.657+0000 7f9824fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f982013ff80 0x7f982016a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:23.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.657+0000 7f9824fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98200c1dc0 con 0x7f98200cace0 2024-01-11T08:23:23.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.657+0000 7f9824fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98200c1520 con 0x7f982013fbb0 2024-01-11T08:23:23.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.657+0000 7f9824fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98200c1800 con 0x7f982013ff80 2024-01-11T08:23:23.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.658+0000 7f981e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98200cace0 0x7f98200c27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:23.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.658+0000 7f981e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98200cace0 0x7f98200c27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38302/0 (socket says 172.21.15.27:38302) 2024-01-11T08:23:23.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.658+0000 7f981e59c700 1 -- 172.21.15.27:0/2850591800 learned_addr learned my addr 172.21.15.27:0/2850591800 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:23:23.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.658+0000 7f981dd9b700 1 --2- 172.21.15.27:0/2850591800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f982013fbb0 0x7f98200c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:23.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.658+0000 7f981ed9d700 1 --2- 172.21.15.27:0/2850591800 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f982013ff80 0x7f982016a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:23.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.658+0000 7f981e59c700 1 -- 172.21.15.27:0/2850591800 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f982013ff80 msgr2=0x7f982016a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:23.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.658+0000 7f981e59c700 1 --2- 172.21.15.27:0/2850591800 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f982013ff80 0x7f982016a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:23.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.658+0000 7f981e59c700 1 -- 172.21.15.27:0/2850591800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f982013fbb0 msgr2=0x7f98200c5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:23.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.658+0000 7f981e59c700 1 --2- 172.21.15.27:0/2850591800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f982013fbb0 0x7f98200c5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:23.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.658+0000 7f981e59c700 1 -- 172.21.15.27:0/2850591800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98100093c0 con 0x7f98200cace0 2024-01-11T08:23:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.658+0000 7f981ed9d700 1 --2- 172.21.15.27:0/2850591800 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f982013ff80 0x7f982016a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:23:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.658+0000 7f981e59c700 1 --2- 172.21.15.27:0/2850591800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98200cace0 0x7f98200c27a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f981000ec80 tx=0x7f9810009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.659+0000 7f980e7fc700 1 -- 172.21.15.27:0/2850591800 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810005a00 con 0x7f98200cace0 2024-01-11T08:23:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.659+0000 7f980e7fc700 1 -- 172.21.15.27:0/2850591800 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9810005b60 con 0x7f98200cace0 2024-01-11T08:23:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.659+0000 7f980e7fc700 1 -- 172.21.15.27:0/2850591800 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981000c9d0 con 0x7f98200cace0 2024-01-11T08:23:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.659+0000 7f9824fff700 1 -- 172.21.15.27:0/2850591800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9814009420 con 0x7f98200cace0 2024-01-11T08:23:23.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.659+0000 7f9824fff700 1 -- 172.21.15.27:0/2850591800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98201696b0 con 0x7f98200cace0 2024-01-11T08:23:23.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.660+0000 7f9824fff700 1 -- 172.21.15.27:0/2850591800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9820169a20 con 0x7f98200cace0 2024-01-11T08:23:23.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.661+0000 7f980e7fc700 1 -- 172.21.15.27:0/2850591800 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f981000cd80 con 0x7f98200cace0 2024-01-11T08:23:23.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.662+0000 7f980e7fc700 1 --2- 172.21.15.27:0/2850591800 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9800066dd0 0x7f9800069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:23:23.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.662+0000 7f980e7fc700 1 -- 172.21.15.27:0/2850591800 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5963+0+0 (secure 0 0 0) 0x7f9810084a40 con 0x7f98200cace0 2024-01-11T08:23:23.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.662+0000 7f981dd9b700 1 --2- 172.21.15.27:0/2850591800 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9800066dd0 0x7f9800069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:23:23.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.663+0000 7f981dd9b700 1 --2- 172.21.15.27:0/2850591800 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9800066dd0 0x7f9800069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f98140091e0 tx=0x7f9814017000 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:23:23.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.666+0000 7f980e7fc700 1 -- 172.21.15.27:0/2850591800 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9810051d90 con 0x7f98200cace0 2024-01-11T08:23:23.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.835+0000 7f980e7fc700 1 -- 172.21.15.27:0/2850591800 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 22) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f9810052160 con 0x7f98200cace0 2024-01-11T08:23:23.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.898+0000 7f9824fff700 1 -- 172.21.15.27:0/2850591800 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f982004c3c0 con 0x7f9800066dd0 2024-01-11T08:23:23.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.906+0000 7f980e7fc700 1 -- 172.21.15.27:0/2850591800 <== mgr.5791 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+76 (secure 0 0 0) 0x7f982004c3c0 con 0x7f9800066dd0 2024-01-11T08:23:23.905 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:23:23.905 INFO:teuthology.orchestra.run.smithi027.stdout:{"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2024-01-11T08:23:23.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.908+0000 7f9824fff700 1 -- 172.21.15.27:0/2850591800 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9800066dd0 msgr2=0x7f9800069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:23.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.908+0000 7f9824fff700 1 --2- 172.21.15.27:0/2850591800 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9800066dd0 0x7f9800069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f98140091e0 tx=0x7f9814017000 comp rx=0 tx=0).stop 2024-01-11T08:23:23.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.908+0000 7f9824fff700 1 -- 172.21.15.27:0/2850591800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98200cace0 msgr2=0x7f98200c27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:23:23.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.908+0000 7f9824fff700 1 --2- 172.21.15.27:0/2850591800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98200cace0 0x7f98200c27a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f981000ec80 tx=0x7f9810009120 comp rx=0 tx=0).stop 2024-01-11T08:23:23.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.908+0000 7f9824fff700 1 -- 172.21.15.27:0/2850591800 shutdown_connections 2024-01-11T08:23:23.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.908+0000 7f9824fff700 1 --2- 172.21.15.27:0/2850591800 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9800066dd0 0x7f9800069280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:23.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.908+0000 7f9824fff700 1 --2- 172.21.15.27:0/2850591800 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f982013ff80 0x7f982016a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:23.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.908+0000 7f9824fff700 1 --2- 172.21.15.27:0/2850591800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98200cace0 0x7f98200c27a0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:23.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.908+0000 7f9824fff700 1 --2- 172.21.15.27:0/2850591800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f982013fbb0 0x7f98200c5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:23:23.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.908+0000 7f9824fff700 1 -- 172.21.15.27:0/2850591800 >> 172.21.15.27:0/2850591800 conn(0x7f98200680a0 msgr2=0x7f98200cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:23:23.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.909+0000 7f9824fff700 1 -- 172.21.15.27:0/2850591800 shutdown_connections 2024-01-11T08:23:23.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:23:23.909+0000 7f9824fff700 1 -- 172.21.15.27:0/2850591800 wait complete. 2024-01-11T08:23:23.926 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2024-01-11T08:23:23.926 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m', '2m'], 'retention': [{}, {}]} 2024-01-11T08:23:23.926 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:23.926 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:23.976 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:23.976 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:23.976 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:24.977 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:24.977 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:25.034 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:25.034 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:25.034 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:26.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:26.035 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:26.091 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:26.091 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:26.091 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:27.092 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:27.092 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:27.148 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:27.148 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:27.148 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:28.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:28.149 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:28.207 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:28.207 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:28.207 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:29.208 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:29.208 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:29.264 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:29.264 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:29.264 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:30.265 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:30.265 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:30.320 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:30.321 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:30.321 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:31.322 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:31.322 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:31.378 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:31.378 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:31.378 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:32.379 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:32.379 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:32.435 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:32.435 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:32.435 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:33.436 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:33.437 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:33.492 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:33.492 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:33.492 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:34.493 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:34.494 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:34.549 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:34.549 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:34.549 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:35.550 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:35.550 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:35.606 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:35.606 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:35.606 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:36.607 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:36.607 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:36.663 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:36.663 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:36.663 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:37.664 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:37.664 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:37.719 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:37.719 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:37.720 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:38.721 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:38.721 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:38.776 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:38.776 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:38.776 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:39.777 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:39.777 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:39.833 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:39.833 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:39.833 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:40.834 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:40.834 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:40.889 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:40.889 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:40.889 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:41.890 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:41.890 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:41.946 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:41.946 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:41.946 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:42.947 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:42.947 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:43.003 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:43.003 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:43.003 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:44.004 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:44.004 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:44.059 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:44.059 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:44.060 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:45.060 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:45.061 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:45.116 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:45.117 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:45.117 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:46.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:46.118 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:46.173 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:46.173 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:46.173 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:47.174 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:47.174 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:47.230 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:47.230 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:47.230 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:48.231 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:48.232 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:48.286 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:48.286 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:48.286 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:49.287 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:49.288 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:49.343 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:49.343 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:49.343 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:50.344 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:50.344 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:50.400 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:50.400 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:50.400 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:51.401 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:51.401 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:51.456 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:51.456 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:51.456 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:51.907 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:23:51.912 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:23:52.457 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:52.492 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:52.547 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:52.547 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:52.547 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:53.548 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:53.548 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:53.603 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:53.603 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:53.603 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:54.604 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:54.604 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:54.660 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:54.660 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:54.660 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:55.661 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:55.661 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:55.717 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:55.717 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:55.717 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:56.718 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:56.718 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:56.773 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:56.773 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:56.773 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:57.774 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:57.774 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:57.830 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:57.830 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:57.830 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:58.831 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:58.831 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:58.886 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:58.886 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:58.886 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:23:59.887 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:23:59.888 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:23:59.943 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:23:59.943 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:23:59.943 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:00.440 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:24:00.440+0000 7f1f4881d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:24:00.440 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:24:00.440+0000 7f1f4881d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:24:00.440 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:24:00.440+0000 7f1f4881d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:24:00.440 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:24:00.440+0000 7f1f4881d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:24:00.440 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:24:00.440+0000 7f1f4881d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:24:00.944 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:00.945 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:00.999 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:01.000 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:01.000 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:01.000 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:01.000 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-01-11-08_24_00_UTC'] 2024-01-11T08:24:01.001 DEBUG:tasks.cephfs.test_snap_schedules:exec=1704961403.4482062, now = 1704961441.0010715, timo = 37 2024-01-11T08:24:02.002 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:02.002 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:02.056 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:02.057 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:02.057 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:02.057 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:03.058 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:03.058 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:03.112 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:03.113 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:03.113 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:03.113 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:04.114 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:04.114 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:04.169 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:04.170 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:04.170 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:04.170 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:05.171 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:05.171 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:05.225 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:05.226 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:05.227 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:05.227 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:06.227 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:06.227 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:06.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:06.283 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:06.283 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:06.283 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:07.284 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:07.284 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:07.338 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:07.339 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:07.339 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:07.339 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:08.340 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:08.341 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:08.396 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:08.397 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:08.397 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:08.398 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:09.398 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:09.398 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:09.452 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:09.453 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:09.454 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:09.454 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:10.454 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:10.455 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:10.505 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:10.506 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:10.506 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:10.507 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:11.507 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:11.508 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:11.562 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:11.563 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:11.563 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:11.563 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:12.565 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:12.565 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:12.619 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:12.620 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:12.620 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:12.620 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:13.622 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:13.622 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:13.676 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:13.677 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:13.677 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:13.677 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:14.677 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:14.678 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:14.731 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:14.732 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:14.732 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:14.733 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:15.733 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:15.734 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:15.788 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:15.790 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:15.790 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:15.790 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:16.791 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:16.791 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:16.845 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:16.846 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:16.846 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:16.846 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:17.847 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:17.847 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:17.905 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:17.906 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:17.906 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:17.906 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:18.907 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:18.907 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:18.961 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:18.962 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:18.962 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:18.962 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:19.963 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:19.963 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:20.017 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:20.018 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:20.018 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:20.018 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:21.019 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:21.019 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:21.073 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:21.074 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:21.074 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:21.074 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:22.070 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:24:22.074 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:22.075 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:22.075 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:24:22.130 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:22.131 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:22.131 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:22.132 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:23.132 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:23.132 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:23.187 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:23.188 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:23.188 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:23.188 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:24.190 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:24.190 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:24.244 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:24.245 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:24.245 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:24.245 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:25.246 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:25.246 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:25.300 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:25.301 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:25.302 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:25.302 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:26.302 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:26.303 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:26.357 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:26.358 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:26.358 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:26.358 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:27.359 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:27.359 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:27.414 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:27.415 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:27.415 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:27.415 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:28.415 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:28.416 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:28.472 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:28.473 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:28.473 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:28.473 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:29.473 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:29.474 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:29.527 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:29.529 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:29.529 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:29.529 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:30.530 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:30.530 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:30.584 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:30.585 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:30.585 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:30.585 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:31.586 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:31.586 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:31.640 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:31.641 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:31.641 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:31.641 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:32.642 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:32.642 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:32.696 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:32.697 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:32.697 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:32.697 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:33.698 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:33.698 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:33.751 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:33.753 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:33.753 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:33.753 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:34.754 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:34.754 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:34.808 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:34.809 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:34.809 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:34.809 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:35.809 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:35.810 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:35.864 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:35.865 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:35.865 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:35.865 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:36.866 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:36.866 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:36.920 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:36.921 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:36.921 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:36.921 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:37.922 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:37.922 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:37.976 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:37.977 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:37.977 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:37.977 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:38.978 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:38.978 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:39.031 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:39.032 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:39.032 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:39.032 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:40.034 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:40.034 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:40.088 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:40.089 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:40.089 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:40.089 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:41.090 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:41.090 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:41.144 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:41.145 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:41.145 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:41.145 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:42.147 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:42.147 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:42.201 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:42.202 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:42.202 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:42.202 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:43.203 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:43.203 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:43.257 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:43.258 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:43.258 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:43.259 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:44.259 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:44.260 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:44.314 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:44.315 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:44.316 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:44.316 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:45.319 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:45.319 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:45.373 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:45.374 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:45.374 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:45.374 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:46.376 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:46.376 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:46.430 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:46.431 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:46.431 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:46.431 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:47.432 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:47.432 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:47.486 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:47.487 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:47.487 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:47.487 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:48.488 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:48.488 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:48.542 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:48.543 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:48.543 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:48.543 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:49.544 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:49.544 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:49.599 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:49.600 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:49.600 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:49.600 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:50.601 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:50.601 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:50.655 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:50.657 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:50.657 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:50.657 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:51.658 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:51.658 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:51.712 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:51.713 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:51.713 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:51.713 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:52.207 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:24:52.213 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:24:52.714 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:52.714 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:52.768 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:52.769 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:52.769 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:52.769 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:53.770 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:53.770 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:53.824 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:53.825 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:53.826 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:53.826 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:54.827 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:54.827 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:54.884 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:54.884 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:54.884 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:54.885 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:55.886 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:55.886 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:55.941 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:55.942 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:55.942 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:55.942 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:56.944 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:56.944 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:56.997 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:56.998 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:56.999 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:56.999 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:57.999 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:57.999 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:58.053 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:58.054 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:58.054 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:58.054 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:24:59.054 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:24:59.055 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:24:59.109 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:24:59.110 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:24:59.110 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:24:59.110 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:25:00.111 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:25:00.111 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:25:00.166 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:25:00.167 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC'} 2024-01-11T08:25:00.168 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:25:00.168 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:25:01.169 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:25:01.169 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:25:01.223 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:25:01.223 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_25_00_UTC 2024-01-11T08:25:01.224 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_24_00_UTC', 'scheduled-2024-01-11-08_25_00_UTC'} 2024-01-11T08:25:01.224 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-01-11-08_25_00_UTC'} 2024-01-11T08:25:01.224 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:25:01.224 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-01-11-08_25_00_UTC'] 2024-01-11T08:25:01.224 DEBUG:tasks.cephfs.test_snap_schedules:exec=1704961403.4482062, now = 1704961501.224817, timo = 97 2024-01-11T08:25:02.226 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-01-11T08:25:02.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.438+0000 7f28171ff700 1 Processor -- start 2024-01-11T08:25:02.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.438+0000 7f28171ff700 1 -- start start 2024-01-11T08:25:02.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.439+0000 7f28171ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2810142dc0 0x7f28100cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:02.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.440+0000 7f28171ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28100cb2e0 0x7f28100cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:02.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.440+0000 7f28171ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28100ccc70 con 0x7f2810143190 2024-01-11T08:25:02.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.440+0000 7f28171ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f281015df30 con 0x7f2810142dc0 2024-01-11T08:25:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.440+0000 7f28171ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28100cb740 con 0x7f28100cb2e0 2024-01-11T08:25:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.440+0000 7f2814f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2810142dc0 0x7f28100cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.440+0000 7f280ffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2810143190 0x7f28100cc020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53158/0 (socket says 172.21.15.27:53158) 2024-01-11T08:25:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.440+0000 7f280ffff700 1 -- 172.21.15.27:0/123808548 learned_addr learned my addr 172.21.15.27:0/123808548 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.440+0000 7f281579c700 1 --2- 172.21.15.27:0/123808548 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28100cb2e0 0x7f28100cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.440+0000 7f2814f9b700 1 -- 172.21.15.27:0/123808548 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28100cb2e0 msgr2=0x7f28100cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.440+0000 7f2814f9b700 1 --2- 172.21.15.27:0/123808548 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28100cb2e0 0x7f28100cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.440+0000 7f2814f9b700 1 -- 172.21.15.27:0/123808548 >> v1:172.21.15.27:6789/0 conn(0x7f2810143190 legacy=0x7f28100cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.440+0000 7f280f7fe700 1 -- 172.21.15.27:0/123808548 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178979188 0 0) 0x7f28100ccc70 con 0x7f2810143190 2024-01-11T08:25:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.440+0000 7f2814f9b700 1 -- 172.21.15.27:0/123808548 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281015d320 con 0x7f2810142dc0 2024-01-11T08:25:02.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.441+0000 7f2814f9b700 1 --2- 172.21.15.27:0/123808548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2810142dc0 0x7f28100cada0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f28040098e0 tx=0x7f28040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d99da3429240bd2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:02.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.441+0000 7f280f7fe700 1 -- 172.21.15.27:0/123808548 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2804009e80 con 0x7f2810142dc0 2024-01-11T08:25:02.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.441+0000 7f280f7fe700 1 -- 172.21.15.27:0/123808548 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280400b860 con 0x7f2810142dc0 2024-01-11T08:25:02.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.441+0000 7f280f7fe700 1 -- 172.21.15.27:0/123808548 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280400bbe0 con 0x7f2810142dc0 2024-01-11T08:25:02.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.441+0000 7f28171ff700 1 -- 172.21.15.27:0/123808548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2810142dc0 msgr2=0x7f28100cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:02.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.441+0000 7f28171ff700 1 --2- 172.21.15.27:0/123808548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2810142dc0 0x7f28100cada0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f28040098e0 tx=0x7f28040091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:02.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.441+0000 7f28171ff700 1 -- 172.21.15.27:0/123808548 shutdown_connections 2024-01-11T08:25:02.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.441+0000 7f28171ff700 1 --2- 172.21.15.27:0/123808548 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28100cb2e0 0x7f28100cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:02.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.441+0000 7f28171ff700 1 --2- 172.21.15.27:0/123808548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2810142dc0 0x7f28100cada0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:02.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.442+0000 7f28171ff700 1 -- 172.21.15.27:0/123808548 >> 172.21.15.27:0/123808548 conn(0x7f2810063ca0 msgr2=0x7f281006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:02.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.442+0000 7f28171ff700 1 -- 172.21.15.27:0/123808548 shutdown_connections 2024-01-11T08:25:02.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.442+0000 7f28171ff700 1 -- 172.21.15.27:0/123808548 wait complete. 2024-01-11T08:25:02.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.443+0000 7f28171ff700 1 Processor -- start 2024-01-11T08:25:02.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.443+0000 7f28171ff700 1 -- start start 2024-01-11T08:25:02.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.444+0000 7f28171ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28100cb2e0 0x7f2810157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:02.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.444+0000 7f28171ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2810142dc0 0x7f2810157db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:02.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.444+0000 7f28171ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2810143190 0x7f2810158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:02.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.444+0000 7f28171ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28101588c0 con 0x7f2810142dc0 2024-01-11T08:25:02.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.444+0000 7f28171ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2810154640 con 0x7f2810143190 2024-01-11T08:25:02.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.444+0000 7f28171ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28101585e0 con 0x7f28100cb2e0 2024-01-11T08:25:02.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.444+0000 7f280ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2810142dc0 0x7f2810157db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:02.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.444+0000 7f280ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2810142dc0 0x7f2810157db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55882/0 (socket says 172.21.15.27:55882) 2024-01-11T08:25:02.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.444+0000 7f280ffff700 1 -- 172.21.15.27:0/2150901028 learned_addr learned my addr 172.21.15.27:0/2150901028 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:02.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.445+0000 7f281579c700 1 --2- 172.21.15.27:0/2150901028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2810143190 0x7f2810158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:02.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.445+0000 7f2814f9b700 1 --2- 172.21.15.27:0/2150901028 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28100cb2e0 0x7f2810157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:02.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.445+0000 7f280ffff700 1 -- 172.21.15.27:0/2150901028 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28100cb2e0 msgr2=0x7f2810157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:02.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.445+0000 7f280ffff700 1 --2- 172.21.15.27:0/2150901028 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28100cb2e0 0x7f2810157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:02.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.445+0000 7f280ffff700 1 -- 172.21.15.27:0/2150901028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2810143190 msgr2=0x7f2810158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:02.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.445+0000 7f280ffff700 1 --2- 172.21.15.27:0/2150901028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2810143190 0x7f2810158ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:02.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.445+0000 7f280ffff700 1 -- 172.21.15.27:0/2150901028 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2804009420 con 0x7f2810142dc0 2024-01-11T08:25:02.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.445+0000 7f280ffff700 1 --2- 172.21.15.27:0/2150901028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2810142dc0 0x7f2810157db0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00c5b0 tx=0x7f27fc001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:02.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.445+0000 7f280cff9700 1 -- 172.21.15.27:0/2150901028 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27fc005b30 con 0x7f2810142dc0 2024-01-11T08:25:02.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.445+0000 7f280cff9700 1 -- 172.21.15.27:0/2150901028 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27fc005c90 con 0x7f2810142dc0 2024-01-11T08:25:02.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.446+0000 7f28171ff700 1 -- 172.21.15.27:0/2150901028 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2810153ff0 con 0x7f2810142dc0 2024-01-11T08:25:02.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.446+0000 7f280cff9700 1 -- 172.21.15.27:0/2150901028 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27fc007320 con 0x7f2810142dc0 2024-01-11T08:25:02.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.446+0000 7f28171ff700 1 -- 172.21.15.27:0/2150901028 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28101582f0 con 0x7f2810142dc0 2024-01-11T08:25:02.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.447+0000 7f28171ff700 1 -- 172.21.15.27:0/2150901028 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28101547b0 con 0x7f2810142dc0 2024-01-11T08:25:02.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.448+0000 7f280cff9700 1 -- 172.21.15.27:0/2150901028 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 84873+0+0 (secure 0 0 0) 0x7f27fc01b3b0 con 0x7f2810142dc0 2024-01-11T08:25:02.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.449+0000 7f280cff9700 1 --2- 172.21.15.27:0/2150901028 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f27f40670a0 0x7f27f4069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:02.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.449+0000 7f280cff9700 1 -- 172.21.15.27:0/2150901028 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6021+0+0 (secure 0 0 0) 0x7f27fc083ce0 con 0x7f2810142dc0 2024-01-11T08:25:02.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.449+0000 7f2814f9b700 1 --2- 172.21.15.27:0/2150901028 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f27f40670a0 0x7f27f4069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:02.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.450+0000 7f2814f9b700 1 --2- 172.21.15.27:0/2150901028 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f27f40670a0 0x7f27f4069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f28040091e0 tx=0x7f2804017000 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:02.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.454+0000 7f280cff9700 1 -- 172.21.15.27:0/2150901028 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f27fc050c60 con 0x7f2810142dc0 2024-01-11T08:25:02.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.682+0000 7f28171ff700 1 -- 172.21.15.27:0/2150901028 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f281004c3c0 con 0x7f27f40670a0 2024-01-11T08:25:02.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.761+0000 7f280cff9700 1 -- 172.21.15.27:0/2150901028 <== mgr.5791 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f281004c3c0 con 0x7f27f40670a0 2024-01-11T08:25:02.760 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule removed for path snap_test_dir1 2024-01-11T08:25:02.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.764+0000 7f28171ff700 1 -- 172.21.15.27:0/2150901028 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f27f40670a0 msgr2=0x7f27f4069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:02.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.764+0000 7f28171ff700 1 --2- 172.21.15.27:0/2150901028 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f27f40670a0 0x7f27f4069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f28040091e0 tx=0x7f2804017000 comp rx=0 tx=0).stop 2024-01-11T08:25:02.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.764+0000 7f28171ff700 1 -- 172.21.15.27:0/2150901028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2810142dc0 msgr2=0x7f2810157db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:02.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.764+0000 7f28171ff700 1 --2- 172.21.15.27:0/2150901028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2810142dc0 0x7f2810157db0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00c5b0 tx=0x7f27fc001070 comp rx=0 tx=0).stop 2024-01-11T08:25:02.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.764+0000 7f28171ff700 1 -- 172.21.15.27:0/2150901028 shutdown_connections 2024-01-11T08:25:02.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.764+0000 7f28171ff700 1 --2- 172.21.15.27:0/2150901028 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f27f40670a0 0x7f27f4069550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:02.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.764+0000 7f28171ff700 1 --2- 172.21.15.27:0/2150901028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2810143190 0x7f2810158ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:02.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.764+0000 7f28171ff700 1 --2- 172.21.15.27:0/2150901028 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28100cb2e0 0x7f2810157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:02.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.764+0000 7f28171ff700 1 --2- 172.21.15.27:0/2150901028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2810142dc0 0x7f2810157db0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:02.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.764+0000 7f28171ff700 1 -- 172.21.15.27:0/2150901028 >> 172.21.15.27:0/2150901028 conn(0x7f2810063ca0 msgr2=0x7f2810141df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:02.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.764+0000 7f28171ff700 1 -- 172.21.15.27:0/2150901028 shutdown_connections 2024-01-11T08:25:02.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:02.764+0000 7f28171ff700 1 -- 172.21.15.27:0/2150901028 wait complete. 2024-01-11T08:25:02.781 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-01-11T08:25:02.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:25:02.782 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:25:02.830 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:25:02.830 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_25_00_UTC 2024-01-11T08:25:02.831 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_24_00_UTC 2024-01-11T08:25:02.831 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:25:02.831 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_24_00_UTC) 2024-01-11T08:25:02.888 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_25_00_UTC 2024-01-11T08:25:02.888 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:25:02.888 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_25_00_UTC) 2024-01-11T08:25:02.944 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:25:02.944 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-01-11T08:25:02.995 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-11T08:25:03.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.197+0000 7f6971bff700 1 Processor -- start 2024-01-11T08:25:03.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.197+0000 7f6971bff700 1 -- start start 2024-01-11T08:25:03.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.198+0000 7f6971bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f696c142dc0 0x7f696c0cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:03.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.198+0000 7f6971bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f696c143190 0x7f696c0cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:03.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.198+0000 7f6971bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f696c0cc020 0x7f696c0cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:03.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.198+0000 7f6971bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696c15df30 con 0x7f696c0cc020 2024-01-11T08:25:03.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.198+0000 7f6971bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696c15d320 con 0x7f696c143190 2024-01-11T08:25:03.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.198+0000 7f6971bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696c15d590 con 0x7f696c142dc0 2024-01-11T08:25:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.198+0000 7f696bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f696c0cc020 0x7f696c0cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.198+0000 7f696bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f696c0cc020 0x7f696c0cc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55888/0 (socket says 172.21.15.27:55888) 2024-01-11T08:25:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.198+0000 7f696bfff700 1 -- 172.21.15.27:0/589096324 learned_addr learned my addr 172.21.15.27:0/589096324 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.199+0000 7f696b7fe700 1 --2- 172.21.15.27:0/589096324 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f696c142dc0 0x7f696c0cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.199+0000 7f696affd700 1 --2- 172.21.15.27:0/589096324 >> v2:172.21.15.77:3300/0 conn(0x7f696c143190 0x7f696c0cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.199+0000 7f696bfff700 1 -- 172.21.15.27:0/589096324 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f696c142dc0 msgr2=0x7f696c0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.199+0000 7f696bfff700 1 --2- 172.21.15.27:0/589096324 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f696c142dc0 0x7f696c0cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.199+0000 7f696bfff700 1 -- 172.21.15.27:0/589096324 >> v2:172.21.15.77:3300/0 conn(0x7f696c143190 msgr2=0x7f696c0cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.199+0000 7f696bfff700 1 --2- 172.21.15.27:0/589096324 >> v2:172.21.15.77:3300/0 conn(0x7f696c143190 0x7f696c0cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.199+0000 7f696bfff700 1 -- 172.21.15.27:0/589096324 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c15d870 con 0x7f696c0cc020 2024-01-11T08:25:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.199+0000 7f696bfff700 1 --2- 172.21.15.27:0/589096324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f696c0cc020 0x7f696c0cc440 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f696c0cf480 tx=0x7f6964009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9fce90eae60d74e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.199+0000 7f696a7fc700 1 -- 172.21.15.27:0/589096324 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696400e070 con 0x7f696c0cc020 2024-01-11T08:25:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.199+0000 7f696a7fc700 1 -- 172.21.15.27:0/589096324 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696400ba50 con 0x7f696c0cc020 2024-01-11T08:25:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.199+0000 7f696a7fc700 1 -- 172.21.15.27:0/589096324 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69640092e0 con 0x7f696c0cc020 2024-01-11T08:25:03.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.200+0000 7f6971bff700 1 -- 172.21.15.27:0/589096324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f696c0cc020 msgr2=0x7f696c0cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:03.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.200+0000 7f6971bff700 1 --2- 172.21.15.27:0/589096324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f696c0cc020 0x7f696c0cc440 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f696c0cf480 tx=0x7f6964009840 comp rx=0 tx=0).stop 2024-01-11T08:25:03.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.200+0000 7f6971bff700 1 -- 172.21.15.27:0/589096324 shutdown_connections 2024-01-11T08:25:03.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.200+0000 7f6971bff700 1 --2- 172.21.15.27:0/589096324 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f696c142dc0 0x7f696c0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:03.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.200+0000 7f6971bff700 1 --2- 172.21.15.27:0/589096324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f696c0cc020 0x7f696c0cc440 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:03.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.200+0000 7f6971bff700 1 --2- 172.21.15.27:0/589096324 >> v2:172.21.15.77:3300/0 conn(0x7f696c143190 0x7f696c0cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:03.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.200+0000 7f6971bff700 1 -- 172.21.15.27:0/589096324 >> 172.21.15.27:0/589096324 conn(0x7f696c063ca0 msgr2=0x7f696c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:03.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.200+0000 7f6971bff700 1 -- 172.21.15.27:0/589096324 shutdown_connections 2024-01-11T08:25:03.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.200+0000 7f6971bff700 1 -- 172.21.15.27:0/589096324 wait complete. 2024-01-11T08:25:03.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.201+0000 7f6971bff700 1 Processor -- start 2024-01-11T08:25:03.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.202+0000 7f6971bff700 1 -- start start 2024-01-11T08:25:03.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.202+0000 7f6971bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f696c0cc020 0x7f696c141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:03.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.202+0000 7f6971bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f696c142dc0 0x7f696c0c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:03.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.202+0000 7f6971bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f696c143190 0x7f696c0aa2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:03.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.202+0000 7f6971bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c1423f0 con 0x7f696c142dc0 2024-01-11T08:25:03.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.202+0000 7f6971bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c169c10 con 0x7f696c143190 2024-01-11T08:25:03.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.202+0000 7f6971bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696c0c7680 con 0x7f696c0cc020 2024-01-11T08:25:03.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.203+0000 7f696affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f696c142dc0 0x7f696c0c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:03.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.203+0000 7f696affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f696c142dc0 0x7f696c0c6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55894/0 (socket says 172.21.15.27:55894) 2024-01-11T08:25:03.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.203+0000 7f696affd700 1 -- 172.21.15.27:0/2236839773 learned_addr learned my addr 172.21.15.27:0/2236839773 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:03.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.203+0000 7f696bfff700 1 --2- 172.21.15.27:0/2236839773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f696c143190 0x7f696c0aa2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:03.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.203+0000 7f696b7fe700 1 --2- 172.21.15.27:0/2236839773 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f696c0cc020 0x7f696c141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:03.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.203+0000 7f696affd700 1 -- 172.21.15.27:0/2236839773 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f696c0cc020 msgr2=0x7f696c141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:03.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.203+0000 7f696affd700 1 --2- 172.21.15.27:0/2236839773 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f696c0cc020 0x7f696c141eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:03.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.203+0000 7f696affd700 1 -- 172.21.15.27:0/2236839773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f696c143190 msgr2=0x7f696c0aa2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:03.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.203+0000 7f696affd700 1 --2- 172.21.15.27:0/2236839773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f696c143190 0x7f696c0aa2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:03.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.203+0000 7f696affd700 1 -- 172.21.15.27:0/2236839773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6964009ae0 con 0x7f696c142dc0 2024-01-11T08:25:03.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.204+0000 7f696affd700 1 --2- 172.21.15.27:0/2236839773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f696c142dc0 0x7f696c0c6e40 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f696000d860 tx=0x7f6960008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:03.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.204+0000 7f69537fe700 1 -- 172.21.15.27:0/2236839773 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69600125f0 con 0x7f696c142dc0 2024-01-11T08:25:03.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.204+0000 7f69537fe700 1 -- 172.21.15.27:0/2236839773 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6960012750 con 0x7f696c142dc0 2024-01-11T08:25:03.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.204+0000 7f6971bff700 1 -- 172.21.15.27:0/2236839773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696c0c7960 con 0x7f696c142dc0 2024-01-11T08:25:03.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.204+0000 7f69537fe700 1 -- 172.21.15.27:0/2236839773 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6960012b10 con 0x7f696c142dc0 2024-01-11T08:25:03.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.204+0000 7f6971bff700 1 -- 172.21.15.27:0/2236839773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696c0c7f20 con 0x7f696c142dc0 2024-01-11T08:25:03.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.205+0000 7f6971bff700 1 -- 172.21.15.27:0/2236839773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696c169da0 con 0x7f696c142dc0 2024-01-11T08:25:03.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.206+0000 7f69537fe700 1 -- 172.21.15.27:0/2236839773 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 84873+0+0 (secure 0 0 0) 0x7f696001e460 con 0x7f696c142dc0 2024-01-11T08:25:03.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.207+0000 7f69537fe700 1 --2- 172.21.15.27:0/2236839773 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f69580670a0 0x7f6958069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:03.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.207+0000 7f69537fe700 1 -- 172.21.15.27:0/2236839773 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6021+0+0 (secure 0 0 0) 0x7f6960084d00 con 0x7f696c142dc0 2024-01-11T08:25:03.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.207+0000 7f696b7fe700 1 --2- 172.21.15.27:0/2236839773 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f69580670a0 0x7f6958069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:03.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.208+0000 7f696b7fe700 1 --2- 172.21.15.27:0/2236839773 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f69580670a0 0x7f6958069550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f695c006fd0 tx=0x7f695c008040 comp rx=0 tx=0).ready entity=mgr.5791 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:03.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.211+0000 7f69537fe700 1 -- 172.21.15.27:0/2236839773 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6960055790 con 0x7f696c142dc0 2024-01-11T08:25:03.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.470+0000 7f6971bff700 1 -- 172.21.15.27:0/2236839773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f696c04c3c0 con 0x7f696c142dc0 2024-01-11T08:25:03.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.740+0000 7f69537fe700 1 -- 172.21.15.27:0/2236839773 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f69600520d0 con 0x7f696c142dc0 2024-01-11T08:25:03.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.741+0000 7f69537fe700 1 -- 172.21.15.27:0/2236839773 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v26) v1 ==== 93+0+0 (secure 0 0 0) 0x7f6960055790 con 0x7f696c142dc0 2024-01-11T08:25:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.744+0000 7f6971bff700 1 -- 172.21.15.27:0/2236839773 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f69580670a0 msgr2=0x7f6958069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.744+0000 7f6971bff700 1 --2- 172.21.15.27:0/2236839773 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f69580670a0 0x7f6958069550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f695c006fd0 tx=0x7f695c008040 comp rx=0 tx=0).stop 2024-01-11T08:25:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.744+0000 7f6971bff700 1 -- 172.21.15.27:0/2236839773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f696c142dc0 msgr2=0x7f696c0c6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.744+0000 7f6971bff700 1 --2- 172.21.15.27:0/2236839773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f696c142dc0 0x7f696c0c6e40 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f696000d860 tx=0x7f6960008040 comp rx=0 tx=0).stop 2024-01-11T08:25:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.744+0000 7f6971bff700 1 -- 172.21.15.27:0/2236839773 shutdown_connections 2024-01-11T08:25:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.744+0000 7f6971bff700 1 --2- 172.21.15.27:0/2236839773 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f69580670a0 0x7f6958069550 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.744+0000 7f6971bff700 1 --2- 172.21.15.27:0/2236839773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f696c143190 0x7f696c0aa2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.744+0000 7f6971bff700 1 --2- 172.21.15.27:0/2236839773 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f696c0cc020 0x7f696c141eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.744+0000 7f6971bff700 1 --2- 172.21.15.27:0/2236839773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f696c142dc0 0x7f696c0c6e40 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:03.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.744+0000 7f6971bff700 1 -- 172.21.15.27:0/2236839773 >> 172.21.15.27:0/2236839773 conn(0x7f696c063ca0 msgr2=0x7f696c0b1520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:03.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.744+0000 7f6971bff700 1 -- 172.21.15.27:0/2236839773 shutdown_connections 2024-01-11T08:25:03.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:03.744+0000 7f6971bff700 1 -- 172.21.15.27:0/2236839773 wait complete. 2024-01-11T08:25:03.763 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T08:25:03.763 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:25:03.862 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:25:03.955 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:03.956+0000 7ff915edc1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:25:03.961 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T08:25:03.961 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-11T08:25:03.961 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:25:03.961 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-01-11T08:25:03.962 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:03.963+0000 7f1feb3201c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:25:04.049 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T08:25:04.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:25:04.049 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-11T08:25:04.123 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:25:04.142 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:04.143+0000 7f1feb3201c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:25:04.156 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:04.157+0000 7ff915edc1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:25:04.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.340+0000 7f3b919ff700 1 Processor -- start 2024-01-11T08:25:04.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.341+0000 7f3b919ff700 1 -- start start 2024-01-11T08:25:04.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.342+0000 7f3b919ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3b8c15a920 0x7f3b8c0b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:04.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.342+0000 7f3b919ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3b8c15acf0 0x7f3b8c0b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:04.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.342+0000 7f3b919ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b8c06ca90 con 0x7f3b8c06f010 2024-01-11T08:25:04.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.342+0000 7f3b919ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b8c0cc2c0 con 0x7f3b8c15acf0 2024-01-11T08:25:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.342+0000 7f3b919ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b8c06bea0 con 0x7f3b8c15a920 2024-01-11T08:25:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.342+0000 7f3b8a7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3b8c15acf0 0x7f3b8c0b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.342+0000 7f3b8b7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3b8c06f010 0x7f3b8c06f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44154/0 (socket says 172.21.15.27:44154) 2024-01-11T08:25:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.342+0000 7f3b8b7fe700 1 -- 172.21.15.27:0/2291003295 learned_addr learned my addr 172.21.15.27:0/2291003295 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.342+0000 7f3b8affd700 1 --2- 172.21.15.27:0/2291003295 >> v2:172.21.15.77:3300/0 conn(0x7f3b8c15a920 0x7f3b8c0b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.343+0000 7f3b8a7fc700 1 -- 172.21.15.27:0/2291003295 >> v2:172.21.15.77:3300/0 conn(0x7f3b8c15a920 msgr2=0x7f3b8c0b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.343+0000 7f3b8a7fc700 1 --2- 172.21.15.27:0/2291003295 >> v2:172.21.15.77:3300/0 conn(0x7f3b8c15a920 0x7f3b8c0b23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.343+0000 7f3b8a7fc700 1 -- 172.21.15.27:0/2291003295 >> v1:172.21.15.27:6789/0 conn(0x7f3b8c06f010 legacy=0x7f3b8c06f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.343+0000 7f3b8a7fc700 1 -- 172.21.15.27:0/2291003295 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8c06c160 con 0x7f3b8c15acf0 2024-01-11T08:25:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.343+0000 7f3b8a7fc700 1 --2- 172.21.15.27:0/2291003295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b8c15acf0 0x7f3b8c0b2910 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c06e110 tx=0x7f3b800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=702274d18bbeb453 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.343+0000 7f3b89ffb700 1 -- 172.21.15.27:0/2291003295 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b80011630 con 0x7f3b8c15acf0 2024-01-11T08:25:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.343+0000 7f3b89ffb700 1 -- 172.21.15.27:0/2291003295 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b80011790 con 0x7f3b8c15acf0 2024-01-11T08:25:04.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.343+0000 7f3b89ffb700 1 -- 172.21.15.27:0/2291003295 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b80011b30 con 0x7f3b8c15acf0 2024-01-11T08:25:04.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.344+0000 7f3b919ff700 1 -- 172.21.15.27:0/2291003295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b8c15acf0 msgr2=0x7f3b8c0b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:04.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.344+0000 7f3b919ff700 1 --2- 172.21.15.27:0/2291003295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b8c15acf0 0x7f3b8c0b2910 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c06e110 tx=0x7f3b800091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:04.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.344+0000 7f3b919ff700 1 -- 172.21.15.27:0/2291003295 shutdown_connections 2024-01-11T08:25:04.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.344+0000 7f3b919ff700 1 --2- 172.21.15.27:0/2291003295 >> v2:172.21.15.77:3300/0 conn(0x7f3b8c15a920 0x7f3b8c0b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:04.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.344+0000 7f3b919ff700 1 --2- 172.21.15.27:0/2291003295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b8c15acf0 0x7f3b8c0b2910 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:04.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.344+0000 7f3b919ff700 1 -- 172.21.15.27:0/2291003295 >> 172.21.15.27:0/2291003295 conn(0x7f3b8c06b200 msgr2=0x7f3b8c0aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:04.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.344+0000 7f3b919ff700 1 -- 172.21.15.27:0/2291003295 shutdown_connections 2024-01-11T08:25:04.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.344+0000 7f3b919ff700 1 -- 172.21.15.27:0/2291003295 wait complete. 2024-01-11T08:25:04.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.346+0000 7f3b919ff700 1 Processor -- start 2024-01-11T08:25:04.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.346+0000 7f3b919ff700 1 -- start start 2024-01-11T08:25:04.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.347+0000 7f3b919ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b8c06f010 0x7f3b8c152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:04.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.348+0000 7f3b919ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b8c15a920 0x7f3b8c156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:04.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.348+0000 7f3b919ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b8c15acf0 0x7f3b8c0ab0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:04.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.348+0000 7f3b919ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c0b01c0 con 0x7f3b8c15a920 2024-01-11T08:25:04.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.348+0000 7f3b919ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c0b0d40 con 0x7f3b8c06f010 2024-01-11T08:25:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.348+0000 7f3b919ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c0b0410 con 0x7f3b8c15acf0 2024-01-11T08:25:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.348+0000 7f3b8a7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b8c15a920 0x7f3b8c156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.348+0000 7f3b8affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b8c06f010 0x7f3b8c152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.348+0000 7f3b8a7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b8c15a920 0x7f3b8c156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36482/0 (socket says 172.21.15.27:36482) 2024-01-11T08:25:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.348+0000 7f3b8affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b8c06f010 0x7f3b8c152090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:48408/0 (socket says 172.21.15.27:48408) 2024-01-11T08:25:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.348+0000 7f3b8a7fc700 1 -- 172.21.15.27:0/718818240 learned_addr learned my addr 172.21.15.27:0/718818240 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.348+0000 7f3b8b7fe700 1 --2- 172.21.15.27:0/718818240 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b8c15acf0 0x7f3b8c0ab0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.348+0000 7f3b8a7fc700 1 -- 172.21.15.27:0/718818240 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b8c15acf0 msgr2=0x7f3b8c0ab0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.348+0000 7f3b8a7fc700 1 --2- 172.21.15.27:0/718818240 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b8c15acf0 0x7f3b8c0ab0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.348+0000 7f3b8a7fc700 1 -- 172.21.15.27:0/718818240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b8c06f010 msgr2=0x7f3b8c152090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.348+0000 7f3b8a7fc700 1 --2- 172.21.15.27:0/718818240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b8c06f010 0x7f3b8c152090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.348+0000 7f3b8a7fc700 1 -- 172.21.15.27:0/718818240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b80009420 con 0x7f3b8c15a920 2024-01-11T08:25:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.348+0000 7f3b8affd700 1 --2- 172.21.15.27:0/718818240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b8c06f010 0x7f3b8c152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:04.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.349+0000 7f3b8a7fc700 1 --2- 172.21.15.27:0/718818240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b8c15a920 0x7f3b8c156d00 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c063d40 tx=0x7f3b80017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.349+0000 7f3b72ffd700 1 -- 172.21.15.27:0/718818240 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b80000d10 con 0x7f3b8c15a920 2024-01-11T08:25:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.349+0000 7f3b72ffd700 1 -- 172.21.15.27:0/718818240 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b80011b10 con 0x7f3b8c15a920 2024-01-11T08:25:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.349+0000 7f3b72ffd700 1 -- 172.21.15.27:0/718818240 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b80025640 con 0x7f3b8c15a920 2024-01-11T08:25:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.349+0000 7f3b919ff700 1 -- 172.21.15.27:0/718818240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b8c0b06f0 con 0x7f3b8c15a920 2024-01-11T08:25:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.349+0000 7f3b919ff700 1 -- 172.21.15.27:0/718818240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b8c1525d0 con 0x7f3b8c15a920 2024-01-11T08:25:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.350+0000 7f3b919ff700 1 -- 172.21.15.27:0/718818240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b8c0718d0 con 0x7f3b8c15a920 2024-01-11T08:25:04.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.351+0000 7f3b72ffd700 1 -- 172.21.15.27:0/718818240 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f3b80025a30 con 0x7f3b8c15a920 2024-01-11T08:25:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.352+0000 7f3b72ffd700 1 --2- 172.21.15.27:0/718818240 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3b78067050 0x7f3b78069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.353+0000 7f3b72ffd700 1 -- 172.21.15.27:0/718818240 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6021+0+0 (secure 0 0 0) 0x7f3b80085fe0 con 0x7f3b8c15a920 2024-01-11T08:25:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.353+0000 7f3b8affd700 1 -- 172.21.15.27:0/718818240 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3b78067050 msgr2=0x7f3b78069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:04.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.353+0000 7f3b8affd700 1 --2- 172.21.15.27:0/718818240 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3b78067050 0x7f3b78069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:04.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.360+0000 7f3b72ffd700 1 -- 172.21.15.27:0/718818240 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3b80052fc0 con 0x7f3b8c15a920 2024-01-11T08:25:04.457 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:04.459+0000 7f1feb3201c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:25:04.505 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:04.506+0000 7ff915edc1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:25:04.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.554+0000 7f3b8affd700 1 -- 172.21.15.27:0/718818240 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3b78067050 msgr2=0x7f3b78069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:04.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.554+0000 7f3b8affd700 1 --2- 172.21.15.27:0/718818240 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3b78067050 0x7f3b78069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:04.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.609+0000 7f3b919ff700 1 -- 172.21.15.27:0/718818240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b8c06b600 con 0x7f3b8c15a920 2024-01-11T08:25:04.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.610+0000 7f3b72ffd700 1 -- 172.21.15.27:0/718818240 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4594 (secure 0 0 0) 0x7f3b800565e0 con 0x7f3b8c15a920 2024-01-11T08:25:04.609 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:04.610 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5243,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2418922463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2418922463},{"type":"v1","addr":"172.21.15.27:6835","nonce":2418922463}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5246,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3919152683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3919152683},{"type":"v1","addr":"172.21.15.77:6836","nonce":3919152683}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5641,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/329572459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":329572459},{"type":"v1","addr":"172.21.15.77:6837","nonce":329572459}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:23:05.344753+0000","modified":"2024-01-11T08:23:08.864031+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/4173068012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":4173068012},{"type":"v1","addr":"172.21.15.27:6834","nonce":4173068012}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2024-01-11T08:25:04.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.613+0000 7f3b919ff700 1 -- 172.21.15.27:0/718818240 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3b78067050 msgr2=0x7f3b78069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:04.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.613+0000 7f3b919ff700 1 --2- 172.21.15.27:0/718818240 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3b78067050 0x7f3b78069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:04.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.613+0000 7f3b919ff700 1 -- 172.21.15.27:0/718818240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b8c15a920 msgr2=0x7f3b8c156d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:04.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.613+0000 7f3b919ff700 1 --2- 172.21.15.27:0/718818240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b8c15a920 0x7f3b8c156d00 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c063d40 tx=0x7f3b80017000 comp rx=0 tx=0).stop 2024-01-11T08:25:04.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.613+0000 7f3b919ff700 1 -- 172.21.15.27:0/718818240 shutdown_connections 2024-01-11T08:25:04.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.613+0000 7f3b919ff700 1 --2- 172.21.15.27:0/718818240 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3b78067050 0x7f3b78069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:04.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.613+0000 7f3b919ff700 1 --2- 172.21.15.27:0/718818240 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b8c15acf0 0x7f3b8c0ab0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:04.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.613+0000 7f3b919ff700 1 --2- 172.21.15.27:0/718818240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b8c06f010 0x7f3b8c152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:04.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.613+0000 7f3b919ff700 1 --2- 172.21.15.27:0/718818240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b8c15a920 0x7f3b8c156d00 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:04.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.613+0000 7f3b919ff700 1 -- 172.21.15.27:0/718818240 >> 172.21.15.27:0/718818240 conn(0x7f3b8c06b200 msgr2=0x7f3b8c152ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:04.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.613+0000 7f3b919ff700 1 -- 172.21.15.27:0/718818240 shutdown_connections 2024-01-11T08:25:04.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.613+0000 7f3b919ff700 1 -- 172.21.15.27:0/718818240 wait complete. 2024-01-11T08:25:04.613 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-01-11T08:25:04.632 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:25:04.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.887+0000 7f9acf5ff700 1 Processor -- start 2024-01-11T08:25:04.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.888+0000 7f9acf5ff700 1 -- start start 2024-01-11T08:25:04.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.889+0000 7f9acf5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ac806b610 0x7f9ac8158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:04.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.889+0000 7f9acf5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac8158b60 con 0x7f9ac81575e0 2024-01-11T08:25:04.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.889+0000 7f9acf5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac81599b0 con 0x7f9ac80b19d0 2024-01-11T08:25:04.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.889+0000 7f9acf5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac80cb5e0 con 0x7f9ac806b610 2024-01-11T08:25:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.889+0000 7f9acd39b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9ac81575e0 0x7f9ac80b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44170/0 (socket says 172.21.15.27:44170) 2024-01-11T08:25:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.889+0000 7f9acd39b700 1 -- 172.21.15.27:0/311968458 learned_addr learned my addr 172.21.15.27:0/311968458 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:04.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.889+0000 7f9acdb9c700 1 --2- 172.21.15.27:0/311968458 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ac806b610 0x7f9ac8158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.890+0000 7f9abffff700 1 -- 172.21.15.27:0/311968458 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834773917 0 0) 0x7f9ac4001110 con 0x7f9ac81575e0 2024-01-11T08:25:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.890+0000 7f9abffff700 1 -- 172.21.15.27:0/311968458 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab4003570 con 0x7f9ac81575e0 2024-01-11T08:25:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.890+0000 7f9abffff700 1 -- 172.21.15.27:0/311968458 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793832484 0 0) 0x7f9ac81599b0 con 0x7f9ac80b19d0 2024-01-11T08:25:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.890+0000 7f9abffff700 1 -- 172.21.15.27:0/311968458 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac4001110 con 0x7f9ac80b19d0 2024-01-11T08:25:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.890+0000 7f9acdb9c700 1 -- 172.21.15.27:0/311968458 >> v1:172.21.15.77:6789/0 conn(0x7f9ac80b19d0 legacy=0x7f9ac8159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.890+0000 7f9acdb9c700 1 -- 172.21.15.27:0/311968458 >> v1:172.21.15.27:6789/0 conn(0x7f9ac81575e0 legacy=0x7f9ac80b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.890+0000 7f9acdb9c700 1 -- 172.21.15.27:0/311968458 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac80c0f90 con 0x7f9ac806b610 2024-01-11T08:25:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.890+0000 7f9acdb9c700 1 --2- 172.21.15.27:0/311968458 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ac806b610 0x7f9ac8158620 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0009e90 tx=0x7f9ac0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=633f78342add0914 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.891+0000 7f9abffff700 1 -- 172.21.15.27:0/311968458 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac000e070 con 0x7f9ac806b610 2024-01-11T08:25:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.891+0000 7f9abffff700 1 -- 172.21.15.27:0/311968458 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac000bb80 con 0x7f9ac806b610 2024-01-11T08:25:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.891+0000 7f9abffff700 1 -- 172.21.15.27:0/311968458 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac0012430 con 0x7f9ac806b610 2024-01-11T08:25:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.891+0000 7f9acf5ff700 1 -- 172.21.15.27:0/311968458 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ac806b610 msgr2=0x7f9ac8158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.891+0000 7f9acf5ff700 1 --2- 172.21.15.27:0/311968458 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ac806b610 0x7f9ac8158620 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0009e90 tx=0x7f9ac0009780 comp rx=0 tx=0).stop 2024-01-11T08:25:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.891+0000 7f9acf5ff700 1 -- 172.21.15.27:0/311968458 shutdown_connections 2024-01-11T08:25:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.891+0000 7f9acf5ff700 1 --2- 172.21.15.27:0/311968458 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ac806b610 0x7f9ac8158620 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:04.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.891+0000 7f9acf5ff700 1 -- 172.21.15.27:0/311968458 >> 172.21.15.27:0/311968458 conn(0x7f9ac80680e0 msgr2=0x7f9ac8069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:04.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.891+0000 7f9acf5ff700 1 -- 172.21.15.27:0/311968458 shutdown_connections 2024-01-11T08:25:04.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.892+0000 7f9acf5ff700 1 -- 172.21.15.27:0/311968458 wait complete. 2024-01-11T08:25:04.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.893+0000 7f9acf5ff700 1 Processor -- start 2024-01-11T08:25:04.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.893+0000 7f9acf5ff700 1 -- start start 2024-01-11T08:25:04.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.894+0000 7f9acf5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ac806b610 0x7f9ac816cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:04.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.894+0000 7f9acf5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ac80b19d0 0x7f9ac815b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:04.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.894+0000 7f9acf5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ac81575e0 0x7f9ac815b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:04.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.894+0000 7f9acf5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac80c56e0 con 0x7f9ac80b19d0 2024-01-11T08:25:04.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.894+0000 7f9acf5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac80affc0 con 0x7f9ac806b610 2024-01-11T08:25:04.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.894+0000 7f9acf5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac80b02a0 con 0x7f9ac81575e0 2024-01-11T08:25:04.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.894+0000 7f9accb9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ac80b19d0 0x7f9ac815b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:04.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.895+0000 7f9acd39b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ac806b610 0x7f9ac816cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:04.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.895+0000 7f9accb9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ac80b19d0 0x7f9ac815b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36498/0 (socket says 172.21.15.27:36498) 2024-01-11T08:25:04.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.895+0000 7f9accb9a700 1 -- 172.21.15.27:0/1525931114 learned_addr learned my addr 172.21.15.27:0/1525931114 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:04.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.895+0000 7f9acdb9c700 1 --2- 172.21.15.27:0/1525931114 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ac81575e0 0x7f9ac815b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:04.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.895+0000 7f9acd39b700 1 -- 172.21.15.27:0/1525931114 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ac81575e0 msgr2=0x7f9ac815b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:04.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.895+0000 7f9acd39b700 1 --2- 172.21.15.27:0/1525931114 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ac81575e0 0x7f9ac815b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:04.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.895+0000 7f9acd39b700 1 -- 172.21.15.27:0/1525931114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ac80b19d0 msgr2=0x7f9ac815b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:04.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.895+0000 7f9acd39b700 1 --2- 172.21.15.27:0/1525931114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ac80b19d0 0x7f9ac815b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:04.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.895+0000 7f9acd39b700 1 -- 172.21.15.27:0/1525931114 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac0009a20 con 0x7f9ac806b610 2024-01-11T08:25:04.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.895+0000 7f9accb9a700 1 --2- 172.21.15.27:0/1525931114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ac80b19d0 0x7f9ac815b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:04.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.895+0000 7f9acd39b700 1 --2- 172.21.15.27:0/1525931114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ac806b610 0x7f9ac816cf40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400c790 tx=0x7f9ac4007d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:04.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.896+0000 7f9abd7fa700 1 -- 172.21.15.27:0/1525931114 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4005c00 con 0x7f9ac806b610 2024-01-11T08:25:04.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.896+0000 7f9abd7fa700 1 -- 172.21.15.27:0/1525931114 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac4005d80 con 0x7f9ac806b610 2024-01-11T08:25:04.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.896+0000 7f9abd7fa700 1 -- 172.21.15.27:0/1525931114 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4007320 con 0x7f9ac806b610 2024-01-11T08:25:04.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.896+0000 7f9acf5ff700 1 -- 172.21.15.27:0/1525931114 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac80b0580 con 0x7f9ac806b610 2024-01-11T08:25:04.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.896+0000 7f9acf5ff700 1 -- 172.21.15.27:0/1525931114 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac816d480 con 0x7f9ac806b610 2024-01-11T08:25:04.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.897+0000 7f9acf5ff700 1 -- 172.21.15.27:0/1525931114 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac80718d0 con 0x7f9ac806b610 2024-01-11T08:25:04.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.898+0000 7f9abd7fa700 1 -- 172.21.15.27:0/1525931114 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f9ac401c440 con 0x7f9ac806b610 2024-01-11T08:25:04.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.899+0000 7f9abd7fa700 1 --2- 172.21.15.27:0/1525931114 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9aac067050 0x7f9aac069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:04.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.899+0000 7f9abd7fa700 1 -- 172.21.15.27:0/1525931114 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6077+0+0 (secure 0 0 0) 0x7f9ac4083ed0 con 0x7f9ac806b610 2024-01-11T08:25:04.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.899+0000 7f9accb9a700 1 -- 172.21.15.27:0/1525931114 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9aac067050 msgr2=0x7f9aac069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:04.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.899+0000 7f9accb9a700 1 --2- 172.21.15.27:0/1525931114 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9aac067050 0x7f9aac069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:04.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:04.902+0000 7f9abd7fa700 1 -- 172.21.15.27:0/1525931114 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9ac4050ea0 con 0x7f9ac806b610 2024-01-11T08:25:04.993 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:04.994+0000 7f1feb3201c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:25:05.094 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:05.095+0000 7ff915edc1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:25:05.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.100+0000 7f9accb9a700 1 -- 172.21.15.27:0/1525931114 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9aac067050 msgr2=0x7f9aac069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:05.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.100+0000 7f9accb9a700 1 --2- 172.21.15.27:0/1525931114 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9aac067050 0x7f9aac069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:05.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.114+0000 7f9acf5ff700 1 -- 172.21.15.27:0/1525931114 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ac8158b60 con 0x7f9ac806b610 2024-01-11T08:25:05.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.115+0000 7f9abd7fa700 1 -- 172.21.15.27:0/1525931114 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+15871 (secure 0 0 0) 0x7f9ac8158b60 con 0x7f9ac806b610 2024-01-11T08:25:05.114 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:05.114 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":63,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:25:04.748651+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:23:03.012171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-11T08:23:04.337714+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.7","mappings":[{"from":0,"to":4}]},{"pgid":"7.1b","mappings":[{"from":0,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/3145516755":"2024-01-12T08:23:21.771322+0000","172.21.15.77:6833/41381":"2024-01-12T08:23:21.771322+0000","172.21.15.77:6832/41381":"2024-01-12T08:23:21.771322+0000","172.21.15.77:0/434568822":"2024-01-12T08:23:21.771322+0000","172.21.15.77:0/1366514973":"2024-01-12T08:23:21.771322+0000","172.21.15.77:0/3742894353":"2024-01-12T08:23:21.771322+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:25:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.117+0000 7f9acf5ff700 1 -- 172.21.15.27:0/1525931114 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9aac067050 msgr2=0x7f9aac069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.117+0000 7f9acf5ff700 1 --2- 172.21.15.27:0/1525931114 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9aac067050 0x7f9aac069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.117+0000 7f9acf5ff700 1 -- 172.21.15.27:0/1525931114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ac806b610 msgr2=0x7f9ac816cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.117+0000 7f9acf5ff700 1 --2- 172.21.15.27:0/1525931114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ac806b610 0x7f9ac816cf40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400c790 tx=0x7f9ac4007d70 comp rx=0 tx=0).stop 2024-01-11T08:25:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.117+0000 7f9acf5ff700 1 -- 172.21.15.27:0/1525931114 shutdown_connections 2024-01-11T08:25:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.117+0000 7f9acf5ff700 1 --2- 172.21.15.27:0/1525931114 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9aac067050 0x7f9aac069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.117+0000 7f9acf5ff700 1 --2- 172.21.15.27:0/1525931114 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ac81575e0 0x7f9ac815b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.117+0000 7f9acf5ff700 1 --2- 172.21.15.27:0/1525931114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ac806b610 0x7f9ac816cf40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.117+0000 7f9acf5ff700 1 --2- 172.21.15.27:0/1525931114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ac80b19d0 0x7f9ac815b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.117+0000 7f9acf5ff700 1 -- 172.21.15.27:0/1525931114 >> 172.21.15.27:0/1525931114 conn(0x7f9ac80680e0 msgr2=0x7f9ac815a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.118+0000 7f9acf5ff700 1 -- 172.21.15.27:0/1525931114 shutdown_connections 2024-01-11T08:25:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.118+0000 7f9acf5ff700 1 -- 172.21.15.27:0/1525931114 wait complete. 2024-01-11T08:25:05.135 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T08:23:03.012171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T08:23:04.337714+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T08:25:05.136 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:25:05.164 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:05.166+0000 7f1feb3201c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:25:05.283 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:05.284+0000 7ff915edc1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:25:05.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.343+0000 7f98cbbff700 1 Processor -- start 2024-01-11T08:25:05.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.343+0000 7f98cbbff700 1 -- start start 2024-01-11T08:25:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.345+0000 7f98cbbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98c415a1b0 0x7f98c415a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.345+0000 7f98cbbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f98c415ae70 0x7f98c415b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.345+0000 7f98cbbff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c40cf850 con 0x7f98c40ccd70 2024-01-11T08:25:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.345+0000 7f98cbbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c40cec40 con 0x7f98c415ae70 2024-01-11T08:25:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.345+0000 7f98cbbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c40cef20 con 0x7f98c415a1b0 2024-01-11T08:25:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.345+0000 7f98c999b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98c415a1b0 0x7f98c415a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.345+0000 7f98ca19c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f98c415ae70 0x7f98c415b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.345+0000 7f98c999b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98c415a1b0 0x7f98c415a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:41706/0 (socket says 172.21.15.27:41706) 2024-01-11T08:25:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.345+0000 7f98ca19c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f98c415ae70 0x7f98c415b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:48424/0 (socket says 172.21.15.27:48424) 2024-01-11T08:25:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.345+0000 7f98c999b700 1 -- 172.21.15.27:0/3462884537 learned_addr learned my addr 172.21.15.27:0/3462884537 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:05.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.346+0000 7f98c999b700 1 -- 172.21.15.27:0/3462884537 >> v2:172.21.15.77:3300/0 conn(0x7f98c415ae70 msgr2=0x7f98c415b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:05.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.346+0000 7f98c999b700 1 --2- 172.21.15.27:0/3462884537 >> v2:172.21.15.77:3300/0 conn(0x7f98c415ae70 0x7f98c415b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:05.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.346+0000 7f98c999b700 1 -- 172.21.15.27:0/3462884537 >> v1:172.21.15.77:6789/0 conn(0x7f98c40ccd70 legacy=0x7f98c40cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:05.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.346+0000 7f98c999b700 1 -- 172.21.15.27:0/3462884537 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c40cf200 con 0x7f98c415a1b0 2024-01-11T08:25:05.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.346+0000 7f98c8999700 1 -- 172.21.15.27:0/3462884537 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277386425 0 0) 0x7f98c40cf850 con 0x7f98c40ccd70 2024-01-11T08:25:05.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.346+0000 7f98c999b700 1 --2- 172.21.15.27:0/3462884537 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98c415a1b0 0x7f98c415a580 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f98c00099b0 tx=0x7f98c0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb60573c16804c72 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:05.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.346+0000 7f98c8999700 1 -- 172.21.15.27:0/3462884537 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c0009ea0 con 0x7f98c415a1b0 2024-01-11T08:25:05.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.346+0000 7f98c8999700 1 -- 172.21.15.27:0/3462884537 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c000b860 con 0x7f98c415a1b0 2024-01-11T08:25:05.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.346+0000 7f98c8999700 1 -- 172.21.15.27:0/3462884537 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c000bc00 con 0x7f98c415a1b0 2024-01-11T08:25:05.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.347+0000 7f98cbbff700 1 -- 172.21.15.27:0/3462884537 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98c415a1b0 msgr2=0x7f98c415a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:05.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.347+0000 7f98cbbff700 1 --2- 172.21.15.27:0/3462884537 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98c415a1b0 0x7f98c415a580 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f98c00099b0 tx=0x7f98c0009250 comp rx=0 tx=0).stop 2024-01-11T08:25:05.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.347+0000 7f98cbbff700 1 -- 172.21.15.27:0/3462884537 shutdown_connections 2024-01-11T08:25:05.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.347+0000 7f98cbbff700 1 --2- 172.21.15.27:0/3462884537 >> v2:172.21.15.77:3300/0 conn(0x7f98c415ae70 0x7f98c415b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:05.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.347+0000 7f98cbbff700 1 --2- 172.21.15.27:0/3462884537 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98c415a1b0 0x7f98c415a580 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:05.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.347+0000 7f98cbbff700 1 -- 172.21.15.27:0/3462884537 >> 172.21.15.27:0/3462884537 conn(0x7f98c406ed40 msgr2=0x7f98c406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:05.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.347+0000 7f98cbbff700 1 -- 172.21.15.27:0/3462884537 shutdown_connections 2024-01-11T08:25:05.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.347+0000 7f98cbbff700 1 -- 172.21.15.27:0/3462884537 wait complete. 2024-01-11T08:25:05.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.349+0000 7f98cbbff700 1 Processor -- start 2024-01-11T08:25:05.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.349+0000 7f98cbbff700 1 -- start start 2024-01-11T08:25:05.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.350+0000 7f98cbbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98c40ccd70 0x7f98c40a6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:05.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.350+0000 7f98cbbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98c415a1b0 0x7f98c4166870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:05.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.350+0000 7f98cbbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98c415ae70 0x7f98c416cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:05.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.350+0000 7f98cbbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c40c9b40 con 0x7f98c415ae70 2024-01-11T08:25:05.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.350+0000 7f98cbbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c416d9c0 con 0x7f98c40ccd70 2024-01-11T08:25:05.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.350+0000 7f98cbbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c40b63a0 con 0x7f98c415a1b0 2024-01-11T08:25:05.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.350+0000 7f98ca19c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98c415ae70 0x7f98c416cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:05.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.351+0000 7f98c999b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98c40ccd70 0x7f98c40a6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:05.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.351+0000 7f98c919a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98c415a1b0 0x7f98c4166870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:05.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.351+0000 7f98c999b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98c40ccd70 0x7f98c40a6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:48440/0 (socket says 172.21.15.27:48440) 2024-01-11T08:25:05.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.351+0000 7f98c999b700 1 -- 172.21.15.27:0/80031343 learned_addr learned my addr 172.21.15.27:0/80031343 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:05.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.351+0000 7f98c999b700 1 -- 172.21.15.27:0/80031343 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98c415a1b0 msgr2=0x7f98c4166870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:05.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.351+0000 7f98c999b700 1 --2- 172.21.15.27:0/80031343 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98c415a1b0 0x7f98c4166870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:05.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.351+0000 7f98c999b700 1 -- 172.21.15.27:0/80031343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98c415ae70 msgr2=0x7f98c416cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:05.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.351+0000 7f98c999b700 1 --2- 172.21.15.27:0/80031343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98c415ae70 0x7f98c416cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:05.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.351+0000 7f98c999b700 1 -- 172.21.15.27:0/80031343 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c00094f0 con 0x7f98c40ccd70 2024-01-11T08:25:05.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.351+0000 7f98c999b700 1 --2- 172.21.15.27:0/80031343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98c40ccd70 0x7f98c40a6f10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f98c0006fd0 tx=0x7f98c0017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:05.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.352+0000 7f98b9ffb700 1 -- 172.21.15.27:0/80031343 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c0000d10 con 0x7f98c40ccd70 2024-01-11T08:25:05.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.352+0000 7f98b9ffb700 1 -- 172.21.15.27:0/80031343 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c000bbe0 con 0x7f98c40ccd70 2024-01-11T08:25:05.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.352+0000 7f98b9ffb700 1 -- 172.21.15.27:0/80031343 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c0011690 con 0x7f98c40ccd70 2024-01-11T08:25:05.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.352+0000 7f98cbbff700 1 -- 172.21.15.27:0/80031343 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c40c9f60 con 0x7f98c40ccd70 2024-01-11T08:25:05.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.352+0000 7f98cbbff700 1 -- 172.21.15.27:0/80031343 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c40c9480 con 0x7f98c40ccd70 2024-01-11T08:25:05.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.353+0000 7f98cbbff700 1 -- 172.21.15.27:0/80031343 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c404c3c0 con 0x7f98c40ccd70 2024-01-11T08:25:05.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.354+0000 7f98b9ffb700 1 -- 172.21.15.27:0/80031343 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f98c0011a80 con 0x7f98c40ccd70 2024-01-11T08:25:05.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.355+0000 7f98b9ffb700 1 --2- 172.21.15.27:0/80031343 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f98a8067050 0x7f98a8069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:05.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.355+0000 7f98b9ffb700 1 -- 172.21.15.27:0/80031343 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6077+0+0 (secure 0 0 0) 0x7f98c0086060 con 0x7f98c40ccd70 2024-01-11T08:25:05.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.355+0000 7f98c919a700 1 -- 172.21.15.27:0/80031343 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f98a8067050 msgr2=0x7f98a8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:05.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.355+0000 7f98c919a700 1 --2- 172.21.15.27:0/80031343 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f98a8067050 0x7f98a8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:05.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.359+0000 7f98b9ffb700 1 -- 172.21.15.27:0/80031343 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f98c00b75a0 con 0x7f98c40ccd70 2024-01-11T08:25:05.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.556+0000 7f98c919a700 1 -- 172.21.15.27:0/80031343 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f98a8067050 msgr2=0x7f98a8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:05.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.556+0000 7f98c919a700 1 --2- 172.21.15.27:0/80031343 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f98a8067050 0x7f98a8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:05.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.571+0000 7f98cbbff700 1 -- 172.21.15.27:0/80031343 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f98c40cddc0 con 0x7f98c40ccd70 2024-01-11T08:25:05.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.572+0000 7f98b9ffb700 1 -- 172.21.15.27:0/80031343 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) v1 ==== 93+0+10 (secure 0 0 0) 0x7f98c00150a0 con 0x7f98c40ccd70 2024-01-11T08:25:05.571 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:25:05.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.575+0000 7f98cbbff700 1 -- 172.21.15.27:0/80031343 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f98a8067050 msgr2=0x7f98a8069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:05.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.575+0000 7f98cbbff700 1 --2- 172.21.15.27:0/80031343 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f98a8067050 0x7f98a8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:05.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.575+0000 7f98cbbff700 1 -- 172.21.15.27:0/80031343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98c40ccd70 msgr2=0x7f98c40a6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:05.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.575+0000 7f98cbbff700 1 --2- 172.21.15.27:0/80031343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98c40ccd70 0x7f98c40a6f10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f98c0006fd0 tx=0x7f98c0017000 comp rx=0 tx=0).stop 2024-01-11T08:25:05.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.575+0000 7f98cbbff700 1 -- 172.21.15.27:0/80031343 shutdown_connections 2024-01-11T08:25:05.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.575+0000 7f98cbbff700 1 --2- 172.21.15.27:0/80031343 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f98a8067050 0x7f98a8069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:05.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.575+0000 7f98cbbff700 1 --2- 172.21.15.27:0/80031343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98c415ae70 0x7f98c416cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:05.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.575+0000 7f98cbbff700 1 --2- 172.21.15.27:0/80031343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98c40ccd70 0x7f98c40a6f10 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:05.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.575+0000 7f98cbbff700 1 --2- 172.21.15.27:0/80031343 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98c415a1b0 0x7f98c4166870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:05.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.575+0000 7f98cbbff700 1 -- 172.21.15.27:0/80031343 >> 172.21.15.27:0/80031343 conn(0x7f98c406ed40 msgr2=0x7f98c4158850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:05.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.575+0000 7f98cbbff700 1 -- 172.21.15.27:0/80031343 shutdown_connections 2024-01-11T08:25:05.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.575+0000 7f98cbbff700 1 -- 172.21.15.27:0/80031343 wait complete. 2024-01-11T08:25:05.587 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:05.588+0000 7f1feb3201c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:25:05.592 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T08:25:05.741 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:05.742+0000 7ff915edc1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:25:05.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.801+0000 7f11749ff700 1 Processor -- start 2024-01-11T08:25:05.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.801+0000 7f11749ff700 1 -- start start 2024-01-11T08:25:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.802+0000 7f11749ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1170159ac0 0x7f1170068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.802+0000 7f11749ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11700c80a0 con 0x7f11700ca640 2024-01-11T08:25:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.802+0000 7f11749ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11700cbac0 con 0x7f11701596f0 2024-01-11T08:25:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.802+0000 7f11749ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11700a9fb0 con 0x7f1170159ac0 2024-01-11T08:25:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.803+0000 7f116dd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1170159ac0 0x7f1170068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.803+0000 7f116dd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1170159ac0 0x7f1170068620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36514/0 (socket says 172.21.15.27:36514) 2024-01-11T08:25:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.803+0000 7f116dd9b700 1 -- 172.21.15.27:0/3553172320 learned_addr learned my addr 172.21.15.27:0/3553172320 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.803+0000 7f116dd9b700 1 -- 172.21.15.27:0/3553172320 >> v1:172.21.15.77:6789/0 conn(0x7f11701596f0 legacy=0x7f11700a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.803+0000 7f116dd9b700 1 -- 172.21.15.27:0/3553172320 >> v1:172.21.15.27:6789/0 conn(0x7f11700ca640 legacy=0x7f11700c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.803+0000 7f116dd9b700 1 -- 172.21.15.27:0/3553172320 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11700c7490 con 0x7f1170159ac0 2024-01-11T08:25:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.803+0000 7f116dd9b700 1 --2- 172.21.15.27:0/3553172320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1170159ac0 0x7f1170068620 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f11640099b0 tx=0x7f1164009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bbeeaae17483497 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.804+0000 7f116d59a700 1 -- 172.21.15.27:0/3553172320 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116400b860 con 0x7f1170159ac0 2024-01-11T08:25:05.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.804+0000 7f116d59a700 1 -- 172.21.15.27:0/3553172320 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116400b9c0 con 0x7f1170159ac0 2024-01-11T08:25:05.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.804+0000 7f116d59a700 1 -- 172.21.15.27:0/3553172320 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116400bd60 con 0x7f1170159ac0 2024-01-11T08:25:05.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.804+0000 7f11749ff700 1 -- 172.21.15.27:0/3553172320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1170159ac0 msgr2=0x7f1170068620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:05.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.804+0000 7f11749ff700 1 --2- 172.21.15.27:0/3553172320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1170159ac0 0x7f1170068620 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f11640099b0 tx=0x7f1164009250 comp rx=0 tx=0).stop 2024-01-11T08:25:05.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.805+0000 7f11749ff700 1 -- 172.21.15.27:0/3553172320 shutdown_connections 2024-01-11T08:25:05.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.805+0000 7f11749ff700 1 --2- 172.21.15.27:0/3553172320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1170159ac0 0x7f1170068620 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:05.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.805+0000 7f11749ff700 1 -- 172.21.15.27:0/3553172320 >> 172.21.15.27:0/3553172320 conn(0x7f11700c9fb0 msgr2=0x7f117006ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:05.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.805+0000 7f11749ff700 1 -- 172.21.15.27:0/3553172320 shutdown_connections 2024-01-11T08:25:05.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.805+0000 7f11749ff700 1 -- 172.21.15.27:0/3553172320 wait complete. 2024-01-11T08:25:05.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.806+0000 7f11749ff700 1 Processor -- start 2024-01-11T08:25:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.807+0000 7f11749ff700 1 -- start start 2024-01-11T08:25:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.807+0000 7f11749ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11700ca640 0x7f11700b0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.807+0000 7f11749ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11701596f0 0x7f11700b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.807+0000 7f11749ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1170159ac0 0x7f11700c6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.807+0000 7f11749ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117016a410 con 0x7f1170159ac0 2024-01-11T08:25:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.808+0000 7f11749ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117016b230 con 0x7f11701596f0 2024-01-11T08:25:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.808+0000 7f11749ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11700b1ce0 con 0x7f11700ca640 2024-01-11T08:25:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.808+0000 7f116ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1170159ac0 0x7f11700c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.808+0000 7f116ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1170159ac0 0x7f11700c6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36516/0 (socket says 172.21.15.27:36516) 2024-01-11T08:25:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.808+0000 7f116ed9d700 1 -- 172.21.15.27:0/861694435 learned_addr learned my addr 172.21.15.27:0/861694435 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.808+0000 7f116dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11701596f0 0x7f11700b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.808+0000 7f116e59c700 1 --2- 172.21.15.27:0/861694435 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11700ca640 0x7f11700b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.808+0000 7f116ed9d700 1 -- 172.21.15.27:0/861694435 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11700ca640 msgr2=0x7f11700b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.808+0000 7f116ed9d700 1 --2- 172.21.15.27:0/861694435 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11700ca640 0x7f11700b0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.808+0000 7f116ed9d700 1 -- 172.21.15.27:0/861694435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11701596f0 msgr2=0x7f11700b17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.808+0000 7f116ed9d700 1 --2- 172.21.15.27:0/861694435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11701596f0 0x7f11700b17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.808+0000 7f116ed9d700 1 -- 172.21.15.27:0/861694435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11640094f0 con 0x7f1170159ac0 2024-01-11T08:25:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.809+0000 7f116ed9d700 1 --2- 172.21.15.27:0/861694435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1170159ac0 0x7f11700c6560 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f116800c910 tx=0x7f11680016e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.809+0000 7f115e7fc700 1 -- 172.21.15.27:0/861694435 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1168012630 con 0x7f1170159ac0 2024-01-11T08:25:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.809+0000 7f115e7fc700 1 -- 172.21.15.27:0/861694435 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11680127b0 con 0x7f1170159ac0 2024-01-11T08:25:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.809+0000 7f11749ff700 1 -- 172.21.15.27:0/861694435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11700c3db0 con 0x7f1170159ac0 2024-01-11T08:25:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.809+0000 7f115e7fc700 1 -- 172.21.15.27:0/861694435 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1168012b70 con 0x7f1170159ac0 2024-01-11T08:25:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.809+0000 7f11749ff700 1 -- 172.21.15.27:0/861694435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117016a900 con 0x7f1170159ac0 2024-01-11T08:25:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.810+0000 7f11749ff700 1 -- 172.21.15.27:0/861694435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f117004c3c0 con 0x7f1170159ac0 2024-01-11T08:25:05.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.811+0000 7f115e7fc700 1 -- 172.21.15.27:0/861694435 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f1168003030 con 0x7f1170159ac0 2024-01-11T08:25:05.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.812+0000 7f115e7fc700 1 --2- 172.21.15.27:0/861694435 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1150067050 0x7f1150069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:05.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.812+0000 7f115e7fc700 1 -- 172.21.15.27:0/861694435 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6077+0+0 (secure 0 0 0) 0x7f1168084810 con 0x7f1170159ac0 2024-01-11T08:25:05.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.813+0000 7f116e59c700 1 -- 172.21.15.27:0/861694435 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1150067050 msgr2=0x7f1150069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:05.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.813+0000 7f116e59c700 1 --2- 172.21.15.27:0/861694435 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1150067050 0x7f1150069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:05.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:05.816+0000 7f115e7fc700 1 -- 172.21.15.27:0/861694435 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1168051760 con 0x7f1170159ac0 2024-01-11T08:25:06.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.014+0000 7f116e59c700 1 -- 172.21.15.27:0/861694435 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1150067050 msgr2=0x7f1150069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:06.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.014+0000 7f116e59c700 1 --2- 172.21.15.27:0/861694435 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1150067050 0x7f1150069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:06.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.030+0000 7f11749ff700 1 -- 172.21.15.27:0/861694435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f11700c7770 con 0x7f1170159ac0 2024-01-11T08:25:06.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.030+0000 7f115e7fc700 1 -- 172.21.15.27:0/861694435 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v63) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1168054d80 con 0x7f1170159ac0 2024-01-11T08:25:06.030 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T08:25:06.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.033+0000 7f11749ff700 1 -- 172.21.15.27:0/861694435 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1150067050 msgr2=0x7f1150069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:06.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.033+0000 7f11749ff700 1 --2- 172.21.15.27:0/861694435 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1150067050 0x7f1150069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.033+0000 7f11749ff700 1 -- 172.21.15.27:0/861694435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1170159ac0 msgr2=0x7f11700c6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:06.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.033+0000 7f11749ff700 1 --2- 172.21.15.27:0/861694435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1170159ac0 0x7f11700c6560 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f116800c910 tx=0x7f11680016e0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.033+0000 7f11749ff700 1 -- 172.21.15.27:0/861694435 shutdown_connections 2024-01-11T08:25:06.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.033+0000 7f11749ff700 1 --2- 172.21.15.27:0/861694435 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1150067050 0x7f1150069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.033+0000 7f11749ff700 1 --2- 172.21.15.27:0/861694435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1170159ac0 0x7f11700c6560 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.033+0000 7f11749ff700 1 --2- 172.21.15.27:0/861694435 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11700ca640 0x7f11700b0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.033+0000 7f11749ff700 1 --2- 172.21.15.27:0/861694435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11701596f0 0x7f11700b17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.033+0000 7f11749ff700 1 -- 172.21.15.27:0/861694435 >> 172.21.15.27:0/861694435 conn(0x7f11700c9fb0 msgr2=0x7f11700cb0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:06.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.033+0000 7f11749ff700 1 -- 172.21.15.27:0/861694435 shutdown_connections 2024-01-11T08:25:06.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.033+0000 7f11749ff700 1 -- 172.21.15.27:0/861694435 wait complete. 2024-01-11T08:25:06.051 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T08:25:06.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.255+0000 7f5e851ff700 1 Processor -- start 2024-01-11T08:25:06.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.255+0000 7f5e851ff700 1 -- start start 2024-01-11T08:25:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.257+0000 7f5e851ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5e80157410 0x7f5e800ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.257+0000 7f5e851ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5e80069540 0x7f5e800c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.257+0000 7f5e851ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e800699d0 con 0x7f5e801577e0 2024-01-11T08:25:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.257+0000 7f5e851ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e80069cb0 con 0x7f5e80069540 2024-01-11T08:25:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.257+0000 7f5e851ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e80069f90 con 0x7f5e80157410 2024-01-11T08:25:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.257+0000 7f5e7f59e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5e80069540 0x7f5e800c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.257+0000 7f5e7f59e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5e80069540 0x7f5e800c6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36530/0 (socket says 172.21.15.27:36530) 2024-01-11T08:25:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.257+0000 7f5e7f59e700 1 -- 172.21.15.27:0/2869648379 learned_addr learned my addr 172.21.15.27:0/2869648379 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.257+0000 7f5e7ed9d700 1 --2- 172.21.15.27:0/2869648379 >> v2:172.21.15.77:3300/0 conn(0x7f5e80157410 0x7f5e800ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.257+0000 7f5e7f59e700 1 -- 172.21.15.27:0/2869648379 >> v2:172.21.15.77:3300/0 conn(0x7f5e80157410 msgr2=0x7f5e800ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.258+0000 7f5e7f59e700 1 --2- 172.21.15.27:0/2869648379 >> v2:172.21.15.77:3300/0 conn(0x7f5e80157410 0x7f5e800ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.258+0000 7f5e7f59e700 1 -- 172.21.15.27:0/2869648379 >> v1:172.21.15.77:6789/0 conn(0x7f5e801577e0 legacy=0x7f5e800ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.258+0000 7f5e7f59e700 1 -- 172.21.15.27:0/2869648379 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8006a270 con 0x7f5e80069540 2024-01-11T08:25:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.258+0000 7f5e7f59e700 1 --2- 172.21.15.27:0/2869648379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e80069540 0x7f5e800c6ba0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5e80070c90 tx=0x7f5e78009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52af4c1f9aa17b04 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.258+0000 7f5e7dd9b700 1 -- 172.21.15.27:0/2869648379 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7800e070 con 0x7f5e80069540 2024-01-11T08:25:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.258+0000 7f5e7dd9b700 1 -- 172.21.15.27:0/2869648379 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7800ba20 con 0x7f5e80069540 2024-01-11T08:25:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.258+0000 7f5e7dd9b700 1 -- 172.21.15.27:0/2869648379 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e780092e0 con 0x7f5e80069540 2024-01-11T08:25:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.259+0000 7f5e851ff700 1 -- 172.21.15.27:0/2869648379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e80069540 msgr2=0x7f5e800c6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.259+0000 7f5e851ff700 1 --2- 172.21.15.27:0/2869648379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e80069540 0x7f5e800c6ba0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5e80070c90 tx=0x7f5e78009840 comp rx=0 tx=0).stop 2024-01-11T08:25:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.259+0000 7f5e851ff700 1 -- 172.21.15.27:0/2869648379 shutdown_connections 2024-01-11T08:25:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.259+0000 7f5e851ff700 1 --2- 172.21.15.27:0/2869648379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e80069540 0x7f5e800c6ba0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.259+0000 7f5e851ff700 1 --2- 172.21.15.27:0/2869648379 >> v2:172.21.15.77:3300/0 conn(0x7f5e80157410 0x7f5e800ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.259+0000 7f5e851ff700 1 -- 172.21.15.27:0/2869648379 >> 172.21.15.27:0/2869648379 conn(0x7f5e800680a0 msgr2=0x7f5e800cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.259+0000 7f5e851ff700 1 -- 172.21.15.27:0/2869648379 shutdown_connections 2024-01-11T08:25:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.259+0000 7f5e851ff700 1 -- 172.21.15.27:0/2869648379 wait complete. 2024-01-11T08:25:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.260+0000 7f5e851ff700 1 Processor -- start 2024-01-11T08:25:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.261+0000 7f5e851ff700 1 -- start start 2024-01-11T08:25:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.261+0000 7f5e851ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e80069540 0x7f5e80140aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.261+0000 7f5e851ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e80157410 0x7f5e80154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.261+0000 7f5e851ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e801577e0 0x7f5e80154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.261+0000 7f5e851ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e801554b0 con 0x7f5e801577e0 2024-01-11T08:25:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.261+0000 7f5e851ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8016ae10 con 0x7f5e80069540 2024-01-11T08:25:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.261+0000 7f5e851ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e80154470 con 0x7f5e80157410 2024-01-11T08:25:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.262+0000 7f5e7f59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e801577e0 0x7f5e80154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.262+0000 7f5e7f59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e801577e0 0x7f5e80154f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36544/0 (socket says 172.21.15.27:36544) 2024-01-11T08:25:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.262+0000 7f5e7f59e700 1 -- 172.21.15.27:0/3054312989 learned_addr learned my addr 172.21.15.27:0/3054312989 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.262+0000 7f5e7ed9d700 1 --2- 172.21.15.27:0/3054312989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e80069540 0x7f5e80140aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.262+0000 7f5e7e59c700 1 --2- 172.21.15.27:0/3054312989 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e80157410 0x7f5e80154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.262+0000 7f5e7f59e700 1 -- 172.21.15.27:0/3054312989 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e80157410 msgr2=0x7f5e80154a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.262+0000 7f5e7f59e700 1 --2- 172.21.15.27:0/3054312989 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e80157410 0x7f5e80154a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.262+0000 7f5e7f59e700 1 -- 172.21.15.27:0/3054312989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e80069540 msgr2=0x7f5e80140aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.262+0000 7f5e7f59e700 1 --2- 172.21.15.27:0/3054312989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e80069540 0x7f5e80140aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.262+0000 7f5e7f59e700 1 -- 172.21.15.27:0/3054312989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e78009ae0 con 0x7f5e801577e0 2024-01-11T08:25:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.263+0000 7f5e7f59e700 1 --2- 172.21.15.27:0/3054312989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e801577e0 0x7f5e80154f70 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f5e80069490 tx=0x7f5e78018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.263+0000 7f5e66ffd700 1 -- 172.21.15.27:0/3054312989 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e78008a80 con 0x7f5e801577e0 2024-01-11T08:25:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.263+0000 7f5e66ffd700 1 -- 172.21.15.27:0/3054312989 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e78008c00 con 0x7f5e801577e0 2024-01-11T08:25:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.263+0000 7f5e66ffd700 1 -- 172.21.15.27:0/3054312989 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7801c410 con 0x7f5e801577e0 2024-01-11T08:25:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.263+0000 7f5e851ff700 1 -- 172.21.15.27:0/3054312989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e80154750 con 0x7f5e801577e0 2024-01-11T08:25:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.263+0000 7f5e851ff700 1 -- 172.21.15.27:0/3054312989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8016b360 con 0x7f5e801577e0 2024-01-11T08:25:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.265+0000 7f5e851ff700 1 -- 172.21.15.27:0/3054312989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e8006b8c0 con 0x7f5e801577e0 2024-01-11T08:25:06.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.265+0000 7f5e66ffd700 1 -- 172.21.15.27:0/3054312989 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f5e78003040 con 0x7f5e801577e0 2024-01-11T08:25:06.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.266+0000 7f5e66ffd700 1 --2- 172.21.15.27:0/3054312989 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5e6c067050 0x7f5e6c069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:06.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.266+0000 7f5e66ffd700 1 -- 172.21.15.27:0/3054312989 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6077+0+0 (secure 0 0 0) 0x7f5e78086160 con 0x7f5e801577e0 2024-01-11T08:25:06.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.266+0000 7f5e7ed9d700 1 -- 172.21.15.27:0/3054312989 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5e6c067050 msgr2=0x7f5e6c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:06.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.266+0000 7f5e7ed9d700 1 --2- 172.21.15.27:0/3054312989 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5e6c067050 0x7f5e6c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:06.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.270+0000 7f5e66ffd700 1 -- 172.21.15.27:0/3054312989 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5e78052fa0 con 0x7f5e801577e0 2024-01-11T08:25:06.371 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:06.372+0000 7f1feb3201c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:25:06.459 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:06.460+0000 7f1feb3201c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:25:06.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.468+0000 7f5e7ed9d700 1 -- 172.21.15.27:0/3054312989 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5e6c067050 msgr2=0x7f5e6c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:06.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.468+0000 7f5e7ed9d700 1 --2- 172.21.15.27:0/3054312989 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5e6c067050 0x7f5e6c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:06.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.475+0000 7f5e851ff700 1 -- 172.21.15.27:0/3054312989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5e8016b580 con 0x7f5e801577e0 2024-01-11T08:25:06.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.476+0000 7f5e66ffd700 1 -- 172.21.15.27:0/3054312989 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v63) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5e780565c0 con 0x7f5e801577e0 2024-01-11T08:25:06.475 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T08:25:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.478+0000 7f5e851ff700 1 -- 172.21.15.27:0/3054312989 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5e6c067050 msgr2=0x7f5e6c069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.478+0000 7f5e851ff700 1 --2- 172.21.15.27:0/3054312989 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5e6c067050 0x7f5e6c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.478+0000 7f5e851ff700 1 -- 172.21.15.27:0/3054312989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e801577e0 msgr2=0x7f5e80154f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.478+0000 7f5e851ff700 1 --2- 172.21.15.27:0/3054312989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e801577e0 0x7f5e80154f70 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f5e80069490 tx=0x7f5e78018000 comp rx=0 tx=0).stop 2024-01-11T08:25:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.478+0000 7f5e851ff700 1 -- 172.21.15.27:0/3054312989 shutdown_connections 2024-01-11T08:25:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.478+0000 7f5e851ff700 1 --2- 172.21.15.27:0/3054312989 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f5e6c067050 0x7f5e6c069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.478+0000 7f5e851ff700 1 --2- 172.21.15.27:0/3054312989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e801577e0 0x7f5e80154f70 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.478+0000 7f5e851ff700 1 --2- 172.21.15.27:0/3054312989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e80069540 0x7f5e80140aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.478+0000 7f5e851ff700 1 --2- 172.21.15.27:0/3054312989 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e80157410 0x7f5e80154a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.479+0000 7f5e851ff700 1 -- 172.21.15.27:0/3054312989 >> 172.21.15.27:0/3054312989 conn(0x7f5e800680a0 msgr2=0x7f5e800af200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:06.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.479+0000 7f5e851ff700 1 -- 172.21.15.27:0/3054312989 shutdown_connections 2024-01-11T08:25:06.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.479+0000 7f5e851ff700 1 -- 172.21.15.27:0/3054312989 wait complete. 2024-01-11T08:25:06.497 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:25:06.594 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:06.595+0000 7ff915edc1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:25:06.615 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:06.616+0000 7f1feb3201c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:25:06.689 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:06.690+0000 7ff915edc1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:25:06.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.699+0000 7fee217ff700 1 Processor -- start 2024-01-11T08:25:06.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.700+0000 7fee217ff700 1 -- start start 2024-01-11T08:25:06.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.702+0000 7fee217ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fee1c0ca7b0 0x7fee1c0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:06.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.702+0000 7fee217ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fee1c159e80 0x7fee1c15dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:06.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.702+0000 7fee217ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee1c15d640 con 0x7fee1c0ca3e0 2024-01-11T08:25:06.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.702+0000 7fee217ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee1c15ae70 con 0x7fee1c0ca7b0 2024-01-11T08:25:06.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.702+0000 7fee217ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee1c0cc4f0 con 0x7fee1c159e80 2024-01-11T08:25:06.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.702+0000 7fee1a7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fee1c0ca7b0 0x7fee1c0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:06.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.702+0000 7fee1affd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fee1c0ca3e0 0x7fee1c0c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44192/0 (socket says 172.21.15.27:44192) 2024-01-11T08:25:06.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.702+0000 7fee1a7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fee1c0ca7b0 0x7fee1c0cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36548/0 (socket says 172.21.15.27:36548) 2024-01-11T08:25:06.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.702+0000 7fee1affd700 1 -- 172.21.15.27:0/258943159 learned_addr learned my addr 172.21.15.27:0/258943159 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:06.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.702+0000 7fee1b7fe700 1 --2- 172.21.15.27:0/258943159 >> v2:172.21.15.77:3300/0 conn(0x7fee1c159e80 0x7fee1c15dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:06.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.702+0000 7fee1a7fc700 1 -- 172.21.15.27:0/258943159 >> v2:172.21.15.77:3300/0 conn(0x7fee1c159e80 msgr2=0x7fee1c15dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:06.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.702+0000 7fee1a7fc700 1 --2- 172.21.15.27:0/258943159 >> v2:172.21.15.77:3300/0 conn(0x7fee1c159e80 0x7fee1c15dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.702+0000 7fee1a7fc700 1 -- 172.21.15.27:0/258943159 >> v1:172.21.15.27:6789/0 conn(0x7fee1c0ca3e0 legacy=0x7fee1c0c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:06.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.703+0000 7fee1a7fc700 1 -- 172.21.15.27:0/258943159 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1c15a5d0 con 0x7fee1c0ca7b0 2024-01-11T08:25:06.703 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:06.703+0000 7f1feb3201c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:25:06.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.703+0000 7fee1a7fc700 1 --2- 172.21.15.27:0/258943159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee1c0ca7b0 0x7fee1c0cca20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fee10009910 tx=0x7fee100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f742bfb4a45565 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:06.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.703+0000 7fee19ffb700 1 -- 172.21.15.27:0/258943159 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee10011630 con 0x7fee1c0ca7b0 2024-01-11T08:25:06.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.703+0000 7fee19ffb700 1 -- 172.21.15.27:0/258943159 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee10011790 con 0x7fee1c0ca7b0 2024-01-11T08:25:06.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.703+0000 7fee19ffb700 1 -- 172.21.15.27:0/258943159 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee10011b50 con 0x7fee1c0ca7b0 2024-01-11T08:25:06.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.704+0000 7fee217ff700 1 -- 172.21.15.27:0/258943159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee1c0ca7b0 msgr2=0x7fee1c0cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.704+0000 7fee217ff700 1 --2- 172.21.15.27:0/258943159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee1c0ca7b0 0x7fee1c0cca20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fee10009910 tx=0x7fee100091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.704+0000 7fee217ff700 1 -- 172.21.15.27:0/258943159 shutdown_connections 2024-01-11T08:25:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.704+0000 7fee217ff700 1 --2- 172.21.15.27:0/258943159 >> v2:172.21.15.77:3300/0 conn(0x7fee1c159e80 0x7fee1c15dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.704+0000 7fee217ff700 1 --2- 172.21.15.27:0/258943159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee1c0ca7b0 0x7fee1c0cca20 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.704+0000 7fee217ff700 1 -- 172.21.15.27:0/258943159 >> 172.21.15.27:0/258943159 conn(0x7fee1c0ce3e0 msgr2=0x7fee1c0ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.704+0000 7fee217ff700 1 -- 172.21.15.27:0/258943159 shutdown_connections 2024-01-11T08:25:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.704+0000 7fee217ff700 1 -- 172.21.15.27:0/258943159 wait complete. 2024-01-11T08:25:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.706+0000 7fee217ff700 1 Processor -- start 2024-01-11T08:25:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.706+0000 7fee217ff700 1 -- start start 2024-01-11T08:25:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.707+0000 7fee217ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee1c0ca3e0 0x7fee1c14cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.707+0000 7fee217ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee1c0ca7b0 0x7fee1c14d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.708+0000 7fee217ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee1c159e80 0x7fee1c14d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.708+0000 7fee217ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c14c830 con 0x7fee1c159e80 2024-01-11T08:25:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.708+0000 7fee217ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c14c490 con 0x7fee1c0ca3e0 2024-01-11T08:25:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.708+0000 7fee217ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c0b23f0 con 0x7fee1c0ca7b0 2024-01-11T08:25:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.708+0000 7fee1b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee1c159e80 0x7fee1c14d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.708+0000 7fee1b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee1c159e80 0x7fee1c14d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36562/0 (socket says 172.21.15.27:36562) 2024-01-11T08:25:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.708+0000 7fee1b7fe700 1 -- 172.21.15.27:0/662215473 learned_addr learned my addr 172.21.15.27:0/662215473 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.708+0000 7fee1affd700 1 --2- 172.21.15.27:0/662215473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee1c0ca3e0 0x7fee1c14cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.708+0000 7fee1a7fc700 1 --2- 172.21.15.27:0/662215473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee1c0ca7b0 0x7fee1c14d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.708+0000 7fee1b7fe700 1 -- 172.21.15.27:0/662215473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee1c0ca7b0 msgr2=0x7fee1c14d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.708+0000 7fee1b7fe700 1 --2- 172.21.15.27:0/662215473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee1c0ca7b0 0x7fee1c14d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.708+0000 7fee1b7fe700 1 -- 172.21.15.27:0/662215473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee1c0ca3e0 msgr2=0x7fee1c14cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.708+0000 7fee1b7fe700 1 --2- 172.21.15.27:0/662215473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee1c0ca3e0 0x7fee1c14cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.708+0000 7fee1b7fe700 1 -- 172.21.15.27:0/662215473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee10009420 con 0x7fee1c159e80 2024-01-11T08:25:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.709+0000 7fee1b7fe700 1 --2- 172.21.15.27:0/662215473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee1c159e80 0x7fee1c14d6d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fee14009df0 tx=0x7fee1400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.709+0000 7fee02ffd700 1 -- 172.21.15.27:0/662215473 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1400caa0 con 0x7fee1c159e80 2024-01-11T08:25:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.709+0000 7fee02ffd700 1 -- 172.21.15.27:0/662215473 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1400f040 con 0x7fee1c159e80 2024-01-11T08:25:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.709+0000 7fee217ff700 1 -- 172.21.15.27:0/662215473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1c0b26d0 con 0x7fee1c159e80 2024-01-11T08:25:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.709+0000 7fee02ffd700 1 -- 172.21.15.27:0/662215473 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1400ce50 con 0x7fee1c159e80 2024-01-11T08:25:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.709+0000 7fee217ff700 1 -- 172.21.15.27:0/662215473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1c0b20a0 con 0x7fee1c159e80 2024-01-11T08:25:06.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.710+0000 7fee217ff700 1 -- 172.21.15.27:0/662215473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee1c14c970 con 0x7fee1c159e80 2024-01-11T08:25:06.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.711+0000 7fee02ffd700 1 -- 172.21.15.27:0/662215473 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fee14014690 con 0x7fee1c159e80 2024-01-11T08:25:06.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.713+0000 7fee02ffd700 1 --2- 172.21.15.27:0/662215473 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fee08067050 0x7fee08069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:06.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.713+0000 7fee02ffd700 1 -- 172.21.15.27:0/662215473 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6077+0+0 (secure 0 0 0) 0x7fee14085460 con 0x7fee1c159e80 2024-01-11T08:25:06.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.713+0000 7fee1affd700 1 -- 172.21.15.27:0/662215473 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fee08067050 msgr2=0x7fee08069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:06.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.713+0000 7fee1affd700 1 --2- 172.21.15.27:0/662215473 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fee08067050 0x7fee08069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:06.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.716+0000 7fee02ffd700 1 -- 172.21.15.27:0/662215473 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fee1c14c970 con 0x7fee1c159e80 2024-01-11T08:25:06.860 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:06.861+0000 7ff915edc1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:25:06.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.914+0000 7fee1affd700 1 -- 172.21.15.27:0/662215473 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fee08067050 msgr2=0x7fee08069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:06.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.914+0000 7fee1affd700 1 --2- 172.21.15.27:0/662215473 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fee08067050 0x7fee08069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:06.918 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:06.920+0000 7f1feb3201c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:25:06.953 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:06.954+0000 7ff915edc1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:25:06.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.963+0000 7fee217ff700 1 -- 172.21.15.27:0/662215473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee1c14c970 con 0x7fee1c159e80 2024-01-11T08:25:06.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.964+0000 7fee02ffd700 1 -- 172.21.15.27:0/662215473 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4594 (secure 0 0 0) 0x7fee1c14c970 con 0x7fee1c159e80 2024-01-11T08:25:06.963 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:06.963 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5243,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2418922463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2418922463},{"type":"v1","addr":"172.21.15.27:6835","nonce":2418922463}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5246,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3919152683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3919152683},{"type":"v1","addr":"172.21.15.77:6836","nonce":3919152683}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5641,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/329572459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":329572459},{"type":"v1","addr":"172.21.15.77:6837","nonce":329572459}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:23:05.344753+0000","modified":"2024-01-11T08:23:08.864031+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/4173068012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":4173068012},{"type":"v1","addr":"172.21.15.27:6834","nonce":4173068012}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2024-01-11T08:25:06.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.966+0000 7fee217ff700 1 -- 172.21.15.27:0/662215473 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fee08067050 msgr2=0x7fee08069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:06.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.966+0000 7fee217ff700 1 --2- 172.21.15.27:0/662215473 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fee08067050 0x7fee08069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.966+0000 7fee217ff700 1 -- 172.21.15.27:0/662215473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee1c159e80 msgr2=0x7fee1c14d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:06.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.966+0000 7fee217ff700 1 --2- 172.21.15.27:0/662215473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee1c159e80 0x7fee1c14d6d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fee14009df0 tx=0x7fee1400b040 comp rx=0 tx=0).stop 2024-01-11T08:25:06.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.966+0000 7fee217ff700 1 -- 172.21.15.27:0/662215473 shutdown_connections 2024-01-11T08:25:06.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.966+0000 7fee217ff700 1 --2- 172.21.15.27:0/662215473 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fee08067050 0x7fee08069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.966+0000 7fee217ff700 1 --2- 172.21.15.27:0/662215473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee1c159e80 0x7fee1c14d6d0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.966+0000 7fee217ff700 1 --2- 172.21.15.27:0/662215473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee1c0ca3e0 0x7fee1c14cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.966+0000 7fee217ff700 1 --2- 172.21.15.27:0/662215473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee1c0ca7b0 0x7fee1c14d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:06.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.966+0000 7fee217ff700 1 -- 172.21.15.27:0/662215473 >> 172.21.15.27:0/662215473 conn(0x7fee1c0ce3e0 msgr2=0x7fee1c06d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:06.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.966+0000 7fee217ff700 1 -- 172.21.15.27:0/662215473 shutdown_connections 2024-01-11T08:25:06.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:06.966+0000 7fee217ff700 1 -- 172.21.15.27:0/662215473 wait complete. 2024-01-11T08:25:06.967 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-01-11T08:25:06.984 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:25:07.015 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:07.017+0000 7f1feb3201c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:25:07.111 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:07.112+0000 7f1feb3201c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:25:07.190 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:07.191+0000 7ff915edc1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:25:07.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.195+0000 7f3b673ff700 1 Processor -- start 2024-01-11T08:25:07.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.195+0000 7f3b673ff700 1 -- start start 2024-01-11T08:25:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.197+0000 7f3b673ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3b600ca3e0 0x7f3b600cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.197+0000 7f3b673ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b60159e80 0x7f3b6015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.197+0000 7f3b673ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b6015d640 con 0x7f3b600ca7b0 2024-01-11T08:25:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.197+0000 7f3b673ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6015ae70 con 0x7f3b600ca3e0 2024-01-11T08:25:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.197+0000 7f3b673ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b600cc4f0 con 0x7f3b60159e80 2024-01-11T08:25:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.197+0000 7f3b6519b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3b600ca3e0 0x7f3b600cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.197+0000 7f3b6499a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3b600ca7b0 0x7f3b600c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44198/0 (socket says 172.21.15.27:44198) 2024-01-11T08:25:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.197+0000 7f3b6499a700 1 -- 172.21.15.27:0/3738974712 learned_addr learned my addr 172.21.15.27:0/3738974712 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.197+0000 7f3b6519b700 1 --2- 172.21.15.27:0/3738974712 >> v2:172.21.15.27:3300/0 conn(0x7f3b600ca3e0 0x7f3b600cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36574/0 (socket says 172.21.15.27:36574) 2024-01-11T08:25:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.197+0000 7f3b6599c700 1 --2- 172.21.15.27:0/3738974712 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b60159e80 0x7f3b6015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.198+0000 7f3b6519b700 1 -- 172.21.15.27:0/3738974712 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b60159e80 msgr2=0x7f3b6015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.198+0000 7f3b6519b700 1 --2- 172.21.15.27:0/3738974712 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b60159e80 0x7f3b6015dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.198+0000 7f3b6519b700 1 -- 172.21.15.27:0/3738974712 >> v1:172.21.15.27:6789/0 conn(0x7f3b600ca7b0 legacy=0x7f3b600c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.198+0000 7f3b6519b700 1 -- 172.21.15.27:0/3738974712 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6015a5d0 con 0x7f3b600ca3e0 2024-01-11T08:25:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.198+0000 7f3b6519b700 1 --2- 172.21.15.27:0/3738974712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b600ca3e0 0x7f3b600cca20 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c009940 tx=0x7f3b5c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6dcd1209adb81c7 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.198+0000 7f3b57fff700 1 -- 172.21.15.27:0/3738974712 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c011630 con 0x7f3b600ca3e0 2024-01-11T08:25:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.198+0000 7f3b57fff700 1 -- 172.21.15.27:0/3738974712 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5c011790 con 0x7f3b600ca3e0 2024-01-11T08:25:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.198+0000 7f3b57fff700 1 -- 172.21.15.27:0/3738974712 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c011b50 con 0x7f3b600ca3e0 2024-01-11T08:25:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.199+0000 7f3b673ff700 1 -- 172.21.15.27:0/3738974712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b600ca3e0 msgr2=0x7f3b600cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.199+0000 7f3b673ff700 1 --2- 172.21.15.27:0/3738974712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b600ca3e0 0x7f3b600cca20 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c009940 tx=0x7f3b5c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.199+0000 7f3b673ff700 1 -- 172.21.15.27:0/3738974712 shutdown_connections 2024-01-11T08:25:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.199+0000 7f3b673ff700 1 --2- 172.21.15.27:0/3738974712 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b60159e80 0x7f3b6015dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.199+0000 7f3b673ff700 1 --2- 172.21.15.27:0/3738974712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b600ca3e0 0x7f3b600cca20 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.199+0000 7f3b673ff700 1 -- 172.21.15.27:0/3738974712 >> 172.21.15.27:0/3738974712 conn(0x7f3b600ce3e0 msgr2=0x7f3b600ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.199+0000 7f3b673ff700 1 -- 172.21.15.27:0/3738974712 shutdown_connections 2024-01-11T08:25:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.199+0000 7f3b673ff700 1 -- 172.21.15.27:0/3738974712 wait complete. 2024-01-11T08:25:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.201+0000 7f3b673ff700 1 Processor -- start 2024-01-11T08:25:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.202+0000 7f3b673ff700 1 -- start start 2024-01-11T08:25:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.203+0000 7f3b673ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b600ca3e0 0x7f3b600b3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.203+0000 7f3b673ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b600ca7b0 0x7f3b60143d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.203+0000 7f3b673ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b60159e80 0x7f3b600c4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.203+0000 7f3b673ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6016a240 con 0x7f3b60159e80 2024-01-11T08:25:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.203+0000 7f3b673ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6016a520 con 0x7f3b600ca7b0 2024-01-11T08:25:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.203+0000 7f3b673ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6016a800 con 0x7f3b600ca3e0 2024-01-11T08:25:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.204+0000 7f3b6519b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b600ca3e0 0x7f3b600b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.204+0000 7f3b6499a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b600ca7b0 0x7f3b60143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.204+0000 7f3b6599c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b60159e80 0x7f3b600c4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.204+0000 7f3b6519b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b600ca3e0 0x7f3b600b3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:41764/0 (socket says 172.21.15.27:41764) 2024-01-11T08:25:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.204+0000 7f3b6519b700 1 -- 172.21.15.27:0/160400883 learned_addr learned my addr 172.21.15.27:0/160400883 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:07.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.204+0000 7f3b6599c700 1 -- 172.21.15.27:0/160400883 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b600ca3e0 msgr2=0x7f3b600b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:07.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.204+0000 7f3b6599c700 1 --2- 172.21.15.27:0/160400883 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b600ca3e0 0x7f3b600b3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.204+0000 7f3b6599c700 1 -- 172.21.15.27:0/160400883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b600ca7b0 msgr2=0x7f3b60143d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:07.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.204+0000 7f3b6599c700 1 --2- 172.21.15.27:0/160400883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b600ca7b0 0x7f3b60143d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.204+0000 7f3b6599c700 1 -- 172.21.15.27:0/160400883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c009420 con 0x7f3b60159e80 2024-01-11T08:25:07.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.204+0000 7f3b6519b700 1 --2- 172.21.15.27:0/160400883 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b600ca3e0 0x7f3b600b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:07.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.204+0000 7f3b6599c700 1 --2- 172.21.15.27:0/160400883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b60159e80 0x7f3b600c4150 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3b5800ecc0 tx=0x7f3b5800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:07.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.205+0000 7f3b557fa700 1 -- 172.21.15.27:0/160400883 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5800c8c0 con 0x7f3b60159e80 2024-01-11T08:25:07.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.205+0000 7f3b557fa700 1 -- 172.21.15.27:0/160400883 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5800f040 con 0x7f3b60159e80 2024-01-11T08:25:07.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.205+0000 7f3b673ff700 1 -- 172.21.15.27:0/160400883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6016aae0 con 0x7f3b60159e80 2024-01-11T08:25:07.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.205+0000 7f3b557fa700 1 -- 172.21.15.27:0/160400883 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5800cc90 con 0x7f3b60159e80 2024-01-11T08:25:07.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.205+0000 7f3b673ff700 1 -- 172.21.15.27:0/160400883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b60169f40 con 0x7f3b60159e80 2024-01-11T08:25:07.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.206+0000 7f3b673ff700 1 -- 172.21.15.27:0/160400883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b600c8610 con 0x7f3b60159e80 2024-01-11T08:25:07.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.207+0000 7f3b557fa700 1 -- 172.21.15.27:0/160400883 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f3b58014690 con 0x7f3b60159e80 2024-01-11T08:25:07.207 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:07.208+0000 7f1feb3201c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:25:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.208+0000 7f3b557fa700 1 --2- 172.21.15.27:0/160400883 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3b44067050 0x7f3b44069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.209+0000 7f3b557fa700 1 -- 172.21.15.27:0/160400883 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6077+0+0 (secure 0 0 0) 0x7f3b580853a0 con 0x7f3b60159e80 2024-01-11T08:25:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.209+0000 7f3b6519b700 1 -- 172.21.15.27:0/160400883 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3b44067050 msgr2=0x7f3b44069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.209+0000 7f3b6519b700 1 --2- 172.21.15.27:0/160400883 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3b44067050 0x7f3b44069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:07.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.212+0000 7f3b557fa700 1 -- 172.21.15.27:0/160400883 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3b580521e0 con 0x7f3b60159e80 2024-01-11T08:25:07.296 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:07.297+0000 7ff915edc1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:25:07.396 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:07.397+0000 7f1feb3201c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:25:07.397 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:07.398+0000 7ff915edc1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:25:07.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.410+0000 7f3b6519b700 1 -- 172.21.15.27:0/160400883 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3b44067050 msgr2=0x7f3b44069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:07.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.410+0000 7f3b6519b700 1 --2- 172.21.15.27:0/160400883 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3b44067050 0x7f3b44069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:07.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.417+0000 7f3b673ff700 1 -- 172.21.15.27:0/160400883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b600718d0 con 0x7f3b60159e80 2024-01-11T08:25:07.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.418+0000 7f3b557fa700 1 -- 172.21.15.27:0/160400883 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+15871 (secure 0 0 0) 0x7f3b58055800 con 0x7f3b60159e80 2024-01-11T08:25:07.417 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:07.417 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":63,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:25:04.748651+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:23:03.012171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-11T08:23:04.337714+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.7","mappings":[{"from":0,"to":4}]},{"pgid":"7.1b","mappings":[{"from":0,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/3145516755":"2024-01-12T08:23:21.771322+0000","172.21.15.77:6833/41381":"2024-01-12T08:23:21.771322+0000","172.21.15.77:6832/41381":"2024-01-12T08:23:21.771322+0000","172.21.15.77:0/434568822":"2024-01-12T08:23:21.771322+0000","172.21.15.77:0/1366514973":"2024-01-12T08:23:21.771322+0000","172.21.15.77:0/3742894353":"2024-01-12T08:23:21.771322+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:25:07.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.420+0000 7f3b673ff700 1 -- 172.21.15.27:0/160400883 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3b44067050 msgr2=0x7f3b44069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:07.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.420+0000 7f3b673ff700 1 --2- 172.21.15.27:0/160400883 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3b44067050 0x7f3b44069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.420+0000 7f3b673ff700 1 -- 172.21.15.27:0/160400883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b60159e80 msgr2=0x7f3b600c4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:07.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.420+0000 7f3b673ff700 1 --2- 172.21.15.27:0/160400883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b60159e80 0x7f3b600c4150 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3b5800ecc0 tx=0x7f3b5800b040 comp rx=0 tx=0).stop 2024-01-11T08:25:07.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.420+0000 7f3b673ff700 1 -- 172.21.15.27:0/160400883 shutdown_connections 2024-01-11T08:25:07.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.420+0000 7f3b673ff700 1 --2- 172.21.15.27:0/160400883 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3b44067050 0x7f3b44069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.420+0000 7f3b673ff700 1 --2- 172.21.15.27:0/160400883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b60159e80 0x7f3b600c4150 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.420+0000 7f3b673ff700 1 --2- 172.21.15.27:0/160400883 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b600ca3e0 0x7f3b600b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.420+0000 7f3b673ff700 1 --2- 172.21.15.27:0/160400883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b600ca7b0 0x7f3b60143d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.420+0000 7f3b673ff700 1 -- 172.21.15.27:0/160400883 >> 172.21.15.27:0/160400883 conn(0x7f3b600ce3e0 msgr2=0x7f3b600691a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:07.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.420+0000 7f3b673ff700 1 -- 172.21.15.27:0/160400883 shutdown_connections 2024-01-11T08:25:07.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.421+0000 7f3b673ff700 1 -- 172.21.15.27:0/160400883 wait complete. 2024-01-11T08:25:07.438 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-11T08:25:07.438 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:25:07.494 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:07.495+0000 7f1feb3201c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:25:07.500 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:07.501+0000 7ff915edc1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:25:07.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.641+0000 7f52d73ff700 1 Processor -- start 2024-01-11T08:25:07.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.641+0000 7f52d73ff700 1 -- start start 2024-01-11T08:25:07.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.642+0000 7f52d73ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f52d0159650 0x7f52d006bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:07.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.642+0000 7f52d73ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52d0159a20 0x7f52d00b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:07.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.643+0000 7f52d73ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d006fba0 con 0x7f52d00b2730 2024-01-11T08:25:07.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.643+0000 7f52d73ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d006e700 con 0x7f52d0159650 2024-01-11T08:25:07.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.643+0000 7f52d73ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00b11a0 con 0x7f52d0159a20 2024-01-11T08:25:07.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.643+0000 7f52d519b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f52d0159650 0x7f52d006bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:07.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.643+0000 7f52d519b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f52d0159650 0x7f52d006bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36588/0 (socket says 172.21.15.27:36588) 2024-01-11T08:25:07.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.643+0000 7f52d519b700 1 -- 172.21.15.27:0/3429328167 learned_addr learned my addr 172.21.15.27:0/3429328167 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:07.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.643+0000 7f52d599c700 1 --1- 172.21.15.27:0/3429328167 >> v1:172.21.15.27:6789/0 conn(0x7f52d00b2730 0x7f52d00b2bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44206/0 (socket says 172.21.15.27:44206) 2024-01-11T08:25:07.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.643+0000 7f52d499a700 1 --2- 172.21.15.27:0/3429328167 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52d0159a20 0x7f52d00b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:07.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.643+0000 7f52d519b700 1 -- 172.21.15.27:0/3429328167 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52d0159a20 msgr2=0x7f52d00b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:07.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.643+0000 7f52d519b700 1 --2- 172.21.15.27:0/3429328167 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52d0159a20 0x7f52d00b0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.643+0000 7f52d519b700 1 -- 172.21.15.27:0/3429328167 >> v1:172.21.15.27:6789/0 conn(0x7f52d00b2730 legacy=0x7f52d00b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:07.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.643+0000 7f52d519b700 1 -- 172.21.15.27:0/3429328167 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d006c360 con 0x7f52d0159650 2024-01-11T08:25:07.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.644+0000 7f52d519b700 1 --2- 172.21.15.27:0/3429328167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52d0159650 0x7f52d006bd80 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0099b0 tx=0x7f52cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d529892d5f375acb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:07.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.644+0000 7f52c7fff700 1 -- 172.21.15.27:0/3429328167 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc009ea0 con 0x7f52d0159650 2024-01-11T08:25:07.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.644+0000 7f52c7fff700 1 -- 172.21.15.27:0/3429328167 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52cc00b860 con 0x7f52d0159650 2024-01-11T08:25:07.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.644+0000 7f52c7fff700 1 -- 172.21.15.27:0/3429328167 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc00bc00 con 0x7f52d0159650 2024-01-11T08:25:07.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.645+0000 7f52d73ff700 1 -- 172.21.15.27:0/3429328167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52d0159650 msgr2=0x7f52d006bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:07.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.645+0000 7f52d73ff700 1 --2- 172.21.15.27:0/3429328167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52d0159650 0x7f52d006bd80 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0099b0 tx=0x7f52cc009250 comp rx=0 tx=0).stop 2024-01-11T08:25:07.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.645+0000 7f52d73ff700 1 -- 172.21.15.27:0/3429328167 shutdown_connections 2024-01-11T08:25:07.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.645+0000 7f52d73ff700 1 --2- 172.21.15.27:0/3429328167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52d0159650 0x7f52d006bd80 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.645+0000 7f52d73ff700 1 --2- 172.21.15.27:0/3429328167 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52d0159a20 0x7f52d00b0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.645+0000 7f52d73ff700 1 -- 172.21.15.27:0/3429328167 >> 172.21.15.27:0/3429328167 conn(0x7f52d006b200 msgr2=0x7f52d00aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:07.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.645+0000 7f52d73ff700 1 -- 172.21.15.27:0/3429328167 shutdown_connections 2024-01-11T08:25:07.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.645+0000 7f52d73ff700 1 -- 172.21.15.27:0/3429328167 wait complete. 2024-01-11T08:25:07.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.647+0000 7f52d73ff700 1 Processor -- start 2024-01-11T08:25:07.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.648+0000 7f52d73ff700 1 -- start start 2024-01-11T08:25:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.649+0000 7f52d73ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52d00b2730 0x7f52d00aa260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.649+0000 7f52d73ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52d0159650 0x7f52d0169ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.649+0000 7f52d73ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52d0159a20 0x7f52d0157660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.649+0000 7f52d73ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00abdc0 con 0x7f52d0159a20 2024-01-11T08:25:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.649+0000 7f52d73ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00aaf10 con 0x7f52d0159650 2024-01-11T08:25:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.649+0000 7f52d73ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00ab1b0 con 0x7f52d00b2730 2024-01-11T08:25:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.649+0000 7f52d599c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52d0159a20 0x7f52d0157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.649+0000 7f52d519b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52d00b2730 0x7f52d00aa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.649+0000 7f52d499a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52d0159650 0x7f52d0169ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.649+0000 7f52d519b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52d00b2730 0x7f52d00aa260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:41786/0 (socket says 172.21.15.27:41786) 2024-01-11T08:25:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.649+0000 7f52d499a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52d0159650 0x7f52d0169ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:48526/0 (socket says 172.21.15.27:48526) 2024-01-11T08:25:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.649+0000 7f52d519b700 1 -- 172.21.15.27:0/3111387190 learned_addr learned my addr 172.21.15.27:0/3111387190 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.649+0000 7f52d599c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52d0159a20 0x7f52d0157660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36596/0 (socket says 172.21.15.27:36596) 2024-01-11T08:25:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.650+0000 7f52d599c700 1 -- 172.21.15.27:0/3111387190 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52d00b2730 msgr2=0x7f52d00aa260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.650+0000 7f52d599c700 1 --2- 172.21.15.27:0/3111387190 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52d00b2730 0x7f52d00aa260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.650+0000 7f52d599c700 1 -- 172.21.15.27:0/3111387190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52d0159650 msgr2=0x7f52d0169ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.650+0000 7f52d599c700 1 --2- 172.21.15.27:0/3111387190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52d0159650 0x7f52d0169ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.650+0000 7f52d599c700 1 -- 172.21.15.27:0/3111387190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52cc0094f0 con 0x7f52d0159a20 2024-01-11T08:25:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.650+0000 7f52d499a700 1 --2- 172.21.15.27:0/3111387190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52d0159650 0x7f52d0169ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.650+0000 7f52d599c700 1 --2- 172.21.15.27:0/3111387190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52d0159a20 0x7f52d0157660 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f52c800cc40 tx=0x7f52c8001a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.650+0000 7f52c57fa700 1 -- 172.21.15.27:0/3111387190 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8005df0 con 0x7f52d0159a20 2024-01-11T08:25:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.650+0000 7f52c57fa700 1 -- 172.21.15.27:0/3111387190 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c800a6a0 con 0x7f52d0159a20 2024-01-11T08:25:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.651+0000 7f52c57fa700 1 -- 172.21.15.27:0/3111387190 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c800aa60 con 0x7f52d0159a20 2024-01-11T08:25:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.651+0000 7f52d73ff700 1 -- 172.21.15.27:0/3111387190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d00ab490 con 0x7f52d0159a20 2024-01-11T08:25:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.651+0000 7f52d73ff700 1 -- 172.21.15.27:0/3111387190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d00aba50 con 0x7f52d0159a20 2024-01-11T08:25:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.652+0000 7f52d73ff700 1 -- 172.21.15.27:0/3111387190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d00718d0 con 0x7f52d0159a20 2024-01-11T08:25:07.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.653+0000 7f52c57fa700 1 -- 172.21.15.27:0/3111387190 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f52c80070f0 con 0x7f52d0159a20 2024-01-11T08:25:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.655+0000 7f52c57fa700 1 --2- 172.21.15.27:0/3111387190 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f52b4067050 0x7f52b4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.655+0000 7f52c57fa700 1 -- 172.21.15.27:0/3111387190 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6077+0+0 (secure 0 0 0) 0x7f52c8084830 con 0x7f52d0159a20 2024-01-11T08:25:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.655+0000 7f52d519b700 1 -- 172.21.15.27:0/3111387190 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f52b4067050 msgr2=0x7f52b4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.655+0000 7f52d519b700 1 --2- 172.21.15.27:0/3111387190 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f52b4067050 0x7f52b4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:07.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.658+0000 7f52c57fa700 1 -- 172.21.15.27:0/3111387190 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f52c80518b0 con 0x7f52d0159a20 2024-01-11T08:25:07.708 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:07.708+0000 7ff915edc1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:25:07.812 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:07.813+0000 7ff915edc1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:25:07.838 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:07.839+0000 7f1feb3201c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:25:07.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.856+0000 7f52d519b700 1 -- 172.21.15.27:0/3111387190 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f52b4067050 msgr2=0x7f52b4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:07.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.856+0000 7f52d519b700 1 --2- 172.21.15.27:0/3111387190 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f52b4067050 0x7f52b4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:07.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.911+0000 7f52d73ff700 1 -- 172.21.15.27:0/3111387190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52d00b0f80 con 0x7f52d0159a20 2024-01-11T08:25:07.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.911+0000 7f52c57fa700 1 -- 172.21.15.27:0/3111387190 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4594 (secure 0 0 0) 0x7f52c8054ed0 con 0x7f52d0159a20 2024-01-11T08:25:07.910 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:07.910 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5243,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2418922463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2418922463},{"type":"v1","addr":"172.21.15.27:6835","nonce":2418922463}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5246,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3919152683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3919152683},{"type":"v1","addr":"172.21.15.77:6836","nonce":3919152683}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5641,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/329572459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":329572459},{"type":"v1","addr":"172.21.15.77:6837","nonce":329572459}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:23:05.344753+0000","modified":"2024-01-11T08:23:08.864031+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/4173068012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":4173068012},{"type":"v1","addr":"172.21.15.27:6834","nonce":4173068012}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2024-01-11T08:25:07.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.914+0000 7f52d73ff700 1 -- 172.21.15.27:0/3111387190 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f52b4067050 msgr2=0x7f52b4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:07.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.914+0000 7f52d73ff700 1 --2- 172.21.15.27:0/3111387190 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f52b4067050 0x7f52b4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.914+0000 7f52d73ff700 1 -- 172.21.15.27:0/3111387190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52d0159a20 msgr2=0x7f52d0157660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:07.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.914+0000 7f52d73ff700 1 --2- 172.21.15.27:0/3111387190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52d0159a20 0x7f52d0157660 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f52c800cc40 tx=0x7f52c8001a50 comp rx=0 tx=0).stop 2024-01-11T08:25:07.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.914+0000 7f52d73ff700 1 -- 172.21.15.27:0/3111387190 shutdown_connections 2024-01-11T08:25:07.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.914+0000 7f52d73ff700 1 --2- 172.21.15.27:0/3111387190 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f52b4067050 0x7f52b4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.914+0000 7f52d73ff700 1 --2- 172.21.15.27:0/3111387190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52d0159a20 0x7f52d0157660 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.914+0000 7f52d73ff700 1 --2- 172.21.15.27:0/3111387190 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52d00b2730 0x7f52d00aa260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.914+0000 7f52d73ff700 1 --2- 172.21.15.27:0/3111387190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52d0159650 0x7f52d0169ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:07.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.914+0000 7f52d73ff700 1 -- 172.21.15.27:0/3111387190 >> 172.21.15.27:0/3111387190 conn(0x7f52d006b200 msgr2=0x7f52d0153ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:07.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.914+0000 7f52d73ff700 1 -- 172.21.15.27:0/3111387190 shutdown_connections 2024-01-11T08:25:07.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:07.914+0000 7f52d73ff700 1 -- 172.21.15.27:0/3111387190 wait complete. 2024-01-11T08:25:07.914 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-01-11T08:25:07.927 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:07.929+0000 7f1feb3201c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:25:07.931 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:25:08.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.139+0000 7f01f579d700 1 Processor -- start 2024-01-11T08:25:08.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.139+0000 7f01f579d700 1 -- start start 2024-01-11T08:25:08.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.140+0000 7f01f579d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01f00b0260 0x7f01f00b2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:08.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.140+0000 7f01f579d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f01f00b2840 0x7f01f00af1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:08.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.140+0000 7f01f579d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f01f015df30 0x7f01f00af720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:08.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.140+0000 7f01f579d700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f00b2d70 con 0x7f01f00b2840 2024-01-11T08:25:08.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.140+0000 7f01f579d700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f00afc60 con 0x7f01f015df30 2024-01-11T08:25:08.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.140+0000 7f01f579d700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f00cbcc0 con 0x7f01f00b0260 2024-01-11T08:25:08.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.141+0000 7f01ef7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f01f00b2840 0x7f01f00af1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:08.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.141+0000 7f01ef7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f01f00b2840 0x7f01f00af1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36612/0 (socket says 172.21.15.27:36612) 2024-01-11T08:25:08.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.141+0000 7f01ef7fe700 1 -- 172.21.15.27:0/564716372 learned_addr learned my addr 172.21.15.27:0/564716372 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:08.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.141+0000 7f01effff700 1 --2- 172.21.15.27:0/564716372 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01f00b0260 0x7f01f00b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:08.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.141+0000 7f01f4f9c700 1 --2- 172.21.15.27:0/564716372 >> v2:172.21.15.77:3300/0 conn(0x7f01f015df30 0x7f01f00af720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:08.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.141+0000 7f01ef7fe700 1 -- 172.21.15.27:0/564716372 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01f00b0260 msgr2=0x7f01f00b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:08.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.141+0000 7f01ef7fe700 1 --2- 172.21.15.27:0/564716372 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01f00b0260 0x7f01f00b2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.141+0000 7f01ef7fe700 1 -- 172.21.15.27:0/564716372 >> v2:172.21.15.77:3300/0 conn(0x7f01f015df30 msgr2=0x7f01f00af720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:08.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.141+0000 7f01ef7fe700 1 --2- 172.21.15.27:0/564716372 >> v2:172.21.15.77:3300/0 conn(0x7f01f015df30 0x7f01f00af720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.141+0000 7f01ef7fe700 1 -- 172.21.15.27:0/564716372 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f00cb390 con 0x7f01f00b2840 2024-01-11T08:25:08.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.141+0000 7f01ef7fe700 1 --2- 172.21.15.27:0/564716372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01f00b2840 0x7f01f00af1e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f01f00b5d00 tx=0x7f01e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd1bd103a602ea73 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:08.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.142+0000 7f01eeffd700 1 -- 172.21.15.27:0/564716372 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e400b6d0 con 0x7f01f00b2840 2024-01-11T08:25:08.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.142+0000 7f01eeffd700 1 -- 172.21.15.27:0/564716372 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e400b830 con 0x7f01f00b2840 2024-01-11T08:25:08.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.142+0000 7f01eeffd700 1 -- 172.21.15.27:0/564716372 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e400bbf0 con 0x7f01f00b2840 2024-01-11T08:25:08.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.142+0000 7f01f579d700 1 -- 172.21.15.27:0/564716372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01f00b2840 msgr2=0x7f01f00af1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:08.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.142+0000 7f01f579d700 1 --2- 172.21.15.27:0/564716372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01f00b2840 0x7f01f00af1e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f01f00b5d00 tx=0x7f01e4009250 comp rx=0 tx=0).stop 2024-01-11T08:25:08.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.142+0000 7f01f579d700 1 -- 172.21.15.27:0/564716372 shutdown_connections 2024-01-11T08:25:08.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.142+0000 7f01f579d700 1 --2- 172.21.15.27:0/564716372 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01f00b0260 0x7f01f00b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.142+0000 7f01f579d700 1 --2- 172.21.15.27:0/564716372 >> v2:172.21.15.77:3300/0 conn(0x7f01f015df30 0x7f01f00af720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.142+0000 7f01f579d700 1 --2- 172.21.15.27:0/564716372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01f00b2840 0x7f01f00af1e0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.142+0000 7f01f579d700 1 -- 172.21.15.27:0/564716372 >> 172.21.15.27:0/564716372 conn(0x7f01f006ed40 msgr2=0x7f01f006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:08.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.143+0000 7f01f579d700 1 -- 172.21.15.27:0/564716372 shutdown_connections 2024-01-11T08:25:08.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.143+0000 7f01f579d700 1 -- 172.21.15.27:0/564716372 wait complete. 2024-01-11T08:25:08.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.144+0000 7f01f579d700 1 Processor -- start 2024-01-11T08:25:08.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.144+0000 7f01f579d700 1 -- start start 2024-01-11T08:25:08.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.145+0000 7f01f579d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01f00b0260 0x7f01f01489f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:08.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.146+0000 7f01f579d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01f00b2840 0x7f01f00b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:08.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.146+0000 7f01f579d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f015df30 0x7f01f0159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:08.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.146+0000 7f01f579d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f00b5710 con 0x7f01f00b2840 2024-01-11T08:25:08.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.146+0000 7f01f579d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f00c7a00 con 0x7f01f015df30 2024-01-11T08:25:08.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.146+0000 7f01f579d700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f00c7ce0 con 0x7f01f00b0260 2024-01-11T08:25:08.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.146+0000 7f01ef7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01f00b2840 0x7f01f00b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:08.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.146+0000 7f01effff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01f00b0260 0x7f01f01489f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:08.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.146+0000 7f01effff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01f00b0260 0x7f01f01489f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:41794/0 (socket says 172.21.15.27:41794) 2024-01-11T08:25:08.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.146+0000 7f01effff700 1 -- 172.21.15.27:0/3511985600 learned_addr learned my addr 172.21.15.27:0/3511985600 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:08.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.146+0000 7f01f4f9c700 1 --2- 172.21.15.27:0/3511985600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f015df30 0x7f01f0159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:08.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.146+0000 7f01effff700 1 -- 172.21.15.27:0/3511985600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f015df30 msgr2=0x7f01f0159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:08.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.146+0000 7f01effff700 1 --2- 172.21.15.27:0/3511985600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f015df30 0x7f01f0159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.146+0000 7f01effff700 1 -- 172.21.15.27:0/3511985600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01f00b2840 msgr2=0x7f01f00b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:08.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.146+0000 7f01effff700 1 --2- 172.21.15.27:0/3511985600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01f00b2840 0x7f01f00b51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.146+0000 7f01effff700 1 -- 172.21.15.27:0/3511985600 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e40094f0 con 0x7f01f00b0260 2024-01-11T08:25:08.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.147+0000 7f01effff700 1 --2- 172.21.15.27:0/3511985600 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01f00b0260 0x7f01f01489f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f01e000ec80 tx=0x7f01e00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:08.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.147+0000 7f01d7fff700 1 -- 172.21.15.27:0/3511985600 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e000c850 con 0x7f01f00b0260 2024-01-11T08:25:08.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.147+0000 7f01d7fff700 1 -- 172.21.15.27:0/3511985600 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e000c9b0 con 0x7f01f00b0260 2024-01-11T08:25:08.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.147+0000 7f01f579d700 1 -- 172.21.15.27:0/3511985600 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f00c7fc0 con 0x7f01f00b0260 2024-01-11T08:25:08.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.147+0000 7f01d7fff700 1 -- 172.21.15.27:0/3511985600 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e000cd50 con 0x7f01f00b0260 2024-01-11T08:25:08.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.147+0000 7f01f579d700 1 -- 172.21.15.27:0/3511985600 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f00c8580 con 0x7f01f00b0260 2024-01-11T08:25:08.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.148+0000 7f01f579d700 1 -- 172.21.15.27:0/3511985600 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f00c88f0 con 0x7f01f00b0260 2024-01-11T08:25:08.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.149+0000 7f01d7fff700 1 -- 172.21.15.27:0/3511985600 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f01e0013680 con 0x7f01f00b0260 2024-01-11T08:25:08.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.150+0000 7f01d7fff700 1 --2- 172.21.15.27:0/3511985600 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f01dc067050 0x7f01dc069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:08.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.150+0000 7f01d7fff700 1 -- 172.21.15.27:0/3511985600 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6077+0+0 (secure 0 0 0) 0x7f01e0084ab0 con 0x7f01f00b0260 2024-01-11T08:25:08.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.150+0000 7f01ef7fe700 1 -- 172.21.15.27:0/3511985600 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f01dc067050 msgr2=0x7f01dc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:08.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.150+0000 7f01ef7fe700 1 --2- 172.21.15.27:0/3511985600 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f01dc067050 0x7f01dc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:08.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.154+0000 7f01d7fff700 1 -- 172.21.15.27:0/3511985600 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f01e0051a00 con 0x7f01f00b0260 2024-01-11T08:25:08.183 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:08.184+0000 7ff915edc1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:25:08.278 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:08.279+0000 7ff915edc1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:25:08.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.351+0000 7f01ef7fe700 1 -- 172.21.15.27:0/3511985600 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f01dc067050 msgr2=0x7f01dc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:08.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.351+0000 7f01ef7fe700 1 --2- 172.21.15.27:0/3511985600 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f01dc067050 0x7f01dc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:08.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.404+0000 7f01f579d700 1 -- 172.21.15.27:0/3511985600 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01f00c7e20 con 0x7f01f00b0260 2024-01-11T08:25:08.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.405+0000 7f01d7fff700 1 -- 172.21.15.27:0/3511985600 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4594 (secure 0 0 0) 0x7f01f00c7e20 con 0x7f01f00b0260 2024-01-11T08:25:08.404 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:08.404 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5243,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2418922463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2418922463},{"type":"v1","addr":"172.21.15.27:6835","nonce":2418922463}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5246,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3919152683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3919152683},{"type":"v1","addr":"172.21.15.77:6836","nonce":3919152683}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5641,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/329572459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":329572459},{"type":"v1","addr":"172.21.15.77:6837","nonce":329572459}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:23:05.344753+0000","modified":"2024-01-11T08:23:08.864031+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/4173068012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":4173068012},{"type":"v1","addr":"172.21.15.27:6834","nonce":4173068012}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2024-01-11T08:25:08.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.407+0000 7f01f579d700 1 -- 172.21.15.27:0/3511985600 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f01dc067050 msgr2=0x7f01dc069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:08.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.407+0000 7f01f579d700 1 --2- 172.21.15.27:0/3511985600 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f01dc067050 0x7f01dc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.407+0000 7f01f579d700 1 -- 172.21.15.27:0/3511985600 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01f00b0260 msgr2=0x7f01f01489f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:08.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.407+0000 7f01f579d700 1 --2- 172.21.15.27:0/3511985600 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01f00b0260 0x7f01f01489f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f01e000ec80 tx=0x7f01e00091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.407+0000 7f01f579d700 1 -- 172.21.15.27:0/3511985600 shutdown_connections 2024-01-11T08:25:08.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.407+0000 7f01f579d700 1 --2- 172.21.15.27:0/3511985600 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f01dc067050 0x7f01dc069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.407+0000 7f01f579d700 1 --2- 172.21.15.27:0/3511985600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f015df30 0x7f01f0159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.407+0000 7f01f579d700 1 --2- 172.21.15.27:0/3511985600 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01f00b0260 0x7f01f01489f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.407+0000 7f01f579d700 1 --2- 172.21.15.27:0/3511985600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01f00b2840 0x7f01f00b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.407+0000 7f01f579d700 1 -- 172.21.15.27:0/3511985600 >> 172.21.15.27:0/3511985600 conn(0x7f01f006ed40 msgr2=0x7f01f00cdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:08.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.408+0000 7f01f579d700 1 -- 172.21.15.27:0/3511985600 shutdown_connections 2024-01-11T08:25:08.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.408+0000 7f01f579d700 1 -- 172.21.15.27:0/3511985600 wait complete. 2024-01-11T08:25:08.408 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-01-11T08:25:08.425 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:25:08.471 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:08.472+0000 7f1feb3201c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:25:08.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.631+0000 7fede4fff700 1 Processor -- start 2024-01-11T08:25:08.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.632+0000 7fede4fff700 1 -- start start 2024-01-11T08:25:08.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.633+0000 7fede4fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fede00c8620 0x7fede00af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.633+0000 7fede4fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fede0069d40 0x7fede00b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.633+0000 7fede4fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fede00c89f0 0x7fede00b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.633+0000 7fede4fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fede0157390 con 0x7fede00c89f0 2024-01-11T08:25:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.633+0000 7fede4fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fede00afeb0 con 0x7fede0069d40 2024-01-11T08:25:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.633+0000 7fede4fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede006a200 con 0x7fede00c8620 2024-01-11T08:25:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.634+0000 7fedde59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fede00c8620 0x7fede00af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.634+0000 7fedded9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fede00c89f0 0x7fede00b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.634+0000 7fedddd9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fede0069d40 0x7fede00b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.634+0000 7fedde59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fede00c8620 0x7fede00af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:41796/0 (socket says 172.21.15.27:41796) 2024-01-11T08:25:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.634+0000 7fedde59c700 1 -- 172.21.15.27:0/1300291707 learned_addr learned my addr 172.21.15.27:0/1300291707 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.634+0000 7fedded9d700 1 -- 172.21.15.27:0/1300291707 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fede00c8620 msgr2=0x7fede00af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.634+0000 7fedded9d700 1 --2- 172.21.15.27:0/1300291707 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fede00c8620 0x7fede00af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.634+0000 7fedded9d700 1 -- 172.21.15.27:0/1300291707 >> v2:172.21.15.77:3300/0 conn(0x7fede0069d40 msgr2=0x7fede00b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.634+0000 7fedded9d700 1 --2- 172.21.15.27:0/1300291707 >> v2:172.21.15.77:3300/0 conn(0x7fede0069d40 0x7fede00b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.634+0000 7fedded9d700 1 -- 172.21.15.27:0/1300291707 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede0156d40 con 0x7fede00c89f0 2024-01-11T08:25:08.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.634+0000 7fedddd9b700 1 --2- 172.21.15.27:0/1300291707 >> v2:172.21.15.77:3300/0 conn(0x7fede0069d40 0x7fede00b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:08.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.635+0000 7fedded9d700 1 --2- 172.21.15.27:0/1300291707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fede00c89f0 0x7fede00b0810 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fede0069180 tx=0x7fedd8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26176caf7f63e39a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:08.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.635+0000 7feddd59a700 1 -- 172.21.15.27:0/1300291707 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd800e070 con 0x7fede00c89f0 2024-01-11T08:25:08.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.635+0000 7feddd59a700 1 -- 172.21.15.27:0/1300291707 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd800bab0 con 0x7fede00c89f0 2024-01-11T08:25:08.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.635+0000 7feddd59a700 1 -- 172.21.15.27:0/1300291707 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd800be70 con 0x7fede00c89f0 2024-01-11T08:25:08.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.636+0000 7fede4fff700 1 -- 172.21.15.27:0/1300291707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fede00c89f0 msgr2=0x7fede00b0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:08.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.636+0000 7fede4fff700 1 --2- 172.21.15.27:0/1300291707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fede00c89f0 0x7fede00b0810 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fede0069180 tx=0x7fedd8009780 comp rx=0 tx=0).stop 2024-01-11T08:25:08.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.636+0000 7fede4fff700 1 -- 172.21.15.27:0/1300291707 shutdown_connections 2024-01-11T08:25:08.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.636+0000 7fede4fff700 1 --2- 172.21.15.27:0/1300291707 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fede00c8620 0x7fede00af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.636+0000 7fede4fff700 1 --2- 172.21.15.27:0/1300291707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fede00c89f0 0x7fede00b0810 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.636+0000 7fede4fff700 1 --2- 172.21.15.27:0/1300291707 >> v2:172.21.15.77:3300/0 conn(0x7fede0069d40 0x7fede00b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.636+0000 7fede4fff700 1 -- 172.21.15.27:0/1300291707 >> 172.21.15.27:0/1300291707 conn(0x7fede00680e0 msgr2=0x7fede0069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:08.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.636+0000 7fede4fff700 1 -- 172.21.15.27:0/1300291707 shutdown_connections 2024-01-11T08:25:08.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.636+0000 7fede4fff700 1 -- 172.21.15.27:0/1300291707 wait complete. 2024-01-11T08:25:08.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.638+0000 7fede4fff700 1 Processor -- start 2024-01-11T08:25:08.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.638+0000 7fede4fff700 1 -- start start 2024-01-11T08:25:08.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.639+0000 7fede4fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fede0069d40 0x7fede00caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:08.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.640+0000 7fede4fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fede00c8620 0x7fede00c9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:08.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.640+0000 7fede4fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fede00c89f0 0x7fede0151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:08.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.640+0000 7fede4fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede015a100 con 0x7fede00c8620 2024-01-11T08:25:08.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.640+0000 7fede4fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede00cb1a0 con 0x7fede00c89f0 2024-01-11T08:25:08.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.640+0000 7fede4fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede00cb480 con 0x7fede0069d40 2024-01-11T08:25:08.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.640+0000 7fedddd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fede00c8620 0x7fede00c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:08.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.640+0000 7fedddd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fede00c8620 0x7fede00c9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36638/0 (socket says 172.21.15.27:36638) 2024-01-11T08:25:08.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.640+0000 7fedddd9b700 1 -- 172.21.15.27:0/4184261889 learned_addr learned my addr 172.21.15.27:0/4184261889 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:08.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.640+0000 7fedde59c700 1 --2- 172.21.15.27:0/4184261889 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fede0069d40 0x7fede00caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:08.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.640+0000 7fedded9d700 1 --2- 172.21.15.27:0/4184261889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fede00c89f0 0x7fede0151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:08.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.640+0000 7fedddd9b700 1 -- 172.21.15.27:0/4184261889 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fede0069d40 msgr2=0x7fede00caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:08.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.640+0000 7fedddd9b700 1 --2- 172.21.15.27:0/4184261889 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fede0069d40 0x7fede00caba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.640+0000 7fedddd9b700 1 -- 172.21.15.27:0/4184261889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fede00c89f0 msgr2=0x7fede0151a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:08.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.640+0000 7fedddd9b700 1 --2- 172.21.15.27:0/4184261889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fede00c89f0 0x7fede0151a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.640+0000 7fedddd9b700 1 -- 172.21.15.27:0/4184261889 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd4009420 con 0x7fede00c8620 2024-01-11T08:25:08.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.640+0000 7fedded9d700 1 --2- 172.21.15.27:0/4184261889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fede00c89f0 0x7fede0151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:08.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.641+0000 7fedddd9b700 1 --2- 172.21.15.27:0/4184261889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fede00c8620 0x7fede00c9040 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fedd400ec80 tx=0x7fedd40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:08.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.641+0000 7fedce7fc700 1 -- 172.21.15.27:0/4184261889 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd400c850 con 0x7fede00c8620 2024-01-11T08:25:08.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.641+0000 7fedce7fc700 1 -- 172.21.15.27:0/4184261889 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd400c9b0 con 0x7fede00c8620 2024-01-11T08:25:08.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.641+0000 7fede4fff700 1 -- 172.21.15.27:0/4184261889 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd8009a20 con 0x7fede00c8620 2024-01-11T08:25:08.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.641+0000 7fedce7fc700 1 -- 172.21.15.27:0/4184261889 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd400cd50 con 0x7fede00c8620 2024-01-11T08:25:08.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.641+0000 7fede4fff700 1 -- 172.21.15.27:0/4184261889 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede00c9580 con 0x7fede00c8620 2024-01-11T08:25:08.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.642+0000 7fede4fff700 1 -- 172.21.15.27:0/4184261889 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede015a320 con 0x7fede00c8620 2024-01-11T08:25:08.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.643+0000 7fedce7fc700 1 -- 172.21.15.27:0/4184261889 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fedd4013680 con 0x7fede00c8620 2024-01-11T08:25:08.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.644+0000 7fedce7fc700 1 --2- 172.21.15.27:0/4184261889 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fedc0067050 0x7fedc0069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:08.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.645+0000 7fedce7fc700 1 -- 172.21.15.27:0/4184261889 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6077+0+0 (secure 0 0 0) 0x7fedd4084db0 con 0x7fede00c8620 2024-01-11T08:25:08.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.645+0000 7fedde59c700 1 -- 172.21.15.27:0/4184261889 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fedc0067050 msgr2=0x7fedc0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:08.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.645+0000 7fedde59c700 1 --2- 172.21.15.27:0/4184261889 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fedc0067050 0x7fedc0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:08.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.647+0000 7fedce7fc700 1 -- 172.21.15.27:0/4184261889 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fedd4051d80 con 0x7fede00c8620 2024-01-11T08:25:08.656 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:08.658+0000 7f1feb3201c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:25:08.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.846+0000 7fedde59c700 1 -- 172.21.15.27:0/4184261889 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fedc0067050 msgr2=0x7fedc0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:08.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.846+0000 7fedde59c700 1 --2- 172.21.15.27:0/4184261889 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fedc0067050 0x7fedc0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:08.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.852+0000 7fede4fff700 1 -- 172.21.15.27:0/4184261889 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fede015a320 con 0x7fede00c8620 2024-01-11T08:25:08.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.853+0000 7fedce7fc700 1 -- 172.21.15.27:0/4184261889 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+15871 (secure 0 0 0) 0x7fede015a320 con 0x7fede00c8620 2024-01-11T08:25:08.852 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:08.852 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":63,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:25:04.748651+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:23:03.012171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-11T08:23:04.337714+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.7","mappings":[{"from":0,"to":4}]},{"pgid":"7.1b","mappings":[{"from":0,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/3145516755":"2024-01-12T08:23:21.771322+0000","172.21.15.77:6833/41381":"2024-01-12T08:23:21.771322+0000","172.21.15.77:6832/41381":"2024-01-12T08:23:21.771322+0000","172.21.15.77:0/434568822":"2024-01-12T08:23:21.771322+0000","172.21.15.77:0/1366514973":"2024-01-12T08:23:21.771322+0000","172.21.15.77:0/3742894353":"2024-01-12T08:23:21.771322+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:25:08.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.855+0000 7fede4fff700 1 -- 172.21.15.27:0/4184261889 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fedc0067050 msgr2=0x7fedc0069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:08.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.855+0000 7fede4fff700 1 --2- 172.21.15.27:0/4184261889 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fedc0067050 0x7fedc0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.855+0000 7fede4fff700 1 -- 172.21.15.27:0/4184261889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fede00c8620 msgr2=0x7fede00c9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:08.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.855+0000 7fede4fff700 1 --2- 172.21.15.27:0/4184261889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fede00c8620 0x7fede00c9040 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fedd400ec80 tx=0x7fedd40091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.856+0000 7fede4fff700 1 -- 172.21.15.27:0/4184261889 shutdown_connections 2024-01-11T08:25:08.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.856+0000 7fede4fff700 1 --2- 172.21.15.27:0/4184261889 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fedc0067050 0x7fedc0069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.856+0000 7fede4fff700 1 --2- 172.21.15.27:0/4184261889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fede00c89f0 0x7fede0151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.856+0000 7fede4fff700 1 --2- 172.21.15.27:0/4184261889 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fede0069d40 0x7fede00caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.856+0000 7fede4fff700 1 --2- 172.21.15.27:0/4184261889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fede00c8620 0x7fede00c9040 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:08.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.856+0000 7fede4fff700 1 -- 172.21.15.27:0/4184261889 >> 172.21.15.27:0/4184261889 conn(0x7fede00680e0 msgr2=0x7fede00b02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:08.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.856+0000 7fede4fff700 1 -- 172.21.15.27:0/4184261889 shutdown_connections 2024-01-11T08:25:08.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:08.856+0000 7fede4fff700 1 -- 172.21.15.27:0/4184261889 wait complete. 2024-01-11T08:25:08.871 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:08.872+0000 7ff915edc1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:25:08.873 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-11T08:25:09.033 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:09.034+0000 7f1feb3201c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:25:09.071 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:09.072+0000 7ff915edc1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:25:09.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.077+0000 7f35341ff700 1 Processor -- start 2024-01-11T08:25:09.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.077+0000 7f35341ff700 1 -- start start 2024-01-11T08:25:09.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.079+0000 7f35341ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f352c1596a0 0x7f352c0abd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:09.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.079+0000 7f35341ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f352c067f90 0x7f352c068410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.079+0000 7f35341ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c0ac250 con 0x7f352c068ba0 2024-01-11T08:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.079+0000 7f35341ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f352c068950 con 0x7f352c1596a0 2024-01-11T08:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.079+0000 7f35341ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f352c067b00 con 0x7f352c067f90 2024-01-11T08:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.079+0000 7f3531f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f352c1596a0 0x7f352c0abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.079+0000 7f3531f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f352c1596a0 0x7f352c0abd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36652/0 (socket says 172.21.15.27:36652) 2024-01-11T08:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.079+0000 7f3531f9b700 1 -- 172.21.15.27:0/1514568881 learned_addr learned my addr 172.21.15.27:0/1514568881 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.079+0000 7f353279c700 1 --2- 172.21.15.27:0/1514568881 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f352c067f90 0x7f352c068410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.080+0000 7f3531f9b700 1 -- 172.21.15.27:0/1514568881 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f352c067f90 msgr2=0x7f352c068410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.080+0000 7f3531f9b700 1 --2- 172.21.15.27:0/1514568881 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f352c067f90 0x7f352c068410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.080+0000 7f3531f9b700 1 -- 172.21.15.27:0/1514568881 >> v1:172.21.15.77:6789/0 conn(0x7f352c068ba0 legacy=0x7f352c0af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:09.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.080+0000 7f3531f9b700 1 -- 172.21.15.27:0/1514568881 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352c06bf60 con 0x7f352c1596a0 2024-01-11T08:25:09.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.080+0000 7f3531f9b700 1 --2- 172.21.15.27:0/1514568881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f352c1596a0 0x7f352c0abd10 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f35280099b0 tx=0x7f3528009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b41191b90aba83d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:09.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.080+0000 7f3530f99700 1 -- 172.21.15.27:0/1514568881 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3528009ea0 con 0x7f352c1596a0 2024-01-11T08:25:09.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.080+0000 7f3530f99700 1 -- 172.21.15.27:0/1514568881 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352800b860 con 0x7f352c1596a0 2024-01-11T08:25:09.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.080+0000 7f3530f99700 1 -- 172.21.15.27:0/1514568881 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352800bc00 con 0x7f352c1596a0 2024-01-11T08:25:09.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.081+0000 7f35341ff700 1 -- 172.21.15.27:0/1514568881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f352c1596a0 msgr2=0x7f352c0abd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:09.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.081+0000 7f35341ff700 1 --2- 172.21.15.27:0/1514568881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f352c1596a0 0x7f352c0abd10 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f35280099b0 tx=0x7f3528009250 comp rx=0 tx=0).stop 2024-01-11T08:25:09.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.081+0000 7f35341ff700 1 -- 172.21.15.27:0/1514568881 shutdown_connections 2024-01-11T08:25:09.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.081+0000 7f35341ff700 1 --2- 172.21.15.27:0/1514568881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f352c1596a0 0x7f352c0abd10 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:09.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.081+0000 7f35341ff700 1 --2- 172.21.15.27:0/1514568881 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f352c067f90 0x7f352c068410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:09.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.081+0000 7f35341ff700 1 -- 172.21.15.27:0/1514568881 >> 172.21.15.27:0/1514568881 conn(0x7f352c06a8b0 msgr2=0x7f352c0ae520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:09.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.081+0000 7f35341ff700 1 -- 172.21.15.27:0/1514568881 shutdown_connections 2024-01-11T08:25:09.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.081+0000 7f35341ff700 1 -- 172.21.15.27:0/1514568881 wait complete. 2024-01-11T08:25:09.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.083+0000 7f35341ff700 1 Processor -- start 2024-01-11T08:25:09.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.083+0000 7f35341ff700 1 -- start start 2024-01-11T08:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.084+0000 7f35341ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f352c067f90 0x7f352c158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.084+0000 7f35341ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f352c068ba0 0x7f352c0cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.084+0000 7f35341ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f352c1596a0 0x7f352c1503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.084+0000 7f35341ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352c159dc0 con 0x7f352c1596a0 2024-01-11T08:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.084+0000 7f35341ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352c15a0a0 con 0x7f352c067f90 2024-01-11T08:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.084+0000 7f35341ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f352c15a380 con 0x7f352c068ba0 2024-01-11T08:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.084+0000 7f3531f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f352c067f90 0x7f352c158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.084+0000 7f353279c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f352c1596a0 0x7f352c1503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.084+0000 7f3531f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f352c067f90 0x7f352c158290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:48554/0 (socket says 172.21.15.27:48554) 2024-01-11T08:25:09.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.084+0000 7f3531f9b700 1 -- 172.21.15.27:0/6265058 learned_addr learned my addr 172.21.15.27:0/6265058 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:09.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.084+0000 7f353179a700 1 --2- 172.21.15.27:0/6265058 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f352c068ba0 0x7f352c0cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.084+0000 7f3531f9b700 1 -- 172.21.15.27:0/6265058 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f352c068ba0 msgr2=0x7f352c0cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.084+0000 7f3531f9b700 1 --2- 172.21.15.27:0/6265058 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f352c068ba0 0x7f352c0cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.084+0000 7f3531f9b700 1 -- 172.21.15.27:0/6265058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f352c1596a0 msgr2=0x7f352c1503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.084+0000 7f3531f9b700 1 --2- 172.21.15.27:0/6265058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f352c1596a0 0x7f352c1503c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.084+0000 7f3531f9b700 1 -- 172.21.15.27:0/6265058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35280094f0 con 0x7f352c067f90 2024-01-11T08:25:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.085+0000 7f353279c700 1 --2- 172.21.15.27:0/6265058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f352c1596a0 0x7f352c1503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.085+0000 7f3531f9b700 1 --2- 172.21.15.27:0/6265058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f352c067f90 0x7f352c158290 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f352c06d500 tx=0x7f3528017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.085+0000 7f3521ffb700 1 -- 172.21.15.27:0/6265058 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3528000d10 con 0x7f352c067f90 2024-01-11T08:25:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.085+0000 7f3521ffb700 1 -- 172.21.15.27:0/6265058 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352800bbe0 con 0x7f352c067f90 2024-01-11T08:25:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.085+0000 7f35341ff700 1 -- 172.21.15.27:0/6265058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f352c15a660 con 0x7f352c067f90 2024-01-11T08:25:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.085+0000 7f3521ffb700 1 -- 172.21.15.27:0/6265058 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3528011670 con 0x7f352c067f90 2024-01-11T08:25:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.085+0000 7f35341ff700 1 -- 172.21.15.27:0/6265058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f352c159a70 con 0x7f352c067f90 2024-01-11T08:25:09.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.086+0000 7f35341ff700 1 -- 172.21.15.27:0/6265058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f352c04c3c0 con 0x7f352c067f90 2024-01-11T08:25:09.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.087+0000 7f3521ffb700 1 -- 172.21.15.27:0/6265058 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f3528011aa0 con 0x7f352c067f90 2024-01-11T08:25:09.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.088+0000 7f3521ffb700 1 --2- 172.21.15.27:0/6265058 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3510067050 0x7f3510069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:09.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.088+0000 7f3521ffb700 1 -- 172.21.15.27:0/6265058 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6077+0+0 (secure 0 0 0) 0x7f3528085fd0 con 0x7f352c067f90 2024-01-11T08:25:09.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.088+0000 7f353179a700 1 -- 172.21.15.27:0/6265058 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3510067050 msgr2=0x7f3510069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:09.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.088+0000 7f353179a700 1 --2- 172.21.15.27:0/6265058 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3510067050 0x7f3510069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:09.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.092+0000 7f3521ffb700 1 -- 172.21.15.27:0/6265058 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3528052f20 con 0x7f352c067f90 2024-01-11T08:25:09.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.289+0000 7f353179a700 1 -- 172.21.15.27:0/6265058 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3510067050 msgr2=0x7f3510069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:09.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.289+0000 7f353179a700 1 --2- 172.21.15.27:0/6265058 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3510067050 0x7f3510069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:09.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.345+0000 7f35341ff700 1 -- 172.21.15.27:0/6265058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f352c06c240 con 0x7f352c067f90 2024-01-11T08:25:09.345 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-11T08:25:09.346+0000 7f27065f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-11T08:25:09.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.354+0000 7f3521ffb700 1 -- 172.21.15.27:0/6265058 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3528056540 con 0x7f352c067f90 2024-01-11T08:25:09.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.357+0000 7f35341ff700 1 -- 172.21.15.27:0/6265058 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3510067050 msgr2=0x7f3510069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:09.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.357+0000 7f35341ff700 1 --2- 172.21.15.27:0/6265058 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3510067050 0x7f3510069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:09.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.357+0000 7f35341ff700 1 -- 172.21.15.27:0/6265058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f352c067f90 msgr2=0x7f352c158290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:09.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.357+0000 7f35341ff700 1 --2- 172.21.15.27:0/6265058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f352c067f90 0x7f352c158290 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f352c06d500 tx=0x7f3528017000 comp rx=0 tx=0).stop 2024-01-11T08:25:09.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.357+0000 7f35341ff700 1 -- 172.21.15.27:0/6265058 shutdown_connections 2024-01-11T08:25:09.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.357+0000 7f35341ff700 1 --2- 172.21.15.27:0/6265058 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f3510067050 0x7f3510069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:09.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.357+0000 7f35341ff700 1 --2- 172.21.15.27:0/6265058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f352c1596a0 0x7f352c1503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:09.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.357+0000 7f35341ff700 1 --2- 172.21.15.27:0/6265058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f352c067f90 0x7f352c158290 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:09.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.357+0000 7f35341ff700 1 --2- 172.21.15.27:0/6265058 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f352c068ba0 0x7f352c0cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:09.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.357+0000 7f35341ff700 1 -- 172.21.15.27:0/6265058 >> 172.21.15.27:0/6265058 conn(0x7f352c06a8b0 msgr2=0x7f352c156eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:09.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.357+0000 7f35341ff700 1 -- 172.21.15.27:0/6265058 shutdown_connections 2024-01-11T08:25:09.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.357+0000 7f35341ff700 1 -- 172.21.15.27:0/6265058 wait complete. 2024-01-11T08:25:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-11T08:25:09.375 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-11T08:25:09.414 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T08:25:09.457 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:09.459+0000 7f1feb3201c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:25:09.482 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:09.483+0000 7ff915edc1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:25:09.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.584+0000 7f6bb2fff700 1 Processor -- start 2024-01-11T08:25:09.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.584+0000 7f6bb2fff700 1 -- start start 2024-01-11T08:25:09.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.585+0000 7f6bb2fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6bac142dc0 0x7f6bac0cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:09.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.585+0000 7f6bb2fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6bac0cb2e0 0x7f6bac0cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:09.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.585+0000 7f6bb2fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bac0ccc70 con 0x7f6bac143190 2024-01-11T08:25:09.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.585+0000 7f6bb2fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bac15df30 con 0x7f6bac0cb2e0 2024-01-11T08:25:09.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.585+0000 7f6bb2fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac15d320 con 0x7f6bac142dc0 2024-01-11T08:25:09.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.586+0000 7f6bb0d9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6bac142dc0 0x7f6bac0cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:09.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.586+0000 7f6babfff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6bac143190 0x7f6bac0cc020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:45720/0 (socket says 172.21.15.27:45720) 2024-01-11T08:25:09.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.586+0000 7f6babfff700 1 -- 172.21.15.27:0/399141307 learned_addr learned my addr 172.21.15.27:0/399141307 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:09.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.586+0000 7f6bb159c700 1 --2- 172.21.15.27:0/399141307 >> v2:172.21.15.77:3300/0 conn(0x7f6bac0cb2e0 0x7f6bac0cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:09.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.586+0000 7f6bab7fe700 1 -- 172.21.15.27:0/399141307 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692879368 0 0) 0x7f6bac0ccc70 con 0x7f6bac143190 2024-01-11T08:25:09.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.586+0000 7f6bab7fe700 1 -- 172.21.15.27:0/399141307 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b9c003570 con 0x7f6bac143190 2024-01-11T08:25:09.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.586+0000 7f6bb0d9b700 1 -- 172.21.15.27:0/399141307 >> v2:172.21.15.77:3300/0 conn(0x7f6bac0cb2e0 msgr2=0x7f6bac0cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:09.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.586+0000 7f6bb0d9b700 1 --2- 172.21.15.27:0/399141307 >> v2:172.21.15.77:3300/0 conn(0x7f6bac0cb2e0 0x7f6bac0cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:09.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.586+0000 7f6bb0d9b700 1 -- 172.21.15.27:0/399141307 >> v1:172.21.15.77:6789/0 conn(0x7f6bac143190 legacy=0x7f6bac0cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:09.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.586+0000 7f6bb0d9b700 1 -- 172.21.15.27:0/399141307 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bac0cb700 con 0x7f6bac142dc0 2024-01-11T08:25:09.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.587+0000 7f6bb0d9b700 1 --2- 172.21.15.27:0/399141307 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6bac142dc0 0x7f6bac0cada0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00098e0 tx=0x7f6ba00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2e08c017c8ff132 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:09.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.587+0000 7f6bab7fe700 1 -- 172.21.15.27:0/399141307 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba0011630 con 0x7f6bac142dc0 2024-01-11T08:25:09.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.587+0000 7f6bab7fe700 1 -- 172.21.15.27:0/399141307 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba0011790 con 0x7f6bac142dc0 2024-01-11T08:25:09.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.587+0000 7f6bab7fe700 1 -- 172.21.15.27:0/399141307 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba0011b30 con 0x7f6bac142dc0 2024-01-11T08:25:09.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.588+0000 7f6bb2fff700 1 -- 172.21.15.27:0/399141307 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6bac142dc0 msgr2=0x7f6bac0cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:09.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.588+0000 7f6bb2fff700 1 --2- 172.21.15.27:0/399141307 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6bac142dc0 0x7f6bac0cada0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00098e0 tx=0x7f6ba00091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:09.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.588+0000 7f6bb2fff700 1 -- 172.21.15.27:0/399141307 shutdown_connections 2024-01-11T08:25:09.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.588+0000 7f6bb2fff700 1 --2- 172.21.15.27:0/399141307 >> v2:172.21.15.77:3300/0 conn(0x7f6bac0cb2e0 0x7f6bac0cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:09.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.588+0000 7f6bb2fff700 1 --2- 172.21.15.27:0/399141307 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6bac142dc0 0x7f6bac0cada0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:09.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.588+0000 7f6bb2fff700 1 -- 172.21.15.27:0/399141307 >> 172.21.15.27:0/399141307 conn(0x7f6bac063ca0 msgr2=0x7f6bac06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:09.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.588+0000 7f6bb2fff700 1 -- 172.21.15.27:0/399141307 shutdown_connections 2024-01-11T08:25:09.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.588+0000 7f6bb2fff700 1 -- 172.21.15.27:0/399141307 wait complete. 2024-01-11T08:25:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.589+0000 7f6bb2fff700 1 Processor -- start 2024-01-11T08:25:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.590+0000 7f6bb2fff700 1 -- start start 2024-01-11T08:25:09.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.591+0000 7f6bb2fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bac0cb2e0 0x7f6bac0c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.591+0000 7f6bb2fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bac142dc0 0x7f6bac140f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.591+0000 7f6bb2fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6bac143190 0x7f6bac0b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.591+0000 7f6bb2fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac0b20b0 con 0x7f6bac142dc0 2024-01-11T08:25:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.591+0000 7f6bb2fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac0c59a0 con 0x7f6bac0cb2e0 2024-01-11T08:25:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.591+0000 7f6bb2fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac0b1af0 con 0x7f6bac143190 2024-01-11T08:25:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.591+0000 7f6babfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bac142dc0 0x7f6bac140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.591+0000 7f6bb0d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bac0cb2e0 0x7f6bac0c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.591+0000 7f6babfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bac142dc0 0x7f6bac140f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36682/0 (socket says 172.21.15.27:36682) 2024-01-11T08:25:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.591+0000 7f6bb0d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bac0cb2e0 0x7f6bac0c5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:48600/0 (socket says 172.21.15.27:48600) 2024-01-11T08:25:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.591+0000 7f6babfff700 1 -- 172.21.15.27:0/1310535623 learned_addr learned my addr 172.21.15.27:0/1310535623 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.591+0000 7f6bb159c700 1 --2- 172.21.15.27:0/1310535623 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6bac143190 0x7f6bac0b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.591+0000 7f6bb159c700 1 -- 172.21.15.27:0/1310535623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bac0cb2e0 msgr2=0x7f6bac0c5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.591+0000 7f6bb159c700 1 --2- 172.21.15.27:0/1310535623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bac0cb2e0 0x7f6bac0c5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.591+0000 7f6bb159c700 1 -- 172.21.15.27:0/1310535623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bac142dc0 msgr2=0x7f6bac140f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.591+0000 7f6bb159c700 1 --2- 172.21.15.27:0/1310535623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bac142dc0 0x7f6bac140f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.591+0000 7f6bb159c700 1 -- 172.21.15.27:0/1310535623 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba0009420 con 0x7f6bac143190 2024-01-11T08:25:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.592+0000 7f6bb159c700 1 --2- 172.21.15.27:0/1310535623 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6bac143190 0x7f6bac0b1180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4009e50 tx=0x7f6ba400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:09.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.592+0000 7f6ba8ff9700 1 -- 172.21.15.27:0/1310535623 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba400caa0 con 0x7f6bac143190 2024-01-11T08:25:09.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.592+0000 7f6ba8ff9700 1 -- 172.21.15.27:0/1310535623 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba400f040 con 0x7f6bac143190 2024-01-11T08:25:09.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.592+0000 7f6bb2fff700 1 -- 172.21.15.27:0/1310535623 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bac0b1dd0 con 0x7f6bac143190 2024-01-11T08:25:09.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.592+0000 7f6ba8ff9700 1 -- 172.21.15.27:0/1310535623 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba400ce50 con 0x7f6bac143190 2024-01-11T08:25:09.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.592+0000 7f6bb2fff700 1 -- 172.21.15.27:0/1310535623 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bac0c6370 con 0x7f6bac143190 2024-01-11T08:25:09.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.593+0000 7f6bb2fff700 1 -- 172.21.15.27:0/1310535623 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bac0c5490 con 0x7f6bac143190 2024-01-11T08:25:09.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.595+0000 7f6ba8ff9700 1 -- 172.21.15.27:0/1310535623 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 26) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f6ba4014690 con 0x7f6bac143190 2024-01-11T08:25:09.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.596+0000 7f6ba8ff9700 1 --2- 172.21.15.27:0/1310535623 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6b90067050 0x7f6b90069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:09.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.596+0000 7f6ba8ff9700 1 -- 172.21.15.27:0/1310535623 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6135+0+0 (secure 0 0 0) 0x7f6ba4085540 con 0x7f6bac143190 2024-01-11T08:25:09.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.596+0000 7f6bb0d9b700 1 -- 172.21.15.27:0/1310535623 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6b90067050 msgr2=0x7f6b90069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:09.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.596+0000 7f6bb0d9b700 1 --2- 172.21.15.27:0/1310535623 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6b90067050 0x7f6b90069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:09.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.604+0000 7f6ba8ff9700 1 -- 172.21.15.27:0/1310535623 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6bac0c5490 con 0x7f6bac143190 2024-01-11T08:25:09.642 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:09.643+0000 7f1feb3201c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:25:09.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.797+0000 7f6bb0d9b700 1 -- 172.21.15.27:0/1310535623 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6b90067050 msgr2=0x7f6b90069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:09.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.797+0000 7f6bb0d9b700 1 --2- 172.21.15.27:0/1310535623 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6b90067050 0x7f6b90069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:09.832 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:09.833+0000 7f1feb3201c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:25:09.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.856+0000 7f6bb2fff700 1 -- 172.21.15.27:0/1310535623 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6bac0c5490 con 0x7f6bac143190 2024-01-11T08:25:09.938 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:09.939+0000 7ff915edc1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:25:09.942 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:09.943+0000 7f1feb3201c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:25:09.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.959+0000 7f6ba8ff9700 1 -- 172.21.15.27:0/1310535623 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mgrmap(e 27) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f6ba40838a0 con 0x7f6bac143190 2024-01-11T08:25:09.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.959+0000 7f6ba8ff9700 1 -- 172.21.15.27:0/1310535623 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6b90067050 msgr2=0x7f6b90069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:09.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:09.959+0000 7f6ba8ff9700 1 --2- 172.21.15.27:0/1310535623 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6b90067050 0x7f6b90069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:10.019 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:10.020+0000 7f1f66144700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:10.019 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:10.020+0000 7f1f66144700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:10.019 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:10.020+0000 7f1f66144700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:10.019 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:10.020+0000 7f1f66144700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:10.019 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:10.020+0000 7f1f66144700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:10.020 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:10.020+0000 7f1f6914a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:10.020 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:10.020+0000 7f1f6914a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:10.020 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:10.020+0000 7f1f6914a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:10.020 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:10.020+0000 7f1f6914a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:10.020 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:10.020+0000 7f1f6914a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:10.135 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:10.136+0000 7ff915edc1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:25:10.329 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:10.330+0000 7ff915edc1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:25:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.354+0000 7f6ba8ff9700 1 -- 172.21.15.27:0/1310535623 <== mon.2 v2:172.21.15.77:3301/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6ba4070930 con 0x7f6bac143190 2024-01-11T08:25:10.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.357+0000 7f6bb2fff700 1 -- 172.21.15.27:0/1310535623 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6bac143190 msgr2=0x7f6bac0b1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:10.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.357+0000 7f6bb2fff700 1 --2- 172.21.15.27:0/1310535623 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6bac143190 0x7f6bac0b1180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4009e50 tx=0x7f6ba400b040 comp rx=0 tx=0).stop 2024-01-11T08:25:10.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.357+0000 7f6bb2fff700 1 -- 172.21.15.27:0/1310535623 shutdown_connections 2024-01-11T08:25:10.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.357+0000 7f6bb2fff700 1 --2- 172.21.15.27:0/1310535623 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6b90067050 0x7f6b90069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:10.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.357+0000 7f6bb2fff700 1 --2- 172.21.15.27:0/1310535623 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6bac143190 0x7f6bac0b1180 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:10.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.357+0000 7f6bb2fff700 1 --2- 172.21.15.27:0/1310535623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bac0cb2e0 0x7f6bac0c5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:10.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.357+0000 7f6bb2fff700 1 --2- 172.21.15.27:0/1310535623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bac142dc0 0x7f6bac140f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:10.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.357+0000 7f6bb2fff700 1 -- 172.21.15.27:0/1310535623 >> 172.21.15.27:0/1310535623 conn(0x7f6bac063ca0 msgr2=0x7f6bac15c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:10.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.357+0000 7f6bb2fff700 1 -- 172.21.15.27:0/1310535623 shutdown_connections 2024-01-11T08:25:10.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.357+0000 7f6bb2fff700 1 -- 172.21.15.27:0/1310535623 wait complete. 2024-01-11T08:25:10.374 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-11T08:25:10.449 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:10.450+0000 7ff915edc1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:25:10.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.623+0000 7f46f1fff700 1 Processor -- start 2024-01-11T08:25:10.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.623+0000 7f46f1fff700 1 -- start start 2024-01-11T08:25:10.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.624+0000 7f46f1fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46ec15a7a0 0x7f46ec15c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:10.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.625+0000 7f46f1fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f46ec0685e0 0x7f46ec068a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:10.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.625+0000 7f46f1fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ec15c4b0 con 0x7f46ec159e60 2024-01-11T08:25:10.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.625+0000 7f46f1fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46ec068fc0 con 0x7f46ec0685e0 2024-01-11T08:25:10.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.625+0000 7f46f1fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec15b5d0 con 0x7f46ec15a7a0 2024-01-11T08:25:10.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.625+0000 7f46ebfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f46ec0685e0 0x7f46ec068a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:10.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.625+0000 7f46eaffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f46ec159e60 0x7f46ec15aec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44220/0 (socket says 172.21.15.27:44220) 2024-01-11T08:25:10.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.625+0000 7f46eaffd700 1 -- 172.21.15.27:0/1471283825 learned_addr learned my addr 172.21.15.27:0/1471283825 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:10.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.625+0000 7f46eb7fe700 1 --2- 172.21.15.27:0/1471283825 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46ec15a7a0 0x7f46ec15c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:10.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.625+0000 7f46ebfff700 1 -- 172.21.15.27:0/1471283825 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46ec15a7a0 msgr2=0x7f46ec15c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:10.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.625+0000 7f46ebfff700 1 --2- 172.21.15.27:0/1471283825 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46ec15a7a0 0x7f46ec15c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:10.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.625+0000 7f46ebfff700 1 -- 172.21.15.27:0/1471283825 >> v1:172.21.15.27:6789/0 conn(0x7f46ec159e60 legacy=0x7f46ec15aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:10.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.625+0000 7f46ea7fc700 1 -- 172.21.15.27:0/1471283825 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187206607 0 0) 0x7f46ec15c4b0 con 0x7f46ec159e60 2024-01-11T08:25:10.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.625+0000 7f46ebfff700 1 -- 172.21.15.27:0/1471283825 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ec15b8a0 con 0x7f46ec0685e0 2024-01-11T08:25:10.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.626+0000 7f46ebfff700 1 --2- 172.21.15.27:0/1471283825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46ec0685e0 0x7f46ec068a80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009e60 tx=0x7f46e4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f297dda5d6874e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:10.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.626+0000 7f46ea7fc700 1 -- 172.21.15.27:0/1471283825 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e400e070 con 0x7f46ec0685e0 2024-01-11T08:25:10.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.626+0000 7f46ea7fc700 1 -- 172.21.15.27:0/1471283825 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e400bc30 con 0x7f46ec0685e0 2024-01-11T08:25:10.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.626+0000 7f46ea7fc700 1 -- 172.21.15.27:0/1471283825 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e4012430 con 0x7f46ec0685e0 2024-01-11T08:25:10.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.626+0000 7f46f1fff700 1 -- 172.21.15.27:0/1471283825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46ec0685e0 msgr2=0x7f46ec068a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:10.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.626+0000 7f46f1fff700 1 --2- 172.21.15.27:0/1471283825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46ec0685e0 0x7f46ec068a80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009e60 tx=0x7f46e4009780 comp rx=0 tx=0).stop 2024-01-11T08:25:10.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.627+0000 7f46f1fff700 1 -- 172.21.15.27:0/1471283825 shutdown_connections 2024-01-11T08:25:10.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.627+0000 7f46f1fff700 1 --2- 172.21.15.27:0/1471283825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46ec0685e0 0x7f46ec068a80 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:10.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.627+0000 7f46f1fff700 1 --2- 172.21.15.27:0/1471283825 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46ec15a7a0 0x7f46ec15c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:10.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.627+0000 7f46f1fff700 1 -- 172.21.15.27:0/1471283825 >> 172.21.15.27:0/1471283825 conn(0x7f46ec06ed40 msgr2=0x7f46ec06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:10.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.627+0000 7f46f1fff700 1 -- 172.21.15.27:0/1471283825 shutdown_connections 2024-01-11T08:25:10.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.627+0000 7f46f1fff700 1 -- 172.21.15.27:0/1471283825 wait complete. 2024-01-11T08:25:10.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.628+0000 7f46f1fff700 1 Processor -- start 2024-01-11T08:25:10.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.628+0000 7f46f1fff700 1 -- start start 2024-01-11T08:25:10.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.629+0000 7f46f1fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46ec0685e0 0x7f46ec0b2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:10.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.629+0000 7f46f1fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46ec15a7a0 0x7f46ec0cf470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:10.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.629+0000 7f46f1fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46ec169460 0x7f46ec1698c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:10.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.629+0000 7f46f1fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec0aff70 con 0x7f46ec169460 2024-01-11T08:25:10.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.629+0000 7f46f1fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec1574f0 con 0x7f46ec0685e0 2024-01-11T08:25:10.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.629+0000 7f46f1fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec0cfca0 con 0x7f46ec15a7a0 2024-01-11T08:25:10.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.630+0000 7f46ebfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46ec169460 0x7f46ec1698c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.630+0000 7f46eaffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46ec15a7a0 0x7f46ec0cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.630+0000 7f46eb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46ec0685e0 0x7f46ec0b2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.630+0000 7f46ebfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46ec169460 0x7f46ec1698c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36692/0 (socket says 172.21.15.27:36692) 2024-01-11T08:25:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.630+0000 7f46ebfff700 1 -- 172.21.15.27:0/2305136137 learned_addr learned my addr 172.21.15.27:0/2305136137 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.630+0000 7f46ebfff700 1 -- 172.21.15.27:0/2305136137 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46ec15a7a0 msgr2=0x7f46ec0cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.630+0000 7f46ebfff700 1 --2- 172.21.15.27:0/2305136137 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46ec15a7a0 0x7f46ec0cf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.630+0000 7f46ebfff700 1 -- 172.21.15.27:0/2305136137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46ec0685e0 msgr2=0x7f46ec0b2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.630+0000 7f46ebfff700 1 --2- 172.21.15.27:0/2305136137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46ec0685e0 0x7f46ec0b2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.630+0000 7f46ebfff700 1 -- 172.21.15.27:0/2305136137 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e4009a20 con 0x7f46ec169460 2024-01-11T08:25:10.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.631+0000 7f46ebfff700 1 --2- 172.21.15.27:0/2305136137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46ec169460 0x7f46ec1698c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f46e4013000 tx=0x7f46e4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:10.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.631+0000 7f46d37fe700 1 -- 172.21.15.27:0/2305136137 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e4027470 con 0x7f46ec169460 2024-01-11T08:25:10.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.631+0000 7f46d37fe700 1 -- 172.21.15.27:0/2305136137 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e4027610 con 0x7f46ec169460 2024-01-11T08:25:10.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.631+0000 7f46f1fff700 1 -- 172.21.15.27:0/2305136137 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46ec0cff80 con 0x7f46ec169460 2024-01-11T08:25:10.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.631+0000 7f46d37fe700 1 -- 172.21.15.27:0/2305136137 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e40279d0 con 0x7f46ec169460 2024-01-11T08:25:10.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.631+0000 7f46f1fff700 1 -- 172.21.15.27:0/2305136137 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46ec0b01c0 con 0x7f46ec169460 2024-01-11T08:25:10.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.632+0000 7f46f1fff700 1 -- 172.21.15.27:0/2305136137 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46ec04c3c0 con 0x7f46ec169460 2024-01-11T08:25:10.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.633+0000 7f46d37fe700 1 -- 172.21.15.27:0/2305136137 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f46e4024070 con 0x7f46ec169460 2024-01-11T08:25:10.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.634+0000 7f46d37fe700 1 -- 172.21.15.27:0/2305136137 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f46e4086a30 con 0x7f46ec169460 2024-01-11T08:25:10.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.640+0000 7f46d37fe700 1 -- 172.21.15.27:0/2305136137 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f46e4053d70 con 0x7f46ec169460 2024-01-11T08:25:10.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.845+0000 7f46f1fff700 1 -- 172.21.15.27:0/2305136137 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f46ec15bb80 con 0x7f46ec169460 2024-01-11T08:25:10.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.955+0000 7f46d37fe700 1 -- 172.21.15.27:0/2305136137 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 28) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f46e4003040 con 0x7f46ec169460 2024-01-11T08:25:10.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.955+0000 7f46d37fe700 1 --2- 172.21.15.27:0/2305136137 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f46d806c790 0x7f46d806eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:10.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.956+0000 7f46eb7fe700 1 --2- 172.21.15.27:0/2305136137 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f46d806c790 0x7f46d806eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:10.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:10.957+0000 7f46eb7fe700 1 --2- 172.21.15.27:0/2305136137 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f46d806c790 0x7f46d806eb70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f46dc006fd0 tx=0x7f46dc008040 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:11.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:11.782+0000 7f46d37fe700 1 -- 172.21.15.27:0/2305136137 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v66) v1 ==== 181+0+0 (secure 0 0 0) 0x7f46e4029c60 con 0x7f46ec169460 2024-01-11T08:25:11.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:11.787+0000 7f46f1fff700 1 -- 172.21.15.27:0/2305136137 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f46d806c790 msgr2=0x7f46d806eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:11.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:11.787+0000 7f46f1fff700 1 --2- 172.21.15.27:0/2305136137 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f46d806c790 0x7f46d806eb70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f46dc006fd0 tx=0x7f46dc008040 comp rx=0 tx=0).stop 2024-01-11T08:25:11.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:11.787+0000 7f46f1fff700 1 -- 172.21.15.27:0/2305136137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46ec169460 msgr2=0x7f46ec1698c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:11.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:11.787+0000 7f46f1fff700 1 --2- 172.21.15.27:0/2305136137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46ec169460 0x7f46ec1698c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f46e4013000 tx=0x7f46e4019000 comp rx=0 tx=0).stop 2024-01-11T08:25:11.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:11.788+0000 7f46f1fff700 1 -- 172.21.15.27:0/2305136137 shutdown_connections 2024-01-11T08:25:11.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:11.788+0000 7f46f1fff700 1 --2- 172.21.15.27:0/2305136137 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f46d806c790 0x7f46d806eb70 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:11.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:11.788+0000 7f46f1fff700 1 --2- 172.21.15.27:0/2305136137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46ec169460 0x7f46ec1698c0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:11.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:11.788+0000 7f46f1fff700 1 --2- 172.21.15.27:0/2305136137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46ec0685e0 0x7f46ec0b2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:11.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:11.788+0000 7f46f1fff700 1 --2- 172.21.15.27:0/2305136137 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46ec15a7a0 0x7f46ec0cf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:11.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:11.788+0000 7f46f1fff700 1 -- 172.21.15.27:0/2305136137 >> 172.21.15.27:0/2305136137 conn(0x7f46ec06ed40 msgr2=0x7f46ec14df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:11.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:11.789+0000 7f46f1fff700 1 -- 172.21.15.27:0/2305136137 shutdown_connections 2024-01-11T08:25:11.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:11.789+0000 7f46f1fff700 1 -- 172.21.15.27:0/2305136137 wait complete. 2024-01-11T08:25:11.790 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-01-11T08:25:11.810 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-11T08:25:12.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.060+0000 7f7e27fff700 1 Processor -- start 2024-01-11T08:25:12.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.060+0000 7f7e27fff700 1 -- start start 2024-01-11T08:25:12.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.061+0000 7f7e27fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e20159250 0x7f7e200b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:12.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.061+0000 7f7e27fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7e200cb410 0x7f7e200cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:12.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.061+0000 7f7e27fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2015a440 con 0x7f7e200684d0 2024-01-11T08:25:12.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.061+0000 7f7e27fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e2015ace0 con 0x7f7e200cb410 2024-01-11T08:25:12.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.061+0000 7f7e27fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e200b0de0 con 0x7f7e20159250 2024-01-11T08:25:12.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.061+0000 7f7e2659c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7e200684d0 0x7f7e200ce2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44230/0 (socket says 172.21.15.27:44230) 2024-01-11T08:25:12.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.061+0000 7f7e2659c700 1 -- 172.21.15.27:0/2006465566 learned_addr learned my addr 172.21.15.27:0/2006465566 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:12.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.062+0000 7f7e25d9b700 1 --2- 172.21.15.27:0/2006465566 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e20159250 0x7f7e200b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:12.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.062+0000 7f7e2559a700 1 --2- 172.21.15.27:0/2006465566 >> v2:172.21.15.77:3300/0 conn(0x7f7e200cb410 0x7f7e200cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:12.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.062+0000 7f7e24d99700 1 -- 172.21.15.27:0/2006465566 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092136824 0 0) 0x7f7e2015a440 con 0x7f7e200684d0 2024-01-11T08:25:12.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.062+0000 7f7e24d99700 1 -- 172.21.15.27:0/2006465566 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e0c003570 con 0x7f7e200684d0 2024-01-11T08:25:12.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.062+0000 7f7e25d9b700 1 -- 172.21.15.27:0/2006465566 >> v2:172.21.15.77:3300/0 conn(0x7f7e200cb410 msgr2=0x7f7e200cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:12.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.062+0000 7f7e25d9b700 1 --2- 172.21.15.27:0/2006465566 >> v2:172.21.15.77:3300/0 conn(0x7f7e200cb410 0x7f7e200cb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:12.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.062+0000 7f7e25d9b700 1 -- 172.21.15.27:0/2006465566 >> v1:172.21.15.27:6789/0 conn(0x7f7e200684d0 legacy=0x7f7e200ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:12.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.062+0000 7f7e25d9b700 1 -- 172.21.15.27:0/2006465566 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2015a720 con 0x7f7e20159250 2024-01-11T08:25:12.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.062+0000 7f7e2559a700 1 --2- 172.21.15.27:0/2006465566 >> v2:172.21.15.77:3300/0 conn(0x7f7e200cb410 0x7f7e200cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:12.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.062+0000 7f7e25d9b700 1 --2- 172.21.15.27:0/2006465566 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e20159250 0x7f7e200b08a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c0098e0 tx=0x7f7e1c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=872d2b63304489d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:12.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.063+0000 7f7e24d99700 1 -- 172.21.15.27:0/2006465566 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c011630 con 0x7f7e20159250 2024-01-11T08:25:12.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.063+0000 7f7e24d99700 1 -- 172.21.15.27:0/2006465566 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1c011790 con 0x7f7e20159250 2024-01-11T08:25:12.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.063+0000 7f7e24d99700 1 -- 172.21.15.27:0/2006465566 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c011b30 con 0x7f7e20159250 2024-01-11T08:25:12.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.063+0000 7f7e27fff700 1 -- 172.21.15.27:0/2006465566 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e20159250 msgr2=0x7f7e200b08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:12.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.063+0000 7f7e27fff700 1 --2- 172.21.15.27:0/2006465566 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e20159250 0x7f7e200b08a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c0098e0 tx=0x7f7e1c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:12.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.063+0000 7f7e27fff700 1 -- 172.21.15.27:0/2006465566 shutdown_connections 2024-01-11T08:25:12.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.063+0000 7f7e27fff700 1 --2- 172.21.15.27:0/2006465566 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e20159250 0x7f7e200b08a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:12.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.063+0000 7f7e27fff700 1 --2- 172.21.15.27:0/2006465566 >> v2:172.21.15.77:3300/0 conn(0x7f7e200cb410 0x7f7e200cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:12.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.063+0000 7f7e27fff700 1 -- 172.21.15.27:0/2006465566 >> 172.21.15.27:0/2006465566 conn(0x7f7e20067610 msgr2=0x7f7e20067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:12.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.063+0000 7f7e27fff700 1 -- 172.21.15.27:0/2006465566 shutdown_connections 2024-01-11T08:25:12.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.063+0000 7f7e27fff700 1 -- 172.21.15.27:0/2006465566 wait complete. 2024-01-11T08:25:12.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.066+0000 7f7e27fff700 1 Processor -- start 2024-01-11T08:25:12.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.066+0000 7f7e27fff700 1 -- start start 2024-01-11T08:25:12.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.067+0000 7f7e27fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e200684d0 0x7f7e200b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:12.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.067+0000 7f7e27fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e200cb410 0x7f7e200aead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:12.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.067+0000 7f7e27fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e20159250 0x7f7e200af010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:12.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.067+0000 7f7e27fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e20169520 con 0x7f7e200684d0 2024-01-11T08:25:12.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.067+0000 7f7e27fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e200b3470 con 0x7f7e20159250 2024-01-11T08:25:12.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.067+0000 7f7e27fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e200b3750 con 0x7f7e200cb410 2024-01-11T08:25:12.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.067+0000 7f7e25d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e200684d0 0x7f7e200b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:12.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.067+0000 7f7e25d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e200684d0 0x7f7e200b2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36700/0 (socket says 172.21.15.27:36700) 2024-01-11T08:25:12.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.067+0000 7f7e25d9b700 1 -- 172.21.15.27:0/2066228842 learned_addr learned my addr 172.21.15.27:0/2066228842 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:12.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.067+0000 7f7e2559a700 1 --2- 172.21.15.27:0/2066228842 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e200cb410 0x7f7e200aead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:12.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.067+0000 7f7e2659c700 1 --2- 172.21.15.27:0/2066228842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e20159250 0x7f7e200af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:12.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.068+0000 7f7e25d9b700 1 -- 172.21.15.27:0/2066228842 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e200cb410 msgr2=0x7f7e200aead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:12.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.068+0000 7f7e25d9b700 1 --2- 172.21.15.27:0/2066228842 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e200cb410 0x7f7e200aead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:12.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.068+0000 7f7e25d9b700 1 -- 172.21.15.27:0/2066228842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e20159250 msgr2=0x7f7e200af010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:12.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.068+0000 7f7e25d9b700 1 --2- 172.21.15.27:0/2066228842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e20159250 0x7f7e200af010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:12.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.068+0000 7f7e25d9b700 1 -- 172.21.15.27:0/2066228842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e10009420 con 0x7f7e200684d0 2024-01-11T08:25:12.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.068+0000 7f7e25d9b700 1 --2- 172.21.15.27:0/2066228842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e200684d0 0x7f7e200b2b70 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7e200cf420 tx=0x7f7e1c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:12.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.068+0000 7f7e15ffb700 1 -- 172.21.15.27:0/2066228842 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c000d10 con 0x7f7e200684d0 2024-01-11T08:25:12.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.068+0000 7f7e15ffb700 1 -- 172.21.15.27:0/2066228842 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1c011b10 con 0x7f7e200684d0 2024-01-11T08:25:12.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.068+0000 7f7e15ffb700 1 -- 172.21.15.27:0/2066228842 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c025660 con 0x7f7e200684d0 2024-01-11T08:25:12.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.068+0000 7f7e27fff700 1 -- 172.21.15.27:0/2066228842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1c009420 con 0x7f7e200684d0 2024-01-11T08:25:12.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.069+0000 7f7e27fff700 1 -- 172.21.15.27:0/2066228842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e200b3d10 con 0x7f7e200684d0 2024-01-11T08:25:12.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.070+0000 7f7e27fff700 1 -- 172.21.15.27:0/2066228842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2004c3c0 con 0x7f7e200684d0 2024-01-11T08:25:12.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.070+0000 7f7e15ffb700 1 -- 172.21.15.27:0/2066228842 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f7e1c025a50 con 0x7f7e200684d0 2024-01-11T08:25:12.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.071+0000 7f7e15ffb700 1 --2- 172.21.15.27:0/2066228842 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7e04066d80 0x7f7e04069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:12.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.071+0000 7f7e15ffb700 1 -- 172.21.15.27:0/2066228842 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5955+0+0 (secure 0 0 0) 0x7f7e1c085b70 con 0x7f7e200684d0 2024-01-11T08:25:12.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.072+0000 7f7e2559a700 1 --2- 172.21.15.27:0/2066228842 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7e04066d80 0x7f7e04069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:12.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.072+0000 7f7e2559a700 1 --2- 172.21.15.27:0/2066228842 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7e04066d80 0x7f7e04069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7e201585b0 tx=0x7f7e100091a0 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:12.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.076+0000 7f7e15ffb700 1 -- 172.21.15.27:0/2066228842 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7e1c052ec0 con 0x7f7e200684d0 2024-01-11T08:25:12.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.277+0000 7f7e27fff700 1 -- 172.21.15.27:0/2066228842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7e200b2950 con 0x7f7e200684d0 2024-01-11T08:25:12.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.788+0000 7f7e15ffb700 1 -- 172.21.15.27:0/2066228842 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 29) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f7e1c053290 con 0x7f7e200684d0 2024-01-11T08:25:12.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.963+0000 7f7e15ffb700 1 -- 172.21.15.27:0/2066228842 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v67) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7e1c054f20 con 0x7f7e200684d0 2024-01-11T08:25:12.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.967+0000 7f7e27fff700 1 -- 172.21.15.27:0/2066228842 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7e04066d80 msgr2=0x7f7e04069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:12.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.967+0000 7f7e27fff700 1 --2- 172.21.15.27:0/2066228842 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7e04066d80 0x7f7e04069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7e201585b0 tx=0x7f7e100091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:12.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.967+0000 7f7e27fff700 1 -- 172.21.15.27:0/2066228842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e200684d0 msgr2=0x7f7e200b2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.967+0000 7f7e27fff700 1 --2- 172.21.15.27:0/2066228842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e200684d0 0x7f7e200b2b70 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7e200cf420 tx=0x7f7e1c017000 comp rx=0 tx=0).stop 2024-01-11T08:25:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.967+0000 7f7e27fff700 1 -- 172.21.15.27:0/2066228842 shutdown_connections 2024-01-11T08:25:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.967+0000 7f7e27fff700 1 --2- 172.21.15.27:0/2066228842 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7e04066d80 0x7f7e04069230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.967+0000 7f7e27fff700 1 --2- 172.21.15.27:0/2066228842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e20159250 0x7f7e200af010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.967+0000 7f7e27fff700 1 --2- 172.21.15.27:0/2066228842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e200684d0 0x7f7e200b2b70 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.967+0000 7f7e27fff700 1 --2- 172.21.15.27:0/2066228842 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e200cb410 0x7f7e200aead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.967+0000 7f7e27fff700 1 -- 172.21.15.27:0/2066228842 >> 172.21.15.27:0/2066228842 conn(0x7f7e20067610 msgr2=0x7f7e200cce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.967+0000 7f7e27fff700 1 -- 172.21.15.27:0/2066228842 shutdown_connections 2024-01-11T08:25:12.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:12.967+0000 7f7e27fff700 1 -- 172.21.15.27:0/2066228842 wait complete. 2024-01-11T08:25:12.968 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-01-11T08:25:12.987 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-11T08:25:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.193+0000 7f0249fff700 1 Processor -- start 2024-01-11T08:25:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.193+0000 7f0249fff700 1 -- start start 2024-01-11T08:25:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.194+0000 7f0249fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0244157410 0x7f02440ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.194+0000 7f0249fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0244069540 0x7f02440c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.194+0000 7f0249fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02440699d0 con 0x7f02441577e0 2024-01-11T08:25:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.194+0000 7f0249fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0244069cb0 con 0x7f0244069540 2024-01-11T08:25:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.194+0000 7f0249fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0244069f90 con 0x7f0244157410 2024-01-11T08:25:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.194+0000 7f0243fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0244069540 0x7f02440c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.195+0000 7f0243fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0244069540 0x7f02440c6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36714/0 (socket says 172.21.15.27:36714) 2024-01-11T08:25:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.195+0000 7f0243fff700 1 -- 172.21.15.27:0/1213340107 learned_addr learned my addr 172.21.15.27:0/1213340107 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.195+0000 7f02437fe700 1 --2- 172.21.15.27:0/1213340107 >> v2:172.21.15.77:3300/0 conn(0x7f0244157410 0x7f02440ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.195+0000 7f0243fff700 1 -- 172.21.15.27:0/1213340107 >> v2:172.21.15.77:3300/0 conn(0x7f0244157410 msgr2=0x7f02440ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.195+0000 7f0243fff700 1 --2- 172.21.15.27:0/1213340107 >> v2:172.21.15.77:3300/0 conn(0x7f0244157410 0x7f02440ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.195+0000 7f0243fff700 1 -- 172.21.15.27:0/1213340107 >> v1:172.21.15.27:6789/0 conn(0x7f02441577e0 legacy=0x7f02440ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.195+0000 7f0243fff700 1 -- 172.21.15.27:0/1213340107 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024406a270 con 0x7f0244069540 2024-01-11T08:25:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.195+0000 7f0243fff700 1 --2- 172.21.15.27:0/1213340107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0244069540 0x7f02440c6ba0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0244070c90 tx=0x7f023c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbe122de43ef61d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.196+0000 7f02427fc700 1 -- 172.21.15.27:0/1213340107 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023c00e070 con 0x7f0244069540 2024-01-11T08:25:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.196+0000 7f02427fc700 1 -- 172.21.15.27:0/1213340107 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023c00ba20 con 0x7f0244069540 2024-01-11T08:25:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.196+0000 7f02427fc700 1 -- 172.21.15.27:0/1213340107 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023c0092e0 con 0x7f0244069540 2024-01-11T08:25:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.196+0000 7f0249fff700 1 -- 172.21.15.27:0/1213340107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0244069540 msgr2=0x7f02440c6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.196+0000 7f0249fff700 1 --2- 172.21.15.27:0/1213340107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0244069540 0x7f02440c6ba0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0244070c90 tx=0x7f023c009840 comp rx=0 tx=0).stop 2024-01-11T08:25:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.196+0000 7f0249fff700 1 -- 172.21.15.27:0/1213340107 shutdown_connections 2024-01-11T08:25:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.196+0000 7f0249fff700 1 --2- 172.21.15.27:0/1213340107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0244069540 0x7f02440c6ba0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.196+0000 7f0249fff700 1 --2- 172.21.15.27:0/1213340107 >> v2:172.21.15.77:3300/0 conn(0x7f0244157410 0x7f02440ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.196+0000 7f0249fff700 1 -- 172.21.15.27:0/1213340107 >> 172.21.15.27:0/1213340107 conn(0x7f02440680a0 msgr2=0x7f02440cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.196+0000 7f0249fff700 1 -- 172.21.15.27:0/1213340107 shutdown_connections 2024-01-11T08:25:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.196+0000 7f0249fff700 1 -- 172.21.15.27:0/1213340107 wait complete. 2024-01-11T08:25:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.198+0000 7f0249fff700 1 Processor -- start 2024-01-11T08:25:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.198+0000 7f0249fff700 1 -- start start 2024-01-11T08:25:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.199+0000 7f0249fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0244069540 0x7f02440aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.199+0000 7f0249fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0244157410 0x7f024415f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.199+0000 7f0249fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02441577e0 0x7f024414be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.199+0000 7f0249fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02440b01e0 con 0x7f0244069540 2024-01-11T08:25:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.199+0000 7f0249fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024415fa70 con 0x7f02441577e0 2024-01-11T08:25:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.199+0000 7f0249fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02440af5d0 con 0x7f0244157410 2024-01-11T08:25:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.199+0000 7f02437fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0244069540 0x7f02440aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.199+0000 7f02437fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0244069540 0x7f02440aef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36724/0 (socket says 172.21.15.27:36724) 2024-01-11T08:25:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.199+0000 7f02437fe700 1 -- 172.21.15.27:0/1838819914 learned_addr learned my addr 172.21.15.27:0/1838819914 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.199+0000 7f0243fff700 1 --2- 172.21.15.27:0/1838819914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02441577e0 0x7f024414be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.199+0000 7f0242ffd700 1 --2- 172.21.15.27:0/1838819914 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0244157410 0x7f024415f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.200+0000 7f02437fe700 1 -- 172.21.15.27:0/1838819914 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0244157410 msgr2=0x7f024415f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.200+0000 7f02437fe700 1 --2- 172.21.15.27:0/1838819914 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0244157410 0x7f024415f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.200+0000 7f02437fe700 1 -- 172.21.15.27:0/1838819914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02441577e0 msgr2=0x7f024414be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.200+0000 7f02437fe700 1 --2- 172.21.15.27:0/1838819914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02441577e0 0x7f024414be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.200+0000 7f02437fe700 1 -- 172.21.15.27:0/1838819914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023c009ae0 con 0x7f0244069540 2024-01-11T08:25:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.200+0000 7f0243fff700 1 --2- 172.21.15.27:0/1838819914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02441577e0 0x7f024414be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.200+0000 7f02437fe700 1 --2- 172.21.15.27:0/1838819914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0244069540 0x7f02440aef60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f023400ec80 tx=0x7f0234009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.200+0000 7f022b7fe700 1 -- 172.21.15.27:0/1838819914 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023400c9d0 con 0x7f0244069540 2024-01-11T08:25:13.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.200+0000 7f022b7fe700 1 -- 172.21.15.27:0/1838819914 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023400cb30 con 0x7f0244069540 2024-01-11T08:25:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.200+0000 7f0249fff700 1 -- 172.21.15.27:0/1838819914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02440af8b0 con 0x7f0244069540 2024-01-11T08:25:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.200+0000 7f022b7fe700 1 -- 172.21.15.27:0/1838819914 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0234013470 con 0x7f0244069540 2024-01-11T08:25:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.200+0000 7f0249fff700 1 -- 172.21.15.27:0/1838819914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02440afe70 con 0x7f0244069540 2024-01-11T08:25:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.201+0000 7f0249fff700 1 -- 172.21.15.27:0/1838819914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02440b04c0 con 0x7f0244069540 2024-01-11T08:25:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.202+0000 7f022b7fe700 1 -- 172.21.15.27:0/1838819914 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f0234013860 con 0x7f0244069540 2024-01-11T08:25:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.203+0000 7f022b7fe700 1 --2- 172.21.15.27:0/1838819914 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0230066ea0 0x7f0230069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.203+0000 7f022b7fe700 1 -- 172.21.15.27:0/1838819914 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5483+0+0 (secure 0 0 0) 0x7f02340849e0 con 0x7f0244069540 2024-01-11T08:25:13.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.204+0000 7f0242ffd700 1 --2- 172.21.15.27:0/1838819914 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0230066ea0 0x7f0230069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:13.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.204+0000 7f0242ffd700 1 --2- 172.21.15.27:0/1838819914 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0230066ea0 0x7f0230069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f02440c6b70 tx=0x7f0238009040 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:13.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.208+0000 7f022b7fe700 1 -- 172.21.15.27:0/1838819914 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0234051e10 con 0x7f0244069540 2024-01-11T08:25:13.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.408+0000 7f0249fff700 1 -- 172.21.15.27:0/1838819914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f024404c3c0 con 0x7f0244069540 2024-01-11T08:25:13.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.412+0000 7f022b7fe700 1 -- 172.21.15.27:0/1838819914 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v19) v1 ==== 120+0+0 (secure 0 0 0) 0x7f02340170b0 con 0x7f0244069540 2024-01-11T08:25:13.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.414+0000 7f0249fff700 1 -- 172.21.15.27:0/1838819914 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0230066ea0 msgr2=0x7f0230069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:13.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.414+0000 7f0249fff700 1 --2- 172.21.15.27:0/1838819914 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0230066ea0 0x7f0230069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f02440c6b70 tx=0x7f0238009040 comp rx=0 tx=0).stop 2024-01-11T08:25:13.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.414+0000 7f0249fff700 1 -- 172.21.15.27:0/1838819914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0244069540 msgr2=0x7f02440aef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:13.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.414+0000 7f0249fff700 1 --2- 172.21.15.27:0/1838819914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0244069540 0x7f02440aef60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f023400ec80 tx=0x7f0234009120 comp rx=0 tx=0).stop 2024-01-11T08:25:13.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.415+0000 7f0249fff700 1 -- 172.21.15.27:0/1838819914 shutdown_connections 2024-01-11T08:25:13.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.415+0000 7f0249fff700 1 --2- 172.21.15.27:0/1838819914 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0230066ea0 0x7f0230069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:13.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.415+0000 7f0249fff700 1 --2- 172.21.15.27:0/1838819914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02441577e0 0x7f024414be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:13.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.415+0000 7f0249fff700 1 --2- 172.21.15.27:0/1838819914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0244069540 0x7f02440aef60 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:13.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.415+0000 7f0249fff700 1 --2- 172.21.15.27:0/1838819914 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0244157410 0x7f024415f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:13.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.415+0000 7f0249fff700 1 -- 172.21.15.27:0/1838819914 >> 172.21.15.27:0/1838819914 conn(0x7f02440680a0 msgr2=0x7f02440ca1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:13.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.415+0000 7f0249fff700 1 -- 172.21.15.27:0/1838819914 shutdown_connections 2024-01-11T08:25:13.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.415+0000 7f0249fff700 1 -- 172.21.15.27:0/1838819914 wait complete. 2024-01-11T08:25:13.433 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-11T08:25:13.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.636+0000 7fd8c8dff700 1 Processor -- start 2024-01-11T08:25:13.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.636+0000 7fd8c8dff700 1 -- start start 2024-01-11T08:25:13.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.637+0000 7fd8c8dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd8c40cd5f0 0x7fd8c41591e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:13.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.637+0000 7fd8c8dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c415aef0 con 0x7fd8c4159720 2024-01-11T08:25:13.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.638+0000 7fd8c8dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c40cee00 con 0x7fd8c40cd220 2024-01-11T08:25:13.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.638+0000 7fd8c8dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8c415dd70 con 0x7fd8c40cd5f0 2024-01-11T08:25:13.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.638+0000 7fd8c1d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd8c40cd5f0 0x7fd8c41591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:13.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.638+0000 7fd8c1d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd8c40cd5f0 0x7fd8c41591e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36738/0 (socket says 172.21.15.27:36738) 2024-01-11T08:25:13.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.638+0000 7fd8c1d9b700 1 -- 172.21.15.27:0/3145100704 learned_addr learned my addr 172.21.15.27:0/3145100704 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:13.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.638+0000 7fd8c1d9b700 1 -- 172.21.15.27:0/3145100704 >> v1:172.21.15.77:6789/0 conn(0x7fd8c40cd220 legacy=0x7fd8c415d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:13.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.638+0000 7fd8c1d9b700 1 -- 172.21.15.27:0/3145100704 >> v1:172.21.15.27:6789/0 conn(0x7fd8c4159720 legacy=0x7fd8c40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:13.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.638+0000 7fd8c1d9b700 1 -- 172.21.15.27:0/3145100704 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c415e050 con 0x7fd8c40cd5f0 2024-01-11T08:25:13.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.638+0000 7fd8c1d9b700 1 --2- 172.21.15.27:0/3145100704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8c40cd5f0 0x7fd8c41591e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd8b80099b0 tx=0x7fd8b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40523f38ae5baea server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:13.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.639+0000 7fd8c159a700 1 -- 172.21.15.27:0/3145100704 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b800b860 con 0x7fd8c40cd5f0 2024-01-11T08:25:13.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.639+0000 7fd8c159a700 1 -- 172.21.15.27:0/3145100704 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b800b9c0 con 0x7fd8c40cd5f0 2024-01-11T08:25:13.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.639+0000 7fd8c159a700 1 -- 172.21.15.27:0/3145100704 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b800bd60 con 0x7fd8c40cd5f0 2024-01-11T08:25:13.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.639+0000 7fd8c8dff700 1 -- 172.21.15.27:0/3145100704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8c40cd5f0 msgr2=0x7fd8c41591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:13.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.639+0000 7fd8c8dff700 1 --2- 172.21.15.27:0/3145100704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8c40cd5f0 0x7fd8c41591e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd8b80099b0 tx=0x7fd8b8009250 comp rx=0 tx=0).stop 2024-01-11T08:25:13.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.639+0000 7fd8c8dff700 1 -- 172.21.15.27:0/3145100704 shutdown_connections 2024-01-11T08:25:13.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.639+0000 7fd8c8dff700 1 --2- 172.21.15.27:0/3145100704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8c40cd5f0 0x7fd8c41591e0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:13.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.639+0000 7fd8c8dff700 1 -- 172.21.15.27:0/3145100704 >> 172.21.15.27:0/3145100704 conn(0x7fd8c406ed40 msgr2=0x7fd8c406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:13.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.639+0000 7fd8c8dff700 1 -- 172.21.15.27:0/3145100704 shutdown_connections 2024-01-11T08:25:13.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.639+0000 7fd8c8dff700 1 -- 172.21.15.27:0/3145100704 wait complete. 2024-01-11T08:25:13.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.641+0000 7fd8c8dff700 1 Processor -- start 2024-01-11T08:25:13.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.641+0000 7fd8c8dff700 1 -- start start 2024-01-11T08:25:13.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.642+0000 7fd8c8dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8c40cd220 0x7fd8c416a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:13.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.642+0000 7fd8c8dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd8c40cd5f0 0x7fd8c40b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:13.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.642+0000 7fd8c8dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8c4159720 0x7fd8c415c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:13.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.642+0000 7fd8c8dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c416a300 con 0x7fd8c40cd220 2024-01-11T08:25:13.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.642+0000 7fd8c8dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c4152810 con 0x7fd8c4159720 2024-01-11T08:25:13.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.642+0000 7fd8c8dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c40c88a0 con 0x7fd8c40cd5f0 2024-01-11T08:25:13.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.642+0000 7fd8c259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8c40cd220 0x7fd8c416a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:13.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.642+0000 7fd8c259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8c40cd220 0x7fd8c416a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36742/0 (socket says 172.21.15.27:36742) 2024-01-11T08:25:13.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.642+0000 7fd8c259c700 1 -- 172.21.15.27:0/1057654112 learned_addr learned my addr 172.21.15.27:0/1057654112 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:13.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.642+0000 7fd8c2d9d700 1 --2- 172.21.15.27:0/1057654112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8c4159720 0x7fd8c415c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:13.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.642+0000 7fd8c1d9b700 1 --2- 172.21.15.27:0/1057654112 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd8c40cd5f0 0x7fd8c40b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:13.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.643+0000 7fd8c259c700 1 -- 172.21.15.27:0/1057654112 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd8c40cd5f0 msgr2=0x7fd8c40b4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:13.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.643+0000 7fd8c259c700 1 --2- 172.21.15.27:0/1057654112 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd8c40cd5f0 0x7fd8c40b4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:13.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.643+0000 7fd8c259c700 1 -- 172.21.15.27:0/1057654112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8c4159720 msgr2=0x7fd8c415c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:13.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.643+0000 7fd8c259c700 1 --2- 172.21.15.27:0/1057654112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8c4159720 0x7fd8c415c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:13.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.643+0000 7fd8c259c700 1 -- 172.21.15.27:0/1057654112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b80094f0 con 0x7fd8c40cd220 2024-01-11T08:25:13.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.643+0000 7fd8c259c700 1 --2- 172.21.15.27:0/1057654112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8c40cd220 0x7fd8c416a790 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400c590 tx=0x7fd8b400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:13.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.643+0000 7fd8b27fc700 1 -- 172.21.15.27:0/1057654112 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4011590 con 0x7fd8c40cd220 2024-01-11T08:25:13.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.643+0000 7fd8b27fc700 1 -- 172.21.15.27:0/1057654112 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b4011710 con 0x7fd8c40cd220 2024-01-11T08:25:13.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.643+0000 7fd8c8dff700 1 -- 172.21.15.27:0/1057654112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c40c8b80 con 0x7fd8c40cd220 2024-01-11T08:25:13.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.643+0000 7fd8b27fc700 1 -- 172.21.15.27:0/1057654112 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4011ad0 con 0x7fd8c40cd220 2024-01-11T08:25:13.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.644+0000 7fd8c8dff700 1 -- 172.21.15.27:0/1057654112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c4169f20 con 0x7fd8c40cd220 2024-01-11T08:25:13.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.644+0000 7fd8c8dff700 1 -- 172.21.15.27:0/1057654112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c404c3c0 con 0x7fd8c40cd220 2024-01-11T08:25:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.645+0000 7fd8b27fc700 1 -- 172.21.15.27:0/1057654112 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fd8b401b3b0 con 0x7fd8c40cd220 2024-01-11T08:25:13.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.647+0000 7fd8b27fc700 1 --2- 172.21.15.27:0/1057654112 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd8a4066ea0 0x7fd8a4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:13.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.647+0000 7fd8b27fc700 1 -- 172.21.15.27:0/1057654112 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5483+0+0 (secure 0 0 0) 0x7fd8b40837c0 con 0x7fd8c40cd220 2024-01-11T08:25:13.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.648+0000 7fd8c1d9b700 1 --2- 172.21.15.27:0/1057654112 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd8a4066ea0 0x7fd8a4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:13.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.648+0000 7fd8c1d9b700 1 --2- 172.21.15.27:0/1057654112 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd8a4066ea0 0x7fd8a4069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd8b80099b0 tx=0x7fd8b8017000 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:13.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.651+0000 7fd8b27fc700 1 -- 172.21.15.27:0/1057654112 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd8b4050bf0 con 0x7fd8c40cd220 2024-01-11T08:25:13.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.863+0000 7fd8c8dff700 1 -- 172.21.15.27:0/1057654112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fd8c416a5e0 con 0x7fd8c40cd220 2024-01-11T08:25:13.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.867+0000 7fd8b27fc700 1 -- 172.21.15.27:0/1057654112 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v20) v1 ==== 115+0+0 (secure 0 0 0) 0x7fd8b4017020 con 0x7fd8c40cd220 2024-01-11T08:25:13.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.869+0000 7fd8c8dff700 1 -- 172.21.15.27:0/1057654112 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd8a4066ea0 msgr2=0x7fd8a4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:13.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.869+0000 7fd8c8dff700 1 --2- 172.21.15.27:0/1057654112 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd8a4066ea0 0x7fd8a4069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd8b80099b0 tx=0x7fd8b8017000 comp rx=0 tx=0).stop 2024-01-11T08:25:13.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.869+0000 7fd8c8dff700 1 -- 172.21.15.27:0/1057654112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8c40cd220 msgr2=0x7fd8c416a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:13.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.869+0000 7fd8c8dff700 1 --2- 172.21.15.27:0/1057654112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8c40cd220 0x7fd8c416a790 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400c590 tx=0x7fd8b400c8a0 comp rx=0 tx=0).stop 2024-01-11T08:25:13.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.870+0000 7fd8c8dff700 1 -- 172.21.15.27:0/1057654112 shutdown_connections 2024-01-11T08:25:13.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.870+0000 7fd8c8dff700 1 --2- 172.21.15.27:0/1057654112 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd8a4066ea0 0x7fd8a4069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:13.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.870+0000 7fd8c8dff700 1 --2- 172.21.15.27:0/1057654112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8c4159720 0x7fd8c415c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:13.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.870+0000 7fd8c8dff700 1 --2- 172.21.15.27:0/1057654112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8c40cd220 0x7fd8c416a790 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:13.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.870+0000 7fd8c8dff700 1 --2- 172.21.15.27:0/1057654112 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd8c40cd5f0 0x7fd8c40b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:13.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.870+0000 7fd8c8dff700 1 -- 172.21.15.27:0/1057654112 >> 172.21.15.27:0/1057654112 conn(0x7fd8c406ed40 msgr2=0x7fd8c40c5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:13.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.870+0000 7fd8c8dff700 1 -- 172.21.15.27:0/1057654112 shutdown_connections 2024-01-11T08:25:13.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:13.870+0000 7fd8c8dff700 1 -- 172.21.15.27:0/1057654112 wait complete. 2024-01-11T08:25:13.887 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-11T08:25:14.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.090+0000 7fa735dff700 1 Processor -- start 2024-01-11T08:25:14.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.091+0000 7fa735dff700 1 -- start start 2024-01-11T08:25:14.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.092+0000 7fa735dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa73013fbb0 0x7fa7300ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:14.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.092+0000 7fa735dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa73013ff80 0x7fa73015a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:14.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.092+0000 7fa735dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7300c6b30 con 0x7fa7300cace0 2024-01-11T08:25:14.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.092+0000 7fa735dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7300c6800 con 0x7fa73013fbb0 2024-01-11T08:25:14.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.092+0000 7fa735dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7300cb240 con 0x7fa73013ff80 2024-01-11T08:25:14.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.092+0000 7fa72f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa73013fbb0 0x7fa7300ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:14.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.092+0000 7fa72f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa73013fbb0 0x7fa7300ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47104/0 (socket says 172.21.15.27:47104) 2024-01-11T08:25:14.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.092+0000 7fa72f7fe700 1 -- 172.21.15.27:0/1347226480 learned_addr learned my addr 172.21.15.27:0/1347226480 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:14.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.092+0000 7fa72effd700 1 --2- 172.21.15.27:0/1347226480 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa73013ff80 0x7fa73015a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:14.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.093+0000 7fa72f7fe700 1 -- 172.21.15.27:0/1347226480 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa73013ff80 msgr2=0x7fa73015a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:14.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.093+0000 7fa72f7fe700 1 --2- 172.21.15.27:0/1347226480 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa73013ff80 0x7fa73015a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:14.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.093+0000 7fa72f7fe700 1 -- 172.21.15.27:0/1347226480 >> v1:172.21.15.77:6789/0 conn(0x7fa7300cace0 legacy=0x7fa73006e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:14.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.093+0000 7fa72f7fe700 1 -- 172.21.15.27:0/1347226480 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7300cb520 con 0x7fa73013fbb0 2024-01-11T08:25:14.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.093+0000 7fa72f7fe700 1 --2- 172.21.15.27:0/1347226480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa73013fbb0 0x7fa7300ca7a0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa720009990 tx=0x7fa720009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f56511af2747776f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:14.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.093+0000 7fa72e7fc700 1 -- 172.21.15.27:0/1347226480 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa720009e80 con 0x7fa73013fbb0 2024-01-11T08:25:14.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.093+0000 7fa72e7fc700 1 -- 172.21.15.27:0/1347226480 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa720005c10 con 0x7fa73013fbb0 2024-01-11T08:25:14.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.094+0000 7fa72e7fc700 1 -- 172.21.15.27:0/1347226480 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72000b650 con 0x7fa73013fbb0 2024-01-11T08:25:14.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.094+0000 7fa735dff700 1 -- 172.21.15.27:0/1347226480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa73013fbb0 msgr2=0x7fa7300ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:14.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.094+0000 7fa735dff700 1 --2- 172.21.15.27:0/1347226480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa73013fbb0 0x7fa7300ca7a0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa720009990 tx=0x7fa720009220 comp rx=0 tx=0).stop 2024-01-11T08:25:14.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.094+0000 7fa735dff700 1 -- 172.21.15.27:0/1347226480 shutdown_connections 2024-01-11T08:25:14.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.094+0000 7fa735dff700 1 --2- 172.21.15.27:0/1347226480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa73013fbb0 0x7fa7300ca7a0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:14.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.094+0000 7fa735dff700 1 --2- 172.21.15.27:0/1347226480 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa73013ff80 0x7fa73015a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:14.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.094+0000 7fa735dff700 1 -- 172.21.15.27:0/1347226480 >> 172.21.15.27:0/1347226480 conn(0x7fa7300680a0 msgr2=0x7fa7300cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:14.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.094+0000 7fa735dff700 1 -- 172.21.15.27:0/1347226480 shutdown_connections 2024-01-11T08:25:14.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.094+0000 7fa735dff700 1 -- 172.21.15.27:0/1347226480 wait complete. 2024-01-11T08:25:14.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.096+0000 7fa735dff700 1 Processor -- start 2024-01-11T08:25:14.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.096+0000 7fa735dff700 1 -- start start 2024-01-11T08:25:14.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.097+0000 7fa735dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7300cace0 0x7fa730169500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:14.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.097+0000 7fa735dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa73013fbb0 0x7fa730159f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:14.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.097+0000 7fa735dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa73013ff80 0x7fa73015a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:14.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.097+0000 7fa735dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73013a990 con 0x7fa73013ff80 2024-01-11T08:25:14.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.097+0000 7fa735dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa730139d80 con 0x7fa7300cace0 2024-01-11T08:25:14.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.097+0000 7fa735dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa73013a060 con 0x7fa73013fbb0 2024-01-11T08:25:14.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.097+0000 7fa72ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa73013ff80 0x7fa73015a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:14.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.097+0000 7fa72ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa73013ff80 0x7fa73015a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47110/0 (socket says 172.21.15.27:47110) 2024-01-11T08:25:14.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.097+0000 7fa72ffff700 1 -- 172.21.15.27:0/2626940568 learned_addr learned my addr 172.21.15.27:0/2626940568 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:14.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.097+0000 7fa72effd700 1 --2- 172.21.15.27:0/2626940568 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa73013fbb0 0x7fa730159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:14.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.097+0000 7fa72f7fe700 1 --2- 172.21.15.27:0/2626940568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7300cace0 0x7fa730169500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:14.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.098+0000 7fa72ffff700 1 -- 172.21.15.27:0/2626940568 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa73013fbb0 msgr2=0x7fa730159f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:14.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.098+0000 7fa72ffff700 1 --2- 172.21.15.27:0/2626940568 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa73013fbb0 0x7fa730159f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:14.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.098+0000 7fa72ffff700 1 -- 172.21.15.27:0/2626940568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7300cace0 msgr2=0x7fa730169500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:14.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.098+0000 7fa72ffff700 1 --2- 172.21.15.27:0/2626940568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7300cace0 0x7fa730169500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:14.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.098+0000 7fa72ffff700 1 -- 172.21.15.27:0/2626940568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa720009480 con 0x7fa73013ff80 2024-01-11T08:25:14.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.098+0000 7fa72ffff700 1 --2- 172.21.15.27:0/2626940568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa73013ff80 0x7fa73015a460 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa72800c8e0 tx=0x7fa728001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:14.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.098+0000 7fa7177fe700 1 -- 172.21.15.27:0/2626940568 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa728012630 con 0x7fa73013ff80 2024-01-11T08:25:14.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.098+0000 7fa7177fe700 1 -- 172.21.15.27:0/2626940568 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa728012790 con 0x7fa73013ff80 2024-01-11T08:25:14.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.098+0000 7fa7177fe700 1 -- 172.21.15.27:0/2626940568 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa728012b50 con 0x7fa73013ff80 2024-01-11T08:25:14.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.098+0000 7fa735dff700 1 -- 172.21.15.27:0/2626940568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa73013a340 con 0x7fa73013ff80 2024-01-11T08:25:14.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.098+0000 7fa735dff700 1 -- 172.21.15.27:0/2626940568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa730138ff0 con 0x7fa73013ff80 2024-01-11T08:25:14.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.099+0000 7fa735dff700 1 -- 172.21.15.27:0/2626940568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa73013ab40 con 0x7fa73013ff80 2024-01-11T08:25:14.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.100+0000 7fa7177fe700 1 -- 172.21.15.27:0/2626940568 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fa72801a3a0 con 0x7fa73013ff80 2024-01-11T08:25:14.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.101+0000 7fa7177fe700 1 --2- 172.21.15.27:0/2626940568 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa71c066ea0 0x7fa71c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:14.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.101+0000 7fa7177fe700 1 -- 172.21.15.27:0/2626940568 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5483+0+0 (secure 0 0 0) 0x7fa728084510 con 0x7fa73013ff80 2024-01-11T08:25:14.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.102+0000 7fa72f7fe700 1 --2- 172.21.15.27:0/2626940568 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa71c066ea0 0x7fa71c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:14.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.102+0000 7fa72f7fe700 1 --2- 172.21.15.27:0/2626940568 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa71c066ea0 0x7fa71c069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa720009260 tx=0x7fa720016000 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:14.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.105+0000 7fa7177fe700 1 -- 172.21.15.27:0/2626940568 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa73013ab40 con 0x7fa73013ff80 2024-01-11T08:25:14.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.308+0000 7fa735dff700 1 -- 172.21.15.27:0/2626940568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa73004c3c0 con 0x7fa73013ff80 2024-01-11T08:25:14.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.313+0000 7fa7177fe700 1 -- 172.21.15.27:0/2626940568 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa7280518a0 con 0x7fa73013ff80 2024-01-11T08:25:14.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.315+0000 7fa735dff700 1 -- 172.21.15.27:0/2626940568 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa71c066ea0 msgr2=0x7fa71c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:14.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.315+0000 7fa735dff700 1 --2- 172.21.15.27:0/2626940568 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa71c066ea0 0x7fa71c069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa720009260 tx=0x7fa720016000 comp rx=0 tx=0).stop 2024-01-11T08:25:14.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.315+0000 7fa735dff700 1 -- 172.21.15.27:0/2626940568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa73013ff80 msgr2=0x7fa73015a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:14.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.315+0000 7fa735dff700 1 --2- 172.21.15.27:0/2626940568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa73013ff80 0x7fa73015a460 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa72800c8e0 tx=0x7fa728001710 comp rx=0 tx=0).stop 2024-01-11T08:25:14.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.315+0000 7fa735dff700 1 -- 172.21.15.27:0/2626940568 shutdown_connections 2024-01-11T08:25:14.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.315+0000 7fa735dff700 1 --2- 172.21.15.27:0/2626940568 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa71c066ea0 0x7fa71c069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:14.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.315+0000 7fa735dff700 1 --2- 172.21.15.27:0/2626940568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa73013ff80 0x7fa73015a460 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:14.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.315+0000 7fa735dff700 1 --2- 172.21.15.27:0/2626940568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7300cace0 0x7fa730169500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:14.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.315+0000 7fa735dff700 1 --2- 172.21.15.27:0/2626940568 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa73013fbb0 0x7fa730159f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:14.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.315+0000 7fa735dff700 1 -- 172.21.15.27:0/2626940568 >> 172.21.15.27:0/2626940568 conn(0x7fa7300680a0 msgr2=0x7fa730069de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:14.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.315+0000 7fa735dff700 1 -- 172.21.15.27:0/2626940568 shutdown_connections 2024-01-11T08:25:14.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.315+0000 7fa735dff700 1 -- 172.21.15.27:0/2626940568 wait complete. 2024-01-11T08:25:14.333 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2024-01-11T08:25:14.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.534+0000 7f59c35ff700 1 Processor -- start 2024-01-11T08:25:14.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.534+0000 7f59c35ff700 1 -- start start 2024-01-11T08:25:14.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.536+0000 7f59c35ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59bc0b2b90 0x7f59bc159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:14.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.536+0000 7f59c35ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f59bc0b07b0 0x7f59bc159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:14.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.536+0000 7f59c35ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59bc0cd730 con 0x7f59bc0b03e0 2024-01-11T08:25:14.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.536+0000 7f59c35ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59bc0ccb20 con 0x7f59bc0b07b0 2024-01-11T08:25:14.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.536+0000 7f59c35ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59bc0cce00 con 0x7f59bc0b2b90 2024-01-11T08:25:14.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.536+0000 7f59c0b9a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59bc0b2b90 0x7f59bc159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:14.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.536+0000 7f59c1b9c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f59bc0b07b0 0x7f59bc159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:14.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.536+0000 7f59c0b9a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59bc0b2b90 0x7f59bc159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:45724/0 (socket says 172.21.15.27:45724) 2024-01-11T08:25:14.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.536+0000 7f59c1b9c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f59bc0b07b0 0x7f59bc159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:58710/0 (socket says 172.21.15.27:58710) 2024-01-11T08:25:14.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.536+0000 7f59c139b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f59bc0b03e0 0x7f59bc0b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:49182/0 (socket says 172.21.15.27:49182) 2024-01-11T08:25:14.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.536+0000 7f59c0b9a700 1 -- 172.21.15.27:0/678327719 learned_addr learned my addr 172.21.15.27:0/678327719 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:14.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.537+0000 7f59b3fff700 1 -- 172.21.15.27:0/678327719 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618869064 0 0) 0x7f59bc0cd730 con 0x7f59bc0b03e0 2024-01-11T08:25:14.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.537+0000 7f59c0b9a700 1 -- 172.21.15.27:0/678327719 >> v2:172.21.15.77:3300/0 conn(0x7f59bc0b07b0 msgr2=0x7f59bc159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:14.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.537+0000 7f59c0b9a700 1 --2- 172.21.15.27:0/678327719 >> v2:172.21.15.77:3300/0 conn(0x7f59bc0b07b0 0x7f59bc159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:14.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.537+0000 7f59c0b9a700 1 -- 172.21.15.27:0/678327719 >> v1:172.21.15.77:6789/0 conn(0x7f59bc0b03e0 legacy=0x7f59bc0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:14.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.537+0000 7f59c0b9a700 1 -- 172.21.15.27:0/678327719 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59bc0cd0e0 con 0x7f59bc0b2b90 2024-01-11T08:25:14.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.537+0000 7f59c1b9c700 1 --2- 172.21.15.27:0/678327719 >> v2:172.21.15.77:3300/0 conn(0x7f59bc0b07b0 0x7f59bc159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:14.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.537+0000 7f59c0b9a700 1 --2- 172.21.15.27:0/678327719 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59bc0b2b90 0x7f59bc159200 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0099b0 tx=0x7f59ac009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34da0e02dc344f61 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.537+0000 7f59b3fff700 1 -- 172.21.15.27:0/678327719 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ac009ea0 con 0x7f59bc0b2b90 2024-01-11T08:25:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.537+0000 7f59b3fff700 1 -- 172.21.15.27:0/678327719 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59ac00b860 con 0x7f59bc0b2b90 2024-01-11T08:25:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.537+0000 7f59b3fff700 1 -- 172.21.15.27:0/678327719 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ac00bc00 con 0x7f59bc0b2b90 2024-01-11T08:25:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.538+0000 7f59c35ff700 1 -- 172.21.15.27:0/678327719 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59bc0b2b90 msgr2=0x7f59bc159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.538+0000 7f59c35ff700 1 --2- 172.21.15.27:0/678327719 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59bc0b2b90 0x7f59bc159200 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0099b0 tx=0x7f59ac009250 comp rx=0 tx=0).stop 2024-01-11T08:25:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.538+0000 7f59c35ff700 1 -- 172.21.15.27:0/678327719 shutdown_connections 2024-01-11T08:25:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.538+0000 7f59c35ff700 1 --2- 172.21.15.27:0/678327719 >> v2:172.21.15.77:3300/0 conn(0x7f59bc0b07b0 0x7f59bc159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.538+0000 7f59c35ff700 1 --2- 172.21.15.27:0/678327719 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59bc0b2b90 0x7f59bc159200 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.538+0000 7f59c35ff700 1 -- 172.21.15.27:0/678327719 >> 172.21.15.27:0/678327719 conn(0x7f59bc06ed40 msgr2=0x7f59bc06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.538+0000 7f59c35ff700 1 -- 172.21.15.27:0/678327719 shutdown_connections 2024-01-11T08:25:14.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.538+0000 7f59c35ff700 1 -- 172.21.15.27:0/678327719 wait complete. 2024-01-11T08:25:14.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.540+0000 7f59c35ff700 1 Processor -- start 2024-01-11T08:25:14.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.540+0000 7f59c35ff700 1 -- start start 2024-01-11T08:25:14.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.541+0000 7f59c35ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59bc0b03e0 0x7f59bc0c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:14.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.541+0000 7f59c35ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59bc0b07b0 0x7f59bc158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:14.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.541+0000 7f59c35ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59bc0b2b90 0x7f59bc16d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.541+0000 7f59c35ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59bc0a6f40 con 0x7f59bc0b03e0 2024-01-11T08:25:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.541+0000 7f59c35ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59bc0c7f50 con 0x7f59bc0b07b0 2024-01-11T08:25:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.541+0000 7f59c35ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59bc0c7660 con 0x7f59bc0b2b90 2024-01-11T08:25:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.541+0000 7f59c139b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59bc0b03e0 0x7f59bc0c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.541+0000 7f59c139b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59bc0b03e0 0x7f59bc0c62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47114/0 (socket says 172.21.15.27:47114) 2024-01-11T08:25:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.541+0000 7f59c139b700 1 -- 172.21.15.27:0/338456992 learned_addr learned my addr 172.21.15.27:0/338456992 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.541+0000 7f59c0b9a700 1 --2- 172.21.15.27:0/338456992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59bc0b07b0 0x7f59bc158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.541+0000 7f59c1b9c700 1 --2- 172.21.15.27:0/338456992 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59bc0b2b90 0x7f59bc16d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.542+0000 7f59c139b700 1 -- 172.21.15.27:0/338456992 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59bc0b2b90 msgr2=0x7f59bc16d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.542+0000 7f59c139b700 1 --2- 172.21.15.27:0/338456992 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59bc0b2b90 0x7f59bc16d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.542+0000 7f59c139b700 1 -- 172.21.15.27:0/338456992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59bc0b07b0 msgr2=0x7f59bc158bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.542+0000 7f59c139b700 1 --2- 172.21.15.27:0/338456992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59bc0b07b0 0x7f59bc158bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.542+0000 7f59c139b700 1 -- 172.21.15.27:0/338456992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b4009a20 con 0x7f59bc0b03e0 2024-01-11T08:25:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.542+0000 7f59c0b9a700 1 --2- 172.21.15.27:0/338456992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59bc0b07b0 0x7f59bc158bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.542+0000 7f59c139b700 1 --2- 172.21.15.27:0/338456992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59bc0b03e0 0x7f59bc0c62f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f59b80011e0 tx=0x7f59b8007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.542+0000 7f59b17fa700 1 -- 172.21.15.27:0/338456992 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b8007300 con 0x7f59bc0b03e0 2024-01-11T08:25:14.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.542+0000 7f59b17fa700 1 -- 172.21.15.27:0/338456992 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b8007460 con 0x7f59bc0b03e0 2024-01-11T08:25:14.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.542+0000 7f59b17fa700 1 -- 172.21.15.27:0/338456992 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b8004830 con 0x7f59bc0b03e0 2024-01-11T08:25:14.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.542+0000 7f59c35ff700 1 -- 172.21.15.27:0/338456992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59ac0094f0 con 0x7f59bc0b03e0 2024-01-11T08:25:14.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.542+0000 7f59c35ff700 1 -- 172.21.15.27:0/338456992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59bc0c79b0 con 0x7f59bc0b03e0 2024-01-11T08:25:14.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.544+0000 7f59c35ff700 1 -- 172.21.15.27:0/338456992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59bc0718d0 con 0x7f59bc0b03e0 2024-01-11T08:25:14.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.544+0000 7f59b17fa700 1 -- 172.21.15.27:0/338456992 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f59b8004be0 con 0x7f59bc0b03e0 2024-01-11T08:25:14.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.545+0000 7f59b17fa700 1 --2- 172.21.15.27:0/338456992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f59a0066ea0 0x7f59a0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:14.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.545+0000 7f59b17fa700 1 -- 172.21.15.27:0/338456992 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5483+0+0 (secure 0 0 0) 0x7f59b8083870 con 0x7f59bc0b03e0 2024-01-11T08:25:14.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.546+0000 7f59c0b9a700 1 --2- 172.21.15.27:0/338456992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f59a0066ea0 0x7f59a0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:14.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.546+0000 7f59c0b9a700 1 --2- 172.21.15.27:0/338456992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f59a0066ea0 0x7f59a0069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0099b0 tx=0x7f59ac017000 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:14.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.549+0000 7f59b17fa700 1 -- 172.21.15.27:0/338456992 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f59b8050d20 con 0x7f59bc0b03e0 2024-01-11T08:25:14.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.834+0000 7f59c35ff700 1 -- 172.21.15.27:0/338456992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7f59bc0cd3c0 con 0x7f59bc0b03e0 2024-01-11T08:25:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:14.873+0000 7f59b17fa700 1 -- 172.21.15.27:0/338456992 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f59b80510f0 con 0x7f59bc0b03e0 2024-01-11T08:25:15.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:15.793+0000 7f59b17fa700 1 -- 172.21.15.27:0/338456992 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v530) v1 ==== 154+0+0 (secure 0 0 0) 0x7f59b805bbd0 con 0x7f59bc0b03e0 2024-01-11T08:25:15.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:15.797+0000 7f59c35ff700 1 -- 172.21.15.27:0/338456992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f59a0066ea0 msgr2=0x7f59a0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:15.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:15.797+0000 7f59c35ff700 1 --2- 172.21.15.27:0/338456992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f59a0066ea0 0x7f59a0069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0099b0 tx=0x7f59ac017000 comp rx=0 tx=0).stop 2024-01-11T08:25:15.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:15.797+0000 7f59c35ff700 1 -- 172.21.15.27:0/338456992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59bc0b03e0 msgr2=0x7f59bc0c62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:15.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:15.797+0000 7f59c35ff700 1 --2- 172.21.15.27:0/338456992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59bc0b03e0 0x7f59bc0c62f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f59b80011e0 tx=0x7f59b8007e10 comp rx=0 tx=0).stop 2024-01-11T08:25:15.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:15.797+0000 7f59c35ff700 1 -- 172.21.15.27:0/338456992 shutdown_connections 2024-01-11T08:25:15.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:15.797+0000 7f59c35ff700 1 --2- 172.21.15.27:0/338456992 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f59a0066ea0 0x7f59a0069350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:15.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:15.797+0000 7f59c35ff700 1 --2- 172.21.15.27:0/338456992 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59bc0b2b90 0x7f59bc16d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:15.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:15.797+0000 7f59c35ff700 1 --2- 172.21.15.27:0/338456992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59bc0b03e0 0x7f59bc0c62f0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:15.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:15.797+0000 7f59c35ff700 1 --2- 172.21.15.27:0/338456992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59bc0b07b0 0x7f59bc158bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:15.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:15.797+0000 7f59c35ff700 1 -- 172.21.15.27:0/338456992 >> 172.21.15.27:0/338456992 conn(0x7f59bc06ed40 msgr2=0x7f59bc0b3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:15.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:15.797+0000 7f59c35ff700 1 -- 172.21.15.27:0/338456992 shutdown_connections 2024-01-11T08:25:15.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:15.797+0000 7f59c35ff700 1 -- 172.21.15.27:0/338456992 wait complete. 2024-01-11T08:25:15.817 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2024-01-11T08:25:15.817 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2024-01-11T08:25:15.817 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-11T08:25:15.817 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2024-01-11T08:25:16.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.024+0000 7f839b7ff700 1 Processor -- start 2024-01-11T08:25:16.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.024+0000 7f839b7ff700 1 -- start start 2024-01-11T08:25:16.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.025+0000 7f839b7ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f83940cc410 0x7f8394159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:16.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.025+0000 7f839b7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83940cd730 con 0x7f83940b03e0 2024-01-11T08:25:16.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.025+0000 7f839b7ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83940ccb20 con 0x7f83940b2b90 2024-01-11T08:25:16.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.025+0000 7f839b7ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83940cce00 con 0x7f83940cc410 2024-01-11T08:25:16.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.025+0000 7f839959b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f83940b03e0 0x7f83940b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38974/0 (socket says 172.21.15.27:38974) 2024-01-11T08:25:16.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.025+0000 7f839959b700 1 -- 172.21.15.27:0/845335350 learned_addr learned my addr 172.21.15.27:0/845335350 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:16.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.026+0000 7f8399d9c700 1 --2- 172.21.15.27:0/845335350 >> v2:172.21.15.27:3300/0 conn(0x7f83940cc410 0x7f8394159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:16.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.026+0000 7f8399d9c700 1 -- 172.21.15.27:0/845335350 >> v1:172.21.15.77:6789/0 conn(0x7f83940b2b90 legacy=0x7f8394159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:16.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.026+0000 7f8399d9c700 1 -- 172.21.15.27:0/845335350 >> v1:172.21.15.27:6789/0 conn(0x7f83940b03e0 legacy=0x7f83940b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:16.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.026+0000 7f8399d9c700 1 -- 172.21.15.27:0/845335350 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83940cd0e0 con 0x7f83940cc410 2024-01-11T08:25:16.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.026+0000 7f8399d9c700 1 --2- 172.21.15.27:0/845335350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83940cc410 0x7f8394159910 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f838c009e90 tx=0x7f838c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c57437298bd58986 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:16.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.026+0000 7f838bfff700 1 -- 172.21.15.27:0/845335350 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838c00e070 con 0x7f83940cc410 2024-01-11T08:25:16.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.027+0000 7f838bfff700 1 -- 172.21.15.27:0/845335350 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838c00bbf0 con 0x7f83940cc410 2024-01-11T08:25:16.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.027+0000 7f838bfff700 1 -- 172.21.15.27:0/845335350 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838c012430 con 0x7f83940cc410 2024-01-11T08:25:16.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.027+0000 7f839b7ff700 1 -- 172.21.15.27:0/845335350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83940cc410 msgr2=0x7f8394159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:16.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.027+0000 7f839b7ff700 1 --2- 172.21.15.27:0/845335350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83940cc410 0x7f8394159910 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f838c009e90 tx=0x7f838c009780 comp rx=0 tx=0).stop 2024-01-11T08:25:16.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.027+0000 7f839b7ff700 1 -- 172.21.15.27:0/845335350 shutdown_connections 2024-01-11T08:25:16.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.027+0000 7f839b7ff700 1 --2- 172.21.15.27:0/845335350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83940cc410 0x7f8394159910 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:16.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.027+0000 7f839b7ff700 1 -- 172.21.15.27:0/845335350 >> 172.21.15.27:0/845335350 conn(0x7f839406ed40 msgr2=0x7f839406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:16.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.027+0000 7f839b7ff700 1 -- 172.21.15.27:0/845335350 shutdown_connections 2024-01-11T08:25:16.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.027+0000 7f839b7ff700 1 -- 172.21.15.27:0/845335350 wait complete. 2024-01-11T08:25:16.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.029+0000 7f839b7ff700 1 Processor -- start 2024-01-11T08:25:16.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.029+0000 7f839b7ff700 1 -- start start 2024-01-11T08:25:16.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.030+0000 7f839b7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83940b03e0 0x7f83940d0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:16.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.030+0000 7f839b7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f83940b2b90 0x7f83940c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:16.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.030+0000 7f839b7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83940cc410 0x7f8394169990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:16.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.030+0000 7f839b7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83940d0c90 con 0x7f83940b03e0 2024-01-11T08:25:16.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.030+0000 7f839b7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83940d0f70 con 0x7f83940cc410 2024-01-11T08:25:16.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.030+0000 7f839b7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83940ce0e0 con 0x7f83940b2b90 2024-01-11T08:25:16.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.030+0000 7f839959b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83940b03e0 0x7f83940d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:16.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.030+0000 7f839959b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83940b03e0 0x7f83940d0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47138/0 (socket says 172.21.15.27:47138) 2024-01-11T08:25:16.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.030+0000 7f839959b700 1 -- 172.21.15.27:0/2502938039 learned_addr learned my addr 172.21.15.27:0/2502938039 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:16.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.030+0000 7f8399d9c700 1 --2- 172.21.15.27:0/2502938039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83940cc410 0x7f8394169990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:16.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.030+0000 7f8398d9a700 1 --2- 172.21.15.27:0/2502938039 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f83940b2b90 0x7f83940c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:16.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.031+0000 7f839959b700 1 -- 172.21.15.27:0/2502938039 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f83940b2b90 msgr2=0x7f83940c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:16.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.031+0000 7f839959b700 1 --2- 172.21.15.27:0/2502938039 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f83940b2b90 0x7f83940c8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:16.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.031+0000 7f839959b700 1 -- 172.21.15.27:0/2502938039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83940cc410 msgr2=0x7f8394169990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:16.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.031+0000 7f839959b700 1 --2- 172.21.15.27:0/2502938039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83940cc410 0x7f8394169990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:16.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.031+0000 7f839959b700 1 -- 172.21.15.27:0/2502938039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838c009a20 con 0x7f83940b03e0 2024-01-11T08:25:16.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.031+0000 7f8399d9c700 1 --2- 172.21.15.27:0/2502938039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83940cc410 0x7f8394169990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:16.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.031+0000 7f839959b700 1 --2- 172.21.15.27:0/2502938039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83940b03e0 0x7f83940d0690 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f839000c7c0 tx=0x7f8390007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:16.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.031+0000 7f83897fa700 1 -- 172.21.15.27:0/2502938039 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8390005b30 con 0x7f83940b03e0 2024-01-11T08:25:16.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.031+0000 7f83897fa700 1 -- 172.21.15.27:0/2502938039 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8390005cb0 con 0x7f83940b03e0 2024-01-11T08:25:16.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.031+0000 7f83897fa700 1 -- 172.21.15.27:0/2502938039 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8390004a80 con 0x7f83940b03e0 2024-01-11T08:25:16.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.031+0000 7f839b7ff700 1 -- 172.21.15.27:0/2502938039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83940ce3c0 con 0x7f83940b03e0 2024-01-11T08:25:16.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.031+0000 7f839b7ff700 1 -- 172.21.15.27:0/2502938039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8394158e00 con 0x7f83940b03e0 2024-01-11T08:25:16.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.032+0000 7f839b7ff700 1 -- 172.21.15.27:0/2502938039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83940ce8e0 con 0x7f83940b03e0 2024-01-11T08:25:16.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.033+0000 7f83897fa700 1 -- 172.21.15.27:0/2502938039 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f83900070f0 con 0x7f83940b03e0 2024-01-11T08:25:16.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.035+0000 7f83897fa700 1 --2- 172.21.15.27:0/2502938039 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8378066f30 0x7f83780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:16.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.035+0000 7f83897fa700 1 -- 172.21.15.27:0/2502938039 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5483+0+0 (secure 0 0 0) 0x7f83900838d0 con 0x7f83940b03e0 2024-01-11T08:25:16.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.035+0000 7f8398d9a700 1 --2- 172.21.15.27:0/2502938039 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8378066f30 0x7f83780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:16.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.036+0000 7f8398d9a700 1 --2- 172.21.15.27:0/2502938039 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8378066f30 0x7f83780693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f83940cbfa0 tx=0x7f8384009040 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:16.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.039+0000 7f83897fa700 1 -- 172.21.15.27:0/2502938039 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8390050ca0 con 0x7f83940b03e0 2024-01-11T08:25:16.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.324+0000 7f839b7ff700 1 -- 172.21.15.27:0/2502938039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f83940718d0 con 0x7f83940b03e0 2024-01-11T08:25:16.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.799+0000 7f83897fa700 1 -- 172.21.15.27:0/2502938039 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v531) v1 ==== 159+0+0 (secure 0 0 0) 0x7f8390018070 con 0x7f83940b03e0 2024-01-11T08:25:16.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.801+0000 7f839b7ff700 1 -- 172.21.15.27:0/2502938039 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8378066f30 msgr2=0x7f83780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:16.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.801+0000 7f839b7ff700 1 --2- 172.21.15.27:0/2502938039 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8378066f30 0x7f83780693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f83940cbfa0 tx=0x7f8384009040 comp rx=0 tx=0).stop 2024-01-11T08:25:16.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.801+0000 7f839b7ff700 1 -- 172.21.15.27:0/2502938039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83940b03e0 msgr2=0x7f83940d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:16.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.801+0000 7f839b7ff700 1 --2- 172.21.15.27:0/2502938039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83940b03e0 0x7f83940d0690 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f839000c7c0 tx=0x7f8390007f70 comp rx=0 tx=0).stop 2024-01-11T08:25:16.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.801+0000 7f839b7ff700 1 -- 172.21.15.27:0/2502938039 shutdown_connections 2024-01-11T08:25:16.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.801+0000 7f839b7ff700 1 --2- 172.21.15.27:0/2502938039 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8378066f30 0x7f83780693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:16.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.801+0000 7f839b7ff700 1 --2- 172.21.15.27:0/2502938039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83940cc410 0x7f8394169990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:16.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.801+0000 7f839b7ff700 1 --2- 172.21.15.27:0/2502938039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83940b03e0 0x7f83940d0690 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:16.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.801+0000 7f839b7ff700 1 --2- 172.21.15.27:0/2502938039 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f83940b2b90 0x7f83940c8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:16.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.801+0000 7f839b7ff700 1 -- 172.21.15.27:0/2502938039 >> 172.21.15.27:0/2502938039 conn(0x7f839406ed40 msgr2=0x7f83940cb400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:16.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.801+0000 7f839b7ff700 1 -- 172.21.15.27:0/2502938039 shutdown_connections 2024-01-11T08:25:16.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:16.801+0000 7f839b7ff700 1 -- 172.21.15.27:0/2502938039 wait complete. 2024-01-11T08:25:16.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-11T08:25:17.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.024+0000 7f1c771ff700 1 Processor -- start 2024-01-11T08:25:17.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.024+0000 7f1c771ff700 1 -- start start 2024-01-11T08:25:17.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.025+0000 7f1c771ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1c70157410 0x7f1c700ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:17.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.025+0000 7f1c771ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c700699d0 con 0x7f1c701577e0 2024-01-11T08:25:17.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.025+0000 7f1c771ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c70069cb0 con 0x7f1c70069540 2024-01-11T08:25:17.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.025+0000 7f1c771ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c70069f90 con 0x7f1c70157410 2024-01-11T08:25:17.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.025+0000 7f1c6ffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1c701577e0 0x7f1c700ae000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38990/0 (socket says 172.21.15.27:38990) 2024-01-11T08:25:17.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.025+0000 7f1c6ffff700 1 -- 172.21.15.27:0/2265034584 learned_addr learned my addr 172.21.15.27:0/2265034584 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:17.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.026+0000 7f1c74f9b700 1 --2- 172.21.15.27:0/2265034584 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1c70157410 0x7f1c700ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.026+0000 7f1c6f7fe700 1 -- 172.21.15.27:0/2265034584 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844105999 0 0) 0x7f1c700699d0 con 0x7f1c701577e0 2024-01-11T08:25:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.026+0000 7f1c6f7fe700 1 -- 172.21.15.27:0/2265034584 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c60003570 con 0x7f1c701577e0 2024-01-11T08:25:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.026+0000 7f1c6f7fe700 1 -- 172.21.15.27:0/2265034584 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117314107 0 0) 0x7f1c70069cb0 con 0x7f1c70069540 2024-01-11T08:25:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.026+0000 7f1c6f7fe700 1 -- 172.21.15.27:0/2265034584 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c700699d0 con 0x7f1c70069540 2024-01-11T08:25:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.026+0000 7f1c74f9b700 1 -- 172.21.15.27:0/2265034584 >> v1:172.21.15.77:6789/0 conn(0x7f1c70069540 legacy=0x7f1c700c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.026+0000 7f1c74f9b700 1 -- 172.21.15.27:0/2265034584 >> v1:172.21.15.27:6789/0 conn(0x7f1c701577e0 legacy=0x7f1c700ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.026+0000 7f1c74f9b700 1 -- 172.21.15.27:0/2265034584 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7006a270 con 0x7f1c70157410 2024-01-11T08:25:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.027+0000 7f1c74f9b700 1 --2- 172.21.15.27:0/2265034584 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1c70157410 0x7f1c700ccae0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1c64009900 tx=0x7f1c64009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e287ca38f7d6b00 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.027+0000 7f1c6f7fe700 1 -- 172.21.15.27:0/2265034584 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c64005c10 con 0x7f1c70157410 2024-01-11T08:25:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.027+0000 7f1c6f7fe700 1 -- 172.21.15.27:0/2265034584 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c64005d70 con 0x7f1c70157410 2024-01-11T08:25:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.027+0000 7f1c6f7fe700 1 -- 172.21.15.27:0/2265034584 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c64006260 con 0x7f1c70157410 2024-01-11T08:25:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.027+0000 7f1c771ff700 1 -- 172.21.15.27:0/2265034584 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1c70157410 msgr2=0x7f1c700ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:17.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.027+0000 7f1c771ff700 1 --2- 172.21.15.27:0/2265034584 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1c70157410 0x7f1c700ccae0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1c64009900 tx=0x7f1c64009120 comp rx=0 tx=0).stop 2024-01-11T08:25:17.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.027+0000 7f1c771ff700 1 -- 172.21.15.27:0/2265034584 shutdown_connections 2024-01-11T08:25:17.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.027+0000 7f1c771ff700 1 --2- 172.21.15.27:0/2265034584 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1c70157410 0x7f1c700ccae0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:17.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.027+0000 7f1c771ff700 1 -- 172.21.15.27:0/2265034584 >> 172.21.15.27:0/2265034584 conn(0x7f1c700680a0 msgr2=0x7f1c700cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:17.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.028+0000 7f1c771ff700 1 -- 172.21.15.27:0/2265034584 shutdown_connections 2024-01-11T08:25:17.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.028+0000 7f1c771ff700 1 -- 172.21.15.27:0/2265034584 wait complete. 2024-01-11T08:25:17.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.029+0000 7f1c771ff700 1 Processor -- start 2024-01-11T08:25:17.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.029+0000 7f1c771ff700 1 -- start start 2024-01-11T08:25:17.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.030+0000 7f1c771ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c70069540 0x7f1c7014de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.030+0000 7f1c771ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1c70157410 0x7f1c700ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.030+0000 7f1c771ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c701577e0 0x7f1c7015e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.030+0000 7f1c771ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c700b00a0 con 0x7f1c701577e0 2024-01-11T08:25:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.030+0000 7f1c771ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7014e340 con 0x7f1c70069540 2024-01-11T08:25:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.030+0000 7f1c771ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c700af490 con 0x7f1c70157410 2024-01-11T08:25:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.030+0000 7f1c7579c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c701577e0 0x7f1c7015e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.030+0000 7f1c7579c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c701577e0 0x7f1c7015e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47142/0 (socket says 172.21.15.27:47142) 2024-01-11T08:25:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.030+0000 7f1c7579c700 1 -- 172.21.15.27:0/198816445 learned_addr learned my addr 172.21.15.27:0/198816445 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.030+0000 7f1c74f9b700 1 --2- 172.21.15.27:0/198816445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c70069540 0x7f1c7014de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.031+0000 7f1c6ffff700 1 --2- 172.21.15.27:0/198816445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1c70157410 0x7f1c700ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.031+0000 7f1c7579c700 1 -- 172.21.15.27:0/198816445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1c70157410 msgr2=0x7f1c700ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.031+0000 7f1c7579c700 1 --2- 172.21.15.27:0/198816445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1c70157410 0x7f1c700ca740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.031+0000 7f1c7579c700 1 -- 172.21.15.27:0/198816445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c70069540 msgr2=0x7f1c7014de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:17.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.031+0000 7f1c7579c700 1 --2- 172.21.15.27:0/198816445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c70069540 0x7f1c7014de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:17.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.031+0000 7f1c7579c700 1 -- 172.21.15.27:0/198816445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c640093c0 con 0x7f1c701577e0 2024-01-11T08:25:17.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.031+0000 7f1c7579c700 1 --2- 172.21.15.27:0/198816445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c701577e0 0x7f1c7015e470 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800c910 tx=0x7f1c68007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:17.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.031+0000 7f1c6cff9700 1 -- 172.21.15.27:0/198816445 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c68012630 con 0x7f1c701577e0 2024-01-11T08:25:17.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.031+0000 7f1c6cff9700 1 -- 172.21.15.27:0/198816445 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c680127b0 con 0x7f1c701577e0 2024-01-11T08:25:17.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.031+0000 7f1c771ff700 1 -- 172.21.15.27:0/198816445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c700af770 con 0x7f1c701577e0 2024-01-11T08:25:17.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.032+0000 7f1c6cff9700 1 -- 172.21.15.27:0/198816445 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c68012b70 con 0x7f1c701577e0 2024-01-11T08:25:17.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.032+0000 7f1c771ff700 1 -- 172.21.15.27:0/198816445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c700afd30 con 0x7f1c701577e0 2024-01-11T08:25:17.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.033+0000 7f1c771ff700 1 -- 172.21.15.27:0/198816445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c700b0250 con 0x7f1c701577e0 2024-01-11T08:25:17.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.034+0000 7f1c6cff9700 1 -- 172.21.15.27:0/198816445 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1c6801b400 con 0x7f1c701577e0 2024-01-11T08:25:17.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.035+0000 7f1c6cff9700 1 --2- 172.21.15.27:0/198816445 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1c54066f30 0x7f1c540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:17.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.035+0000 7f1c6cff9700 1 -- 172.21.15.27:0/198816445 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5483+0+0 (secure 0 0 0) 0x7f1c68084770 con 0x7f1c701577e0 2024-01-11T08:25:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.036+0000 7f1c74f9b700 1 --2- 172.21.15.27:0/198816445 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1c54066f30 0x7f1c540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:17.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.036+0000 7f1c74f9b700 1 --2- 172.21.15.27:0/198816445 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1c54066f30 0x7f1c540693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1c64009390 tx=0x7f1c64016000 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:17.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.040+0000 7f1c6cff9700 1 -- 172.21.15.27:0/198816445 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1c700b0250 con 0x7f1c701577e0 2024-01-11T08:25:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.245+0000 7f1c771ff700 1 -- 172.21.15.27:0/198816445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1c7004c3c0 con 0x7f1c701577e0 2024-01-11T08:25:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.249+0000 7f1c6cff9700 1 -- 172.21.15.27:0/198816445 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1c680160c0 con 0x7f1c701577e0 2024-01-11T08:25:17.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.251+0000 7f1c771ff700 1 -- 172.21.15.27:0/198816445 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1c54066f30 msgr2=0x7f1c540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:17.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.251+0000 7f1c771ff700 1 --2- 172.21.15.27:0/198816445 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1c54066f30 0x7f1c540693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1c64009390 tx=0x7f1c64016000 comp rx=0 tx=0).stop 2024-01-11T08:25:17.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.251+0000 7f1c771ff700 1 -- 172.21.15.27:0/198816445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c701577e0 msgr2=0x7f1c7015e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:17.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.251+0000 7f1c771ff700 1 --2- 172.21.15.27:0/198816445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c701577e0 0x7f1c7015e470 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800c910 tx=0x7f1c68007f90 comp rx=0 tx=0).stop 2024-01-11T08:25:17.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.251+0000 7f1c771ff700 1 -- 172.21.15.27:0/198816445 shutdown_connections 2024-01-11T08:25:17.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.251+0000 7f1c771ff700 1 --2- 172.21.15.27:0/198816445 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f1c54066f30 0x7f1c540693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:17.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.251+0000 7f1c771ff700 1 --2- 172.21.15.27:0/198816445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c701577e0 0x7f1c7015e470 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:17.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.251+0000 7f1c771ff700 1 --2- 172.21.15.27:0/198816445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c70069540 0x7f1c7014de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:17.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.251+0000 7f1c771ff700 1 --2- 172.21.15.27:0/198816445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1c70157410 0x7f1c700ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:17.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.251+0000 7f1c771ff700 1 -- 172.21.15.27:0/198816445 >> 172.21.15.27:0/198816445 conn(0x7f1c700680a0 msgr2=0x7f1c700c1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:17.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.251+0000 7f1c771ff700 1 -- 172.21.15.27:0/198816445 shutdown_connections 2024-01-11T08:25:17.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.251+0000 7f1c771ff700 1 -- 172.21.15.27:0/198816445 wait complete. 2024-01-11T08:25:17.268 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T08:25:17.268 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:25:17.360 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:25:17.455 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T08:25:17.455 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-11T08:25:17.455 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:25:17.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.656+0000 7fbacafff700 1 Processor -- start 2024-01-11T08:25:17.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.656+0000 7fbacafff700 1 -- start start 2024-01-11T08:25:17.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.657+0000 7fbacafff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbac41575e0 0x7fbac4158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:17.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.657+0000 7fbacafff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbac406b610 0x7fbac40b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:17.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.658+0000 7fbacafff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbac40b1800 0x7fbac40ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:17.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.658+0000 7fbacafff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac40cb5e0 con 0x7fbac406b610 2024-01-11T08:25:17.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.658+0000 7fbacafff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac4158b60 con 0x7fbac41575e0 2024-01-11T08:25:17.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.658+0000 7fbacafff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac40b1c60 con 0x7fbac40b1800 2024-01-11T08:25:17.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.658+0000 7fbac3fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbac406b610 0x7fbac40b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:17.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.658+0000 7fbac3fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbac406b610 0x7fbac40b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47156/0 (socket says 172.21.15.27:47156) 2024-01-11T08:25:17.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.658+0000 7fbac3fff700 1 -- 172.21.15.27:0/1620919724 learned_addr learned my addr 172.21.15.27:0/1620919724 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:17.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.658+0000 7fbac8d9b700 1 --2- 172.21.15.27:0/1620919724 >> v2:172.21.15.77:3300/0 conn(0x7fbac41575e0 0x7fbac4158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:17.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.658+0000 7fbac959c700 1 --2- 172.21.15.27:0/1620919724 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbac40b1800 0x7fbac40ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:17.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.658+0000 7fbac3fff700 1 -- 172.21.15.27:0/1620919724 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbac40b1800 msgr2=0x7fbac40ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:17.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.658+0000 7fbac3fff700 1 --2- 172.21.15.27:0/1620919724 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbac40b1800 0x7fbac40ca1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:17.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.658+0000 7fbac3fff700 1 -- 172.21.15.27:0/1620919724 >> v2:172.21.15.77:3300/0 conn(0x7fbac41575e0 msgr2=0x7fbac4158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:17.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.658+0000 7fbac3fff700 1 --2- 172.21.15.27:0/1620919724 >> v2:172.21.15.77:3300/0 conn(0x7fbac41575e0 0x7fbac4158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:17.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.658+0000 7fbac3fff700 1 -- 172.21.15.27:0/1620919724 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac41599b0 con 0x7fbac406b610 2024-01-11T08:25:17.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.658+0000 7fbac8d9b700 1 --2- 172.21.15.27:0/1620919724 >> v2:172.21.15.77:3300/0 conn(0x7fbac41575e0 0x7fbac4158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:17.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.659+0000 7fbac3fff700 1 --2- 172.21.15.27:0/1620919724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac406b610 0x7fbac40b12c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fbac4064b00 tx=0x7fbab0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bdee35fbcc94eff server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:17.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.659+0000 7fbac37fe700 1 -- 172.21.15.27:0/1620919724 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab000b860 con 0x7fbac406b610 2024-01-11T08:25:17.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.659+0000 7fbac37fe700 1 -- 172.21.15.27:0/1620919724 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab000b9c0 con 0x7fbac406b610 2024-01-11T08:25:17.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.659+0000 7fbac37fe700 1 -- 172.21.15.27:0/1620919724 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab000bd60 con 0x7fbac406b610 2024-01-11T08:25:17.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.659+0000 7fbacafff700 1 -- 172.21.15.27:0/1620919724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac406b610 msgr2=0x7fbac40b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:17.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.659+0000 7fbacafff700 1 --2- 172.21.15.27:0/1620919724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac406b610 0x7fbac40b12c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fbac4064b00 tx=0x7fbab0009250 comp rx=0 tx=0).stop 2024-01-11T08:25:17.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.660+0000 7fbacafff700 1 -- 172.21.15.27:0/1620919724 shutdown_connections 2024-01-11T08:25:17.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.660+0000 7fbacafff700 1 --2- 172.21.15.27:0/1620919724 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbac40b1800 0x7fbac40ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:17.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.660+0000 7fbacafff700 1 --2- 172.21.15.27:0/1620919724 >> v2:172.21.15.77:3300/0 conn(0x7fbac41575e0 0x7fbac4158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:17.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.660+0000 7fbacafff700 1 --2- 172.21.15.27:0/1620919724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac406b610 0x7fbac40b12c0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:17.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.660+0000 7fbacafff700 1 -- 172.21.15.27:0/1620919724 >> 172.21.15.27:0/1620919724 conn(0x7fbac40680e0 msgr2=0x7fbac4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:17.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.660+0000 7fbacafff700 1 -- 172.21.15.27:0/1620919724 shutdown_connections 2024-01-11T08:25:17.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.660+0000 7fbacafff700 1 -- 172.21.15.27:0/1620919724 wait complete. 2024-01-11T08:25:17.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.661+0000 7fbacafff700 1 Processor -- start 2024-01-11T08:25:17.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.661+0000 7fbacafff700 1 -- start start 2024-01-11T08:25:17.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.662+0000 7fbacafff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac406b610 0x7fbac40c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:17.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.662+0000 7fbacafff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbac40b1800 0x7fbac4148c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:17.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.662+0000 7fbacafff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbac41575e0 0x7fbac416a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:17.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.662+0000 7fbacafff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac4169fa0 con 0x7fbac406b610 2024-01-11T08:25:17.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.662+0000 7fbacafff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac40c88a0 con 0x7fbac41575e0 2024-01-11T08:25:17.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.662+0000 7fbacafff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac4169390 con 0x7fbac40b1800 2024-01-11T08:25:17.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.662+0000 7fbac8d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac406b610 0x7fbac40c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:17.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.662+0000 7fbac8d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac406b610 0x7fbac40c8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47172/0 (socket says 172.21.15.27:47172) 2024-01-11T08:25:17.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.662+0000 7fbac8d9b700 1 -- 172.21.15.27:0/4281321736 learned_addr learned my addr 172.21.15.27:0/4281321736 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:17.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.663+0000 7fbac3fff700 1 --2- 172.21.15.27:0/4281321736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbac40b1800 0x7fbac4148c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:17.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.663+0000 7fbac959c700 1 --2- 172.21.15.27:0/4281321736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbac41575e0 0x7fbac416a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:17.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.663+0000 7fbac8d9b700 1 -- 172.21.15.27:0/4281321736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbac40b1800 msgr2=0x7fbac4148c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:17.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.663+0000 7fbac8d9b700 1 --2- 172.21.15.27:0/4281321736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbac40b1800 0x7fbac4148c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:17.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.663+0000 7fbac8d9b700 1 -- 172.21.15.27:0/4281321736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbac41575e0 msgr2=0x7fbac416a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:17.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.663+0000 7fbac8d9b700 1 --2- 172.21.15.27:0/4281321736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbac41575e0 0x7fbac416a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:17.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.663+0000 7fbac8d9b700 1 -- 172.21.15.27:0/4281321736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab00094f0 con 0x7fbac406b610 2024-01-11T08:25:17.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.663+0000 7fbac8d9b700 1 --2- 172.21.15.27:0/4281321736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac406b610 0x7fbac40c8360 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fbab800d890 tx=0x7fbab8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:17.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.663+0000 7fbac0ff9700 1 -- 172.21.15.27:0/4281321736 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab80125f0 con 0x7fbac406b610 2024-01-11T08:25:17.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.663+0000 7fbac0ff9700 1 -- 172.21.15.27:0/4281321736 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab8012750 con 0x7fbac406b610 2024-01-11T08:25:17.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.663+0000 7fbac0ff9700 1 -- 172.21.15.27:0/4281321736 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab8012af0 con 0x7fbac406b610 2024-01-11T08:25:17.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.663+0000 7fbacafff700 1 -- 172.21.15.27:0/4281321736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac4169670 con 0x7fbac406b610 2024-01-11T08:25:17.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.664+0000 7fbacafff700 1 -- 172.21.15.27:0/4281321736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac4169c30 con 0x7fbac406b610 2024-01-11T08:25:17.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.665+0000 7fbacafff700 1 -- 172.21.15.27:0/4281321736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac416a280 con 0x7fbac406b610 2024-01-11T08:25:17.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.665+0000 7fbac0ff9700 1 -- 172.21.15.27:0/4281321736 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fbab8014430 con 0x7fbac406b610 2024-01-11T08:25:17.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.666+0000 7fbac0ff9700 1 --2- 172.21.15.27:0/4281321736 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fbaa8066f30 0x7fbaa80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:17.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.667+0000 7fbac0ff9700 1 -- 172.21.15.27:0/4281321736 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5483+0+0 (secure 0 0 0) 0x7fbab8084820 con 0x7fbac406b610 2024-01-11T08:25:17.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.667+0000 7fbac3fff700 1 --2- 172.21.15.27:0/4281321736 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fbaa8066f30 0x7fbaa80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:17.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.667+0000 7fbac3fff700 1 --2- 172.21.15.27:0/4281321736 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fbaa8066f30 0x7fbaa80693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbac4064b00 tx=0x7fbab0017000 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:17.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.670+0000 7fbac0ff9700 1 -- 172.21.15.27:0/4281321736 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbab8051ad0 con 0x7fbac406b610 2024-01-11T08:25:17.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.918+0000 7fbacafff700 1 -- 172.21.15.27:0/4281321736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbac40718d0 con 0x7fbac406b610 2024-01-11T08:25:17.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.919+0000 7fbac0ff9700 1 -- 172.21.15.27:0/4281321736 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+3523 (secure 0 0 0) 0x7fbab80550f0 con 0x7fbac406b610 2024-01-11T08:25:17.918 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:17.918 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5243,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2418922463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2418922463},{"type":"v1","addr":"172.21.15.27:6835","nonce":2418922463}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5246,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6836/3919152683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3919152683},{"type":"v1","addr":"172.21.15.77:6836","nonce":3919152683}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5641,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/329572459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":329572459},{"type":"v1","addr":"172.21.15.77:6837","nonce":329572459}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5968,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/585478493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":585478493},{"type":"v1","addr":"172.21.15.27:6834","nonce":585478493}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[]} 2024-01-11T08:25:17.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.921+0000 7fbacafff700 1 -- 172.21.15.27:0/4281321736 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fbaa8066f30 msgr2=0x7fbaa80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:17.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.921+0000 7fbacafff700 1 --2- 172.21.15.27:0/4281321736 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fbaa8066f30 0x7fbaa80693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbac4064b00 tx=0x7fbab0017000 comp rx=0 tx=0).stop 2024-01-11T08:25:17.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.921+0000 7fbacafff700 1 -- 172.21.15.27:0/4281321736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac406b610 msgr2=0x7fbac40c8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:17.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.921+0000 7fbacafff700 1 --2- 172.21.15.27:0/4281321736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac406b610 0x7fbac40c8360 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fbab800d890 tx=0x7fbab8008040 comp rx=0 tx=0).stop 2024-01-11T08:25:17.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.921+0000 7fbacafff700 1 -- 172.21.15.27:0/4281321736 shutdown_connections 2024-01-11T08:25:17.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.921+0000 7fbacafff700 1 --2- 172.21.15.27:0/4281321736 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fbaa8066f30 0x7fbaa80693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:17.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.921+0000 7fbacafff700 1 --2- 172.21.15.27:0/4281321736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbac41575e0 0x7fbac416a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:17.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.921+0000 7fbacafff700 1 --2- 172.21.15.27:0/4281321736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac406b610 0x7fbac40c8360 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:17.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.921+0000 7fbacafff700 1 --2- 172.21.15.27:0/4281321736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbac40b1800 0x7fbac4148c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:17.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.921+0000 7fbacafff700 1 -- 172.21.15.27:0/4281321736 >> 172.21.15.27:0/4281321736 conn(0x7fbac40680e0 msgr2=0x7fbac4159b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:17.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.921+0000 7fbacafff700 1 -- 172.21.15.27:0/4281321736 shutdown_connections 2024-01-11T08:25:17.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:17.921+0000 7fbacafff700 1 -- 172.21.15.27:0/4281321736 wait complete. 2024-01-11T08:25:17.921 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-01-11T08:25:17.938 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-11T08:25:17.938 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-11T08:25:17.939 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-11T08:25:17.939 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:25:17.939 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-11T08:25:17.939 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-11T08:25:17.939 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-11T08:25:17.939 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:25:17.939 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-11T08:25:17.939 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-11T08:25:17.940 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-11T08:25:17.940 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:25:17.940 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-11T08:25:17.940 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-11T08:25:17.940 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-11T08:25:17.940 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:25:17.952 INFO:tasks.ceph.mds.a:Stopped 2024-01-11T08:25:17.952 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-11T08:25:17.953 INFO:tasks.ceph.mds.c:Stopped 2024-01-11T08:25:17.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-11T08:25:17.956 INFO:tasks.ceph.mds.b:Stopped 2024-01-11T08:25:17.956 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-11T08:25:17.957 INFO:tasks.ceph.mds.d:Stopped 2024-01-11T08:25:17.958 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-11T08:25:17.997 INFO:tasks.ceph.mds.a:Started 2024-01-11T08:25:17.998 DEBUG:teuthology.parallel:result is None 2024-01-11T08:25:18.003 INFO:tasks.ceph.mds.b:Started 2024-01-11T08:25:18.004 DEBUG:teuthology.parallel:result is None 2024-01-11T08:25:18.005 INFO:tasks.ceph.mds.c:Started 2024-01-11T08:25:18.005 DEBUG:teuthology.parallel:result is None 2024-01-11T08:25:18.007 INFO:tasks.ceph.mds.d:Started 2024-01-11T08:25:18.007 DEBUG:teuthology.parallel:result is None 2024-01-11T08:25:18.007 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-11T08:25:18.150 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T08:25:18.151 INFO:tasks.ceph.mds.b.smithi077.stdout:starting mds.b at 2024-01-11T08:25:18.151 INFO:tasks.ceph.mds.d.smithi077.stdout:starting mds.d at 2024-01-11T08:25:18.163 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T08:25:18.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.218+0000 7f4febdff700 1 Processor -- start 2024-01-11T08:25:18.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.218+0000 7f4febdff700 1 -- start start 2024-01-11T08:25:18.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.220+0000 7f4febdff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4fe415ca10 0x7f4fe415a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:18.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.220+0000 7f4febdff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4fe4068cd0 0x7f4fe40b5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:18.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.220+0000 7f4febdff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe40b4bd0 con 0x7f4fe415cde0 2024-01-11T08:25:18.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.220+0000 7f4febdff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe415ac70 con 0x7f4fe4068cd0 2024-01-11T08:25:18.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.220+0000 7f4febdff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe40b3fc0 con 0x7f4fe415ca10 2024-01-11T08:25:18.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.220+0000 7f4fe939a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4fe415cde0 0x7f4fe40685c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38998/0 (socket says 172.21.15.27:38998) 2024-01-11T08:25:18.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.220+0000 7f4fe939a700 1 -- 172.21.15.27:0/3799281544 learned_addr learned my addr 172.21.15.27:0/3799281544 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:18.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.220+0000 7f4fe9b9b700 1 --2- 172.21.15.27:0/3799281544 >> v2:172.21.15.77:3300/0 conn(0x7f4fe415ca10 0x7f4fe415a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:18.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.220+0000 7f4fea39c700 1 --2- 172.21.15.27:0/3799281544 >> v2:172.21.15.27:3300/0 conn(0x7f4fe4068cd0 0x7f4fe40b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:18.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.220+0000 7f4fe8b99700 1 -- 172.21.15.27:0/3799281544 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060898084 0 0) 0x7f4fe40b4bd0 con 0x7f4fe415cde0 2024-01-11T08:25:18.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.221+0000 7f4fe8b99700 1 -- 172.21.15.27:0/3799281544 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd0003570 con 0x7f4fe415cde0 2024-01-11T08:25:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.221+0000 7f4fea39c700 1 -- 172.21.15.27:0/3799281544 >> v2:172.21.15.77:3300/0 conn(0x7f4fe415ca10 msgr2=0x7f4fe415a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.221+0000 7f4fea39c700 1 --2- 172.21.15.27:0/3799281544 >> v2:172.21.15.77:3300/0 conn(0x7f4fe415ca10 0x7f4fe415a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.221+0000 7f4fea39c700 1 -- 172.21.15.27:0/3799281544 >> v1:172.21.15.27:6789/0 conn(0x7f4fe415cde0 legacy=0x7f4fe40685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.221+0000 7f4fea39c700 1 -- 172.21.15.27:0/3799281544 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe40b42a0 con 0x7f4fe4068cd0 2024-01-11T08:25:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.221+0000 7f4fe9b9b700 1 --2- 172.21.15.27:0/3799281544 >> v2:172.21.15.77:3300/0 conn(0x7f4fe415ca10 0x7f4fe415a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.221+0000 7f4fea39c700 1 --2- 172.21.15.27:0/3799281544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fe4068cd0 0x7f4fe40b5060 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40b6630 tx=0x7f4fdc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7fd21275725e7e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.221+0000 7f4fe8b99700 1 -- 172.21.15.27:0/3799281544 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc00e070 con 0x7f4fe4068cd0 2024-01-11T08:25:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.221+0000 7f4fe8b99700 1 -- 172.21.15.27:0/3799281544 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fdc00bab0 con 0x7f4fe4068cd0 2024-01-11T08:25:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.221+0000 7f4fe8b99700 1 -- 172.21.15.27:0/3799281544 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc00be70 con 0x7f4fe4068cd0 2024-01-11T08:25:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.222+0000 7f4febdff700 1 -- 172.21.15.27:0/3799281544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fe4068cd0 msgr2=0x7f4fe40b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.222+0000 7f4febdff700 1 --2- 172.21.15.27:0/3799281544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fe4068cd0 0x7f4fe40b5060 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40b6630 tx=0x7f4fdc009780 comp rx=0 tx=0).stop 2024-01-11T08:25:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.222+0000 7f4febdff700 1 -- 172.21.15.27:0/3799281544 shutdown_connections 2024-01-11T08:25:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.222+0000 7f4febdff700 1 --2- 172.21.15.27:0/3799281544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fe4068cd0 0x7f4fe40b5060 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:18.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.222+0000 7f4febdff700 1 --2- 172.21.15.27:0/3799281544 >> v2:172.21.15.77:3300/0 conn(0x7f4fe415ca10 0x7f4fe415a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:18.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.222+0000 7f4febdff700 1 -- 172.21.15.27:0/3799281544 >> 172.21.15.27:0/3799281544 conn(0x7f4fe406ed40 msgr2=0x7f4fe406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:18.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.222+0000 7f4febdff700 1 -- 172.21.15.27:0/3799281544 shutdown_connections 2024-01-11T08:25:18.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.222+0000 7f4febdff700 1 -- 172.21.15.27:0/3799281544 wait complete. 2024-01-11T08:25:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.223+0000 7f4febdff700 1 Processor -- start 2024-01-11T08:25:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.224+0000 7f4febdff700 1 -- start start 2024-01-11T08:25:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.224+0000 7f4febdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fe4068cd0 0x7f4fe41450c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.224+0000 7f4febdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4fe415ca10 0x7f4fe4156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.224+0000 7f4febdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fe415cde0 0x7f4fe4159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.224+0000 7f4febdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe4145600 con 0x7f4fe415cde0 2024-01-11T08:25:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.224+0000 7f4febdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe4158b30 con 0x7f4fe4068cd0 2024-01-11T08:25:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.224+0000 7f4febdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe41544a0 con 0x7f4fe415ca10 2024-01-11T08:25:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.225+0000 7f4fea39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fe415cde0 0x7f4fe4159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.225+0000 7f4fe939a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4fe415ca10 0x7f4fe4156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.225+0000 7f4fe939a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4fe415ca10 0x7f4fe4156c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:45804/0 (socket says 172.21.15.27:45804) 2024-01-11T08:25:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.225+0000 7f4fea39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fe415cde0 0x7f4fe4159520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47210/0 (socket says 172.21.15.27:47210) 2024-01-11T08:25:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.225+0000 7f4fe939a700 1 -- 172.21.15.27:0/3977364705 learned_addr learned my addr 172.21.15.27:0/3977364705 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.225+0000 7f4fe9b9b700 1 --2- 172.21.15.27:0/3977364705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fe4068cd0 0x7f4fe41450c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.225+0000 7f4fe939a700 1 -- 172.21.15.27:0/3977364705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fe4068cd0 msgr2=0x7f4fe41450c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.225+0000 7f4fe939a700 1 --2- 172.21.15.27:0/3977364705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fe4068cd0 0x7f4fe41450c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.225+0000 7f4fe939a700 1 -- 172.21.15.27:0/3977364705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fe415cde0 msgr2=0x7f4fe4159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.225+0000 7f4fe939a700 1 --2- 172.21.15.27:0/3977364705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fe415cde0 0x7f4fe4159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.225+0000 7f4fe939a700 1 -- 172.21.15.27:0/3977364705 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe0009420 con 0x7f4fe415ca10 2024-01-11T08:25:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.225+0000 7f4fea39c700 1 --2- 172.21.15.27:0/3977364705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fe415cde0 0x7f4fe4159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.226+0000 7f4fe939a700 1 --2- 172.21.15.27:0/3977364705 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4fe415ca10 0x7f4fe4156c10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4fd400c590 tx=0x7f4fd4007dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.226+0000 7f4fd9ffb700 1 -- 172.21.15.27:0/3977364705 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd4005b30 con 0x7f4fe415ca10 2024-01-11T08:25:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.226+0000 7f4fd9ffb700 1 -- 172.21.15.27:0/3977364705 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd4005cb0 con 0x7f4fe415ca10 2024-01-11T08:25:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.226+0000 7f4febdff700 1 -- 172.21.15.27:0/3977364705 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fdc009a20 con 0x7f4fe415ca10 2024-01-11T08:25:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.226+0000 7f4fd9ffb700 1 -- 172.21.15.27:0/3977364705 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd4007320 con 0x7f4fe415ca10 2024-01-11T08:25:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.226+0000 7f4febdff700 1 -- 172.21.15.27:0/3977364705 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe4158850 con 0x7f4fe415ca10 2024-01-11T08:25:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.227+0000 7f4febdff700 1 -- 172.21.15.27:0/3977364705 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe4154660 con 0x7f4fe415ca10 2024-01-11T08:25:18.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.228+0000 7f4fd9ffb700 1 -- 172.21.15.27:0/3977364705 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4fd401b3b0 con 0x7f4fe415ca10 2024-01-11T08:25:18.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.229+0000 7f4fd9ffb700 1 --2- 172.21.15.27:0/3977364705 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4fc8066f60 0x7f4fc8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:18.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.229+0000 7f4fd9ffb700 1 -- 172.21.15.27:0/3977364705 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5483+0+0 (secure 0 0 0) 0x7f4fd4083840 con 0x7f4fe415ca10 2024-01-11T08:25:18.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.229+0000 7f4fe9b9b700 1 --2- 172.21.15.27:0/3977364705 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4fc8066f60 0x7f4fc8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:18.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.230+0000 7f4fe9b9b700 1 --2- 172.21.15.27:0/3977364705 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4fc8066f60 0x7f4fc8069410 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4fe415ac40 tx=0x7f4fe00091a0 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.234+0000 7f4fd9ffb700 1 -- 172.21.15.27:0/3977364705 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4fd4050af0 con 0x7f4fe415ca10 2024-01-11T08:25:18.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.450+0000 7f4febdff700 1 -- 172.21.15.27:0/3977364705 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f4fe404c3c0 con 0x7f4fe415ca10 2024-01-11T08:25:18.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.814+0000 7f4fd9ffb700 1 -- 172.21.15.27:0/3977364705 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v69) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4fd4017080 con 0x7f4fe415ca10 2024-01-11T08:25:18.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.816+0000 7f4febdff700 1 -- 172.21.15.27:0/3977364705 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4fc8066f60 msgr2=0x7f4fc8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:18.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.816+0000 7f4febdff700 1 --2- 172.21.15.27:0/3977364705 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4fc8066f60 0x7f4fc8069410 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4fe415ac40 tx=0x7f4fe00091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:18.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.816+0000 7f4febdff700 1 -- 172.21.15.27:0/3977364705 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4fe415ca10 msgr2=0x7f4fe4156c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:18.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.816+0000 7f4febdff700 1 --2- 172.21.15.27:0/3977364705 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4fe415ca10 0x7f4fe4156c10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4fd400c590 tx=0x7f4fd4007dd0 comp rx=0 tx=0).stop 2024-01-11T08:25:18.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.816+0000 7f4febdff700 1 -- 172.21.15.27:0/3977364705 shutdown_connections 2024-01-11T08:25:18.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.816+0000 7f4febdff700 1 --2- 172.21.15.27:0/3977364705 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4fc8066f60 0x7f4fc8069410 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:18.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.816+0000 7f4febdff700 1 --2- 172.21.15.27:0/3977364705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fe415cde0 0x7f4fe4159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:18.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.816+0000 7f4febdff700 1 --2- 172.21.15.27:0/3977364705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fe4068cd0 0x7f4fe41450c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:18.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.816+0000 7f4febdff700 1 --2- 172.21.15.27:0/3977364705 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4fe415ca10 0x7f4fe4156c10 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:18.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.816+0000 7f4febdff700 1 -- 172.21.15.27:0/3977364705 >> 172.21.15.27:0/3977364705 conn(0x7f4fe406ed40 msgr2=0x7f4fe40b54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:18.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.817+0000 7f4febdff700 1 -- 172.21.15.27:0/3977364705 shutdown_connections 2024-01-11T08:25:18.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:18.817+0000 7f4febdff700 1 -- 172.21.15.27:0/3977364705 wait complete. 2024-01-11T08:25:18.817 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-11T08:25:18.836 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-11T08:25:19.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.039+0000 7fb9265ff700 1 Processor -- start 2024-01-11T08:25:19.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.039+0000 7fb9265ff700 1 -- start start 2024-01-11T08:25:19.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.040+0000 7fb9265ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb9200b03e0 0x7fb9200b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:19.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.041+0000 7fb9265ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb9200b29c0 0x7fb920159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:19.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.041+0000 7fb9265ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9200cd730 con 0x7fb9200b07b0 2024-01-11T08:25:19.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.041+0000 7fb9265ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9200b2de0 con 0x7fb9200b29c0 2024-01-11T08:25:19.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.041+0000 7fb9265ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9200ccb20 con 0x7fb9200b03e0 2024-01-11T08:25:19.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.041+0000 7fb91f7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb9200b07b0 0x7fb920159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39004/0 (socket says 172.21.15.27:39004) 2024-01-11T08:25:19.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.041+0000 7fb91f7fe700 1 -- 172.21.15.27:0/1984917528 learned_addr learned my addr 172.21.15.27:0/1984917528 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:19.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.041+0000 7fb924b9c700 1 --2- 172.21.15.27:0/1984917528 >> v2:172.21.15.27:3300/0 conn(0x7fb9200b29c0 0x7fb920159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:19.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.041+0000 7fb91ffff700 1 --2- 172.21.15.27:0/1984917528 >> v2:172.21.15.77:3300/0 conn(0x7fb9200b03e0 0x7fb9200b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:19.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.041+0000 7fb924b9c700 1 -- 172.21.15.27:0/1984917528 >> v2:172.21.15.77:3300/0 conn(0x7fb9200b03e0 msgr2=0x7fb9200b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:19.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.041+0000 7fb924b9c700 1 --2- 172.21.15.27:0/1984917528 >> v2:172.21.15.77:3300/0 conn(0x7fb9200b03e0 0x7fb9200b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.041+0000 7fb924b9c700 1 -- 172.21.15.27:0/1984917528 >> v1:172.21.15.27:6789/0 conn(0x7fb9200b07b0 legacy=0x7fb920159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:19.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.041+0000 7fb924b9c700 1 -- 172.21.15.27:0/1984917528 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9200cce00 con 0x7fb9200b29c0 2024-01-11T08:25:19.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.042+0000 7fb924b9c700 1 --2- 172.21.15.27:0/1984917528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9200b29c0 0x7fb920159910 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb920089470 tx=0x7fb918009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a80ed9e72dceeb59 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:19.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.042+0000 7fb91effd700 1 -- 172.21.15.27:0/1984917528 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91800e070 con 0x7fb9200b29c0 2024-01-11T08:25:19.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.042+0000 7fb91effd700 1 -- 172.21.15.27:0/1984917528 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91800bb60 con 0x7fb9200b29c0 2024-01-11T08:25:19.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.042+0000 7fb91effd700 1 -- 172.21.15.27:0/1984917528 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb918012430 con 0x7fb9200b29c0 2024-01-11T08:25:19.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.042+0000 7fb9265ff700 1 -- 172.21.15.27:0/1984917528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9200b29c0 msgr2=0x7fb920159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:19.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.042+0000 7fb9265ff700 1 --2- 172.21.15.27:0/1984917528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9200b29c0 0x7fb920159910 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb920089470 tx=0x7fb918009780 comp rx=0 tx=0).stop 2024-01-11T08:25:19.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.042+0000 7fb9265ff700 1 -- 172.21.15.27:0/1984917528 shutdown_connections 2024-01-11T08:25:19.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.042+0000 7fb9265ff700 1 --2- 172.21.15.27:0/1984917528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9200b29c0 0x7fb920159910 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.042+0000 7fb9265ff700 1 --2- 172.21.15.27:0/1984917528 >> v2:172.21.15.77:3300/0 conn(0x7fb9200b03e0 0x7fb9200b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.042+0000 7fb9265ff700 1 -- 172.21.15.27:0/1984917528 >> 172.21.15.27:0/1984917528 conn(0x7fb92006ed40 msgr2=0x7fb92006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:19.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.043+0000 7fb9265ff700 1 -- 172.21.15.27:0/1984917528 shutdown_connections 2024-01-11T08:25:19.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.043+0000 7fb9265ff700 1 -- 172.21.15.27:0/1984917528 wait complete. 2024-01-11T08:25:19.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.044+0000 7fb9265ff700 1 Processor -- start 2024-01-11T08:25:19.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.044+0000 7fb9265ff700 1 -- start start 2024-01-11T08:25:19.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.045+0000 7fb9265ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9200b03e0 0x7fb920149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:19.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.045+0000 7fb9265ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb9200b07b0 0x7fb9200c9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:19.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.045+0000 7fb9265ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9200b29c0 0x7fb9200c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:19.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.045+0000 7fb9265ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9200c6e50 con 0x7fb9200b03e0 2024-01-11T08:25:19.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.045+0000 7fb9265ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9200d0c10 con 0x7fb9200b29c0 2024-01-11T08:25:19.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.045+0000 7fb9265ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9200d0000 con 0x7fb9200b07b0 2024-01-11T08:25:19.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.046+0000 7fb91ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9200b03e0 0x7fb920149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:19.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.046+0000 7fb91ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9200b03e0 0x7fb920149470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47226/0 (socket says 172.21.15.27:47226) 2024-01-11T08:25:19.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.046+0000 7fb91ffff700 1 -- 172.21.15.27:0/2116928462 learned_addr learned my addr 172.21.15.27:0/2116928462 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:19.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.046+0000 7fb91f7fe700 1 --2- 172.21.15.27:0/2116928462 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb9200b07b0 0x7fb9200c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:19.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.046+0000 7fb924b9c700 1 --2- 172.21.15.27:0/2116928462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9200b29c0 0x7fb9200c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:19.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.046+0000 7fb91ffff700 1 -- 172.21.15.27:0/2116928462 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb9200b07b0 msgr2=0x7fb9200c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:19.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.046+0000 7fb91ffff700 1 --2- 172.21.15.27:0/2116928462 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb9200b07b0 0x7fb9200c9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.046+0000 7fb91ffff700 1 -- 172.21.15.27:0/2116928462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9200b29c0 msgr2=0x7fb9200c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:19.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.046+0000 7fb91ffff700 1 --2- 172.21.15.27:0/2116928462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9200b29c0 0x7fb9200c9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.046+0000 7fb91ffff700 1 -- 172.21.15.27:0/2116928462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb918009a20 con 0x7fb9200b03e0 2024-01-11T08:25:19.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.046+0000 7fb91ffff700 1 --2- 172.21.15.27:0/2116928462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9200b03e0 0x7fb920149470 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb91000ec80 tx=0x7fb9100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:19.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.046+0000 7fb907fff700 1 -- 172.21.15.27:0/2116928462 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91000c850 con 0x7fb9200b03e0 2024-01-11T08:25:19.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.047+0000 7fb907fff700 1 -- 172.21.15.27:0/2116928462 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91000c9b0 con 0x7fb9200b03e0 2024-01-11T08:25:19.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.047+0000 7fb907fff700 1 -- 172.21.15.27:0/2116928462 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91000cd50 con 0x7fb9200b03e0 2024-01-11T08:25:19.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.047+0000 7fb9265ff700 1 -- 172.21.15.27:0/2116928462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9200d02e0 con 0x7fb9200b03e0 2024-01-11T08:25:19.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.047+0000 7fb9265ff700 1 -- 172.21.15.27:0/2116928462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9200d08a0 con 0x7fb9200b03e0 2024-01-11T08:25:19.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.048+0000 7fb9265ff700 1 -- 172.21.15.27:0/2116928462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9200d0ef0 con 0x7fb9200b03e0 2024-01-11T08:25:19.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.049+0000 7fb907fff700 1 -- 172.21.15.27:0/2116928462 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb910013680 con 0x7fb9200b03e0 2024-01-11T08:25:19.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.050+0000 7fb907fff700 1 --2- 172.21.15.27:0/2116928462 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb90c066f30 0x7fb90c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:19.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.050+0000 7fb907fff700 1 -- 172.21.15.27:0/2116928462 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 4881+0+0 (secure 0 0 0) 0x7fb910084480 con 0x7fb9200b03e0 2024-01-11T08:25:19.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.050+0000 7fb91f7fe700 1 --2- 172.21.15.27:0/2116928462 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb90c066f30 0x7fb90c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:19.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.051+0000 7fb91f7fe700 1 --2- 172.21.15.27:0/2116928462 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb90c066f30 0x7fb90c0693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb92015acb0 tx=0x7fb914007eb0 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.054+0000 7fb907fff700 1 -- 172.21.15.27:0/2116928462 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb910051a20 con 0x7fb9200b03e0 2024-01-11T08:25:19.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.334+0000 7fb9265ff700 1 -- 172.21.15.27:0/2116928462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb9200718d0 con 0x7fb9200b03e0 2024-01-11T08:25:19.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.335+0000 7fb907fff700 1 -- 172.21.15.27:0/2116928462 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fb910055040 con 0x7fb9200b03e0 2024-01-11T08:25:19.334 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:19.334 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T08:25:19.334 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-11T08:25:19.334 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:19.334 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-11T08:25:19.334 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lQm3yLhAAv6erz5hMfat8EUzRA5luMA==", 2024-01-11T08:25:19.335 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:19.335 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:25:19.335 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:25:19.335 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:25:19.335 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:19.335 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.335 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:19.335 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:19.335 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-11T08:25:19.335 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9l+FG8ChAACHndtVLx7qS3sidPniSycA==", 2024-01-11T08:25:19.335 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:19.335 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:25:19.335 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:25:19.335 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:25:19.335 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:19.336 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.336 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:19.336 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:19.336 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-11T08:25:19.336 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l1NnEOhAALHXsc5es7hufCthGdRzkyw==", 2024-01-11T08:25:19.336 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:19.336 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:25:19.336 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:25:19.336 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:25:19.336 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:19.336 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.336 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:19.336 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:19.336 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-11T08:25:19.336 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lruoWFhAAm8AmrZtTJhYI1+19F79RPQ==", 2024-01-11T08:25:19.337 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:19.337 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:25:19.337 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:25:19.337 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:25:19.337 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:19.337 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.337 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:19.337 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:19.337 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-11T08:25:19.337 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA2op9ltZygORAAZr1X1nCN0ce8rIVaGDpFyQ==", 2024-01-11T08:25:19.337 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:19.337 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:25:19.337 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:25:19.337 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:19.337 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.338 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:19.338 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:19.338 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-11T08:25:19.338 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA6op9lqSRPAhAAmtyCsyK5ZVGX2wfhOCMuFQ==", 2024-01-11T08:25:19.338 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:19.338 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:25:19.338 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:25:19.338 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:19.338 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.338 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:19.338 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:19.338 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-11T08:25:19.338 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA9op9lZcs8BhAA8j+U+jPOoy+fliVWZAPFYg==", 2024-01-11T08:25:19.338 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:19.339 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:25:19.339 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:25:19.339 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:19.339 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.339 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:19.339 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:19.339 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-11T08:25:19.339 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBAop9lFuCUChAAP6si/2W1mxH4ecQJvuumkw==", 2024-01-11T08:25:19.339 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:19.339 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:25:19.339 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:25:19.339 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:19.339 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.339 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:19.340 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:19.340 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-11T08:25:19.340 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBGop9lOpwBIRAAcpcsjwmX2IldcKHJWAgHuA==", 2024-01-11T08:25:19.340 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:19.340 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:25:19.340 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:25:19.340 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:19.340 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.340 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:19.340 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:19.340 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-11T08:25:19.340 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBJop9l4q8hJRAAK8mAYVRQMoWS6wBOU69HmA==", 2024-01-11T08:25:19.340 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:19.340 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:25:19.341 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:25:19.341 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:19.341 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:19.341 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:19.341 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-11T08:25:19.341 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBMop9lqMF9KBAAFgFFI2GWt9AGif4AoScThw==", 2024-01-11T08:25:19.341 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:19.341 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:25:19.341 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:25:19.341 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:19.341 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:19.341 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:19.342 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-11T08:25:19.342 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBPop9lN7eDLBAAKswPZFfs0g2AaSe6V+pbbQ==", 2024-01-11T08:25:19.342 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:19.342 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:25:19.342 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:25:19.342 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:19.342 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.342 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:19.342 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:19.342 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-11T08:25:19.342 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lT5okIBAABy8PbJJbA062VNmb1Y8qhg==", 2024-01-11T08:25:19.342 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:19.342 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:25:19.342 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-01-11T08:25:19.342 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-11T08:25:19.343 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.343 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:19.343 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:19.343 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-11T08:25:19.343 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lDf37LhAAqV+EabKXJEW1sS5xtu1tXw==", 2024-01-11T08:25:19.343 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:19.343 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:25:19.343 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:25:19.343 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:25:19.343 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:25:19.343 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.343 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:19.343 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:19.343 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-11T08:25:19.343 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lWayxJxAAcalM8QSOGtbCXxjYZGaNQA==", 2024-01-11T08:25:19.343 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:19.344 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:25:19.344 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:25:19.344 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:25:19.344 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:25:19.344 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:19.344 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:19.344 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-11T08:25:19.344 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lHD4tNhAAdmnFBmMrK+QhQYdpbzW0mg==", 2024-01-11T08:25:19.344 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:19.344 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:25:19.344 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:25:19.344 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:25:19.344 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:25:19.344 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:19.345 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:19.345 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-11T08:25:19.345 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAwop9lGPtDMxAAJ8FjrSEdCtQL8AhtAyqnTw==", 2024-01-11T08:25:19.345 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:19.345 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:25:19.345 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:25:19.345 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:25:19.345 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:19.345 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.345 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:19.345 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:19.345 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-11T08:25:19.345 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lXBZLJxAAXa1S8Vkm3MiV4igtNbg/1Q==", 2024-01-11T08:25:19.345 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:19.345 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:25:19.345 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T08:25:19.346 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:19.346 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.346 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:19.346 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:19.346 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-11T08:25:19.346 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l3ZGoHxAANqzytTTvWYSomG+RFsFDyg==", 2024-01-11T08:25:19.346 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:19.346 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:25:19.346 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T08:25:19.346 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:19.346 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.346 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:19.346 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-11T08:25:19.346 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T08:25:19.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.337+0000 7fb9265ff700 1 -- 172.21.15.27:0/2116928462 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb90c066f30 msgr2=0x7fb90c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:19.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.337+0000 7fb9265ff700 1 --2- 172.21.15.27:0/2116928462 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb90c066f30 0x7fb90c0693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb92015acb0 tx=0x7fb914007eb0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.337+0000 7fb9265ff700 1 -- 172.21.15.27:0/2116928462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9200b03e0 msgr2=0x7fb920149470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:19.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.337+0000 7fb9265ff700 1 --2- 172.21.15.27:0/2116928462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9200b03e0 0x7fb920149470 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb91000ec80 tx=0x7fb9100091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.337+0000 7fb9265ff700 1 -- 172.21.15.27:0/2116928462 shutdown_connections 2024-01-11T08:25:19.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.337+0000 7fb9265ff700 1 --2- 172.21.15.27:0/2116928462 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb90c066f30 0x7fb90c0693e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.337+0000 7fb9265ff700 1 --2- 172.21.15.27:0/2116928462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9200b29c0 0x7fb9200c9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.337+0000 7fb9265ff700 1 --2- 172.21.15.27:0/2116928462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9200b03e0 0x7fb920149470 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.337+0000 7fb9265ff700 1 --2- 172.21.15.27:0/2116928462 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb9200b07b0 0x7fb9200c9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.337+0000 7fb9265ff700 1 -- 172.21.15.27:0/2116928462 >> 172.21.15.27:0/2116928462 conn(0x7fb92006ed40 msgr2=0x7fb9201592a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:19.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.337+0000 7fb9265ff700 1 -- 172.21.15.27:0/2116928462 shutdown_connections 2024-01-11T08:25:19.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.338+0000 7fb9265ff700 1 -- 172.21.15.27:0/2116928462 wait complete. 2024-01-11T08:25:19.354 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:25:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.554+0000 7f93ae5ff700 1 Processor -- start 2024-01-11T08:25:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.554+0000 7f93ae5ff700 1 -- start start 2024-01-11T08:25:19.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.556+0000 7f93ae5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93a80ca7b0 0x7f93a80cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:19.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.556+0000 7f93ae5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f93a8159e80 0x7f93a815dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:19.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.556+0000 7f93ae5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a815d640 con 0x7f93a80ca3e0 2024-01-11T08:25:19.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.556+0000 7f93ae5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a815ae70 con 0x7f93a8159e80 2024-01-11T08:25:19.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.556+0000 7f93ae5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a80cc4f0 con 0x7f93a80ca7b0 2024-01-11T08:25:19.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.556+0000 7f93a77fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93a80ca7b0 0x7f93a80cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:19.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.556+0000 7f93acb9c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f93a8159e80 0x7f93a815dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:19.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.556+0000 7f93acb9c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f93a8159e80 0x7f93a815dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:58864/0 (socket says 172.21.15.27:58864) 2024-01-11T08:25:19.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.556+0000 7f93a77fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93a80ca7b0 0x7f93a80cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:45836/0 (socket says 172.21.15.27:45836) 2024-01-11T08:25:19.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.556+0000 7f93acb9c700 1 -- 172.21.15.27:0/2872752678 learned_addr learned my addr 172.21.15.27:0/2872752678 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:19.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.556+0000 7f93a7fff700 1 --1- 172.21.15.27:0/2872752678 >> v1:172.21.15.77:6789/0 conn(0x7f93a80ca3e0 0x7f93a80c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:49224/0 (socket says 172.21.15.27:49224) 2024-01-11T08:25:19.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.557+0000 7f93a77fe700 1 -- 172.21.15.27:0/2872752678 >> v2:172.21.15.77:3300/0 conn(0x7f93a8159e80 msgr2=0x7f93a815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:19.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.557+0000 7f93a77fe700 1 --2- 172.21.15.27:0/2872752678 >> v2:172.21.15.77:3300/0 conn(0x7f93a8159e80 0x7f93a815dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.557+0000 7f93a77fe700 1 -- 172.21.15.27:0/2872752678 >> v1:172.21.15.77:6789/0 conn(0x7f93a80ca3e0 legacy=0x7f93a80c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:19.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.557+0000 7f93a77fe700 1 -- 172.21.15.27:0/2872752678 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a815a5d0 con 0x7f93a80ca7b0 2024-01-11T08:25:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.557+0000 7f93a77fe700 1 --2- 172.21.15.27:0/2872752678 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93a80ca7b0 0x7f93a80cca20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f939c0098e0 tx=0x7f939c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3276fbcff940f130 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.557+0000 7f93a6ffd700 1 -- 172.21.15.27:0/2872752678 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c00b640 con 0x7f93a80ca7b0 2024-01-11T08:25:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.557+0000 7f93a6ffd700 1 -- 172.21.15.27:0/2872752678 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939c00b7a0 con 0x7f93a80ca7b0 2024-01-11T08:25:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.557+0000 7f93a6ffd700 1 -- 172.21.15.27:0/2872752678 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c00bb40 con 0x7f93a80ca7b0 2024-01-11T08:25:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.558+0000 7f93ae5ff700 1 -- 172.21.15.27:0/2872752678 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93a80ca7b0 msgr2=0x7f93a80cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.558+0000 7f93ae5ff700 1 --2- 172.21.15.27:0/2872752678 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93a80ca7b0 0x7f93a80cca20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f939c0098e0 tx=0x7f939c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.558+0000 7f93ae5ff700 1 -- 172.21.15.27:0/2872752678 shutdown_connections 2024-01-11T08:25:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.558+0000 7f93ae5ff700 1 --2- 172.21.15.27:0/2872752678 >> v2:172.21.15.77:3300/0 conn(0x7f93a8159e80 0x7f93a815dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.558+0000 7f93ae5ff700 1 --2- 172.21.15.27:0/2872752678 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93a80ca7b0 0x7f93a80cca20 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.558+0000 7f93ae5ff700 1 -- 172.21.15.27:0/2872752678 >> 172.21.15.27:0/2872752678 conn(0x7f93a80ce3e0 msgr2=0x7f93a80ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.558+0000 7f93ae5ff700 1 -- 172.21.15.27:0/2872752678 shutdown_connections 2024-01-11T08:25:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.558+0000 7f93ae5ff700 1 -- 172.21.15.27:0/2872752678 wait complete. 2024-01-11T08:25:19.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.560+0000 7f93ae5ff700 1 Processor -- start 2024-01-11T08:25:19.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.560+0000 7f93ae5ff700 1 -- start start 2024-01-11T08:25:19.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.561+0000 7f93ae5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93a80ca3e0 0x7f93a8148040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:19.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.561+0000 7f93ae5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93a80ca7b0 0x7f93a80b3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:19.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.561+0000 7f93ae5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93a8159e80 0x7f93a80b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:19.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.561+0000 7f93ae5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a8148580 con 0x7f93a80ca3e0 2024-01-11T08:25:19.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.561+0000 7f93ae5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a80b1180 con 0x7f93a80ca7b0 2024-01-11T08:25:19.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.561+0000 7f93ae5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a80b1fa0 con 0x7f93a8159e80 2024-01-11T08:25:19.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.561+0000 7f93a7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93a80ca3e0 0x7f93a8148040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:19.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.561+0000 7f93a7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93a80ca3e0 0x7f93a8148040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47232/0 (socket says 172.21.15.27:47232) 2024-01-11T08:25:19.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.561+0000 7f93a7fff700 1 -- 172.21.15.27:0/1149281714 learned_addr learned my addr 172.21.15.27:0/1149281714 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:19.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.561+0000 7f93acb9c700 1 --2- 172.21.15.27:0/1149281714 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93a8159e80 0x7f93a80b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:19.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.561+0000 7f93a77fe700 1 --2- 172.21.15.27:0/1149281714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93a80ca7b0 0x7f93a80b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:19.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.562+0000 7f93a7fff700 1 -- 172.21.15.27:0/1149281714 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93a8159e80 msgr2=0x7f93a80b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:19.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.562+0000 7f93a7fff700 1 --2- 172.21.15.27:0/1149281714 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93a8159e80 0x7f93a80b3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.562+0000 7f93a7fff700 1 -- 172.21.15.27:0/1149281714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93a80ca7b0 msgr2=0x7f93a80b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:19.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.562+0000 7f93a7fff700 1 --2- 172.21.15.27:0/1149281714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93a80ca7b0 0x7f93a80b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.562+0000 7f93a7fff700 1 -- 172.21.15.27:0/1149281714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939c009420 con 0x7f93a80ca3e0 2024-01-11T08:25:19.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.562+0000 7f93a7fff700 1 --2- 172.21.15.27:0/1149281714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93a80ca3e0 0x7f93a8148040 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f939800c7c0 tx=0x7f9398007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:19.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.562+0000 7f938ffff700 1 -- 172.21.15.27:0/1149281714 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9398005a70 con 0x7f93a80ca3e0 2024-01-11T08:25:19.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.562+0000 7f938ffff700 1 -- 172.21.15.27:0/1149281714 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9398005bd0 con 0x7f93a80ca3e0 2024-01-11T08:25:19.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.562+0000 7f93ae5ff700 1 -- 172.21.15.27:0/1149281714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a80b1390 con 0x7f93a80ca3e0 2024-01-11T08:25:19.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.562+0000 7f938ffff700 1 -- 172.21.15.27:0/1149281714 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93980070d0 con 0x7f93a80ca3e0 2024-01-11T08:25:19.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.562+0000 7f93ae5ff700 1 -- 172.21.15.27:0/1149281714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a80b1950 con 0x7f93a80ca3e0 2024-01-11T08:25:19.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.564+0000 7f93ae5ff700 1 -- 172.21.15.27:0/1149281714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a80718d0 con 0x7f93a80ca3e0 2024-01-11T08:25:19.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.564+0000 7f938ffff700 1 -- 172.21.15.27:0/1149281714 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9398007440 con 0x7f93a80ca3e0 2024-01-11T08:25:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.565+0000 7f938ffff700 1 --2- 172.21.15.27:0/1149281714 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9394066f30 0x7f93940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.565+0000 7f938ffff700 1 -- 172.21.15.27:0/1149281714 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 4881+0+0 (secure 0 0 0) 0x7f9398083610 con 0x7f93a80ca3e0 2024-01-11T08:25:19.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.566+0000 7f93a77fe700 1 --2- 172.21.15.27:0/1149281714 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9394066f30 0x7f93940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:19.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.566+0000 7f93a77fe700 1 --2- 172.21.15.27:0/1149281714 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9394066f30 0x7f93940693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f939c0091e0 tx=0x7f939c017000 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:19.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.569+0000 7f938ffff700 1 -- 172.21.15.27:0/1149281714 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9398050bb0 con 0x7f93a80ca3e0 2024-01-11T08:25:19.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.776+0000 7f93ae5ff700 1 -- 172.21.15.27:0/1149281714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93a815a8b0 con 0x7f93a80ca3e0 2024-01-11T08:25:19.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.777+0000 7f938ffff700 1 -- 172.21.15.27:0/1149281714 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+12129 (secure 0 0 0) 0x7f9398017090 con 0x7f93a80ca3e0 2024-01-11T08:25:19.776 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:19.776 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":71,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:25:18.819242+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:25:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.779+0000 7f93ae5ff700 1 -- 172.21.15.27:0/1149281714 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9394066f30 msgr2=0x7f93940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.779+0000 7f93ae5ff700 1 --2- 172.21.15.27:0/1149281714 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9394066f30 0x7f93940693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f939c0091e0 tx=0x7f939c017000 comp rx=0 tx=0).stop 2024-01-11T08:25:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.779+0000 7f93ae5ff700 1 -- 172.21.15.27:0/1149281714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93a80ca3e0 msgr2=0x7f93a8148040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.779+0000 7f93ae5ff700 1 --2- 172.21.15.27:0/1149281714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93a80ca3e0 0x7f93a8148040 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f939800c7c0 tx=0x7f9398007f70 comp rx=0 tx=0).stop 2024-01-11T08:25:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.779+0000 7f93ae5ff700 1 -- 172.21.15.27:0/1149281714 shutdown_connections 2024-01-11T08:25:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.779+0000 7f93ae5ff700 1 --2- 172.21.15.27:0/1149281714 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9394066f30 0x7f93940693e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.779+0000 7f93ae5ff700 1 --2- 172.21.15.27:0/1149281714 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93a8159e80 0x7f93a80b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.779+0000 7f93ae5ff700 1 --2- 172.21.15.27:0/1149281714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93a80ca3e0 0x7f93a8148040 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.779+0000 7f93ae5ff700 1 --2- 172.21.15.27:0/1149281714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93a80ca7b0 0x7f93a80b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.779+0000 7f93ae5ff700 1 -- 172.21.15.27:0/1149281714 >> 172.21.15.27:0/1149281714 conn(0x7f93a80ce3e0 msgr2=0x7f93a80693f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.780+0000 7f93ae5ff700 1 -- 172.21.15.27:0/1149281714 shutdown_connections 2024-01-11T08:25:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:19.780+0000 7f93ae5ff700 1 -- 172.21.15.27:0/1149281714 wait complete. 2024-01-11T08:25:19.796 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-11T08:25:19.797 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:25:20.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.001+0000 7fb2d7dff700 1 Processor -- start 2024-01-11T08:25:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.001+0000 7fb2d7dff700 1 -- start start 2024-01-11T08:25:20.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.003+0000 7fb2d7dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb2d00ccd70 0x7fb2d00cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.003+0000 7fb2d7dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb2d015ae70 0x7fb2d015b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.003+0000 7fb2d7dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d00cf850 con 0x7fb2d015a1b0 2024-01-11T08:25:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.003+0000 7fb2d7dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d015ac90 con 0x7fb2d00ccd70 2024-01-11T08:25:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.003+0000 7fb2d7dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d00cec40 con 0x7fb2d015ae70 2024-01-11T08:25:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.003+0000 7fb2d5b9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb2d015a1b0 0x7fb2d015a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39020/0 (socket says 172.21.15.27:39020) 2024-01-11T08:25:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.003+0000 7fb2d5b9b700 1 -- 172.21.15.27:0/839383395 learned_addr learned my addr 172.21.15.27:0/839383395 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.003+0000 7fb2d539a700 1 --2- 172.21.15.27:0/839383395 >> v2:172.21.15.27:3300/0 conn(0x7fb2d00ccd70 0x7fb2d00cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.003+0000 7fb2d639c700 1 --2- 172.21.15.27:0/839383395 >> v2:172.21.15.77:3300/0 conn(0x7fb2d015ae70 0x7fb2d015b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.003+0000 7fb2d4b99700 1 -- 172.21.15.27:0/839383395 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013097035 0 0) 0x7fb2d00cf850 con 0x7fb2d015a1b0 2024-01-11T08:25:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.004+0000 7fb2d4b99700 1 -- 172.21.15.27:0/839383395 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2bc003570 con 0x7fb2d015a1b0 2024-01-11T08:25:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.004+0000 7fb2d539a700 1 -- 172.21.15.27:0/839383395 >> v2:172.21.15.77:3300/0 conn(0x7fb2d015ae70 msgr2=0x7fb2d015b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.004+0000 7fb2d539a700 1 --2- 172.21.15.27:0/839383395 >> v2:172.21.15.77:3300/0 conn(0x7fb2d015ae70 0x7fb2d015b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.004+0000 7fb2d539a700 1 -- 172.21.15.27:0/839383395 >> v1:172.21.15.27:6789/0 conn(0x7fb2d015a1b0 legacy=0x7fb2d015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:20.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.004+0000 7fb2d539a700 1 -- 172.21.15.27:0/839383395 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d00cef20 con 0x7fb2d00ccd70 2024-01-11T08:25:20.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.004+0000 7fb2d539a700 1 --2- 172.21.15.27:0/839383395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2d00ccd70 0x7fb2d00cd210 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0089810 tx=0x7fb2c00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b35e1aed047393 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:20.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.004+0000 7fb2d4b99700 1 -- 172.21.15.27:0/839383395 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0011630 con 0x7fb2d00ccd70 2024-01-11T08:25:20.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.004+0000 7fb2d4b99700 1 -- 172.21.15.27:0/839383395 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c00117b0 con 0x7fb2d00ccd70 2024-01-11T08:25:20.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.004+0000 7fb2d4b99700 1 -- 172.21.15.27:0/839383395 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0011b70 con 0x7fb2d00ccd70 2024-01-11T08:25:20.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.005+0000 7fb2d7dff700 1 -- 172.21.15.27:0/839383395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2d00ccd70 msgr2=0x7fb2d00cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:20.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.005+0000 7fb2d7dff700 1 --2- 172.21.15.27:0/839383395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2d00ccd70 0x7fb2d00cd210 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0089810 tx=0x7fb2c00091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.005+0000 7fb2d7dff700 1 -- 172.21.15.27:0/839383395 shutdown_connections 2024-01-11T08:25:20.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.005+0000 7fb2d7dff700 1 --2- 172.21.15.27:0/839383395 >> v2:172.21.15.77:3300/0 conn(0x7fb2d015ae70 0x7fb2d015b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.005+0000 7fb2d7dff700 1 --2- 172.21.15.27:0/839383395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2d00ccd70 0x7fb2d00cd210 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.005+0000 7fb2d7dff700 1 -- 172.21.15.27:0/839383395 >> 172.21.15.27:0/839383395 conn(0x7fb2d006ed40 msgr2=0x7fb2d006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:20.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.005+0000 7fb2d7dff700 1 -- 172.21.15.27:0/839383395 shutdown_connections 2024-01-11T08:25:20.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.005+0000 7fb2d7dff700 1 -- 172.21.15.27:0/839383395 wait complete. 2024-01-11T08:25:20.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.007+0000 7fb2d7dff700 1 Processor -- start 2024-01-11T08:25:20.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.007+0000 7fb2d7dff700 1 -- start start 2024-01-11T08:25:20.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.009+0000 7fb2d7dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2d00ccd70 0x7fb2d00b5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.009+0000 7fb2d7dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb2d015a1b0 0x7fb2d00c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.009+0000 7fb2d7dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2d015ae70 0x7fb2d00c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.009+0000 7fb2d7dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d00b63b0 con 0x7fb2d015ae70 2024-01-11T08:25:20.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.009+0000 7fb2d7dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d0169fc0 con 0x7fb2d00ccd70 2024-01-11T08:25:20.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.009+0000 7fb2d7dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d016a2a0 con 0x7fb2d015a1b0 2024-01-11T08:25:20.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.009+0000 7fb2d639c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2d015ae70 0x7fb2d00c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.009+0000 7fb2d639c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2d015ae70 0x7fb2d00c7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47246/0 (socket says 172.21.15.27:47246) 2024-01-11T08:25:20.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.009+0000 7fb2d639c700 1 -- 172.21.15.27:0/2866672897 learned_addr learned my addr 172.21.15.27:0/2866672897 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:20.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.009+0000 7fb2d5b9b700 1 --2- 172.21.15.27:0/2866672897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2d00ccd70 0x7fb2d00b5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.009+0000 7fb2d539a700 1 --2- 172.21.15.27:0/2866672897 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb2d015a1b0 0x7fb2d00c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.009+0000 7fb2d639c700 1 -- 172.21.15.27:0/2866672897 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb2d015a1b0 msgr2=0x7fb2d00c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:20.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.009+0000 7fb2d639c700 1 --2- 172.21.15.27:0/2866672897 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb2d015a1b0 0x7fb2d00c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.009+0000 7fb2d639c700 1 -- 172.21.15.27:0/2866672897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2d00ccd70 msgr2=0x7fb2d00b5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:20.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.009+0000 7fb2d639c700 1 --2- 172.21.15.27:0/2866672897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2d00ccd70 0x7fb2d00b5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.009+0000 7fb2d639c700 1 -- 172.21.15.27:0/2866672897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c0009420 con 0x7fb2d015ae70 2024-01-11T08:25:20.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.010+0000 7fb2d639c700 1 --2- 172.21.15.27:0/2866672897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2d015ae70 0x7fb2d00c7690 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009e90 tx=0x7fb2c800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:20.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.010+0000 7fb2c5ffb700 1 -- 172.21.15.27:0/2866672897 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c800cc20 con 0x7fb2d015ae70 2024-01-11T08:25:20.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.010+0000 7fb2c5ffb700 1 -- 172.21.15.27:0/2866672897 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c800f040 con 0x7fb2d015ae70 2024-01-11T08:25:20.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.010+0000 7fb2d7dff700 1 -- 172.21.15.27:0/2866672897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d016a580 con 0x7fb2d015ae70 2024-01-11T08:25:20.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.010+0000 7fb2c5ffb700 1 -- 172.21.15.27:0/2866672897 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8014480 con 0x7fb2d015ae70 2024-01-11T08:25:20.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.010+0000 7fb2d7dff700 1 -- 172.21.15.27:0/2866672897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d016a860 con 0x7fb2d015ae70 2024-01-11T08:25:20.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.011+0000 7fb2d7dff700 1 -- 172.21.15.27:0/2866672897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d016d430 con 0x7fb2d015ae70 2024-01-11T08:25:20.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.012+0000 7fb2c5ffb700 1 -- 172.21.15.27:0/2866672897 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb2c8014830 con 0x7fb2d015ae70 2024-01-11T08:25:20.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.013+0000 7fb2c5ffb700 1 --2- 172.21.15.27:0/2866672897 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb2b4066f30 0x7fb2b40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.013+0000 7fb2c5ffb700 1 -- 172.21.15.27:0/2866672897 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 4881+0+0 (secure 0 0 0) 0x7fb2c8084cc0 con 0x7fb2d015ae70 2024-01-11T08:25:20.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.013+0000 7fb2d5b9b700 1 --2- 172.21.15.27:0/2866672897 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb2b4066f30 0x7fb2b40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.014+0000 7fb2d5b9b700 1 --2- 172.21.15.27:0/2866672897 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb2b4066f30 0x7fb2b40693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc0058d0 tx=0x7fb2cc009040 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:20.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.017+0000 7fb2c5ffb700 1 -- 172.21.15.27:0/2866672897 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb2c808b080 con 0x7fb2d015ae70 2024-01-11T08:25:20.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.220+0000 7fb2d7dff700 1 -- 172.21.15.27:0/2866672897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb2d004c3c0 con 0x7fb2d015ae70 2024-01-11T08:25:20.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.220+0000 7fb2c5ffb700 1 -- 172.21.15.27:0/2866672897 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb2c8018020 con 0x7fb2d015ae70 2024-01-11T08:25:20.219 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:25:20.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.222+0000 7fb2d7dff700 1 -- 172.21.15.27:0/2866672897 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb2b4066f30 msgr2=0x7fb2b40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:20.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.222+0000 7fb2d7dff700 1 --2- 172.21.15.27:0/2866672897 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb2b4066f30 0x7fb2b40693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc0058d0 tx=0x7fb2cc009040 comp rx=0 tx=0).stop 2024-01-11T08:25:20.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.222+0000 7fb2d7dff700 1 -- 172.21.15.27:0/2866672897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2d015ae70 msgr2=0x7fb2d00c7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:20.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.222+0000 7fb2d7dff700 1 --2- 172.21.15.27:0/2866672897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2d015ae70 0x7fb2d00c7690 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009e90 tx=0x7fb2c800b040 comp rx=0 tx=0).stop 2024-01-11T08:25:20.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.222+0000 7fb2d7dff700 1 -- 172.21.15.27:0/2866672897 shutdown_connections 2024-01-11T08:25:20.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.222+0000 7fb2d7dff700 1 --2- 172.21.15.27:0/2866672897 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb2b4066f30 0x7fb2b40693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.222+0000 7fb2d7dff700 1 --2- 172.21.15.27:0/2866672897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2d015ae70 0x7fb2d00c7690 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.222+0000 7fb2d7dff700 1 --2- 172.21.15.27:0/2866672897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2d00ccd70 0x7fb2d00b5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.222+0000 7fb2d7dff700 1 --2- 172.21.15.27:0/2866672897 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb2d015a1b0 0x7fb2d00c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.222+0000 7fb2d7dff700 1 -- 172.21.15.27:0/2866672897 >> 172.21.15.27:0/2866672897 conn(0x7fb2d006ed40 msgr2=0x7fb2d0158b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:20.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.222+0000 7fb2d7dff700 1 -- 172.21.15.27:0/2866672897 shutdown_connections 2024-01-11T08:25:20.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.223+0000 7fb2d7dff700 1 -- 172.21.15.27:0/2866672897 wait complete. 2024-01-11T08:25:20.239 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-11T08:25:20.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.440+0000 7f6d9adff700 1 Processor -- start 2024-01-11T08:25:20.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.440+0000 7f6d9adff700 1 -- start start 2024-01-11T08:25:20.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.442+0000 7f6d9adff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6d940cca20 0x7f6d94159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.442+0000 7f6d9adff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9415d640 con 0x7f6d940ca7b0 2024-01-11T08:25:20.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.442+0000 7f6d9adff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d940cce60 con 0x7f6d940ca3e0 2024-01-11T08:25:20.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.442+0000 7f6d9adff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d9415ae70 con 0x7f6d940cca20 2024-01-11T08:25:20.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.442+0000 7f6d93fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6d940ca7b0 0x7f6d9415dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39034/0 (socket says 172.21.15.27:39034) 2024-01-11T08:25:20.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.442+0000 7f6d93fff700 1 -- 172.21.15.27:0/259894430 learned_addr learned my addr 172.21.15.27:0/259894430 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:20.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.442+0000 7f6d9939c700 1 --2- 172.21.15.27:0/259894430 >> v2:172.21.15.27:3300/0 conn(0x7f6d940cca20 0x7f6d94159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.442+0000 7f6d937fe700 1 -- 172.21.15.27:0/259894430 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966526628 0 0) 0x7f6d9415d640 con 0x7f6d940ca7b0 2024-01-11T08:25:20.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.442+0000 7f6d937fe700 1 -- 172.21.15.27:0/259894430 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d84003570 con 0x7f6d940ca7b0 2024-01-11T08:25:20.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.442+0000 7f6d9939c700 1 -- 172.21.15.27:0/259894430 >> v1:172.21.15.77:6789/0 conn(0x7f6d940ca3e0 legacy=0x7f6d940c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:20.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.442+0000 7f6d9939c700 1 -- 172.21.15.27:0/259894430 >> v1:172.21.15.27:6789/0 conn(0x7f6d940ca7b0 legacy=0x7f6d9415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:20.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.442+0000 7f6d9939c700 1 -- 172.21.15.27:0/259894430 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9415a5d0 con 0x7f6d940cca20 2024-01-11T08:25:20.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.443+0000 7f6d9939c700 1 --2- 172.21.15.27:0/259894430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d940cca20 0x7f6d94159e80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009e90 tx=0x7f6d8c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2428b73b49bc1a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:20.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.443+0000 7f6d937fe700 1 -- 172.21.15.27:0/259894430 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c00e070 con 0x7f6d940cca20 2024-01-11T08:25:20.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.443+0000 7f6d937fe700 1 -- 172.21.15.27:0/259894430 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d8c00bc40 con 0x7f6d940cca20 2024-01-11T08:25:20.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.443+0000 7f6d937fe700 1 -- 172.21.15.27:0/259894430 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c012430 con 0x7f6d940cca20 2024-01-11T08:25:20.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.443+0000 7f6d9adff700 1 -- 172.21.15.27:0/259894430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d940cca20 msgr2=0x7f6d94159e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:20.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.443+0000 7f6d9adff700 1 --2- 172.21.15.27:0/259894430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d940cca20 0x7f6d94159e80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009e90 tx=0x7f6d8c009780 comp rx=0 tx=0).stop 2024-01-11T08:25:20.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.444+0000 7f6d9adff700 1 -- 172.21.15.27:0/259894430 shutdown_connections 2024-01-11T08:25:20.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.444+0000 7f6d9adff700 1 --2- 172.21.15.27:0/259894430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d940cca20 0x7f6d94159e80 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.444+0000 7f6d9adff700 1 -- 172.21.15.27:0/259894430 >> 172.21.15.27:0/259894430 conn(0x7f6d940ce3e0 msgr2=0x7f6d940ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:20.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.444+0000 7f6d9adff700 1 -- 172.21.15.27:0/259894430 shutdown_connections 2024-01-11T08:25:20.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.444+0000 7f6d9adff700 1 -- 172.21.15.27:0/259894430 wait complete. 2024-01-11T08:25:20.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.445+0000 7f6d9adff700 1 Processor -- start 2024-01-11T08:25:20.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.445+0000 7f6d9adff700 1 -- start start 2024-01-11T08:25:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.446+0000 7f6d9adff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d940ca3e0 0x7f6d9415dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.446+0000 7f6d9adff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d940ca7b0 0x7f6d94150a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.446+0000 7f6d9adff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d940cca20 0x7f6d94150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.446+0000 7f6d9adff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9414c920 con 0x7f6d940cca20 2024-01-11T08:25:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.446+0000 7f6d9adff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9414c610 con 0x7f6d940ca3e0 2024-01-11T08:25:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.446+0000 7f6d9adff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9415e440 con 0x7f6d940ca7b0 2024-01-11T08:25:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.446+0000 7f6d9939c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d940cca20 0x7f6d94150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.447+0000 7f6d9939c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d940cca20 0x7f6d94150f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47272/0 (socket says 172.21.15.27:47272) 2024-01-11T08:25:20.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.447+0000 7f6d9939c700 1 -- 172.21.15.27:0/3358527651 learned_addr learned my addr 172.21.15.27:0/3358527651 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:20.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.447+0000 7f6d93fff700 1 --2- 172.21.15.27:0/3358527651 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d940ca7b0 0x7f6d94150a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.447+0000 7f6d98b9b700 1 --2- 172.21.15.27:0/3358527651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d940ca3e0 0x7f6d9415dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.447+0000 7f6d9939c700 1 -- 172.21.15.27:0/3358527651 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d940ca7b0 msgr2=0x7f6d94150a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:20.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.447+0000 7f6d9939c700 1 --2- 172.21.15.27:0/3358527651 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d940ca7b0 0x7f6d94150a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.447+0000 7f6d9939c700 1 -- 172.21.15.27:0/3358527651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d940ca3e0 msgr2=0x7f6d9415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:20.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.447+0000 7f6d9939c700 1 --2- 172.21.15.27:0/3358527651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d940ca3e0 0x7f6d9415dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.447+0000 7f6d9939c700 1 -- 172.21.15.27:0/3358527651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c009a20 con 0x7f6d940cca20 2024-01-11T08:25:20.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.447+0000 7f6d9939c700 1 --2- 172.21.15.27:0/3358527651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d940cca20 0x7f6d94150f90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f6d94064980 tx=0x7f6d8c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:20.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.447+0000 7f6d90ff9700 1 -- 172.21.15.27:0/3358527651 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c012e80 con 0x7f6d940cca20 2024-01-11T08:25:20.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.447+0000 7f6d90ff9700 1 -- 172.21.15.27:0/3358527651 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d8c01b630 con 0x7f6d940cca20 2024-01-11T08:25:20.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.448+0000 7f6d9adff700 1 -- 172.21.15.27:0/3358527651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9415e720 con 0x7f6d940cca20 2024-01-11T08:25:20.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.448+0000 7f6d90ff9700 1 -- 172.21.15.27:0/3358527651 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c01b9d0 con 0x7f6d940cca20 2024-01-11T08:25:20.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.448+0000 7f6d9adff700 1 -- 172.21.15.27:0/3358527651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9415e010 con 0x7f6d940cca20 2024-01-11T08:25:20.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.449+0000 7f6d9adff700 1 -- 172.21.15.27:0/3358527651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d9415e200 con 0x7f6d940cca20 2024-01-11T08:25:20.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.449+0000 7f6d90ff9700 1 -- 172.21.15.27:0/3358527651 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6d8c01c400 con 0x7f6d940cca20 2024-01-11T08:25:20.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.450+0000 7f6d90ff9700 1 --2- 172.21.15.27:0/3358527651 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6d78066f30 0x7f6d780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.451+0000 7f6d90ff9700 1 -- 172.21.15.27:0/3358527651 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 4881+0+0 (secure 0 0 0) 0x7f6d8c086000 con 0x7f6d940cca20 2024-01-11T08:25:20.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.451+0000 7f6d98b9b700 1 --2- 172.21.15.27:0/3358527651 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6d78066f30 0x7f6d780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.452+0000 7f6d98b9b700 1 --2- 172.21.15.27:0/3358527651 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6d78066f30 0x7f6d780693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6d88003860 tx=0x7f6d88009040 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:20.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.455+0000 7f6d90ff9700 1 -- 172.21.15.27:0/3358527651 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6d9415e200 con 0x7f6d940cca20 2024-01-11T08:25:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.661+0000 7f6d9adff700 1 -- 172.21.15.27:0/3358527651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6d9415e200 con 0x7f6d940cca20 2024-01-11T08:25:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.661+0000 7f6d90ff9700 1 -- 172.21.15.27:0/3358527651 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v71) v1 ==== 84+0+65 (secure 0 0 0) 0x7f6d8c02a060 con 0x7f6d940cca20 2024-01-11T08:25:20.660 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:20.660 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T08:25:20.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:20.661 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-11T08:25:20.661 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-11T08:25:20.661 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:20.661 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T08:25:20.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.663+0000 7f6d9adff700 1 -- 172.21.15.27:0/3358527651 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6d78066f30 msgr2=0x7f6d780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:20.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.663+0000 7f6d9adff700 1 --2- 172.21.15.27:0/3358527651 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6d78066f30 0x7f6d780693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6d88003860 tx=0x7f6d88009040 comp rx=0 tx=0).stop 2024-01-11T08:25:20.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.663+0000 7f6d9adff700 1 -- 172.21.15.27:0/3358527651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d940cca20 msgr2=0x7f6d94150f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:20.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.663+0000 7f6d9adff700 1 --2- 172.21.15.27:0/3358527651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d940cca20 0x7f6d94150f90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f6d94064980 tx=0x7f6d8c019000 comp rx=0 tx=0).stop 2024-01-11T08:25:20.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.664+0000 7f6d9adff700 1 -- 172.21.15.27:0/3358527651 shutdown_connections 2024-01-11T08:25:20.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.664+0000 7f6d9adff700 1 --2- 172.21.15.27:0/3358527651 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6d78066f30 0x7f6d780693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.664+0000 7f6d9adff700 1 --2- 172.21.15.27:0/3358527651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d940cca20 0x7f6d94150f90 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.664+0000 7f6d9adff700 1 --2- 172.21.15.27:0/3358527651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d940ca3e0 0x7f6d9415dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.664+0000 7f6d9adff700 1 --2- 172.21.15.27:0/3358527651 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d940ca7b0 0x7f6d94150a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.664+0000 7f6d9adff700 1 -- 172.21.15.27:0/3358527651 >> 172.21.15.27:0/3358527651 conn(0x7f6d940ce3e0 msgr2=0x7f6d9406ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:20.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.664+0000 7f6d9adff700 1 -- 172.21.15.27:0/3358527651 shutdown_connections 2024-01-11T08:25:20.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.664+0000 7f6d9adff700 1 -- 172.21.15.27:0/3358527651 wait complete. 2024-01-11T08:25:20.680 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-11T08:25:20.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-11T08:25:20.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.886+0000 7f0d45bff700 1 Processor -- start 2024-01-11T08:25:20.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.887+0000 7f0d45bff700 1 -- start start 2024-01-11T08:25:20.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.888+0000 7f0d45bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0d4015a7a0 0x7f0d4015c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.888+0000 7f0d45bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0d400685e0 0x7f0d40068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.888+0000 7f0d45bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d40068f80 con 0x7f0d40159e60 2024-01-11T08:25:20.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.888+0000 7f0d45bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d4015c4b0 con 0x7f0d4015a7a0 2024-01-11T08:25:20.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.888+0000 7f0d45bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4015b5d0 con 0x7f0d400685e0 2024-01-11T08:25:20.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.888+0000 7f0d3effd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0d40159e60 0x7f0d4015aec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39042/0 (socket says 172.21.15.27:39042) 2024-01-11T08:25:20.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.888+0000 7f0d3effd700 1 -- 172.21.15.27:0/3357059967 learned_addr learned my addr 172.21.15.27:0/3357059967 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:20.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.888+0000 7f0d3f7fe700 1 --2- 172.21.15.27:0/3357059967 >> v2:172.21.15.77:3300/0 conn(0x7f0d4015a7a0 0x7f0d4015c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.888+0000 7f0d3ffff700 1 --2- 172.21.15.27:0/3357059967 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0d400685e0 0x7f0d40068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.889+0000 7f0d3e7fc700 1 -- 172.21.15.27:0/3357059967 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554637807 0 0) 0x7f0d34001110 con 0x7f0d40159e60 2024-01-11T08:25:20.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.889+0000 7f0d3e7fc700 1 -- 172.21.15.27:0/3357059967 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d28003570 con 0x7f0d40159e60 2024-01-11T08:25:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.889+0000 7f0d3e7fc700 1 -- 172.21.15.27:0/3357059967 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1467247906 0 0) 0x7f0d28003570 con 0x7f0d40159e60 2024-01-11T08:25:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.889+0000 7f0d3e7fc700 1 -- 172.21.15.27:0/3357059967 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0d400685e0 msgr2=0x7f0d40068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.889+0000 7f0d3e7fc700 1 --2- 172.21.15.27:0/3357059967 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0d400685e0 0x7f0d40068a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.889+0000 7f0d3e7fc700 1 -- 172.21.15.27:0/3357059967 >> v2:172.21.15.77:3300/0 conn(0x7f0d4015a7a0 msgr2=0x7f0d4015c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.889+0000 7f0d3e7fc700 1 --2- 172.21.15.27:0/3357059967 >> v2:172.21.15.77:3300/0 conn(0x7f0d4015a7a0 0x7f0d4015c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.889+0000 7f0d3e7fc700 1 -- 172.21.15.27:0/3357059967 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4015b8a0 con 0x7f0d40159e60 2024-01-11T08:25:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.889+0000 7f0d3e7fc700 1 -- 172.21.15.27:0/3357059967 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f0d34001600 con 0x7f0d40159e60 2024-01-11T08:25:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.889+0000 7f0d3e7fc700 1 -- 172.21.15.27:0/3357059967 >> v1:172.21.15.27:6789/0 conn(0x7f0d40159e60 legacy=0x7f0d4015aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.889+0000 7f0d3e7fc700 1 --2- 172.21.15.27:0/3357059967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d28003a60 0x7f0d28003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.889+0000 7f0d3e7fc700 1 --2- 172.21.15.27:0/3357059967 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0d28004810 0x7f0d28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.889+0000 7f0d3e7fc700 1 --2- 172.21.15.27:0/3357059967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d280051c0 0x7f0d2800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.889+0000 7f0d3e7fc700 1 -- 172.21.15.27:0/3357059967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4015c4b0 con 0x7f0d28003a60 2024-01-11T08:25:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.889+0000 7f0d3e7fc700 1 -- 172.21.15.27:0/3357059967 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4015b5d0 con 0x7f0d280051c0 2024-01-11T08:25:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.889+0000 7f0d3e7fc700 1 -- 172.21.15.27:0/3357059967 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2800bfa0 con 0x7f0d28004810 2024-01-11T08:25:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.889+0000 7f0d3ffff700 1 --2- 172.21.15.27:0/3357059967 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0d400685e0 0x7f0d40068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:20.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.889+0000 7f0d3f7fe700 1 --2- 172.21.15.27:0/3357059967 >> v2:172.21.15.77:3300/0 conn(0x7f0d4015a7a0 0x7f0d4015c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:20.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.890+0000 7f0d3f7fe700 1 --2- 172.21.15.27:0/3357059967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d28003a60 0x7f0d28003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.890+0000 7f0d3effd700 1 --2- 172.21.15.27:0/3357059967 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0d28004810 0x7f0d28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.890+0000 7f0d3ffff700 1 --2- 172.21.15.27:0/3357059967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d280051c0 0x7f0d2800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.890+0000 7f0d3f7fe700 1 -- 172.21.15.27:0/3357059967 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0d28004810 msgr2=0x7f0d28004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:20.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.890+0000 7f0d3f7fe700 1 --2- 172.21.15.27:0/3357059967 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0d28004810 0x7f0d28004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.890+0000 7f0d3f7fe700 1 -- 172.21.15.27:0/3357059967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d280051c0 msgr2=0x7f0d2800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:20.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.890+0000 7f0d3f7fe700 1 --2- 172.21.15.27:0/3357059967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d280051c0 0x7f0d2800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.890+0000 7f0d3f7fe700 1 -- 172.21.15.27:0/3357059967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d28003570 con 0x7f0d28003a60 2024-01-11T08:25:20.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.890+0000 7f0d3ffff700 1 --2- 172.21.15.27:0/3357059967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d280051c0 0x7f0d2800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:20.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.890+0000 7f0d3f7fe700 1 --2- 172.21.15.27:0/3357059967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d28003a60 0x7f0d28003e70 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0d30009db0 tx=0x7f0d300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d30487005089cbe server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:20.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.891+0000 7f0d3e7fc700 1 -- 172.21.15.27:0/3357059967 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d30009740 con 0x7f0d28003a60 2024-01-11T08:25:20.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.891+0000 7f0d3e7fc700 1 -- 172.21.15.27:0/3357059967 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3000e920 con 0x7f0d28003a60 2024-01-11T08:25:20.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.891+0000 7f0d3e7fc700 1 -- 172.21.15.27:0/3357059967 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3000eca0 con 0x7f0d28003a60 2024-01-11T08:25:20.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.891+0000 7f0d45bff700 1 -- 172.21.15.27:0/3357059967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d28003a60 msgr2=0x7f0d28003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:20.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.891+0000 7f0d45bff700 1 --2- 172.21.15.27:0/3357059967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d28003a60 0x7f0d28003e70 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0d30009db0 tx=0x7f0d300091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.891+0000 7f0d3ffff700 1 -- 172.21.15.27:0/3357059967 reap_dead start 2024-01-11T08:25:20.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.891+0000 7f0d45bff700 1 -- 172.21.15.27:0/3357059967 shutdown_connections 2024-01-11T08:25:20.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.891+0000 7f0d45bff700 1 -- 172.21.15.27:0/3357059967 >> 172.21.15.27:0/3357059967 conn(0x7f0d4006ed40 msgr2=0x7f0d4006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:20.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.891+0000 7f0d45bff700 1 -- 172.21.15.27:0/3357059967 shutdown_connections 2024-01-11T08:25:20.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.891+0000 7f0d45bff700 1 -- 172.21.15.27:0/3357059967 wait complete. 2024-01-11T08:25:20.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.893+0000 7f0d45bff700 1 Processor -- start 2024-01-11T08:25:20.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.893+0000 7f0d45bff700 1 -- start start 2024-01-11T08:25:20.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.894+0000 7f0d45bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0d28003a60 0x7f0d4014dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.894+0000 7f0d45bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d40150df0 0x7f0d40151260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.894+0000 7f0d45bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d400b2210 0x7f0d400b2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.894+0000 7f0d45bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4014e160 con 0x7f0d400b2210 2024-01-11T08:25:20.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.894+0000 7f0d45bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d400b2bc0 con 0x7f0d40150df0 2024-01-11T08:25:20.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.894+0000 7f0d45bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d40151e00 con 0x7f0d28003a60 2024-01-11T08:25:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.894+0000 7f0d3ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d400b2210 0x7f0d400b2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.894+0000 7f0d3ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d400b2210 0x7f0d400b2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47302/0 (socket says 172.21.15.27:47302) 2024-01-11T08:25:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.894+0000 7f0d3ffff700 1 -- 172.21.15.27:0/4183813621 learned_addr learned my addr 172.21.15.27:0/4183813621 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.894+0000 7f0d3f7fe700 1 --2- 172.21.15.27:0/4183813621 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0d28003a60 0x7f0d4014dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.895+0000 7f0d3effd700 1 --2- 172.21.15.27:0/4183813621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d40150df0 0x7f0d40151260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.895+0000 7f0d3ffff700 1 -- 172.21.15.27:0/4183813621 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0d28003a60 msgr2=0x7f0d4014dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.895+0000 7f0d3ffff700 1 --2- 172.21.15.27:0/4183813621 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0d28003a60 0x7f0d4014dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.895+0000 7f0d3ffff700 1 -- 172.21.15.27:0/4183813621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d40150df0 msgr2=0x7f0d40151260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.895+0000 7f0d3ffff700 1 --2- 172.21.15.27:0/4183813621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d40150df0 0x7f0d40151260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.895+0000 7f0d3ffff700 1 -- 172.21.15.27:0/4183813621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d400b02b0 con 0x7f0d400b2210 2024-01-11T08:25:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.895+0000 7f0d3ffff700 1 --2- 172.21.15.27:0/4183813621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d400b2210 0x7f0d400b2680 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f0d400899d0 tx=0x7f0d3800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.895+0000 7f0d277fe700 1 -- 172.21.15.27:0/4183813621 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3800e070 con 0x7f0d400b2210 2024-01-11T08:25:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.895+0000 7f0d277fe700 1 -- 172.21.15.27:0/4183813621 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d38011c50 con 0x7f0d400b2210 2024-01-11T08:25:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.895+0000 7f0d277fe700 1 -- 172.21.15.27:0/4183813621 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d38016470 con 0x7f0d400b2210 2024-01-11T08:25:20.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.895+0000 7f0d45bff700 1 -- 172.21.15.27:0/4183813621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d4016a870 con 0x7f0d400b2210 2024-01-11T08:25:20.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.896+0000 7f0d45bff700 1 -- 172.21.15.27:0/4183813621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d40151b20 con 0x7f0d400b2210 2024-01-11T08:25:20.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.897+0000 7f0d45bff700 1 -- 172.21.15.27:0/4183813621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d400b0440 con 0x7f0d400b2210 2024-01-11T08:25:20.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.897+0000 7f0d277fe700 1 -- 172.21.15.27:0/4183813621 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0d380168a0 con 0x7f0d400b2210 2024-01-11T08:25:20.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.898+0000 7f0d277fe700 1 --2- 172.21.15.27:0/4183813621 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0d2c066f30 0x7f0d2c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:20.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.898+0000 7f0d277fe700 1 -- 172.21.15.27:0/4183813621 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 4881+0+0 (secure 0 0 0) 0x7f0d38085060 con 0x7f0d400b2210 2024-01-11T08:25:20.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.899+0000 7f0d3f7fe700 1 --2- 172.21.15.27:0/4183813621 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0d2c066f30 0x7f0d2c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:20.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.900+0000 7f0d3f7fe700 1 --2- 172.21.15.27:0/4183813621 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0d2c066f30 0x7f0d2c0693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0d300091e0 tx=0x7f0d30019000 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:20.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:20.902+0000 7f0d277fe700 1 -- 172.21.15.27:0/4183813621 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0d38052560 con 0x7f0d400b2210 2024-01-11T08:25:21.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:21.107+0000 7f0d45bff700 1 -- 172.21.15.27:0/4183813621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0d4004c3c0 con 0x7f0d400b2210 2024-01-11T08:25:21.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:21.830+0000 7f0d277fe700 1 -- 172.21.15.27:0/4183813621 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v72) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0d38055b80 con 0x7f0d400b2210 2024-01-11T08:25:21.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:21.835+0000 7f0d45bff700 1 -- 172.21.15.27:0/4183813621 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0d2c066f30 msgr2=0x7f0d2c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:21.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:21.835+0000 7f0d45bff700 1 --2- 172.21.15.27:0/4183813621 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0d2c066f30 0x7f0d2c0693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0d300091e0 tx=0x7f0d30019000 comp rx=0 tx=0).stop 2024-01-11T08:25:21.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:21.835+0000 7f0d45bff700 1 -- 172.21.15.27:0/4183813621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d400b2210 msgr2=0x7f0d400b2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:21.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:21.835+0000 7f0d45bff700 1 --2- 172.21.15.27:0/4183813621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d400b2210 0x7f0d400b2680 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f0d400899d0 tx=0x7f0d3800b040 comp rx=0 tx=0).stop 2024-01-11T08:25:21.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:21.835+0000 7f0d45bff700 1 -- 172.21.15.27:0/4183813621 shutdown_connections 2024-01-11T08:25:21.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:21.835+0000 7f0d45bff700 1 --2- 172.21.15.27:0/4183813621 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0d2c066f30 0x7f0d2c0693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:21.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:21.835+0000 7f0d45bff700 1 --2- 172.21.15.27:0/4183813621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d400b2210 0x7f0d400b2680 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:21.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:21.835+0000 7f0d45bff700 1 --2- 172.21.15.27:0/4183813621 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0d28003a60 0x7f0d4014dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:21.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:21.835+0000 7f0d45bff700 1 --2- 172.21.15.27:0/4183813621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d40150df0 0x7f0d40151260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:21.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:21.835+0000 7f0d45bff700 1 -- 172.21.15.27:0/4183813621 >> 172.21.15.27:0/4183813621 conn(0x7f0d4006ed40 msgr2=0x7f0d401500d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:21.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:21.836+0000 7f0d45bff700 1 -- 172.21.15.27:0/4183813621 shutdown_connections 2024-01-11T08:25:21.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:21.836+0000 7f0d45bff700 1 -- 172.21.15.27:0/4183813621 wait complete. 2024-01-11T08:25:21.838 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-11T08:25:21.858 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-11T08:25:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.069+0000 7f7087bff700 1 Processor -- start 2024-01-11T08:25:22.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.069+0000 7f7087bff700 1 -- start start 2024-01-11T08:25:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.070+0000 7f7087bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f70800b03e0 0x7f70800b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.071+0000 7f7087bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f70800b07b0 0x7f70800b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.071+0000 7f7087bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7080159200 0x7f7080159620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.071+0000 7f7087bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70800cd730 con 0x7f70800b07b0 2024-01-11T08:25:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.071+0000 7f7087bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70800ccb20 con 0x7f70800b03e0 2024-01-11T08:25:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.071+0000 7f7087bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70800cce00 con 0x7f7080159200 2024-01-11T08:25:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.071+0000 7f708519a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f70800b07b0 0x7f70800b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:22.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.071+0000 7f708519a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f70800b07b0 0x7f70800b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47306/0 (socket says 172.21.15.27:47306) 2024-01-11T08:25:22.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.071+0000 7f708519a700 1 -- 172.21.15.27:0/707330999 learned_addr learned my addr 172.21.15.27:0/707330999 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:22.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.071+0000 7f708599b700 1 --2- 172.21.15.27:0/707330999 >> v2:172.21.15.77:3300/0 conn(0x7f70800b03e0 0x7f70800b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:22.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.071+0000 7f708619c700 1 --2- 172.21.15.27:0/707330999 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7080159200 0x7f7080159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:22.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.071+0000 7f708519a700 1 -- 172.21.15.27:0/707330999 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7080159200 msgr2=0x7f7080159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:22.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.071+0000 7f708519a700 1 --2- 172.21.15.27:0/707330999 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7080159200 0x7f7080159620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:22.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.071+0000 7f708519a700 1 -- 172.21.15.27:0/707330999 >> v2:172.21.15.77:3300/0 conn(0x7f70800b03e0 msgr2=0x7f70800b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:22.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.071+0000 7f708519a700 1 --2- 172.21.15.27:0/707330999 >> v2:172.21.15.77:3300/0 conn(0x7f70800b03e0 0x7f70800b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:22.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.071+0000 7f708519a700 1 -- 172.21.15.27:0/707330999 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70800cd0e0 con 0x7f70800b07b0 2024-01-11T08:25:22.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.071+0000 7f708619c700 1 --2- 172.21.15.27:0/707330999 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7080159200 0x7f7080159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:22.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.072+0000 7f708519a700 1 --2- 172.21.15.27:0/707330999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70800b07b0 0x7f70800b29c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f7080089470 tx=0x7f7070009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73253761d41ae6af server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:22.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.072+0000 7f7084999700 1 -- 172.21.15.27:0/707330999 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7070009e70 con 0x7f70800b07b0 2024-01-11T08:25:22.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.072+0000 7f7084999700 1 -- 172.21.15.27:0/707330999 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707000b860 con 0x7f70800b07b0 2024-01-11T08:25:22.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.072+0000 7f7084999700 1 -- 172.21.15.27:0/707330999 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707000bc00 con 0x7f70800b07b0 2024-01-11T08:25:22.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.073+0000 7f7087bff700 1 -- 172.21.15.27:0/707330999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70800b07b0 msgr2=0x7f70800b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.073+0000 7f7087bff700 1 --2- 172.21.15.27:0/707330999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70800b07b0 0x7f70800b29c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f7080089470 tx=0x7f7070009250 comp rx=0 tx=0).stop 2024-01-11T08:25:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.073+0000 7f7087bff700 1 -- 172.21.15.27:0/707330999 shutdown_connections 2024-01-11T08:25:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.073+0000 7f7087bff700 1 --2- 172.21.15.27:0/707330999 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7080159200 0x7f7080159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.073+0000 7f7087bff700 1 --2- 172.21.15.27:0/707330999 >> v2:172.21.15.77:3300/0 conn(0x7f70800b03e0 0x7f70800b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.073+0000 7f7087bff700 1 --2- 172.21.15.27:0/707330999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70800b07b0 0x7f70800b29c0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.073+0000 7f7087bff700 1 -- 172.21.15.27:0/707330999 >> 172.21.15.27:0/707330999 conn(0x7f708006ed40 msgr2=0x7f708006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.073+0000 7f7087bff700 1 -- 172.21.15.27:0/707330999 shutdown_connections 2024-01-11T08:25:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.073+0000 7f7087bff700 1 -- 172.21.15.27:0/707330999 wait complete. 2024-01-11T08:25:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.074+0000 7f7087bff700 1 Processor -- start 2024-01-11T08:25:22.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.074+0000 7f7087bff700 1 -- start start 2024-01-11T08:25:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.075+0000 7f7087bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70800b03e0 0x7f7080157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.075+0000 7f7087bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70800b07b0 0x7f70800c5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.075+0000 7f7087bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7080159200 0x7f708015bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.075+0000 7f7087bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708015cd70 con 0x7f7080159200 2024-01-11T08:25:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.075+0000 7f7087bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70800c4650 con 0x7f70800b03e0 2024-01-11T08:25:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.075+0000 7f7087bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708015ca90 con 0x7f70800b07b0 2024-01-11T08:25:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.076+0000 7f708619c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7080159200 0x7f708015bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.076+0000 7f708599b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70800b03e0 0x7f7080157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.076+0000 7f708619c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7080159200 0x7f708015bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47308/0 (socket says 172.21.15.27:47308) 2024-01-11T08:25:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.076+0000 7f708599b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70800b03e0 0x7f7080157a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:58958/0 (socket says 172.21.15.27:58958) 2024-01-11T08:25:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.076+0000 7f708619c700 1 -- 172.21.15.27:0/2429048311 learned_addr learned my addr 172.21.15.27:0/2429048311 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.076+0000 7f708519a700 1 --2- 172.21.15.27:0/2429048311 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70800b07b0 0x7f70800c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.076+0000 7f708619c700 1 -- 172.21.15.27:0/2429048311 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70800b07b0 msgr2=0x7f70800c5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.076+0000 7f708619c700 1 --2- 172.21.15.27:0/2429048311 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70800b07b0 0x7f70800c5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.076+0000 7f708619c700 1 -- 172.21.15.27:0/2429048311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70800b03e0 msgr2=0x7f7080157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.076+0000 7f708619c700 1 --2- 172.21.15.27:0/2429048311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70800b03e0 0x7f7080157a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.076+0000 7f708619c700 1 -- 172.21.15.27:0/2429048311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70700094f0 con 0x7f7080159200 2024-01-11T08:25:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.076+0000 7f708619c700 1 --2- 172.21.15.27:0/2429048311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7080159200 0x7f708015bf40 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7078009fc0 tx=0x7f70780099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.077+0000 7f7075ffb700 1 -- 172.21.15.27:0/2429048311 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707800dbe0 con 0x7f7080159200 2024-01-11T08:25:22.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.077+0000 7f7075ffb700 1 -- 172.21.15.27:0/2429048311 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707800f040 con 0x7f7080159200 2024-01-11T08:25:22.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.077+0000 7f7075ffb700 1 -- 172.21.15.27:0/2429048311 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7078015440 con 0x7f7080159200 2024-01-11T08:25:22.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.077+0000 7f7087bff700 1 -- 172.21.15.27:0/2429048311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70800c4000 con 0x7f7080159200 2024-01-11T08:25:22.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.077+0000 7f7087bff700 1 -- 172.21.15.27:0/2429048311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7080157f90 con 0x7f7080159200 2024-01-11T08:25:22.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.078+0000 7f7087bff700 1 -- 172.21.15.27:0/2429048311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70800c47c0 con 0x7f7080159200 2024-01-11T08:25:22.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.079+0000 7f7075ffb700 1 -- 172.21.15.27:0/2429048311 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f707801e3b0 con 0x7f7080159200 2024-01-11T08:25:22.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.080+0000 7f7075ffb700 1 --2- 172.21.15.27:0/2429048311 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7064066f30 0x7f70640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:22.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.080+0000 7f7075ffb700 1 -- 172.21.15.27:0/2429048311 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5251+0+0 (secure 0 0 0) 0x7f7078084750 con 0x7f7080159200 2024-01-11T08:25:22.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.080+0000 7f708599b700 1 --2- 172.21.15.27:0/2429048311 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7064066f30 0x7f70640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:22.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.081+0000 7f708599b700 1 --2- 172.21.15.27:0/2429048311 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7064066f30 0x7f70640693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f70800b1850 tx=0x7f707c0091a0 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:22.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.085+0000 7f7075ffb700 1 -- 172.21.15.27:0/2429048311 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7078051b80 con 0x7f7080159200 2024-01-11T08:25:22.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.289+0000 7f7087bff700 1 -- 172.21.15.27:0/2429048311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f708004c3c0 con 0x7f7080159200 2024-01-11T08:25:22.332 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:25:22.337 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:25:22.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.838+0000 7f7075ffb700 1 -- 172.21.15.27:0/2429048311 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v73) v1 ==== 170+0+0 (secure 0 0 0) 0x7f70780551a0 con 0x7f7080159200 2024-01-11T08:25:22.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.844+0000 7f7087bff700 1 -- 172.21.15.27:0/2429048311 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7064066f30 msgr2=0x7f70640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:22.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.844+0000 7f7087bff700 1 --2- 172.21.15.27:0/2429048311 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7064066f30 0x7f70640693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f70800b1850 tx=0x7f707c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:22.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.844+0000 7f7087bff700 1 -- 172.21.15.27:0/2429048311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7080159200 msgr2=0x7f708015bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:22.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.844+0000 7f7087bff700 1 --2- 172.21.15.27:0/2429048311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7080159200 0x7f708015bf40 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7078009fc0 tx=0x7f70780099b0 comp rx=0 tx=0).stop 2024-01-11T08:25:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.845+0000 7f7087bff700 1 -- 172.21.15.27:0/2429048311 shutdown_connections 2024-01-11T08:25:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.845+0000 7f7087bff700 1 --2- 172.21.15.27:0/2429048311 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7064066f30 0x7f70640693e0 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f70800b1850 tx=0x7f707c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.845+0000 7f7087bff700 1 --2- 172.21.15.27:0/2429048311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7080159200 0x7f708015bf40 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:22.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.845+0000 7f7087bff700 1 --2- 172.21.15.27:0/2429048311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70800b03e0 0x7f7080157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:22.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.845+0000 7f7087bff700 1 --2- 172.21.15.27:0/2429048311 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70800b07b0 0x7f70800c5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:22.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.845+0000 7f7087bff700 1 -- 172.21.15.27:0/2429048311 >> 172.21.15.27:0/2429048311 conn(0x7f708006ed40 msgr2=0x7f70800ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:22.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.845+0000 7f7087bff700 1 -- 172.21.15.27:0/2429048311 shutdown_connections 2024-01-11T08:25:22.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:22.846+0000 7f7087bff700 1 -- 172.21.15.27:0/2429048311 wait complete. 2024-01-11T08:25:22.848 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-11T08:25:22.874 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-11T08:25:23.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.078+0000 7fd0c07ff700 1 Processor -- start 2024-01-11T08:25:23.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.078+0000 7fd0c07ff700 1 -- start start 2024-01-11T08:25:23.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.080+0000 7fd0c07ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd0b8142dc0 0x7fd0b80cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.080+0000 7fd0c07ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd0b8143190 0x7fd0b80cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.080+0000 7fd0c07ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b815d320 con 0x7fd0b80cc020 2024-01-11T08:25:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.080+0000 7fd0c07ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b815df30 con 0x7fd0b8142dc0 2024-01-11T08:25:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.080+0000 7fd0c07ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b815d590 con 0x7fd0b8143190 2024-01-11T08:25:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.080+0000 7fd0be59b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd0b8142dc0 0x7fd0b80cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.080+0000 7fd0be59b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd0b8142dc0 0x7fd0b80cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47314/0 (socket says 172.21.15.27:47314) 2024-01-11T08:25:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.080+0000 7fd0be59b700 1 -- 172.21.15.27:0/2038850201 learned_addr learned my addr 172.21.15.27:0/2038850201 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.080+0000 7fd0bdd9a700 1 --2- 172.21.15.27:0/2038850201 >> v2:172.21.15.77:3300/0 conn(0x7fd0b8143190 0x7fd0b80cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.081+0000 7fd0be59b700 1 -- 172.21.15.27:0/2038850201 >> v2:172.21.15.77:3300/0 conn(0x7fd0b8143190 msgr2=0x7fd0b80cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.081+0000 7fd0be59b700 1 --2- 172.21.15.27:0/2038850201 >> v2:172.21.15.77:3300/0 conn(0x7fd0b8143190 0x7fd0b80cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.081+0000 7fd0be59b700 1 -- 172.21.15.27:0/2038850201 >> v1:172.21.15.77:6789/0 conn(0x7fd0b80cc020 legacy=0x7fd0b80cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.081+0000 7fd0be59b700 1 -- 172.21.15.27:0/2038850201 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b815d870 con 0x7fd0b8142dc0 2024-01-11T08:25:23.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.081+0000 7fd0be59b700 1 --2- 172.21.15.27:0/2038850201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8142dc0 0x7fd0b80cada0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd0b806c480 tx=0x7fd0b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ca0a5fa5baf3a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:23.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.081+0000 7fd0bd599700 1 -- 172.21.15.27:0/2038850201 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b4009e70 con 0x7fd0b8142dc0 2024-01-11T08:25:23.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.081+0000 7fd0bd599700 1 -- 172.21.15.27:0/2038850201 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b400b860 con 0x7fd0b8142dc0 2024-01-11T08:25:23.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.081+0000 7fd0bd599700 1 -- 172.21.15.27:0/2038850201 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b400bc00 con 0x7fd0b8142dc0 2024-01-11T08:25:23.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.082+0000 7fd0c07ff700 1 -- 172.21.15.27:0/2038850201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8142dc0 msgr2=0x7fd0b80cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:23.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.082+0000 7fd0c07ff700 1 --2- 172.21.15.27:0/2038850201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8142dc0 0x7fd0b80cada0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd0b806c480 tx=0x7fd0b4009250 comp rx=0 tx=0).stop 2024-01-11T08:25:23.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.082+0000 7fd0c07ff700 1 -- 172.21.15.27:0/2038850201 shutdown_connections 2024-01-11T08:25:23.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.082+0000 7fd0c07ff700 1 --2- 172.21.15.27:0/2038850201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8142dc0 0x7fd0b80cada0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:23.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.082+0000 7fd0c07ff700 1 --2- 172.21.15.27:0/2038850201 >> v2:172.21.15.77:3300/0 conn(0x7fd0b8143190 0x7fd0b80cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:23.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.082+0000 7fd0c07ff700 1 -- 172.21.15.27:0/2038850201 >> 172.21.15.27:0/2038850201 conn(0x7fd0b8063ca0 msgr2=0x7fd0b806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:23.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.082+0000 7fd0c07ff700 1 -- 172.21.15.27:0/2038850201 shutdown_connections 2024-01-11T08:25:23.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.082+0000 7fd0c07ff700 1 -- 172.21.15.27:0/2038850201 wait complete. 2024-01-11T08:25:23.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.083+0000 7fd0c07ff700 1 Processor -- start 2024-01-11T08:25:23.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.084+0000 7fd0c07ff700 1 -- start start 2024-01-11T08:25:23.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.084+0000 7fd0c07ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0b80cc020 0x7fd0b815a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:23.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.084+0000 7fd0c07ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0b8142dc0 0x7fd0b80c56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:23.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.084+0000 7fd0c07ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8143190 0x7fd0b8140e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:23.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.084+0000 7fd0c07ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b80c6d40 con 0x7fd0b8143190 2024-01-11T08:25:23.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.084+0000 7fd0c07ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b80c7ff0 con 0x7fd0b8142dc0 2024-01-11T08:25:23.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.084+0000 7fd0c07ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b815a860 con 0x7fd0b80cc020 2024-01-11T08:25:23.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.085+0000 7fd0bed9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8143190 0x7fd0b8140e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:23.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.085+0000 7fd0bed9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8143190 0x7fd0b8140e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47326/0 (socket says 172.21.15.27:47326) 2024-01-11T08:25:23.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.085+0000 7fd0bed9c700 1 -- 172.21.15.27:0/480126501 learned_addr learned my addr 172.21.15.27:0/480126501 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:23.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.085+0000 7fd0bdd9a700 1 --2- 172.21.15.27:0/480126501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0b8142dc0 0x7fd0b80c56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:23.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.085+0000 7fd0be59b700 1 --2- 172.21.15.27:0/480126501 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0b80cc020 0x7fd0b815a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:23.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.085+0000 7fd0bed9c700 1 -- 172.21.15.27:0/480126501 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0b80cc020 msgr2=0x7fd0b815a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:23.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.085+0000 7fd0bed9c700 1 --2- 172.21.15.27:0/480126501 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0b80cc020 0x7fd0b815a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:23.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.085+0000 7fd0bed9c700 1 -- 172.21.15.27:0/480126501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0b8142dc0 msgr2=0x7fd0b80c56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:23.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.085+0000 7fd0bed9c700 1 --2- 172.21.15.27:0/480126501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0b8142dc0 0x7fd0b80c56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:23.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.085+0000 7fd0bed9c700 1 -- 172.21.15.27:0/480126501 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b40094f0 con 0x7fd0b8143190 2024-01-11T08:25:23.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.085+0000 7fd0bed9c700 1 --2- 172.21.15.27:0/480126501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8143190 0x7fd0b8140e90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd0b000c8e0 tx=0x7fd0b0001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:23.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.086+0000 7fd0ae7fc700 1 -- 172.21.15.27:0/480126501 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b000ce80 con 0x7fd0b8143190 2024-01-11T08:25:23.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.086+0000 7fd0ae7fc700 1 -- 172.21.15.27:0/480126501 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b000a7e0 con 0x7fd0b8143190 2024-01-11T08:25:23.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.086+0000 7fd0ae7fc700 1 -- 172.21.15.27:0/480126501 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b000ab80 con 0x7fd0b8143190 2024-01-11T08:25:23.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.086+0000 7fd0c07ff700 1 -- 172.21.15.27:0/480126501 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b80c6f50 con 0x7fd0b8143190 2024-01-11T08:25:23.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.086+0000 7fd0c07ff700 1 -- 172.21.15.27:0/480126501 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b80c77f0 con 0x7fd0b8143190 2024-01-11T08:25:23.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.087+0000 7fd0c07ff700 1 -- 172.21.15.27:0/480126501 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b804c3c0 con 0x7fd0b8143190 2024-01-11T08:25:23.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.088+0000 7fd0ae7fc700 1 -- 172.21.15.27:0/480126501 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd0b0012420 con 0x7fd0b8143190 2024-01-11T08:25:23.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.089+0000 7fd0ae7fc700 1 --2- 172.21.15.27:0/480126501 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd09c066f30 0x7fd09c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:23.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.089+0000 7fd0ae7fc700 1 -- 172.21.15.27:0/480126501 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5633+0+0 (secure 0 0 0) 0x7fd0b0084840 con 0x7fd0b8143190 2024-01-11T08:25:23.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.089+0000 7fd0be59b700 1 --2- 172.21.15.27:0/480126501 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd09c066f30 0x7fd09c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:23.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.091+0000 7fd0be59b700 1 --2- 172.21.15.27:0/480126501 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd09c066f30 0x7fd09c0693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd0b806c480 tx=0x7fd0b4017000 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:23.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.094+0000 7fd0ae7fc700 1 -- 172.21.15.27:0/480126501 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd0b0051a50 con 0x7fd0b8143190 2024-01-11T08:25:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.344+0000 7fd0c07ff700 1 -- 172.21.15.27:0/480126501 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd0b815b8a0 con 0x7fd0b8143190 2024-01-11T08:25:23.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.842+0000 7fd0ae7fc700 1 -- 172.21.15.27:0/480126501 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v30) v1 ==== 172+0+0 (secure 0 0 0) 0x7fd0b0055070 con 0x7fd0b8143190 2024-01-11T08:25:23.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.846+0000 7fd0c07ff700 1 -- 172.21.15.27:0/480126501 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd09c066f30 msgr2=0x7fd09c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:23.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.846+0000 7fd0c07ff700 1 --2- 172.21.15.27:0/480126501 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd09c066f30 0x7fd09c0693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd0b806c480 tx=0x7fd0b4017000 comp rx=0 tx=0).stop 2024-01-11T08:25:23.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.846+0000 7fd0c07ff700 1 -- 172.21.15.27:0/480126501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8143190 msgr2=0x7fd0b8140e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:23.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.846+0000 7fd0c07ff700 1 --2- 172.21.15.27:0/480126501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8143190 0x7fd0b8140e90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd0b000c8e0 tx=0x7fd0b0001710 comp rx=0 tx=0).stop 2024-01-11T08:25:23.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.846+0000 7fd0c07ff700 1 -- 172.21.15.27:0/480126501 shutdown_connections 2024-01-11T08:25:23.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.846+0000 7fd0c07ff700 1 --2- 172.21.15.27:0/480126501 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd09c066f30 0x7fd09c0693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:23.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.846+0000 7fd0c07ff700 1 --2- 172.21.15.27:0/480126501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0b8143190 0x7fd0b8140e90 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:23.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.846+0000 7fd0c07ff700 1 --2- 172.21.15.27:0/480126501 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0b80cc020 0x7fd0b815a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:23.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.846+0000 7fd0c07ff700 1 --2- 172.21.15.27:0/480126501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0b8142dc0 0x7fd0b80c56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:23.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.846+0000 7fd0c07ff700 1 -- 172.21.15.27:0/480126501 >> 172.21.15.27:0/480126501 conn(0x7fd0b8063ca0 msgr2=0x7fd0b81420c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:23.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.846+0000 7fd0c07ff700 1 -- 172.21.15.27:0/480126501 shutdown_connections 2024-01-11T08:25:23.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:23.847+0000 7fd0c07ff700 1 -- 172.21.15.27:0/480126501 wait complete. 2024-01-11T08:25:23.856 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 8 and data pool 9 2024-01-11T08:25:23.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:25:24.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.088+0000 7f2ba89ff700 1 Processor -- start 2024-01-11T08:25:24.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.088+0000 7f2ba89ff700 1 -- start start 2024-01-11T08:25:24.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.089+0000 7f2ba89ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2ba40b2b90 0x7f2ba4159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:24.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.089+0000 7f2ba89ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2ba40b07b0 0x7f2ba4159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:24.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.089+0000 7f2ba89ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba40cd730 con 0x7f2ba40b03e0 2024-01-11T08:25:24.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.089+0000 7f2ba89ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ba40ccb20 con 0x7f2ba40b2b90 2024-01-11T08:25:24.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.089+0000 7f2ba89ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ba40cce00 con 0x7f2ba40b07b0 2024-01-11T08:25:24.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.090+0000 7f2ba1d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2ba40b2b90 0x7f2ba4159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:24.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.090+0000 7f2ba259c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2ba40b03e0 0x7f2ba40b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46354/0 (socket says 172.21.15.27:46354) 2024-01-11T08:25:24.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.090+0000 7f2ba259c700 1 -- 172.21.15.27:0/2049855306 learned_addr learned my addr 172.21.15.27:0/2049855306 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:24.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.090+0000 7f2ba2d9d700 1 --2- 172.21.15.27:0/2049855306 >> v2:172.21.15.77:3300/0 conn(0x7f2ba40b07b0 0x7f2ba4159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:24.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.090+0000 7f2ba1d9b700 1 -- 172.21.15.27:0/2049855306 >> v2:172.21.15.77:3300/0 conn(0x7f2ba40b07b0 msgr2=0x7f2ba4159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:24.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.090+0000 7f2ba1d9b700 1 --2- 172.21.15.27:0/2049855306 >> v2:172.21.15.77:3300/0 conn(0x7f2ba40b07b0 0x7f2ba4159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.090+0000 7f2ba1d9b700 1 -- 172.21.15.27:0/2049855306 >> v1:172.21.15.27:6789/0 conn(0x7f2ba40b03e0 legacy=0x7f2ba40b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:24.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.090+0000 7f2ba1d9b700 1 -- 172.21.15.27:0/2049855306 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba40cd0e0 con 0x7f2ba40b2b90 2024-01-11T08:25:24.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.091+0000 7f2ba1d9b700 1 --2- 172.21.15.27:0/2049855306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40b2b90 0x7f2ba4159200 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40b1d10 tx=0x7f2b980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a01ebbfc04624f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:24.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.091+0000 7f2ba159a700 1 -- 172.21.15.27:0/2049855306 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9800b6b0 con 0x7f2ba40b2b90 2024-01-11T08:25:24.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.091+0000 7f2ba159a700 1 -- 172.21.15.27:0/2049855306 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9800b810 con 0x7f2ba40b2b90 2024-01-11T08:25:24.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.091+0000 7f2ba159a700 1 -- 172.21.15.27:0/2049855306 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9800bbb0 con 0x7f2ba40b2b90 2024-01-11T08:25:24.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.091+0000 7f2ba89ff700 1 -- 172.21.15.27:0/2049855306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40b2b90 msgr2=0x7f2ba4159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:24.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.091+0000 7f2ba89ff700 1 --2- 172.21.15.27:0/2049855306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40b2b90 0x7f2ba4159200 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40b1d10 tx=0x7f2b980091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.091+0000 7f2ba89ff700 1 -- 172.21.15.27:0/2049855306 shutdown_connections 2024-01-11T08:25:24.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.091+0000 7f2ba89ff700 1 --2- 172.21.15.27:0/2049855306 >> v2:172.21.15.77:3300/0 conn(0x7f2ba40b07b0 0x7f2ba4159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.091+0000 7f2ba89ff700 1 --2- 172.21.15.27:0/2049855306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40b2b90 0x7f2ba4159200 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.092+0000 7f2ba89ff700 1 -- 172.21.15.27:0/2049855306 >> 172.21.15.27:0/2049855306 conn(0x7f2ba406ed40 msgr2=0x7f2ba406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:24.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.092+0000 7f2ba89ff700 1 -- 172.21.15.27:0/2049855306 shutdown_connections 2024-01-11T08:25:24.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.092+0000 7f2ba89ff700 1 -- 172.21.15.27:0/2049855306 wait complete. 2024-01-11T08:25:24.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.093+0000 7f2ba89ff700 1 Processor -- start 2024-01-11T08:25:24.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.093+0000 7f2ba89ff700 1 -- start start 2024-01-11T08:25:24.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.094+0000 7f2ba89ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba40b03e0 0x7f2ba4158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:24.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.094+0000 7f2ba89ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40b07b0 0x7f2ba40c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:24.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.094+0000 7f2ba89ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba40b2b90 0x7f2ba40ae460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:24.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.094+0000 7f2ba89ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba40c4e20 con 0x7f2ba40b07b0 2024-01-11T08:25:24.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.094+0000 7f2ba89ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba40c5100 con 0x7f2ba40b03e0 2024-01-11T08:25:24.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.094+0000 7f2ba89ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba40c53e0 con 0x7f2ba40b2b90 2024-01-11T08:25:24.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.094+0000 7f2ba1d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40b07b0 0x7f2ba40c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:24.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.095+0000 7f2ba1d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40b07b0 0x7f2ba40c5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56830/0 (socket says 172.21.15.27:56830) 2024-01-11T08:25:24.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.095+0000 7f2ba1d9b700 1 -- 172.21.15.27:0/4024373778 learned_addr learned my addr 172.21.15.27:0/4024373778 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:24.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.095+0000 7f2ba259c700 1 --2- 172.21.15.27:0/4024373778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba40b03e0 0x7f2ba4158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:24.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.095+0000 7f2ba2d9d700 1 --2- 172.21.15.27:0/4024373778 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba40b2b90 0x7f2ba40ae460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:24.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.095+0000 7f2ba1d9b700 1 -- 172.21.15.27:0/4024373778 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba40b2b90 msgr2=0x7f2ba40ae460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:24.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.095+0000 7f2ba1d9b700 1 --2- 172.21.15.27:0/4024373778 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba40b2b90 0x7f2ba40ae460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.095+0000 7f2ba1d9b700 1 -- 172.21.15.27:0/4024373778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba40b03e0 msgr2=0x7f2ba4158bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:24.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.095+0000 7f2ba1d9b700 1 --2- 172.21.15.27:0/4024373778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba40b03e0 0x7f2ba4158bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.095+0000 7f2ba1d9b700 1 -- 172.21.15.27:0/4024373778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b98009420 con 0x7f2ba40b07b0 2024-01-11T08:25:24.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.095+0000 7f2ba1d9b700 1 --2- 172.21.15.27:0/4024373778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40b07b0 0x7f2ba40c5fc0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4069540 tx=0x7f2b98017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:24.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.096+0000 7f2b927fc700 1 -- 172.21.15.27:0/4024373778 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9800bb90 con 0x7f2ba40b07b0 2024-01-11T08:25:24.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.096+0000 7f2b927fc700 1 -- 172.21.15.27:0/4024373778 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b98011420 con 0x7f2ba40b07b0 2024-01-11T08:25:24.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.096+0000 7f2b927fc700 1 -- 172.21.15.27:0/4024373778 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b980117c0 con 0x7f2ba40b07b0 2024-01-11T08:25:24.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.096+0000 7f2ba89ff700 1 -- 172.21.15.27:0/4024373778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba40c56c0 con 0x7f2ba40b07b0 2024-01-11T08:25:24.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.096+0000 7f2ba89ff700 1 -- 172.21.15.27:0/4024373778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba40c4860 con 0x7f2ba40b07b0 2024-01-11T08:25:24.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.097+0000 7f2ba89ff700 1 -- 172.21.15.27:0/4024373778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba40718d0 con 0x7f2ba40b07b0 2024-01-11T08:25:24.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.098+0000 7f2b927fc700 1 -- 172.21.15.27:0/4024373778 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2b98011bb0 con 0x7f2ba40b07b0 2024-01-11T08:25:24.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.099+0000 7f2b927fc700 1 --2- 172.21.15.27:0/4024373778 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2b84066f30 0x7f2b840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:24.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.099+0000 7f2b927fc700 1 -- 172.21.15.27:0/4024373778 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f2b98085e30 con 0x7f2ba40b07b0 2024-01-11T08:25:24.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.100+0000 7f2ba259c700 1 --2- 172.21.15.27:0/4024373778 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2b84066f30 0x7f2b840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:24.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.100+0000 7f2ba259c700 1 --2- 172.21.15.27:0/4024373778 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2b84066f30 0x7f2b840693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2b94007130 tx=0x7f2b94007eb0 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:24.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.103+0000 7f2b927fc700 1 -- 172.21.15.27:0/4024373778 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2b98052fe0 con 0x7f2ba40b07b0 2024-01-11T08:25:24.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.307+0000 7f2ba89ff700 1 -- 172.21.15.27:0/4024373778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ba40cd3c0 con 0x7f2ba40b07b0 2024-01-11T08:25:24.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.308+0000 7f2b927fc700 1 -- 172.21.15.27:0/4024373778 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+15358 (secure 0 0 0) 0x7f2b98011e60 con 0x7f2ba40b07b0 2024-01-11T08:25:24.307 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:24.307 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":74,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:25:23.345193+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:25:21.160295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-11T08:25:22.342780+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:25:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.310+0000 7f2ba89ff700 1 -- 172.21.15.27:0/4024373778 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2b84066f30 msgr2=0x7f2b840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.310+0000 7f2ba89ff700 1 --2- 172.21.15.27:0/4024373778 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2b84066f30 0x7f2b840693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2b94007130 tx=0x7f2b94007eb0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.310+0000 7f2ba89ff700 1 -- 172.21.15.27:0/4024373778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40b07b0 msgr2=0x7f2ba40c5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.310+0000 7f2ba89ff700 1 --2- 172.21.15.27:0/4024373778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40b07b0 0x7f2ba40c5fc0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4069540 tx=0x7f2b98017000 comp rx=0 tx=0).stop 2024-01-11T08:25:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.311+0000 7f2ba89ff700 1 -- 172.21.15.27:0/4024373778 shutdown_connections 2024-01-11T08:25:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.311+0000 7f2ba89ff700 1 --2- 172.21.15.27:0/4024373778 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2b84066f30 0x7f2b840693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.311+0000 7f2ba89ff700 1 --2- 172.21.15.27:0/4024373778 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba40b2b90 0x7f2ba40ae460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.311+0000 7f2ba89ff700 1 --2- 172.21.15.27:0/4024373778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba40b03e0 0x7f2ba4158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.311+0000 7f2ba89ff700 1 --2- 172.21.15.27:0/4024373778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40b07b0 0x7f2ba40c5fc0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.311+0000 7f2ba89ff700 1 -- 172.21.15.27:0/4024373778 >> 172.21.15.27:0/4024373778 conn(0x7f2ba406ed40 msgr2=0x7f2ba40d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.311+0000 7f2ba89ff700 1 -- 172.21.15.27:0/4024373778 shutdown_connections 2024-01-11T08:25:24.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.311+0000 7f2ba89ff700 1 -- 172.21.15.27:0/4024373778 wait complete. 2024-01-11T08:25:24.329 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:25:24.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.528+0000 7f4e683ff700 1 Processor -- start 2024-01-11T08:25:24.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.529+0000 7f4e683ff700 1 -- start start 2024-01-11T08:25:24.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.530+0000 7f4e683ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4e600b03e0 0x7f4e600b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:24.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.530+0000 7f4e683ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e600b29c0 0x7f4e60159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:24.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.530+0000 7f4e683ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e600cd730 con 0x7f4e600b07b0 2024-01-11T08:25:24.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.530+0000 7f4e683ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e600b2de0 con 0x7f4e600b03e0 2024-01-11T08:25:24.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.530+0000 7f4e683ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e600ccb20 con 0x7f4e600b29c0 2024-01-11T08:25:24.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.530+0000 7f4e6619b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4e600b03e0 0x7f4e600b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:24.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.530+0000 7f4e6619b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4e600b03e0 0x7f4e600b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56844/0 (socket says 172.21.15.27:56844) 2024-01-11T08:25:24.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.530+0000 7f4e6619b700 1 -- 172.21.15.27:0/2248497661 learned_addr learned my addr 172.21.15.27:0/2248497661 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:24.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.530+0000 7f4e6699c700 1 --2- 172.21.15.27:0/2248497661 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e600b29c0 0x7f4e60159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:24.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.531+0000 7f4e6619b700 1 -- 172.21.15.27:0/2248497661 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e600b29c0 msgr2=0x7f4e60159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:24.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.531+0000 7f4e6619b700 1 --2- 172.21.15.27:0/2248497661 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e600b29c0 0x7f4e60159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.531+0000 7f4e6619b700 1 -- 172.21.15.27:0/2248497661 >> v1:172.21.15.77:6789/0 conn(0x7f4e600b07b0 legacy=0x7f4e60159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:24.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.531+0000 7f4e6619b700 1 -- 172.21.15.27:0/2248497661 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e600cce00 con 0x7f4e600b03e0 2024-01-11T08:25:24.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.531+0000 7f4e6619b700 1 --2- 172.21.15.27:0/2248497661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e600b03e0 0x7f4e600b2480 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c0099b0 tx=0x7f4e5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e8b77f5d2b548bf server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:24.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.531+0000 7f4e65199700 1 -- 172.21.15.27:0/2248497661 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c009ea0 con 0x7f4e600b03e0 2024-01-11T08:25:24.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.531+0000 7f4e65199700 1 -- 172.21.15.27:0/2248497661 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5c00b860 con 0x7f4e600b03e0 2024-01-11T08:25:24.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.531+0000 7f4e65199700 1 -- 172.21.15.27:0/2248497661 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c00bc00 con 0x7f4e600b03e0 2024-01-11T08:25:24.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.532+0000 7f4e683ff700 1 -- 172.21.15.27:0/2248497661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e600b03e0 msgr2=0x7f4e600b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:24.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.532+0000 7f4e683ff700 1 --2- 172.21.15.27:0/2248497661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e600b03e0 0x7f4e600b2480 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c0099b0 tx=0x7f4e5c009250 comp rx=0 tx=0).stop 2024-01-11T08:25:24.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.532+0000 7f4e683ff700 1 -- 172.21.15.27:0/2248497661 shutdown_connections 2024-01-11T08:25:24.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.532+0000 7f4e683ff700 1 --2- 172.21.15.27:0/2248497661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e600b03e0 0x7f4e600b2480 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.532+0000 7f4e683ff700 1 --2- 172.21.15.27:0/2248497661 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e600b29c0 0x7f4e60159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.532+0000 7f4e683ff700 1 -- 172.21.15.27:0/2248497661 >> 172.21.15.27:0/2248497661 conn(0x7f4e6006ed40 msgr2=0x7f4e6006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:24.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.532+0000 7f4e683ff700 1 -- 172.21.15.27:0/2248497661 shutdown_connections 2024-01-11T08:25:24.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.532+0000 7f4e683ff700 1 -- 172.21.15.27:0/2248497661 wait complete. 2024-01-11T08:25:24.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.534+0000 7f4e683ff700 1 Processor -- start 2024-01-11T08:25:24.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.534+0000 7f4e683ff700 1 -- start start 2024-01-11T08:25:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.536+0000 7f4e683ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e600b03e0 0x7f4e600c8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.536+0000 7f4e683ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e600b07b0 0x7f4e600b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.536+0000 7f4e683ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e600b29c0 0x7f4e600b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.536+0000 7f4e683ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e60169950 con 0x7f4e600b29c0 2024-01-11T08:25:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.536+0000 7f4e683ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6016a7a0 con 0x7f4e600b03e0 2024-01-11T08:25:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.536+0000 7f4e683ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e60169b90 con 0x7f4e600b07b0 2024-01-11T08:25:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.536+0000 7f4e6699c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e600b29c0 0x7f4e600b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.536+0000 7f4e6699c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e600b29c0 0x7f4e600b5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56846/0 (socket says 172.21.15.27:56846) 2024-01-11T08:25:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.536+0000 7f4e6699c700 1 -- 172.21.15.27:0/1228923963 learned_addr learned my addr 172.21.15.27:0/1228923963 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.536+0000 7f4e6619b700 1 --2- 172.21.15.27:0/1228923963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e600b03e0 0x7f4e600c8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.536+0000 7f4e6599a700 1 --2- 172.21.15.27:0/1228923963 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e600b07b0 0x7f4e600b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.537+0000 7f4e6699c700 1 -- 172.21.15.27:0/1228923963 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e600b07b0 msgr2=0x7f4e600b4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.537+0000 7f4e6699c700 1 --2- 172.21.15.27:0/1228923963 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e600b07b0 0x7f4e600b4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.537+0000 7f4e6699c700 1 -- 172.21.15.27:0/1228923963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e600b03e0 msgr2=0x7f4e600c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.537+0000 7f4e6699c700 1 --2- 172.21.15.27:0/1228923963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e600b03e0 0x7f4e600c8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.537+0000 7f4e6699c700 1 -- 172.21.15.27:0/1228923963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c0094f0 con 0x7f4e600b29c0 2024-01-11T08:25:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.537+0000 7f4e6699c700 1 --2- 172.21.15.27:0/1228923963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e600b29c0 0x7f4e600b5040 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f4e58009df0 tx=0x7f4e5800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.537+0000 7f4e567fc700 1 -- 172.21.15.27:0/1228923963 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5800caa0 con 0x7f4e600b29c0 2024-01-11T08:25:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.537+0000 7f4e567fc700 1 -- 172.21.15.27:0/1228923963 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5800f040 con 0x7f4e600b29c0 2024-01-11T08:25:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.537+0000 7f4e567fc700 1 -- 172.21.15.27:0/1228923963 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5800ce70 con 0x7f4e600b29c0 2024-01-11T08:25:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.537+0000 7f4e683ff700 1 -- 172.21.15.27:0/1228923963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e60169e70 con 0x7f4e600b29c0 2024-01-11T08:25:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.537+0000 7f4e683ff700 1 -- 172.21.15.27:0/1228923963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e6016a430 con 0x7f4e600b29c0 2024-01-11T08:25:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.538+0000 7f4e683ff700 1 -- 172.21.15.27:0/1228923963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e6016a930 con 0x7f4e600b29c0 2024-01-11T08:25:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.540+0000 7f4e567fc700 1 -- 172.21.15.27:0/1228923963 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4e58014690 con 0x7f4e600b29c0 2024-01-11T08:25:24.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.540+0000 7f4e567fc700 1 --2- 172.21.15.27:0/1228923963 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4e44066f30 0x7f4e440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:24.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.541+0000 7f4e567fc700 1 -- 172.21.15.27:0/1228923963 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f4e58085070 con 0x7f4e600b29c0 2024-01-11T08:25:24.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.541+0000 7f4e6619b700 1 --2- 172.21.15.27:0/1228923963 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4e44066f30 0x7f4e440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:24.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.542+0000 7f4e6619b700 1 --2- 172.21.15.27:0/1228923963 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4e44066f30 0x7f4e440693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c0099b0 tx=0x7f4e5c017000 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.544+0000 7f4e567fc700 1 -- 172.21.15.27:0/1228923963 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4e58055d50 con 0x7f4e600b29c0 2024-01-11T08:25:24.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.751+0000 7f4e683ff700 1 -- 172.21.15.27:0/1228923963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e600718d0 con 0x7f4e600b29c0 2024-01-11T08:25:24.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.751+0000 7f4e567fc700 1 -- 172.21.15.27:0/1228923963 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+15358 (secure 0 0 0) 0x7f4e580558e0 con 0x7f4e600b29c0 2024-01-11T08:25:24.752 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:24.752 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":74,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:25:23.345193+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:25:21.160295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-11T08:25:22.342780+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:25:24.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.754+0000 7f4e683ff700 1 -- 172.21.15.27:0/1228923963 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4e44066f30 msgr2=0x7f4e440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:24.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.754+0000 7f4e683ff700 1 --2- 172.21.15.27:0/1228923963 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4e44066f30 0x7f4e440693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c0099b0 tx=0x7f4e5c017000 comp rx=0 tx=0).stop 2024-01-11T08:25:24.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.754+0000 7f4e683ff700 1 -- 172.21.15.27:0/1228923963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e600b29c0 msgr2=0x7f4e600b5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:24.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.754+0000 7f4e683ff700 1 --2- 172.21.15.27:0/1228923963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e600b29c0 0x7f4e600b5040 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f4e58009df0 tx=0x7f4e5800b040 comp rx=0 tx=0).stop 2024-01-11T08:25:24.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.754+0000 7f4e683ff700 1 -- 172.21.15.27:0/1228923963 shutdown_connections 2024-01-11T08:25:24.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.754+0000 7f4e683ff700 1 --2- 172.21.15.27:0/1228923963 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4e44066f30 0x7f4e440693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.754+0000 7f4e683ff700 1 --2- 172.21.15.27:0/1228923963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e600b29c0 0x7f4e600b5040 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.754+0000 7f4e683ff700 1 --2- 172.21.15.27:0/1228923963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e600b03e0 0x7f4e600c8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.754+0000 7f4e683ff700 1 --2- 172.21.15.27:0/1228923963 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e600b07b0 0x7f4e600b4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.754+0000 7f4e683ff700 1 -- 172.21.15.27:0/1228923963 >> 172.21.15.27:0/1228923963 conn(0x7f4e6006ed40 msgr2=0x7f4e6015a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:24.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.754+0000 7f4e683ff700 1 -- 172.21.15.27:0/1228923963 shutdown_connections 2024-01-11T08:25:24.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.754+0000 7f4e683ff700 1 -- 172.21.15.27:0/1228923963 wait complete. 2024-01-11T08:25:24.772 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-11T08:25:24.818 INFO:tasks.ceph.mds.d.smithi077.stderr:2024-01-11T08:25:24.818+0000 7f23283f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T08:25:24.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.979+0000 7f2c22fff700 1 Processor -- start 2024-01-11T08:25:24.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.979+0000 7f2c22fff700 1 -- start start 2024-01-11T08:25:24.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.980+0000 7f2c22fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2c1c157410 0x7f2c1c0ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:24.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.980+0000 7f2c22fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c1c1577e0 0x7f2c1c0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:24.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.980+0000 7f2c22fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c1c0699d0 con 0x7f2c1c069540 2024-01-11T08:25:24.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.980+0000 7f2c22fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c1c069cb0 con 0x7f2c1c1577e0 2024-01-11T08:25:24.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.980+0000 7f2c22fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c069f90 con 0x7f2c1c157410 2024-01-11T08:25:24.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.980+0000 7f2c1bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c1c1577e0 0x7f2c1c0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:24.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.981+0000 7f2c1bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c1c1577e0 0x7f2c1c0ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56860/0 (socket says 172.21.15.27:56860) 2024-01-11T08:25:24.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.981+0000 7f2c1bfff700 1 -- 172.21.15.27:0/3219596846 learned_addr learned my addr 172.21.15.27:0/3219596846 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:24.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.981+0000 7f2c20d9b700 1 --2- 172.21.15.27:0/3219596846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2c1c157410 0x7f2c1c0ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:24.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.981+0000 7f2c1bfff700 1 -- 172.21.15.27:0/3219596846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2c1c157410 msgr2=0x7f2c1c0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:24.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.981+0000 7f2c1bfff700 1 --2- 172.21.15.27:0/3219596846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2c1c157410 0x7f2c1c0ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.981+0000 7f2c1bfff700 1 -- 172.21.15.27:0/3219596846 >> v1:172.21.15.77:6789/0 conn(0x7f2c1c069540 legacy=0x7f2c1c15b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:24.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.981+0000 7f2c1bfff700 1 -- 172.21.15.27:0/3219596846 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1c06a270 con 0x7f2c1c1577e0 2024-01-11T08:25:24.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.981+0000 7f2c1bfff700 1 --2- 172.21.15.27:0/3219596846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c1c1577e0 0x7f2c1c0ae000 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f2c080099b0 tx=0x7f2c08009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2aa13a8ed3c926e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:24.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.982+0000 7f2c1b7fe700 1 -- 172.21.15.27:0/3219596846 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0800b860 con 0x7f2c1c1577e0 2024-01-11T08:25:24.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.982+0000 7f2c1b7fe700 1 -- 172.21.15.27:0/3219596846 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0800b9c0 con 0x7f2c1c1577e0 2024-01-11T08:25:24.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.982+0000 7f2c1b7fe700 1 -- 172.21.15.27:0/3219596846 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0800bd60 con 0x7f2c1c1577e0 2024-01-11T08:25:24.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.982+0000 7f2c22fff700 1 -- 172.21.15.27:0/3219596846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c1c1577e0 msgr2=0x7f2c1c0ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:24.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.982+0000 7f2c22fff700 1 --2- 172.21.15.27:0/3219596846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c1c1577e0 0x7f2c1c0ae000 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f2c080099b0 tx=0x7f2c08009250 comp rx=0 tx=0).stop 2024-01-11T08:25:24.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.982+0000 7f2c22fff700 1 -- 172.21.15.27:0/3219596846 shutdown_connections 2024-01-11T08:25:24.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.982+0000 7f2c22fff700 1 --2- 172.21.15.27:0/3219596846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2c1c157410 0x7f2c1c0ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.982+0000 7f2c22fff700 1 --2- 172.21.15.27:0/3219596846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c1c1577e0 0x7f2c1c0ae000 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.982+0000 7f2c22fff700 1 -- 172.21.15.27:0/3219596846 >> 172.21.15.27:0/3219596846 conn(0x7f2c1c0680a0 msgr2=0x7f2c1c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:24.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.982+0000 7f2c22fff700 1 -- 172.21.15.27:0/3219596846 shutdown_connections 2024-01-11T08:25:24.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.982+0000 7f2c22fff700 1 -- 172.21.15.27:0/3219596846 wait complete. 2024-01-11T08:25:24.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.984+0000 7f2c22fff700 1 Processor -- start 2024-01-11T08:25:24.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.984+0000 7f2c22fff700 1 -- start start 2024-01-11T08:25:24.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.985+0000 7f2c22fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c1c069540 0x7f2c1c0b0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:24.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.985+0000 7f2c22fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c1c157410 0x7f2c1c15bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:24.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.985+0000 7f2c22fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2c1c1577e0 0x7f2c1c15c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:24.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.985+0000 7f2c22fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0caaa0 con 0x7f2c1c157410 2024-01-11T08:25:24.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.985+0000 7f2c22fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0c6120 con 0x7f2c1c069540 2024-01-11T08:25:24.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.985+0000 7f2c22fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0b0f10 con 0x7f2c1c1577e0 2024-01-11T08:25:24.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.985+0000 7f2c1bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c1c157410 0x7f2c1c15bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:24.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.985+0000 7f2c1bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c1c157410 0x7f2c1c15bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56868/0 (socket says 172.21.15.27:56868) 2024-01-11T08:25:24.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.985+0000 7f2c1bfff700 1 -- 172.21.15.27:0/4213825334 learned_addr learned my addr 172.21.15.27:0/4213825334 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:24.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.985+0000 7f2c2159c700 1 --2- 172.21.15.27:0/4213825334 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2c1c1577e0 0x7f2c1c15c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:24.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.985+0000 7f2c20d9b700 1 --2- 172.21.15.27:0/4213825334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c1c069540 0x7f2c1c0b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:24.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.985+0000 7f2c1bfff700 1 -- 172.21.15.27:0/4213825334 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2c1c1577e0 msgr2=0x7f2c1c15c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:24.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.985+0000 7f2c1bfff700 1 --2- 172.21.15.27:0/4213825334 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2c1c1577e0 0x7f2c1c15c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.985+0000 7f2c1bfff700 1 -- 172.21.15.27:0/4213825334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c1c069540 msgr2=0x7f2c1c0b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:24.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.985+0000 7f2c1bfff700 1 --2- 172.21.15.27:0/4213825334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c1c069540 0x7f2c1c0b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:24.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.986+0000 7f2c1bfff700 1 -- 172.21.15.27:0/4213825334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c080094f0 con 0x7f2c1c157410 2024-01-11T08:25:24.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.986+0000 7f2c1bfff700 1 --2- 172.21.15.27:0/4213825334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c1c157410 0x7f2c1c15bfc0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009290 tx=0x7f2c08017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:24.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.986+0000 7f2c18ff9700 1 -- 172.21.15.27:0/4213825334 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c08024440 con 0x7f2c1c157410 2024-01-11T08:25:24.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.986+0000 7f2c18ff9700 1 -- 172.21.15.27:0/4213825334 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c080245c0 con 0x7f2c1c157410 2024-01-11T08:25:24.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.986+0000 7f2c22fff700 1 -- 172.21.15.27:0/4213825334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1c0b11f0 con 0x7f2c1c157410 2024-01-11T08:25:24.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.986+0000 7f2c18ff9700 1 -- 172.21.15.27:0/4213825334 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c08024980 con 0x7f2c1c157410 2024-01-11T08:25:24.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.986+0000 7f2c22fff700 1 -- 172.21.15.27:0/4213825334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c1c0b17b0 con 0x7f2c1c157410 2024-01-11T08:25:24.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.987+0000 7f2c22fff700 1 -- 172.21.15.27:0/4213825334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c1c04c3c0 con 0x7f2c1c157410 2024-01-11T08:25:24.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.988+0000 7f2c18ff9700 1 -- 172.21.15.27:0/4213825334 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2c0801a420 con 0x7f2c1c157410 2024-01-11T08:25:24.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.990+0000 7f2c18ff9700 1 --2- 172.21.15.27:0/4213825334 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2c00066f30 0x7f2c000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:24.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.990+0000 7f2c18ff9700 1 -- 172.21.15.27:0/4213825334 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f2c08086370 con 0x7f2c1c157410 2024-01-11T08:25:24.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.990+0000 7f2c20d9b700 1 --2- 172.21.15.27:0/4213825334 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2c00066f30 0x7f2c000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:24.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.991+0000 7f2c20d9b700 1 --2- 172.21.15.27:0/4213825334 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2c00066f30 0x7f2c000693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2c10006aa0 tx=0x7f2c10008040 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:24.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:24.994+0000 7f2c18ff9700 1 -- 172.21.15.27:0/4213825334 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2c08053520 con 0x7f2c1c157410 2024-01-11T08:25:25.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:25.244+0000 7f2c22fff700 1 -- 172.21.15.27:0/4213825334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2c1c0cabe0 con 0x7f2c1c157410 2024-01-11T08:25:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:25.866+0000 7f2c18ff9700 1 -- 172.21.15.27:0/4213825334 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v32) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2c0805c070 con 0x7f2c1c157410 2024-01-11T08:25:25.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:25.869+0000 7f2c22fff700 1 -- 172.21.15.27:0/4213825334 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2c00066f30 msgr2=0x7f2c000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:25.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:25.869+0000 7f2c22fff700 1 --2- 172.21.15.27:0/4213825334 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2c00066f30 0x7f2c000693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2c10006aa0 tx=0x7f2c10008040 comp rx=0 tx=0).stop 2024-01-11T08:25:25.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:25.869+0000 7f2c22fff700 1 -- 172.21.15.27:0/4213825334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c1c157410 msgr2=0x7f2c1c15bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:25.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:25.869+0000 7f2c22fff700 1 --2- 172.21.15.27:0/4213825334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c1c157410 0x7f2c1c15bfc0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009290 tx=0x7f2c08017000 comp rx=0 tx=0).stop 2024-01-11T08:25:25.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:25.869+0000 7f2c22fff700 1 -- 172.21.15.27:0/4213825334 shutdown_connections 2024-01-11T08:25:25.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:25.869+0000 7f2c22fff700 1 --2- 172.21.15.27:0/4213825334 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2c00066f30 0x7f2c000693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:25.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:25.869+0000 7f2c22fff700 1 --2- 172.21.15.27:0/4213825334 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2c1c1577e0 0x7f2c1c15c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:25.869+0000 7f2c22fff700 1 --2- 172.21.15.27:0/4213825334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c1c069540 0x7f2c1c0b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:25.869+0000 7f2c22fff700 1 --2- 172.21.15.27:0/4213825334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c1c157410 0x7f2c1c15bfc0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:25.869+0000 7f2c22fff700 1 -- 172.21.15.27:0/4213825334 >> 172.21.15.27:0/4213825334 conn(0x7f2c1c0680a0 msgr2=0x7f2c1c0c7970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:25.869+0000 7f2c22fff700 1 -- 172.21.15.27:0/4213825334 shutdown_connections 2024-01-11T08:25:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:25.869+0000 7f2c22fff700 1 -- 172.21.15.27:0/4213825334 wait complete. 2024-01-11T08:25:25.889 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-11T08:25:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.096+0000 7f663dbff700 1 Processor -- start 2024-01-11T08:25:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.096+0000 7f663dbff700 1 -- start start 2024-01-11T08:25:26.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.097+0000 7f663dbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66380cd220 0x7f663815d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:26.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.097+0000 7f663dbff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f663815d860 0x7f66381591e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:26.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.097+0000 7f663dbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f66380cd5f0 0x7f6638159720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:26.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.097+0000 7f663dbff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f663815aef0 con 0x7f663815d860 2024-01-11T08:25:26.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.097+0000 7f663dbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66380cee00 con 0x7f66380cd5f0 2024-01-11T08:25:26.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.097+0000 7f663dbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f663815dd70 con 0x7f66380cd220 2024-01-11T08:25:26.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.097+0000 7f6636ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f663815d860 0x7f66381591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:26.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.098+0000 7f6636ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f663815d860 0x7f66381591e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56884/0 (socket says 172.21.15.27:56884) 2024-01-11T08:25:26.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.098+0000 7f6636ffd700 1 -- 172.21.15.27:0/4259007413 learned_addr learned my addr 172.21.15.27:0/4259007413 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:26.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.098+0000 7f6637fff700 1 --2- 172.21.15.27:0/4259007413 >> v2:172.21.15.77:3300/0 conn(0x7f66380cd5f0 0x7f6638159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.098+0000 7f66377fe700 1 --2- 172.21.15.27:0/4259007413 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66380cd220 0x7f663815d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.098+0000 7f6636ffd700 1 -- 172.21.15.27:0/4259007413 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66380cd220 msgr2=0x7f663815d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.098+0000 7f6636ffd700 1 --2- 172.21.15.27:0/4259007413 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66380cd220 0x7f663815d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.098+0000 7f6636ffd700 1 -- 172.21.15.27:0/4259007413 >> v2:172.21.15.77:3300/0 conn(0x7f66380cd5f0 msgr2=0x7f6638159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.098+0000 7f6636ffd700 1 --2- 172.21.15.27:0/4259007413 >> v2:172.21.15.77:3300/0 conn(0x7f66380cd5f0 0x7f6638159720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.098+0000 7f6636ffd700 1 -- 172.21.15.27:0/4259007413 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663815e050 con 0x7f663815d860 2024-01-11T08:25:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.098+0000 7f6637fff700 1 --2- 172.21.15.27:0/4259007413 >> v2:172.21.15.77:3300/0 conn(0x7f66380cd5f0 0x7f6638159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.098+0000 7f6636ffd700 1 --2- 172.21.15.27:0/4259007413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f663815d860 0x7f66381591e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f66380ce330 tx=0x7f662c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5372a05c47330c6f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.098+0000 7f66367fc700 1 -- 172.21.15.27:0/4259007413 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c00b860 con 0x7f663815d860 2024-01-11T08:25:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.098+0000 7f66367fc700 1 -- 172.21.15.27:0/4259007413 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662c00b9c0 con 0x7f663815d860 2024-01-11T08:25:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.098+0000 7f66367fc700 1 -- 172.21.15.27:0/4259007413 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c00bd60 con 0x7f663815d860 2024-01-11T08:25:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.099+0000 7f663dbff700 1 -- 172.21.15.27:0/4259007413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f663815d860 msgr2=0x7f66381591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.099+0000 7f663dbff700 1 --2- 172.21.15.27:0/4259007413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f663815d860 0x7f66381591e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f66380ce330 tx=0x7f662c009250 comp rx=0 tx=0).stop 2024-01-11T08:25:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.099+0000 7f663dbff700 1 -- 172.21.15.27:0/4259007413 shutdown_connections 2024-01-11T08:25:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.099+0000 7f663dbff700 1 --2- 172.21.15.27:0/4259007413 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66380cd220 0x7f663815d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.099+0000 7f663dbff700 1 --2- 172.21.15.27:0/4259007413 >> v2:172.21.15.77:3300/0 conn(0x7f66380cd5f0 0x7f6638159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.099+0000 7f663dbff700 1 --2- 172.21.15.27:0/4259007413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f663815d860 0x7f66381591e0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.099+0000 7f663dbff700 1 -- 172.21.15.27:0/4259007413 >> 172.21.15.27:0/4259007413 conn(0x7f663806ed40 msgr2=0x7f663806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.099+0000 7f663dbff700 1 -- 172.21.15.27:0/4259007413 shutdown_connections 2024-01-11T08:25:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.099+0000 7f663dbff700 1 -- 172.21.15.27:0/4259007413 wait complete. 2024-01-11T08:25:26.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.101+0000 7f663dbff700 1 Processor -- start 2024-01-11T08:25:26.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.101+0000 7f663dbff700 1 -- start start 2024-01-11T08:25:26.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.102+0000 7f663dbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66380cd220 0x7f6638152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.102+0000 7f663dbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66380cd5f0 0x7f66380b3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.102+0000 7f663dbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f663815d860 0x7f66380b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.102+0000 7f663dbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66380b1d60 con 0x7f66380cd220 2024-01-11T08:25:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.102+0000 7f663dbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6638152750 con 0x7f66380cd5f0 2024-01-11T08:25:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.102+0000 7f663dbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66380b28a0 con 0x7f663815d860 2024-01-11T08:25:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.102+0000 7f66377fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66380cd220 0x7f6638152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.102+0000 7f66377fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66380cd220 0x7f6638152210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56898/0 (socket says 172.21.15.27:56898) 2024-01-11T08:25:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.102+0000 7f66377fe700 1 -- 172.21.15.27:0/995650902 learned_addr learned my addr 172.21.15.27:0/995650902 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.102+0000 7f66377fe700 1 -- 172.21.15.27:0/995650902 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f663815d860 msgr2=0x7f66380b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.102+0000 7f66377fe700 1 --2- 172.21.15.27:0/995650902 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f663815d860 0x7f66380b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.102+0000 7f66377fe700 1 -- 172.21.15.27:0/995650902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66380cd5f0 msgr2=0x7f66380b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.102+0000 7f66377fe700 1 --2- 172.21.15.27:0/995650902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66380cd5f0 0x7f66380b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.102+0000 7f66377fe700 1 -- 172.21.15.27:0/995650902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662c0094f0 con 0x7f66380cd220 2024-01-11T08:25:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.103+0000 7f66377fe700 1 --2- 172.21.15.27:0/995650902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66380cd220 0x7f6638152210 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f662800d890 tx=0x7f6628008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:26.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.103+0000 7f661f7fe700 1 -- 172.21.15.27:0/995650902 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66280125f0 con 0x7f66380cd220 2024-01-11T08:25:26.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.103+0000 7f661f7fe700 1 -- 172.21.15.27:0/995650902 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6628012750 con 0x7f66380cd220 2024-01-11T08:25:26.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.103+0000 7f661f7fe700 1 -- 172.21.15.27:0/995650902 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6628012b10 con 0x7f66380cd220 2024-01-11T08:25:26.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.103+0000 7f663dbff700 1 -- 172.21.15.27:0/995650902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66380b2d30 con 0x7f66380cd220 2024-01-11T08:25:26.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.103+0000 7f663dbff700 1 -- 172.21.15.27:0/995650902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66380b1f70 con 0x7f66380cd220 2024-01-11T08:25:26.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.104+0000 7f663dbff700 1 -- 172.21.15.27:0/995650902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663804c3c0 con 0x7f66380cd220 2024-01-11T08:25:26.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.105+0000 7f661f7fe700 1 -- 172.21.15.27:0/995650902 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f662801e460 con 0x7f66380cd220 2024-01-11T08:25:26.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.106+0000 7f661f7fe700 1 --2- 172.21.15.27:0/995650902 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6624066f30 0x7f66240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:26.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.106+0000 7f661f7fe700 1 -- 172.21.15.27:0/995650902 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f6628084b30 con 0x7f66380cd220 2024-01-11T08:25:26.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.106+0000 7f6636ffd700 1 --2- 172.21.15.27:0/995650902 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6624066f30 0x7f66240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:26.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.107+0000 7f6636ffd700 1 --2- 172.21.15.27:0/995650902 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6624066f30 0x7f66240693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f66380ce330 tx=0x7f662c017000 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:26.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.110+0000 7f661f7fe700 1 -- 172.21.15.27:0/995650902 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6628055810 con 0x7f66380cd220 2024-01-11T08:25:26.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.362+0000 7f663dbff700 1 -- 172.21.15.27:0/995650902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f66380ce380 con 0x7f66380cd220 2024-01-11T08:25:26.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.875+0000 7f661f7fe700 1 -- 172.21.15.27:0/995650902 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v33) v1 ==== 122+0+0 (secure 0 0 0) 0x7f66280553a0 con 0x7f66380cd220 2024-01-11T08:25:26.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.877+0000 7f663dbff700 1 -- 172.21.15.27:0/995650902 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6624066f30 msgr2=0x7f66240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:26.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.877+0000 7f663dbff700 1 --2- 172.21.15.27:0/995650902 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6624066f30 0x7f66240693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f66380ce330 tx=0x7f662c017000 comp rx=0 tx=0).stop 2024-01-11T08:25:26.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.877+0000 7f663dbff700 1 -- 172.21.15.27:0/995650902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66380cd220 msgr2=0x7f6638152210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:26.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.877+0000 7f663dbff700 1 --2- 172.21.15.27:0/995650902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66380cd220 0x7f6638152210 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f662800d890 tx=0x7f6628008040 comp rx=0 tx=0).stop 2024-01-11T08:25:26.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.877+0000 7f663dbff700 1 -- 172.21.15.27:0/995650902 shutdown_connections 2024-01-11T08:25:26.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.877+0000 7f663dbff700 1 --2- 172.21.15.27:0/995650902 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f6624066f30 0x7f66240693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:26.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.877+0000 7f663dbff700 1 --2- 172.21.15.27:0/995650902 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f663815d860 0x7f66380b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:26.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.877+0000 7f663dbff700 1 --2- 172.21.15.27:0/995650902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66380cd220 0x7f6638152210 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:26.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.877+0000 7f663dbff700 1 --2- 172.21.15.27:0/995650902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66380cd5f0 0x7f66380b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:26.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.877+0000 7f663dbff700 1 -- 172.21.15.27:0/995650902 >> 172.21.15.27:0/995650902 conn(0x7f663806ed40 msgr2=0x7f6638152dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:26.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.878+0000 7f663dbff700 1 -- 172.21.15.27:0/995650902 shutdown_connections 2024-01-11T08:25:26.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:26.878+0000 7f663dbff700 1 -- 172.21.15.27:0/995650902 wait complete. 2024-01-11T08:25:26.897 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:25:27.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.097+0000 7fac6cfff700 1 Processor -- start 2024-01-11T08:25:27.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.098+0000 7fac6cfff700 1 -- start start 2024-01-11T08:25:27.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.099+0000 7fac6cfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fac680b03e0 0x7fac680b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:27.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.099+0000 7fac6cfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fac680b29c0 0x7fac68159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:27.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.099+0000 7fac6cfff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac680cd730 con 0x7fac680b07b0 2024-01-11T08:25:27.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.099+0000 7fac6cfff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac680b2de0 con 0x7fac680b03e0 2024-01-11T08:25:27.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.099+0000 7fac6cfff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac680ccb20 con 0x7fac680b29c0 2024-01-11T08:25:27.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.099+0000 7fac6659c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fac680b03e0 0x7fac680b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:27.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.099+0000 7fac6659c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fac680b03e0 0x7fac680b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56902/0 (socket says 172.21.15.27:56902) 2024-01-11T08:25:27.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.099+0000 7fac6659c700 1 -- 172.21.15.27:0/2181186180 learned_addr learned my addr 172.21.15.27:0/2181186180 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:27.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.100+0000 7fac66d9d700 1 --2- 172.21.15.27:0/2181186180 >> v2:172.21.15.77:3300/0 conn(0x7fac680b29c0 0x7fac68159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:27.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.100+0000 7fac6659c700 1 -- 172.21.15.27:0/2181186180 >> v2:172.21.15.77:3300/0 conn(0x7fac680b29c0 msgr2=0x7fac68159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:27.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.100+0000 7fac6659c700 1 --2- 172.21.15.27:0/2181186180 >> v2:172.21.15.77:3300/0 conn(0x7fac680b29c0 0x7fac68159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:27.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.100+0000 7fac6659c700 1 -- 172.21.15.27:0/2181186180 >> v1:172.21.15.77:6789/0 conn(0x7fac680b07b0 legacy=0x7fac68159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:27.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.100+0000 7fac6659c700 1 -- 172.21.15.27:0/2181186180 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac680cce00 con 0x7fac680b03e0 2024-01-11T08:25:27.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.100+0000 7fac6659c700 1 --2- 172.21.15.27:0/2181186180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680b03e0 0x7fac680b2480 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fac680b1d10 tx=0x7fac58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d18b77c6f0c1bc3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:27.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.100+0000 7fac6559a700 1 -- 172.21.15.27:0/2181186180 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5800b820 con 0x7fac680b03e0 2024-01-11T08:25:27.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.100+0000 7fac6559a700 1 -- 172.21.15.27:0/2181186180 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5800b980 con 0x7fac680b03e0 2024-01-11T08:25:27.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.100+0000 7fac6559a700 1 -- 172.21.15.27:0/2181186180 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5800bd20 con 0x7fac680b03e0 2024-01-11T08:25:27.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.101+0000 7fac6cfff700 1 -- 172.21.15.27:0/2181186180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680b03e0 msgr2=0x7fac680b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:27.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.101+0000 7fac6cfff700 1 --2- 172.21.15.27:0/2181186180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680b03e0 0x7fac680b2480 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fac680b1d10 tx=0x7fac58009250 comp rx=0 tx=0).stop 2024-01-11T08:25:27.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.101+0000 7fac6cfff700 1 -- 172.21.15.27:0/2181186180 shutdown_connections 2024-01-11T08:25:27.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.101+0000 7fac6cfff700 1 --2- 172.21.15.27:0/2181186180 >> v2:172.21.15.77:3300/0 conn(0x7fac680b29c0 0x7fac68159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:27.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.101+0000 7fac6cfff700 1 --2- 172.21.15.27:0/2181186180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680b03e0 0x7fac680b2480 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:27.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.101+0000 7fac6cfff700 1 -- 172.21.15.27:0/2181186180 >> 172.21.15.27:0/2181186180 conn(0x7fac6806ed40 msgr2=0x7fac6806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:27.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.101+0000 7fac6cfff700 1 -- 172.21.15.27:0/2181186180 shutdown_connections 2024-01-11T08:25:27.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.101+0000 7fac6cfff700 1 -- 172.21.15.27:0/2181186180 wait complete. 2024-01-11T08:25:27.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.102+0000 7fac6cfff700 1 Processor -- start 2024-01-11T08:25:27.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.103+0000 7fac6cfff700 1 -- start start 2024-01-11T08:25:27.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.103+0000 7fac6cfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680b03e0 0x7fac68169e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:27.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.104+0000 7fac6cfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac680b07b0 0x7fac680c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:27.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.104+0000 7fac6cfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac680b29c0 0x7fac680ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:27.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.104+0000 7fac6cfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac680afd50 con 0x7fac680b03e0 2024-01-11T08:25:27.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.104+0000 7fac6cfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6816ae10 con 0x7fac680b29c0 2024-01-11T08:25:27.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.104+0000 7fac6cfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac680c5340 con 0x7fac680b07b0 2024-01-11T08:25:27.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.104+0000 7fac6659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680b03e0 0x7fac68169e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:27.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.104+0000 7fac6659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680b03e0 0x7fac68169e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56908/0 (socket says 172.21.15.27:56908) 2024-01-11T08:25:27.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.104+0000 7fac6659c700 1 -- 172.21.15.27:0/1611043346 learned_addr learned my addr 172.21.15.27:0/1611043346 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:27.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.104+0000 7fac66d9d700 1 --2- 172.21.15.27:0/1611043346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac680b29c0 0x7fac680ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:27.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.104+0000 7fac65d9b700 1 --2- 172.21.15.27:0/1611043346 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac680b07b0 0x7fac680c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:27.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.104+0000 7fac6659c700 1 -- 172.21.15.27:0/1611043346 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac680b07b0 msgr2=0x7fac680c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:27.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.104+0000 7fac6659c700 1 --2- 172.21.15.27:0/1611043346 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac680b07b0 0x7fac680c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:27.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.104+0000 7fac6659c700 1 -- 172.21.15.27:0/1611043346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac680b29c0 msgr2=0x7fac680ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:27.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.104+0000 7fac6659c700 1 --2- 172.21.15.27:0/1611043346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac680b29c0 0x7fac680ce5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:27.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.104+0000 7fac6659c700 1 -- 172.21.15.27:0/1611043346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac580094f0 con 0x7fac680b03e0 2024-01-11T08:25:27.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.104+0000 7fac66d9d700 1 --2- 172.21.15.27:0/1611043346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac680b29c0 0x7fac680ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:27.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.105+0000 7fac6659c700 1 --2- 172.21.15.27:0/1611043346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680b03e0 0x7fac68169e30 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fac580030c0 tx=0x7fac58016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:27.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.105+0000 7fac567fc700 1 -- 172.21.15.27:0/1611043346 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac58011e60 con 0x7fac680b03e0 2024-01-11T08:25:27.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.105+0000 7fac567fc700 1 -- 172.21.15.27:0/1611043346 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5801b680 con 0x7fac680b03e0 2024-01-11T08:25:27.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.105+0000 7fac6cfff700 1 -- 172.21.15.27:0/1611043346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6816a850 con 0x7fac680b03e0 2024-01-11T08:25:27.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.105+0000 7fac567fc700 1 -- 172.21.15.27:0/1611043346 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5801ba20 con 0x7fac680b03e0 2024-01-11T08:25:27.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.105+0000 7fac6cfff700 1 -- 172.21.15.27:0/1611043346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac680c4fd0 con 0x7fac680b03e0 2024-01-11T08:25:27.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.106+0000 7fac6cfff700 1 -- 172.21.15.27:0/1611043346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac680c5620 con 0x7fac680b03e0 2024-01-11T08:25:27.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.107+0000 7fac567fc700 1 -- 172.21.15.27:0/1611043346 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fac5801c450 con 0x7fac680b03e0 2024-01-11T08:25:27.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.108+0000 7fac567fc700 1 --2- 172.21.15.27:0/1611043346 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fac48066f30 0x7fac480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:27.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.108+0000 7fac567fc700 1 -- 172.21.15.27:0/1611043346 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5733+0+0 (secure 0 0 0) 0x7fac58086350 con 0x7fac680b03e0 2024-01-11T08:25:27.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.109+0000 7fac65d9b700 1 --2- 172.21.15.27:0/1611043346 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fac48066f30 0x7fac480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:27.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.109+0000 7fac65d9b700 1 --2- 172.21.15.27:0/1611043346 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fac48066f30 0x7fac480693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fac680ca870 tx=0x7fac5c009040 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:27.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.112+0000 7fac567fc700 1 -- 172.21.15.27:0/1611043346 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fac58053500 con 0x7fac680b03e0 2024-01-11T08:25:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.363+0000 7fac6cfff700 1 -- 172.21.15.27:0/1611043346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac680718d0 con 0x7fac680b03e0 2024-01-11T08:25:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.363+0000 7fac567fc700 1 -- 172.21.15.27:0/1611043346 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4597 (secure 0 0 0) 0x7fac58056b20 con 0x7fac680b03e0 2024-01-11T08:25:27.363 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:27.364 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5606,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2845124438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2845124438},{"type":"v1","addr":"172.21.15.77:6837","nonce":2845124438}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5618,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3958104902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3958104902},{"type":"v1","addr":"172.21.15.27:6835","nonce":3958104902}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":6091,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/2612413022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2612413022},{"type":"v1","addr":"172.21.15.27:6833","nonce":2612413022}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:25:23.345532+0000","modified":"2024-01-11T08:25:26.867139+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6174},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6174":{"gid":6174,"name":"d","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6836/3056318454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3056318454},{"type":"v1","addr":"172.21.15.77:6836","nonce":3056318454}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2024-01-11T08:25:27.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.366+0000 7fac6cfff700 1 -- 172.21.15.27:0/1611043346 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fac48066f30 msgr2=0x7fac480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:27.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.366+0000 7fac6cfff700 1 --2- 172.21.15.27:0/1611043346 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fac48066f30 0x7fac480693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fac680ca870 tx=0x7fac5c009040 comp rx=0 tx=0).stop 2024-01-11T08:25:27.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.366+0000 7fac6cfff700 1 -- 172.21.15.27:0/1611043346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680b03e0 msgr2=0x7fac68169e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:27.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.366+0000 7fac6cfff700 1 --2- 172.21.15.27:0/1611043346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680b03e0 0x7fac68169e30 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fac580030c0 tx=0x7fac58016000 comp rx=0 tx=0).stop 2024-01-11T08:25:27.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.366+0000 7fac6cfff700 1 -- 172.21.15.27:0/1611043346 shutdown_connections 2024-01-11T08:25:27.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.366+0000 7fac6cfff700 1 --2- 172.21.15.27:0/1611043346 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fac48066f30 0x7fac480693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:27.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.366+0000 7fac6cfff700 1 --2- 172.21.15.27:0/1611043346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac680b29c0 0x7fac680ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:27.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.366+0000 7fac6cfff700 1 --2- 172.21.15.27:0/1611043346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680b03e0 0x7fac68169e30 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:27.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.366+0000 7fac6cfff700 1 --2- 172.21.15.27:0/1611043346 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac680b07b0 0x7fac680c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:27.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.366+0000 7fac6cfff700 1 -- 172.21.15.27:0/1611043346 >> 172.21.15.27:0/1611043346 conn(0x7fac6806ed40 msgr2=0x7fac680b4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:27.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.366+0000 7fac6cfff700 1 -- 172.21.15.27:0/1611043346 shutdown_connections 2024-01-11T08:25:27.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.366+0000 7fac6cfff700 1 -- 172.21.15.27:0/1611043346 wait complete. 2024-01-11T08:25:27.367 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 33 2024-01-11T08:25:27.384 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:25:27.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.583+0000 7f0480fff700 1 Processor -- start 2024-01-11T08:25:27.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.583+0000 7f0480fff700 1 -- start start 2024-01-11T08:25:27.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.585+0000 7f0480fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f047c159db0 0x7f047c15a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:27.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.585+0000 7f0480fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f047c14fc20 0x7f047c150080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:27.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.585+0000 7f0480fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f047c157e30 con 0x7f047c1599e0 2024-01-11T08:25:27.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.585+0000 7f0480fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f047c157500 con 0x7f047c14fc20 2024-01-11T08:25:27.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.585+0000 7f0480fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f047c1577e0 con 0x7f047c159db0 2024-01-11T08:25:27.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.585+0000 7f047ad9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f047c14fc20 0x7f047c150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:27.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.585+0000 7f047ad9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f047c14fc20 0x7f047c150080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56918/0 (socket says 172.21.15.27:56918) 2024-01-11T08:25:27.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.585+0000 7f047ad9d700 1 -- 172.21.15.27:0/1267009429 learned_addr learned my addr 172.21.15.27:0/1267009429 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:27.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.585+0000 7f0479d9b700 1 --2- 172.21.15.27:0/1267009429 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f047c159db0 0x7f047c15a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:27.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.586+0000 7f047ad9d700 1 -- 172.21.15.27:0/1267009429 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f047c159db0 msgr2=0x7f047c15a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:27.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.586+0000 7f047ad9d700 1 --2- 172.21.15.27:0/1267009429 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f047c159db0 0x7f047c15a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:27.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.586+0000 7f047ad9d700 1 -- 172.21.15.27:0/1267009429 >> v1:172.21.15.77:6789/0 conn(0x7f047c1599e0 legacy=0x7f047c0b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:27.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.586+0000 7f047ad9d700 1 -- 172.21.15.27:0/1267009429 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f047c157ac0 con 0x7f047c14fc20 2024-01-11T08:25:27.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.586+0000 7f047ad9d700 1 --2- 172.21.15.27:0/1267009429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f047c14fc20 0x7f047c150080 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0474009f20 tx=0x7f0474009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aacbd2df2bd5cdd8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:27.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.586+0000 7f047959a700 1 -- 172.21.15.27:0/1267009429 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047400e070 con 0x7f047c14fc20 2024-01-11T08:25:27.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.586+0000 7f047959a700 1 -- 172.21.15.27:0/1267009429 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047400b9d0 con 0x7f047c14fc20 2024-01-11T08:25:27.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.586+0000 7f047959a700 1 -- 172.21.15.27:0/1267009429 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04740092e0 con 0x7f047c14fc20 2024-01-11T08:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.587+0000 7f0480fff700 1 -- 172.21.15.27:0/1267009429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f047c14fc20 msgr2=0x7f047c150080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.587+0000 7f0480fff700 1 --2- 172.21.15.27:0/1267009429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f047c14fc20 0x7f047c150080 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0474009f20 tx=0x7f0474009840 comp rx=0 tx=0).stop 2024-01-11T08:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.587+0000 7f0480fff700 1 -- 172.21.15.27:0/1267009429 shutdown_connections 2024-01-11T08:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.587+0000 7f0480fff700 1 --2- 172.21.15.27:0/1267009429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f047c14fc20 0x7f047c150080 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.587+0000 7f0480fff700 1 --2- 172.21.15.27:0/1267009429 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f047c159db0 0x7f047c15a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.587+0000 7f0480fff700 1 -- 172.21.15.27:0/1267009429 >> 172.21.15.27:0/1267009429 conn(0x7f047c0680e0 msgr2=0x7f047c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.587+0000 7f0480fff700 1 -- 172.21.15.27:0/1267009429 shutdown_connections 2024-01-11T08:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.587+0000 7f0480fff700 1 -- 172.21.15.27:0/1267009429 wait complete. 2024-01-11T08:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.588+0000 7f0480fff700 1 Processor -- start 2024-01-11T08:25:27.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.589+0000 7f0480fff700 1 -- start start 2024-01-11T08:25:27.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.590+0000 7f0480fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f047c14fc20 0x7f047c0ac460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:27.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.590+0000 7f0480fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f047c1599e0 0x7f047c0adf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:27.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.590+0000 7f0480fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f047c159db0 0x7f047c148c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:27.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.590+0000 7f0480fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047c153df0 con 0x7f047c14fc20 2024-01-11T08:25:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.590+0000 7f0480fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047c0acd20 con 0x7f047c159db0 2024-01-11T08:25:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.590+0000 7f0480fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f047c0ad000 con 0x7f047c1599e0 2024-01-11T08:25:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.590+0000 7f047a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f047c14fc20 0x7f047c0ac460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.590+0000 7f047a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f047c14fc20 0x7f047c0ac460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56924/0 (socket says 172.21.15.27:56924) 2024-01-11T08:25:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.590+0000 7f047a59c700 1 -- 172.21.15.27:0/671044471 learned_addr learned my addr 172.21.15.27:0/671044471 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.590+0000 7f047ad9d700 1 --2- 172.21.15.27:0/671044471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f047c159db0 0x7f047c148c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.590+0000 7f0479d9b700 1 --2- 172.21.15.27:0/671044471 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f047c1599e0 0x7f047c0adf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.590+0000 7f047a59c700 1 -- 172.21.15.27:0/671044471 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f047c1599e0 msgr2=0x7f047c0adf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.590+0000 7f047a59c700 1 --2- 172.21.15.27:0/671044471 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f047c1599e0 0x7f047c0adf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.590+0000 7f047a59c700 1 -- 172.21.15.27:0/671044471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f047c159db0 msgr2=0x7f047c148c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.590+0000 7f047a59c700 1 --2- 172.21.15.27:0/671044471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f047c159db0 0x7f047c148c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.590+0000 7f047a59c700 1 -- 172.21.15.27:0/671044471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0474009ae0 con 0x7f047c14fc20 2024-01-11T08:25:27.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.591+0000 7f047a59c700 1 --2- 172.21.15.27:0/671044471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f047c14fc20 0x7f047c0ac460 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f046c00c590 tx=0x7f046c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:27.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.591+0000 7f046a7fc700 1 -- 172.21.15.27:0/671044471 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c011590 con 0x7f047c14fc20 2024-01-11T08:25:27.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.591+0000 7f046a7fc700 1 -- 172.21.15.27:0/671044471 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046c0116f0 con 0x7f047c14fc20 2024-01-11T08:25:27.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.591+0000 7f046a7fc700 1 -- 172.21.15.27:0/671044471 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c011ab0 con 0x7f047c14fc20 2024-01-11T08:25:27.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.591+0000 7f0480fff700 1 -- 172.21.15.27:0/671044471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f047c0ad2e0 con 0x7f047c14fc20 2024-01-11T08:25:27.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.591+0000 7f0480fff700 1 -- 172.21.15.27:0/671044471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f047c0ac9a0 con 0x7f047c14fc20 2024-01-11T08:25:27.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.592+0000 7f0480fff700 1 -- 172.21.15.27:0/671044471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f047c0718d0 con 0x7f047c14fc20 2024-01-11T08:25:27.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.593+0000 7f046a7fc700 1 -- 172.21.15.27:0/671044471 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f046c01b3b0 con 0x7f047c14fc20 2024-01-11T08:25:27.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.594+0000 7f046a7fc700 1 --2- 172.21.15.27:0/671044471 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f045c066f30 0x7f045c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:27.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.594+0000 7f046a7fc700 1 -- 172.21.15.27:0/671044471 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f046c083ba0 con 0x7f047c14fc20 2024-01-11T08:25:27.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.595+0000 7f0479d9b700 1 --2- 172.21.15.27:0/671044471 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f045c066f30 0x7f045c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:27.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.595+0000 7f0479d9b700 1 --2- 172.21.15.27:0/671044471 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f045c066f30 0x7f045c0693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f047c156a40 tx=0x7f0470008040 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:27.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.598+0000 7f046a7fc700 1 -- 172.21.15.27:0/671044471 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f046c050d50 con 0x7f047c14fc20 2024-01-11T08:25:27.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.800+0000 7f0480fff700 1 -- 172.21.15.27:0/671044471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f047c150810 con 0x7f047c14fc20 2024-01-11T08:25:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.801+0000 7f046a7fc700 1 -- 172.21.15.27:0/671044471 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+15345 (secure 0 0 0) 0x7f046c017020 con 0x7f047c14fc20 2024-01-11T08:25:27.801 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:27.801 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":75,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:25:24.849990+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:25:21.160295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-11T08:25:22.342780+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:25:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.803+0000 7f0480fff700 1 -- 172.21.15.27:0/671044471 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f045c066f30 msgr2=0x7f045c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.803+0000 7f0480fff700 1 --2- 172.21.15.27:0/671044471 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f045c066f30 0x7f045c0693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f047c156a40 tx=0x7f0470008040 comp rx=0 tx=0).stop 2024-01-11T08:25:27.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.803+0000 7f0480fff700 1 -- 172.21.15.27:0/671044471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f047c14fc20 msgr2=0x7f047c0ac460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:27.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.803+0000 7f0480fff700 1 --2- 172.21.15.27:0/671044471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f047c14fc20 0x7f047c0ac460 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f046c00c590 tx=0x7f046c00c950 comp rx=0 tx=0).stop 2024-01-11T08:25:27.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.803+0000 7f0480fff700 1 -- 172.21.15.27:0/671044471 shutdown_connections 2024-01-11T08:25:27.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.803+0000 7f0480fff700 1 --2- 172.21.15.27:0/671044471 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f045c066f30 0x7f045c0693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:27.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.803+0000 7f0480fff700 1 --2- 172.21.15.27:0/671044471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f047c159db0 0x7f047c148c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:27.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.803+0000 7f0480fff700 1 --2- 172.21.15.27:0/671044471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f047c14fc20 0x7f047c0ac460 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:27.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.803+0000 7f0480fff700 1 --2- 172.21.15.27:0/671044471 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f047c1599e0 0x7f047c0adf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:27.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.803+0000 7f0480fff700 1 -- 172.21.15.27:0/671044471 >> 172.21.15.27:0/671044471 conn(0x7f047c0680e0 msgr2=0x7f047c06e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:27.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.803+0000 7f0480fff700 1 -- 172.21.15.27:0/671044471 shutdown_connections 2024-01-11T08:25:27.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:27.803+0000 7f0480fff700 1 -- 172.21.15.27:0/671044471 wait complete. 2024-01-11T08:25:27.821 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-11T08:25:27.822 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:25:28.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.023+0000 7f978fdff700 1 Processor -- start 2024-01-11T08:25:28.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.023+0000 7f978fdff700 1 -- start start 2024-01-11T08:25:28.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.025+0000 7f978fdff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9788069d40 0x7f97880b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:28.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.025+0000 7f978fdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97880c89f0 0x7f97880b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:28.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.025+0000 7f978fdff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9788157390 con 0x7f97880c8620 2024-01-11T08:25:28.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.025+0000 7f978fdff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97880adcd0 con 0x7f9788069d40 2024-01-11T08:25:28.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.025+0000 7f978fdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978806a200 con 0x7f97880c89f0 2024-01-11T08:25:28.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.025+0000 7f978db9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f97880c8620 0x7f97880af970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46360/0 (socket says 172.21.15.27:46360) 2024-01-11T08:25:28.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.025+0000 7f978db9b700 1 -- 172.21.15.27:0/3184536910 learned_addr learned my addr 172.21.15.27:0/3184536910 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.025+0000 7f978e39c700 1 --2- 172.21.15.27:0/3184536910 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97880c89f0 0x7f97880b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.025+0000 7f978d39a700 1 --2- 172.21.15.27:0/3184536910 >> v2:172.21.15.77:3300/0 conn(0x7f9788069d40 0x7f97880b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.025+0000 7f978cb99700 1 -- 172.21.15.27:0/3184536910 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869057301 0 0) 0x7f9788157390 con 0x7f97880c8620 2024-01-11T08:25:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.025+0000 7f978cb99700 1 -- 172.21.15.27:0/3184536910 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9774005780 con 0x7f97880c8620 2024-01-11T08:25:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.026+0000 7f978e39c700 1 -- 172.21.15.27:0/3184536910 >> v2:172.21.15.77:3300/0 conn(0x7f9788069d40 msgr2=0x7f97880b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:28.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.026+0000 7f978e39c700 1 --2- 172.21.15.27:0/3184536910 >> v2:172.21.15.77:3300/0 conn(0x7f9788069d40 0x7f97880b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.026+0000 7f978e39c700 1 -- 172.21.15.27:0/3184536910 >> v1:172.21.15.27:6789/0 conn(0x7f97880c8620 legacy=0x7f97880af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:28.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.026+0000 7f978e39c700 1 -- 172.21.15.27:0/3184536910 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9788156d40 con 0x7f97880c89f0 2024-01-11T08:25:28.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.026+0000 7f978d39a700 1 --2- 172.21.15.27:0/3184536910 >> v2:172.21.15.77:3300/0 conn(0x7f9788069d40 0x7f97880b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:28.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.026+0000 7f978e39c700 1 --2- 172.21.15.27:0/3184536910 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97880c89f0 0x7f97880b0810 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9780009e60 tx=0x7f9780009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eeb8be038aa67e84 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:28.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.026+0000 7f978cb99700 1 -- 172.21.15.27:0/3184536910 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978000e070 con 0x7f97880c89f0 2024-01-11T08:25:28.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.026+0000 7f978cb99700 1 -- 172.21.15.27:0/3184536910 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978000bab0 con 0x7f97880c89f0 2024-01-11T08:25:28.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.026+0000 7f978cb99700 1 -- 172.21.15.27:0/3184536910 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978000be70 con 0x7f97880c89f0 2024-01-11T08:25:28.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.027+0000 7f978fdff700 1 -- 172.21.15.27:0/3184536910 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97880c89f0 msgr2=0x7f97880b0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:28.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.027+0000 7f978fdff700 1 --2- 172.21.15.27:0/3184536910 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97880c89f0 0x7f97880b0810 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9780009e60 tx=0x7f9780009780 comp rx=0 tx=0).stop 2024-01-11T08:25:28.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.027+0000 7f978fdff700 1 -- 172.21.15.27:0/3184536910 shutdown_connections 2024-01-11T08:25:28.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.027+0000 7f978fdff700 1 --2- 172.21.15.27:0/3184536910 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97880c89f0 0x7f97880b0810 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.027+0000 7f978fdff700 1 --2- 172.21.15.27:0/3184536910 >> v2:172.21.15.77:3300/0 conn(0x7f9788069d40 0x7f97880b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.027+0000 7f978fdff700 1 -- 172.21.15.27:0/3184536910 >> 172.21.15.27:0/3184536910 conn(0x7f97880680e0 msgr2=0x7f9788069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:28.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.027+0000 7f978fdff700 1 -- 172.21.15.27:0/3184536910 shutdown_connections 2024-01-11T08:25:28.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.027+0000 7f978fdff700 1 -- 172.21.15.27:0/3184536910 wait complete. 2024-01-11T08:25:28.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.028+0000 7f978fdff700 1 Processor -- start 2024-01-11T08:25:28.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.029+0000 7f978fdff700 1 -- start start 2024-01-11T08:25:28.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.029+0000 7f978fdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9788069d40 0x7f97881528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:28.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.029+0000 7f978fdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97880c8620 0x7f97880af470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:28.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.030+0000 7f978fdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97880c89f0 0x7f97880af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:28.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.030+0000 7f978fdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97880caa60 con 0x7f9788069d40 2024-01-11T08:25:28.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.030+0000 7f978fdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97880c9e50 con 0x7f97880c8620 2024-01-11T08:25:28.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.030+0000 7f978fdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97880ca130 con 0x7f97880c89f0 2024-01-11T08:25:28.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.030+0000 7f978db9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9788069d40 0x7f97881528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:28.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.030+0000 7f978db9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9788069d40 0x7f97881528a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56940/0 (socket says 172.21.15.27:56940) 2024-01-11T08:25:28.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.030+0000 7f978db9b700 1 -- 172.21.15.27:0/2982626299 learned_addr learned my addr 172.21.15.27:0/2982626299 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:28.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.030+0000 7f978d39a700 1 --2- 172.21.15.27:0/2982626299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97880c8620 0x7f97880af470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:28.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.030+0000 7f978e39c700 1 --2- 172.21.15.27:0/2982626299 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97880c89f0 0x7f97880af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:28.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.030+0000 7f978db9b700 1 -- 172.21.15.27:0/2982626299 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97880c89f0 msgr2=0x7f97880af9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:28.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.030+0000 7f978db9b700 1 --2- 172.21.15.27:0/2982626299 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97880c89f0 0x7f97880af9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.030+0000 7f978db9b700 1 -- 172.21.15.27:0/2982626299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97880c8620 msgr2=0x7f97880af470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:28.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.030+0000 7f978db9b700 1 --2- 172.21.15.27:0/2982626299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97880c8620 0x7f97880af470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.030+0000 7f978db9b700 1 -- 172.21.15.27:0/2982626299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9778009420 con 0x7f9788069d40 2024-01-11T08:25:28.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.030+0000 7f978db9b700 1 --2- 172.21.15.27:0/2982626299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9788069d40 0x7f97881528a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f978400c7c0 tx=0x7f9784007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:28.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.031+0000 7f977dffb700 1 -- 172.21.15.27:0/2982626299 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784005b30 con 0x7f9788069d40 2024-01-11T08:25:28.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.031+0000 7f977dffb700 1 -- 172.21.15.27:0/2982626299 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9784005c90 con 0x7f9788069d40 2024-01-11T08:25:28.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.031+0000 7f978fdff700 1 -- 172.21.15.27:0/2982626299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9780009a20 con 0x7f9788069d40 2024-01-11T08:25:28.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.031+0000 7f977dffb700 1 -- 172.21.15.27:0/2982626299 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784007340 con 0x7f9788069d40 2024-01-11T08:25:28.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.031+0000 7f978fdff700 1 -- 172.21.15.27:0/2982626299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97880ca6f0 con 0x7f9788069d40 2024-01-11T08:25:28.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.032+0000 7f978fdff700 1 -- 172.21.15.27:0/2982626299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97880718d0 con 0x7f9788069d40 2024-01-11T08:25:28.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.033+0000 7f977dffb700 1 -- 172.21.15.27:0/2982626299 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f978401b400 con 0x7f9788069d40 2024-01-11T08:25:28.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.034+0000 7f977dffb700 1 --2- 172.21.15.27:0/2982626299 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f976c066f30 0x7f976c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:28.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.034+0000 7f977dffb700 1 -- 172.21.15.27:0/2982626299 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f9784084140 con 0x7f9788069d40 2024-01-11T08:25:28.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.034+0000 7f978d39a700 1 --2- 172.21.15.27:0/2982626299 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f976c066f30 0x7f976c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:28.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.035+0000 7f978d39a700 1 --2- 172.21.15.27:0/2982626299 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f976c066f30 0x7f976c0693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f97780093f0 tx=0x7f9778008040 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:28.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.038+0000 7f977dffb700 1 -- 172.21.15.27:0/2982626299 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9784051370 con 0x7f9788069d40 2024-01-11T08:25:28.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.236+0000 7f978fdff700 1 -- 172.21.15.27:0/2982626299 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f97880caba0 con 0x7f976c066f30 2024-01-11T08:25:28.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.247+0000 7f977dffb700 1 -- 172.21.15.27:0/2982626299 <== mgr.5450 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289029 (secure 0 0 0) 0x7f9778011a20 con 0x7f976c066f30 2024-01-11T08:25:28.251 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.251+0000 7f978fdff700 1 -- 172.21.15.27:0/2982626299 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f976c066f30 msgr2=0x7f976c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.251+0000 7f978fdff700 1 --2- 172.21.15.27:0/2982626299 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f976c066f30 0x7f976c0693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f97780093f0 tx=0x7f9778008040 comp rx=0 tx=0).stop 2024-01-11T08:25:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.251+0000 7f978fdff700 1 -- 172.21.15.27:0/2982626299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9788069d40 msgr2=0x7f97881528a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.251+0000 7f978fdff700 1 --2- 172.21.15.27:0/2982626299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9788069d40 0x7f97881528a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f978400c7c0 tx=0x7f9784007e00 comp rx=0 tx=0).stop 2024-01-11T08:25:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.251+0000 7f978fdff700 1 -- 172.21.15.27:0/2982626299 shutdown_connections 2024-01-11T08:25:28.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.251+0000 7f978fdff700 1 --2- 172.21.15.27:0/2982626299 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f976c066f30 0x7f976c0693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.251+0000 7f978fdff700 1 --2- 172.21.15.27:0/2982626299 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97880c89f0 0x7f97880af9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.251+0000 7f978fdff700 1 --2- 172.21.15.27:0/2982626299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9788069d40 0x7f97881528a0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.251+0000 7f978fdff700 1 --2- 172.21.15.27:0/2982626299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97880c8620 0x7f97880af470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.252+0000 7f978fdff700 1 -- 172.21.15.27:0/2982626299 >> 172.21.15.27:0/2982626299 conn(0x7f97880680e0 msgr2=0x7f97881541a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:28.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.252+0000 7f978fdff700 1 -- 172.21.15.27:0/2982626299 shutdown_connections 2024-01-11T08:25:28.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.252+0000 7f978fdff700 1 -- 172.21.15.27:0/2982626299 wait complete. 2024-01-11T08:25:28.255 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:25:28.269 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-11T08:25:27.972373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":120,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2431608,"kb_used_data":8248,"kb_used_omap":0,"kb_used_meta":2423296,"kb_avail":752543112,"statfs":{"total":773094113280,"available":770604146688,"internally_reserved":0,"allocated":8445952,"data_stored":4492396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2481455104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,9],"upper_bound":32},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001231"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.872924+0000","last_change":"2024-01-11T08:25:23.860736+0000","last_active":"2024-01-11T08:25:23.872924+0000","last_peered":"2024-01-11T08:25:23.872924+0000","last_clean":"2024-01-11T08:25:23.872924+0000","last_became_active":"2024-01-11T08:25:23.860459+0000","last_became_peered":"2024-01-11T08:25:23.860459+0000","last_unstale":"2024-01-11T08:25:23.872924+0000","last_undegraded":"2024-01-11T08:25:23.872924+0000","last_fullsized":"2024-01-11T08:25:23.872924+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:13:56.163286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.876258+0000","last_change":"2024-01-11T08:25:23.872814+0000","last_active":"2024-01-11T08:25:23.876258+0000","last_peered":"2024-01-11T08:25:23.876258+0000","last_clean":"2024-01-11T08:25:23.876258+0000","last_became_active":"2024-01-11T08:25:23.872421+0000","last_became_peered":"2024-01-11T08:25:23.872421+0000","last_unstale":"2024-01-11T08:25:23.876258+0000","last_undegraded":"2024-01-11T08:25:23.876258+0000","last_fullsized":"2024-01-11T08:25:23.876258+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:20:24.901979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.877566+0000","last_change":"2024-01-11T08:25:23.874981+0000","last_active":"2024-01-11T08:25:23.877566+0000","last_peered":"2024-01-11T08:25:23.877566+0000","last_clean":"2024-01-11T08:25:23.877566+0000","last_became_active":"2024-01-11T08:25:23.874541+0000","last_became_peered":"2024-01-11T08:25:23.874541+0000","last_unstale":"2024-01-11T08:25:23.877566+0000","last_undegraded":"2024-01-11T08:25:23.877566+0000","last_fullsized":"2024-01-11T08:25:23.877566+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:55:26.350726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.857632+0000","last_change":"2024-01-11T08:25:23.857632+0000","last_active":"2024-01-11T08:25:23.857632+0000","last_peered":"2024-01-11T08:25:23.857632+0000","last_clean":"2024-01-11T08:25:23.857632+0000","last_became_active":"2024-01-11T08:25:23.857328+0000","last_became_peered":"2024-01-11T08:25:23.857328+0000","last_unstale":"2024-01-11T08:25:23.857632+0000","last_undegraded":"2024-01-11T08:25:23.857632+0000","last_fullsized":"2024-01-11T08:25:23.857632+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:01:30.482699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.863582+0000","last_change":"2024-01-11T08:25:23.855695+0000","last_active":"2024-01-11T08:25:23.863582+0000","last_peered":"2024-01-11T08:25:23.863582+0000","last_clean":"2024-01-11T08:25:23.863582+0000","last_became_active":"2024-01-11T08:25:23.854931+0000","last_became_peered":"2024-01-11T08:25:23.854931+0000","last_unstale":"2024-01-11T08:25:23.863582+0000","last_undegraded":"2024-01-11T08:25:23.863582+0000","last_fullsized":"2024-01-11T08:25:23.863582+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:19:21.377793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.867231+0000","last_change":"2024-01-11T08:25:23.867231+0000","last_active":"2024-01-11T08:25:23.867231+0000","last_peered":"2024-01-11T08:25:23.867231+0000","last_clean":"2024-01-11T08:25:23.867231+0000","last_became_active":"2024-01-11T08:25:23.866921+0000","last_became_peered":"2024-01-11T08:25:23.866921+0000","last_unstale":"2024-01-11T08:25:23.867231+0000","last_undegraded":"2024-01-11T08:25:23.867231+0000","last_fullsized":"2024-01-11T08:25:23.867231+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:50:16.386930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.856105+0000","last_change":"2024-01-11T08:25:23.856105+0000","last_active":"2024-01-11T08:25:23.856105+0000","last_peered":"2024-01-11T08:25:23.856105+0000","last_clean":"2024-01-11T08:25:23.856105+0000","last_became_active":"2024-01-11T08:25:23.855835+0000","last_became_peered":"2024-01-11T08:25:23.855835+0000","last_unstale":"2024-01-11T08:25:23.856105+0000","last_undegraded":"2024-01-11T08:25:23.856105+0000","last_fullsized":"2024-01-11T08:25:23.856105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:47:40.738909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.855832+0000","last_change":"2024-01-11T08:25:23.851288+0000","last_active":"2024-01-11T08:25:23.855832+0000","last_peered":"2024-01-11T08:25:23.855832+0000","last_clean":"2024-01-11T08:25:23.855832+0000","last_became_active":"2024-01-11T08:25:23.851020+0000","last_became_peered":"2024-01-11T08:25:23.851020+0000","last_unstale":"2024-01-11T08:25:23.855832+0000","last_undegraded":"2024-01-11T08:25:23.855832+0000","last_fullsized":"2024-01-11T08:25:23.855832+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:36:58.150280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.861565+0000","last_change":"2024-01-11T08:25:23.861565+0000","last_active":"2024-01-11T08:25:23.861565+0000","last_peered":"2024-01-11T08:25:23.861565+0000","last_clean":"2024-01-11T08:25:23.861565+0000","last_became_active":"2024-01-11T08:25:23.861250+0000","last_became_peered":"2024-01-11T08:25:23.861250+0000","last_unstale":"2024-01-11T08:25:23.861565+0000","last_undegraded":"2024-01-11T08:25:23.861565+0000","last_fullsized":"2024-01-11T08:25:23.861565+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:02.344677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.875445+0000","last_change":"2024-01-11T08:25:23.861222+0000","last_active":"2024-01-11T08:25:23.875445+0000","last_peered":"2024-01-11T08:25:23.875445+0000","last_clean":"2024-01-11T08:25:23.875445+0000","last_became_active":"2024-01-11T08:25:23.860921+0000","last_became_peered":"2024-01-11T08:25:23.860921+0000","last_unstale":"2024-01-11T08:25:23.875445+0000","last_undegraded":"2024-01-11T08:25:23.875445+0000","last_fullsized":"2024-01-11T08:25:23.875445+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:21:38.471310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.879228+0000","last_change":"2024-01-11T08:25:23.873460+0000","last_active":"2024-01-11T08:25:23.879228+0000","last_peered":"2024-01-11T08:25:23.879228+0000","last_clean":"2024-01-11T08:25:23.879228+0000","last_became_active":"2024-01-11T08:25:23.873209+0000","last_became_peered":"2024-01-11T08:25:23.873209+0000","last_unstale":"2024-01-11T08:25:23.879228+0000","last_undegraded":"2024-01-11T08:25:23.879228+0000","last_fullsized":"2024-01-11T08:25:23.879228+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:22:19.702797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.861035+0000","last_change":"2024-01-11T08:25:23.861035+0000","last_active":"2024-01-11T08:25:23.861035+0000","last_peered":"2024-01-11T08:25:23.861035+0000","last_clean":"2024-01-11T08:25:23.861035+0000","last_became_active":"2024-01-11T08:25:23.860737+0000","last_became_peered":"2024-01-11T08:25:23.860737+0000","last_unstale":"2024-01-11T08:25:23.861035+0000","last_undegraded":"2024-01-11T08:25:23.861035+0000","last_fullsized":"2024-01-11T08:25:23.861035+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:43:11.246835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.876486+0000","last_change":"2024-01-11T08:25:23.873249+0000","last_active":"2024-01-11T08:25:23.876486+0000","last_peered":"2024-01-11T08:25:23.876486+0000","last_clean":"2024-01-11T08:25:23.876486+0000","last_became_active":"2024-01-11T08:25:23.872956+0000","last_became_peered":"2024-01-11T08:25:23.872956+0000","last_unstale":"2024-01-11T08:25:23.876486+0000","last_undegraded":"2024-01-11T08:25:23.876486+0000","last_fullsized":"2024-01-11T08:25:23.876486+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:21:38.471310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.860036+0000","last_change":"2024-01-11T08:25:23.860036+0000","last_active":"2024-01-11T08:25:23.860036+0000","last_peered":"2024-01-11T08:25:23.860036+0000","last_clean":"2024-01-11T08:25:23.860036+0000","last_became_active":"2024-01-11T08:25:23.858022+0000","last_became_peered":"2024-01-11T08:25:23.858022+0000","last_unstale":"2024-01-11T08:25:23.860036+0000","last_undegraded":"2024-01-11T08:25:23.860036+0000","last_fullsized":"2024-01-11T08:25:23.860036+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:23:55.749223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.863580+0000","last_change":"2024-01-11T08:25:23.863580+0000","last_active":"2024-01-11T08:25:23.863580+0000","last_peered":"2024-01-11T08:25:23.863580+0000","last_clean":"2024-01-11T08:25:23.863580+0000","last_became_active":"2024-01-11T08:25:23.863269+0000","last_became_peered":"2024-01-11T08:25:23.863269+0000","last_unstale":"2024-01-11T08:25:23.863580+0000","last_undegraded":"2024-01-11T08:25:23.863580+0000","last_fullsized":"2024-01-11T08:25:23.863580+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:47:03.454915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.869698+0000","last_change":"2024-01-11T08:25:23.869698+0000","last_active":"2024-01-11T08:25:23.869698+0000","last_peered":"2024-01-11T08:25:23.869698+0000","last_clean":"2024-01-11T08:25:23.869698+0000","last_became_active":"2024-01-11T08:25:23.868588+0000","last_became_peered":"2024-01-11T08:25:23.868588+0000","last_unstale":"2024-01-11T08:25:23.869698+0000","last_undegraded":"2024-01-11T08:25:23.869698+0000","last_fullsized":"2024-01-11T08:25:23.869698+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:54:45.299302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.868011+0000","last_change":"2024-01-11T08:25:23.861461+0000","last_active":"2024-01-11T08:25:23.868011+0000","last_peered":"2024-01-11T08:25:23.868011+0000","last_clean":"2024-01-11T08:25:23.868011+0000","last_became_active":"2024-01-11T08:25:23.861196+0000","last_became_peered":"2024-01-11T08:25:23.861196+0000","last_unstale":"2024-01-11T08:25:23.868011+0000","last_undegraded":"2024-01-11T08:25:23.868011+0000","last_fullsized":"2024-01-11T08:25:23.868011+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:18:45.387043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.880082+0000","last_change":"2024-01-11T08:25:23.879128+0000","last_active":"2024-01-11T08:25:23.880082+0000","last_peered":"2024-01-11T08:25:23.880082+0000","last_clean":"2024-01-11T08:25:23.880082+0000","last_became_active":"2024-01-11T08:25:23.878862+0000","last_became_peered":"2024-01-11T08:25:23.878862+0000","last_unstale":"2024-01-11T08:25:23.880082+0000","last_undegraded":"2024-01-11T08:25:23.880082+0000","last_fullsized":"2024-01-11T08:25:23.880082+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:00:58.175007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.873020+0000","last_change":"2024-01-11T08:25:23.860722+0000","last_active":"2024-01-11T08:25:23.873020+0000","last_peered":"2024-01-11T08:25:23.873020+0000","last_clean":"2024-01-11T08:25:23.873020+0000","last_became_active":"2024-01-11T08:25:23.860420+0000","last_became_peered":"2024-01-11T08:25:23.860420+0000","last_unstale":"2024-01-11T08:25:23.873020+0000","last_undegraded":"2024-01-11T08:25:23.873020+0000","last_fullsized":"2024-01-11T08:25:23.873020+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:15:41.820195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.873957+0000","last_change":"2024-01-11T08:25:23.867445+0000","last_active":"2024-01-11T08:25:23.873957+0000","last_peered":"2024-01-11T08:25:23.873957+0000","last_clean":"2024-01-11T08:25:23.873957+0000","last_became_active":"2024-01-11T08:25:23.867180+0000","last_became_peered":"2024-01-11T08:25:23.867180+0000","last_unstale":"2024-01-11T08:25:23.873957+0000","last_undegraded":"2024-01-11T08:25:23.873957+0000","last_fullsized":"2024-01-11T08:25:23.873957+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:08:45.304781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.873728+0000","last_change":"2024-01-11T08:25:23.869311+0000","last_active":"2024-01-11T08:25:23.873728+0000","last_peered":"2024-01-11T08:25:23.873728+0000","last_clean":"2024-01-11T08:25:23.873728+0000","last_became_active":"2024-01-11T08:25:23.869043+0000","last_became_peered":"2024-01-11T08:25:23.869043+0000","last_unstale":"2024-01-11T08:25:23.873728+0000","last_undegraded":"2024-01-11T08:25:23.873728+0000","last_fullsized":"2024-01-11T08:25:23.873728+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:47:03.454915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.874066+0000","last_change":"2024-01-11T08:25:23.868231+0000","last_active":"2024-01-11T08:25:23.874066+0000","last_peered":"2024-01-11T08:25:23.874066+0000","last_clean":"2024-01-11T08:25:23.874066+0000","last_became_active":"2024-01-11T08:25:23.861599+0000","last_became_peered":"2024-01-11T08:25:23.861599+0000","last_unstale":"2024-01-11T08:25:23.874066+0000","last_undegraded":"2024-01-11T08:25:23.874066+0000","last_fullsized":"2024-01-11T08:25:23.874066+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:10:22.895007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.877492+0000","last_change":"2024-01-11T08:25:23.874598+0000","last_active":"2024-01-11T08:25:23.877492+0000","last_peered":"2024-01-11T08:25:23.877492+0000","last_clean":"2024-01-11T08:25:23.877492+0000","last_became_active":"2024-01-11T08:25:23.872952+0000","last_became_peered":"2024-01-11T08:25:23.872952+0000","last_unstale":"2024-01-11T08:25:23.877492+0000","last_undegraded":"2024-01-11T08:25:23.877492+0000","last_fullsized":"2024-01-11T08:25:23.877492+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:21:33.358382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.876541+0000","last_change":"2024-01-11T08:25:23.873989+0000","last_active":"2024-01-11T08:25:23.876541+0000","last_peered":"2024-01-11T08:25:23.876541+0000","last_clean":"2024-01-11T08:25:23.876541+0000","last_became_active":"2024-01-11T08:25:23.873673+0000","last_became_peered":"2024-01-11T08:25:23.873673+0000","last_unstale":"2024-01-11T08:25:23.876541+0000","last_undegraded":"2024-01-11T08:25:23.876541+0000","last_fullsized":"2024-01-11T08:25:23.876541+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:58:35.728365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.872946+0000","last_change":"2024-01-11T08:25:23.861407+0000","last_active":"2024-01-11T08:25:23.872946+0000","last_peered":"2024-01-11T08:25:23.872946+0000","last_clean":"2024-01-11T08:25:23.872946+0000","last_became_active":"2024-01-11T08:25:23.861042+0000","last_became_peered":"2024-01-11T08:25:23.861042+0000","last_unstale":"2024-01-11T08:25:23.872946+0000","last_undegraded":"2024-01-11T08:25:23.872946+0000","last_fullsized":"2024-01-11T08:25:23.872946+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:20:08.135199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.858847+0000","last_change":"2024-01-11T08:25:23.858847+0000","last_active":"2024-01-11T08:25:23.858847+0000","last_peered":"2024-01-11T08:25:23.858847+0000","last_clean":"2024-01-11T08:25:23.858847+0000","last_became_active":"2024-01-11T08:25:23.858557+0000","last_became_peered":"2024-01-11T08:25:23.858557+0000","last_unstale":"2024-01-11T08:25:23.858847+0000","last_undegraded":"2024-01-11T08:25:23.858847+0000","last_fullsized":"2024-01-11T08:25:23.858847+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:54:45.299302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.867927+0000","last_change":"2024-01-11T08:25:23.867927+0000","last_active":"2024-01-11T08:25:23.867927+0000","last_peered":"2024-01-11T08:25:23.867927+0000","last_clean":"2024-01-11T08:25:23.867927+0000","last_became_active":"2024-01-11T08:25:23.867654+0000","last_became_peered":"2024-01-11T08:25:23.867654+0000","last_unstale":"2024-01-11T08:25:23.867927+0000","last_undegraded":"2024-01-11T08:25:23.867927+0000","last_fullsized":"2024-01-11T08:25:23.867927+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:43:11.246835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.874140+0000","last_change":"2024-01-11T08:25:23.866937+0000","last_active":"2024-01-11T08:25:23.874140+0000","last_peered":"2024-01-11T08:25:23.874140+0000","last_clean":"2024-01-11T08:25:23.874140+0000","last_became_active":"2024-01-11T08:25:23.865250+0000","last_became_peered":"2024-01-11T08:25:23.865250+0000","last_unstale":"2024-01-11T08:25:23.874140+0000","last_undegraded":"2024-01-11T08:25:23.874140+0000","last_fullsized":"2024-01-11T08:25:23.874140+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:25:40.491820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.866155+0000","last_change":"2024-01-11T08:25:23.852786+0000","last_active":"2024-01-11T08:25:23.866155+0000","last_peered":"2024-01-11T08:25:23.866155+0000","last_clean":"2024-01-11T08:25:23.866155+0000","last_became_active":"2024-01-11T08:25:23.852460+0000","last_became_peered":"2024-01-11T08:25:23.852460+0000","last_unstale":"2024-01-11T08:25:23.866155+0000","last_undegraded":"2024-01-11T08:25:23.866155+0000","last_fullsized":"2024-01-11T08:25:23.866155+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:14:12.366345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.877446+0000","last_change":"2024-01-11T08:25:23.873169+0000","last_active":"2024-01-11T08:25:23.877446+0000","last_peered":"2024-01-11T08:25:23.877446+0000","last_clean":"2024-01-11T08:25:23.877446+0000","last_became_active":"2024-01-11T08:25:23.872841+0000","last_became_peered":"2024-01-11T08:25:23.872841+0000","last_unstale":"2024-01-11T08:25:23.877446+0000","last_undegraded":"2024-01-11T08:25:23.877446+0000","last_fullsized":"2024-01-11T08:25:23.877446+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:11:46.534149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.873295+0000","last_change":"2024-01-11T08:25:23.864748+0000","last_active":"2024-01-11T08:25:23.873295+0000","last_peered":"2024-01-11T08:25:23.873295+0000","last_clean":"2024-01-11T08:25:23.873295+0000","last_became_active":"2024-01-11T08:25:23.864474+0000","last_became_peered":"2024-01-11T08:25:23.864474+0000","last_unstale":"2024-01-11T08:25:23.873295+0000","last_undegraded":"2024-01-11T08:25:23.873295+0000","last_fullsized":"2024-01-11T08:25:23.873295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:08:10.968900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.864146+0000","last_change":"2024-01-11T08:25:23.864146+0000","last_active":"2024-01-11T08:25:23.864146+0000","last_peered":"2024-01-11T08:25:23.864146+0000","last_clean":"2024-01-11T08:25:23.864146+0000","last_became_active":"2024-01-11T08:25:23.863890+0000","last_became_peered":"2024-01-11T08:25:23.863890+0000","last_unstale":"2024-01-11T08:25:23.864146+0000","last_undegraded":"2024-01-11T08:25:23.864146+0000","last_fullsized":"2024-01-11T08:25:23.864146+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:45:12.148806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.862496+0000","last_change":"2024-01-11T08:25:23.862496+0000","last_active":"2024-01-11T08:25:23.862496+0000","last_peered":"2024-01-11T08:25:23.862496+0000","last_clean":"2024-01-11T08:25:23.862496+0000","last_became_active":"2024-01-11T08:25:23.862234+0000","last_became_peered":"2024-01-11T08:25:23.862234+0000","last_unstale":"2024-01-11T08:25:23.862496+0000","last_undegraded":"2024-01-11T08:25:23.862496+0000","last_fullsized":"2024-01-11T08:25:23.862496+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:01:30.482699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"74'4","reported_seq":13,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.881153+0000","last_change":"2024-01-11T08:25:22.851110+0000","last_active":"2024-01-11T08:25:23.881153+0000","last_peered":"2024-01-11T08:25:23.881153+0000","last_clean":"2024-01-11T08:25:23.881153+0000","last_became_active":"2024-01-11T08:25:22.850230+0000","last_became_peered":"2024-01-11T08:25:22.850230+0000","last_unstale":"2024-01-11T08:25:23.881153+0000","last_undegraded":"2024-01-11T08:25:23.881153+0000","last_fullsized":"2024-01-11T08:25:23.881153+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:22:18.693300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.877128+0000","last_change":"2024-01-11T08:25:23.872201+0000","last_active":"2024-01-11T08:25:23.877128+0000","last_peered":"2024-01-11T08:25:23.877128+0000","last_clean":"2024-01-11T08:25:23.877128+0000","last_became_active":"2024-01-11T08:25:23.871950+0000","last_became_peered":"2024-01-11T08:25:23.871950+0000","last_unstale":"2024-01-11T08:25:23.877128+0000","last_undegraded":"2024-01-11T08:25:23.877128+0000","last_fullsized":"2024-01-11T08:25:23.877128+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:41:23.867564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"74'1","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.872632+0000","last_change":"2024-01-11T08:25:22.857442+0000","last_active":"2024-01-11T08:25:23.872632+0000","last_peered":"2024-01-11T08:25:23.872632+0000","last_clean":"2024-01-11T08:25:23.872632+0000","last_became_active":"2024-01-11T08:25:22.856309+0000","last_became_peered":"2024-01-11T08:25:22.856309+0000","last_unstale":"2024-01-11T08:25:23.872632+0000","last_undegraded":"2024-01-11T08:25:23.872632+0000","last_fullsized":"2024-01-11T08:25:23.872632+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:39:14.987410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.862306+0000","last_change":"2024-01-11T08:25:23.862306+0000","last_active":"2024-01-11T08:25:23.862306+0000","last_peered":"2024-01-11T08:25:23.862306+0000","last_clean":"2024-01-11T08:25:23.862306+0000","last_became_active":"2024-01-11T08:25:23.861889+0000","last_became_peered":"2024-01-11T08:25:23.861889+0000","last_unstale":"2024-01-11T08:25:23.862306+0000","last_undegraded":"2024-01-11T08:25:23.862306+0000","last_fullsized":"2024-01-11T08:25:23.862306+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:59:37.011537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"74'2","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.879808+0000","last_change":"2024-01-11T08:25:22.864486+0000","last_active":"2024-01-11T08:25:23.879808+0000","last_peered":"2024-01-11T08:25:23.879808+0000","last_clean":"2024-01-11T08:25:23.879808+0000","last_became_active":"2024-01-11T08:25:22.864194+0000","last_became_peered":"2024-01-11T08:25:22.864194+0000","last_unstale":"2024-01-11T08:25:23.879808+0000","last_undegraded":"2024-01-11T08:25:23.879808+0000","last_fullsized":"2024-01-11T08:25:23.879808+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:12:37.683775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.873819+0000","last_change":"2024-01-11T08:25:23.873819+0000","last_active":"2024-01-11T08:25:23.873819+0000","last_peered":"2024-01-11T08:25:23.873819+0000","last_clean":"2024-01-11T08:25:23.873819+0000","last_became_active":"2024-01-11T08:25:23.873293+0000","last_became_peered":"2024-01-11T08:25:23.873293+0000","last_unstale":"2024-01-11T08:25:23.873819+0000","last_undegraded":"2024-01-11T08:25:23.873819+0000","last_fullsized":"2024-01-11T08:25:23.873819+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:52:36.524494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"74'2","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.867065+0000","last_change":"2024-01-11T08:25:22.854672+0000","last_active":"2024-01-11T08:25:23.867065+0000","last_peered":"2024-01-11T08:25:23.867065+0000","last_clean":"2024-01-11T08:25:23.867065+0000","last_became_active":"2024-01-11T08:25:22.854384+0000","last_became_peered":"2024-01-11T08:25:22.854384+0000","last_unstale":"2024-01-11T08:25:23.867065+0000","last_undegraded":"2024-01-11T08:25:23.867065+0000","last_fullsized":"2024-01-11T08:25:23.867065+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:47:02.445419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.870165+0000","last_change":"2024-01-11T08:25:23.865456+0000","last_active":"2024-01-11T08:25:23.870165+0000","last_peered":"2024-01-11T08:25:23.870165+0000","last_clean":"2024-01-11T08:25:23.870165+0000","last_became_active":"2024-01-11T08:25:23.863331+0000","last_became_peered":"2024-01-11T08:25:23.863331+0000","last_unstale":"2024-01-11T08:25:23.870165+0000","last_undegraded":"2024-01-11T08:25:23.870165+0000","last_fullsized":"2024-01-11T08:25:23.870165+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:36:44.218033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"74'1","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.846982+0000","last_change":"2024-01-11T08:25:22.856142+0000","last_active":"2024-01-11T08:25:23.846982+0000","last_peered":"2024-01-11T08:25:23.846982+0000","last_clean":"2024-01-11T08:25:23.846982+0000","last_became_active":"2024-01-11T08:25:22.855635+0000","last_became_peered":"2024-01-11T08:25:22.855635+0000","last_unstale":"2024-01-11T08:25:23.846982+0000","last_undegraded":"2024-01-11T08:25:23.846982+0000","last_fullsized":"2024-01-11T08:25:23.846982+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:42:52.255488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.870564+0000","last_change":"2024-01-11T08:25:23.866332+0000","last_active":"2024-01-11T08:25:23.870564+0000","last_peered":"2024-01-11T08:25:23.870564+0000","last_clean":"2024-01-11T08:25:23.870564+0000","last_became_active":"2024-01-11T08:25:23.861779+0000","last_became_peered":"2024-01-11T08:25:23.861779+0000","last_unstale":"2024-01-11T08:25:23.870564+0000","last_undegraded":"2024-01-11T08:25:23.870564+0000","last_fullsized":"2024-01-11T08:25:23.870564+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:49:28.354463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"74'1","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.852670+0000","last_change":"2024-01-11T08:25:22.854267+0000","last_active":"2024-01-11T08:25:23.852670+0000","last_peered":"2024-01-11T08:25:23.852670+0000","last_clean":"2024-01-11T08:25:23.852670+0000","last_became_active":"2024-01-11T08:25:22.853884+0000","last_became_peered":"2024-01-11T08:25:22.853884+0000","last_unstale":"2024-01-11T08:25:23.852670+0000","last_undegraded":"2024-01-11T08:25:23.852670+0000","last_fullsized":"2024-01-11T08:25:23.852670+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:35:21.406141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.874157+0000","last_change":"2024-01-11T08:25:23.869344+0000","last_active":"2024-01-11T08:25:23.874157+0000","last_peered":"2024-01-11T08:25:23.874157+0000","last_clean":"2024-01-11T08:25:23.874157+0000","last_became_active":"2024-01-11T08:25:23.869088+0000","last_became_peered":"2024-01-11T08:25:23.869088+0000","last_unstale":"2024-01-11T08:25:23.874157+0000","last_undegraded":"2024-01-11T08:25:23.874157+0000","last_fullsized":"2024-01-11T08:25:23.874157+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:42:53.264984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-11T08:25:22.856781+0000","last_change":"2024-01-11T08:25:22.852104+0000","last_active":"2024-01-11T08:25:22.856781+0000","last_peered":"2024-01-11T08:25:22.856781+0000","last_clean":"2024-01-11T08:25:22.856781+0000","last_became_active":"2024-01-11T08:25:22.851787+0000","last_became_peered":"2024-01-11T08:25:22.851787+0000","last_unstale":"2024-01-11T08:25:22.856781+0000","last_undegraded":"2024-01-11T08:25:22.856781+0000","last_fullsized":"2024-01-11T08:25:22.856781+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:23:54.739727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.870600+0000","last_change":"2024-01-11T08:25:23.866696+0000","last_active":"2024-01-11T08:25:23.870600+0000","last_peered":"2024-01-11T08:25:23.870600+0000","last_clean":"2024-01-11T08:25:23.870600+0000","last_became_active":"2024-01-11T08:25:23.864469+0000","last_became_peered":"2024-01-11T08:25:23.864469+0000","last_unstale":"2024-01-11T08:25:23.870600+0000","last_undegraded":"2024-01-11T08:25:23.870600+0000","last_fullsized":"2024-01-11T08:25:23.870600+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:16.536682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-11T08:25:22.860954+0000","last_change":"2024-01-11T08:25:22.859339+0000","last_active":"2024-01-11T08:25:22.860954+0000","last_peered":"2024-01-11T08:25:22.860954+0000","last_clean":"2024-01-11T08:25:22.860954+0000","last_became_active":"2024-01-11T08:25:22.859026+0000","last_became_peered":"2024-01-11T08:25:22.859026+0000","last_unstale":"2024-01-11T08:25:22.860954+0000","last_undegraded":"2024-01-11T08:25:22.860954+0000","last_fullsized":"2024-01-11T08:25:22.860954+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:21:37.461813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.862643+0000","last_change":"2024-01-11T08:25:23.862643+0000","last_active":"2024-01-11T08:25:23.862643+0000","last_peered":"2024-01-11T08:25:23.862643+0000","last_clean":"2024-01-11T08:25:23.862643+0000","last_became_active":"2024-01-11T08:25:23.861028+0000","last_became_peered":"2024-01-11T08:25:23.861028+0000","last_unstale":"2024-01-11T08:25:23.862643+0000","last_undegraded":"2024-01-11T08:25:23.862643+0000","last_fullsized":"2024-01-11T08:25:23.862643+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:11:46.534149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.861770+0000","last_change":"2024-01-11T08:25:22.863437+0000","last_active":"2024-01-11T08:25:23.861770+0000","last_peered":"2024-01-11T08:25:23.861770+0000","last_clean":"2024-01-11T08:25:23.861770+0000","last_became_active":"2024-01-11T08:25:22.863054+0000","last_became_peered":"2024-01-11T08:25:22.863054+0000","last_unstale":"2024-01-11T08:25:23.861770+0000","last_undegraded":"2024-01-11T08:25:23.861770+0000","last_fullsized":"2024-01-11T08:25:23.861770+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:20:30.108010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.879652+0000","last_change":"2024-01-11T08:25:23.878865+0000","last_active":"2024-01-11T08:25:23.879652+0000","last_peered":"2024-01-11T08:25:23.879652+0000","last_clean":"2024-01-11T08:25:23.879652+0000","last_became_active":"2024-01-11T08:25:23.878598+0000","last_became_peered":"2024-01-11T08:25:23.878598+0000","last_unstale":"2024-01-11T08:25:23.879652+0000","last_undegraded":"2024-01-11T08:25:23.879652+0000","last_fullsized":"2024-01-11T08:25:23.879652+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:24:45.942689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.842617+0000","last_change":"2024-01-11T08:25:22.852556+0000","last_active":"2024-01-11T08:25:23.842617+0000","last_peered":"2024-01-11T08:25:23.842617+0000","last_clean":"2024-01-11T08:25:23.842617+0000","last_became_active":"2024-01-11T08:25:22.851861+0000","last_became_peered":"2024-01-11T08:25:22.851861+0000","last_unstale":"2024-01-11T08:25:23.842617+0000","last_undegraded":"2024-01-11T08:25:23.842617+0000","last_fullsized":"2024-01-11T08:25:23.842617+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:21:32.348886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.876370+0000","last_change":"2024-01-11T08:25:23.869012+0000","last_active":"2024-01-11T08:25:23.876370+0000","last_peered":"2024-01-11T08:25:23.876370+0000","last_clean":"2024-01-11T08:25:23.876370+0000","last_became_active":"2024-01-11T08:25:23.868581+0000","last_became_peered":"2024-01-11T08:25:23.868581+0000","last_unstale":"2024-01-11T08:25:23.876370+0000","last_undegraded":"2024-01-11T08:25:23.876370+0000","last_fullsized":"2024-01-11T08:25:23.876370+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:37:20.723150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"74'3","reported_seq":13,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.876961+0000","last_change":"2024-01-11T08:25:22.868809+0000","last_active":"2024-01-11T08:25:23.876961+0000","last_peered":"2024-01-11T08:25:23.876961+0000","last_clean":"2024-01-11T08:25:23.876961+0000","last_became_active":"2024-01-11T08:25:22.868572+0000","last_became_peered":"2024-01-11T08:25:22.868572+0000","last_unstale":"2024-01-11T08:25:23.876961+0000","last_undegraded":"2024-01-11T08:25:23.876961+0000","last_fullsized":"2024-01-11T08:25:23.876961+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:43:47.124370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.878149+0000","last_change":"2024-01-11T08:25:23.877485+0000","last_active":"2024-01-11T08:25:23.878149+0000","last_peered":"2024-01-11T08:25:23.878149+0000","last_clean":"2024-01-11T08:25:23.878149+0000","last_became_active":"2024-01-11T08:25:23.877189+0000","last_became_peered":"2024-01-11T08:25:23.877189+0000","last_unstale":"2024-01-11T08:25:23.878149+0000","last_undegraded":"2024-01-11T08:25:23.878149+0000","last_fullsized":"2024-01-11T08:25:23.878149+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:50:16.386930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"74'6","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.871722+0000","last_change":"2024-01-11T08:25:22.858712+0000","last_active":"2024-01-11T08:25:23.871722+0000","last_peered":"2024-01-11T08:25:23.871722+0000","last_clean":"2024-01-11T08:25:23.871722+0000","last_became_active":"2024-01-11T08:25:22.858429+0000","last_became_peered":"2024-01-11T08:25:22.858429+0000","last_unstale":"2024-01-11T08:25:23.871722+0000","last_undegraded":"2024-01-11T08:25:23.871722+0000","last_fullsized":"2024-01-11T08:25:23.871722+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:20:07.177322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.866341+0000","last_change":"2024-01-11T08:25:23.866341+0000","last_active":"2024-01-11T08:25:23.866341+0000","last_peered":"2024-01-11T08:25:23.866341+0000","last_clean":"2024-01-11T08:25:23.866341+0000","last_became_active":"2024-01-11T08:25:23.866036+0000","last_became_peered":"2024-01-11T08:25:23.866036+0000","last_unstale":"2024-01-11T08:25:23.866341+0000","last_undegraded":"2024-01-11T08:25:23.866341+0000","last_fullsized":"2024-01-11T08:25:23.866341+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:56:15.839817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"74'2","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.874460+0000","last_change":"2024-01-11T08:25:22.852070+0000","last_active":"2024-01-11T08:25:23.874460+0000","last_peered":"2024-01-11T08:25:23.874460+0000","last_clean":"2024-01-11T08:25:23.874460+0000","last_became_active":"2024-01-11T08:25:22.851774+0000","last_became_peered":"2024-01-11T08:25:22.851774+0000","last_unstale":"2024-01-11T08:25:23.874460+0000","last_undegraded":"2024-01-11T08:25:23.874460+0000","last_fullsized":"2024-01-11T08:25:23.874460+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:00:57.165511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.869504+0000","last_change":"2024-01-11T08:25:23.863936+0000","last_active":"2024-01-11T08:25:23.869504+0000","last_peered":"2024-01-11T08:25:23.869504+0000","last_clean":"2024-01-11T08:25:23.869504+0000","last_became_active":"2024-01-11T08:25:23.863655+0000","last_became_peered":"2024-01-11T08:25:23.863655+0000","last_unstale":"2024-01-11T08:25:23.869504+0000","last_undegraded":"2024-01-11T08:25:23.869504+0000","last_fullsized":"2024-01-11T08:25:23.869504+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:02.344677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.854354+0000","last_change":"2024-01-11T08:25:22.858764+0000","last_active":"2024-01-11T08:25:23.854354+0000","last_peered":"2024-01-11T08:25:23.854354+0000","last_clean":"2024-01-11T08:25:23.854354+0000","last_became_active":"2024-01-11T08:25:22.857409+0000","last_became_peered":"2024-01-11T08:25:22.857409+0000","last_unstale":"2024-01-11T08:25:23.854354+0000","last_undegraded":"2024-01-11T08:25:23.854354+0000","last_fullsized":"2024-01-11T08:25:23.854354+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:41:40.154137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.869005+0000","last_change":"2024-01-11T08:25:23.869005+0000","last_active":"2024-01-11T08:25:23.869005+0000","last_peered":"2024-01-11T08:25:23.869005+0000","last_clean":"2024-01-11T08:25:23.869005+0000","last_became_active":"2024-01-11T08:25:23.868691+0000","last_became_peered":"2024-01-11T08:25:23.868691+0000","last_unstale":"2024-01-11T08:25:23.869005+0000","last_undegraded":"2024-01-11T08:25:23.869005+0000","last_fullsized":"2024-01-11T08:25:23.869005+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:07:23.998750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-11T08:25:22.857407+0000","last_change":"2024-01-11T08:25:22.854594+0000","last_active":"2024-01-11T08:25:22.857407+0000","last_peered":"2024-01-11T08:25:22.857407+0000","last_clean":"2024-01-11T08:25:22.857407+0000","last_became_active":"2024-01-11T08:25:22.854299+0000","last_became_peered":"2024-01-11T08:25:22.854299+0000","last_unstale":"2024-01-11T08:25:22.857407+0000","last_undegraded":"2024-01-11T08:25:22.857407+0000","last_fullsized":"2024-01-11T08:25:22.857407+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:01.335181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.877508+0000","last_change":"2024-01-11T08:25:23.875606+0000","last_active":"2024-01-11T08:25:23.877508+0000","last_peered":"2024-01-11T08:25:23.877508+0000","last_clean":"2024-01-11T08:25:23.877508+0000","last_became_active":"2024-01-11T08:25:23.875281+0000","last_became_peered":"2024-01-11T08:25:23.875281+0000","last_unstale":"2024-01-11T08:25:23.877508+0000","last_undegraded":"2024-01-11T08:25:23.877508+0000","last_fullsized":"2024-01-11T08:25:23.877508+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:12:22.491644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"74'8","reported_seq":18,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.875202+0000","last_change":"2024-01-11T08:25:22.860304+0000","last_active":"2024-01-11T08:25:23.875202+0000","last_peered":"2024-01-11T08:25:23.875202+0000","last_clean":"2024-01-11T08:25:23.875202+0000","last_became_active":"2024-01-11T08:25:22.860063+0000","last_became_peered":"2024-01-11T08:25:22.860063+0000","last_unstale":"2024-01-11T08:25:23.875202+0000","last_undegraded":"2024-01-11T08:25:23.875202+0000","last_fullsized":"2024-01-11T08:25:23.875202+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:20:07.177322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.877453+0000","last_change":"2024-01-11T08:25:23.874224+0000","last_active":"2024-01-11T08:25:23.877453+0000","last_peered":"2024-01-11T08:25:23.877453+0000","last_clean":"2024-01-11T08:25:23.877453+0000","last_became_active":"2024-01-11T08:25:23.873913+0000","last_became_peered":"2024-01-11T08:25:23.873913+0000","last_unstale":"2024-01-11T08:25:23.877453+0000","last_undegraded":"2024-01-11T08:25:23.877453+0000","last_fullsized":"2024-01-11T08:25:23.877453+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:52:36.524494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"74'1","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.874537+0000","last_change":"2024-01-11T08:25:22.860048+0000","last_active":"2024-01-11T08:25:23.874537+0000","last_peered":"2024-01-11T08:25:23.874537+0000","last_clean":"2024-01-11T08:25:23.874537+0000","last_became_active":"2024-01-11T08:25:22.859794+0000","last_became_peered":"2024-01-11T08:25:22.859794+0000","last_unstale":"2024-01-11T08:25:23.874537+0000","last_undegraded":"2024-01-11T08:25:23.874537+0000","last_fullsized":"2024-01-11T08:25:23.874537+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:13:42.223026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.862041+0000","last_change":"2024-01-11T08:25:23.862041+0000","last_active":"2024-01-11T08:25:23.862041+0000","last_peered":"2024-01-11T08:25:23.862041+0000","last_clean":"2024-01-11T08:25:23.862041+0000","last_became_active":"2024-01-11T08:25:23.861740+0000","last_became_peered":"2024-01-11T08:25:23.861740+0000","last_unstale":"2024-01-11T08:25:23.862041+0000","last_undegraded":"2024-01-11T08:25:23.862041+0000","last_fullsized":"2024-01-11T08:25:23.862041+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:27:58.197947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"74'2","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.880302+0000","last_change":"2024-01-11T08:25:22.856459+0000","last_active":"2024-01-11T08:25:23.880302+0000","last_peered":"2024-01-11T08:25:23.880302+0000","last_clean":"2024-01-11T08:25:23.880302+0000","last_became_active":"2024-01-11T08:25:22.856212+0000","last_became_peered":"2024-01-11T08:25:22.856212+0000","last_unstale":"2024-01-11T08:25:23.880302+0000","last_undegraded":"2024-01-11T08:25:23.880302+0000","last_fullsized":"2024-01-11T08:25:23.880302+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:33:01.469771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.857559+0000","last_change":"2024-01-11T08:25:23.856416+0000","last_active":"2024-01-11T08:25:23.857559+0000","last_peered":"2024-01-11T08:25:23.857559+0000","last_clean":"2024-01-11T08:25:23.857559+0000","last_became_active":"2024-01-11T08:25:23.856103+0000","last_became_peered":"2024-01-11T08:25:23.856103+0000","last_unstale":"2024-01-11T08:25:23.857559+0000","last_undegraded":"2024-01-11T08:25:23.857559+0000","last_fullsized":"2024-01-11T08:25:23.857559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:52:31.741170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"74'1","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.844649+0000","last_change":"2024-01-11T08:25:22.856103+0000","last_active":"2024-01-11T08:25:23.844649+0000","last_peered":"2024-01-11T08:25:23.844649+0000","last_clean":"2024-01-11T08:25:23.844649+0000","last_became_active":"2024-01-11T08:25:22.855851+0000","last_became_peered":"2024-01-11T08:25:22.855851+0000","last_unstale":"2024-01-11T08:25:23.844649+0000","last_undegraded":"2024-01-11T08:25:23.844649+0000","last_fullsized":"2024-01-11T08:25:23.844649+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:11:45.524653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.878131+0000","last_change":"2024-01-11T08:25:23.877382+0000","last_active":"2024-01-11T08:25:23.878131+0000","last_peered":"2024-01-11T08:25:23.878131+0000","last_clean":"2024-01-11T08:25:23.878131+0000","last_became_active":"2024-01-11T08:25:23.877132+0000","last_became_peered":"2024-01-11T08:25:23.877132+0000","last_unstale":"2024-01-11T08:25:23.878131+0000","last_undegraded":"2024-01-11T08:25:23.878131+0000","last_fullsized":"2024-01-11T08:25:23.878131+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:45:12.148806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-11T08:25:22.853614+0000","last_change":"2024-01-11T08:25:22.849646+0000","last_active":"2024-01-11T08:25:22.853614+0000","last_peered":"2024-01-11T08:25:22.853614+0000","last_clean":"2024-01-11T08:25:22.853614+0000","last_became_active":"2024-01-11T08:25:22.849364+0000","last_became_peered":"2024-01-11T08:25:22.849364+0000","last_unstale":"2024-01-11T08:25:22.853614+0000","last_undegraded":"2024-01-11T08:25:22.853614+0000","last_fullsized":"2024-01-11T08:25:22.853614+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:37:19.713653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.866272+0000","last_change":"2024-01-11T08:25:23.866272+0000","last_active":"2024-01-11T08:25:23.866272+0000","last_peered":"2024-01-11T08:25:23.866272+0000","last_clean":"2024-01-11T08:25:23.866272+0000","last_became_active":"2024-01-11T08:25:23.865969+0000","last_became_peered":"2024-01-11T08:25:23.865969+0000","last_unstale":"2024-01-11T08:25:23.866272+0000","last_undegraded":"2024-01-11T08:25:23.866272+0000","last_fullsized":"2024-01-11T08:25:23.866272+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:00:41.682079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"74'2","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.872667+0000","last_change":"2024-01-11T08:25:22.856273+0000","last_active":"2024-01-11T08:25:23.872667+0000","last_peered":"2024-01-11T08:25:23.872667+0000","last_clean":"2024-01-11T08:25:23.872667+0000","last_became_active":"2024-01-11T08:25:22.854621+0000","last_became_peered":"2024-01-11T08:25:22.854621+0000","last_unstale":"2024-01-11T08:25:23.872667+0000","last_undegraded":"2024-01-11T08:25:23.872667+0000","last_fullsized":"2024-01-11T08:25:23.872667+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:12:21.482148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.867008+0000","last_change":"2024-01-11T08:25:23.867008+0000","last_active":"2024-01-11T08:25:23.867008+0000","last_peered":"2024-01-11T08:25:23.867008+0000","last_clean":"2024-01-11T08:25:23.867008+0000","last_became_active":"2024-01-11T08:25:23.866577+0000","last_became_peered":"2024-01-11T08:25:23.866577+0000","last_unstale":"2024-01-11T08:25:23.867008+0000","last_undegraded":"2024-01-11T08:25:23.867008+0000","last_fullsized":"2024-01-11T08:25:23.867008+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:56:15.839817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.856472+0000","last_change":"2024-01-11T08:25:22.864331+0000","last_active":"2024-01-11T08:25:23.856472+0000","last_peered":"2024-01-11T08:25:23.856472+0000","last_clean":"2024-01-11T08:25:23.856472+0000","last_became_active":"2024-01-11T08:25:22.863989+0000","last_became_peered":"2024-01-11T08:25:22.863989+0000","last_unstale":"2024-01-11T08:25:23.856472+0000","last_undegraded":"2024-01-11T08:25:23.856472+0000","last_fullsized":"2024-01-11T08:25:23.856472+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:05:35.853054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.861329+0000","last_change":"2024-01-11T08:25:23.861329+0000","last_active":"2024-01-11T08:25:23.861329+0000","last_peered":"2024-01-11T08:25:23.861329+0000","last_clean":"2024-01-11T08:25:23.861329+0000","last_became_active":"2024-01-11T08:25:23.861073+0000","last_became_peered":"2024-01-11T08:25:23.861073+0000","last_unstale":"2024-01-11T08:25:23.861329+0000","last_undegraded":"2024-01-11T08:25:23.861329+0000","last_fullsized":"2024-01-11T08:25:23.861329+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:24:45.942689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-11T08:25:22.861169+0000","last_change":"2024-01-11T08:25:22.859051+0000","last_active":"2024-01-11T08:25:22.861169+0000","last_peered":"2024-01-11T08:25:22.861169+0000","last_clean":"2024-01-11T08:25:22.861169+0000","last_became_active":"2024-01-11T08:25:22.858810+0000","last_became_peered":"2024-01-11T08:25:22.858810+0000","last_unstale":"2024-01-11T08:25:22.861169+0000","last_undegraded":"2024-01-11T08:25:22.861169+0000","last_fullsized":"2024-01-11T08:25:22.861169+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:07:22.989253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.856041+0000","last_change":"2024-01-11T08:25:23.856041+0000","last_active":"2024-01-11T08:25:23.856041+0000","last_peered":"2024-01-11T08:25:23.856041+0000","last_clean":"2024-01-11T08:25:23.856041+0000","last_became_active":"2024-01-11T08:25:23.855774+0000","last_became_peered":"2024-01-11T08:25:23.855774+0000","last_unstale":"2024-01-11T08:25:23.856041+0000","last_undegraded":"2024-01-11T08:25:23.856041+0000","last_fullsized":"2024-01-11T08:25:23.856041+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:59:37.011537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"74'2","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.876889+0000","last_change":"2024-01-11T08:25:22.860486+0000","last_active":"2024-01-11T08:25:23.876889+0000","last_peered":"2024-01-11T08:25:23.876889+0000","last_clean":"2024-01-11T08:25:23.876889+0000","last_became_active":"2024-01-11T08:25:22.860182+0000","last_became_peered":"2024-01-11T08:25:22.860182+0000","last_unstale":"2024-01-11T08:25:23.876889+0000","last_undegraded":"2024-01-11T08:25:23.876889+0000","last_fullsized":"2024-01-11T08:25:23.876889+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:31:17.972125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.864323+0000","last_change":"2024-01-11T08:25:23.864323+0000","last_active":"2024-01-11T08:25:23.864323+0000","last_peered":"2024-01-11T08:25:23.864323+0000","last_clean":"2024-01-11T08:25:23.864323+0000","last_became_active":"2024-01-11T08:25:23.864072+0000","last_became_peered":"2024-01-11T08:25:23.864072+0000","last_unstale":"2024-01-11T08:25:23.864323+0000","last_undegraded":"2024-01-11T08:25:23.864323+0000","last_fullsized":"2024-01-11T08:25:23.864323+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:52:31.481386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-11T08:25:22.861915+0000","last_change":"2024-01-11T08:25:22.860575+0000","last_active":"2024-01-11T08:25:22.861915+0000","last_peered":"2024-01-11T08:25:22.861915+0000","last_clean":"2024-01-11T08:25:22.861915+0000","last_became_active":"2024-01-11T08:25:22.858134+0000","last_became_peered":"2024-01-11T08:25:22.858134+0000","last_unstale":"2024-01-11T08:25:22.861915+0000","last_undegraded":"2024-01-11T08:25:22.861915+0000","last_fullsized":"2024-01-11T08:25:22.861915+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:41:22.858068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.876350+0000","last_change":"2024-01-11T08:25:23.868846+0000","last_active":"2024-01-11T08:25:23.876350+0000","last_peered":"2024-01-11T08:25:23.876350+0000","last_clean":"2024-01-11T08:25:23.876350+0000","last_became_active":"2024-01-11T08:25:23.868230+0000","last_became_peered":"2024-01-11T08:25:23.868230+0000","last_unstale":"2024-01-11T08:25:23.876350+0000","last_undegraded":"2024-01-11T08:25:23.876350+0000","last_fullsized":"2024-01-11T08:25:23.876350+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:58:35.728365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.854627+0000","last_change":"2024-01-11T08:25:22.860300+0000","last_active":"2024-01-11T08:25:23.854627+0000","last_peered":"2024-01-11T08:25:23.854627+0000","last_clean":"2024-01-11T08:25:23.854627+0000","last_became_active":"2024-01-11T08:25:22.859664+0000","last_became_peered":"2024-01-11T08:25:22.859664+0000","last_unstale":"2024-01-11T08:25:23.854627+0000","last_undegraded":"2024-01-11T08:25:23.854627+0000","last_fullsized":"2024-01-11T08:25:23.854627+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:57:31.238419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.873578+0000","last_change":"2024-01-11T08:25:23.873578+0000","last_active":"2024-01-11T08:25:23.873578+0000","last_peered":"2024-01-11T08:25:23.873578+0000","last_clean":"2024-01-11T08:25:23.873578+0000","last_became_active":"2024-01-11T08:25:23.870192+0000","last_became_peered":"2024-01-11T08:25:23.870192+0000","last_unstale":"2024-01-11T08:25:23.873578+0000","last_undegraded":"2024-01-11T08:25:23.873578+0000","last_fullsized":"2024-01-11T08:25:23.873578+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:55:26.350726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-11T08:25:22.857672+0000","last_change":"2024-01-11T08:25:22.855065+0000","last_active":"2024-01-11T08:25:22.857672+0000","last_peered":"2024-01-11T08:25:22.857672+0000","last_clean":"2024-01-11T08:25:22.857672+0000","last_became_active":"2024-01-11T08:25:22.852962+0000","last_became_peered":"2024-01-11T08:25:22.852962+0000","last_unstale":"2024-01-11T08:25:22.857672+0000","last_undegraded":"2024-01-11T08:25:22.857672+0000","last_fullsized":"2024-01-11T08:25:22.857672+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:47:39.729413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.870190+0000","last_change":"2024-01-11T08:25:23.865663+0000","last_active":"2024-01-11T08:25:23.870190+0000","last_peered":"2024-01-11T08:25:23.870190+0000","last_clean":"2024-01-11T08:25:23.870190+0000","last_became_active":"2024-01-11T08:25:23.865174+0000","last_became_peered":"2024-01-11T08:25:23.865174+0000","last_unstale":"2024-01-11T08:25:23.870190+0000","last_undegraded":"2024-01-11T08:25:23.870190+0000","last_fullsized":"2024-01-11T08:25:23.870190+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:50:34.445851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.855983+0000","last_change":"2024-01-11T08:25:22.855435+0000","last_active":"2024-01-11T08:25:23.855983+0000","last_peered":"2024-01-11T08:25:23.855983+0000","last_clean":"2024-01-11T08:25:23.855983+0000","last_became_active":"2024-01-11T08:25:22.855192+0000","last_became_peered":"2024-01-11T08:25:22.855192+0000","last_unstale":"2024-01-11T08:25:23.855983+0000","last_undegraded":"2024-01-11T08:25:23.855983+0000","last_fullsized":"2024-01-11T08:25:23.855983+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:19:49.995942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.876134+0000","last_change":"2024-01-11T08:25:23.872878+0000","last_active":"2024-01-11T08:25:23.876134+0000","last_peered":"2024-01-11T08:25:23.876134+0000","last_clean":"2024-01-11T08:25:23.876134+0000","last_became_active":"2024-01-11T08:25:23.872474+0000","last_became_peered":"2024-01-11T08:25:23.872474+0000","last_unstale":"2024-01-11T08:25:23.876134+0000","last_undegraded":"2024-01-11T08:25:23.876134+0000","last_fullsized":"2024-01-11T08:25:23.876134+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:39:15.996906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-11T08:25:22.862890+0000","last_change":"2024-01-11T08:25:22.860552+0000","last_active":"2024-01-11T08:25:22.862890+0000","last_peered":"2024-01-11T08:25:22.862890+0000","last_clean":"2024-01-11T08:25:22.862890+0000","last_became_active":"2024-01-11T08:25:22.860297+0000","last_became_peered":"2024-01-11T08:25:22.860297+0000","last_unstale":"2024-01-11T08:25:22.862890+0000","last_undegraded":"2024-01-11T08:25:22.862890+0000","last_fullsized":"2024-01-11T08:25:22.862890+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:20:30.108010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.875884+0000","last_change":"2024-01-11T08:25:23.863643+0000","last_active":"2024-01-11T08:25:23.875884+0000","last_peered":"2024-01-11T08:25:23.875884+0000","last_clean":"2024-01-11T08:25:23.875884+0000","last_became_active":"2024-01-11T08:25:23.861786+0000","last_became_peered":"2024-01-11T08:25:23.861786+0000","last_unstale":"2024-01-11T08:25:23.875884+0000","last_undegraded":"2024-01-11T08:25:23.875884+0000","last_fullsized":"2024-01-11T08:25:23.875884+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:42:53.264984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.858272+0000","last_change":"2024-01-11T08:25:22.860686+0000","last_active":"2024-01-11T08:25:23.858272+0000","last_peered":"2024-01-11T08:25:23.858272+0000","last_clean":"2024-01-11T08:25:23.858272+0000","last_became_active":"2024-01-11T08:25:22.860417+0000","last_became_peered":"2024-01-11T08:25:22.860417+0000","last_unstale":"2024-01-11T08:25:23.858272+0000","last_undegraded":"2024-01-11T08:25:23.858272+0000","last_fullsized":"2024-01-11T08:25:23.858272+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:43:04.458768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.867426+0000","last_change":"2024-01-11T08:25:23.859836+0000","last_active":"2024-01-11T08:25:23.867426+0000","last_peered":"2024-01-11T08:25:23.867426+0000","last_clean":"2024-01-11T08:25:23.867426+0000","last_became_active":"2024-01-11T08:25:23.858584+0000","last_became_peered":"2024-01-11T08:25:23.858584+0000","last_unstale":"2024-01-11T08:25:23.867426+0000","last_undegraded":"2024-01-11T08:25:23.867426+0000","last_fullsized":"2024-01-11T08:25:23.867426+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:53:58.821688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.841751+0000","last_change":"2024-01-11T08:25:22.856991+0000","last_active":"2024-01-11T08:25:23.841751+0000","last_peered":"2024-01-11T08:25:23.841751+0000","last_clean":"2024-01-11T08:25:23.841751+0000","last_became_active":"2024-01-11T08:25:22.854384+0000","last_became_peered":"2024-01-11T08:25:22.854384+0000","last_unstale":"2024-01-11T08:25:23.841751+0000","last_undegraded":"2024-01-11T08:25:23.841751+0000","last_fullsized":"2024-01-11T08:25:23.841751+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:11:45.524653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.867961+0000","last_change":"2024-01-11T08:25:23.860721+0000","last_active":"2024-01-11T08:25:23.867961+0000","last_peered":"2024-01-11T08:25:23.867961+0000","last_clean":"2024-01-11T08:25:23.867961+0000","last_became_active":"2024-01-11T08:25:23.860438+0000","last_became_peered":"2024-01-11T08:25:23.860438+0000","last_unstale":"2024-01-11T08:25:23.867961+0000","last_undegraded":"2024-01-11T08:25:23.867961+0000","last_fullsized":"2024-01-11T08:25:23.867961+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:59.235791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.854360+0000","last_change":"2024-01-11T08:25:22.858950+0000","last_active":"2024-01-11T08:25:23.854360+0000","last_peered":"2024-01-11T08:25:23.854360+0000","last_clean":"2024-01-11T08:25:23.854360+0000","last_became_active":"2024-01-11T08:25:22.858632+0000","last_became_peered":"2024-01-11T08:25:22.858632+0000","last_unstale":"2024-01-11T08:25:23.854360+0000","last_undegraded":"2024-01-11T08:25:23.854360+0000","last_fullsized":"2024-01-11T08:25:23.854360+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:20:11.856429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"66'161","reported_seq":361,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.867154+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:25:23.867154+0000","last_peered":"2024-01-11T08:25:23.867154+0000","last_clean":"2024-01-11T08:25:23.867154+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:25:23.867154+0000","last_undegraded":"2024-01-11T08:25:23.867154+0000","last_fullsized":"2024-01-11T08:25:23.867154+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575030,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302856,"kb_used_data":1472,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068984,"statfs":{"total":96636764160,"available":96326639616,"internally_reserved":0,"allocated":1507328,"data_stored":1007067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.71899999999999997,"15min":0.70799999999999996},"min":{"1min":0.45300000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.95899999999999996,"5min":0.996,"15min":2.8300000000000001},"last":0.62},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.71499999999999997,"15min":0.70999999999999996},"min":{"1min":0.55600000000000005,"5min":0.40500000000000003,"15min":0.372},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.54500000000000004}]},{"osd":1,"last update":"Thu Jan 11 08:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.71399999999999997,"15min":0.70399999999999996},"min":{"1min":0.53300000000000003,"5min":0.41899999999999998,"15min":0.35799999999999998},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.998},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71999999999999997,"15min":0.72099999999999997},"min":{"1min":0.58299999999999996,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93999999999999995,"5min":0.95899999999999996,"15min":1.153},"last":0.59199999999999997}]},{"osd":2,"last update":"Thu Jan 11 08:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.73099999999999998,"15min":0.72799999999999998},"min":{"1min":0.68700000000000006,"5min":0.436,"15min":0.436},"max":{"1min":0.96499999999999997,"5min":1.03,"15min":1.034},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.73099999999999998,"15min":0.72099999999999997},"min":{"1min":0.54800000000000004,"5min":0.46999999999999997,"15min":0.41099999999999998},"max":{"1min":0.98299999999999998,"5min":0.99399999999999999,"15min":1.0049999999999999},"last":0.55700000000000005}]},{"osd":3,"last update":"Thu Jan 11 08:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.746,"15min":0.75600000000000001},"min":{"1min":0.56899999999999995,"5min":0.48299999999999998,"15min":0.40300000000000002},"max":{"1min":0.876,"5min":1.032,"15min":2.9119999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.73299999999999998,"15min":0.74199999999999999},"min":{"1min":0.64700000000000002,"5min":0.50800000000000001,"15min":0.48999999999999999},"max":{"1min":0.92000000000000004,"5min":1.008,"15min":1.008},"last":0.64300000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.47099999999999997,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":1.089},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.63500000000000001,"15min":0.63600000000000001},"min":{"1min":0.48099999999999998,"5min":0.42699999999999999,"15min":0.39500000000000002},"max":{"1min":0.85999999999999999,"5min":0.90700000000000003,"15min":1.1020000000000001},"last":0.52600000000000002}]},{"osd":5,"last update":"Thu Jan 11 08:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.63200000000000001,"15min":0.64700000000000002},"min":{"1min":0.42299999999999999,"5min":0.42099999999999999,"15min":0.34599999999999997},"max":{"1min":0.85399999999999998,"5min":0.92200000000000004,"15min":1.0589999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.67200000000000004,"15min":0.68300000000000005},"min":{"1min":0.47799999999999998,"5min":0.47199999999999998,"15min":0.40400000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.032},"last":0.63200000000000001}]},{"osd":6,"last update":"Thu Jan 11 08:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.748,"15min":0.76300000000000001},"min":{"1min":0.64100000000000001,"5min":0.52000000000000002,"15min":0.46999999999999997},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.1359999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73199999999999998,"15min":0.74399999999999999},"min":{"1min":0.44900000000000001,"5min":0.437,"15min":0.42099999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.1299999999999999},"last":0.70299999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575029,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302256,"kb_used_data":872,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069584,"statfs":{"total":96636764160,"available":96327254016,"internally_reserved":0,"allocated":892928,"data_stored":410223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.66800000000000004,"15min":0.66000000000000003},"min":{"1min":0.56499999999999995,"5min":0.379,"15min":0.29999999999999999},"max":{"1min":0.87,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68200000000000005,"15min":0.68100000000000005},"min":{"1min":0.41499999999999998,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":0.92800000000000005,"5min":0.96999999999999997,"15min":1},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Jan 11 08:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68999999999999995,"15min":0.68100000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.35899999999999999},"max":{"1min":0.93200000000000005,"5min":1.163,"15min":1.163},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.69199999999999995,"15min":0.68999999999999995},"min":{"1min":0.53500000000000003,"5min":0.435,"15min":0.39900000000000002},"max":{"1min":0.81000000000000005,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.81599999999999995}]},{"osd":2,"last update":"Thu Jan 11 08:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70699999999999996,"15min":0.70299999999999996},"min":{"1min":0.501,"5min":0.46899999999999997,"15min":0.38600000000000001},"max":{"1min":0.95399999999999996,"5min":0.97199999999999998,"15min":1.016},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71799999999999997,"15min":0.71999999999999997},"min":{"1min":0.51900000000000002,"5min":0.46200000000000002,"15min":0.373},"max":{"1min":0.91100000000000003,"5min":0.96699999999999997,"15min":0.995},"last":0.64700000000000002}]},{"osd":3,"last update":"Thu Jan 11 08:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72399999999999998,"15min":0.73399999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.42799999999999999},"max":{"1min":0.86899999999999999,"5min":0.95799999999999996,"15min":1.3260000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.73999999999999999,"15min":0.73399999999999999},"min":{"1min":0.55000000000000004,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.90900000000000003,"5min":0.97999999999999998,"15min":1.0049999999999999},"last":0.84199999999999997}]},{"osd":4,"last update":"Thu Jan 11 08:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.54500000000000004,"15min":0.57299999999999995},"min":{"1min":0.41899999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.61399999999999999,"5min":0.874,"15min":1.212},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58099999999999996,"15min":0.61599999999999999},"min":{"1min":0.438,"5min":0.38900000000000001,"15min":0.374},"max":{"1min":0.70999999999999996,"5min":0.96399999999999997,"15min":1.8320000000000001},"last":0.60099999999999998}]},{"osd":5,"last update":"Thu Jan 11 08:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63700000000000001,"15min":0.65800000000000003},"min":{"1min":0.432,"5min":0.432,"15min":0.39900000000000002},"max":{"1min":0.84999999999999998,"5min":0.95499999999999996,"15min":1.0209999999999999},"last":1.03},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.68300000000000005,"15min":0.70799999999999996},"min":{"1min":0.50600000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88200000000000001,"5min":0.89500000000000002,"15min":1.7929999999999999},"last":0.90800000000000003}]},{"osd":7,"last update":"Thu Jan 11 08:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.69399999999999995,"15min":0.71899999999999997},"min":{"1min":0.53200000000000003,"5min":0.51800000000000002,"15min":0.41099999999999998},"max":{"1min":0.86699999999999999,"5min":0.879,"15min":1.278},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.73399999999999999,"15min":0.751},"min":{"1min":0.46700000000000003,"5min":0.45900000000000002,"15min":0.44700000000000001},"max":{"1min":0.98999999999999999,"5min":1.022,"15min":1.1599999999999999},"last":1.0009999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575031,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302272,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069568,"statfs":{"total":96636764160,"available":96327237632,"internally_reserved":0,"allocated":909312,"data_stored":416629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.501,"15min":0.49099999999999999},"min":{"1min":0.42199999999999999,"5min":0.30599999999999999,"15min":0.26900000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.97699999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.58599999999999997,"15min":0.57899999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.373},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.497,"15min":0.48999999999999999},"min":{"1min":0.36099999999999999,"5min":0.312,"15min":0.312},"max":{"1min":1.0960000000000001,"5min":1.113,"15min":1.113},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.56399999999999995,"15min":0.56899999999999995},"min":{"1min":0.42399999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76700000000000002,"5min":0.82399999999999995,"15min":1.25},"last":0.69699999999999995}]},{"osd":3,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.51000000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55400000000000005,"15min":0.56200000000000006},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.34899999999999998},"max":{"1min":0.78100000000000003,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.73199999999999998}]},{"osd":4,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.78100000000000003,"15min":0.78700000000000003},"min":{"1min":0.53300000000000003,"5min":0.439,"15min":0.439},"max":{"1min":1.0800000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.82799999999999996,"15min":0.83199999999999996},"min":{"1min":0.63500000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.0329999999999999,"5min":1.246,"15min":1.3680000000000001},"last":0.76800000000000002}]},{"osd":5,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78600000000000003,"15min":0.80300000000000005},"min":{"1min":0.57799999999999996,"5min":0.502,"15min":0.46999999999999997},"max":{"1min":1.0649999999999999,"5min":1.1759999999999999,"15min":1.7849999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.79300000000000004,"15min":0.80100000000000005},"min":{"1min":0.52900000000000003,"5min":0.52500000000000002,"15min":0.50700000000000001},"max":{"1min":1.1339999999999999,"5min":1.1579999999999999,"15min":1.371},"last":0.83699999999999997}]},{"osd":6,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.81899999999999995,"15min":0.82199999999999995},"min":{"1min":0.70899999999999996,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0289999999999999,"5min":1.0840000000000001,"15min":1.413},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.82899999999999996,"15min":0.82599999999999996},"min":{"1min":0.621,"5min":0.621,"15min":0.55500000000000005},"max":{"1min":1.0469999999999999,"5min":1.228,"15min":1.228},"last":0.77900000000000003}]},{"osd":7,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84599999999999997},"min":{"1min":0.625,"5min":0.59199999999999997,"15min":0.52700000000000002},"max":{"1min":1.0109999999999999,"5min":1.147,"15min":1.226},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.84199999999999997,"15min":0.84099999999999997},"min":{"1min":0.65800000000000003,"5min":0.58599999999999997,"15min":0.51400000000000001},"max":{"1min":1.0669999999999999,"5min":1.258,"15min":1.321},"last":0.92200000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575030,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298756,"kb_used_data":1468,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073084,"statfs":{"total":96636764160,"available":96330838016,"internally_reserved":0,"allocated":1503232,"data_stored":1002681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,9],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.495,"15min":0.499},"min":{"1min":0.34899999999999998,"5min":0.29999999999999999,"15min":0.24299999999999999},"max":{"1min":0.74099999999999999,"5min":0.77700000000000002,"15min":0.92700000000000005},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.502,"5min":0.41299999999999998,"15min":0.35299999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.92100000000000004},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54400000000000004,"15min":0.54900000000000004},"min":{"1min":0.42799999999999999,"5min":0.34799999999999998,"15min":0.29099999999999998},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.86799999999999999},"last":1.335},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.53900000000000003,"15min":0.54900000000000004},"min":{"1min":0.44600000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69999999999999996,"5min":0.78500000000000003,"15min":0.83999999999999997},"last":0.68600000000000005}]},{"osd":3,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.52000000000000002,"15min":0.52200000000000002},"min":{"1min":0.39200000000000002,"5min":0.35999999999999999,"15min":0.33600000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.621,"15min":0.622},"min":{"1min":0.48899999999999999,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84899999999999998,"5min":0.877,"15min":0.91000000000000003},"last":1.4119999999999999}]},{"osd":4,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.79400000000000004,"15min":0.78900000000000003},"min":{"1min":0.64300000000000002,"5min":0.45700000000000002,"15min":0.437},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.79800000000000004,"15min":0.79000000000000004},"min":{"1min":0.64900000000000002,"5min":0.53100000000000003,"15min":0.46899999999999997},"max":{"1min":1.0209999999999999,"5min":1.038,"15min":1.2350000000000001},"last":0.84499999999999997}]},{"osd":5,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.79800000000000004,"15min":0.80600000000000005},"min":{"1min":0.69399999999999995,"5min":0.52500000000000002,"15min":0.505},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.2929999999999999},"last":1.518},{"interface":"front","average":{"1min":0.88,"5min":0.82099999999999995,"15min":0.81799999999999995},"min":{"1min":0.68300000000000005,"5min":0.49099999999999999,"15min":0.47799999999999998},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.85899999999999999}]},{"osd":6,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.80700000000000005,"15min":0.80100000000000005},"min":{"1min":0.70399999999999996,"5min":0.50900000000000001,"15min":0.497},"max":{"1min":1.163,"5min":1.163,"15min":1.2689999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.82699999999999996,"15min":0.83399999999999996},"min":{"1min":0.73899999999999999,"5min":0.54700000000000004,"15min":0.44900000000000001},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.2749999999999999},"last":1.5589999999999999}]},{"osd":7,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.83799999999999997,"15min":0.84099999999999997},"min":{"1min":0.63200000000000001,"5min":0.57999999999999996,"15min":0.51400000000000001},"max":{"1min":1.0329999999999999,"5min":1.071,"15min":1.446},"last":1.024},{"interface":"front","average":{"1min":0.877,"5min":0.84499999999999997,"15min":0.84299999999999997},"min":{"1min":0.64100000000000001,"5min":0.60499999999999998,"15min":0.57699999999999996},"max":{"1min":1.1539999999999999,"5min":1.1659999999999999,"15min":1.298},"last":1.0760000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306372,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065468,"statfs":{"total":96636764160,"available":96323039232,"internally_reserved":0,"allocated":913408,"data_stored":416595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:24:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.47899999999999998,"15min":0.47099999999999997},"min":{"1min":0.38900000000000001,"5min":0.29899999999999999,"15min":0.26000000000000001},"max":{"1min":0.70499999999999996,"5min":0.81399999999999995,"15min":0.86099999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53600000000000003,"15min":0.55200000000000005},"min":{"1min":0.34399999999999997,"5min":0.33100000000000002,"15min":0.28899999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":1.4119999999999999},"last":0.45900000000000002}]},{"osd":1,"last update":"Thu Jan 11 08:24:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54100000000000004,"15min":0.54900000000000004},"min":{"1min":0.374,"5min":0.36399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68500000000000005,"5min":0.80300000000000005,"15min":0.90000000000000002},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.57599999999999996,"15min":0.59099999999999997},"min":{"1min":0.47599999999999998,"5min":0.39200000000000002,"15min":0.38100000000000001},"max":{"1min":0.75700000000000001,"5min":0.92900000000000005,"15min":1.5640000000000001},"last":0.52500000000000002}]},{"osd":3,"last update":"Thu Jan 11 08:24:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53900000000000003,"15min":0.54900000000000004},"min":{"1min":0.39400000000000002,"5min":0.36199999999999999,"15min":0.32900000000000001},"max":{"1min":0.71299999999999997,"5min":0.82499999999999996,"15min":1.1220000000000001},"last":0.499},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.624,"15min":0.64500000000000002},"min":{"1min":0.47099999999999997,"5min":0.45300000000000001,"15min":0.41799999999999998},"max":{"1min":0.80700000000000005,"5min":0.95299999999999996,"15min":1.22},"last":0.435}]},{"osd":4,"last update":"Thu Jan 11 08:24:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.78700000000000003,"15min":0.79500000000000004},"min":{"1min":0.55400000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.98699999999999999,"5min":1.0680000000000001,"15min":1.488},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.78200000000000003,"15min":0.78600000000000003},"min":{"1min":0.57799999999999996,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0660000000000001,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.74399999999999999}]},{"osd":5,"last update":"Thu Jan 11 08:24:42 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.83199999999999996,"15min":0.85099999999999998},"min":{"1min":0.70299999999999996,"5min":0.622,"15min":0.51600000000000001},"max":{"1min":1.087,"5min":1.214,"15min":2.9209999999999998},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.878,"5min":0.84499999999999997,"15min":0.85599999999999998},"min":{"1min":0.67200000000000004,"5min":0.55500000000000005,"15min":0.55400000000000005},"max":{"1min":1.2,"5min":1.2,"15min":1.845},"last":0.872}]},{"osd":6,"last update":"Thu Jan 11 08:24:42 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.81000000000000005,"15min":0.80600000000000005},"min":{"1min":0.54300000000000004,"5min":0.54200000000000004,"15min":0.51700000000000002},"max":{"1min":1.131,"5min":1.1919999999999999,"15min":1.3560000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82299999999999995,"15min":0.84299999999999997},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0529999999999999,"5min":1.123,"15min":2.968},"last":0.72399999999999998}]},{"osd":7,"last update":"Thu Jan 11 08:24:42 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.86599999999999999,"15min":0.879},"min":{"1min":0.71799999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.6359999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.85299999999999998,"15min":0.86899999999999999},"min":{"1min":0.63700000000000001,"5min":0.58699999999999997,"15min":0.53800000000000003},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.391},"last":0.85899999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575029,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306372,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065468,"statfs":{"total":96636764160,"available":96323039232,"internally_reserved":0,"allocated":913408,"data_stored":416595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.502,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.29499999999999998,"15min":0.27600000000000002},"max":{"1min":0.79800000000000004,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53600000000000003,"15min":0.54600000000000004},"min":{"1min":0.42199999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67400000000000004,"5min":0.72199999999999998,"15min":1.0109999999999999},"last":0.49299999999999999}]},{"osd":1,"last update":"Thu Jan 11 08:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46300000000000002,"15min":0.48599999999999999},"min":{"1min":0.34000000000000002,"5min":0.30299999999999999,"15min":0.25900000000000001},"max":{"1min":0.626,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58799999999999997,"15min":0.57099999999999995},"min":{"1min":0.45400000000000001,"5min":0.36299999999999999,"15min":0.32300000000000001},"max":{"1min":0.74299999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.40699999999999997}]},{"osd":2,"last update":"Thu Jan 11 08:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.46300000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.74099999999999999,"5min":0.90900000000000003,"15min":0.97599999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63500000000000001,"15min":0.63100000000000001},"min":{"1min":0.48599999999999999,"5min":0.44700000000000001,"15min":0.32000000000000001},"max":{"1min":0.86199999999999999,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.45900000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.71799999999999997,"15min":0.749},"min":{"1min":0.51700000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.038,"5min":1.038,"15min":1.351},"last":0.622},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.77200000000000002,"15min":0.78200000000000003},"min":{"1min":0.56599999999999995,"5min":0.503,"15min":0.503},"max":{"1min":0.92100000000000004,"5min":1.1819999999999999,"15min":1.198},"last":0.67600000000000005}]},{"osd":5,"last update":"Thu Jan 11 08:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.76800000000000002,"15min":0.79300000000000004},"min":{"1min":0.55800000000000005,"5min":0.53500000000000003,"15min":0.51900000000000002},"max":{"1min":0.96499999999999997,"5min":1.139,"15min":1.3779999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79400000000000004,"15min":0.81000000000000005},"min":{"1min":0.64300000000000002,"5min":0.49199999999999999,"15min":0.44900000000000001},"max":{"1min":1.016,"5min":1.2809999999999999,"15min":1.4239999999999999},"last":0.63600000000000001}]},{"osd":6,"last update":"Thu Jan 11 08:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.82399999999999995,"15min":0.83999999999999997},"min":{"1min":0.625,"5min":0.56799999999999995,"15min":0.52600000000000002},"max":{"1min":1.093,"5min":1.335,"15min":1.927},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.81999999999999995},"min":{"1min":0.61099999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.97599999999999998,"5min":1.2230000000000001,"15min":3.5579999999999998},"last":0.57999999999999996}]},{"osd":7,"last update":"Thu Jan 11 08:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81799999999999995,"15min":0.85499999999999998},"min":{"1min":0.55000000000000004,"5min":0.52800000000000002,"15min":0.46600000000000003},"max":{"1min":1.081,"5min":1.1579999999999999,"15min":3.5089999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.88,"5min":0.84699999999999998,"15min":0.86199999999999999},"min":{"1min":0.70299999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.0609999999999999,"5min":1.1879999999999999,"15min":1.21},"last":0.69899999999999995}]}]},{"osd":4,"up_from":13,"seq":55834575030,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310460,"kb_used_data":884,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94061380,"statfs":{"total":96636764160,"available":96318853120,"internally_reserved":0,"allocated":905216,"data_stored":411759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.748,"15min":0.74099999999999999},"min":{"1min":0.47299999999999998,"5min":0.38900000000000001,"15min":0.28100000000000003},"max":{"1min":0.89600000000000002,"5min":0.95999999999999996,"15min":1.524},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.74299999999999999,"15min":0.748},"min":{"1min":0.54800000000000004,"5min":0.51600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.177},"last":0.72299999999999998}]},{"osd":1,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.70199999999999996,"15min":0.70299999999999996},"min":{"1min":0.52100000000000002,"5min":0.42199999999999999,"15min":0.33300000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.1499999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.70899999999999996,"15min":0.73499999999999999},"min":{"1min":0.52500000000000002,"5min":0.379,"15min":0.36299999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":2.4740000000000002},"last":0.43099999999999999}]},{"osd":2,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.72199999999999998,"15min":0.71699999999999997},"min":{"1min":0.54800000000000004,"5min":0.46000000000000002,"15min":0.38900000000000001},"max":{"1min":0.97499999999999998,"5min":0.98599999999999999,"15min":1.022},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.747,"5min":0.751,"15min":0.73499999999999999},"min":{"1min":0.54000000000000004,"5min":0.48499999999999999,"15min":0.47199999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.212},"last":0.53700000000000003}]},{"osd":3,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70299999999999996,"15min":0.71199999999999997},"min":{"1min":0.497,"5min":0.433,"15min":0.38100000000000001},"max":{"1min":0.81599999999999995,"5min":1.0189999999999999,"15min":1.2649999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.75,"15min":0.75800000000000001},"min":{"1min":0.63800000000000001,"5min":0.50900000000000001,"15min":0.47499999999999998},"max":{"1min":1.026,"5min":1.026,"15min":1.069},"last":0.502}]},{"osd":5,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.60699999999999998,"15min":0.626},"min":{"1min":0.45000000000000001,"5min":0.38700000000000001,"15min":0.35899999999999999},"max":{"1min":0.873,"5min":0.873,"15min":1.0700000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.621,"15min":0.63900000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.32600000000000001},"max":{"1min":0.80100000000000005,"5min":0.97799999999999998,"15min":0.98199999999999998},"last":0.78100000000000003}]},{"osd":6,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.65000000000000002,"15min":0.66200000000000003},"min":{"1min":0.439,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93100000000000005,"5min":1.133,"15min":1.133},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.68600000000000005,"15min":0.70099999999999996},"min":{"1min":0.497,"5min":0.42799999999999999,"15min":0.40400000000000003},"max":{"1min":0.88300000000000001,"5min":1.0269999999999999,"15min":1.119},"last":0.68600000000000005}]},{"osd":7,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68100000000000005,"15min":0.69999999999999996},"min":{"1min":0.47099999999999997,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85099999999999998,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.747,"5min":0.74299999999999999,"15min":0.752},"min":{"1min":0.49299999999999999,"5min":0.46200000000000002,"15min":0.42299999999999999},"max":{"1min":1.127,"5min":1.2350000000000001,"15min":1.4279999999999999},"last":0.75600000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575030,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302264,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069576,"statfs":{"total":96636764160,"available":96327245824,"internally_reserved":0,"allocated":901120,"data_stored":410847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.71399999999999997,"15min":0.70899999999999996},"min":{"1min":0.52700000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.95399999999999996,"5min":1.0249999999999999,"15min":1.032},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.70899999999999996,"15min":0.70799999999999996},"min":{"1min":0.54500000000000004,"5min":0.437,"15min":0.437},"max":{"1min":0.96199999999999997,"5min":1.0609999999999999,"15min":1.696},"last":0.58199999999999996}]},{"osd":1,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.68600000000000005,"15min":0.70099999999999996},"min":{"1min":0.48999999999999999,"5min":0.39400000000000002,"15min":0.36399999999999999},"max":{"1min":0.84799999999999998,"5min":1.0089999999999999,"15min":1.4359999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.72999999999999998,"15min":0.73499999999999999},"min":{"1min":0.5,"5min":0.48299999999999998,"15min":0.47299999999999998},"max":{"1min":0.90300000000000002,"5min":0.97399999999999998,"15min":4.4969999999999999},"last":0.61099999999999999}]},{"osd":2,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.72099999999999997,"15min":0.70899999999999996},"min":{"1min":0.48099999999999998,"5min":0.435,"15min":0.435},"max":{"1min":0.95999999999999996,"5min":1.028,"15min":1.238},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.70999999999999996,"15min":0.749},"min":{"1min":0.52200000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.90200000000000002,"5min":1.0049999999999999,"15min":8.0429999999999993},"last":0.55500000000000005}]},{"osd":3,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.72199999999999998,"15min":0.76000000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.442},"max":{"1min":0.88700000000000001,"5min":1.0449999999999999,"15min":8.0839999999999996},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.746,"15min":0.748},"min":{"1min":0.439,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.98499999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.64600000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.60399999999999998,"15min":0.61799999999999999},"min":{"1min":0.41699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.71799999999999997,"5min":0.96599999999999997,"15min":1.113},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":0.82499999999999996,"5min":0.88600000000000001,"15min":1.002},"last":0.752}]},{"osd":6,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.65200000000000002,"15min":0.65400000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80800000000000005,"5min":0.86899999999999999,"15min":1.034},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.68400000000000005,"15min":0.69399999999999995},"min":{"1min":0.499,"5min":0.499,"15min":0.45400000000000001},"max":{"1min":0.77800000000000002,"5min":0.94599999999999995,"15min":1.143},"last":0.67800000000000005}]},{"osd":7,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.73699999999999999,"15min":0.747},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.437},"max":{"1min":0.875,"5min":1.028,"15min":1.1399999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.747,"5min":0.77800000000000002,"15min":0.77500000000000002},"min":{"1min":0.51200000000000001,"5min":0.5,"15min":0.5},"max":{"1min":0.92500000000000004,"5min":0.96699999999999997,"15min":1.079},"last":0.76000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:25:28.274 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:25:28.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.510+0000 7f8265bff700 1 Processor -- start 2024-01-11T08:25:28.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.510+0000 7f8265bff700 1 -- start start 2024-01-11T08:25:28.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.511+0000 7f8265bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f82600ca7b0 0x7f82600cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:28.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.511+0000 7f8265bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8260159e80 0x7f826015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:28.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.511+0000 7f8265bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826015d640 con 0x7f82600ca3e0 2024-01-11T08:25:28.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.511+0000 7f8265bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f826015ae70 con 0x7f82600ca7b0 2024-01-11T08:25:28.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.511+0000 7f8265bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82600cc4f0 con 0x7f8260159e80 2024-01-11T08:25:28.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.512+0000 7f825effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f82600ca7b0 0x7f82600cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:28.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.512+0000 7f825f7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f82600ca3e0 0x7f82600c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46364/0 (socket says 172.21.15.27:46364) 2024-01-11T08:25:28.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.512+0000 7f825f7fe700 1 -- 172.21.15.27:0/3543240239 learned_addr learned my addr 172.21.15.27:0/3543240239 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:28.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.512+0000 7f825ffff700 1 --2- 172.21.15.27:0/3543240239 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8260159e80 0x7f826015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:28.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.512+0000 7f825effd700 1 -- 172.21.15.27:0/3543240239 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8260159e80 msgr2=0x7f826015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:28.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.512+0000 7f825effd700 1 --2- 172.21.15.27:0/3543240239 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8260159e80 0x7f826015dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.512+0000 7f825effd700 1 -- 172.21.15.27:0/3543240239 >> v1:172.21.15.27:6789/0 conn(0x7f82600ca3e0 legacy=0x7f82600c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:28.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.512+0000 7f825effd700 1 -- 172.21.15.27:0/3543240239 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826015a5d0 con 0x7f82600ca7b0 2024-01-11T08:25:28.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.512+0000 7f825e7fc700 1 -- 172.21.15.27:0/3543240239 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523053169 0 0) 0x7f826015d640 con 0x7f82600ca3e0 2024-01-11T08:25:28.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.512+0000 7f825effd700 1 --2- 172.21.15.27:0/3543240239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82600ca7b0 0x7f82600cca20 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f8254009940 tx=0x7f82540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a383d51fba35e4f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:28.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.513+0000 7f825e7fc700 1 -- 172.21.15.27:0/3543240239 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825400b7e0 con 0x7f82600ca7b0 2024-01-11T08:25:28.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.513+0000 7f825e7fc700 1 -- 172.21.15.27:0/3543240239 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825400b940 con 0x7f82600ca7b0 2024-01-11T08:25:28.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.513+0000 7f825e7fc700 1 -- 172.21.15.27:0/3543240239 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825400bce0 con 0x7f82600ca7b0 2024-01-11T08:25:28.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.513+0000 7f8265bff700 1 -- 172.21.15.27:0/3543240239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82600ca7b0 msgr2=0x7f82600cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:28.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.513+0000 7f8265bff700 1 --2- 172.21.15.27:0/3543240239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82600ca7b0 0x7f82600cca20 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f8254009940 tx=0x7f82540091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.513+0000 7f8265bff700 1 -- 172.21.15.27:0/3543240239 shutdown_connections 2024-01-11T08:25:28.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.513+0000 7f8265bff700 1 --2- 172.21.15.27:0/3543240239 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8260159e80 0x7f826015dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.513+0000 7f8265bff700 1 --2- 172.21.15.27:0/3543240239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82600ca7b0 0x7f82600cca20 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.513+0000 7f8265bff700 1 -- 172.21.15.27:0/3543240239 >> 172.21.15.27:0/3543240239 conn(0x7f82600ce3e0 msgr2=0x7f82600ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:28.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.513+0000 7f8265bff700 1 -- 172.21.15.27:0/3543240239 shutdown_connections 2024-01-11T08:25:28.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.513+0000 7f8265bff700 1 -- 172.21.15.27:0/3543240239 wait complete. 2024-01-11T08:25:28.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.515+0000 7f8265bff700 1 Processor -- start 2024-01-11T08:25:28.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.515+0000 7f8265bff700 1 -- start start 2024-01-11T08:25:28.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.516+0000 7f8265bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82600ca3e0 0x7f82600b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:28.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.516+0000 7f8265bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f82600ca7b0 0x7f82600c4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:28.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.516+0000 7f8265bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8260159e80 0x7f826013a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:28.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.516+0000 7f8265bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82600b1530 con 0x7f82600ca3e0 2024-01-11T08:25:28.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.516+0000 7f8265bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826014a340 con 0x7f8260159e80 2024-01-11T08:25:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.516+0000 7f8265bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826016e1e0 con 0x7f82600ca7b0 2024-01-11T08:25:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.516+0000 7f825f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82600ca3e0 0x7f82600b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.516+0000 7f825f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82600ca3e0 0x7f82600b3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56964/0 (socket says 172.21.15.27:56964) 2024-01-11T08:25:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.516+0000 7f825f7fe700 1 -- 172.21.15.27:0/246454418 learned_addr learned my addr 172.21.15.27:0/246454418 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.516+0000 7f825effd700 1 --2- 172.21.15.27:0/246454418 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f82600ca7b0 0x7f82600c4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.517+0000 7f825ffff700 1 --2- 172.21.15.27:0/246454418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8260159e80 0x7f826013a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.517+0000 7f825f7fe700 1 -- 172.21.15.27:0/246454418 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f82600ca7b0 msgr2=0x7f82600c4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:28.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.517+0000 7f825f7fe700 1 --2- 172.21.15.27:0/246454418 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f82600ca7b0 0x7f82600c4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.517+0000 7f825f7fe700 1 -- 172.21.15.27:0/246454418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8260159e80 msgr2=0x7f826013a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:28.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.517+0000 7f825f7fe700 1 --2- 172.21.15.27:0/246454418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8260159e80 0x7f826013a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.517+0000 7f825f7fe700 1 -- 172.21.15.27:0/246454418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8254009420 con 0x7f82600ca3e0 2024-01-11T08:25:28.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.517+0000 7f825f7fe700 1 --2- 172.21.15.27:0/246454418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82600ca3e0 0x7f82600b3c60 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f825000c7e0 tx=0x7f8250007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:28.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.517+0000 7f82477fe700 1 -- 172.21.15.27:0/246454418 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8250005b30 con 0x7f82600ca3e0 2024-01-11T08:25:28.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.517+0000 7f82477fe700 1 -- 172.21.15.27:0/246454418 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8250005c90 con 0x7f82600ca3e0 2024-01-11T08:25:28.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.517+0000 7f82477fe700 1 -- 172.21.15.27:0/246454418 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8250007300 con 0x7f82600ca3e0 2024-01-11T08:25:28.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.517+0000 7f8265bff700 1 -- 172.21.15.27:0/246454418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826016d5d0 con 0x7f82600ca3e0 2024-01-11T08:25:28.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.517+0000 7f8265bff700 1 -- 172.21.15.27:0/246454418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f826016db90 con 0x7f82600ca3e0 2024-01-11T08:25:28.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.518+0000 7f8265bff700 1 -- 172.21.15.27:0/246454418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82600718d0 con 0x7f82600ca3e0 2024-01-11T08:25:28.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.519+0000 7f82477fe700 1 -- 172.21.15.27:0/246454418 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f825001b3b0 con 0x7f82600ca3e0 2024-01-11T08:25:28.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.520+0000 7f82477fe700 1 --2- 172.21.15.27:0/246454418 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f824c066f30 0x7f824c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:28.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.520+0000 7f82477fe700 1 -- 172.21.15.27:0/246454418 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f8250083b10 con 0x7f82600ca3e0 2024-01-11T08:25:28.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.521+0000 7f825effd700 1 --2- 172.21.15.27:0/246454418 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f824c066f30 0x7f824c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:28.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.521+0000 7f825effd700 1 --2- 172.21.15.27:0/246454418 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f824c066f30 0x7f824c0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8254009860 tx=0x7f8254017000 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:28.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.524+0000 7f82477fe700 1 -- 172.21.15.27:0/246454418 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8250050cc0 con 0x7f82600ca3e0 2024-01-11T08:25:28.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.723+0000 7f8265bff700 1 -- 172.21.15.27:0/246454418 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f826015a8b0 con 0x7f824c066f30 2024-01-11T08:25:28.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.732+0000 7f82477fe700 1 -- 172.21.15.27:0/246454418 <== mgr.5450 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289029 (secure 0 0 0) 0x7f826015a8b0 con 0x7f824c066f30 2024-01-11T08:25:28.736 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.736+0000 7f8265bff700 1 -- 172.21.15.27:0/246454418 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f824c066f30 msgr2=0x7f824c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.736+0000 7f8265bff700 1 --2- 172.21.15.27:0/246454418 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f824c066f30 0x7f824c0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8254009860 tx=0x7f8254017000 comp rx=0 tx=0).stop 2024-01-11T08:25:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.736+0000 7f8265bff700 1 -- 172.21.15.27:0/246454418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82600ca3e0 msgr2=0x7f82600b3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.736+0000 7f8265bff700 1 --2- 172.21.15.27:0/246454418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82600ca3e0 0x7f82600b3c60 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f825000c7e0 tx=0x7f8250007df0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.736+0000 7f8265bff700 1 -- 172.21.15.27:0/246454418 shutdown_connections 2024-01-11T08:25:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.736+0000 7f8265bff700 1 --2- 172.21.15.27:0/246454418 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f824c066f30 0x7f824c0693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.736+0000 7f8265bff700 1 --2- 172.21.15.27:0/246454418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8260159e80 0x7f826013a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.736+0000 7f8265bff700 1 --2- 172.21.15.27:0/246454418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82600ca3e0 0x7f82600b3c60 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.736+0000 7f8265bff700 1 --2- 172.21.15.27:0/246454418 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f82600ca7b0 0x7f82600c4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.736+0000 7f8265bff700 1 -- 172.21.15.27:0/246454418 >> 172.21.15.27:0/246454418 conn(0x7f82600ce3e0 msgr2=0x7f82601527a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.736+0000 7f8265bff700 1 -- 172.21.15.27:0/246454418 shutdown_connections 2024-01-11T08:25:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:28.736+0000 7f8265bff700 1 -- 172.21.15.27:0/246454418 wait complete. 2024-01-11T08:25:28.739 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:25:28.754 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-11T08:25:27.972373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":120,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2431608,"kb_used_data":8248,"kb_used_omap":0,"kb_used_meta":2423296,"kb_avail":752543112,"statfs":{"total":773094113280,"available":770604146688,"internally_reserved":0,"allocated":8445952,"data_stored":4492396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2481455104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,9],"upper_bound":32},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001231"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.872924+0000","last_change":"2024-01-11T08:25:23.860736+0000","last_active":"2024-01-11T08:25:23.872924+0000","last_peered":"2024-01-11T08:25:23.872924+0000","last_clean":"2024-01-11T08:25:23.872924+0000","last_became_active":"2024-01-11T08:25:23.860459+0000","last_became_peered":"2024-01-11T08:25:23.860459+0000","last_unstale":"2024-01-11T08:25:23.872924+0000","last_undegraded":"2024-01-11T08:25:23.872924+0000","last_fullsized":"2024-01-11T08:25:23.872924+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:13:56.163286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.876258+0000","last_change":"2024-01-11T08:25:23.872814+0000","last_active":"2024-01-11T08:25:23.876258+0000","last_peered":"2024-01-11T08:25:23.876258+0000","last_clean":"2024-01-11T08:25:23.876258+0000","last_became_active":"2024-01-11T08:25:23.872421+0000","last_became_peered":"2024-01-11T08:25:23.872421+0000","last_unstale":"2024-01-11T08:25:23.876258+0000","last_undegraded":"2024-01-11T08:25:23.876258+0000","last_fullsized":"2024-01-11T08:25:23.876258+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:20:24.901979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.877566+0000","last_change":"2024-01-11T08:25:23.874981+0000","last_active":"2024-01-11T08:25:23.877566+0000","last_peered":"2024-01-11T08:25:23.877566+0000","last_clean":"2024-01-11T08:25:23.877566+0000","last_became_active":"2024-01-11T08:25:23.874541+0000","last_became_peered":"2024-01-11T08:25:23.874541+0000","last_unstale":"2024-01-11T08:25:23.877566+0000","last_undegraded":"2024-01-11T08:25:23.877566+0000","last_fullsized":"2024-01-11T08:25:23.877566+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:55:26.350726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.857632+0000","last_change":"2024-01-11T08:25:23.857632+0000","last_active":"2024-01-11T08:25:23.857632+0000","last_peered":"2024-01-11T08:25:23.857632+0000","last_clean":"2024-01-11T08:25:23.857632+0000","last_became_active":"2024-01-11T08:25:23.857328+0000","last_became_peered":"2024-01-11T08:25:23.857328+0000","last_unstale":"2024-01-11T08:25:23.857632+0000","last_undegraded":"2024-01-11T08:25:23.857632+0000","last_fullsized":"2024-01-11T08:25:23.857632+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:01:30.482699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.863582+0000","last_change":"2024-01-11T08:25:23.855695+0000","last_active":"2024-01-11T08:25:23.863582+0000","last_peered":"2024-01-11T08:25:23.863582+0000","last_clean":"2024-01-11T08:25:23.863582+0000","last_became_active":"2024-01-11T08:25:23.854931+0000","last_became_peered":"2024-01-11T08:25:23.854931+0000","last_unstale":"2024-01-11T08:25:23.863582+0000","last_undegraded":"2024-01-11T08:25:23.863582+0000","last_fullsized":"2024-01-11T08:25:23.863582+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:19:21.377793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.867231+0000","last_change":"2024-01-11T08:25:23.867231+0000","last_active":"2024-01-11T08:25:23.867231+0000","last_peered":"2024-01-11T08:25:23.867231+0000","last_clean":"2024-01-11T08:25:23.867231+0000","last_became_active":"2024-01-11T08:25:23.866921+0000","last_became_peered":"2024-01-11T08:25:23.866921+0000","last_unstale":"2024-01-11T08:25:23.867231+0000","last_undegraded":"2024-01-11T08:25:23.867231+0000","last_fullsized":"2024-01-11T08:25:23.867231+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:50:16.386930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.856105+0000","last_change":"2024-01-11T08:25:23.856105+0000","last_active":"2024-01-11T08:25:23.856105+0000","last_peered":"2024-01-11T08:25:23.856105+0000","last_clean":"2024-01-11T08:25:23.856105+0000","last_became_active":"2024-01-11T08:25:23.855835+0000","last_became_peered":"2024-01-11T08:25:23.855835+0000","last_unstale":"2024-01-11T08:25:23.856105+0000","last_undegraded":"2024-01-11T08:25:23.856105+0000","last_fullsized":"2024-01-11T08:25:23.856105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:47:40.738909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.855832+0000","last_change":"2024-01-11T08:25:23.851288+0000","last_active":"2024-01-11T08:25:23.855832+0000","last_peered":"2024-01-11T08:25:23.855832+0000","last_clean":"2024-01-11T08:25:23.855832+0000","last_became_active":"2024-01-11T08:25:23.851020+0000","last_became_peered":"2024-01-11T08:25:23.851020+0000","last_unstale":"2024-01-11T08:25:23.855832+0000","last_undegraded":"2024-01-11T08:25:23.855832+0000","last_fullsized":"2024-01-11T08:25:23.855832+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:36:58.150280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.861565+0000","last_change":"2024-01-11T08:25:23.861565+0000","last_active":"2024-01-11T08:25:23.861565+0000","last_peered":"2024-01-11T08:25:23.861565+0000","last_clean":"2024-01-11T08:25:23.861565+0000","last_became_active":"2024-01-11T08:25:23.861250+0000","last_became_peered":"2024-01-11T08:25:23.861250+0000","last_unstale":"2024-01-11T08:25:23.861565+0000","last_undegraded":"2024-01-11T08:25:23.861565+0000","last_fullsized":"2024-01-11T08:25:23.861565+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:02.344677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.875445+0000","last_change":"2024-01-11T08:25:23.861222+0000","last_active":"2024-01-11T08:25:23.875445+0000","last_peered":"2024-01-11T08:25:23.875445+0000","last_clean":"2024-01-11T08:25:23.875445+0000","last_became_active":"2024-01-11T08:25:23.860921+0000","last_became_peered":"2024-01-11T08:25:23.860921+0000","last_unstale":"2024-01-11T08:25:23.875445+0000","last_undegraded":"2024-01-11T08:25:23.875445+0000","last_fullsized":"2024-01-11T08:25:23.875445+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:21:38.471310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.879228+0000","last_change":"2024-01-11T08:25:23.873460+0000","last_active":"2024-01-11T08:25:23.879228+0000","last_peered":"2024-01-11T08:25:23.879228+0000","last_clean":"2024-01-11T08:25:23.879228+0000","last_became_active":"2024-01-11T08:25:23.873209+0000","last_became_peered":"2024-01-11T08:25:23.873209+0000","last_unstale":"2024-01-11T08:25:23.879228+0000","last_undegraded":"2024-01-11T08:25:23.879228+0000","last_fullsized":"2024-01-11T08:25:23.879228+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:22:19.702797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.861035+0000","last_change":"2024-01-11T08:25:23.861035+0000","last_active":"2024-01-11T08:25:23.861035+0000","last_peered":"2024-01-11T08:25:23.861035+0000","last_clean":"2024-01-11T08:25:23.861035+0000","last_became_active":"2024-01-11T08:25:23.860737+0000","last_became_peered":"2024-01-11T08:25:23.860737+0000","last_unstale":"2024-01-11T08:25:23.861035+0000","last_undegraded":"2024-01-11T08:25:23.861035+0000","last_fullsized":"2024-01-11T08:25:23.861035+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:43:11.246835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.876486+0000","last_change":"2024-01-11T08:25:23.873249+0000","last_active":"2024-01-11T08:25:23.876486+0000","last_peered":"2024-01-11T08:25:23.876486+0000","last_clean":"2024-01-11T08:25:23.876486+0000","last_became_active":"2024-01-11T08:25:23.872956+0000","last_became_peered":"2024-01-11T08:25:23.872956+0000","last_unstale":"2024-01-11T08:25:23.876486+0000","last_undegraded":"2024-01-11T08:25:23.876486+0000","last_fullsized":"2024-01-11T08:25:23.876486+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:21:38.471310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.860036+0000","last_change":"2024-01-11T08:25:23.860036+0000","last_active":"2024-01-11T08:25:23.860036+0000","last_peered":"2024-01-11T08:25:23.860036+0000","last_clean":"2024-01-11T08:25:23.860036+0000","last_became_active":"2024-01-11T08:25:23.858022+0000","last_became_peered":"2024-01-11T08:25:23.858022+0000","last_unstale":"2024-01-11T08:25:23.860036+0000","last_undegraded":"2024-01-11T08:25:23.860036+0000","last_fullsized":"2024-01-11T08:25:23.860036+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:23:55.749223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.863580+0000","last_change":"2024-01-11T08:25:23.863580+0000","last_active":"2024-01-11T08:25:23.863580+0000","last_peered":"2024-01-11T08:25:23.863580+0000","last_clean":"2024-01-11T08:25:23.863580+0000","last_became_active":"2024-01-11T08:25:23.863269+0000","last_became_peered":"2024-01-11T08:25:23.863269+0000","last_unstale":"2024-01-11T08:25:23.863580+0000","last_undegraded":"2024-01-11T08:25:23.863580+0000","last_fullsized":"2024-01-11T08:25:23.863580+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:47:03.454915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.869698+0000","last_change":"2024-01-11T08:25:23.869698+0000","last_active":"2024-01-11T08:25:23.869698+0000","last_peered":"2024-01-11T08:25:23.869698+0000","last_clean":"2024-01-11T08:25:23.869698+0000","last_became_active":"2024-01-11T08:25:23.868588+0000","last_became_peered":"2024-01-11T08:25:23.868588+0000","last_unstale":"2024-01-11T08:25:23.869698+0000","last_undegraded":"2024-01-11T08:25:23.869698+0000","last_fullsized":"2024-01-11T08:25:23.869698+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:54:45.299302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.868011+0000","last_change":"2024-01-11T08:25:23.861461+0000","last_active":"2024-01-11T08:25:23.868011+0000","last_peered":"2024-01-11T08:25:23.868011+0000","last_clean":"2024-01-11T08:25:23.868011+0000","last_became_active":"2024-01-11T08:25:23.861196+0000","last_became_peered":"2024-01-11T08:25:23.861196+0000","last_unstale":"2024-01-11T08:25:23.868011+0000","last_undegraded":"2024-01-11T08:25:23.868011+0000","last_fullsized":"2024-01-11T08:25:23.868011+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:18:45.387043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.880082+0000","last_change":"2024-01-11T08:25:23.879128+0000","last_active":"2024-01-11T08:25:23.880082+0000","last_peered":"2024-01-11T08:25:23.880082+0000","last_clean":"2024-01-11T08:25:23.880082+0000","last_became_active":"2024-01-11T08:25:23.878862+0000","last_became_peered":"2024-01-11T08:25:23.878862+0000","last_unstale":"2024-01-11T08:25:23.880082+0000","last_undegraded":"2024-01-11T08:25:23.880082+0000","last_fullsized":"2024-01-11T08:25:23.880082+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:00:58.175007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.873020+0000","last_change":"2024-01-11T08:25:23.860722+0000","last_active":"2024-01-11T08:25:23.873020+0000","last_peered":"2024-01-11T08:25:23.873020+0000","last_clean":"2024-01-11T08:25:23.873020+0000","last_became_active":"2024-01-11T08:25:23.860420+0000","last_became_peered":"2024-01-11T08:25:23.860420+0000","last_unstale":"2024-01-11T08:25:23.873020+0000","last_undegraded":"2024-01-11T08:25:23.873020+0000","last_fullsized":"2024-01-11T08:25:23.873020+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:15:41.820195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.873957+0000","last_change":"2024-01-11T08:25:23.867445+0000","last_active":"2024-01-11T08:25:23.873957+0000","last_peered":"2024-01-11T08:25:23.873957+0000","last_clean":"2024-01-11T08:25:23.873957+0000","last_became_active":"2024-01-11T08:25:23.867180+0000","last_became_peered":"2024-01-11T08:25:23.867180+0000","last_unstale":"2024-01-11T08:25:23.873957+0000","last_undegraded":"2024-01-11T08:25:23.873957+0000","last_fullsized":"2024-01-11T08:25:23.873957+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:08:45.304781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.873728+0000","last_change":"2024-01-11T08:25:23.869311+0000","last_active":"2024-01-11T08:25:23.873728+0000","last_peered":"2024-01-11T08:25:23.873728+0000","last_clean":"2024-01-11T08:25:23.873728+0000","last_became_active":"2024-01-11T08:25:23.869043+0000","last_became_peered":"2024-01-11T08:25:23.869043+0000","last_unstale":"2024-01-11T08:25:23.873728+0000","last_undegraded":"2024-01-11T08:25:23.873728+0000","last_fullsized":"2024-01-11T08:25:23.873728+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:47:03.454915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.874066+0000","last_change":"2024-01-11T08:25:23.868231+0000","last_active":"2024-01-11T08:25:23.874066+0000","last_peered":"2024-01-11T08:25:23.874066+0000","last_clean":"2024-01-11T08:25:23.874066+0000","last_became_active":"2024-01-11T08:25:23.861599+0000","last_became_peered":"2024-01-11T08:25:23.861599+0000","last_unstale":"2024-01-11T08:25:23.874066+0000","last_undegraded":"2024-01-11T08:25:23.874066+0000","last_fullsized":"2024-01-11T08:25:23.874066+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:10:22.895007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.877492+0000","last_change":"2024-01-11T08:25:23.874598+0000","last_active":"2024-01-11T08:25:23.877492+0000","last_peered":"2024-01-11T08:25:23.877492+0000","last_clean":"2024-01-11T08:25:23.877492+0000","last_became_active":"2024-01-11T08:25:23.872952+0000","last_became_peered":"2024-01-11T08:25:23.872952+0000","last_unstale":"2024-01-11T08:25:23.877492+0000","last_undegraded":"2024-01-11T08:25:23.877492+0000","last_fullsized":"2024-01-11T08:25:23.877492+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:21:33.358382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.876541+0000","last_change":"2024-01-11T08:25:23.873989+0000","last_active":"2024-01-11T08:25:23.876541+0000","last_peered":"2024-01-11T08:25:23.876541+0000","last_clean":"2024-01-11T08:25:23.876541+0000","last_became_active":"2024-01-11T08:25:23.873673+0000","last_became_peered":"2024-01-11T08:25:23.873673+0000","last_unstale":"2024-01-11T08:25:23.876541+0000","last_undegraded":"2024-01-11T08:25:23.876541+0000","last_fullsized":"2024-01-11T08:25:23.876541+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:58:35.728365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.872946+0000","last_change":"2024-01-11T08:25:23.861407+0000","last_active":"2024-01-11T08:25:23.872946+0000","last_peered":"2024-01-11T08:25:23.872946+0000","last_clean":"2024-01-11T08:25:23.872946+0000","last_became_active":"2024-01-11T08:25:23.861042+0000","last_became_peered":"2024-01-11T08:25:23.861042+0000","last_unstale":"2024-01-11T08:25:23.872946+0000","last_undegraded":"2024-01-11T08:25:23.872946+0000","last_fullsized":"2024-01-11T08:25:23.872946+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:20:08.135199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.858847+0000","last_change":"2024-01-11T08:25:23.858847+0000","last_active":"2024-01-11T08:25:23.858847+0000","last_peered":"2024-01-11T08:25:23.858847+0000","last_clean":"2024-01-11T08:25:23.858847+0000","last_became_active":"2024-01-11T08:25:23.858557+0000","last_became_peered":"2024-01-11T08:25:23.858557+0000","last_unstale":"2024-01-11T08:25:23.858847+0000","last_undegraded":"2024-01-11T08:25:23.858847+0000","last_fullsized":"2024-01-11T08:25:23.858847+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:54:45.299302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.867927+0000","last_change":"2024-01-11T08:25:23.867927+0000","last_active":"2024-01-11T08:25:23.867927+0000","last_peered":"2024-01-11T08:25:23.867927+0000","last_clean":"2024-01-11T08:25:23.867927+0000","last_became_active":"2024-01-11T08:25:23.867654+0000","last_became_peered":"2024-01-11T08:25:23.867654+0000","last_unstale":"2024-01-11T08:25:23.867927+0000","last_undegraded":"2024-01-11T08:25:23.867927+0000","last_fullsized":"2024-01-11T08:25:23.867927+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:43:11.246835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.874140+0000","last_change":"2024-01-11T08:25:23.866937+0000","last_active":"2024-01-11T08:25:23.874140+0000","last_peered":"2024-01-11T08:25:23.874140+0000","last_clean":"2024-01-11T08:25:23.874140+0000","last_became_active":"2024-01-11T08:25:23.865250+0000","last_became_peered":"2024-01-11T08:25:23.865250+0000","last_unstale":"2024-01-11T08:25:23.874140+0000","last_undegraded":"2024-01-11T08:25:23.874140+0000","last_fullsized":"2024-01-11T08:25:23.874140+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:25:40.491820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.866155+0000","last_change":"2024-01-11T08:25:23.852786+0000","last_active":"2024-01-11T08:25:23.866155+0000","last_peered":"2024-01-11T08:25:23.866155+0000","last_clean":"2024-01-11T08:25:23.866155+0000","last_became_active":"2024-01-11T08:25:23.852460+0000","last_became_peered":"2024-01-11T08:25:23.852460+0000","last_unstale":"2024-01-11T08:25:23.866155+0000","last_undegraded":"2024-01-11T08:25:23.866155+0000","last_fullsized":"2024-01-11T08:25:23.866155+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:14:12.366345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.877446+0000","last_change":"2024-01-11T08:25:23.873169+0000","last_active":"2024-01-11T08:25:23.877446+0000","last_peered":"2024-01-11T08:25:23.877446+0000","last_clean":"2024-01-11T08:25:23.877446+0000","last_became_active":"2024-01-11T08:25:23.872841+0000","last_became_peered":"2024-01-11T08:25:23.872841+0000","last_unstale":"2024-01-11T08:25:23.877446+0000","last_undegraded":"2024-01-11T08:25:23.877446+0000","last_fullsized":"2024-01-11T08:25:23.877446+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:11:46.534149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.873295+0000","last_change":"2024-01-11T08:25:23.864748+0000","last_active":"2024-01-11T08:25:23.873295+0000","last_peered":"2024-01-11T08:25:23.873295+0000","last_clean":"2024-01-11T08:25:23.873295+0000","last_became_active":"2024-01-11T08:25:23.864474+0000","last_became_peered":"2024-01-11T08:25:23.864474+0000","last_unstale":"2024-01-11T08:25:23.873295+0000","last_undegraded":"2024-01-11T08:25:23.873295+0000","last_fullsized":"2024-01-11T08:25:23.873295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:08:10.968900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.864146+0000","last_change":"2024-01-11T08:25:23.864146+0000","last_active":"2024-01-11T08:25:23.864146+0000","last_peered":"2024-01-11T08:25:23.864146+0000","last_clean":"2024-01-11T08:25:23.864146+0000","last_became_active":"2024-01-11T08:25:23.863890+0000","last_became_peered":"2024-01-11T08:25:23.863890+0000","last_unstale":"2024-01-11T08:25:23.864146+0000","last_undegraded":"2024-01-11T08:25:23.864146+0000","last_fullsized":"2024-01-11T08:25:23.864146+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:45:12.148806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.862496+0000","last_change":"2024-01-11T08:25:23.862496+0000","last_active":"2024-01-11T08:25:23.862496+0000","last_peered":"2024-01-11T08:25:23.862496+0000","last_clean":"2024-01-11T08:25:23.862496+0000","last_became_active":"2024-01-11T08:25:23.862234+0000","last_became_peered":"2024-01-11T08:25:23.862234+0000","last_unstale":"2024-01-11T08:25:23.862496+0000","last_undegraded":"2024-01-11T08:25:23.862496+0000","last_fullsized":"2024-01-11T08:25:23.862496+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:01:30.482699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"74'4","reported_seq":13,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.881153+0000","last_change":"2024-01-11T08:25:22.851110+0000","last_active":"2024-01-11T08:25:23.881153+0000","last_peered":"2024-01-11T08:25:23.881153+0000","last_clean":"2024-01-11T08:25:23.881153+0000","last_became_active":"2024-01-11T08:25:22.850230+0000","last_became_peered":"2024-01-11T08:25:22.850230+0000","last_unstale":"2024-01-11T08:25:23.881153+0000","last_undegraded":"2024-01-11T08:25:23.881153+0000","last_fullsized":"2024-01-11T08:25:23.881153+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:22:18.693300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.877128+0000","last_change":"2024-01-11T08:25:23.872201+0000","last_active":"2024-01-11T08:25:23.877128+0000","last_peered":"2024-01-11T08:25:23.877128+0000","last_clean":"2024-01-11T08:25:23.877128+0000","last_became_active":"2024-01-11T08:25:23.871950+0000","last_became_peered":"2024-01-11T08:25:23.871950+0000","last_unstale":"2024-01-11T08:25:23.877128+0000","last_undegraded":"2024-01-11T08:25:23.877128+0000","last_fullsized":"2024-01-11T08:25:23.877128+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:41:23.867564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"74'1","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.872632+0000","last_change":"2024-01-11T08:25:22.857442+0000","last_active":"2024-01-11T08:25:23.872632+0000","last_peered":"2024-01-11T08:25:23.872632+0000","last_clean":"2024-01-11T08:25:23.872632+0000","last_became_active":"2024-01-11T08:25:22.856309+0000","last_became_peered":"2024-01-11T08:25:22.856309+0000","last_unstale":"2024-01-11T08:25:23.872632+0000","last_undegraded":"2024-01-11T08:25:23.872632+0000","last_fullsized":"2024-01-11T08:25:23.872632+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:39:14.987410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.862306+0000","last_change":"2024-01-11T08:25:23.862306+0000","last_active":"2024-01-11T08:25:23.862306+0000","last_peered":"2024-01-11T08:25:23.862306+0000","last_clean":"2024-01-11T08:25:23.862306+0000","last_became_active":"2024-01-11T08:25:23.861889+0000","last_became_peered":"2024-01-11T08:25:23.861889+0000","last_unstale":"2024-01-11T08:25:23.862306+0000","last_undegraded":"2024-01-11T08:25:23.862306+0000","last_fullsized":"2024-01-11T08:25:23.862306+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:59:37.011537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"74'2","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.879808+0000","last_change":"2024-01-11T08:25:22.864486+0000","last_active":"2024-01-11T08:25:23.879808+0000","last_peered":"2024-01-11T08:25:23.879808+0000","last_clean":"2024-01-11T08:25:23.879808+0000","last_became_active":"2024-01-11T08:25:22.864194+0000","last_became_peered":"2024-01-11T08:25:22.864194+0000","last_unstale":"2024-01-11T08:25:23.879808+0000","last_undegraded":"2024-01-11T08:25:23.879808+0000","last_fullsized":"2024-01-11T08:25:23.879808+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:12:37.683775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.873819+0000","last_change":"2024-01-11T08:25:23.873819+0000","last_active":"2024-01-11T08:25:23.873819+0000","last_peered":"2024-01-11T08:25:23.873819+0000","last_clean":"2024-01-11T08:25:23.873819+0000","last_became_active":"2024-01-11T08:25:23.873293+0000","last_became_peered":"2024-01-11T08:25:23.873293+0000","last_unstale":"2024-01-11T08:25:23.873819+0000","last_undegraded":"2024-01-11T08:25:23.873819+0000","last_fullsized":"2024-01-11T08:25:23.873819+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:52:36.524494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"74'2","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.867065+0000","last_change":"2024-01-11T08:25:22.854672+0000","last_active":"2024-01-11T08:25:23.867065+0000","last_peered":"2024-01-11T08:25:23.867065+0000","last_clean":"2024-01-11T08:25:23.867065+0000","last_became_active":"2024-01-11T08:25:22.854384+0000","last_became_peered":"2024-01-11T08:25:22.854384+0000","last_unstale":"2024-01-11T08:25:23.867065+0000","last_undegraded":"2024-01-11T08:25:23.867065+0000","last_fullsized":"2024-01-11T08:25:23.867065+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:47:02.445419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.870165+0000","last_change":"2024-01-11T08:25:23.865456+0000","last_active":"2024-01-11T08:25:23.870165+0000","last_peered":"2024-01-11T08:25:23.870165+0000","last_clean":"2024-01-11T08:25:23.870165+0000","last_became_active":"2024-01-11T08:25:23.863331+0000","last_became_peered":"2024-01-11T08:25:23.863331+0000","last_unstale":"2024-01-11T08:25:23.870165+0000","last_undegraded":"2024-01-11T08:25:23.870165+0000","last_fullsized":"2024-01-11T08:25:23.870165+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:36:44.218033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"74'1","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.846982+0000","last_change":"2024-01-11T08:25:22.856142+0000","last_active":"2024-01-11T08:25:23.846982+0000","last_peered":"2024-01-11T08:25:23.846982+0000","last_clean":"2024-01-11T08:25:23.846982+0000","last_became_active":"2024-01-11T08:25:22.855635+0000","last_became_peered":"2024-01-11T08:25:22.855635+0000","last_unstale":"2024-01-11T08:25:23.846982+0000","last_undegraded":"2024-01-11T08:25:23.846982+0000","last_fullsized":"2024-01-11T08:25:23.846982+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:42:52.255488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.870564+0000","last_change":"2024-01-11T08:25:23.866332+0000","last_active":"2024-01-11T08:25:23.870564+0000","last_peered":"2024-01-11T08:25:23.870564+0000","last_clean":"2024-01-11T08:25:23.870564+0000","last_became_active":"2024-01-11T08:25:23.861779+0000","last_became_peered":"2024-01-11T08:25:23.861779+0000","last_unstale":"2024-01-11T08:25:23.870564+0000","last_undegraded":"2024-01-11T08:25:23.870564+0000","last_fullsized":"2024-01-11T08:25:23.870564+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:49:28.354463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"74'1","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.852670+0000","last_change":"2024-01-11T08:25:22.854267+0000","last_active":"2024-01-11T08:25:23.852670+0000","last_peered":"2024-01-11T08:25:23.852670+0000","last_clean":"2024-01-11T08:25:23.852670+0000","last_became_active":"2024-01-11T08:25:22.853884+0000","last_became_peered":"2024-01-11T08:25:22.853884+0000","last_unstale":"2024-01-11T08:25:23.852670+0000","last_undegraded":"2024-01-11T08:25:23.852670+0000","last_fullsized":"2024-01-11T08:25:23.852670+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:35:21.406141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.874157+0000","last_change":"2024-01-11T08:25:23.869344+0000","last_active":"2024-01-11T08:25:23.874157+0000","last_peered":"2024-01-11T08:25:23.874157+0000","last_clean":"2024-01-11T08:25:23.874157+0000","last_became_active":"2024-01-11T08:25:23.869088+0000","last_became_peered":"2024-01-11T08:25:23.869088+0000","last_unstale":"2024-01-11T08:25:23.874157+0000","last_undegraded":"2024-01-11T08:25:23.874157+0000","last_fullsized":"2024-01-11T08:25:23.874157+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:42:53.264984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-11T08:25:22.856781+0000","last_change":"2024-01-11T08:25:22.852104+0000","last_active":"2024-01-11T08:25:22.856781+0000","last_peered":"2024-01-11T08:25:22.856781+0000","last_clean":"2024-01-11T08:25:22.856781+0000","last_became_active":"2024-01-11T08:25:22.851787+0000","last_became_peered":"2024-01-11T08:25:22.851787+0000","last_unstale":"2024-01-11T08:25:22.856781+0000","last_undegraded":"2024-01-11T08:25:22.856781+0000","last_fullsized":"2024-01-11T08:25:22.856781+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:23:54.739727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.870600+0000","last_change":"2024-01-11T08:25:23.866696+0000","last_active":"2024-01-11T08:25:23.870600+0000","last_peered":"2024-01-11T08:25:23.870600+0000","last_clean":"2024-01-11T08:25:23.870600+0000","last_became_active":"2024-01-11T08:25:23.864469+0000","last_became_peered":"2024-01-11T08:25:23.864469+0000","last_unstale":"2024-01-11T08:25:23.870600+0000","last_undegraded":"2024-01-11T08:25:23.870600+0000","last_fullsized":"2024-01-11T08:25:23.870600+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:16.536682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-11T08:25:22.860954+0000","last_change":"2024-01-11T08:25:22.859339+0000","last_active":"2024-01-11T08:25:22.860954+0000","last_peered":"2024-01-11T08:25:22.860954+0000","last_clean":"2024-01-11T08:25:22.860954+0000","last_became_active":"2024-01-11T08:25:22.859026+0000","last_became_peered":"2024-01-11T08:25:22.859026+0000","last_unstale":"2024-01-11T08:25:22.860954+0000","last_undegraded":"2024-01-11T08:25:22.860954+0000","last_fullsized":"2024-01-11T08:25:22.860954+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:21:37.461813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.862643+0000","last_change":"2024-01-11T08:25:23.862643+0000","last_active":"2024-01-11T08:25:23.862643+0000","last_peered":"2024-01-11T08:25:23.862643+0000","last_clean":"2024-01-11T08:25:23.862643+0000","last_became_active":"2024-01-11T08:25:23.861028+0000","last_became_peered":"2024-01-11T08:25:23.861028+0000","last_unstale":"2024-01-11T08:25:23.862643+0000","last_undegraded":"2024-01-11T08:25:23.862643+0000","last_fullsized":"2024-01-11T08:25:23.862643+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:11:46.534149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.861770+0000","last_change":"2024-01-11T08:25:22.863437+0000","last_active":"2024-01-11T08:25:23.861770+0000","last_peered":"2024-01-11T08:25:23.861770+0000","last_clean":"2024-01-11T08:25:23.861770+0000","last_became_active":"2024-01-11T08:25:22.863054+0000","last_became_peered":"2024-01-11T08:25:22.863054+0000","last_unstale":"2024-01-11T08:25:23.861770+0000","last_undegraded":"2024-01-11T08:25:23.861770+0000","last_fullsized":"2024-01-11T08:25:23.861770+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:20:30.108010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.879652+0000","last_change":"2024-01-11T08:25:23.878865+0000","last_active":"2024-01-11T08:25:23.879652+0000","last_peered":"2024-01-11T08:25:23.879652+0000","last_clean":"2024-01-11T08:25:23.879652+0000","last_became_active":"2024-01-11T08:25:23.878598+0000","last_became_peered":"2024-01-11T08:25:23.878598+0000","last_unstale":"2024-01-11T08:25:23.879652+0000","last_undegraded":"2024-01-11T08:25:23.879652+0000","last_fullsized":"2024-01-11T08:25:23.879652+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:24:45.942689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.842617+0000","last_change":"2024-01-11T08:25:22.852556+0000","last_active":"2024-01-11T08:25:23.842617+0000","last_peered":"2024-01-11T08:25:23.842617+0000","last_clean":"2024-01-11T08:25:23.842617+0000","last_became_active":"2024-01-11T08:25:22.851861+0000","last_became_peered":"2024-01-11T08:25:22.851861+0000","last_unstale":"2024-01-11T08:25:23.842617+0000","last_undegraded":"2024-01-11T08:25:23.842617+0000","last_fullsized":"2024-01-11T08:25:23.842617+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:21:32.348886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.876370+0000","last_change":"2024-01-11T08:25:23.869012+0000","last_active":"2024-01-11T08:25:23.876370+0000","last_peered":"2024-01-11T08:25:23.876370+0000","last_clean":"2024-01-11T08:25:23.876370+0000","last_became_active":"2024-01-11T08:25:23.868581+0000","last_became_peered":"2024-01-11T08:25:23.868581+0000","last_unstale":"2024-01-11T08:25:23.876370+0000","last_undegraded":"2024-01-11T08:25:23.876370+0000","last_fullsized":"2024-01-11T08:25:23.876370+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:37:20.723150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"74'3","reported_seq":13,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.876961+0000","last_change":"2024-01-11T08:25:22.868809+0000","last_active":"2024-01-11T08:25:23.876961+0000","last_peered":"2024-01-11T08:25:23.876961+0000","last_clean":"2024-01-11T08:25:23.876961+0000","last_became_active":"2024-01-11T08:25:22.868572+0000","last_became_peered":"2024-01-11T08:25:22.868572+0000","last_unstale":"2024-01-11T08:25:23.876961+0000","last_undegraded":"2024-01-11T08:25:23.876961+0000","last_fullsized":"2024-01-11T08:25:23.876961+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:43:47.124370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.878149+0000","last_change":"2024-01-11T08:25:23.877485+0000","last_active":"2024-01-11T08:25:23.878149+0000","last_peered":"2024-01-11T08:25:23.878149+0000","last_clean":"2024-01-11T08:25:23.878149+0000","last_became_active":"2024-01-11T08:25:23.877189+0000","last_became_peered":"2024-01-11T08:25:23.877189+0000","last_unstale":"2024-01-11T08:25:23.878149+0000","last_undegraded":"2024-01-11T08:25:23.878149+0000","last_fullsized":"2024-01-11T08:25:23.878149+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:50:16.386930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"74'6","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.871722+0000","last_change":"2024-01-11T08:25:22.858712+0000","last_active":"2024-01-11T08:25:23.871722+0000","last_peered":"2024-01-11T08:25:23.871722+0000","last_clean":"2024-01-11T08:25:23.871722+0000","last_became_active":"2024-01-11T08:25:22.858429+0000","last_became_peered":"2024-01-11T08:25:22.858429+0000","last_unstale":"2024-01-11T08:25:23.871722+0000","last_undegraded":"2024-01-11T08:25:23.871722+0000","last_fullsized":"2024-01-11T08:25:23.871722+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:20:07.177322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.866341+0000","last_change":"2024-01-11T08:25:23.866341+0000","last_active":"2024-01-11T08:25:23.866341+0000","last_peered":"2024-01-11T08:25:23.866341+0000","last_clean":"2024-01-11T08:25:23.866341+0000","last_became_active":"2024-01-11T08:25:23.866036+0000","last_became_peered":"2024-01-11T08:25:23.866036+0000","last_unstale":"2024-01-11T08:25:23.866341+0000","last_undegraded":"2024-01-11T08:25:23.866341+0000","last_fullsized":"2024-01-11T08:25:23.866341+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:56:15.839817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"74'2","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.874460+0000","last_change":"2024-01-11T08:25:22.852070+0000","last_active":"2024-01-11T08:25:23.874460+0000","last_peered":"2024-01-11T08:25:23.874460+0000","last_clean":"2024-01-11T08:25:23.874460+0000","last_became_active":"2024-01-11T08:25:22.851774+0000","last_became_peered":"2024-01-11T08:25:22.851774+0000","last_unstale":"2024-01-11T08:25:23.874460+0000","last_undegraded":"2024-01-11T08:25:23.874460+0000","last_fullsized":"2024-01-11T08:25:23.874460+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:00:57.165511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.869504+0000","last_change":"2024-01-11T08:25:23.863936+0000","last_active":"2024-01-11T08:25:23.869504+0000","last_peered":"2024-01-11T08:25:23.869504+0000","last_clean":"2024-01-11T08:25:23.869504+0000","last_became_active":"2024-01-11T08:25:23.863655+0000","last_became_peered":"2024-01-11T08:25:23.863655+0000","last_unstale":"2024-01-11T08:25:23.869504+0000","last_undegraded":"2024-01-11T08:25:23.869504+0000","last_fullsized":"2024-01-11T08:25:23.869504+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:02.344677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.854354+0000","last_change":"2024-01-11T08:25:22.858764+0000","last_active":"2024-01-11T08:25:23.854354+0000","last_peered":"2024-01-11T08:25:23.854354+0000","last_clean":"2024-01-11T08:25:23.854354+0000","last_became_active":"2024-01-11T08:25:22.857409+0000","last_became_peered":"2024-01-11T08:25:22.857409+0000","last_unstale":"2024-01-11T08:25:23.854354+0000","last_undegraded":"2024-01-11T08:25:23.854354+0000","last_fullsized":"2024-01-11T08:25:23.854354+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:41:40.154137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.869005+0000","last_change":"2024-01-11T08:25:23.869005+0000","last_active":"2024-01-11T08:25:23.869005+0000","last_peered":"2024-01-11T08:25:23.869005+0000","last_clean":"2024-01-11T08:25:23.869005+0000","last_became_active":"2024-01-11T08:25:23.868691+0000","last_became_peered":"2024-01-11T08:25:23.868691+0000","last_unstale":"2024-01-11T08:25:23.869005+0000","last_undegraded":"2024-01-11T08:25:23.869005+0000","last_fullsized":"2024-01-11T08:25:23.869005+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:07:23.998750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-11T08:25:22.857407+0000","last_change":"2024-01-11T08:25:22.854594+0000","last_active":"2024-01-11T08:25:22.857407+0000","last_peered":"2024-01-11T08:25:22.857407+0000","last_clean":"2024-01-11T08:25:22.857407+0000","last_became_active":"2024-01-11T08:25:22.854299+0000","last_became_peered":"2024-01-11T08:25:22.854299+0000","last_unstale":"2024-01-11T08:25:22.857407+0000","last_undegraded":"2024-01-11T08:25:22.857407+0000","last_fullsized":"2024-01-11T08:25:22.857407+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:01.335181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.877508+0000","last_change":"2024-01-11T08:25:23.875606+0000","last_active":"2024-01-11T08:25:23.877508+0000","last_peered":"2024-01-11T08:25:23.877508+0000","last_clean":"2024-01-11T08:25:23.877508+0000","last_became_active":"2024-01-11T08:25:23.875281+0000","last_became_peered":"2024-01-11T08:25:23.875281+0000","last_unstale":"2024-01-11T08:25:23.877508+0000","last_undegraded":"2024-01-11T08:25:23.877508+0000","last_fullsized":"2024-01-11T08:25:23.877508+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:12:22.491644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"74'8","reported_seq":18,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.875202+0000","last_change":"2024-01-11T08:25:22.860304+0000","last_active":"2024-01-11T08:25:23.875202+0000","last_peered":"2024-01-11T08:25:23.875202+0000","last_clean":"2024-01-11T08:25:23.875202+0000","last_became_active":"2024-01-11T08:25:22.860063+0000","last_became_peered":"2024-01-11T08:25:22.860063+0000","last_unstale":"2024-01-11T08:25:23.875202+0000","last_undegraded":"2024-01-11T08:25:23.875202+0000","last_fullsized":"2024-01-11T08:25:23.875202+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:20:07.177322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.877453+0000","last_change":"2024-01-11T08:25:23.874224+0000","last_active":"2024-01-11T08:25:23.877453+0000","last_peered":"2024-01-11T08:25:23.877453+0000","last_clean":"2024-01-11T08:25:23.877453+0000","last_became_active":"2024-01-11T08:25:23.873913+0000","last_became_peered":"2024-01-11T08:25:23.873913+0000","last_unstale":"2024-01-11T08:25:23.877453+0000","last_undegraded":"2024-01-11T08:25:23.877453+0000","last_fullsized":"2024-01-11T08:25:23.877453+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:52:36.524494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"74'1","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.874537+0000","last_change":"2024-01-11T08:25:22.860048+0000","last_active":"2024-01-11T08:25:23.874537+0000","last_peered":"2024-01-11T08:25:23.874537+0000","last_clean":"2024-01-11T08:25:23.874537+0000","last_became_active":"2024-01-11T08:25:22.859794+0000","last_became_peered":"2024-01-11T08:25:22.859794+0000","last_unstale":"2024-01-11T08:25:23.874537+0000","last_undegraded":"2024-01-11T08:25:23.874537+0000","last_fullsized":"2024-01-11T08:25:23.874537+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:13:42.223026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.862041+0000","last_change":"2024-01-11T08:25:23.862041+0000","last_active":"2024-01-11T08:25:23.862041+0000","last_peered":"2024-01-11T08:25:23.862041+0000","last_clean":"2024-01-11T08:25:23.862041+0000","last_became_active":"2024-01-11T08:25:23.861740+0000","last_became_peered":"2024-01-11T08:25:23.861740+0000","last_unstale":"2024-01-11T08:25:23.862041+0000","last_undegraded":"2024-01-11T08:25:23.862041+0000","last_fullsized":"2024-01-11T08:25:23.862041+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:27:58.197947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"74'2","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.880302+0000","last_change":"2024-01-11T08:25:22.856459+0000","last_active":"2024-01-11T08:25:23.880302+0000","last_peered":"2024-01-11T08:25:23.880302+0000","last_clean":"2024-01-11T08:25:23.880302+0000","last_became_active":"2024-01-11T08:25:22.856212+0000","last_became_peered":"2024-01-11T08:25:22.856212+0000","last_unstale":"2024-01-11T08:25:23.880302+0000","last_undegraded":"2024-01-11T08:25:23.880302+0000","last_fullsized":"2024-01-11T08:25:23.880302+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:33:01.469771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.857559+0000","last_change":"2024-01-11T08:25:23.856416+0000","last_active":"2024-01-11T08:25:23.857559+0000","last_peered":"2024-01-11T08:25:23.857559+0000","last_clean":"2024-01-11T08:25:23.857559+0000","last_became_active":"2024-01-11T08:25:23.856103+0000","last_became_peered":"2024-01-11T08:25:23.856103+0000","last_unstale":"2024-01-11T08:25:23.857559+0000","last_undegraded":"2024-01-11T08:25:23.857559+0000","last_fullsized":"2024-01-11T08:25:23.857559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:52:31.741170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"74'1","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.844649+0000","last_change":"2024-01-11T08:25:22.856103+0000","last_active":"2024-01-11T08:25:23.844649+0000","last_peered":"2024-01-11T08:25:23.844649+0000","last_clean":"2024-01-11T08:25:23.844649+0000","last_became_active":"2024-01-11T08:25:22.855851+0000","last_became_peered":"2024-01-11T08:25:22.855851+0000","last_unstale":"2024-01-11T08:25:23.844649+0000","last_undegraded":"2024-01-11T08:25:23.844649+0000","last_fullsized":"2024-01-11T08:25:23.844649+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:11:45.524653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.878131+0000","last_change":"2024-01-11T08:25:23.877382+0000","last_active":"2024-01-11T08:25:23.878131+0000","last_peered":"2024-01-11T08:25:23.878131+0000","last_clean":"2024-01-11T08:25:23.878131+0000","last_became_active":"2024-01-11T08:25:23.877132+0000","last_became_peered":"2024-01-11T08:25:23.877132+0000","last_unstale":"2024-01-11T08:25:23.878131+0000","last_undegraded":"2024-01-11T08:25:23.878131+0000","last_fullsized":"2024-01-11T08:25:23.878131+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:45:12.148806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-11T08:25:22.853614+0000","last_change":"2024-01-11T08:25:22.849646+0000","last_active":"2024-01-11T08:25:22.853614+0000","last_peered":"2024-01-11T08:25:22.853614+0000","last_clean":"2024-01-11T08:25:22.853614+0000","last_became_active":"2024-01-11T08:25:22.849364+0000","last_became_peered":"2024-01-11T08:25:22.849364+0000","last_unstale":"2024-01-11T08:25:22.853614+0000","last_undegraded":"2024-01-11T08:25:22.853614+0000","last_fullsized":"2024-01-11T08:25:22.853614+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:37:19.713653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.866272+0000","last_change":"2024-01-11T08:25:23.866272+0000","last_active":"2024-01-11T08:25:23.866272+0000","last_peered":"2024-01-11T08:25:23.866272+0000","last_clean":"2024-01-11T08:25:23.866272+0000","last_became_active":"2024-01-11T08:25:23.865969+0000","last_became_peered":"2024-01-11T08:25:23.865969+0000","last_unstale":"2024-01-11T08:25:23.866272+0000","last_undegraded":"2024-01-11T08:25:23.866272+0000","last_fullsized":"2024-01-11T08:25:23.866272+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:00:41.682079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"74'2","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.872667+0000","last_change":"2024-01-11T08:25:22.856273+0000","last_active":"2024-01-11T08:25:23.872667+0000","last_peered":"2024-01-11T08:25:23.872667+0000","last_clean":"2024-01-11T08:25:23.872667+0000","last_became_active":"2024-01-11T08:25:22.854621+0000","last_became_peered":"2024-01-11T08:25:22.854621+0000","last_unstale":"2024-01-11T08:25:23.872667+0000","last_undegraded":"2024-01-11T08:25:23.872667+0000","last_fullsized":"2024-01-11T08:25:23.872667+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:12:21.482148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.867008+0000","last_change":"2024-01-11T08:25:23.867008+0000","last_active":"2024-01-11T08:25:23.867008+0000","last_peered":"2024-01-11T08:25:23.867008+0000","last_clean":"2024-01-11T08:25:23.867008+0000","last_became_active":"2024-01-11T08:25:23.866577+0000","last_became_peered":"2024-01-11T08:25:23.866577+0000","last_unstale":"2024-01-11T08:25:23.867008+0000","last_undegraded":"2024-01-11T08:25:23.867008+0000","last_fullsized":"2024-01-11T08:25:23.867008+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:56:15.839817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.856472+0000","last_change":"2024-01-11T08:25:22.864331+0000","last_active":"2024-01-11T08:25:23.856472+0000","last_peered":"2024-01-11T08:25:23.856472+0000","last_clean":"2024-01-11T08:25:23.856472+0000","last_became_active":"2024-01-11T08:25:22.863989+0000","last_became_peered":"2024-01-11T08:25:22.863989+0000","last_unstale":"2024-01-11T08:25:23.856472+0000","last_undegraded":"2024-01-11T08:25:23.856472+0000","last_fullsized":"2024-01-11T08:25:23.856472+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:05:35.853054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.861329+0000","last_change":"2024-01-11T08:25:23.861329+0000","last_active":"2024-01-11T08:25:23.861329+0000","last_peered":"2024-01-11T08:25:23.861329+0000","last_clean":"2024-01-11T08:25:23.861329+0000","last_became_active":"2024-01-11T08:25:23.861073+0000","last_became_peered":"2024-01-11T08:25:23.861073+0000","last_unstale":"2024-01-11T08:25:23.861329+0000","last_undegraded":"2024-01-11T08:25:23.861329+0000","last_fullsized":"2024-01-11T08:25:23.861329+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:24:45.942689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-11T08:25:22.861169+0000","last_change":"2024-01-11T08:25:22.859051+0000","last_active":"2024-01-11T08:25:22.861169+0000","last_peered":"2024-01-11T08:25:22.861169+0000","last_clean":"2024-01-11T08:25:22.861169+0000","last_became_active":"2024-01-11T08:25:22.858810+0000","last_became_peered":"2024-01-11T08:25:22.858810+0000","last_unstale":"2024-01-11T08:25:22.861169+0000","last_undegraded":"2024-01-11T08:25:22.861169+0000","last_fullsized":"2024-01-11T08:25:22.861169+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:07:22.989253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.856041+0000","last_change":"2024-01-11T08:25:23.856041+0000","last_active":"2024-01-11T08:25:23.856041+0000","last_peered":"2024-01-11T08:25:23.856041+0000","last_clean":"2024-01-11T08:25:23.856041+0000","last_became_active":"2024-01-11T08:25:23.855774+0000","last_became_peered":"2024-01-11T08:25:23.855774+0000","last_unstale":"2024-01-11T08:25:23.856041+0000","last_undegraded":"2024-01-11T08:25:23.856041+0000","last_fullsized":"2024-01-11T08:25:23.856041+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:59:37.011537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"74'2","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.876889+0000","last_change":"2024-01-11T08:25:22.860486+0000","last_active":"2024-01-11T08:25:23.876889+0000","last_peered":"2024-01-11T08:25:23.876889+0000","last_clean":"2024-01-11T08:25:23.876889+0000","last_became_active":"2024-01-11T08:25:22.860182+0000","last_became_peered":"2024-01-11T08:25:22.860182+0000","last_unstale":"2024-01-11T08:25:23.876889+0000","last_undegraded":"2024-01-11T08:25:23.876889+0000","last_fullsized":"2024-01-11T08:25:23.876889+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:31:17.972125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.864323+0000","last_change":"2024-01-11T08:25:23.864323+0000","last_active":"2024-01-11T08:25:23.864323+0000","last_peered":"2024-01-11T08:25:23.864323+0000","last_clean":"2024-01-11T08:25:23.864323+0000","last_became_active":"2024-01-11T08:25:23.864072+0000","last_became_peered":"2024-01-11T08:25:23.864072+0000","last_unstale":"2024-01-11T08:25:23.864323+0000","last_undegraded":"2024-01-11T08:25:23.864323+0000","last_fullsized":"2024-01-11T08:25:23.864323+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:52:31.481386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-11T08:25:22.861915+0000","last_change":"2024-01-11T08:25:22.860575+0000","last_active":"2024-01-11T08:25:22.861915+0000","last_peered":"2024-01-11T08:25:22.861915+0000","last_clean":"2024-01-11T08:25:22.861915+0000","last_became_active":"2024-01-11T08:25:22.858134+0000","last_became_peered":"2024-01-11T08:25:22.858134+0000","last_unstale":"2024-01-11T08:25:22.861915+0000","last_undegraded":"2024-01-11T08:25:22.861915+0000","last_fullsized":"2024-01-11T08:25:22.861915+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:41:22.858068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.876350+0000","last_change":"2024-01-11T08:25:23.868846+0000","last_active":"2024-01-11T08:25:23.876350+0000","last_peered":"2024-01-11T08:25:23.876350+0000","last_clean":"2024-01-11T08:25:23.876350+0000","last_became_active":"2024-01-11T08:25:23.868230+0000","last_became_peered":"2024-01-11T08:25:23.868230+0000","last_unstale":"2024-01-11T08:25:23.876350+0000","last_undegraded":"2024-01-11T08:25:23.876350+0000","last_fullsized":"2024-01-11T08:25:23.876350+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:58:35.728365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.854627+0000","last_change":"2024-01-11T08:25:22.860300+0000","last_active":"2024-01-11T08:25:23.854627+0000","last_peered":"2024-01-11T08:25:23.854627+0000","last_clean":"2024-01-11T08:25:23.854627+0000","last_became_active":"2024-01-11T08:25:22.859664+0000","last_became_peered":"2024-01-11T08:25:22.859664+0000","last_unstale":"2024-01-11T08:25:23.854627+0000","last_undegraded":"2024-01-11T08:25:23.854627+0000","last_fullsized":"2024-01-11T08:25:23.854627+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:57:31.238419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":8,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.873578+0000","last_change":"2024-01-11T08:25:23.873578+0000","last_active":"2024-01-11T08:25:23.873578+0000","last_peered":"2024-01-11T08:25:23.873578+0000","last_clean":"2024-01-11T08:25:23.873578+0000","last_became_active":"2024-01-11T08:25:23.870192+0000","last_became_peered":"2024-01-11T08:25:23.870192+0000","last_unstale":"2024-01-11T08:25:23.873578+0000","last_undegraded":"2024-01-11T08:25:23.873578+0000","last_fullsized":"2024-01-11T08:25:23.873578+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:55:26.350726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-11T08:25:22.857672+0000","last_change":"2024-01-11T08:25:22.855065+0000","last_active":"2024-01-11T08:25:22.857672+0000","last_peered":"2024-01-11T08:25:22.857672+0000","last_clean":"2024-01-11T08:25:22.857672+0000","last_became_active":"2024-01-11T08:25:22.852962+0000","last_became_peered":"2024-01-11T08:25:22.852962+0000","last_unstale":"2024-01-11T08:25:22.857672+0000","last_undegraded":"2024-01-11T08:25:22.857672+0000","last_fullsized":"2024-01-11T08:25:22.857672+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:47:39.729413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.870190+0000","last_change":"2024-01-11T08:25:23.865663+0000","last_active":"2024-01-11T08:25:23.870190+0000","last_peered":"2024-01-11T08:25:23.870190+0000","last_clean":"2024-01-11T08:25:23.870190+0000","last_became_active":"2024-01-11T08:25:23.865174+0000","last_became_peered":"2024-01-11T08:25:23.865174+0000","last_unstale":"2024-01-11T08:25:23.870190+0000","last_undegraded":"2024-01-11T08:25:23.870190+0000","last_fullsized":"2024-01-11T08:25:23.870190+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:50:34.445851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.855983+0000","last_change":"2024-01-11T08:25:22.855435+0000","last_active":"2024-01-11T08:25:23.855983+0000","last_peered":"2024-01-11T08:25:23.855983+0000","last_clean":"2024-01-11T08:25:23.855983+0000","last_became_active":"2024-01-11T08:25:22.855192+0000","last_became_peered":"2024-01-11T08:25:22.855192+0000","last_unstale":"2024-01-11T08:25:23.855983+0000","last_undegraded":"2024-01-11T08:25:23.855983+0000","last_fullsized":"2024-01-11T08:25:23.855983+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:19:49.995942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.876134+0000","last_change":"2024-01-11T08:25:23.872878+0000","last_active":"2024-01-11T08:25:23.876134+0000","last_peered":"2024-01-11T08:25:23.876134+0000","last_clean":"2024-01-11T08:25:23.876134+0000","last_became_active":"2024-01-11T08:25:23.872474+0000","last_became_peered":"2024-01-11T08:25:23.872474+0000","last_unstale":"2024-01-11T08:25:23.876134+0000","last_undegraded":"2024-01-11T08:25:23.876134+0000","last_fullsized":"2024-01-11T08:25:23.876134+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:39:15.996906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-01-11T08:25:22.862890+0000","last_change":"2024-01-11T08:25:22.860552+0000","last_active":"2024-01-11T08:25:22.862890+0000","last_peered":"2024-01-11T08:25:22.862890+0000","last_clean":"2024-01-11T08:25:22.862890+0000","last_became_active":"2024-01-11T08:25:22.860297+0000","last_became_peered":"2024-01-11T08:25:22.860297+0000","last_unstale":"2024-01-11T08:25:22.862890+0000","last_undegraded":"2024-01-11T08:25:22.862890+0000","last_fullsized":"2024-01-11T08:25:22.862890+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:20:30.108010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.875884+0000","last_change":"2024-01-11T08:25:23.863643+0000","last_active":"2024-01-11T08:25:23.875884+0000","last_peered":"2024-01-11T08:25:23.875884+0000","last_clean":"2024-01-11T08:25:23.875884+0000","last_became_active":"2024-01-11T08:25:23.861786+0000","last_became_peered":"2024-01-11T08:25:23.861786+0000","last_unstale":"2024-01-11T08:25:23.875884+0000","last_undegraded":"2024-01-11T08:25:23.875884+0000","last_fullsized":"2024-01-11T08:25:23.875884+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:42:53.264984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.858272+0000","last_change":"2024-01-11T08:25:22.860686+0000","last_active":"2024-01-11T08:25:23.858272+0000","last_peered":"2024-01-11T08:25:23.858272+0000","last_clean":"2024-01-11T08:25:23.858272+0000","last_became_active":"2024-01-11T08:25:22.860417+0000","last_became_peered":"2024-01-11T08:25:22.860417+0000","last_unstale":"2024-01-11T08:25:23.858272+0000","last_undegraded":"2024-01-11T08:25:23.858272+0000","last_fullsized":"2024-01-11T08:25:23.858272+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:43:04.458768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.867426+0000","last_change":"2024-01-11T08:25:23.859836+0000","last_active":"2024-01-11T08:25:23.867426+0000","last_peered":"2024-01-11T08:25:23.867426+0000","last_clean":"2024-01-11T08:25:23.867426+0000","last_became_active":"2024-01-11T08:25:23.858584+0000","last_became_peered":"2024-01-11T08:25:23.858584+0000","last_unstale":"2024-01-11T08:25:23.867426+0000","last_undegraded":"2024-01-11T08:25:23.867426+0000","last_fullsized":"2024-01-11T08:25:23.867426+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:53:58.821688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.841751+0000","last_change":"2024-01-11T08:25:22.856991+0000","last_active":"2024-01-11T08:25:23.841751+0000","last_peered":"2024-01-11T08:25:23.841751+0000","last_clean":"2024-01-11T08:25:23.841751+0000","last_became_active":"2024-01-11T08:25:22.854384+0000","last_became_peered":"2024-01-11T08:25:22.854384+0000","last_unstale":"2024-01-11T08:25:23.841751+0000","last_undegraded":"2024-01-11T08:25:23.841751+0000","last_fullsized":"2024-01-11T08:25:23.841751+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:11:45.524653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.867961+0000","last_change":"2024-01-11T08:25:23.860721+0000","last_active":"2024-01-11T08:25:23.867961+0000","last_peered":"2024-01-11T08:25:23.867961+0000","last_clean":"2024-01-11T08:25:23.867961+0000","last_became_active":"2024-01-11T08:25:23.860438+0000","last_became_peered":"2024-01-11T08:25:23.860438+0000","last_unstale":"2024-01-11T08:25:23.867961+0000","last_undegraded":"2024-01-11T08:25:23.867961+0000","last_fullsized":"2024-01-11T08:25:23.867961+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:22.829720+0000","last_clean_scrub_stamp":"2024-01-11T08:25:22.829720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:59.235791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.854360+0000","last_change":"2024-01-11T08:25:22.858950+0000","last_active":"2024-01-11T08:25:23.854360+0000","last_peered":"2024-01-11T08:25:23.854360+0000","last_clean":"2024-01-11T08:25:23.854360+0000","last_became_active":"2024-01-11T08:25:22.858632+0000","last_became_peered":"2024-01-11T08:25:22.858632+0000","last_unstale":"2024-01-11T08:25:23.854360+0000","last_undegraded":"2024-01-11T08:25:23.854360+0000","last_fullsized":"2024-01-11T08:25:23.854360+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:21.820224+0000","last_clean_scrub_stamp":"2024-01-11T08:25:21.820224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:20:11.856429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"66'161","reported_seq":361,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-01-11T08:25:23.867154+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:25:23.867154+0000","last_peered":"2024-01-11T08:25:23.867154+0000","last_clean":"2024-01-11T08:25:23.867154+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:25:23.867154+0000","last_undegraded":"2024-01-11T08:25:23.867154+0000","last_fullsized":"2024-01-11T08:25:23.867154+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575030,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302856,"kb_used_data":1472,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068984,"statfs":{"total":96636764160,"available":96326639616,"internally_reserved":0,"allocated":1507328,"data_stored":1007067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.71899999999999997,"15min":0.70799999999999996},"min":{"1min":0.45300000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.95899999999999996,"5min":0.996,"15min":2.8300000000000001},"last":0.62},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.71499999999999997,"15min":0.70999999999999996},"min":{"1min":0.55600000000000005,"5min":0.40500000000000003,"15min":0.372},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.54500000000000004}]},{"osd":1,"last update":"Thu Jan 11 08:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.71399999999999997,"15min":0.70399999999999996},"min":{"1min":0.53300000000000003,"5min":0.41899999999999998,"15min":0.35799999999999998},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.998},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71999999999999997,"15min":0.72099999999999997},"min":{"1min":0.58299999999999996,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93999999999999995,"5min":0.95899999999999996,"15min":1.153},"last":0.59199999999999997}]},{"osd":2,"last update":"Thu Jan 11 08:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.73099999999999998,"15min":0.72799999999999998},"min":{"1min":0.68700000000000006,"5min":0.436,"15min":0.436},"max":{"1min":0.96499999999999997,"5min":1.03,"15min":1.034},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.73099999999999998,"15min":0.72099999999999997},"min":{"1min":0.54800000000000004,"5min":0.46999999999999997,"15min":0.41099999999999998},"max":{"1min":0.98299999999999998,"5min":0.99399999999999999,"15min":1.0049999999999999},"last":0.55700000000000005}]},{"osd":3,"last update":"Thu Jan 11 08:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.746,"15min":0.75600000000000001},"min":{"1min":0.56899999999999995,"5min":0.48299999999999998,"15min":0.40300000000000002},"max":{"1min":0.876,"5min":1.032,"15min":2.9119999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.73299999999999998,"15min":0.74199999999999999},"min":{"1min":0.64700000000000002,"5min":0.50800000000000001,"15min":0.48999999999999999},"max":{"1min":0.92000000000000004,"5min":1.008,"15min":1.008},"last":0.64300000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.47099999999999997,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":1.089},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.63500000000000001,"15min":0.63600000000000001},"min":{"1min":0.48099999999999998,"5min":0.42699999999999999,"15min":0.39500000000000002},"max":{"1min":0.85999999999999999,"5min":0.90700000000000003,"15min":1.1020000000000001},"last":0.52600000000000002}]},{"osd":5,"last update":"Thu Jan 11 08:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.63200000000000001,"15min":0.64700000000000002},"min":{"1min":0.42299999999999999,"5min":0.42099999999999999,"15min":0.34599999999999997},"max":{"1min":0.85399999999999998,"5min":0.92200000000000004,"15min":1.0589999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.67200000000000004,"15min":0.68300000000000005},"min":{"1min":0.47799999999999998,"5min":0.47199999999999998,"15min":0.40400000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.032},"last":0.63200000000000001}]},{"osd":6,"last update":"Thu Jan 11 08:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.748,"15min":0.76300000000000001},"min":{"1min":0.64100000000000001,"5min":0.52000000000000002,"15min":0.46999999999999997},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.1359999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73199999999999998,"15min":0.74399999999999999},"min":{"1min":0.44900000000000001,"5min":0.437,"15min":0.42099999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.1299999999999999},"last":0.70299999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575029,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302256,"kb_used_data":872,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069584,"statfs":{"total":96636764160,"available":96327254016,"internally_reserved":0,"allocated":892928,"data_stored":410223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.66800000000000004,"15min":0.66000000000000003},"min":{"1min":0.56499999999999995,"5min":0.379,"15min":0.29999999999999999},"max":{"1min":0.87,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68200000000000005,"15min":0.68100000000000005},"min":{"1min":0.41499999999999998,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":0.92800000000000005,"5min":0.96999999999999997,"15min":1},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Jan 11 08:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68999999999999995,"15min":0.68100000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.35899999999999999},"max":{"1min":0.93200000000000005,"5min":1.163,"15min":1.163},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.69199999999999995,"15min":0.68999999999999995},"min":{"1min":0.53500000000000003,"5min":0.435,"15min":0.39900000000000002},"max":{"1min":0.81000000000000005,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.81599999999999995}]},{"osd":2,"last update":"Thu Jan 11 08:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70699999999999996,"15min":0.70299999999999996},"min":{"1min":0.501,"5min":0.46899999999999997,"15min":0.38600000000000001},"max":{"1min":0.95399999999999996,"5min":0.97199999999999998,"15min":1.016},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71799999999999997,"15min":0.71999999999999997},"min":{"1min":0.51900000000000002,"5min":0.46200000000000002,"15min":0.373},"max":{"1min":0.91100000000000003,"5min":0.96699999999999997,"15min":0.995},"last":0.64700000000000002}]},{"osd":3,"last update":"Thu Jan 11 08:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72399999999999998,"15min":0.73399999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.42799999999999999},"max":{"1min":0.86899999999999999,"5min":0.95799999999999996,"15min":1.3260000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.73999999999999999,"15min":0.73399999999999999},"min":{"1min":0.55000000000000004,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.90900000000000003,"5min":0.97999999999999998,"15min":1.0049999999999999},"last":0.84199999999999997}]},{"osd":4,"last update":"Thu Jan 11 08:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.54500000000000004,"15min":0.57299999999999995},"min":{"1min":0.41899999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.61399999999999999,"5min":0.874,"15min":1.212},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58099999999999996,"15min":0.61599999999999999},"min":{"1min":0.438,"5min":0.38900000000000001,"15min":0.374},"max":{"1min":0.70999999999999996,"5min":0.96399999999999997,"15min":1.8320000000000001},"last":0.60099999999999998}]},{"osd":5,"last update":"Thu Jan 11 08:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63700000000000001,"15min":0.65800000000000003},"min":{"1min":0.432,"5min":0.432,"15min":0.39900000000000002},"max":{"1min":0.84999999999999998,"5min":0.95499999999999996,"15min":1.0209999999999999},"last":1.03},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.68300000000000005,"15min":0.70799999999999996},"min":{"1min":0.50600000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88200000000000001,"5min":0.89500000000000002,"15min":1.7929999999999999},"last":0.90800000000000003}]},{"osd":7,"last update":"Thu Jan 11 08:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.69399999999999995,"15min":0.71899999999999997},"min":{"1min":0.53200000000000003,"5min":0.51800000000000002,"15min":0.41099999999999998},"max":{"1min":0.86699999999999999,"5min":0.879,"15min":1.278},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.73399999999999999,"15min":0.751},"min":{"1min":0.46700000000000003,"5min":0.45900000000000002,"15min":0.44700000000000001},"max":{"1min":0.98999999999999999,"5min":1.022,"15min":1.1599999999999999},"last":1.0009999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575031,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302272,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069568,"statfs":{"total":96636764160,"available":96327237632,"internally_reserved":0,"allocated":909312,"data_stored":416629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.501,"15min":0.49099999999999999},"min":{"1min":0.42199999999999999,"5min":0.30599999999999999,"15min":0.26900000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.97699999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.58599999999999997,"15min":0.57899999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.373},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.497,"15min":0.48999999999999999},"min":{"1min":0.36099999999999999,"5min":0.312,"15min":0.312},"max":{"1min":1.0960000000000001,"5min":1.113,"15min":1.113},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.56399999999999995,"15min":0.56899999999999995},"min":{"1min":0.42399999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76700000000000002,"5min":0.82399999999999995,"15min":1.25},"last":0.69699999999999995}]},{"osd":3,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.51000000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55400000000000005,"15min":0.56200000000000006},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.34899999999999998},"max":{"1min":0.78100000000000003,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.73199999999999998}]},{"osd":4,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.78100000000000003,"15min":0.78700000000000003},"min":{"1min":0.53300000000000003,"5min":0.439,"15min":0.439},"max":{"1min":1.0800000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.82799999999999996,"15min":0.83199999999999996},"min":{"1min":0.63500000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.0329999999999999,"5min":1.246,"15min":1.3680000000000001},"last":0.76800000000000002}]},{"osd":5,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78600000000000003,"15min":0.80300000000000005},"min":{"1min":0.57799999999999996,"5min":0.502,"15min":0.46999999999999997},"max":{"1min":1.0649999999999999,"5min":1.1759999999999999,"15min":1.7849999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.79300000000000004,"15min":0.80100000000000005},"min":{"1min":0.52900000000000003,"5min":0.52500000000000002,"15min":0.50700000000000001},"max":{"1min":1.1339999999999999,"5min":1.1579999999999999,"15min":1.371},"last":0.83699999999999997}]},{"osd":6,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.81899999999999995,"15min":0.82199999999999995},"min":{"1min":0.70899999999999996,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0289999999999999,"5min":1.0840000000000001,"15min":1.413},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.82899999999999996,"15min":0.82599999999999996},"min":{"1min":0.621,"5min":0.621,"15min":0.55500000000000005},"max":{"1min":1.0469999999999999,"5min":1.228,"15min":1.228},"last":0.77900000000000003}]},{"osd":7,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84599999999999997},"min":{"1min":0.625,"5min":0.59199999999999997,"15min":0.52700000000000002},"max":{"1min":1.0109999999999999,"5min":1.147,"15min":1.226},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.84199999999999997,"15min":0.84099999999999997},"min":{"1min":0.65800000000000003,"5min":0.58599999999999997,"15min":0.51400000000000001},"max":{"1min":1.0669999999999999,"5min":1.258,"15min":1.321},"last":0.92200000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575030,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298756,"kb_used_data":1468,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073084,"statfs":{"total":96636764160,"available":96330838016,"internally_reserved":0,"allocated":1503232,"data_stored":1002681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,9],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.495,"15min":0.499},"min":{"1min":0.34899999999999998,"5min":0.29999999999999999,"15min":0.24299999999999999},"max":{"1min":0.74099999999999999,"5min":0.77700000000000002,"15min":0.92700000000000005},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.502,"5min":0.41299999999999998,"15min":0.35299999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.92100000000000004},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54400000000000004,"15min":0.54900000000000004},"min":{"1min":0.42799999999999999,"5min":0.34799999999999998,"15min":0.29099999999999998},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.86799999999999999},"last":1.335},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.53900000000000003,"15min":0.54900000000000004},"min":{"1min":0.44600000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69999999999999996,"5min":0.78500000000000003,"15min":0.83999999999999997},"last":0.68600000000000005}]},{"osd":3,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.52000000000000002,"15min":0.52200000000000002},"min":{"1min":0.39200000000000002,"5min":0.35999999999999999,"15min":0.33600000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.621,"15min":0.622},"min":{"1min":0.48899999999999999,"5min":0.45600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84899999999999998,"5min":0.877,"15min":0.91000000000000003},"last":1.4119999999999999}]},{"osd":4,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.79400000000000004,"15min":0.78900000000000003},"min":{"1min":0.64300000000000002,"5min":0.45700000000000002,"15min":0.437},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.79800000000000004,"15min":0.79000000000000004},"min":{"1min":0.64900000000000002,"5min":0.53100000000000003,"15min":0.46899999999999997},"max":{"1min":1.0209999999999999,"5min":1.038,"15min":1.2350000000000001},"last":0.84499999999999997}]},{"osd":5,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.79800000000000004,"15min":0.80600000000000005},"min":{"1min":0.69399999999999995,"5min":0.52500000000000002,"15min":0.505},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.2929999999999999},"last":1.518},{"interface":"front","average":{"1min":0.88,"5min":0.82099999999999995,"15min":0.81799999999999995},"min":{"1min":0.68300000000000005,"5min":0.49099999999999999,"15min":0.47799999999999998},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.85899999999999999}]},{"osd":6,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.80700000000000005,"15min":0.80100000000000005},"min":{"1min":0.70399999999999996,"5min":0.50900000000000001,"15min":0.497},"max":{"1min":1.163,"5min":1.163,"15min":1.2689999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.82699999999999996,"15min":0.83399999999999996},"min":{"1min":0.73899999999999999,"5min":0.54700000000000004,"15min":0.44900000000000001},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.2749999999999999},"last":1.5589999999999999}]},{"osd":7,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.83799999999999997,"15min":0.84099999999999997},"min":{"1min":0.63200000000000001,"5min":0.57999999999999996,"15min":0.51400000000000001},"max":{"1min":1.0329999999999999,"5min":1.071,"15min":1.446},"last":1.024},{"interface":"front","average":{"1min":0.877,"5min":0.84499999999999997,"15min":0.84299999999999997},"min":{"1min":0.64100000000000001,"5min":0.60499999999999998,"15min":0.57699999999999996},"max":{"1min":1.1539999999999999,"5min":1.1659999999999999,"15min":1.298},"last":1.0760000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306372,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065468,"statfs":{"total":96636764160,"available":96323039232,"internally_reserved":0,"allocated":913408,"data_stored":416595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:24:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.47899999999999998,"15min":0.47099999999999997},"min":{"1min":0.38900000000000001,"5min":0.29899999999999999,"15min":0.26000000000000001},"max":{"1min":0.70499999999999996,"5min":0.81399999999999995,"15min":0.86099999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53600000000000003,"15min":0.55200000000000005},"min":{"1min":0.34399999999999997,"5min":0.33100000000000002,"15min":0.28899999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":1.4119999999999999},"last":0.45900000000000002}]},{"osd":1,"last update":"Thu Jan 11 08:24:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54100000000000004,"15min":0.54900000000000004},"min":{"1min":0.374,"5min":0.36399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68500000000000005,"5min":0.80300000000000005,"15min":0.90000000000000002},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.57599999999999996,"15min":0.59099999999999997},"min":{"1min":0.47599999999999998,"5min":0.39200000000000002,"15min":0.38100000000000001},"max":{"1min":0.75700000000000001,"5min":0.92900000000000005,"15min":1.5640000000000001},"last":0.52500000000000002}]},{"osd":3,"last update":"Thu Jan 11 08:24:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53900000000000003,"15min":0.54900000000000004},"min":{"1min":0.39400000000000002,"5min":0.36199999999999999,"15min":0.32900000000000001},"max":{"1min":0.71299999999999997,"5min":0.82499999999999996,"15min":1.1220000000000001},"last":0.499},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.624,"15min":0.64500000000000002},"min":{"1min":0.47099999999999997,"5min":0.45300000000000001,"15min":0.41799999999999998},"max":{"1min":0.80700000000000005,"5min":0.95299999999999996,"15min":1.22},"last":0.435}]},{"osd":4,"last update":"Thu Jan 11 08:24:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.78700000000000003,"15min":0.79500000000000004},"min":{"1min":0.55400000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.98699999999999999,"5min":1.0680000000000001,"15min":1.488},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.78200000000000003,"15min":0.78600000000000003},"min":{"1min":0.57799999999999996,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0660000000000001,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.74399999999999999}]},{"osd":5,"last update":"Thu Jan 11 08:24:42 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.83199999999999996,"15min":0.85099999999999998},"min":{"1min":0.70299999999999996,"5min":0.622,"15min":0.51600000000000001},"max":{"1min":1.087,"5min":1.214,"15min":2.9209999999999998},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.878,"5min":0.84499999999999997,"15min":0.85599999999999998},"min":{"1min":0.67200000000000004,"5min":0.55500000000000005,"15min":0.55400000000000005},"max":{"1min":1.2,"5min":1.2,"15min":1.845},"last":0.872}]},{"osd":6,"last update":"Thu Jan 11 08:24:42 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.81000000000000005,"15min":0.80600000000000005},"min":{"1min":0.54300000000000004,"5min":0.54200000000000004,"15min":0.51700000000000002},"max":{"1min":1.131,"5min":1.1919999999999999,"15min":1.3560000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82299999999999995,"15min":0.84299999999999997},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0529999999999999,"5min":1.123,"15min":2.968},"last":0.72399999999999998}]},{"osd":7,"last update":"Thu Jan 11 08:24:42 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.86599999999999999,"15min":0.879},"min":{"1min":0.71799999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.6359999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.85299999999999998,"15min":0.86899999999999999},"min":{"1min":0.63700000000000001,"5min":0.58699999999999997,"15min":0.53800000000000003},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.391},"last":0.85899999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575029,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306372,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065468,"statfs":{"total":96636764160,"available":96323039232,"internally_reserved":0,"allocated":913408,"data_stored":416595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.502,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.29499999999999998,"15min":0.27600000000000002},"max":{"1min":0.79800000000000004,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53600000000000003,"15min":0.54600000000000004},"min":{"1min":0.42199999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67400000000000004,"5min":0.72199999999999998,"15min":1.0109999999999999},"last":0.49299999999999999}]},{"osd":1,"last update":"Thu Jan 11 08:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46300000000000002,"15min":0.48599999999999999},"min":{"1min":0.34000000000000002,"5min":0.30299999999999999,"15min":0.25900000000000001},"max":{"1min":0.626,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58799999999999997,"15min":0.57099999999999995},"min":{"1min":0.45400000000000001,"5min":0.36299999999999999,"15min":0.32300000000000001},"max":{"1min":0.74299999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.40699999999999997}]},{"osd":2,"last update":"Thu Jan 11 08:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.46300000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.74099999999999999,"5min":0.90900000000000003,"15min":0.97599999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63500000000000001,"15min":0.63100000000000001},"min":{"1min":0.48599999999999999,"5min":0.44700000000000001,"15min":0.32000000000000001},"max":{"1min":0.86199999999999999,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.45900000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.71799999999999997,"15min":0.749},"min":{"1min":0.51700000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.038,"5min":1.038,"15min":1.351},"last":0.622},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.77200000000000002,"15min":0.78200000000000003},"min":{"1min":0.56599999999999995,"5min":0.503,"15min":0.503},"max":{"1min":0.92100000000000004,"5min":1.1819999999999999,"15min":1.198},"last":0.67600000000000005}]},{"osd":5,"last update":"Thu Jan 11 08:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.76800000000000002,"15min":0.79300000000000004},"min":{"1min":0.55800000000000005,"5min":0.53500000000000003,"15min":0.51900000000000002},"max":{"1min":0.96499999999999997,"5min":1.139,"15min":1.3779999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79400000000000004,"15min":0.81000000000000005},"min":{"1min":0.64300000000000002,"5min":0.49199999999999999,"15min":0.44900000000000001},"max":{"1min":1.016,"5min":1.2809999999999999,"15min":1.4239999999999999},"last":0.63600000000000001}]},{"osd":6,"last update":"Thu Jan 11 08:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.82399999999999995,"15min":0.83999999999999997},"min":{"1min":0.625,"5min":0.56799999999999995,"15min":0.52600000000000002},"max":{"1min":1.093,"5min":1.335,"15min":1.927},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.81999999999999995},"min":{"1min":0.61099999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.97599999999999998,"5min":1.2230000000000001,"15min":3.5579999999999998},"last":0.57999999999999996}]},{"osd":7,"last update":"Thu Jan 11 08:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81799999999999995,"15min":0.85499999999999998},"min":{"1min":0.55000000000000004,"5min":0.52800000000000002,"15min":0.46600000000000003},"max":{"1min":1.081,"5min":1.1579999999999999,"15min":3.5089999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.88,"5min":0.84699999999999998,"15min":0.86199999999999999},"min":{"1min":0.70299999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.0609999999999999,"5min":1.1879999999999999,"15min":1.21},"last":0.69899999999999995}]}]},{"osd":4,"up_from":13,"seq":55834575030,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310460,"kb_used_data":884,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94061380,"statfs":{"total":96636764160,"available":96318853120,"internally_reserved":0,"allocated":905216,"data_stored":411759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.748,"15min":0.74099999999999999},"min":{"1min":0.47299999999999998,"5min":0.38900000000000001,"15min":0.28100000000000003},"max":{"1min":0.89600000000000002,"5min":0.95999999999999996,"15min":1.524},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.74299999999999999,"15min":0.748},"min":{"1min":0.54800000000000004,"5min":0.51600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.177},"last":0.72299999999999998}]},{"osd":1,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.70199999999999996,"15min":0.70299999999999996},"min":{"1min":0.52100000000000002,"5min":0.42199999999999999,"15min":0.33300000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.1499999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.70899999999999996,"15min":0.73499999999999999},"min":{"1min":0.52500000000000002,"5min":0.379,"15min":0.36299999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":2.4740000000000002},"last":0.43099999999999999}]},{"osd":2,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.72199999999999998,"15min":0.71699999999999997},"min":{"1min":0.54800000000000004,"5min":0.46000000000000002,"15min":0.38900000000000001},"max":{"1min":0.97499999999999998,"5min":0.98599999999999999,"15min":1.022},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.747,"5min":0.751,"15min":0.73499999999999999},"min":{"1min":0.54000000000000004,"5min":0.48499999999999999,"15min":0.47199999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.212},"last":0.53700000000000003}]},{"osd":3,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70299999999999996,"15min":0.71199999999999997},"min":{"1min":0.497,"5min":0.433,"15min":0.38100000000000001},"max":{"1min":0.81599999999999995,"5min":1.0189999999999999,"15min":1.2649999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.75,"15min":0.75800000000000001},"min":{"1min":0.63800000000000001,"5min":0.50900000000000001,"15min":0.47499999999999998},"max":{"1min":1.026,"5min":1.026,"15min":1.069},"last":0.502}]},{"osd":5,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.60699999999999998,"15min":0.626},"min":{"1min":0.45000000000000001,"5min":0.38700000000000001,"15min":0.35899999999999999},"max":{"1min":0.873,"5min":0.873,"15min":1.0700000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.621,"15min":0.63900000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.32600000000000001},"max":{"1min":0.80100000000000005,"5min":0.97799999999999998,"15min":0.98199999999999998},"last":0.78100000000000003}]},{"osd":6,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.65000000000000002,"15min":0.66200000000000003},"min":{"1min":0.439,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93100000000000005,"5min":1.133,"15min":1.133},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.68600000000000005,"15min":0.70099999999999996},"min":{"1min":0.497,"5min":0.42799999999999999,"15min":0.40400000000000003},"max":{"1min":0.88300000000000001,"5min":1.0269999999999999,"15min":1.119},"last":0.68600000000000005}]},{"osd":7,"last update":"Thu Jan 11 08:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68100000000000005,"15min":0.69999999999999996},"min":{"1min":0.47099999999999997,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85099999999999998,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.747,"5min":0.74299999999999999,"15min":0.752},"min":{"1min":0.49299999999999999,"5min":0.46200000000000002,"15min":0.42299999999999999},"max":{"1min":1.127,"5min":1.2350000000000001,"15min":1.4279999999999999},"last":0.75600000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575030,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302264,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069576,"statfs":{"total":96636764160,"available":96327245824,"internally_reserved":0,"allocated":901120,"data_stored":410847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.71399999999999997,"15min":0.70899999999999996},"min":{"1min":0.52700000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.95399999999999996,"5min":1.0249999999999999,"15min":1.032},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.70899999999999996,"15min":0.70799999999999996},"min":{"1min":0.54500000000000004,"5min":0.437,"15min":0.437},"max":{"1min":0.96199999999999997,"5min":1.0609999999999999,"15min":1.696},"last":0.58199999999999996}]},{"osd":1,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.68600000000000005,"15min":0.70099999999999996},"min":{"1min":0.48999999999999999,"5min":0.39400000000000002,"15min":0.36399999999999999},"max":{"1min":0.84799999999999998,"5min":1.0089999999999999,"15min":1.4359999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.72999999999999998,"15min":0.73499999999999999},"min":{"1min":0.5,"5min":0.48299999999999998,"15min":0.47299999999999998},"max":{"1min":0.90300000000000002,"5min":0.97399999999999998,"15min":4.4969999999999999},"last":0.61099999999999999}]},{"osd":2,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.72099999999999997,"15min":0.70899999999999996},"min":{"1min":0.48099999999999998,"5min":0.435,"15min":0.435},"max":{"1min":0.95999999999999996,"5min":1.028,"15min":1.238},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.70999999999999996,"15min":0.749},"min":{"1min":0.52200000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.90200000000000002,"5min":1.0049999999999999,"15min":8.0429999999999993},"last":0.55500000000000005}]},{"osd":3,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.72199999999999998,"15min":0.76000000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.442},"max":{"1min":0.88700000000000001,"5min":1.0449999999999999,"15min":8.0839999999999996},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.746,"15min":0.748},"min":{"1min":0.439,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.98499999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.64600000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.60399999999999998,"15min":0.61799999999999999},"min":{"1min":0.41699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.71799999999999997,"5min":0.96599999999999997,"15min":1.113},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":0.82499999999999996,"5min":0.88600000000000001,"15min":1.002},"last":0.752}]},{"osd":6,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.65200000000000002,"15min":0.65400000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80800000000000005,"5min":0.86899999999999999,"15min":1.034},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.68400000000000005,"15min":0.69399999999999995},"min":{"1min":0.499,"5min":0.499,"15min":0.45400000000000001},"max":{"1min":0.77800000000000002,"5min":0.94599999999999995,"15min":1.143},"last":0.67800000000000005}]},{"osd":7,"last update":"Thu Jan 11 08:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.73699999999999999,"15min":0.747},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.437},"max":{"1min":0.875,"5min":1.028,"15min":1.1399999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.747,"5min":0.77800000000000002,"15min":0.77500000000000002},"min":{"1min":0.51200000000000001,"5min":0.5,"15min":0.5},"max":{"1min":0.92500000000000004,"5min":0.96699999999999997,"15min":1.079},"last":0.76000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:25:28.758 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-11T08:25:28.759 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-11T08:25:29.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.001+0000 7f7d293ff700 1 Processor -- start 2024-01-11T08:25:29.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.001+0000 7f7d293ff700 1 -- start start 2024-01-11T08:25:29.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.002+0000 7f7d293ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d240b03e0 0x7f7d240b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:29.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.002+0000 7f7d293ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7d240b07b0 0x7f7d240b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:29.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.002+0000 7f7d293ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7d24159200 0x7f7d24159640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:29.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.002+0000 7f7d293ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d240cd730 con 0x7f7d24159200 2024-01-11T08:25:29.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.002+0000 7f7d293ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d240ccb20 con 0x7f7d240b07b0 2024-01-11T08:25:29.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.002+0000 7f7d293ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d240cce00 con 0x7f7d240b03e0 2024-01-11T08:25:29.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.002+0000 7f7d237fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7d24159200 0x7f7d24159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:29.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.002+0000 7f7d237fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7d24159200 0x7f7d24159640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56972/0 (socket says 172.21.15.27:56972) 2024-01-11T08:25:29.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.002+0000 7f7d237fe700 1 -- 172.21.15.27:0/705313833 learned_addr learned my addr 172.21.15.27:0/705313833 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:29.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.002+0000 7f7d22ffd700 1 --2- 172.21.15.27:0/705313833 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d240b03e0 0x7f7d240b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:29.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.003+0000 7f7d227fc700 1 --2- 172.21.15.27:0/705313833 >> v2:172.21.15.77:3300/0 conn(0x7f7d240b07b0 0x7f7d240b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:29.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.003+0000 7f7d237fe700 1 -- 172.21.15.27:0/705313833 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d240b03e0 msgr2=0x7f7d240b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:29.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.003+0000 7f7d237fe700 1 --2- 172.21.15.27:0/705313833 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d240b03e0 0x7f7d240b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.003+0000 7f7d237fe700 1 -- 172.21.15.27:0/705313833 >> v2:172.21.15.77:3300/0 conn(0x7f7d240b07b0 msgr2=0x7f7d240b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:29.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.003+0000 7f7d237fe700 1 --2- 172.21.15.27:0/705313833 >> v2:172.21.15.77:3300/0 conn(0x7f7d240b07b0 0x7f7d240b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.003+0000 7f7d237fe700 1 -- 172.21.15.27:0/705313833 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d240cd0e0 con 0x7f7d24159200 2024-01-11T08:25:29.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.003+0000 7f7d227fc700 1 --2- 172.21.15.27:0/705313833 >> v2:172.21.15.77:3300/0 conn(0x7f7d240b07b0 0x7f7d240b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:29.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.003+0000 7f7d237fe700 1 --2- 172.21.15.27:0/705313833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d24159200 0x7f7d24159640 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7d240893d0 tx=0x7f7d1c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c03d3258e2797ba server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:29.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.003+0000 7f7d21ffb700 1 -- 172.21.15.27:0/705313833 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c00e070 con 0x7f7d24159200 2024-01-11T08:25:29.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.003+0000 7f7d21ffb700 1 -- 172.21.15.27:0/705313833 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1c00ba20 con 0x7f7d24159200 2024-01-11T08:25:29.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.003+0000 7f7d21ffb700 1 -- 172.21.15.27:0/705313833 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c0092e0 con 0x7f7d24159200 2024-01-11T08:25:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.004+0000 7f7d293ff700 1 -- 172.21.15.27:0/705313833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d24159200 msgr2=0x7f7d24159640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.004+0000 7f7d293ff700 1 --2- 172.21.15.27:0/705313833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d24159200 0x7f7d24159640 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7d240893d0 tx=0x7f7d1c009840 comp rx=0 tx=0).stop 2024-01-11T08:25:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.004+0000 7f7d293ff700 1 -- 172.21.15.27:0/705313833 shutdown_connections 2024-01-11T08:25:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.004+0000 7f7d293ff700 1 --2- 172.21.15.27:0/705313833 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d240b03e0 0x7f7d240b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.004+0000 7f7d293ff700 1 --2- 172.21.15.27:0/705313833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d24159200 0x7f7d24159640 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.004+0000 7f7d293ff700 1 --2- 172.21.15.27:0/705313833 >> v2:172.21.15.77:3300/0 conn(0x7f7d240b07b0 0x7f7d240b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.004+0000 7f7d293ff700 1 -- 172.21.15.27:0/705313833 >> 172.21.15.27:0/705313833 conn(0x7f7d2406ed40 msgr2=0x7f7d2406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.004+0000 7f7d293ff700 1 -- 172.21.15.27:0/705313833 shutdown_connections 2024-01-11T08:25:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.004+0000 7f7d293ff700 1 -- 172.21.15.27:0/705313833 wait complete. 2024-01-11T08:25:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.005+0000 7f7d293ff700 1 Processor -- start 2024-01-11T08:25:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.006+0000 7f7d293ff700 1 -- start start 2024-01-11T08:25:29.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.006+0000 7f7d293ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d240b03e0 0x7f7d240af8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:29.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.006+0000 7f7d293ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d240b07b0 0x7f7d240c9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:29.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.006+0000 7f7d293ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d24159200 0x7f7d240c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:29.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.006+0000 7f7d293ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d240afde0 con 0x7f7d240b07b0 2024-01-11T08:25:29.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.006+0000 7f7d293ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d241697d0 con 0x7f7d240b03e0 2024-01-11T08:25:29.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.007+0000 7f7d293ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d24169ab0 con 0x7f7d24159200 2024-01-11T08:25:29.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.007+0000 7f7d227fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d240b07b0 0x7f7d240c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:29.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.007+0000 7f7d227fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d240b07b0 0x7f7d240c9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56978/0 (socket says 172.21.15.27:56978) 2024-01-11T08:25:29.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.007+0000 7f7d227fc700 1 -- 172.21.15.27:0/256398070 learned_addr learned my addr 172.21.15.27:0/256398070 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:29.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.007+0000 7f7d22ffd700 1 --2- 172.21.15.27:0/256398070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d240b03e0 0x7f7d240af8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:29.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.007+0000 7f7d237fe700 1 --2- 172.21.15.27:0/256398070 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d24159200 0x7f7d240c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:29.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.007+0000 7f7d227fc700 1 -- 172.21.15.27:0/256398070 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d24159200 msgr2=0x7f7d240c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:29.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.007+0000 7f7d227fc700 1 --2- 172.21.15.27:0/256398070 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d24159200 0x7f7d240c9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.007+0000 7f7d227fc700 1 -- 172.21.15.27:0/256398070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d240b03e0 msgr2=0x7f7d240af8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:29.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.007+0000 7f7d227fc700 1 --2- 172.21.15.27:0/256398070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d240b03e0 0x7f7d240af8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.007+0000 7f7d227fc700 1 -- 172.21.15.27:0/256398070 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1c009ae0 con 0x7f7d240b07b0 2024-01-11T08:25:29.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.008+0000 7f7d227fc700 1 --2- 172.21.15.27:0/256398070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d240b07b0 0x7f7d240c9710 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800e8d0 tx=0x7f7d18008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:29.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.008+0000 7f7d0affd700 1 -- 172.21.15.27:0/256398070 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d18013410 con 0x7f7d240b07b0 2024-01-11T08:25:29.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.008+0000 7f7d0affd700 1 -- 172.21.15.27:0/256398070 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d18013570 con 0x7f7d240b07b0 2024-01-11T08:25:29.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.008+0000 7f7d293ff700 1 -- 172.21.15.27:0/256398070 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d24169d90 con 0x7f7d240b07b0 2024-01-11T08:25:29.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.008+0000 7f7d0affd700 1 -- 172.21.15.27:0/256398070 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d18013910 con 0x7f7d240b07b0 2024-01-11T08:25:29.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.008+0000 7f7d293ff700 1 -- 172.21.15.27:0/256398070 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d240c52c0 con 0x7f7d240b07b0 2024-01-11T08:25:29.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.009+0000 7f7d293ff700 1 -- 172.21.15.27:0/256398070 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d2416a3e0 con 0x7f7d240b07b0 2024-01-11T08:25:29.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.010+0000 7f7d0affd700 1 -- 172.21.15.27:0/256398070 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7d18013d00 con 0x7f7d240b07b0 2024-01-11T08:25:29.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.011+0000 7f7d0affd700 1 --2- 172.21.15.27:0/256398070 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7d10066f30 0x7f7d100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:29.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.011+0000 7f7d0affd700 1 -- 172.21.15.27:0/256398070 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f7d18084290 con 0x7f7d240b07b0 2024-01-11T08:25:29.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.011+0000 7f7d22ffd700 1 --2- 172.21.15.27:0/256398070 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7d10066f30 0x7f7d100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:29.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.012+0000 7f7d22ffd700 1 --2- 172.21.15.27:0/256398070 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7d10066f30 0x7f7d100693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7d240caa20 tx=0x7f7d140091a0 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.015+0000 7f7d0affd700 1 -- 172.21.15.27:0/256398070 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7d180514e0 con 0x7f7d240b07b0 2024-01-11T08:25:29.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.294+0000 7f7d293ff700 1 -- 172.21.15.27:0/256398070 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f7d2416a590 con 0x7f7d240b07b0 2024-01-11T08:25:29.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.298+0000 7f7d0affd700 1 -- 172.21.15.27:0/256398070 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+0 (secure 0 0 0) 0x7f7d2416a590 con 0x7f7d240b07b0 2024-01-11T08:25:29.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.301+0000 7f7d293ff700 1 -- 172.21.15.27:0/256398070 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7d10066f30 msgr2=0x7f7d100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:29.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.301+0000 7f7d293ff700 1 --2- 172.21.15.27:0/256398070 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7d10066f30 0x7f7d100693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7d240caa20 tx=0x7f7d140091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.301+0000 7f7d293ff700 1 -- 172.21.15.27:0/256398070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d240b07b0 msgr2=0x7f7d240c9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:29.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.301+0000 7f7d293ff700 1 --2- 172.21.15.27:0/256398070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d240b07b0 0x7f7d240c9710 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800e8d0 tx=0x7f7d18008040 comp rx=0 tx=0).stop 2024-01-11T08:25:29.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.301+0000 7f7d293ff700 1 -- 172.21.15.27:0/256398070 shutdown_connections 2024-01-11T08:25:29.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.301+0000 7f7d293ff700 1 --2- 172.21.15.27:0/256398070 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7d10066f30 0x7f7d100693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.301+0000 7f7d293ff700 1 --2- 172.21.15.27:0/256398070 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d24159200 0x7f7d240c9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.301+0000 7f7d293ff700 1 --2- 172.21.15.27:0/256398070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d240b03e0 0x7f7d240af8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.301+0000 7f7d293ff700 1 --2- 172.21.15.27:0/256398070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d240b07b0 0x7f7d240c9710 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.301+0000 7f7d293ff700 1 -- 172.21.15.27:0/256398070 >> 172.21.15.27:0/256398070 conn(0x7f7d2406ed40 msgr2=0x7f7d24158a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:29.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.301+0000 7f7d293ff700 1 -- 172.21.15.27:0/256398070 shutdown_connections 2024-01-11T08:25:29.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.301+0000 7f7d293ff700 1 -- 172.21.15.27:0/256398070 wait complete. 2024-01-11T08:25:29.303 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-11T08:25:29.319 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:25:29.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.517+0000 7f486ddff700 1 Processor -- start 2024-01-11T08:25:29.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.518+0000 7f486ddff700 1 -- start start 2024-01-11T08:25:29.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.519+0000 7f486ddff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f48680b19d0 0x7f4868158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:29.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.519+0000 7f486ddff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f486806b610 0x7f4868159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:29.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.519+0000 7f486ddff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4868158b60 con 0x7f48681575e0 2024-01-11T08:25:29.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.519+0000 7f486ddff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48680cb5e0 con 0x7f486806b610 2024-01-11T08:25:29.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.519+0000 7f486ddff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48681599b0 con 0x7f48680b19d0 2024-01-11T08:25:29.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.519+0000 7f4867fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f486806b610 0x7f4868159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:29.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.519+0000 7f48677fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f48681575e0 0x7f48680b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46380/0 (socket says 172.21.15.27:46380) 2024-01-11T08:25:29.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.519+0000 7f48677fe700 1 -- 172.21.15.27:0/3790233013 learned_addr learned my addr 172.21.15.27:0/3790233013 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:29.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.519+0000 7f4867fff700 1 --2- 172.21.15.27:0/3790233013 >> v2:172.21.15.27:3300/0 conn(0x7f486806b610 0x7f4868159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56986/0 (socket says 172.21.15.27:56986) 2024-01-11T08:25:29.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.519+0000 7f4866ffd700 1 --2- 172.21.15.27:0/3790233013 >> v2:172.21.15.77:3300/0 conn(0x7f48680b19d0 0x7f4868158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:29.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.520+0000 7f4867fff700 1 -- 172.21.15.27:0/3790233013 >> v2:172.21.15.77:3300/0 conn(0x7f48680b19d0 msgr2=0x7f4868158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:29.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.520+0000 7f4867fff700 1 --2- 172.21.15.27:0/3790233013 >> v2:172.21.15.77:3300/0 conn(0x7f48680b19d0 0x7f4868158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.520+0000 7f4867fff700 1 -- 172.21.15.27:0/3790233013 >> v1:172.21.15.27:6789/0 conn(0x7f48681575e0 legacy=0x7f48680b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:29.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.520+0000 7f4867fff700 1 -- 172.21.15.27:0/3790233013 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48680c0f90 con 0x7f486806b610 2024-01-11T08:25:29.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.520+0000 7f4866ffd700 1 --2- 172.21.15.27:0/3790233013 >> v2:172.21.15.77:3300/0 conn(0x7f48680b19d0 0x7f4868158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:29.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.520+0000 7f48667fc700 1 -- 172.21.15.27:0/3790233013 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371074762 0 0) 0x7f4858001110 con 0x7f48681575e0 2024-01-11T08:25:29.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.520+0000 7f4867fff700 1 --2- 172.21.15.27:0/3790233013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f486806b610 0x7f4868159e40 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f4868064fe0 tx=0x7f4860009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82a2fd532af92d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:29.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.520+0000 7f48667fc700 1 -- 172.21.15.27:0/3790233013 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486000e070 con 0x7f486806b610 2024-01-11T08:25:29.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.520+0000 7f48667fc700 1 -- 172.21.15.27:0/3790233013 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486000bc30 con 0x7f486806b610 2024-01-11T08:25:29.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.520+0000 7f48667fc700 1 -- 172.21.15.27:0/3790233013 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860012430 con 0x7f486806b610 2024-01-11T08:25:29.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.521+0000 7f486ddff700 1 -- 172.21.15.27:0/3790233013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f486806b610 msgr2=0x7f4868159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:29.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.521+0000 7f486ddff700 1 --2- 172.21.15.27:0/3790233013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f486806b610 0x7f4868159e40 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f4868064fe0 tx=0x7f4860009780 comp rx=0 tx=0).stop 2024-01-11T08:25:29.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.521+0000 7f486ddff700 1 -- 172.21.15.27:0/3790233013 shutdown_connections 2024-01-11T08:25:29.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.521+0000 7f486ddff700 1 --2- 172.21.15.27:0/3790233013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f486806b610 0x7f4868159e40 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.521+0000 7f486ddff700 1 --2- 172.21.15.27:0/3790233013 >> v2:172.21.15.77:3300/0 conn(0x7f48680b19d0 0x7f4868158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.521+0000 7f486ddff700 1 -- 172.21.15.27:0/3790233013 >> 172.21.15.27:0/3790233013 conn(0x7f48680680e0 msgr2=0x7f4868069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:29.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.521+0000 7f486ddff700 1 -- 172.21.15.27:0/3790233013 shutdown_connections 2024-01-11T08:25:29.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.521+0000 7f486ddff700 1 -- 172.21.15.27:0/3790233013 wait complete. 2024-01-11T08:25:29.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.522+0000 7f486ddff700 1 Processor -- start 2024-01-11T08:25:29.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.523+0000 7f486ddff700 1 -- start start 2024-01-11T08:25:29.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.523+0000 7f486ddff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f486806b610 0x7f4868153e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:29.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.523+0000 7f486ddff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48680b19d0 0x7f486816a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:29.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.523+0000 7f486ddff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48681575e0 0x7f486815e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:29.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.523+0000 7f486ddff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48680c6c10 con 0x7f48680b19d0 2024-01-11T08:25:29.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.523+0000 7f486ddff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4868169600 con 0x7f48681575e0 2024-01-11T08:25:29.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.523+0000 7f486ddff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4868154340 con 0x7f486806b610 2024-01-11T08:25:29.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.524+0000 7f4866ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48680b19d0 0x7f486816a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:29.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.524+0000 7f4866ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48680b19d0 0x7f486816a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56996/0 (socket says 172.21.15.27:56996) 2024-01-11T08:25:29.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.524+0000 7f4866ffd700 1 -- 172.21.15.27:0/3110597064 learned_addr learned my addr 172.21.15.27:0/3110597064 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:29.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.524+0000 7f48677fe700 1 --2- 172.21.15.27:0/3110597064 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f486806b610 0x7f4868153e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:29.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.524+0000 7f4867fff700 1 --2- 172.21.15.27:0/3110597064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48681575e0 0x7f486815e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:29.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.524+0000 7f4866ffd700 1 -- 172.21.15.27:0/3110597064 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f486806b610 msgr2=0x7f4868153e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:29.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.524+0000 7f4866ffd700 1 --2- 172.21.15.27:0/3110597064 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f486806b610 0x7f4868153e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.524+0000 7f4866ffd700 1 -- 172.21.15.27:0/3110597064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48681575e0 msgr2=0x7f486815e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:29.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.524+0000 7f4866ffd700 1 --2- 172.21.15.27:0/3110597064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48681575e0 0x7f486815e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.524+0000 7f4866ffd700 1 -- 172.21.15.27:0/3110597064 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4860009a20 con 0x7f48680b19d0 2024-01-11T08:25:29.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.524+0000 7f4867fff700 1 --2- 172.21.15.27:0/3110597064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48681575e0 0x7f486815e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:25:29.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.524+0000 7f48677fe700 1 --2- 172.21.15.27:0/3110597064 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f486806b610 0x7f4868153e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:25:29.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.524+0000 7f4866ffd700 1 --2- 172.21.15.27:0/3110597064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48680b19d0 0x7f486816a7d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f485c00e8d0 tx=0x7f485c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:29.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.525+0000 7f484f7fe700 1 -- 172.21.15.27:0/3110597064 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c00ee70 con 0x7f48680b19d0 2024-01-11T08:25:29.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.525+0000 7f484f7fe700 1 -- 172.21.15.27:0/3110597064 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f485c00c5e0 con 0x7f48680b19d0 2024-01-11T08:25:29.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.525+0000 7f486ddff700 1 -- 172.21.15.27:0/3110597064 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48680c6090 con 0x7f48680b19d0 2024-01-11T08:25:29.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.525+0000 7f484f7fe700 1 -- 172.21.15.27:0/3110597064 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c00c980 con 0x7f48680b19d0 2024-01-11T08:25:29.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.525+0000 7f486ddff700 1 -- 172.21.15.27:0/3110597064 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48680c6650 con 0x7f48680b19d0 2024-01-11T08:25:29.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.526+0000 7f486ddff700 1 -- 172.21.15.27:0/3110597064 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4868158840 con 0x7f48680b19d0 2024-01-11T08:25:29.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.527+0000 7f484f7fe700 1 -- 172.21.15.27:0/3110597064 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f485c013410 con 0x7f48680b19d0 2024-01-11T08:25:29.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.528+0000 7f484f7fe700 1 --2- 172.21.15.27:0/3110597064 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4854066f30 0x7f48540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:29.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.528+0000 7f484f7fe700 1 -- 172.21.15.27:0/3110597064 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f485c084400 con 0x7f48680b19d0 2024-01-11T08:25:29.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.528+0000 7f48677fe700 1 --2- 172.21.15.27:0/3110597064 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4854066f30 0x7f48540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:29.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.529+0000 7f48677fe700 1 --2- 172.21.15.27:0/3110597064 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4854066f30 0x7f48540693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4858001810 tx=0x7f4858009040 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:29.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.532+0000 7f484f7fe700 1 -- 172.21.15.27:0/3110597064 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f485c0515b0 con 0x7f48680b19d0 2024-01-11T08:25:29.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.780+0000 7f486ddff700 1 -- 172.21.15.27:0/3110597064 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48680718d0 con 0x7f48680b19d0 2024-01-11T08:25:29.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.781+0000 7f484f7fe700 1 -- 172.21.15.27:0/3110597064 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4597 (secure 0 0 0) 0x7f485c054bd0 con 0x7f48680b19d0 2024-01-11T08:25:29.781 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:29.781 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5606,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2845124438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2845124438},{"type":"v1","addr":"172.21.15.77:6837","nonce":2845124438}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5618,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3958104902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3958104902},{"type":"v1","addr":"172.21.15.27:6835","nonce":3958104902}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":6091,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/2612413022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2612413022},{"type":"v1","addr":"172.21.15.27:6833","nonce":2612413022}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:25:23.345532+0000","modified":"2024-01-11T08:25:26.867139+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6174},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6174":{"gid":6174,"name":"d","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6836/3056318454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3056318454},{"type":"v1","addr":"172.21.15.77:6836","nonce":3056318454}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2024-01-11T08:25:29.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.783+0000 7f486ddff700 1 -- 172.21.15.27:0/3110597064 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4854066f30 msgr2=0x7f48540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:29.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.783+0000 7f486ddff700 1 --2- 172.21.15.27:0/3110597064 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4854066f30 0x7f48540693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4858001810 tx=0x7f4858009040 comp rx=0 tx=0).stop 2024-01-11T08:25:29.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.783+0000 7f486ddff700 1 -- 172.21.15.27:0/3110597064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48680b19d0 msgr2=0x7f486816a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:29.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.783+0000 7f486ddff700 1 --2- 172.21.15.27:0/3110597064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48680b19d0 0x7f486816a7d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f485c00e8d0 tx=0x7f485c008040 comp rx=0 tx=0).stop 2024-01-11T08:25:29.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.783+0000 7f486ddff700 1 -- 172.21.15.27:0/3110597064 shutdown_connections 2024-01-11T08:25:29.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.783+0000 7f486ddff700 1 --2- 172.21.15.27:0/3110597064 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f4854066f30 0x7f48540693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.783+0000 7f486ddff700 1 --2- 172.21.15.27:0/3110597064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48681575e0 0x7f486815e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.783+0000 7f486ddff700 1 --2- 172.21.15.27:0/3110597064 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f486806b610 0x7f4868153e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.783+0000 7f486ddff700 1 --2- 172.21.15.27:0/3110597064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48680b19d0 0x7f486816a7d0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:29.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.783+0000 7f486ddff700 1 -- 172.21.15.27:0/3110597064 >> 172.21.15.27:0/3110597064 conn(0x7f48680680e0 msgr2=0x7f486806e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:29.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.783+0000 7f486ddff700 1 -- 172.21.15.27:0/3110597064 shutdown_connections 2024-01-11T08:25:29.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:29.783+0000 7f486ddff700 1 -- 172.21.15.27:0/3110597064 wait complete. 2024-01-11T08:25:29.784 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 33 2024-01-11T08:25:30.204 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 33, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-11T08:25:23.345532+0000', 'modified': '2024-01-11T08:25:26.867139+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6174}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6174': {'gid': 6174, 'name': 'd', 'rank': 0, 'incarnation': 30, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6836/3056318454', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 3056318454}, {'type': 'v1', 'addr': '172.21.15.77:6836', 'nonce': 3056318454}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-11T08:25:30.204 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-11T08:25:30.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-01-11T08:25:30.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.407+0000 7f02729ff700 1 Processor -- start 2024-01-11T08:25:30.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.408+0000 7f02729ff700 1 -- start start 2024-01-11T08:25:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.409+0000 7f02729ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f026c15cde0 0x7f026c15a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.409+0000 7f02729ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f026c0b4bd0 con 0x7f026c068cd0 2024-01-11T08:25:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.409+0000 7f02729ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f026c15ac70 con 0x7f026c15ca10 2024-01-11T08:25:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.409+0000 7f02729ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f026c0b3fc0 con 0x7f026c15cde0 2024-01-11T08:25:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.409+0000 7f0270f9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f026c15cde0 0x7f026c15a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.409+0000 7f026b7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f026c068cd0 0x7f026c0b5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46386/0 (socket says 172.21.15.27:46386) 2024-01-11T08:25:30.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.409+0000 7f026b7fe700 1 -- 172.21.15.27:0/4174996831 learned_addr learned my addr 172.21.15.27:0/4174996831 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:30.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.410+0000 7f0270f9c700 1 -- 172.21.15.27:0/4174996831 >> v1:172.21.15.77:6789/0 conn(0x7f026c15ca10 legacy=0x7f026c0685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:30.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.410+0000 7f026affd700 1 -- 172.21.15.27:0/4174996831 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12580761 0 0) 0x7f026c0b4bd0 con 0x7f026c068cd0 2024-01-11T08:25:30.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.410+0000 7f0270f9c700 1 -- 172.21.15.27:0/4174996831 >> v1:172.21.15.27:6789/0 conn(0x7f026c068cd0 legacy=0x7f026c0b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:30.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.410+0000 7f0270f9c700 1 -- 172.21.15.27:0/4174996831 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c0b42a0 con 0x7f026c15cde0 2024-01-11T08:25:30.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.410+0000 7f0270f9c700 1 --2- 172.21.15.27:0/4174996831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f026c15cde0 0x7f026c15a730 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f0264009e90 tx=0x7f0264009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24609369ef5a8341 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:30.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.410+0000 7f026affd700 1 -- 172.21.15.27:0/4174996831 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026400e070 con 0x7f026c15cde0 2024-01-11T08:25:30.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.410+0000 7f026affd700 1 -- 172.21.15.27:0/4174996831 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026400bbf0 con 0x7f026c15cde0 2024-01-11T08:25:30.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.410+0000 7f026affd700 1 -- 172.21.15.27:0/4174996831 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0264012430 con 0x7f026c15cde0 2024-01-11T08:25:30.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.411+0000 7f02729ff700 1 -- 172.21.15.27:0/4174996831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f026c15cde0 msgr2=0x7f026c15a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:30.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.411+0000 7f02729ff700 1 --2- 172.21.15.27:0/4174996831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f026c15cde0 0x7f026c15a730 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f0264009e90 tx=0x7f0264009780 comp rx=0 tx=0).stop 2024-01-11T08:25:30.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.411+0000 7f02729ff700 1 -- 172.21.15.27:0/4174996831 shutdown_connections 2024-01-11T08:25:30.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.411+0000 7f02729ff700 1 --2- 172.21.15.27:0/4174996831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f026c15cde0 0x7f026c15a730 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.411+0000 7f02729ff700 1 -- 172.21.15.27:0/4174996831 >> 172.21.15.27:0/4174996831 conn(0x7f026c06ed40 msgr2=0x7f026c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:30.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.411+0000 7f02729ff700 1 -- 172.21.15.27:0/4174996831 shutdown_connections 2024-01-11T08:25:30.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.411+0000 7f02729ff700 1 -- 172.21.15.27:0/4174996831 wait complete. 2024-01-11T08:25:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.413+0000 7f02729ff700 1 Processor -- start 2024-01-11T08:25:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.413+0000 7f02729ff700 1 -- start start 2024-01-11T08:25:30.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.414+0000 7f02729ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f026c068cd0 0x7f026c155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.414+0000 7f02729ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f026c15ca10 0x7f026c0ca630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.414+0000 7f02729ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f026c15cde0 0x7f026c0cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.414+0000 7f02729ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c0c6070 con 0x7f026c15cde0 2024-01-11T08:25:30.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.414+0000 7f02729ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c0c6350 con 0x7f026c068cd0 2024-01-11T08:25:30.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.414+0000 7f02729ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026c0c6630 con 0x7f026c15ca10 2024-01-11T08:25:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.414+0000 7f0270f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f026c15cde0 0x7f026c0cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.414+0000 7f0270f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f026c15cde0 0x7f026c0cab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57024/0 (socket says 172.21.15.27:57024) 2024-01-11T08:25:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.414+0000 7f0270f9c700 1 -- 172.21.15.27:0/728725275 learned_addr learned my addr 172.21.15.27:0/728725275 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.414+0000 7f026bfff700 1 --2- 172.21.15.27:0/728725275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f026c068cd0 0x7f026c155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.414+0000 7f026b7fe700 1 --2- 172.21.15.27:0/728725275 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f026c15ca10 0x7f026c0ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.414+0000 7f0270f9c700 1 -- 172.21.15.27:0/728725275 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f026c15ca10 msgr2=0x7f026c0ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.414+0000 7f0270f9c700 1 --2- 172.21.15.27:0/728725275 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f026c15ca10 0x7f026c0ca630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.414+0000 7f0270f9c700 1 -- 172.21.15.27:0/728725275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f026c068cd0 msgr2=0x7f026c155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.414+0000 7f0270f9c700 1 --2- 172.21.15.27:0/728725275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f026c068cd0 0x7f026c155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.414+0000 7f0270f9c700 1 -- 172.21.15.27:0/728725275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0264009a20 con 0x7f026c15cde0 2024-01-11T08:25:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.415+0000 7f0270f9c700 1 --2- 172.21.15.27:0/728725275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f026c15cde0 0x7f026c0cab70 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0264013000 tx=0x7f0264019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.415+0000 7f0253fff700 1 -- 172.21.15.27:0/728725275 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0264012e90 con 0x7f026c15cde0 2024-01-11T08:25:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.415+0000 7f0253fff700 1 -- 172.21.15.27:0/728725275 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026401b630 con 0x7f026c15cde0 2024-01-11T08:25:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.415+0000 7f02729ff700 1 -- 172.21.15.27:0/728725275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f026c0c6910 con 0x7f026c15cde0 2024-01-11T08:25:30.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.415+0000 7f0253fff700 1 -- 172.21.15.27:0/728725275 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026401b9f0 con 0x7f026c15cde0 2024-01-11T08:25:30.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.415+0000 7f02729ff700 1 -- 172.21.15.27:0/728725275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f026c156250 con 0x7f026c15cde0 2024-01-11T08:25:30.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.417+0000 7f0253fff700 1 -- 172.21.15.27:0/728725275 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f026401c400 con 0x7f026c15cde0 2024-01-11T08:25:30.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.418+0000 7f0253fff700 1 --2- 172.21.15.27:0/728725275 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0258066f30 0x7f02580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.418+0000 7f0253fff700 1 -- 172.21.15.27:0/728725275 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f0264086130 con 0x7f026c15cde0 2024-01-11T08:25:30.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.419+0000 7f026bfff700 1 --2- 172.21.15.27:0/728725275 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0258066f30 0x7f02580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.419+0000 7f026bfff700 1 --2- 172.21.15.27:0/728725275 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0258066f30 0x7f02580693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f025c003a80 tx=0x7f025c007ec0 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:30.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.420+0000 7f028105cb80 1 Processor -- start 2024-01-11T08:25:30.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.421+0000 7f028105cb80 1 -- start start 2024-01-11T08:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.422+0000 7f028105cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 0x56529749fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.422+0000 7f028105cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529749a3a0 0x5652974c6860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.422+0000 7f028105cb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5652974beef0 0x5652974c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.422+0000 7f028105cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652973b7310 con 0x5652974beb20 2024-01-11T08:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.422+0000 7f028105cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652973f4130 con 0x56529749a3a0 2024-01-11T08:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.422+0000 7f028105cb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x5652973f1af0 con 0x5652974beef0 2024-01-11T08:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.422+0000 7f026bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 0x56529749fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.422+0000 7f026bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 0x56529749fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57038/0 (socket says 172.21.15.27:57038) 2024-01-11T08:25:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.422+0000 7f026bfff700 1 -- 172.21.15.27:0/463305222 learned_addr learned my addr 172.21.15.27:0/463305222 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.422+0000 7f026b7fe700 1 --2- 172.21.15.27:0/463305222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529749a3a0 0x5652974c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.422+0000 7f0270f9c700 1 --2- 172.21.15.27:0/463305222 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5652974beef0 0x5652974c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.422+0000 7f026bfff700 1 -- 172.21.15.27:0/463305222 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5652974beef0 msgr2=0x5652974c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.422+0000 7f026bfff700 1 --2- 172.21.15.27:0/463305222 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5652974beef0 0x5652974c6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.422+0000 7f026bfff700 1 -- 172.21.15.27:0/463305222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529749a3a0 msgr2=0x5652974c6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.422+0000 7f026bfff700 1 --2- 172.21.15.27:0/463305222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529749a3a0 0x5652974c6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.422+0000 7f026bfff700 1 -- 172.21.15.27:0/463305222 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652972e00a0 con 0x5652974beb20 2024-01-11T08:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.422+0000 7f026b7fe700 1 --2- 172.21.15.27:0/463305222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529749a3a0 0x5652974c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.422+0000 7f0270f9c700 1 --2- 172.21.15.27:0/463305222 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5652974beef0 0x5652974c6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.423+0000 7f026bfff700 1 --2- 172.21.15.27:0/463305222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 0x56529749fdf0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f025c002730 tx=0x7f025c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c70348aa353b699 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.423+0000 7f02517fa700 1 -- 172.21.15.27:0/463305222 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c014450 con 0x5652974beb20 2024-01-11T08:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.423+0000 7f02517fa700 1 -- 172.21.15.27:0/463305222 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025c0145b0 con 0x5652974beb20 2024-01-11T08:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.423+0000 7f02517fa700 1 -- 172.21.15.27:0/463305222 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c014930 con 0x5652974beb20 2024-01-11T08:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.423+0000 7f028105cb80 1 -- 172.21.15.27:0/463305222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 msgr2=0x56529749fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.424+0000 7f028105cb80 1 --2- 172.21.15.27:0/463305222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 0x56529749fdf0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f025c002730 tx=0x7f025c015000 comp rx=0 tx=0).stop 2024-01-11T08:25:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.424+0000 7f028105cb80 1 -- 172.21.15.27:0/463305222 shutdown_connections 2024-01-11T08:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.424+0000 7f028105cb80 1 --2- 172.21.15.27:0/463305222 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5652974beef0 0x5652974c6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.424+0000 7f028105cb80 1 --2- 172.21.15.27:0/463305222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 0x56529749fdf0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.424+0000 7f028105cb80 1 --2- 172.21.15.27:0/463305222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529749a3a0 0x5652974c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.424+0000 7f028105cb80 1 -- 172.21.15.27:0/463305222 >> 172.21.15.27:0/463305222 conn(0x56529749c300 msgr2=0x56529749ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.424+0000 7f028105cb80 1 -- 172.21.15.27:0/463305222 shutdown_connections 2024-01-11T08:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.424+0000 7f028105cb80 1 -- 172.21.15.27:0/463305222 wait complete. 2024-01-11T08:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.424+0000 7f028105cb80 1 -- start start 2024-01-11T08:25:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.425+0000 7f028105cb80 1 Processor -- start 2024-01-11T08:25:30.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.426+0000 7f028105cb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529749a3a0 0x5652974e8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.426+0000 7f028105cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 0x5652974bde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.426+0000 7f028105cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652974beef0 0x5652974be370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.426+0000 7f028105cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652973ed8e0 con 0x5652974beb20 2024-01-11T08:25:30.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.426+0000 7f028105cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x565297399980 con 0x5652974beef0 2024-01-11T08:25:30.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.426+0000 7f028105cb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x565297399620 con 0x56529749a3a0 2024-01-11T08:25:30.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.426+0000 7f028105cb80 10 client.0 _ll_register_callbacks cb 0x5652971fca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T08:25:30.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.426+0000 7f026b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 0x5652974bde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.426+0000 7f026b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 0x5652974bde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57050/0 (socket says 172.21.15.27:57050) 2024-01-11T08:25:30.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.426+0000 7f026b7fe700 1 -- 172.21.15.27:0/2443916484 learned_addr learned my addr 172.21.15.27:0/2443916484 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:30.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.426+0000 7f026bfff700 1 --2- 172.21.15.27:0/2443916484 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529749a3a0 0x5652974e8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.426+0000 7f0270f9c700 1 --2- 172.21.15.27:0/2443916484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652974beef0 0x5652974be370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.426+0000 7f026b7fe700 1 -- 172.21.15.27:0/2443916484 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529749a3a0 msgr2=0x5652974e8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.427+0000 7f026b7fe700 1 --2- 172.21.15.27:0/2443916484 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529749a3a0 0x5652974e8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.427+0000 7f026b7fe700 1 -- 172.21.15.27:0/2443916484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652974beef0 msgr2=0x5652974be370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.427+0000 7f026b7fe700 1 --2- 172.21.15.27:0/2443916484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652974beef0 0x5652974be370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.427+0000 7f026b7fe700 1 -- 172.21.15.27:0/2443916484 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f026c15ac70 con 0x5652974beb20 2024-01-11T08:25:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.427+0000 7f026b7fe700 1 --2- 172.21.15.27:0/2443916484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 0x5652974bde30 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f026c0690d0 tx=0x7f0260009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1e80d9fcc29493 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.427+0000 7f024a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T08:25:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.427+0000 7f024a7fc700 1 -- 172.21.15.27:0/2443916484 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0260006b60 con 0x5652974beb20 2024-01-11T08:25:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.427+0000 7f024a7fc700 1 -- 172.21.15.27:0/2443916484 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(75..75 src has 1..75) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f0260001c90 con 0x5652974beb20 2024-01-11T08:25:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.427+0000 7f028105cb80 1 -- 172.21.15.27:0/2443916484 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565297492560 con 0x5652974beb20 2024-01-11T08:25:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.427+0000 7f024a7fc700 1 -- 172.21.15.27:0/2443916484 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0260006120 con 0x5652974beb20 2024-01-11T08:25:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.427+0000 7f028105cb80 10 client.6324 fetch_fsmap learned FSMap version 33 2024-01-11T08:25:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.427+0000 7f028105cb80 1 -- 172.21.15.27:0/2443916484 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x5652973bc3c0 con 0x5652974beb20 2024-01-11T08:25:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.427+0000 7f024a7fc700 1 -- 172.21.15.27:0/2443916484 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0260024020 con 0x5652974beb20 2024-01-11T08:25:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.428+0000 7f024a7fc700 1 -- 172.21.15.27:0/2443916484 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 33) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f0260006280 con 0x5652974beb20 2024-01-11T08:25:30.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.428+0000 7f028105cb80 10 client.6324 fetch_fsmap finished waiting for FSMap version 33 2024-01-11T08:25:30.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.428+0000 7f028105cb80 10 client.6324 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-11T08:25:30.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.428+0000 7f028105cb80 20 client.6324 populate_metadata read hostname 'smithi027' 2024-01-11T08:25:30.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.428+0000 7f028105cb80 1 --2- 172.21.15.27:0/2443916484 >> [v2:172.21.15.77:6834/3056318454,v1:172.21.15.77:6836/3056318454] conn(0x5652974bf8e0 0x5652974e6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.428+0000 7f028105cb80 4 client.6324 mds_command: new command op to 6174 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-11T08:25:30.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.428+0000 7f028105cb80 1 -- 172.21.15.27:0/2443916484 --> [v2:172.21.15.77:6834/3056318454,v1:172.21.15.77:6836/3056318454] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56529746f0e0 con 0x5652974bf8e0 2024-01-11T08:25:30.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.428+0000 7f026bfff700 1 --2- 172.21.15.27:0/2443916484 >> [v2:172.21.15.77:6834/3056318454,v1:172.21.15.77:6836/3056318454] conn(0x5652974bf8e0 0x5652974e6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.429+0000 7f026bfff700 1 --2- 172.21.15.27:0/2443916484 >> [v2:172.21.15.77:6834/3056318454,v1:172.21.15.77:6836/3056318454] conn(0x5652974bf8e0 0x5652974e6f60 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=480ee427f30fbc13 server_cookie=d2c7eb95bb5b440f in_seq=0 out_seq=0 2024-01-11T08:25:30.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.429+0000 7f024a7fc700 10 client.6324 ms_handle_connect on v2:172.21.15.77:6834/3056318454 2024-01-11T08:25:30.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.429+0000 7f024a7fc700 1 -- 172.21.15.27:0/2443916484 <== mds.0 v2:172.21.15.77:6834/3056318454 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x56529746f0e0 con 0x5652974bf8e0 2024-01-11T08:25:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.430+0000 7f024a7fc700 10 client.6324 handle_command_reply: tid=0 2024-01-11T08:25:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.430+0000 7f028105cb80 1 client.6324 shutdown 2024-01-11T08:25:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.430+0000 7f028105cb80 1 -- 172.21.15.27:0/2443916484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 msgr2=0x5652974bde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.430+0000 7f028105cb80 1 --2- 172.21.15.27:0/2443916484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 0x5652974bde30 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f026c0690d0 tx=0x7f0260009040 comp rx=0 tx=0).stop 2024-01-11T08:25:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.430+0000 7f028105cb80 1 -- 172.21.15.27:0/2443916484 shutdown_connections 2024-01-11T08:25:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.430+0000 7f028105cb80 1 --2- 172.21.15.27:0/2443916484 >> [v2:172.21.15.77:6834/3056318454,v1:172.21.15.77:6836/3056318454] conn(0x5652974bf8e0 0x5652974e6f60 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.430+0000 7f028105cb80 1 --2- 172.21.15.27:0/2443916484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652974beef0 0x5652974be370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.430+0000 7f028105cb80 1 --2- 172.21.15.27:0/2443916484 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529749a3a0 0x5652974e8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.430+0000 7f028105cb80 1 --2- 172.21.15.27:0/2443916484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 0x5652974bde30 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.430+0000 7f024a7fc700 0 client.6324 ms_handle_reset on v2:172.21.15.77:6834/3056318454 2024-01-11T08:25:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.430+0000 7f028105cb80 1 -- 172.21.15.27:0/2443916484 >> 172.21.15.27:0/2443916484 conn(0x56529749c300 msgr2=0x5652974cd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.430+0000 7f028105cb80 1 -- 172.21.15.27:0/2443916484 shutdown_connections 2024-01-11T08:25:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.430+0000 7f028105cb80 1 -- 172.21.15.27:0/2443916484 wait complete. 2024-01-11T08:25:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.430+0000 7f028105cb80 20 client.6324 trim_cache size 0 max 16384 2024-01-11T08:25:30.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.448+0000 7f028105cb80 1 Processor -- start 2024-01-11T08:25:30.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.448+0000 7f028105cb80 1 -- start start 2024-01-11T08:25:30.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.449+0000 7f028105cb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529749a3a0 0x56529753f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.449+0000 7f028105cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 0x5652974c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.449+0000 7f028105cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652974beef0 0x5652974c5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.449+0000 7f028105cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652972851e0 con 0x5652974beb20 2024-01-11T08:25:30.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.449+0000 7f028105cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652974cdad0 con 0x5652974beef0 2024-01-11T08:25:30.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.449+0000 7f028105cb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x5652974ccc30 con 0x56529749a3a0 2024-01-11T08:25:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.449+0000 7f026b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 0x5652974c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.449+0000 7f026b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 0x5652974c5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57052/0 (socket says 172.21.15.27:57052) 2024-01-11T08:25:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.449+0000 7f026b7fe700 1 -- 172.21.15.27:0/2145151960 learned_addr learned my addr 172.21.15.27:0/2145151960 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.449+0000 7f0270f9c700 1 --2- 172.21.15.27:0/2145151960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652974beef0 0x5652974c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.449+0000 7f026bfff700 1 --2- 172.21.15.27:0/2145151960 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529749a3a0 0x56529753f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.449+0000 7f026b7fe700 1 -- 172.21.15.27:0/2145151960 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529749a3a0 msgr2=0x56529753f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.449+0000 7f026b7fe700 1 --2- 172.21.15.27:0/2145151960 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529749a3a0 0x56529753f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.449+0000 7f026b7fe700 1 -- 172.21.15.27:0/2145151960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652974beef0 msgr2=0x5652974c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.449+0000 7f026b7fe700 1 --2- 172.21.15.27:0/2145151960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652974beef0 0x5652974c5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.450+0000 7f026b7fe700 1 -- 172.21.15.27:0/2145151960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652972541d0 con 0x5652974beb20 2024-01-11T08:25:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.450+0000 7f026bfff700 1 --2- 172.21.15.27:0/2145151960 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529749a3a0 0x56529753f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:25:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.450+0000 7f026b7fe700 1 --2- 172.21.15.27:0/2145151960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 0x5652974c5220 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0260000fb0 tx=0x7f0260019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe8c304f515a97b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.450+0000 7f0251ffb700 1 -- 172.21.15.27:0/2145151960 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0260006ad0 con 0x5652974beb20 2024-01-11T08:25:30.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.450+0000 7f0251ffb700 1 -- 172.21.15.27:0/2145151960 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0260024040 con 0x5652974beb20 2024-01-11T08:25:30.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.450+0000 7f0251ffb700 1 -- 172.21.15.27:0/2145151960 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0260000bc0 con 0x5652974beb20 2024-01-11T08:25:30.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.451+0000 7f028105cb80 1 -- 172.21.15.27:0/2145151960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 msgr2=0x5652974c5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:30.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.451+0000 7f028105cb80 1 --2- 172.21.15.27:0/2145151960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 0x5652974c5220 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0260000fb0 tx=0x7f0260019000 comp rx=0 tx=0).stop 2024-01-11T08:25:30.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.451+0000 7f028105cb80 1 -- 172.21.15.27:0/2145151960 shutdown_connections 2024-01-11T08:25:30.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.451+0000 7f028105cb80 1 --2- 172.21.15.27:0/2145151960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652974beef0 0x5652974c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.451+0000 7f028105cb80 1 --2- 172.21.15.27:0/2145151960 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529749a3a0 0x56529753f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.451+0000 7f028105cb80 1 --2- 172.21.15.27:0/2145151960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652974beb20 0x5652974c5220 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.451+0000 7f028105cb80 1 -- 172.21.15.27:0/2145151960 >> 172.21.15.27:0/2145151960 conn(0x56529749c300 msgr2=0x5652974c9150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:30.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.451+0000 7f028105cb80 1 -- 172.21.15.27:0/2145151960 shutdown_connections 2024-01-11T08:25:30.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.451+0000 7f028105cb80 1 -- 172.21.15.27:0/2145151960 wait complete. 2024-01-11T08:25:30.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.451+0000 7f028105cb80 1 -- start start 2024-01-11T08:25:30.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.452+0000 7f028105cb80 1 Processor -- start 2024-01-11T08:25:30.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.453+0000 7f028105cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529749a3a0 0x56529753fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.453+0000 7f028105cb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5652974beb20 0x5652974c5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.453+0000 7f028105cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652974beef0 0x5652974e4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.453+0000 7f028105cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652974e6b50 con 0x56529749a3a0 2024-01-11T08:25:30.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.453+0000 7f028105cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x565297540670 con 0x5652974beef0 2024-01-11T08:25:30.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.453+0000 7f028105cb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x5652974e6cc0 con 0x5652974beb20 2024-01-11T08:25:30.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.453+0000 7f028105cb80 10 client.0 _ll_register_callbacks cb 0x5652971fca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T08:25:30.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.453+0000 7f026bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529749a3a0 0x56529753fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.453+0000 7f026bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529749a3a0 0x56529753fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57062/0 (socket says 172.21.15.27:57062) 2024-01-11T08:25:30.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.453+0000 7f026bfff700 1 -- 172.21.15.27:0/820116252 learned_addr learned my addr 172.21.15.27:0/820116252 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:30.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.453+0000 7f026b7fe700 1 --2- 172.21.15.27:0/820116252 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5652974beb20 0x5652974c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.453+0000 7f0270f9c700 1 --2- 172.21.15.27:0/820116252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652974beef0 0x5652974e4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.453+0000 7f026bfff700 1 -- 172.21.15.27:0/820116252 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5652974beb20 msgr2=0x5652974c5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:30.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.453+0000 7f026bfff700 1 --2- 172.21.15.27:0/820116252 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5652974beb20 0x5652974c5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.453+0000 7f026bfff700 1 -- 172.21.15.27:0/820116252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652974beef0 msgr2=0x5652974e4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:30.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.453+0000 7f026bfff700 1 --2- 172.21.15.27:0/820116252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652974beef0 0x5652974e4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.453+0000 7f026bfff700 1 -- 172.21.15.27:0/820116252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652973bc3c0 con 0x56529749a3a0 2024-01-11T08:25:30.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.453+0000 7f0270f9c700 1 --2- 172.21.15.27:0/820116252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652974beef0 0x5652974e4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:30.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.454+0000 7f026bfff700 1 --2- 172.21.15.27:0/820116252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529749a3a0 0x56529753fc60 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f025c020000 tx=0x7f025c032250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b48cd3bc4117565 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:30.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.454+0000 7f024a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T08:25:30.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.454+0000 7f024a7fc700 1 -- 172.21.15.27:0/820116252 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c014930 con 0x56529749a3a0 2024-01-11T08:25:30.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.454+0000 7f024a7fc700 1 -- 172.21.15.27:0/820116252 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(75..75 src has 1..75) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f025c01c040 con 0x56529749a3a0 2024-01-11T08:25:30.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.454+0000 7f028105cb80 1 -- 172.21.15.27:0/820116252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f025c0145b0 con 0x56529749a3a0 2024-01-11T08:25:30.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.454+0000 7f024a7fc700 1 -- 172.21.15.27:0/820116252 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f025c020620 con 0x56529749a3a0 2024-01-11T08:25:30.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.454+0000 7f028105cb80 10 client.6330 fetch_fsmap learned FSMap version 33 2024-01-11T08:25:30.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.454+0000 7f028105cb80 1 -- 172.21.15.27:0/820116252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x56529722dd60 con 0x56529749a3a0 2024-01-11T08:25:30.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.454+0000 7f024a7fc700 1 -- 172.21.15.27:0/820116252 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025c0145b0 con 0x56529749a3a0 2024-01-11T08:25:30.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.455+0000 7f024a7fc700 1 -- 172.21.15.27:0/820116252 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 33) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f025c002050 con 0x56529749a3a0 2024-01-11T08:25:30.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.455+0000 7f028105cb80 10 client.6330 fetch_fsmap finished waiting for FSMap version 33 2024-01-11T08:25:30.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.455+0000 7f028105cb80 10 client.6330 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-11T08:25:30.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.455+0000 7f028105cb80 20 client.6330 populate_metadata read hostname 'smithi027' 2024-01-11T08:25:30.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.455+0000 7f028105cb80 1 --2- 172.21.15.27:0/820116252 >> [v2:172.21.15.77:6834/3056318454,v1:172.21.15.77:6836/3056318454] conn(0x5652974bd860 0x5652974bdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.455+0000 7f028105cb80 4 client.6330 mds_command: new command op to 6174 tid=0[{"prefix": "status"}] 2024-01-11T08:25:30.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.455+0000 7f028105cb80 1 -- 172.21.15.27:0/820116252 --> [v2:172.21.15.77:6834/3056318454,v1:172.21.15.77:6836/3056318454] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5652973908d0 con 0x5652974bd860 2024-01-11T08:25:30.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.455+0000 7f026b7fe700 1 --2- 172.21.15.27:0/820116252 >> [v2:172.21.15.77:6834/3056318454,v1:172.21.15.77:6836/3056318454] conn(0x5652974bd860 0x5652974bdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.456+0000 7f024a7fc700 10 client.6330 ms_handle_connect on v2:172.21.15.77:6834/3056318454 2024-01-11T08:25:30.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.456+0000 7f026b7fe700 1 --2- 172.21.15.27:0/820116252 >> [v2:172.21.15.77:6834/3056318454,v1:172.21.15.77:6836/3056318454] conn(0x5652974bd860 0x5652974bdc30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e42038266e328b0 server_cookie=27e885980594122c in_seq=0 out_seq=0 2024-01-11T08:25:30.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.456+0000 7f024a7fc700 1 -- 172.21.15.27:0/820116252 <== mds.0 v2:172.21.15.77:6834/3056318454 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5652973908d0 con 0x5652974bd860 2024-01-11T08:25:30.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.456+0000 7f024a7fc700 10 client.6330 handle_command_reply: tid=0 2024-01-11T08:25:30.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.456+0000 7f028105cb80 1 client.6330 shutdown 2024-01-11T08:25:30.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.456+0000 7f028105cb80 1 -- 172.21.15.27:0/820116252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529749a3a0 msgr2=0x56529753fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:30.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.456+0000 7f028105cb80 1 --2- 172.21.15.27:0/820116252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529749a3a0 0x56529753fc60 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f025c020000 tx=0x7f025c032250 comp rx=0 tx=0).stop 2024-01-11T08:25:30.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.456+0000 7f028105cb80 1 -- 172.21.15.27:0/820116252 shutdown_connections 2024-01-11T08:25:30.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.456+0000 7f028105cb80 1 --2- 172.21.15.27:0/820116252 >> [v2:172.21.15.77:6834/3056318454,v1:172.21.15.77:6836/3056318454] conn(0x5652974bd860 0x5652974bdc30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.456+0000 7f028105cb80 1 --2- 172.21.15.27:0/820116252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5652974beef0 0x5652974e4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.456+0000 7f028105cb80 1 --2- 172.21.15.27:0/820116252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529749a3a0 0x56529753fc60 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.456+0000 7f024a7fc700 0 client.6330 ms_handle_reset on v2:172.21.15.77:6834/3056318454 2024-01-11T08:25:30.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.456+0000 7f028105cb80 1 --2- 172.21.15.27:0/820116252 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5652974beb20 0x5652974c5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.456+0000 7f028105cb80 1 -- 172.21.15.27:0/820116252 >> 172.21.15.27:0/820116252 conn(0x56529749c300 msgr2=0x56529749aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:30.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.457+0000 7f028105cb80 1 -- 172.21.15.27:0/820116252 shutdown_connections 2024-01-11T08:25:30.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.457+0000 7f028105cb80 1 -- 172.21.15.27:0/820116252 wait complete. 2024-01-11T08:25:30.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.457+0000 7f028105cb80 20 client.6330 trim_cache size 0 max 16384 2024-01-11T08:25:30.462 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T08:25:30.462 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "8fddb4a7-9d66-40f1-892d-6fe86c728ee4", 2024-01-11T08:25:30.462 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-11T08:25:30.462 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 6174, 2024-01-11T08:25:30.462 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-11T08:25:30.462 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-11T08:25:30.462 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-11T08:25:30.462 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 6.6188066839999999, 2024-01-11T08:25:30.463 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 33, 2024-01-11T08:25:30.463 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 75, 2024-01-11T08:25:30.463 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 75, 2024-01-11T08:25:30.463 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 12.301725009 2024-01-11T08:25:30.463 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T08:25:30.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.457+0000 7f02729ff700 1 -- 172.21.15.27:0/728725275 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0258066f30 msgr2=0x7f02580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:30.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.457+0000 7f02729ff700 1 --2- 172.21.15.27:0/728725275 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0258066f30 0x7f02580693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f025c003a80 tx=0x7f025c007ec0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.457+0000 7f02729ff700 1 -- 172.21.15.27:0/728725275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f026c15cde0 msgr2=0x7f026c0cab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:30.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.457+0000 7f02729ff700 1 --2- 172.21.15.27:0/728725275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f026c15cde0 0x7f026c0cab70 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0264013000 tx=0x7f0264019000 comp rx=0 tx=0).stop 2024-01-11T08:25:30.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.457+0000 7f02729ff700 1 -- 172.21.15.27:0/728725275 shutdown_connections 2024-01-11T08:25:30.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.457+0000 7f02729ff700 1 --2- 172.21.15.27:0/728725275 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f0258066f30 0x7f02580693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.457+0000 7f02729ff700 1 --2- 172.21.15.27:0/728725275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f026c15cde0 0x7f026c0cab70 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.457+0000 7f02729ff700 1 --2- 172.21.15.27:0/728725275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f026c068cd0 0x7f026c155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.457+0000 7f02729ff700 1 --2- 172.21.15.27:0/728725275 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f026c15ca10 0x7f026c0ca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.457+0000 7f02729ff700 1 -- 172.21.15.27:0/728725275 >> 172.21.15.27:0/728725275 conn(0x7f026c06ed40 msgr2=0x7f026c157be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:30.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.458+0000 7f02729ff700 1 -- 172.21.15.27:0/728725275 shutdown_connections 2024-01-11T08:25:30.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.458+0000 7f02729ff700 1 -- 172.21.15.27:0/728725275 wait complete. 2024-01-11T08:25:30.475 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-11T08:25:30.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.679+0000 7fcfdbfff700 1 Processor -- start 2024-01-11T08:25:30.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.679+0000 7fcfdbfff700 1 -- start start 2024-01-11T08:25:30.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.680+0000 7fcfdbfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcfd40b2b90 0x7fcfd4159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.680+0000 7fcfdbfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcfd40b07b0 0x7fcfd4159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.680+0000 7fcfdbfff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd40cd730 con 0x7fcfd40b03e0 2024-01-11T08:25:30.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.680+0000 7fcfdbfff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd40ccb20 con 0x7fcfd40b2b90 2024-01-11T08:25:30.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.680+0000 7fcfdbfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd40cce00 con 0x7fcfd40b07b0 2024-01-11T08:25:30.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.680+0000 7fcfd959a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcfd40b2b90 0x7fcfd4159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.680+0000 7fcfd9d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcfd40b03e0 0x7fcfd40b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46398/0 (socket says 172.21.15.27:46398) 2024-01-11T08:25:30.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.680+0000 7fcfd9d9b700 1 -- 172.21.15.27:0/3823689253 learned_addr learned my addr 172.21.15.27:0/3823689253 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:30.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.681+0000 7fcfda59c700 1 --2- 172.21.15.27:0/3823689253 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcfd40b07b0 0x7fcfd4159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.681+0000 7fcfd959a700 1 -- 172.21.15.27:0/3823689253 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcfd40b07b0 msgr2=0x7fcfd4159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:30.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.681+0000 7fcfd8d99700 1 -- 172.21.15.27:0/3823689253 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776039863 0 0) 0x7fcfd40cd730 con 0x7fcfd40b03e0 2024-01-11T08:25:30.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.681+0000 7fcfd959a700 1 --2- 172.21.15.27:0/3823689253 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcfd40b07b0 0x7fcfd4159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.681+0000 7fcfd959a700 1 -- 172.21.15.27:0/3823689253 >> v1:172.21.15.27:6789/0 conn(0x7fcfd40b03e0 legacy=0x7fcfd40b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:30.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.681+0000 7fcfd959a700 1 -- 172.21.15.27:0/3823689253 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd40cd0e0 con 0x7fcfd40b2b90 2024-01-11T08:25:30.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.681+0000 7fcfd959a700 1 --2- 172.21.15.27:0/3823689253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd40b2b90 0x7fcfd4159200 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fcfc40098e0 tx=0x7fcfc40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7020a25d1c864124 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:30.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.681+0000 7fcfd8d99700 1 -- 172.21.15.27:0/3823689253 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc400b640 con 0x7fcfd40b2b90 2024-01-11T08:25:30.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.682+0000 7fcfd8d99700 1 -- 172.21.15.27:0/3823689253 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc400b7a0 con 0x7fcfd40b2b90 2024-01-11T08:25:30.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.682+0000 7fcfd8d99700 1 -- 172.21.15.27:0/3823689253 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc400bb40 con 0x7fcfd40b2b90 2024-01-11T08:25:30.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.682+0000 7fcfdbfff700 1 -- 172.21.15.27:0/3823689253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd40b2b90 msgr2=0x7fcfd4159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:30.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.682+0000 7fcfdbfff700 1 --2- 172.21.15.27:0/3823689253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd40b2b90 0x7fcfd4159200 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fcfc40098e0 tx=0x7fcfc40091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.682+0000 7fcfdbfff700 1 -- 172.21.15.27:0/3823689253 shutdown_connections 2024-01-11T08:25:30.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.682+0000 7fcfdbfff700 1 --2- 172.21.15.27:0/3823689253 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcfd40b07b0 0x7fcfd4159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.682+0000 7fcfdbfff700 1 --2- 172.21.15.27:0/3823689253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd40b2b90 0x7fcfd4159200 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.682+0000 7fcfdbfff700 1 -- 172.21.15.27:0/3823689253 >> 172.21.15.27:0/3823689253 conn(0x7fcfd406ed40 msgr2=0x7fcfd406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:30.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.682+0000 7fcfdbfff700 1 -- 172.21.15.27:0/3823689253 shutdown_connections 2024-01-11T08:25:30.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.682+0000 7fcfdbfff700 1 -- 172.21.15.27:0/3823689253 wait complete. 2024-01-11T08:25:30.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.684+0000 7fcfdbfff700 1 Processor -- start 2024-01-11T08:25:30.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.684+0000 7fcfdbfff700 1 -- start start 2024-01-11T08:25:30.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.685+0000 7fcfdbfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcfd40b03e0 0x7fcfd414a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.685+0000 7fcfdbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd40b07b0 0x7fcfd40c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.685+0000 7fcfdbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfd40b2b90 0x7fcfd40c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.685+0000 7fcfdbfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd40c85b0 con 0x7fcfd40b07b0 2024-01-11T08:25:30.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.685+0000 7fcfdbfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd40c79a0 con 0x7fcfd40b2b90 2024-01-11T08:25:30.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.685+0000 7fcfdbfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd40c7c80 con 0x7fcfd40b03e0 2024-01-11T08:25:30.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.685+0000 7fcfd959a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd40b07b0 0x7fcfd40c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.685+0000 7fcfd959a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd40b07b0 0x7fcfd40c9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57082/0 (socket says 172.21.15.27:57082) 2024-01-11T08:25:30.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.685+0000 7fcfd959a700 1 -- 172.21.15.27:0/4122067071 learned_addr learned my addr 172.21.15.27:0/4122067071 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:30.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.685+0000 7fcfd9d9b700 1 --2- 172.21.15.27:0/4122067071 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcfd40b03e0 0x7fcfd414a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.685+0000 7fcfda59c700 1 --2- 172.21.15.27:0/4122067071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfd40b2b90 0x7fcfd40c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.685+0000 7fcfd959a700 1 -- 172.21.15.27:0/4122067071 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcfd40b03e0 msgr2=0x7fcfd414a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:30.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.685+0000 7fcfd959a700 1 --2- 172.21.15.27:0/4122067071 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcfd40b03e0 0x7fcfd414a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.685+0000 7fcfd959a700 1 -- 172.21.15.27:0/4122067071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfd40b2b90 msgr2=0x7fcfd40c8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:30.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.685+0000 7fcfd959a700 1 --2- 172.21.15.27:0/4122067071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfd40b2b90 0x7fcfd40c8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.686+0000 7fcfd959a700 1 -- 172.21.15.27:0/4122067071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc4009420 con 0x7fcfd40b07b0 2024-01-11T08:25:30.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.686+0000 7fcfd959a700 1 --2- 172.21.15.27:0/4122067071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd40b07b0 0x7fcfd40c9ee0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4069540 tx=0x7fcfc4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:30.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.686+0000 7fcfc9ffb700 1 -- 172.21.15.27:0/4122067071 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc400bb20 con 0x7fcfd40b07b0 2024-01-11T08:25:30.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.686+0000 7fcfc9ffb700 1 -- 172.21.15.27:0/4122067071 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc4011420 con 0x7fcfd40b07b0 2024-01-11T08:25:30.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.686+0000 7fcfc9ffb700 1 -- 172.21.15.27:0/4122067071 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc40117c0 con 0x7fcfd40b07b0 2024-01-11T08:25:30.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.686+0000 7fcfdbfff700 1 -- 172.21.15.27:0/4122067071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd40c7f60 con 0x7fcfd40b07b0 2024-01-11T08:25:30.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.686+0000 7fcfdbfff700 1 -- 172.21.15.27:0/4122067071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd414ae10 con 0x7fcfd40b07b0 2024-01-11T08:25:30.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.687+0000 7fcfdbfff700 1 -- 172.21.15.27:0/4122067071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd40c8720 con 0x7fcfd40b07b0 2024-01-11T08:25:30.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.688+0000 7fcfc9ffb700 1 -- 172.21.15.27:0/4122067071 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fcfc4011b70 con 0x7fcfd40b07b0 2024-01-11T08:25:30.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.689+0000 7fcfc9ffb700 1 --2- 172.21.15.27:0/4122067071 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcfb8066f30 0x7fcfb80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:30.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.689+0000 7fcfc9ffb700 1 -- 172.21.15.27:0/4122067071 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5733+0+0 (secure 0 0 0) 0x7fcfc4085b20 con 0x7fcfd40b07b0 2024-01-11T08:25:30.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.690+0000 7fcfd9d9b700 1 --2- 172.21.15.27:0/4122067071 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcfb8066f30 0x7fcfb80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:30.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.690+0000 7fcfd9d9b700 1 --2- 172.21.15.27:0/4122067071 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcfb8066f30 0x7fcfb80693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00011e0 tx=0x7fcfd0009040 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:30.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.694+0000 7fcfc9ffb700 1 -- 172.21.15.27:0/4122067071 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcfc4052d70 con 0x7fcfd40b07b0 2024-01-11T08:25:30.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.945+0000 7fcfdbfff700 1 -- 172.21.15.27:0/4122067071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcfd40718d0 con 0x7fcfd40b07b0 2024-01-11T08:25:30.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.946+0000 7fcfc9ffb700 1 -- 172.21.15.27:0/4122067071 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v33) v1 ==== 53+0+73 (secure 0 0 0) 0x7fcfc4015060 con 0x7fcfd40b07b0 2024-01-11T08:25:30.946 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-11T08:25:30.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.948+0000 7fcfdbfff700 1 -- 172.21.15.27:0/4122067071 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcfb8066f30 msgr2=0x7fcfb80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:30.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.948+0000 7fcfdbfff700 1 --2- 172.21.15.27:0/4122067071 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcfb8066f30 0x7fcfb80693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00011e0 tx=0x7fcfd0009040 comp rx=0 tx=0).stop 2024-01-11T08:25:30.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.948+0000 7fcfdbfff700 1 -- 172.21.15.27:0/4122067071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd40b07b0 msgr2=0x7fcfd40c9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:30.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.948+0000 7fcfdbfff700 1 --2- 172.21.15.27:0/4122067071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd40b07b0 0x7fcfd40c9ee0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4069540 tx=0x7fcfc4017000 comp rx=0 tx=0).stop 2024-01-11T08:25:30.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.948+0000 7fcfdbfff700 1 -- 172.21.15.27:0/4122067071 shutdown_connections 2024-01-11T08:25:30.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.948+0000 7fcfdbfff700 1 --2- 172.21.15.27:0/4122067071 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fcfb8066f30 0x7fcfb80693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.948+0000 7fcfdbfff700 1 --2- 172.21.15.27:0/4122067071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfd40b2b90 0x7fcfd40c8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.948+0000 7fcfdbfff700 1 --2- 172.21.15.27:0/4122067071 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcfd40b03e0 0x7fcfd414a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.948+0000 7fcfdbfff700 1 --2- 172.21.15.27:0/4122067071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd40b07b0 0x7fcfd40c9ee0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:30.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.948+0000 7fcfdbfff700 1 -- 172.21.15.27:0/4122067071 >> 172.21.15.27:0/4122067071 conn(0x7fcfd406ed40 msgr2=0x7fcfd40d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:30.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.948+0000 7fcfdbfff700 1 -- 172.21.15.27:0/4122067071 shutdown_connections 2024-01-11T08:25:30.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:30.949+0000 7fcfdbfff700 1 -- 172.21.15.27:0/4122067071 wait complete. 2024-01-11T08:25:30.967 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-11T08:25:30.967 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-11T08:25:30.967 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-11T08:25:30.967 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-11T08:25:30.967 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-11T08:25:30.967 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-11T08:25:30.967 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-11T08:25:30.967 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-11T08:25:30.968 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:25:30.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:25:30.968 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-11T08:25:31.017 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T08:25:31.017 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T08:25:31.017 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T08:25:31.017 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:25:31.017 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-11T08:25:31.017 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:25:31.017 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:25:31.017 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:25:31.017 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-11T08:25:31.017 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:25:31.017 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:25:31.017 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-11T08:25:31.017 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-11T08:25:31.018 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-11T08:25:31.018 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-11T08:25:31.018 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-11T08:25:31.018 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 41591sec preferred_lft 41591sec 2024-01-11T08:25:31.018 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-11T08:25:31.018 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:25:31.018 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:25:31.018 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-11T08:25:31.018 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-11T08:25:31.018 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T08:25:31.018 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff 2024-01-11T08:25:31.018 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-11T08:25:31.018 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:25:31.018 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::acd2:b3ff:fe6a:db4b/64 scope link 2024-01-11T08:25:31.018 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:25:31.019 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T08:25:31.019 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:fa:85:1b:3a:bc brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-11T08:25:31.019 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ecfa:85ff:fe1b:3abc/64 scope link 2024-01-11T08:25:31.019 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:25:31.019 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T08:25:31.019 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-11T08:25:31.019 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ec88:faff:fe17:41aa/64 scope link 2024-01-11T08:25:31.019 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:25:31.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:25:31.019 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-11T08:25:31.090 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-11T08:25:31.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:25:31.093 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-11T08:25:31.144 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-11T08:25:31.144 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-11T08:25:31.144 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-11T08:25:31.144 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-11T08:25:31.144 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-11T08:25:31.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:25:31.145 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T08:25:31.145 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T08:25:31.145 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-11T08:25:31.145 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-11T08:25:31.145 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-11T08:25:31.145 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T08:25:31.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:25:31.321 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T08:25:31.368 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-11T08:25:31.368 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-11T08:25:31.369 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:25:31.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:25:31.370 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-11T08:25:31.442 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:25:31.442 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-11T08:25:31.490 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:25:31.490 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:25:31.490 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-11T08:25:31.537 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-11T08:25:31.537 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-11T08:25:31.537 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:25:31.641 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:25:31.641 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:25:31.641 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:25:31.656 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:25:31.656 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:25:31.656 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:25:31.656 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:25:31.656 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:25:31.678 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:25:31.678 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:25:31.678 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:25:31.678 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:25:31.679 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:25:31.679 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790" 2024-01-11T08:25:31.679 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790:/ 2024-01-11T08:25:31.679 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=8fddb4a7-9d66-40f1-892d-6fe86c728ee4" will pass to kernel 2024-01-11T08:25:31.682 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-11T08:25:31.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:25:31.683 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-11T08:25:31.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-11T08:25:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.961+0000 7fa0efbff700 1 Processor -- start 2024-01-11T08:25:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.962+0000 7fa0efbff700 1 -- start start 2024-01-11T08:25:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.963+0000 7fa0efbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0e815ca10 0x7fa0e815a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.963+0000 7fa0efbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa0e815cde0 0x7fa0e80685c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.963+0000 7fa0efbff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e80b4bd0 con 0x7fa0e815aed0 2024-01-11T08:25:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.963+0000 7fa0efbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e815ac70 con 0x7fa0e815cde0 2024-01-11T08:25:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.963+0000 7fa0efbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e80b3fc0 con 0x7fa0e815ca10 2024-01-11T08:25:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.963+0000 7fa0ed19a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa0e815cde0 0x7fa0e80685c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.963+0000 7fa0ed99b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0e815ca10 0x7fa0e815a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.964+0000 7fa0ed99b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0e815ca10 0x7fa0e815a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:40786/0 (socket says 172.21.15.27:40786) 2024-01-11T08:25:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.964+0000 7fa0ed19a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa0e815cde0 0x7fa0e80685c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:60150/0 (socket says 172.21.15.27:60150) 2024-01-11T08:25:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.964+0000 7fa0ed99b700 1 -- 172.21.15.27:0/3237260212 learned_addr learned my addr 172.21.15.27:0/3237260212 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.964+0000 7fa0ee19c700 1 --1- 172.21.15.27:0/3237260212 >> v1:172.21.15.77:6789/0 conn(0x7fa0e815aed0 0x7fa0e815d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:34398/0 (socket says 172.21.15.27:34398) 2024-01-11T08:25:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.964+0000 7fa0ed99b700 1 -- 172.21.15.27:0/3237260212 >> v2:172.21.15.77:3300/0 conn(0x7fa0e815cde0 msgr2=0x7fa0e80685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.964+0000 7fa0ed99b700 1 --2- 172.21.15.27:0/3237260212 >> v2:172.21.15.77:3300/0 conn(0x7fa0e815cde0 0x7fa0e80685c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.964+0000 7fa0ed99b700 1 -- 172.21.15.27:0/3237260212 >> v1:172.21.15.77:6789/0 conn(0x7fa0e815aed0 legacy=0x7fa0e815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.964+0000 7fa0ed99b700 1 -- 172.21.15.27:0/3237260212 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e80b42a0 con 0x7fa0e815ca10 2024-01-11T08:25:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.964+0000 7fa0ed19a700 1 --2- 172.21.15.27:0/3237260212 >> v2:172.21.15.77:3300/0 conn(0x7fa0e815cde0 0x7fa0e80685c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.964+0000 7fa0ec999700 1 -- 172.21.15.27:0/3237260212 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994870012 0 0) 0x7fa0e80b4bd0 con 0x7fa0e815aed0 2024-01-11T08:25:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.964+0000 7fa0ed99b700 1 --2- 172.21.15.27:0/3237260212 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0e815ca10 0x7fa0e815a730 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa0e40099b0 tx=0x7fa0e4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36c123d1a9298c2b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.965+0000 7fa0ec999700 1 -- 172.21.15.27:0/3237260212 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4009ea0 con 0x7fa0e815ca10 2024-01-11T08:25:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.965+0000 7fa0ec999700 1 -- 172.21.15.27:0/3237260212 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e400b860 con 0x7fa0e815ca10 2024-01-11T08:25:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.965+0000 7fa0ec999700 1 -- 172.21.15.27:0/3237260212 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e400bc00 con 0x7fa0e815ca10 2024-01-11T08:25:31.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.965+0000 7fa0efbff700 1 -- 172.21.15.27:0/3237260212 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0e815ca10 msgr2=0x7fa0e815a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:31.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.965+0000 7fa0efbff700 1 --2- 172.21.15.27:0/3237260212 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0e815ca10 0x7fa0e815a730 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa0e40099b0 tx=0x7fa0e4009250 comp rx=0 tx=0).stop 2024-01-11T08:25:31.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.965+0000 7fa0efbff700 1 -- 172.21.15.27:0/3237260212 shutdown_connections 2024-01-11T08:25:31.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.965+0000 7fa0efbff700 1 --2- 172.21.15.27:0/3237260212 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0e815ca10 0x7fa0e815a730 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:31.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.965+0000 7fa0efbff700 1 --2- 172.21.15.27:0/3237260212 >> v2:172.21.15.77:3300/0 conn(0x7fa0e815cde0 0x7fa0e80685c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:31.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.965+0000 7fa0efbff700 1 -- 172.21.15.27:0/3237260212 >> 172.21.15.27:0/3237260212 conn(0x7fa0e806ed40 msgr2=0x7fa0e806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:31.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.965+0000 7fa0efbff700 1 -- 172.21.15.27:0/3237260212 shutdown_connections 2024-01-11T08:25:31.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.966+0000 7fa0efbff700 1 -- 172.21.15.27:0/3237260212 wait complete. 2024-01-11T08:25:31.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.967+0000 7fa0efbff700 1 Processor -- start 2024-01-11T08:25:31.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.967+0000 7fa0efbff700 1 -- start start 2024-01-11T08:25:31.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.968+0000 7fa0efbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0e815aed0 0x7fa0e81441e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:31.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.968+0000 7fa0efbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0e815ca10 0x7fa0e8154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:31.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.968+0000 7fa0efbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0e815cde0 0x7fa0e8156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:31.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.968+0000 7fa0efbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e8156480 con 0x7fa0e815ca10 2024-01-11T08:25:31.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.968+0000 7fa0efbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e8155870 con 0x7fa0e815cde0 2024-01-11T08:25:31.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.968+0000 7fa0efbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e8155b50 con 0x7fa0e815aed0 2024-01-11T08:25:31.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.968+0000 7fa0ed19a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0e815ca10 0x7fa0e8154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:31.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.968+0000 7fa0ed19a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0e815ca10 0x7fa0e8154460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57090/0 (socket says 172.21.15.27:57090) 2024-01-11T08:25:31.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.968+0000 7fa0ed19a700 1 -- 172.21.15.27:0/1493425853 learned_addr learned my addr 172.21.15.27:0/1493425853 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:31.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.969+0000 7fa0ed99b700 1 --2- 172.21.15.27:0/1493425853 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0e815aed0 0x7fa0e81441e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:31.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.969+0000 7fa0ee19c700 1 --2- 172.21.15.27:0/1493425853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0e815cde0 0x7fa0e8156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:31.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.969+0000 7fa0ed19a700 1 -- 172.21.15.27:0/1493425853 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0e815aed0 msgr2=0x7fa0e81441e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:31.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.969+0000 7fa0ed19a700 1 --2- 172.21.15.27:0/1493425853 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0e815aed0 0x7fa0e81441e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:31.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.969+0000 7fa0ed19a700 1 -- 172.21.15.27:0/1493425853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0e815cde0 msgr2=0x7fa0e8156d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:31.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.969+0000 7fa0ed19a700 1 --2- 172.21.15.27:0/1493425853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0e815cde0 0x7fa0e8156d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:31.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.969+0000 7fa0ed19a700 1 -- 172.21.15.27:0/1493425853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d8009420 con 0x7fa0e815ca10 2024-01-11T08:25:31.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.969+0000 7fa0ed19a700 1 --2- 172.21.15.27:0/1493425853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0e815ca10 0x7fa0e8154460 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa0d800ec80 tx=0x7fa0d80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:31.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.969+0000 7fa0ddffb700 1 -- 172.21.15.27:0/1493425853 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d800c850 con 0x7fa0e815ca10 2024-01-11T08:25:31.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.970+0000 7fa0ddffb700 1 -- 172.21.15.27:0/1493425853 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d800c9b0 con 0x7fa0e815ca10 2024-01-11T08:25:31.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.970+0000 7fa0efbff700 1 -- 172.21.15.27:0/1493425853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e40094f0 con 0x7fa0e815ca10 2024-01-11T08:25:31.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.970+0000 7fa0ddffb700 1 -- 172.21.15.27:0/1493425853 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d800cd30 con 0x7fa0e815ca10 2024-01-11T08:25:31.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.970+0000 7fa0efbff700 1 -- 172.21.15.27:0/1493425853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e8156110 con 0x7fa0e815ca10 2024-01-11T08:25:31.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.971+0000 7fa0efbff700 1 -- 172.21.15.27:0/1493425853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e81565c0 con 0x7fa0e815ca10 2024-01-11T08:25:31.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.972+0000 7fa0ddffb700 1 -- 172.21.15.27:0/1493425853 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa0d8013680 con 0x7fa0e815ca10 2024-01-11T08:25:31.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.973+0000 7fa0ddffb700 1 --2- 172.21.15.27:0/1493425853 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa0cc066f30 0x7fa0cc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:31.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.973+0000 7fa0ddffb700 1 -- 172.21.15.27:0/1493425853 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5733+0+0 (secure 0 0 0) 0x7fa0d8084b10 con 0x7fa0e815ca10 2024-01-11T08:25:31.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.973+0000 7fa0ed99b700 1 --2- 172.21.15.27:0/1493425853 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa0cc066f30 0x7fa0cc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:31.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.974+0000 7fa0ed99b700 1 --2- 172.21.15.27:0/1493425853 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa0cc066f30 0x7fa0cc0693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa0e40099b0 tx=0x7fa0e4017000 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:31.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:31.977+0000 7fa0ddffb700 1 -- 172.21.15.27:0/1493425853 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa0e81565c0 con 0x7fa0e815ca10 2024-01-11T08:25:32.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.206+0000 7fa0efbff700 1 -- 172.21.15.27:0/1493425853 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0e81565c0 con 0x7fa0cc066f30 2024-01-11T08:25:32.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.209+0000 7fa0ddffb700 1 -- 172.21.15.27:0/1493425853 <== mgr.5450 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fa0e81565c0 con 0x7fa0cc066f30 2024-01-11T08:25:32.209 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T08:25:32.209 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:32.209 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-11T08:25:32.209 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:32.209 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T08:25:32.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.211+0000 7fa0efbff700 1 -- 172.21.15.27:0/1493425853 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa0cc066f30 msgr2=0x7fa0cc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:32.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.211+0000 7fa0efbff700 1 --2- 172.21.15.27:0/1493425853 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa0cc066f30 0x7fa0cc0693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa0e40099b0 tx=0x7fa0e4017000 comp rx=0 tx=0).stop 2024-01-11T08:25:32.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.211+0000 7fa0efbff700 1 -- 172.21.15.27:0/1493425853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0e815ca10 msgr2=0x7fa0e8154460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:32.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.211+0000 7fa0efbff700 1 --2- 172.21.15.27:0/1493425853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0e815ca10 0x7fa0e8154460 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa0d800ec80 tx=0x7fa0d80091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:32.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.211+0000 7fa0efbff700 1 -- 172.21.15.27:0/1493425853 shutdown_connections 2024-01-11T08:25:32.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.211+0000 7fa0efbff700 1 --2- 172.21.15.27:0/1493425853 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fa0cc066f30 0x7fa0cc0693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:32.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.211+0000 7fa0efbff700 1 --2- 172.21.15.27:0/1493425853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0e815cde0 0x7fa0e8156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:32.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.211+0000 7fa0efbff700 1 --2- 172.21.15.27:0/1493425853 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0e815aed0 0x7fa0e81441e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:32.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.211+0000 7fa0efbff700 1 --2- 172.21.15.27:0/1493425853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0e815ca10 0x7fa0e8154460 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:32.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.211+0000 7fa0efbff700 1 -- 172.21.15.27:0/1493425853 >> 172.21.15.27:0/1493425853 conn(0x7fa0e806ed40 msgr2=0x7fa0e81523a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:32.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.211+0000 7fa0efbff700 1 -- 172.21.15.27:0/1493425853 shutdown_connections 2024-01-11T08:25:32.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.211+0000 7fa0efbff700 1 -- 172.21.15.27:0/1493425853 wait complete. 2024-01-11T08:25:32.229 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-11T08:25:32.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.430+0000 7feb9c1ff700 1 Processor -- start 2024-01-11T08:25:32.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.430+0000 7feb9c1ff700 1 -- start start 2024-01-11T08:25:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.431+0000 7feb9c1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feb9413fbb0 0x7feb940ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.431+0000 7feb9c1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb9415a730 0x7feb940c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.431+0000 7feb9c1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb940c6b30 con 0x7feb9413ff80 2024-01-11T08:25:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.431+0000 7feb9c1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb940c6800 con 0x7feb9413fbb0 2024-01-11T08:25:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.432+0000 7feb9c1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb940cb240 con 0x7feb9415a730 2024-01-11T08:25:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.432+0000 7feb99f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feb9413fbb0 0x7feb940ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.432+0000 7feb99f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feb9413fbb0 0x7feb940ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57106/0 (socket says 172.21.15.27:57106) 2024-01-11T08:25:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.432+0000 7feb99f9b700 1 -- 172.21.15.27:0/3006724477 learned_addr learned my addr 172.21.15.27:0/3006724477 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.432+0000 7feb9a79c700 1 --2- 172.21.15.27:0/3006724477 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb9415a730 0x7feb940c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.432+0000 7feb99f9b700 1 -- 172.21.15.27:0/3006724477 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb9415a730 msgr2=0x7feb940c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.432+0000 7feb99f9b700 1 --2- 172.21.15.27:0/3006724477 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb9415a730 0x7feb940c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.432+0000 7feb99f9b700 1 -- 172.21.15.27:0/3006724477 >> v1:172.21.15.77:6789/0 conn(0x7feb9413ff80 legacy=0x7feb9415a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.432+0000 7feb98f99700 1 -- 172.21.15.27:0/3006724477 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773367168 0 0) 0x7feb940c6b30 con 0x7feb9413ff80 2024-01-11T08:25:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.433+0000 7feb99f9b700 1 -- 172.21.15.27:0/3006724477 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb940cb520 con 0x7feb9413fbb0 2024-01-11T08:25:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.433+0000 7feb99f9b700 1 --2- 172.21.15.27:0/3006724477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb9413fbb0 0x7feb940ca7a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7feb9000bc40 tx=0x7feb9000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809405b3e9b6cfd1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.433+0000 7feb98f99700 1 -- 172.21.15.27:0/3006724477 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9000dba0 con 0x7feb9413fbb0 2024-01-11T08:25:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.433+0000 7feb98f99700 1 -- 172.21.15.27:0/3006724477 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9000dd00 con 0x7feb9413fbb0 2024-01-11T08:25:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.433+0000 7feb98f99700 1 -- 172.21.15.27:0/3006724477 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb900196d0 con 0x7feb9413fbb0 2024-01-11T08:25:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.434+0000 7feb9c1ff700 1 -- 172.21.15.27:0/3006724477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb9413fbb0 msgr2=0x7feb940ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.434+0000 7feb9c1ff700 1 --2- 172.21.15.27:0/3006724477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb9413fbb0 0x7feb940ca7a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7feb9000bc40 tx=0x7feb9000b4f0 comp rx=0 tx=0).stop 2024-01-11T08:25:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.434+0000 7feb9c1ff700 1 -- 172.21.15.27:0/3006724477 shutdown_connections 2024-01-11T08:25:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.434+0000 7feb9c1ff700 1 --2- 172.21.15.27:0/3006724477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb9413fbb0 0x7feb940ca7a0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.434+0000 7feb9c1ff700 1 --2- 172.21.15.27:0/3006724477 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb9415a730 0x7feb940c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.434+0000 7feb9c1ff700 1 -- 172.21.15.27:0/3006724477 >> 172.21.15.27:0/3006724477 conn(0x7feb940680a0 msgr2=0x7feb940cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.434+0000 7feb9c1ff700 1 -- 172.21.15.27:0/3006724477 shutdown_connections 2024-01-11T08:25:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.434+0000 7feb9c1ff700 1 -- 172.21.15.27:0/3006724477 wait complete. 2024-01-11T08:25:32.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.435+0000 7feb9c1ff700 1 Processor -- start 2024-01-11T08:25:32.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.435+0000 7feb9c1ff700 1 -- start start 2024-01-11T08:25:32.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.436+0000 7feb9c1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb9413fbb0 0x7feb940c7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:32.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.436+0000 7feb9c1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb9413ff80 0x7feb940c3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:32.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.436+0000 7feb9c1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb9415a730 0x7feb940b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:32.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.436+0000 7feb9c1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb940b0b30 con 0x7feb9413ff80 2024-01-11T08:25:32.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.436+0000 7feb9c1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb940b0e10 con 0x7feb9415a730 2024-01-11T08:25:32.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.436+0000 7feb9c1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb940c4fa0 con 0x7feb9413fbb0 2024-01-11T08:25:32.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.437+0000 7feb9979a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb9413ff80 0x7feb940c3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:32.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.437+0000 7feb9979a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb9413ff80 0x7feb940c3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57112/0 (socket says 172.21.15.27:57112) 2024-01-11T08:25:32.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.437+0000 7feb9979a700 1 -- 172.21.15.27:0/226758424 learned_addr learned my addr 172.21.15.27:0/226758424 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:32.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.437+0000 7feb9a79c700 1 --2- 172.21.15.27:0/226758424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb9415a730 0x7feb940b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:32.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.437+0000 7feb99f9b700 1 --2- 172.21.15.27:0/226758424 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb9413fbb0 0x7feb940c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:32.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.437+0000 7feb9979a700 1 -- 172.21.15.27:0/226758424 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb9413fbb0 msgr2=0x7feb940c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:32.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.437+0000 7feb9979a700 1 --2- 172.21.15.27:0/226758424 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb9413fbb0 0x7feb940c7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:32.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.437+0000 7feb9979a700 1 -- 172.21.15.27:0/226758424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb9415a730 msgr2=0x7feb940b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:32.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.437+0000 7feb9979a700 1 --2- 172.21.15.27:0/226758424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb9415a730 0x7feb940b00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:32.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.437+0000 7feb9979a700 1 -- 172.21.15.27:0/226758424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb9000b730 con 0x7feb9413ff80 2024-01-11T08:25:32.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.437+0000 7feb9a79c700 1 --2- 172.21.15.27:0/226758424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb9415a730 0x7feb940b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:32.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.437+0000 7feb99f9b700 1 --2- 172.21.15.27:0/226758424 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb9413fbb0 0x7feb940c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:32.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.437+0000 7feb9979a700 1 --2- 172.21.15.27:0/226758424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb9413ff80 0x7feb940c3270 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7feb8400c7e0 tx=0x7feb84007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:32.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.437+0000 7feb89ffb700 1 -- 172.21.15.27:0/226758424 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb84012700 con 0x7feb9413ff80 2024-01-11T08:25:32.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.438+0000 7feb89ffb700 1 -- 172.21.15.27:0/226758424 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb84012880 con 0x7feb9413ff80 2024-01-11T08:25:32.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.438+0000 7feb9c1ff700 1 -- 172.21.15.27:0/226758424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb940c5280 con 0x7feb9413ff80 2024-01-11T08:25:32.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.438+0000 7feb89ffb700 1 -- 172.21.15.27:0/226758424 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb84012c40 con 0x7feb9413ff80 2024-01-11T08:25:32.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.438+0000 7feb9c1ff700 1 -- 172.21.15.27:0/226758424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb940b1e80 con 0x7feb9413ff80 2024-01-11T08:25:32.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.439+0000 7feb9c1ff700 1 -- 172.21.15.27:0/226758424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb940c5470 con 0x7feb9413ff80 2024-01-11T08:25:32.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.439+0000 7feb89ffb700 1 -- 172.21.15.27:0/226758424 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 84742+0+0 (secure 0 0 0) 0x7feb8401b400 con 0x7feb9413ff80 2024-01-11T08:25:32.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.440+0000 7feb89ffb700 1 --2- 172.21.15.27:0/226758424 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7feb78066f30 0x7feb780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:32.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.440+0000 7feb89ffb700 1 -- 172.21.15.27:0/226758424 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5733+0+0 (secure 0 0 0) 0x7feb840841c0 con 0x7feb9413ff80 2024-01-11T08:25:32.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.441+0000 7feb99f9b700 1 --2- 172.21.15.27:0/226758424 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7feb78066f30 0x7feb780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:32.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.442+0000 7feb99f9b700 1 --2- 172.21.15.27:0/226758424 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7feb78066f30 0x7feb780693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feb9000bc40 tx=0x7feb9001b000 comp rx=0 tx=0).ready entity=mgr.5450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:32.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.444+0000 7feb89ffb700 1 -- 172.21.15.27:0/226758424 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feb84051370 con 0x7feb9413ff80 2024-01-11T08:25:32.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.703+0000 7feb9c1ff700 1 -- 172.21.15.27:0/226758424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7feb9404c3c0 con 0x7feb9413ff80 2024-01-11T08:25:32.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.902+0000 7feb89ffb700 1 -- 172.21.15.27:0/226758424 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 31) v1 ==== 84759+0+0 (secure 0 0 0) 0x7feb84051740 con 0x7feb9413ff80 2024-01-11T08:25:32.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.903+0000 7feb89ffb700 1 -- 172.21.15.27:0/226758424 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v31) v1 ==== 92+0+0 (secure 0 0 0) 0x7feb84054e00 con 0x7feb9413ff80 2024-01-11T08:25:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.906+0000 7feb9c1ff700 1 -- 172.21.15.27:0/226758424 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7feb78066f30 msgr2=0x7feb780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.906+0000 7feb9c1ff700 1 --2- 172.21.15.27:0/226758424 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7feb78066f30 0x7feb780693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feb9000bc40 tx=0x7feb9001b000 comp rx=0 tx=0).stop 2024-01-11T08:25:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.906+0000 7feb9c1ff700 1 -- 172.21.15.27:0/226758424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb9413ff80 msgr2=0x7feb940c3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.906+0000 7feb9c1ff700 1 --2- 172.21.15.27:0/226758424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb9413ff80 0x7feb940c3270 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7feb8400c7e0 tx=0x7feb84007df0 comp rx=0 tx=0).stop 2024-01-11T08:25:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.906+0000 7feb9c1ff700 1 -- 172.21.15.27:0/226758424 shutdown_connections 2024-01-11T08:25:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.906+0000 7feb9c1ff700 1 --2- 172.21.15.27:0/226758424 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7feb78066f30 0x7feb780693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.906+0000 7feb9c1ff700 1 --2- 172.21.15.27:0/226758424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb9415a730 0x7feb940b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.906+0000 7feb9c1ff700 1 --2- 172.21.15.27:0/226758424 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb9413fbb0 0x7feb940c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.906+0000 7feb9c1ff700 1 --2- 172.21.15.27:0/226758424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb9413ff80 0x7feb940c3270 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.906+0000 7feb9c1ff700 1 -- 172.21.15.27:0/226758424 >> 172.21.15.27:0/226758424 conn(0x7feb940680a0 msgr2=0x7feb940c8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.907+0000 7feb9c1ff700 1 -- 172.21.15.27:0/226758424 shutdown_connections 2024-01-11T08:25:32.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:32.907+0000 7feb9c1ff700 1 -- 172.21.15.27:0/226758424 wait complete. 2024-01-11T08:25:32.925 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-11T08:25:33.116 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:33.116+0000 7f79e88fb1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:25:33.120 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:33.120+0000 7fb6fb0fe1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:25:33.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.135+0000 7f9559bff700 1 Processor -- start 2024-01-11T08:25:33.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.135+0000 7f9559bff700 1 -- start start 2024-01-11T08:25:33.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.136+0000 7f9559bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9554142dc0 0x7f95540cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:33.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.136+0000 7f9559bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9554143190 0x7f95540cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:33.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.136+0000 7f9559bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f95540cc020 0x7f95540cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:33.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.136+0000 7f9559bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f955415df30 con 0x7f9554142dc0 2024-01-11T08:25:33.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.136+0000 7f9559bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f955415d320 con 0x7f95540cc020 2024-01-11T08:25:33.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.136+0000 7f9559bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f955415d590 con 0x7f9554143190 2024-01-11T08:25:33.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.137+0000 7f95537fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9554142dc0 0x7f95540cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:33.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.137+0000 7f95537fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9554142dc0 0x7f95540cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57130/0 (socket says 172.21.15.27:57130) 2024-01-11T08:25:33.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.137+0000 7f95537fe700 1 -- 172.21.15.27:0/1525146277 learned_addr learned my addr 172.21.15.27:0/1525146277 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:33.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.137+0000 7f9552ffd700 1 --2- 172.21.15.27:0/1525146277 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9554143190 0x7f95540cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:33.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.137+0000 7f9553fff700 1 --2- 172.21.15.27:0/1525146277 >> v2:172.21.15.77:3300/0 conn(0x7f95540cc020 0x7f95540cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.137+0000 7f95537fe700 1 -- 172.21.15.27:0/1525146277 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9554143190 msgr2=0x7f95540cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.137+0000 7f95537fe700 1 --2- 172.21.15.27:0/1525146277 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9554143190 0x7f95540cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.137+0000 7f95537fe700 1 -- 172.21.15.27:0/1525146277 >> v2:172.21.15.77:3300/0 conn(0x7f95540cc020 msgr2=0x7f95540cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.137+0000 7f95537fe700 1 --2- 172.21.15.27:0/1525146277 >> v2:172.21.15.77:3300/0 conn(0x7f95540cc020 0x7f95540cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.137+0000 7f95537fe700 1 -- 172.21.15.27:0/1525146277 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955415d870 con 0x7f9554142dc0 2024-01-11T08:25:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.137+0000 7f9553fff700 1 --2- 172.21.15.27:0/1525146277 >> v2:172.21.15.77:3300/0 conn(0x7f95540cc020 0x7f95540cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.137+0000 7f9552ffd700 1 --2- 172.21.15.27:0/1525146277 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9554143190 0x7f95540cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.138+0000 7f95537fe700 1 --2- 172.21.15.27:0/1525146277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9554142dc0 0x7f95540cada0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f95540ce390 tx=0x7f9544009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be41d1e12d6fe008 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.138+0000 7f95527fc700 1 -- 172.21.15.27:0/1525146277 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954400b640 con 0x7f9554142dc0 2024-01-11T08:25:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.138+0000 7f95527fc700 1 -- 172.21.15.27:0/1525146277 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954400b7a0 con 0x7f9554142dc0 2024-01-11T08:25:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.138+0000 7f95527fc700 1 -- 172.21.15.27:0/1525146277 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954400bb60 con 0x7f9554142dc0 2024-01-11T08:25:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.139+0000 7f9559bff700 1 -- 172.21.15.27:0/1525146277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9554142dc0 msgr2=0x7f95540cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.139+0000 7f9559bff700 1 --2- 172.21.15.27:0/1525146277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9554142dc0 0x7f95540cada0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f95540ce390 tx=0x7f9544009250 comp rx=0 tx=0).stop 2024-01-11T08:25:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.139+0000 7f9559bff700 1 -- 172.21.15.27:0/1525146277 shutdown_connections 2024-01-11T08:25:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.139+0000 7f9559bff700 1 --2- 172.21.15.27:0/1525146277 >> v2:172.21.15.77:3300/0 conn(0x7f95540cc020 0x7f95540cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.139+0000 7f9559bff700 1 --2- 172.21.15.27:0/1525146277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9554142dc0 0x7f95540cada0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.139+0000 7f9559bff700 1 --2- 172.21.15.27:0/1525146277 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9554143190 0x7f95540cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.139+0000 7f9559bff700 1 -- 172.21.15.27:0/1525146277 >> 172.21.15.27:0/1525146277 conn(0x7f9554063ca0 msgr2=0x7f955406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.139+0000 7f9559bff700 1 -- 172.21.15.27:0/1525146277 shutdown_connections 2024-01-11T08:25:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.139+0000 7f9559bff700 1 -- 172.21.15.27:0/1525146277 wait complete. 2024-01-11T08:25:33.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.140+0000 7f9559bff700 1 Processor -- start 2024-01-11T08:25:33.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.140+0000 7f9559bff700 1 -- start start 2024-01-11T08:25:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.141+0000 7f9559bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95540cc020 0x7f955416a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.141+0000 7f9559bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9554142dc0 0x7f95540b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.141+0000 7f9559bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9554143190 0x7f9554139a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.141+0000 7f9559bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955416b200 con 0x7f9554143190 2024-01-11T08:25:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.141+0000 7f9559bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95540b3590 con 0x7f95540cc020 2024-01-11T08:25:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.141+0000 7f9559bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9554139f50 con 0x7f9554142dc0 2024-01-11T08:25:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.142+0000 7f9553fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9554143190 0x7f9554139a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.142+0000 7f9553fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9554143190 0x7f9554139a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57144/0 (socket says 172.21.15.27:57144) 2024-01-11T08:25:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.142+0000 7f9553fff700 1 -- 172.21.15.27:0/3695327403 learned_addr learned my addr 172.21.15.27:0/3695327403 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.142+0000 7f95537fe700 1 --2- 172.21.15.27:0/3695327403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95540cc020 0x7f955416a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.142+0000 7f9552ffd700 1 --2- 172.21.15.27:0/3695327403 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9554142dc0 0x7f95540b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.142+0000 7f9553fff700 1 -- 172.21.15.27:0/3695327403 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9554142dc0 msgr2=0x7f95540b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.142+0000 7f9553fff700 1 --2- 172.21.15.27:0/3695327403 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9554142dc0 0x7f95540b3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.142+0000 7f9553fff700 1 -- 172.21.15.27:0/3695327403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95540cc020 msgr2=0x7f955416a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.142+0000 7f9553fff700 1 --2- 172.21.15.27:0/3695327403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95540cc020 0x7f955416a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.142+0000 7f9553fff700 1 -- 172.21.15.27:0/3695327403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954c009a20 con 0x7f9554143190 2024-01-11T08:25:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.142+0000 7f9553fff700 1 --2- 172.21.15.27:0/3695327403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9554143190 0x7f9554139a10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f954c009fc0 tx=0x7f954c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.143+0000 7f953b7fe700 1 -- 172.21.15.27:0/3695327403 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c00cbd0 con 0x7f9554143190 2024-01-11T08:25:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.143+0000 7f953b7fe700 1 -- 172.21.15.27:0/3695327403 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954c00f040 con 0x7f9554143190 2024-01-11T08:25:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.143+0000 7f953b7fe700 1 -- 172.21.15.27:0/3695327403 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c014480 con 0x7f9554143190 2024-01-11T08:25:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.143+0000 7f9559bff700 1 -- 172.21.15.27:0/3695327403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9548009420 con 0x7f9554143190 2024-01-11T08:25:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.143+0000 7f9559bff700 1 -- 172.21.15.27:0/3695327403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95440094f0 con 0x7f9554143190 2024-01-11T08:25:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.144+0000 7f9559bff700 1 -- 172.21.15.27:0/3695327403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f955416b370 con 0x7f9554143190 2024-01-11T08:25:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.145+0000 7f953b7fe700 1 -- 172.21.15.27:0/3695327403 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 31) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f954c003020 con 0x7f9554143190 2024-01-11T08:25:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.146+0000 7f953b7fe700 1 --2- 172.21.15.27:0/3695327403 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9540066f80 0x7f9540069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.146+0000 7f953b7fe700 1 -- 172.21.15.27:0/3695327403 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f954c084fe0 con 0x7f9554143190 2024-01-11T08:25:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.146+0000 7f95537fe700 1 -- 172.21.15.27:0/3695327403 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9540066f80 msgr2=0x7f9540069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.146+0000 7f95537fe700 1 --2- 172.21.15.27:0/3695327403 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9540066f80 0x7f9540069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:33.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.150+0000 7f953b7fe700 1 -- 172.21.15.27:0/3695327403 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f954c052190 con 0x7f9554143190 2024-01-11T08:25:33.296 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:33.296+0000 7f79e88fb1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:25:33.319 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:33.319+0000 7fb6fb0fe1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:25:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.347+0000 7f95537fe700 1 -- 172.21.15.27:0/3695327403 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9540066f80 msgr2=0x7f9540069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.347+0000 7f95537fe700 1 --2- 172.21.15.27:0/3695327403 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9540066f80 0x7f9540069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:33.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.397+0000 7f9559bff700 1 -- 172.21.15.27:0/3695327403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f955415db50 con 0x7f9554143190 2024-01-11T08:25:33.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.401+0000 7f953b7fe700 1 -- 172.21.15.27:0/3695327403 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7f954c0557b0 con 0x7f9554143190 2024-01-11T08:25:33.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.403+0000 7f9559bff700 1 -- 172.21.15.27:0/3695327403 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9540066f80 msgr2=0x7f9540069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:33.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.403+0000 7f9559bff700 1 --2- 172.21.15.27:0/3695327403 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9540066f80 0x7f9540069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.403+0000 7f9559bff700 1 -- 172.21.15.27:0/3695327403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9554143190 msgr2=0x7f9554139a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:33.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.403+0000 7f9559bff700 1 --2- 172.21.15.27:0/3695327403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9554143190 0x7f9554139a10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f954c009fc0 tx=0x7f954c00b040 comp rx=0 tx=0).stop 2024-01-11T08:25:33.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.403+0000 7f9559bff700 1 -- 172.21.15.27:0/3695327403 shutdown_connections 2024-01-11T08:25:33.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.403+0000 7f9559bff700 1 --2- 172.21.15.27:0/3695327403 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f9540066f80 0x7f9540069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.403+0000 7f9559bff700 1 --2- 172.21.15.27:0/3695327403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9554143190 0x7f9554139a10 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.403+0000 7f9559bff700 1 --2- 172.21.15.27:0/3695327403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95540cc020 0x7f955416a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.403+0000 7f9559bff700 1 --2- 172.21.15.27:0/3695327403 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9554142dc0 0x7f95540b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.403+0000 7f9559bff700 1 -- 172.21.15.27:0/3695327403 >> 172.21.15.27:0/3695327403 conn(0x7f9554063ca0 msgr2=0x7f95540c39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:33.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.403+0000 7f9559bff700 1 -- 172.21.15.27:0/3695327403 shutdown_connections 2024-01-11T08:25:33.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.403+0000 7f9559bff700 1 -- 172.21.15.27:0/3695327403 wait complete. 2024-01-11T08:25:33.422 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-11T08:25:33.613 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:33.614+0000 7f79e88fb1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:25:33.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.629+0000 7f81e1fff700 1 Processor -- start 2024-01-11T08:25:33.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.629+0000 7f81e1fff700 1 -- start start 2024-01-11T08:25:33.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.630+0000 7f81e1fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f81dc140dc0 0x7f81dc0adc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:33.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.630+0000 7f81e1fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f81dc141190 0x7f81dc0c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:33.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.630+0000 7f81e1fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81dc15c060 con 0x7f81dc0c8f60 2024-01-11T08:25:33.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.630+0000 7f81e1fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81dc0cc9b0 con 0x7f81dc140dc0 2024-01-11T08:25:33.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.630+0000 7f81e1fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81dc0c9370 con 0x7f81dc141190 2024-01-11T08:25:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.630+0000 7f81db7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f81dc140dc0 0x7f81dc0adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.630+0000 7f81db7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f81dc140dc0 0x7f81dc0adc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57154/0 (socket says 172.21.15.27:57154) 2024-01-11T08:25:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.630+0000 7f81db7fe700 1 -- 172.21.15.27:0/1203995966 learned_addr learned my addr 172.21.15.27:0/1203995966 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.630+0000 7f81daffd700 1 --2- 172.21.15.27:0/1203995966 >> v2:172.21.15.77:3300/0 conn(0x7f81dc141190 0x7f81dc0c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.630+0000 7f81db7fe700 1 -- 172.21.15.27:0/1203995966 >> v2:172.21.15.77:3300/0 conn(0x7f81dc141190 msgr2=0x7f81dc0c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.630+0000 7f81db7fe700 1 --2- 172.21.15.27:0/1203995966 >> v2:172.21.15.77:3300/0 conn(0x7f81dc141190 0x7f81dc0c8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.630+0000 7f81db7fe700 1 -- 172.21.15.27:0/1203995966 >> v1:172.21.15.77:6789/0 conn(0x7f81dc0c8f60 legacy=0x7f81dc15b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.630+0000 7f81db7fe700 1 -- 172.21.15.27:0/1203995966 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc15bd80 con 0x7f81dc140dc0 2024-01-11T08:25:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.630+0000 7f81da7fc700 1 -- 172.21.15.27:0/1203995966 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538614369 0 0) 0x7f81d4001490 con 0x7f81dc0c8f60 2024-01-11T08:25:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.630+0000 7f81db7fe700 1 --2- 172.21.15.27:0/1203995966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81dc140dc0 0x7f81dc0adc80 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0c5410 tx=0x7f81cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b60a2b94d3409ffd server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:33.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.632+0000 7f81da7fc700 1 -- 172.21.15.27:0/1203995966 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc009e70 con 0x7f81dc140dc0 2024-01-11T08:25:33.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.632+0000 7f81da7fc700 1 -- 172.21.15.27:0/1203995966 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81cc00b860 con 0x7f81dc140dc0 2024-01-11T08:25:33.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.632+0000 7f81da7fc700 1 -- 172.21.15.27:0/1203995966 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc00bc00 con 0x7f81dc140dc0 2024-01-11T08:25:33.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.632+0000 7f81e1fff700 1 -- 172.21.15.27:0/1203995966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81dc140dc0 msgr2=0x7f81dc0adc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:33.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.632+0000 7f81e1fff700 1 --2- 172.21.15.27:0/1203995966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81dc140dc0 0x7f81dc0adc80 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0c5410 tx=0x7f81cc009250 comp rx=0 tx=0).stop 2024-01-11T08:25:33.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.632+0000 7f81e1fff700 1 -- 172.21.15.27:0/1203995966 shutdown_connections 2024-01-11T08:25:33.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.632+0000 7f81e1fff700 1 --2- 172.21.15.27:0/1203995966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81dc140dc0 0x7f81dc0adc80 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.632+0000 7f81e1fff700 1 --2- 172.21.15.27:0/1203995966 >> v2:172.21.15.77:3300/0 conn(0x7f81dc141190 0x7f81dc0c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.632+0000 7f81e1fff700 1 -- 172.21.15.27:0/1203995966 >> 172.21.15.27:0/1203995966 conn(0x7f81dc063ca0 msgr2=0x7f81dc06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.632+0000 7f81e1fff700 1 -- 172.21.15.27:0/1203995966 shutdown_connections 2024-01-11T08:25:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.632+0000 7f81e1fff700 1 -- 172.21.15.27:0/1203995966 wait complete. 2024-01-11T08:25:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.633+0000 7f81e1fff700 1 Processor -- start 2024-01-11T08:25:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.633+0000 7f81e1fff700 1 -- start start 2024-01-11T08:25:33.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.634+0000 7f81e1fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81dc0c8f60 0x7f81dc156450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:33.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.634+0000 7f81e1fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f81dc140dc0 0x7f81dc0b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:33.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.634+0000 7f81e1fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81dc141190 0x7f81dc0b16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:33.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.634+0000 7f81e1fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc0b3f90 con 0x7f81dc0c8f60 2024-01-11T08:25:33.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.634+0000 7f81e1fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc0b3380 con 0x7f81dc141190 2024-01-11T08:25:33.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.634+0000 7f81e1fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc0b3660 con 0x7f81dc140dc0 2024-01-11T08:25:33.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.634+0000 7f81db7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81dc0c8f60 0x7f81dc156450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:33.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.634+0000 7f81db7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81dc0c8f60 0x7f81dc156450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57156/0 (socket says 172.21.15.27:57156) 2024-01-11T08:25:33.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.634+0000 7f81db7fe700 1 -- 172.21.15.27:0/2791169595 learned_addr learned my addr 172.21.15.27:0/2791169595 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:33.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.634+0000 7f81daffd700 1 --2- 172.21.15.27:0/2791169595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f81dc140dc0 0x7f81dc0b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:33.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.634+0000 7f81dbfff700 1 --2- 172.21.15.27:0/2791169595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81dc141190 0x7f81dc0b16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.635+0000 7f81db7fe700 1 -- 172.21.15.27:0/2791169595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f81dc140dc0 msgr2=0x7f81dc0b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.635+0000 7f81db7fe700 1 --2- 172.21.15.27:0/2791169595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f81dc140dc0 0x7f81dc0b1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.635+0000 7f81db7fe700 1 -- 172.21.15.27:0/2791169595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81dc141190 msgr2=0x7f81dc0b16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.635+0000 7f81db7fe700 1 --2- 172.21.15.27:0/2791169595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81dc141190 0x7f81dc0b16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.635+0000 7f81db7fe700 1 -- 172.21.15.27:0/2791169595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc0094f0 con 0x7f81dc0c8f60 2024-01-11T08:25:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.635+0000 7f81db7fe700 1 --2- 172.21.15.27:0/2791169595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81dc0c8f60 0x7f81dc156450 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f81cc0094c0 tx=0x7f81cc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.635+0000 7f81c37fe700 1 -- 172.21.15.27:0/2791169595 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc000c90 con 0x7f81dc0c8f60 2024-01-11T08:25:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.637+0000 7f81c37fe700 1 -- 172.21.15.27:0/2791169595 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81cc00b9d0 con 0x7f81dc0c8f60 2024-01-11T08:25:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.637+0000 7f81e1fff700 1 -- 172.21.15.27:0/2791169595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81dc0b3940 con 0x7f81dc0c8f60 2024-01-11T08:25:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.637+0000 7f81c37fe700 1 -- 172.21.15.27:0/2791169595 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc025420 con 0x7f81dc0c8f60 2024-01-11T08:25:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.637+0000 7f81e1fff700 1 -- 172.21.15.27:0/2791169595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81dc0b2350 con 0x7f81dc0c8f60 2024-01-11T08:25:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.638+0000 7f81e1fff700 1 -- 172.21.15.27:0/2791169595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81dc04c3c0 con 0x7f81dc0c8f60 2024-01-11T08:25:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.639+0000 7f81c37fe700 1 -- 172.21.15.27:0/2791169595 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 31) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f81cc023070 con 0x7f81dc0c8f60 2024-01-11T08:25:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.639+0000 7f81c37fe700 1 --2- 172.21.15.27:0/2791169595 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f81c8066f80 0x7f81c8069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.639+0000 7f81c37fe700 1 -- 172.21.15.27:0/2791169595 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f81cc085b30 con 0x7f81dc0c8f60 2024-01-11T08:25:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.639+0000 7f81daffd700 1 -- 172.21.15.27:0/2791169595 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f81c8066f80 msgr2=0x7f81c8069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.639+0000 7f81daffd700 1 --2- 172.21.15.27:0/2791169595 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f81c8066f80 0x7f81c8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:33.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.645+0000 7f81c37fe700 1 -- 172.21.15.27:0/2791169595 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f81cc052ce0 con 0x7f81dc0c8f60 2024-01-11T08:25:33.662 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:33.663+0000 7fb6fb0fe1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:25:33.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.841+0000 7f81daffd700 1 -- 172.21.15.27:0/2791169595 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f81c8066f80 msgr2=0x7f81c8069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:33.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.841+0000 7f81daffd700 1 --2- 172.21.15.27:0/2791169595 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f81c8066f80 0x7f81c8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.850+0000 7f81e1fff700 1 -- 172.21.15.27:0/2791169595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f81dc0c51c0 con 0x7f81dc0c8f60 2024-01-11T08:25:33.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.854+0000 7f81c37fe700 1 -- 172.21.15.27:0/2791169595 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7f81cc0150c0 con 0x7f81dc0c8f60 2024-01-11T08:25:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.856+0000 7f81e1fff700 1 -- 172.21.15.27:0/2791169595 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f81c8066f80 msgr2=0x7f81c8069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.856+0000 7f81e1fff700 1 --2- 172.21.15.27:0/2791169595 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f81c8066f80 0x7f81c8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.856+0000 7f81e1fff700 1 -- 172.21.15.27:0/2791169595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81dc0c8f60 msgr2=0x7f81dc156450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.856+0000 7f81e1fff700 1 --2- 172.21.15.27:0/2791169595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81dc0c8f60 0x7f81dc156450 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f81cc0094c0 tx=0x7f81cc017000 comp rx=0 tx=0).stop 2024-01-11T08:25:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.857+0000 7f81e1fff700 1 -- 172.21.15.27:0/2791169595 shutdown_connections 2024-01-11T08:25:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.857+0000 7f81e1fff700 1 --2- 172.21.15.27:0/2791169595 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f81c8066f80 0x7f81c8069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.857+0000 7f81e1fff700 1 --2- 172.21.15.27:0/2791169595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81dc141190 0x7f81dc0b16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.857+0000 7f81e1fff700 1 --2- 172.21.15.27:0/2791169595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81dc0c8f60 0x7f81dc156450 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.857+0000 7f81e1fff700 1 --2- 172.21.15.27:0/2791169595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f81dc140dc0 0x7f81dc0b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.857+0000 7f81e1fff700 1 -- 172.21.15.27:0/2791169595 >> 172.21.15.27:0/2791169595 conn(0x7f81dc063ca0 msgr2=0x7f81dc15b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.857+0000 7f81e1fff700 1 -- 172.21.15.27:0/2791169595 shutdown_connections 2024-01-11T08:25:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:33.857+0000 7f81e1fff700 1 -- 172.21.15.27:0/2791169595 wait complete. 2024-01-11T08:25:33.874 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:25:33.874 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-01-11T08:25:33.927 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-01-11T08:25:34.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.134+0000 7f731f1ff700 1 Processor -- start 2024-01-11T08:25:34.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.134+0000 7f731f1ff700 1 -- start start 2024-01-11T08:25:34.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.136+0000 7f731f1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7318142dc0 0x7f73180cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:34.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.136+0000 7f731f1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731815d320 con 0x7f7318143190 2024-01-11T08:25:34.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.136+0000 7f731f1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73180cb700 con 0x7f73180cb2e0 2024-01-11T08:25:34.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.136+0000 7f731f1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731815df30 con 0x7f7318142dc0 2024-01-11T08:25:34.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.136+0000 7f7317fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7318143190 0x7f73180cc020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36522/0 (socket says 172.21.15.27:36522) 2024-01-11T08:25:34.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.136+0000 7f7317fff700 1 -- 172.21.15.27:0/1727795991 learned_addr learned my addr 172.21.15.27:0/1727795991 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:34.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.136+0000 7f731cf9b700 1 --2- 172.21.15.27:0/1727795991 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7318142dc0 0x7f73180cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:34.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.137+0000 7f73177fe700 1 -- 172.21.15.27:0/1727795991 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365811786 0 0) 0x7f731815d320 con 0x7f7318143190 2024-01-11T08:25:34.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.137+0000 7f73177fe700 1 -- 172.21.15.27:0/1727795991 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7308003570 con 0x7f7318143190 2024-01-11T08:25:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.137+0000 7f73177fe700 1 -- 172.21.15.27:0/1727795991 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012032278 0 0) 0x7f73180cb700 con 0x7f73180cb2e0 2024-01-11T08:25:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.137+0000 7f73177fe700 1 -- 172.21.15.27:0/1727795991 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f731815d320 con 0x7f73180cb2e0 2024-01-11T08:25:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.137+0000 7f731cf9b700 1 -- 172.21.15.27:0/1727795991 >> v1:172.21.15.77:6789/0 conn(0x7f73180cb2e0 legacy=0x7f7318068720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.137+0000 7f731cf9b700 1 -- 172.21.15.27:0/1727795991 >> v1:172.21.15.27:6789/0 conn(0x7f7318143190 legacy=0x7f73180cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.137+0000 7f731cf9b700 1 -- 172.21.15.27:0/1727795991 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731815d590 con 0x7f7318142dc0 2024-01-11T08:25:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.137+0000 7f731cf9b700 1 --2- 172.21.15.27:0/1727795991 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7318142dc0 0x7f73180cada0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f730c009910 tx=0x7f730c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edd218c4c16b7c61 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.137+0000 7f73177fe700 1 -- 172.21.15.27:0/1727795991 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c011630 con 0x7f7318142dc0 2024-01-11T08:25:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.138+0000 7f73177fe700 1 -- 172.21.15.27:0/1727795991 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730c011790 con 0x7f7318142dc0 2024-01-11T08:25:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.138+0000 7f73177fe700 1 -- 172.21.15.27:0/1727795991 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c011b30 con 0x7f7318142dc0 2024-01-11T08:25:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.138+0000 7f731f1ff700 1 -- 172.21.15.27:0/1727795991 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7318142dc0 msgr2=0x7f73180cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.138+0000 7f731f1ff700 1 --2- 172.21.15.27:0/1727795991 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7318142dc0 0x7f73180cada0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f730c009910 tx=0x7f730c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.138+0000 7f731f1ff700 1 -- 172.21.15.27:0/1727795991 shutdown_connections 2024-01-11T08:25:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.138+0000 7f731f1ff700 1 --2- 172.21.15.27:0/1727795991 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7318142dc0 0x7f73180cada0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.138+0000 7f731f1ff700 1 -- 172.21.15.27:0/1727795991 >> 172.21.15.27:0/1727795991 conn(0x7f7318063ca0 msgr2=0x7f731806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.138+0000 7f731f1ff700 1 -- 172.21.15.27:0/1727795991 shutdown_connections 2024-01-11T08:25:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.138+0000 7f731f1ff700 1 -- 172.21.15.27:0/1727795991 wait complete. 2024-01-11T08:25:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.140+0000 7f731f1ff700 1 Processor -- start 2024-01-11T08:25:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.140+0000 7f731f1ff700 1 -- start start 2024-01-11T08:25:34.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.141+0000 7f731f1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73180cb2e0 0x7f731815a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:34.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.141+0000 7f731f1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7318142dc0 0x7f731813e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:34.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.141+0000 7f731f1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7318143190 0x7f7318169b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:34.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.141+0000 7f731f1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731815a690 con 0x7f7318142dc0 2024-01-11T08:25:34.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.141+0000 7f731f1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73180b1c30 con 0x7f73180cb2e0 2024-01-11T08:25:34.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.141+0000 7f731f1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73180b1180 con 0x7f7318143190 2024-01-11T08:25:34.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.141+0000 7f7317fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7318142dc0 0x7f731813e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:34.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.141+0000 7f7317fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7318142dc0 0x7f731813e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47706/0 (socket says 172.21.15.27:47706) 2024-01-11T08:25:34.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.141+0000 7f7317fff700 1 -- 172.21.15.27:0/2432587657 learned_addr learned my addr 172.21.15.27:0/2432587657 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:34.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.141+0000 7f731cf9b700 1 --2- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73180cb2e0 0x7f731815a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:34.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.141+0000 7f731d79c700 1 --2- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7318143190 0x7f7318169b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:34.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.142+0000 7f7317fff700 1 -- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7318143190 msgr2=0x7f7318169b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:34.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.142+0000 7f7317fff700 1 --2- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7318143190 0x7f7318169b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:34.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.142+0000 7f7317fff700 1 -- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73180cb2e0 msgr2=0x7f731815a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:34.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.142+0000 7f7317fff700 1 --2- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73180cb2e0 0x7f731815a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:34.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.142+0000 7f7317fff700 1 -- 172.21.15.27:0/2432587657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c009420 con 0x7f7318142dc0 2024-01-11T08:25:34.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.142+0000 7f731d79c700 1 --2- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7318143190 0x7f7318169b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:34.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.142+0000 7f7317fff700 1 --2- 172.21.15.27:0/2432587657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7318142dc0 0x7f731813e230 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f730400c790 tx=0x7f7304007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:34.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.142+0000 7f7314ff9700 1 -- 172.21.15.27:0/2432587657 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7304005b30 con 0x7f7318142dc0 2024-01-11T08:25:34.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.142+0000 7f7314ff9700 1 -- 172.21.15.27:0/2432587657 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7304005cd0 con 0x7f7318142dc0 2024-01-11T08:25:34.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.142+0000 7f7314ff9700 1 -- 172.21.15.27:0/2432587657 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7304007320 con 0x7f7318142dc0 2024-01-11T08:25:34.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.142+0000 7f731f1ff700 1 -- 172.21.15.27:0/2432587657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73180b1460 con 0x7f7318142dc0 2024-01-11T08:25:34.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.143+0000 7f731f1ff700 1 -- 172.21.15.27:0/2432587657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731813e770 con 0x7f7318142dc0 2024-01-11T08:25:34.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.144+0000 7f731f1ff700 1 -- 172.21.15.27:0/2432587657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f731804c3c0 con 0x7f7318142dc0 2024-01-11T08:25:34.143 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:34.144+0000 7f79e88fb1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:25:34.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.145+0000 7f7314ff9700 1 -- 172.21.15.27:0/2432587657 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 31) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f7304012400 con 0x7f7318142dc0 2024-01-11T08:25:34.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.146+0000 7f7314ff9700 1 --2- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f72fc066f80 0x7f72fc069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:34.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.146+0000 7f7314ff9700 1 -- 172.21.15.27:0/2432587657 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f73040838b0 con 0x7f7318142dc0 2024-01-11T08:25:34.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.146+0000 7f731cf9b700 1 -- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f72fc066f80 msgr2=0x7f72fc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:34.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.146+0000 7f731cf9b700 1 --2- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f72fc066f80 0x7f72fc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:34.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.150+0000 7f7314ff9700 1 -- 172.21.15.27:0/2432587657 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f730404b5a0 con 0x7f7318142dc0 2024-01-11T08:25:34.234 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:34.235+0000 7fb6fb0fe1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:25:34.314 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:34.314+0000 7f79e88fb1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:25:34.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.347+0000 7f731cf9b700 1 -- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f72fc066f80 msgr2=0x7f72fc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:34.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.347+0000 7f731cf9b700 1 --2- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f72fc066f80 0x7f72fc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:34.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.379+0000 7f731f1ff700 1 -- 172.21.15.27:0/2432587657 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f731815d8e0 con 0x7f72fc066f80 2024-01-11T08:25:34.415 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:34.416+0000 7fb6fb0fe1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:25:34.731 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:34.731+0000 7f79e88fb1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:25:34.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.748+0000 7f731cf9b700 1 -- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f72fc066f80 msgr2=0x7f72fc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:34.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:34.748+0000 7f731cf9b700 1 --2- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f72fc066f80 0x7f72fc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-11T08:25:34.854 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:34.855+0000 7fb6fb0fe1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:25:35.508 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:35.508+0000 7f79e88fb1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:25:35.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:35.549+0000 7f731cf9b700 1 -- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f72fc066f80 msgr2=0x7f72fc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:35.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:35.549+0000 7f731cf9b700 1 --2- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f72fc066f80 0x7f72fc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-11T08:25:35.596 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:35.597+0000 7f79e88fb1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:25:35.679 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:35.680+0000 7fb6fb0fe1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:25:35.753 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:35.753+0000 7f79e88fb1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:25:35.772 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:35.774+0000 7fb6fb0fe1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:25:35.841 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:35.841+0000 7f79e88fb1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:25:35.938 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:35.939+0000 7fb6fb0fe1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:25:36.031 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:36.032+0000 7fb6fb0fe1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:25:36.058 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:36.058+0000 7f79e88fb1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:25:36.153 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:36.153+0000 7f79e88fb1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:25:36.248 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:36.249+0000 7f79e88fb1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:25:36.257 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:36.258+0000 7fb6fb0fe1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:25:36.343 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:36.344+0000 7f79e88fb1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:25:36.358 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:36.359+0000 7fb6fb0fe1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:25:36.460 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:36.461+0000 7fb6fb0fe1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:25:36.531 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:36.532+0000 7f79e88fb1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:25:36.561 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:36.562+0000 7fb6fb0fe1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:25:36.628 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:36.628+0000 7f79e88fb1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:25:36.760 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:36.761+0000 7fb6fb0fe1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:25:36.864 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:36.865+0000 7fb6fb0fe1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:25:36.968 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:36.968+0000 7f79e88fb1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:25:37.057 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:37.057+0000 7f79e88fb1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:25:37.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:37.151+0000 7f731cf9b700 1 -- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f72fc066f80 msgr2=0x7f72fc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:37.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:37.151+0000 7f731cf9b700 1 --2- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f72fc066f80 0x7f72fc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-11T08:25:37.223 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:37.224+0000 7fb6fb0fe1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:25:37.317 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:37.318+0000 7fb6fb0fe1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:25:37.594 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:37.594+0000 7f79e88fb1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:25:37.778 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:37.779+0000 7f79e88fb1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:25:37.893 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:37.893+0000 7fb6fb0fe1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:25:38.089 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:38.090+0000 7fb6fb0fe1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:25:38.154 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:38.154+0000 7f79e88fb1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:25:38.483 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:38.484+0000 7fb6fb0fe1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:25:38.572 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:38.572+0000 7f79e88fb1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:25:38.752 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:38.752+0000 7f79e88fb1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:25:38.932 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:38.933+0000 7fb6fb0fe1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:25:38.938 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:38.939+0000 7f79e88fb1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:25:39.047 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:39.048+0000 7f79e88fb1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:25:39.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:39.058+0000 7f7314ff9700 1 -- 172.21.15.27:0/2432587657 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 32) v1 ==== 84470+0+0 (secure 0 0 0) 0x7f7304050f80 con 0x7f7318142dc0 2024-01-11T08:25:39.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:39.058+0000 7f7314ff9700 1 -- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f72fc066f80 msgr2=0x7f72fc069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:39.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:39.058+0000 7f7314ff9700 1 --2- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f72fc066f80 0x7f72fc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:39.132 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:39.133+0000 7fb6fb0fe1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:25:39.183 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:39.183+0000 7f7960f3f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:39.183 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:39.183+0000 7f7960f3f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:39.183 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:39.183+0000 7f7960f3f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:39.183 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:39.183+0000 7f7960f3f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:39.183 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:39.183+0000 7f7960f3f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:39.184 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:39.184+0000 7f795d738700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:39.184 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:39.184+0000 7f795d738700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:39.184 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:39.184+0000 7f795d738700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:39.184 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:39.184+0000 7f795d738700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:39.184 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:39.184+0000 7f795d738700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:39.335 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:39.336+0000 7fb6fb0fe1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:25:39.450 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:39.451+0000 7fb6fb0fe1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:25:40.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.062+0000 7f7314ff9700 1 -- 172.21.15.27:0/2432587657 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 33) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f7304053d60 con 0x7f7318142dc0 2024-01-11T08:25:40.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.062+0000 7f7314ff9700 1 --2- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f72fc066f80 0x7f72fc0693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:40.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.062+0000 7f7314ff9700 1 -- 172.21.15.27:0/2432587657 --> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f731815d8e0 con 0x7f72fc066f80 2024-01-11T08:25:40.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.063+0000 7f731cf9b700 1 --2- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f72fc066f80 0x7f72fc0693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:40.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.069+0000 7f731cf9b700 1 --2- 172.21.15.27:0/2432587657 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f72fc066f80 0x7f72fc0693b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f730c009860 tx=0x7f730c017000 comp rx=0 tx=0).ready entity=mgr.5735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:40.079 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:40.079+0000 7f7967f4d700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-01-11T08:25:40.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.080+0000 7f7314ff9700 1 -- 172.21.15.27:0/2432587657 <== mgr.5735 v2:172.21.15.77:6832/41381 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f731815d8e0 con 0x7f72fc066f80 2024-01-11T08:25:40.081 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-01-11T08:25:40.090 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-11T08:25:40.091 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:25:40.091 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-01-11T08:25:40.143 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-11T08:25:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.345+0000 7fdbe41ff700 1 Processor -- start 2024-01-11T08:25:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.345+0000 7fdbe41ff700 1 -- start start 2024-01-11T08:25:40.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.347+0000 7fdbe41ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdbdc0ccd70 0x7fdbdc0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:40.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.347+0000 7fdbe41ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbdc0cf850 con 0x7fdbdc15a1b0 2024-01-11T08:25:40.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.347+0000 7fdbe41ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbdc0cec40 con 0x7fdbdc15ae70 2024-01-11T08:25:40.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.347+0000 7fdbe41ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbdc15ac90 con 0x7fdbdc0ccd70 2024-01-11T08:25:40.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.347+0000 7fdbe1f9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdbdc15a1b0 0x7fdbdc15a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36526/0 (socket says 172.21.15.27:36526) 2024-01-11T08:25:40.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.347+0000 7fdbe1f9b700 1 -- 172.21.15.27:0/29453010 learned_addr learned my addr 172.21.15.27:0/29453010 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:40.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.347+0000 7fdbe179a700 1 --2- 172.21.15.27:0/29453010 >> v2:172.21.15.77:3300/0 conn(0x7fdbdc0ccd70 0x7fdbdc0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:40.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.347+0000 7fdbe0f99700 1 -- 172.21.15.27:0/29453010 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604307654 0 0) 0x7fdbdc0cf850 con 0x7fdbdc15a1b0 2024-01-11T08:25:40.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.347+0000 7fdbe0f99700 1 -- 172.21.15.27:0/29453010 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc8003570 con 0x7fdbdc15a1b0 2024-01-11T08:25:40.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.348+0000 7fdbe0f99700 1 -- 172.21.15.27:0/29453010 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524014028 0 0) 0x7fdbdc0cec40 con 0x7fdbdc15ae70 2024-01-11T08:25:40.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.348+0000 7fdbe0f99700 1 -- 172.21.15.27:0/29453010 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbdc0cf850 con 0x7fdbdc15ae70 2024-01-11T08:25:40.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.348+0000 7fdbe0f99700 1 -- 172.21.15.27:0/29453010 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1881825892 0 0) 0x7fdbc8003570 con 0x7fdbdc15a1b0 2024-01-11T08:25:40.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.348+0000 7fdbe0f99700 1 -- 172.21.15.27:0/29453010 >> v2:172.21.15.77:3300/0 conn(0x7fdbdc0ccd70 msgr2=0x7fdbdc0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:40.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.348+0000 7fdbe0f99700 1 --2- 172.21.15.27:0/29453010 >> v2:172.21.15.77:3300/0 conn(0x7fdbdc0ccd70 0x7fdbdc0cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:40.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.348+0000 7fdbe0f99700 1 -- 172.21.15.27:0/29453010 >> v1:172.21.15.77:6789/0 conn(0x7fdbdc15ae70 legacy=0x7fdbdc15b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:40.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.348+0000 7fdbe0f99700 1 -- 172.21.15.27:0/29453010 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbdc0cef20 con 0x7fdbdc15a1b0 2024-01-11T08:25:40.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.348+0000 7fdbe0f99700 1 -- 172.21.15.27:0/29453010 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fdbd8001470 con 0x7fdbdc15a1b0 2024-01-11T08:25:40.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.348+0000 7fdbe0f99700 1 -- 172.21.15.27:0/29453010 >> v1:172.21.15.27:6789/0 conn(0x7fdbdc15a1b0 legacy=0x7fdbdc15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:40.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.348+0000 7fdbe0f99700 1 --2- 172.21.15.27:0/29453010 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbc8003c40 0x7fdbc8004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:40.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.348+0000 7fdbe0f99700 1 --2- 172.21.15.27:0/29453010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbc8004810 0x7fdbc8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:40.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.348+0000 7fdbe0f99700 1 --2- 172.21.15.27:0/29453010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc8005210 0x7fdbc800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:40.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.348+0000 7fdbe0f99700 1 -- 172.21.15.27:0/29453010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc15ac90 con 0x7fdbc8005210 2024-01-11T08:25:40.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.348+0000 7fdbe0f99700 1 -- 172.21.15.27:0/29453010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc800c130 con 0x7fdbc8004810 2024-01-11T08:25:40.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.348+0000 7fdbe0f99700 1 -- 172.21.15.27:0/29453010 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc800c2c0 con 0x7fdbc8003c40 2024-01-11T08:25:40.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.348+0000 7fdbe1f9b700 1 --2- 172.21.15.27:0/29453010 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbc8003c40 0x7fdbc8004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:40.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.348+0000 7fdbe279c700 1 --2- 172.21.15.27:0/29453010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc8005210 0x7fdbc800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:40.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.348+0000 7fdbe179a700 1 --2- 172.21.15.27:0/29453010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbc8004810 0x7fdbc8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:40.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.349+0000 7fdbe279c700 1 -- 172.21.15.27:0/29453010 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbc8003c40 msgr2=0x7fdbc8004030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:40.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.349+0000 7fdbe279c700 1 --2- 172.21.15.27:0/29453010 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbc8003c40 0x7fdbc8004030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:40.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.349+0000 7fdbe279c700 1 -- 172.21.15.27:0/29453010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbc8004810 msgr2=0x7fdbc8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:40.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.349+0000 7fdbe279c700 1 --2- 172.21.15.27:0/29453010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbc8004810 0x7fdbc8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:40.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.349+0000 7fdbe279c700 1 -- 172.21.15.27:0/29453010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc8003570 con 0x7fdbc8005210 2024-01-11T08:25:40.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.349+0000 7fdbe279c700 1 -- 172.21.15.27:0/29453010 reap_dead start 2024-01-11T08:25:40.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.349+0000 7fdbe279c700 1 --2- 172.21.15.27:0/29453010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc8005210 0x7fdbc800ba60 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fdbd4002150 tx=0x7fdbd40017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0dbdff1c32aa69f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:40.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.349+0000 7fdbe0f99700 1 -- 172.21.15.27:0/29453010 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4001ab0 con 0x7fdbc8005210 2024-01-11T08:25:40.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.349+0000 7fdbe0f99700 1 -- 172.21.15.27:0/29453010 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd400c940 con 0x7fdbc8005210 2024-01-11T08:25:40.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.349+0000 7fdbe0f99700 1 -- 172.21.15.27:0/29453010 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd400ccc0 con 0x7fdbc8005210 2024-01-11T08:25:40.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.350+0000 7fdbe41ff700 1 -- 172.21.15.27:0/29453010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc8005210 msgr2=0x7fdbc800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:40.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.350+0000 7fdbe41ff700 1 --2- 172.21.15.27:0/29453010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc8005210 0x7fdbc800ba60 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fdbd4002150 tx=0x7fdbd40017a0 comp rx=0 tx=0).stop 2024-01-11T08:25:40.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.350+0000 7fdbe41ff700 1 -- 172.21.15.27:0/29453010 shutdown_connections 2024-01-11T08:25:40.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.350+0000 7fdbe41ff700 1 --2- 172.21.15.27:0/29453010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc8005210 0x7fdbc800ba60 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:40.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.350+0000 7fdbe41ff700 1 -- 172.21.15.27:0/29453010 >> 172.21.15.27:0/29453010 conn(0x7fdbdc06ed40 msgr2=0x7fdbdc06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:40.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.350+0000 7fdbe41ff700 1 -- 172.21.15.27:0/29453010 shutdown_connections 2024-01-11T08:25:40.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.350+0000 7fdbe41ff700 1 -- 172.21.15.27:0/29453010 wait complete. 2024-01-11T08:25:40.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.351+0000 7fdbe41ff700 1 Processor -- start 2024-01-11T08:25:40.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.352+0000 7fdbe41ff700 1 -- start start 2024-01-11T08:25:40.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.353+0000 7fdbe41ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbc8005210 0x7fdbdc145f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:40.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.353+0000 7fdbe41ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbdc1549d0 0x7fdbdc154e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:40.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.353+0000 7fdbe41ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbdc0c90c0 0x7fdbdc0c9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:40.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.353+0000 7fdbe41ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc155380 con 0x7fdbdc0c90c0 2024-01-11T08:25:40.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.353+0000 7fdbe41ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc0c7830 con 0x7fdbc8005210 2024-01-11T08:25:40.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.353+0000 7fdbe41ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc0c7ad0 con 0x7fdbdc1549d0 2024-01-11T08:25:40.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.353+0000 7fdbe279c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbdc0c90c0 0x7fdbdc0c9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:40.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.353+0000 7fdbe279c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbdc0c90c0 0x7fdbdc0c9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47728/0 (socket says 172.21.15.27:47728) 2024-01-11T08:25:40.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.353+0000 7fdbe279c700 1 -- 172.21.15.27:0/2861816011 learned_addr learned my addr 172.21.15.27:0/2861816011 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:40.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.353+0000 7fdbe1f9b700 1 --2- 172.21.15.27:0/2861816011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbc8005210 0x7fdbdc145f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:40.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.353+0000 7fdbe179a700 1 --2- 172.21.15.27:0/2861816011 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbdc1549d0 0x7fdbdc154e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:40.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.353+0000 7fdbe279c700 1 -- 172.21.15.27:0/2861816011 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbdc1549d0 msgr2=0x7fdbdc154e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:40.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.353+0000 7fdbe279c700 1 --2- 172.21.15.27:0/2861816011 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbdc1549d0 0x7fdbdc154e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:40.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.353+0000 7fdbe279c700 1 -- 172.21.15.27:0/2861816011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbc8005210 msgr2=0x7fdbdc145f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:40.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.353+0000 7fdbe279c700 1 --2- 172.21.15.27:0/2861816011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbc8005210 0x7fdbdc145f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:40.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.353+0000 7fdbe279c700 1 -- 172.21.15.27:0/2861816011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbdc0c7db0 con 0x7fdbdc0c90c0 2024-01-11T08:25:40.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.353+0000 7fdbe1f9b700 1 --2- 172.21.15.27:0/2861816011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbc8005210 0x7fdbdc145f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:40.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.353+0000 7fdbe179a700 1 --2- 172.21.15.27:0/2861816011 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbdc1549d0 0x7fdbdc154e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:40.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.354+0000 7fdbe279c700 1 --2- 172.21.15.27:0/2861816011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbdc0c90c0 0x7fdbdc0c9540 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc06fd20 tx=0x7fdbd4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:40.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.354+0000 7fdbd1ffb700 1 -- 172.21.15.27:0/2861816011 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4001ab0 con 0x7fdbdc0c90c0 2024-01-11T08:25:40.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.354+0000 7fdbd1ffb700 1 -- 172.21.15.27:0/2861816011 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd4006db0 con 0x7fdbdc0c90c0 2024-01-11T08:25:40.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.354+0000 7fdbd1ffb700 1 -- 172.21.15.27:0/2861816011 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd401e680 con 0x7fdbdc0c90c0 2024-01-11T08:25:40.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.354+0000 7fdbe41ff700 1 -- 172.21.15.27:0/2861816011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbdc0c8090 con 0x7fdbdc0c90c0 2024-01-11T08:25:40.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.354+0000 7fdbe41ff700 1 -- 172.21.15.27:0/2861816011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbdc0c9a80 con 0x7fdbdc0c90c0 2024-01-11T08:25:40.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.355+0000 7fdbe41ff700 1 -- 172.21.15.27:0/2861816011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbdc0c85b0 con 0x7fdbdc0c90c0 2024-01-11T08:25:40.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.356+0000 7fdbd1ffb700 1 -- 172.21.15.27:0/2861816011 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 33) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fdbd4003030 con 0x7fdbdc0c90c0 2024-01-11T08:25:40.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.357+0000 7fdbd1ffb700 1 --2- 172.21.15.27:0/2861816011 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdbc0066dd0 0x7fdbc0069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:40.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.357+0000 7fdbd1ffb700 1 -- 172.21.15.27:0/2861816011 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5991+0+0 (secure 0 0 0) 0x7fdbd40861d0 con 0x7fdbdc0c90c0 2024-01-11T08:25:40.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.357+0000 7fdbe1f9b700 1 --2- 172.21.15.27:0/2861816011 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdbc0066dd0 0x7fdbc0069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:40.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.358+0000 7fdbe1f9b700 1 --2- 172.21.15.27:0/2861816011 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdbc0066dd0 0x7fdbc0069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8000e10 tx=0x7fdbd8009040 comp rx=0 tx=0).ready entity=mgr.5735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:40.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.361+0000 7fdbd1ffb700 1 -- 172.21.15.27:0/2861816011 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdbdc0c85b0 con 0x7fdbdc0c90c0 2024-01-11T08:25:40.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:40.623+0000 7fdbe41ff700 1 -- 172.21.15.27:0/2861816011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fdbdc0c85b0 con 0x7fdbdc0c90c0 2024-01-11T08:25:41.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.115+0000 7fdbd1ffb700 1 -- 172.21.15.27:0/2861816011 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fdbd40538d0 con 0x7fdbdc0c90c0 2024-01-11T08:25:41.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.116+0000 7fdbd1ffb700 1 -- 172.21.15.27:0/2861816011 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v34) v1 ==== 93+0+0 (secure 0 0 0) 0x7fdbd4056b20 con 0x7fdbdc0c90c0 2024-01-11T08:25:41.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.119+0000 7fdbe41ff700 1 -- 172.21.15.27:0/2861816011 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdbc0066dd0 msgr2=0x7fdbc0069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:41.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.119+0000 7fdbe41ff700 1 --2- 172.21.15.27:0/2861816011 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdbc0066dd0 0x7fdbc0069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8000e10 tx=0x7fdbd8009040 comp rx=0 tx=0).stop 2024-01-11T08:25:41.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.119+0000 7fdbe41ff700 1 -- 172.21.15.27:0/2861816011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbdc0c90c0 msgr2=0x7fdbdc0c9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:41.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.119+0000 7fdbe41ff700 1 --2- 172.21.15.27:0/2861816011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbdc0c90c0 0x7fdbdc0c9540 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc06fd20 tx=0x7fdbd4016000 comp rx=0 tx=0).stop 2024-01-11T08:25:41.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.119+0000 7fdbe41ff700 1 -- 172.21.15.27:0/2861816011 shutdown_connections 2024-01-11T08:25:41.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.119+0000 7fdbe41ff700 1 --2- 172.21.15.27:0/2861816011 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fdbc0066dd0 0x7fdbc0069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:41.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.119+0000 7fdbe41ff700 1 --2- 172.21.15.27:0/2861816011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbdc0c90c0 0x7fdbdc0c9540 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:41.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.119+0000 7fdbe41ff700 1 --2- 172.21.15.27:0/2861816011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbc8005210 0x7fdbdc145f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:41.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.119+0000 7fdbe41ff700 1 --2- 172.21.15.27:0/2861816011 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdbdc1549d0 0x7fdbdc154e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:41.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.119+0000 7fdbe41ff700 1 -- 172.21.15.27:0/2861816011 >> 172.21.15.27:0/2861816011 conn(0x7fdbdc06ed40 msgr2=0x7fdbdc156080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:41.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.120+0000 7fdbe41ff700 1 -- 172.21.15.27:0/2861816011 shutdown_connections 2024-01-11T08:25:41.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.120+0000 7fdbe41ff700 1 -- 172.21.15.27:0/2861816011 wait complete. 2024-01-11T08:25:41.139 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T08:25:41.139 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:25:41.243 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:25:41.331 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T08:25:41.331 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-11T08:25:41.332 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:25:41.332 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-01-11T08:25:41.332 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:41.333+0000 7ff67d0f81c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:25:41.340 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:41.341+0000 7f01ec3381c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:25:41.416 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T08:25:41.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:25:41.416 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-11T08:25:41.484 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:25:41.521 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:41.522+0000 7f01ec3381c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:25:41.533 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:41.534+0000 7ff67d0f81c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:25:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.704+0000 7f8bbf1ff700 1 Processor -- start 2024-01-11T08:25:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.705+0000 7f8bbf1ff700 1 -- start start 2024-01-11T08:25:41.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.706+0000 7f8bbf1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8bb8159890 0x7f8bb80af320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:41.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.706+0000 7f8bbf1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8bb80ac370 0x7f8bb80aa660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:41.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.706+0000 7f8bbf1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb80aa1d0 con 0x7f8bb80ad020 2024-01-11T08:25:41.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.706+0000 7f8bbf1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bb80af860 con 0x7f8bb8159890 2024-01-11T08:25:41.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.706+0000 7f8bbf1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb80ac8d0 con 0x7f8bb80ac370 2024-01-11T08:25:41.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.706+0000 7f8bb7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8bb8159890 0x7f8bb80af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:41.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.706+0000 7f8bb7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8bb8159890 0x7f8bb80af320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47750/0 (socket says 172.21.15.27:47750) 2024-01-11T08:25:41.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.707+0000 7f8bb7fff700 1 -- 172.21.15.27:0/4185917839 learned_addr learned my addr 172.21.15.27:0/4185917839 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:41.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.707+0000 7f8bbcf9b700 1 --1- 172.21.15.27:0/4185917839 >> v1:172.21.15.27:6789/0 conn(0x7f8bb80ad020 0x7f8bb8159180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36550/0 (socket says 172.21.15.27:36550) 2024-01-11T08:25:41.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.707+0000 7f8bbd79c700 1 --2- 172.21.15.27:0/4185917839 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8bb80ac370 0x7f8bb80aa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:41.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.707+0000 7f8bb7fff700 1 -- 172.21.15.27:0/4185917839 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8bb80ac370 msgr2=0x7f8bb80aa660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:41.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.707+0000 7f8bb7fff700 1 --2- 172.21.15.27:0/4185917839 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8bb80ac370 0x7f8bb80aa660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:41.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.707+0000 7f8bb7fff700 1 -- 172.21.15.27:0/4185917839 >> v1:172.21.15.27:6789/0 conn(0x7f8bb80ad020 legacy=0x7f8bb8159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:41.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.707+0000 7f8bb7fff700 1 -- 172.21.15.27:0/4185917839 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb80a98a0 con 0x7f8bb8159890 2024-01-11T08:25:41.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.707+0000 7f8bb77fe700 1 -- 172.21.15.27:0/4185917839 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56353536 0 0) 0x7f8bb80aa1d0 con 0x7f8bb80ad020 2024-01-11T08:25:41.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.707+0000 7f8bb7fff700 1 --2- 172.21.15.27:0/4185917839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb8159890 0x7f8bb80af320 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40099b0 tx=0x7f8ba4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f14cb68a03e116 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:41.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.708+0000 7f8bb77fe700 1 -- 172.21.15.27:0/4185917839 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4009ea0 con 0x7f8bb8159890 2024-01-11T08:25:41.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.708+0000 7f8bb77fe700 1 -- 172.21.15.27:0/4185917839 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba400b860 con 0x7f8bb8159890 2024-01-11T08:25:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.708+0000 7f8bb77fe700 1 -- 172.21.15.27:0/4185917839 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba400bc00 con 0x7f8bb8159890 2024-01-11T08:25:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.708+0000 7f8bbf1ff700 1 -- 172.21.15.27:0/4185917839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb8159890 msgr2=0x7f8bb80af320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.708+0000 7f8bbf1ff700 1 --2- 172.21.15.27:0/4185917839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb8159890 0x7f8bb80af320 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40099b0 tx=0x7f8ba4009250 comp rx=0 tx=0).stop 2024-01-11T08:25:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.708+0000 7f8bbf1ff700 1 -- 172.21.15.27:0/4185917839 shutdown_connections 2024-01-11T08:25:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.708+0000 7f8bbf1ff700 1 --2- 172.21.15.27:0/4185917839 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8bb80ac370 0x7f8bb80aa660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.708+0000 7f8bbf1ff700 1 --2- 172.21.15.27:0/4185917839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb8159890 0x7f8bb80af320 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.708+0000 7f8bbf1ff700 1 -- 172.21.15.27:0/4185917839 >> 172.21.15.27:0/4185917839 conn(0x7f8bb80680e0 msgr2=0x7f8bb8069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.708+0000 7f8bbf1ff700 1 -- 172.21.15.27:0/4185917839 shutdown_connections 2024-01-11T08:25:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.708+0000 7f8bbf1ff700 1 -- 172.21.15.27:0/4185917839 wait complete. 2024-01-11T08:25:41.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.710+0000 7f8bbf1ff700 1 Processor -- start 2024-01-11T08:25:41.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.710+0000 7f8bbf1ff700 1 -- start start 2024-01-11T08:25:41.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.711+0000 7f8bbf1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bb80ac370 0x7f8bb816a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.711+0000 7f8bbf1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8bb80ad020 0x7f8bb81567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.711+0000 7f8bbf1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb8159890 0x7f8bb815d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.711+0000 7f8bbf1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb815e630 con 0x7f8bb8159890 2024-01-11T08:25:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.711+0000 7f8bbf1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb815d7a0 con 0x7f8bb80ac370 2024-01-11T08:25:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.711+0000 7f8bbf1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb815da20 con 0x7f8bb80ad020 2024-01-11T08:25:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.711+0000 7f8bbd79c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb8159890 0x7f8bb815d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.712+0000 7f8bb7fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8bb80ad020 0x7f8bb81567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.712+0000 7f8bbd79c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb8159890 0x7f8bb815d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47754/0 (socket says 172.21.15.27:47754) 2024-01-11T08:25:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.712+0000 7f8bbd79c700 1 -- 172.21.15.27:0/183507616 learned_addr learned my addr 172.21.15.27:0/183507616 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.712+0000 7f8bbcf9b700 1 --2- 172.21.15.27:0/183507616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bb80ac370 0x7f8bb816a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.712+0000 7f8bbd79c700 1 -- 172.21.15.27:0/183507616 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8bb80ad020 msgr2=0x7f8bb81567e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:41.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.712+0000 7f8bbd79c700 1 --2- 172.21.15.27:0/183507616 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8bb80ad020 0x7f8bb81567e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:41.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.712+0000 7f8bbd79c700 1 -- 172.21.15.27:0/183507616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bb80ac370 msgr2=0x7f8bb816a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:41.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.712+0000 7f8bbd79c700 1 --2- 172.21.15.27:0/183507616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bb80ac370 0x7f8bb816a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:41.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.712+0000 7f8bbd79c700 1 -- 172.21.15.27:0/183507616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba40094f0 con 0x7f8bb8159890 2024-01-11T08:25:41.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.712+0000 7f8bbd79c700 1 --2- 172.21.15.27:0/183507616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb8159890 0x7f8bb815d260 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0009e90 tx=0x7f8bb000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:41.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.712+0000 7f8bb4ff9700 1 -- 172.21.15.27:0/183507616 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb000cb70 con 0x7f8bb8159890 2024-01-11T08:25:41.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.712+0000 7f8bb4ff9700 1 -- 172.21.15.27:0/183507616 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb000f040 con 0x7f8bb8159890 2024-01-11T08:25:41.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.712+0000 7f8bbf1ff700 1 -- 172.21.15.27:0/183507616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb815dd00 con 0x7f8bb8159890 2024-01-11T08:25:41.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.713+0000 7f8bb4ff9700 1 -- 172.21.15.27:0/183507616 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb0014480 con 0x7f8bb8159890 2024-01-11T08:25:41.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.713+0000 7f8bbf1ff700 1 -- 172.21.15.27:0/183507616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb815e2c0 con 0x7f8bb8159890 2024-01-11T08:25:41.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.714+0000 7f8bbf1ff700 1 -- 172.21.15.27:0/183507616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bb80718d0 con 0x7f8bb8159890 2024-01-11T08:25:41.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.715+0000 7f8bb4ff9700 1 -- 172.21.15.27:0/183507616 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f8bb0014830 con 0x7f8bb8159890 2024-01-11T08:25:41.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.716+0000 7f8bb4ff9700 1 --2- 172.21.15.27:0/183507616 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8b9c067050 0x7f8b9c069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:41.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.716+0000 7f8bb4ff9700 1 -- 172.21.15.27:0/183507616 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5991+0+0 (secure 0 0 0) 0x7f8bb00852a0 con 0x7f8bb8159890 2024-01-11T08:25:41.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.716+0000 7f8bbcf9b700 1 -- 172.21.15.27:0/183507616 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8b9c067050 msgr2=0x7f8b9c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:41.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.716+0000 7f8bbcf9b700 1 --2- 172.21.15.27:0/183507616 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8b9c067050 0x7f8b9c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:41.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.721+0000 7f8bb4ff9700 1 -- 172.21.15.27:0/183507616 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8bb00521d0 con 0x7f8bb8159890 2024-01-11T08:25:41.836 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:41.836+0000 7f01ec3381c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:25:41.878 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:41.879+0000 7ff67d0f81c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:25:41.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.917+0000 7f8bbcf9b700 1 -- 172.21.15.27:0/183507616 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8b9c067050 msgr2=0x7f8b9c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:41.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.917+0000 7f8bbcf9b700 1 --2- 172.21.15.27:0/183507616 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8b9c067050 0x7f8b9c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:41.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.973+0000 7f8bbf1ff700 1 -- 172.21.15.27:0/183507616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8bb80a9b80 con 0x7f8bb8159890 2024-01-11T08:25:41.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.973+0000 7f8bb4ff9700 1 -- 172.21.15.27:0/183507616 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4597 (secure 0 0 0) 0x7f8bb00557f0 con 0x7f8bb8159890 2024-01-11T08:25:41.972 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:41.973 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5606,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2845124438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2845124438},{"type":"v1","addr":"172.21.15.77:6837","nonce":2845124438}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5618,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3958104902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3958104902},{"type":"v1","addr":"172.21.15.27:6835","nonce":3958104902}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":6091,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/2612413022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2612413022},{"type":"v1","addr":"172.21.15.27:6833","nonce":2612413022}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:25:23.345532+0000","modified":"2024-01-11T08:25:26.867139+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6174},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6174":{"gid":6174,"name":"d","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6836/3056318454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3056318454},{"type":"v1","addr":"172.21.15.77:6836","nonce":3056318454}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2024-01-11T08:25:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.976+0000 7f8bbf1ff700 1 -- 172.21.15.27:0/183507616 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8b9c067050 msgr2=0x7f8b9c069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.976+0000 7f8bbf1ff700 1 --2- 172.21.15.27:0/183507616 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8b9c067050 0x7f8b9c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.976+0000 7f8bbf1ff700 1 -- 172.21.15.27:0/183507616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb8159890 msgr2=0x7f8bb815d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.976+0000 7f8bbf1ff700 1 --2- 172.21.15.27:0/183507616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb8159890 0x7f8bb815d260 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0009e90 tx=0x7f8bb000b040 comp rx=0 tx=0).stop 2024-01-11T08:25:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.976+0000 7f8bbf1ff700 1 -- 172.21.15.27:0/183507616 shutdown_connections 2024-01-11T08:25:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.976+0000 7f8bbf1ff700 1 --2- 172.21.15.27:0/183507616 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f8b9c067050 0x7f8b9c069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.976+0000 7f8bbf1ff700 1 --2- 172.21.15.27:0/183507616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb8159890 0x7f8bb815d260 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.976+0000 7f8bbf1ff700 1 --2- 172.21.15.27:0/183507616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bb80ac370 0x7f8bb816a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.976+0000 7f8bbf1ff700 1 --2- 172.21.15.27:0/183507616 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8bb80ad020 0x7f8bb81567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.976+0000 7f8bbf1ff700 1 -- 172.21.15.27:0/183507616 >> 172.21.15.27:0/183507616 conn(0x7f8bb80680e0 msgr2=0x7f8bb80afac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.976+0000 7f8bbf1ff700 1 -- 172.21.15.27:0/183507616 shutdown_connections 2024-01-11T08:25:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:41.976+0000 7f8bbf1ff700 1 -- 172.21.15.27:0/183507616 wait complete. 2024-01-11T08:25:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 33 2024-01-11T08:25:41.994 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:25:42.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.242+0000 7fb0b59ff700 1 Processor -- start 2024-01-11T08:25:42.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.242+0000 7fb0b59ff700 1 -- start start 2024-01-11T08:25:42.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.243+0000 7fb0b59ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0b01575e0 0x7fb0b0158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:42.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.243+0000 7fb0b59ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb0b006b610 0x7fb0b00b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:42.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.243+0000 7fb0b59ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b0158b60 con 0x7fb0b00b1800 2024-01-11T08:25:42.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.243+0000 7fb0b59ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b00cb5e0 con 0x7fb0b01575e0 2024-01-11T08:25:42.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.243+0000 7fb0b59ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b01599b0 con 0x7fb0b006b610 2024-01-11T08:25:42.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.244+0000 7fb0aeffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0b01575e0 0x7fb0b0158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:42.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.244+0000 7fb0af7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb0b00b1800 0x7fb0b00ca1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36564/0 (socket says 172.21.15.27:36564) 2024-01-11T08:25:42.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.244+0000 7fb0af7fe700 1 -- 172.21.15.27:0/3520677106 learned_addr learned my addr 172.21.15.27:0/3520677106 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:42.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.244+0000 7fb0ae7fc700 1 --2- 172.21.15.27:0/3520677106 >> v2:172.21.15.77:3300/0 conn(0x7fb0b006b610 0x7fb0b00b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:42.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.244+0000 7fb0aeffd700 1 -- 172.21.15.27:0/3520677106 >> v2:172.21.15.77:3300/0 conn(0x7fb0b006b610 msgr2=0x7fb0b00b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:42.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.244+0000 7fb0aeffd700 1 --2- 172.21.15.27:0/3520677106 >> v2:172.21.15.77:3300/0 conn(0x7fb0b006b610 0x7fb0b00b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.244+0000 7fb0aeffd700 1 -- 172.21.15.27:0/3520677106 >> v1:172.21.15.27:6789/0 conn(0x7fb0b00b1800 legacy=0x7fb0b00ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:42.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.244+0000 7fb0adffb700 1 -- 172.21.15.27:0/3520677106 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129554294 0 0) 0x7fb0a8001490 con 0x7fb0b00b1800 2024-01-11T08:25:42.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.244+0000 7fb0aeffd700 1 -- 172.21.15.27:0/3520677106 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b00b1c10 con 0x7fb0b01575e0 2024-01-11T08:25:42.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.245+0000 7fb0aeffd700 1 --2- 172.21.15.27:0/3520677106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0b01575e0 0x7fb0b0158620 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0064fe0 tx=0x7fb0a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb8f6f0d1d4dd694 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:42.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.245+0000 7fb0adffb700 1 -- 172.21.15.27:0/3520677106 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0011630 con 0x7fb0b01575e0 2024-01-11T08:25:42.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.245+0000 7fb0adffb700 1 -- 172.21.15.27:0/3520677106 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a0011790 con 0x7fb0b01575e0 2024-01-11T08:25:42.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.245+0000 7fb0adffb700 1 -- 172.21.15.27:0/3520677106 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0011b30 con 0x7fb0b01575e0 2024-01-11T08:25:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.246+0000 7fb0b59ff700 1 -- 172.21.15.27:0/3520677106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0b01575e0 msgr2=0x7fb0b0158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.246+0000 7fb0b59ff700 1 --2- 172.21.15.27:0/3520677106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0b01575e0 0x7fb0b0158620 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0064fe0 tx=0x7fb0a00091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.246+0000 7fb0b59ff700 1 -- 172.21.15.27:0/3520677106 shutdown_connections 2024-01-11T08:25:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.246+0000 7fb0b59ff700 1 --2- 172.21.15.27:0/3520677106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0b01575e0 0x7fb0b0158620 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.246+0000 7fb0b59ff700 1 --2- 172.21.15.27:0/3520677106 >> v2:172.21.15.77:3300/0 conn(0x7fb0b006b610 0x7fb0b00b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.246+0000 7fb0b59ff700 1 -- 172.21.15.27:0/3520677106 >> 172.21.15.27:0/3520677106 conn(0x7fb0b00680e0 msgr2=0x7fb0b0069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.246+0000 7fb0b59ff700 1 -- 172.21.15.27:0/3520677106 shutdown_connections 2024-01-11T08:25:42.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.246+0000 7fb0b59ff700 1 -- 172.21.15.27:0/3520677106 wait complete. 2024-01-11T08:25:42.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.248+0000 7fb0b59ff700 1 Processor -- start 2024-01-11T08:25:42.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.248+0000 7fb0b59ff700 1 -- start start 2024-01-11T08:25:42.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.250+0000 7fb0b59ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0b006b610 0x7fb0b015b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:42.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.250+0000 7fb0b59ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0b00b1800 0x7fb0b015bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:42.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.250+0000 7fb0b59ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0b01575e0 0x7fb0b015c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:42.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.250+0000 7fb0b59ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b006f0e0 con 0x7fb0b00b1800 2024-01-11T08:25:42.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.250+0000 7fb0b59ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b015b330 con 0x7fb0b01575e0 2024-01-11T08:25:42.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.250+0000 7fb0b59ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b006e840 con 0x7fb0b006b610 2024-01-11T08:25:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.250+0000 7fb0aeffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0b006b610 0x7fb0b015b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.250+0000 7fb0ae7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0b00b1800 0x7fb0b015bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.250+0000 7fb0aeffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0b006b610 0x7fb0b015b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:47308/0 (socket says 172.21.15.27:47308) 2024-01-11T08:25:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.250+0000 7fb0aeffd700 1 -- 172.21.15.27:0/4153495358 learned_addr learned my addr 172.21.15.27:0/4153495358 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.250+0000 7fb0af7fe700 1 --2- 172.21.15.27:0/4153495358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0b01575e0 0x7fb0b015c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.251+0000 7fb0aeffd700 1 -- 172.21.15.27:0/4153495358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0b01575e0 msgr2=0x7fb0b015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.251+0000 7fb0aeffd700 1 --2- 172.21.15.27:0/4153495358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0b01575e0 0x7fb0b015c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.251+0000 7fb0aeffd700 1 -- 172.21.15.27:0/4153495358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0b00b1800 msgr2=0x7fb0b015bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.251+0000 7fb0aeffd700 1 --2- 172.21.15.27:0/4153495358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0b00b1800 0x7fb0b015bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.251+0000 7fb0aeffd700 1 -- 172.21.15.27:0/4153495358 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a0009420 con 0x7fb0b006b610 2024-01-11T08:25:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.251+0000 7fb0aeffd700 1 --2- 172.21.15.27:0/4153495358 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0b006b610 0x7fb0b015b7c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0066180 tx=0x7fb0a0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.252+0000 7fb096ffd700 1 -- 172.21.15.27:0/4153495358 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0000d10 con 0x7fb0b006b610 2024-01-11T08:25:42.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.252+0000 7fb096ffd700 1 -- 172.21.15.27:0/4153495358 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a0011900 con 0x7fb0b006b610 2024-01-11T08:25:42.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.252+0000 7fb096ffd700 1 -- 172.21.15.27:0/4153495358 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0025420 con 0x7fb0b006b610 2024-01-11T08:25:42.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.252+0000 7fb0b59ff700 1 -- 172.21.15.27:0/4153495358 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b006eb20 con 0x7fb0b006b610 2024-01-11T08:25:42.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.252+0000 7fb0b59ff700 1 -- 172.21.15.27:0/4153495358 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b0147df0 con 0x7fb0b006b610 2024-01-11T08:25:42.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.253+0000 7fb0b59ff700 1 -- 172.21.15.27:0/4153495358 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b00718d0 con 0x7fb0b006b610 2024-01-11T08:25:42.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.254+0000 7fb096ffd700 1 -- 172.21.15.27:0/4153495358 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fb0a0023070 con 0x7fb0b006b610 2024-01-11T08:25:42.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.255+0000 7fb096ffd700 1 --2- 172.21.15.27:0/4153495358 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb09c067050 0x7fb09c069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:42.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.255+0000 7fb096ffd700 1 -- 172.21.15.27:0/4153495358 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5991+0+0 (secure 0 0 0) 0x7fb0a0085c20 con 0x7fb0b006b610 2024-01-11T08:25:42.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.255+0000 7fb0ae7fc700 1 -- 172.21.15.27:0/4153495358 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb09c067050 msgr2=0x7fb09c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:42.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.255+0000 7fb0ae7fc700 1 --2- 172.21.15.27:0/4153495358 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb09c067050 0x7fb09c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:42.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.259+0000 7fb096ffd700 1 -- 172.21.15.27:0/4153495358 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb0a0052bc0 con 0x7fb0b006b610 2024-01-11T08:25:42.361 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:42.362+0000 7f01ec3381c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:25:42.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.456+0000 7fb0ae7fc700 1 -- 172.21.15.27:0/4153495358 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb09c067050 msgr2=0x7fb09c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:42.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.456+0000 7fb0ae7fc700 1 --2- 172.21.15.27:0/4153495358 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb09c067050 0x7fb09c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:42.470 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:42.472+0000 7ff67d0f81c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:25:42.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.490+0000 7fb0b59ff700 1 -- 172.21.15.27:0/4153495358 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0b0069b20 con 0x7fb0b006b610 2024-01-11T08:25:42.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.491+0000 7fb096ffd700 1 -- 172.21.15.27:0/4153495358 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+15710 (secure 0 0 0) 0x7fb0a00561e0 con 0x7fb0b006b610 2024-01-11T08:25:42.490 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:42.491 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":76,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:25:39.051951+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:25:21.160295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-11T08:25:22.342780+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/32753357":"2024-01-12T08:25:39.051906+0000","172.21.15.77:6832/41381":"2024-01-12T08:25:39.051906+0000","172.21.15.77:0/3311724933":"2024-01-12T08:25:39.051906+0000","172.21.15.77:6833/41381":"2024-01-12T08:25:39.051906+0000","172.21.15.77:0/2372882029":"2024-01-12T08:25:39.051906+0000","172.21.15.77:0/3065749352":"2024-01-12T08:25:39.051906+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:25:42.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.494+0000 7fb0b59ff700 1 -- 172.21.15.27:0/4153495358 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb09c067050 msgr2=0x7fb09c069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:42.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.494+0000 7fb0b59ff700 1 --2- 172.21.15.27:0/4153495358 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb09c067050 0x7fb09c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.494+0000 7fb0b59ff700 1 -- 172.21.15.27:0/4153495358 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0b006b610 msgr2=0x7fb0b015b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:42.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.494+0000 7fb0b59ff700 1 --2- 172.21.15.27:0/4153495358 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0b006b610 0x7fb0b015b7c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0066180 tx=0x7fb0a0017000 comp rx=0 tx=0).stop 2024-01-11T08:25:42.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.494+0000 7fb0b59ff700 1 -- 172.21.15.27:0/4153495358 shutdown_connections 2024-01-11T08:25:42.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.494+0000 7fb0b59ff700 1 --2- 172.21.15.27:0/4153495358 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb09c067050 0x7fb09c069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.494+0000 7fb0b59ff700 1 --2- 172.21.15.27:0/4153495358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0b01575e0 0x7fb0b015c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.494+0000 7fb0b59ff700 1 --2- 172.21.15.27:0/4153495358 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0b006b610 0x7fb0b015b7c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.494+0000 7fb0b59ff700 1 --2- 172.21.15.27:0/4153495358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0b00b1800 0x7fb0b015bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.494+0000 7fb0b59ff700 1 -- 172.21.15.27:0/4153495358 >> 172.21.15.27:0/4153495358 conn(0x7fb0b00680e0 msgr2=0x7fb0b0153b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:42.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.494+0000 7fb0b59ff700 1 -- 172.21.15.27:0/4153495358 shutdown_connections 2024-01-11T08:25:42.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.494+0000 7fb0b59ff700 1 -- 172.21.15.27:0/4153495358 wait complete. 2024-01-11T08:25:42.512 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T08:25:21.160295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T08:25:22.342780+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T08:25:42.512 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:25:42.528 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:42.529+0000 7f01ec3381c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:25:42.658 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:42.658+0000 7ff67d0f81c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:25:42.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.721+0000 7f580cbff700 1 Processor -- start 2024-01-11T08:25:42.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.721+0000 7f580cbff700 1 -- start start 2024-01-11T08:25:42.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.722+0000 7f580cbff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f580815a1b0 0x7f580815a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:42.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.722+0000 7f580cbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f58080ccd70 0x7f58080cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:42.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.722+0000 7f580cbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f58080685c0 0x7f5808068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:42.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.722+0000 7f580cbff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58080cf850 con 0x7f580815a1b0 2024-01-11T08:25:42.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.722+0000 7f580cbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58080cec40 con 0x7f58080ccd70 2024-01-11T08:25:42.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.722+0000 7f580cbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58080cef20 con 0x7f58080685c0 2024-01-11T08:25:42.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.723+0000 7f580659c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f580815a1b0 0x7f580815a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:42.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.723+0000 7f580659c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f580815a1b0 0x7f580815a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47772/0 (socket says 172.21.15.27:47772) 2024-01-11T08:25:42.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.723+0000 7f580659c700 1 -- 172.21.15.27:0/2199985490 learned_addr learned my addr 172.21.15.27:0/2199985490 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:42.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.723+0000 7f5805d9b700 1 --2- 172.21.15.27:0/2199985490 >> v2:172.21.15.77:3300/0 conn(0x7f58080ccd70 0x7f58080cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:42.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.723+0000 7f5806d9d700 1 --2- 172.21.15.27:0/2199985490 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f58080685c0 0x7f5808068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:42.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.723+0000 7f580659c700 1 -- 172.21.15.27:0/2199985490 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f58080685c0 msgr2=0x7f5808068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:42.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.723+0000 7f580659c700 1 --2- 172.21.15.27:0/2199985490 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f58080685c0 0x7f5808068a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.723+0000 7f580659c700 1 -- 172.21.15.27:0/2199985490 >> v2:172.21.15.77:3300/0 conn(0x7f58080ccd70 msgr2=0x7f58080cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:42.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.723+0000 7f580659c700 1 --2- 172.21.15.27:0/2199985490 >> v2:172.21.15.77:3300/0 conn(0x7f58080ccd70 0x7f58080cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.723+0000 7f580659c700 1 -- 172.21.15.27:0/2199985490 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58080cf200 con 0x7f580815a1b0 2024-01-11T08:25:42.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.724+0000 7f580659c700 1 --2- 172.21.15.27:0/2199985490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f580815a1b0 0x7f580815a580 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f58080696f0 tx=0x7f57f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b01c0f41456c8d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:42.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.724+0000 7f580559a700 1 -- 172.21.15.27:0/2199985490 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f800b640 con 0x7f580815a1b0 2024-01-11T08:25:42.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.724+0000 7f580559a700 1 -- 172.21.15.27:0/2199985490 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f800b7a0 con 0x7f580815a1b0 2024-01-11T08:25:42.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.724+0000 7f580559a700 1 -- 172.21.15.27:0/2199985490 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f800bb60 con 0x7f580815a1b0 2024-01-11T08:25:42.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.725+0000 7f580cbff700 1 -- 172.21.15.27:0/2199985490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f580815a1b0 msgr2=0x7f580815a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:42.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.725+0000 7f580cbff700 1 --2- 172.21.15.27:0/2199985490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f580815a1b0 0x7f580815a580 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f58080696f0 tx=0x7f57f8009250 comp rx=0 tx=0).stop 2024-01-11T08:25:42.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.725+0000 7f580cbff700 1 -- 172.21.15.27:0/2199985490 shutdown_connections 2024-01-11T08:25:42.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.725+0000 7f580cbff700 1 --2- 172.21.15.27:0/2199985490 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f58080685c0 0x7f5808068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.725+0000 7f580cbff700 1 --2- 172.21.15.27:0/2199985490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f580815a1b0 0x7f580815a580 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.725+0000 7f580cbff700 1 --2- 172.21.15.27:0/2199985490 >> v2:172.21.15.77:3300/0 conn(0x7f58080ccd70 0x7f58080cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.725+0000 7f580cbff700 1 -- 172.21.15.27:0/2199985490 >> 172.21.15.27:0/2199985490 conn(0x7f580806ed40 msgr2=0x7f580806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:42.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.725+0000 7f580cbff700 1 -- 172.21.15.27:0/2199985490 shutdown_connections 2024-01-11T08:25:42.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.725+0000 7f580cbff700 1 -- 172.21.15.27:0/2199985490 wait complete. 2024-01-11T08:25:42.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.726+0000 7f580cbff700 1 Processor -- start 2024-01-11T08:25:42.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.727+0000 7f580cbff700 1 -- start start 2024-01-11T08:25:42.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.728+0000 7f580cbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f58080685c0 0x7f580815af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:42.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.728+0000 7f580cbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58080ccd70 0x7f580815c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:42.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.728+0000 7f580cbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f580815a1b0 0x7f58081558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:42.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.728+0000 7f580cbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580815cdd0 con 0x7f58080ccd70 2024-01-11T08:25:42.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.728+0000 7f580cbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580815b600 con 0x7f580815a1b0 2024-01-11T08:25:42.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.728+0000 7f580cbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f580815b8e0 con 0x7f58080685c0 2024-01-11T08:25:42.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.728+0000 7f580659c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f58080685c0 0x7f580815af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:42.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.728+0000 7f5805d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58080ccd70 0x7f580815c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:42.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.728+0000 7f580659c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f58080685c0 0x7f580815af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:47336/0 (socket says 172.21.15.27:47336) 2024-01-11T08:25:42.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.728+0000 7f580659c700 1 -- 172.21.15.27:0/1469285786 learned_addr learned my addr 172.21.15.27:0/1469285786 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:42.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.728+0000 7f5806d9d700 1 --2- 172.21.15.27:0/1469285786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f580815a1b0 0x7f58081558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:42.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.728+0000 7f580659c700 1 -- 172.21.15.27:0/1469285786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f580815a1b0 msgr2=0x7f58081558e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:42.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.728+0000 7f580659c700 1 --2- 172.21.15.27:0/1469285786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f580815a1b0 0x7f58081558e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.728+0000 7f580659c700 1 -- 172.21.15.27:0/1469285786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58080ccd70 msgr2=0x7f580815c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:42.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.728+0000 7f580659c700 1 --2- 172.21.15.27:0/1469285786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58080ccd70 0x7f580815c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.728+0000 7f580659c700 1 -- 172.21.15.27:0/1469285786 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f80094f0 con 0x7f58080685c0 2024-01-11T08:25:42.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.729+0000 7f580659c700 1 --2- 172.21.15.27:0/1469285786 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f58080685c0 0x7f580815af40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5808069770 tx=0x7f57f8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:42.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.729+0000 7f57f67fc700 1 -- 172.21.15.27:0/1469285786 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f800b910 con 0x7f58080685c0 2024-01-11T08:25:42.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.729+0000 7f57f67fc700 1 -- 172.21.15.27:0/1469285786 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f800ba70 con 0x7f58080685c0 2024-01-11T08:25:42.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.729+0000 7f580cbff700 1 -- 172.21.15.27:0/1469285786 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f580815bbc0 con 0x7f58080685c0 2024-01-11T08:25:42.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.729+0000 7f57f67fc700 1 -- 172.21.15.27:0/1469285786 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f8011630 con 0x7f58080685c0 2024-01-11T08:25:42.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.729+0000 7f580cbff700 1 -- 172.21.15.27:0/1469285786 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58080c8880 con 0x7f58080685c0 2024-01-11T08:25:42.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.730+0000 7f580cbff700 1 -- 172.21.15.27:0/1469285786 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f580804c3c0 con 0x7f58080685c0 2024-01-11T08:25:42.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.731+0000 7f57f67fc700 1 -- 172.21.15.27:0/1469285786 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f57f80119a0 con 0x7f58080685c0 2024-01-11T08:25:42.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.733+0000 7f57f67fc700 1 --2- 172.21.15.27:0/1469285786 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f57e8067050 0x7f57e8069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:42.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.733+0000 7f57f67fc700 1 -- 172.21.15.27:0/1469285786 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5991+0+0 (secure 0 0 0) 0x7f57f8085f80 con 0x7f58080685c0 2024-01-11T08:25:42.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.733+0000 7f5805d9b700 1 -- 172.21.15.27:0/1469285786 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f57e8067050 msgr2=0x7f57e8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:42.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.733+0000 7f5805d9b700 1 --2- 172.21.15.27:0/1469285786 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f57e8067050 0x7f57e8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:42.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.737+0000 7f57f67fc700 1 -- 172.21.15.27:0/1469285786 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f57f8052f20 con 0x7f58080685c0 2024-01-11T08:25:42.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.934+0000 7f5805d9b700 1 -- 172.21.15.27:0/1469285786 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f57e8067050 msgr2=0x7f57e8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:42.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.934+0000 7f5805d9b700 1 --2- 172.21.15.27:0/1469285786 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f57e8067050 0x7f57e8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:42.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.942+0000 7f580cbff700 1 -- 172.21.15.27:0/1469285786 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f58080b2290 con 0x7f58080685c0 2024-01-11T08:25:42.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.943+0000 7f57f67fc700 1 -- 172.21.15.27:0/1469285786 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7f57f8056540 con 0x7f58080685c0 2024-01-11T08:25:42.942 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:25:42.944 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:42.945+0000 7f01ec3381c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:25:42.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.946+0000 7f580cbff700 1 -- 172.21.15.27:0/1469285786 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f57e8067050 msgr2=0x7f57e8069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:42.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.946+0000 7f580cbff700 1 --2- 172.21.15.27:0/1469285786 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f57e8067050 0x7f57e8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.946+0000 7f580cbff700 1 -- 172.21.15.27:0/1469285786 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f58080685c0 msgr2=0x7f580815af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:42.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.946+0000 7f580cbff700 1 --2- 172.21.15.27:0/1469285786 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f58080685c0 0x7f580815af40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5808069770 tx=0x7f57f8017000 comp rx=0 tx=0).stop 2024-01-11T08:25:42.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.946+0000 7f580cbff700 1 -- 172.21.15.27:0/1469285786 shutdown_connections 2024-01-11T08:25:42.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.946+0000 7f580cbff700 1 --2- 172.21.15.27:0/1469285786 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f57e8067050 0x7f57e8069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.946+0000 7f580cbff700 1 --2- 172.21.15.27:0/1469285786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f580815a1b0 0x7f58081558e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.946+0000 7f580cbff700 1 --2- 172.21.15.27:0/1469285786 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f58080685c0 0x7f580815af40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.946+0000 7f580cbff700 1 --2- 172.21.15.27:0/1469285786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58080ccd70 0x7f580815c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:42.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.946+0000 7f580cbff700 1 -- 172.21.15.27:0/1469285786 >> 172.21.15.27:0/1469285786 conn(0x7f580806ed40 msgr2=0x7f580815d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:42.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.946+0000 7f580cbff700 1 -- 172.21.15.27:0/1469285786 shutdown_connections 2024-01-11T08:25:42.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:42.946+0000 7f580cbff700 1 -- 172.21.15.27:0/1469285786 wait complete. 2024-01-11T08:25:42.965 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T08:25:43.105 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:43.106+0000 7ff67d0f81c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:25:43.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.173+0000 7f74af3ff700 1 Processor -- start 2024-01-11T08:25:43.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.174+0000 7f74af3ff700 1 -- start start 2024-01-11T08:25:43.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.175+0000 7f74af3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74a8142dc0 0x7f74a80cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:43.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.175+0000 7f74af3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74a8143190 0x7f74a80cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:43.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.175+0000 7f74af3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a815d320 con 0x7f74a80cc020 2024-01-11T08:25:43.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.175+0000 7f74af3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a815df30 con 0x7f74a8142dc0 2024-01-11T08:25:43.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.175+0000 7f74af3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a815d590 con 0x7f74a8143190 2024-01-11T08:25:43.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.175+0000 7f74ad19b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74a8142dc0 0x7f74a80cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:43.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.175+0000 7f74ad19b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74a8142dc0 0x7f74a80cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47786/0 (socket says 172.21.15.27:47786) 2024-01-11T08:25:43.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.175+0000 7f74ad19b700 1 -- 172.21.15.27:0/3124534846 learned_addr learned my addr 172.21.15.27:0/3124534846 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:43.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.175+0000 7f74ac99a700 1 --2- 172.21.15.27:0/3124534846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74a8143190 0x7f74a80cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:43.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.176+0000 7f74ad19b700 1 -- 172.21.15.27:0/3124534846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74a8143190 msgr2=0x7f74a80cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:43.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.176+0000 7f74ad19b700 1 --2- 172.21.15.27:0/3124534846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74a8143190 0x7f74a80cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:43.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.176+0000 7f74ad19b700 1 -- 172.21.15.27:0/3124534846 >> v1:172.21.15.77:6789/0 conn(0x7f74a80cc020 legacy=0x7f74a80cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:43.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.176+0000 7f74ad19b700 1 -- 172.21.15.27:0/3124534846 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a815d870 con 0x7f74a8142dc0 2024-01-11T08:25:43.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.176+0000 7f74ad19b700 1 --2- 172.21.15.27:0/3124534846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74a8142dc0 0x7f74a80cada0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f74a40099b0 tx=0x7f74a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45f4415ab60d8203 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:43.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.176+0000 7f749ffff700 1 -- 172.21.15.27:0/3124534846 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a4009ea0 con 0x7f74a8142dc0 2024-01-11T08:25:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.176+0000 7f749ffff700 1 -- 172.21.15.27:0/3124534846 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a400b860 con 0x7f74a8142dc0 2024-01-11T08:25:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.176+0000 7f749ffff700 1 -- 172.21.15.27:0/3124534846 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a400bc00 con 0x7f74a8142dc0 2024-01-11T08:25:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.177+0000 7f74af3ff700 1 -- 172.21.15.27:0/3124534846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74a8142dc0 msgr2=0x7f74a80cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.177+0000 7f74af3ff700 1 --2- 172.21.15.27:0/3124534846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74a8142dc0 0x7f74a80cada0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f74a40099b0 tx=0x7f74a4009250 comp rx=0 tx=0).stop 2024-01-11T08:25:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.177+0000 7f74af3ff700 1 -- 172.21.15.27:0/3124534846 shutdown_connections 2024-01-11T08:25:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.177+0000 7f74af3ff700 1 --2- 172.21.15.27:0/3124534846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74a8142dc0 0x7f74a80cada0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.177+0000 7f74af3ff700 1 --2- 172.21.15.27:0/3124534846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74a8143190 0x7f74a80cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.177+0000 7f74af3ff700 1 -- 172.21.15.27:0/3124534846 >> 172.21.15.27:0/3124534846 conn(0x7f74a8063ca0 msgr2=0x7f74a806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.177+0000 7f74af3ff700 1 -- 172.21.15.27:0/3124534846 shutdown_connections 2024-01-11T08:25:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.177+0000 7f74af3ff700 1 -- 172.21.15.27:0/3124534846 wait complete. 2024-01-11T08:25:43.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.179+0000 7f74af3ff700 1 Processor -- start 2024-01-11T08:25:43.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.179+0000 7f74af3ff700 1 -- start start 2024-01-11T08:25:43.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.180+0000 7f74af3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74a80cc020 0x7f74a80c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:43.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.180+0000 7f74af3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74a8142dc0 0x7f74a80b0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:43.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.180+0000 7f74af3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74a8143190 0x7f74a8141bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:43.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.180+0000 7f74af3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a80b06b0 con 0x7f74a80cc020 2024-01-11T08:25:43.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.180+0000 7f74af3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a816d9f0 con 0x7f74a8143190 2024-01-11T08:25:43.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.180+0000 7f74af3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a816dcd0 con 0x7f74a8142dc0 2024-01-11T08:25:43.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.180+0000 7f74ad19b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74a80cc020 0x7f74a80c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:43.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.180+0000 7f74ad19b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74a80cc020 0x7f74a80c8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47788/0 (socket says 172.21.15.27:47788) 2024-01-11T08:25:43.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.180+0000 7f74ad19b700 1 -- 172.21.15.27:0/3146890994 learned_addr learned my addr 172.21.15.27:0/3146890994 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:43.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.180+0000 7f74ac99a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74a8142dc0 0x7f74a80b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:43.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.180+0000 7f74ad99c700 1 --2- 172.21.15.27:0/3146890994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74a8143190 0x7f74a8141bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:43.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.181+0000 7f74ac99a700 1 -- 172.21.15.27:0/3146890994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74a8143190 msgr2=0x7f74a8141bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:43.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.181+0000 7f74ac99a700 1 --2- 172.21.15.27:0/3146890994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74a8143190 0x7f74a8141bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:43.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.181+0000 7f74ac99a700 1 -- 172.21.15.27:0/3146890994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74a80cc020 msgr2=0x7f74a80c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:43.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.181+0000 7f74ac99a700 1 --2- 172.21.15.27:0/3146890994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74a80cc020 0x7f74a80c8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:43.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.181+0000 7f74ac99a700 1 -- 172.21.15.27:0/3146890994 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a40094f0 con 0x7f74a8142dc0 2024-01-11T08:25:43.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.181+0000 7f74ad19b700 1 --2- 172.21.15.27:0/3146890994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74a80cc020 0x7f74a80c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:43.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.181+0000 7f74ac99a700 1 --2- 172.21.15.27:0/3146890994 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74a8142dc0 0x7f74a80b0170 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f749800e900 tx=0x7f74980091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:43.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.181+0000 7f749d7fa700 1 -- 172.21.15.27:0/3146890994 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7498013680 con 0x7f74a8142dc0 2024-01-11T08:25:43.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.181+0000 7f749d7fa700 1 -- 172.21.15.27:0/3146890994 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74980137e0 con 0x7f74a8142dc0 2024-01-11T08:25:43.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.181+0000 7f749d7fa700 1 -- 172.21.15.27:0/3146890994 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7498013ba0 con 0x7f74a8142dc0 2024-01-11T08:25:43.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.182+0000 7f74af3ff700 1 -- 172.21.15.27:0/3146890994 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a816dfb0 con 0x7f74a8142dc0 2024-01-11T08:25:43.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.182+0000 7f74af3ff700 1 -- 172.21.15.27:0/3146890994 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a80c85d0 con 0x7f74a8142dc0 2024-01-11T08:25:43.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.183+0000 7f74af3ff700 1 -- 172.21.15.27:0/3146890994 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a804c3c0 con 0x7f74a8142dc0 2024-01-11T08:25:43.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.184+0000 7f749d7fa700 1 -- 172.21.15.27:0/3146890994 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f74980123d0 con 0x7f74a8142dc0 2024-01-11T08:25:43.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.184+0000 7f749d7fa700 1 --2- 172.21.15.27:0/3146890994 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f748c067050 0x7f748c069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:43.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.185+0000 7f749d7fa700 1 -- 172.21.15.27:0/3146890994 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5991+0+0 (secure 0 0 0) 0x7f74980848c0 con 0x7f74a8142dc0 2024-01-11T08:25:43.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.185+0000 7f74ad19b700 1 -- 172.21.15.27:0/3146890994 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f748c067050 msgr2=0x7f748c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:43.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.185+0000 7f74ad19b700 1 --2- 172.21.15.27:0/3146890994 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f748c067050 0x7f748c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:43.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.189+0000 7f749d7fa700 1 -- 172.21.15.27:0/3146890994 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7498051860 con 0x7f74a8142dc0 2024-01-11T08:25:43.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.386+0000 7f74ad19b700 1 -- 172.21.15.27:0/3146890994 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f748c067050 msgr2=0x7f748c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:43.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.386+0000 7f74ad19b700 1 --2- 172.21.15.27:0/3146890994 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f748c067050 0x7f748c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:43.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.393+0000 7f74af3ff700 1 -- 172.21.15.27:0/3146890994 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f74a815db50 con 0x7f74a8142dc0 2024-01-11T08:25:43.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.394+0000 7f749d7fa700 1 -- 172.21.15.27:0/3146890994 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v76) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7498021020 con 0x7f74a8142dc0 2024-01-11T08:25:43.393 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T08:25:43.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.396+0000 7f74af3ff700 1 -- 172.21.15.27:0/3146890994 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f748c067050 msgr2=0x7f748c069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:43.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.396+0000 7f74af3ff700 1 --2- 172.21.15.27:0/3146890994 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f748c067050 0x7f748c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.396+0000 7f74af3ff700 1 -- 172.21.15.27:0/3146890994 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74a8142dc0 msgr2=0x7f74a80b0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.396+0000 7f74af3ff700 1 --2- 172.21.15.27:0/3146890994 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74a8142dc0 0x7f74a80b0170 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f749800e900 tx=0x7f74980091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.396+0000 7f74af3ff700 1 -- 172.21.15.27:0/3146890994 shutdown_connections 2024-01-11T08:25:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.396+0000 7f74af3ff700 1 --2- 172.21.15.27:0/3146890994 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f748c067050 0x7f748c069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.396+0000 7f74af3ff700 1 --2- 172.21.15.27:0/3146890994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74a8143190 0x7f74a8141bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.396+0000 7f74af3ff700 1 --2- 172.21.15.27:0/3146890994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74a80cc020 0x7f74a80c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.396+0000 7f74af3ff700 1 --2- 172.21.15.27:0/3146890994 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74a8142dc0 0x7f74a80b0170 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.396+0000 7f74af3ff700 1 -- 172.21.15.27:0/3146890994 >> 172.21.15.27:0/3146890994 conn(0x7f74a8063ca0 msgr2=0x7f74a80c4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.396+0000 7f74af3ff700 1 -- 172.21.15.27:0/3146890994 shutdown_connections 2024-01-11T08:25:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.396+0000 7f74af3ff700 1 -- 172.21.15.27:0/3146890994 wait complete. 2024-01-11T08:25:43.414 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T08:25:43.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.618+0000 7fb7915ff700 1 Processor -- start 2024-01-11T08:25:43.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.618+0000 7fb7915ff700 1 -- start start 2024-01-11T08:25:43.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.620+0000 7fb7915ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb78c0cb4b0 0x7fb78c0cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:43.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.620+0000 7fb7915ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78c0ccc70 con 0x7fb78c142dc0 2024-01-11T08:25:43.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.620+0000 7fb7915ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78c15d320 con 0x7fb78c143190 2024-01-11T08:25:43.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.620+0000 7fb7915ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb78c15df30 con 0x7fb78c0cb4b0 2024-01-11T08:25:43.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.620+0000 7fb78b7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb78c0cb4b0 0x7fb78c0cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:43.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.620+0000 7fb78affd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb78c142dc0 0x7fb78c0cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36572/0 (socket says 172.21.15.27:36572) 2024-01-11T08:25:43.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.620+0000 7fb78affd700 1 -- 172.21.15.27:0/1066064266 learned_addr learned my addr 172.21.15.27:0/1066064266 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:43.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.620+0000 7fb789ffb700 1 -- 172.21.15.27:0/1066064266 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230528989 0 0) 0x7fb78c0ccc70 con 0x7fb78c142dc0 2024-01-11T08:25:43.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.620+0000 7fb78b7fe700 1 -- 172.21.15.27:0/1066064266 >> v1:172.21.15.77:6789/0 conn(0x7fb78c143190 legacy=0x7fb78c0cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:43.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.620+0000 7fb78b7fe700 1 -- 172.21.15.27:0/1066064266 >> v1:172.21.15.27:6789/0 conn(0x7fb78c142dc0 legacy=0x7fb78c0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:43.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.620+0000 7fb78b7fe700 1 -- 172.21.15.27:0/1066064266 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c15d590 con 0x7fb78c0cb4b0 2024-01-11T08:25:43.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.621+0000 7fb78b7fe700 1 --2- 172.21.15.27:0/1066064266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb78c0cb4b0 0x7fb78c0cc730 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb784009de0 tx=0x7fb784009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=462ed95e42c73dc2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:43.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.621+0000 7fb789ffb700 1 -- 172.21.15.27:0/1066064266 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78400e070 con 0x7fb78c0cb4b0 2024-01-11T08:25:43.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.621+0000 7fb789ffb700 1 -- 172.21.15.27:0/1066064266 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78400bbb0 con 0x7fb78c0cb4b0 2024-01-11T08:25:43.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.621+0000 7fb789ffb700 1 -- 172.21.15.27:0/1066064266 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784012430 con 0x7fb78c0cb4b0 2024-01-11T08:25:43.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.622+0000 7fb7915ff700 1 -- 172.21.15.27:0/1066064266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb78c0cb4b0 msgr2=0x7fb78c0cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:43.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.622+0000 7fb7915ff700 1 --2- 172.21.15.27:0/1066064266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb78c0cb4b0 0x7fb78c0cc730 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb784009de0 tx=0x7fb784009780 comp rx=0 tx=0).stop 2024-01-11T08:25:43.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.622+0000 7fb7915ff700 1 -- 172.21.15.27:0/1066064266 shutdown_connections 2024-01-11T08:25:43.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.622+0000 7fb7915ff700 1 --2- 172.21.15.27:0/1066064266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb78c0cb4b0 0x7fb78c0cc730 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:43.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.622+0000 7fb7915ff700 1 -- 172.21.15.27:0/1066064266 >> 172.21.15.27:0/1066064266 conn(0x7fb78c063ca0 msgr2=0x7fb78c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:43.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.622+0000 7fb7915ff700 1 -- 172.21.15.27:0/1066064266 shutdown_connections 2024-01-11T08:25:43.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.622+0000 7fb7915ff700 1 -- 172.21.15.27:0/1066064266 wait complete. 2024-01-11T08:25:43.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.624+0000 7fb7915ff700 1 Processor -- start 2024-01-11T08:25:43.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.624+0000 7fb7915ff700 1 -- start start 2024-01-11T08:25:43.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.625+0000 7fb7915ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb78c0cb4b0 0x7fb78c15ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:43.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.625+0000 7fb7915ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb78c142dc0 0x7fb78c0b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:43.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.625+0000 7fb7915ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb78c143190 0x7fb78c141dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:43.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.625+0000 7fb7915ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c15ff20 con 0x7fb78c143190 2024-01-11T08:25:43.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.625+0000 7fb7915ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c15f310 con 0x7fb78c0cb4b0 2024-01-11T08:25:43.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.625+0000 7fb7915ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c15f5f0 con 0x7fb78c142dc0 2024-01-11T08:25:43.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.625+0000 7fb78b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb78c143190 0x7fb78c141dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:43.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.625+0000 7fb78affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb78c0cb4b0 0x7fb78c15ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:43.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.625+0000 7fb78b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb78c143190 0x7fb78c141dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47820/0 (socket says 172.21.15.27:47820) 2024-01-11T08:25:43.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.625+0000 7fb78b7fe700 1 -- 172.21.15.27:0/1412850275 learned_addr learned my addr 172.21.15.27:0/1412850275 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:43.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.625+0000 7fb78a7fc700 1 --2- 172.21.15.27:0/1412850275 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb78c142dc0 0x7fb78c0b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:43.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.626+0000 7fb78b7fe700 1 -- 172.21.15.27:0/1412850275 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb78c142dc0 msgr2=0x7fb78c0b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:43.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.626+0000 7fb78b7fe700 1 --2- 172.21.15.27:0/1412850275 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb78c142dc0 0x7fb78c0b2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:43.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.626+0000 7fb78b7fe700 1 -- 172.21.15.27:0/1412850275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb78c0cb4b0 msgr2=0x7fb78c15ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:43.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.626+0000 7fb78b7fe700 1 --2- 172.21.15.27:0/1412850275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb78c0cb4b0 0x7fb78c15ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:43.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.626+0000 7fb78b7fe700 1 -- 172.21.15.27:0/1412850275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb784009a20 con 0x7fb78c143190 2024-01-11T08:25:43.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.626+0000 7fb78b7fe700 1 --2- 172.21.15.27:0/1412850275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb78c143190 0x7fb78c141dd0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fb7840099f0 tx=0x7fb784018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:43.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.626+0000 7fb772ffd700 1 -- 172.21.15.27:0/1412850275 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784027470 con 0x7fb78c143190 2024-01-11T08:25:43.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.626+0000 7fb772ffd700 1 -- 172.21.15.27:0/1412850275 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7840275f0 con 0x7fb78c143190 2024-01-11T08:25:43.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.626+0000 7fb772ffd700 1 -- 172.21.15.27:0/1412850275 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7840279b0 con 0x7fb78c143190 2024-01-11T08:25:43.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.626+0000 7fb7915ff700 1 -- 172.21.15.27:0/1412850275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78c15f8d0 con 0x7fb78c143190 2024-01-11T08:25:43.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.627+0000 7fb7915ff700 1 -- 172.21.15.27:0/1412850275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb78c0b34a0 con 0x7fb78c143190 2024-01-11T08:25:43.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.628+0000 7fb7915ff700 1 -- 172.21.15.27:0/1412850275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78c1600d0 con 0x7fb78c143190 2024-01-11T08:25:43.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.629+0000 7fb772ffd700 1 -- 172.21.15.27:0/1412850275 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fb784024070 con 0x7fb78c143190 2024-01-11T08:25:43.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.630+0000 7fb772ffd700 1 --2- 172.21.15.27:0/1412850275 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb778067050 0x7fb778069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:43.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.630+0000 7fb772ffd700 1 -- 172.21.15.27:0/1412850275 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5991+0+0 (secure 0 0 0) 0x7fb784086ff0 con 0x7fb78c143190 2024-01-11T08:25:43.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.630+0000 7fb78affd700 1 -- 172.21.15.27:0/1412850275 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb778067050 msgr2=0x7fb778069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:43.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.630+0000 7fb78affd700 1 --2- 172.21.15.27:0/1412850275 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb778067050 0x7fb778069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:43.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.634+0000 7fb772ffd700 1 -- 172.21.15.27:0/1412850275 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb78c1600d0 con 0x7fb78c143190 2024-01-11T08:25:43.724 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:43.726+0000 7f01ec3381c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:25:43.812 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:43.814+0000 7f01ec3381c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:25:43.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.831+0000 7fb78affd700 1 -- 172.21.15.27:0/1412850275 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb778067050 msgr2=0x7fb778069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:43.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.831+0000 7fb78affd700 1 --2- 172.21.15.27:0/1412850275 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb778067050 0x7fb778069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:43.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.836+0000 7fb7915ff700 1 -- 172.21.15.27:0/1412850275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb78c04c3c0 con 0x7fb78c143190 2024-01-11T08:25:43.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.836+0000 7fb772ffd700 1 -- 172.21.15.27:0/1412850275 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v76) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb784053ff0 con 0x7fb78c143190 2024-01-11T08:25:43.835 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T08:25:43.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.839+0000 7fb7915ff700 1 -- 172.21.15.27:0/1412850275 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb778067050 msgr2=0x7fb778069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:43.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.839+0000 7fb7915ff700 1 --2- 172.21.15.27:0/1412850275 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb778067050 0x7fb778069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:43.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.839+0000 7fb7915ff700 1 -- 172.21.15.27:0/1412850275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb78c143190 msgr2=0x7fb78c141dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:43.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.839+0000 7fb7915ff700 1 --2- 172.21.15.27:0/1412850275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb78c143190 0x7fb78c141dd0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fb7840099f0 tx=0x7fb784018000 comp rx=0 tx=0).stop 2024-01-11T08:25:43.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.839+0000 7fb7915ff700 1 -- 172.21.15.27:0/1412850275 shutdown_connections 2024-01-11T08:25:43.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.839+0000 7fb7915ff700 1 --2- 172.21.15.27:0/1412850275 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb778067050 0x7fb778069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:43.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.839+0000 7fb7915ff700 1 --2- 172.21.15.27:0/1412850275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb78c143190 0x7fb78c141dd0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:43.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.839+0000 7fb7915ff700 1 --2- 172.21.15.27:0/1412850275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb78c0cb4b0 0x7fb78c15ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:43.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.839+0000 7fb7915ff700 1 --2- 172.21.15.27:0/1412850275 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb78c142dc0 0x7fb78c0b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:43.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.839+0000 7fb7915ff700 1 -- 172.21.15.27:0/1412850275 >> 172.21.15.27:0/1412850275 conn(0x7fb78c063ca0 msgr2=0x7fb78c0703a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:43.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.839+0000 7fb7915ff700 1 -- 172.21.15.27:0/1412850275 shutdown_connections 2024-01-11T08:25:43.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:43.839+0000 7fb7915ff700 1 -- 172.21.15.27:0/1412850275 wait complete. 2024-01-11T08:25:43.857 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:25:43.959 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:43.960+0000 7ff67d0f81c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:25:43.967 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:43.968+0000 7f01ec3381c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:25:44.053 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:44.054+0000 7f01ec3381c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:25:44.055 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:44.056+0000 7ff67d0f81c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:25:44.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.068+0000 7f22bc5ff700 1 Processor -- start 2024-01-11T08:25:44.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.069+0000 7f22bc5ff700 1 -- start start 2024-01-11T08:25:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.070+0000 7f22bc5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f22b40b03e0 0x7f22b40b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.070+0000 7f22bc5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22b40b07b0 0x7f22b40b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.070+0000 7f22bc5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b40cd730 con 0x7f22b4159200 2024-01-11T08:25:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.070+0000 7f22bc5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22b4159d50 con 0x7f22b40b03e0 2024-01-11T08:25:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.070+0000 7f22bc5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22b40ccb20 con 0x7f22b40b07b0 2024-01-11T08:25:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.070+0000 7f22ba39b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f22b40b03e0 0x7f22b40b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.070+0000 7f22b9b9a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22b40b07b0 0x7f22b40b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.070+0000 7f22ba39b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f22b40b03e0 0x7f22b40b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41594/0 (socket says 172.21.15.27:41594) 2024-01-11T08:25:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.070+0000 7f22ba39b700 1 -- 172.21.15.27:0/136678536 learned_addr learned my addr 172.21.15.27:0/136678536 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.071+0000 7f22ba39b700 1 -- 172.21.15.27:0/136678536 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22b40b07b0 msgr2=0x7f22b40b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.071+0000 7f22ba39b700 1 --2- 172.21.15.27:0/136678536 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22b40b07b0 0x7f22b40b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.071+0000 7f22ba39b700 1 -- 172.21.15.27:0/136678536 >> v1:172.21.15.77:6789/0 conn(0x7f22b4159200 legacy=0x7f22b4159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:44.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.071+0000 7f22ba39b700 1 -- 172.21.15.27:0/136678536 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b40cce00 con 0x7f22b40b03e0 2024-01-11T08:25:44.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.071+0000 7f22ba39b700 1 --2- 172.21.15.27:0/136678536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22b40b03e0 0x7f22b40b2480 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f22b00099b0 tx=0x7f22b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8a27f2666407f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:44.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.071+0000 7f22b9399700 1 -- 172.21.15.27:0/136678536 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b0009ea0 con 0x7f22b40b03e0 2024-01-11T08:25:44.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.071+0000 7f22b9399700 1 -- 172.21.15.27:0/136678536 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b000b860 con 0x7f22b40b03e0 2024-01-11T08:25:44.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.071+0000 7f22b9399700 1 -- 172.21.15.27:0/136678536 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b000bc00 con 0x7f22b40b03e0 2024-01-11T08:25:44.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.072+0000 7f22bc5ff700 1 -- 172.21.15.27:0/136678536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22b40b03e0 msgr2=0x7f22b40b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:44.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.072+0000 7f22bc5ff700 1 --2- 172.21.15.27:0/136678536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22b40b03e0 0x7f22b40b2480 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f22b00099b0 tx=0x7f22b0009250 comp rx=0 tx=0).stop 2024-01-11T08:25:44.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.072+0000 7f22bc5ff700 1 -- 172.21.15.27:0/136678536 shutdown_connections 2024-01-11T08:25:44.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.072+0000 7f22bc5ff700 1 --2- 172.21.15.27:0/136678536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22b40b03e0 0x7f22b40b2480 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.072+0000 7f22bc5ff700 1 --2- 172.21.15.27:0/136678536 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22b40b07b0 0x7f22b40b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.072+0000 7f22bc5ff700 1 -- 172.21.15.27:0/136678536 >> 172.21.15.27:0/136678536 conn(0x7f22b406ed40 msgr2=0x7f22b406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:44.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.072+0000 7f22bc5ff700 1 -- 172.21.15.27:0/136678536 shutdown_connections 2024-01-11T08:25:44.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.072+0000 7f22bc5ff700 1 -- 172.21.15.27:0/136678536 wait complete. 2024-01-11T08:25:44.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.074+0000 7f22bc5ff700 1 Processor -- start 2024-01-11T08:25:44.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.074+0000 7f22bc5ff700 1 -- start start 2024-01-11T08:25:44.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.075+0000 7f22bc5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22b40b03e0 0x7f22b40ca640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:44.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.075+0000 7f22bc5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22b40b07b0 0x7f22b4158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:44.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.075+0000 7f22bc5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22b4159200 0x7f22b40c5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:44.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.075+0000 7f22bc5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b40cab80 con 0x7f22b40b07b0 2024-01-11T08:25:44.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.075+0000 7f22bc5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b40c7ad0 con 0x7f22b40b03e0 2024-01-11T08:25:44.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.075+0000 7f22bc5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22b40c86c0 con 0x7f22b4159200 2024-01-11T08:25:44.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.075+0000 7f22b9b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22b40b07b0 0x7f22b4158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:44.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.075+0000 7f22b9b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22b40b07b0 0x7f22b4158130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41608/0 (socket says 172.21.15.27:41608) 2024-01-11T08:25:44.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.075+0000 7f22b9b9a700 1 -- 172.21.15.27:0/973666590 learned_addr learned my addr 172.21.15.27:0/973666590 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:44.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.075+0000 7f22ba39b700 1 --2- 172.21.15.27:0/973666590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22b40b03e0 0x7f22b40ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:44.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.075+0000 7f22bab9c700 1 --2- 172.21.15.27:0/973666590 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22b4159200 0x7f22b40c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:44.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.075+0000 7f22b9b9a700 1 -- 172.21.15.27:0/973666590 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22b4159200 msgr2=0x7f22b40c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:44.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.075+0000 7f22b9b9a700 1 --2- 172.21.15.27:0/973666590 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22b4159200 0x7f22b40c5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.075+0000 7f22b9b9a700 1 -- 172.21.15.27:0/973666590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22b40b03e0 msgr2=0x7f22b40ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:44.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.075+0000 7f22b9b9a700 1 --2- 172.21.15.27:0/973666590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22b40b03e0 0x7f22b40ca640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.075+0000 7f22b9b9a700 1 -- 172.21.15.27:0/973666590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b00094f0 con 0x7f22b40b07b0 2024-01-11T08:25:44.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.076+0000 7f22b9b9a700 1 --2- 172.21.15.27:0/973666590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22b40b07b0 0x7f22b4158130 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f22a400ec80 tx=0x7f22a40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:44.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.076+0000 7f22aa7fc700 1 -- 172.21.15.27:0/973666590 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a400c850 con 0x7f22b40b07b0 2024-01-11T08:25:44.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.076+0000 7f22aa7fc700 1 -- 172.21.15.27:0/973666590 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a400c9b0 con 0x7f22b40b07b0 2024-01-11T08:25:44.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.076+0000 7f22aa7fc700 1 -- 172.21.15.27:0/973666590 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a400cd50 con 0x7f22b40b07b0 2024-01-11T08:25:44.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.076+0000 7f22bc5ff700 1 -- 172.21.15.27:0/973666590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b40c8100 con 0x7f22b40b07b0 2024-01-11T08:25:44.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.076+0000 7f22bc5ff700 1 -- 172.21.15.27:0/973666590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b40c7760 con 0x7f22b40b07b0 2024-01-11T08:25:44.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.077+0000 7f22bc5ff700 1 -- 172.21.15.27:0/973666590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b40c75c0 con 0x7f22b40b07b0 2024-01-11T08:25:44.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.078+0000 7f22aa7fc700 1 -- 172.21.15.27:0/973666590 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f22a4013680 con 0x7f22b40b07b0 2024-01-11T08:25:44.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.079+0000 7f22aa7fc700 1 --2- 172.21.15.27:0/973666590 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2298067050 0x7f2298069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:44.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.079+0000 7f22aa7fc700 1 -- 172.21.15.27:0/973666590 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5991+0+0 (secure 0 0 0) 0x7f22a40849f0 con 0x7f22b40b07b0 2024-01-11T08:25:44.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.079+0000 7f22ba39b700 1 -- 172.21.15.27:0/973666590 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2298067050 msgr2=0x7f2298069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:44.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.079+0000 7f22ba39b700 1 --2- 172.21.15.27:0/973666590 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2298067050 0x7f2298069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:44.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.087+0000 7f22aa7fc700 1 -- 172.21.15.27:0/973666590 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f22b40c75c0 con 0x7f22b40b07b0 2024-01-11T08:25:44.229 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:44.230+0000 7ff67d0f81c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:25:44.271 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:44.273+0000 7f01ec3381c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:25:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.280+0000 7f22ba39b700 1 -- 172.21.15.27:0/973666590 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2298067050 msgr2=0x7f2298069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.280+0000 7f22ba39b700 1 --2- 172.21.15.27:0/973666590 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2298067050 0x7f2298069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:44.322 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:44.323+0000 7ff67d0f81c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:25:44.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.335+0000 7f22bc5ff700 1 -- 172.21.15.27:0/973666590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22b40c75c0 con 0x7f22b40b07b0 2024-01-11T08:25:44.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.336+0000 7f22aa7fc700 1 -- 172.21.15.27:0/973666590 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4597 (secure 0 0 0) 0x7f22b40c75c0 con 0x7f22b40b07b0 2024-01-11T08:25:44.335 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:44.335 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5606,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2845124438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2845124438},{"type":"v1","addr":"172.21.15.77:6837","nonce":2845124438}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5618,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3958104902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3958104902},{"type":"v1","addr":"172.21.15.27:6835","nonce":3958104902}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":6091,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/2612413022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2612413022},{"type":"v1","addr":"172.21.15.27:6833","nonce":2612413022}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:25:23.345532+0000","modified":"2024-01-11T08:25:26.867139+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6174},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6174":{"gid":6174,"name":"d","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6836/3056318454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3056318454},{"type":"v1","addr":"172.21.15.77:6836","nonce":3056318454}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2024-01-11T08:25:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.338+0000 7f22bc5ff700 1 -- 172.21.15.27:0/973666590 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2298067050 msgr2=0x7f2298069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.338+0000 7f22bc5ff700 1 --2- 172.21.15.27:0/973666590 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2298067050 0x7f2298069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.338+0000 7f22bc5ff700 1 -- 172.21.15.27:0/973666590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22b40b07b0 msgr2=0x7f22b4158130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.338+0000 7f22bc5ff700 1 --2- 172.21.15.27:0/973666590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22b40b07b0 0x7f22b4158130 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f22a400ec80 tx=0x7f22a40091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.338+0000 7f22bc5ff700 1 -- 172.21.15.27:0/973666590 shutdown_connections 2024-01-11T08:25:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.338+0000 7f22bc5ff700 1 --2- 172.21.15.27:0/973666590 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f2298067050 0x7f2298069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.338+0000 7f22bc5ff700 1 --2- 172.21.15.27:0/973666590 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22b4159200 0x7f22b40c5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.338+0000 7f22bc5ff700 1 --2- 172.21.15.27:0/973666590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22b40b03e0 0x7f22b40ca640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.338+0000 7f22bc5ff700 1 --2- 172.21.15.27:0/973666590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22b40b07b0 0x7f22b4158130 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.338+0000 7f22bc5ff700 1 -- 172.21.15.27:0/973666590 >> 172.21.15.27:0/973666590 conn(0x7f22b406ed40 msgr2=0x7f22b40cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.338+0000 7f22bc5ff700 1 -- 172.21.15.27:0/973666590 shutdown_connections 2024-01-11T08:25:44.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.339+0000 7f22bc5ff700 1 -- 172.21.15.27:0/973666590 wait complete. 2024-01-11T08:25:44.339 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 33 2024-01-11T08:25:44.356 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:25:44.366 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:44.367+0000 7f01ec3381c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:25:44.462 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:44.463+0000 7f01ec3381c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:25:44.557 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:44.558+0000 7f01ec3381c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:25:44.561 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:44.562+0000 7ff67d0f81c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:25:44.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.570+0000 7fecee3ff700 1 Processor -- start 2024-01-11T08:25:44.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.570+0000 7fecee3ff700 1 -- start start 2024-01-11T08:25:44.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.571+0000 7fecee3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fece81575e0 0x7fece8158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:44.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.571+0000 7fecee3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fece806b610 0x7fece80b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:44.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.571+0000 7fecee3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fece80b1800 0x7fece80ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:44.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.572+0000 7fecee3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece80cb5e0 con 0x7fece806b610 2024-01-11T08:25:44.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.572+0000 7fecee3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece8158b60 con 0x7fece81575e0 2024-01-11T08:25:44.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.572+0000 7fecee3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece80b1c60 con 0x7fece80b1800 2024-01-11T08:25:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.572+0000 7fece77fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fece806b610 0x7fece80b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.572+0000 7fece7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fece81575e0 0x7fece8158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.572+0000 7fece77fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fece806b610 0x7fece80b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41616/0 (socket says 172.21.15.27:41616) 2024-01-11T08:25:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.572+0000 7fece7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fece81575e0 0x7fece8158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:39038/0 (socket says 172.21.15.27:39038) 2024-01-11T08:25:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.572+0000 7fece77fe700 1 -- 172.21.15.27:0/2219722122 learned_addr learned my addr 172.21.15.27:0/2219722122 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.572+0000 7fecec99c700 1 --2- 172.21.15.27:0/2219722122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fece80b1800 0x7fece80ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.572+0000 7fece77fe700 1 -- 172.21.15.27:0/2219722122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fece80b1800 msgr2=0x7fece80ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.572+0000 7fece77fe700 1 --2- 172.21.15.27:0/2219722122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fece80b1800 0x7fece80ca1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.572+0000 7fece77fe700 1 -- 172.21.15.27:0/2219722122 >> v2:172.21.15.77:3300/0 conn(0x7fece81575e0 msgr2=0x7fece8158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.572+0000 7fece77fe700 1 --2- 172.21.15.27:0/2219722122 >> v2:172.21.15.77:3300/0 conn(0x7fece81575e0 0x7fece8158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.573+0000 7fece77fe700 1 -- 172.21.15.27:0/2219722122 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece81599b0 con 0x7fece806b610 2024-01-11T08:25:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.573+0000 7fece7fff700 1 --2- 172.21.15.27:0/2219722122 >> v2:172.21.15.77:3300/0 conn(0x7fece81575e0 0x7fece8158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.573+0000 7fecec99c700 1 --2- 172.21.15.27:0/2219722122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fece80b1800 0x7fece80ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.573+0000 7fece77fe700 1 --2- 172.21.15.27:0/2219722122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fece806b610 0x7fece80b12c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fece8064b00 tx=0x7fecdc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b404e8618c57d687 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:44.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.573+0000 7fece6ffd700 1 -- 172.21.15.27:0/2219722122 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecdc00b640 con 0x7fece806b610 2024-01-11T08:25:44.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.573+0000 7fece6ffd700 1 -- 172.21.15.27:0/2219722122 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecdc00b7a0 con 0x7fece806b610 2024-01-11T08:25:44.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.573+0000 7fece6ffd700 1 -- 172.21.15.27:0/2219722122 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecdc00bb60 con 0x7fece806b610 2024-01-11T08:25:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.574+0000 7fecee3ff700 1 -- 172.21.15.27:0/2219722122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fece806b610 msgr2=0x7fece80b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.574+0000 7fecee3ff700 1 --2- 172.21.15.27:0/2219722122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fece806b610 0x7fece80b12c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fece8064b00 tx=0x7fecdc009250 comp rx=0 tx=0).stop 2024-01-11T08:25:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.574+0000 7fecee3ff700 1 -- 172.21.15.27:0/2219722122 shutdown_connections 2024-01-11T08:25:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.574+0000 7fecee3ff700 1 --2- 172.21.15.27:0/2219722122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fece80b1800 0x7fece80ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.574+0000 7fecee3ff700 1 --2- 172.21.15.27:0/2219722122 >> v2:172.21.15.77:3300/0 conn(0x7fece81575e0 0x7fece8158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.574+0000 7fecee3ff700 1 --2- 172.21.15.27:0/2219722122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fece806b610 0x7fece80b12c0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.574+0000 7fecee3ff700 1 -- 172.21.15.27:0/2219722122 >> 172.21.15.27:0/2219722122 conn(0x7fece80680e0 msgr2=0x7fece8069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.574+0000 7fecee3ff700 1 -- 172.21.15.27:0/2219722122 shutdown_connections 2024-01-11T08:25:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.574+0000 7fecee3ff700 1 -- 172.21.15.27:0/2219722122 wait complete. 2024-01-11T08:25:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.575+0000 7fecee3ff700 1 Processor -- start 2024-01-11T08:25:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.576+0000 7fecee3ff700 1 -- start start 2024-01-11T08:25:44.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.576+0000 7fecee3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fece806b610 0x7fece80c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:44.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.577+0000 7fecee3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fece80b1800 0x7fece816b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:44.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.577+0000 7fecee3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fece81575e0 0x7fece815d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:44.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.577+0000 7fecee3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece80c6160 con 0x7fece80b1800 2024-01-11T08:25:44.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.577+0000 7fecee3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece816b130 con 0x7fece806b610 2024-01-11T08:25:44.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.577+0000 7fecee3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece80c6390 con 0x7fece81575e0 2024-01-11T08:25:44.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.577+0000 7fece77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fece80b1800 0x7fece816b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:44.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.577+0000 7fece7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fece806b610 0x7fece80c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:44.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.577+0000 7fece7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fece806b610 0x7fece80c5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:39050/0 (socket says 172.21.15.27:39050) 2024-01-11T08:25:44.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.577+0000 7fece7fff700 1 -- 172.21.15.27:0/67103262 learned_addr learned my addr 172.21.15.27:0/67103262 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:44.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.577+0000 7fece77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fece80b1800 0x7fece816b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41620/0 (socket says 172.21.15.27:41620) 2024-01-11T08:25:44.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.577+0000 7fecec99c700 1 --2- 172.21.15.27:0/67103262 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fece81575e0 0x7fece815d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:44.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.577+0000 7fece7fff700 1 -- 172.21.15.27:0/67103262 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fece81575e0 msgr2=0x7fece815d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:44.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.577+0000 7fece7fff700 1 --2- 172.21.15.27:0/67103262 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fece81575e0 0x7fece815d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.577+0000 7fece7fff700 1 -- 172.21.15.27:0/67103262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fece80b1800 msgr2=0x7fece816b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:44.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.577+0000 7fece7fff700 1 --2- 172.21.15.27:0/67103262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fece80b1800 0x7fece816b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.577+0000 7fece7fff700 1 -- 172.21.15.27:0/67103262 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece0009a20 con 0x7fece806b610 2024-01-11T08:25:44.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.578+0000 7fece7fff700 1 --2- 172.21.15.27:0/67103262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fece806b610 0x7fece80c5c20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fecd800ec80 tx=0x7fecd80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:44.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.578+0000 7feccffff700 1 -- 172.21.15.27:0/67103262 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd800c850 con 0x7fece806b610 2024-01-11T08:25:44.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.578+0000 7feccffff700 1 -- 172.21.15.27:0/67103262 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd800c9b0 con 0x7fece806b610 2024-01-11T08:25:44.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.578+0000 7fecee3ff700 1 -- 172.21.15.27:0/67103262 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd8009420 con 0x7fece806b610 2024-01-11T08:25:44.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.578+0000 7feccffff700 1 -- 172.21.15.27:0/67103262 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd800cd30 con 0x7fece806b610 2024-01-11T08:25:44.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.578+0000 7fecee3ff700 1 -- 172.21.15.27:0/67103262 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecdc0094f0 con 0x7fece806b610 2024-01-11T08:25:44.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.579+0000 7fecee3ff700 1 -- 172.21.15.27:0/67103262 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece80718d0 con 0x7fece806b610 2024-01-11T08:25:44.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.581+0000 7feccffff700 1 -- 172.21.15.27:0/67103262 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fecd8013680 con 0x7fece806b610 2024-01-11T08:25:44.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.582+0000 7feccffff700 1 --2- 172.21.15.27:0/67103262 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fecd4067050 0x7fecd4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:44.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.582+0000 7feccffff700 1 -- 172.21.15.27:0/67103262 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5991+0+0 (secure 0 0 0) 0x7fecd8084d30 con 0x7fece806b610 2024-01-11T08:25:44.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.582+0000 7fece77fe700 1 -- 172.21.15.27:0/67103262 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fecd4067050 msgr2=0x7fecd4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:44.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.582+0000 7fece77fe700 1 --2- 172.21.15.27:0/67103262 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fecd4067050 0x7fecd4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:44.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.587+0000 7feccffff700 1 -- 172.21.15.27:0/67103262 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fecd80558c0 con 0x7fece806b610 2024-01-11T08:25:44.667 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:44.668+0000 7ff67d0f81c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:25:44.748 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:44.749+0000 7f01ec3381c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:25:44.769 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:44.771+0000 7ff67d0f81c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:25:44.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.783+0000 7fece77fe700 1 -- 172.21.15.27:0/67103262 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fecd4067050 msgr2=0x7fecd4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:44.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.783+0000 7fece77fe700 1 --2- 172.21.15.27:0/67103262 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fecd4067050 0x7fecd4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:44.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.793+0000 7fecee3ff700 1 -- 172.21.15.27:0/67103262 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fece8069880 con 0x7fece806b610 2024-01-11T08:25:44.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.794+0000 7feccffff700 1 -- 172.21.15.27:0/67103262 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+15710 (secure 0 0 0) 0x7fecd8017090 con 0x7fece806b610 2024-01-11T08:25:44.793 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:44.793 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":76,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:25:39.051951+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:25:21.160295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-11T08:25:22.342780+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/32753357":"2024-01-12T08:25:39.051906+0000","172.21.15.77:6832/41381":"2024-01-12T08:25:39.051906+0000","172.21.15.77:0/3311724933":"2024-01-12T08:25:39.051906+0000","172.21.15.77:6833/41381":"2024-01-12T08:25:39.051906+0000","172.21.15.77:0/2372882029":"2024-01-12T08:25:39.051906+0000","172.21.15.77:0/3065749352":"2024-01-12T08:25:39.051906+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:25:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.796+0000 7fecee3ff700 1 -- 172.21.15.27:0/67103262 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fecd4067050 msgr2=0x7fecd4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.796+0000 7fecee3ff700 1 --2- 172.21.15.27:0/67103262 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fecd4067050 0x7fecd4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.796+0000 7fecee3ff700 1 -- 172.21.15.27:0/67103262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fece806b610 msgr2=0x7fece80c5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.796+0000 7fecee3ff700 1 --2- 172.21.15.27:0/67103262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fece806b610 0x7fece80c5c20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fecd800ec80 tx=0x7fecd80091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.796+0000 7fecee3ff700 1 -- 172.21.15.27:0/67103262 shutdown_connections 2024-01-11T08:25:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.796+0000 7fecee3ff700 1 --2- 172.21.15.27:0/67103262 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fecd4067050 0x7fecd4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.796+0000 7fecee3ff700 1 --2- 172.21.15.27:0/67103262 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fece81575e0 0x7fece815d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.796+0000 7fecee3ff700 1 --2- 172.21.15.27:0/67103262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fece806b610 0x7fece80c5c20 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.796+0000 7fecee3ff700 1 --2- 172.21.15.27:0/67103262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fece80b1800 0x7fece816b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.796+0000 7fecee3ff700 1 -- 172.21.15.27:0/67103262 >> 172.21.15.27:0/67103262 conn(0x7fece80680e0 msgr2=0x7fece8159d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.796+0000 7fecee3ff700 1 -- 172.21.15.27:0/67103262 shutdown_connections 2024-01-11T08:25:44.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:44.796+0000 7fecee3ff700 1 -- 172.21.15.27:0/67103262 wait complete. 2024-01-11T08:25:44.814 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-11T08:25:44.814 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:25:44.846 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:44.847+0000 7f01ec3381c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:25:44.877 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:44.878+0000 7ff67d0f81c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:25:45.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.020+0000 7f7ceb9ff700 1 Processor -- start 2024-01-11T08:25:45.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.020+0000 7f7ceb9ff700 1 -- start start 2024-01-11T08:25:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.022+0000 7f7ceb9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7ce406b610 0x7f7ce4158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.022+0000 7f7ceb9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce4158b60 con 0x7f7ce40b19d0 2024-01-11T08:25:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.022+0000 7f7ceb9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce41599b0 con 0x7f7ce41575e0 2024-01-11T08:25:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.022+0000 7f7ceb9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ce40cb5e0 con 0x7f7ce406b610 2024-01-11T08:25:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.022+0000 7f7ce8f9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7ce40b19d0 0x7f7ce4159e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36278/0 (socket says 172.21.15.27:36278) 2024-01-11T08:25:45.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.022+0000 7f7ce8f9a700 1 -- 172.21.15.27:0/2039786600 learned_addr learned my addr 172.21.15.27:0/2039786600 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:45.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.022+0000 7f7ce9f9c700 1 --2- 172.21.15.27:0/2039786600 >> v2:172.21.15.77:3300/0 conn(0x7f7ce406b610 0x7f7ce4158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.022+0000 7f7cdbfff700 1 -- 172.21.15.27:0/2039786600 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714235809 0 0) 0x7f7cd4001110 con 0x7f7ce40b19d0 2024-01-11T08:25:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.023+0000 7f7cdbfff700 1 -- 172.21.15.27:0/2039786600 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cd0003570 con 0x7f7ce40b19d0 2024-01-11T08:25:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.023+0000 7f7ce9f9c700 1 -- 172.21.15.27:0/2039786600 >> v1:172.21.15.77:6789/0 conn(0x7f7ce41575e0 legacy=0x7f7ce40b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:45.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.023+0000 7f7cdbfff700 1 -- 172.21.15.27:0/2039786600 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554258865 0 0) 0x7f7ce41599b0 con 0x7f7ce41575e0 2024-01-11T08:25:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.023+0000 7f7ce9f9c700 1 -- 172.21.15.27:0/2039786600 >> v1:172.21.15.27:6789/0 conn(0x7f7ce40b19d0 legacy=0x7f7ce4159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.023+0000 7f7ce9f9c700 1 -- 172.21.15.27:0/2039786600 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce40c0f90 con 0x7f7ce406b610 2024-01-11T08:25:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.023+0000 7f7ce9f9c700 1 --2- 172.21.15.27:0/2039786600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce406b610 0x7f7ce4158620 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc009e60 tx=0x7f7cdc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e58d79243fe62e01 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.023+0000 7f7cdbfff700 1 -- 172.21.15.27:0/2039786600 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc00e070 con 0x7f7ce406b610 2024-01-11T08:25:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.023+0000 7f7cdbfff700 1 -- 172.21.15.27:0/2039786600 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cdc00bb60 con 0x7f7ce406b610 2024-01-11T08:25:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.023+0000 7f7cdbfff700 1 -- 172.21.15.27:0/2039786600 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc012430 con 0x7f7ce406b610 2024-01-11T08:25:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.024+0000 7f7ceb9ff700 1 -- 172.21.15.27:0/2039786600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce406b610 msgr2=0x7f7ce4158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.024+0000 7f7ceb9ff700 1 --2- 172.21.15.27:0/2039786600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce406b610 0x7f7ce4158620 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc009e60 tx=0x7f7cdc009780 comp rx=0 tx=0).stop 2024-01-11T08:25:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.024+0000 7f7ceb9ff700 1 -- 172.21.15.27:0/2039786600 shutdown_connections 2024-01-11T08:25:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.024+0000 7f7ceb9ff700 1 --2- 172.21.15.27:0/2039786600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce406b610 0x7f7ce4158620 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.024+0000 7f7ceb9ff700 1 -- 172.21.15.27:0/2039786600 >> 172.21.15.27:0/2039786600 conn(0x7f7ce40680e0 msgr2=0x7f7ce4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.024+0000 7f7ceb9ff700 1 -- 172.21.15.27:0/2039786600 shutdown_connections 2024-01-11T08:25:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.024+0000 7f7ceb9ff700 1 -- 172.21.15.27:0/2039786600 wait complete. 2024-01-11T08:25:45.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.026+0000 7f7ceb9ff700 1 Processor -- start 2024-01-11T08:25:45.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.026+0000 7f7ceb9ff700 1 -- start start 2024-01-11T08:25:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.027+0000 7f7ceb9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce406b610 0x7f7ce416cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.027+0000 7f7ceb9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ce40b19d0 0x7f7ce40c6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.027+0000 7f7ceb9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce41575e0 0x7f7ce415d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.027+0000 7f7ceb9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce40c7080 con 0x7f7ce406b610 2024-01-11T08:25:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.027+0000 7f7ceb9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce416a5c0 con 0x7f7ce41575e0 2024-01-11T08:25:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.027+0000 7f7ceb9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce40c6ac0 con 0x7f7ce40b19d0 2024-01-11T08:25:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.027+0000 7f7ce979b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce406b610 0x7f7ce416cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.027+0000 7f7ce979b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce406b610 0x7f7ce416cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41628/0 (socket says 172.21.15.27:41628) 2024-01-11T08:25:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.027+0000 7f7ce979b700 1 -- 172.21.15.27:0/3844968166 learned_addr learned my addr 172.21.15.27:0/3844968166 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.027+0000 7f7ce8f9a700 1 --2- 172.21.15.27:0/3844968166 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ce40b19d0 0x7f7ce40c6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.027+0000 7f7ce9f9c700 1 --2- 172.21.15.27:0/3844968166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce41575e0 0x7f7ce415d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:45.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.028+0000 7f7ce979b700 1 -- 172.21.15.27:0/3844968166 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ce40b19d0 msgr2=0x7f7ce40c6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:45.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.028+0000 7f7ce979b700 1 --2- 172.21.15.27:0/3844968166 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ce40b19d0 0x7f7ce40c6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:45.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.028+0000 7f7ce979b700 1 -- 172.21.15.27:0/3844968166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce41575e0 msgr2=0x7f7ce415d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:45.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.028+0000 7f7ce979b700 1 --2- 172.21.15.27:0/3844968166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce41575e0 0x7f7ce415d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:45.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.028+0000 7f7ce979b700 1 -- 172.21.15.27:0/3844968166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cdc009a20 con 0x7f7ce406b610 2024-01-11T08:25:45.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.028+0000 7f7ce979b700 1 --2- 172.21.15.27:0/3844968166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce406b610 0x7f7ce416cbb0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000c530 tx=0x7f7ce000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:45.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.028+0000 7f7cd97fa700 1 -- 172.21.15.27:0/3844968166 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0011590 con 0x7f7ce406b610 2024-01-11T08:25:45.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.028+0000 7f7cd97fa700 1 -- 172.21.15.27:0/3844968166 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce0011710 con 0x7f7ce406b610 2024-01-11T08:25:45.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.028+0000 7f7ceb9ff700 1 -- 172.21.15.27:0/3844968166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce40c6da0 con 0x7f7ce406b610 2024-01-11T08:25:45.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.028+0000 7f7cd97fa700 1 -- 172.21.15.27:0/3844968166 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0011ad0 con 0x7f7ce406b610 2024-01-11T08:25:45.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.028+0000 7f7ceb9ff700 1 -- 172.21.15.27:0/3844968166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce416d0f0 con 0x7f7ce406b610 2024-01-11T08:25:45.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.030+0000 7f7ceb9ff700 1 -- 172.21.15.27:0/3844968166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce416d310 con 0x7f7ce406b610 2024-01-11T08:25:45.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.031+0000 7f7cd97fa700 1 -- 172.21.15.27:0/3844968166 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f7ce001b3b0 con 0x7f7ce406b610 2024-01-11T08:25:45.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.032+0000 7f7cd97fa700 1 --2- 172.21.15.27:0/3844968166 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7cc8067050 0x7f7cc8069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:45.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.032+0000 7f7cd97fa700 1 -- 172.21.15.27:0/3844968166 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5991+0+0 (secure 0 0 0) 0x7f7ce0083c50 con 0x7f7ce406b610 2024-01-11T08:25:45.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.032+0000 7f7ce8f9a700 1 -- 172.21.15.27:0/3844968166 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7cc8067050 msgr2=0x7f7cc8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:45.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.032+0000 7f7ce8f9a700 1 --2- 172.21.15.27:0/3844968166 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7cc8067050 0x7f7cc8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:45.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.037+0000 7f7cd97fa700 1 -- 172.21.15.27:0/3844968166 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7ce0089020 con 0x7f7ce406b610 2024-01-11T08:25:45.083 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:45.084+0000 7ff67d0f81c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:25:45.188 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:45.189+0000 7ff67d0f81c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:25:45.191 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:45.190+0000 7f01ec3381c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:25:45.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.233+0000 7f7ce8f9a700 1 -- 172.21.15.27:0/3844968166 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7cc8067050 msgr2=0x7f7cc8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:45.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.233+0000 7f7ce8f9a700 1 --2- 172.21.15.27:0/3844968166 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7cc8067050 0x7f7cc8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:45.278 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:45.279+0000 7f01ec3381c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:25:45.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.289+0000 7f7ceb9ff700 1 -- 172.21.15.27:0/3844968166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ce40718d0 con 0x7f7ce406b610 2024-01-11T08:25:45.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.289+0000 7f7cd97fa700 1 -- 172.21.15.27:0/3844968166 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4597 (secure 0 0 0) 0x7f7ce0017070 con 0x7f7ce406b610 2024-01-11T08:25:45.288 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:45.289 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5606,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2845124438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2845124438},{"type":"v1","addr":"172.21.15.77:6837","nonce":2845124438}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5618,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3958104902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3958104902},{"type":"v1","addr":"172.21.15.27:6835","nonce":3958104902}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":6091,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/2612413022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2612413022},{"type":"v1","addr":"172.21.15.27:6833","nonce":2612413022}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:25:23.345532+0000","modified":"2024-01-11T08:25:26.867139+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6174},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6174":{"gid":6174,"name":"d","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6836/3056318454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3056318454},{"type":"v1","addr":"172.21.15.77:6836","nonce":3056318454}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2024-01-11T08:25:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.292+0000 7f7ceb9ff700 1 -- 172.21.15.27:0/3844968166 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7cc8067050 msgr2=0x7f7cc8069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.292+0000 7f7ceb9ff700 1 --2- 172.21.15.27:0/3844968166 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7cc8067050 0x7f7cc8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.292+0000 7f7ceb9ff700 1 -- 172.21.15.27:0/3844968166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce406b610 msgr2=0x7f7ce416cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.292+0000 7f7ceb9ff700 1 --2- 172.21.15.27:0/3844968166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce406b610 0x7f7ce416cbb0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000c530 tx=0x7f7ce000c840 comp rx=0 tx=0).stop 2024-01-11T08:25:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.292+0000 7f7ceb9ff700 1 -- 172.21.15.27:0/3844968166 shutdown_connections 2024-01-11T08:25:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.292+0000 7f7ceb9ff700 1 --2- 172.21.15.27:0/3844968166 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7f7cc8067050 0x7f7cc8069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.292+0000 7f7ceb9ff700 1 --2- 172.21.15.27:0/3844968166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce41575e0 0x7f7ce415d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.292+0000 7f7ceb9ff700 1 --2- 172.21.15.27:0/3844968166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce406b610 0x7f7ce416cbb0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.292+0000 7f7ceb9ff700 1 --2- 172.21.15.27:0/3844968166 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ce40b19d0 0x7f7ce40c6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.292+0000 7f7ceb9ff700 1 -- 172.21.15.27:0/3844968166 >> 172.21.15.27:0/3844968166 conn(0x7f7ce40680e0 msgr2=0x7f7ce406e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.292+0000 7f7ceb9ff700 1 -- 172.21.15.27:0/3844968166 shutdown_connections 2024-01-11T08:25:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.292+0000 7f7ceb9ff700 1 -- 172.21.15.27:0/3844968166 wait complete. 2024-01-11T08:25:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 33 2024-01-11T08:25:45.309 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:25:45.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.517+0000 7fd0d17ff700 1 Processor -- start 2024-01-11T08:25:45.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.517+0000 7fd0d17ff700 1 -- start start 2024-01-11T08:25:45.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.518+0000 7fd0d17ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd0cc069d40 0x7fd0cc0b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:45.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.518+0000 7fd0d17ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0cc0c89f0 0x7fd0cc0b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:45.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.518+0000 7fd0d17ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0cc157390 con 0x7fd0cc0c8620 2024-01-11T08:25:45.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.518+0000 7fd0d17ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0cc0adcd0 con 0x7fd0cc069d40 2024-01-11T08:25:45.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.518+0000 7fd0d17ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc06a200 con 0x7fd0cc0c89f0 2024-01-11T08:25:45.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.518+0000 7fd0ca7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd0cc069d40 0x7fd0cc0b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:45.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.519+0000 7fd0ca7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd0cc069d40 0x7fd0cc0b02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41644/0 (socket says 172.21.15.27:41644) 2024-01-11T08:25:45.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.519+0000 7fd0ca7fc700 1 -- 172.21.15.27:0/2530768040 learned_addr learned my addr 172.21.15.27:0/2530768040 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:45.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.519+0000 7fd0cb7fe700 1 --2- 172.21.15.27:0/2530768040 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0cc0c89f0 0x7fd0cc0b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:45.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.519+0000 7fd0ca7fc700 1 -- 172.21.15.27:0/2530768040 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0cc0c89f0 msgr2=0x7fd0cc0b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:45.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.519+0000 7fd0ca7fc700 1 --2- 172.21.15.27:0/2530768040 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0cc0c89f0 0x7fd0cc0b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.519+0000 7fd0ca7fc700 1 -- 172.21.15.27:0/2530768040 >> v1:172.21.15.77:6789/0 conn(0x7fd0cc0c8620 legacy=0x7fd0cc0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.519+0000 7fd0ca7fc700 1 -- 172.21.15.27:0/2530768040 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0cc156d40 con 0x7fd0cc069d40 2024-01-11T08:25:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.519+0000 7fd0ca7fc700 1 --2- 172.21.15.27:0/2530768040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc069d40 0x7fd0cc0b02d0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000bc50 tx=0x7fd0c000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ac0e9f433af443 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.519+0000 7fd0c9ffb700 1 -- 172.21.15.27:0/2530768040 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c000dbf0 con 0x7fd0cc069d40 2024-01-11T08:25:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.520+0000 7fd0c9ffb700 1 -- 172.21.15.27:0/2530768040 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c000dd50 con 0x7fd0cc069d40 2024-01-11T08:25:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.520+0000 7fd0c9ffb700 1 -- 172.21.15.27:0/2530768040 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c00196b0 con 0x7fd0cc069d40 2024-01-11T08:25:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.520+0000 7fd0d17ff700 1 -- 172.21.15.27:0/2530768040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc069d40 msgr2=0x7fd0cc0b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.520+0000 7fd0d17ff700 1 --2- 172.21.15.27:0/2530768040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc069d40 0x7fd0cc0b02d0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000bc50 tx=0x7fd0c000b4f0 comp rx=0 tx=0).stop 2024-01-11T08:25:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.520+0000 7fd0d17ff700 1 -- 172.21.15.27:0/2530768040 shutdown_connections 2024-01-11T08:25:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.520+0000 7fd0d17ff700 1 --2- 172.21.15.27:0/2530768040 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0cc0c89f0 0x7fd0cc0b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.520+0000 7fd0d17ff700 1 --2- 172.21.15.27:0/2530768040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc069d40 0x7fd0cc0b02d0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.520+0000 7fd0d17ff700 1 -- 172.21.15.27:0/2530768040 >> 172.21.15.27:0/2530768040 conn(0x7fd0cc0680e0 msgr2=0x7fd0cc069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.520+0000 7fd0d17ff700 1 -- 172.21.15.27:0/2530768040 shutdown_connections 2024-01-11T08:25:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.520+0000 7fd0d17ff700 1 -- 172.21.15.27:0/2530768040 wait complete. 2024-01-11T08:25:45.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.522+0000 7fd0d17ff700 1 Processor -- start 2024-01-11T08:25:45.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.522+0000 7fd0d17ff700 1 -- start start 2024-01-11T08:25:45.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.523+0000 7fd0d17ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc069d40 0x7fd0cc0a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:45.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.523+0000 7fd0d17ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0cc0c8620 0x7fd0cc0ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:45.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.523+0000 7fd0d17ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0cc0c89f0 0x7fd0cc0c7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:45.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.523+0000 7fd0d17ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc16af60 con 0x7fd0cc069d40 2024-01-11T08:25:45.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.523+0000 7fd0d17ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc16a3e0 con 0x7fd0cc0c89f0 2024-01-11T08:25:45.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.523+0000 7fd0d17ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc16a6c0 con 0x7fd0cc0c8620 2024-01-11T08:25:45.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.523+0000 7fd0caffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc069d40 0x7fd0cc0a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:45.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.523+0000 7fd0caffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc069d40 0x7fd0cc0a5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41658/0 (socket says 172.21.15.27:41658) 2024-01-11T08:25:45.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.523+0000 7fd0ca7fc700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0cc0c8620 0x7fd0cc0ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:45.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.523+0000 7fd0caffd700 1 -- 172.21.15.27:0/4194967999 learned_addr learned my addr 172.21.15.27:0/4194967999 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:45.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.523+0000 7fd0cb7fe700 1 --2- 172.21.15.27:0/4194967999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0cc0c89f0 0x7fd0cc0c7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:45.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.524+0000 7fd0ca7fc700 1 -- 172.21.15.27:0/4194967999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0cc0c89f0 msgr2=0x7fd0cc0c7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:45.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.524+0000 7fd0ca7fc700 1 --2- 172.21.15.27:0/4194967999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0cc0c89f0 0x7fd0cc0c7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:45.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.524+0000 7fd0ca7fc700 1 -- 172.21.15.27:0/4194967999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc069d40 msgr2=0x7fd0cc0a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:45.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.524+0000 7fd0ca7fc700 1 --2- 172.21.15.27:0/4194967999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc069d40 0x7fd0cc0a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:45.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.524+0000 7fd0ca7fc700 1 -- 172.21.15.27:0/4194967999 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c000b790 con 0x7fd0cc0c8620 2024-01-11T08:25:45.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.524+0000 7fd0caffd700 1 --2- 172.21.15.27:0/4194967999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc069d40 0x7fd0cc0a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:45.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.524+0000 7fd0ca7fc700 1 --2- 172.21.15.27:0/4194967999 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0cc0c8620 0x7fd0cc0ca340 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc064ad0 tx=0x7fd0c001b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:45.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.524+0000 7fd0b2ffd700 1 -- 172.21.15.27:0/4194967999 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0010680 con 0x7fd0cc0c8620 2024-01-11T08:25:45.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.524+0000 7fd0b2ffd700 1 -- 172.21.15.27:0/4194967999 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c00107e0 con 0x7fd0cc0c8620 2024-01-11T08:25:45.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.524+0000 7fd0d17ff700 1 -- 172.21.15.27:0/4194967999 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0cc16a9a0 con 0x7fd0cc0c8620 2024-01-11T08:25:45.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.524+0000 7fd0b2ffd700 1 -- 172.21.15.27:0/4194967999 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0010ba0 con 0x7fd0cc0c8620 2024-01-11T08:25:45.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.524+0000 7fd0d17ff700 1 -- 172.21.15.27:0/4194967999 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0cc0a59d0 con 0x7fd0cc0c8620 2024-01-11T08:25:45.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.525+0000 7fd0d17ff700 1 -- 172.21.15.27:0/4194967999 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0cc0718d0 con 0x7fd0cc0c8620 2024-01-11T08:25:45.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.527+0000 7fd0b2ffd700 1 -- 172.21.15.27:0/4194967999 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fd0c002c430 con 0x7fd0cc0c8620 2024-01-11T08:25:45.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.527+0000 7fd0b2ffd700 1 --2- 172.21.15.27:0/4194967999 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd0b8067050 0x7fd0b8069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:45.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.527+0000 7fd0b2ffd700 1 -- 172.21.15.27:0/4194967999 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5991+0+0 (secure 0 0 0) 0x7fd0c0014070 con 0x7fd0cc0c8620 2024-01-11T08:25:45.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.528+0000 7fd0caffd700 1 -- 172.21.15.27:0/4194967999 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd0b8067050 msgr2=0x7fd0b8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:45.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.528+0000 7fd0caffd700 1 --2- 172.21.15.27:0/4194967999 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd0b8067050 0x7fd0b8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:45.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.532+0000 7fd0b2ffd700 1 -- 172.21.15.27:0/4194967999 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd0c0053490 con 0x7fd0cc0c8620 2024-01-11T08:25:45.560 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:45.561+0000 7ff67d0f81c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:25:45.655 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:45.656+0000 7ff67d0f81c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:25:45.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.729+0000 7fd0caffd700 1 -- 172.21.15.27:0/4194967999 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd0b8067050 msgr2=0x7fd0b8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:45.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.729+0000 7fd0caffd700 1 --2- 172.21.15.27:0/4194967999 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd0b8067050 0x7fd0b8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.781+0000 7fd0d17ff700 1 -- 172.21.15.27:0/4194967999 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0cc157020 con 0x7fd0cc0c8620 2024-01-11T08:25:45.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.782+0000 7fd0b2ffd700 1 -- 172.21.15.27:0/4194967999 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4597 (secure 0 0 0) 0x7fd0c0056ab0 con 0x7fd0cc0c8620 2024-01-11T08:25:45.781 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:45.781 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5606,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2845124438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2845124438},{"type":"v1","addr":"172.21.15.77:6837","nonce":2845124438}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5618,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3958104902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3958104902},{"type":"v1","addr":"172.21.15.27:6835","nonce":3958104902}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":6091,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/2612413022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2612413022},{"type":"v1","addr":"172.21.15.27:6833","nonce":2612413022}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:25:23.345532+0000","modified":"2024-01-11T08:25:26.867139+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6174},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6174":{"gid":6174,"name":"d","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6836/3056318454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":3056318454},{"type":"v1","addr":"172.21.15.77:6836","nonce":3056318454}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2024-01-11T08:25:45.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.784+0000 7fd0d17ff700 1 -- 172.21.15.27:0/4194967999 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd0b8067050 msgr2=0x7fd0b8069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:45.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.785+0000 7fd0d17ff700 1 --2- 172.21.15.27:0/4194967999 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd0b8067050 0x7fd0b8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:45.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.785+0000 7fd0d17ff700 1 -- 172.21.15.27:0/4194967999 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0cc0c8620 msgr2=0x7fd0cc0ca340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:45.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.785+0000 7fd0d17ff700 1 --2- 172.21.15.27:0/4194967999 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0cc0c8620 0x7fd0cc0ca340 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc064ad0 tx=0x7fd0c001b000 comp rx=0 tx=0).stop 2024-01-11T08:25:45.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.785+0000 7fd0d17ff700 1 -- 172.21.15.27:0/4194967999 shutdown_connections 2024-01-11T08:25:45.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.785+0000 7fd0d17ff700 1 --2- 172.21.15.27:0/4194967999 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd0b8067050 0x7fd0b8069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:45.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.785+0000 7fd0d17ff700 1 --2- 172.21.15.27:0/4194967999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0cc0c89f0 0x7fd0cc0c7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:45.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.785+0000 7fd0d17ff700 1 --2- 172.21.15.27:0/4194967999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0cc069d40 0x7fd0cc0a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:45.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.785+0000 7fd0d17ff700 1 --2- 172.21.15.27:0/4194967999 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0cc0c8620 0x7fd0cc0ca340 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:45.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.785+0000 7fd0d17ff700 1 -- 172.21.15.27:0/4194967999 >> 172.21.15.27:0/4194967999 conn(0x7fd0cc0680e0 msgr2=0x7fd0cc155570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:45.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.785+0000 7fd0d17ff700 1 -- 172.21.15.27:0/4194967999 shutdown_connections 2024-01-11T08:25:45.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:45.785+0000 7fd0d17ff700 1 -- 172.21.15.27:0/4194967999 wait complete. 2024-01-11T08:25:45.785 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 33 2024-01-11T08:25:45.803 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:25:45.817 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:45.818+0000 7f01ec3381c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:25:46.005 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:46.006+0000 7f01ec3381c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:25:46.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.008+0000 7fd5537ff700 1 Processor -- start 2024-01-11T08:25:46.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.008+0000 7fd5537ff700 1 -- start start 2024-01-11T08:25:46.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.009+0000 7fd5537ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd54c0ca3e0 0x7fd54c0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:46.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.010+0000 7fd5537ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd54c159e80 0x7fd54c15dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:46.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.010+0000 7fd5537ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c15d640 con 0x7fd54c0ca7b0 2024-01-11T08:25:46.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.010+0000 7fd5537ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54c15ae70 con 0x7fd54c0ca3e0 2024-01-11T08:25:46.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.010+0000 7fd5537ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0cc4f0 con 0x7fd54c159e80 2024-01-11T08:25:46.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.010+0000 7fd55159b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd54c0ca3e0 0x7fd54c0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:46.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.010+0000 7fd551d9c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd54c159e80 0x7fd54c15dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:46.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.010+0000 7fd55159b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd54c0ca3e0 0x7fd54c0cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:39070/0 (socket says 172.21.15.27:39070) 2024-01-11T08:25:46.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.010+0000 7fd55159b700 1 -- 172.21.15.27:0/2956587876 learned_addr learned my addr 172.21.15.27:0/2956587876 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:46.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.011+0000 7fd551d9c700 1 -- 172.21.15.27:0/2956587876 >> v2:172.21.15.77:3300/0 conn(0x7fd54c0ca3e0 msgr2=0x7fd54c0cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:46.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.011+0000 7fd551d9c700 1 --2- 172.21.15.27:0/2956587876 >> v2:172.21.15.77:3300/0 conn(0x7fd54c0ca3e0 0x7fd54c0cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.011+0000 7fd551d9c700 1 -- 172.21.15.27:0/2956587876 >> v1:172.21.15.77:6789/0 conn(0x7fd54c0ca7b0 legacy=0x7fd54c0c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:46.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.011+0000 7fd551d9c700 1 -- 172.21.15.27:0/2956587876 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c15a5d0 con 0x7fd54c159e80 2024-01-11T08:25:46.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.011+0000 7fd55159b700 1 --2- 172.21.15.27:0/2956587876 >> v2:172.21.15.77:3300/0 conn(0x7fd54c0ca3e0 0x7fd54c0cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:46.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.011+0000 7fd543fff700 1 -- 172.21.15.27:0/2956587876 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463589157 0 0) 0x7fd54c15d640 con 0x7fd54c0ca7b0 2024-01-11T08:25:46.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.011+0000 7fd551d9c700 1 --2- 172.21.15.27:0/2956587876 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd54c159e80 0x7fd54c15dad0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd544009e90 tx=0x7fd544009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=935db0238a397777 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:46.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.011+0000 7fd543fff700 1 -- 172.21.15.27:0/2956587876 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54400e070 con 0x7fd54c159e80 2024-01-11T08:25:46.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.011+0000 7fd543fff700 1 -- 172.21.15.27:0/2956587876 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54400bbf0 con 0x7fd54c159e80 2024-01-11T08:25:46.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.011+0000 7fd543fff700 1 -- 172.21.15.27:0/2956587876 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd544012430 con 0x7fd54c159e80 2024-01-11T08:25:46.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.012+0000 7fd5537ff700 1 -- 172.21.15.27:0/2956587876 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd54c159e80 msgr2=0x7fd54c15dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:46.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.012+0000 7fd5537ff700 1 --2- 172.21.15.27:0/2956587876 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd54c159e80 0x7fd54c15dad0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd544009e90 tx=0x7fd544009780 comp rx=0 tx=0).stop 2024-01-11T08:25:46.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.012+0000 7fd5537ff700 1 -- 172.21.15.27:0/2956587876 shutdown_connections 2024-01-11T08:25:46.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.012+0000 7fd5537ff700 1 --2- 172.21.15.27:0/2956587876 >> v2:172.21.15.77:3300/0 conn(0x7fd54c0ca3e0 0x7fd54c0cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.012+0000 7fd5537ff700 1 --2- 172.21.15.27:0/2956587876 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd54c159e80 0x7fd54c15dad0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.012+0000 7fd5537ff700 1 -- 172.21.15.27:0/2956587876 >> 172.21.15.27:0/2956587876 conn(0x7fd54c0ce3e0 msgr2=0x7fd54c0ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:46.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.012+0000 7fd5537ff700 1 -- 172.21.15.27:0/2956587876 shutdown_connections 2024-01-11T08:25:46.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.012+0000 7fd5537ff700 1 -- 172.21.15.27:0/2956587876 wait complete. 2024-01-11T08:25:46.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.014+0000 7fd5537ff700 1 Processor -- start 2024-01-11T08:25:46.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.014+0000 7fd5537ff700 1 -- start start 2024-01-11T08:25:46.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.015+0000 7fd5537ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd54c0ca3e0 0x7fd54c150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:46.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.015+0000 7fd5537ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd54c0ca7b0 0x7fd54c169320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:46.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.015+0000 7fd5537ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd54c159e80 0x7fd54c0b2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:46.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.016+0000 7fd5537ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c151010 con 0x7fd54c0ca7b0 2024-01-11T08:25:46.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.016+0000 7fd5537ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c16c780 con 0x7fd54c159e80 2024-01-11T08:25:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.016+0000 7fd5537ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0b2270 con 0x7fd54c0ca3e0 2024-01-11T08:25:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.016+0000 7fd55159b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd54c0ca3e0 0x7fd54c150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.016+0000 7fd55159b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd54c0ca3e0 0x7fd54c150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:55140/0 (socket says 172.21.15.27:55140) 2024-01-11T08:25:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.016+0000 7fd55159b700 1 -- 172.21.15.27:0/2810689130 learned_addr learned my addr 172.21.15.27:0/2810689130 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.016+0000 7fd551d9c700 1 --2- 172.21.15.27:0/2810689130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd54c159e80 0x7fd54c0b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.016+0000 7fd550d9a700 1 --2- 172.21.15.27:0/2810689130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd54c0ca7b0 0x7fd54c169320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.016+0000 7fd551d9c700 1 -- 172.21.15.27:0/2810689130 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd54c0ca3e0 msgr2=0x7fd54c150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.016+0000 7fd551d9c700 1 --2- 172.21.15.27:0/2810689130 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd54c0ca3e0 0x7fd54c150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.016+0000 7fd551d9c700 1 -- 172.21.15.27:0/2810689130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd54c0ca7b0 msgr2=0x7fd54c169320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.016+0000 7fd551d9c700 1 --2- 172.21.15.27:0/2810689130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd54c0ca7b0 0x7fd54c169320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.016+0000 7fd551d9c700 1 -- 172.21.15.27:0/2810689130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5480094f0 con 0x7fd54c159e80 2024-01-11T08:25:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.016+0000 7fd55159b700 1 --2- 172.21.15.27:0/2810689130 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd54c0ca3e0 0x7fd54c150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.017+0000 7fd551d9c700 1 --2- 172.21.15.27:0/2810689130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd54c159e80 0x7fd54c0b2700 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd544016000 tx=0x7fd544019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.017+0000 7fd5417fa700 1 -- 172.21.15.27:0/2810689130 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54401b420 con 0x7fd54c159e80 2024-01-11T08:25:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.017+0000 7fd5417fa700 1 -- 172.21.15.27:0/2810689130 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54401b580 con 0x7fd54c159e80 2024-01-11T08:25:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.017+0000 7fd5417fa700 1 -- 172.21.15.27:0/2810689130 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54401b940 con 0x7fd54c159e80 2024-01-11T08:25:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.017+0000 7fd5537ff700 1 -- 172.21.15.27:0/2810689130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd544009a20 con 0x7fd54c159e80 2024-01-11T08:25:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.017+0000 7fd5537ff700 1 -- 172.21.15.27:0/2810689130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54c0b1940 con 0x7fd54c159e80 2024-01-11T08:25:46.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.018+0000 7fd5537ff700 1 -- 172.21.15.27:0/2810689130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54c0b2420 con 0x7fd54c159e80 2024-01-11T08:25:46.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.019+0000 7fd5417fa700 1 -- 172.21.15.27:0/2810689130 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fd54401bcb0 con 0x7fd54c159e80 2024-01-11T08:25:46.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.020+0000 7fd5417fa700 1 --2- 172.21.15.27:0/2810689130 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd530067050 0x7fd530069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:46.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.020+0000 7fd5417fa700 1 -- 172.21.15.27:0/2810689130 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5991+0+0 (secure 0 0 0) 0x7fd5440864e0 con 0x7fd54c159e80 2024-01-11T08:25:46.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.020+0000 7fd55159b700 1 -- 172.21.15.27:0/2810689130 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd530067050 msgr2=0x7fd530069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:46.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.020+0000 7fd55159b700 1 --2- 172.21.15.27:0/2810689130 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd530067050 0x7fd530069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:46.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.025+0000 7fd5417fa700 1 -- 172.21.15.27:0/2810689130 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd544053410 con 0x7fd54c159e80 2024-01-11T08:25:46.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.221+0000 7fd55159b700 1 -- 172.21.15.27:0/2810689130 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd530067050 msgr2=0x7fd530069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:46.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.221+0000 7fd55159b700 1 --2- 172.21.15.27:0/2810689130 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd530067050 0x7fd530069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:46.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.229+0000 7fd5537ff700 1 -- 172.21.15.27:0/2810689130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd54c06c840 con 0x7fd54c159e80 2024-01-11T08:25:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.230+0000 7fd5417fa700 1 -- 172.21.15.27:0/2810689130 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+15710 (secure 0 0 0) 0x7fd544056a30 con 0x7fd54c159e80 2024-01-11T08:25:46.229 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:46.229 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":76,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:25:39.051951+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:25:21.160295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-11T08:25:22.342780+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/32753357":"2024-01-12T08:25:39.051906+0000","172.21.15.77:6832/41381":"2024-01-12T08:25:39.051906+0000","172.21.15.77:0/3311724933":"2024-01-12T08:25:39.051906+0000","172.21.15.77:6833/41381":"2024-01-12T08:25:39.051906+0000","172.21.15.77:0/2372882029":"2024-01-12T08:25:39.051906+0000","172.21.15.77:0/3065749352":"2024-01-12T08:25:39.051906+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:25:46.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.232+0000 7fd5537ff700 1 -- 172.21.15.27:0/2810689130 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd530067050 msgr2=0x7fd530069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.232+0000 7fd5537ff700 1 --2- 172.21.15.27:0/2810689130 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd530067050 0x7fd530069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.232+0000 7fd5537ff700 1 -- 172.21.15.27:0/2810689130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd54c159e80 msgr2=0x7fd54c0b2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.232+0000 7fd5537ff700 1 --2- 172.21.15.27:0/2810689130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd54c159e80 0x7fd54c0b2700 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd544016000 tx=0x7fd544019000 comp rx=0 tx=0).stop 2024-01-11T08:25:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.232+0000 7fd5537ff700 1 -- 172.21.15.27:0/2810689130 shutdown_connections 2024-01-11T08:25:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.232+0000 7fd5537ff700 1 --2- 172.21.15.27:0/2810689130 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fd530067050 0x7fd530069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.232+0000 7fd5537ff700 1 --2- 172.21.15.27:0/2810689130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd54c159e80 0x7fd54c0b2700 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.232+0000 7fd5537ff700 1 --2- 172.21.15.27:0/2810689130 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd54c0ca3e0 0x7fd54c150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.232+0000 7fd5537ff700 1 --2- 172.21.15.27:0/2810689130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd54c0ca7b0 0x7fd54c169320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.232+0000 7fd5537ff700 1 -- 172.21.15.27:0/2810689130 >> 172.21.15.27:0/2810689130 conn(0x7fd54c0ce3e0 msgr2=0x7fd54c0cacd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.232+0000 7fd5537ff700 1 -- 172.21.15.27:0/2810689130 shutdown_connections 2024-01-11T08:25:46.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.232+0000 7fd5537ff700 1 -- 172.21.15.27:0/2810689130 wait complete. 2024-01-11T08:25:46.251 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:46.251+0000 7ff67d0f81c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:25:46.252 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-11T08:25:46.384 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:46.385+0000 7f01ec3381c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:25:46.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.454+0000 7fb2e11ff700 1 Processor -- start 2024-01-11T08:25:46.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.454+0000 7fb2e11ff700 1 -- start start 2024-01-11T08:25:46.454 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:46.455+0000 7ff67d0f81c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:25:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.455+0000 7fb2e11ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb2dc155e30 0x7fb2dc0afc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.455+0000 7fb2e11ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc0aeb50 con 0x7fb2dc155a60 2024-01-11T08:25:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.455+0000 7fb2e11ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc0abf20 con 0x7fb2dc0b0190 2024-01-11T08:25:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.455+0000 7fb2e11ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2dc0b05d0 con 0x7fb2dc155e30 2024-01-11T08:25:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.456+0000 7fb2dad9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb2dc155a60 0x7fb2dc0aef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36282/0 (socket says 172.21.15.27:36282) 2024-01-11T08:25:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.456+0000 7fb2dad9d700 1 -- 172.21.15.27:0/2891903632 learned_addr learned my addr 172.21.15.27:0/2891903632 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.456+0000 7fb2da59c700 1 --2- 172.21.15.27:0/2891903632 >> v2:172.21.15.77:3300/0 conn(0x7fb2dc155e30 0x7fb2dc0afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.456+0000 7fb2d9d9b700 1 -- 172.21.15.27:0/2891903632 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672348772 0 0) 0x7fb2dc0aeb50 con 0x7fb2dc155a60 2024-01-11T08:25:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.456+0000 7fb2d9d9b700 1 -- 172.21.15.27:0/2891903632 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c4003570 con 0x7fb2dc155a60 2024-01-11T08:25:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.456+0000 7fb2da59c700 1 -- 172.21.15.27:0/2891903632 >> v1:172.21.15.77:6789/0 conn(0x7fb2dc0b0190 legacy=0x7fb2dc0ac3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.456+0000 7fb2da59c700 1 -- 172.21.15.27:0/2891903632 >> v1:172.21.15.27:6789/0 conn(0x7fb2dc155a60 legacy=0x7fb2dc0aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.456+0000 7fb2da59c700 1 -- 172.21.15.27:0/2891903632 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc0ae590 con 0x7fb2dc155e30 2024-01-11T08:25:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.457+0000 7fb2da59c700 1 --2- 172.21.15.27:0/2891903632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2dc155e30 0x7fb2dc0afc50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb2d00098e0 tx=0x7fb2d00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b96695c71652830 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.457+0000 7fb2d9d9b700 1 -- 172.21.15.27:0/2891903632 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d000b640 con 0x7fb2dc155e30 2024-01-11T08:25:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.457+0000 7fb2d9d9b700 1 -- 172.21.15.27:0/2891903632 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d000b7a0 con 0x7fb2dc155e30 2024-01-11T08:25:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.457+0000 7fb2d9d9b700 1 -- 172.21.15.27:0/2891903632 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d000bb40 con 0x7fb2dc155e30 2024-01-11T08:25:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.458+0000 7fb2e11ff700 1 -- 172.21.15.27:0/2891903632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2dc155e30 msgr2=0x7fb2dc0afc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.458+0000 7fb2e11ff700 1 --2- 172.21.15.27:0/2891903632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2dc155e30 0x7fb2dc0afc50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb2d00098e0 tx=0x7fb2d00091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.458+0000 7fb2e11ff700 1 -- 172.21.15.27:0/2891903632 shutdown_connections 2024-01-11T08:25:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.458+0000 7fb2e11ff700 1 --2- 172.21.15.27:0/2891903632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2dc155e30 0x7fb2dc0afc50 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.458+0000 7fb2e11ff700 1 -- 172.21.15.27:0/2891903632 >> 172.21.15.27:0/2891903632 conn(0x7fb2dc0680b0 msgr2=0x7fb2dc0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.458+0000 7fb2e11ff700 1 -- 172.21.15.27:0/2891903632 shutdown_connections 2024-01-11T08:25:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.458+0000 7fb2e11ff700 1 -- 172.21.15.27:0/2891903632 wait complete. 2024-01-11T08:25:46.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.459+0000 7fb2e11ff700 1 Processor -- start 2024-01-11T08:25:46.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.460+0000 7fb2e11ff700 1 -- start start 2024-01-11T08:25:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.460+0000 7fb2e11ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2dc0b0190 0x7fb2dc143cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.460+0000 7fb2e11ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb2dc155a60 0x7fb2dc169e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.460+0000 7fb2e11ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc155e30 0x7fb2dc1576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.460+0000 7fb2e11ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc144230 con 0x7fb2dc155e30 2024-01-11T08:25:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.460+0000 7fb2e11ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc1563c0 con 0x7fb2dc0b0190 2024-01-11T08:25:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.460+0000 7fb2e11ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc157220 con 0x7fb2dc155a60 2024-01-11T08:25:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.461+0000 7fb2dad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2dc0b0190 0x7fb2dc143cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.461+0000 7fb2db59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc155e30 0x7fb2dc1576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.461+0000 7fb2db59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc155e30 0x7fb2dc1576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41672/0 (socket says 172.21.15.27:41672) 2024-01-11T08:25:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.461+0000 7fb2db59e700 1 -- 172.21.15.27:0/2797501742 learned_addr learned my addr 172.21.15.27:0/2797501742 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.461+0000 7fb2da59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb2dc155a60 0x7fb2dc169e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.461+0000 7fb2db59e700 1 -- 172.21.15.27:0/2797501742 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb2dc155a60 msgr2=0x7fb2dc169e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.461+0000 7fb2db59e700 1 --2- 172.21.15.27:0/2797501742 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb2dc155a60 0x7fb2dc169e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.461+0000 7fb2db59e700 1 -- 172.21.15.27:0/2797501742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2dc0b0190 msgr2=0x7fb2dc143cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.461+0000 7fb2db59e700 1 --2- 172.21.15.27:0/2797501742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2dc0b0190 0x7fb2dc143cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.461+0000 7fb2db59e700 1 -- 172.21.15.27:0/2797501742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d0009420 con 0x7fb2dc155e30 2024-01-11T08:25:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.461+0000 7fb2dad9d700 1 --2- 172.21.15.27:0/2797501742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2dc0b0190 0x7fb2dc143cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.462+0000 7fb2db59e700 1 --2- 172.21.15.27:0/2797501742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc155e30 0x7fb2dc1576b0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb2d400c960 tx=0x7fb2d4001750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.462+0000 7fb2c2ffd700 1 -- 172.21.15.27:0/2797501742 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4012630 con 0x7fb2dc155e30 2024-01-11T08:25:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.462+0000 7fb2c2ffd700 1 -- 172.21.15.27:0/2797501742 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d4012790 con 0x7fb2dc155e30 2024-01-11T08:25:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.462+0000 7fb2c2ffd700 1 -- 172.21.15.27:0/2797501742 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4012b50 con 0x7fb2dc155e30 2024-01-11T08:25:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.462+0000 7fb2e11ff700 1 -- 172.21.15.27:0/2797501742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2dc156610 con 0x7fb2dc155e30 2024-01-11T08:25:46.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.462+0000 7fb2e11ff700 1 -- 172.21.15.27:0/2797501742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2dc156bd0 con 0x7fb2dc155e30 2024-01-11T08:25:46.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.463+0000 7fb2e11ff700 1 -- 172.21.15.27:0/2797501742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2dc1573b0 con 0x7fb2dc155e30 2024-01-11T08:25:46.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.465+0000 7fb2c2ffd700 1 -- 172.21.15.27:0/2797501742 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fb2d401a3a0 con 0x7fb2dc155e30 2024-01-11T08:25:46.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.466+0000 7fb2c2ffd700 1 --2- 172.21.15.27:0/2797501742 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb2c8067050 0x7fb2c8069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:46.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.466+0000 7fb2c2ffd700 1 -- 172.21.15.27:0/2797501742 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5991+0+0 (secure 0 0 0) 0x7fb2d4084a40 con 0x7fb2dc155e30 2024-01-11T08:25:46.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.467+0000 7fb2dad9d700 1 -- 172.21.15.27:0/2797501742 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb2c8067050 msgr2=0x7fb2c8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:46.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.467+0000 7fb2dad9d700 1 --2- 172.21.15.27:0/2797501742 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb2c8067050 0x7fb2c8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:25:46.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.471+0000 7fb2c2ffd700 1 -- 172.21.15.27:0/2797501742 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb2d4051a60 con 0x7fb2dc155e30 2024-01-11T08:25:46.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.668+0000 7fb2dad9d700 1 -- 172.21.15.27:0/2797501742 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb2c8067050 msgr2=0x7fb2c8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6832/41381 2024-01-11T08:25:46.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.668+0000 7fb2dad9d700 1 --2- 172.21.15.27:0/2797501742 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb2c8067050 0x7fb2c8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:25:46.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.721+0000 7fb2e11ff700 1 -- 172.21.15.27:0/2797501742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb2dc04c3c0 con 0x7fb2dc155e30 2024-01-11T08:25:46.721 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-11T08:25:46.722+0000 7f27065f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-11T08:25:46.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.726+0000 7fb2c2ffd700 1 -- 172.21.15.27:0/2797501742 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v34) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb2d4055080 con 0x7fb2dc155e30 2024-01-11T08:25:46.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.730+0000 7fb2e11ff700 1 -- 172.21.15.27:0/2797501742 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb2c8067050 msgr2=0x7fb2c8069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:25:46.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.730+0000 7fb2e11ff700 1 --2- 172.21.15.27:0/2797501742 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb2c8067050 0x7fb2c8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.730+0000 7fb2e11ff700 1 -- 172.21.15.27:0/2797501742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc155e30 msgr2=0x7fb2dc1576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:46.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.730+0000 7fb2e11ff700 1 --2- 172.21.15.27:0/2797501742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc155e30 0x7fb2dc1576b0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb2d400c960 tx=0x7fb2d4001750 comp rx=0 tx=0).stop 2024-01-11T08:25:46.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.730+0000 7fb2e11ff700 1 -- 172.21.15.27:0/2797501742 shutdown_connections 2024-01-11T08:25:46.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.730+0000 7fb2e11ff700 1 --2- 172.21.15.27:0/2797501742 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb2c8067050 0x7fb2c8069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.730+0000 7fb2e11ff700 1 --2- 172.21.15.27:0/2797501742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc155e30 0x7fb2dc1576b0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.730+0000 7fb2e11ff700 1 --2- 172.21.15.27:0/2797501742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2dc0b0190 0x7fb2dc143cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.730+0000 7fb2e11ff700 1 --2- 172.21.15.27:0/2797501742 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb2dc155a60 0x7fb2dc169e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.730+0000 7fb2e11ff700 1 -- 172.21.15.27:0/2797501742 >> 172.21.15.27:0/2797501742 conn(0x7fb2dc0680b0 msgr2=0x7fb2dc1526c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:46.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.730+0000 7fb2e11ff700 1 -- 172.21.15.27:0/2797501742 shutdown_connections 2024-01-11T08:25:46.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.730+0000 7fb2e11ff700 1 -- 172.21.15.27:0/2797501742 wait complete. 2024-01-11T08:25:46.730 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-11T08:25:46.731 INFO:tasks.ceph.mds.d.smithi077.stderr: -2287> 2024-01-11T08:25:24.818+0000 7f23283f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T08:25:46.750 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-11T08:25:46.777 INFO:tasks.ceph.mds.d.smithi077.stdout:starting mds.d at 2024-01-11T08:25:46.812 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:46.813+0000 7f01ec3381c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:25:46.866 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:46.867+0000 7ff67d0f81c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:25:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.959+0000 7fb31a3ff700 1 Processor -- start 2024-01-11T08:25:46.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.960+0000 7fb31a3ff700 1 -- start start 2024-01-11T08:25:46.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.961+0000 7fb31a3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb3140ccd70 0x7fb3140cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:46.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.961+0000 7fb31a3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3140cf850 con 0x7fb31415a1b0 2024-01-11T08:25:46.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.961+0000 7fb31a3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3140cec40 con 0x7fb31415ae70 2024-01-11T08:25:46.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.961+0000 7fb31a3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb31415ac90 con 0x7fb3140ccd70 2024-01-11T08:25:46.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.962+0000 7fb313fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb31415a1b0 0x7fb31415a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36286/0 (socket says 172.21.15.27:36286) 2024-01-11T08:25:46.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.962+0000 7fb313fff700 1 -- 172.21.15.27:0/970565843 learned_addr learned my addr 172.21.15.27:0/970565843 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:46.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.962+0000 7fb3137fe700 1 --2- 172.21.15.27:0/970565843 >> v2:172.21.15.77:3300/0 conn(0x7fb3140ccd70 0x7fb3140cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:46.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.962+0000 7fb312ffd700 1 -- 172.21.15.27:0/970565843 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450788180 0 0) 0x7fb3140cf850 con 0x7fb31415a1b0 2024-01-11T08:25:46.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.962+0000 7fb312ffd700 1 -- 172.21.15.27:0/970565843 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2fc003570 con 0x7fb31415a1b0 2024-01-11T08:25:46.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.962+0000 7fb312ffd700 1 -- 172.21.15.27:0/970565843 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017390412 0 0) 0x7fb3140cec40 con 0x7fb31415ae70 2024-01-11T08:25:46.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.962+0000 7fb312ffd700 1 -- 172.21.15.27:0/970565843 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3140cf850 con 0x7fb31415ae70 2024-01-11T08:25:46.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.962+0000 7fb3137fe700 1 -- 172.21.15.27:0/970565843 >> v1:172.21.15.77:6789/0 conn(0x7fb31415ae70 legacy=0x7fb31415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:46.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.962+0000 7fb3137fe700 1 -- 172.21.15.27:0/970565843 >> v1:172.21.15.27:6789/0 conn(0x7fb31415a1b0 legacy=0x7fb31415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:46.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.962+0000 7fb3137fe700 1 -- 172.21.15.27:0/970565843 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3140cef20 con 0x7fb3140ccd70 2024-01-11T08:25:46.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.963+0000 7fb3137fe700 1 --2- 172.21.15.27:0/970565843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3140ccd70 0x7fb3140cd210 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb3080098e0 tx=0x7fb3080091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82f94221e8e69412 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:46.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.963+0000 7fb312ffd700 1 -- 172.21.15.27:0/970565843 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308011630 con 0x7fb3140ccd70 2024-01-11T08:25:46.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.963+0000 7fb312ffd700 1 -- 172.21.15.27:0/970565843 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb308011790 con 0x7fb3140ccd70 2024-01-11T08:25:46.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.963+0000 7fb312ffd700 1 -- 172.21.15.27:0/970565843 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308011b30 con 0x7fb3140ccd70 2024-01-11T08:25:46.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.964+0000 7fb31a3ff700 1 -- 172.21.15.27:0/970565843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3140ccd70 msgr2=0x7fb3140cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:46.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.964+0000 7fb31a3ff700 1 --2- 172.21.15.27:0/970565843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3140ccd70 0x7fb3140cd210 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb3080098e0 tx=0x7fb3080091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.964+0000 7fb31a3ff700 1 -- 172.21.15.27:0/970565843 shutdown_connections 2024-01-11T08:25:46.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.964+0000 7fb31a3ff700 1 --2- 172.21.15.27:0/970565843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3140ccd70 0x7fb3140cd210 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.964+0000 7fb31a3ff700 1 -- 172.21.15.27:0/970565843 >> 172.21.15.27:0/970565843 conn(0x7fb31406ed40 msgr2=0x7fb31406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:46.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.964+0000 7fb31a3ff700 1 -- 172.21.15.27:0/970565843 shutdown_connections 2024-01-11T08:25:46.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.964+0000 7fb31a3ff700 1 -- 172.21.15.27:0/970565843 wait complete. 2024-01-11T08:25:46.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.965+0000 7fb31a3ff700 1 Processor -- start 2024-01-11T08:25:46.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.966+0000 7fb31a3ff700 1 -- start start 2024-01-11T08:25:46.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.966+0000 7fb31a3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3140ccd70 0x7fb31416a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:46.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.966+0000 7fb31a3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb31415a1b0 0x7fb3140c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:46.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.966+0000 7fb31a3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb31415ae70 0x7fb31414dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:46.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.966+0000 7fb31a3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31416af50 con 0x7fb3140ccd70 2024-01-11T08:25:46.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.966+0000 7fb31a3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31416b230 con 0x7fb31415a1b0 2024-01-11T08:25:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.966+0000 7fb31a3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31416b510 con 0x7fb31415ae70 2024-01-11T08:25:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.967+0000 7fb313fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3140ccd70 0x7fb31416a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.967+0000 7fb313fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3140ccd70 0x7fb31416a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41680/0 (socket says 172.21.15.27:41680) 2024-01-11T08:25:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.967+0000 7fb313fff700 1 -- 172.21.15.27:0/2003676385 learned_addr learned my addr 172.21.15.27:0/2003676385 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.967+0000 7fb3137fe700 1 --2- 172.21.15.27:0/2003676385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb31415a1b0 0x7fb3140c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.967+0000 7fb31899c700 1 --2- 172.21.15.27:0/2003676385 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb31415ae70 0x7fb31414dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.967+0000 7fb313fff700 1 -- 172.21.15.27:0/2003676385 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb31415ae70 msgr2=0x7fb31414dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.967+0000 7fb313fff700 1 --2- 172.21.15.27:0/2003676385 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb31415ae70 0x7fb31414dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.967+0000 7fb313fff700 1 -- 172.21.15.27:0/2003676385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb31415a1b0 msgr2=0x7fb3140c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.967+0000 7fb313fff700 1 --2- 172.21.15.27:0/2003676385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb31415a1b0 0x7fb3140c5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.967+0000 7fb313fff700 1 -- 172.21.15.27:0/2003676385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb308009420 con 0x7fb3140ccd70 2024-01-11T08:25:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.967+0000 7fb3137fe700 1 --2- 172.21.15.27:0/2003676385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb31415a1b0 0x7fb3140c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:46.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.968+0000 7fb313fff700 1 --2- 172.21.15.27:0/2003676385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3140ccd70 0x7fb31416a640 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb30400c790 tx=0x7fb304007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:46.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.968+0000 7fb2fbfff700 1 -- 172.21.15.27:0/2003676385 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb304005b30 con 0x7fb3140ccd70 2024-01-11T08:25:46.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.968+0000 7fb2fbfff700 1 -- 172.21.15.27:0/2003676385 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb304005c90 con 0x7fb3140ccd70 2024-01-11T08:25:46.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.968+0000 7fb31a3ff700 1 -- 172.21.15.27:0/2003676385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31416b7f0 con 0x7fb3140ccd70 2024-01-11T08:25:46.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.968+0000 7fb2fbfff700 1 -- 172.21.15.27:0/2003676385 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb304007300 con 0x7fb3140ccd70 2024-01-11T08:25:46.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.968+0000 7fb31a3ff700 1 -- 172.21.15.27:0/2003676385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3141582b0 con 0x7fb3140ccd70 2024-01-11T08:25:46.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.969+0000 7fb31a3ff700 1 -- 172.21.15.27:0/2003676385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31416ba30 con 0x7fb3140ccd70 2024-01-11T08:25:46.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.970+0000 7fb2fbfff700 1 -- 172.21.15.27:0/2003676385 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fb30401b3b0 con 0x7fb3140ccd70 2024-01-11T08:25:46.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.971+0000 7fb2fbfff700 1 --2- 172.21.15.27:0/2003676385 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb300067050 0x7fb300069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:46.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.971+0000 7fb2fbfff700 1 -- 172.21.15.27:0/2003676385 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6077+0+0 (secure 0 0 0) 0x7fb304083d90 con 0x7fb3140ccd70 2024-01-11T08:25:46.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.971+0000 7fb3137fe700 1 --2- 172.21.15.27:0/2003676385 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb300067050 0x7fb300069500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:46.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.971+0000 7fb3137fe700 1 --2- 172.21.15.27:0/2003676385 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb300067050 0x7fb300069500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello connection peer type does not match what peer advertises 16 != 2 2024-01-11T08:25:46.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.971+0000 7fb3137fe700 1 --2- 172.21.15.27:0/2003676385 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb300067050 0x7fb300069500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.972+0000 7fb2fbfff700 1 -- 172.21.15.27:0/2003676385 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb300067050 msgr2=0x7fb300069500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-01-11T08:25:46.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.972+0000 7fb2fbfff700 1 --2- 172.21.15.27:0/2003676385 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb300067050 0x7fb300069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:46.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:46.976+0000 7fb2fbfff700 1 -- 172.21.15.27:0/2003676385 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb31416ba30 con 0x7fb3140ccd70 2024-01-11T08:25:46.995 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:46.996+0000 7f01ec3381c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:25:47.181 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:47.183+0000 7f01ec3381c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:25:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.224+0000 7fb31a3ff700 1 -- 172.21.15.27:0/2003676385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb31416ba30 con 0x7fb3140ccd70 2024-01-11T08:25:47.289 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:47.290+0000 7f01ec3381c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:25:47.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.302+0000 7fb2fbfff700 1 -- 172.21.15.27:0/2003676385 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 35) v1 ==== 84453+0+0 (secure 0 0 0) 0x7fb304051150 con 0x7fb3140ccd70 2024-01-11T08:25:47.326 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:47.327+0000 7ff67d0f81c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:25:47.363 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:47.363+0000 7f0166983700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:47.363 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:47.363+0000 7f0166983700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:47.363 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:47.363+0000 7f0166983700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:47.363 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:47.363+0000 7f0166983700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:47.363 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:47.363+0000 7f0166983700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:47.363 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:47.364+0000 7f0168987700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:47.363 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:47.364+0000 7f0168987700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:47.363 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:47.364+0000 7f0168987700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:47.363 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:47.364+0000 7f0168987700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:47.363 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:25:47.364+0000 7f0168987700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:25:47.521 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:47.522+0000 7ff67d0f81c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:25:47.720 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:47.720+0000 7ff67d0f81c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:25:47.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.727+0000 7fb2fbfff700 1 -- 172.21.15.27:0/2003676385 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb3040076f0 con 0x7fb3140ccd70 2024-01-11T08:25:47.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.729+0000 7fb31a3ff700 1 -- 172.21.15.27:0/2003676385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3140ccd70 msgr2=0x7fb31416a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:47.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.729+0000 7fb31a3ff700 1 --2- 172.21.15.27:0/2003676385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3140ccd70 0x7fb31416a640 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb30400c790 tx=0x7fb304007ea0 comp rx=0 tx=0).stop 2024-01-11T08:25:47.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.729+0000 7fb31a3ff700 1 -- 172.21.15.27:0/2003676385 shutdown_connections 2024-01-11T08:25:47.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.729+0000 7fb31a3ff700 1 --2- 172.21.15.27:0/2003676385 >> [v2:172.21.15.77:6832/41381,v1:172.21.15.77:6833/41381] conn(0x7fb300067050 0x7fb300069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:47.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.729+0000 7fb31a3ff700 1 --2- 172.21.15.27:0/2003676385 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb31415ae70 0x7fb31414dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:47.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.729+0000 7fb31a3ff700 1 --2- 172.21.15.27:0/2003676385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3140ccd70 0x7fb31416a640 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:47.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.729+0000 7fb31a3ff700 1 --2- 172.21.15.27:0/2003676385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb31415a1b0 0x7fb3140c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:47.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.729+0000 7fb31a3ff700 1 -- 172.21.15.27:0/2003676385 >> 172.21.15.27:0/2003676385 conn(0x7fb31406ed40 msgr2=0x7fb314158bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:47.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.730+0000 7fb31a3ff700 1 -- 172.21.15.27:0/2003676385 shutdown_connections 2024-01-11T08:25:47.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.730+0000 7fb31a3ff700 1 -- 172.21.15.27:0/2003676385 wait complete. 2024-01-11T08:25:47.748 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-11T08:25:47.839 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:25:47.840+0000 7ff67d0f81c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:25:47.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.957+0000 7f3cf93ff700 1 Processor -- start 2024-01-11T08:25:47.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.957+0000 7f3cf93ff700 1 -- start start 2024-01-11T08:25:47.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.958+0000 7f3cf93ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3cf4156810 0x7f3cf406e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:47.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.958+0000 7f3cf93ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3cf40ccac0 0x7f3cf406ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:47.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.958+0000 7f3cf93ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf40c8590 con 0x7f3cf4156440 2024-01-11T08:25:47.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.958+0000 7f3cf93ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cf40c7980 con 0x7f3cf4156810 2024-01-11T08:25:47.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.959+0000 7f3cf93ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf40c7c60 con 0x7f3cf40ccac0 2024-01-11T08:25:47.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.959+0000 7f3cf27fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3cf4156810 0x7f3cf406e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:47.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.959+0000 7f3cf27fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3cf4156810 0x7f3cf406e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41688/0 (socket says 172.21.15.27:41688) 2024-01-11T08:25:47.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.959+0000 7f3cf27fc700 1 -- 172.21.15.27:0/3522541122 learned_addr learned my addr 172.21.15.27:0/3522541122 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:47.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.959+0000 7f3cf37fe700 1 --2- 172.21.15.27:0/3522541122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3cf40ccac0 0x7f3cf406ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:47.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.959+0000 7f3cf27fc700 1 -- 172.21.15.27:0/3522541122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3cf40ccac0 msgr2=0x7f3cf406ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:47.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.959+0000 7f3cf27fc700 1 --2- 172.21.15.27:0/3522541122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3cf40ccac0 0x7f3cf406ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:47.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.959+0000 7f3cf27fc700 1 -- 172.21.15.27:0/3522541122 >> v1:172.21.15.77:6789/0 conn(0x7f3cf4156440 legacy=0x7f3cf40c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:47.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.959+0000 7f3cf27fc700 1 -- 172.21.15.27:0/3522541122 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf40c7f40 con 0x7f3cf4156810 2024-01-11T08:25:47.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.959+0000 7f3cf37fe700 1 --2- 172.21.15.27:0/3522541122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3cf40ccac0 0x7f3cf406ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:47.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.960+0000 7f3cf27fc700 1 --2- 172.21.15.27:0/3522541122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4156810 0x7f3cf406e720 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3ce80099b0 tx=0x7f3ce8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfe0417b78589654 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:47.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.960+0000 7f3cf1ffb700 1 -- 172.21.15.27:0/3522541122 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce800b860 con 0x7f3cf4156810 2024-01-11T08:25:47.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.960+0000 7f3cf1ffb700 1 -- 172.21.15.27:0/3522541122 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce800b9c0 con 0x7f3cf4156810 2024-01-11T08:25:47.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.960+0000 7f3cf1ffb700 1 -- 172.21.15.27:0/3522541122 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce800bd60 con 0x7f3cf4156810 2024-01-11T08:25:47.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.960+0000 7f3cf93ff700 1 -- 172.21.15.27:0/3522541122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4156810 msgr2=0x7f3cf406e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:47.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.960+0000 7f3cf93ff700 1 --2- 172.21.15.27:0/3522541122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4156810 0x7f3cf406e720 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3ce80099b0 tx=0x7f3ce8009250 comp rx=0 tx=0).stop 2024-01-11T08:25:47.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.961+0000 7f3cf93ff700 1 -- 172.21.15.27:0/3522541122 shutdown_connections 2024-01-11T08:25:47.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.961+0000 7f3cf93ff700 1 --2- 172.21.15.27:0/3522541122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3cf40ccac0 0x7f3cf406ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:47.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.961+0000 7f3cf93ff700 1 --2- 172.21.15.27:0/3522541122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4156810 0x7f3cf406e720 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:47.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.961+0000 7f3cf93ff700 1 -- 172.21.15.27:0/3522541122 >> 172.21.15.27:0/3522541122 conn(0x7f3cf40680e0 msgr2=0x7f3cf40cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:47.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.961+0000 7f3cf93ff700 1 -- 172.21.15.27:0/3522541122 shutdown_connections 2024-01-11T08:25:47.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.961+0000 7f3cf93ff700 1 -- 172.21.15.27:0/3522541122 wait complete. 2024-01-11T08:25:47.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.962+0000 7f3cf93ff700 1 Processor -- start 2024-01-11T08:25:47.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.962+0000 7f3cf93ff700 1 -- start start 2024-01-11T08:25:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.963+0000 7f3cf93ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3cf40ccac0 0x7f3cf4156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.963+0000 7f3cf93ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4156440 0x7f3cf41582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.963+0000 7f3cf93ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3cf4156810 0x7f3cf4155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.963+0000 7f3cf93ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf4157e20 con 0x7f3cf4156440 2024-01-11T08:25:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.963+0000 7f3cf93ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf41587f0 con 0x7f3cf4156810 2024-01-11T08:25:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.963+0000 7f3cf93ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf4155a70 con 0x7f3cf40ccac0 2024-01-11T08:25:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.964+0000 7f3cf27fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4156440 0x7f3cf41582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.964+0000 7f3cf27fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4156440 0x7f3cf41582b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41694/0 (socket says 172.21.15.27:41694) 2024-01-11T08:25:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.964+0000 7f3cf27fc700 1 -- 172.21.15.27:0/2410347039 learned_addr learned my addr 172.21.15.27:0/2410347039 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.964+0000 7f3cf37fe700 1 --2- 172.21.15.27:0/2410347039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3cf4156810 0x7f3cf4155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.964+0000 7f3cf2ffd700 1 --2- 172.21.15.27:0/2410347039 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3cf40ccac0 0x7f3cf4156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.964+0000 7f3cf27fc700 1 -- 172.21.15.27:0/2410347039 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3cf40ccac0 msgr2=0x7f3cf4156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.964+0000 7f3cf27fc700 1 --2- 172.21.15.27:0/2410347039 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3cf40ccac0 0x7f3cf4156c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.964+0000 7f3cf27fc700 1 -- 172.21.15.27:0/2410347039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3cf4156810 msgr2=0x7f3cf4155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.964+0000 7f3cf27fc700 1 --2- 172.21.15.27:0/2410347039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3cf4156810 0x7f3cf4155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.964+0000 7f3cf27fc700 1 -- 172.21.15.27:0/2410347039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce80094f0 con 0x7f3cf4156440 2024-01-11T08:25:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.964+0000 7f3cf2ffd700 1 --2- 172.21.15.27:0/2410347039 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3cf40ccac0 0x7f3cf4156c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.964+0000 7f3cf27fc700 1 --2- 172.21.15.27:0/2410347039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4156440 0x7f3cf41582b0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8009290 tx=0x7f3ce8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.965+0000 7f3cdaffd700 1 -- 172.21.15.27:0/2410347039 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8026480 con 0x7f3cf4156440 2024-01-11T08:25:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.965+0000 7f3cdaffd700 1 -- 172.21.15.27:0/2410347039 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce801f070 con 0x7f3cf4156440 2024-01-11T08:25:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.965+0000 7f3cdaffd700 1 -- 172.21.15.27:0/2410347039 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8026830 con 0x7f3cf4156440 2024-01-11T08:25:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.965+0000 7f3cf93ff700 1 -- 172.21.15.27:0/2410347039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf4157210 con 0x7f3cf4156440 2024-01-11T08:25:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.965+0000 7f3cf93ff700 1 -- 172.21.15.27:0/2410347039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf41577d0 con 0x7f3cf4156440 2024-01-11T08:25:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.966+0000 7f3cf93ff700 1 -- 172.21.15.27:0/2410347039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf4157fd0 con 0x7f3cf4156440 2024-01-11T08:25:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.967+0000 7f3cdaffd700 1 -- 172.21.15.27:0/2410347039 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 35) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f3ce8026ba0 con 0x7f3cf4156440 2024-01-11T08:25:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.968+0000 7f3cdaffd700 1 -- 172.21.15.27:0/2410347039 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6335+0+0 (secure 0 0 0) 0x7f3ce8021070 con 0x7f3cf4156440 2024-01-11T08:25:47.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:47.972+0000 7f3cdaffd700 1 -- 172.21.15.27:0/2410347039 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3cf4157fd0 con 0x7f3cf4156440 2024-01-11T08:25:48.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:48.176+0000 7f3cf93ff700 1 -- 172.21.15.27:0/2410347039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3cf404c3c0 con 0x7f3cf4156440 2024-01-11T08:25:48.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:48.302+0000 7f3cdaffd700 1 -- 172.21.15.27:0/2410347039 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 36) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f3ce8052e50 con 0x7f3cf4156440 2024-01-11T08:25:48.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:48.302+0000 7f3cdaffd700 1 --2- 172.21.15.27:0/2410347039 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f3ce006c6d0 0x7f3ce006eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:48.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:48.303+0000 7f3cf2ffd700 1 --2- 172.21.15.27:0/2410347039 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f3ce006c6d0 0x7f3ce006eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:48.303+0000 7f3cf2ffd700 1 --2- 172.21.15.27:0/2410347039 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f3ce006c6d0 0x7f3ce006eab0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40018f0 tx=0x7f3ce4009040 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:49.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.135+0000 7f3cdaffd700 1 -- 172.21.15.27:0/2410347039 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v79) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3ce80698b0 con 0x7f3cf4156440 2024-01-11T08:25:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.140+0000 7f3cf93ff700 1 -- 172.21.15.27:0/2410347039 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f3ce006c6d0 msgr2=0x7f3ce006eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.140+0000 7f3cf93ff700 1 --2- 172.21.15.27:0/2410347039 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f3ce006c6d0 0x7f3ce006eab0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40018f0 tx=0x7f3ce4009040 comp rx=0 tx=0).stop 2024-01-11T08:25:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.140+0000 7f3cf93ff700 1 -- 172.21.15.27:0/2410347039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4156440 msgr2=0x7f3cf41582b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.140+0000 7f3cf93ff700 1 --2- 172.21.15.27:0/2410347039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4156440 0x7f3cf41582b0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8009290 tx=0x7f3ce8017000 comp rx=0 tx=0).stop 2024-01-11T08:25:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.140+0000 7f3cf93ff700 1 -- 172.21.15.27:0/2410347039 shutdown_connections 2024-01-11T08:25:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.140+0000 7f3cf93ff700 1 --2- 172.21.15.27:0/2410347039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3cf4156810 0x7f3cf4155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.140+0000 7f3cf93ff700 1 --2- 172.21.15.27:0/2410347039 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f3ce006c6d0 0x7f3ce006eab0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.140+0000 7f3cf93ff700 1 --2- 172.21.15.27:0/2410347039 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3cf40ccac0 0x7f3cf4156c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:49.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.140+0000 7f3cf93ff700 1 --2- 172.21.15.27:0/2410347039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cf4156440 0x7f3cf41582b0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:49.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.140+0000 7f3cf93ff700 1 -- 172.21.15.27:0/2410347039 >> 172.21.15.27:0/2410347039 conn(0x7f3cf40680e0 msgr2=0x7f3cf40c89b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:49.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.140+0000 7f3cf93ff700 1 -- 172.21.15.27:0/2410347039 shutdown_connections 2024-01-11T08:25:49.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.140+0000 7f3cf93ff700 1 -- 172.21.15.27:0/2410347039 wait complete. 2024-01-11T08:25:49.141 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-01-11T08:25:49.162 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-11T08:25:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.377+0000 7f5a6e79d700 1 Processor -- start 2024-01-11T08:25:49.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.377+0000 7f5a6e79d700 1 -- start start 2024-01-11T08:25:49.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.379+0000 7f5a6e79d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5a68153d10 0x7f5a680b1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:49.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.379+0000 7f5a6e79d700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6815e0c0 con 0x7f5a680b2370 2024-01-11T08:25:49.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.379+0000 7f5a6e79d700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a680b2780 con 0x7f5a68153940 2024-01-11T08:25:49.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.379+0000 7f5a6e79d700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a680ceaf0 con 0x7f5a68153d10 2024-01-11T08:25:49.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.379+0000 7f5a6df9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5a680b2370 0x7f5a6815d350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36290/0 (socket says 172.21.15.27:36290) 2024-01-11T08:25:49.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.379+0000 7f5a6df9c700 1 -- 172.21.15.27:0/2000760669 learned_addr learned my addr 172.21.15.27:0/2000760669 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:49.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.379+0000 7f5a6cf9a700 1 --2- 172.21.15.27:0/2000760669 >> v2:172.21.15.77:3300/0 conn(0x7f5a68153d10 0x7f5a680b1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:49.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.379+0000 7f5a5ffff700 1 -- 172.21.15.27:0/2000760669 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453437650 0 0) 0x7f5a6815e0c0 con 0x7f5a680b2370 2024-01-11T08:25:49.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.379+0000 7f5a5ffff700 1 -- 172.21.15.27:0/2000760669 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a54003570 con 0x7f5a680b2370 2024-01-11T08:25:49.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.380+0000 7f5a5ffff700 1 -- 172.21.15.27:0/2000760669 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567831439 0 0) 0x7f5a680b2780 con 0x7f5a68153940 2024-01-11T08:25:49.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.380+0000 7f5a5ffff700 1 -- 172.21.15.27:0/2000760669 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a6815e0c0 con 0x7f5a68153940 2024-01-11T08:25:49.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.380+0000 7f5a6cf9a700 1 -- 172.21.15.27:0/2000760669 >> v1:172.21.15.77:6789/0 conn(0x7f5a68153940 legacy=0x7f5a680cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:49.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.380+0000 7f5a6cf9a700 1 -- 172.21.15.27:0/2000760669 >> v1:172.21.15.27:6789/0 conn(0x7f5a680b2370 legacy=0x7f5a6815d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:49.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.380+0000 7f5a6cf9a700 1 -- 172.21.15.27:0/2000760669 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a680cedd0 con 0x7f5a68153d10 2024-01-11T08:25:49.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.380+0000 7f5a6cf9a700 1 --2- 172.21.15.27:0/2000760669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a68153d10 0x7f5a680b1e30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5a580098e0 tx=0x7f5a580091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e2c54e84844db7e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:49.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.380+0000 7f5a5ffff700 1 -- 172.21.15.27:0/2000760669 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a58011630 con 0x7f5a68153d10 2024-01-11T08:25:49.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.380+0000 7f5a5ffff700 1 -- 172.21.15.27:0/2000760669 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a58011790 con 0x7f5a68153d10 2024-01-11T08:25:49.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.380+0000 7f5a5ffff700 1 -- 172.21.15.27:0/2000760669 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a58011b10 con 0x7f5a68153d10 2024-01-11T08:25:49.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.381+0000 7f5a6e79d700 1 -- 172.21.15.27:0/2000760669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a68153d10 msgr2=0x7f5a680b1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:49.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.381+0000 7f5a6e79d700 1 --2- 172.21.15.27:0/2000760669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a68153d10 0x7f5a680b1e30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5a580098e0 tx=0x7f5a580091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:49.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.381+0000 7f5a6e79d700 1 -- 172.21.15.27:0/2000760669 shutdown_connections 2024-01-11T08:25:49.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.381+0000 7f5a6e79d700 1 --2- 172.21.15.27:0/2000760669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a68153d10 0x7f5a680b1e30 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:49.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.381+0000 7f5a6e79d700 1 -- 172.21.15.27:0/2000760669 >> 172.21.15.27:0/2000760669 conn(0x7f5a6806ed40 msgr2=0x7f5a6806f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:49.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.381+0000 7f5a6e79d700 1 -- 172.21.15.27:0/2000760669 shutdown_connections 2024-01-11T08:25:49.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.381+0000 7f5a6e79d700 1 -- 172.21.15.27:0/2000760669 wait complete. 2024-01-11T08:25:49.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.382+0000 7f5a6e79d700 1 Processor -- start 2024-01-11T08:25:49.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.383+0000 7f5a6e79d700 1 -- start start 2024-01-11T08:25:49.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.383+0000 7f5a6e79d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a680b2370 0x7f5a6815aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:49.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.383+0000 7f5a6e79d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a68153940 0x7f5a68149e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.383+0000 7f5a6e79d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a68153d10 0x7f5a68154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.383+0000 7f5a6e79d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a68156010 con 0x7f5a680b2370 2024-01-11T08:25:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.383+0000 7f5a6e79d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a68154e70 con 0x7f5a68153940 2024-01-11T08:25:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.383+0000 7f5a6e79d700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a68155150 con 0x7f5a68153d10 2024-01-11T08:25:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.384+0000 7f5a6d79b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a680b2370 0x7f5a6815aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.384+0000 7f5a6d79b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a680b2370 0x7f5a6815aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41704/0 (socket says 172.21.15.27:41704) 2024-01-11T08:25:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.384+0000 7f5a6d79b700 1 -- 172.21.15.27:0/3009190018 learned_addr learned my addr 172.21.15.27:0/3009190018 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.384+0000 7f5a6cf9a700 1 --2- 172.21.15.27:0/3009190018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a68153940 0x7f5a68149e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.384+0000 7f5a6df9c700 1 --2- 172.21.15.27:0/3009190018 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a68153d10 0x7f5a68154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.384+0000 7f5a6d79b700 1 -- 172.21.15.27:0/3009190018 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a68153d10 msgr2=0x7f5a68154500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.384+0000 7f5a6d79b700 1 --2- 172.21.15.27:0/3009190018 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a68153d10 0x7f5a68154500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.384+0000 7f5a6d79b700 1 -- 172.21.15.27:0/3009190018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a68153940 msgr2=0x7f5a68149e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.384+0000 7f5a6d79b700 1 --2- 172.21.15.27:0/3009190018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a68153940 0x7f5a68149e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:49.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.384+0000 7f5a6d79b700 1 -- 172.21.15.27:0/3009190018 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a58009420 con 0x7f5a680b2370 2024-01-11T08:25:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.384+0000 7f5a6df9c700 1 --2- 172.21.15.27:0/3009190018 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a68153d10 0x7f5a68154500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:25:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.384+0000 7f5a6d79b700 1 --2- 172.21.15.27:0/3009190018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a680b2370 0x7f5a6815aee0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400c7c0 tx=0x7f5a64007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.385+0000 7f5a5d7fa700 1 -- 172.21.15.27:0/3009190018 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a64001530 con 0x7f5a680b2370 2024-01-11T08:25:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.385+0000 7f5a5d7fa700 1 -- 172.21.15.27:0/3009190018 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a64001690 con 0x7f5a680b2370 2024-01-11T08:25:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.385+0000 7f5a6e79d700 1 -- 172.21.15.27:0/3009190018 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a68155430 con 0x7f5a680b2370 2024-01-11T08:25:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.385+0000 7f5a5d7fa700 1 -- 172.21.15.27:0/3009190018 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a64001a50 con 0x7f5a680b2370 2024-01-11T08:25:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.385+0000 7f5a6e79d700 1 -- 172.21.15.27:0/3009190018 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6814a340 con 0x7f5a680b2370 2024-01-11T08:25:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.386+0000 7f5a6e79d700 1 -- 172.21.15.27:0/3009190018 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6804c3c0 con 0x7f5a680b2370 2024-01-11T08:25:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.387+0000 7f5a5d7fa700 1 -- 172.21.15.27:0/3009190018 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 36) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f5a6401b400 con 0x7f5a680b2370 2024-01-11T08:25:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.388+0000 7f5a5d7fa700 1 --2- 172.21.15.27:0/3009190018 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5a4c066d80 0x7f5a4c069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.388+0000 7f5a5d7fa700 1 -- 172.21.15.27:0/3009190018 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5897+0+0 (secure 0 0 0) 0x7f5a64083e90 con 0x7f5a680b2370 2024-01-11T08:25:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.388+0000 7f5a6cf9a700 1 --2- 172.21.15.27:0/3009190018 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5a4c066d80 0x7f5a4c069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:49.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.389+0000 7f5a6cf9a700 1 --2- 172.21.15.27:0/3009190018 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5a4c066d80 0x7f5a4c069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5a580093f0 tx=0x7f5a58017000 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:49.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.392+0000 7f5a5d7fa700 1 -- 172.21.15.27:0/3009190018 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a640512e0 con 0x7f5a680b2370 2024-01-11T08:25:49.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:49.599+0000 7f5a6e79d700 1 -- 172.21.15.27:0/3009190018 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5a680cf0b0 con 0x7f5a680b2370 2024-01-11T08:25:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.139+0000 7f5a5d7fa700 1 -- 172.21.15.27:0/3009190018 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f5a640516b0 con 0x7f5a680b2370 2024-01-11T08:25:50.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.314+0000 7f5a5d7fa700 1 -- 172.21.15.27:0/3009190018 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v80) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5a64054d70 con 0x7f5a680b2370 2024-01-11T08:25:50.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.318+0000 7f5a6e79d700 1 -- 172.21.15.27:0/3009190018 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5a4c066d80 msgr2=0x7f5a4c069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.318+0000 7f5a6e79d700 1 --2- 172.21.15.27:0/3009190018 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5a4c066d80 0x7f5a4c069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5a580093f0 tx=0x7f5a58017000 comp rx=0 tx=0).stop 2024-01-11T08:25:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.318+0000 7f5a6e79d700 1 -- 172.21.15.27:0/3009190018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a680b2370 msgr2=0x7f5a6815aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.318+0000 7f5a6e79d700 1 --2- 172.21.15.27:0/3009190018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a680b2370 0x7f5a6815aee0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400c7c0 tx=0x7f5a64007db0 comp rx=0 tx=0).stop 2024-01-11T08:25:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.318+0000 7f5a6e79d700 1 -- 172.21.15.27:0/3009190018 shutdown_connections 2024-01-11T08:25:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.318+0000 7f5a6e79d700 1 --2- 172.21.15.27:0/3009190018 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5a4c066d80 0x7f5a4c069230 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.318+0000 7f5a6e79d700 1 --2- 172.21.15.27:0/3009190018 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a68153d10 0x7f5a68154500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.318+0000 7f5a6e79d700 1 --2- 172.21.15.27:0/3009190018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a680b2370 0x7f5a6815aee0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.318+0000 7f5a6e79d700 1 --2- 172.21.15.27:0/3009190018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a68153940 0x7f5a68149e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.318+0000 7f5a6e79d700 1 -- 172.21.15.27:0/3009190018 >> 172.21.15.27:0/3009190018 conn(0x7f5a6806ed40 msgr2=0x7f5a6814ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.319+0000 7f5a6e79d700 1 -- 172.21.15.27:0/3009190018 shutdown_connections 2024-01-11T08:25:50.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.319+0000 7f5a6e79d700 1 -- 172.21.15.27:0/3009190018 wait complete. 2024-01-11T08:25:50.320 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-01-11T08:25:50.340 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-11T08:25:50.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.546+0000 7f28bd9ff700 1 Processor -- start 2024-01-11T08:25:50.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.547+0000 7f28bd9ff700 1 -- start start 2024-01-11T08:25:50.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.548+0000 7f28bd9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f28b8157410 0x7f28b80ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:50.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.548+0000 7f28bd9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f28b81577e0 0x7f28b80ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:50.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.548+0000 7f28bd9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b80699d0 con 0x7f28b8069540 2024-01-11T08:25:50.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.548+0000 7f28bd9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28b8069cb0 con 0x7f28b8157410 2024-01-11T08:25:50.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.548+0000 7f28bd9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28b8069f90 con 0x7f28b81577e0 2024-01-11T08:25:50.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.548+0000 7f28b6ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f28b8157410 0x7f28b80ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:50.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.548+0000 7f28b6ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f28b8157410 0x7f28b80ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41708/0 (socket says 172.21.15.27:41708) 2024-01-11T08:25:50.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.548+0000 7f28b6ffd700 1 -- 172.21.15.27:0/2657374913 learned_addr learned my addr 172.21.15.27:0/2657374913 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:50.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.548+0000 7f28b67fc700 1 --2- 172.21.15.27:0/2657374913 >> v2:172.21.15.77:3300/0 conn(0x7f28b81577e0 0x7f28b80ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:50.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.549+0000 7f28b6ffd700 1 -- 172.21.15.27:0/2657374913 >> v2:172.21.15.77:3300/0 conn(0x7f28b81577e0 msgr2=0x7f28b80ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:50.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.549+0000 7f28b6ffd700 1 --2- 172.21.15.27:0/2657374913 >> v2:172.21.15.77:3300/0 conn(0x7f28b81577e0 0x7f28b80ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:50.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.549+0000 7f28b6ffd700 1 -- 172.21.15.27:0/2657374913 >> v1:172.21.15.77:6789/0 conn(0x7f28b8069540 legacy=0x7f28b815b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:50.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.549+0000 7f28b6ffd700 1 -- 172.21.15.27:0/2657374913 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b806a270 con 0x7f28b8157410 2024-01-11T08:25:50.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.549+0000 7f28b67fc700 1 --2- 172.21.15.27:0/2657374913 >> v2:172.21.15.77:3300/0 conn(0x7f28b81577e0 0x7f28b80ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:50.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.549+0000 7f28b6ffd700 1 --2- 172.21.15.27:0/2657374913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b8157410 0x7f28b80ccae0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f28b806d2a0 tx=0x7f28a8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec497b27e746d612 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:50.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.549+0000 7f28b5ffb700 1 -- 172.21.15.27:0/2657374913 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a8005c10 con 0x7f28b8157410 2024-01-11T08:25:50.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.549+0000 7f28b5ffb700 1 -- 172.21.15.27:0/2657374913 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a8005d70 con 0x7f28b8157410 2024-01-11T08:25:50.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.549+0000 7f28b5ffb700 1 -- 172.21.15.27:0/2657374913 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a800b880 con 0x7f28b8157410 2024-01-11T08:25:50.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.550+0000 7f28bd9ff700 1 -- 172.21.15.27:0/2657374913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b8157410 msgr2=0x7f28b80ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.550+0000 7f28bd9ff700 1 --2- 172.21.15.27:0/2657374913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b8157410 0x7f28b80ccae0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f28b806d2a0 tx=0x7f28a8009220 comp rx=0 tx=0).stop 2024-01-11T08:25:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.550+0000 7f28bd9ff700 1 -- 172.21.15.27:0/2657374913 shutdown_connections 2024-01-11T08:25:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.550+0000 7f28bd9ff700 1 --2- 172.21.15.27:0/2657374913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b8157410 0x7f28b80ccae0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.550+0000 7f28bd9ff700 1 --2- 172.21.15.27:0/2657374913 >> v2:172.21.15.77:3300/0 conn(0x7f28b81577e0 0x7f28b80ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.550+0000 7f28bd9ff700 1 -- 172.21.15.27:0/2657374913 >> 172.21.15.27:0/2657374913 conn(0x7f28b80680a0 msgr2=0x7f28b80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.550+0000 7f28bd9ff700 1 -- 172.21.15.27:0/2657374913 shutdown_connections 2024-01-11T08:25:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.550+0000 7f28bd9ff700 1 -- 172.21.15.27:0/2657374913 wait complete. 2024-01-11T08:25:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.551+0000 7f28bd9ff700 1 Processor -- start 2024-01-11T08:25:50.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.552+0000 7f28bd9ff700 1 -- start start 2024-01-11T08:25:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.553+0000 7f28bd9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28b8069540 0x7f28b80af890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.553+0000 7f28bd9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28b8157410 0x7f28b80b0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.553+0000 7f28bd9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b81577e0 0x7f28b80c1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.553+0000 7f28bd9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b80afdd0 con 0x7f28b81577e0 2024-01-11T08:25:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.553+0000 7f28bd9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b80c4c70 con 0x7f28b8157410 2024-01-11T08:25:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.553+0000 7f28bd9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b80af400 con 0x7f28b8069540 2024-01-11T08:25:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.553+0000 7f28b77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b81577e0 0x7f28b80c1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.553+0000 7f28b6ffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28b8069540 0x7f28b80af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.553+0000 7f28b77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b81577e0 0x7f28b80c1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41712/0 (socket says 172.21.15.27:41712) 2024-01-11T08:25:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.553+0000 7f28b77fe700 1 -- 172.21.15.27:0/2235801361 learned_addr learned my addr 172.21.15.27:0/2235801361 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.553+0000 7f28b67fc700 1 --2- 172.21.15.27:0/2235801361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28b8157410 0x7f28b80b0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.553+0000 7f28b77fe700 1 -- 172.21.15.27:0/2235801361 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28b8069540 msgr2=0x7f28b80af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.553+0000 7f28b77fe700 1 --2- 172.21.15.27:0/2235801361 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28b8069540 0x7f28b80af890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.553+0000 7f28b77fe700 1 -- 172.21.15.27:0/2235801361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28b8157410 msgr2=0x7f28b80b0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.553+0000 7f28b77fe700 1 --2- 172.21.15.27:0/2235801361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28b8157410 0x7f28b80b0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.553+0000 7f28b77fe700 1 -- 172.21.15.27:0/2235801361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a8009480 con 0x7f28b81577e0 2024-01-11T08:25:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.554+0000 7f28b77fe700 1 --2- 172.21.15.27:0/2235801361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b81577e0 0x7f28b80c1f30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f28b000c870 tx=0x7f28b0001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.554+0000 7f289effd700 1 -- 172.21.15.27:0/2235801361 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b0012630 con 0x7f28b81577e0 2024-01-11T08:25:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.554+0000 7f289effd700 1 -- 172.21.15.27:0/2235801361 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b00127b0 con 0x7f28b81577e0 2024-01-11T08:25:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.554+0000 7f289effd700 1 -- 172.21.15.27:0/2235801361 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b0012b70 con 0x7f28b81577e0 2024-01-11T08:25:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.554+0000 7f28bd9ff700 1 -- 172.21.15.27:0/2235801361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b80c46b0 con 0x7f28b81577e0 2024-01-11T08:25:50.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.554+0000 7f28bd9ff700 1 -- 172.21.15.27:0/2235801361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b80af090 con 0x7f28b81577e0 2024-01-11T08:25:50.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.555+0000 7f28bd9ff700 1 -- 172.21.15.27:0/2235801361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b804c3c0 con 0x7f28b81577e0 2024-01-11T08:25:50.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.556+0000 7f289effd700 1 -- 172.21.15.27:0/2235801361 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f28b0003030 con 0x7f28b81577e0 2024-01-11T08:25:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.557+0000 7f289effd700 1 --2- 172.21.15.27:0/2235801361 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f28a4066ea0 0x7f28a4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.557+0000 7f289effd700 1 -- 172.21.15.27:0/2235801361 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5483+0+0 (secure 0 0 0) 0x7f28b00841f0 con 0x7f28b81577e0 2024-01-11T08:25:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.558+0000 7f28b6ffd700 1 --2- 172.21.15.27:0/2235801361 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f28a4066ea0 0x7f28a4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.559+0000 7f28b6ffd700 1 --2- 172.21.15.27:0/2235801361 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f28a4066ea0 0x7f28a4069350 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f28b8153160 tx=0x7f28a8016000 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:50.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.561+0000 7f289effd700 1 -- 172.21.15.27:0/2235801361 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f28b0051580 con 0x7f28b81577e0 2024-01-11T08:25:50.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.767+0000 7f28bd9ff700 1 -- 172.21.15.27:0/2235801361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f28b80af540 con 0x7f28b81577e0 2024-01-11T08:25:50.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.773+0000 7f289effd700 1 -- 172.21.15.27:0/2235801361 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v25) v1 ==== 120+0+0 (secure 0 0 0) 0x7f28b0054ba0 con 0x7f28b81577e0 2024-01-11T08:25:50.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.775+0000 7f28bd9ff700 1 -- 172.21.15.27:0/2235801361 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f28a4066ea0 msgr2=0x7f28a4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:50.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.775+0000 7f28bd9ff700 1 --2- 172.21.15.27:0/2235801361 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f28a4066ea0 0x7f28a4069350 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f28b8153160 tx=0x7f28a8016000 comp rx=0 tx=0).stop 2024-01-11T08:25:50.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.775+0000 7f28bd9ff700 1 -- 172.21.15.27:0/2235801361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b81577e0 msgr2=0x7f28b80c1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:50.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.775+0000 7f28bd9ff700 1 --2- 172.21.15.27:0/2235801361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b81577e0 0x7f28b80c1f30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f28b000c870 tx=0x7f28b0001670 comp rx=0 tx=0).stop 2024-01-11T08:25:50.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.775+0000 7f28bd9ff700 1 -- 172.21.15.27:0/2235801361 shutdown_connections 2024-01-11T08:25:50.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.775+0000 7f28bd9ff700 1 --2- 172.21.15.27:0/2235801361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b81577e0 0x7f28b80c1f30 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:50.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.775+0000 7f28bd9ff700 1 --2- 172.21.15.27:0/2235801361 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f28a4066ea0 0x7f28a4069350 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:50.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.775+0000 7f28bd9ff700 1 --2- 172.21.15.27:0/2235801361 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28b8069540 0x7f28b80af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:50.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.775+0000 7f28bd9ff700 1 --2- 172.21.15.27:0/2235801361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28b8157410 0x7f28b80b0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:50.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.775+0000 7f28bd9ff700 1 -- 172.21.15.27:0/2235801361 >> 172.21.15.27:0/2235801361 conn(0x7f28b80680a0 msgr2=0x7f28b80c6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:50.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.775+0000 7f28bd9ff700 1 -- 172.21.15.27:0/2235801361 shutdown_connections 2024-01-11T08:25:50.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:50.776+0000 7f28bd9ff700 1 -- 172.21.15.27:0/2235801361 wait complete. 2024-01-11T08:25:50.792 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-11T08:25:51.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.002+0000 7f564d5ff700 1 Processor -- start 2024-01-11T08:25:51.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.002+0000 7f564d5ff700 1 -- start start 2024-01-11T08:25:51.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.003+0000 7f564d5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f564815a1b0 0x7f564815a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.003+0000 7f564d5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f564815ae70 0x7f564815b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.003+0000 7f564d5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56480cf850 con 0x7f56480ccd70 2024-01-11T08:25:51.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.004+0000 7f564d5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56480cec40 con 0x7f564815a1b0 2024-01-11T08:25:51.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.004+0000 7f564d5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56480cef20 con 0x7f564815ae70 2024-01-11T08:25:51.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.004+0000 7f5646ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f564815a1b0 0x7f564815a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.004+0000 7f56467fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f56480ccd70 0x7f56480cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36298/0 (socket says 172.21.15.27:36298) 2024-01-11T08:25:51.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.004+0000 7f56467fc700 1 -- 172.21.15.27:0/4194577271 learned_addr learned my addr 172.21.15.27:0/4194577271 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:51.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.004+0000 7f5645ffb700 1 -- 172.21.15.27:0/4194577271 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202624089 0 0) 0x7f56480cf850 con 0x7f56480ccd70 2024-01-11T08:25:51.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.004+0000 7f5646ffd700 1 -- 172.21.15.27:0/4194577271 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f564815ae70 msgr2=0x7f564815b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:51.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.004+0000 7f5646ffd700 1 --2- 172.21.15.27:0/4194577271 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f564815ae70 0x7f564815b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.004+0000 7f5646ffd700 1 -- 172.21.15.27:0/4194577271 >> v1:172.21.15.27:6789/0 conn(0x7f56480ccd70 legacy=0x7f56480cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:51.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.004+0000 7f5646ffd700 1 -- 172.21.15.27:0/4194577271 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56480cf200 con 0x7f564815a1b0 2024-01-11T08:25:51.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.005+0000 7f5646ffd700 1 --2- 172.21.15.27:0/4194577271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f564815a1b0 0x7f564815a580 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f56380098e0 tx=0x7f56380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7218aa4e9a492378 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:51.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.005+0000 7f5645ffb700 1 -- 172.21.15.27:0/4194577271 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638009e80 con 0x7f564815a1b0 2024-01-11T08:25:51.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.005+0000 7f5645ffb700 1 -- 172.21.15.27:0/4194577271 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563800b860 con 0x7f564815a1b0 2024-01-11T08:25:51.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.005+0000 7f5645ffb700 1 -- 172.21.15.27:0/4194577271 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563800bbe0 con 0x7f564815a1b0 2024-01-11T08:25:51.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.005+0000 7f564d5ff700 1 -- 172.21.15.27:0/4194577271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f564815a1b0 msgr2=0x7f564815a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:51.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.005+0000 7f564d5ff700 1 --2- 172.21.15.27:0/4194577271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f564815a1b0 0x7f564815a580 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f56380098e0 tx=0x7f56380091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.006+0000 7f564d5ff700 1 -- 172.21.15.27:0/4194577271 shutdown_connections 2024-01-11T08:25:51.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.006+0000 7f564d5ff700 1 --2- 172.21.15.27:0/4194577271 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f564815ae70 0x7f564815b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.006+0000 7f564d5ff700 1 --2- 172.21.15.27:0/4194577271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f564815a1b0 0x7f564815a580 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.006+0000 7f564d5ff700 1 -- 172.21.15.27:0/4194577271 >> 172.21.15.27:0/4194577271 conn(0x7f564806ed40 msgr2=0x7f564806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:51.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.006+0000 7f564d5ff700 1 -- 172.21.15.27:0/4194577271 shutdown_connections 2024-01-11T08:25:51.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.006+0000 7f564d5ff700 1 -- 172.21.15.27:0/4194577271 wait complete. 2024-01-11T08:25:51.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.007+0000 7f564d5ff700 1 Processor -- start 2024-01-11T08:25:51.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.007+0000 7f564d5ff700 1 -- start start 2024-01-11T08:25:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.008+0000 7f564d5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56480ccd70 0x7f56480ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.008+0000 7f564d5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f564815a1b0 0x7f56480b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.008+0000 7f564d5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f564815ae70 0x7f564814f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.008+0000 7f564d5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56480b38c0 con 0x7f56480ccd70 2024-01-11T08:25:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.008+0000 7f564d5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56480b3af0 con 0x7f564815a1b0 2024-01-11T08:25:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.008+0000 7f564d5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56480b3dd0 con 0x7f564815ae70 2024-01-11T08:25:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.009+0000 7f5646ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56480ccd70 0x7f56480ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.009+0000 7f5646ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56480ccd70 0x7f56480ca080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41732/0 (socket says 172.21.15.27:41732) 2024-01-11T08:25:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.009+0000 7f5646ffd700 1 -- 172.21.15.27:0/78771747 learned_addr learned my addr 172.21.15.27:0/78771747 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.009+0000 7f56467fc700 1 --2- 172.21.15.27:0/78771747 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f564815a1b0 0x7f56480b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.009+0000 7f56477fe700 1 --2- 172.21.15.27:0/78771747 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f564815ae70 0x7f564814f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.009+0000 7f5646ffd700 1 -- 172.21.15.27:0/78771747 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f564815ae70 msgr2=0x7f564814f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.009+0000 7f5646ffd700 1 --2- 172.21.15.27:0/78771747 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f564815ae70 0x7f564814f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.009+0000 7f5646ffd700 1 -- 172.21.15.27:0/78771747 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f564815a1b0 msgr2=0x7f56480b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.009+0000 7f5646ffd700 1 --2- 172.21.15.27:0/78771747 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f564815a1b0 0x7f56480b3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.009+0000 7f5646ffd700 1 -- 172.21.15.27:0/78771747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5638009420 con 0x7f56480ccd70 2024-01-11T08:25:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.009+0000 7f5646ffd700 1 --2- 172.21.15.27:0/78771747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56480ccd70 0x7f56480ca080 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5648069770 tx=0x7f5638017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.010+0000 7f562effd700 1 -- 172.21.15.27:0/78771747 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638000cc0 con 0x7f56480ccd70 2024-01-11T08:25:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.010+0000 7f562effd700 1 -- 172.21.15.27:0/78771747 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563800b9d0 con 0x7f56480ccd70 2024-01-11T08:25:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.010+0000 7f562effd700 1 -- 172.21.15.27:0/78771747 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638026680 con 0x7f56480ccd70 2024-01-11T08:25:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.010+0000 7f564d5ff700 1 -- 172.21.15.27:0/78771747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56480b40b0 con 0x7f56480ccd70 2024-01-11T08:25:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.010+0000 7f564d5ff700 1 -- 172.21.15.27:0/78771747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56481541c0 con 0x7f56480ccd70 2024-01-11T08:25:51.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.011+0000 7f564d5ff700 1 -- 172.21.15.27:0/78771747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56481543e0 con 0x7f56480ccd70 2024-01-11T08:25:51.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.012+0000 7f562effd700 1 -- 172.21.15.27:0/78771747 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f5638023030 con 0x7f56480ccd70 2024-01-11T08:25:51.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.013+0000 7f562effd700 1 --2- 172.21.15.27:0/78771747 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5634066ea0 0x7f5634069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.013+0000 7f562effd700 1 -- 172.21.15.27:0/78771747 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5483+0+0 (secure 0 0 0) 0x7f5638085c40 con 0x7f56480ccd70 2024-01-11T08:25:51.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.014+0000 7f56467fc700 1 --2- 172.21.15.27:0/78771747 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5634066ea0 0x7f5634069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.014+0000 7f56467fc700 1 --2- 172.21.15.27:0/78771747 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5634066ea0 0x7f5634069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f564816d300 tx=0x7f563c007f50 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:51.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.017+0000 7f562effd700 1 -- 172.21.15.27:0/78771747 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f56481543e0 con 0x7f56480ccd70 2024-01-11T08:25:51.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.226+0000 7f564d5ff700 1 -- 172.21.15.27:0/78771747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f56481543e0 con 0x7f56480ccd70 2024-01-11T08:25:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.230+0000 7f562effd700 1 -- 172.21.15.27:0/78771747 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v26) v1 ==== 115+0+0 (secure 0 0 0) 0x7f56481543e0 con 0x7f56480ccd70 2024-01-11T08:25:51.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.233+0000 7f564d5ff700 1 -- 172.21.15.27:0/78771747 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5634066ea0 msgr2=0x7f5634069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:51.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.233+0000 7f564d5ff700 1 --2- 172.21.15.27:0/78771747 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5634066ea0 0x7f5634069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f564816d300 tx=0x7f563c007f50 comp rx=0 tx=0).stop 2024-01-11T08:25:51.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.233+0000 7f564d5ff700 1 -- 172.21.15.27:0/78771747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56480ccd70 msgr2=0x7f56480ca080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:51.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.233+0000 7f564d5ff700 1 --2- 172.21.15.27:0/78771747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56480ccd70 0x7f56480ca080 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5648069770 tx=0x7f5638017000 comp rx=0 tx=0).stop 2024-01-11T08:25:51.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.233+0000 7f564d5ff700 1 -- 172.21.15.27:0/78771747 shutdown_connections 2024-01-11T08:25:51.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.233+0000 7f564d5ff700 1 --2- 172.21.15.27:0/78771747 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5634066ea0 0x7f5634069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.233+0000 7f564d5ff700 1 --2- 172.21.15.27:0/78771747 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f564815ae70 0x7f564814f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.233+0000 7f564d5ff700 1 --2- 172.21.15.27:0/78771747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56480ccd70 0x7f56480ca080 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.233+0000 7f564d5ff700 1 --2- 172.21.15.27:0/78771747 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f564815a1b0 0x7f56480b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.233+0000 7f564d5ff700 1 -- 172.21.15.27:0/78771747 >> 172.21.15.27:0/78771747 conn(0x7f564806ed40 msgr2=0x7f56481586a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:51.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.233+0000 7f564d5ff700 1 -- 172.21.15.27:0/78771747 shutdown_connections 2024-01-11T08:25:51.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.233+0000 7f564d5ff700 1 -- 172.21.15.27:0/78771747 wait complete. 2024-01-11T08:25:51.250 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-11T08:25:51.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.451+0000 7f7e6dbff700 1 Processor -- start 2024-01-11T08:25:51.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.452+0000 7f7e6dbff700 1 -- start start 2024-01-11T08:25:51.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.453+0000 7f7e6dbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7e6815a1b0 0x7f7e6815a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.453+0000 7f7e6dbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e6815ae70 0x7f7e6815b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.454+0000 7f7e6dbff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e680cf850 con 0x7f7e680ccd70 2024-01-11T08:25:51.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.454+0000 7f7e6dbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e680cec40 con 0x7f7e6815a1b0 2024-01-11T08:25:51.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.454+0000 7f7e6dbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e680cef20 con 0x7f7e6815ae70 2024-01-11T08:25:51.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.454+0000 7f7e66ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7e680ccd70 0x7f7e680cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36300/0 (socket says 172.21.15.27:36300) 2024-01-11T08:25:51.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.454+0000 7f7e66ffd700 1 -- 172.21.15.27:0/2485188084 learned_addr learned my addr 172.21.15.27:0/2485188084 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:51.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.454+0000 7f7e67fff700 1 --2- 172.21.15.27:0/2485188084 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e6815ae70 0x7f7e6815b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.454+0000 7f7e677fe700 1 --2- 172.21.15.27:0/2485188084 >> v2:172.21.15.77:3300/0 conn(0x7f7e6815a1b0 0x7f7e6815a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.454+0000 7f7e667fc700 1 -- 172.21.15.27:0/2485188084 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450436014 0 0) 0x7f7e680cf850 con 0x7f7e680ccd70 2024-01-11T08:25:51.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.454+0000 7f7e667fc700 1 -- 172.21.15.27:0/2485188084 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e50003570 con 0x7f7e680ccd70 2024-01-11T08:25:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e667fc700 1 -- 172.21.15.27:0/2485188084 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1039884830 0 0) 0x7f7e50003570 con 0x7f7e680ccd70 2024-01-11T08:25:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e667fc700 1 -- 172.21.15.27:0/2485188084 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e6815ae70 msgr2=0x7f7e6815b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e667fc700 1 --2- 172.21.15.27:0/2485188084 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e6815ae70 0x7f7e6815b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e667fc700 1 -- 172.21.15.27:0/2485188084 >> v2:172.21.15.77:3300/0 conn(0x7f7e6815a1b0 msgr2=0x7f7e6815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e667fc700 1 --2- 172.21.15.27:0/2485188084 >> v2:172.21.15.77:3300/0 conn(0x7f7e6815a1b0 0x7f7e6815a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e667fc700 1 -- 172.21.15.27:0/2485188084 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e680cf200 con 0x7f7e680ccd70 2024-01-11T08:25:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e667fc700 1 -- 172.21.15.27:0/2485188084 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f7e5c001470 con 0x7f7e680ccd70 2024-01-11T08:25:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e667fc700 1 -- 172.21.15.27:0/2485188084 >> v1:172.21.15.27:6789/0 conn(0x7f7e680ccd70 legacy=0x7f7e680cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e667fc700 1 --2- 172.21.15.27:0/2485188084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e50003ae0 0x7f7e50003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e667fc700 1 --2- 172.21.15.27:0/2485188084 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e6815ae70 0x7f7e50004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e667fc700 1 --2- 172.21.15.27:0/2485188084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e50004d50 0x7f7e500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e667fc700 1 -- 172.21.15.27:0/2485188084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e680cec40 con 0x7f7e50004d50 2024-01-11T08:25:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e667fc700 1 -- 172.21.15.27:0/2485188084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e680cef20 con 0x7f7e50003ae0 2024-01-11T08:25:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e667fc700 1 -- 172.21.15.27:0/2485188084 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5000bb10 con 0x7f7e6815ae70 2024-01-11T08:25:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e677fe700 1 --2- 172.21.15.27:0/2485188084 >> v2:172.21.15.77:3300/0 conn(0x7f7e6815a1b0 0x7f7e6815a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e67fff700 1 --2- 172.21.15.27:0/2485188084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e50004d50 0x7f7e500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e66ffd700 1 --2- 172.21.15.27:0/2485188084 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e6815ae70 0x7f7e50004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e677fe700 1 --2- 172.21.15.27:0/2485188084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e50003ae0 0x7f7e50003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e67fff700 1 -- 172.21.15.27:0/2485188084 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e6815ae70 msgr2=0x7f7e50004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e67fff700 1 --2- 172.21.15.27:0/2485188084 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e6815ae70 0x7f7e50004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e67fff700 1 -- 172.21.15.27:0/2485188084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e50003ae0 msgr2=0x7f7e50003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e67fff700 1 --2- 172.21.15.27:0/2485188084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e50003ae0 0x7f7e50003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.455+0000 7f7e67fff700 1 -- 172.21.15.27:0/2485188084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e50003570 con 0x7f7e50004d50 2024-01-11T08:25:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.456+0000 7f7e67fff700 1 --2- 172.21.15.27:0/2485188084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e50004d50 0x7f7e500051c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000e240 tx=0x7f7e6000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68cbb7c189ecd62c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.456+0000 7f7e667fc700 1 -- 172.21.15.27:0/2485188084 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6000eaf0 con 0x7f7e50004d50 2024-01-11T08:25:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.456+0000 7f7e667fc700 1 -- 172.21.15.27:0/2485188084 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6000f040 con 0x7f7e50004d50 2024-01-11T08:25:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.456+0000 7f7e667fc700 1 -- 172.21.15.27:0/2485188084 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6000eea0 con 0x7f7e50004d50 2024-01-11T08:25:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.456+0000 7f7e6dbff700 1 -- 172.21.15.27:0/2485188084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e50004d50 msgr2=0x7f7e500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.456+0000 7f7e6dbff700 1 --2- 172.21.15.27:0/2485188084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e50004d50 0x7f7e500051c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000e240 tx=0x7f7e6000b040 comp rx=0 tx=0).stop 2024-01-11T08:25:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.456+0000 7f7e67fff700 1 -- 172.21.15.27:0/2485188084 reap_dead start 2024-01-11T08:25:51.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.456+0000 7f7e6dbff700 1 -- 172.21.15.27:0/2485188084 shutdown_connections 2024-01-11T08:25:51.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.456+0000 7f7e6dbff700 1 -- 172.21.15.27:0/2485188084 >> 172.21.15.27:0/2485188084 conn(0x7f7e6806ed40 msgr2=0x7f7e6806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:51.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.457+0000 7f7e6dbff700 1 -- 172.21.15.27:0/2485188084 shutdown_connections 2024-01-11T08:25:51.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.457+0000 7f7e6dbff700 1 -- 172.21.15.27:0/2485188084 wait complete. 2024-01-11T08:25:51.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.458+0000 7f7e6dbff700 1 Processor -- start 2024-01-11T08:25:51.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.458+0000 7f7e6dbff700 1 -- start start 2024-01-11T08:25:51.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.459+0000 7f7e6dbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e680b4b40 0x7f7e680c8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.459+0000 7f7e6dbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e680c9110 0x7f7e6814ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.459+0000 7f7e6dbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e680b4f10 0x7f7e6814b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.459+0000 7f7e6dbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6814b5b0 con 0x7f7e680c9110 2024-01-11T08:25:51.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.459+0000 7f7e6dbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e680c8e80 con 0x7f7e680b4b40 2024-01-11T08:25:51.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.459+0000 7f7e6dbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e680c95f0 con 0x7f7e680b4f10 2024-01-11T08:25:51.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.459+0000 7f7e66ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e680c9110 0x7f7e6814ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.459+0000 7f7e66ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e680c9110 0x7f7e6814ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41752/0 (socket says 172.21.15.27:41752) 2024-01-11T08:25:51.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.459+0000 7f7e66ffd700 1 -- 172.21.15.27:0/2990253666 learned_addr learned my addr 172.21.15.27:0/2990253666 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:51.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.460+0000 7f7e67fff700 1 --2- 172.21.15.27:0/2990253666 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e680b4f10 0x7f7e6814b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.460+0000 7f7e677fe700 1 --2- 172.21.15.27:0/2990253666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e680b4b40 0x7f7e680c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.460+0000 7f7e66ffd700 1 -- 172.21.15.27:0/2990253666 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e680b4f10 msgr2=0x7f7e6814b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:51.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.460+0000 7f7e66ffd700 1 --2- 172.21.15.27:0/2990253666 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e680b4f10 0x7f7e6814b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.460+0000 7f7e66ffd700 1 -- 172.21.15.27:0/2990253666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e680b4b40 msgr2=0x7f7e680c8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:51.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.460+0000 7f7e66ffd700 1 --2- 172.21.15.27:0/2990253666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e680b4b40 0x7f7e680c8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.460+0000 7f7e66ffd700 1 -- 172.21.15.27:0/2990253666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e680c98d0 con 0x7f7e680c9110 2024-01-11T08:25:51.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.460+0000 7f7e66ffd700 1 --2- 172.21.15.27:0/2990253666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e680c9110 0x7f7e6814ab30 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c001ed0 tx=0x7f7e5c001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:51.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.460+0000 7f7e4f7fe700 1 -- 172.21.15.27:0/2990253666 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c018030 con 0x7f7e680c9110 2024-01-11T08:25:51.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.461+0000 7f7e4f7fe700 1 -- 172.21.15.27:0/2990253666 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5c00f690 con 0x7f7e680c9110 2024-01-11T08:25:51.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.461+0000 7f7e4f7fe700 1 -- 172.21.15.27:0/2990253666 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c00fa50 con 0x7f7e680c9110 2024-01-11T08:25:51.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.461+0000 7f7e6dbff700 1 -- 172.21.15.27:0/2990253666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e680c9bb0 con 0x7f7e680c9110 2024-01-11T08:25:51.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.461+0000 7f7e6dbff700 1 -- 172.21.15.27:0/2990253666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e680ca170 con 0x7f7e680c9110 2024-01-11T08:25:51.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.462+0000 7f7e6dbff700 1 -- 172.21.15.27:0/2990253666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e6804c3c0 con 0x7f7e680c9110 2024-01-11T08:25:51.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.463+0000 7f7e4f7fe700 1 -- 172.21.15.27:0/2990253666 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f7e5c01b070 con 0x7f7e680c9110 2024-01-11T08:25:51.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.464+0000 7f7e4f7fe700 1 --2- 172.21.15.27:0/2990253666 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f7e54066ea0 0x7f7e54069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.464+0000 7f7e4f7fe700 1 -- 172.21.15.27:0/2990253666 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5483+0+0 (secure 0 0 0) 0x7f7e5c019070 con 0x7f7e680c9110 2024-01-11T08:25:51.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.464+0000 7f7e677fe700 1 --2- 172.21.15.27:0/2990253666 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f7e54066ea0 0x7f7e54069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.465+0000 7f7e677fe700 1 --2- 172.21.15.27:0/2990253666 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f7e54066ea0 0x7f7e54069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e580093f0 tx=0x7f7e58008040 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:51.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.468+0000 7f7e4f7fe700 1 -- 172.21.15.27:0/2990253666 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7e5c051540 con 0x7f7e680c9110 2024-01-11T08:25:51.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.672+0000 7f7e6dbff700 1 -- 172.21.15.27:0/2990253666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7e6815c810 con 0x7f7e680c9110 2024-01-11T08:25:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.676+0000 7f7e4f7fe700 1 -- 172.21.15.27:0/2990253666 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7e5c054b60 con 0x7f7e680c9110 2024-01-11T08:25:51.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.678+0000 7f7e6dbff700 1 -- 172.21.15.27:0/2990253666 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f7e54066ea0 msgr2=0x7f7e54069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:51.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.678+0000 7f7e6dbff700 1 --2- 172.21.15.27:0/2990253666 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f7e54066ea0 0x7f7e54069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e580093f0 tx=0x7f7e58008040 comp rx=0 tx=0).stop 2024-01-11T08:25:51.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.678+0000 7f7e6dbff700 1 -- 172.21.15.27:0/2990253666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e680c9110 msgr2=0x7f7e6814ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:51.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.678+0000 7f7e6dbff700 1 --2- 172.21.15.27:0/2990253666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e680c9110 0x7f7e6814ab30 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c001ed0 tx=0x7f7e5c001dc0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.678+0000 7f7e6dbff700 1 -- 172.21.15.27:0/2990253666 shutdown_connections 2024-01-11T08:25:51.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.678+0000 7f7e6dbff700 1 --2- 172.21.15.27:0/2990253666 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f7e54066ea0 0x7f7e54069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.678+0000 7f7e6dbff700 1 --2- 172.21.15.27:0/2990253666 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e680b4f10 0x7f7e6814b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.679+0000 7f7e6dbff700 1 --2- 172.21.15.27:0/2990253666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e680b4b40 0x7f7e680c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.679+0000 7f7e6dbff700 1 --2- 172.21.15.27:0/2990253666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e680c9110 0x7f7e6814ab30 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.679+0000 7f7e6dbff700 1 -- 172.21.15.27:0/2990253666 >> 172.21.15.27:0/2990253666 conn(0x7f7e6806ed40 msgr2=0x7f7e6806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.679+0000 7f7e6dbff700 1 -- 172.21.15.27:0/2990253666 shutdown_connections 2024-01-11T08:25:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.679+0000 7f7e6dbff700 1 -- 172.21.15.27:0/2990253666 wait complete. 2024-01-11T08:25:51.695 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2024-01-11T08:25:51.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.896+0000 7f9e9f3ff700 1 Processor -- start 2024-01-11T08:25:51.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.896+0000 7f9e9f3ff700 1 -- start start 2024-01-11T08:25:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.898+0000 7f9e9f3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9e981596c0 0x7f9e9815d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.898+0000 7f9e9f3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e980ca670 0x7f9e98068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.898+0000 7f9e9f3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e980a9cb0 con 0x7f9e98068b60 2024-01-11T08:25:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.898+0000 7f9e9f3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e98068fe0 con 0x7f9e981596c0 2024-01-11T08:25:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.898+0000 7f9e9f3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9815e210 con 0x7f9e980ca670 2024-01-11T08:25:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.898+0000 7f9e9d19b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9e981596c0 0x7f9e9815d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.898+0000 7f9e9d19b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9e981596c0 0x7f9e9815d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41758/0 (socket says 172.21.15.27:41758) 2024-01-11T08:25:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.898+0000 7f9e9d19b700 1 -- 172.21.15.27:0/879680430 learned_addr learned my addr 172.21.15.27:0/879680430 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.898+0000 7f9e9c99a700 1 --2- 172.21.15.27:0/879680430 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e980ca670 0x7f9e98068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.899+0000 7f9e9d19b700 1 -- 172.21.15.27:0/879680430 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e980ca670 msgr2=0x7f9e98068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.899+0000 7f9e9d19b700 1 --2- 172.21.15.27:0/879680430 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e980ca670 0x7f9e98068620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.899+0000 7f9e9d19b700 1 -- 172.21.15.27:0/879680430 >> v1:172.21.15.77:6789/0 conn(0x7f9e98068b60 legacy=0x7f9e98159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.899+0000 7f9e9d19b700 1 -- 172.21.15.27:0/879680430 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9815dc50 con 0x7f9e981596c0 2024-01-11T08:25:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.899+0000 7f9e9d19b700 1 --2- 172.21.15.27:0/879680430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e981596c0 0x7f9e9815d300 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f9e940099b0 tx=0x7f9e94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bad638dd2108955e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.899+0000 7f9e8ffff700 1 -- 172.21.15.27:0/879680430 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9400b860 con 0x7f9e981596c0 2024-01-11T08:25:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.899+0000 7f9e8ffff700 1 -- 172.21.15.27:0/879680430 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9400b9c0 con 0x7f9e981596c0 2024-01-11T08:25:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.899+0000 7f9e8ffff700 1 -- 172.21.15.27:0/879680430 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9400bd60 con 0x7f9e981596c0 2024-01-11T08:25:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.900+0000 7f9e9f3ff700 1 -- 172.21.15.27:0/879680430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e981596c0 msgr2=0x7f9e9815d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.900+0000 7f9e9f3ff700 1 --2- 172.21.15.27:0/879680430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e981596c0 0x7f9e9815d300 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f9e940099b0 tx=0x7f9e94009250 comp rx=0 tx=0).stop 2024-01-11T08:25:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.900+0000 7f9e9f3ff700 1 -- 172.21.15.27:0/879680430 shutdown_connections 2024-01-11T08:25:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.900+0000 7f9e9f3ff700 1 --2- 172.21.15.27:0/879680430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e981596c0 0x7f9e9815d300 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.900+0000 7f9e9f3ff700 1 --2- 172.21.15.27:0/879680430 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e980ca670 0x7f9e98068620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.900+0000 7f9e9f3ff700 1 -- 172.21.15.27:0/879680430 >> 172.21.15.27:0/879680430 conn(0x7f9e980c9f70 msgr2=0x7f9e9806deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.900+0000 7f9e9f3ff700 1 -- 172.21.15.27:0/879680430 shutdown_connections 2024-01-11T08:25:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.900+0000 7f9e9f3ff700 1 -- 172.21.15.27:0/879680430 wait complete. 2024-01-11T08:25:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.901+0000 7f9e9f3ff700 1 Processor -- start 2024-01-11T08:25:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.901+0000 7f9e9f3ff700 1 -- start start 2024-01-11T08:25:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.902+0000 7f9e9f3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e98068b60 0x7f9e980ac260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.902+0000 7f9e9f3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e980ca670 0x7f9e981564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.902+0000 7f9e9f3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e981596c0 0x7f9e9816d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.902+0000 7f9e9f3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e980abdd0 con 0x7f9e98068b60 2024-01-11T08:25:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.902+0000 7f9e9f3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9816e250 con 0x7f9e981596c0 2024-01-11T08:25:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.902+0000 7f9e9f3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9816e530 con 0x7f9e980ca670 2024-01-11T08:25:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.903+0000 7f9e9d19b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e98068b60 0x7f9e980ac260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.903+0000 7f9e9d19b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e98068b60 0x7f9e980ac260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41774/0 (socket says 172.21.15.27:41774) 2024-01-11T08:25:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.903+0000 7f9e9d19b700 1 -- 172.21.15.27:0/2015362789 learned_addr learned my addr 172.21.15.27:0/2015362789 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.903+0000 7f9e9c99a700 1 --2- 172.21.15.27:0/2015362789 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e980ca670 0x7f9e981564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.903+0000 7f9e9d19b700 1 -- 172.21.15.27:0/2015362789 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e980ca670 msgr2=0x7f9e981564f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.903+0000 7f9e9d99c700 1 --2- 172.21.15.27:0/2015362789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e981596c0 0x7f9e9816d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.903+0000 7f9e9d19b700 1 --2- 172.21.15.27:0/2015362789 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e980ca670 0x7f9e981564f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.903+0000 7f9e9d19b700 1 -- 172.21.15.27:0/2015362789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e981596c0 msgr2=0x7f9e9816d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.903+0000 7f9e9d19b700 1 --2- 172.21.15.27:0/2015362789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e981596c0 0x7f9e9816d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.903+0000 7f9e9d19b700 1 -- 172.21.15.27:0/2015362789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e940094f0 con 0x7f9e98068b60 2024-01-11T08:25:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.903+0000 7f9e9c99a700 1 --2- 172.21.15.27:0/2015362789 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e980ca670 0x7f9e981564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:25:51.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.903+0000 7f9e9d99c700 1 --2- 172.21.15.27:0/2015362789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e981596c0 0x7f9e9816d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:25:51.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.903+0000 7f9e9d19b700 1 --2- 172.21.15.27:0/2015362789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e98068b60 0x7f9e980ac260 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f9e9806d490 tx=0x7f9e94017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:51.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.904+0000 7f9e8d7fa700 1 -- 172.21.15.27:0/2015362789 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94024440 con 0x7f9e98068b60 2024-01-11T08:25:51.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.904+0000 7f9e8d7fa700 1 -- 172.21.15.27:0/2015362789 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e940245c0 con 0x7f9e98068b60 2024-01-11T08:25:51.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.904+0000 7f9e8d7fa700 1 -- 172.21.15.27:0/2015362789 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94024980 con 0x7f9e98068b60 2024-01-11T08:25:51.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.904+0000 7f9e9f3ff700 1 -- 172.21.15.27:0/2015362789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e980aba60 con 0x7f9e98068b60 2024-01-11T08:25:51.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.904+0000 7f9e9f3ff700 1 -- 172.21.15.27:0/2015362789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e980ac7a0 con 0x7f9e98068b60 2024-01-11T08:25:51.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.905+0000 7f9e9f3ff700 1 -- 172.21.15.27:0/2015362789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e980ab8c0 con 0x7f9e98068b60 2024-01-11T08:25:51.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.906+0000 7f9e8d7fa700 1 -- 172.21.15.27:0/2015362789 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 37) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f9e9401a420 con 0x7f9e98068b60 2024-01-11T08:25:51.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.907+0000 7f9e8d7fa700 1 --2- 172.21.15.27:0/2015362789 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e7c066ea0 0x7f9e7c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:51.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.907+0000 7f9e8d7fa700 1 -- 172.21.15.27:0/2015362789 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5483+0+0 (secure 0 0 0) 0x7f9e94086150 con 0x7f9e98068b60 2024-01-11T08:25:51.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.907+0000 7f9e9c99a700 1 --2- 172.21.15.27:0/2015362789 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e7c066ea0 0x7f9e7c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:51.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.908+0000 7f9e9c99a700 1 --2- 172.21.15.27:0/2015362789 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e7c066ea0 0x7f9e7c069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9e980acb40 tx=0x7f9e88008040 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:51.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:51.911+0000 7f9e8d7fa700 1 -- 172.21.15.27:0/2015362789 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9e980ab8c0 con 0x7f9e98068b60 2024-01-11T08:25:52.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:52.193+0000 7f9e9f3ff700 1 -- 172.21.15.27:0/2015362789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f9e980ab8c0 con 0x7f9e98068b60 2024-01-11T08:25:52.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:52.240+0000 7f9e8d7fa700 1 -- 172.21.15.27:0/2015362789 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9e940538b0 con 0x7f9e98068b60 2024-01-11T08:25:52.447 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:25:52.452 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:25:53.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.157+0000 7f9e8d7fa700 1 -- 172.21.15.27:0/2015362789 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v563) v1 ==== 156+0+0 (secure 0 0 0) 0x7f9e9407eed0 con 0x7f9e98068b60 2024-01-11T08:25:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.160+0000 7f9e9f3ff700 1 -- 172.21.15.27:0/2015362789 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e7c066ea0 msgr2=0x7f9e7c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.160+0000 7f9e9f3ff700 1 --2- 172.21.15.27:0/2015362789 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e7c066ea0 0x7f9e7c069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9e980acb40 tx=0x7f9e88008040 comp rx=0 tx=0).stop 2024-01-11T08:25:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.160+0000 7f9e9f3ff700 1 -- 172.21.15.27:0/2015362789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e98068b60 msgr2=0x7f9e980ac260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.160+0000 7f9e9f3ff700 1 --2- 172.21.15.27:0/2015362789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e98068b60 0x7f9e980ac260 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f9e9806d490 tx=0x7f9e94017000 comp rx=0 tx=0).stop 2024-01-11T08:25:53.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.160+0000 7f9e9f3ff700 1 -- 172.21.15.27:0/2015362789 shutdown_connections 2024-01-11T08:25:53.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.160+0000 7f9e9f3ff700 1 --2- 172.21.15.27:0/2015362789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e981596c0 0x7f9e9816d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:53.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.160+0000 7f9e9f3ff700 1 --2- 172.21.15.27:0/2015362789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e98068b60 0x7f9e980ac260 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:53.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.160+0000 7f9e9f3ff700 1 --2- 172.21.15.27:0/2015362789 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e7c066ea0 0x7f9e7c069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:53.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.160+0000 7f9e9f3ff700 1 --2- 172.21.15.27:0/2015362789 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e980ca670 0x7f9e981564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:53.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.160+0000 7f9e9f3ff700 1 -- 172.21.15.27:0/2015362789 >> 172.21.15.27:0/2015362789 conn(0x7f9e980c9f70 msgr2=0x7f9e9806a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:53.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.160+0000 7f9e9f3ff700 1 -- 172.21.15.27:0/2015362789 shutdown_connections 2024-01-11T08:25:53.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.160+0000 7f9e9f3ff700 1 -- 172.21.15.27:0/2015362789 wait complete. 2024-01-11T08:25:53.180 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2024-01-11T08:25:53.181 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2024-01-11T08:25:53.181 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-11T08:25:53.181 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2024-01-11T08:25:53.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.386+0000 7f3c20fff700 1 Processor -- start 2024-01-11T08:25:53.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.386+0000 7f3c20fff700 1 -- start start 2024-01-11T08:25:53.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.387+0000 7f3c20fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c1c0ca7b0 0x7f3c1c0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:53.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.387+0000 7f3c20fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3c1c159e80 0x7f3c1c15dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:53.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.387+0000 7f3c20fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c15d640 con 0x7f3c1c0ca3e0 2024-01-11T08:25:53.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.387+0000 7f3c20fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c1c15ae70 con 0x7f3c1c159e80 2024-01-11T08:25:53.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.387+0000 7f3c20fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c0cc4f0 con 0x7f3c1c0ca7b0 2024-01-11T08:25:53.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.388+0000 7f3c19d9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c1c0ca7b0 0x7f3c1c0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:53.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.388+0000 7f3c1ad9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3c1c159e80 0x7f3c1c15dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:53.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.388+0000 7f3c1ad9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3c1c159e80 0x7f3c1c15dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:39184/0 (socket says 172.21.15.27:39184) 2024-01-11T08:25:53.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.388+0000 7f3c1ad9d700 1 -- 172.21.15.27:0/50654662 learned_addr learned my addr 172.21.15.27:0/50654662 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:53.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.388+0000 7f3c1a59c700 1 --1- 172.21.15.27:0/50654662 >> v1:172.21.15.77:6789/0 conn(0x7f3c1c0ca3e0 0x7f3c1c0c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:58570/0 (socket says 172.21.15.27:58570) 2024-01-11T08:25:53.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.389+0000 7f3c1ad9d700 1 -- 172.21.15.27:0/50654662 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c1c0ca7b0 msgr2=0x7f3c1c0cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:53.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.389+0000 7f3c1ad9d700 1 --2- 172.21.15.27:0/50654662 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c1c0ca7b0 0x7f3c1c0cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:53.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.389+0000 7f3c1ad9d700 1 -- 172.21.15.27:0/50654662 >> v1:172.21.15.77:6789/0 conn(0x7f3c1c0ca3e0 legacy=0x7f3c1c0c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:53.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.389+0000 7f3c1ad9d700 1 -- 172.21.15.27:0/50654662 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1c15a5d0 con 0x7f3c1c159e80 2024-01-11T08:25:53.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.389+0000 7f3c1ad9d700 1 --2- 172.21.15.27:0/50654662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c1c159e80 0x7f3c1c15dad0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3c14009f50 tx=0x7f3c14009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=921b490eb247eb1a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:53.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.389+0000 7f3c1959a700 1 -- 172.21.15.27:0/50654662 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1400e070 con 0x7f3c1c159e80 2024-01-11T08:25:53.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.389+0000 7f3c1959a700 1 -- 172.21.15.27:0/50654662 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1400b9c0 con 0x7f3c1c159e80 2024-01-11T08:25:53.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.389+0000 7f3c1959a700 1 -- 172.21.15.27:0/50654662 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c140092e0 con 0x7f3c1c159e80 2024-01-11T08:25:53.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.390+0000 7f3c20fff700 1 -- 172.21.15.27:0/50654662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c1c159e80 msgr2=0x7f3c1c15dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:53.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.390+0000 7f3c20fff700 1 --2- 172.21.15.27:0/50654662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c1c159e80 0x7f3c1c15dad0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3c14009f50 tx=0x7f3c14009840 comp rx=0 tx=0).stop 2024-01-11T08:25:53.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.390+0000 7f3c20fff700 1 -- 172.21.15.27:0/50654662 shutdown_connections 2024-01-11T08:25:53.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.390+0000 7f3c20fff700 1 --2- 172.21.15.27:0/50654662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c1c159e80 0x7f3c1c15dad0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:53.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.390+0000 7f3c20fff700 1 --2- 172.21.15.27:0/50654662 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c1c0ca7b0 0x7f3c1c0cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:53.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.390+0000 7f3c20fff700 1 -- 172.21.15.27:0/50654662 >> 172.21.15.27:0/50654662 conn(0x7f3c1c0ce3e0 msgr2=0x7f3c1c0ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:53.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.390+0000 7f3c20fff700 1 -- 172.21.15.27:0/50654662 shutdown_connections 2024-01-11T08:25:53.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.390+0000 7f3c20fff700 1 -- 172.21.15.27:0/50654662 wait complete. 2024-01-11T08:25:53.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.391+0000 7f3c20fff700 1 Processor -- start 2024-01-11T08:25:53.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.392+0000 7f3c20fff700 1 -- start start 2024-01-11T08:25:53.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.393+0000 7f3c20fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c1c0ca3e0 0x7f3c1c14be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:53.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.393+0000 7f3c20fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c1c0ca7b0 0x7f3c1c14f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:53.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.393+0000 7f3c20fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c1c159e80 0x7f3c1c13add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:53.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.393+0000 7f3c20fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c0b2020 con 0x7f3c1c159e80 2024-01-11T08:25:53.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.393+0000 7f3c20fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c13b310 con 0x7f3c1c0ca7b0 2024-01-11T08:25:53.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.393+0000 7f3c20fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c0b1a10 con 0x7f3c1c0ca3e0 2024-01-11T08:25:53.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.393+0000 7f3c1ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c1c159e80 0x7f3c1c13add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:53.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.393+0000 7f3c1ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c1c159e80 0x7f3c1c13add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41782/0 (socket says 172.21.15.27:41782) 2024-01-11T08:25:53.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.393+0000 7f3c1ad9d700 1 -- 172.21.15.27:0/1481720473 learned_addr learned my addr 172.21.15.27:0/1481720473 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:53.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.393+0000 7f3c1a59c700 1 --2- 172.21.15.27:0/1481720473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c1c0ca3e0 0x7f3c1c14be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:53.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.393+0000 7f3c19d9b700 1 --2- 172.21.15.27:0/1481720473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c1c0ca7b0 0x7f3c1c14f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:53.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.393+0000 7f3c1ad9d700 1 -- 172.21.15.27:0/1481720473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c1c0ca3e0 msgr2=0x7f3c1c14be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:53.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.393+0000 7f3c1ad9d700 1 --2- 172.21.15.27:0/1481720473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c1c0ca3e0 0x7f3c1c14be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:53.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.393+0000 7f3c1ad9d700 1 -- 172.21.15.27:0/1481720473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c1c0ca7b0 msgr2=0x7f3c1c14f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:53.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.393+0000 7f3c1ad9d700 1 --2- 172.21.15.27:0/1481720473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c1c0ca7b0 0x7f3c1c14f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:53.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.393+0000 7f3c1ad9d700 1 -- 172.21.15.27:0/1481720473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c14009ae0 con 0x7f3c1c159e80 2024-01-11T08:25:53.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.394+0000 7f3c1ad9d700 1 --2- 172.21.15.27:0/1481720473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c1c159e80 0x7f3c1c13add0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f3c14009880 tx=0x7f3c14018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:53.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.394+0000 7f3c0a7fc700 1 -- 172.21.15.27:0/1481720473 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c14008ba0 con 0x7f3c1c159e80 2024-01-11T08:25:53.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.394+0000 7f3c0a7fc700 1 -- 172.21.15.27:0/1481720473 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c14008d00 con 0x7f3c1c159e80 2024-01-11T08:25:53.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.394+0000 7f3c20fff700 1 -- 172.21.15.27:0/1481720473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c1c1496c0 con 0x7f3c1c159e80 2024-01-11T08:25:53.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.394+0000 7f3c0a7fc700 1 -- 172.21.15.27:0/1481720473 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c14025690 con 0x7f3c1c159e80 2024-01-11T08:25:53.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.394+0000 7f3c20fff700 1 -- 172.21.15.27:0/1481720473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c1c1499a0 con 0x7f3c1c159e80 2024-01-11T08:25:53.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.395+0000 7f3c20fff700 1 -- 172.21.15.27:0/1481720473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c1c0b21d0 con 0x7f3c1c159e80 2024-01-11T08:25:53.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.396+0000 7f3c0a7fc700 1 -- 172.21.15.27:0/1481720473 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3c14008020 con 0x7f3c1c159e80 2024-01-11T08:25:53.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.397+0000 7f3c0a7fc700 1 --2- 172.21.15.27:0/1481720473 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f3bfc066f30 0x7f3bfc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:53.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.397+0000 7f3c0a7fc700 1 -- 172.21.15.27:0/1481720473 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5483+0+0 (secure 0 0 0) 0x7f3c14085c60 con 0x7f3c1c159e80 2024-01-11T08:25:53.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.398+0000 7f3c1a59c700 1 --2- 172.21.15.27:0/1481720473 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f3bfc066f30 0x7f3bfc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:53.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.398+0000 7f3c1a59c700 1 --2- 172.21.15.27:0/1481720473 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f3bfc066f30 0x7f3bfc0693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c004fd0 tx=0x7f3c0c007eb0 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:53.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.401+0000 7f3c0a7fc700 1 -- 172.21.15.27:0/1481720473 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3c1c0b21d0 con 0x7f3c1c159e80 2024-01-11T08:25:53.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:53.686+0000 7f3c20fff700 1 -- 172.21.15.27:0/1481720473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7f3c1c0718d0 con 0x7f3c1c159e80 2024-01-11T08:25:54.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.164+0000 7f3c0a7fc700 1 -- 172.21.15.27:0/1481720473 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v564) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3c14052f10 con 0x7f3c1c159e80 2024-01-11T08:25:54.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.167+0000 7f3c20fff700 1 -- 172.21.15.27:0/1481720473 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f3bfc066f30 msgr2=0x7f3bfc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:54.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.167+0000 7f3c20fff700 1 --2- 172.21.15.27:0/1481720473 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f3bfc066f30 0x7f3bfc0693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c004fd0 tx=0x7f3c0c007eb0 comp rx=0 tx=0).stop 2024-01-11T08:25:54.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.167+0000 7f3c20fff700 1 -- 172.21.15.27:0/1481720473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c1c159e80 msgr2=0x7f3c1c13add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:54.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.167+0000 7f3c20fff700 1 --2- 172.21.15.27:0/1481720473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c1c159e80 0x7f3c1c13add0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f3c14009880 tx=0x7f3c14018000 comp rx=0 tx=0).stop 2024-01-11T08:25:54.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.167+0000 7f3c20fff700 1 -- 172.21.15.27:0/1481720473 shutdown_connections 2024-01-11T08:25:54.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.167+0000 7f3c20fff700 1 --2- 172.21.15.27:0/1481720473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c1c159e80 0x7f3c1c13add0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:54.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.167+0000 7f3c20fff700 1 --2- 172.21.15.27:0/1481720473 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f3bfc066f30 0x7f3bfc0693e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:54.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.167+0000 7f3c20fff700 1 --2- 172.21.15.27:0/1481720473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c1c0ca3e0 0x7f3c1c14be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:54.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.167+0000 7f3c20fff700 1 --2- 172.21.15.27:0/1481720473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c1c0ca7b0 0x7f3c1c14f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:54.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.167+0000 7f3c20fff700 1 -- 172.21.15.27:0/1481720473 >> 172.21.15.27:0/1481720473 conn(0x7f3c1c0ce3e0 msgr2=0x7f3c1c151de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:54.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.167+0000 7f3c20fff700 1 -- 172.21.15.27:0/1481720473 shutdown_connections 2024-01-11T08:25:54.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.167+0000 7f3c20fff700 1 -- 172.21.15.27:0/1481720473 wait complete. 2024-01-11T08:25:54.188 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-11T08:25:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.392+0000 7f8a161ff700 1 Processor -- start 2024-01-11T08:25:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.392+0000 7f8a161ff700 1 -- start start 2024-01-11T08:25:54.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.393+0000 7f8a161ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a10157410 0x7f8a100ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:54.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.393+0000 7f8a161ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8a10069540 0x7f8a100c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:54.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.393+0000 7f8a161ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a100699d0 con 0x7f8a101577e0 2024-01-11T08:25:54.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.393+0000 7f8a161ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a10069cb0 con 0x7f8a10069540 2024-01-11T08:25:54.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.393+0000 7f8a161ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a10069f90 con 0x7f8a10157410 2024-01-11T08:25:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.394+0000 7f8a0ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8a10069540 0x7f8a100c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.394+0000 7f8a0ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8a10069540 0x7f8a100c6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35272/0 (socket says 172.21.15.27:35272) 2024-01-11T08:25:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.394+0000 7f8a0ffff700 1 -- 172.21.15.27:0/3276376686 learned_addr learned my addr 172.21.15.27:0/3276376686 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.394+0000 7f8a0f7fe700 1 --2- 172.21.15.27:0/3276376686 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a10157410 0x7f8a100ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.394+0000 7f8a0ffff700 1 -- 172.21.15.27:0/3276376686 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a10157410 msgr2=0x7f8a100ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.394+0000 7f8a0ffff700 1 --2- 172.21.15.27:0/3276376686 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a10157410 0x7f8a100ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.394+0000 7f8a0ffff700 1 -- 172.21.15.27:0/3276376686 >> v1:172.21.15.77:6789/0 conn(0x7f8a101577e0 legacy=0x7f8a100ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.394+0000 7f8a0ffff700 1 -- 172.21.15.27:0/3276376686 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1006a270 con 0x7f8a10069540 2024-01-11T08:25:54.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.394+0000 7f8a0ffff700 1 --2- 172.21.15.27:0/3276376686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a10069540 0x7f8a100c6ba0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f8a08009f20 tx=0x7f8a08009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2aae9d3bb1d30b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:54.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.395+0000 7f8a0e7fc700 1 -- 172.21.15.27:0/3276376686 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0800e070 con 0x7f8a10069540 2024-01-11T08:25:54.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.395+0000 7f8a0e7fc700 1 -- 172.21.15.27:0/3276376686 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0800ba80 con 0x7f8a10069540 2024-01-11T08:25:54.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.395+0000 7f8a0e7fc700 1 -- 172.21.15.27:0/3276376686 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a080092e0 con 0x7f8a10069540 2024-01-11T08:25:54.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.395+0000 7f8a161ff700 1 -- 172.21.15.27:0/3276376686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a10069540 msgr2=0x7f8a100c6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:54.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.395+0000 7f8a161ff700 1 --2- 172.21.15.27:0/3276376686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a10069540 0x7f8a100c6ba0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f8a08009f20 tx=0x7f8a08009840 comp rx=0 tx=0).stop 2024-01-11T08:25:54.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.395+0000 7f8a161ff700 1 -- 172.21.15.27:0/3276376686 shutdown_connections 2024-01-11T08:25:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.395+0000 7f8a161ff700 1 --2- 172.21.15.27:0/3276376686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a10069540 0x7f8a100c6ba0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.395+0000 7f8a161ff700 1 --2- 172.21.15.27:0/3276376686 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a10157410 0x7f8a100ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.395+0000 7f8a161ff700 1 -- 172.21.15.27:0/3276376686 >> 172.21.15.27:0/3276376686 conn(0x7f8a100680a0 msgr2=0x7f8a100cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.395+0000 7f8a161ff700 1 -- 172.21.15.27:0/3276376686 shutdown_connections 2024-01-11T08:25:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.395+0000 7f8a161ff700 1 -- 172.21.15.27:0/3276376686 wait complete. 2024-01-11T08:25:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.397+0000 7f8a161ff700 1 Processor -- start 2024-01-11T08:25:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.397+0000 7f8a161ff700 1 -- start start 2024-01-11T08:25:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.398+0000 7f8a161ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a10069540 0x7f8a100ab390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.398+0000 7f8a161ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a10157410 0x7f8a10143750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.398+0000 7f8a161ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a101577e0 0x7f8a100b0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.398+0000 7f8a161ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a100ac5e0 con 0x7f8a10069540 2024-01-11T08:25:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.398+0000 7f8a161ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a100ab9d0 con 0x7f8a10157410 2024-01-11T08:25:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.398+0000 7f8a161ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a100abcb0 con 0x7f8a101577e0 2024-01-11T08:25:54.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.398+0000 7f8a0f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a10069540 0x7f8a100ab390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.398+0000 7f8a0f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a10069540 0x7f8a100ab390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35288/0 (socket says 172.21.15.27:35288) 2024-01-11T08:25:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.398+0000 7f8a0f7fe700 1 -- 172.21.15.27:0/3033283372 learned_addr learned my addr 172.21.15.27:0/3033283372 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.398+0000 7f8a0effd700 1 --2- 172.21.15.27:0/3033283372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a10157410 0x7f8a10143750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.398+0000 7f8a0ffff700 1 --2- 172.21.15.27:0/3033283372 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a101577e0 0x7f8a100b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.398+0000 7f8a0f7fe700 1 -- 172.21.15.27:0/3033283372 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a101577e0 msgr2=0x7f8a100b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.398+0000 7f8a0f7fe700 1 --2- 172.21.15.27:0/3033283372 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a101577e0 0x7f8a100b0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.398+0000 7f8a0f7fe700 1 -- 172.21.15.27:0/3033283372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a10157410 msgr2=0x7f8a10143750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.398+0000 7f8a0f7fe700 1 --2- 172.21.15.27:0/3033283372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a10157410 0x7f8a10143750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.398+0000 7f8a0f7fe700 1 -- 172.21.15.27:0/3033283372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a08009ae0 con 0x7f8a10069540 2024-01-11T08:25:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.399+0000 7f8a0f7fe700 1 --2- 172.21.15.27:0/3033283372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a10069540 0x7f8a100ab390 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f8a0000d810 tx=0x7f8a00008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.399+0000 7f89f77fe700 1 -- 172.21.15.27:0/3033283372 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a000125f0 con 0x7f8a10069540 2024-01-11T08:25:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.399+0000 7f89f77fe700 1 -- 172.21.15.27:0/3033283372 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a00012750 con 0x7f8a10069540 2024-01-11T08:25:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.399+0000 7f89f77fe700 1 -- 172.21.15.27:0/3033283372 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a00012af0 con 0x7f8a10069540 2024-01-11T08:25:54.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.399+0000 7f8a161ff700 1 -- 172.21.15.27:0/3033283372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a100abf90 con 0x7f8a10069540 2024-01-11T08:25:54.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.399+0000 7f8a161ff700 1 -- 172.21.15.27:0/3033283372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a100b09d0 con 0x7f8a10069540 2024-01-11T08:25:54.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.400+0000 7f8a161ff700 1 -- 172.21.15.27:0/3033283372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a1004c3c0 con 0x7f8a10069540 2024-01-11T08:25:54.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.401+0000 7f89f77fe700 1 -- 172.21.15.27:0/3033283372 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8a0001e460 con 0x7f8a10069540 2024-01-11T08:25:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.402+0000 7f89f77fe700 1 --2- 172.21.15.27:0/3033283372 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f89fc066f30 0x7f89fc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.402+0000 7f89f77fe700 1 -- 172.21.15.27:0/3033283372 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5483+0+0 (secure 0 0 0) 0x7f8a00084b50 con 0x7f8a10069540 2024-01-11T08:25:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.402+0000 7f8a0effd700 1 --2- 172.21.15.27:0/3033283372 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f89fc066f30 0x7f89fc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:54.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.403+0000 7f8a0effd700 1 --2- 172.21.15.27:0/3033283372 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f89fc066f30 0x7f89fc0693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8a1006d560 tx=0x7f8a04007ec0 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:54.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.406+0000 7f89f77fe700 1 -- 172.21.15.27:0/3033283372 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8a00051e00 con 0x7f8a10069540 2024-01-11T08:25:54.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.612+0000 7f8a161ff700 1 -- 172.21.15.27:0/3033283372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f8a100c9840 con 0x7f8a10069540 2024-01-11T08:25:54.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.617+0000 7f89f77fe700 1 -- 172.21.15.27:0/3033283372 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8a00055420 con 0x7f8a10069540 2024-01-11T08:25:54.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.619+0000 7f8a161ff700 1 -- 172.21.15.27:0/3033283372 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f89fc066f30 msgr2=0x7f89fc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:54.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.619+0000 7f8a161ff700 1 --2- 172.21.15.27:0/3033283372 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f89fc066f30 0x7f89fc0693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8a1006d560 tx=0x7f8a04007ec0 comp rx=0 tx=0).stop 2024-01-11T08:25:54.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.619+0000 7f8a161ff700 1 -- 172.21.15.27:0/3033283372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a10069540 msgr2=0x7f8a100ab390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:54.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.619+0000 7f8a161ff700 1 --2- 172.21.15.27:0/3033283372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a10069540 0x7f8a100ab390 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f8a0000d810 tx=0x7f8a00008040 comp rx=0 tx=0).stop 2024-01-11T08:25:54.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.619+0000 7f8a161ff700 1 -- 172.21.15.27:0/3033283372 shutdown_connections 2024-01-11T08:25:54.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.619+0000 7f8a161ff700 1 --2- 172.21.15.27:0/3033283372 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f89fc066f30 0x7f89fc0693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:54.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.619+0000 7f8a161ff700 1 --2- 172.21.15.27:0/3033283372 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a101577e0 0x7f8a100b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:54.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.619+0000 7f8a161ff700 1 --2- 172.21.15.27:0/3033283372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a10069540 0x7f8a100ab390 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:54.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.619+0000 7f8a161ff700 1 --2- 172.21.15.27:0/3033283372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a10157410 0x7f8a10143750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:54.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.619+0000 7f8a161ff700 1 -- 172.21.15.27:0/3033283372 >> 172.21.15.27:0/3033283372 conn(0x7f8a100680a0 msgr2=0x7f8a10152b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:54.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.619+0000 7f8a161ff700 1 -- 172.21.15.27:0/3033283372 shutdown_connections 2024-01-11T08:25:54.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:54.619+0000 7f8a161ff700 1 -- 172.21.15.27:0/3033283372 wait complete. 2024-01-11T08:25:54.636 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T08:25:54.636 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:25:54.732 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:25:54.845 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T08:25:54.845 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-11T08:25:54.846 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:25:55.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.046+0000 7f25bbfff700 1 Processor -- start 2024-01-11T08:25:55.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.046+0000 7f25bbfff700 1 -- start start 2024-01-11T08:25:55.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.048+0000 7f25bbfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f25b40ca3e0 0x7f25b40cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:55.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.048+0000 7f25bbfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f25b40ca7b0 0x7f25b4159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:55.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.048+0000 7f25bbfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f25b40c8a20 0x7f25b40c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:55.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.048+0000 7f25bbfff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b415ae70 con 0x7f25b40ca7b0 2024-01-11T08:25:55.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.048+0000 7f25bbfff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b415a3c0 con 0x7f25b40c8a20 2024-01-11T08:25:55.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.048+0000 7f25bbfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b40cc4f0 con 0x7f25b40ca3e0 2024-01-11T08:25:55.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.048+0000 7f25b959a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f25b40ca7b0 0x7f25b4159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:55.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.048+0000 7f25b959a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f25b40ca7b0 0x7f25b4159e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35298/0 (socket says 172.21.15.27:35298) 2024-01-11T08:25:55.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.048+0000 7f25b959a700 1 -- 172.21.15.27:0/3544999621 learned_addr learned my addr 172.21.15.27:0/3544999621 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:55.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.048+0000 7f25ba59c700 1 --2- 172.21.15.27:0/3544999621 >> v2:172.21.15.77:3300/0 conn(0x7f25b40c8a20 0x7f25b40c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:55.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.048+0000 7f25b9d9b700 1 --2- 172.21.15.27:0/3544999621 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f25b40ca3e0 0x7f25b40cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:55.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.049+0000 7f25b959a700 1 -- 172.21.15.27:0/3544999621 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f25b40ca3e0 msgr2=0x7f25b40cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:55.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.049+0000 7f25b959a700 1 --2- 172.21.15.27:0/3544999621 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f25b40ca3e0 0x7f25b40cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:55.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.049+0000 7f25b959a700 1 -- 172.21.15.27:0/3544999621 >> v2:172.21.15.77:3300/0 conn(0x7f25b40c8a20 msgr2=0x7f25b40c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:55.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.049+0000 7f25b959a700 1 --2- 172.21.15.27:0/3544999621 >> v2:172.21.15.77:3300/0 conn(0x7f25b40c8a20 0x7f25b40c8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:55.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.049+0000 7f25b959a700 1 -- 172.21.15.27:0/3544999621 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b415d640 con 0x7f25b40ca7b0 2024-01-11T08:25:55.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.049+0000 7f25b959a700 1 --2- 172.21.15.27:0/3544999621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b40ca7b0 0x7f25b4159e80 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f25b40ce2b0 tx=0x7f25a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc6fefc3c01fd2b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:55.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.049+0000 7f25b8d99700 1 -- 172.21.15.27:0/3544999621 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a400b640 con 0x7f25b40ca7b0 2024-01-11T08:25:55.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.049+0000 7f25b8d99700 1 -- 172.21.15.27:0/3544999621 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a400b7a0 con 0x7f25b40ca7b0 2024-01-11T08:25:55.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.049+0000 7f25b8d99700 1 -- 172.21.15.27:0/3544999621 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a400bb60 con 0x7f25b40ca7b0 2024-01-11T08:25:55.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.050+0000 7f25bbfff700 1 -- 172.21.15.27:0/3544999621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b40ca7b0 msgr2=0x7f25b4159e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:55.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.050+0000 7f25bbfff700 1 --2- 172.21.15.27:0/3544999621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b40ca7b0 0x7f25b4159e80 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f25b40ce2b0 tx=0x7f25a4009250 comp rx=0 tx=0).stop 2024-01-11T08:25:55.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.050+0000 7f25bbfff700 1 -- 172.21.15.27:0/3544999621 shutdown_connections 2024-01-11T08:25:55.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.050+0000 7f25bbfff700 1 --2- 172.21.15.27:0/3544999621 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f25b40ca3e0 0x7f25b40cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:55.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.050+0000 7f25bbfff700 1 --2- 172.21.15.27:0/3544999621 >> v2:172.21.15.77:3300/0 conn(0x7f25b40c8a20 0x7f25b40c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:55.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.050+0000 7f25bbfff700 1 --2- 172.21.15.27:0/3544999621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b40ca7b0 0x7f25b4159e80 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:55.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.050+0000 7f25bbfff700 1 -- 172.21.15.27:0/3544999621 >> 172.21.15.27:0/3544999621 conn(0x7f25b40ce3e0 msgr2=0x7f25b40ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:55.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.050+0000 7f25bbfff700 1 -- 172.21.15.27:0/3544999621 shutdown_connections 2024-01-11T08:25:55.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.050+0000 7f25bbfff700 1 -- 172.21.15.27:0/3544999621 wait complete. 2024-01-11T08:25:55.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.052+0000 7f25bbfff700 1 Processor -- start 2024-01-11T08:25:55.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.052+0000 7f25bbfff700 1 -- start start 2024-01-11T08:25:55.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.053+0000 7f25bbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25b40c8a20 0x7f25b416ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:55.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.053+0000 7f25bbfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f25b40ca3e0 0x7f25b4150690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:55.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.053+0000 7f25bbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b40ca7b0 0x7f25b40b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:55.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.053+0000 7f25bbfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b40b3f30 con 0x7f25b40ca7b0 2024-01-11T08:25:55.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.053+0000 7f25bbfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b4150e50 con 0x7f25b40c8a20 2024-01-11T08:25:55.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.053+0000 7f25bbfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b4150bd0 con 0x7f25b40ca3e0 2024-01-11T08:25:55.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.053+0000 7f25ba59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b40ca7b0 0x7f25b40b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:55.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.053+0000 7f25ba59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b40ca7b0 0x7f25b40b1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35312/0 (socket says 172.21.15.27:35312) 2024-01-11T08:25:55.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.053+0000 7f25ba59c700 1 -- 172.21.15.27:0/1684817732 learned_addr learned my addr 172.21.15.27:0/1684817732 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:55.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.053+0000 7f25b959a700 1 --2- 172.21.15.27:0/1684817732 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f25b40ca3e0 0x7f25b4150690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:55.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.053+0000 7f25b9d9b700 1 --2- 172.21.15.27:0/1684817732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25b40c8a20 0x7f25b416ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:55.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.053+0000 7f25ba59c700 1 -- 172.21.15.27:0/1684817732 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f25b40ca3e0 msgr2=0x7f25b4150690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:55.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.053+0000 7f25ba59c700 1 --2- 172.21.15.27:0/1684817732 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f25b40ca3e0 0x7f25b4150690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:55.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.053+0000 7f25ba59c700 1 -- 172.21.15.27:0/1684817732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25b40c8a20 msgr2=0x7f25b416ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:55.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.053+0000 7f25ba59c700 1 --2- 172.21.15.27:0/1684817732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25b40c8a20 0x7f25b416ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:55.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.053+0000 7f25ba59c700 1 -- 172.21.15.27:0/1684817732 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a40094f0 con 0x7f25b40ca7b0 2024-01-11T08:25:55.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.053+0000 7f25b959a700 1 --2- 172.21.15.27:0/1684817732 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f25b40ca3e0 0x7f25b4150690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:25:55.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.054+0000 7f25b9d9b700 1 --2- 172.21.15.27:0/1684817732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25b40c8a20 0x7f25b416ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:25:55.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.054+0000 7f25ba59c700 1 --2- 172.21.15.27:0/1684817732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b40ca7b0 0x7f25b40b1180 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f25ac009e20 tx=0x7f25ac00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:55.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.054+0000 7f25a9ffb700 1 -- 172.21.15.27:0/1684817732 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac00cc20 con 0x7f25b40ca7b0 2024-01-11T08:25:55.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.054+0000 7f25a9ffb700 1 -- 172.21.15.27:0/1684817732 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ac00f040 con 0x7f25b40ca7b0 2024-01-11T08:25:55.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.054+0000 7f25bbfff700 1 -- 172.21.15.27:0/1684817732 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b415b160 con 0x7f25b40ca7b0 2024-01-11T08:25:55.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.054+0000 7f25a9ffb700 1 -- 172.21.15.27:0/1684817732 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac014480 con 0x7f25b40ca7b0 2024-01-11T08:25:55.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.054+0000 7f25bbfff700 1 -- 172.21.15.27:0/1684817732 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b415ba00 con 0x7f25b40ca7b0 2024-01-11T08:25:55.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.055+0000 7f25bbfff700 1 -- 172.21.15.27:0/1684817732 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b415b560 con 0x7f25b40ca7b0 2024-01-11T08:25:55.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.056+0000 7f25a9ffb700 1 -- 172.21.15.27:0/1684817732 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f25ac014830 con 0x7f25b40ca7b0 2024-01-11T08:25:55.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.057+0000 7f25a9ffb700 1 --2- 172.21.15.27:0/1684817732 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2598066f30 0x7f25980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:55.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.057+0000 7f25a9ffb700 1 -- 172.21.15.27:0/1684817732 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5483+0+0 (secure 0 0 0) 0x7f25ac085040 con 0x7f25b40ca7b0 2024-01-11T08:25:55.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.058+0000 7f25b9d9b700 1 --2- 172.21.15.27:0/1684817732 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2598066f30 0x7f25980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:55.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.058+0000 7f25b9d9b700 1 --2- 172.21.15.27:0/1684817732 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2598066f30 0x7f25980693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f25b0006f00 tx=0x7f25b0008040 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:55.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.061+0000 7f25a9ffb700 1 -- 172.21.15.27:0/1684817732 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f25ac052390 con 0x7f25b40ca7b0 2024-01-11T08:25:55.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.312+0000 7f25bbfff700 1 -- 172.21.15.27:0/1684817732 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25b415b350 con 0x7f25b40ca7b0 2024-01-11T08:25:55.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.313+0000 7f25a9ffb700 1 -- 172.21.15.27:0/1684817732 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+3529 (secure 0 0 0) 0x7f25ac018de0 con 0x7f25b40ca7b0 2024-01-11T08:25:55.312 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:55.312 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":35,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5606,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2845124438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6835","nonce":2845124438},{"type":"v1","addr":"172.21.15.77:6837","nonce":2845124438}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5618,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3958104902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3958104902},{"type":"v1","addr":"172.21.15.27:6835","nonce":3958104902}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":5852,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/2235501160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2235501160},{"type":"v1","addr":"172.21.15.77:6833","nonce":2235501160}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6091,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/2612413022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2612413022},{"type":"v1","addr":"172.21.15.27:6833","nonce":2612413022}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[]} 2024-01-11T08:25:55.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.315+0000 7f25bbfff700 1 -- 172.21.15.27:0/1684817732 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2598066f30 msgr2=0x7f25980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:55.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.315+0000 7f25bbfff700 1 --2- 172.21.15.27:0/1684817732 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2598066f30 0x7f25980693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f25b0006f00 tx=0x7f25b0008040 comp rx=0 tx=0).stop 2024-01-11T08:25:55.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.315+0000 7f25bbfff700 1 -- 172.21.15.27:0/1684817732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b40ca7b0 msgr2=0x7f25b40b1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:55.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.315+0000 7f25bbfff700 1 --2- 172.21.15.27:0/1684817732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b40ca7b0 0x7f25b40b1180 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f25ac009e20 tx=0x7f25ac00b040 comp rx=0 tx=0).stop 2024-01-11T08:25:55.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.315+0000 7f25bbfff700 1 -- 172.21.15.27:0/1684817732 shutdown_connections 2024-01-11T08:25:55.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.315+0000 7f25bbfff700 1 --2- 172.21.15.27:0/1684817732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b40ca7b0 0x7f25b40b1180 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:55.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.315+0000 7f25bbfff700 1 --2- 172.21.15.27:0/1684817732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25b40c8a20 0x7f25b416ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:55.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.315+0000 7f25bbfff700 1 --2- 172.21.15.27:0/1684817732 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2598066f30 0x7f25980693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:55.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.315+0000 7f25bbfff700 1 --2- 172.21.15.27:0/1684817732 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f25b40ca3e0 0x7f25b4150690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:55.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.315+0000 7f25bbfff700 1 -- 172.21.15.27:0/1684817732 >> 172.21.15.27:0/1684817732 conn(0x7f25b40ce3e0 msgr2=0x7f25b40c4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:55.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.316+0000 7f25bbfff700 1 -- 172.21.15.27:0/1684817732 shutdown_connections 2024-01-11T08:25:55.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.316+0000 7f25bbfff700 1 -- 172.21.15.27:0/1684817732 wait complete. 2024-01-11T08:25:55.316 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 35 2024-01-11T08:25:55.333 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-11T08:25:55.333 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-11T08:25:55.333 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-11T08:25:55.333 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:25:55.333 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-11T08:25:55.333 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-11T08:25:55.334 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-11T08:25:55.334 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:25:55.334 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-11T08:25:55.334 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-11T08:25:55.334 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-11T08:25:55.334 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:25:55.334 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-11T08:25:55.334 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-11T08:25:55.335 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-11T08:25:55.335 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:25:55.348 INFO:tasks.ceph.mds.c:Stopped 2024-01-11T08:25:55.349 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-11T08:25:55.349 INFO:tasks.ceph.mds.a:Stopped 2024-01-11T08:25:55.349 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-11T08:25:55.350 INFO:tasks.ceph.mds.b:Stopped 2024-01-11T08:25:55.351 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-11T08:25:55.351 INFO:tasks.ceph.mds.d:Stopped 2024-01-11T08:25:55.352 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-11T08:25:55.394 INFO:tasks.ceph.mds.c:Started 2024-01-11T08:25:55.394 DEBUG:teuthology.parallel:result is None 2024-01-11T08:25:55.395 INFO:tasks.ceph.mds.b:Started 2024-01-11T08:25:55.396 DEBUG:teuthology.parallel:result is None 2024-01-11T08:25:55.397 INFO:tasks.ceph.mds.a:Started 2024-01-11T08:25:55.397 DEBUG:teuthology.parallel:result is None 2024-01-11T08:25:55.399 INFO:tasks.ceph.mds.d:Started 2024-01-11T08:25:55.399 DEBUG:teuthology.parallel:result is None 2024-01-11T08:25:55.399 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-11T08:25:55.544 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T08:25:55.545 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T08:25:55.545 INFO:tasks.ceph.mds.d.smithi077.stdout:starting mds.d at 2024-01-11T08:25:55.545 INFO:tasks.ceph.mds.b.smithi077.stdout:starting mds.b at 2024-01-11T08:25:55.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.614+0000 7f18647ff700 1 Processor -- start 2024-01-11T08:25:55.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.614+0000 7f18647ff700 1 -- start start 2024-01-11T08:25:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.615+0000 7f18647ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f185c156ab0 0x7f185c0ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.616+0000 7f18647ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185c0adc80 con 0x7f185c0b0550 2024-01-11T08:25:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.616+0000 7f18647ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185c0adf60 con 0x7f185c0cc9b0 2024-01-11T08:25:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.616+0000 7f18647ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f185c0ae240 con 0x7f185c156ab0 2024-01-11T08:25:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.616+0000 7f1862d9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f185c0b0550 0x7f185c160070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45740/0 (socket says 172.21.15.27:45740) 2024-01-11T08:25:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.616+0000 7f1862d9c700 1 -- 172.21.15.27:0/1037274402 learned_addr learned my addr 172.21.15.27:0/1037274402 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.616+0000 7f186259b700 1 --2- 172.21.15.27:0/1037274402 >> v2:172.21.15.77:3300/0 conn(0x7f185c156ab0 0x7f185c0ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.616+0000 7f1861599700 1 -- 172.21.15.27:0/1037274402 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016625120 0 0) 0x7f185c0adc80 con 0x7f185c0b0550 2024-01-11T08:25:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.616+0000 7f1861599700 1 -- 172.21.15.27:0/1037274402 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1848003570 con 0x7f185c0b0550 2024-01-11T08:25:55.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.617+0000 7f186259b700 1 -- 172.21.15.27:0/1037274402 >> v1:172.21.15.77:6789/0 conn(0x7f185c0cc9b0 legacy=0x7f185c156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:55.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.617+0000 7f186259b700 1 -- 172.21.15.27:0/1037274402 >> v1:172.21.15.27:6789/0 conn(0x7f185c0b0550 legacy=0x7f185c160070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:55.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.617+0000 7f186259b700 1 -- 172.21.15.27:0/1037274402 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c0ae520 con 0x7f185c156ab0 2024-01-11T08:25:55.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.617+0000 7f186259b700 1 --2- 172.21.15.27:0/1037274402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c156ab0 0x7f185c0ad650 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f18580098e0 tx=0x7f18580091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6958565927ba3507 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:55.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.617+0000 7f1861599700 1 -- 172.21.15.27:0/1037274402 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185800b640 con 0x7f185c156ab0 2024-01-11T08:25:55.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.617+0000 7f1861599700 1 -- 172.21.15.27:0/1037274402 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185800b7a0 con 0x7f185c156ab0 2024-01-11T08:25:55.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.617+0000 7f1861599700 1 -- 172.21.15.27:0/1037274402 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185800bb40 con 0x7f185c156ab0 2024-01-11T08:25:55.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.618+0000 7f18647ff700 1 -- 172.21.15.27:0/1037274402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c156ab0 msgr2=0x7f185c0ad650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:55.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.618+0000 7f18647ff700 1 --2- 172.21.15.27:0/1037274402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c156ab0 0x7f185c0ad650 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f18580098e0 tx=0x7f18580091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:55.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.618+0000 7f18647ff700 1 -- 172.21.15.27:0/1037274402 shutdown_connections 2024-01-11T08:25:55.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.618+0000 7f18647ff700 1 --2- 172.21.15.27:0/1037274402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c156ab0 0x7f185c0ad650 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:55.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.618+0000 7f18647ff700 1 -- 172.21.15.27:0/1037274402 >> 172.21.15.27:0/1037274402 conn(0x7f185c0680b0 msgr2=0x7f185c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:55.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.618+0000 7f18647ff700 1 -- 172.21.15.27:0/1037274402 shutdown_connections 2024-01-11T08:25:55.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.618+0000 7f18647ff700 1 -- 172.21.15.27:0/1037274402 wait complete. 2024-01-11T08:25:55.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.619+0000 7f18647ff700 1 Processor -- start 2024-01-11T08:25:55.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.619+0000 7f18647ff700 1 -- start start 2024-01-11T08:25:55.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.620+0000 7f18647ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c0b0550 0x7f185c06d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:55.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.620+0000 7f18647ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f185c0cc9b0 0x7f185c141a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:55.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.620+0000 7f18647ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f185c156ab0 0x7f185c1581c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:55.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.620+0000 7f18647ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c06daa0 con 0x7f185c0cc9b0 2024-01-11T08:25:55.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.620+0000 7f18647ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c0cb720 con 0x7f185c0b0550 2024-01-11T08:25:55.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.620+0000 7f18647ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185c158c60 con 0x7f185c156ab0 2024-01-11T08:25:55.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.621+0000 7f1861d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f185c0cc9b0 0x7f185c141a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:55.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.621+0000 7f1861d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f185c0cc9b0 0x7f185c141a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35360/0 (socket says 172.21.15.27:35360) 2024-01-11T08:25:55.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.621+0000 7f1861d9a700 1 -- 172.21.15.27:0/2993142492 learned_addr learned my addr 172.21.15.27:0/2993142492 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:55.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.621+0000 7f186259b700 1 --2- 172.21.15.27:0/2993142492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c0b0550 0x7f185c06d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:55.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.621+0000 7f1862d9c700 1 --2- 172.21.15.27:0/2993142492 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f185c156ab0 0x7f185c1581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:55.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.621+0000 7f1861d9a700 1 -- 172.21.15.27:0/2993142492 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f185c156ab0 msgr2=0x7f185c1581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.621+0000 7f1861d9a700 1 --2- 172.21.15.27:0/2993142492 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f185c156ab0 0x7f185c1581c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.621+0000 7f1861d9a700 1 -- 172.21.15.27:0/2993142492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c0b0550 msgr2=0x7f185c06d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.621+0000 7f1861d9a700 1 --2- 172.21.15.27:0/2993142492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c0b0550 0x7f185c06d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.621+0000 7f1861d9a700 1 -- 172.21.15.27:0/2993142492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1858009420 con 0x7f185c0cc9b0 2024-01-11T08:25:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.621+0000 7f1861d9a700 1 --2- 172.21.15.27:0/2993142492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f185c0cc9b0 0x7f185c141a90 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f184c00c590 tx=0x7f184c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.622+0000 7f18527fc700 1 -- 172.21.15.27:0/2993142492 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c011590 con 0x7f185c0cc9b0 2024-01-11T08:25:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.622+0000 7f18527fc700 1 -- 172.21.15.27:0/2993142492 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184c011710 con 0x7f185c0cc9b0 2024-01-11T08:25:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.622+0000 7f18527fc700 1 -- 172.21.15.27:0/2993142492 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c011ad0 con 0x7f185c0cc9b0 2024-01-11T08:25:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.622+0000 7f18647ff700 1 -- 172.21.15.27:0/2993142492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f185c158f40 con 0x7f185c0cc9b0 2024-01-11T08:25:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.622+0000 7f18647ff700 1 -- 172.21.15.27:0/2993142492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185c0c9000 con 0x7f185c0cc9b0 2024-01-11T08:25:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.623+0000 7f18647ff700 1 -- 172.21.15.27:0/2993142492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f185c0cb890 con 0x7f185c0cc9b0 2024-01-11T08:25:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.624+0000 7f18527fc700 1 -- 172.21.15.27:0/2993142492 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f184c01b3b0 con 0x7f185c0cc9b0 2024-01-11T08:25:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.625+0000 7f18527fc700 1 --2- 172.21.15.27:0/2993142492 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f1840066f30 0x7f18400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.625+0000 7f18527fc700 1 -- 172.21.15.27:0/2993142492 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5483+0+0 (secure 0 0 0) 0x7f184c083960 con 0x7f185c0cc9b0 2024-01-11T08:25:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.625+0000 7f186259b700 1 --2- 172.21.15.27:0/2993142492 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f1840066f30 0x7f18400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.626+0000 7f186259b700 1 --2- 172.21.15.27:0/2993142492 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f1840066f30 0x7f18400693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f18580093f0 tx=0x7f1858017000 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:55.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.629+0000 7f18527fc700 1 -- 172.21.15.27:0/2993142492 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f184c050c10 con 0x7f185c0cc9b0 2024-01-11T08:25:55.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:55.847+0000 7f18647ff700 1 -- 172.21.15.27:0/2993142492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f185c04c3c0 con 0x7f185c0cc9b0 2024-01-11T08:25:56.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.177+0000 7f18527fc700 1 -- 172.21.15.27:0/2993142492 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v82) v1 ==== 97+0+0 (secure 0 0 0) 0x7f184c017070 con 0x7f185c0cc9b0 2024-01-11T08:25:56.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.180+0000 7f18647ff700 1 -- 172.21.15.27:0/2993142492 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f1840066f30 msgr2=0x7f18400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:56.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.180+0000 7f18647ff700 1 --2- 172.21.15.27:0/2993142492 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f1840066f30 0x7f18400693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f18580093f0 tx=0x7f1858017000 comp rx=0 tx=0).stop 2024-01-11T08:25:56.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.180+0000 7f18647ff700 1 -- 172.21.15.27:0/2993142492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f185c0cc9b0 msgr2=0x7f185c141a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:56.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.180+0000 7f18647ff700 1 --2- 172.21.15.27:0/2993142492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f185c0cc9b0 0x7f185c141a90 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f184c00c590 tx=0x7f184c00c8a0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.180+0000 7f18647ff700 1 -- 172.21.15.27:0/2993142492 shutdown_connections 2024-01-11T08:25:56.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.180+0000 7f18647ff700 1 --2- 172.21.15.27:0/2993142492 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f1840066f30 0x7f18400693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.180+0000 7f18647ff700 1 --2- 172.21.15.27:0/2993142492 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f185c156ab0 0x7f185c1581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.180+0000 7f18647ff700 1 --2- 172.21.15.27:0/2993142492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c0b0550 0x7f185c06d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.180+0000 7f18647ff700 1 --2- 172.21.15.27:0/2993142492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f185c0cc9b0 0x7f185c141a90 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.180+0000 7f18647ff700 1 -- 172.21.15.27:0/2993142492 >> 172.21.15.27:0/2993142492 conn(0x7f185c0680b0 msgr2=0x7f185c14eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:56.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.180+0000 7f18647ff700 1 -- 172.21.15.27:0/2993142492 shutdown_connections 2024-01-11T08:25:56.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.180+0000 7f18647ff700 1 -- 172.21.15.27:0/2993142492 wait complete. 2024-01-11T08:25:56.180 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-11T08:25:56.200 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-11T08:25:56.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.401+0000 7f2e6e5ff700 1 Processor -- start 2024-01-11T08:25:56.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.401+0000 7f2e6e5ff700 1 -- start start 2024-01-11T08:25:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.402+0000 7f2e6e5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2e68159650 0x7f2e6806bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.402+0000 7f2e6e5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e68159a20 0x7f2e680b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.402+0000 7f2e6e5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2e680b2730 0x7f2e680b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.402+0000 7f2e6e5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e6806e700 con 0x7f2e680b2730 2024-01-11T08:25:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.402+0000 7f2e6e5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e680b11a0 con 0x7f2e68159650 2024-01-11T08:25:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.402+0000 7f2e6e5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6806c360 con 0x7f2e68159a20 2024-01-11T08:25:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.402+0000 7f2e6cb9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2e680b2730 0x7f2e680b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.403+0000 7f2e6cb9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2e680b2730 0x7f2e680b2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35370/0 (socket says 172.21.15.27:35370) 2024-01-11T08:25:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.403+0000 7f2e6cb9c700 1 -- 172.21.15.27:0/2677917351 learned_addr learned my addr 172.21.15.27:0/2677917351 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.403+0000 7f2e67fff700 1 --2- 172.21.15.27:0/2677917351 >> v2:172.21.15.77:3300/0 conn(0x7f2e68159650 0x7f2e6806bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.403+0000 7f2e677fe700 1 --2- 172.21.15.27:0/2677917351 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e68159a20 0x7f2e680b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.403+0000 7f2e6cb9c700 1 -- 172.21.15.27:0/2677917351 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e68159a20 msgr2=0x7f2e680b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.403+0000 7f2e6cb9c700 1 --2- 172.21.15.27:0/2677917351 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e68159a20 0x7f2e680b0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.403+0000 7f2e6cb9c700 1 -- 172.21.15.27:0/2677917351 >> v2:172.21.15.77:3300/0 conn(0x7f2e68159650 msgr2=0x7f2e6806bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.403+0000 7f2e6cb9c700 1 --2- 172.21.15.27:0/2677917351 >> v2:172.21.15.77:3300/0 conn(0x7f2e68159650 0x7f2e6806bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.403+0000 7f2e6cb9c700 1 -- 172.21.15.27:0/2677917351 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6806c640 con 0x7f2e680b2730 2024-01-11T08:25:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.403+0000 7f2e677fe700 1 --2- 172.21.15.27:0/2677917351 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e68159a20 0x7f2e680b0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.403+0000 7f2e6cb9c700 1 --2- 172.21.15.27:0/2677917351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e680b2730 0x7f2e680b2bb0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f2e6806e500 tx=0x7f2e60009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee978035fe6a787b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.404+0000 7f2e66ffd700 1 -- 172.21.15.27:0/2677917351 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6000e070 con 0x7f2e680b2730 2024-01-11T08:25:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.404+0000 7f2e66ffd700 1 -- 172.21.15.27:0/2677917351 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6000ba20 con 0x7f2e680b2730 2024-01-11T08:25:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.404+0000 7f2e66ffd700 1 -- 172.21.15.27:0/2677917351 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e600092e0 con 0x7f2e680b2730 2024-01-11T08:25:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.404+0000 7f2e6e5ff700 1 -- 172.21.15.27:0/2677917351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e680b2730 msgr2=0x7f2e680b2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.404+0000 7f2e6e5ff700 1 --2- 172.21.15.27:0/2677917351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e680b2730 0x7f2e680b2bb0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f2e6806e500 tx=0x7f2e60009840 comp rx=0 tx=0).stop 2024-01-11T08:25:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.404+0000 7f2e6e5ff700 1 -- 172.21.15.27:0/2677917351 shutdown_connections 2024-01-11T08:25:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.404+0000 7f2e6e5ff700 1 --2- 172.21.15.27:0/2677917351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e680b2730 0x7f2e680b2bb0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.404+0000 7f2e6e5ff700 1 --2- 172.21.15.27:0/2677917351 >> v2:172.21.15.77:3300/0 conn(0x7f2e68159650 0x7f2e6806bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.404+0000 7f2e6e5ff700 1 --2- 172.21.15.27:0/2677917351 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e68159a20 0x7f2e680b0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.404+0000 7f2e6e5ff700 1 -- 172.21.15.27:0/2677917351 >> 172.21.15.27:0/2677917351 conn(0x7f2e6806b200 msgr2=0x7f2e680aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.404+0000 7f2e6e5ff700 1 -- 172.21.15.27:0/2677917351 shutdown_connections 2024-01-11T08:25:56.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.404+0000 7f2e6e5ff700 1 -- 172.21.15.27:0/2677917351 wait complete. 2024-01-11T08:25:56.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.406+0000 7f2e6e5ff700 1 Processor -- start 2024-01-11T08:25:56.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.406+0000 7f2e6e5ff700 1 -- start start 2024-01-11T08:25:56.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.407+0000 7f2e6e5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e680b2730 0x7f2e680aa320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:56.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.407+0000 7f2e6e5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e68159650 0x7f2e6815f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:56.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.407+0000 7f2e6e5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e68159a20 0x7f2e68150280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:56.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.407+0000 7f2e6e5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e680aded0 con 0x7f2e68159a20 2024-01-11T08:25:56.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.407+0000 7f2e6e5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e680ad2c0 con 0x7f2e680b2730 2024-01-11T08:25:56.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.407+0000 7f2e6e5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e680ad5a0 con 0x7f2e68159650 2024-01-11T08:25:56.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.407+0000 7f2e6cb9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e68159a20 0x7f2e68150280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:56.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.407+0000 7f2e6cb9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e68159a20 0x7f2e68150280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35386/0 (socket says 172.21.15.27:35386) 2024-01-11T08:25:56.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.407+0000 7f2e6cb9c700 1 -- 172.21.15.27:0/3656297116 learned_addr learned my addr 172.21.15.27:0/3656297116 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:56.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.408+0000 7f2e677fe700 1 --2- 172.21.15.27:0/3656297116 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e68159650 0x7f2e6815f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:56.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.408+0000 7f2e67fff700 1 --2- 172.21.15.27:0/3656297116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e680b2730 0x7f2e680aa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:56.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.408+0000 7f2e6cb9c700 1 -- 172.21.15.27:0/3656297116 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e68159650 msgr2=0x7f2e6815f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:56.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.408+0000 7f2e6cb9c700 1 --2- 172.21.15.27:0/3656297116 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e68159650 0x7f2e6815f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.408+0000 7f2e6cb9c700 1 -- 172.21.15.27:0/3656297116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e680b2730 msgr2=0x7f2e680aa320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:56.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.408+0000 7f2e6cb9c700 1 --2- 172.21.15.27:0/3656297116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e680b2730 0x7f2e680aa320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.408+0000 7f2e6cb9c700 1 -- 172.21.15.27:0/3656297116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e60009ae0 con 0x7f2e68159a20 2024-01-11T08:25:56.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.408+0000 7f2e677fe700 1 --2- 172.21.15.27:0/3656297116 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e68159650 0x7f2e6815f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:25:56.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.408+0000 7f2e67fff700 1 --2- 172.21.15.27:0/3656297116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e680b2730 0x7f2e680aa320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:25:56.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.408+0000 7f2e6cb9c700 1 --2- 172.21.15.27:0/3656297116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e68159a20 0x7f2e68150280 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f2e68063d40 tx=0x7f2e60018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:56.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.408+0000 7f2e4ffff700 1 -- 172.21.15.27:0/3656297116 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e600032f0 con 0x7f2e68159a20 2024-01-11T08:25:56.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.408+0000 7f2e4ffff700 1 -- 172.21.15.27:0/3656297116 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e60026460 con 0x7f2e68159a20 2024-01-11T08:25:56.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.408+0000 7f2e4ffff700 1 -- 172.21.15.27:0/3656297116 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e60026800 con 0x7f2e68159a20 2024-01-11T08:25:56.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.408+0000 7f2e6e5ff700 1 -- 172.21.15.27:0/3656297116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e680ad880 con 0x7f2e68159a20 2024-01-11T08:25:56.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.408+0000 7f2e6e5ff700 1 -- 172.21.15.27:0/3656297116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e680aa860 con 0x7f2e68159a20 2024-01-11T08:25:56.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.410+0000 7f2e6e5ff700 1 -- 172.21.15.27:0/3656297116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e680718d0 con 0x7f2e68159a20 2024-01-11T08:25:56.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.410+0000 7f2e4ffff700 1 -- 172.21.15.27:0/3656297116 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2e60003040 con 0x7f2e68159a20 2024-01-11T08:25:56.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.412+0000 7f2e4ffff700 1 --2- 172.21.15.27:0/3656297116 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2e54066f30 0x7f2e540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:56.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.412+0000 7f2e4ffff700 1 -- 172.21.15.27:0/3656297116 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4881+0+0 (secure 0 0 0) 0x7f2e60086340 con 0x7f2e68159a20 2024-01-11T08:25:56.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.412+0000 7f2e67fff700 1 --2- 172.21.15.27:0/3656297116 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2e54066f30 0x7f2e540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:56.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.413+0000 7f2e67fff700 1 --2- 172.21.15.27:0/3656297116 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2e54066f30 0x7f2e540693e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2e58006f00 tx=0x7f2e58008040 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:56.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.416+0000 7f2e4ffff700 1 -- 172.21.15.27:0/3656297116 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2e600538e0 con 0x7f2e68159a20 2024-01-11T08:25:56.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.703+0000 7f2e6e5ff700 1 -- 172.21.15.27:0/3656297116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f2e6806c920 con 0x7f2e68159a20 2024-01-11T08:25:56.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.703+0000 7f2e4ffff700 1 -- 172.21.15.27:0/3656297116 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f2e60056f00 con 0x7f2e68159a20 2024-01-11T08:25:56.702 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:56.702 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T08:25:56.703 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-11T08:25:56.703 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:56.703 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-11T08:25:56.703 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lQm3yLhAAv6erz5hMfat8EUzRA5luMA==", 2024-01-11T08:25:56.703 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:56.703 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:25:56.703 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:25:56.703 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:25:56.703 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:56.703 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:56.703 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:56.703 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-11T08:25:56.703 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9l+FG8ChAACHndtVLx7qS3sidPniSycA==", 2024-01-11T08:25:56.703 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:56.704 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:25:56.704 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:25:56.704 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:25:56.704 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:56.704 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:56.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:56.704 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-11T08:25:56.704 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l1NnEOhAALHXsc5es7hufCthGdRzkyw==", 2024-01-11T08:25:56.704 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:56.704 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:25:56.704 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:25:56.704 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:25:56.704 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:56.705 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:56.705 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:56.705 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-11T08:25:56.705 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lruoWFhAAm8AmrZtTJhYI1+19F79RPQ==", 2024-01-11T08:25:56.705 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:56.705 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:25:56.705 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:25:56.705 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:25:56.705 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:56.705 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:56.705 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:56.705 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-11T08:25:56.705 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA2op9ltZygORAAZr1X1nCN0ce8rIVaGDpFyQ==", 2024-01-11T08:25:56.706 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:56.706 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:25:56.706 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:25:56.706 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:56.706 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:56.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:56.706 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-11T08:25:56.706 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA6op9lqSRPAhAAmtyCsyK5ZVGX2wfhOCMuFQ==", 2024-01-11T08:25:56.706 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:56.706 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:25:56.706 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:25:56.706 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:56.706 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:56.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:56.707 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-11T08:25:56.707 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA9op9lZcs8BhAA8j+U+jPOoy+fliVWZAPFYg==", 2024-01-11T08:25:56.707 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:56.707 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:25:56.707 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:25:56.707 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:56.707 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:56.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:56.707 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-11T08:25:56.707 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBAop9lFuCUChAAP6si/2W1mxH4ecQJvuumkw==", 2024-01-11T08:25:56.707 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:56.707 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:25:56.707 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:25:56.708 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:56.708 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:56.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:56.708 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-11T08:25:56.708 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBGop9lOpwBIRAAcpcsjwmX2IldcKHJWAgHuA==", 2024-01-11T08:25:56.708 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:56.708 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:25:56.708 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:25:56.708 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:56.708 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:56.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:56.708 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-11T08:25:56.709 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBJop9l4q8hJRAAK8mAYVRQMoWS6wBOU69HmA==", 2024-01-11T08:25:56.709 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:56.709 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:25:56.709 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:25:56.709 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:56.709 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:56.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:56.709 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-11T08:25:56.709 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBMop9lqMF9KBAAFgFFI2GWt9AGif4AoScThw==", 2024-01-11T08:25:56.709 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:56.709 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:25:56.709 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:25:56.709 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:56.709 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:56.710 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:56.710 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-11T08:25:56.710 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBPop9lN7eDLBAAKswPZFfs0g2AaSe6V+pbbQ==", 2024-01-11T08:25:56.710 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:56.710 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:25:56.710 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:25:56.710 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:56.710 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:56.710 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:56.710 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-11T08:25:56.710 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lT5okIBAABy8PbJJbA062VNmb1Y8qhg==", 2024-01-11T08:25:56.710 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:56.710 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:25:56.710 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-01-11T08:25:56.711 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-11T08:25:56.711 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:56.711 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:56.711 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-11T08:25:56.711 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lDf37LhAAqV+EabKXJEW1sS5xtu1tXw==", 2024-01-11T08:25:56.711 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:56.711 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:25:56.711 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:25:56.711 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:25:56.711 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:25:56.711 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:56.711 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:56.711 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-11T08:25:56.712 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lWayxJxAAcalM8QSOGtbCXxjYZGaNQA==", 2024-01-11T08:25:56.712 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:56.712 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:25:56.712 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:25:56.712 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:25:56.712 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:25:56.712 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:56.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:56.712 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-11T08:25:56.712 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lHD4tNhAAdmnFBmMrK+QhQYdpbzW0mg==", 2024-01-11T08:25:56.712 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:56.712 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:25:56.712 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:25:56.712 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:25:56.713 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:25:56.713 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:56.713 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:56.713 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-11T08:25:56.713 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAwop9lGPtDMxAAJ8FjrSEdCtQL8AhtAyqnTw==", 2024-01-11T08:25:56.713 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:56.713 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:25:56.713 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:25:56.713 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:25:56.713 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:56.713 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:56.713 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:56.713 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-11T08:25:56.714 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lXBZLJxAAXa1S8Vkm3MiV4igtNbg/1Q==", 2024-01-11T08:25:56.714 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:56.714 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:25:56.714 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T08:25:56.714 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:56.714 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:25:56.714 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:56.714 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-11T08:25:56.714 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l3ZGoHxAANqzytTTvWYSomG+RFsFDyg==", 2024-01-11T08:25:56.714 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:25:56.714 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:25:56.714 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T08:25:56.714 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:25:56.714 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.715 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:56.715 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-11T08:25:56.715 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T08:25:56.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.706+0000 7f2e6e5ff700 1 -- 172.21.15.27:0/3656297116 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2e54066f30 msgr2=0x7f2e540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:56.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.706+0000 7f2e6e5ff700 1 --2- 172.21.15.27:0/3656297116 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2e54066f30 0x7f2e540693e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2e58006f00 tx=0x7f2e58008040 comp rx=0 tx=0).stop 2024-01-11T08:25:56.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.706+0000 7f2e6e5ff700 1 -- 172.21.15.27:0/3656297116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e68159a20 msgr2=0x7f2e68150280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:56.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.706+0000 7f2e6e5ff700 1 --2- 172.21.15.27:0/3656297116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e68159a20 0x7f2e68150280 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f2e68063d40 tx=0x7f2e60018000 comp rx=0 tx=0).stop 2024-01-11T08:25:56.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.706+0000 7f2e6e5ff700 1 -- 172.21.15.27:0/3656297116 shutdown_connections 2024-01-11T08:25:56.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.706+0000 7f2e6e5ff700 1 --2- 172.21.15.27:0/3656297116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e68159a20 0x7f2e68150280 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.706+0000 7f2e6e5ff700 1 --2- 172.21.15.27:0/3656297116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e680b2730 0x7f2e680aa320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.706+0000 7f2e6e5ff700 1 --2- 172.21.15.27:0/3656297116 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2e54066f30 0x7f2e540693e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.706+0000 7f2e6e5ff700 1 --2- 172.21.15.27:0/3656297116 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e68159650 0x7f2e6815f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.706+0000 7f2e6e5ff700 1 -- 172.21.15.27:0/3656297116 >> 172.21.15.27:0/3656297116 conn(0x7f2e6806b200 msgr2=0x7f2e68158830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:56.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.706+0000 7f2e6e5ff700 1 -- 172.21.15.27:0/3656297116 shutdown_connections 2024-01-11T08:25:56.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.706+0000 7f2e6e5ff700 1 -- 172.21.15.27:0/3656297116 wait complete. 2024-01-11T08:25:56.723 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:25:56.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.922+0000 7f812d3ff700 1 Processor -- start 2024-01-11T08:25:56.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.922+0000 7f812d3ff700 1 -- start start 2024-01-11T08:25:56.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.923+0000 7f812d3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f81280c8620 0x7f81280af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:56.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.923+0000 7f812d3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8128069d40 0x7f81280b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:56.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.923+0000 7f812d3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f81280c89f0 0x7f81280b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:56.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.923+0000 7f812d3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8128157390 con 0x7f81280c8620 2024-01-11T08:25:56.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.923+0000 7f812d3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81280afeb0 con 0x7f81280c89f0 2024-01-11T08:25:56.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.923+0000 7f812d3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812806a200 con 0x7f8128069d40 2024-01-11T08:25:56.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.924+0000 7f8126ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f81280c8620 0x7f81280af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:56.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.924+0000 7f8126ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f81280c8620 0x7f81280af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35402/0 (socket says 172.21.15.27:35402) 2024-01-11T08:25:56.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.924+0000 7f8126ffd700 1 -- 172.21.15.27:0/2706840006 learned_addr learned my addr 172.21.15.27:0/2706840006 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:56.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.924+0000 7f81267fc700 1 --2- 172.21.15.27:0/2706840006 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8128069d40 0x7f81280b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:56.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.924+0000 7f81277fe700 1 --2- 172.21.15.27:0/2706840006 >> v2:172.21.15.77:3300/0 conn(0x7f81280c89f0 0x7f81280b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:56.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.924+0000 7f8126ffd700 1 -- 172.21.15.27:0/2706840006 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8128069d40 msgr2=0x7f81280b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:56.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.924+0000 7f8126ffd700 1 --2- 172.21.15.27:0/2706840006 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8128069d40 0x7f81280b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.924+0000 7f8126ffd700 1 -- 172.21.15.27:0/2706840006 >> v2:172.21.15.77:3300/0 conn(0x7f81280c89f0 msgr2=0x7f81280b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:56.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.924+0000 7f8126ffd700 1 --2- 172.21.15.27:0/2706840006 >> v2:172.21.15.77:3300/0 conn(0x7f81280c89f0 0x7f81280b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.924+0000 7f8126ffd700 1 -- 172.21.15.27:0/2706840006 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8128156d40 con 0x7f81280c8620 2024-01-11T08:25:56.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.924+0000 7f81277fe700 1 --2- 172.21.15.27:0/2706840006 >> v2:172.21.15.77:3300/0 conn(0x7f81280c89f0 0x7f81280b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:56.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.925+0000 7f8126ffd700 1 --2- 172.21.15.27:0/2706840006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81280c8620 0x7f81280af970 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f81281545c0 tx=0x7f8118009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f88666289f6dc8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:56.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.925+0000 7f8125ffb700 1 -- 172.21.15.27:0/2706840006 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8118009e70 con 0x7f81280c8620 2024-01-11T08:25:56.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.925+0000 7f8125ffb700 1 -- 172.21.15.27:0/2706840006 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811800b860 con 0x7f81280c8620 2024-01-11T08:25:56.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.925+0000 7f8125ffb700 1 -- 172.21.15.27:0/2706840006 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811800bc00 con 0x7f81280c8620 2024-01-11T08:25:56.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.925+0000 7f812d3ff700 1 -- 172.21.15.27:0/2706840006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81280c8620 msgr2=0x7f81280af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:56.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.925+0000 7f812d3ff700 1 --2- 172.21.15.27:0/2706840006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81280c8620 0x7f81280af970 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f81281545c0 tx=0x7f8118009250 comp rx=0 tx=0).stop 2024-01-11T08:25:56.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.926+0000 7f812d3ff700 1 -- 172.21.15.27:0/2706840006 shutdown_connections 2024-01-11T08:25:56.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.926+0000 7f812d3ff700 1 --2- 172.21.15.27:0/2706840006 >> v2:172.21.15.77:3300/0 conn(0x7f81280c89f0 0x7f81280b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.926+0000 7f812d3ff700 1 --2- 172.21.15.27:0/2706840006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81280c8620 0x7f81280af970 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.926+0000 7f812d3ff700 1 --2- 172.21.15.27:0/2706840006 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8128069d40 0x7f81280b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.926+0000 7f812d3ff700 1 -- 172.21.15.27:0/2706840006 >> 172.21.15.27:0/2706840006 conn(0x7f81280680e0 msgr2=0x7f8128069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:56.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.926+0000 7f812d3ff700 1 -- 172.21.15.27:0/2706840006 shutdown_connections 2024-01-11T08:25:56.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.926+0000 7f812d3ff700 1 -- 172.21.15.27:0/2706840006 wait complete. 2024-01-11T08:25:56.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.927+0000 7f812d3ff700 1 Processor -- start 2024-01-11T08:25:56.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.927+0000 7f812d3ff700 1 -- start start 2024-01-11T08:25:56.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.928+0000 7f812d3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8128069d40 0x7f8128147d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.928+0000 7f812d3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f81280c8620 0x7f81280c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.928+0000 7f812d3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81280c89f0 0x7f81280aa870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.928+0000 7f812d3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8128148280 con 0x7f81280c89f0 2024-01-11T08:25:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.928+0000 7f812d3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81280cce80 con 0x7f8128069d40 2024-01-11T08:25:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.928+0000 7f812d3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81280cc270 con 0x7f81280c8620 2024-01-11T08:25:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.928+0000 7f81277fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81280c89f0 0x7f81280aa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.929+0000 7f81277fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81280c89f0 0x7f81280aa870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35406/0 (socket says 172.21.15.27:35406) 2024-01-11T08:25:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.929+0000 7f81277fe700 1 -- 172.21.15.27:0/1038466110 learned_addr learned my addr 172.21.15.27:0/1038466110 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.929+0000 7f81267fc700 1 --2- 172.21.15.27:0/1038466110 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f81280c8620 0x7f81280c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.929+0000 7f8126ffd700 1 --2- 172.21.15.27:0/1038466110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8128069d40 0x7f8128147d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.929+0000 7f81277fe700 1 -- 172.21.15.27:0/1038466110 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f81280c8620 msgr2=0x7f81280c99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.929+0000 7f81277fe700 1 --2- 172.21.15.27:0/1038466110 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f81280c8620 0x7f81280c99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.929+0000 7f81277fe700 1 -- 172.21.15.27:0/1038466110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8128069d40 msgr2=0x7f8128147d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.929+0000 7f81277fe700 1 --2- 172.21.15.27:0/1038466110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8128069d40 0x7f8128147d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.929+0000 7f81277fe700 1 -- 172.21.15.27:0/1038466110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81180094f0 con 0x7f81280c89f0 2024-01-11T08:25:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.929+0000 7f81277fe700 1 --2- 172.21.15.27:0/1038466110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81280c89f0 0x7f81280aa870 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f812000d150 tx=0x7f8120009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.929+0000 7f810effd700 1 -- 172.21.15.27:0/1038466110 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812000dc90 con 0x7f81280c89f0 2024-01-11T08:25:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.929+0000 7f810effd700 1 -- 172.21.15.27:0/1038466110 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812000f040 con 0x7f81280c89f0 2024-01-11T08:25:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.929+0000 7f810effd700 1 -- 172.21.15.27:0/1038466110 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8120015690 con 0x7f81280c89f0 2024-01-11T08:25:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.930+0000 7f812d3ff700 1 -- 172.21.15.27:0/1038466110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81280cc550 con 0x7f81280c89f0 2024-01-11T08:25:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.930+0000 7f812d3ff700 1 -- 172.21.15.27:0/1038466110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81280ccb10 con 0x7f81280c89f0 2024-01-11T08:25:56.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.931+0000 7f812d3ff700 1 -- 172.21.15.27:0/1038466110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81280718d0 con 0x7f81280c89f0 2024-01-11T08:25:56.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.932+0000 7f810effd700 1 -- 172.21.15.27:0/1038466110 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f812001e3b0 con 0x7f81280c89f0 2024-01-11T08:25:56.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.933+0000 7f810effd700 1 --2- 172.21.15.27:0/1038466110 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f8114066f30 0x7f81140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:56.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.933+0000 7f810effd700 1 -- 172.21.15.27:0/1038466110 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4881+0+0 (secure 0 0 0) 0x7f8120084660 con 0x7f81280c89f0 2024-01-11T08:25:56.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.933+0000 7f8126ffd700 1 --2- 172.21.15.27:0/1038466110 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f8114066f30 0x7f81140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.934+0000 7f8126ffd700 1 --2- 172.21.15.27:0/1038466110 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f8114066f30 0x7f81140693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f81281545c0 tx=0x7f8118017000 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:56.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:56.937+0000 7f810effd700 1 -- 172.21.15.27:0/1038466110 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8120051c00 con 0x7f81280c89f0 2024-01-11T08:25:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.141+0000 7f812d3ff700 1 -- 172.21.15.27:0/1038466110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8128159e70 con 0x7f81280c89f0 2024-01-11T08:25:57.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.142+0000 7f810effd700 1 -- 172.21.15.27:0/1038466110 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+12129 (secure 0 0 0) 0x7f8120055220 con 0x7f81280c89f0 2024-01-11T08:25:57.141 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:57.141 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":84,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:25:56.190709+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:25:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.145+0000 7f812d3ff700 1 -- 172.21.15.27:0/1038466110 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f8114066f30 msgr2=0x7f81140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.145+0000 7f812d3ff700 1 --2- 172.21.15.27:0/1038466110 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f8114066f30 0x7f81140693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f81281545c0 tx=0x7f8118017000 comp rx=0 tx=0).stop 2024-01-11T08:25:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.145+0000 7f812d3ff700 1 -- 172.21.15.27:0/1038466110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81280c89f0 msgr2=0x7f81280aa870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.145+0000 7f812d3ff700 1 --2- 172.21.15.27:0/1038466110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81280c89f0 0x7f81280aa870 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f812000d150 tx=0x7f8120009a50 comp rx=0 tx=0).stop 2024-01-11T08:25:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.145+0000 7f812d3ff700 1 -- 172.21.15.27:0/1038466110 shutdown_connections 2024-01-11T08:25:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.145+0000 7f812d3ff700 1 --2- 172.21.15.27:0/1038466110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81280c89f0 0x7f81280aa870 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.145+0000 7f812d3ff700 1 --2- 172.21.15.27:0/1038466110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8128069d40 0x7f8128147d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.145+0000 7f812d3ff700 1 --2- 172.21.15.27:0/1038466110 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f8114066f30 0x7f81140693e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.145+0000 7f812d3ff700 1 --2- 172.21.15.27:0/1038466110 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f81280c8620 0x7f81280c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.145+0000 7f812d3ff700 1 -- 172.21.15.27:0/1038466110 >> 172.21.15.27:0/1038466110 conn(0x7f81280680e0 msgr2=0x7f8128156410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:57.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.145+0000 7f812d3ff700 1 -- 172.21.15.27:0/1038466110 shutdown_connections 2024-01-11T08:25:57.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.145+0000 7f812d3ff700 1 -- 172.21.15.27:0/1038466110 wait complete. 2024-01-11T08:25:57.162 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-11T08:25:57.162 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:25:57.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.368+0000 7fed387ff700 1 Processor -- start 2024-01-11T08:25:57.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.368+0000 7fed387ff700 1 -- start start 2024-01-11T08:25:57.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.370+0000 7fed387ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fed3015a1b0 0x7fed3015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:57.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.370+0000 7fed387ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fed3015ae70 0x7fed3015b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:57.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.370+0000 7fed387ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed300cf850 con 0x7fed300ccd70 2024-01-11T08:25:57.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.370+0000 7fed387ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed300cec40 con 0x7fed3015a1b0 2024-01-11T08:25:57.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.370+0000 7fed387ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed300cef20 con 0x7fed3015ae70 2024-01-11T08:25:57.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.370+0000 7fed35d9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fed300ccd70 0x7fed300cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45748/0 (socket says 172.21.15.27:45748) 2024-01-11T08:25:57.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.370+0000 7fed35d9a700 1 -- 172.21.15.27:0/919142461 learned_addr learned my addr 172.21.15.27:0/919142461 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:57.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.370+0000 7fed36d9c700 1 --2- 172.21.15.27:0/919142461 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fed3015ae70 0x7fed3015b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:57.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.370+0000 7fed3659b700 1 --2- 172.21.15.27:0/919142461 >> v2:172.21.15.77:3300/0 conn(0x7fed3015a1b0 0x7fed3015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.370+0000 7fed35599700 1 -- 172.21.15.27:0/919142461 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983126978 0 0) 0x7fed300cf850 con 0x7fed300ccd70 2024-01-11T08:25:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.370+0000 7fed35599700 1 -- 172.21.15.27:0/919142461 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed1c003570 con 0x7fed300ccd70 2024-01-11T08:25:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed35599700 1 -- 172.21.15.27:0/919142461 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 534622604 0 0) 0x7fed1c003570 con 0x7fed300ccd70 2024-01-11T08:25:57.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed35599700 1 -- 172.21.15.27:0/919142461 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fed3015ae70 msgr2=0x7fed3015b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:57.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed35599700 1 --2- 172.21.15.27:0/919142461 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fed3015ae70 0x7fed3015b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed35599700 1 -- 172.21.15.27:0/919142461 >> v2:172.21.15.77:3300/0 conn(0x7fed3015a1b0 msgr2=0x7fed3015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:57.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed35599700 1 --2- 172.21.15.27:0/919142461 >> v2:172.21.15.77:3300/0 conn(0x7fed3015a1b0 0x7fed3015a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed35599700 1 -- 172.21.15.27:0/919142461 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed300cf200 con 0x7fed300ccd70 2024-01-11T08:25:57.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed35599700 1 -- 172.21.15.27:0/919142461 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fed20001470 con 0x7fed300ccd70 2024-01-11T08:25:57.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed35599700 1 -- 172.21.15.27:0/919142461 >> v1:172.21.15.27:6789/0 conn(0x7fed300ccd70 legacy=0x7fed300cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:57.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed35599700 1 --2- 172.21.15.27:0/919142461 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fed3015ae70 0x7fed1c003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:57.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed35599700 1 --2- 172.21.15.27:0/919142461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed1c0040b0 0x7fed1c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed35599700 1 --2- 172.21.15.27:0/919142461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed1c004d50 0x7fed1c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed35599700 1 -- 172.21.15.27:0/919142461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed300cec40 con 0x7fed1c0040b0 2024-01-11T08:25:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed35599700 1 -- 172.21.15.27:0/919142461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed300cef20 con 0x7fed1c004d50 2024-01-11T08:25:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed35599700 1 -- 172.21.15.27:0/919142461 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c00bb60 con 0x7fed3015ae70 2024-01-11T08:25:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed35d9a700 1 --2- 172.21.15.27:0/919142461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed1c0040b0 0x7fed1c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed36d9c700 1 --2- 172.21.15.27:0/919142461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed1c004d50 0x7fed1c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed3659b700 1 --2- 172.21.15.27:0/919142461 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fed3015ae70 0x7fed1c003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed35d9a700 1 -- 172.21.15.27:0/919142461 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fed3015ae70 msgr2=0x7fed1c003ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed35d9a700 1 --2- 172.21.15.27:0/919142461 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fed3015ae70 0x7fed1c003ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed35d9a700 1 -- 172.21.15.27:0/919142461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed1c004d50 msgr2=0x7fed1c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.371+0000 7fed35d9a700 1 --2- 172.21.15.27:0/919142461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed1c004d50 0x7fed1c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.372+0000 7fed35d9a700 1 -- 172.21.15.27:0/919142461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1c003570 con 0x7fed1c0040b0 2024-01-11T08:25:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.372+0000 7fed36d9c700 1 --2- 172.21.15.27:0/919142461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed1c004d50 0x7fed1c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:25:57.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.372+0000 7fed35d9a700 1 --2- 172.21.15.27:0/919142461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed1c0040b0 0x7fed1c004810 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fed2000c2a0 tx=0x7fed20001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cffb21e1a102c40 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:57.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.372+0000 7fed35599700 1 -- 172.21.15.27:0/919142461 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2000cb70 con 0x7fed1c0040b0 2024-01-11T08:25:57.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.372+0000 7fed35599700 1 -- 172.21.15.27:0/919142461 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2000ccd0 con 0x7fed1c0040b0 2024-01-11T08:25:57.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.372+0000 7fed35599700 1 -- 172.21.15.27:0/919142461 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed20005b70 con 0x7fed1c0040b0 2024-01-11T08:25:57.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.372+0000 7fed387ff700 1 -- 172.21.15.27:0/919142461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed1c0040b0 msgr2=0x7fed1c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:57.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.372+0000 7fed387ff700 1 --2- 172.21.15.27:0/919142461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed1c0040b0 0x7fed1c004810 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fed2000c2a0 tx=0x7fed20001dc0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.372+0000 7fed36d9c700 1 -- 172.21.15.27:0/919142461 reap_dead start 2024-01-11T08:25:57.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.373+0000 7fed387ff700 1 -- 172.21.15.27:0/919142461 shutdown_connections 2024-01-11T08:25:57.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.373+0000 7fed387ff700 1 -- 172.21.15.27:0/919142461 >> 172.21.15.27:0/919142461 conn(0x7fed3006ed40 msgr2=0x7fed3006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:57.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.373+0000 7fed387ff700 1 -- 172.21.15.27:0/919142461 shutdown_connections 2024-01-11T08:25:57.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.373+0000 7fed387ff700 1 -- 172.21.15.27:0/919142461 wait complete. 2024-01-11T08:25:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.375+0000 7fed387ff700 1 Processor -- start 2024-01-11T08:25:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.375+0000 7fed387ff700 1 -- start start 2024-01-11T08:25:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.376+0000 7fed387ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fed1c0040b0 0x7fed3014b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:57.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.376+0000 7fed387ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed30169710 0x7fed30169b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:57.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.376+0000 7fed387ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed300c9fe0 0x7fed300ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:57.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.376+0000 7fed387ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed30155a10 con 0x7fed300c9fe0 2024-01-11T08:25:57.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.376+0000 7fed387ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed30157b10 con 0x7fed30169710 2024-01-11T08:25:57.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.376+0000 7fed387ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed30158970 con 0x7fed1c0040b0 2024-01-11T08:25:57.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.377+0000 7fed36d9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed300c9fe0 0x7fed300ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:57.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.377+0000 7fed3659b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fed1c0040b0 0x7fed3014b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:57.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.377+0000 7fed3659b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fed1c0040b0 0x7fed3014b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:58324/0 (socket says 172.21.15.27:58324) 2024-01-11T08:25:57.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.377+0000 7fed36d9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed300c9fe0 0x7fed300ca450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35418/0 (socket says 172.21.15.27:35418) 2024-01-11T08:25:57.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.377+0000 7fed3659b700 1 -- 172.21.15.27:0/3885266936 learned_addr learned my addr 172.21.15.27:0/3885266936 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:57.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.377+0000 7fed35d9a700 1 --2- 172.21.15.27:0/3885266936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed30169710 0x7fed30169b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:57.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.377+0000 7fed3659b700 1 -- 172.21.15.27:0/3885266936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed30169710 msgr2=0x7fed30169b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:57.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.377+0000 7fed3659b700 1 --2- 172.21.15.27:0/3885266936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed30169710 0x7fed30169b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.377+0000 7fed3659b700 1 -- 172.21.15.27:0/3885266936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed300c9fe0 msgr2=0x7fed300ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.377+0000 7fed3659b700 1 --2- 172.21.15.27:0/3885266936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed300c9fe0 0x7fed300ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.377+0000 7fed3659b700 1 -- 172.21.15.27:0/3885266936 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3014b5f0 con 0x7fed1c0040b0 2024-01-11T08:25:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.377+0000 7fed36d9c700 1 --2- 172.21.15.27:0/3885266936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed300c9fe0 0x7fed300ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:25:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.377+0000 7fed3659b700 1 --2- 172.21.15.27:0/3885266936 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fed1c0040b0 0x7fed3014b0b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fed2c011d40 tx=0x7fed2c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.378+0000 7fed267fc700 1 -- 172.21.15.27:0/3885266936 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2c00fa10 con 0x7fed1c0040b0 2024-01-11T08:25:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.378+0000 7fed267fc700 1 -- 172.21.15.27:0/3885266936 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2c00fb70 con 0x7fed1c0040b0 2024-01-11T08:25:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.378+0000 7fed267fc700 1 -- 172.21.15.27:0/3885266936 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2c014440 con 0x7fed1c0040b0 2024-01-11T08:25:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.378+0000 7fed387ff700 1 -- 172.21.15.27:0/3885266936 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed30157d60 con 0x7fed1c0040b0 2024-01-11T08:25:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.378+0000 7fed387ff700 1 -- 172.21.15.27:0/3885266936 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed30158320 con 0x7fed1c0040b0 2024-01-11T08:25:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.379+0000 7fed387ff700 1 -- 172.21.15.27:0/3885266936 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed3004c3c0 con 0x7fed1c0040b0 2024-01-11T08:25:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.380+0000 7fed267fc700 1 -- 172.21.15.27:0/3885266936 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fed2c014870 con 0x7fed1c0040b0 2024-01-11T08:25:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.381+0000 7fed267fc700 1 --2- 172.21.15.27:0/3885266936 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fed14066f30 0x7fed140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.381+0000 7fed267fc700 1 -- 172.21.15.27:0/3885266936 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4881+0+0 (secure 0 0 0) 0x7fed2c018030 con 0x7fed1c0040b0 2024-01-11T08:25:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.381+0000 7fed35d9a700 1 --2- 172.21.15.27:0/3885266936 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fed14066f30 0x7fed140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.382+0000 7fed35d9a700 1 --2- 172.21.15.27:0/3885266936 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fed14066f30 0x7fed140693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fed20015040 tx=0x7fed20012000 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.385+0000 7fed267fc700 1 -- 172.21.15.27:0/3885266936 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fed2c0524a0 con 0x7fed1c0040b0 2024-01-11T08:25:57.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.585+0000 7fed387ff700 1 -- 172.21.15.27:0/3885266936 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fed3015c3b0 con 0x7fed1c0040b0 2024-01-11T08:25:57.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.586+0000 7fed267fc700 1 -- 172.21.15.27:0/3885266936 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7fed2c0220a0 con 0x7fed1c0040b0 2024-01-11T08:25:57.585 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:25:57.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.588+0000 7fed387ff700 1 -- 172.21.15.27:0/3885266936 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fed14066f30 msgr2=0x7fed140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:57.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.588+0000 7fed387ff700 1 --2- 172.21.15.27:0/3885266936 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fed14066f30 0x7fed140693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fed20015040 tx=0x7fed20012000 comp rx=0 tx=0).stop 2024-01-11T08:25:57.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.588+0000 7fed387ff700 1 -- 172.21.15.27:0/3885266936 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fed1c0040b0 msgr2=0x7fed3014b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:57.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.588+0000 7fed387ff700 1 --2- 172.21.15.27:0/3885266936 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fed1c0040b0 0x7fed3014b0b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fed2c011d40 tx=0x7fed2c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.588+0000 7fed387ff700 1 -- 172.21.15.27:0/3885266936 shutdown_connections 2024-01-11T08:25:57.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.588+0000 7fed387ff700 1 --2- 172.21.15.27:0/3885266936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed300c9fe0 0x7fed300ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.588+0000 7fed387ff700 1 --2- 172.21.15.27:0/3885266936 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fed14066f30 0x7fed140693e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.588+0000 7fed387ff700 1 --2- 172.21.15.27:0/3885266936 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fed1c0040b0 0x7fed3014b0b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.588+0000 7fed387ff700 1 --2- 172.21.15.27:0/3885266936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed30169710 0x7fed30169b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.588+0000 7fed387ff700 1 -- 172.21.15.27:0/3885266936 >> 172.21.15.27:0/3885266936 conn(0x7fed3006ed40 msgr2=0x7fed300c5ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:57.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.588+0000 7fed387ff700 1 -- 172.21.15.27:0/3885266936 shutdown_connections 2024-01-11T08:25:57.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.589+0000 7fed387ff700 1 -- 172.21.15.27:0/3885266936 wait complete. 2024-01-11T08:25:57.605 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-11T08:25:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.805+0000 7f33001ff700 1 Processor -- start 2024-01-11T08:25:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.805+0000 7f33001ff700 1 -- start start 2024-01-11T08:25:57.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.806+0000 7f33001ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f32f80cc410 0x7f32f8159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:57.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.806+0000 7f33001ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f80cd730 con 0x7f32f80b03e0 2024-01-11T08:25:57.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.806+0000 7f33001ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f80ccb20 con 0x7f32f80b2b90 2024-01-11T08:25:57.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.806+0000 7f33001ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32f80cce00 con 0x7f32f80cc410 2024-01-11T08:25:57.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.807+0000 7f32fdf9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f32f80b03e0 0x7f32f80b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45762/0 (socket says 172.21.15.27:45762) 2024-01-11T08:25:57.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.807+0000 7f32fdf9b700 1 -- 172.21.15.27:0/3047823869 learned_addr learned my addr 172.21.15.27:0/3047823869 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.807+0000 7f32fe79c700 1 --2- 172.21.15.27:0/3047823869 >> v2:172.21.15.77:3300/0 conn(0x7f32f80cc410 0x7f32f8159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.807+0000 7f32fcf99700 1 -- 172.21.15.27:0/3047823869 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3517334986 0 0) 0x7f32f80cd730 con 0x7f32f80b03e0 2024-01-11T08:25:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.807+0000 7f32fcf99700 1 -- 172.21.15.27:0/3047823869 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e4003570 con 0x7f32f80b03e0 2024-01-11T08:25:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.807+0000 7f32fcf99700 1 -- 172.21.15.27:0/3047823869 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4040881091 0 0) 0x7f32e4003570 con 0x7f32f80b03e0 2024-01-11T08:25:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.807+0000 7f32fcf99700 1 -- 172.21.15.27:0/3047823869 >> v2:172.21.15.77:3300/0 conn(0x7f32f80cc410 msgr2=0x7f32f8159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.807+0000 7f32fcf99700 1 --2- 172.21.15.27:0/3047823869 >> v2:172.21.15.77:3300/0 conn(0x7f32f80cc410 0x7f32f8159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.807+0000 7f32fcf99700 1 -- 172.21.15.27:0/3047823869 >> v1:172.21.15.77:6789/0 conn(0x7f32f80b2b90 legacy=0x7f32f8159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.807+0000 7f32fcf99700 1 -- 172.21.15.27:0/3047823869 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f80cd0e0 con 0x7f32f80b03e0 2024-01-11T08:25:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.807+0000 7f32fcf99700 1 -- 172.21.15.27:0/3047823869 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f32f4001470 con 0x7f32f80b03e0 2024-01-11T08:25:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.808+0000 7f32fcf99700 1 -- 172.21.15.27:0/3047823869 >> v1:172.21.15.27:6789/0 conn(0x7f32f80b03e0 legacy=0x7f32f80b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.808+0000 7f32fcf99700 1 --2- 172.21.15.27:0/3047823869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e4003ae0 0x7f32e4003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.808+0000 7f32fcf99700 1 --2- 172.21.15.27:0/3047823869 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32e4004810 0x7f32e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.808+0000 7f32fcf99700 1 --2- 172.21.15.27:0/3047823869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e4005230 0x7f32e400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.808+0000 7f32fcf99700 1 -- 172.21.15.27:0/3047823869 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80cce00 con 0x7f32e4003ae0 2024-01-11T08:25:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.808+0000 7f32fcf99700 1 -- 172.21.15.27:0/3047823869 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e400c160 con 0x7f32e4005230 2024-01-11T08:25:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.808+0000 7f32fcf99700 1 -- 172.21.15.27:0/3047823869 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e400c320 con 0x7f32e4004810 2024-01-11T08:25:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.808+0000 7f32fdf9b700 1 --2- 172.21.15.27:0/3047823869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e4003ae0 0x7f32e4003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.808+0000 7f32fe79c700 1 --2- 172.21.15.27:0/3047823869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e4005230 0x7f32e400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.808+0000 7f32fd79a700 1 --2- 172.21.15.27:0/3047823869 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32e4004810 0x7f32e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.808+0000 7f32fdf9b700 1 -- 172.21.15.27:0/3047823869 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32e4004810 msgr2=0x7f32e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.808+0000 7f32fdf9b700 1 --2- 172.21.15.27:0/3047823869 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32e4004810 0x7f32e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.808+0000 7f32fdf9b700 1 -- 172.21.15.27:0/3047823869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e4005230 msgr2=0x7f32e400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.808+0000 7f32fdf9b700 1 --2- 172.21.15.27:0/3047823869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e4005230 0x7f32e400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.808+0000 7f32fdf9b700 1 -- 172.21.15.27:0/3047823869 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e4003570 con 0x7f32e4003ae0 2024-01-11T08:25:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.808+0000 7f32fe79c700 1 -- 172.21.15.27:0/3047823869 reap_dead start 2024-01-11T08:25:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.808+0000 7f32fdf9b700 1 --2- 172.21.15.27:0/3047823869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e4003ae0 0x7f32e4003ef0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f32f400c2d0 tx=0x7f32f4001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=680613c62de1bc2e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.809+0000 7f32fcf99700 1 -- 172.21.15.27:0/3047823869 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f400cba0 con 0x7f32e4003ae0 2024-01-11T08:25:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.809+0000 7f32fcf99700 1 -- 172.21.15.27:0/3047823869 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f400cd20 con 0x7f32e4003ae0 2024-01-11T08:25:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.809+0000 7f32fcf99700 1 -- 172.21.15.27:0/3047823869 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4005b70 con 0x7f32e4003ae0 2024-01-11T08:25:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.809+0000 7f33001ff700 1 -- 172.21.15.27:0/3047823869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e4003ae0 msgr2=0x7f32e4003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.809+0000 7f33001ff700 1 --2- 172.21.15.27:0/3047823869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e4003ae0 0x7f32e4003ef0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f32f400c2d0 tx=0x7f32f4001d80 comp rx=0 tx=0).stop 2024-01-11T08:25:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.809+0000 7f33001ff700 1 -- 172.21.15.27:0/3047823869 shutdown_connections 2024-01-11T08:25:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.809+0000 7f33001ff700 1 --2- 172.21.15.27:0/3047823869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e4003ae0 0x7f32e4003ef0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.809+0000 7f33001ff700 1 -- 172.21.15.27:0/3047823869 >> 172.21.15.27:0/3047823869 conn(0x7f32f806ed40 msgr2=0x7f32f806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.809+0000 7f33001ff700 1 -- 172.21.15.27:0/3047823869 shutdown_connections 2024-01-11T08:25:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.809+0000 7f33001ff700 1 -- 172.21.15.27:0/3047823869 wait complete. 2024-01-11T08:25:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.811+0000 7f33001ff700 1 Processor -- start 2024-01-11T08:25:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.811+0000 7f33001ff700 1 -- start start 2024-01-11T08:25:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.812+0000 7f33001ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e4003ae0 0x7f32f80a6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.812+0000 7f33001ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32f80c96b0 0x7f32f80ce930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.812+0000 7f33001ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f80c9b20 0x7f32f80a7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.812+0000 7f33001ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f816ad70 con 0x7f32f80c9b20 2024-01-11T08:25:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.812+0000 7f33001ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f816a1f0 con 0x7f32e4003ae0 2024-01-11T08:25:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.812+0000 7f33001ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f816a4d0 con 0x7f32f80c96b0 2024-01-11T08:25:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.812+0000 7f32fe79c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f80c9b20 0x7f32f80a7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.812+0000 7f32fe79c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f80c9b20 0x7f32f80a7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35444/0 (socket says 172.21.15.27:35444) 2024-01-11T08:25:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.813+0000 7f32fe79c700 1 -- 172.21.15.27:0/1088852495 learned_addr learned my addr 172.21.15.27:0/1088852495 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.813+0000 7f32fdf9b700 1 --2- 172.21.15.27:0/1088852495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e4003ae0 0x7f32f80a6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.813+0000 7f32fd79a700 1 --2- 172.21.15.27:0/1088852495 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32f80c96b0 0x7f32f80ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.813+0000 7f32fe79c700 1 -- 172.21.15.27:0/1088852495 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32f80c96b0 msgr2=0x7f32f80ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.813+0000 7f32fe79c700 1 --2- 172.21.15.27:0/1088852495 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32f80c96b0 0x7f32f80ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.813+0000 7f32fe79c700 1 -- 172.21.15.27:0/1088852495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e4003ae0 msgr2=0x7f32f80a6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.813+0000 7f32fe79c700 1 --2- 172.21.15.27:0/1088852495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e4003ae0 0x7f32f80a6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.813+0000 7f32fe79c700 1 -- 172.21.15.27:0/1088852495 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f816a7b0 con 0x7f32f80c9b20 2024-01-11T08:25:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.813+0000 7f32fe79c700 1 --2- 172.21.15.27:0/1088852495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f80c9b20 0x7f32f80a7a20 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f32f80895b0 tx=0x7f32f000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.813+0000 7f32edffb700 1 -- 172.21.15.27:0/1088852495 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f000e070 con 0x7f32f80c9b20 2024-01-11T08:25:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.813+0000 7f32edffb700 1 -- 172.21.15.27:0/1088852495 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f0011b80 con 0x7f32f80c9b20 2024-01-11T08:25:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.813+0000 7f33001ff700 1 -- 172.21.15.27:0/1088852495 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f816aa90 con 0x7f32f80c9b20 2024-01-11T08:25:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.813+0000 7f32edffb700 1 -- 172.21.15.27:0/1088852495 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f0016470 con 0x7f32f80c9b20 2024-01-11T08:25:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.814+0000 7f33001ff700 1 -- 172.21.15.27:0/1088852495 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f8169da0 con 0x7f32f80c9b20 2024-01-11T08:25:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.815+0000 7f33001ff700 1 -- 172.21.15.27:0/1088852495 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f80718d0 con 0x7f32f80c9b20 2024-01-11T08:25:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.816+0000 7f32edffb700 1 -- 172.21.15.27:0/1088852495 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f32f00168a0 con 0x7f32f80c9b20 2024-01-11T08:25:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.816+0000 7f32edffb700 1 --2- 172.21.15.27:0/1088852495 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f32dc066f30 0x7f32dc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.817+0000 7f32edffb700 1 -- 172.21.15.27:0/1088852495 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4881+0+0 (secure 0 0 0) 0x7f32f0084e70 con 0x7f32f80c9b20 2024-01-11T08:25:57.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.817+0000 7f32fdf9b700 1 --2- 172.21.15.27:0/1088852495 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f32dc066f30 0x7f32dc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:57.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.818+0000 7f32fdf9b700 1 --2- 172.21.15.27:0/1088852495 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f32dc066f30 0x7f32dc0693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f32f4007c00 tx=0x7f32f4015000 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:57.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:57.821+0000 7f32edffb700 1 -- 172.21.15.27:0/1088852495 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f32f0052370 con 0x7f32f80c9b20 2024-01-11T08:25:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.029+0000 7f33001ff700 1 -- 172.21.15.27:0/1088852495 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f32f816d3a0 con 0x7f32f80c9b20 2024-01-11T08:25:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.029+0000 7f32edffb700 1 -- 172.21.15.27:0/1088852495 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v84) v1 ==== 84+0+65 (secure 0 0 0) 0x7f32f0055990 con 0x7f32f80c9b20 2024-01-11T08:25:58.029 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:25:58.029 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T08:25:58.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:25:58.029 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-11T08:25:58.029 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-11T08:25:58.029 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:25:58.029 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T08:25:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.032+0000 7f33001ff700 1 -- 172.21.15.27:0/1088852495 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f32dc066f30 msgr2=0x7f32dc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.032+0000 7f33001ff700 1 --2- 172.21.15.27:0/1088852495 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f32dc066f30 0x7f32dc0693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f32f4007c00 tx=0x7f32f4015000 comp rx=0 tx=0).stop 2024-01-11T08:25:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.032+0000 7f33001ff700 1 -- 172.21.15.27:0/1088852495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f80c9b20 msgr2=0x7f32f80a7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.032+0000 7f33001ff700 1 --2- 172.21.15.27:0/1088852495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f80c9b20 0x7f32f80a7a20 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f32f80895b0 tx=0x7f32f000b040 comp rx=0 tx=0).stop 2024-01-11T08:25:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.032+0000 7f33001ff700 1 -- 172.21.15.27:0/1088852495 shutdown_connections 2024-01-11T08:25:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.032+0000 7f33001ff700 1 --2- 172.21.15.27:0/1088852495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32f80c9b20 0x7f32f80a7a20 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.032+0000 7f33001ff700 1 --2- 172.21.15.27:0/1088852495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e4003ae0 0x7f32f80a6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.032+0000 7f33001ff700 1 --2- 172.21.15.27:0/1088852495 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f32dc066f30 0x7f32dc0693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.032+0000 7f33001ff700 1 --2- 172.21.15.27:0/1088852495 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32f80c96b0 0x7f32f80ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.032+0000 7f33001ff700 1 -- 172.21.15.27:0/1088852495 >> 172.21.15.27:0/1088852495 conn(0x7f32f806ed40 msgr2=0x7f32f80b09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.032+0000 7f33001ff700 1 -- 172.21.15.27:0/1088852495 shutdown_connections 2024-01-11T08:25:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.032+0000 7f33001ff700 1 -- 172.21.15.27:0/1088852495 wait complete. 2024-01-11T08:25:58.049 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-11T08:25:58.049 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-11T08:25:58.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.253+0000 7f9d909ff700 1 Processor -- start 2024-01-11T08:25:58.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.254+0000 7f9d909ff700 1 -- start start 2024-01-11T08:25:58.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.255+0000 7f9d909ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9d8c0b08a0 0x7f9d8c0684d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:58.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.255+0000 7f9d909ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8c068e10 con 0x7f9d8c159250 2024-01-11T08:25:58.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.255+0000 7f9d909ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8c0b0d20 con 0x7f9d8c068a10 2024-01-11T08:25:58.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.255+0000 7f9d909ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d8c15ace0 con 0x7f9d8c0b08a0 2024-01-11T08:25:58.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.255+0000 7f9d8a59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9d8c159250 0x7f9d8c0cb410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45764/0 (socket says 172.21.15.27:45764) 2024-01-11T08:25:58.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.255+0000 7f9d8a59c700 1 -- 172.21.15.27:0/2746822197 learned_addr learned my addr 172.21.15.27:0/2746822197 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:25:58.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.255+0000 7f9d89d9b700 1 --2- 172.21.15.27:0/2746822197 >> v2:172.21.15.27:3300/0 conn(0x7f9d8c0b08a0 0x7f9d8c0684d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.256+0000 7f9d8959a700 1 -- 172.21.15.27:0/2746822197 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145071566 0 0) 0x7f9d8c068e10 con 0x7f9d8c159250 2024-01-11T08:25:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.256+0000 7f9d8959a700 1 -- 172.21.15.27:0/2746822197 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d74003570 con 0x7f9d8c159250 2024-01-11T08:25:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.256+0000 7f9d8959a700 1 -- 172.21.15.27:0/2746822197 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556111801 0 0) 0x7f9d8c0b0d20 con 0x7f9d8c068a10 2024-01-11T08:25:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.256+0000 7f9d8959a700 1 -- 172.21.15.27:0/2746822197 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d8c068e10 con 0x7f9d8c068a10 2024-01-11T08:25:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.256+0000 7f9d89d9b700 1 -- 172.21.15.27:0/2746822197 >> v1:172.21.15.77:6789/0 conn(0x7f9d8c068a10 legacy=0x7f9d8c0ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.256+0000 7f9d89d9b700 1 -- 172.21.15.27:0/2746822197 >> v1:172.21.15.27:6789/0 conn(0x7f9d8c159250 legacy=0x7f9d8c0cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.256+0000 7f9d89d9b700 1 -- 172.21.15.27:0/2746822197 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c15a440 con 0x7f9d8c0b08a0 2024-01-11T08:25:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.256+0000 7f9d89d9b700 1 --2- 172.21.15.27:0/2746822197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d8c0b08a0 0x7f9d8c0684d0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f9d800098e0 tx=0x7f9d800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90fbe6264f989ae4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.256+0000 7f9d8959a700 1 -- 172.21.15.27:0/2746822197 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d80011630 con 0x7f9d8c0b08a0 2024-01-11T08:25:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.256+0000 7f9d8959a700 1 -- 172.21.15.27:0/2746822197 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d80011790 con 0x7f9d8c0b08a0 2024-01-11T08:25:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.256+0000 7f9d8959a700 1 -- 172.21.15.27:0/2746822197 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d80011b30 con 0x7f9d8c0b08a0 2024-01-11T08:25:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.257+0000 7f9d909ff700 1 -- 172.21.15.27:0/2746822197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d8c0b08a0 msgr2=0x7f9d8c0684d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.257+0000 7f9d909ff700 1 --2- 172.21.15.27:0/2746822197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d8c0b08a0 0x7f9d8c0684d0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f9d800098e0 tx=0x7f9d800091a0 comp rx=0 tx=0).stop 2024-01-11T08:25:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.257+0000 7f9d909ff700 1 -- 172.21.15.27:0/2746822197 shutdown_connections 2024-01-11T08:25:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.257+0000 7f9d909ff700 1 --2- 172.21.15.27:0/2746822197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d8c0b08a0 0x7f9d8c0684d0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.257+0000 7f9d909ff700 1 -- 172.21.15.27:0/2746822197 >> 172.21.15.27:0/2746822197 conn(0x7f9d8c067610 msgr2=0x7f9d8c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.257+0000 7f9d909ff700 1 -- 172.21.15.27:0/2746822197 shutdown_connections 2024-01-11T08:25:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.257+0000 7f9d909ff700 1 -- 172.21.15.27:0/2746822197 wait complete. 2024-01-11T08:25:58.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.259+0000 7f9d909ff700 1 Processor -- start 2024-01-11T08:25:58.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.259+0000 7f9d909ff700 1 -- start start 2024-01-11T08:25:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.260+0000 7f9d909ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9d8c068a10 0x7f9d8c14a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.260+0000 7f9d909ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d8c0b08a0 0x7f9d8c0b2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.260+0000 7f9d909ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d8c159250 0x7f9d8c0c1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.260+0000 7f9d909ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c0c5de0 con 0x7f9d8c0b08a0 2024-01-11T08:25:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.260+0000 7f9d909ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c0c63a0 con 0x7f9d8c159250 2024-01-11T08:25:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.260+0000 7f9d909ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c0b3600 con 0x7f9d8c068a10 2024-01-11T08:25:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.260+0000 7f9d89d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d8c0b08a0 0x7f9d8c0b2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.260+0000 7f9d89d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d8c0b08a0 0x7f9d8c0b2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35462/0 (socket says 172.21.15.27:35462) 2024-01-11T08:25:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.260+0000 7f9d89d9b700 1 -- 172.21.15.27:0/1332730096 learned_addr learned my addr 172.21.15.27:0/1332730096 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.260+0000 7f9d8a59c700 1 --2- 172.21.15.27:0/1332730096 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9d8c068a10 0x7f9d8c14a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.260+0000 7f9d8ad9d700 1 --2- 172.21.15.27:0/1332730096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d8c159250 0x7f9d8c0c1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.260+0000 7f9d89d9b700 1 -- 172.21.15.27:0/1332730096 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9d8c068a10 msgr2=0x7f9d8c14a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.260+0000 7f9d89d9b700 1 --2- 172.21.15.27:0/1332730096 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9d8c068a10 0x7f9d8c14a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.260+0000 7f9d89d9b700 1 -- 172.21.15.27:0/1332730096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d8c159250 msgr2=0x7f9d8c0c1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.260+0000 7f9d89d9b700 1 --2- 172.21.15.27:0/1332730096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d8c159250 0x7f9d8c0c1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.260+0000 7f9d89d9b700 1 -- 172.21.15.27:0/1332730096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d80009420 con 0x7f9d8c0b08a0 2024-01-11T08:25:58.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.261+0000 7f9d89d9b700 1 --2- 172.21.15.27:0/1332730096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d8c0b08a0 0x7f9d8c0b2af0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c068fa0 tx=0x7f9d80017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:58.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.261+0000 7f9d7a7fc700 1 -- 172.21.15.27:0/1332730096 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8001f070 con 0x7f9d8c0b08a0 2024-01-11T08:25:58.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.261+0000 7f9d7a7fc700 1 -- 172.21.15.27:0/1332730096 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d80000c90 con 0x7f9d8c0b08a0 2024-01-11T08:25:58.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.261+0000 7f9d909ff700 1 -- 172.21.15.27:0/1332730096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8c0b38e0 con 0x7f9d8c0b08a0 2024-01-11T08:25:58.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.261+0000 7f9d7a7fc700 1 -- 172.21.15.27:0/1332730096 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d80025420 con 0x7f9d8c0b08a0 2024-01-11T08:25:58.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.261+0000 7f9d909ff700 1 -- 172.21.15.27:0/1332730096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8c157130 con 0x7f9d8c0b08a0 2024-01-11T08:25:58.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.262+0000 7f9d909ff700 1 -- 172.21.15.27:0/1332730096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8c0c6510 con 0x7f9d8c0b08a0 2024-01-11T08:25:58.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.263+0000 7f9d7a7fc700 1 -- 172.21.15.27:0/1332730096 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9d80023070 con 0x7f9d8c0b08a0 2024-01-11T08:25:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.264+0000 7f9d7a7fc700 1 --2- 172.21.15.27:0/1332730096 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9d6c066f30 0x7f9d6c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.264+0000 7f9d7a7fc700 1 -- 172.21.15.27:0/1332730096 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4881+0+0 (secure 0 0 0) 0x7f9d800855e0 con 0x7f9d8c0b08a0 2024-01-11T08:25:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.265+0000 7f9d8a59c700 1 --2- 172.21.15.27:0/1332730096 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9d6c066f30 0x7f9d6c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.265+0000 7f9d8a59c700 1 --2- 172.21.15.27:0/1332730096 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9d6c066f30 0x7f9d6c0693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c0058d0 tx=0x7f9d7c009040 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:58.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.268+0000 7f9d7a7fc700 1 -- 172.21.15.27:0/1332730096 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9d80052b80 con 0x7f9d8c0b08a0 2024-01-11T08:25:58.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:58.470+0000 7f9d909ff700 1 -- 172.21.15.27:0/1332730096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9d8c04c3c0 con 0x7f9d8c0b08a0 2024-01-11T08:25:59.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.198+0000 7f9d7a7fc700 1 -- 172.21.15.27:0/1332730096 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v85) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9d800150b0 con 0x7f9d8c0b08a0 2024-01-11T08:25:59.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.201+0000 7f9d909ff700 1 -- 172.21.15.27:0/1332730096 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9d6c066f30 msgr2=0x7f9d6c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:59.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.201+0000 7f9d909ff700 1 --2- 172.21.15.27:0/1332730096 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9d6c066f30 0x7f9d6c0693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c0058d0 tx=0x7f9d7c009040 comp rx=0 tx=0).stop 2024-01-11T08:25:59.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.201+0000 7f9d909ff700 1 -- 172.21.15.27:0/1332730096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d8c0b08a0 msgr2=0x7f9d8c0b2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:59.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.201+0000 7f9d909ff700 1 --2- 172.21.15.27:0/1332730096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d8c0b08a0 0x7f9d8c0b2af0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c068fa0 tx=0x7f9d80017000 comp rx=0 tx=0).stop 2024-01-11T08:25:59.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.202+0000 7f9d909ff700 1 -- 172.21.15.27:0/1332730096 shutdown_connections 2024-01-11T08:25:59.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.202+0000 7f9d909ff700 1 --2- 172.21.15.27:0/1332730096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d8c159250 0x7f9d8c0c1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:59.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.202+0000 7f9d909ff700 1 --2- 172.21.15.27:0/1332730096 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9d6c066f30 0x7f9d6c0693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:59.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.202+0000 7f9d909ff700 1 --2- 172.21.15.27:0/1332730096 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9d8c068a10 0x7f9d8c14a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:59.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.202+0000 7f9d909ff700 1 --2- 172.21.15.27:0/1332730096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d8c0b08a0 0x7f9d8c0b2af0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:59.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.202+0000 7f9d909ff700 1 -- 172.21.15.27:0/1332730096 >> 172.21.15.27:0/1332730096 conn(0x7f9d8c067610 msgr2=0x7f9d8c0ce280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:59.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.202+0000 7f9d909ff700 1 -- 172.21.15.27:0/1332730096 shutdown_connections 2024-01-11T08:25:59.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.202+0000 7f9d909ff700 1 -- 172.21.15.27:0/1332730096 wait complete. 2024-01-11T08:25:59.203 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-11T08:25:59.223 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-11T08:25:59.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.433+0000 7f92e0bff700 1 Processor -- start 2024-01-11T08:25:59.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.433+0000 7f92e0bff700 1 -- start start 2024-01-11T08:25:59.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.434+0000 7f92e0bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f92dc0b03e0 0x7f92dc0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:59.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.434+0000 7f92e0bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f92dc0b29c0 0x7f92dc159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:59.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.434+0000 7f92e0bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92dc0cd730 con 0x7f92dc0b07b0 2024-01-11T08:25:59.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.434+0000 7f92e0bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92dc0b2de0 con 0x7f92dc0b03e0 2024-01-11T08:25:59.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.434+0000 7f92e0bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc0ccb20 con 0x7f92dc0b29c0 2024-01-11T08:25:59.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.434+0000 7f92da59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f92dc0b03e0 0x7f92dc0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:59.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.435+0000 7f92da59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f92dc0b03e0 0x7f92dc0b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35478/0 (socket says 172.21.15.27:35478) 2024-01-11T08:25:59.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.435+0000 7f92da59c700 1 -- 172.21.15.27:0/2684163391 learned_addr learned my addr 172.21.15.27:0/2684163391 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:59.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.435+0000 7f92dad9d700 1 --2- 172.21.15.27:0/2684163391 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f92dc0b29c0 0x7f92dc159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:59.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.435+0000 7f92da59c700 1 -- 172.21.15.27:0/2684163391 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f92dc0b29c0 msgr2=0x7f92dc159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:25:59.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.435+0000 7f92da59c700 1 --2- 172.21.15.27:0/2684163391 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f92dc0b29c0 0x7f92dc159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:59.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.435+0000 7f92da59c700 1 -- 172.21.15.27:0/2684163391 >> v1:172.21.15.77:6789/0 conn(0x7f92dc0b07b0 legacy=0x7f92dc159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:59.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.435+0000 7f92da59c700 1 -- 172.21.15.27:0/2684163391 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92dc0cce00 con 0x7f92dc0b03e0 2024-01-11T08:25:59.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.435+0000 7f92da59c700 1 --2- 172.21.15.27:0/2684163391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92dc0b03e0 0x7f92dc0b2480 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f92cc0099b0 tx=0x7f92cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de55a41f17c10428 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:59.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.436+0000 7f92d959a700 1 -- 172.21.15.27:0/2684163391 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc009ea0 con 0x7f92dc0b03e0 2024-01-11T08:25:59.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.436+0000 7f92d959a700 1 -- 172.21.15.27:0/2684163391 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92cc00b860 con 0x7f92dc0b03e0 2024-01-11T08:25:59.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.436+0000 7f92d959a700 1 -- 172.21.15.27:0/2684163391 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc00bc00 con 0x7f92dc0b03e0 2024-01-11T08:25:59.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.436+0000 7f92e0bff700 1 -- 172.21.15.27:0/2684163391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92dc0b03e0 msgr2=0x7f92dc0b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:59.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.436+0000 7f92e0bff700 1 --2- 172.21.15.27:0/2684163391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92dc0b03e0 0x7f92dc0b2480 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f92cc0099b0 tx=0x7f92cc009250 comp rx=0 tx=0).stop 2024-01-11T08:25:59.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.436+0000 7f92e0bff700 1 -- 172.21.15.27:0/2684163391 shutdown_connections 2024-01-11T08:25:59.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.436+0000 7f92e0bff700 1 --2- 172.21.15.27:0/2684163391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92dc0b03e0 0x7f92dc0b2480 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:59.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.436+0000 7f92e0bff700 1 --2- 172.21.15.27:0/2684163391 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f92dc0b29c0 0x7f92dc159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:59.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.436+0000 7f92e0bff700 1 -- 172.21.15.27:0/2684163391 >> 172.21.15.27:0/2684163391 conn(0x7f92dc06ed40 msgr2=0x7f92dc06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:25:59.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.437+0000 7f92e0bff700 1 -- 172.21.15.27:0/2684163391 shutdown_connections 2024-01-11T08:25:59.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.437+0000 7f92e0bff700 1 -- 172.21.15.27:0/2684163391 wait complete. 2024-01-11T08:25:59.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.438+0000 7f92e0bff700 1 Processor -- start 2024-01-11T08:25:59.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.438+0000 7f92e0bff700 1 -- start start 2024-01-11T08:25:59.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.439+0000 7f92e0bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92dc0b03e0 0x7f92dc158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:59.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.439+0000 7f92e0bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc0b07b0 0x7f92dc0c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:59.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.439+0000 7f92e0bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f92dc0b29c0 0x7f92dc0c7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:59.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.439+0000 7f92e0bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc16e550 con 0x7f92dc0b03e0 2024-01-11T08:25:59.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.439+0000 7f92e0bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc169f80 con 0x7f92dc0b07b0 2024-01-11T08:25:59.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.439+0000 7f92e0bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc16a7e0 con 0x7f92dc0b29c0 2024-01-11T08:25:59.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.439+0000 7f92da59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92dc0b03e0 0x7f92dc158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:59.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.439+0000 7f92da59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92dc0b03e0 0x7f92dc158d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35490/0 (socket says 172.21.15.27:35490) 2024-01-11T08:25:59.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.439+0000 7f92da59c700 1 -- 172.21.15.27:0/2940887178 learned_addr learned my addr 172.21.15.27:0/2940887178 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:25:59.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.440+0000 7f92dad9d700 1 --2- 172.21.15.27:0/2940887178 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f92dc0b29c0 0x7f92dc0c7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:59.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.440+0000 7f92d9d9b700 1 --2- 172.21.15.27:0/2940887178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc0b07b0 0x7f92dc0c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:59.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.440+0000 7f92da59c700 1 -- 172.21.15.27:0/2940887178 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f92dc0b29c0 msgr2=0x7f92dc0c7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:59.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.440+0000 7f92da59c700 1 --2- 172.21.15.27:0/2940887178 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f92dc0b29c0 0x7f92dc0c7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:59.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.440+0000 7f92da59c700 1 -- 172.21.15.27:0/2940887178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc0b07b0 msgr2=0x7f92dc0c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:25:59.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.440+0000 7f92da59c700 1 --2- 172.21.15.27:0/2940887178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc0b07b0 0x7f92dc0c75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:25:59.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.440+0000 7f92da59c700 1 -- 172.21.15.27:0/2940887178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc0094f0 con 0x7f92dc0b03e0 2024-01-11T08:25:59.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.440+0000 7f92da59c700 1 --2- 172.21.15.27:0/2940887178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92dc0b03e0 0x7f92dc158d00 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f92dc069540 tx=0x7f92cc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:59.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.440+0000 7f92ca7fc700 1 -- 172.21.15.27:0/2940887178 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc00b9d0 con 0x7f92dc0b03e0 2024-01-11T08:25:59.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.440+0000 7f92ca7fc700 1 -- 172.21.15.27:0/2940887178 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92cc00bb50 con 0x7f92dc0b03e0 2024-01-11T08:25:59.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.440+0000 7f92e0bff700 1 -- 172.21.15.27:0/2940887178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92dc16e270 con 0x7f92dc0b03e0 2024-01-11T08:25:59.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.441+0000 7f92ca7fc700 1 -- 172.21.15.27:0/2940887178 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc011670 con 0x7f92dc0b03e0 2024-01-11T08:25:59.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.441+0000 7f92e0bff700 1 -- 172.21.15.27:0/2940887178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92dc16a470 con 0x7f92dc0b03e0 2024-01-11T08:25:59.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.442+0000 7f92e0bff700 1 -- 172.21.15.27:0/2940887178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92dc15d390 con 0x7f92dc0b03e0 2024-01-11T08:25:59.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.443+0000 7f92ca7fc700 1 -- 172.21.15.27:0/2940887178 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f92cc011a20 con 0x7f92dc0b03e0 2024-01-11T08:25:59.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.445+0000 7f92ca7fc700 1 --2- 172.21.15.27:0/2940887178 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f92bc066f30 0x7f92bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:25:59.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.445+0000 7f92ca7fc700 1 -- 172.21.15.27:0/2940887178 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5251+0+0 (secure 0 0 0) 0x7f92cc085c30 con 0x7f92dc0b03e0 2024-01-11T08:25:59.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.446+0000 7f92d9d9b700 1 --2- 172.21.15.27:0/2940887178 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f92bc066f30 0x7f92bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:25:59.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.446+0000 7f92d9d9b700 1 --2- 172.21.15.27:0/2940887178 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f92bc066f30 0x7f92bc0693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f92dc159a60 tx=0x7f92d0000bc0 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:25:59.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.449+0000 7f92ca7fc700 1 -- 172.21.15.27:0/2940887178 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f92cc0530c0 con 0x7f92dc0b03e0 2024-01-11T08:25:59.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:25:59.654+0000 7f92e0bff700 1 -- 172.21.15.27:0/2940887178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f92dc04c3c0 con 0x7f92dc0b03e0 2024-01-11T08:26:00.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.206+0000 7f92ca7fc700 1 -- 172.21.15.27:0/2940887178 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v86) v1 ==== 170+0+0 (secure 0 0 0) 0x7f92cc0566e0 con 0x7f92dc0b03e0 2024-01-11T08:26:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.211+0000 7f92e0bff700 1 -- 172.21.15.27:0/2940887178 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f92bc066f30 msgr2=0x7f92bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.211+0000 7f92e0bff700 1 --2- 172.21.15.27:0/2940887178 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f92bc066f30 0x7f92bc0693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f92dc159a60 tx=0x7f92d0000bc0 comp rx=0 tx=0).stop 2024-01-11T08:26:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.211+0000 7f92e0bff700 1 -- 172.21.15.27:0/2940887178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92dc0b03e0 msgr2=0x7f92dc158d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.211+0000 7f92e0bff700 1 --2- 172.21.15.27:0/2940887178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92dc0b03e0 0x7f92dc158d00 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f92dc069540 tx=0x7f92cc017000 comp rx=0 tx=0).stop 2024-01-11T08:26:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.211+0000 7f92e0bff700 1 -- 172.21.15.27:0/2940887178 shutdown_connections 2024-01-11T08:26:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.211+0000 7f92e0bff700 1 --2- 172.21.15.27:0/2940887178 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f92bc066f30 0x7f92bc0693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.211+0000 7f92e0bff700 1 --2- 172.21.15.27:0/2940887178 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f92dc0b29c0 0x7f92dc0c7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.211+0000 7f92e0bff700 1 --2- 172.21.15.27:0/2940887178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92dc0b03e0 0x7f92dc158d00 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.211+0000 7f92e0bff700 1 --2- 172.21.15.27:0/2940887178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92dc0b07b0 0x7f92dc0c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.211+0000 7f92e0bff700 1 -- 172.21.15.27:0/2940887178 >> 172.21.15.27:0/2940887178 conn(0x7f92dc06ed40 msgr2=0x7f92dc0b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.212+0000 7f92e0bff700 1 -- 172.21.15.27:0/2940887178 shutdown_connections 2024-01-11T08:26:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.212+0000 7f92e0bff700 1 -- 172.21.15.27:0/2940887178 wait complete. 2024-01-11T08:26:00.219 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-11T08:26:00.243 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-11T08:26:00.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.446+0000 7f93401ff700 1 Processor -- start 2024-01-11T08:26:00.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.446+0000 7f93401ff700 1 -- start start 2024-01-11T08:26:00.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.448+0000 7f93401ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93380ccd70 0x7f93380cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:00.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.448+0000 7f93401ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f933815ae70 0x7f933815b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:00.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.448+0000 7f93401ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93380cf850 con 0x7f933815a1b0 2024-01-11T08:26:00.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.448+0000 7f93401ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f933815ac90 con 0x7f933815ae70 2024-01-11T08:26:00.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.448+0000 7f93401ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93380cec40 con 0x7f93380ccd70 2024-01-11T08:26:00.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.448+0000 7f933e79c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f933815ae70 0x7f933815b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:00.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.448+0000 7f933e79c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f933815ae70 0x7f933815b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35496/0 (socket says 172.21.15.27:35496) 2024-01-11T08:26:00.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.448+0000 7f933e79c700 1 -- 172.21.15.27:0/2311225099 learned_addr learned my addr 172.21.15.27:0/2311225099 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:00.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.448+0000 7f933d79a700 1 --2- 172.21.15.27:0/2311225099 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93380ccd70 0x7f93380cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:00.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.449+0000 7f933e79c700 1 -- 172.21.15.27:0/2311225099 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93380ccd70 msgr2=0x7f93380cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:00.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.449+0000 7f933e79c700 1 --2- 172.21.15.27:0/2311225099 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93380ccd70 0x7f93380cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:00.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.449+0000 7f933e79c700 1 -- 172.21.15.27:0/2311225099 >> v1:172.21.15.77:6789/0 conn(0x7f933815a1b0 legacy=0x7f933815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:00.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.449+0000 7f933e79c700 1 -- 172.21.15.27:0/2311225099 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93380cef20 con 0x7f933815ae70 2024-01-11T08:26:00.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.449+0000 7f933e79c700 1 --2- 172.21.15.27:0/2311225099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f933815ae70 0x7f933815b2f0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f9330009f20 tx=0x7f9330009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96771b98f18fab1c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:00.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.449+0000 7f933cf99700 1 -- 172.21.15.27:0/2311225099 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933000e070 con 0x7f933815ae70 2024-01-11T08:26:00.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.449+0000 7f933cf99700 1 -- 172.21.15.27:0/2311225099 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933000b9d0 con 0x7f933815ae70 2024-01-11T08:26:00.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.449+0000 7f933cf99700 1 -- 172.21.15.27:0/2311225099 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93300092e0 con 0x7f933815ae70 2024-01-11T08:26:00.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.450+0000 7f93401ff700 1 -- 172.21.15.27:0/2311225099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f933815ae70 msgr2=0x7f933815b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:00.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.450+0000 7f93401ff700 1 --2- 172.21.15.27:0/2311225099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f933815ae70 0x7f933815b2f0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f9330009f20 tx=0x7f9330009840 comp rx=0 tx=0).stop 2024-01-11T08:26:00.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.450+0000 7f93401ff700 1 -- 172.21.15.27:0/2311225099 shutdown_connections 2024-01-11T08:26:00.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.450+0000 7f93401ff700 1 --2- 172.21.15.27:0/2311225099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f933815ae70 0x7f933815b2f0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:00.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.450+0000 7f93401ff700 1 --2- 172.21.15.27:0/2311225099 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93380ccd70 0x7f93380cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:00.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.450+0000 7f93401ff700 1 -- 172.21.15.27:0/2311225099 >> 172.21.15.27:0/2311225099 conn(0x7f933806ed40 msgr2=0x7f933806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:00.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.450+0000 7f93401ff700 1 -- 172.21.15.27:0/2311225099 shutdown_connections 2024-01-11T08:26:00.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.450+0000 7f93401ff700 1 -- 172.21.15.27:0/2311225099 wait complete. 2024-01-11T08:26:00.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.452+0000 7f93401ff700 1 Processor -- start 2024-01-11T08:26:00.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.452+0000 7f93401ff700 1 -- start start 2024-01-11T08:26:00.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.453+0000 7f93401ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93380ccd70 0x7f9338157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:00.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.454+0000 7f93401ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f933815a1b0 0x7f93380b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:00.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.454+0000 7f93401ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f933815ae70 0x7f933816a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:00.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.454+0000 7f93401ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93380b5010 con 0x7f933815ae70 2024-01-11T08:26:00.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.454+0000 7f93401ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93380b37e0 con 0x7f933815a1b0 2024-01-11T08:26:00.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.454+0000 7f93401ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93380b4640 con 0x7f93380ccd70 2024-01-11T08:26:00.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.454+0000 7f933e79c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f933815ae70 0x7f933816a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:00.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.454+0000 7f933d79a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f933815a1b0 0x7f93380b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:00.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.454+0000 7f933e79c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f933815ae70 0x7f933816a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35506/0 (socket says 172.21.15.27:35506) 2024-01-11T08:26:00.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.454+0000 7f933e79c700 1 -- 172.21.15.27:0/3659795184 learned_addr learned my addr 172.21.15.27:0/3659795184 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:00.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.454+0000 7f933df9b700 1 --2- 172.21.15.27:0/3659795184 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93380ccd70 0x7f9338157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:00.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.454+0000 7f933e79c700 1 -- 172.21.15.27:0/3659795184 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93380ccd70 msgr2=0x7f9338157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:00.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.454+0000 7f933e79c700 1 --2- 172.21.15.27:0/3659795184 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93380ccd70 0x7f9338157640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:00.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.454+0000 7f933e79c700 1 -- 172.21.15.27:0/3659795184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f933815a1b0 msgr2=0x7f93380b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:00.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.454+0000 7f933e79c700 1 --2- 172.21.15.27:0/3659795184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f933815a1b0 0x7f93380b4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:00.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.454+0000 7f933e79c700 1 -- 172.21.15.27:0/3659795184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9330009ae0 con 0x7f933815ae70 2024-01-11T08:26:00.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.454+0000 7f933d79a700 1 --2- 172.21.15.27:0/3659795184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f933815a1b0 0x7f93380b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:00.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.455+0000 7f933e79c700 1 --2- 172.21.15.27:0/3659795184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f933815ae70 0x7f933816a260 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f9330009430 tx=0x7f9330018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:00.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.455+0000 7f932dffb700 1 -- 172.21.15.27:0/3659795184 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9330008b00 con 0x7f933815ae70 2024-01-11T08:26:00.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.455+0000 7f932dffb700 1 -- 172.21.15.27:0/3659795184 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9330008c60 con 0x7f933815ae70 2024-01-11T08:26:00.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.455+0000 7f93401ff700 1 -- 172.21.15.27:0/3659795184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933816a7a0 con 0x7f933815ae70 2024-01-11T08:26:00.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.455+0000 7f932dffb700 1 -- 172.21.15.27:0/3659795184 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933001c410 con 0x7f933815ae70 2024-01-11T08:26:00.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.455+0000 7f93401ff700 1 -- 172.21.15.27:0/3659795184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93380b3d10 con 0x7f933815ae70 2024-01-11T08:26:00.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.456+0000 7f93401ff700 1 -- 172.21.15.27:0/3659795184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93380b47d0 con 0x7f933815ae70 2024-01-11T08:26:00.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.457+0000 7f932dffb700 1 -- 172.21.15.27:0/3659795184 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9330003040 con 0x7f933815ae70 2024-01-11T08:26:00.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.458+0000 7f932dffb700 1 --2- 172.21.15.27:0/3659795184 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f931c066f30 0x7f931c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:00.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.458+0000 7f932dffb700 1 -- 172.21.15.27:0/3659795184 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5633+0+0 (secure 0 0 0) 0x7f9330086440 con 0x7f933815ae70 2024-01-11T08:26:00.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.459+0000 7f933df9b700 1 --2- 172.21.15.27:0/3659795184 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f931c066f30 0x7f931c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:00.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.459+0000 7f933df9b700 1 --2- 172.21.15.27:0/3659795184 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f931c066f30 0x7f931c0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f93380ca310 tx=0x7f9334009040 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:00.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.462+0000 7f932dffb700 1 -- 172.21.15.27:0/3659795184 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f93300536d0 con 0x7f933815ae70 2024-01-11T08:26:00.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:00.708+0000 7f93401ff700 1 -- 172.21.15.27:0/3659795184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f933804c3c0 con 0x7f933815ae70 2024-01-11T08:26:01.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.209+0000 7f932dffb700 1 -- 172.21.15.27:0/3659795184 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v38) v1 ==== 174+0+0 (secure 0 0 0) 0x7f9330056cf0 con 0x7f933815ae70 2024-01-11T08:26:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.215+0000 7f93401ff700 1 -- 172.21.15.27:0/3659795184 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f931c066f30 msgr2=0x7f931c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.215+0000 7f93401ff700 1 --2- 172.21.15.27:0/3659795184 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f931c066f30 0x7f931c0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f93380ca310 tx=0x7f9334009040 comp rx=0 tx=0).stop 2024-01-11T08:26:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.215+0000 7f93401ff700 1 -- 172.21.15.27:0/3659795184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f933815ae70 msgr2=0x7f933816a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.215+0000 7f93401ff700 1 --2- 172.21.15.27:0/3659795184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f933815ae70 0x7f933816a260 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f9330009430 tx=0x7f9330018000 comp rx=0 tx=0).stop 2024-01-11T08:26:01.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.216+0000 7f93401ff700 1 -- 172.21.15.27:0/3659795184 shutdown_connections 2024-01-11T08:26:01.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.216+0000 7f93401ff700 1 --2- 172.21.15.27:0/3659795184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f933815ae70 0x7f933816a260 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:01.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.216+0000 7f93401ff700 1 --2- 172.21.15.27:0/3659795184 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f931c066f30 0x7f931c0693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:01.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.216+0000 7f93401ff700 1 --2- 172.21.15.27:0/3659795184 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93380ccd70 0x7f9338157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:01.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.216+0000 7f93401ff700 1 --2- 172.21.15.27:0/3659795184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f933815a1b0 0x7f93380b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:01.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.216+0000 7f93401ff700 1 -- 172.21.15.27:0/3659795184 >> 172.21.15.27:0/3659795184 conn(0x7f933806ed40 msgr2=0x7f9338158730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:01.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.218+0000 7f93401ff700 1 -- 172.21.15.27:0/3659795184 shutdown_connections 2024-01-11T08:26:01.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.219+0000 7f93401ff700 1 -- 172.21.15.27:0/3659795184 wait complete. 2024-01-11T08:26:01.224 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 10 and data pool 11 2024-01-11T08:26:01.254 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:26:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.456+0000 7f262efff700 1 Processor -- start 2024-01-11T08:26:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.456+0000 7f262efff700 1 -- start start 2024-01-11T08:26:01.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.458+0000 7f262efff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26280ca7b0 0x7f26280cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:01.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.458+0000 7f262efff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262815d640 con 0x7f2628159e80 2024-01-11T08:26:01.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.458+0000 7f262efff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262815a5d0 con 0x7f26280ca3e0 2024-01-11T08:26:01.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.458+0000 7f262efff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f262815ae70 con 0x7f26280ca7b0 2024-01-11T08:26:01.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.458+0000 7f262d59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2628159e80 0x7f262815dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45770/0 (socket says 172.21.15.27:45770) 2024-01-11T08:26:01.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.458+0000 7f262d59c700 1 -- 172.21.15.27:0/3306295233 learned_addr learned my addr 172.21.15.27:0/3306295233 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.458+0000 7f2627fff700 1 --2- 172.21.15.27:0/3306295233 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26280ca7b0 0x7f26280cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.458+0000 7f26277fe700 1 -- 172.21.15.27:0/3306295233 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616819539 0 0) 0x7f262815d640 con 0x7f2628159e80 2024-01-11T08:26:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.458+0000 7f26277fe700 1 -- 172.21.15.27:0/3306295233 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2618003570 con 0x7f2628159e80 2024-01-11T08:26:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.458+0000 7f26277fe700 1 -- 172.21.15.27:0/3306295233 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473121237 0 0) 0x7f262815a5d0 con 0x7f26280ca3e0 2024-01-11T08:26:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.459+0000 7f26277fe700 1 -- 172.21.15.27:0/3306295233 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f262815d640 con 0x7f26280ca3e0 2024-01-11T08:26:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.459+0000 7f2627fff700 1 -- 172.21.15.27:0/3306295233 >> v1:172.21.15.77:6789/0 conn(0x7f26280ca3e0 legacy=0x7f26280c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.459+0000 7f2627fff700 1 -- 172.21.15.27:0/3306295233 >> v1:172.21.15.27:6789/0 conn(0x7f2628159e80 legacy=0x7f262815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.459+0000 7f2627fff700 1 -- 172.21.15.27:0/3306295233 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262815a8b0 con 0x7f26280ca7b0 2024-01-11T08:26:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.459+0000 7f2627fff700 1 --2- 172.21.15.27:0/3306295233 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26280ca7b0 0x7f26280cca20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2614009910 tx=0x7f26140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1564c3343f04a246 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.459+0000 7f26277fe700 1 -- 172.21.15.27:0/3306295233 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2614011630 con 0x7f26280ca7b0 2024-01-11T08:26:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.459+0000 7f26277fe700 1 -- 172.21.15.27:0/3306295233 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2614011790 con 0x7f26280ca7b0 2024-01-11T08:26:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.459+0000 7f26277fe700 1 -- 172.21.15.27:0/3306295233 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2614011b30 con 0x7f26280ca7b0 2024-01-11T08:26:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.460+0000 7f262efff700 1 -- 172.21.15.27:0/3306295233 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26280ca7b0 msgr2=0x7f26280cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.460+0000 7f262efff700 1 --2- 172.21.15.27:0/3306295233 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26280ca7b0 0x7f26280cca20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2614009910 tx=0x7f26140091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.460+0000 7f262efff700 1 -- 172.21.15.27:0/3306295233 shutdown_connections 2024-01-11T08:26:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.460+0000 7f262efff700 1 --2- 172.21.15.27:0/3306295233 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26280ca7b0 0x7f26280cca20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.460+0000 7f262efff700 1 -- 172.21.15.27:0/3306295233 >> 172.21.15.27:0/3306295233 conn(0x7f26280ce3e0 msgr2=0x7f26280ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.460+0000 7f262efff700 1 -- 172.21.15.27:0/3306295233 shutdown_connections 2024-01-11T08:26:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.460+0000 7f262efff700 1 -- 172.21.15.27:0/3306295233 wait complete. 2024-01-11T08:26:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.461+0000 7f262efff700 1 Processor -- start 2024-01-11T08:26:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.462+0000 7f262efff700 1 -- start start 2024-01-11T08:26:01.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.463+0000 7f262efff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26280ca3e0 0x7f262815d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.463+0000 7f262efff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26280ca7b0 0x7f2628156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.463+0000 7f262efff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2628159e80 0x7f2628150ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.463+0000 7f262efff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262815df10 con 0x7f26280ca7b0 2024-01-11T08:26:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.463+0000 7f262efff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262815e1b0 con 0x7f2628159e80 2024-01-11T08:26:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.463+0000 7f262efff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f262815e490 con 0x7f26280ca3e0 2024-01-11T08:26:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.463+0000 7f2627fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26280ca7b0 0x7f2628156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.463+0000 7f2627fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26280ca7b0 0x7f2628156ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35512/0 (socket says 172.21.15.27:35512) 2024-01-11T08:26:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.463+0000 7f2627fff700 1 -- 172.21.15.27:0/479673556 learned_addr learned my addr 172.21.15.27:0/479673556 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.463+0000 7f262cd9b700 1 --2- 172.21.15.27:0/479673556 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26280ca3e0 0x7f262815d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.463+0000 7f262d59c700 1 --2- 172.21.15.27:0/479673556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2628159e80 0x7f2628150ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.463+0000 7f2627fff700 1 -- 172.21.15.27:0/479673556 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26280ca3e0 msgr2=0x7f262815d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.463+0000 7f2627fff700 1 --2- 172.21.15.27:0/479673556 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26280ca3e0 0x7f262815d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.463+0000 7f2627fff700 1 -- 172.21.15.27:0/479673556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2628159e80 msgr2=0x7f2628150ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.463+0000 7f2627fff700 1 --2- 172.21.15.27:0/479673556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2628159e80 0x7f2628150ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.463+0000 7f2627fff700 1 -- 172.21.15.27:0/479673556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2614009420 con 0x7f26280ca7b0 2024-01-11T08:26:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.463+0000 7f262d59c700 1 --2- 172.21.15.27:0/479673556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2628159e80 0x7f2628150ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:26:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.464+0000 7f2627fff700 1 --2- 172.21.15.27:0/479673556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26280ca7b0 0x7f2628156ce0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f2614009f00 tx=0x7f2614017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.464+0000 7f2624ff9700 1 -- 172.21.15.27:0/479673556 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261401f070 con 0x7f26280ca7b0 2024-01-11T08:26:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.464+0000 7f2624ff9700 1 -- 172.21.15.27:0/479673556 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2614011900 con 0x7f26280ca7b0 2024-01-11T08:26:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.464+0000 7f2624ff9700 1 -- 172.21.15.27:0/479673556 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2614023410 con 0x7f26280ca7b0 2024-01-11T08:26:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.464+0000 7f262efff700 1 -- 172.21.15.27:0/479673556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f262815e770 con 0x7f26280ca7b0 2024-01-11T08:26:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.464+0000 7f262efff700 1 -- 172.21.15.27:0/479673556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2628157220 con 0x7f26280ca7b0 2024-01-11T08:26:01.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.465+0000 7f262efff700 1 -- 172.21.15.27:0/479673556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26280718d0 con 0x7f26280ca7b0 2024-01-11T08:26:01.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.466+0000 7f2624ff9700 1 -- 172.21.15.27:0/479673556 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2614023800 con 0x7f26280ca7b0 2024-01-11T08:26:01.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.467+0000 7f2624ff9700 1 --2- 172.21.15.27:0/479673556 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f260c066f30 0x7f260c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:01.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.467+0000 7f2624ff9700 1 -- 172.21.15.27:0/479673556 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f2614085620 con 0x7f26280ca7b0 2024-01-11T08:26:01.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.468+0000 7f262cd9b700 1 --2- 172.21.15.27:0/479673556 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f260c066f30 0x7f260c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:01.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.468+0000 7f262cd9b700 1 --2- 172.21.15.27:0/479673556 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f260c066f30 0x7f260c0693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f261c007fd0 tx=0x7f261c007d50 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:01.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.471+0000 7f2624ff9700 1 -- 172.21.15.27:0/479673556 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2614052870 con 0x7f26280ca7b0 2024-01-11T08:26:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.678+0000 7f262efff700 1 -- 172.21.15.27:0/479673556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2628159c60 con 0x7f26280ca7b0 2024-01-11T08:26:01.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.679+0000 7f2624ff9700 1 -- 172.21.15.27:0/479673556 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+15361 (secure 0 0 0) 0x7f2614055e90 con 0x7f26280ca7b0 2024-01-11T08:26:01.678 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:01.678 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":87,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:26:00.709739+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:25:58.523510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-11T08:25:59.706779+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:26:01.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.681+0000 7f262efff700 1 -- 172.21.15.27:0/479673556 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f260c066f30 msgr2=0x7f260c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:01.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.681+0000 7f262efff700 1 --2- 172.21.15.27:0/479673556 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f260c066f30 0x7f260c0693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f261c007fd0 tx=0x7f261c007d50 comp rx=0 tx=0).stop 2024-01-11T08:26:01.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.681+0000 7f262efff700 1 -- 172.21.15.27:0/479673556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26280ca7b0 msgr2=0x7f2628156ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:01.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.681+0000 7f262efff700 1 --2- 172.21.15.27:0/479673556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26280ca7b0 0x7f2628156ce0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f2614009f00 tx=0x7f2614017000 comp rx=0 tx=0).stop 2024-01-11T08:26:01.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.681+0000 7f262efff700 1 -- 172.21.15.27:0/479673556 shutdown_connections 2024-01-11T08:26:01.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.681+0000 7f262efff700 1 --2- 172.21.15.27:0/479673556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2628159e80 0x7f2628150ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:01.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.681+0000 7f262efff700 1 --2- 172.21.15.27:0/479673556 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f260c066f30 0x7f260c0693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:01.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.681+0000 7f262efff700 1 --2- 172.21.15.27:0/479673556 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26280ca3e0 0x7f262815d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:01.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.681+0000 7f262efff700 1 --2- 172.21.15.27:0/479673556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26280ca7b0 0x7f2628156ce0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:01.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.681+0000 7f262efff700 1 -- 172.21.15.27:0/479673556 >> 172.21.15.27:0/479673556 conn(0x7f26280ce3e0 msgr2=0x7f2628068f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:01.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.681+0000 7f262efff700 1 -- 172.21.15.27:0/479673556 shutdown_connections 2024-01-11T08:26:01.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.681+0000 7f262efff700 1 -- 172.21.15.27:0/479673556 wait complete. 2024-01-11T08:26:01.699 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:26:01.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.899+0000 7fcc88fff700 1 Processor -- start 2024-01-11T08:26:01.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.899+0000 7fcc88fff700 1 -- start start 2024-01-11T08:26:01.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.900+0000 7fcc88fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc840b19d0 0x7fcc84158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:01.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.900+0000 7fcc88fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc84158b60 con 0x7fcc8406b610 2024-01-11T08:26:01.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.900+0000 7fcc88fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc841599b0 con 0x7fcc841575e0 2024-01-11T08:26:01.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.900+0000 7fcc88fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc840cb5e0 con 0x7fcc840b19d0 2024-01-11T08:26:01.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.901+0000 7fcc81d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc840b19d0 0x7fcc84158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:01.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.901+0000 7fcc82d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcc8406b610 0x7fcc84159e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45782/0 (socket says 172.21.15.27:45782) 2024-01-11T08:26:01.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.901+0000 7fcc82d9d700 1 -- 172.21.15.27:0/2207183987 learned_addr learned my addr 172.21.15.27:0/2207183987 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:01.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.901+0000 7fcc8159a700 1 -- 172.21.15.27:0/2207183987 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946501034 0 0) 0x7fcc7c001490 con 0x7fcc8406b610 2024-01-11T08:26:01.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.901+0000 7fcc81d9b700 1 -- 172.21.15.27:0/2207183987 >> v1:172.21.15.77:6789/0 conn(0x7fcc841575e0 legacy=0x7fcc840b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:01.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.901+0000 7fcc81d9b700 1 -- 172.21.15.27:0/2207183987 >> v1:172.21.15.27:6789/0 conn(0x7fcc8406b610 legacy=0x7fcc84159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:01.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.901+0000 7fcc81d9b700 1 -- 172.21.15.27:0/2207183987 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc840c0f90 con 0x7fcc840b19d0 2024-01-11T08:26:01.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.902+0000 7fcc81d9b700 1 --2- 172.21.15.27:0/2207183987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc840b19d0 0x7fcc84158620 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fcc78009860 tx=0x7fcc780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca9cba3f449f7387 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:01.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.902+0000 7fcc8159a700 1 -- 172.21.15.27:0/2207183987 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78011630 con 0x7fcc840b19d0 2024-01-11T08:26:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.902+0000 7fcc8159a700 1 -- 172.21.15.27:0/2207183987 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc78011790 con 0x7fcc840b19d0 2024-01-11T08:26:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.902+0000 7fcc8159a700 1 -- 172.21.15.27:0/2207183987 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78011b50 con 0x7fcc840b19d0 2024-01-11T08:26:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.902+0000 7fcc88fff700 1 -- 172.21.15.27:0/2207183987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc840b19d0 msgr2=0x7fcc84158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.902+0000 7fcc88fff700 1 --2- 172.21.15.27:0/2207183987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc840b19d0 0x7fcc84158620 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fcc78009860 tx=0x7fcc780091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.902+0000 7fcc88fff700 1 -- 172.21.15.27:0/2207183987 shutdown_connections 2024-01-11T08:26:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.902+0000 7fcc88fff700 1 --2- 172.21.15.27:0/2207183987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc840b19d0 0x7fcc84158620 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.902+0000 7fcc88fff700 1 -- 172.21.15.27:0/2207183987 >> 172.21.15.27:0/2207183987 conn(0x7fcc840680e0 msgr2=0x7fcc84069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.903+0000 7fcc88fff700 1 -- 172.21.15.27:0/2207183987 shutdown_connections 2024-01-11T08:26:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.903+0000 7fcc88fff700 1 -- 172.21.15.27:0/2207183987 wait complete. 2024-01-11T08:26:01.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.904+0000 7fcc88fff700 1 Processor -- start 2024-01-11T08:26:01.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.904+0000 7fcc88fff700 1 -- start start 2024-01-11T08:26:01.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.905+0000 7fcc88fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc8406b610 0x7fcc8416c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:01.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.905+0000 7fcc88fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc840b19d0 0x7fcc8406e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:01.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.905+0000 7fcc88fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc841575e0 0x7fcc8406eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:01.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.905+0000 7fcc88fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc840c5e20 con 0x7fcc8406b610 2024-01-11T08:26:01.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.905+0000 7fcc88fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8416ab80 con 0x7fcc840b19d0 2024-01-11T08:26:01.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.905+0000 7fcc88fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8416a8a0 con 0x7fcc841575e0 2024-01-11T08:26:01.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.905+0000 7fcc8259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc8406b610 0x7fcc8416c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:01.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.905+0000 7fcc8259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc8406b610 0x7fcc8416c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35542/0 (socket says 172.21.15.27:35542) 2024-01-11T08:26:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.906+0000 7fcc8259c700 1 -- 172.21.15.27:0/2950722130 learned_addr learned my addr 172.21.15.27:0/2950722130 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.906+0000 7fcc81d9b700 1 --2- 172.21.15.27:0/2950722130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc840b19d0 0x7fcc8406e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.906+0000 7fcc82d9d700 1 --2- 172.21.15.27:0/2950722130 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc841575e0 0x7fcc8406eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.906+0000 7fcc8259c700 1 -- 172.21.15.27:0/2950722130 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc841575e0 msgr2=0x7fcc8406eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.906+0000 7fcc8259c700 1 --2- 172.21.15.27:0/2950722130 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc841575e0 0x7fcc8406eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.906+0000 7fcc8259c700 1 -- 172.21.15.27:0/2950722130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc840b19d0 msgr2=0x7fcc8406e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.906+0000 7fcc8259c700 1 --2- 172.21.15.27:0/2950722130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc840b19d0 0x7fcc8406e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.906+0000 7fcc8259c700 1 -- 172.21.15.27:0/2950722130 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc78009420 con 0x7fcc8406b610 2024-01-11T08:26:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.906+0000 7fcc8259c700 1 --2- 172.21.15.27:0/2950722130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc8406b610 0x7fcc8416c930 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fcc7400c810 tx=0x7fcc74007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.906+0000 7fcc727fc700 1 -- 172.21.15.27:0/2950722130 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc74012630 con 0x7fcc8406b610 2024-01-11T08:26:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.906+0000 7fcc727fc700 1 -- 172.21.15.27:0/2950722130 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc740127d0 con 0x7fcc8406b610 2024-01-11T08:26:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.906+0000 7fcc727fc700 1 -- 172.21.15.27:0/2950722130 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc74012b90 con 0x7fcc8406b610 2024-01-11T08:26:01.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.906+0000 7fcc88fff700 1 -- 172.21.15.27:0/2950722130 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc840c5760 con 0x7fcc8406b610 2024-01-11T08:26:01.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.907+0000 7fcc88fff700 1 -- 172.21.15.27:0/2950722130 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc8416ce70 con 0x7fcc8406b610 2024-01-11T08:26:01.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.908+0000 7fcc88fff700 1 -- 172.21.15.27:0/2950722130 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc840c55c0 con 0x7fcc8406b610 2024-01-11T08:26:01.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.909+0000 7fcc727fc700 1 -- 172.21.15.27:0/2950722130 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fcc7401b400 con 0x7fcc8406b610 2024-01-11T08:26:01.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.910+0000 7fcc727fc700 1 --2- 172.21.15.27:0/2950722130 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc64066f30 0x7fcc640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:01.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.911+0000 7fcc727fc700 1 -- 172.21.15.27:0/2950722130 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5733+0+0 (secure 0 0 0) 0x7fcc740842b0 con 0x7fcc8406b610 2024-01-11T08:26:01.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.911+0000 7fcc81d9b700 1 --2- 172.21.15.27:0/2950722130 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc64066f30 0x7fcc640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:01.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.912+0000 7fcc81d9b700 1 --2- 172.21.15.27:0/2950722130 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc64066f30 0x7fcc640693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcc780093f0 tx=0x7fcc78016000 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:01.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:01.914+0000 7fcc727fc700 1 -- 172.21.15.27:0/2950722130 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcc840c55c0 con 0x7fcc8406b610 2024-01-11T08:26:02.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.118+0000 7fcc88fff700 1 -- 172.21.15.27:0/2950722130 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc840c55c0 con 0x7fcc8406b610 2024-01-11T08:26:02.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.119+0000 7fcc727fc700 1 -- 172.21.15.27:0/2950722130 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+15361 (secure 0 0 0) 0x7fcc840c55c0 con 0x7fcc8406b610 2024-01-11T08:26:02.118 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:02.118 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":87,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:26:00.709739+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:25:58.523510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-11T08:25:59.706779+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:26:02.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.121+0000 7fcc88fff700 1 -- 172.21.15.27:0/2950722130 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc64066f30 msgr2=0x7fcc640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:02.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.121+0000 7fcc88fff700 1 --2- 172.21.15.27:0/2950722130 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc64066f30 0x7fcc640693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcc780093f0 tx=0x7fcc78016000 comp rx=0 tx=0).stop 2024-01-11T08:26:02.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.121+0000 7fcc88fff700 1 -- 172.21.15.27:0/2950722130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc8406b610 msgr2=0x7fcc8416c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:02.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.121+0000 7fcc88fff700 1 --2- 172.21.15.27:0/2950722130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc8406b610 0x7fcc8416c930 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fcc7400c810 tx=0x7fcc74007f30 comp rx=0 tx=0).stop 2024-01-11T08:26:02.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.121+0000 7fcc88fff700 1 -- 172.21.15.27:0/2950722130 shutdown_connections 2024-01-11T08:26:02.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.121+0000 7fcc88fff700 1 --2- 172.21.15.27:0/2950722130 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc64066f30 0x7fcc640693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:02.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.121+0000 7fcc88fff700 1 --2- 172.21.15.27:0/2950722130 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc841575e0 0x7fcc8406eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:02.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.121+0000 7fcc88fff700 1 --2- 172.21.15.27:0/2950722130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc8406b610 0x7fcc8416c930 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:02.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.121+0000 7fcc88fff700 1 --2- 172.21.15.27:0/2950722130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc840b19d0 0x7fcc8406e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:02.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.121+0000 7fcc88fff700 1 -- 172.21.15.27:0/2950722130 >> 172.21.15.27:0/2950722130 conn(0x7fcc840680e0 msgr2=0x7fcc84152a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:02.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.121+0000 7fcc88fff700 1 -- 172.21.15.27:0/2950722130 shutdown_connections 2024-01-11T08:26:02.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.121+0000 7fcc88fff700 1 -- 172.21.15.27:0/2950722130 wait complete. 2024-01-11T08:26:02.138 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-11T08:26:02.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.344+0000 7f2bab9ff700 1 Processor -- start 2024-01-11T08:26:02.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.344+0000 7f2bab9ff700 1 -- start start 2024-01-11T08:26:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.346+0000 7f2bab9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba406f2b0 0x7f2ba41610a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.346+0000 7f2bab9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba40c6ff0 con 0x7f2ba40c8490 2024-01-11T08:26:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.346+0000 7f2bab9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba40cb700 con 0x7f2ba406e7d0 2024-01-11T08:26:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.346+0000 7f2bab9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba40cb9e0 con 0x7f2ba406f2b0 2024-01-11T08:26:02.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.346+0000 7f2ba8f9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2ba40c8490 0x7f2ba406eba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45798/0 (socket says 172.21.15.27:45798) 2024-01-11T08:26:02.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.346+0000 7f2ba8f9a700 1 -- 172.21.15.27:0/3624210715 learned_addr learned my addr 172.21.15.27:0/3624210715 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:02.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.346+0000 7f2ba9f9c700 1 --2- 172.21.15.27:0/3624210715 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba406f2b0 0x7f2ba41610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:02.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.347+0000 7f2b9bfff700 1 -- 172.21.15.27:0/3624210715 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101910056 0 0) 0x7f2ba40c6ff0 con 0x7f2ba40c8490 2024-01-11T08:26:02.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.347+0000 7f2b9bfff700 1 -- 172.21.15.27:0/3624210715 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b90003570 con 0x7f2ba40c8490 2024-01-11T08:26:02.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.347+0000 7f2b9bfff700 1 -- 172.21.15.27:0/3624210715 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623533628 0 0) 0x7f2ba40cb700 con 0x7f2ba406e7d0 2024-01-11T08:26:02.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.347+0000 7f2b9bfff700 1 -- 172.21.15.27:0/3624210715 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ba40c6ff0 con 0x7f2ba406e7d0 2024-01-11T08:26:02.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.347+0000 7f2ba9f9c700 1 -- 172.21.15.27:0/3624210715 >> v1:172.21.15.77:6789/0 conn(0x7f2ba406e7d0 legacy=0x7f2ba40cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:02.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.347+0000 7f2ba9f9c700 1 -- 172.21.15.27:0/3624210715 >> v1:172.21.15.27:6789/0 conn(0x7f2ba40c8490 legacy=0x7f2ba406eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:02.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.347+0000 7f2ba9f9c700 1 -- 172.21.15.27:0/3624210715 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba40c69a0 con 0x7f2ba406f2b0 2024-01-11T08:26:02.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.347+0000 7f2ba9f9c700 1 --2- 172.21.15.27:0/3624210715 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba406f2b0 0x7f2ba41610a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c009e90 tx=0x7f2b9c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b040931bfe0505a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:02.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.348+0000 7f2b9bfff700 1 -- 172.21.15.27:0/3624210715 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c00e070 con 0x7f2ba406f2b0 2024-01-11T08:26:02.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.348+0000 7f2b9bfff700 1 -- 172.21.15.27:0/3624210715 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9c00bb80 con 0x7f2ba406f2b0 2024-01-11T08:26:02.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.348+0000 7f2b9bfff700 1 -- 172.21.15.27:0/3624210715 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c012430 con 0x7f2ba406f2b0 2024-01-11T08:26:02.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.348+0000 7f2bab9ff700 1 -- 172.21.15.27:0/3624210715 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba406f2b0 msgr2=0x7f2ba41610a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:02.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.348+0000 7f2bab9ff700 1 --2- 172.21.15.27:0/3624210715 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba406f2b0 0x7f2ba41610a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c009e90 tx=0x7f2b9c009780 comp rx=0 tx=0).stop 2024-01-11T08:26:02.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.348+0000 7f2bab9ff700 1 -- 172.21.15.27:0/3624210715 shutdown_connections 2024-01-11T08:26:02.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.348+0000 7f2bab9ff700 1 --2- 172.21.15.27:0/3624210715 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba406f2b0 0x7f2ba41610a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:02.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.348+0000 7f2bab9ff700 1 -- 172.21.15.27:0/3624210715 >> 172.21.15.27:0/3624210715 conn(0x7f2ba40680a0 msgr2=0x7f2ba40cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:02.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.348+0000 7f2bab9ff700 1 -- 172.21.15.27:0/3624210715 shutdown_connections 2024-01-11T08:26:02.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.348+0000 7f2bab9ff700 1 -- 172.21.15.27:0/3624210715 wait complete. 2024-01-11T08:26:02.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.350+0000 7f2bab9ff700 1 Processor -- start 2024-01-11T08:26:02.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.350+0000 7f2bab9ff700 1 -- start start 2024-01-11T08:26:02.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.351+0000 7f2bab9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba406e7d0 0x7f2ba40c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:02.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.351+0000 7f2bab9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba406f2b0 0x7f2ba413bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:02.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.351+0000 7f2bab9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40c8490 0x7f2ba40c5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:02.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.351+0000 7f2bab9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba40c3a50 con 0x7f2ba40c8490 2024-01-11T08:26:02.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.351+0000 7f2bab9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba40aa350 con 0x7f2ba406f2b0 2024-01-11T08:26:02.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.351+0000 7f2bab9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba40aa630 con 0x7f2ba406e7d0 2024-01-11T08:26:02.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.351+0000 7f2ba9f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40c8490 0x7f2ba40c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:02.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.351+0000 7f2ba9f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40c8490 0x7f2ba40c5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35548/0 (socket says 172.21.15.27:35548) 2024-01-11T08:26:02.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.351+0000 7f2ba9f9c700 1 -- 172.21.15.27:0/4220716268 learned_addr learned my addr 172.21.15.27:0/4220716268 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:02.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.351+0000 7f2ba8f9a700 1 --2- 172.21.15.27:0/4220716268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba406f2b0 0x7f2ba413bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:02.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.351+0000 7f2ba979b700 1 --2- 172.21.15.27:0/4220716268 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba406e7d0 0x7f2ba40c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:02.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.352+0000 7f2ba9f9c700 1 -- 172.21.15.27:0/4220716268 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba406e7d0 msgr2=0x7f2ba40c3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:02.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.352+0000 7f2ba9f9c700 1 --2- 172.21.15.27:0/4220716268 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba406e7d0 0x7f2ba40c3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:02.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.352+0000 7f2ba9f9c700 1 -- 172.21.15.27:0/4220716268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba406f2b0 msgr2=0x7f2ba413bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:02.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.352+0000 7f2ba9f9c700 1 --2- 172.21.15.27:0/4220716268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba406f2b0 0x7f2ba413bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:02.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.352+0000 7f2ba9f9c700 1 -- 172.21.15.27:0/4220716268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9c009a20 con 0x7f2ba40c8490 2024-01-11T08:26:02.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.352+0000 7f2ba9f9c700 1 --2- 172.21.15.27:0/4220716268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40c8490 0x7f2ba40c5670 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4069490 tx=0x7f2b9c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:02.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.352+0000 7f2b997fa700 1 -- 172.21.15.27:0/4220716268 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c01b420 con 0x7f2ba40c8490 2024-01-11T08:26:02.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.352+0000 7f2b997fa700 1 -- 172.21.15.27:0/4220716268 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9c01b5a0 con 0x7f2ba40c8490 2024-01-11T08:26:02.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.352+0000 7f2bab9ff700 1 -- 172.21.15.27:0/4220716268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba40aa910 con 0x7f2ba40c8490 2024-01-11T08:26:02.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.352+0000 7f2b997fa700 1 -- 172.21.15.27:0/4220716268 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c01b960 con 0x7f2ba40c8490 2024-01-11T08:26:02.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.352+0000 7f2bab9ff700 1 -- 172.21.15.27:0/4220716268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba40aaed0 con 0x7f2ba40c8490 2024-01-11T08:26:02.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.353+0000 7f2bab9ff700 1 -- 172.21.15.27:0/4220716268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba40ab0c0 con 0x7f2ba40c8490 2024-01-11T08:26:02.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.354+0000 7f2b997fa700 1 -- 172.21.15.27:0/4220716268 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2b9c01bd50 con 0x7f2ba40c8490 2024-01-11T08:26:02.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.355+0000 7f2b997fa700 1 --2- 172.21.15.27:0/4220716268 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2b88066f30 0x7f2b880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:02.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.355+0000 7f2b997fa700 1 -- 172.21.15.27:0/4220716268 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f2b9c086460 con 0x7f2ba40c8490 2024-01-11T08:26:02.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.356+0000 7f2ba979b700 1 --2- 172.21.15.27:0/4220716268 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2b88066f30 0x7f2b880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:02.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.357+0000 7f2ba979b700 1 --2- 172.21.15.27:0/4220716268 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2b88066f30 0x7f2b880693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0001170 tx=0x7f2ba0009040 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:02.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.359+0000 7f2b997fa700 1 -- 172.21.15.27:0/4220716268 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2b9c0570a0 con 0x7f2ba40c8490 2024-01-11T08:26:02.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:02.608+0000 7f2bab9ff700 1 -- 172.21.15.27:0/4220716268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2ba404c3c0 con 0x7f2ba40c8490 2024-01-11T08:26:03.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.230+0000 7f2b997fa700 1 -- 172.21.15.27:0/4220716268 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v41) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2b9c056c30 con 0x7f2ba40c8490 2024-01-11T08:26:03.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.233+0000 7f2bab9ff700 1 -- 172.21.15.27:0/4220716268 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2b88066f30 msgr2=0x7f2b880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:03.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.233+0000 7f2bab9ff700 1 --2- 172.21.15.27:0/4220716268 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2b88066f30 0x7f2b880693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0001170 tx=0x7f2ba0009040 comp rx=0 tx=0).stop 2024-01-11T08:26:03.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.233+0000 7f2bab9ff700 1 -- 172.21.15.27:0/4220716268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40c8490 msgr2=0x7f2ba40c5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:03.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.233+0000 7f2bab9ff700 1 --2- 172.21.15.27:0/4220716268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40c8490 0x7f2ba40c5670 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4069490 tx=0x7f2b9c019000 comp rx=0 tx=0).stop 2024-01-11T08:26:03.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.233+0000 7f2bab9ff700 1 -- 172.21.15.27:0/4220716268 shutdown_connections 2024-01-11T08:26:03.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.233+0000 7f2bab9ff700 1 --2- 172.21.15.27:0/4220716268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40c8490 0x7f2ba40c5670 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:03.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.233+0000 7f2bab9ff700 1 --2- 172.21.15.27:0/4220716268 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2b88066f30 0x7f2b880693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:03.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.233+0000 7f2bab9ff700 1 --2- 172.21.15.27:0/4220716268 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba406e7d0 0x7f2ba40c3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:03.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.233+0000 7f2bab9ff700 1 --2- 172.21.15.27:0/4220716268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba406f2b0 0x7f2ba413bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:03.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.233+0000 7f2bab9ff700 1 -- 172.21.15.27:0/4220716268 >> 172.21.15.27:0/4220716268 conn(0x7f2ba40680a0 msgr2=0x7f2ba4069d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:03.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.233+0000 7f2bab9ff700 1 -- 172.21.15.27:0/4220716268 shutdown_connections 2024-01-11T08:26:03.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.233+0000 7f2bab9ff700 1 -- 172.21.15.27:0/4220716268 wait complete. 2024-01-11T08:26:03.252 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-11T08:26:03.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.457+0000 7f1ae49ff700 1 Processor -- start 2024-01-11T08:26:03.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.457+0000 7f1ae49ff700 1 -- start start 2024-01-11T08:26:03.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.458+0000 7f1ae49ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ae0157410 0x7f1ae00ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:03.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.458+0000 7f1ae49ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1ae01577e0 0x7f1ae00ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:03.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.458+0000 7f1ae49ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae00699d0 con 0x7f1ae0069540 2024-01-11T08:26:03.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.458+0000 7f1ae49ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae0069cb0 con 0x7f1ae01577e0 2024-01-11T08:26:03.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.458+0000 7f1ae49ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae0069f90 con 0x7f1ae0157410 2024-01-11T08:26:03.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.459+0000 7f1ade59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ae0157410 0x7f1ae00ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:03.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.459+0000 7f1ade59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ae0157410 0x7f1ae00ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:58422/0 (socket says 172.21.15.27:58422) 2024-01-11T08:26:03.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.459+0000 7f1ade59c700 1 -- 172.21.15.27:0/110972476 learned_addr learned my addr 172.21.15.27:0/110972476 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:03.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.459+0000 7f1addd9b700 1 --2- 172.21.15.27:0/110972476 >> v2:172.21.15.77:3300/0 conn(0x7f1ae01577e0 0x7f1ae00ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:03.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.459+0000 7f1add59a700 1 -- 172.21.15.27:0/110972476 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405230700 0 0) 0x7f1ae00699d0 con 0x7f1ae0069540 2024-01-11T08:26:03.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.459+0000 7f1add59a700 1 -- 172.21.15.27:0/110972476 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac8003570 con 0x7f1ae0069540 2024-01-11T08:26:03.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.459+0000 7f1ade59c700 1 -- 172.21.15.27:0/110972476 >> v2:172.21.15.77:3300/0 conn(0x7f1ae01577e0 msgr2=0x7f1ae00ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:03.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.459+0000 7f1ade59c700 1 --2- 172.21.15.27:0/110972476 >> v2:172.21.15.77:3300/0 conn(0x7f1ae01577e0 0x7f1ae00ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:03.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.459+0000 7f1ade59c700 1 -- 172.21.15.27:0/110972476 >> v1:172.21.15.77:6789/0 conn(0x7f1ae0069540 legacy=0x7f1ae015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:03.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.459+0000 7f1ade59c700 1 -- 172.21.15.27:0/110972476 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae006a270 con 0x7f1ae0157410 2024-01-11T08:26:03.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.460+0000 7f1ade59c700 1 --2- 172.21.15.27:0/110972476 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ae0157410 0x7f1ae00ccae0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0009990 tx=0x7f1ad0009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b46a30d67dce19 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:03.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.460+0000 7f1add59a700 1 -- 172.21.15.27:0/110972476 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0005c10 con 0x7f1ae0157410 2024-01-11T08:26:03.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.460+0000 7f1add59a700 1 -- 172.21.15.27:0/110972476 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad0005d70 con 0x7f1ae0157410 2024-01-11T08:26:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.460+0000 7f1add59a700 1 -- 172.21.15.27:0/110972476 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0006260 con 0x7f1ae0157410 2024-01-11T08:26:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.460+0000 7f1ae49ff700 1 -- 172.21.15.27:0/110972476 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ae0157410 msgr2=0x7f1ae00ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.460+0000 7f1ae49ff700 1 --2- 172.21.15.27:0/110972476 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ae0157410 0x7f1ae00ccae0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0009990 tx=0x7f1ad0009220 comp rx=0 tx=0).stop 2024-01-11T08:26:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.460+0000 7f1ae49ff700 1 -- 172.21.15.27:0/110972476 shutdown_connections 2024-01-11T08:26:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.460+0000 7f1ae49ff700 1 --2- 172.21.15.27:0/110972476 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ae0157410 0x7f1ae00ccae0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.460+0000 7f1ae49ff700 1 --2- 172.21.15.27:0/110972476 >> v2:172.21.15.77:3300/0 conn(0x7f1ae01577e0 0x7f1ae00ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.460+0000 7f1ae49ff700 1 -- 172.21.15.27:0/110972476 >> 172.21.15.27:0/110972476 conn(0x7f1ae00680a0 msgr2=0x7f1ae00cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.461+0000 7f1ae49ff700 1 -- 172.21.15.27:0/110972476 shutdown_connections 2024-01-11T08:26:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.461+0000 7f1ae49ff700 1 -- 172.21.15.27:0/110972476 wait complete. 2024-01-11T08:26:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.462+0000 7f1ae49ff700 1 Processor -- start 2024-01-11T08:26:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.462+0000 7f1ae49ff700 1 -- start start 2024-01-11T08:26:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.463+0000 7f1ae49ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ae0069540 0x7f1ae00cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.463+0000 7f1ae49ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae0157410 0x7f1ae0146d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.463+0000 7f1ae49ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ae01577e0 0x7f1ae016a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.463+0000 7f1ae49ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae00cb650 con 0x7f1ae0157410 2024-01-11T08:26:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.463+0000 7f1ae49ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae0147240 con 0x7f1ae01577e0 2024-01-11T08:26:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.463+0000 7f1ae49ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae016a710 con 0x7f1ae0069540 2024-01-11T08:26:03.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.464+0000 7f1addd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae0157410 0x7f1ae0146d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.464+0000 7f1ade59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ae0069540 0x7f1ae00cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.464+0000 7f1addd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae0157410 0x7f1ae0146d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35558/0 (socket says 172.21.15.27:35558) 2024-01-11T08:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.464+0000 7f1addd9b700 1 -- 172.21.15.27:0/3872926772 learned_addr learned my addr 172.21.15.27:0/3872926772 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.464+0000 7f1aded9d700 1 --2- 172.21.15.27:0/3872926772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ae01577e0 0x7f1ae016a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.464+0000 7f1addd9b700 1 -- 172.21.15.27:0/3872926772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ae0069540 msgr2=0x7f1ae00cb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.464+0000 7f1addd9b700 1 --2- 172.21.15.27:0/3872926772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ae0069540 0x7f1ae00cb110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.464+0000 7f1addd9b700 1 -- 172.21.15.27:0/3872926772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ae01577e0 msgr2=0x7f1ae016a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.464+0000 7f1addd9b700 1 --2- 172.21.15.27:0/3872926772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ae01577e0 0x7f1ae016a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.464+0000 7f1addd9b700 1 -- 172.21.15.27:0/3872926772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad0009480 con 0x7f1ae0157410 2024-01-11T08:26:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.464+0000 7f1addd9b700 1 --2- 172.21.15.27:0/3872926772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae0157410 0x7f1ae0146d00 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f1ad400e930 tx=0x7f1ad40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.465+0000 7f1ace7fc700 1 -- 172.21.15.27:0/3872926772 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4013680 con 0x7f1ae0157410 2024-01-11T08:26:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.465+0000 7f1ace7fc700 1 -- 172.21.15.27:0/3872926772 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad40137e0 con 0x7f1ae0157410 2024-01-11T08:26:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.465+0000 7f1ae49ff700 1 -- 172.21.15.27:0/3872926772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae00cac80 con 0x7f1ae0157410 2024-01-11T08:26:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.465+0000 7f1ace7fc700 1 -- 172.21.15.27:0/3872926772 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4013ba0 con 0x7f1ae0157410 2024-01-11T08:26:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.465+0000 7f1ae49ff700 1 -- 172.21.15.27:0/3872926772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae00ca070 con 0x7f1ae0157410 2024-01-11T08:26:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.466+0000 7f1ae49ff700 1 -- 172.21.15.27:0/3872926772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae00cae10 con 0x7f1ae0157410 2024-01-11T08:26:03.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.467+0000 7f1ace7fc700 1 -- 172.21.15.27:0/3872926772 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1ad40123d0 con 0x7f1ae0157410 2024-01-11T08:26:03.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.468+0000 7f1ace7fc700 1 --2- 172.21.15.27:0/3872926772 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f1ac0066f30 0x7f1ac00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:03.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.468+0000 7f1ace7fc700 1 -- 172.21.15.27:0/3872926772 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f1ad4084390 con 0x7f1ae0157410 2024-01-11T08:26:03.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.468+0000 7f1ade59c700 1 --2- 172.21.15.27:0/3872926772 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f1ac0066f30 0x7f1ac00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:03.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.469+0000 7f1ade59c700 1 --2- 172.21.15.27:0/3872926772 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f1ac0066f30 0x7f1ac00693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0009260 tx=0x7f1ad0016000 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:03.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.472+0000 7f1ace7fc700 1 -- 172.21.15.27:0/3872926772 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1ad4051540 con 0x7f1ae0157410 2024-01-11T08:26:03.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:03.719+0000 7f1ae49ff700 1 -- 172.21.15.27:0/3872926772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1ae004c3c0 con 0x7f1ae0157410 2024-01-11T08:26:04.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.238+0000 7f1ace7fc700 1 -- 172.21.15.27:0/3872926772 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1ad4054b60 con 0x7f1ae0157410 2024-01-11T08:26:04.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.241+0000 7f1ae49ff700 1 -- 172.21.15.27:0/3872926772 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f1ac0066f30 msgr2=0x7f1ac00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:04.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.241+0000 7f1ae49ff700 1 --2- 172.21.15.27:0/3872926772 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f1ac0066f30 0x7f1ac00693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0009260 tx=0x7f1ad0016000 comp rx=0 tx=0).stop 2024-01-11T08:26:04.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.241+0000 7f1ae49ff700 1 -- 172.21.15.27:0/3872926772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae0157410 msgr2=0x7f1ae0146d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:04.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.241+0000 7f1ae49ff700 1 --2- 172.21.15.27:0/3872926772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae0157410 0x7f1ae0146d00 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f1ad400e930 tx=0x7f1ad40091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.241+0000 7f1ae49ff700 1 -- 172.21.15.27:0/3872926772 shutdown_connections 2024-01-11T08:26:04.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.241+0000 7f1ae49ff700 1 --2- 172.21.15.27:0/3872926772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ae01577e0 0x7f1ae016a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.241+0000 7f1ae49ff700 1 --2- 172.21.15.27:0/3872926772 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f1ac0066f30 0x7f1ac00693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.241+0000 7f1ae49ff700 1 --2- 172.21.15.27:0/3872926772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ae0069540 0x7f1ae00cb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.241+0000 7f1ae49ff700 1 --2- 172.21.15.27:0/3872926772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae0157410 0x7f1ae0146d00 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.241+0000 7f1ae49ff700 1 -- 172.21.15.27:0/3872926772 >> 172.21.15.27:0/3872926772 conn(0x7f1ae00680a0 msgr2=0x7f1ae015cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:04.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.241+0000 7f1ae49ff700 1 -- 172.21.15.27:0/3872926772 shutdown_connections 2024-01-11T08:26:04.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.241+0000 7f1ae49ff700 1 -- 172.21.15.27:0/3872926772 wait complete. 2024-01-11T08:26:04.261 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:26:04.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.463+0000 7fdaa87ff700 1 Processor -- start 2024-01-11T08:26:04.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.464+0000 7fdaa87ff700 1 -- start start 2024-01-11T08:26:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.465+0000 7fdaa87ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdaa00ca7b0 0x7fdaa00cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.465+0000 7fdaa87ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdaa0159e80 0x7fdaa015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.465+0000 7fdaa87ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa015d640 con 0x7fdaa00ca3e0 2024-01-11T08:26:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.465+0000 7fdaa87ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaa015ae70 con 0x7fdaa00ca7b0 2024-01-11T08:26:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.465+0000 7fdaa87ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa00cc4f0 con 0x7fdaa0159e80 2024-01-11T08:26:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.465+0000 7fdaa659b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdaa00ca3e0 0x7fdaa00c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58420/0 (socket says 172.21.15.27:58420) 2024-01-11T08:26:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.465+0000 7fdaa659b700 1 -- 172.21.15.27:0/2257133079 learned_addr learned my addr 172.21.15.27:0/2257133079 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.465+0000 7fdaa6d9c700 1 --2- 172.21.15.27:0/2257133079 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdaa0159e80 0x7fdaa015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.465+0000 7fdaa5d9a700 1 --2- 172.21.15.27:0/2257133079 >> v2:172.21.15.77:3300/0 conn(0x7fdaa00ca7b0 0x7fdaa00cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.465+0000 7fdaa5599700 1 -- 172.21.15.27:0/2257133079 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837539808 0 0) 0x7fdaa015d640 con 0x7fdaa00ca3e0 2024-01-11T08:26:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.466+0000 7fdaa5599700 1 -- 172.21.15.27:0/2257133079 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda8c003570 con 0x7fdaa00ca3e0 2024-01-11T08:26:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.466+0000 7fdaa5599700 1 -- 172.21.15.27:0/2257133079 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1209372967 0 0) 0x7fda8c003570 con 0x7fdaa00ca3e0 2024-01-11T08:26:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.466+0000 7fdaa5599700 1 -- 172.21.15.27:0/2257133079 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdaa0159e80 msgr2=0x7fdaa015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.466+0000 7fdaa5599700 1 --2- 172.21.15.27:0/2257133079 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdaa0159e80 0x7fdaa015dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.466+0000 7fdaa5599700 1 -- 172.21.15.27:0/2257133079 >> v2:172.21.15.77:3300/0 conn(0x7fdaa00ca7b0 msgr2=0x7fdaa00cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.466+0000 7fdaa5599700 1 --2- 172.21.15.27:0/2257133079 >> v2:172.21.15.77:3300/0 conn(0x7fdaa00ca7b0 0x7fdaa00cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.466+0000 7fdaa5599700 1 -- 172.21.15.27:0/2257133079 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa015a5d0 con 0x7fdaa00ca3e0 2024-01-11T08:26:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.466+0000 7fdaa5599700 1 -- 172.21.15.27:0/2257133079 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fda9c001470 con 0x7fdaa00ca3e0 2024-01-11T08:26:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.466+0000 7fdaa5599700 1 -- 172.21.15.27:0/2257133079 >> v1:172.21.15.27:6789/0 conn(0x7fdaa00ca3e0 legacy=0x7fdaa00c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.466+0000 7fdaa5599700 1 --2- 172.21.15.27:0/2257133079 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fda8c003a90 0x7fda8c003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.466+0000 7fdaa5599700 1 --2- 172.21.15.27:0/2257133079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda8c004810 0x7fda8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.466+0000 7fdaa5599700 1 --2- 172.21.15.27:0/2257133079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda8c0051c0 0x7fda8c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.466+0000 7fdaa5599700 1 -- 172.21.15.27:0/2257133079 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa015ae70 con 0x7fda8c0051c0 2024-01-11T08:26:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.466+0000 7fdaa5599700 1 -- 172.21.15.27:0/2257133079 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa00cc4f0 con 0x7fda8c004810 2024-01-11T08:26:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.466+0000 7fdaa5599700 1 -- 172.21.15.27:0/2257133079 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c00bfd0 con 0x7fda8c003a90 2024-01-11T08:26:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.466+0000 7fdaa6d9c700 1 --2- 172.21.15.27:0/2257133079 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdaa0159e80 0x7fdaa015dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.466+0000 7fdaa659b700 1 --2- 172.21.15.27:0/2257133079 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fda8c003a90 0x7fda8c003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.466+0000 7fdaa6d9c700 1 --2- 172.21.15.27:0/2257133079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda8c0051c0 0x7fda8c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.466+0000 7fdaa5d9a700 1 --2- 172.21.15.27:0/2257133079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda8c004810 0x7fda8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.467+0000 7fdaa6d9c700 1 -- 172.21.15.27:0/2257133079 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fda8c003a90 msgr2=0x7fda8c003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.467+0000 7fdaa6d9c700 1 --2- 172.21.15.27:0/2257133079 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fda8c003a90 0x7fda8c003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.467+0000 7fdaa6d9c700 1 -- 172.21.15.27:0/2257133079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda8c004810 msgr2=0x7fda8c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.467+0000 7fdaa6d9c700 1 --2- 172.21.15.27:0/2257133079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda8c004810 0x7fda8c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.467+0000 7fdaa6d9c700 1 -- 172.21.15.27:0/2257133079 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8c003570 con 0x7fda8c0051c0 2024-01-11T08:26:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.467+0000 7fdaa6d9c700 1 --2- 172.21.15.27:0/2257133079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda8c0051c0 0x7fda8c00ba60 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fdaa015a250 tx=0x7fda9800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74dabcbab5cfff2a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.467+0000 7fdaa5599700 1 -- 172.21.15.27:0/2257133079 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9800ea40 con 0x7fda8c0051c0 2024-01-11T08:26:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.467+0000 7fdaa5599700 1 -- 172.21.15.27:0/2257133079 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9800f040 con 0x7fda8c0051c0 2024-01-11T08:26:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.467+0000 7fdaa5599700 1 -- 172.21.15.27:0/2257133079 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9800edd0 con 0x7fda8c0051c0 2024-01-11T08:26:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.468+0000 7fdaa87ff700 1 -- 172.21.15.27:0/2257133079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda8c0051c0 msgr2=0x7fda8c00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.468+0000 7fdaa87ff700 1 --2- 172.21.15.27:0/2257133079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda8c0051c0 0x7fda8c00ba60 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fdaa015a250 tx=0x7fda9800b040 comp rx=0 tx=0).stop 2024-01-11T08:26:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.468+0000 7fdaa6d9c700 1 -- 172.21.15.27:0/2257133079 reap_dead start 2024-01-11T08:26:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.468+0000 7fdaa87ff700 1 -- 172.21.15.27:0/2257133079 shutdown_connections 2024-01-11T08:26:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.468+0000 7fdaa87ff700 1 -- 172.21.15.27:0/2257133079 >> 172.21.15.27:0/2257133079 conn(0x7fdaa00ce3e0 msgr2=0x7fdaa00ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.468+0000 7fdaa87ff700 1 -- 172.21.15.27:0/2257133079 shutdown_connections 2024-01-11T08:26:04.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.468+0000 7fdaa87ff700 1 -- 172.21.15.27:0/2257133079 wait complete. 2024-01-11T08:26:04.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.469+0000 7fdaa87ff700 1 Processor -- start 2024-01-11T08:26:04.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.470+0000 7fdaa87ff700 1 -- start start 2024-01-11T08:26:04.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.470+0000 7fdaa87ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda8c0051c0 0x7fdaa0154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:04.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.470+0000 7fdaa87ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdaa00c7390 0x7fdaa00c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:04.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.471+0000 7fdaa87ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaa01514e0 0x7fdaa00693f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:04.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.471+0000 7fdaa87ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa01595c0 con 0x7fda8c0051c0 2024-01-11T08:26:04.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.471+0000 7fdaa87ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa00c8540 con 0x7fdaa01514e0 2024-01-11T08:26:04.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.471+0000 7fdaa87ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa0154640 con 0x7fdaa00c7390 2024-01-11T08:26:04.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.471+0000 7fdaa659b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda8c0051c0 0x7fdaa0154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:04.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.471+0000 7fdaa659b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda8c0051c0 0x7fdaa0154100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50810/0 (socket says 172.21.15.27:50810) 2024-01-11T08:26:04.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.471+0000 7fdaa659b700 1 -- 172.21.15.27:0/2383736694 learned_addr learned my addr 172.21.15.27:0/2383736694 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:04.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.471+0000 7fdaa659b700 1 -- 172.21.15.27:0/2383736694 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdaa00c7390 msgr2=0x7fdaa00c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:04.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.471+0000 7fdaa659b700 1 --2- 172.21.15.27:0/2383736694 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdaa00c7390 0x7fdaa00c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.471+0000 7fdaa6d9c700 1 --2- 172.21.15.27:0/2383736694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaa01514e0 0x7fdaa00693f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:04.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.471+0000 7fdaa659b700 1 -- 172.21.15.27:0/2383736694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaa01514e0 msgr2=0x7fdaa00693f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:04.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.471+0000 7fdaa659b700 1 --2- 172.21.15.27:0/2383736694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaa01514e0 0x7fdaa00693f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.471+0000 7fdaa659b700 1 -- 172.21.15.27:0/2383736694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa00c7930 con 0x7fda8c0051c0 2024-01-11T08:26:04.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.471+0000 7fdaa6d9c700 1 --2- 172.21.15.27:0/2383736694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaa01514e0 0x7fdaa00693f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:04.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.471+0000 7fdaa659b700 1 --2- 172.21.15.27:0/2383736694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda8c0051c0 0x7fdaa0154100 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fda9c0073c0 tx=0x7fda9c001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:04.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.472+0000 7fda967fc700 1 -- 172.21.15.27:0/2383736694 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9c018030 con 0x7fda8c0051c0 2024-01-11T08:26:04.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.472+0000 7fda967fc700 1 -- 172.21.15.27:0/2383736694 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9c00f8d0 con 0x7fda8c0051c0 2024-01-11T08:26:04.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.472+0000 7fdaa87ff700 1 -- 172.21.15.27:0/2383736694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa00c7c10 con 0x7fda8c0051c0 2024-01-11T08:26:04.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.472+0000 7fda967fc700 1 -- 172.21.15.27:0/2383736694 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9c00fc90 con 0x7fda8c0051c0 2024-01-11T08:26:04.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.472+0000 7fdaa87ff700 1 -- 172.21.15.27:0/2383736694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa00c81d0 con 0x7fda8c0051c0 2024-01-11T08:26:04.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.473+0000 7fdaa87ff700 1 -- 172.21.15.27:0/2383736694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa00718d0 con 0x7fda8c0051c0 2024-01-11T08:26:04.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.474+0000 7fda967fc700 1 -- 172.21.15.27:0/2383736694 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fda9c01b070 con 0x7fda8c0051c0 2024-01-11T08:26:04.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.475+0000 7fda967fc700 1 --2- 172.21.15.27:0/2383736694 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fda84066f30 0x7fda840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:04.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.475+0000 7fda967fc700 1 -- 172.21.15.27:0/2383736694 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5733+0+0 (secure 0 0 0) 0x7fda9c019070 con 0x7fda8c0051c0 2024-01-11T08:26:04.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.475+0000 7fdaa5d9a700 1 --2- 172.21.15.27:0/2383736694 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fda84066f30 0x7fda840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:04.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.476+0000 7fdaa5d9a700 1 --2- 172.21.15.27:0/2383736694 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fda84066f30 0x7fda840693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fda90009170 tx=0x7fda900091a0 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.479+0000 7fda967fc700 1 -- 172.21.15.27:0/2383736694 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fda9c0517d0 con 0x7fda8c0051c0 2024-01-11T08:26:04.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.728+0000 7fdaa87ff700 1 -- 172.21.15.27:0/2383736694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdaa015a8b0 con 0x7fda8c0051c0 2024-01-11T08:26:04.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.728+0000 7fda967fc700 1 -- 172.21.15.27:0/2383736694 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4596 (secure 0 0 0) 0x7fda9c00fdf0 con 0x7fda8c0051c0 2024-01-11T08:26:04.728 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:04.729 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5945,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1366930493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1366930493},{"type":"v1","addr":"172.21.15.77:6837","nonce":1366930493}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/516828593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":516828593},{"type":"v1","addr":"172.21.15.27:6835","nonce":516828593}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6624,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1462523541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1462523541},{"type":"v1","addr":"172.21.15.27:6834","nonce":1462523541}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:26:00.710104+0000","modified":"2024-01-11T08:26:04.230562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6627":{"gid":6627,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6835/3498364963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3498364963},{"type":"v1","addr":"172.21.15.77:6835","nonce":3498364963}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2024-01-11T08:26:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.731+0000 7fdaa87ff700 1 -- 172.21.15.27:0/2383736694 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fda84066f30 msgr2=0x7fda840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.731+0000 7fdaa87ff700 1 --2- 172.21.15.27:0/2383736694 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fda84066f30 0x7fda840693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fda90009170 tx=0x7fda900091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.731+0000 7fdaa87ff700 1 -- 172.21.15.27:0/2383736694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda8c0051c0 msgr2=0x7fdaa0154100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.731+0000 7fdaa87ff700 1 --2- 172.21.15.27:0/2383736694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda8c0051c0 0x7fdaa0154100 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fda9c0073c0 tx=0x7fda9c001e50 comp rx=0 tx=0).stop 2024-01-11T08:26:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.731+0000 7fdaa87ff700 1 -- 172.21.15.27:0/2383736694 shutdown_connections 2024-01-11T08:26:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.731+0000 7fdaa87ff700 1 --2- 172.21.15.27:0/2383736694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaa01514e0 0x7fdaa00693f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.731+0000 7fdaa87ff700 1 --2- 172.21.15.27:0/2383736694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda8c0051c0 0x7fdaa0154100 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.731+0000 7fdaa87ff700 1 --2- 172.21.15.27:0/2383736694 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fda84066f30 0x7fda840693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.731+0000 7fdaa87ff700 1 --2- 172.21.15.27:0/2383736694 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdaa00c7390 0x7fdaa00c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.731+0000 7fdaa87ff700 1 -- 172.21.15.27:0/2383736694 >> 172.21.15.27:0/2383736694 conn(0x7fdaa00ce3e0 msgr2=0x7fdaa0151d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.731+0000 7fdaa87ff700 1 -- 172.21.15.27:0/2383736694 shutdown_connections 2024-01-11T08:26:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.731+0000 7fdaa87ff700 1 -- 172.21.15.27:0/2383736694 wait complete. 2024-01-11T08:26:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 42 2024-01-11T08:26:04.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:26:04.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.949+0000 7f93cebff700 1 Processor -- start 2024-01-11T08:26:04.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.950+0000 7f93cebff700 1 -- start start 2024-01-11T08:26:04.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.951+0000 7f93cebff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93c80b03e0 0x7f93c80b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:04.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.951+0000 7f93cebff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f93c80b07b0 0x7f93c80b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:04.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.951+0000 7f93cebff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93c80cd730 con 0x7f93c8159200 2024-01-11T08:26:04.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.951+0000 7f93cebff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c8159d50 con 0x7f93c80b07b0 2024-01-11T08:26:04.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.951+0000 7f93cebff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c80ccb20 con 0x7f93c80b03e0 2024-01-11T08:26:04.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.951+0000 7f93c7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f93c80b07b0 0x7f93c80b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:04.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.951+0000 7f93c7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f93c80b07b0 0x7f93c80b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50820/0 (socket says 172.21.15.27:50820) 2024-01-11T08:26:04.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.951+0000 7f93c7fff700 1 -- 172.21.15.27:0/2660395010 learned_addr learned my addr 172.21.15.27:0/2660395010 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.951+0000 7f93cc99b700 1 --2- 172.21.15.27:0/2660395010 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93c80b03e0 0x7f93c80b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.952+0000 7f93c7fff700 1 -- 172.21.15.27:0/2660395010 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93c80b03e0 msgr2=0x7f93c80b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.952+0000 7f93c7fff700 1 --2- 172.21.15.27:0/2660395010 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93c80b03e0 0x7f93c80b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.952+0000 7f93c7fff700 1 -- 172.21.15.27:0/2660395010 >> v1:172.21.15.77:6789/0 conn(0x7f93c8159200 legacy=0x7f93c8159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.952+0000 7f93c7fff700 1 -- 172.21.15.27:0/2660395010 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c80cce00 con 0x7f93c80b07b0 2024-01-11T08:26:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.952+0000 7f93c7fff700 1 --2- 172.21.15.27:0/2660395010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93c80b07b0 0x7f93c80b29c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f93b40099b0 tx=0x7f93b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7cb027ca3338b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.952+0000 7f93c77fe700 1 -- 172.21.15.27:0/2660395010 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b400b860 con 0x7f93c80b07b0 2024-01-11T08:26:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.952+0000 7f93c77fe700 1 -- 172.21.15.27:0/2660395010 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b400b9c0 con 0x7f93c80b07b0 2024-01-11T08:26:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.952+0000 7f93c77fe700 1 -- 172.21.15.27:0/2660395010 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b400bd60 con 0x7f93c80b07b0 2024-01-11T08:26:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.953+0000 7f93cebff700 1 -- 172.21.15.27:0/2660395010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93c80b07b0 msgr2=0x7f93c80b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.953+0000 7f93cebff700 1 --2- 172.21.15.27:0/2660395010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93c80b07b0 0x7f93c80b29c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f93b40099b0 tx=0x7f93b4009250 comp rx=0 tx=0).stop 2024-01-11T08:26:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.953+0000 7f93cebff700 1 -- 172.21.15.27:0/2660395010 shutdown_connections 2024-01-11T08:26:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.953+0000 7f93cebff700 1 --2- 172.21.15.27:0/2660395010 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93c80b03e0 0x7f93c80b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.953+0000 7f93cebff700 1 --2- 172.21.15.27:0/2660395010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93c80b07b0 0x7f93c80b29c0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.953+0000 7f93cebff700 1 -- 172.21.15.27:0/2660395010 >> 172.21.15.27:0/2660395010 conn(0x7f93c806ed40 msgr2=0x7f93c806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.953+0000 7f93cebff700 1 -- 172.21.15.27:0/2660395010 shutdown_connections 2024-01-11T08:26:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.953+0000 7f93cebff700 1 -- 172.21.15.27:0/2660395010 wait complete. 2024-01-11T08:26:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.954+0000 7f93cebff700 1 Processor -- start 2024-01-11T08:26:04.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.955+0000 7f93cebff700 1 -- start start 2024-01-11T08:26:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.955+0000 7f93cebff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93c80b03e0 0x7f93c8157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.955+0000 7f93cebff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93c80b07b0 0x7f93c80c4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.955+0000 7f93cebff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c8159200 0x7f93c8169fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.955+0000 7f93cebff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c81583a0 con 0x7f93c80b07b0 2024-01-11T08:26:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.955+0000 7f93cebff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c80c4e40 con 0x7f93c8159200 2024-01-11T08:26:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.955+0000 7f93cebff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c816a4f0 con 0x7f93c80b03e0 2024-01-11T08:26:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.956+0000 7f93c7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93c80b07b0 0x7f93c80c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.956+0000 7f93c7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93c80b07b0 0x7f93c80c4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50824/0 (socket says 172.21.15.27:50824) 2024-01-11T08:26:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.956+0000 7f93c7fff700 1 -- 172.21.15.27:0/399212916 learned_addr learned my addr 172.21.15.27:0/399212916 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.956+0000 7f93cc99b700 1 --2- 172.21.15.27:0/399212916 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93c80b03e0 0x7f93c8157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.956+0000 7f93cd19c700 1 --2- 172.21.15.27:0/399212916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c8159200 0x7f93c8169fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.956+0000 7f93c7fff700 1 -- 172.21.15.27:0/399212916 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93c80b03e0 msgr2=0x7f93c8157e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.956+0000 7f93c7fff700 1 --2- 172.21.15.27:0/399212916 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93c80b03e0 0x7f93c8157e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.956+0000 7f93c7fff700 1 -- 172.21.15.27:0/399212916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c8159200 msgr2=0x7f93c8169fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.956+0000 7f93c7fff700 1 --2- 172.21.15.27:0/399212916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c8159200 0x7f93c8169fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.956+0000 7f93c7fff700 1 -- 172.21.15.27:0/399212916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b40094f0 con 0x7f93c80b07b0 2024-01-11T08:26:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.956+0000 7f93c7fff700 1 --2- 172.21.15.27:0/399212916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93c80b07b0 0x7f93c80c4900 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f93b400b3c0 tx=0x7f93b4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.957+0000 7f93c4ff9700 1 -- 172.21.15.27:0/399212916 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b4011df0 con 0x7f93c80b07b0 2024-01-11T08:26:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.957+0000 7f93c4ff9700 1 -- 172.21.15.27:0/399212916 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b4024440 con 0x7f93c80b07b0 2024-01-11T08:26:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.957+0000 7f93c4ff9700 1 -- 172.21.15.27:0/399212916 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b4024800 con 0x7f93c80b07b0 2024-01-11T08:26:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.957+0000 7f93cebff700 1 -- 172.21.15.27:0/399212916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c816afd0 con 0x7f93c80b07b0 2024-01-11T08:26:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.957+0000 7f93cebff700 1 -- 172.21.15.27:0/399212916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c80c59c0 con 0x7f93c80b07b0 2024-01-11T08:26:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.958+0000 7f93cebff700 1 -- 172.21.15.27:0/399212916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c80c5b50 con 0x7f93c80b07b0 2024-01-11T08:26:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.959+0000 7f93c4ff9700 1 -- 172.21.15.27:0/399212916 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f93b401a420 con 0x7f93c80b07b0 2024-01-11T08:26:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.960+0000 7f93c4ff9700 1 --2- 172.21.15.27:0/399212916 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f93ac066f30 0x7f93ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.960+0000 7f93c4ff9700 1 -- 172.21.15.27:0/399212916 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f93b4086470 con 0x7f93c80b07b0 2024-01-11T08:26:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.960+0000 7f93cc99b700 1 --2- 172.21.15.27:0/399212916 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f93ac066f30 0x7f93ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:04.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.961+0000 7f93cc99b700 1 --2- 172.21.15.27:0/399212916 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f93ac066f30 0x7f93ac0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f93bc006fd0 tx=0x7f93bc008040 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:04.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:04.964+0000 7f93c4ff9700 1 -- 172.21.15.27:0/399212916 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f93b4053620 con 0x7f93c80b07b0 2024-01-11T08:26:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.172+0000 7f93cebff700 1 -- 172.21.15.27:0/399212916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93c80cd0e0 con 0x7f93c80b07b0 2024-01-11T08:26:05.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.173+0000 7f93c4ff9700 1 -- 172.21.15.27:0/399212916 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+15348 (secure 0 0 0) 0x7f93b4056c40 con 0x7f93c80b07b0 2024-01-11T08:26:05.173 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:05.173 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":88,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:26:02.215725+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:25:58.523510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-11T08:25:59.706779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:26:05.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.175+0000 7f93cebff700 1 -- 172.21.15.27:0/399212916 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f93ac066f30 msgr2=0x7f93ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:05.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.175+0000 7f93cebff700 1 --2- 172.21.15.27:0/399212916 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f93ac066f30 0x7f93ac0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f93bc006fd0 tx=0x7f93bc008040 comp rx=0 tx=0).stop 2024-01-11T08:26:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.175+0000 7f93cebff700 1 -- 172.21.15.27:0/399212916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93c80b07b0 msgr2=0x7f93c80c4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.175+0000 7f93cebff700 1 --2- 172.21.15.27:0/399212916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93c80b07b0 0x7f93c80c4900 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f93b400b3c0 tx=0x7f93b4017000 comp rx=0 tx=0).stop 2024-01-11T08:26:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.175+0000 7f93cebff700 1 -- 172.21.15.27:0/399212916 shutdown_connections 2024-01-11T08:26:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.175+0000 7f93cebff700 1 --2- 172.21.15.27:0/399212916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c8159200 0x7f93c8169fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.175+0000 7f93cebff700 1 --2- 172.21.15.27:0/399212916 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f93ac066f30 0x7f93ac0693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.175+0000 7f93cebff700 1 --2- 172.21.15.27:0/399212916 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f93c80b03e0 0x7f93c8157e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.175+0000 7f93cebff700 1 --2- 172.21.15.27:0/399212916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93c80b07b0 0x7f93c80c4900 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.175+0000 7f93cebff700 1 -- 172.21.15.27:0/399212916 >> 172.21.15.27:0/399212916 conn(0x7f93c806ed40 msgr2=0x7f93c80b46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.175+0000 7f93cebff700 1 -- 172.21.15.27:0/399212916 shutdown_connections 2024-01-11T08:26:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.175+0000 7f93cebff700 1 -- 172.21.15.27:0/399212916 wait complete. 2024-01-11T08:26:05.194 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-11T08:26:05.194 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:26:05.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.393+0000 7fecac7ff700 1 Processor -- start 2024-01-11T08:26:05.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.394+0000 7fecac7ff700 1 -- start start 2024-01-11T08:26:05.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.395+0000 7fecac7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feca41575e0 0x7feca4158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:05.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.395+0000 7fecac7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feca40b19d0 0x7feca4159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:05.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.395+0000 7fecac7ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca4158b60 con 0x7feca406b610 2024-01-11T08:26:05.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.395+0000 7fecac7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca40cb5e0 con 0x7feca41575e0 2024-01-11T08:26:05.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.395+0000 7fecac7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca41599b0 con 0x7feca40b19d0 2024-01-11T08:26:05.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.395+0000 7fecaad9c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feca40b19d0 0x7feca4159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:05.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.395+0000 7fecaa59b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feca41575e0 0x7feca4158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:05.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.395+0000 7fecaad9c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feca40b19d0 0x7feca4159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:45110/0 (socket says 172.21.15.27:45110) 2024-01-11T08:26:05.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.395+0000 7fecaa59b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feca41575e0 0x7feca4158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:44816/0 (socket says 172.21.15.27:44816) 2024-01-11T08:26:05.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.395+0000 7fecaad9c700 1 -- 172.21.15.27:0/2853192121 learned_addr learned my addr 172.21.15.27:0/2853192121 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:05.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.396+0000 7fecaad9c700 1 -- 172.21.15.27:0/2853192121 >> v2:172.21.15.77:3300/0 conn(0x7feca41575e0 msgr2=0x7feca4158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:05.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.396+0000 7fecaad9c700 1 --2- 172.21.15.27:0/2853192121 >> v2:172.21.15.77:3300/0 conn(0x7feca41575e0 0x7feca4158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:05.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.396+0000 7fecaad9c700 1 -- 172.21.15.27:0/2853192121 >> v1:172.21.15.77:6789/0 conn(0x7feca406b610 legacy=0x7feca40b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:05.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.396+0000 7fecaad9c700 1 -- 172.21.15.27:0/2853192121 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca40c0f90 con 0x7feca40b19d0 2024-01-11T08:26:05.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.396+0000 7feca9599700 1 -- 172.21.15.27:0/2853192121 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972237523 0 0) 0x7fec94001110 con 0x7feca406b610 2024-01-11T08:26:05.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.396+0000 7fecaa59b700 1 --2- 172.21.15.27:0/2853192121 >> v2:172.21.15.77:3300/0 conn(0x7feca41575e0 0x7feca4158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:05.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.396+0000 7fecaad9c700 1 --2- 172.21.15.27:0/2853192121 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feca40b19d0 0x7feca4159e40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fec9c009f20 tx=0x7fec9c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e926085fdb4b0b64 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:05.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.396+0000 7feca9599700 1 -- 172.21.15.27:0/2853192121 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c00e070 con 0x7feca40b19d0 2024-01-11T08:26:05.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.397+0000 7feca9599700 1 -- 172.21.15.27:0/2853192121 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9c00b9d0 con 0x7feca40b19d0 2024-01-11T08:26:05.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.397+0000 7feca9599700 1 -- 172.21.15.27:0/2853192121 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c0092e0 con 0x7feca40b19d0 2024-01-11T08:26:05.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.397+0000 7fecac7ff700 1 -- 172.21.15.27:0/2853192121 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feca40b19d0 msgr2=0x7feca4159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:05.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.397+0000 7fecac7ff700 1 --2- 172.21.15.27:0/2853192121 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feca40b19d0 0x7feca4159e40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fec9c009f20 tx=0x7fec9c009840 comp rx=0 tx=0).stop 2024-01-11T08:26:05.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.397+0000 7fecac7ff700 1 -- 172.21.15.27:0/2853192121 shutdown_connections 2024-01-11T08:26:05.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.397+0000 7fecac7ff700 1 --2- 172.21.15.27:0/2853192121 >> v2:172.21.15.77:3300/0 conn(0x7feca41575e0 0x7feca4158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:05.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.397+0000 7fecac7ff700 1 --2- 172.21.15.27:0/2853192121 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feca40b19d0 0x7feca4159e40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:05.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.397+0000 7fecac7ff700 1 -- 172.21.15.27:0/2853192121 >> 172.21.15.27:0/2853192121 conn(0x7feca40680e0 msgr2=0x7feca4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:05.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.397+0000 7fecac7ff700 1 -- 172.21.15.27:0/2853192121 shutdown_connections 2024-01-11T08:26:05.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.397+0000 7fecac7ff700 1 -- 172.21.15.27:0/2853192121 wait complete. 2024-01-11T08:26:05.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.399+0000 7fecac7ff700 1 Processor -- start 2024-01-11T08:26:05.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.399+0000 7fecac7ff700 1 -- start start 2024-01-11T08:26:05.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.400+0000 7fecac7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca406b610 0x7feca4148970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:05.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.400+0000 7fecac7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feca40b19d0 0x7feca40c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:05.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.400+0000 7fecac7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feca41575e0 0x7feca40c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.400+0000 7fecac7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca40c8270 con 0x7feca406b610 2024-01-11T08:26:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.400+0000 7fecac7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca4148eb0 con 0x7feca40b19d0 2024-01-11T08:26:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.400+0000 7fecac7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca41463c0 con 0x7feca41575e0 2024-01-11T08:26:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.400+0000 7fecaa59b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca406b610 0x7feca4148970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.400+0000 7fecaa59b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca406b610 0x7feca4148970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50826/0 (socket says 172.21.15.27:50826) 2024-01-11T08:26:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.400+0000 7fecaa59b700 1 -- 172.21.15.27:0/1252622866 learned_addr learned my addr 172.21.15.27:0/1252622866 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.400+0000 7fecaad9c700 1 --2- 172.21.15.27:0/1252622866 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feca41575e0 0x7feca40c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.400+0000 7feca9d9a700 1 --2- 172.21.15.27:0/1252622866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feca40b19d0 0x7feca40c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.400+0000 7fecaa59b700 1 -- 172.21.15.27:0/1252622866 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feca41575e0 msgr2=0x7feca40c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.400+0000 7fecaa59b700 1 --2- 172.21.15.27:0/1252622866 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feca41575e0 0x7feca40c5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.400+0000 7fecaa59b700 1 -- 172.21.15.27:0/1252622866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feca40b19d0 msgr2=0x7feca40c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.400+0000 7fecaa59b700 1 --2- 172.21.15.27:0/1252622866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feca40b19d0 0x7feca40c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.400+0000 7fecaa59b700 1 -- 172.21.15.27:0/1252622866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca0009420 con 0x7feca406b610 2024-01-11T08:26:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.400+0000 7feca9d9a700 1 --2- 172.21.15.27:0/1252622866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feca40b19d0 0x7feca40c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.401+0000 7fecaad9c700 1 --2- 172.21.15.27:0/1252622866 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feca41575e0 0x7feca40c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.401+0000 7fecaa59b700 1 --2- 172.21.15.27:0/1252622866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca406b610 0x7feca4148970 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7feca000ec80 tx=0x7feca00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.401+0000 7fec9a7fc700 1 -- 172.21.15.27:0/1252622866 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca000c970 con 0x7feca406b610 2024-01-11T08:26:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.401+0000 7fec9a7fc700 1 -- 172.21.15.27:0/1252622866 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca000cad0 con 0x7feca406b610 2024-01-11T08:26:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.401+0000 7fec9a7fc700 1 -- 172.21.15.27:0/1252622866 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca000ce50 con 0x7feca406b610 2024-01-11T08:26:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.401+0000 7fecac7ff700 1 -- 172.21.15.27:0/1252622866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec9c009ae0 con 0x7feca406b610 2024-01-11T08:26:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.401+0000 7fecac7ff700 1 -- 172.21.15.27:0/1252622866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca4146980 con 0x7feca406b610 2024-01-11T08:26:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.402+0000 7fecac7ff700 1 -- 172.21.15.27:0/1252622866 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca41472b0 con 0x7feca406b610 2024-01-11T08:26:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.403+0000 7fec9a7fc700 1 -- 172.21.15.27:0/1252622866 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7feca0013680 con 0x7feca406b610 2024-01-11T08:26:05.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.404+0000 7fec9a7fc700 1 --2- 172.21.15.27:0/1252622866 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fec88066f30 0x7fec880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:05.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.404+0000 7fec9a7fc700 1 -- 172.21.15.27:0/1252622866 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5733+0+0 (secure 0 0 0) 0x7feca0084960 con 0x7feca406b610 2024-01-11T08:26:05.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.404+0000 7feca9d9a700 1 --2- 172.21.15.27:0/1252622866 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fec88066f30 0x7fec880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:05.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.405+0000 7feca9d9a700 1 --2- 172.21.15.27:0/1252622866 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fec88066f30 0x7fec880693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7feca40b0f90 tx=0x7fec94009040 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.408+0000 7fec9a7fc700 1 -- 172.21.15.27:0/1252622866 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feca0051b10 con 0x7feca406b610 2024-01-11T08:26:05.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.605+0000 7fecac7ff700 1 -- 172.21.15.27:0/1252622866 --> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feca40b20d0 con 0x7fec88066f30 2024-01-11T08:26:05.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.617+0000 7fec9a7fc700 1 -- 172.21.15.27:0/1252622866 <== mgr.5795 v2:172.21.15.77:6834/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289059 (secure 0 0 0) 0x7feca40b20d0 con 0x7fec88066f30 2024-01-11T08:26:05.620 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:05.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.621+0000 7fecac7ff700 1 -- 172.21.15.27:0/1252622866 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fec88066f30 msgr2=0x7fec880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:05.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.621+0000 7fecac7ff700 1 --2- 172.21.15.27:0/1252622866 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fec88066f30 0x7fec880693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7feca40b0f90 tx=0x7fec94009040 comp rx=0 tx=0).stop 2024-01-11T08:26:05.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.621+0000 7fecac7ff700 1 -- 172.21.15.27:0/1252622866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca406b610 msgr2=0x7feca4148970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:05.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.621+0000 7fecac7ff700 1 --2- 172.21.15.27:0/1252622866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca406b610 0x7feca4148970 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7feca000ec80 tx=0x7feca00091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:05.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.621+0000 7fecac7ff700 1 -- 172.21.15.27:0/1252622866 shutdown_connections 2024-01-11T08:26:05.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.621+0000 7fecac7ff700 1 --2- 172.21.15.27:0/1252622866 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fec88066f30 0x7fec880693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:05.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.621+0000 7fecac7ff700 1 --2- 172.21.15.27:0/1252622866 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feca41575e0 0x7feca40c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:05.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.621+0000 7fecac7ff700 1 --2- 172.21.15.27:0/1252622866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca406b610 0x7feca4148970 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:05.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.621+0000 7fecac7ff700 1 --2- 172.21.15.27:0/1252622866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feca40b19d0 0x7feca40c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:05.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.621+0000 7fecac7ff700 1 -- 172.21.15.27:0/1252622866 >> 172.21.15.27:0/1252622866 conn(0x7feca40680e0 msgr2=0x7feca4159e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:05.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.621+0000 7fecac7ff700 1 -- 172.21.15.27:0/1252622866 shutdown_connections 2024-01-11T08:26:05.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.621+0000 7fecac7ff700 1 -- 172.21.15.27:0/1252622866 wait complete. 2024-01-11T08:26:05.624 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:26:05.639 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-11T08:26:05.316497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":674,"num_read_kb":1321,"num_write":319,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2437052,"kb_used_data":9596,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752537668,"statfs":{"total":773094113280,"available":770598572032,"internally_reserved":0,"allocated":9826304,"data_stored":5142656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001391"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.238182+0000","last_change":"2024-01-11T08:26:01.235430+0000","last_active":"2024-01-11T08:26:01.238182+0000","last_peered":"2024-01-11T08:26:01.238182+0000","last_clean":"2024-01-11T08:26:01.238182+0000","last_became_active":"2024-01-11T08:26:01.231860+0000","last_became_peered":"2024-01-11T08:26:01.231860+0000","last_unstale":"2024-01-11T08:26:01.238182+0000","last_undegraded":"2024-01-11T08:26:01.238182+0000","last_fullsized":"2024-01-11T08:26:01.238182+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:55:56.568599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.251616+0000","last_change":"2024-01-11T08:26:01.250130+0000","last_active":"2024-01-11T08:26:01.251616+0000","last_peered":"2024-01-11T08:26:01.251616+0000","last_clean":"2024-01-11T08:26:01.251616+0000","last_became_active":"2024-01-11T08:26:01.249817+0000","last_became_peered":"2024-01-11T08:26:01.249817+0000","last_unstale":"2024-01-11T08:26:01.251616+0000","last_undegraded":"2024-01-11T08:26:01.251616+0000","last_fullsized":"2024-01-11T08:26:01.251616+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:14:56.858740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.229889+0000","last_change":"2024-01-11T08:26:01.227701+0000","last_active":"2024-01-11T08:26:01.229889+0000","last_peered":"2024-01-11T08:26:01.229889+0000","last_clean":"2024-01-11T08:26:01.229889+0000","last_became_active":"2024-01-11T08:26:01.227388+0000","last_became_peered":"2024-01-11T08:26:01.227388+0000","last_unstale":"2024-01-11T08:26:01.229889+0000","last_undegraded":"2024-01-11T08:26:01.229889+0000","last_fullsized":"2024-01-11T08:26:01.229889+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:53:16.623590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.243226+0000","last_change":"2024-01-11T08:26:01.240669+0000","last_active":"2024-01-11T08:26:01.243226+0000","last_peered":"2024-01-11T08:26:01.243226+0000","last_clean":"2024-01-11T08:26:01.243226+0000","last_became_active":"2024-01-11T08:26:01.238255+0000","last_became_peered":"2024-01-11T08:26:01.238255+0000","last_unstale":"2024-01-11T08:26:01.243226+0000","last_undegraded":"2024-01-11T08:26:01.243226+0000","last_fullsized":"2024-01-11T08:26:01.243226+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:26:24.873546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.229166+0000","last_change":"2024-01-11T08:26:01.226105+0000","last_active":"2024-01-11T08:26:01.229166+0000","last_peered":"2024-01-11T08:26:01.229166+0000","last_clean":"2024-01-11T08:26:01.229166+0000","last_became_active":"2024-01-11T08:26:01.225365+0000","last_became_peered":"2024-01-11T08:26:01.225365+0000","last_unstale":"2024-01-11T08:26:01.229166+0000","last_undegraded":"2024-01-11T08:26:01.229166+0000","last_fullsized":"2024-01-11T08:26:01.229166+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:37:35.515669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.248957+0000","last_change":"2024-01-11T08:26:01.245817+0000","last_active":"2024-01-11T08:26:01.248957+0000","last_peered":"2024-01-11T08:26:01.248957+0000","last_clean":"2024-01-11T08:26:01.248957+0000","last_became_active":"2024-01-11T08:26:01.245416+0000","last_became_peered":"2024-01-11T08:26:01.245416+0000","last_unstale":"2024-01-11T08:26:01.248957+0000","last_undegraded":"2024-01-11T08:26:01.248957+0000","last_fullsized":"2024-01-11T08:26:01.248957+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:51:02.992576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.243449+0000","last_change":"2024-01-11T08:26:01.236309+0000","last_active":"2024-01-11T08:26:01.243449+0000","last_peered":"2024-01-11T08:26:01.243449+0000","last_clean":"2024-01-11T08:26:01.243449+0000","last_became_active":"2024-01-11T08:26:01.236027+0000","last_became_peered":"2024-01-11T08:26:01.236027+0000","last_unstale":"2024-01-11T08:26:01.243449+0000","last_undegraded":"2024-01-11T08:26:01.243449+0000","last_fullsized":"2024-01-11T08:26:01.243449+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:00:08.280288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.236669+0000","last_change":"2024-01-11T08:26:01.234620+0000","last_active":"2024-01-11T08:26:01.236669+0000","last_peered":"2024-01-11T08:26:01.236669+0000","last_clean":"2024-01-11T08:26:01.236669+0000","last_became_active":"2024-01-11T08:26:01.233940+0000","last_became_peered":"2024-01-11T08:26:01.233940+0000","last_unstale":"2024-01-11T08:26:01.236669+0000","last_undegraded":"2024-01-11T08:26:01.236669+0000","last_fullsized":"2024-01-11T08:26:01.236669+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:20:06.654183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.230472+0000","last_change":"2024-01-11T08:26:01.225434+0000","last_active":"2024-01-11T08:26:01.230472+0000","last_peered":"2024-01-11T08:26:01.230472+0000","last_clean":"2024-01-11T08:26:01.230472+0000","last_became_active":"2024-01-11T08:26:01.225046+0000","last_became_peered":"2024-01-11T08:26:01.225046+0000","last_unstale":"2024-01-11T08:26:01.230472+0000","last_undegraded":"2024-01-11T08:26:01.230472+0000","last_fullsized":"2024-01-11T08:26:01.230472+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:44:42.746687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.248687+0000","last_change":"2024-01-11T08:26:01.242456+0000","last_active":"2024-01-11T08:26:01.248687+0000","last_peered":"2024-01-11T08:26:01.248687+0000","last_clean":"2024-01-11T08:26:01.248687+0000","last_became_active":"2024-01-11T08:26:01.242090+0000","last_became_peered":"2024-01-11T08:26:01.242090+0000","last_unstale":"2024-01-11T08:26:01.248687+0000","last_undegraded":"2024-01-11T08:26:01.248687+0000","last_fullsized":"2024-01-11T08:26:01.248687+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:51:02.992576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.237998+0000","last_change":"2024-01-11T08:26:01.232219+0000","last_active":"2024-01-11T08:26:01.237998+0000","last_peered":"2024-01-11T08:26:01.237998+0000","last_clean":"2024-01-11T08:26:01.237998+0000","last_became_active":"2024-01-11T08:26:01.231975+0000","last_became_peered":"2024-01-11T08:26:01.231975+0000","last_unstale":"2024-01-11T08:26:01.237998+0000","last_undegraded":"2024-01-11T08:26:01.237998+0000","last_fullsized":"2024-01-11T08:26:01.237998+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:18:48.110131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.235732+0000","last_change":"2024-01-11T08:26:01.231566+0000","last_active":"2024-01-11T08:26:01.235732+0000","last_peered":"2024-01-11T08:26:01.235732+0000","last_clean":"2024-01-11T08:26:01.235732+0000","last_became_active":"2024-01-11T08:26:01.231272+0000","last_became_peered":"2024-01-11T08:26:01.231272+0000","last_unstale":"2024-01-11T08:26:01.235732+0000","last_undegraded":"2024-01-11T08:26:01.235732+0000","last_fullsized":"2024-01-11T08:26:01.235732+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:47:41.531786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.241908+0000","last_change":"2024-01-11T08:26:01.236516+0000","last_active":"2024-01-11T08:26:01.241908+0000","last_peered":"2024-01-11T08:26:01.241908+0000","last_clean":"2024-01-11T08:26:01.241908+0000","last_became_active":"2024-01-11T08:26:01.236205+0000","last_became_peered":"2024-01-11T08:26:01.236205+0000","last_unstale":"2024-01-11T08:26:01.241908+0000","last_undegraded":"2024-01-11T08:26:01.241908+0000","last_fullsized":"2024-01-11T08:26:01.241908+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:33:31.939713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.235061+0000","last_change":"2024-01-11T08:26:01.234005+0000","last_active":"2024-01-11T08:26:01.235061+0000","last_peered":"2024-01-11T08:26:01.235061+0000","last_clean":"2024-01-11T08:26:01.235061+0000","last_became_active":"2024-01-11T08:26:01.233646+0000","last_became_peered":"2024-01-11T08:26:01.233646+0000","last_unstale":"2024-01-11T08:26:01.235061+0000","last_undegraded":"2024-01-11T08:26:01.235061+0000","last_fullsized":"2024-01-11T08:26:01.235061+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:24:53.427601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.242470+0000","last_change":"2024-01-11T08:26:01.239241+0000","last_active":"2024-01-11T08:26:01.242470+0000","last_peered":"2024-01-11T08:26:01.242470+0000","last_clean":"2024-01-11T08:26:01.242470+0000","last_became_active":"2024-01-11T08:26:01.238928+0000","last_became_peered":"2024-01-11T08:26:01.238928+0000","last_unstale":"2024-01-11T08:26:01.242470+0000","last_undegraded":"2024-01-11T08:26:01.242470+0000","last_fullsized":"2024-01-11T08:26:01.242470+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:40.073394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.249269+0000","last_change":"2024-01-11T08:26:01.243153+0000","last_active":"2024-01-11T08:26:01.249269+0000","last_peered":"2024-01-11T08:26:01.249269+0000","last_clean":"2024-01-11T08:26:01.249269+0000","last_became_active":"2024-01-11T08:26:01.242324+0000","last_became_peered":"2024-01-11T08:26:01.242324+0000","last_unstale":"2024-01-11T08:26:01.249269+0000","last_undegraded":"2024-01-11T08:26:01.249269+0000","last_fullsized":"2024-01-11T08:26:01.249269+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:04.047849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.230455+0000","last_change":"2024-01-11T08:26:01.225414+0000","last_active":"2024-01-11T08:26:01.230455+0000","last_peered":"2024-01-11T08:26:01.230455+0000","last_clean":"2024-01-11T08:26:01.230455+0000","last_became_active":"2024-01-11T08:26:01.224575+0000","last_became_peered":"2024-01-11T08:26:01.224575+0000","last_unstale":"2024-01-11T08:26:01.230455+0000","last_undegraded":"2024-01-11T08:26:01.230455+0000","last_fullsized":"2024-01-11T08:26:01.230455+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:23:36.026867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.243101+0000","last_change":"2024-01-11T08:26:01.239000+0000","last_active":"2024-01-11T08:26:01.243101+0000","last_peered":"2024-01-11T08:26:01.243101+0000","last_clean":"2024-01-11T08:26:01.243101+0000","last_became_active":"2024-01-11T08:26:01.237898+0000","last_became_peered":"2024-01-11T08:26:01.237898+0000","last_unstale":"2024-01-11T08:26:01.243101+0000","last_undegraded":"2024-01-11T08:26:01.243101+0000","last_fullsized":"2024-01-11T08:26:01.243101+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:41.268243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.249044+0000","last_change":"2024-01-11T08:26:01.245329+0000","last_active":"2024-01-11T08:26:01.249044+0000","last_peered":"2024-01-11T08:26:01.249044+0000","last_clean":"2024-01-11T08:26:01.249044+0000","last_became_active":"2024-01-11T08:26:01.241392+0000","last_became_peered":"2024-01-11T08:26:01.241392+0000","last_unstale":"2024-01-11T08:26:01.249044+0000","last_undegraded":"2024-01-11T08:26:01.249044+0000","last_fullsized":"2024-01-11T08:26:01.249044+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:24:32.790847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.234032+0000","last_change":"2024-01-11T08:26:01.232687+0000","last_active":"2024-01-11T08:26:01.234032+0000","last_peered":"2024-01-11T08:26:01.234032+0000","last_clean":"2024-01-11T08:26:01.234032+0000","last_became_active":"2024-01-11T08:26:01.232329+0000","last_became_peered":"2024-01-11T08:26:01.232329+0000","last_unstale":"2024-01-11T08:26:01.234032+0000","last_undegraded":"2024-01-11T08:26:01.234032+0000","last_fullsized":"2024-01-11T08:26:01.234032+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:26:39.112106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.248534+0000","last_change":"2024-01-11T08:26:01.243060+0000","last_active":"2024-01-11T08:26:01.248534+0000","last_peered":"2024-01-11T08:26:01.248534+0000","last_clean":"2024-01-11T08:26:01.248534+0000","last_became_active":"2024-01-11T08:26:01.242691+0000","last_became_peered":"2024-01-11T08:26:01.242691+0000","last_unstale":"2024-01-11T08:26:01.248534+0000","last_undegraded":"2024-01-11T08:26:01.248534+0000","last_fullsized":"2024-01-11T08:26:01.248534+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:05:50.354398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.247957+0000","last_change":"2024-01-11T08:26:01.242760+0000","last_active":"2024-01-11T08:26:01.247957+0000","last_peered":"2024-01-11T08:26:01.247957+0000","last_clean":"2024-01-11T08:26:01.247957+0000","last_became_active":"2024-01-11T08:26:01.241982+0000","last_became_peered":"2024-01-11T08:26:01.241982+0000","last_unstale":"2024-01-11T08:26:01.247957+0000","last_undegraded":"2024-01-11T08:26:01.247957+0000","last_fullsized":"2024-01-11T08:26:01.247957+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:20:06.654183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.238093+0000","last_change":"2024-01-11T08:26:01.235037+0000","last_active":"2024-01-11T08:26:01.238093+0000","last_peered":"2024-01-11T08:26:01.238093+0000","last_clean":"2024-01-11T08:26:01.238093+0000","last_became_active":"2024-01-11T08:26:01.232284+0000","last_became_peered":"2024-01-11T08:26:01.232284+0000","last_unstale":"2024-01-11T08:26:01.238093+0000","last_undegraded":"2024-01-11T08:26:01.238093+0000","last_fullsized":"2024-01-11T08:26:01.238093+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:05:50.354398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.241635+0000","last_change":"2024-01-11T08:26:01.238789+0000","last_active":"2024-01-11T08:26:01.241635+0000","last_peered":"2024-01-11T08:26:01.241635+0000","last_clean":"2024-01-11T08:26:01.241635+0000","last_became_active":"2024-01-11T08:26:01.238439+0000","last_became_peered":"2024-01-11T08:26:01.238439+0000","last_unstale":"2024-01-11T08:26:01.241635+0000","last_undegraded":"2024-01-11T08:26:01.241635+0000","last_fullsized":"2024-01-11T08:26:01.241635+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:50:29.330776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.251019+0000","last_change":"2024-01-11T08:26:01.248895+0000","last_active":"2024-01-11T08:26:01.251019+0000","last_peered":"2024-01-11T08:26:01.251019+0000","last_clean":"2024-01-11T08:26:01.251019+0000","last_became_active":"2024-01-11T08:26:01.243940+0000","last_became_peered":"2024-01-11T08:26:01.243940+0000","last_unstale":"2024-01-11T08:26:01.251019+0000","last_undegraded":"2024-01-11T08:26:01.251019+0000","last_fullsized":"2024-01-11T08:26:01.251019+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:54.716686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.250890+0000","last_change":"2024-01-11T08:26:01.248393+0000","last_active":"2024-01-11T08:26:01.250890+0000","last_peered":"2024-01-11T08:26:01.250890+0000","last_clean":"2024-01-11T08:26:01.250890+0000","last_became_active":"2024-01-11T08:26:01.244506+0000","last_became_peered":"2024-01-11T08:26:01.244506+0000","last_unstale":"2024-01-11T08:26:01.250890+0000","last_undegraded":"2024-01-11T08:26:01.250890+0000","last_fullsized":"2024-01-11T08:26:01.250890+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:33:19.528587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.229866+0000","last_change":"2024-01-11T08:26:01.227396+0000","last_active":"2024-01-11T08:26:01.229866+0000","last_peered":"2024-01-11T08:26:01.229866+0000","last_clean":"2024-01-11T08:26:01.229866+0000","last_became_active":"2024-01-11T08:26:01.226983+0000","last_became_peered":"2024-01-11T08:26:01.226983+0000","last_unstale":"2024-01-11T08:26:01.229866+0000","last_undegraded":"2024-01-11T08:26:01.229866+0000","last_fullsized":"2024-01-11T08:26:01.229866+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:28:09.565070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.248570+0000","last_change":"2024-01-11T08:26:01.243009+0000","last_active":"2024-01-11T08:26:01.248570+0000","last_peered":"2024-01-11T08:26:01.248570+0000","last_clean":"2024-01-11T08:26:01.248570+0000","last_became_active":"2024-01-11T08:26:01.242032+0000","last_became_peered":"2024-01-11T08:26:01.242032+0000","last_unstale":"2024-01-11T08:26:01.248570+0000","last_undegraded":"2024-01-11T08:26:01.248570+0000","last_fullsized":"2024-01-11T08:26:01.248570+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:24:32.790847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.234742+0000","last_change":"2024-01-11T08:26:01.232234+0000","last_active":"2024-01-11T08:26:01.234742+0000","last_peered":"2024-01-11T08:26:01.234742+0000","last_clean":"2024-01-11T08:26:01.234742+0000","last_became_active":"2024-01-11T08:26:01.231957+0000","last_became_peered":"2024-01-11T08:26:01.231957+0000","last_unstale":"2024-01-11T08:26:01.234742+0000","last_undegraded":"2024-01-11T08:26:01.234742+0000","last_fullsized":"2024-01-11T08:26:01.234742+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:27.288434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.250898+0000","last_change":"2024-01-11T08:26:01.246116+0000","last_active":"2024-01-11T08:26:01.250898+0000","last_peered":"2024-01-11T08:26:01.250898+0000","last_clean":"2024-01-11T08:26:01.250898+0000","last_became_active":"2024-01-11T08:26:01.243231+0000","last_became_peered":"2024-01-11T08:26:01.243231+0000","last_unstale":"2024-01-11T08:26:01.250898+0000","last_undegraded":"2024-01-11T08:26:01.250898+0000","last_fullsized":"2024-01-11T08:26:01.250898+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:05:38.883911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.243915+0000","last_change":"2024-01-11T08:26:01.241089+0000","last_active":"2024-01-11T08:26:01.243915+0000","last_peered":"2024-01-11T08:26:01.243915+0000","last_clean":"2024-01-11T08:26:01.243915+0000","last_became_active":"2024-01-11T08:26:01.240602+0000","last_became_peered":"2024-01-11T08:26:01.240602+0000","last_unstale":"2024-01-11T08:26:01.243915+0000","last_undegraded":"2024-01-11T08:26:01.243915+0000","last_fullsized":"2024-01-11T08:26:01.243915+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:40:51.946197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.236650+0000","last_change":"2024-01-11T08:26:01.234629+0000","last_active":"2024-01-11T08:26:01.236650+0000","last_peered":"2024-01-11T08:26:01.236650+0000","last_clean":"2024-01-11T08:26:01.236650+0000","last_became_active":"2024-01-11T08:26:01.233972+0000","last_became_peered":"2024-01-11T08:26:01.233972+0000","last_unstale":"2024-01-11T08:26:01.236650+0000","last_undegraded":"2024-01-11T08:26:01.236650+0000","last_fullsized":"2024-01-11T08:26:01.236650+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:42:25.046785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.241873+0000","last_change":"2024-01-11T08:26:01.240107+0000","last_active":"2024-01-11T08:26:01.241873+0000","last_peered":"2024-01-11T08:26:01.241873+0000","last_clean":"2024-01-11T08:26:01.241873+0000","last_became_active":"2024-01-11T08:26:01.239581+0000","last_became_peered":"2024-01-11T08:26:01.239581+0000","last_unstale":"2024-01-11T08:26:01.241873+0000","last_undegraded":"2024-01-11T08:26:01.241873+0000","last_fullsized":"2024-01-11T08:26:01.241873+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:33:19.528587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"87'4","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.244037+0000","last_change":"2024-01-11T08:26:00.234871+0000","last_active":"2024-01-11T08:26:01.244037+0000","last_peered":"2024-01-11T08:26:01.244037+0000","last_clean":"2024-01-11T08:26:01.244037+0000","last_became_active":"2024-01-11T08:26:00.234017+0000","last_became_peered":"2024-01-11T08:26:00.234017+0000","last_unstale":"2024-01-11T08:26:01.244037+0000","last_undegraded":"2024-01-11T08:26:01.244037+0000","last_fullsized":"2024-01-11T08:26:01.244037+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:37:59.184802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.228928+0000","last_change":"2024-01-11T08:26:01.225152+0000","last_active":"2024-01-11T08:26:01.228928+0000","last_peered":"2024-01-11T08:26:01.228928+0000","last_clean":"2024-01-11T08:26:01.228928+0000","last_became_active":"2024-01-11T08:26:01.221478+0000","last_became_peered":"2024-01-11T08:26:01.221478+0000","last_unstale":"2024-01-11T08:26:01.228928+0000","last_undegraded":"2024-01-11T08:26:01.228928+0000","last_fullsized":"2024-01-11T08:26:01.228928+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:43:14.381961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.240531+0000","last_change":"2024-01-11T08:26:00.223271+0000","last_active":"2024-01-11T08:26:01.240531+0000","last_peered":"2024-01-11T08:26:01.240531+0000","last_clean":"2024-01-11T08:26:01.240531+0000","last_became_active":"2024-01-11T08:26:00.223002+0000","last_became_peered":"2024-01-11T08:26:00.223002+0000","last_unstale":"2024-01-11T08:26:01.240531+0000","last_undegraded":"2024-01-11T08:26:01.240531+0000","last_fullsized":"2024-01-11T08:26:01.240531+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:01:23.093342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.236692+0000","last_change":"2024-01-11T08:26:01.235756+0000","last_active":"2024-01-11T08:26:01.236692+0000","last_peered":"2024-01-11T08:26:01.236692+0000","last_clean":"2024-01-11T08:26:01.236692+0000","last_became_active":"2024-01-11T08:26:01.235437+0000","last_became_peered":"2024-01-11T08:26:01.235437+0000","last_unstale":"2024-01-11T08:26:01.236692+0000","last_undegraded":"2024-01-11T08:26:01.236692+0000","last_fullsized":"2024-01-11T08:26:01.236692+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:04.047849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.247849+0000","last_change":"2024-01-11T08:26:00.234044+0000","last_active":"2024-01-11T08:26:01.247849+0000","last_peered":"2024-01-11T08:26:01.247849+0000","last_clean":"2024-01-11T08:26:01.247849+0000","last_became_active":"2024-01-11T08:26:00.233470+0000","last_became_peered":"2024-01-11T08:26:00.233470+0000","last_unstale":"2024-01-11T08:26:01.247849+0000","last_undegraded":"2024-01-11T08:26:01.247849+0000","last_fullsized":"2024-01-11T08:26:01.247849+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:26:23.864262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.240953+0000","last_change":"2024-01-11T08:26:01.237160+0000","last_active":"2024-01-11T08:26:01.240953+0000","last_peered":"2024-01-11T08:26:01.240953+0000","last_clean":"2024-01-11T08:26:01.240953+0000","last_became_active":"2024-01-11T08:26:01.236729+0000","last_became_peered":"2024-01-11T08:26:01.236729+0000","last_unstale":"2024-01-11T08:26:01.240953+0000","last_undegraded":"2024-01-11T08:26:01.240953+0000","last_fullsized":"2024-01-11T08:26:01.240953+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:57:27.207174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.244765+0000","last_change":"2024-01-11T08:26:00.227011+0000","last_active":"2024-01-11T08:26:01.244765+0000","last_peered":"2024-01-11T08:26:01.244765+0000","last_clean":"2024-01-11T08:26:01.244765+0000","last_became_active":"2024-01-11T08:26:00.225851+0000","last_became_peered":"2024-01-11T08:26:00.225851+0000","last_unstale":"2024-01-11T08:26:01.244765+0000","last_undegraded":"2024-01-11T08:26:01.244765+0000","last_fullsized":"2024-01-11T08:26:01.244765+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:54:17.965201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.243361+0000","last_change":"2024-01-11T08:26:01.238452+0000","last_active":"2024-01-11T08:26:01.243361+0000","last_peered":"2024-01-11T08:26:01.243361+0000","last_clean":"2024-01-11T08:26:01.243361+0000","last_became_active":"2024-01-11T08:26:01.237948+0000","last_became_peered":"2024-01-11T08:26:01.237948+0000","last_unstale":"2024-01-11T08:26:01.243361+0000","last_undegraded":"2024-01-11T08:26:01.243361+0000","last_fullsized":"2024-01-11T08:26:01.243361+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:57:27.207174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.244936+0000","last_change":"2024-01-11T08:26:00.234239+0000","last_active":"2024-01-11T08:26:01.244936+0000","last_peered":"2024-01-11T08:26:01.244936+0000","last_clean":"2024-01-11T08:26:01.244936+0000","last_became_active":"2024-01-11T08:26:00.233953+0000","last_became_peered":"2024-01-11T08:26:00.233953+0000","last_unstale":"2024-01-11T08:26:01.244936+0000","last_undegraded":"2024-01-11T08:26:01.244936+0000","last_fullsized":"2024-01-11T08:26:01.244936+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:14:57.866489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.248499+0000","last_change":"2024-01-11T08:26:01.243022+0000","last_active":"2024-01-11T08:26:01.248499+0000","last_peered":"2024-01-11T08:26:01.248499+0000","last_clean":"2024-01-11T08:26:01.248499+0000","last_became_active":"2024-01-11T08:26:01.242698+0000","last_became_peered":"2024-01-11T08:26:01.242698+0000","last_unstale":"2024-01-11T08:26:01.248499+0000","last_undegraded":"2024-01-11T08:26:01.248499+0000","last_fullsized":"2024-01-11T08:26:01.248499+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:40.073394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.235775+0000","last_change":"2024-01-11T08:26:00.222170+0000","last_active":"2024-01-11T08:26:01.235775+0000","last_peered":"2024-01-11T08:26:01.235775+0000","last_clean":"2024-01-11T08:26:01.235775+0000","last_became_active":"2024-01-11T08:26:00.221902+0000","last_became_peered":"2024-01-11T08:26:00.221902+0000","last_unstale":"2024-01-11T08:26:01.235775+0000","last_undegraded":"2024-01-11T08:26:01.235775+0000","last_fullsized":"2024-01-11T08:26:01.235775+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:42:56.576888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.244517+0000","last_change":"2024-01-11T08:26:01.242392+0000","last_active":"2024-01-11T08:26:01.244517+0000","last_peered":"2024-01-11T08:26:01.244517+0000","last_clean":"2024-01-11T08:26:01.244517+0000","last_became_active":"2024-01-11T08:26:01.242058+0000","last_became_peered":"2024-01-11T08:26:01.242058+0000","last_unstale":"2024-01-11T08:26:01.244517+0000","last_undegraded":"2024-01-11T08:26:01.244517+0000","last_fullsized":"2024-01-11T08:26:01.244517+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:56:22.332113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.219450+0000","last_change":"2024-01-11T08:26:00.226695+0000","last_active":"2024-01-11T08:26:01.219450+0000","last_peered":"2024-01-11T08:26:01.219450+0000","last_clean":"2024-01-11T08:26:01.219450+0000","last_became_active":"2024-01-11T08:26:00.226287+0000","last_became_peered":"2024-01-11T08:26:00.226287+0000","last_unstale":"2024-01-11T08:26:01.219450+0000","last_undegraded":"2024-01-11T08:26:01.219450+0000","last_fullsized":"2024-01-11T08:26:01.219450+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:18:51.052145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.230094+0000","last_change":"2024-01-11T08:26:01.225073+0000","last_active":"2024-01-11T08:26:01.230094+0000","last_peered":"2024-01-11T08:26:01.230094+0000","last_clean":"2024-01-11T08:26:01.230094+0000","last_became_active":"2024-01-11T08:26:01.224752+0000","last_became_peered":"2024-01-11T08:26:01.224752+0000","last_unstale":"2024-01-11T08:26:01.230094+0000","last_undegraded":"2024-01-11T08:26:01.230094+0000","last_fullsized":"2024-01-11T08:26:01.230094+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:33:19.528587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.215486+0000","last_change":"2024-01-11T08:26:00.226531+0000","last_active":"2024-01-11T08:26:01.215486+0000","last_peered":"2024-01-11T08:26:01.215486+0000","last_clean":"2024-01-11T08:26:01.215486+0000","last_became_active":"2024-01-11T08:26:00.224791+0000","last_became_peered":"2024-01-11T08:26:00.224791+0000","last_unstale":"2024-01-11T08:26:01.215486+0000","last_undegraded":"2024-01-11T08:26:01.215486+0000","last_fullsized":"2024-01-11T08:26:01.215486+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:28:34.554052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.241112+0000","last_change":"2024-01-11T08:26:01.233468+0000","last_active":"2024-01-11T08:26:01.241112+0000","last_peered":"2024-01-11T08:26:01.241112+0000","last_clean":"2024-01-11T08:26:01.241112+0000","last_became_active":"2024-01-11T08:26:01.233192+0000","last_became_peered":"2024-01-11T08:26:01.233192+0000","last_unstale":"2024-01-11T08:26:01.241112+0000","last_undegraded":"2024-01-11T08:26:01.241112+0000","last_fullsized":"2024-01-11T08:26:01.241112+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:21:55.045490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.223856+0000","last_change":"2024-01-11T08:26:00.230451+0000","last_active":"2024-01-11T08:26:01.223856+0000","last_peered":"2024-01-11T08:26:01.223856+0000","last_clean":"2024-01-11T08:26:01.223856+0000","last_became_active":"2024-01-11T08:26:00.230157+0000","last_became_peered":"2024-01-11T08:26:00.230157+0000","last_unstale":"2024-01-11T08:26:01.223856+0000","last_undegraded":"2024-01-11T08:26:01.223856+0000","last_fullsized":"2024-01-11T08:26:01.223856+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:01:31.347510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.238165+0000","last_change":"2024-01-11T08:26:01.231563+0000","last_active":"2024-01-11T08:26:01.238165+0000","last_peered":"2024-01-11T08:26:01.238165+0000","last_clean":"2024-01-11T08:26:01.238165+0000","last_became_active":"2024-01-11T08:26:01.230672+0000","last_became_peered":"2024-01-11T08:26:01.230672+0000","last_unstale":"2024-01-11T08:26:01.238165+0000","last_undegraded":"2024-01-11T08:26:01.238165+0000","last_fullsized":"2024-01-11T08:26:01.238165+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:33:31.939713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.220184+0000","last_change":"2024-01-11T08:26:00.229539+0000","last_active":"2024-01-11T08:26:01.220184+0000","last_peered":"2024-01-11T08:26:01.220184+0000","last_clean":"2024-01-11T08:26:01.220184+0000","last_became_active":"2024-01-11T08:26:00.227734+0000","last_became_peered":"2024-01-11T08:26:00.227734+0000","last_unstale":"2024-01-11T08:26:01.220184+0000","last_undegraded":"2024-01-11T08:26:01.220184+0000","last_fullsized":"2024-01-11T08:26:01.220184+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:51:42.400479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.249665+0000","last_change":"2024-01-11T08:26:01.246726+0000","last_active":"2024-01-11T08:26:01.249665+0000","last_peered":"2024-01-11T08:26:01.249665+0000","last_clean":"2024-01-11T08:26:01.249665+0000","last_became_active":"2024-01-11T08:26:01.246304+0000","last_became_peered":"2024-01-11T08:26:01.246304+0000","last_unstale":"2024-01-11T08:26:01.249665+0000","last_undegraded":"2024-01-11T08:26:01.249665+0000","last_fullsized":"2024-01-11T08:26:01.249665+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:54.716686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"87'3","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.238190+0000","last_change":"2024-01-11T08:26:00.230358+0000","last_active":"2024-01-11T08:26:01.238190+0000","last_peered":"2024-01-11T08:26:01.238190+0000","last_clean":"2024-01-11T08:26:01.238190+0000","last_became_active":"2024-01-11T08:26:00.230098+0000","last_became_peered":"2024-01-11T08:26:00.230098+0000","last_unstale":"2024-01-11T08:26:01.238190+0000","last_undegraded":"2024-01-11T08:26:01.238190+0000","last_fullsized":"2024-01-11T08:26:01.238190+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:33:30.930430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.230422+0000","last_change":"2024-01-11T08:26:01.227066+0000","last_active":"2024-01-11T08:26:01.230422+0000","last_peered":"2024-01-11T08:26:01.230422+0000","last_clean":"2024-01-11T08:26:01.230422+0000","last_became_active":"2024-01-11T08:26:01.226733+0000","last_became_peered":"2024-01-11T08:26:01.226733+0000","last_unstale":"2024-01-11T08:26:01.230422+0000","last_undegraded":"2024-01-11T08:26:01.230422+0000","last_fullsized":"2024-01-11T08:26:01.230422+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:11:00.260395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"87'6","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.248087+0000","last_change":"2024-01-11T08:26:00.223314+0000","last_active":"2024-01-11T08:26:01.248087+0000","last_peered":"2024-01-11T08:26:01.248087+0000","last_clean":"2024-01-11T08:26:01.248087+0000","last_became_active":"2024-01-11T08:26:00.222895+0000","last_became_peered":"2024-01-11T08:26:00.222895+0000","last_unstale":"2024-01-11T08:26:01.248087+0000","last_undegraded":"2024-01-11T08:26:01.248087+0000","last_fullsized":"2024-01-11T08:26:01.248087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:11:43.581266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.238311+0000","last_change":"2024-01-11T08:26:01.236582+0000","last_active":"2024-01-11T08:26:01.238311+0000","last_peered":"2024-01-11T08:26:01.238311+0000","last_clean":"2024-01-11T08:26:01.238311+0000","last_became_active":"2024-01-11T08:26:01.236309+0000","last_became_peered":"2024-01-11T08:26:01.236309+0000","last_unstale":"2024-01-11T08:26:01.238311+0000","last_undegraded":"2024-01-11T08:26:01.238311+0000","last_fullsized":"2024-01-11T08:26:01.238311+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:51:02.992576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.241558+0000","last_change":"2024-01-11T08:26:00.226277+0000","last_active":"2024-01-11T08:26:01.241558+0000","last_peered":"2024-01-11T08:26:01.241558+0000","last_clean":"2024-01-11T08:26:01.241558+0000","last_became_active":"2024-01-11T08:26:00.225621+0000","last_became_peered":"2024-01-11T08:26:00.225621+0000","last_unstale":"2024-01-11T08:26:01.241558+0000","last_undegraded":"2024-01-11T08:26:01.241558+0000","last_fullsized":"2024-01-11T08:26:01.241558+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:51:42.400479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.250937+0000","last_change":"2024-01-11T08:26:01.248903+0000","last_active":"2024-01-11T08:26:01.250937+0000","last_peered":"2024-01-11T08:26:01.250937+0000","last_clean":"2024-01-11T08:26:01.250937+0000","last_became_active":"2024-01-11T08:26:01.246057+0000","last_became_peered":"2024-01-11T08:26:01.246057+0000","last_unstale":"2024-01-11T08:26:01.250937+0000","last_undegraded":"2024-01-11T08:26:01.250937+0000","last_fullsized":"2024-01-11T08:26:01.250937+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:18:48.110131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.218346+0000","last_change":"2024-01-11T08:26:00.222361+0000","last_active":"2024-01-11T08:26:01.218346+0000","last_peered":"2024-01-11T08:26:01.218346+0000","last_clean":"2024-01-11T08:26:01.218346+0000","last_became_active":"2024-01-11T08:26:00.222048+0000","last_became_peered":"2024-01-11T08:26:00.222048+0000","last_unstale":"2024-01-11T08:26:01.218346+0000","last_undegraded":"2024-01-11T08:26:01.218346+0000","last_fullsized":"2024-01-11T08:26:01.218346+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:14:18.949492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.242069+0000","last_change":"2024-01-11T08:26:01.237930+0000","last_active":"2024-01-11T08:26:01.242069+0000","last_peered":"2024-01-11T08:26:01.242069+0000","last_clean":"2024-01-11T08:26:01.242069+0000","last_became_active":"2024-01-11T08:26:01.237679+0000","last_became_peered":"2024-01-11T08:26:01.237679+0000","last_unstale":"2024-01-11T08:26:01.242069+0000","last_undegraded":"2024-01-11T08:26:01.242069+0000","last_fullsized":"2024-01-11T08:26:01.242069+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:55:56.568599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.221847+0000","last_change":"2024-01-11T08:26:00.223109+0000","last_active":"2024-01-11T08:26:01.221847+0000","last_peered":"2024-01-11T08:26:01.221847+0000","last_clean":"2024-01-11T08:26:01.221847+0000","last_became_active":"2024-01-11T08:26:00.222837+0000","last_became_peered":"2024-01-11T08:26:00.222837+0000","last_unstale":"2024-01-11T08:26:01.221847+0000","last_undegraded":"2024-01-11T08:26:01.221847+0000","last_fullsized":"2024-01-11T08:26:01.221847+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:24:31.781564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.223211+0000","last_change":"2024-01-11T08:26:01.218373+0000","last_active":"2024-01-11T08:26:01.223211+0000","last_peered":"2024-01-11T08:26:01.223211+0000","last_clean":"2024-01-11T08:26:01.223211+0000","last_became_active":"2024-01-11T08:26:01.218101+0000","last_became_peered":"2024-01-11T08:26:01.218101+0000","last_unstale":"2024-01-11T08:26:01.223211+0000","last_undegraded":"2024-01-11T08:26:01.223211+0000","last_fullsized":"2024-01-11T08:26:01.223211+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:09:22.670170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"87'8","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.249225+0000","last_change":"2024-01-11T08:26:00.229585+0000","last_active":"2024-01-11T08:26:01.249225+0000","last_peered":"2024-01-11T08:26:01.249225+0000","last_clean":"2024-01-11T08:26:01.249225+0000","last_became_active":"2024-01-11T08:26:00.228203+0000","last_became_peered":"2024-01-11T08:26:00.228203+0000","last_unstale":"2024-01-11T08:26:01.249225+0000","last_undegraded":"2024-01-11T08:26:01.249225+0000","last_fullsized":"2024-01-11T08:26:01.249225+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:12.248111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.251038+0000","last_change":"2024-01-11T08:26:01.249327+0000","last_active":"2024-01-11T08:26:01.251038+0000","last_peered":"2024-01-11T08:26:01.251038+0000","last_clean":"2024-01-11T08:26:01.251038+0000","last_became_active":"2024-01-11T08:26:01.244394+0000","last_became_peered":"2024-01-11T08:26:01.244394+0000","last_unstale":"2024-01-11T08:26:01.251038+0000","last_undegraded":"2024-01-11T08:26:01.251038+0000","last_fullsized":"2024-01-11T08:26:01.251038+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:14:56.858740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.233742+0000","last_change":"2024-01-11T08:26:00.226725+0000","last_active":"2024-01-11T08:26:01.233742+0000","last_peered":"2024-01-11T08:26:01.233742+0000","last_clean":"2024-01-11T08:26:01.233742+0000","last_became_active":"2024-01-11T08:26:00.226435+0000","last_became_peered":"2024-01-11T08:26:00.226435+0000","last_unstale":"2024-01-11T08:26:01.233742+0000","last_undegraded":"2024-01-11T08:26:01.233742+0000","last_fullsized":"2024-01-11T08:26:01.233742+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:03:22.145834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.235616+0000","last_change":"2024-01-11T08:26:01.229961+0000","last_active":"2024-01-11T08:26:01.235616+0000","last_peered":"2024-01-11T08:26:01.235616+0000","last_clean":"2024-01-11T08:26:01.235616+0000","last_became_active":"2024-01-11T08:26:01.229709+0000","last_became_peered":"2024-01-11T08:26:01.229709+0000","last_unstale":"2024-01-11T08:26:01.235616+0000","last_undegraded":"2024-01-11T08:26:01.235616+0000","last_fullsized":"2024-01-11T08:26:01.235616+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:40.073394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.248638+0000","last_change":"2024-01-11T08:26:00.233384+0000","last_active":"2024-01-11T08:26:01.248638+0000","last_peered":"2024-01-11T08:26:01.248638+0000","last_clean":"2024-01-11T08:26:01.248638+0000","last_became_active":"2024-01-11T08:26:00.232542+0000","last_became_peered":"2024-01-11T08:26:00.232542+0000","last_unstale":"2024-01-11T08:26:01.248638+0000","last_undegraded":"2024-01-11T08:26:01.248638+0000","last_fullsized":"2024-01-11T08:26:01.248638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:40.258959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.235751+0000","last_change":"2024-01-11T08:26:01.231746+0000","last_active":"2024-01-11T08:26:01.235751+0000","last_peered":"2024-01-11T08:26:01.235751+0000","last_clean":"2024-01-11T08:26:01.235751+0000","last_became_active":"2024-01-11T08:26:01.231306+0000","last_became_peered":"2024-01-11T08:26:01.231306+0000","last_unstale":"2024-01-11T08:26:01.235751+0000","last_undegraded":"2024-01-11T08:26:01.235751+0000","last_fullsized":"2024-01-11T08:26:01.235751+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:13.257395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.248161+0000","last_change":"2024-01-11T08:26:00.231398+0000","last_active":"2024-01-11T08:26:01.248161+0000","last_peered":"2024-01-11T08:26:01.248161+0000","last_clean":"2024-01-11T08:26:01.248161+0000","last_became_active":"2024-01-11T08:26:00.229449+0000","last_became_peered":"2024-01-11T08:26:00.229449+0000","last_unstale":"2024-01-11T08:26:01.248161+0000","last_undegraded":"2024-01-11T08:26:01.248161+0000","last_fullsized":"2024-01-11T08:26:01.248161+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:40:46.547648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.247504+0000","last_change":"2024-01-11T08:26:01.245175+0000","last_active":"2024-01-11T08:26:01.247504+0000","last_peered":"2024-01-11T08:26:01.247504+0000","last_clean":"2024-01-11T08:26:01.247504+0000","last_became_active":"2024-01-11T08:26:01.244848+0000","last_became_peered":"2024-01-11T08:26:01.244848+0000","last_unstale":"2024-01-11T08:26:01.247504+0000","last_undegraded":"2024-01-11T08:26:01.247504+0000","last_fullsized":"2024-01-11T08:26:01.247504+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:13:10.860956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.216249+0000","last_change":"2024-01-11T08:26:00.236493+0000","last_active":"2024-01-11T08:26:01.216249+0000","last_peered":"2024-01-11T08:26:01.216249+0000","last_clean":"2024-01-11T08:26:01.216249+0000","last_became_active":"2024-01-11T08:26:00.236170+0000","last_became_peered":"2024-01-11T08:26:00.236170+0000","last_unstale":"2024-01-11T08:26:01.216249+0000","last_undegraded":"2024-01-11T08:26:01.216249+0000","last_fullsized":"2024-01-11T08:26:01.216249+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:17:29.795584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.234560+0000","last_change":"2024-01-11T08:26:01.229458+0000","last_active":"2024-01-11T08:26:01.234560+0000","last_peered":"2024-01-11T08:26:01.234560+0000","last_clean":"2024-01-11T08:26:01.234560+0000","last_became_active":"2024-01-11T08:26:01.229031+0000","last_became_peered":"2024-01-11T08:26:01.229031+0000","last_unstale":"2024-01-11T08:26:01.234560+0000","last_undegraded":"2024-01-11T08:26:01.234560+0000","last_fullsized":"2024-01-11T08:26:01.234560+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:41:16.624634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.218413+0000","last_change":"2024-01-11T08:26:00.220912+0000","last_active":"2024-01-11T08:26:01.218413+0000","last_peered":"2024-01-11T08:26:01.218413+0000","last_clean":"2024-01-11T08:26:01.218413+0000","last_became_active":"2024-01-11T08:26:00.220382+0000","last_became_peered":"2024-01-11T08:26:00.220382+0000","last_unstale":"2024-01-11T08:26:01.218413+0000","last_undegraded":"2024-01-11T08:26:01.218413+0000","last_fullsized":"2024-01-11T08:26:01.218413+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:18:51.052145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.234465+0000","last_change":"2024-01-11T08:26:01.229535+0000","last_active":"2024-01-11T08:26:01.234465+0000","last_peered":"2024-01-11T08:26:01.234465+0000","last_clean":"2024-01-11T08:26:01.234465+0000","last_became_active":"2024-01-11T08:26:01.229211+0000","last_became_peered":"2024-01-11T08:26:01.229211+0000","last_unstale":"2024-01-11T08:26:01.234465+0000","last_undegraded":"2024-01-11T08:26:01.234465+0000","last_fullsized":"2024-01-11T08:26:01.234465+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:57:12.855745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.221944+0000","last_change":"2024-01-11T08:26:00.226532+0000","last_active":"2024-01-11T08:26:01.221944+0000","last_peered":"2024-01-11T08:26:01.221944+0000","last_clean":"2024-01-11T08:26:01.221944+0000","last_became_active":"2024-01-11T08:26:00.226218+0000","last_became_peered":"2024-01-11T08:26:00.226218+0000","last_unstale":"2024-01-11T08:26:01.221944+0000","last_undegraded":"2024-01-11T08:26:01.221944+0000","last_fullsized":"2024-01-11T08:26:01.221944+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:51:01.983293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.233922+0000","last_change":"2024-01-11T08:26:01.229252+0000","last_active":"2024-01-11T08:26:01.233922+0000","last_peered":"2024-01-11T08:26:01.233922+0000","last_clean":"2024-01-11T08:26:01.233922+0000","last_became_active":"2024-01-11T08:26:01.228886+0000","last_became_peered":"2024-01-11T08:26:01.228886+0000","last_unstale":"2024-01-11T08:26:01.233922+0000","last_undegraded":"2024-01-11T08:26:01.233922+0000","last_fullsized":"2024-01-11T08:26:01.233922+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:51:38.890046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.216718+0000","last_change":"2024-01-11T08:26:00.223025+0000","last_active":"2024-01-11T08:26:01.216718+0000","last_peered":"2024-01-11T08:26:01.216718+0000","last_clean":"2024-01-11T08:26:01.216718+0000","last_became_active":"2024-01-11T08:26:00.222727+0000","last_became_peered":"2024-01-11T08:26:00.222727+0000","last_unstale":"2024-01-11T08:26:01.216718+0000","last_undegraded":"2024-01-11T08:26:01.216718+0000","last_fullsized":"2024-01-11T08:26:01.216718+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:42:24.037501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.233394+0000","last_change":"2024-01-11T08:26:01.230966+0000","last_active":"2024-01-11T08:26:01.233394+0000","last_peered":"2024-01-11T08:26:01.233394+0000","last_clean":"2024-01-11T08:26:01.233394+0000","last_became_active":"2024-01-11T08:26:01.230657+0000","last_became_peered":"2024-01-11T08:26:01.230657+0000","last_unstale":"2024-01-11T08:26:01.233394+0000","last_undegraded":"2024-01-11T08:26:01.233394+0000","last_fullsized":"2024-01-11T08:26:01.233394+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:14:58.875773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.232677+0000","last_change":"2024-01-11T08:26:00.226758+0000","last_active":"2024-01-11T08:26:01.232677+0000","last_peered":"2024-01-11T08:26:01.232677+0000","last_clean":"2024-01-11T08:26:01.232677+0000","last_became_active":"2024-01-11T08:26:00.226468+0000","last_became_peered":"2024-01-11T08:26:00.226468+0000","last_unstale":"2024-01-11T08:26:01.232677+0000","last_undegraded":"2024-01-11T08:26:01.232677+0000","last_fullsized":"2024-01-11T08:26:01.232677+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:16:05.400026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.236997+0000","last_change":"2024-01-11T08:26:01.231397+0000","last_active":"2024-01-11T08:26:01.236997+0000","last_peered":"2024-01-11T08:26:01.236997+0000","last_clean":"2024-01-11T08:26:01.236997+0000","last_became_active":"2024-01-11T08:26:01.231114+0000","last_became_peered":"2024-01-11T08:26:01.231114+0000","last_unstale":"2024-01-11T08:26:01.236997+0000","last_undegraded":"2024-01-11T08:26:01.236997+0000","last_fullsized":"2024-01-11T08:26:01.236997+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:40:29.446808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.218981+0000","last_change":"2024-01-11T08:26:00.227918+0000","last_active":"2024-01-11T08:26:01.218981+0000","last_peered":"2024-01-11T08:26:01.218981+0000","last_clean":"2024-01-11T08:26:01.218981+0000","last_became_active":"2024-01-11T08:26:00.227663+0000","last_became_peered":"2024-01-11T08:26:00.227663+0000","last_unstale":"2024-01-11T08:26:01.218981+0000","last_undegraded":"2024-01-11T08:26:01.218981+0000","last_fullsized":"2024-01-11T08:26:01.218981+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:20:05.644899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.241661+0000","last_change":"2024-01-11T08:26:01.239266+0000","last_active":"2024-01-11T08:26:01.241661+0000","last_peered":"2024-01-11T08:26:01.241661+0000","last_clean":"2024-01-11T08:26:01.241661+0000","last_became_active":"2024-01-11T08:26:01.238875+0000","last_became_peered":"2024-01-11T08:26:01.238875+0000","last_unstale":"2024-01-11T08:26:01.241661+0000","last_undegraded":"2024-01-11T08:26:01.241661+0000","last_fullsized":"2024-01-11T08:26:01.241661+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:14:58.875773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.215131+0000","last_change":"2024-01-11T08:26:00.228618+0000","last_active":"2024-01-11T08:26:01.215131+0000","last_peered":"2024-01-11T08:26:01.215131+0000","last_clean":"2024-01-11T08:26:01.215131+0000","last_became_active":"2024-01-11T08:26:00.227538+0000","last_became_peered":"2024-01-11T08:26:00.227538+0000","last_unstale":"2024-01-11T08:26:01.215131+0000","last_undegraded":"2024-01-11T08:26:01.215131+0000","last_fullsized":"2024-01-11T08:26:01.215131+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:53:07.837492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.238021+0000","last_change":"2024-01-11T08:26:01.234732+0000","last_active":"2024-01-11T08:26:01.238021+0000","last_peered":"2024-01-11T08:26:01.238021+0000","last_clean":"2024-01-11T08:26:01.238021+0000","last_became_active":"2024-01-11T08:26:01.234453+0000","last_became_peered":"2024-01-11T08:26:01.234453+0000","last_unstale":"2024-01-11T08:26:01.238021+0000","last_undegraded":"2024-01-11T08:26:01.238021+0000","last_fullsized":"2024-01-11T08:26:01.238021+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:05:38.883911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.219103+0000","last_change":"2024-01-11T08:26:00.227131+0000","last_active":"2024-01-11T08:26:01.219103+0000","last_peered":"2024-01-11T08:26:01.219103+0000","last_clean":"2024-01-11T08:26:01.219103+0000","last_became_active":"2024-01-11T08:26:00.226874+0000","last_became_peered":"2024-01-11T08:26:00.226874+0000","last_unstale":"2024-01-11T08:26:01.219103+0000","last_undegraded":"2024-01-11T08:26:01.219103+0000","last_fullsized":"2024-01-11T08:26:01.219103+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:56:27.206807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.227772+0000","last_change":"2024-01-11T08:26:01.222304+0000","last_active":"2024-01-11T08:26:01.227772+0000","last_peered":"2024-01-11T08:26:01.227772+0000","last_clean":"2024-01-11T08:26:01.227772+0000","last_became_active":"2024-01-11T08:26:01.221431+0000","last_became_peered":"2024-01-11T08:26:01.221431+0000","last_unstale":"2024-01-11T08:26:01.227772+0000","last_undegraded":"2024-01-11T08:26:01.227772+0000","last_fullsized":"2024-01-11T08:26:01.227772+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:53:09.106559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.215235+0000","last_change":"2024-01-11T08:26:00.230136+0000","last_active":"2024-01-11T08:26:01.215235+0000","last_peered":"2024-01-11T08:26:01.215235+0000","last_clean":"2024-01-11T08:26:01.215235+0000","last_became_active":"2024-01-11T08:26:00.229888+0000","last_became_peered":"2024-01-11T08:26:00.229888+0000","last_unstale":"2024-01-11T08:26:01.215235+0000","last_undegraded":"2024-01-11T08:26:01.215235+0000","last_fullsized":"2024-01-11T08:26:01.215235+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:01:18.038184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.248796+0000","last_change":"2024-01-11T08:26:01.242830+0000","last_active":"2024-01-11T08:26:01.248796+0000","last_peered":"2024-01-11T08:26:01.248796+0000","last_clean":"2024-01-11T08:26:01.248796+0000","last_became_active":"2024-01-11T08:26:01.241012+0000","last_became_peered":"2024-01-11T08:26:01.241012+0000","last_unstale":"2024-01-11T08:26:01.248796+0000","last_undegraded":"2024-01-11T08:26:01.248796+0000","last_fullsized":"2024-01-11T08:26:01.248796+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:01:24.102625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.219469+0000","last_change":"2024-01-11T08:26:00.227828+0000","last_active":"2024-01-11T08:26:01.219469+0000","last_peered":"2024-01-11T08:26:01.219469+0000","last_clean":"2024-01-11T08:26:01.219469+0000","last_became_active":"2024-01-11T08:26:00.227378+0000","last_became_peered":"2024-01-11T08:26:00.227378+0000","last_unstale":"2024-01-11T08:26:01.219469+0000","last_undegraded":"2024-01-11T08:26:01.219469+0000","last_fullsized":"2024-01-11T08:26:01.219469+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:16:05.400026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.227060+0000","last_change":"2024-01-11T08:26:01.224394+0000","last_active":"2024-01-11T08:26:01.227060+0000","last_peered":"2024-01-11T08:26:01.227060+0000","last_clean":"2024-01-11T08:26:01.227060+0000","last_became_active":"2024-01-11T08:26:01.224115+0000","last_became_peered":"2024-01-11T08:26:01.224115+0000","last_unstale":"2024-01-11T08:26:01.227060+0000","last_undegraded":"2024-01-11T08:26:01.227060+0000","last_fullsized":"2024-01-11T08:26:01.227060+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:01:32.356793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.222735+0000","last_change":"2024-01-11T08:26:00.235792+0000","last_active":"2024-01-11T08:26:01.222735+0000","last_peered":"2024-01-11T08:26:01.222735+0000","last_clean":"2024-01-11T08:26:01.222735+0000","last_became_active":"2024-01-11T08:26:00.235147+0000","last_became_peered":"2024-01-11T08:26:00.235147+0000","last_unstale":"2024-01-11T08:26:01.222735+0000","last_undegraded":"2024-01-11T08:26:01.222735+0000","last_fullsized":"2024-01-11T08:26:01.222735+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:47:40.522503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.236243+0000","last_change":"2024-01-11T08:26:01.229847+0000","last_active":"2024-01-11T08:26:01.236243+0000","last_peered":"2024-01-11T08:26:01.236243+0000","last_clean":"2024-01-11T08:26:01.236243+0000","last_became_active":"2024-01-11T08:26:01.229075+0000","last_became_peered":"2024-01-11T08:26:01.229075+0000","last_unstale":"2024-01-11T08:26:01.236243+0000","last_undegraded":"2024-01-11T08:26:01.236243+0000","last_fullsized":"2024-01-11T08:26:01.236243+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:56:28.216090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.218066+0000","last_change":"2024-01-11T08:26:00.226657+0000","last_active":"2024-01-11T08:26:01.218066+0000","last_peered":"2024-01-11T08:26:01.218066+0000","last_clean":"2024-01-11T08:26:01.218066+0000","last_became_active":"2024-01-11T08:26:00.226257+0000","last_became_peered":"2024-01-11T08:26:00.226257+0000","last_unstale":"2024-01-11T08:26:01.218066+0000","last_undegraded":"2024-01-11T08:26:01.218066+0000","last_fullsized":"2024-01-11T08:26:01.218066+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:58:18.541149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.239263+0000","last_change":"2024-01-11T08:26:01.237279+0000","last_active":"2024-01-11T08:26:01.239263+0000","last_peered":"2024-01-11T08:26:01.239263+0000","last_clean":"2024-01-11T08:26:01.239263+0000","last_became_active":"2024-01-11T08:26:01.236984+0000","last_became_peered":"2024-01-11T08:26:01.236984+0000","last_unstale":"2024-01-11T08:26:01.239263+0000","last_undegraded":"2024-01-11T08:26:01.239263+0000","last_fullsized":"2024-01-11T08:26:01.239263+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:57:18.739762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.236591+0000","last_change":"2024-01-11T08:26:00.224059+0000","last_active":"2024-01-11T08:26:01.236591+0000","last_peered":"2024-01-11T08:26:01.236591+0000","last_clean":"2024-01-11T08:26:01.236591+0000","last_became_active":"2024-01-11T08:26:00.223795+0000","last_became_peered":"2024-01-11T08:26:00.223795+0000","last_unstale":"2024-01-11T08:26:01.236591+0000","last_undegraded":"2024-01-11T08:26:01.236591+0000","last_fullsized":"2024-01-11T08:26:01.236591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:32:55.355542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"79'181","reported_seq":443,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.216780+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:26:01.216780+0000","last_peered":"2024-01-11T08:26:01.216780+0000","last_clean":"2024-01-11T08:26:01.216780+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:26:01.216780+0000","last_undegraded":"2024-01-11T08:26:01.216780+0000","last_fullsized":"2024-01-11T08:26:01.216780+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":674,"num_read_kb":1321,"num_write":275,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":674,"num_read_kb":1321,"num_write":275,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181,"ondisk_log_size":181,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575036,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303020,"kb_used_data":1636,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068820,"statfs":{"total":96636764160,"available":96326471680,"internally_reserved":0,"allocated":1675264,"data_stored":1086409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.69399999999999995,"15min":0.70199999999999996},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.79000000000000004,"5min":0.996,"15min":2.8300000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.69199999999999995,"15min":0.70699999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.372},"max":{"1min":0.91600000000000004,"5min":1.073,"15min":1.073},"last":0.59299999999999997}]},{"osd":1,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.45700000000000002,"5min":0.41899999999999998,"15min":0.35799999999999998},"max":{"1min":0.93400000000000005,"5min":0.99099999999999999,"15min":0.998},"last":0.62},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.70599999999999996,"15min":0.71999999999999997},"min":{"1min":0.41999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.94999999999999996,"5min":0.95899999999999996,"15min":1.153},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.71999999999999997,"15min":0.72899999999999998},"min":{"1min":0.54200000000000004,"5min":0.436,"15min":0.436},"max":{"1min":0.872,"5min":1.03,"15min":1.034},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.70499999999999996,"15min":0.71599999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.0049999999999999},"last":0.68500000000000005}]},{"osd":3,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.71999999999999997,"15min":0.74399999999999999},"min":{"1min":0.496,"5min":0.48299999999999998,"15min":0.40300000000000002},"max":{"1min":0.91700000000000004,"5min":1.032,"15min":2.9119999999999999},"last":0.875},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.71899999999999997,"15min":0.73899999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.95899999999999996,"5min":1.008,"15min":1.008},"last":0.72099999999999997}]},{"osd":4,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.42299999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":1.089},"last":0.75},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63400000000000001,"15min":0.63200000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.85299999999999998,"5min":0.90700000000000003,"15min":1.1020000000000001},"last":1.002}]},{"osd":5,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.34599999999999997},"max":{"1min":0.90400000000000003,"5min":0.92200000000000004,"15min":1.0589999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.66600000000000004,"15min":0.67800000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.40400000000000003},"max":{"1min":0.94499999999999995,"5min":1.0089999999999999,"15min":1.032},"last":0.83499999999999996}]},{"osd":6,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.74099999999999999,"15min":0.75800000000000001},"min":{"1min":0.54800000000000004,"5min":0.52000000000000002,"15min":0.46999999999999997},"max":{"1min":0.97499999999999998,"5min":1.0469999999999999,"15min":1.1359999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.72399999999999998,"15min":0.73899999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42099999999999999},"max":{"1min":0.90800000000000003,"5min":1.028,"15min":1.1299999999999999},"last":0.97599999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575035,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302440,"kb_used_data":1056,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069400,"statfs":{"total":96636764160,"available":96327065600,"internally_reserved":0,"allocated":1081344,"data_stored":495545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.66100000000000003,"15min":0.66500000000000004},"min":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.29999999999999999},"max":{"1min":0.81100000000000005,"5min":0.89300000000000002,"15min":0.94799999999999995},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.66600000000000004,"15min":0.68400000000000005},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.68999999999999995}]},{"osd":1,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.68200000000000005,"15min":0.68700000000000006},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.35899999999999999},"max":{"1min":0.997,"5min":1.163,"15min":1.163},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68400000000000005,"15min":0.69099999999999995},"min":{"1min":0.49299999999999999,"5min":0.435,"15min":0.39900000000000002},"max":{"1min":0.92600000000000005,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.61899999999999999}]},{"osd":2,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.70599999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.38600000000000001},"max":{"1min":0.90200000000000002,"5min":0.97199999999999998,"15min":1.016},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.70599999999999996,"15min":0.72099999999999997},"min":{"1min":0.48999999999999999,"5min":0.47799999999999998,"15min":0.373},"max":{"1min":0.89500000000000002,"5min":0.96699999999999997,"15min":0.995},"last":0.56000000000000005}]},{"osd":3,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71699999999999997,"15min":0.73299999999999998},"min":{"1min":0.48099999999999998,"5min":0.46600000000000003,"15min":0.42799999999999999},"max":{"1min":0.94799999999999995,"5min":0.95799999999999996,"15min":1.3260000000000001},"last":0.503},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.73299999999999998,"15min":0.73699999999999999},"min":{"1min":0.52000000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.88800000000000001,"5min":0.95899999999999996,"15min":1.0049999999999999},"last":0.58899999999999997}]},{"osd":4,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.42999999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84199999999999997,"5min":0.874,"15min":1.212},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57699999999999996,"15min":0.60499999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.374},"max":{"1min":0.76700000000000002,"5min":0.96399999999999997,"15min":1.8320000000000001},"last":0.63900000000000001}]},{"osd":5,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64200000000000002,"15min":0.65400000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68799999999999994,"15min":0.70499999999999996},"min":{"1min":0.52300000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":1.7929999999999999},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69099999999999995,"15min":0.71299999999999997},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.41099999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":1.278},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.73099999999999998,"15min":0.749},"min":{"1min":0.46700000000000003,"5min":0.45900000000000002,"15min":0.44700000000000001},"max":{"1min":1.0009999999999999,"5min":1.022,"15min":1.1599999999999999},"last":0.753}]}]},{"osd":1,"up_from":13,"seq":55834575037,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302452,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069388,"statfs":{"total":96636764160,"available":96327053312,"internally_reserved":0,"allocated":1093632,"data_stored":497577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.309,"5min":0.30599999999999999,"15min":0.26900000000000002},"max":{"1min":0.755,"5min":0.85999999999999999,"15min":0.97699999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56999999999999995,"15min":0.57999999999999996},"min":{"1min":0.42699999999999999,"5min":0.41199999999999998,"15min":0.373},"max":{"1min":0.82299999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.504}]},{"osd":2,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48299999999999998,"15min":0.48899999999999999},"min":{"1min":0.33100000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.65800000000000003,"5min":1.113,"15min":1.113},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.375,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71499999999999997,"5min":0.82399999999999995,"15min":1.25},"last":0.33700000000000002}]},{"osd":3,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.60399999999999998,"15min":0.61699999999999999},"min":{"1min":0.41499999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.77600000000000002,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.371,"15min":0.34899999999999998},"max":{"1min":0.88100000000000001,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.45400000000000001}]},{"osd":4,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.75900000000000001,"15min":0.78600000000000003},"min":{"1min":0.504,"5min":0.439,"15min":0.439},"max":{"1min":1.0620000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.80900000000000005,"15min":0.82799999999999996},"min":{"1min":0.65100000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.069,"5min":1.246,"15min":1.3680000000000001},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76900000000000002,"15min":0.79900000000000004},"min":{"1min":0.54300000000000004,"5min":0.502,"15min":0.46999999999999997},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.7849999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.76800000000000002,"15min":0.79400000000000004},"min":{"1min":0.56299999999999994,"5min":0.52700000000000002,"15min":0.50700000000000001},"max":{"1min":1.052,"5min":1.1339999999999999,"15min":1.371},"last":0.72499999999999998}]},{"osd":6,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.79800000000000004,"15min":0.81899999999999995},"min":{"1min":0.56100000000000005,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.413},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.80500000000000005,"15min":0.82499999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.93899999999999995,"5min":1.155,"15min":1.228},"last":0.70999999999999996}]},{"osd":7,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.81999999999999995,"15min":0.84599999999999997},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.52700000000000002},"max":{"1min":1.0149999999999999,"5min":1.129,"15min":1.226},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.82099999999999995,"15min":0.83799999999999997},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.51400000000000001},"max":{"1min":1.099,"5min":1.1339999999999999,"15min":1.321},"last":0.68799999999999994}]}]},{"osd":0,"up_from":13,"seq":55834575036,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303000,"kb_used_data":1616,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068840,"statfs":{"total":96636764160,"available":96326492160,"internally_reserved":0,"allocated":1654784,"data_stored":1079505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.5,"15min":0.5},"min":{"1min":0.317,"5min":0.29999999999999999,"15min":0.24299999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.92700000000000005},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56599999999999995,"15min":0.57399999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.35299999999999998},"max":{"1min":0.89600000000000002,"5min":0.90500000000000003,"15min":0.92100000000000004},"last":0.42699999999999999}]},{"osd":2,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.29099999999999998},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54400000000000004,"15min":0.55300000000000005},"min":{"1min":0.41699999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.32100000000000001}]},{"osd":3,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.33600000000000002},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.629,"15min":0.627},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.39600000000000002},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.44700000000000001}]},{"osd":4,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.79100000000000004,"15min":0.78900000000000003},"min":{"1min":0.502,"5min":0.45700000000000002,"15min":0.437},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.80200000000000005,"15min":0.79100000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.46899999999999997},"max":{"1min":1.095,"5min":1.095,"15min":1.2350000000000001},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.80200000000000005,"15min":0.80800000000000005},"min":{"1min":0.57699999999999996,"5min":0.52500000000000002,"15min":0.505},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.82199999999999995,"15min":0.81699999999999995},"min":{"1min":0.58799999999999997,"5min":0.49099999999999999,"15min":0.47799999999999998},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.59899999999999998}]},{"osd":6,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81100000000000005,"15min":0.80200000000000005},"min":{"1min":0.60199999999999998,"5min":0.50900000000000001,"15min":0.497},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.2689999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82699999999999996,"15min":0.83399999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.44900000000000001},"max":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.621}]},{"osd":7,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.84199999999999997,"15min":0.84099999999999997},"min":{"1min":0.61399999999999999,"5min":0.57999999999999996,"15min":0.51400000000000001},"max":{"1min":1.371,"5min":1.371,"15min":1.446},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.85099999999999998,"15min":0.84299999999999997},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.57699999999999996},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.298},"last":0.66700000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575035,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306540,"kb_used_data":1060,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065300,"statfs":{"total":96636764160,"available":96322867200,"internally_reserved":0,"allocated":1085440,"data_stored":496989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47599999999999998,"15min":0.47199999999999998},"min":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.26000000000000001},"max":{"1min":0.67900000000000005,"5min":0.73499999999999999,"15min":0.86099999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51800000000000002,"15min":0.54500000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.28899999999999998},"max":{"1min":0.67100000000000004,"5min":0.81699999999999995,"15min":1.4119999999999999},"last":0.36699999999999999}]},{"osd":1,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54200000000000004,"15min":0.55000000000000004},"min":{"1min":0.40400000000000003,"5min":0.36399999999999999,"15min":0.30399999999999999},"max":{"1min":0.73199999999999998,"5min":0.80300000000000005,"15min":0.90000000000000002},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.58599999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38100000000000001},"max":{"1min":0.79100000000000004,"5min":0.92900000000000005,"15min":1.5640000000000001},"last":0.44400000000000001}]},{"osd":3,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.38100000000000001,"5min":0.36199999999999999,"15min":0.32900000000000001},"max":{"1min":0.72399999999999998,"5min":0.82499999999999996,"15min":1.1220000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.61199999999999999,"15min":0.63800000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.41799999999999998},"max":{"1min":0.77500000000000002,"5min":0.95299999999999996,"15min":1.22},"last":0.41099999999999998}]},{"osd":4,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.77100000000000002,"15min":0.78900000000000003},"min":{"1min":0.58699999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.026,"5min":1.0580000000000001,"15min":1.488},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.76100000000000001,"15min":0.78100000000000003},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89200000000000002,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.68200000000000005}]},{"osd":5,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.81499999999999995,"15min":0.83999999999999997},"min":{"1min":0.63500000000000001,"5min":0.622,"15min":0.51600000000000001},"max":{"1min":1.105,"5min":1.214,"15min":2.9209999999999998},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.83099999999999996,"15min":0.84999999999999998},"min":{"1min":0.63500000000000001,"5min":0.55500000000000005,"15min":0.55400000000000005},"max":{"1min":1.036,"5min":1.2,"15min":1.845},"last":0.70099999999999996}]},{"osd":6,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.79800000000000004,"15min":0.80200000000000005},"min":{"1min":0.56299999999999994,"5min":0.54200000000000004,"15min":0.51700000000000002},"max":{"1min":0.91400000000000003,"5min":1.1919999999999999,"15min":1.3560000000000001},"last":0.752},{"interface":"front","average":{"1min":0.747,"5min":0.80700000000000005,"15min":0.83299999999999996},"min":{"1min":0.55800000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.012,"5min":1.123,"15min":2.968},"last":0.53800000000000003}]},{"osd":7,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.85299999999999998,"15min":0.872},"min":{"1min":0.62,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.056,"5min":1.2629999999999999,"15min":1.6359999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.83199999999999996,"15min":0.86199999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.0720000000000001,"5min":1.2270000000000001,"15min":1.391},"last":0.72399999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575035,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306536,"kb_used_data":1056,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065304,"statfs":{"total":96636764160,"available":96322871296,"internally_reserved":0,"allocated":1081344,"data_stored":495565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.499,"15min":0.48499999999999999},"min":{"1min":0.34999999999999998,"5min":0.29499999999999998,"15min":0.27600000000000002},"max":{"1min":0.80300000000000005,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52500000000000002,"15min":0.54600000000000004},"min":{"1min":0.39300000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.873,"5min":0.873,"15min":1.0109999999999999},"last":0.52900000000000003}]},{"osd":1,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.47799999999999998,"15min":0.49099999999999999},"min":{"1min":0.33900000000000002,"5min":0.30299999999999999,"15min":0.25900000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.39600000000000002,"5min":0.36299999999999999,"15min":0.32300000000000001},"max":{"1min":0.92500000000000004,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.48799999999999999}]},{"osd":2,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57399999999999995,"15min":0.58399999999999996},"min":{"1min":0.42799999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81299999999999994,"5min":0.90900000000000003,"15min":0.97599999999999998},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.61699999999999999,"15min":0.628},"min":{"1min":0.45900000000000002,"5min":0.44700000000000001,"15min":0.32000000000000001},"max":{"1min":0.74299999999999999,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.64000000000000001}]},{"osd":4,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.71899999999999997,"15min":0.74299999999999999},"min":{"1min":0.54100000000000004,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.012,"5min":1.038,"15min":1.351},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77400000000000002,"15min":0.78200000000000003},"min":{"1min":0.60199999999999998,"5min":0.503,"15min":0.503},"max":{"1min":1.1459999999999999,"5min":1.1819999999999999,"15min":1.198},"last":0.61499999999999999}]},{"osd":5,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76400000000000001,"15min":0.78900000000000003},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.51900000000000002},"max":{"1min":1.0649999999999999,"5min":1.139,"15min":1.3779999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.79400000000000004,"15min":0.80900000000000005},"min":{"1min":0.623,"5min":0.49199999999999999,"15min":0.44900000000000001},"max":{"1min":1.24,"5min":1.2809999999999999,"15min":1.4239999999999999},"last":0.68300000000000005}]},{"osd":6,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.81399999999999995,"15min":0.83499999999999996},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.52600000000000002},"max":{"1min":1.0820000000000001,"5min":1.335,"15min":1.927},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.78700000000000003,"15min":0.80600000000000005},"min":{"1min":0.57999999999999996,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.93999999999999995,"5min":1.2230000000000001,"15min":3.5579999999999998},"last":0.73499999999999999}]},{"osd":7,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81000000000000005,"15min":0.84299999999999997},"min":{"1min":0.59199999999999997,"5min":0.52800000000000002,"15min":0.46600000000000003},"max":{"1min":1.0660000000000001,"5min":1.1579999999999999,"15min":3.5089999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.84599999999999997,"15min":0.86099999999999999},"min":{"1min":0.60599999999999998,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.21},"last":0.77600000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575036,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310628,"kb_used_data":1052,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94061212,"statfs":{"total":96636764160,"available":96318681088,"internally_reserved":0,"allocated":1077248,"data_stored":495523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.72699999999999998,"15min":0.73799999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.28100000000000003},"max":{"1min":0.874,"5min":0.95999999999999996,"15min":1.524},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72799999999999998,"15min":0.74299999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.38600000000000001},"max":{"1min":0.91500000000000004,"5min":1.0049999999999999,"15min":1.177},"last":0.83899999999999997}]},{"osd":1,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.69899999999999995,"15min":0.70399999999999996},"min":{"1min":0.47399999999999998,"5min":0.46200000000000002,"15min":0.33300000000000002},"max":{"1min":0.89400000000000002,"5min":1.0069999999999999,"15min":1.1499999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.69199999999999995,"15min":0.72299999999999998},"min":{"1min":0.43099999999999999,"5min":0.379,"15min":0.36299999999999999},"max":{"1min":0.94899999999999995,"5min":0.99199999999999999,"15min":2.4740000000000002},"last":0.59199999999999997}]},{"osd":2,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.71699999999999997},"min":{"1min":0.46600000000000003,"5min":0.46000000000000002,"15min":0.38900000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.72699999999999998,"15min":0.72999999999999998},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.89400000000000002,"5min":0.97699999999999998,"15min":1.212},"last":0.81599999999999995}]},{"osd":3,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.68700000000000006,"15min":0.70599999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.38100000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":1.2649999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.73999999999999999,"15min":0.754},"min":{"1min":0.502,"5min":0.502,"15min":0.47499999999999998},"max":{"1min":0.84399999999999997,"5min":1.026,"15min":1.069},"last":0.85199999999999998}]},{"osd":5,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60399999999999998,"15min":0.61799999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.85399999999999998,"5min":0.873,"15min":1.0700000000000001},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.622,"15min":0.63300000000000001},"min":{"1min":0.45000000000000001,"5min":0.38200000000000001,"15min":0.32600000000000001},"max":{"1min":0.94299999999999995,"5min":0.97799999999999998,"15min":0.98199999999999998},"last":0.92100000000000004}]},{"osd":6,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.65700000000000003,"15min":0.65600000000000003},"min":{"1min":0.40500000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0680000000000001,"5min":1.133,"15min":1.133},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.69199999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95099999999999996,"5min":1.0269999999999999,"15min":1.119},"last":0.90400000000000003}]},{"osd":7,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.69299999999999995},"min":{"1min":0.51500000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.99399999999999999,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.73999999999999999,"15min":0.74299999999999999},"min":{"1min":0.56999999999999995,"5min":0.46200000000000002,"15min":0.42299999999999999},"max":{"1min":0.91000000000000003,"5min":1.2350000000000001,"15min":1.4279999999999999},"last":0.93799999999999994}]}]},{"osd":5,"up_from":13,"seq":55834575036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302436,"kb_used_data":1052,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069404,"statfs":{"total":96636764160,"available":96327069696,"internally_reserved":0,"allocated":1077248,"data_stored":495543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.69199999999999995,"15min":0.70799999999999996},"min":{"1min":0.39300000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85499999999999998,"5min":1.0249999999999999,"15min":1.032},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.69599999999999995,"15min":0.70999999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.879,"5min":0.96199999999999997,"15min":1.696},"last":0.80000000000000004}]},{"osd":1,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66400000000000003,"15min":0.69899999999999995},"min":{"1min":0.42799999999999999,"5min":0.39400000000000002,"15min":0.36399999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":1.4359999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.70799999999999996,"15min":0.73199999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.05,"5min":1.05,"15min":4.4969999999999999},"last":0.746}]},{"osd":2,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70699999999999996,"15min":0.71099999999999997},"min":{"1min":0.55100000000000005,"5min":0.435,"15min":0.435},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.238},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69499999999999995,"15min":0.746},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.41599999999999998},"max":{"1min":0.94199999999999995,"5min":1.0049999999999999,"15min":8.0429999999999993},"last":0.86399999999999999}]},{"osd":3,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.70199999999999996,"15min":0.755},"min":{"1min":0.47599999999999998,"5min":0.46200000000000002,"15min":0.442},"max":{"1min":1.208,"5min":1.208,"15min":8.0839999999999996},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.748,"15min":0.747},"min":{"1min":0.56399999999999995,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.84999999999999998}]},{"osd":4,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.38300000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":1.113},"last":0.498},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64500000000000002,"15min":0.64000000000000001},"min":{"1min":0.41499999999999998,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.626}]},{"osd":6,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.45700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.034},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67700000000000005,"15min":0.68600000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.085,"5min":1.085,"15min":1.143},"last":0.67100000000000004}]},{"osd":7,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.73399999999999999,"15min":0.73699999999999999},"min":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.437},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77300000000000002,"15min":0.77000000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.88800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:26:05.644 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:26:05.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.879+0000 7f1c23bff700 1 Processor -- start 2024-01-11T08:26:05.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.879+0000 7f1c23bff700 1 -- start start 2024-01-11T08:26:05.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.880+0000 7f1c23bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1c1c0b03e0 0x7f1c1c0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:05.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.880+0000 7f1c23bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1c1c0b07b0 0x7f1c1c0b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:05.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.880+0000 7f1c23bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c1c0cd730 con 0x7f1c1c159200 2024-01-11T08:26:05.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.880+0000 7f1c23bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c1c159d30 con 0x7f1c1c0b07b0 2024-01-11T08:26:05.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.880+0000 7f1c23bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c1c0ccb20 con 0x7f1c1c0b03e0 2024-01-11T08:26:05.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.880+0000 7f1c2119a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1c1c0b07b0 0x7f1c1c0b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.880+0000 7f1c2119a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1c1c0b07b0 0x7f1c1c0b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50838/0 (socket says 172.21.15.27:50838) 2024-01-11T08:26:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.880+0000 7f1c2119a700 1 -- 172.21.15.27:0/1348584386 learned_addr learned my addr 172.21.15.27:0/1348584386 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.881+0000 7f1c2119a700 1 -- 172.21.15.27:0/1348584386 >> v2:172.21.15.77:3300/0 conn(0x7f1c1c0b03e0 msgr2=0x7f1c1c0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.881+0000 7f1c2199b700 1 --2- 172.21.15.27:0/1348584386 >> v2:172.21.15.77:3300/0 conn(0x7f1c1c0b03e0 0x7f1c1c0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.881+0000 7f1c2119a700 1 --2- 172.21.15.27:0/1348584386 >> v2:172.21.15.77:3300/0 conn(0x7f1c1c0b03e0 0x7f1c1c0b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.881+0000 7f1c2119a700 1 -- 172.21.15.27:0/1348584386 >> v1:172.21.15.77:6789/0 conn(0x7f1c1c159200 legacy=0x7f1c1c159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.881+0000 7f1c2119a700 1 -- 172.21.15.27:0/1348584386 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1c0cce00 con 0x7f1c1c0b07b0 2024-01-11T08:26:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.881+0000 7f1c2199b700 1 --2- 172.21.15.27:0/1348584386 >> v2:172.21.15.77:3300/0 conn(0x7f1c1c0b03e0 0x7f1c1c0b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.881+0000 7f1c2119a700 1 --2- 172.21.15.27:0/1348584386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c0b07b0 0x7f1c1c0b29c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c089470 tx=0x7f1c0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e95330db040ff1c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.881+0000 7f1c20999700 1 -- 172.21.15.27:0/1348584386 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c00b820 con 0x7f1c1c0b07b0 2024-01-11T08:26:05.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.881+0000 7f1c20999700 1 -- 172.21.15.27:0/1348584386 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0c00b980 con 0x7f1c1c0b07b0 2024-01-11T08:26:05.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.881+0000 7f1c20999700 1 -- 172.21.15.27:0/1348584386 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c00bd20 con 0x7f1c1c0b07b0 2024-01-11T08:26:05.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.882+0000 7f1c23bff700 1 -- 172.21.15.27:0/1348584386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c0b07b0 msgr2=0x7f1c1c0b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:05.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.882+0000 7f1c23bff700 1 --2- 172.21.15.27:0/1348584386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c0b07b0 0x7f1c1c0b29c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c089470 tx=0x7f1c0c009250 comp rx=0 tx=0).stop 2024-01-11T08:26:05.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.882+0000 7f1c23bff700 1 -- 172.21.15.27:0/1348584386 shutdown_connections 2024-01-11T08:26:05.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.882+0000 7f1c23bff700 1 --2- 172.21.15.27:0/1348584386 >> v2:172.21.15.77:3300/0 conn(0x7f1c1c0b03e0 0x7f1c1c0b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:05.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.882+0000 7f1c23bff700 1 --2- 172.21.15.27:0/1348584386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c0b07b0 0x7f1c1c0b29c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:05.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.882+0000 7f1c23bff700 1 -- 172.21.15.27:0/1348584386 >> 172.21.15.27:0/1348584386 conn(0x7f1c1c06ed40 msgr2=0x7f1c1c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:05.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.882+0000 7f1c23bff700 1 -- 172.21.15.27:0/1348584386 shutdown_connections 2024-01-11T08:26:05.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.882+0000 7f1c23bff700 1 -- 172.21.15.27:0/1348584386 wait complete. 2024-01-11T08:26:05.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.883+0000 7f1c23bff700 1 Processor -- start 2024-01-11T08:26:05.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.884+0000 7f1c23bff700 1 -- start start 2024-01-11T08:26:05.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.884+0000 7f1c23bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c1c0b03e0 0x7f1c1c0a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:05.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.884+0000 7f1c23bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c0b07b0 0x7f1c1c0af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:05.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.884+0000 7f1c23bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1c1c159200 0x7f1c1c1576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:05.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.884+0000 7f1c23bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c16bf50 con 0x7f1c1c0b07b0 2024-01-11T08:26:05.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.884+0000 7f1c23bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c16b0d0 con 0x7f1c1c0b03e0 2024-01-11T08:26:05.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.885+0000 7f1c23bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c16b340 con 0x7f1c1c159200 2024-01-11T08:26:05.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.885+0000 7f1c2119a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c0b07b0 0x7f1c1c0af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:05.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.885+0000 7f1c2119a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c0b07b0 0x7f1c1c0af4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50846/0 (socket says 172.21.15.27:50846) 2024-01-11T08:26:05.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.885+0000 7f1c2119a700 1 -- 172.21.15.27:0/2532017479 learned_addr learned my addr 172.21.15.27:0/2532017479 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:05.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.885+0000 7f1c2119a700 1 -- 172.21.15.27:0/2532017479 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1c1c159200 msgr2=0x7f1c1c1576e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:05.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.885+0000 7f1c2119a700 1 --2- 172.21.15.27:0/2532017479 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1c1c159200 0x7f1c1c1576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:05.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.885+0000 7f1c2119a700 1 -- 172.21.15.27:0/2532017479 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c1c0b03e0 msgr2=0x7f1c1c0a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:05.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.885+0000 7f1c2119a700 1 --2- 172.21.15.27:0/2532017479 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c1c0b03e0 0x7f1c1c0a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:05.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.885+0000 7f1c2119a700 1 -- 172.21.15.27:0/2532017479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0c0094f0 con 0x7f1c1c0b07b0 2024-01-11T08:26:05.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.885+0000 7f1c2119a700 1 --2- 172.21.15.27:0/2532017479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c0b07b0 0x7f1c1c0af4b0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0030c0 tx=0x7f1c0c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:05.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.886+0000 7f1c11ffb700 1 -- 172.21.15.27:0/2532017479 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c011e60 con 0x7f1c1c0b07b0 2024-01-11T08:26:05.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.886+0000 7f1c23bff700 1 -- 172.21.15.27:0/2532017479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c1c16b620 con 0x7f1c1c0b07b0 2024-01-11T08:26:05.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.886+0000 7f1c11ffb700 1 -- 172.21.15.27:0/2532017479 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0c020070 con 0x7f1c1c0b07b0 2024-01-11T08:26:05.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.886+0000 7f1c11ffb700 1 -- 172.21.15.27:0/2532017479 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c01b6a0 con 0x7f1c1c0b07b0 2024-01-11T08:26:05.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.886+0000 7f1c23bff700 1 -- 172.21.15.27:0/2532017479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c1c16bbe0 con 0x7f1c1c0b07b0 2024-01-11T08:26:05.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.887+0000 7f1c23bff700 1 -- 172.21.15.27:0/2532017479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c1c0718d0 con 0x7f1c1c0b07b0 2024-01-11T08:26:05.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.888+0000 7f1c11ffb700 1 -- 172.21.15.27:0/2532017479 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1c0c022030 con 0x7f1c1c0b07b0 2024-01-11T08:26:05.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.889+0000 7f1c11ffb700 1 --2- 172.21.15.27:0/2532017479 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f1c00066f30 0x7f1c000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:05.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.889+0000 7f1c11ffb700 1 -- 172.21.15.27:0/2532017479 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f1c0c086670 con 0x7f1c1c0b07b0 2024-01-11T08:26:05.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.889+0000 7f1c2199b700 1 --2- 172.21.15.27:0/2532017479 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f1c00066f30 0x7f1c000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:05.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.890+0000 7f1c2199b700 1 --2- 172.21.15.27:0/2532017479 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f1c00066f30 0x7f1c000693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1c18007aa0 tx=0x7f1c18008040 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:05.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:05.893+0000 7f1c11ffb700 1 -- 172.21.15.27:0/2532017479 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1c0c0538d0 con 0x7f1c1c0b07b0 2024-01-11T08:26:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.093+0000 7f1c23bff700 1 -- 172.21.15.27:0/2532017479 --> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c1c0718d0 con 0x7f1c00066f30 2024-01-11T08:26:06.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.103+0000 7f1c11ffb700 1 -- 172.21.15.27:0/2532017479 <== mgr.5795 v2:172.21.15.77:6834/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289059 (secure 0 0 0) 0x7f1c1c0718d0 con 0x7f1c00066f30 2024-01-11T08:26:06.107 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.107+0000 7f1c23bff700 1 -- 172.21.15.27:0/2532017479 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f1c00066f30 msgr2=0x7f1c000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.107+0000 7f1c23bff700 1 --2- 172.21.15.27:0/2532017479 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f1c00066f30 0x7f1c000693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1c18007aa0 tx=0x7f1c18008040 comp rx=0 tx=0).stop 2024-01-11T08:26:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.107+0000 7f1c23bff700 1 -- 172.21.15.27:0/2532017479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c0b07b0 msgr2=0x7f1c1c0af4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.107+0000 7f1c23bff700 1 --2- 172.21.15.27:0/2532017479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c0b07b0 0x7f1c1c0af4b0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0030c0 tx=0x7f1c0c016000 comp rx=0 tx=0).stop 2024-01-11T08:26:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.107+0000 7f1c23bff700 1 -- 172.21.15.27:0/2532017479 shutdown_connections 2024-01-11T08:26:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.107+0000 7f1c23bff700 1 --2- 172.21.15.27:0/2532017479 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f1c00066f30 0x7f1c000693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.107+0000 7f1c23bff700 1 --2- 172.21.15.27:0/2532017479 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1c1c159200 0x7f1c1c1576e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.107+0000 7f1c23bff700 1 --2- 172.21.15.27:0/2532017479 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c1c0b03e0 0x7f1c1c0a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.107+0000 7f1c23bff700 1 --2- 172.21.15.27:0/2532017479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c0b07b0 0x7f1c1c0af4b0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.107+0000 7f1c23bff700 1 -- 172.21.15.27:0/2532017479 >> 172.21.15.27:0/2532017479 conn(0x7f1c1c06ed40 msgr2=0x7f1c1c0b4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.107+0000 7f1c23bff700 1 -- 172.21.15.27:0/2532017479 shutdown_connections 2024-01-11T08:26:06.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.107+0000 7f1c23bff700 1 -- 172.21.15.27:0/2532017479 wait complete. 2024-01-11T08:26:06.110 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:26:06.125 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-11T08:26:05.316497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":674,"num_read_kb":1321,"num_write":319,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2437052,"kb_used_data":9596,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752537668,"statfs":{"total":773094113280,"available":770598572032,"internally_reserved":0,"allocated":9826304,"data_stored":5142656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001391"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.238182+0000","last_change":"2024-01-11T08:26:01.235430+0000","last_active":"2024-01-11T08:26:01.238182+0000","last_peered":"2024-01-11T08:26:01.238182+0000","last_clean":"2024-01-11T08:26:01.238182+0000","last_became_active":"2024-01-11T08:26:01.231860+0000","last_became_peered":"2024-01-11T08:26:01.231860+0000","last_unstale":"2024-01-11T08:26:01.238182+0000","last_undegraded":"2024-01-11T08:26:01.238182+0000","last_fullsized":"2024-01-11T08:26:01.238182+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:55:56.568599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.251616+0000","last_change":"2024-01-11T08:26:01.250130+0000","last_active":"2024-01-11T08:26:01.251616+0000","last_peered":"2024-01-11T08:26:01.251616+0000","last_clean":"2024-01-11T08:26:01.251616+0000","last_became_active":"2024-01-11T08:26:01.249817+0000","last_became_peered":"2024-01-11T08:26:01.249817+0000","last_unstale":"2024-01-11T08:26:01.251616+0000","last_undegraded":"2024-01-11T08:26:01.251616+0000","last_fullsized":"2024-01-11T08:26:01.251616+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:14:56.858740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.229889+0000","last_change":"2024-01-11T08:26:01.227701+0000","last_active":"2024-01-11T08:26:01.229889+0000","last_peered":"2024-01-11T08:26:01.229889+0000","last_clean":"2024-01-11T08:26:01.229889+0000","last_became_active":"2024-01-11T08:26:01.227388+0000","last_became_peered":"2024-01-11T08:26:01.227388+0000","last_unstale":"2024-01-11T08:26:01.229889+0000","last_undegraded":"2024-01-11T08:26:01.229889+0000","last_fullsized":"2024-01-11T08:26:01.229889+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:53:16.623590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.243226+0000","last_change":"2024-01-11T08:26:01.240669+0000","last_active":"2024-01-11T08:26:01.243226+0000","last_peered":"2024-01-11T08:26:01.243226+0000","last_clean":"2024-01-11T08:26:01.243226+0000","last_became_active":"2024-01-11T08:26:01.238255+0000","last_became_peered":"2024-01-11T08:26:01.238255+0000","last_unstale":"2024-01-11T08:26:01.243226+0000","last_undegraded":"2024-01-11T08:26:01.243226+0000","last_fullsized":"2024-01-11T08:26:01.243226+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:26:24.873546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.229166+0000","last_change":"2024-01-11T08:26:01.226105+0000","last_active":"2024-01-11T08:26:01.229166+0000","last_peered":"2024-01-11T08:26:01.229166+0000","last_clean":"2024-01-11T08:26:01.229166+0000","last_became_active":"2024-01-11T08:26:01.225365+0000","last_became_peered":"2024-01-11T08:26:01.225365+0000","last_unstale":"2024-01-11T08:26:01.229166+0000","last_undegraded":"2024-01-11T08:26:01.229166+0000","last_fullsized":"2024-01-11T08:26:01.229166+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:37:35.515669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.248957+0000","last_change":"2024-01-11T08:26:01.245817+0000","last_active":"2024-01-11T08:26:01.248957+0000","last_peered":"2024-01-11T08:26:01.248957+0000","last_clean":"2024-01-11T08:26:01.248957+0000","last_became_active":"2024-01-11T08:26:01.245416+0000","last_became_peered":"2024-01-11T08:26:01.245416+0000","last_unstale":"2024-01-11T08:26:01.248957+0000","last_undegraded":"2024-01-11T08:26:01.248957+0000","last_fullsized":"2024-01-11T08:26:01.248957+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:51:02.992576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.243449+0000","last_change":"2024-01-11T08:26:01.236309+0000","last_active":"2024-01-11T08:26:01.243449+0000","last_peered":"2024-01-11T08:26:01.243449+0000","last_clean":"2024-01-11T08:26:01.243449+0000","last_became_active":"2024-01-11T08:26:01.236027+0000","last_became_peered":"2024-01-11T08:26:01.236027+0000","last_unstale":"2024-01-11T08:26:01.243449+0000","last_undegraded":"2024-01-11T08:26:01.243449+0000","last_fullsized":"2024-01-11T08:26:01.243449+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:00:08.280288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.236669+0000","last_change":"2024-01-11T08:26:01.234620+0000","last_active":"2024-01-11T08:26:01.236669+0000","last_peered":"2024-01-11T08:26:01.236669+0000","last_clean":"2024-01-11T08:26:01.236669+0000","last_became_active":"2024-01-11T08:26:01.233940+0000","last_became_peered":"2024-01-11T08:26:01.233940+0000","last_unstale":"2024-01-11T08:26:01.236669+0000","last_undegraded":"2024-01-11T08:26:01.236669+0000","last_fullsized":"2024-01-11T08:26:01.236669+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:20:06.654183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.230472+0000","last_change":"2024-01-11T08:26:01.225434+0000","last_active":"2024-01-11T08:26:01.230472+0000","last_peered":"2024-01-11T08:26:01.230472+0000","last_clean":"2024-01-11T08:26:01.230472+0000","last_became_active":"2024-01-11T08:26:01.225046+0000","last_became_peered":"2024-01-11T08:26:01.225046+0000","last_unstale":"2024-01-11T08:26:01.230472+0000","last_undegraded":"2024-01-11T08:26:01.230472+0000","last_fullsized":"2024-01-11T08:26:01.230472+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:44:42.746687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.248687+0000","last_change":"2024-01-11T08:26:01.242456+0000","last_active":"2024-01-11T08:26:01.248687+0000","last_peered":"2024-01-11T08:26:01.248687+0000","last_clean":"2024-01-11T08:26:01.248687+0000","last_became_active":"2024-01-11T08:26:01.242090+0000","last_became_peered":"2024-01-11T08:26:01.242090+0000","last_unstale":"2024-01-11T08:26:01.248687+0000","last_undegraded":"2024-01-11T08:26:01.248687+0000","last_fullsized":"2024-01-11T08:26:01.248687+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:51:02.992576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.237998+0000","last_change":"2024-01-11T08:26:01.232219+0000","last_active":"2024-01-11T08:26:01.237998+0000","last_peered":"2024-01-11T08:26:01.237998+0000","last_clean":"2024-01-11T08:26:01.237998+0000","last_became_active":"2024-01-11T08:26:01.231975+0000","last_became_peered":"2024-01-11T08:26:01.231975+0000","last_unstale":"2024-01-11T08:26:01.237998+0000","last_undegraded":"2024-01-11T08:26:01.237998+0000","last_fullsized":"2024-01-11T08:26:01.237998+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:18:48.110131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.235732+0000","last_change":"2024-01-11T08:26:01.231566+0000","last_active":"2024-01-11T08:26:01.235732+0000","last_peered":"2024-01-11T08:26:01.235732+0000","last_clean":"2024-01-11T08:26:01.235732+0000","last_became_active":"2024-01-11T08:26:01.231272+0000","last_became_peered":"2024-01-11T08:26:01.231272+0000","last_unstale":"2024-01-11T08:26:01.235732+0000","last_undegraded":"2024-01-11T08:26:01.235732+0000","last_fullsized":"2024-01-11T08:26:01.235732+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:47:41.531786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.241908+0000","last_change":"2024-01-11T08:26:01.236516+0000","last_active":"2024-01-11T08:26:01.241908+0000","last_peered":"2024-01-11T08:26:01.241908+0000","last_clean":"2024-01-11T08:26:01.241908+0000","last_became_active":"2024-01-11T08:26:01.236205+0000","last_became_peered":"2024-01-11T08:26:01.236205+0000","last_unstale":"2024-01-11T08:26:01.241908+0000","last_undegraded":"2024-01-11T08:26:01.241908+0000","last_fullsized":"2024-01-11T08:26:01.241908+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:33:31.939713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.235061+0000","last_change":"2024-01-11T08:26:01.234005+0000","last_active":"2024-01-11T08:26:01.235061+0000","last_peered":"2024-01-11T08:26:01.235061+0000","last_clean":"2024-01-11T08:26:01.235061+0000","last_became_active":"2024-01-11T08:26:01.233646+0000","last_became_peered":"2024-01-11T08:26:01.233646+0000","last_unstale":"2024-01-11T08:26:01.235061+0000","last_undegraded":"2024-01-11T08:26:01.235061+0000","last_fullsized":"2024-01-11T08:26:01.235061+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:24:53.427601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.242470+0000","last_change":"2024-01-11T08:26:01.239241+0000","last_active":"2024-01-11T08:26:01.242470+0000","last_peered":"2024-01-11T08:26:01.242470+0000","last_clean":"2024-01-11T08:26:01.242470+0000","last_became_active":"2024-01-11T08:26:01.238928+0000","last_became_peered":"2024-01-11T08:26:01.238928+0000","last_unstale":"2024-01-11T08:26:01.242470+0000","last_undegraded":"2024-01-11T08:26:01.242470+0000","last_fullsized":"2024-01-11T08:26:01.242470+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:40.073394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.249269+0000","last_change":"2024-01-11T08:26:01.243153+0000","last_active":"2024-01-11T08:26:01.249269+0000","last_peered":"2024-01-11T08:26:01.249269+0000","last_clean":"2024-01-11T08:26:01.249269+0000","last_became_active":"2024-01-11T08:26:01.242324+0000","last_became_peered":"2024-01-11T08:26:01.242324+0000","last_unstale":"2024-01-11T08:26:01.249269+0000","last_undegraded":"2024-01-11T08:26:01.249269+0000","last_fullsized":"2024-01-11T08:26:01.249269+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:04.047849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.230455+0000","last_change":"2024-01-11T08:26:01.225414+0000","last_active":"2024-01-11T08:26:01.230455+0000","last_peered":"2024-01-11T08:26:01.230455+0000","last_clean":"2024-01-11T08:26:01.230455+0000","last_became_active":"2024-01-11T08:26:01.224575+0000","last_became_peered":"2024-01-11T08:26:01.224575+0000","last_unstale":"2024-01-11T08:26:01.230455+0000","last_undegraded":"2024-01-11T08:26:01.230455+0000","last_fullsized":"2024-01-11T08:26:01.230455+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:23:36.026867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.243101+0000","last_change":"2024-01-11T08:26:01.239000+0000","last_active":"2024-01-11T08:26:01.243101+0000","last_peered":"2024-01-11T08:26:01.243101+0000","last_clean":"2024-01-11T08:26:01.243101+0000","last_became_active":"2024-01-11T08:26:01.237898+0000","last_became_peered":"2024-01-11T08:26:01.237898+0000","last_unstale":"2024-01-11T08:26:01.243101+0000","last_undegraded":"2024-01-11T08:26:01.243101+0000","last_fullsized":"2024-01-11T08:26:01.243101+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:41.268243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.249044+0000","last_change":"2024-01-11T08:26:01.245329+0000","last_active":"2024-01-11T08:26:01.249044+0000","last_peered":"2024-01-11T08:26:01.249044+0000","last_clean":"2024-01-11T08:26:01.249044+0000","last_became_active":"2024-01-11T08:26:01.241392+0000","last_became_peered":"2024-01-11T08:26:01.241392+0000","last_unstale":"2024-01-11T08:26:01.249044+0000","last_undegraded":"2024-01-11T08:26:01.249044+0000","last_fullsized":"2024-01-11T08:26:01.249044+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:24:32.790847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.234032+0000","last_change":"2024-01-11T08:26:01.232687+0000","last_active":"2024-01-11T08:26:01.234032+0000","last_peered":"2024-01-11T08:26:01.234032+0000","last_clean":"2024-01-11T08:26:01.234032+0000","last_became_active":"2024-01-11T08:26:01.232329+0000","last_became_peered":"2024-01-11T08:26:01.232329+0000","last_unstale":"2024-01-11T08:26:01.234032+0000","last_undegraded":"2024-01-11T08:26:01.234032+0000","last_fullsized":"2024-01-11T08:26:01.234032+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:26:39.112106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.248534+0000","last_change":"2024-01-11T08:26:01.243060+0000","last_active":"2024-01-11T08:26:01.248534+0000","last_peered":"2024-01-11T08:26:01.248534+0000","last_clean":"2024-01-11T08:26:01.248534+0000","last_became_active":"2024-01-11T08:26:01.242691+0000","last_became_peered":"2024-01-11T08:26:01.242691+0000","last_unstale":"2024-01-11T08:26:01.248534+0000","last_undegraded":"2024-01-11T08:26:01.248534+0000","last_fullsized":"2024-01-11T08:26:01.248534+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:05:50.354398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.247957+0000","last_change":"2024-01-11T08:26:01.242760+0000","last_active":"2024-01-11T08:26:01.247957+0000","last_peered":"2024-01-11T08:26:01.247957+0000","last_clean":"2024-01-11T08:26:01.247957+0000","last_became_active":"2024-01-11T08:26:01.241982+0000","last_became_peered":"2024-01-11T08:26:01.241982+0000","last_unstale":"2024-01-11T08:26:01.247957+0000","last_undegraded":"2024-01-11T08:26:01.247957+0000","last_fullsized":"2024-01-11T08:26:01.247957+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:20:06.654183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.238093+0000","last_change":"2024-01-11T08:26:01.235037+0000","last_active":"2024-01-11T08:26:01.238093+0000","last_peered":"2024-01-11T08:26:01.238093+0000","last_clean":"2024-01-11T08:26:01.238093+0000","last_became_active":"2024-01-11T08:26:01.232284+0000","last_became_peered":"2024-01-11T08:26:01.232284+0000","last_unstale":"2024-01-11T08:26:01.238093+0000","last_undegraded":"2024-01-11T08:26:01.238093+0000","last_fullsized":"2024-01-11T08:26:01.238093+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:05:50.354398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.241635+0000","last_change":"2024-01-11T08:26:01.238789+0000","last_active":"2024-01-11T08:26:01.241635+0000","last_peered":"2024-01-11T08:26:01.241635+0000","last_clean":"2024-01-11T08:26:01.241635+0000","last_became_active":"2024-01-11T08:26:01.238439+0000","last_became_peered":"2024-01-11T08:26:01.238439+0000","last_unstale":"2024-01-11T08:26:01.241635+0000","last_undegraded":"2024-01-11T08:26:01.241635+0000","last_fullsized":"2024-01-11T08:26:01.241635+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:50:29.330776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.251019+0000","last_change":"2024-01-11T08:26:01.248895+0000","last_active":"2024-01-11T08:26:01.251019+0000","last_peered":"2024-01-11T08:26:01.251019+0000","last_clean":"2024-01-11T08:26:01.251019+0000","last_became_active":"2024-01-11T08:26:01.243940+0000","last_became_peered":"2024-01-11T08:26:01.243940+0000","last_unstale":"2024-01-11T08:26:01.251019+0000","last_undegraded":"2024-01-11T08:26:01.251019+0000","last_fullsized":"2024-01-11T08:26:01.251019+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:54.716686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.250890+0000","last_change":"2024-01-11T08:26:01.248393+0000","last_active":"2024-01-11T08:26:01.250890+0000","last_peered":"2024-01-11T08:26:01.250890+0000","last_clean":"2024-01-11T08:26:01.250890+0000","last_became_active":"2024-01-11T08:26:01.244506+0000","last_became_peered":"2024-01-11T08:26:01.244506+0000","last_unstale":"2024-01-11T08:26:01.250890+0000","last_undegraded":"2024-01-11T08:26:01.250890+0000","last_fullsized":"2024-01-11T08:26:01.250890+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:33:19.528587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.229866+0000","last_change":"2024-01-11T08:26:01.227396+0000","last_active":"2024-01-11T08:26:01.229866+0000","last_peered":"2024-01-11T08:26:01.229866+0000","last_clean":"2024-01-11T08:26:01.229866+0000","last_became_active":"2024-01-11T08:26:01.226983+0000","last_became_peered":"2024-01-11T08:26:01.226983+0000","last_unstale":"2024-01-11T08:26:01.229866+0000","last_undegraded":"2024-01-11T08:26:01.229866+0000","last_fullsized":"2024-01-11T08:26:01.229866+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:28:09.565070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.248570+0000","last_change":"2024-01-11T08:26:01.243009+0000","last_active":"2024-01-11T08:26:01.248570+0000","last_peered":"2024-01-11T08:26:01.248570+0000","last_clean":"2024-01-11T08:26:01.248570+0000","last_became_active":"2024-01-11T08:26:01.242032+0000","last_became_peered":"2024-01-11T08:26:01.242032+0000","last_unstale":"2024-01-11T08:26:01.248570+0000","last_undegraded":"2024-01-11T08:26:01.248570+0000","last_fullsized":"2024-01-11T08:26:01.248570+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:24:32.790847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.234742+0000","last_change":"2024-01-11T08:26:01.232234+0000","last_active":"2024-01-11T08:26:01.234742+0000","last_peered":"2024-01-11T08:26:01.234742+0000","last_clean":"2024-01-11T08:26:01.234742+0000","last_became_active":"2024-01-11T08:26:01.231957+0000","last_became_peered":"2024-01-11T08:26:01.231957+0000","last_unstale":"2024-01-11T08:26:01.234742+0000","last_undegraded":"2024-01-11T08:26:01.234742+0000","last_fullsized":"2024-01-11T08:26:01.234742+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:52:27.288434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.250898+0000","last_change":"2024-01-11T08:26:01.246116+0000","last_active":"2024-01-11T08:26:01.250898+0000","last_peered":"2024-01-11T08:26:01.250898+0000","last_clean":"2024-01-11T08:26:01.250898+0000","last_became_active":"2024-01-11T08:26:01.243231+0000","last_became_peered":"2024-01-11T08:26:01.243231+0000","last_unstale":"2024-01-11T08:26:01.250898+0000","last_undegraded":"2024-01-11T08:26:01.250898+0000","last_fullsized":"2024-01-11T08:26:01.250898+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:05:38.883911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.243915+0000","last_change":"2024-01-11T08:26:01.241089+0000","last_active":"2024-01-11T08:26:01.243915+0000","last_peered":"2024-01-11T08:26:01.243915+0000","last_clean":"2024-01-11T08:26:01.243915+0000","last_became_active":"2024-01-11T08:26:01.240602+0000","last_became_peered":"2024-01-11T08:26:01.240602+0000","last_unstale":"2024-01-11T08:26:01.243915+0000","last_undegraded":"2024-01-11T08:26:01.243915+0000","last_fullsized":"2024-01-11T08:26:01.243915+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:40:51.946197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.236650+0000","last_change":"2024-01-11T08:26:01.234629+0000","last_active":"2024-01-11T08:26:01.236650+0000","last_peered":"2024-01-11T08:26:01.236650+0000","last_clean":"2024-01-11T08:26:01.236650+0000","last_became_active":"2024-01-11T08:26:01.233972+0000","last_became_peered":"2024-01-11T08:26:01.233972+0000","last_unstale":"2024-01-11T08:26:01.236650+0000","last_undegraded":"2024-01-11T08:26:01.236650+0000","last_fullsized":"2024-01-11T08:26:01.236650+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:42:25.046785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.241873+0000","last_change":"2024-01-11T08:26:01.240107+0000","last_active":"2024-01-11T08:26:01.241873+0000","last_peered":"2024-01-11T08:26:01.241873+0000","last_clean":"2024-01-11T08:26:01.241873+0000","last_became_active":"2024-01-11T08:26:01.239581+0000","last_became_peered":"2024-01-11T08:26:01.239581+0000","last_unstale":"2024-01-11T08:26:01.241873+0000","last_undegraded":"2024-01-11T08:26:01.241873+0000","last_fullsized":"2024-01-11T08:26:01.241873+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:33:19.528587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"87'4","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.244037+0000","last_change":"2024-01-11T08:26:00.234871+0000","last_active":"2024-01-11T08:26:01.244037+0000","last_peered":"2024-01-11T08:26:01.244037+0000","last_clean":"2024-01-11T08:26:01.244037+0000","last_became_active":"2024-01-11T08:26:00.234017+0000","last_became_peered":"2024-01-11T08:26:00.234017+0000","last_unstale":"2024-01-11T08:26:01.244037+0000","last_undegraded":"2024-01-11T08:26:01.244037+0000","last_fullsized":"2024-01-11T08:26:01.244037+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:37:59.184802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.228928+0000","last_change":"2024-01-11T08:26:01.225152+0000","last_active":"2024-01-11T08:26:01.228928+0000","last_peered":"2024-01-11T08:26:01.228928+0000","last_clean":"2024-01-11T08:26:01.228928+0000","last_became_active":"2024-01-11T08:26:01.221478+0000","last_became_peered":"2024-01-11T08:26:01.221478+0000","last_unstale":"2024-01-11T08:26:01.228928+0000","last_undegraded":"2024-01-11T08:26:01.228928+0000","last_fullsized":"2024-01-11T08:26:01.228928+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:43:14.381961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.240531+0000","last_change":"2024-01-11T08:26:00.223271+0000","last_active":"2024-01-11T08:26:01.240531+0000","last_peered":"2024-01-11T08:26:01.240531+0000","last_clean":"2024-01-11T08:26:01.240531+0000","last_became_active":"2024-01-11T08:26:00.223002+0000","last_became_peered":"2024-01-11T08:26:00.223002+0000","last_unstale":"2024-01-11T08:26:01.240531+0000","last_undegraded":"2024-01-11T08:26:01.240531+0000","last_fullsized":"2024-01-11T08:26:01.240531+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:01:23.093342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.236692+0000","last_change":"2024-01-11T08:26:01.235756+0000","last_active":"2024-01-11T08:26:01.236692+0000","last_peered":"2024-01-11T08:26:01.236692+0000","last_clean":"2024-01-11T08:26:01.236692+0000","last_became_active":"2024-01-11T08:26:01.235437+0000","last_became_peered":"2024-01-11T08:26:01.235437+0000","last_unstale":"2024-01-11T08:26:01.236692+0000","last_undegraded":"2024-01-11T08:26:01.236692+0000","last_fullsized":"2024-01-11T08:26:01.236692+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:04.047849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.247849+0000","last_change":"2024-01-11T08:26:00.234044+0000","last_active":"2024-01-11T08:26:01.247849+0000","last_peered":"2024-01-11T08:26:01.247849+0000","last_clean":"2024-01-11T08:26:01.247849+0000","last_became_active":"2024-01-11T08:26:00.233470+0000","last_became_peered":"2024-01-11T08:26:00.233470+0000","last_unstale":"2024-01-11T08:26:01.247849+0000","last_undegraded":"2024-01-11T08:26:01.247849+0000","last_fullsized":"2024-01-11T08:26:01.247849+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:26:23.864262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.240953+0000","last_change":"2024-01-11T08:26:01.237160+0000","last_active":"2024-01-11T08:26:01.240953+0000","last_peered":"2024-01-11T08:26:01.240953+0000","last_clean":"2024-01-11T08:26:01.240953+0000","last_became_active":"2024-01-11T08:26:01.236729+0000","last_became_peered":"2024-01-11T08:26:01.236729+0000","last_unstale":"2024-01-11T08:26:01.240953+0000","last_undegraded":"2024-01-11T08:26:01.240953+0000","last_fullsized":"2024-01-11T08:26:01.240953+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:57:27.207174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.244765+0000","last_change":"2024-01-11T08:26:00.227011+0000","last_active":"2024-01-11T08:26:01.244765+0000","last_peered":"2024-01-11T08:26:01.244765+0000","last_clean":"2024-01-11T08:26:01.244765+0000","last_became_active":"2024-01-11T08:26:00.225851+0000","last_became_peered":"2024-01-11T08:26:00.225851+0000","last_unstale":"2024-01-11T08:26:01.244765+0000","last_undegraded":"2024-01-11T08:26:01.244765+0000","last_fullsized":"2024-01-11T08:26:01.244765+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:54:17.965201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.243361+0000","last_change":"2024-01-11T08:26:01.238452+0000","last_active":"2024-01-11T08:26:01.243361+0000","last_peered":"2024-01-11T08:26:01.243361+0000","last_clean":"2024-01-11T08:26:01.243361+0000","last_became_active":"2024-01-11T08:26:01.237948+0000","last_became_peered":"2024-01-11T08:26:01.237948+0000","last_unstale":"2024-01-11T08:26:01.243361+0000","last_undegraded":"2024-01-11T08:26:01.243361+0000","last_fullsized":"2024-01-11T08:26:01.243361+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:57:27.207174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.244936+0000","last_change":"2024-01-11T08:26:00.234239+0000","last_active":"2024-01-11T08:26:01.244936+0000","last_peered":"2024-01-11T08:26:01.244936+0000","last_clean":"2024-01-11T08:26:01.244936+0000","last_became_active":"2024-01-11T08:26:00.233953+0000","last_became_peered":"2024-01-11T08:26:00.233953+0000","last_unstale":"2024-01-11T08:26:01.244936+0000","last_undegraded":"2024-01-11T08:26:01.244936+0000","last_fullsized":"2024-01-11T08:26:01.244936+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:14:57.866489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.248499+0000","last_change":"2024-01-11T08:26:01.243022+0000","last_active":"2024-01-11T08:26:01.248499+0000","last_peered":"2024-01-11T08:26:01.248499+0000","last_clean":"2024-01-11T08:26:01.248499+0000","last_became_active":"2024-01-11T08:26:01.242698+0000","last_became_peered":"2024-01-11T08:26:01.242698+0000","last_unstale":"2024-01-11T08:26:01.248499+0000","last_undegraded":"2024-01-11T08:26:01.248499+0000","last_fullsized":"2024-01-11T08:26:01.248499+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:40.073394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.235775+0000","last_change":"2024-01-11T08:26:00.222170+0000","last_active":"2024-01-11T08:26:01.235775+0000","last_peered":"2024-01-11T08:26:01.235775+0000","last_clean":"2024-01-11T08:26:01.235775+0000","last_became_active":"2024-01-11T08:26:00.221902+0000","last_became_peered":"2024-01-11T08:26:00.221902+0000","last_unstale":"2024-01-11T08:26:01.235775+0000","last_undegraded":"2024-01-11T08:26:01.235775+0000","last_fullsized":"2024-01-11T08:26:01.235775+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:42:56.576888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.244517+0000","last_change":"2024-01-11T08:26:01.242392+0000","last_active":"2024-01-11T08:26:01.244517+0000","last_peered":"2024-01-11T08:26:01.244517+0000","last_clean":"2024-01-11T08:26:01.244517+0000","last_became_active":"2024-01-11T08:26:01.242058+0000","last_became_peered":"2024-01-11T08:26:01.242058+0000","last_unstale":"2024-01-11T08:26:01.244517+0000","last_undegraded":"2024-01-11T08:26:01.244517+0000","last_fullsized":"2024-01-11T08:26:01.244517+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:56:22.332113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.219450+0000","last_change":"2024-01-11T08:26:00.226695+0000","last_active":"2024-01-11T08:26:01.219450+0000","last_peered":"2024-01-11T08:26:01.219450+0000","last_clean":"2024-01-11T08:26:01.219450+0000","last_became_active":"2024-01-11T08:26:00.226287+0000","last_became_peered":"2024-01-11T08:26:00.226287+0000","last_unstale":"2024-01-11T08:26:01.219450+0000","last_undegraded":"2024-01-11T08:26:01.219450+0000","last_fullsized":"2024-01-11T08:26:01.219450+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:18:51.052145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.230094+0000","last_change":"2024-01-11T08:26:01.225073+0000","last_active":"2024-01-11T08:26:01.230094+0000","last_peered":"2024-01-11T08:26:01.230094+0000","last_clean":"2024-01-11T08:26:01.230094+0000","last_became_active":"2024-01-11T08:26:01.224752+0000","last_became_peered":"2024-01-11T08:26:01.224752+0000","last_unstale":"2024-01-11T08:26:01.230094+0000","last_undegraded":"2024-01-11T08:26:01.230094+0000","last_fullsized":"2024-01-11T08:26:01.230094+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:33:19.528587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.215486+0000","last_change":"2024-01-11T08:26:00.226531+0000","last_active":"2024-01-11T08:26:01.215486+0000","last_peered":"2024-01-11T08:26:01.215486+0000","last_clean":"2024-01-11T08:26:01.215486+0000","last_became_active":"2024-01-11T08:26:00.224791+0000","last_became_peered":"2024-01-11T08:26:00.224791+0000","last_unstale":"2024-01-11T08:26:01.215486+0000","last_undegraded":"2024-01-11T08:26:01.215486+0000","last_fullsized":"2024-01-11T08:26:01.215486+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:28:34.554052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.241112+0000","last_change":"2024-01-11T08:26:01.233468+0000","last_active":"2024-01-11T08:26:01.241112+0000","last_peered":"2024-01-11T08:26:01.241112+0000","last_clean":"2024-01-11T08:26:01.241112+0000","last_became_active":"2024-01-11T08:26:01.233192+0000","last_became_peered":"2024-01-11T08:26:01.233192+0000","last_unstale":"2024-01-11T08:26:01.241112+0000","last_undegraded":"2024-01-11T08:26:01.241112+0000","last_fullsized":"2024-01-11T08:26:01.241112+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:21:55.045490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.223856+0000","last_change":"2024-01-11T08:26:00.230451+0000","last_active":"2024-01-11T08:26:01.223856+0000","last_peered":"2024-01-11T08:26:01.223856+0000","last_clean":"2024-01-11T08:26:01.223856+0000","last_became_active":"2024-01-11T08:26:00.230157+0000","last_became_peered":"2024-01-11T08:26:00.230157+0000","last_unstale":"2024-01-11T08:26:01.223856+0000","last_undegraded":"2024-01-11T08:26:01.223856+0000","last_fullsized":"2024-01-11T08:26:01.223856+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:01:31.347510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.238165+0000","last_change":"2024-01-11T08:26:01.231563+0000","last_active":"2024-01-11T08:26:01.238165+0000","last_peered":"2024-01-11T08:26:01.238165+0000","last_clean":"2024-01-11T08:26:01.238165+0000","last_became_active":"2024-01-11T08:26:01.230672+0000","last_became_peered":"2024-01-11T08:26:01.230672+0000","last_unstale":"2024-01-11T08:26:01.238165+0000","last_undegraded":"2024-01-11T08:26:01.238165+0000","last_fullsized":"2024-01-11T08:26:01.238165+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:33:31.939713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.220184+0000","last_change":"2024-01-11T08:26:00.229539+0000","last_active":"2024-01-11T08:26:01.220184+0000","last_peered":"2024-01-11T08:26:01.220184+0000","last_clean":"2024-01-11T08:26:01.220184+0000","last_became_active":"2024-01-11T08:26:00.227734+0000","last_became_peered":"2024-01-11T08:26:00.227734+0000","last_unstale":"2024-01-11T08:26:01.220184+0000","last_undegraded":"2024-01-11T08:26:01.220184+0000","last_fullsized":"2024-01-11T08:26:01.220184+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:51:42.400479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.249665+0000","last_change":"2024-01-11T08:26:01.246726+0000","last_active":"2024-01-11T08:26:01.249665+0000","last_peered":"2024-01-11T08:26:01.249665+0000","last_clean":"2024-01-11T08:26:01.249665+0000","last_became_active":"2024-01-11T08:26:01.246304+0000","last_became_peered":"2024-01-11T08:26:01.246304+0000","last_unstale":"2024-01-11T08:26:01.249665+0000","last_undegraded":"2024-01-11T08:26:01.249665+0000","last_fullsized":"2024-01-11T08:26:01.249665+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:54.716686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"87'3","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.238190+0000","last_change":"2024-01-11T08:26:00.230358+0000","last_active":"2024-01-11T08:26:01.238190+0000","last_peered":"2024-01-11T08:26:01.238190+0000","last_clean":"2024-01-11T08:26:01.238190+0000","last_became_active":"2024-01-11T08:26:00.230098+0000","last_became_peered":"2024-01-11T08:26:00.230098+0000","last_unstale":"2024-01-11T08:26:01.238190+0000","last_undegraded":"2024-01-11T08:26:01.238190+0000","last_fullsized":"2024-01-11T08:26:01.238190+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:33:30.930430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.230422+0000","last_change":"2024-01-11T08:26:01.227066+0000","last_active":"2024-01-11T08:26:01.230422+0000","last_peered":"2024-01-11T08:26:01.230422+0000","last_clean":"2024-01-11T08:26:01.230422+0000","last_became_active":"2024-01-11T08:26:01.226733+0000","last_became_peered":"2024-01-11T08:26:01.226733+0000","last_unstale":"2024-01-11T08:26:01.230422+0000","last_undegraded":"2024-01-11T08:26:01.230422+0000","last_fullsized":"2024-01-11T08:26:01.230422+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:11:00.260395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"87'6","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.248087+0000","last_change":"2024-01-11T08:26:00.223314+0000","last_active":"2024-01-11T08:26:01.248087+0000","last_peered":"2024-01-11T08:26:01.248087+0000","last_clean":"2024-01-11T08:26:01.248087+0000","last_became_active":"2024-01-11T08:26:00.222895+0000","last_became_peered":"2024-01-11T08:26:00.222895+0000","last_unstale":"2024-01-11T08:26:01.248087+0000","last_undegraded":"2024-01-11T08:26:01.248087+0000","last_fullsized":"2024-01-11T08:26:01.248087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:11:43.581266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.238311+0000","last_change":"2024-01-11T08:26:01.236582+0000","last_active":"2024-01-11T08:26:01.238311+0000","last_peered":"2024-01-11T08:26:01.238311+0000","last_clean":"2024-01-11T08:26:01.238311+0000","last_became_active":"2024-01-11T08:26:01.236309+0000","last_became_peered":"2024-01-11T08:26:01.236309+0000","last_unstale":"2024-01-11T08:26:01.238311+0000","last_undegraded":"2024-01-11T08:26:01.238311+0000","last_fullsized":"2024-01-11T08:26:01.238311+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:51:02.992576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.241558+0000","last_change":"2024-01-11T08:26:00.226277+0000","last_active":"2024-01-11T08:26:01.241558+0000","last_peered":"2024-01-11T08:26:01.241558+0000","last_clean":"2024-01-11T08:26:01.241558+0000","last_became_active":"2024-01-11T08:26:00.225621+0000","last_became_peered":"2024-01-11T08:26:00.225621+0000","last_unstale":"2024-01-11T08:26:01.241558+0000","last_undegraded":"2024-01-11T08:26:01.241558+0000","last_fullsized":"2024-01-11T08:26:01.241558+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:51:42.400479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.250937+0000","last_change":"2024-01-11T08:26:01.248903+0000","last_active":"2024-01-11T08:26:01.250937+0000","last_peered":"2024-01-11T08:26:01.250937+0000","last_clean":"2024-01-11T08:26:01.250937+0000","last_became_active":"2024-01-11T08:26:01.246057+0000","last_became_peered":"2024-01-11T08:26:01.246057+0000","last_unstale":"2024-01-11T08:26:01.250937+0000","last_undegraded":"2024-01-11T08:26:01.250937+0000","last_fullsized":"2024-01-11T08:26:01.250937+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:18:48.110131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.218346+0000","last_change":"2024-01-11T08:26:00.222361+0000","last_active":"2024-01-11T08:26:01.218346+0000","last_peered":"2024-01-11T08:26:01.218346+0000","last_clean":"2024-01-11T08:26:01.218346+0000","last_became_active":"2024-01-11T08:26:00.222048+0000","last_became_peered":"2024-01-11T08:26:00.222048+0000","last_unstale":"2024-01-11T08:26:01.218346+0000","last_undegraded":"2024-01-11T08:26:01.218346+0000","last_fullsized":"2024-01-11T08:26:01.218346+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:14:18.949492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.242069+0000","last_change":"2024-01-11T08:26:01.237930+0000","last_active":"2024-01-11T08:26:01.242069+0000","last_peered":"2024-01-11T08:26:01.242069+0000","last_clean":"2024-01-11T08:26:01.242069+0000","last_became_active":"2024-01-11T08:26:01.237679+0000","last_became_peered":"2024-01-11T08:26:01.237679+0000","last_unstale":"2024-01-11T08:26:01.242069+0000","last_undegraded":"2024-01-11T08:26:01.242069+0000","last_fullsized":"2024-01-11T08:26:01.242069+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:55:56.568599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.221847+0000","last_change":"2024-01-11T08:26:00.223109+0000","last_active":"2024-01-11T08:26:01.221847+0000","last_peered":"2024-01-11T08:26:01.221847+0000","last_clean":"2024-01-11T08:26:01.221847+0000","last_became_active":"2024-01-11T08:26:00.222837+0000","last_became_peered":"2024-01-11T08:26:00.222837+0000","last_unstale":"2024-01-11T08:26:01.221847+0000","last_undegraded":"2024-01-11T08:26:01.221847+0000","last_fullsized":"2024-01-11T08:26:01.221847+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:24:31.781564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.223211+0000","last_change":"2024-01-11T08:26:01.218373+0000","last_active":"2024-01-11T08:26:01.223211+0000","last_peered":"2024-01-11T08:26:01.223211+0000","last_clean":"2024-01-11T08:26:01.223211+0000","last_became_active":"2024-01-11T08:26:01.218101+0000","last_became_peered":"2024-01-11T08:26:01.218101+0000","last_unstale":"2024-01-11T08:26:01.223211+0000","last_undegraded":"2024-01-11T08:26:01.223211+0000","last_fullsized":"2024-01-11T08:26:01.223211+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:09:22.670170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"87'8","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.249225+0000","last_change":"2024-01-11T08:26:00.229585+0000","last_active":"2024-01-11T08:26:01.249225+0000","last_peered":"2024-01-11T08:26:01.249225+0000","last_clean":"2024-01-11T08:26:01.249225+0000","last_became_active":"2024-01-11T08:26:00.228203+0000","last_became_peered":"2024-01-11T08:26:00.228203+0000","last_unstale":"2024-01-11T08:26:01.249225+0000","last_undegraded":"2024-01-11T08:26:01.249225+0000","last_fullsized":"2024-01-11T08:26:01.249225+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:12.248111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.251038+0000","last_change":"2024-01-11T08:26:01.249327+0000","last_active":"2024-01-11T08:26:01.251038+0000","last_peered":"2024-01-11T08:26:01.251038+0000","last_clean":"2024-01-11T08:26:01.251038+0000","last_became_active":"2024-01-11T08:26:01.244394+0000","last_became_peered":"2024-01-11T08:26:01.244394+0000","last_unstale":"2024-01-11T08:26:01.251038+0000","last_undegraded":"2024-01-11T08:26:01.251038+0000","last_fullsized":"2024-01-11T08:26:01.251038+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:14:56.858740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.233742+0000","last_change":"2024-01-11T08:26:00.226725+0000","last_active":"2024-01-11T08:26:01.233742+0000","last_peered":"2024-01-11T08:26:01.233742+0000","last_clean":"2024-01-11T08:26:01.233742+0000","last_became_active":"2024-01-11T08:26:00.226435+0000","last_became_peered":"2024-01-11T08:26:00.226435+0000","last_unstale":"2024-01-11T08:26:01.233742+0000","last_undegraded":"2024-01-11T08:26:01.233742+0000","last_fullsized":"2024-01-11T08:26:01.233742+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:03:22.145834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.235616+0000","last_change":"2024-01-11T08:26:01.229961+0000","last_active":"2024-01-11T08:26:01.235616+0000","last_peered":"2024-01-11T08:26:01.235616+0000","last_clean":"2024-01-11T08:26:01.235616+0000","last_became_active":"2024-01-11T08:26:01.229709+0000","last_became_peered":"2024-01-11T08:26:01.229709+0000","last_unstale":"2024-01-11T08:26:01.235616+0000","last_undegraded":"2024-01-11T08:26:01.235616+0000","last_fullsized":"2024-01-11T08:26:01.235616+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:40.073394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.248638+0000","last_change":"2024-01-11T08:26:00.233384+0000","last_active":"2024-01-11T08:26:01.248638+0000","last_peered":"2024-01-11T08:26:01.248638+0000","last_clean":"2024-01-11T08:26:01.248638+0000","last_became_active":"2024-01-11T08:26:00.232542+0000","last_became_peered":"2024-01-11T08:26:00.232542+0000","last_unstale":"2024-01-11T08:26:01.248638+0000","last_undegraded":"2024-01-11T08:26:01.248638+0000","last_fullsized":"2024-01-11T08:26:01.248638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:40.258959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.235751+0000","last_change":"2024-01-11T08:26:01.231746+0000","last_active":"2024-01-11T08:26:01.235751+0000","last_peered":"2024-01-11T08:26:01.235751+0000","last_clean":"2024-01-11T08:26:01.235751+0000","last_became_active":"2024-01-11T08:26:01.231306+0000","last_became_peered":"2024-01-11T08:26:01.231306+0000","last_unstale":"2024-01-11T08:26:01.235751+0000","last_undegraded":"2024-01-11T08:26:01.235751+0000","last_fullsized":"2024-01-11T08:26:01.235751+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:13.257395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.248161+0000","last_change":"2024-01-11T08:26:00.231398+0000","last_active":"2024-01-11T08:26:01.248161+0000","last_peered":"2024-01-11T08:26:01.248161+0000","last_clean":"2024-01-11T08:26:01.248161+0000","last_became_active":"2024-01-11T08:26:00.229449+0000","last_became_peered":"2024-01-11T08:26:00.229449+0000","last_unstale":"2024-01-11T08:26:01.248161+0000","last_undegraded":"2024-01-11T08:26:01.248161+0000","last_fullsized":"2024-01-11T08:26:01.248161+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:40:46.547648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.247504+0000","last_change":"2024-01-11T08:26:01.245175+0000","last_active":"2024-01-11T08:26:01.247504+0000","last_peered":"2024-01-11T08:26:01.247504+0000","last_clean":"2024-01-11T08:26:01.247504+0000","last_became_active":"2024-01-11T08:26:01.244848+0000","last_became_peered":"2024-01-11T08:26:01.244848+0000","last_unstale":"2024-01-11T08:26:01.247504+0000","last_undegraded":"2024-01-11T08:26:01.247504+0000","last_fullsized":"2024-01-11T08:26:01.247504+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:13:10.860956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.216249+0000","last_change":"2024-01-11T08:26:00.236493+0000","last_active":"2024-01-11T08:26:01.216249+0000","last_peered":"2024-01-11T08:26:01.216249+0000","last_clean":"2024-01-11T08:26:01.216249+0000","last_became_active":"2024-01-11T08:26:00.236170+0000","last_became_peered":"2024-01-11T08:26:00.236170+0000","last_unstale":"2024-01-11T08:26:01.216249+0000","last_undegraded":"2024-01-11T08:26:01.216249+0000","last_fullsized":"2024-01-11T08:26:01.216249+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:17:29.795584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.234560+0000","last_change":"2024-01-11T08:26:01.229458+0000","last_active":"2024-01-11T08:26:01.234560+0000","last_peered":"2024-01-11T08:26:01.234560+0000","last_clean":"2024-01-11T08:26:01.234560+0000","last_became_active":"2024-01-11T08:26:01.229031+0000","last_became_peered":"2024-01-11T08:26:01.229031+0000","last_unstale":"2024-01-11T08:26:01.234560+0000","last_undegraded":"2024-01-11T08:26:01.234560+0000","last_fullsized":"2024-01-11T08:26:01.234560+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:41:16.624634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.218413+0000","last_change":"2024-01-11T08:26:00.220912+0000","last_active":"2024-01-11T08:26:01.218413+0000","last_peered":"2024-01-11T08:26:01.218413+0000","last_clean":"2024-01-11T08:26:01.218413+0000","last_became_active":"2024-01-11T08:26:00.220382+0000","last_became_peered":"2024-01-11T08:26:00.220382+0000","last_unstale":"2024-01-11T08:26:01.218413+0000","last_undegraded":"2024-01-11T08:26:01.218413+0000","last_fullsized":"2024-01-11T08:26:01.218413+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:18:51.052145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.234465+0000","last_change":"2024-01-11T08:26:01.229535+0000","last_active":"2024-01-11T08:26:01.234465+0000","last_peered":"2024-01-11T08:26:01.234465+0000","last_clean":"2024-01-11T08:26:01.234465+0000","last_became_active":"2024-01-11T08:26:01.229211+0000","last_became_peered":"2024-01-11T08:26:01.229211+0000","last_unstale":"2024-01-11T08:26:01.234465+0000","last_undegraded":"2024-01-11T08:26:01.234465+0000","last_fullsized":"2024-01-11T08:26:01.234465+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:57:12.855745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.221944+0000","last_change":"2024-01-11T08:26:00.226532+0000","last_active":"2024-01-11T08:26:01.221944+0000","last_peered":"2024-01-11T08:26:01.221944+0000","last_clean":"2024-01-11T08:26:01.221944+0000","last_became_active":"2024-01-11T08:26:00.226218+0000","last_became_peered":"2024-01-11T08:26:00.226218+0000","last_unstale":"2024-01-11T08:26:01.221944+0000","last_undegraded":"2024-01-11T08:26:01.221944+0000","last_fullsized":"2024-01-11T08:26:01.221944+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:51:01.983293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.233922+0000","last_change":"2024-01-11T08:26:01.229252+0000","last_active":"2024-01-11T08:26:01.233922+0000","last_peered":"2024-01-11T08:26:01.233922+0000","last_clean":"2024-01-11T08:26:01.233922+0000","last_became_active":"2024-01-11T08:26:01.228886+0000","last_became_peered":"2024-01-11T08:26:01.228886+0000","last_unstale":"2024-01-11T08:26:01.233922+0000","last_undegraded":"2024-01-11T08:26:01.233922+0000","last_fullsized":"2024-01-11T08:26:01.233922+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:51:38.890046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.216718+0000","last_change":"2024-01-11T08:26:00.223025+0000","last_active":"2024-01-11T08:26:01.216718+0000","last_peered":"2024-01-11T08:26:01.216718+0000","last_clean":"2024-01-11T08:26:01.216718+0000","last_became_active":"2024-01-11T08:26:00.222727+0000","last_became_peered":"2024-01-11T08:26:00.222727+0000","last_unstale":"2024-01-11T08:26:01.216718+0000","last_undegraded":"2024-01-11T08:26:01.216718+0000","last_fullsized":"2024-01-11T08:26:01.216718+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:42:24.037501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.233394+0000","last_change":"2024-01-11T08:26:01.230966+0000","last_active":"2024-01-11T08:26:01.233394+0000","last_peered":"2024-01-11T08:26:01.233394+0000","last_clean":"2024-01-11T08:26:01.233394+0000","last_became_active":"2024-01-11T08:26:01.230657+0000","last_became_peered":"2024-01-11T08:26:01.230657+0000","last_unstale":"2024-01-11T08:26:01.233394+0000","last_undegraded":"2024-01-11T08:26:01.233394+0000","last_fullsized":"2024-01-11T08:26:01.233394+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:14:58.875773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.232677+0000","last_change":"2024-01-11T08:26:00.226758+0000","last_active":"2024-01-11T08:26:01.232677+0000","last_peered":"2024-01-11T08:26:01.232677+0000","last_clean":"2024-01-11T08:26:01.232677+0000","last_became_active":"2024-01-11T08:26:00.226468+0000","last_became_peered":"2024-01-11T08:26:00.226468+0000","last_unstale":"2024-01-11T08:26:01.232677+0000","last_undegraded":"2024-01-11T08:26:01.232677+0000","last_fullsized":"2024-01-11T08:26:01.232677+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:16:05.400026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.236997+0000","last_change":"2024-01-11T08:26:01.231397+0000","last_active":"2024-01-11T08:26:01.236997+0000","last_peered":"2024-01-11T08:26:01.236997+0000","last_clean":"2024-01-11T08:26:01.236997+0000","last_became_active":"2024-01-11T08:26:01.231114+0000","last_became_peered":"2024-01-11T08:26:01.231114+0000","last_unstale":"2024-01-11T08:26:01.236997+0000","last_undegraded":"2024-01-11T08:26:01.236997+0000","last_fullsized":"2024-01-11T08:26:01.236997+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:40:29.446808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.218981+0000","last_change":"2024-01-11T08:26:00.227918+0000","last_active":"2024-01-11T08:26:01.218981+0000","last_peered":"2024-01-11T08:26:01.218981+0000","last_clean":"2024-01-11T08:26:01.218981+0000","last_became_active":"2024-01-11T08:26:00.227663+0000","last_became_peered":"2024-01-11T08:26:00.227663+0000","last_unstale":"2024-01-11T08:26:01.218981+0000","last_undegraded":"2024-01-11T08:26:01.218981+0000","last_fullsized":"2024-01-11T08:26:01.218981+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:20:05.644899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.241661+0000","last_change":"2024-01-11T08:26:01.239266+0000","last_active":"2024-01-11T08:26:01.241661+0000","last_peered":"2024-01-11T08:26:01.241661+0000","last_clean":"2024-01-11T08:26:01.241661+0000","last_became_active":"2024-01-11T08:26:01.238875+0000","last_became_peered":"2024-01-11T08:26:01.238875+0000","last_unstale":"2024-01-11T08:26:01.241661+0000","last_undegraded":"2024-01-11T08:26:01.241661+0000","last_fullsized":"2024-01-11T08:26:01.241661+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:14:58.875773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.215131+0000","last_change":"2024-01-11T08:26:00.228618+0000","last_active":"2024-01-11T08:26:01.215131+0000","last_peered":"2024-01-11T08:26:01.215131+0000","last_clean":"2024-01-11T08:26:01.215131+0000","last_became_active":"2024-01-11T08:26:00.227538+0000","last_became_peered":"2024-01-11T08:26:00.227538+0000","last_unstale":"2024-01-11T08:26:01.215131+0000","last_undegraded":"2024-01-11T08:26:01.215131+0000","last_fullsized":"2024-01-11T08:26:01.215131+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:53:07.837492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.238021+0000","last_change":"2024-01-11T08:26:01.234732+0000","last_active":"2024-01-11T08:26:01.238021+0000","last_peered":"2024-01-11T08:26:01.238021+0000","last_clean":"2024-01-11T08:26:01.238021+0000","last_became_active":"2024-01-11T08:26:01.234453+0000","last_became_peered":"2024-01-11T08:26:01.234453+0000","last_unstale":"2024-01-11T08:26:01.238021+0000","last_undegraded":"2024-01-11T08:26:01.238021+0000","last_fullsized":"2024-01-11T08:26:01.238021+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:05:38.883911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.219103+0000","last_change":"2024-01-11T08:26:00.227131+0000","last_active":"2024-01-11T08:26:01.219103+0000","last_peered":"2024-01-11T08:26:01.219103+0000","last_clean":"2024-01-11T08:26:01.219103+0000","last_became_active":"2024-01-11T08:26:00.226874+0000","last_became_peered":"2024-01-11T08:26:00.226874+0000","last_unstale":"2024-01-11T08:26:01.219103+0000","last_undegraded":"2024-01-11T08:26:01.219103+0000","last_fullsized":"2024-01-11T08:26:01.219103+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:56:27.206807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.227772+0000","last_change":"2024-01-11T08:26:01.222304+0000","last_active":"2024-01-11T08:26:01.227772+0000","last_peered":"2024-01-11T08:26:01.227772+0000","last_clean":"2024-01-11T08:26:01.227772+0000","last_became_active":"2024-01-11T08:26:01.221431+0000","last_became_peered":"2024-01-11T08:26:01.221431+0000","last_unstale":"2024-01-11T08:26:01.227772+0000","last_undegraded":"2024-01-11T08:26:01.227772+0000","last_fullsized":"2024-01-11T08:26:01.227772+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:53:09.106559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.215235+0000","last_change":"2024-01-11T08:26:00.230136+0000","last_active":"2024-01-11T08:26:01.215235+0000","last_peered":"2024-01-11T08:26:01.215235+0000","last_clean":"2024-01-11T08:26:01.215235+0000","last_became_active":"2024-01-11T08:26:00.229888+0000","last_became_peered":"2024-01-11T08:26:00.229888+0000","last_unstale":"2024-01-11T08:26:01.215235+0000","last_undegraded":"2024-01-11T08:26:01.215235+0000","last_fullsized":"2024-01-11T08:26:01.215235+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:01:18.038184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.248796+0000","last_change":"2024-01-11T08:26:01.242830+0000","last_active":"2024-01-11T08:26:01.248796+0000","last_peered":"2024-01-11T08:26:01.248796+0000","last_clean":"2024-01-11T08:26:01.248796+0000","last_became_active":"2024-01-11T08:26:01.241012+0000","last_became_peered":"2024-01-11T08:26:01.241012+0000","last_unstale":"2024-01-11T08:26:01.248796+0000","last_undegraded":"2024-01-11T08:26:01.248796+0000","last_fullsized":"2024-01-11T08:26:01.248796+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:01:24.102625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.219469+0000","last_change":"2024-01-11T08:26:00.227828+0000","last_active":"2024-01-11T08:26:01.219469+0000","last_peered":"2024-01-11T08:26:01.219469+0000","last_clean":"2024-01-11T08:26:01.219469+0000","last_became_active":"2024-01-11T08:26:00.227378+0000","last_became_peered":"2024-01-11T08:26:00.227378+0000","last_unstale":"2024-01-11T08:26:01.219469+0000","last_undegraded":"2024-01-11T08:26:01.219469+0000","last_fullsized":"2024-01-11T08:26:01.219469+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:16:05.400026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.227060+0000","last_change":"2024-01-11T08:26:01.224394+0000","last_active":"2024-01-11T08:26:01.227060+0000","last_peered":"2024-01-11T08:26:01.227060+0000","last_clean":"2024-01-11T08:26:01.227060+0000","last_became_active":"2024-01-11T08:26:01.224115+0000","last_became_peered":"2024-01-11T08:26:01.224115+0000","last_unstale":"2024-01-11T08:26:01.227060+0000","last_undegraded":"2024-01-11T08:26:01.227060+0000","last_fullsized":"2024-01-11T08:26:01.227060+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:01:32.356793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.222735+0000","last_change":"2024-01-11T08:26:00.235792+0000","last_active":"2024-01-11T08:26:01.222735+0000","last_peered":"2024-01-11T08:26:01.222735+0000","last_clean":"2024-01-11T08:26:01.222735+0000","last_became_active":"2024-01-11T08:26:00.235147+0000","last_became_peered":"2024-01-11T08:26:00.235147+0000","last_unstale":"2024-01-11T08:26:01.222735+0000","last_undegraded":"2024-01-11T08:26:01.222735+0000","last_fullsized":"2024-01-11T08:26:01.222735+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:47:40.522503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.236243+0000","last_change":"2024-01-11T08:26:01.229847+0000","last_active":"2024-01-11T08:26:01.236243+0000","last_peered":"2024-01-11T08:26:01.236243+0000","last_clean":"2024-01-11T08:26:01.236243+0000","last_became_active":"2024-01-11T08:26:01.229075+0000","last_became_peered":"2024-01-11T08:26:01.229075+0000","last_unstale":"2024-01-11T08:26:01.236243+0000","last_undegraded":"2024-01-11T08:26:01.236243+0000","last_fullsized":"2024-01-11T08:26:01.236243+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:56:28.216090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.218066+0000","last_change":"2024-01-11T08:26:00.226657+0000","last_active":"2024-01-11T08:26:01.218066+0000","last_peered":"2024-01-11T08:26:01.218066+0000","last_clean":"2024-01-11T08:26:01.218066+0000","last_became_active":"2024-01-11T08:26:00.226257+0000","last_became_peered":"2024-01-11T08:26:00.226257+0000","last_unstale":"2024-01-11T08:26:01.218066+0000","last_undegraded":"2024-01-11T08:26:01.218066+0000","last_fullsized":"2024-01-11T08:26:01.218066+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:58:18.541149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.239263+0000","last_change":"2024-01-11T08:26:01.237279+0000","last_active":"2024-01-11T08:26:01.239263+0000","last_peered":"2024-01-11T08:26:01.239263+0000","last_clean":"2024-01-11T08:26:01.239263+0000","last_became_active":"2024-01-11T08:26:01.236984+0000","last_became_peered":"2024-01-11T08:26:01.236984+0000","last_unstale":"2024-01-11T08:26:01.239263+0000","last_undegraded":"2024-01-11T08:26:01.239263+0000","last_fullsized":"2024-01-11T08:26:01.239263+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:00.195109+0000","last_clean_scrub_stamp":"2024-01-11T08:26:00.195109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:57:18.739762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.236591+0000","last_change":"2024-01-11T08:26:00.224059+0000","last_active":"2024-01-11T08:26:01.236591+0000","last_peered":"2024-01-11T08:26:01.236591+0000","last_clean":"2024-01-11T08:26:01.236591+0000","last_became_active":"2024-01-11T08:26:00.223795+0000","last_became_peered":"2024-01-11T08:26:00.223795+0000","last_unstale":"2024-01-11T08:26:01.236591+0000","last_undegraded":"2024-01-11T08:26:01.236591+0000","last_fullsized":"2024-01-11T08:26:01.236591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:25:59.185825+0000","last_clean_scrub_stamp":"2024-01-11T08:25:59.185825+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:32:55.355542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"79'181","reported_seq":443,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-11T08:26:01.216780+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:26:01.216780+0000","last_peered":"2024-01-11T08:26:01.216780+0000","last_clean":"2024-01-11T08:26:01.216780+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:26:01.216780+0000","last_undegraded":"2024-01-11T08:26:01.216780+0000","last_fullsized":"2024-01-11T08:26:01.216780+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":674,"num_read_kb":1321,"num_write":275,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":674,"num_read_kb":1321,"num_write":275,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181,"ondisk_log_size":181,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575036,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303020,"kb_used_data":1636,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068820,"statfs":{"total":96636764160,"available":96326471680,"internally_reserved":0,"allocated":1675264,"data_stored":1086409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.69399999999999995,"15min":0.70199999999999996},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.79000000000000004,"5min":0.996,"15min":2.8300000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.69199999999999995,"15min":0.70699999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.372},"max":{"1min":0.91600000000000004,"5min":1.073,"15min":1.073},"last":0.59299999999999997}]},{"osd":1,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.45700000000000002,"5min":0.41899999999999998,"15min":0.35799999999999998},"max":{"1min":0.93400000000000005,"5min":0.99099999999999999,"15min":0.998},"last":0.62},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.70599999999999996,"15min":0.71999999999999997},"min":{"1min":0.41999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.94999999999999996,"5min":0.95899999999999996,"15min":1.153},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.71999999999999997,"15min":0.72899999999999998},"min":{"1min":0.54200000000000004,"5min":0.436,"15min":0.436},"max":{"1min":0.872,"5min":1.03,"15min":1.034},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.70499999999999996,"15min":0.71599999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.0049999999999999},"last":0.68500000000000005}]},{"osd":3,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.71999999999999997,"15min":0.74399999999999999},"min":{"1min":0.496,"5min":0.48299999999999998,"15min":0.40300000000000002},"max":{"1min":0.91700000000000004,"5min":1.032,"15min":2.9119999999999999},"last":0.875},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.71899999999999997,"15min":0.73899999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.95899999999999996,"5min":1.008,"15min":1.008},"last":0.72099999999999997}]},{"osd":4,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.42299999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":1.089},"last":0.75},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63400000000000001,"15min":0.63200000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.85299999999999998,"5min":0.90700000000000003,"15min":1.1020000000000001},"last":1.002}]},{"osd":5,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.34599999999999997},"max":{"1min":0.90400000000000003,"5min":0.92200000000000004,"15min":1.0589999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.66600000000000004,"15min":0.67800000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.40400000000000003},"max":{"1min":0.94499999999999995,"5min":1.0089999999999999,"15min":1.032},"last":0.83499999999999996}]},{"osd":6,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.74099999999999999,"15min":0.75800000000000001},"min":{"1min":0.54800000000000004,"5min":0.52000000000000002,"15min":0.46999999999999997},"max":{"1min":0.97499999999999998,"5min":1.0469999999999999,"15min":1.1359999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.72399999999999998,"15min":0.73899999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42099999999999999},"max":{"1min":0.90800000000000003,"5min":1.028,"15min":1.1299999999999999},"last":0.97599999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575035,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302440,"kb_used_data":1056,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069400,"statfs":{"total":96636764160,"available":96327065600,"internally_reserved":0,"allocated":1081344,"data_stored":495545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.66100000000000003,"15min":0.66500000000000004},"min":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.29999999999999999},"max":{"1min":0.81100000000000005,"5min":0.89300000000000002,"15min":0.94799999999999995},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.66600000000000004,"15min":0.68400000000000005},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.68999999999999995}]},{"osd":1,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.68200000000000005,"15min":0.68700000000000006},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.35899999999999999},"max":{"1min":0.997,"5min":1.163,"15min":1.163},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68400000000000005,"15min":0.69099999999999995},"min":{"1min":0.49299999999999999,"5min":0.435,"15min":0.39900000000000002},"max":{"1min":0.92600000000000005,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.61899999999999999}]},{"osd":2,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.70599999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.38600000000000001},"max":{"1min":0.90200000000000002,"5min":0.97199999999999998,"15min":1.016},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.70599999999999996,"15min":0.72099999999999997},"min":{"1min":0.48999999999999999,"5min":0.47799999999999998,"15min":0.373},"max":{"1min":0.89500000000000002,"5min":0.96699999999999997,"15min":0.995},"last":0.56000000000000005}]},{"osd":3,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71699999999999997,"15min":0.73299999999999998},"min":{"1min":0.48099999999999998,"5min":0.46600000000000003,"15min":0.42799999999999999},"max":{"1min":0.94799999999999995,"5min":0.95799999999999996,"15min":1.3260000000000001},"last":0.503},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.73299999999999998,"15min":0.73699999999999999},"min":{"1min":0.52000000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.88800000000000001,"5min":0.95899999999999996,"15min":1.0049999999999999},"last":0.58899999999999997}]},{"osd":4,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.42999999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84199999999999997,"5min":0.874,"15min":1.212},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57699999999999996,"15min":0.60499999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.374},"max":{"1min":0.76700000000000002,"5min":0.96399999999999997,"15min":1.8320000000000001},"last":0.63900000000000001}]},{"osd":5,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64200000000000002,"15min":0.65400000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68799999999999994,"15min":0.70499999999999996},"min":{"1min":0.52300000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":1.7929999999999999},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69099999999999995,"15min":0.71299999999999997},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.41099999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":1.278},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.73099999999999998,"15min":0.749},"min":{"1min":0.46700000000000003,"5min":0.45900000000000002,"15min":0.44700000000000001},"max":{"1min":1.0009999999999999,"5min":1.022,"15min":1.1599999999999999},"last":0.753}]}]},{"osd":1,"up_from":13,"seq":55834575037,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302452,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069388,"statfs":{"total":96636764160,"available":96327053312,"internally_reserved":0,"allocated":1093632,"data_stored":497577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.309,"5min":0.30599999999999999,"15min":0.26900000000000002},"max":{"1min":0.755,"5min":0.85999999999999999,"15min":0.97699999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56999999999999995,"15min":0.57999999999999996},"min":{"1min":0.42699999999999999,"5min":0.41199999999999998,"15min":0.373},"max":{"1min":0.82299999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.504}]},{"osd":2,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48299999999999998,"15min":0.48899999999999999},"min":{"1min":0.33100000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.65800000000000003,"5min":1.113,"15min":1.113},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.375,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71499999999999997,"5min":0.82399999999999995,"15min":1.25},"last":0.33700000000000002}]},{"osd":3,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.60399999999999998,"15min":0.61699999999999999},"min":{"1min":0.41499999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.77600000000000002,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.371,"15min":0.34899999999999998},"max":{"1min":0.88100000000000001,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.45400000000000001}]},{"osd":4,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.75900000000000001,"15min":0.78600000000000003},"min":{"1min":0.504,"5min":0.439,"15min":0.439},"max":{"1min":1.0620000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.80900000000000005,"15min":0.82799999999999996},"min":{"1min":0.65100000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.069,"5min":1.246,"15min":1.3680000000000001},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76900000000000002,"15min":0.79900000000000004},"min":{"1min":0.54300000000000004,"5min":0.502,"15min":0.46999999999999997},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.7849999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.76800000000000002,"15min":0.79400000000000004},"min":{"1min":0.56299999999999994,"5min":0.52700000000000002,"15min":0.50700000000000001},"max":{"1min":1.052,"5min":1.1339999999999999,"15min":1.371},"last":0.72499999999999998}]},{"osd":6,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.79800000000000004,"15min":0.81899999999999995},"min":{"1min":0.56100000000000005,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.413},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.80500000000000005,"15min":0.82499999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.93899999999999995,"5min":1.155,"15min":1.228},"last":0.70999999999999996}]},{"osd":7,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.81999999999999995,"15min":0.84599999999999997},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.52700000000000002},"max":{"1min":1.0149999999999999,"5min":1.129,"15min":1.226},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.82099999999999995,"15min":0.83799999999999997},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.51400000000000001},"max":{"1min":1.099,"5min":1.1339999999999999,"15min":1.321},"last":0.68799999999999994}]}]},{"osd":0,"up_from":13,"seq":55834575036,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303000,"kb_used_data":1616,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068840,"statfs":{"total":96636764160,"available":96326492160,"internally_reserved":0,"allocated":1654784,"data_stored":1079505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.5,"15min":0.5},"min":{"1min":0.317,"5min":0.29999999999999999,"15min":0.24299999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.92700000000000005},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56599999999999995,"15min":0.57399999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.35299999999999998},"max":{"1min":0.89600000000000002,"5min":0.90500000000000003,"15min":0.92100000000000004},"last":0.42699999999999999}]},{"osd":2,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.29099999999999998},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54400000000000004,"15min":0.55300000000000005},"min":{"1min":0.41699999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.32100000000000001}]},{"osd":3,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.33600000000000002},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.629,"15min":0.627},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.39600000000000002},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.44700000000000001}]},{"osd":4,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.79100000000000004,"15min":0.78900000000000003},"min":{"1min":0.502,"5min":0.45700000000000002,"15min":0.437},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.80200000000000005,"15min":0.79100000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.46899999999999997},"max":{"1min":1.095,"5min":1.095,"15min":1.2350000000000001},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.80200000000000005,"15min":0.80800000000000005},"min":{"1min":0.57699999999999996,"5min":0.52500000000000002,"15min":0.505},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.82199999999999995,"15min":0.81699999999999995},"min":{"1min":0.58799999999999997,"5min":0.49099999999999999,"15min":0.47799999999999998},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.59899999999999998}]},{"osd":6,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81100000000000005,"15min":0.80200000000000005},"min":{"1min":0.60199999999999998,"5min":0.50900000000000001,"15min":0.497},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.2689999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82699999999999996,"15min":0.83399999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.44900000000000001},"max":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.621}]},{"osd":7,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.84199999999999997,"15min":0.84099999999999997},"min":{"1min":0.61399999999999999,"5min":0.57999999999999996,"15min":0.51400000000000001},"max":{"1min":1.371,"5min":1.371,"15min":1.446},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.85099999999999998,"15min":0.84299999999999997},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.57699999999999996},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.298},"last":0.66700000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575035,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306540,"kb_used_data":1060,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065300,"statfs":{"total":96636764160,"available":96322867200,"internally_reserved":0,"allocated":1085440,"data_stored":496989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47599999999999998,"15min":0.47199999999999998},"min":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.26000000000000001},"max":{"1min":0.67900000000000005,"5min":0.73499999999999999,"15min":0.86099999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51800000000000002,"15min":0.54500000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.28899999999999998},"max":{"1min":0.67100000000000004,"5min":0.81699999999999995,"15min":1.4119999999999999},"last":0.36699999999999999}]},{"osd":1,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54200000000000004,"15min":0.55000000000000004},"min":{"1min":0.40400000000000003,"5min":0.36399999999999999,"15min":0.30399999999999999},"max":{"1min":0.73199999999999998,"5min":0.80300000000000005,"15min":0.90000000000000002},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.58599999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38100000000000001},"max":{"1min":0.79100000000000004,"5min":0.92900000000000005,"15min":1.5640000000000001},"last":0.44400000000000001}]},{"osd":3,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.38100000000000001,"5min":0.36199999999999999,"15min":0.32900000000000001},"max":{"1min":0.72399999999999998,"5min":0.82499999999999996,"15min":1.1220000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.61199999999999999,"15min":0.63800000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.41799999999999998},"max":{"1min":0.77500000000000002,"5min":0.95299999999999996,"15min":1.22},"last":0.41099999999999998}]},{"osd":4,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.77100000000000002,"15min":0.78900000000000003},"min":{"1min":0.58699999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.026,"5min":1.0580000000000001,"15min":1.488},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.76100000000000001,"15min":0.78100000000000003},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89200000000000002,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.68200000000000005}]},{"osd":5,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.81499999999999995,"15min":0.83999999999999997},"min":{"1min":0.63500000000000001,"5min":0.622,"15min":0.51600000000000001},"max":{"1min":1.105,"5min":1.214,"15min":2.9209999999999998},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.83099999999999996,"15min":0.84999999999999998},"min":{"1min":0.63500000000000001,"5min":0.55500000000000005,"15min":0.55400000000000005},"max":{"1min":1.036,"5min":1.2,"15min":1.845},"last":0.70099999999999996}]},{"osd":6,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.79800000000000004,"15min":0.80200000000000005},"min":{"1min":0.56299999999999994,"5min":0.54200000000000004,"15min":0.51700000000000002},"max":{"1min":0.91400000000000003,"5min":1.1919999999999999,"15min":1.3560000000000001},"last":0.752},{"interface":"front","average":{"1min":0.747,"5min":0.80700000000000005,"15min":0.83299999999999996},"min":{"1min":0.55800000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.012,"5min":1.123,"15min":2.968},"last":0.53800000000000003}]},{"osd":7,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.85299999999999998,"15min":0.872},"min":{"1min":0.62,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.056,"5min":1.2629999999999999,"15min":1.6359999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.83199999999999996,"15min":0.86199999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.0720000000000001,"5min":1.2270000000000001,"15min":1.391},"last":0.72399999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575035,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306536,"kb_used_data":1056,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065304,"statfs":{"total":96636764160,"available":96322871296,"internally_reserved":0,"allocated":1081344,"data_stored":495565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.499,"15min":0.48499999999999999},"min":{"1min":0.34999999999999998,"5min":0.29499999999999998,"15min":0.27600000000000002},"max":{"1min":0.80300000000000005,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52500000000000002,"15min":0.54600000000000004},"min":{"1min":0.39300000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.873,"5min":0.873,"15min":1.0109999999999999},"last":0.52900000000000003}]},{"osd":1,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.47799999999999998,"15min":0.49099999999999999},"min":{"1min":0.33900000000000002,"5min":0.30299999999999999,"15min":0.25900000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.39600000000000002,"5min":0.36299999999999999,"15min":0.32300000000000001},"max":{"1min":0.92500000000000004,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.48799999999999999}]},{"osd":2,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57399999999999995,"15min":0.58399999999999996},"min":{"1min":0.42799999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81299999999999994,"5min":0.90900000000000003,"15min":0.97599999999999998},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.61699999999999999,"15min":0.628},"min":{"1min":0.45900000000000002,"5min":0.44700000000000001,"15min":0.32000000000000001},"max":{"1min":0.74299999999999999,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.64000000000000001}]},{"osd":4,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.71899999999999997,"15min":0.74299999999999999},"min":{"1min":0.54100000000000004,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.012,"5min":1.038,"15min":1.351},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77400000000000002,"15min":0.78200000000000003},"min":{"1min":0.60199999999999998,"5min":0.503,"15min":0.503},"max":{"1min":1.1459999999999999,"5min":1.1819999999999999,"15min":1.198},"last":0.61499999999999999}]},{"osd":5,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76400000000000001,"15min":0.78900000000000003},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.51900000000000002},"max":{"1min":1.0649999999999999,"5min":1.139,"15min":1.3779999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.79400000000000004,"15min":0.80900000000000005},"min":{"1min":0.623,"5min":0.49199999999999999,"15min":0.44900000000000001},"max":{"1min":1.24,"5min":1.2809999999999999,"15min":1.4239999999999999},"last":0.68300000000000005}]},{"osd":6,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.81399999999999995,"15min":0.83499999999999996},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.52600000000000002},"max":{"1min":1.0820000000000001,"5min":1.335,"15min":1.927},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.78700000000000003,"15min":0.80600000000000005},"min":{"1min":0.57999999999999996,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.93999999999999995,"5min":1.2230000000000001,"15min":3.5579999999999998},"last":0.73499999999999999}]},{"osd":7,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81000000000000005,"15min":0.84299999999999997},"min":{"1min":0.59199999999999997,"5min":0.52800000000000002,"15min":0.46600000000000003},"max":{"1min":1.0660000000000001,"5min":1.1579999999999999,"15min":3.5089999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.84599999999999997,"15min":0.86099999999999999},"min":{"1min":0.60599999999999998,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.21},"last":0.77600000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575036,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310628,"kb_used_data":1052,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94061212,"statfs":{"total":96636764160,"available":96318681088,"internally_reserved":0,"allocated":1077248,"data_stored":495523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.72699999999999998,"15min":0.73799999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.28100000000000003},"max":{"1min":0.874,"5min":0.95999999999999996,"15min":1.524},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72799999999999998,"15min":0.74299999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.38600000000000001},"max":{"1min":0.91500000000000004,"5min":1.0049999999999999,"15min":1.177},"last":0.83899999999999997}]},{"osd":1,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.69899999999999995,"15min":0.70399999999999996},"min":{"1min":0.47399999999999998,"5min":0.46200000000000002,"15min":0.33300000000000002},"max":{"1min":0.89400000000000002,"5min":1.0069999999999999,"15min":1.1499999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.69199999999999995,"15min":0.72299999999999998},"min":{"1min":0.43099999999999999,"5min":0.379,"15min":0.36299999999999999},"max":{"1min":0.94899999999999995,"5min":0.99199999999999999,"15min":2.4740000000000002},"last":0.59199999999999997}]},{"osd":2,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.71699999999999997},"min":{"1min":0.46600000000000003,"5min":0.46000000000000002,"15min":0.38900000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.72699999999999998,"15min":0.72999999999999998},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.89400000000000002,"5min":0.97699999999999998,"15min":1.212},"last":0.81599999999999995}]},{"osd":3,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.68700000000000006,"15min":0.70599999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.38100000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":1.2649999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.73999999999999999,"15min":0.754},"min":{"1min":0.502,"5min":0.502,"15min":0.47499999999999998},"max":{"1min":0.84399999999999997,"5min":1.026,"15min":1.069},"last":0.85199999999999998}]},{"osd":5,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60399999999999998,"15min":0.61799999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.85399999999999998,"5min":0.873,"15min":1.0700000000000001},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.622,"15min":0.63300000000000001},"min":{"1min":0.45000000000000001,"5min":0.38200000000000001,"15min":0.32600000000000001},"max":{"1min":0.94299999999999995,"5min":0.97799999999999998,"15min":0.98199999999999998},"last":0.92100000000000004}]},{"osd":6,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.65700000000000003,"15min":0.65600000000000003},"min":{"1min":0.40500000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0680000000000001,"5min":1.133,"15min":1.133},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.69199999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95099999999999996,"5min":1.0269999999999999,"15min":1.119},"last":0.90400000000000003}]},{"osd":7,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.69299999999999995},"min":{"1min":0.51500000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.99399999999999999,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.73999999999999999,"15min":0.74299999999999999},"min":{"1min":0.56999999999999995,"5min":0.46200000000000002,"15min":0.42299999999999999},"max":{"1min":0.91000000000000003,"5min":1.2350000000000001,"15min":1.4279999999999999},"last":0.93799999999999994}]}]},{"osd":5,"up_from":13,"seq":55834575036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302436,"kb_used_data":1052,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069404,"statfs":{"total":96636764160,"available":96327069696,"internally_reserved":0,"allocated":1077248,"data_stored":495543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.69199999999999995,"15min":0.70799999999999996},"min":{"1min":0.39300000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85499999999999998,"5min":1.0249999999999999,"15min":1.032},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.69599999999999995,"15min":0.70999999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.879,"5min":0.96199999999999997,"15min":1.696},"last":0.80000000000000004}]},{"osd":1,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66400000000000003,"15min":0.69899999999999995},"min":{"1min":0.42799999999999999,"5min":0.39400000000000002,"15min":0.36399999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":1.4359999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.70799999999999996,"15min":0.73199999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.05,"5min":1.05,"15min":4.4969999999999999},"last":0.746}]},{"osd":2,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70699999999999996,"15min":0.71099999999999997},"min":{"1min":0.55100000000000005,"5min":0.435,"15min":0.435},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.238},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69499999999999995,"15min":0.746},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.41599999999999998},"max":{"1min":0.94199999999999995,"5min":1.0049999999999999,"15min":8.0429999999999993},"last":0.86399999999999999}]},{"osd":3,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.70199999999999996,"15min":0.755},"min":{"1min":0.47599999999999998,"5min":0.46200000000000002,"15min":0.442},"max":{"1min":1.208,"5min":1.208,"15min":8.0839999999999996},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.748,"15min":0.747},"min":{"1min":0.56399999999999995,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.84999999999999998}]},{"osd":4,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.38300000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":1.113},"last":0.498},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64500000000000002,"15min":0.64000000000000001},"min":{"1min":0.41499999999999998,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.626}]},{"osd":6,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.45700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.034},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67700000000000005,"15min":0.68600000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.085,"5min":1.085,"15min":1.143},"last":0.67100000000000004}]},{"osd":7,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.73399999999999999,"15min":0.73699999999999999},"min":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.437},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77300000000000002,"15min":0.77000000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.88800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:26:06.129 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-11T08:26:06.130 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-11T08:26:06.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.375+0000 7f922cdff700 1 Processor -- start 2024-01-11T08:26:06.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.375+0000 7f922cdff700 1 -- start start 2024-01-11T08:26:06.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.376+0000 7f922cdff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f92280b2b90 0x7f9228159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:06.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.376+0000 7f922cdff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92280cd730 con 0x7f92280b03e0 2024-01-11T08:26:06.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.376+0000 7f922cdff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92280ccb20 con 0x7f92280b07b0 2024-01-11T08:26:06.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.376+0000 7f922cdff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92280cce00 con 0x7f92280b2b90 2024-01-11T08:26:06.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.376+0000 7f922659c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f92280b03e0 0x7f92280b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58430/0 (socket says 172.21.15.27:58430) 2024-01-11T08:26:06.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.376+0000 7f922659c700 1 -- 172.21.15.27:0/1259724132 learned_addr learned my addr 172.21.15.27:0/1259724132 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:06.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.376+0000 7f9225d9b700 1 --2- 172.21.15.27:0/1259724132 >> v2:172.21.15.27:3300/0 conn(0x7f92280b2b90 0x7f9228159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:06.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.377+0000 7f922559a700 1 -- 172.21.15.27:0/1259724132 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637754963 0 0) 0x7f92280cd730 con 0x7f92280b03e0 2024-01-11T08:26:06.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.377+0000 7f922559a700 1 -- 172.21.15.27:0/1259724132 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9210003570 con 0x7f92280b03e0 2024-01-11T08:26:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.377+0000 7f9225d9b700 1 -- 172.21.15.27:0/1259724132 >> v1:172.21.15.77:6789/0 conn(0x7f92280b07b0 legacy=0x7f9228159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.377+0000 7f9225d9b700 1 -- 172.21.15.27:0/1259724132 >> v1:172.21.15.27:6789/0 conn(0x7f92280b03e0 legacy=0x7f92280b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.377+0000 7f9225d9b700 1 -- 172.21.15.27:0/1259724132 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92280cd0e0 con 0x7f92280b2b90 2024-01-11T08:26:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.377+0000 7f9225d9b700 1 --2- 172.21.15.27:0/1259724132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92280b2b90 0x7f9228159200 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f921c0098e0 tx=0x7f921c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=504b8478da9f8ef0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.377+0000 7f922559a700 1 -- 172.21.15.27:0/1259724132 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c00b640 con 0x7f92280b2b90 2024-01-11T08:26:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.378+0000 7f922559a700 1 -- 172.21.15.27:0/1259724132 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f921c00b7a0 con 0x7f92280b2b90 2024-01-11T08:26:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.378+0000 7f922559a700 1 -- 172.21.15.27:0/1259724132 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c00bb40 con 0x7f92280b2b90 2024-01-11T08:26:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.378+0000 7f922cdff700 1 -- 172.21.15.27:0/1259724132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92280b2b90 msgr2=0x7f9228159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.378+0000 7f922cdff700 1 --2- 172.21.15.27:0/1259724132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92280b2b90 0x7f9228159200 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f921c0098e0 tx=0x7f921c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.378+0000 7f922cdff700 1 -- 172.21.15.27:0/1259724132 shutdown_connections 2024-01-11T08:26:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.378+0000 7f922cdff700 1 --2- 172.21.15.27:0/1259724132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92280b2b90 0x7f9228159200 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.378+0000 7f922cdff700 1 -- 172.21.15.27:0/1259724132 >> 172.21.15.27:0/1259724132 conn(0x7f922806ed40 msgr2=0x7f922806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:06.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.378+0000 7f922cdff700 1 -- 172.21.15.27:0/1259724132 shutdown_connections 2024-01-11T08:26:06.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.378+0000 7f922cdff700 1 -- 172.21.15.27:0/1259724132 wait complete. 2024-01-11T08:26:06.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.380+0000 7f922cdff700 1 Processor -- start 2024-01-11T08:26:06.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.381+0000 7f922cdff700 1 -- start start 2024-01-11T08:26:06.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.382+0000 7f922cdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92280b03e0 0x7f922815b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.382+0000 7f922cdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f92280b07b0 0x7f92281470e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.382+0000 7f922cdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92280b2b90 0x7f92280cf290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.382+0000 7f922cdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922815cb80 con 0x7f92280b03e0 2024-01-11T08:26:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.382+0000 7f922cdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922815bf70 con 0x7f92280b2b90 2024-01-11T08:26:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.382+0000 7f922cdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922815c250 con 0x7f92280b07b0 2024-01-11T08:26:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.382+0000 7f922659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92280b03e0 0x7f922815b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.382+0000 7f922659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92280b03e0 0x7f922815b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50866/0 (socket says 172.21.15.27:50866) 2024-01-11T08:26:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.382+0000 7f922659c700 1 -- 172.21.15.27:0/3895211881 learned_addr learned my addr 172.21.15.27:0/3895211881 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.382+0000 7f9226d9d700 1 --2- 172.21.15.27:0/3895211881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92280b2b90 0x7f92280cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.383+0000 7f9225d9b700 1 --2- 172.21.15.27:0/3895211881 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f92280b07b0 0x7f92281470e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.383+0000 7f922659c700 1 -- 172.21.15.27:0/3895211881 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f92280b07b0 msgr2=0x7f92281470e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.383+0000 7f922659c700 1 --2- 172.21.15.27:0/3895211881 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f92280b07b0 0x7f92281470e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.383+0000 7f922659c700 1 -- 172.21.15.27:0/3895211881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92280b2b90 msgr2=0x7f92280cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.383+0000 7f922659c700 1 --2- 172.21.15.27:0/3895211881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92280b2b90 0x7f92280cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:06.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.383+0000 7f922659c700 1 -- 172.21.15.27:0/3895211881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921c009420 con 0x7f92280b03e0 2024-01-11T08:26:06.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.383+0000 7f9226d9d700 1 --2- 172.21.15.27:0/3895211881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92280b2b90 0x7f92280cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:26:06.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.383+0000 7f922659c700 1 --2- 172.21.15.27:0/3895211881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92280b03e0 0x7f922815b8e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f921800c7c0 tx=0x7f9218007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:06.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.383+0000 7f92167fc700 1 -- 172.21.15.27:0/3895211881 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9218005b30 con 0x7f92280b03e0 2024-01-11T08:26:06.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.383+0000 7f92167fc700 1 -- 172.21.15.27:0/3895211881 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9218005c90 con 0x7f92280b03e0 2024-01-11T08:26:06.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.383+0000 7f922cdff700 1 -- 172.21.15.27:0/3895211881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f922815c530 con 0x7f92280b03e0 2024-01-11T08:26:06.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.383+0000 7f92167fc700 1 -- 172.21.15.27:0/3895211881 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9218007340 con 0x7f92280b03e0 2024-01-11T08:26:06.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.383+0000 7f922cdff700 1 -- 172.21.15.27:0/3895211881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9228147620 con 0x7f92280b03e0 2024-01-11T08:26:06.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.385+0000 7f922cdff700 1 -- 172.21.15.27:0/3895211881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f922804c3c0 con 0x7f92280b03e0 2024-01-11T08:26:06.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.385+0000 7f92167fc700 1 -- 172.21.15.27:0/3895211881 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f921801b400 con 0x7f92280b03e0 2024-01-11T08:26:06.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.387+0000 7f92167fc700 1 --2- 172.21.15.27:0/3895211881 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9208066f30 0x7f92080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:06.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.387+0000 7f92167fc700 1 -- 172.21.15.27:0/3895211881 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f9218084110 con 0x7f92280b03e0 2024-01-11T08:26:06.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.387+0000 7f9225d9b700 1 --2- 172.21.15.27:0/3895211881 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9208066f30 0x7f92080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:06.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.388+0000 7f9225d9b700 1 --2- 172.21.15.27:0/3895211881 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9208066f30 0x7f92080693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f921c0093f0 tx=0x7f921c017000 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:06.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.390+0000 7f92167fc700 1 -- 172.21.15.27:0/3895211881 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9218051340 con 0x7f92280b03e0 2024-01-11T08:26:06.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.672+0000 7f922cdff700 1 -- 172.21.15.27:0/3895211881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f92280cd3c0 con 0x7f92280b03e0 2024-01-11T08:26:06.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.678+0000 7f92167fc700 1 -- 172.21.15.27:0/3895211881 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+0 (secure 0 0 0) 0x7f9218054960 con 0x7f92280b03e0 2024-01-11T08:26:06.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.681+0000 7f922cdff700 1 -- 172.21.15.27:0/3895211881 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9208066f30 msgr2=0x7f92080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:06.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.681+0000 7f922cdff700 1 --2- 172.21.15.27:0/3895211881 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9208066f30 0x7f92080693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f921c0093f0 tx=0x7f921c017000 comp rx=0 tx=0).stop 2024-01-11T08:26:06.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.681+0000 7f922cdff700 1 -- 172.21.15.27:0/3895211881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92280b03e0 msgr2=0x7f922815b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:06.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.681+0000 7f922cdff700 1 --2- 172.21.15.27:0/3895211881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92280b03e0 0x7f922815b8e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f921800c7c0 tx=0x7f9218007e00 comp rx=0 tx=0).stop 2024-01-11T08:26:06.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.681+0000 7f922cdff700 1 -- 172.21.15.27:0/3895211881 shutdown_connections 2024-01-11T08:26:06.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.681+0000 7f922cdff700 1 --2- 172.21.15.27:0/3895211881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92280b2b90 0x7f92280cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:06.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.681+0000 7f922cdff700 1 --2- 172.21.15.27:0/3895211881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92280b03e0 0x7f922815b8e0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:06.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.681+0000 7f922cdff700 1 --2- 172.21.15.27:0/3895211881 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9208066f30 0x7f92080693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:06.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.681+0000 7f922cdff700 1 --2- 172.21.15.27:0/3895211881 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f92280b07b0 0x7f92281470e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:06.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.681+0000 7f922cdff700 1 -- 172.21.15.27:0/3895211881 >> 172.21.15.27:0/3895211881 conn(0x7f922806ed40 msgr2=0x7f92280703a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:06.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.681+0000 7f922cdff700 1 -- 172.21.15.27:0/3895211881 shutdown_connections 2024-01-11T08:26:06.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.681+0000 7f922cdff700 1 -- 172.21.15.27:0/3895211881 wait complete. 2024-01-11T08:26:06.682 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-11T08:26:06.699 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:26:06.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.897+0000 7f01047ff700 1 Processor -- start 2024-01-11T08:26:06.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.898+0000 7f01047ff700 1 -- start start 2024-01-11T08:26:06.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.899+0000 7f01047ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f00fc0b03e0 0x7f00fc0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:06.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.899+0000 7f01047ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f00fc0b07b0 0x7f00fc0b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:06.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.899+0000 7f01047ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00fc0cd730 con 0x7f00fc159200 2024-01-11T08:26:06.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.899+0000 7f01047ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00fc159d30 con 0x7f00fc0b07b0 2024-01-11T08:26:06.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.899+0000 7f01047ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00fc0ccb20 con 0x7f00fc0b03e0 2024-01-11T08:26:06.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.899+0000 7f0101d9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f00fc0b07b0 0x7f00fc0b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:06.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.899+0000 7f0101d9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f00fc0b07b0 0x7f00fc0b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50868/0 (socket says 172.21.15.27:50868) 2024-01-11T08:26:06.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.899+0000 7f0101d9a700 1 -- 172.21.15.27:0/1456278117 learned_addr learned my addr 172.21.15.27:0/1456278117 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:06.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.899+0000 7f010259b700 1 --2- 172.21.15.27:0/1456278117 >> v2:172.21.15.77:3300/0 conn(0x7f00fc0b03e0 0x7f00fc0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:06.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.900+0000 7f0101d9a700 1 -- 172.21.15.27:0/1456278117 >> v2:172.21.15.77:3300/0 conn(0x7f00fc0b03e0 msgr2=0x7f00fc0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:06.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.900+0000 7f0101d9a700 1 --2- 172.21.15.27:0/1456278117 >> v2:172.21.15.77:3300/0 conn(0x7f00fc0b03e0 0x7f00fc0b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:06.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.900+0000 7f0101d9a700 1 -- 172.21.15.27:0/1456278117 >> v1:172.21.15.77:6789/0 conn(0x7f00fc159200 legacy=0x7f00fc159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:06.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.900+0000 7f0101d9a700 1 -- 172.21.15.27:0/1456278117 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00fc0cce00 con 0x7f00fc0b07b0 2024-01-11T08:26:06.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.900+0000 7f010259b700 1 --2- 172.21.15.27:0/1456278117 >> v2:172.21.15.77:3300/0 conn(0x7f00fc0b03e0 0x7f00fc0b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:26:06.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.900+0000 7f0101d9a700 1 --2- 172.21.15.27:0/1456278117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00fc0b07b0 0x7f00fc0b29c0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f00fc089470 tx=0x7f00ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11348c0cdcdb760f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:06.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.900+0000 7f0101599700 1 -- 172.21.15.27:0/1456278117 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec00b860 con 0x7f00fc0b07b0 2024-01-11T08:26:06.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.900+0000 7f0101599700 1 -- 172.21.15.27:0/1456278117 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00ec00b9c0 con 0x7f00fc0b07b0 2024-01-11T08:26:06.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.900+0000 7f0101599700 1 -- 172.21.15.27:0/1456278117 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec00bd60 con 0x7f00fc0b07b0 2024-01-11T08:26:06.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.901+0000 7f01047ff700 1 -- 172.21.15.27:0/1456278117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00fc0b07b0 msgr2=0x7f00fc0b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:06.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.901+0000 7f01047ff700 1 --2- 172.21.15.27:0/1456278117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00fc0b07b0 0x7f00fc0b29c0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f00fc089470 tx=0x7f00ec009250 comp rx=0 tx=0).stop 2024-01-11T08:26:06.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.901+0000 7f01047ff700 1 -- 172.21.15.27:0/1456278117 shutdown_connections 2024-01-11T08:26:06.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.901+0000 7f01047ff700 1 --2- 172.21.15.27:0/1456278117 >> v2:172.21.15.77:3300/0 conn(0x7f00fc0b03e0 0x7f00fc0b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:06.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.901+0000 7f01047ff700 1 --2- 172.21.15.27:0/1456278117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00fc0b07b0 0x7f00fc0b29c0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:06.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.901+0000 7f01047ff700 1 -- 172.21.15.27:0/1456278117 >> 172.21.15.27:0/1456278117 conn(0x7f00fc06ed40 msgr2=0x7f00fc06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:06.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.901+0000 7f01047ff700 1 -- 172.21.15.27:0/1456278117 shutdown_connections 2024-01-11T08:26:06.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.901+0000 7f01047ff700 1 -- 172.21.15.27:0/1456278117 wait complete. 2024-01-11T08:26:06.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.903+0000 7f01047ff700 1 Processor -- start 2024-01-11T08:26:06.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.903+0000 7f01047ff700 1 -- start start 2024-01-11T08:26:06.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.904+0000 7f01047ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00fc0b03e0 0x7f00fc0ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:06.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.904+0000 7f01047ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00fc0b07b0 0x7f00fc0c7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:06.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.904+0000 7f01047ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00fc159200 0x7f00fc157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:06.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.904+0000 7f01047ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc0c5660 con 0x7f00fc0b03e0 2024-01-11T08:26:06.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.904+0000 7f01047ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc0c8020 con 0x7f00fc159200 2024-01-11T08:26:06.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.904+0000 7f01047ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc0c7b70 con 0x7f00fc0b07b0 2024-01-11T08:26:06.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.904+0000 7f010259b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00fc0b03e0 0x7f00fc0ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:06.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.904+0000 7f010259b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00fc0b03e0 0x7f00fc0ca4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50884/0 (socket says 172.21.15.27:50884) 2024-01-11T08:26:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.904+0000 7f010259b700 1 -- 172.21.15.27:0/1099778010 learned_addr learned my addr 172.21.15.27:0/1099778010 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.904+0000 7f0101d9a700 1 --2- 172.21.15.27:0/1099778010 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00fc0b07b0 0x7f00fc0c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.904+0000 7f0102d9c700 1 --2- 172.21.15.27:0/1099778010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00fc159200 0x7f00fc157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.904+0000 7f010259b700 1 -- 172.21.15.27:0/1099778010 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00fc0b07b0 msgr2=0x7f00fc0c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.904+0000 7f010259b700 1 --2- 172.21.15.27:0/1099778010 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00fc0b07b0 0x7f00fc0c7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.904+0000 7f010259b700 1 -- 172.21.15.27:0/1099778010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00fc159200 msgr2=0x7f00fc157e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.904+0000 7f010259b700 1 --2- 172.21.15.27:0/1099778010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00fc159200 0x7f00fc157e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.904+0000 7f010259b700 1 -- 172.21.15.27:0/1099778010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec0094f0 con 0x7f00fc0b03e0 2024-01-11T08:26:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.904+0000 7f0102d9c700 1 --2- 172.21.15.27:0/1099778010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00fc159200 0x7f00fc157e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.905+0000 7f010259b700 1 --2- 172.21.15.27:0/1099778010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00fc0b03e0 0x7f00fc0ca4e0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f00f800e900 tx=0x7f00f8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.905+0000 7f00f27fc700 1 -- 172.21.15.27:0/1099778010 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f800eea0 con 0x7f00fc0b03e0 2024-01-11T08:26:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.905+0000 7f00f27fc700 1 -- 172.21.15.27:0/1099778010 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f800c5e0 con 0x7f00fc0b03e0 2024-01-11T08:26:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.905+0000 7f01047ff700 1 -- 172.21.15.27:0/1099778010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00fc0c84b0 con 0x7f00fc0b03e0 2024-01-11T08:26:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.905+0000 7f00f27fc700 1 -- 172.21.15.27:0/1099778010 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f800c980 con 0x7f00fc0b03e0 2024-01-11T08:26:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.905+0000 7f01047ff700 1 -- 172.21.15.27:0/1099778010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00fc0c5380 con 0x7f00fc0b03e0 2024-01-11T08:26:06.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.906+0000 7f01047ff700 1 -- 172.21.15.27:0/1099778010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00fc0718d0 con 0x7f00fc0b03e0 2024-01-11T08:26:06.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.907+0000 7f00f27fc700 1 -- 172.21.15.27:0/1099778010 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f00f8013410 con 0x7f00fc0b03e0 2024-01-11T08:26:06.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.908+0000 7f00f27fc700 1 --2- 172.21.15.27:0/1099778010 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f00e0066f30 0x7f00e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:06.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.908+0000 7f00f27fc700 1 -- 172.21.15.27:0/1099778010 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f00f8084350 con 0x7f00fc0b03e0 2024-01-11T08:26:06.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.908+0000 7f0101d9a700 1 --2- 172.21.15.27:0/1099778010 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f00e0066f30 0x7f00e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:06.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.909+0000 7f0101d9a700 1 --2- 172.21.15.27:0/1099778010 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f00e0066f30 0x7f00e00693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f00fc089470 tx=0x7f00ec017000 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:06.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:06.912+0000 7f00f27fc700 1 -- 172.21.15.27:0/1099778010 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f00f80515a0 con 0x7f00fc0b03e0 2024-01-11T08:26:07.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.171+0000 7f01047ff700 1 -- 172.21.15.27:0/1099778010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00fc0c8300 con 0x7f00fc0b03e0 2024-01-11T08:26:07.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.171+0000 7f00f27fc700 1 -- 172.21.15.27:0/1099778010 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4596 (secure 0 0 0) 0x7f00f8054bc0 con 0x7f00fc0b03e0 2024-01-11T08:26:07.172 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:07.172 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5945,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1366930493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1366930493},{"type":"v1","addr":"172.21.15.77:6837","nonce":1366930493}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/516828593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":516828593},{"type":"v1","addr":"172.21.15.27:6835","nonce":516828593}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6624,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1462523541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1462523541},{"type":"v1","addr":"172.21.15.27:6834","nonce":1462523541}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:26:00.710104+0000","modified":"2024-01-11T08:26:04.230562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6627":{"gid":6627,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6835/3498364963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3498364963},{"type":"v1","addr":"172.21.15.77:6835","nonce":3498364963}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2024-01-11T08:26:07.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.174+0000 7f01047ff700 1 -- 172.21.15.27:0/1099778010 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f00e0066f30 msgr2=0x7f00e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:07.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.174+0000 7f01047ff700 1 --2- 172.21.15.27:0/1099778010 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f00e0066f30 0x7f00e00693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f00fc089470 tx=0x7f00ec017000 comp rx=0 tx=0).stop 2024-01-11T08:26:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.174+0000 7f01047ff700 1 -- 172.21.15.27:0/1099778010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00fc0b03e0 msgr2=0x7f00fc0ca4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.174+0000 7f01047ff700 1 --2- 172.21.15.27:0/1099778010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00fc0b03e0 0x7f00fc0ca4e0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f00f800e900 tx=0x7f00f8008040 comp rx=0 tx=0).stop 2024-01-11T08:26:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.174+0000 7f01047ff700 1 -- 172.21.15.27:0/1099778010 shutdown_connections 2024-01-11T08:26:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.174+0000 7f01047ff700 1 --2- 172.21.15.27:0/1099778010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00fc159200 0x7f00fc157e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.174+0000 7f01047ff700 1 --2- 172.21.15.27:0/1099778010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00fc0b03e0 0x7f00fc0ca4e0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.174+0000 7f01047ff700 1 --2- 172.21.15.27:0/1099778010 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f00e0066f30 0x7f00e00693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.174+0000 7f01047ff700 1 --2- 172.21.15.27:0/1099778010 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00fc0b07b0 0x7f00fc0c7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.174+0000 7f01047ff700 1 -- 172.21.15.27:0/1099778010 >> 172.21.15.27:0/1099778010 conn(0x7f00fc06ed40 msgr2=0x7f00fc0b2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.174+0000 7f01047ff700 1 -- 172.21.15.27:0/1099778010 shutdown_connections 2024-01-11T08:26:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.174+0000 7f01047ff700 1 -- 172.21.15.27:0/1099778010 wait complete. 2024-01-11T08:26:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 42 2024-01-11T08:26:07.182 INFO:tasks.ceph.mds.d.smithi077.stderr:2024-01-11T08:26:07.182+0000 7fac26ff1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T08:26:07.595 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 42, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-11T08:26:00.710104+0000', 'modified': '2024-01-11T08:26:04.230562+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6627}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6627': {'gid': 6627, 'name': 'd', 'rank': 0, 'incarnation': 38, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6835/3498364963', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6832', 'nonce': 3498364963}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 3498364963}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-11T08:26:07.595 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-11T08:26:07.595 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-01-11T08:26:07.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.796+0000 7fefee3ff700 1 Processor -- start 2024-01-11T08:26:07.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.796+0000 7fefee3ff700 1 -- start start 2024-01-11T08:26:07.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.797+0000 7fefee3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fefe815ca10 0x7fefe815a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.797+0000 7fefee3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fefe8068cd0 0x7fefe80b5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.797+0000 7fefee3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe80b4bd0 con 0x7fefe815cde0 2024-01-11T08:26:07.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.797+0000 7fefee3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe815ac70 con 0x7fefe815ca10 2024-01-11T08:26:07.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.797+0000 7fefee3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe80b3fc0 con 0x7fefe8068cd0 2024-01-11T08:26:07.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.797+0000 7fefe7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fefe815ca10 0x7fefe815a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.797+0000 7fefe7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fefe815ca10 0x7fefe815a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50890/0 (socket says 172.21.15.27:50890) 2024-01-11T08:26:07.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.797+0000 7fefe7fff700 1 -- 172.21.15.27:0/836737756 learned_addr learned my addr 172.21.15.27:0/836737756 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:07.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.798+0000 7fefec99c700 1 --2- 172.21.15.27:0/836737756 >> v2:172.21.15.77:3300/0 conn(0x7fefe8068cd0 0x7fefe80b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.798+0000 7fefe7fff700 1 -- 172.21.15.27:0/836737756 >> v2:172.21.15.77:3300/0 conn(0x7fefe8068cd0 msgr2=0x7fefe80b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:07.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.798+0000 7fefe7fff700 1 --2- 172.21.15.27:0/836737756 >> v2:172.21.15.77:3300/0 conn(0x7fefe8068cd0 0x7fefe80b5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.798+0000 7fefe7fff700 1 -- 172.21.15.27:0/836737756 >> v1:172.21.15.77:6789/0 conn(0x7fefe815cde0 legacy=0x7fefe80685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:07.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.798+0000 7fefe7fff700 1 -- 172.21.15.27:0/836737756 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe80b42a0 con 0x7fefe815ca10 2024-01-11T08:26:07.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.798+0000 7fefe7fff700 1 --2- 172.21.15.27:0/836737756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe815ca10 0x7fefe815a730 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fefe8159d00 tx=0x7fefd8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6c0266d8737dc63 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:07.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.799+0000 7fefe6ffd700 1 -- 172.21.15.27:0/836737756 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd800b860 con 0x7fefe815ca10 2024-01-11T08:26:07.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.799+0000 7fefe6ffd700 1 -- 172.21.15.27:0/836737756 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd800b9c0 con 0x7fefe815ca10 2024-01-11T08:26:07.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.799+0000 7fefe6ffd700 1 -- 172.21.15.27:0/836737756 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd800bd60 con 0x7fefe815ca10 2024-01-11T08:26:07.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.799+0000 7fefee3ff700 1 -- 172.21.15.27:0/836737756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe815ca10 msgr2=0x7fefe815a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:07.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.799+0000 7fefee3ff700 1 --2- 172.21.15.27:0/836737756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe815ca10 0x7fefe815a730 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fefe8159d00 tx=0x7fefd8009250 comp rx=0 tx=0).stop 2024-01-11T08:26:07.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.799+0000 7fefee3ff700 1 -- 172.21.15.27:0/836737756 shutdown_connections 2024-01-11T08:26:07.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.799+0000 7fefee3ff700 1 --2- 172.21.15.27:0/836737756 >> v2:172.21.15.77:3300/0 conn(0x7fefe8068cd0 0x7fefe80b5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.799+0000 7fefee3ff700 1 --2- 172.21.15.27:0/836737756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe815ca10 0x7fefe815a730 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.799+0000 7fefee3ff700 1 -- 172.21.15.27:0/836737756 >> 172.21.15.27:0/836737756 conn(0x7fefe806ed40 msgr2=0x7fefe806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:07.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.799+0000 7fefee3ff700 1 -- 172.21.15.27:0/836737756 shutdown_connections 2024-01-11T08:26:07.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.799+0000 7fefee3ff700 1 -- 172.21.15.27:0/836737756 wait complete. 2024-01-11T08:26:07.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.801+0000 7fefee3ff700 1 Processor -- start 2024-01-11T08:26:07.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.802+0000 7fefee3ff700 1 -- start start 2024-01-11T08:26:07.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.803+0000 7fefee3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefe8068cd0 0x7fefe816cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.803+0000 7fefee3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefe815ca10 0x7fefe81555e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.803+0000 7fefee3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe815cde0 0x7fefe8155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.803+0000 7fefee3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe814f7f0 con 0x7fefe815cde0 2024-01-11T08:26:07.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.803+0000 7fefee3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe816a950 con 0x7fefe8068cd0 2024-01-11T08:26:07.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.803+0000 7fefee3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe814f1a0 con 0x7fefe815ca10 2024-01-11T08:26:07.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.803+0000 7fefec99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe815cde0 0x7fefe8155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.803+0000 7fefec99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe815cde0 0x7fefe8155b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50892/0 (socket says 172.21.15.27:50892) 2024-01-11T08:26:07.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.803+0000 7fefec99c700 1 -- 172.21.15.27:0/2371414684 learned_addr learned my addr 172.21.15.27:0/2371414684 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:07.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.803+0000 7fefe7fff700 1 --2- 172.21.15.27:0/2371414684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefe8068cd0 0x7fefe816cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.803+0000 7fefe77fe700 1 --2- 172.21.15.27:0/2371414684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefe815ca10 0x7fefe81555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.804+0000 7fefec99c700 1 -- 172.21.15.27:0/2371414684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefe815ca10 msgr2=0x7fefe81555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:07.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.804+0000 7fefec99c700 1 --2- 172.21.15.27:0/2371414684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefe815ca10 0x7fefe81555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.804+0000 7fefec99c700 1 -- 172.21.15.27:0/2371414684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefe8068cd0 msgr2=0x7fefe816cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:07.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.804+0000 7fefec99c700 1 --2- 172.21.15.27:0/2371414684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefe8068cd0 0x7fefe816cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.804+0000 7fefec99c700 1 -- 172.21.15.27:0/2371414684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd80094f0 con 0x7fefe815cde0 2024-01-11T08:26:07.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.804+0000 7fefec99c700 1 --2- 172.21.15.27:0/2371414684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe815cde0 0x7fefe8155b20 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fefe0007f30 tx=0x7fefe00098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:07.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.804+0000 7fefcffff700 1 -- 172.21.15.27:0/2371414684 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe000eb90 con 0x7fefe815cde0 2024-01-11T08:26:07.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.804+0000 7fefcffff700 1 -- 172.21.15.27:0/2371414684 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe000ecf0 con 0x7fefe815cde0 2024-01-11T08:26:07.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.804+0000 7fefcffff700 1 -- 172.21.15.27:0/2371414684 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe0014670 con 0x7fefe815cde0 2024-01-11T08:26:07.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.804+0000 7fefee3ff700 1 -- 172.21.15.27:0/2371414684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe814f480 con 0x7fefe815cde0 2024-01-11T08:26:07.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.805+0000 7fefee3ff700 1 -- 172.21.15.27:0/2371414684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe816af20 con 0x7fefe815cde0 2024-01-11T08:26:07.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.807+0000 7fefcffff700 1 -- 172.21.15.27:0/2371414684 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fefe0014a20 con 0x7fefe815cde0 2024-01-11T08:26:07.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.807+0000 7fefcffff700 1 --2- 172.21.15.27:0/2371414684 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fefd4066f30 0x7fefd40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.808+0000 7fefcffff700 1 -- 172.21.15.27:0/2371414684 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5733+0+0 (secure 0 0 0) 0x7fefe0084c30 con 0x7fefe815cde0 2024-01-11T08:26:07.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.808+0000 7fefe7fff700 1 --2- 172.21.15.27:0/2371414684 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fefd4066f30 0x7fefd40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.809+0000 7fefe7fff700 1 --2- 172.21.15.27:0/2371414684 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fefd4066f30 0x7fefd40693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fefe8159d00 tx=0x7fefd8017000 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:07.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.809+0000 7feffcbbeb80 1 Processor -- start 2024-01-11T08:26:07.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.809+0000 7feffcbbeb80 1 -- start start 2024-01-11T08:26:07.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.810+0000 7feffcbbeb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e095b20 0x55c55e076df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.810+0000 7feffcbbeb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e0713a0 0x55c55e09d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.810+0000 7feffcbbeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095ef0 0x55c55e09dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.810+0000 7feffcbbeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c55df8e310 con 0x55c55e095ef0 2024-01-11T08:26:07.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.810+0000 7feffcbbeb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c55dfcb130 con 0x55c55e095b20 2024-01-11T08:26:07.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.810+0000 7feffcbbeb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c55dfc8af0 con 0x55c55e0713a0 2024-01-11T08:26:07.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.811+0000 7fefec99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095ef0 0x55c55e09dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.811+0000 7fefe7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e095b20 0x55c55e076df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.811+0000 7fefec99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095ef0 0x55c55e09dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50902/0 (socket says 172.21.15.27:50902) 2024-01-11T08:26:07.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.811+0000 7fefe7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e095b20 0x55c55e076df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:44888/0 (socket says 172.21.15.27:44888) 2024-01-11T08:26:07.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.811+0000 7fefec99c700 1 -- 172.21.15.27:0/520839011 learned_addr learned my addr 172.21.15.27:0/520839011 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:07.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.811+0000 7fefe77fe700 1 --2- 172.21.15.27:0/520839011 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e0713a0 0x55c55e09d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.811+0000 7fefec99c700 1 -- 172.21.15.27:0/520839011 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e0713a0 msgr2=0x55c55e09d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.811+0000 7fefec99c700 1 --2- 172.21.15.27:0/520839011 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e0713a0 0x55c55e09d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.811+0000 7fefec99c700 1 -- 172.21.15.27:0/520839011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e095b20 msgr2=0x55c55e076df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.811+0000 7fefec99c700 1 --2- 172.21.15.27:0/520839011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e095b20 0x55c55e076df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.811+0000 7fefec99c700 1 -- 172.21.15.27:0/520839011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c55deb70a0 con 0x55c55e095ef0 2024-01-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.811+0000 7fefe77fe700 1 --2- 172.21.15.27:0/520839011 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e0713a0 0x55c55e09d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.811+0000 7fefec99c700 1 --2- 172.21.15.27:0/520839011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095ef0 0x55c55e09dda0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fefe000c040 tx=0x7fefe0082ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b14ebc0a8dbeee server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.812+0000 7fefcd7fa700 1 -- 172.21.15.27:0/520839011 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe0055aa0 con 0x55c55e095ef0 2024-01-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.812+0000 7fefcd7fa700 1 -- 172.21.15.27:0/520839011 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe0058cf0 con 0x55c55e095ef0 2024-01-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.812+0000 7fefcd7fa700 1 -- 172.21.15.27:0/520839011 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe005ca40 con 0x55c55e095ef0 2024-01-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.812+0000 7feffcbbeb80 1 -- 172.21.15.27:0/520839011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095ef0 msgr2=0x55c55e09dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.812+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/520839011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095ef0 0x55c55e09dda0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fefe000c040 tx=0x7fefe0082ff0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.812+0000 7feffcbbeb80 1 -- 172.21.15.27:0/520839011 shutdown_connections 2024-01-11T08:26:07.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.812+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/520839011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095ef0 0x55c55e09dda0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.812+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/520839011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e095b20 0x55c55e076df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.812+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/520839011 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e0713a0 0x55c55e09d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.812+0000 7feffcbbeb80 1 -- 172.21.15.27:0/520839011 >> 172.21.15.27:0/520839011 conn(0x55c55e073300 msgr2=0x55c55e075bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:07.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.812+0000 7feffcbbeb80 1 -- 172.21.15.27:0/520839011 shutdown_connections 2024-01-11T08:26:07.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.812+0000 7feffcbbeb80 1 -- 172.21.15.27:0/520839011 wait complete. 2024-01-11T08:26:07.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.813+0000 7feffcbbeb80 1 -- start start 2024-01-11T08:26:07.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.814+0000 7feffcbbeb80 1 Processor -- start 2024-01-11T08:26:07.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.815+0000 7feffcbbeb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e0713a0 0x55c55e095070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.815+0000 7feffcbbeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095b20 0x55c55e0a4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.815+0000 7feffcbbeb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e095ef0 0x55c55e09f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.815+0000 7feffcbbeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c55dfc48e0 con 0x55c55e095b20 2024-01-11T08:26:07.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.815+0000 7feffcbbeb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c55df70980 con 0x55c55e095ef0 2024-01-11T08:26:07.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.815+0000 7feffcbbeb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c55df70620 con 0x55c55e0713a0 2024-01-11T08:26:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.815+0000 7feffcbbeb80 10 client.0 _ll_register_callbacks cb 0x55c55ddd3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T08:26:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.816+0000 7fefe77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095b20 0x55c55e0a4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.816+0000 7fefe77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095b20 0x55c55e0a4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50916/0 (socket says 172.21.15.27:50916) 2024-01-11T08:26:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.816+0000 7fefe77fe700 1 -- 172.21.15.27:0/2364341556 learned_addr learned my addr 172.21.15.27:0/2364341556 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.816+0000 7fefe7fff700 1 --2- 172.21.15.27:0/2364341556 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e0713a0 0x55c55e095070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.816+0000 7fefec99c700 1 --2- 172.21.15.27:0/2364341556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e095ef0 0x55c55e09f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.816+0000 7fefe77fe700 1 -- 172.21.15.27:0/2364341556 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e0713a0 msgr2=0x55c55e095070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.816+0000 7fefe77fe700 1 --2- 172.21.15.27:0/2364341556 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e0713a0 0x55c55e095070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.816+0000 7fefe77fe700 1 -- 172.21.15.27:0/2364341556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e095ef0 msgr2=0x55c55e09f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.816+0000 7fefe77fe700 1 --2- 172.21.15.27:0/2364341556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e095ef0 0x55c55e09f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.816+0000 7fefe77fe700 1 -- 172.21.15.27:0/2364341556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe816af20 con 0x55c55e095b20 2024-01-11T08:26:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.816+0000 7fefe77fe700 1 --2- 172.21.15.27:0/2364341556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095b20 0x55c55e0a4200 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fefe815d260 tx=0x7fefdc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffc173f5ec463f96 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.816+0000 7fefc67fc700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T08:26:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.817+0000 7fefc67fc700 1 -- 172.21.15.27:0/2364341556 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefdc01a070 con 0x55c55e095b20 2024-01-11T08:26:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.817+0000 7fefc67fc700 1 -- 172.21.15.27:0/2364341556 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 5733+0+0 (secure 0 0 0) 0x7fefdc011e50 con 0x55c55e095b20 2024-01-11T08:26:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.817+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2364341556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c55e069560 con 0x55c55e095b20 2024-01-11T08:26:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.817+0000 7fefc67fc700 1 -- 172.21.15.27:0/2364341556 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7fefdc006c40 con 0x55c55e095b20 2024-01-11T08:26:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.817+0000 7feffcbbeb80 10 client.6756 fetch_fsmap learned FSMap version 42 2024-01-11T08:26:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.817+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2364341556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x55c55df933c0 con 0x55c55e095b20 2024-01-11T08:26:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.817+0000 7fefc67fc700 1 -- 172.21.15.27:0/2364341556 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefdc024020 con 0x55c55e095b20 2024-01-11T08:26:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.817+0000 7fefc67fc700 1 -- 172.21.15.27:0/2364341556 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 42) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fefdc01d020 con 0x55c55e095b20 2024-01-11T08:26:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.817+0000 7feffcbbeb80 10 client.6756 fetch_fsmap finished waiting for FSMap version 42 2024-01-11T08:26:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.817+0000 7feffcbbeb80 10 client.6756 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-11T08:26:07.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.817+0000 7feffcbbeb80 20 client.6756 populate_metadata read hostname 'smithi027' 2024-01-11T08:26:07.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.817+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/2364341556 >> [v2:172.21.15.77:6832/3498364963,v1:172.21.15.77:6835/3498364963] conn(0x55c55e0a3260 0x55c55e0a3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.817+0000 7feffcbbeb80 4 client.6756 mds_command: new command op to 6627 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-11T08:26:07.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.817+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2364341556 --> [v2:172.21.15.77:6832/3498364963,v1:172.21.15.77:6835/3498364963] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c55e0460e0 con 0x55c55e0a3260 2024-01-11T08:26:07.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.818+0000 7fefe7fff700 1 --2- 172.21.15.27:0/2364341556 >> [v2:172.21.15.77:6832/3498364963,v1:172.21.15.77:6835/3498364963] conn(0x55c55e0a3260 0x55c55e0a3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.818+0000 7fefe7fff700 1 --2- 172.21.15.27:0/2364341556 >> [v2:172.21.15.77:6832/3498364963,v1:172.21.15.77:6835/3498364963] conn(0x55c55e0a3260 0x55c55e0a3630 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f664a3beb909864 server_cookie=941a7f63dedb625e in_seq=0 out_seq=0 2024-01-11T08:26:07.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.818+0000 7fefc67fc700 10 client.6756 ms_handle_connect on v2:172.21.15.77:6832/3498364963 2024-01-11T08:26:07.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.819+0000 7fefc67fc700 1 -- 172.21.15.27:0/2364341556 <== mds.0 v2:172.21.15.77:6832/3498364963 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x55c55e0460e0 con 0x55c55e0a3260 2024-01-11T08:26:07.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.819+0000 7fefc67fc700 10 client.6756 handle_command_reply: tid=0 2024-01-11T08:26:07.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.819+0000 7feffcbbeb80 1 client.6756 shutdown 2024-01-11T08:26:07.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.819+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2364341556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095b20 msgr2=0x55c55e0a4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:07.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.819+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/2364341556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095b20 0x55c55e0a4200 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fefe815d260 tx=0x7fefdc009040 comp rx=0 tx=0).stop 2024-01-11T08:26:07.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.819+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2364341556 shutdown_connections 2024-01-11T08:26:07.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.819+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/2364341556 >> [v2:172.21.15.77:6832/3498364963,v1:172.21.15.77:6835/3498364963] conn(0x55c55e0a3260 0x55c55e0a3630 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.819+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/2364341556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e095ef0 0x55c55e09f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.819+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/2364341556 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e0713a0 0x55c55e095070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.819+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/2364341556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095b20 0x55c55e0a4200 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.819+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2364341556 >> 172.21.15.27:0/2364341556 conn(0x55c55e073300 msgr2=0x55c55e0763a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:07.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.819+0000 7fefc67fc700 0 client.6756 ms_handle_reset on v2:172.21.15.77:6832/3498364963 2024-01-11T08:26:07.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.820+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2364341556 shutdown_connections 2024-01-11T08:26:07.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.820+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2364341556 wait complete. 2024-01-11T08:26:07.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.820+0000 7feffcbbeb80 20 client.6756 trim_cache size 0 max 16384 2024-01-11T08:26:07.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.837+0000 7feffcbbeb80 1 Processor -- start 2024-01-11T08:26:07.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.837+0000 7feffcbbeb80 1 -- start start 2024-01-11T08:26:07.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.838+0000 7feffcbbeb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e073300 0x55c55e0a3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.838+0000 7feffcbbeb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e0713a0 0x55c55e0940a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.838+0000 7feffcbbeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095b20 0x55c55e0945e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.838+0000 7feffcbbeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c55de5c1e0 con 0x55c55e095b20 2024-01-11T08:26:07.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.838+0000 7feffcbbeb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c55e0cb400 con 0x55c55e0713a0 2024-01-11T08:26:07.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.838+0000 7feffcbbeb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c55e09d0d0 con 0x55c55e073300 2024-01-11T08:26:07.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.838+0000 7fefec99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095b20 0x55c55e0945e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.838+0000 7fefec99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095b20 0x55c55e0945e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50918/0 (socket says 172.21.15.27:50918) 2024-01-11T08:26:07.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.838+0000 7fefec99c700 1 -- 172.21.15.27:0/2919587615 learned_addr learned my addr 172.21.15.27:0/2919587615 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:07.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.839+0000 7fefe7fff700 1 --2- 172.21.15.27:0/2919587615 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e073300 0x55c55e0a3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.839+0000 7fefe77fe700 1 --2- 172.21.15.27:0/2919587615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e0713a0 0x55c55e0940a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.839+0000 7fefec99c700 1 -- 172.21.15.27:0/2919587615 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e073300 msgr2=0x55c55e0a3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:07.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.839+0000 7fefec99c700 1 --2- 172.21.15.27:0/2919587615 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e073300 0x55c55e0a3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.839+0000 7fefec99c700 1 -- 172.21.15.27:0/2919587615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e0713a0 msgr2=0x55c55e0940a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:07.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.839+0000 7fefec99c700 1 --2- 172.21.15.27:0/2919587615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e0713a0 0x55c55e0940a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.839+0000 7fefec99c700 1 -- 172.21.15.27:0/2919587615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c55de2b1d0 con 0x55c55e095b20 2024-01-11T08:26:07.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.839+0000 7fefec99c700 1 --2- 172.21.15.27:0/2919587615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095b20 0x55c55e0945e0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fefe0083120 tx=0x7fefe006f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a98e268cc0e3ed2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:07.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.839+0000 7fefcdffb700 1 -- 172.21.15.27:0/2919587615 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe0055870 con 0x55c55e095b20 2024-01-11T08:26:07.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.839+0000 7fefcdffb700 1 -- 172.21.15.27:0/2919587615 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe0055aa0 con 0x55c55e095b20 2024-01-11T08:26:07.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.839+0000 7fefcdffb700 1 -- 172.21.15.27:0/2919587615 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe005ca40 con 0x55c55e095b20 2024-01-11T08:26:07.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.840+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2919587615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095b20 msgr2=0x55c55e0945e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:07.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.840+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/2919587615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095b20 0x55c55e0945e0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fefe0083120 tx=0x7fefe006f000 comp rx=0 tx=0).stop 2024-01-11T08:26:07.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.840+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2919587615 shutdown_connections 2024-01-11T08:26:07.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.840+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/2919587615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e095b20 0x55c55e0945e0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.840+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/2919587615 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e073300 0x55c55e0a3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.840+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/2919587615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e0713a0 0x55c55e0940a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.840+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2919587615 >> 172.21.15.27:0/2919587615 conn(0x55c55e09d400 msgr2=0x55c55e09c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:07.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.840+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2919587615 shutdown_connections 2024-01-11T08:26:07.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.840+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2919587615 wait complete. 2024-01-11T08:26:07.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.841+0000 7feffcbbeb80 1 -- start start 2024-01-11T08:26:07.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.841+0000 7feffcbbeb80 1 Processor -- start 2024-01-11T08:26:07.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.842+0000 7feffcbbeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e0713a0 0x55c55e0a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.842+0000 7feffcbbeb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e073300 0x55c55e0a1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.842+0000 7feffcbbeb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e095b20 0x55c55e0ca050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.842+0000 7feffcbbeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c55e1218e0 con 0x55c55e0713a0 2024-01-11T08:26:07.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.842+0000 7feffcbbeb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c55e120110 con 0x55c55e095b20 2024-01-11T08:26:07.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.842+0000 7feffcbbeb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c55e0949c0 con 0x55c55e073300 2024-01-11T08:26:07.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.842+0000 7feffcbbeb80 10 client.0 _ll_register_callbacks cb 0x55c55ddd3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T08:26:07.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.842+0000 7fefe7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e0713a0 0x55c55e0a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.842+0000 7fefe7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e0713a0 0x55c55e0a1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50932/0 (socket says 172.21.15.27:50932) 2024-01-11T08:26:07.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.842+0000 7fefe7fff700 1 -- 172.21.15.27:0/2666933444 learned_addr learned my addr 172.21.15.27:0/2666933444 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:07.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.842+0000 7fefe77fe700 1 --2- 172.21.15.27:0/2666933444 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e073300 0x55c55e0a1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.842+0000 7fefec99c700 1 --2- 172.21.15.27:0/2666933444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e095b20 0x55c55e0ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.843+0000 7fefe7fff700 1 -- 172.21.15.27:0/2666933444 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e073300 msgr2=0x55c55e0a1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:07.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.843+0000 7fefe7fff700 1 --2- 172.21.15.27:0/2666933444 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e073300 0x55c55e0a1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.843+0000 7fefe7fff700 1 -- 172.21.15.27:0/2666933444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e095b20 msgr2=0x55c55e0ca050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:07.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.843+0000 7fefe7fff700 1 --2- 172.21.15.27:0/2666933444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e095b20 0x55c55e0ca050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.843+0000 7fefe7fff700 1 -- 172.21.15.27:0/2666933444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c55deb70a0 con 0x55c55e0713a0 2024-01-11T08:26:07.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.843+0000 7fefe7fff700 1 --2- 172.21.15.27:0/2666933444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e0713a0 0x55c55e0a1a10 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fefe8088fe0 tx=0x7fefd802b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceeafd920fb266f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:07.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.843+0000 7fefc67fc700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T08:26:07.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.843+0000 7fefc67fc700 1 -- 172.21.15.27:0/2666933444 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd8021070 con 0x55c55e0713a0 2024-01-11T08:26:07.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.843+0000 7fefc67fc700 1 -- 172.21.15.27:0/2666933444 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 5733+0+0 (secure 0 0 0) 0x7fefd8036500 con 0x55c55e0713a0 2024-01-11T08:26:07.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.843+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2666933444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fefe0058cf0 con 0x55c55e0713a0 2024-01-11T08:26:07.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.844+0000 7fefc67fc700 1 -- 172.21.15.27:0/2666933444 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7fefd80368d0 con 0x55c55e0713a0 2024-01-11T08:26:07.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.844+0000 7feffcbbeb80 10 client.6762 fetch_fsmap learned FSMap version 42 2024-01-11T08:26:07.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.844+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2666933444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x55c55de04d60 con 0x55c55e0713a0 2024-01-11T08:26:07.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.844+0000 7fefc67fc700 1 -- 172.21.15.27:0/2666933444 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe0058cf0 con 0x55c55e0713a0 2024-01-11T08:26:07.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.844+0000 7fefc67fc700 1 -- 172.21.15.27:0/2666933444 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 42) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fefd8036a30 con 0x55c55e0713a0 2024-01-11T08:26:07.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.844+0000 7feffcbbeb80 10 client.6762 fetch_fsmap finished waiting for FSMap version 42 2024-01-11T08:26:07.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.844+0000 7feffcbbeb80 10 client.6762 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-11T08:26:07.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.844+0000 7feffcbbeb80 20 client.6762 populate_metadata read hostname 'smithi027' 2024-01-11T08:26:07.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.844+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/2666933444 >> [v2:172.21.15.77:6832/3498364963,v1:172.21.15.77:6835/3498364963] conn(0x55c55e095ef0 0x55c55e0ca590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:07.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.844+0000 7feffcbbeb80 4 client.6762 mds_command: new command op to 6627 tid=0[{"prefix": "status"}] 2024-01-11T08:26:07.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.844+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2666933444 --> [v2:172.21.15.77:6832/3498364963,v1:172.21.15.77:6835/3498364963] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c55df678d0 con 0x55c55e095ef0 2024-01-11T08:26:07.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.845+0000 7fefe77fe700 1 --2- 172.21.15.27:0/2666933444 >> [v2:172.21.15.77:6832/3498364963,v1:172.21.15.77:6835/3498364963] conn(0x55c55e095ef0 0x55c55e0ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:07.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.845+0000 7fefe77fe700 1 --2- 172.21.15.27:0/2666933444 >> [v2:172.21.15.77:6832/3498364963,v1:172.21.15.77:6835/3498364963] conn(0x55c55e095ef0 0x55c55e0ca590 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fef0e11774d3eeef server_cookie=563ec9350f1da762 in_seq=0 out_seq=0 2024-01-11T08:26:07.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.845+0000 7fefc67fc700 10 client.6762 ms_handle_connect on v2:172.21.15.77:6832/3498364963 2024-01-11T08:26:07.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.846+0000 7fefc67fc700 1 -- 172.21.15.27:0/2666933444 <== mds.0 v2:172.21.15.77:6832/3498364963 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55c55df678d0 con 0x55c55e095ef0 2024-01-11T08:26:07.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.846+0000 7fefc67fc700 10 client.6762 handle_command_reply: tid=0 2024-01-11T08:26:07.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.846+0000 7feffcbbeb80 1 client.6762 shutdown 2024-01-11T08:26:07.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.846+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2666933444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e0713a0 msgr2=0x55c55e0a1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:07.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.846+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/2666933444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e0713a0 0x55c55e0a1a10 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fefe8088fe0 tx=0x7fefd802b040 comp rx=0 tx=0).stop 2024-01-11T08:26:07.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.846+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2666933444 shutdown_connections 2024-01-11T08:26:07.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.846+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/2666933444 >> [v2:172.21.15.77:6832/3498364963,v1:172.21.15.77:6835/3498364963] conn(0x55c55e095ef0 0x55c55e0ca590 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.846+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/2666933444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c55e095b20 0x55c55e0ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.846+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/2666933444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c55e0713a0 0x55c55e0a1a10 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.846+0000 7feffcbbeb80 1 --2- 172.21.15.27:0/2666933444 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55c55e073300 0x55c55e0a1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.846+0000 7fefc67fc700 0 client.6762 ms_handle_reset on v2:172.21.15.77:6832/3498364963 2024-01-11T08:26:07.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.846+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2666933444 >> 172.21.15.27:0/2666933444 conn(0x55c55e09d400 msgr2=0x55c55e0a2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:07.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.846+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2666933444 shutdown_connections 2024-01-11T08:26:07.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.846+0000 7feffcbbeb80 1 -- 172.21.15.27:0/2666933444 wait complete. 2024-01-11T08:26:07.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.846+0000 7feffcbbeb80 20 client.6762 trim_cache size 0 max 16384 2024-01-11T08:26:07.852 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T08:26:07.852 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "8fddb4a7-9d66-40f1-892d-6fe86c728ee4", 2024-01-11T08:26:07.852 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-11T08:26:07.852 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 6627, 2024-01-11T08:26:07.852 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-11T08:26:07.852 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-11T08:26:07.853 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-11T08:26:07.853 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 6.6401978010000002, 2024-01-11T08:26:07.853 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 42, 2024-01-11T08:26:07.853 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 88, 2024-01-11T08:26:07.853 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 88, 2024-01-11T08:26:07.853 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 12.296786021000001 2024-01-11T08:26:07.853 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T08:26:07.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.847+0000 7fefee3ff700 1 -- 172.21.15.27:0/2371414684 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fefd4066f30 msgr2=0x7fefd40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:07.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.847+0000 7fefee3ff700 1 --2- 172.21.15.27:0/2371414684 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fefd4066f30 0x7fefd40693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fefe8159d00 tx=0x7fefd8017000 comp rx=0 tx=0).stop 2024-01-11T08:26:07.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.847+0000 7fefee3ff700 1 -- 172.21.15.27:0/2371414684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe815cde0 msgr2=0x7fefe8155b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:07.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.847+0000 7fefee3ff700 1 --2- 172.21.15.27:0/2371414684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe815cde0 0x7fefe8155b20 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fefe0007f30 tx=0x7fefe00098e0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.847+0000 7fefee3ff700 1 -- 172.21.15.27:0/2371414684 shutdown_connections 2024-01-11T08:26:07.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.847+0000 7fefee3ff700 1 --2- 172.21.15.27:0/2371414684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe815cde0 0x7fefe8155b20 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.847+0000 7fefee3ff700 1 --2- 172.21.15.27:0/2371414684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefe8068cd0 0x7fefe816cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.847+0000 7fefee3ff700 1 --2- 172.21.15.27:0/2371414684 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fefd4066f30 0x7fefd40693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.847+0000 7fefee3ff700 1 --2- 172.21.15.27:0/2371414684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefe815ca10 0x7fefe81555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:07.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.847+0000 7fefee3ff700 1 -- 172.21.15.27:0/2371414684 >> 172.21.15.27:0/2371414684 conn(0x7fefe806ed40 msgr2=0x7fefe80afe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:07.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.847+0000 7fefee3ff700 1 -- 172.21.15.27:0/2371414684 shutdown_connections 2024-01-11T08:26:07.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:07.847+0000 7fefee3ff700 1 -- 172.21.15.27:0/2371414684 wait complete. 2024-01-11T08:26:07.864 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-11T08:26:08.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.063+0000 7f03a47ff700 1 Processor -- start 2024-01-11T08:26:08.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.064+0000 7f03a47ff700 1 -- start start 2024-01-11T08:26:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.065+0000 7f03a47ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f039c0b2e40 0x7f039c06bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.065+0000 7f03a47ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f039c06fba0 con 0x7f039c159a20 2024-01-11T08:26:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.065+0000 7f03a47ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f039c06e700 con 0x7f039c159650 2024-01-11T08:26:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.065+0000 7f03a47ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f039c06c360 con 0x7f039c0b2e40 2024-01-11T08:26:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.065+0000 7f03a1d9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f039c159a20 0x7f039c0b2730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58438/0 (socket says 172.21.15.27:58438) 2024-01-11T08:26:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.065+0000 7f03a1d9a700 1 -- 172.21.15.27:0/3386482133 learned_addr learned my addr 172.21.15.27:0/3386482133 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.065+0000 7f03a2d9c700 1 --2- 172.21.15.27:0/3386482133 >> v2:172.21.15.77:3300/0 conn(0x7f039c0b2e40 0x7f039c06bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.066+0000 7f03a1599700 1 -- 172.21.15.27:0/3386482133 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524492694 0 0) 0x7f039c06fba0 con 0x7f039c159a20 2024-01-11T08:26:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.066+0000 7f03a1599700 1 -- 172.21.15.27:0/3386482133 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0388003570 con 0x7f039c159a20 2024-01-11T08:26:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.066+0000 7f03a2d9c700 1 -- 172.21.15.27:0/3386482133 >> v1:172.21.15.77:6789/0 conn(0x7f039c159650 legacy=0x7f039c0b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:08.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.066+0000 7f03a2d9c700 1 -- 172.21.15.27:0/3386482133 >> v1:172.21.15.27:6789/0 conn(0x7f039c159a20 legacy=0x7f039c0b2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:08.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.066+0000 7f03a2d9c700 1 -- 172.21.15.27:0/3386482133 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039c06c640 con 0x7f039c0b2e40 2024-01-11T08:26:08.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.066+0000 7f03a2d9c700 1 --2- 172.21.15.27:0/3386482133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f039c0b2e40 0x7f039c06bd80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0394009e60 tx=0x7f0394009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a08a8b594d759f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:08.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.067+0000 7f03a1599700 1 -- 172.21.15.27:0/3386482133 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039400e070 con 0x7f039c0b2e40 2024-01-11T08:26:08.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.067+0000 7f03a1599700 1 -- 172.21.15.27:0/3386482133 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039400bb60 con 0x7f039c0b2e40 2024-01-11T08:26:08.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.067+0000 7f03a1599700 1 -- 172.21.15.27:0/3386482133 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0394012430 con 0x7f039c0b2e40 2024-01-11T08:26:08.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.067+0000 7f03a47ff700 1 -- 172.21.15.27:0/3386482133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f039c0b2e40 msgr2=0x7f039c06bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:08.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.067+0000 7f03a47ff700 1 --2- 172.21.15.27:0/3386482133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f039c0b2e40 0x7f039c06bd80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0394009e60 tx=0x7f0394009780 comp rx=0 tx=0).stop 2024-01-11T08:26:08.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.067+0000 7f03a47ff700 1 -- 172.21.15.27:0/3386482133 shutdown_connections 2024-01-11T08:26:08.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.067+0000 7f03a47ff700 1 --2- 172.21.15.27:0/3386482133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f039c0b2e40 0x7f039c06bd80 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:08.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.067+0000 7f03a47ff700 1 -- 172.21.15.27:0/3386482133 >> 172.21.15.27:0/3386482133 conn(0x7f039c06b200 msgr2=0x7f039c0aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:08.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.067+0000 7f03a47ff700 1 -- 172.21.15.27:0/3386482133 shutdown_connections 2024-01-11T08:26:08.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.067+0000 7f03a47ff700 1 -- 172.21.15.27:0/3386482133 wait complete. 2024-01-11T08:26:08.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.069+0000 7f03a47ff700 1 Processor -- start 2024-01-11T08:26:08.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.069+0000 7f03a47ff700 1 -- start start 2024-01-11T08:26:08.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.070+0000 7f03a47ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f039c0b2e40 0x7f039c14f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:08.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.070+0000 7f03a47ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f039c159650 0x7f039c1530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:08.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.070+0000 7f03a47ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f039c159a20 0x7f039c153610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:08.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.070+0000 7f03a47ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c150930 con 0x7f039c0b2e40 2024-01-11T08:26:08.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.070+0000 7f03a47ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c14fd20 con 0x7f039c159650 2024-01-11T08:26:08.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.070+0000 7f03a47ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f039c150000 con 0x7f039c159a20 2024-01-11T08:26:08.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.070+0000 7f03a259b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f039c0b2e40 0x7f039c14f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:08.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.070+0000 7f03a259b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f039c0b2e40 0x7f039c14f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50944/0 (socket says 172.21.15.27:50944) 2024-01-11T08:26:08.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.070+0000 7f03a259b700 1 -- 172.21.15.27:0/3654969335 learned_addr learned my addr 172.21.15.27:0/3654969335 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:08.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.070+0000 7f03a1d9a700 1 --2- 172.21.15.27:0/3654969335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f039c159650 0x7f039c1530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:08.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.070+0000 7f03a2d9c700 1 --2- 172.21.15.27:0/3654969335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f039c159a20 0x7f039c153610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:08.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.071+0000 7f03a259b700 1 -- 172.21.15.27:0/3654969335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f039c159a20 msgr2=0x7f039c153610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:08.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.071+0000 7f03a259b700 1 --2- 172.21.15.27:0/3654969335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f039c159a20 0x7f039c153610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:08.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.071+0000 7f03a259b700 1 -- 172.21.15.27:0/3654969335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f039c159650 msgr2=0x7f039c1530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:08.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.071+0000 7f03a259b700 1 --2- 172.21.15.27:0/3654969335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f039c159650 0x7f039c1530d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:08.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.071+0000 7f03a259b700 1 -- 172.21.15.27:0/3654969335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0394009a20 con 0x7f039c0b2e40 2024-01-11T08:26:08.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.071+0000 7f03a2d9c700 1 --2- 172.21.15.27:0/3654969335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f039c159a20 0x7f039c153610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:08.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.071+0000 7f03a259b700 1 --2- 172.21.15.27:0/3654969335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f039c0b2e40 0x7f039c14f380 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f039800c790 tx=0x7f0398007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:08.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.071+0000 7f03927fc700 1 -- 172.21.15.27:0/3654969335 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0398012630 con 0x7f039c0b2e40 2024-01-11T08:26:08.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.071+0000 7f03927fc700 1 -- 172.21.15.27:0/3654969335 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03980127b0 con 0x7f039c0b2e40 2024-01-11T08:26:08.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.071+0000 7f03a47ff700 1 -- 172.21.15.27:0/3654969335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f039c1502e0 con 0x7f039c0b2e40 2024-01-11T08:26:08.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.071+0000 7f03927fc700 1 -- 172.21.15.27:0/3654969335 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0398012b70 con 0x7f039c0b2e40 2024-01-11T08:26:08.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.071+0000 7f03a47ff700 1 -- 172.21.15.27:0/3654969335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f039c14f8c0 con 0x7f039c0b2e40 2024-01-11T08:26:08.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.072+0000 7f03a47ff700 1 -- 172.21.15.27:0/3654969335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f039c1694f0 con 0x7f039c0b2e40 2024-01-11T08:26:08.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.073+0000 7f03927fc700 1 -- 172.21.15.27:0/3654969335 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f039801b400 con 0x7f039c0b2e40 2024-01-11T08:26:08.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.074+0000 7f03927fc700 1 --2- 172.21.15.27:0/3654969335 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f0380066f30 0x7f03800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:08.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.074+0000 7f03927fc700 1 -- 172.21.15.27:0/3654969335 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f03980840e0 con 0x7f039c0b2e40 2024-01-11T08:26:08.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.075+0000 7f03a1d9a700 1 --2- 172.21.15.27:0/3654969335 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f0380066f30 0x7f03800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:08.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.075+0000 7f03a1d9a700 1 --2- 172.21.15.27:0/3654969335 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f0380066f30 0x7f03800693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f039c153c80 tx=0x7f038c009040 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:08.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.078+0000 7f03927fc700 1 -- 172.21.15.27:0/3654969335 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0398051330 con 0x7f039c0b2e40 2024-01-11T08:26:08.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.326+0000 7f03a47ff700 1 -- 172.21.15.27:0/3654969335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f039c0718d0 con 0x7f039c0b2e40 2024-01-11T08:26:08.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.326+0000 7f03927fc700 1 -- 172.21.15.27:0/3654969335 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v42) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0398054950 con 0x7f039c0b2e40 2024-01-11T08:26:08.326 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-11T08:26:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.328+0000 7f03a47ff700 1 -- 172.21.15.27:0/3654969335 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f0380066f30 msgr2=0x7f03800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.328+0000 7f03a47ff700 1 --2- 172.21.15.27:0/3654969335 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f0380066f30 0x7f03800693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f039c153c80 tx=0x7f038c009040 comp rx=0 tx=0).stop 2024-01-11T08:26:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.328+0000 7f03a47ff700 1 -- 172.21.15.27:0/3654969335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f039c0b2e40 msgr2=0x7f039c14f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.328+0000 7f03a47ff700 1 --2- 172.21.15.27:0/3654969335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f039c0b2e40 0x7f039c14f380 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f039800c790 tx=0x7f0398007f60 comp rx=0 tx=0).stop 2024-01-11T08:26:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.329+0000 7f03a47ff700 1 -- 172.21.15.27:0/3654969335 shutdown_connections 2024-01-11T08:26:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.329+0000 7f03a47ff700 1 --2- 172.21.15.27:0/3654969335 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f0380066f30 0x7f03800693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.329+0000 7f03a47ff700 1 --2- 172.21.15.27:0/3654969335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f039c159a20 0x7f039c153610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.329+0000 7f03a47ff700 1 --2- 172.21.15.27:0/3654969335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f039c0b2e40 0x7f039c14f380 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.329+0000 7f03a47ff700 1 --2- 172.21.15.27:0/3654969335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f039c159650 0x7f039c1530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.329+0000 7f03a47ff700 1 -- 172.21.15.27:0/3654969335 >> 172.21.15.27:0/3654969335 conn(0x7f039c06b200 msgr2=0x7f039c156220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.329+0000 7f03a47ff700 1 -- 172.21.15.27:0/3654969335 shutdown_connections 2024-01-11T08:26:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:08.329+0000 7f03a47ff700 1 -- 172.21.15.27:0/3654969335 wait complete. 2024-01-11T08:26:08.346 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-11T08:26:08.346 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-11T08:26:08.347 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-11T08:26:08.347 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-11T08:26:08.347 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-11T08:26:08.347 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-11T08:26:08.347 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-11T08:26:08.347 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-11T08:26:08.347 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:26:08.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:26:08.347 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-11T08:26:08.396 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T08:26:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T08:26:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T08:26:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:26:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-11T08:26:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:26:08.396 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:26:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:26:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-11T08:26:08.397 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:26:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:26:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-11T08:26:08.397 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-11T08:26:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-11T08:26:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-11T08:26:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-11T08:26:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 41553sec preferred_lft 41553sec 2024-01-11T08:26:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-11T08:26:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:26:08.397 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:26:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-11T08:26:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-11T08:26:08.397 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T08:26:08.397 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff 2024-01-11T08:26:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-11T08:26:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:26:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::acd2:b3ff:fe6a:db4b/64 scope link 2024-01-11T08:26:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:26:08.398 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T08:26:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:fa:85:1b:3a:bc brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-11T08:26:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ecfa:85ff:fe1b:3abc/64 scope link 2024-01-11T08:26:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:26:08.398 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T08:26:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-11T08:26:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ec88:faff:fe17:41aa/64 scope link 2024-01-11T08:26:08.398 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:26:08.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:26:08.399 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-11T08:26:08.469 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-11T08:26:08.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:26:08.472 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-11T08:26:08.522 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-11T08:26:08.523 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-11T08:26:08.523 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-11T08:26:08.523 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-11T08:26:08.523 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-11T08:26:08.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:26:08.524 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T08:26:08.525 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T08:26:08.525 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-11T08:26:08.525 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-11T08:26:08.525 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-11T08:26:08.525 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T08:26:08.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:26:08.701 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T08:26:08.749 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-11T08:26:08.749 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-11T08:26:08.750 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:26:08.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:26:08.750 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-11T08:26:08.823 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:26:08.823 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-11T08:26:08.870 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:26:08.870 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:26:08.871 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-11T08:26:08.918 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-11T08:26:08.918 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-11T08:26:08.918 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:26:09.061 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:26:09.061 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:26:09.061 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:26:09.076 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:26:09.076 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:26:09.076 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:26:09.076 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:26:09.076 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:26:09.098 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:26:09.098 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:26:09.098 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:26:09.098 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:26:09.098 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:26:09.098 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790" 2024-01-11T08:26:09.098 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790:/ 2024-01-11T08:26:09.098 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=8fddb4a7-9d66-40f1-892d-6fe86c728ee4" will pass to kernel 2024-01-11T08:26:09.102 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-11T08:26:09.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:26:09.103 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-11T08:26:09.175 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-11T08:26:09.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.379+0000 7fb31efff700 1 Processor -- start 2024-01-11T08:26:09.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.379+0000 7fb31efff700 1 -- start start 2024-01-11T08:26:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.380+0000 7fb31efff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb318156ab0 0x7fb3180ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.380+0000 7fb31efff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3180b0550 0x7fb318160070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.380+0000 7fb31efff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3180adc80 con 0x7fb3180cc9b0 2024-01-11T08:26:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.380+0000 7fb31efff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3180adf60 con 0x7fb318156ab0 2024-01-11T08:26:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.380+0000 7fb31efff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3180ae240 con 0x7fb3180b0550 2024-01-11T08:26:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.381+0000 7fb31cd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb318156ab0 0x7fb3180ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.381+0000 7fb317fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb3180cc9b0 0x7fb318156e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58452/0 (socket says 172.21.15.27:58452) 2024-01-11T08:26:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.381+0000 7fb317fff700 1 -- 172.21.15.27:0/3716159186 learned_addr learned my addr 172.21.15.27:0/3716159186 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.381+0000 7fb31d59c700 1 --2- 172.21.15.27:0/3716159186 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3180b0550 0x7fb318160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.381+0000 7fb31cd9b700 1 -- 172.21.15.27:0/3716159186 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3180b0550 msgr2=0x7fb318160070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.381+0000 7fb3177fe700 1 -- 172.21.15.27:0/3716159186 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200268541 0 0) 0x7fb3180adc80 con 0x7fb3180cc9b0 2024-01-11T08:26:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.381+0000 7fb31cd9b700 1 --2- 172.21.15.27:0/3716159186 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3180b0550 0x7fb318160070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.381+0000 7fb31cd9b700 1 -- 172.21.15.27:0/3716159186 >> v1:172.21.15.27:6789/0 conn(0x7fb3180cc9b0 legacy=0x7fb318156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.381+0000 7fb31cd9b700 1 -- 172.21.15.27:0/3716159186 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3180ae520 con 0x7fb318156ab0 2024-01-11T08:26:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.381+0000 7fb31cd9b700 1 --2- 172.21.15.27:0/3716159186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318156ab0 0x7fb3180ad650 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0098e0 tx=0x7fb30c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351641f03c2ddac6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.382+0000 7fb3177fe700 1 -- 172.21.15.27:0/3716159186 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c009e80 con 0x7fb318156ab0 2024-01-11T08:26:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.382+0000 7fb3177fe700 1 -- 172.21.15.27:0/3716159186 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30c00b860 con 0x7fb318156ab0 2024-01-11T08:26:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.382+0000 7fb3177fe700 1 -- 172.21.15.27:0/3716159186 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c00bbe0 con 0x7fb318156ab0 2024-01-11T08:26:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.382+0000 7fb31efff700 1 -- 172.21.15.27:0/3716159186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318156ab0 msgr2=0x7fb3180ad650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.382+0000 7fb31efff700 1 --2- 172.21.15.27:0/3716159186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318156ab0 0x7fb3180ad650 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0098e0 tx=0x7fb30c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.382+0000 7fb31efff700 1 -- 172.21.15.27:0/3716159186 shutdown_connections 2024-01-11T08:26:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.382+0000 7fb31efff700 1 --2- 172.21.15.27:0/3716159186 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3180b0550 0x7fb318160070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.382+0000 7fb31efff700 1 --2- 172.21.15.27:0/3716159186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318156ab0 0x7fb3180ad650 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.382+0000 7fb31efff700 1 -- 172.21.15.27:0/3716159186 >> 172.21.15.27:0/3716159186 conn(0x7fb3180680b0 msgr2=0x7fb3180cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.382+0000 7fb31efff700 1 -- 172.21.15.27:0/3716159186 shutdown_connections 2024-01-11T08:26:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.383+0000 7fb31efff700 1 -- 172.21.15.27:0/3716159186 wait complete. 2024-01-11T08:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.384+0000 7fb31efff700 1 Processor -- start 2024-01-11T08:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.384+0000 7fb31efff700 1 -- start start 2024-01-11T08:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.385+0000 7fb31efff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3180b0550 0x7fb318149330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.385+0000 7fb31efff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3180cc9b0 0x7fb3180c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.385+0000 7fb31efff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb318156ab0 0x7fb318157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.385+0000 7fb31efff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3180c9fe0 con 0x7fb3180cc9b0 2024-01-11T08:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.385+0000 7fb31efff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3180ca3c0 con 0x7fb3180b0550 2024-01-11T08:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.385+0000 7fb31efff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3180c9740 con 0x7fb318156ab0 2024-01-11T08:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.385+0000 7fb317fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3180cc9b0 0x7fb3180c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.385+0000 7fb317fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3180cc9b0 0x7fb3180c8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50964/0 (socket says 172.21.15.27:50964) 2024-01-11T08:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.385+0000 7fb317fff700 1 -- 172.21.15.27:0/3814629235 learned_addr learned my addr 172.21.15.27:0/3814629235 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.385+0000 7fb31cd9b700 1 --2- 172.21.15.27:0/3814629235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3180b0550 0x7fb318149330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.385+0000 7fb31d59c700 1 --2- 172.21.15.27:0/3814629235 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb318156ab0 0x7fb318157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.386+0000 7fb317fff700 1 -- 172.21.15.27:0/3814629235 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb318156ab0 msgr2=0x7fb318157b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.386+0000 7fb317fff700 1 --2- 172.21.15.27:0/3814629235 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb318156ab0 0x7fb318157b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.386+0000 7fb317fff700 1 -- 172.21.15.27:0/3814629235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3180b0550 msgr2=0x7fb318149330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.386+0000 7fb317fff700 1 --2- 172.21.15.27:0/3814629235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3180b0550 0x7fb318149330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.386+0000 7fb317fff700 1 -- 172.21.15.27:0/3814629235 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb30c009420 con 0x7fb3180cc9b0 2024-01-11T08:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.386+0000 7fb31d59c700 1 --2- 172.21.15.27:0/3814629235 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb318156ab0 0x7fb318157b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.386+0000 7fb317fff700 1 --2- 172.21.15.27:0/3814629235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3180cc9b0 0x7fb3180c8dd0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fb30400c5e0 tx=0x7fb304001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.386+0000 7fb314ff9700 1 -- 172.21.15.27:0/3814629235 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb304005b30 con 0x7fb3180cc9b0 2024-01-11T08:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.386+0000 7fb314ff9700 1 -- 172.21.15.27:0/3814629235 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb304005c90 con 0x7fb3180cc9b0 2024-01-11T08:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.386+0000 7fb314ff9700 1 -- 172.21.15.27:0/3814629235 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb304007300 con 0x7fb3180cc9b0 2024-01-11T08:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.386+0000 7fb31efff700 1 -- 172.21.15.27:0/3814629235 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3180c9a20 con 0x7fb3180cc9b0 2024-01-11T08:26:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.386+0000 7fb31efff700 1 -- 172.21.15.27:0/3814629235 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb318149870 con 0x7fb3180cc9b0 2024-01-11T08:26:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.387+0000 7fb31efff700 1 -- 172.21.15.27:0/3814629235 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31804c3c0 con 0x7fb3180cc9b0 2024-01-11T08:26:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.388+0000 7fb314ff9700 1 -- 172.21.15.27:0/3814629235 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb30401b3b0 con 0x7fb3180cc9b0 2024-01-11T08:26:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.389+0000 7fb314ff9700 1 --2- 172.21.15.27:0/3814629235 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fb2fc066f30 0x7fb2fc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.389+0000 7fb314ff9700 1 -- 172.21.15.27:0/3814629235 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5733+0+0 (secure 0 0 0) 0x7fb304083a80 con 0x7fb3180cc9b0 2024-01-11T08:26:09.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.390+0000 7fb31cd9b700 1 --2- 172.21.15.27:0/3814629235 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fb2fc066f30 0x7fb2fc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:09.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.390+0000 7fb31cd9b700 1 --2- 172.21.15.27:0/3814629235 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fb2fc066f30 0x7fb2fc0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0091e0 tx=0x7fb30c017000 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:09.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.393+0000 7fb314ff9700 1 -- 172.21.15.27:0/3814629235 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb304050c30 con 0x7fb3180cc9b0 2024-01-11T08:26:09.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.625+0000 7fb31efff700 1 -- 172.21.15.27:0/3814629235 --> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3180ca500 con 0x7fb2fc066f30 2024-01-11T08:26:09.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.628+0000 7fb314ff9700 1 -- 172.21.15.27:0/3814629235 <== mgr.5795 v2:172.21.15.77:6834/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fb3180ca500 con 0x7fb2fc066f30 2024-01-11T08:26:09.627 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T08:26:09.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:09.627 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-11T08:26:09.627 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:09.627 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T08:26:09.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.629+0000 7fb31efff700 1 -- 172.21.15.27:0/3814629235 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fb2fc066f30 msgr2=0x7fb2fc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:09.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.629+0000 7fb31efff700 1 --2- 172.21.15.27:0/3814629235 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fb2fc066f30 0x7fb2fc0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0091e0 tx=0x7fb30c017000 comp rx=0 tx=0).stop 2024-01-11T08:26:09.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.629+0000 7fb31efff700 1 -- 172.21.15.27:0/3814629235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3180cc9b0 msgr2=0x7fb3180c8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:09.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.629+0000 7fb31efff700 1 --2- 172.21.15.27:0/3814629235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3180cc9b0 0x7fb3180c8dd0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fb30400c5e0 tx=0x7fb304001070 comp rx=0 tx=0).stop 2024-01-11T08:26:09.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.629+0000 7fb31efff700 1 -- 172.21.15.27:0/3814629235 shutdown_connections 2024-01-11T08:26:09.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.629+0000 7fb31efff700 1 --2- 172.21.15.27:0/3814629235 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fb2fc066f30 0x7fb2fc0693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:09.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.629+0000 7fb31efff700 1 --2- 172.21.15.27:0/3814629235 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb318156ab0 0x7fb318157b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:09.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.629+0000 7fb31efff700 1 --2- 172.21.15.27:0/3814629235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3180b0550 0x7fb318149330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:09.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.629+0000 7fb31efff700 1 --2- 172.21.15.27:0/3814629235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3180cc9b0 0x7fb3180c8dd0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:09.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.629+0000 7fb31efff700 1 -- 172.21.15.27:0/3814629235 >> 172.21.15.27:0/3814629235 conn(0x7fb3180680b0 msgr2=0x7fb3180c8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:09.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.629+0000 7fb31efff700 1 -- 172.21.15.27:0/3814629235 shutdown_connections 2024-01-11T08:26:09.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.629+0000 7fb31efff700 1 -- 172.21.15.27:0/3814629235 wait complete. 2024-01-11T08:26:09.647 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-11T08:26:09.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.849+0000 7fee11bff700 1 Processor -- start 2024-01-11T08:26:09.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.850+0000 7fee11bff700 1 -- start start 2024-01-11T08:26:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.851+0000 7fee11bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fee0c15a730 0x7fee0c1613a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.851+0000 7fee11bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c0c6b30 con 0x7fee0c13fbb0 2024-01-11T08:26:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.851+0000 7fee11bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c0c6800 con 0x7fee0c13ff80 2024-01-11T08:26:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.851+0000 7fee11bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee0c0cb240 con 0x7fee0c15a730 2024-01-11T08:26:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.851+0000 7fee0b7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fee0c13fbb0 0x7fee0c0ca7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58460/0 (socket says 172.21.15.27:58460) 2024-01-11T08:26:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.851+0000 7fee0b7fe700 1 -- 172.21.15.27:0/2473857262 learned_addr learned my addr 172.21.15.27:0/2473857262 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.851+0000 7fee0bfff700 1 --2- 172.21.15.27:0/2473857262 >> v2:172.21.15.27:3300/0 conn(0x7fee0c15a730 0x7fee0c1613a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.851+0000 7fee0a7fc700 1 -- 172.21.15.27:0/2473857262 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385383954 0 0) 0x7fee0c0c6b30 con 0x7fee0c13fbb0 2024-01-11T08:26:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.852+0000 7fee0a7fc700 1 -- 172.21.15.27:0/2473857262 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf4003570 con 0x7fee0c13fbb0 2024-01-11T08:26:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.852+0000 7fee0bfff700 1 -- 172.21.15.27:0/2473857262 >> v1:172.21.15.77:6789/0 conn(0x7fee0c13ff80 legacy=0x7fee0c15a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.852+0000 7fee0bfff700 1 -- 172.21.15.27:0/2473857262 >> v1:172.21.15.27:6789/0 conn(0x7fee0c13fbb0 legacy=0x7fee0c0ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.852+0000 7fee0bfff700 1 -- 172.21.15.27:0/2473857262 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c0cb520 con 0x7fee0c15a730 2024-01-11T08:26:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.852+0000 7fee0bfff700 1 --2- 172.21.15.27:0/2473857262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0c15a730 0x7fee0c1613a0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fee04009e60 tx=0x7fee04009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d0a8d788b75e76 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.852+0000 7fee0a7fc700 1 -- 172.21.15.27:0/2473857262 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0400e070 con 0x7fee0c15a730 2024-01-11T08:26:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.852+0000 7fee0a7fc700 1 -- 172.21.15.27:0/2473857262 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0400bb60 con 0x7fee0c15a730 2024-01-11T08:26:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.852+0000 7fee0a7fc700 1 -- 172.21.15.27:0/2473857262 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04012430 con 0x7fee0c15a730 2024-01-11T08:26:09.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.853+0000 7fee11bff700 1 -- 172.21.15.27:0/2473857262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0c15a730 msgr2=0x7fee0c1613a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:09.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.853+0000 7fee11bff700 1 --2- 172.21.15.27:0/2473857262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0c15a730 0x7fee0c1613a0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fee04009e60 tx=0x7fee04009780 comp rx=0 tx=0).stop 2024-01-11T08:26:09.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.853+0000 7fee11bff700 1 -- 172.21.15.27:0/2473857262 shutdown_connections 2024-01-11T08:26:09.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.853+0000 7fee11bff700 1 --2- 172.21.15.27:0/2473857262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0c15a730 0x7fee0c1613a0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:09.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.853+0000 7fee11bff700 1 -- 172.21.15.27:0/2473857262 >> 172.21.15.27:0/2473857262 conn(0x7fee0c0680a0 msgr2=0x7fee0c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:09.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.853+0000 7fee11bff700 1 -- 172.21.15.27:0/2473857262 shutdown_connections 2024-01-11T08:26:09.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.853+0000 7fee11bff700 1 -- 172.21.15.27:0/2473857262 wait complete. 2024-01-11T08:26:09.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.854+0000 7fee11bff700 1 Processor -- start 2024-01-11T08:26:09.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.855+0000 7fee11bff700 1 -- start start 2024-01-11T08:26:09.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.856+0000 7fee11bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0c13fbb0 0x7fee0c157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:09.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.856+0000 7fee11bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee0c13ff80 0x7fee0c13cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:09.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.856+0000 7fee11bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee0c15a730 0x7fee0c0ca480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:09.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.856+0000 7fee11bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c1573f0 con 0x7fee0c13fbb0 2024-01-11T08:26:09.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.856+0000 7fee11bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c156b30 con 0x7fee0c13ff80 2024-01-11T08:26:09.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.856+0000 7fee11bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c156da0 con 0x7fee0c15a730 2024-01-11T08:26:09.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.856+0000 7fee0b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0c13fbb0 0x7fee0c157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:09.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.856+0000 7fee0b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0c13fbb0 0x7fee0c157880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50982/0 (socket says 172.21.15.27:50982) 2024-01-11T08:26:09.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.856+0000 7fee0b7fe700 1 -- 172.21.15.27:0/3798966426 learned_addr learned my addr 172.21.15.27:0/3798966426 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:09.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.856+0000 7fee0bfff700 1 --2- 172.21.15.27:0/3798966426 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee0c15a730 0x7fee0c0ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:09.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.856+0000 7fee0affd700 1 --2- 172.21.15.27:0/3798966426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee0c13ff80 0x7fee0c13cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:09.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.856+0000 7fee0b7fe700 1 -- 172.21.15.27:0/3798966426 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee0c15a730 msgr2=0x7fee0c0ca480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:09.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.856+0000 7fee0b7fe700 1 --2- 172.21.15.27:0/3798966426 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee0c15a730 0x7fee0c0ca480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:09.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.856+0000 7fee0b7fe700 1 -- 172.21.15.27:0/3798966426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee0c13ff80 msgr2=0x7fee0c13cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:09.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.856+0000 7fee0b7fe700 1 --2- 172.21.15.27:0/3798966426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee0c13ff80 0x7fee0c13cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:09.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.856+0000 7fee0b7fe700 1 -- 172.21.15.27:0/3798966426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee04009a20 con 0x7fee0c13fbb0 2024-01-11T08:26:09.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.857+0000 7fee0b7fe700 1 --2- 172.21.15.27:0/3798966426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0c13fbb0 0x7fee0c157880 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00c7c0 tx=0x7fedfc007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:09.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.857+0000 7fedf37fe700 1 -- 172.21.15.27:0/3798966426 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc0072e0 con 0x7fee0c13fbb0 2024-01-11T08:26:09.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.857+0000 7fedf37fe700 1 -- 172.21.15.27:0/3798966426 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedfc007440 con 0x7fee0c13fbb0 2024-01-11T08:26:09.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.857+0000 7fedf37fe700 1 -- 172.21.15.27:0/3798966426 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc0057f0 con 0x7fee0c13fbb0 2024-01-11T08:26:09.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.857+0000 7fee11bff700 1 -- 172.21.15.27:0/3798966426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee0c157080 con 0x7fee0c13fbb0 2024-01-11T08:26:09.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.857+0000 7fee11bff700 1 -- 172.21.15.27:0/3798966426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee0c13d000 con 0x7fee0c13fbb0 2024-01-11T08:26:09.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.858+0000 7fee11bff700 1 -- 172.21.15.27:0/3798966426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee0c1576d0 con 0x7fee0c13fbb0 2024-01-11T08:26:09.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.859+0000 7fedf37fe700 1 -- 172.21.15.27:0/3798966426 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fedfc005c20 con 0x7fee0c13fbb0 2024-01-11T08:26:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.860+0000 7fedf37fe700 1 --2- 172.21.15.27:0/3798966426 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fedf8066f30 0x7fedf80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.860+0000 7fedf37fe700 1 -- 172.21.15.27:0/3798966426 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5733+0+0 (secure 0 0 0) 0x7fedfc083a90 con 0x7fee0c13fbb0 2024-01-11T08:26:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.860+0000 7fee0affd700 1 --2- 172.21.15.27:0/3798966426 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fedf8066f30 0x7fedf80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.861+0000 7fee0affd700 1 --2- 172.21.15.27:0/3798966426 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fedf8066f30 0x7fedf80693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0b0b00 tx=0x7fee00009040 comp rx=0 tx=0).ready entity=mgr.5795 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:09.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:09.864+0000 7fedf37fe700 1 -- 172.21.15.27:0/3798966426 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fedfc04b5a0 con 0x7fee0c13fbb0 2024-01-11T08:26:10.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.128+0000 7fee11bff700 1 -- 172.21.15.27:0/3798966426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fee0c04c3c0 con 0x7fee0c13fbb0 2024-01-11T08:26:10.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.729+0000 7fedf37fe700 1 -- 172.21.15.27:0/3798966426 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 39) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fedfc0510c0 con 0x7fee0c13fbb0 2024-01-11T08:26:10.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.731+0000 7fedf37fe700 1 -- 172.21.15.27:0/3798966426 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v39) v1 ==== 92+0+0 (secure 0 0 0) 0x7fedfc050cf0 con 0x7fee0c13fbb0 2024-01-11T08:26:10.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.734+0000 7fee11bff700 1 -- 172.21.15.27:0/3798966426 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fedf8066f30 msgr2=0x7fedf80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:10.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.734+0000 7fee11bff700 1 --2- 172.21.15.27:0/3798966426 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fedf8066f30 0x7fedf80693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0b0b00 tx=0x7fee00009040 comp rx=0 tx=0).stop 2024-01-11T08:26:10.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.734+0000 7fee11bff700 1 -- 172.21.15.27:0/3798966426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0c13fbb0 msgr2=0x7fee0c157880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:10.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.734+0000 7fee11bff700 1 --2- 172.21.15.27:0/3798966426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0c13fbb0 0x7fee0c157880 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00c7c0 tx=0x7fedfc007c80 comp rx=0 tx=0).stop 2024-01-11T08:26:10.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.734+0000 7fee11bff700 1 -- 172.21.15.27:0/3798966426 shutdown_connections 2024-01-11T08:26:10.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.734+0000 7fee11bff700 1 --2- 172.21.15.27:0/3798966426 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fedf8066f30 0x7fedf80693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:10.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.734+0000 7fee11bff700 1 --2- 172.21.15.27:0/3798966426 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee0c15a730 0x7fee0c0ca480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:10.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.734+0000 7fee11bff700 1 --2- 172.21.15.27:0/3798966426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0c13fbb0 0x7fee0c157880 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:10.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.734+0000 7fee11bff700 1 --2- 172.21.15.27:0/3798966426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee0c13ff80 0x7fee0c13cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:10.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.734+0000 7fee11bff700 1 -- 172.21.15.27:0/3798966426 >> 172.21.15.27:0/3798966426 conn(0x7fee0c0680a0 msgr2=0x7fee0c0c0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:10.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.734+0000 7fee11bff700 1 -- 172.21.15.27:0/3798966426 shutdown_connections 2024-01-11T08:26:10.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.734+0000 7fee11bff700 1 -- 172.21.15.27:0/3798966426 wait complete. 2024-01-11T08:26:10.754 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-11T08:26:10.940 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:10.941+0000 7faf24d661c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:26:10.947 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:10.947+0000 7f500fb711c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:26:10.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.964+0000 7fdcfd3ff700 1 Processor -- start 2024-01-11T08:26:10.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.964+0000 7fdcfd3ff700 1 -- start start 2024-01-11T08:26:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.966+0000 7fdcfd3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdcf8141190 0x7fdcf80c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.966+0000 7fdcfd3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdcf80c8f60 0x7fdcf815b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.966+0000 7fdcfd3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf815c060 con 0x7fdcf8140dc0 2024-01-11T08:26:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.966+0000 7fdcfd3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcf80cc9b0 con 0x7fdcf8141190 2024-01-11T08:26:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.966+0000 7fdcfd3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf80ae390 con 0x7fdcf80c8f60 2024-01-11T08:26:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.966+0000 7fdcf67fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdcf8141190 0x7fdcf80c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.966+0000 7fdcf67fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdcf8141190 0x7fdcf80c8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50998/0 (socket says 172.21.15.27:50998) 2024-01-11T08:26:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.966+0000 7fdcf67fc700 1 -- 172.21.15.27:0/2255133941 learned_addr learned my addr 172.21.15.27:0/2255133941 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.966+0000 7fdcf77fe700 1 --2- 172.21.15.27:0/2255133941 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdcf80c8f60 0x7fdcf815b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.966+0000 7fdcf6ffd700 1 --1- 172.21.15.27:0/2255133941 >> v1:172.21.15.77:6789/0 conn(0x7fdcf8140dc0 0x7fdcf80adc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:38900/0 (socket says 172.21.15.27:38900) 2024-01-11T08:26:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.967+0000 7fdcf67fc700 1 -- 172.21.15.27:0/2255133941 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdcf80c8f60 msgr2=0x7fdcf815b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.967+0000 7fdcf67fc700 1 --2- 172.21.15.27:0/2255133941 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdcf80c8f60 0x7fdcf815b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.967+0000 7fdcf67fc700 1 -- 172.21.15.27:0/2255133941 >> v1:172.21.15.77:6789/0 conn(0x7fdcf8140dc0 legacy=0x7fdcf80adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.967+0000 7fdcf5ffb700 1 -- 172.21.15.27:0/2255133941 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547443815 0 0) 0x7fdce8001110 con 0x7fdcf8140dc0 2024-01-11T08:26:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.967+0000 7fdcf67fc700 1 -- 172.21.15.27:0/2255133941 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf80c9370 con 0x7fdcf8141190 2024-01-11T08:26:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.967+0000 7fdcf67fc700 1 --2- 172.21.15.27:0/2255133941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf8141190 0x7fdcf80c8a20 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fdcec0099b0 tx=0x7fdcec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcc0f54e8d51c8c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.967+0000 7fdcf5ffb700 1 -- 172.21.15.27:0/2255133941 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcec009ea0 con 0x7fdcf8141190 2024-01-11T08:26:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.967+0000 7fdcf5ffb700 1 -- 172.21.15.27:0/2255133941 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcec00b860 con 0x7fdcf8141190 2024-01-11T08:26:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.967+0000 7fdcf5ffb700 1 -- 172.21.15.27:0/2255133941 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcec00bc00 con 0x7fdcf8141190 2024-01-11T08:26:10.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.968+0000 7fdcfd3ff700 1 -- 172.21.15.27:0/2255133941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf8141190 msgr2=0x7fdcf80c8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:10.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.968+0000 7fdcfd3ff700 1 --2- 172.21.15.27:0/2255133941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf8141190 0x7fdcf80c8a20 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fdcec0099b0 tx=0x7fdcec009250 comp rx=0 tx=0).stop 2024-01-11T08:26:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.968+0000 7fdcfd3ff700 1 -- 172.21.15.27:0/2255133941 shutdown_connections 2024-01-11T08:26:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.968+0000 7fdcfd3ff700 1 --2- 172.21.15.27:0/2255133941 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdcf80c8f60 0x7fdcf815b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.968+0000 7fdcfd3ff700 1 --2- 172.21.15.27:0/2255133941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf8141190 0x7fdcf80c8a20 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.968+0000 7fdcfd3ff700 1 -- 172.21.15.27:0/2255133941 >> 172.21.15.27:0/2255133941 conn(0x7fdcf8063ca0 msgr2=0x7fdcf806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.968+0000 7fdcfd3ff700 1 -- 172.21.15.27:0/2255133941 shutdown_connections 2024-01-11T08:26:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.968+0000 7fdcfd3ff700 1 -- 172.21.15.27:0/2255133941 wait complete. 2024-01-11T08:26:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.970+0000 7fdcfd3ff700 1 Processor -- start 2024-01-11T08:26:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.970+0000 7fdcfd3ff700 1 -- start start 2024-01-11T08:26:10.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.971+0000 7fdcfd3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf80c8f60 0x7fdcf8157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:10.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.971+0000 7fdcfd3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcf8140dc0 0x7fdcf80b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:10.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.971+0000 7fdcfd3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdcf8141190 0x7fdcf80b2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:10.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.971+0000 7fdcfd3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf8157570 con 0x7fdcf80c8f60 2024-01-11T08:26:10.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.971+0000 7fdcfd3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf80c6560 con 0x7fdcf8140dc0 2024-01-11T08:26:10.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.971+0000 7fdcfd3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf80c5950 con 0x7fdcf8141190 2024-01-11T08:26:10.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.971+0000 7fdcf6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf80c8f60 0x7fdcf8157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.971+0000 7fdcf6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf80c8f60 0x7fdcf8157030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51000/0 (socket says 172.21.15.27:51000) 2024-01-11T08:26:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.971+0000 7fdcf6ffd700 1 -- 172.21.15.27:0/1178169928 learned_addr learned my addr 172.21.15.27:0/1178169928 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.971+0000 7fdcf77fe700 1 --2- 172.21.15.27:0/1178169928 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdcf8141190 0x7fdcf80b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.971+0000 7fdcf67fc700 1 --2- 172.21.15.27:0/1178169928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcf8140dc0 0x7fdcf80b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.971+0000 7fdcf6ffd700 1 -- 172.21.15.27:0/1178169928 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdcf8141190 msgr2=0x7fdcf80b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.971+0000 7fdcf6ffd700 1 --2- 172.21.15.27:0/1178169928 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdcf8141190 0x7fdcf80b2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.971+0000 7fdcf6ffd700 1 -- 172.21.15.27:0/1178169928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcf8140dc0 msgr2=0x7fdcf80b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.972+0000 7fdcf6ffd700 1 --2- 172.21.15.27:0/1178169928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcf8140dc0 0x7fdcf80b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.972+0000 7fdcf6ffd700 1 -- 172.21.15.27:0/1178169928 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcec0094f0 con 0x7fdcf80c8f60 2024-01-11T08:26:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.972+0000 7fdcf6ffd700 1 --2- 172.21.15.27:0/1178169928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf80c8f60 0x7fdcf8157030 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fdce800c840 tx=0x7fdce8007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.972+0000 7fdcdeffd700 1 -- 172.21.15.27:0/1178169928 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce8005a70 con 0x7fdcf80c8f60 2024-01-11T08:26:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.972+0000 7fdcdeffd700 1 -- 172.21.15.27:0/1178169928 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce8005bf0 con 0x7fdcf80c8f60 2024-01-11T08:26:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.972+0000 7fdcfd3ff700 1 -- 172.21.15.27:0/1178169928 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf80c5c30 con 0x7fdcf80c8f60 2024-01-11T08:26:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.972+0000 7fdcdeffd700 1 -- 172.21.15.27:0/1178169928 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce80070d0 con 0x7fdcf80c8f60 2024-01-11T08:26:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.972+0000 7fdcfd3ff700 1 -- 172.21.15.27:0/1178169928 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf80c61f0 con 0x7fdcf80c8f60 2024-01-11T08:26:10.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.974+0000 7fdcfd3ff700 1 -- 172.21.15.27:0/1178169928 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcf804c3c0 con 0x7fdcf80c8f60 2024-01-11T08:26:10.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.974+0000 7fdcdeffd700 1 -- 172.21.15.27:0/1178169928 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fdce80074c0 con 0x7fdcf80c8f60 2024-01-11T08:26:10.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.975+0000 7fdcdeffd700 1 --2- 172.21.15.27:0/1178169928 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fdce4066f80 0x7fdce4069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:10.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.975+0000 7fdcdeffd700 1 -- 172.21.15.27:0/1178169928 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5733+0+0 (secure 0 0 0) 0x7fdce8083940 con 0x7fdcf80c8f60 2024-01-11T08:26:10.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.975+0000 7fdcf67fc700 1 -- 172.21.15.27:0/1178169928 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fdce4066f80 msgr2=0x7fdce4069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:10.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.975+0000 7fdcf67fc700 1 --2- 172.21.15.27:0/1178169928 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fdce4066f80 0x7fdce4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:26:10.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:10.979+0000 7fdcdeffd700 1 -- 172.21.15.27:0/1178169928 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdce8050b90 con 0x7fdcf80c8f60 2024-01-11T08:26:11.120 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:11.120+0000 7faf24d661c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:26:11.152 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:11.152+0000 7f500fb711c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:26:11.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.176+0000 7fdcf67fc700 1 -- 172.21.15.27:0/1178169928 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fdce4066f80 msgr2=0x7fdce4069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:11.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.176+0000 7fdcf67fc700 1 --2- 172.21.15.27:0/1178169928 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fdce4066f80 0x7fdce4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:26:11.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.224+0000 7fdcfd3ff700 1 -- 172.21.15.27:0/1178169928 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fdcf815c060 con 0x7fdcf80c8f60 2024-01-11T08:26:11.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.228+0000 7fdcdeffd700 1 -- 172.21.15.27:0/1178169928 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdcf815c060 con 0x7fdcf80c8f60 2024-01-11T08:26:11.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.231+0000 7fdcfd3ff700 1 -- 172.21.15.27:0/1178169928 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fdce4066f80 msgr2=0x7fdce4069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:26:11.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.231+0000 7fdcfd3ff700 1 --2- 172.21.15.27:0/1178169928 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fdce4066f80 0x7fdce4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.231+0000 7fdcfd3ff700 1 -- 172.21.15.27:0/1178169928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf80c8f60 msgr2=0x7fdcf8157030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.231+0000 7fdcfd3ff700 1 --2- 172.21.15.27:0/1178169928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf80c8f60 0x7fdcf8157030 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fdce800c840 tx=0x7fdce8007c50 comp rx=0 tx=0).stop 2024-01-11T08:26:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.231+0000 7fdcfd3ff700 1 -- 172.21.15.27:0/1178169928 shutdown_connections 2024-01-11T08:26:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.231+0000 7fdcfd3ff700 1 --2- 172.21.15.27:0/1178169928 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fdce4066f80 0x7fdce4069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.231+0000 7fdcfd3ff700 1 --2- 172.21.15.27:0/1178169928 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdcf8141190 0x7fdcf80b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.231+0000 7fdcfd3ff700 1 --2- 172.21.15.27:0/1178169928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcf80c8f60 0x7fdcf8157030 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.231+0000 7fdcfd3ff700 1 --2- 172.21.15.27:0/1178169928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcf8140dc0 0x7fdcf80b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.231+0000 7fdcfd3ff700 1 -- 172.21.15.27:0/1178169928 >> 172.21.15.27:0/1178169928 conn(0x7fdcf8063ca0 msgr2=0x7fdcf8068720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.231+0000 7fdcfd3ff700 1 -- 172.21.15.27:0/1178169928 shutdown_connections 2024-01-11T08:26:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.231+0000 7fdcfd3ff700 1 -- 172.21.15.27:0/1178169928 wait complete. 2024-01-11T08:26:11.250 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-11T08:26:11.432 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:11.432+0000 7faf24d661c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:26:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.456+0000 7f08869ff700 1 Processor -- start 2024-01-11T08:26:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.456+0000 7f08869ff700 1 -- start start 2024-01-11T08:26:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.457+0000 7f08869ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0880157380 0x7f08800af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.457+0000 7f08869ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0880157750 0x7f088006e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.457+0000 7f08869ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08800cb3f0 con 0x7f0880068620 2024-01-11T08:26:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.457+0000 7f08869ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08800afdf0 con 0x7f0880157380 2024-01-11T08:26:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.457+0000 7f08869ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08800ca7e0 con 0x7f0880157750 2024-01-11T08:26:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.458+0000 7f087ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0880157380 0x7f08800af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.458+0000 7f087ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0880157380 0x7f08800af8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51006/0 (socket says 172.21.15.27:51006) 2024-01-11T08:26:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.458+0000 7f087ffff700 1 -- 172.21.15.27:0/315167235 learned_addr learned my addr 172.21.15.27:0/315167235 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.458+0000 7f087f7fe700 1 --2- 172.21.15.27:0/315167235 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0880157750 0x7f088006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.458+0000 7f087ffff700 1 -- 172.21.15.27:0/315167235 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0880157750 msgr2=0x7f088006e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.458+0000 7f087ffff700 1 --2- 172.21.15.27:0/315167235 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0880157750 0x7f088006e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.458+0000 7f087ffff700 1 -- 172.21.15.27:0/315167235 >> v1:172.21.15.77:6789/0 conn(0x7f0880068620 legacy=0x7f08800cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.458+0000 7f087ffff700 1 -- 172.21.15.27:0/315167235 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08800caac0 con 0x7f0880157380 2024-01-11T08:26:11.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.458+0000 7f087f7fe700 1 --2- 172.21.15.27:0/315167235 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0880157750 0x7f088006e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:26:11.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.458+0000 7f087ffff700 1 --2- 172.21.15.27:0/315167235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0880157380 0x7f08800af8b0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f08700099b0 tx=0x7f0870009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6a745ee653a4d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:11.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.459+0000 7f087effd700 1 -- 172.21.15.27:0/315167235 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087000b860 con 0x7f0880157380 2024-01-11T08:26:11.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.459+0000 7f087effd700 1 -- 172.21.15.27:0/315167235 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087000b9c0 con 0x7f0880157380 2024-01-11T08:26:11.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.459+0000 7f087effd700 1 -- 172.21.15.27:0/315167235 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087000bd60 con 0x7f0880157380 2024-01-11T08:26:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.459+0000 7f08869ff700 1 -- 172.21.15.27:0/315167235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0880157380 msgr2=0x7f08800af8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.459+0000 7f08869ff700 1 --2- 172.21.15.27:0/315167235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0880157380 0x7f08800af8b0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f08700099b0 tx=0x7f0870009250 comp rx=0 tx=0).stop 2024-01-11T08:26:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.459+0000 7f08869ff700 1 -- 172.21.15.27:0/315167235 shutdown_connections 2024-01-11T08:26:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.459+0000 7f08869ff700 1 --2- 172.21.15.27:0/315167235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0880157380 0x7f08800af8b0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.459+0000 7f08869ff700 1 --2- 172.21.15.27:0/315167235 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0880157750 0x7f088006e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.459+0000 7f08869ff700 1 -- 172.21.15.27:0/315167235 >> 172.21.15.27:0/315167235 conn(0x7f08800c9fb0 msgr2=0x7f088006ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.460+0000 7f08869ff700 1 -- 172.21.15.27:0/315167235 shutdown_connections 2024-01-11T08:26:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.460+0000 7f08869ff700 1 -- 172.21.15.27:0/315167235 wait complete. 2024-01-11T08:26:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.461+0000 7f08869ff700 1 Processor -- start 2024-01-11T08:26:11.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.461+0000 7f08869ff700 1 -- start start 2024-01-11T08:26:11.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.462+0000 7f08869ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0880068620 0x7f08800c64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:11.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.462+0000 7f08869ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0880157380 0x7f08801545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:11.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.462+0000 7f08869ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0880157750 0x7f08800c4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:11.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.462+0000 7f08869ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08800c6a20 con 0x7f0880157380 2024-01-11T08:26:11.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.462+0000 7f08869ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08800c5440 con 0x7f0880068620 2024-01-11T08:26:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.462+0000 7f08869ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08800c5720 con 0x7f0880157750 2024-01-11T08:26:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.463+0000 7f087f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0880157380 0x7f08801545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.463+0000 7f087ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0880068620 0x7f08800c64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.463+0000 7f087ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0880068620 0x7f08800c64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:44966/0 (socket says 172.21.15.27:44966) 2024-01-11T08:26:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.463+0000 7f087ffff700 1 -- 172.21.15.27:0/745273157 learned_addr learned my addr 172.21.15.27:0/745273157 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.463+0000 7f0884f9c700 1 --2- 172.21.15.27:0/745273157 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0880157750 0x7f08800c4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.463+0000 7f087f7fe700 1 -- 172.21.15.27:0/745273157 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0880157750 msgr2=0x7f08800c4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.463+0000 7f087f7fe700 1 --2- 172.21.15.27:0/745273157 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0880157750 0x7f08800c4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.463+0000 7f087f7fe700 1 -- 172.21.15.27:0/745273157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0880068620 msgr2=0x7f08800c64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.463+0000 7f087f7fe700 1 --2- 172.21.15.27:0/745273157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0880068620 0x7f08800c64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.463+0000 7f087f7fe700 1 -- 172.21.15.27:0/745273157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08700094f0 con 0x7f0880157380 2024-01-11T08:26:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.463+0000 7f087ffff700 1 --2- 172.21.15.27:0/745273157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0880068620 0x7f08800c64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.463+0000 7f0884f9c700 1 --2- 172.21.15.27:0/745273157 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0880157750 0x7f08800c4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.463+0000 7f087f7fe700 1 --2- 172.21.15.27:0/745273157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0880157380 0x7f08801545a0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f0874006fd0 tx=0x7f0874008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:11.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.464+0000 7f0867fff700 1 -- 172.21.15.27:0/745273157 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0874013410 con 0x7f0880157380 2024-01-11T08:26:11.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.464+0000 7f0867fff700 1 -- 172.21.15.27:0/745273157 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0874013570 con 0x7f0880157380 2024-01-11T08:26:11.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.464+0000 7f08869ff700 1 -- 172.21.15.27:0/745273157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08800c5a00 con 0x7f0880157380 2024-01-11T08:26:11.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.464+0000 7f0867fff700 1 -- 172.21.15.27:0/745273157 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08740138f0 con 0x7f0880157380 2024-01-11T08:26:11.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.464+0000 7f08869ff700 1 -- 172.21.15.27:0/745273157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08800c5fc0 con 0x7f0880157380 2024-01-11T08:26:11.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.465+0000 7f08869ff700 1 -- 172.21.15.27:0/745273157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08800c6330 con 0x7f0880157380 2024-01-11T08:26:11.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.466+0000 7f0867fff700 1 -- 172.21.15.27:0/745273157 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f0874013ce0 con 0x7f0880157380 2024-01-11T08:26:11.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.467+0000 7f0867fff700 1 --2- 172.21.15.27:0/745273157 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f086c066f80 0x7f086c069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:11.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.467+0000 7f0867fff700 1 -- 172.21.15.27:0/745273157 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f08740841d0 con 0x7f0880157380 2024-01-11T08:26:11.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.467+0000 7f087ffff700 1 -- 172.21.15.27:0/745273157 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f086c066f80 msgr2=0x7f086c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:11.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.468+0000 7f087ffff700 1 --2- 172.21.15.27:0/745273157 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f086c066f80 0x7f086c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:26:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.472+0000 7f0867fff700 1 -- 172.21.15.27:0/745273157 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0874051420 con 0x7f0880157380 2024-01-11T08:26:11.495 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:11.495+0000 7f500fb711c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:26:11.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.669+0000 7f087ffff700 1 -- 172.21.15.27:0/745273157 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f086c066f80 msgr2=0x7f086c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:11.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.669+0000 7f087ffff700 1 --2- 172.21.15.27:0/745273157 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f086c066f80 0x7f086c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:26:11.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.674+0000 7f08869ff700 1 -- 172.21.15.27:0/745273157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f088004c3c0 con 0x7f0880157380 2024-01-11T08:26:11.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.680+0000 7f0867fff700 1 -- 172.21.15.27:0/745273157 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0874054a40 con 0x7f0880157380 2024-01-11T08:26:11.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.682+0000 7f08869ff700 1 -- 172.21.15.27:0/745273157 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f086c066f80 msgr2=0x7f086c069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:26:11.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.682+0000 7f08869ff700 1 --2- 172.21.15.27:0/745273157 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f086c066f80 0x7f086c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.682+0000 7f08869ff700 1 -- 172.21.15.27:0/745273157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0880157380 msgr2=0x7f08801545a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:11.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.682+0000 7f08869ff700 1 --2- 172.21.15.27:0/745273157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0880157380 0x7f08801545a0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f0874006fd0 tx=0x7f0874008040 comp rx=0 tx=0).stop 2024-01-11T08:26:11.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.682+0000 7f08869ff700 1 -- 172.21.15.27:0/745273157 shutdown_connections 2024-01-11T08:26:11.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.682+0000 7f08869ff700 1 --2- 172.21.15.27:0/745273157 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f086c066f80 0x7f086c069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.682+0000 7f08869ff700 1 --2- 172.21.15.27:0/745273157 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0880157750 0x7f08800c4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.682+0000 7f08869ff700 1 --2- 172.21.15.27:0/745273157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0880068620 0x7f08800c64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.682+0000 7f08869ff700 1 --2- 172.21.15.27:0/745273157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0880157380 0x7f08801545a0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.682+0000 7f08869ff700 1 -- 172.21.15.27:0/745273157 >> 172.21.15.27:0/745273157 conn(0x7f08800c9fb0 msgr2=0x7f0880156470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:11.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.682+0000 7f08869ff700 1 -- 172.21.15.27:0/745273157 shutdown_connections 2024-01-11T08:26:11.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.682+0000 7f08869ff700 1 -- 172.21.15.27:0/745273157 wait complete. 2024-01-11T08:26:11.701 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-01-11T08:26:11.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.908+0000 7f4167fff700 1 Processor -- start 2024-01-11T08:26:11.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.908+0000 7f4167fff700 1 -- start start 2024-01-11T08:26:11.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.910+0000 7f4167fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4160142dc0 0x7f41600cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:11.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.910+0000 7f4167fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4160143190 0x7f41600cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:11.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.910+0000 7f4167fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416015d320 con 0x7f41600cc020 2024-01-11T08:26:11.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.910+0000 7f4167fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f416015df30 con 0x7f4160143190 2024-01-11T08:26:11.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.910+0000 7f4167fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416015d590 con 0x7f4160142dc0 2024-01-11T08:26:11.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.910+0000 7f416559a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4160143190 0x7f41600cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.910+0000 7f416559a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4160143190 0x7f41600cb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51032/0 (socket says 172.21.15.27:51032) 2024-01-11T08:26:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.910+0000 7f416559a700 1 -- 172.21.15.27:0/2046911028 learned_addr learned my addr 172.21.15.27:0/2046911028 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:11.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.910+0000 7f4165d9b700 1 --2- 172.21.15.27:0/2046911028 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4160142dc0 0x7f41600cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.911+0000 7f416559a700 1 -- 172.21.15.27:0/2046911028 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4160142dc0 msgr2=0x7f41600cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.911+0000 7f416559a700 1 --2- 172.21.15.27:0/2046911028 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4160142dc0 0x7f41600cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.911+0000 7f416559a700 1 -- 172.21.15.27:0/2046911028 >> v1:172.21.15.77:6789/0 conn(0x7f41600cc020 legacy=0x7f41600cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.911+0000 7f416559a700 1 -- 172.21.15.27:0/2046911028 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416015d870 con 0x7f4160143190 2024-01-11T08:26:11.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.911+0000 7f4164d99700 1 -- 172.21.15.27:0/2046911028 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542467240 0 0) 0x7f416015d320 con 0x7f41600cc020 2024-01-11T08:26:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.911+0000 7f416559a700 1 --2- 172.21.15.27:0/2046911028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4160143190 0x7f41600cb2e0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f41500099b0 tx=0x7f4150009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6594fc0d9a0d74d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.911+0000 7f4164d99700 1 -- 172.21.15.27:0/2046911028 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150009ea0 con 0x7f4160143190 2024-01-11T08:26:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.911+0000 7f4164d99700 1 -- 172.21.15.27:0/2046911028 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415000b860 con 0x7f4160143190 2024-01-11T08:26:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.911+0000 7f4164d99700 1 -- 172.21.15.27:0/2046911028 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415000bc00 con 0x7f4160143190 2024-01-11T08:26:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.912+0000 7f4167fff700 1 -- 172.21.15.27:0/2046911028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4160143190 msgr2=0x7f41600cb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.912+0000 7f4167fff700 1 --2- 172.21.15.27:0/2046911028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4160143190 0x7f41600cb2e0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f41500099b0 tx=0x7f4150009250 comp rx=0 tx=0).stop 2024-01-11T08:26:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.912+0000 7f4167fff700 1 -- 172.21.15.27:0/2046911028 shutdown_connections 2024-01-11T08:26:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.912+0000 7f4167fff700 1 --2- 172.21.15.27:0/2046911028 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4160142dc0 0x7f41600cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.912+0000 7f4167fff700 1 --2- 172.21.15.27:0/2046911028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4160143190 0x7f41600cb2e0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.912+0000 7f4167fff700 1 -- 172.21.15.27:0/2046911028 >> 172.21.15.27:0/2046911028 conn(0x7f4160063ca0 msgr2=0x7f416006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.912+0000 7f4167fff700 1 -- 172.21.15.27:0/2046911028 shutdown_connections 2024-01-11T08:26:11.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.912+0000 7f4167fff700 1 -- 172.21.15.27:0/2046911028 wait complete. 2024-01-11T08:26:11.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.914+0000 7f4167fff700 1 Processor -- start 2024-01-11T08:26:11.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.914+0000 7f4167fff700 1 -- start start 2024-01-11T08:26:11.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.916+0000 7f4167fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41600cc020 0x7f41600c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:11.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.916+0000 7f4167fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4160142dc0 0x7f41600c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:11.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.916+0000 7f4167fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4160143190 0x7f41601528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:11.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.916+0000 7f4167fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4160169ed0 con 0x7f4160143190 2024-01-11T08:26:11.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.916+0000 7f4167fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41600c7700 con 0x7f41600cc020 2024-01-11T08:26:11.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.916+0000 7f4167fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41600c79e0 con 0x7f4160142dc0 2024-01-11T08:26:11.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.916+0000 7f416659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4160143190 0x7f41601528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:11.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.916+0000 7f4165d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41600cc020 0x7f41600c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:11.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.916+0000 7f416559a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4160142dc0 0x7f41600c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:11.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.916+0000 7f4165d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41600cc020 0x7f41600c8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:44968/0 (socket says 172.21.15.27:44968) 2024-01-11T08:26:11.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.916+0000 7f4165d9b700 1 -- 172.21.15.27:0/655618248 learned_addr learned my addr 172.21.15.27:0/655618248 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:11.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.916+0000 7f416559a700 1 -- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41600cc020 msgr2=0x7f41600c8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:11.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.917+0000 7f416559a700 1 --2- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41600cc020 0x7f41600c8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.917+0000 7f416559a700 1 -- 172.21.15.27:0/655618248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4160143190 msgr2=0x7f41601528a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:11.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.917+0000 7f416559a700 1 --2- 172.21.15.27:0/655618248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4160143190 0x7f41601528a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:11.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.917+0000 7f416559a700 1 -- 172.21.15.27:0/655618248 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41500094f0 con 0x7f4160142dc0 2024-01-11T08:26:11.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.917+0000 7f4165d9b700 1 --2- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41600cc020 0x7f41600c8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:11.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.917+0000 7f416659c700 1 --2- 172.21.15.27:0/655618248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4160143190 0x7f41601528a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:11.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.917+0000 7f416559a700 1 --2- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4160142dc0 0x7f41600c6cb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f41600ce390 tx=0x7f4150017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:11.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.917+0000 7f4155ffb700 1 -- 172.21.15.27:0/655618248 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150000d10 con 0x7f4160142dc0 2024-01-11T08:26:11.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.917+0000 7f4155ffb700 1 -- 172.21.15.27:0/655618248 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415000bbe0 con 0x7f4160142dc0 2024-01-11T08:26:11.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.917+0000 7f4167fff700 1 -- 172.21.15.27:0/655618248 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41600c7cc0 con 0x7f4160142dc0 2024-01-11T08:26:11.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.917+0000 7f4155ffb700 1 -- 172.21.15.27:0/655618248 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150011630 con 0x7f4160142dc0 2024-01-11T08:26:11.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.917+0000 7f4167fff700 1 -- 172.21.15.27:0/655618248 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41600c8280 con 0x7f4160142dc0 2024-01-11T08:26:11.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.919+0000 7f4167fff700 1 -- 172.21.15.27:0/655618248 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f416004c3c0 con 0x7f4160142dc0 2024-01-11T08:26:11.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.919+0000 7f4155ffb700 1 -- 172.21.15.27:0/655618248 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 39) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f4150011a60 con 0x7f4160142dc0 2024-01-11T08:26:11.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.920+0000 7f4155ffb700 1 --2- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f4144066f80 0x7f4144069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:11.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.920+0000 7f4155ffb700 1 -- 172.21.15.27:0/655618248 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f4150085fe0 con 0x7f4160142dc0 2024-01-11T08:26:11.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.920+0000 7f4165d9b700 1 -- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f4144066f80 msgr2=0x7f4144069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:11.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.920+0000 7f4165d9b700 1 --2- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f4144066f80 0x7f4144069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:26:11.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:11.925+0000 7f4155ffb700 1 -- 172.21.15.27:0/655618248 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f41500531d0 con 0x7f4160142dc0 2024-01-11T08:26:11.958 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:11.959+0000 7faf24d661c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:26:12.108 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:12.109+0000 7f500fb711c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:26:12.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:12.121+0000 7f4165d9b700 1 -- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f4144066f80 msgr2=0x7f4144069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:12.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:12.121+0000 7f4165d9b700 1 --2- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f4144066f80 0x7f4144069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:26:12.127 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:12.127+0000 7faf24d661c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:26:12.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:12.218+0000 7f4167fff700 1 -- 172.21.15.27:0/655618248 --> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f416015db50 con 0x7f4144066f80 2024-01-11T08:26:12.289 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:12.290+0000 7f500fb711c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:26:12.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:12.522+0000 7f4165d9b700 1 -- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f4144066f80 msgr2=0x7f4144069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:12.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:12.522+0000 7f4165d9b700 1 --2- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f4144066f80 0x7f4144069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-11T08:26:12.543 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:12.543+0000 7faf24d661c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:26:12.727 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:12.728+0000 7f500fb711c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:26:13.317 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:13.317+0000 7faf24d661c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:26:13.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:13.323+0000 7f4165d9b700 1 -- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f4144066f80 msgr2=0x7f4144069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:13.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:13.323+0000 7f4165d9b700 1 --2- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f4144066f80 0x7f4144069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-11T08:26:13.404 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:13.404+0000 7faf24d661c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:26:13.552 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:13.553+0000 7f500fb711c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:26:13.558 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:13.559+0000 7faf24d661c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:26:13.645 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:13.645+0000 7faf24d661c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:26:13.646 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:13.646+0000 7f500fb711c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:26:13.811 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:13.812+0000 7f500fb711c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:26:13.860 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:13.860+0000 7faf24d661c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:26:13.904 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:13.905+0000 7f500fb711c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:26:13.955 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:13.955+0000 7faf24d661c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:26:14.050 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:14.050+0000 7faf24d661c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:26:14.130 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:14.131+0000 7f500fb711c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:26:14.144 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:14.144+0000 7faf24d661c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:26:14.231 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:14.232+0000 7f500fb711c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:26:14.331 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:14.332+0000 7f500fb711c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:26:14.332 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:14.333+0000 7faf24d661c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:26:14.429 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:14.429+0000 7faf24d661c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:26:14.431 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:14.433+0000 7f500fb711c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:26:14.631 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:14.632+0000 7f500fb711c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:26:14.736 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:14.738+0000 7f500fb711c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:26:14.768 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:14.768+0000 7faf24d661c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:26:14.857 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:14.857+0000 7faf24d661c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:26:14.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:14.924+0000 7f4165d9b700 1 -- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f4144066f80 msgr2=0x7f4144069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:14.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:14.924+0000 7f4165d9b700 1 --2- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f4144066f80 0x7f4144069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-11T08:26:15.095 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:15.096+0000 7f500fb711c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:26:15.189 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:15.191+0000 7f500fb711c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:26:15.396 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:15.397+0000 7faf24d661c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:26:15.580 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:15.581+0000 7faf24d661c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:26:15.764 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:15.765+0000 7f500fb711c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:26:15.952 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:15.952+0000 7faf24d661c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:26:15.960 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:15.961+0000 7f500fb711c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:26:16.357 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:16.358+0000 7f500fb711c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:26:16.375 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:16.376+0000 7faf24d661c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:26:16.556 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:16.556+0000 7faf24d661c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:26:16.740 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:16.741+0000 7faf24d661c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:26:16.801 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:16.802+0000 7f500fb711c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:26:16.848 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:16.848+0000 7faf24d661c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:26:16.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:16.863+0000 7f4155ffb700 1 -- 172.21.15.27:0/655618248 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mgrmap(e 40) v1 ==== 84470+0+0 (secure 0 0 0) 0x7f415000fdc0 con 0x7f4160142dc0 2024-01-11T08:26:16.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:16.863+0000 7f4155ffb700 1 -- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f4144066f80 msgr2=0x7f4144069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:26:16.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:16.863+0000 7f4155ffb700 1 --2- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f4144066f80 0x7f4144069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:16.978 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:16.978+0000 7fae9a3b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:16.978 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:16.978+0000 7fae9a3b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:16.978 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:16.978+0000 7fae9a3b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:16.979 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:16.978+0000 7fae9a3b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:16.979 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:16.978+0000 7fae9a3b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:16.979 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:16.978+0000 7fae9d3bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:16.979 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:16.979+0000 7fae9d3bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:16.979 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:16.979+0000 7fae9d3bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:16.979 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:16.979+0000 7fae9d3bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:16.979 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:16.979+0000 7fae9d3bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:17.005 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:17.006+0000 7f500fb711c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:26:17.202 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:17.203+0000 7f500fb711c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:26:17.320 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:17.321+0000 7f500fb711c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:26:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:17.872+0000 7f4155ffb700 1 -- 172.21.15.27:0/655618248 <== mon.2 v2:172.21.15.77:3301/0 8 ==== mgrmap(e 41) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f41500535a0 con 0x7f4160142dc0 2024-01-11T08:26:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:17.872+0000 7f4155ffb700 1 --2- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f4144066f80 0x7f41440693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:17.872+0000 7f4155ffb700 1 -- 172.21.15.27:0/655618248 --> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f416015db50 con 0x7f4144066f80 2024-01-11T08:26:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:17.873+0000 7f4165d9b700 1 --2- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f4144066f80 0x7f41440693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:17.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:17.875+0000 7f4165d9b700 1 --2- 172.21.15.27:0/655618248 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f4144066f80 0x7f41440693b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f415c009c40 tx=0x7f415c0091a0 comp rx=0 tx=0).ready entity=mgr.6059 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:17.882 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:17.882+0000 7faea43cd700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-01-11T08:26:17.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:17.883+0000 7f4155ffb700 1 -- 172.21.15.27:0/655618248 <== mgr.6059 v2:172.21.15.77:6834/41381 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f416015db50 con 0x7f4144066f80 2024-01-11T08:26:17.884 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-01-11T08:26:17.892 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-11T08:26:17.892 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-11T08:26:18.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.095+0000 7f6abcfff700 1 Processor -- start 2024-01-11T08:26:18.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.096+0000 7f6abcfff700 1 -- start start 2024-01-11T08:26:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.097+0000 7f6abcfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6ab8143190 0x7f6ab80cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.097+0000 7f6abcfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6ab80cb4b0 0x7f6ab80cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.097+0000 7f6abcfff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab815d320 con 0x7f6ab8142dc0 2024-01-11T08:26:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.097+0000 7f6abcfff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab815df30 con 0x7f6ab8143190 2024-01-11T08:26:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.097+0000 7f6abcfff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab815d590 con 0x7f6ab80cb4b0 2024-01-11T08:26:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.097+0000 7f6ab659c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6ab8142dc0 0x7f6ab80cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35280/0 (socket says 172.21.15.27:35280) 2024-01-11T08:26:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.097+0000 7f6ab5d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6ab8143190 0x7f6ab80cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.097+0000 7f6ab659c700 1 -- 172.21.15.27:0/3537927236 learned_addr learned my addr 172.21.15.27:0/3537927236 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:18.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.097+0000 7f6ab6d9d700 1 --2- 172.21.15.27:0/3537927236 >> v2:172.21.15.77:3300/0 conn(0x7f6ab80cb4b0 0x7f6ab80cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:18.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.098+0000 7f6ab5d9b700 1 -- 172.21.15.27:0/3537927236 >> v2:172.21.15.77:3300/0 conn(0x7f6ab80cb4b0 msgr2=0x7f6ab80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:18.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.098+0000 7f6ab5d9b700 1 --2- 172.21.15.27:0/3537927236 >> v2:172.21.15.77:3300/0 conn(0x7f6ab80cb4b0 0x7f6ab80cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:18.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.098+0000 7f6ab5d9b700 1 -- 172.21.15.27:0/3537927236 >> v1:172.21.15.27:6789/0 conn(0x7f6ab8142dc0 legacy=0x7f6ab80cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:18.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.098+0000 7f6ab5d9b700 1 -- 172.21.15.27:0/3537927236 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab815d870 con 0x7f6ab8143190 2024-01-11T08:26:18.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.098+0000 7f6ab5d9b700 1 --2- 172.21.15.27:0/3537927236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab8143190 0x7f6ab80cc020 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f6ab81577b0 tx=0x7f6aac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47ef3841b4f117d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:18.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.098+0000 7f6ab559a700 1 -- 172.21.15.27:0/3537927236 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac00b6b0 con 0x7f6ab8143190 2024-01-11T08:26:18.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.098+0000 7f6ab559a700 1 -- 172.21.15.27:0/3537927236 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aac00b810 con 0x7f6ab8143190 2024-01-11T08:26:18.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.098+0000 7f6ab559a700 1 -- 172.21.15.27:0/3537927236 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac00bbb0 con 0x7f6ab8143190 2024-01-11T08:26:18.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.099+0000 7f6abcfff700 1 -- 172.21.15.27:0/3537927236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab8143190 msgr2=0x7f6ab80cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:18.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.099+0000 7f6abcfff700 1 --2- 172.21.15.27:0/3537927236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab8143190 0x7f6ab80cc020 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f6ab81577b0 tx=0x7f6aac0091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:18.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.099+0000 7f6abcfff700 1 -- 172.21.15.27:0/3537927236 shutdown_connections 2024-01-11T08:26:18.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.099+0000 7f6abcfff700 1 --2- 172.21.15.27:0/3537927236 >> v2:172.21.15.77:3300/0 conn(0x7f6ab80cb4b0 0x7f6ab80cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:18.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.099+0000 7f6abcfff700 1 --2- 172.21.15.27:0/3537927236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab8143190 0x7f6ab80cc020 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:18.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.099+0000 7f6abcfff700 1 -- 172.21.15.27:0/3537927236 >> 172.21.15.27:0/3537927236 conn(0x7f6ab8063ca0 msgr2=0x7f6ab806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:18.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.099+0000 7f6abcfff700 1 -- 172.21.15.27:0/3537927236 shutdown_connections 2024-01-11T08:26:18.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.099+0000 7f6abcfff700 1 -- 172.21.15.27:0/3537927236 wait complete. 2024-01-11T08:26:18.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.101+0000 7f6abcfff700 1 Processor -- start 2024-01-11T08:26:18.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.101+0000 7f6abcfff700 1 -- start start 2024-01-11T08:26:18.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.102+0000 7f6abcfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ab80cb4b0 0x7f6ab81404a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:18.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.102+0000 7f6abcfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab8142dc0 0x7f6ab81558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:18.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.102+0000 7f6abcfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6ab8143190 0x7f6ab80b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:18.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.102+0000 7f6abcfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab815bf80 con 0x7f6ab8142dc0 2024-01-11T08:26:18.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.102+0000 7f6abcfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab815b650 con 0x7f6ab80cb4b0 2024-01-11T08:26:18.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.102+0000 7f6abcfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab815b930 con 0x7f6ab8143190 2024-01-11T08:26:18.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.102+0000 7f6ab5d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab8142dc0 0x7f6ab81558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:18.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.102+0000 7f6ab5d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab8142dc0 0x7f6ab81558d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59348/0 (socket says 172.21.15.27:59348) 2024-01-11T08:26:18.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.102+0000 7f6ab5d9b700 1 -- 172.21.15.27:0/4231999231 learned_addr learned my addr 172.21.15.27:0/4231999231 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:18.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.102+0000 7f6ab5d9b700 1 -- 172.21.15.27:0/4231999231 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6ab8143190 msgr2=0x7f6ab80b3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:18.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.102+0000 7f6ab6d9d700 1 --2- 172.21.15.27:0/4231999231 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6ab8143190 0x7f6ab80b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:18.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.102+0000 7f6ab5d9b700 1 --2- 172.21.15.27:0/4231999231 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6ab8143190 0x7f6ab80b3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:18.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.102+0000 7f6ab659c700 1 --2- 172.21.15.27:0/4231999231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ab80cb4b0 0x7f6ab81404a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:18.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.102+0000 7f6ab5d9b700 1 -- 172.21.15.27:0/4231999231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ab80cb4b0 msgr2=0x7f6ab81404a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:18.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.102+0000 7f6ab5d9b700 1 --2- 172.21.15.27:0/4231999231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ab80cb4b0 0x7f6ab81404a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:18.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.102+0000 7f6ab5d9b700 1 -- 172.21.15.27:0/4231999231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aac009420 con 0x7f6ab8142dc0 2024-01-11T08:26:18.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.102+0000 7f6ab6d9d700 1 --2- 172.21.15.27:0/4231999231 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6ab8143190 0x7f6ab80b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:18.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.102+0000 7f6ab659c700 1 --2- 172.21.15.27:0/4231999231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ab80cb4b0 0x7f6ab81404a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:18.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.103+0000 7f6ab5d9b700 1 --2- 172.21.15.27:0/4231999231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab8142dc0 0x7f6ab81558d0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8064a70 tx=0x7f6aac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:18.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.103+0000 7f6aa67fc700 1 -- 172.21.15.27:0/4231999231 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac011420 con 0x7f6ab8142dc0 2024-01-11T08:26:18.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.103+0000 7f6aa67fc700 1 -- 172.21.15.27:0/4231999231 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aac011580 con 0x7f6ab8142dc0 2024-01-11T08:26:18.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.103+0000 7f6aa67fc700 1 -- 172.21.15.27:0/4231999231 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac011940 con 0x7f6ab8142dc0 2024-01-11T08:26:18.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.103+0000 7f6abcfff700 1 -- 172.21.15.27:0/4231999231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab815bc10 con 0x7f6ab8142dc0 2024-01-11T08:26:18.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.103+0000 7f6abcfff700 1 -- 172.21.15.27:0/4231999231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab8155e10 con 0x7f6ab8142dc0 2024-01-11T08:26:18.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.104+0000 7f6abcfff700 1 -- 172.21.15.27:0/4231999231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab815c130 con 0x7f6ab8142dc0 2024-01-11T08:26:18.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.105+0000 7f6aa67fc700 1 -- 172.21.15.27:0/4231999231 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 41) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f6aac011d30 con 0x7f6ab8142dc0 2024-01-11T08:26:18.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.106+0000 7f6aa67fc700 1 --2- 172.21.15.27:0/4231999231 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f6a98066dd0 0x7f6a98069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:18.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.106+0000 7f6aa67fc700 1 -- 172.21.15.27:0/4231999231 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5991+0+0 (secure 0 0 0) 0x7f6aac085ce0 con 0x7f6ab8142dc0 2024-01-11T08:26:18.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.106+0000 7f6ab659c700 1 --2- 172.21.15.27:0/4231999231 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f6a98066dd0 0x7f6a98069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:18.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.107+0000 7f6ab659c700 1 --2- 172.21.15.27:0/4231999231 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f6a98066dd0 0x7f6a98069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8001820 tx=0x7f6aa8009040 comp rx=0 tx=0).ready entity=mgr.6059 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:18.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.111+0000 7f6aa67fc700 1 -- 172.21.15.27:0/4231999231 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6ab815c130 con 0x7f6ab8142dc0 2024-01-11T08:26:18.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.370+0000 7f6abcfff700 1 -- 172.21.15.27:0/4231999231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f6ab804c3c0 con 0x7f6ab8142dc0 2024-01-11T08:26:18.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.917+0000 7f6aa67fc700 1 -- 172.21.15.27:0/4231999231 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 42) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f6aac0534b0 con 0x7f6ab8142dc0 2024-01-11T08:26:18.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.918+0000 7f6aa67fc700 1 -- 172.21.15.27:0/4231999231 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v42) v1 ==== 93+0+0 (secure 0 0 0) 0x7f6aac056700 con 0x7f6ab8142dc0 2024-01-11T08:26:18.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.921+0000 7f6abcfff700 1 -- 172.21.15.27:0/4231999231 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f6a98066dd0 msgr2=0x7f6a98069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:18.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.921+0000 7f6abcfff700 1 --2- 172.21.15.27:0/4231999231 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f6a98066dd0 0x7f6a98069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8001820 tx=0x7f6aa8009040 comp rx=0 tx=0).stop 2024-01-11T08:26:18.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.921+0000 7f6abcfff700 1 -- 172.21.15.27:0/4231999231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab8142dc0 msgr2=0x7f6ab81558d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:18.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.921+0000 7f6abcfff700 1 --2- 172.21.15.27:0/4231999231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab8142dc0 0x7f6ab81558d0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8064a70 tx=0x7f6aac017000 comp rx=0 tx=0).stop 2024-01-11T08:26:18.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.921+0000 7f6abcfff700 1 -- 172.21.15.27:0/4231999231 shutdown_connections 2024-01-11T08:26:18.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.921+0000 7f6abcfff700 1 --2- 172.21.15.27:0/4231999231 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f6a98066dd0 0x7f6a98069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:18.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.921+0000 7f6abcfff700 1 --2- 172.21.15.27:0/4231999231 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6ab8143190 0x7f6ab80b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:18.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.921+0000 7f6abcfff700 1 --2- 172.21.15.27:0/4231999231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ab80cb4b0 0x7f6ab81404a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:18.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.921+0000 7f6abcfff700 1 --2- 172.21.15.27:0/4231999231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ab8142dc0 0x7f6ab81558d0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:18.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.921+0000 7f6abcfff700 1 -- 172.21.15.27:0/4231999231 >> 172.21.15.27:0/4231999231 conn(0x7f6ab8063ca0 msgr2=0x7f6ab81420c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:18.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.921+0000 7f6abcfff700 1 -- 172.21.15.27:0/4231999231 shutdown_connections 2024-01-11T08:26:18.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:18.921+0000 7f6abcfff700 1 -- 172.21.15.27:0/4231999231 wait complete. 2024-01-11T08:26:18.940 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T08:26:18.940 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:26:19.039 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:26:19.137 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:19.137+0000 7fb6b233c1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:26:19.143 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:19.144+0000 7f267415b1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:26:19.145 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T08:26:19.145 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-11T08:26:19.145 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:26:19.145 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-01-11T08:26:19.226 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T08:26:19.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:26:19.226 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-11T08:26:19.302 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:26:19.317 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:19.317+0000 7fb6b233c1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:26:19.340 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:19.341+0000 7f267415b1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:26:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.520+0000 7f6dd99ff700 1 Processor -- start 2024-01-11T08:26:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.520+0000 7f6dd99ff700 1 -- start start 2024-01-11T08:26:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.521+0000 7f6dd99ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6dd4069d40 0x7f6dd40b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.521+0000 7f6dd99ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6dd40c89f0 0x7f6dd40b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.521+0000 7f6dd99ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd4157390 con 0x7f6dd40c8620 2024-01-11T08:26:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.521+0000 7f6dd99ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dd40adcd0 con 0x7f6dd40c89f0 2024-01-11T08:26:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.521+0000 7f6dd99ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dd406a200 con 0x7f6dd4069d40 2024-01-11T08:26:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.521+0000 7f6dd37fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6dd40c89f0 0x7f6dd40b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.521+0000 7f6dd37fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6dd40c89f0 0x7f6dd40b0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59380/0 (socket says 172.21.15.27:59380) 2024-01-11T08:26:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.521+0000 7f6dd37fe700 1 -- 172.21.15.27:0/2852522306 learned_addr learned my addr 172.21.15.27:0/2852522306 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.522+0000 7f6dd27fc700 1 --2- 172.21.15.27:0/2852522306 >> v2:172.21.15.77:3300/0 conn(0x7f6dd4069d40 0x7f6dd40b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.521+0000 7f6dd2ffd700 1 --1- 172.21.15.27:0/2852522306 >> v1:172.21.15.27:6789/0 conn(0x7f6dd40c8620 0x7f6dd40af970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35282/0 (socket says 172.21.15.27:35282) 2024-01-11T08:26:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.522+0000 7f6dd37fe700 1 -- 172.21.15.27:0/2852522306 >> v2:172.21.15.77:3300/0 conn(0x7f6dd4069d40 msgr2=0x7f6dd40b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.522+0000 7f6dd37fe700 1 --2- 172.21.15.27:0/2852522306 >> v2:172.21.15.77:3300/0 conn(0x7f6dd4069d40 0x7f6dd40b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.522+0000 7f6dd37fe700 1 -- 172.21.15.27:0/2852522306 >> v1:172.21.15.27:6789/0 conn(0x7f6dd40c8620 legacy=0x7f6dd40af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.522+0000 7f6dd37fe700 1 -- 172.21.15.27:0/2852522306 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd4156d40 con 0x7f6dd40c89f0 2024-01-11T08:26:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.522+0000 7f6dd37fe700 1 --2- 172.21.15.27:0/2852522306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd40c89f0 0x7f6dd40b0810 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4157300 tx=0x7f6dcc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7948537d11a2ddc8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.523+0000 7f6dd1ffb700 1 -- 172.21.15.27:0/2852522306 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc00e070 con 0x7f6dd40c89f0 2024-01-11T08:26:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.523+0000 7f6dd1ffb700 1 -- 172.21.15.27:0/2852522306 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dcc00ba20 con 0x7f6dd40c89f0 2024-01-11T08:26:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.523+0000 7f6dd1ffb700 1 -- 172.21.15.27:0/2852522306 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc0092e0 con 0x7f6dd40c89f0 2024-01-11T08:26:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.523+0000 7f6dd99ff700 1 -- 172.21.15.27:0/2852522306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd40c89f0 msgr2=0x7f6dd40b0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.523+0000 7f6dd99ff700 1 --2- 172.21.15.27:0/2852522306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd40c89f0 0x7f6dd40b0810 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4157300 tx=0x7f6dcc009840 comp rx=0 tx=0).stop 2024-01-11T08:26:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.523+0000 7f6dd99ff700 1 -- 172.21.15.27:0/2852522306 shutdown_connections 2024-01-11T08:26:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.523+0000 7f6dd99ff700 1 --2- 172.21.15.27:0/2852522306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd40c89f0 0x7f6dd40b0810 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.523+0000 7f6dd99ff700 1 --2- 172.21.15.27:0/2852522306 >> v2:172.21.15.77:3300/0 conn(0x7f6dd4069d40 0x7f6dd40b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:19.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.523+0000 7f6dd99ff700 1 -- 172.21.15.27:0/2852522306 >> 172.21.15.27:0/2852522306 conn(0x7f6dd40680e0 msgr2=0x7f6dd4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:19.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.523+0000 7f6dd99ff700 1 -- 172.21.15.27:0/2852522306 shutdown_connections 2024-01-11T08:26:19.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.523+0000 7f6dd99ff700 1 -- 172.21.15.27:0/2852522306 wait complete. 2024-01-11T08:26:19.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.525+0000 7f6dd99ff700 1 Processor -- start 2024-01-11T08:26:19.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.525+0000 7f6dd99ff700 1 -- start start 2024-01-11T08:26:19.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.526+0000 7f6dd99ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6dd4069d40 0x7f6dd4148c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:19.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.526+0000 7f6dd99ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd40c8620 0x7f6dd4169670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:19.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.526+0000 7f6dd99ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dd40c89f0 0x7f6dd40cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:19.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.526+0000 7f6dd99ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd416a580 con 0x7f6dd40c8620 2024-01-11T08:26:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.526+0000 7f6dd99ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd416d110 con 0x7f6dd40c89f0 2024-01-11T08:26:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.526+0000 7f6dd99ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd4169fc0 con 0x7f6dd4069d40 2024-01-11T08:26:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.526+0000 7f6dd27fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd40c8620 0x7f6dd4169670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.526+0000 7f6dd27fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd40c8620 0x7f6dd4169670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59386/0 (socket says 172.21.15.27:59386) 2024-01-11T08:26:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.526+0000 7f6dd27fc700 1 -- 172.21.15.27:0/1727461431 learned_addr learned my addr 172.21.15.27:0/1727461431 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.526+0000 7f6dd37fe700 1 --2- 172.21.15.27:0/1727461431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dd40c89f0 0x7f6dd40cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.526+0000 7f6dd2ffd700 1 --2- 172.21.15.27:0/1727461431 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6dd4069d40 0x7f6dd4148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.527+0000 7f6dd27fc700 1 -- 172.21.15.27:0/1727461431 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6dd4069d40 msgr2=0x7f6dd4148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.527+0000 7f6dd27fc700 1 --2- 172.21.15.27:0/1727461431 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6dd4069d40 0x7f6dd4148c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.527+0000 7f6dd27fc700 1 -- 172.21.15.27:0/1727461431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dd40c89f0 msgr2=0x7f6dd40cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.527+0000 7f6dd27fc700 1 --2- 172.21.15.27:0/1727461431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dd40c89f0 0x7f6dd40cbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.527+0000 7f6dd27fc700 1 -- 172.21.15.27:0/1727461431 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dcc009ae0 con 0x7f6dd40c8620 2024-01-11T08:26:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.527+0000 7f6dd27fc700 1 --2- 172.21.15.27:0/1727461431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd40c8620 0x7f6dd4169670 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800ec80 tx=0x7f6dc80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:19.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.527+0000 7f6dbaffd700 1 -- 172.21.15.27:0/1727461431 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc800c850 con 0x7f6dd40c8620 2024-01-11T08:26:19.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.527+0000 7f6dbaffd700 1 -- 172.21.15.27:0/1727461431 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dc800c9b0 con 0x7f6dd40c8620 2024-01-11T08:26:19.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.527+0000 7f6dbaffd700 1 -- 172.21.15.27:0/1727461431 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc800cd50 con 0x7f6dd40c8620 2024-01-11T08:26:19.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.527+0000 7f6dd99ff700 1 -- 172.21.15.27:0/1727461431 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dd416a2a0 con 0x7f6dd40c8620 2024-01-11T08:26:19.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.528+0000 7f6dd99ff700 1 -- 172.21.15.27:0/1727461431 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dd41491b0 con 0x7f6dd40c8620 2024-01-11T08:26:19.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.529+0000 7f6dd99ff700 1 -- 172.21.15.27:0/1727461431 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dd416d280 con 0x7f6dd40c8620 2024-01-11T08:26:19.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.529+0000 7f6dbaffd700 1 -- 172.21.15.27:0/1727461431 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 42) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f6dc8013680 con 0x7f6dd40c8620 2024-01-11T08:26:19.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.531+0000 7f6dbaffd700 1 --2- 172.21.15.27:0/1727461431 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f6dc0067050 0x7f6dc0069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:19.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.531+0000 7f6dbaffd700 1 -- 172.21.15.27:0/1727461431 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5991+0+0 (secure 0 0 0) 0x7f6dc80849f0 con 0x7f6dd40c8620 2024-01-11T08:26:19.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.531+0000 7f6dd2ffd700 1 -- 172.21.15.27:0/1727461431 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f6dc0067050 msgr2=0x7f6dc0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:19.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.531+0000 7f6dd2ffd700 1 --2- 172.21.15.27:0/1727461431 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f6dc0067050 0x7f6dc0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:26:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.535+0000 7f6dbaffd700 1 -- 172.21.15.27:0/1727461431 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6dc8051a30 con 0x7f6dd40c8620 2024-01-11T08:26:19.633 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:19.633+0000 7fb6b233c1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:26:19.687 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:19.688+0000 7f267415b1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:26:19.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.732+0000 7f6dd2ffd700 1 -- 172.21.15.27:0/1727461431 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f6dc0067050 msgr2=0x7f6dc0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:19.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.732+0000 7f6dd2ffd700 1 --2- 172.21.15.27:0/1727461431 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f6dc0067050 0x7f6dc0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:26:19.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.791+0000 7f6dd99ff700 1 -- 172.21.15.27:0/1727461431 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6dd40718d0 con 0x7f6dd40c8620 2024-01-11T08:26:19.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.791+0000 7f6dbaffd700 1 -- 172.21.15.27:0/1727461431 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4596 (secure 0 0 0) 0x7f6dc8055050 con 0x7f6dd40c8620 2024-01-11T08:26:19.790 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:19.790 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5945,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1366930493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1366930493},{"type":"v1","addr":"172.21.15.77:6837","nonce":1366930493}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/516828593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":516828593},{"type":"v1","addr":"172.21.15.27:6835","nonce":516828593}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6624,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1462523541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1462523541},{"type":"v1","addr":"172.21.15.27:6834","nonce":1462523541}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:26:00.710104+0000","modified":"2024-01-11T08:26:04.230562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6627":{"gid":6627,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6835/3498364963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3498364963},{"type":"v1","addr":"172.21.15.77:6835","nonce":3498364963}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2024-01-11T08:26:19.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.794+0000 7f6dd99ff700 1 -- 172.21.15.27:0/1727461431 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f6dc0067050 msgr2=0x7f6dc0069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:26:19.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.794+0000 7f6dd99ff700 1 --2- 172.21.15.27:0/1727461431 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f6dc0067050 0x7f6dc0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:19.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.794+0000 7f6dd99ff700 1 -- 172.21.15.27:0/1727461431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd40c8620 msgr2=0x7f6dd4169670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:19.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.794+0000 7f6dd99ff700 1 --2- 172.21.15.27:0/1727461431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd40c8620 0x7f6dd4169670 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800ec80 tx=0x7f6dc80091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:19.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.794+0000 7f6dd99ff700 1 -- 172.21.15.27:0/1727461431 shutdown_connections 2024-01-11T08:26:19.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.794+0000 7f6dd99ff700 1 --2- 172.21.15.27:0/1727461431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dd40c89f0 0x7f6dd40cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:19.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.794+0000 7f6dd99ff700 1 --2- 172.21.15.27:0/1727461431 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f6dc0067050 0x7f6dc0069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:19.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.794+0000 7f6dd99ff700 1 --2- 172.21.15.27:0/1727461431 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6dd4069d40 0x7f6dd4148c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:19.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.794+0000 7f6dd99ff700 1 --2- 172.21.15.27:0/1727461431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd40c8620 0x7f6dd4169670 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:19.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.794+0000 7f6dd99ff700 1 -- 172.21.15.27:0/1727461431 >> 172.21.15.27:0/1727461431 conn(0x7f6dd40680e0 msgr2=0x7f6dd40cb350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:19.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.794+0000 7f6dd99ff700 1 -- 172.21.15.27:0/1727461431 shutdown_connections 2024-01-11T08:26:19.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:19.794+0000 7f6dd99ff700 1 -- 172.21.15.27:0/1727461431 wait complete. 2024-01-11T08:26:19.794 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 42 2024-01-11T08:26:19.817 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:26:20.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.062+0000 7fe582fff700 1 Processor -- start 2024-01-11T08:26:20.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.062+0000 7fe582fff700 1 -- start start 2024-01-11T08:26:20.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.064+0000 7fe582fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe57c0b09e0 0x7fe57c1610a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:20.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.064+0000 7fe582fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57c157390 con 0x7fe57c069d40 2024-01-11T08:26:20.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.064+0000 7fe582fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57c06a200 con 0x7fe57c0c8620 2024-01-11T08:26:20.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.064+0000 7fe582fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe57c0adcd0 con 0x7fe57c0b09e0 2024-01-11T08:26:20.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.064+0000 7fe58159c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe57c0b09e0 0x7fe57c1610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:20.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.064+0000 7fe58159c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe57c0b09e0 0x7fe57c1610a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59396/0 (socket says 172.21.15.27:59396) 2024-01-11T08:26:20.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.064+0000 7fe58159c700 1 -- 172.21.15.27:0/441788387 learned_addr learned my addr 172.21.15.27:0/441788387 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:20.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.065+0000 7fe58159c700 1 -- 172.21.15.27:0/441788387 >> v1:172.21.15.77:6789/0 conn(0x7fe57c0c8620 legacy=0x7fe57c0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:20.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.065+0000 7fe58159c700 1 -- 172.21.15.27:0/441788387 >> v1:172.21.15.27:6789/0 conn(0x7fe57c069d40 legacy=0x7fe57c0b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:20.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.065+0000 7fe58159c700 1 -- 172.21.15.27:0/441788387 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57c156d40 con 0x7fe57c0b09e0 2024-01-11T08:26:20.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.065+0000 7fe58159c700 1 --2- 172.21.15.27:0/441788387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0b09e0 0x7fe57c1610a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fe574009f20 tx=0x7fe574009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=280762325962a4cf server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:20.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.065+0000 7fe57b7fe700 1 -- 172.21.15.27:0/441788387 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57400e070 con 0x7fe57c0b09e0 2024-01-11T08:26:20.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.065+0000 7fe57b7fe700 1 -- 172.21.15.27:0/441788387 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe57400ba80 con 0x7fe57c0b09e0 2024-01-11T08:26:20.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.065+0000 7fe57b7fe700 1 -- 172.21.15.27:0/441788387 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5740092e0 con 0x7fe57c0b09e0 2024-01-11T08:26:20.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.066+0000 7fe582fff700 1 -- 172.21.15.27:0/441788387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0b09e0 msgr2=0x7fe57c1610a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:20.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.066+0000 7fe582fff700 1 --2- 172.21.15.27:0/441788387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0b09e0 0x7fe57c1610a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fe574009f20 tx=0x7fe574009840 comp rx=0 tx=0).stop 2024-01-11T08:26:20.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.066+0000 7fe582fff700 1 -- 172.21.15.27:0/441788387 shutdown_connections 2024-01-11T08:26:20.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.066+0000 7fe582fff700 1 --2- 172.21.15.27:0/441788387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0b09e0 0x7fe57c1610a0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.066+0000 7fe582fff700 1 -- 172.21.15.27:0/441788387 >> 172.21.15.27:0/441788387 conn(0x7fe57c0680e0 msgr2=0x7fe57c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:20.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.066+0000 7fe582fff700 1 -- 172.21.15.27:0/441788387 shutdown_connections 2024-01-11T08:26:20.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.066+0000 7fe582fff700 1 -- 172.21.15.27:0/441788387 wait complete. 2024-01-11T08:26:20.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.068+0000 7fe582fff700 1 Processor -- start 2024-01-11T08:26:20.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.068+0000 7fe582fff700 1 -- start start 2024-01-11T08:26:20.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.069+0000 7fe582fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe57c069d40 0x7fe57c159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:20.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.069+0000 7fe582fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0b09e0 0x7fe57c16a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:20.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.069+0000 7fe582fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe57c0c8620 0x7fe57c16a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:20.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.069+0000 7fe582fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c159840 con 0x7fe57c0b09e0 2024-01-11T08:26:20.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.069+0000 7fe582fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c0ca1a0 con 0x7fe57c069d40 2024-01-11T08:26:20.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.069+0000 7fe582fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c158e70 con 0x7fe57c0c8620 2024-01-11T08:26:20.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.069+0000 7fe57bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0b09e0 0x7fe57c16a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:20.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.069+0000 7fe580d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe57c069d40 0x7fe57c159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:20.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.069+0000 7fe58159c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe57c0c8620 0x7fe57c16a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:20.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.069+0000 7fe57bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0b09e0 0x7fe57c16a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59398/0 (socket says 172.21.15.27:59398) 2024-01-11T08:26:20.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.069+0000 7fe57bfff700 1 -- 172.21.15.27:0/1552915053 learned_addr learned my addr 172.21.15.27:0/1552915053 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:20.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.069+0000 7fe580d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe57c069d40 0x7fe57c159300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:37334/0 (socket says 172.21.15.27:37334) 2024-01-11T08:26:20.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.070+0000 7fe57bfff700 1 -- 172.21.15.27:0/1552915053 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe57c0c8620 msgr2=0x7fe57c16a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:20.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.070+0000 7fe57bfff700 1 --2- 172.21.15.27:0/1552915053 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe57c0c8620 0x7fe57c16a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.070+0000 7fe57bfff700 1 -- 172.21.15.27:0/1552915053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe57c069d40 msgr2=0x7fe57c159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:20.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.070+0000 7fe57bfff700 1 --2- 172.21.15.27:0/1552915053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe57c069d40 0x7fe57c159300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.070+0000 7fe57bfff700 1 -- 172.21.15.27:0/1552915053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe574009ae0 con 0x7fe57c0b09e0 2024-01-11T08:26:20.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.070+0000 7fe580d9b700 1 --2- 172.21.15.27:0/1552915053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe57c069d40 0x7fe57c159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:20.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.070+0000 7fe57bfff700 1 --2- 172.21.15.27:0/1552915053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0b09e0 0x7fe57c16a2b0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fe56800c590 tx=0x7fe56800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:20.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.070+0000 7fe578ff9700 1 -- 172.21.15.27:0/1552915053 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe568011590 con 0x7fe57c0b09e0 2024-01-11T08:26:20.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.070+0000 7fe578ff9700 1 -- 172.21.15.27:0/1552915053 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe568011710 con 0x7fe57c0b09e0 2024-01-11T08:26:20.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.070+0000 7fe578ff9700 1 -- 172.21.15.27:0/1552915053 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe568011ad0 con 0x7fe57c0b09e0 2024-01-11T08:26:20.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.070+0000 7fe582fff700 1 -- 172.21.15.27:0/1552915053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57c0ab5d0 con 0x7fe57c0b09e0 2024-01-11T08:26:20.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.070+0000 7fe582fff700 1 -- 172.21.15.27:0/1552915053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57c158820 con 0x7fe57c0b09e0 2024-01-11T08:26:20.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.071+0000 7fe582fff700 1 -- 172.21.15.27:0/1552915053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe57c158fe0 con 0x7fe57c0b09e0 2024-01-11T08:26:20.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.072+0000 7fe578ff9700 1 -- 172.21.15.27:0/1552915053 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 42) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fe56801b3b0 con 0x7fe57c0b09e0 2024-01-11T08:26:20.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.074+0000 7fe578ff9700 1 --2- 172.21.15.27:0/1552915053 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fe560067050 0x7fe560069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:20.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.074+0000 7fe578ff9700 1 -- 172.21.15.27:0/1552915053 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5991+0+0 (secure 0 0 0) 0x7fe568083c70 con 0x7fe57c0b09e0 2024-01-11T08:26:20.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.074+0000 7fe580d9b700 1 -- 172.21.15.27:0/1552915053 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fe560067050 msgr2=0x7fe560069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:20.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.074+0000 7fe580d9b700 1 --2- 172.21.15.27:0/1552915053 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fe560067050 0x7fe560069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:26:20.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.079+0000 7fe578ff9700 1 -- 172.21.15.27:0/1552915053 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe568050c10 con 0x7fe57c0b09e0 2024-01-11T08:26:20.162 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:20.163+0000 7fb6b233c1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:26:20.262 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:20.263+0000 7f267415b1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:26:20.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.275+0000 7fe580d9b700 1 -- 172.21.15.27:0/1552915053 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fe560067050 msgr2=0x7fe560069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:20.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.275+0000 7fe580d9b700 1 --2- 172.21.15.27:0/1552915053 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fe560067050 0x7fe560069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:26:20.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.285+0000 7fe582fff700 1 -- 172.21.15.27:0/1552915053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe57c0718d0 con 0x7fe57c0b09e0 2024-01-11T08:26:20.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.287+0000 7fe578ff9700 1 -- 172.21.15.27:0/1552915053 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+15714 (secure 0 0 0) 0x7fe568017070 con 0x7fe57c0b09e0 2024-01-11T08:26:20.286 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:20.286 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:26:16.852286+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:25:58.523510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-11T08:25:59.706779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/3506515452":"2024-01-12T08:26:16.852256+0000","172.21.15.77:0/1728351455":"2024-01-12T08:26:16.852256+0000","172.21.15.77:0/2065913513":"2024-01-12T08:26:16.852256+0000","172.21.15.77:6834/41381":"2024-01-12T08:26:16.852256+0000","172.21.15.77:6836/41381":"2024-01-12T08:26:16.852256+0000","172.21.15.77:0/663229260":"2024-01-12T08:26:16.852256+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:26:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.289+0000 7fe582fff700 1 -- 172.21.15.27:0/1552915053 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fe560067050 msgr2=0x7fe560069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:26:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.289+0000 7fe582fff700 1 --2- 172.21.15.27:0/1552915053 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fe560067050 0x7fe560069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.289+0000 7fe582fff700 1 -- 172.21.15.27:0/1552915053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0b09e0 msgr2=0x7fe57c16a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.289+0000 7fe582fff700 1 --2- 172.21.15.27:0/1552915053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0b09e0 0x7fe57c16a2b0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fe56800c590 tx=0x7fe56800c8a0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.289+0000 7fe582fff700 1 -- 172.21.15.27:0/1552915053 shutdown_connections 2024-01-11T08:26:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.289+0000 7fe582fff700 1 --2- 172.21.15.27:0/1552915053 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fe560067050 0x7fe560069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.289+0000 7fe582fff700 1 --2- 172.21.15.27:0/1552915053 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe57c0c8620 0x7fe57c16a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.289+0000 7fe582fff700 1 --2- 172.21.15.27:0/1552915053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe57c069d40 0x7fe57c159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.289+0000 7fe582fff700 1 --2- 172.21.15.27:0/1552915053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0b09e0 0x7fe57c16a2b0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.289+0000 7fe582fff700 1 -- 172.21.15.27:0/1552915053 >> 172.21.15.27:0/1552915053 conn(0x7fe57c0680e0 msgr2=0x7fe57c1526e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.290+0000 7fe582fff700 1 -- 172.21.15.27:0/1552915053 shutdown_connections 2024-01-11T08:26:20.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.290+0000 7fe582fff700 1 -- 172.21.15.27:0/1552915053 wait complete. 2024-01-11T08:26:20.307 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T08:25:58.523510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T08:25:59.706779+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T08:26:20.308 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:26:20.332 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:20.332+0000 7fb6b233c1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:26:20.451 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:20.452+0000 7f267415b1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:26:20.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.519+0000 7f5e449ff700 1 Processor -- start 2024-01-11T08:26:20.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.519+0000 7f5e449ff700 1 -- start start 2024-01-11T08:26:20.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.520+0000 7f5e449ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e40142dc0 0x7f5e400cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:20.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.520+0000 7f5e449ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5e400cb2e0 0x7f5e400cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:20.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.520+0000 7f5e449ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e400ccc70 con 0x7f5e40143190 2024-01-11T08:26:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.520+0000 7f5e449ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e4015df30 con 0x7f5e400cb2e0 2024-01-11T08:26:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.520+0000 7f5e449ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4015d320 con 0x7f5e40142dc0 2024-01-11T08:26:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.521+0000 7f5e3dd9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5e40143190 0x7f5e400cc020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35294/0 (socket says 172.21.15.27:35294) 2024-01-11T08:26:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.521+0000 7f5e3dd9b700 1 -- 172.21.15.27:0/1777804451 learned_addr learned my addr 172.21.15.27:0/1777804451 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.521+0000 7f5e3ed9d700 1 --2- 172.21.15.27:0/1777804451 >> v2:172.21.15.27:3300/0 conn(0x7f5e400cb2e0 0x7f5e400cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.521+0000 7f5e3e59c700 1 --2- 172.21.15.27:0/1777804451 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e40142dc0 0x7f5e400cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.521+0000 7f5e3ed9d700 1 -- 172.21.15.27:0/1777804451 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e40142dc0 msgr2=0x7f5e400cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.521+0000 7f5e3ed9d700 1 --2- 172.21.15.27:0/1777804451 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e40142dc0 0x7f5e400cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.521+0000 7f5e3ed9d700 1 -- 172.21.15.27:0/1777804451 >> v1:172.21.15.27:6789/0 conn(0x7f5e40143190 legacy=0x7f5e400cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.521+0000 7f5e3ed9d700 1 -- 172.21.15.27:0/1777804451 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e400cb700 con 0x7f5e400cb2e0 2024-01-11T08:26:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.521+0000 7f5e3d59a700 1 -- 172.21.15.27:0/1777804451 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016408790 0 0) 0x7f5e400ccc70 con 0x7f5e40143190 2024-01-11T08:26:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.522+0000 7f5e3ed9d700 1 --2- 172.21.15.27:0/1777804451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e400cb2e0 0x7f5e400cc730 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5e38009e60 tx=0x7f5e38009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a83064f1e1a7bdaa server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:20.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.522+0000 7f5e3d59a700 1 -- 172.21.15.27:0/1777804451 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3800e070 con 0x7f5e400cb2e0 2024-01-11T08:26:20.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.522+0000 7f5e3d59a700 1 -- 172.21.15.27:0/1777804451 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3800bb60 con 0x7f5e400cb2e0 2024-01-11T08:26:20.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.522+0000 7f5e3d59a700 1 -- 172.21.15.27:0/1777804451 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e38012430 con 0x7f5e400cb2e0 2024-01-11T08:26:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.522+0000 7f5e449ff700 1 -- 172.21.15.27:0/1777804451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e400cb2e0 msgr2=0x7f5e400cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.522+0000 7f5e449ff700 1 --2- 172.21.15.27:0/1777804451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e400cb2e0 0x7f5e400cc730 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5e38009e60 tx=0x7f5e38009780 comp rx=0 tx=0).stop 2024-01-11T08:26:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.523+0000 7f5e449ff700 1 -- 172.21.15.27:0/1777804451 shutdown_connections 2024-01-11T08:26:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.523+0000 7f5e449ff700 1 --2- 172.21.15.27:0/1777804451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e400cb2e0 0x7f5e400cc730 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.523+0000 7f5e449ff700 1 --2- 172.21.15.27:0/1777804451 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e40142dc0 0x7f5e400cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.523+0000 7f5e449ff700 1 -- 172.21.15.27:0/1777804451 >> 172.21.15.27:0/1777804451 conn(0x7f5e40063ca0 msgr2=0x7f5e4006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.523+0000 7f5e449ff700 1 -- 172.21.15.27:0/1777804451 shutdown_connections 2024-01-11T08:26:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.523+0000 7f5e449ff700 1 -- 172.21.15.27:0/1777804451 wait complete. 2024-01-11T08:26:20.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.525+0000 7f5e449ff700 1 Processor -- start 2024-01-11T08:26:20.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.526+0000 7f5e449ff700 1 -- start start 2024-01-11T08:26:20.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.527+0000 7f5e449ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e400cb2e0 0x7f5e40141630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:20.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.527+0000 7f5e449ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e40142dc0 0x7f5e400c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:20.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.527+0000 7f5e449ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e40143190 0x7f5e400c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:20.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.527+0000 7f5e449ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e400b2840 con 0x7f5e40142dc0 2024-01-11T08:26:20.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.528+0000 7f5e449ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e400c6150 con 0x7f5e40143190 2024-01-11T08:26:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.528+0000 7f5e449ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e400c7500 con 0x7f5e400cb2e0 2024-01-11T08:26:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.528+0000 7f5e3dd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e40142dc0 0x7f5e400c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.528+0000 7f5e3e59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e400cb2e0 0x7f5e40141630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.528+0000 7f5e3dd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e40142dc0 0x7f5e400c6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59414/0 (socket says 172.21.15.27:59414) 2024-01-11T08:26:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.528+0000 7f5e3dd9b700 1 -- 172.21.15.27:0/3620572867 learned_addr learned my addr 172.21.15.27:0/3620572867 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.528+0000 7f5e3ed9d700 1 --2- 172.21.15.27:0/3620572867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e40143190 0x7f5e400c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.528+0000 7f5e3dd9b700 1 -- 172.21.15.27:0/3620572867 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e400cb2e0 msgr2=0x7f5e40141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.528+0000 7f5e3dd9b700 1 --2- 172.21.15.27:0/3620572867 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e400cb2e0 0x7f5e40141630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.528+0000 7f5e3dd9b700 1 -- 172.21.15.27:0/3620572867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e40143190 msgr2=0x7f5e400c5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.528+0000 7f5e3dd9b700 1 --2- 172.21.15.27:0/3620572867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e40143190 0x7f5e400c5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.528+0000 7f5e3dd9b700 1 -- 172.21.15.27:0/3620572867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e38009a20 con 0x7f5e40142dc0 2024-01-11T08:26:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.528+0000 7f5e3dd9b700 1 --2- 172.21.15.27:0/3620572867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e40142dc0 0x7f5e400c6e40 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f5e3400c5b0 tx=0x7f5e34001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.529+0000 7f5e2e7fc700 1 -- 172.21.15.27:0/3620572867 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e34005b30 con 0x7f5e40142dc0 2024-01-11T08:26:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.529+0000 7f5e2e7fc700 1 -- 172.21.15.27:0/3620572867 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e34005c90 con 0x7f5e40142dc0 2024-01-11T08:26:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.529+0000 7f5e449ff700 1 -- 172.21.15.27:0/3620572867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e400c77e0 con 0x7f5e40142dc0 2024-01-11T08:26:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.529+0000 7f5e2e7fc700 1 -- 172.21.15.27:0/3620572867 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e34007320 con 0x7f5e40142dc0 2024-01-11T08:26:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.529+0000 7f5e449ff700 1 -- 172.21.15.27:0/3620572867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e400b24d0 con 0x7f5e40142dc0 2024-01-11T08:26:20.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.530+0000 7f5e449ff700 1 -- 172.21.15.27:0/3620572867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e4004c3c0 con 0x7f5e40142dc0 2024-01-11T08:26:20.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.531+0000 7f5e2e7fc700 1 -- 172.21.15.27:0/3620572867 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 42) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f5e3401b3b0 con 0x7f5e40142dc0 2024-01-11T08:26:20.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.532+0000 7f5e2e7fc700 1 --2- 172.21.15.27:0/3620572867 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5e20067050 0x7f5e20069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:20.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.532+0000 7f5e2e7fc700 1 -- 172.21.15.27:0/3620572867 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5991+0+0 (secure 0 0 0) 0x7f5e34083b90 con 0x7f5e40142dc0 2024-01-11T08:26:20.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.532+0000 7f5e3e59c700 1 -- 172.21.15.27:0/3620572867 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5e20067050 msgr2=0x7f5e20069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:20.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.532+0000 7f5e3e59c700 1 --2- 172.21.15.27:0/3620572867 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5e20067050 0x7f5e20069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:26:20.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.536+0000 7f5e2e7fc700 1 -- 172.21.15.27:0/3620572867 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5e34050bd0 con 0x7f5e40142dc0 2024-01-11T08:26:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.733+0000 7f5e3e59c700 1 -- 172.21.15.27:0/3620572867 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5e20067050 msgr2=0x7f5e20069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.733+0000 7f5e3e59c700 1 --2- 172.21.15.27:0/3620572867 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5e20067050 0x7f5e20069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:26:20.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.739+0000 7f5e449ff700 1 -- 172.21.15.27:0/3620572867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5e400b2980 con 0x7f5e40142dc0 2024-01-11T08:26:20.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.739+0000 7f5e2e7fc700 1 -- 172.21.15.27:0/3620572867 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5e340170a0 con 0x7f5e40142dc0 2024-01-11T08:26:20.738 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:26:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.742+0000 7f5e449ff700 1 -- 172.21.15.27:0/3620572867 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5e20067050 msgr2=0x7f5e20069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:26:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.742+0000 7f5e449ff700 1 --2- 172.21.15.27:0/3620572867 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5e20067050 0x7f5e20069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.742+0000 7f5e449ff700 1 -- 172.21.15.27:0/3620572867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e40142dc0 msgr2=0x7f5e400c6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.742+0000 7f5e449ff700 1 --2- 172.21.15.27:0/3620572867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e40142dc0 0x7f5e400c6e40 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f5e3400c5b0 tx=0x7f5e34001070 comp rx=0 tx=0).stop 2024-01-11T08:26:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.742+0000 7f5e449ff700 1 -- 172.21.15.27:0/3620572867 shutdown_connections 2024-01-11T08:26:20.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.742+0000 7f5e449ff700 1 --2- 172.21.15.27:0/3620572867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e40143190 0x7f5e400c5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.742+0000 7f5e449ff700 1 --2- 172.21.15.27:0/3620572867 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f5e20067050 0x7f5e20069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.742+0000 7f5e449ff700 1 --2- 172.21.15.27:0/3620572867 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e400cb2e0 0x7f5e40141630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.742+0000 7f5e449ff700 1 --2- 172.21.15.27:0/3620572867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e40142dc0 0x7f5e400c6e40 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.742+0000 7f5e449ff700 1 -- 172.21.15.27:0/3620572867 >> 172.21.15.27:0/3620572867 conn(0x7f5e40063ca0 msgr2=0x7f5e4006cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:20.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.742+0000 7f5e449ff700 1 -- 172.21.15.27:0/3620572867 shutdown_connections 2024-01-11T08:26:20.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.742+0000 7f5e449ff700 1 -- 172.21.15.27:0/3620572867 wait complete. 2024-01-11T08:26:20.747 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:20.747+0000 7fb6b233c1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:26:20.759 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T08:26:20.900 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:20.901+0000 7f267415b1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:26:20.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.966+0000 7f71469ff700 1 Processor -- start 2024-01-11T08:26:20.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.967+0000 7f71469ff700 1 -- start start 2024-01-11T08:26:20.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.968+0000 7f71469ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7140157410 0x7f71400ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:20.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.968+0000 7f71469ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71400699d0 con 0x7f71401577e0 2024-01-11T08:26:20.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.968+0000 7f71469ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7140069cb0 con 0x7f7140069540 2024-01-11T08:26:20.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.968+0000 7f71469ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7140069f90 con 0x7f7140157410 2024-01-11T08:26:20.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.968+0000 7f713ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7140157410 0x7f71400ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:20.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.968+0000 7f713ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7140157410 0x7f71400ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59420/0 (socket says 172.21.15.27:59420) 2024-01-11T08:26:20.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.968+0000 7f713ffff700 1 -- 172.21.15.27:0/2185965913 learned_addr learned my addr 172.21.15.27:0/2185965913 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:20.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.969+0000 7f713ffff700 1 -- 172.21.15.27:0/2185965913 >> v1:172.21.15.77:6789/0 conn(0x7f7140069540 legacy=0x7f71400c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:20.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.969+0000 7f713ffff700 1 -- 172.21.15.27:0/2185965913 >> v1:172.21.15.27:6789/0 conn(0x7f71401577e0 legacy=0x7f71400ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:20.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.969+0000 7f713effd700 1 -- 172.21.15.27:0/2185965913 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524656314 0 0) 0x7f7140069cb0 con 0x7f7140069540 2024-01-11T08:26:20.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.969+0000 7f713ffff700 1 -- 172.21.15.27:0/2185965913 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714006a270 con 0x7f7140157410 2024-01-11T08:26:20.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.969+0000 7f713ffff700 1 --2- 172.21.15.27:0/2185965913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7140157410 0x7f71400ccae0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f7130009990 tx=0x7f7130009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79969dbf38445cd2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:20.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.969+0000 7f713effd700 1 -- 172.21.15.27:0/2185965913 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713000b760 con 0x7f7140157410 2024-01-11T08:26:20.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.970+0000 7f713effd700 1 -- 172.21.15.27:0/2185965913 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713000b8c0 con 0x7f7140157410 2024-01-11T08:26:20.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.970+0000 7f713effd700 1 -- 172.21.15.27:0/2185965913 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713000bc60 con 0x7f7140157410 2024-01-11T08:26:20.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.970+0000 7f71469ff700 1 -- 172.21.15.27:0/2185965913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7140157410 msgr2=0x7f71400ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:20.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.970+0000 7f71469ff700 1 --2- 172.21.15.27:0/2185965913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7140157410 0x7f71400ccae0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f7130009990 tx=0x7f7130009220 comp rx=0 tx=0).stop 2024-01-11T08:26:20.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.970+0000 7f71469ff700 1 -- 172.21.15.27:0/2185965913 shutdown_connections 2024-01-11T08:26:20.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.970+0000 7f71469ff700 1 --2- 172.21.15.27:0/2185965913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7140157410 0x7f71400ccae0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.970+0000 7f71469ff700 1 -- 172.21.15.27:0/2185965913 >> 172.21.15.27:0/2185965913 conn(0x7f71400680a0 msgr2=0x7f71400cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:20.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.970+0000 7f71469ff700 1 -- 172.21.15.27:0/2185965913 shutdown_connections 2024-01-11T08:26:20.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.971+0000 7f71469ff700 1 -- 172.21.15.27:0/2185965913 wait complete. 2024-01-11T08:26:20.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.972+0000 7f71469ff700 1 Processor -- start 2024-01-11T08:26:20.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.972+0000 7f71469ff700 1 -- start start 2024-01-11T08:26:20.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.973+0000 7f71469ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7140069540 0x7f714016a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:20.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.973+0000 7f71469ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7140157410 0x7f71400c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:20.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.973+0000 7f71469ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71401577e0 0x7f7140146c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:20.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.973+0000 7f71469ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71400ca2d0 con 0x7f7140069540 2024-01-11T08:26:20.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.973+0000 7f71469ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71400ca4e0 con 0x7f71401577e0 2024-01-11T08:26:20.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.973+0000 7f71469ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71400ca7c0 con 0x7f7140157410 2024-01-11T08:26:20.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.973+0000 7f713ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7140069540 0x7f714016a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:20.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.974+0000 7f713ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7140069540 0x7f714016a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59428/0 (socket says 172.21.15.27:59428) 2024-01-11T08:26:20.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.974+0000 7f713ffff700 1 -- 172.21.15.27:0/1525760769 learned_addr learned my addr 172.21.15.27:0/1525760769 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.974+0000 7f713f7fe700 1 --2- 172.21.15.27:0/1525760769 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7140157410 0x7f71400c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.974+0000 7f7144f9c700 1 --2- 172.21.15.27:0/1525760769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71401577e0 0x7f7140146c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.974+0000 7f713ffff700 1 -- 172.21.15.27:0/1525760769 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7140157410 msgr2=0x7f71400c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.974+0000 7f713ffff700 1 --2- 172.21.15.27:0/1525760769 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7140157410 0x7f71400c9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.974+0000 7f713ffff700 1 -- 172.21.15.27:0/1525760769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71401577e0 msgr2=0x7f7140146c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.974+0000 7f713ffff700 1 --2- 172.21.15.27:0/1525760769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71401577e0 0x7f7140146c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.974+0000 7f713ffff700 1 -- 172.21.15.27:0/1525760769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7130009480 con 0x7f7140069540 2024-01-11T08:26:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.974+0000 7f713ffff700 1 --2- 172.21.15.27:0/1525760769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7140069540 0x7f714016a250 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f7130006f10 tx=0x7f7130016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.974+0000 7f7127fff700 1 -- 172.21.15.27:0/1525760769 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7130011d40 con 0x7f7140069540 2024-01-11T08:26:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.975+0000 7f7127fff700 1 -- 172.21.15.27:0/1525760769 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713001c460 con 0x7f7140069540 2024-01-11T08:26:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.975+0000 7f71469ff700 1 -- 172.21.15.27:0/1525760769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71400caaa0 con 0x7f7140069540 2024-01-11T08:26:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.975+0000 7f7127fff700 1 -- 172.21.15.27:0/1525760769 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713001c820 con 0x7f7140069540 2024-01-11T08:26:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.975+0000 7f71469ff700 1 -- 172.21.15.27:0/1525760769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71400cb060 con 0x7f7140069540 2024-01-11T08:26:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.976+0000 7f71469ff700 1 -- 172.21.15.27:0/1525760769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714004c3c0 con 0x7f7140069540 2024-01-11T08:26:20.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.977+0000 7f7127fff700 1 -- 172.21.15.27:0/1525760769 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 42) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f713001cb90 con 0x7f7140069540 2024-01-11T08:26:20.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.978+0000 7f7127fff700 1 --2- 172.21.15.27:0/1525760769 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f712c067050 0x7f712c069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:20.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.978+0000 7f7127fff700 1 -- 172.21.15.27:0/1525760769 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5991+0+0 (secure 0 0 0) 0x7f7130086c10 con 0x7f7140069540 2024-01-11T08:26:20.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.978+0000 7f713f7fe700 1 -- 172.21.15.27:0/1525760769 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f712c067050 msgr2=0x7f712c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:20.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.978+0000 7f713f7fe700 1 --2- 172.21.15.27:0/1525760769 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f712c067050 0x7f712c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:26:20.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:20.985+0000 7f7127fff700 1 -- 172.21.15.27:0/1525760769 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7130053c50 con 0x7f7140069540 2024-01-11T08:26:21.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.179+0000 7f713f7fe700 1 -- 172.21.15.27:0/1525760769 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f712c067050 msgr2=0x7f712c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:21.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.179+0000 7f713f7fe700 1 --2- 172.21.15.27:0/1525760769 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f712c067050 0x7f712c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:26:21.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.216+0000 7f71469ff700 1 -- 172.21.15.27:0/1525760769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f714004c3c0 con 0x7f7140069540 2024-01-11T08:26:21.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.216+0000 7f7127fff700 1 -- 172.21.15.27:0/1525760769 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7130057270 con 0x7f7140069540 2024-01-11T08:26:21.216 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T08:26:21.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.219+0000 7f71469ff700 1 -- 172.21.15.27:0/1525760769 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f712c067050 msgr2=0x7f712c069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:26:21.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.219+0000 7f71469ff700 1 --2- 172.21.15.27:0/1525760769 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f712c067050 0x7f712c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.219+0000 7f71469ff700 1 -- 172.21.15.27:0/1525760769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7140069540 msgr2=0x7f714016a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:21.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.219+0000 7f71469ff700 1 --2- 172.21.15.27:0/1525760769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7140069540 0x7f714016a250 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f7130006f10 tx=0x7f7130016000 comp rx=0 tx=0).stop 2024-01-11T08:26:21.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.219+0000 7f71469ff700 1 -- 172.21.15.27:0/1525760769 shutdown_connections 2024-01-11T08:26:21.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.219+0000 7f71469ff700 1 --2- 172.21.15.27:0/1525760769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71401577e0 0x7f7140146c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.219+0000 7f71469ff700 1 --2- 172.21.15.27:0/1525760769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7140069540 0x7f714016a250 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.219+0000 7f71469ff700 1 --2- 172.21.15.27:0/1525760769 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f712c067050 0x7f712c069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.219+0000 7f71469ff700 1 --2- 172.21.15.27:0/1525760769 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7140157410 0x7f71400c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.219+0000 7f71469ff700 1 -- 172.21.15.27:0/1525760769 >> 172.21.15.27:0/1525760769 conn(0x7f71400680a0 msgr2=0x7f71400af500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:21.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.219+0000 7f71469ff700 1 -- 172.21.15.27:0/1525760769 shutdown_connections 2024-01-11T08:26:21.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.219+0000 7f71469ff700 1 -- 172.21.15.27:0/1525760769 wait complete. 2024-01-11T08:26:21.237 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T08:26:21.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.442+0000 7f300bfff700 1 Processor -- start 2024-01-11T08:26:21.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.442+0000 7f300bfff700 1 -- start start 2024-01-11T08:26:21.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.443+0000 7f300bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3004069790 0x7f30040c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:21.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.443+0000 7f300bfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f30040c8720 0x7f30040c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:21.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.443+0000 7f300bfff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30040c7420 con 0x7f3004069b60 2024-01-11T08:26:21.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.443+0000 7f300bfff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30040cd0a0 con 0x7f3004069790 2024-01-11T08:26:21.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.443+0000 7f300bfff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30040c7680 con 0x7f30040c8720 2024-01-11T08:26:21.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.444+0000 7f3009d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3004069790 0x7f30040c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:21.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.444+0000 7f3009d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3004069790 0x7f30040c69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59438/0 (socket says 172.21.15.27:59438) 2024-01-11T08:26:21.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.444+0000 7f3009d9b700 1 -- 172.21.15.27:0/101886584 learned_addr learned my addr 172.21.15.27:0/101886584 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.444+0000 7f300959a700 1 --2- 172.21.15.27:0/101886584 >> v2:172.21.15.77:3300/0 conn(0x7f30040c8720 0x7f30040c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.444+0000 7f3009d9b700 1 -- 172.21.15.27:0/101886584 >> v2:172.21.15.77:3300/0 conn(0x7f30040c8720 msgr2=0x7f30040c6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.444+0000 7f3009d9b700 1 --2- 172.21.15.27:0/101886584 >> v2:172.21.15.77:3300/0 conn(0x7f30040c8720 0x7f30040c6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.444+0000 7f3009d9b700 1 -- 172.21.15.27:0/101886584 >> v1:172.21.15.77:6789/0 conn(0x7f3004069b60 legacy=0x7f30041591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.444+0000 7f3009d9b700 1 -- 172.21.15.27:0/101886584 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30040c7960 con 0x7f3004069790 2024-01-11T08:26:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.444+0000 7f3009d9b700 1 --2- 172.21.15.27:0/101886584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3004069790 0x7f30040c69a0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f30040c9d20 tx=0x7f3000009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9de965e214da542 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.445+0000 7f3008d99700 1 -- 172.21.15.27:0/101886584 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3000005c10 con 0x7f3004069790 2024-01-11T08:26:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.445+0000 7f3008d99700 1 -- 172.21.15.27:0/101886584 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3000005d70 con 0x7f3004069790 2024-01-11T08:26:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.445+0000 7f3008d99700 1 -- 172.21.15.27:0/101886584 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3000006260 con 0x7f3004069790 2024-01-11T08:26:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.445+0000 7f300bfff700 1 -- 172.21.15.27:0/101886584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3004069790 msgr2=0x7f30040c69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.445+0000 7f300bfff700 1 --2- 172.21.15.27:0/101886584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3004069790 0x7f30040c69a0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f30040c9d20 tx=0x7f3000009220 comp rx=0 tx=0).stop 2024-01-11T08:26:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.445+0000 7f300bfff700 1 -- 172.21.15.27:0/101886584 shutdown_connections 2024-01-11T08:26:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.445+0000 7f300bfff700 1 --2- 172.21.15.27:0/101886584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3004069790 0x7f30040c69a0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.445+0000 7f300bfff700 1 --2- 172.21.15.27:0/101886584 >> v2:172.21.15.77:3300/0 conn(0x7f30040c8720 0x7f30040c6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.445+0000 7f300bfff700 1 -- 172.21.15.27:0/101886584 >> 172.21.15.27:0/101886584 conn(0x7f30040680a0 msgr2=0x7f30040cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.446+0000 7f300bfff700 1 -- 172.21.15.27:0/101886584 shutdown_connections 2024-01-11T08:26:21.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.446+0000 7f300bfff700 1 -- 172.21.15.27:0/101886584 wait complete. 2024-01-11T08:26:21.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.447+0000 7f300bfff700 1 Processor -- start 2024-01-11T08:26:21.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.447+0000 7f300bfff700 1 -- start start 2024-01-11T08:26:21.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.448+0000 7f300bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3004069790 0x7f300414c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:21.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.448+0000 7f300bfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3004069b60 0x7f30041545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:21.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.448+0000 7f300bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30040c8720 0x7f3004154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:21.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.448+0000 7f300bfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3004151cd0 con 0x7f3004069790 2024-01-11T08:26:21.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.448+0000 7f300bfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30041519f0 con 0x7f30040c8720 2024-01-11T08:26:21.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.448+0000 7f300bfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30040af100 con 0x7f3004069b60 2024-01-11T08:26:21.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.448+0000 7f3009d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3004069790 0x7f300414c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.448+0000 7f3009d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3004069790 0x7f300414c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59448/0 (socket says 172.21.15.27:59448) 2024-01-11T08:26:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.448+0000 7f3009d9b700 1 -- 172.21.15.27:0/805448102 learned_addr learned my addr 172.21.15.27:0/805448102 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.448+0000 7f300959a700 1 --2- 172.21.15.27:0/805448102 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3004069b60 0x7f30041545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.448+0000 7f300a59c700 1 --2- 172.21.15.27:0/805448102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30040c8720 0x7f3004154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.449+0000 7f3009d9b700 1 -- 172.21.15.27:0/805448102 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3004069b60 msgr2=0x7f30041545a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.449+0000 7f3009d9b700 1 --2- 172.21.15.27:0/805448102 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3004069b60 0x7f30041545a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.449+0000 7f3009d9b700 1 -- 172.21.15.27:0/805448102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30040c8720 msgr2=0x7f3004154ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.449+0000 7f3009d9b700 1 --2- 172.21.15.27:0/805448102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30040c8720 0x7f3004154ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.449+0000 7f3009d9b700 1 -- 172.21.15.27:0/805448102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3000009480 con 0x7f3004069790 2024-01-11T08:26:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.449+0000 7f300a59c700 1 --2- 172.21.15.27:0/805448102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30040c8720 0x7f3004154ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:21.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.449+0000 7f3009d9b700 1 --2- 172.21.15.27:0/805448102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3004069790 0x7f300414c9d0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f300000b0e0 tx=0x7f3000016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:21.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.450+0000 7f2ff9ffb700 1 -- 172.21.15.27:0/805448102 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300000b9e0 con 0x7f3004069790 2024-01-11T08:26:21.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.450+0000 7f2ff9ffb700 1 -- 172.21.15.27:0/805448102 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300000bb40 con 0x7f3004069790 2024-01-11T08:26:21.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.450+0000 7f2ff9ffb700 1 -- 172.21.15.27:0/805448102 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300001b430 con 0x7f3004069790 2024-01-11T08:26:21.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.450+0000 7f300bfff700 1 -- 172.21.15.27:0/805448102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30040af3e0 con 0x7f3004069790 2024-01-11T08:26:21.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.450+0000 7f300bfff700 1 -- 172.21.15.27:0/805448102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f300414cf10 con 0x7f3004069790 2024-01-11T08:26:21.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.451+0000 7f300bfff700 1 -- 172.21.15.27:0/805448102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30040aef60 con 0x7f3004069790 2024-01-11T08:26:21.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.452+0000 7f2ff9ffb700 1 -- 172.21.15.27:0/805448102 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 42) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f300001b860 con 0x7f3004069790 2024-01-11T08:26:21.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.453+0000 7f2ff9ffb700 1 --2- 172.21.15.27:0/805448102 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2fe8067050 0x7f2fe8069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:21.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.453+0000 7f2ff9ffb700 1 -- 172.21.15.27:0/805448102 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5991+0+0 (secure 0 0 0) 0x7f3000086440 con 0x7f3004069790 2024-01-11T08:26:21.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.453+0000 7f300959a700 1 -- 172.21.15.27:0/805448102 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2fe8067050 msgr2=0x7f2fe8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:21.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.453+0000 7f300959a700 1 --2- 172.21.15.27:0/805448102 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2fe8067050 0x7f2fe8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:26:21.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.457+0000 7f2ff9ffb700 1 -- 172.21.15.27:0/805448102 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f30040aef60 con 0x7f3004069790 2024-01-11T08:26:21.529 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:21.529+0000 7fb6b233c1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:26:21.617 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:21.617+0000 7fb6b233c1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:26:21.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.654+0000 7f300959a700 1 -- 172.21.15.27:0/805448102 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2fe8067050 msgr2=0x7f2fe8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:21.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.654+0000 7f300959a700 1 --2- 172.21.15.27:0/805448102 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2fe8067050 0x7f2fe8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:26:21.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.676+0000 7f300bfff700 1 -- 172.21.15.27:0/805448102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f30040aef60 con 0x7f3004069790 2024-01-11T08:26:21.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.677+0000 7f2ff9ffb700 1 -- 172.21.15.27:0/805448102 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f30040aef60 con 0x7f3004069790 2024-01-11T08:26:21.676 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T08:26:21.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.680+0000 7f300bfff700 1 -- 172.21.15.27:0/805448102 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2fe8067050 msgr2=0x7f2fe8069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:26:21.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.680+0000 7f300bfff700 1 --2- 172.21.15.27:0/805448102 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2fe8067050 0x7f2fe8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.680+0000 7f300bfff700 1 -- 172.21.15.27:0/805448102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3004069790 msgr2=0x7f300414c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:21.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.680+0000 7f300bfff700 1 --2- 172.21.15.27:0/805448102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3004069790 0x7f300414c9d0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f300000b0e0 tx=0x7f3000016000 comp rx=0 tx=0).stop 2024-01-11T08:26:21.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.680+0000 7f300bfff700 1 -- 172.21.15.27:0/805448102 shutdown_connections 2024-01-11T08:26:21.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.680+0000 7f300bfff700 1 --2- 172.21.15.27:0/805448102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30040c8720 0x7f3004154ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.680+0000 7f300bfff700 1 --2- 172.21.15.27:0/805448102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3004069790 0x7f300414c9d0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.680+0000 7f300bfff700 1 --2- 172.21.15.27:0/805448102 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2fe8067050 0x7f2fe8069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.680+0000 7f300bfff700 1 --2- 172.21.15.27:0/805448102 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3004069b60 0x7f30041545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.680+0000 7f300bfff700 1 -- 172.21.15.27:0/805448102 >> 172.21.15.27:0/805448102 conn(0x7f30040680a0 msgr2=0x7f30040c9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:21.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.681+0000 7f300bfff700 1 -- 172.21.15.27:0/805448102 shutdown_connections 2024-01-11T08:26:21.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.681+0000 7f300bfff700 1 -- 172.21.15.27:0/805448102 wait complete. 2024-01-11T08:26:21.704 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:26:21.769 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:21.769+0000 7fb6b233c1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:26:21.781 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:21.782+0000 7f267415b1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:26:21.856 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:21.857+0000 7fb6b233c1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:26:21.877 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:21.878+0000 7f267415b1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:26:21.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.907+0000 7fee52fff700 1 Processor -- start 2024-01-11T08:26:21.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.907+0000 7fee52fff700 1 -- start start 2024-01-11T08:26:21.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.909+0000 7fee52fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fee4c0ca3e0 0x7fee4c0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:21.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.909+0000 7fee52fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fee4c0ca7b0 0x7fee4c159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:21.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.910+0000 7fee52fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee4c15d640 con 0x7fee4c0c8a20 2024-01-11T08:26:21.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.910+0000 7fee52fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee4c15ae70 con 0x7fee4c0ca7b0 2024-01-11T08:26:21.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.910+0000 7fee52fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee4c15a3c0 con 0x7fee4c0ca3e0 2024-01-11T08:26:21.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.910+0000 7fee4bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fee4c0ca7b0 0x7fee4c159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:21.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.910+0000 7fee4bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fee4c0ca7b0 0x7fee4c159e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59460/0 (socket says 172.21.15.27:59460) 2024-01-11T08:26:21.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.910+0000 7fee4bfff700 1 -- 172.21.15.27:0/3667839512 learned_addr learned my addr 172.21.15.27:0/3667839512 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:21.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.910+0000 7fee50d9b700 1 --2- 172.21.15.27:0/3667839512 >> v2:172.21.15.77:3300/0 conn(0x7fee4c0ca3e0 0x7fee4c0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:21.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.910+0000 7fee4bfff700 1 -- 172.21.15.27:0/3667839512 >> v2:172.21.15.77:3300/0 conn(0x7fee4c0ca3e0 msgr2=0x7fee4c0cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:21.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.910+0000 7fee4bfff700 1 --2- 172.21.15.27:0/3667839512 >> v2:172.21.15.77:3300/0 conn(0x7fee4c0ca3e0 0x7fee4c0cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.910+0000 7fee4bfff700 1 -- 172.21.15.27:0/3667839512 >> v1:172.21.15.27:6789/0 conn(0x7fee4c0c8a20 legacy=0x7fee4c15dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:21.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.910+0000 7fee4bfff700 1 -- 172.21.15.27:0/3667839512 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c15a5d0 con 0x7fee4c0ca7b0 2024-01-11T08:26:21.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.911+0000 7fee4bfff700 1 --2- 172.21.15.27:0/3667839512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee4c0ca7b0 0x7fee4c159e80 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fee380099b0 tx=0x7fee38009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c6e1111cac7dc server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:21.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.911+0000 7fee4b7fe700 1 -- 172.21.15.27:0/3667839512 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee38009ea0 con 0x7fee4c0ca7b0 2024-01-11T08:26:21.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.911+0000 7fee4b7fe700 1 -- 172.21.15.27:0/3667839512 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee3800b820 con 0x7fee4c0ca7b0 2024-01-11T08:26:21.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.911+0000 7fee4b7fe700 1 -- 172.21.15.27:0/3667839512 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3800bbc0 con 0x7fee4c0ca7b0 2024-01-11T08:26:21.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.911+0000 7fee52fff700 1 -- 172.21.15.27:0/3667839512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee4c0ca7b0 msgr2=0x7fee4c159e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:21.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.911+0000 7fee52fff700 1 --2- 172.21.15.27:0/3667839512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee4c0ca7b0 0x7fee4c159e80 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fee380099b0 tx=0x7fee38009250 comp rx=0 tx=0).stop 2024-01-11T08:26:21.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.912+0000 7fee52fff700 1 -- 172.21.15.27:0/3667839512 shutdown_connections 2024-01-11T08:26:21.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.912+0000 7fee52fff700 1 --2- 172.21.15.27:0/3667839512 >> v2:172.21.15.77:3300/0 conn(0x7fee4c0ca3e0 0x7fee4c0cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.912+0000 7fee52fff700 1 --2- 172.21.15.27:0/3667839512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee4c0ca7b0 0x7fee4c159e80 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.912+0000 7fee52fff700 1 -- 172.21.15.27:0/3667839512 >> 172.21.15.27:0/3667839512 conn(0x7fee4c0ce3e0 msgr2=0x7fee4c0ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:21.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.912+0000 7fee52fff700 1 -- 172.21.15.27:0/3667839512 shutdown_connections 2024-01-11T08:26:21.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.912+0000 7fee52fff700 1 -- 172.21.15.27:0/3667839512 wait complete. 2024-01-11T08:26:21.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.914+0000 7fee52fff700 1 Processor -- start 2024-01-11T08:26:21.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.914+0000 7fee52fff700 1 -- start start 2024-01-11T08:26:21.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.916+0000 7fee52fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee4c0c8a20 0x7fee4c0b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:21.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.916+0000 7fee52fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee4c0ca3e0 0x7fee4c1604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:21.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.916+0000 7fee52fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee4c0ca7b0 0x7fee4c1610c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:21.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.916+0000 7fee52fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c0c5e70 con 0x7fee4c0ca7b0 2024-01-11T08:26:21.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.916+0000 7fee52fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c0b2000 con 0x7fee4c0ca3e0 2024-01-11T08:26:21.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.916+0000 7fee52fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c0c6130 con 0x7fee4c0c8a20 2024-01-11T08:26:21.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.916+0000 7fee5159c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee4c0ca7b0 0x7fee4c1610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:21.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.916+0000 7fee5159c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee4c0ca7b0 0x7fee4c1610c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59466/0 (socket says 172.21.15.27:59466) 2024-01-11T08:26:21.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.916+0000 7fee5159c700 1 -- 172.21.15.27:0/4107652815 learned_addr learned my addr 172.21.15.27:0/4107652815 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:21.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.916+0000 7fee4bfff700 1 --2- 172.21.15.27:0/4107652815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee4c0ca3e0 0x7fee4c1604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:21.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.916+0000 7fee50d9b700 1 --2- 172.21.15.27:0/4107652815 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee4c0c8a20 0x7fee4c0b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:21.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.917+0000 7fee5159c700 1 -- 172.21.15.27:0/4107652815 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee4c0c8a20 msgr2=0x7fee4c0b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:21.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.917+0000 7fee5159c700 1 --2- 172.21.15.27:0/4107652815 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee4c0c8a20 0x7fee4c0b1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.917+0000 7fee5159c700 1 -- 172.21.15.27:0/4107652815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee4c0ca3e0 msgr2=0x7fee4c1604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:21.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.917+0000 7fee5159c700 1 --2- 172.21.15.27:0/4107652815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee4c0ca3e0 0x7fee4c1604f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:21.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.917+0000 7fee5159c700 1 -- 172.21.15.27:0/4107652815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee380094f0 con 0x7fee4c0ca7b0 2024-01-11T08:26:21.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.917+0000 7fee5159c700 1 --2- 172.21.15.27:0/4107652815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee4c0ca7b0 0x7fee4c1610c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fee4400c910 tx=0x7fee44001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:21.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.917+0000 7fee48ff9700 1 -- 172.21.15.27:0/4107652815 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee44012630 con 0x7fee4c0ca7b0 2024-01-11T08:26:21.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.917+0000 7fee48ff9700 1 -- 172.21.15.27:0/4107652815 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee44012790 con 0x7fee4c0ca7b0 2024-01-11T08:26:21.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.917+0000 7fee52fff700 1 -- 172.21.15.27:0/4107652815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee4c160a30 con 0x7fee4c0ca7b0 2024-01-11T08:26:21.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.917+0000 7fee48ff9700 1 -- 172.21.15.27:0/4107652815 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee44012b50 con 0x7fee4c0ca7b0 2024-01-11T08:26:21.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.918+0000 7fee52fff700 1 -- 172.21.15.27:0/4107652815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4c0b1a40 con 0x7fee4c0ca7b0 2024-01-11T08:26:21.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.919+0000 7fee52fff700 1 -- 172.21.15.27:0/4107652815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4c0c62c0 con 0x7fee4c0ca7b0 2024-01-11T08:26:21.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.921+0000 7fee48ff9700 1 -- 172.21.15.27:0/4107652815 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 42) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fee4401a3a0 con 0x7fee4c0ca7b0 2024-01-11T08:26:21.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.921+0000 7fee48ff9700 1 --2- 172.21.15.27:0/4107652815 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fee30067050 0x7fee30069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:21.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.922+0000 7fee48ff9700 1 -- 172.21.15.27:0/4107652815 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5991+0+0 (secure 0 0 0) 0x7fee440849b0 con 0x7fee4c0ca7b0 2024-01-11T08:26:21.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.922+0000 7fee50d9b700 1 -- 172.21.15.27:0/4107652815 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fee30067050 msgr2=0x7fee30069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:21.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.922+0000 7fee50d9b700 1 --2- 172.21.15.27:0/4107652815 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fee30067050 0x7fee30069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:26:21.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:21.926+0000 7fee48ff9700 1 -- 172.21.15.27:0/4107652815 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fee440519f0 con 0x7fee4c0ca7b0 2024-01-11T08:26:22.052 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:22.052+0000 7f267415b1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:26:22.074 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:22.074+0000 7fb6b233c1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:26:22.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.123+0000 7fee50d9b700 1 -- 172.21.15.27:0/4107652815 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fee30067050 msgr2=0x7fee30069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:22.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.123+0000 7fee50d9b700 1 --2- 172.21.15.27:0/4107652815 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fee30067050 0x7fee30069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:26:22.144 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:22.145+0000 7f267415b1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:26:22.170 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:22.170+0000 7fb6b233c1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:26:22.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.185+0000 7fee52fff700 1 -- 172.21.15.27:0/4107652815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee4c0718d0 con 0x7fee4c0ca7b0 2024-01-11T08:26:22.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.185+0000 7fee48ff9700 1 -- 172.21.15.27:0/4107652815 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4596 (secure 0 0 0) 0x7fee44055010 con 0x7fee4c0ca7b0 2024-01-11T08:26:22.185 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:22.185 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5945,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1366930493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1366930493},{"type":"v1","addr":"172.21.15.77:6837","nonce":1366930493}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/516828593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":516828593},{"type":"v1","addr":"172.21.15.27:6835","nonce":516828593}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6624,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1462523541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1462523541},{"type":"v1","addr":"172.21.15.27:6834","nonce":1462523541}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:26:00.710104+0000","modified":"2024-01-11T08:26:04.230562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6627":{"gid":6627,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6835/3498364963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3498364963},{"type":"v1","addr":"172.21.15.77:6835","nonce":3498364963}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2024-01-11T08:26:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.188+0000 7fee52fff700 1 -- 172.21.15.27:0/4107652815 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fee30067050 msgr2=0x7fee30069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:26:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.188+0000 7fee52fff700 1 --2- 172.21.15.27:0/4107652815 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fee30067050 0x7fee30069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.188+0000 7fee52fff700 1 -- 172.21.15.27:0/4107652815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee4c0ca7b0 msgr2=0x7fee4c1610c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.188+0000 7fee52fff700 1 --2- 172.21.15.27:0/4107652815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee4c0ca7b0 0x7fee4c1610c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fee4400c910 tx=0x7fee44001710 comp rx=0 tx=0).stop 2024-01-11T08:26:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.188+0000 7fee52fff700 1 -- 172.21.15.27:0/4107652815 shutdown_connections 2024-01-11T08:26:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.188+0000 7fee52fff700 1 --2- 172.21.15.27:0/4107652815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee4c0ca7b0 0x7fee4c1610c0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.188+0000 7fee52fff700 1 --2- 172.21.15.27:0/4107652815 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fee30067050 0x7fee30069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.188+0000 7fee52fff700 1 --2- 172.21.15.27:0/4107652815 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee4c0c8a20 0x7fee4c0b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.188+0000 7fee52fff700 1 --2- 172.21.15.27:0/4107652815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee4c0ca3e0 0x7fee4c1604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.188+0000 7fee52fff700 1 -- 172.21.15.27:0/4107652815 >> 172.21.15.27:0/4107652815 conn(0x7fee4c0ce3e0 msgr2=0x7fee4c15c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.188+0000 7fee52fff700 1 -- 172.21.15.27:0/4107652815 shutdown_connections 2024-01-11T08:26:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.188+0000 7fee52fff700 1 -- 172.21.15.27:0/4107652815 wait complete. 2024-01-11T08:26:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 42 2024-01-11T08:26:22.207 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:26:22.265 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:22.266+0000 7fb6b233c1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:26:22.361 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:22.361+0000 7fb6b233c1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:26:22.378 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:22.379+0000 7f267415b1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:26:22.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.413+0000 7f239cfff700 1 Processor -- start 2024-01-11T08:26:22.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.413+0000 7f239cfff700 1 -- start start 2024-01-11T08:26:22.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.414+0000 7f239cfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f23980b2b90 0x7f2398159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:22.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.414+0000 7f239cfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f23980b07b0 0x7f2398159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:22.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.414+0000 7f239cfff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23980cd730 con 0x7f23980b03e0 2024-01-11T08:26:22.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.414+0000 7f239cfff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23980ccb20 con 0x7f23980b07b0 2024-01-11T08:26:22.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.414+0000 7f239cfff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23980cce00 con 0x7f23980b2b90 2024-01-11T08:26:22.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.415+0000 7f2396d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f23980b07b0 0x7f2398159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:22.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.415+0000 7f2396d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f23980b07b0 0x7f2398159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59470/0 (socket says 172.21.15.27:59470) 2024-01-11T08:26:22.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.415+0000 7f2396d9d700 1 -- 172.21.15.27:0/2724389299 learned_addr learned my addr 172.21.15.27:0/2724389299 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:22.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.415+0000 7f2395d9b700 1 --2- 172.21.15.27:0/2724389299 >> v2:172.21.15.77:3300/0 conn(0x7f23980b2b90 0x7f2398159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:22.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.415+0000 7f2396d9d700 1 -- 172.21.15.27:0/2724389299 >> v2:172.21.15.77:3300/0 conn(0x7f23980b2b90 msgr2=0x7f2398159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:22.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.415+0000 7f2396d9d700 1 --2- 172.21.15.27:0/2724389299 >> v2:172.21.15.77:3300/0 conn(0x7f23980b2b90 0x7f2398159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.415+0000 7f2396d9d700 1 -- 172.21.15.27:0/2724389299 >> v1:172.21.15.27:6789/0 conn(0x7f23980b03e0 legacy=0x7f23980b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:22.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.415+0000 7f2396d9d700 1 -- 172.21.15.27:0/2724389299 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23980cd0e0 con 0x7f23980b07b0 2024-01-11T08:26:22.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.415+0000 7f2396d9d700 1 --2- 172.21.15.27:0/2724389299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980b07b0 0x7f2398159740 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f23980cdfc0 tx=0x7f2390009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf44a02a07cf2f3c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:22.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.416+0000 7f239559a700 1 -- 172.21.15.27:0/2724389299 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239000e070 con 0x7f23980b07b0 2024-01-11T08:26:22.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.416+0000 7f239559a700 1 -- 172.21.15.27:0/2724389299 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239000ba20 con 0x7f23980b07b0 2024-01-11T08:26:22.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.416+0000 7f239559a700 1 -- 172.21.15.27:0/2724389299 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23900092e0 con 0x7f23980b07b0 2024-01-11T08:26:22.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.416+0000 7f239cfff700 1 -- 172.21.15.27:0/2724389299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980b07b0 msgr2=0x7f2398159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:22.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.416+0000 7f239cfff700 1 --2- 172.21.15.27:0/2724389299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980b07b0 0x7f2398159740 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f23980cdfc0 tx=0x7f2390009840 comp rx=0 tx=0).stop 2024-01-11T08:26:22.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.416+0000 7f239cfff700 1 -- 172.21.15.27:0/2724389299 shutdown_connections 2024-01-11T08:26:22.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.416+0000 7f239cfff700 1 --2- 172.21.15.27:0/2724389299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980b07b0 0x7f2398159740 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.416+0000 7f239cfff700 1 --2- 172.21.15.27:0/2724389299 >> v2:172.21.15.77:3300/0 conn(0x7f23980b2b90 0x7f2398159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.416+0000 7f239cfff700 1 -- 172.21.15.27:0/2724389299 >> 172.21.15.27:0/2724389299 conn(0x7f239806ed40 msgr2=0x7f239806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:22.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.416+0000 7f239cfff700 1 -- 172.21.15.27:0/2724389299 shutdown_connections 2024-01-11T08:26:22.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.416+0000 7f239cfff700 1 -- 172.21.15.27:0/2724389299 wait complete. 2024-01-11T08:26:22.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.418+0000 7f239cfff700 1 Processor -- start 2024-01-11T08:26:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.418+0000 7f239cfff700 1 -- start start 2024-01-11T08:26:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.419+0000 7f239cfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980b03e0 0x7f2398169620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.419+0000 7f239cfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23980b07b0 0x7f23980c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.419+0000 7f239cfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23980b2b90 0x7f23980c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.419+0000 7f239cfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23980c9f70 con 0x7f23980b03e0 2024-01-11T08:26:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.419+0000 7f239cfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23980c5030 con 0x7f23980b07b0 2024-01-11T08:26:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.419+0000 7f239cfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239816a5c0 con 0x7f23980b2b90 2024-01-11T08:26:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.419+0000 7f239659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980b03e0 0x7f2398169620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.419+0000 7f239659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980b03e0 0x7f2398169620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59480/0 (socket says 172.21.15.27:59480) 2024-01-11T08:26:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.419+0000 7f239659c700 1 -- 172.21.15.27:0/2170575845 learned_addr learned my addr 172.21.15.27:0/2170575845 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.419+0000 7f2395d9b700 1 --2- 172.21.15.27:0/2170575845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23980b07b0 0x7f23980c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:22.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.419+0000 7f2396d9d700 1 --2- 172.21.15.27:0/2170575845 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23980b2b90 0x7f23980c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:22.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.420+0000 7f239659c700 1 -- 172.21.15.27:0/2170575845 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23980b2b90 msgr2=0x7f23980c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:22.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.420+0000 7f239659c700 1 --2- 172.21.15.27:0/2170575845 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23980b2b90 0x7f23980c9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.420+0000 7f239659c700 1 -- 172.21.15.27:0/2170575845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23980b07b0 msgr2=0x7f23980c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:22.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.420+0000 7f239659c700 1 --2- 172.21.15.27:0/2170575845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23980b07b0 0x7f23980c6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.420+0000 7f239659c700 1 -- 172.21.15.27:0/2170575845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2390009ae0 con 0x7f23980b03e0 2024-01-11T08:26:22.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.420+0000 7f239659c700 1 --2- 172.21.15.27:0/2170575845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980b03e0 0x7f2398169620 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f238800c590 tx=0x7f238800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:22.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.420+0000 7f23867fc700 1 -- 172.21.15.27:0/2170575845 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2388011590 con 0x7f23980b03e0 2024-01-11T08:26:22.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.420+0000 7f23867fc700 1 -- 172.21.15.27:0/2170575845 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2388011710 con 0x7f23980b03e0 2024-01-11T08:26:22.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.420+0000 7f23867fc700 1 -- 172.21.15.27:0/2170575845 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2388011ad0 con 0x7f23980b03e0 2024-01-11T08:26:22.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.420+0000 7f239cfff700 1 -- 172.21.15.27:0/2170575845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2398169d20 con 0x7f23980b03e0 2024-01-11T08:26:22.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.420+0000 7f239cfff700 1 -- 172.21.15.27:0/2170575845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f239816a2e0 con 0x7f23980b03e0 2024-01-11T08:26:22.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.421+0000 7f239cfff700 1 -- 172.21.15.27:0/2170575845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23980718d0 con 0x7f23980b03e0 2024-01-11T08:26:22.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.422+0000 7f23867fc700 1 -- 172.21.15.27:0/2170575845 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 42) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f238801b3b0 con 0x7f23980b03e0 2024-01-11T08:26:22.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.424+0000 7f23867fc700 1 --2- 172.21.15.27:0/2170575845 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2378067050 0x7f2378069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:22.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.424+0000 7f23867fc700 1 -- 172.21.15.27:0/2170575845 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5991+0+0 (secure 0 0 0) 0x7f2388083c70 con 0x7f23980b03e0 2024-01-11T08:26:22.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.424+0000 7f2395d9b700 1 -- 172.21.15.27:0/2170575845 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2378067050 msgr2=0x7f2378069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:22.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.424+0000 7f2395d9b700 1 --2- 172.21.15.27:0/2170575845 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2378067050 0x7f2378069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:26:22.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.427+0000 7f23867fc700 1 -- 172.21.15.27:0/2170575845 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2388050c10 con 0x7f23980b03e0 2024-01-11T08:26:22.485 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:22.486+0000 7f267415b1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:26:22.550 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:22.550+0000 7fb6b233c1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:26:22.562 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:26:22.566 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:26:22.589 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:22.590+0000 7f267415b1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:26:22.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.625+0000 7f2395d9b700 1 -- 172.21.15.27:0/2170575845 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2378067050 msgr2=0x7f2378069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:22.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.625+0000 7f2395d9b700 1 --2- 172.21.15.27:0/2170575845 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2378067050 0x7f2378069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:26:22.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.630+0000 7f239cfff700 1 -- 172.21.15.27:0/2170575845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23980cd3c0 con 0x7f23980b03e0 2024-01-11T08:26:22.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.631+0000 7f23867fc700 1 -- 172.21.15.27:0/2170575845 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+15714 (secure 0 0 0) 0x7f2388017070 con 0x7f23980b03e0 2024-01-11T08:26:22.631 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:22.631 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:26:16.852286+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:25:58.523510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-11T08:25:59.706779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/3506515452":"2024-01-12T08:26:16.852256+0000","172.21.15.77:0/1728351455":"2024-01-12T08:26:16.852256+0000","172.21.15.77:0/2065913513":"2024-01-12T08:26:16.852256+0000","172.21.15.77:6834/41381":"2024-01-12T08:26:16.852256+0000","172.21.15.77:6836/41381":"2024-01-12T08:26:16.852256+0000","172.21.15.77:0/663229260":"2024-01-12T08:26:16.852256+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:26:22.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.634+0000 7f239cfff700 1 -- 172.21.15.27:0/2170575845 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2378067050 msgr2=0x7f2378069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:26:22.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.634+0000 7f239cfff700 1 --2- 172.21.15.27:0/2170575845 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2378067050 0x7f2378069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.634+0000 7f239cfff700 1 -- 172.21.15.27:0/2170575845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980b03e0 msgr2=0x7f2398169620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:22.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.634+0000 7f239cfff700 1 --2- 172.21.15.27:0/2170575845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980b03e0 0x7f2398169620 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f238800c590 tx=0x7f238800c8a0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.634+0000 7f239cfff700 1 -- 172.21.15.27:0/2170575845 shutdown_connections 2024-01-11T08:26:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.634+0000 7f239cfff700 1 --2- 172.21.15.27:0/2170575845 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f2378067050 0x7f2378069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.634+0000 7f239cfff700 1 --2- 172.21.15.27:0/2170575845 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23980b2b90 0x7f23980c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.634+0000 7f239cfff700 1 --2- 172.21.15.27:0/2170575845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980b03e0 0x7f2398169620 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.634+0000 7f239cfff700 1 --2- 172.21.15.27:0/2170575845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23980b07b0 0x7f23980c6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.634+0000 7f239cfff700 1 -- 172.21.15.27:0/2170575845 >> 172.21.15.27:0/2170575845 conn(0x7f239806ed40 msgr2=0x7f23980b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.634+0000 7f239cfff700 1 -- 172.21.15.27:0/2170575845 shutdown_connections 2024-01-11T08:26:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.634+0000 7f239cfff700 1 -- 172.21.15.27:0/2170575845 wait complete. 2024-01-11T08:26:22.652 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-11T08:26:22.652 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:26:22.653 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:22.652+0000 7fb6b233c1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:26:22.694 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:22.695+0000 7f267415b1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:26:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.895+0000 7fc2d5dff700 1 Processor -- start 2024-01-11T08:26:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.895+0000 7fc2d5dff700 1 -- start start 2024-01-11T08:26:22.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.897+0000 7fc2d5dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2d0159650 0x7fc2d006bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:22.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.897+0000 7fc2d5dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2d00b2730 0x7fc2d00b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:22.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.897+0000 7fc2d5dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d006fba0 con 0x7fc2d0159a20 2024-01-11T08:26:22.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.897+0000 7fc2d5dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2d006e700 con 0x7fc2d00b2730 2024-01-11T08:26:22.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.897+0000 7fc2d5dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d006c360 con 0x7fc2d0159650 2024-01-11T08:26:22.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.897+0000 7fc2cffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2d00b2730 0x7fc2d00b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:22.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.897+0000 7fc2ceffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc2d0159a20 0x7fc2d00b0c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35330/0 (socket says 172.21.15.27:35330) 2024-01-11T08:26:22.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.897+0000 7fc2cffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2d00b2730 0x7fc2d00b2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59492/0 (socket says 172.21.15.27:59492) 2024-01-11T08:26:22.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.897+0000 7fc2cf7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2d0159650 0x7fc2d006bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:22.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.897+0000 7fc2ceffd700 1 -- 172.21.15.27:0/1057550319 learned_addr learned my addr 172.21.15.27:0/1057550319 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:22.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.898+0000 7fc2cf7fe700 1 -- 172.21.15.27:0/1057550319 >> v2:172.21.15.27:3300/0 conn(0x7fc2d00b2730 msgr2=0x7fc2d00b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:22.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.898+0000 7fc2cf7fe700 1 --2- 172.21.15.27:0/1057550319 >> v2:172.21.15.27:3300/0 conn(0x7fc2d00b2730 0x7fc2d00b2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.898+0000 7fc2cf7fe700 1 -- 172.21.15.27:0/1057550319 >> v1:172.21.15.27:6789/0 conn(0x7fc2d0159a20 legacy=0x7fc2d00b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:22.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.898+0000 7fc2cf7fe700 1 -- 172.21.15.27:0/1057550319 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d006c640 con 0x7fc2d0159650 2024-01-11T08:26:22.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.898+0000 7fc2cf7fe700 1 --2- 172.21.15.27:0/1057550319 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2d0159650 0x7fc2d006bd80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc2c00098e0 tx=0x7fc2c00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c23a860a97857641 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:22.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.898+0000 7fc2ce7fc700 1 -- 172.21.15.27:0/1057550319 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c000b640 con 0x7fc2d0159650 2024-01-11T08:26:22.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.898+0000 7fc2ce7fc700 1 -- 172.21.15.27:0/1057550319 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c000b7a0 con 0x7fc2d0159650 2024-01-11T08:26:22.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.898+0000 7fc2ce7fc700 1 -- 172.21.15.27:0/1057550319 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c000bb40 con 0x7fc2d0159650 2024-01-11T08:26:22.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.899+0000 7fc2d5dff700 1 -- 172.21.15.27:0/1057550319 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2d0159650 msgr2=0x7fc2d006bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:22.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.899+0000 7fc2d5dff700 1 --2- 172.21.15.27:0/1057550319 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2d0159650 0x7fc2d006bd80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc2c00098e0 tx=0x7fc2c00091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.899+0000 7fc2d5dff700 1 -- 172.21.15.27:0/1057550319 shutdown_connections 2024-01-11T08:26:22.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.899+0000 7fc2d5dff700 1 --2- 172.21.15.27:0/1057550319 >> v2:172.21.15.27:3300/0 conn(0x7fc2d00b2730 0x7fc2d00b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.899+0000 7fc2d5dff700 1 --2- 172.21.15.27:0/1057550319 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2d0159650 0x7fc2d006bd80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.899+0000 7fc2d5dff700 1 -- 172.21.15.27:0/1057550319 >> 172.21.15.27:0/1057550319 conn(0x7fc2d006b200 msgr2=0x7fc2d00aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:22.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.899+0000 7fc2d5dff700 1 -- 172.21.15.27:0/1057550319 shutdown_connections 2024-01-11T08:26:22.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.899+0000 7fc2d5dff700 1 -- 172.21.15.27:0/1057550319 wait complete. 2024-01-11T08:26:22.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.901+0000 7fc2d5dff700 1 Processor -- start 2024-01-11T08:26:22.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.901+0000 7fc2d5dff700 1 -- start start 2024-01-11T08:26:22.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.903+0000 7fc2d5dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2d00b2730 0x7fc2d00cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:22.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.903+0000 7fc2d5dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d0159650 0x7fc2d016ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:22.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.903+0000 7fc2d5dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2d0159a20 0x7fc2d0169c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:22.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.903+0000 7fc2d5dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d00ac4e0 con 0x7fc2d0159650 2024-01-11T08:26:22.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.903+0000 7fc2d5dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d016a780 con 0x7fc2d00b2730 2024-01-11T08:26:22.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.903+0000 7fc2d5dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d016aa60 con 0x7fc2d0159a20 2024-01-11T08:26:22.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.903+0000 7fc2ceffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d0159650 0x7fc2d016ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:22.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.903+0000 7fc2ceffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d0159650 0x7fc2d016ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59504/0 (socket says 172.21.15.27:59504) 2024-01-11T08:26:22.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.903+0000 7fc2ceffd700 1 -- 172.21.15.27:0/3329278096 learned_addr learned my addr 172.21.15.27:0/3329278096 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:22.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.903+0000 7fc2cffff700 1 --2- 172.21.15.27:0/3329278096 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2d0159a20 0x7fc2d0169c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:22.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.903+0000 7fc2cf7fe700 1 --2- 172.21.15.27:0/3329278096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2d00b2730 0x7fc2d00cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:22.903 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:22.903+0000 7f267415b1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:26:22.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.904+0000 7fc2ceffd700 1 -- 172.21.15.27:0/3329278096 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2d0159a20 msgr2=0x7fc2d0169c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:22.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.904+0000 7fc2ceffd700 1 --2- 172.21.15.27:0/3329278096 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2d0159a20 0x7fc2d0169c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.904+0000 7fc2ceffd700 1 -- 172.21.15.27:0/3329278096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2d00b2730 msgr2=0x7fc2d00cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:22.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.904+0000 7fc2ceffd700 1 --2- 172.21.15.27:0/3329278096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2d00b2730 0x7fc2d00cdce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:22.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.904+0000 7fc2ceffd700 1 -- 172.21.15.27:0/3329278096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c0009420 con 0x7fc2d0159650 2024-01-11T08:26:22.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.904+0000 7fc2cf7fe700 1 --2- 172.21.15.27:0/3329278096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2d00b2730 0x7fc2d00cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:22.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.904+0000 7fc2cffff700 1 --2- 172.21.15.27:0/3329278096 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2d0159a20 0x7fc2d0169c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:22.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.904+0000 7fc2ceffd700 1 --2- 172.21.15.27:0/3329278096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d0159650 0x7fc2d016ae90 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fc2c400c560 tx=0x7fc2c4001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:22.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.904+0000 7fc2b77fe700 1 -- 172.21.15.27:0/3329278096 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4005b30 con 0x7fc2d0159650 2024-01-11T08:26:22.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.904+0000 7fc2b77fe700 1 -- 172.21.15.27:0/3329278096 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c4005c90 con 0x7fc2d0159650 2024-01-11T08:26:22.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.904+0000 7fc2b77fe700 1 -- 172.21.15.27:0/3329278096 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4007320 con 0x7fc2d0159650 2024-01-11T08:26:22.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.904+0000 7fc2d5dff700 1 -- 172.21.15.27:0/3329278096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d00ac170 con 0x7fc2d0159650 2024-01-11T08:26:22.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.904+0000 7fc2d5dff700 1 -- 172.21.15.27:0/3329278096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d016b3d0 con 0x7fc2d0159650 2024-01-11T08:26:22.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.906+0000 7fc2d5dff700 1 -- 172.21.15.27:0/3329278096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2d00ac690 con 0x7fc2d0159650 2024-01-11T08:26:22.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.906+0000 7fc2b77fe700 1 -- 172.21.15.27:0/3329278096 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 42) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fc2c401b3b0 con 0x7fc2d0159650 2024-01-11T08:26:22.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.907+0000 7fc2b77fe700 1 --2- 172.21.15.27:0/3329278096 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fc2bc067050 0x7fc2bc069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:22.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.907+0000 7fc2b77fe700 1 -- 172.21.15.27:0/3329278096 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5991+0+0 (secure 0 0 0) 0x7fc2c4083b30 con 0x7fc2d0159650 2024-01-11T08:26:22.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.907+0000 7fc2cf7fe700 1 -- 172.21.15.27:0/3329278096 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fc2bc067050 msgr2=0x7fc2bc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:22.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.908+0000 7fc2cf7fe700 1 --2- 172.21.15.27:0/3329278096 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fc2bc067050 0x7fc2bc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:26:22.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:22.911+0000 7fc2b77fe700 1 -- 172.21.15.27:0/3329278096 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc2d00ac690 con 0x7fc2d0159650 2024-01-11T08:26:23.000 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:23.001+0000 7fb6b233c1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:26:23.009 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:23.010+0000 7f267415b1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:26:23.089 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:23.090+0000 7fb6b233c1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:26:23.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.109+0000 7fc2cf7fe700 1 -- 172.21.15.27:0/3329278096 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fc2bc067050 msgr2=0x7fc2bc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:23.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.109+0000 7fc2cf7fe700 1 --2- 172.21.15.27:0/3329278096 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fc2bc067050 0x7fc2bc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:26:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.161+0000 7fc2d5dff700 1 -- 172.21.15.27:0/3329278096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2d016ac80 con 0x7fc2d0159650 2024-01-11T08:26:23.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.162+0000 7fc2b77fe700 1 -- 172.21.15.27:0/3329278096 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4596 (secure 0 0 0) 0x7fc2c4017090 con 0x7fc2d0159650 2024-01-11T08:26:23.161 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:23.161 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5945,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1366930493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1366930493},{"type":"v1","addr":"172.21.15.77:6837","nonce":1366930493}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/516828593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":516828593},{"type":"v1","addr":"172.21.15.27:6835","nonce":516828593}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6624,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1462523541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1462523541},{"type":"v1","addr":"172.21.15.27:6834","nonce":1462523541}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:26:00.710104+0000","modified":"2024-01-11T08:26:04.230562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6627":{"gid":6627,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6835/3498364963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3498364963},{"type":"v1","addr":"172.21.15.77:6835","nonce":3498364963}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2024-01-11T08:26:23.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.164+0000 7fc2d5dff700 1 -- 172.21.15.27:0/3329278096 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fc2bc067050 msgr2=0x7fc2bc069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:26:23.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.164+0000 7fc2d5dff700 1 --2- 172.21.15.27:0/3329278096 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fc2bc067050 0x7fc2bc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.164+0000 7fc2d5dff700 1 -- 172.21.15.27:0/3329278096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d0159650 msgr2=0x7fc2d016ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:23.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.164+0000 7fc2d5dff700 1 --2- 172.21.15.27:0/3329278096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d0159650 0x7fc2d016ae90 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fc2c400c560 tx=0x7fc2c4001020 comp rx=0 tx=0).stop 2024-01-11T08:26:23.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.165+0000 7fc2d5dff700 1 -- 172.21.15.27:0/3329278096 shutdown_connections 2024-01-11T08:26:23.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.165+0000 7fc2d5dff700 1 --2- 172.21.15.27:0/3329278096 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fc2bc067050 0x7fc2bc069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.165+0000 7fc2d5dff700 1 --2- 172.21.15.27:0/3329278096 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2d0159a20 0x7fc2d0169c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.165+0000 7fc2d5dff700 1 --2- 172.21.15.27:0/3329278096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2d00b2730 0x7fc2d00cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.165+0000 7fc2d5dff700 1 --2- 172.21.15.27:0/3329278096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2d0159650 0x7fc2d016ae90 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.165+0000 7fc2d5dff700 1 -- 172.21.15.27:0/3329278096 >> 172.21.15.27:0/3329278096 conn(0x7fc2d006b200 msgr2=0x7fc2d015c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.165+0000 7fc2d5dff700 1 -- 172.21.15.27:0/3329278096 shutdown_connections 2024-01-11T08:26:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.165+0000 7fc2d5dff700 1 -- 172.21.15.27:0/3329278096 wait complete. 2024-01-11T08:26:23.166 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 42 2024-01-11T08:26:23.185 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:26:23.376 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:23.377+0000 7f267415b1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:26:23.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.390+0000 7fcc33bff700 1 Processor -- start 2024-01-11T08:26:23.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.391+0000 7fcc33bff700 1 -- start start 2024-01-11T08:26:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.392+0000 7fcc33bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcc2c1599e0 0x7fcc2c0b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.392+0000 7fcc33bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc2c159db0 0x7fcc2c15a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.392+0000 7fcc33bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc2c14fc20 0x7fcc2c150080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.392+0000 7fcc33bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc2c157e30 con 0x7fcc2c14fc20 2024-01-11T08:26:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.392+0000 7fcc33bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc2c157500 con 0x7fcc2c1599e0 2024-01-11T08:26:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.392+0000 7fcc33bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c1577e0 con 0x7fcc2c159db0 2024-01-11T08:26:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.393+0000 7fcc3219c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc2c14fc20 0x7fcc2c150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.393+0000 7fcc3119a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc2c159db0 0x7fcc2c15a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.393+0000 7fcc3199b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcc2c1599e0 0x7fcc2c0b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:23.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.393+0000 7fcc3119a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc2c159db0 0x7fcc2c15a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:42006/0 (socket says 172.21.15.27:42006) 2024-01-11T08:26:23.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.393+0000 7fcc3219c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc2c14fc20 0x7fcc2c150080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59508/0 (socket says 172.21.15.27:59508) 2024-01-11T08:26:23.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.393+0000 7fcc3119a700 1 -- 172.21.15.27:0/3340745772 learned_addr learned my addr 172.21.15.27:0/3340745772 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:23.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.393+0000 7fcc3119a700 1 -- 172.21.15.27:0/3340745772 >> v2:172.21.15.77:3300/0 conn(0x7fcc2c1599e0 msgr2=0x7fcc2c0b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:23.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.393+0000 7fcc3119a700 1 --2- 172.21.15.27:0/3340745772 >> v2:172.21.15.77:3300/0 conn(0x7fcc2c1599e0 0x7fcc2c0b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.393+0000 7fcc3119a700 1 -- 172.21.15.27:0/3340745772 >> v2:172.21.15.27:3300/0 conn(0x7fcc2c14fc20 msgr2=0x7fcc2c150080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:23.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.393+0000 7fcc3119a700 1 --2- 172.21.15.27:0/3340745772 >> v2:172.21.15.27:3300/0 conn(0x7fcc2c14fc20 0x7fcc2c150080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.393+0000 7fcc3119a700 1 -- 172.21.15.27:0/3340745772 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2c157ac0 con 0x7fcc2c159db0 2024-01-11T08:26:23.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.393+0000 7fcc3199b700 1 --2- 172.21.15.27:0/3340745772 >> v2:172.21.15.77:3300/0 conn(0x7fcc2c1599e0 0x7fcc2c0b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:23.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.394+0000 7fcc3119a700 1 --2- 172.21.15.27:0/3340745772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc2c159db0 0x7fcc2c15a200 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c070d50 tx=0x7fcc1c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6139ac659de571d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:23.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.394+0000 7fcc30999700 1 -- 172.21.15.27:0/3340745772 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c00b640 con 0x7fcc2c159db0 2024-01-11T08:26:23.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.394+0000 7fcc30999700 1 -- 172.21.15.27:0/3340745772 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1c00b7a0 con 0x7fcc2c159db0 2024-01-11T08:26:23.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.394+0000 7fcc30999700 1 -- 172.21.15.27:0/3340745772 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c00bb60 con 0x7fcc2c159db0 2024-01-11T08:26:23.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.395+0000 7fcc33bff700 1 -- 172.21.15.27:0/3340745772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc2c159db0 msgr2=0x7fcc2c15a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:23.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.395+0000 7fcc33bff700 1 --2- 172.21.15.27:0/3340745772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc2c159db0 0x7fcc2c15a200 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c070d50 tx=0x7fcc1c009250 comp rx=0 tx=0).stop 2024-01-11T08:26:23.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.395+0000 7fcc33bff700 1 -- 172.21.15.27:0/3340745772 shutdown_connections 2024-01-11T08:26:23.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.395+0000 7fcc33bff700 1 --2- 172.21.15.27:0/3340745772 >> v2:172.21.15.27:3300/0 conn(0x7fcc2c14fc20 0x7fcc2c150080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.395+0000 7fcc33bff700 1 --2- 172.21.15.27:0/3340745772 >> v2:172.21.15.77:3300/0 conn(0x7fcc2c1599e0 0x7fcc2c0b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.395+0000 7fcc33bff700 1 --2- 172.21.15.27:0/3340745772 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc2c159db0 0x7fcc2c15a200 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.395+0000 7fcc33bff700 1 -- 172.21.15.27:0/3340745772 >> 172.21.15.27:0/3340745772 conn(0x7fcc2c0680e0 msgr2=0x7fcc2c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:23.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.395+0000 7fcc33bff700 1 -- 172.21.15.27:0/3340745772 shutdown_connections 2024-01-11T08:26:23.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.395+0000 7fcc33bff700 1 -- 172.21.15.27:0/3340745772 wait complete. 2024-01-11T08:26:23.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.397+0000 7fcc33bff700 1 Processor -- start 2024-01-11T08:26:23.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.397+0000 7fcc33bff700 1 -- start start 2024-01-11T08:26:23.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.398+0000 7fcc33bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc2c14fc20 0x7fcc2c0adc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:23.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.398+0000 7fcc33bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2c1599e0 0x7fcc2c1582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:23.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.398+0000 7fcc33bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc2c159db0 0x7fcc2c158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:23.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.399+0000 7fcc33bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c16af70 con 0x7fcc2c1599e0 2024-01-11T08:26:23.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.399+0000 7fcc33bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c16a1d0 con 0x7fcc2c14fc20 2024-01-11T08:26:23.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.399+0000 7fcc33bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c16a4b0 con 0x7fcc2c159db0 2024-01-11T08:26:23.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.399+0000 7fcc3119a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2c1599e0 0x7fcc2c1582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:23.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.399+0000 7fcc3119a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2c1599e0 0x7fcc2c1582c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59524/0 (socket says 172.21.15.27:59524) 2024-01-11T08:26:23.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.399+0000 7fcc3119a700 1 -- 172.21.15.27:0/288874691 learned_addr learned my addr 172.21.15.27:0/288874691 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:23.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.399+0000 7fcc3219c700 1 --2- 172.21.15.27:0/288874691 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc2c159db0 0x7fcc2c158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:23.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.399+0000 7fcc3199b700 1 --2- 172.21.15.27:0/288874691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc2c14fc20 0x7fcc2c0adc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:23.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.399+0000 7fcc3119a700 1 -- 172.21.15.27:0/288874691 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc2c159db0 msgr2=0x7fcc2c158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:23.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.399+0000 7fcc3119a700 1 --2- 172.21.15.27:0/288874691 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc2c159db0 0x7fcc2c158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.399+0000 7fcc3119a700 1 -- 172.21.15.27:0/288874691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc2c14fc20 msgr2=0x7fcc2c0adc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:23.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.399+0000 7fcc3119a700 1 --2- 172.21.15.27:0/288874691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc2c14fc20 0x7fcc2c0adc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.399+0000 7fcc3119a700 1 -- 172.21.15.27:0/288874691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc28009420 con 0x7fcc2c1599e0 2024-01-11T08:26:23.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.399+0000 7fcc3199b700 1 --2- 172.21.15.27:0/288874691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc2c14fc20 0x7fcc2c0adc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:23.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.400+0000 7fcc3119a700 1 --2- 172.21.15.27:0/288874691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2c1599e0 0x7fcc2c1582c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c157920 tx=0x7fcc1c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:23.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.400+0000 7fcc21ffb700 1 -- 172.21.15.27:0/288874691 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c00b910 con 0x7fcc2c1599e0 2024-01-11T08:26:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.400+0000 7fcc21ffb700 1 -- 172.21.15.27:0/288874691 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1c00ba70 con 0x7fcc2c1599e0 2024-01-11T08:26:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.400+0000 7fcc21ffb700 1 -- 172.21.15.27:0/288874691 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c01a680 con 0x7fcc2c1599e0 2024-01-11T08:26:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.400+0000 7fcc33bff700 1 -- 172.21.15.27:0/288874691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1c0094f0 con 0x7fcc2c1599e0 2024-01-11T08:26:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.400+0000 7fcc33bff700 1 -- 172.21.15.27:0/288874691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc2c16aa70 con 0x7fcc2c1599e0 2024-01-11T08:26:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.401+0000 7fcc33bff700 1 -- 172.21.15.27:0/288874691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc2c154500 con 0x7fcc2c1599e0 2024-01-11T08:26:23.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.402+0000 7fcc21ffb700 1 -- 172.21.15.27:0/288874691 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 42) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fcc1c01aa70 con 0x7fcc2c1599e0 2024-01-11T08:26:23.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.403+0000 7fcc21ffb700 1 --2- 172.21.15.27:0/288874691 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc10067050 0x7fcc10069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:23.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.403+0000 7fcc21ffb700 1 -- 172.21.15.27:0/288874691 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5991+0+0 (secure 0 0 0) 0x7fcc1c085f80 con 0x7fcc2c1599e0 2024-01-11T08:26:23.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.403+0000 7fcc3199b700 1 -- 172.21.15.27:0/288874691 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc10067050 msgr2=0x7fcc10069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:23.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.403+0000 7fcc3199b700 1 --2- 172.21.15.27:0/288874691 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc10067050 0x7fcc10069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:26:23.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.407+0000 7fcc21ffb700 1 -- 172.21.15.27:0/288874691 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcc1c0b75a0 con 0x7fcc2c1599e0 2024-01-11T08:26:23.475 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:23.476+0000 7f267415b1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:26:23.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.604+0000 7fcc3199b700 1 -- 172.21.15.27:0/288874691 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc10067050 msgr2=0x7fcc10069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:23.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.604+0000 7fcc3199b700 1 --2- 172.21.15.27:0/288874691 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc10067050 0x7fcc10069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:26:23.630 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:23.630+0000 7fb6b233c1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:26:23.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.654+0000 7fcc33bff700 1 -- 172.21.15.27:0/288874691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc2c0718d0 con 0x7fcc2c1599e0 2024-01-11T08:26:23.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.655+0000 7fcc21ffb700 1 -- 172.21.15.27:0/288874691 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4596 (secure 0 0 0) 0x7fcc2c154500 con 0x7fcc2c1599e0 2024-01-11T08:26:23.654 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:23.654 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5945,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1366930493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1366930493},{"type":"v1","addr":"172.21.15.77:6837","nonce":1366930493}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/516828593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":516828593},{"type":"v1","addr":"172.21.15.27:6835","nonce":516828593}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6624,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1462523541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1462523541},{"type":"v1","addr":"172.21.15.27:6834","nonce":1462523541}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:26:00.710104+0000","modified":"2024-01-11T08:26:04.230562+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6627":{"gid":6627,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6835/3498364963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3498364963},{"type":"v1","addr":"172.21.15.77:6835","nonce":3498364963}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2024-01-11T08:26:23.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.657+0000 7fcc33bff700 1 -- 172.21.15.27:0/288874691 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc10067050 msgr2=0x7fcc10069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:26:23.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.657+0000 7fcc33bff700 1 --2- 172.21.15.27:0/288874691 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc10067050 0x7fcc10069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.657+0000 7fcc33bff700 1 -- 172.21.15.27:0/288874691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2c1599e0 msgr2=0x7fcc2c1582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:23.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.657+0000 7fcc33bff700 1 --2- 172.21.15.27:0/288874691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2c1599e0 0x7fcc2c1582c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c157920 tx=0x7fcc1c017000 comp rx=0 tx=0).stop 2024-01-11T08:26:23.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.657+0000 7fcc33bff700 1 -- 172.21.15.27:0/288874691 shutdown_connections 2024-01-11T08:26:23.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.657+0000 7fcc33bff700 1 --2- 172.21.15.27:0/288874691 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc10067050 0x7fcc10069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.657+0000 7fcc33bff700 1 --2- 172.21.15.27:0/288874691 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc2c159db0 0x7fcc2c158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.657+0000 7fcc33bff700 1 --2- 172.21.15.27:0/288874691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc2c14fc20 0x7fcc2c0adc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.657+0000 7fcc33bff700 1 --2- 172.21.15.27:0/288874691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc2c1599e0 0x7fcc2c1582c0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.657+0000 7fcc33bff700 1 -- 172.21.15.27:0/288874691 >> 172.21.15.27:0/288874691 conn(0x7fcc2c0680e0 msgr2=0x7fcc2c14d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:23.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.657+0000 7fcc33bff700 1 -- 172.21.15.27:0/288874691 shutdown_connections 2024-01-11T08:26:23.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.657+0000 7fcc33bff700 1 -- 172.21.15.27:0/288874691 wait complete. 2024-01-11T08:26:23.658 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 42 2024-01-11T08:26:23.675 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:26:23.816 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:23.816+0000 7fb6b233c1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:26:23.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.878+0000 7f9e9b7ff700 1 Processor -- start 2024-01-11T08:26:23.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.878+0000 7f9e9b7ff700 1 -- start start 2024-01-11T08:26:23.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.879+0000 7f9e9b7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9e940ca7b0 0x7f9e940cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:23.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.879+0000 7f9e9b7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9415d640 con 0x7f9e94159e80 2024-01-11T08:26:23.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.879+0000 7f9e9b7ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9415a5d0 con 0x7f9e940ca3e0 2024-01-11T08:26:23.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.879+0000 7f9e9b7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e9415ae70 con 0x7f9e940ca7b0 2024-01-11T08:26:23.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.880+0000 7f9e99d9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9e94159e80 0x7f9e9415dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56098/0 (socket says 172.21.15.27:56098) 2024-01-11T08:26:23.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.880+0000 7f9e99d9c700 1 -- 172.21.15.27:0/3582891177 learned_addr learned my addr 172.21.15.27:0/3582891177 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:23.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.880+0000 7f9e98d9a700 1 --2- 172.21.15.27:0/3582891177 >> v2:172.21.15.77:3300/0 conn(0x7f9e940ca7b0 0x7f9e940cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:23.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.880+0000 7f9e8bfff700 1 -- 172.21.15.27:0/3582891177 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349550060 0 0) 0x7f9e9415d640 con 0x7f9e94159e80 2024-01-11T08:26:23.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.880+0000 7f9e8bfff700 1 -- 172.21.15.27:0/3582891177 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e80003570 con 0x7f9e94159e80 2024-01-11T08:26:23.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.880+0000 7f9e8bfff700 1 -- 172.21.15.27:0/3582891177 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903537198 0 0) 0x7f9e9415a5d0 con 0x7f9e940ca3e0 2024-01-11T08:26:23.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.880+0000 7f9e8bfff700 1 -- 172.21.15.27:0/3582891177 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e9415d640 con 0x7f9e940ca3e0 2024-01-11T08:26:23.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.880+0000 7f9e98d9a700 1 -- 172.21.15.27:0/3582891177 >> v1:172.21.15.77:6789/0 conn(0x7f9e940ca3e0 legacy=0x7f9e940c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:23.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.880+0000 7f9e98d9a700 1 -- 172.21.15.27:0/3582891177 >> v1:172.21.15.27:6789/0 conn(0x7f9e94159e80 legacy=0x7f9e9415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:23.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.880+0000 7f9e98d9a700 1 -- 172.21.15.27:0/3582891177 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9415a8b0 con 0x7f9e940ca7b0 2024-01-11T08:26:23.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.881+0000 7f9e98d9a700 1 --2- 172.21.15.27:0/3582891177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e940ca7b0 0x7f9e940cca20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9e84009910 tx=0x7f9e840091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2ba1a7ec55e1ec4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:23.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.881+0000 7f9e8bfff700 1 -- 172.21.15.27:0/3582891177 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e84011630 con 0x7f9e940ca7b0 2024-01-11T08:26:23.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.881+0000 7f9e8bfff700 1 -- 172.21.15.27:0/3582891177 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e84011790 con 0x7f9e940ca7b0 2024-01-11T08:26:23.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.881+0000 7f9e8bfff700 1 -- 172.21.15.27:0/3582891177 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e84011b30 con 0x7f9e940ca7b0 2024-01-11T08:26:23.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.882+0000 7f9e9b7ff700 1 -- 172.21.15.27:0/3582891177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e940ca7b0 msgr2=0x7f9e940cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:23.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.882+0000 7f9e9b7ff700 1 --2- 172.21.15.27:0/3582891177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e940ca7b0 0x7f9e940cca20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9e84009910 tx=0x7f9e840091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.882+0000 7f9e9b7ff700 1 -- 172.21.15.27:0/3582891177 shutdown_connections 2024-01-11T08:26:23.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.882+0000 7f9e9b7ff700 1 --2- 172.21.15.27:0/3582891177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e940ca7b0 0x7f9e940cca20 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.882+0000 7f9e9b7ff700 1 -- 172.21.15.27:0/3582891177 >> 172.21.15.27:0/3582891177 conn(0x7f9e940ce3e0 msgr2=0x7f9e940ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:23.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.882+0000 7f9e9b7ff700 1 -- 172.21.15.27:0/3582891177 shutdown_connections 2024-01-11T08:26:23.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.882+0000 7f9e9b7ff700 1 -- 172.21.15.27:0/3582891177 wait complete. 2024-01-11T08:26:23.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.883+0000 7f9e9b7ff700 1 Processor -- start 2024-01-11T08:26:23.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.883+0000 7f9e9b7ff700 1 -- start start 2024-01-11T08:26:23.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.884+0000 7f9e9b7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e940ca3e0 0x7f9e9414b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:23.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.884+0000 7f9e9b7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e940ca7b0 0x7f9e940b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:23.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.884+0000 7f9e9b7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e94159e80 0x7f9e940b38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:23.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.884+0000 7f9e9b7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9414bc80 con 0x7f9e940ca7b0 2024-01-11T08:26:23.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.884+0000 7f9e9b7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e940b3430 con 0x7f9e94159e80 2024-01-11T08:26:23.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.884+0000 7f9e9b7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e940b2820 con 0x7f9e940ca3e0 2024-01-11T08:26:23.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.885+0000 7f9e98d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e940ca7b0 0x7f9e940b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:23.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.885+0000 7f9e9959b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e940ca3e0 0x7f9e9414b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:23.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.885+0000 7f9e98d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e940ca7b0 0x7f9e940b1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53930/0 (socket says 172.21.15.27:53930) 2024-01-11T08:26:23.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.885+0000 7f9e98d9a700 1 -- 172.21.15.27:0/2356093796 learned_addr learned my addr 172.21.15.27:0/2356093796 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:23.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.885+0000 7f9e99d9c700 1 --2- 172.21.15.27:0/2356093796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e94159e80 0x7f9e940b38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:23.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.885+0000 7f9e98d9a700 1 -- 172.21.15.27:0/2356093796 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e940ca3e0 msgr2=0x7f9e9414b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:23.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.885+0000 7f9e98d9a700 1 --2- 172.21.15.27:0/2356093796 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e940ca3e0 0x7f9e9414b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.885+0000 7f9e98d9a700 1 -- 172.21.15.27:0/2356093796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e94159e80 msgr2=0x7f9e940b38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:23.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.885+0000 7f9e98d9a700 1 --2- 172.21.15.27:0/2356093796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e94159e80 0x7f9e940b38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:23.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.885+0000 7f9e98d9a700 1 -- 172.21.15.27:0/2356093796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e84009420 con 0x7f9e940ca7b0 2024-01-11T08:26:23.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.885+0000 7f9e98d9a700 1 --2- 172.21.15.27:0/2356093796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e940ca7b0 0x7f9e940b1f80 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f9e94064980 tx=0x7f9e84017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:23.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.886+0000 7f9e897fa700 1 -- 172.21.15.27:0/2356093796 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8401f070 con 0x7f9e940ca7b0 2024-01-11T08:26:23.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.886+0000 7f9e897fa700 1 -- 172.21.15.27:0/2356093796 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e84006af0 con 0x7f9e940ca7b0 2024-01-11T08:26:23.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.886+0000 7f9e897fa700 1 -- 172.21.15.27:0/2356093796 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e84023410 con 0x7f9e940ca7b0 2024-01-11T08:26:23.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.886+0000 7f9e9b7ff700 1 -- 172.21.15.27:0/2356093796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e940b2b00 con 0x7f9e940ca7b0 2024-01-11T08:26:23.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.886+0000 7f9e9b7ff700 1 -- 172.21.15.27:0/2356093796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e940b30c0 con 0x7f9e940ca7b0 2024-01-11T08:26:23.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.887+0000 7f9e9b7ff700 1 -- 172.21.15.27:0/2356093796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e940718d0 con 0x7f9e940ca7b0 2024-01-11T08:26:23.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.888+0000 7f9e897fa700 1 -- 172.21.15.27:0/2356093796 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 42) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f9e8402c3b0 con 0x7f9e940ca7b0 2024-01-11T08:26:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.889+0000 7f9e897fa700 1 --2- 172.21.15.27:0/2356093796 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e78067050 0x7f9e78069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.889+0000 7f9e897fa700 1 -- 172.21.15.27:0/2356093796 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5991+0+0 (secure 0 0 0) 0x7f9e84085860 con 0x7f9e940ca7b0 2024-01-11T08:26:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.889+0000 7f9e9959b700 1 -- 172.21.15.27:0/2356093796 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e78067050 msgr2=0x7f9e78069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.889+0000 7f9e9959b700 1 --2- 172.21.15.27:0/2356093796 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e78067050 0x7f9e78069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:26:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:23.893+0000 7f9e897fa700 1 -- 172.21.15.27:0/2356093796 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9e840528a0 con 0x7f9e940ca7b0 2024-01-11T08:26:24.065 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:24.066+0000 7f267415b1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:26:24.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.090+0000 7f9e9959b700 1 -- 172.21.15.27:0/2356093796 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e78067050 msgr2=0x7f9e78069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:24.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.090+0000 7f9e9959b700 1 --2- 172.21.15.27:0/2356093796 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e78067050 0x7f9e78069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:26:24.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.098+0000 7f9e9b7ff700 1 -- 172.21.15.27:0/2356093796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e9406c840 con 0x7f9e940ca7b0 2024-01-11T08:26:24.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.099+0000 7f9e897fa700 1 -- 172.21.15.27:0/2356093796 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+15714 (secure 0 0 0) 0x7f9e84015090 con 0x7f9e940ca7b0 2024-01-11T08:26:24.098 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:24.099 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:26:16.852286+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:25:58.523510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-11T08:25:59.706779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/3506515452":"2024-01-12T08:26:16.852256+0000","172.21.15.77:0/1728351455":"2024-01-12T08:26:16.852256+0000","172.21.15.77:0/2065913513":"2024-01-12T08:26:16.852256+0000","172.21.15.77:6834/41381":"2024-01-12T08:26:16.852256+0000","172.21.15.77:6836/41381":"2024-01-12T08:26:16.852256+0000","172.21.15.77:0/663229260":"2024-01-12T08:26:16.852256+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:26:24.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.101+0000 7f9e9b7ff700 1 -- 172.21.15.27:0/2356093796 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e78067050 msgr2=0x7f9e78069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:26:24.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.101+0000 7f9e9b7ff700 1 --2- 172.21.15.27:0/2356093796 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e78067050 0x7f9e78069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.101+0000 7f9e9b7ff700 1 -- 172.21.15.27:0/2356093796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e940ca7b0 msgr2=0x7f9e940b1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:24.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.101+0000 7f9e9b7ff700 1 --2- 172.21.15.27:0/2356093796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e940ca7b0 0x7f9e940b1f80 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f9e94064980 tx=0x7f9e84017000 comp rx=0 tx=0).stop 2024-01-11T08:26:24.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.101+0000 7f9e9b7ff700 1 -- 172.21.15.27:0/2356093796 shutdown_connections 2024-01-11T08:26:24.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.101+0000 7f9e9b7ff700 1 --2- 172.21.15.27:0/2356093796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e94159e80 0x7f9e940b38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.101+0000 7f9e9b7ff700 1 --2- 172.21.15.27:0/2356093796 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e78067050 0x7f9e78069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.101+0000 7f9e9b7ff700 1 --2- 172.21.15.27:0/2356093796 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e940ca3e0 0x7f9e9414b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.101+0000 7f9e9b7ff700 1 --2- 172.21.15.27:0/2356093796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e940ca7b0 0x7f9e940b1f80 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.101+0000 7f9e9b7ff700 1 -- 172.21.15.27:0/2356093796 >> 172.21.15.27:0/2356093796 conn(0x7f9e940ce3e0 msgr2=0x7f9e94152090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:24.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.102+0000 7f9e9b7ff700 1 -- 172.21.15.27:0/2356093796 shutdown_connections 2024-01-11T08:26:24.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.102+0000 7f9e9b7ff700 1 -- 172.21.15.27:0/2356093796 wait complete. 2024-01-11T08:26:24.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-11T08:26:24.191 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:24.191+0000 7fb6b233c1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:26:24.272 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:24.272+0000 7f267415b1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:26:24.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.325+0000 7fac7ffff700 1 Processor -- start 2024-01-11T08:26:24.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.325+0000 7fac7ffff700 1 -- start start 2024-01-11T08:26:24.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.326+0000 7fac7ffff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac7806bf10 0x7fac78159c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:24.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.326+0000 7fac7ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fac780b0a40 0x7fac7815da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:24.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.326+0000 7fac7ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fac7815df40 0x7fac78068720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:24.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.326+0000 7fac7ffff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac7815d290 con 0x7fac780b0a40 2024-01-11T08:26:24.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.326+0000 7fac7ffff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac7815d570 con 0x7fac7815df40 2024-01-11T08:26:24.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.326+0000 7fac7ffff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac78154180 con 0x7fac7806bf10 2024-01-11T08:26:24.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.327+0000 7fac7d59a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fac780b0a40 0x7fac7815da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:24.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.327+0000 7fac7d59a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fac780b0a40 0x7fac7815da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53942/0 (socket says 172.21.15.27:53942) 2024-01-11T08:26:24.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.327+0000 7fac7d59a700 1 -- 172.21.15.27:0/2991331260 learned_addr learned my addr 172.21.15.27:0/2991331260 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:24.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.327+0000 7fac7dd9b700 1 --2- 172.21.15.27:0/2991331260 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac7806bf10 0x7fac78159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:24.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.327+0000 7fac7e59c700 1 --2- 172.21.15.27:0/2991331260 >> v2:172.21.15.77:3300/0 conn(0x7fac7815df40 0x7fac78068720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:24.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.327+0000 7fac7d59a700 1 -- 172.21.15.27:0/2991331260 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac7806bf10 msgr2=0x7fac78159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:24.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.327+0000 7fac7d59a700 1 --2- 172.21.15.27:0/2991331260 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac7806bf10 0x7fac78159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.327+0000 7fac7d59a700 1 -- 172.21.15.27:0/2991331260 >> v2:172.21.15.77:3300/0 conn(0x7fac7815df40 msgr2=0x7fac78068720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:24.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.327+0000 7fac7d59a700 1 --2- 172.21.15.27:0/2991331260 >> v2:172.21.15.77:3300/0 conn(0x7fac7815df40 0x7fac78068720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.327+0000 7fac7d59a700 1 -- 172.21.15.27:0/2991331260 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac7815a510 con 0x7fac780b0a40 2024-01-11T08:26:24.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.328+0000 7fac7d59a700 1 --2- 172.21.15.27:0/2991331260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac780b0a40 0x7fac7815da00 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fac78157860 tx=0x7fac68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7df002e038db7fd2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:24.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.328+0000 7fac7cd99700 1 -- 172.21.15.27:0/2991331260 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6800b640 con 0x7fac780b0a40 2024-01-11T08:26:24.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.328+0000 7fac7cd99700 1 -- 172.21.15.27:0/2991331260 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6800b7a0 con 0x7fac780b0a40 2024-01-11T08:26:24.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.328+0000 7fac7cd99700 1 -- 172.21.15.27:0/2991331260 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6800bb40 con 0x7fac780b0a40 2024-01-11T08:26:24.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.328+0000 7fac7ffff700 1 -- 172.21.15.27:0/2991331260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac780b0a40 msgr2=0x7fac7815da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:24.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.328+0000 7fac7ffff700 1 --2- 172.21.15.27:0/2991331260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac780b0a40 0x7fac7815da00 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fac78157860 tx=0x7fac68009250 comp rx=0 tx=0).stop 2024-01-11T08:26:24.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.328+0000 7fac7ffff700 1 -- 172.21.15.27:0/2991331260 shutdown_connections 2024-01-11T08:26:24.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.329+0000 7fac7ffff700 1 --2- 172.21.15.27:0/2991331260 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac7806bf10 0x7fac78159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.329+0000 7fac7ffff700 1 --2- 172.21.15.27:0/2991331260 >> v2:172.21.15.77:3300/0 conn(0x7fac7815df40 0x7fac78068720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.329+0000 7fac7ffff700 1 --2- 172.21.15.27:0/2991331260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac780b0a40 0x7fac7815da00 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.329+0000 7fac7ffff700 1 -- 172.21.15.27:0/2991331260 >> 172.21.15.27:0/2991331260 conn(0x7fac780ce3c0 msgr2=0x7fac780ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:24.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.329+0000 7fac7ffff700 1 -- 172.21.15.27:0/2991331260 shutdown_connections 2024-01-11T08:26:24.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.329+0000 7fac7ffff700 1 -- 172.21.15.27:0/2991331260 wait complete. 2024-01-11T08:26:24.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.330+0000 7fac7ffff700 1 Processor -- start 2024-01-11T08:26:24.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.330+0000 7fac7ffff700 1 -- start start 2024-01-11T08:26:24.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.331+0000 7fac7ffff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac7806bf10 0x7fac78153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:24.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.331+0000 7fac7ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac780b0a40 0x7fac78155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:24.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.331+0000 7fac7ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac7815df40 0x7fac780b3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:24.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.331+0000 7fac7ffff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac78155940 con 0x7fac780b0a40 2024-01-11T08:26:24.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.331+0000 7fac7ffff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac780b1390 con 0x7fac7815df40 2024-01-11T08:26:24.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.331+0000 7fac7ffff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac7816a420 con 0x7fac7806bf10 2024-01-11T08:26:24.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.331+0000 7fac7d59a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac780b0a40 0x7fac78155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:24.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.331+0000 7fac7dd9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac7806bf10 0x7fac78153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:24.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.331+0000 7fac7d59a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac780b0a40 0x7fac78155dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53954/0 (socket says 172.21.15.27:53954) 2024-01-11T08:26:24.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.332+0000 7fac7dd9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac7806bf10 0x7fac78153ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:37710/0 (socket says 172.21.15.27:37710) 2024-01-11T08:26:24.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.332+0000 7fac7d59a700 1 -- 172.21.15.27:0/3134321396 learned_addr learned my addr 172.21.15.27:0/3134321396 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:24.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.332+0000 7fac7e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac7815df40 0x7fac780b3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:24.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.332+0000 7fac7dd9b700 1 -- 172.21.15.27:0/3134321396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac7815df40 msgr2=0x7fac780b3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:24.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.332+0000 7fac7dd9b700 1 --2- 172.21.15.27:0/3134321396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac7815df40 0x7fac780b3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.332+0000 7fac7dd9b700 1 -- 172.21.15.27:0/3134321396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac780b0a40 msgr2=0x7fac78155dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:24.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.332+0000 7fac7dd9b700 1 --2- 172.21.15.27:0/3134321396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac780b0a40 0x7fac78155dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.332+0000 7fac7dd9b700 1 -- 172.21.15.27:0/3134321396 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac680094f0 con 0x7fac7806bf10 2024-01-11T08:26:24.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.332+0000 7fac7d59a700 1 --2- 172.21.15.27:0/3134321396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac780b0a40 0x7fac78155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:24.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.332+0000 7fac7dd9b700 1 --2- 172.21.15.27:0/3134321396 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac7806bf10 0x7fac78153ae0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fac7400e900 tx=0x7fac740091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:24.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.333+0000 7fac6dffb700 1 -- 172.21.15.27:0/3134321396 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac74013680 con 0x7fac7806bf10 2024-01-11T08:26:24.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.333+0000 7fac6dffb700 1 -- 172.21.15.27:0/3134321396 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac740137e0 con 0x7fac7806bf10 2024-01-11T08:26:24.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.333+0000 7fac6dffb700 1 -- 172.21.15.27:0/3134321396 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac74013ba0 con 0x7fac7806bf10 2024-01-11T08:26:24.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.333+0000 7fac7ffff700 1 -- 172.21.15.27:0/3134321396 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac7816a700 con 0x7fac7806bf10 2024-01-11T08:26:24.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.333+0000 7fac7ffff700 1 -- 172.21.15.27:0/3134321396 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac781555d0 con 0x7fac7806bf10 2024-01-11T08:26:24.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.334+0000 7fac7ffff700 1 -- 172.21.15.27:0/3134321396 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac7804c3c0 con 0x7fac7806bf10 2024-01-11T08:26:24.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.335+0000 7fac6dffb700 1 -- 172.21.15.27:0/3134321396 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 42) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fac740123d0 con 0x7fac7806bf10 2024-01-11T08:26:24.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.336+0000 7fac6dffb700 1 --2- 172.21.15.27:0/3134321396 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fac5c067050 0x7fac5c069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:24.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.336+0000 7fac6dffb700 1 -- 172.21.15.27:0/3134321396 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5991+0+0 (secure 0 0 0) 0x7fac740848c0 con 0x7fac7806bf10 2024-01-11T08:26:24.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.336+0000 7fac7d59a700 1 -- 172.21.15.27:0/3134321396 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fac5c067050 msgr2=0x7fac5c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:24.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.336+0000 7fac7d59a700 1 --2- 172.21.15.27:0/3134321396 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fac5c067050 0x7fac5c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:26:24.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.340+0000 7fac6dffb700 1 -- 172.21.15.27:0/3134321396 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fac74051860 con 0x7fac7806bf10 2024-01-11T08:26:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.537+0000 7fac7d59a700 1 -- 172.21.15.27:0/3134321396 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fac5c067050 msgr2=0x7fac5c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6834/41381 2024-01-11T08:26:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.537+0000 7fac7d59a700 1 --2- 172.21.15.27:0/3134321396 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fac5c067050 0x7fac5c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:26:24.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.591+0000 7fac7ffff700 1 -- 172.21.15.27:0/3134321396 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fac7815a7f0 con 0x7fac7806bf10 2024-01-11T08:26:24.591 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-11T08:26:24.592+0000 7f27065f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-11T08:26:24.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.597+0000 7fac6dffb700 1 -- 172.21.15.27:0/3134321396 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) v1 ==== 161+0+0 (secure 0 0 0) 0x7fac74021090 con 0x7fac7806bf10 2024-01-11T08:26:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.601+0000 7fac7ffff700 1 -- 172.21.15.27:0/3134321396 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fac5c067050 msgr2=0x7fac5c069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:26:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.601+0000 7fac7ffff700 1 --2- 172.21.15.27:0/3134321396 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fac5c067050 0x7fac5c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.601+0000 7fac7ffff700 1 -- 172.21.15.27:0/3134321396 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac7806bf10 msgr2=0x7fac78153ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.601+0000 7fac7ffff700 1 --2- 172.21.15.27:0/3134321396 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac7806bf10 0x7fac78153ae0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fac7400e900 tx=0x7fac740091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.601+0000 7fac7ffff700 1 -- 172.21.15.27:0/3134321396 shutdown_connections 2024-01-11T08:26:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.601+0000 7fac7ffff700 1 --2- 172.21.15.27:0/3134321396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac7815df40 0x7fac780b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.601+0000 7fac7ffff700 1 --2- 172.21.15.27:0/3134321396 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fac5c067050 0x7fac5c069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.601+0000 7fac7ffff700 1 --2- 172.21.15.27:0/3134321396 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac7806bf10 0x7fac78153ae0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.601+0000 7fac7ffff700 1 --2- 172.21.15.27:0/3134321396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac780b0a40 0x7fac78155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.601+0000 7fac7ffff700 1 -- 172.21.15.27:0/3134321396 >> 172.21.15.27:0/3134321396 conn(0x7fac780ce3c0 msgr2=0x7fac780cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:24.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.601+0000 7fac7ffff700 1 -- 172.21.15.27:0/3134321396 shutdown_connections 2024-01-11T08:26:24.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.601+0000 7fac7ffff700 1 -- 172.21.15.27:0/3134321396 wait complete. 2024-01-11T08:26:24.601 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-11T08:26:24.605 INFO:tasks.ceph.mds.d.smithi077.stderr: -1178> 2024-01-11T08:26:07.182+0000 7fac26ff1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T08:26:24.621 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:24.621+0000 7fb6b233c1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:26:24.626 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-11T08:26:24.645 INFO:tasks.ceph.mds.d.smithi077.stdout:starting mds.d at 2024-01-11T08:26:24.688 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:24.689+0000 7f267415b1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:26:24.806 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:24.806+0000 7fb6b233c1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:26:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.835+0000 7fb50bdff700 1 Processor -- start 2024-01-11T08:26:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.835+0000 7fb50bdff700 1 -- start start 2024-01-11T08:26:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.837+0000 7fb50bdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb50406a040 0x7fb50415a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.837+0000 7fb50bdff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb50406a410 0x7fb5040cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.837+0000 7fb50bdff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb5040cc450 0x7fb504153600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.837+0000 7fb50bdff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb50415ab90 con 0x7fb50406a410 2024-01-11T08:26:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.837+0000 7fb50bdff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5040cc850 con 0x7fb5040cc450 2024-01-11T08:26:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.837+0000 7fb50bdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb50406cdb0 con 0x7fb50406a040 2024-01-11T08:26:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.837+0000 7fb50939a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb50406a410 0x7fb5040cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.837+0000 7fb509b9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb50406a040 0x7fb50415a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.837+0000 7fb50939a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb50406a410 0x7fb5040cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53970/0 (socket says 172.21.15.27:53970) 2024-01-11T08:26:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.837+0000 7fb50939a700 1 -- 172.21.15.27:0/75224784 learned_addr learned my addr 172.21.15.27:0/75224784 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.837+0000 7fb50a39c700 1 --2- 172.21.15.27:0/75224784 >> v2:172.21.15.77:3300/0 conn(0x7fb5040cc450 0x7fb504153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.838+0000 7fb509b9b700 1 -- 172.21.15.27:0/75224784 >> v2:172.21.15.77:3300/0 conn(0x7fb5040cc450 msgr2=0x7fb504153600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.838+0000 7fb509b9b700 1 --2- 172.21.15.27:0/75224784 >> v2:172.21.15.77:3300/0 conn(0x7fb5040cc450 0x7fb504153600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.838+0000 7fb509b9b700 1 -- 172.21.15.27:0/75224784 >> v2:172.21.15.27:3300/0 conn(0x7fb50406a410 msgr2=0x7fb5040cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.838+0000 7fb509b9b700 1 --2- 172.21.15.27:0/75224784 >> v2:172.21.15.27:3300/0 conn(0x7fb50406a410 0x7fb5040cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.838+0000 7fb509b9b700 1 -- 172.21.15.27:0/75224784 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb50406d090 con 0x7fb50406a040 2024-01-11T08:26:24.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.838+0000 7fb50a39c700 1 --2- 172.21.15.27:0/75224784 >> v2:172.21.15.77:3300/0 conn(0x7fb5040cc450 0x7fb504153600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:24.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.838+0000 7fb50939a700 1 --2- 172.21.15.27:0/75224784 >> v2:172.21.15.27:3300/0 conn(0x7fb50406a410 0x7fb5040cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:24.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.838+0000 7fb509b9b700 1 --2- 172.21.15.27:0/75224784 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb50406a040 0x7fb50415a650 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb50406f6a0 tx=0x7fb5000091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c90501652b2c74e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:24.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.838+0000 7fb508b99700 1 -- 172.21.15.27:0/75224784 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb500011630 con 0x7fb50406a040 2024-01-11T08:26:24.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.839+0000 7fb508b99700 1 -- 172.21.15.27:0/75224784 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5000117b0 con 0x7fb50406a040 2024-01-11T08:26:24.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.839+0000 7fb508b99700 1 -- 172.21.15.27:0/75224784 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb500011b70 con 0x7fb50406a040 2024-01-11T08:26:24.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.839+0000 7fb50bdff700 1 -- 172.21.15.27:0/75224784 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb50406a040 msgr2=0x7fb50415a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:24.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.839+0000 7fb50bdff700 1 --2- 172.21.15.27:0/75224784 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb50406a040 0x7fb50415a650 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb50406f6a0 tx=0x7fb5000091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.839+0000 7fb50bdff700 1 -- 172.21.15.27:0/75224784 shutdown_connections 2024-01-11T08:26:24.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.839+0000 7fb50bdff700 1 --2- 172.21.15.27:0/75224784 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb50406a040 0x7fb50415a650 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.839+0000 7fb50bdff700 1 --2- 172.21.15.27:0/75224784 >> v2:172.21.15.77:3300/0 conn(0x7fb5040cc450 0x7fb504153600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.839+0000 7fb50bdff700 1 --2- 172.21.15.27:0/75224784 >> v2:172.21.15.27:3300/0 conn(0x7fb50406a410 0x7fb5040cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.839+0000 7fb50bdff700 1 -- 172.21.15.27:0/75224784 >> 172.21.15.27:0/75224784 conn(0x7fb5040ce340 msgr2=0x7fb5040ce740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:24.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.839+0000 7fb50bdff700 1 -- 172.21.15.27:0/75224784 shutdown_connections 2024-01-11T08:26:24.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.839+0000 7fb50bdff700 1 -- 172.21.15.27:0/75224784 wait complete. 2024-01-11T08:26:24.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.841+0000 7fb50bdff700 1 Processor -- start 2024-01-11T08:26:24.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.841+0000 7fb50bdff700 1 -- start start 2024-01-11T08:26:24.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.842+0000 7fb50bdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb50406a040 0x7fb504169850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:24.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.842+0000 7fb50bdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb50406a410 0x7fb50413b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:24.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.842+0000 7fb50bdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5040cc450 0x7fb50413d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:24.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.842+0000 7fb50bdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5040afac0 con 0x7fb5040cc450 2024-01-11T08:26:24.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.842+0000 7fb50bdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb504169d90 con 0x7fb50406a410 2024-01-11T08:26:24.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.842+0000 7fb50bdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb50416a040 con 0x7fb50406a040 2024-01-11T08:26:24.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.842+0000 7fb50a39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5040cc450 0x7fb50413d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:24.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.842+0000 7fb50a39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5040cc450 0x7fb50413d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53978/0 (socket says 172.21.15.27:53978) 2024-01-11T08:26:24.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.842+0000 7fb509b9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb50406a040 0x7fb504169850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:24.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.842+0000 7fb50a39c700 1 -- 172.21.15.27:0/2997385648 learned_addr learned my addr 172.21.15.27:0/2997385648 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:24.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.842+0000 7fb50939a700 1 --2- 172.21.15.27:0/2997385648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb50406a410 0x7fb50413b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:24.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.843+0000 7fb50a39c700 1 -- 172.21.15.27:0/2997385648 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb50406a040 msgr2=0x7fb504169850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:24.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.843+0000 7fb50a39c700 1 --2- 172.21.15.27:0/2997385648 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb50406a040 0x7fb504169850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.843+0000 7fb50a39c700 1 -- 172.21.15.27:0/2997385648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb50406a410 msgr2=0x7fb50413b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:24.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.843+0000 7fb50a39c700 1 --2- 172.21.15.27:0/2997385648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb50406a410 0x7fb50413b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:24.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.843+0000 7fb50a39c700 1 -- 172.21.15.27:0/2997385648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f40094f0 con 0x7fb5040cc450 2024-01-11T08:26:24.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.843+0000 7fb509b9b700 1 --2- 172.21.15.27:0/2997385648 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb50406a040 0x7fb504169850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:24.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.843+0000 7fb50a39c700 1 --2- 172.21.15.27:0/2997385648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5040cc450 0x7fb50413d130 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc009f20 tx=0x7fb4fc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:24.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.843+0000 7fb4f9ffb700 1 -- 172.21.15.27:0/2997385648 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc00cb70 con 0x7fb5040cc450 2024-01-11T08:26:24.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.843+0000 7fb4f9ffb700 1 -- 172.21.15.27:0/2997385648 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4fc00f040 con 0x7fb5040cc450 2024-01-11T08:26:24.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.843+0000 7fb50bdff700 1 -- 172.21.15.27:0/2997385648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4fc009a20 con 0x7fb5040cc450 2024-01-11T08:26:24.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.843+0000 7fb4f9ffb700 1 -- 172.21.15.27:0/2997385648 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc014480 con 0x7fb5040cc450 2024-01-11T08:26:24.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.843+0000 7fb50bdff700 1 -- 172.21.15.27:0/2997385648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb500009420 con 0x7fb5040cc450 2024-01-11T08:26:24.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.845+0000 7fb50bdff700 1 -- 172.21.15.27:0/2997385648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb50416a720 con 0x7fb5040cc450 2024-01-11T08:26:24.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.845+0000 7fb4f9ffb700 1 -- 172.21.15.27:0/2997385648 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 42) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fb4fc003020 con 0x7fb5040cc450 2024-01-11T08:26:24.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.847+0000 7fb4f9ffb700 1 --2- 172.21.15.27:0/2997385648 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fb4e8067050 0x7fb4e8069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:24.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.847+0000 7fb4f9ffb700 1 -- 172.21.15.27:0/2997385648 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6077+0+0 (secure 0 0 0) 0x7fb4fc085230 con 0x7fb5040cc450 2024-01-11T08:26:24.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.847+0000 7fb509b9b700 -1 --2- 172.21.15.27:0/2997385648 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fb4e8067050 0x7fb4e8069500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] is using msgr V1 protocol 2024-01-11T08:26:24.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.847+0000 7fb509b9b700 1 --2- 172.21.15.27:0/2997385648 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fb4e8067050 0x7fb4e8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:26:24.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:24.852+0000 7fb4f9ffb700 1 -- 172.21.15.27:0/2997385648 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb4fc052110 con 0x7fb5040cc450 2024-01-11T08:26:24.995 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:24.996+0000 7fb6b233c1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:26:25.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.048+0000 7fb509b9b700 -1 --2- 172.21.15.27:0/2997385648 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fb4e8067050 0x7fb4e8069500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] is using msgr V1 protocol 2024-01-11T08:26:25.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.048+0000 7fb509b9b700 1 --2- 172.21.15.27:0/2997385648 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fb4e8067050 0x7fb4e8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:26:25.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.104+0000 7fb50bdff700 1 -- 172.21.15.27:0/2997385648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb50404c3f0 con 0x7fb5040cc450 2024-01-11T08:26:25.106 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:25.107+0000 7fb6b233c1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:26:25.109 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:25.109+0000 7fb6a83fc700 -1 --2- 172.21.15.77:0/41381 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x55eff13d0000 0x55eff13d6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] is using msgr V1 protocol 2024-01-11T08:26:25.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.117+0000 7fb4f9ffb700 1 -- 172.21.15.27:0/2997385648 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 43) v1 ==== 84453+0+0 (secure 0 0 0) 0x7fb4fc0835d0 con 0x7fb5040cc450 2024-01-11T08:26:25.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.117+0000 7fb4f9ffb700 1 -- 172.21.15.27:0/2997385648 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fb4e8067050 msgr2=0x7fb4e8069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:26:25.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.117+0000 7fb4f9ffb700 1 --2- 172.21.15.27:0/2997385648 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fb4e8067050 0x7fb4e8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:25.148 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:25.149+0000 7f267415b1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:26:25.188 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:25.189+0000 7fb63018a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:25.189 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:25.189+0000 7fb63018a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:25.189 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:25.189+0000 7fb63018a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:25.189 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:25.189+0000 7fb63018a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:25.189 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:25.189+0000 7fb63018a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:25.189 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:25.190+0000 7fb62d184700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:25.190 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:25.190+0000 7fb62d184700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:25.190 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:25.190+0000 7fb62d184700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:25.190 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:25.190+0000 7fb62d184700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:25.190 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:25.190+0000 7fb62d184700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:25.342 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:25.343+0000 7f267415b1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:26:25.539 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:25.540+0000 7f267415b1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:26:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.597+0000 7fb4f9ffb700 1 -- 172.21.15.27:0/2997385648 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb4fc07efd0 con 0x7fb5040cc450 2024-01-11T08:26:25.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.600+0000 7fb50bdff700 1 -- 172.21.15.27:0/2997385648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5040cc450 msgr2=0x7fb50413d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:25.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.600+0000 7fb50bdff700 1 --2- 172.21.15.27:0/2997385648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5040cc450 0x7fb50413d130 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc009f20 tx=0x7fb4fc00b040 comp rx=0 tx=0).stop 2024-01-11T08:26:25.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.600+0000 7fb50bdff700 1 -- 172.21.15.27:0/2997385648 shutdown_connections 2024-01-11T08:26:25.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.600+0000 7fb50bdff700 1 --2- 172.21.15.27:0/2997385648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5040cc450 0x7fb50413d130 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:25.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.600+0000 7fb50bdff700 1 --2- 172.21.15.27:0/2997385648 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x7fb4e8067050 0x7fb4e8069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:25.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.600+0000 7fb50bdff700 1 --2- 172.21.15.27:0/2997385648 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb50406a040 0x7fb504169850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:25.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.600+0000 7fb50bdff700 1 --2- 172.21.15.27:0/2997385648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb50406a410 0x7fb50413b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:25.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.600+0000 7fb50bdff700 1 -- 172.21.15.27:0/2997385648 >> 172.21.15.27:0/2997385648 conn(0x7fb5040ce340 msgr2=0x7fb50406bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:25.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.600+0000 7fb50bdff700 1 -- 172.21.15.27:0/2997385648 shutdown_connections 2024-01-11T08:26:25.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.600+0000 7fb50bdff700 1 -- 172.21.15.27:0/2997385648 wait complete. 2024-01-11T08:26:25.617 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-11T08:26:25.659 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:25.661+0000 7f267415b1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:26:25.663 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:25.664+0000 7f266a1fc700 -1 --2- 172.21.15.27:0/41514 >> [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] conn(0x5626f91d8000 0x5626f91de000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:172.21.15.77:6834/41381,v1:172.21.15.77:6836/41381] is using msgr V1 protocol 2024-01-11T08:26:25.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.823+0000 7f17e07ff700 1 Processor -- start 2024-01-11T08:26:25.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.823+0000 7f17e07ff700 1 -- start start 2024-01-11T08:26:25.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.825+0000 7f17e07ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f17d80c71a0 0x7f17d806abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:25.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.825+0000 7f17e07ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17d80c7b90 0x7f17d80c7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:25.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.825+0000 7f17e07ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d815e670 con 0x7f17d80c9bc0 2024-01-11T08:26:25.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.825+0000 7f17e07ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d8069e30 con 0x7f17d80c71a0 2024-01-11T08:26:25.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.825+0000 7f17e07ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d806a110 con 0x7f17d80c7b90 2024-01-11T08:26:25.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.825+0000 7f17de59b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f17d80c71a0 0x7f17d806abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:25.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.825+0000 7f17de59b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f17d80c71a0 0x7f17d806abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53990/0 (socket says 172.21.15.27:53990) 2024-01-11T08:26:25.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.825+0000 7f17de59b700 1 -- 172.21.15.27:0/342179546 learned_addr learned my addr 172.21.15.27:0/342179546 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:25.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.825+0000 7f17ded9c700 1 --1- 172.21.15.27:0/342179546 >> v1:172.21.15.27:6789/0 conn(0x7f17d80c9bc0 0x7f17d80af810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56110/0 (socket says 172.21.15.27:56110) 2024-01-11T08:26:25.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.826+0000 7f17ddd9a700 1 --2- 172.21.15.27:0/342179546 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17d80c7b90 0x7f17d80c7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:25.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.826+0000 7f17de59b700 1 -- 172.21.15.27:0/342179546 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17d80c7b90 msgr2=0x7f17d80c7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:25.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.826+0000 7f17de59b700 1 --2- 172.21.15.27:0/342179546 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17d80c7b90 0x7f17d80c7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:25.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.826+0000 7f17de59b700 1 -- 172.21.15.27:0/342179546 >> v1:172.21.15.27:6789/0 conn(0x7f17d80c9bc0 legacy=0x7f17d80af810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:25.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.826+0000 7f17de59b700 1 -- 172.21.15.27:0/342179546 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d806a3f0 con 0x7f17d80c71a0 2024-01-11T08:26:25.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.826+0000 7f17de59b700 1 --2- 172.21.15.27:0/342179546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17d80c71a0 0x7f17d806abf0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f17d40099b0 tx=0x7f17d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca56774bbc0d7e5c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:25.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.826+0000 7f17dd599700 1 -- 172.21.15.27:0/342179546 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d4009ea0 con 0x7f17d80c71a0 2024-01-11T08:26:25.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.826+0000 7f17dd599700 1 -- 172.21.15.27:0/342179546 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d400b860 con 0x7f17d80c71a0 2024-01-11T08:26:25.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.826+0000 7f17dd599700 1 -- 172.21.15.27:0/342179546 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d400bc00 con 0x7f17d80c71a0 2024-01-11T08:26:25.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.827+0000 7f17e07ff700 1 -- 172.21.15.27:0/342179546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17d80c71a0 msgr2=0x7f17d806abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:25.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.827+0000 7f17e07ff700 1 --2- 172.21.15.27:0/342179546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17d80c71a0 0x7f17d806abf0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f17d40099b0 tx=0x7f17d4009250 comp rx=0 tx=0).stop 2024-01-11T08:26:25.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.827+0000 7f17e07ff700 1 -- 172.21.15.27:0/342179546 shutdown_connections 2024-01-11T08:26:25.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.827+0000 7f17e07ff700 1 --2- 172.21.15.27:0/342179546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17d80c71a0 0x7f17d806abf0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:25.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.827+0000 7f17e07ff700 1 --2- 172.21.15.27:0/342179546 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17d80c7b90 0x7f17d80c7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:25.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.827+0000 7f17e07ff700 1 -- 172.21.15.27:0/342179546 >> 172.21.15.27:0/342179546 conn(0x7f17d80cd4d0 msgr2=0x7f17d80c94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:25.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.827+0000 7f17e07ff700 1 -- 172.21.15.27:0/342179546 shutdown_connections 2024-01-11T08:26:25.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.827+0000 7f17e07ff700 1 -- 172.21.15.27:0/342179546 wait complete. 2024-01-11T08:26:25.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.829+0000 7f17e07ff700 1 Processor -- start 2024-01-11T08:26:25.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.829+0000 7f17e07ff700 1 -- start start 2024-01-11T08:26:25.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.830+0000 7f17e07ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17d80c71a0 0x7f17d80c2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:25.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.830+0000 7f17e07ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17d80c7b90 0x7f17d80c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:25.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.830+0000 7f17e07ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17d80c9bc0 0x7f17d815c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:25.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.830+0000 7f17e07ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d80c5980 con 0x7f17d80c9bc0 2024-01-11T08:26:25.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.830+0000 7f17e07ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d80c4ad0 con 0x7f17d80c71a0 2024-01-11T08:26:25.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.830+0000 7f17e07ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d80c3380 con 0x7f17d80c7b90 2024-01-11T08:26:25.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.830+0000 7f17ded9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17d80c9bc0 0x7f17d815c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:25.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.830+0000 7f17ded9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17d80c9bc0 0x7f17d815c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54006/0 (socket says 172.21.15.27:54006) 2024-01-11T08:26:25.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.830+0000 7f17ded9c700 1 -- 172.21.15.27:0/1176449476 learned_addr learned my addr 172.21.15.27:0/1176449476 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:25.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.830+0000 7f17ddd9a700 1 --2- 172.21.15.27:0/1176449476 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17d80c7b90 0x7f17d80c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:25.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.830+0000 7f17de59b700 1 --2- 172.21.15.27:0/1176449476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17d80c71a0 0x7f17d80c2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:25.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.830+0000 7f17ded9c700 1 -- 172.21.15.27:0/1176449476 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17d80c7b90 msgr2=0x7f17d80c5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:25.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.830+0000 7f17ded9c700 1 --2- 172.21.15.27:0/1176449476 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17d80c7b90 0x7f17d80c5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:25.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.830+0000 7f17ded9c700 1 -- 172.21.15.27:0/1176449476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17d80c71a0 msgr2=0x7f17d80c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:25.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.830+0000 7f17ded9c700 1 --2- 172.21.15.27:0/1176449476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17d80c71a0 0x7f17d80c2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:25.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.830+0000 7f17ded9c700 1 -- 172.21.15.27:0/1176449476 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d40094f0 con 0x7f17d80c9bc0 2024-01-11T08:26:25.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.830+0000 7f17ddd9a700 1 --2- 172.21.15.27:0/1176449476 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17d80c7b90 0x7f17d80c5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:25.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.830+0000 7f17de59b700 1 --2- 172.21.15.27:0/1176449476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17d80c71a0 0x7f17d80c2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:25.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.831+0000 7f17ded9c700 1 --2- 172.21.15.27:0/1176449476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17d80c9bc0 0x7f17d815c610 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f17d000cc70 tx=0x7f17d0001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:25.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.831+0000 7f17ce7fc700 1 -- 172.21.15.27:0/1176449476 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0004bb0 con 0x7f17d80c9bc0 2024-01-11T08:26:25.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.831+0000 7f17ce7fc700 1 -- 172.21.15.27:0/1176449476 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d0004d30 con 0x7f17d80c9bc0 2024-01-11T08:26:25.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.831+0000 7f17e07ff700 1 -- 172.21.15.27:0/1176449476 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d80c4d70 con 0x7f17d80c9bc0 2024-01-11T08:26:25.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.831+0000 7f17ce7fc700 1 -- 172.21.15.27:0/1176449476 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0004ea0 con 0x7f17d80c9bc0 2024-01-11T08:26:25.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.831+0000 7f17e07ff700 1 -- 172.21.15.27:0/1176449476 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d80c5330 con 0x7f17d80c9bc0 2024-01-11T08:26:25.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.832+0000 7f17e07ff700 1 -- 172.21.15.27:0/1176449476 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d804c3c0 con 0x7f17d80c9bc0 2024-01-11T08:26:25.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.833+0000 7f17ce7fc700 1 -- 172.21.15.27:0/1176449476 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f17d00070f0 con 0x7f17d80c9bc0 2024-01-11T08:26:25.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.834+0000 7f17ce7fc700 1 -- 172.21.15.27:0/1176449476 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6335+0+0 (secure 0 0 0) 0x7f17d0083e00 con 0x7f17d80c9bc0 2024-01-11T08:26:25.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:25.838+0000 7f17ce7fc700 1 -- 172.21.15.27:0/1176449476 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f17d0051150 con 0x7f17d80c9bc0 2024-01-11T08:26:26.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:26.044+0000 7f17e07ff700 1 -- 172.21.15.27:0/1176449476 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f17d80637a0 con 0x7f17d80c9bc0 2024-01-11T08:26:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:26.120+0000 7f17ce7fc700 1 -- 172.21.15.27:0/1176449476 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 44) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f17d0050c00 con 0x7f17d80c9bc0 2024-01-11T08:26:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:26.120+0000 7f17ce7fc700 1 --2- 172.21.15.27:0/1176449476 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f17bc06c6d0 0x7f17bc06eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:26.121+0000 7f17de59b700 1 --2- 172.21.15.27:0/1176449476 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f17bc06c6d0 0x7f17bc06eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:26.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:26.122+0000 7f17de59b700 1 --2- 172.21.15.27:0/1176449476 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f17bc06c6d0 0x7f17bc06eab0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f17d40099b0 tx=0x7f17d4017000 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:26.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:26.936+0000 7f17ce7fc700 1 -- 172.21.15.27:0/1176449476 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v92) v1 ==== 181+0+0 (secure 0 0 0) 0x7f17d0080790 con 0x7f17d80c9bc0 2024-01-11T08:26:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:26.940+0000 7f17e07ff700 1 -- 172.21.15.27:0/1176449476 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f17bc06c6d0 msgr2=0x7f17bc06eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:26.940+0000 7f17e07ff700 1 --2- 172.21.15.27:0/1176449476 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f17bc06c6d0 0x7f17bc06eab0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f17d40099b0 tx=0x7f17d4017000 comp rx=0 tx=0).stop 2024-01-11T08:26:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:26.940+0000 7f17e07ff700 1 -- 172.21.15.27:0/1176449476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17d80c9bc0 msgr2=0x7f17d815c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:26.940+0000 7f17e07ff700 1 --2- 172.21.15.27:0/1176449476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17d80c9bc0 0x7f17d815c610 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f17d000cc70 tx=0x7f17d0001a70 comp rx=0 tx=0).stop 2024-01-11T08:26:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:26.941+0000 7f17e07ff700 1 -- 172.21.15.27:0/1176449476 shutdown_connections 2024-01-11T08:26:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:26.941+0000 7f17e07ff700 1 --2- 172.21.15.27:0/1176449476 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f17bc06c6d0 0x7f17bc06eab0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:27.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:26.941+0000 7f17e07ff700 1 --2- 172.21.15.27:0/1176449476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17d80c9bc0 0x7f17d815c610 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:27.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:26.941+0000 7f17e07ff700 1 --2- 172.21.15.27:0/1176449476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17d80c71a0 0x7f17d80c2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:27.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:26.941+0000 7f17e07ff700 1 --2- 172.21.15.27:0/1176449476 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17d80c7b90 0x7f17d80c5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:27.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:26.941+0000 7f17e07ff700 1 -- 172.21.15.27:0/1176449476 >> 172.21.15.27:0/1176449476 conn(0x7f17d80cd4d0 msgr2=0x7f17d806dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:27.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:26.942+0000 7f17e07ff700 1 -- 172.21.15.27:0/1176449476 shutdown_connections 2024-01-11T08:26:27.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:26.942+0000 7f17e07ff700 1 -- 172.21.15.27:0/1176449476 wait complete. 2024-01-11T08:26:27.014 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-01-11T08:26:27.014 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-11T08:26:27.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.270+0000 7f590fbff700 1 Processor -- start 2024-01-11T08:26:27.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.270+0000 7f590fbff700 1 -- start start 2024-01-11T08:26:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.271+0000 7f590fbff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5908157440 0x7f59080ae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.271+0000 7f590fbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5908157810 0x7f5908069660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.271+0000 7f590fbff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f590806d420 con 0x7f590806a570 2024-01-11T08:26:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.271+0000 7f590fbff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f590806d700 con 0x7f5908157440 2024-01-11T08:26:27.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.271+0000 7f590fbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f590806d9e0 con 0x7f5908157810 2024-01-11T08:26:27.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.272+0000 7f590d99b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5908157440 0x7f59080ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:27.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.272+0000 7f590d99b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5908157440 0x7f59080ae430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54008/0 (socket says 172.21.15.27:54008) 2024-01-11T08:26:27.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.272+0000 7f590d99b700 1 -- 172.21.15.27:0/2417921201 learned_addr learned my addr 172.21.15.27:0/2417921201 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:27.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.272+0000 7f590d19a700 1 --2- 172.21.15.27:0/2417921201 >> v2:172.21.15.77:3300/0 conn(0x7f5908157810 0x7f5908069660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:27.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.272+0000 7f590d99b700 1 -- 172.21.15.27:0/2417921201 >> v2:172.21.15.77:3300/0 conn(0x7f5908157810 msgr2=0x7f5908069660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:27.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.272+0000 7f590d99b700 1 --2- 172.21.15.27:0/2417921201 >> v2:172.21.15.77:3300/0 conn(0x7f5908157810 0x7f5908069660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:27.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.272+0000 7f590d99b700 1 -- 172.21.15.27:0/2417921201 >> v1:172.21.15.27:6789/0 conn(0x7f590806a570 legacy=0x7f590806e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:27.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.272+0000 7f590d99b700 1 -- 172.21.15.27:0/2417921201 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f590806dcc0 con 0x7f5908157440 2024-01-11T08:26:27.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.272+0000 7f590c999700 1 -- 172.21.15.27:0/2417921201 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790753883 0 0) 0x7f590806d420 con 0x7f590806a570 2024-01-11T08:26:27.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.273+0000 7f590d99b700 1 --2- 172.21.15.27:0/2417921201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5908157440 0x7f59080ae430 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f590815b090 tx=0x7f5904009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca4780ef0fdd426 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:27.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.273+0000 7f590c999700 1 -- 172.21.15.27:0/2417921201 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5904005c10 con 0x7f5908157440 2024-01-11T08:26:27.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.273+0000 7f590c999700 1 -- 172.21.15.27:0/2417921201 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5904005d70 con 0x7f5908157440 2024-01-11T08:26:27.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.273+0000 7f590c999700 1 -- 172.21.15.27:0/2417921201 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5904006260 con 0x7f5908157440 2024-01-11T08:26:27.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.273+0000 7f590fbff700 1 -- 172.21.15.27:0/2417921201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5908157440 msgr2=0x7f59080ae430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:27.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.273+0000 7f590fbff700 1 --2- 172.21.15.27:0/2417921201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5908157440 0x7f59080ae430 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f590815b090 tx=0x7f5904009220 comp rx=0 tx=0).stop 2024-01-11T08:26:27.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.273+0000 7f590fbff700 1 -- 172.21.15.27:0/2417921201 shutdown_connections 2024-01-11T08:26:27.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.273+0000 7f590fbff700 1 --2- 172.21.15.27:0/2417921201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5908157440 0x7f59080ae430 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:27.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.273+0000 7f590fbff700 1 --2- 172.21.15.27:0/2417921201 >> v2:172.21.15.77:3300/0 conn(0x7f5908157810 0x7f5908069660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:27.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.273+0000 7f590fbff700 1 -- 172.21.15.27:0/2417921201 >> 172.21.15.27:0/2417921201 conn(0x7f59080680e0 msgr2=0x7f59080cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:27.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.273+0000 7f590fbff700 1 -- 172.21.15.27:0/2417921201 shutdown_connections 2024-01-11T08:26:27.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.274+0000 7f590fbff700 1 -- 172.21.15.27:0/2417921201 wait complete. 2024-01-11T08:26:27.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.276+0000 7f590fbff700 1 Processor -- start 2024-01-11T08:26:27.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.276+0000 7f590fbff700 1 -- start start 2024-01-11T08:26:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.277+0000 7f590fbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f590806a570 0x7f5908158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.277+0000 7f590fbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5908157440 0x7f5908159050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:27.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.277+0000 7f590fbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5908157810 0x7f590815c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.277+0000 7f590fbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59080c0300 con 0x7f5908157440 2024-01-11T08:26:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.277+0000 7f590fbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59080bf6f0 con 0x7f590806a570 2024-01-11T08:26:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.277+0000 7f590fbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59080bf9d0 con 0x7f5908157810 2024-01-11T08:26:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.277+0000 7f590d19a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5908157440 0x7f5908159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.277+0000 7f590d19a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5908157440 0x7f5908159050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54024/0 (socket says 172.21.15.27:54024) 2024-01-11T08:26:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.277+0000 7f590d19a700 1 -- 172.21.15.27:0/1734551907 learned_addr learned my addr 172.21.15.27:0/1734551907 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.277+0000 7f590d99b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f590806a570 0x7f5908158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.278+0000 7f590e19c700 1 --2- 172.21.15.27:0/1734551907 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5908157810 0x7f590815c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.278+0000 7f590d19a700 1 -- 172.21.15.27:0/1734551907 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5908157810 msgr2=0x7f590815c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.278+0000 7f590d19a700 1 --2- 172.21.15.27:0/1734551907 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5908157810 0x7f590815c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.278+0000 7f590d19a700 1 -- 172.21.15.27:0/1734551907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f590806a570 msgr2=0x7f5908158b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.278+0000 7f590d19a700 1 --2- 172.21.15.27:0/1734551907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f590806a570 0x7f5908158b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.278+0000 7f590d19a700 1 -- 172.21.15.27:0/1734551907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5904009480 con 0x7f5908157440 2024-01-11T08:26:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.278+0000 7f590d99b700 1 --2- 172.21.15.27:0/1734551907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f590806a570 0x7f5908158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.278+0000 7f590d19a700 1 --2- 172.21.15.27:0/1734551907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5908157440 0x7f5908159050 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f58f800e900 tx=0x7f58f80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.278+0000 7f58fdffb700 1 -- 172.21.15.27:0/1734551907 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f8013680 con 0x7f5908157440 2024-01-11T08:26:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.278+0000 7f58fdffb700 1 -- 172.21.15.27:0/1734551907 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f80137e0 con 0x7f5908157440 2024-01-11T08:26:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.278+0000 7f590fbff700 1 -- 172.21.15.27:0/1734551907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59080bfcb0 con 0x7f5908157440 2024-01-11T08:26:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.278+0000 7f58fdffb700 1 -- 172.21.15.27:0/1734551907 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f8013ba0 con 0x7f5908157440 2024-01-11T08:26:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.279+0000 7f590fbff700 1 -- 172.21.15.27:0/1734551907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59080c2670 con 0x7f5908157440 2024-01-11T08:26:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.280+0000 7f590fbff700 1 -- 172.21.15.27:0/1734551907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59080c04b0 con 0x7f5908157440 2024-01-11T08:26:27.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.280+0000 7f58fdffb700 1 -- 172.21.15.27:0/1734551907 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 44) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f58f80123d0 con 0x7f5908157440 2024-01-11T08:26:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.281+0000 7f58fdffb700 1 --2- 172.21.15.27:0/1734551907 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f58ec066d80 0x7f58ec069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.281+0000 7f58fdffb700 1 -- 172.21.15.27:0/1734551907 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5897+0+0 (secure 0 0 0) 0x7f58f8084320 con 0x7f5908157440 2024-01-11T08:26:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.282+0000 7f590d99b700 1 --2- 172.21.15.27:0/1734551907 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f58ec066d80 0x7f58ec069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.282+0000 7f590d99b700 1 --2- 172.21.15.27:0/1734551907 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f58ec066d80 0x7f58ec069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5904009260 tx=0x7f5904016000 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.285+0000 7f58fdffb700 1 -- 172.21.15.27:0/1734551907 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f59080c04b0 con 0x7f5908157440 2024-01-11T08:26:27.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.488+0000 7f590fbff700 1 -- 172.21.15.27:0/1734551907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f590804c3c0 con 0x7f5908157440 2024-01-11T08:26:27.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:27.938+0000 7f58fdffb700 1 -- 172.21.15.27:0/1734551907 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 45) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f58f8051b50 con 0x7f5908157440 2024-01-11T08:26:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.131+0000 7f58fdffb700 1 -- 172.21.15.27:0/1734551907 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v93) v1 ==== 169+0+0 (secure 0 0 0) 0x7f58f80840d0 con 0x7f5908157440 2024-01-11T08:26:28.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.135+0000 7f590fbff700 1 -- 172.21.15.27:0/1734551907 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f58ec066d80 msgr2=0x7f58ec069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:28.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.135+0000 7f590fbff700 1 --2- 172.21.15.27:0/1734551907 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f58ec066d80 0x7f58ec069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5904009260 tx=0x7f5904016000 comp rx=0 tx=0).stop 2024-01-11T08:26:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.135+0000 7f590fbff700 1 -- 172.21.15.27:0/1734551907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5908157440 msgr2=0x7f5908159050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.135+0000 7f590fbff700 1 --2- 172.21.15.27:0/1734551907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5908157440 0x7f5908159050 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f58f800e900 tx=0x7f58f80091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.135+0000 7f590fbff700 1 -- 172.21.15.27:0/1734551907 shutdown_connections 2024-01-11T08:26:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.135+0000 7f590fbff700 1 --2- 172.21.15.27:0/1734551907 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f58ec066d80 0x7f58ec069230 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.135+0000 7f590fbff700 1 --2- 172.21.15.27:0/1734551907 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5908157810 0x7f590815c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.135+0000 7f590fbff700 1 --2- 172.21.15.27:0/1734551907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f590806a570 0x7f5908158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.135+0000 7f590fbff700 1 --2- 172.21.15.27:0/1734551907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5908157440 0x7f5908159050 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.135+0000 7f590fbff700 1 -- 172.21.15.27:0/1734551907 >> 172.21.15.27:0/1734551907 conn(0x7f59080680e0 msgr2=0x7f590806e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.135+0000 7f590fbff700 1 -- 172.21.15.27:0/1734551907 shutdown_connections 2024-01-11T08:26:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.135+0000 7f590fbff700 1 -- 172.21.15.27:0/1734551907 wait complete. 2024-01-11T08:26:28.137 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-01-11T08:26:28.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-11T08:26:28.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.370+0000 7f2a19bff700 1 Processor -- start 2024-01-11T08:26:28.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.370+0000 7f2a19bff700 1 -- start start 2024-01-11T08:26:28.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.371+0000 7f2a19bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2a14157410 0x7f2a140ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:28.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.371+0000 7f2a19bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2a14069540 0x7f2a140c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:28.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.371+0000 7f2a19bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a140699d0 con 0x7f2a141577e0 2024-01-11T08:26:28.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.371+0000 7f2a19bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a14069cb0 con 0x7f2a14157410 2024-01-11T08:26:28.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.371+0000 7f2a19bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a14069f90 con 0x7f2a14069540 2024-01-11T08:26:28.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.372+0000 7f2a137fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2a14157410 0x7f2a140ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:28.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.372+0000 7f2a137fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2a14157410 0x7f2a140ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54034/0 (socket says 172.21.15.27:54034) 2024-01-11T08:26:28.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.372+0000 7f2a137fe700 1 -- 172.21.15.27:0/867399818 learned_addr learned my addr 172.21.15.27:0/867399818 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:28.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.372+0000 7f2a13fff700 1 --2- 172.21.15.27:0/867399818 >> v2:172.21.15.77:3300/0 conn(0x7f2a14069540 0x7f2a140c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.372+0000 7f2a137fe700 1 -- 172.21.15.27:0/867399818 >> v2:172.21.15.77:3300/0 conn(0x7f2a14069540 msgr2=0x7f2a140c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.372+0000 7f2a137fe700 1 --2- 172.21.15.27:0/867399818 >> v2:172.21.15.77:3300/0 conn(0x7f2a14069540 0x7f2a140c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.372+0000 7f2a137fe700 1 -- 172.21.15.27:0/867399818 >> v1:172.21.15.77:6789/0 conn(0x7f2a141577e0 legacy=0x7f2a140ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.372+0000 7f2a137fe700 1 -- 172.21.15.27:0/867399818 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1406a270 con 0x7f2a14157410 2024-01-11T08:26:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.373+0000 7f2a137fe700 1 --2- 172.21.15.27:0/867399818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a14157410 0x7f2a140ccae0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2a1406d030 tx=0x7f2a04009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303bcebc8c16351 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.373+0000 7f2a127fc700 1 -- 172.21.15.27:0/867399818 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a04005c10 con 0x7f2a14157410 2024-01-11T08:26:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.373+0000 7f2a127fc700 1 -- 172.21.15.27:0/867399818 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a04005d70 con 0x7f2a14157410 2024-01-11T08:26:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.373+0000 7f2a127fc700 1 -- 172.21.15.27:0/867399818 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a04006260 con 0x7f2a14157410 2024-01-11T08:26:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.373+0000 7f2a19bff700 1 -- 172.21.15.27:0/867399818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a14157410 msgr2=0x7f2a140ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.373+0000 7f2a19bff700 1 --2- 172.21.15.27:0/867399818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a14157410 0x7f2a140ccae0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2a1406d030 tx=0x7f2a04009220 comp rx=0 tx=0).stop 2024-01-11T08:26:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.373+0000 7f2a19bff700 1 -- 172.21.15.27:0/867399818 shutdown_connections 2024-01-11T08:26:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.373+0000 7f2a19bff700 1 --2- 172.21.15.27:0/867399818 >> v2:172.21.15.77:3300/0 conn(0x7f2a14069540 0x7f2a140c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.373+0000 7f2a19bff700 1 --2- 172.21.15.27:0/867399818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a14157410 0x7f2a140ccae0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.373+0000 7f2a19bff700 1 -- 172.21.15.27:0/867399818 >> 172.21.15.27:0/867399818 conn(0x7f2a140680a0 msgr2=0x7f2a140cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.373+0000 7f2a19bff700 1 -- 172.21.15.27:0/867399818 shutdown_connections 2024-01-11T08:26:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.374+0000 7f2a19bff700 1 -- 172.21.15.27:0/867399818 wait complete. 2024-01-11T08:26:28.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.375+0000 7f2a19bff700 1 Processor -- start 2024-01-11T08:26:28.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.375+0000 7f2a19bff700 1 -- start start 2024-01-11T08:26:28.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.376+0000 7f2a19bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a14069540 0x7f2a1414dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.376+0000 7f2a19bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a14157410 0x7f2a140c50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.376+0000 7f2a19bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a141577e0 0x7f2a140c1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.376+0000 7f2a19bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a140c3ee0 con 0x7f2a141577e0 2024-01-11T08:26:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.376+0000 7f2a19bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a140c3310 con 0x7f2a14069540 2024-01-11T08:26:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.376+0000 7f2a19bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a140c35b0 con 0x7f2a14157410 2024-01-11T08:26:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.376+0000 7f2a13fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a141577e0 0x7f2a140c1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.376+0000 7f2a137fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a14069540 0x7f2a1414dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.376+0000 7f2a13fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a141577e0 0x7f2a140c1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54038/0 (socket says 172.21.15.27:54038) 2024-01-11T08:26:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.376+0000 7f2a13fff700 1 -- 172.21.15.27:0/2810920912 learned_addr learned my addr 172.21.15.27:0/2810920912 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.377+0000 7f2a12ffd700 1 --2- 172.21.15.27:0/2810920912 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a14157410 0x7f2a140c50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.377+0000 7f2a13fff700 1 -- 172.21.15.27:0/2810920912 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a14157410 msgr2=0x7f2a140c50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.377+0000 7f2a13fff700 1 --2- 172.21.15.27:0/2810920912 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a14157410 0x7f2a140c50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.377+0000 7f2a13fff700 1 -- 172.21.15.27:0/2810920912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a14069540 msgr2=0x7f2a1414dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.377+0000 7f2a13fff700 1 --2- 172.21.15.27:0/2810920912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a14069540 0x7f2a1414dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.377+0000 7f2a13fff700 1 -- 172.21.15.27:0/2810920912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a04009480 con 0x7f2a141577e0 2024-01-11T08:26:28.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.377+0000 7f2a13fff700 1 --2- 172.21.15.27:0/2810920912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a141577e0 0x7f2a140c1a30 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c009df0 tx=0x7f2a0c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:28.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.377+0000 7f29fb7fe700 1 -- 172.21.15.27:0/2810920912 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c00caa0 con 0x7f2a141577e0 2024-01-11T08:26:28.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.377+0000 7f29fb7fe700 1 -- 172.21.15.27:0/2810920912 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0c00f040 con 0x7f2a141577e0 2024-01-11T08:26:28.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.377+0000 7f2a19bff700 1 -- 172.21.15.27:0/2810920912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a140c3890 con 0x7f2a141577e0 2024-01-11T08:26:28.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.377+0000 7f29fb7fe700 1 -- 172.21.15.27:0/2810920912 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c00ce70 con 0x7f2a141577e0 2024-01-11T08:26:28.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.378+0000 7f2a19bff700 1 -- 172.21.15.27:0/2810920912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a140c5630 con 0x7f2a141577e0 2024-01-11T08:26:28.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.379+0000 7f2a19bff700 1 -- 172.21.15.27:0/2810920912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a140c58b0 con 0x7f2a141577e0 2024-01-11T08:26:28.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.379+0000 7f29fb7fe700 1 -- 172.21.15.27:0/2810920912 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f2a0c014690 con 0x7f2a141577e0 2024-01-11T08:26:28.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.380+0000 7f29fb7fe700 1 --2- 172.21.15.27:0/2810920912 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2a00066ea0 0x7f2a00069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:28.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.380+0000 7f29fb7fe700 1 -- 172.21.15.27:0/2810920912 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5483+0+0 (secure 0 0 0) 0x7f2a0c084ef0 con 0x7f2a141577e0 2024-01-11T08:26:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.381+0000 7f2a137fe700 1 --2- 172.21.15.27:0/2810920912 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2a00066ea0 0x7f2a00069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.381+0000 7f2a137fe700 1 --2- 172.21.15.27:0/2810920912 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2a00066ea0 0x7f2a00069350 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2a140b20e0 tx=0x7f2a04016000 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:28.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.385+0000 7f29fb7fe700 1 -- 172.21.15.27:0/2810920912 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2a0c052280 con 0x7f2a141577e0 2024-01-11T08:26:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.589+0000 7f2a19bff700 1 -- 172.21.15.27:0/2810920912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f2a1404c3c0 con 0x7f2a141577e0 2024-01-11T08:26:28.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.594+0000 7f29fb7fe700 1 -- 172.21.15.27:0/2810920912 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v31) v1 ==== 120+0+0 (secure 0 0 0) 0x7f2a0c0558a0 con 0x7f2a141577e0 2024-01-11T08:26:28.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.596+0000 7f2a19bff700 1 -- 172.21.15.27:0/2810920912 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2a00066ea0 msgr2=0x7f2a00069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:28.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.596+0000 7f2a19bff700 1 --2- 172.21.15.27:0/2810920912 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2a00066ea0 0x7f2a00069350 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2a140b20e0 tx=0x7f2a04016000 comp rx=0 tx=0).stop 2024-01-11T08:26:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.596+0000 7f2a19bff700 1 -- 172.21.15.27:0/2810920912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a141577e0 msgr2=0x7f2a140c1a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.596+0000 7f2a19bff700 1 --2- 172.21.15.27:0/2810920912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a141577e0 0x7f2a140c1a30 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c009df0 tx=0x7f2a0c00b040 comp rx=0 tx=0).stop 2024-01-11T08:26:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.596+0000 7f2a19bff700 1 -- 172.21.15.27:0/2810920912 shutdown_connections 2024-01-11T08:26:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.596+0000 7f2a19bff700 1 --2- 172.21.15.27:0/2810920912 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2a00066ea0 0x7f2a00069350 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.596+0000 7f2a19bff700 1 --2- 172.21.15.27:0/2810920912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a141577e0 0x7f2a140c1a30 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.596+0000 7f2a19bff700 1 --2- 172.21.15.27:0/2810920912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a14069540 0x7f2a1414dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.596+0000 7f2a19bff700 1 --2- 172.21.15.27:0/2810920912 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a14157410 0x7f2a140c50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.596+0000 7f2a19bff700 1 -- 172.21.15.27:0/2810920912 >> 172.21.15.27:0/2810920912 conn(0x7f2a140680a0 msgr2=0x7f2a1415f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.596+0000 7f2a19bff700 1 -- 172.21.15.27:0/2810920912 shutdown_connections 2024-01-11T08:26:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.596+0000 7f2a19bff700 1 -- 172.21.15.27:0/2810920912 wait complete. 2024-01-11T08:26:28.613 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-11T08:26:28.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.815+0000 7f80ca1ff700 1 Processor -- start 2024-01-11T08:26:28.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.815+0000 7f80ca1ff700 1 -- start start 2024-01-11T08:26:28.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.816+0000 7f80ca1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f80c4142dc0 0x7f80c40cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:28.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.817+0000 7f80ca1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f80c40cb2e0 0x7f80c40cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:28.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.817+0000 7f80ca1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80c40ccc70 con 0x7f80c4143190 2024-01-11T08:26:28.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.817+0000 7f80ca1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80c415df30 con 0x7f80c40cb2e0 2024-01-11T08:26:28.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.817+0000 7f80ca1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80c415d320 con 0x7f80c4142dc0 2024-01-11T08:26:28.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.817+0000 7f80c2ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f80c4143190 0x7f80c40cc020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56128/0 (socket says 172.21.15.27:56128) 2024-01-11T08:26:28.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.817+0000 7f80c2ffd700 1 -- 172.21.15.27:0/2687969216 learned_addr learned my addr 172.21.15.27:0/2687969216 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:28.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.817+0000 7f80c3fff700 1 --2- 172.21.15.27:0/2687969216 >> v2:172.21.15.27:3300/0 conn(0x7f80c40cb2e0 0x7f80c40cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:28.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.817+0000 7f80c37fe700 1 --2- 172.21.15.27:0/2687969216 >> v2:172.21.15.77:3300/0 conn(0x7f80c4142dc0 0x7f80c40cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:28.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.817+0000 7f80c27fc700 1 -- 172.21.15.27:0/2687969216 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194602088 0 0) 0x7f80c40ccc70 con 0x7f80c4143190 2024-01-11T08:26:28.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.817+0000 7f80c27fc700 1 -- 172.21.15.27:0/2687969216 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80ac003570 con 0x7f80c4143190 2024-01-11T08:26:28.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.817+0000 7f80c3fff700 1 -- 172.21.15.27:0/2687969216 >> v2:172.21.15.77:3300/0 conn(0x7f80c4142dc0 msgr2=0x7f80c40cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:28.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.817+0000 7f80c3fff700 1 --2- 172.21.15.27:0/2687969216 >> v2:172.21.15.77:3300/0 conn(0x7f80c4142dc0 0x7f80c40cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:28.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.817+0000 7f80c3fff700 1 -- 172.21.15.27:0/2687969216 >> v1:172.21.15.27:6789/0 conn(0x7f80c4143190 legacy=0x7f80c40cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:28.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.817+0000 7f80c3fff700 1 -- 172.21.15.27:0/2687969216 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c40cb700 con 0x7f80c40cb2e0 2024-01-11T08:26:28.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.818+0000 7f80c3fff700 1 --2- 172.21.15.27:0/2687969216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c40cb2e0 0x7f80c40cc730 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f80c406f4d0 tx=0x7f80bc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caac4c4ec0dd9a92 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:28.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.818+0000 7f80c27fc700 1 -- 172.21.15.27:0/2687969216 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80bc00e070 con 0x7f80c40cb2e0 2024-01-11T08:26:28.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.818+0000 7f80c27fc700 1 -- 172.21.15.27:0/2687969216 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80bc00bab0 con 0x7f80c40cb2e0 2024-01-11T08:26:28.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.818+0000 7f80c27fc700 1 -- 172.21.15.27:0/2687969216 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80bc00be30 con 0x7f80c40cb2e0 2024-01-11T08:26:28.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.818+0000 7f80ca1ff700 1 -- 172.21.15.27:0/2687969216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c40cb2e0 msgr2=0x7f80c40cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:28.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.818+0000 7f80ca1ff700 1 --2- 172.21.15.27:0/2687969216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c40cb2e0 0x7f80c40cc730 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f80c406f4d0 tx=0x7f80bc009780 comp rx=0 tx=0).stop 2024-01-11T08:26:28.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.819+0000 7f80ca1ff700 1 -- 172.21.15.27:0/2687969216 shutdown_connections 2024-01-11T08:26:28.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.819+0000 7f80ca1ff700 1 --2- 172.21.15.27:0/2687969216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c40cb2e0 0x7f80c40cc730 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:28.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.819+0000 7f80ca1ff700 1 --2- 172.21.15.27:0/2687969216 >> v2:172.21.15.77:3300/0 conn(0x7f80c4142dc0 0x7f80c40cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:28.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.819+0000 7f80ca1ff700 1 -- 172.21.15.27:0/2687969216 >> 172.21.15.27:0/2687969216 conn(0x7f80c4063ca0 msgr2=0x7f80c406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:28.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.819+0000 7f80ca1ff700 1 -- 172.21.15.27:0/2687969216 shutdown_connections 2024-01-11T08:26:28.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.819+0000 7f80ca1ff700 1 -- 172.21.15.27:0/2687969216 wait complete. 2024-01-11T08:26:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.820+0000 7f80ca1ff700 1 Processor -- start 2024-01-11T08:26:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.820+0000 7f80ca1ff700 1 -- start start 2024-01-11T08:26:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.821+0000 7f80ca1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c40cb2e0 0x7f80c416d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.821+0000 7f80ca1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f80c4142dc0 0x7f80c4140da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.821+0000 7f80ca1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c4143190 0x7f80c40c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.821+0000 7f80ca1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c40c6430 con 0x7f80c4143190 2024-01-11T08:26:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.821+0000 7f80ca1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c40c68c0 con 0x7f80c40cb2e0 2024-01-11T08:26:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.821+0000 7f80ca1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c40c5b00 con 0x7f80c4142dc0 2024-01-11T08:26:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.821+0000 7f80c3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c4143190 0x7f80c40c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.822+0000 7f80c37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c40cb2e0 0x7f80c416d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.822+0000 7f80c3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c4143190 0x7f80c40c9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54058/0 (socket says 172.21.15.27:54058) 2024-01-11T08:26:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.822+0000 7f80c3fff700 1 -- 172.21.15.27:0/369219081 learned_addr learned my addr 172.21.15.27:0/369219081 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.822+0000 7f80c2ffd700 1 --2- 172.21.15.27:0/369219081 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f80c4142dc0 0x7f80c4140da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.822+0000 7f80c3fff700 1 -- 172.21.15.27:0/369219081 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f80c4142dc0 msgr2=0x7f80c4140da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.822+0000 7f80c3fff700 1 --2- 172.21.15.27:0/369219081 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f80c4142dc0 0x7f80c4140da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.822+0000 7f80c3fff700 1 -- 172.21.15.27:0/369219081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c40cb2e0 msgr2=0x7f80c416d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.822+0000 7f80c3fff700 1 --2- 172.21.15.27:0/369219081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c40cb2e0 0x7f80c416d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.822+0000 7f80c3fff700 1 -- 172.21.15.27:0/369219081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80bc009a20 con 0x7f80c4143190 2024-01-11T08:26:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.822+0000 7f80c3fff700 1 --2- 172.21.15.27:0/369219081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c4143190 0x7f80c40c9680 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0099f0 tx=0x7f80bc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.822+0000 7f80ab7fe700 1 -- 172.21.15.27:0/369219081 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80bc012d00 con 0x7f80c4143190 2024-01-11T08:26:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.823+0000 7f80ab7fe700 1 -- 172.21.15.27:0/369219081 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80bc027470 con 0x7f80c4143190 2024-01-11T08:26:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.823+0000 7f80ab7fe700 1 -- 172.21.15.27:0/369219081 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80bc012ea0 con 0x7f80c4143190 2024-01-11T08:26:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.823+0000 7f80ca1ff700 1 -- 172.21.15.27:0/369219081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c40c5de0 con 0x7f80c4143190 2024-01-11T08:26:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.823+0000 7f80ca1ff700 1 -- 172.21.15.27:0/369219081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c41412e0 con 0x7f80c4143190 2024-01-11T08:26:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.824+0000 7f80ca1ff700 1 -- 172.21.15.27:0/369219081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c404c3c0 con 0x7f80c4143190 2024-01-11T08:26:28.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.825+0000 7f80ab7fe700 1 -- 172.21.15.27:0/369219081 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f80bc024070 con 0x7f80c4143190 2024-01-11T08:26:28.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.826+0000 7f80ab7fe700 1 --2- 172.21.15.27:0/369219081 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f80b0066ea0 0x7f80b0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:28.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.826+0000 7f80ab7fe700 1 -- 172.21.15.27:0/369219081 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5483+0+0 (secure 0 0 0) 0x7f80bc086970 con 0x7f80c4143190 2024-01-11T08:26:28.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.826+0000 7f80c37fe700 1 --2- 172.21.15.27:0/369219081 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f80b0066ea0 0x7f80b0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:28.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.827+0000 7f80c37fe700 1 --2- 172.21.15.27:0/369219081 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f80b0066ea0 0x7f80b0069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f80b4009db0 tx=0x7f80b40091a0 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:28.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:28.830+0000 7f80ab7fe700 1 -- 172.21.15.27:0/369219081 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f80bc053da0 con 0x7f80c4143190 2024-01-11T08:26:29.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.037+0000 7f80ca1ff700 1 -- 172.21.15.27:0/369219081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f80c40c6570 con 0x7f80c4143190 2024-01-11T08:26:29.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.043+0000 7f80ab7fe700 1 -- 172.21.15.27:0/369219081 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v32) v1 ==== 115+0+0 (secure 0 0 0) 0x7f80bc02c0b0 con 0x7f80c4143190 2024-01-11T08:26:29.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.045+0000 7f80ca1ff700 1 -- 172.21.15.27:0/369219081 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f80b0066ea0 msgr2=0x7f80b0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:29.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.045+0000 7f80ca1ff700 1 --2- 172.21.15.27:0/369219081 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f80b0066ea0 0x7f80b0069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f80b4009db0 tx=0x7f80b40091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:29.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.045+0000 7f80ca1ff700 1 -- 172.21.15.27:0/369219081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c4143190 msgr2=0x7f80c40c9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:29.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.045+0000 7f80ca1ff700 1 --2- 172.21.15.27:0/369219081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c4143190 0x7f80c40c9680 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0099f0 tx=0x7f80bc019000 comp rx=0 tx=0).stop 2024-01-11T08:26:29.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.045+0000 7f80ca1ff700 1 -- 172.21.15.27:0/369219081 shutdown_connections 2024-01-11T08:26:29.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.045+0000 7f80ca1ff700 1 --2- 172.21.15.27:0/369219081 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f80b0066ea0 0x7f80b0069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:29.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.045+0000 7f80ca1ff700 1 --2- 172.21.15.27:0/369219081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c4143190 0x7f80c40c9680 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:29.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.045+0000 7f80ca1ff700 1 --2- 172.21.15.27:0/369219081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c40cb2e0 0x7f80c416d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:29.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.045+0000 7f80ca1ff700 1 --2- 172.21.15.27:0/369219081 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f80c4142dc0 0x7f80c4140da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:29.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.045+0000 7f80ca1ff700 1 -- 172.21.15.27:0/369219081 >> 172.21.15.27:0/369219081 conn(0x7f80c4063ca0 msgr2=0x7f80c40c3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:29.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.045+0000 7f80ca1ff700 1 -- 172.21.15.27:0/369219081 shutdown_connections 2024-01-11T08:26:29.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.045+0000 7f80ca1ff700 1 -- 172.21.15.27:0/369219081 wait complete. 2024-01-11T08:26:29.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-11T08:26:29.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.264+0000 7fa3ddbff700 1 Processor -- start 2024-01-11T08:26:29.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.265+0000 7fa3ddbff700 1 -- start start 2024-01-11T08:26:29.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.266+0000 7fa3ddbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3d815a1b0 0x7fa3d815a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:29.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.266+0000 7fa3ddbff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa3d80ccd70 0x7fa3d80cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:29.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.266+0000 7fa3ddbff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d80cf850 con 0x7fa3d80685c0 2024-01-11T08:26:29.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.266+0000 7fa3ddbff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d80cec40 con 0x7fa3d80ccd70 2024-01-11T08:26:29.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.266+0000 7fa3ddbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d80cef20 con 0x7fa3d815a1b0 2024-01-11T08:26:29.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.266+0000 7fa3d6ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa3d80ccd70 0x7fa3d80cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:29.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.266+0000 7fa3d6ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa3d80ccd70 0x7fa3d80cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54072/0 (socket says 172.21.15.27:54072) 2024-01-11T08:26:29.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.266+0000 7fa3d6ffd700 1 -- 172.21.15.27:0/2721448723 learned_addr learned my addr 172.21.15.27:0/2721448723 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:29.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.267+0000 7fa3d77fe700 1 --2- 172.21.15.27:0/2721448723 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3d815a1b0 0x7fa3d815a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:29.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.267+0000 7fa3d6ffd700 1 -- 172.21.15.27:0/2721448723 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3d815a1b0 msgr2=0x7fa3d815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:29.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.267+0000 7fa3d6ffd700 1 --2- 172.21.15.27:0/2721448723 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3d815a1b0 0x7fa3d815a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:29.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.267+0000 7fa3d6ffd700 1 -- 172.21.15.27:0/2721448723 >> v1:172.21.15.77:6789/0 conn(0x7fa3d80685c0 legacy=0x7fa3d8068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:29.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.267+0000 7fa3d6ffd700 1 -- 172.21.15.27:0/2721448723 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d80cf200 con 0x7fa3d80ccd70 2024-01-11T08:26:29.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.267+0000 7fa3d77fe700 1 --2- 172.21.15.27:0/2721448723 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3d815a1b0 0x7fa3d815a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:26:29.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.267+0000 7fa3d6ffd700 1 --2- 172.21.15.27:0/2721448723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3d80ccd70 0x7fa3d80cd1e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc0099b0 tx=0x7fa3cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c420c693771a3ace server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:29.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.267+0000 7fa3d67fc700 1 -- 172.21.15.27:0/2721448723 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc00b860 con 0x7fa3d80ccd70 2024-01-11T08:26:29.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.268+0000 7fa3d67fc700 1 -- 172.21.15.27:0/2721448723 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3cc00b9c0 con 0x7fa3d80ccd70 2024-01-11T08:26:29.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.268+0000 7fa3d67fc700 1 -- 172.21.15.27:0/2721448723 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc00bd60 con 0x7fa3d80ccd70 2024-01-11T08:26:29.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.268+0000 7fa3ddbff700 1 -- 172.21.15.27:0/2721448723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3d80ccd70 msgr2=0x7fa3d80cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:29.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.268+0000 7fa3ddbff700 1 --2- 172.21.15.27:0/2721448723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3d80ccd70 0x7fa3d80cd1e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc0099b0 tx=0x7fa3cc009250 comp rx=0 tx=0).stop 2024-01-11T08:26:29.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.268+0000 7fa3ddbff700 1 -- 172.21.15.27:0/2721448723 shutdown_connections 2024-01-11T08:26:29.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.268+0000 7fa3ddbff700 1 --2- 172.21.15.27:0/2721448723 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3d815a1b0 0x7fa3d815a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:29.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.268+0000 7fa3ddbff700 1 --2- 172.21.15.27:0/2721448723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3d80ccd70 0x7fa3d80cd1e0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:29.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.268+0000 7fa3ddbff700 1 -- 172.21.15.27:0/2721448723 >> 172.21.15.27:0/2721448723 conn(0x7fa3d806ed40 msgr2=0x7fa3d806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:29.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.268+0000 7fa3ddbff700 1 -- 172.21.15.27:0/2721448723 shutdown_connections 2024-01-11T08:26:29.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.268+0000 7fa3ddbff700 1 -- 172.21.15.27:0/2721448723 wait complete. 2024-01-11T08:26:29.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.270+0000 7fa3ddbff700 1 Processor -- start 2024-01-11T08:26:29.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.270+0000 7fa3ddbff700 1 -- start start 2024-01-11T08:26:29.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.271+0000 7fa3ddbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3d80685c0 0x7fa3d81540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:29.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.271+0000 7fa3ddbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3d80ccd70 0x7fa3d8145130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:29.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.271+0000 7fa3ddbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3d815a1b0 0x7fa3d8151470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.271+0000 7fa3ddbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d8155dc0 con 0x7fa3d80ccd70 2024-01-11T08:26:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.271+0000 7fa3ddbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d8154720 con 0x7fa3d80685c0 2024-01-11T08:26:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.271+0000 7fa3ddbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d8155490 con 0x7fa3d815a1b0 2024-01-11T08:26:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.271+0000 7fa3d6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3d80ccd70 0x7fa3d8145130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.271+0000 7fa3d6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3d80ccd70 0x7fa3d8145130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54084/0 (socket says 172.21.15.27:54084) 2024-01-11T08:26:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.271+0000 7fa3d6ffd700 1 -- 172.21.15.27:0/3963389177 learned_addr learned my addr 172.21.15.27:0/3963389177 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.271+0000 7fa3d7fff700 1 --2- 172.21.15.27:0/3963389177 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3d815a1b0 0x7fa3d8151470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.271+0000 7fa3d77fe700 1 --2- 172.21.15.27:0/3963389177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3d80685c0 0x7fa3d81540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.272+0000 7fa3d6ffd700 1 -- 172.21.15.27:0/3963389177 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3d815a1b0 msgr2=0x7fa3d8151470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.272+0000 7fa3d6ffd700 1 --2- 172.21.15.27:0/3963389177 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3d815a1b0 0x7fa3d8151470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.272+0000 7fa3d6ffd700 1 -- 172.21.15.27:0/3963389177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3d80685c0 msgr2=0x7fa3d81540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.272+0000 7fa3d6ffd700 1 --2- 172.21.15.27:0/3963389177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3d80685c0 0x7fa3d81540b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.272+0000 7fa3d6ffd700 1 -- 172.21.15.27:0/3963389177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc0094f0 con 0x7fa3d80ccd70 2024-01-11T08:26:29.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.272+0000 7fa3d6ffd700 1 --2- 172.21.15.27:0/3963389177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3d80ccd70 0x7fa3d8145130 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009290 tx=0x7fa3cc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:29.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.272+0000 7fa3bf7fe700 1 -- 172.21.15.27:0/3963389177 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc011e00 con 0x7fa3d80ccd70 2024-01-11T08:26:29.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.272+0000 7fa3bf7fe700 1 -- 172.21.15.27:0/3963389177 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3cc01f070 con 0x7fa3d80ccd70 2024-01-11T08:26:29.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.272+0000 7fa3bf7fe700 1 -- 172.21.15.27:0/3963389177 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc0266d0 con 0x7fa3d80ccd70 2024-01-11T08:26:29.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.272+0000 7fa3ddbff700 1 -- 172.21.15.27:0/3963389177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d8155770 con 0x7fa3d80ccd70 2024-01-11T08:26:29.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.272+0000 7fa3ddbff700 1 -- 172.21.15.27:0/3963389177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d81519b0 con 0x7fa3d80ccd70 2024-01-11T08:26:29.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.273+0000 7fa3ddbff700 1 -- 172.21.15.27:0/3963389177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d8155f70 con 0x7fa3d80ccd70 2024-01-11T08:26:29.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.274+0000 7fa3bf7fe700 1 -- 172.21.15.27:0/3963389177 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fa3cc026a40 con 0x7fa3d80ccd70 2024-01-11T08:26:29.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.275+0000 7fa3bf7fe700 1 --2- 172.21.15.27:0/3963389177 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa3c4066ea0 0x7fa3c4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:29.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.275+0000 7fa3bf7fe700 1 -- 172.21.15.27:0/3963389177 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5483+0+0 (secure 0 0 0) 0x7fa3cc021070 con 0x7fa3d80ccd70 2024-01-11T08:26:29.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.276+0000 7fa3d77fe700 1 --2- 172.21.15.27:0/3963389177 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa3c4066ea0 0x7fa3c4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:29.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.276+0000 7fa3d77fe700 1 --2- 172.21.15.27:0/3963389177 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa3c4066ea0 0x7fa3c4069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8154040 tx=0x7fa3c8008040 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:29.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.279+0000 7fa3bf7fe700 1 -- 172.21.15.27:0/3963389177 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa3cc053470 con 0x7fa3d80ccd70 2024-01-11T08:26:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.479+0000 7fa3ddbff700 1 -- 172.21.15.27:0/3963389177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa3d804c3c0 con 0x7fa3d80ccd70 2024-01-11T08:26:29.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.485+0000 7fa3bf7fe700 1 -- 172.21.15.27:0/3963389177 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa3cc02b0a0 con 0x7fa3d80ccd70 2024-01-11T08:26:29.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.487+0000 7fa3ddbff700 1 -- 172.21.15.27:0/3963389177 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa3c4066ea0 msgr2=0x7fa3c4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:29.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.487+0000 7fa3ddbff700 1 --2- 172.21.15.27:0/3963389177 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa3c4066ea0 0x7fa3c4069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8154040 tx=0x7fa3c8008040 comp rx=0 tx=0).stop 2024-01-11T08:26:29.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.487+0000 7fa3ddbff700 1 -- 172.21.15.27:0/3963389177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3d80ccd70 msgr2=0x7fa3d8145130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:29.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.487+0000 7fa3ddbff700 1 --2- 172.21.15.27:0/3963389177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3d80ccd70 0x7fa3d8145130 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009290 tx=0x7fa3cc017000 comp rx=0 tx=0).stop 2024-01-11T08:26:29.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.488+0000 7fa3ddbff700 1 -- 172.21.15.27:0/3963389177 shutdown_connections 2024-01-11T08:26:29.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.488+0000 7fa3ddbff700 1 --2- 172.21.15.27:0/3963389177 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa3c4066ea0 0x7fa3c4069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:29.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.488+0000 7fa3ddbff700 1 --2- 172.21.15.27:0/3963389177 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa3d815a1b0 0x7fa3d8151470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:29.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.488+0000 7fa3ddbff700 1 --2- 172.21.15.27:0/3963389177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3d80685c0 0x7fa3d81540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:29.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.488+0000 7fa3ddbff700 1 --2- 172.21.15.27:0/3963389177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3d80ccd70 0x7fa3d8145130 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:29.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.488+0000 7fa3ddbff700 1 -- 172.21.15.27:0/3963389177 >> 172.21.15.27:0/3963389177 conn(0x7fa3d806ed40 msgr2=0x7fa3d80c5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:29.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.488+0000 7fa3ddbff700 1 -- 172.21.15.27:0/3963389177 shutdown_connections 2024-01-11T08:26:29.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.488+0000 7fa3ddbff700 1 -- 172.21.15.27:0/3963389177 wait complete. 2024-01-11T08:26:29.504 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2024-01-11T08:26:29.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.706+0000 7fdb763ff700 1 Processor -- start 2024-01-11T08:26:29.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.706+0000 7fdb763ff700 1 -- start start 2024-01-11T08:26:29.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.707+0000 7fdb763ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb700cc410 0x7fdb70159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:29.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.707+0000 7fdb763ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb700cd730 con 0x7fdb700b03e0 2024-01-11T08:26:29.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.707+0000 7fdb763ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb700ccb20 con 0x7fdb700b2b90 2024-01-11T08:26:29.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.707+0000 7fdb763ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb700cce00 con 0x7fdb700cc410 2024-01-11T08:26:29.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.708+0000 7fdb6ffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdb700b03e0 0x7fdb700b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56130/0 (socket says 172.21.15.27:56130) 2024-01-11T08:26:29.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.708+0000 7fdb6ffff700 1 -- 172.21.15.27:0/515080252 learned_addr learned my addr 172.21.15.27:0/515080252 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:29.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.708+0000 7fdb7499c700 1 --2- 172.21.15.27:0/515080252 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb700cc410 0x7fdb70159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.708+0000 7fdb6effd700 1 -- 172.21.15.27:0/515080252 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300049403 0 0) 0x7fdb700cd730 con 0x7fdb700b03e0 2024-01-11T08:26:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.708+0000 7fdb6effd700 1 -- 172.21.15.27:0/515080252 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb58003570 con 0x7fdb700b03e0 2024-01-11T08:26:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.708+0000 7fdb6effd700 1 -- 172.21.15.27:0/515080252 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581471845 0 0) 0x7fdb700ccb20 con 0x7fdb700b2b90 2024-01-11T08:26:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.708+0000 7fdb6effd700 1 -- 172.21.15.27:0/515080252 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb700cd730 con 0x7fdb700b2b90 2024-01-11T08:26:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.708+0000 7fdb7499c700 1 -- 172.21.15.27:0/515080252 >> v1:172.21.15.77:6789/0 conn(0x7fdb700b2b90 legacy=0x7fdb70159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.709+0000 7fdb7499c700 1 -- 172.21.15.27:0/515080252 >> v1:172.21.15.27:6789/0 conn(0x7fdb700b03e0 legacy=0x7fdb700b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.709+0000 7fdb7499c700 1 -- 172.21.15.27:0/515080252 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb700cd0e0 con 0x7fdb700cc410 2024-01-11T08:26:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.709+0000 7fdb7499c700 1 --2- 172.21.15.27:0/515080252 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb700cc410 0x7fdb70159910 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009e90 tx=0x7fdb68009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c558682448dbe152 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.709+0000 7fdb6effd700 1 -- 172.21.15.27:0/515080252 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6800e070 con 0x7fdb700cc410 2024-01-11T08:26:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.709+0000 7fdb6effd700 1 -- 172.21.15.27:0/515080252 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb6800bb80 con 0x7fdb700cc410 2024-01-11T08:26:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.709+0000 7fdb6effd700 1 -- 172.21.15.27:0/515080252 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb68012430 con 0x7fdb700cc410 2024-01-11T08:26:29.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.710+0000 7fdb763ff700 1 -- 172.21.15.27:0/515080252 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb700cc410 msgr2=0x7fdb70159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:29.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.710+0000 7fdb763ff700 1 --2- 172.21.15.27:0/515080252 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb700cc410 0x7fdb70159910 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009e90 tx=0x7fdb68009780 comp rx=0 tx=0).stop 2024-01-11T08:26:29.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.710+0000 7fdb763ff700 1 -- 172.21.15.27:0/515080252 shutdown_connections 2024-01-11T08:26:29.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.710+0000 7fdb763ff700 1 --2- 172.21.15.27:0/515080252 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb700cc410 0x7fdb70159910 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:29.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.710+0000 7fdb763ff700 1 -- 172.21.15.27:0/515080252 >> 172.21.15.27:0/515080252 conn(0x7fdb7006ed40 msgr2=0x7fdb7006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:29.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.710+0000 7fdb763ff700 1 -- 172.21.15.27:0/515080252 shutdown_connections 2024-01-11T08:26:29.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.710+0000 7fdb763ff700 1 -- 172.21.15.27:0/515080252 wait complete. 2024-01-11T08:26:29.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.711+0000 7fdb763ff700 1 Processor -- start 2024-01-11T08:26:29.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.712+0000 7fdb763ff700 1 -- start start 2024-01-11T08:26:29.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.712+0000 7fdb763ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb700b03e0 0x7fdb7014da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:29.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.712+0000 7fdb763ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb700b2b90 0x7fdb700d0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:29.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.712+0000 7fdb763ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb700cc410 0x7fdb700c9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:29.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.712+0000 7fdb763ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb700cfba0 con 0x7fdb700cc410 2024-01-11T08:26:29.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.712+0000 7fdb763ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb701696c0 con 0x7fdb700b2b90 2024-01-11T08:26:29.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.712+0000 7fdb763ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7014df40 con 0x7fdb700b03e0 2024-01-11T08:26:29.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.713+0000 7fdb7499c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb700cc410 0x7fdb700c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:29.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.713+0000 7fdb7499c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb700cc410 0x7fdb700c9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54092/0 (socket says 172.21.15.27:54092) 2024-01-11T08:26:29.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.713+0000 7fdb7499c700 1 -- 172.21.15.27:0/2943849433 learned_addr learned my addr 172.21.15.27:0/2943849433 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:29.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.713+0000 7fdb6ffff700 1 --2- 172.21.15.27:0/2943849433 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb700b03e0 0x7fdb7014da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:29.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.713+0000 7fdb6f7fe700 1 --2- 172.21.15.27:0/2943849433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb700b2b90 0x7fdb700d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:29.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.713+0000 7fdb7499c700 1 -- 172.21.15.27:0/2943849433 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb700b03e0 msgr2=0x7fdb7014da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:29.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.713+0000 7fdb7499c700 1 --2- 172.21.15.27:0/2943849433 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb700b03e0 0x7fdb7014da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:29.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.713+0000 7fdb7499c700 1 -- 172.21.15.27:0/2943849433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb700b2b90 msgr2=0x7fdb700d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:29.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.713+0000 7fdb7499c700 1 --2- 172.21.15.27:0/2943849433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb700b2b90 0x7fdb700d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:29.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.713+0000 7fdb7499c700 1 -- 172.21.15.27:0/2943849433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb68009a20 con 0x7fdb700cc410 2024-01-11T08:26:29.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.714+0000 7fdb7499c700 1 --2- 172.21.15.27:0/2943849433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb700cc410 0x7fdb700c9000 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fdb70069540 tx=0x7fdb68019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:29.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.714+0000 7fdb57fff700 1 -- 172.21.15.27:0/2943849433 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6801b420 con 0x7fdb700cc410 2024-01-11T08:26:29.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.714+0000 7fdb57fff700 1 -- 172.21.15.27:0/2943849433 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb6801b5a0 con 0x7fdb700cc410 2024-01-11T08:26:29.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.714+0000 7fdb57fff700 1 -- 172.21.15.27:0/2943849433 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6801b960 con 0x7fdb700cc410 2024-01-11T08:26:29.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.714+0000 7fdb763ff700 1 -- 172.21.15.27:0/2943849433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb700cf020 con 0x7fdb700cc410 2024-01-11T08:26:29.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.714+0000 7fdb763ff700 1 -- 172.21.15.27:0/2943849433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb700cf5e0 con 0x7fdb700cc410 2024-01-11T08:26:29.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.715+0000 7fdb763ff700 1 -- 172.21.15.27:0/2943849433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb70169870 con 0x7fdb700cc410 2024-01-11T08:26:29.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.716+0000 7fdb57fff700 1 -- 172.21.15.27:0/2943849433 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fdb6801bd50 con 0x7fdb700cc410 2024-01-11T08:26:29.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.717+0000 7fdb57fff700 1 --2- 172.21.15.27:0/2943849433 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fdb5c066ea0 0x7fdb5c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:29.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.718+0000 7fdb57fff700 1 -- 172.21.15.27:0/2943849433 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5483+0+0 (secure 0 0 0) 0x7fdb68085c60 con 0x7fdb700cc410 2024-01-11T08:26:29.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.718+0000 7fdb6ffff700 1 --2- 172.21.15.27:0/2943849433 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fdb5c066ea0 0x7fdb5c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:29.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.718+0000 7fdb6ffff700 1 --2- 172.21.15.27:0/2943849433 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fdb5c066ea0 0x7fdb5c069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdb60007fd0 tx=0x7fdb60009040 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:29.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:29.722+0000 7fdb57fff700 1 -- 172.21.15.27:0/2943849433 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdb70169870 con 0x7fdb700cc410 2024-01-11T08:26:30.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:30.006+0000 7fdb763ff700 1 -- 172.21.15.27:0/2943849433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7fdb700b1440 con 0x7fdb700cc410 2024-01-11T08:26:30.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:30.052+0000 7fdb57fff700 1 -- 172.21.15.27:0/2943849433 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fdb68053510 con 0x7fdb700cc410 2024-01-11T08:26:30.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:30.957+0000 7fdb57fff700 1 -- 172.21.15.27:0/2943849433 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v596) v1 ==== 166+0+0 (secure 0 0 0) 0x7fdb6808b190 con 0x7fdb700cc410 2024-01-11T08:26:30.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:30.960+0000 7fdb763ff700 1 -- 172.21.15.27:0/2943849433 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fdb5c066ea0 msgr2=0x7fdb5c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:30.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:30.960+0000 7fdb763ff700 1 --2- 172.21.15.27:0/2943849433 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fdb5c066ea0 0x7fdb5c069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdb60007fd0 tx=0x7fdb60009040 comp rx=0 tx=0).stop 2024-01-11T08:26:30.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:30.960+0000 7fdb763ff700 1 -- 172.21.15.27:0/2943849433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb700cc410 msgr2=0x7fdb700c9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:30.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:30.960+0000 7fdb763ff700 1 --2- 172.21.15.27:0/2943849433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb700cc410 0x7fdb700c9000 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fdb70069540 tx=0x7fdb68019000 comp rx=0 tx=0).stop 2024-01-11T08:26:30.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:30.960+0000 7fdb763ff700 1 -- 172.21.15.27:0/2943849433 shutdown_connections 2024-01-11T08:26:30.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:30.960+0000 7fdb763ff700 1 --2- 172.21.15.27:0/2943849433 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fdb5c066ea0 0x7fdb5c069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:30.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:30.960+0000 7fdb763ff700 1 --2- 172.21.15.27:0/2943849433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb700cc410 0x7fdb700c9000 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:30.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:30.960+0000 7fdb763ff700 1 --2- 172.21.15.27:0/2943849433 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb700b03e0 0x7fdb7014da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:30.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:30.960+0000 7fdb763ff700 1 --2- 172.21.15.27:0/2943849433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb700b2b90 0x7fdb700d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:30.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:30.960+0000 7fdb763ff700 1 -- 172.21.15.27:0/2943849433 >> 172.21.15.27:0/2943849433 conn(0x7fdb7006ed40 msgr2=0x7fdb700b2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:30.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:30.960+0000 7fdb763ff700 1 -- 172.21.15.27:0/2943849433 shutdown_connections 2024-01-11T08:26:30.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:30.960+0000 7fdb763ff700 1 -- 172.21.15.27:0/2943849433 wait complete. 2024-01-11T08:26:30.980 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2024-01-11T08:26:30.980 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that a non-existent path leads to schedule deactivation after a few retries. 2024-01-11T08:26:30.980 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-11T08:26:30.980 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2024-01-11T08:26:31.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.186+0000 7fb5a93ff700 1 Processor -- start 2024-01-11T08:26:31.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.186+0000 7fb5a93ff700 1 -- start start 2024-01-11T08:26:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.188+0000 7fb5a93ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb5a4069d40 0x7fb5a40b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.188+0000 7fb5a93ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a4157390 con 0x7fb5a40c8620 2024-01-11T08:26:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.188+0000 7fb5a93ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a406a200 con 0x7fb5a40c89f0 2024-01-11T08:26:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.188+0000 7fb5a93ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a4156d40 con 0x7fb5a4069d40 2024-01-11T08:26:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.188+0000 7fb5a2ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb5a40c8620 0x7fb5a40af970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56138/0 (socket says 172.21.15.27:56138) 2024-01-11T08:26:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.188+0000 7fb5a2ffd700 1 -- 172.21.15.27:0/380919151 learned_addr learned my addr 172.21.15.27:0/380919151 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.188+0000 7fb5a27fc700 1 --2- 172.21.15.27:0/380919151 >> v2:172.21.15.77:3300/0 conn(0x7fb5a4069d40 0x7fb5a40b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.188+0000 7fb5a1ffb700 1 -- 172.21.15.27:0/380919151 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179632190 0 0) 0x7fb5a4157390 con 0x7fb5a40c8620 2024-01-11T08:26:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.188+0000 7fb5a1ffb700 1 -- 172.21.15.27:0/380919151 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb58c003570 con 0x7fb5a40c8620 2024-01-11T08:26:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.189+0000 7fb5a1ffb700 1 -- 172.21.15.27:0/380919151 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205478809 0 0) 0x7fb5a406a200 con 0x7fb5a40c89f0 2024-01-11T08:26:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.189+0000 7fb5a1ffb700 1 -- 172.21.15.27:0/380919151 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5a4157390 con 0x7fb5a40c89f0 2024-01-11T08:26:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.189+0000 7fb5a27fc700 1 -- 172.21.15.27:0/380919151 >> v1:172.21.15.77:6789/0 conn(0x7fb5a40c89f0 legacy=0x7fb5a40accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.189+0000 7fb5a27fc700 1 -- 172.21.15.27:0/380919151 >> v1:172.21.15.27:6789/0 conn(0x7fb5a40c8620 legacy=0x7fb5a40af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.189+0000 7fb5a27fc700 1 -- 172.21.15.27:0/380919151 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a4157020 con 0x7fb5a4069d40 2024-01-11T08:26:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.189+0000 7fb5a27fc700 1 --2- 172.21.15.27:0/380919151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a4069d40 0x7fb5a40b02d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb5980098e0 tx=0x7fb5980091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84e8c7b9897ffbff server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.189+0000 7fb5a1ffb700 1 -- 172.21.15.27:0/380919151 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb598011630 con 0x7fb5a4069d40 2024-01-11T08:26:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.189+0000 7fb5a1ffb700 1 -- 172.21.15.27:0/380919151 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb598011790 con 0x7fb5a4069d40 2024-01-11T08:26:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.189+0000 7fb5a1ffb700 1 -- 172.21.15.27:0/380919151 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb598011b10 con 0x7fb5a4069d40 2024-01-11T08:26:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.190+0000 7fb5a93ff700 1 -- 172.21.15.27:0/380919151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a4069d40 msgr2=0x7fb5a40b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.190+0000 7fb5a93ff700 1 --2- 172.21.15.27:0/380919151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a4069d40 0x7fb5a40b02d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb5980098e0 tx=0x7fb5980091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.190+0000 7fb5a93ff700 1 -- 172.21.15.27:0/380919151 shutdown_connections 2024-01-11T08:26:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.190+0000 7fb5a93ff700 1 --2- 172.21.15.27:0/380919151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a4069d40 0x7fb5a40b02d0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.190+0000 7fb5a93ff700 1 -- 172.21.15.27:0/380919151 >> 172.21.15.27:0/380919151 conn(0x7fb5a40680e0 msgr2=0x7fb5a4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.190+0000 7fb5a93ff700 1 -- 172.21.15.27:0/380919151 shutdown_connections 2024-01-11T08:26:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.190+0000 7fb5a93ff700 1 -- 172.21.15.27:0/380919151 wait complete. 2024-01-11T08:26:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.192+0000 7fb5a93ff700 1 Processor -- start 2024-01-11T08:26:31.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.192+0000 7fb5a93ff700 1 -- start start 2024-01-11T08:26:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.193+0000 7fb5a93ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a4069d40 0x7fb5a4148be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.193+0000 7fb5a93ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5a40c8620 0x7fb5a415c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.193+0000 7fb5a93ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5a40c89f0 0x7fb5a406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.193+0000 7fb5a93ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a415a950 con 0x7fb5a40c89f0 2024-01-11T08:26:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.193+0000 7fb5a93ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a415ce60 con 0x7fb5a4069d40 2024-01-11T08:26:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.193+0000 7fb5a93ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a415a020 con 0x7fb5a40c8620 2024-01-11T08:26:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.193+0000 7fb5a37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5a40c89f0 0x7fb5a406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.193+0000 7fb5a37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5a40c89f0 0x7fb5a406e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54102/0 (socket says 172.21.15.27:54102) 2024-01-11T08:26:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.193+0000 7fb5a37fe700 1 -- 172.21.15.27:0/2241085810 learned_addr learned my addr 172.21.15.27:0/2241085810 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.193+0000 7fb5a2ffd700 1 --2- 172.21.15.27:0/2241085810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a4069d40 0x7fb5a4148be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.193+0000 7fb5a27fc700 1 --2- 172.21.15.27:0/2241085810 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5a40c8620 0x7fb5a415c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.194+0000 7fb5a37fe700 1 -- 172.21.15.27:0/2241085810 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5a40c8620 msgr2=0x7fb5a415c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.194+0000 7fb5a37fe700 1 --2- 172.21.15.27:0/2241085810 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5a40c8620 0x7fb5a415c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.194+0000 7fb5a37fe700 1 -- 172.21.15.27:0/2241085810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a4069d40 msgr2=0x7fb5a4148be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.194+0000 7fb5a37fe700 1 --2- 172.21.15.27:0/2241085810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a4069d40 0x7fb5a4148be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.194+0000 7fb5a37fe700 1 -- 172.21.15.27:0/2241085810 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb598009420 con 0x7fb5a40c89f0 2024-01-11T08:26:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.194+0000 7fb5a37fe700 1 --2- 172.21.15.27:0/2241085810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5a40c89f0 0x7fb5a406e770 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00c910 tx=0x7fb59c007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.194+0000 7fb58affd700 1 -- 172.21.15.27:0/2241085810 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c012630 con 0x7fb5a40c89f0 2024-01-11T08:26:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.194+0000 7fb58affd700 1 -- 172.21.15.27:0/2241085810 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c0127b0 con 0x7fb5a40c89f0 2024-01-11T08:26:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.194+0000 7fb5a93ff700 1 -- 172.21.15.27:0/2241085810 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a415a300 con 0x7fb5a40c89f0 2024-01-11T08:26:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.194+0000 7fb58affd700 1 -- 172.21.15.27:0/2241085810 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c012b70 con 0x7fb5a40c89f0 2024-01-11T08:26:31.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.194+0000 7fb5a93ff700 1 -- 172.21.15.27:0/2241085810 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a4149120 con 0x7fb5a40c89f0 2024-01-11T08:26:31.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.195+0000 7fb5a93ff700 1 -- 172.21.15.27:0/2241085810 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a415aac0 con 0x7fb5a40c89f0 2024-01-11T08:26:31.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.196+0000 7fb58affd700 1 -- 172.21.15.27:0/2241085810 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb59c01b400 con 0x7fb5a40c89f0 2024-01-11T08:26:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.198+0000 7fb58affd700 1 --2- 172.21.15.27:0/2241085810 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb590066f30 0x7fb5900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.198+0000 7fb58affd700 1 -- 172.21.15.27:0/2241085810 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5483+0+0 (secure 0 0 0) 0x7fb59c084710 con 0x7fb5a40c89f0 2024-01-11T08:26:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.198+0000 7fb5a2ffd700 1 --2- 172.21.15.27:0/2241085810 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb590066f30 0x7fb5900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.199+0000 7fb5a2ffd700 1 --2- 172.21.15.27:0/2241085810 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb590066f30 0x7fb5900693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb594007fd0 tx=0x7fb594009040 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:31.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.202+0000 7fb58affd700 1 -- 172.21.15.27:0/2241085810 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb59c051a60 con 0x7fb5a40c89f0 2024-01-11T08:26:31.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.489+0000 7fb5a93ff700 1 -- 172.21.15.27:0/2241085810 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7fb5a40718d0 con 0x7fb5a40c89f0 2024-01-11T08:26:31.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.962+0000 7fb58affd700 1 -- 172.21.15.27:0/2241085810 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v597) v1 ==== 186+0+0 (secure 0 0 0) 0x7fb59c016100 con 0x7fb5a40c89f0 2024-01-11T08:26:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.965+0000 7fb5a93ff700 1 -- 172.21.15.27:0/2241085810 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb590066f30 msgr2=0x7fb5900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.965+0000 7fb5a93ff700 1 --2- 172.21.15.27:0/2241085810 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb590066f30 0x7fb5900693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb594007fd0 tx=0x7fb594009040 comp rx=0 tx=0).stop 2024-01-11T08:26:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.965+0000 7fb5a93ff700 1 -- 172.21.15.27:0/2241085810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5a40c89f0 msgr2=0x7fb5a406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.965+0000 7fb5a93ff700 1 --2- 172.21.15.27:0/2241085810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5a40c89f0 0x7fb5a406e770 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00c910 tx=0x7fb59c007f90 comp rx=0 tx=0).stop 2024-01-11T08:26:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.965+0000 7fb5a93ff700 1 -- 172.21.15.27:0/2241085810 shutdown_connections 2024-01-11T08:26:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.965+0000 7fb5a93ff700 1 --2- 172.21.15.27:0/2241085810 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb590066f30 0x7fb5900693e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.965+0000 7fb5a93ff700 1 --2- 172.21.15.27:0/2241085810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5a40c89f0 0x7fb5a406e770 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.965+0000 7fb5a93ff700 1 --2- 172.21.15.27:0/2241085810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a4069d40 0x7fb5a4148be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.965+0000 7fb5a93ff700 1 --2- 172.21.15.27:0/2241085810 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5a40c8620 0x7fb5a415c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.965+0000 7fb5a93ff700 1 -- 172.21.15.27:0/2241085810 >> 172.21.15.27:0/2241085810 conn(0x7fb5a40680e0 msgr2=0x7fb5a41537e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.965+0000 7fb5a93ff700 1 -- 172.21.15.27:0/2241085810 shutdown_connections 2024-01-11T08:26:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:31.965+0000 7fb5a93ff700 1 -- 172.21.15.27:0/2241085810 wait complete. 2024-01-11T08:26:31.983 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-11T08:26:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.189+0000 7fc7c29ff700 1 Processor -- start 2024-01-11T08:26:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.189+0000 7fc7c29ff700 1 -- start start 2024-01-11T08:26:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.190+0000 7fc7c29ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7bc13fbb0 0x7fc7bc0ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.190+0000 7fc7c29ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7bc13ff80 0x7fc7bc15a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.190+0000 7fc7c29ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7bc0c6b30 con 0x7fc7bc0cace0 2024-01-11T08:26:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.190+0000 7fc7c29ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7bc0c6800 con 0x7fc7bc13fbb0 2024-01-11T08:26:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.190+0000 7fc7c29ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0cb240 con 0x7fc7bc13ff80 2024-01-11T08:26:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.191+0000 7fc7bbfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7bc13fbb0 0x7fc7bc0ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.191+0000 7fc7bbfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7bc13fbb0 0x7fc7bc0ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54104/0 (socket says 172.21.15.27:54104) 2024-01-11T08:26:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.191+0000 7fc7bbfff700 1 -- 172.21.15.27:0/531313593 learned_addr learned my addr 172.21.15.27:0/531313593 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.191+0000 7fc7bb7fe700 1 --2- 172.21.15.27:0/531313593 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7bc13ff80 0x7fc7bc15a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.191+0000 7fc7bbfff700 1 -- 172.21.15.27:0/531313593 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7bc13ff80 msgr2=0x7fc7bc15a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.191+0000 7fc7bbfff700 1 --2- 172.21.15.27:0/531313593 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7bc13ff80 0x7fc7bc15a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.191+0000 7fc7bbfff700 1 -- 172.21.15.27:0/531313593 >> v1:172.21.15.27:6789/0 conn(0x7fc7bc0cace0 legacy=0x7fc7bc06e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.191+0000 7fc7bbfff700 1 -- 172.21.15.27:0/531313593 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7bc0cb520 con 0x7fc7bc13fbb0 2024-01-11T08:26:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.191+0000 7fc7baffd700 1 -- 172.21.15.27:0/531313593 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665467750 0 0) 0x7fc7bc0c6b30 con 0x7fc7bc0cace0 2024-01-11T08:26:32.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.191+0000 7fc7bbfff700 1 --2- 172.21.15.27:0/531313593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc13fbb0 0x7fc7bc0ca7a0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009990 tx=0x7fc7ac009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c67d219f687c0f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:32.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.192+0000 7fc7baffd700 1 -- 172.21.15.27:0/531313593 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac009e80 con 0x7fc7bc13fbb0 2024-01-11T08:26:32.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.192+0000 7fc7baffd700 1 -- 172.21.15.27:0/531313593 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ac005c10 con 0x7fc7bc13fbb0 2024-01-11T08:26:32.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.192+0000 7fc7baffd700 1 -- 172.21.15.27:0/531313593 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac00b650 con 0x7fc7bc13fbb0 2024-01-11T08:26:32.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.192+0000 7fc7c29ff700 1 -- 172.21.15.27:0/531313593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc13fbb0 msgr2=0x7fc7bc0ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:32.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.192+0000 7fc7c29ff700 1 --2- 172.21.15.27:0/531313593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc13fbb0 0x7fc7bc0ca7a0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009990 tx=0x7fc7ac009220 comp rx=0 tx=0).stop 2024-01-11T08:26:32.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.192+0000 7fc7c29ff700 1 -- 172.21.15.27:0/531313593 shutdown_connections 2024-01-11T08:26:32.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.192+0000 7fc7c29ff700 1 --2- 172.21.15.27:0/531313593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc13fbb0 0x7fc7bc0ca7a0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:32.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.192+0000 7fc7c29ff700 1 --2- 172.21.15.27:0/531313593 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7bc13ff80 0x7fc7bc15a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:32.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.192+0000 7fc7c29ff700 1 -- 172.21.15.27:0/531313593 >> 172.21.15.27:0/531313593 conn(0x7fc7bc0680a0 msgr2=0x7fc7bc0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:32.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.192+0000 7fc7c29ff700 1 -- 172.21.15.27:0/531313593 shutdown_connections 2024-01-11T08:26:32.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.192+0000 7fc7c29ff700 1 -- 172.21.15.27:0/531313593 wait complete. 2024-01-11T08:26:32.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.194+0000 7fc7c29ff700 1 Processor -- start 2024-01-11T08:26:32.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.194+0000 7fc7c29ff700 1 -- start start 2024-01-11T08:26:32.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.195+0000 7fc7c29ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7bc0cace0 0x7fc7bc0b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:32.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.195+0000 7fc7c29ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc13fbb0 0x7fc7bc155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:32.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.195+0000 7fc7c29ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7bc13ff80 0x7fc7bc0af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:32.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.195+0000 7fc7c29ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0c27f0 con 0x7fc7bc13fbb0 2024-01-11T08:26:32.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.195+0000 7fc7c29ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc156330 con 0x7fc7bc13ff80 2024-01-11T08:26:32.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.195+0000 7fc7c29ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0c1be0 con 0x7fc7bc0cace0 2024-01-11T08:26:32.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.195+0000 7fc7bb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc13fbb0 0x7fc7bc155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:32.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.195+0000 7fc7bb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc13fbb0 0x7fc7bc155df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54110/0 (socket says 172.21.15.27:54110) 2024-01-11T08:26:32.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.195+0000 7fc7bb7fe700 1 -- 172.21.15.27:0/1867796308 learned_addr learned my addr 172.21.15.27:0/1867796308 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:32.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.195+0000 7fc7bbfff700 1 --2- 172.21.15.27:0/1867796308 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7bc0cace0 0x7fc7bc0b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:32.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.195+0000 7fc7c0f9c700 1 --2- 172.21.15.27:0/1867796308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7bc13ff80 0x7fc7bc0af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:32.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.196+0000 7fc7bb7fe700 1 -- 172.21.15.27:0/1867796308 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7bc0cace0 msgr2=0x7fc7bc0b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:32.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.196+0000 7fc7bb7fe700 1 --2- 172.21.15.27:0/1867796308 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7bc0cace0 0x7fc7bc0b02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:32.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.196+0000 7fc7bb7fe700 1 -- 172.21.15.27:0/1867796308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7bc13ff80 msgr2=0x7fc7bc0af950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:32.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.196+0000 7fc7bb7fe700 1 --2- 172.21.15.27:0/1867796308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7bc13ff80 0x7fc7bc0af950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:32.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.196+0000 7fc7bb7fe700 1 -- 172.21.15.27:0/1867796308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac009480 con 0x7fc7bc13fbb0 2024-01-11T08:26:32.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.196+0000 7fc7c0f9c700 1 --2- 172.21.15.27:0/1867796308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7bc13ff80 0x7fc7bc0af950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:32.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.196+0000 7fc7bb7fe700 1 --2- 172.21.15.27:0/1867796308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc13fbb0 0x7fc7bc155df0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc7b000e930 tx=0x7fc7b00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:32.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.196+0000 7fc7a3fff700 1 -- 172.21.15.27:0/1867796308 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b000eed0 con 0x7fc7bc13fbb0 2024-01-11T08:26:32.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.196+0000 7fc7a3fff700 1 -- 172.21.15.27:0/1867796308 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b000c850 con 0x7fc7bc13fbb0 2024-01-11T08:26:32.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.196+0000 7fc7a3fff700 1 -- 172.21.15.27:0/1867796308 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b000cbf0 con 0x7fc7bc13fbb0 2024-01-11T08:26:32.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.196+0000 7fc7c29ff700 1 -- 172.21.15.27:0/1867796308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7bc0c1ec0 con 0x7fc7bc13fbb0 2024-01-11T08:26:32.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.196+0000 7fc7c29ff700 1 -- 172.21.15.27:0/1867796308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7bc0c2480 con 0x7fc7bc13fbb0 2024-01-11T08:26:32.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.197+0000 7fc7c29ff700 1 -- 172.21.15.27:0/1867796308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7bc04c3c0 con 0x7fc7bc13fbb0 2024-01-11T08:26:32.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.199+0000 7fc7a3fff700 1 -- 172.21.15.27:0/1867796308 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc7b0013470 con 0x7fc7bc13fbb0 2024-01-11T08:26:32.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.200+0000 7fc7a3fff700 1 --2- 172.21.15.27:0/1867796308 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc7a8066f30 0x7fc7a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:32.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.200+0000 7fc7a3fff700 1 -- 172.21.15.27:0/1867796308 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5483+0+0 (secure 0 0 0) 0x7fc7b00846e0 con 0x7fc7bc13fbb0 2024-01-11T08:26:32.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.200+0000 7fc7bbfff700 1 --2- 172.21.15.27:0/1867796308 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc7a8066f30 0x7fc7a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:32.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.201+0000 7fc7bbfff700 1 --2- 172.21.15.27:0/1867796308 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc7a8066f30 0x7fc7a80693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009260 tx=0x7fc7ac016000 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:32.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.204+0000 7fc7a3fff700 1 -- 172.21.15.27:0/1867796308 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc7b0051990 con 0x7fc7bc13fbb0 2024-01-11T08:26:32.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.409+0000 7fc7c29ff700 1 -- 172.21.15.27:0/1867796308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc7bc0cb800 con 0x7fc7bc13fbb0 2024-01-11T08:26:32.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.413+0000 7fc7a3fff700 1 -- 172.21.15.27:0/1867796308 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc7b0054fb0 con 0x7fc7bc13fbb0 2024-01-11T08:26:32.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.416+0000 7fc7c29ff700 1 -- 172.21.15.27:0/1867796308 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc7a8066f30 msgr2=0x7fc7a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.416+0000 7fc7c29ff700 1 --2- 172.21.15.27:0/1867796308 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc7a8066f30 0x7fc7a80693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009260 tx=0x7fc7ac016000 comp rx=0 tx=0).stop 2024-01-11T08:26:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.416+0000 7fc7c29ff700 1 -- 172.21.15.27:0/1867796308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc13fbb0 msgr2=0x7fc7bc155df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.416+0000 7fc7c29ff700 1 --2- 172.21.15.27:0/1867796308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc13fbb0 0x7fc7bc155df0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc7b000e930 tx=0x7fc7b00091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.416+0000 7fc7c29ff700 1 -- 172.21.15.27:0/1867796308 shutdown_connections 2024-01-11T08:26:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.416+0000 7fc7c29ff700 1 --2- 172.21.15.27:0/1867796308 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc7a8066f30 0x7fc7a80693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.416+0000 7fc7c29ff700 1 --2- 172.21.15.27:0/1867796308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7bc13ff80 0x7fc7bc0af950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.416+0000 7fc7c29ff700 1 --2- 172.21.15.27:0/1867796308 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7bc0cace0 0x7fc7bc0b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.416+0000 7fc7c29ff700 1 --2- 172.21.15.27:0/1867796308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7bc13fbb0 0x7fc7bc155df0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.416+0000 7fc7c29ff700 1 -- 172.21.15.27:0/1867796308 >> 172.21.15.27:0/1867796308 conn(0x7fc7bc0680a0 msgr2=0x7fc7bc13dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.416+0000 7fc7c29ff700 1 -- 172.21.15.27:0/1867796308 shutdown_connections 2024-01-11T08:26:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.416+0000 7fc7c29ff700 1 -- 172.21.15.27:0/1867796308 wait complete. 2024-01-11T08:26:32.433 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T08:26:32.433 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:26:32.536 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:26:32.630 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T08:26:32.630 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-11T08:26:32.631 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:26:32.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.830+0000 7fa05edff700 1 Processor -- start 2024-01-11T08:26:32.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.831+0000 7fa05edff700 1 -- start start 2024-01-11T08:26:32.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.832+0000 7fa05edff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0581575e0 0x7fa058158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:32.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.832+0000 7fa05edff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa0580b19d0 0x7fa058159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:32.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.832+0000 7fa05edff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa058158b60 con 0x7fa05806b610 2024-01-11T08:26:32.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.832+0000 7fa05edff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0580cb5e0 con 0x7fa0580b19d0 2024-01-11T08:26:32.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.832+0000 7fa05edff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0581599b0 con 0x7fa0581575e0 2024-01-11T08:26:32.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.832+0000 7fa05cb9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0581575e0 0x7fa058158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:32.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.833+0000 7fa05d39c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa0580b19d0 0x7fa058159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:32.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.833+0000 7fa057fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa05806b610 0x7fa0580b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:34128/0 (socket says 172.21.15.27:34128) 2024-01-11T08:26:32.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.833+0000 7fa05cb9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0581575e0 0x7fa058158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:37832/0 (socket says 172.21.15.27:37832) 2024-01-11T08:26:32.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.833+0000 7fa057fff700 1 -- 172.21.15.27:0/3384987264 learned_addr learned my addr 172.21.15.27:0/3384987264 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:32.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.833+0000 7fa05cb9b700 1 -- 172.21.15.27:0/3384987264 >> v2:172.21.15.77:3300/0 conn(0x7fa0580b19d0 msgr2=0x7fa058159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:32.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.833+0000 7fa05cb9b700 1 --2- 172.21.15.27:0/3384987264 >> v2:172.21.15.77:3300/0 conn(0x7fa0580b19d0 0x7fa058159e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:32.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.833+0000 7fa05cb9b700 1 -- 172.21.15.27:0/3384987264 >> v1:172.21.15.77:6789/0 conn(0x7fa05806b610 legacy=0x7fa0580b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:32.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.833+0000 7fa05cb9b700 1 -- 172.21.15.27:0/3384987264 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0580c0f90 con 0x7fa0581575e0 2024-01-11T08:26:32.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.833+0000 7fa05d39c700 1 --2- 172.21.15.27:0/3384987264 >> v2:172.21.15.77:3300/0 conn(0x7fa0580b19d0 0x7fa058159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:32.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.833+0000 7fa0577fe700 1 -- 172.21.15.27:0/3384987264 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273803868 0 0) 0x7fa044001110 con 0x7fa05806b610 2024-01-11T08:26:32.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.833+0000 7fa05cb9b700 1 --2- 172.21.15.27:0/3384987264 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0581575e0 0x7fa058158620 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0098e0 tx=0x7fa04c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=693d5e11d2e1d9c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:32.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.834+0000 7fa0577fe700 1 -- 172.21.15.27:0/3384987264 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c00b640 con 0x7fa0581575e0 2024-01-11T08:26:32.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.834+0000 7fa0577fe700 1 -- 172.21.15.27:0/3384987264 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04c00b7a0 con 0x7fa0581575e0 2024-01-11T08:26:32.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.834+0000 7fa0577fe700 1 -- 172.21.15.27:0/3384987264 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c00bb40 con 0x7fa0581575e0 2024-01-11T08:26:32.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.834+0000 7fa05edff700 1 -- 172.21.15.27:0/3384987264 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0581575e0 msgr2=0x7fa058158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:32.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.834+0000 7fa05edff700 1 --2- 172.21.15.27:0/3384987264 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0581575e0 0x7fa058158620 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0098e0 tx=0x7fa04c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:32.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.834+0000 7fa05edff700 1 -- 172.21.15.27:0/3384987264 shutdown_connections 2024-01-11T08:26:32.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.834+0000 7fa05edff700 1 --2- 172.21.15.27:0/3384987264 >> v2:172.21.15.77:3300/0 conn(0x7fa0580b19d0 0x7fa058159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:32.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.834+0000 7fa05edff700 1 --2- 172.21.15.27:0/3384987264 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0581575e0 0x7fa058158620 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:32.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.834+0000 7fa05edff700 1 -- 172.21.15.27:0/3384987264 >> 172.21.15.27:0/3384987264 conn(0x7fa0580680e0 msgr2=0x7fa058069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:32.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.834+0000 7fa05edff700 1 -- 172.21.15.27:0/3384987264 shutdown_connections 2024-01-11T08:26:32.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.834+0000 7fa05edff700 1 -- 172.21.15.27:0/3384987264 wait complete. 2024-01-11T08:26:32.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.836+0000 7fa05edff700 1 Processor -- start 2024-01-11T08:26:32.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.836+0000 7fa05edff700 1 -- start start 2024-01-11T08:26:32.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.837+0000 7fa05edff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa05806b610 0x7fa058150e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:32.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.837+0000 7fa05edff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0580b19d0 0x7fa058144da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:32.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.837+0000 7fa05edff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0581575e0 0x7fa0580c2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:32.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.837+0000 7fa05edff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0580c2d80 con 0x7fa0580b19d0 2024-01-11T08:26:32.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.837+0000 7fa05edff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05806fb90 con 0x7fa05806b610 2024-01-11T08:26:32.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.837+0000 7fa05edff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0580c2590 con 0x7fa0581575e0 2024-01-11T08:26:32.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.837+0000 7fa057fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0580b19d0 0x7fa058144da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:32.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.837+0000 7fa057fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0580b19d0 0x7fa058144da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54116/0 (socket says 172.21.15.27:54116) 2024-01-11T08:26:32.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.837+0000 7fa057fff700 1 -- 172.21.15.27:0/372674604 learned_addr learned my addr 172.21.15.27:0/372674604 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:32.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.837+0000 7fa05d39c700 1 --2- 172.21.15.27:0/372674604 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0581575e0 0x7fa0580c2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:32.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.837+0000 7fa05cb9b700 1 --2- 172.21.15.27:0/372674604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa05806b610 0x7fa058150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:32.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.838+0000 7fa057fff700 1 -- 172.21.15.27:0/372674604 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0581575e0 msgr2=0x7fa0580c2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:32.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.838+0000 7fa057fff700 1 --2- 172.21.15.27:0/372674604 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0581575e0 0x7fa0580c2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:32.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.838+0000 7fa057fff700 1 -- 172.21.15.27:0/372674604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa05806b610 msgr2=0x7fa058150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:32.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.838+0000 7fa057fff700 1 --2- 172.21.15.27:0/372674604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa05806b610 0x7fa058150e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.838+0000 7fa057fff700 1 -- 172.21.15.27:0/372674604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa050009a20 con 0x7fa0580b19d0 2024-01-11T08:26:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.838+0000 7fa05d39c700 1 --2- 172.21.15.27:0/372674604 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0581575e0 0x7fa0580c2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.838+0000 7fa057fff700 1 --2- 172.21.15.27:0/372674604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0580b19d0 0x7fa058144da0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fa04400c840 tx=0x7fa044007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.838+0000 7fa054ff9700 1 -- 172.21.15.27:0/372674604 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa044005b30 con 0x7fa0580b19d0 2024-01-11T08:26:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.838+0000 7fa054ff9700 1 -- 172.21.15.27:0/372674604 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa044005c90 con 0x7fa0580b19d0 2024-01-11T08:26:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.838+0000 7fa054ff9700 1 -- 172.21.15.27:0/372674604 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa044007300 con 0x7fa0580b19d0 2024-01-11T08:26:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.838+0000 7fa05edff700 1 -- 172.21.15.27:0/372674604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04c009420 con 0x7fa0580b19d0 2024-01-11T08:26:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.839+0000 7fa05edff700 1 -- 172.21.15.27:0/372674604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa05806db00 con 0x7fa0580b19d0 2024-01-11T08:26:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.839+0000 7fa05edff700 1 -- 172.21.15.27:0/372674604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0580718d0 con 0x7fa0580b19d0 2024-01-11T08:26:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.840+0000 7fa054ff9700 1 -- 172.21.15.27:0/372674604 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa04401b3b0 con 0x7fa0580b19d0 2024-01-11T08:26:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.841+0000 7fa054ff9700 1 --2- 172.21.15.27:0/372674604 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa03c066f30 0x7fa03c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.841+0000 7fa054ff9700 1 -- 172.21.15.27:0/372674604 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5483+0+0 (secure 0 0 0) 0x7fa044083770 con 0x7fa0580b19d0 2024-01-11T08:26:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.842+0000 7fa05cb9b700 1 --2- 172.21.15.27:0/372674604 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa03c066f30 0x7fa03c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:32.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.843+0000 7fa05cb9b700 1 --2- 172.21.15.27:0/372674604 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa03c066f30 0x7fa03c0693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0093f0 tx=0x7fa04c017000 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:32.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:32.846+0000 7fa054ff9700 1 -- 172.21.15.27:0/372674604 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa044050aa0 con 0x7fa0580b19d0 2024-01-11T08:26:33.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.094+0000 7fa05edff700 1 -- 172.21.15.27:0/372674604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa058158b60 con 0x7fa0580b19d0 2024-01-11T08:26:33.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.095+0000 7fa054ff9700 1 -- 172.21.15.27:0/372674604 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+3526 (secure 0 0 0) 0x7fa058158b60 con 0x7fa0580b19d0 2024-01-11T08:26:33.096 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:33.096 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":44,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5945,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1366930493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1366930493},{"type":"v1","addr":"172.21.15.77:6837","nonce":1366930493}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/516828593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":516828593},{"type":"v1","addr":"172.21.15.27:6835","nonce":516828593}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6173,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1860074671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1860074671},{"type":"v1","addr":"172.21.15.77:6834","nonce":1860074671}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":6624,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1462523541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1462523541},{"type":"v1","addr":"172.21.15.27:6834","nonce":1462523541}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[]} 2024-01-11T08:26:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.097+0000 7fa05edff700 1 -- 172.21.15.27:0/372674604 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa03c066f30 msgr2=0x7fa03c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.097+0000 7fa05edff700 1 --2- 172.21.15.27:0/372674604 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa03c066f30 0x7fa03c0693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0093f0 tx=0x7fa04c017000 comp rx=0 tx=0).stop 2024-01-11T08:26:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.097+0000 7fa05edff700 1 -- 172.21.15.27:0/372674604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0580b19d0 msgr2=0x7fa058144da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.097+0000 7fa05edff700 1 --2- 172.21.15.27:0/372674604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0580b19d0 0x7fa058144da0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fa04400c840 tx=0x7fa044007c50 comp rx=0 tx=0).stop 2024-01-11T08:26:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.097+0000 7fa05edff700 1 -- 172.21.15.27:0/372674604 shutdown_connections 2024-01-11T08:26:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.097+0000 7fa05edff700 1 --2- 172.21.15.27:0/372674604 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa03c066f30 0x7fa03c0693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.097+0000 7fa05edff700 1 --2- 172.21.15.27:0/372674604 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0581575e0 0x7fa0580c2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.097+0000 7fa05edff700 1 --2- 172.21.15.27:0/372674604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa05806b610 0x7fa058150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.097+0000 7fa05edff700 1 --2- 172.21.15.27:0/372674604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0580b19d0 0x7fa058144da0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.097+0000 7fa05edff700 1 -- 172.21.15.27:0/372674604 >> 172.21.15.27:0/372674604 conn(0x7fa0580680e0 msgr2=0x7fa058152bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.097+0000 7fa05edff700 1 -- 172.21.15.27:0/372674604 shutdown_connections 2024-01-11T08:26:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.097+0000 7fa05edff700 1 -- 172.21.15.27:0/372674604 wait complete. 2024-01-11T08:26:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 44 2024-01-11T08:26:33.114 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-11T08:26:33.114 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-11T08:26:33.114 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-11T08:26:33.114 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:26:33.115 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-11T08:26:33.115 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-11T08:26:33.115 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-11T08:26:33.115 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:26:33.115 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-11T08:26:33.115 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-11T08:26:33.115 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-11T08:26:33.115 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:26:33.116 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-11T08:26:33.116 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-11T08:26:33.116 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-11T08:26:33.116 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:26:33.128 INFO:tasks.ceph.mds.a:Stopped 2024-01-11T08:26:33.128 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-11T08:26:33.129 INFO:tasks.ceph.mds.c:Stopped 2024-01-11T08:26:33.129 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-11T08:26:33.131 INFO:tasks.ceph.mds.b:Stopped 2024-01-11T08:26:33.131 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-11T08:26:33.132 INFO:tasks.ceph.mds.d:Stopped 2024-01-11T08:26:33.133 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-11T08:26:33.172 INFO:tasks.ceph.mds.a:Started 2024-01-11T08:26:33.172 DEBUG:teuthology.parallel:result is None 2024-01-11T08:26:33.175 INFO:tasks.ceph.mds.c:Started 2024-01-11T08:26:33.175 DEBUG:teuthology.parallel:result is None 2024-01-11T08:26:33.176 INFO:tasks.ceph.mds.b:Started 2024-01-11T08:26:33.177 DEBUG:teuthology.parallel:result is None 2024-01-11T08:26:33.180 INFO:tasks.ceph.mds.d:Started 2024-01-11T08:26:33.180 DEBUG:teuthology.parallel:result is None 2024-01-11T08:26:33.181 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-11T08:26:33.322 INFO:tasks.ceph.mds.b.smithi077.stdout:starting mds.b at 2024-01-11T08:26:33.322 INFO:tasks.ceph.mds.d.smithi077.stdout:starting mds.d at 2024-01-11T08:26:33.325 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T08:26:33.325 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T08:26:33.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.399+0000 7fba043ff700 1 Processor -- start 2024-01-11T08:26:33.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.399+0000 7fba043ff700 1 -- start start 2024-01-11T08:26:33.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.400+0000 7fba043ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb9fc157730 0x7fb9fc06f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:33.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.400+0000 7fba043ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9fc0cb540 con 0x7fb9fc157360 2024-01-11T08:26:33.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.400+0000 7fba043ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9fc0ca960 con 0x7fb9fc0b19a0 2024-01-11T08:26:33.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.400+0000 7fba043ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc0cac10 con 0x7fb9fc157730 2024-01-11T08:26:33.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.401+0000 7fba0219b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb9fc157360 0x7fb9fc0b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56156/0 (socket says 172.21.15.27:56156) 2024-01-11T08:26:33.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.401+0000 7fba0219b700 1 -- 172.21.15.27:0/4047432427 learned_addr learned my addr 172.21.15.27:0/4047432427 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:33.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.401+0000 7fba0299c700 1 --2- 172.21.15.27:0/4047432427 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb9fc157730 0x7fb9fc06f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:33.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.401+0000 7fba01199700 1 -- 172.21.15.27:0/4047432427 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385305413 0 0) 0x7fb9fc0cb540 con 0x7fb9fc157360 2024-01-11T08:26:33.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.401+0000 7fba01199700 1 -- 172.21.15.27:0/4047432427 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9e8003570 con 0x7fb9fc157360 2024-01-11T08:26:33.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.401+0000 7fba01199700 1 -- 172.21.15.27:0/4047432427 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343386186 0 0) 0x7fb9fc0ca960 con 0x7fb9fc0b19a0 2024-01-11T08:26:33.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.401+0000 7fba01199700 1 -- 172.21.15.27:0/4047432427 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9fc0cb540 con 0x7fb9fc0b19a0 2024-01-11T08:26:33.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.401+0000 7fba0299c700 1 -- 172.21.15.27:0/4047432427 >> v1:172.21.15.77:6789/0 conn(0x7fb9fc0b19a0 legacy=0x7fb9fc0abb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:33.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.401+0000 7fba0299c700 1 -- 172.21.15.27:0/4047432427 >> v1:172.21.15.27:6789/0 conn(0x7fb9fc157360 legacy=0x7fb9fc0b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:33.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.401+0000 7fba0299c700 1 -- 172.21.15.27:0/4047432427 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9fc0caef0 con 0x7fb9fc157730 2024-01-11T08:26:33.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.402+0000 7fba0299c700 1 --2- 172.21.15.27:0/4047432427 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb9fc157730 0x7fb9fc06f570 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4009e90 tx=0x7fb9f4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dc01bcc67c41bf8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:33.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.402+0000 7fba01199700 1 -- 172.21.15.27:0/4047432427 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f400e070 con 0x7fb9fc157730 2024-01-11T08:26:33.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.402+0000 7fba01199700 1 -- 172.21.15.27:0/4047432427 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f400bb80 con 0x7fb9fc157730 2024-01-11T08:26:33.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.402+0000 7fba01199700 1 -- 172.21.15.27:0/4047432427 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4012430 con 0x7fb9fc157730 2024-01-11T08:26:33.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.403+0000 7fba043ff700 1 -- 172.21.15.27:0/4047432427 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb9fc157730 msgr2=0x7fb9fc06f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:33.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.403+0000 7fba043ff700 1 --2- 172.21.15.27:0/4047432427 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb9fc157730 0x7fb9fc06f570 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4009e90 tx=0x7fb9f4009780 comp rx=0 tx=0).stop 2024-01-11T08:26:33.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.403+0000 7fba043ff700 1 -- 172.21.15.27:0/4047432427 shutdown_connections 2024-01-11T08:26:33.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.403+0000 7fba043ff700 1 --2- 172.21.15.27:0/4047432427 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb9fc157730 0x7fb9fc06f570 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:33.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.403+0000 7fba043ff700 1 -- 172.21.15.27:0/4047432427 >> 172.21.15.27:0/4047432427 conn(0x7fb9fc0680b0 msgr2=0x7fb9fc0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:33.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.403+0000 7fba043ff700 1 -- 172.21.15.27:0/4047432427 shutdown_connections 2024-01-11T08:26:33.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.403+0000 7fba043ff700 1 -- 172.21.15.27:0/4047432427 wait complete. 2024-01-11T08:26:33.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.404+0000 7fba043ff700 1 Processor -- start 2024-01-11T08:26:33.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.404+0000 7fba043ff700 1 -- start start 2024-01-11T08:26:33.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.405+0000 7fba043ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9fc0b19a0 0x7fb9fc150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:33.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.405+0000 7fba043ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb9fc157360 0x7fb9fc154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:33.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.405+0000 7fba043ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9fc157730 0x7fb9fc0aaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:33.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.405+0000 7fba043ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc0b03f0 con 0x7fb9fc0b19a0 2024-01-11T08:26:33.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.405+0000 7fba043ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc1514b0 con 0x7fb9fc157730 2024-01-11T08:26:33.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.405+0000 7fba043ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc1546d0 con 0x7fb9fc157360 2024-01-11T08:26:33.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.406+0000 7fba0219b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9fc0b19a0 0x7fb9fc150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:33.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.406+0000 7fba0219b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9fc0b19a0 0x7fb9fc150f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54178/0 (socket says 172.21.15.27:54178) 2024-01-11T08:26:33.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.406+0000 7fba0219b700 1 -- 172.21.15.27:0/2240982969 learned_addr learned my addr 172.21.15.27:0/2240982969 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:33.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.406+0000 7fba0219b700 1 -- 172.21.15.27:0/2240982969 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb9fc157360 msgr2=0x7fb9fc154190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:33.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.406+0000 7fba0219b700 1 --2- 172.21.15.27:0/2240982969 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb9fc157360 0x7fb9fc154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:33.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.406+0000 7fba0219b700 1 -- 172.21.15.27:0/2240982969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9fc157730 msgr2=0x7fb9fc0aaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:33.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.406+0000 7fba0219b700 1 --2- 172.21.15.27:0/2240982969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9fc157730 0x7fb9fc0aaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:33.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.406+0000 7fba0219b700 1 -- 172.21.15.27:0/2240982969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f4009a20 con 0x7fb9fc0b19a0 2024-01-11T08:26:33.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.406+0000 7fba0219b700 1 --2- 172.21.15.27:0/2240982969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9fc0b19a0 0x7fb9fc150f70 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800c7c0 tx=0x7fb9f8007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:33.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.407+0000 7fb9f27fc700 1 -- 172.21.15.27:0/2240982969 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8005b30 con 0x7fb9fc0b19a0 2024-01-11T08:26:33.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.407+0000 7fb9f27fc700 1 -- 172.21.15.27:0/2240982969 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f8005c90 con 0x7fb9fc0b19a0 2024-01-11T08:26:33.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.407+0000 7fba043ff700 1 -- 172.21.15.27:0/2240982969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9fc0af7e0 con 0x7fb9fc0b19a0 2024-01-11T08:26:33.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.407+0000 7fb9f27fc700 1 -- 172.21.15.27:0/2240982969 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8007300 con 0x7fb9fc0b19a0 2024-01-11T08:26:33.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.407+0000 7fba043ff700 1 -- 172.21.15.27:0/2240982969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9fc0afe20 con 0x7fb9fc0b19a0 2024-01-11T08:26:33.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.408+0000 7fba043ff700 1 -- 172.21.15.27:0/2240982969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9fc070ce0 con 0x7fb9fc0b19a0 2024-01-11T08:26:33.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.409+0000 7fb9f27fc700 1 -- 172.21.15.27:0/2240982969 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb9f801b400 con 0x7fb9fc0b19a0 2024-01-11T08:26:33.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.410+0000 7fb9f27fc700 1 --2- 172.21.15.27:0/2240982969 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb9e0066f30 0x7fb9e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:33.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.410+0000 7fb9f27fc700 1 -- 172.21.15.27:0/2240982969 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5483+0+0 (secure 0 0 0) 0x7fb9f8083fb0 con 0x7fb9fc0b19a0 2024-01-11T08:26:33.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.410+0000 7fba0199a700 1 --2- 172.21.15.27:0/2240982969 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb9e0066f30 0x7fb9e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:33.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.411+0000 7fba0199a700 1 --2- 172.21.15.27:0/2240982969 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb9e0066f30 0x7fb9e00693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc0ca8a0 tx=0x7fb9ec0079d0 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:33.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.414+0000 7fb9f27fc700 1 -- 172.21.15.27:0/2240982969 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb9fc070ce0 con 0x7fb9fc0b19a0 2024-01-11T08:26:33.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.631+0000 7fba043ff700 1 -- 172.21.15.27:0/2240982969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb9fc070ce0 con 0x7fb9fc0b19a0 2024-01-11T08:26:33.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.978+0000 7fb9f27fc700 1 -- 172.21.15.27:0/2240982969 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v95) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb9fc070ce0 con 0x7fb9fc0b19a0 2024-01-11T08:26:33.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.981+0000 7fba043ff700 1 -- 172.21.15.27:0/2240982969 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb9e0066f30 msgr2=0x7fb9e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:33.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.981+0000 7fba043ff700 1 --2- 172.21.15.27:0/2240982969 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb9e0066f30 0x7fb9e00693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc0ca8a0 tx=0x7fb9ec0079d0 comp rx=0 tx=0).stop 2024-01-11T08:26:33.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.981+0000 7fba043ff700 1 -- 172.21.15.27:0/2240982969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9fc0b19a0 msgr2=0x7fb9fc150f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:33.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.981+0000 7fba043ff700 1 --2- 172.21.15.27:0/2240982969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9fc0b19a0 0x7fb9fc150f70 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800c7c0 tx=0x7fb9f8007ea0 comp rx=0 tx=0).stop 2024-01-11T08:26:33.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.981+0000 7fba043ff700 1 -- 172.21.15.27:0/2240982969 shutdown_connections 2024-01-11T08:26:33.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.981+0000 7fba043ff700 1 --2- 172.21.15.27:0/2240982969 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb9e0066f30 0x7fb9e00693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:33.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.981+0000 7fba043ff700 1 --2- 172.21.15.27:0/2240982969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9fc157730 0x7fb9fc0aaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:33.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.981+0000 7fba043ff700 1 --2- 172.21.15.27:0/2240982969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9fc0b19a0 0x7fb9fc150f70 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:33.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.981+0000 7fba043ff700 1 --2- 172.21.15.27:0/2240982969 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb9fc157360 0x7fb9fc154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:33.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.981+0000 7fba043ff700 1 -- 172.21.15.27:0/2240982969 >> 172.21.15.27:0/2240982969 conn(0x7fb9fc0680b0 msgr2=0x7fb9fc1564a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:33.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.981+0000 7fba043ff700 1 -- 172.21.15.27:0/2240982969 shutdown_connections 2024-01-11T08:26:33.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:33.982+0000 7fba043ff700 1 -- 172.21.15.27:0/2240982969 wait complete. 2024-01-11T08:26:33.982 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-11T08:26:34.000 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-11T08:26:34.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.201+0000 7efdc93ff700 1 Processor -- start 2024-01-11T08:26:34.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.201+0000 7efdc93ff700 1 -- start start 2024-01-11T08:26:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.202+0000 7efdc93ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efdc40ca3e0 0x7efdc40cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.202+0000 7efdc93ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdc40ca7b0 0x7efdc4159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.202+0000 7efdc93ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc415d640 con 0x7efdc40c8a20 2024-01-11T08:26:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.202+0000 7efdc93ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc415ae70 con 0x7efdc40ca3e0 2024-01-11T08:26:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.202+0000 7efdc93ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc415a3c0 con 0x7efdc40ca7b0 2024-01-11T08:26:34.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.202+0000 7efdc37fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efdc40c8a20 0x7efdc415dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47390/0 (socket says 172.21.15.27:47390) 2024-01-11T08:26:34.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.202+0000 7efdc37fe700 1 -- 172.21.15.27:0/3409877599 learned_addr learned my addr 172.21.15.27:0/3409877599 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.203+0000 7efdc27fc700 1 --2- 172.21.15.27:0/3409877599 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdc40ca7b0 0x7efdc4159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.203+0000 7efdc2ffd700 1 --2- 172.21.15.27:0/3409877599 >> v2:172.21.15.77:3300/0 conn(0x7efdc40ca3e0 0x7efdc40cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.203+0000 7efdc1ffb700 1 -- 172.21.15.27:0/3409877599 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101349934 0 0) 0x7efdc415d640 con 0x7efdc40c8a20 2024-01-11T08:26:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.203+0000 7efdc1ffb700 1 -- 172.21.15.27:0/3409877599 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdac003570 con 0x7efdc40c8a20 2024-01-11T08:26:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.203+0000 7efdc1ffb700 1 -- 172.21.15.27:0/3409877599 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1208616811 0 0) 0x7efdac003570 con 0x7efdc40c8a20 2024-01-11T08:26:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.203+0000 7efdc1ffb700 1 -- 172.21.15.27:0/3409877599 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdc40ca7b0 msgr2=0x7efdc4159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.203+0000 7efdc1ffb700 1 --2- 172.21.15.27:0/3409877599 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdc40ca7b0 0x7efdc4159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.203+0000 7efdc1ffb700 1 -- 172.21.15.27:0/3409877599 >> v2:172.21.15.77:3300/0 conn(0x7efdc40ca3e0 msgr2=0x7efdc40cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.203+0000 7efdc1ffb700 1 --2- 172.21.15.27:0/3409877599 >> v2:172.21.15.77:3300/0 conn(0x7efdc40ca3e0 0x7efdc40cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.203+0000 7efdc1ffb700 1 -- 172.21.15.27:0/3409877599 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc415a5d0 con 0x7efdc40c8a20 2024-01-11T08:26:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.203+0000 7efdc1ffb700 1 -- 172.21.15.27:0/3409877599 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7efdbc0017f0 con 0x7efdc40c8a20 2024-01-11T08:26:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.203+0000 7efdc1ffb700 1 -- 172.21.15.27:0/3409877599 >> v1:172.21.15.27:6789/0 conn(0x7efdc40c8a20 legacy=0x7efdc415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.203+0000 7efdc1ffb700 1 --2- 172.21.15.27:0/3409877599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdac003ae0 0x7efdac003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.203+0000 7efdc1ffb700 1 --2- 172.21.15.27:0/3409877599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdac004810 0x7efdac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.203+0000 7efdc1ffb700 1 --2- 172.21.15.27:0/3409877599 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdc40ca7b0 0x7efdac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.203+0000 7efdc1ffb700 1 -- 172.21.15.27:0/3409877599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc415ae70 con 0x7efdac004810 2024-01-11T08:26:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.203+0000 7efdc1ffb700 1 -- 172.21.15.27:0/3409877599 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc415a3c0 con 0x7efdac003ae0 2024-01-11T08:26:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.203+0000 7efdc1ffb700 1 -- 172.21.15.27:0/3409877599 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdac00bb40 con 0x7efdc40ca7b0 2024-01-11T08:26:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.204+0000 7efdc27fc700 1 --2- 172.21.15.27:0/3409877599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdac004810 0x7efdac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.204+0000 7efdc2ffd700 1 --2- 172.21.15.27:0/3409877599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdac003ae0 0x7efdac003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.204+0000 7efdc37fe700 1 --2- 172.21.15.27:0/3409877599 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdc40ca7b0 0x7efdac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.204+0000 7efdc27fc700 1 -- 172.21.15.27:0/3409877599 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdc40ca7b0 msgr2=0x7efdac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.204+0000 7efdc27fc700 1 --2- 172.21.15.27:0/3409877599 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdc40ca7b0 0x7efdac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.204+0000 7efdc27fc700 1 -- 172.21.15.27:0/3409877599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdac003ae0 msgr2=0x7efdac003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.204+0000 7efdc27fc700 1 --2- 172.21.15.27:0/3409877599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdac003ae0 0x7efdac003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.204+0000 7efdc27fc700 1 -- 172.21.15.27:0/3409877599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdac003570 con 0x7efdac004810 2024-01-11T08:26:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.204+0000 7efdc27fc700 1 --2- 172.21.15.27:0/3409877599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdac004810 0x7efdac004c80 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7efdb8009c00 tx=0x7efdb80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2827033d11efd31e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.204+0000 7efdc1ffb700 1 -- 172.21.15.27:0/3409877599 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb800e710 con 0x7efdac004810 2024-01-11T08:26:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.204+0000 7efdc1ffb700 1 -- 172.21.15.27:0/3409877599 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb800e870 con 0x7efdac004810 2024-01-11T08:26:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.204+0000 7efdc1ffb700 1 -- 172.21.15.27:0/3409877599 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb800ec30 con 0x7efdac004810 2024-01-11T08:26:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.205+0000 7efdc93ff700 1 -- 172.21.15.27:0/3409877599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdac004810 msgr2=0x7efdac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.205+0000 7efdc93ff700 1 --2- 172.21.15.27:0/3409877599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdac004810 0x7efdac004c80 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7efdb8009c00 tx=0x7efdb80091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.205+0000 7efdc37fe700 1 -- 172.21.15.27:0/3409877599 reap_dead start 2024-01-11T08:26:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.205+0000 7efdc93ff700 1 -- 172.21.15.27:0/3409877599 shutdown_connections 2024-01-11T08:26:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.205+0000 7efdc93ff700 1 -- 172.21.15.27:0/3409877599 >> 172.21.15.27:0/3409877599 conn(0x7efdc40ce3e0 msgr2=0x7efdc40ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.205+0000 7efdc93ff700 1 -- 172.21.15.27:0/3409877599 shutdown_connections 2024-01-11T08:26:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.205+0000 7efdc93ff700 1 -- 172.21.15.27:0/3409877599 wait complete. 2024-01-11T08:26:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.206+0000 7efdc93ff700 1 Processor -- start 2024-01-11T08:26:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.207+0000 7efdc93ff700 1 -- start start 2024-01-11T08:26:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.207+0000 7efdc93ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdac004810 0x7efdc41506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.207+0000 7efdc93ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc416dce0 0x7efdc40b2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.207+0000 7efdc93ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdc415bf50 0x7efdc415c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.208+0000 7efdc93ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc41518a0 con 0x7efdc415bf50 2024-01-11T08:26:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.208+0000 7efdc93ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc415c900 con 0x7efdc416dce0 2024-01-11T08:26:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.208+0000 7efdc93ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc41515c0 con 0x7efdac004810 2024-01-11T08:26:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.208+0000 7efdc37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdc415bf50 0x7efdc415c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.208+0000 7efdc2ffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdac004810 0x7efdc41506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.208+0000 7efdc37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdc415bf50 0x7efdc415c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41312/0 (socket says 172.21.15.27:41312) 2024-01-11T08:26:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.208+0000 7efdc37fe700 1 -- 172.21.15.27:0/3443489802 learned_addr learned my addr 172.21.15.27:0/3443489802 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.208+0000 7efdc2ffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdac004810 0x7efdc41506b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:60736/0 (socket says 172.21.15.27:60736) 2024-01-11T08:26:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.208+0000 7efdc27fc700 1 --2- 172.21.15.27:0/3443489802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc416dce0 0x7efdc40b2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.208+0000 7efdc37fe700 1 -- 172.21.15.27:0/3443489802 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdac004810 msgr2=0x7efdc41506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.208+0000 7efdc37fe700 1 --2- 172.21.15.27:0/3443489802 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdac004810 0x7efdc41506b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.208+0000 7efdc37fe700 1 -- 172.21.15.27:0/3443489802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc416dce0 msgr2=0x7efdc40b2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.208+0000 7efdc37fe700 1 --2- 172.21.15.27:0/3443489802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc416dce0 0x7efdc40b2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.208+0000 7efdc37fe700 1 -- 172.21.15.27:0/3443489802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc416e150 con 0x7efdc415bf50 2024-01-11T08:26:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.209+0000 7efdc37fe700 1 --2- 172.21.15.27:0/3443489802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdc415bf50 0x7efdc415c3c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7efdc406f370 tx=0x7efdbc001380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.209+0000 7efdaaffd700 1 -- 172.21.15.27:0/3443489802 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdbc00e900 con 0x7efdc415bf50 2024-01-11T08:26:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.209+0000 7efdaaffd700 1 -- 172.21.15.27:0/3443489802 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdbc00ea60 con 0x7efdc415bf50 2024-01-11T08:26:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.209+0000 7efdaaffd700 1 -- 172.21.15.27:0/3443489802 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdbc00ee20 con 0x7efdc415bf50 2024-01-11T08:26:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.209+0000 7efdc93ff700 1 -- 172.21.15.27:0/3443489802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc4150c90 con 0x7efdc415bf50 2024-01-11T08:26:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.209+0000 7efdc93ff700 1 -- 172.21.15.27:0/3443489802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc4151250 con 0x7efdc415bf50 2024-01-11T08:26:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.210+0000 7efdc93ff700 1 -- 172.21.15.27:0/3443489802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc4151700 con 0x7efdc415bf50 2024-01-11T08:26:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.211+0000 7efdaaffd700 1 -- 172.21.15.27:0/3443489802 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7efdbc013630 con 0x7efdc415bf50 2024-01-11T08:26:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.212+0000 7efdaaffd700 1 --2- 172.21.15.27:0/3443489802 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efdb0066f30 0x7efdb00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.212+0000 7efdaaffd700 1 -- 172.21.15.27:0/3443489802 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 4881+0+0 (secure 0 0 0) 0x7efdbc084680 con 0x7efdc415bf50 2024-01-11T08:26:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.213+0000 7efdc2ffd700 1 --2- 172.21.15.27:0/3443489802 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efdb0066f30 0x7efdb00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.213+0000 7efdc2ffd700 1 --2- 172.21.15.27:0/3443489802 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efdb0066f30 0x7efdb00693e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efdb4006f00 tx=0x7efdb40091a0 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:34.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.216+0000 7efdaaffd700 1 -- 172.21.15.27:0/3443489802 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efdc4151700 con 0x7efdc415bf50 2024-01-11T08:26:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.497+0000 7efdc93ff700 1 -- 172.21.15.27:0/3443489802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7efdc4151700 con 0x7efdc415bf50 2024-01-11T08:26:34.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.498+0000 7efdaaffd700 1 -- 172.21.15.27:0/3443489802 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7efdc4151700 con 0x7efdc415bf50 2024-01-11T08:26:34.497 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:34.497 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T08:26:34.497 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-11T08:26:34.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:34.497 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-11T08:26:34.497 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lQm3yLhAAv6erz5hMfat8EUzRA5luMA==", 2024-01-11T08:26:34.498 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:26:34.498 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:26:34.498 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:26:34.498 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:26:34.498 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:26:34.498 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:26:34.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:34.498 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-11T08:26:34.498 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9l+FG8ChAACHndtVLx7qS3sidPniSycA==", 2024-01-11T08:26:34.498 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:26:34.498 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:26:34.498 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:26:34.498 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:26:34.498 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:26:34.499 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:26:34.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:34.499 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-11T08:26:34.499 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l1NnEOhAALHXsc5es7hufCthGdRzkyw==", 2024-01-11T08:26:34.499 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:26:34.499 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:26:34.499 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:26:34.499 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:26:34.499 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:26:34.499 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:26:34.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:34.500 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-11T08:26:34.500 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lruoWFhAAm8AmrZtTJhYI1+19F79RPQ==", 2024-01-11T08:26:34.500 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:26:34.500 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:26:34.500 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:26:34.500 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:26:34.500 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:26:34.500 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:26:34.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:34.500 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-11T08:26:34.500 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA2op9ltZygORAAZr1X1nCN0ce8rIVaGDpFyQ==", 2024-01-11T08:26:34.500 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:26:34.500 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:26:34.501 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:26:34.501 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:26:34.501 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:26:34.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:34.501 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-11T08:26:34.501 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA6op9lqSRPAhAAmtyCsyK5ZVGX2wfhOCMuFQ==", 2024-01-11T08:26:34.501 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:26:34.501 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:26:34.501 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:26:34.501 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:26:34.501 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:26:34.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:34.502 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-11T08:26:34.502 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA9op9lZcs8BhAA8j+U+jPOoy+fliVWZAPFYg==", 2024-01-11T08:26:34.502 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:26:34.502 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:26:34.502 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:26:34.502 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:26:34.502 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:26:34.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:34.502 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-11T08:26:34.502 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBAop9lFuCUChAAP6si/2W1mxH4ecQJvuumkw==", 2024-01-11T08:26:34.502 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:26:34.502 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:26:34.502 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:26:34.503 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:26:34.503 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:26:34.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:34.503 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-11T08:26:34.503 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBGop9lOpwBIRAAcpcsjwmX2IldcKHJWAgHuA==", 2024-01-11T08:26:34.503 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:26:34.503 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:26:34.503 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:26:34.503 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:26:34.503 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:26:34.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:34.503 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-11T08:26:34.504 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBJop9l4q8hJRAAK8mAYVRQMoWS6wBOU69HmA==", 2024-01-11T08:26:34.504 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:26:34.504 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:26:34.504 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:26:34.504 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:26:34.504 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:26:34.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:34.504 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-11T08:26:34.504 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBMop9lqMF9KBAAFgFFI2GWt9AGif4AoScThw==", 2024-01-11T08:26:34.504 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:26:34.504 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:26:34.504 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:26:34.505 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:26:34.505 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:26:34.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:34.505 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-11T08:26:34.505 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBPop9lN7eDLBAAKswPZFfs0g2AaSe6V+pbbQ==", 2024-01-11T08:26:34.505 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:26:34.505 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:26:34.505 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:26:34.505 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:26:34.505 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:26:34.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:34.505 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-11T08:26:34.506 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lT5okIBAABy8PbJJbA062VNmb1Y8qhg==", 2024-01-11T08:26:34.506 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:26:34.506 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:26:34.506 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-01-11T08:26:34.506 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-11T08:26:34.506 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:26:34.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:34.506 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-11T08:26:34.506 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lDf37LhAAqV+EabKXJEW1sS5xtu1tXw==", 2024-01-11T08:26:34.506 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:26:34.506 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:26:34.506 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:26:34.506 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:26:34.506 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:26:34.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:26:34.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:34.507 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-11T08:26:34.507 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lWayxJxAAcalM8QSOGtbCXxjYZGaNQA==", 2024-01-11T08:26:34.507 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:26:34.507 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:26:34.507 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:26:34.507 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:26:34.507 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:26:34.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:26:34.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:34.507 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-11T08:26:34.507 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lHD4tNhAAdmnFBmMrK+QhQYdpbzW0mg==", 2024-01-11T08:26:34.507 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:26:34.508 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:26:34.508 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:26:34.508 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:26:34.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:26:34.508 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:26:34.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:34.508 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-11T08:26:34.508 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAwop9lGPtDMxAAJ8FjrSEdCtQL8AhtAyqnTw==", 2024-01-11T08:26:34.508 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:26:34.508 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:26:34.508 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:26:34.508 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:26:34.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:26:34.508 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:26:34.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:34.509 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-11T08:26:34.509 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lXBZLJxAAXa1S8Vkm3MiV4igtNbg/1Q==", 2024-01-11T08:26:34.509 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:26:34.509 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:26:34.509 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T08:26:34.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:26:34.509 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:26:34.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:34.509 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-11T08:26:34.509 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l3ZGoHxAANqzytTTvWYSomG+RFsFDyg==", 2024-01-11T08:26:34.509 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:26:34.509 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:26:34.509 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T08:26:34.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:26:34.510 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.510 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:34.510 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-11T08:26:34.510 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T08:26:34.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.500+0000 7efdc93ff700 1 -- 172.21.15.27:0/3443489802 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efdb0066f30 msgr2=0x7efdb00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:34.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.500+0000 7efdc93ff700 1 --2- 172.21.15.27:0/3443489802 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efdb0066f30 0x7efdb00693e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efdb4006f00 tx=0x7efdb40091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.500+0000 7efdc93ff700 1 -- 172.21.15.27:0/3443489802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdc415bf50 msgr2=0x7efdc415c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:34.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.500+0000 7efdc93ff700 1 --2- 172.21.15.27:0/3443489802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdc415bf50 0x7efdc415c3c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7efdc406f370 tx=0x7efdbc001380 comp rx=0 tx=0).stop 2024-01-11T08:26:34.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.500+0000 7efdc93ff700 1 -- 172.21.15.27:0/3443489802 shutdown_connections 2024-01-11T08:26:34.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.500+0000 7efdc93ff700 1 --2- 172.21.15.27:0/3443489802 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efdb0066f30 0x7efdb00693e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.500+0000 7efdc93ff700 1 --2- 172.21.15.27:0/3443489802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdc415bf50 0x7efdc415c3c0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.501+0000 7efdc93ff700 1 --2- 172.21.15.27:0/3443489802 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdac004810 0x7efdc41506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.501+0000 7efdc93ff700 1 --2- 172.21.15.27:0/3443489802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc416dce0 0x7efdc40b2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.501+0000 7efdc93ff700 1 -- 172.21.15.27:0/3443489802 >> 172.21.15.27:0/3443489802 conn(0x7efdc40ce3e0 msgr2=0x7efdc406a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:34.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.501+0000 7efdc93ff700 1 -- 172.21.15.27:0/3443489802 shutdown_connections 2024-01-11T08:26:34.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.501+0000 7efdc93ff700 1 -- 172.21.15.27:0/3443489802 wait complete. 2024-01-11T08:26:34.518 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:26:34.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.719+0000 7f1eb81ff700 1 Processor -- start 2024-01-11T08:26:34.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.719+0000 7f1eb81ff700 1 -- start start 2024-01-11T08:26:34.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.721+0000 7f1eb81ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1eb00b03e0 0x7f1eb00b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:34.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.721+0000 7f1eb81ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb00cd730 con 0x7f1eb00b07b0 2024-01-11T08:26:34.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.721+0000 7f1eb81ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb00b2de0 con 0x7f1eb00b29c0 2024-01-11T08:26:34.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.721+0000 7f1eb81ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb00ccb20 con 0x7f1eb00b03e0 2024-01-11T08:26:34.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.721+0000 7f1eb579a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1eb00b07b0 0x7f1eb0159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47396/0 (socket says 172.21.15.27:47396) 2024-01-11T08:26:34.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.721+0000 7f1eb579a700 1 -- 172.21.15.27:0/3724751874 learned_addr learned my addr 172.21.15.27:0/3724751874 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:34.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.721+0000 7f1eb5f9b700 1 --2- 172.21.15.27:0/3724751874 >> v2:172.21.15.77:3300/0 conn(0x7f1eb00b03e0 0x7f1eb00b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:34.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.721+0000 7f1eb4f99700 1 -- 172.21.15.27:0/3724751874 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931168919 0 0) 0x7f1eb00cd730 con 0x7f1eb00b07b0 2024-01-11T08:26:34.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.722+0000 7f1eb4f99700 1 -- 172.21.15.27:0/3724751874 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e9c003570 con 0x7f1eb00b07b0 2024-01-11T08:26:34.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.722+0000 7f1eb4f99700 1 -- 172.21.15.27:0/3724751874 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572810665 0 0) 0x7f1eb00b2de0 con 0x7f1eb00b29c0 2024-01-11T08:26:34.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.722+0000 7f1eb4f99700 1 -- 172.21.15.27:0/3724751874 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eb00cd730 con 0x7f1eb00b29c0 2024-01-11T08:26:34.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.722+0000 7f1eb5f9b700 1 -- 172.21.15.27:0/3724751874 >> v1:172.21.15.77:6789/0 conn(0x7f1eb00b29c0 legacy=0x7f1eb00aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:34.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.722+0000 7f1eb5f9b700 1 -- 172.21.15.27:0/3724751874 >> v1:172.21.15.27:6789/0 conn(0x7f1eb00b07b0 legacy=0x7f1eb0159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:34.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.722+0000 7f1eb5f9b700 1 -- 172.21.15.27:0/3724751874 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb00cce00 con 0x7f1eb00b03e0 2024-01-11T08:26:34.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.722+0000 7f1eb5f9b700 1 --2- 172.21.15.27:0/3724751874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1eb00b03e0 0x7f1eb00b2480 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f1eac0098e0 tx=0x7f1eac0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f7d559edd0cd791 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:34.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.722+0000 7f1eb4f99700 1 -- 172.21.15.27:0/3724751874 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac011630 con 0x7f1eb00b03e0 2024-01-11T08:26:34.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.722+0000 7f1eb4f99700 1 -- 172.21.15.27:0/3724751874 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac011790 con 0x7f1eb00b03e0 2024-01-11T08:26:34.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.722+0000 7f1eb4f99700 1 -- 172.21.15.27:0/3724751874 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac011b30 con 0x7f1eb00b03e0 2024-01-11T08:26:34.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.723+0000 7f1eb81ff700 1 -- 172.21.15.27:0/3724751874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1eb00b03e0 msgr2=0x7f1eb00b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:34.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.723+0000 7f1eb81ff700 1 --2- 172.21.15.27:0/3724751874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1eb00b03e0 0x7f1eb00b2480 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f1eac0098e0 tx=0x7f1eac0091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.723+0000 7f1eb81ff700 1 -- 172.21.15.27:0/3724751874 shutdown_connections 2024-01-11T08:26:34.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.723+0000 7f1eb81ff700 1 --2- 172.21.15.27:0/3724751874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1eb00b03e0 0x7f1eb00b2480 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.723+0000 7f1eb81ff700 1 -- 172.21.15.27:0/3724751874 >> 172.21.15.27:0/3724751874 conn(0x7f1eb006ed40 msgr2=0x7f1eb006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:34.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.723+0000 7f1eb81ff700 1 -- 172.21.15.27:0/3724751874 shutdown_connections 2024-01-11T08:26:34.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.723+0000 7f1eb81ff700 1 -- 172.21.15.27:0/3724751874 wait complete. 2024-01-11T08:26:34.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.725+0000 7f1eb81ff700 1 Processor -- start 2024-01-11T08:26:34.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.725+0000 7f1eb81ff700 1 -- start start 2024-01-11T08:26:34.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.726+0000 7f1eb81ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eb00b03e0 0x7f1eb014e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:34.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.726+0000 7f1eb81ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1eb00b07b0 0x7f1eb00c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:34.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.726+0000 7f1eb81ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1eb00b29c0 0x7f1eb00c6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:34.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.726+0000 7f1eb81ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb016d150 con 0x7f1eb00b03e0 2024-01-11T08:26:34.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.726+0000 7f1eb81ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb016df70 con 0x7f1eb00b07b0 2024-01-11T08:26:34.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.726+0000 7f1eb81ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb016d360 con 0x7f1eb00b29c0 2024-01-11T08:26:34.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.726+0000 7f1eb5f9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eb00b03e0 0x7f1eb014e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:34.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.726+0000 7f1eb5f9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eb00b03e0 0x7f1eb014e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41324/0 (socket says 172.21.15.27:41324) 2024-01-11T08:26:34.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.726+0000 7f1eb5f9b700 1 -- 172.21.15.27:0/1108625497 learned_addr learned my addr 172.21.15.27:0/1108625497 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:34.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.726+0000 7f1eb579a700 1 --2- 172.21.15.27:0/1108625497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1eb00b07b0 0x7f1eb00c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.726+0000 7f1eb5f9b700 1 -- 172.21.15.27:0/1108625497 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1eb00b29c0 msgr2=0x7f1eb00c6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.726+0000 7f1eb679c700 1 --2- 172.21.15.27:0/1108625497 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1eb00b29c0 0x7f1eb00c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.726+0000 7f1eb5f9b700 1 --2- 172.21.15.27:0/1108625497 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1eb00b29c0 0x7f1eb00c6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.726+0000 7f1eb5f9b700 1 -- 172.21.15.27:0/1108625497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1eb00b07b0 msgr2=0x7f1eb00c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.726+0000 7f1eb5f9b700 1 --2- 172.21.15.27:0/1108625497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1eb00b07b0 0x7f1eb00c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.726+0000 7f1eb5f9b700 1 -- 172.21.15.27:0/1108625497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eac009420 con 0x7f1eb00b03e0 2024-01-11T08:26:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.726+0000 7f1eb579a700 1 --2- 172.21.15.27:0/1108625497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1eb00b07b0 0x7f1eb00c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.726+0000 7f1eb679c700 1 --2- 172.21.15.27:0/1108625497 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1eb00b29c0 0x7f1eb00c6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.727+0000 7f1eb5f9b700 1 --2- 172.21.15.27:0/1108625497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eb00b03e0 0x7f1eb014e250 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0069540 tx=0x7f1eac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.727+0000 7f1ea5ffb700 1 -- 172.21.15.27:0/1108625497 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac011b10 con 0x7f1eb00b03e0 2024-01-11T08:26:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.727+0000 7f1ea5ffb700 1 -- 172.21.15.27:0/1108625497 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac000cc0 con 0x7f1eb00b03e0 2024-01-11T08:26:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.727+0000 7f1eb81ff700 1 -- 172.21.15.27:0/1108625497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb016d640 con 0x7f1eb00b03e0 2024-01-11T08:26:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.727+0000 7f1ea5ffb700 1 -- 172.21.15.27:0/1108625497 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac01a680 con 0x7f1eb00b03e0 2024-01-11T08:26:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.727+0000 7f1eb81ff700 1 -- 172.21.15.27:0/1108625497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb016dc00 con 0x7f1eb00b03e0 2024-01-11T08:26:34.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.728+0000 7f1eb81ff700 1 -- 172.21.15.27:0/1108625497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb00718d0 con 0x7f1eb00b03e0 2024-01-11T08:26:34.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.729+0000 7f1ea5ffb700 1 -- 172.21.15.27:0/1108625497 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1eac01a9f0 con 0x7f1eb00b03e0 2024-01-11T08:26:34.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.730+0000 7f1ea5ffb700 1 --2- 172.21.15.27:0/1108625497 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1e94066f30 0x7f1e940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:34.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.730+0000 7f1ea5ffb700 1 -- 172.21.15.27:0/1108625497 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 4881+0+0 (secure 0 0 0) 0x7f1eac085f90 con 0x7f1eb00b03e0 2024-01-11T08:26:34.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.730+0000 7f1eb579a700 1 --2- 172.21.15.27:0/1108625497 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1e94066f30 0x7f1e940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:34.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.731+0000 7f1eb579a700 1 --2- 172.21.15.27:0/1108625497 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1e94066f30 0x7f1e940693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0007fd0 tx=0x7f1ea0009040 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:34.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.734+0000 7f1ea5ffb700 1 -- 172.21.15.27:0/1108625497 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1eac053490 con 0x7f1eb00b03e0 2024-01-11T08:26:34.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.944+0000 7f1eb81ff700 1 -- 172.21.15.27:0/1108625497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1eb00cd0e0 con 0x7f1eb00b03e0 2024-01-11T08:26:34.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.945+0000 7f1ea5ffb700 1 -- 172.21.15.27:0/1108625497 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+12130 (secure 0 0 0) 0x7f1eac056ab0 con 0x7f1eb00b03e0 2024-01-11T08:26:34.944 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:34.944 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:26:33.984731+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:26:34.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.947+0000 7f1eb81ff700 1 -- 172.21.15.27:0/1108625497 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1e94066f30 msgr2=0x7f1e940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:34.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.947+0000 7f1eb81ff700 1 --2- 172.21.15.27:0/1108625497 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1e94066f30 0x7f1e940693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0007fd0 tx=0x7f1ea0009040 comp rx=0 tx=0).stop 2024-01-11T08:26:34.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.947+0000 7f1eb81ff700 1 -- 172.21.15.27:0/1108625497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eb00b03e0 msgr2=0x7f1eb014e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:34.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.947+0000 7f1eb81ff700 1 --2- 172.21.15.27:0/1108625497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eb00b03e0 0x7f1eb014e250 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0069540 tx=0x7f1eac017000 comp rx=0 tx=0).stop 2024-01-11T08:26:34.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.947+0000 7f1eb81ff700 1 -- 172.21.15.27:0/1108625497 shutdown_connections 2024-01-11T08:26:34.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.947+0000 7f1eb81ff700 1 --2- 172.21.15.27:0/1108625497 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1e94066f30 0x7f1e940693e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.947+0000 7f1eb81ff700 1 --2- 172.21.15.27:0/1108625497 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1eb00b29c0 0x7f1eb00c6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.947+0000 7f1eb81ff700 1 --2- 172.21.15.27:0/1108625497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eb00b03e0 0x7f1eb014e250 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.947+0000 7f1eb81ff700 1 --2- 172.21.15.27:0/1108625497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1eb00b07b0 0x7f1eb00c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:34.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.947+0000 7f1eb81ff700 1 -- 172.21.15.27:0/1108625497 >> 172.21.15.27:0/1108625497 conn(0x7f1eb006ed40 msgr2=0x7f1eb00ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:34.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.947+0000 7f1eb81ff700 1 -- 172.21.15.27:0/1108625497 shutdown_connections 2024-01-11T08:26:34.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:34.947+0000 7f1eb81ff700 1 -- 172.21.15.27:0/1108625497 wait complete. 2024-01-11T08:26:34.964 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-11T08:26:34.965 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:26:35.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.171+0000 7f8d791ff700 1 Processor -- start 2024-01-11T08:26:35.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.171+0000 7f8d791ff700 1 -- start start 2024-01-11T08:26:35.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.173+0000 7f8d791ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d74142dc0 0x7f8d740cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:35.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.173+0000 7f8d791ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d74143190 0x7f8d740cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:35.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.173+0000 7f8d791ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7415d320 con 0x7f8d740cc020 2024-01-11T08:26:35.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.173+0000 7f8d791ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d7415df30 con 0x7f8d74142dc0 2024-01-11T08:26:35.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.173+0000 7f8d791ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7415d590 con 0x7f8d74143190 2024-01-11T08:26:35.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.173+0000 7f8d72d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d74142dc0 0x7f8d740cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:35.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.173+0000 7f8d7359e700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8d740cc020 0x7f8d740cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47412/0 (socket says 172.21.15.27:47412) 2024-01-11T08:26:35.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.173+0000 7f8d7359e700 1 -- 172.21.15.27:0/2514250305 learned_addr learned my addr 172.21.15.27:0/2514250305 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:35.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.173+0000 7f8d7259c700 1 --2- 172.21.15.27:0/2514250305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d74143190 0x7f8d740cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:35.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.173+0000 7f8d72d9d700 1 -- 172.21.15.27:0/2514250305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d74143190 msgr2=0x7f8d740cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:35.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.173+0000 7f8d72d9d700 1 --2- 172.21.15.27:0/2514250305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d74143190 0x7f8d740cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:35.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.173+0000 7f8d72d9d700 1 -- 172.21.15.27:0/2514250305 >> v1:172.21.15.27:6789/0 conn(0x7f8d740cc020 legacy=0x7f8d740cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:35.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.174+0000 7f8d72d9d700 1 -- 172.21.15.27:0/2514250305 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7415d870 con 0x7f8d74142dc0 2024-01-11T08:26:35.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.173+0000 7f8d71d9b700 1 -- 172.21.15.27:0/2514250305 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146740175 0 0) 0x7f8d7415d320 con 0x7f8d740cc020 2024-01-11T08:26:35.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.174+0000 7f8d72d9d700 1 --2- 172.21.15.27:0/2514250305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d74142dc0 0x7f8d740cada0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f8d640098e0 tx=0x7f8d640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1693bdec20e50307 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:35.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.174+0000 7f8d71d9b700 1 -- 172.21.15.27:0/2514250305 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d64009e80 con 0x7f8d74142dc0 2024-01-11T08:26:35.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.174+0000 7f8d71d9b700 1 -- 172.21.15.27:0/2514250305 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6400b860 con 0x7f8d74142dc0 2024-01-11T08:26:35.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.174+0000 7f8d71d9b700 1 -- 172.21.15.27:0/2514250305 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6400bbe0 con 0x7f8d74142dc0 2024-01-11T08:26:35.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.174+0000 7f8d791ff700 1 -- 172.21.15.27:0/2514250305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d74142dc0 msgr2=0x7f8d740cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:35.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.174+0000 7f8d791ff700 1 --2- 172.21.15.27:0/2514250305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d74142dc0 0x7f8d740cada0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f8d640098e0 tx=0x7f8d640091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:35.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.175+0000 7f8d791ff700 1 -- 172.21.15.27:0/2514250305 shutdown_connections 2024-01-11T08:26:35.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.175+0000 7f8d791ff700 1 --2- 172.21.15.27:0/2514250305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d74142dc0 0x7f8d740cada0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:35.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.175+0000 7f8d791ff700 1 --2- 172.21.15.27:0/2514250305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d74143190 0x7f8d740cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:35.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.175+0000 7f8d791ff700 1 -- 172.21.15.27:0/2514250305 >> 172.21.15.27:0/2514250305 conn(0x7f8d74063ca0 msgr2=0x7f8d7406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:35.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.175+0000 7f8d791ff700 1 -- 172.21.15.27:0/2514250305 shutdown_connections 2024-01-11T08:26:35.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.175+0000 7f8d791ff700 1 -- 172.21.15.27:0/2514250305 wait complete. 2024-01-11T08:26:35.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.176+0000 7f8d791ff700 1 Processor -- start 2024-01-11T08:26:35.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.176+0000 7f8d791ff700 1 -- start start 2024-01-11T08:26:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.177+0000 7f8d791ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d740cc020 0x7f8d740c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.177+0000 7f8d791ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d74142dc0 0x7f8d741596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.177+0000 7f8d791ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d74143190 0x7f8d740c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.177+0000 7f8d791ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d740c96d0 con 0x7f8d740cc020 2024-01-11T08:26:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.177+0000 7f8d791ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7416c150 con 0x7f8d74143190 2024-01-11T08:26:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.177+0000 7f8d791ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7416b540 con 0x7f8d74142dc0 2024-01-11T08:26:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.178+0000 7f8d72d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d740cc020 0x7f8d740c68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.178+0000 7f8d72d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d740cc020 0x7f8d740c68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41340/0 (socket says 172.21.15.27:41340) 2024-01-11T08:26:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.178+0000 7f8d72d9d700 1 -- 172.21.15.27:0/1120276933 learned_addr learned my addr 172.21.15.27:0/1120276933 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.178+0000 7f8d7259c700 1 --2- 172.21.15.27:0/1120276933 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d74142dc0 0x7f8d741596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.178+0000 7f8d7359e700 1 --2- 172.21.15.27:0/1120276933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d74143190 0x7f8d740c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.178+0000 7f8d72d9d700 1 -- 172.21.15.27:0/1120276933 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d74142dc0 msgr2=0x7f8d741596d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.178+0000 7f8d72d9d700 1 --2- 172.21.15.27:0/1120276933 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d74142dc0 0x7f8d741596d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.178+0000 7f8d72d9d700 1 -- 172.21.15.27:0/1120276933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d74143190 msgr2=0x7f8d740c5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.178+0000 7f8d72d9d700 1 --2- 172.21.15.27:0/1120276933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d74143190 0x7f8d740c5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.178+0000 7f8d72d9d700 1 -- 172.21.15.27:0/1120276933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d64009420 con 0x7f8d740cc020 2024-01-11T08:26:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.178+0000 7f8d7359e700 1 --2- 172.21.15.27:0/1120276933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d74143190 0x7f8d740c5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:26:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.178+0000 7f8d7259c700 1 --2- 172.21.15.27:0/1120276933 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d74142dc0 0x7f8d741596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:26:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.178+0000 7f8d72d9d700 1 --2- 172.21.15.27:0/1120276933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d740cc020 0x7f8d740c68e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f8d74064a70 tx=0x7f8d64017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.179+0000 7f8d5affd700 1 -- 172.21.15.27:0/1120276933 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6400bbe0 con 0x7f8d740cc020 2024-01-11T08:26:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.179+0000 7f8d5affd700 1 -- 172.21.15.27:0/1120276933 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d64011420 con 0x7f8d740cc020 2024-01-11T08:26:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.179+0000 7f8d5affd700 1 -- 172.21.15.27:0/1120276933 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d640117c0 con 0x7f8d740cc020 2024-01-11T08:26:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.179+0000 7f8d791ff700 1 -- 172.21.15.27:0/1120276933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7416b820 con 0x7f8d740cc020 2024-01-11T08:26:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.179+0000 7f8d791ff700 1 -- 172.21.15.27:0/1120276933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7416bde0 con 0x7f8d740cc020 2024-01-11T08:26:35.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.180+0000 7f8d791ff700 1 -- 172.21.15.27:0/1120276933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d7404c3c0 con 0x7f8d740cc020 2024-01-11T08:26:35.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.181+0000 7f8d5affd700 1 -- 172.21.15.27:0/1120276933 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8d64011bb0 con 0x7f8d740cc020 2024-01-11T08:26:35.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.182+0000 7f8d5affd700 1 --2- 172.21.15.27:0/1120276933 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8d60066f30 0x7f8d600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:35.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.182+0000 7f8d5affd700 1 -- 172.21.15.27:0/1120276933 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 4881+0+0 (secure 0 0 0) 0x7f8d64085c70 con 0x7f8d740cc020 2024-01-11T08:26:35.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.182+0000 7f8d7259c700 1 --2- 172.21.15.27:0/1120276933 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8d60066f30 0x7f8d600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:35.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.183+0000 7f8d7259c700 1 --2- 172.21.15.27:0/1120276933 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8d60066f30 0x7f8d600693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8d740c6870 tx=0x7f8d68008040 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.186+0000 7f8d5affd700 1 -- 172.21.15.27:0/1120276933 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8d64053170 con 0x7f8d740cc020 2024-01-11T08:26:35.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.391+0000 7f8d791ff700 1 -- 172.21.15.27:0/1120276933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8d740cbd80 con 0x7f8d740cc020 2024-01-11T08:26:35.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.391+0000 7f8d5affd700 1 -- 172.21.15.27:0/1120276933 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8d64011e60 con 0x7f8d740cc020 2024-01-11T08:26:35.390 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:26:35.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.393+0000 7f8d791ff700 1 -- 172.21.15.27:0/1120276933 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8d60066f30 msgr2=0x7f8d600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:35.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.393+0000 7f8d791ff700 1 --2- 172.21.15.27:0/1120276933 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8d60066f30 0x7f8d600693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8d740c6870 tx=0x7f8d68008040 comp rx=0 tx=0).stop 2024-01-11T08:26:35.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.393+0000 7f8d791ff700 1 -- 172.21.15.27:0/1120276933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d740cc020 msgr2=0x7f8d740c68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:35.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.393+0000 7f8d791ff700 1 --2- 172.21.15.27:0/1120276933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d740cc020 0x7f8d740c68e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f8d74064a70 tx=0x7f8d64017000 comp rx=0 tx=0).stop 2024-01-11T08:26:35.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.393+0000 7f8d791ff700 1 -- 172.21.15.27:0/1120276933 shutdown_connections 2024-01-11T08:26:35.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.393+0000 7f8d791ff700 1 --2- 172.21.15.27:0/1120276933 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8d60066f30 0x7f8d600693e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:35.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.393+0000 7f8d791ff700 1 --2- 172.21.15.27:0/1120276933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d74143190 0x7f8d740c5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:35.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.393+0000 7f8d791ff700 1 --2- 172.21.15.27:0/1120276933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d740cc020 0x7f8d740c68e0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:35.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.393+0000 7f8d791ff700 1 --2- 172.21.15.27:0/1120276933 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d74142dc0 0x7f8d741596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:35.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.393+0000 7f8d791ff700 1 -- 172.21.15.27:0/1120276933 >> 172.21.15.27:0/1120276933 conn(0x7f8d74063ca0 msgr2=0x7f8d74141fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:35.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.394+0000 7f8d791ff700 1 -- 172.21.15.27:0/1120276933 shutdown_connections 2024-01-11T08:26:35.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.394+0000 7f8d791ff700 1 -- 172.21.15.27:0/1120276933 wait complete. 2024-01-11T08:26:35.411 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-11T08:26:35.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.613+0000 7fa572fff700 1 Processor -- start 2024-01-11T08:26:35.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.613+0000 7fa572fff700 1 -- start start 2024-01-11T08:26:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.614+0000 7fa572fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa56c069d40 0x7fa56c0b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.614+0000 7fa572fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56c157390 con 0x7fa56c0c89f0 2024-01-11T08:26:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.614+0000 7fa572fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56c06a200 con 0x7fa56c0c8620 2024-01-11T08:26:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.614+0000 7fa572fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa56c156d40 con 0x7fa56c069d40 2024-01-11T08:26:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.615+0000 7fa57159c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa56c0c89f0 0x7fa56c0accc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47422/0 (socket says 172.21.15.27:47422) 2024-01-11T08:26:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.615+0000 7fa57159c700 1 -- 172.21.15.27:0/3181543024 learned_addr learned my addr 172.21.15.27:0/3181543024 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.615+0000 7fa56bfff700 1 --2- 172.21.15.27:0/3181543024 >> v2:172.21.15.77:3300/0 conn(0x7fa56c069d40 0x7fa56c0b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:35.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.615+0000 7fa56b7fe700 1 -- 172.21.15.27:0/3181543024 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827885723 0 0) 0x7fa56c157390 con 0x7fa56c0c89f0 2024-01-11T08:26:35.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.615+0000 7fa56b7fe700 1 -- 172.21.15.27:0/3181543024 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa55c003570 con 0x7fa56c0c89f0 2024-01-11T08:26:35.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.615+0000 7fa56b7fe700 1 -- 172.21.15.27:0/3181543024 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478566475 0 0) 0x7fa56c06a200 con 0x7fa56c0c8620 2024-01-11T08:26:35.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.615+0000 7fa56b7fe700 1 -- 172.21.15.27:0/3181543024 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa56c157390 con 0x7fa56c0c8620 2024-01-11T08:26:35.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.616+0000 7fa56bfff700 1 -- 172.21.15.27:0/3181543024 >> v1:172.21.15.77:6789/0 conn(0x7fa56c0c8620 legacy=0x7fa56c0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.616+0000 7fa56bfff700 1 -- 172.21.15.27:0/3181543024 >> v1:172.21.15.27:6789/0 conn(0x7fa56c0c89f0 legacy=0x7fa56c0accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.616+0000 7fa56bfff700 1 -- 172.21.15.27:0/3181543024 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c157020 con 0x7fa56c069d40 2024-01-11T08:26:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.616+0000 7fa56bfff700 1 --2- 172.21.15.27:0/3181543024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa56c069d40 0x7fa56c0b02d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa5580098e0 tx=0x7fa5580091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=794aa4ebc1fa34e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.616+0000 7fa56b7fe700 1 -- 172.21.15.27:0/3181543024 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa558011630 con 0x7fa56c069d40 2024-01-11T08:26:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.616+0000 7fa56b7fe700 1 -- 172.21.15.27:0/3181543024 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa558011790 con 0x7fa56c069d40 2024-01-11T08:26:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.616+0000 7fa56b7fe700 1 -- 172.21.15.27:0/3181543024 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa558011b30 con 0x7fa56c069d40 2024-01-11T08:26:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.617+0000 7fa572fff700 1 -- 172.21.15.27:0/3181543024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa56c069d40 msgr2=0x7fa56c0b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.617+0000 7fa572fff700 1 --2- 172.21.15.27:0/3181543024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa56c069d40 0x7fa56c0b02d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa5580098e0 tx=0x7fa5580091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.617+0000 7fa572fff700 1 -- 172.21.15.27:0/3181543024 shutdown_connections 2024-01-11T08:26:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.617+0000 7fa572fff700 1 --2- 172.21.15.27:0/3181543024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa56c069d40 0x7fa56c0b02d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.617+0000 7fa572fff700 1 -- 172.21.15.27:0/3181543024 >> 172.21.15.27:0/3181543024 conn(0x7fa56c0680e0 msgr2=0x7fa56c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.617+0000 7fa572fff700 1 -- 172.21.15.27:0/3181543024 shutdown_connections 2024-01-11T08:26:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.617+0000 7fa572fff700 1 -- 172.21.15.27:0/3181543024 wait complete. 2024-01-11T08:26:35.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.618+0000 7fa572fff700 1 Processor -- start 2024-01-11T08:26:35.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.619+0000 7fa572fff700 1 -- start start 2024-01-11T08:26:35.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.619+0000 7fa572fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa56c069d40 0x7fa56c153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:35.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.619+0000 7fa572fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa56c0c8620 0x7fa56c15c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:35.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.620+0000 7fa572fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa56c0c89f0 0x7fa56c0ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:35.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.620+0000 7fa572fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c1604a0 con 0x7fa56c0c89f0 2024-01-11T08:26:35.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.620+0000 7fa572fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c15c4b0 con 0x7fa56c0c8620 2024-01-11T08:26:35.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.620+0000 7fa572fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c154360 con 0x7fa56c069d40 2024-01-11T08:26:35.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.620+0000 7fa57159c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa56c0c89f0 0x7fa56c0ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:35.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.620+0000 7fa57159c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa56c0c89f0 0x7fa56c0ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41350/0 (socket says 172.21.15.27:41350) 2024-01-11T08:26:35.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.620+0000 7fa57159c700 1 -- 172.21.15.27:0/944605892 learned_addr learned my addr 172.21.15.27:0/944605892 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:35.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.620+0000 7fa56bfff700 1 --2- 172.21.15.27:0/944605892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa56c0c8620 0x7fa56c15c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:35.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.620+0000 7fa570d9b700 1 --2- 172.21.15.27:0/944605892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa56c069d40 0x7fa56c153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:35.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.620+0000 7fa57159c700 1 -- 172.21.15.27:0/944605892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa56c069d40 msgr2=0x7fa56c153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:35.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.620+0000 7fa57159c700 1 --2- 172.21.15.27:0/944605892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa56c069d40 0x7fa56c153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:35.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.620+0000 7fa57159c700 1 -- 172.21.15.27:0/944605892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa56c0c8620 msgr2=0x7fa56c15c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:35.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.620+0000 7fa57159c700 1 --2- 172.21.15.27:0/944605892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa56c0c8620 0x7fa56c15c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:35.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.620+0000 7fa57159c700 1 -- 172.21.15.27:0/944605892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa558009420 con 0x7fa56c0c89f0 2024-01-11T08:26:35.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.621+0000 7fa57159c700 1 --2- 172.21.15.27:0/944605892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa56c0c89f0 0x7fa56c0ca2a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fa56400cd00 tx=0x7fa564001ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:35.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.621+0000 7fa568ff9700 1 -- 172.21.15.27:0/944605892 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56400ce00 con 0x7fa56c0c89f0 2024-01-11T08:26:35.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.621+0000 7fa568ff9700 1 -- 172.21.15.27:0/944605892 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56400aa80 con 0x7fa56c0c89f0 2024-01-11T08:26:35.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.621+0000 7fa572fff700 1 -- 172.21.15.27:0/944605892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa56c15b5c0 con 0x7fa56c0c89f0 2024-01-11T08:26:35.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.621+0000 7fa568ff9700 1 -- 172.21.15.27:0/944605892 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56400ae40 con 0x7fa56c0c89f0 2024-01-11T08:26:35.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.621+0000 7fa572fff700 1 -- 172.21.15.27:0/944605892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56c15bb80 con 0x7fa56c0c89f0 2024-01-11T08:26:35.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.622+0000 7fa572fff700 1 -- 172.21.15.27:0/944605892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa56c15c640 con 0x7fa56c0c89f0 2024-01-11T08:26:35.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.623+0000 7fa568ff9700 1 -- 172.21.15.27:0/944605892 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa564007300 con 0x7fa56c0c89f0 2024-01-11T08:26:35.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.624+0000 7fa568ff9700 1 --2- 172.21.15.27:0/944605892 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa550066f30 0x7fa5500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:35.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.624+0000 7fa568ff9700 1 -- 172.21.15.27:0/944605892 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 4881+0+0 (secure 0 0 0) 0x7fa564084550 con 0x7fa56c0c89f0 2024-01-11T08:26:35.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.624+0000 7fa570d9b700 1 --2- 172.21.15.27:0/944605892 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa550066f30 0x7fa5500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:35.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.625+0000 7fa570d9b700 1 --2- 172.21.15.27:0/944605892 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa550066f30 0x7fa5500693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa560007fd0 tx=0x7fa560007d50 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.628+0000 7fa568ff9700 1 -- 172.21.15.27:0/944605892 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa564051b70 con 0x7fa56c0c89f0 2024-01-11T08:26:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.835+0000 7fa572fff700 1 -- 172.21.15.27:0/944605892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa56c0718d0 con 0x7fa56c0c89f0 2024-01-11T08:26:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.835+0000 7fa568ff9700 1 -- 172.21.15.27:0/944605892 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v97) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa564055190 con 0x7fa56c0c89f0 2024-01-11T08:26:35.835 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:35.835 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T08:26:35.835 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:35.835 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-11T08:26:35.835 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-11T08:26:35.835 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:35.835 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T08:26:35.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.838+0000 7fa572fff700 1 -- 172.21.15.27:0/944605892 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa550066f30 msgr2=0x7fa5500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:35.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.838+0000 7fa572fff700 1 --2- 172.21.15.27:0/944605892 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa550066f30 0x7fa5500693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa560007fd0 tx=0x7fa560007d50 comp rx=0 tx=0).stop 2024-01-11T08:26:35.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.838+0000 7fa572fff700 1 -- 172.21.15.27:0/944605892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa56c0c89f0 msgr2=0x7fa56c0ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.838+0000 7fa572fff700 1 --2- 172.21.15.27:0/944605892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa56c0c89f0 0x7fa56c0ca2a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fa56400cd00 tx=0x7fa564001ac0 comp rx=0 tx=0).stop 2024-01-11T08:26:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.838+0000 7fa572fff700 1 -- 172.21.15.27:0/944605892 shutdown_connections 2024-01-11T08:26:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.838+0000 7fa572fff700 1 --2- 172.21.15.27:0/944605892 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa550066f30 0x7fa5500693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.838+0000 7fa572fff700 1 --2- 172.21.15.27:0/944605892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa56c0c89f0 0x7fa56c0ca2a0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.838+0000 7fa572fff700 1 --2- 172.21.15.27:0/944605892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa56c069d40 0x7fa56c153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.838+0000 7fa572fff700 1 --2- 172.21.15.27:0/944605892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa56c0c8620 0x7fa56c15c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.838+0000 7fa572fff700 1 -- 172.21.15.27:0/944605892 >> 172.21.15.27:0/944605892 conn(0x7fa56c0680e0 msgr2=0x7fa56c0b02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.838+0000 7fa572fff700 1 -- 172.21.15.27:0/944605892 shutdown_connections 2024-01-11T08:26:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:35.838+0000 7fa572fff700 1 -- 172.21.15.27:0/944605892 wait complete. 2024-01-11T08:26:35.855 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-11T08:26:35.855 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-11T08:26:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.059+0000 7feb317ff700 1 Processor -- start 2024-01-11T08:26:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.059+0000 7feb317ff700 1 -- start start 2024-01-11T08:26:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.060+0000 7feb317ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c15a7a0 0x7feb2c15c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.060+0000 7feb317ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feb2c0685e0 0x7feb2c068a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.060+0000 7feb317ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2c15c4b0 con 0x7feb2c159e60 2024-01-11T08:26:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.061+0000 7feb317ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2c068fc0 con 0x7feb2c0685e0 2024-01-11T08:26:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.061+0000 7feb317ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c15b5d0 con 0x7feb2c15a7a0 2024-01-11T08:26:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.061+0000 7feb2b7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feb2c0685e0 0x7feb2c068a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.061+0000 7feb2b7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feb2c0685e0 0x7feb2c068a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41352/0 (socket says 172.21.15.27:41352) 2024-01-11T08:26:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.061+0000 7feb2b7fe700 1 -- 172.21.15.27:0/2389767833 learned_addr learned my addr 172.21.15.27:0/2389767833 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.061+0000 7feb2a7fc700 1 --1- 172.21.15.27:0/2389767833 >> v1:172.21.15.27:6789/0 conn(0x7feb2c159e60 0x7feb2c15aec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47432/0 (socket says 172.21.15.27:47432) 2024-01-11T08:26:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.061+0000 7feb2affd700 1 --2- 172.21.15.27:0/2389767833 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c15a7a0 0x7feb2c15c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.061+0000 7feb2b7fe700 1 -- 172.21.15.27:0/2389767833 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c15a7a0 msgr2=0x7feb2c15c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.061+0000 7feb2b7fe700 1 --2- 172.21.15.27:0/2389767833 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c15a7a0 0x7feb2c15c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.061+0000 7feb2b7fe700 1 -- 172.21.15.27:0/2389767833 >> v1:172.21.15.27:6789/0 conn(0x7feb2c159e60 legacy=0x7feb2c15aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.061+0000 7feb2b7fe700 1 -- 172.21.15.27:0/2389767833 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c15b8a0 con 0x7feb2c0685e0 2024-01-11T08:26:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.061+0000 7feb29ffb700 1 -- 172.21.15.27:0/2389767833 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023933469 0 0) 0x7feb2c15c4b0 con 0x7feb2c159e60 2024-01-11T08:26:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.062+0000 7feb2b7fe700 1 --2- 172.21.15.27:0/2389767833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb2c0685e0 0x7feb2c068a80 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7feb24009f20 tx=0x7feb24009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2db97a41215860f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.062+0000 7feb29ffb700 1 -- 172.21.15.27:0/2389767833 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2400e070 con 0x7feb2c0685e0 2024-01-11T08:26:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.062+0000 7feb29ffb700 1 -- 172.21.15.27:0/2389767833 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2400b9d0 con 0x7feb2c0685e0 2024-01-11T08:26:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.062+0000 7feb29ffb700 1 -- 172.21.15.27:0/2389767833 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb240092e0 con 0x7feb2c0685e0 2024-01-11T08:26:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.062+0000 7feb317ff700 1 -- 172.21.15.27:0/2389767833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb2c0685e0 msgr2=0x7feb2c068a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.062+0000 7feb317ff700 1 --2- 172.21.15.27:0/2389767833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb2c0685e0 0x7feb2c068a80 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7feb24009f20 tx=0x7feb24009840 comp rx=0 tx=0).stop 2024-01-11T08:26:36.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.062+0000 7feb317ff700 1 -- 172.21.15.27:0/2389767833 shutdown_connections 2024-01-11T08:26:36.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.062+0000 7feb317ff700 1 --2- 172.21.15.27:0/2389767833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb2c0685e0 0x7feb2c068a80 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.062+0000 7feb317ff700 1 --2- 172.21.15.27:0/2389767833 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c15a7a0 0x7feb2c15c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.062+0000 7feb317ff700 1 -- 172.21.15.27:0/2389767833 >> 172.21.15.27:0/2389767833 conn(0x7feb2c06ed40 msgr2=0x7feb2c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.063+0000 7feb317ff700 1 -- 172.21.15.27:0/2389767833 shutdown_connections 2024-01-11T08:26:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.063+0000 7feb317ff700 1 -- 172.21.15.27:0/2389767833 wait complete. 2024-01-11T08:26:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.064+0000 7feb317ff700 1 Processor -- start 2024-01-11T08:26:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.064+0000 7feb317ff700 1 -- start start 2024-01-11T08:26:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.065+0000 7feb317ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c0685e0 0x7feb2c16aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.065+0000 7feb317ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb2c15a7a0 0x7feb2c13ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.065+0000 7feb317ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb2c14cd30 0x7feb2c14d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.065+0000 7feb317ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c14d6f0 con 0x7feb2c14cd30 2024-01-11T08:26:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.065+0000 7feb317ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c0b09b0 con 0x7feb2c15a7a0 2024-01-11T08:26:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.065+0000 7feb317ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c0d0c10 con 0x7feb2c0685e0 2024-01-11T08:26:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.065+0000 7feb2b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb2c14cd30 0x7feb2c14d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.065+0000 7feb2b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb2c14cd30 0x7feb2c14d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41366/0 (socket says 172.21.15.27:41366) 2024-01-11T08:26:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.065+0000 7feb2b7fe700 1 -- 172.21.15.27:0/2391508261 learned_addr learned my addr 172.21.15.27:0/2391508261 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.065+0000 7feb2affd700 1 --2- 172.21.15.27:0/2391508261 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c0685e0 0x7feb2c16aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.065+0000 7feb2a7fc700 1 --2- 172.21.15.27:0/2391508261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb2c15a7a0 0x7feb2c13ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.066+0000 7feb2b7fe700 1 -- 172.21.15.27:0/2391508261 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c0685e0 msgr2=0x7feb2c16aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.066+0000 7feb2b7fe700 1 --2- 172.21.15.27:0/2391508261 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c0685e0 0x7feb2c16aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.066+0000 7feb2b7fe700 1 -- 172.21.15.27:0/2391508261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb2c15a7a0 msgr2=0x7feb2c13ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.066+0000 7feb2b7fe700 1 --2- 172.21.15.27:0/2391508261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb2c15a7a0 0x7feb2c13ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.066+0000 7feb2b7fe700 1 -- 172.21.15.27:0/2391508261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb24009ae0 con 0x7feb2c14cd30 2024-01-11T08:26:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.066+0000 7feb2a7fc700 1 --2- 172.21.15.27:0/2391508261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb2c15a7a0 0x7feb2c13ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:26:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.066+0000 7feb2b7fe700 1 --2- 172.21.15.27:0/2391508261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb2c14cd30 0x7feb2c14d1b0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7feb2400b610 tx=0x7feb24018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.066+0000 7feb12ffd700 1 -- 172.21.15.27:0/2391508261 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24008b80 con 0x7feb2c14cd30 2024-01-11T08:26:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.066+0000 7feb12ffd700 1 -- 172.21.15.27:0/2391508261 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb24008d20 con 0x7feb2c14cd30 2024-01-11T08:26:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.066+0000 7feb12ffd700 1 -- 172.21.15.27:0/2391508261 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2401c660 con 0x7feb2c14cd30 2024-01-11T08:26:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.066+0000 7feb317ff700 1 -- 172.21.15.27:0/2391508261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2c0b0080 con 0x7feb2c14cd30 2024-01-11T08:26:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.067+0000 7feb317ff700 1 -- 172.21.15.27:0/2391508261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2c0b0640 con 0x7feb2c14cd30 2024-01-11T08:26:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.068+0000 7feb317ff700 1 -- 172.21.15.27:0/2391508261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2c04c3c0 con 0x7feb2c14cd30 2024-01-11T08:26:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.068+0000 7feb12ffd700 1 -- 172.21.15.27:0/2391508261 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7feb24003040 con 0x7feb2c14cd30 2024-01-11T08:26:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.070+0000 7feb12ffd700 1 --2- 172.21.15.27:0/2391508261 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7feb18066f30 0x7feb180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.070+0000 7feb12ffd700 1 -- 172.21.15.27:0/2391508261 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 4881+0+0 (secure 0 0 0) 0x7feb24085b40 con 0x7feb2c14cd30 2024-01-11T08:26:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.070+0000 7feb2affd700 1 --2- 172.21.15.27:0/2391508261 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7feb18066f30 0x7feb180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.071+0000 7feb2affd700 1 --2- 172.21.15.27:0/2391508261 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7feb18066f30 0x7feb180693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feb1c009db0 tx=0x7feb1c0091a0 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.074+0000 7feb12ffd700 1 -- 172.21.15.27:0/2391508261 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feb24053040 con 0x7feb2c14cd30 2024-01-11T08:26:36.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.276+0000 7feb317ff700 1 -- 172.21.15.27:0/2391508261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7feb2c0b0af0 con 0x7feb2c14cd30 2024-01-11T08:26:36.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.992+0000 7feb12ffd700 1 -- 172.21.15.27:0/2391508261 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v98) v1 ==== 138+0+0 (secure 0 0 0) 0x7feb24056660 con 0x7feb2c14cd30 2024-01-11T08:26:36.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.996+0000 7feb317ff700 1 -- 172.21.15.27:0/2391508261 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7feb18066f30 msgr2=0x7feb180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:36.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.996+0000 7feb317ff700 1 --2- 172.21.15.27:0/2391508261 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7feb18066f30 0x7feb180693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feb1c009db0 tx=0x7feb1c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:36.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.996+0000 7feb317ff700 1 -- 172.21.15.27:0/2391508261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb2c14cd30 msgr2=0x7feb2c14d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:36.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.996+0000 7feb317ff700 1 --2- 172.21.15.27:0/2391508261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb2c14cd30 0x7feb2c14d1b0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7feb2400b610 tx=0x7feb24018000 comp rx=0 tx=0).stop 2024-01-11T08:26:36.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.997+0000 7feb317ff700 1 -- 172.21.15.27:0/2391508261 shutdown_connections 2024-01-11T08:26:36.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.997+0000 7feb317ff700 1 --2- 172.21.15.27:0/2391508261 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7feb18066f30 0x7feb180693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:36.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.997+0000 7feb317ff700 1 --2- 172.21.15.27:0/2391508261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb2c14cd30 0x7feb2c14d1b0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:36.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.997+0000 7feb317ff700 1 --2- 172.21.15.27:0/2391508261 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c0685e0 0x7feb2c16aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:36.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.997+0000 7feb317ff700 1 --2- 172.21.15.27:0/2391508261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb2c15a7a0 0x7feb2c13ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:36.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.997+0000 7feb317ff700 1 -- 172.21.15.27:0/2391508261 >> 172.21.15.27:0/2391508261 conn(0x7feb2c06ed40 msgr2=0x7feb2c0b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:36.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.997+0000 7feb317ff700 1 -- 172.21.15.27:0/2391508261 shutdown_connections 2024-01-11T08:26:36.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:36.998+0000 7feb317ff700 1 -- 172.21.15.27:0/2391508261 wait complete. 2024-01-11T08:26:37.000 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-11T08:26:37.019 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-11T08:26:37.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.230+0000 7f4e261ff700 1 Processor -- start 2024-01-11T08:26:37.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.230+0000 7f4e261ff700 1 -- start start 2024-01-11T08:26:37.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.231+0000 7f4e261ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e20069d40 0x7f4e200b0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.231+0000 7f4e261ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4e200ab700 0x7f4e200b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.231+0000 7f4e261ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e200af4c0 con 0x7f4e20153e90 2024-01-11T08:26:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.231+0000 7f4e261ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e201574d0 con 0x7f4e200ab700 2024-01-11T08:26:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.231+0000 7f4e261ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e201577b0 con 0x7f4e20069d40 2024-01-11T08:26:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.231+0000 7f4e1ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4e200ab700 0x7f4e200b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.232+0000 7f4e1ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4e200ab700 0x7f4e200b0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41370/0 (socket says 172.21.15.27:41370) 2024-01-11T08:26:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.232+0000 7f4e1ffff700 1 -- 172.21.15.27:0/2680942143 learned_addr learned my addr 172.21.15.27:0/2680942143 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.232+0000 7f4e1f7fe700 1 --1- 172.21.15.27:0/2680942143 >> v1:172.21.15.77:6789/0 conn(0x7f4e20153e90 0x7f4e200accc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:54532/0 (socket says 172.21.15.27:54532) 2024-01-11T08:26:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.232+0000 7f4e1effd700 1 --2- 172.21.15.27:0/2680942143 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e20069d40 0x7f4e200b0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.232+0000 7f4e1ffff700 1 -- 172.21.15.27:0/2680942143 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e20069d40 msgr2=0x7f4e200b0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.232+0000 7f4e1ffff700 1 --2- 172.21.15.27:0/2680942143 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e20069d40 0x7f4e200b0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.232+0000 7f4e1ffff700 1 -- 172.21.15.27:0/2680942143 >> v1:172.21.15.77:6789/0 conn(0x7f4e20153e90 legacy=0x7f4e200accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.232+0000 7f4e1ffff700 1 -- 172.21.15.27:0/2680942143 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e200af150 con 0x7f4e200ab700 2024-01-11T08:26:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.232+0000 7f4e1ffff700 1 --2- 172.21.15.27:0/2680942143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e200ab700 0x7f4e200b0c60 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800d1a0 tx=0x7f4e1800ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2367d8ec602ea45 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.232+0000 7f4e1e7fc700 1 -- 172.21.15.27:0/2680942143 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e18010070 con 0x7f4e200ab700 2024-01-11T08:26:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.233+0000 7f4e1e7fc700 1 -- 172.21.15.27:0/2680942143 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e1800de10 con 0x7f4e200ab700 2024-01-11T08:26:37.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.233+0000 7f4e1e7fc700 1 -- 172.21.15.27:0/2680942143 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1800b320 con 0x7f4e200ab700 2024-01-11T08:26:37.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.233+0000 7f4e261ff700 1 -- 172.21.15.27:0/2680942143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e200ab700 msgr2=0x7f4e200b0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:37.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.233+0000 7f4e261ff700 1 --2- 172.21.15.27:0/2680942143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e200ab700 0x7f4e200b0c60 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800d1a0 tx=0x7f4e1800ba50 comp rx=0 tx=0).stop 2024-01-11T08:26:37.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.233+0000 7f4e261ff700 1 -- 172.21.15.27:0/2680942143 shutdown_connections 2024-01-11T08:26:37.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.233+0000 7f4e261ff700 1 --2- 172.21.15.27:0/2680942143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e200ab700 0x7f4e200b0c60 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:37.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.233+0000 7f4e261ff700 1 --2- 172.21.15.27:0/2680942143 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e20069d40 0x7f4e200b0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:37.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.233+0000 7f4e261ff700 1 -- 172.21.15.27:0/2680942143 >> 172.21.15.27:0/2680942143 conn(0x7f4e200680a0 msgr2=0x7f4e20069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:37.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.233+0000 7f4e261ff700 1 -- 172.21.15.27:0/2680942143 shutdown_connections 2024-01-11T08:26:37.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.233+0000 7f4e261ff700 1 -- 172.21.15.27:0/2680942143 wait complete. 2024-01-11T08:26:37.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.235+0000 7f4e261ff700 1 Processor -- start 2024-01-11T08:26:37.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.235+0000 7f4e261ff700 1 -- start start 2024-01-11T08:26:37.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.236+0000 7f4e261ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e20069d40 0x7f4e200ac590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:37.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.236+0000 7f4e261ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e200ab700 0x7f4e20150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:37.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.236+0000 7f4e261ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e20153e90 0x7f4e20159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:37.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.236+0000 7f4e261ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e20165480 con 0x7f4e20069d40 2024-01-11T08:26:37.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.236+0000 7f4e261ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e20165750 con 0x7f4e200ab700 2024-01-11T08:26:37.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.236+0000 7f4e261ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e200acad0 con 0x7f4e20153e90 2024-01-11T08:26:37.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.236+0000 7f4e1f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e20069d40 0x7f4e200ac590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:37.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.236+0000 7f4e1f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e20069d40 0x7f4e200ac590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41384/0 (socket says 172.21.15.27:41384) 2024-01-11T08:26:37.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.236+0000 7f4e1f7fe700 1 -- 172.21.15.27:0/1032814824 learned_addr learned my addr 172.21.15.27:0/1032814824 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:37.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.236+0000 7f4e1ffff700 1 --2- 172.21.15.27:0/1032814824 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e20153e90 0x7f4e20159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:37.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.236+0000 7f4e1effd700 1 --2- 172.21.15.27:0/1032814824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e200ab700 0x7f4e20150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:37.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.237+0000 7f4e1f7fe700 1 -- 172.21.15.27:0/1032814824 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e20153e90 msgr2=0x7f4e20159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:37.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.237+0000 7f4e1f7fe700 1 --2- 172.21.15.27:0/1032814824 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e20153e90 0x7f4e20159dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:37.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.237+0000 7f4e1f7fe700 1 -- 172.21.15.27:0/1032814824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e200ab700 msgr2=0x7f4e20150b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:37.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.237+0000 7f4e1f7fe700 1 --2- 172.21.15.27:0/1032814824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e200ab700 0x7f4e20150b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:37.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.237+0000 7f4e1f7fe700 1 -- 172.21.15.27:0/1032814824 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1800bcf0 con 0x7f4e20069d40 2024-01-11T08:26:37.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.237+0000 7f4e1f7fe700 1 --2- 172.21.15.27:0/1032814824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e20069d40 0x7f4e200ac590 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f4e1000c560 tx=0x7f4e10007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:37.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.237+0000 7f4e077fe700 1 -- 172.21.15.27:0/1032814824 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e10005a70 con 0x7f4e20069d40 2024-01-11T08:26:37.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.237+0000 7f4e077fe700 1 -- 172.21.15.27:0/1032814824 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e10005bd0 con 0x7f4e20069d40 2024-01-11T08:26:37.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.237+0000 7f4e077fe700 1 -- 172.21.15.27:0/1032814824 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e100070d0 con 0x7f4e20069d40 2024-01-11T08:26:37.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.237+0000 7f4e261ff700 1 -- 172.21.15.27:0/1032814824 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2015aba0 con 0x7f4e20069d40 2024-01-11T08:26:37.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.237+0000 7f4e261ff700 1 -- 172.21.15.27:0/1032814824 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e200ad010 con 0x7f4e20069d40 2024-01-11T08:26:37.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.238+0000 7f4e261ff700 1 -- 172.21.15.27:0/1032814824 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e2004c3c0 con 0x7f4e20069d40 2024-01-11T08:26:37.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.240+0000 7f4e077fe700 1 -- 172.21.15.27:0/1032814824 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4e10007480 con 0x7f4e20069d40 2024-01-11T08:26:37.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.241+0000 7f4e077fe700 1 --2- 172.21.15.27:0/1032814824 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4e0c066f30 0x7f4e0c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:37.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.241+0000 7f4e077fe700 1 -- 172.21.15.27:0/1032814824 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5251+0+0 (secure 0 0 0) 0x7f4e10083c10 con 0x7f4e20069d40 2024-01-11T08:26:37.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.241+0000 7f4e1effd700 1 --2- 172.21.15.27:0/1032814824 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4e0c066f30 0x7f4e0c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:37.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.242+0000 7f4e1effd700 1 --2- 172.21.15.27:0/1032814824 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4e0c066f30 0x7f4e0c0693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4e20159380 tx=0x7f4e140091a0 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:37.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.244+0000 7f4e077fe700 1 -- 172.21.15.27:0/1032814824 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4e10051020 con 0x7f4e20069d40 2024-01-11T08:26:37.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:37.452+0000 7f4e261ff700 1 -- 172.21.15.27:0/1032814824 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4e2006a4f0 con 0x7f4e20069d40 2024-01-11T08:26:38.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.004+0000 7f4e077fe700 1 -- 172.21.15.27:0/1032814824 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v99) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4e100170d0 con 0x7f4e20069d40 2024-01-11T08:26:38.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.008+0000 7f4e261ff700 1 -- 172.21.15.27:0/1032814824 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4e0c066f30 msgr2=0x7f4e0c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:38.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.008+0000 7f4e261ff700 1 --2- 172.21.15.27:0/1032814824 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4e0c066f30 0x7f4e0c0693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4e20159380 tx=0x7f4e140091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:38.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.009+0000 7f4e261ff700 1 -- 172.21.15.27:0/1032814824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e20069d40 msgr2=0x7f4e200ac590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:38.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.009+0000 7f4e261ff700 1 --2- 172.21.15.27:0/1032814824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e20069d40 0x7f4e200ac590 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f4e1000c560 tx=0x7f4e10007f70 comp rx=0 tx=0).stop 2024-01-11T08:26:38.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.009+0000 7f4e261ff700 1 -- 172.21.15.27:0/1032814824 shutdown_connections 2024-01-11T08:26:38.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.009+0000 7f4e261ff700 1 --2- 172.21.15.27:0/1032814824 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4e0c066f30 0x7f4e0c0693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:38.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.009+0000 7f4e261ff700 1 --2- 172.21.15.27:0/1032814824 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e20153e90 0x7f4e20159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:38.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.009+0000 7f4e261ff700 1 --2- 172.21.15.27:0/1032814824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e20069d40 0x7f4e200ac590 secure :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f4e1000c560 tx=0x7f4e10007f70 comp rx=0 tx=0).stop 2024-01-11T08:26:38.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.009+0000 7f4e261ff700 1 --2- 172.21.15.27:0/1032814824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e200ab700 0x7f4e20150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:38.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.009+0000 7f4e261ff700 1 -- 172.21.15.27:0/1032814824 >> 172.21.15.27:0/1032814824 conn(0x7f4e200680a0 msgr2=0x7f4e20155540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:38.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.010+0000 7f4e261ff700 1 -- 172.21.15.27:0/1032814824 shutdown_connections 2024-01-11T08:26:38.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.010+0000 7f4e261ff700 1 -- 172.21.15.27:0/1032814824 wait complete. 2024-01-11T08:26:38.014 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-11T08:26:38.043 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-11T08:26:38.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.249+0000 7f3eb57ff700 1 Processor -- start 2024-01-11T08:26:38.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.249+0000 7f3eb57ff700 1 -- start start 2024-01-11T08:26:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.250+0000 7f3eb57ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3eb015a1b0 0x7f3eb015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.250+0000 7f3eb57ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3eb015ae70 0x7f3eb015b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.250+0000 7f3eb57ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb00cf850 con 0x7f3eb00ccd70 2024-01-11T08:26:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.250+0000 7f3eb57ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb00cec40 con 0x7f3eb015ae70 2024-01-11T08:26:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.250+0000 7f3eb57ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb00cef20 con 0x7f3eb015a1b0 2024-01-11T08:26:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.250+0000 7f3eae7fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3eb00ccd70 0x7f3eb00cd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47442/0 (socket says 172.21.15.27:47442) 2024-01-11T08:26:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.250+0000 7f3eae7fc700 1 -- 172.21.15.27:0/725712814 learned_addr learned my addr 172.21.15.27:0/725712814 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.251+0000 7f3eaeffd700 1 --2- 172.21.15.27:0/725712814 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3eb015a1b0 0x7f3eb015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.251+0000 7f3eaf7fe700 1 --2- 172.21.15.27:0/725712814 >> v2:172.21.15.77:3300/0 conn(0x7f3eb015ae70 0x7f3eb015b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.251+0000 7f3eadffb700 1 -- 172.21.15.27:0/725712814 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928901004 0 0) 0x7f3eb00cf850 con 0x7f3eb00ccd70 2024-01-11T08:26:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.251+0000 7f3eadffb700 1 -- 172.21.15.27:0/725712814 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e98003570 con 0x7f3eb00ccd70 2024-01-11T08:26:38.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.251+0000 7f3eaf7fe700 1 -- 172.21.15.27:0/725712814 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3eb015a1b0 msgr2=0x7f3eb015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:38.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.251+0000 7f3eadffb700 1 -- 172.21.15.27:0/725712814 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3050168619 0 0) 0x7f3e98003570 con 0x7f3eb00ccd70 2024-01-11T08:26:38.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.251+0000 7f3eaf7fe700 1 --2- 172.21.15.27:0/725712814 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3eb015a1b0 0x7f3eb015a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:38.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.251+0000 7f3eaf7fe700 1 -- 172.21.15.27:0/725712814 >> v1:172.21.15.27:6789/0 conn(0x7f3eb00ccd70 legacy=0x7f3eb00cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:38.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.251+0000 7f3eaf7fe700 1 -- 172.21.15.27:0/725712814 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb00cf200 con 0x7f3eb015ae70 2024-01-11T08:26:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.251+0000 7f3eaeffd700 1 --2- 172.21.15.27:0/725712814 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3eb015a1b0 0x7f3eb015a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.252+0000 7f3eaf7fe700 1 --2- 172.21.15.27:0/725712814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb015ae70 0x7f3eb015b2f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8009e60 tx=0x7f3ea8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98f4b4676f0f84db server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.252+0000 7f3eadffb700 1 -- 172.21.15.27:0/725712814 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea800e070 con 0x7f3eb015ae70 2024-01-11T08:26:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.252+0000 7f3eadffb700 1 -- 172.21.15.27:0/725712814 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea800b7a0 con 0x7f3eb015ae70 2024-01-11T08:26:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.252+0000 7f3eadffb700 1 -- 172.21.15.27:0/725712814 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4001470 con 0x7f3eb015ae70 2024-01-11T08:26:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.252+0000 7f3eb57ff700 1 -- 172.21.15.27:0/725712814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb015ae70 msgr2=0x7f3eb015b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.252+0000 7f3eb57ff700 1 --2- 172.21.15.27:0/725712814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb015ae70 0x7f3eb015b2f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8009e60 tx=0x7f3ea8009780 comp rx=0 tx=0).stop 2024-01-11T08:26:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.252+0000 7f3eb57ff700 1 -- 172.21.15.27:0/725712814 shutdown_connections 2024-01-11T08:26:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.252+0000 7f3eb57ff700 1 --2- 172.21.15.27:0/725712814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb015ae70 0x7f3eb015b2f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.252+0000 7f3eb57ff700 1 --2- 172.21.15.27:0/725712814 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3eb015a1b0 0x7f3eb015a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.252+0000 7f3eb57ff700 1 -- 172.21.15.27:0/725712814 >> 172.21.15.27:0/725712814 conn(0x7f3eb006ed40 msgr2=0x7f3eb006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.252+0000 7f3eb57ff700 1 -- 172.21.15.27:0/725712814 shutdown_connections 2024-01-11T08:26:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.253+0000 7f3eb57ff700 1 -- 172.21.15.27:0/725712814 wait complete. 2024-01-11T08:26:38.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.254+0000 7f3eb57ff700 1 Processor -- start 2024-01-11T08:26:38.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.254+0000 7f3eb57ff700 1 -- start start 2024-01-11T08:26:38.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.255+0000 7f3eb57ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eb00ccd70 0x7f3eb0155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:38.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.255+0000 7f3eb57ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3eb015a1b0 0x7f3eb0158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:38.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.255+0000 7f3eb57ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb015ae70 0x7f3eb00c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:38.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.255+0000 7f3eb57ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb0157a90 con 0x7f3eb00ccd70 2024-01-11T08:26:38.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.255+0000 7f3eb57ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb0157d70 con 0x7f3eb015ae70 2024-01-11T08:26:38.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.255+0000 7f3eb57ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb0158050 con 0x7f3eb015a1b0 2024-01-11T08:26:38.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.255+0000 7f3eaeffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eb00ccd70 0x7f3eb0155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:38.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.255+0000 7f3eaeffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eb00ccd70 0x7f3eb0155490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41386/0 (socket says 172.21.15.27:41386) 2024-01-11T08:26:38.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.255+0000 7f3eaeffd700 1 -- 172.21.15.27:0/712697593 learned_addr learned my addr 172.21.15.27:0/712697593 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.255+0000 7f3eaf7fe700 1 --2- 172.21.15.27:0/712697593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb015ae70 0x7f3eb00c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.255+0000 7f3eae7fc700 1 --2- 172.21.15.27:0/712697593 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3eb015a1b0 0x7f3eb0158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.256+0000 7f3eaeffd700 1 -- 172.21.15.27:0/712697593 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3eb015a1b0 msgr2=0x7f3eb0158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.256+0000 7f3eaeffd700 1 --2- 172.21.15.27:0/712697593 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3eb015a1b0 0x7f3eb0158b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.256+0000 7f3eaeffd700 1 -- 172.21.15.27:0/712697593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb015ae70 msgr2=0x7f3eb00c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.256+0000 7f3eaeffd700 1 --2- 172.21.15.27:0/712697593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb015ae70 0x7f3eb00c98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.256+0000 7f3eaeffd700 1 -- 172.21.15.27:0/712697593 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea0009420 con 0x7f3eb00ccd70 2024-01-11T08:26:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.256+0000 7f3eaeffd700 1 --2- 172.21.15.27:0/712697593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eb00ccd70 0x7f3eb0155490 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000ec80 tx=0x7f3ea00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.256+0000 7f3e96ffd700 1 -- 172.21.15.27:0/712697593 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea000ca60 con 0x7f3eb00ccd70 2024-01-11T08:26:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.256+0000 7f3e96ffd700 1 -- 172.21.15.27:0/712697593 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea000cbc0 con 0x7f3eb00ccd70 2024-01-11T08:26:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.256+0000 7f3e96ffd700 1 -- 172.21.15.27:0/712697593 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0013470 con 0x7f3eb00ccd70 2024-01-11T08:26:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.256+0000 7f3eb57ff700 1 -- 172.21.15.27:0/712697593 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea8009a20 con 0x7f3eb00ccd70 2024-01-11T08:26:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.256+0000 7f3eb57ff700 1 -- 172.21.15.27:0/712697593 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb0158610 con 0x7f3eb00ccd70 2024-01-11T08:26:38.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.257+0000 7f3eb57ff700 1 -- 172.21.15.27:0/712697593 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb0158800 con 0x7f3eb00ccd70 2024-01-11T08:26:38.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.259+0000 7f3e96ffd700 1 -- 172.21.15.27:0/712697593 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3ea0013820 con 0x7f3eb00ccd70 2024-01-11T08:26:38.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.261+0000 7f3e96ffd700 1 --2- 172.21.15.27:0/712697593 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3e9c066f30 0x7f3e9c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:38.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.261+0000 7f3e96ffd700 1 -- 172.21.15.27:0/712697593 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5633+0+0 (secure 0 0 0) 0x7f3ea0085160 con 0x7f3eb00ccd70 2024-01-11T08:26:38.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.261+0000 7f3eae7fc700 1 --2- 172.21.15.27:0/712697593 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3e9c066f30 0x7f3e9c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:38.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.262+0000 7f3eae7fc700 1 --2- 172.21.15.27:0/712697593 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3e9c066f30 0x7f3e9c0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4001ed0 tx=0x7f3ea4001520 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.265+0000 7f3e96ffd700 1 -- 172.21.15.27:0/712697593 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3ea0052490 con 0x7f3eb00ccd70 2024-01-11T08:26:38.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:38.512+0000 7f3eb57ff700 1 -- 172.21.15.27:0/712697593 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f3eb004c3c0 con 0x7f3eb00ccd70 2024-01-11T08:26:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.005+0000 7f3e96ffd700 1 -- 172.21.15.27:0/712697593 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v47) v1 ==== 174+0+0 (secure 0 0 0) 0x7f3ea0055ab0 con 0x7f3eb00ccd70 2024-01-11T08:26:39.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.010+0000 7f3eb57ff700 1 -- 172.21.15.27:0/712697593 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3e9c066f30 msgr2=0x7f3e9c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:39.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.010+0000 7f3eb57ff700 1 --2- 172.21.15.27:0/712697593 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3e9c066f30 0x7f3e9c0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4001ed0 tx=0x7f3ea4001520 comp rx=0 tx=0).stop 2024-01-11T08:26:39.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.010+0000 7f3eb57ff700 1 -- 172.21.15.27:0/712697593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eb00ccd70 msgr2=0x7f3eb0155490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.010+0000 7f3eb57ff700 1 --2- 172.21.15.27:0/712697593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eb00ccd70 0x7f3eb0155490 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000ec80 tx=0x7f3ea00091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.011+0000 7f3eb57ff700 1 -- 172.21.15.27:0/712697593 shutdown_connections 2024-01-11T08:26:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.011+0000 7f3eb57ff700 1 --2- 172.21.15.27:0/712697593 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3e9c066f30 0x7f3e9c0693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.011+0000 7f3eb57ff700 1 --2- 172.21.15.27:0/712697593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb015ae70 0x7f3eb00c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.011+0000 7f3eb57ff700 1 --2- 172.21.15.27:0/712697593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eb00ccd70 0x7f3eb0155490 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.011+0000 7f3eb57ff700 1 --2- 172.21.15.27:0/712697593 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3eb015a1b0 0x7f3eb0158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.011+0000 7f3eb57ff700 1 -- 172.21.15.27:0/712697593 >> 172.21.15.27:0/712697593 conn(0x7f3eb006ed40 msgr2=0x7f3eb0155e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.012+0000 7f3eb57ff700 1 -- 172.21.15.27:0/712697593 shutdown_connections 2024-01-11T08:26:39.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.012+0000 7f3eb57ff700 1 -- 172.21.15.27:0/712697593 wait complete. 2024-01-11T08:26:39.017 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 12 and data pool 13 2024-01-11T08:26:39.052 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:26:39.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.254+0000 7fe8225ff700 1 Processor -- start 2024-01-11T08:26:39.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.254+0000 7fe8225ff700 1 -- start start 2024-01-11T08:26:39.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.255+0000 7fe8225ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe81c0b03e0 0x7fe81c0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:39.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.256+0000 7fe8225ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe81c0b07b0 0x7fe81c0b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:39.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.256+0000 7fe8225ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe81c0cd730 con 0x7fe81c159200 2024-01-11T08:26:39.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.256+0000 7fe8225ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe81c159d30 con 0x7fe81c0b03e0 2024-01-11T08:26:39.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.256+0000 7fe8225ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe81c0ccb20 con 0x7fe81c0b07b0 2024-01-11T08:26:39.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.256+0000 7fe81bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe81c0b03e0 0x7fe81c0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:39.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.256+0000 7fe820b9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe81c159200 0x7fe81c159620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47444/0 (socket says 172.21.15.27:47444) 2024-01-11T08:26:39.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.256+0000 7fe820b9c700 1 -- 172.21.15.27:0/3306996081 learned_addr learned my addr 172.21.15.27:0/3306996081 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:39.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.256+0000 7fe81bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe81c0b03e0 0x7fe81c0b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41402/0 (socket says 172.21.15.27:41402) 2024-01-11T08:26:39.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.256+0000 7fe81b7fe700 1 --2- 172.21.15.27:0/3306996081 >> v2:172.21.15.77:3300/0 conn(0x7fe81c0b07b0 0x7fe81c0b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:39.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.256+0000 7fe81bfff700 1 -- 172.21.15.27:0/3306996081 >> v2:172.21.15.77:3300/0 conn(0x7fe81c0b07b0 msgr2=0x7fe81c0b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:39.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.256+0000 7fe81bfff700 1 --2- 172.21.15.27:0/3306996081 >> v2:172.21.15.77:3300/0 conn(0x7fe81c0b07b0 0x7fe81c0b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.256+0000 7fe81bfff700 1 -- 172.21.15.27:0/3306996081 >> v1:172.21.15.27:6789/0 conn(0x7fe81c159200 legacy=0x7fe81c159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:39.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.256+0000 7fe81bfff700 1 -- 172.21.15.27:0/3306996081 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81c0cce00 con 0x7fe81c0b03e0 2024-01-11T08:26:39.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.257+0000 7fe81bfff700 1 --2- 172.21.15.27:0/3306996081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c0b03e0 0x7fe81c0b2480 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0cdfc0 tx=0x7fe80c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77dd500165eba821 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:39.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.257+0000 7fe81affd700 1 -- 172.21.15.27:0/3306996081 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c00b6b0 con 0x7fe81c0b03e0 2024-01-11T08:26:39.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.257+0000 7fe81affd700 1 -- 172.21.15.27:0/3306996081 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80c00b810 con 0x7fe81c0b03e0 2024-01-11T08:26:39.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.257+0000 7fe81affd700 1 -- 172.21.15.27:0/3306996081 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c00bbb0 con 0x7fe81c0b03e0 2024-01-11T08:26:39.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.257+0000 7fe8225ff700 1 -- 172.21.15.27:0/3306996081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c0b03e0 msgr2=0x7fe81c0b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:39.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.257+0000 7fe8225ff700 1 --2- 172.21.15.27:0/3306996081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c0b03e0 0x7fe81c0b2480 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0cdfc0 tx=0x7fe80c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.257+0000 7fe8225ff700 1 -- 172.21.15.27:0/3306996081 shutdown_connections 2024-01-11T08:26:39.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.257+0000 7fe8225ff700 1 --2- 172.21.15.27:0/3306996081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c0b03e0 0x7fe81c0b2480 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.257+0000 7fe8225ff700 1 --2- 172.21.15.27:0/3306996081 >> v2:172.21.15.77:3300/0 conn(0x7fe81c0b07b0 0x7fe81c0b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.257+0000 7fe8225ff700 1 -- 172.21.15.27:0/3306996081 >> 172.21.15.27:0/3306996081 conn(0x7fe81c06ed40 msgr2=0x7fe81c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:39.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.258+0000 7fe8225ff700 1 -- 172.21.15.27:0/3306996081 shutdown_connections 2024-01-11T08:26:39.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.258+0000 7fe8225ff700 1 -- 172.21.15.27:0/3306996081 wait complete. 2024-01-11T08:26:39.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.259+0000 7fe8225ff700 1 Processor -- start 2024-01-11T08:26:39.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.259+0000 7fe8225ff700 1 -- start start 2024-01-11T08:26:39.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.260+0000 7fe8225ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe81c0b03e0 0x7fe81c1581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:39.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.260+0000 7fe8225ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c0b07b0 0x7fe81c169930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:39.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.260+0000 7fe8225ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe81c159200 0x7fe81c0c6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:39.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.260+0000 7fe8225ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c158730 con 0x7fe81c0b07b0 2024-01-11T08:26:39.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.260+0000 7fe8225ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c0cfe30 con 0x7fe81c0b03e0 2024-01-11T08:26:39.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.260+0000 7fe8225ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c169e70 con 0x7fe81c159200 2024-01-11T08:26:39.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.260+0000 7fe81b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c0b07b0 0x7fe81c169930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:39.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.260+0000 7fe81b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c0b07b0 0x7fe81c169930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41404/0 (socket says 172.21.15.27:41404) 2024-01-11T08:26:39.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.260+0000 7fe81b7fe700 1 -- 172.21.15.27:0/2608018684 learned_addr learned my addr 172.21.15.27:0/2608018684 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:39.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.261+0000 7fe81bfff700 1 --2- 172.21.15.27:0/2608018684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe81c0b03e0 0x7fe81c1581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:39.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.261+0000 7fe820b9c700 1 --2- 172.21.15.27:0/2608018684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe81c159200 0x7fe81c0c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:39.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.261+0000 7fe81b7fe700 1 -- 172.21.15.27:0/2608018684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe81c159200 msgr2=0x7fe81c0c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:39.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.261+0000 7fe81b7fe700 1 --2- 172.21.15.27:0/2608018684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe81c159200 0x7fe81c0c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.261+0000 7fe81b7fe700 1 -- 172.21.15.27:0/2608018684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe81c0b03e0 msgr2=0x7fe81c1581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:39.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.261+0000 7fe81b7fe700 1 --2- 172.21.15.27:0/2608018684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe81c0b03e0 0x7fe81c1581f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.261+0000 7fe81b7fe700 1 -- 172.21.15.27:0/2608018684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c009420 con 0x7fe81c0b07b0 2024-01-11T08:26:39.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.261+0000 7fe81b7fe700 1 --2- 172.21.15.27:0/2608018684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c0b07b0 0x7fe81c169930 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe81000c7d0 tx=0x7fe810008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:39.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.261+0000 7fe803fff700 1 -- 172.21.15.27:0/2608018684 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe810011600 con 0x7fe81c0b07b0 2024-01-11T08:26:39.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.261+0000 7fe803fff700 1 -- 172.21.15.27:0/2608018684 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe810011760 con 0x7fe81c0b07b0 2024-01-11T08:26:39.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.261+0000 7fe803fff700 1 -- 172.21.15.27:0/2608018684 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe810011b20 con 0x7fe81c0b07b0 2024-01-11T08:26:39.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.261+0000 7fe8225ff700 1 -- 172.21.15.27:0/2608018684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe81c0cf220 con 0x7fe81c0b07b0 2024-01-11T08:26:39.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.262+0000 7fe8225ff700 1 -- 172.21.15.27:0/2608018684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe81c0cf7e0 con 0x7fe81c0b07b0 2024-01-11T08:26:39.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.263+0000 7fe8225ff700 1 -- 172.21.15.27:0/2608018684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe81c0cffa0 con 0x7fe81c0b07b0 2024-01-11T08:26:39.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.264+0000 7fe803fff700 1 -- 172.21.15.27:0/2608018684 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe8100193f0 con 0x7fe81c0b07b0 2024-01-11T08:26:39.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.265+0000 7fe803fff700 1 --2- 172.21.15.27:0/2608018684 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe808066f30 0x7fe8080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:39.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.265+0000 7fe803fff700 1 -- 172.21.15.27:0/2608018684 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5733+0+0 (secure 0 0 0) 0x7fe810084850 con 0x7fe81c0b07b0 2024-01-11T08:26:39.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.265+0000 7fe81bfff700 1 --2- 172.21.15.27:0/2608018684 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe808066f30 0x7fe8080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:39.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.266+0000 7fe81bfff700 1 --2- 172.21.15.27:0/2608018684 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe808066f30 0x7fe8080693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe80c0093f0 tx=0x7fe80c017000 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:39.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.269+0000 7fe803fff700 1 -- 172.21.15.27:0/2608018684 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe810051aa0 con 0x7fe81c0b07b0 2024-01-11T08:26:39.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.472+0000 7fe8225ff700 1 -- 172.21.15.27:0/2608018684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe81c0718d0 con 0x7fe81c0b07b0 2024-01-11T08:26:39.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.473+0000 7fe803fff700 1 -- 172.21.15.27:0/2608018684 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+15364 (secure 0 0 0) 0x7fe810084070 con 0x7fe81c0b07b0 2024-01-11T08:26:39.472 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:39.472 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":100,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:26:38.513969+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:26:36.329359+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-11T08:26:37.504889+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:26:39.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.475+0000 7fe8225ff700 1 -- 172.21.15.27:0/2608018684 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe808066f30 msgr2=0x7fe8080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:39.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.475+0000 7fe8225ff700 1 --2- 172.21.15.27:0/2608018684 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe808066f30 0x7fe8080693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe80c0093f0 tx=0x7fe80c017000 comp rx=0 tx=0).stop 2024-01-11T08:26:39.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.475+0000 7fe8225ff700 1 -- 172.21.15.27:0/2608018684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c0b07b0 msgr2=0x7fe81c169930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:39.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.475+0000 7fe8225ff700 1 --2- 172.21.15.27:0/2608018684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c0b07b0 0x7fe81c169930 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe81000c7d0 tx=0x7fe810008040 comp rx=0 tx=0).stop 2024-01-11T08:26:39.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.475+0000 7fe8225ff700 1 -- 172.21.15.27:0/2608018684 shutdown_connections 2024-01-11T08:26:39.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.475+0000 7fe8225ff700 1 --2- 172.21.15.27:0/2608018684 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe808066f30 0x7fe8080693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.475+0000 7fe8225ff700 1 --2- 172.21.15.27:0/2608018684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe81c159200 0x7fe81c0c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.475+0000 7fe8225ff700 1 --2- 172.21.15.27:0/2608018684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe81c0b03e0 0x7fe81c1581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.475+0000 7fe8225ff700 1 --2- 172.21.15.27:0/2608018684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe81c0b07b0 0x7fe81c169930 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.475+0000 7fe8225ff700 1 -- 172.21.15.27:0/2608018684 >> 172.21.15.27:0/2608018684 conn(0x7fe81c06ed40 msgr2=0x7fe81c15c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:39.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.475+0000 7fe8225ff700 1 -- 172.21.15.27:0/2608018684 shutdown_connections 2024-01-11T08:26:39.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.475+0000 7fe8225ff700 1 -- 172.21.15.27:0/2608018684 wait complete. 2024-01-11T08:26:39.492 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:26:39.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.693+0000 7f167d9ff700 1 Processor -- start 2024-01-11T08:26:39.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.693+0000 7f167d9ff700 1 -- start start 2024-01-11T08:26:39.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.694+0000 7f167d9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f16780b2b90 0x7f1678159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:39.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.694+0000 7f167d9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16780b07b0 0x7f1678159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:39.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.694+0000 7f167d9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16780cd730 con 0x7f16780b03e0 2024-01-11T08:26:39.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.694+0000 7f167d9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16780ccb20 con 0x7f16780b2b90 2024-01-11T08:26:39.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.694+0000 7f167d9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16780cce00 con 0x7f16780b07b0 2024-01-11T08:26:39.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.695+0000 7f16777fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16780b07b0 0x7f1678159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:39.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.695+0000 7f1676ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f16780b03e0 0x7f16780b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47460/0 (socket says 172.21.15.27:47460) 2024-01-11T08:26:39.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.695+0000 7f1676ffd700 1 -- 172.21.15.27:0/4040954073 learned_addr learned my addr 172.21.15.27:0/4040954073 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:39.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.695+0000 7f16767fc700 1 --2- 172.21.15.27:0/4040954073 >> v2:172.21.15.77:3300/0 conn(0x7f16780b2b90 0x7f1678159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:39.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.695+0000 7f1675ffb700 1 -- 172.21.15.27:0/4040954073 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579232103 0 0) 0x7f16780cd730 con 0x7f16780b03e0 2024-01-11T08:26:39.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.695+0000 7f16777fe700 1 -- 172.21.15.27:0/4040954073 >> v2:172.21.15.77:3300/0 conn(0x7f16780b2b90 msgr2=0x7f1678159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:39.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.695+0000 7f16777fe700 1 --2- 172.21.15.27:0/4040954073 >> v2:172.21.15.77:3300/0 conn(0x7f16780b2b90 0x7f1678159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.695+0000 7f16777fe700 1 -- 172.21.15.27:0/4040954073 >> v1:172.21.15.27:6789/0 conn(0x7f16780b03e0 legacy=0x7f16780b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:39.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.695+0000 7f16777fe700 1 -- 172.21.15.27:0/4040954073 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16780cd0e0 con 0x7f16780b07b0 2024-01-11T08:26:39.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.695+0000 7f16777fe700 1 --2- 172.21.15.27:0/4040954073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16780b07b0 0x7f1678159740 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1670009e60 tx=0x7f1670009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60a6db59527b9e1c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:39.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.696+0000 7f1675ffb700 1 -- 172.21.15.27:0/4040954073 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167000e070 con 0x7f16780b07b0 2024-01-11T08:26:39.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.696+0000 7f1675ffb700 1 -- 172.21.15.27:0/4040954073 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167000bb60 con 0x7f16780b07b0 2024-01-11T08:26:39.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.696+0000 7f1675ffb700 1 -- 172.21.15.27:0/4040954073 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1670012430 con 0x7f16780b07b0 2024-01-11T08:26:39.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.696+0000 7f167d9ff700 1 -- 172.21.15.27:0/4040954073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16780b07b0 msgr2=0x7f1678159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:39.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.696+0000 7f167d9ff700 1 --2- 172.21.15.27:0/4040954073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16780b07b0 0x7f1678159740 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1670009e60 tx=0x7f1670009780 comp rx=0 tx=0).stop 2024-01-11T08:26:39.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.696+0000 7f167d9ff700 1 -- 172.21.15.27:0/4040954073 shutdown_connections 2024-01-11T08:26:39.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.696+0000 7f167d9ff700 1 --2- 172.21.15.27:0/4040954073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16780b07b0 0x7f1678159740 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.696+0000 7f167d9ff700 1 --2- 172.21.15.27:0/4040954073 >> v2:172.21.15.77:3300/0 conn(0x7f16780b2b90 0x7f1678159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.696+0000 7f167d9ff700 1 -- 172.21.15.27:0/4040954073 >> 172.21.15.27:0/4040954073 conn(0x7f167806ed40 msgr2=0x7f167806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:39.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.697+0000 7f167d9ff700 1 -- 172.21.15.27:0/4040954073 shutdown_connections 2024-01-11T08:26:39.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.697+0000 7f167d9ff700 1 -- 172.21.15.27:0/4040954073 wait complete. 2024-01-11T08:26:39.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.698+0000 7f167d9ff700 1 Processor -- start 2024-01-11T08:26:39.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.698+0000 7f167d9ff700 1 -- start start 2024-01-11T08:26:39.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.699+0000 7f167d9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16780b03e0 0x7f1678169790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:39.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.699+0000 7f167d9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16780b07b0 0x7f16780c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:39.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.699+0000 7f167d9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16780b2b90 0x7f1678147310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:39.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.699+0000 7f167d9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167816a700 con 0x7f16780b03e0 2024-01-11T08:26:39.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.699+0000 7f167d9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16780c5310 con 0x7f16780b2b90 2024-01-11T08:26:39.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.699+0000 7f167d9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f167816a140 con 0x7f16780b07b0 2024-01-11T08:26:39.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.699+0000 7f1676ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16780b03e0 0x7f1678169790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:39.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.699+0000 7f1676ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16780b03e0 0x7f1678169790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41406/0 (socket says 172.21.15.27:41406) 2024-01-11T08:26:39.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.699+0000 7f1676ffd700 1 -- 172.21.15.27:0/2608788187 learned_addr learned my addr 172.21.15.27:0/2608788187 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:39.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.700+0000 7f16767fc700 1 --2- 172.21.15.27:0/2608788187 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16780b07b0 0x7f16780c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:39.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.700+0000 7f16777fe700 1 --2- 172.21.15.27:0/2608788187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16780b2b90 0x7f1678147310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:39.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.700+0000 7f1676ffd700 1 -- 172.21.15.27:0/2608788187 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16780b07b0 msgr2=0x7f16780c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:39.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.700+0000 7f1676ffd700 1 --2- 172.21.15.27:0/2608788187 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16780b07b0 0x7f16780c57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.700+0000 7f1676ffd700 1 -- 172.21.15.27:0/2608788187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16780b2b90 msgr2=0x7f1678147310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:39.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.700+0000 7f1676ffd700 1 --2- 172.21.15.27:0/2608788187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16780b2b90 0x7f1678147310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.700+0000 7f1676ffd700 1 -- 172.21.15.27:0/2608788187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1670009a20 con 0x7f16780b03e0 2024-01-11T08:26:39.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.700+0000 7f16777fe700 1 --2- 172.21.15.27:0/2608788187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16780b2b90 0x7f1678147310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:39.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.700+0000 7f1676ffd700 1 --2- 172.21.15.27:0/2608788187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16780b03e0 0x7f1678169790 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f166800c810 tx=0x7f1668007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:39.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.700+0000 7f165effd700 1 -- 172.21.15.27:0/2608788187 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1668005b30 con 0x7f16780b03e0 2024-01-11T08:26:39.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.700+0000 7f165effd700 1 -- 172.21.15.27:0/2608788187 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1668005c90 con 0x7f16780b03e0 2024-01-11T08:26:39.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.700+0000 7f165effd700 1 -- 172.21.15.27:0/2608788187 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1668007300 con 0x7f16780b03e0 2024-01-11T08:26:39.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.700+0000 7f167d9ff700 1 -- 172.21.15.27:0/2608788187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f167816a420 con 0x7f16780b03e0 2024-01-11T08:26:39.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.701+0000 7f167d9ff700 1 -- 172.21.15.27:0/2608788187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1678169cd0 con 0x7f16780b03e0 2024-01-11T08:26:39.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.702+0000 7f167d9ff700 1 -- 172.21.15.27:0/2608788187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16780c55f0 con 0x7f16780b03e0 2024-01-11T08:26:39.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.702+0000 7f165effd700 1 -- 172.21.15.27:0/2608788187 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f166801b400 con 0x7f16780b03e0 2024-01-11T08:26:39.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.704+0000 7f165effd700 1 --2- 172.21.15.27:0/2608788187 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1664066f30 0x7f16640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:39.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.704+0000 7f165effd700 1 -- 172.21.15.27:0/2608788187 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f1668084060 con 0x7f16780b03e0 2024-01-11T08:26:39.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.704+0000 7f16767fc700 1 --2- 172.21.15.27:0/2608788187 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1664066f30 0x7f16640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:39.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.705+0000 7f16767fc700 1 --2- 172.21.15.27:0/2608788187 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1664066f30 0x7f16640693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f16781593c0 tx=0x7f166c008040 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:39.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.707+0000 7f165effd700 1 -- 172.21.15.27:0/2608788187 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1668051290 con 0x7f16780b03e0 2024-01-11T08:26:39.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.915+0000 7f167d9ff700 1 -- 172.21.15.27:0/2608788187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16780718d0 con 0x7f16780b03e0 2024-01-11T08:26:39.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.916+0000 7f165effd700 1 -- 172.21.15.27:0/2608788187 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+15364 (secure 0 0 0) 0x7f16680548b0 con 0x7f16780b03e0 2024-01-11T08:26:39.915 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:39.915 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":100,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:26:38.513969+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:26:36.329359+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-11T08:26:37.504889+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:26:39.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.918+0000 7f167d9ff700 1 -- 172.21.15.27:0/2608788187 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1664066f30 msgr2=0x7f16640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:39.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.918+0000 7f167d9ff700 1 --2- 172.21.15.27:0/2608788187 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1664066f30 0x7f16640693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f16781593c0 tx=0x7f166c008040 comp rx=0 tx=0).stop 2024-01-11T08:26:39.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.918+0000 7f167d9ff700 1 -- 172.21.15.27:0/2608788187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16780b03e0 msgr2=0x7f1678169790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:39.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.918+0000 7f167d9ff700 1 --2- 172.21.15.27:0/2608788187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16780b03e0 0x7f1678169790 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f166800c810 tx=0x7f1668007df0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.918+0000 7f167d9ff700 1 -- 172.21.15.27:0/2608788187 shutdown_connections 2024-01-11T08:26:39.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.918+0000 7f167d9ff700 1 --2- 172.21.15.27:0/2608788187 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1664066f30 0x7f16640693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.918+0000 7f167d9ff700 1 --2- 172.21.15.27:0/2608788187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16780b2b90 0x7f1678147310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.918+0000 7f167d9ff700 1 --2- 172.21.15.27:0/2608788187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16780b03e0 0x7f1678169790 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.918+0000 7f167d9ff700 1 --2- 172.21.15.27:0/2608788187 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16780b07b0 0x7f16780c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:39.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.918+0000 7f167d9ff700 1 -- 172.21.15.27:0/2608788187 >> 172.21.15.27:0/2608788187 conn(0x7f167806ed40 msgr2=0x7f16780b4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:39.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.918+0000 7f167d9ff700 1 -- 172.21.15.27:0/2608788187 shutdown_connections 2024-01-11T08:26:39.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:39.918+0000 7f167d9ff700 1 -- 172.21.15.27:0/2608788187 wait complete. 2024-01-11T08:26:39.935 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-11T08:26:39.981 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-01-11T08:26:39.982+0000 7feddfbf1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T08:26:40.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.139+0000 7fa7bebff700 1 Processor -- start 2024-01-11T08:26:40.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.140+0000 7fa7bebff700 1 -- start start 2024-01-11T08:26:40.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.141+0000 7fa7bebff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7b8069bd0 0x7fa7b80aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:40.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.142+0000 7fa7bebff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa7b80c6de0 0x7fa7b80c7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:40.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.142+0000 7fa7bebff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b806e480 con 0x7fa7b8155c50 2024-01-11T08:26:40.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.142+0000 7fa7bebff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b80cb5a0 con 0x7fa7b80c6de0 2024-01-11T08:26:40.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.142+0000 7fa7bebff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b806df00 con 0x7fa7b8069bd0 2024-01-11T08:26:40.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.142+0000 7fa7bd19c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa7b80c6de0 0x7fa7b80c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:40.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.142+0000 7fa7b7fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7b8069bd0 0x7fa7b80aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:40.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.142+0000 7fa7bc99b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa7b8155c50 0x7fa7b80694c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:54538/0 (socket says 172.21.15.27:54538) 2024-01-11T08:26:40.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.142+0000 7fa7bc99b700 1 -- 172.21.15.27:0/2071974527 learned_addr learned my addr 172.21.15.27:0/2071974527 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:40.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.143+0000 7fa7b77fe700 1 -- 172.21.15.27:0/2071974527 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133714466 0 0) 0x7fa7b806e480 con 0x7fa7b8155c50 2024-01-11T08:26:40.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.143+0000 7fa7b77fe700 1 -- 172.21.15.27:0/2071974527 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a8003570 con 0x7fa7b8155c50 2024-01-11T08:26:40.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.143+0000 7fa7b7fff700 1 -- 172.21.15.27:0/2071974527 >> v2:172.21.15.77:3300/0 conn(0x7fa7b80c6de0 msgr2=0x7fa7b80c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:40.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.143+0000 7fa7b7fff700 1 --2- 172.21.15.27:0/2071974527 >> v2:172.21.15.77:3300/0 conn(0x7fa7b80c6de0 0x7fa7b80c7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:40.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.143+0000 7fa7b7fff700 1 -- 172.21.15.27:0/2071974527 >> v1:172.21.15.77:6789/0 conn(0x7fa7b8155c50 legacy=0x7fa7b80694c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:40.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.143+0000 7fa7b7fff700 1 -- 172.21.15.27:0/2071974527 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b80c77c0 con 0x7fa7b8069bd0 2024-01-11T08:26:40.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.143+0000 7fa7bd19c700 1 --2- 172.21.15.27:0/2071974527 >> v2:172.21.15.77:3300/0 conn(0x7fa7b80c6de0 0x7fa7b80c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:40.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.143+0000 7fa7b7fff700 1 --2- 172.21.15.27:0/2071974527 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7b8069bd0 0x7fa7b80aef80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40098e0 tx=0x7fa7a40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5af21831ac5df857 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:40.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.143+0000 7fa7b77fe700 1 -- 172.21.15.27:0/2071974527 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4011630 con 0x7fa7b8069bd0 2024-01-11T08:26:40.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.143+0000 7fa7b77fe700 1 -- 172.21.15.27:0/2071974527 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a4011790 con 0x7fa7b8069bd0 2024-01-11T08:26:40.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.143+0000 7fa7b77fe700 1 -- 172.21.15.27:0/2071974527 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4011b30 con 0x7fa7b8069bd0 2024-01-11T08:26:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.144+0000 7fa7bebff700 1 -- 172.21.15.27:0/2071974527 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7b8069bd0 msgr2=0x7fa7b80aef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.144+0000 7fa7bebff700 1 --2- 172.21.15.27:0/2071974527 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7b8069bd0 0x7fa7b80aef80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40098e0 tx=0x7fa7a40091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.144+0000 7fa7bebff700 1 -- 172.21.15.27:0/2071974527 shutdown_connections 2024-01-11T08:26:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.144+0000 7fa7bebff700 1 --2- 172.21.15.27:0/2071974527 >> v2:172.21.15.77:3300/0 conn(0x7fa7b80c6de0 0x7fa7b80c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.144+0000 7fa7bebff700 1 --2- 172.21.15.27:0/2071974527 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7b8069bd0 0x7fa7b80aef80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.144+0000 7fa7bebff700 1 -- 172.21.15.27:0/2071974527 >> 172.21.15.27:0/2071974527 conn(0x7fa7b80680a0 msgr2=0x7fa7b80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.144+0000 7fa7bebff700 1 -- 172.21.15.27:0/2071974527 shutdown_connections 2024-01-11T08:26:40.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.144+0000 7fa7bebff700 1 -- 172.21.15.27:0/2071974527 wait complete. 2024-01-11T08:26:40.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.146+0000 7fa7bebff700 1 Processor -- start 2024-01-11T08:26:40.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.146+0000 7fa7bebff700 1 -- start start 2024-01-11T08:26:40.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.148+0000 7fa7bebff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7b8069bd0 0x7fa7b81652b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:40.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.148+0000 7fa7bebff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7b80c6de0 0x7fa7b8141360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:40.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.148+0000 7fa7bebff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7b8155c50 0x7fa7b81418a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:40.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.148+0000 7fa7bebff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b8141fe0 con 0x7fa7b8069bd0 2024-01-11T08:26:40.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.148+0000 7fa7bebff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b8166d90 con 0x7fa7b80c6de0 2024-01-11T08:26:40.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.148+0000 7fa7bebff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b8166180 con 0x7fa7b8155c50 2024-01-11T08:26:40.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.148+0000 7fa7bc99b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7b8069bd0 0x7fa7b81652b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:40.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.148+0000 7fa7bc99b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7b8069bd0 0x7fa7b81652b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41412/0 (socket says 172.21.15.27:41412) 2024-01-11T08:26:40.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.148+0000 7fa7bc99b700 1 -- 172.21.15.27:0/1593600445 learned_addr learned my addr 172.21.15.27:0/1593600445 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:40.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.148+0000 7fa7bd19c700 1 --2- 172.21.15.27:0/1593600445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7b8155c50 0x7fa7b81418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:40.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.148+0000 7fa7b7fff700 1 --2- 172.21.15.27:0/1593600445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7b80c6de0 0x7fa7b8141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:40.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.148+0000 7fa7bc99b700 1 -- 172.21.15.27:0/1593600445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7b8155c50 msgr2=0x7fa7b81418a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:40.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.148+0000 7fa7bc99b700 1 --2- 172.21.15.27:0/1593600445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7b8155c50 0x7fa7b81418a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:40.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.148+0000 7fa7bc99b700 1 -- 172.21.15.27:0/1593600445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7b80c6de0 msgr2=0x7fa7b8141360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:40.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.148+0000 7fa7bc99b700 1 --2- 172.21.15.27:0/1593600445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7b80c6de0 0x7fa7b8141360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:40.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.148+0000 7fa7bc99b700 1 -- 172.21.15.27:0/1593600445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b000ab30 con 0x7fa7b8069bd0 2024-01-11T08:26:40.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.149+0000 7fa7bc99b700 1 --2- 172.21.15.27:0/1593600445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7b8069bd0 0x7fa7b81652b0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac001230 tx=0x7fa7ac007e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:40.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.149+0000 7fa7b4ff9700 1 -- 172.21.15.27:0/1593600445 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac005b80 con 0x7fa7b8069bd0 2024-01-11T08:26:40.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.149+0000 7fa7b4ff9700 1 -- 172.21.15.27:0/1593600445 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7ac005ce0 con 0x7fa7b8069bd0 2024-01-11T08:26:40.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.149+0000 7fa7bebff700 1 -- 172.21.15.27:0/1593600445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a4009420 con 0x7fa7b8069bd0 2024-01-11T08:26:40.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.149+0000 7fa7b4ff9700 1 -- 172.21.15.27:0/1593600445 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac007320 con 0x7fa7b8069bd0 2024-01-11T08:26:40.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.149+0000 7fa7bebff700 1 -- 172.21.15.27:0/1593600445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b8166740 con 0x7fa7b8069bd0 2024-01-11T08:26:40.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.150+0000 7fa7bebff700 1 -- 172.21.15.27:0/1593600445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b804c3c0 con 0x7fa7b8069bd0 2024-01-11T08:26:40.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.151+0000 7fa7b4ff9700 1 -- 172.21.15.27:0/1593600445 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa7ac0076d0 con 0x7fa7b8069bd0 2024-01-11T08:26:40.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.152+0000 7fa7b4ff9700 1 --2- 172.21.15.27:0/1593600445 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa79c066f30 0x7fa79c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:40.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.152+0000 7fa7b4ff9700 1 -- 172.21.15.27:0/1593600445 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5733+0+0 (secure 0 0 0) 0x7fa7ac0867b0 con 0x7fa7b8069bd0 2024-01-11T08:26:40.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.153+0000 7fa7b7fff700 1 --2- 172.21.15.27:0/1593600445 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa79c066f30 0x7fa79c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:40.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.153+0000 7fa7b7fff700 1 --2- 172.21.15.27:0/1593600445 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa79c066f30 0x7fa79c0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40091e0 tx=0x7fa7a4017000 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:40.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.157+0000 7fa7b4ff9700 1 -- 172.21.15.27:0/1593600445 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa7ac053a80 con 0x7fa7b8069bd0 2024-01-11T08:26:40.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:40.406+0000 7fa7bebff700 1 -- 172.21.15.27:0/1593600445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa7b80b1810 con 0x7fa7b8069bd0 2024-01-11T08:26:41.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.030+0000 7fa7b4ff9700 1 -- 172.21.15.27:0/1593600445 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v49) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa7ac0150b0 con 0x7fa7b8069bd0 2024-01-11T08:26:41.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.033+0000 7fa7bebff700 1 -- 172.21.15.27:0/1593600445 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa79c066f30 msgr2=0x7fa79c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:41.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.033+0000 7fa7bebff700 1 --2- 172.21.15.27:0/1593600445 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa79c066f30 0x7fa79c0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40091e0 tx=0x7fa7a4017000 comp rx=0 tx=0).stop 2024-01-11T08:26:41.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.033+0000 7fa7bebff700 1 -- 172.21.15.27:0/1593600445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7b8069bd0 msgr2=0x7fa7b81652b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:41.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.033+0000 7fa7bebff700 1 --2- 172.21.15.27:0/1593600445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7b8069bd0 0x7fa7b81652b0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac001230 tx=0x7fa7ac007e90 comp rx=0 tx=0).stop 2024-01-11T08:26:41.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.033+0000 7fa7bebff700 1 -- 172.21.15.27:0/1593600445 shutdown_connections 2024-01-11T08:26:41.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.033+0000 7fa7bebff700 1 --2- 172.21.15.27:0/1593600445 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa79c066f30 0x7fa79c0693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:41.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.033+0000 7fa7bebff700 1 --2- 172.21.15.27:0/1593600445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7b8155c50 0x7fa7b81418a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:41.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.033+0000 7fa7bebff700 1 --2- 172.21.15.27:0/1593600445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7b8069bd0 0x7fa7b81652b0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:41.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.033+0000 7fa7bebff700 1 --2- 172.21.15.27:0/1593600445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7b80c6de0 0x7fa7b8141360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:41.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.033+0000 7fa7bebff700 1 -- 172.21.15.27:0/1593600445 >> 172.21.15.27:0/1593600445 conn(0x7fa7b80680a0 msgr2=0x7fa7b8154dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:41.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.033+0000 7fa7bebff700 1 -- 172.21.15.27:0/1593600445 shutdown_connections 2024-01-11T08:26:41.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.033+0000 7fa7bebff700 1 -- 172.21.15.27:0/1593600445 wait complete. 2024-01-11T08:26:41.053 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-11T08:26:41.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.301+0000 7f1cb07ff700 1 Processor -- start 2024-01-11T08:26:41.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.301+0000 7f1cb07ff700 1 -- start start 2024-01-11T08:26:41.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.302+0000 7f1cb07ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ca8142dc0 0x7f1ca80cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:41.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.302+0000 7f1cb07ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1ca80cb2e0 0x7f1ca80cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:41.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.302+0000 7f1cb07ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca80ccc70 con 0x7f1ca8143190 2024-01-11T08:26:41.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.302+0000 7f1cb07ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ca815df30 con 0x7f1ca80cb2e0 2024-01-11T08:26:41.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.302+0000 7f1cb07ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca815d320 con 0x7f1ca8142dc0 2024-01-11T08:26:41.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.303+0000 7f1cae59b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ca8142dc0 0x7f1ca80cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:41.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.303+0000 7f1cadd9a700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1ca8143190 0x7f1ca80cc020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:54544/0 (socket says 172.21.15.27:54544) 2024-01-11T08:26:41.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.303+0000 7f1cadd9a700 1 -- 172.21.15.27:0/2073170549 learned_addr learned my addr 172.21.15.27:0/2073170549 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:41.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.303+0000 7f1caed9c700 1 --2- 172.21.15.27:0/2073170549 >> v2:172.21.15.77:3300/0 conn(0x7f1ca80cb2e0 0x7f1ca80cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:41.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.304+0000 7f1cae59b700 1 -- 172.21.15.27:0/2073170549 >> v2:172.21.15.77:3300/0 conn(0x7f1ca80cb2e0 msgr2=0x7f1ca80cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:41.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.304+0000 7f1cae59b700 1 --2- 172.21.15.27:0/2073170549 >> v2:172.21.15.77:3300/0 conn(0x7f1ca80cb2e0 0x7f1ca80cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:41.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.304+0000 7f1cae59b700 1 -- 172.21.15.27:0/2073170549 >> v1:172.21.15.77:6789/0 conn(0x7f1ca8143190 legacy=0x7f1ca80cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:41.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.304+0000 7f1cae59b700 1 -- 172.21.15.27:0/2073170549 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca80cb700 con 0x7f1ca8142dc0 2024-01-11T08:26:41.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.304+0000 7f1caed9c700 1 --2- 172.21.15.27:0/2073170549 >> v2:172.21.15.77:3300/0 conn(0x7f1ca80cb2e0 0x7f1ca80cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:41.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.304+0000 7f1cae59b700 1 --2- 172.21.15.27:0/2073170549 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ca8142dc0 0x7f1ca80cada0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1ca40098e0 tx=0x7f1ca40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=418dbc327a4787f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.304+0000 7f1cad599700 1 -- 172.21.15.27:0/2073170549 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4011630 con 0x7f1ca8142dc0 2024-01-11T08:26:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.305+0000 7f1cad599700 1 -- 172.21.15.27:0/2073170549 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca4011790 con 0x7f1ca8142dc0 2024-01-11T08:26:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.305+0000 7f1cad599700 1 -- 172.21.15.27:0/2073170549 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4011b30 con 0x7f1ca8142dc0 2024-01-11T08:26:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.305+0000 7f1cb07ff700 1 -- 172.21.15.27:0/2073170549 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ca8142dc0 msgr2=0x7f1ca80cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.305+0000 7f1cb07ff700 1 --2- 172.21.15.27:0/2073170549 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ca8142dc0 0x7f1ca80cada0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1ca40098e0 tx=0x7f1ca40091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.305+0000 7f1cb07ff700 1 -- 172.21.15.27:0/2073170549 shutdown_connections 2024-01-11T08:26:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.305+0000 7f1cb07ff700 1 --2- 172.21.15.27:0/2073170549 >> v2:172.21.15.77:3300/0 conn(0x7f1ca80cb2e0 0x7f1ca80cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.305+0000 7f1cb07ff700 1 --2- 172.21.15.27:0/2073170549 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ca8142dc0 0x7f1ca80cada0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.305+0000 7f1cb07ff700 1 -- 172.21.15.27:0/2073170549 >> 172.21.15.27:0/2073170549 conn(0x7f1ca8063ca0 msgr2=0x7f1ca806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.305+0000 7f1cb07ff700 1 -- 172.21.15.27:0/2073170549 shutdown_connections 2024-01-11T08:26:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.306+0000 7f1cb07ff700 1 -- 172.21.15.27:0/2073170549 wait complete. 2024-01-11T08:26:41.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.307+0000 7f1cb07ff700 1 Processor -- start 2024-01-11T08:26:41.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.307+0000 7f1cb07ff700 1 -- start start 2024-01-11T08:26:41.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.308+0000 7f1cb07ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ca80cb2e0 0x7f1ca812e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:41.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.308+0000 7f1cb07ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ca8142dc0 0x7f1ca816d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:41.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.308+0000 7f1cb07ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ca8143190 0x7f1ca816d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:41.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.308+0000 7f1cb07ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca80c5fd0 con 0x7f1ca8142dc0 2024-01-11T08:26:41.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.308+0000 7f1cb07ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca816b5c0 con 0x7f1ca80cb2e0 2024-01-11T08:26:41.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.308+0000 7f1cb07ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca816afb0 con 0x7f1ca8143190 2024-01-11T08:26:41.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.308+0000 7f1cadd9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ca8142dc0 0x7f1ca816d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:41.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.308+0000 7f1cadd9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ca8142dc0 0x7f1ca816d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41428/0 (socket says 172.21.15.27:41428) 2024-01-11T08:26:41.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.309+0000 7f1cadd9a700 1 -- 172.21.15.27:0/806558254 learned_addr learned my addr 172.21.15.27:0/806558254 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:41.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.309+0000 7f1cae59b700 1 --2- 172.21.15.27:0/806558254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ca80cb2e0 0x7f1ca812e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:41.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.309+0000 7f1caed9c700 1 --2- 172.21.15.27:0/806558254 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ca8143190 0x7f1ca816d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:41.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.309+0000 7f1cadd9a700 1 -- 172.21.15.27:0/806558254 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ca8143190 msgr2=0x7f1ca816d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:41.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.309+0000 7f1cadd9a700 1 --2- 172.21.15.27:0/806558254 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ca8143190 0x7f1ca816d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:41.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.309+0000 7f1cadd9a700 1 -- 172.21.15.27:0/806558254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ca80cb2e0 msgr2=0x7f1ca812e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:41.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.309+0000 7f1cadd9a700 1 --2- 172.21.15.27:0/806558254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ca80cb2e0 0x7f1ca812e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:41.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.309+0000 7f1cadd9a700 1 -- 172.21.15.27:0/806558254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca0009a20 con 0x7f1ca8142dc0 2024-01-11T08:26:41.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.309+0000 7f1caed9c700 1 --2- 172.21.15.27:0/806558254 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ca8143190 0x7f1ca816d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:41.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.309+0000 7f1cae59b700 1 --2- 172.21.15.27:0/806558254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ca80cb2e0 0x7f1ca812e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:41.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.309+0000 7f1cadd9a700 1 --2- 172.21.15.27:0/806558254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ca8142dc0 0x7f1ca816d340 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800b5e0 tx=0x7f1c98009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:41.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.310+0000 7f1c9e7fc700 1 -- 172.21.15.27:0/806558254 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c98011630 con 0x7f1ca8142dc0 2024-01-11T08:26:41.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.310+0000 7f1c9e7fc700 1 -- 172.21.15.27:0/806558254 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c98011790 con 0x7f1ca8142dc0 2024-01-11T08:26:41.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.310+0000 7f1cb07ff700 1 -- 172.21.15.27:0/806558254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca4009420 con 0x7f1ca8142dc0 2024-01-11T08:26:41.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.310+0000 7f1c9e7fc700 1 -- 172.21.15.27:0/806558254 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c98011b50 con 0x7f1ca8142dc0 2024-01-11T08:26:41.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.310+0000 7f1cb07ff700 1 -- 172.21.15.27:0/806558254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca816b2d0 con 0x7f1ca8142dc0 2024-01-11T08:26:41.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.311+0000 7f1cb07ff700 1 -- 172.21.15.27:0/806558254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca816b770 con 0x7f1ca8142dc0 2024-01-11T08:26:41.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.312+0000 7f1c9e7fc700 1 -- 172.21.15.27:0/806558254 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1c9801a400 con 0x7f1ca8142dc0 2024-01-11T08:26:41.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.313+0000 7f1c9e7fc700 1 --2- 172.21.15.27:0/806558254 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1c8c066f30 0x7f1c8c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:41.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.313+0000 7f1c9e7fc700 1 -- 172.21.15.27:0/806558254 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f1c98084490 con 0x7f1ca8142dc0 2024-01-11T08:26:41.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.314+0000 7f1cae59b700 1 --2- 172.21.15.27:0/806558254 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1c8c066f30 0x7f1c8c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:41.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.315+0000 7f1cae59b700 1 --2- 172.21.15.27:0/806558254 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1c8c066f30 0x7f1c8c0693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1ca40030c0 tx=0x7f1ca4016000 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:41.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.318+0000 7f1c9e7fc700 1 -- 172.21.15.27:0/806558254 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1ca816b770 con 0x7f1ca8142dc0 2024-01-11T08:26:41.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:41.566+0000 7f1cb07ff700 1 -- 172.21.15.27:0/806558254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1ca816b730 con 0x7f1ca8142dc0 2024-01-11T08:26:42.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.036+0000 7f1c9e7fc700 1 -- 172.21.15.27:0/806558254 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v50) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1c98055230 con 0x7f1ca8142dc0 2024-01-11T08:26:42.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.038+0000 7f1cb07ff700 1 -- 172.21.15.27:0/806558254 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1c8c066f30 msgr2=0x7f1c8c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:42.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.038+0000 7f1cb07ff700 1 --2- 172.21.15.27:0/806558254 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1c8c066f30 0x7f1c8c0693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1ca40030c0 tx=0x7f1ca4016000 comp rx=0 tx=0).stop 2024-01-11T08:26:42.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.038+0000 7f1cb07ff700 1 -- 172.21.15.27:0/806558254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ca8142dc0 msgr2=0x7f1ca816d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:42.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.038+0000 7f1cb07ff700 1 --2- 172.21.15.27:0/806558254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ca8142dc0 0x7f1ca816d340 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800b5e0 tx=0x7f1c98009120 comp rx=0 tx=0).stop 2024-01-11T08:26:42.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.038+0000 7f1cb07ff700 1 -- 172.21.15.27:0/806558254 shutdown_connections 2024-01-11T08:26:42.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.038+0000 7f1cb07ff700 1 --2- 172.21.15.27:0/806558254 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1c8c066f30 0x7f1c8c0693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.038+0000 7f1cb07ff700 1 --2- 172.21.15.27:0/806558254 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ca8143190 0x7f1ca816d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.038+0000 7f1cb07ff700 1 --2- 172.21.15.27:0/806558254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ca80cb2e0 0x7f1ca812e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.038+0000 7f1cb07ff700 1 --2- 172.21.15.27:0/806558254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ca8142dc0 0x7f1ca816d340 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.038+0000 7f1cb07ff700 1 -- 172.21.15.27:0/806558254 >> 172.21.15.27:0/806558254 conn(0x7f1ca8063ca0 msgr2=0x7f1ca81420d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:42.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.038+0000 7f1cb07ff700 1 -- 172.21.15.27:0/806558254 shutdown_connections 2024-01-11T08:26:42.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.039+0000 7f1cb07ff700 1 -- 172.21.15.27:0/806558254 wait complete. 2024-01-11T08:26:42.057 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:26:42.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.259+0000 7f07f7dff700 1 Processor -- start 2024-01-11T08:26:42.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.259+0000 7f07f7dff700 1 -- start start 2024-01-11T08:26:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.261+0000 7f07f7dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f07f00ca7b0 0x7f07f00cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.261+0000 7f07f7dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f015d640 con 0x7f07f0159e80 2024-01-11T08:26:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.261+0000 7f07f7dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f015a5d0 con 0x7f07f00ca3e0 2024-01-11T08:26:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.261+0000 7f07f7dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07f015ae70 con 0x7f07f00ca7b0 2024-01-11T08:26:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.261+0000 7f07f639c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f07f0159e80 0x7f07f015dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47468/0 (socket says 172.21.15.27:47468) 2024-01-11T08:26:42.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.261+0000 7f07f639c700 1 -- 172.21.15.27:0/1447943996 learned_addr learned my addr 172.21.15.27:0/1447943996 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.261+0000 7f07f539a700 1 --2- 172.21.15.27:0/1447943996 >> v2:172.21.15.77:3300/0 conn(0x7f07f00ca7b0 0x7f07f00cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.262+0000 7f07f4b99700 1 -- 172.21.15.27:0/1447943996 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037533601 0 0) 0x7f07f015d640 con 0x7f07f0159e80 2024-01-11T08:26:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.262+0000 7f07f4b99700 1 -- 172.21.15.27:0/1447943996 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07dc003570 con 0x7f07f0159e80 2024-01-11T08:26:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.262+0000 7f07f4b99700 1 -- 172.21.15.27:0/1447943996 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685873685 0 0) 0x7f07f015a5d0 con 0x7f07f00ca3e0 2024-01-11T08:26:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.262+0000 7f07f4b99700 1 -- 172.21.15.27:0/1447943996 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07f015d640 con 0x7f07f00ca3e0 2024-01-11T08:26:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.262+0000 7f07f539a700 1 -- 172.21.15.27:0/1447943996 >> v1:172.21.15.77:6789/0 conn(0x7f07f00ca3e0 legacy=0x7f07f00c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.262+0000 7f07f539a700 1 -- 172.21.15.27:0/1447943996 >> v1:172.21.15.27:6789/0 conn(0x7f07f0159e80 legacy=0x7f07f015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.262+0000 7f07f539a700 1 -- 172.21.15.27:0/1447943996 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f015a8b0 con 0x7f07f00ca7b0 2024-01-11T08:26:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.262+0000 7f07f539a700 1 --2- 172.21.15.27:0/1447943996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f00ca7b0 0x7f07f00cca20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f07e0009910 tx=0x7f07e00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c4a66107fbcb369 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.263+0000 7f07f4b99700 1 -- 172.21.15.27:0/1447943996 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e0011630 con 0x7f07f00ca7b0 2024-01-11T08:26:42.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.263+0000 7f07f4b99700 1 -- 172.21.15.27:0/1447943996 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e0011790 con 0x7f07f00ca7b0 2024-01-11T08:26:42.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.263+0000 7f07f4b99700 1 -- 172.21.15.27:0/1447943996 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e0011b30 con 0x7f07f00ca7b0 2024-01-11T08:26:42.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.263+0000 7f07f7dff700 1 -- 172.21.15.27:0/1447943996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f00ca7b0 msgr2=0x7f07f00cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:42.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.263+0000 7f07f7dff700 1 --2- 172.21.15.27:0/1447943996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f00ca7b0 0x7f07f00cca20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f07e0009910 tx=0x7f07e00091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.263+0000 7f07f7dff700 1 -- 172.21.15.27:0/1447943996 shutdown_connections 2024-01-11T08:26:42.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.263+0000 7f07f7dff700 1 --2- 172.21.15.27:0/1447943996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f00ca7b0 0x7f07f00cca20 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.263+0000 7f07f7dff700 1 -- 172.21.15.27:0/1447943996 >> 172.21.15.27:0/1447943996 conn(0x7f07f00ce3e0 msgr2=0x7f07f00ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:42.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.263+0000 7f07f7dff700 1 -- 172.21.15.27:0/1447943996 shutdown_connections 2024-01-11T08:26:42.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.263+0000 7f07f7dff700 1 -- 172.21.15.27:0/1447943996 wait complete. 2024-01-11T08:26:42.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.265+0000 7f07f7dff700 1 Processor -- start 2024-01-11T08:26:42.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.266+0000 7f07f7dff700 1 -- start start 2024-01-11T08:26:42.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.267+0000 7f07f7dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07f00ca3e0 0x7f07f014b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:42.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.267+0000 7f07f7dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07f00ca7b0 0x7f07f00b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:42.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.267+0000 7f07f7dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f0159e80 0x7f07f00b38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:42.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.267+0000 7f07f7dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f014bc80 con 0x7f07f00ca7b0 2024-01-11T08:26:42.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.267+0000 7f07f7dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f00b3430 con 0x7f07f0159e80 2024-01-11T08:26:42.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.267+0000 7f07f7dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f00b2820 con 0x7f07f00ca3e0 2024-01-11T08:26:42.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.267+0000 7f07f539a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07f00ca7b0 0x7f07f00b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:42.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.267+0000 7f07f539a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07f00ca7b0 0x7f07f00b1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41444/0 (socket says 172.21.15.27:41444) 2024-01-11T08:26:42.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.267+0000 7f07f539a700 1 -- 172.21.15.27:0/84759878 learned_addr learned my addr 172.21.15.27:0/84759878 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:42.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.268+0000 7f07f639c700 1 --2- 172.21.15.27:0/84759878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f0159e80 0x7f07f00b38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:42.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.268+0000 7f07f5b9b700 1 --2- 172.21.15.27:0/84759878 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07f00ca3e0 0x7f07f014b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:42.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.268+0000 7f07f539a700 1 -- 172.21.15.27:0/84759878 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07f00ca3e0 msgr2=0x7f07f014b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:42.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.268+0000 7f07f539a700 1 --2- 172.21.15.27:0/84759878 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07f00ca3e0 0x7f07f014b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.268+0000 7f07f539a700 1 -- 172.21.15.27:0/84759878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f0159e80 msgr2=0x7f07f00b38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:42.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.268+0000 7f07f539a700 1 --2- 172.21.15.27:0/84759878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f0159e80 0x7f07f00b38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.268+0000 7f07f539a700 1 -- 172.21.15.27:0/84759878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e0009420 con 0x7f07f00ca7b0 2024-01-11T08:26:42.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.268+0000 7f07f639c700 1 --2- 172.21.15.27:0/84759878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f0159e80 0x7f07f00b38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:42.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.268+0000 7f07f5b9b700 1 --2- 172.21.15.27:0/84759878 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07f00ca3e0 0x7f07f014b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:42.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.268+0000 7f07f539a700 1 --2- 172.21.15.27:0/84759878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07f00ca7b0 0x7f07f00b1f80 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f07f0064980 tx=0x7f07e0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:42.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.268+0000 7f07e5ffb700 1 -- 172.21.15.27:0/84759878 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e001f070 con 0x7f07f00ca7b0 2024-01-11T08:26:42.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.268+0000 7f07e5ffb700 1 -- 172.21.15.27:0/84759878 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e0011900 con 0x7f07f00ca7b0 2024-01-11T08:26:42.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.268+0000 7f07f7dff700 1 -- 172.21.15.27:0/84759878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f00b2b00 con 0x7f07f00ca7b0 2024-01-11T08:26:42.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.268+0000 7f07e5ffb700 1 -- 172.21.15.27:0/84759878 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e0023410 con 0x7f07f00ca7b0 2024-01-11T08:26:42.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.269+0000 7f07f7dff700 1 -- 172.21.15.27:0/84759878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f00b30c0 con 0x7f07f00ca7b0 2024-01-11T08:26:42.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.270+0000 7f07f7dff700 1 -- 172.21.15.27:0/84759878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f00718d0 con 0x7f07f00ca7b0 2024-01-11T08:26:42.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.270+0000 7f07e5ffb700 1 -- 172.21.15.27:0/84759878 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f07e002c3b0 con 0x7f07f00ca7b0 2024-01-11T08:26:42.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.271+0000 7f07e5ffb700 1 --2- 172.21.15.27:0/84759878 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f07d4066f30 0x7f07d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:42.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.271+0000 7f07e5ffb700 1 -- 172.21.15.27:0/84759878 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f07e00857c0 con 0x7f07f00ca7b0 2024-01-11T08:26:42.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.272+0000 7f07f5b9b700 1 --2- 172.21.15.27:0/84759878 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f07d4066f30 0x7f07d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:42.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.273+0000 7f07f5b9b700 1 --2- 172.21.15.27:0/84759878 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f07d4066f30 0x7f07d40693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f07ec007fd0 tx=0x7f07ec009040 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:42.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.276+0000 7f07e5ffb700 1 -- 172.21.15.27:0/84759878 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f07e00529f0 con 0x7f07f00ca7b0 2024-01-11T08:26:42.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.525+0000 7f07f7dff700 1 -- 172.21.15.27:0/84759878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07f006c840 con 0x7f07f00ca7b0 2024-01-11T08:26:42.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.526+0000 7f07e5ffb700 1 -- 172.21.15.27:0/84759878 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4593 (secure 0 0 0) 0x7f07e0055f90 con 0x7f07f00ca7b0 2024-01-11T08:26:42.526 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:42.526 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":50,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/921942372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":921942372},{"type":"v1","addr":"172.21.15.27:6834","nonce":921942372}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7080,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/3844286168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3844286168},{"type":"v1","addr":"172.21.15.77:6834","nonce":3844286168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7083,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3222788452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":3222788452},{"type":"v1","addr":"172.21.15.77:6837","nonce":3222788452}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:26:38.514309+0000","modified":"2024-01-11T08:26:42.031140+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7089},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7089":{"gid":7089,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/751423743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":751423743},{"type":"v1","addr":"172.21.15.27:6835","nonce":751423743}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2024-01-11T08:26:42.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.528+0000 7f07f7dff700 1 -- 172.21.15.27:0/84759878 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f07d4066f30 msgr2=0x7f07d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.528+0000 7f07f7dff700 1 --2- 172.21.15.27:0/84759878 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f07d4066f30 0x7f07d40693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f07ec007fd0 tx=0x7f07ec009040 comp rx=0 tx=0).stop 2024-01-11T08:26:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.528+0000 7f07f7dff700 1 -- 172.21.15.27:0/84759878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07f00ca7b0 msgr2=0x7f07f00b1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.528+0000 7f07f7dff700 1 --2- 172.21.15.27:0/84759878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07f00ca7b0 0x7f07f00b1f80 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f07f0064980 tx=0x7f07e0017000 comp rx=0 tx=0).stop 2024-01-11T08:26:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.528+0000 7f07f7dff700 1 -- 172.21.15.27:0/84759878 shutdown_connections 2024-01-11T08:26:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.528+0000 7f07f7dff700 1 --2- 172.21.15.27:0/84759878 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f07d4066f30 0x7f07d40693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.528+0000 7f07f7dff700 1 --2- 172.21.15.27:0/84759878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f0159e80 0x7f07f00b38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.528+0000 7f07f7dff700 1 --2- 172.21.15.27:0/84759878 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07f00ca3e0 0x7f07f014b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.528+0000 7f07f7dff700 1 --2- 172.21.15.27:0/84759878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07f00ca7b0 0x7f07f00b1f80 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.528+0000 7f07f7dff700 1 -- 172.21.15.27:0/84759878 >> 172.21.15.27:0/84759878 conn(0x7f07f00ce3e0 msgr2=0x7f07f0152090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.528+0000 7f07f7dff700 1 -- 172.21.15.27:0/84759878 shutdown_connections 2024-01-11T08:26:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.528+0000 7f07f7dff700 1 -- 172.21.15.27:0/84759878 wait complete. 2024-01-11T08:26:42.530 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 50 2024-01-11T08:26:42.546 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:26:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.745+0000 7fa6185ff700 1 Processor -- start 2024-01-11T08:26:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.745+0000 7fa6185ff700 1 -- start start 2024-01-11T08:26:42.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.746+0000 7fa6185ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa610159db0 0x7fa61015a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:42.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.746+0000 7fa6185ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa61014fc20 0x7fa610150080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:42.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.746+0000 7fa6185ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa610157e30 con 0x7fa6101599e0 2024-01-11T08:26:42.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.746+0000 7fa6185ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa610157500 con 0x7fa61014fc20 2024-01-11T08:26:42.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.746+0000 7fa6185ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6101577e0 con 0x7fa610159db0 2024-01-11T08:26:42.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.747+0000 7fa61639b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa6101599e0 0x7fa6100b0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47480/0 (socket says 172.21.15.27:47480) 2024-01-11T08:26:42.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.747+0000 7fa61639b700 1 -- 172.21.15.27:0/414419629 learned_addr learned my addr 172.21.15.27:0/414419629 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:42.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.747+0000 7fa615b9a700 1 --2- 172.21.15.27:0/414419629 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa610159db0 0x7fa61015a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:42.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.747+0000 7fa616b9c700 1 --2- 172.21.15.27:0/414419629 >> v2:172.21.15.77:3300/0 conn(0x7fa61014fc20 0x7fa610150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.747+0000 7fa615399700 1 -- 172.21.15.27:0/414419629 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222075765 0 0) 0x7fa610157e30 con 0x7fa6101599e0 2024-01-11T08:26:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.747+0000 7fa615399700 1 -- 172.21.15.27:0/414419629 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5fc003570 con 0x7fa6101599e0 2024-01-11T08:26:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.747+0000 7fa615399700 1 -- 172.21.15.27:0/414419629 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2270843322 0 0) 0x7fa5fc003570 con 0x7fa6101599e0 2024-01-11T08:26:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.747+0000 7fa615399700 1 -- 172.21.15.27:0/414419629 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa610159db0 msgr2=0x7fa61015a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.747+0000 7fa615399700 1 --2- 172.21.15.27:0/414419629 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa610159db0 0x7fa61015a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.747+0000 7fa615399700 1 -- 172.21.15.27:0/414419629 >> v2:172.21.15.77:3300/0 conn(0x7fa61014fc20 msgr2=0x7fa610150080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.747+0000 7fa615399700 1 --2- 172.21.15.27:0/414419629 >> v2:172.21.15.77:3300/0 conn(0x7fa61014fc20 0x7fa610150080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.748+0000 7fa615399700 1 -- 172.21.15.27:0/414419629 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa610157ac0 con 0x7fa6101599e0 2024-01-11T08:26:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.748+0000 7fa615399700 1 -- 172.21.15.27:0/414419629 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fa60c001470 con 0x7fa6101599e0 2024-01-11T08:26:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.748+0000 7fa615399700 1 -- 172.21.15.27:0/414419629 >> v1:172.21.15.27:6789/0 conn(0x7fa6101599e0 legacy=0x7fa6100b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.748+0000 7fa615399700 1 --2- 172.21.15.27:0/414419629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc003a60 0x7fa5fc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.748+0000 7fa615399700 1 --2- 172.21.15.27:0/414419629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5fc004810 0x7fa5fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.748+0000 7fa615399700 1 --2- 172.21.15.27:0/414419629 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa610159db0 0x7fa5fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.748+0000 7fa615399700 1 -- 172.21.15.27:0/414419629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa610157500 con 0x7fa5fc003a60 2024-01-11T08:26:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.748+0000 7fa615399700 1 -- 172.21.15.27:0/414419629 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6101577e0 con 0x7fa5fc004810 2024-01-11T08:26:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.748+0000 7fa615399700 1 -- 172.21.15.27:0/414419629 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc00dc70 con 0x7fa610159db0 2024-01-11T08:26:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.748+0000 7fa61639b700 1 --2- 172.21.15.27:0/414419629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc003a60 0x7fa5fc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.748+0000 7fa616b9c700 1 --2- 172.21.15.27:0/414419629 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa610159db0 0x7fa5fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.748+0000 7fa615b9a700 1 --2- 172.21.15.27:0/414419629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5fc004810 0x7fa5fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.748+0000 7fa61639b700 1 -- 172.21.15.27:0/414419629 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa610159db0 msgr2=0x7fa5fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.748+0000 7fa61639b700 1 --2- 172.21.15.27:0/414419629 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa610159db0 0x7fa5fc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.748+0000 7fa61639b700 1 -- 172.21.15.27:0/414419629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5fc004810 msgr2=0x7fa5fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.748+0000 7fa61639b700 1 --2- 172.21.15.27:0/414419629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5fc004810 0x7fa5fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.748+0000 7fa61639b700 1 -- 172.21.15.27:0/414419629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc003570 con 0x7fa5fc003a60 2024-01-11T08:26:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.749+0000 7fa61639b700 1 --2- 172.21.15.27:0/414419629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc003a60 0x7fa5fc003e70 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fa60c00c3f0 tx=0x7fa60c001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db88ab4de52de039 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.749+0000 7fa615399700 1 -- 172.21.15.27:0/414419629 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c00cc80 con 0x7fa5fc003a60 2024-01-11T08:26:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.749+0000 7fa615399700 1 -- 172.21.15.27:0/414419629 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60c00cde0 con 0x7fa5fc003a60 2024-01-11T08:26:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.749+0000 7fa615399700 1 -- 172.21.15.27:0/414419629 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c005b70 con 0x7fa5fc003a60 2024-01-11T08:26:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.749+0000 7fa6185ff700 1 -- 172.21.15.27:0/414419629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc003a60 msgr2=0x7fa5fc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.749+0000 7fa6185ff700 1 --2- 172.21.15.27:0/414419629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc003a60 0x7fa5fc003e70 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fa60c00c3f0 tx=0x7fa60c001e90 comp rx=0 tx=0).stop 2024-01-11T08:26:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.749+0000 7fa616b9c700 1 -- 172.21.15.27:0/414419629 reap_dead start 2024-01-11T08:26:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.749+0000 7fa6185ff700 1 -- 172.21.15.27:0/414419629 shutdown_connections 2024-01-11T08:26:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.749+0000 7fa6185ff700 1 -- 172.21.15.27:0/414419629 >> 172.21.15.27:0/414419629 conn(0x7fa6100680e0 msgr2=0x7fa610069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.750+0000 7fa6185ff700 1 -- 172.21.15.27:0/414419629 shutdown_connections 2024-01-11T08:26:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.750+0000 7fa6185ff700 1 -- 172.21.15.27:0/414419629 wait complete. 2024-01-11T08:26:42.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.751+0000 7fa6185ff700 1 Processor -- start 2024-01-11T08:26:42.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.751+0000 7fa6185ff700 1 -- start start 2024-01-11T08:26:42.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.752+0000 7fa6185ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc003a60 0x7fa61014a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:42.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.752+0000 7fa6185ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6100adab0 0x7fa6100adf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:42.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.752+0000 7fa6185ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa610155a20 0x7fa610155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:42.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.752+0000 7fa6185ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6100ae460 con 0x7fa5fc003a60 2024-01-11T08:26:42.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.752+0000 7fa6185ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa610156400 con 0x7fa6100adab0 2024-01-11T08:26:42.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.752+0000 7fa6185ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6101548d0 con 0x7fa610155a20 2024-01-11T08:26:42.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.752+0000 7fa61639b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc003a60 0x7fa61014a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:42.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.752+0000 7fa61639b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc003a60 0x7fa61014a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41454/0 (socket says 172.21.15.27:41454) 2024-01-11T08:26:42.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.752+0000 7fa61639b700 1 -- 172.21.15.27:0/1585783355 learned_addr learned my addr 172.21.15.27:0/1585783355 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:42.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.753+0000 7fa616b9c700 1 --2- 172.21.15.27:0/1585783355 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa610155a20 0x7fa610155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:42.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.753+0000 7fa615b9a700 1 --2- 172.21.15.27:0/1585783355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6100adab0 0x7fa6100adf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:42.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.753+0000 7fa61639b700 1 -- 172.21.15.27:0/1585783355 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa610155a20 msgr2=0x7fa610155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:42.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.753+0000 7fa61639b700 1 --2- 172.21.15.27:0/1585783355 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa610155a20 0x7fa610155ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.753+0000 7fa61639b700 1 -- 172.21.15.27:0/1585783355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6100adab0 msgr2=0x7fa6100adf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:42.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.753+0000 7fa61639b700 1 --2- 172.21.15.27:0/1585783355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6100adab0 0x7fa6100adf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.753+0000 7fa61639b700 1 -- 172.21.15.27:0/1585783355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61014ade0 con 0x7fa5fc003a60 2024-01-11T08:26:42.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.753+0000 7fa615b9a700 1 --2- 172.21.15.27:0/1585783355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6100adab0 0x7fa6100adf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:26:42.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.753+0000 7fa61639b700 1 --2- 172.21.15.27:0/1585783355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc003a60 0x7fa61014a8a0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0046d0 tx=0x7fa60c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:42.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.753+0000 7fa6067fc700 1 -- 172.21.15.27:0/1585783355 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c00cc80 con 0x7fa5fc003a60 2024-01-11T08:26:42.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.754+0000 7fa6067fc700 1 -- 172.21.15.27:0/1585783355 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60c0072e0 con 0x7fa5fc003a60 2024-01-11T08:26:42.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.754+0000 7fa6067fc700 1 -- 172.21.15.27:0/1585783355 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c0076a0 con 0x7fa5fc003a60 2024-01-11T08:26:42.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.754+0000 7fa6185ff700 1 -- 172.21.15.27:0/1585783355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa610153fe0 con 0x7fa5fc003a60 2024-01-11T08:26:42.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.754+0000 7fa6185ff700 1 -- 172.21.15.27:0/1585783355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa610154280 con 0x7fa5fc003a60 2024-01-11T08:26:42.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.755+0000 7fa6185ff700 1 -- 172.21.15.27:0/1585783355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6100718d0 con 0x7fa5fc003a60 2024-01-11T08:26:42.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.756+0000 7fa6067fc700 1 -- 172.21.15.27:0/1585783355 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa60c019030 con 0x7fa5fc003a60 2024-01-11T08:26:42.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.756+0000 7fa6067fc700 1 --2- 172.21.15.27:0/1585783355 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa5f4066f30 0x7fa5f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:42.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.756+0000 7fa6067fc700 1 -- 172.21.15.27:0/1585783355 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5733+0+0 (secure 0 0 0) 0x7fa60c0855e0 con 0x7fa5fc003a60 2024-01-11T08:26:42.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.757+0000 7fa615b9a700 1 --2- 172.21.15.27:0/1585783355 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa5f4066f30 0x7fa5f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:42.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.758+0000 7fa615b9a700 1 --2- 172.21.15.27:0/1585783355 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa5f4066f30 0x7fa5f40693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa6101551a0 tx=0x7fa600008040 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:42.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.761+0000 7fa6067fc700 1 -- 172.21.15.27:0/1585783355 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa60c052830 con 0x7fa5fc003a60 2024-01-11T08:26:42.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.966+0000 7fa6185ff700 1 -- 172.21.15.27:0/1585783355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6101505c0 con 0x7fa5fc003a60 2024-01-11T08:26:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.967+0000 7fa6067fc700 1 -- 172.21.15.27:0/1585783355 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+15351 (secure 0 0 0) 0x7fa60c055e50 con 0x7fa5fc003a60 2024-01-11T08:26:42.967 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:42.968 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:26:40.016923+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:26:36.329359+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-11T08:26:37.504889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:26:42.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.969+0000 7fa6185ff700 1 -- 172.21.15.27:0/1585783355 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa5f4066f30 msgr2=0x7fa5f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:42.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.969+0000 7fa6185ff700 1 --2- 172.21.15.27:0/1585783355 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa5f4066f30 0x7fa5f40693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa6101551a0 tx=0x7fa600008040 comp rx=0 tx=0).stop 2024-01-11T08:26:42.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.969+0000 7fa6185ff700 1 -- 172.21.15.27:0/1585783355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc003a60 msgr2=0x7fa61014a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:42.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.969+0000 7fa6185ff700 1 --2- 172.21.15.27:0/1585783355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc003a60 0x7fa61014a8a0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0046d0 tx=0x7fa60c015000 comp rx=0 tx=0).stop 2024-01-11T08:26:42.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.969+0000 7fa6185ff700 1 -- 172.21.15.27:0/1585783355 shutdown_connections 2024-01-11T08:26:42.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.969+0000 7fa6185ff700 1 --2- 172.21.15.27:0/1585783355 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa5f4066f30 0x7fa5f40693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.969+0000 7fa6185ff700 1 --2- 172.21.15.27:0/1585783355 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa610155a20 0x7fa610155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.970+0000 7fa6185ff700 1 --2- 172.21.15.27:0/1585783355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc003a60 0x7fa61014a8a0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.970+0000 7fa6185ff700 1 --2- 172.21.15.27:0/1585783355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6100adab0 0x7fa6100adf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:42.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.970+0000 7fa6185ff700 1 -- 172.21.15.27:0/1585783355 >> 172.21.15.27:0/1585783355 conn(0x7fa6100680e0 msgr2=0x7fa61014c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:42.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.970+0000 7fa6185ff700 1 -- 172.21.15.27:0/1585783355 shutdown_connections 2024-01-11T08:26:42.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:42.970+0000 7fa6185ff700 1 -- 172.21.15.27:0/1585783355 wait complete. 2024-01-11T08:26:42.988 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-11T08:26:42.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:26:43.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.188+0000 7ff3ebbff700 1 Processor -- start 2024-01-11T08:26:43.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.188+0000 7ff3ebbff700 1 -- start start 2024-01-11T08:26:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.190+0000 7ff3ebbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff3e4159a20 0x7ff3e406bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.190+0000 7ff3ebbff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff3e40b2730 0x7ff3e40b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.190+0000 7ff3ebbff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e406fba0 con 0x7ff3e4159650 2024-01-11T08:26:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.190+0000 7ff3ebbff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e406e700 con 0x7ff3e40b2730 2024-01-11T08:26:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.190+0000 7ff3ebbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e406c360 con 0x7ff3e4159a20 2024-01-11T08:26:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.190+0000 7ff3ea19c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff3e40b2730 0x7ff3e40b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.190+0000 7ff3ea19c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff3e40b2730 0x7ff3e40b2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41470/0 (socket says 172.21.15.27:41470) 2024-01-11T08:26:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.190+0000 7ff3ea19c700 1 -- 172.21.15.27:0/2413334158 learned_addr learned my addr 172.21.15.27:0/2413334158 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.190+0000 7ff3e999b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff3e4159650 0x7ff3e40b0c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47490/0 (socket says 172.21.15.27:47490) 2024-01-11T08:26:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.190+0000 7ff3e919a700 1 --2- 172.21.15.27:0/2413334158 >> v2:172.21.15.77:3300/0 conn(0x7ff3e4159a20 0x7ff3e406bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.191+0000 7ff3ea19c700 1 -- 172.21.15.27:0/2413334158 >> v2:172.21.15.77:3300/0 conn(0x7ff3e4159a20 msgr2=0x7ff3e406bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.191+0000 7ff3ea19c700 1 --2- 172.21.15.27:0/2413334158 >> v2:172.21.15.77:3300/0 conn(0x7ff3e4159a20 0x7ff3e406bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.191+0000 7ff3ea19c700 1 -- 172.21.15.27:0/2413334158 >> v1:172.21.15.27:6789/0 conn(0x7ff3e4159650 legacy=0x7ff3e40b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.191+0000 7ff3ea19c700 1 -- 172.21.15.27:0/2413334158 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e406c640 con 0x7ff3e40b2730 2024-01-11T08:26:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.191+0000 7ff3e8999700 1 -- 172.21.15.27:0/2413334158 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388926155 0 0) 0x7ff3e406fba0 con 0x7ff3e4159650 2024-01-11T08:26:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.191+0000 7ff3ea19c700 1 --2- 172.21.15.27:0/2413334158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e40b2730 0x7ff3e40b2b70 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4069c00 tx=0x7ff3dc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=868ddc37fcb4a5c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.191+0000 7ff3e8999700 1 -- 172.21.15.27:0/2413334158 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc00e070 con 0x7ff3e40b2730 2024-01-11T08:26:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.191+0000 7ff3e8999700 1 -- 172.21.15.27:0/2413334158 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3dc00ba50 con 0x7ff3e40b2730 2024-01-11T08:26:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.191+0000 7ff3e8999700 1 -- 172.21.15.27:0/2413334158 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc0092e0 con 0x7ff3e40b2730 2024-01-11T08:26:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.192+0000 7ff3ebbff700 1 -- 172.21.15.27:0/2413334158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e40b2730 msgr2=0x7ff3e40b2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.192+0000 7ff3ebbff700 1 --2- 172.21.15.27:0/2413334158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e40b2730 0x7ff3e40b2b70 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4069c00 tx=0x7ff3dc009840 comp rx=0 tx=0).stop 2024-01-11T08:26:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.192+0000 7ff3ebbff700 1 -- 172.21.15.27:0/2413334158 shutdown_connections 2024-01-11T08:26:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.192+0000 7ff3ebbff700 1 --2- 172.21.15.27:0/2413334158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e40b2730 0x7ff3e40b2b70 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.192+0000 7ff3ebbff700 1 --2- 172.21.15.27:0/2413334158 >> v2:172.21.15.77:3300/0 conn(0x7ff3e4159a20 0x7ff3e406bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.192+0000 7ff3ebbff700 1 -- 172.21.15.27:0/2413334158 >> 172.21.15.27:0/2413334158 conn(0x7ff3e406b200 msgr2=0x7ff3e40aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:43.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.192+0000 7ff3ebbff700 1 -- 172.21.15.27:0/2413334158 shutdown_connections 2024-01-11T08:26:43.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.192+0000 7ff3ebbff700 1 -- 172.21.15.27:0/2413334158 wait complete. 2024-01-11T08:26:43.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.193+0000 7ff3ebbff700 1 Processor -- start 2024-01-11T08:26:43.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.194+0000 7ff3ebbff700 1 -- start start 2024-01-11T08:26:43.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.194+0000 7ff3ebbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff3e40b2730 0x7ff3e4145db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:43.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.195+0000 7ff3ebbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3e4159650 0x7ff3e40ad1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:43.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.195+0000 7ff3ebbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e4159a20 0x7ff3e40ad6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:43.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.195+0000 7ff3ebbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e41695e0 con 0x7ff3e4159a20 2024-01-11T08:26:43.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.195+0000 7ff3ebbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e40ab2e0 con 0x7ff3e4159650 2024-01-11T08:26:43.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.195+0000 7ff3ebbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e40aa6d0 con 0x7ff3e40b2730 2024-01-11T08:26:43.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.195+0000 7ff3ea19c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e4159a20 0x7ff3e40ad6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:43.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.195+0000 7ff3ea19c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e4159a20 0x7ff3e40ad6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41480/0 (socket says 172.21.15.27:41480) 2024-01-11T08:26:43.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.195+0000 7ff3ea19c700 1 -- 172.21.15.27:0/3716503857 learned_addr learned my addr 172.21.15.27:0/3716503857 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:43.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.195+0000 7ff3e999b700 1 --2- 172.21.15.27:0/3716503857 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff3e40b2730 0x7ff3e4145db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:43.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.195+0000 7ff3e919a700 1 --2- 172.21.15.27:0/3716503857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3e4159650 0x7ff3e40ad1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:43.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.195+0000 7ff3ea19c700 1 -- 172.21.15.27:0/3716503857 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff3e40b2730 msgr2=0x7ff3e4145db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:43.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.195+0000 7ff3ea19c700 1 --2- 172.21.15.27:0/3716503857 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff3e40b2730 0x7ff3e4145db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:43.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.195+0000 7ff3ea19c700 1 -- 172.21.15.27:0/3716503857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3e4159650 msgr2=0x7ff3e40ad1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:43.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.195+0000 7ff3ea19c700 1 --2- 172.21.15.27:0/3716503857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3e4159650 0x7ff3e40ad1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:43.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.195+0000 7ff3ea19c700 1 -- 172.21.15.27:0/3716503857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3dc009ae0 con 0x7ff3e4159a20 2024-01-11T08:26:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.196+0000 7ff3ea19c700 1 --2- 172.21.15.27:0/3716503857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e4159a20 0x7ff3e40ad6f0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4063d40 tx=0x7ff3dc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.196+0000 7ff3d9ffb700 1 -- 172.21.15.27:0/3716503857 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc008d20 con 0x7ff3e4159a20 2024-01-11T08:26:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.196+0000 7ff3d9ffb700 1 -- 172.21.15.27:0/3716503857 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3dc026460 con 0x7ff3e4159a20 2024-01-11T08:26:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.196+0000 7ff3ebbff700 1 -- 172.21.15.27:0/3716503857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e40aa9b0 con 0x7ff3e4159a20 2024-01-11T08:26:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.196+0000 7ff3d9ffb700 1 -- 172.21.15.27:0/3716503857 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc026820 con 0x7ff3e4159a20 2024-01-11T08:26:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.196+0000 7ff3ebbff700 1 -- 172.21.15.27:0/3716503857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e40aaf70 con 0x7ff3e4159a20 2024-01-11T08:26:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.197+0000 7ff3ebbff700 1 -- 172.21.15.27:0/3716503857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e40718d0 con 0x7ff3e4159a20 2024-01-11T08:26:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.198+0000 7ff3d9ffb700 1 -- 172.21.15.27:0/3716503857 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff3dc003040 con 0x7ff3e4159a20 2024-01-11T08:26:43.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.199+0000 7ff3d9ffb700 1 --2- 172.21.15.27:0/3716503857 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff3c8066f30 0x7ff3c80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:43.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.199+0000 7ff3d9ffb700 1 -- 172.21.15.27:0/3716503857 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5733+0+0 (secure 0 0 0) 0x7ff3dc086560 con 0x7ff3e4159a20 2024-01-11T08:26:43.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.200+0000 7ff3e999b700 1 --2- 172.21.15.27:0/3716503857 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff3c8066f30 0x7ff3c80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:43.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.200+0000 7ff3e999b700 1 --2- 172.21.15.27:0/3716503857 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff3c8066f30 0x7ff3c80693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0007fd0 tx=0x7ff3e0007d50 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:43.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.203+0000 7ff3d9ffb700 1 -- 172.21.15.27:0/3716503857 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff3dc053710 con 0x7ff3e4159a20 2024-01-11T08:26:43.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.399+0000 7ff3ebbff700 1 -- 172.21.15.27:0/3716503857 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff3e406c970 con 0x7ff3c8066f30 2024-01-11T08:26:43.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.411+0000 7ff3d9ffb700 1 -- 172.21.15.27:0/3716503857 <== mgr.6745 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289244 (secure 0 0 0) 0x7ff3e406c970 con 0x7ff3c8066f30 2024-01-11T08:26:43.415 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:43.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.415+0000 7ff3ebbff700 1 -- 172.21.15.27:0/3716503857 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff3c8066f30 msgr2=0x7ff3c80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:43.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.415+0000 7ff3ebbff700 1 --2- 172.21.15.27:0/3716503857 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff3c8066f30 0x7ff3c80693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0007fd0 tx=0x7ff3e0007d50 comp rx=0 tx=0).stop 2024-01-11T08:26:43.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.415+0000 7ff3ebbff700 1 -- 172.21.15.27:0/3716503857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e4159a20 msgr2=0x7ff3e40ad6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:43.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.415+0000 7ff3ebbff700 1 --2- 172.21.15.27:0/3716503857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e4159a20 0x7ff3e40ad6f0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4063d40 tx=0x7ff3dc018000 comp rx=0 tx=0).stop 2024-01-11T08:26:43.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.415+0000 7ff3ebbff700 1 -- 172.21.15.27:0/3716503857 shutdown_connections 2024-01-11T08:26:43.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.415+0000 7ff3ebbff700 1 --2- 172.21.15.27:0/3716503857 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff3c8066f30 0x7ff3c80693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:43.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.415+0000 7ff3ebbff700 1 --2- 172.21.15.27:0/3716503857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3e4159a20 0x7ff3e40ad6f0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:43.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.415+0000 7ff3ebbff700 1 --2- 172.21.15.27:0/3716503857 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff3e40b2730 0x7ff3e4145db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:43.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.415+0000 7ff3ebbff700 1 --2- 172.21.15.27:0/3716503857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3e4159650 0x7ff3e40ad1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:43.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.415+0000 7ff3ebbff700 1 -- 172.21.15.27:0/3716503857 >> 172.21.15.27:0/3716503857 conn(0x7ff3e406b200 msgr2=0x7ff3e4067610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:43.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.415+0000 7ff3ebbff700 1 -- 172.21.15.27:0/3716503857 shutdown_connections 2024-01-11T08:26:43.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.415+0000 7ff3ebbff700 1 -- 172.21.15.27:0/3716503857 wait complete. 2024-01-11T08:26:43.418 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:26:43.433 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-11T08:26:43.139056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":848,"num_read_kb":1718,"num_write":355,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":239,"ondisk_log_size":239,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":75,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2438328,"kb_used_data":10872,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752536392,"statfs":{"total":773094113280,"available":770597265408,"internally_reserved":0,"allocated":11132928,"data_stored":5755020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001780"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.026453+0000","last_change":"2024-01-11T08:26:39.026453+0000","last_active":"2024-01-11T08:26:39.026453+0000","last_peered":"2024-01-11T08:26:39.026453+0000","last_clean":"2024-01-11T08:26:39.026453+0000","last_became_active":"2024-01-11T08:26:39.026117+0000","last_became_peered":"2024-01-11T08:26:39.026117+0000","last_unstale":"2024-01-11T08:26:39.026453+0000","last_undegraded":"2024-01-11T08:26:39.026453+0000","last_fullsized":"2024-01-11T08:26:39.026453+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:26:01.025745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.044291+0000","last_change":"2024-01-11T08:26:39.040377+0000","last_active":"2024-01-11T08:26:39.044291+0000","last_peered":"2024-01-11T08:26:39.044291+0000","last_clean":"2024-01-11T08:26:39.044291+0000","last_became_active":"2024-01-11T08:26:39.040083+0000","last_became_peered":"2024-01-11T08:26:39.040083+0000","last_unstale":"2024-01-11T08:26:39.044291+0000","last_undegraded":"2024-01-11T08:26:39.044291+0000","last_fullsized":"2024-01-11T08:26:39.044291+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:24:59.410081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.033071+0000","last_change":"2024-01-11T08:26:39.030880+0000","last_active":"2024-01-11T08:26:39.033071+0000","last_peered":"2024-01-11T08:26:39.033071+0000","last_clean":"2024-01-11T08:26:39.033071+0000","last_became_active":"2024-01-11T08:26:39.030170+0000","last_became_peered":"2024-01-11T08:26:39.030170+0000","last_unstale":"2024-01-11T08:26:39.033071+0000","last_undegraded":"2024-01-11T08:26:39.033071+0000","last_fullsized":"2024-01-11T08:26:39.033071+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:51:42.937889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.039254+0000","last_change":"2024-01-11T08:26:39.039254+0000","last_active":"2024-01-11T08:26:39.039254+0000","last_peered":"2024-01-11T08:26:39.039254+0000","last_clean":"2024-01-11T08:26:39.039254+0000","last_became_active":"2024-01-11T08:26:39.038895+0000","last_became_peered":"2024-01-11T08:26:39.038895+0000","last_unstale":"2024-01-11T08:26:39.039254+0000","last_undegraded":"2024-01-11T08:26:39.039254+0000","last_fullsized":"2024-01-11T08:26:39.039254+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:47:14.391963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.036900+0000","last_change":"2024-01-11T08:26:39.035301+0000","last_active":"2024-01-11T08:26:39.036900+0000","last_peered":"2024-01-11T08:26:39.036900+0000","last_clean":"2024-01-11T08:26:39.036900+0000","last_became_active":"2024-01-11T08:26:39.034869+0000","last_became_peered":"2024-01-11T08:26:39.034869+0000","last_unstale":"2024-01-11T08:26:39.036900+0000","last_undegraded":"2024-01-11T08:26:39.036900+0000","last_fullsized":"2024-01-11T08:26:39.036900+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:26:01.025745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.022104+0000","last_change":"2024-01-11T08:26:39.022104+0000","last_active":"2024-01-11T08:26:39.022104+0000","last_peered":"2024-01-11T08:26:39.022104+0000","last_clean":"2024-01-11T08:26:39.022104+0000","last_became_active":"2024-01-11T08:26:39.020953+0000","last_became_peered":"2024-01-11T08:26:39.020953+0000","last_unstale":"2024-01-11T08:26:39.022104+0000","last_undegraded":"2024-01-11T08:26:39.022104+0000","last_fullsized":"2024-01-11T08:26:39.022104+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:41:19.595025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.025189+0000","last_change":"2024-01-11T08:26:39.025189+0000","last_active":"2024-01-11T08:26:39.025189+0000","last_peered":"2024-01-11T08:26:39.025189+0000","last_clean":"2024-01-11T08:26:39.025189+0000","last_became_active":"2024-01-11T08:26:39.024909+0000","last_became_peered":"2024-01-11T08:26:39.024909+0000","last_unstale":"2024-01-11T08:26:39.025189+0000","last_undegraded":"2024-01-11T08:26:39.025189+0000","last_fullsized":"2024-01-11T08:26:39.025189+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:08:02.532479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.040501+0000","last_change":"2024-01-11T08:26:39.040501+0000","last_active":"2024-01-11T08:26:39.040501+0000","last_peered":"2024-01-11T08:26:39.040501+0000","last_clean":"2024-01-11T08:26:39.040501+0000","last_became_active":"2024-01-11T08:26:39.040152+0000","last_became_peered":"2024-01-11T08:26:39.040152+0000","last_unstale":"2024-01-11T08:26:39.040501+0000","last_undegraded":"2024-01-11T08:26:39.040501+0000","last_fullsized":"2024-01-11T08:26:39.040501+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:59:43.220997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.035617+0000","last_change":"2024-01-11T08:26:39.035617+0000","last_active":"2024-01-11T08:26:39.035617+0000","last_peered":"2024-01-11T08:26:39.035617+0000","last_clean":"2024-01-11T08:26:39.035617+0000","last_became_active":"2024-01-11T08:26:39.035303+0000","last_became_peered":"2024-01-11T08:26:39.035303+0000","last_unstale":"2024-01-11T08:26:39.035617+0000","last_undegraded":"2024-01-11T08:26:39.035617+0000","last_fullsized":"2024-01-11T08:26:39.035617+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:21:29.172470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.015655+0000","last_change":"2024-01-11T08:26:39.015655+0000","last_active":"2024-01-11T08:26:39.015655+0000","last_peered":"2024-01-11T08:26:39.015655+0000","last_clean":"2024-01-11T08:26:39.015655+0000","last_became_active":"2024-01-11T08:26:39.015333+0000","last_became_peered":"2024-01-11T08:26:39.015333+0000","last_unstale":"2024-01-11T08:26:39.015655+0000","last_undegraded":"2024-01-11T08:26:39.015655+0000","last_fullsized":"2024-01-11T08:26:39.015655+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:10:31.552465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.039793+0000","last_change":"2024-01-11T08:26:39.035866+0000","last_active":"2024-01-11T08:26:39.039793+0000","last_peered":"2024-01-11T08:26:39.039793+0000","last_clean":"2024-01-11T08:26:39.039793+0000","last_became_active":"2024-01-11T08:26:39.035538+0000","last_became_peered":"2024-01-11T08:26:39.035538+0000","last_unstale":"2024-01-11T08:26:39.039793+0000","last_undegraded":"2024-01-11T08:26:39.039793+0000","last_fullsized":"2024-01-11T08:26:39.039793+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:41:52.664240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.014776+0000","last_change":"2024-01-11T08:26:39.014776+0000","last_active":"2024-01-11T08:26:39.014776+0000","last_peered":"2024-01-11T08:26:39.014776+0000","last_clean":"2024-01-11T08:26:39.014776+0000","last_became_active":"2024-01-11T08:26:39.014212+0000","last_became_peered":"2024-01-11T08:26:39.014212+0000","last_unstale":"2024-01-11T08:26:39.014776+0000","last_undegraded":"2024-01-11T08:26:39.014776+0000","last_fullsized":"2024-01-11T08:26:39.014776+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:41:19.595025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.026903+0000","last_change":"2024-01-11T08:26:39.025545+0000","last_active":"2024-01-11T08:26:39.026903+0000","last_peered":"2024-01-11T08:26:39.026903+0000","last_clean":"2024-01-11T08:26:39.026903+0000","last_became_active":"2024-01-11T08:26:39.025228+0000","last_became_peered":"2024-01-11T08:26:39.025228+0000","last_unstale":"2024-01-11T08:26:39.026903+0000","last_undegraded":"2024-01-11T08:26:39.026903+0000","last_fullsized":"2024-01-11T08:26:39.026903+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:41:19.595025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.039746+0000","last_change":"2024-01-11T08:26:39.039746+0000","last_active":"2024-01-11T08:26:39.039746+0000","last_peered":"2024-01-11T08:26:39.039746+0000","last_clean":"2024-01-11T08:26:39.039746+0000","last_became_active":"2024-01-11T08:26:39.038891+0000","last_became_peered":"2024-01-11T08:26:39.038891+0000","last_unstale":"2024-01-11T08:26:39.039746+0000","last_undegraded":"2024-01-11T08:26:39.039746+0000","last_fullsized":"2024-01-11T08:26:39.039746+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:23:35.971073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.036008+0000","last_change":"2024-01-11T08:26:39.029787+0000","last_active":"2024-01-11T08:26:39.036008+0000","last_peered":"2024-01-11T08:26:39.036008+0000","last_clean":"2024-01-11T08:26:39.036008+0000","last_became_active":"2024-01-11T08:26:39.026721+0000","last_became_peered":"2024-01-11T08:26:39.026721+0000","last_unstale":"2024-01-11T08:26:39.036008+0000","last_undegraded":"2024-01-11T08:26:39.036008+0000","last_fullsized":"2024-01-11T08:26:39.036008+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:29.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.024416+0000","last_change":"2024-01-11T08:26:39.017023+0000","last_active":"2024-01-11T08:26:39.024416+0000","last_peered":"2024-01-11T08:26:39.024416+0000","last_clean":"2024-01-11T08:26:39.024416+0000","last_became_active":"2024-01-11T08:26:39.014421+0000","last_became_peered":"2024-01-11T08:26:39.014421+0000","last_unstale":"2024-01-11T08:26:39.024416+0000","last_undegraded":"2024-01-11T08:26:39.024416+0000","last_fullsized":"2024-01-11T08:26:39.024416+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:29.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.028040+0000","last_change":"2024-01-11T08:26:39.023050+0000","last_active":"2024-01-11T08:26:39.028040+0000","last_peered":"2024-01-11T08:26:39.028040+0000","last_clean":"2024-01-11T08:26:39.028040+0000","last_became_active":"2024-01-11T08:26:39.022784+0000","last_became_peered":"2024-01-11T08:26:39.022784+0000","last_unstale":"2024-01-11T08:26:39.028040+0000","last_undegraded":"2024-01-11T08:26:39.028040+0000","last_fullsized":"2024-01-11T08:26:39.028040+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:40:47.685006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.028004+0000","last_change":"2024-01-11T08:26:39.028004+0000","last_active":"2024-01-11T08:26:39.028004+0000","last_peered":"2024-01-11T08:26:39.028004+0000","last_clean":"2024-01-11T08:26:39.028004+0000","last_became_active":"2024-01-11T08:26:39.027382+0000","last_became_peered":"2024-01-11T08:26:39.027382+0000","last_unstale":"2024-01-11T08:26:39.028004+0000","last_undegraded":"2024-01-11T08:26:39.028004+0000","last_fullsized":"2024-01-11T08:26:39.028004+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:31:17.740400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.037841+0000","last_change":"2024-01-11T08:26:39.037841+0000","last_active":"2024-01-11T08:26:39.037841+0000","last_peered":"2024-01-11T08:26:39.037841+0000","last_clean":"2024-01-11T08:26:39.037841+0000","last_became_active":"2024-01-11T08:26:39.037447+0000","last_became_peered":"2024-01-11T08:26:39.037447+0000","last_unstale":"2024-01-11T08:26:39.037841+0000","last_undegraded":"2024-01-11T08:26:39.037841+0000","last_fullsized":"2024-01-11T08:26:39.037841+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:21:29.172470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.031822+0000","last_change":"2024-01-11T08:26:39.031822+0000","last_active":"2024-01-11T08:26:39.031822+0000","last_peered":"2024-01-11T08:26:39.031822+0000","last_clean":"2024-01-11T08:26:39.031822+0000","last_became_active":"2024-01-11T08:26:39.031506+0000","last_became_peered":"2024-01-11T08:26:39.031506+0000","last_unstale":"2024-01-11T08:26:39.031822+0000","last_undegraded":"2024-01-11T08:26:39.031822+0000","last_fullsized":"2024-01-11T08:26:39.031822+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:01:21.366890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.033968+0000","last_change":"2024-01-11T08:26:39.033968+0000","last_active":"2024-01-11T08:26:39.033968+0000","last_peered":"2024-01-11T08:26:39.033968+0000","last_clean":"2024-01-11T08:26:39.033968+0000","last_became_active":"2024-01-11T08:26:39.033642+0000","last_became_peered":"2024-01-11T08:26:39.033642+0000","last_unstale":"2024-01-11T08:26:39.033968+0000","last_undegraded":"2024-01-11T08:26:39.033968+0000","last_fullsized":"2024-01-11T08:26:39.033968+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:09:26.014852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.051386+0000","last_change":"2024-01-11T08:26:39.047029+0000","last_active":"2024-01-11T08:26:39.051386+0000","last_peered":"2024-01-11T08:26:39.051386+0000","last_clean":"2024-01-11T08:26:39.051386+0000","last_became_active":"2024-01-11T08:26:39.046202+0000","last_became_peered":"2024-01-11T08:26:39.046202+0000","last_unstale":"2024-01-11T08:26:39.051386+0000","last_undegraded":"2024-01-11T08:26:39.051386+0000","last_fullsized":"2024-01-11T08:26:39.051386+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:06:23.950780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.039659+0000","last_change":"2024-01-11T08:26:39.039659+0000","last_active":"2024-01-11T08:26:39.039659+0000","last_peered":"2024-01-11T08:26:39.039659+0000","last_clean":"2024-01-11T08:26:39.039659+0000","last_became_active":"2024-01-11T08:26:39.039356+0000","last_became_peered":"2024-01-11T08:26:39.039356+0000","last_unstale":"2024-01-11T08:26:39.039659+0000","last_undegraded":"2024-01-11T08:26:39.039659+0000","last_fullsized":"2024-01-11T08:26:39.039659+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:24:22.444046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.045653+0000","last_change":"2024-01-11T08:26:39.045653+0000","last_active":"2024-01-11T08:26:39.045653+0000","last_peered":"2024-01-11T08:26:39.045653+0000","last_clean":"2024-01-11T08:26:39.045653+0000","last_became_active":"2024-01-11T08:26:39.045358+0000","last_became_peered":"2024-01-11T08:26:39.045358+0000","last_unstale":"2024-01-11T08:26:39.045653+0000","last_undegraded":"2024-01-11T08:26:39.045653+0000","last_fullsized":"2024-01-11T08:26:39.045653+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:22:57.054076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.044176+0000","last_change":"2024-01-11T08:26:39.037990+0000","last_active":"2024-01-11T08:26:39.044176+0000","last_peered":"2024-01-11T08:26:39.044176+0000","last_clean":"2024-01-11T08:26:39.044176+0000","last_became_active":"2024-01-11T08:26:39.036034+0000","last_became_peered":"2024-01-11T08:26:39.036034+0000","last_unstale":"2024-01-11T08:26:39.044176+0000","last_undegraded":"2024-01-11T08:26:39.044176+0000","last_fullsized":"2024-01-11T08:26:39.044176+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:15:20.506477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.036929+0000","last_change":"2024-01-11T08:26:39.035691+0000","last_active":"2024-01-11T08:26:39.036929+0000","last_peered":"2024-01-11T08:26:39.036929+0000","last_clean":"2024-01-11T08:26:39.036929+0000","last_became_active":"2024-01-11T08:26:39.034981+0000","last_became_peered":"2024-01-11T08:26:39.034981+0000","last_unstale":"2024-01-11T08:26:39.036929+0000","last_undegraded":"2024-01-11T08:26:39.036929+0000","last_fullsized":"2024-01-11T08:26:39.036929+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:40:47.685006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.028441+0000","last_change":"2024-01-11T08:26:39.023632+0000","last_active":"2024-01-11T08:26:39.028441+0000","last_peered":"2024-01-11T08:26:39.028441+0000","last_clean":"2024-01-11T08:26:39.028441+0000","last_became_active":"2024-01-11T08:26:39.023294+0000","last_became_peered":"2024-01-11T08:26:39.023294+0000","last_unstale":"2024-01-11T08:26:39.028441+0000","last_undegraded":"2024-01-11T08:26:39.028441+0000","last_fullsized":"2024-01-11T08:26:39.028441+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:35.454673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.017576+0000","last_change":"2024-01-11T08:26:39.017576+0000","last_active":"2024-01-11T08:26:39.017576+0000","last_peered":"2024-01-11T08:26:39.017576+0000","last_clean":"2024-01-11T08:26:39.017576+0000","last_became_active":"2024-01-11T08:26:39.017275+0000","last_became_peered":"2024-01-11T08:26:39.017275+0000","last_unstale":"2024-01-11T08:26:39.017576+0000","last_undegraded":"2024-01-11T08:26:39.017576+0000","last_fullsized":"2024-01-11T08:26:39.017576+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:13:27.550996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.041371+0000","last_change":"2024-01-11T08:26:39.034358+0000","last_active":"2024-01-11T08:26:39.041371+0000","last_peered":"2024-01-11T08:26:39.041371+0000","last_clean":"2024-01-11T08:26:39.041371+0000","last_became_active":"2024-01-11T08:26:39.033994+0000","last_became_peered":"2024-01-11T08:26:39.033994+0000","last_unstale":"2024-01-11T08:26:39.041371+0000","last_undegraded":"2024-01-11T08:26:39.041371+0000","last_fullsized":"2024-01-11T08:26:39.041371+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:15:40.371755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.051403+0000","last_change":"2024-01-11T08:26:39.047201+0000","last_active":"2024-01-11T08:26:39.051403+0000","last_peered":"2024-01-11T08:26:39.051403+0000","last_clean":"2024-01-11T08:26:39.051403+0000","last_became_active":"2024-01-11T08:26:39.045817+0000","last_became_peered":"2024-01-11T08:26:39.045817+0000","last_unstale":"2024-01-11T08:26:39.051403+0000","last_undegraded":"2024-01-11T08:26:39.051403+0000","last_fullsized":"2024-01-11T08:26:39.051403+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:19:25.941718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.033729+0000","last_change":"2024-01-11T08:26:39.030895+0000","last_active":"2024-01-11T08:26:39.033729+0000","last_peered":"2024-01-11T08:26:39.033729+0000","last_clean":"2024-01-11T08:26:39.033729+0000","last_became_active":"2024-01-11T08:26:39.030619+0000","last_became_peered":"2024-01-11T08:26:39.030619+0000","last_unstale":"2024-01-11T08:26:39.033729+0000","last_undegraded":"2024-01-11T08:26:39.033729+0000","last_fullsized":"2024-01-11T08:26:39.033729+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:56:34.267505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.039191+0000","last_change":"2024-01-11T08:26:39.039191+0000","last_active":"2024-01-11T08:26:39.039191+0000","last_peered":"2024-01-11T08:26:39.039191+0000","last_clean":"2024-01-11T08:26:39.039191+0000","last_became_active":"2024-01-11T08:26:39.038809+0000","last_became_peered":"2024-01-11T08:26:39.038809+0000","last_unstale":"2024-01-11T08:26:39.039191+0000","last_undegraded":"2024-01-11T08:26:39.039191+0000","last_fullsized":"2024-01-11T08:26:39.039191+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:59:43.220997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.027681+0000","last_change":"2024-01-11T08:26:39.020588+0000","last_active":"2024-01-11T08:26:39.027681+0000","last_peered":"2024-01-11T08:26:39.027681+0000","last_clean":"2024-01-11T08:26:39.027681+0000","last_became_active":"2024-01-11T08:26:39.020306+0000","last_became_peered":"2024-01-11T08:26:39.020306+0000","last_unstale":"2024-01-11T08:26:39.027681+0000","last_undegraded":"2024-01-11T08:26:39.027681+0000","last_fullsized":"2024-01-11T08:26:39.027681+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:41:19.595025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"100'4","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.044056+0000","last_change":"2024-01-11T08:26:38.029051+0000","last_active":"2024-01-11T08:26:39.044056+0000","last_peered":"2024-01-11T08:26:39.044056+0000","last_clean":"2024-01-11T08:26:39.044056+0000","last_became_active":"2024-01-11T08:26:38.028640+0000","last_became_peered":"2024-01-11T08:26:38.028640+0000","last_unstale":"2024-01-11T08:26:39.044056+0000","last_undegraded":"2024-01-11T08:26:39.044056+0000","last_fullsized":"2024-01-11T08:26:39.044056+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:44:56.089483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.036461+0000","last_change":"2024-01-11T08:26:39.036461+0000","last_active":"2024-01-11T08:26:39.036461+0000","last_peered":"2024-01-11T08:26:39.036461+0000","last_clean":"2024-01-11T08:26:39.036461+0000","last_became_active":"2024-01-11T08:26:39.036132+0000","last_became_peered":"2024-01-11T08:26:39.036132+0000","last_unstale":"2024-01-11T08:26:39.036461+0000","last_undegraded":"2024-01-11T08:26:39.036461+0000","last_fullsized":"2024-01-11T08:26:39.036461+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:28:14.292547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"100'1","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.028425+0000","last_change":"2024-01-11T08:26:38.028014+0000","last_active":"2024-01-11T08:26:39.028425+0000","last_peered":"2024-01-11T08:26:39.028425+0000","last_clean":"2024-01-11T08:26:39.028425+0000","last_became_active":"2024-01-11T08:26:38.027309+0000","last_became_peered":"2024-01-11T08:26:38.027309+0000","last_unstale":"2024-01-11T08:26:39.028425+0000","last_undegraded":"2024-01-11T08:26:39.028425+0000","last_fullsized":"2024-01-11T08:26:39.028425+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:51:04.670324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.037481+0000","last_change":"2024-01-11T08:26:39.037481+0000","last_active":"2024-01-11T08:26:39.037481+0000","last_peered":"2024-01-11T08:26:39.037481+0000","last_clean":"2024-01-11T08:26:39.037481+0000","last_became_active":"2024-01-11T08:26:39.037222+0000","last_became_peered":"2024-01-11T08:26:39.037222+0000","last_unstale":"2024-01-11T08:26:39.037481+0000","last_undegraded":"2024-01-11T08:26:39.037481+0000","last_fullsized":"2024-01-11T08:26:39.037481+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:49:22.217423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"100'2","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.048410+0000","last_change":"2024-01-11T08:26:38.029327+0000","last_active":"2024-01-11T08:26:39.048410+0000","last_peered":"2024-01-11T08:26:39.048410+0000","last_clean":"2024-01-11T08:26:39.048410+0000","last_became_active":"2024-01-11T08:26:38.029007+0000","last_became_peered":"2024-01-11T08:26:38.029007+0000","last_unstale":"2024-01-11T08:26:39.048410+0000","last_undegraded":"2024-01-11T08:26:39.048410+0000","last_fullsized":"2024-01-11T08:26:39.048410+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:59:39.984404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.026871+0000","last_change":"2024-01-11T08:26:39.025023+0000","last_active":"2024-01-11T08:26:39.026871+0000","last_peered":"2024-01-11T08:26:39.026871+0000","last_clean":"2024-01-11T08:26:39.026871+0000","last_became_active":"2024-01-11T08:26:39.022587+0000","last_became_peered":"2024-01-11T08:26:39.022587+0000","last_unstale":"2024-01-11T08:26:39.026871+0000","last_undegraded":"2024-01-11T08:26:39.026871+0000","last_fullsized":"2024-01-11T08:26:39.026871+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:59:43.220997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"100'2","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.046912+0000","last_change":"2024-01-11T08:26:38.026888+0000","last_active":"2024-01-11T08:26:39.046912+0000","last_peered":"2024-01-11T08:26:39.046912+0000","last_clean":"2024-01-11T08:26:39.046912+0000","last_became_active":"2024-01-11T08:26:38.026593+0000","last_became_peered":"2024-01-11T08:26:38.026593+0000","last_unstale":"2024-01-11T08:26:39.046912+0000","last_undegraded":"2024-01-11T08:26:39.046912+0000","last_fullsized":"2024-01-11T08:26:39.046912+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:49:35.838083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.051418+0000","last_change":"2024-01-11T08:26:39.047205+0000","last_active":"2024-01-11T08:26:39.051418+0000","last_peered":"2024-01-11T08:26:39.051418+0000","last_clean":"2024-01-11T08:26:39.051418+0000","last_became_active":"2024-01-11T08:26:39.046555+0000","last_became_peered":"2024-01-11T08:26:39.046555+0000","last_unstale":"2024-01-11T08:26:39.051418+0000","last_undegraded":"2024-01-11T08:26:39.051418+0000","last_fullsized":"2024-01-11T08:26:39.051418+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:45:30.088103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"100'1","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.038636+0000","last_change":"2024-01-11T08:26:38.031001+0000","last_active":"2024-01-11T08:26:39.038636+0000","last_peered":"2024-01-11T08:26:39.038636+0000","last_clean":"2024-01-11T08:26:39.038636+0000","last_became_active":"2024-01-11T08:26:38.030566+0000","last_became_peered":"2024-01-11T08:26:38.030566+0000","last_unstale":"2024-01-11T08:26:39.038636+0000","last_undegraded":"2024-01-11T08:26:39.038636+0000","last_fullsized":"2024-01-11T08:26:39.038636+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:10:12.714352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.031349+0000","last_change":"2024-01-11T08:26:39.022783+0000","last_active":"2024-01-11T08:26:39.031349+0000","last_peered":"2024-01-11T08:26:39.031349+0000","last_clean":"2024-01-11T08:26:39.031349+0000","last_became_active":"2024-01-11T08:26:39.022489+0000","last_became_peered":"2024-01-11T08:26:39.022489+0000","last_unstale":"2024-01-11T08:26:39.031349+0000","last_undegraded":"2024-01-11T08:26:39.031349+0000","last_fullsized":"2024-01-11T08:26:39.031349+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:51:42.937889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"100'1","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.030598+0000","last_change":"2024-01-11T08:26:38.033436+0000","last_active":"2024-01-11T08:26:39.030598+0000","last_peered":"2024-01-11T08:26:39.030598+0000","last_clean":"2024-01-11T08:26:39.030598+0000","last_became_active":"2024-01-11T08:26:38.032987+0000","last_became_peered":"2024-01-11T08:26:38.032987+0000","last_unstale":"2024-01-11T08:26:39.030598+0000","last_undegraded":"2024-01-11T08:26:39.030598+0000","last_fullsized":"2024-01-11T08:26:39.030598+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:06:22.942835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.046428+0000","last_change":"2024-01-11T08:26:39.045832+0000","last_active":"2024-01-11T08:26:39.046428+0000","last_peered":"2024-01-11T08:26:39.046428+0000","last_clean":"2024-01-11T08:26:39.046428+0000","last_became_active":"2024-01-11T08:26:39.045410+0000","last_became_peered":"2024-01-11T08:26:39.045410+0000","last_unstale":"2024-01-11T08:26:39.046428+0000","last_undegraded":"2024-01-11T08:26:39.046428+0000","last_fullsized":"2024-01-11T08:26:39.046428+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:28:14.292547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.032253+0000","last_change":"2024-01-11T08:26:38.028785+0000","last_active":"2024-01-11T08:26:38.032253+0000","last_peered":"2024-01-11T08:26:38.032253+0000","last_clean":"2024-01-11T08:26:38.032253+0000","last_became_active":"2024-01-11T08:26:38.026668+0000","last_became_peered":"2024-01-11T08:26:38.026668+0000","last_unstale":"2024-01-11T08:26:38.032253+0000","last_undegraded":"2024-01-11T08:26:38.032253+0000","last_fullsized":"2024-01-11T08:26:38.032253+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:01:34.920162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.032870+0000","last_change":"2024-01-11T08:26:39.027019+0000","last_active":"2024-01-11T08:26:39.032870+0000","last_peered":"2024-01-11T08:26:39.032870+0000","last_clean":"2024-01-11T08:26:39.032870+0000","last_became_active":"2024-01-11T08:26:39.026297+0000","last_became_peered":"2024-01-11T08:26:39.026297+0000","last_unstale":"2024-01-11T08:26:39.032870+0000","last_undegraded":"2024-01-11T08:26:39.032870+0000","last_fullsized":"2024-01-11T08:26:39.032870+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:21:29.172470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":10,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.030185+0000","last_change":"2024-01-11T08:26:38.027465+0000","last_active":"2024-01-11T08:26:39.030185+0000","last_peered":"2024-01-11T08:26:39.030185+0000","last_clean":"2024-01-11T08:26:39.030185+0000","last_became_active":"2024-01-11T08:26:38.027171+0000","last_became_peered":"2024-01-11T08:26:38.027171+0000","last_unstale":"2024-01-11T08:26:39.030185+0000","last_undegraded":"2024-01-11T08:26:39.030185+0000","last_fullsized":"2024-01-11T08:26:39.030185+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:31:16.732455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.037160+0000","last_change":"2024-01-11T08:26:39.035404+0000","last_active":"2024-01-11T08:26:39.037160+0000","last_peered":"2024-01-11T08:26:39.037160+0000","last_clean":"2024-01-11T08:26:39.037160+0000","last_became_active":"2024-01-11T08:26:39.035040+0000","last_became_peered":"2024-01-11T08:26:39.035040+0000","last_unstale":"2024-01-11T08:26:39.037160+0000","last_undegraded":"2024-01-11T08:26:39.037160+0000","last_fullsized":"2024-01-11T08:26:39.037160+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:41:19.595025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.032375+0000","last_change":"2024-01-11T08:26:38.028124+0000","last_active":"2024-01-11T08:26:38.032375+0000","last_peered":"2024-01-11T08:26:38.032375+0000","last_clean":"2024-01-11T08:26:38.032375+0000","last_became_active":"2024-01-11T08:26:38.027678+0000","last_became_peered":"2024-01-11T08:26:38.027678+0000","last_unstale":"2024-01-11T08:26:38.032375+0000","last_undegraded":"2024-01-11T08:26:38.032375+0000","last_fullsized":"2024-01-11T08:26:38.032375+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:46:57.348974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.023827+0000","last_change":"2024-01-11T08:26:39.023827+0000","last_active":"2024-01-11T08:26:39.023827+0000","last_peered":"2024-01-11T08:26:39.023827+0000","last_clean":"2024-01-11T08:26:39.023827+0000","last_became_active":"2024-01-11T08:26:39.023587+0000","last_became_peered":"2024-01-11T08:26:39.023587+0000","last_unstale":"2024-01-11T08:26:39.023827+0000","last_undegraded":"2024-01-11T08:26:39.023827+0000","last_fullsized":"2024-01-11T08:26:39.023827+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:40:47.685006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.033010+0000","last_change":"2024-01-11T08:26:38.030607+0000","last_active":"2024-01-11T08:26:38.033010+0000","last_peered":"2024-01-11T08:26:38.033010+0000","last_clean":"2024-01-11T08:26:38.033010+0000","last_became_active":"2024-01-11T08:26:38.030282+0000","last_became_peered":"2024-01-11T08:26:38.030282+0000","last_unstale":"2024-01-11T08:26:38.033010+0000","last_undegraded":"2024-01-11T08:26:38.033010+0000","last_fullsized":"2024-01-11T08:26:38.033010+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:06:22.942835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.033781+0000","last_change":"2024-01-11T08:26:39.033781+0000","last_active":"2024-01-11T08:26:39.033781+0000","last_peered":"2024-01-11T08:26:39.033781+0000","last_clean":"2024-01-11T08:26:39.033781+0000","last_became_active":"2024-01-11T08:26:39.032278+0000","last_became_peered":"2024-01-11T08:26:39.032278+0000","last_unstale":"2024-01-11T08:26:39.033781+0000","last_undegraded":"2024-01-11T08:26:39.033781+0000","last_fullsized":"2024-01-11T08:26:39.033781+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:51:42.937889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"100'3","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.031227+0000","last_change":"2024-01-11T08:26:38.027117+0000","last_active":"2024-01-11T08:26:39.031227+0000","last_peered":"2024-01-11T08:26:39.031227+0000","last_clean":"2024-01-11T08:26:39.031227+0000","last_became_active":"2024-01-11T08:26:38.026420+0000","last_became_peered":"2024-01-11T08:26:38.026420+0000","last_unstale":"2024-01-11T08:26:39.031227+0000","last_undegraded":"2024-01-11T08:26:39.031227+0000","last_fullsized":"2024-01-11T08:26:39.031227+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:02:42.005111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.036143+0000","last_change":"2024-01-11T08:26:39.034558+0000","last_active":"2024-01-11T08:26:39.036143+0000","last_peered":"2024-01-11T08:26:39.036143+0000","last_clean":"2024-01-11T08:26:39.036143+0000","last_became_active":"2024-01-11T08:26:39.034194+0000","last_became_peered":"2024-01-11T08:26:39.034194+0000","last_unstale":"2024-01-11T08:26:39.036143+0000","last_undegraded":"2024-01-11T08:26:39.036143+0000","last_fullsized":"2024-01-11T08:26:39.036143+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:10:31.552465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"100'6","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.033546+0000","last_change":"2024-01-11T08:26:38.024428+0000","last_active":"2024-01-11T08:26:39.033546+0000","last_peered":"2024-01-11T08:26:39.033546+0000","last_clean":"2024-01-11T08:26:39.033546+0000","last_became_active":"2024-01-11T08:26:38.022550+0000","last_became_peered":"2024-01-11T08:26:38.022550+0000","last_unstale":"2024-01-11T08:26:39.033546+0000","last_undegraded":"2024-01-11T08:26:39.033546+0000","last_fullsized":"2024-01-11T08:26:39.033546+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:46:57.348974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.033077+0000","last_change":"2024-01-11T08:26:39.033077+0000","last_active":"2024-01-11T08:26:39.033077+0000","last_peered":"2024-01-11T08:26:39.033077+0000","last_clean":"2024-01-11T08:26:39.033077+0000","last_became_active":"2024-01-11T08:26:39.032641+0000","last_became_peered":"2024-01-11T08:26:39.032641+0000","last_unstale":"2024-01-11T08:26:39.033077+0000","last_undegraded":"2024-01-11T08:26:39.033077+0000","last_fullsized":"2024-01-11T08:26:39.033077+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:29.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"100'2","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.049574+0000","last_change":"2024-01-11T08:26:38.031060+0000","last_active":"2024-01-11T08:26:39.049574+0000","last_peered":"2024-01-11T08:26:39.049574+0000","last_clean":"2024-01-11T08:26:39.049574+0000","last_became_active":"2024-01-11T08:26:38.030786+0000","last_became_peered":"2024-01-11T08:26:38.030786+0000","last_unstale":"2024-01-11T08:26:39.049574+0000","last_undegraded":"2024-01-11T08:26:39.049574+0000","last_fullsized":"2024-01-11T08:26:39.049574+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:09:25.006907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.043245+0000","last_change":"2024-01-11T08:26:39.042572+0000","last_active":"2024-01-11T08:26:39.043245+0000","last_peered":"2024-01-11T08:26:39.043245+0000","last_clean":"2024-01-11T08:26:39.043245+0000","last_became_active":"2024-01-11T08:26:39.042273+0000","last_became_peered":"2024-01-11T08:26:39.042273+0000","last_unstale":"2024-01-11T08:26:39.043245+0000","last_undegraded":"2024-01-11T08:26:39.043245+0000","last_fullsized":"2024-01-11T08:26:39.043245+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:29.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":10,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.005005+0000","last_change":"2024-01-11T08:26:38.014454+0000","last_active":"2024-01-11T08:26:39.005005+0000","last_peered":"2024-01-11T08:26:39.005005+0000","last_clean":"2024-01-11T08:26:39.005005+0000","last_became_active":"2024-01-11T08:26:38.014198+0000","last_became_peered":"2024-01-11T08:26:38.014198+0000","last_unstale":"2024-01-11T08:26:39.005005+0000","last_undegraded":"2024-01-11T08:26:39.005005+0000","last_fullsized":"2024-01-11T08:26:39.005005+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:37:07.604504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.048878+0000","last_change":"2024-01-11T08:26:39.046399+0000","last_active":"2024-01-11T08:26:39.048878+0000","last_peered":"2024-01-11T08:26:39.048878+0000","last_clean":"2024-01-11T08:26:39.048878+0000","last_became_active":"2024-01-11T08:26:39.046121+0000","last_became_peered":"2024-01-11T08:26:39.046121+0000","last_unstale":"2024-01-11T08:26:39.048878+0000","last_undegraded":"2024-01-11T08:26:39.048878+0000","last_fullsized":"2024-01-11T08:26:39.048878+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:42:07.465080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.032164+0000","last_change":"2024-01-11T08:26:38.027586+0000","last_active":"2024-01-11T08:26:38.032164+0000","last_peered":"2024-01-11T08:26:38.032164+0000","last_clean":"2024-01-11T08:26:38.032164+0000","last_became_active":"2024-01-11T08:26:38.027130+0000","last_became_peered":"2024-01-11T08:26:38.027130+0000","last_unstale":"2024-01-11T08:26:38.032164+0000","last_undegraded":"2024-01-11T08:26:38.032164+0000","last_fullsized":"2024-01-11T08:26:38.032164+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:44:56.089483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.039828+0000","last_change":"2024-01-11T08:26:39.032670+0000","last_active":"2024-01-11T08:26:39.039828+0000","last_peered":"2024-01-11T08:26:39.039828+0000","last_clean":"2024-01-11T08:26:39.039828+0000","last_became_active":"2024-01-11T08:26:39.031923+0000","last_became_peered":"2024-01-11T08:26:39.031923+0000","last_unstale":"2024-01-11T08:26:39.039828+0000","last_undegraded":"2024-01-11T08:26:39.039828+0000","last_fullsized":"2024-01-11T08:26:39.039828+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:51:05.678270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"100'8","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.049199+0000","last_change":"2024-01-11T08:26:38.024297+0000","last_active":"2024-01-11T08:26:39.049199+0000","last_peered":"2024-01-11T08:26:39.049199+0000","last_clean":"2024-01-11T08:26:39.049199+0000","last_became_active":"2024-01-11T08:26:38.024009+0000","last_became_peered":"2024-01-11T08:26:38.024009+0000","last_unstale":"2024-01-11T08:26:39.049199+0000","last_undegraded":"2024-01-11T08:26:39.049199+0000","last_fullsized":"2024-01-11T08:26:39.049199+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:46:57.348974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.027317+0000","last_change":"2024-01-11T08:26:39.026352+0000","last_active":"2024-01-11T08:26:39.027317+0000","last_peered":"2024-01-11T08:26:39.027317+0000","last_clean":"2024-01-11T08:26:39.027317+0000","last_became_active":"2024-01-11T08:26:39.026055+0000","last_became_peered":"2024-01-11T08:26:39.026055+0000","last_unstale":"2024-01-11T08:26:39.027317+0000","last_undegraded":"2024-01-11T08:26:39.027317+0000","last_fullsized":"2024-01-11T08:26:39.027317+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:53:13.699008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"100'1","reported_seq":10,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.011375+0000","last_change":"2024-01-11T08:26:38.028437+0000","last_active":"2024-01-11T08:26:39.011375+0000","last_peered":"2024-01-11T08:26:39.011375+0000","last_clean":"2024-01-11T08:26:39.011375+0000","last_became_active":"2024-01-11T08:26:38.027230+0000","last_became_peered":"2024-01-11T08:26:38.027230+0000","last_unstale":"2024-01-11T08:26:39.011375+0000","last_undegraded":"2024-01-11T08:26:39.011375+0000","last_fullsized":"2024-01-11T08:26:39.011375+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:11:41.070507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.028545+0000","last_change":"2024-01-11T08:26:39.028545+0000","last_active":"2024-01-11T08:26:39.028545+0000","last_peered":"2024-01-11T08:26:39.028545+0000","last_clean":"2024-01-11T08:26:39.028545+0000","last_became_active":"2024-01-11T08:26:39.028270+0000","last_became_peered":"2024-01-11T08:26:39.028270+0000","last_unstale":"2024-01-11T08:26:39.028545+0000","last_undegraded":"2024-01-11T08:26:39.028545+0000","last_fullsized":"2024-01-11T08:26:39.028545+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:29.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"100'2","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.047379+0000","last_change":"2024-01-11T08:26:38.029024+0000","last_active":"2024-01-11T08:26:39.047379+0000","last_peered":"2024-01-11T08:26:39.047379+0000","last_clean":"2024-01-11T08:26:39.047379+0000","last_became_active":"2024-01-11T08:26:38.028745+0000","last_became_peered":"2024-01-11T08:26:38.028745+0000","last_unstale":"2024-01-11T08:26:39.047379+0000","last_undegraded":"2024-01-11T08:26:39.047379+0000","last_fullsized":"2024-01-11T08:26:39.047379+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:45:36.644341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.040414+0000","last_change":"2024-01-11T08:26:39.039302+0000","last_active":"2024-01-11T08:26:39.040414+0000","last_peered":"2024-01-11T08:26:39.040414+0000","last_clean":"2024-01-11T08:26:39.040414+0000","last_became_active":"2024-01-11T08:26:39.038858+0000","last_became_peered":"2024-01-11T08:26:39.038858+0000","last_unstale":"2024-01-11T08:26:39.040414+0000","last_undegraded":"2024-01-11T08:26:39.040414+0000","last_fullsized":"2024-01-11T08:26:39.040414+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:21:29.172470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.030930+0000","last_change":"2024-01-11T08:26:38.025828+0000","last_active":"2024-01-11T08:26:38.030930+0000","last_peered":"2024-01-11T08:26:38.030930+0000","last_clean":"2024-01-11T08:26:38.030930+0000","last_became_active":"2024-01-11T08:26:38.025207+0000","last_became_peered":"2024-01-11T08:26:38.025207+0000","last_unstale":"2024-01-11T08:26:38.030930+0000","last_undegraded":"2024-01-11T08:26:38.030930+0000","last_fullsized":"2024-01-11T08:26:38.030930+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:29:45.372046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.040024+0000","last_change":"2024-01-11T08:26:39.040024+0000","last_active":"2024-01-11T08:26:39.040024+0000","last_peered":"2024-01-11T08:26:39.040024+0000","last_clean":"2024-01-11T08:26:39.040024+0000","last_became_active":"2024-01-11T08:26:39.039494+0000","last_became_peered":"2024-01-11T08:26:39.039494+0000","last_unstale":"2024-01-11T08:26:39.040024+0000","last_undegraded":"2024-01-11T08:26:39.040024+0000","last_fullsized":"2024-01-11T08:26:39.040024+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:01:54.270267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"100'2","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.022160+0000","last_change":"2024-01-11T08:26:38.017810+0000","last_active":"2024-01-11T08:26:39.022160+0000","last_peered":"2024-01-11T08:26:39.022160+0000","last_clean":"2024-01-11T08:26:39.022160+0000","last_became_active":"2024-01-11T08:26:38.017522+0000","last_became_peered":"2024-01-11T08:26:38.017522+0000","last_unstale":"2024-01-11T08:26:39.022160+0000","last_undegraded":"2024-01-11T08:26:39.022160+0000","last_fullsized":"2024-01-11T08:26:39.022160+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:53:12.691063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.018145+0000","last_change":"2024-01-11T08:26:39.018145+0000","last_active":"2024-01-11T08:26:39.018145+0000","last_peered":"2024-01-11T08:26:39.018145+0000","last_clean":"2024-01-11T08:26:39.018145+0000","last_became_active":"2024-01-11T08:26:39.017838+0000","last_became_peered":"2024-01-11T08:26:39.017838+0000","last_unstale":"2024-01-11T08:26:39.018145+0000","last_undegraded":"2024-01-11T08:26:39.018145+0000","last_fullsized":"2024-01-11T08:26:39.018145+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:51:42.937889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.031282+0000","last_change":"2024-01-11T08:26:38.027479+0000","last_active":"2024-01-11T08:26:38.031282+0000","last_peered":"2024-01-11T08:26:38.031282+0000","last_clean":"2024-01-11T08:26:38.031282+0000","last_became_active":"2024-01-11T08:26:38.027200+0000","last_became_peered":"2024-01-11T08:26:38.027200+0000","last_unstale":"2024-01-11T08:26:38.031282+0000","last_undegraded":"2024-01-11T08:26:38.031282+0000","last_fullsized":"2024-01-11T08:26:38.031282+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:00:45.069014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.026743+0000","last_change":"2024-01-11T08:26:39.026743+0000","last_active":"2024-01-11T08:26:39.026743+0000","last_peered":"2024-01-11T08:26:39.026743+0000","last_clean":"2024-01-11T08:26:39.026743+0000","last_became_active":"2024-01-11T08:26:39.026488+0000","last_became_peered":"2024-01-11T08:26:39.026488+0000","last_unstale":"2024-01-11T08:26:39.026743+0000","last_undegraded":"2024-01-11T08:26:39.026743+0000","last_fullsized":"2024-01-11T08:26:39.026743+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:21:31.207621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.027169+0000","last_change":"2024-01-11T08:26:38.023219+0000","last_active":"2024-01-11T08:26:38.027169+0000","last_peered":"2024-01-11T08:26:38.027169+0000","last_clean":"2024-01-11T08:26:38.027169+0000","last_became_active":"2024-01-11T08:26:38.022963+0000","last_became_peered":"2024-01-11T08:26:38.022963+0000","last_unstale":"2024-01-11T08:26:38.027169+0000","last_undegraded":"2024-01-11T08:26:38.027169+0000","last_fullsized":"2024-01-11T08:26:38.027169+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:41:53.413360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.028292+0000","last_change":"2024-01-11T08:26:39.028292+0000","last_active":"2024-01-11T08:26:39.028292+0000","last_peered":"2024-01-11T08:26:39.028292+0000","last_clean":"2024-01-11T08:26:39.028292+0000","last_became_active":"2024-01-11T08:26:39.027473+0000","last_became_peered":"2024-01-11T08:26:39.027473+0000","last_unstale":"2024-01-11T08:26:39.028292+0000","last_undegraded":"2024-01-11T08:26:39.028292+0000","last_fullsized":"2024-01-11T08:26:39.028292+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:28:14.292547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":10,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.004709+0000","last_change":"2024-01-11T08:26:38.023339+0000","last_active":"2024-01-11T08:26:39.004709+0000","last_peered":"2024-01-11T08:26:39.004709+0000","last_clean":"2024-01-11T08:26:39.004709+0000","last_became_active":"2024-01-11T08:26:38.022815+0000","last_became_peered":"2024-01-11T08:26:38.022815+0000","last_unstale":"2024-01-11T08:26:39.004709+0000","last_undegraded":"2024-01-11T08:26:39.004709+0000","last_fullsized":"2024-01-11T08:26:39.004709+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:27:28.154658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.027659+0000","last_change":"2024-01-11T08:26:39.019719+0000","last_active":"2024-01-11T08:26:39.027659+0000","last_peered":"2024-01-11T08:26:39.027659+0000","last_clean":"2024-01-11T08:26:39.027659+0000","last_became_active":"2024-01-11T08:26:39.019454+0000","last_became_peered":"2024-01-11T08:26:39.019454+0000","last_unstale":"2024-01-11T08:26:39.027659+0000","last_undegraded":"2024-01-11T08:26:39.027659+0000","last_fullsized":"2024-01-11T08:26:39.027659+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:28:14.292547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"100'2","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.042441+0000","last_change":"2024-01-11T08:26:38.019388+0000","last_active":"2024-01-11T08:26:39.042441+0000","last_peered":"2024-01-11T08:26:39.042441+0000","last_clean":"2024-01-11T08:26:39.042441+0000","last_became_active":"2024-01-11T08:26:38.019116+0000","last_became_peered":"2024-01-11T08:26:38.019116+0000","last_unstale":"2024-01-11T08:26:39.042441+0000","last_undegraded":"2024-01-11T08:26:39.042441+0000","last_fullsized":"2024-01-11T08:26:39.042441+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:12:33.643792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.033022+0000","last_change":"2024-01-11T08:26:39.030601+0000","last_active":"2024-01-11T08:26:39.033022+0000","last_peered":"2024-01-11T08:26:39.033022+0000","last_clean":"2024-01-11T08:26:39.033022+0000","last_became_active":"2024-01-11T08:26:39.029663+0000","last_became_peered":"2024-01-11T08:26:39.029663+0000","last_unstale":"2024-01-11T08:26:39.033022+0000","last_undegraded":"2024-01-11T08:26:39.033022+0000","last_fullsized":"2024-01-11T08:26:39.033022+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:21:29.172470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.033748+0000","last_change":"2024-01-11T08:26:38.029559+0000","last_active":"2024-01-11T08:26:39.033748+0000","last_peered":"2024-01-11T08:26:39.033748+0000","last_clean":"2024-01-11T08:26:39.033748+0000","last_became_active":"2024-01-11T08:26:38.029268+0000","last_became_peered":"2024-01-11T08:26:38.029268+0000","last_unstale":"2024-01-11T08:26:39.033748+0000","last_undegraded":"2024-01-11T08:26:39.033748+0000","last_fullsized":"2024-01-11T08:26:39.033748+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:38:36.065914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.028113+0000","last_change":"2024-01-11T08:26:39.028113+0000","last_active":"2024-01-11T08:26:39.028113+0000","last_peered":"2024-01-11T08:26:39.028113+0000","last_clean":"2024-01-11T08:26:39.028113+0000","last_became_active":"2024-01-11T08:26:39.027608+0000","last_became_peered":"2024-01-11T08:26:39.027608+0000","last_unstale":"2024-01-11T08:26:39.028113+0000","last_undegraded":"2024-01-11T08:26:39.028113+0000","last_fullsized":"2024-01-11T08:26:39.028113+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:56:34.267505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.035303+0000","last_change":"2024-01-11T08:26:38.032356+0000","last_active":"2024-01-11T08:26:38.035303+0000","last_peered":"2024-01-11T08:26:38.035303+0000","last_clean":"2024-01-11T08:26:38.035303+0000","last_became_active":"2024-01-11T08:26:38.031798+0000","last_became_peered":"2024-01-11T08:26:38.031798+0000","last_unstale":"2024-01-11T08:26:38.035303+0000","last_undegraded":"2024-01-11T08:26:38.035303+0000","last_fullsized":"2024-01-11T08:26:38.035303+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:24:58.402136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.044906+0000","last_change":"2024-01-11T08:26:39.044000+0000","last_active":"2024-01-11T08:26:39.044906+0000","last_peered":"2024-01-11T08:26:39.044906+0000","last_clean":"2024-01-11T08:26:39.044906+0000","last_became_active":"2024-01-11T08:26:39.043498+0000","last_became_peered":"2024-01-11T08:26:39.043498+0000","last_unstale":"2024-01-11T08:26:39.044906+0000","last_undegraded":"2024-01-11T08:26:39.044906+0000","last_fullsized":"2024-01-11T08:26:39.044906+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:59:43.220997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.026181+0000","last_change":"2024-01-11T08:26:38.022488+0000","last_active":"2024-01-11T08:26:38.026181+0000","last_peered":"2024-01-11T08:26:38.026181+0000","last_clean":"2024-01-11T08:26:38.026181+0000","last_became_active":"2024-01-11T08:26:38.022007+0000","last_became_peered":"2024-01-11T08:26:38.022007+0000","last_unstale":"2024-01-11T08:26:38.026181+0000","last_undegraded":"2024-01-11T08:26:38.026181+0000","last_fullsized":"2024-01-11T08:26:38.026181+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:51:04.670324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.018493+0000","last_change":"2024-01-11T08:26:39.018493+0000","last_active":"2024-01-11T08:26:39.018493+0000","last_peered":"2024-01-11T08:26:39.018493+0000","last_clean":"2024-01-11T08:26:39.018493+0000","last_became_active":"2024-01-11T08:26:39.018183+0000","last_became_peered":"2024-01-11T08:26:39.018183+0000","last_unstale":"2024-01-11T08:26:39.018493+0000","last_undegraded":"2024-01-11T08:26:39.018493+0000","last_fullsized":"2024-01-11T08:26:39.018493+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:29.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.025595+0000","last_change":"2024-01-11T08:26:38.022216+0000","last_active":"2024-01-11T08:26:38.025595+0000","last_peered":"2024-01-11T08:26:38.025595+0000","last_clean":"2024-01-11T08:26:38.025595+0000","last_became_active":"2024-01-11T08:26:38.021945+0000","last_became_peered":"2024-01-11T08:26:38.021945+0000","last_unstale":"2024-01-11T08:26:38.025595+0000","last_undegraded":"2024-01-11T08:26:38.025595+0000","last_fullsized":"2024-01-11T08:26:38.025595+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:44:56.089483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.044928+0000","last_change":"2024-01-11T08:26:39.044928+0000","last_active":"2024-01-11T08:26:39.044928+0000","last_peered":"2024-01-11T08:26:39.044928+0000","last_clean":"2024-01-11T08:26:39.044928+0000","last_became_active":"2024-01-11T08:26:39.044353+0000","last_became_peered":"2024-01-11T08:26:39.044353+0000","last_unstale":"2024-01-11T08:26:39.044928+0000","last_undegraded":"2024-01-11T08:26:39.044928+0000","last_fullsized":"2024-01-11T08:26:39.044928+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:28:14.292547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.015276+0000","last_change":"2024-01-11T08:26:38.030355+0000","last_active":"2024-01-11T08:26:39.015276+0000","last_peered":"2024-01-11T08:26:39.015276+0000","last_clean":"2024-01-11T08:26:39.015276+0000","last_became_active":"2024-01-11T08:26:38.030099+0000","last_became_peered":"2024-01-11T08:26:38.030099+0000","last_unstale":"2024-01-11T08:26:39.015276+0000","last_undegraded":"2024-01-11T08:26:39.015276+0000","last_fullsized":"2024-01-11T08:26:39.015276+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:15:19.498532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.041476+0000","last_change":"2024-01-11T08:26:39.041011+0000","last_active":"2024-01-11T08:26:39.041476+0000","last_peered":"2024-01-11T08:26:39.041476+0000","last_clean":"2024-01-11T08:26:39.041476+0000","last_became_active":"2024-01-11T08:26:39.040733+0000","last_became_peered":"2024-01-11T08:26:39.040733+0000","last_unstale":"2024-01-11T08:26:39.041476+0000","last_undegraded":"2024-01-11T08:26:39.041476+0000","last_fullsized":"2024-01-11T08:26:39.041476+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:37:08.612450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.025460+0000","last_change":"2024-01-11T08:26:38.023632+0000","last_active":"2024-01-11T08:26:38.025460+0000","last_peered":"2024-01-11T08:26:38.025460+0000","last_clean":"2024-01-11T08:26:38.025460+0000","last_became_active":"2024-01-11T08:26:38.021109+0000","last_became_peered":"2024-01-11T08:26:38.021109+0000","last_unstale":"2024-01-11T08:26:38.025460+0000","last_undegraded":"2024-01-11T08:26:38.025460+0000","last_fullsized":"2024-01-11T08:26:38.025460+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:45:29.080157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.020206+0000","last_change":"2024-01-11T08:26:39.020206+0000","last_active":"2024-01-11T08:26:39.020206+0000","last_peered":"2024-01-11T08:26:39.020206+0000","last_clean":"2024-01-11T08:26:39.020206+0000","last_became_active":"2024-01-11T08:26:39.019952+0000","last_became_peered":"2024-01-11T08:26:39.019952+0000","last_unstale":"2024-01-11T08:26:39.020206+0000","last_undegraded":"2024-01-11T08:26:39.020206+0000","last_fullsized":"2024-01-11T08:26:39.020206+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:35.454673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.013675+0000","last_change":"2024-01-11T08:26:38.026264+0000","last_active":"2024-01-11T08:26:39.013675+0000","last_peered":"2024-01-11T08:26:39.013675+0000","last_clean":"2024-01-11T08:26:39.013675+0000","last_became_active":"2024-01-11T08:26:38.025907+0000","last_became_peered":"2024-01-11T08:26:38.025907+0000","last_unstale":"2024-01-11T08:26:39.013675+0000","last_undegraded":"2024-01-11T08:26:39.013675+0000","last_fullsized":"2024-01-11T08:26:39.013675+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:11:41.070507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.040164+0000","last_change":"2024-01-11T08:26:39.035601+0000","last_active":"2024-01-11T08:26:39.040164+0000","last_peered":"2024-01-11T08:26:39.040164+0000","last_clean":"2024-01-11T08:26:39.040164+0000","last_became_active":"2024-01-11T08:26:39.035305+0000","last_became_peered":"2024-01-11T08:26:39.035305+0000","last_unstale":"2024-01-11T08:26:39.040164+0000","last_undegraded":"2024-01-11T08:26:39.040164+0000","last_fullsized":"2024-01-11T08:26:39.040164+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:43:43.522488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"100'1","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.040224+0000","last_change":"2024-01-11T08:26:38.025554+0000","last_active":"2024-01-11T08:26:39.040224+0000","last_peered":"2024-01-11T08:26:39.040224+0000","last_clean":"2024-01-11T08:26:39.040224+0000","last_became_active":"2024-01-11T08:26:38.025077+0000","last_became_peered":"2024-01-11T08:26:38.025077+0000","last_unstale":"2024-01-11T08:26:39.040224+0000","last_undegraded":"2024-01-11T08:26:39.040224+0000","last_fullsized":"2024-01-11T08:26:39.040224+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:11:41.070507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"92'201","reported_seq":524,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.012222+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:26:39.012222+0000","last_peered":"2024-01-11T08:26:39.012222+0000","last_clean":"2024-01-11T08:26:39.012222+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:26:39.012222+0000","last_undegraded":"2024-01-11T08:26:39.012222+0000","last_fullsized":"2024-01-11T08:26:39.012222+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":848,"num_read_kb":1718,"num_write":311,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":848,"num_read_kb":1718,"num_write":311,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":201,"ondisk_log_size":201,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575042,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303180,"kb_used_data":1796,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068660,"statfs":{"total":96636764160,"available":96326307840,"internally_reserved":0,"allocated":1839104,"data_stored":1164805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.69399999999999995,"15min":0.70199999999999996},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.79000000000000004,"5min":0.996,"15min":2.8300000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.69199999999999995,"15min":0.70699999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.372},"max":{"1min":0.91600000000000004,"5min":1.073,"15min":1.073},"last":0.48699999999999999}]},{"osd":1,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.45700000000000002,"5min":0.41899999999999998,"15min":0.35799999999999998},"max":{"1min":0.93400000000000005,"5min":0.99099999999999999,"15min":0.998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.70599999999999996,"15min":0.71999999999999997},"min":{"1min":0.41999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.94999999999999996,"5min":0.95899999999999996,"15min":1.153},"last":0.56399999999999995}]},{"osd":2,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.71999999999999997,"15min":0.72899999999999998},"min":{"1min":0.54200000000000004,"5min":0.436,"15min":0.436},"max":{"1min":0.872,"5min":1.03,"15min":1.034},"last":0.623},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.70499999999999996,"15min":0.71599999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.0049999999999999},"last":0.58499999999999996}]},{"osd":3,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.71999999999999997,"15min":0.74399999999999999},"min":{"1min":0.496,"5min":0.48299999999999998,"15min":0.40300000000000002},"max":{"1min":0.91700000000000004,"5min":1.032,"15min":2.9119999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.71899999999999997,"15min":0.73899999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.95899999999999996,"5min":1.008,"15min":1.008},"last":0.64400000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.42299999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":1.089},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63400000000000001,"15min":0.63200000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.85299999999999998,"5min":0.90700000000000003,"15min":1.1020000000000001},"last":0.69599999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.34599999999999997},"max":{"1min":0.90400000000000003,"5min":0.92200000000000004,"15min":1.0589999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.66600000000000004,"15min":0.67800000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.40400000000000003},"max":{"1min":0.94499999999999995,"5min":1.0089999999999999,"15min":1.032},"last":0.67500000000000004}]},{"osd":6,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.74099999999999999,"15min":0.75800000000000001},"min":{"1min":0.54800000000000004,"5min":0.52000000000000002,"15min":0.46999999999999997},"max":{"1min":0.97499999999999998,"5min":1.0469999999999999,"15min":1.1359999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.72399999999999998,"15min":0.73899999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42099999999999999},"max":{"1min":0.90800000000000003,"5min":1.028,"15min":1.1299999999999999},"last":0.78600000000000003}]}]},{"osd":6,"up_from":13,"seq":55834575041,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302596,"kb_used_data":1212,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069244,"statfs":{"total":96636764160,"available":96326905856,"internally_reserved":0,"allocated":1241088,"data_stored":568717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.66100000000000003,"15min":0.66500000000000004},"min":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.29999999999999999},"max":{"1min":0.81100000000000005,"5min":0.89300000000000002,"15min":0.94799999999999995},"last":0.628},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.66600000000000004,"15min":0.68400000000000005},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.65800000000000003}]},{"osd":1,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.68200000000000005,"15min":0.68700000000000006},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.35899999999999999},"max":{"1min":0.997,"5min":1.163,"15min":1.163},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68400000000000005,"15min":0.69099999999999995},"min":{"1min":0.49299999999999999,"5min":0.435,"15min":0.39900000000000002},"max":{"1min":0.92600000000000005,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.72899999999999998}]},{"osd":2,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.70599999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.38600000000000001},"max":{"1min":0.90200000000000002,"5min":0.97199999999999998,"15min":1.016},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.70599999999999996,"15min":0.72099999999999997},"min":{"1min":0.48999999999999999,"5min":0.47799999999999998,"15min":0.373},"max":{"1min":0.89500000000000002,"5min":0.96699999999999997,"15min":0.995},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71699999999999997,"15min":0.73299999999999998},"min":{"1min":0.48099999999999998,"5min":0.46600000000000003,"15min":0.42799999999999999},"max":{"1min":0.94799999999999995,"5min":0.95799999999999996,"15min":1.3260000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.73299999999999998,"15min":0.73699999999999999},"min":{"1min":0.52000000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.88800000000000001,"5min":0.95899999999999996,"15min":1.0049999999999999},"last":0.63600000000000001}]},{"osd":4,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.42999999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84199999999999997,"5min":0.874,"15min":1.212},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57699999999999996,"15min":0.60499999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.374},"max":{"1min":0.76700000000000002,"5min":0.96399999999999997,"15min":1.8320000000000001},"last":0.69899999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64200000000000002,"15min":0.65400000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68799999999999994,"15min":0.70499999999999996},"min":{"1min":0.52300000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":1.7929999999999999},"last":0.55900000000000005}]},{"osd":7,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69099999999999995,"15min":0.71299999999999997},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.41099999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":1.278},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.73099999999999998,"15min":0.749},"min":{"1min":0.46700000000000003,"5min":0.45900000000000002,"15min":0.44700000000000001},"max":{"1min":1.0009999999999999,"5min":1.022,"15min":1.1599999999999999},"last":0.71399999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575043,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302608,"kb_used_data":1224,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069232,"statfs":{"total":96636764160,"available":96326893568,"internally_reserved":0,"allocated":1253376,"data_stored":574561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.309,"5min":0.30599999999999999,"15min":0.26900000000000002},"max":{"1min":0.755,"5min":0.85999999999999999,"15min":0.97699999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56999999999999995,"15min":0.57999999999999996},"min":{"1min":0.42699999999999999,"5min":0.41199999999999998,"15min":0.373},"max":{"1min":0.82299999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.47299999999999998}]},{"osd":2,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48299999999999998,"15min":0.48899999999999999},"min":{"1min":0.33100000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.65800000000000003,"5min":1.113,"15min":1.113},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.375,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71499999999999997,"5min":0.82399999999999995,"15min":1.25},"last":0.35599999999999998}]},{"osd":3,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.60399999999999998,"15min":0.61699999999999999},"min":{"1min":0.41499999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.77600000000000002,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.371,"15min":0.34899999999999998},"max":{"1min":0.88100000000000001,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.74099999999999999}]},{"osd":4,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.75900000000000001,"15min":0.78600000000000003},"min":{"1min":0.504,"5min":0.439,"15min":0.439},"max":{"1min":1.0620000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.80900000000000005,"15min":0.82799999999999996},"min":{"1min":0.65100000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.069,"5min":1.246,"15min":1.3680000000000001},"last":0.85199999999999998}]},{"osd":5,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76900000000000002,"15min":0.79900000000000004},"min":{"1min":0.54300000000000004,"5min":0.502,"15min":0.46999999999999997},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.7849999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.76800000000000002,"15min":0.79400000000000004},"min":{"1min":0.56299999999999994,"5min":0.52700000000000002,"15min":0.50700000000000001},"max":{"1min":1.052,"5min":1.1339999999999999,"15min":1.371},"last":0.752}]},{"osd":6,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.79800000000000004,"15min":0.81899999999999995},"min":{"1min":0.56100000000000005,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.413},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.80500000000000005,"15min":0.82499999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.93899999999999995,"5min":1.155,"15min":1.228},"last":0.61799999999999999}]},{"osd":7,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.81999999999999995,"15min":0.84599999999999997},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.52700000000000002},"max":{"1min":1.0149999999999999,"5min":1.129,"15min":1.226},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.82099999999999995,"15min":0.83799999999999997},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.51400000000000001},"max":{"1min":1.099,"5min":1.1339999999999999,"15min":1.321},"last":0.83399999999999996}]}]},{"osd":0,"up_from":13,"seq":55834575042,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303168,"kb_used_data":1784,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068672,"statfs":{"total":96636764160,"available":96326320128,"internally_reserved":0,"allocated":1826816,"data_stored":1158453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.5,"15min":0.5},"min":{"1min":0.317,"5min":0.29999999999999999,"15min":0.24299999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.92700000000000005},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56599999999999995,"15min":0.57399999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.35299999999999998},"max":{"1min":0.89600000000000002,"5min":0.90500000000000003,"15min":0.92100000000000004},"last":0.60999999999999999}]},{"osd":2,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.29099999999999998},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54400000000000004,"15min":0.55300000000000005},"min":{"1min":0.41699999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.67800000000000005}]},{"osd":3,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.33600000000000002},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.629,"15min":0.627},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.39600000000000002},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.64600000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.79100000000000004,"15min":0.78900000000000003},"min":{"1min":0.502,"5min":0.45700000000000002,"15min":0.437},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.80200000000000005,"15min":0.79100000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.46899999999999997},"max":{"1min":1.095,"5min":1.095,"15min":1.2350000000000001},"last":0.69199999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.80200000000000005,"15min":0.80800000000000005},"min":{"1min":0.57699999999999996,"5min":0.52500000000000002,"15min":0.505},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.82199999999999995,"15min":0.81699999999999995},"min":{"1min":0.58799999999999997,"5min":0.49099999999999999,"15min":0.47799999999999998},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.77200000000000002}]},{"osd":6,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81100000000000005,"15min":0.80200000000000005},"min":{"1min":0.60199999999999998,"5min":0.50900000000000001,"15min":0.497},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.2689999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82699999999999996,"15min":0.83399999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.44900000000000001},"max":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.748}]},{"osd":7,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.84199999999999997,"15min":0.84099999999999997},"min":{"1min":0.61399999999999999,"5min":0.57999999999999996,"15min":0.51400000000000001},"max":{"1min":1.371,"5min":1.371,"15min":1.446},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.85099999999999998,"15min":0.84299999999999997},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.57699999999999996},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.298},"last":0.66500000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575041,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306684,"kb_used_data":1204,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065156,"statfs":{"total":96636764160,"available":96322719744,"internally_reserved":0,"allocated":1232896,"data_stored":568605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47599999999999998,"15min":0.47199999999999998},"min":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.26000000000000001},"max":{"1min":0.67900000000000005,"5min":0.73499999999999999,"15min":0.86099999999999999},"last":0.443},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51800000000000002,"15min":0.54500000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.28899999999999998},"max":{"1min":0.67100000000000004,"5min":0.81699999999999995,"15min":1.4119999999999999},"last":0.48199999999999998}]},{"osd":1,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54200000000000004,"15min":0.55000000000000004},"min":{"1min":0.40400000000000003,"5min":0.36399999999999999,"15min":0.30399999999999999},"max":{"1min":0.73199999999999998,"5min":0.80300000000000005,"15min":0.90000000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.58599999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38100000000000001},"max":{"1min":0.79100000000000004,"5min":0.92900000000000005,"15min":1.5640000000000001},"last":0.54800000000000004}]},{"osd":3,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.38100000000000001,"5min":0.36199999999999999,"15min":0.32900000000000001},"max":{"1min":0.72399999999999998,"5min":0.82499999999999996,"15min":1.1220000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.61199999999999999,"15min":0.63800000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.41799999999999998},"max":{"1min":0.77500000000000002,"5min":0.95299999999999996,"15min":1.22},"last":0.57999999999999996}]},{"osd":4,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.77100000000000002,"15min":0.78900000000000003},"min":{"1min":0.58699999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.026,"5min":1.0580000000000001,"15min":1.488},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.76100000000000001,"15min":0.78100000000000003},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89200000000000002,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.69599999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.81499999999999995,"15min":0.83999999999999997},"min":{"1min":0.63500000000000001,"5min":0.622,"15min":0.51600000000000001},"max":{"1min":1.105,"5min":1.214,"15min":2.9209999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.83099999999999996,"15min":0.84999999999999998},"min":{"1min":0.63500000000000001,"5min":0.55500000000000005,"15min":0.55400000000000005},"max":{"1min":1.036,"5min":1.2,"15min":1.845},"last":0.874}]},{"osd":6,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.79800000000000004,"15min":0.80200000000000005},"min":{"1min":0.56299999999999994,"5min":0.54200000000000004,"15min":0.51700000000000002},"max":{"1min":0.91400000000000003,"5min":1.1919999999999999,"15min":1.3560000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.747,"5min":0.80700000000000005,"15min":0.83299999999999996},"min":{"1min":0.55800000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.012,"5min":1.123,"15min":2.968},"last":0.84499999999999997}]},{"osd":7,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.85299999999999998,"15min":0.872},"min":{"1min":0.62,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.056,"5min":1.2629999999999999,"15min":1.6359999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.83199999999999996,"15min":0.86199999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.0720000000000001,"5min":1.2270000000000001,"15min":1.391},"last":0.82299999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575041,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306704,"kb_used_data":1224,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065136,"statfs":{"total":96636764160,"available":96322699264,"internally_reserved":0,"allocated":1253376,"data_stored":575903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.499,"15min":0.48499999999999999},"min":{"1min":0.34999999999999998,"5min":0.29499999999999998,"15min":0.27600000000000002},"max":{"1min":0.80300000000000005,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.436},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52500000000000002,"15min":0.54600000000000004},"min":{"1min":0.39300000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.873,"5min":0.873,"15min":1.0109999999999999},"last":0.41899999999999998}]},{"osd":1,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.47799999999999998,"15min":0.49099999999999999},"min":{"1min":0.33900000000000002,"5min":0.30299999999999999,"15min":0.25900000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.39600000000000002,"5min":0.36299999999999999,"15min":0.32300000000000001},"max":{"1min":0.92500000000000004,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57399999999999995,"15min":0.58399999999999996},"min":{"1min":0.42799999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81299999999999994,"5min":0.90900000000000003,"15min":0.97599999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.61699999999999999,"15min":0.628},"min":{"1min":0.45900000000000002,"5min":0.44700000000000001,"15min":0.32000000000000001},"max":{"1min":0.74299999999999999,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.50700000000000001}]},{"osd":4,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.71899999999999997,"15min":0.74299999999999999},"min":{"1min":0.54100000000000004,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.012,"5min":1.038,"15min":1.351},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77400000000000002,"15min":0.78200000000000003},"min":{"1min":0.60199999999999998,"5min":0.503,"15min":0.503},"max":{"1min":1.1459999999999999,"5min":1.1819999999999999,"15min":1.198},"last":0.57299999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76400000000000001,"15min":0.78900000000000003},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.51900000000000002},"max":{"1min":1.0649999999999999,"5min":1.139,"15min":1.3779999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.79400000000000004,"15min":0.80900000000000005},"min":{"1min":0.623,"5min":0.49199999999999999,"15min":0.44900000000000001},"max":{"1min":1.24,"5min":1.2809999999999999,"15min":1.4239999999999999},"last":0.68200000000000005}]},{"osd":6,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.81399999999999995,"15min":0.83499999999999996},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.52600000000000002},"max":{"1min":1.0820000000000001,"5min":1.335,"15min":1.927},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.78700000000000003,"15min":0.80600000000000005},"min":{"1min":0.57999999999999996,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.93999999999999995,"5min":1.2230000000000001,"15min":3.5579999999999998},"last":0.73699999999999999}]},{"osd":7,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81000000000000005,"15min":0.84299999999999997},"min":{"1min":0.59199999999999997,"5min":0.52800000000000002,"15min":0.46600000000000003},"max":{"1min":1.0660000000000001,"5min":1.1579999999999999,"15min":3.5089999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.84599999999999997,"15min":0.86099999999999999},"min":{"1min":0.60599999999999998,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.21},"last":0.64900000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310788,"kb_used_data":1212,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94061052,"statfs":{"total":96636764160,"available":96318517248,"internally_reserved":0,"allocated":1241088,"data_stored":574367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.72699999999999998,"15min":0.73799999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.28100000000000003},"max":{"1min":0.874,"5min":0.95999999999999996,"15min":1.524},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72799999999999998,"15min":0.74299999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.38600000000000001},"max":{"1min":0.91500000000000004,"5min":1.0049999999999999,"15min":1.177},"last":0.77400000000000002}]},{"osd":1,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.69899999999999995,"15min":0.70399999999999996},"min":{"1min":0.47399999999999998,"5min":0.46200000000000002,"15min":0.33300000000000002},"max":{"1min":0.89400000000000002,"5min":1.0069999999999999,"15min":1.1499999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.69199999999999995,"15min":0.72299999999999998},"min":{"1min":0.43099999999999999,"5min":0.379,"15min":0.36299999999999999},"max":{"1min":0.94899999999999995,"5min":0.99199999999999999,"15min":2.4740000000000002},"last":0.44}]},{"osd":2,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.71699999999999997},"min":{"1min":0.46600000000000003,"5min":0.46000000000000002,"15min":0.38900000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.72699999999999998,"15min":0.72999999999999998},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.89400000000000002,"5min":0.97699999999999998,"15min":1.212},"last":0.50800000000000001}]},{"osd":3,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.68700000000000006,"15min":0.70599999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.38100000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":1.2649999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.73999999999999999,"15min":0.754},"min":{"1min":0.502,"5min":0.502,"15min":0.47499999999999998},"max":{"1min":0.84399999999999997,"5min":1.026,"15min":1.069},"last":0.621}]},{"osd":5,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60399999999999998,"15min":0.61799999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.85399999999999998,"5min":0.873,"15min":1.0700000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.622,"15min":0.63300000000000001},"min":{"1min":0.45000000000000001,"5min":0.38200000000000001,"15min":0.32600000000000001},"max":{"1min":0.94299999999999995,"5min":0.97799999999999998,"15min":0.98199999999999998},"last":0.69999999999999996}]},{"osd":6,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.65700000000000003,"15min":0.65600000000000003},"min":{"1min":0.40500000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0680000000000001,"5min":1.133,"15min":1.133},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.69199999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95099999999999996,"5min":1.0269999999999999,"15min":1.119},"last":0.66000000000000003}]},{"osd":7,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.69299999999999995},"min":{"1min":0.51500000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.99399999999999999,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.73999999999999999,"15min":0.74299999999999999},"min":{"1min":0.56999999999999995,"5min":0.46200000000000002,"15min":0.42299999999999999},"max":{"1min":0.91000000000000003,"5min":1.2350000000000001,"15min":1.4279999999999999},"last":0.59299999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575042,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302600,"kb_used_data":1216,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069240,"statfs":{"total":96636764160,"available":96326901760,"internally_reserved":0,"allocated":1245184,"data_stored":569609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.69199999999999995,"15min":0.70799999999999996},"min":{"1min":0.39300000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85499999999999998,"5min":1.0249999999999999,"15min":1.032},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.69599999999999995,"15min":0.70999999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.879,"5min":0.96199999999999997,"15min":1.696},"last":0.70299999999999996}]},{"osd":1,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66400000000000003,"15min":0.69899999999999995},"min":{"1min":0.42799999999999999,"5min":0.39400000000000002,"15min":0.36399999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":1.4359999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.70799999999999996,"15min":0.73199999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.05,"5min":1.05,"15min":4.4969999999999999},"last":0.79900000000000004}]},{"osd":2,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70699999999999996,"15min":0.71099999999999997},"min":{"1min":0.55100000000000005,"5min":0.435,"15min":0.435},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.238},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69499999999999995,"15min":0.746},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.41599999999999998},"max":{"1min":0.94199999999999995,"5min":1.0049999999999999,"15min":8.0429999999999993},"last":0.73199999999999998}]},{"osd":3,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.70199999999999996,"15min":0.755},"min":{"1min":0.47599999999999998,"5min":0.46200000000000002,"15min":0.442},"max":{"1min":1.208,"5min":1.208,"15min":8.0839999999999996},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.748,"15min":0.747},"min":{"1min":0.56399999999999995,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.80800000000000005}]},{"osd":4,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.38300000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":1.113},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64500000000000002,"15min":0.64000000000000001},"min":{"1min":0.41499999999999998,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.755}]},{"osd":6,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.45700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.034},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67700000000000005,"15min":0.68600000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.085,"5min":1.085,"15min":1.143},"last":0.79300000000000004}]},{"osd":7,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.73399999999999999,"15min":0.73699999999999999},"min":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.437},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77300000000000002,"15min":0.77000000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.93700000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:26:43.439 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:26:43.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.673+0000 7f48efdff700 1 Processor -- start 2024-01-11T08:26:43.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.673+0000 7f48efdff700 1 -- start start 2024-01-11T08:26:43.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.675+0000 7f48efdff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f48e80b2b90 0x7f48e8159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:43.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.675+0000 7f48efdff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f48e80b07b0 0x7f48e8159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:43.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.675+0000 7f48efdff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48e80cd730 con 0x7f48e80b03e0 2024-01-11T08:26:43.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.675+0000 7f48efdff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48e80ccb20 con 0x7f48e80b07b0 2024-01-11T08:26:43.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.675+0000 7f48efdff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48e80cce00 con 0x7f48e80b2b90 2024-01-11T08:26:43.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.675+0000 7f48ee39c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f48e80b07b0 0x7f48e8159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:43.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.675+0000 7f48ee39c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f48e80b07b0 0x7f48e8159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41486/0 (socket says 172.21.15.27:41486) 2024-01-11T08:26:43.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.675+0000 7f48ee39c700 1 -- 172.21.15.27:0/2973571355 learned_addr learned my addr 172.21.15.27:0/2973571355 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:43.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.675+0000 7f48ed39a700 1 --2- 172.21.15.27:0/2973571355 >> v2:172.21.15.77:3300/0 conn(0x7f48e80b2b90 0x7f48e8159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:43.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.675+0000 7f48ee39c700 1 -- 172.21.15.27:0/2973571355 >> v2:172.21.15.77:3300/0 conn(0x7f48e80b2b90 msgr2=0x7f48e8159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:43.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.675+0000 7f48ee39c700 1 --2- 172.21.15.27:0/2973571355 >> v2:172.21.15.77:3300/0 conn(0x7f48e80b2b90 0x7f48e8159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:43.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.675+0000 7f48ee39c700 1 -- 172.21.15.27:0/2973571355 >> v1:172.21.15.77:6789/0 conn(0x7f48e80b03e0 legacy=0x7f48e80b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:43.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.675+0000 7f48ee39c700 1 -- 172.21.15.27:0/2973571355 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e80cd0e0 con 0x7f48e80b07b0 2024-01-11T08:26:43.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.676+0000 7f48ee39c700 1 --2- 172.21.15.27:0/2973571355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48e80b07b0 0x7f48e8159740 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f48e80cdfc0 tx=0x7f48e0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d3f352d8c82916 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:43.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.676+0000 7f48ecb99700 1 -- 172.21.15.27:0/2973571355 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e000e070 con 0x7f48e80b07b0 2024-01-11T08:26:43.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.676+0000 7f48ecb99700 1 -- 172.21.15.27:0/2973571355 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e000ba50 con 0x7f48e80b07b0 2024-01-11T08:26:43.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.676+0000 7f48ecb99700 1 -- 172.21.15.27:0/2973571355 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e00092e0 con 0x7f48e80b07b0 2024-01-11T08:26:43.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.676+0000 7f48efdff700 1 -- 172.21.15.27:0/2973571355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48e80b07b0 msgr2=0x7f48e8159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:43.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.676+0000 7f48efdff700 1 --2- 172.21.15.27:0/2973571355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48e80b07b0 0x7f48e8159740 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f48e80cdfc0 tx=0x7f48e0009840 comp rx=0 tx=0).stop 2024-01-11T08:26:43.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.677+0000 7f48efdff700 1 -- 172.21.15.27:0/2973571355 shutdown_connections 2024-01-11T08:26:43.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.677+0000 7f48efdff700 1 --2- 172.21.15.27:0/2973571355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48e80b07b0 0x7f48e8159740 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:43.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.677+0000 7f48efdff700 1 --2- 172.21.15.27:0/2973571355 >> v2:172.21.15.77:3300/0 conn(0x7f48e80b2b90 0x7f48e8159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:43.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.677+0000 7f48efdff700 1 -- 172.21.15.27:0/2973571355 >> 172.21.15.27:0/2973571355 conn(0x7f48e806ed40 msgr2=0x7f48e806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:43.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.677+0000 7f48efdff700 1 -- 172.21.15.27:0/2973571355 shutdown_connections 2024-01-11T08:26:43.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.677+0000 7f48efdff700 1 -- 172.21.15.27:0/2973571355 wait complete. 2024-01-11T08:26:43.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.678+0000 7f48efdff700 1 Processor -- start 2024-01-11T08:26:43.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.678+0000 7f48efdff700 1 -- start start 2024-01-11T08:26:43.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.679+0000 7f48efdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48e80b03e0 0x7f48e8169620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:43.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.679+0000 7f48efdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48e80b07b0 0x7f48e80c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:43.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.679+0000 7f48efdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e80b2b90 0x7f48e80c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:43.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.679+0000 7f48efdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e80c9f70 con 0x7f48e80b07b0 2024-01-11T08:26:43.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.679+0000 7f48efdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e80c5030 con 0x7f48e80b2b90 2024-01-11T08:26:43.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.679+0000 7f48efdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48e816a5c0 con 0x7f48e80b03e0 2024-01-11T08:26:43.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.680+0000 7f48ed39a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48e80b07b0 0x7f48e80c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:43.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.680+0000 7f48ed39a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48e80b07b0 0x7f48e80c6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41488/0 (socket says 172.21.15.27:41488) 2024-01-11T08:26:43.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.680+0000 7f48ed39a700 1 -- 172.21.15.27:0/1272533103 learned_addr learned my addr 172.21.15.27:0/1272533103 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:43.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.680+0000 7f48ed39a700 1 -- 172.21.15.27:0/1272533103 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48e80b03e0 msgr2=0x7f48e8169620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:43.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.680+0000 7f48ee39c700 1 --2- 172.21.15.27:0/1272533103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e80b2b90 0x7f48e80c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:43.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.680+0000 7f48edb9b700 1 --2- 172.21.15.27:0/1272533103 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48e80b03e0 0x7f48e8169620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:43.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.680+0000 7f48ed39a700 1 --2- 172.21.15.27:0/1272533103 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48e80b03e0 0x7f48e8169620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:43.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.680+0000 7f48ed39a700 1 -- 172.21.15.27:0/1272533103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e80b2b90 msgr2=0x7f48e80c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:43.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.680+0000 7f48ed39a700 1 --2- 172.21.15.27:0/1272533103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e80b2b90 0x7f48e80c9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:43.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.680+0000 7f48ed39a700 1 -- 172.21.15.27:0/1272533103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e0009ae0 con 0x7f48e80b07b0 2024-01-11T08:26:43.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.680+0000 7f48ee39c700 1 --2- 172.21.15.27:0/1272533103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e80b2b90 0x7f48e80c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:43.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.680+0000 7f48edb9b700 1 --2- 172.21.15.27:0/1272533103 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48e80b03e0 0x7f48e8169620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:43.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.680+0000 7f48ed39a700 1 --2- 172.21.15.27:0/1272533103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48e80b07b0 0x7f48e80c6ef0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f48d800d860 tx=0x7f48d8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:43.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.681+0000 7f48ddffb700 1 -- 172.21.15.27:0/1272533103 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d80125f0 con 0x7f48e80b07b0 2024-01-11T08:26:43.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.681+0000 7f48ddffb700 1 -- 172.21.15.27:0/1272533103 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d8012750 con 0x7f48e80b07b0 2024-01-11T08:26:43.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.681+0000 7f48efdff700 1 -- 172.21.15.27:0/1272533103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48e8169d20 con 0x7f48e80b07b0 2024-01-11T08:26:43.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.681+0000 7f48ddffb700 1 -- 172.21.15.27:0/1272533103 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d8012b10 con 0x7f48e80b07b0 2024-01-11T08:26:43.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.681+0000 7f48efdff700 1 -- 172.21.15.27:0/1272533103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48e816a2e0 con 0x7f48e80b07b0 2024-01-11T08:26:43.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.682+0000 7f48efdff700 1 -- 172.21.15.27:0/1272533103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48e80718d0 con 0x7f48e80b07b0 2024-01-11T08:26:43.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.683+0000 7f48ddffb700 1 -- 172.21.15.27:0/1272533103 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f48d801e460 con 0x7f48e80b07b0 2024-01-11T08:26:43.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.684+0000 7f48ddffb700 1 --2- 172.21.15.27:0/1272533103 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48cc066f30 0x7f48cc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:43.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.684+0000 7f48ddffb700 1 -- 172.21.15.27:0/1272533103 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f48d8084c50 con 0x7f48e80b07b0 2024-01-11T08:26:43.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.684+0000 7f48edb9b700 1 --2- 172.21.15.27:0/1272533103 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48cc066f30 0x7f48cc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:43.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.685+0000 7f48edb9b700 1 --2- 172.21.15.27:0/1272533103 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48cc066f30 0x7f48cc0693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f48e4001840 tx=0x7f48e4009040 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:43.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.688+0000 7f48ddffb700 1 -- 172.21.15.27:0/1272533103 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f48d8051e00 con 0x7f48e80b07b0 2024-01-11T08:26:43.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.887+0000 7f48efdff700 1 -- 172.21.15.27:0/1272533103 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f48e80c5170 con 0x7f48cc066f30 2024-01-11T08:26:43.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.896+0000 7f48ddffb700 1 -- 172.21.15.27:0/1272533103 <== mgr.6745 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289244 (secure 0 0 0) 0x7f48e80c5170 con 0x7f48cc066f30 2024-01-11T08:26:43.900 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:43.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.900+0000 7f48efdff700 1 -- 172.21.15.27:0/1272533103 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48cc066f30 msgr2=0x7f48cc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:43.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.900+0000 7f48efdff700 1 --2- 172.21.15.27:0/1272533103 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48cc066f30 0x7f48cc0693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f48e4001840 tx=0x7f48e4009040 comp rx=0 tx=0).stop 2024-01-11T08:26:43.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.900+0000 7f48efdff700 1 -- 172.21.15.27:0/1272533103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48e80b07b0 msgr2=0x7f48e80c6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.900+0000 7f48efdff700 1 --2- 172.21.15.27:0/1272533103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48e80b07b0 0x7f48e80c6ef0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f48d800d860 tx=0x7f48d8008040 comp rx=0 tx=0).stop 2024-01-11T08:26:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.900+0000 7f48efdff700 1 -- 172.21.15.27:0/1272533103 shutdown_connections 2024-01-11T08:26:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.900+0000 7f48efdff700 1 --2- 172.21.15.27:0/1272533103 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48cc066f30 0x7f48cc0693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.900+0000 7f48efdff700 1 --2- 172.21.15.27:0/1272533103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e80b2b90 0x7f48e80c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.900+0000 7f48efdff700 1 --2- 172.21.15.27:0/1272533103 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48e80b03e0 0x7f48e8169620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.900+0000 7f48efdff700 1 --2- 172.21.15.27:0/1272533103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48e80b07b0 0x7f48e80c6ef0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.900+0000 7f48efdff700 1 -- 172.21.15.27:0/1272533103 >> 172.21.15.27:0/1272533103 conn(0x7f48e806ed40 msgr2=0x7f48e80b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.900+0000 7f48efdff700 1 -- 172.21.15.27:0/1272533103 shutdown_connections 2024-01-11T08:26:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:43.900+0000 7f48efdff700 1 -- 172.21.15.27:0/1272533103 wait complete. 2024-01-11T08:26:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:26:43.919 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-11T08:26:43.139056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":848,"num_read_kb":1718,"num_write":355,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":239,"ondisk_log_size":239,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":75,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2438328,"kb_used_data":10872,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752536392,"statfs":{"total":773094113280,"available":770597265408,"internally_reserved":0,"allocated":11132928,"data_stored":5755020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001780"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.026453+0000","last_change":"2024-01-11T08:26:39.026453+0000","last_active":"2024-01-11T08:26:39.026453+0000","last_peered":"2024-01-11T08:26:39.026453+0000","last_clean":"2024-01-11T08:26:39.026453+0000","last_became_active":"2024-01-11T08:26:39.026117+0000","last_became_peered":"2024-01-11T08:26:39.026117+0000","last_unstale":"2024-01-11T08:26:39.026453+0000","last_undegraded":"2024-01-11T08:26:39.026453+0000","last_fullsized":"2024-01-11T08:26:39.026453+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:26:01.025745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.044291+0000","last_change":"2024-01-11T08:26:39.040377+0000","last_active":"2024-01-11T08:26:39.044291+0000","last_peered":"2024-01-11T08:26:39.044291+0000","last_clean":"2024-01-11T08:26:39.044291+0000","last_became_active":"2024-01-11T08:26:39.040083+0000","last_became_peered":"2024-01-11T08:26:39.040083+0000","last_unstale":"2024-01-11T08:26:39.044291+0000","last_undegraded":"2024-01-11T08:26:39.044291+0000","last_fullsized":"2024-01-11T08:26:39.044291+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:24:59.410081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.033071+0000","last_change":"2024-01-11T08:26:39.030880+0000","last_active":"2024-01-11T08:26:39.033071+0000","last_peered":"2024-01-11T08:26:39.033071+0000","last_clean":"2024-01-11T08:26:39.033071+0000","last_became_active":"2024-01-11T08:26:39.030170+0000","last_became_peered":"2024-01-11T08:26:39.030170+0000","last_unstale":"2024-01-11T08:26:39.033071+0000","last_undegraded":"2024-01-11T08:26:39.033071+0000","last_fullsized":"2024-01-11T08:26:39.033071+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:51:42.937889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.039254+0000","last_change":"2024-01-11T08:26:39.039254+0000","last_active":"2024-01-11T08:26:39.039254+0000","last_peered":"2024-01-11T08:26:39.039254+0000","last_clean":"2024-01-11T08:26:39.039254+0000","last_became_active":"2024-01-11T08:26:39.038895+0000","last_became_peered":"2024-01-11T08:26:39.038895+0000","last_unstale":"2024-01-11T08:26:39.039254+0000","last_undegraded":"2024-01-11T08:26:39.039254+0000","last_fullsized":"2024-01-11T08:26:39.039254+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:47:14.391963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.036900+0000","last_change":"2024-01-11T08:26:39.035301+0000","last_active":"2024-01-11T08:26:39.036900+0000","last_peered":"2024-01-11T08:26:39.036900+0000","last_clean":"2024-01-11T08:26:39.036900+0000","last_became_active":"2024-01-11T08:26:39.034869+0000","last_became_peered":"2024-01-11T08:26:39.034869+0000","last_unstale":"2024-01-11T08:26:39.036900+0000","last_undegraded":"2024-01-11T08:26:39.036900+0000","last_fullsized":"2024-01-11T08:26:39.036900+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:26:01.025745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.022104+0000","last_change":"2024-01-11T08:26:39.022104+0000","last_active":"2024-01-11T08:26:39.022104+0000","last_peered":"2024-01-11T08:26:39.022104+0000","last_clean":"2024-01-11T08:26:39.022104+0000","last_became_active":"2024-01-11T08:26:39.020953+0000","last_became_peered":"2024-01-11T08:26:39.020953+0000","last_unstale":"2024-01-11T08:26:39.022104+0000","last_undegraded":"2024-01-11T08:26:39.022104+0000","last_fullsized":"2024-01-11T08:26:39.022104+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:41:19.595025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.025189+0000","last_change":"2024-01-11T08:26:39.025189+0000","last_active":"2024-01-11T08:26:39.025189+0000","last_peered":"2024-01-11T08:26:39.025189+0000","last_clean":"2024-01-11T08:26:39.025189+0000","last_became_active":"2024-01-11T08:26:39.024909+0000","last_became_peered":"2024-01-11T08:26:39.024909+0000","last_unstale":"2024-01-11T08:26:39.025189+0000","last_undegraded":"2024-01-11T08:26:39.025189+0000","last_fullsized":"2024-01-11T08:26:39.025189+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:08:02.532479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.040501+0000","last_change":"2024-01-11T08:26:39.040501+0000","last_active":"2024-01-11T08:26:39.040501+0000","last_peered":"2024-01-11T08:26:39.040501+0000","last_clean":"2024-01-11T08:26:39.040501+0000","last_became_active":"2024-01-11T08:26:39.040152+0000","last_became_peered":"2024-01-11T08:26:39.040152+0000","last_unstale":"2024-01-11T08:26:39.040501+0000","last_undegraded":"2024-01-11T08:26:39.040501+0000","last_fullsized":"2024-01-11T08:26:39.040501+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:59:43.220997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.035617+0000","last_change":"2024-01-11T08:26:39.035617+0000","last_active":"2024-01-11T08:26:39.035617+0000","last_peered":"2024-01-11T08:26:39.035617+0000","last_clean":"2024-01-11T08:26:39.035617+0000","last_became_active":"2024-01-11T08:26:39.035303+0000","last_became_peered":"2024-01-11T08:26:39.035303+0000","last_unstale":"2024-01-11T08:26:39.035617+0000","last_undegraded":"2024-01-11T08:26:39.035617+0000","last_fullsized":"2024-01-11T08:26:39.035617+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:21:29.172470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.015655+0000","last_change":"2024-01-11T08:26:39.015655+0000","last_active":"2024-01-11T08:26:39.015655+0000","last_peered":"2024-01-11T08:26:39.015655+0000","last_clean":"2024-01-11T08:26:39.015655+0000","last_became_active":"2024-01-11T08:26:39.015333+0000","last_became_peered":"2024-01-11T08:26:39.015333+0000","last_unstale":"2024-01-11T08:26:39.015655+0000","last_undegraded":"2024-01-11T08:26:39.015655+0000","last_fullsized":"2024-01-11T08:26:39.015655+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:10:31.552465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.039793+0000","last_change":"2024-01-11T08:26:39.035866+0000","last_active":"2024-01-11T08:26:39.039793+0000","last_peered":"2024-01-11T08:26:39.039793+0000","last_clean":"2024-01-11T08:26:39.039793+0000","last_became_active":"2024-01-11T08:26:39.035538+0000","last_became_peered":"2024-01-11T08:26:39.035538+0000","last_unstale":"2024-01-11T08:26:39.039793+0000","last_undegraded":"2024-01-11T08:26:39.039793+0000","last_fullsized":"2024-01-11T08:26:39.039793+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:41:52.664240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.014776+0000","last_change":"2024-01-11T08:26:39.014776+0000","last_active":"2024-01-11T08:26:39.014776+0000","last_peered":"2024-01-11T08:26:39.014776+0000","last_clean":"2024-01-11T08:26:39.014776+0000","last_became_active":"2024-01-11T08:26:39.014212+0000","last_became_peered":"2024-01-11T08:26:39.014212+0000","last_unstale":"2024-01-11T08:26:39.014776+0000","last_undegraded":"2024-01-11T08:26:39.014776+0000","last_fullsized":"2024-01-11T08:26:39.014776+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:41:19.595025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.026903+0000","last_change":"2024-01-11T08:26:39.025545+0000","last_active":"2024-01-11T08:26:39.026903+0000","last_peered":"2024-01-11T08:26:39.026903+0000","last_clean":"2024-01-11T08:26:39.026903+0000","last_became_active":"2024-01-11T08:26:39.025228+0000","last_became_peered":"2024-01-11T08:26:39.025228+0000","last_unstale":"2024-01-11T08:26:39.026903+0000","last_undegraded":"2024-01-11T08:26:39.026903+0000","last_fullsized":"2024-01-11T08:26:39.026903+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:41:19.595025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.039746+0000","last_change":"2024-01-11T08:26:39.039746+0000","last_active":"2024-01-11T08:26:39.039746+0000","last_peered":"2024-01-11T08:26:39.039746+0000","last_clean":"2024-01-11T08:26:39.039746+0000","last_became_active":"2024-01-11T08:26:39.038891+0000","last_became_peered":"2024-01-11T08:26:39.038891+0000","last_unstale":"2024-01-11T08:26:39.039746+0000","last_undegraded":"2024-01-11T08:26:39.039746+0000","last_fullsized":"2024-01-11T08:26:39.039746+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:23:35.971073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.036008+0000","last_change":"2024-01-11T08:26:39.029787+0000","last_active":"2024-01-11T08:26:39.036008+0000","last_peered":"2024-01-11T08:26:39.036008+0000","last_clean":"2024-01-11T08:26:39.036008+0000","last_became_active":"2024-01-11T08:26:39.026721+0000","last_became_peered":"2024-01-11T08:26:39.026721+0000","last_unstale":"2024-01-11T08:26:39.036008+0000","last_undegraded":"2024-01-11T08:26:39.036008+0000","last_fullsized":"2024-01-11T08:26:39.036008+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:29.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.024416+0000","last_change":"2024-01-11T08:26:39.017023+0000","last_active":"2024-01-11T08:26:39.024416+0000","last_peered":"2024-01-11T08:26:39.024416+0000","last_clean":"2024-01-11T08:26:39.024416+0000","last_became_active":"2024-01-11T08:26:39.014421+0000","last_became_peered":"2024-01-11T08:26:39.014421+0000","last_unstale":"2024-01-11T08:26:39.024416+0000","last_undegraded":"2024-01-11T08:26:39.024416+0000","last_fullsized":"2024-01-11T08:26:39.024416+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:29.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.028040+0000","last_change":"2024-01-11T08:26:39.023050+0000","last_active":"2024-01-11T08:26:39.028040+0000","last_peered":"2024-01-11T08:26:39.028040+0000","last_clean":"2024-01-11T08:26:39.028040+0000","last_became_active":"2024-01-11T08:26:39.022784+0000","last_became_peered":"2024-01-11T08:26:39.022784+0000","last_unstale":"2024-01-11T08:26:39.028040+0000","last_undegraded":"2024-01-11T08:26:39.028040+0000","last_fullsized":"2024-01-11T08:26:39.028040+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:40:47.685006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.028004+0000","last_change":"2024-01-11T08:26:39.028004+0000","last_active":"2024-01-11T08:26:39.028004+0000","last_peered":"2024-01-11T08:26:39.028004+0000","last_clean":"2024-01-11T08:26:39.028004+0000","last_became_active":"2024-01-11T08:26:39.027382+0000","last_became_peered":"2024-01-11T08:26:39.027382+0000","last_unstale":"2024-01-11T08:26:39.028004+0000","last_undegraded":"2024-01-11T08:26:39.028004+0000","last_fullsized":"2024-01-11T08:26:39.028004+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:31:17.740400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.037841+0000","last_change":"2024-01-11T08:26:39.037841+0000","last_active":"2024-01-11T08:26:39.037841+0000","last_peered":"2024-01-11T08:26:39.037841+0000","last_clean":"2024-01-11T08:26:39.037841+0000","last_became_active":"2024-01-11T08:26:39.037447+0000","last_became_peered":"2024-01-11T08:26:39.037447+0000","last_unstale":"2024-01-11T08:26:39.037841+0000","last_undegraded":"2024-01-11T08:26:39.037841+0000","last_fullsized":"2024-01-11T08:26:39.037841+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:21:29.172470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.031822+0000","last_change":"2024-01-11T08:26:39.031822+0000","last_active":"2024-01-11T08:26:39.031822+0000","last_peered":"2024-01-11T08:26:39.031822+0000","last_clean":"2024-01-11T08:26:39.031822+0000","last_became_active":"2024-01-11T08:26:39.031506+0000","last_became_peered":"2024-01-11T08:26:39.031506+0000","last_unstale":"2024-01-11T08:26:39.031822+0000","last_undegraded":"2024-01-11T08:26:39.031822+0000","last_fullsized":"2024-01-11T08:26:39.031822+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:01:21.366890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.033968+0000","last_change":"2024-01-11T08:26:39.033968+0000","last_active":"2024-01-11T08:26:39.033968+0000","last_peered":"2024-01-11T08:26:39.033968+0000","last_clean":"2024-01-11T08:26:39.033968+0000","last_became_active":"2024-01-11T08:26:39.033642+0000","last_became_peered":"2024-01-11T08:26:39.033642+0000","last_unstale":"2024-01-11T08:26:39.033968+0000","last_undegraded":"2024-01-11T08:26:39.033968+0000","last_fullsized":"2024-01-11T08:26:39.033968+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:09:26.014852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.051386+0000","last_change":"2024-01-11T08:26:39.047029+0000","last_active":"2024-01-11T08:26:39.051386+0000","last_peered":"2024-01-11T08:26:39.051386+0000","last_clean":"2024-01-11T08:26:39.051386+0000","last_became_active":"2024-01-11T08:26:39.046202+0000","last_became_peered":"2024-01-11T08:26:39.046202+0000","last_unstale":"2024-01-11T08:26:39.051386+0000","last_undegraded":"2024-01-11T08:26:39.051386+0000","last_fullsized":"2024-01-11T08:26:39.051386+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:06:23.950780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.039659+0000","last_change":"2024-01-11T08:26:39.039659+0000","last_active":"2024-01-11T08:26:39.039659+0000","last_peered":"2024-01-11T08:26:39.039659+0000","last_clean":"2024-01-11T08:26:39.039659+0000","last_became_active":"2024-01-11T08:26:39.039356+0000","last_became_peered":"2024-01-11T08:26:39.039356+0000","last_unstale":"2024-01-11T08:26:39.039659+0000","last_undegraded":"2024-01-11T08:26:39.039659+0000","last_fullsized":"2024-01-11T08:26:39.039659+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:24:22.444046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.045653+0000","last_change":"2024-01-11T08:26:39.045653+0000","last_active":"2024-01-11T08:26:39.045653+0000","last_peered":"2024-01-11T08:26:39.045653+0000","last_clean":"2024-01-11T08:26:39.045653+0000","last_became_active":"2024-01-11T08:26:39.045358+0000","last_became_peered":"2024-01-11T08:26:39.045358+0000","last_unstale":"2024-01-11T08:26:39.045653+0000","last_undegraded":"2024-01-11T08:26:39.045653+0000","last_fullsized":"2024-01-11T08:26:39.045653+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:22:57.054076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.044176+0000","last_change":"2024-01-11T08:26:39.037990+0000","last_active":"2024-01-11T08:26:39.044176+0000","last_peered":"2024-01-11T08:26:39.044176+0000","last_clean":"2024-01-11T08:26:39.044176+0000","last_became_active":"2024-01-11T08:26:39.036034+0000","last_became_peered":"2024-01-11T08:26:39.036034+0000","last_unstale":"2024-01-11T08:26:39.044176+0000","last_undegraded":"2024-01-11T08:26:39.044176+0000","last_fullsized":"2024-01-11T08:26:39.044176+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:15:20.506477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.036929+0000","last_change":"2024-01-11T08:26:39.035691+0000","last_active":"2024-01-11T08:26:39.036929+0000","last_peered":"2024-01-11T08:26:39.036929+0000","last_clean":"2024-01-11T08:26:39.036929+0000","last_became_active":"2024-01-11T08:26:39.034981+0000","last_became_peered":"2024-01-11T08:26:39.034981+0000","last_unstale":"2024-01-11T08:26:39.036929+0000","last_undegraded":"2024-01-11T08:26:39.036929+0000","last_fullsized":"2024-01-11T08:26:39.036929+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:40:47.685006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.028441+0000","last_change":"2024-01-11T08:26:39.023632+0000","last_active":"2024-01-11T08:26:39.028441+0000","last_peered":"2024-01-11T08:26:39.028441+0000","last_clean":"2024-01-11T08:26:39.028441+0000","last_became_active":"2024-01-11T08:26:39.023294+0000","last_became_peered":"2024-01-11T08:26:39.023294+0000","last_unstale":"2024-01-11T08:26:39.028441+0000","last_undegraded":"2024-01-11T08:26:39.028441+0000","last_fullsized":"2024-01-11T08:26:39.028441+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:35.454673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.017576+0000","last_change":"2024-01-11T08:26:39.017576+0000","last_active":"2024-01-11T08:26:39.017576+0000","last_peered":"2024-01-11T08:26:39.017576+0000","last_clean":"2024-01-11T08:26:39.017576+0000","last_became_active":"2024-01-11T08:26:39.017275+0000","last_became_peered":"2024-01-11T08:26:39.017275+0000","last_unstale":"2024-01-11T08:26:39.017576+0000","last_undegraded":"2024-01-11T08:26:39.017576+0000","last_fullsized":"2024-01-11T08:26:39.017576+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:13:27.550996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.041371+0000","last_change":"2024-01-11T08:26:39.034358+0000","last_active":"2024-01-11T08:26:39.041371+0000","last_peered":"2024-01-11T08:26:39.041371+0000","last_clean":"2024-01-11T08:26:39.041371+0000","last_became_active":"2024-01-11T08:26:39.033994+0000","last_became_peered":"2024-01-11T08:26:39.033994+0000","last_unstale":"2024-01-11T08:26:39.041371+0000","last_undegraded":"2024-01-11T08:26:39.041371+0000","last_fullsized":"2024-01-11T08:26:39.041371+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:15:40.371755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.051403+0000","last_change":"2024-01-11T08:26:39.047201+0000","last_active":"2024-01-11T08:26:39.051403+0000","last_peered":"2024-01-11T08:26:39.051403+0000","last_clean":"2024-01-11T08:26:39.051403+0000","last_became_active":"2024-01-11T08:26:39.045817+0000","last_became_peered":"2024-01-11T08:26:39.045817+0000","last_unstale":"2024-01-11T08:26:39.051403+0000","last_undegraded":"2024-01-11T08:26:39.051403+0000","last_fullsized":"2024-01-11T08:26:39.051403+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:19:25.941718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.033729+0000","last_change":"2024-01-11T08:26:39.030895+0000","last_active":"2024-01-11T08:26:39.033729+0000","last_peered":"2024-01-11T08:26:39.033729+0000","last_clean":"2024-01-11T08:26:39.033729+0000","last_became_active":"2024-01-11T08:26:39.030619+0000","last_became_peered":"2024-01-11T08:26:39.030619+0000","last_unstale":"2024-01-11T08:26:39.033729+0000","last_undegraded":"2024-01-11T08:26:39.033729+0000","last_fullsized":"2024-01-11T08:26:39.033729+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:56:34.267505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.039191+0000","last_change":"2024-01-11T08:26:39.039191+0000","last_active":"2024-01-11T08:26:39.039191+0000","last_peered":"2024-01-11T08:26:39.039191+0000","last_clean":"2024-01-11T08:26:39.039191+0000","last_became_active":"2024-01-11T08:26:39.038809+0000","last_became_peered":"2024-01-11T08:26:39.038809+0000","last_unstale":"2024-01-11T08:26:39.039191+0000","last_undegraded":"2024-01-11T08:26:39.039191+0000","last_fullsized":"2024-01-11T08:26:39.039191+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:59:43.220997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.027681+0000","last_change":"2024-01-11T08:26:39.020588+0000","last_active":"2024-01-11T08:26:39.027681+0000","last_peered":"2024-01-11T08:26:39.027681+0000","last_clean":"2024-01-11T08:26:39.027681+0000","last_became_active":"2024-01-11T08:26:39.020306+0000","last_became_peered":"2024-01-11T08:26:39.020306+0000","last_unstale":"2024-01-11T08:26:39.027681+0000","last_undegraded":"2024-01-11T08:26:39.027681+0000","last_fullsized":"2024-01-11T08:26:39.027681+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:41:19.595025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"100'4","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.044056+0000","last_change":"2024-01-11T08:26:38.029051+0000","last_active":"2024-01-11T08:26:39.044056+0000","last_peered":"2024-01-11T08:26:39.044056+0000","last_clean":"2024-01-11T08:26:39.044056+0000","last_became_active":"2024-01-11T08:26:38.028640+0000","last_became_peered":"2024-01-11T08:26:38.028640+0000","last_unstale":"2024-01-11T08:26:39.044056+0000","last_undegraded":"2024-01-11T08:26:39.044056+0000","last_fullsized":"2024-01-11T08:26:39.044056+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:44:56.089483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.036461+0000","last_change":"2024-01-11T08:26:39.036461+0000","last_active":"2024-01-11T08:26:39.036461+0000","last_peered":"2024-01-11T08:26:39.036461+0000","last_clean":"2024-01-11T08:26:39.036461+0000","last_became_active":"2024-01-11T08:26:39.036132+0000","last_became_peered":"2024-01-11T08:26:39.036132+0000","last_unstale":"2024-01-11T08:26:39.036461+0000","last_undegraded":"2024-01-11T08:26:39.036461+0000","last_fullsized":"2024-01-11T08:26:39.036461+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:28:14.292547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"100'1","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.028425+0000","last_change":"2024-01-11T08:26:38.028014+0000","last_active":"2024-01-11T08:26:39.028425+0000","last_peered":"2024-01-11T08:26:39.028425+0000","last_clean":"2024-01-11T08:26:39.028425+0000","last_became_active":"2024-01-11T08:26:38.027309+0000","last_became_peered":"2024-01-11T08:26:38.027309+0000","last_unstale":"2024-01-11T08:26:39.028425+0000","last_undegraded":"2024-01-11T08:26:39.028425+0000","last_fullsized":"2024-01-11T08:26:39.028425+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:51:04.670324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.037481+0000","last_change":"2024-01-11T08:26:39.037481+0000","last_active":"2024-01-11T08:26:39.037481+0000","last_peered":"2024-01-11T08:26:39.037481+0000","last_clean":"2024-01-11T08:26:39.037481+0000","last_became_active":"2024-01-11T08:26:39.037222+0000","last_became_peered":"2024-01-11T08:26:39.037222+0000","last_unstale":"2024-01-11T08:26:39.037481+0000","last_undegraded":"2024-01-11T08:26:39.037481+0000","last_fullsized":"2024-01-11T08:26:39.037481+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:49:22.217423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"100'2","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.048410+0000","last_change":"2024-01-11T08:26:38.029327+0000","last_active":"2024-01-11T08:26:39.048410+0000","last_peered":"2024-01-11T08:26:39.048410+0000","last_clean":"2024-01-11T08:26:39.048410+0000","last_became_active":"2024-01-11T08:26:38.029007+0000","last_became_peered":"2024-01-11T08:26:38.029007+0000","last_unstale":"2024-01-11T08:26:39.048410+0000","last_undegraded":"2024-01-11T08:26:39.048410+0000","last_fullsized":"2024-01-11T08:26:39.048410+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:59:39.984404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.026871+0000","last_change":"2024-01-11T08:26:39.025023+0000","last_active":"2024-01-11T08:26:39.026871+0000","last_peered":"2024-01-11T08:26:39.026871+0000","last_clean":"2024-01-11T08:26:39.026871+0000","last_became_active":"2024-01-11T08:26:39.022587+0000","last_became_peered":"2024-01-11T08:26:39.022587+0000","last_unstale":"2024-01-11T08:26:39.026871+0000","last_undegraded":"2024-01-11T08:26:39.026871+0000","last_fullsized":"2024-01-11T08:26:39.026871+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:59:43.220997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"100'2","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.046912+0000","last_change":"2024-01-11T08:26:38.026888+0000","last_active":"2024-01-11T08:26:39.046912+0000","last_peered":"2024-01-11T08:26:39.046912+0000","last_clean":"2024-01-11T08:26:39.046912+0000","last_became_active":"2024-01-11T08:26:38.026593+0000","last_became_peered":"2024-01-11T08:26:38.026593+0000","last_unstale":"2024-01-11T08:26:39.046912+0000","last_undegraded":"2024-01-11T08:26:39.046912+0000","last_fullsized":"2024-01-11T08:26:39.046912+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:49:35.838083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.051418+0000","last_change":"2024-01-11T08:26:39.047205+0000","last_active":"2024-01-11T08:26:39.051418+0000","last_peered":"2024-01-11T08:26:39.051418+0000","last_clean":"2024-01-11T08:26:39.051418+0000","last_became_active":"2024-01-11T08:26:39.046555+0000","last_became_peered":"2024-01-11T08:26:39.046555+0000","last_unstale":"2024-01-11T08:26:39.051418+0000","last_undegraded":"2024-01-11T08:26:39.051418+0000","last_fullsized":"2024-01-11T08:26:39.051418+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:45:30.088103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"100'1","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.038636+0000","last_change":"2024-01-11T08:26:38.031001+0000","last_active":"2024-01-11T08:26:39.038636+0000","last_peered":"2024-01-11T08:26:39.038636+0000","last_clean":"2024-01-11T08:26:39.038636+0000","last_became_active":"2024-01-11T08:26:38.030566+0000","last_became_peered":"2024-01-11T08:26:38.030566+0000","last_unstale":"2024-01-11T08:26:39.038636+0000","last_undegraded":"2024-01-11T08:26:39.038636+0000","last_fullsized":"2024-01-11T08:26:39.038636+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:10:12.714352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.031349+0000","last_change":"2024-01-11T08:26:39.022783+0000","last_active":"2024-01-11T08:26:39.031349+0000","last_peered":"2024-01-11T08:26:39.031349+0000","last_clean":"2024-01-11T08:26:39.031349+0000","last_became_active":"2024-01-11T08:26:39.022489+0000","last_became_peered":"2024-01-11T08:26:39.022489+0000","last_unstale":"2024-01-11T08:26:39.031349+0000","last_undegraded":"2024-01-11T08:26:39.031349+0000","last_fullsized":"2024-01-11T08:26:39.031349+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:51:42.937889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"100'1","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.030598+0000","last_change":"2024-01-11T08:26:38.033436+0000","last_active":"2024-01-11T08:26:39.030598+0000","last_peered":"2024-01-11T08:26:39.030598+0000","last_clean":"2024-01-11T08:26:39.030598+0000","last_became_active":"2024-01-11T08:26:38.032987+0000","last_became_peered":"2024-01-11T08:26:38.032987+0000","last_unstale":"2024-01-11T08:26:39.030598+0000","last_undegraded":"2024-01-11T08:26:39.030598+0000","last_fullsized":"2024-01-11T08:26:39.030598+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:06:22.942835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.046428+0000","last_change":"2024-01-11T08:26:39.045832+0000","last_active":"2024-01-11T08:26:39.046428+0000","last_peered":"2024-01-11T08:26:39.046428+0000","last_clean":"2024-01-11T08:26:39.046428+0000","last_became_active":"2024-01-11T08:26:39.045410+0000","last_became_peered":"2024-01-11T08:26:39.045410+0000","last_unstale":"2024-01-11T08:26:39.046428+0000","last_undegraded":"2024-01-11T08:26:39.046428+0000","last_fullsized":"2024-01-11T08:26:39.046428+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:28:14.292547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.032253+0000","last_change":"2024-01-11T08:26:38.028785+0000","last_active":"2024-01-11T08:26:38.032253+0000","last_peered":"2024-01-11T08:26:38.032253+0000","last_clean":"2024-01-11T08:26:38.032253+0000","last_became_active":"2024-01-11T08:26:38.026668+0000","last_became_peered":"2024-01-11T08:26:38.026668+0000","last_unstale":"2024-01-11T08:26:38.032253+0000","last_undegraded":"2024-01-11T08:26:38.032253+0000","last_fullsized":"2024-01-11T08:26:38.032253+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:01:34.920162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.032870+0000","last_change":"2024-01-11T08:26:39.027019+0000","last_active":"2024-01-11T08:26:39.032870+0000","last_peered":"2024-01-11T08:26:39.032870+0000","last_clean":"2024-01-11T08:26:39.032870+0000","last_became_active":"2024-01-11T08:26:39.026297+0000","last_became_peered":"2024-01-11T08:26:39.026297+0000","last_unstale":"2024-01-11T08:26:39.032870+0000","last_undegraded":"2024-01-11T08:26:39.032870+0000","last_fullsized":"2024-01-11T08:26:39.032870+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:21:29.172470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":10,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.030185+0000","last_change":"2024-01-11T08:26:38.027465+0000","last_active":"2024-01-11T08:26:39.030185+0000","last_peered":"2024-01-11T08:26:39.030185+0000","last_clean":"2024-01-11T08:26:39.030185+0000","last_became_active":"2024-01-11T08:26:38.027171+0000","last_became_peered":"2024-01-11T08:26:38.027171+0000","last_unstale":"2024-01-11T08:26:39.030185+0000","last_undegraded":"2024-01-11T08:26:39.030185+0000","last_fullsized":"2024-01-11T08:26:39.030185+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:31:16.732455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.037160+0000","last_change":"2024-01-11T08:26:39.035404+0000","last_active":"2024-01-11T08:26:39.037160+0000","last_peered":"2024-01-11T08:26:39.037160+0000","last_clean":"2024-01-11T08:26:39.037160+0000","last_became_active":"2024-01-11T08:26:39.035040+0000","last_became_peered":"2024-01-11T08:26:39.035040+0000","last_unstale":"2024-01-11T08:26:39.037160+0000","last_undegraded":"2024-01-11T08:26:39.037160+0000","last_fullsized":"2024-01-11T08:26:39.037160+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:41:19.595025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.032375+0000","last_change":"2024-01-11T08:26:38.028124+0000","last_active":"2024-01-11T08:26:38.032375+0000","last_peered":"2024-01-11T08:26:38.032375+0000","last_clean":"2024-01-11T08:26:38.032375+0000","last_became_active":"2024-01-11T08:26:38.027678+0000","last_became_peered":"2024-01-11T08:26:38.027678+0000","last_unstale":"2024-01-11T08:26:38.032375+0000","last_undegraded":"2024-01-11T08:26:38.032375+0000","last_fullsized":"2024-01-11T08:26:38.032375+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:46:57.348974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.023827+0000","last_change":"2024-01-11T08:26:39.023827+0000","last_active":"2024-01-11T08:26:39.023827+0000","last_peered":"2024-01-11T08:26:39.023827+0000","last_clean":"2024-01-11T08:26:39.023827+0000","last_became_active":"2024-01-11T08:26:39.023587+0000","last_became_peered":"2024-01-11T08:26:39.023587+0000","last_unstale":"2024-01-11T08:26:39.023827+0000","last_undegraded":"2024-01-11T08:26:39.023827+0000","last_fullsized":"2024-01-11T08:26:39.023827+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:40:47.685006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.033010+0000","last_change":"2024-01-11T08:26:38.030607+0000","last_active":"2024-01-11T08:26:38.033010+0000","last_peered":"2024-01-11T08:26:38.033010+0000","last_clean":"2024-01-11T08:26:38.033010+0000","last_became_active":"2024-01-11T08:26:38.030282+0000","last_became_peered":"2024-01-11T08:26:38.030282+0000","last_unstale":"2024-01-11T08:26:38.033010+0000","last_undegraded":"2024-01-11T08:26:38.033010+0000","last_fullsized":"2024-01-11T08:26:38.033010+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:06:22.942835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.033781+0000","last_change":"2024-01-11T08:26:39.033781+0000","last_active":"2024-01-11T08:26:39.033781+0000","last_peered":"2024-01-11T08:26:39.033781+0000","last_clean":"2024-01-11T08:26:39.033781+0000","last_became_active":"2024-01-11T08:26:39.032278+0000","last_became_peered":"2024-01-11T08:26:39.032278+0000","last_unstale":"2024-01-11T08:26:39.033781+0000","last_undegraded":"2024-01-11T08:26:39.033781+0000","last_fullsized":"2024-01-11T08:26:39.033781+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:51:42.937889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"100'3","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.031227+0000","last_change":"2024-01-11T08:26:38.027117+0000","last_active":"2024-01-11T08:26:39.031227+0000","last_peered":"2024-01-11T08:26:39.031227+0000","last_clean":"2024-01-11T08:26:39.031227+0000","last_became_active":"2024-01-11T08:26:38.026420+0000","last_became_peered":"2024-01-11T08:26:38.026420+0000","last_unstale":"2024-01-11T08:26:39.031227+0000","last_undegraded":"2024-01-11T08:26:39.031227+0000","last_fullsized":"2024-01-11T08:26:39.031227+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:02:42.005111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.036143+0000","last_change":"2024-01-11T08:26:39.034558+0000","last_active":"2024-01-11T08:26:39.036143+0000","last_peered":"2024-01-11T08:26:39.036143+0000","last_clean":"2024-01-11T08:26:39.036143+0000","last_became_active":"2024-01-11T08:26:39.034194+0000","last_became_peered":"2024-01-11T08:26:39.034194+0000","last_unstale":"2024-01-11T08:26:39.036143+0000","last_undegraded":"2024-01-11T08:26:39.036143+0000","last_fullsized":"2024-01-11T08:26:39.036143+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:10:31.552465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"100'6","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.033546+0000","last_change":"2024-01-11T08:26:38.024428+0000","last_active":"2024-01-11T08:26:39.033546+0000","last_peered":"2024-01-11T08:26:39.033546+0000","last_clean":"2024-01-11T08:26:39.033546+0000","last_became_active":"2024-01-11T08:26:38.022550+0000","last_became_peered":"2024-01-11T08:26:38.022550+0000","last_unstale":"2024-01-11T08:26:39.033546+0000","last_undegraded":"2024-01-11T08:26:39.033546+0000","last_fullsized":"2024-01-11T08:26:39.033546+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:46:57.348974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.033077+0000","last_change":"2024-01-11T08:26:39.033077+0000","last_active":"2024-01-11T08:26:39.033077+0000","last_peered":"2024-01-11T08:26:39.033077+0000","last_clean":"2024-01-11T08:26:39.033077+0000","last_became_active":"2024-01-11T08:26:39.032641+0000","last_became_peered":"2024-01-11T08:26:39.032641+0000","last_unstale":"2024-01-11T08:26:39.033077+0000","last_undegraded":"2024-01-11T08:26:39.033077+0000","last_fullsized":"2024-01-11T08:26:39.033077+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:29.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"100'2","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.049574+0000","last_change":"2024-01-11T08:26:38.031060+0000","last_active":"2024-01-11T08:26:39.049574+0000","last_peered":"2024-01-11T08:26:39.049574+0000","last_clean":"2024-01-11T08:26:39.049574+0000","last_became_active":"2024-01-11T08:26:38.030786+0000","last_became_peered":"2024-01-11T08:26:38.030786+0000","last_unstale":"2024-01-11T08:26:39.049574+0000","last_undegraded":"2024-01-11T08:26:39.049574+0000","last_fullsized":"2024-01-11T08:26:39.049574+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:09:25.006907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.043245+0000","last_change":"2024-01-11T08:26:39.042572+0000","last_active":"2024-01-11T08:26:39.043245+0000","last_peered":"2024-01-11T08:26:39.043245+0000","last_clean":"2024-01-11T08:26:39.043245+0000","last_became_active":"2024-01-11T08:26:39.042273+0000","last_became_peered":"2024-01-11T08:26:39.042273+0000","last_unstale":"2024-01-11T08:26:39.043245+0000","last_undegraded":"2024-01-11T08:26:39.043245+0000","last_fullsized":"2024-01-11T08:26:39.043245+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:29.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":10,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.005005+0000","last_change":"2024-01-11T08:26:38.014454+0000","last_active":"2024-01-11T08:26:39.005005+0000","last_peered":"2024-01-11T08:26:39.005005+0000","last_clean":"2024-01-11T08:26:39.005005+0000","last_became_active":"2024-01-11T08:26:38.014198+0000","last_became_peered":"2024-01-11T08:26:38.014198+0000","last_unstale":"2024-01-11T08:26:39.005005+0000","last_undegraded":"2024-01-11T08:26:39.005005+0000","last_fullsized":"2024-01-11T08:26:39.005005+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:37:07.604504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.048878+0000","last_change":"2024-01-11T08:26:39.046399+0000","last_active":"2024-01-11T08:26:39.048878+0000","last_peered":"2024-01-11T08:26:39.048878+0000","last_clean":"2024-01-11T08:26:39.048878+0000","last_became_active":"2024-01-11T08:26:39.046121+0000","last_became_peered":"2024-01-11T08:26:39.046121+0000","last_unstale":"2024-01-11T08:26:39.048878+0000","last_undegraded":"2024-01-11T08:26:39.048878+0000","last_fullsized":"2024-01-11T08:26:39.048878+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:42:07.465080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.032164+0000","last_change":"2024-01-11T08:26:38.027586+0000","last_active":"2024-01-11T08:26:38.032164+0000","last_peered":"2024-01-11T08:26:38.032164+0000","last_clean":"2024-01-11T08:26:38.032164+0000","last_became_active":"2024-01-11T08:26:38.027130+0000","last_became_peered":"2024-01-11T08:26:38.027130+0000","last_unstale":"2024-01-11T08:26:38.032164+0000","last_undegraded":"2024-01-11T08:26:38.032164+0000","last_fullsized":"2024-01-11T08:26:38.032164+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:44:56.089483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.039828+0000","last_change":"2024-01-11T08:26:39.032670+0000","last_active":"2024-01-11T08:26:39.039828+0000","last_peered":"2024-01-11T08:26:39.039828+0000","last_clean":"2024-01-11T08:26:39.039828+0000","last_became_active":"2024-01-11T08:26:39.031923+0000","last_became_peered":"2024-01-11T08:26:39.031923+0000","last_unstale":"2024-01-11T08:26:39.039828+0000","last_undegraded":"2024-01-11T08:26:39.039828+0000","last_fullsized":"2024-01-11T08:26:39.039828+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:51:05.678270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"100'8","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.049199+0000","last_change":"2024-01-11T08:26:38.024297+0000","last_active":"2024-01-11T08:26:39.049199+0000","last_peered":"2024-01-11T08:26:39.049199+0000","last_clean":"2024-01-11T08:26:39.049199+0000","last_became_active":"2024-01-11T08:26:38.024009+0000","last_became_peered":"2024-01-11T08:26:38.024009+0000","last_unstale":"2024-01-11T08:26:39.049199+0000","last_undegraded":"2024-01-11T08:26:39.049199+0000","last_fullsized":"2024-01-11T08:26:39.049199+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:46:57.348974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.027317+0000","last_change":"2024-01-11T08:26:39.026352+0000","last_active":"2024-01-11T08:26:39.027317+0000","last_peered":"2024-01-11T08:26:39.027317+0000","last_clean":"2024-01-11T08:26:39.027317+0000","last_became_active":"2024-01-11T08:26:39.026055+0000","last_became_peered":"2024-01-11T08:26:39.026055+0000","last_unstale":"2024-01-11T08:26:39.027317+0000","last_undegraded":"2024-01-11T08:26:39.027317+0000","last_fullsized":"2024-01-11T08:26:39.027317+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:53:13.699008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"100'1","reported_seq":10,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.011375+0000","last_change":"2024-01-11T08:26:38.028437+0000","last_active":"2024-01-11T08:26:39.011375+0000","last_peered":"2024-01-11T08:26:39.011375+0000","last_clean":"2024-01-11T08:26:39.011375+0000","last_became_active":"2024-01-11T08:26:38.027230+0000","last_became_peered":"2024-01-11T08:26:38.027230+0000","last_unstale":"2024-01-11T08:26:39.011375+0000","last_undegraded":"2024-01-11T08:26:39.011375+0000","last_fullsized":"2024-01-11T08:26:39.011375+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:11:41.070507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.028545+0000","last_change":"2024-01-11T08:26:39.028545+0000","last_active":"2024-01-11T08:26:39.028545+0000","last_peered":"2024-01-11T08:26:39.028545+0000","last_clean":"2024-01-11T08:26:39.028545+0000","last_became_active":"2024-01-11T08:26:39.028270+0000","last_became_peered":"2024-01-11T08:26:39.028270+0000","last_unstale":"2024-01-11T08:26:39.028545+0000","last_undegraded":"2024-01-11T08:26:39.028545+0000","last_fullsized":"2024-01-11T08:26:39.028545+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:29.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"100'2","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.047379+0000","last_change":"2024-01-11T08:26:38.029024+0000","last_active":"2024-01-11T08:26:39.047379+0000","last_peered":"2024-01-11T08:26:39.047379+0000","last_clean":"2024-01-11T08:26:39.047379+0000","last_became_active":"2024-01-11T08:26:38.028745+0000","last_became_peered":"2024-01-11T08:26:38.028745+0000","last_unstale":"2024-01-11T08:26:39.047379+0000","last_undegraded":"2024-01-11T08:26:39.047379+0000","last_fullsized":"2024-01-11T08:26:39.047379+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:45:36.644341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.040414+0000","last_change":"2024-01-11T08:26:39.039302+0000","last_active":"2024-01-11T08:26:39.040414+0000","last_peered":"2024-01-11T08:26:39.040414+0000","last_clean":"2024-01-11T08:26:39.040414+0000","last_became_active":"2024-01-11T08:26:39.038858+0000","last_became_peered":"2024-01-11T08:26:39.038858+0000","last_unstale":"2024-01-11T08:26:39.040414+0000","last_undegraded":"2024-01-11T08:26:39.040414+0000","last_fullsized":"2024-01-11T08:26:39.040414+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:21:29.172470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.030930+0000","last_change":"2024-01-11T08:26:38.025828+0000","last_active":"2024-01-11T08:26:38.030930+0000","last_peered":"2024-01-11T08:26:38.030930+0000","last_clean":"2024-01-11T08:26:38.030930+0000","last_became_active":"2024-01-11T08:26:38.025207+0000","last_became_peered":"2024-01-11T08:26:38.025207+0000","last_unstale":"2024-01-11T08:26:38.030930+0000","last_undegraded":"2024-01-11T08:26:38.030930+0000","last_fullsized":"2024-01-11T08:26:38.030930+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:29:45.372046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.040024+0000","last_change":"2024-01-11T08:26:39.040024+0000","last_active":"2024-01-11T08:26:39.040024+0000","last_peered":"2024-01-11T08:26:39.040024+0000","last_clean":"2024-01-11T08:26:39.040024+0000","last_became_active":"2024-01-11T08:26:39.039494+0000","last_became_peered":"2024-01-11T08:26:39.039494+0000","last_unstale":"2024-01-11T08:26:39.040024+0000","last_undegraded":"2024-01-11T08:26:39.040024+0000","last_fullsized":"2024-01-11T08:26:39.040024+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:01:54.270267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"100'2","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.022160+0000","last_change":"2024-01-11T08:26:38.017810+0000","last_active":"2024-01-11T08:26:39.022160+0000","last_peered":"2024-01-11T08:26:39.022160+0000","last_clean":"2024-01-11T08:26:39.022160+0000","last_became_active":"2024-01-11T08:26:38.017522+0000","last_became_peered":"2024-01-11T08:26:38.017522+0000","last_unstale":"2024-01-11T08:26:39.022160+0000","last_undegraded":"2024-01-11T08:26:39.022160+0000","last_fullsized":"2024-01-11T08:26:39.022160+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:53:12.691063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.018145+0000","last_change":"2024-01-11T08:26:39.018145+0000","last_active":"2024-01-11T08:26:39.018145+0000","last_peered":"2024-01-11T08:26:39.018145+0000","last_clean":"2024-01-11T08:26:39.018145+0000","last_became_active":"2024-01-11T08:26:39.017838+0000","last_became_peered":"2024-01-11T08:26:39.017838+0000","last_unstale":"2024-01-11T08:26:39.018145+0000","last_undegraded":"2024-01-11T08:26:39.018145+0000","last_fullsized":"2024-01-11T08:26:39.018145+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:51:42.937889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.031282+0000","last_change":"2024-01-11T08:26:38.027479+0000","last_active":"2024-01-11T08:26:38.031282+0000","last_peered":"2024-01-11T08:26:38.031282+0000","last_clean":"2024-01-11T08:26:38.031282+0000","last_became_active":"2024-01-11T08:26:38.027200+0000","last_became_peered":"2024-01-11T08:26:38.027200+0000","last_unstale":"2024-01-11T08:26:38.031282+0000","last_undegraded":"2024-01-11T08:26:38.031282+0000","last_fullsized":"2024-01-11T08:26:38.031282+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:00:45.069014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.026743+0000","last_change":"2024-01-11T08:26:39.026743+0000","last_active":"2024-01-11T08:26:39.026743+0000","last_peered":"2024-01-11T08:26:39.026743+0000","last_clean":"2024-01-11T08:26:39.026743+0000","last_became_active":"2024-01-11T08:26:39.026488+0000","last_became_peered":"2024-01-11T08:26:39.026488+0000","last_unstale":"2024-01-11T08:26:39.026743+0000","last_undegraded":"2024-01-11T08:26:39.026743+0000","last_fullsized":"2024-01-11T08:26:39.026743+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:21:31.207621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.027169+0000","last_change":"2024-01-11T08:26:38.023219+0000","last_active":"2024-01-11T08:26:38.027169+0000","last_peered":"2024-01-11T08:26:38.027169+0000","last_clean":"2024-01-11T08:26:38.027169+0000","last_became_active":"2024-01-11T08:26:38.022963+0000","last_became_peered":"2024-01-11T08:26:38.022963+0000","last_unstale":"2024-01-11T08:26:38.027169+0000","last_undegraded":"2024-01-11T08:26:38.027169+0000","last_fullsized":"2024-01-11T08:26:38.027169+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:41:53.413360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.028292+0000","last_change":"2024-01-11T08:26:39.028292+0000","last_active":"2024-01-11T08:26:39.028292+0000","last_peered":"2024-01-11T08:26:39.028292+0000","last_clean":"2024-01-11T08:26:39.028292+0000","last_became_active":"2024-01-11T08:26:39.027473+0000","last_became_peered":"2024-01-11T08:26:39.027473+0000","last_unstale":"2024-01-11T08:26:39.028292+0000","last_undegraded":"2024-01-11T08:26:39.028292+0000","last_fullsized":"2024-01-11T08:26:39.028292+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:28:14.292547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":10,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.004709+0000","last_change":"2024-01-11T08:26:38.023339+0000","last_active":"2024-01-11T08:26:39.004709+0000","last_peered":"2024-01-11T08:26:39.004709+0000","last_clean":"2024-01-11T08:26:39.004709+0000","last_became_active":"2024-01-11T08:26:38.022815+0000","last_became_peered":"2024-01-11T08:26:38.022815+0000","last_unstale":"2024-01-11T08:26:39.004709+0000","last_undegraded":"2024-01-11T08:26:39.004709+0000","last_fullsized":"2024-01-11T08:26:39.004709+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:27:28.154658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.027659+0000","last_change":"2024-01-11T08:26:39.019719+0000","last_active":"2024-01-11T08:26:39.027659+0000","last_peered":"2024-01-11T08:26:39.027659+0000","last_clean":"2024-01-11T08:26:39.027659+0000","last_became_active":"2024-01-11T08:26:39.019454+0000","last_became_peered":"2024-01-11T08:26:39.019454+0000","last_unstale":"2024-01-11T08:26:39.027659+0000","last_undegraded":"2024-01-11T08:26:39.027659+0000","last_fullsized":"2024-01-11T08:26:39.027659+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:28:14.292547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"100'2","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.042441+0000","last_change":"2024-01-11T08:26:38.019388+0000","last_active":"2024-01-11T08:26:39.042441+0000","last_peered":"2024-01-11T08:26:39.042441+0000","last_clean":"2024-01-11T08:26:39.042441+0000","last_became_active":"2024-01-11T08:26:38.019116+0000","last_became_peered":"2024-01-11T08:26:38.019116+0000","last_unstale":"2024-01-11T08:26:39.042441+0000","last_undegraded":"2024-01-11T08:26:39.042441+0000","last_fullsized":"2024-01-11T08:26:39.042441+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:12:33.643792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.033022+0000","last_change":"2024-01-11T08:26:39.030601+0000","last_active":"2024-01-11T08:26:39.033022+0000","last_peered":"2024-01-11T08:26:39.033022+0000","last_clean":"2024-01-11T08:26:39.033022+0000","last_became_active":"2024-01-11T08:26:39.029663+0000","last_became_peered":"2024-01-11T08:26:39.029663+0000","last_unstale":"2024-01-11T08:26:39.033022+0000","last_undegraded":"2024-01-11T08:26:39.033022+0000","last_fullsized":"2024-01-11T08:26:39.033022+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:21:29.172470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.033748+0000","last_change":"2024-01-11T08:26:38.029559+0000","last_active":"2024-01-11T08:26:39.033748+0000","last_peered":"2024-01-11T08:26:39.033748+0000","last_clean":"2024-01-11T08:26:39.033748+0000","last_became_active":"2024-01-11T08:26:38.029268+0000","last_became_peered":"2024-01-11T08:26:38.029268+0000","last_unstale":"2024-01-11T08:26:39.033748+0000","last_undegraded":"2024-01-11T08:26:39.033748+0000","last_fullsized":"2024-01-11T08:26:39.033748+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:38:36.065914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.028113+0000","last_change":"2024-01-11T08:26:39.028113+0000","last_active":"2024-01-11T08:26:39.028113+0000","last_peered":"2024-01-11T08:26:39.028113+0000","last_clean":"2024-01-11T08:26:39.028113+0000","last_became_active":"2024-01-11T08:26:39.027608+0000","last_became_peered":"2024-01-11T08:26:39.027608+0000","last_unstale":"2024-01-11T08:26:39.028113+0000","last_undegraded":"2024-01-11T08:26:39.028113+0000","last_fullsized":"2024-01-11T08:26:39.028113+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:56:34.267505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.035303+0000","last_change":"2024-01-11T08:26:38.032356+0000","last_active":"2024-01-11T08:26:38.035303+0000","last_peered":"2024-01-11T08:26:38.035303+0000","last_clean":"2024-01-11T08:26:38.035303+0000","last_became_active":"2024-01-11T08:26:38.031798+0000","last_became_peered":"2024-01-11T08:26:38.031798+0000","last_unstale":"2024-01-11T08:26:38.035303+0000","last_undegraded":"2024-01-11T08:26:38.035303+0000","last_fullsized":"2024-01-11T08:26:38.035303+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:24:58.402136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.044906+0000","last_change":"2024-01-11T08:26:39.044000+0000","last_active":"2024-01-11T08:26:39.044906+0000","last_peered":"2024-01-11T08:26:39.044906+0000","last_clean":"2024-01-11T08:26:39.044906+0000","last_became_active":"2024-01-11T08:26:39.043498+0000","last_became_peered":"2024-01-11T08:26:39.043498+0000","last_unstale":"2024-01-11T08:26:39.044906+0000","last_undegraded":"2024-01-11T08:26:39.044906+0000","last_fullsized":"2024-01-11T08:26:39.044906+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:59:43.220997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.026181+0000","last_change":"2024-01-11T08:26:38.022488+0000","last_active":"2024-01-11T08:26:38.026181+0000","last_peered":"2024-01-11T08:26:38.026181+0000","last_clean":"2024-01-11T08:26:38.026181+0000","last_became_active":"2024-01-11T08:26:38.022007+0000","last_became_peered":"2024-01-11T08:26:38.022007+0000","last_unstale":"2024-01-11T08:26:38.026181+0000","last_undegraded":"2024-01-11T08:26:38.026181+0000","last_fullsized":"2024-01-11T08:26:38.026181+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:51:04.670324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.018493+0000","last_change":"2024-01-11T08:26:39.018493+0000","last_active":"2024-01-11T08:26:39.018493+0000","last_peered":"2024-01-11T08:26:39.018493+0000","last_clean":"2024-01-11T08:26:39.018493+0000","last_became_active":"2024-01-11T08:26:39.018183+0000","last_became_peered":"2024-01-11T08:26:39.018183+0000","last_unstale":"2024-01-11T08:26:39.018493+0000","last_undegraded":"2024-01-11T08:26:39.018493+0000","last_fullsized":"2024-01-11T08:26:39.018493+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:29.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.025595+0000","last_change":"2024-01-11T08:26:38.022216+0000","last_active":"2024-01-11T08:26:38.025595+0000","last_peered":"2024-01-11T08:26:38.025595+0000","last_clean":"2024-01-11T08:26:38.025595+0000","last_became_active":"2024-01-11T08:26:38.021945+0000","last_became_peered":"2024-01-11T08:26:38.021945+0000","last_unstale":"2024-01-11T08:26:38.025595+0000","last_undegraded":"2024-01-11T08:26:38.025595+0000","last_fullsized":"2024-01-11T08:26:38.025595+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:44:56.089483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.044928+0000","last_change":"2024-01-11T08:26:39.044928+0000","last_active":"2024-01-11T08:26:39.044928+0000","last_peered":"2024-01-11T08:26:39.044928+0000","last_clean":"2024-01-11T08:26:39.044928+0000","last_became_active":"2024-01-11T08:26:39.044353+0000","last_became_peered":"2024-01-11T08:26:39.044353+0000","last_unstale":"2024-01-11T08:26:39.044928+0000","last_undegraded":"2024-01-11T08:26:39.044928+0000","last_fullsized":"2024-01-11T08:26:39.044928+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:28:14.292547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.015276+0000","last_change":"2024-01-11T08:26:38.030355+0000","last_active":"2024-01-11T08:26:39.015276+0000","last_peered":"2024-01-11T08:26:39.015276+0000","last_clean":"2024-01-11T08:26:39.015276+0000","last_became_active":"2024-01-11T08:26:38.030099+0000","last_became_peered":"2024-01-11T08:26:38.030099+0000","last_unstale":"2024-01-11T08:26:39.015276+0000","last_undegraded":"2024-01-11T08:26:39.015276+0000","last_fullsized":"2024-01-11T08:26:39.015276+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:15:19.498532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.041476+0000","last_change":"2024-01-11T08:26:39.041011+0000","last_active":"2024-01-11T08:26:39.041476+0000","last_peered":"2024-01-11T08:26:39.041476+0000","last_clean":"2024-01-11T08:26:39.041476+0000","last_became_active":"2024-01-11T08:26:39.040733+0000","last_became_peered":"2024-01-11T08:26:39.040733+0000","last_unstale":"2024-01-11T08:26:39.041476+0000","last_undegraded":"2024-01-11T08:26:39.041476+0000","last_fullsized":"2024-01-11T08:26:39.041476+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:37:08.612450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-11T08:26:38.025460+0000","last_change":"2024-01-11T08:26:38.023632+0000","last_active":"2024-01-11T08:26:38.025460+0000","last_peered":"2024-01-11T08:26:38.025460+0000","last_clean":"2024-01-11T08:26:38.025460+0000","last_became_active":"2024-01-11T08:26:38.021109+0000","last_became_peered":"2024-01-11T08:26:38.021109+0000","last_unstale":"2024-01-11T08:26:38.025460+0000","last_undegraded":"2024-01-11T08:26:38.025460+0000","last_fullsized":"2024-01-11T08:26:38.025460+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:45:29.080157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":8,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.020206+0000","last_change":"2024-01-11T08:26:39.020206+0000","last_active":"2024-01-11T08:26:39.020206+0000","last_peered":"2024-01-11T08:26:39.020206+0000","last_clean":"2024-01-11T08:26:39.020206+0000","last_became_active":"2024-01-11T08:26:39.019952+0000","last_became_peered":"2024-01-11T08:26:39.019952+0000","last_unstale":"2024-01-11T08:26:39.020206+0000","last_undegraded":"2024-01-11T08:26:39.020206+0000","last_fullsized":"2024-01-11T08:26:39.020206+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:54:35.454673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.013675+0000","last_change":"2024-01-11T08:26:38.026264+0000","last_active":"2024-01-11T08:26:39.013675+0000","last_peered":"2024-01-11T08:26:39.013675+0000","last_clean":"2024-01-11T08:26:39.013675+0000","last_became_active":"2024-01-11T08:26:38.025907+0000","last_became_peered":"2024-01-11T08:26:38.025907+0000","last_unstale":"2024-01-11T08:26:39.013675+0000","last_undegraded":"2024-01-11T08:26:39.013675+0000","last_fullsized":"2024-01-11T08:26:39.013675+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:11:41.070507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.040164+0000","last_change":"2024-01-11T08:26:39.035601+0000","last_active":"2024-01-11T08:26:39.040164+0000","last_peered":"2024-01-11T08:26:39.040164+0000","last_clean":"2024-01-11T08:26:39.040164+0000","last_became_active":"2024-01-11T08:26:39.035305+0000","last_became_peered":"2024-01-11T08:26:39.035305+0000","last_unstale":"2024-01-11T08:26:39.040164+0000","last_undegraded":"2024-01-11T08:26:39.040164+0000","last_fullsized":"2024-01-11T08:26:39.040164+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:37.991780+0000","last_clean_scrub_stamp":"2024-01-11T08:26:37.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:43:43.522488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"100'1","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.040224+0000","last_change":"2024-01-11T08:26:38.025554+0000","last_active":"2024-01-11T08:26:39.040224+0000","last_peered":"2024-01-11T08:26:39.040224+0000","last_clean":"2024-01-11T08:26:39.040224+0000","last_became_active":"2024-01-11T08:26:38.025077+0000","last_became_peered":"2024-01-11T08:26:38.025077+0000","last_unstale":"2024-01-11T08:26:39.040224+0000","last_undegraded":"2024-01-11T08:26:39.040224+0000","last_fullsized":"2024-01-11T08:26:39.040224+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:26:36.983835+0000","last_clean_scrub_stamp":"2024-01-11T08:26:36.983835+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:11:41.070507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"92'201","reported_seq":524,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-01-11T08:26:39.012222+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:26:39.012222+0000","last_peered":"2024-01-11T08:26:39.012222+0000","last_clean":"2024-01-11T08:26:39.012222+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:26:39.012222+0000","last_undegraded":"2024-01-11T08:26:39.012222+0000","last_fullsized":"2024-01-11T08:26:39.012222+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":848,"num_read_kb":1718,"num_write":311,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":848,"num_read_kb":1718,"num_write":311,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":201,"ondisk_log_size":201,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575042,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303180,"kb_used_data":1796,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068660,"statfs":{"total":96636764160,"available":96326307840,"internally_reserved":0,"allocated":1839104,"data_stored":1164805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.69399999999999995,"15min":0.70199999999999996},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.79000000000000004,"5min":0.996,"15min":2.8300000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.69199999999999995,"15min":0.70699999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.372},"max":{"1min":0.91600000000000004,"5min":1.073,"15min":1.073},"last":0.48699999999999999}]},{"osd":1,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.45700000000000002,"5min":0.41899999999999998,"15min":0.35799999999999998},"max":{"1min":0.93400000000000005,"5min":0.99099999999999999,"15min":0.998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.70599999999999996,"15min":0.71999999999999997},"min":{"1min":0.41999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.94999999999999996,"5min":0.95899999999999996,"15min":1.153},"last":0.56399999999999995}]},{"osd":2,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.71999999999999997,"15min":0.72899999999999998},"min":{"1min":0.54200000000000004,"5min":0.436,"15min":0.436},"max":{"1min":0.872,"5min":1.03,"15min":1.034},"last":0.623},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.70499999999999996,"15min":0.71599999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.0049999999999999},"last":0.58499999999999996}]},{"osd":3,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.71999999999999997,"15min":0.74399999999999999},"min":{"1min":0.496,"5min":0.48299999999999998,"15min":0.40300000000000002},"max":{"1min":0.91700000000000004,"5min":1.032,"15min":2.9119999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.71899999999999997,"15min":0.73899999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.95899999999999996,"5min":1.008,"15min":1.008},"last":0.64400000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.42299999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":1.089},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63400000000000001,"15min":0.63200000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.85299999999999998,"5min":0.90700000000000003,"15min":1.1020000000000001},"last":0.69599999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.34599999999999997},"max":{"1min":0.90400000000000003,"5min":0.92200000000000004,"15min":1.0589999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.66600000000000004,"15min":0.67800000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.40400000000000003},"max":{"1min":0.94499999999999995,"5min":1.0089999999999999,"15min":1.032},"last":0.67500000000000004}]},{"osd":6,"last update":"Thu Jan 11 08:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.74099999999999999,"15min":0.75800000000000001},"min":{"1min":0.54800000000000004,"5min":0.52000000000000002,"15min":0.46999999999999997},"max":{"1min":0.97499999999999998,"5min":1.0469999999999999,"15min":1.1359999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.72399999999999998,"15min":0.73899999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42099999999999999},"max":{"1min":0.90800000000000003,"5min":1.028,"15min":1.1299999999999999},"last":0.78600000000000003}]}]},{"osd":6,"up_from":13,"seq":55834575041,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302596,"kb_used_data":1212,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069244,"statfs":{"total":96636764160,"available":96326905856,"internally_reserved":0,"allocated":1241088,"data_stored":568717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.66100000000000003,"15min":0.66500000000000004},"min":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.29999999999999999},"max":{"1min":0.81100000000000005,"5min":0.89300000000000002,"15min":0.94799999999999995},"last":0.628},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.66600000000000004,"15min":0.68400000000000005},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.65800000000000003}]},{"osd":1,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.68200000000000005,"15min":0.68700000000000006},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.35899999999999999},"max":{"1min":0.997,"5min":1.163,"15min":1.163},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68400000000000005,"15min":0.69099999999999995},"min":{"1min":0.49299999999999999,"5min":0.435,"15min":0.39900000000000002},"max":{"1min":0.92600000000000005,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.72899999999999998}]},{"osd":2,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.70599999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.38600000000000001},"max":{"1min":0.90200000000000002,"5min":0.97199999999999998,"15min":1.016},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.70599999999999996,"15min":0.72099999999999997},"min":{"1min":0.48999999999999999,"5min":0.47799999999999998,"15min":0.373},"max":{"1min":0.89500000000000002,"5min":0.96699999999999997,"15min":0.995},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71699999999999997,"15min":0.73299999999999998},"min":{"1min":0.48099999999999998,"5min":0.46600000000000003,"15min":0.42799999999999999},"max":{"1min":0.94799999999999995,"5min":0.95799999999999996,"15min":1.3260000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.73299999999999998,"15min":0.73699999999999999},"min":{"1min":0.52000000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.88800000000000001,"5min":0.95899999999999996,"15min":1.0049999999999999},"last":0.63600000000000001}]},{"osd":4,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.42999999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84199999999999997,"5min":0.874,"15min":1.212},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57699999999999996,"15min":0.60499999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.374},"max":{"1min":0.76700000000000002,"5min":0.96399999999999997,"15min":1.8320000000000001},"last":0.69899999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64200000000000002,"15min":0.65400000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68799999999999994,"15min":0.70499999999999996},"min":{"1min":0.52300000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":1.7929999999999999},"last":0.55900000000000005}]},{"osd":7,"last update":"Thu Jan 11 08:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69099999999999995,"15min":0.71299999999999997},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.41099999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":1.278},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.73099999999999998,"15min":0.749},"min":{"1min":0.46700000000000003,"5min":0.45900000000000002,"15min":0.44700000000000001},"max":{"1min":1.0009999999999999,"5min":1.022,"15min":1.1599999999999999},"last":0.71399999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575043,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302608,"kb_used_data":1224,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069232,"statfs":{"total":96636764160,"available":96326893568,"internally_reserved":0,"allocated":1253376,"data_stored":574561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.309,"5min":0.30599999999999999,"15min":0.26900000000000002},"max":{"1min":0.755,"5min":0.85999999999999999,"15min":0.97699999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56999999999999995,"15min":0.57999999999999996},"min":{"1min":0.42699999999999999,"5min":0.41199999999999998,"15min":0.373},"max":{"1min":0.82299999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.47299999999999998}]},{"osd":2,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48299999999999998,"15min":0.48899999999999999},"min":{"1min":0.33100000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.65800000000000003,"5min":1.113,"15min":1.113},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.375,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71499999999999997,"5min":0.82399999999999995,"15min":1.25},"last":0.35599999999999998}]},{"osd":3,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.60399999999999998,"15min":0.61699999999999999},"min":{"1min":0.41499999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.77600000000000002,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.371,"15min":0.34899999999999998},"max":{"1min":0.88100000000000001,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.74099999999999999}]},{"osd":4,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.75900000000000001,"15min":0.78600000000000003},"min":{"1min":0.504,"5min":0.439,"15min":0.439},"max":{"1min":1.0620000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.80900000000000005,"15min":0.82799999999999996},"min":{"1min":0.65100000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.069,"5min":1.246,"15min":1.3680000000000001},"last":0.85199999999999998}]},{"osd":5,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76900000000000002,"15min":0.79900000000000004},"min":{"1min":0.54300000000000004,"5min":0.502,"15min":0.46999999999999997},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.7849999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.76800000000000002,"15min":0.79400000000000004},"min":{"1min":0.56299999999999994,"5min":0.52700000000000002,"15min":0.50700000000000001},"max":{"1min":1.052,"5min":1.1339999999999999,"15min":1.371},"last":0.752}]},{"osd":6,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.79800000000000004,"15min":0.81899999999999995},"min":{"1min":0.56100000000000005,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.413},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.80500000000000005,"15min":0.82499999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.93899999999999995,"5min":1.155,"15min":1.228},"last":0.61799999999999999}]},{"osd":7,"last update":"Thu Jan 11 08:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.81999999999999995,"15min":0.84599999999999997},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.52700000000000002},"max":{"1min":1.0149999999999999,"5min":1.129,"15min":1.226},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.82099999999999995,"15min":0.83799999999999997},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.51400000000000001},"max":{"1min":1.099,"5min":1.1339999999999999,"15min":1.321},"last":0.83399999999999996}]}]},{"osd":0,"up_from":13,"seq":55834575042,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303168,"kb_used_data":1784,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068672,"statfs":{"total":96636764160,"available":96326320128,"internally_reserved":0,"allocated":1826816,"data_stored":1158453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.5,"15min":0.5},"min":{"1min":0.317,"5min":0.29999999999999999,"15min":0.24299999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.92700000000000005},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56599999999999995,"15min":0.57399999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.35299999999999998},"max":{"1min":0.89600000000000002,"5min":0.90500000000000003,"15min":0.92100000000000004},"last":0.60999999999999999}]},{"osd":2,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.29099999999999998},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54400000000000004,"15min":0.55300000000000005},"min":{"1min":0.41699999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.67800000000000005}]},{"osd":3,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.33600000000000002},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.629,"15min":0.627},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.39600000000000002},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.64600000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.79100000000000004,"15min":0.78900000000000003},"min":{"1min":0.502,"5min":0.45700000000000002,"15min":0.437},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.80200000000000005,"15min":0.79100000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.46899999999999997},"max":{"1min":1.095,"5min":1.095,"15min":1.2350000000000001},"last":0.69199999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.80200000000000005,"15min":0.80800000000000005},"min":{"1min":0.57699999999999996,"5min":0.52500000000000002,"15min":0.505},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.82199999999999995,"15min":0.81699999999999995},"min":{"1min":0.58799999999999997,"5min":0.49099999999999999,"15min":0.47799999999999998},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.77200000000000002}]},{"osd":6,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81100000000000005,"15min":0.80200000000000005},"min":{"1min":0.60199999999999998,"5min":0.50900000000000001,"15min":0.497},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.2689999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82699999999999996,"15min":0.83399999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.44900000000000001},"max":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.748}]},{"osd":7,"last update":"Thu Jan 11 08:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.84199999999999997,"15min":0.84099999999999997},"min":{"1min":0.61399999999999999,"5min":0.57999999999999996,"15min":0.51400000000000001},"max":{"1min":1.371,"5min":1.371,"15min":1.446},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.85099999999999998,"15min":0.84299999999999997},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.57699999999999996},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.298},"last":0.66500000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575041,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306684,"kb_used_data":1204,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065156,"statfs":{"total":96636764160,"available":96322719744,"internally_reserved":0,"allocated":1232896,"data_stored":568605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47599999999999998,"15min":0.47199999999999998},"min":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.26000000000000001},"max":{"1min":0.67900000000000005,"5min":0.73499999999999999,"15min":0.86099999999999999},"last":0.443},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51800000000000002,"15min":0.54500000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.28899999999999998},"max":{"1min":0.67100000000000004,"5min":0.81699999999999995,"15min":1.4119999999999999},"last":0.48199999999999998}]},{"osd":1,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54200000000000004,"15min":0.55000000000000004},"min":{"1min":0.40400000000000003,"5min":0.36399999999999999,"15min":0.30399999999999999},"max":{"1min":0.73199999999999998,"5min":0.80300000000000005,"15min":0.90000000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.58599999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38100000000000001},"max":{"1min":0.79100000000000004,"5min":0.92900000000000005,"15min":1.5640000000000001},"last":0.54800000000000004}]},{"osd":3,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.38100000000000001,"5min":0.36199999999999999,"15min":0.32900000000000001},"max":{"1min":0.72399999999999998,"5min":0.82499999999999996,"15min":1.1220000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.61199999999999999,"15min":0.63800000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.41799999999999998},"max":{"1min":0.77500000000000002,"5min":0.95299999999999996,"15min":1.22},"last":0.57999999999999996}]},{"osd":4,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.77100000000000002,"15min":0.78900000000000003},"min":{"1min":0.58699999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.026,"5min":1.0580000000000001,"15min":1.488},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.76100000000000001,"15min":0.78100000000000003},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89200000000000002,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.69599999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.81499999999999995,"15min":0.83999999999999997},"min":{"1min":0.63500000000000001,"5min":0.622,"15min":0.51600000000000001},"max":{"1min":1.105,"5min":1.214,"15min":2.9209999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.83099999999999996,"15min":0.84999999999999998},"min":{"1min":0.63500000000000001,"5min":0.55500000000000005,"15min":0.55400000000000005},"max":{"1min":1.036,"5min":1.2,"15min":1.845},"last":0.874}]},{"osd":6,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.79800000000000004,"15min":0.80200000000000005},"min":{"1min":0.56299999999999994,"5min":0.54200000000000004,"15min":0.51700000000000002},"max":{"1min":0.91400000000000003,"5min":1.1919999999999999,"15min":1.3560000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.747,"5min":0.80700000000000005,"15min":0.83299999999999996},"min":{"1min":0.55800000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.012,"5min":1.123,"15min":2.968},"last":0.84499999999999997}]},{"osd":7,"last update":"Thu Jan 11 08:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.85299999999999998,"15min":0.872},"min":{"1min":0.62,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.056,"5min":1.2629999999999999,"15min":1.6359999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.83199999999999996,"15min":0.86199999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.0720000000000001,"5min":1.2270000000000001,"15min":1.391},"last":0.82299999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575041,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306704,"kb_used_data":1224,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065136,"statfs":{"total":96636764160,"available":96322699264,"internally_reserved":0,"allocated":1253376,"data_stored":575903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.499,"15min":0.48499999999999999},"min":{"1min":0.34999999999999998,"5min":0.29499999999999998,"15min":0.27600000000000002},"max":{"1min":0.80300000000000005,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.436},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52500000000000002,"15min":0.54600000000000004},"min":{"1min":0.39300000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.873,"5min":0.873,"15min":1.0109999999999999},"last":0.41899999999999998}]},{"osd":1,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.47799999999999998,"15min":0.49099999999999999},"min":{"1min":0.33900000000000002,"5min":0.30299999999999999,"15min":0.25900000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.39600000000000002,"5min":0.36299999999999999,"15min":0.32300000000000001},"max":{"1min":0.92500000000000004,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57399999999999995,"15min":0.58399999999999996},"min":{"1min":0.42799999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81299999999999994,"5min":0.90900000000000003,"15min":0.97599999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.61699999999999999,"15min":0.628},"min":{"1min":0.45900000000000002,"5min":0.44700000000000001,"15min":0.32000000000000001},"max":{"1min":0.74299999999999999,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.50700000000000001}]},{"osd":4,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.71899999999999997,"15min":0.74299999999999999},"min":{"1min":0.54100000000000004,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.012,"5min":1.038,"15min":1.351},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77400000000000002,"15min":0.78200000000000003},"min":{"1min":0.60199999999999998,"5min":0.503,"15min":0.503},"max":{"1min":1.1459999999999999,"5min":1.1819999999999999,"15min":1.198},"last":0.57299999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76400000000000001,"15min":0.78900000000000003},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.51900000000000002},"max":{"1min":1.0649999999999999,"5min":1.139,"15min":1.3779999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.79400000000000004,"15min":0.80900000000000005},"min":{"1min":0.623,"5min":0.49199999999999999,"15min":0.44900000000000001},"max":{"1min":1.24,"5min":1.2809999999999999,"15min":1.4239999999999999},"last":0.68200000000000005}]},{"osd":6,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.81399999999999995,"15min":0.83499999999999996},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.52600000000000002},"max":{"1min":1.0820000000000001,"5min":1.335,"15min":1.927},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.78700000000000003,"15min":0.80600000000000005},"min":{"1min":0.57999999999999996,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.93999999999999995,"5min":1.2230000000000001,"15min":3.5579999999999998},"last":0.73699999999999999}]},{"osd":7,"last update":"Thu Jan 11 08:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81000000000000005,"15min":0.84299999999999997},"min":{"1min":0.59199999999999997,"5min":0.52800000000000002,"15min":0.46600000000000003},"max":{"1min":1.0660000000000001,"5min":1.1579999999999999,"15min":3.5089999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.84599999999999997,"15min":0.86099999999999999},"min":{"1min":0.60599999999999998,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.21},"last":0.64900000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310788,"kb_used_data":1212,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94061052,"statfs":{"total":96636764160,"available":96318517248,"internally_reserved":0,"allocated":1241088,"data_stored":574367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.72699999999999998,"15min":0.73799999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.28100000000000003},"max":{"1min":0.874,"5min":0.95999999999999996,"15min":1.524},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72799999999999998,"15min":0.74299999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.38600000000000001},"max":{"1min":0.91500000000000004,"5min":1.0049999999999999,"15min":1.177},"last":0.77400000000000002}]},{"osd":1,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.69899999999999995,"15min":0.70399999999999996},"min":{"1min":0.47399999999999998,"5min":0.46200000000000002,"15min":0.33300000000000002},"max":{"1min":0.89400000000000002,"5min":1.0069999999999999,"15min":1.1499999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.69199999999999995,"15min":0.72299999999999998},"min":{"1min":0.43099999999999999,"5min":0.379,"15min":0.36299999999999999},"max":{"1min":0.94899999999999995,"5min":0.99199999999999999,"15min":2.4740000000000002},"last":0.44}]},{"osd":2,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.71699999999999997},"min":{"1min":0.46600000000000003,"5min":0.46000000000000002,"15min":0.38900000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.72699999999999998,"15min":0.72999999999999998},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.89400000000000002,"5min":0.97699999999999998,"15min":1.212},"last":0.50800000000000001}]},{"osd":3,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.68700000000000006,"15min":0.70599999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.38100000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":1.2649999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.73999999999999999,"15min":0.754},"min":{"1min":0.502,"5min":0.502,"15min":0.47499999999999998},"max":{"1min":0.84399999999999997,"5min":1.026,"15min":1.069},"last":0.621}]},{"osd":5,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60399999999999998,"15min":0.61799999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.85399999999999998,"5min":0.873,"15min":1.0700000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.622,"15min":0.63300000000000001},"min":{"1min":0.45000000000000001,"5min":0.38200000000000001,"15min":0.32600000000000001},"max":{"1min":0.94299999999999995,"5min":0.97799999999999998,"15min":0.98199999999999998},"last":0.69999999999999996}]},{"osd":6,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.65700000000000003,"15min":0.65600000000000003},"min":{"1min":0.40500000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0680000000000001,"5min":1.133,"15min":1.133},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.69199999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95099999999999996,"5min":1.0269999999999999,"15min":1.119},"last":0.66000000000000003}]},{"osd":7,"last update":"Thu Jan 11 08:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.69299999999999995},"min":{"1min":0.51500000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.99399999999999999,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.73999999999999999,"15min":0.74299999999999999},"min":{"1min":0.56999999999999995,"5min":0.46200000000000002,"15min":0.42299999999999999},"max":{"1min":0.91000000000000003,"5min":1.2350000000000001,"15min":1.4279999999999999},"last":0.59299999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575042,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302600,"kb_used_data":1216,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069240,"statfs":{"total":96636764160,"available":96326901760,"internally_reserved":0,"allocated":1245184,"data_stored":569609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.69199999999999995,"15min":0.70799999999999996},"min":{"1min":0.39300000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85499999999999998,"5min":1.0249999999999999,"15min":1.032},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.69599999999999995,"15min":0.70999999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.879,"5min":0.96199999999999997,"15min":1.696},"last":0.70299999999999996}]},{"osd":1,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66400000000000003,"15min":0.69899999999999995},"min":{"1min":0.42799999999999999,"5min":0.39400000000000002,"15min":0.36399999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":1.4359999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.70799999999999996,"15min":0.73199999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.05,"5min":1.05,"15min":4.4969999999999999},"last":0.79900000000000004}]},{"osd":2,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70699999999999996,"15min":0.71099999999999997},"min":{"1min":0.55100000000000005,"5min":0.435,"15min":0.435},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.238},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69499999999999995,"15min":0.746},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.41599999999999998},"max":{"1min":0.94199999999999995,"5min":1.0049999999999999,"15min":8.0429999999999993},"last":0.73199999999999998}]},{"osd":3,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.70199999999999996,"15min":0.755},"min":{"1min":0.47599999999999998,"5min":0.46200000000000002,"15min":0.442},"max":{"1min":1.208,"5min":1.208,"15min":8.0839999999999996},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.748,"15min":0.747},"min":{"1min":0.56399999999999995,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.80800000000000005}]},{"osd":4,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.38300000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":1.113},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64500000000000002,"15min":0.64000000000000001},"min":{"1min":0.41499999999999998,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.755}]},{"osd":6,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.45700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.034},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67700000000000005,"15min":0.68600000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.085,"5min":1.085,"15min":1.143},"last":0.79300000000000004}]},{"osd":7,"last update":"Thu Jan 11 08:25:43 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.73399999999999999,"15min":0.73699999999999999},"min":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.437},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77300000000000002,"15min":0.77000000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.93700000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:26:43.924 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-11T08:26:43.924 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-11T08:26:44.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.166+0000 7f6b4bfff700 1 Processor -- start 2024-01-11T08:26:44.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.167+0000 7f6b4bfff700 1 -- start start 2024-01-11T08:26:44.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.168+0000 7f6b4bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6b440c8860 0x7f6b44158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:44.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.168+0000 7f6b4bfff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b440cabe0 con 0x7f6b440b19d0 2024-01-11T08:26:44.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.168+0000 7f6b4bfff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b440ae960 con 0x7f6b44158600 2024-01-11T08:26:44.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.168+0000 7f6b4bfff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b440ca590 con 0x7f6b440c8860 2024-01-11T08:26:44.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.168+0000 7f6b49d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6b440c8860 0x7f6b44158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:44.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.168+0000 7f6b49d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6b440c8860 0x7f6b44158ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35440/0 (socket says 172.21.15.27:35440) 2024-01-11T08:26:44.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.168+0000 7f6b49d9b700 1 -- 172.21.15.27:0/3012591003 learned_addr learned my addr 172.21.15.27:0/3012591003 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:44.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.169+0000 7f6b49d9b700 1 -- 172.21.15.27:0/3012591003 >> v1:172.21.15.77:6789/0 conn(0x7f6b44158600 legacy=0x7f6b440b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:44.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.169+0000 7f6b49d9b700 1 -- 172.21.15.27:0/3012591003 >> v1:172.21.15.27:6789/0 conn(0x7f6b440b19d0 legacy=0x7f6b440cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:44.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.169+0000 7f6b49d9b700 1 -- 172.21.15.27:0/3012591003 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b440ca870 con 0x7f6b440c8860 2024-01-11T08:26:44.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.169+0000 7f6b49d9b700 1 --2- 172.21.15.27:0/3012591003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b440c8860 0x7f6b44158ca0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f6b400099b0 tx=0x7f6b40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f875e7de27c2c6a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:44.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.169+0000 7f6b48d99700 1 -- 172.21.15.27:0/3012591003 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4000b860 con 0x7f6b440c8860 2024-01-11T08:26:44.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.169+0000 7f6b48d99700 1 -- 172.21.15.27:0/3012591003 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4000b9c0 con 0x7f6b440c8860 2024-01-11T08:26:44.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.169+0000 7f6b48d99700 1 -- 172.21.15.27:0/3012591003 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4000bd60 con 0x7f6b440c8860 2024-01-11T08:26:44.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.170+0000 7f6b4bfff700 1 -- 172.21.15.27:0/3012591003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b440c8860 msgr2=0x7f6b44158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:44.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.170+0000 7f6b4bfff700 1 --2- 172.21.15.27:0/3012591003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b440c8860 0x7f6b44158ca0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f6b400099b0 tx=0x7f6b40009250 comp rx=0 tx=0).stop 2024-01-11T08:26:44.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.170+0000 7f6b4bfff700 1 -- 172.21.15.27:0/3012591003 shutdown_connections 2024-01-11T08:26:44.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.170+0000 7f6b4bfff700 1 --2- 172.21.15.27:0/3012591003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b440c8860 0x7f6b44158ca0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:44.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.170+0000 7f6b4bfff700 1 -- 172.21.15.27:0/3012591003 >> 172.21.15.27:0/3012591003 conn(0x7f6b440680a0 msgr2=0x7f6b44069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:44.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.170+0000 7f6b4bfff700 1 -- 172.21.15.27:0/3012591003 shutdown_connections 2024-01-11T08:26:44.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.170+0000 7f6b4bfff700 1 -- 172.21.15.27:0/3012591003 wait complete. 2024-01-11T08:26:44.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.172+0000 7f6b4bfff700 1 Processor -- start 2024-01-11T08:26:44.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.172+0000 7f6b4bfff700 1 -- start start 2024-01-11T08:26:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.173+0000 7f6b4bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b440b19d0 0x7f6b4414d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.173+0000 7f6b4bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b440c8860 0x7f6b440ab700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.173+0000 7f6b4bfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6b44158600 0x7f6b4415a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.173+0000 7f6b4bfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4414d8c0 con 0x7f6b440c8860 2024-01-11T08:26:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.173+0000 7f6b4bfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b440cc0e0 con 0x7f6b440b19d0 2024-01-11T08:26:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.173+0000 7f6b4bfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b440b0cb0 con 0x7f6b44158600 2024-01-11T08:26:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.173+0000 7f6b4959a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b440c8860 0x7f6b440ab700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.173+0000 7f6b4959a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b440c8860 0x7f6b440ab700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35446/0 (socket says 172.21.15.27:35446) 2024-01-11T08:26:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.173+0000 7f6b4959a700 1 -- 172.21.15.27:0/278299877 learned_addr learned my addr 172.21.15.27:0/278299877 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.173+0000 7f6b49d9b700 1 --2- 172.21.15.27:0/278299877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b440b19d0 0x7f6b4414d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.173+0000 7f6b4a59c700 1 --2- 172.21.15.27:0/278299877 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6b44158600 0x7f6b4415a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.173+0000 7f6b4959a700 1 -- 172.21.15.27:0/278299877 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6b44158600 msgr2=0x7f6b4415a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.173+0000 7f6b4959a700 1 --2- 172.21.15.27:0/278299877 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6b44158600 0x7f6b4415a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.173+0000 7f6b4959a700 1 -- 172.21.15.27:0/278299877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b440b19d0 msgr2=0x7f6b4414d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.173+0000 7f6b4959a700 1 --2- 172.21.15.27:0/278299877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b440b19d0 0x7f6b4414d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.173+0000 7f6b4959a700 1 -- 172.21.15.27:0/278299877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b400094f0 con 0x7f6b440c8860 2024-01-11T08:26:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.174+0000 7f6b4a59c700 1 --2- 172.21.15.27:0/278299877 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6b44158600 0x7f6b4415a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:26:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.174+0000 7f6b4959a700 1 --2- 172.21.15.27:0/278299877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b440c8860 0x7f6b440ab700 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f6b3400c510 tx=0x7f6b3400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.174+0000 7f6b39ffb700 1 -- 172.21.15.27:0/278299877 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b34011590 con 0x7f6b440c8860 2024-01-11T08:26:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.174+0000 7f6b39ffb700 1 -- 172.21.15.27:0/278299877 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b34011710 con 0x7f6b440c8860 2024-01-11T08:26:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.174+0000 7f6b39ffb700 1 -- 172.21.15.27:0/278299877 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b34011ad0 con 0x7f6b440c8860 2024-01-11T08:26:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.174+0000 7f6b4bfff700 1 -- 172.21.15.27:0/278299877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b440cc350 con 0x7f6b440c8860 2024-01-11T08:26:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.174+0000 7f6b4bfff700 1 -- 172.21.15.27:0/278299877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b440b0f90 con 0x7f6b440c8860 2024-01-11T08:26:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.175+0000 7f6b4bfff700 1 -- 172.21.15.27:0/278299877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b4404c3c0 con 0x7f6b440c8860 2024-01-11T08:26:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.176+0000 7f6b39ffb700 1 -- 172.21.15.27:0/278299877 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6b3401b3b0 con 0x7f6b440c8860 2024-01-11T08:26:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.177+0000 7f6b39ffb700 1 --2- 172.21.15.27:0/278299877 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6b28066f30 0x7f6b280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.177+0000 7f6b39ffb700 1 -- 172.21.15.27:0/278299877 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f6b34083a70 con 0x7f6b440c8860 2024-01-11T08:26:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.177+0000 7f6b49d9b700 1 --2- 172.21.15.27:0/278299877 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6b28066f30 0x7f6b280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.178+0000 7f6b49d9b700 1 --2- 172.21.15.27:0/278299877 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6b28066f30 0x7f6b280693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6b400099b0 tx=0x7f6b40017000 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.181+0000 7f6b39ffb700 1 -- 172.21.15.27:0/278299877 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6b34050c20 con 0x7f6b440c8860 2024-01-11T08:26:44.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.462+0000 7f6b4bfff700 1 -- 172.21.15.27:0/278299877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f6b44158a30 con 0x7f6b440c8860 2024-01-11T08:26:44.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.467+0000 7f6b39ffb700 1 -- 172.21.15.27:0/278299877 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+0 (secure 0 0 0) 0x7f6b34017070 con 0x7f6b440c8860 2024-01-11T08:26:44.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.469+0000 7f6b4bfff700 1 -- 172.21.15.27:0/278299877 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6b28066f30 msgr2=0x7f6b280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:44.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.469+0000 7f6b4bfff700 1 --2- 172.21.15.27:0/278299877 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6b28066f30 0x7f6b280693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6b400099b0 tx=0x7f6b40017000 comp rx=0 tx=0).stop 2024-01-11T08:26:44.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.469+0000 7f6b4bfff700 1 -- 172.21.15.27:0/278299877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b440c8860 msgr2=0x7f6b440ab700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:44.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.469+0000 7f6b4bfff700 1 --2- 172.21.15.27:0/278299877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b440c8860 0x7f6b440ab700 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f6b3400c510 tx=0x7f6b3400c820 comp rx=0 tx=0).stop 2024-01-11T08:26:44.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.469+0000 7f6b4bfff700 1 -- 172.21.15.27:0/278299877 shutdown_connections 2024-01-11T08:26:44.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.469+0000 7f6b4bfff700 1 --2- 172.21.15.27:0/278299877 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6b28066f30 0x7f6b280693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:44.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.469+0000 7f6b4bfff700 1 --2- 172.21.15.27:0/278299877 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6b44158600 0x7f6b4415a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:44.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.469+0000 7f6b4bfff700 1 --2- 172.21.15.27:0/278299877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b440b19d0 0x7f6b4414d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:44.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.469+0000 7f6b4bfff700 1 --2- 172.21.15.27:0/278299877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b440c8860 0x7f6b440ab700 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:44.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.469+0000 7f6b4bfff700 1 -- 172.21.15.27:0/278299877 >> 172.21.15.27:0/278299877 conn(0x7f6b440680a0 msgr2=0x7f6b4406ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:44.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.469+0000 7f6b4bfff700 1 -- 172.21.15.27:0/278299877 shutdown_connections 2024-01-11T08:26:44.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.469+0000 7f6b4bfff700 1 -- 172.21.15.27:0/278299877 wait complete. 2024-01-11T08:26:44.470 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-11T08:26:44.487 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:26:44.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.687+0000 7f04489ff700 1 Processor -- start 2024-01-11T08:26:44.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.687+0000 7f04489ff700 1 -- start start 2024-01-11T08:26:44.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.688+0000 7f04489ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f04441575e0 0x7f0444158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:44.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.688+0000 7f04489ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f044406b610 0x7f04440b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:44.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.688+0000 7f04489ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f04440b1800 0x7f04440ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:44.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.688+0000 7f04489ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04440cb5e0 con 0x7f04441575e0 2024-01-11T08:26:44.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.688+0000 7f04489ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04440b1c80 con 0x7f04440b1800 2024-01-11T08:26:44.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.688+0000 7f04489ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0444158b60 con 0x7f044406b610 2024-01-11T08:26:44.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.689+0000 7f044259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f04441575e0 0x7f0444158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:44.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.689+0000 7f044259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f04441575e0 0x7f0444158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35456/0 (socket says 172.21.15.27:35456) 2024-01-11T08:26:44.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.689+0000 7f044259c700 1 -- 172.21.15.27:0/899996436 learned_addr learned my addr 172.21.15.27:0/899996436 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:44.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.689+0000 7f0442d9d700 1 --2- 172.21.15.27:0/899996436 >> v2:172.21.15.77:3300/0 conn(0x7f04440b1800 0x7f04440ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:44.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.689+0000 7f0441d9b700 1 --2- 172.21.15.27:0/899996436 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f044406b610 0x7f04440b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:44.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.689+0000 7f044259c700 1 -- 172.21.15.27:0/899996436 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f044406b610 msgr2=0x7f04440b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:44.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.689+0000 7f044259c700 1 --2- 172.21.15.27:0/899996436 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f044406b610 0x7f04440b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:44.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.689+0000 7f044259c700 1 -- 172.21.15.27:0/899996436 >> v2:172.21.15.77:3300/0 conn(0x7f04440b1800 msgr2=0x7f04440ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:44.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.689+0000 7f044259c700 1 --2- 172.21.15.27:0/899996436 >> v2:172.21.15.77:3300/0 conn(0x7f04440b1800 0x7f04440ca1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:44.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.689+0000 7f044259c700 1 -- 172.21.15.27:0/899996436 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04441599b0 con 0x7f04441575e0 2024-01-11T08:26:44.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.689+0000 7f0442d9d700 1 --2- 172.21.15.27:0/899996436 >> v2:172.21.15.77:3300/0 conn(0x7f04440b1800 0x7f04440ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:26:44.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.689+0000 7f044259c700 1 --2- 172.21.15.27:0/899996436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04441575e0 0x7f0444158620 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f044406d0a0 tx=0x7f0434009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7536c35be7ce4d12 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:44.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.690+0000 7f044159a700 1 -- 172.21.15.27:0/899996436 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043400b860 con 0x7f04441575e0 2024-01-11T08:26:44.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.690+0000 7f044159a700 1 -- 172.21.15.27:0/899996436 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043400b9c0 con 0x7f04441575e0 2024-01-11T08:26:44.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.690+0000 7f044159a700 1 -- 172.21.15.27:0/899996436 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043400bd60 con 0x7f04441575e0 2024-01-11T08:26:44.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.690+0000 7f04489ff700 1 -- 172.21.15.27:0/899996436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04441575e0 msgr2=0x7f0444158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:44.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.690+0000 7f04489ff700 1 --2- 172.21.15.27:0/899996436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04441575e0 0x7f0444158620 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f044406d0a0 tx=0x7f0434009250 comp rx=0 tx=0).stop 2024-01-11T08:26:44.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.690+0000 7f04489ff700 1 -- 172.21.15.27:0/899996436 shutdown_connections 2024-01-11T08:26:44.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.690+0000 7f04489ff700 1 --2- 172.21.15.27:0/899996436 >> v2:172.21.15.77:3300/0 conn(0x7f04440b1800 0x7f04440ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:44.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.690+0000 7f04489ff700 1 --2- 172.21.15.27:0/899996436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04441575e0 0x7f0444158620 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:44.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.690+0000 7f04489ff700 1 --2- 172.21.15.27:0/899996436 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f044406b610 0x7f04440b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:44.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.690+0000 7f04489ff700 1 -- 172.21.15.27:0/899996436 >> 172.21.15.27:0/899996436 conn(0x7f04440680e0 msgr2=0x7f0444069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:44.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.690+0000 7f04489ff700 1 -- 172.21.15.27:0/899996436 shutdown_connections 2024-01-11T08:26:44.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.690+0000 7f04489ff700 1 -- 172.21.15.27:0/899996436 wait complete. 2024-01-11T08:26:44.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.692+0000 7f04489ff700 1 Processor -- start 2024-01-11T08:26:44.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.692+0000 7f04489ff700 1 -- start start 2024-01-11T08:26:44.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.693+0000 7f04489ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f044406b610 0x7f044414fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:44.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.693+0000 7f04489ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04440b1800 0x7f044414b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:44.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.693+0000 7f04489ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04441575e0 0x7f04441506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:44.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.693+0000 7f04489ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044414f8f0 con 0x7f044406b610 2024-01-11T08:26:44.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.693+0000 7f04489ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044414b670 con 0x7f04440b1800 2024-01-11T08:26:44.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.693+0000 7f04489ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044414efc0 con 0x7f04441575e0 2024-01-11T08:26:44.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.693+0000 7f044259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f044406b610 0x7f044414fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:44.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.693+0000 7f044259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f044406b610 0x7f044414fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35464/0 (socket says 172.21.15.27:35464) 2024-01-11T08:26:44.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.693+0000 7f044259c700 1 -- 172.21.15.27:0/1570552802 learned_addr learned my addr 172.21.15.27:0/1570552802 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:44.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.693+0000 7f0442d9d700 1 --2- 172.21.15.27:0/1570552802 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04441575e0 0x7f04441506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:44.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.693+0000 7f0441d9b700 1 --2- 172.21.15.27:0/1570552802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04440b1800 0x7f044414b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:44.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.694+0000 7f044259c700 1 -- 172.21.15.27:0/1570552802 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04441575e0 msgr2=0x7f04441506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:44.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.694+0000 7f044259c700 1 --2- 172.21.15.27:0/1570552802 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04441575e0 0x7f04441506b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:44.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.694+0000 7f044259c700 1 -- 172.21.15.27:0/1570552802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04440b1800 msgr2=0x7f044414b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:44.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.694+0000 7f044259c700 1 --2- 172.21.15.27:0/1570552802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04440b1800 0x7f044414b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:44.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.694+0000 7f044259c700 1 -- 172.21.15.27:0/1570552802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04340094f0 con 0x7f044406b610 2024-01-11T08:26:44.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.694+0000 7f044259c700 1 --2- 172.21.15.27:0/1570552802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f044406b610 0x7f044414fd80 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f0434009290 tx=0x7f0434017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:44.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.694+0000 7f04327fc700 1 -- 172.21.15.27:0/1570552802 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0434026480 con 0x7f044406b610 2024-01-11T08:26:44.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.694+0000 7f04327fc700 1 -- 172.21.15.27:0/1570552802 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043401f070 con 0x7f044406b610 2024-01-11T08:26:44.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.694+0000 7f04489ff700 1 -- 172.21.15.27:0/1570552802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f044414f2a0 con 0x7f044406b610 2024-01-11T08:26:44.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.694+0000 7f04327fc700 1 -- 172.21.15.27:0/1570552802 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0434026830 con 0x7f044406b610 2024-01-11T08:26:44.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.694+0000 7f04489ff700 1 -- 172.21.15.27:0/1570552802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04441680d0 con 0x7f044406b610 2024-01-11T08:26:44.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.695+0000 7f04489ff700 1 -- 172.21.15.27:0/1570552802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04440718d0 con 0x7f044406b610 2024-01-11T08:26:44.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.696+0000 7f04327fc700 1 -- 172.21.15.27:0/1570552802 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0434026ba0 con 0x7f044406b610 2024-01-11T08:26:44.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.698+0000 7f04327fc700 1 --2- 172.21.15.27:0/1570552802 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0424066f30 0x7f04240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:44.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.698+0000 7f04327fc700 1 -- 172.21.15.27:0/1570552802 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f0434021070 con 0x7f044406b610 2024-01-11T08:26:44.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.699+0000 7f0441d9b700 1 --2- 172.21.15.27:0/1570552802 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0424066f30 0x7f04240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:44.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.700+0000 7f0441d9b700 1 --2- 172.21.15.27:0/1570552802 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0424066f30 0x7f04240693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f044414edd0 tx=0x7f0438008040 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:44.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.702+0000 7f04327fc700 1 -- 172.21.15.27:0/1570552802 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0434053600 con 0x7f044406b610 2024-01-11T08:26:44.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.952+0000 7f04489ff700 1 -- 172.21.15.27:0/1570552802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f044414fbd0 con 0x7f044406b610 2024-01-11T08:26:44.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.952+0000 7f04327fc700 1 -- 172.21.15.27:0/1570552802 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4593 (secure 0 0 0) 0x7f0434056c20 con 0x7f044406b610 2024-01-11T08:26:44.952 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:44.953 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":50,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/921942372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":921942372},{"type":"v1","addr":"172.21.15.27:6834","nonce":921942372}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7080,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/3844286168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3844286168},{"type":"v1","addr":"172.21.15.77:6834","nonce":3844286168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7083,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3222788452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":3222788452},{"type":"v1","addr":"172.21.15.77:6837","nonce":3222788452}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:26:38.514309+0000","modified":"2024-01-11T08:26:42.031140+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7089},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7089":{"gid":7089,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/751423743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":751423743},{"type":"v1","addr":"172.21.15.27:6835","nonce":751423743}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2024-01-11T08:26:44.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.954+0000 7f04489ff700 1 -- 172.21.15.27:0/1570552802 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0424066f30 msgr2=0x7f04240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:44.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.954+0000 7f04489ff700 1 --2- 172.21.15.27:0/1570552802 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0424066f30 0x7f04240693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f044414edd0 tx=0x7f0438008040 comp rx=0 tx=0).stop 2024-01-11T08:26:44.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.954+0000 7f04489ff700 1 -- 172.21.15.27:0/1570552802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f044406b610 msgr2=0x7f044414fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:44.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.954+0000 7f04489ff700 1 --2- 172.21.15.27:0/1570552802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f044406b610 0x7f044414fd80 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f0434009290 tx=0x7f0434017000 comp rx=0 tx=0).stop 2024-01-11T08:26:44.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.955+0000 7f04489ff700 1 -- 172.21.15.27:0/1570552802 shutdown_connections 2024-01-11T08:26:44.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.955+0000 7f04489ff700 1 --2- 172.21.15.27:0/1570552802 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0424066f30 0x7f04240693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:44.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.955+0000 7f04489ff700 1 --2- 172.21.15.27:0/1570552802 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04441575e0 0x7f04441506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:44.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.955+0000 7f04489ff700 1 --2- 172.21.15.27:0/1570552802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f044406b610 0x7f044414fd80 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:44.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.955+0000 7f04489ff700 1 --2- 172.21.15.27:0/1570552802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04440b1800 0x7f044414b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:44.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.955+0000 7f04489ff700 1 -- 172.21.15.27:0/1570552802 >> 172.21.15.27:0/1570552802 conn(0x7f04440680e0 msgr2=0x7f04440ca8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:44.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.955+0000 7f04489ff700 1 -- 172.21.15.27:0/1570552802 shutdown_connections 2024-01-11T08:26:44.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:44.955+0000 7f04489ff700 1 -- 172.21.15.27:0/1570552802 wait complete. 2024-01-11T08:26:44.956 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 50 2024-01-11T08:26:45.375 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-11T08:26:38.514309+0000', 'modified': '2024-01-11T08:26:42.031140+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7089}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7089': {'gid': 7089, 'name': 'c', 'rank': 0, 'incarnation': 47, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6835/751423743', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 751423743}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 751423743}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-11T08:26:45.376 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-11T08:26:45.376 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-11T08:26:45.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.578+0000 7f8e1bdff700 1 Processor -- start 2024-01-11T08:26:45.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.578+0000 7f8e1bdff700 1 -- start start 2024-01-11T08:26:45.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.579+0000 7f8e1bdff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8e14156ab0 0x7f8e140ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.579+0000 7f8e1bdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8e140b0550 0x7f8e14160070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.580+0000 7f8e1bdff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e140adc80 con 0x7f8e140cc9b0 2024-01-11T08:26:45.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.580+0000 7f8e1bdff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e140adf60 con 0x7f8e14156ab0 2024-01-11T08:26:45.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.580+0000 7f8e1bdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e140ae240 con 0x7f8e140b0550 2024-01-11T08:26:45.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.580+0000 7f8e19b9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8e14156ab0 0x7f8e140ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.580+0000 7f8e19b9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8e14156ab0 0x7f8e140ad650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35466/0 (socket says 172.21.15.27:35466) 2024-01-11T08:26:45.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.580+0000 7f8e19b9b700 1 -- 172.21.15.27:0/3537868951 learned_addr learned my addr 172.21.15.27:0/3537868951 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:45.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.580+0000 7f8e1a39c700 1 --2- 172.21.15.27:0/3537868951 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8e140b0550 0x7f8e14160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.580+0000 7f8e19b9b700 1 -- 172.21.15.27:0/3537868951 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8e140b0550 msgr2=0x7f8e14160070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:45.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.580+0000 7f8e19b9b700 1 --2- 172.21.15.27:0/3537868951 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8e140b0550 0x7f8e14160070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.580+0000 7f8e19b9b700 1 -- 172.21.15.27:0/3537868951 >> v1:172.21.15.77:6789/0 conn(0x7f8e140cc9b0 legacy=0x7f8e14156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:45.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.580+0000 7f8e19b9b700 1 -- 172.21.15.27:0/3537868951 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e140ae520 con 0x7f8e14156ab0 2024-01-11T08:26:45.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.581+0000 7f8e19b9b700 1 --2- 172.21.15.27:0/3537868951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e14156ab0 0x7f8e140ad650 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f8e100099b0 tx=0x7f8e10009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca5c281ea72e9460 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:45.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.581+0000 7f8e18b99700 1 -- 172.21.15.27:0/3537868951 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1000b860 con 0x7f8e14156ab0 2024-01-11T08:26:45.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.581+0000 7f8e18b99700 1 -- 172.21.15.27:0/3537868951 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1000b9c0 con 0x7f8e14156ab0 2024-01-11T08:26:45.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.581+0000 7f8e18b99700 1 -- 172.21.15.27:0/3537868951 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1000bd60 con 0x7f8e14156ab0 2024-01-11T08:26:45.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.581+0000 7f8e1bdff700 1 -- 172.21.15.27:0/3537868951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e14156ab0 msgr2=0x7f8e140ad650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.581+0000 7f8e1bdff700 1 --2- 172.21.15.27:0/3537868951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e14156ab0 0x7f8e140ad650 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f8e100099b0 tx=0x7f8e10009250 comp rx=0 tx=0).stop 2024-01-11T08:26:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.581+0000 7f8e1bdff700 1 -- 172.21.15.27:0/3537868951 shutdown_connections 2024-01-11T08:26:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.581+0000 7f8e1bdff700 1 --2- 172.21.15.27:0/3537868951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e14156ab0 0x7f8e140ad650 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.581+0000 7f8e1bdff700 1 --2- 172.21.15.27:0/3537868951 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8e140b0550 0x7f8e14160070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.581+0000 7f8e1bdff700 1 -- 172.21.15.27:0/3537868951 >> 172.21.15.27:0/3537868951 conn(0x7f8e140680b0 msgr2=0x7f8e140cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.582+0000 7f8e1bdff700 1 -- 172.21.15.27:0/3537868951 shutdown_connections 2024-01-11T08:26:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.582+0000 7f8e1bdff700 1 -- 172.21.15.27:0/3537868951 wait complete. 2024-01-11T08:26:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.583+0000 7f8e1bdff700 1 Processor -- start 2024-01-11T08:26:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.583+0000 7f8e1bdff700 1 -- start start 2024-01-11T08:26:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.584+0000 7f8e1bdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e140b0550 0x7f8e14159140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.584+0000 7f8e1bdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e140cc9b0 0x7f8e1414a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.584+0000 7f8e1bdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8e14156ab0 0x7f8e14156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.584+0000 7f8e1bdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e14149db0 con 0x7f8e140b0550 2024-01-11T08:26:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.584+0000 7f8e1bdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e14149a80 con 0x7f8e140cc9b0 2024-01-11T08:26:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.584+0000 7f8e1bdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1415ea60 con 0x7f8e14156ab0 2024-01-11T08:26:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.584+0000 7f8e19b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e140b0550 0x7f8e14159140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.584+0000 7f8e19b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e140b0550 0x7f8e14159140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35480/0 (socket says 172.21.15.27:35480) 2024-01-11T08:26:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.584+0000 7f8e19b9b700 1 -- 172.21.15.27:0/55157509 learned_addr learned my addr 172.21.15.27:0/55157509 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:45.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.584+0000 7f8e1939a700 1 --2- 172.21.15.27:0/55157509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e140cc9b0 0x7f8e1414a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.585+0000 7f8e1a39c700 1 --2- 172.21.15.27:0/55157509 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8e14156ab0 0x7f8e14156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.585+0000 7f8e19b9b700 1 -- 172.21.15.27:0/55157509 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8e14156ab0 msgr2=0x7f8e14156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:45.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.585+0000 7f8e19b9b700 1 --2- 172.21.15.27:0/55157509 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8e14156ab0 0x7f8e14156e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.585+0000 7f8e19b9b700 1 -- 172.21.15.27:0/55157509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e140cc9b0 msgr2=0x7f8e1414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:45.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.585+0000 7f8e19b9b700 1 --2- 172.21.15.27:0/55157509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e140cc9b0 0x7f8e1414a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.585+0000 7f8e19b9b700 1 -- 172.21.15.27:0/55157509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e100094f0 con 0x7f8e140b0550 2024-01-11T08:26:45.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.585+0000 7f8e19b9b700 1 --2- 172.21.15.27:0/55157509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e140b0550 0x7f8e14159140 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f8e1406fe90 tx=0x7f8e10017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:45.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.585+0000 7f8e09ffb700 1 -- 172.21.15.27:0/55157509 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e10011e40 con 0x7f8e140b0550 2024-01-11T08:26:45.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.585+0000 7f8e09ffb700 1 -- 172.21.15.27:0/55157509 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e10024440 con 0x7f8e140b0550 2024-01-11T08:26:45.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.585+0000 7f8e09ffb700 1 -- 172.21.15.27:0/55157509 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e100247e0 con 0x7f8e140b0550 2024-01-11T08:26:45.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.585+0000 7f8e1bdff700 1 -- 172.21.15.27:0/55157509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1415ed40 con 0x7f8e140b0550 2024-01-11T08:26:45.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.586+0000 7f8e1bdff700 1 -- 172.21.15.27:0/55157509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e14159680 con 0x7f8e140b0550 2024-01-11T08:26:45.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.588+0000 7f8e09ffb700 1 -- 172.21.15.27:0/55157509 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8e1001a420 con 0x7f8e140b0550 2024-01-11T08:26:45.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.588+0000 7f8e09ffb700 1 --2- 172.21.15.27:0/55157509 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8df8066f30 0x7f8df80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.589+0000 7f8e09ffb700 1 -- 172.21.15.27:0/55157509 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f8e100863e0 con 0x7f8e140b0550 2024-01-11T08:26:45.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.589+0000 7f8e1939a700 1 --2- 172.21.15.27:0/55157509 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8df8066f30 0x7f8df80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.590+0000 7f8e1939a700 1 --2- 172.21.15.27:0/55157509 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8df8066f30 0x7f8df80693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8e140cc010 tx=0x7f8e04009040 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:45.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.590+0000 7f8e2a5f4b80 1 Processor -- start 2024-01-11T08:26:45.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.590+0000 7f8e2a5f4b80 1 -- start start 2024-01-11T08:26:45.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.591+0000 7f8e2a5f4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdb20 0x560d7a9dedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.591+0000 7f8e2a5f4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7a9d93a0 0x560d7aa05860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.591+0000 7f8e2a5f4b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x560d7a9fdef0 0x560d7aa05da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.591+0000 7f8e2a5f4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d7a8f6310 con 0x560d7a9fdb20 2024-01-11T08:26:45.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.591+0000 7f8e2a5f4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d7a933130 con 0x560d7a9d93a0 2024-01-11T08:26:45.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.591+0000 7f8e2a5f4b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x560d7a930af0 con 0x560d7a9fdef0 2024-01-11T08:26:45.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.591+0000 7f8e19b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdb20 0x560d7a9dedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.591+0000 7f8e19b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdb20 0x560d7a9dedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35492/0 (socket says 172.21.15.27:35492) 2024-01-11T08:26:45.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.592+0000 7f8e19b9b700 1 -- 172.21.15.27:0/3179988265 learned_addr learned my addr 172.21.15.27:0/3179988265 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:45.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.592+0000 7f8e19b9b700 1 -- 172.21.15.27:0/3179988265 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x560d7a9fdef0 msgr2=0x560d7aa05da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:45.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.592+0000 7f8e19b9b700 1 --2- 172.21.15.27:0/3179988265 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x560d7a9fdef0 0x560d7aa05da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.592+0000 7f8e19b9b700 1 -- 172.21.15.27:0/3179988265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7a9d93a0 msgr2=0x560d7aa05860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:45.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.592+0000 7f8e19b9b700 1 --2- 172.21.15.27:0/3179988265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7a9d93a0 0x560d7aa05860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.592+0000 7f8e19b9b700 1 -- 172.21.15.27:0/3179988265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560d7a81f0a0 con 0x560d7a9fdb20 2024-01-11T08:26:45.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.592+0000 7f8e19b9b700 1 --2- 172.21.15.27:0/3179988265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdb20 0x560d7a9dedf0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f8e10009fd0 tx=0x7f8e100847a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1a5e669bedf9c0e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:45.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.592+0000 7f8df6ffd700 1 -- 172.21.15.27:0/3179988265 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1005a520 con 0x560d7a9fdb20 2024-01-11T08:26:45.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.592+0000 7f8df6ffd700 1 -- 172.21.15.27:0/3179988265 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1005e270 con 0x560d7a9fdb20 2024-01-11T08:26:45.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.592+0000 7f8df6ffd700 1 -- 172.21.15.27:0/3179988265 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e10055670 con 0x560d7a9fdb20 2024-01-11T08:26:45.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.593+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/3179988265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdb20 msgr2=0x560d7a9dedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:45.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.593+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/3179988265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdb20 0x560d7a9dedf0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f8e10009fd0 tx=0x7f8e100847a0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.593+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/3179988265 shutdown_connections 2024-01-11T08:26:45.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.593+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/3179988265 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x560d7a9fdef0 0x560d7aa05da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.593+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/3179988265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdb20 0x560d7a9dedf0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.593+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/3179988265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7a9d93a0 0x560d7aa05860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.593+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/3179988265 >> 172.21.15.27:0/3179988265 conn(0x560d7a9db300 msgr2=0x560d7a9ddbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:45.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.593+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/3179988265 shutdown_connections 2024-01-11T08:26:45.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.593+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/3179988265 wait complete. 2024-01-11T08:26:45.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.594+0000 7f8e2a5f4b80 1 -- start start 2024-01-11T08:26:45.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.594+0000 7f8e2a5f4b80 1 Processor -- start 2024-01-11T08:26:45.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.595+0000 7f8e2a5f4b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x560d7a9d93a0 0x560d7a9fd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.595+0000 7f8e2a5f4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdb20 0x560d7aa26c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.595+0000 7f8e2a5f4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7a9fdef0 0x560d7a9d8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.595+0000 7f8e2a5f4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d7a92c8e0 con 0x560d7a9fdb20 2024-01-11T08:26:45.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.595+0000 7f8e2a5f4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d7a8d8980 con 0x560d7a9fdef0 2024-01-11T08:26:45.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.595+0000 7f8e2a5f4b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x560d7a8d8620 con 0x560d7a9d93a0 2024-01-11T08:26:45.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.595+0000 7f8e2a5f4b80 10 client.0 _ll_register_callbacks cb 0x560d7a73ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T08:26:45.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.595+0000 7f8e1939a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdb20 0x560d7aa26c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.595+0000 7f8e1939a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdb20 0x560d7aa26c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35508/0 (socket says 172.21.15.27:35508) 2024-01-11T08:26:45.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.595+0000 7f8e1939a700 1 -- 172.21.15.27:0/611890547 learned_addr learned my addr 172.21.15.27:0/611890547 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:45.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.595+0000 7f8e19b9b700 1 --2- 172.21.15.27:0/611890547 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x560d7a9d93a0 0x560d7a9fd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.595+0000 7f8e1a39c700 1 --2- 172.21.15.27:0/611890547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7a9fdef0 0x560d7a9d8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.596+0000 7f8e1939a700 1 -- 172.21.15.27:0/611890547 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x560d7a9d93a0 msgr2=0x560d7a9fd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:45.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.596+0000 7f8e1939a700 1 --2- 172.21.15.27:0/611890547 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x560d7a9d93a0 0x560d7a9fd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.596+0000 7f8e1939a700 1 -- 172.21.15.27:0/611890547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7a9fdef0 msgr2=0x560d7a9d8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:45.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.596+0000 7f8e1939a700 1 --2- 172.21.15.27:0/611890547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7a9fdef0 0x560d7a9d8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.596+0000 7f8e1939a700 1 -- 172.21.15.27:0/611890547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e14159680 con 0x560d7a9fdb20 2024-01-11T08:26:45.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.596+0000 7f8e1939a700 1 --2- 172.21.15.27:0/611890547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdb20 0x560d7aa26c00 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f8e14149070 tx=0x7f8e04015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60eeb9d42e1adef4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:45.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.596+0000 7f8deffff700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T08:26:45.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.596+0000 7f8deffff700 1 -- 172.21.15.27:0/611890547 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e04014c10 con 0x560d7a9fdb20 2024-01-11T08:26:45.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.596+0000 7f8deffff700 1 -- 172.21.15.27:0/611890547 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f8e0401c040 con 0x560d7a9fdb20 2024-01-11T08:26:45.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.596+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/611890547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560d7a9d1560 con 0x560d7a9fdb20 2024-01-11T08:26:45.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.596+0000 7f8deffff700 1 -- 172.21.15.27:0/611890547 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e04020410 con 0x560d7a9fdb20 2024-01-11T08:26:45.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.596+0000 7f8e2a5f4b80 10 client.7218 fetch_fsmap learned FSMap version 50 2024-01-11T08:26:45.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.596+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/611890547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) v3 -- 0x560d7a8fb3c0 con 0x560d7a9fdb20 2024-01-11T08:26:45.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.597+0000 7f8deffff700 1 -- 172.21.15.27:0/611890547 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560d7a9d1560 con 0x560d7a9fdb20 2024-01-11T08:26:45.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.597+0000 7f8deffff700 1 -- 172.21.15.27:0/611890547 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 50) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f8e04020570 con 0x560d7a9fdb20 2024-01-11T08:26:45.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.597+0000 7f8e2a5f4b80 10 client.7218 fetch_fsmap finished waiting for FSMap version 50 2024-01-11T08:26:45.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.597+0000 7f8e2a5f4b80 10 client.7218 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-11T08:26:45.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.597+0000 7f8e2a5f4b80 20 client.7218 populate_metadata read hostname 'smithi027' 2024-01-11T08:26:45.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.597+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/611890547 >> [v2:172.21.15.27:6833/751423743,v1:172.21.15.27:6835/751423743] conn(0x560d7aa25460 0x560d7aa25830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.597+0000 7f8e2a5f4b80 4 client.7218 mds_command: new command op to 7089 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-11T08:26:45.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.597+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/611890547 --> [v2:172.21.15.27:6833/751423743,v1:172.21.15.27:6835/751423743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560d7a9ae0e0 con 0x560d7aa25460 2024-01-11T08:26:45.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.597+0000 7f8e19b9b700 1 --2- 172.21.15.27:0/611890547 >> [v2:172.21.15.27:6833/751423743,v1:172.21.15.27:6835/751423743] conn(0x560d7aa25460 0x560d7aa25830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.598+0000 7f8e19b9b700 1 --2- 172.21.15.27:0/611890547 >> [v2:172.21.15.27:6833/751423743,v1:172.21.15.27:6835/751423743] conn(0x560d7aa25460 0x560d7aa25830 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24b3d6a222d199c6 server_cookie=cf1a23ae3dd3bf38 in_seq=0 out_seq=0 2024-01-11T08:26:45.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.598+0000 7f8deffff700 10 client.7218 ms_handle_connect on v2:172.21.15.27:6833/751423743 2024-01-11T08:26:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.598+0000 7f8deffff700 1 -- 172.21.15.27:0/611890547 <== mds.0 v2:172.21.15.27:6833/751423743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x560d7a9ae0e0 con 0x560d7aa25460 2024-01-11T08:26:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.598+0000 7f8deffff700 10 client.7218 handle_command_reply: tid=0 2024-01-11T08:26:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.598+0000 7f8e2a5f4b80 1 client.7218 shutdown 2024-01-11T08:26:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.599+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/611890547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdb20 msgr2=0x560d7aa26c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.599+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/611890547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdb20 0x560d7aa26c00 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f8e14149070 tx=0x7f8e04015000 comp rx=0 tx=0).stop 2024-01-11T08:26:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.599+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/611890547 shutdown_connections 2024-01-11T08:26:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.599+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/611890547 >> [v2:172.21.15.27:6833/751423743,v1:172.21.15.27:6835/751423743] conn(0x560d7aa25460 0x560d7aa25830 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.599+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/611890547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7a9fdef0 0x560d7a9d8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.599+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/611890547 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x560d7a9d93a0 0x560d7a9fd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.599+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/611890547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdb20 0x560d7aa26c00 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.599+0000 7f8deffff700 0 client.7218 ms_handle_reset on v2:172.21.15.27:6833/751423743 2024-01-11T08:26:45.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.599+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/611890547 >> 172.21.15.27:0/611890547 conn(0x560d7a9db300 msgr2=0x560d7aa0c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:45.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.599+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/611890547 shutdown_connections 2024-01-11T08:26:45.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.599+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/611890547 wait complete. 2024-01-11T08:26:45.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.599+0000 7f8e2a5f4b80 20 client.7218 trim_cache size 0 max 16384 2024-01-11T08:26:45.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.616+0000 7f8e2a5f4b80 1 Processor -- start 2024-01-11T08:26:45.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.617+0000 7f8e2a5f4b80 1 -- start start 2024-01-11T08:26:45.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.617+0000 7f8e2a5f4b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x560d7a9d93a0 0x560d7a9fbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.617+0000 7f8e2a5f4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7a9fdb20 0x560d7aa0af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.617+0000 7f8e2a5f4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdef0 0x560d7aa7f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.617+0000 7f8e2a5f4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d7a7c41e0 con 0x560d7a9fdef0 2024-01-11T08:26:45.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.617+0000 7f8e2a5f4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d7a9dd710 con 0x560d7a9fdb20 2024-01-11T08:26:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.617+0000 7f8e2a5f4b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x560d7a9d86f0 con 0x560d7a9d93a0 2024-01-11T08:26:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.618+0000 7f8e1a39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdef0 0x560d7aa7f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.618+0000 7f8e1a39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdef0 0x560d7aa7f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35510/0 (socket says 172.21.15.27:35510) 2024-01-11T08:26:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.618+0000 7f8e1a39c700 1 -- 172.21.15.27:0/816898897 learned_addr learned my addr 172.21.15.27:0/816898897 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.618+0000 7f8e1939a700 1 --2- 172.21.15.27:0/816898897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7a9fdb20 0x560d7aa0af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.618+0000 7f8e19b9b700 1 --2- 172.21.15.27:0/816898897 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x560d7a9d93a0 0x560d7a9fbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.618+0000 7f8e1a39c700 1 -- 172.21.15.27:0/816898897 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x560d7a9d93a0 msgr2=0x560d7a9fbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.618+0000 7f8e1a39c700 1 --2- 172.21.15.27:0/816898897 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x560d7a9d93a0 0x560d7a9fbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.618+0000 7f8e1a39c700 1 -- 172.21.15.27:0/816898897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7a9fdb20 msgr2=0x560d7aa0af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.618+0000 7f8e1a39c700 1 --2- 172.21.15.27:0/816898897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7a9fdb20 0x560d7aa0af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.618+0000 7f8e1a39c700 1 -- 172.21.15.27:0/816898897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560d7a7931d0 con 0x560d7a9fdef0 2024-01-11T08:26:45.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.618+0000 7f8e1a39c700 1 --2- 172.21.15.27:0/816898897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdef0 0x560d7aa7f6b0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c0073d0 tx=0x7f8e0c0097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5d51bcedcd862aa server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.619+0000 7f8df77fe700 1 -- 172.21.15.27:0/816898897 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c017470 con 0x560d7a9fdef0 2024-01-11T08:26:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.619+0000 7f8df77fe700 1 -- 172.21.15.27:0/816898897 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0c0175d0 con 0x560d7a9fdef0 2024-01-11T08:26:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.619+0000 7f8df77fe700 1 -- 172.21.15.27:0/816898897 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c017950 con 0x560d7a9fdef0 2024-01-11T08:26:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.619+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/816898897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdef0 msgr2=0x560d7aa7f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.619+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/816898897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdef0 0x560d7aa7f6b0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c0073d0 tx=0x7f8e0c0097b0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.619+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/816898897 shutdown_connections 2024-01-11T08:26:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.619+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/816898897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9fdef0 0x560d7aa7f6b0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.619+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/816898897 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x560d7a9d93a0 0x560d7a9fbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.619+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/816898897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7a9fdb20 0x560d7aa0af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.619+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/816898897 >> 172.21.15.27:0/816898897 conn(0x560d7a9db300 msgr2=0x560d7aa0bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:45.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.619+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/816898897 shutdown_connections 2024-01-11T08:26:45.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.619+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/816898897 wait complete. 2024-01-11T08:26:45.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.620+0000 7f8e2a5f4b80 1 -- start start 2024-01-11T08:26:45.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.620+0000 7f8e2a5f4b80 1 Processor -- start 2024-01-11T08:26:45.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.621+0000 7f8e2a5f4b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9d93a0 0x560d7aa05420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.621+0000 7f8e2a5f4b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x560d7a9fdb20 0x560d7aa32ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.621+0000 7f8e2a5f4b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7a9fdef0 0x560d7aa80330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.621+0000 7f8e2a5f4b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d7aa811e0 con 0x560d7a9d93a0 2024-01-11T08:26:45.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.621+0000 7f8e2a5f4b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d7aa05960 con 0x560d7a9fdef0 2024-01-11T08:26:45.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.621+0000 7f8e2a5f4b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x560d7aa23a70 con 0x560d7a9fdb20 2024-01-11T08:26:45.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.621+0000 7f8e2a5f4b80 10 client.0 _ll_register_callbacks cb 0x560d7a73ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T08:26:45.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.621+0000 7f8e19b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9d93a0 0x560d7aa05420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.621+0000 7f8e19b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9d93a0 0x560d7aa05420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35526/0 (socket says 172.21.15.27:35526) 2024-01-11T08:26:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.621+0000 7f8e19b9b700 1 -- 172.21.15.27:0/1869099995 learned_addr learned my addr 172.21.15.27:0/1869099995 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.622+0000 7f8e1939a700 1 --2- 172.21.15.27:0/1869099995 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x560d7a9fdb20 0x560d7aa32ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.622+0000 7f8e1a39c700 1 --2- 172.21.15.27:0/1869099995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7a9fdef0 0x560d7aa80330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.622+0000 7f8e19b9b700 1 -- 172.21.15.27:0/1869099995 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x560d7a9fdb20 msgr2=0x560d7aa32ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.622+0000 7f8e19b9b700 1 --2- 172.21.15.27:0/1869099995 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x560d7a9fdb20 0x560d7aa32ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.622+0000 7f8e19b9b700 1 -- 172.21.15.27:0/1869099995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7a9fdef0 msgr2=0x560d7aa80330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.622+0000 7f8e19b9b700 1 --2- 172.21.15.27:0/1869099995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7a9fdef0 0x560d7aa80330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.622+0000 7f8e19b9b700 1 -- 172.21.15.27:0/1869099995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e0c011040 con 0x560d7a9d93a0 2024-01-11T08:26:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.622+0000 7f8e19b9b700 1 --2- 172.21.15.27:0/1869099995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9d93a0 0x560d7aa05420 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f8e10056fd0 tx=0x7f8e10080040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f521cb6e26e1e560 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.622+0000 7f8deffff700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T08:26:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.622+0000 7f8deffff700 1 -- 172.21.15.27:0/1869099995 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1005a520 con 0x560d7a9d93a0 2024-01-11T08:26:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.622+0000 7f8deffff700 1 -- 172.21.15.27:0/1869099995 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f8e10055410 con 0x560d7a9d93a0 2024-01-11T08:26:45.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.622+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/1869099995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e1005e270 con 0x560d7a9d93a0 2024-01-11T08:26:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.623+0000 7f8deffff700 1 -- 172.21.15.27:0/1869099995 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e10024940 con 0x560d7a9d93a0 2024-01-11T08:26:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.623+0000 7f8e2a5f4b80 10 client.7224 fetch_fsmap learned FSMap version 50 2024-01-11T08:26:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.623+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/1869099995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) v3 -- 0x560d7a76cd60 con 0x560d7a9d93a0 2024-01-11T08:26:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.623+0000 7f8deffff700 1 -- 172.21.15.27:0/1869099995 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1005e270 con 0x560d7a9d93a0 2024-01-11T08:26:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.623+0000 7f8deffff700 1 -- 172.21.15.27:0/1869099995 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 50) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f8e10066070 con 0x560d7a9d93a0 2024-01-11T08:26:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.623+0000 7f8e2a5f4b80 10 client.7224 fetch_fsmap finished waiting for FSMap version 50 2024-01-11T08:26:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.623+0000 7f8e2a5f4b80 10 client.7224 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-11T08:26:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.623+0000 7f8e2a5f4b80 20 client.7224 populate_metadata read hostname 'smithi027' 2024-01-11T08:26:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.623+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/1869099995 >> [v2:172.21.15.27:6833/751423743,v1:172.21.15.27:6835/751423743] conn(0x560d7aa808e0 0x560d7aa7bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.623+0000 7f8e2a5f4b80 4 client.7224 mds_command: new command op to 7089 tid=0[{"prefix": "status"}] 2024-01-11T08:26:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.623+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/1869099995 --> [v2:172.21.15.27:6833/751423743,v1:172.21.15.27:6835/751423743] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560d7a8cf8d0 con 0x560d7aa808e0 2024-01-11T08:26:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.623+0000 7f8e1939a700 1 --2- 172.21.15.27:0/1869099995 >> [v2:172.21.15.27:6833/751423743,v1:172.21.15.27:6835/751423743] conn(0x560d7aa808e0 0x560d7aa7bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.624+0000 7f8e1939a700 1 --2- 172.21.15.27:0/1869099995 >> [v2:172.21.15.27:6833/751423743,v1:172.21.15.27:6835/751423743] conn(0x560d7aa808e0 0x560d7aa7bdc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab024214967a5407 server_cookie=1e96508ac2aaca35 in_seq=0 out_seq=0 2024-01-11T08:26:45.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.624+0000 7f8deffff700 10 client.7224 ms_handle_connect on v2:172.21.15.27:6833/751423743 2024-01-11T08:26:45.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.624+0000 7f8deffff700 1 -- 172.21.15.27:0/1869099995 <== mds.0 v2:172.21.15.27:6833/751423743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x560d7a8cf8d0 con 0x560d7aa808e0 2024-01-11T08:26:45.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.624+0000 7f8deffff700 10 client.7224 handle_command_reply: tid=0 2024-01-11T08:26:45.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.624+0000 7f8e2a5f4b80 1 client.7224 shutdown 2024-01-11T08:26:45.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.624+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/1869099995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9d93a0 msgr2=0x560d7aa05420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:45.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.624+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/1869099995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9d93a0 0x560d7aa05420 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f8e10056fd0 tx=0x7f8e10080040 comp rx=0 tx=0).stop 2024-01-11T08:26:45.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.624+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/1869099995 shutdown_connections 2024-01-11T08:26:45.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.624+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/1869099995 >> [v2:172.21.15.27:6833/751423743,v1:172.21.15.27:6835/751423743] conn(0x560d7aa808e0 0x560d7aa7bdc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.624+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/1869099995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7a9fdef0 0x560d7aa80330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.624+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/1869099995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d7a9d93a0 0x560d7aa05420 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.624+0000 7f8deffff700 0 client.7224 ms_handle_reset on v2:172.21.15.27:6833/751423743 2024-01-11T08:26:45.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.624+0000 7f8e2a5f4b80 1 --2- 172.21.15.27:0/1869099995 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x560d7a9fdb20 0x560d7aa32ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.624+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/1869099995 >> 172.21.15.27:0/1869099995 conn(0x560d7a9db300 msgr2=0x560d7aa0c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:45.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.624+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/1869099995 shutdown_connections 2024-01-11T08:26:45.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.625+0000 7f8e2a5f4b80 1 -- 172.21.15.27:0/1869099995 wait complete. 2024-01-11T08:26:45.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.625+0000 7f8e2a5f4b80 20 client.7224 trim_cache size 0 max 16384 2024-01-11T08:26:45.629 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T08:26:45.629 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "8fddb4a7-9d66-40f1-892d-6fe86c728ee4", 2024-01-11T08:26:45.629 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-11T08:26:45.629 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 7089, 2024-01-11T08:26:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-11T08:26:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-11T08:26:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-11T08:26:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 6.6234836259999996, 2024-01-11T08:26:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 50, 2024-01-11T08:26:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 101, 2024-01-11T08:26:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 101, 2024-01-11T08:26:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 12.29251509 2024-01-11T08:26:45.630 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T08:26:45.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.625+0000 7f8e1bdff700 1 -- 172.21.15.27:0/55157509 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8df8066f30 msgr2=0x7f8df80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:45.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.625+0000 7f8e1bdff700 1 --2- 172.21.15.27:0/55157509 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8df8066f30 0x7f8df80693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8e140cc010 tx=0x7f8e04009040 comp rx=0 tx=0).stop 2024-01-11T08:26:45.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.625+0000 7f8e1bdff700 1 -- 172.21.15.27:0/55157509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e140b0550 msgr2=0x7f8e14159140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:45.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.625+0000 7f8e1bdff700 1 --2- 172.21.15.27:0/55157509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e140b0550 0x7f8e14159140 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f8e1406fe90 tx=0x7f8e10017000 comp rx=0 tx=0).stop 2024-01-11T08:26:45.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.625+0000 7f8e1bdff700 1 -- 172.21.15.27:0/55157509 shutdown_connections 2024-01-11T08:26:45.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.625+0000 7f8e1bdff700 1 --2- 172.21.15.27:0/55157509 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8df8066f30 0x7f8df80693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.625+0000 7f8e1bdff700 1 --2- 172.21.15.27:0/55157509 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8e14156ab0 0x7f8e14156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.625+0000 7f8e1bdff700 1 --2- 172.21.15.27:0/55157509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e140b0550 0x7f8e14159140 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.625+0000 7f8e1bdff700 1 --2- 172.21.15.27:0/55157509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e140cc9b0 0x7f8e1414a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.625+0000 7f8e1bdff700 1 -- 172.21.15.27:0/55157509 >> 172.21.15.27:0/55157509 conn(0x7f8e140680b0 msgr2=0x7f8e140ace30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:45.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.626+0000 7f8e1bdff700 1 -- 172.21.15.27:0/55157509 shutdown_connections 2024-01-11T08:26:45.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.626+0000 7f8e1bdff700 1 -- 172.21.15.27:0/55157509 wait complete. 2024-01-11T08:26:45.643 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-11T08:26:45.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.843+0000 7ff1b49ff700 1 Processor -- start 2024-01-11T08:26:45.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.843+0000 7ff1b49ff700 1 -- start start 2024-01-11T08:26:45.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.845+0000 7ff1b49ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1b00b19d0 0x7ff1b0158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.845+0000 7ff1b49ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff1b006b610 0x7ff1b0159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.845+0000 7ff1b49ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1b0158b60 con 0x7ff1b01575e0 2024-01-11T08:26:45.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.845+0000 7ff1b49ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1b00cb5e0 con 0x7ff1b006b610 2024-01-11T08:26:45.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.845+0000 7ff1b49ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b01599b0 con 0x7ff1b00b19d0 2024-01-11T08:26:45.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.845+0000 7ff1aed9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff1b006b610 0x7ff1b0159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.845+0000 7ff1ae59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff1b01575e0 0x7ff1b00b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35940/0 (socket says 172.21.15.27:35940) 2024-01-11T08:26:45.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.845+0000 7ff1ae59c700 1 -- 172.21.15.27:0/3169111172 learned_addr learned my addr 172.21.15.27:0/3169111172 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:45.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.845+0000 7ff1aed9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff1b006b610 0x7ff1b0159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35530/0 (socket says 172.21.15.27:35530) 2024-01-11T08:26:45.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.845+0000 7ff1add9b700 1 --2- 172.21.15.27:0/3169111172 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1b00b19d0 0x7ff1b0158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.845+0000 7ff1aed9d700 1 -- 172.21.15.27:0/3169111172 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1b00b19d0 msgr2=0x7ff1b0158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:45.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.845+0000 7ff1aed9d700 1 --2- 172.21.15.27:0/3169111172 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1b00b19d0 0x7ff1b0158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.845+0000 7ff1aed9d700 1 -- 172.21.15.27:0/3169111172 >> v1:172.21.15.27:6789/0 conn(0x7ff1b01575e0 legacy=0x7ff1b00b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:45.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.846+0000 7ff1aed9d700 1 -- 172.21.15.27:0/3169111172 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b00c0f90 con 0x7ff1b006b610 2024-01-11T08:26:45.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.846+0000 7ff1add9b700 1 --2- 172.21.15.27:0/3169111172 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1b00b19d0 0x7ff1b0158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:26:45.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.845+0000 7ff1ad59a700 1 -- 172.21.15.27:0/3169111172 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941297802 0 0) 0x7ff1a0001110 con 0x7ff1b01575e0 2024-01-11T08:26:45.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.846+0000 7ff1aed9d700 1 --2- 172.21.15.27:0/3169111172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b006b610 0x7ff1b0159e40 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8009e60 tx=0x7ff1a8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=871b8052209f63d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:45.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.846+0000 7ff1ad59a700 1 -- 172.21.15.27:0/3169111172 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a800e070 con 0x7ff1b006b610 2024-01-11T08:26:45.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.846+0000 7ff1ad59a700 1 -- 172.21.15.27:0/3169111172 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a800bc30 con 0x7ff1b006b610 2024-01-11T08:26:45.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.846+0000 7ff1ad59a700 1 -- 172.21.15.27:0/3169111172 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8012430 con 0x7ff1b006b610 2024-01-11T08:26:45.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.847+0000 7ff1b49ff700 1 -- 172.21.15.27:0/3169111172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b006b610 msgr2=0x7ff1b0159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:45.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.847+0000 7ff1b49ff700 1 --2- 172.21.15.27:0/3169111172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b006b610 0x7ff1b0159e40 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8009e60 tx=0x7ff1a8009780 comp rx=0 tx=0).stop 2024-01-11T08:26:45.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.847+0000 7ff1b49ff700 1 -- 172.21.15.27:0/3169111172 shutdown_connections 2024-01-11T08:26:45.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.847+0000 7ff1b49ff700 1 --2- 172.21.15.27:0/3169111172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b006b610 0x7ff1b0159e40 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.847+0000 7ff1b49ff700 1 --2- 172.21.15.27:0/3169111172 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1b00b19d0 0x7ff1b0158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.847+0000 7ff1b49ff700 1 -- 172.21.15.27:0/3169111172 >> 172.21.15.27:0/3169111172 conn(0x7ff1b00680e0 msgr2=0x7ff1b0069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:45.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.847+0000 7ff1b49ff700 1 -- 172.21.15.27:0/3169111172 shutdown_connections 2024-01-11T08:26:45.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.847+0000 7ff1b49ff700 1 -- 172.21.15.27:0/3169111172 wait complete. 2024-01-11T08:26:45.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.848+0000 7ff1b49ff700 1 Processor -- start 2024-01-11T08:26:45.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.848+0000 7ff1b49ff700 1 -- start start 2024-01-11T08:26:45.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.849+0000 7ff1b49ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1b006b610 0x7ff1b01407e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.849+0000 7ff1b49ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b00b19d0 0x7ff1b0140d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.849+0000 7ff1b49ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1b01575e0 0x7ff1b00c11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.849+0000 7ff1b49ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b01541d0 con 0x7ff1b00b19d0 2024-01-11T08:26:45.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.849+0000 7ff1b49ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b01413a0 con 0x7ff1b01575e0 2024-01-11T08:26:45.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.849+0000 7ff1b49ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b0141680 con 0x7ff1b006b610 2024-01-11T08:26:45.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.850+0000 7ff1add9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b00b19d0 0x7ff1b0140d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.850+0000 7ff1add9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b00b19d0 0x7ff1b0140d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35532/0 (socket says 172.21.15.27:35532) 2024-01-11T08:26:45.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.850+0000 7ff1add9b700 1 -- 172.21.15.27:0/2506056047 learned_addr learned my addr 172.21.15.27:0/2506056047 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:45.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.850+0000 7ff1ae59c700 1 --2- 172.21.15.27:0/2506056047 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1b006b610 0x7ff1b01407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.850+0000 7ff1aed9d700 1 --2- 172.21.15.27:0/2506056047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1b01575e0 0x7ff1b00c11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.850+0000 7ff1add9b700 1 -- 172.21.15.27:0/2506056047 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1b006b610 msgr2=0x7ff1b01407e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:45.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.850+0000 7ff1add9b700 1 --2- 172.21.15.27:0/2506056047 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1b006b610 0x7ff1b01407e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.850+0000 7ff1add9b700 1 -- 172.21.15.27:0/2506056047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1b01575e0 msgr2=0x7ff1b00c11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:45.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.850+0000 7ff1add9b700 1 --2- 172.21.15.27:0/2506056047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1b01575e0 0x7ff1b00c11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:45.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.850+0000 7ff1add9b700 1 -- 172.21.15.27:0/2506056047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a8009a20 con 0x7ff1b00b19d0 2024-01-11T08:26:45.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.850+0000 7ff1aed9d700 1 --2- 172.21.15.27:0/2506056047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1b01575e0 0x7ff1b00c11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:26:45.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.850+0000 7ff1add9b700 1 --2- 172.21.15.27:0/2506056047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b00b19d0 0x7ff1b0140d20 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400e8d0 tx=0x7ff1a4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:45.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.850+0000 7ff19e7fc700 1 -- 172.21.15.27:0/2506056047 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4013410 con 0x7ff1b00b19d0 2024-01-11T08:26:45.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.851+0000 7ff19e7fc700 1 -- 172.21.15.27:0/2506056047 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a4013570 con 0x7ff1b00b19d0 2024-01-11T08:26:45.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.851+0000 7ff19e7fc700 1 -- 172.21.15.27:0/2506056047 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4013910 con 0x7ff1b00b19d0 2024-01-11T08:26:45.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.851+0000 7ff1b49ff700 1 -- 172.21.15.27:0/2506056047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b0141960 con 0x7ff1b00b19d0 2024-01-11T08:26:45.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.851+0000 7ff1b49ff700 1 -- 172.21.15.27:0/2506056047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1b0141f20 con 0x7ff1b00b19d0 2024-01-11T08:26:45.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.852+0000 7ff1b49ff700 1 -- 172.21.15.27:0/2506056047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1b0142290 con 0x7ff1b00b19d0 2024-01-11T08:26:45.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.853+0000 7ff19e7fc700 1 -- 172.21.15.27:0/2506056047 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff1a4013d00 con 0x7ff1b00b19d0 2024-01-11T08:26:45.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.853+0000 7ff19e7fc700 1 --2- 172.21.15.27:0/2506056047 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff190066f30 0x7ff1900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:45.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.854+0000 7ff19e7fc700 1 -- 172.21.15.27:0/2506056047 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5733+0+0 (secure 0 0 0) 0x7ff1a40842c0 con 0x7ff1b00b19d0 2024-01-11T08:26:45.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.854+0000 7ff1ae59c700 1 --2- 172.21.15.27:0/2506056047 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff190066f30 0x7ff1900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:45.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.855+0000 7ff1ae59c700 1 --2- 172.21.15.27:0/2506056047 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff190066f30 0x7ff1900693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff1b00c86a0 tx=0x7ff1a0007f50 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:45.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:45.858+0000 7ff19e7fc700 1 -- 172.21.15.27:0/2506056047 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff1a4051510 con 0x7ff1b00b19d0 2024-01-11T08:26:46.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:46.116+0000 7ff1b49ff700 1 -- 172.21.15.27:0/2506056047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff1b00718d0 con 0x7ff1b00b19d0 2024-01-11T08:26:46.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:46.116+0000 7ff19e7fc700 1 -- 172.21.15.27:0/2506056047 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v50) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff1a4054b30 con 0x7ff1b00b19d0 2024-01-11T08:26:46.117 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-11T08:26:46.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:46.119+0000 7ff1b49ff700 1 -- 172.21.15.27:0/2506056047 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff190066f30 msgr2=0x7ff1900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:46.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:46.119+0000 7ff1b49ff700 1 --2- 172.21.15.27:0/2506056047 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff190066f30 0x7ff1900693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff1b00c86a0 tx=0x7ff1a0007f50 comp rx=0 tx=0).stop 2024-01-11T08:26:46.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:46.119+0000 7ff1b49ff700 1 -- 172.21.15.27:0/2506056047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b00b19d0 msgr2=0x7ff1b0140d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:46.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:46.119+0000 7ff1b49ff700 1 --2- 172.21.15.27:0/2506056047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b00b19d0 0x7ff1b0140d20 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400e8d0 tx=0x7ff1a4008040 comp rx=0 tx=0).stop 2024-01-11T08:26:46.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:46.119+0000 7ff1b49ff700 1 -- 172.21.15.27:0/2506056047 shutdown_connections 2024-01-11T08:26:46.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:46.119+0000 7ff1b49ff700 1 --2- 172.21.15.27:0/2506056047 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff190066f30 0x7ff1900693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:46.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:46.119+0000 7ff1b49ff700 1 --2- 172.21.15.27:0/2506056047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1b01575e0 0x7ff1b00c11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:46.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:46.119+0000 7ff1b49ff700 1 --2- 172.21.15.27:0/2506056047 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff1b006b610 0x7ff1b01407e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:46.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:46.119+0000 7ff1b49ff700 1 --2- 172.21.15.27:0/2506056047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b00b19d0 0x7ff1b0140d20 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:46.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:46.119+0000 7ff1b49ff700 1 -- 172.21.15.27:0/2506056047 >> 172.21.15.27:0/2506056047 conn(0x7ff1b00680e0 msgr2=0x7ff1b006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:46.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:46.119+0000 7ff1b49ff700 1 -- 172.21.15.27:0/2506056047 shutdown_connections 2024-01-11T08:26:46.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:46.119+0000 7ff1b49ff700 1 -- 172.21.15.27:0/2506056047 wait complete. 2024-01-11T08:26:46.137 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-11T08:26:46.137 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-11T08:26:46.137 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-11T08:26:46.137 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-11T08:26:46.137 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-11T08:26:46.137 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-11T08:26:46.137 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-11T08:26:46.137 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-11T08:26:46.138 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:26:46.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:26:46.138 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-11T08:26:46.186 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T08:26:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T08:26:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T08:26:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:26:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-11T08:26:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:26:46.186 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:26:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:26:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-11T08:26:46.186 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:26:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:26:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-11T08:26:46.187 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-11T08:26:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-11T08:26:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-11T08:26:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-11T08:26:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 41516sec preferred_lft 41516sec 2024-01-11T08:26:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-11T08:26:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:26:46.187 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:26:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-11T08:26:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-11T08:26:46.187 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T08:26:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff 2024-01-11T08:26:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-11T08:26:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:26:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::acd2:b3ff:fe6a:db4b/64 scope link 2024-01-11T08:26:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:26:46.188 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T08:26:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:fa:85:1b:3a:bc brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-11T08:26:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ecfa:85ff:fe1b:3abc/64 scope link 2024-01-11T08:26:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:26:46.188 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T08:26:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-11T08:26:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ec88:faff:fe17:41aa/64 scope link 2024-01-11T08:26:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:26:46.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:26:46.189 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-11T08:26:46.260 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-11T08:26:46.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:26:46.264 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-11T08:26:46.313 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-11T08:26:46.313 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-11T08:26:46.313 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-11T08:26:46.314 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-11T08:26:46.314 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-11T08:26:46.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:26:46.315 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T08:26:46.315 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T08:26:46.315 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-11T08:26:46.315 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-11T08:26:46.315 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-11T08:26:46.315 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T08:26:46.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:26:46.491 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T08:26:46.538 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-11T08:26:46.538 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-11T08:26:46.539 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:26:46.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:26:46.539 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-11T08:26:46.613 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:26:46.613 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-11T08:26:46.660 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:26:46.660 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:26:46.660 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-11T08:26:46.707 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-11T08:26:46.708 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-11T08:26:46.708 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:26:46.811 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:26:46.811 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:26:46.811 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:26:46.825 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:26:46.826 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:26:46.826 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:26:46.826 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:26:46.826 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:26:46.846 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:26:46.847 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:26:46.847 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:26:46.847 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:26:46.847 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:26:46.847 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790" 2024-01-11T08:26:46.847 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790:/ 2024-01-11T08:26:46.847 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=8fddb4a7-9d66-40f1-892d-6fe86c728ee4" will pass to kernel 2024-01-11T08:26:46.851 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-11T08:26:46.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:26:46.851 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-11T08:26:46.924 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-11T08:26:47.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.128+0000 7fd2ccbff700 1 Processor -- start 2024-01-11T08:26:47.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.128+0000 7fd2ccbff700 1 -- start start 2024-01-11T08:26:47.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.129+0000 7fd2ccbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd2c815ca10 0x7fd2c815a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:47.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.129+0000 7fd2ccbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2c815cde0 0x7fd2c80685c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.129+0000 7fd2ccbff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd2c815aed0 0x7fd2c8068b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.129+0000 7fd2ccbff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2c80b4bd0 con 0x7fd2c815aed0 2024-01-11T08:26:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.129+0000 7fd2ccbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2c815ac70 con 0x7fd2c815ca10 2024-01-11T08:26:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.129+0000 7fd2ccbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c80b3fc0 con 0x7fd2c815cde0 2024-01-11T08:26:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.129+0000 7fd2c6d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd2c815aed0 0x7fd2c8068b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.129+0000 7fd2c6d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd2c815aed0 0x7fd2c8068b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35536/0 (socket says 172.21.15.27:35536) 2024-01-11T08:26:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.129+0000 7fd2c6d9d700 1 -- 172.21.15.27:0/3853906899 learned_addr learned my addr 172.21.15.27:0/3853906899 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.130+0000 7fd2c5d9b700 1 --2- 172.21.15.27:0/3853906899 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2c815cde0 0x7fd2c80685c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.130+0000 7fd2c659c700 1 --2- 172.21.15.27:0/3853906899 >> v2:172.21.15.77:3300/0 conn(0x7fd2c815ca10 0x7fd2c815a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.130+0000 7fd2c6d9d700 1 -- 172.21.15.27:0/3853906899 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2c815cde0 msgr2=0x7fd2c80685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.130+0000 7fd2c6d9d700 1 --2- 172.21.15.27:0/3853906899 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2c815cde0 0x7fd2c80685c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.130+0000 7fd2c6d9d700 1 -- 172.21.15.27:0/3853906899 >> v2:172.21.15.77:3300/0 conn(0x7fd2c815ca10 msgr2=0x7fd2c815a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.130+0000 7fd2c6d9d700 1 --2- 172.21.15.27:0/3853906899 >> v2:172.21.15.77:3300/0 conn(0x7fd2c815ca10 0x7fd2c815a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.130+0000 7fd2c6d9d700 1 -- 172.21.15.27:0/3853906899 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c80b42a0 con 0x7fd2c815aed0 2024-01-11T08:26:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.130+0000 7fd2c659c700 1 --2- 172.21.15.27:0/3853906899 >> v2:172.21.15.77:3300/0 conn(0x7fd2c815ca10 0x7fd2c815a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:26:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.130+0000 7fd2c6d9d700 1 --2- 172.21.15.27:0/3853906899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2c815aed0 0x7fd2c8068b00 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80b6630 tx=0x7fd2c0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c649c3e31098458 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.130+0000 7fd2c559a700 1 -- 172.21.15.27:0/3853906899 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c000e070 con 0x7fd2c815aed0 2024-01-11T08:26:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.131+0000 7fd2c559a700 1 -- 172.21.15.27:0/3853906899 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c000ba20 con 0x7fd2c815aed0 2024-01-11T08:26:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.131+0000 7fd2c559a700 1 -- 172.21.15.27:0/3853906899 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c00092e0 con 0x7fd2c815aed0 2024-01-11T08:26:47.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.131+0000 7fd2ccbff700 1 -- 172.21.15.27:0/3853906899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2c815aed0 msgr2=0x7fd2c8068b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:47.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.131+0000 7fd2ccbff700 1 --2- 172.21.15.27:0/3853906899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2c815aed0 0x7fd2c8068b00 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80b6630 tx=0x7fd2c0009840 comp rx=0 tx=0).stop 2024-01-11T08:26:47.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.131+0000 7fd2ccbff700 1 -- 172.21.15.27:0/3853906899 shutdown_connections 2024-01-11T08:26:47.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.131+0000 7fd2ccbff700 1 --2- 172.21.15.27:0/3853906899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2c815aed0 0x7fd2c8068b00 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:47.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.131+0000 7fd2ccbff700 1 --2- 172.21.15.27:0/3853906899 >> v2:172.21.15.77:3300/0 conn(0x7fd2c815ca10 0x7fd2c815a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:47.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.131+0000 7fd2ccbff700 1 --2- 172.21.15.27:0/3853906899 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2c815cde0 0x7fd2c80685c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:47.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.131+0000 7fd2ccbff700 1 -- 172.21.15.27:0/3853906899 >> 172.21.15.27:0/3853906899 conn(0x7fd2c806ed40 msgr2=0x7fd2c806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:47.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.131+0000 7fd2ccbff700 1 -- 172.21.15.27:0/3853906899 shutdown_connections 2024-01-11T08:26:47.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.131+0000 7fd2ccbff700 1 -- 172.21.15.27:0/3853906899 wait complete. 2024-01-11T08:26:47.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.133+0000 7fd2ccbff700 1 Processor -- start 2024-01-11T08:26:47.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.134+0000 7fd2ccbff700 1 -- start start 2024-01-11T08:26:47.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.135+0000 7fd2ccbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2c815aed0 0x7fd2c8155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:47.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.135+0000 7fd2ccbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2c815ca10 0x7fd2c81562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:47.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.135+0000 7fd2ccbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2c815cde0 0x7fd2c8151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:47.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.135+0000 7fd2ccbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c8157710 con 0x7fd2c815aed0 2024-01-11T08:26:47.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.135+0000 7fd2ccbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c8156830 con 0x7fd2c815cde0 2024-01-11T08:26:47.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.135+0000 7fd2ccbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c8156b00 con 0x7fd2c815ca10 2024-01-11T08:26:47.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.135+0000 7fd2c659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2c815aed0 0x7fd2c8155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:47.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.135+0000 7fd2c659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2c815aed0 0x7fd2c8155c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35550/0 (socket says 172.21.15.27:35550) 2024-01-11T08:26:47.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.135+0000 7fd2c659c700 1 -- 172.21.15.27:0/2327257667 learned_addr learned my addr 172.21.15.27:0/2327257667 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:47.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.135+0000 7fd2c5d9b700 1 --2- 172.21.15.27:0/2327257667 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2c815ca10 0x7fd2c81562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:47.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.135+0000 7fd2c659c700 1 -- 172.21.15.27:0/2327257667 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2c815ca10 msgr2=0x7fd2c81562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:47.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.135+0000 7fd2c6d9d700 1 --2- 172.21.15.27:0/2327257667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2c815cde0 0x7fd2c8151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:47.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.136+0000 7fd2c659c700 1 --2- 172.21.15.27:0/2327257667 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2c815ca10 0x7fd2c81562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:47.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.136+0000 7fd2c659c700 1 -- 172.21.15.27:0/2327257667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2c815cde0 msgr2=0x7fd2c8151cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:47.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.136+0000 7fd2c659c700 1 --2- 172.21.15.27:0/2327257667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2c815cde0 0x7fd2c8151cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:47.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.136+0000 7fd2c659c700 1 -- 172.21.15.27:0/2327257667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c0009ae0 con 0x7fd2c815aed0 2024-01-11T08:26:47.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.136+0000 7fd2c5d9b700 1 --2- 172.21.15.27:0/2327257667 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2c815ca10 0x7fd2c81562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:47.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.136+0000 7fd2c6d9d700 1 --2- 172.21.15.27:0/2327257667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2c815cde0 0x7fd2c8151cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:26:47.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.136+0000 7fd2c659c700 1 --2- 172.21.15.27:0/2327257667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2c815aed0 0x7fd2c8155c80 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fd2b800e8d0 tx=0x7fd2b8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:47.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.136+0000 7fd2b67fc700 1 -- 172.21.15.27:0/2327257667 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b800ee70 con 0x7fd2c815aed0 2024-01-11T08:26:47.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.136+0000 7fd2b67fc700 1 -- 172.21.15.27:0/2327257667 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b800c5e0 con 0x7fd2c815aed0 2024-01-11T08:26:47.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.136+0000 7fd2b67fc700 1 -- 172.21.15.27:0/2327257667 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b800c980 con 0x7fd2c815aed0 2024-01-11T08:26:47.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.136+0000 7fd2ccbff700 1 -- 172.21.15.27:0/2327257667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c8156de0 con 0x7fd2c815aed0 2024-01-11T08:26:47.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.136+0000 7fd2ccbff700 1 -- 172.21.15.27:0/2327257667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c81573a0 con 0x7fd2c815aed0 2024-01-11T08:26:47.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.137+0000 7fd2ccbff700 1 -- 172.21.15.27:0/2327257667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c8169b60 con 0x7fd2c815aed0 2024-01-11T08:26:47.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.138+0000 7fd2b67fc700 1 -- 172.21.15.27:0/2327257667 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd2b8013410 con 0x7fd2c815aed0 2024-01-11T08:26:47.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.139+0000 7fd2b67fc700 1 --2- 172.21.15.27:0/2327257667 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd2a8066f30 0x7fd2a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:47.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.140+0000 7fd2b67fc700 1 -- 172.21.15.27:0/2327257667 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5733+0+0 (secure 0 0 0) 0x7fd2b8084450 con 0x7fd2c815aed0 2024-01-11T08:26:47.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.140+0000 7fd2c5d9b700 1 --2- 172.21.15.27:0/2327257667 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd2a8066f30 0x7fd2a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:47.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.141+0000 7fd2c5d9b700 1 --2- 172.21.15.27:0/2327257667 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd2a8066f30 0x7fd2a80693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc006f00 tx=0x7fd2bc008040 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:47.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.144+0000 7fd2b67fc700 1 -- 172.21.15.27:0/2327257667 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd2b8051600 con 0x7fd2c815aed0 2024-01-11T08:26:47.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.375+0000 7fd2ccbff700 1 -- 172.21.15.27:0/2327257667 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2c804c3c0 con 0x7fd2a8066f30 2024-01-11T08:26:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.378+0000 7fd2b67fc700 1 -- 172.21.15.27:0/2327257667 <== mgr.6745 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fd2c804c3c0 con 0x7fd2a8066f30 2024-01-11T08:26:47.377 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T08:26:47.378 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:26:47.378 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-11T08:26:47.378 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:26:47.378 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T08:26:47.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.379+0000 7fd2ccbff700 1 -- 172.21.15.27:0/2327257667 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd2a8066f30 msgr2=0x7fd2a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:47.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.379+0000 7fd2ccbff700 1 --2- 172.21.15.27:0/2327257667 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd2a8066f30 0x7fd2a80693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc006f00 tx=0x7fd2bc008040 comp rx=0 tx=0).stop 2024-01-11T08:26:47.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.379+0000 7fd2ccbff700 1 -- 172.21.15.27:0/2327257667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2c815aed0 msgr2=0x7fd2c8155c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:47.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.379+0000 7fd2ccbff700 1 --2- 172.21.15.27:0/2327257667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2c815aed0 0x7fd2c8155c80 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fd2b800e8d0 tx=0x7fd2b8008040 comp rx=0 tx=0).stop 2024-01-11T08:26:47.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.379+0000 7fd2ccbff700 1 -- 172.21.15.27:0/2327257667 shutdown_connections 2024-01-11T08:26:47.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.379+0000 7fd2ccbff700 1 --2- 172.21.15.27:0/2327257667 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd2a8066f30 0x7fd2a80693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:47.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.379+0000 7fd2ccbff700 1 --2- 172.21.15.27:0/2327257667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2c815cde0 0x7fd2c8151cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:47.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.379+0000 7fd2ccbff700 1 --2- 172.21.15.27:0/2327257667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2c815aed0 0x7fd2c8155c80 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:47.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.379+0000 7fd2ccbff700 1 --2- 172.21.15.27:0/2327257667 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2c815ca10 0x7fd2c81562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:47.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.379+0000 7fd2ccbff700 1 -- 172.21.15.27:0/2327257667 >> 172.21.15.27:0/2327257667 conn(0x7fd2c806ed40 msgr2=0x7fd2c80b5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:47.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.380+0000 7fd2ccbff700 1 -- 172.21.15.27:0/2327257667 shutdown_connections 2024-01-11T08:26:47.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.380+0000 7fd2ccbff700 1 -- 172.21.15.27:0/2327257667 wait complete. 2024-01-11T08:26:47.397 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-11T08:26:47.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.598+0000 7f6fdbfff700 1 Processor -- start 2024-01-11T08:26:47.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.598+0000 7f6fdbfff700 1 -- start start 2024-01-11T08:26:47.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.599+0000 7f6fdbfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fd415a1b0 0x7f6fd415a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:47.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.599+0000 7f6fdbfff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd40cf850 con 0x7f6fd40ccd70 2024-01-11T08:26:47.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.599+0000 7f6fdbfff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd40cec40 con 0x7f6fd415ae70 2024-01-11T08:26:47.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.599+0000 7f6fdbfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40cef20 con 0x7f6fd415a1b0 2024-01-11T08:26:47.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.600+0000 7f6fd959a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6fd40ccd70 0x7f6fd40cd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35944/0 (socket says 172.21.15.27:35944) 2024-01-11T08:26:47.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.600+0000 7f6fd959a700 1 -- 172.21.15.27:0/1143089113 learned_addr learned my addr 172.21.15.27:0/1143089113 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.600+0000 7f6fd9d9b700 1 --2- 172.21.15.27:0/1143089113 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fd415a1b0 0x7f6fd415a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.600+0000 7f6fd8d99700 1 -- 172.21.15.27:0/1143089113 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947619293 0 0) 0x7f6fd40cf850 con 0x7f6fd40ccd70 2024-01-11T08:26:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.600+0000 7f6fd8d99700 1 -- 172.21.15.27:0/1143089113 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fc0003570 con 0x7f6fd40ccd70 2024-01-11T08:26:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.600+0000 7f6fd8d99700 1 -- 172.21.15.27:0/1143089113 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174505330 0 0) 0x7f6fd40cec40 con 0x7f6fd415ae70 2024-01-11T08:26:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.600+0000 7f6fd8d99700 1 -- 172.21.15.27:0/1143089113 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd40cf850 con 0x7f6fd415ae70 2024-01-11T08:26:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.600+0000 7f6fd8d99700 1 -- 172.21.15.27:0/1143089113 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3866987949 0 0) 0x7f6fc0003570 con 0x7f6fd40ccd70 2024-01-11T08:26:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.600+0000 7f6fd8d99700 1 -- 172.21.15.27:0/1143089113 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fd415a1b0 msgr2=0x7f6fd415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.600+0000 7f6fd8d99700 1 --2- 172.21.15.27:0/1143089113 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fd415a1b0 0x7f6fd415a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.600+0000 7f6fd8d99700 1 -- 172.21.15.27:0/1143089113 >> v1:172.21.15.77:6789/0 conn(0x7f6fd415ae70 legacy=0x7f6fd415b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:47.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.600+0000 7f6fd8d99700 1 -- 172.21.15.27:0/1143089113 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd40cf200 con 0x7f6fd40ccd70 2024-01-11T08:26:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.600+0000 7f6fd8d99700 1 -- 172.21.15.27:0/1143089113 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f6fc4001470 con 0x7f6fd40ccd70 2024-01-11T08:26:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.601+0000 7f6fd8d99700 1 -- 172.21.15.27:0/1143089113 >> v1:172.21.15.27:6789/0 conn(0x7f6fd40ccd70 legacy=0x7f6fd40cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.601+0000 7f6fd8d99700 1 --2- 172.21.15.27:0/1143089113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc0003c20 0x7f6fc0004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.601+0000 7f6fd8d99700 1 --2- 172.21.15.27:0/1143089113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc0004810 0x7f6fc0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.601+0000 7f6fd8d99700 1 --2- 172.21.15.27:0/1143089113 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fd415a1b0 0x7f6fc00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.601+0000 7f6fd8d99700 1 -- 172.21.15.27:0/1143089113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40cef20 con 0x7f6fc0004810 2024-01-11T08:26:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.601+0000 7f6fd8d99700 1 -- 172.21.15.27:0/1143089113 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc000bb40 con 0x7f6fc0003c20 2024-01-11T08:26:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.601+0000 7f6fd8d99700 1 -- 172.21.15.27:0/1143089113 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc000bd00 con 0x7f6fd415a1b0 2024-01-11T08:26:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.601+0000 7f6fd959a700 1 --2- 172.21.15.27:0/1143089113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc0004810 0x7f6fc0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.601+0000 7f6fd9d9b700 1 --2- 172.21.15.27:0/1143089113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc0003c20 0x7f6fc0004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.601+0000 7f6fda59c700 1 --2- 172.21.15.27:0/1143089113 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fd415a1b0 0x7f6fc00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.601+0000 7f6fd959a700 1 -- 172.21.15.27:0/1143089113 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fd415a1b0 msgr2=0x7f6fc00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.601+0000 7f6fd959a700 1 --2- 172.21.15.27:0/1143089113 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fd415a1b0 0x7f6fc00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:47.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.601+0000 7f6fd959a700 1 -- 172.21.15.27:0/1143089113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc0003c20 msgr2=0x7f6fc0004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:47.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.601+0000 7f6fd959a700 1 --2- 172.21.15.27:0/1143089113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc0003c20 0x7f6fc0004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:47.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.601+0000 7f6fd959a700 1 -- 172.21.15.27:0/1143089113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc0003570 con 0x7f6fc0004810 2024-01-11T08:26:47.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.602+0000 7f6fd959a700 1 --2- 172.21.15.27:0/1143089113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc0004810 0x7f6fc0004c80 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f6fc400c2d0 tx=0x7f6fc4001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd8bff2aee0d930c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:47.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.602+0000 7f6fd8d99700 1 -- 172.21.15.27:0/1143089113 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc400cbc0 con 0x7f6fc0004810 2024-01-11T08:26:47.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.602+0000 7f6fd8d99700 1 -- 172.21.15.27:0/1143089113 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc400cd20 con 0x7f6fc0004810 2024-01-11T08:26:47.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.602+0000 7f6fd8d99700 1 -- 172.21.15.27:0/1143089113 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc4005b70 con 0x7f6fc0004810 2024-01-11T08:26:47.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.602+0000 7f6fdbfff700 1 -- 172.21.15.27:0/1143089113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc0004810 msgr2=0x7f6fc0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:47.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.602+0000 7f6fdbfff700 1 --2- 172.21.15.27:0/1143089113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc0004810 0x7f6fc0004c80 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f6fc400c2d0 tx=0x7f6fc4001e10 comp rx=0 tx=0).stop 2024-01-11T08:26:47.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.602+0000 7f6fda59c700 1 -- 172.21.15.27:0/1143089113 reap_dead start 2024-01-11T08:26:47.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.603+0000 7f6fdbfff700 1 -- 172.21.15.27:0/1143089113 shutdown_connections 2024-01-11T08:26:47.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.603+0000 7f6fdbfff700 1 -- 172.21.15.27:0/1143089113 >> 172.21.15.27:0/1143089113 conn(0x7f6fd406ed40 msgr2=0x7f6fd406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:47.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.603+0000 7f6fdbfff700 1 -- 172.21.15.27:0/1143089113 shutdown_connections 2024-01-11T08:26:47.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.603+0000 7f6fdbfff700 1 -- 172.21.15.27:0/1143089113 wait complete. 2024-01-11T08:26:47.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.604+0000 7f6fdbfff700 1 Processor -- start 2024-01-11T08:26:47.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.604+0000 7f6fdbfff700 1 -- start start 2024-01-11T08:26:47.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.605+0000 7f6fdbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc0004810 0x7f6fd40c8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:47.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.605+0000 7f6fdbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd40a7990 0x7f6fd4157d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:47.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.605+0000 7f6fdbfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fd4169b50 0x7f6fd40c5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.605+0000 7f6fdbfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40c9340 con 0x7f6fd40a7990 2024-01-11T08:26:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.605+0000 7f6fdbfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40a7e00 con 0x7f6fc0004810 2024-01-11T08:26:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.605+0000 7f6fdbfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40c8730 con 0x7f6fd4169b50 2024-01-11T08:26:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.606+0000 7f6fd959a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd40a7990 0x7f6fd4157d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.606+0000 7f6fd9d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc0004810 0x7f6fd40c8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.606+0000 7f6fd959a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd40a7990 0x7f6fd4157d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35574/0 (socket says 172.21.15.27:35574) 2024-01-11T08:26:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.606+0000 7f6fd959a700 1 -- 172.21.15.27:0/2131093649 learned_addr learned my addr 172.21.15.27:0/2131093649 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.606+0000 7f6fda59c700 1 --2- 172.21.15.27:0/2131093649 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fd4169b50 0x7f6fd40c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.606+0000 7f6fd959a700 1 -- 172.21.15.27:0/2131093649 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fd4169b50 msgr2=0x7f6fd40c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.606+0000 7f6fd959a700 1 --2- 172.21.15.27:0/2131093649 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fd4169b50 0x7f6fd40c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.606+0000 7f6fd959a700 1 -- 172.21.15.27:0/2131093649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc0004810 msgr2=0x7f6fd40c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.606+0000 7f6fd959a700 1 --2- 172.21.15.27:0/2131093649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc0004810 0x7f6fd40c8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.606+0000 7f6fd959a700 1 -- 172.21.15.27:0/2131093649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd40c8a10 con 0x7f6fd40a7990 2024-01-11T08:26:47.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.606+0000 7f6fda59c700 1 --2- 172.21.15.27:0/2131093649 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fd4169b50 0x7f6fd40c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:26:47.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.606+0000 7f6fd959a700 1 --2- 172.21.15.27:0/2131093649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd40a7990 0x7f6fd4157d20 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4011000 tx=0x7f6fc4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:47.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.607+0000 7f6fc9ffb700 1 -- 172.21.15.27:0/2131093649 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc4001720 con 0x7f6fd40a7990 2024-01-11T08:26:47.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.607+0000 7f6fc9ffb700 1 -- 172.21.15.27:0/2131093649 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc40070d0 con 0x7f6fd40a7990 2024-01-11T08:26:47.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.607+0000 7f6fc9ffb700 1 -- 172.21.15.27:0/2131093649 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc4007490 con 0x7f6fd40a7990 2024-01-11T08:26:47.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.607+0000 7f6fdbfff700 1 -- 172.21.15.27:0/2131093649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd40c8cf0 con 0x7f6fd40a7990 2024-01-11T08:26:47.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.607+0000 7f6fdbfff700 1 -- 172.21.15.27:0/2131093649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd4158260 con 0x7f6fd40a7990 2024-01-11T08:26:47.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.608+0000 7f6fdbfff700 1 -- 172.21.15.27:0/2131093649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd40c94b0 con 0x7f6fd40a7990 2024-01-11T08:26:47.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.609+0000 7f6fc9ffb700 1 -- 172.21.15.27:0/2131093649 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6fc4019030 con 0x7f6fd40a7990 2024-01-11T08:26:47.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.610+0000 7f6fc9ffb700 1 --2- 172.21.15.27:0/2131093649 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6fb8066f30 0x7f6fb80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:47.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.610+0000 7f6fc9ffb700 1 -- 172.21.15.27:0/2131093649 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f6fc4085bc0 con 0x7f6fd40a7990 2024-01-11T08:26:47.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.610+0000 7f6fd9d9b700 1 --2- 172.21.15.27:0/2131093649 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6fb8066f30 0x7f6fb80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:47.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.611+0000 7f6fd9d9b700 1 --2- 172.21.15.27:0/2131093649 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6fb8066f30 0x7f6fb80693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6fd40a70c0 tx=0x7f6fd00091a0 comp rx=0 tx=0).ready entity=mgr.6745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:47.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.614+0000 7f6fc9ffb700 1 -- 172.21.15.27:0/2131093649 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6fc4052d70 con 0x7f6fd40a7990 2024-01-11T08:26:47.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:47.872+0000 7f6fdbfff700 1 -- 172.21.15.27:0/2131093649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f6fd404c3c0 con 0x7f6fd40a7990 2024-01-11T08:26:48.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.063+0000 7f6fc9ffb700 1 -- 172.21.15.27:0/2131093649 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 47) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f6fc4053140 con 0x7f6fd40a7990 2024-01-11T08:26:48.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.064+0000 7f6fc9ffb700 1 -- 172.21.15.27:0/2131093649 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v47) v1 ==== 92+0+0 (secure 0 0 0) 0x7f6fc4056800 con 0x7f6fd40a7990 2024-01-11T08:26:48.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.067+0000 7f6fdbfff700 1 -- 172.21.15.27:0/2131093649 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6fb8066f30 msgr2=0x7f6fb80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:48.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.067+0000 7f6fdbfff700 1 --2- 172.21.15.27:0/2131093649 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6fb8066f30 0x7f6fb80693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6fd40a70c0 tx=0x7f6fd00091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.067+0000 7f6fdbfff700 1 -- 172.21.15.27:0/2131093649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd40a7990 msgr2=0x7f6fd4157d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:48.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.067+0000 7f6fdbfff700 1 --2- 172.21.15.27:0/2131093649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd40a7990 0x7f6fd4157d20 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4011000 tx=0x7f6fc4015000 comp rx=0 tx=0).stop 2024-01-11T08:26:48.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.068+0000 7f6fdbfff700 1 -- 172.21.15.27:0/2131093649 shutdown_connections 2024-01-11T08:26:48.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.068+0000 7f6fdbfff700 1 --2- 172.21.15.27:0/2131093649 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6fb8066f30 0x7f6fb80693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.068+0000 7f6fdbfff700 1 --2- 172.21.15.27:0/2131093649 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fd4169b50 0x7f6fd40c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.068+0000 7f6fdbfff700 1 --2- 172.21.15.27:0/2131093649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fc0004810 0x7f6fd40c8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.068+0000 7f6fdbfff700 1 --2- 172.21.15.27:0/2131093649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd40a7990 0x7f6fd4157d20 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.068+0000 7f6fdbfff700 1 -- 172.21.15.27:0/2131093649 >> 172.21.15.27:0/2131093649 conn(0x7f6fd406ed40 msgr2=0x7f6fd40b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:48.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.068+0000 7f6fdbfff700 1 -- 172.21.15.27:0/2131093649 shutdown_connections 2024-01-11T08:26:48.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.068+0000 7f6fdbfff700 1 -- 172.21.15.27:0/2131093649 wait complete. 2024-01-11T08:26:48.092 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-11T08:26:48.276 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:48.277+0000 7f90b82c11c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:26:48.290 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:48.290+0000 7ff4ffc5e1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:26:48.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.299+0000 7f86943ff700 1 Processor -- start 2024-01-11T08:26:48.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.300+0000 7f86943ff700 1 -- start start 2024-01-11T08:26:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.301+0000 7f86943ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f868c1577e0 0x7f868c0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.301+0000 7f86943ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f868c069540 0x7f868c0ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.301+0000 7f86943ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f868c0699d0 con 0x7f868c157410 2024-01-11T08:26:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.301+0000 7f86943ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f868c069cb0 con 0x7f868c1577e0 2024-01-11T08:26:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.301+0000 7f86943ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f868c069f90 con 0x7f868c069540 2024-01-11T08:26:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.301+0000 7f869199a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f868c1577e0 0x7f868c0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.301+0000 7f869199a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f868c1577e0 0x7f868c0ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35598/0 (socket says 172.21.15.27:35598) 2024-01-11T08:26:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.301+0000 7f869199a700 1 -- 172.21.15.27:0/1890746044 learned_addr learned my addr 172.21.15.27:0/1890746044 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.301+0000 7f869219b700 1 --1- 172.21.15.27:0/1890746044 >> v1:172.21.15.27:6789/0 conn(0x7f868c157410 0x7f868c0ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35966/0 (socket says 172.21.15.27:35966) 2024-01-11T08:26:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.301+0000 7f869299c700 1 --2- 172.21.15.27:0/1890746044 >> v2:172.21.15.77:3300/0 conn(0x7f868c069540 0x7f868c0ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.302+0000 7f869199a700 1 -- 172.21.15.27:0/1890746044 >> v2:172.21.15.77:3300/0 conn(0x7f868c069540 msgr2=0x7f868c0ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.302+0000 7f869199a700 1 --2- 172.21.15.27:0/1890746044 >> v2:172.21.15.77:3300/0 conn(0x7f868c069540 0x7f868c0ae540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.302+0000 7f869199a700 1 -- 172.21.15.27:0/1890746044 >> v1:172.21.15.27:6789/0 conn(0x7f868c157410 legacy=0x7f868c0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.302+0000 7f869199a700 1 -- 172.21.15.27:0/1890746044 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868c06a270 con 0x7f868c1577e0 2024-01-11T08:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.302+0000 7f8691199700 1 -- 172.21.15.27:0/1890746044 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959428711 0 0) 0x7f868c0699d0 con 0x7f868c157410 2024-01-11T08:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.302+0000 7f869199a700 1 --2- 172.21.15.27:0/1890746044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868c1577e0 0x7f868c0ae000 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f868c06d030 tx=0x7f867c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=793bdf12ae76f918 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:48.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.302+0000 7f8691199700 1 -- 172.21.15.27:0/1890746044 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c009e70 con 0x7f868c1577e0 2024-01-11T08:26:48.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.303+0000 7f8691199700 1 -- 172.21.15.27:0/1890746044 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867c00b860 con 0x7f868c1577e0 2024-01-11T08:26:48.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.303+0000 7f8691199700 1 -- 172.21.15.27:0/1890746044 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c00bc00 con 0x7f868c1577e0 2024-01-11T08:26:48.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.303+0000 7f86943ff700 1 -- 172.21.15.27:0/1890746044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868c1577e0 msgr2=0x7f868c0ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:48.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.303+0000 7f86943ff700 1 --2- 172.21.15.27:0/1890746044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868c1577e0 0x7f868c0ae000 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f868c06d030 tx=0x7f867c009250 comp rx=0 tx=0).stop 2024-01-11T08:26:48.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.303+0000 7f86943ff700 1 -- 172.21.15.27:0/1890746044 shutdown_connections 2024-01-11T08:26:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.303+0000 7f86943ff700 1 --2- 172.21.15.27:0/1890746044 >> v2:172.21.15.77:3300/0 conn(0x7f868c069540 0x7f868c0ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.303+0000 7f86943ff700 1 --2- 172.21.15.27:0/1890746044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868c1577e0 0x7f868c0ae000 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.303+0000 7f86943ff700 1 -- 172.21.15.27:0/1890746044 >> 172.21.15.27:0/1890746044 conn(0x7f868c0680a0 msgr2=0x7f868c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.303+0000 7f86943ff700 1 -- 172.21.15.27:0/1890746044 shutdown_connections 2024-01-11T08:26:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.303+0000 7f86943ff700 1 -- 172.21.15.27:0/1890746044 wait complete. 2024-01-11T08:26:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.305+0000 7f86943ff700 1 Processor -- start 2024-01-11T08:26:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.305+0000 7f86943ff700 1 -- start start 2024-01-11T08:26:48.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.306+0000 7f86943ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c069540 0x7f868c0c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.306+0000 7f86943ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868c157410 0x7f868c159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.306+0000 7f86943ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f868c1577e0 0x7f868c14b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.306+0000 7f86943ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868c159850 con 0x7f868c157410 2024-01-11T08:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.306+0000 7f86943ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868c15a690 con 0x7f868c069540 2024-01-11T08:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.306+0000 7f86943ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868c15ab20 con 0x7f868c1577e0 2024-01-11T08:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.306+0000 7f869199a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868c157410 0x7f868c159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.306+0000 7f869199a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868c157410 0x7f868c159310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35610/0 (socket says 172.21.15.27:35610) 2024-01-11T08:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.306+0000 7f869199a700 1 -- 172.21.15.27:0/2645640884 learned_addr learned my addr 172.21.15.27:0/2645640884 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.306+0000 7f869219b700 1 --2- 172.21.15.27:0/2645640884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c069540 0x7f868c0c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.307+0000 7f869299c700 1 --2- 172.21.15.27:0/2645640884 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f868c1577e0 0x7f868c14b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.307+0000 7f869199a700 1 -- 172.21.15.27:0/2645640884 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f868c1577e0 msgr2=0x7f868c14b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.307+0000 7f869199a700 1 --2- 172.21.15.27:0/2645640884 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f868c1577e0 0x7f868c14b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.307+0000 7f869199a700 1 -- 172.21.15.27:0/2645640884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c069540 msgr2=0x7f868c0c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.307+0000 7f869199a700 1 --2- 172.21.15.27:0/2645640884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c069540 0x7f868c0c3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.307+0000 7f869199a700 1 -- 172.21.15.27:0/2645640884 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c0094f0 con 0x7f868c157410 2024-01-11T08:26:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.307+0000 7f869299c700 1 --2- 172.21.15.27:0/2645640884 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f868c1577e0 0x7f868c14b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:26:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.307+0000 7f869199a700 1 --2- 172.21.15.27:0/2645640884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868c157410 0x7f868c159310 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f867c0094c0 tx=0x7f867c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.307+0000 7f86827fc700 1 -- 172.21.15.27:0/2645640884 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c00b9d0 con 0x7f868c157410 2024-01-11T08:26:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.307+0000 7f86827fc700 1 -- 172.21.15.27:0/2645640884 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867c00bb50 con 0x7f868c157410 2024-01-11T08:26:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.307+0000 7f86827fc700 1 -- 172.21.15.27:0/2645640884 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c025670 con 0x7f868c157410 2024-01-11T08:26:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.308+0000 7f86943ff700 1 -- 172.21.15.27:0/2645640884 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f868c14b700 con 0x7f868c157410 2024-01-11T08:26:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.308+0000 7f86943ff700 1 -- 172.21.15.27:0/2645640884 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f868c159d60 con 0x7f868c157410 2024-01-11T08:26:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.309+0000 7f86943ff700 1 -- 172.21.15.27:0/2645640884 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f868c15a800 con 0x7f868c157410 2024-01-11T08:26:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.310+0000 7f86827fc700 1 -- 172.21.15.27:0/2645640884 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 47) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f867c023070 con 0x7f868c157410 2024-01-11T08:26:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.311+0000 7f86827fc700 1 --2- 172.21.15.27:0/2645640884 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8670066f80 0x7f8670069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.311+0000 7f86827fc700 1 -- 172.21.15.27:0/2645640884 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f867c085b20 con 0x7f868c157410 2024-01-11T08:26:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.311+0000 7f869219b700 1 -- 172.21.15.27:0/2645640884 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8670066f80 msgr2=0x7f8670069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:26:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.311+0000 7f869219b700 1 --2- 172.21.15.27:0/2645640884 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8670066f80 0x7f8670069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:26:48.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.315+0000 7f86827fc700 1 -- 172.21.15.27:0/2645640884 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f867c052cd0 con 0x7f868c157410 2024-01-11T08:26:48.454 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:48.455+0000 7f90b82c11c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:26:48.489 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:48.489+0000 7ff4ffc5e1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:26:48.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.512+0000 7f869219b700 1 -- 172.21.15.27:0/2645640884 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8670066f80 msgr2=0x7f8670069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:26:48.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.512+0000 7f869219b700 1 --2- 172.21.15.27:0/2645640884 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8670066f80 0x7f8670069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:26:48.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.561+0000 7f86943ff700 1 -- 172.21.15.27:0/2645640884 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f868c04c3c0 con 0x7f868c157410 2024-01-11T08:26:48.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.565+0000 7f86827fc700 1 -- 172.21.15.27:0/2645640884 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v35)=0 v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7f867c0562f0 con 0x7f868c157410 2024-01-11T08:26:48.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.568+0000 7f86943ff700 1 -- 172.21.15.27:0/2645640884 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8670066f80 msgr2=0x7f8670069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:26:48.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.568+0000 7f86943ff700 1 --2- 172.21.15.27:0/2645640884 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8670066f80 0x7f8670069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.568+0000 7f86943ff700 1 -- 172.21.15.27:0/2645640884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868c157410 msgr2=0x7f868c159310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:48.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.568+0000 7f86943ff700 1 --2- 172.21.15.27:0/2645640884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868c157410 0x7f868c159310 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f867c0094c0 tx=0x7f867c017000 comp rx=0 tx=0).stop 2024-01-11T08:26:48.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.568+0000 7f86943ff700 1 -- 172.21.15.27:0/2645640884 shutdown_connections 2024-01-11T08:26:48.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.568+0000 7f86943ff700 1 --2- 172.21.15.27:0/2645640884 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8670066f80 0x7f8670069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.568+0000 7f86943ff700 1 --2- 172.21.15.27:0/2645640884 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f868c1577e0 0x7f868c14b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.568+0000 7f86943ff700 1 --2- 172.21.15.27:0/2645640884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f868c069540 0x7f868c0c3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.568+0000 7f86943ff700 1 --2- 172.21.15.27:0/2645640884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868c157410 0x7f868c159310 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.568+0000 7f86943ff700 1 -- 172.21.15.27:0/2645640884 >> 172.21.15.27:0/2645640884 conn(0x7f868c0680a0 msgr2=0x7f868c0ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:48.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.568+0000 7f86943ff700 1 -- 172.21.15.27:0/2645640884 shutdown_connections 2024-01-11T08:26:48.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.568+0000 7f86943ff700 1 -- 172.21.15.27:0/2645640884 wait complete. 2024-01-11T08:26:48.587 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-11T08:26:48.768 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:48.768+0000 7f90b82c11c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:26:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.792+0000 7f7f021ff700 1 Processor -- start 2024-01-11T08:26:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.792+0000 7f7f021ff700 1 -- start start 2024-01-11T08:26:48.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.794+0000 7f7f021ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7efc067c30 0x7f7efc0cae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:48.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.794+0000 7f7f021ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7efc0cb3d0 0x7f7efc0c7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:48.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.794+0000 7f7f021ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7efc06c3c0 con 0x7f7efc067860 2024-01-11T08:26:48.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.794+0000 7f7f021ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7efc0cb9c0 con 0x7f7efc0cb3d0 2024-01-11T08:26:48.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.794+0000 7f7f021ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc06f650 con 0x7f7efc067c30 2024-01-11T08:26:48.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.794+0000 7f7efb7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7efc067860 0x7f7efc06c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35980/0 (socket says 172.21.15.27:35980) 2024-01-11T08:26:48.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.794+0000 7f7efb7fe700 1 -- 172.21.15.27:0/3209602925 learned_addr learned my addr 172.21.15.27:0/3209602925 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:48.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.794+0000 7f7efaffd700 1 --2- 172.21.15.27:0/3209602925 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7efc067c30 0x7f7efc0cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:48.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.794+0000 7f7efbfff700 1 --2- 172.21.15.27:0/3209602925 >> v2:172.21.15.77:3300/0 conn(0x7f7efc0cb3d0 0x7f7efc0c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:48.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.795+0000 7f7efa7fc700 1 -- 172.21.15.27:0/3209602925 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832557919 0 0) 0x7f7efc06c3c0 con 0x7f7efc067860 2024-01-11T08:26:48.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.795+0000 7f7efa7fc700 1 -- 172.21.15.27:0/3209602925 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ee4003570 con 0x7f7efc067860 2024-01-11T08:26:48.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.795+0000 7f7efbfff700 1 -- 172.21.15.27:0/3209602925 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7efc067c30 msgr2=0x7f7efc0cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:48.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.795+0000 7f7efbfff700 1 --2- 172.21.15.27:0/3209602925 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7efc067c30 0x7f7efc0cae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.795+0000 7f7efbfff700 1 -- 172.21.15.27:0/3209602925 >> v1:172.21.15.27:6789/0 conn(0x7f7efc067860 legacy=0x7f7efc06c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:48.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.795+0000 7f7efbfff700 1 -- 172.21.15.27:0/3209602925 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7efc06ba90 con 0x7f7efc0cb3d0 2024-01-11T08:26:48.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.795+0000 7f7efbfff700 1 --2- 172.21.15.27:0/3209602925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7efc0cb3d0 0x7f7efc0c7900 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009e60 tx=0x7f7ef4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57557aa3fc78bf96 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:48.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.795+0000 7f7efa7fc700 1 -- 172.21.15.27:0/3209602925 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef400e070 con 0x7f7efc0cb3d0 2024-01-11T08:26:48.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.795+0000 7f7efa7fc700 1 -- 172.21.15.27:0/3209602925 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef400bab0 con 0x7f7efc0cb3d0 2024-01-11T08:26:48.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.795+0000 7f7efa7fc700 1 -- 172.21.15.27:0/3209602925 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef400be30 con 0x7f7efc0cb3d0 2024-01-11T08:26:48.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.796+0000 7f7f021ff700 1 -- 172.21.15.27:0/3209602925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7efc0cb3d0 msgr2=0x7f7efc0c7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:48.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.796+0000 7f7f021ff700 1 --2- 172.21.15.27:0/3209602925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7efc0cb3d0 0x7f7efc0c7900 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009e60 tx=0x7f7ef4009780 comp rx=0 tx=0).stop 2024-01-11T08:26:48.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.796+0000 7f7f021ff700 1 -- 172.21.15.27:0/3209602925 shutdown_connections 2024-01-11T08:26:48.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.796+0000 7f7f021ff700 1 --2- 172.21.15.27:0/3209602925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7efc0cb3d0 0x7f7efc0c7900 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.796+0000 7f7f021ff700 1 --2- 172.21.15.27:0/3209602925 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7efc067c30 0x7f7efc0cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.796+0000 7f7f021ff700 1 -- 172.21.15.27:0/3209602925 >> 172.21.15.27:0/3209602925 conn(0x7f7efc06b1f0 msgr2=0x7f7efc0cd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:48.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.796+0000 7f7f021ff700 1 -- 172.21.15.27:0/3209602925 shutdown_connections 2024-01-11T08:26:48.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.796+0000 7f7f021ff700 1 -- 172.21.15.27:0/3209602925 wait complete. 2024-01-11T08:26:48.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.798+0000 7f7f021ff700 1 Processor -- start 2024-01-11T08:26:48.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.798+0000 7f7f021ff700 1 -- start start 2024-01-11T08:26:48.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.799+0000 7f7f021ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7efc067860 0x7f7efc13b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:48.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.799+0000 7f7f021ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7efc067c30 0x7f7efc158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:48.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.799+0000 7f7f021ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7efc0cb3d0 0x7f7efc15f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:48.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.799+0000 7f7f021ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc0b0060 con 0x7f7efc067c30 2024-01-11T08:26:48.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.799+0000 7f7f021ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc13bbd0 con 0x7f7efc0cb3d0 2024-01-11T08:26:48.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.799+0000 7f7f021ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc0aeaf0 con 0x7f7efc067860 2024-01-11T08:26:48.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.799+0000 7f7efaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7efc067c30 0x7f7efc158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:48.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.799+0000 7f7efaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7efc067c30 0x7f7efc158ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35618/0 (socket says 172.21.15.27:35618) 2024-01-11T08:26:48.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.799+0000 7f7efaffd700 1 -- 172.21.15.27:0/2138743610 learned_addr learned my addr 172.21.15.27:0/2138743610 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:48.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.799+0000 7f7efbfff700 1 --2- 172.21.15.27:0/2138743610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7efc0cb3d0 0x7f7efc15f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:48.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.799+0000 7f7efb7fe700 1 --2- 172.21.15.27:0/2138743610 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7efc067860 0x7f7efc13b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:48.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.800+0000 7f7efaffd700 1 -- 172.21.15.27:0/2138743610 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7efc067860 msgr2=0x7f7efc13b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:48.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.800+0000 7f7efaffd700 1 --2- 172.21.15.27:0/2138743610 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7efc067860 0x7f7efc13b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.800+0000 7f7efaffd700 1 -- 172.21.15.27:0/2138743610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7efc0cb3d0 msgr2=0x7f7efc15f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:48.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.800+0000 7f7efaffd700 1 --2- 172.21.15.27:0/2138743610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7efc0cb3d0 0x7f7efc15f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:48.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.800+0000 7f7efaffd700 1 -- 172.21.15.27:0/2138743610 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef4009a20 con 0x7f7efc067c30 2024-01-11T08:26:48.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.800+0000 7f7efaffd700 1 --2- 172.21.15.27:0/2138743610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7efc067c30 0x7f7efc158ee0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000ec80 tx=0x7f7ef00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:48.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.800+0000 7f7ee37fe700 1 -- 172.21.15.27:0/2138743610 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef000c850 con 0x7f7efc067c30 2024-01-11T08:26:48.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.800+0000 7f7ee37fe700 1 -- 172.21.15.27:0/2138743610 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef000c9b0 con 0x7f7efc067c30 2024-01-11T08:26:48.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.800+0000 7f7ee37fe700 1 -- 172.21.15.27:0/2138743610 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef000cd50 con 0x7f7efc067c30 2024-01-11T08:26:48.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.800+0000 7f7f021ff700 1 -- 172.21.15.27:0/2138743610 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7efc0aedd0 con 0x7f7efc067c30 2024-01-11T08:26:48.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.800+0000 7f7f021ff700 1 -- 172.21.15.27:0/2138743610 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7efc0b0590 con 0x7f7efc067c30 2024-01-11T08:26:48.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.802+0000 7f7f021ff700 1 -- 172.21.15.27:0/2138743610 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7efc04c3c0 con 0x7f7efc067c30 2024-01-11T08:26:48.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.803+0000 7f7ee37fe700 1 -- 172.21.15.27:0/2138743610 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 47) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f7ef0013680 con 0x7f7efc067c30 2024-01-11T08:26:48.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.803+0000 7f7ee37fe700 1 --2- 172.21.15.27:0/2138743610 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7ee8066f80 0x7f7ee8069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:48.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.804+0000 7f7ee37fe700 1 -- 172.21.15.27:0/2138743610 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f7ef00847e0 con 0x7f7efc067c30 2024-01-11T08:26:48.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.804+0000 7f7efb7fe700 1 -- 172.21.15.27:0/2138743610 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7ee8066f80 msgr2=0x7f7ee8069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:26:48.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.804+0000 7f7efb7fe700 1 --2- 172.21.15.27:0/2138743610 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7ee8066f80 0x7f7ee8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:26:48.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:48.808+0000 7f7ee37fe700 1 -- 172.21.15.27:0/2138743610 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7ef0051a30 con 0x7f7efc067c30 2024-01-11T08:26:48.837 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:48.837+0000 7ff4ffc5e1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:26:49.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.005+0000 7f7efb7fe700 1 -- 172.21.15.27:0/2138743610 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7ee8066f80 msgr2=0x7f7ee8069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:26:49.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.005+0000 7f7efb7fe700 1 --2- 172.21.15.27:0/2138743610 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7ee8066f80 0x7f7ee8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:26:49.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.013+0000 7f7f021ff700 1 -- 172.21.15.27:0/2138743610 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f7efc06bd70 con 0x7f7efc067c30 2024-01-11T08:26:49.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.017+0000 7f7ee37fe700 1 -- 172.21.15.27:0/2138743610 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v36)=0 v36) v1 ==== 133+0+0 (secure 0 0 0) 0x7f7ef0055050 con 0x7f7efc067c30 2024-01-11T08:26:49.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.020+0000 7f7f021ff700 1 -- 172.21.15.27:0/2138743610 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7ee8066f80 msgr2=0x7f7ee8069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:26:49.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.020+0000 7f7f021ff700 1 --2- 172.21.15.27:0/2138743610 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7ee8066f80 0x7f7ee8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:49.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.020+0000 7f7f021ff700 1 -- 172.21.15.27:0/2138743610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7efc067c30 msgr2=0x7f7efc158ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:49.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.020+0000 7f7f021ff700 1 --2- 172.21.15.27:0/2138743610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7efc067c30 0x7f7efc158ee0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000ec80 tx=0x7f7ef00091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:49.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.020+0000 7f7f021ff700 1 -- 172.21.15.27:0/2138743610 shutdown_connections 2024-01-11T08:26:49.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.020+0000 7f7f021ff700 1 --2- 172.21.15.27:0/2138743610 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7ee8066f80 0x7f7ee8069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:49.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.020+0000 7f7f021ff700 1 --2- 172.21.15.27:0/2138743610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7efc0cb3d0 0x7f7efc15f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:49.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.020+0000 7f7f021ff700 1 --2- 172.21.15.27:0/2138743610 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7efc067860 0x7f7efc13b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:49.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.020+0000 7f7f021ff700 1 --2- 172.21.15.27:0/2138743610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7efc067c30 0x7f7efc158ee0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:49.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.020+0000 7f7f021ff700 1 -- 172.21.15.27:0/2138743610 >> 172.21.15.27:0/2138743610 conn(0x7f7efc06b1f0 msgr2=0x7f7efc06e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:49.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.020+0000 7f7f021ff700 1 -- 172.21.15.27:0/2138743610 shutdown_connections 2024-01-11T08:26:49.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.020+0000 7f7f021ff700 1 -- 172.21.15.27:0/2138743610 wait complete. 2024-01-11T08:26:49.039 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /bad-path 1m 2024-01-11T08:26:49.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.245+0000 7fd0cedff700 1 Processor -- start 2024-01-11T08:26:49.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.245+0000 7fd0cedff700 1 -- start start 2024-01-11T08:26:49.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.246+0000 7fd0cedff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd0c8155c50 0x7fd0c80aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:49.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.246+0000 7fd0cedff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0c80694c0 0x7fd0c8069910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:49.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.246+0000 7fd0cedff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c806e480 con 0x7fd0c80c6de0 2024-01-11T08:26:49.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.246+0000 7fd0cedff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0c80cb5a0 con 0x7fd0c8155c50 2024-01-11T08:26:49.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.246+0000 7fd0cedff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c806df00 con 0x7fd0c80694c0 2024-01-11T08:26:49.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.247+0000 7fd0ccb9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd0c8155c50 0x7fd0c80aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:49.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.247+0000 7fd0c7fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0c80694c0 0x7fd0c8069910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:49.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.247+0000 7fd0c7fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0c80694c0 0x7fd0c8069910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:41646/0 (socket says 172.21.15.27:41646) 2024-01-11T08:26:49.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.247+0000 7fd0c7fff700 1 -- 172.21.15.27:0/3212631038 learned_addr learned my addr 172.21.15.27:0/3212631038 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:49.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.247+0000 7fd0ccb9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd0c8155c50 0x7fd0c80aef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:54874/0 (socket says 172.21.15.27:54874) 2024-01-11T08:26:49.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.247+0000 7fd0cd39c700 1 --1- 172.21.15.27:0/3212631038 >> v1:172.21.15.77:6789/0 conn(0x7fd0c80c6de0 0x7fd0c80c7260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:38760/0 (socket says 172.21.15.27:38760) 2024-01-11T08:26:49.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.247+0000 7fd0c7fff700 1 -- 172.21.15.27:0/3212631038 >> v2:172.21.15.77:3300/0 conn(0x7fd0c8155c50 msgr2=0x7fd0c80aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:49.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.247+0000 7fd0c7fff700 1 --2- 172.21.15.27:0/3212631038 >> v2:172.21.15.77:3300/0 conn(0x7fd0c8155c50 0x7fd0c80aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:49.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.247+0000 7fd0c7fff700 1 -- 172.21.15.27:0/3212631038 >> v1:172.21.15.77:6789/0 conn(0x7fd0c80c6de0 legacy=0x7fd0c80c7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:49.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.247+0000 7fd0c7fff700 1 -- 172.21.15.27:0/3212631038 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c8069e50 con 0x7fd0c80694c0 2024-01-11T08:26:49.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.248+0000 7fd0c7fff700 1 --2- 172.21.15.27:0/3212631038 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0c80694c0 0x7fd0c8069910 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400bc50 tx=0x7fd0b400b4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b098d61b41c0b8c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:49.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.248+0000 7fd0c77fe700 1 -- 172.21.15.27:0/3212631038 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b400dc80 con 0x7fd0c80694c0 2024-01-11T08:26:49.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.248+0000 7fd0c77fe700 1 -- 172.21.15.27:0/3212631038 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b400dde0 con 0x7fd0c80694c0 2024-01-11T08:26:49.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.248+0000 7fd0c77fe700 1 -- 172.21.15.27:0/3212631038 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b40196b0 con 0x7fd0c80694c0 2024-01-11T08:26:49.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.249+0000 7fd0cedff700 1 -- 172.21.15.27:0/3212631038 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0c80694c0 msgr2=0x7fd0c8069910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:49.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.249+0000 7fd0cedff700 1 --2- 172.21.15.27:0/3212631038 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0c80694c0 0x7fd0c8069910 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400bc50 tx=0x7fd0b400b4f0 comp rx=0 tx=0).stop 2024-01-11T08:26:49.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.249+0000 7fd0cedff700 1 -- 172.21.15.27:0/3212631038 shutdown_connections 2024-01-11T08:26:49.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.249+0000 7fd0cedff700 1 --2- 172.21.15.27:0/3212631038 >> v2:172.21.15.77:3300/0 conn(0x7fd0c8155c50 0x7fd0c80aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:49.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.249+0000 7fd0cedff700 1 --2- 172.21.15.27:0/3212631038 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0c80694c0 0x7fd0c8069910 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:49.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.249+0000 7fd0cedff700 1 -- 172.21.15.27:0/3212631038 >> 172.21.15.27:0/3212631038 conn(0x7fd0c80680a0 msgr2=0x7fd0c80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:49.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.249+0000 7fd0cedff700 1 -- 172.21.15.27:0/3212631038 shutdown_connections 2024-01-11T08:26:49.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.249+0000 7fd0cedff700 1 -- 172.21.15.27:0/3212631038 wait complete. 2024-01-11T08:26:49.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.250+0000 7fd0cedff700 1 Processor -- start 2024-01-11T08:26:49.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.250+0000 7fd0cedff700 1 -- start start 2024-01-11T08:26:49.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.251+0000 7fd0cedff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0c80694c0 0x7fd0c8158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:49.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.251+0000 7fd0cedff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0c80c6de0 0x7fd0c8141b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:49.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.251+0000 7fd0cedff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c8155c50 0x7fd0c80b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:49.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.251+0000 7fd0cedff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c81592c0 con 0x7fd0c8155c50 2024-01-11T08:26:49.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.251+0000 7fd0cedff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c81586b0 con 0x7fd0c80c6de0 2024-01-11T08:26:49.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.251+0000 7fd0cedff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c8158990 con 0x7fd0c80694c0 2024-01-11T08:26:49.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.252+0000 7fd0ccb9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0c80694c0 0x7fd0c8158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:49.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.252+0000 7fd0ccb9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0c80694c0 0x7fd0c8158080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:41652/0 (socket says 172.21.15.27:41652) 2024-01-11T08:26:49.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.252+0000 7fd0ccb9b700 1 -- 172.21.15.27:0/3462124322 learned_addr learned my addr 172.21.15.27:0/3462124322 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:49.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.252+0000 7fd0cd39c700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c8155c50 0x7fd0c80b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:49.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.252+0000 7fd0c7fff700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0c80c6de0 0x7fd0c8141b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:49.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.252+0000 7fd0ccb9b700 1 -- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0c80c6de0 msgr2=0x7fd0c8141b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:49.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.252+0000 7fd0ccb9b700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0c80c6de0 0x7fd0c8141b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:49.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.252+0000 7fd0ccb9b700 1 -- 172.21.15.27:0/3462124322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c8155c50 msgr2=0x7fd0c80b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:49.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.252+0000 7fd0ccb9b700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c8155c50 0x7fd0c80b05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:49.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.252+0000 7fd0ccb9b700 1 -- 172.21.15.27:0/3462124322 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b400b790 con 0x7fd0c80694c0 2024-01-11T08:26:49.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.252+0000 7fd0ccb9b700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0c80694c0 0x7fd0c8158080 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc00ea00 tx=0x7fd0bc009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:49.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.253+0000 7fd0c4ff9700 1 -- 172.21.15.27:0/3462124322 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc00c6a0 con 0x7fd0c80694c0 2024-01-11T08:26:49.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.253+0000 7fd0c4ff9700 1 -- 172.21.15.27:0/3462124322 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0bc00c800 con 0x7fd0c80694c0 2024-01-11T08:26:49.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.253+0000 7fd0c4ff9700 1 -- 172.21.15.27:0/3462124322 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc00cb80 con 0x7fd0c80694c0 2024-01-11T08:26:49.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.253+0000 7fd0cedff700 1 -- 172.21.15.27:0/3462124322 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c8158c70 con 0x7fd0c80694c0 2024-01-11T08:26:49.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.253+0000 7fd0cedff700 1 -- 172.21.15.27:0/3462124322 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c80b1aa0 con 0x7fd0c80694c0 2024-01-11T08:26:49.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.254+0000 7fd0cedff700 1 -- 172.21.15.27:0/3462124322 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c804c3c0 con 0x7fd0c80694c0 2024-01-11T08:26:49.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.255+0000 7fd0c4ff9700 1 -- 172.21.15.27:0/3462124322 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 47) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fd0bc013470 con 0x7fd0c80694c0 2024-01-11T08:26:49.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.256+0000 7fd0c4ff9700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0ac066f80 0x7fd0ac069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:49.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.256+0000 7fd0c4ff9700 1 -- 172.21.15.27:0/3462124322 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5733+0+0 (secure 0 0 0) 0x7fd0bc084b80 con 0x7fd0c80694c0 2024-01-11T08:26:49.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.256+0000 7fd0c7fff700 1 -- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0ac066f80 msgr2=0x7fd0ac069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:26:49.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.256+0000 7fd0c7fff700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0ac066f80 0x7fd0ac069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:26:49.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.260+0000 7fd0c4ff9700 1 -- 172.21.15.27:0/3462124322 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd0bc051d30 con 0x7fd0c80694c0 2024-01-11T08:26:49.295 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:49.296+0000 7f90b82c11c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:26:49.411 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:49.411+0000 7ff4ffc5e1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:26:49.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.457+0000 7fd0c7fff700 1 -- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0ac066f80 msgr2=0x7fd0ac069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:26:49.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.457+0000 7fd0c7fff700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0ac066f80 0x7fd0ac069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:26:49.466 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:49.467+0000 7f90b82c11c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:26:49.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.491+0000 7fd0cedff700 1 -- 172.21.15.27:0/3462124322 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0c80af080 con 0x7fd0ac066f80 2024-01-11T08:26:49.591 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:49.591+0000 7ff4ffc5e1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:26:49.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.858+0000 7fd0c7fff700 1 -- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0ac066f80 msgr2=0x7fd0ac069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:26:49.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:49.858+0000 7fd0c7fff700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0ac066f80 0x7fd0ac069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-11T08:26:49.883 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:49.883+0000 7f90b82c11c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:26:50.030 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:50.030+0000 7ff4ffc5e1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:26:50.657 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:50.657+0000 7f90b82c11c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:26:50.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:50.660+0000 7fd0c7fff700 1 -- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0ac066f80 msgr2=0x7fd0ac069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:26:50.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:50.660+0000 7fd0c7fff700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0ac066f80 0x7fd0ac069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-11T08:26:50.747 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:50.747+0000 7f90b82c11c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:26:50.851 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:50.852+0000 7ff4ffc5e1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:26:50.903 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:50.904+0000 7f90b82c11c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:26:50.944 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:50.946+0000 7ff4ffc5e1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:26:50.991 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:50.991+0000 7f90b82c11c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:26:51.109 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:51.110+0000 7ff4ffc5e1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:26:51.202 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:51.203+0000 7ff4ffc5e1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:26:51.207 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:51.207+0000 7f90b82c11c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:26:51.303 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:51.303+0000 7f90b82c11c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:26:51.398 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:51.398+0000 7f90b82c11c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:26:51.428 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:51.429+0000 7ff4ffc5e1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:26:51.492 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:51.493+0000 7f90b82c11c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:26:51.529 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:51.530+0000 7ff4ffc5e1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:26:51.630 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:51.631+0000 7ff4ffc5e1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:26:51.681 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:51.681+0000 7f90b82c11c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:26:51.731 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:51.732+0000 7ff4ffc5e1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:26:51.778 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:51.778+0000 7f90b82c11c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:26:51.930 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:51.931+0000 7ff4ffc5e1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:26:52.034 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:52.034+0000 7ff4ffc5e1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:26:52.120 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:52.120+0000 7f90b82c11c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:26:52.209 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:52.209+0000 7f90b82c11c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:26:52.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:52.262+0000 7fd0c7fff700 1 -- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0ac066f80 msgr2=0x7fd0ac069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:26:52.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:52.262+0000 7fd0c7fff700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0ac066f80 0x7fd0ac069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-11T08:26:52.393 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:52.394+0000 7ff4ffc5e1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:26:52.487 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:52.488+0000 7ff4ffc5e1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:26:52.699 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:26:52.704 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:26:52.747 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:52.747+0000 7f90b82c11c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:26:52.937 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:52.937+0000 7f90b82c11c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:26:53.063 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:53.064+0000 7ff4ffc5e1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:26:53.261 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:53.262+0000 7ff4ffc5e1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:26:53.314 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:53.314+0000 7f90b82c11c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:26:53.655 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:53.656+0000 7ff4ffc5e1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:26:53.733 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:53.733+0000 7f90b82c11c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:26:53.912 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:53.912+0000 7f90b82c11c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:26:54.101 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:54.101+0000 7f90b82c11c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:26:54.104 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:54.105+0000 7ff4ffc5e1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:26:54.210 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:54.211+0000 7f90b82c11c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:26:54.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:54.223+0000 7fd0c4ff9700 1 -- 172.21.15.27:0/3462124322 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mgrmap(e 48) v1 ==== 84470+0+0 (secure 0 0 0) 0x7fd0bc052100 con 0x7fd0c80694c0 2024-01-11T08:26:54.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:54.224+0000 7fd0c4ff9700 1 -- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0ac066f80 msgr2=0x7fd0ac069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:26:54.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:54.224+0000 7fd0c4ff9700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0ac066f80 0x7fd0ac069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:54.309 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:54.310+0000 7ff4ffc5e1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:26:54.346 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:54.345+0000 7f902d8b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:54.346 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:54.345+0000 7f902d8b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:54.346 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:54.345+0000 7f902d8b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:54.346 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:54.345+0000 7f902d8b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:54.346 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:54.345+0000 7f902d8b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:54.346 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:54.346+0000 7f90308bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:54.346 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:54.346+0000 7f90308bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:54.346 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:54.346+0000 7f90308bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:54.347 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:54.346+0000 7f90308bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:54.347 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:26:54.346+0000 7f90308bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:26:54.511 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:54.512+0000 7ff4ffc5e1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:26:54.626 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:26:54.627+0000 7ff4ffc5e1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:26:55.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.231+0000 7fd0c4ff9700 1 -- 172.21.15.27:0/3462124322 <== mon.2 v2:172.21.15.77:3301/0 8 ==== mgrmap(e 49) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fd0bc0609c0 con 0x7fd0c80694c0 2024-01-11T08:26:55.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.231+0000 7fd0c4ff9700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0ac066f80 0x7fd0ac0693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:55.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.231+0000 7fd0c4ff9700 1 -- 172.21.15.27:0/3462124322 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0c80af080 con 0x7fd0ac066f80 2024-01-11T08:26:55.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.232+0000 7fd0c7fff700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0ac066f80 0x7fd0ac0693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:55.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.233+0000 7fd0c7fff700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0ac066f80 0x7fd0ac0693b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400bc50 tx=0x7fd0b401b000 comp rx=0 tx=0).ready entity=mgr.7066 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:55.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.300+0000 7fd0c4ff9700 1 -- 172.21.15.27:0/3462124322 <== mgr.7066 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fd0c80af080 con 0x7fd0ac066f80 2024-01-11T08:26:55.299 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path /bad-path 2024-01-11T08:26:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.302+0000 7fd0cedff700 1 -- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0ac066f80 msgr2=0x7fd0ac0693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.302+0000 7fd0cedff700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0ac066f80 0x7fd0ac0693b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400bc50 tx=0x7fd0b401b000 comp rx=0 tx=0).stop 2024-01-11T08:26:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.302+0000 7fd0cedff700 1 -- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0c80694c0 msgr2=0x7fd0c8158080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.302+0000 7fd0cedff700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0c80694c0 0x7fd0c8158080 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc00ea00 tx=0x7fd0bc009120 comp rx=0 tx=0).stop 2024-01-11T08:26:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.302+0000 7fd0cedff700 1 -- 172.21.15.27:0/3462124322 shutdown_connections 2024-01-11T08:26:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.302+0000 7fd0cedff700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0ac066f80 0x7fd0ac0693b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.302+0000 7fd0cedff700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c8155c50 0x7fd0c80b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.302+0000 7fd0cedff700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0c80694c0 0x7fd0c8158080 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.302+0000 7fd0cedff700 1 --2- 172.21.15.27:0/3462124322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0c80c6de0 0x7fd0c8141b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.302+0000 7fd0cedff700 1 -- 172.21.15.27:0/3462124322 >> 172.21.15.27:0/3462124322 conn(0x7fd0c80680a0 msgr2=0x7fd0c80c9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.303+0000 7fd0cedff700 1 -- 172.21.15.27:0/3462124322 shutdown_connections 2024-01-11T08:26:55.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.303+0000 7fd0cedff700 1 -- 172.21.15.27:0/3462124322 wait complete. 2024-01-11T08:26:55.320 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2024-01-11T08:26:55.321 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2024-01-11T08:26:55.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.523+0000 7f3ef91ff700 1 Processor -- start 2024-01-11T08:26:55.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.523+0000 7f3ef91ff700 1 -- start start 2024-01-11T08:26:55.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.525+0000 7f3ef91ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3ef4143190 0x7f3ef40cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:55.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.525+0000 7f3ef91ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3ef40cb4b0 0x7f3ef40cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:55.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.525+0000 7f3ef91ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ef415d320 con 0x7f3ef4142dc0 2024-01-11T08:26:55.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.525+0000 7f3ef91ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ef415df30 con 0x7f3ef40cb4b0 2024-01-11T08:26:55.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.525+0000 7f3ef91ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef415d590 con 0x7f3ef4143190 2024-01-11T08:26:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.525+0000 7f3ef359e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3ef40cb4b0 0x7f3ef40cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.525+0000 7f3ef2d9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3ef4142dc0 0x7f3ef40cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:44872/0 (socket says 172.21.15.27:44872) 2024-01-11T08:26:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.525+0000 7f3ef259c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3ef4143190 0x7f3ef40cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.525+0000 7f3ef2d9d700 1 -- 172.21.15.27:0/29966843 learned_addr learned my addr 172.21.15.27:0/29966843 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:26:55.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.526+0000 7f3ef359e700 1 -- 172.21.15.27:0/29966843 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3ef4143190 msgr2=0x7f3ef40cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:26:55.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.526+0000 7f3ef359e700 1 --2- 172.21.15.27:0/29966843 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3ef4143190 0x7f3ef40cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:55.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.526+0000 7f3ef359e700 1 -- 172.21.15.27:0/29966843 >> v1:172.21.15.77:6789/0 conn(0x7f3ef4142dc0 legacy=0x7f3ef40cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:55.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.526+0000 7f3ef359e700 1 -- 172.21.15.27:0/29966843 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef415d870 con 0x7f3ef40cb4b0 2024-01-11T08:26:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.526+0000 7f3ef259c700 1 --2- 172.21.15.27:0/29966843 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3ef4143190 0x7f3ef40cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:26:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.526+0000 7f3ef1d9b700 1 -- 172.21.15.27:0/29966843 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19697831 0 0) 0x7f3ef415d320 con 0x7f3ef4142dc0 2024-01-11T08:26:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.526+0000 7f3ef359e700 1 --2- 172.21.15.27:0/29966843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ef40cb4b0 0x7f3ef40cc560 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3eec009e60 tx=0x7f3eec009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a18c35e7721f7cab server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.526+0000 7f3ef1d9b700 1 -- 172.21.15.27:0/29966843 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eec00e070 con 0x7f3ef40cb4b0 2024-01-11T08:26:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.527+0000 7f3ef1d9b700 1 -- 172.21.15.27:0/29966843 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eec00bb60 con 0x7f3ef40cb4b0 2024-01-11T08:26:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.527+0000 7f3ef1d9b700 1 -- 172.21.15.27:0/29966843 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eec012430 con 0x7f3ef40cb4b0 2024-01-11T08:26:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.527+0000 7f3ef91ff700 1 -- 172.21.15.27:0/29966843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ef40cb4b0 msgr2=0x7f3ef40cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.527+0000 7f3ef91ff700 1 --2- 172.21.15.27:0/29966843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ef40cb4b0 0x7f3ef40cc560 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3eec009e60 tx=0x7f3eec009780 comp rx=0 tx=0).stop 2024-01-11T08:26:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.527+0000 7f3ef91ff700 1 -- 172.21.15.27:0/29966843 shutdown_connections 2024-01-11T08:26:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.527+0000 7f3ef91ff700 1 --2- 172.21.15.27:0/29966843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ef40cb4b0 0x7f3ef40cc560 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.527+0000 7f3ef91ff700 1 --2- 172.21.15.27:0/29966843 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3ef4143190 0x7f3ef40cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.527+0000 7f3ef91ff700 1 -- 172.21.15.27:0/29966843 >> 172.21.15.27:0/29966843 conn(0x7f3ef4063ca0 msgr2=0x7f3ef406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.527+0000 7f3ef91ff700 1 -- 172.21.15.27:0/29966843 shutdown_connections 2024-01-11T08:26:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.527+0000 7f3ef91ff700 1 -- 172.21.15.27:0/29966843 wait complete. 2024-01-11T08:26:55.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.529+0000 7f3ef91ff700 1 Processor -- start 2024-01-11T08:26:55.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.530+0000 7f3ef91ff700 1 -- start start 2024-01-11T08:26:55.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.531+0000 7f3ef91ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ef40cb4b0 0x7f3ef416ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:55.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.531+0000 7f3ef91ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ef4142dc0 0x7f3ef41419b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:55.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.531+0000 7f3ef91ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3ef4143190 0x7f3ef40c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:55.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.531+0000 7f3ef91ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef416cf50 con 0x7f3ef40cb4b0 2024-01-11T08:26:55.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.531+0000 7f3ef91ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef40c6d70 con 0x7f3ef4142dc0 2024-01-11T08:26:55.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.531+0000 7f3ef91ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef40c6160 con 0x7f3ef4143190 2024-01-11T08:26:55.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.531+0000 7f3ef2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ef40cb4b0 0x7f3ef416ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:55.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.531+0000 7f3ef259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ef4142dc0 0x7f3ef41419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:55.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.531+0000 7f3ef2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ef40cb4b0 0x7f3ef416ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41748/0 (socket says 172.21.15.27:41748) 2024-01-11T08:26:55.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.531+0000 7f3ef2d9d700 1 -- 172.21.15.27:0/557776843 learned_addr learned my addr 172.21.15.27:0/557776843 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:26:55.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.531+0000 7f3ef359e700 1 --2- 172.21.15.27:0/557776843 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3ef4143190 0x7f3ef40c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:55.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.532+0000 7f3ef259c700 1 -- 172.21.15.27:0/557776843 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3ef4143190 msgr2=0x7f3ef40c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:55.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.532+0000 7f3ef259c700 1 --2- 172.21.15.27:0/557776843 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3ef4143190 0x7f3ef40c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:55.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.532+0000 7f3ef259c700 1 -- 172.21.15.27:0/557776843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ef40cb4b0 msgr2=0x7f3ef416ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:55.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.532+0000 7f3ef259c700 1 --2- 172.21.15.27:0/557776843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ef40cb4b0 0x7f3ef416ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:55.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.532+0000 7f3ef259c700 1 -- 172.21.15.27:0/557776843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee8009420 con 0x7f3ef4142dc0 2024-01-11T08:26:55.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.532+0000 7f3ef259c700 1 --2- 172.21.15.27:0/557776843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ef4142dc0 0x7f3ef41419b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3ee800ea30 tx=0x7f3ee80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:55.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.532+0000 7f3edaffd700 1 -- 172.21.15.27:0/557776843 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8009660 con 0x7f3ef4142dc0 2024-01-11T08:26:55.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.532+0000 7f3edaffd700 1 -- 172.21.15.27:0/557776843 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee800ee70 con 0x7f3ef4142dc0 2024-01-11T08:26:55.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.532+0000 7f3ef91ff700 1 -- 172.21.15.27:0/557776843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eec009a20 con 0x7f3ef4142dc0 2024-01-11T08:26:55.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.532+0000 7f3edaffd700 1 -- 172.21.15.27:0/557776843 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee800ca80 con 0x7f3ef4142dc0 2024-01-11T08:26:55.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.532+0000 7f3ef91ff700 1 -- 172.21.15.27:0/557776843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ef40c64f0 con 0x7f3ef4142dc0 2024-01-11T08:26:55.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.533+0000 7f3ef91ff700 1 -- 172.21.15.27:0/557776843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ef40c6f20 con 0x7f3ef4142dc0 2024-01-11T08:26:55.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.534+0000 7f3edaffd700 1 -- 172.21.15.27:0/557776843 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 49) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f3ee8013470 con 0x7f3ef4142dc0 2024-01-11T08:26:55.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.535+0000 7f3edaffd700 1 --2- 172.21.15.27:0/557776843 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3ee0066dd0 0x7f3ee0069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:26:55.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.535+0000 7f3edaffd700 1 -- 172.21.15.27:0/557776843 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5991+0+0 (secure 0 0 0) 0x7f3ee8084190 con 0x7f3ef4142dc0 2024-01-11T08:26:55.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.536+0000 7f3ef2d9d700 1 --2- 172.21.15.27:0/557776843 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3ee0066dd0 0x7f3ee0069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:26:55.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.536+0000 7f3ef2d9d700 1 --2- 172.21.15.27:0/557776843 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3ee0066dd0 0x7f3ee0069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4007fd0 tx=0x7f3ee4007df0 comp rx=0 tx=0).ready entity=mgr.7066 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:26:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.540+0000 7f3edaffd700 1 -- 172.21.15.27:0/557776843 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3ef40c6f20 con 0x7f3ef4142dc0 2024-01-11T08:26:55.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.769+0000 7f3ef91ff700 1 -- 172.21.15.27:0/557776843 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3ef404c3c0 con 0x7f3ee0066dd0 2024-01-11T08:26:55.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.782+0000 7f3edaffd700 1 -- 172.21.15.27:0/557776843 <== mgr.7066 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+287 (secure 0 0 0) 0x7f3ef404c3c0 con 0x7f3ee0066dd0 2024-01-11T08:26:55.782 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:26:55.782 INFO:teuthology.orchestra.run.smithi027.stdout:[{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:26:55", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-01-11T08:26:55.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.784+0000 7f3ef91ff700 1 -- 172.21.15.27:0/557776843 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3ee0066dd0 msgr2=0x7f3ee0069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:55.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.784+0000 7f3ef91ff700 1 --2- 172.21.15.27:0/557776843 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3ee0066dd0 0x7f3ee0069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4007fd0 tx=0x7f3ee4007df0 comp rx=0 tx=0).stop 2024-01-11T08:26:55.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.784+0000 7f3ef91ff700 1 -- 172.21.15.27:0/557776843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ef4142dc0 msgr2=0x7f3ef41419b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:26:55.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.784+0000 7f3ef91ff700 1 --2- 172.21.15.27:0/557776843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ef4142dc0 0x7f3ef41419b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3ee800ea30 tx=0x7f3ee80091a0 comp rx=0 tx=0).stop 2024-01-11T08:26:55.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.785+0000 7f3ef91ff700 1 -- 172.21.15.27:0/557776843 shutdown_connections 2024-01-11T08:26:55.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.785+0000 7f3ef91ff700 1 --2- 172.21.15.27:0/557776843 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3ee0066dd0 0x7f3ee0069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:55.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.785+0000 7f3ef91ff700 1 --2- 172.21.15.27:0/557776843 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3ef4143190 0x7f3ef40c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:55.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.785+0000 7f3ef91ff700 1 --2- 172.21.15.27:0/557776843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ef40cb4b0 0x7f3ef416ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:55.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.785+0000 7f3ef91ff700 1 --2- 172.21.15.27:0/557776843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ef4142dc0 0x7f3ef41419b0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:26:55.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.785+0000 7f3ef91ff700 1 -- 172.21.15.27:0/557776843 >> 172.21.15.27:0/557776843 conn(0x7f3ef4063ca0 msgr2=0x7f3ef406cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:26:55.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.785+0000 7f3ef91ff700 1 -- 172.21.15.27:0/557776843 shutdown_connections 2024-01-11T08:26:55.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:26:55.785+0000 7f3ef91ff700 1 -- 172.21.15.27:0/557776843 wait complete. 2024-01-11T08:26:55.802 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:26:55", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-01-11T08:27:00.312 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:27:00.312+0000 7f902009e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:27:00.312 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:27:00.312+0000 7f902009e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:27:00.312 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:27:00.312+0000 7f902009e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:27:00.313 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:27:00.312+0000 7f902009e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:27:00.313 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:27:00.312+0000 7f902009e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:27:22.810 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:27:22.816 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:27:52.926 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:27:52.931 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:27:55.803 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2024-01-11T08:27:56.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.018+0000 7f5ec87ff700 1 Processor -- start 2024-01-11T08:27:56.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.019+0000 7f5ec87ff700 1 -- start start 2024-01-11T08:27:56.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.020+0000 7f5ec87ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ec0142dc0 0x7f5ec00cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:56.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.020+0000 7f5ec87ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ec0143190 0x7f5ec00cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:56.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.020+0000 7f5ec87ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec015d320 con 0x7f5ec00cc020 2024-01-11T08:27:56.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.020+0000 7f5ec87ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ec015df30 con 0x7f5ec0143190 2024-01-11T08:27:56.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.020+0000 7f5ec87ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec015d590 con 0x7f5ec0142dc0 2024-01-11T08:27:56.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.020+0000 7f5ec5d9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ec0143190 0x7f5ec00cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:56.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.020+0000 7f5ec5d9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ec0143190 0x7f5ec00cb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56104/0 (socket says 172.21.15.27:56104) 2024-01-11T08:27:56.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.020+0000 7f5ec5d9a700 1 -- 172.21.15.27:0/1882265251 learned_addr learned my addr 172.21.15.27:0/1882265251 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:27:56.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.020+0000 7f5ec659b700 1 --2- 172.21.15.27:0/1882265251 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ec0142dc0 0x7f5ec00cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:56.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.021+0000 7f5ec5d9a700 1 -- 172.21.15.27:0/1882265251 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ec0142dc0 msgr2=0x7f5ec00cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:27:56.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.021+0000 7f5ec5d9a700 1 --2- 172.21.15.27:0/1882265251 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ec0142dc0 0x7f5ec00cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:56.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.021+0000 7f5ec5d9a700 1 -- 172.21.15.27:0/1882265251 >> v1:172.21.15.27:6789/0 conn(0x7f5ec00cc020 legacy=0x7f5ec00cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:56.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.021+0000 7f5ec5d9a700 1 -- 172.21.15.27:0/1882265251 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec015d870 con 0x7f5ec0143190 2024-01-11T08:27:56.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.021+0000 7f5ec5599700 1 -- 172.21.15.27:0/1882265251 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031310494 0 0) 0x7f5ec015d320 con 0x7f5ec00cc020 2024-01-11T08:27:56.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.021+0000 7f5ec5d9a700 1 --2- 172.21.15.27:0/1882265251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec0143190 0x7f5ec00cb2e0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f5eb00099b0 tx=0x7f5eb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e650eb7bff4002 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:27:56.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.021+0000 7f5ec5599700 1 -- 172.21.15.27:0/1882265251 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb0009ea0 con 0x7f5ec0143190 2024-01-11T08:27:56.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.021+0000 7f5ec5599700 1 -- 172.21.15.27:0/1882265251 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb000b860 con 0x7f5ec0143190 2024-01-11T08:27:56.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.021+0000 7f5ec5599700 1 -- 172.21.15.27:0/1882265251 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb000bc00 con 0x7f5ec0143190 2024-01-11T08:27:56.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.022+0000 7f5ec87ff700 1 -- 172.21.15.27:0/1882265251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec0143190 msgr2=0x7f5ec00cb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:56.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.022+0000 7f5ec87ff700 1 --2- 172.21.15.27:0/1882265251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec0143190 0x7f5ec00cb2e0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f5eb00099b0 tx=0x7f5eb0009250 comp rx=0 tx=0).stop 2024-01-11T08:27:56.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.022+0000 7f5ec87ff700 1 -- 172.21.15.27:0/1882265251 shutdown_connections 2024-01-11T08:27:56.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.022+0000 7f5ec87ff700 1 --2- 172.21.15.27:0/1882265251 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ec0142dc0 0x7f5ec00cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:56.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.022+0000 7f5ec87ff700 1 --2- 172.21.15.27:0/1882265251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec0143190 0x7f5ec00cb2e0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:56.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.022+0000 7f5ec87ff700 1 -- 172.21.15.27:0/1882265251 >> 172.21.15.27:0/1882265251 conn(0x7f5ec0063ca0 msgr2=0x7f5ec006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:27:56.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.022+0000 7f5ec87ff700 1 -- 172.21.15.27:0/1882265251 shutdown_connections 2024-01-11T08:27:56.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.022+0000 7f5ec87ff700 1 -- 172.21.15.27:0/1882265251 wait complete. 2024-01-11T08:27:56.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.024+0000 7f5ec87ff700 1 Processor -- start 2024-01-11T08:27:56.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.024+0000 7f5ec87ff700 1 -- start start 2024-01-11T08:27:56.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.025+0000 7f5ec87ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec00cc020 0x7f5ec00c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:56.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.025+0000 7f5ec87ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ec0142dc0 0x7f5ec00c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:56.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.025+0000 7f5ec87ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ec0143190 0x7f5ec01528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:56.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.025+0000 7f5ec87ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec0169ed0 con 0x7f5ec00cc020 2024-01-11T08:27:56.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.025+0000 7f5ec87ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec00c7700 con 0x7f5ec0142dc0 2024-01-11T08:27:56.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.025+0000 7f5ec87ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec00c79e0 con 0x7f5ec0143190 2024-01-11T08:27:56.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.025+0000 7f5ec659b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec00cc020 0x7f5ec00c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:56.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.025+0000 7f5ec659b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec00cc020 0x7f5ec00c8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56106/0 (socket says 172.21.15.27:56106) 2024-01-11T08:27:56.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.025+0000 7f5ec659b700 1 -- 172.21.15.27:0/4042244705 learned_addr learned my addr 172.21.15.27:0/4042244705 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:27:56.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.025+0000 7f5ec6d9c700 1 --2- 172.21.15.27:0/4042244705 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ec0143190 0x7f5ec01528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:56.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.025+0000 7f5ec5d9a700 1 --2- 172.21.15.27:0/4042244705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ec0142dc0 0x7f5ec00c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:56.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.026+0000 7f5ec659b700 1 -- 172.21.15.27:0/4042244705 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ec0143190 msgr2=0x7f5ec01528a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:56.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.026+0000 7f5ec659b700 1 --2- 172.21.15.27:0/4042244705 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ec0143190 0x7f5ec01528a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:56.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.026+0000 7f5ec659b700 1 -- 172.21.15.27:0/4042244705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ec0142dc0 msgr2=0x7f5ec00c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:56.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.026+0000 7f5ec659b700 1 --2- 172.21.15.27:0/4042244705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ec0142dc0 0x7f5ec00c6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:56.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.026+0000 7f5ec659b700 1 -- 172.21.15.27:0/4042244705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb00094f0 con 0x7f5ec00cc020 2024-01-11T08:27:56.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.026+0000 7f5ec6d9c700 1 --2- 172.21.15.27:0/4042244705 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ec0143190 0x7f5ec01528a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:27:56.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.026+0000 7f5ec5d9a700 1 --2- 172.21.15.27:0/4042244705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ec0142dc0 0x7f5ec00c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:27:56.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.026+0000 7f5ec659b700 1 --2- 172.21.15.27:0/4042244705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec00cc020 0x7f5ec00c8e80 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc00ec80 tx=0x7f5ebc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:27:56.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.026+0000 7f5eb67fc700 1 -- 172.21.15.27:0/4042244705 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ebc00c9d0 con 0x7f5ec00cc020 2024-01-11T08:27:56.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.026+0000 7f5eb67fc700 1 -- 172.21.15.27:0/4042244705 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ebc00cb30 con 0x7f5ec00cc020 2024-01-11T08:27:56.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.026+0000 7f5ec87ff700 1 -- 172.21.15.27:0/4042244705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec00c7cc0 con 0x7f5ec00cc020 2024-01-11T08:27:56.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.026+0000 7f5eb67fc700 1 -- 172.21.15.27:0/4042244705 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ebc013470 con 0x7f5ec00cc020 2024-01-11T08:27:56.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.026+0000 7f5ec87ff700 1 -- 172.21.15.27:0/4042244705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec00c8280 con 0x7f5ec00cc020 2024-01-11T08:27:56.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.027+0000 7f5ec87ff700 1 -- 172.21.15.27:0/4042244705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec004c3c0 con 0x7f5ec00cc020 2024-01-11T08:27:56.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.028+0000 7f5eb67fc700 1 -- 172.21.15.27:0/4042244705 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 52) v1 ==== 84975+0+0 (secure 0 0 0) 0x7f5ebc013820 con 0x7f5ec00cc020 2024-01-11T08:27:56.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.029+0000 7f5eb67fc700 1 --2- 172.21.15.27:0/4042244705 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5ea40671c0 0x7f5ea4069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:56.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.029+0000 7f5eb67fc700 1 -- 172.21.15.27:0/4042244705 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6020+0+0 (secure 0 0 0) 0x7f5ebc084e60 con 0x7f5ec00cc020 2024-01-11T08:27:56.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.029+0000 7f5ec5d9a700 1 --2- 172.21.15.27:0/4042244705 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5ea40671c0 0x7f5ea4069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:56.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.030+0000 7f5ec5d9a700 1 --2- 172.21.15.27:0/4042244705 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5ea40671c0 0x7f5ea4069670 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5eb00099b0 tx=0x7f5eb0017000 comp rx=0 tx=0).ready entity=mgr.7066 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:27:56.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.034+0000 7f5eb67fc700 1 -- 172.21.15.27:0/4042244705 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5ebc051c60 con 0x7f5ec00cc020 2024-01-11T08:27:56.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.262+0000 7f5ec87ff700 1 -- 172.21.15.27:0/4042244705 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5ec015db50 con 0x7f5ea40671c0 2024-01-11T08:27:56.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.273+0000 7f5eb67fc700 1 -- 172.21.15.27:0/4042244705 <== mgr.7066 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+288 (secure 0 0 0) 0x7f5ec015db50 con 0x7f5ea40671c0 2024-01-11T08:27:56.272 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:27:56.273 INFO:teuthology.orchestra.run.smithi027.stdout:[{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:26:55", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2024-01-11T08:27:56.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.276+0000 7f5ec87ff700 1 -- 172.21.15.27:0/4042244705 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5ea40671c0 msgr2=0x7f5ea4069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:56.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.276+0000 7f5ec87ff700 1 --2- 172.21.15.27:0/4042244705 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5ea40671c0 0x7f5ea4069670 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5eb00099b0 tx=0x7f5eb0017000 comp rx=0 tx=0).stop 2024-01-11T08:27:56.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.276+0000 7f5ec87ff700 1 -- 172.21.15.27:0/4042244705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec00cc020 msgr2=0x7f5ec00c8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:56.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.276+0000 7f5ec87ff700 1 --2- 172.21.15.27:0/4042244705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec00cc020 0x7f5ec00c8e80 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc00ec80 tx=0x7f5ebc0091a0 comp rx=0 tx=0).stop 2024-01-11T08:27:56.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.276+0000 7f5ec87ff700 1 -- 172.21.15.27:0/4042244705 shutdown_connections 2024-01-11T08:27:56.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.276+0000 7f5ec87ff700 1 --2- 172.21.15.27:0/4042244705 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5ea40671c0 0x7f5ea4069670 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:56.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.276+0000 7f5ec87ff700 1 --2- 172.21.15.27:0/4042244705 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ec0143190 0x7f5ec01528a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:56.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.276+0000 7f5ec87ff700 1 --2- 172.21.15.27:0/4042244705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec00cc020 0x7f5ec00c8e80 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:56.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.276+0000 7f5ec87ff700 1 --2- 172.21.15.27:0/4042244705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ec0142dc0 0x7f5ec00c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:56.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.276+0000 7f5ec87ff700 1 -- 172.21.15.27:0/4042244705 >> 172.21.15.27:0/4042244705 conn(0x7f5ec0063ca0 msgr2=0x7f5ec0068720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:27:56.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.276+0000 7f5ec87ff700 1 -- 172.21.15.27:0/4042244705 shutdown_connections 2024-01-11T08:27:56.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.276+0000 7f5ec87ff700 1 -- 172.21.15.27:0/4042244705 wait complete. 2024-01-11T08:27:56.293 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:26:55", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2024-01-11T08:27:56.293 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /bad-path 2024-01-11T08:27:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.497+0000 7fb5d2fff700 1 Processor -- start 2024-01-11T08:27:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.497+0000 7fb5d2fff700 1 -- start start 2024-01-11T08:27:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.498+0000 7fb5d2fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb5cc155c50 0x7fb5cc0aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.498+0000 7fb5d2fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5cc0c74f0 0x7fb5cc069910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.498+0000 7fb5d2fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5cc06e480 con 0x7fb5cc0694c0 2024-01-11T08:27:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.498+0000 7fb5d2fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5cc0cb5a0 con 0x7fb5cc0c74f0 2024-01-11T08:27:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.498+0000 7fb5d2fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5cc06df00 con 0x7fb5cc155c50 2024-01-11T08:27:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.498+0000 7fb5d159c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5cc0c74f0 0x7fb5cc069910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.499+0000 7fb5cbfff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb5cc0694c0 0x7fb5cc0c6de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53514/0 (socket says 172.21.15.27:53514) 2024-01-11T08:27:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.499+0000 7fb5cbfff700 1 -- 172.21.15.27:0/766659669 learned_addr learned my addr 172.21.15.27:0/766659669 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:27:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.499+0000 7fb5d0d9b700 1 --2- 172.21.15.27:0/766659669 >> v2:172.21.15.77:3300/0 conn(0x7fb5cc155c50 0x7fb5cc0aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.499+0000 7fb5d159c700 1 -- 172.21.15.27:0/766659669 >> v2:172.21.15.77:3300/0 conn(0x7fb5cc155c50 msgr2=0x7fb5cc0aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:27:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.499+0000 7fb5d159c700 1 --2- 172.21.15.27:0/766659669 >> v2:172.21.15.77:3300/0 conn(0x7fb5cc155c50 0x7fb5cc0aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.499+0000 7fb5d159c700 1 -- 172.21.15.27:0/766659669 >> v1:172.21.15.27:6789/0 conn(0x7fb5cc0694c0 legacy=0x7fb5cc0c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.499+0000 7fb5d159c700 1 -- 172.21.15.27:0/766659669 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5cc069e50 con 0x7fb5cc0c74f0 2024-01-11T08:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.499+0000 7fb5d159c700 1 --2- 172.21.15.27:0/766659669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5cc0c74f0 0x7fb5cc069910 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc06dcd0 tx=0x7fb5c400a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fcfed49d88acab server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.500+0000 7fb5cb7fe700 1 -- 172.21.15.27:0/766659669 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c400f070 con 0x7fb5cc0c74f0 2024-01-11T08:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.500+0000 7fb5cb7fe700 1 -- 172.21.15.27:0/766659669 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c400a2e0 con 0x7fb5cc0c74f0 2024-01-11T08:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.500+0000 7fb5cb7fe700 1 -- 172.21.15.27:0/766659669 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c400cd80 con 0x7fb5cc0c74f0 2024-01-11T08:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.500+0000 7fb5d2fff700 1 -- 172.21.15.27:0/766659669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5cc0c74f0 msgr2=0x7fb5cc069910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.500+0000 7fb5d2fff700 1 --2- 172.21.15.27:0/766659669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5cc0c74f0 0x7fb5cc069910 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc06dcd0 tx=0x7fb5c400a890 comp rx=0 tx=0).stop 2024-01-11T08:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.500+0000 7fb5d2fff700 1 -- 172.21.15.27:0/766659669 shutdown_connections 2024-01-11T08:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.500+0000 7fb5d2fff700 1 --2- 172.21.15.27:0/766659669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5cc0c74f0 0x7fb5cc069910 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.500+0000 7fb5d2fff700 1 --2- 172.21.15.27:0/766659669 >> v2:172.21.15.77:3300/0 conn(0x7fb5cc155c50 0x7fb5cc0aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.500+0000 7fb5d2fff700 1 -- 172.21.15.27:0/766659669 >> 172.21.15.27:0/766659669 conn(0x7fb5cc0680a0 msgr2=0x7fb5cc0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.500+0000 7fb5d2fff700 1 -- 172.21.15.27:0/766659669 shutdown_connections 2024-01-11T08:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.500+0000 7fb5d2fff700 1 -- 172.21.15.27:0/766659669 wait complete. 2024-01-11T08:27:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.502+0000 7fb5d2fff700 1 Processor -- start 2024-01-11T08:27:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.502+0000 7fb5d2fff700 1 -- start start 2024-01-11T08:27:56.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.503+0000 7fb5d2fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5cc0694c0 0x7fb5cc14cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:56.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.503+0000 7fb5d2fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5cc0c74f0 0x7fb5cc0c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:56.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.503+0000 7fb5d2fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5cc155c50 0x7fb5cc0abed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:56.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.503+0000 7fb5d2fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5cc14d1b0 con 0x7fb5cc0c74f0 2024-01-11T08:27:56.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.503+0000 7fb5d2fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5cc0c4510 con 0x7fb5cc155c50 2024-01-11T08:27:56.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.503+0000 7fb5d2fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5cc0c47f0 con 0x7fb5cc0694c0 2024-01-11T08:27:56.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.503+0000 7fb5cbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5cc0c74f0 0x7fb5cc0c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:56.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.504+0000 7fb5cbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5cc0c74f0 0x7fb5cc0c3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56124/0 (socket says 172.21.15.27:56124) 2024-01-11T08:27:56.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.504+0000 7fb5cbfff700 1 -- 172.21.15.27:0/3120459543 learned_addr learned my addr 172.21.15.27:0/3120459543 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:27:56.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.504+0000 7fb5d159c700 1 --2- 172.21.15.27:0/3120459543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5cc155c50 0x7fb5cc0abed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:56.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.504+0000 7fb5d0d9b700 1 --2- 172.21.15.27:0/3120459543 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5cc0694c0 0x7fb5cc14cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:56.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.504+0000 7fb5cbfff700 1 -- 172.21.15.27:0/3120459543 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5cc0694c0 msgr2=0x7fb5cc14cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:56.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.504+0000 7fb5cbfff700 1 --2- 172.21.15.27:0/3120459543 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5cc0694c0 0x7fb5cc14cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:56.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.504+0000 7fb5cbfff700 1 -- 172.21.15.27:0/3120459543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5cc155c50 msgr2=0x7fb5cc0abed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:56.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.504+0000 7fb5cbfff700 1 --2- 172.21.15.27:0/3120459543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5cc155c50 0x7fb5cc0abed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:56.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.504+0000 7fb5cbfff700 1 -- 172.21.15.27:0/3120459543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c400ab30 con 0x7fb5cc0c74f0 2024-01-11T08:27:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.504+0000 7fb5cbfff700 1 --2- 172.21.15.27:0/3120459543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5cc0c74f0 0x7fb5cc0c3c80 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800c7c0 tx=0x7fb5b8007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:27:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.505+0000 7fb5c8ff9700 1 -- 172.21.15.27:0/3120459543 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8005b30 con 0x7fb5cc0c74f0 2024-01-11T08:27:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.505+0000 7fb5c8ff9700 1 -- 172.21.15.27:0/3120459543 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b8005c90 con 0x7fb5cc0c74f0 2024-01-11T08:27:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.505+0000 7fb5c8ff9700 1 -- 172.21.15.27:0/3120459543 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8004a80 con 0x7fb5cc0c74f0 2024-01-11T08:27:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.505+0000 7fb5d2fff700 1 -- 172.21.15.27:0/3120459543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5cc0c4ad0 con 0x7fb5cc0c74f0 2024-01-11T08:27:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.505+0000 7fb5d2fff700 1 -- 172.21.15.27:0/3120459543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5cc0aa190 con 0x7fb5cc0c74f0 2024-01-11T08:27:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.506+0000 7fb5d2fff700 1 -- 172.21.15.27:0/3120459543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5cc0aa380 con 0x7fb5cc0c74f0 2024-01-11T08:27:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.507+0000 7fb5c8ff9700 1 -- 172.21.15.27:0/3120459543 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 52) v1 ==== 84975+0+0 (secure 0 0 0) 0x7fb5b80070d0 con 0x7fb5cc0c74f0 2024-01-11T08:27:56.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.507+0000 7fb5c8ff9700 1 --2- 172.21.15.27:0/3120459543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb5b00671c0 0x7fb5b0069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:56.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.508+0000 7fb5c8ff9700 1 -- 172.21.15.27:0/3120459543 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6020+0+0 (secure 0 0 0) 0x7fb5b8083dc0 con 0x7fb5cc0c74f0 2024-01-11T08:27:56.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.508+0000 7fb5d0d9b700 1 --2- 172.21.15.27:0/3120459543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb5b00671c0 0x7fb5b0069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:56.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.509+0000 7fb5d0d9b700 1 --2- 172.21.15.27:0/3120459543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb5b00671c0 0x7fb5b0069670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb5c00061f0 tx=0x7fb5c0008040 comp rx=0 tx=0).ready entity=mgr.7066 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:27:56.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.512+0000 7fb5c8ff9700 1 -- 172.21.15.27:0/3120459543 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb5b804b5a0 con 0x7fb5cc0c74f0 2024-01-11T08:27:56.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.743+0000 7fb5d2fff700 1 -- 172.21.15.27:0/3120459543 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/bad-path", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5cc04c3c0 con 0x7fb5b00671c0 2024-01-11T08:27:56.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.813+0000 7fb5c8ff9700 1 -- 172.21.15.27:0/3120459543 <== mgr.7066 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+36 (secure 0 0 0) 0x7fb5cc04c3c0 con 0x7fb5b00671c0 2024-01-11T08:27:56.813 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule removed for path /bad-path 2024-01-11T08:27:56.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.816+0000 7fb5d2fff700 1 -- 172.21.15.27:0/3120459543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb5b00671c0 msgr2=0x7fb5b0069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:56.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.816+0000 7fb5d2fff700 1 --2- 172.21.15.27:0/3120459543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb5b00671c0 0x7fb5b0069670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb5c00061f0 tx=0x7fb5c0008040 comp rx=0 tx=0).stop 2024-01-11T08:27:56.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.816+0000 7fb5d2fff700 1 -- 172.21.15.27:0/3120459543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5cc0c74f0 msgr2=0x7fb5cc0c3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:56.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.816+0000 7fb5d2fff700 1 --2- 172.21.15.27:0/3120459543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5cc0c74f0 0x7fb5cc0c3c80 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800c7c0 tx=0x7fb5b8007f70 comp rx=0 tx=0).stop 2024-01-11T08:27:56.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.816+0000 7fb5d2fff700 1 -- 172.21.15.27:0/3120459543 shutdown_connections 2024-01-11T08:27:56.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.816+0000 7fb5d2fff700 1 --2- 172.21.15.27:0/3120459543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb5b00671c0 0x7fb5b0069670 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:56.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.816+0000 7fb5d2fff700 1 --2- 172.21.15.27:0/3120459543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5cc155c50 0x7fb5cc0abed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:56.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.816+0000 7fb5d2fff700 1 --2- 172.21.15.27:0/3120459543 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5cc0694c0 0x7fb5cc14cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:56.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.816+0000 7fb5d2fff700 1 --2- 172.21.15.27:0/3120459543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5cc0c74f0 0x7fb5cc0c3c80 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:56.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.816+0000 7fb5d2fff700 1 -- 172.21.15.27:0/3120459543 >> 172.21.15.27:0/3120459543 conn(0x7fb5cc0680a0 msgr2=0x7fb5cc06e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:27:56.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.816+0000 7fb5d2fff700 1 -- 172.21.15.27:0/3120459543 shutdown_connections 2024-01-11T08:27:56.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:56.816+0000 7fb5d2fff700 1 -- 172.21.15.27:0/3120459543 wait complete. 2024-01-11T08:27:56.832 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2024-01-11T08:27:56.833 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-11T08:27:57.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.035+0000 7f97f0dff700 1 Processor -- start 2024-01-11T08:27:57.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.036+0000 7f97f0dff700 1 -- start start 2024-01-11T08:27:57.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.037+0000 7f97f0dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f97ec15a1b0 0x7f97ec15a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:57.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.037+0000 7f97f0dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97ec0ccd70 0x7f97ec0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:57.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.037+0000 7f97f0dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f97ec0685c0 0x7f97ec068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:57.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.037+0000 7f97f0dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97ec0cf850 con 0x7f97ec0685c0 2024-01-11T08:27:57.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.037+0000 7f97f0dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97ec0cec40 con 0x7f97ec15a1b0 2024-01-11T08:27:57.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.037+0000 7f97f0dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec0cef20 con 0x7f97ec0ccd70 2024-01-11T08:27:57.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.037+0000 7f97ead9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f97ec0685c0 0x7f97ec068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:57.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.037+0000 7f97ead9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f97ec0685c0 0x7f97ec068a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56126/0 (socket says 172.21.15.27:56126) 2024-01-11T08:27:57.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.037+0000 7f97ead9d700 1 -- 172.21.15.27:0/2126119645 learned_addr learned my addr 172.21.15.27:0/2126119645 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:27:57.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.037+0000 7f97ea59c700 1 --2- 172.21.15.27:0/2126119645 >> v2:172.21.15.77:3300/0 conn(0x7f97ec15a1b0 0x7f97ec15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:57.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.037+0000 7f97e9d9b700 1 --2- 172.21.15.27:0/2126119645 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97ec0ccd70 0x7f97ec0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:57.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.038+0000 7f97ead9d700 1 -- 172.21.15.27:0/2126119645 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97ec0ccd70 msgr2=0x7f97ec0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:27:57.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.038+0000 7f97ead9d700 1 --2- 172.21.15.27:0/2126119645 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97ec0ccd70 0x7f97ec0cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:57.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.038+0000 7f97ead9d700 1 -- 172.21.15.27:0/2126119645 >> v2:172.21.15.77:3300/0 conn(0x7f97ec15a1b0 msgr2=0x7f97ec15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:27:57.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.038+0000 7f97ead9d700 1 --2- 172.21.15.27:0/2126119645 >> v2:172.21.15.77:3300/0 conn(0x7f97ec15a1b0 0x7f97ec15a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:57.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.038+0000 7f97ead9d700 1 -- 172.21.15.27:0/2126119645 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec0cf200 con 0x7f97ec0685c0 2024-01-11T08:27:57.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.038+0000 7f97ead9d700 1 --2- 172.21.15.27:0/2126119645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec0685c0 0x7f97ec068a40 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f97ec0d16f0 tx=0x7f97e4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a5ba5232433a43 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:27:57.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.038+0000 7f97e959a700 1 -- 172.21.15.27:0/2126119645 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e400e070 con 0x7f97ec0685c0 2024-01-11T08:27:57.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.038+0000 7f97e959a700 1 -- 172.21.15.27:0/2126119645 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e400b970 con 0x7f97ec0685c0 2024-01-11T08:27:57.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.038+0000 7f97e959a700 1 -- 172.21.15.27:0/2126119645 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e40092e0 con 0x7f97ec0685c0 2024-01-11T08:27:57.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.039+0000 7f97f0dff700 1 -- 172.21.15.27:0/2126119645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec0685c0 msgr2=0x7f97ec068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:57.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.039+0000 7f97f0dff700 1 --2- 172.21.15.27:0/2126119645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec0685c0 0x7f97ec068a40 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f97ec0d16f0 tx=0x7f97e4009840 comp rx=0 tx=0).stop 2024-01-11T08:27:57.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.039+0000 7f97f0dff700 1 -- 172.21.15.27:0/2126119645 shutdown_connections 2024-01-11T08:27:57.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.039+0000 7f97f0dff700 1 --2- 172.21.15.27:0/2126119645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec0685c0 0x7f97ec068a40 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:57.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.039+0000 7f97f0dff700 1 --2- 172.21.15.27:0/2126119645 >> v2:172.21.15.77:3300/0 conn(0x7f97ec15a1b0 0x7f97ec15a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:57.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.039+0000 7f97f0dff700 1 --2- 172.21.15.27:0/2126119645 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97ec0ccd70 0x7f97ec0cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:57.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.039+0000 7f97f0dff700 1 -- 172.21.15.27:0/2126119645 >> 172.21.15.27:0/2126119645 conn(0x7f97ec06ed40 msgr2=0x7f97ec06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:27:57.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.039+0000 7f97f0dff700 1 -- 172.21.15.27:0/2126119645 shutdown_connections 2024-01-11T08:27:57.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.039+0000 7f97f0dff700 1 -- 172.21.15.27:0/2126119645 wait complete. 2024-01-11T08:27:57.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.041+0000 7f97f0dff700 1 Processor -- start 2024-01-11T08:27:57.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.041+0000 7f97f0dff700 1 -- start start 2024-01-11T08:27:57.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.042+0000 7f97f0dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97ec0685c0 0x7f97ec143690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:57.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.043+0000 7f97f0dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec0ccd70 0x7f97ec0c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:57.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.043+0000 7f97f0dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97ec15a1b0 0x7f97ec157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:57.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.043+0000 7f97f0dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec15d950 con 0x7f97ec0ccd70 2024-01-11T08:27:57.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.043+0000 7f97f0dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec15e1f0 con 0x7f97ec15a1b0 2024-01-11T08:27:57.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.043+0000 7f97f0dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec0ca3d0 con 0x7f97ec0685c0 2024-01-11T08:27:57.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.043+0000 7f97e9d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec0ccd70 0x7f97ec0c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:57.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.043+0000 7f97e9d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec0ccd70 0x7f97ec0c98c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56138/0 (socket says 172.21.15.27:56138) 2024-01-11T08:27:57.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.043+0000 7f97e9d9b700 1 -- 172.21.15.27:0/2305873714 learned_addr learned my addr 172.21.15.27:0/2305873714 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:27:57.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.043+0000 7f97ea59c700 1 --2- 172.21.15.27:0/2305873714 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97ec0685c0 0x7f97ec143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:57.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.043+0000 7f97ead9d700 1 --2- 172.21.15.27:0/2305873714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97ec15a1b0 0x7f97ec157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:57.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.043+0000 7f97e9d9b700 1 -- 172.21.15.27:0/2305873714 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97ec0685c0 msgr2=0x7f97ec143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:57.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.043+0000 7f97e9d9b700 1 --2- 172.21.15.27:0/2305873714 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97ec0685c0 0x7f97ec143690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:57.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.043+0000 7f97e9d9b700 1 -- 172.21.15.27:0/2305873714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97ec15a1b0 msgr2=0x7f97ec157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:57.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.043+0000 7f97e9d9b700 1 --2- 172.21.15.27:0/2305873714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97ec15a1b0 0x7f97ec157e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:57.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.043+0000 7f97e9d9b700 1 -- 172.21.15.27:0/2305873714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e4009ae0 con 0x7f97ec0ccd70 2024-01-11T08:27:57.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.044+0000 7f97e9d9b700 1 --2- 172.21.15.27:0/2305873714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec0ccd70 0x7f97ec0c98c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f97e000ecb0 tx=0x7f97e00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:27:57.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.044+0000 7f97da7fc700 1 -- 172.21.15.27:0/2305873714 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e000c9d0 con 0x7f97ec0ccd70 2024-01-11T08:27:57.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.044+0000 7f97da7fc700 1 -- 172.21.15.27:0/2305873714 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e000cb30 con 0x7f97ec0ccd70 2024-01-11T08:27:57.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.044+0000 7f97f0dff700 1 -- 172.21.15.27:0/2305873714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ec15dba0 con 0x7f97ec0ccd70 2024-01-11T08:27:57.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.044+0000 7f97da7fc700 1 -- 172.21.15.27:0/2305873714 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e0013470 con 0x7f97ec0ccd70 2024-01-11T08:27:57.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.044+0000 7f97f0dff700 1 -- 172.21.15.27:0/2305873714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ec143bd0 con 0x7f97ec0ccd70 2024-01-11T08:27:57.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.045+0000 7f97f0dff700 1 -- 172.21.15.27:0/2305873714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ec15e380 con 0x7f97ec0ccd70 2024-01-11T08:27:57.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.046+0000 7f97da7fc700 1 -- 172.21.15.27:0/2305873714 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 52) v1 ==== 84975+0+0 (secure 0 0 0) 0x7f97e0013820 con 0x7f97ec0ccd70 2024-01-11T08:27:57.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.047+0000 7f97da7fc700 1 --2- 172.21.15.27:0/2305873714 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f97cc0671c0 0x7f97cc069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:57.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.048+0000 7f97da7fc700 1 -- 172.21.15.27:0/2305873714 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6020+0+0 (secure 0 0 0) 0x7f97e0084dd0 con 0x7f97ec0ccd70 2024-01-11T08:27:57.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.048+0000 7f97ea59c700 1 --2- 172.21.15.27:0/2305873714 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f97cc0671c0 0x7f97cc069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:57.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.049+0000 7f97ea59c700 1 --2- 172.21.15.27:0/2305873714 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f97cc0671c0 0x7f97cc069670 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009d40 tx=0x7f97dc008040 comp rx=0 tx=0).ready entity=mgr.7066 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:27:57.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.052+0000 7f97da7fc700 1 -- 172.21.15.27:0/2305873714 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f97e0051bd0 con 0x7f97ec0ccd70 2024-01-11T08:27:57.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.310+0000 7f97f0dff700 1 -- 172.21.15.27:0/2305873714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f97ec04c3c0 con 0x7f97ec0ccd70 2024-01-11T08:27:57.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.873+0000 7f97da7fc700 1 -- 172.21.15.27:0/2305873714 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 53) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f97e008b080 con 0x7f97ec0ccd70 2024-01-11T08:27:57.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.874+0000 7f97da7fc700 1 -- 172.21.15.27:0/2305873714 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v53) v1 ==== 93+0+0 (secure 0 0 0) 0x7f97e0051bd0 con 0x7f97ec0ccd70 2024-01-11T08:27:57.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.877+0000 7f97f0dff700 1 -- 172.21.15.27:0/2305873714 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f97cc0671c0 msgr2=0x7f97cc069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:57.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.877+0000 7f97f0dff700 1 --2- 172.21.15.27:0/2305873714 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f97cc0671c0 0x7f97cc069670 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009d40 tx=0x7f97dc008040 comp rx=0 tx=0).stop 2024-01-11T08:27:57.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.877+0000 7f97f0dff700 1 -- 172.21.15.27:0/2305873714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec0ccd70 msgr2=0x7f97ec0c98c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:57.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.877+0000 7f97f0dff700 1 --2- 172.21.15.27:0/2305873714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec0ccd70 0x7f97ec0c98c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f97e000ecb0 tx=0x7f97e00091a0 comp rx=0 tx=0).stop 2024-01-11T08:27:57.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.878+0000 7f97f0dff700 1 -- 172.21.15.27:0/2305873714 shutdown_connections 2024-01-11T08:27:57.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.878+0000 7f97f0dff700 1 --2- 172.21.15.27:0/2305873714 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f97cc0671c0 0x7f97cc069670 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:57.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.878+0000 7f97f0dff700 1 --2- 172.21.15.27:0/2305873714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97ec15a1b0 0x7f97ec157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:57.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.878+0000 7f97f0dff700 1 --2- 172.21.15.27:0/2305873714 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f97ec0685c0 0x7f97ec143690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:57.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.878+0000 7f97f0dff700 1 --2- 172.21.15.27:0/2305873714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec0ccd70 0x7f97ec0c98c0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:57.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.878+0000 7f97f0dff700 1 -- 172.21.15.27:0/2305873714 >> 172.21.15.27:0/2305873714 conn(0x7f97ec06ed40 msgr2=0x7f97ec0b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:27:57.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.878+0000 7f97f0dff700 1 -- 172.21.15.27:0/2305873714 shutdown_connections 2024-01-11T08:27:57.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:57.878+0000 7f97f0dff700 1 -- 172.21.15.27:0/2305873714 wait complete. 2024-01-11T08:27:57.898 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T08:27:57.898 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:27:57.996 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:27:58.085 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T08:27:58.086 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-11T08:27:58.086 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:27:58.086 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-01-11T08:27:58.094 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:27:58.094+0000 7f54a36501c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:27:58.095 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:27:58.097+0000 7fcdba6941c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:27:58.165 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T08:27:58.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:27:58.165 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-11T08:27:58.275 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:27:58.276+0000 7fcdba6941c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:27:58.292 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:27:58.297 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:27:58.295+0000 7f54a36501c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:27:58.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.507+0000 7f38207ff700 1 Processor -- start 2024-01-11T08:27:58.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.508+0000 7f38207ff700 1 -- start start 2024-01-11T08:27:58.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.509+0000 7f38207ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f38180c8c70 0x7f381806b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:58.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.509+0000 7f38207ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f38180c9040 0x7f381806be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:58.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.509+0000 7f38207ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381815bc30 con 0x7f38180cd410 2024-01-11T08:27:58.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.509+0000 7f38207ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3818158c40 con 0x7f38180c9040 2024-01-11T08:27:58.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.509+0000 7f38207ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381806c5a0 con 0x7f38180c8c70 2024-01-11T08:27:58.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.509+0000 7f381ed9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f38180cd410 0x7f38180cd850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53530/0 (socket says 172.21.15.27:53530) 2024-01-11T08:27:58.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.509+0000 7f381ed9c700 1 -- 172.21.15.27:0/4092959684 learned_addr learned my addr 172.21.15.27:0/4092959684 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:27:58.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.510+0000 7f381dd9a700 1 --2- 172.21.15.27:0/4092959684 >> v2:172.21.15.77:3300/0 conn(0x7f38180c9040 0x7f381806be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:58.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.509+0000 7f381e59b700 1 --2- 172.21.15.27:0/4092959684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f38180c8c70 0x7f381806b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:58.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.510+0000 7f381d599700 1 -- 172.21.15.27:0/4092959684 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988810855 0 0) 0x7f381815bc30 con 0x7f38180cd410 2024-01-11T08:27:58.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.510+0000 7f381d599700 1 -- 172.21.15.27:0/4092959684 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3804003570 con 0x7f38180cd410 2024-01-11T08:27:58.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.510+0000 7f381dd9a700 1 -- 172.21.15.27:0/4092959684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f38180c8c70 msgr2=0x7f381806b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:27:58.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.510+0000 7f381dd9a700 1 --2- 172.21.15.27:0/4092959684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f38180c8c70 0x7f381806b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:58.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.510+0000 7f381dd9a700 1 -- 172.21.15.27:0/4092959684 >> v1:172.21.15.27:6789/0 conn(0x7f38180cd410 legacy=0x7f38180cd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:58.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.510+0000 7f381dd9a700 1 -- 172.21.15.27:0/4092959684 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381815b300 con 0x7f38180c9040 2024-01-11T08:27:58.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.510+0000 7f381e59b700 1 --2- 172.21.15.27:0/4092959684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f38180c8c70 0x7f381806b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:27:58.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.510+0000 7f381dd9a700 1 --2- 172.21.15.27:0/4092959684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38180c9040 0x7f381806be90 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f38080098e0 tx=0x7f38080091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7b707b243dc6154 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:27:58.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.511+0000 7f381d599700 1 -- 172.21.15.27:0/4092959684 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808011630 con 0x7f38180c9040 2024-01-11T08:27:58.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.511+0000 7f381d599700 1 -- 172.21.15.27:0/4092959684 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3808011790 con 0x7f38180c9040 2024-01-11T08:27:58.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.511+0000 7f381d599700 1 -- 172.21.15.27:0/4092959684 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808011b10 con 0x7f38180c9040 2024-01-11T08:27:58.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.511+0000 7f38207ff700 1 -- 172.21.15.27:0/4092959684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38180c9040 msgr2=0x7f381806be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:58.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.511+0000 7f38207ff700 1 --2- 172.21.15.27:0/4092959684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38180c9040 0x7f381806be90 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f38080098e0 tx=0x7f38080091a0 comp rx=0 tx=0).stop 2024-01-11T08:27:58.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.511+0000 7f38207ff700 1 -- 172.21.15.27:0/4092959684 shutdown_connections 2024-01-11T08:27:58.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.511+0000 7f38207ff700 1 --2- 172.21.15.27:0/4092959684 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f38180c8c70 0x7f381806b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:58.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.511+0000 7f38207ff700 1 --2- 172.21.15.27:0/4092959684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38180c9040 0x7f381806be90 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:58.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.511+0000 7f38207ff700 1 -- 172.21.15.27:0/4092959684 >> 172.21.15.27:0/4092959684 conn(0x7f38180ce3e0 msgr2=0x7f38180ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:27:58.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.511+0000 7f38207ff700 1 -- 172.21.15.27:0/4092959684 shutdown_connections 2024-01-11T08:27:58.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.512+0000 7f38207ff700 1 -- 172.21.15.27:0/4092959684 wait complete. 2024-01-11T08:27:58.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.513+0000 7f38207ff700 1 Processor -- start 2024-01-11T08:27:58.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.513+0000 7f38207ff700 1 -- start start 2024-01-11T08:27:58.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.514+0000 7f38207ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f38180c8c70 0x7f3818159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:58.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.514+0000 7f38207ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38180c9040 0x7f38181502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:58.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.514+0000 7f38207ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38180cd410 0x7f381814e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:58.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.514+0000 7f38207ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381814dba0 con 0x7f38180c9040 2024-01-11T08:27:58.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.514+0000 7f38207ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381814a500 con 0x7f38180cd410 2024-01-11T08:27:58.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.514+0000 7f38207ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3818149f50 con 0x7f38180c8c70 2024-01-11T08:27:58.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.514+0000 7f381dd9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38180c9040 0x7f38181502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:58.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.514+0000 7f381dd9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38180c9040 0x7f38181502b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56148/0 (socket says 172.21.15.27:56148) 2024-01-11T08:27:58.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.514+0000 7f381dd9a700 1 -- 172.21.15.27:0/3110507318 learned_addr learned my addr 172.21.15.27:0/3110507318 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:27:58.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.514+0000 7f381e59b700 1 --2- 172.21.15.27:0/3110507318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f38180c8c70 0x7f3818159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:58.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.514+0000 7f381ed9c700 1 --2- 172.21.15.27:0/3110507318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38180cd410 0x7f381814e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:58.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.515+0000 7f381dd9a700 1 -- 172.21.15.27:0/3110507318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f38180c8c70 msgr2=0x7f3818159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:58.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.515+0000 7f381dd9a700 1 --2- 172.21.15.27:0/3110507318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f38180c8c70 0x7f3818159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:58.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.515+0000 7f381dd9a700 1 -- 172.21.15.27:0/3110507318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38180cd410 msgr2=0x7f381814e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:58.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.515+0000 7f381dd9a700 1 --2- 172.21.15.27:0/3110507318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38180cd410 0x7f381814e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:58.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.515+0000 7f381dd9a700 1 -- 172.21.15.27:0/3110507318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3814009420 con 0x7f38180c9040 2024-01-11T08:27:58.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.515+0000 7f381dd9a700 1 --2- 172.21.15.27:0/3110507318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38180c9040 0x7f38181502b0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f3818064980 tx=0x7f3808017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:27:58.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.515+0000 7f380e7fc700 1 -- 172.21.15.27:0/3110507318 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808011900 con 0x7f38180c9040 2024-01-11T08:27:58.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.515+0000 7f380e7fc700 1 -- 172.21.15.27:0/3110507318 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3808011a80 con 0x7f38180c9040 2024-01-11T08:27:58.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.515+0000 7f380e7fc700 1 -- 172.21.15.27:0/3110507318 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380801a680 con 0x7f38180c9040 2024-01-11T08:27:58.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.515+0000 7f38207ff700 1 -- 172.21.15.27:0/3110507318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3808009420 con 0x7f38180c9040 2024-01-11T08:27:58.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.515+0000 7f38207ff700 1 -- 172.21.15.27:0/3110507318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38180b32b0 con 0x7f38180c9040 2024-01-11T08:27:58.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.517+0000 7f38207ff700 1 -- 172.21.15.27:0/3110507318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f381814dd10 con 0x7f38180c9040 2024-01-11T08:27:58.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.518+0000 7f380e7fc700 1 -- 172.21.15.27:0/3110507318 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 53) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f380801aab0 con 0x7f38180c9040 2024-01-11T08:27:58.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.518+0000 7f380e7fc700 1 --2- 172.21.15.27:0/3110507318 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f37fc067170 0x7f37fc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:58.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.519+0000 7f380e7fc700 1 -- 172.21.15.27:0/3110507318 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6020+0+0 (secure 0 0 0) 0x7f3808086570 con 0x7f38180c9040 2024-01-11T08:27:58.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.519+0000 7f381e59b700 1 -- 172.21.15.27:0/3110507318 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f37fc067170 msgr2=0x7f37fc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:27:58.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.519+0000 7f381e59b700 1 --2- 172.21.15.27:0/3110507318 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f37fc067170 0x7f37fc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:27:58.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.523+0000 7f380e7fc700 1 -- 172.21.15.27:0/3110507318 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3808053410 con 0x7f38180c9040 2024-01-11T08:27:58.588 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:27:58.589+0000 7fcdba6941c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:27:58.648 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:27:58.648+0000 7f54a36501c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:27:58.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.720+0000 7f381e59b700 1 -- 172.21.15.27:0/3110507318 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f37fc067170 msgr2=0x7f37fc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:27:58.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.720+0000 7f381e59b700 1 --2- 172.21.15.27:0/3110507318 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f37fc067170 0x7f37fc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:27:58.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.768+0000 7f38207ff700 1 -- 172.21.15.27:0/3110507318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f381806c840 con 0x7f38180c9040 2024-01-11T08:27:58.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.769+0000 7f380e7fc700 1 -- 172.21.15.27:0/3110507318 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4593 (secure 0 0 0) 0x7f3808056a30 con 0x7f38180c9040 2024-01-11T08:27:58.769 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:27:58.769 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":50,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/921942372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":921942372},{"type":"v1","addr":"172.21.15.27:6834","nonce":921942372}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7080,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/3844286168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3844286168},{"type":"v1","addr":"172.21.15.77:6834","nonce":3844286168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7083,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3222788452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":3222788452},{"type":"v1","addr":"172.21.15.77:6837","nonce":3222788452}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:26:38.514309+0000","modified":"2024-01-11T08:26:42.031140+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7089},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7089":{"gid":7089,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/751423743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":751423743},{"type":"v1","addr":"172.21.15.27:6835","nonce":751423743}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2024-01-11T08:27:58.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.771+0000 7f38207ff700 1 -- 172.21.15.27:0/3110507318 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f37fc067170 msgr2=0x7f37fc069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:27:58.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.771+0000 7f38207ff700 1 --2- 172.21.15.27:0/3110507318 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f37fc067170 0x7f37fc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:58.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.771+0000 7f38207ff700 1 -- 172.21.15.27:0/3110507318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38180c9040 msgr2=0x7f38181502b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:58.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.771+0000 7f38207ff700 1 --2- 172.21.15.27:0/3110507318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38180c9040 0x7f38181502b0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f3818064980 tx=0x7f3808017000 comp rx=0 tx=0).stop 2024-01-11T08:27:58.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.772+0000 7f38207ff700 1 -- 172.21.15.27:0/3110507318 shutdown_connections 2024-01-11T08:27:58.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.772+0000 7f38207ff700 1 --2- 172.21.15.27:0/3110507318 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f37fc067170 0x7f37fc069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:58.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.772+0000 7f38207ff700 1 --2- 172.21.15.27:0/3110507318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38180cd410 0x7f381814e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:58.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.772+0000 7f38207ff700 1 --2- 172.21.15.27:0/3110507318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f38180c8c70 0x7f3818159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:58.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.772+0000 7f38207ff700 1 --2- 172.21.15.27:0/3110507318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38180c9040 0x7f38181502b0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:58.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.772+0000 7f38207ff700 1 -- 172.21.15.27:0/3110507318 >> 172.21.15.27:0/3110507318 conn(0x7f38180ce3e0 msgr2=0x7f381814cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:27:58.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.772+0000 7f38207ff700 1 -- 172.21.15.27:0/3110507318 shutdown_connections 2024-01-11T08:27:58.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.772+0000 7f38207ff700 1 -- 172.21.15.27:0/3110507318 wait complete. 2024-01-11T08:27:58.774 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 50 2024-01-11T08:27:58.790 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:27:58.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.993+0000 7fc3ebbff700 1 Processor -- start 2024-01-11T08:27:58.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.993+0000 7fc3ebbff700 1 -- start start 2024-01-11T08:27:58.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.994+0000 7fc3ebbff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3e40ad020 0x7fc3e40af320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:58.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.994+0000 7fc3ebbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc3e4159890 0x7fc3e40aa660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:58.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.994+0000 7fc3ebbff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e40aa1d0 con 0x7fc3e40ac370 2024-01-11T08:27:58.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.994+0000 7fc3ebbff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e40af860 con 0x7fc3e40ad020 2024-01-11T08:27:58.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.994+0000 7fc3ebbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e40ac8d0 con 0x7fc3e4159890 2024-01-11T08:27:58.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.995+0000 7fc3e999b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3e40ad020 0x7fc3e40af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:58.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.995+0000 7fc3e999b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3e40ad020 0x7fc3e40af320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56150/0 (socket says 172.21.15.27:56150) 2024-01-11T08:27:58.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.995+0000 7fc3e999b700 1 -- 172.21.15.27:0/1900138251 learned_addr learned my addr 172.21.15.27:0/1900138251 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:27:58.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.995+0000 7fc3ea19c700 1 --2- 172.21.15.27:0/1900138251 >> v2:172.21.15.77:3300/0 conn(0x7fc3e4159890 0x7fc3e40aa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:58.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.995+0000 7fc3e999b700 1 -- 172.21.15.27:0/1900138251 >> v2:172.21.15.77:3300/0 conn(0x7fc3e4159890 msgr2=0x7fc3e40aa660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:27:58.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.995+0000 7fc3e999b700 1 --2- 172.21.15.27:0/1900138251 >> v2:172.21.15.77:3300/0 conn(0x7fc3e4159890 0x7fc3e40aa660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:58.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.995+0000 7fc3e999b700 1 -- 172.21.15.27:0/1900138251 >> v1:172.21.15.77:6789/0 conn(0x7fc3e40ac370 legacy=0x7fc3e4159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:58.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.995+0000 7fc3e999b700 1 -- 172.21.15.27:0/1900138251 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e40a98a0 con 0x7fc3e40ad020 2024-01-11T08:27:58.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.995+0000 7fc3e8999700 1 -- 172.21.15.27:0/1900138251 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601620808 0 0) 0x7fc3e40aa1d0 con 0x7fc3e40ac370 2024-01-11T08:27:58.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.996+0000 7fc3e999b700 1 --2- 172.21.15.27:0/1900138251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3e40ad020 0x7fc3e40af320 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fc3e00099b0 tx=0x7fc3e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e86fe59e72dc3be0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:27:58.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.996+0000 7fc3e8999700 1 -- 172.21.15.27:0/1900138251 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0009ea0 con 0x7fc3e40ad020 2024-01-11T08:27:58.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.996+0000 7fc3e8999700 1 -- 172.21.15.27:0/1900138251 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e000b820 con 0x7fc3e40ad020 2024-01-11T08:27:58.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.996+0000 7fc3e8999700 1 -- 172.21.15.27:0/1900138251 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e000bbc0 con 0x7fc3e40ad020 2024-01-11T08:27:58.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.997+0000 7fc3ebbff700 1 -- 172.21.15.27:0/1900138251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3e40ad020 msgr2=0x7fc3e40af320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:58.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.997+0000 7fc3ebbff700 1 --2- 172.21.15.27:0/1900138251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3e40ad020 0x7fc3e40af320 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fc3e00099b0 tx=0x7fc3e0009250 comp rx=0 tx=0).stop 2024-01-11T08:27:58.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.997+0000 7fc3ebbff700 1 -- 172.21.15.27:0/1900138251 shutdown_connections 2024-01-11T08:27:58.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.997+0000 7fc3ebbff700 1 --2- 172.21.15.27:0/1900138251 >> v2:172.21.15.77:3300/0 conn(0x7fc3e4159890 0x7fc3e40aa660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:58.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.997+0000 7fc3ebbff700 1 --2- 172.21.15.27:0/1900138251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3e40ad020 0x7fc3e40af320 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:58.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.997+0000 7fc3ebbff700 1 -- 172.21.15.27:0/1900138251 >> 172.21.15.27:0/1900138251 conn(0x7fc3e40680e0 msgr2=0x7fc3e4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:27:58.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.997+0000 7fc3ebbff700 1 -- 172.21.15.27:0/1900138251 shutdown_connections 2024-01-11T08:27:58.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.997+0000 7fc3ebbff700 1 -- 172.21.15.27:0/1900138251 wait complete. 2024-01-11T08:27:58.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.998+0000 7fc3ebbff700 1 Processor -- start 2024-01-11T08:27:58.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:58.999+0000 7fc3ebbff700 1 -- start start 2024-01-11T08:27:59.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.000+0000 7fc3ebbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc3e40ac370 0x7fc3e4157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:59.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.000+0000 7fc3ebbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3e40ad020 0x7fc3e4152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:59.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.000+0000 7fc3ebbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3e4159890 0x7fc3e40cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:59.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.000+0000 7fc3ebbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e4155570 con 0x7fc3e4159890 2024-01-11T08:27:59.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.000+0000 7fc3ebbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e41557c0 con 0x7fc3e40ad020 2024-01-11T08:27:59.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.000+0000 7fc3ebbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e4155aa0 con 0x7fc3e40ac370 2024-01-11T08:27:59.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.000+0000 7fc3ea19c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3e4159890 0x7fc3e40cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:59.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.000+0000 7fc3e999b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc3e40ac370 0x7fc3e4157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:59.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.000+0000 7fc3ea19c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3e4159890 0x7fc3e40cbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56164/0 (socket says 172.21.15.27:56164) 2024-01-11T08:27:59.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.000+0000 7fc3ea19c700 1 -- 172.21.15.27:0/3974458109 learned_addr learned my addr 172.21.15.27:0/3974458109 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:27:59.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.000+0000 7fc3e919a700 1 --2- 172.21.15.27:0/3974458109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3e40ad020 0x7fc3e4152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:59.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.001+0000 7fc3e999b700 1 -- 172.21.15.27:0/3974458109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3e40ad020 msgr2=0x7fc3e4152ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:59.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.001+0000 7fc3e999b700 1 --2- 172.21.15.27:0/3974458109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3e40ad020 0x7fc3e4152ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.001+0000 7fc3e999b700 1 -- 172.21.15.27:0/3974458109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3e4159890 msgr2=0x7fc3e40cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:59.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.001+0000 7fc3e999b700 1 --2- 172.21.15.27:0/3974458109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3e4159890 0x7fc3e40cbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.001+0000 7fc3e999b700 1 -- 172.21.15.27:0/3974458109 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e00094f0 con 0x7fc3e40ac370 2024-01-11T08:27:59.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.001+0000 7fc3ea19c700 1 --2- 172.21.15.27:0/3974458109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3e4159890 0x7fc3e40cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:27:59.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.001+0000 7fc3e999b700 1 --2- 172.21.15.27:0/3974458109 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc3e40ac370 0x7fc3e4157a70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4066130 tx=0x7fc3e0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:27:59.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.001+0000 7fc3d9ffb700 1 -- 172.21.15.27:0/3974458109 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0000d10 con 0x7fc3e40ac370 2024-01-11T08:27:59.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.001+0000 7fc3d9ffb700 1 -- 172.21.15.27:0/3974458109 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e000bba0 con 0x7fc3e40ac370 2024-01-11T08:27:59.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.001+0000 7fc3d9ffb700 1 -- 172.21.15.27:0/3974458109 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0011650 con 0x7fc3e40ac370 2024-01-11T08:27:59.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.001+0000 7fc3ebbff700 1 -- 172.21.15.27:0/3974458109 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e4155d80 con 0x7fc3e40ac370 2024-01-11T08:27:59.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.002+0000 7fc3ebbff700 1 -- 172.21.15.27:0/3974458109 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e4156340 con 0x7fc3e40ac370 2024-01-11T08:27:59.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.003+0000 7fc3ebbff700 1 -- 172.21.15.27:0/3974458109 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e4156530 con 0x7fc3e40ac370 2024-01-11T08:27:59.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.004+0000 7fc3d9ffb700 1 -- 172.21.15.27:0/3974458109 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 53) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fc3e0011a00 con 0x7fc3e40ac370 2024-01-11T08:27:59.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.004+0000 7fc3d9ffb700 1 --2- 172.21.15.27:0/3974458109 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc3c8067170 0x7fc3c8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:59.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.004+0000 7fc3d9ffb700 1 -- 172.21.15.27:0/3974458109 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6020+0+0 (secure 0 0 0) 0x7fc3e0086250 con 0x7fc3e40ac370 2024-01-11T08:27:59.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.005+0000 7fc3e919a700 1 -- 172.21.15.27:0/3974458109 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc3c8067170 msgr2=0x7fc3c8069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:27:59.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.005+0000 7fc3e919a700 1 --2- 172.21.15.27:0/3974458109 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc3c8067170 0x7fc3c8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:27:59.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.009+0000 7fc3d9ffb700 1 -- 172.21.15.27:0/3974458109 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc3e00530d0 con 0x7fc3e40ac370 2024-01-11T08:27:59.113 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:27:59.114+0000 7fcdba6941c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:27:59.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.206+0000 7fc3e919a700 1 -- 172.21.15.27:0/3974458109 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc3c8067170 msgr2=0x7fc3c8069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:27:59.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.206+0000 7fc3e919a700 1 --2- 172.21.15.27:0/3974458109 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc3c8067170 0x7fc3c8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:27:59.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.220+0000 7fc3ebbff700 1 -- 172.21.15.27:0/3974458109 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3e40718d0 con 0x7fc3e40ac370 2024-01-11T08:27:59.220 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:27:59.221+0000 7f54a36501c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:27:59.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.221+0000 7fc3d9ffb700 1 -- 172.21.15.27:0/3974458109 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+15767 (secure 0 0 0) 0x7fc3e0011cb0 con 0x7fc3e40ac370 2024-01-11T08:27:59.221 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:27:59.222 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":104,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:27:55.854427+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:26:36.329359+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-11T08:26:37.504889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.d","mappings":[{"from":1,"to":3}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6836/41381":"2024-01-12T08:26:54.214924+0000","172.21.15.77:6835/41381":"2024-01-12T08:26:54.214924+0000","172.21.15.77:0/1971990124":"2024-01-12T08:26:54.214924+0000","172.21.15.77:0/673641283":"2024-01-12T08:26:54.214924+0000","172.21.15.77:0/3650013349":"2024-01-12T08:26:54.214924+0000","172.21.15.77:0/4016583196":"2024-01-12T08:26:54.214924+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:27:59.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.224+0000 7fc3ebbff700 1 -- 172.21.15.27:0/3974458109 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc3c8067170 msgr2=0x7fc3c8069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:27:59.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.224+0000 7fc3ebbff700 1 --2- 172.21.15.27:0/3974458109 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc3c8067170 0x7fc3c8069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.224+0000 7fc3ebbff700 1 -- 172.21.15.27:0/3974458109 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc3e40ac370 msgr2=0x7fc3e4157a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:59.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.224+0000 7fc3ebbff700 1 --2- 172.21.15.27:0/3974458109 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc3e40ac370 0x7fc3e4157a70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4066130 tx=0x7fc3e0017000 comp rx=0 tx=0).stop 2024-01-11T08:27:59.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.224+0000 7fc3ebbff700 1 -- 172.21.15.27:0/3974458109 shutdown_connections 2024-01-11T08:27:59.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.224+0000 7fc3ebbff700 1 --2- 172.21.15.27:0/3974458109 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc3c8067170 0x7fc3c8069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.224+0000 7fc3ebbff700 1 --2- 172.21.15.27:0/3974458109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3e4159890 0x7fc3e40cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.224+0000 7fc3ebbff700 1 --2- 172.21.15.27:0/3974458109 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc3e40ac370 0x7fc3e4157a70 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.224+0000 7fc3ebbff700 1 --2- 172.21.15.27:0/3974458109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3e40ad020 0x7fc3e4152ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.224+0000 7fc3ebbff700 1 -- 172.21.15.27:0/3974458109 >> 172.21.15.27:0/3974458109 conn(0x7fc3e40680e0 msgr2=0x7fc3e4159180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:27:59.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.224+0000 7fc3ebbff700 1 -- 172.21.15.27:0/3974458109 shutdown_connections 2024-01-11T08:27:59.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.224+0000 7fc3ebbff700 1 -- 172.21.15.27:0/3974458109 wait complete. 2024-01-11T08:27:59.243 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T08:26:36.329359+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T08:26:37.504889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T08:27:59.244 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:27:59.283 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:27:59.285+0000 7fcdba6941c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:27:59.404 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:27:59.404+0000 7f54a36501c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:27:59.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.449+0000 7f225e3ff700 1 Processor -- start 2024-01-11T08:27:59.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.449+0000 7f225e3ff700 1 -- start start 2024-01-11T08:27:59.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.451+0000 7f225e3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2258143190 0x7f22580cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:59.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.451+0000 7f225e3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22580cb4b0 0x7f22580cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:59.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.451+0000 7f225e3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225815d320 con 0x7f2258142dc0 2024-01-11T08:27:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.451+0000 7f225e3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f225815df30 con 0x7f2258143190 2024-01-11T08:27:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.451+0000 7f225e3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225815d590 con 0x7f22580cb4b0 2024-01-11T08:27:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.451+0000 7f22577fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2258143190 0x7f22580cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.451+0000 7f22577fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2258143190 0x7f22580cc020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56178/0 (socket says 172.21.15.27:56178) 2024-01-11T08:27:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.451+0000 7f22577fe700 1 -- 172.21.15.27:0/2583894737 learned_addr learned my addr 172.21.15.27:0/2583894737 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:27:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.451+0000 7f225c99c700 1 --2- 172.21.15.27:0/2583894737 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22580cb4b0 0x7f22580cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.452+0000 7f22577fe700 1 -- 172.21.15.27:0/2583894737 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22580cb4b0 msgr2=0x7f22580cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:27:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.452+0000 7f22577fe700 1 --2- 172.21.15.27:0/2583894737 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22580cb4b0 0x7f22580cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.452+0000 7f22577fe700 1 -- 172.21.15.27:0/2583894737 >> v1:172.21.15.27:6789/0 conn(0x7f2258142dc0 legacy=0x7f22580cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.452+0000 7f22577fe700 1 -- 172.21.15.27:0/2583894737 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225815d870 con 0x7f2258143190 2024-01-11T08:27:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.452+0000 7f22577fe700 1 --2- 172.21.15.27:0/2583894737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2258143190 0x7f22580cc020 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f224c0099b0 tx=0x7f224c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1459f09b681f75fa server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:27:59.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.452+0000 7f2256ffd700 1 -- 172.21.15.27:0/2583894737 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c009ea0 con 0x7f2258143190 2024-01-11T08:27:59.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.452+0000 7f2256ffd700 1 -- 172.21.15.27:0/2583894737 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224c00b860 con 0x7f2258143190 2024-01-11T08:27:59.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.452+0000 7f2256ffd700 1 -- 172.21.15.27:0/2583894737 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c00bc00 con 0x7f2258143190 2024-01-11T08:27:59.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.453+0000 7f225e3ff700 1 -- 172.21.15.27:0/2583894737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2258143190 msgr2=0x7f22580cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:59.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.453+0000 7f225e3ff700 1 --2- 172.21.15.27:0/2583894737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2258143190 0x7f22580cc020 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f224c0099b0 tx=0x7f224c009250 comp rx=0 tx=0).stop 2024-01-11T08:27:59.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.453+0000 7f225e3ff700 1 -- 172.21.15.27:0/2583894737 shutdown_connections 2024-01-11T08:27:59.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.453+0000 7f225e3ff700 1 --2- 172.21.15.27:0/2583894737 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22580cb4b0 0x7f22580cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.453+0000 7f225e3ff700 1 --2- 172.21.15.27:0/2583894737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2258143190 0x7f22580cc020 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.453+0000 7f225e3ff700 1 -- 172.21.15.27:0/2583894737 >> 172.21.15.27:0/2583894737 conn(0x7f2258063ca0 msgr2=0x7f225806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:27:59.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.453+0000 7f225e3ff700 1 -- 172.21.15.27:0/2583894737 shutdown_connections 2024-01-11T08:27:59.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.453+0000 7f225e3ff700 1 -- 172.21.15.27:0/2583894737 wait complete. 2024-01-11T08:27:59.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.455+0000 7f225e3ff700 1 Processor -- start 2024-01-11T08:27:59.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.455+0000 7f225e3ff700 1 -- start start 2024-01-11T08:27:59.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.456+0000 7f225e3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22580cb4b0 0x7f22580c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:59.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.456+0000 7f225e3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2258142dc0 0x7f225815b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:59.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.456+0000 7f225e3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2258143190 0x7f225815be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.456+0000 7f225e3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225816a160 con 0x7f2258143190 2024-01-11T08:27:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.456+0000 7f225e3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22580c7750 con 0x7f2258142dc0 2024-01-11T08:27:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.456+0000 7f225e3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22580c7a30 con 0x7f22580cb4b0 2024-01-11T08:27:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.456+0000 7f225c99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2258143190 0x7f225815be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.456+0000 7f22577fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2258142dc0 0x7f225815b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.456+0000 7f2257fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22580cb4b0 0x7f22580c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.456+0000 7f225c99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2258143190 0x7f225815be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56184/0 (socket says 172.21.15.27:56184) 2024-01-11T08:27:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.456+0000 7f225c99c700 1 -- 172.21.15.27:0/3630237640 learned_addr learned my addr 172.21.15.27:0/3630237640 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:27:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.457+0000 7f2257fff700 1 -- 172.21.15.27:0/3630237640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2258142dc0 msgr2=0x7f225815b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.457+0000 7f2257fff700 1 --2- 172.21.15.27:0/3630237640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2258142dc0 0x7f225815b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.457+0000 7f2257fff700 1 -- 172.21.15.27:0/3630237640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2258143190 msgr2=0x7f225815be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.457+0000 7f2257fff700 1 --2- 172.21.15.27:0/3630237640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2258143190 0x7f225815be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.457+0000 7f2257fff700 1 -- 172.21.15.27:0/3630237640 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224c0094f0 con 0x7f22580cb4b0 2024-01-11T08:27:59.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.457+0000 7f225c99c700 1 --2- 172.21.15.27:0/3630237640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2258143190 0x7f225815be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:27:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.457+0000 7f2257fff700 1 --2- 172.21.15.27:0/3630237640 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22580cb4b0 0x7f22580c9770 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f224800c590 tx=0x7f2248007ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:27:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.457+0000 7f223ffff700 1 -- 172.21.15.27:0/3630237640 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2248011590 con 0x7f22580cb4b0 2024-01-11T08:27:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.457+0000 7f223ffff700 1 -- 172.21.15.27:0/3630237640 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2248011710 con 0x7f22580cb4b0 2024-01-11T08:27:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.457+0000 7f225e3ff700 1 -- 172.21.15.27:0/3630237640 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22580c7d10 con 0x7f22580cb4b0 2024-01-11T08:27:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.457+0000 7f223ffff700 1 -- 172.21.15.27:0/3630237640 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2248011ad0 con 0x7f22580cb4b0 2024-01-11T08:27:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.457+0000 7f225e3ff700 1 -- 172.21.15.27:0/3630237640 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22580c9cb0 con 0x7f22580cb4b0 2024-01-11T08:27:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.458+0000 7f225e3ff700 1 -- 172.21.15.27:0/3630237640 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f225804c3c0 con 0x7f22580cb4b0 2024-01-11T08:27:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.459+0000 7f223ffff700 1 -- 172.21.15.27:0/3630237640 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 53) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f224801b3b0 con 0x7f22580cb4b0 2024-01-11T08:27:59.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.460+0000 7f223ffff700 1 --2- 172.21.15.27:0/3630237640 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2244067170 0x7f2244069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:59.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.460+0000 7f223ffff700 1 -- 172.21.15.27:0/3630237640 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6020+0+0 (secure 0 0 0) 0x7f2248083e20 con 0x7f22580cb4b0 2024-01-11T08:27:59.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.460+0000 7f22577fe700 1 -- 172.21.15.27:0/3630237640 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2244067170 msgr2=0x7f2244069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:27:59.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.460+0000 7f22577fe700 1 --2- 172.21.15.27:0/3630237640 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2244067170 0x7f2244069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:27:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.464+0000 7f223ffff700 1 -- 172.21.15.27:0/3630237640 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2248050c20 con 0x7f22580cb4b0 2024-01-11T08:27:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.661+0000 7f22577fe700 1 -- 172.21.15.27:0/3630237640 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2244067170 msgr2=0x7f2244069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:27:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.661+0000 7f22577fe700 1 --2- 172.21.15.27:0/3630237640 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2244067170 0x7f2244069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:27:59.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.668+0000 7f225e3ff700 1 -- 172.21.15.27:0/3630237640 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f22580c7b70 con 0x7f22580cb4b0 2024-01-11T08:27:59.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.669+0000 7f223ffff700 1 -- 172.21.15.27:0/3630237640 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7f22580c7b70 con 0x7f22580cb4b0 2024-01-11T08:27:59.669 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:27:59.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.671+0000 7f225e3ff700 1 -- 172.21.15.27:0/3630237640 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2244067170 msgr2=0x7f2244069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:27:59.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.671+0000 7f225e3ff700 1 --2- 172.21.15.27:0/3630237640 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2244067170 0x7f2244069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.671+0000 7f225e3ff700 1 -- 172.21.15.27:0/3630237640 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22580cb4b0 msgr2=0x7f22580c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:59.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.671+0000 7f225e3ff700 1 --2- 172.21.15.27:0/3630237640 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22580cb4b0 0x7f22580c9770 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f224800c590 tx=0x7f2248007ec0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.671+0000 7f225e3ff700 1 -- 172.21.15.27:0/3630237640 shutdown_connections 2024-01-11T08:27:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.671+0000 7f225e3ff700 1 --2- 172.21.15.27:0/3630237640 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2244067170 0x7f2244069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.671+0000 7f225e3ff700 1 --2- 172.21.15.27:0/3630237640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2258143190 0x7f225815be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.671+0000 7f225e3ff700 1 --2- 172.21.15.27:0/3630237640 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f22580cb4b0 0x7f22580c9770 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.671+0000 7f225e3ff700 1 --2- 172.21.15.27:0/3630237640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2258142dc0 0x7f225815b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.671+0000 7f225e3ff700 1 -- 172.21.15.27:0/3630237640 >> 172.21.15.27:0/3630237640 conn(0x7f2258063ca0 msgr2=0x7f2258141fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:27:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.671+0000 7f225e3ff700 1 -- 172.21.15.27:0/3630237640 shutdown_connections 2024-01-11T08:27:59.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.671+0000 7f225e3ff700 1 -- 172.21.15.27:0/3630237640 wait complete. 2024-01-11T08:27:59.690 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T08:27:59.699 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:27:59.700+0000 7fcdba6941c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:27:59.866 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:27:59.866+0000 7f54a36501c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:27:59.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.898+0000 7f8c745ff700 1 Processor -- start 2024-01-11T08:27:59.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.898+0000 7f8c745ff700 1 -- start start 2024-01-11T08:27:59.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.899+0000 7f8c745ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c6c13ff80 0x7f8c6c15a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:59.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.899+0000 7f8c745ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6c0c6b30 con 0x7f8c6c15a560 2024-01-11T08:27:59.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.899+0000 7f8c745ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6c0c6800 con 0x7f8c6c13fbb0 2024-01-11T08:27:59.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.899+0000 7f8c745ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c6c15a960 con 0x7f8c6c13ff80 2024-01-11T08:27:59.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.900+0000 7f8c71b9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c6c13ff80 0x7f8c6c15a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:59.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.900+0000 7f8c7239b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8c6c13fbb0 0x7f8c6c0ca7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:37546/0 (socket says 172.21.15.27:37546) 2024-01-11T08:27:59.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.900+0000 7f8c7239b700 1 -- 172.21.15.27:0/1068839967 learned_addr learned my addr 172.21.15.27:0/1068839967 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:27:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.900+0000 7f8c71399700 1 -- 172.21.15.27:0/1068839967 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842680138 0 0) 0x7f8c6c0c6800 con 0x7f8c6c13fbb0 2024-01-11T08:27:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.900+0000 7f8c71399700 1 -- 172.21.15.27:0/1068839967 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c58003570 con 0x7f8c6c13fbb0 2024-01-11T08:27:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.900+0000 7f8c71b9a700 1 -- 172.21.15.27:0/1068839967 >> v1:172.21.15.77:6789/0 conn(0x7f8c6c13fbb0 legacy=0x7f8c6c0ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.900+0000 7f8c71b9a700 1 -- 172.21.15.27:0/1068839967 >> v1:172.21.15.27:6789/0 conn(0x7f8c6c15a560 legacy=0x7f8c6c0c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.900+0000 7f8c71b9a700 1 -- 172.21.15.27:0/1068839967 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6c0cb240 con 0x7f8c6c13ff80 2024-01-11T08:27:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.901+0000 7f8c71b9a700 1 --2- 172.21.15.27:0/1068839967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c13ff80 0x7f8c6c15a020 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c0098e0 tx=0x7f8c5c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44cdccd4b34f4019 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:27:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.901+0000 7f8c71399700 1 -- 172.21.15.27:0/1068839967 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c00b640 con 0x7f8c6c13ff80 2024-01-11T08:27:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.901+0000 7f8c71399700 1 -- 172.21.15.27:0/1068839967 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c5c00b7a0 con 0x7f8c6c13ff80 2024-01-11T08:27:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.901+0000 7f8c71399700 1 -- 172.21.15.27:0/1068839967 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c00bb40 con 0x7f8c6c13ff80 2024-01-11T08:27:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.901+0000 7f8c745ff700 1 -- 172.21.15.27:0/1068839967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c13ff80 msgr2=0x7f8c6c15a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:59.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.901+0000 7f8c745ff700 1 --2- 172.21.15.27:0/1068839967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c13ff80 0x7f8c6c15a020 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c0098e0 tx=0x7f8c5c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.902+0000 7f8c745ff700 1 -- 172.21.15.27:0/1068839967 shutdown_connections 2024-01-11T08:27:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.902+0000 7f8c745ff700 1 --2- 172.21.15.27:0/1068839967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c13ff80 0x7f8c6c15a020 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.902+0000 7f8c745ff700 1 -- 172.21.15.27:0/1068839967 >> 172.21.15.27:0/1068839967 conn(0x7f8c6c0680a0 msgr2=0x7f8c6c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:27:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.902+0000 7f8c745ff700 1 -- 172.21.15.27:0/1068839967 shutdown_connections 2024-01-11T08:27:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.902+0000 7f8c745ff700 1 -- 172.21.15.27:0/1068839967 wait complete. 2024-01-11T08:27:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.903+0000 7f8c745ff700 1 Processor -- start 2024-01-11T08:27:59.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.903+0000 7f8c745ff700 1 -- start start 2024-01-11T08:27:59.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.904+0000 7f8c745ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c6c13fbb0 0x7f8c6c069790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:59.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.904+0000 7f8c745ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c13ff80 0x7f8c6c157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:59.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.904+0000 7f8c745ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c6c15a560 0x7f8c6c169230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:59.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.904+0000 7f8c745ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c0ae390 con 0x7f8c6c13ff80 2024-01-11T08:27:59.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.904+0000 7f8c745ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c0c3270 con 0x7f8c6c15a560 2024-01-11T08:27:59.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.904+0000 7f8c745ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c0c3550 con 0x7f8c6c13fbb0 2024-01-11T08:27:59.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.905+0000 7f8c71b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c13ff80 0x7f8c6c157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:59.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.905+0000 7f8c7239b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c6c13fbb0 0x7f8c6c069790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:59.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.905+0000 7f8c71b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c13ff80 0x7f8c6c157530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56198/0 (socket says 172.21.15.27:56198) 2024-01-11T08:27:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.905+0000 7f8c71b9a700 1 -- 172.21.15.27:0/1697516952 learned_addr learned my addr 172.21.15.27:0/1697516952 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:27:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.905+0000 7f8c72b9c700 1 --2- 172.21.15.27:0/1697516952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c6c15a560 0x7f8c6c169230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:27:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.905+0000 7f8c71b9a700 1 -- 172.21.15.27:0/1697516952 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c6c13fbb0 msgr2=0x7f8c6c069790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.905+0000 7f8c71b9a700 1 --2- 172.21.15.27:0/1697516952 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c6c13fbb0 0x7f8c6c069790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.905+0000 7f8c71b9a700 1 -- 172.21.15.27:0/1697516952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c6c15a560 msgr2=0x7f8c6c169230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:27:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.905+0000 7f8c71b9a700 1 --2- 172.21.15.27:0/1697516952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c6c15a560 0x7f8c6c169230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:27:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.905+0000 7f8c71b9a700 1 -- 172.21.15.27:0/1697516952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5c009420 con 0x7f8c6c13ff80 2024-01-11T08:27:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.905+0000 7f8c71b9a700 1 --2- 172.21.15.27:0/1697516952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c13ff80 0x7f8c6c157530 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c069490 tx=0x7f8c5c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:27:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.906+0000 7f8c627fc700 1 -- 172.21.15.27:0/1697516952 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c000d10 con 0x7f8c6c13ff80 2024-01-11T08:27:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.906+0000 7f8c627fc700 1 -- 172.21.15.27:0/1697516952 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c5c00bb20 con 0x7f8c6c13ff80 2024-01-11T08:27:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.906+0000 7f8c627fc700 1 -- 172.21.15.27:0/1697516952 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c011670 con 0x7f8c6c13ff80 2024-01-11T08:27:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.906+0000 7f8c745ff700 1 -- 172.21.15.27:0/1697516952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c6c0c3830 con 0x7f8c6c13ff80 2024-01-11T08:27:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.906+0000 7f8c745ff700 1 -- 172.21.15.27:0/1697516952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c6c0c3df0 con 0x7f8c6c13ff80 2024-01-11T08:27:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.907+0000 7f8c745ff700 1 -- 172.21.15.27:0/1697516952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c6c04c3c0 con 0x7f8c6c13ff80 2024-01-11T08:27:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.908+0000 7f8c627fc700 1 -- 172.21.15.27:0/1697516952 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 53) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f8c5c0119e0 con 0x7f8c6c13ff80 2024-01-11T08:27:59.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.909+0000 7f8c627fc700 1 --2- 172.21.15.27:0/1697516952 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8c50067170 0x7f8c50069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:27:59.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.909+0000 7f8c627fc700 1 -- 172.21.15.27:0/1697516952 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6020+0+0 (secure 0 0 0) 0x7f8c5c085ff0 con 0x7f8c6c13ff80 2024-01-11T08:27:59.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.909+0000 7f8c7239b700 1 -- 172.21.15.27:0/1697516952 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8c50067170 msgr2=0x7f8c50069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:27:59.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.909+0000 7f8c7239b700 1 --2- 172.21.15.27:0/1697516952 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8c50067170 0x7f8c50069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:27:59.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:27:59.913+0000 7f8c627fc700 1 -- 172.21.15.27:0/1697516952 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8c5c052e10 con 0x7f8c6c13ff80 2024-01-11T08:28:00.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.110+0000 7f8c7239b700 1 -- 172.21.15.27:0/1697516952 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8c50067170 msgr2=0x7f8c50069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:00.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.110+0000 7f8c7239b700 1 --2- 172.21.15.27:0/1697516952 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8c50067170 0x7f8c50069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:28:00.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.119+0000 7f8c745ff700 1 -- 172.21.15.27:0/1697516952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8c6c0cb520 con 0x7f8c6c13ff80 2024-01-11T08:28:00.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.119+0000 7f8c627fc700 1 -- 172.21.15.27:0/1697516952 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v104) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8c5c056430 con 0x7f8c6c13ff80 2024-01-11T08:28:00.119 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T08:28:00.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.122+0000 7f8c745ff700 1 -- 172.21.15.27:0/1697516952 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8c50067170 msgr2=0x7f8c50069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:28:00.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.122+0000 7f8c745ff700 1 --2- 172.21.15.27:0/1697516952 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8c50067170 0x7f8c50069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:00.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.122+0000 7f8c745ff700 1 -- 172.21.15.27:0/1697516952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c13ff80 msgr2=0x7f8c6c157530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:00.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.122+0000 7f8c745ff700 1 --2- 172.21.15.27:0/1697516952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c13ff80 0x7f8c6c157530 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c069490 tx=0x7f8c5c017000 comp rx=0 tx=0).stop 2024-01-11T08:28:00.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.122+0000 7f8c745ff700 1 -- 172.21.15.27:0/1697516952 shutdown_connections 2024-01-11T08:28:00.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.122+0000 7f8c745ff700 1 --2- 172.21.15.27:0/1697516952 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8c50067170 0x7f8c50069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:00.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.122+0000 7f8c745ff700 1 --2- 172.21.15.27:0/1697516952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c6c15a560 0x7f8c6c169230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:00.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.122+0000 7f8c745ff700 1 --2- 172.21.15.27:0/1697516952 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c6c13fbb0 0x7f8c6c069790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:00.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.122+0000 7f8c745ff700 1 --2- 172.21.15.27:0/1697516952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c13ff80 0x7f8c6c157530 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:00.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.122+0000 7f8c745ff700 1 -- 172.21.15.27:0/1697516952 >> 172.21.15.27:0/1697516952 conn(0x7f8c6c0680a0 msgr2=0x7f8c6c0c0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:00.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.122+0000 7f8c745ff700 1 -- 172.21.15.27:0/1697516952 shutdown_connections 2024-01-11T08:28:00.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.122+0000 7f8c745ff700 1 -- 172.21.15.27:0/1697516952 wait complete. 2024-01-11T08:28:00.146 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T08:28:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.357+0000 7fb1971ff700 1 Processor -- start 2024-01-11T08:28:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.358+0000 7fb1971ff700 1 -- start start 2024-01-11T08:28:00.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.359+0000 7fb1971ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb190067860 0x7fb19006c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:00.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.359+0000 7fb1971ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb190067c30 0x7fb1900c7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:00.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.359+0000 7fb1971ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19006c3c0 con 0x7fb19006cd90 2024-01-11T08:28:00.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.359+0000 7fb1971ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1900cb9c0 con 0x7fb190067860 2024-01-11T08:28:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.359+0000 7fb1971ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb19006f650 con 0x7fb190067c30 2024-01-11T08:28:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.359+0000 7fb194f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb190067860 0x7fb19006c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.359+0000 7fb194f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb190067860 0x7fb19006c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56200/0 (socket says 172.21.15.27:56200) 2024-01-11T08:28:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.359+0000 7fb194f9b700 1 -- 172.21.15.27:0/1759800076 learned_addr learned my addr 172.21.15.27:0/1759800076 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.359+0000 7fb18ffff700 1 --1- 172.21.15.27:0/1759800076 >> v1:172.21.15.27:6789/0 conn(0x7fb19006cd90 0x7fb1900cae90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53554/0 (socket says 172.21.15.27:53554) 2024-01-11T08:28:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.359+0000 7fb19579c700 1 --2- 172.21.15.27:0/1759800076 >> v2:172.21.15.77:3300/0 conn(0x7fb190067c30 0x7fb1900c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.360+0000 7fb194f9b700 1 -- 172.21.15.27:0/1759800076 >> v2:172.21.15.77:3300/0 conn(0x7fb190067c30 msgr2=0x7fb1900c7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.360+0000 7fb194f9b700 1 --2- 172.21.15.27:0/1759800076 >> v2:172.21.15.77:3300/0 conn(0x7fb190067c30 0x7fb1900c7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.360+0000 7fb194f9b700 1 -- 172.21.15.27:0/1759800076 >> v1:172.21.15.27:6789/0 conn(0x7fb19006cd90 legacy=0x7fb1900cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.360+0000 7fb194f9b700 1 -- 172.21.15.27:0/1759800076 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19006ba90 con 0x7fb190067860 2024-01-11T08:28:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.360+0000 7fb194f9b700 1 --2- 172.21.15.27:0/1759800076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb190067860 0x7fb19006c850 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fb1900cd260 tx=0x7fb184009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c22e8ce973934e05 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.360+0000 7fb18f7fe700 1 -- 172.21.15.27:0/1759800076 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb184009e70 con 0x7fb190067860 2024-01-11T08:28:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.360+0000 7fb18f7fe700 1 -- 172.21.15.27:0/1759800076 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18400b860 con 0x7fb190067860 2024-01-11T08:28:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.360+0000 7fb18f7fe700 1 -- 172.21.15.27:0/1759800076 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18400bc00 con 0x7fb190067860 2024-01-11T08:28:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.361+0000 7fb1971ff700 1 -- 172.21.15.27:0/1759800076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb190067860 msgr2=0x7fb19006c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.361+0000 7fb1971ff700 1 --2- 172.21.15.27:0/1759800076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb190067860 0x7fb19006c850 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fb1900cd260 tx=0x7fb184009250 comp rx=0 tx=0).stop 2024-01-11T08:28:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.361+0000 7fb1971ff700 1 -- 172.21.15.27:0/1759800076 shutdown_connections 2024-01-11T08:28:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.361+0000 7fb1971ff700 1 --2- 172.21.15.27:0/1759800076 >> v2:172.21.15.77:3300/0 conn(0x7fb190067c30 0x7fb1900c7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.361+0000 7fb1971ff700 1 --2- 172.21.15.27:0/1759800076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb190067860 0x7fb19006c850 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.361+0000 7fb1971ff700 1 -- 172.21.15.27:0/1759800076 >> 172.21.15.27:0/1759800076 conn(0x7fb19006b1f0 msgr2=0x7fb1900cd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.361+0000 7fb1971ff700 1 -- 172.21.15.27:0/1759800076 shutdown_connections 2024-01-11T08:28:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.361+0000 7fb1971ff700 1 -- 172.21.15.27:0/1759800076 wait complete. 2024-01-11T08:28:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.363+0000 7fb1971ff700 1 Processor -- start 2024-01-11T08:28:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.363+0000 7fb1971ff700 1 -- start start 2024-01-11T08:28:00.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.364+0000 7fb1971ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb190067860 0x7fb19016a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:00.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.364+0000 7fb1971ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb190067c30 0x7fb19015cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:00.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.364+0000 7fb1971ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb19006cd90 0x7fb1900b0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:00.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.364+0000 7fb1971ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1900ad9d0 con 0x7fb19006cd90 2024-01-11T08:28:00.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.364+0000 7fb1971ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19016a7b0 con 0x7fb190067c30 2024-01-11T08:28:00.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.364+0000 7fb1971ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1900acae0 con 0x7fb190067860 2024-01-11T08:28:00.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.364+0000 7fb19579c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb19006cd90 0x7fb1900b0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:00.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.364+0000 7fb19579c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb19006cd90 0x7fb1900b0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56212/0 (socket says 172.21.15.27:56212) 2024-01-11T08:28:00.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.364+0000 7fb19579c700 1 -- 172.21.15.27:0/291170244 learned_addr learned my addr 172.21.15.27:0/291170244 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:00.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.364+0000 7fb194f9b700 1 --2- 172.21.15.27:0/291170244 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb190067860 0x7fb19016a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:00.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.364+0000 7fb18ffff700 1 --2- 172.21.15.27:0/291170244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb190067c30 0x7fb19015cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:00.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.364+0000 7fb19579c700 1 -- 172.21.15.27:0/291170244 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb190067860 msgr2=0x7fb19016a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:00.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.364+0000 7fb19579c700 1 --2- 172.21.15.27:0/291170244 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb190067860 0x7fb19016a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:00.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.364+0000 7fb19579c700 1 -- 172.21.15.27:0/291170244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb190067c30 msgr2=0x7fb19015cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:00.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.364+0000 7fb19579c700 1 --2- 172.21.15.27:0/291170244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb190067c30 0x7fb19015cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:00.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.364+0000 7fb19579c700 1 -- 172.21.15.27:0/291170244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1840094f0 con 0x7fb19006cd90 2024-01-11T08:28:00.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.365+0000 7fb19579c700 1 --2- 172.21.15.27:0/291170244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb19006cd90 0x7fb1900b0060 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fb188009e90 tx=0x7fb18800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:00.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.365+0000 7fb18cff9700 1 -- 172.21.15.27:0/291170244 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18800cb70 con 0x7fb19006cd90 2024-01-11T08:28:00.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.365+0000 7fb18cff9700 1 -- 172.21.15.27:0/291170244 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18800f040 con 0x7fb19006cd90 2024-01-11T08:28:00.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.365+0000 7fb1971ff700 1 -- 172.21.15.27:0/291170244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1900acdc0 con 0x7fb19006cd90 2024-01-11T08:28:00.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.365+0000 7fb18cff9700 1 -- 172.21.15.27:0/291170244 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb188014480 con 0x7fb19006cd90 2024-01-11T08:28:00.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.365+0000 7fb1971ff700 1 -- 172.21.15.27:0/291170244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1900ad380 con 0x7fb19006cd90 2024-01-11T08:28:00.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.366+0000 7fb17a7fc700 1 -- 172.21.15.27:0/291170244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb19004c3c0 con 0x7fb19006cd90 2024-01-11T08:28:00.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.367+0000 7fb18cff9700 1 -- 172.21.15.27:0/291170244 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 53) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fb188014830 con 0x7fb19006cd90 2024-01-11T08:28:00.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.368+0000 7fb18cff9700 1 --2- 172.21.15.27:0/291170244 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb174067170 0x7fb174069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:00.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.368+0000 7fb18cff9700 1 -- 172.21.15.27:0/291170244 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6020+0+0 (secure 0 0 0) 0x7fb1880854b0 con 0x7fb19006cd90 2024-01-11T08:28:00.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.368+0000 7fb194f9b700 1 -- 172.21.15.27:0/291170244 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb174067170 msgr2=0x7fb174069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:00.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.368+0000 7fb194f9b700 1 --2- 172.21.15.27:0/291170244 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb174067170 0x7fb174069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:28:00.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.373+0000 7fb18cff9700 1 -- 172.21.15.27:0/291170244 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb188052240 con 0x7fb19006cd90 2024-01-11T08:28:00.483 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:00.485+0000 7fcdba6941c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:28:00.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.569+0000 7fb194f9b700 1 -- 172.21.15.27:0/291170244 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb174067170 msgr2=0x7fb174069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:00.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.569+0000 7fb194f9b700 1 --2- 172.21.15.27:0/291170244 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb174067170 0x7fb174069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:28:00.571 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:00.573+0000 7fcdba6941c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:28:00.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.573+0000 7fb17a7fc700 1 -- 172.21.15.27:0/291170244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb19006bd70 con 0x7fb19006cd90 2024-01-11T08:28:00.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.574+0000 7fb18cff9700 1 -- 172.21.15.27:0/291170244 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v104) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb188055860 con 0x7fb19006cd90 2024-01-11T08:28:00.574 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T08:28:00.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.576+0000 7fb17a7fc700 1 -- 172.21.15.27:0/291170244 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb174067170 msgr2=0x7fb174069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:28:00.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.576+0000 7fb17a7fc700 1 --2- 172.21.15.27:0/291170244 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb174067170 0x7fb174069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.576+0000 7fb17a7fc700 1 -- 172.21.15.27:0/291170244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb19006cd90 msgr2=0x7fb1900b0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.576+0000 7fb17a7fc700 1 --2- 172.21.15.27:0/291170244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb19006cd90 0x7fb1900b0060 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fb188009e90 tx=0x7fb18800b040 comp rx=0 tx=0).stop 2024-01-11T08:28:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.576+0000 7fb17a7fc700 1 -- 172.21.15.27:0/291170244 shutdown_connections 2024-01-11T08:28:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.576+0000 7fb17a7fc700 1 --2- 172.21.15.27:0/291170244 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb174067170 0x7fb174069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.576+0000 7fb17a7fc700 1 --2- 172.21.15.27:0/291170244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb19006cd90 0x7fb1900b0060 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.576+0000 7fb17a7fc700 1 --2- 172.21.15.27:0/291170244 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb190067860 0x7fb19016a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.576+0000 7fb17a7fc700 1 --2- 172.21.15.27:0/291170244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb190067c30 0x7fb19015cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.576+0000 7fb17a7fc700 1 -- 172.21.15.27:0/291170244 >> 172.21.15.27:0/291170244 conn(0x7fb19006b1f0 msgr2=0x7fb19006e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.576+0000 7fb17a7fc700 1 -- 172.21.15.27:0/291170244 shutdown_connections 2024-01-11T08:28:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.576+0000 7fb17a7fc700 1 -- 172.21.15.27:0/291170244 wait complete. 2024-01-11T08:28:00.596 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:28:00.723 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:00.723+0000 7f54a36501c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:28:00.727 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:00.728+0000 7fcdba6941c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:28:00.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.799+0000 7ff3947ff700 1 Processor -- start 2024-01-11T08:28:00.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.799+0000 7ff3947ff700 1 -- start start 2024-01-11T08:28:00.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.800+0000 7ff3947ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff38c0cc410 0x7ff38c159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:00.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.800+0000 7ff3947ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38c0cd730 con 0x7ff38c0b03e0 2024-01-11T08:28:00.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.800+0000 7ff3947ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38c0ccb20 con 0x7ff38c0b2b90 2024-01-11T08:28:00.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.800+0000 7ff3947ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff38c0cce00 con 0x7ff38c0cc410 2024-01-11T08:28:00.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.801+0000 7ff392d9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff38c0cc410 0x7ff38c159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:00.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.801+0000 7ff392d9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff38c0cc410 0x7ff38c159910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56228/0 (socket says 172.21.15.27:56228) 2024-01-11T08:28:00.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.801+0000 7ff392d9c700 1 -- 172.21.15.27:0/2363445100 learned_addr learned my addr 172.21.15.27:0/2363445100 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:00.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.801+0000 7ff391599700 1 -- 172.21.15.27:0/2363445100 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115933558 0 0) 0x7ff38c0ccb20 con 0x7ff38c0b2b90 2024-01-11T08:28:00.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.801+0000 7ff392d9c700 1 -- 172.21.15.27:0/2363445100 >> v1:172.21.15.77:6789/0 conn(0x7ff38c0b2b90 legacy=0x7ff38c159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:00.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.801+0000 7ff392d9c700 1 -- 172.21.15.27:0/2363445100 >> v1:172.21.15.27:6789/0 conn(0x7ff38c0b03e0 legacy=0x7ff38c0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:00.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.801+0000 7ff392d9c700 1 -- 172.21.15.27:0/2363445100 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c0cd0e0 con 0x7ff38c0cc410 2024-01-11T08:28:00.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.802+0000 7ff392d9c700 1 --2- 172.21.15.27:0/2363445100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff38c0cc410 0x7ff38c159910 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7ff384009f50 tx=0x7ff384009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2c80a1c8b034ce8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:00.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.802+0000 7ff391599700 1 -- 172.21.15.27:0/2363445100 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38400e070 con 0x7ff38c0cc410 2024-01-11T08:28:00.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.802+0000 7ff391599700 1 -- 172.21.15.27:0/2363445100 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38400ba70 con 0x7ff38c0cc410 2024-01-11T08:28:00.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.802+0000 7ff391599700 1 -- 172.21.15.27:0/2363445100 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3840092e0 con 0x7ff38c0cc410 2024-01-11T08:28:00.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.802+0000 7ff3947ff700 1 -- 172.21.15.27:0/2363445100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff38c0cc410 msgr2=0x7ff38c159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:00.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.802+0000 7ff3947ff700 1 --2- 172.21.15.27:0/2363445100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff38c0cc410 0x7ff38c159910 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7ff384009f50 tx=0x7ff384009840 comp rx=0 tx=0).stop 2024-01-11T08:28:00.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.802+0000 7ff3947ff700 1 -- 172.21.15.27:0/2363445100 shutdown_connections 2024-01-11T08:28:00.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.802+0000 7ff3947ff700 1 --2- 172.21.15.27:0/2363445100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff38c0cc410 0x7ff38c159910 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:00.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.802+0000 7ff3947ff700 1 -- 172.21.15.27:0/2363445100 >> 172.21.15.27:0/2363445100 conn(0x7ff38c06ed40 msgr2=0x7ff38c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:00.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.802+0000 7ff3947ff700 1 -- 172.21.15.27:0/2363445100 shutdown_connections 2024-01-11T08:28:00.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.803+0000 7ff3947ff700 1 -- 172.21.15.27:0/2363445100 wait complete. 2024-01-11T08:28:00.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.804+0000 7ff3947ff700 1 Processor -- start 2024-01-11T08:28:00.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.804+0000 7ff3947ff700 1 -- start start 2024-01-11T08:28:00.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.805+0000 7ff3947ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff38c0b03e0 0x7ff38c0c6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:00.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.805+0000 7ff3947ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff38c0b2b90 0x7ff38c0c4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:00.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.805+0000 7ff3947ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c0cc410 0x7ff38c0c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:00.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.805+0000 7ff3947ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c0c7760 con 0x7ff38c0b2b90 2024-01-11T08:28:00.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.805+0000 7ff3947ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c0c7a40 con 0x7ff38c0cc410 2024-01-11T08:28:00.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.805+0000 7ff3947ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c0c7d20 con 0x7ff38c0b03e0 2024-01-11T08:28:00.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.805+0000 7ff391d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff38c0b2b90 0x7ff38c0c4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:00.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.805+0000 7ff391d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff38c0b2b90 0x7ff38c0c4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56240/0 (socket says 172.21.15.27:56240) 2024-01-11T08:28:00.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.805+0000 7ff391d9a700 1 -- 172.21.15.27:0/4169753319 learned_addr learned my addr 172.21.15.27:0/4169753319 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:00.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.805+0000 7ff39259b700 1 --2- 172.21.15.27:0/4169753319 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff38c0b03e0 0x7ff38c0c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:00.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.805+0000 7ff392d9c700 1 --2- 172.21.15.27:0/4169753319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c0cc410 0x7ff38c0c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:00.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.806+0000 7ff391d9a700 1 -- 172.21.15.27:0/4169753319 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff38c0b03e0 msgr2=0x7ff38c0c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:00.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.806+0000 7ff391d9a700 1 --2- 172.21.15.27:0/4169753319 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff38c0b03e0 0x7ff38c0c6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:00.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.806+0000 7ff391d9a700 1 -- 172.21.15.27:0/4169753319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c0cc410 msgr2=0x7ff38c0c53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:00.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.806+0000 7ff391d9a700 1 --2- 172.21.15.27:0/4169753319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c0cc410 0x7ff38c0c53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:00.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.806+0000 7ff391d9a700 1 -- 172.21.15.27:0/4169753319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff384009ae0 con 0x7ff38c0b2b90 2024-01-11T08:28:00.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.806+0000 7ff391d9a700 1 --2- 172.21.15.27:0/4169753319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff38c0b2b90 0x7ff38c0c4e70 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00c5e0 tx=0x7ff37c007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:00.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.806+0000 7ff3827fc700 1 -- 172.21.15.27:0/4169753319 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c011590 con 0x7ff38c0b2b90 2024-01-11T08:28:00.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.806+0000 7ff3827fc700 1 -- 172.21.15.27:0/4169753319 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37c0116f0 con 0x7ff38c0b2b90 2024-01-11T08:28:00.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.806+0000 7ff3827fc700 1 -- 172.21.15.27:0/4169753319 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c011ab0 con 0x7ff38c0b2b90 2024-01-11T08:28:00.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.806+0000 7ff3947ff700 1 -- 172.21.15.27:0/4169753319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38c0c8000 con 0x7ff38c0b2b90 2024-01-11T08:28:00.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.806+0000 7ff3947ff700 1 -- 172.21.15.27:0/4169753319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38c159000 con 0x7ff38c0b2b90 2024-01-11T08:28:00.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.808+0000 7ff3947ff700 1 -- 172.21.15.27:0/4169753319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff38c0c8520 con 0x7ff38c0b2b90 2024-01-11T08:28:00.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.808+0000 7ff3827fc700 1 -- 172.21.15.27:0/4169753319 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 53) v1 ==== 84958+0+0 (secure 0 0 0) 0x7ff37c01b3b0 con 0x7ff38c0b2b90 2024-01-11T08:28:00.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.809+0000 7ff3827fc700 1 --2- 172.21.15.27:0/4169753319 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff370067170 0x7ff370069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:00.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.809+0000 7ff3827fc700 1 -- 172.21.15.27:0/4169753319 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6020+0+0 (secure 0 0 0) 0x7ff37c083e90 con 0x7ff38c0b2b90 2024-01-11T08:28:00.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.810+0000 7ff39259b700 1 -- 172.21.15.27:0/4169753319 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff370067170 msgr2=0x7ff370069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:00.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.810+0000 7ff39259b700 1 --2- 172.21.15.27:0/4169753319 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff370067170 0x7ff370069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:28:00.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:00.813+0000 7ff3827fc700 1 -- 172.21.15.27:0/4169753319 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff38c0c8520 con 0x7ff38c0b2b90 2024-01-11T08:28:00.814 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:00.815+0000 7fcdba6941c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:28:00.822 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:00.822+0000 7f54a36501c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:28:00.990 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:00.990+0000 7f54a36501c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:28:01.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.011+0000 7ff39259b700 1 -- 172.21.15.27:0/4169753319 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff370067170 msgr2=0x7ff370069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:01.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.011+0000 7ff39259b700 1 --2- 172.21.15.27:0/4169753319 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff370067170 0x7ff370069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:28:01.033 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:01.035+0000 7fcdba6941c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:28:01.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.063+0000 7ff3947ff700 1 -- 172.21.15.27:0/4169753319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff38c0c8520 con 0x7ff38c0b2b90 2024-01-11T08:28:01.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.064+0000 7ff3827fc700 1 -- 172.21.15.27:0/4169753319 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4593 (secure 0 0 0) 0x7ff38c0c8520 con 0x7ff38c0b2b90 2024-01-11T08:28:01.064 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:28:01.064 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":50,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/921942372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":921942372},{"type":"v1","addr":"172.21.15.27:6834","nonce":921942372}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7080,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/3844286168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3844286168},{"type":"v1","addr":"172.21.15.77:6834","nonce":3844286168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7083,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3222788452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":3222788452},{"type":"v1","addr":"172.21.15.77:6837","nonce":3222788452}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:26:38.514309+0000","modified":"2024-01-11T08:26:42.031140+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7089},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7089":{"gid":7089,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/751423743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":751423743},{"type":"v1","addr":"172.21.15.27:6835","nonce":751423743}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2024-01-11T08:28:01.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.066+0000 7ff3947ff700 1 -- 172.21.15.27:0/4169753319 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff370067170 msgr2=0x7ff370069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:28:01.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.066+0000 7ff3947ff700 1 --2- 172.21.15.27:0/4169753319 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff370067170 0x7ff370069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.066+0000 7ff3947ff700 1 -- 172.21.15.27:0/4169753319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff38c0b2b90 msgr2=0x7ff38c0c4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:01.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.066+0000 7ff3947ff700 1 --2- 172.21.15.27:0/4169753319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff38c0b2b90 0x7ff38c0c4e70 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00c5e0 tx=0x7ff37c007e60 comp rx=0 tx=0).stop 2024-01-11T08:28:01.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.067+0000 7ff3947ff700 1 -- 172.21.15.27:0/4169753319 shutdown_connections 2024-01-11T08:28:01.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.067+0000 7ff3947ff700 1 --2- 172.21.15.27:0/4169753319 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff370067170 0x7ff370069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.067+0000 7ff3947ff700 1 --2- 172.21.15.27:0/4169753319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff38c0cc410 0x7ff38c0c53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.067+0000 7ff3947ff700 1 --2- 172.21.15.27:0/4169753319 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff38c0b03e0 0x7ff38c0c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.067+0000 7ff3947ff700 1 --2- 172.21.15.27:0/4169753319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff38c0b2b90 0x7ff38c0c4e70 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.067+0000 7ff3947ff700 1 -- 172.21.15.27:0/4169753319 >> 172.21.15.27:0/4169753319 conn(0x7ff38c06ed40 msgr2=0x7ff38c0b2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:01.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.067+0000 7ff3947ff700 1 -- 172.21.15.27:0/4169753319 shutdown_connections 2024-01-11T08:28:01.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.067+0000 7ff3947ff700 1 -- 172.21.15.27:0/4169753319 wait complete. 2024-01-11T08:28:01.070 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 50 2024-01-11T08:28:01.085 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:28:01.086 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:01.085+0000 7f54a36501c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:28:01.129 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:01.130+0000 7fcdba6941c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:28:01.225 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:01.226+0000 7fcdba6941c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:28:01.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.291+0000 7fae849ff700 1 Processor -- start 2024-01-11T08:28:01.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.291+0000 7fae849ff700 1 -- start start 2024-01-11T08:28:01.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.292+0000 7fae849ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fae800ad020 0x7fae800af320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:01.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.292+0000 7fae849ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae800ac370 0x7fae80159180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:01.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.292+0000 7fae849ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fae8006b460 0x7fae801596c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:01.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.292+0000 7fae849ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae80159c00 con 0x7fae800ad020 2024-01-11T08:28:01.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.292+0000 7fae849ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae800aa1d0 con 0x7fae8006b460 2024-01-11T08:28:01.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.292+0000 7fae849ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae800af860 con 0x7fae800ac370 2024-01-11T08:28:01.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.292+0000 7fae7e59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fae800ad020 0x7fae800af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:01.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.293+0000 7fae7e59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fae800ad020 0x7fae800af320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56242/0 (socket says 172.21.15.27:56242) 2024-01-11T08:28:01.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.293+0000 7fae7e59c700 1 -- 172.21.15.27:0/3442912775 learned_addr learned my addr 172.21.15.27:0/3442912775 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:01.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.293+0000 7fae7ed9d700 1 --2- 172.21.15.27:0/3442912775 >> v2:172.21.15.77:3300/0 conn(0x7fae8006b460 0x7fae801596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:01.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.293+0000 7fae7dd9b700 1 --2- 172.21.15.27:0/3442912775 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae800ac370 0x7fae80159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:01.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.293+0000 7fae7e59c700 1 -- 172.21.15.27:0/3442912775 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae800ac370 msgr2=0x7fae80159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:01.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.293+0000 7fae7e59c700 1 --2- 172.21.15.27:0/3442912775 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae800ac370 0x7fae80159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.293+0000 7fae7e59c700 1 -- 172.21.15.27:0/3442912775 >> v2:172.21.15.77:3300/0 conn(0x7fae8006b460 msgr2=0x7fae801596c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:01.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.293+0000 7fae7e59c700 1 --2- 172.21.15.27:0/3442912775 >> v2:172.21.15.77:3300/0 conn(0x7fae8006b460 0x7fae801596c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.293+0000 7fae7e59c700 1 -- 172.21.15.27:0/3442912775 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae800ac8d0 con 0x7fae800ad020 2024-01-11T08:28:01.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.293+0000 7fae7ed9d700 1 --2- 172.21.15.27:0/3442912775 >> v2:172.21.15.77:3300/0 conn(0x7fae8006b460 0x7fae801596c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:28:01.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.293+0000 7fae7e59c700 1 --2- 172.21.15.27:0/3442912775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae800ad020 0x7fae800af320 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fae80066130 tx=0x7fae70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=378c3fe2d33c5ce5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:01.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.294+0000 7fae7d59a700 1 -- 172.21.15.27:0/3442912775 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7000b640 con 0x7fae800ad020 2024-01-11T08:28:01.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.294+0000 7fae7d59a700 1 -- 172.21.15.27:0/3442912775 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7000b7a0 con 0x7fae800ad020 2024-01-11T08:28:01.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.294+0000 7fae7d59a700 1 -- 172.21.15.27:0/3442912775 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7000bb60 con 0x7fae800ad020 2024-01-11T08:28:01.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.294+0000 7fae849ff700 1 -- 172.21.15.27:0/3442912775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae800ad020 msgr2=0x7fae800af320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:01.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.294+0000 7fae849ff700 1 --2- 172.21.15.27:0/3442912775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae800ad020 0x7fae800af320 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fae80066130 tx=0x7fae70009250 comp rx=0 tx=0).stop 2024-01-11T08:28:01.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.294+0000 7fae849ff700 1 -- 172.21.15.27:0/3442912775 shutdown_connections 2024-01-11T08:28:01.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.294+0000 7fae849ff700 1 --2- 172.21.15.27:0/3442912775 >> v2:172.21.15.77:3300/0 conn(0x7fae8006b460 0x7fae801596c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.294+0000 7fae849ff700 1 --2- 172.21.15.27:0/3442912775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae800ad020 0x7fae800af320 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.294+0000 7fae849ff700 1 --2- 172.21.15.27:0/3442912775 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae800ac370 0x7fae80159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.294+0000 7fae849ff700 1 -- 172.21.15.27:0/3442912775 >> 172.21.15.27:0/3442912775 conn(0x7fae800680e0 msgr2=0x7fae80069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:01.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.295+0000 7fae849ff700 1 -- 172.21.15.27:0/3442912775 shutdown_connections 2024-01-11T08:28:01.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.295+0000 7fae849ff700 1 -- 172.21.15.27:0/3442912775 wait complete. 2024-01-11T08:28:01.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.296+0000 7fae849ff700 1 Processor -- start 2024-01-11T08:28:01.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.296+0000 7fae849ff700 1 -- start start 2024-01-11T08:28:01.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.297+0000 7fae849ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae8006b460 0x7fae80143830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:01.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.297+0000 7fae849ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae800ac370 0x7fae800c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:01.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.297+0000 7fae849ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae800ad020 0x7fae800c0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:01.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.297+0000 7fae849ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae800c4050 con 0x7fae800ac370 2024-01-11T08:28:01.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.297+0000 7fae849ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae800c2580 con 0x7fae8006b460 2024-01-11T08:28:01.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.297+0000 7fae849ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae80143d70 con 0x7fae800ad020 2024-01-11T08:28:01.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.298+0000 7fae7e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae8006b460 0x7fae80143830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:01.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.298+0000 7fae7dd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae800ac370 0x7fae800c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:01.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.298+0000 7fae7e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae8006b460 0x7fae80143830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:47750/0 (socket says 172.21.15.27:47750) 2024-01-11T08:28:01.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.298+0000 7fae7e59c700 1 -- 172.21.15.27:0/712132202 learned_addr learned my addr 172.21.15.27:0/712132202 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:01.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.298+0000 7fae7ed9d700 1 --2- 172.21.15.27:0/712132202 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae800ad020 0x7fae800c0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:01.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.298+0000 7fae7dd9b700 1 -- 172.21.15.27:0/712132202 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae800ad020 msgr2=0x7fae800c0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:01.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.298+0000 7fae7dd9b700 1 --2- 172.21.15.27:0/712132202 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae800ad020 0x7fae800c0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.298+0000 7fae7dd9b700 1 -- 172.21.15.27:0/712132202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae8006b460 msgr2=0x7fae80143830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:01.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.298+0000 7fae7dd9b700 1 --2- 172.21.15.27:0/712132202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae8006b460 0x7fae80143830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.298+0000 7fae7dd9b700 1 -- 172.21.15.27:0/712132202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae78009a20 con 0x7fae800ac370 2024-01-11T08:28:01.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.298+0000 7fae7e59c700 1 --2- 172.21.15.27:0/712132202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae8006b460 0x7fae80143830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:28:01.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.298+0000 7fae7dd9b700 1 --2- 172.21.15.27:0/712132202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae800ac370 0x7fae800c2a10 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fae7400e9d0 tx=0x7fae740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:01.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.299+0000 7fae6e7fc700 1 -- 172.21.15.27:0/712132202 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7400eec0 con 0x7fae800ac370 2024-01-11T08:28:01.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.299+0000 7fae6e7fc700 1 -- 172.21.15.27:0/712132202 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7400c850 con 0x7fae800ac370 2024-01-11T08:28:01.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.299+0000 7fae6e7fc700 1 -- 172.21.15.27:0/712132202 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7400cc10 con 0x7fae800ac370 2024-01-11T08:28:01.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.299+0000 7fae849ff700 1 -- 172.21.15.27:0/712132202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae700094f0 con 0x7fae800ac370 2024-01-11T08:28:01.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.299+0000 7fae849ff700 1 -- 172.21.15.27:0/712132202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae800c1c50 con 0x7fae800ac370 2024-01-11T08:28:01.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.300+0000 7fae849ff700 1 -- 172.21.15.27:0/712132202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae800c2710 con 0x7fae800ac370 2024-01-11T08:28:01.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.301+0000 7fae6e7fc700 1 -- 172.21.15.27:0/712132202 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 53) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fae74013470 con 0x7fae800ac370 2024-01-11T08:28:01.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.302+0000 7fae6e7fc700 1 --2- 172.21.15.27:0/712132202 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fae60067170 0x7fae60069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.302+0000 7fae6e7fc700 1 -- 172.21.15.27:0/712132202 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6020+0+0 (secure 0 0 0) 0x7fae74084a40 con 0x7fae800ac370 2024-01-11T08:28:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.302+0000 7fae7e59c700 1 -- 172.21.15.27:0/712132202 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fae60067170 msgr2=0x7fae60069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.302+0000 7fae7e59c700 1 --2- 172.21.15.27:0/712132202 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fae60067170 0x7fae60069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:28:01.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.312+0000 7fae6e7fc700 1 -- 172.21.15.27:0/712132202 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fae74051840 con 0x7fae800ac370 2024-01-11T08:28:01.319 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:01.319+0000 7f54a36501c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:28:01.320 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:01.322+0000 7fcdba6941c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:28:01.422 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:01.422+0000 7f54a36501c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:28:01.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.503+0000 7fae7e59c700 1 -- 172.21.15.27:0/712132202 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fae60067170 msgr2=0x7fae60069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:01.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.503+0000 7fae7e59c700 1 --2- 172.21.15.27:0/712132202 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fae60067170 0x7fae60069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:28:01.509 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:01.510+0000 7fcdba6941c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:28:01.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.518+0000 7fae849ff700 1 -- 172.21.15.27:0/712132202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae800ab7a0 con 0x7fae800ac370 2024-01-11T08:28:01.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.519+0000 7fae6e7fc700 1 -- 172.21.15.27:0/712132202 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+15767 (secure 0 0 0) 0x7fae74017090 con 0x7fae800ac370 2024-01-11T08:28:01.520 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:28:01.520 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":104,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:27:55.854427+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:26:36.329359+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-11T08:26:37.504889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.d","mappings":[{"from":1,"to":3}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6836/41381":"2024-01-12T08:26:54.214924+0000","172.21.15.77:6835/41381":"2024-01-12T08:26:54.214924+0000","172.21.15.77:0/1971990124":"2024-01-12T08:26:54.214924+0000","172.21.15.77:0/673641283":"2024-01-12T08:26:54.214924+0000","172.21.15.77:0/3650013349":"2024-01-12T08:26:54.214924+0000","172.21.15.77:0/4016583196":"2024-01-12T08:26:54.214924+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:28:01.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.522+0000 7fae849ff700 1 -- 172.21.15.27:0/712132202 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fae60067170 msgr2=0x7fae60069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:28:01.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.522+0000 7fae849ff700 1 --2- 172.21.15.27:0/712132202 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fae60067170 0x7fae60069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.522+0000 7fae849ff700 1 -- 172.21.15.27:0/712132202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae800ac370 msgr2=0x7fae800c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:01.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.522+0000 7fae849ff700 1 --2- 172.21.15.27:0/712132202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae800ac370 0x7fae800c2a10 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fae7400e9d0 tx=0x7fae740091a0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.522+0000 7fae849ff700 1 -- 172.21.15.27:0/712132202 shutdown_connections 2024-01-11T08:28:01.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.522+0000 7fae849ff700 1 --2- 172.21.15.27:0/712132202 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fae60067170 0x7fae60069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.522+0000 7fae849ff700 1 --2- 172.21.15.27:0/712132202 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae800ad020 0x7fae800c0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.522+0000 7fae849ff700 1 --2- 172.21.15.27:0/712132202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae8006b460 0x7fae80143830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.522+0000 7fae849ff700 1 --2- 172.21.15.27:0/712132202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae800ac370 0x7fae800c2a10 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.522+0000 7fae849ff700 1 -- 172.21.15.27:0/712132202 >> 172.21.15.27:0/712132202 conn(0x7fae800680e0 msgr2=0x7fae8006e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:01.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.522+0000 7fae849ff700 1 -- 172.21.15.27:0/712132202 shutdown_connections 2024-01-11T08:28:01.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.522+0000 7fae849ff700 1 -- 172.21.15.27:0/712132202 wait complete. 2024-01-11T08:28:01.527 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:01.527+0000 7f54a36501c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:28:01.542 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-11T08:28:01.542 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:28:01.605 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:01.607+0000 7fcdba6941c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:28:01.637 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:01.637+0000 7f54a36501c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:28:01.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.746+0000 7fbf68dff700 1 Processor -- start 2024-01-11T08:28:01.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.746+0000 7fbf68dff700 1 -- start start 2024-01-11T08:28:01.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.747+0000 7fbf68dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbf641575e0 0x7fbf64158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:01.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.748+0000 7fbf68dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbf640b19d0 0x7fbf64159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:01.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.748+0000 7fbf68dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf64158b60 con 0x7fbf6406b610 2024-01-11T08:28:01.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.748+0000 7fbf68dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf640cb5e0 con 0x7fbf641575e0 2024-01-11T08:28:01.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.748+0000 7fbf68dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf641599b0 con 0x7fbf640b19d0 2024-01-11T08:28:01.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.748+0000 7fbf6259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbf641575e0 0x7fbf64158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:01.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.748+0000 7fbf6259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbf641575e0 0x7fbf64158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56268/0 (socket says 172.21.15.27:56268) 2024-01-11T08:28:01.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.748+0000 7fbf6259c700 1 -- 172.21.15.27:0/1718955569 learned_addr learned my addr 172.21.15.27:0/1718955569 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:01.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.748+0000 7fbf62d9d700 1 --2- 172.21.15.27:0/1718955569 >> v2:172.21.15.77:3300/0 conn(0x7fbf640b19d0 0x7fbf64159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:01.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.748+0000 7fbf6259c700 1 -- 172.21.15.27:0/1718955569 >> v2:172.21.15.77:3300/0 conn(0x7fbf640b19d0 msgr2=0x7fbf64159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:01.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.748+0000 7fbf6259c700 1 --2- 172.21.15.27:0/1718955569 >> v2:172.21.15.77:3300/0 conn(0x7fbf640b19d0 0x7fbf64159e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.749+0000 7fbf6259c700 1 -- 172.21.15.27:0/1718955569 >> v1:172.21.15.77:6789/0 conn(0x7fbf6406b610 legacy=0x7fbf640b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:01.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.748+0000 7fbf6159a700 1 -- 172.21.15.27:0/1718955569 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478322689 0 0) 0x7fbf58001110 con 0x7fbf6406b610 2024-01-11T08:28:01.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.749+0000 7fbf6259c700 1 -- 172.21.15.27:0/1718955569 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf640c0f90 con 0x7fbf641575e0 2024-01-11T08:28:01.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.749+0000 7fbf6259c700 1 --2- 172.21.15.27:0/1718955569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf641575e0 0x7fbf64158620 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fbf6406d070 tx=0x7fbf54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117e05528b2714f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.749+0000 7fbf6159a700 1 -- 172.21.15.27:0/1718955569 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf54009e70 con 0x7fbf641575e0 2024-01-11T08:28:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.749+0000 7fbf6159a700 1 -- 172.21.15.27:0/1718955569 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5400b860 con 0x7fbf641575e0 2024-01-11T08:28:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.749+0000 7fbf6159a700 1 -- 172.21.15.27:0/1718955569 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5400bc00 con 0x7fbf641575e0 2024-01-11T08:28:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.750+0000 7fbf68dff700 1 -- 172.21.15.27:0/1718955569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf641575e0 msgr2=0x7fbf64158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.750+0000 7fbf68dff700 1 --2- 172.21.15.27:0/1718955569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf641575e0 0x7fbf64158620 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fbf6406d070 tx=0x7fbf54009250 comp rx=0 tx=0).stop 2024-01-11T08:28:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.750+0000 7fbf68dff700 1 -- 172.21.15.27:0/1718955569 shutdown_connections 2024-01-11T08:28:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.750+0000 7fbf68dff700 1 --2- 172.21.15.27:0/1718955569 >> v2:172.21.15.77:3300/0 conn(0x7fbf640b19d0 0x7fbf64159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.750+0000 7fbf68dff700 1 --2- 172.21.15.27:0/1718955569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf641575e0 0x7fbf64158620 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.750+0000 7fbf68dff700 1 -- 172.21.15.27:0/1718955569 >> 172.21.15.27:0/1718955569 conn(0x7fbf640680e0 msgr2=0x7fbf64069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.750+0000 7fbf68dff700 1 -- 172.21.15.27:0/1718955569 shutdown_connections 2024-01-11T08:28:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.750+0000 7fbf68dff700 1 -- 172.21.15.27:0/1718955569 wait complete. 2024-01-11T08:28:01.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.751+0000 7fbf68dff700 1 Processor -- start 2024-01-11T08:28:01.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.752+0000 7fbf68dff700 1 -- start start 2024-01-11T08:28:01.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.753+0000 7fbf68dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf6406b610 0x7fbf64153da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:01.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.753+0000 7fbf68dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf640b19d0 0x7fbf640b01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:01.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.753+0000 7fbf68dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf641575e0 0x7fbf640c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:01.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.753+0000 7fbf68dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf640afd20 con 0x7fbf640b19d0 2024-01-11T08:28:01.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.753+0000 7fbf68dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf640af150 con 0x7fbf641575e0 2024-01-11T08:28:01.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.753+0000 7fbf68dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf640af3f0 con 0x7fbf6406b610 2024-01-11T08:28:01.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.753+0000 7fbf61d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf640b19d0 0x7fbf640b01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:01.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.753+0000 7fbf62d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf641575e0 0x7fbf640c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:01.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.753+0000 7fbf6259c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf6406b610 0x7fbf64153da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:01.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.753+0000 7fbf61d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf640b19d0 0x7fbf640b01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56270/0 (socket says 172.21.15.27:56270) 2024-01-11T08:28:01.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.753+0000 7fbf61d9b700 1 -- 172.21.15.27:0/2141338518 learned_addr learned my addr 172.21.15.27:0/2141338518 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.753+0000 7fbf62d9d700 1 -- 172.21.15.27:0/2141338518 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf6406b610 msgr2=0x7fbf64153da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.753+0000 7fbf62d9d700 1 --2- 172.21.15.27:0/2141338518 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf6406b610 0x7fbf64153da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.753+0000 7fbf62d9d700 1 -- 172.21.15.27:0/2141338518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf640b19d0 msgr2=0x7fbf640b01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.753+0000 7fbf62d9d700 1 --2- 172.21.15.27:0/2141338518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf640b19d0 0x7fbf640b01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.754+0000 7fbf62d9d700 1 -- 172.21.15.27:0/2141338518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf540094f0 con 0x7fbf641575e0 2024-01-11T08:28:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.754+0000 7fbf61d9b700 1 --2- 172.21.15.27:0/2141338518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf640b19d0 0x7fbf640b01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:28:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.754+0000 7fbf62d9d700 1 --2- 172.21.15.27:0/2141338518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf641575e0 0x7fbf640c76b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009e50 tx=0x7fbf5c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.754+0000 7fbf527fc700 1 -- 172.21.15.27:0/2141338518 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c00caa0 con 0x7fbf641575e0 2024-01-11T08:28:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.754+0000 7fbf527fc700 1 -- 172.21.15.27:0/2141338518 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5c00f040 con 0x7fbf641575e0 2024-01-11T08:28:01.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.754+0000 7fbf527fc700 1 -- 172.21.15.27:0/2141338518 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c00ce50 con 0x7fbf641575e0 2024-01-11T08:28:01.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.754+0000 7fbf68dff700 1 -- 172.21.15.27:0/2141338518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf640af6d0 con 0x7fbf641575e0 2024-01-11T08:28:01.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.754+0000 7fbf68dff700 1 -- 172.21.15.27:0/2141338518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf641542e0 con 0x7fbf641575e0 2024-01-11T08:28:01.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.755+0000 7fbf68dff700 1 -- 172.21.15.27:0/2141338518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf640afed0 con 0x7fbf641575e0 2024-01-11T08:28:01.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.756+0000 7fbf527fc700 1 -- 172.21.15.27:0/2141338518 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 53) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fbf5c014690 con 0x7fbf641575e0 2024-01-11T08:28:01.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.757+0000 7fbf527fc700 1 --2- 172.21.15.27:0/2141338518 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbf44067170 0x7fbf44069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:01.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.757+0000 7fbf527fc700 1 -- 172.21.15.27:0/2141338518 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6020+0+0 (secure 0 0 0) 0x7fbf5c085700 con 0x7fbf641575e0 2024-01-11T08:28:01.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.758+0000 7fbf6259c700 1 -- 172.21.15.27:0/2141338518 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbf44067170 msgr2=0x7fbf44069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:01.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.758+0000 7fbf6259c700 1 --2- 172.21.15.27:0/2141338518 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbf44067170 0x7fbf44069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:28:01.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.761+0000 7fbf527fc700 1 -- 172.21.15.27:0/2141338518 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbf640afed0 con 0x7fbf641575e0 2024-01-11T08:28:01.841 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:01.841+0000 7f54a36501c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:28:01.946 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:01.946+0000 7f54a36501c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:28:01.948 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:01.950+0000 7fcdba6941c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:28:01.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.959+0000 7fbf6259c700 1 -- 172.21.15.27:0/2141338518 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbf44067170 msgr2=0x7fbf44069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:01.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:01.959+0000 7fbf6259c700 1 --2- 172.21.15.27:0/2141338518 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbf44067170 0x7fbf44069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:28:02.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.009+0000 7fbf68dff700 1 -- 172.21.15.27:0/2141338518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf640afe90 con 0x7fbf641575e0 2024-01-11T08:28:02.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.010+0000 7fbf527fc700 1 -- 172.21.15.27:0/2141338518 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4593 (secure 0 0 0) 0x7fbf5c083e60 con 0x7fbf641575e0 2024-01-11T08:28:02.010 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:28:02.010 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":50,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/921942372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":921942372},{"type":"v1","addr":"172.21.15.27:6834","nonce":921942372}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7080,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/3844286168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3844286168},{"type":"v1","addr":"172.21.15.77:6834","nonce":3844286168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7083,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3222788452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":3222788452},{"type":"v1","addr":"172.21.15.77:6837","nonce":3222788452}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:26:38.514309+0000","modified":"2024-01-11T08:26:42.031140+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7089},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7089":{"gid":7089,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/751423743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":751423743},{"type":"v1","addr":"172.21.15.27:6835","nonce":751423743}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2024-01-11T08:28:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.013+0000 7fbf68dff700 1 -- 172.21.15.27:0/2141338518 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbf44067170 msgr2=0x7fbf44069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:28:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.013+0000 7fbf68dff700 1 --2- 172.21.15.27:0/2141338518 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbf44067170 0x7fbf44069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.013+0000 7fbf68dff700 1 -- 172.21.15.27:0/2141338518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf641575e0 msgr2=0x7fbf640c76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.013+0000 7fbf68dff700 1 --2- 172.21.15.27:0/2141338518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf641575e0 0x7fbf640c76b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009e50 tx=0x7fbf5c00b040 comp rx=0 tx=0).stop 2024-01-11T08:28:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.013+0000 7fbf68dff700 1 -- 172.21.15.27:0/2141338518 shutdown_connections 2024-01-11T08:28:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.013+0000 7fbf68dff700 1 --2- 172.21.15.27:0/2141338518 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbf44067170 0x7fbf44069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.013+0000 7fbf68dff700 1 --2- 172.21.15.27:0/2141338518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf641575e0 0x7fbf640c76b0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.013+0000 7fbf68dff700 1 --2- 172.21.15.27:0/2141338518 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf6406b610 0x7fbf64153da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.013+0000 7fbf68dff700 1 --2- 172.21.15.27:0/2141338518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf640b19d0 0x7fbf640b01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.013+0000 7fbf68dff700 1 -- 172.21.15.27:0/2141338518 >> 172.21.15.27:0/2141338518 conn(0x7fbf640680e0 msgr2=0x7fbf64155f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:02.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.013+0000 7fbf68dff700 1 -- 172.21.15.27:0/2141338518 shutdown_connections 2024-01-11T08:28:02.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.013+0000 7fbf68dff700 1 -- 172.21.15.27:0/2141338518 wait complete. 2024-01-11T08:28:02.015 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 50 2024-01-11T08:28:02.033 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:28:02.038 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:02.040+0000 7fcdba6941c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:28:02.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.238+0000 7f67727ff700 1 Processor -- start 2024-01-11T08:28:02.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.238+0000 7f67727ff700 1 -- start start 2024-01-11T08:28:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.240+0000 7f67727ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f676c1575e0 0x7f676c158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.240+0000 7f67727ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f676c06b610 0x7f676c0b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.240+0000 7f67727ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f676c0b1800 0x7f676c0ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.240+0000 7f67727ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f676c0cb5e0 con 0x7f676c1575e0 2024-01-11T08:28:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.240+0000 7f67727ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f676c0b1c80 con 0x7f676c0b1800 2024-01-11T08:28:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.240+0000 7f67727ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f676c158b60 con 0x7f676c06b610 2024-01-11T08:28:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.240+0000 7f676bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f676c1575e0 0x7f676c158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.240+0000 7f676bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f676c1575e0 0x7f676c158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56274/0 (socket says 172.21.15.27:56274) 2024-01-11T08:28:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.240+0000 7f676bfff700 1 -- 172.21.15.27:0/2542216736 learned_addr learned my addr 172.21.15.27:0/2542216736 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.240+0000 7f676b7fe700 1 --2- 172.21.15.27:0/2542216736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f676c06b610 0x7f676c0b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.240+0000 7f6770d9c700 1 --2- 172.21.15.27:0/2542216736 >> v2:172.21.15.77:3300/0 conn(0x7f676c0b1800 0x7f676c0ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:02.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.241+0000 7f676bfff700 1 -- 172.21.15.27:0/2542216736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f676c06b610 msgr2=0x7f676c0b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:02.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.241+0000 7f676bfff700 1 --2- 172.21.15.27:0/2542216736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f676c06b610 0x7f676c0b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.241+0000 7f676bfff700 1 -- 172.21.15.27:0/2542216736 >> v2:172.21.15.77:3300/0 conn(0x7f676c0b1800 msgr2=0x7f676c0ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:02.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.241+0000 7f676bfff700 1 --2- 172.21.15.27:0/2542216736 >> v2:172.21.15.77:3300/0 conn(0x7f676c0b1800 0x7f676c0ca1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.241+0000 7f676bfff700 1 -- 172.21.15.27:0/2542216736 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676c1599b0 con 0x7f676c1575e0 2024-01-11T08:28:02.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.241+0000 7f676bfff700 1 --2- 172.21.15.27:0/2542216736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f676c1575e0 0x7f676c158620 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f676c06d0a0 tx=0x7f675c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5adac80fb79f686b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:02.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.241+0000 7f676affd700 1 -- 172.21.15.27:0/2542216736 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c00b640 con 0x7f676c1575e0 2024-01-11T08:28:02.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.241+0000 7f676affd700 1 -- 172.21.15.27:0/2542216736 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675c00b7a0 con 0x7f676c1575e0 2024-01-11T08:28:02.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.241+0000 7f676affd700 1 -- 172.21.15.27:0/2542216736 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c00bb60 con 0x7f676c1575e0 2024-01-11T08:28:02.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.242+0000 7f67727ff700 1 -- 172.21.15.27:0/2542216736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f676c1575e0 msgr2=0x7f676c158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:02.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.242+0000 7f67727ff700 1 --2- 172.21.15.27:0/2542216736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f676c1575e0 0x7f676c158620 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f676c06d0a0 tx=0x7f675c009250 comp rx=0 tx=0).stop 2024-01-11T08:28:02.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.242+0000 7f67727ff700 1 -- 172.21.15.27:0/2542216736 shutdown_connections 2024-01-11T08:28:02.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.242+0000 7f67727ff700 1 --2- 172.21.15.27:0/2542216736 >> v2:172.21.15.77:3300/0 conn(0x7f676c0b1800 0x7f676c0ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.242+0000 7f67727ff700 1 --2- 172.21.15.27:0/2542216736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f676c1575e0 0x7f676c158620 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.242+0000 7f67727ff700 1 --2- 172.21.15.27:0/2542216736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f676c06b610 0x7f676c0b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.242+0000 7f67727ff700 1 -- 172.21.15.27:0/2542216736 >> 172.21.15.27:0/2542216736 conn(0x7f676c0680e0 msgr2=0x7f676c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:02.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.242+0000 7f67727ff700 1 -- 172.21.15.27:0/2542216736 shutdown_connections 2024-01-11T08:28:02.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.242+0000 7f67727ff700 1 -- 172.21.15.27:0/2542216736 wait complete. 2024-01-11T08:28:02.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.243+0000 7f67727ff700 1 Processor -- start 2024-01-11T08:28:02.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.244+0000 7f67727ff700 1 -- start start 2024-01-11T08:28:02.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.245+0000 7f67727ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f676c06b610 0x7f676c16c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:02.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.245+0000 7f67727ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f676c0b1800 0x7f676c0c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:02.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.245+0000 7f67727ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f676c1575e0 0x7f676c169dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:02.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.245+0000 7f67727ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c16cf10 con 0x7f676c06b610 2024-01-11T08:28:02.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.245+0000 7f67727ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c0c6e10 con 0x7f676c1575e0 2024-01-11T08:28:02.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.245+0000 7f67727ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f676c0c70f0 con 0x7f676c0b1800 2024-01-11T08:28:02.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.245+0000 7f676bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f676c06b610 0x7f676c16c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:02.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.245+0000 7f676bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f676c06b610 0x7f676c16c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56282/0 (socket says 172.21.15.27:56282) 2024-01-11T08:28:02.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.245+0000 7f676bfff700 1 -- 172.21.15.27:0/1252470842 learned_addr learned my addr 172.21.15.27:0/1252470842 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:02.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.245+0000 7f676b7fe700 1 --2- 172.21.15.27:0/1252470842 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f676c0b1800 0x7f676c0c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:02.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.245+0000 7f6770d9c700 1 --2- 172.21.15.27:0/1252470842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f676c1575e0 0x7f676c169dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:02.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.245+0000 7f676bfff700 1 -- 172.21.15.27:0/1252470842 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f676c0b1800 msgr2=0x7f676c0c63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:02.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.245+0000 7f676bfff700 1 --2- 172.21.15.27:0/1252470842 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f676c0b1800 0x7f676c0c63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.245+0000 7f676bfff700 1 -- 172.21.15.27:0/1252470842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f676c1575e0 msgr2=0x7f676c169dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:02.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.245+0000 7f676bfff700 1 --2- 172.21.15.27:0/1252470842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f676c1575e0 0x7f676c169dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.245+0000 7f676bfff700 1 -- 172.21.15.27:0/1252470842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675c0094f0 con 0x7f676c06b610 2024-01-11T08:28:02.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.246+0000 7f676bfff700 1 --2- 172.21.15.27:0/1252470842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f676c06b610 0x7f676c16c9d0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f676c066180 tx=0x7f675c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:02.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.246+0000 7f6753fff700 1 -- 172.21.15.27:0/1252470842 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c00b910 con 0x7f676c06b610 2024-01-11T08:28:02.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.246+0000 7f6753fff700 1 -- 172.21.15.27:0/1252470842 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675c00ba70 con 0x7f676c06b610 2024-01-11T08:28:02.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.246+0000 7f6753fff700 1 -- 172.21.15.27:0/1252470842 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c011630 con 0x7f676c06b610 2024-01-11T08:28:02.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.246+0000 7f67727ff700 1 -- 172.21.15.27:0/1252470842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676c0c73d0 con 0x7f676c06b610 2024-01-11T08:28:02.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.246+0000 7f67727ff700 1 -- 172.21.15.27:0/1252470842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676c0c7990 con 0x7f676c06b610 2024-01-11T08:28:02.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.247+0000 7f67727ff700 1 -- 172.21.15.27:0/1252470842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f676c0718d0 con 0x7f676c06b610 2024-01-11T08:28:02.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.248+0000 7f6753fff700 1 -- 172.21.15.27:0/1252470842 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 53) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f675c0119a0 con 0x7f676c06b610 2024-01-11T08:28:02.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.250+0000 7f6753fff700 1 --2- 172.21.15.27:0/1252470842 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6758067170 0x7f6758069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:02.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.250+0000 7f6753fff700 1 -- 172.21.15.27:0/1252470842 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6020+0+0 (secure 0 0 0) 0x7f675c0860a0 con 0x7f676c06b610 2024-01-11T08:28:02.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.250+0000 7f676b7fe700 1 -- 172.21.15.27:0/1252470842 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6758067170 msgr2=0x7f6758069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:02.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.250+0000 7f676b7fe700 1 --2- 172.21.15.27:0/1252470842 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6758067170 0x7f6758069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:28:02.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.254+0000 7f6753fff700 1 -- 172.21.15.27:0/1252470842 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f675c052ea0 con 0x7f676c06b610 2024-01-11T08:28:02.332 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:02.332+0000 7f54a36501c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:28:02.426 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:02.426+0000 7f54a36501c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:28:02.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.451+0000 7f676b7fe700 1 -- 172.21.15.27:0/1252470842 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6758067170 msgr2=0x7f6758069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:02.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.451+0000 7f676b7fe700 1 --2- 172.21.15.27:0/1252470842 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6758067170 0x7f6758069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:28:02.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.520+0000 7f67727ff700 1 -- 172.21.15.27:0/1252470842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f676c070d20 con 0x7f676c06b610 2024-01-11T08:28:02.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.521+0000 7f6753fff700 1 -- 172.21.15.27:0/1252470842 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4593 (secure 0 0 0) 0x7f675c0564c0 con 0x7f676c06b610 2024-01-11T08:28:02.521 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:28:02.521 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":50,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/921942372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":921942372},{"type":"v1","addr":"172.21.15.27:6834","nonce":921942372}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7080,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/3844286168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3844286168},{"type":"v1","addr":"172.21.15.77:6834","nonce":3844286168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7083,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3222788452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":3222788452},{"type":"v1","addr":"172.21.15.77:6837","nonce":3222788452}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:26:38.514309+0000","modified":"2024-01-11T08:26:42.031140+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7089},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7089":{"gid":7089,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/751423743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":751423743},{"type":"v1","addr":"172.21.15.27:6835","nonce":751423743}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2024-01-11T08:28:02.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.523+0000 7f67727ff700 1 -- 172.21.15.27:0/1252470842 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6758067170 msgr2=0x7f6758069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:28:02.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.523+0000 7f67727ff700 1 --2- 172.21.15.27:0/1252470842 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6758067170 0x7f6758069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.524+0000 7f67727ff700 1 -- 172.21.15.27:0/1252470842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f676c06b610 msgr2=0x7f676c16c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:02.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.524+0000 7f67727ff700 1 --2- 172.21.15.27:0/1252470842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f676c06b610 0x7f676c16c9d0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f676c066180 tx=0x7f675c017000 comp rx=0 tx=0).stop 2024-01-11T08:28:02.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.524+0000 7f67727ff700 1 -- 172.21.15.27:0/1252470842 shutdown_connections 2024-01-11T08:28:02.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.524+0000 7f67727ff700 1 --2- 172.21.15.27:0/1252470842 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6758067170 0x7f6758069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.524+0000 7f67727ff700 1 --2- 172.21.15.27:0/1252470842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f676c1575e0 0x7f676c169dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.524+0000 7f67727ff700 1 --2- 172.21.15.27:0/1252470842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f676c06b610 0x7f676c16c9d0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.524+0000 7f67727ff700 1 --2- 172.21.15.27:0/1252470842 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f676c0b1800 0x7f676c0c63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.524+0000 7f67727ff700 1 -- 172.21.15.27:0/1252470842 >> 172.21.15.27:0/1252470842 conn(0x7f676c0680e0 msgr2=0x7f676c06e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:02.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.524+0000 7f67727ff700 1 -- 172.21.15.27:0/1252470842 shutdown_connections 2024-01-11T08:28:02.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.524+0000 7f67727ff700 1 -- 172.21.15.27:0/1252470842 wait complete. 2024-01-11T08:28:02.525 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 50 2024-01-11T08:28:02.542 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:28:02.575 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:02.576+0000 7fcdba6941c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:28:02.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.742+0000 7f9db49ff700 1 Processor -- start 2024-01-11T08:28:02.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.742+0000 7f9db49ff700 1 -- start start 2024-01-11T08:28:02.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.743+0000 7f9db49ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9db0157500 0x7f9db006b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:02.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.743+0000 7f9db49ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9db006bed0 0x7f9db00c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:02.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.743+0000 7f9db49ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db00cd290 con 0x7f9db00c8f70 2024-01-11T08:28:02.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.743+0000 7f9db49ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db00c93f0 con 0x7f9db0157500 2024-01-11T08:28:02.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.743+0000 7f9db49ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db006c430 con 0x7f9db006bed0 2024-01-11T08:28:02.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.744+0000 7f9dae59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9db0157500 0x7f9db006b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:02.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.744+0000 7f9dadd9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9db006bed0 0x7f9db00c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:02.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.744+0000 7f9daed9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9db00c8f70 0x7f9db006ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53580/0 (socket says 172.21.15.27:53580) 2024-01-11T08:28:02.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.744+0000 7f9daed9d700 1 -- 172.21.15.27:0/2385950754 learned_addr learned my addr 172.21.15.27:0/2385950754 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:28:02.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.744+0000 7f9dae59c700 1 --2- 172.21.15.27:0/2385950754 >> v2:172.21.15.27:3300/0 conn(0x7f9db0157500 0x7f9db006b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56296/0 (socket says 172.21.15.27:56296) 2024-01-11T08:28:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.744+0000 7f9dae59c700 1 -- 172.21.15.27:0/2385950754 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9db006bed0 msgr2=0x7f9db00c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.744+0000 7f9dae59c700 1 --2- 172.21.15.27:0/2385950754 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9db006bed0 0x7f9db00c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.744+0000 7f9dae59c700 1 -- 172.21.15.27:0/2385950754 >> v1:172.21.15.27:6789/0 conn(0x7f9db00c8f70 legacy=0x7f9db006ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.744+0000 7f9dae59c700 1 -- 172.21.15.27:0/2385950754 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db0157c00 con 0x7f9db0157500 2024-01-11T08:28:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.744+0000 7f9dadd9b700 1 --2- 172.21.15.27:0/2385950754 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9db006bed0 0x7f9db00c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:28:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.745+0000 7f9dae59c700 1 --2- 172.21.15.27:0/2385950754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9db0157500 0x7f9db006b990 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f9da00098e0 tx=0x7f9da00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=225ba6bf54d296c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.745+0000 7f9dad59a700 1 -- 172.21.15.27:0/2385950754 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da000b640 con 0x7f9db0157500 2024-01-11T08:28:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.745+0000 7f9dad59a700 1 -- 172.21.15.27:0/2385950754 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da000b7a0 con 0x7f9db0157500 2024-01-11T08:28:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.745+0000 7f9dad59a700 1 -- 172.21.15.27:0/2385950754 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da000bb40 con 0x7f9db0157500 2024-01-11T08:28:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.745+0000 7f9db49ff700 1 -- 172.21.15.27:0/2385950754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9db0157500 msgr2=0x7f9db006b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.745+0000 7f9db49ff700 1 --2- 172.21.15.27:0/2385950754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9db0157500 0x7f9db006b990 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f9da00098e0 tx=0x7f9da00091a0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.746+0000 7f9db49ff700 1 -- 172.21.15.27:0/2385950754 shutdown_connections 2024-01-11T08:28:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.746+0000 7f9db49ff700 1 --2- 172.21.15.27:0/2385950754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9db0157500 0x7f9db006b990 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.746+0000 7f9db49ff700 1 --2- 172.21.15.27:0/2385950754 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9db006bed0 0x7f9db00c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.746+0000 7f9db49ff700 1 -- 172.21.15.27:0/2385950754 >> 172.21.15.27:0/2385950754 conn(0x7f9db00ce3f0 msgr2=0x7f9db00ce7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.746+0000 7f9db49ff700 1 -- 172.21.15.27:0/2385950754 shutdown_connections 2024-01-11T08:28:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.746+0000 7f9db49ff700 1 -- 172.21.15.27:0/2385950754 wait complete. 2024-01-11T08:28:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.747+0000 7f9db49ff700 1 Processor -- start 2024-01-11T08:28:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.747+0000 7f9db49ff700 1 -- start start 2024-01-11T08:28:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.748+0000 7f9db49ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9db006bed0 0x7f9db00b1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.748+0000 7f9db49ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9db00c8f70 0x7f9db00c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.748+0000 7f9db49ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9db0157500 0x7f9db00c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.748+0000 7f9db49ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db00c5860 con 0x7f9db006bed0 2024-01-11T08:28:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.748+0000 7f9db49ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db00b16d0 con 0x7f9db00c8f70 2024-01-11T08:28:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.748+0000 7f9db49ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db00b2510 con 0x7f9db0157500 2024-01-11T08:28:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.748+0000 7f9dae59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9db006bed0 0x7f9db00b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.749+0000 7f9dae59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9db006bed0 0x7f9db00b1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56302/0 (socket says 172.21.15.27:56302) 2024-01-11T08:28:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.749+0000 7f9dae59c700 1 -- 172.21.15.27:0/1897457788 learned_addr learned my addr 172.21.15.27:0/1897457788 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.749+0000 7f9daed9d700 1 --2- 172.21.15.27:0/1897457788 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9db0157500 0x7f9db00c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.749+0000 7f9dadd9b700 1 --2- 172.21.15.27:0/1897457788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9db00c8f70 0x7f9db00c5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.749+0000 7f9dae59c700 1 -- 172.21.15.27:0/1897457788 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9db0157500 msgr2=0x7f9db00c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.749+0000 7f9dae59c700 1 --2- 172.21.15.27:0/1897457788 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9db0157500 0x7f9db00c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.749+0000 7f9dae59c700 1 -- 172.21.15.27:0/1897457788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9db00c8f70 msgr2=0x7f9db00c5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.749+0000 7f9dae59c700 1 --2- 172.21.15.27:0/1897457788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9db00c8f70 0x7f9db00c5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.749+0000 7f9dae59c700 1 -- 172.21.15.27:0/1897457788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da4009420 con 0x7f9db006bed0 2024-01-11T08:28:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.749+0000 7f9dae59c700 1 --2- 172.21.15.27:0/1897457788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9db006bed0 0x7f9db00b1190 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f9da000a010 tx=0x7f9da0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.750+0000 7f9d9e7fc700 1 -- 172.21.15.27:0/1897457788 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da001bc80 con 0x7f9db006bed0 2024-01-11T08:28:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.750+0000 7f9d9e7fc700 1 -- 172.21.15.27:0/1897457788 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da001be20 con 0x7f9db006bed0 2024-01-11T08:28:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.750+0000 7f9d9e7fc700 1 -- 172.21.15.27:0/1897457788 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da0011670 con 0x7f9db006bed0 2024-01-11T08:28:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.750+0000 7f9db49ff700 1 -- 172.21.15.27:0/1897457788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da0009420 con 0x7f9db006bed0 2024-01-11T08:28:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.750+0000 7f9db49ff700 1 -- 172.21.15.27:0/1897457788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db00b1ec0 con 0x7f9db006bed0 2024-01-11T08:28:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.751+0000 7f9db49ff700 1 -- 172.21.15.27:0/1897457788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db00718d0 con 0x7f9db006bed0 2024-01-11T08:28:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.752+0000 7f9d9e7fc700 1 -- 172.21.15.27:0/1897457788 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 53) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f9da0011a60 con 0x7f9db006bed0 2024-01-11T08:28:02.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.753+0000 7f9d9e7fc700 1 --2- 172.21.15.27:0/1897457788 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9d90067170 0x7f9d90069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:02.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.753+0000 7f9d9e7fc700 1 -- 172.21.15.27:0/1897457788 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6020+0+0 (secure 0 0 0) 0x7f9da0086270 con 0x7f9db006bed0 2024-01-11T08:28:02.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.753+0000 7f9dadd9b700 1 -- 172.21.15.27:0/1897457788 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9d90067170 msgr2=0x7f9d90069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:02.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.753+0000 7f9dadd9b700 1 --2- 172.21.15.27:0/1897457788 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9d90067170 0x7f9d90069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:28:02.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.757+0000 7f9d9e7fc700 1 -- 172.21.15.27:0/1897457788 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9da0053070 con 0x7f9db006bed0 2024-01-11T08:28:02.762 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:02.763+0000 7fcdba6941c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:28:02.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.954+0000 7f9dadd9b700 1 -- 172.21.15.27:0/1897457788 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9d90067170 msgr2=0x7f9d90069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:02.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.954+0000 7f9dadd9b700 1 --2- 172.21.15.27:0/1897457788 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9d90067170 0x7f9d90069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:28:02.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.965+0000 7f9db49ff700 1 -- 172.21.15.27:0/1897457788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9db006c6f0 con 0x7f9db006bed0 2024-01-11T08:28:02.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.966+0000 7f9d9e7fc700 1 -- 172.21.15.27:0/1897457788 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+15767 (secure 0 0 0) 0x7f9da0056690 con 0x7f9db006bed0 2024-01-11T08:28:02.966 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:28:02.966 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":104,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:27:55.854427+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:26:36.329359+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-11T08:26:37.504889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.d","mappings":[{"from":1,"to":3}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6836/41381":"2024-01-12T08:26:54.214924+0000","172.21.15.77:6835/41381":"2024-01-12T08:26:54.214924+0000","172.21.15.77:0/1971990124":"2024-01-12T08:26:54.214924+0000","172.21.15.77:0/673641283":"2024-01-12T08:26:54.214924+0000","172.21.15.77:0/3650013349":"2024-01-12T08:26:54.214924+0000","172.21.15.77:0/4016583196":"2024-01-12T08:26:54.214924+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:28:02.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.968+0000 7f9db49ff700 1 -- 172.21.15.27:0/1897457788 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9d90067170 msgr2=0x7f9d90069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:28:02.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.968+0000 7f9db49ff700 1 --2- 172.21.15.27:0/1897457788 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9d90067170 0x7f9d90069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.969+0000 7f9db49ff700 1 -- 172.21.15.27:0/1897457788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9db006bed0 msgr2=0x7f9db00b1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:02.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.969+0000 7f9db49ff700 1 --2- 172.21.15.27:0/1897457788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9db006bed0 0x7f9db00b1190 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f9da000a010 tx=0x7f9da0017000 comp rx=0 tx=0).stop 2024-01-11T08:28:02.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.969+0000 7f9db49ff700 1 -- 172.21.15.27:0/1897457788 shutdown_connections 2024-01-11T08:28:02.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.969+0000 7f9db49ff700 1 --2- 172.21.15.27:0/1897457788 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9d90067170 0x7f9d90069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.969+0000 7f9db49ff700 1 --2- 172.21.15.27:0/1897457788 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9db0157500 0x7f9db00c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.969+0000 7f9db49ff700 1 --2- 172.21.15.27:0/1897457788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9db006bed0 0x7f9db00b1190 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.969+0000 7f9db49ff700 1 --2- 172.21.15.27:0/1897457788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9db00c8f70 0x7f9db00c5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:02.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.969+0000 7f9db49ff700 1 -- 172.21.15.27:0/1897457788 >> 172.21.15.27:0/1897457788 conn(0x7f9db00ce3f0 msgr2=0x7f9db0155c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:02.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.969+0000 7f9db49ff700 1 -- 172.21.15.27:0/1897457788 shutdown_connections 2024-01-11T08:28:02.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:02.969+0000 7f9db49ff700 1 -- 172.21.15.27:0/1897457788 wait complete. 2024-01-11T08:28:02.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-11T08:28:03.015 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:03.015+0000 7f54a36501c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:28:03.142 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:03.143+0000 7fcdba6941c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:28:03.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.190+0000 7fc5d47ff700 1 Processor -- start 2024-01-11T08:28:03.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.190+0000 7fc5d47ff700 1 -- start start 2024-01-11T08:28:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.191+0000 7fc5d47ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc5cc0adec0 0x7fc5cc15bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.191+0000 7fc5d47ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc5cc0ae290 0x7fc5cc06b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.191+0000 7fc5d47ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc5cc06bed0 0x7fc5cc157a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.192+0000 7fc5d47ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5cc06c350 con 0x7fc5cc0adec0 2024-01-11T08:28:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.192+0000 7fc5d47ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5cc0b3c50 con 0x7fc5cc0ae290 2024-01-11T08:28:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.192+0000 7fc5d47ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc158680 con 0x7fc5cc06bed0 2024-01-11T08:28:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.192+0000 7fc5d259b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc5cc0adec0 0x7fc5cc15bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.192+0000 7fc5d1d9a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc5cc0ae290 0x7fc5cc06b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.192+0000 7fc5d2d9c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc5cc06bed0 0x7fc5cc157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.192+0000 7fc5d259b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc5cc0adec0 0x7fc5cc15bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56304/0 (socket says 172.21.15.27:56304) 2024-01-11T08:28:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.192+0000 7fc5d259b700 1 -- 172.21.15.27:0/2810289560 learned_addr learned my addr 172.21.15.27:0/2810289560 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.192+0000 7fc5d259b700 1 -- 172.21.15.27:0/2810289560 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc5cc06bed0 msgr2=0x7fc5cc157a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.192+0000 7fc5d259b700 1 --2- 172.21.15.27:0/2810289560 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc5cc06bed0 0x7fc5cc157a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.192+0000 7fc5d259b700 1 -- 172.21.15.27:0/2810289560 >> v2:172.21.15.77:3300/0 conn(0x7fc5cc0ae290 msgr2=0x7fc5cc06b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.192+0000 7fc5d259b700 1 --2- 172.21.15.27:0/2810289560 >> v2:172.21.15.77:3300/0 conn(0x7fc5cc0ae290 0x7fc5cc06b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.192+0000 7fc5d259b700 1 -- 172.21.15.27:0/2810289560 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc158960 con 0x7fc5cc0adec0 2024-01-11T08:28:03.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.193+0000 7fc5d259b700 1 --2- 172.21.15.27:0/2810289560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5cc0adec0 0x7fc5cc15bbf0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc0af770 tx=0x7fc5c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10cb792579cf8838 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:03.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.193+0000 7fc5d1599700 1 -- 172.21.15.27:0/2810289560 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c800b640 con 0x7fc5cc0adec0 2024-01-11T08:28:03.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.193+0000 7fc5d1599700 1 -- 172.21.15.27:0/2810289560 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c800b7a0 con 0x7fc5cc0adec0 2024-01-11T08:28:03.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.193+0000 7fc5d1599700 1 -- 172.21.15.27:0/2810289560 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c800bb40 con 0x7fc5cc0adec0 2024-01-11T08:28:03.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.194+0000 7fc5d47ff700 1 -- 172.21.15.27:0/2810289560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5cc0adec0 msgr2=0x7fc5cc15bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:03.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.194+0000 7fc5d47ff700 1 --2- 172.21.15.27:0/2810289560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5cc0adec0 0x7fc5cc15bbf0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc0af770 tx=0x7fc5c8009250 comp rx=0 tx=0).stop 2024-01-11T08:28:03.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.194+0000 7fc5d47ff700 1 -- 172.21.15.27:0/2810289560 shutdown_connections 2024-01-11T08:28:03.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.194+0000 7fc5d47ff700 1 --2- 172.21.15.27:0/2810289560 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc5cc06bed0 0x7fc5cc157a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:03.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.194+0000 7fc5d47ff700 1 --2- 172.21.15.27:0/2810289560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5cc0adec0 0x7fc5cc15bbf0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:03.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.194+0000 7fc5d47ff700 1 --2- 172.21.15.27:0/2810289560 >> v2:172.21.15.77:3300/0 conn(0x7fc5cc0ae290 0x7fc5cc06b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:03.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.194+0000 7fc5d47ff700 1 -- 172.21.15.27:0/2810289560 >> 172.21.15.27:0/2810289560 conn(0x7fc5cc0ce3c0 msgr2=0x7fc5cc0ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:03.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.194+0000 7fc5d47ff700 1 -- 172.21.15.27:0/2810289560 shutdown_connections 2024-01-11T08:28:03.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.194+0000 7fc5d47ff700 1 -- 172.21.15.27:0/2810289560 wait complete. 2024-01-11T08:28:03.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.196+0000 7fc5d47ff700 1 Processor -- start 2024-01-11T08:28:03.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.196+0000 7fc5d47ff700 1 -- start start 2024-01-11T08:28:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.198+0000 7fc5d47ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5cc06bed0 0x7fc5cc153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.198+0000 7fc5d47ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc5cc0adec0 0x7fc5cc0a5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.198+0000 7fc5d47ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5cc0ae290 0x7fc5cc16abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.198+0000 7fc5d47ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc16ced0 con 0x7fc5cc06bed0 2024-01-11T08:28:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.198+0000 7fc5d47ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc169aa0 con 0x7fc5cc0ae290 2024-01-11T08:28:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.198+0000 7fc5d47ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc153990 con 0x7fc5cc0adec0 2024-01-11T08:28:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.198+0000 7fc5d259b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5cc06bed0 0x7fc5cc153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.198+0000 7fc5d259b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5cc06bed0 0x7fc5cc153450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56310/0 (socket says 172.21.15.27:56310) 2024-01-11T08:28:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.198+0000 7fc5d259b700 1 -- 172.21.15.27:0/1030985249 learned_addr learned my addr 172.21.15.27:0/1030985249 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.198+0000 7fc5d1d9a700 1 --2- 172.21.15.27:0/1030985249 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc5cc0adec0 0x7fc5cc0a5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.199+0000 7fc5d2d9c700 1 --2- 172.21.15.27:0/1030985249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5cc0ae290 0x7fc5cc16abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.199+0000 7fc5d259b700 1 -- 172.21.15.27:0/1030985249 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc5cc0adec0 msgr2=0x7fc5cc0a5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:03.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.199+0000 7fc5d259b700 1 --2- 172.21.15.27:0/1030985249 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc5cc0adec0 0x7fc5cc0a5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:03.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.199+0000 7fc5d259b700 1 -- 172.21.15.27:0/1030985249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5cc0ae290 msgr2=0x7fc5cc16abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:03.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.199+0000 7fc5d259b700 1 --2- 172.21.15.27:0/1030985249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5cc0ae290 0x7fc5cc16abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:03.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.199+0000 7fc5d259b700 1 -- 172.21.15.27:0/1030985249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c80094f0 con 0x7fc5cc06bed0 2024-01-11T08:28:03.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.199+0000 7fc5d259b700 1 --2- 172.21.15.27:0/1030985249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5cc06bed0 0x7fc5cc153450 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80099b0 tx=0x7fc5c8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:03.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.199+0000 7fc5c27fc700 1 -- 172.21.15.27:0/1030985249 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c800bb20 con 0x7fc5cc06bed0 2024-01-11T08:28:03.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.199+0000 7fc5c27fc700 1 -- 172.21.15.27:0/1030985249 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c8000c90 con 0x7fc5cc06bed0 2024-01-11T08:28:03.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.199+0000 7fc5c27fc700 1 -- 172.21.15.27:0/1030985249 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8011650 con 0x7fc5cc06bed0 2024-01-11T08:28:03.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.199+0000 7fc5d47ff700 1 -- 172.21.15.27:0/1030985249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5cc153ee0 con 0x7fc5cc06bed0 2024-01-11T08:28:03.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.200+0000 7fc5d47ff700 1 -- 172.21.15.27:0/1030985249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5cc16d790 con 0x7fc5cc06bed0 2024-01-11T08:28:03.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.201+0000 7fc5d47ff700 1 -- 172.21.15.27:0/1030985249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5cc04c3c0 con 0x7fc5cc06bed0 2024-01-11T08:28:03.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.202+0000 7fc5c27fc700 1 -- 172.21.15.27:0/1030985249 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 53) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fc5c8011a40 con 0x7fc5cc06bed0 2024-01-11T08:28:03.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.202+0000 7fc5c27fc700 1 --2- 172.21.15.27:0/1030985249 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc5b0067170 0x7fc5b0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:03.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.203+0000 7fc5c27fc700 1 -- 172.21.15.27:0/1030985249 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6020+0+0 (secure 0 0 0) 0x7fc5c80863a0 con 0x7fc5cc06bed0 2024-01-11T08:28:03.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.203+0000 7fc5d1d9a700 1 -- 172.21.15.27:0/1030985249 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc5b0067170 msgr2=0x7fc5b0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:03.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.203+0000 7fc5d1d9a700 1 --2- 172.21.15.27:0/1030985249 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc5b0067170 0x7fc5b0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:28:03.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.207+0000 7fc5c27fc700 1 -- 172.21.15.27:0/1030985249 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc5c80531a0 con 0x7fc5cc06bed0 2024-01-11T08:28:03.221 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:03.221+0000 7f54a36501c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:28:03.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.404+0000 7fc5d1d9a700 1 -- 172.21.15.27:0/1030985249 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc5b0067170 msgr2=0x7fc5b0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:03.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.404+0000 7fc5d1d9a700 1 --2- 172.21.15.27:0/1030985249 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc5b0067170 0x7fc5b0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:28:03.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.457+0000 7fc5d47ff700 1 -- 172.21.15.27:0/1030985249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc5cc15b110 con 0x7fc5cc06bed0 2024-01-11T08:28:03.458 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-11T08:28:03.458+0000 7f27065f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-11T08:28:03.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.463+0000 7fc5c27fc700 1 -- 172.21.15.27:0/1030985249 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v51) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc5c80567c0 con 0x7fc5cc06bed0 2024-01-11T08:28:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.465+0000 7fc5d47ff700 1 -- 172.21.15.27:0/1030985249 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc5b0067170 msgr2=0x7fc5b0069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:28:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.465+0000 7fc5d47ff700 1 --2- 172.21.15.27:0/1030985249 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc5b0067170 0x7fc5b0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.465+0000 7fc5d47ff700 1 -- 172.21.15.27:0/1030985249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5cc06bed0 msgr2=0x7fc5cc153450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.465+0000 7fc5d47ff700 1 --2- 172.21.15.27:0/1030985249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5cc06bed0 0x7fc5cc153450 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80099b0 tx=0x7fc5c8017000 comp rx=0 tx=0).stop 2024-01-11T08:28:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.465+0000 7fc5d47ff700 1 -- 172.21.15.27:0/1030985249 shutdown_connections 2024-01-11T08:28:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.465+0000 7fc5d47ff700 1 --2- 172.21.15.27:0/1030985249 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc5b0067170 0x7fc5b0069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.465+0000 7fc5d47ff700 1 --2- 172.21.15.27:0/1030985249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5cc0ae290 0x7fc5cc16abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.465+0000 7fc5d47ff700 1 --2- 172.21.15.27:0/1030985249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5cc06bed0 0x7fc5cc153450 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.465+0000 7fc5d47ff700 1 --2- 172.21.15.27:0/1030985249 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc5cc0adec0 0x7fc5cc0a5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.465+0000 7fc5d47ff700 1 -- 172.21.15.27:0/1030985249 >> 172.21.15.27:0/1030985249 conn(0x7fc5cc0ce3c0 msgr2=0x7fc5cc154930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.466+0000 7fc5d47ff700 1 -- 172.21.15.27:0/1030985249 shutdown_connections 2024-01-11T08:28:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.466+0000 7fc5d47ff700 1 -- 172.21.15.27:0/1030985249 wait complete. 2024-01-11T08:28:03.467 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-11T08:28:03.474 INFO:tasks.ceph.mds.c.smithi027.stderr: -6307> 2024-01-11T08:26:39.982+0000 7feddfbf1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T08:28:03.485 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-11T08:28:03.521 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T08:28:03.567 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:03.569+0000 7fcdba6941c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:28:03.627 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:03.627+0000 7f54a36501c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:28:03.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.725+0000 7f3c26dff700 1 Processor -- start 2024-01-11T08:28:03.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.726+0000 7f3c26dff700 1 -- start start 2024-01-11T08:28:03.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.727+0000 7f3c26dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c20142dc0 0x7f3c200cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:03.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.727+0000 7f3c26dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3c20143190 0x7f3c200cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:03.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.727+0000 7f3c26dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3c200cc020 0x7f3c200cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:03.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.727+0000 7f3c26dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c2015df30 con 0x7f3c20143190 2024-01-11T08:28:03.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.727+0000 7f3c26dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c2015d320 con 0x7f3c200cc020 2024-01-11T08:28:03.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.727+0000 7f3c26dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2015d590 con 0x7f3c20142dc0 2024-01-11T08:28:03.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.727+0000 7f3c1ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3c20143190 0x7f3c200cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:03.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.728+0000 7f3c2539c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3c200cc020 0x7f3c200cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:03.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.728+0000 7f3c24b9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c20142dc0 0x7f3c200cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:03.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.728+0000 7f3c1ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3c20143190 0x7f3c200cb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56324/0 (socket says 172.21.15.27:56324) 2024-01-11T08:28:03.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.728+0000 7f3c1ffff700 1 -- 172.21.15.27:0/848285357 learned_addr learned my addr 172.21.15.27:0/848285357 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:03.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.728+0000 7f3c2539c700 1 -- 172.21.15.27:0/848285357 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c20142dc0 msgr2=0x7f3c200cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:03.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.728+0000 7f3c2539c700 1 --2- 172.21.15.27:0/848285357 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c20142dc0 0x7f3c200cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:03.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.728+0000 7f3c2539c700 1 -- 172.21.15.27:0/848285357 >> v2:172.21.15.27:3300/0 conn(0x7f3c20143190 msgr2=0x7f3c200cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:03.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.728+0000 7f3c2539c700 1 --2- 172.21.15.27:0/848285357 >> v2:172.21.15.27:3300/0 conn(0x7f3c20143190 0x7f3c200cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:03.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.728+0000 7f3c2539c700 1 -- 172.21.15.27:0/848285357 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2015d870 con 0x7f3c200cc020 2024-01-11T08:28:03.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.728+0000 7f3c1ffff700 1 --2- 172.21.15.27:0/848285357 >> v2:172.21.15.27:3300/0 conn(0x7f3c20143190 0x7f3c200cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:28:03.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.728+0000 7f3c2539c700 1 --2- 172.21.15.27:0/848285357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c200cc020 0x7f3c200cc440 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3c200ce390 tx=0x7f3c18009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9f678d0e114ab5b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:03.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.729+0000 7f3c1f7fe700 1 -- 172.21.15.27:0/848285357 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1800e070 con 0x7f3c200cc020 2024-01-11T08:28:03.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.729+0000 7f3c1f7fe700 1 -- 172.21.15.27:0/848285357 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1800bab0 con 0x7f3c200cc020 2024-01-11T08:28:03.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.729+0000 7f3c1f7fe700 1 -- 172.21.15.27:0/848285357 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1800be70 con 0x7f3c200cc020 2024-01-11T08:28:03.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.729+0000 7f3c26dff700 1 -- 172.21.15.27:0/848285357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c200cc020 msgr2=0x7f3c200cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:03.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.729+0000 7f3c26dff700 1 --2- 172.21.15.27:0/848285357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c200cc020 0x7f3c200cc440 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3c200ce390 tx=0x7f3c18009780 comp rx=0 tx=0).stop 2024-01-11T08:28:03.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.729+0000 7f3c26dff700 1 -- 172.21.15.27:0/848285357 shutdown_connections 2024-01-11T08:28:03.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.729+0000 7f3c26dff700 1 --2- 172.21.15.27:0/848285357 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c20142dc0 0x7f3c200cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:03.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.729+0000 7f3c26dff700 1 --2- 172.21.15.27:0/848285357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c200cc020 0x7f3c200cc440 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:03.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.729+0000 7f3c26dff700 1 --2- 172.21.15.27:0/848285357 >> v2:172.21.15.27:3300/0 conn(0x7f3c20143190 0x7f3c200cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:03.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.729+0000 7f3c26dff700 1 -- 172.21.15.27:0/848285357 >> 172.21.15.27:0/848285357 conn(0x7f3c20063ca0 msgr2=0x7f3c2006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:03.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.729+0000 7f3c26dff700 1 -- 172.21.15.27:0/848285357 shutdown_connections 2024-01-11T08:28:03.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.729+0000 7f3c26dff700 1 -- 172.21.15.27:0/848285357 wait complete. 2024-01-11T08:28:03.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.731+0000 7f3c26dff700 1 Processor -- start 2024-01-11T08:28:03.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.731+0000 7f3c26dff700 1 -- start start 2024-01-11T08:28:03.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.732+0000 7f3c26dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c200cc020 0x7f3c200c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:03.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.732+0000 7f3c26dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c20142dc0 0x7f3c200c9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:03.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.732+0000 7f3c26dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c20143190 0x7f3c20141cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:03.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.732+0000 7f3c26dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c200b2b60 con 0x7f3c20143190 2024-01-11T08:28:03.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.732+0000 7f3c26dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c200b2db0 con 0x7f3c200cc020 2024-01-11T08:28:03.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.732+0000 7f3c26dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c200b3090 con 0x7f3c20142dc0 2024-01-11T08:28:03.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.732+0000 7f3c2539c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c20143190 0x7f3c20141cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.732+0000 7f3c1ffff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c20142dc0 0x7f3c200c9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.732+0000 7f3c24b9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c200cc020 0x7f3c200c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.732+0000 7f3c1ffff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c20142dc0 0x7f3c200c9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:54964/0 (socket says 172.21.15.27:54964) 2024-01-11T08:28:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.732+0000 7f3c2539c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c20143190 0x7f3c20141cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56332/0 (socket says 172.21.15.27:56332) 2024-01-11T08:28:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.732+0000 7f3c1ffff700 1 -- 172.21.15.27:0/3674961231 learned_addr learned my addr 172.21.15.27:0/3674961231 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:03.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.733+0000 7f3c2539c700 1 -- 172.21.15.27:0/3674961231 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c20142dc0 msgr2=0x7f3c200c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:03.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.733+0000 7f3c2539c700 1 --2- 172.21.15.27:0/3674961231 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c20142dc0 0x7f3c200c9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:03.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.733+0000 7f3c2539c700 1 -- 172.21.15.27:0/3674961231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c200cc020 msgr2=0x7f3c200c8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:03.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.733+0000 7f3c2539c700 1 --2- 172.21.15.27:0/3674961231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c200cc020 0x7f3c200c8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:03.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.733+0000 7f3c2539c700 1 -- 172.21.15.27:0/3674961231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c0094f0 con 0x7f3c20143190 2024-01-11T08:28:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.733+0000 7f3c24b9b700 1 --2- 172.21.15.27:0/3674961231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c200cc020 0x7f3c200c8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:28:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.733+0000 7f3c1ffff700 1 --2- 172.21.15.27:0/3674961231 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c20142dc0 0x7f3c200c9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:28:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.733+0000 7f3c2539c700 1 --2- 172.21.15.27:0/3674961231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c20143190 0x7f3c20141cb0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f3c18016000 tx=0x7f3c18019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.733+0000 7f3c1cff9700 1 -- 172.21.15.27:0/3674961231 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c18012dd0 con 0x7f3c20143190 2024-01-11T08:28:03.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.734+0000 7f3c1cff9700 1 -- 172.21.15.27:0/3674961231 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c18027470 con 0x7f3c20143190 2024-01-11T08:28:03.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.734+0000 7f3c1cff9700 1 -- 172.21.15.27:0/3674961231 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c18027830 con 0x7f3c20143190 2024-01-11T08:28:03.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.734+0000 7f3c26dff700 1 -- 172.21.15.27:0/3674961231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c18009a20 con 0x7f3c20143190 2024-01-11T08:28:03.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.734+0000 7f3c26dff700 1 -- 172.21.15.27:0/3674961231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c200b3650 con 0x7f3c20143190 2024-01-11T08:28:03.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.735+0000 7f3c26dff700 1 -- 172.21.15.27:0/3674961231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c200b3890 con 0x7f3c20143190 2024-01-11T08:28:03.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.736+0000 7f3c1cff9700 1 -- 172.21.15.27:0/3674961231 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 53) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f3c18024070 con 0x7f3c20143190 2024-01-11T08:28:03.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.737+0000 7f3c1cff9700 1 --2- 172.21.15.27:0/3674961231 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3c04067170 0x7f3c04069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:03.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.737+0000 7f3c1cff9700 1 -- 172.21.15.27:0/3674961231 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6106+0+0 (secure 0 0 0) 0x7f3c18086f00 con 0x7f3c20143190 2024-01-11T08:28:03.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.737+0000 7f3c24b9b700 1 -- 172.21.15.27:0/3674961231 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3c04067170 msgr2=0x7f3c04069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:03.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.737+0000 7f3c24b9b700 1 --2- 172.21.15.27:0/3674961231 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3c04067170 0x7f3c04069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:28:03.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.741+0000 7f3c1cff9700 1 -- 172.21.15.27:0/3674961231 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3c200b3890 con 0x7f3c20143190 2024-01-11T08:28:03.750 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:03.752+0000 7fcdba6941c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:28:03.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.938+0000 7f3c24b9b700 1 -- 172.21.15.27:0/3674961231 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3c04067170 msgr2=0x7f3c04069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:03.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.938+0000 7f3c24b9b700 1 --2- 172.21.15.27:0/3674961231 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3c04067170 0x7f3c04069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:28:03.939 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:03.941+0000 7fcdba6941c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:28:03.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:03.990+0000 7f3c26dff700 1 -- 172.21.15.27:0/3674961231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3c200b3890 con 0x7f3c20143190 2024-01-11T08:28:04.049 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:04.051+0000 7fcdba6941c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:28:04.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.062+0000 7f3c1cff9700 1 -- 172.21.15.27:0/3674961231 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 54) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f3c18003040 con 0x7f3c20143190 2024-01-11T08:28:04.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.062+0000 7f3c1cff9700 1 -- 172.21.15.27:0/3674961231 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3c04067170 msgr2=0x7f3c04069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:28:04.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.062+0000 7f3c1cff9700 1 --2- 172.21.15.27:0/3674961231 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3c04067170 0x7f3c04069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:04.089 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:04.089+0000 7f54a36501c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:28:04.126 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:04.127+0000 7fcd3848a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:04.126 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:04.127+0000 7fcd3848a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:04.126 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:04.127+0000 7fcd3848a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:04.126 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:04.127+0000 7fcd3848a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:04.126 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:04.127+0000 7fcd3848a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:04.127 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:04.128+0000 7fcd35484700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:04.127 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:04.128+0000 7fcd35484700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:04.127 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:04.128+0000 7fcd35484700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:04.127 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:04.128+0000 7fcd35484700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:04.127 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:04.128+0000 7fcd35484700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:04.283 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:04.283+0000 7f54a36501c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:28:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.463+0000 7f3c1cff9700 1 -- 172.21.15.27:0/3674961231 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v52) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3c1805a450 con 0x7f3c20143190 2024-01-11T08:28:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.465+0000 7f3c26dff700 1 -- 172.21.15.27:0/3674961231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c20143190 msgr2=0x7f3c20141cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.465+0000 7f3c26dff700 1 --2- 172.21.15.27:0/3674961231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c20143190 0x7f3c20141cb0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f3c18016000 tx=0x7f3c18019000 comp rx=0 tx=0).stop 2024-01-11T08:28:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.465+0000 7f3c26dff700 1 -- 172.21.15.27:0/3674961231 shutdown_connections 2024-01-11T08:28:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.465+0000 7f3c26dff700 1 --2- 172.21.15.27:0/3674961231 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3c04067170 0x7f3c04069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.465+0000 7f3c26dff700 1 --2- 172.21.15.27:0/3674961231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c20143190 0x7f3c20141cb0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.465+0000 7f3c26dff700 1 --2- 172.21.15.27:0/3674961231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c200cc020 0x7f3c200c8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.465+0000 7f3c26dff700 1 --2- 172.21.15.27:0/3674961231 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c20142dc0 0x7f3c200c9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.465+0000 7f3c26dff700 1 -- 172.21.15.27:0/3674961231 >> 172.21.15.27:0/3674961231 conn(0x7f3c20063ca0 msgr2=0x7f3c200c4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.466+0000 7f3c26dff700 1 -- 172.21.15.27:0/3674961231 shutdown_connections 2024-01-11T08:28:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.466+0000 7f3c26dff700 1 -- 172.21.15.27:0/3674961231 wait complete. 2024-01-11T08:28:04.483 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:04.483+0000 7f54a36501c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:28:04.485 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-11T08:28:04.600 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:04.600+0000 7f54a36501c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:28:04.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.691+0000 7f12f7dff700 1 Processor -- start 2024-01-11T08:28:04.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.691+0000 7f12f7dff700 1 -- start start 2024-01-11T08:28:04.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.692+0000 7f12f7dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f12f015a7a0 0x7f12f015c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:04.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.692+0000 7f12f7dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12f00685e0 0x7f12f0068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.692+0000 7f12f7dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f0068f80 con 0x7f12f0159e60 2024-01-11T08:28:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.692+0000 7f12f7dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f015c4b0 con 0x7f12f015a7a0 2024-01-11T08:28:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.692+0000 7f12f7dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f015b5d0 con 0x7f12f00685e0 2024-01-11T08:28:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.692+0000 7f12f5b9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f12f015a7a0 0x7f12f015c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.692+0000 7f12f5b9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f12f015a7a0 0x7f12f015c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56142/0 (socket says 172.21.15.27:56142) 2024-01-11T08:28:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.692+0000 7f12f5b9b700 1 -- 172.21.15.27:0/2010710605 learned_addr learned my addr 172.21.15.27:0/2010710605 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.692+0000 7f12f539a700 1 --1- 172.21.15.27:0/2010710605 >> v1:172.21.15.27:6789/0 conn(0x7f12f0159e60 0x7f12f015aec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34000/0 (socket says 172.21.15.27:34000) 2024-01-11T08:28:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.693+0000 7f12f639c700 1 --2- 172.21.15.27:0/2010710605 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12f00685e0 0x7f12f0068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.693+0000 7f12f5b9b700 1 -- 172.21.15.27:0/2010710605 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12f00685e0 msgr2=0x7f12f0068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.693+0000 7f12f5b9b700 1 --2- 172.21.15.27:0/2010710605 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12f00685e0 0x7f12f0068a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.693+0000 7f12f5b9b700 1 -- 172.21.15.27:0/2010710605 >> v1:172.21.15.27:6789/0 conn(0x7f12f0159e60 legacy=0x7f12f015aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.693+0000 7f12f5b9b700 1 -- 172.21.15.27:0/2010710605 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f015b8a0 con 0x7f12f015a7a0 2024-01-11T08:28:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.693+0000 7f12f4b99700 1 -- 172.21.15.27:0/2010710605 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115710329 0 0) 0x7f12e0001110 con 0x7f12f0159e60 2024-01-11T08:28:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.693+0000 7f12f5b9b700 1 --2- 172.21.15.27:0/2010710605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f015a7a0 0x7f12f015c940 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f12ec0099b0 tx=0x7f12ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e04c51c3e40cc3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:04.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.693+0000 7f12f4b99700 1 -- 172.21.15.27:0/2010710605 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec009ea0 con 0x7f12f015a7a0 2024-01-11T08:28:04.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.693+0000 7f12f4b99700 1 -- 172.21.15.27:0/2010710605 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ec00b860 con 0x7f12f015a7a0 2024-01-11T08:28:04.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.693+0000 7f12f4b99700 1 -- 172.21.15.27:0/2010710605 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec00bc00 con 0x7f12f015a7a0 2024-01-11T08:28:04.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.694+0000 7f12f7dff700 1 -- 172.21.15.27:0/2010710605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f015a7a0 msgr2=0x7f12f015c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:04.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.694+0000 7f12f7dff700 1 --2- 172.21.15.27:0/2010710605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f015a7a0 0x7f12f015c940 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f12ec0099b0 tx=0x7f12ec009250 comp rx=0 tx=0).stop 2024-01-11T08:28:04.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.694+0000 7f12f7dff700 1 -- 172.21.15.27:0/2010710605 shutdown_connections 2024-01-11T08:28:04.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.694+0000 7f12f7dff700 1 --2- 172.21.15.27:0/2010710605 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12f00685e0 0x7f12f0068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:04.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.694+0000 7f12f7dff700 1 --2- 172.21.15.27:0/2010710605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f015a7a0 0x7f12f015c940 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:04.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.694+0000 7f12f7dff700 1 -- 172.21.15.27:0/2010710605 >> 172.21.15.27:0/2010710605 conn(0x7f12f006ed40 msgr2=0x7f12f006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:04.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.694+0000 7f12f7dff700 1 -- 172.21.15.27:0/2010710605 shutdown_connections 2024-01-11T08:28:04.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.694+0000 7f12f7dff700 1 -- 172.21.15.27:0/2010710605 wait complete. 2024-01-11T08:28:04.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.696+0000 7f12f7dff700 1 Processor -- start 2024-01-11T08:28:04.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.696+0000 7f12f7dff700 1 -- start start 2024-01-11T08:28:04.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.697+0000 7f12f7dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12f00685e0 0x7f12f0140b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.697+0000 7f12f7dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12f015a7a0 0x7f12f0151820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.697+0000 7f12f7dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f014e4b0 0x7f12f00b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.697+0000 7f12f7dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f0141e30 con 0x7f12f014e4b0 2024-01-11T08:28:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.697+0000 7f12f7dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f0151d60 con 0x7f12f015a7a0 2024-01-11T08:28:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.697+0000 7f12f7dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f0141220 con 0x7f12f00685e0 2024-01-11T08:28:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.697+0000 7f12f639c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f014e4b0 0x7f12f00b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.697+0000 7f12f639c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f014e4b0 0x7f12f00b20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56154/0 (socket says 172.21.15.27:56154) 2024-01-11T08:28:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.697+0000 7f12f639c700 1 -- 172.21.15.27:0/2192118130 learned_addr learned my addr 172.21.15.27:0/2192118130 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.697+0000 7f12f539a700 1 --2- 172.21.15.27:0/2192118130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12f015a7a0 0x7f12f0151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.697+0000 7f12f5b9b700 1 --2- 172.21.15.27:0/2192118130 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12f00685e0 0x7f12f0140b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.697+0000 7f12f639c700 1 -- 172.21.15.27:0/2192118130 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12f00685e0 msgr2=0x7f12f0140b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.698+0000 7f12f639c700 1 --2- 172.21.15.27:0/2192118130 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12f00685e0 0x7f12f0140b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.698+0000 7f12f639c700 1 -- 172.21.15.27:0/2192118130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12f015a7a0 msgr2=0x7f12f0151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:04.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.698+0000 7f12f639c700 1 --2- 172.21.15.27:0/2192118130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12f015a7a0 0x7f12f0151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:04.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.698+0000 7f12f639c700 1 -- 172.21.15.27:0/2192118130 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ec0094f0 con 0x7f12f014e4b0 2024-01-11T08:28:04.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.698+0000 7f12f5b9b700 1 --2- 172.21.15.27:0/2192118130 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12f00685e0 0x7f12f0140b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:28:04.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.698+0000 7f12f639c700 1 --2- 172.21.15.27:0/2192118130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f014e4b0 0x7f12f00b20c0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f12e8009e20 tx=0x7f12e800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:04.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.698+0000 7f12e5ffb700 1 -- 172.21.15.27:0/2192118130 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e800cc20 con 0x7f12f014e4b0 2024-01-11T08:28:04.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.698+0000 7f12e5ffb700 1 -- 172.21.15.27:0/2192118130 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e800f040 con 0x7f12f014e4b0 2024-01-11T08:28:04.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.698+0000 7f12e5ffb700 1 -- 172.21.15.27:0/2192118130 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e8014480 con 0x7f12f014e4b0 2024-01-11T08:28:04.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.698+0000 7f12f7dff700 1 -- 172.21.15.27:0/2192118130 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f0141500 con 0x7f12f014e4b0 2024-01-11T08:28:04.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.698+0000 7f12f7dff700 1 -- 172.21.15.27:0/2192118130 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f0141ac0 con 0x7f12f014e4b0 2024-01-11T08:28:04.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.699+0000 7f12f7dff700 1 -- 172.21.15.27:0/2192118130 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f0141fe0 con 0x7f12f014e4b0 2024-01-11T08:28:04.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.700+0000 7f12e5ffb700 1 -- 172.21.15.27:0/2192118130 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 54) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f12e8014830 con 0x7f12f014e4b0 2024-01-11T08:28:04.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.701+0000 7f12e5ffb700 1 -- 172.21.15.27:0/2192118130 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6450+0+0 (secure 0 0 0) 0x7f12e80850d0 con 0x7f12f014e4b0 2024-01-11T08:28:04.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.705+0000 7f12e5ffb700 1 -- 172.21.15.27:0/2192118130 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f12f0141fe0 con 0x7f12f014e4b0 2024-01-11T08:28:04.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:04.910+0000 7f12f7dff700 1 -- 172.21.15.27:0/2192118130 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f12f004c3c0 con 0x7f12f014e4b0 2024-01-11T08:28:05.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:05.063+0000 7f12e5ffb700 1 -- 172.21.15.27:0/2192118130 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 55) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f12e8051dd0 con 0x7f12f014e4b0 2024-01-11T08:28:05.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:05.064+0000 7f12e5ffb700 1 --2- 172.21.15.27:0/2192118130 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f12d406c7d0 0x7f12d406ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:05.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:05.064+0000 7f12f5b9b700 1 --2- 172.21.15.27:0/2192118130 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f12d406c7d0 0x7f12d406ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:05.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:05.067+0000 7f12f5b9b700 1 --2- 172.21.15.27:0/2192118130 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f12d406c7d0 0x7f12d406ebb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f12ec0099b0 tx=0x7f12ec017000 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:05.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:05.891+0000 7f12e5ffb700 1 -- 172.21.15.27:0/2192118130 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v107) v1 ==== 181+0+0 (secure 0 0 0) 0x7f12e8070dc0 con 0x7f12f014e4b0 2024-01-11T08:28:05.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:05.896+0000 7f12f7dff700 1 -- 172.21.15.27:0/2192118130 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f12d406c7d0 msgr2=0x7f12d406ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:05.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:05.896+0000 7f12f7dff700 1 --2- 172.21.15.27:0/2192118130 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f12d406c7d0 0x7f12d406ebb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f12ec0099b0 tx=0x7f12ec017000 comp rx=0 tx=0).stop 2024-01-11T08:28:05.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:05.896+0000 7f12f7dff700 1 -- 172.21.15.27:0/2192118130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f014e4b0 msgr2=0x7f12f00b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:05.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:05.896+0000 7f12f7dff700 1 --2- 172.21.15.27:0/2192118130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f014e4b0 0x7f12f00b20c0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f12e8009e20 tx=0x7f12e800b040 comp rx=0 tx=0).stop 2024-01-11T08:28:05.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:05.897+0000 7f12f7dff700 1 -- 172.21.15.27:0/2192118130 shutdown_connections 2024-01-11T08:28:05.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:05.897+0000 7f12f7dff700 1 --2- 172.21.15.27:0/2192118130 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f12d406c7d0 0x7f12d406ebb0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:05.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:05.897+0000 7f12f7dff700 1 --2- 172.21.15.27:0/2192118130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f014e4b0 0x7f12f00b20c0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:05.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:05.897+0000 7f12f7dff700 1 --2- 172.21.15.27:0/2192118130 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12f00685e0 0x7f12f0140b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:05.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:05.897+0000 7f12f7dff700 1 --2- 172.21.15.27:0/2192118130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12f015a7a0 0x7f12f0151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:05.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:05.897+0000 7f12f7dff700 1 -- 172.21.15.27:0/2192118130 >> 172.21.15.27:0/2192118130 conn(0x7f12f006ed40 msgr2=0x7f12f00cf970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:05.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:05.897+0000 7f12f7dff700 1 -- 172.21.15.27:0/2192118130 shutdown_connections 2024-01-11T08:28:05.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:05.897+0000 7f12f7dff700 1 -- 172.21.15.27:0/2192118130 wait complete. 2024-01-11T08:28:05.899 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-01-11T08:28:05.918 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-11T08:28:06.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.131+0000 7fe2239ff700 1 Processor -- start 2024-01-11T08:28:06.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.131+0000 7fe2239ff700 1 -- start start 2024-01-11T08:28:06.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.133+0000 7fe2239ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe21c0685e0 0x7fe21c068a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:06.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.133+0000 7fe2239ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21c15c4b0 con 0x7fe21c159e60 2024-01-11T08:28:06.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.133+0000 7fe2239ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21c15d050 con 0x7fe21c15a7a0 2024-01-11T08:28:06.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.133+0000 7fe2239ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe21c068f40 con 0x7fe21c0685e0 2024-01-11T08:28:06.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.133+0000 7fe221f9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe21c0685e0 0x7fe21c068a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:06.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.133+0000 7fe221f9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe21c0685e0 0x7fe21c068a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56162/0 (socket says 172.21.15.27:56162) 2024-01-11T08:28:06.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.133+0000 7fe221f9c700 1 -- 172.21.15.27:0/494374306 learned_addr learned my addr 172.21.15.27:0/494374306 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:06.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.133+0000 7fe220f9a700 1 --1- 172.21.15.27:0/494374306 >> v1:172.21.15.27:6789/0 conn(0x7fe21c159e60 0x7fe21c15aec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34008/0 (socket says 172.21.15.27:34008) 2024-01-11T08:28:06.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.133+0000 7fe22179b700 1 --1- 172.21.15.27:0/494374306 >> v1:172.21.15.77:6789/0 conn(0x7fe21c15a7a0 0x7fe21c15c940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:38698/0 (socket says 172.21.15.27:38698) 2024-01-11T08:28:06.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.133+0000 7fe221f9c700 1 -- 172.21.15.27:0/494374306 >> v1:172.21.15.77:6789/0 conn(0x7fe21c15a7a0 legacy=0x7fe21c15c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:06.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.133+0000 7fe221f9c700 1 -- 172.21.15.27:0/494374306 >> v1:172.21.15.27:6789/0 conn(0x7fe21c159e60 legacy=0x7fe21c15aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:06.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.133+0000 7fe221f9c700 1 -- 172.21.15.27:0/494374306 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c15b5d0 con 0x7fe21c0685e0 2024-01-11T08:28:06.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.133+0000 7fe221f9c700 1 --2- 172.21.15.27:0/494374306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c0685e0 0x7fe21c068a00 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fe214009f20 tx=0x7fe214009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd1a13982193d4cf server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:06.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.134+0000 7fe213fff700 1 -- 172.21.15.27:0/494374306 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21400e070 con 0x7fe21c0685e0 2024-01-11T08:28:06.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.134+0000 7fe213fff700 1 -- 172.21.15.27:0/494374306 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21400ba80 con 0x7fe21c0685e0 2024-01-11T08:28:06.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.134+0000 7fe213fff700 1 -- 172.21.15.27:0/494374306 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2140092e0 con 0x7fe21c0685e0 2024-01-11T08:28:06.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.134+0000 7fe2239ff700 1 -- 172.21.15.27:0/494374306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c0685e0 msgr2=0x7fe21c068a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:06.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.134+0000 7fe2239ff700 1 --2- 172.21.15.27:0/494374306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c0685e0 0x7fe21c068a00 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fe214009f20 tx=0x7fe214009840 comp rx=0 tx=0).stop 2024-01-11T08:28:06.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.134+0000 7fe2239ff700 1 -- 172.21.15.27:0/494374306 shutdown_connections 2024-01-11T08:28:06.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.134+0000 7fe2239ff700 1 --2- 172.21.15.27:0/494374306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c0685e0 0x7fe21c068a00 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:06.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.134+0000 7fe2239ff700 1 -- 172.21.15.27:0/494374306 >> 172.21.15.27:0/494374306 conn(0x7fe21c06ed40 msgr2=0x7fe21c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:06.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.134+0000 7fe2239ff700 1 -- 172.21.15.27:0/494374306 shutdown_connections 2024-01-11T08:28:06.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.135+0000 7fe2239ff700 1 -- 172.21.15.27:0/494374306 wait complete. 2024-01-11T08:28:06.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.136+0000 7fe2239ff700 1 Processor -- start 2024-01-11T08:28:06.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.136+0000 7fe2239ff700 1 -- start start 2024-01-11T08:28:06.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.137+0000 7fe2239ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c0685e0 0x7fe21c169cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:06.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.137+0000 7fe2239ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe21c15a7a0 0x7fe21c0b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:06.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.137+0000 7fe2239ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe21c0b2680 0x7fe21c14df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:06.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.137+0000 7fe2239ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c0cbfa0 con 0x7fe21c0685e0 2024-01-11T08:28:06.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.137+0000 7fe2239ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c0cb9e0 con 0x7fe21c15a7a0 2024-01-11T08:28:06.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.137+0000 7fe2239ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c0cbcc0 con 0x7fe21c0b2680 2024-01-11T08:28:06.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.137+0000 7fe22179b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c0685e0 0x7fe21c169cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:06.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.137+0000 7fe22179b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c0685e0 0x7fe21c169cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56168/0 (socket says 172.21.15.27:56168) 2024-01-11T08:28:06.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.137+0000 7fe22179b700 1 -- 172.21.15.27:0/1080742588 learned_addr learned my addr 172.21.15.27:0/1080742588 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:06.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.137+0000 7fe220f9a700 1 --2- 172.21.15.27:0/1080742588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe21c15a7a0 0x7fe21c0b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:06.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.138+0000 7fe221f9c700 1 --2- 172.21.15.27:0/1080742588 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe21c0b2680 0x7fe21c14df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:06.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.138+0000 7fe22179b700 1 -- 172.21.15.27:0/1080742588 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe21c0b2680 msgr2=0x7fe21c14df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:06.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.138+0000 7fe22179b700 1 --2- 172.21.15.27:0/1080742588 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe21c0b2680 0x7fe21c14df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:06.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.138+0000 7fe22179b700 1 -- 172.21.15.27:0/1080742588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe21c15a7a0 msgr2=0x7fe21c0b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:06.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.138+0000 7fe22179b700 1 --2- 172.21.15.27:0/1080742588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe21c15a7a0 0x7fe21c0b2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:06.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.138+0000 7fe22179b700 1 -- 172.21.15.27:0/1080742588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe214009ae0 con 0x7fe21c0685e0 2024-01-11T08:28:06.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.138+0000 7fe22179b700 1 --2- 172.21.15.27:0/1080742588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c0685e0 0x7fe21c169cc0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fe21800c7c0 tx=0x7fe218007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:06.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.138+0000 7fe2117fa700 1 -- 172.21.15.27:0/1080742588 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe218005a70 con 0x7fe21c0685e0 2024-01-11T08:28:06.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.138+0000 7fe2117fa700 1 -- 172.21.15.27:0/1080742588 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe218005bd0 con 0x7fe21c0685e0 2024-01-11T08:28:06.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.138+0000 7fe2239ff700 1 -- 172.21.15.27:0/1080742588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21c0c8cd0 con 0x7fe21c0685e0 2024-01-11T08:28:06.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.138+0000 7fe2117fa700 1 -- 172.21.15.27:0/1080742588 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2180070d0 con 0x7fe21c0685e0 2024-01-11T08:28:06.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.139+0000 7fe2239ff700 1 -- 172.21.15.27:0/1080742588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21c14e4d0 con 0x7fe21c0685e0 2024-01-11T08:28:06.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.140+0000 7fe2239ff700 1 -- 172.21.15.27:0/1080742588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21c0c8b30 con 0x7fe21c0685e0 2024-01-11T08:28:06.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.141+0000 7fe2117fa700 1 -- 172.21.15.27:0/1080742588 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 55) v1 ==== 84580+0+0 (secure 0 0 0) 0x7fe218007480 con 0x7fe21c0685e0 2024-01-11T08:28:06.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.142+0000 7fe2117fa700 1 --2- 172.21.15.27:0/1080742588 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe200066d80 0x7fe200069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:06.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.142+0000 7fe2117fa700 1 -- 172.21.15.27:0/1080742588 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6012+0+0 (secure 0 0 0) 0x7fe2180837f0 con 0x7fe21c0685e0 2024-01-11T08:28:06.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.142+0000 7fe220f9a700 1 --2- 172.21.15.27:0/1080742588 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe200066d80 0x7fe200069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:06.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.143+0000 7fe220f9a700 1 --2- 172.21.15.27:0/1080742588 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe200066d80 0x7fe200069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe21c0b4bc0 tx=0x7fe20c007eb0 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:06.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.146+0000 7fe2117fa700 1 -- 172.21.15.27:0/1080742588 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe21c0c8b30 con 0x7fe21c0685e0 2024-01-11T08:28:06.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.355+0000 7fe2239ff700 1 -- 172.21.15.27:0/1080742588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe21c0c8b30 con 0x7fe21c0685e0 2024-01-11T08:28:06.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:06.892+0000 7fe2117fa700 1 -- 172.21.15.27:0/1080742588 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 56) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fe218050ee0 con 0x7fe21c0685e0 2024-01-11T08:28:07.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.076+0000 7fe2117fa700 1 -- 172.21.15.27:0/1080742588 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v108) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe2180545a0 con 0x7fe21c0685e0 2024-01-11T08:28:07.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.080+0000 7fe2239ff700 1 -- 172.21.15.27:0/1080742588 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe200066d80 msgr2=0x7fe200069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:07.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.080+0000 7fe2239ff700 1 --2- 172.21.15.27:0/1080742588 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe200066d80 0x7fe200069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe21c0b4bc0 tx=0x7fe20c007eb0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.080+0000 7fe2239ff700 1 -- 172.21.15.27:0/1080742588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c0685e0 msgr2=0x7fe21c169cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:07.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.080+0000 7fe2239ff700 1 --2- 172.21.15.27:0/1080742588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c0685e0 0x7fe21c169cc0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fe21800c7c0 tx=0x7fe218007f70 comp rx=0 tx=0).stop 2024-01-11T08:28:07.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.081+0000 7fe2239ff700 1 -- 172.21.15.27:0/1080742588 shutdown_connections 2024-01-11T08:28:07.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.081+0000 7fe2239ff700 1 --2- 172.21.15.27:0/1080742588 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe200066d80 0x7fe200069230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.081+0000 7fe2239ff700 1 --2- 172.21.15.27:0/1080742588 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe21c0b2680 0x7fe21c14df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.081+0000 7fe2239ff700 1 --2- 172.21.15.27:0/1080742588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c0685e0 0x7fe21c169cc0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.081+0000 7fe2239ff700 1 --2- 172.21.15.27:0/1080742588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe21c15a7a0 0x7fe21c0b2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.081+0000 7fe2239ff700 1 -- 172.21.15.27:0/1080742588 >> 172.21.15.27:0/1080742588 conn(0x7fe21c06ed40 msgr2=0x7fe21c14f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:07.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.081+0000 7fe2239ff700 1 -- 172.21.15.27:0/1080742588 shutdown_connections 2024-01-11T08:28:07.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.081+0000 7fe2239ff700 1 -- 172.21.15.27:0/1080742588 wait complete. 2024-01-11T08:28:07.082 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-01-11T08:28:07.102 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-11T08:28:07.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.304+0000 7f1098bff700 1 Processor -- start 2024-01-11T08:28:07.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.305+0000 7f1098bff700 1 -- start start 2024-01-11T08:28:07.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.306+0000 7f1098bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f10940ccd70 0x7f10940cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:07.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.306+0000 7f1098bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10940cf850 con 0x7f109415a1b0 2024-01-11T08:28:07.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.306+0000 7f1098bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10940cec40 con 0x7f109415ae70 2024-01-11T08:28:07.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.306+0000 7f1098bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f109415ac90 con 0x7f10940ccd70 2024-01-11T08:28:07.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.306+0000 7f109259c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f109415a1b0 0x7f109415a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34018/0 (socket says 172.21.15.27:34018) 2024-01-11T08:28:07.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.306+0000 7f109259c700 1 -- 172.21.15.27:0/2910428058 learned_addr learned my addr 172.21.15.27:0/2910428058 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:28:07.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.307+0000 7f1091d9b700 1 --2- 172.21.15.27:0/2910428058 >> v2:172.21.15.77:3300/0 conn(0x7f10940ccd70 0x7f10940cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:07.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.307+0000 7f109159a700 1 -- 172.21.15.27:0/2910428058 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551313893 0 0) 0x7f10940cf850 con 0x7f109415a1b0 2024-01-11T08:28:07.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.307+0000 7f109159a700 1 -- 172.21.15.27:0/2910428058 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f107c003570 con 0x7f109415a1b0 2024-01-11T08:28:07.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.307+0000 7f1091d9b700 1 -- 172.21.15.27:0/2910428058 >> v1:172.21.15.77:6789/0 conn(0x7f109415ae70 legacy=0x7f109415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:07.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.307+0000 7f1091d9b700 1 -- 172.21.15.27:0/2910428058 >> v1:172.21.15.27:6789/0 conn(0x7f109415a1b0 legacy=0x7f109415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:07.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.307+0000 7f1091d9b700 1 -- 172.21.15.27:0/2910428058 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10940cef20 con 0x7f10940ccd70 2024-01-11T08:28:07.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.307+0000 7f1091d9b700 1 --2- 172.21.15.27:0/2910428058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10940ccd70 0x7f10940cd210 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f10880098e0 tx=0x7f10880091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52b6c0e36d8ad00 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:07.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.308+0000 7f109159a700 1 -- 172.21.15.27:0/2910428058 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108800b640 con 0x7f10940ccd70 2024-01-11T08:28:07.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.308+0000 7f109159a700 1 -- 172.21.15.27:0/2910428058 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f108800b7a0 con 0x7f10940ccd70 2024-01-11T08:28:07.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.308+0000 7f109159a700 1 -- 172.21.15.27:0/2910428058 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108800bb40 con 0x7f10940ccd70 2024-01-11T08:28:07.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.308+0000 7f1098bff700 1 -- 172.21.15.27:0/2910428058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10940ccd70 msgr2=0x7f10940cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:07.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.308+0000 7f1098bff700 1 --2- 172.21.15.27:0/2910428058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10940ccd70 0x7f10940cd210 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f10880098e0 tx=0x7f10880091a0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.308+0000 7f1098bff700 1 -- 172.21.15.27:0/2910428058 shutdown_connections 2024-01-11T08:28:07.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.308+0000 7f1098bff700 1 --2- 172.21.15.27:0/2910428058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10940ccd70 0x7f10940cd210 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.308+0000 7f1098bff700 1 -- 172.21.15.27:0/2910428058 >> 172.21.15.27:0/2910428058 conn(0x7f109406ed40 msgr2=0x7f109406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:07.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.308+0000 7f1098bff700 1 -- 172.21.15.27:0/2910428058 shutdown_connections 2024-01-11T08:28:07.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.308+0000 7f1098bff700 1 -- 172.21.15.27:0/2910428058 wait complete. 2024-01-11T08:28:07.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.310+0000 7f1098bff700 1 Processor -- start 2024-01-11T08:28:07.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.310+0000 7f1098bff700 1 -- start start 2024-01-11T08:28:07.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.311+0000 7f1098bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10940ccd70 0x7f10941481c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:07.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.311+0000 7f1098bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f109415a1b0 0x7f10941488b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:07.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.311+0000 7f1098bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f109415ae70 0x7f10940c5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:07.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.311+0000 7f1098bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10940c8cc0 con 0x7f109415ae70 2024-01-11T08:28:07.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.311+0000 7f1098bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10940b6110 con 0x7f10940ccd70 2024-01-11T08:28:07.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.311+0000 7f1098bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10940c8ed0 con 0x7f109415a1b0 2024-01-11T08:28:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.311+0000 7f1092d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f109415ae70 0x7f10940c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.311+0000 7f1092d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f109415ae70 0x7f10940c5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56180/0 (socket says 172.21.15.27:56180) 2024-01-11T08:28:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.311+0000 7f1092d9d700 1 -- 172.21.15.27:0/4154310619 learned_addr learned my addr 172.21.15.27:0/4154310619 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.311+0000 7f109259c700 1 --2- 172.21.15.27:0/4154310619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10940ccd70 0x7f10941481c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.311+0000 7f1091d9b700 1 --2- 172.21.15.27:0/4154310619 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f109415a1b0 0x7f10941488b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.312+0000 7f1092d9d700 1 -- 172.21.15.27:0/4154310619 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f109415a1b0 msgr2=0x7f10941488b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.312+0000 7f1092d9d700 1 --2- 172.21.15.27:0/4154310619 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f109415a1b0 0x7f10941488b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.312+0000 7f1092d9d700 1 -- 172.21.15.27:0/4154310619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10940ccd70 msgr2=0x7f10941481c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.312+0000 7f1092d9d700 1 --2- 172.21.15.27:0/4154310619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10940ccd70 0x7f10941481c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.312+0000 7f1092d9d700 1 -- 172.21.15.27:0/4154310619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1088009420 con 0x7f109415ae70 2024-01-11T08:28:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.312+0000 7f1092d9d700 1 --2- 172.21.15.27:0/4154310619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f109415ae70 0x7f10940c5750 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f108c00c8e0 tx=0x7f108c001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.312+0000 7f10827fc700 1 -- 172.21.15.27:0/4154310619 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108c012630 con 0x7f109415ae70 2024-01-11T08:28:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.312+0000 7f10827fc700 1 -- 172.21.15.27:0/4154310619 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f108c012790 con 0x7f109415ae70 2024-01-11T08:28:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.312+0000 7f10827fc700 1 -- 172.21.15.27:0/4154310619 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108c012b50 con 0x7f109415ae70 2024-01-11T08:28:07.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.312+0000 7f1098bff700 1 -- 172.21.15.27:0/4154310619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10940c91b0 con 0x7f109415ae70 2024-01-11T08:28:07.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.312+0000 7f1098bff700 1 -- 172.21.15.27:0/4154310619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10941556a0 con 0x7f109415ae70 2024-01-11T08:28:07.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.314+0000 7f1098bff700 1 -- 172.21.15.27:0/4154310619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f109404c3c0 con 0x7f109415ae70 2024-01-11T08:28:07.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.315+0000 7f10827fc700 1 -- 172.21.15.27:0/4154310619 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 56) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f108c01a3a0 con 0x7f109415ae70 2024-01-11T08:28:07.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.316+0000 7f10827fc700 1 --2- 172.21.15.27:0/4154310619 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1074066ea0 0x7f1074069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:07.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.316+0000 7f10827fc700 1 -- 172.21.15.27:0/4154310619 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5569+0+0 (secure 0 0 0) 0x7f108c084610 con 0x7f109415ae70 2024-01-11T08:28:07.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.316+0000 7f109259c700 1 --2- 172.21.15.27:0/4154310619 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1074066ea0 0x7f1074069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:07.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.317+0000 7f109259c700 1 --2- 172.21.15.27:0/4154310619 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1074066ea0 0x7f1074069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1084007fd0 tx=0x7f1084009040 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:07.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.320+0000 7f10827fc700 1 -- 172.21.15.27:0/4154310619 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f108c051940 con 0x7f109415ae70 2024-01-11T08:28:07.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.522+0000 7f1098bff700 1 -- 172.21.15.27:0/4154310619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f109415aa40 con 0x7f109415ae70 2024-01-11T08:28:07.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.527+0000 7f10827fc700 1 -- 172.21.15.27:0/4154310619 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v37) v1 ==== 120+0+0 (secure 0 0 0) 0x7f108c054f60 con 0x7f109415ae70 2024-01-11T08:28:07.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.529+0000 7f1098bff700 1 -- 172.21.15.27:0/4154310619 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1074066ea0 msgr2=0x7f1074069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:07.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.529+0000 7f1098bff700 1 --2- 172.21.15.27:0/4154310619 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1074066ea0 0x7f1074069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1084007fd0 tx=0x7f1084009040 comp rx=0 tx=0).stop 2024-01-11T08:28:07.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.529+0000 7f1098bff700 1 -- 172.21.15.27:0/4154310619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f109415ae70 msgr2=0x7f10940c5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:07.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.529+0000 7f1098bff700 1 --2- 172.21.15.27:0/4154310619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f109415ae70 0x7f10940c5750 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f108c00c8e0 tx=0x7f108c001710 comp rx=0 tx=0).stop 2024-01-11T08:28:07.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.529+0000 7f1098bff700 1 -- 172.21.15.27:0/4154310619 shutdown_connections 2024-01-11T08:28:07.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.529+0000 7f1098bff700 1 --2- 172.21.15.27:0/4154310619 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1074066ea0 0x7f1074069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.529+0000 7f1098bff700 1 --2- 172.21.15.27:0/4154310619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f109415ae70 0x7f10940c5750 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.529+0000 7f1098bff700 1 --2- 172.21.15.27:0/4154310619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10940ccd70 0x7f10941481c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.529+0000 7f1098bff700 1 --2- 172.21.15.27:0/4154310619 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f109415a1b0 0x7f10941488b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.529+0000 7f1098bff700 1 -- 172.21.15.27:0/4154310619 >> 172.21.15.27:0/4154310619 conn(0x7f109406ed40 msgr2=0x7f10940d0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:07.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.529+0000 7f1098bff700 1 -- 172.21.15.27:0/4154310619 shutdown_connections 2024-01-11T08:28:07.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.529+0000 7f1098bff700 1 -- 172.21.15.27:0/4154310619 wait complete. 2024-01-11T08:28:07.546 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-11T08:28:07.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.748+0000 7f3c5afff700 1 Processor -- start 2024-01-11T08:28:07.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.749+0000 7f3c5afff700 1 -- start start 2024-01-11T08:28:07.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.750+0000 7f3c5afff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3c54069540 0x7f3c54162270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:07.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.750+0000 7f3c5afff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c540699d0 con 0x7f3c541577e0 2024-01-11T08:28:07.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.750+0000 7f3c5afff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c54069cb0 con 0x7f3c54157410 2024-01-11T08:28:07.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.750+0000 7f3c5afff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c54069f90 con 0x7f3c54069540 2024-01-11T08:28:07.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.750+0000 7f3c5959c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3c54069540 0x7f3c54162270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:07.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.750+0000 7f3c5959c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3c54069540 0x7f3c54162270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56190/0 (socket says 172.21.15.27:56190) 2024-01-11T08:28:07.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.750+0000 7f3c5959c700 1 -- 172.21.15.27:0/3860191271 learned_addr learned my addr 172.21.15.27:0/3860191271 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:07.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.751+0000 7f3c5959c700 1 -- 172.21.15.27:0/3860191271 >> v1:172.21.15.77:6789/0 conn(0x7f3c54157410 legacy=0x7f3c540ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:07.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.751+0000 7f3c5959c700 1 -- 172.21.15.27:0/3860191271 >> v1:172.21.15.27:6789/0 conn(0x7f3c541577e0 legacy=0x7f3c540ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:07.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.751+0000 7f3c5959c700 1 -- 172.21.15.27:0/3860191271 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5406a270 con 0x7f3c54069540 2024-01-11T08:28:07.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.751+0000 7f3c5959c700 1 --2- 172.21.15.27:0/3860191271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c54069540 0x7f3c54162270 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c009f20 tx=0x7f3c4c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb32765eef45c9d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:07.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.751+0000 7f3c537fe700 1 -- 172.21.15.27:0/3860191271 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c00e070 con 0x7f3c54069540 2024-01-11T08:28:07.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.751+0000 7f3c537fe700 1 -- 172.21.15.27:0/3860191271 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c4c00ba80 con 0x7f3c54069540 2024-01-11T08:28:07.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.751+0000 7f3c537fe700 1 -- 172.21.15.27:0/3860191271 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c0092e0 con 0x7f3c54069540 2024-01-11T08:28:07.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.752+0000 7f3c5afff700 1 -- 172.21.15.27:0/3860191271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c54069540 msgr2=0x7f3c54162270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:07.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.752+0000 7f3c5afff700 1 --2- 172.21.15.27:0/3860191271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c54069540 0x7f3c54162270 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c009f20 tx=0x7f3c4c009840 comp rx=0 tx=0).stop 2024-01-11T08:28:07.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.752+0000 7f3c5afff700 1 -- 172.21.15.27:0/3860191271 shutdown_connections 2024-01-11T08:28:07.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.752+0000 7f3c5afff700 1 --2- 172.21.15.27:0/3860191271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c54069540 0x7f3c54162270 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.752+0000 7f3c5afff700 1 -- 172.21.15.27:0/3860191271 >> 172.21.15.27:0/3860191271 conn(0x7f3c540680a0 msgr2=0x7f3c540cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:07.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.752+0000 7f3c5afff700 1 -- 172.21.15.27:0/3860191271 shutdown_connections 2024-01-11T08:28:07.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.752+0000 7f3c5afff700 1 -- 172.21.15.27:0/3860191271 wait complete. 2024-01-11T08:28:07.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.754+0000 7f3c5afff700 1 Processor -- start 2024-01-11T08:28:07.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.754+0000 7f3c5afff700 1 -- start start 2024-01-11T08:28:07.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.755+0000 7f3c5afff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c54069540 0x7f3c541431c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:07.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.755+0000 7f3c5afff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c54157410 0x7f3c540c1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:07.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.755+0000 7f3c5afff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c541577e0 0x7f3c540c1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:07.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.755+0000 7f3c5afff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c540b1ad0 con 0x7f3c54157410 2024-01-11T08:28:07.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.755+0000 7f3c5afff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c540b0ec0 con 0x7f3c54069540 2024-01-11T08:28:07.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.755+0000 7f3c5afff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c540b11a0 con 0x7f3c541577e0 2024-01-11T08:28:07.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.755+0000 7f3c53fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c54157410 0x7f3c540c1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:07.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.755+0000 7f3c53fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c54157410 0x7f3c540c1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56202/0 (socket says 172.21.15.27:56202) 2024-01-11T08:28:07.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.755+0000 7f3c53fff700 1 -- 172.21.15.27:0/1555486819 learned_addr learned my addr 172.21.15.27:0/1555486819 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:07.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.755+0000 7f3c58d9b700 1 --2- 172.21.15.27:0/1555486819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c54069540 0x7f3c541431c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:07.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.755+0000 7f3c5959c700 1 --2- 172.21.15.27:0/1555486819 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c541577e0 0x7f3c540c1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:07.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.755+0000 7f3c53fff700 1 -- 172.21.15.27:0/1555486819 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c541577e0 msgr2=0x7f3c540c1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:07.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.755+0000 7f3c53fff700 1 --2- 172.21.15.27:0/1555486819 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c541577e0 0x7f3c540c1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.755+0000 7f3c53fff700 1 -- 172.21.15.27:0/1555486819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c54069540 msgr2=0x7f3c541431c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:07.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.755+0000 7f3c53fff700 1 --2- 172.21.15.27:0/1555486819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c54069540 0x7f3c541431c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.756+0000 7f3c53fff700 1 -- 172.21.15.27:0/1555486819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4c009ae0 con 0x7f3c54157410 2024-01-11T08:28:07.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.756+0000 7f3c58d9b700 1 --2- 172.21.15.27:0/1555486819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c54069540 0x7f3c541431c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:07.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.756+0000 7f3c53fff700 1 --2- 172.21.15.27:0/1555486819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c54157410 0x7f3c540c1010 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f3c4000c590 tx=0x7f3c40007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:07.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.756+0000 7f3c50ff9700 1 -- 172.21.15.27:0/1555486819 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c40011590 con 0x7f3c54157410 2024-01-11T08:28:07.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.756+0000 7f3c50ff9700 1 -- 172.21.15.27:0/1555486819 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c40011710 con 0x7f3c54157410 2024-01-11T08:28:07.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.756+0000 7f3c50ff9700 1 -- 172.21.15.27:0/1555486819 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c40011ad0 con 0x7f3c54157410 2024-01-11T08:28:07.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.756+0000 7f3c5afff700 1 -- 172.21.15.27:0/1555486819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c540b1480 con 0x7f3c54157410 2024-01-11T08:28:07.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.756+0000 7f3c5afff700 1 -- 172.21.15.27:0/1555486819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c54143700 con 0x7f3c54157410 2024-01-11T08:28:07.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.757+0000 7f3c5afff700 1 -- 172.21.15.27:0/1555486819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c5404c3c0 con 0x7f3c54157410 2024-01-11T08:28:07.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.758+0000 7f3c50ff9700 1 -- 172.21.15.27:0/1555486819 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 56) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f3c4001a360 con 0x7f3c54157410 2024-01-11T08:28:07.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.759+0000 7f3c50ff9700 1 --2- 172.21.15.27:0/1555486819 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3c38066ea0 0x7f3c38069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:07.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.759+0000 7f3c50ff9700 1 -- 172.21.15.27:0/1555486819 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5569+0+0 (secure 0 0 0) 0x7f3c40083450 con 0x7f3c54157410 2024-01-11T08:28:07.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.759+0000 7f3c58d9b700 1 --2- 172.21.15.27:0/1555486819 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3c38066ea0 0x7f3c38069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:07.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.760+0000 7f3c58d9b700 1 --2- 172.21.15.27:0/1555486819 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3c38066ea0 0x7f3c38069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3c480018f0 tx=0x7f3c48009040 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:07.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.763+0000 7f3c50ff9700 1 -- 172.21.15.27:0/1555486819 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3c40050780 con 0x7f3c54157410 2024-01-11T08:28:07.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.977+0000 7f3c5afff700 1 -- 172.21.15.27:0/1555486819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f3c540b1e60 con 0x7f3c54157410 2024-01-11T08:28:07.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.981+0000 7f3c50ff9700 1 -- 172.21.15.27:0/1555486819 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v38) v1 ==== 115+0+0 (secure 0 0 0) 0x7f3c40053da0 con 0x7f3c54157410 2024-01-11T08:28:07.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.983+0000 7f3c5afff700 1 -- 172.21.15.27:0/1555486819 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3c38066ea0 msgr2=0x7f3c38069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:07.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.983+0000 7f3c5afff700 1 --2- 172.21.15.27:0/1555486819 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3c38066ea0 0x7f3c38069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3c480018f0 tx=0x7f3c48009040 comp rx=0 tx=0).stop 2024-01-11T08:28:07.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.983+0000 7f3c5afff700 1 -- 172.21.15.27:0/1555486819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c54157410 msgr2=0x7f3c540c1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:07.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.983+0000 7f3c5afff700 1 --2- 172.21.15.27:0/1555486819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c54157410 0x7f3c540c1010 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f3c4000c590 tx=0x7f3c40007f50 comp rx=0 tx=0).stop 2024-01-11T08:28:07.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.983+0000 7f3c5afff700 1 -- 172.21.15.27:0/1555486819 shutdown_connections 2024-01-11T08:28:07.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.983+0000 7f3c5afff700 1 --2- 172.21.15.27:0/1555486819 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3c38066ea0 0x7f3c38069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.983+0000 7f3c5afff700 1 --2- 172.21.15.27:0/1555486819 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3c541577e0 0x7f3c540c1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.983+0000 7f3c5afff700 1 --2- 172.21.15.27:0/1555486819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c54069540 0x7f3c541431c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.983+0000 7f3c5afff700 1 --2- 172.21.15.27:0/1555486819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c54157410 0x7f3c540c1010 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:07.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.983+0000 7f3c5afff700 1 -- 172.21.15.27:0/1555486819 >> 172.21.15.27:0/1555486819 conn(0x7f3c540680a0 msgr2=0x7f3c540c7620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:07.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.984+0000 7f3c5afff700 1 -- 172.21.15.27:0/1555486819 shutdown_connections 2024-01-11T08:28:07.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:07.984+0000 7f3c5afff700 1 -- 172.21.15.27:0/1555486819 wait complete. 2024-01-11T08:28:08.000 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-11T08:28:08.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.203+0000 7f175b1ff700 1 Processor -- start 2024-01-11T08:28:08.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.203+0000 7f175b1ff700 1 -- start start 2024-01-11T08:28:08.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.204+0000 7f175b1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1754142dc0 0x7f17540cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:08.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.204+0000 7f175b1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17540cb2e0 0x7f17540cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:08.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.204+0000 7f175b1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17540ccc70 con 0x7f1754143190 2024-01-11T08:28:08.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.205+0000 7f175b1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f175415df30 con 0x7f1754142dc0 2024-01-11T08:28:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.205+0000 7f175b1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17540cb740 con 0x7f17540cb2e0 2024-01-11T08:28:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.205+0000 7f1758f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1754142dc0 0x7f17540cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.205+0000 7f1758f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1754142dc0 0x7f17540cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56214/0 (socket says 172.21.15.27:56214) 2024-01-11T08:28:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.205+0000 7f1758f9b700 1 -- 172.21.15.27:0/3484273863 learned_addr learned my addr 172.21.15.27:0/3484273863 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.205+0000 7f1753fff700 1 --1- 172.21.15.27:0/3484273863 >> v1:172.21.15.27:6789/0 conn(0x7f1754143190 0x7f17540cc020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34032/0 (socket says 172.21.15.27:34032) 2024-01-11T08:28:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.205+0000 7f175979c700 1 --2- 172.21.15.27:0/3484273863 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17540cb2e0 0x7f17540cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.205+0000 7f1758f9b700 1 -- 172.21.15.27:0/3484273863 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17540cb2e0 msgr2=0x7f17540cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.205+0000 7f1758f9b700 1 --2- 172.21.15.27:0/3484273863 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17540cb2e0 0x7f17540cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.205+0000 7f1758f9b700 1 -- 172.21.15.27:0/3484273863 >> v1:172.21.15.27:6789/0 conn(0x7f1754143190 legacy=0x7f17540cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.205+0000 7f1758f9b700 1 -- 172.21.15.27:0/3484273863 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f175415d320 con 0x7f1754142dc0 2024-01-11T08:28:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.206+0000 7f1758f9b700 1 --2- 172.21.15.27:0/3484273863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1754142dc0 0x7f17540cada0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f17480099b0 tx=0x7f1748009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd408dff5d9bad server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:08.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.206+0000 7f17537fe700 1 -- 172.21.15.27:0/3484273863 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174800b860 con 0x7f1754142dc0 2024-01-11T08:28:08.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.206+0000 7f17537fe700 1 -- 172.21.15.27:0/3484273863 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174800b9c0 con 0x7f1754142dc0 2024-01-11T08:28:08.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.206+0000 7f17537fe700 1 -- 172.21.15.27:0/3484273863 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174800bd60 con 0x7f1754142dc0 2024-01-11T08:28:08.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.206+0000 7f175b1ff700 1 -- 172.21.15.27:0/3484273863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1754142dc0 msgr2=0x7f17540cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:08.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.206+0000 7f175b1ff700 1 --2- 172.21.15.27:0/3484273863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1754142dc0 0x7f17540cada0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f17480099b0 tx=0x7f1748009250 comp rx=0 tx=0).stop 2024-01-11T08:28:08.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.207+0000 7f175b1ff700 1 -- 172.21.15.27:0/3484273863 shutdown_connections 2024-01-11T08:28:08.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.207+0000 7f175b1ff700 1 --2- 172.21.15.27:0/3484273863 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17540cb2e0 0x7f17540cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:08.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.207+0000 7f175b1ff700 1 --2- 172.21.15.27:0/3484273863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1754142dc0 0x7f17540cada0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:08.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.207+0000 7f175b1ff700 1 -- 172.21.15.27:0/3484273863 >> 172.21.15.27:0/3484273863 conn(0x7f1754063ca0 msgr2=0x7f175406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:08.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.207+0000 7f175b1ff700 1 -- 172.21.15.27:0/3484273863 shutdown_connections 2024-01-11T08:28:08.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.207+0000 7f175b1ff700 1 -- 172.21.15.27:0/3484273863 wait complete. 2024-01-11T08:28:08.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.208+0000 7f175b1ff700 1 Processor -- start 2024-01-11T08:28:08.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.208+0000 7f175b1ff700 1 -- start start 2024-01-11T08:28:08.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.209+0000 7f175b1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17540cb2e0 0x7f1754156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:08.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.209+0000 7f175b1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1754142dc0 0x7f17540cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:08.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.209+0000 7f175b1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1754143190 0x7f17540c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:08.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.209+0000 7f175b1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1754157070 con 0x7f1754142dc0 2024-01-11T08:28:08.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.209+0000 7f175b1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175416cfa0 con 0x7f1754143190 2024-01-11T08:28:08.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.209+0000 7f175b1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1754169d50 con 0x7f17540cb2e0 2024-01-11T08:28:08.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.209+0000 7f1753fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1754142dc0 0x7f17540cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:08.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.209+0000 7f1753fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1754142dc0 0x7f17540cbfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56224/0 (socket says 172.21.15.27:56224) 2024-01-11T08:28:08.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.209+0000 7f1753fff700 1 -- 172.21.15.27:0/1815540173 learned_addr learned my addr 172.21.15.27:0/1815540173 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:08.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.210+0000 7f1758f9b700 1 --2- 172.21.15.27:0/1815540173 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17540cb2e0 0x7f1754156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:08.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.210+0000 7f175979c700 1 --2- 172.21.15.27:0/1815540173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1754143190 0x7f17540c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:08.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.210+0000 7f1753fff700 1 -- 172.21.15.27:0/1815540173 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17540cb2e0 msgr2=0x7f1754156b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:08.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.210+0000 7f1753fff700 1 --2- 172.21.15.27:0/1815540173 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17540cb2e0 0x7f1754156b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:08.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.210+0000 7f1753fff700 1 -- 172.21.15.27:0/1815540173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1754143190 msgr2=0x7f17540c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:08.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.210+0000 7f1753fff700 1 --2- 172.21.15.27:0/1815540173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1754143190 0x7f17540c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:08.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.210+0000 7f1753fff700 1 -- 172.21.15.27:0/1815540173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17480094f0 con 0x7f1754142dc0 2024-01-11T08:28:08.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.210+0000 7f1758f9b700 1 --2- 172.21.15.27:0/1815540173 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17540cb2e0 0x7f1754156b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:08.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.210+0000 7f1753fff700 1 --2- 172.21.15.27:0/1815540173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1754142dc0 0x7f17540cbfb0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f174000c560 tx=0x7f1740007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:08.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.210+0000 7f1750ff9700 1 -- 172.21.15.27:0/1815540173 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1740005a70 con 0x7f1754142dc0 2024-01-11T08:28:08.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.210+0000 7f1750ff9700 1 -- 172.21.15.27:0/1815540173 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1740005bd0 con 0x7f1754142dc0 2024-01-11T08:28:08.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.210+0000 7f175b1ff700 1 -- 172.21.15.27:0/1815540173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1754169ff0 con 0x7f1754142dc0 2024-01-11T08:28:08.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.210+0000 7f1750ff9700 1 -- 172.21.15.27:0/1815540173 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17400070d0 con 0x7f1754142dc0 2024-01-11T08:28:08.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.211+0000 7f175b1ff700 1 -- 172.21.15.27:0/1815540173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f175416a5b0 con 0x7f1754142dc0 2024-01-11T08:28:08.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.212+0000 7f175b1ff700 1 -- 172.21.15.27:0/1815540173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f175404c3c0 con 0x7f1754142dc0 2024-01-11T08:28:08.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.213+0000 7f1750ff9700 1 -- 172.21.15.27:0/1815540173 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 56) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f1740007440 con 0x7f1754142dc0 2024-01-11T08:28:08.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.213+0000 7f1750ff9700 1 --2- 172.21.15.27:0/1815540173 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1738066ea0 0x7f1738069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:08.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.214+0000 7f1750ff9700 1 -- 172.21.15.27:0/1815540173 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5569+0+0 (secure 0 0 0) 0x7f17400838d0 con 0x7f1754142dc0 2024-01-11T08:28:08.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.214+0000 7f1758f9b700 1 --2- 172.21.15.27:0/1815540173 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1738066ea0 0x7f1738069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:08.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.215+0000 7f1758f9b700 1 --2- 172.21.15.27:0/1815540173 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1738066ea0 0x7f1738069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f17480099b0 tx=0x7f1748017000 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:08.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.217+0000 7f1750ff9700 1 -- 172.21.15.27:0/1815540173 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1740050c00 con 0x7f1754142dc0 2024-01-11T08:28:08.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.422+0000 7f175b1ff700 1 -- 172.21.15.27:0/1815540173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f175415d590 con 0x7f1754142dc0 2024-01-11T08:28:08.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.427+0000 7f1750ff9700 1 -- 172.21.15.27:0/1815540173 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v39) v1 ==== 104+0+0 (secure 0 0 0) 0x7f17400170a0 con 0x7f1754142dc0 2024-01-11T08:28:08.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.429+0000 7f175b1ff700 1 -- 172.21.15.27:0/1815540173 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1738066ea0 msgr2=0x7f1738069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:08.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.429+0000 7f175b1ff700 1 --2- 172.21.15.27:0/1815540173 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1738066ea0 0x7f1738069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f17480099b0 tx=0x7f1748017000 comp rx=0 tx=0).stop 2024-01-11T08:28:08.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.429+0000 7f175b1ff700 1 -- 172.21.15.27:0/1815540173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1754142dc0 msgr2=0x7f17540cbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:08.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.429+0000 7f175b1ff700 1 --2- 172.21.15.27:0/1815540173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1754142dc0 0x7f17540cbfb0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f174000c560 tx=0x7f1740007f70 comp rx=0 tx=0).stop 2024-01-11T08:28:08.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.429+0000 7f175b1ff700 1 -- 172.21.15.27:0/1815540173 shutdown_connections 2024-01-11T08:28:08.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.429+0000 7f175b1ff700 1 --2- 172.21.15.27:0/1815540173 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1738066ea0 0x7f1738069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:08.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.429+0000 7f175b1ff700 1 --2- 172.21.15.27:0/1815540173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1754143190 0x7f17540c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:08.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.429+0000 7f175b1ff700 1 --2- 172.21.15.27:0/1815540173 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17540cb2e0 0x7f1754156b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:08.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.429+0000 7f175b1ff700 1 --2- 172.21.15.27:0/1815540173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1754142dc0 0x7f17540cbfb0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:08.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.429+0000 7f175b1ff700 1 -- 172.21.15.27:0/1815540173 >> 172.21.15.27:0/1815540173 conn(0x7f1754063ca0 msgr2=0x7f17540c43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:08.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.429+0000 7f175b1ff700 1 -- 172.21.15.27:0/1815540173 shutdown_connections 2024-01-11T08:28:08.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.429+0000 7f175b1ff700 1 -- 172.21.15.27:0/1815540173 wait complete. 2024-01-11T08:28:08.447 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2024-01-11T08:28:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.651+0000 7f89333ff700 1 Processor -- start 2024-01-11T08:28:08.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.651+0000 7f89333ff700 1 -- start start 2024-01-11T08:28:08.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.652+0000 7f89333ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f892c0b03e0 0x7f892c0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:08.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.652+0000 7f89333ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c0cd730 con 0x7f892c0b29c0 2024-01-11T08:28:08.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.652+0000 7f89333ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c0b2de0 con 0x7f892c0b07b0 2024-01-11T08:28:08.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.652+0000 7f89333ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f892c0ccb20 con 0x7f892c0b03e0 2024-01-11T08:28:08.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.652+0000 7f893119b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f892c0b03e0 0x7f892c0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:08.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.653+0000 7f893119b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f892c0b03e0 0x7f892c0b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56236/0 (socket says 172.21.15.27:56236) 2024-01-11T08:28:08.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.653+0000 7f893119b700 1 -- 172.21.15.27:0/3761950215 learned_addr learned my addr 172.21.15.27:0/3761950215 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:08.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.653+0000 7f893119b700 1 -- 172.21.15.27:0/3761950215 >> v1:172.21.15.77:6789/0 conn(0x7f892c0b07b0 legacy=0x7f892c159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:08.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.653+0000 7f893119b700 1 -- 172.21.15.27:0/3761950215 >> v1:172.21.15.27:6789/0 conn(0x7f892c0b29c0 legacy=0x7f892c0aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:08.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.653+0000 7f893119b700 1 -- 172.21.15.27:0/3761950215 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c0cce00 con 0x7f892c0b03e0 2024-01-11T08:28:08.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.653+0000 7f893119b700 1 --2- 172.21.15.27:0/3761950215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892c0b03e0 0x7f892c0b2480 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f89280099b0 tx=0x7f8928009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c749817838833198 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:08.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.654+0000 7f8923fff700 1 -- 172.21.15.27:0/3761950215 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892800b860 con 0x7f892c0b03e0 2024-01-11T08:28:08.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.654+0000 7f8923fff700 1 -- 172.21.15.27:0/3761950215 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892800b9c0 con 0x7f892c0b03e0 2024-01-11T08:28:08.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.654+0000 7f8923fff700 1 -- 172.21.15.27:0/3761950215 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892800bd60 con 0x7f892c0b03e0 2024-01-11T08:28:08.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.654+0000 7f89333ff700 1 -- 172.21.15.27:0/3761950215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892c0b03e0 msgr2=0x7f892c0b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:08.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.654+0000 7f89333ff700 1 --2- 172.21.15.27:0/3761950215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892c0b03e0 0x7f892c0b2480 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f89280099b0 tx=0x7f8928009250 comp rx=0 tx=0).stop 2024-01-11T08:28:08.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.654+0000 7f89333ff700 1 -- 172.21.15.27:0/3761950215 shutdown_connections 2024-01-11T08:28:08.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.654+0000 7f89333ff700 1 --2- 172.21.15.27:0/3761950215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892c0b03e0 0x7f892c0b2480 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:08.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.654+0000 7f89333ff700 1 -- 172.21.15.27:0/3761950215 >> 172.21.15.27:0/3761950215 conn(0x7f892c06ed40 msgr2=0x7f892c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:08.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.654+0000 7f89333ff700 1 -- 172.21.15.27:0/3761950215 shutdown_connections 2024-01-11T08:28:08.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.654+0000 7f89333ff700 1 -- 172.21.15.27:0/3761950215 wait complete. 2024-01-11T08:28:08.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.656+0000 7f89333ff700 1 Processor -- start 2024-01-11T08:28:08.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.656+0000 7f89333ff700 1 -- start start 2024-01-11T08:28:08.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.657+0000 7f89333ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892c0b03e0 0x7f892c16b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:08.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.657+0000 7f89333ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f892c0b07b0 0x7f892c1696f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:08.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.657+0000 7f89333ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f892c0b29c0 0x7f892c0a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:08.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.657+0000 7f89333ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892c16aeb0 con 0x7f892c0b03e0 2024-01-11T08:28:08.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.657+0000 7f89333ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892c0c4000 con 0x7f892c0b07b0 2024-01-11T08:28:08.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.657+0000 7f89333ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892c0c42e0 con 0x7f892c0b29c0 2024-01-11T08:28:08.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.657+0000 7f893119b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892c0b03e0 0x7f892c16b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:08.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.657+0000 7f893119b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892c0b03e0 0x7f892c16b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56252/0 (socket says 172.21.15.27:56252) 2024-01-11T08:28:08.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.657+0000 7f893119b700 1 -- 172.21.15.27:0/1521009009 learned_addr learned my addr 172.21.15.27:0/1521009009 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:08.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.658+0000 7f893119b700 1 -- 172.21.15.27:0/1521009009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f892c0b29c0 msgr2=0x7f892c0a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:08.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.658+0000 7f893119b700 1 --2- 172.21.15.27:0/1521009009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f892c0b29c0 0x7f892c0a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:08.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.658+0000 7f893119b700 1 -- 172.21.15.27:0/1521009009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f892c0b07b0 msgr2=0x7f892c1696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:08.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.658+0000 7f893119b700 1 --2- 172.21.15.27:0/1521009009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f892c0b07b0 0x7f892c1696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:08.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.658+0000 7f893119b700 1 -- 172.21.15.27:0/1521009009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89280094f0 con 0x7f892c0b03e0 2024-01-11T08:28:08.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.658+0000 7f893119b700 1 --2- 172.21.15.27:0/1521009009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892c0b03e0 0x7f892c16b090 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f892c069540 tx=0x7f8928017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:08.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.658+0000 7f89217fa700 1 -- 172.21.15.27:0/1521009009 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8928011ea0 con 0x7f892c0b03e0 2024-01-11T08:28:08.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.658+0000 7f89217fa700 1 -- 172.21.15.27:0/1521009009 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892801a630 con 0x7f892c0b03e0 2024-01-11T08:28:08.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.658+0000 7f89217fa700 1 -- 172.21.15.27:0/1521009009 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892801a9d0 con 0x7f892c0b03e0 2024-01-11T08:28:08.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.658+0000 7f89333ff700 1 -- 172.21.15.27:0/1521009009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f892c0c45c0 con 0x7f892c0b03e0 2024-01-11T08:28:08.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.658+0000 7f89333ff700 1 -- 172.21.15.27:0/1521009009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f892c16b5d0 con 0x7f892c0b03e0 2024-01-11T08:28:08.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.659+0000 7f89333ff700 1 -- 172.21.15.27:0/1521009009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892c0c4ac0 con 0x7f892c0b03e0 2024-01-11T08:28:08.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.660+0000 7f89217fa700 1 -- 172.21.15.27:0/1521009009 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 56) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f892802d440 con 0x7f892c0b03e0 2024-01-11T08:28:08.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.662+0000 7f89217fa700 1 --2- 172.21.15.27:0/1521009009 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8910066ea0 0x7f8910069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:08.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.662+0000 7f89217fa700 1 -- 172.21.15.27:0/1521009009 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5569+0+0 (secure 0 0 0) 0x7f8928085f40 con 0x7f892c0b03e0 2024-01-11T08:28:08.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.663+0000 7f893099a700 1 --2- 172.21.15.27:0/1521009009 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8910066ea0 0x7f8910069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:08.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.663+0000 7f893099a700 1 --2- 172.21.15.27:0/1521009009 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8910066ea0 0x7f8910069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f892c15daf0 tx=0x7f891c007b40 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:08.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.666+0000 7f89217fa700 1 -- 172.21.15.27:0/1521009009 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8928053310 con 0x7f892c0b03e0 2024-01-11T08:28:08.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.952+0000 7f89333ff700 1 -- 172.21.15.27:0/1521009009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f892c0718d0 con 0x7f892c0b03e0 2024-01-11T08:28:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:08.988+0000 7f89217fa700 1 -- 172.21.15.27:0/1521009009 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f89280536e0 con 0x7f892c0b03e0 2024-01-11T08:28:09.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:09.906+0000 7f89217fa700 1 -- 172.21.15.27:0/1521009009 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v662) v1 ==== 183+0+0 (secure 0 0 0) 0x7f8928056da0 con 0x7f892c0b03e0 2024-01-11T08:28:09.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:09.909+0000 7f89333ff700 1 -- 172.21.15.27:0/1521009009 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8910066ea0 msgr2=0x7f8910069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:09.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:09.909+0000 7f89333ff700 1 --2- 172.21.15.27:0/1521009009 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8910066ea0 0x7f8910069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f892c15daf0 tx=0x7f891c007b40 comp rx=0 tx=0).stop 2024-01-11T08:28:09.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:09.910+0000 7f89333ff700 1 -- 172.21.15.27:0/1521009009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892c0b03e0 msgr2=0x7f892c16b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:09.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:09.910+0000 7f89333ff700 1 --2- 172.21.15.27:0/1521009009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892c0b03e0 0x7f892c16b090 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f892c069540 tx=0x7f8928017000 comp rx=0 tx=0).stop 2024-01-11T08:28:09.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:09.910+0000 7f89333ff700 1 -- 172.21.15.27:0/1521009009 shutdown_connections 2024-01-11T08:28:09.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:09.910+0000 7f89333ff700 1 --2- 172.21.15.27:0/1521009009 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8910066ea0 0x7f8910069350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:09.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:09.910+0000 7f89333ff700 1 --2- 172.21.15.27:0/1521009009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f892c0b29c0 0x7f892c0a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:09.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:09.910+0000 7f89333ff700 1 --2- 172.21.15.27:0/1521009009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892c0b03e0 0x7f892c16b090 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:09.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:09.910+0000 7f89333ff700 1 --2- 172.21.15.27:0/1521009009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f892c0b07b0 0x7f892c1696f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:09.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:09.910+0000 7f89333ff700 1 -- 172.21.15.27:0/1521009009 >> 172.21.15.27:0/1521009009 conn(0x7f892c06ed40 msgr2=0x7f892c15c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:09.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:09.910+0000 7f89333ff700 1 -- 172.21.15.27:0/1521009009 shutdown_connections 2024-01-11T08:28:09.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:09.910+0000 7f89333ff700 1 -- 172.21.15.27:0/1521009009 wait complete. 2024-01-11T08:28:09.930 INFO:tasks.cephfs_test_runner:Test that a non-existent path leads to schedule deactivation after a few retries. ... ok 2024-01-11T08:28:09.930 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2024-01-11T08:28:09.930 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-11T08:28:09.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2024-01-11T08:28:10.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.136+0000 7fa63b3ff700 1 Processor -- start 2024-01-11T08:28:10.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.136+0000 7fa63b3ff700 1 -- start start 2024-01-11T08:28:10.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.138+0000 7fa63b3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa63406b610 0x7fa634158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:10.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.138+0000 7fa63b3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa634158b60 con 0x7fa6341575e0 2024-01-11T08:28:10.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.138+0000 7fa63b3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6341599b0 con 0x7fa6340b19d0 2024-01-11T08:28:10.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.138+0000 7fa63b3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6340cb5e0 con 0x7fa63406b610 2024-01-11T08:28:10.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.138+0000 7fa63919b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa6341575e0 0x7fa6340b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34050/0 (socket says 172.21.15.27:34050) 2024-01-11T08:28:10.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.138+0000 7fa63919b700 1 -- 172.21.15.27:0/2959144917 learned_addr learned my addr 172.21.15.27:0/2959144917 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:28:10.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.138+0000 7fa63999c700 1 --2- 172.21.15.27:0/2959144917 >> v2:172.21.15.27:3300/0 conn(0x7fa63406b610 0x7fa634158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:10.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.138+0000 7fa63999c700 1 -- 172.21.15.27:0/2959144917 >> v1:172.21.15.77:6789/0 conn(0x7fa6340b19d0 legacy=0x7fa634159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:10.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.138+0000 7fa63999c700 1 -- 172.21.15.27:0/2959144917 >> v1:172.21.15.27:6789/0 conn(0x7fa6341575e0 legacy=0x7fa6340b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:10.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.138+0000 7fa63999c700 1 -- 172.21.15.27:0/2959144917 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6340c0f90 con 0x7fa63406b610 2024-01-11T08:28:10.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.139+0000 7fa63999c700 1 --2- 172.21.15.27:0/2959144917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63406b610 0x7fa634158620 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fa62c009e60 tx=0x7fa62c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea38db31cc9d7332 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:10.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.139+0000 7fa62bfff700 1 -- 172.21.15.27:0/2959144917 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c00e070 con 0x7fa63406b610 2024-01-11T08:28:10.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.139+0000 7fa62bfff700 1 -- 172.21.15.27:0/2959144917 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa62c00bc30 con 0x7fa63406b610 2024-01-11T08:28:10.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.139+0000 7fa62bfff700 1 -- 172.21.15.27:0/2959144917 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c012430 con 0x7fa63406b610 2024-01-11T08:28:10.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.139+0000 7fa63b3ff700 1 -- 172.21.15.27:0/2959144917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63406b610 msgr2=0x7fa634158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:10.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.140+0000 7fa63b3ff700 1 --2- 172.21.15.27:0/2959144917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63406b610 0x7fa634158620 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fa62c009e60 tx=0x7fa62c009780 comp rx=0 tx=0).stop 2024-01-11T08:28:10.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.140+0000 7fa63b3ff700 1 -- 172.21.15.27:0/2959144917 shutdown_connections 2024-01-11T08:28:10.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.140+0000 7fa63b3ff700 1 --2- 172.21.15.27:0/2959144917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa63406b610 0x7fa634158620 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:10.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.140+0000 7fa63b3ff700 1 -- 172.21.15.27:0/2959144917 >> 172.21.15.27:0/2959144917 conn(0x7fa6340680e0 msgr2=0x7fa634069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:10.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.140+0000 7fa63b3ff700 1 -- 172.21.15.27:0/2959144917 shutdown_connections 2024-01-11T08:28:10.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.140+0000 7fa63b3ff700 1 -- 172.21.15.27:0/2959144917 wait complete. 2024-01-11T08:28:10.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.141+0000 7fa63b3ff700 1 Processor -- start 2024-01-11T08:28:10.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.141+0000 7fa63b3ff700 1 -- start start 2024-01-11T08:28:10.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.142+0000 7fa63b3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa63406b610 0x7fa634153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:10.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.142+0000 7fa63b3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6340b19d0 0x7fa634169830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:10.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.142+0000 7fa63b3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6341575e0 0x7fa6340c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:10.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.142+0000 7fa63b3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6340b0310 con 0x7fa6341575e0 2024-01-11T08:28:10.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.142+0000 7fa63b3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6340af700 con 0x7fa6340b19d0 2024-01-11T08:28:10.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.142+0000 7fa63b3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6340af9e0 con 0x7fa63406b610 2024-01-11T08:28:10.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.143+0000 7fa63919b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa63406b610 0x7fa634153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:10.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.143+0000 7fa63999c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6341575e0 0x7fa6340c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:10.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.143+0000 7fa63899a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6340b19d0 0x7fa634169830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:10.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.143+0000 7fa63919b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa63406b610 0x7fa634153e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:36696/0 (socket says 172.21.15.27:36696) 2024-01-11T08:28:10.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.143+0000 7fa63919b700 1 -- 172.21.15.27:0/2405012711 learned_addr learned my addr 172.21.15.27:0/2405012711 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:10.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.143+0000 7fa63999c700 1 -- 172.21.15.27:0/2405012711 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa63406b610 msgr2=0x7fa634153e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:10.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.143+0000 7fa63999c700 1 --2- 172.21.15.27:0/2405012711 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa63406b610 0x7fa634153e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:10.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.143+0000 7fa63999c700 1 -- 172.21.15.27:0/2405012711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6340b19d0 msgr2=0x7fa634169830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:10.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.143+0000 7fa63999c700 1 --2- 172.21.15.27:0/2405012711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6340b19d0 0x7fa634169830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:10.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.143+0000 7fa63999c700 1 -- 172.21.15.27:0/2405012711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62c009a20 con 0x7fa6341575e0 2024-01-11T08:28:10.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.144+0000 7fa63999c700 1 --2- 172.21.15.27:0/2405012711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6341575e0 0x7fa6340c6f70 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fa62c016000 tx=0x7fa62c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:10.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.144+0000 7fa6297fa700 1 -- 172.21.15.27:0/2405012711 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c012e10 con 0x7fa6341575e0 2024-01-11T08:28:10.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.144+0000 7fa6297fa700 1 -- 172.21.15.27:0/2405012711 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa62c01e630 con 0x7fa6341575e0 2024-01-11T08:28:10.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.144+0000 7fa63b3ff700 1 -- 172.21.15.27:0/2405012711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6340afcc0 con 0x7fa6341575e0 2024-01-11T08:28:10.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.144+0000 7fa6297fa700 1 -- 172.21.15.27:0/2405012711 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c01e9f0 con 0x7fa6341575e0 2024-01-11T08:28:10.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.144+0000 7fa63b3ff700 1 -- 172.21.15.27:0/2405012711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6340c6290 con 0x7fa6341575e0 2024-01-11T08:28:10.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.145+0000 7fa63b3ff700 1 -- 172.21.15.27:0/2405012711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6340b0480 con 0x7fa6341575e0 2024-01-11T08:28:10.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.146+0000 7fa6297fa700 1 -- 172.21.15.27:0/2405012711 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa62c024070 con 0x7fa6341575e0 2024-01-11T08:28:10.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.147+0000 7fa6297fa700 1 --2- 172.21.15.27:0/2405012711 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa618066f60 0x7fa618069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:10.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.147+0000 7fa6297fa700 1 -- 172.21.15.27:0/2405012711 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5569+0+0 (secure 0 0 0) 0x7fa62c086c00 con 0x7fa6341575e0 2024-01-11T08:28:10.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.147+0000 7fa63919b700 1 --2- 172.21.15.27:0/2405012711 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa618066f60 0x7fa618069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:10.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.148+0000 7fa63919b700 1 --2- 172.21.15.27:0/2405012711 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa618066f60 0x7fa618069410 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa630007fd0 tx=0x7fa630009040 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:10.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.151+0000 7fa6297fa700 1 -- 172.21.15.27:0/2405012711 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa62c053e50 con 0x7fa6341575e0 2024-01-11T08:28:10.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.432+0000 7fa63b3ff700 1 -- 172.21.15.27:0/2405012711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7fa6340718d0 con 0x7fa6341575e0 2024-01-11T08:28:10.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.911+0000 7fa6297fa700 1 -- 172.21.15.27:0/2405012711 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v663) v1 ==== 151+0+0 (secure 0 0 0) 0x7fa62c02c020 con 0x7fa6341575e0 2024-01-11T08:28:10.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.914+0000 7fa63b3ff700 1 -- 172.21.15.27:0/2405012711 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa618066f60 msgr2=0x7fa618069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:10.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.914+0000 7fa63b3ff700 1 --2- 172.21.15.27:0/2405012711 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa618066f60 0x7fa618069410 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa630007fd0 tx=0x7fa630009040 comp rx=0 tx=0).stop 2024-01-11T08:28:10.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.914+0000 7fa63b3ff700 1 -- 172.21.15.27:0/2405012711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6341575e0 msgr2=0x7fa6340c6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:10.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.914+0000 7fa63b3ff700 1 --2- 172.21.15.27:0/2405012711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6341575e0 0x7fa6340c6f70 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fa62c016000 tx=0x7fa62c019000 comp rx=0 tx=0).stop 2024-01-11T08:28:10.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.914+0000 7fa63b3ff700 1 -- 172.21.15.27:0/2405012711 shutdown_connections 2024-01-11T08:28:10.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.914+0000 7fa63b3ff700 1 --2- 172.21.15.27:0/2405012711 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa618066f60 0x7fa618069410 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:10.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.914+0000 7fa63b3ff700 1 --2- 172.21.15.27:0/2405012711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6341575e0 0x7fa6340c6f70 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:10.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.914+0000 7fa63b3ff700 1 --2- 172.21.15.27:0/2405012711 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa63406b610 0x7fa634153e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:10.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.914+0000 7fa63b3ff700 1 --2- 172.21.15.27:0/2405012711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6340b19d0 0x7fa634169830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:10.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.914+0000 7fa63b3ff700 1 -- 172.21.15.27:0/2405012711 >> 172.21.15.27:0/2405012711 conn(0x7fa6340680e0 msgr2=0x7fa63406dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:10.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.914+0000 7fa63b3ff700 1 -- 172.21.15.27:0/2405012711 shutdown_connections 2024-01-11T08:28:10.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:10.914+0000 7fa63b3ff700 1 -- 172.21.15.27:0/2405012711 wait complete. 2024-01-11T08:28:10.932 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-11T08:28:11.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.137+0000 7fe62e1ff700 1 Processor -- start 2024-01-11T08:28:11.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.137+0000 7fe62e1ff700 1 -- start start 2024-01-11T08:28:11.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.138+0000 7fe62e1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe6281622d0 0x7fe6281646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:11.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.138+0000 7fe62e1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6280cf850 con 0x7fe62815a1b0 2024-01-11T08:28:11.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.138+0000 7fe62e1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6280cec40 con 0x7fe6280ccd70 2024-01-11T08:28:11.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.138+0000 7fe62e1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe62815ac90 con 0x7fe6281622d0 2024-01-11T08:28:11.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.139+0000 7fe6277fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe62815a1b0 0x7fe62815a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34064/0 (socket says 172.21.15.27:34064) 2024-01-11T08:28:11.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.139+0000 7fe6277fe700 1 -- 172.21.15.27:0/582034135 learned_addr learned my addr 172.21.15.27:0/582034135 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:28:11.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.139+0000 7fe627fff700 1 --2- 172.21.15.27:0/582034135 >> v2:172.21.15.27:3300/0 conn(0x7fe6281622d0 0x7fe6281646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:11.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.139+0000 7fe627fff700 1 -- 172.21.15.27:0/582034135 >> v1:172.21.15.77:6789/0 conn(0x7fe6280ccd70 legacy=0x7fe6280cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:11.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.139+0000 7fe627fff700 1 -- 172.21.15.27:0/582034135 >> v1:172.21.15.27:6789/0 conn(0x7fe62815a1b0 legacy=0x7fe62815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:11.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.139+0000 7fe627fff700 1 -- 172.21.15.27:0/582034135 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6280cef20 con 0x7fe6281622d0 2024-01-11T08:28:11.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.139+0000 7fe627fff700 1 --2- 172.21.15.27:0/582034135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6281622d0 0x7fe6281646b0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fe620009e60 tx=0x7fe620009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11361779a7d15de server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:11.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.140+0000 7fe6267fc700 1 -- 172.21.15.27:0/582034135 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62000e070 con 0x7fe6281622d0 2024-01-11T08:28:11.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.140+0000 7fe6267fc700 1 -- 172.21.15.27:0/582034135 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62000bc30 con 0x7fe6281622d0 2024-01-11T08:28:11.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.140+0000 7fe6267fc700 1 -- 172.21.15.27:0/582034135 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe620012430 con 0x7fe6281622d0 2024-01-11T08:28:11.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.140+0000 7fe62e1ff700 1 -- 172.21.15.27:0/582034135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6281622d0 msgr2=0x7fe6281646b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:11.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.140+0000 7fe62e1ff700 1 --2- 172.21.15.27:0/582034135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6281622d0 0x7fe6281646b0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fe620009e60 tx=0x7fe620009780 comp rx=0 tx=0).stop 2024-01-11T08:28:11.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.140+0000 7fe62e1ff700 1 -- 172.21.15.27:0/582034135 shutdown_connections 2024-01-11T08:28:11.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.140+0000 7fe62e1ff700 1 --2- 172.21.15.27:0/582034135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6281622d0 0x7fe6281646b0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:11.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.140+0000 7fe62e1ff700 1 -- 172.21.15.27:0/582034135 >> 172.21.15.27:0/582034135 conn(0x7fe62806ed40 msgr2=0x7fe62806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:11.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.140+0000 7fe62e1ff700 1 -- 172.21.15.27:0/582034135 shutdown_connections 2024-01-11T08:28:11.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.140+0000 7fe62e1ff700 1 -- 172.21.15.27:0/582034135 wait complete. 2024-01-11T08:28:11.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.142+0000 7fe62e1ff700 1 Processor -- start 2024-01-11T08:28:11.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.142+0000 7fe62e1ff700 1 -- start start 2024-01-11T08:28:11.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.143+0000 7fe62e1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6280ccd70 0x7fe62814b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:11.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.143+0000 7fe62e1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe62815a1b0 0x7fe6280b2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:11.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.143+0000 7fe62e1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6281622d0 0x7fe6280c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.143+0000 7fe62e1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62814b5f0 con 0x7fe6280ccd70 2024-01-11T08:28:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.143+0000 7fe62e1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6280b2ca0 con 0x7fe6281622d0 2024-01-11T08:28:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.143+0000 7fe62e1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6280b17b0 con 0x7fe62815a1b0 2024-01-11T08:28:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.143+0000 7fe6277fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6280ccd70 0x7fe62814b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.143+0000 7fe6277fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6280ccd70 0x7fe62814b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56290/0 (socket says 172.21.15.27:56290) 2024-01-11T08:28:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.143+0000 7fe6277fe700 1 -- 172.21.15.27:0/2898194860 learned_addr learned my addr 172.21.15.27:0/2898194860 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.143+0000 7fe626ffd700 1 --2- 172.21.15.27:0/2898194860 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe62815a1b0 0x7fe6280b2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.143+0000 7fe627fff700 1 --2- 172.21.15.27:0/2898194860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6281622d0 0x7fe6280c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.144+0000 7fe6277fe700 1 -- 172.21.15.27:0/2898194860 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe62815a1b0 msgr2=0x7fe6280b2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.144+0000 7fe6277fe700 1 --2- 172.21.15.27:0/2898194860 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe62815a1b0 0x7fe6280b2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.144+0000 7fe6277fe700 1 -- 172.21.15.27:0/2898194860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6281622d0 msgr2=0x7fe6280c8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.144+0000 7fe6277fe700 1 --2- 172.21.15.27:0/2898194860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6281622d0 0x7fe6280c8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.144+0000 7fe6277fe700 1 -- 172.21.15.27:0/2898194860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe620009a20 con 0x7fe6280ccd70 2024-01-11T08:28:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.144+0000 7fe627fff700 1 --2- 172.21.15.27:0/2898194860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6281622d0 0x7fe6280c8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:11.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.144+0000 7fe6277fe700 1 --2- 172.21.15.27:0/2898194860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6280ccd70 0x7fe62814b0b0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fe61800c7c0 tx=0x7fe618007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:11.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.144+0000 7fe60f7fe700 1 -- 172.21.15.27:0/2898194860 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe618005b30 con 0x7fe6280ccd70 2024-01-11T08:28:11.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.144+0000 7fe60f7fe700 1 -- 172.21.15.27:0/2898194860 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe618005cb0 con 0x7fe6280ccd70 2024-01-11T08:28:11.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.144+0000 7fe60f7fe700 1 -- 172.21.15.27:0/2898194860 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe618004a80 con 0x7fe6280ccd70 2024-01-11T08:28:11.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.144+0000 7fe62e1ff700 1 -- 172.21.15.27:0/2898194860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6280b1c80 con 0x7fe6280ccd70 2024-01-11T08:28:11.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.144+0000 7fe62e1ff700 1 -- 172.21.15.27:0/2898194860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6280b22c0 con 0x7fe6280ccd70 2024-01-11T08:28:11.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.145+0000 7fe62e1ff700 1 -- 172.21.15.27:0/2898194860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6280b2450 con 0x7fe6280ccd70 2024-01-11T08:28:11.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.146+0000 7fe60f7fe700 1 -- 172.21.15.27:0/2898194860 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe6180070f0 con 0x7fe6280ccd70 2024-01-11T08:28:11.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.147+0000 7fe60f7fe700 1 --2- 172.21.15.27:0/2898194860 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe614066f30 0x7fe6140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:11.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.147+0000 7fe60f7fe700 1 -- 172.21.15.27:0/2898194860 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5569+0+0 (secure 0 0 0) 0x7fe6180838c0 con 0x7fe6280ccd70 2024-01-11T08:28:11.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.148+0000 7fe626ffd700 1 --2- 172.21.15.27:0/2898194860 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe614066f30 0x7fe6140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:11.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.148+0000 7fe626ffd700 1 --2- 172.21.15.27:0/2898194860 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe614066f30 0x7fe6140693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe62806fbd0 tx=0x7fe61c009040 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:11.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.152+0000 7fe60f7fe700 1 -- 172.21.15.27:0/2898194860 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe618050c30 con 0x7fe6280ccd70 2024-01-11T08:28:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.353+0000 7fe62e1ff700 1 -- 172.21.15.27:0/2898194860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe62804c3c0 con 0x7fe6280ccd70 2024-01-11T08:28:11.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.357+0000 7fe60f7fe700 1 -- 172.21.15.27:0/2898194860 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe618018070 con 0x7fe6280ccd70 2024-01-11T08:28:11.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.359+0000 7fe62e1ff700 1 -- 172.21.15.27:0/2898194860 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe614066f30 msgr2=0x7fe6140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:11.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.359+0000 7fe62e1ff700 1 --2- 172.21.15.27:0/2898194860 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe614066f30 0x7fe6140693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe62806fbd0 tx=0x7fe61c009040 comp rx=0 tx=0).stop 2024-01-11T08:28:11.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.359+0000 7fe62e1ff700 1 -- 172.21.15.27:0/2898194860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6280ccd70 msgr2=0x7fe62814b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:11.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.359+0000 7fe62e1ff700 1 --2- 172.21.15.27:0/2898194860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6280ccd70 0x7fe62814b0b0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fe61800c7c0 tx=0x7fe618007f70 comp rx=0 tx=0).stop 2024-01-11T08:28:11.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.359+0000 7fe62e1ff700 1 -- 172.21.15.27:0/2898194860 shutdown_connections 2024-01-11T08:28:11.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.359+0000 7fe62e1ff700 1 --2- 172.21.15.27:0/2898194860 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe614066f30 0x7fe6140693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:11.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.359+0000 7fe62e1ff700 1 --2- 172.21.15.27:0/2898194860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6281622d0 0x7fe6280c8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:11.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.359+0000 7fe62e1ff700 1 --2- 172.21.15.27:0/2898194860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6280ccd70 0x7fe62814b0b0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:11.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.359+0000 7fe62e1ff700 1 --2- 172.21.15.27:0/2898194860 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe62815a1b0 0x7fe6280b2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:11.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.359+0000 7fe62e1ff700 1 -- 172.21.15.27:0/2898194860 >> 172.21.15.27:0/2898194860 conn(0x7fe62806ed40 msgr2=0x7fe6280c6ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:11.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.359+0000 7fe62e1ff700 1 -- 172.21.15.27:0/2898194860 shutdown_connections 2024-01-11T08:28:11.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.359+0000 7fe62e1ff700 1 -- 172.21.15.27:0/2898194860 wait complete. 2024-01-11T08:28:11.377 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T08:28:11.377 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:28:11.469 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:28:11.569 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T08:28:11.569 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-11T08:28:11.570 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:28:11.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.770+0000 7f73269ff700 1 Processor -- start 2024-01-11T08:28:11.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.770+0000 7f73269ff700 1 -- start start 2024-01-11T08:28:11.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.771+0000 7f73269ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f73200ca7b0 0x7f73200cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:11.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.772+0000 7f73269ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f732015d640 con 0x7f7320159e80 2024-01-11T08:28:11.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.772+0000 7f73269ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f732015a5d0 con 0x7f73200ca3e0 2024-01-11T08:28:11.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.772+0000 7f73269ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f732015ae70 con 0x7f73200ca7b0 2024-01-11T08:28:11.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.772+0000 7f731f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f73200ca7b0 0x7f73200cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:11.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.772+0000 7f731f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f73200ca7b0 0x7f73200cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56306/0 (socket says 172.21.15.27:56306) 2024-01-11T08:28:11.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.772+0000 7f731f7fe700 1 -- 172.21.15.27:0/3794058067 learned_addr learned my addr 172.21.15.27:0/3794058067 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:11.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.772+0000 7f7324f9c700 1 --1- 172.21.15.27:0/3794058067 >> v1:172.21.15.27:6789/0 conn(0x7f7320159e80 0x7f732015dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34068/0 (socket says 172.21.15.27:34068) 2024-01-11T08:28:11.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.772+0000 7f731f7fe700 1 -- 172.21.15.27:0/3794058067 >> v1:172.21.15.77:6789/0 conn(0x7f73200ca3e0 legacy=0x7f73200c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:11.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.772+0000 7f731f7fe700 1 -- 172.21.15.27:0/3794058067 >> v1:172.21.15.27:6789/0 conn(0x7f7320159e80 legacy=0x7f732015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:11.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.772+0000 7f731f7fe700 1 -- 172.21.15.27:0/3794058067 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732015a8b0 con 0x7f73200ca7b0 2024-01-11T08:28:11.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.773+0000 7f731f7fe700 1 --2- 172.21.15.27:0/3794058067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73200ca7b0 0x7f73200cca20 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f73140099e0 tx=0x7f7314009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e3c2c7c482954a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:11.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.773+0000 7f731effd700 1 -- 172.21.15.27:0/3794058067 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731400b820 con 0x7f73200ca7b0 2024-01-11T08:28:11.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.773+0000 7f731effd700 1 -- 172.21.15.27:0/3794058067 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f731400b980 con 0x7f73200ca7b0 2024-01-11T08:28:11.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.773+0000 7f731effd700 1 -- 172.21.15.27:0/3794058067 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731400bd20 con 0x7f73200ca7b0 2024-01-11T08:28:11.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.773+0000 7f73269ff700 1 -- 172.21.15.27:0/3794058067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73200ca7b0 msgr2=0x7f73200cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:11.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.773+0000 7f73269ff700 1 --2- 172.21.15.27:0/3794058067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73200ca7b0 0x7f73200cca20 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f73140099e0 tx=0x7f7314009250 comp rx=0 tx=0).stop 2024-01-11T08:28:11.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.774+0000 7f73269ff700 1 -- 172.21.15.27:0/3794058067 shutdown_connections 2024-01-11T08:28:11.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.774+0000 7f73269ff700 1 --2- 172.21.15.27:0/3794058067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73200ca7b0 0x7f73200cca20 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.774+0000 7f73269ff700 1 -- 172.21.15.27:0/3794058067 >> 172.21.15.27:0/3794058067 conn(0x7f73200ce3e0 msgr2=0x7f73200ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.774+0000 7f73269ff700 1 -- 172.21.15.27:0/3794058067 shutdown_connections 2024-01-11T08:28:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.774+0000 7f73269ff700 1 -- 172.21.15.27:0/3794058067 wait complete. 2024-01-11T08:28:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.775+0000 7f73269ff700 1 Processor -- start 2024-01-11T08:28:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.775+0000 7f73269ff700 1 -- start start 2024-01-11T08:28:11.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.776+0000 7f73269ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73200ca3e0 0x7f73200b3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:11.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.776+0000 7f73269ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73200ca7b0 0x7f73200c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:11.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.776+0000 7f73269ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7320159e80 0x7f7320140a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:11.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.776+0000 7f73269ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732016e450 con 0x7f73200ca3e0 2024-01-11T08:28:11.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.776+0000 7f73269ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732016d840 con 0x7f73200ca7b0 2024-01-11T08:28:11.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.776+0000 7f73269ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732016db20 con 0x7f7320159e80 2024-01-11T08:28:11.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.777+0000 7f731ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73200ca3e0 0x7f73200b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:11.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.777+0000 7f731ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73200ca3e0 0x7f73200b3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56320/0 (socket says 172.21.15.27:56320) 2024-01-11T08:28:11.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.777+0000 7f731ffff700 1 -- 172.21.15.27:0/257222745 learned_addr learned my addr 172.21.15.27:0/257222745 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:11.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.777+0000 7f731f7fe700 1 --2- 172.21.15.27:0/257222745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73200ca7b0 0x7f73200c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:11.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.777+0000 7f7324f9c700 1 --2- 172.21.15.27:0/257222745 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7320159e80 0x7f7320140a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:11.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.777+0000 7f731ffff700 1 -- 172.21.15.27:0/257222745 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7320159e80 msgr2=0x7f7320140a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:11.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.777+0000 7f731ffff700 1 --2- 172.21.15.27:0/257222745 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7320159e80 0x7f7320140a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:11.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.777+0000 7f731ffff700 1 -- 172.21.15.27:0/257222745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73200ca7b0 msgr2=0x7f73200c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:11.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.777+0000 7f731ffff700 1 --2- 172.21.15.27:0/257222745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73200ca7b0 0x7f73200c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:11.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.777+0000 7f731ffff700 1 -- 172.21.15.27:0/257222745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73140094f0 con 0x7f73200ca3e0 2024-01-11T08:28:11.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.777+0000 7f7324f9c700 1 --2- 172.21.15.27:0/257222745 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7320159e80 0x7f7320140a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:28:11.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.777+0000 7f731f7fe700 1 --2- 172.21.15.27:0/257222745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73200ca7b0 0x7f73200c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:11.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.777+0000 7f731ffff700 1 --2- 172.21.15.27:0/257222745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73200ca3e0 0x7f73200b3520 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f731000c4e0 tx=0x7f731000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:11.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.777+0000 7f7307fff700 1 -- 172.21.15.27:0/257222745 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7310011590 con 0x7f73200ca3e0 2024-01-11T08:28:11.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.778+0000 7f7307fff700 1 -- 172.21.15.27:0/257222745 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7310011710 con 0x7f73200ca3e0 2024-01-11T08:28:11.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.778+0000 7f7307fff700 1 -- 172.21.15.27:0/257222745 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7310011ad0 con 0x7f73200ca3e0 2024-01-11T08:28:11.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.778+0000 7f73269ff700 1 -- 172.21.15.27:0/257222745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f732016de00 con 0x7f73200ca3e0 2024-01-11T08:28:11.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.778+0000 7f73269ff700 1 -- 172.21.15.27:0/257222745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73201519c0 con 0x7f73200ca3e0 2024-01-11T08:28:11.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.779+0000 7f73269ff700 1 -- 172.21.15.27:0/257222745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73200718d0 con 0x7f73200ca3e0 2024-01-11T08:28:11.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.780+0000 7f7307fff700 1 -- 172.21.15.27:0/257222745 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f731001b3b0 con 0x7f73200ca3e0 2024-01-11T08:28:11.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.780+0000 7f7307fff700 1 --2- 172.21.15.27:0/257222745 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f730c066f30 0x7f730c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:11.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.781+0000 7f7307fff700 1 -- 172.21.15.27:0/257222745 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5569+0+0 (secure 0 0 0) 0x7f73100839e0 con 0x7f73200ca3e0 2024-01-11T08:28:11.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.781+0000 7f731f7fe700 1 --2- 172.21.15.27:0/257222745 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f730c066f30 0x7f730c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:11.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.781+0000 7f731f7fe700 1 --2- 172.21.15.27:0/257222745 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f730c066f30 0x7f730c0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f73140099e0 tx=0x7f7314017000 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:11.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:11.785+0000 7f7307fff700 1 -- 172.21.15.27:0/257222745 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7310050cd0 con 0x7f73200ca3e0 2024-01-11T08:28:12.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.035+0000 7f73269ff700 1 -- 172.21.15.27:0/257222745 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f732006c840 con 0x7f73200ca3e0 2024-01-11T08:28:12.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.035+0000 7f7307fff700 1 -- 172.21.15.27:0/257222745 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+3526 (secure 0 0 0) 0x7f7310017070 con 0x7f73200ca3e0 2024-01-11T08:28:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:28:12.034 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":52,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/921942372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":921942372},{"type":"v1","addr":"172.21.15.27:6834","nonce":921942372}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":6539,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2566970263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2566970263},{"type":"v1","addr":"172.21.15.27:6835","nonce":2566970263}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7080,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/3844286168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3844286168},{"type":"v1","addr":"172.21.15.77:6834","nonce":3844286168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":46},{"gid":7083,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3222788452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":3222788452},{"type":"v1","addr":"172.21.15.77:6837","nonce":3222788452}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[]} 2024-01-11T08:28:12.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.038+0000 7f73269ff700 1 -- 172.21.15.27:0/257222745 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f730c066f30 msgr2=0x7f730c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:12.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.038+0000 7f73269ff700 1 --2- 172.21.15.27:0/257222745 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f730c066f30 0x7f730c0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f73140099e0 tx=0x7f7314017000 comp rx=0 tx=0).stop 2024-01-11T08:28:12.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.038+0000 7f73269ff700 1 -- 172.21.15.27:0/257222745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73200ca3e0 msgr2=0x7f73200b3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:12.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.038+0000 7f73269ff700 1 --2- 172.21.15.27:0/257222745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73200ca3e0 0x7f73200b3520 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f731000c4e0 tx=0x7f731000c8a0 comp rx=0 tx=0).stop 2024-01-11T08:28:12.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.038+0000 7f73269ff700 1 -- 172.21.15.27:0/257222745 shutdown_connections 2024-01-11T08:28:12.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.038+0000 7f73269ff700 1 --2- 172.21.15.27:0/257222745 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f730c066f30 0x7f730c0693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:12.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.038+0000 7f73269ff700 1 --2- 172.21.15.27:0/257222745 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7320159e80 0x7f7320140a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:12.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.038+0000 7f73269ff700 1 --2- 172.21.15.27:0/257222745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73200ca3e0 0x7f73200b3520 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:12.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.038+0000 7f73269ff700 1 --2- 172.21.15.27:0/257222745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73200ca7b0 0x7f73200c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:12.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.038+0000 7f73269ff700 1 -- 172.21.15.27:0/257222745 >> 172.21.15.27:0/257222745 conn(0x7f73200ce3e0 msgr2=0x7f73200692a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:12.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.038+0000 7f73269ff700 1 -- 172.21.15.27:0/257222745 shutdown_connections 2024-01-11T08:28:12.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.038+0000 7f73269ff700 1 -- 172.21.15.27:0/257222745 wait complete. 2024-01-11T08:28:12.038 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 52 2024-01-11T08:28:12.054 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-11T08:28:12.055 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-11T08:28:12.055 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-11T08:28:12.055 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:28:12.055 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-11T08:28:12.055 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-11T08:28:12.056 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-11T08:28:12.056 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:28:12.056 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-11T08:28:12.056 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-11T08:28:12.056 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-11T08:28:12.056 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:28:12.057 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-11T08:28:12.057 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-11T08:28:12.057 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-11T08:28:12.057 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:28:12.070 INFO:tasks.ceph.mds.a:Stopped 2024-01-11T08:28:12.070 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-11T08:28:12.071 INFO:tasks.ceph.mds.c:Stopped 2024-01-11T08:28:12.071 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-11T08:28:12.072 INFO:tasks.ceph.mds.d:Stopped 2024-01-11T08:28:12.072 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-11T08:28:12.073 INFO:tasks.ceph.mds.b:Stopped 2024-01-11T08:28:12.073 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-11T08:28:12.115 INFO:tasks.ceph.mds.a:Started 2024-01-11T08:28:12.115 DEBUG:teuthology.parallel:result is None 2024-01-11T08:28:12.120 INFO:tasks.ceph.mds.d:Started 2024-01-11T08:28:12.121 INFO:tasks.ceph.mds.b:Started 2024-01-11T08:28:12.121 DEBUG:teuthology.parallel:result is None 2024-01-11T08:28:12.121 DEBUG:teuthology.parallel:result is None 2024-01-11T08:28:12.121 INFO:tasks.ceph.mds.c:Started 2024-01-11T08:28:12.122 DEBUG:teuthology.parallel:result is None 2024-01-11T08:28:12.122 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-11T08:28:12.262 INFO:tasks.ceph.mds.d.smithi077.stdout:starting mds.d at 2024-01-11T08:28:12.262 INFO:tasks.ceph.mds.b.smithi077.stdout:starting mds.b at 2024-01-11T08:28:12.269 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T08:28:12.270 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T08:28:12.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.339+0000 7f20749ff700 1 Processor -- start 2024-01-11T08:28:12.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.340+0000 7f20749ff700 1 -- start start 2024-01-11T08:28:12.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.341+0000 7f20749ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f207015ca10 0x7f207015a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:12.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.341+0000 7f20749ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20700b4bd0 con 0x7f2070068cd0 2024-01-11T08:28:12.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.341+0000 7f20749ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f207015ac70 con 0x7f207015cde0 2024-01-11T08:28:12.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.341+0000 7f20749ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20700b3fc0 con 0x7f207015ca10 2024-01-11T08:28:12.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.341+0000 7f206e59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f207015ca10 0x7f207015a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:12.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.341+0000 7f206e59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f207015ca10 0x7f207015a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56346/0 (socket says 172.21.15.27:56346) 2024-01-11T08:28:12.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.341+0000 7f206e59c700 1 -- 172.21.15.27:0/2010870313 learned_addr learned my addr 172.21.15.27:0/2010870313 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:12.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.341+0000 7f206ed9d700 1 --1- 172.21.15.27:0/2010870313 >> v1:172.21.15.27:6789/0 conn(0x7f2070068cd0 0x7f20700b5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34094/0 (socket says 172.21.15.27:34094) 2024-01-11T08:28:12.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.342+0000 7f206e59c700 1 -- 172.21.15.27:0/2010870313 >> v1:172.21.15.77:6789/0 conn(0x7f207015cde0 legacy=0x7f20700685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:12.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.342+0000 7f206e59c700 1 -- 172.21.15.27:0/2010870313 >> v1:172.21.15.27:6789/0 conn(0x7f2070068cd0 legacy=0x7f20700b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.342+0000 7f206e59c700 1 -- 172.21.15.27:0/2010870313 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20700b42a0 con 0x7f207015ca10 2024-01-11T08:28:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.342+0000 7f206d59a700 1 -- 172.21.15.27:0/2010870313 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157444490 0 0) 0x7f20700b4bd0 con 0x7f2070068cd0 2024-01-11T08:28:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.342+0000 7f206e59c700 1 --2- 172.21.15.27:0/2010870313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f207015ca10 0x7f207015a730 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f20600099b0 tx=0x7f2060009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77986b5390fc838e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.342+0000 7f206d59a700 1 -- 172.21.15.27:0/2010870313 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f206000b860 con 0x7f207015ca10 2024-01-11T08:28:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.342+0000 7f206d59a700 1 -- 172.21.15.27:0/2010870313 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f206000b9c0 con 0x7f207015ca10 2024-01-11T08:28:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.342+0000 7f206d59a700 1 -- 172.21.15.27:0/2010870313 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f206000bd60 con 0x7f207015ca10 2024-01-11T08:28:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.343+0000 7f20749ff700 1 -- 172.21.15.27:0/2010870313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f207015ca10 msgr2=0x7f207015a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.343+0000 7f20749ff700 1 --2- 172.21.15.27:0/2010870313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f207015ca10 0x7f207015a730 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f20600099b0 tx=0x7f2060009250 comp rx=0 tx=0).stop 2024-01-11T08:28:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.343+0000 7f20749ff700 1 -- 172.21.15.27:0/2010870313 shutdown_connections 2024-01-11T08:28:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.343+0000 7f20749ff700 1 --2- 172.21.15.27:0/2010870313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f207015ca10 0x7f207015a730 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.343+0000 7f20749ff700 1 -- 172.21.15.27:0/2010870313 >> 172.21.15.27:0/2010870313 conn(0x7f207006ed40 msgr2=0x7f207006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.343+0000 7f20749ff700 1 -- 172.21.15.27:0/2010870313 shutdown_connections 2024-01-11T08:28:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.343+0000 7f20749ff700 1 -- 172.21.15.27:0/2010870313 wait complete. 2024-01-11T08:28:12.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.344+0000 7f20749ff700 1 Processor -- start 2024-01-11T08:28:12.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.345+0000 7f20749ff700 1 -- start start 2024-01-11T08:28:12.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.345+0000 7f20749ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2070068cd0 0x7f207016b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:12.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.345+0000 7f20749ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f207015ca10 0x7f207016cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:12.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.345+0000 7f20749ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f207015cde0 0x7f207016d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:12.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.345+0000 7f20749ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2070152830 con 0x7f2070068cd0 2024-01-11T08:28:12.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.345+0000 7f20749ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207014c5e0 con 0x7f207015ca10 2024-01-11T08:28:12.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.345+0000 7f20749ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20700b12b0 con 0x7f207015cde0 2024-01-11T08:28:12.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.346+0000 7f206e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2070068cd0 0x7f207016b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:12.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.346+0000 7f206e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2070068cd0 0x7f207016b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56360/0 (socket says 172.21.15.27:56360) 2024-01-11T08:28:12.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.346+0000 7f206e59c700 1 -- 172.21.15.27:0/934442662 learned_addr learned my addr 172.21.15.27:0/934442662 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:12.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.346+0000 7f206ed9d700 1 --2- 172.21.15.27:0/934442662 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f207015cde0 0x7f207016d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:12.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.346+0000 7f206dd9b700 1 --2- 172.21.15.27:0/934442662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f207015ca10 0x7f207016cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:12.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.346+0000 7f206e59c700 1 -- 172.21.15.27:0/934442662 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f207015cde0 msgr2=0x7f207016d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:12.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.346+0000 7f206e59c700 1 --2- 172.21.15.27:0/934442662 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f207015cde0 0x7f207016d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:12.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.346+0000 7f206e59c700 1 -- 172.21.15.27:0/934442662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f207015ca10 msgr2=0x7f207016cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:12.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.346+0000 7f206e59c700 1 --2- 172.21.15.27:0/934442662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f207015ca10 0x7f207016cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:12.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.346+0000 7f206e59c700 1 -- 172.21.15.27:0/934442662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20600094f0 con 0x7f2070068cd0 2024-01-11T08:28:12.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.346+0000 7f206dd9b700 1 --2- 172.21.15.27:0/934442662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f207015ca10 0x7f207016cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:12.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.346+0000 7f206ed9d700 1 --2- 172.21.15.27:0/934442662 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f207015cde0 0x7f207016d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:12.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.346+0000 7f206e59c700 1 --2- 172.21.15.27:0/934442662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2070068cd0 0x7f207016b2d0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f20600094c0 tx=0x7f2060017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:12.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.347+0000 7f205e7fc700 1 -- 172.21.15.27:0/934442662 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f206001f070 con 0x7f2070068cd0 2024-01-11T08:28:12.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.347+0000 7f205e7fc700 1 -- 172.21.15.27:0/934442662 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2060011d10 con 0x7f2070068cd0 2024-01-11T08:28:12.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.347+0000 7f205e7fc700 1 -- 172.21.15.27:0/934442662 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2060025650 con 0x7f2070068cd0 2024-01-11T08:28:12.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.347+0000 7f20749ff700 1 -- 172.21.15.27:0/934442662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20700b09c0 con 0x7f2070068cd0 2024-01-11T08:28:12.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.347+0000 7f20749ff700 1 -- 172.21.15.27:0/934442662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20700b0f40 con 0x7f2070068cd0 2024-01-11T08:28:12.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.348+0000 7f20749ff700 1 -- 172.21.15.27:0/934442662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f207004c3c0 con 0x7f2070068cd0 2024-01-11T08:28:12.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.349+0000 7f205e7fc700 1 -- 172.21.15.27:0/934442662 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2060023070 con 0x7f2070068cd0 2024-01-11T08:28:12.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.350+0000 7f205e7fc700 1 --2- 172.21.15.27:0/934442662 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2050066f30 0x7f20500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:12.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.350+0000 7f205e7fc700 1 -- 172.21.15.27:0/934442662 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5569+0+0 (secure 0 0 0) 0x7f2060085bf0 con 0x7f2070068cd0 2024-01-11T08:28:12.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.350+0000 7f206dd9b700 1 --2- 172.21.15.27:0/934442662 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2050066f30 0x7f20500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:12.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.351+0000 7f206dd9b700 1 --2- 172.21.15.27:0/934442662 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2050066f30 0x7f20500693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2064007fd0 tx=0x7f2064009040 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:12.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.354+0000 7f205e7fc700 1 -- 172.21.15.27:0/934442662 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2060052e80 con 0x7f2070068cd0 2024-01-11T08:28:12.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.572+0000 7f20749ff700 1 -- 172.21.15.27:0/934442662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f20700b4580 con 0x7f2070068cd0 2024-01-11T08:28:12.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.924+0000 7f205e7fc700 1 -- 172.21.15.27:0/934442662 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v110) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2060015080 con 0x7f2070068cd0 2024-01-11T08:28:12.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.927+0000 7f20749ff700 1 -- 172.21.15.27:0/934442662 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2050066f30 msgr2=0x7f20500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:12.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.927+0000 7f20749ff700 1 --2- 172.21.15.27:0/934442662 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2050066f30 0x7f20500693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2064007fd0 tx=0x7f2064009040 comp rx=0 tx=0).stop 2024-01-11T08:28:12.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.927+0000 7f20749ff700 1 -- 172.21.15.27:0/934442662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2070068cd0 msgr2=0x7f207016b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:12.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.927+0000 7f20749ff700 1 --2- 172.21.15.27:0/934442662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2070068cd0 0x7f207016b2d0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f20600094c0 tx=0x7f2060017000 comp rx=0 tx=0).stop 2024-01-11T08:28:12.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.927+0000 7f20749ff700 1 -- 172.21.15.27:0/934442662 shutdown_connections 2024-01-11T08:28:12.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.927+0000 7f20749ff700 1 --2- 172.21.15.27:0/934442662 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2050066f30 0x7f20500693e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:12.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.927+0000 7f20749ff700 1 --2- 172.21.15.27:0/934442662 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f207015cde0 0x7f207016d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:12.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.927+0000 7f20749ff700 1 --2- 172.21.15.27:0/934442662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2070068cd0 0x7f207016b2d0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:12.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.927+0000 7f20749ff700 1 --2- 172.21.15.27:0/934442662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f207015ca10 0x7f207016cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:12.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.927+0000 7f20749ff700 1 -- 172.21.15.27:0/934442662 >> 172.21.15.27:0/934442662 conn(0x7f207006ed40 msgr2=0x7f20700d0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:12.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.928+0000 7f20749ff700 1 -- 172.21.15.27:0/934442662 shutdown_connections 2024-01-11T08:28:12.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:12.928+0000 7f20749ff700 1 -- 172.21.15.27:0/934442662 wait complete. 2024-01-11T08:28:12.928 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-11T08:28:12.946 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-11T08:28:13.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.147+0000 7f397e3ff700 1 Processor -- start 2024-01-11T08:28:13.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.147+0000 7f397e3ff700 1 -- start start 2024-01-11T08:28:13.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.148+0000 7f397e3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3978069d40 0x7f39780b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:13.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.148+0000 7f397e3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3978157390 con 0x7f39780c89f0 2024-01-11T08:28:13.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.148+0000 7f397e3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397806a200 con 0x7f39780c8620 2024-01-11T08:28:13.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.148+0000 7f397e3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3978156d40 con 0x7f3978069d40 2024-01-11T08:28:13.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.148+0000 7f39777fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3978069d40 0x7f39780b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:13.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.149+0000 7f397c99c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f39780c89f0 0x7f39780accc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34096/0 (socket says 172.21.15.27:34096) 2024-01-11T08:28:13.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.149+0000 7f39777fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3978069d40 0x7f39780b02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56376/0 (socket says 172.21.15.27:56376) 2024-01-11T08:28:13.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.149+0000 7f397c99c700 1 -- 172.21.15.27:0/3320333250 learned_addr learned my addr 172.21.15.27:0/3320333250 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:28:13.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.149+0000 7f39777fe700 1 -- 172.21.15.27:0/3320333250 >> v1:172.21.15.77:6789/0 conn(0x7f39780c8620 legacy=0x7f39780af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:13.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.149+0000 7f39777fe700 1 -- 172.21.15.27:0/3320333250 >> v1:172.21.15.27:6789/0 conn(0x7f39780c89f0 legacy=0x7f39780accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:13.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.149+0000 7f39777fe700 1 -- 172.21.15.27:0/3320333250 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3978157020 con 0x7f3978069d40 2024-01-11T08:28:13.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.149+0000 7f3976ffd700 1 -- 172.21.15.27:0/3320333250 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789545558 0 0) 0x7f3978157390 con 0x7f39780c89f0 2024-01-11T08:28:13.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.149+0000 7f39777fe700 1 --2- 172.21.15.27:0/3320333250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3978069d40 0x7f39780b02d0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f396c00bb90 tx=0x7f396c00b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b13f12db82e488f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:13.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.149+0000 7f3976ffd700 1 -- 172.21.15.27:0/3320333250 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c00dba0 con 0x7f3978069d40 2024-01-11T08:28:13.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.150+0000 7f3976ffd700 1 -- 172.21.15.27:0/3320333250 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396c00dd00 con 0x7f3978069d40 2024-01-11T08:28:13.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.150+0000 7f3976ffd700 1 -- 172.21.15.27:0/3320333250 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c0196b0 con 0x7f3978069d40 2024-01-11T08:28:13.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.150+0000 7f397e3ff700 1 -- 172.21.15.27:0/3320333250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3978069d40 msgr2=0x7f39780b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:13.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.150+0000 7f397e3ff700 1 --2- 172.21.15.27:0/3320333250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3978069d40 0x7f39780b02d0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f396c00bb90 tx=0x7f396c00b430 comp rx=0 tx=0).stop 2024-01-11T08:28:13.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.150+0000 7f397e3ff700 1 -- 172.21.15.27:0/3320333250 shutdown_connections 2024-01-11T08:28:13.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.150+0000 7f397e3ff700 1 --2- 172.21.15.27:0/3320333250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3978069d40 0x7f39780b02d0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:13.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.150+0000 7f397e3ff700 1 -- 172.21.15.27:0/3320333250 >> 172.21.15.27:0/3320333250 conn(0x7f39780680e0 msgr2=0x7f3978069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:13.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.150+0000 7f397e3ff700 1 -- 172.21.15.27:0/3320333250 shutdown_connections 2024-01-11T08:28:13.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.150+0000 7f397e3ff700 1 -- 172.21.15.27:0/3320333250 wait complete. 2024-01-11T08:28:13.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.152+0000 7f397e3ff700 1 Processor -- start 2024-01-11T08:28:13.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.152+0000 7f397e3ff700 1 -- start start 2024-01-11T08:28:13.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.153+0000 7f397e3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3978069d40 0x7f397816c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:13.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.153+0000 7f397e3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39780c8620 0x7f39780ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:13.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.153+0000 7f397e3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39780c89f0 0x7f397813f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:13.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.153+0000 7f397e3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397816c140 con 0x7f39780c89f0 2024-01-11T08:28:13.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.153+0000 7f397e3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397816b530 con 0x7f3978069d40 2024-01-11T08:28:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.153+0000 7f397e3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397816b810 con 0x7f39780c8620 2024-01-11T08:28:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.153+0000 7f397c99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39780c89f0 0x7f397813f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.153+0000 7f397c99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39780c89f0 0x7f397813f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56380/0 (socket says 172.21.15.27:56380) 2024-01-11T08:28:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.153+0000 7f397c99c700 1 -- 172.21.15.27:0/4019862958 learned_addr learned my addr 172.21.15.27:0/4019862958 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.153+0000 7f3977fff700 1 --2- 172.21.15.27:0/4019862958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3978069d40 0x7f397816c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.154+0000 7f39777fe700 1 --2- 172.21.15.27:0/4019862958 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39780c8620 0x7f39780ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.154+0000 7f397c99c700 1 -- 172.21.15.27:0/4019862958 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39780c8620 msgr2=0x7f39780ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.154+0000 7f397c99c700 1 --2- 172.21.15.27:0/4019862958 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39780c8620 0x7f39780ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.154+0000 7f397c99c700 1 -- 172.21.15.27:0/4019862958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3978069d40 msgr2=0x7f397816c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.154+0000 7f397c99c700 1 --2- 172.21.15.27:0/4019862958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3978069d40 0x7f397816c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.154+0000 7f397c99c700 1 -- 172.21.15.27:0/4019862958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396c00b6d0 con 0x7f39780c89f0 2024-01-11T08:28:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.154+0000 7f3977fff700 1 --2- 172.21.15.27:0/4019862958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3978069d40 0x7f397816c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:28:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.154+0000 7f39777fe700 1 --2- 172.21.15.27:0/4019862958 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39780c8620 0x7f39780ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:28:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.154+0000 7f397c99c700 1 --2- 172.21.15.27:0/4019862958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39780c89f0 0x7f397813f7f0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f397000ccc0 tx=0x7f39700019e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.154+0000 7f395ffff700 1 -- 172.21.15.27:0/4019862958 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397000ce70 con 0x7f39780c89f0 2024-01-11T08:28:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.154+0000 7f395ffff700 1 -- 172.21.15.27:0/4019862958 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397000ab70 con 0x7f39780c89f0 2024-01-11T08:28:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.154+0000 7f397e3ff700 1 -- 172.21.15.27:0/4019862958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397816baf0 con 0x7f39780c89f0 2024-01-11T08:28:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.154+0000 7f395ffff700 1 -- 172.21.15.27:0/4019862958 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39700070f0 con 0x7f39780c89f0 2024-01-11T08:28:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.154+0000 7f397e3ff700 1 -- 172.21.15.27:0/4019862958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39780ca880 con 0x7f39780c89f0 2024-01-11T08:28:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.155+0000 7f397e3ff700 1 -- 172.21.15.27:0/4019862958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f397816c2f0 con 0x7f39780c89f0 2024-01-11T08:28:13.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.156+0000 7f395ffff700 1 -- 172.21.15.27:0/4019862958 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f39700074a0 con 0x7f39780c89f0 2024-01-11T08:28:13.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.157+0000 7f395ffff700 1 --2- 172.21.15.27:0/4019862958 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3964066f30 0x7f39640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:13.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.157+0000 7f395ffff700 1 -- 172.21.15.27:0/4019862958 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4881+0+0 (secure 0 0 0) 0x7f3970084680 con 0x7f39780c89f0 2024-01-11T08:28:13.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.158+0000 7f3977fff700 1 --2- 172.21.15.27:0/4019862958 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3964066f30 0x7f39640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:13.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.158+0000 7f3977fff700 1 --2- 172.21.15.27:0/4019862958 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3964066f30 0x7f39640693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3968001840 tx=0x7f3968009040 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:13.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.161+0000 7f395ffff700 1 -- 172.21.15.27:0/4019862958 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f397816c2f0 con 0x7f39780c89f0 2024-01-11T08:28:13.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.445+0000 7f397e3ff700 1 -- 172.21.15.27:0/4019862958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f39780718d0 con 0x7f39780c89f0 2024-01-11T08:28:13.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.446+0000 7f395ffff700 1 -- 172.21.15.27:0/4019862958 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f3970051c10 con 0x7f39780c89f0 2024-01-11T08:28:13.445 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:28:13.445 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T08:28:13.445 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-11T08:28:13.445 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:13.445 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-11T08:28:13.445 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lQm3yLhAAv6erz5hMfat8EUzRA5luMA==", 2024-01-11T08:28:13.445 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:28:13.446 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:28:13.446 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:28:13.446 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:28:13.446 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:28:13.446 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.446 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:28:13.446 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:13.446 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-11T08:28:13.446 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9l+FG8ChAACHndtVLx7qS3sidPniSycA==", 2024-01-11T08:28:13.446 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:28:13.446 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:28:13.446 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:28:13.446 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:28:13.446 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:28:13.447 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.447 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:28:13.447 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:13.447 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-11T08:28:13.447 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l1NnEOhAALHXsc5es7hufCthGdRzkyw==", 2024-01-11T08:28:13.447 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:28:13.447 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:28:13.447 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:28:13.447 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:28:13.447 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:28:13.447 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.447 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:28:13.447 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:13.447 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-11T08:28:13.447 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lruoWFhAAm8AmrZtTJhYI1+19F79RPQ==", 2024-01-11T08:28:13.448 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:28:13.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:28:13.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:28:13.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:28:13.448 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:28:13.448 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:28:13.448 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:13.448 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-11T08:28:13.448 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA2op9ltZygORAAZr1X1nCN0ce8rIVaGDpFyQ==", 2024-01-11T08:28:13.448 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:28:13.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:28:13.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:28:13.449 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:28:13.449 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:28:13.449 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:13.449 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-11T08:28:13.449 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA6op9lqSRPAhAAmtyCsyK5ZVGX2wfhOCMuFQ==", 2024-01-11T08:28:13.449 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:28:13.449 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:28:13.449 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:28:13.449 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:28:13.449 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:28:13.449 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:13.449 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-11T08:28:13.449 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA9op9lZcs8BhAA8j+U+jPOoy+fliVWZAPFYg==", 2024-01-11T08:28:13.450 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:28:13.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:28:13.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:28:13.450 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:28:13.450 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.450 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:28:13.450 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:13.450 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-11T08:28:13.450 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBAop9lFuCUChAAP6si/2W1mxH4ecQJvuumkw==", 2024-01-11T08:28:13.450 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:28:13.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:28:13.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:28:13.450 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:28:13.450 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.450 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:28:13.451 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:13.451 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-11T08:28:13.451 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBGop9lOpwBIRAAcpcsjwmX2IldcKHJWAgHuA==", 2024-01-11T08:28:13.451 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:28:13.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:28:13.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:28:13.451 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:28:13.451 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.451 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:28:13.451 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:13.451 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-11T08:28:13.451 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBJop9l4q8hJRAAK8mAYVRQMoWS6wBOU69HmA==", 2024-01-11T08:28:13.451 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:28:13.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:28:13.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:28:13.452 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:28:13.452 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:28:13.452 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:13.452 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-11T08:28:13.452 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBMop9lqMF9KBAAFgFFI2GWt9AGif4AoScThw==", 2024-01-11T08:28:13.452 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:28:13.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:28:13.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:28:13.452 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:28:13.452 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:28:13.453 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:13.453 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-11T08:28:13.453 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBPop9lN7eDLBAAKswPZFfs0g2AaSe6V+pbbQ==", 2024-01-11T08:28:13.453 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:28:13.453 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:28:13.453 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:28:13.453 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:28:13.453 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.453 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:28:13.453 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:13.453 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-11T08:28:13.453 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lT5okIBAABy8PbJJbA062VNmb1Y8qhg==", 2024-01-11T08:28:13.453 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:28:13.453 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:28:13.453 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-01-11T08:28:13.454 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-11T08:28:13.454 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:28:13.454 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:13.454 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-11T08:28:13.454 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lDf37LhAAqV+EabKXJEW1sS5xtu1tXw==", 2024-01-11T08:28:13.454 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:28:13.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:28:13.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:28:13.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:28:13.454 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:28:13.454 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:28:13.454 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:13.454 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-11T08:28:13.455 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lWayxJxAAcalM8QSOGtbCXxjYZGaNQA==", 2024-01-11T08:28:13.455 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:28:13.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:28:13.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:28:13.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:28:13.455 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:28:13.455 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:28:13.455 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:13.455 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-11T08:28:13.455 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lHD4tNhAAdmnFBmMrK+QhQYdpbzW0mg==", 2024-01-11T08:28:13.455 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:28:13.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:28:13.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:28:13.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:28:13.456 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:28:13.456 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:28:13.456 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:13.456 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-11T08:28:13.456 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAwop9lGPtDMxAAJ8FjrSEdCtQL8AhtAyqnTw==", 2024-01-11T08:28:13.456 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:28:13.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:28:13.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:28:13.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:28:13.456 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:28:13.456 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:28:13.456 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:13.456 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-11T08:28:13.457 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lXBZLJxAAXa1S8Vkm3MiV4igtNbg/1Q==", 2024-01-11T08:28:13.457 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:28:13.457 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:28:13.457 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T08:28:13.457 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:28:13.457 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:28:13.457 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:13.457 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-11T08:28:13.457 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l3ZGoHxAANqzytTTvWYSomG+RFsFDyg==", 2024-01-11T08:28:13.457 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:28:13.457 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:28:13.457 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T08:28:13.457 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:28:13.457 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.458 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:13.458 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-11T08:28:13.458 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T08:28:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.448+0000 7f397e3ff700 1 -- 172.21.15.27:0/4019862958 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3964066f30 msgr2=0x7f39640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.448+0000 7f397e3ff700 1 --2- 172.21.15.27:0/4019862958 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3964066f30 0x7f39640693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3968001840 tx=0x7f3968009040 comp rx=0 tx=0).stop 2024-01-11T08:28:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.448+0000 7f397e3ff700 1 -- 172.21.15.27:0/4019862958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39780c89f0 msgr2=0x7f397813f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.448+0000 7f397e3ff700 1 --2- 172.21.15.27:0/4019862958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39780c89f0 0x7f397813f7f0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f397000ccc0 tx=0x7f39700019e0 comp rx=0 tx=0).stop 2024-01-11T08:28:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.448+0000 7f397e3ff700 1 -- 172.21.15.27:0/4019862958 shutdown_connections 2024-01-11T08:28:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.448+0000 7f397e3ff700 1 --2- 172.21.15.27:0/4019862958 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3964066f30 0x7f39640693e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.448+0000 7f397e3ff700 1 --2- 172.21.15.27:0/4019862958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39780c89f0 0x7f397813f7f0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.448+0000 7f397e3ff700 1 --2- 172.21.15.27:0/4019862958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3978069d40 0x7f397816c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.448+0000 7f397e3ff700 1 --2- 172.21.15.27:0/4019862958 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39780c8620 0x7f39780ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.448+0000 7f397e3ff700 1 -- 172.21.15.27:0/4019862958 >> 172.21.15.27:0/4019862958 conn(0x7f39780680e0 msgr2=0x7f3978159620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.449+0000 7f397e3ff700 1 -- 172.21.15.27:0/4019862958 shutdown_connections 2024-01-11T08:28:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.449+0000 7f397e3ff700 1 -- 172.21.15.27:0/4019862958 wait complete. 2024-01-11T08:28:13.465 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:28:13.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.666+0000 7f5e881ff700 1 Processor -- start 2024-01-11T08:28:13.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.666+0000 7f5e881ff700 1 -- start start 2024-01-11T08:28:13.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.667+0000 7f5e881ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5e800b03e0 0x7f5e800b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.667+0000 7f5e881ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5e800b07b0 0x7f5e800b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.667+0000 7f5e881ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e800cd730 con 0x7f5e80159200 2024-01-11T08:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.667+0000 7f5e881ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e80159d30 con 0x7f5e800b07b0 2024-01-11T08:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.667+0000 7f5e881ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e800ccb20 con 0x7f5e800b03e0 2024-01-11T08:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.667+0000 7f5e8579a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5e800b07b0 0x7f5e800b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.668+0000 7f5e8679c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5e80159200 0x7f5e80159620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34104/0 (socket says 172.21.15.27:34104) 2024-01-11T08:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.668+0000 7f5e8679c700 1 -- 172.21.15.27:0/693262700 learned_addr learned my addr 172.21.15.27:0/693262700 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.668+0000 7f5e85f9b700 1 --2- 172.21.15.27:0/693262700 >> v2:172.21.15.77:3300/0 conn(0x7f5e800b03e0 0x7f5e800b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.668+0000 7f5e8579a700 1 -- 172.21.15.27:0/693262700 >> v2:172.21.15.77:3300/0 conn(0x7f5e800b03e0 msgr2=0x7f5e800b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.668+0000 7f5e8579a700 1 --2- 172.21.15.27:0/693262700 >> v2:172.21.15.77:3300/0 conn(0x7f5e800b03e0 0x7f5e800b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.668+0000 7f5e8579a700 1 -- 172.21.15.27:0/693262700 >> v1:172.21.15.27:6789/0 conn(0x7f5e80159200 legacy=0x7f5e80159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.668+0000 7f5e8579a700 1 -- 172.21.15.27:0/693262700 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e800cce00 con 0x7f5e800b07b0 2024-01-11T08:28:13.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.668+0000 7f5e84f99700 1 -- 172.21.15.27:0/693262700 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577419993 0 0) 0x7f5e800cd730 con 0x7f5e80159200 2024-01-11T08:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.668+0000 7f5e8579a700 1 --2- 172.21.15.27:0/693262700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e800b07b0 0x7f5e800b29c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f5e80089470 tx=0x7f5e700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d30aafc9792accb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.669+0000 7f5e84f99700 1 -- 172.21.15.27:0/693262700 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e70011630 con 0x7f5e800b07b0 2024-01-11T08:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.669+0000 7f5e84f99700 1 -- 172.21.15.27:0/693262700 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e70011790 con 0x7f5e800b07b0 2024-01-11T08:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.669+0000 7f5e84f99700 1 -- 172.21.15.27:0/693262700 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e70011b30 con 0x7f5e800b07b0 2024-01-11T08:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.669+0000 7f5e881ff700 1 -- 172.21.15.27:0/693262700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e800b07b0 msgr2=0x7f5e800b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.669+0000 7f5e881ff700 1 --2- 172.21.15.27:0/693262700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e800b07b0 0x7f5e800b29c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f5e80089470 tx=0x7f5e700091a0 comp rx=0 tx=0).stop 2024-01-11T08:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.669+0000 7f5e881ff700 1 -- 172.21.15.27:0/693262700 shutdown_connections 2024-01-11T08:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.669+0000 7f5e881ff700 1 --2- 172.21.15.27:0/693262700 >> v2:172.21.15.77:3300/0 conn(0x7f5e800b03e0 0x7f5e800b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.669+0000 7f5e881ff700 1 --2- 172.21.15.27:0/693262700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e800b07b0 0x7f5e800b29c0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.669+0000 7f5e881ff700 1 -- 172.21.15.27:0/693262700 >> 172.21.15.27:0/693262700 conn(0x7f5e8006ed40 msgr2=0x7f5e8006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.669+0000 7f5e881ff700 1 -- 172.21.15.27:0/693262700 shutdown_connections 2024-01-11T08:28:13.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.669+0000 7f5e881ff700 1 -- 172.21.15.27:0/693262700 wait complete. 2024-01-11T08:28:13.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.671+0000 7f5e881ff700 1 Processor -- start 2024-01-11T08:28:13.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.671+0000 7f5e881ff700 1 -- start start 2024-01-11T08:28:13.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.672+0000 7f5e881ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e800b03e0 0x7f5e80146710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:13.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.672+0000 7f5e881ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e800b07b0 0x7f5e800c6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:13.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.672+0000 7f5e881ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e80159200 0x7f5e80169960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:13.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.672+0000 7f5e881ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e800c98c0 con 0x7f5e80159200 2024-01-11T08:28:13.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.672+0000 7f5e881ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e80169ea0 con 0x7f5e800b03e0 2024-01-11T08:28:13.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.672+0000 7f5e881ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8016a140 con 0x7f5e800b07b0 2024-01-11T08:28:13.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.672+0000 7f5e8679c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e80159200 0x7f5e80169960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.672+0000 7f5e8679c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e80159200 0x7f5e80169960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56408/0 (socket says 172.21.15.27:56408) 2024-01-11T08:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.672+0000 7f5e8679c700 1 -- 172.21.15.27:0/2621699915 learned_addr learned my addr 172.21.15.27:0/2621699915 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.672+0000 7f5e85f9b700 1 --2- 172.21.15.27:0/2621699915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e800b03e0 0x7f5e80146710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.672+0000 7f5e8579a700 1 --2- 172.21.15.27:0/2621699915 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e800b07b0 0x7f5e800c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.673+0000 7f5e8679c700 1 -- 172.21.15.27:0/2621699915 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e800b07b0 msgr2=0x7f5e800c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.673+0000 7f5e8679c700 1 --2- 172.21.15.27:0/2621699915 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e800b07b0 0x7f5e800c6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.673+0000 7f5e8679c700 1 -- 172.21.15.27:0/2621699915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e800b03e0 msgr2=0x7f5e80146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.673+0000 7f5e8679c700 1 --2- 172.21.15.27:0/2621699915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e800b03e0 0x7f5e80146710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.673+0000 7f5e8679c700 1 -- 172.21.15.27:0/2621699915 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e70009420 con 0x7f5e80159200 2024-01-11T08:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.673+0000 7f5e8579a700 1 --2- 172.21.15.27:0/2621699915 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e800b07b0 0x7f5e800c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.673+0000 7f5e8679c700 1 --2- 172.21.15.27:0/2621699915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e80159200 0x7f5e80169960 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f5e7800cbf0 tx=0x7f5e780019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.673+0000 7f5e75ffb700 1 -- 172.21.15.27:0/2621699915 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7800a9c0 con 0x7f5e80159200 2024-01-11T08:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.673+0000 7f5e75ffb700 1 -- 172.21.15.27:0/2621699915 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7800ab20 con 0x7f5e80159200 2024-01-11T08:28:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.673+0000 7f5e75ffb700 1 -- 172.21.15.27:0/2621699915 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e780070f0 con 0x7f5e80159200 2024-01-11T08:28:13.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.673+0000 7f5e881ff700 1 -- 172.21.15.27:0/2621699915 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e8016a420 con 0x7f5e80159200 2024-01-11T08:28:13.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.673+0000 7f5e881ff700 1 -- 172.21.15.27:0/2621699915 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e800c9500 con 0x7f5e80159200 2024-01-11T08:28:13.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.674+0000 7f5e881ff700 1 -- 172.21.15.27:0/2621699915 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e800c9a30 con 0x7f5e80159200 2024-01-11T08:28:13.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.675+0000 7f5e75ffb700 1 -- 172.21.15.27:0/2621699915 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5e78007520 con 0x7f5e80159200 2024-01-11T08:28:13.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.676+0000 7f5e75ffb700 1 --2- 172.21.15.27:0/2621699915 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5e64066f30 0x7f5e640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:13.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.676+0000 7f5e75ffb700 1 -- 172.21.15.27:0/2621699915 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4881+0+0 (secure 0 0 0) 0x7f5e780845e0 con 0x7f5e80159200 2024-01-11T08:28:13.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.677+0000 7f5e85f9b700 1 --2- 172.21.15.27:0/2621699915 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5e64066f30 0x7f5e640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:13.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.677+0000 7f5e85f9b700 1 --2- 172.21.15.27:0/2621699915 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5e64066f30 0x7f5e640693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c009d40 tx=0x7f5e7c008040 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:13.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.680+0000 7f5e75ffb700 1 -- 172.21.15.27:0/2621699915 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5e78051c00 con 0x7f5e80159200 2024-01-11T08:28:13.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.887+0000 7f5e881ff700 1 -- 172.21.15.27:0/2621699915 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e800718d0 con 0x7f5e80159200 2024-01-11T08:28:13.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.888+0000 7f5e75ffb700 1 -- 172.21.15.27:0/2621699915 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+12132 (secure 0 0 0) 0x7f5e78055220 con 0x7f5e80159200 2024-01-11T08:28:13.887 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:28:13.887 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":112,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:28:12.930246+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:28:13.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.890+0000 7f5e881ff700 1 -- 172.21.15.27:0/2621699915 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5e64066f30 msgr2=0x7f5e640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.890+0000 7f5e881ff700 1 --2- 172.21.15.27:0/2621699915 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5e64066f30 0x7f5e640693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c009d40 tx=0x7f5e7c008040 comp rx=0 tx=0).stop 2024-01-11T08:28:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.890+0000 7f5e881ff700 1 -- 172.21.15.27:0/2621699915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e80159200 msgr2=0x7f5e80169960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.890+0000 7f5e881ff700 1 --2- 172.21.15.27:0/2621699915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e80159200 0x7f5e80169960 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f5e7800cbf0 tx=0x7f5e780019b0 comp rx=0 tx=0).stop 2024-01-11T08:28:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.890+0000 7f5e881ff700 1 -- 172.21.15.27:0/2621699915 shutdown_connections 2024-01-11T08:28:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.890+0000 7f5e881ff700 1 --2- 172.21.15.27:0/2621699915 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5e64066f30 0x7f5e640693e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.890+0000 7f5e881ff700 1 --2- 172.21.15.27:0/2621699915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e80159200 0x7f5e80169960 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.890+0000 7f5e881ff700 1 --2- 172.21.15.27:0/2621699915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e800b03e0 0x7f5e80146710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.890+0000 7f5e881ff700 1 --2- 172.21.15.27:0/2621699915 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e800b07b0 0x7f5e800c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.890+0000 7f5e881ff700 1 -- 172.21.15.27:0/2621699915 >> 172.21.15.27:0/2621699915 conn(0x7f5e8006ed40 msgr2=0x7f5e8015c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.890+0000 7f5e881ff700 1 -- 172.21.15.27:0/2621699915 shutdown_connections 2024-01-11T08:28:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:13.890+0000 7f5e881ff700 1 -- 172.21.15.27:0/2621699915 wait complete. 2024-01-11T08:28:13.908 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-11T08:28:14.145 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:28:14.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.350+0000 7fcc2d5ff700 1 Processor -- start 2024-01-11T08:28:14.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.350+0000 7fcc2d5ff700 1 -- start start 2024-01-11T08:28:14.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.351+0000 7fcc2d5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc280cb4b0 0x7fcc280cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:14.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.351+0000 7fcc2d5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc280ccc70 con 0x7fcc28143190 2024-01-11T08:28:14.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.351+0000 7fcc2d5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2815d320 con 0x7fcc28142dc0 2024-01-11T08:28:14.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.351+0000 7fcc2d5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc2815df30 con 0x7fcc280cb4b0 2024-01-11T08:28:14.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.351+0000 7fcc267fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcc28143190 0x7fcc280cc020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51512/0 (socket says 172.21.15.27:51512) 2024-01-11T08:28:14.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.351+0000 7fcc267fc700 1 -- 172.21.15.27:0/3834467207 learned_addr learned my addr 172.21.15.27:0/3834467207 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:28:14.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.351+0000 7fcc277fe700 1 --2- 172.21.15.27:0/3834467207 >> v2:172.21.15.27:3300/0 conn(0x7fcc280cb4b0 0x7fcc280cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:14.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.352+0000 7fcc25ffb700 1 -- 172.21.15.27:0/3834467207 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629611696 0 0) 0x7fcc280ccc70 con 0x7fcc28143190 2024-01-11T08:28:14.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.352+0000 7fcc25ffb700 1 -- 172.21.15.27:0/3834467207 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc10003570 con 0x7fcc28143190 2024-01-11T08:28:14.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.352+0000 7fcc277fe700 1 -- 172.21.15.27:0/3834467207 >> v1:172.21.15.77:6789/0 conn(0x7fcc28142dc0 legacy=0x7fcc280cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:14.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.352+0000 7fcc25ffb700 1 -- 172.21.15.27:0/3834467207 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238715968 0 0) 0x7fcc2815d320 con 0x7fcc28142dc0 2024-01-11T08:28:14.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.352+0000 7fcc277fe700 1 -- 172.21.15.27:0/3834467207 >> v1:172.21.15.27:6789/0 conn(0x7fcc28143190 legacy=0x7fcc280cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:14.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.352+0000 7fcc277fe700 1 -- 172.21.15.27:0/3834467207 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2815d590 con 0x7fcc280cb4b0 2024-01-11T08:28:14.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.352+0000 7fcc277fe700 1 --2- 172.21.15.27:0/3834467207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc280cb4b0 0x7fcc280cc730 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fcc20009e60 tx=0x7fcc20009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db335d63a5a69c3e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:14.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.353+0000 7fcc25ffb700 1 -- 172.21.15.27:0/3834467207 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2000e070 con 0x7fcc280cb4b0 2024-01-11T08:28:14.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.353+0000 7fcc25ffb700 1 -- 172.21.15.27:0/3834467207 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2000bb60 con 0x7fcc280cb4b0 2024-01-11T08:28:14.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.353+0000 7fcc25ffb700 1 -- 172.21.15.27:0/3834467207 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc20012430 con 0x7fcc280cb4b0 2024-01-11T08:28:14.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.353+0000 7fcc2d5ff700 1 -- 172.21.15.27:0/3834467207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc280cb4b0 msgr2=0x7fcc280cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:14.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.353+0000 7fcc2d5ff700 1 --2- 172.21.15.27:0/3834467207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc280cb4b0 0x7fcc280cc730 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fcc20009e60 tx=0x7fcc20009780 comp rx=0 tx=0).stop 2024-01-11T08:28:14.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.353+0000 7fcc2d5ff700 1 -- 172.21.15.27:0/3834467207 shutdown_connections 2024-01-11T08:28:14.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.353+0000 7fcc2d5ff700 1 --2- 172.21.15.27:0/3834467207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc280cb4b0 0x7fcc280cc730 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:14.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.353+0000 7fcc2d5ff700 1 -- 172.21.15.27:0/3834467207 >> 172.21.15.27:0/3834467207 conn(0x7fcc28063ca0 msgr2=0x7fcc2806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:14.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.353+0000 7fcc2d5ff700 1 -- 172.21.15.27:0/3834467207 shutdown_connections 2024-01-11T08:28:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.353+0000 7fcc2d5ff700 1 -- 172.21.15.27:0/3834467207 wait complete. 2024-01-11T08:28:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.355+0000 7fcc2d5ff700 1 Processor -- start 2024-01-11T08:28:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.355+0000 7fcc2d5ff700 1 -- start start 2024-01-11T08:28:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.356+0000 7fcc2d5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc280cb4b0 0x7fcc2813c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.356+0000 7fcc2d5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc28142dc0 0x7fcc280cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.356+0000 7fcc2d5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc28143190 0x7fcc28138930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.356+0000 7fcc2d5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2815ac40 con 0x7fcc28142dc0 2024-01-11T08:28:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.356+0000 7fcc2d5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2813cb70 con 0x7fcc280cb4b0 2024-01-11T08:28:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.356+0000 7fcc2d5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2813cd80 con 0x7fcc28143190 2024-01-11T08:28:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.356+0000 7fcc267fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc28142dc0 0x7fcc280cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.356+0000 7fcc267fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc28142dc0 0x7fcc280cbfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33958/0 (socket says 172.21.15.27:33958) 2024-01-11T08:28:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.356+0000 7fcc267fc700 1 -- 172.21.15.27:0/4210156325 learned_addr learned my addr 172.21.15.27:0/4210156325 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.356+0000 7fcc26ffd700 1 --2- 172.21.15.27:0/4210156325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc280cb4b0 0x7fcc2813c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.356+0000 7fcc277fe700 1 --2- 172.21.15.27:0/4210156325 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc28143190 0x7fcc28138930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:14.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.357+0000 7fcc267fc700 1 -- 172.21.15.27:0/4210156325 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc28143190 msgr2=0x7fcc28138930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:14.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.357+0000 7fcc267fc700 1 --2- 172.21.15.27:0/4210156325 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc28143190 0x7fcc28138930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:14.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.357+0000 7fcc267fc700 1 -- 172.21.15.27:0/4210156325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc280cb4b0 msgr2=0x7fcc2813c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:14.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.357+0000 7fcc267fc700 1 --2- 172.21.15.27:0/4210156325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc280cb4b0 0x7fcc2813c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:14.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.357+0000 7fcc267fc700 1 -- 172.21.15.27:0/4210156325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc20009a20 con 0x7fcc28142dc0 2024-01-11T08:28:14.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.357+0000 7fcc267fc700 1 --2- 172.21.15.27:0/4210156325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc28142dc0 0x7fcc280cbfd0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0011e0 tx=0x7fcc1c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:14.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.357+0000 7fcc0effd700 1 -- 172.21.15.27:0/4210156325 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c005b30 con 0x7fcc28142dc0 2024-01-11T08:28:14.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.357+0000 7fcc0effd700 1 -- 172.21.15.27:0/4210156325 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1c005c90 con 0x7fcc28142dc0 2024-01-11T08:28:14.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.357+0000 7fcc2d5ff700 1 -- 172.21.15.27:0/4210156325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc2813d060 con 0x7fcc28142dc0 2024-01-11T08:28:14.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.357+0000 7fcc0effd700 1 -- 172.21.15.27:0/4210156325 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c007320 con 0x7fcc28142dc0 2024-01-11T08:28:14.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.357+0000 7fcc2d5ff700 1 -- 172.21.15.27:0/4210156325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc2813d620 con 0x7fcc28142dc0 2024-01-11T08:28:14.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.359+0000 7fcc2d5ff700 1 -- 172.21.15.27:0/4210156325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc2804c3c0 con 0x7fcc28142dc0 2024-01-11T08:28:14.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.359+0000 7fcc0effd700 1 -- 172.21.15.27:0/4210156325 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fcc1c007690 con 0x7fcc28142dc0 2024-01-11T08:28:14.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.361+0000 7fcc0effd700 1 --2- 172.21.15.27:0/4210156325 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc14066f30 0x7fcc140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:14.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.361+0000 7fcc0effd700 1 -- 172.21.15.27:0/4210156325 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4881+0+0 (secure 0 0 0) 0x7fcc1c083650 con 0x7fcc28142dc0 2024-01-11T08:28:14.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.361+0000 7fcc26ffd700 1 --2- 172.21.15.27:0/4210156325 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc14066f30 0x7fcc140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:14.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.362+0000 7fcc26ffd700 1 --2- 172.21.15.27:0/4210156325 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc14066f30 0x7fcc140693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcc18003a90 tx=0x7fcc18007ce0 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:14.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.365+0000 7fcc0effd700 1 -- 172.21.15.27:0/4210156325 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcc1c050b50 con 0x7fcc28142dc0 2024-01-11T08:28:14.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.570+0000 7fcc2d5ff700 1 -- 172.21.15.27:0/4210156325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcc2815d870 con 0x7fcc28142dc0 2024-01-11T08:28:14.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.570+0000 7fcc0effd700 1 -- 172.21.15.27:0/4210156325 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcc1c0170a0 con 0x7fcc28142dc0 2024-01-11T08:28:14.570 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:28:14.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.572+0000 7fcc2d5ff700 1 -- 172.21.15.27:0/4210156325 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc14066f30 msgr2=0x7fcc140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:14.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.573+0000 7fcc2d5ff700 1 --2- 172.21.15.27:0/4210156325 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc14066f30 0x7fcc140693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcc18003a90 tx=0x7fcc18007ce0 comp rx=0 tx=0).stop 2024-01-11T08:28:14.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.573+0000 7fcc2d5ff700 1 -- 172.21.15.27:0/4210156325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc28142dc0 msgr2=0x7fcc280cbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:14.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.573+0000 7fcc2d5ff700 1 --2- 172.21.15.27:0/4210156325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc28142dc0 0x7fcc280cbfd0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0011e0 tx=0x7fcc1c007df0 comp rx=0 tx=0).stop 2024-01-11T08:28:14.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.573+0000 7fcc2d5ff700 1 -- 172.21.15.27:0/4210156325 shutdown_connections 2024-01-11T08:28:14.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.573+0000 7fcc2d5ff700 1 --2- 172.21.15.27:0/4210156325 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fcc14066f30 0x7fcc140693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:14.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.573+0000 7fcc2d5ff700 1 --2- 172.21.15.27:0/4210156325 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc28143190 0x7fcc28138930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:14.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.573+0000 7fcc2d5ff700 1 --2- 172.21.15.27:0/4210156325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc280cb4b0 0x7fcc2813c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:14.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.573+0000 7fcc2d5ff700 1 --2- 172.21.15.27:0/4210156325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc28142dc0 0x7fcc280cbfd0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:14.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.573+0000 7fcc2d5ff700 1 -- 172.21.15.27:0/4210156325 >> 172.21.15.27:0/4210156325 conn(0x7fcc28063ca0 msgr2=0x7fcc280c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:14.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.573+0000 7fcc2d5ff700 1 -- 172.21.15.27:0/4210156325 shutdown_connections 2024-01-11T08:28:14.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.573+0000 7fcc2d5ff700 1 -- 172.21.15.27:0/4210156325 wait complete. 2024-01-11T08:28:14.590 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-11T08:28:14.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.791+0000 7f9842dff700 1 Processor -- start 2024-01-11T08:28:14.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.791+0000 7f9842dff700 1 -- start start 2024-01-11T08:28:14.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.792+0000 7f9842dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f983c0b03e0 0x7f983c0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:14.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.793+0000 7f9842dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f983c0b07b0 0x7f983c0b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:14.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.793+0000 7f9842dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f983c0cd730 con 0x7f983c159200 2024-01-11T08:28:14.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.793+0000 7f9842dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f983c159d30 con 0x7f983c0b03e0 2024-01-11T08:28:14.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.793+0000 7f9842dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f983c0ccb20 con 0x7f983c0b07b0 2024-01-11T08:28:14.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.793+0000 7f9840b9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f983c0b03e0 0x7f983c0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:14.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.793+0000 7f984139c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f983c159200 0x7f983c159620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51526/0 (socket says 172.21.15.27:51526) 2024-01-11T08:28:14.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.793+0000 7f984139c700 1 -- 172.21.15.27:0/2172560175 learned_addr learned my addr 172.21.15.27:0/2172560175 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:28:14.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.793+0000 7f983bfff700 1 --2- 172.21.15.27:0/2172560175 >> v2:172.21.15.77:3300/0 conn(0x7f983c0b07b0 0x7f983c0b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:14.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.793+0000 7f983b7fe700 1 -- 172.21.15.27:0/2172560175 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394052520 0 0) 0x7f983c0cd730 con 0x7f983c159200 2024-01-11T08:28:14.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.793+0000 7f9840b9b700 1 -- 172.21.15.27:0/2172560175 >> v2:172.21.15.77:3300/0 conn(0x7f983c0b07b0 msgr2=0x7f983c0b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:14.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.793+0000 7f9840b9b700 1 --2- 172.21.15.27:0/2172560175 >> v2:172.21.15.77:3300/0 conn(0x7f983c0b07b0 0x7f983c0b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:14.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.793+0000 7f9840b9b700 1 -- 172.21.15.27:0/2172560175 >> v1:172.21.15.27:6789/0 conn(0x7f983c159200 legacy=0x7f983c159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:14.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.793+0000 7f9840b9b700 1 -- 172.21.15.27:0/2172560175 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f983c0cce00 con 0x7f983c0b03e0 2024-01-11T08:28:14.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.794+0000 7f9840b9b700 1 --2- 172.21.15.27:0/2172560175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c0b03e0 0x7f983c0b2480 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f983c0cdfc0 tx=0x7f98300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55147812115d4912 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:14.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.794+0000 7f983b7fe700 1 -- 172.21.15.27:0/2172560175 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983000b6b0 con 0x7f983c0b03e0 2024-01-11T08:28:14.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.794+0000 7f983b7fe700 1 -- 172.21.15.27:0/2172560175 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983000b810 con 0x7f983c0b03e0 2024-01-11T08:28:14.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.794+0000 7f983b7fe700 1 -- 172.21.15.27:0/2172560175 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983000bbb0 con 0x7f983c0b03e0 2024-01-11T08:28:14.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.794+0000 7f9842dff700 1 -- 172.21.15.27:0/2172560175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c0b03e0 msgr2=0x7f983c0b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:14.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.794+0000 7f9842dff700 1 --2- 172.21.15.27:0/2172560175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c0b03e0 0x7f983c0b2480 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f983c0cdfc0 tx=0x7f98300091a0 comp rx=0 tx=0).stop 2024-01-11T08:28:14.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.795+0000 7f9842dff700 1 -- 172.21.15.27:0/2172560175 shutdown_connections 2024-01-11T08:28:14.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.795+0000 7f9842dff700 1 --2- 172.21.15.27:0/2172560175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c0b03e0 0x7f983c0b2480 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:14.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.795+0000 7f9842dff700 1 --2- 172.21.15.27:0/2172560175 >> v2:172.21.15.77:3300/0 conn(0x7f983c0b07b0 0x7f983c0b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:14.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.795+0000 7f9842dff700 1 -- 172.21.15.27:0/2172560175 >> 172.21.15.27:0/2172560175 conn(0x7f983c06ed40 msgr2=0x7f983c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:14.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.795+0000 7f9842dff700 1 -- 172.21.15.27:0/2172560175 shutdown_connections 2024-01-11T08:28:14.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.795+0000 7f9842dff700 1 -- 172.21.15.27:0/2172560175 wait complete. 2024-01-11T08:28:14.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.796+0000 7f9842dff700 1 Processor -- start 2024-01-11T08:28:14.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.796+0000 7f9842dff700 1 -- start start 2024-01-11T08:28:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.797+0000 7f9842dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f983c0b03e0 0x7f983c0c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.797+0000 7f9842dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f983c0b07b0 0x7f983c14c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.797+0000 7f9842dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c159200 0x7f983c0af180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.797+0000 7f9842dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983c0cab10 con 0x7f983c159200 2024-01-11T08:28:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.797+0000 7f9842dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983c0c9cb0 con 0x7f983c0b07b0 2024-01-11T08:28:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.797+0000 7f9842dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983c0c81f0 con 0x7f983c0b03e0 2024-01-11T08:28:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.797+0000 7f984139c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c159200 0x7f983c0af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.798+0000 7f984139c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c159200 0x7f983c0af180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33978/0 (socket says 172.21.15.27:33978) 2024-01-11T08:28:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.798+0000 7f984139c700 1 -- 172.21.15.27:0/3237894524 learned_addr learned my addr 172.21.15.27:0/3237894524 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.798+0000 7f9840b9b700 1 --2- 172.21.15.27:0/3237894524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f983c0b03e0 0x7f983c0c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.798+0000 7f983bfff700 1 --2- 172.21.15.27:0/3237894524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f983c0b07b0 0x7f983c14c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.798+0000 7f984139c700 1 -- 172.21.15.27:0/3237894524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f983c0b03e0 msgr2=0x7f983c0c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.798+0000 7f984139c700 1 --2- 172.21.15.27:0/3237894524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f983c0b03e0 0x7f983c0c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.798+0000 7f984139c700 1 -- 172.21.15.27:0/3237894524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f983c0b07b0 msgr2=0x7f983c14c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.798+0000 7f984139c700 1 --2- 172.21.15.27:0/3237894524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f983c0b07b0 0x7f983c14c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.798+0000 7f984139c700 1 -- 172.21.15.27:0/3237894524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9830009420 con 0x7f983c159200 2024-01-11T08:28:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.798+0000 7f9840b9b700 1 --2- 172.21.15.27:0/3237894524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f983c0b03e0 0x7f983c0c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.798+0000 7f983bfff700 1 --2- 172.21.15.27:0/3237894524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f983c0b07b0 0x7f983c14c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.798+0000 7f984139c700 1 --2- 172.21.15.27:0/3237894524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c159200 0x7f983c0af180 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f983400cbf0 tx=0x7f98340019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.798+0000 7f9838ff9700 1 -- 172.21.15.27:0/3237894524 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983400a9c0 con 0x7f983c159200 2024-01-11T08:28:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.799+0000 7f9838ff9700 1 -- 172.21.15.27:0/3237894524 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983400ab20 con 0x7f983c159200 2024-01-11T08:28:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.799+0000 7f9842dff700 1 -- 172.21.15.27:0/3237894524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f983c0c9f00 con 0x7f983c159200 2024-01-11T08:28:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.799+0000 7f9838ff9700 1 -- 172.21.15.27:0/3237894524 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98340070f0 con 0x7f983c159200 2024-01-11T08:28:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.799+0000 7f9842dff700 1 -- 172.21.15.27:0/3237894524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f983c0ca4c0 con 0x7f983c159200 2024-01-11T08:28:14.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.800+0000 7f9842dff700 1 -- 172.21.15.27:0/3237894524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f983c0c3ed0 con 0x7f983c159200 2024-01-11T08:28:14.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.801+0000 7f9838ff9700 1 -- 172.21.15.27:0/3237894524 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9834007520 con 0x7f983c159200 2024-01-11T08:28:14.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.802+0000 7f9838ff9700 1 --2- 172.21.15.27:0/3237894524 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9820066f30 0x7f98200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:14.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.802+0000 7f9838ff9700 1 -- 172.21.15.27:0/3237894524 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4881+0+0 (secure 0 0 0) 0x7f9834084600 con 0x7f983c159200 2024-01-11T08:28:14.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.802+0000 7f9840b9b700 1 --2- 172.21.15.27:0/3237894524 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9820066f30 0x7f98200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:14.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.803+0000 7f9840b9b700 1 --2- 172.21.15.27:0/3237894524 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9820066f30 0x7f98200693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f98300093f0 tx=0x7f9830017000 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:14.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:14.806+0000 7f9838ff9700 1 -- 172.21.15.27:0/3237894524 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f983c0c3ed0 con 0x7f983c159200 2024-01-11T08:28:15.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.013+0000 7f9842dff700 1 -- 172.21.15.27:0/3237894524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f983c0c3ed0 con 0x7f983c159200 2024-01-11T08:28:15.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.013+0000 7f9838ff9700 1 -- 172.21.15.27:0/3237894524 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v112) v1 ==== 84+0+65 (secure 0 0 0) 0x7f983c0c3ed0 con 0x7f983c159200 2024-01-11T08:28:15.013 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:28:15.013 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T08:28:15.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:15.013 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-11T08:28:15.013 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-11T08:28:15.013 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:15.013 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T08:28:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.015+0000 7f9842dff700 1 -- 172.21.15.27:0/3237894524 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9820066f30 msgr2=0x7f98200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.015+0000 7f9842dff700 1 --2- 172.21.15.27:0/3237894524 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9820066f30 0x7f98200693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f98300093f0 tx=0x7f9830017000 comp rx=0 tx=0).stop 2024-01-11T08:28:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.015+0000 7f9842dff700 1 -- 172.21.15.27:0/3237894524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c159200 msgr2=0x7f983c0af180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.015+0000 7f9842dff700 1 --2- 172.21.15.27:0/3237894524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c159200 0x7f983c0af180 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f983400cbf0 tx=0x7f98340019b0 comp rx=0 tx=0).stop 2024-01-11T08:28:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.016+0000 7f9842dff700 1 -- 172.21.15.27:0/3237894524 shutdown_connections 2024-01-11T08:28:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.016+0000 7f9842dff700 1 --2- 172.21.15.27:0/3237894524 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9820066f30 0x7f98200693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.016+0000 7f9842dff700 1 --2- 172.21.15.27:0/3237894524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f983c159200 0x7f983c0af180 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.016+0000 7f9842dff700 1 --2- 172.21.15.27:0/3237894524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f983c0b03e0 0x7f983c0c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.016+0000 7f9842dff700 1 --2- 172.21.15.27:0/3237894524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f983c0b07b0 0x7f983c14c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.016+0000 7f9842dff700 1 -- 172.21.15.27:0/3237894524 >> 172.21.15.27:0/3237894524 conn(0x7f983c06ed40 msgr2=0x7f983c158a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.016+0000 7f9842dff700 1 -- 172.21.15.27:0/3237894524 shutdown_connections 2024-01-11T08:28:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.016+0000 7f9842dff700 1 -- 172.21.15.27:0/3237894524 wait complete. 2024-01-11T08:28:15.033 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-11T08:28:15.034 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-11T08:28:15.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.239+0000 7f4509bff700 1 Processor -- start 2024-01-11T08:28:15.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.239+0000 7f4509bff700 1 -- start start 2024-01-11T08:28:15.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.240+0000 7f4509bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f450415a7a0 0x7f450415c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:15.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.240+0000 7f4509bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4504159e60 0x7f450415aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:15.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.240+0000 7f4509bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f450415b400 0x7f45040685e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:15.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.240+0000 7f4509bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f450415c4b0 con 0x7f450415b400 2024-01-11T08:28:15.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.241+0000 7f4509bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f450415b8a0 con 0x7f4504159e60 2024-01-11T08:28:15.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.241+0000 7f4509bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f450415bb80 con 0x7f450415a7a0 2024-01-11T08:28:15.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.241+0000 7f4503fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f450415b400 0x7f45040685e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:15.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.241+0000 7f4503fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f450415b400 0x7f45040685e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33992/0 (socket says 172.21.15.27:33992) 2024-01-11T08:28:15.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.241+0000 7f4503fff700 1 -- 172.21.15.27:0/1599538986 learned_addr learned my addr 172.21.15.27:0/1599538986 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:15.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.241+0000 7f45037fe700 1 --2- 172.21.15.27:0/1599538986 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f450415a7a0 0x7f450415c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:15.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.241+0000 7f4502ffd700 1 --2- 172.21.15.27:0/1599538986 >> v2:172.21.15.77:3300/0 conn(0x7f4504159e60 0x7f450415aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:15.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.241+0000 7f4503fff700 1 -- 172.21.15.27:0/1599538986 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f450415a7a0 msgr2=0x7f450415c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:15.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.241+0000 7f4503fff700 1 --2- 172.21.15.27:0/1599538986 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f450415a7a0 0x7f450415c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:15.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.241+0000 7f4503fff700 1 -- 172.21.15.27:0/1599538986 >> v2:172.21.15.77:3300/0 conn(0x7f4504159e60 msgr2=0x7f450415aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:15.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.241+0000 7f4503fff700 1 --2- 172.21.15.27:0/1599538986 >> v2:172.21.15.77:3300/0 conn(0x7f4504159e60 0x7f450415aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:15.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.241+0000 7f4503fff700 1 -- 172.21.15.27:0/1599538986 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450415be60 con 0x7f450415b400 2024-01-11T08:28:15.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.241+0000 7f4502ffd700 1 --2- 172.21.15.27:0/1599538986 >> v2:172.21.15.77:3300/0 conn(0x7f4504159e60 0x7f450415aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:15.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.242+0000 7f4503fff700 1 --2- 172.21.15.27:0/1599538986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f450415b400 0x7f45040685e0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f4504089450 tx=0x7f44fc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef62e9535000f453 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:15.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.242+0000 7f45027fc700 1 -- 172.21.15.27:0/1599538986 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc00e070 con 0x7f450415b400 2024-01-11T08:28:15.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.242+0000 7f45027fc700 1 -- 172.21.15.27:0/1599538986 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44fc00ba20 con 0x7f450415b400 2024-01-11T08:28:15.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.242+0000 7f45027fc700 1 -- 172.21.15.27:0/1599538986 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc0092e0 con 0x7f450415b400 2024-01-11T08:28:15.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.243+0000 7f4509bff700 1 -- 172.21.15.27:0/1599538986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f450415b400 msgr2=0x7f45040685e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:15.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.243+0000 7f4509bff700 1 --2- 172.21.15.27:0/1599538986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f450415b400 0x7f45040685e0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f4504089450 tx=0x7f44fc009840 comp rx=0 tx=0).stop 2024-01-11T08:28:15.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.243+0000 7f4509bff700 1 -- 172.21.15.27:0/1599538986 shutdown_connections 2024-01-11T08:28:15.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.243+0000 7f4509bff700 1 --2- 172.21.15.27:0/1599538986 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f450415a7a0 0x7f450415c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:15.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.243+0000 7f4509bff700 1 --2- 172.21.15.27:0/1599538986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f450415b400 0x7f45040685e0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:15.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.243+0000 7f4509bff700 1 --2- 172.21.15.27:0/1599538986 >> v2:172.21.15.77:3300/0 conn(0x7f4504159e60 0x7f450415aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:15.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.243+0000 7f4509bff700 1 -- 172.21.15.27:0/1599538986 >> 172.21.15.27:0/1599538986 conn(0x7f450406ed40 msgr2=0x7f450406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:15.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.243+0000 7f4509bff700 1 -- 172.21.15.27:0/1599538986 shutdown_connections 2024-01-11T08:28:15.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.243+0000 7f4509bff700 1 -- 172.21.15.27:0/1599538986 wait complete. 2024-01-11T08:28:15.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.244+0000 7f4509bff700 1 Processor -- start 2024-01-11T08:28:15.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.245+0000 7f4509bff700 1 -- start start 2024-01-11T08:28:15.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.245+0000 7f4509bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f450415a7a0 0x7f4504151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:15.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.246+0000 7f4509bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f450415b400 0x7f450416aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:15.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.246+0000 7f4509bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4504150df0 0x7f450414cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:15.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.246+0000 7f4509bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45040b0660 con 0x7f450415b400 2024-01-11T08:28:15.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.246+0000 7f4509bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45041522f0 con 0x7f450415a7a0 2024-01-11T08:28:15.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.246+0000 7f4509bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45041525d0 con 0x7f4504150df0 2024-01-11T08:28:15.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.246+0000 7f4502ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f450415b400 0x7f450416aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:15.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.246+0000 7f4502ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f450415b400 0x7f450416aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34002/0 (socket says 172.21.15.27:34002) 2024-01-11T08:28:15.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.246+0000 7f4502ffd700 1 -- 172.21.15.27:0/219086569 learned_addr learned my addr 172.21.15.27:0/219086569 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:15.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.246+0000 7f45037fe700 1 --2- 172.21.15.27:0/219086569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f450415a7a0 0x7f4504151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:15.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.246+0000 7f4503fff700 1 --2- 172.21.15.27:0/219086569 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4504150df0 0x7f450414cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:15.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.246+0000 7f4502ffd700 1 -- 172.21.15.27:0/219086569 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4504150df0 msgr2=0x7f450414cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:15.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.246+0000 7f4502ffd700 1 --2- 172.21.15.27:0/219086569 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4504150df0 0x7f450414cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:15.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.246+0000 7f4502ffd700 1 -- 172.21.15.27:0/219086569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f450415a7a0 msgr2=0x7f4504151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:15.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.246+0000 7f4502ffd700 1 --2- 172.21.15.27:0/219086569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f450415a7a0 0x7f4504151ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:15.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.246+0000 7f4502ffd700 1 -- 172.21.15.27:0/219086569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc009ae0 con 0x7f450415b400 2024-01-11T08:28:15.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.247+0000 7f4502ffd700 1 --2- 172.21.15.27:0/219086569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f450415b400 0x7f450416aa10 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f44f800d860 tx=0x7f44f8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:15.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.247+0000 7f44eb7fe700 1 -- 172.21.15.27:0/219086569 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f80125f0 con 0x7f450415b400 2024-01-11T08:28:15.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.247+0000 7f44eb7fe700 1 -- 172.21.15.27:0/219086569 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44f8012750 con 0x7f450415b400 2024-01-11T08:28:15.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.247+0000 7f4509bff700 1 -- 172.21.15.27:0/219086569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45040b0010 con 0x7f450415b400 2024-01-11T08:28:15.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.247+0000 7f44eb7fe700 1 -- 172.21.15.27:0/219086569 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f8012af0 con 0x7f450415b400 2024-01-11T08:28:15.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.247+0000 7f4509bff700 1 -- 172.21.15.27:0/219086569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45040b2df0 con 0x7f450415b400 2024-01-11T08:28:15.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.248+0000 7f4509bff700 1 -- 172.21.15.27:0/219086569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450404c3c0 con 0x7f450415b400 2024-01-11T08:28:15.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.249+0000 7f44eb7fe700 1 -- 172.21.15.27:0/219086569 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f44f8013480 con 0x7f450415b400 2024-01-11T08:28:15.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.250+0000 7f44eb7fe700 1 --2- 172.21.15.27:0/219086569 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f44f0066f30 0x7f44f00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:15.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.250+0000 7f44eb7fe700 1 -- 172.21.15.27:0/219086569 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 4881+0+0 (secure 0 0 0) 0x7f44f8084440 con 0x7f450415b400 2024-01-11T08:28:15.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.250+0000 7f45037fe700 1 --2- 172.21.15.27:0/219086569 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f44f0066f30 0x7f44f00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:15.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.251+0000 7f45037fe700 1 --2- 172.21.15.27:0/219086569 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f44f0066f30 0x7f44f00693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009db0 tx=0x7f44f40091a0 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:15.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.254+0000 7f44eb7fe700 1 -- 172.21.15.27:0/219086569 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f44f8051940 con 0x7f450415b400 2024-01-11T08:28:15.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.458+0000 7f4509bff700 1 -- 172.21.15.27:0/219086569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f45040b07a0 con 0x7f450415b400 2024-01-11T08:28:15.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.939+0000 7f44eb7fe700 1 -- 172.21.15.27:0/219086569 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v113) v1 ==== 138+0+0 (secure 0 0 0) 0x7f44f8054f60 con 0x7f450415b400 2024-01-11T08:28:15.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.943+0000 7f4509bff700 1 -- 172.21.15.27:0/219086569 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f44f0066f30 msgr2=0x7f44f00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:15.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.943+0000 7f4509bff700 1 --2- 172.21.15.27:0/219086569 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f44f0066f30 0x7f44f00693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009db0 tx=0x7f44f40091a0 comp rx=0 tx=0).stop 2024-01-11T08:28:15.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.943+0000 7f4509bff700 1 -- 172.21.15.27:0/219086569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f450415b400 msgr2=0x7f450416aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.943+0000 7f4509bff700 1 --2- 172.21.15.27:0/219086569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f450415b400 0x7f450416aa10 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f44f800d860 tx=0x7f44f8008040 comp rx=0 tx=0).stop 2024-01-11T08:28:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.944+0000 7f4509bff700 1 -- 172.21.15.27:0/219086569 shutdown_connections 2024-01-11T08:28:15.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.944+0000 7f4509bff700 1 --2- 172.21.15.27:0/219086569 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f44f0066f30 0x7f44f00693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:15.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.944+0000 7f4509bff700 1 --2- 172.21.15.27:0/219086569 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4504150df0 0x7f450414cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:15.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.944+0000 7f4509bff700 1 --2- 172.21.15.27:0/219086569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f450415a7a0 0x7f4504151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:15.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.944+0000 7f4509bff700 1 --2- 172.21.15.27:0/219086569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f450415b400 0x7f450416aa10 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:15.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.944+0000 7f4509bff700 1 -- 172.21.15.27:0/219086569 >> 172.21.15.27:0/219086569 conn(0x7f450406ed40 msgr2=0x7f450414d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:15.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.945+0000 7f4509bff700 1 -- 172.21.15.27:0/219086569 shutdown_connections 2024-01-11T08:28:15.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:15.945+0000 7f4509bff700 1 -- 172.21.15.27:0/219086569 wait complete. 2024-01-11T08:28:15.948 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-11T08:28:15.966 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-11T08:28:16.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.179+0000 7f10b05ff700 1 Processor -- start 2024-01-11T08:28:16.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.180+0000 7f10b05ff700 1 -- start start 2024-01-11T08:28:16.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.181+0000 7f10b05ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f10a80c8860 0x7f10a8158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:16.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.181+0000 7f10b05ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10a80b19d0 0x7f10a80ae230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:16.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.181+0000 7f10b05ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a80cabe0 con 0x7f10a8158600 2024-01-11T08:28:16.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.181+0000 7f10b05ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a80ae770 con 0x7f10a80c8860 2024-01-11T08:28:16.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.181+0000 7f10b05ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a80ae960 con 0x7f10a80b19d0 2024-01-11T08:28:16.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.181+0000 7f10ae39b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f10a80c8860 0x7f10a8158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:16.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.181+0000 7f10ae39b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f10a80c8860 0x7f10a8158ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34012/0 (socket says 172.21.15.27:34012) 2024-01-11T08:28:16.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.181+0000 7f10ae39b700 1 -- 172.21.15.27:0/1110679347 learned_addr learned my addr 172.21.15.27:0/1110679347 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:16.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.181+0000 7f10aeb9c700 1 --2- 172.21.15.27:0/1110679347 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10a80b19d0 0x7f10a80ae230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:16.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.182+0000 7f10ae39b700 1 -- 172.21.15.27:0/1110679347 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10a80b19d0 msgr2=0x7f10a80ae230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:16.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.182+0000 7f10ae39b700 1 --2- 172.21.15.27:0/1110679347 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10a80b19d0 0x7f10a80ae230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:16.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.182+0000 7f10ae39b700 1 -- 172.21.15.27:0/1110679347 >> v1:172.21.15.77:6789/0 conn(0x7f10a8158600 legacy=0x7f10a80b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:16.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.182+0000 7f10ae39b700 1 -- 172.21.15.27:0/1110679347 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a80ca590 con 0x7f10a80c8860 2024-01-11T08:28:16.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.182+0000 7f10ae39b700 1 --2- 172.21.15.27:0/1110679347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10a80c8860 0x7f10a8158ca0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f10a40099b0 tx=0x7f10a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e971b7ef2c0239ad server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:16.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.182+0000 7f10ad399700 1 -- 172.21.15.27:0/1110679347 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a4009ea0 con 0x7f10a80c8860 2024-01-11T08:28:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.182+0000 7f10ad399700 1 -- 172.21.15.27:0/1110679347 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a400b860 con 0x7f10a80c8860 2024-01-11T08:28:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.183+0000 7f10ad399700 1 -- 172.21.15.27:0/1110679347 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a400bc00 con 0x7f10a80c8860 2024-01-11T08:28:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.183+0000 7f10b05ff700 1 -- 172.21.15.27:0/1110679347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10a80c8860 msgr2=0x7f10a8158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.183+0000 7f10b05ff700 1 --2- 172.21.15.27:0/1110679347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10a80c8860 0x7f10a8158ca0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f10a40099b0 tx=0x7f10a4009250 comp rx=0 tx=0).stop 2024-01-11T08:28:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.183+0000 7f10b05ff700 1 -- 172.21.15.27:0/1110679347 shutdown_connections 2024-01-11T08:28:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.183+0000 7f10b05ff700 1 --2- 172.21.15.27:0/1110679347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10a80c8860 0x7f10a8158ca0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.183+0000 7f10b05ff700 1 --2- 172.21.15.27:0/1110679347 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10a80b19d0 0x7f10a80ae230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.183+0000 7f10b05ff700 1 -- 172.21.15.27:0/1110679347 >> 172.21.15.27:0/1110679347 conn(0x7f10a80680a0 msgr2=0x7f10a8069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.183+0000 7f10b05ff700 1 -- 172.21.15.27:0/1110679347 shutdown_connections 2024-01-11T08:28:16.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.183+0000 7f10b05ff700 1 -- 172.21.15.27:0/1110679347 wait complete. 2024-01-11T08:28:16.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.185+0000 7f10b05ff700 1 Processor -- start 2024-01-11T08:28:16.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.185+0000 7f10b05ff700 1 -- start start 2024-01-11T08:28:16.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.186+0000 7f10b05ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10a80b19d0 0x7f10a80ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:16.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.186+0000 7f10b05ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10a80c8860 0x7f10a815c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:16.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.186+0000 7f10b05ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10a8158600 0x7f10a8143b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:16.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.186+0000 7f10b05ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a80ad3b0 con 0x7f10a80b19d0 2024-01-11T08:28:16.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.186+0000 7f10b05ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a815cc90 con 0x7f10a80c8860 2024-01-11T08:28:16.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.186+0000 7f10b05ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a80ac7a0 con 0x7f10a8158600 2024-01-11T08:28:16.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.186+0000 7f10ae39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10a80b19d0 0x7f10a80ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:16.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.186+0000 7f10ae39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10a80b19d0 0x7f10a80ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34014/0 (socket says 172.21.15.27:34014) 2024-01-11T08:28:16.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.186+0000 7f10ae39b700 1 -- 172.21.15.27:0/2816794567 learned_addr learned my addr 172.21.15.27:0/2816794567 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:16.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.186+0000 7f10aeb9c700 1 --2- 172.21.15.27:0/2816794567 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10a8158600 0x7f10a8143b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:16.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.186+0000 7f10adb9a700 1 --2- 172.21.15.27:0/2816794567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10a80c8860 0x7f10a815c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:16.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.187+0000 7f10ae39b700 1 -- 172.21.15.27:0/2816794567 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10a8158600 msgr2=0x7f10a8143b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:16.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.187+0000 7f10ae39b700 1 --2- 172.21.15.27:0/2816794567 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10a8158600 0x7f10a8143b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:16.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.187+0000 7f10ae39b700 1 -- 172.21.15.27:0/2816794567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10a80c8860 msgr2=0x7f10a815c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:16.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.187+0000 7f10ae39b700 1 --2- 172.21.15.27:0/2816794567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10a80c8860 0x7f10a815c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:16.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.187+0000 7f10ae39b700 1 -- 172.21.15.27:0/2816794567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a40094f0 con 0x7f10a80b19d0 2024-01-11T08:28:16.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.187+0000 7f10aeb9c700 1 --2- 172.21.15.27:0/2816794567 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10a8158600 0x7f10a8143b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:16.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.187+0000 7f10ae39b700 1 --2- 172.21.15.27:0/2816794567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10a80b19d0 0x7f10a80ac0c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f10a4006fd0 tx=0x7f10a4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:16.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.187+0000 7f109e7fc700 1 -- 172.21.15.27:0/2816794567 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a400bbe0 con 0x7f10a80b19d0 2024-01-11T08:28:16.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.187+0000 7f109e7fc700 1 -- 172.21.15.27:0/2816794567 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a4011420 con 0x7f10a80b19d0 2024-01-11T08:28:16.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.187+0000 7f10b05ff700 1 -- 172.21.15.27:0/2816794567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a80aca80 con 0x7f10a80b19d0 2024-01-11T08:28:16.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.187+0000 7f109e7fc700 1 -- 172.21.15.27:0/2816794567 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a40117c0 con 0x7f10a80b19d0 2024-01-11T08:28:16.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.187+0000 7f10b05ff700 1 -- 172.21.15.27:0/2816794567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a80ad040 con 0x7f10a80b19d0 2024-01-11T08:28:16.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.188+0000 7f10b05ff700 1 -- 172.21.15.27:0/2816794567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a804c3c0 con 0x7f10a80b19d0 2024-01-11T08:28:16.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.190+0000 7f109e7fc700 1 -- 172.21.15.27:0/2816794567 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f10a4011bf0 con 0x7f10a80b19d0 2024-01-11T08:28:16.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.191+0000 7f109e7fc700 1 --2- 172.21.15.27:0/2816794567 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f108c066f30 0x7f108c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:16.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.191+0000 7f109e7fc700 1 -- 172.21.15.27:0/2816794567 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5251+0+0 (secure 0 0 0) 0x7f10a4085c90 con 0x7f10a80b19d0 2024-01-11T08:28:16.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.192+0000 7f10adb9a700 1 --2- 172.21.15.27:0/2816794567 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f108c066f30 0x7f108c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.193+0000 7f10adb9a700 1 --2- 172.21.15.27:0/2816794567 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f108c066f30 0x7f108c0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f10a8155510 tx=0x7f1098009040 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:16.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.195+0000 7f109e7fc700 1 -- 172.21.15.27:0/2816794567 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f10a4053220 con 0x7f10a80b19d0 2024-01-11T08:28:16.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.397+0000 7f10b05ff700 1 -- 172.21.15.27:0/2816794567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f10a80ca870 con 0x7f10a80b19d0 2024-01-11T08:28:16.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.948+0000 7f109e7fc700 1 -- 172.21.15.27:0/2816794567 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v114) v1 ==== 170+0+0 (secure 0 0 0) 0x7f10a4056840 con 0x7f10a80b19d0 2024-01-11T08:28:16.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.952+0000 7f10b05ff700 1 -- 172.21.15.27:0/2816794567 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f108c066f30 msgr2=0x7f108c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:16.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.952+0000 7f10b05ff700 1 --2- 172.21.15.27:0/2816794567 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f108c066f30 0x7f108c0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f10a8155510 tx=0x7f1098009040 comp rx=0 tx=0).stop 2024-01-11T08:28:16.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.952+0000 7f10b05ff700 1 -- 172.21.15.27:0/2816794567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10a80b19d0 msgr2=0x7f10a80ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:16.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.952+0000 7f10b05ff700 1 --2- 172.21.15.27:0/2816794567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10a80b19d0 0x7f10a80ac0c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f10a4006fd0 tx=0x7f10a4017000 comp rx=0 tx=0).stop 2024-01-11T08:28:16.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.953+0000 7f10b05ff700 1 -- 172.21.15.27:0/2816794567 shutdown_connections 2024-01-11T08:28:16.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.953+0000 7f10b05ff700 1 --2- 172.21.15.27:0/2816794567 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f108c066f30 0x7f108c0693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:16.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.953+0000 7f10b05ff700 1 --2- 172.21.15.27:0/2816794567 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10a8158600 0x7f10a8143b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:16.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.953+0000 7f10b05ff700 1 --2- 172.21.15.27:0/2816794567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10a80b19d0 0x7f10a80ac0c0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:16.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.953+0000 7f10b05ff700 1 --2- 172.21.15.27:0/2816794567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10a80c8860 0x7f10a815c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:16.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.953+0000 7f10b05ff700 1 -- 172.21.15.27:0/2816794567 >> 172.21.15.27:0/2816794567 conn(0x7f10a80680a0 msgr2=0x7f10a81578b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:16.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.954+0000 7f10b05ff700 1 -- 172.21.15.27:0/2816794567 shutdown_connections 2024-01-11T08:28:16.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:16.954+0000 7f10b05ff700 1 -- 172.21.15.27:0/2816794567 wait complete. 2024-01-11T08:28:16.967 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-11T08:28:16.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-11T08:28:17.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.191+0000 7f077a3ff700 1 Processor -- start 2024-01-11T08:28:17.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.191+0000 7f077a3ff700 1 -- start start 2024-01-11T08:28:17.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.192+0000 7f077a3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f077415a1b0 0x7f077415a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:17.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.192+0000 7f077a3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f07740ccd70 0x7f07740cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:17.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.192+0000 7f077a3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07740685c0 0x7f0774068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:17.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.192+0000 7f077a3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07740cf850 con 0x7f077415a1b0 2024-01-11T08:28:17.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.192+0000 7f077a3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07740cec40 con 0x7f07740ccd70 2024-01-11T08:28:17.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.192+0000 7f077a3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07740cef20 con 0x7f07740685c0 2024-01-11T08:28:17.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.192+0000 7f0773fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f077415a1b0 0x7f077415a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:17.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.192+0000 7f07737fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f07740ccd70 0x7f07740cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:17.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.192+0000 7f0773fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f077415a1b0 0x7f077415a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34018/0 (socket says 172.21.15.27:34018) 2024-01-11T08:28:17.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.192+0000 7f077899c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07740685c0 0x7f0774068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:17.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.192+0000 7f0773fff700 1 -- 172.21.15.27:0/1971138158 learned_addr learned my addr 172.21.15.27:0/1971138158 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:17.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.193+0000 7f0773fff700 1 -- 172.21.15.27:0/1971138158 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07740685c0 msgr2=0x7f0774068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.193+0000 7f0773fff700 1 --2- 172.21.15.27:0/1971138158 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07740685c0 0x7f0774068a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.193+0000 7f0773fff700 1 -- 172.21.15.27:0/1971138158 >> v2:172.21.15.77:3300/0 conn(0x7f07740ccd70 msgr2=0x7f07740cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.193+0000 7f0773fff700 1 --2- 172.21.15.27:0/1971138158 >> v2:172.21.15.77:3300/0 conn(0x7f07740ccd70 0x7f07740cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.193+0000 7f0773fff700 1 -- 172.21.15.27:0/1971138158 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07740cf200 con 0x7f077415a1b0 2024-01-11T08:28:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.193+0000 7f0773fff700 1 --2- 172.21.15.27:0/1971138158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f077415a1b0 0x7f077415a580 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f07740696f0 tx=0x7f0764009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe3a74b0e6a02513 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.193+0000 7f0772ffd700 1 -- 172.21.15.27:0/1971138158 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076400b640 con 0x7f077415a1b0 2024-01-11T08:28:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.194+0000 7f0772ffd700 1 -- 172.21.15.27:0/1971138158 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076400b7a0 con 0x7f077415a1b0 2024-01-11T08:28:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.194+0000 7f0772ffd700 1 -- 172.21.15.27:0/1971138158 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076400bb60 con 0x7f077415a1b0 2024-01-11T08:28:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.194+0000 7f077a3ff700 1 -- 172.21.15.27:0/1971138158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f077415a1b0 msgr2=0x7f077415a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.194+0000 7f077a3ff700 1 --2- 172.21.15.27:0/1971138158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f077415a1b0 0x7f077415a580 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f07740696f0 tx=0x7f0764009250 comp rx=0 tx=0).stop 2024-01-11T08:28:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.194+0000 7f077a3ff700 1 -- 172.21.15.27:0/1971138158 shutdown_connections 2024-01-11T08:28:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.194+0000 7f077a3ff700 1 --2- 172.21.15.27:0/1971138158 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07740685c0 0x7f0774068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.194+0000 7f077a3ff700 1 --2- 172.21.15.27:0/1971138158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f077415a1b0 0x7f077415a580 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.194+0000 7f077a3ff700 1 --2- 172.21.15.27:0/1971138158 >> v2:172.21.15.77:3300/0 conn(0x7f07740ccd70 0x7f07740cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.194+0000 7f077a3ff700 1 -- 172.21.15.27:0/1971138158 >> 172.21.15.27:0/1971138158 conn(0x7f077406ed40 msgr2=0x7f077406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.194+0000 7f077a3ff700 1 -- 172.21.15.27:0/1971138158 shutdown_connections 2024-01-11T08:28:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.194+0000 7f077a3ff700 1 -- 172.21.15.27:0/1971138158 wait complete. 2024-01-11T08:28:17.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.196+0000 7f077a3ff700 1 Processor -- start 2024-01-11T08:28:17.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.196+0000 7f077a3ff700 1 -- start start 2024-01-11T08:28:17.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.197+0000 7f077a3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07740685c0 0x7f0774169b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:17.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.197+0000 7f077a3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07740ccd70 0x7f07740c9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:17.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.197+0000 7f077a3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f077415a1b0 0x7f0774068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:17.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.197+0000 7f077a3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07740c94c0 con 0x7f07740685c0 2024-01-11T08:28:17.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.197+0000 7f077a3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077416a4e0 con 0x7f077415a1b0 2024-01-11T08:28:17.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.197+0000 7f077a3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f077416a7c0 con 0x7f07740ccd70 2024-01-11T08:28:17.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.197+0000 7f0773fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07740685c0 0x7f0774169b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:17.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.197+0000 7f0773fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07740685c0 0x7f0774169b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34032/0 (socket says 172.21.15.27:34032) 2024-01-11T08:28:17.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.197+0000 7f0773fff700 1 -- 172.21.15.27:0/1502590761 learned_addr learned my addr 172.21.15.27:0/1502590761 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:17.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.197+0000 7f07737fe700 1 --2- 172.21.15.27:0/1502590761 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07740ccd70 0x7f07740c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:17.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.197+0000 7f077899c700 1 --2- 172.21.15.27:0/1502590761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f077415a1b0 0x7f0774068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:17.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.197+0000 7f0773fff700 1 -- 172.21.15.27:0/1502590761 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07740ccd70 msgr2=0x7f07740c9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:17.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.197+0000 7f0773fff700 1 --2- 172.21.15.27:0/1502590761 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07740ccd70 0x7f07740c9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:17.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.197+0000 7f0773fff700 1 -- 172.21.15.27:0/1502590761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f077415a1b0 msgr2=0x7f0774068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:17.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.197+0000 7f0773fff700 1 --2- 172.21.15.27:0/1502590761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f077415a1b0 0x7f0774068a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:17.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.197+0000 7f0773fff700 1 -- 172.21.15.27:0/1502590761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07640094f0 con 0x7f07740685c0 2024-01-11T08:28:17.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.197+0000 7f077899c700 1 --2- 172.21.15.27:0/1502590761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f077415a1b0 0x7f0774068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:17.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.198+0000 7f0773fff700 1 --2- 172.21.15.27:0/1502590761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07740685c0 0x7f0774169b20 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f0774069770 tx=0x7f0764017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:17.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.198+0000 7f075bfff700 1 -- 172.21.15.27:0/1502590761 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764000d10 con 0x7f07740685c0 2024-01-11T08:28:17.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.198+0000 7f075bfff700 1 -- 172.21.15.27:0/1502590761 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076400bb20 con 0x7f07740685c0 2024-01-11T08:28:17.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.198+0000 7f077a3ff700 1 -- 172.21.15.27:0/1502590761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f077416aaa0 con 0x7f07740685c0 2024-01-11T08:28:17.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.198+0000 7f075bfff700 1 -- 172.21.15.27:0/1502590761 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764011670 con 0x7f07740685c0 2024-01-11T08:28:17.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.198+0000 7f077a3ff700 1 -- 172.21.15.27:0/1502590761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f077416a060 con 0x7f07740685c0 2024-01-11T08:28:17.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.199+0000 7f077a3ff700 1 -- 172.21.15.27:0/1502590761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f077404c3c0 con 0x7f07740685c0 2024-01-11T08:28:17.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.201+0000 7f075bfff700 1 -- 172.21.15.27:0/1502590761 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f07640119e0 con 0x7f07740685c0 2024-01-11T08:28:17.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.202+0000 7f075bfff700 1 --2- 172.21.15.27:0/1502590761 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0760066f30 0x7f07600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:17.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.202+0000 7f075bfff700 1 -- 172.21.15.27:0/1502590761 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5633+0+0 (secure 0 0 0) 0x7f0764086550 con 0x7f07740685c0 2024-01-11T08:28:17.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.202+0000 7f07737fe700 1 --2- 172.21.15.27:0/1502590761 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0760066f30 0x7f07600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:17.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.203+0000 7f07737fe700 1 --2- 172.21.15.27:0/1502590761 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0760066f30 0x7f07600693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0774167750 tx=0x7f07680091a0 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:17.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.205+0000 7f075bfff700 1 -- 172.21.15.27:0/1502590761 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0764053880 con 0x7f07740685c0 2024-01-11T08:28:17.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.452+0000 7f077a3ff700 1 -- 172.21.15.27:0/1502590761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f077415cd10 con 0x7f07740685c0 2024-01-11T08:28:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.952+0000 7f075bfff700 1 -- 172.21.15.27:0/1502590761 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v55) v1 ==== 174+0+0 (secure 0 0 0) 0x7f0764056ea0 con 0x7f07740685c0 2024-01-11T08:28:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.959+0000 7f077a3ff700 1 -- 172.21.15.27:0/1502590761 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0760066f30 msgr2=0x7f07600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.959+0000 7f077a3ff700 1 --2- 172.21.15.27:0/1502590761 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0760066f30 0x7f07600693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0774167750 tx=0x7f07680091a0 comp rx=0 tx=0).stop 2024-01-11T08:28:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.959+0000 7f077a3ff700 1 -- 172.21.15.27:0/1502590761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07740685c0 msgr2=0x7f0774169b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.959+0000 7f077a3ff700 1 --2- 172.21.15.27:0/1502590761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07740685c0 0x7f0774169b20 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f0774069770 tx=0x7f0764017000 comp rx=0 tx=0).stop 2024-01-11T08:28:17.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.960+0000 7f077a3ff700 1 -- 172.21.15.27:0/1502590761 shutdown_connections 2024-01-11T08:28:17.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.960+0000 7f077a3ff700 1 --2- 172.21.15.27:0/1502590761 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0760066f30 0x7f07600693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:17.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.960+0000 7f077a3ff700 1 --2- 172.21.15.27:0/1502590761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f077415a1b0 0x7f0774068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:17.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.960+0000 7f077a3ff700 1 --2- 172.21.15.27:0/1502590761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07740685c0 0x7f0774169b20 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:17.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.960+0000 7f077a3ff700 1 --2- 172.21.15.27:0/1502590761 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07740ccd70 0x7f07740c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:17.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.960+0000 7f077a3ff700 1 -- 172.21.15.27:0/1502590761 >> 172.21.15.27:0/1502590761 conn(0x7f077406ed40 msgr2=0x7f077415d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:17.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.963+0000 7f077a3ff700 1 -- 172.21.15.27:0/1502590761 shutdown_connections 2024-01-11T08:28:17.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:17.963+0000 7f077a3ff700 1 -- 172.21.15.27:0/1502590761 wait complete. 2024-01-11T08:28:17.970 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 14 and data pool 15 2024-01-11T08:28:18.003 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:28:18.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.204+0000 7fdddd3ff700 1 Processor -- start 2024-01-11T08:28:18.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.205+0000 7fdddd3ff700 1 -- start start 2024-01-11T08:28:18.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.206+0000 7fdddd3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fddd80ca7b0 0x7fddd80cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:18.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.206+0000 7fdddd3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd815d640 con 0x7fddd80ca3e0 2024-01-11T08:28:18.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.206+0000 7fdddd3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd815a5d0 con 0x7fddd8159e80 2024-01-11T08:28:18.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.206+0000 7fdddd3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd815ae70 con 0x7fddd80ca7b0 2024-01-11T08:28:18.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.206+0000 7fddd6ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fddd80ca3e0 0x7fddd80c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51540/0 (socket says 172.21.15.27:51540) 2024-01-11T08:28:18.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.206+0000 7fddd6ffd700 1 -- 172.21.15.27:0/493904604 learned_addr learned my addr 172.21.15.27:0/493904604 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:28:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.207+0000 7fddd67fc700 1 --2- 172.21.15.27:0/493904604 >> v2:172.21.15.77:3300/0 conn(0x7fddd80ca7b0 0x7fddd80cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.207+0000 7fddd5ffb700 1 -- 172.21.15.27:0/493904604 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787767654 0 0) 0x7fddd815d640 con 0x7fddd80ca3e0 2024-01-11T08:28:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.207+0000 7fddd5ffb700 1 -- 172.21.15.27:0/493904604 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc0003570 con 0x7fddd80ca3e0 2024-01-11T08:28:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.207+0000 7fddd5ffb700 1 -- 172.21.15.27:0/493904604 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1773428541 0 0) 0x7fddc0003570 con 0x7fddd80ca3e0 2024-01-11T08:28:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.207+0000 7fddd5ffb700 1 -- 172.21.15.27:0/493904604 >> v2:172.21.15.77:3300/0 conn(0x7fddd80ca7b0 msgr2=0x7fddd80cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:18.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.207+0000 7fddd5ffb700 1 --2- 172.21.15.27:0/493904604 >> v2:172.21.15.77:3300/0 conn(0x7fddd80ca7b0 0x7fddd80cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.207+0000 7fddd5ffb700 1 -- 172.21.15.27:0/493904604 >> v1:172.21.15.77:6789/0 conn(0x7fddd8159e80 legacy=0x7fddd815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:18.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.207+0000 7fddd5ffb700 1 -- 172.21.15.27:0/493904604 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd815a8b0 con 0x7fddd80ca3e0 2024-01-11T08:28:18.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.207+0000 7fddd5ffb700 1 -- 172.21.15.27:0/493904604 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fddc8001470 con 0x7fddd80ca3e0 2024-01-11T08:28:18.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.207+0000 7fddd5ffb700 1 -- 172.21.15.27:0/493904604 >> v1:172.21.15.27:6789/0 conn(0x7fddd80ca3e0 legacy=0x7fddd80c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:18.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.207+0000 7fddd5ffb700 1 --2- 172.21.15.27:0/493904604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddc0003a90 0x7fddc0003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:18.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.207+0000 7fddd5ffb700 1 --2- 172.21.15.27:0/493904604 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddc0004810 0x7fddc0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:18.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.207+0000 7fddd5ffb700 1 --2- 172.21.15.27:0/493904604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddc00051c0 0x7fddc000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:18.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.207+0000 7fddd5ffb700 1 -- 172.21.15.27:0/493904604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd815ae70 con 0x7fddc0003a90 2024-01-11T08:28:18.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.207+0000 7fddd5ffb700 1 -- 172.21.15.27:0/493904604 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc000c160 con 0x7fddc00051c0 2024-01-11T08:28:18.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.207+0000 7fddd5ffb700 1 -- 172.21.15.27:0/493904604 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc000c320 con 0x7fddc0004810 2024-01-11T08:28:18.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.207+0000 7fddd6ffd700 1 --2- 172.21.15.27:0/493904604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddc0003a90 0x7fddc0003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:18.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.208+0000 7fddd77fe700 1 --2- 172.21.15.27:0/493904604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddc00051c0 0x7fddc000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:18.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.208+0000 7fddd67fc700 1 --2- 172.21.15.27:0/493904604 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddc0004810 0x7fddc0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.208+0000 7fddd6ffd700 1 -- 172.21.15.27:0/493904604 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddc0004810 msgr2=0x7fddc0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.208+0000 7fddd6ffd700 1 --2- 172.21.15.27:0/493904604 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddc0004810 0x7fddc0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.208+0000 7fddd6ffd700 1 -- 172.21.15.27:0/493904604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddc00051c0 msgr2=0x7fddc000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.208+0000 7fddd6ffd700 1 --2- 172.21.15.27:0/493904604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddc00051c0 0x7fddc000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.208+0000 7fddd6ffd700 1 -- 172.21.15.27:0/493904604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc0003570 con 0x7fddc0003a90 2024-01-11T08:28:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.208+0000 7fddd77fe700 1 -- 172.21.15.27:0/493904604 reap_dead start 2024-01-11T08:28:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.208+0000 7fddd6ffd700 1 --2- 172.21.15.27:0/493904604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddc0003a90 0x7fddc0003e60 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fddc800c2d0 tx=0x7fddc8001da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de3f4abda18e4962 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.208+0000 7fddd5ffb700 1 -- 172.21.15.27:0/493904604 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc800cba0 con 0x7fddc0003a90 2024-01-11T08:28:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.208+0000 7fddd5ffb700 1 -- 172.21.15.27:0/493904604 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc800cd20 con 0x7fddc0003a90 2024-01-11T08:28:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.208+0000 7fddd5ffb700 1 -- 172.21.15.27:0/493904604 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc8005b70 con 0x7fddc0003a90 2024-01-11T08:28:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.209+0000 7fdddd3ff700 1 -- 172.21.15.27:0/493904604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddc0003a90 msgr2=0x7fddc0003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.209+0000 7fdddd3ff700 1 --2- 172.21.15.27:0/493904604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddc0003a90 0x7fddc0003e60 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fddc800c2d0 tx=0x7fddc8001da0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.209+0000 7fdddd3ff700 1 -- 172.21.15.27:0/493904604 shutdown_connections 2024-01-11T08:28:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.209+0000 7fdddd3ff700 1 --2- 172.21.15.27:0/493904604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddc0003a90 0x7fddc0003e60 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.209+0000 7fdddd3ff700 1 -- 172.21.15.27:0/493904604 >> 172.21.15.27:0/493904604 conn(0x7fddd80ce3e0 msgr2=0x7fddd80ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.209+0000 7fdddd3ff700 1 -- 172.21.15.27:0/493904604 shutdown_connections 2024-01-11T08:28:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.209+0000 7fdddd3ff700 1 -- 172.21.15.27:0/493904604 wait complete. 2024-01-11T08:28:18.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.211+0000 7fdddd3ff700 1 Processor -- start 2024-01-11T08:28:18.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.211+0000 7fdddd3ff700 1 -- start start 2024-01-11T08:28:18.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.212+0000 7fdddd3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddc0003a90 0x7fddd81511d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:18.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.212+0000 7fdddd3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddd815f890 0x7fddd815fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:18.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.212+0000 7fdddd3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd80b1b40 0x7fddd80b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:18.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.212+0000 7fdddd3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd8160240 con 0x7fddd80b1b40 2024-01-11T08:28:18.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.212+0000 7fdddd3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd815d9d0 con 0x7fddd815f890 2024-01-11T08:28:18.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.212+0000 7fdddd3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd815df90 con 0x7fddc0003a90 2024-01-11T08:28:18.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.212+0000 7fddd77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd80b1b40 0x7fddd80b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:18.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.212+0000 7fddd77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd80b1b40 0x7fddd80b1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34038/0 (socket says 172.21.15.27:34038) 2024-01-11T08:28:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.212+0000 7fddd77fe700 1 -- 172.21.15.27:0/4139961185 learned_addr learned my addr 172.21.15.27:0/4139961185 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.212+0000 7fddd6ffd700 1 --2- 172.21.15.27:0/4139961185 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddc0003a90 0x7fddd81511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.212+0000 7fddd67fc700 1 --2- 172.21.15.27:0/4139961185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddd815f890 0x7fddd815fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.212+0000 7fddd77fe700 1 -- 172.21.15.27:0/4139961185 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddc0003a90 msgr2=0x7fddd81511d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.212+0000 7fddd77fe700 1 --2- 172.21.15.27:0/4139961185 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddc0003a90 0x7fddd81511d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.212+0000 7fddd77fe700 1 -- 172.21.15.27:0/4139961185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddd815f890 msgr2=0x7fddd815fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.212+0000 7fddd77fe700 1 --2- 172.21.15.27:0/4139961185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddd815f890 0x7fddd815fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.212+0000 7fddd77fe700 1 -- 172.21.15.27:0/4139961185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd8154350 con 0x7fddd80b1b40 2024-01-11T08:28:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.212+0000 7fddd67fc700 1 --2- 172.21.15.27:0/4139961185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddd815f890 0x7fddd815fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.213+0000 7fddd77fe700 1 --2- 172.21.15.27:0/4139961185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd80b1b40 0x7fddd80b1fc0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fddd806f370 tx=0x7fddd00016d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.213+0000 7fddbeffd700 1 -- 172.21.15.27:0/4139961185 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd0006db0 con 0x7fddd80b1b40 2024-01-11T08:28:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.213+0000 7fddbeffd700 1 -- 172.21.15.27:0/4139961185 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd000c5d0 con 0x7fddd80b1b40 2024-01-11T08:28:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.213+0000 7fdddd3ff700 1 -- 172.21.15.27:0/4139961185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd80b2830 con 0x7fddd80b1b40 2024-01-11T08:28:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.213+0000 7fddbeffd700 1 -- 172.21.15.27:0/4139961185 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd000c990 con 0x7fddd80b1b40 2024-01-11T08:28:18.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.213+0000 7fdddd3ff700 1 -- 172.21.15.27:0/4139961185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd815dc20 con 0x7fddd80b1b40 2024-01-11T08:28:18.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.214+0000 7fdddd3ff700 1 -- 172.21.15.27:0/4139961185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd815e120 con 0x7fddd80b1b40 2024-01-11T08:28:18.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.215+0000 7fddbeffd700 1 -- 172.21.15.27:0/4139961185 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fddd0014480 con 0x7fddd80b1b40 2024-01-11T08:28:18.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.216+0000 7fddbeffd700 1 --2- 172.21.15.27:0/4139961185 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fddc4066f30 0x7fddc40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:18.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.216+0000 7fddbeffd700 1 -- 172.21.15.27:0/4139961185 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5733+0+0 (secure 0 0 0) 0x7fddd0083fe0 con 0x7fddd80b1b40 2024-01-11T08:28:18.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.216+0000 7fddd6ffd700 1 --2- 172.21.15.27:0/4139961185 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fddc4066f30 0x7fddc40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:18.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.217+0000 7fddd6ffd700 1 --2- 172.21.15.27:0/4139961185 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fddc4066f30 0x7fddc40693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fddc8007c20 tx=0x7fddc8015000 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:18.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.220+0000 7fddbeffd700 1 -- 172.21.15.27:0/4139961185 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fddd004d5a0 con 0x7fddd80b1b40 2024-01-11T08:28:18.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.423+0000 7fdddd3ff700 1 -- 172.21.15.27:0/4139961185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fddd80718d0 con 0x7fddd80b1b40 2024-01-11T08:28:18.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.424+0000 7fddbeffd700 1 -- 172.21.15.27:0/4139961185 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+15372 (secure 0 0 0) 0x7fddd0019090 con 0x7fddd80b1b40 2024-01-11T08:28:18.423 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:28:18.423 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":115,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:28:17.453892+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:28:15.512106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-11T08:28:16.451115+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:28:18.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.426+0000 7fdddd3ff700 1 -- 172.21.15.27:0/4139961185 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fddc4066f30 msgr2=0x7fddc40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.426+0000 7fdddd3ff700 1 --2- 172.21.15.27:0/4139961185 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fddc4066f30 0x7fddc40693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fddc8007c20 tx=0x7fddc8015000 comp rx=0 tx=0).stop 2024-01-11T08:28:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.426+0000 7fdddd3ff700 1 -- 172.21.15.27:0/4139961185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd80b1b40 msgr2=0x7fddd80b1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.426+0000 7fdddd3ff700 1 --2- 172.21.15.27:0/4139961185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd80b1b40 0x7fddd80b1fc0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fddd806f370 tx=0x7fddd00016d0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.426+0000 7fdddd3ff700 1 -- 172.21.15.27:0/4139961185 shutdown_connections 2024-01-11T08:28:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.426+0000 7fdddd3ff700 1 --2- 172.21.15.27:0/4139961185 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fddc4066f30 0x7fddc40693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.426+0000 7fdddd3ff700 1 --2- 172.21.15.27:0/4139961185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddd80b1b40 0x7fddd80b1fc0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.426+0000 7fdddd3ff700 1 --2- 172.21.15.27:0/4139961185 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fddc0003a90 0x7fddd81511d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.426+0000 7fdddd3ff700 1 --2- 172.21.15.27:0/4139961185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddd815f890 0x7fddd815fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.426+0000 7fdddd3ff700 1 -- 172.21.15.27:0/4139961185 >> 172.21.15.27:0/4139961185 conn(0x7fddd80ce3e0 msgr2=0x7fddd80cacd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.426+0000 7fdddd3ff700 1 -- 172.21.15.27:0/4139961185 shutdown_connections 2024-01-11T08:28:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.427+0000 7fdddd3ff700 1 -- 172.21.15.27:0/4139961185 wait complete. 2024-01-11T08:28:18.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:28:18.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.646+0000 7f8ba33ff700 1 Processor -- start 2024-01-11T08:28:18.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.647+0000 7f8ba33ff700 1 -- start start 2024-01-11T08:28:18.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.648+0000 7f8ba33ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b9c069d40 0x7f8b9c0b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:18.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.648+0000 7f8ba33ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b9c157390 con 0x7f8b9c0c89f0 2024-01-11T08:28:18.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.648+0000 7f8ba33ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b9c06a200 con 0x7f8b9c0c8620 2024-01-11T08:28:18.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.648+0000 7f8ba33ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c156d40 con 0x7f8b9c069d40 2024-01-11T08:28:18.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.648+0000 7f8ba199c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8b9c0c89f0 0x7f8b9c0accc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51546/0 (socket says 172.21.15.27:51546) 2024-01-11T08:28:18.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.648+0000 7f8ba199c700 1 -- 172.21.15.27:0/3077655310 learned_addr learned my addr 172.21.15.27:0/3077655310 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:28:18.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.648+0000 7f8ba099a700 1 --2- 172.21.15.27:0/3077655310 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b9c069d40 0x7f8b9c0b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:18.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.649+0000 7f8b93fff700 1 -- 172.21.15.27:0/3077655310 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856495093 0 0) 0x7f8b9c157390 con 0x7f8b9c0c89f0 2024-01-11T08:28:18.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.649+0000 7f8b93fff700 1 -- 172.21.15.27:0/3077655310 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b88003570 con 0x7f8b9c0c89f0 2024-01-11T08:28:18.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.649+0000 7f8b93fff700 1 -- 172.21.15.27:0/3077655310 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567136012 0 0) 0x7f8b9c06a200 con 0x7f8b9c0c8620 2024-01-11T08:28:18.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.649+0000 7f8b93fff700 1 -- 172.21.15.27:0/3077655310 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b9c157390 con 0x7f8b9c0c8620 2024-01-11T08:28:18.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.649+0000 7f8ba099a700 1 -- 172.21.15.27:0/3077655310 >> v1:172.21.15.77:6789/0 conn(0x7f8b9c0c8620 legacy=0x7f8b9c0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:18.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.649+0000 7f8ba099a700 1 -- 172.21.15.27:0/3077655310 >> v1:172.21.15.27:6789/0 conn(0x7f8b9c0c89f0 legacy=0x7f8b9c0accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:18.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.649+0000 7f8ba099a700 1 -- 172.21.15.27:0/3077655310 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9c157020 con 0x7f8b9c069d40 2024-01-11T08:28:18.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.649+0000 7f8ba099a700 1 --2- 172.21.15.27:0/3077655310 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b9c069d40 0x7f8b9c0b02d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c009940 tx=0x7f8b8c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=181f7f2f17302afb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:18.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.650+0000 7f8b93fff700 1 -- 172.21.15.27:0/3077655310 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c011630 con 0x7f8b9c069d40 2024-01-11T08:28:18.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.650+0000 7f8b93fff700 1 -- 172.21.15.27:0/3077655310 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8c011790 con 0x7f8b9c069d40 2024-01-11T08:28:18.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.650+0000 7f8b93fff700 1 -- 172.21.15.27:0/3077655310 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c011b10 con 0x7f8b9c069d40 2024-01-11T08:28:18.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.650+0000 7f8ba33ff700 1 -- 172.21.15.27:0/3077655310 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b9c069d40 msgr2=0x7f8b9c0b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:18.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.650+0000 7f8ba33ff700 1 --2- 172.21.15.27:0/3077655310 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b9c069d40 0x7f8b9c0b02d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c009940 tx=0x7f8b8c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.650+0000 7f8ba33ff700 1 -- 172.21.15.27:0/3077655310 shutdown_connections 2024-01-11T08:28:18.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.650+0000 7f8ba33ff700 1 --2- 172.21.15.27:0/3077655310 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b9c069d40 0x7f8b9c0b02d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.650+0000 7f8ba33ff700 1 -- 172.21.15.27:0/3077655310 >> 172.21.15.27:0/3077655310 conn(0x7f8b9c0680e0 msgr2=0x7f8b9c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:18.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.650+0000 7f8ba33ff700 1 -- 172.21.15.27:0/3077655310 shutdown_connections 2024-01-11T08:28:18.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.650+0000 7f8ba33ff700 1 -- 172.21.15.27:0/3077655310 wait complete. 2024-01-11T08:28:18.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.652+0000 7f8ba33ff700 1 Processor -- start 2024-01-11T08:28:18.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.652+0000 7f8ba33ff700 1 -- start start 2024-01-11T08:28:18.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.653+0000 7f8ba33ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b9c069d40 0x7f8b9c0ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:18.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.653+0000 7f8ba33ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b9c0c8620 0x7f8b9c167b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:18.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.653+0000 7f8ba33ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b9c0c89f0 0x7f8b9c1458a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:18.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.653+0000 7f8ba33ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c15b2d0 con 0x7f8b9c0c89f0 2024-01-11T08:28:18.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.653+0000 7f8ba33ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c15be50 con 0x7f8b9c0c8620 2024-01-11T08:28:18.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.653+0000 7f8ba33ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c15b520 con 0x7f8b9c069d40 2024-01-11T08:28:18.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.653+0000 7f8ba199c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b9c0c89f0 0x7f8b9c1458a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:18.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.653+0000 7f8ba199c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b9c0c89f0 0x7f8b9c1458a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34044/0 (socket says 172.21.15.27:34044) 2024-01-11T08:28:18.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.653+0000 7f8ba199c700 1 -- 172.21.15.27:0/3926819059 learned_addr learned my addr 172.21.15.27:0/3926819059 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:18.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.654+0000 7f8ba099a700 1 --2- 172.21.15.27:0/3926819059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b9c0c8620 0x7f8b9c167b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:18.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.654+0000 7f8ba119b700 1 --2- 172.21.15.27:0/3926819059 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b9c069d40 0x7f8b9c0ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:18.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.654+0000 7f8ba199c700 1 -- 172.21.15.27:0/3926819059 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b9c069d40 msgr2=0x7f8b9c0ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:18.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.654+0000 7f8ba199c700 1 --2- 172.21.15.27:0/3926819059 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b9c069d40 0x7f8b9c0ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.654+0000 7f8ba199c700 1 -- 172.21.15.27:0/3926819059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b9c0c8620 msgr2=0x7f8b9c167b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:18.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.654+0000 7f8ba199c700 1 --2- 172.21.15.27:0/3926819059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b9c0c8620 0x7f8b9c167b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.654+0000 7f8ba199c700 1 -- 172.21.15.27:0/3926819059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8c009420 con 0x7f8b9c0c89f0 2024-01-11T08:28:18.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.654+0000 7f8ba119b700 1 --2- 172.21.15.27:0/3926819059 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b9c069d40 0x7f8b9c0ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:18.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.654+0000 7f8ba199c700 1 --2- 172.21.15.27:0/3926819059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b9c0c89f0 0x7f8b9c1458a0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400cb70 tx=0x7f8b940019e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:18.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.655+0000 7f8b917fa700 1 -- 172.21.15.27:0/3926819059 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9400a8b0 con 0x7f8b9c0c89f0 2024-01-11T08:28:18.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.655+0000 7f8b917fa700 1 -- 172.21.15.27:0/3926819059 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9400aa30 con 0x7f8b9c0c89f0 2024-01-11T08:28:18.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.655+0000 7f8ba33ff700 1 -- 172.21.15.27:0/3926819059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b9c15b800 con 0x7f8b9c0c89f0 2024-01-11T08:28:18.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.655+0000 7f8b917fa700 1 -- 172.21.15.27:0/3926819059 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9400adf0 con 0x7f8b9c0c89f0 2024-01-11T08:28:18.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.655+0000 7f8ba33ff700 1 -- 172.21.15.27:0/3926819059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b9c0ca880 con 0x7f8b9c0c89f0 2024-01-11T08:28:18.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.656+0000 7f8ba33ff700 1 -- 172.21.15.27:0/3926819059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b9c15bfc0 con 0x7f8b9c0c89f0 2024-01-11T08:28:18.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.657+0000 7f8b917fa700 1 -- 172.21.15.27:0/3926819059 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8b94007300 con 0x7f8b9c0c89f0 2024-01-11T08:28:18.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.658+0000 7f8b917fa700 1 --2- 172.21.15.27:0/3926819059 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8b80067000 0x7f8b800694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:18.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.658+0000 7f8b917fa700 1 -- 172.21.15.27:0/3926819059 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f8b940847c0 con 0x7f8b9c0c89f0 2024-01-11T08:28:18.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.658+0000 7f8ba119b700 1 --2- 172.21.15.27:0/3926819059 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8b80067000 0x7f8b800694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:18.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.659+0000 7f8ba119b700 1 --2- 172.21.15.27:0/3926819059 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8b80067000 0x7f8b800694b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8b98000c00 tx=0x7f8b98009040 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:18.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.662+0000 7f8b917fa700 1 -- 172.21.15.27:0/3926819059 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8b94051a70 con 0x7f8b9c0c89f0 2024-01-11T08:28:18.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.865+0000 7f8ba33ff700 1 -- 172.21.15.27:0/3926819059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b9c0718d0 con 0x7f8b9c0c89f0 2024-01-11T08:28:18.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.866+0000 7f8b917fa700 1 -- 172.21.15.27:0/3926819059 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+15372 (secure 0 0 0) 0x7f8b94055090 con 0x7f8b9c0c89f0 2024-01-11T08:28:18.866 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:28:18.866 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":115,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:28:17.453892+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:28:15.512106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-11T08:28:16.451115+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:28:18.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.868+0000 7f8ba33ff700 1 -- 172.21.15.27:0/3926819059 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8b80067000 msgr2=0x7f8b800694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:18.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.868+0000 7f8ba33ff700 1 --2- 172.21.15.27:0/3926819059 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8b80067000 0x7f8b800694b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8b98000c00 tx=0x7f8b98009040 comp rx=0 tx=0).stop 2024-01-11T08:28:18.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.868+0000 7f8ba33ff700 1 -- 172.21.15.27:0/3926819059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b9c0c89f0 msgr2=0x7f8b9c1458a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:18.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.868+0000 7f8ba33ff700 1 --2- 172.21.15.27:0/3926819059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b9c0c89f0 0x7f8b9c1458a0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400cb70 tx=0x7f8b940019e0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.869+0000 7f8ba33ff700 1 -- 172.21.15.27:0/3926819059 shutdown_connections 2024-01-11T08:28:18.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.869+0000 7f8ba33ff700 1 --2- 172.21.15.27:0/3926819059 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8b80067000 0x7f8b800694b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.869+0000 7f8ba33ff700 1 --2- 172.21.15.27:0/3926819059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b9c0c89f0 0x7f8b9c1458a0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.869+0000 7f8ba33ff700 1 --2- 172.21.15.27:0/3926819059 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b9c069d40 0x7f8b9c0ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.869+0000 7f8ba33ff700 1 --2- 172.21.15.27:0/3926819059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b9c0c8620 0x7f8b9c167b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:18.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.869+0000 7f8ba33ff700 1 -- 172.21.15.27:0/3926819059 >> 172.21.15.27:0/3926819059 conn(0x7f8b9c0680e0 msgr2=0x7f8b9c1538b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:18.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.869+0000 7f8ba33ff700 1 -- 172.21.15.27:0/3926819059 shutdown_connections 2024-01-11T08:28:18.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:18.869+0000 7f8ba33ff700 1 -- 172.21.15.27:0/3926819059 wait complete. 2024-01-11T08:28:18.886 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-11T08:28:19.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.094+0000 7fbc33bff700 1 Processor -- start 2024-01-11T08:28:19.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.094+0000 7fbc33bff700 1 -- start start 2024-01-11T08:28:19.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.096+0000 7fbc33bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc2c155c50 0x7fbc2c0aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:19.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.096+0000 7fbc33bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbc2c0c74f0 0x7fbc2c069910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:19.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.096+0000 7fbc33bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc2c06e480 con 0x7fbc2c0694c0 2024-01-11T08:28:19.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.096+0000 7fbc33bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc2c0cb5a0 con 0x7fbc2c155c50 2024-01-11T08:28:19.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.096+0000 7fbc33bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc2c06df00 con 0x7fbc2c0c74f0 2024-01-11T08:28:19.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.096+0000 7fbc3199b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc2c155c50 0x7fbc2c0aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:19.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.096+0000 7fbc3199b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc2c155c50 0x7fbc2c0aef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34054/0 (socket says 172.21.15.27:34054) 2024-01-11T08:28:19.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.096+0000 7fbc3199b700 1 -- 172.21.15.27:0/1060291437 learned_addr learned my addr 172.21.15.27:0/1060291437 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:19.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.097+0000 7fbc3199b700 1 -- 172.21.15.27:0/1060291437 >> v2:172.21.15.77:3300/0 conn(0x7fbc2c0c74f0 msgr2=0x7fbc2c069910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:19.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.097+0000 7fbc3199b700 1 --2- 172.21.15.27:0/1060291437 >> v2:172.21.15.77:3300/0 conn(0x7fbc2c0c74f0 0x7fbc2c069910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:19.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.097+0000 7fbc3199b700 1 -- 172.21.15.27:0/1060291437 >> v1:172.21.15.77:6789/0 conn(0x7fbc2c0694c0 legacy=0x7fbc2c0c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:19.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.097+0000 7fbc3199b700 1 -- 172.21.15.27:0/1060291437 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c069e50 con 0x7fbc2c155c50 2024-01-11T08:28:19.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.097+0000 7fbc3199b700 1 --2- 172.21.15.27:0/1060291437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc2c155c50 0x7fbc2c0aef80 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c06d540 tx=0x7fbc2800b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e6da4e06711f1da server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:19.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.097+0000 7fbc30999700 1 -- 172.21.15.27:0/1060291437 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2800dba0 con 0x7fbc2c155c50 2024-01-11T08:28:19.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.098+0000 7fbc30999700 1 -- 172.21.15.27:0/1060291437 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2800dd20 con 0x7fbc2c155c50 2024-01-11T08:28:19.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.098+0000 7fbc30999700 1 -- 172.21.15.27:0/1060291437 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28005c50 con 0x7fbc2c155c50 2024-01-11T08:28:19.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.098+0000 7fbc33bff700 1 -- 172.21.15.27:0/1060291437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc2c155c50 msgr2=0x7fbc2c0aef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:19.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.098+0000 7fbc33bff700 1 --2- 172.21.15.27:0/1060291437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc2c155c50 0x7fbc2c0aef80 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c06d540 tx=0x7fbc2800b4b0 comp rx=0 tx=0).stop 2024-01-11T08:28:19.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.098+0000 7fbc33bff700 1 -- 172.21.15.27:0/1060291437 shutdown_connections 2024-01-11T08:28:19.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.098+0000 7fbc33bff700 1 --2- 172.21.15.27:0/1060291437 >> v2:172.21.15.77:3300/0 conn(0x7fbc2c0c74f0 0x7fbc2c069910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:19.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.098+0000 7fbc33bff700 1 --2- 172.21.15.27:0/1060291437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc2c155c50 0x7fbc2c0aef80 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:19.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.098+0000 7fbc33bff700 1 -- 172.21.15.27:0/1060291437 >> 172.21.15.27:0/1060291437 conn(0x7fbc2c0680a0 msgr2=0x7fbc2c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:19.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.098+0000 7fbc33bff700 1 -- 172.21.15.27:0/1060291437 shutdown_connections 2024-01-11T08:28:19.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.098+0000 7fbc33bff700 1 -- 172.21.15.27:0/1060291437 wait complete. 2024-01-11T08:28:19.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.100+0000 7fbc33bff700 1 Processor -- start 2024-01-11T08:28:19.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.100+0000 7fbc33bff700 1 -- start start 2024-01-11T08:28:19.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.101+0000 7fbc33bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbc2c0694c0 0x7fbc2c14b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:19.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.101+0000 7fbc33bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc2c0c74f0 0x7fbc2c14fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:19.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.101+0000 7fbc33bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc2c155c50 0x7fbc2c0c4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:19.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.101+0000 7fbc33bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c1513d0 con 0x7fbc2c155c50 2024-01-11T08:28:19.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.101+0000 7fbc33bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c1507c0 con 0x7fbc2c0c74f0 2024-01-11T08:28:19.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.101+0000 7fbc33bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c150aa0 con 0x7fbc2c0694c0 2024-01-11T08:28:19.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.101+0000 7fbc3219c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc2c155c50 0x7fbc2c0c4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:19.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.101+0000 7fbc3219c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc2c155c50 0x7fbc2c0c4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34060/0 (socket says 172.21.15.27:34060) 2024-01-11T08:28:19.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.101+0000 7fbc3219c700 1 -- 172.21.15.27:0/3309143330 learned_addr learned my addr 172.21.15.27:0/3309143330 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:19.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.101+0000 7fbc3199b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbc2c0694c0 0x7fbc2c14b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:19.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.101+0000 7fbc3119a700 1 --2- 172.21.15.27:0/3309143330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc2c0c74f0 0x7fbc2c14fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:19.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.102+0000 7fbc3219c700 1 -- 172.21.15.27:0/3309143330 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbc2c0694c0 msgr2=0x7fbc2c14b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:19.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.102+0000 7fbc3219c700 1 --2- 172.21.15.27:0/3309143330 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbc2c0694c0 0x7fbc2c14b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:19.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.102+0000 7fbc3219c700 1 -- 172.21.15.27:0/3309143330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc2c0c74f0 msgr2=0x7fbc2c14fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:19.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.102+0000 7fbc3219c700 1 --2- 172.21.15.27:0/3309143330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc2c0c74f0 0x7fbc2c14fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:19.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.102+0000 7fbc3219c700 1 -- 172.21.15.27:0/3309143330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2800b730 con 0x7fbc2c155c50 2024-01-11T08:28:19.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.102+0000 7fbc3219c700 1 --2- 172.21.15.27:0/3309143330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc2c155c50 0x7fbc2c0c4ee0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400bc30 tx=0x7fbc24009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:19.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.102+0000 7fbc21ffb700 1 -- 172.21.15.27:0/3309143330 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc240092e0 con 0x7fbc2c155c50 2024-01-11T08:28:19.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.102+0000 7fbc21ffb700 1 -- 172.21.15.27:0/3309143330 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2400c630 con 0x7fbc2c155c50 2024-01-11T08:28:19.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.102+0000 7fbc33bff700 1 -- 172.21.15.27:0/3309143330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc2c150d80 con 0x7fbc2c155c50 2024-01-11T08:28:19.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.102+0000 7fbc21ffb700 1 -- 172.21.15.27:0/3309143330 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2400ca10 con 0x7fbc2c155c50 2024-01-11T08:28:19.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.102+0000 7fbc33bff700 1 -- 172.21.15.27:0/3309143330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc2c14bcd0 con 0x7fbc2c155c50 2024-01-11T08:28:19.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.103+0000 7fbc33bff700 1 -- 172.21.15.27:0/3309143330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc2c04c3c0 con 0x7fbc2c155c50 2024-01-11T08:28:19.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.104+0000 7fbc21ffb700 1 -- 172.21.15.27:0/3309143330 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fbc24012070 con 0x7fbc2c155c50 2024-01-11T08:28:19.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.105+0000 7fbc21ffb700 1 --2- 172.21.15.27:0/3309143330 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbc10066f60 0x7fbc10069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:19.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.105+0000 7fbc21ffb700 1 -- 172.21.15.27:0/3309143330 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5733+0+0 (secure 0 0 0) 0x7fbc240840f0 con 0x7fbc2c155c50 2024-01-11T08:28:19.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.106+0000 7fbc3199b700 1 --2- 172.21.15.27:0/3309143330 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbc10066f60 0x7fbc10069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:19.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.106+0000 7fbc3199b700 1 --2- 172.21.15.27:0/3309143330 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbc10066f60 0x7fbc10069410 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c06d540 tx=0x7fbc28018000 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:19.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.110+0000 7fbc21ffb700 1 -- 172.21.15.27:0/3309143330 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbc240512a0 con 0x7fbc2c155c50 2024-01-11T08:28:19.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.361+0000 7fbc33bff700 1 -- 172.21.15.27:0/3309143330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fbc2c068620 con 0x7fbc2c155c50 2024-01-11T08:28:19.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.976+0000 7fbc21ffb700 1 -- 172.21.15.27:0/3309143330 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v58) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbc240548c0 con 0x7fbc2c155c50 2024-01-11T08:28:19.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.979+0000 7fbc33bff700 1 -- 172.21.15.27:0/3309143330 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbc10066f60 msgr2=0x7fbc10069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:19.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.979+0000 7fbc33bff700 1 --2- 172.21.15.27:0/3309143330 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbc10066f60 0x7fbc10069410 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c06d540 tx=0x7fbc28018000 comp rx=0 tx=0).stop 2024-01-11T08:28:19.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.979+0000 7fbc33bff700 1 -- 172.21.15.27:0/3309143330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc2c155c50 msgr2=0x7fbc2c0c4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:19.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.979+0000 7fbc33bff700 1 --2- 172.21.15.27:0/3309143330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc2c155c50 0x7fbc2c0c4ee0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400bc30 tx=0x7fbc24009840 comp rx=0 tx=0).stop 2024-01-11T08:28:19.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.979+0000 7fbc33bff700 1 -- 172.21.15.27:0/3309143330 shutdown_connections 2024-01-11T08:28:19.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.979+0000 7fbc33bff700 1 --2- 172.21.15.27:0/3309143330 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbc10066f60 0x7fbc10069410 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:19.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.979+0000 7fbc33bff700 1 --2- 172.21.15.27:0/3309143330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc2c155c50 0x7fbc2c0c4ee0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:19.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.979+0000 7fbc33bff700 1 --2- 172.21.15.27:0/3309143330 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbc2c0694c0 0x7fbc2c14b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:19.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.979+0000 7fbc33bff700 1 --2- 172.21.15.27:0/3309143330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc2c0c74f0 0x7fbc2c14fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:19.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.979+0000 7fbc33bff700 1 -- 172.21.15.27:0/3309143330 >> 172.21.15.27:0/3309143330 conn(0x7fbc2c0680a0 msgr2=0x7fbc2c1538f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:19.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.979+0000 7fbc33bff700 1 -- 172.21.15.27:0/3309143330 shutdown_connections 2024-01-11T08:28:19.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:19.979+0000 7fbc33bff700 1 -- 172.21.15.27:0/3309143330 wait complete. 2024-01-11T08:28:19.997 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-11T08:28:20.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.202+0000 7f5ff57ff700 1 Processor -- start 2024-01-11T08:28:20.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.203+0000 7f5ff57ff700 1 -- start start 2024-01-11T08:28:20.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.204+0000 7f5ff57ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ff015a1b0 0x7f5ff015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:20.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.204+0000 7f5ff57ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5ff00ccd70 0x7f5ff00cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:20.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.204+0000 7f5ff57ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff00cf850 con 0x7f5ff00685c0 2024-01-11T08:28:20.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.204+0000 7f5ff57ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ff00cec40 con 0x7f5ff015a1b0 2024-01-11T08:28:20.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.204+0000 7f5ff57ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ff00cef20 con 0x7f5ff00ccd70 2024-01-11T08:28:20.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.204+0000 7f5feeffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ff015a1b0 0x7f5ff015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:20.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.204+0000 7f5feeffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ff015a1b0 0x7f5ff015a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34062/0 (socket says 172.21.15.27:34062) 2024-01-11T08:28:20.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.204+0000 7f5feeffd700 1 -- 172.21.15.27:0/1080372699 learned_addr learned my addr 172.21.15.27:0/1080372699 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:20.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.204+0000 7f5fef7fe700 1 --1- 172.21.15.27:0/1080372699 >> v1:172.21.15.27:6789/0 conn(0x7f5ff00685c0 0x7f5ff00689f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51562/0 (socket says 172.21.15.27:51562) 2024-01-11T08:28:20.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.204+0000 7f5fee7fc700 1 --2- 172.21.15.27:0/1080372699 >> v2:172.21.15.77:3300/0 conn(0x7f5ff00ccd70 0x7f5ff00cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:20.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.205+0000 7f5fedffb700 1 -- 172.21.15.27:0/1080372699 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246956537 0 0) 0x7f5ff00cf850 con 0x7f5ff00685c0 2024-01-11T08:28:20.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.205+0000 7f5feeffd700 1 -- 172.21.15.27:0/1080372699 >> v2:172.21.15.77:3300/0 conn(0x7f5ff00ccd70 msgr2=0x7f5ff00cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:20.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.205+0000 7f5feeffd700 1 --2- 172.21.15.27:0/1080372699 >> v2:172.21.15.77:3300/0 conn(0x7f5ff00ccd70 0x7f5ff00cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:20.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.205+0000 7f5feeffd700 1 -- 172.21.15.27:0/1080372699 >> v1:172.21.15.27:6789/0 conn(0x7f5ff00685c0 legacy=0x7f5ff00689f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:20.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.205+0000 7f5feeffd700 1 -- 172.21.15.27:0/1080372699 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff00cf200 con 0x7f5ff015a1b0 2024-01-11T08:28:20.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.205+0000 7f5feeffd700 1 --2- 172.21.15.27:0/1080372699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff015a1b0 0x7f5ff015a580 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00696f0 tx=0x7f5fe0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad57bb0f8017bffa server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:20.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.205+0000 7f5fedffb700 1 -- 172.21.15.27:0/1080372699 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe0009e70 con 0x7f5ff015a1b0 2024-01-11T08:28:20.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.205+0000 7f5fedffb700 1 -- 172.21.15.27:0/1080372699 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe000b860 con 0x7f5ff015a1b0 2024-01-11T08:28:20.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.205+0000 7f5fedffb700 1 -- 172.21.15.27:0/1080372699 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe000bc00 con 0x7f5ff015a1b0 2024-01-11T08:28:20.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.206+0000 7f5ff57ff700 1 -- 172.21.15.27:0/1080372699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff015a1b0 msgr2=0x7f5ff015a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:20.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.206+0000 7f5ff57ff700 1 --2- 172.21.15.27:0/1080372699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff015a1b0 0x7f5ff015a580 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00696f0 tx=0x7f5fe0009250 comp rx=0 tx=0).stop 2024-01-11T08:28:20.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.206+0000 7f5ff57ff700 1 -- 172.21.15.27:0/1080372699 shutdown_connections 2024-01-11T08:28:20.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.206+0000 7f5ff57ff700 1 --2- 172.21.15.27:0/1080372699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff015a1b0 0x7f5ff015a580 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:20.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.206+0000 7f5ff57ff700 1 --2- 172.21.15.27:0/1080372699 >> v2:172.21.15.77:3300/0 conn(0x7f5ff00ccd70 0x7f5ff00cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:20.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.206+0000 7f5ff57ff700 1 -- 172.21.15.27:0/1080372699 >> 172.21.15.27:0/1080372699 conn(0x7f5ff006ed40 msgr2=0x7f5ff006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:20.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.206+0000 7f5ff57ff700 1 -- 172.21.15.27:0/1080372699 shutdown_connections 2024-01-11T08:28:20.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.206+0000 7f5ff57ff700 1 -- 172.21.15.27:0/1080372699 wait complete. 2024-01-11T08:28:20.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.208+0000 7f5ff57ff700 1 Processor -- start 2024-01-11T08:28:20.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.208+0000 7f5ff57ff700 1 -- start start 2024-01-11T08:28:20.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.209+0000 7f5ff57ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ff00685c0 0x7f5ff00a6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:20.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.209+0000 7f5ff57ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff00ccd70 0x7f5ff00c7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:20.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.209+0000 7f5ff57ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ff015a1b0 0x7f5ff01669e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:20.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.209+0000 7f5ff57ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff00a7610 con 0x7f5ff00ccd70 2024-01-11T08:28:20.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.209+0000 7f5ff57ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff00b5970 con 0x7f5ff015a1b0 2024-01-11T08:28:20.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.209+0000 7f5ff57ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff0166f20 con 0x7f5ff00685c0 2024-01-11T08:28:20.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.209+0000 7f5fee7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff00ccd70 0x7f5ff00c7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:20.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.209+0000 7f5fee7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff00ccd70 0x7f5ff00c7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34076/0 (socket says 172.21.15.27:34076) 2024-01-11T08:28:20.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.209+0000 7f5fee7fc700 1 -- 172.21.15.27:0/3881366270 learned_addr learned my addr 172.21.15.27:0/3881366270 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:20.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.209+0000 7f5feeffd700 1 --2- 172.21.15.27:0/3881366270 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ff00685c0 0x7f5ff00a6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:20.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.209+0000 7f5fef7fe700 1 --2- 172.21.15.27:0/3881366270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ff015a1b0 0x7f5ff01669e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:20.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.209+0000 7f5fee7fc700 1 -- 172.21.15.27:0/3881366270 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ff00685c0 msgr2=0x7f5ff00a6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:20.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.209+0000 7f5fee7fc700 1 --2- 172.21.15.27:0/3881366270 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ff00685c0 0x7f5ff00a6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:20.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.209+0000 7f5fee7fc700 1 -- 172.21.15.27:0/3881366270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ff015a1b0 msgr2=0x7f5ff01669e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:20.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.209+0000 7f5fee7fc700 1 --2- 172.21.15.27:0/3881366270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ff015a1b0 0x7f5ff01669e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:20.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.209+0000 7f5fee7fc700 1 -- 172.21.15.27:0/3881366270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe00094f0 con 0x7f5ff00ccd70 2024-01-11T08:28:20.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.209+0000 7f5fef7fe700 1 --2- 172.21.15.27:0/3881366270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ff015a1b0 0x7f5ff01669e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:20.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.210+0000 7f5fee7fc700 1 --2- 172.21.15.27:0/3881366270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff00ccd70 0x7f5ff00c7e40 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400ecb0 tx=0x7f5fe40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:20.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.210+0000 7f5fd6ffd700 1 -- 172.21.15.27:0/3881366270 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe400c920 con 0x7f5ff00ccd70 2024-01-11T08:28:20.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.210+0000 7f5fd6ffd700 1 -- 172.21.15.27:0/3881366270 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe400ca80 con 0x7f5ff00ccd70 2024-01-11T08:28:20.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.210+0000 7f5ff57ff700 1 -- 172.21.15.27:0/3881366270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff00b5040 con 0x7f5ff00ccd70 2024-01-11T08:28:20.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.210+0000 7f5fd6ffd700 1 -- 172.21.15.27:0/3881366270 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe400ce20 con 0x7f5ff00ccd70 2024-01-11T08:28:20.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.210+0000 7f5ff57ff700 1 -- 172.21.15.27:0/3881366270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff00b5600 con 0x7f5ff00ccd70 2024-01-11T08:28:20.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.211+0000 7f5ff57ff700 1 -- 172.21.15.27:0/3881366270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff00b5b20 con 0x7f5ff00ccd70 2024-01-11T08:28:20.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.212+0000 7f5fd6ffd700 1 -- 172.21.15.27:0/3881366270 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5fe4013680 con 0x7f5ff00ccd70 2024-01-11T08:28:20.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.213+0000 7f5fd6ffd700 1 --2- 172.21.15.27:0/3881366270 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5fdc066f30 0x7f5fdc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:20.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.213+0000 7f5fd6ffd700 1 -- 172.21.15.27:0/3881366270 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f5fe40848f0 con 0x7f5ff00ccd70 2024-01-11T08:28:20.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.214+0000 7f5feeffd700 1 --2- 172.21.15.27:0/3881366270 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5fdc066f30 0x7f5fdc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:20.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.214+0000 7f5feeffd700 1 --2- 172.21.15.27:0/3881366270 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5fdc066f30 0x7f5fdc0693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00696f0 tx=0x7f5fe0017000 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:20.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.218+0000 7f5fd6ffd700 1 -- 172.21.15.27:0/3881366270 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5ff00b5b20 con 0x7f5ff00ccd70 2024-01-11T08:28:20.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.465+0000 7f5ff57ff700 1 -- 172.21.15.27:0/3881366270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5ff00b5b20 con 0x7f5ff00ccd70 2024-01-11T08:28:20.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.978+0000 7f5fd6ffd700 1 -- 172.21.15.27:0/3881366270 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v59) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5ff00b5b20 con 0x7f5ff00ccd70 2024-01-11T08:28:20.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.981+0000 7f5ff57ff700 1 -- 172.21.15.27:0/3881366270 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5fdc066f30 msgr2=0x7f5fdc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:20.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.981+0000 7f5ff57ff700 1 --2- 172.21.15.27:0/3881366270 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5fdc066f30 0x7f5fdc0693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00696f0 tx=0x7f5fe0017000 comp rx=0 tx=0).stop 2024-01-11T08:28:20.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.981+0000 7f5ff57ff700 1 -- 172.21.15.27:0/3881366270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff00ccd70 msgr2=0x7f5ff00c7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:20.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.981+0000 7f5ff57ff700 1 --2- 172.21.15.27:0/3881366270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff00ccd70 0x7f5ff00c7e40 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400ecb0 tx=0x7f5fe40091a0 comp rx=0 tx=0).stop 2024-01-11T08:28:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.981+0000 7f5ff57ff700 1 -- 172.21.15.27:0/3881366270 shutdown_connections 2024-01-11T08:28:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.981+0000 7f5ff57ff700 1 --2- 172.21.15.27:0/3881366270 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5fdc066f30 0x7f5fdc0693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.981+0000 7f5ff57ff700 1 --2- 172.21.15.27:0/3881366270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ff015a1b0 0x7f5ff01669e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.981+0000 7f5ff57ff700 1 --2- 172.21.15.27:0/3881366270 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5ff00685c0 0x7f5ff00a6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.981+0000 7f5ff57ff700 1 --2- 172.21.15.27:0/3881366270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff00ccd70 0x7f5ff00c7e40 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.981+0000 7f5ff57ff700 1 -- 172.21.15.27:0/3881366270 >> 172.21.15.27:0/3881366270 conn(0x7f5ff006ed40 msgr2=0x7f5ff0158b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.981+0000 7f5ff57ff700 1 -- 172.21.15.27:0/3881366270 shutdown_connections 2024-01-11T08:28:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:20.981+0000 7f5ff57ff700 1 -- 172.21.15.27:0/3881366270 wait complete. 2024-01-11T08:28:20.999 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:28:21.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.201+0000 7ff7df3ff700 1 Processor -- start 2024-01-11T08:28:21.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.201+0000 7ff7df3ff700 1 -- start start 2024-01-11T08:28:21.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.202+0000 7ff7df3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff7d80ca3e0 0x7ff7d80cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:21.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.202+0000 7ff7df3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff7d80ca7b0 0x7ff7d8159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:21.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.202+0000 7ff7df3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff7d80c8a20 0x7ff7d80c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:21.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.202+0000 7ff7df3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d815ae70 con 0x7ff7d80ca3e0 2024-01-11T08:28:21.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.202+0000 7ff7df3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d815a3c0 con 0x7ff7d80ca7b0 2024-01-11T08:28:21.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.202+0000 7ff7df3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d80cc4f0 con 0x7ff7d80c8a20 2024-01-11T08:28:21.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.203+0000 7ff7dd19b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff7d80ca3e0 0x7ff7d80cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:21.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.203+0000 7ff7dd19b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff7d80ca3e0 0x7ff7d80cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34086/0 (socket says 172.21.15.27:34086) 2024-01-11T08:28:21.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.203+0000 7ff7dd19b700 1 -- 172.21.15.27:0/788497521 learned_addr learned my addr 172.21.15.27:0/788497521 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:21.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.203+0000 7ff7dd99c700 1 --2- 172.21.15.27:0/788497521 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff7d80c8a20 0x7ff7d80c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:21.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.203+0000 7ff7dc99a700 1 --2- 172.21.15.27:0/788497521 >> v2:172.21.15.77:3300/0 conn(0x7ff7d80ca7b0 0x7ff7d8159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:21.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.203+0000 7ff7dd19b700 1 -- 172.21.15.27:0/788497521 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff7d80c8a20 msgr2=0x7ff7d80c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:21.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.203+0000 7ff7dd19b700 1 --2- 172.21.15.27:0/788497521 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff7d80c8a20 0x7ff7d80c8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.203+0000 7ff7dd19b700 1 -- 172.21.15.27:0/788497521 >> v2:172.21.15.77:3300/0 conn(0x7ff7d80ca7b0 msgr2=0x7ff7d8159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:21.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.203+0000 7ff7dd19b700 1 --2- 172.21.15.27:0/788497521 >> v2:172.21.15.77:3300/0 conn(0x7ff7d80ca7b0 0x7ff7d8159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.203+0000 7ff7dd19b700 1 -- 172.21.15.27:0/788497521 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d815d640 con 0x7ff7d80ca3e0 2024-01-11T08:28:21.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.203+0000 7ff7dd99c700 1 --2- 172.21.15.27:0/788497521 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff7d80c8a20 0x7ff7d80c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:21.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.204+0000 7ff7dd19b700 1 --2- 172.21.15.27:0/788497521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7d80ca3e0 0x7ff7d80cca20 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7ff7d40099e0 tx=0x7ff7d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b21a994398209be4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:21.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.204+0000 7ff7cffff700 1 -- 172.21.15.27:0/788497521 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d400b7e0 con 0x7ff7d80ca3e0 2024-01-11T08:28:21.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.204+0000 7ff7cffff700 1 -- 172.21.15.27:0/788497521 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7d400b940 con 0x7ff7d80ca3e0 2024-01-11T08:28:21.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.204+0000 7ff7cffff700 1 -- 172.21.15.27:0/788497521 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d400bce0 con 0x7ff7d80ca3e0 2024-01-11T08:28:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.204+0000 7ff7df3ff700 1 -- 172.21.15.27:0/788497521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7d80ca3e0 msgr2=0x7ff7d80cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.204+0000 7ff7df3ff700 1 --2- 172.21.15.27:0/788497521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7d80ca3e0 0x7ff7d80cca20 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7ff7d40099e0 tx=0x7ff7d4009250 comp rx=0 tx=0).stop 2024-01-11T08:28:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.205+0000 7ff7df3ff700 1 -- 172.21.15.27:0/788497521 shutdown_connections 2024-01-11T08:28:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.205+0000 7ff7df3ff700 1 --2- 172.21.15.27:0/788497521 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff7d80c8a20 0x7ff7d80c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.205+0000 7ff7df3ff700 1 --2- 172.21.15.27:0/788497521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7d80ca3e0 0x7ff7d80cca20 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.205+0000 7ff7df3ff700 1 --2- 172.21.15.27:0/788497521 >> v2:172.21.15.77:3300/0 conn(0x7ff7d80ca7b0 0x7ff7d8159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.205+0000 7ff7df3ff700 1 -- 172.21.15.27:0/788497521 >> 172.21.15.27:0/788497521 conn(0x7ff7d80ce3e0 msgr2=0x7ff7d80ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.205+0000 7ff7df3ff700 1 -- 172.21.15.27:0/788497521 shutdown_connections 2024-01-11T08:28:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.205+0000 7ff7df3ff700 1 -- 172.21.15.27:0/788497521 wait complete. 2024-01-11T08:28:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.206+0000 7ff7df3ff700 1 Processor -- start 2024-01-11T08:28:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.206+0000 7ff7df3ff700 1 -- start start 2024-01-11T08:28:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.207+0000 7ff7df3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7d80c8a20 0x7ff7d8150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.207+0000 7ff7df3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7d80ca3e0 0x7ff7d80b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.207+0000 7ff7df3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff7d80ca7b0 0x7ff7d80c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.207+0000 7ff7df3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d80c63e0 con 0x7ff7d80c8a20 2024-01-11T08:28:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.207+0000 7ff7df3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d80c6680 con 0x7ff7d80ca3e0 2024-01-11T08:28:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.207+0000 7ff7df3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d8158d30 con 0x7ff7d80ca7b0 2024-01-11T08:28:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.208+0000 7ff7dd19b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7d80c8a20 0x7ff7d8150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.208+0000 7ff7dd19b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7d80c8a20 0x7ff7d8150a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34088/0 (socket says 172.21.15.27:34088) 2024-01-11T08:28:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.208+0000 7ff7dd19b700 1 -- 172.21.15.27:0/3922841597 learned_addr learned my addr 172.21.15.27:0/3922841597 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.208+0000 7ff7dd99c700 1 --2- 172.21.15.27:0/3922841597 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff7d80ca7b0 0x7ff7d80c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.208+0000 7ff7dc99a700 1 --2- 172.21.15.27:0/3922841597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7d80ca3e0 0x7ff7d80b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.208+0000 7ff7dd19b700 1 -- 172.21.15.27:0/3922841597 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff7d80ca7b0 msgr2=0x7ff7d80c5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.208+0000 7ff7dd19b700 1 --2- 172.21.15.27:0/3922841597 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff7d80ca7b0 0x7ff7d80c5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.208+0000 7ff7dd19b700 1 -- 172.21.15.27:0/3922841597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7d80ca3e0 msgr2=0x7ff7d80b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.208+0000 7ff7dd19b700 1 --2- 172.21.15.27:0/3922841597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7d80ca3e0 0x7ff7d80b3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.208+0000 7ff7dd19b700 1 -- 172.21.15.27:0/3922841597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d40094f0 con 0x7ff7d80c8a20 2024-01-11T08:28:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.208+0000 7ff7dd99c700 1 --2- 172.21.15.27:0/3922841597 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff7d80ca7b0 0x7ff7d80c5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.208+0000 7ff7dd19b700 1 --2- 172.21.15.27:0/3922841597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7d80c8a20 0x7ff7d8150a10 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7ff7d40090b0 tx=0x7ff7d4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.209+0000 7ff7cd7fa700 1 -- 172.21.15.27:0/3922841597 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d4011e90 con 0x7ff7d80c8a20 2024-01-11T08:28:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.209+0000 7ff7cd7fa700 1 -- 172.21.15.27:0/3922841597 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7d401f070 con 0x7ff7d80c8a20 2024-01-11T08:28:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.209+0000 7ff7df3ff700 1 -- 172.21.15.27:0/3922841597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d8159010 con 0x7ff7d80c8a20 2024-01-11T08:28:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.209+0000 7ff7cd7fa700 1 -- 172.21.15.27:0/3922841597 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d40266d0 con 0x7ff7d80c8a20 2024-01-11T08:28:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.209+0000 7ff7df3ff700 1 -- 172.21.15.27:0/3922841597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d80c6960 con 0x7ff7d80c8a20 2024-01-11T08:28:21.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.210+0000 7ff7df3ff700 1 -- 172.21.15.27:0/3922841597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d80718d0 con 0x7ff7d80c8a20 2024-01-11T08:28:21.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.211+0000 7ff7cd7fa700 1 -- 172.21.15.27:0/3922841597 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff7d4026a40 con 0x7ff7d80c8a20 2024-01-11T08:28:21.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.212+0000 7ff7cd7fa700 1 --2- 172.21.15.27:0/3922841597 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff7bc066f30 0x7ff7bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:21.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.212+0000 7ff7cd7fa700 1 -- 172.21.15.27:0/3922841597 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5733+0+0 (secure 0 0 0) 0x7ff7d4021070 con 0x7ff7d80c8a20 2024-01-11T08:28:21.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.213+0000 7ff7dc99a700 1 --2- 172.21.15.27:0/3922841597 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff7bc066f30 0x7ff7bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:21.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.213+0000 7ff7dc99a700 1 --2- 172.21.15.27:0/3922841597 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff7bc066f30 0x7ff7bc0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff7d80c89f0 tx=0x7ff7c8008040 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:21.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.216+0000 7ff7cd7fa700 1 -- 172.21.15.27:0/3922841597 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff7d4053640 con 0x7ff7d80c8a20 2024-01-11T08:28:21.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.463+0000 7ff7df3ff700 1 -- 172.21.15.27:0/3922841597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7d80c9290 con 0x7ff7d80c8a20 2024-01-11T08:28:21.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.464+0000 7ff7cd7fa700 1 -- 172.21.15.27:0/3922841597 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4596 (secure 0 0 0) 0x7ff7d4056c60 con 0x7ff7d80c8a20 2024-01-11T08:28:21.463 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:28:21.463 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6608,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2001630215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2001630215},{"type":"v1","addr":"172.21.15.77:6834","nonce":2001630215}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7348,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/884469933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":884469933},{"type":"v1","addr":"172.21.15.77:6837","nonce":884469933}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":7584,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/4086481881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4086481881},{"type":"v1","addr":"172.21.15.27:6834","nonce":4086481881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:28:17.454224+0000","modified":"2024-01-11T08:28:20.975892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7587":{"gid":7587,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/2923524035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2923524035},{"type":"v1","addr":"172.21.15.27:6835","nonce":2923524035}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2024-01-11T08:28:21.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.466+0000 7ff7df3ff700 1 -- 172.21.15.27:0/3922841597 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff7bc066f30 msgr2=0x7ff7bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:21.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.466+0000 7ff7df3ff700 1 --2- 172.21.15.27:0/3922841597 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff7bc066f30 0x7ff7bc0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff7d80c89f0 tx=0x7ff7c8008040 comp rx=0 tx=0).stop 2024-01-11T08:28:21.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.466+0000 7ff7df3ff700 1 -- 172.21.15.27:0/3922841597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7d80c8a20 msgr2=0x7ff7d8150a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:21.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.466+0000 7ff7df3ff700 1 --2- 172.21.15.27:0/3922841597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7d80c8a20 0x7ff7d8150a10 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7ff7d40090b0 tx=0x7ff7d4017000 comp rx=0 tx=0).stop 2024-01-11T08:28:21.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.466+0000 7ff7df3ff700 1 -- 172.21.15.27:0/3922841597 shutdown_connections 2024-01-11T08:28:21.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.466+0000 7ff7df3ff700 1 --2- 172.21.15.27:0/3922841597 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff7bc066f30 0x7ff7bc0693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.466+0000 7ff7df3ff700 1 --2- 172.21.15.27:0/3922841597 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff7d80ca7b0 0x7ff7d80c5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.466+0000 7ff7df3ff700 1 --2- 172.21.15.27:0/3922841597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7d80c8a20 0x7ff7d8150a10 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.466+0000 7ff7df3ff700 1 --2- 172.21.15.27:0/3922841597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7d80ca3e0 0x7ff7d80b3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.466+0000 7ff7df3ff700 1 -- 172.21.15.27:0/3922841597 >> 172.21.15.27:0/3922841597 conn(0x7ff7d80ce3e0 msgr2=0x7ff7d815b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:21.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.467+0000 7ff7df3ff700 1 -- 172.21.15.27:0/3922841597 shutdown_connections 2024-01-11T08:28:21.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.467+0000 7ff7df3ff700 1 -- 172.21.15.27:0/3922841597 wait complete. 2024-01-11T08:28:21.467 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 59 2024-01-11T08:28:21.484 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:28:21.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.685+0000 7f33d21ff700 1 Processor -- start 2024-01-11T08:28:21.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.685+0000 7f33d21ff700 1 -- start start 2024-01-11T08:28:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.686+0000 7f33d21ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33cc0b2b90 0x7f33cc159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.686+0000 7f33d21ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc0cd730 con 0x7f33cc0b03e0 2024-01-11T08:28:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.686+0000 7f33d21ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc0ccb20 con 0x7f33cc0b07b0 2024-01-11T08:28:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.686+0000 7f33d21ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0cce00 con 0x7f33cc0b2b90 2024-01-11T08:28:21.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.687+0000 7f33cb7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f33cc0b03e0 0x7f33cc0b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51568/0 (socket says 172.21.15.27:51568) 2024-01-11T08:28:21.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.687+0000 7f33cb7fe700 1 -- 172.21.15.27:0/1581019085 learned_addr learned my addr 172.21.15.27:0/1581019085 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:28:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.687+0000 7f33caffd700 1 --2- 172.21.15.27:0/1581019085 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33cc0b2b90 0x7f33cc159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.687+0000 7f33ca7fc700 1 -- 172.21.15.27:0/1581019085 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571334250 0 0) 0x7f33cc0cd730 con 0x7f33cc0b03e0 2024-01-11T08:28:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.687+0000 7f33ca7fc700 1 -- 172.21.15.27:0/1581019085 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33b4003570 con 0x7f33cc0b03e0 2024-01-11T08:28:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.687+0000 7f33ca7fc700 1 -- 172.21.15.27:0/1581019085 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573184025 0 0) 0x7f33cc0ccb20 con 0x7f33cc0b07b0 2024-01-11T08:28:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.687+0000 7f33ca7fc700 1 -- 172.21.15.27:0/1581019085 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33cc0cd730 con 0x7f33cc0b07b0 2024-01-11T08:28:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.687+0000 7f33ca7fc700 1 -- 172.21.15.27:0/1581019085 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3628150055 0 0) 0x7f33b4003570 con 0x7f33cc0b03e0 2024-01-11T08:28:21.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.687+0000 7f33ca7fc700 1 -- 172.21.15.27:0/1581019085 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33cc0b2b90 msgr2=0x7f33cc159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.687+0000 7f33ca7fc700 1 --2- 172.21.15.27:0/1581019085 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33cc0b2b90 0x7f33cc159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.687+0000 7f33ca7fc700 1 -- 172.21.15.27:0/1581019085 >> v1:172.21.15.77:6789/0 conn(0x7f33cc0b07b0 legacy=0x7f33cc159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.687+0000 7f33ca7fc700 1 -- 172.21.15.27:0/1581019085 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc0cd0e0 con 0x7f33cc0b03e0 2024-01-11T08:28:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.687+0000 7f33ca7fc700 1 -- 172.21.15.27:0/1581019085 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f33bc001470 con 0x7f33cc0b03e0 2024-01-11T08:28:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.687+0000 7f33caffd700 1 --2- 172.21.15.27:0/1581019085 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33cc0b2b90 0x7f33cc159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:28:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.688+0000 7f33ca7fc700 1 -- 172.21.15.27:0/1581019085 >> v1:172.21.15.27:6789/0 conn(0x7f33cc0b03e0 legacy=0x7f33cc0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.688+0000 7f33ca7fc700 1 --2- 172.21.15.27:0/1581019085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33b4003c20 0x7f33b4004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.688+0000 7f33ca7fc700 1 --2- 172.21.15.27:0/1581019085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b4004810 0x7f33b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.688+0000 7f33ca7fc700 1 --2- 172.21.15.27:0/1581019085 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33cc0b2b90 0x7f33b40051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.688+0000 7f33ca7fc700 1 -- 172.21.15.27:0/1581019085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0cce00 con 0x7f33b4003c20 2024-01-11T08:28:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.688+0000 7f33ca7fc700 1 -- 172.21.15.27:0/1581019085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b400bbb0 con 0x7f33b4004810 2024-01-11T08:28:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.688+0000 7f33ca7fc700 1 -- 172.21.15.27:0/1581019085 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b400bd70 con 0x7f33cc0b2b90 2024-01-11T08:28:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.688+0000 7f33cb7fe700 1 --2- 172.21.15.27:0/1581019085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33b4003c20 0x7f33b4004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.688+0000 7f33caffd700 1 --2- 172.21.15.27:0/1581019085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b4004810 0x7f33b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.688+0000 7f33cbfff700 1 --2- 172.21.15.27:0/1581019085 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33cc0b2b90 0x7f33b40051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.688+0000 7f33cb7fe700 1 -- 172.21.15.27:0/1581019085 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33cc0b2b90 msgr2=0x7f33b40051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.688+0000 7f33cb7fe700 1 --2- 172.21.15.27:0/1581019085 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33cc0b2b90 0x7f33b40051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.688+0000 7f33cb7fe700 1 -- 172.21.15.27:0/1581019085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b4004810 msgr2=0x7f33b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.688+0000 7f33cb7fe700 1 --2- 172.21.15.27:0/1581019085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b4004810 0x7f33b4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.688+0000 7f33cb7fe700 1 -- 172.21.15.27:0/1581019085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c0009420 con 0x7f33b4003c20 2024-01-11T08:28:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.689+0000 7f33cb7fe700 1 --2- 172.21.15.27:0/1581019085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33b4003c20 0x7f33b4004010 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00c210 tx=0x7f33bc001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d54538d21440a7 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.689+0000 7f33ca7fc700 1 -- 172.21.15.27:0/1581019085 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc00cc10 con 0x7f33b4003c20 2024-01-11T08:28:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.689+0000 7f33ca7fc700 1 -- 172.21.15.27:0/1581019085 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33bc00cd70 con 0x7f33b4003c20 2024-01-11T08:28:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.689+0000 7f33ca7fc700 1 -- 172.21.15.27:0/1581019085 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc005b70 con 0x7f33b4003c20 2024-01-11T08:28:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.689+0000 7f33d21ff700 1 -- 172.21.15.27:0/1581019085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33b4003c20 msgr2=0x7f33b4004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:21.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.689+0000 7f33d21ff700 1 --2- 172.21.15.27:0/1581019085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33b4003c20 0x7f33b4004010 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00c210 tx=0x7f33bc001d40 comp rx=0 tx=0).stop 2024-01-11T08:28:21.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.689+0000 7f33cbfff700 1 -- 172.21.15.27:0/1581019085 reap_dead start 2024-01-11T08:28:21.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.689+0000 7f33d21ff700 1 -- 172.21.15.27:0/1581019085 shutdown_connections 2024-01-11T08:28:21.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.689+0000 7f33d21ff700 1 -- 172.21.15.27:0/1581019085 >> 172.21.15.27:0/1581019085 conn(0x7f33cc06ed40 msgr2=0x7f33cc06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:21.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.690+0000 7f33d21ff700 1 -- 172.21.15.27:0/1581019085 shutdown_connections 2024-01-11T08:28:21.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.690+0000 7f33d21ff700 1 -- 172.21.15.27:0/1581019085 wait complete. 2024-01-11T08:28:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.691+0000 7f33d21ff700 1 Processor -- start 2024-01-11T08:28:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.691+0000 7f33d21ff700 1 -- start start 2024-01-11T08:28:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.692+0000 7f33d21ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b4003c20 0x7f33cc0c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.692+0000 7f33d21ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33cc15cae0 0x7f33cc14a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.692+0000 7f33d21ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33cc0c9350 0x7f33cc14b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.692+0000 7f33d21ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0c8ec0 con 0x7f33cc0c9350 2024-01-11T08:28:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.692+0000 7f33d21ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0c7fd0 con 0x7f33b4003c20 2024-01-11T08:28:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.692+0000 7f33d21ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0c82b0 con 0x7f33cc15cae0 2024-01-11T08:28:21.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.693+0000 7f33cb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b4003c20 0x7f33cc0c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:21.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.693+0000 7f33cbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33cc0c9350 0x7f33cc14b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:21.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.693+0000 7f33cbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33cc0c9350 0x7f33cc14b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34106/0 (socket says 172.21.15.27:34106) 2024-01-11T08:28:21.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.693+0000 7f33cbfff700 1 -- 172.21.15.27:0/3354086856 learned_addr learned my addr 172.21.15.27:0/3354086856 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:21.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.693+0000 7f33caffd700 1 --2- 172.21.15.27:0/3354086856 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33cc15cae0 0x7f33cc14a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:21.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.693+0000 7f33cbfff700 1 -- 172.21.15.27:0/3354086856 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33cc15cae0 msgr2=0x7f33cc14a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:21.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.693+0000 7f33cbfff700 1 --2- 172.21.15.27:0/3354086856 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33cc15cae0 0x7f33cc14a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.693+0000 7f33cbfff700 1 -- 172.21.15.27:0/3354086856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b4003c20 msgr2=0x7f33cc0c6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:21.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.693+0000 7f33cbfff700 1 --2- 172.21.15.27:0/3354086856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b4003c20 0x7f33cc0c6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.693+0000 7f33cbfff700 1 -- 172.21.15.27:0/3354086856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc0c8590 con 0x7f33cc0c9350 2024-01-11T08:28:21.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.693+0000 7f33cbfff700 1 --2- 172.21.15.27:0/3354086856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33cc0c9350 0x7f33cc14b4f0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f33cc15d3b0 tx=0x7f33c40017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:21.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.693+0000 7f33b37fe700 1 -- 172.21.15.27:0/3354086856 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c4006db0 con 0x7f33cc0c9350 2024-01-11T08:28:21.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.694+0000 7f33b37fe700 1 -- 172.21.15.27:0/3354086856 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c400c540 con 0x7f33cc0c9350 2024-01-11T08:28:21.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.694+0000 7f33b37fe700 1 -- 172.21.15.27:0/3354086856 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c400c900 con 0x7f33cc0c9350 2024-01-11T08:28:21.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.694+0000 7f33d21ff700 1 -- 172.21.15.27:0/3354086856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33cc0c8870 con 0x7f33cc0c9350 2024-01-11T08:28:21.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.694+0000 7f33d21ff700 1 -- 172.21.15.27:0/3354086856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33cc14aea0 con 0x7f33cc0c9350 2024-01-11T08:28:21.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.695+0000 7f33d21ff700 1 -- 172.21.15.27:0/3354086856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33cc0c8d10 con 0x7f33cc0c9350 2024-01-11T08:28:21.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.696+0000 7f33b37fe700 1 -- 172.21.15.27:0/3354086856 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f33c400cc70 con 0x7f33cc0c9350 2024-01-11T08:28:21.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.697+0000 7f33b37fe700 1 --2- 172.21.15.27:0/3354086856 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f33b8066f30 0x7f33b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:21.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.697+0000 7f33b37fe700 1 -- 172.21.15.27:0/3354086856 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f33c4083fc0 con 0x7f33cc0c9350 2024-01-11T08:28:21.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.697+0000 7f33cb7fe700 1 --2- 172.21.15.27:0/3354086856 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f33b8066f30 0x7f33b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:21.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.698+0000 7f33cb7fe700 1 --2- 172.21.15.27:0/3354086856 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f33b8066f30 0x7f33b80693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f33bc001120 tx=0x7f33bc015000 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:21.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.701+0000 7f33b37fe700 1 -- 172.21.15.27:0/3354086856 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f33c4051170 con 0x7f33cc0c9350 2024-01-11T08:28:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.907+0000 7f33d21ff700 1 -- 172.21.15.27:0/3354086856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33cc0b0c90 con 0x7f33cc0c9350 2024-01-11T08:28:21.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.907+0000 7f33b37fe700 1 -- 172.21.15.27:0/3354086856 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+15359 (secure 0 0 0) 0x7f33c4019090 con 0x7f33cc0c9350 2024-01-11T08:28:21.907 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:28:21.907 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":116,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:28:18.960261+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:28:15.512106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-11T08:28:16.451115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:28:21.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.910+0000 7f33d21ff700 1 -- 172.21.15.27:0/3354086856 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f33b8066f30 msgr2=0x7f33b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:21.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.910+0000 7f33d21ff700 1 --2- 172.21.15.27:0/3354086856 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f33b8066f30 0x7f33b80693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f33bc001120 tx=0x7f33bc015000 comp rx=0 tx=0).stop 2024-01-11T08:28:21.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.910+0000 7f33d21ff700 1 -- 172.21.15.27:0/3354086856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33cc0c9350 msgr2=0x7f33cc14b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:21.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.910+0000 7f33d21ff700 1 --2- 172.21.15.27:0/3354086856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33cc0c9350 0x7f33cc14b4f0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f33cc15d3b0 tx=0x7f33c40017a0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.910+0000 7f33d21ff700 1 -- 172.21.15.27:0/3354086856 shutdown_connections 2024-01-11T08:28:21.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.910+0000 7f33d21ff700 1 --2- 172.21.15.27:0/3354086856 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f33b8066f30 0x7f33b80693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.910+0000 7f33d21ff700 1 --2- 172.21.15.27:0/3354086856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33cc0c9350 0x7f33cc14b4f0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.910+0000 7f33d21ff700 1 --2- 172.21.15.27:0/3354086856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33b4003c20 0x7f33cc0c6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.910+0000 7f33d21ff700 1 --2- 172.21.15.27:0/3354086856 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33cc15cae0 0x7f33cc14a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:21.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.910+0000 7f33d21ff700 1 -- 172.21.15.27:0/3354086856 >> 172.21.15.27:0/3354086856 conn(0x7f33cc06ed40 msgr2=0x7f33cc0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:21.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.910+0000 7f33d21ff700 1 -- 172.21.15.27:0/3354086856 shutdown_connections 2024-01-11T08:28:21.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:21.910+0000 7f33d21ff700 1 -- 172.21.15.27:0/3354086856 wait complete. 2024-01-11T08:28:21.928 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-11T08:28:21.928 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:28:22.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.129+0000 7f4dc61ff700 1 Processor -- start 2024-01-11T08:28:22.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.129+0000 7f4dc61ff700 1 -- start start 2024-01-11T08:28:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.130+0000 7f4dc61ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4dc00b19d0 0x7f4dc0158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.130+0000 7f4dc61ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4dc006b610 0x7f4dc0159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.130+0000 7f4dc61ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc0158b60 con 0x7f4dc01575e0 2024-01-11T08:28:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.130+0000 7f4dc61ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc00cb5e0 con 0x7f4dc006b610 2024-01-11T08:28:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.130+0000 7f4dc61ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc01599b0 con 0x7f4dc00b19d0 2024-01-11T08:28:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.131+0000 7f4dbffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4dc006b610 0x7f4dc0159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.131+0000 7f4dbffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4dc006b610 0x7f4dc0159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34116/0 (socket says 172.21.15.27:34116) 2024-01-11T08:28:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.131+0000 7f4dbffff700 1 -- 172.21.15.27:0/4286448105 learned_addr learned my addr 172.21.15.27:0/4286448105 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.131+0000 7f4dbeffd700 1 --2- 172.21.15.27:0/4286448105 >> v2:172.21.15.77:3300/0 conn(0x7f4dc00b19d0 0x7f4dc0158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.131+0000 7f4dbffff700 1 -- 172.21.15.27:0/4286448105 >> v2:172.21.15.77:3300/0 conn(0x7f4dc00b19d0 msgr2=0x7f4dc0158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.131+0000 7f4dbffff700 1 --2- 172.21.15.27:0/4286448105 >> v2:172.21.15.77:3300/0 conn(0x7f4dc00b19d0 0x7f4dc0158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.131+0000 7f4dbffff700 1 -- 172.21.15.27:0/4286448105 >> v1:172.21.15.77:6789/0 conn(0x7f4dc01575e0 legacy=0x7f4dc00b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.131+0000 7f4dbffff700 1 -- 172.21.15.27:0/4286448105 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc00c0f90 con 0x7f4dc006b610 2024-01-11T08:28:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.131+0000 7f4dbeffd700 1 --2- 172.21.15.27:0/4286448105 >> v2:172.21.15.77:3300/0 conn(0x7f4dc00b19d0 0x7f4dc0158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.131+0000 7f4dbffff700 1 --2- 172.21.15.27:0/4286448105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dc006b610 0x7f4dc0159e40 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0064fe0 tx=0x7f4db8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c771e0ad6ce717b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:22.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.132+0000 7f4dbe7fc700 1 -- 172.21.15.27:0/4286448105 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db800e070 con 0x7f4dc006b610 2024-01-11T08:28:22.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.132+0000 7f4dbe7fc700 1 -- 172.21.15.27:0/4286448105 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db800ba50 con 0x7f4dc006b610 2024-01-11T08:28:22.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.132+0000 7f4dbe7fc700 1 -- 172.21.15.27:0/4286448105 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db80092e0 con 0x7f4dc006b610 2024-01-11T08:28:22.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.132+0000 7f4dc61ff700 1 -- 172.21.15.27:0/4286448105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dc006b610 msgr2=0x7f4dc0159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:22.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.132+0000 7f4dc61ff700 1 --2- 172.21.15.27:0/4286448105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dc006b610 0x7f4dc0159e40 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0064fe0 tx=0x7f4db8009840 comp rx=0 tx=0).stop 2024-01-11T08:28:22.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.132+0000 7f4dc61ff700 1 -- 172.21.15.27:0/4286448105 shutdown_connections 2024-01-11T08:28:22.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.132+0000 7f4dc61ff700 1 --2- 172.21.15.27:0/4286448105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dc006b610 0x7f4dc0159e40 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:22.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.132+0000 7f4dc61ff700 1 --2- 172.21.15.27:0/4286448105 >> v2:172.21.15.77:3300/0 conn(0x7f4dc00b19d0 0x7f4dc0158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:22.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.132+0000 7f4dc61ff700 1 -- 172.21.15.27:0/4286448105 >> 172.21.15.27:0/4286448105 conn(0x7f4dc00680e0 msgr2=0x7f4dc0069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:22.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.133+0000 7f4dc61ff700 1 -- 172.21.15.27:0/4286448105 shutdown_connections 2024-01-11T08:28:22.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.133+0000 7f4dc61ff700 1 -- 172.21.15.27:0/4286448105 wait complete. 2024-01-11T08:28:22.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.134+0000 7f4dc61ff700 1 Processor -- start 2024-01-11T08:28:22.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.134+0000 7f4dc61ff700 1 -- start start 2024-01-11T08:28:22.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.135+0000 7f4dc61ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dc006b610 0x7f4dc0153840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:22.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.135+0000 7f4dc61ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4dc00b19d0 0x7f4dc0169a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:22.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.135+0000 7f4dc61ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4dc01575e0 0x7f4dc0169fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:22.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.135+0000 7f4dc61ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc0154360 con 0x7f4dc006b610 2024-01-11T08:28:22.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.135+0000 7f4dc61ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc0155c60 con 0x7f4dc01575e0 2024-01-11T08:28:22.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.135+0000 7f4dc61ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc0155050 con 0x7f4dc00b19d0 2024-01-11T08:28:22.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.135+0000 7f4dbf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dc006b610 0x7f4dc0153840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:22.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.135+0000 7f4dbf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dc006b610 0x7f4dc0153840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34120/0 (socket says 172.21.15.27:34120) 2024-01-11T08:28:22.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.136+0000 7f4dbf7fe700 1 -- 172.21.15.27:0/2446319045 learned_addr learned my addr 172.21.15.27:0/2446319045 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:22.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.136+0000 7f4dbeffd700 1 --2- 172.21.15.27:0/2446319045 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4dc00b19d0 0x7f4dc0169a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:22.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.136+0000 7f4dbffff700 1 --2- 172.21.15.27:0/2446319045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4dc01575e0 0x7f4dc0169fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:22.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.136+0000 7f4dbf7fe700 1 -- 172.21.15.27:0/2446319045 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4dc00b19d0 msgr2=0x7f4dc0169a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:22.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.136+0000 7f4dbf7fe700 1 --2- 172.21.15.27:0/2446319045 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4dc00b19d0 0x7f4dc0169a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:22.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.136+0000 7f4dbf7fe700 1 -- 172.21.15.27:0/2446319045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4dc01575e0 msgr2=0x7f4dc0169fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:22.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.136+0000 7f4dbf7fe700 1 --2- 172.21.15.27:0/2446319045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4dc01575e0 0x7f4dc0169fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:22.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.136+0000 7f4dbf7fe700 1 -- 172.21.15.27:0/2446319045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db8009ae0 con 0x7f4dc006b610 2024-01-11T08:28:22.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.136+0000 7f4dbf7fe700 1 --2- 172.21.15.27:0/2446319045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dc006b610 0x7f4dc0153840 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f4db000c4e0 tx=0x7f4db000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:22.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.136+0000 7f4da77fe700 1 -- 172.21.15.27:0/2446319045 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db0011590 con 0x7f4dc006b610 2024-01-11T08:28:22.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.137+0000 7f4da77fe700 1 -- 172.21.15.27:0/2446319045 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db0011710 con 0x7f4dc006b610 2024-01-11T08:28:22.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.137+0000 7f4da77fe700 1 -- 172.21.15.27:0/2446319045 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db0011ad0 con 0x7f4dc006b610 2024-01-11T08:28:22.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.137+0000 7f4dc61ff700 1 -- 172.21.15.27:0/2446319045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc0155330 con 0x7f4dc006b610 2024-01-11T08:28:22.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.137+0000 7f4dc61ff700 1 -- 172.21.15.27:0/2446319045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc01558f0 con 0x7f4dc006b610 2024-01-11T08:28:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.138+0000 7f4dc61ff700 1 -- 172.21.15.27:0/2446319045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc00718d0 con 0x7f4dc006b610 2024-01-11T08:28:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.139+0000 7f4da77fe700 1 -- 172.21.15.27:0/2446319045 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4db001b3b0 con 0x7f4dc006b610 2024-01-11T08:28:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.140+0000 7f4da77fe700 1 --2- 172.21.15.27:0/2446319045 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4dac066f30 0x7f4dac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.140+0000 7f4da77fe700 1 -- 172.21.15.27:0/2446319045 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f4db0083710 con 0x7f4dc006b610 2024-01-11T08:28:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.140+0000 7f4dbeffd700 1 --2- 172.21.15.27:0/2446319045 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4dac066f30 0x7f4dac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.141+0000 7f4dbeffd700 1 --2- 172.21.15.27:0/2446319045 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4dac066f30 0x7f4dac0693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00c5770 tx=0x7f4db4008040 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:22.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.144+0000 7f4da77fe700 1 -- 172.21.15.27:0/2446319045 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4db0050960 con 0x7f4dc006b610 2024-01-11T08:28:22.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.344+0000 7f4dc61ff700 1 -- 172.21.15.27:0/2446319045 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4dc0158b60 con 0x7f4dac066f30 2024-01-11T08:28:22.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.355+0000 7f4da77fe700 1 -- 172.21.15.27:0/2446319045 <== mgr.7177 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289727 (secure 0 0 0) 0x7f4dc0158b60 con 0x7f4dac066f30 2024-01-11T08:28:22.359 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:28:22.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.359+0000 7f4dc61ff700 1 -- 172.21.15.27:0/2446319045 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4dac066f30 msgr2=0x7f4dac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:22.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.359+0000 7f4dc61ff700 1 --2- 172.21.15.27:0/2446319045 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4dac066f30 0x7f4dac0693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00c5770 tx=0x7f4db4008040 comp rx=0 tx=0).stop 2024-01-11T08:28:22.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.359+0000 7f4dc61ff700 1 -- 172.21.15.27:0/2446319045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dc006b610 msgr2=0x7f4dc0153840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:22.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.359+0000 7f4dc61ff700 1 --2- 172.21.15.27:0/2446319045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dc006b610 0x7f4dc0153840 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f4db000c4e0 tx=0x7f4db000c7f0 comp rx=0 tx=0).stop 2024-01-11T08:28:22.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.360+0000 7f4dc61ff700 1 -- 172.21.15.27:0/2446319045 shutdown_connections 2024-01-11T08:28:22.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.360+0000 7f4dc61ff700 1 --2- 172.21.15.27:0/2446319045 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4dac066f30 0x7f4dac0693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:22.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.360+0000 7f4dc61ff700 1 --2- 172.21.15.27:0/2446319045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4dc01575e0 0x7f4dc0169fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:22.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.360+0000 7f4dc61ff700 1 --2- 172.21.15.27:0/2446319045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dc006b610 0x7f4dc0153840 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:22.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.360+0000 7f4dc61ff700 1 --2- 172.21.15.27:0/2446319045 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4dc00b19d0 0x7f4dc0169a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:22.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.360+0000 7f4dc61ff700 1 -- 172.21.15.27:0/2446319045 >> 172.21.15.27:0/2446319045 conn(0x7f4dc00680e0 msgr2=0x7f4dc015a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:22.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.360+0000 7f4dc61ff700 1 -- 172.21.15.27:0/2446319045 shutdown_connections 2024-01-11T08:28:22.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.360+0000 7f4dc61ff700 1 -- 172.21.15.27:0/2446319045 wait complete. 2024-01-11T08:28:22.362 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:28:22.377 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-11T08:28:22.077222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1040,"num_read_kb":2130,"num_write":395,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":261,"ondisk_log_size":261,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":152,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2443976,"kb_used_data":12424,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752530744,"statfs":{"total":773094113280,"available":770591481856,"internally_reserved":0,"allocated":12722176,"data_stored":6516116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001285"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.990268+0000","last_change":"2024-01-11T08:28:17.984391+0000","last_active":"2024-01-11T08:28:17.990268+0000","last_peered":"2024-01-11T08:28:17.990268+0000","last_clean":"2024-01-11T08:28:17.990268+0000","last_became_active":"2024-01-11T08:28:17.984143+0000","last_became_peered":"2024-01-11T08:28:17.984143+0000","last_unstale":"2024-01-11T08:28:17.990268+0000","last_undegraded":"2024-01-11T08:28:17.990268+0000","last_fullsized":"2024-01-11T08:28:17.990268+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:52:47.412887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.987135+0000","last_change":"2024-01-11T08:28:17.985653+0000","last_active":"2024-01-11T08:28:17.987135+0000","last_peered":"2024-01-11T08:28:17.987135+0000","last_clean":"2024-01-11T08:28:17.987135+0000","last_became_active":"2024-01-11T08:28:17.985140+0000","last_became_peered":"2024-01-11T08:28:17.985140+0000","last_unstale":"2024-01-11T08:28:17.987135+0000","last_undegraded":"2024-01-11T08:28:17.987135+0000","last_fullsized":"2024-01-11T08:28:17.987135+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:01:19.046780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.992281+0000","last_change":"2024-01-11T08:28:17.988746+0000","last_active":"2024-01-11T08:28:17.992281+0000","last_peered":"2024-01-11T08:28:17.992281+0000","last_clean":"2024-01-11T08:28:17.992281+0000","last_became_active":"2024-01-11T08:28:17.987819+0000","last_became_peered":"2024-01-11T08:28:17.987819+0000","last_unstale":"2024-01-11T08:28:17.992281+0000","last_undegraded":"2024-01-11T08:28:17.992281+0000","last_fullsized":"2024-01-11T08:28:17.992281+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:17:37.552135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.986492+0000","last_change":"2024-01-11T08:28:17.982122+0000","last_active":"2024-01-11T08:28:17.986492+0000","last_peered":"2024-01-11T08:28:17.986492+0000","last_clean":"2024-01-11T08:28:17.986492+0000","last_became_active":"2024-01-11T08:28:17.981742+0000","last_became_peered":"2024-01-11T08:28:17.981742+0000","last_unstale":"2024-01-11T08:28:17.986492+0000","last_undegraded":"2024-01-11T08:28:17.986492+0000","last_fullsized":"2024-01-11T08:28:17.986492+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:12:51.077671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.977919+0000","last_change":"2024-01-11T08:28:17.975351+0000","last_active":"2024-01-11T08:28:17.977919+0000","last_peered":"2024-01-11T08:28:17.977919+0000","last_clean":"2024-01-11T08:28:17.977919+0000","last_became_active":"2024-01-11T08:28:17.974419+0000","last_became_peered":"2024-01-11T08:28:17.974419+0000","last_unstale":"2024-01-11T08:28:17.977919+0000","last_undegraded":"2024-01-11T08:28:17.977919+0000","last_fullsized":"2024-01-11T08:28:17.977919+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:40:34.768302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.986188+0000","last_change":"2024-01-11T08:28:17.984079+0000","last_active":"2024-01-11T08:28:17.986188+0000","last_peered":"2024-01-11T08:28:17.986188+0000","last_clean":"2024-01-11T08:28:17.986188+0000","last_became_active":"2024-01-11T08:28:17.983716+0000","last_became_peered":"2024-01-11T08:28:17.983716+0000","last_unstale":"2024-01-11T08:28:17.986188+0000","last_undegraded":"2024-01-11T08:28:17.986188+0000","last_fullsized":"2024-01-11T08:28:17.986188+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:35:44.632717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.985143+0000","last_change":"2024-01-11T08:28:17.982880+0000","last_active":"2024-01-11T08:28:17.985143+0000","last_peered":"2024-01-11T08:28:17.985143+0000","last_clean":"2024-01-11T08:28:17.985143+0000","last_became_active":"2024-01-11T08:28:17.982557+0000","last_became_peered":"2024-01-11T08:28:17.982557+0000","last_unstale":"2024-01-11T08:28:17.985143+0000","last_undegraded":"2024-01-11T08:28:17.985143+0000","last_fullsized":"2024-01-11T08:28:17.985143+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:36:53.350082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.976053+0000","last_change":"2024-01-11T08:28:17.968873+0000","last_active":"2024-01-11T08:28:17.976053+0000","last_peered":"2024-01-11T08:28:17.976053+0000","last_clean":"2024-01-11T08:28:17.976053+0000","last_became_active":"2024-01-11T08:28:17.968506+0000","last_became_peered":"2024-01-11T08:28:17.968506+0000","last_unstale":"2024-01-11T08:28:17.976053+0000","last_undegraded":"2024-01-11T08:28:17.976053+0000","last_fullsized":"2024-01-11T08:28:17.976053+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:08:30.335895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.970953+0000","last_change":"2024-01-11T08:28:17.968244+0000","last_active":"2024-01-11T08:28:17.970953+0000","last_peered":"2024-01-11T08:28:17.970953+0000","last_clean":"2024-01-11T08:28:17.970953+0000","last_became_active":"2024-01-11T08:28:17.966994+0000","last_became_peered":"2024-01-11T08:28:17.966994+0000","last_unstale":"2024-01-11T08:28:17.970953+0000","last_undegraded":"2024-01-11T08:28:17.970953+0000","last_fullsized":"2024-01-11T08:28:17.970953+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:51:42.562966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.978277+0000","last_change":"2024-01-11T08:28:17.976009+0000","last_active":"2024-01-11T08:28:17.978277+0000","last_peered":"2024-01-11T08:28:17.978277+0000","last_clean":"2024-01-11T08:28:17.978277+0000","last_became_active":"2024-01-11T08:28:17.975719+0000","last_became_peered":"2024-01-11T08:28:17.975719+0000","last_unstale":"2024-01-11T08:28:17.978277+0000","last_undegraded":"2024-01-11T08:28:17.978277+0000","last_fullsized":"2024-01-11T08:28:17.978277+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:02:38.249774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.970437+0000","last_change":"2024-01-11T08:28:17.967873+0000","last_active":"2024-01-11T08:28:17.970437+0000","last_peered":"2024-01-11T08:28:17.970437+0000","last_clean":"2024-01-11T08:28:17.970437+0000","last_became_active":"2024-01-11T08:28:17.966819+0000","last_became_peered":"2024-01-11T08:28:17.966819+0000","last_unstale":"2024-01-11T08:28:17.970437+0000","last_undegraded":"2024-01-11T08:28:17.970437+0000","last_fullsized":"2024-01-11T08:28:17.970437+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:01:23.824852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.975465+0000","last_change":"2024-01-11T08:28:17.972272+0000","last_active":"2024-01-11T08:28:17.975465+0000","last_peered":"2024-01-11T08:28:17.975465+0000","last_clean":"2024-01-11T08:28:17.975465+0000","last_became_active":"2024-01-11T08:28:17.971704+0000","last_became_peered":"2024-01-11T08:28:17.971704+0000","last_unstale":"2024-01-11T08:28:17.975465+0000","last_undegraded":"2024-01-11T08:28:17.975465+0000","last_fullsized":"2024-01-11T08:28:17.975465+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:01:19.046780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.968779+0000","last_change":"2024-01-11T08:28:17.967441+0000","last_active":"2024-01-11T08:28:17.968779+0000","last_peered":"2024-01-11T08:28:17.968779+0000","last_clean":"2024-01-11T08:28:17.968779+0000","last_became_active":"2024-01-11T08:28:17.966709+0000","last_became_peered":"2024-01-11T08:28:17.966709+0000","last_unstale":"2024-01-11T08:28:17.968779+0000","last_undegraded":"2024-01-11T08:28:17.968779+0000","last_fullsized":"2024-01-11T08:28:17.968779+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:19:53.198221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.984566+0000","last_change":"2024-01-11T08:28:17.977606+0000","last_active":"2024-01-11T08:28:17.984566+0000","last_peered":"2024-01-11T08:28:17.984566+0000","last_clean":"2024-01-11T08:28:17.984566+0000","last_became_active":"2024-01-11T08:28:17.977377+0000","last_became_peered":"2024-01-11T08:28:17.977377+0000","last_unstale":"2024-01-11T08:28:17.984566+0000","last_undegraded":"2024-01-11T08:28:17.984566+0000","last_fullsized":"2024-01-11T08:28:17.984566+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:07:27.285241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.977164+0000","last_change":"2024-01-11T08:28:17.972733+0000","last_active":"2024-01-11T08:28:17.977164+0000","last_peered":"2024-01-11T08:28:17.977164+0000","last_clean":"2024-01-11T08:28:17.977164+0000","last_became_active":"2024-01-11T08:28:17.972154+0000","last_became_peered":"2024-01-11T08:28:17.972154+0000","last_unstale":"2024-01-11T08:28:17.977164+0000","last_undegraded":"2024-01-11T08:28:17.977164+0000","last_fullsized":"2024-01-11T08:28:17.977164+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:28:37.430387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.973281+0000","last_change":"2024-01-11T08:28:17.970813+0000","last_active":"2024-01-11T08:28:17.973281+0000","last_peered":"2024-01-11T08:28:17.973281+0000","last_clean":"2024-01-11T08:28:17.973281+0000","last_became_active":"2024-01-11T08:28:17.969930+0000","last_became_peered":"2024-01-11T08:28:17.969930+0000","last_unstale":"2024-01-11T08:28:17.973281+0000","last_undegraded":"2024-01-11T08:28:17.973281+0000","last_fullsized":"2024-01-11T08:28:17.973281+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:53:55.216533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.977834+0000","last_change":"2024-01-11T08:28:17.970347+0000","last_active":"2024-01-11T08:28:17.977834+0000","last_peered":"2024-01-11T08:28:17.977834+0000","last_clean":"2024-01-11T08:28:17.977834+0000","last_became_active":"2024-01-11T08:28:17.970104+0000","last_became_peered":"2024-01-11T08:28:17.970104+0000","last_unstale":"2024-01-11T08:28:17.977834+0000","last_undegraded":"2024-01-11T08:28:17.977834+0000","last_fullsized":"2024-01-11T08:28:17.977834+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:39:18.705936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.992320+0000","last_change":"2024-01-11T08:28:17.986999+0000","last_active":"2024-01-11T08:28:17.992320+0000","last_peered":"2024-01-11T08:28:17.992320+0000","last_clean":"2024-01-11T08:28:17.992320+0000","last_became_active":"2024-01-11T08:28:17.986574+0000","last_became_peered":"2024-01-11T08:28:17.986574+0000","last_unstale":"2024-01-11T08:28:17.992320+0000","last_undegraded":"2024-01-11T08:28:17.992320+0000","last_fullsized":"2024-01-11T08:28:17.992320+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:35:44.632717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.977070+0000","last_change":"2024-01-11T08:28:17.971433+0000","last_active":"2024-01-11T08:28:17.977070+0000","last_peered":"2024-01-11T08:28:17.977070+0000","last_clean":"2024-01-11T08:28:17.977070+0000","last_became_active":"2024-01-11T08:28:17.971179+0000","last_became_peered":"2024-01-11T08:28:17.971179+0000","last_unstale":"2024-01-11T08:28:17.977070+0000","last_undegraded":"2024-01-11T08:28:17.977070+0000","last_fullsized":"2024-01-11T08:28:17.977070+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:37:35.976533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.975441+0000","last_change":"2024-01-11T08:28:17.972037+0000","last_active":"2024-01-11T08:28:17.975441+0000","last_peered":"2024-01-11T08:28:17.975441+0000","last_clean":"2024-01-11T08:28:17.975441+0000","last_became_active":"2024-01-11T08:28:17.971177+0000","last_became_peered":"2024-01-11T08:28:17.971177+0000","last_unstale":"2024-01-11T08:28:17.975441+0000","last_undegraded":"2024-01-11T08:28:17.975441+0000","last_fullsized":"2024-01-11T08:28:17.975441+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:40:34.768302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.984626+0000","last_change":"2024-01-11T08:28:17.976823+0000","last_active":"2024-01-11T08:28:17.984626+0000","last_peered":"2024-01-11T08:28:17.984626+0000","last_clean":"2024-01-11T08:28:17.984626+0000","last_became_active":"2024-01-11T08:28:17.976534+0000","last_became_peered":"2024-01-11T08:28:17.976534+0000","last_unstale":"2024-01-11T08:28:17.984626+0000","last_undegraded":"2024-01-11T08:28:17.984626+0000","last_fullsized":"2024-01-11T08:28:17.984626+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:48:49.108760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.984077+0000","last_change":"2024-01-11T08:28:17.982228+0000","last_active":"2024-01-11T08:28:17.984077+0000","last_peered":"2024-01-11T08:28:17.984077+0000","last_clean":"2024-01-11T08:28:17.984077+0000","last_became_active":"2024-01-11T08:28:17.981749+0000","last_became_peered":"2024-01-11T08:28:17.981749+0000","last_unstale":"2024-01-11T08:28:17.984077+0000","last_undegraded":"2024-01-11T08:28:17.984077+0000","last_fullsized":"2024-01-11T08:28:17.984077+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:20:14.569050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.983138+0000","last_change":"2024-01-11T08:28:17.982015+0000","last_active":"2024-01-11T08:28:17.983138+0000","last_peered":"2024-01-11T08:28:17.983138+0000","last_clean":"2024-01-11T08:28:17.983138+0000","last_became_active":"2024-01-11T08:28:17.981770+0000","last_became_peered":"2024-01-11T08:28:17.981770+0000","last_unstale":"2024-01-11T08:28:17.983138+0000","last_undegraded":"2024-01-11T08:28:17.983138+0000","last_fullsized":"2024-01-11T08:28:17.983138+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:11:37.602642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.973493+0000","last_change":"2024-01-11T08:28:17.972434+0000","last_active":"2024-01-11T08:28:17.973493+0000","last_peered":"2024-01-11T08:28:17.973493+0000","last_clean":"2024-01-11T08:28:17.973493+0000","last_became_active":"2024-01-11T08:28:17.972139+0000","last_became_peered":"2024-01-11T08:28:17.972139+0000","last_unstale":"2024-01-11T08:28:17.973493+0000","last_undegraded":"2024-01-11T08:28:17.973493+0000","last_fullsized":"2024-01-11T08:28:17.973493+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:33:46.515399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.981093+0000","last_change":"2024-01-11T08:28:17.978145+0000","last_active":"2024-01-11T08:28:17.981093+0000","last_peered":"2024-01-11T08:28:17.981093+0000","last_clean":"2024-01-11T08:28:17.981093+0000","last_became_active":"2024-01-11T08:28:17.977860+0000","last_became_peered":"2024-01-11T08:28:17.977860+0000","last_unstale":"2024-01-11T08:28:17.981093+0000","last_undegraded":"2024-01-11T08:28:17.981093+0000","last_fullsized":"2024-01-11T08:28:17.981093+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:51:42.562966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.979436+0000","last_change":"2024-01-11T08:28:17.976159+0000","last_active":"2024-01-11T08:28:17.979436+0000","last_peered":"2024-01-11T08:28:17.979436+0000","last_clean":"2024-01-11T08:28:17.979436+0000","last_became_active":"2024-01-11T08:28:17.975867+0000","last_became_peered":"2024-01-11T08:28:17.975867+0000","last_unstale":"2024-01-11T08:28:17.979436+0000","last_undegraded":"2024-01-11T08:28:17.979436+0000","last_fullsized":"2024-01-11T08:28:17.979436+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:47:38.042745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.971700+0000","last_change":"2024-01-11T08:28:17.968397+0000","last_active":"2024-01-11T08:28:17.971700+0000","last_peered":"2024-01-11T08:28:17.971700+0000","last_clean":"2024-01-11T08:28:17.971700+0000","last_became_active":"2024-01-11T08:28:17.967576+0000","last_became_peered":"2024-01-11T08:28:17.967576+0000","last_unstale":"2024-01-11T08:28:17.971700+0000","last_undegraded":"2024-01-11T08:28:17.971700+0000","last_fullsized":"2024-01-11T08:28:17.971700+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:09:13.812238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.980233+0000","last_change":"2024-01-11T08:28:17.977400+0000","last_active":"2024-01-11T08:28:17.980233+0000","last_peered":"2024-01-11T08:28:17.980233+0000","last_clean":"2024-01-11T08:28:17.980233+0000","last_became_active":"2024-01-11T08:28:17.977064+0000","last_became_peered":"2024-01-11T08:28:17.977064+0000","last_unstale":"2024-01-11T08:28:17.980233+0000","last_undegraded":"2024-01-11T08:28:17.980233+0000","last_fullsized":"2024-01-11T08:28:17.980233+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:01:19.046780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.977534+0000","last_change":"2024-01-11T08:28:17.974036+0000","last_active":"2024-01-11T08:28:17.977534+0000","last_peered":"2024-01-11T08:28:17.977534+0000","last_clean":"2024-01-11T08:28:17.977534+0000","last_became_active":"2024-01-11T08:28:17.973756+0000","last_became_peered":"2024-01-11T08:28:17.973756+0000","last_unstale":"2024-01-11T08:28:17.977534+0000","last_undegraded":"2024-01-11T08:28:17.977534+0000","last_fullsized":"2024-01-11T08:28:17.977534+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:40:34.768302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.981315+0000","last_change":"2024-01-11T08:28:17.973583+0000","last_active":"2024-01-11T08:28:17.981315+0000","last_peered":"2024-01-11T08:28:17.981315+0000","last_clean":"2024-01-11T08:28:17.981315+0000","last_became_active":"2024-01-11T08:28:17.973043+0000","last_became_peered":"2024-01-11T08:28:17.973043+0000","last_unstale":"2024-01-11T08:28:17.981315+0000","last_undegraded":"2024-01-11T08:28:17.981315+0000","last_fullsized":"2024-01-11T08:28:17.981315+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:53:27.932566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.980408+0000","last_change":"2024-01-11T08:28:17.978364+0000","last_active":"2024-01-11T08:28:17.980408+0000","last_peered":"2024-01-11T08:28:17.980408+0000","last_clean":"2024-01-11T08:28:17.980408+0000","last_became_active":"2024-01-11T08:28:17.977163+0000","last_became_peered":"2024-01-11T08:28:17.977163+0000","last_unstale":"2024-01-11T08:28:17.980408+0000","last_undegraded":"2024-01-11T08:28:17.980408+0000","last_fullsized":"2024-01-11T08:28:17.980408+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:36:53.350082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.974534+0000","last_change":"2024-01-11T08:28:17.972309+0000","last_active":"2024-01-11T08:28:17.974534+0000","last_peered":"2024-01-11T08:28:17.974534+0000","last_clean":"2024-01-11T08:28:17.974534+0000","last_became_active":"2024-01-11T08:28:17.972002+0000","last_became_peered":"2024-01-11T08:28:17.972002+0000","last_unstale":"2024-01-11T08:28:17.974534+0000","last_undegraded":"2024-01-11T08:28:17.974534+0000","last_fullsized":"2024-01-11T08:28:17.974534+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:57:08.672492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.975485+0000","last_change":"2024-01-11T08:28:17.968542+0000","last_active":"2024-01-11T08:28:17.975485+0000","last_peered":"2024-01-11T08:28:17.975485+0000","last_clean":"2024-01-11T08:28:17.975485+0000","last_became_active":"2024-01-11T08:28:17.967756+0000","last_became_peered":"2024-01-11T08:28:17.967756+0000","last_unstale":"2024-01-11T08:28:17.975485+0000","last_undegraded":"2024-01-11T08:28:17.975485+0000","last_fullsized":"2024-01-11T08:28:17.975485+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:22:27.717871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"115'4","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.991569+0000","last_change":"2024-01-11T08:28:16.970927+0000","last_active":"2024-01-11T08:28:17.991569+0000","last_peered":"2024-01-11T08:28:17.991569+0000","last_clean":"2024-01-11T08:28:17.991569+0000","last_became_active":"2024-01-11T08:28:16.970654+0000","last_became_peered":"2024-01-11T08:28:16.970654+0000","last_unstale":"2024-01-11T08:28:17.991569+0000","last_undegraded":"2024-01-11T08:28:17.991569+0000","last_fullsized":"2024-01-11T08:28:17.991569+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:32:21.280368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.980349+0000","last_change":"2024-01-11T08:28:17.977574+0000","last_active":"2024-01-11T08:28:17.980349+0000","last_peered":"2024-01-11T08:28:17.980349+0000","last_clean":"2024-01-11T08:28:17.980349+0000","last_became_active":"2024-01-11T08:28:17.977321+0000","last_became_peered":"2024-01-11T08:28:17.977321+0000","last_unstale":"2024-01-11T08:28:17.980349+0000","last_undegraded":"2024-01-11T08:28:17.980349+0000","last_fullsized":"2024-01-11T08:28:17.980349+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:09:13.812238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"115'1","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.987119+0000","last_change":"2024-01-11T08:28:16.969153+0000","last_active":"2024-01-11T08:28:17.987119+0000","last_peered":"2024-01-11T08:28:17.987119+0000","last_clean":"2024-01-11T08:28:17.987119+0000","last_became_active":"2024-01-11T08:28:16.968902+0000","last_became_peered":"2024-01-11T08:28:16.968902+0000","last_unstale":"2024-01-11T08:28:17.987119+0000","last_undegraded":"2024-01-11T08:28:17.987119+0000","last_fullsized":"2024-01-11T08:28:17.987119+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:41:30.011035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.972877+0000","last_change":"2024-01-11T08:28:17.969705+0000","last_active":"2024-01-11T08:28:17.972877+0000","last_peered":"2024-01-11T08:28:17.972877+0000","last_clean":"2024-01-11T08:28:17.972877+0000","last_became_active":"2024-01-11T08:28:17.969348+0000","last_became_peered":"2024-01-11T08:28:17.969348+0000","last_unstale":"2024-01-11T08:28:17.972877+0000","last_undegraded":"2024-01-11T08:28:17.972877+0000","last_fullsized":"2024-01-11T08:28:17.972877+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:40:29.251257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"115'2","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.990448+0000","last_change":"2024-01-11T08:28:16.975283+0000","last_active":"2024-01-11T08:28:17.990448+0000","last_peered":"2024-01-11T08:28:17.990448+0000","last_clean":"2024-01-11T08:28:17.990448+0000","last_became_active":"2024-01-11T08:28:16.973391+0000","last_became_peered":"2024-01-11T08:28:16.973391+0000","last_unstale":"2024-01-11T08:28:17.990448+0000","last_undegraded":"2024-01-11T08:28:17.990448+0000","last_fullsized":"2024-01-11T08:28:17.990448+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:01:17.535401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.968424+0000","last_change":"2024-01-11T08:28:17.965959+0000","last_active":"2024-01-11T08:28:17.968424+0000","last_peered":"2024-01-11T08:28:17.968424+0000","last_clean":"2024-01-11T08:28:17.968424+0000","last_became_active":"2024-01-11T08:28:17.965502+0000","last_became_peered":"2024-01-11T08:28:17.965502+0000","last_unstale":"2024-01-11T08:28:17.968424+0000","last_undegraded":"2024-01-11T08:28:17.968424+0000","last_fullsized":"2024-01-11T08:28:17.968424+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:42:56.802832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"115'2","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.987287+0000","last_change":"2024-01-11T08:28:16.970018+0000","last_active":"2024-01-11T08:28:17.987287+0000","last_peered":"2024-01-11T08:28:17.987287+0000","last_clean":"2024-01-11T08:28:17.987287+0000","last_became_active":"2024-01-11T08:28:16.968638+0000","last_became_peered":"2024-01-11T08:28:16.968638+0000","last_unstale":"2024-01-11T08:28:17.987287+0000","last_undegraded":"2024-01-11T08:28:17.987287+0000","last_fullsized":"2024-01-11T08:28:17.987287+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:18:42.004051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.980659+0000","last_change":"2024-01-11T08:28:17.967920+0000","last_active":"2024-01-11T08:28:17.980659+0000","last_peered":"2024-01-11T08:28:17.980659+0000","last_clean":"2024-01-11T08:28:17.980659+0000","last_became_active":"2024-01-11T08:28:17.967637+0000","last_became_peered":"2024-01-11T08:28:17.967637+0000","last_unstale":"2024-01-11T08:28:17.980659+0000","last_undegraded":"2024-01-11T08:28:17.980659+0000","last_fullsized":"2024-01-11T08:28:17.980659+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:23:39.775217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"115'1","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.992178+0000","last_change":"2024-01-11T08:28:16.965022+0000","last_active":"2024-01-11T08:28:17.992178+0000","last_peered":"2024-01-11T08:28:17.992178+0000","last_clean":"2024-01-11T08:28:17.992178+0000","last_became_active":"2024-01-11T08:28:16.964707+0000","last_became_peered":"2024-01-11T08:28:16.964707+0000","last_unstale":"2024-01-11T08:28:17.992178+0000","last_undegraded":"2024-01-11T08:28:17.992178+0000","last_fullsized":"2024-01-11T08:28:17.992178+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:17:29.823696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.980450+0000","last_change":"2024-01-11T08:28:17.977960+0000","last_active":"2024-01-11T08:28:17.980450+0000","last_peered":"2024-01-11T08:28:17.980450+0000","last_clean":"2024-01-11T08:28:17.980450+0000","last_became_active":"2024-01-11T08:28:17.977596+0000","last_became_peered":"2024-01-11T08:28:17.977596+0000","last_unstale":"2024-01-11T08:28:17.980450+0000","last_undegraded":"2024-01-11T08:28:17.980450+0000","last_fullsized":"2024-01-11T08:28:17.980450+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:51:46.671037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"115'1","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.990500+0000","last_change":"2024-01-11T08:28:16.975205+0000","last_active":"2024-01-11T08:28:17.990500+0000","last_peered":"2024-01-11T08:28:17.990500+0000","last_clean":"2024-01-11T08:28:17.990500+0000","last_became_active":"2024-01-11T08:28:16.974114+0000","last_became_peered":"2024-01-11T08:28:16.974114+0000","last_unstale":"2024-01-11T08:28:17.990500+0000","last_undegraded":"2024-01-11T08:28:17.990500+0000","last_fullsized":"2024-01-11T08:28:17.990500+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:26:59.400728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.986543+0000","last_change":"2024-01-11T08:28:17.982333+0000","last_active":"2024-01-11T08:28:17.986543+0000","last_peered":"2024-01-11T08:28:17.986543+0000","last_clean":"2024-01-11T08:28:17.986543+0000","last_became_active":"2024-01-11T08:28:17.976709+0000","last_became_peered":"2024-01-11T08:28:17.976709+0000","last_unstale":"2024-01-11T08:28:17.986543+0000","last_undegraded":"2024-01-11T08:28:17.986543+0000","last_fullsized":"2024-01-11T08:28:17.986543+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:18:43.010725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.961779+0000","last_change":"2024-01-11T08:28:16.968940+0000","last_active":"2024-01-11T08:28:17.961779+0000","last_peered":"2024-01-11T08:28:17.961779+0000","last_clean":"2024-01-11T08:28:17.961779+0000","last_became_active":"2024-01-11T08:28:16.968628+0000","last_became_peered":"2024-01-11T08:28:16.968628+0000","last_unstale":"2024-01-11T08:28:17.961779+0000","last_undegraded":"2024-01-11T08:28:17.961779+0000","last_fullsized":"2024-01-11T08:28:17.961779+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:10:07.272755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.965293+0000","last_change":"2024-01-11T08:28:17.961408+0000","last_active":"2024-01-11T08:28:17.965293+0000","last_peered":"2024-01-11T08:28:17.965293+0000","last_clean":"2024-01-11T08:28:17.965293+0000","last_became_active":"2024-01-11T08:28:17.961001+0000","last_became_peered":"2024-01-11T08:28:17.961001+0000","last_unstale":"2024-01-11T08:28:17.965293+0000","last_undegraded":"2024-01-11T08:28:17.965293+0000","last_fullsized":"2024-01-11T08:28:17.965293+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:02:38.249774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.961275+0000","last_change":"2024-01-11T08:28:16.970181+0000","last_active":"2024-01-11T08:28:17.961275+0000","last_peered":"2024-01-11T08:28:17.961275+0000","last_clean":"2024-01-11T08:28:17.961275+0000","last_became_active":"2024-01-11T08:28:16.969099+0000","last_became_peered":"2024-01-11T08:28:16.969099+0000","last_unstale":"2024-01-11T08:28:17.961275+0000","last_undegraded":"2024-01-11T08:28:17.961275+0000","last_fullsized":"2024-01-11T08:28:17.961275+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:07:55.263842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.979456+0000","last_change":"2024-01-11T08:28:17.976644+0000","last_active":"2024-01-11T08:28:17.979456+0000","last_peered":"2024-01-11T08:28:17.979456+0000","last_clean":"2024-01-11T08:28:17.979456+0000","last_became_active":"2024-01-11T08:28:17.976392+0000","last_became_peered":"2024-01-11T08:28:17.976392+0000","last_unstale":"2024-01-11T08:28:17.979456+0000","last_undegraded":"2024-01-11T08:28:17.979456+0000","last_fullsized":"2024-01-11T08:28:17.979456+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:32:23.513567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.959453+0000","last_change":"2024-01-11T08:28:16.966688+0000","last_active":"2024-01-11T08:28:17.959453+0000","last_peered":"2024-01-11T08:28:17.959453+0000","last_clean":"2024-01-11T08:28:17.959453+0000","last_became_active":"2024-01-11T08:28:16.966425+0000","last_became_peered":"2024-01-11T08:28:16.966425+0000","last_unstale":"2024-01-11T08:28:17.959453+0000","last_undegraded":"2024-01-11T08:28:17.959453+0000","last_fullsized":"2024-01-11T08:28:17.959453+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:55:17.169420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.963547+0000","last_change":"2024-01-11T08:28:17.958893+0000","last_active":"2024-01-11T08:28:17.963547+0000","last_peered":"2024-01-11T08:28:17.963547+0000","last_clean":"2024-01-11T08:28:17.963547+0000","last_became_active":"2024-01-11T08:28:17.958617+0000","last_became_peered":"2024-01-11T08:28:17.958617+0000","last_unstale":"2024-01-11T08:28:17.963547+0000","last_undegraded":"2024-01-11T08:28:17.963547+0000","last_fullsized":"2024-01-11T08:28:17.963547+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:53:55.216533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.952974+0000","last_change":"2024-01-11T08:28:16.966676+0000","last_active":"2024-01-11T08:28:17.952974+0000","last_peered":"2024-01-11T08:28:17.952974+0000","last_clean":"2024-01-11T08:28:17.952974+0000","last_became_active":"2024-01-11T08:28:16.966424+0000","last_became_peered":"2024-01-11T08:28:16.966424+0000","last_unstale":"2024-01-11T08:28:17.952974+0000","last_undegraded":"2024-01-11T08:28:17.952974+0000","last_fullsized":"2024-01-11T08:28:17.952974+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:47:11.453713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.981090+0000","last_change":"2024-01-11T08:28:17.977899+0000","last_active":"2024-01-11T08:28:17.981090+0000","last_peered":"2024-01-11T08:28:17.981090+0000","last_clean":"2024-01-11T08:28:17.981090+0000","last_became_active":"2024-01-11T08:28:17.977616+0000","last_became_peered":"2024-01-11T08:28:17.977616+0000","last_unstale":"2024-01-11T08:28:17.981090+0000","last_undegraded":"2024-01-11T08:28:17.981090+0000","last_fullsized":"2024-01-11T08:28:17.981090+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:53:55.216533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"115'3","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.988758+0000","last_change":"2024-01-11T08:28:16.968103+0000","last_active":"2024-01-11T08:28:17.988758+0000","last_peered":"2024-01-11T08:28:17.988758+0000","last_clean":"2024-01-11T08:28:17.988758+0000","last_became_active":"2024-01-11T08:28:16.967466+0000","last_became_peered":"2024-01-11T08:28:16.967466+0000","last_unstale":"2024-01-11T08:28:17.988758+0000","last_undegraded":"2024-01-11T08:28:17.988758+0000","last_fullsized":"2024-01-11T08:28:17.988758+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:47:20.068330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.983817+0000","last_change":"2024-01-11T08:28:17.981727+0000","last_active":"2024-01-11T08:28:17.983817+0000","last_peered":"2024-01-11T08:28:17.983817+0000","last_clean":"2024-01-11T08:28:17.983817+0000","last_became_active":"2024-01-11T08:28:17.981277+0000","last_became_peered":"2024-01-11T08:28:17.981277+0000","last_unstale":"2024-01-11T08:28:17.983817+0000","last_undegraded":"2024-01-11T08:28:17.983817+0000","last_fullsized":"2024-01-11T08:28:17.983817+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:02:38.249774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"115'6","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.995227+0000","last_change":"2024-01-11T08:28:16.970694+0000","last_active":"2024-01-11T08:28:17.995227+0000","last_peered":"2024-01-11T08:28:17.995227+0000","last_clean":"2024-01-11T08:28:17.995227+0000","last_became_active":"2024-01-11T08:28:16.970351+0000","last_became_peered":"2024-01-11T08:28:16.970351+0000","last_unstale":"2024-01-11T08:28:17.995227+0000","last_undegraded":"2024-01-11T08:28:17.995227+0000","last_fullsized":"2024-01-11T08:28:17.995227+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:18:42.004051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.967325+0000","last_change":"2024-01-11T08:28:17.959607+0000","last_active":"2024-01-11T08:28:17.967325+0000","last_peered":"2024-01-11T08:28:17.967325+0000","last_clean":"2024-01-11T08:28:17.967325+0000","last_became_active":"2024-01-11T08:28:17.959285+0000","last_became_peered":"2024-01-11T08:28:17.959285+0000","last_unstale":"2024-01-11T08:28:17.967325+0000","last_undegraded":"2024-01-11T08:28:17.967325+0000","last_fullsized":"2024-01-11T08:28:17.967325+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:32:22.287042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"115'2","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.985699+0000","last_change":"2024-01-11T08:28:16.967522+0000","last_active":"2024-01-11T08:28:17.985699+0000","last_peered":"2024-01-11T08:28:17.985699+0000","last_clean":"2024-01-11T08:28:17.985699+0000","last_became_active":"2024-01-11T08:28:16.967230+0000","last_became_peered":"2024-01-11T08:28:16.967230+0000","last_unstale":"2024-01-11T08:28:17.985699+0000","last_undegraded":"2024-01-11T08:28:17.985699+0000","last_fullsized":"2024-01-11T08:28:17.985699+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:02:54.383857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.975608+0000","last_change":"2024-01-11T08:28:17.972587+0000","last_active":"2024-01-11T08:28:17.975608+0000","last_peered":"2024-01-11T08:28:17.975608+0000","last_clean":"2024-01-11T08:28:17.975608+0000","last_became_active":"2024-01-11T08:28:17.972260+0000","last_became_peered":"2024-01-11T08:28:17.972260+0000","last_unstale":"2024-01-11T08:28:17.975608+0000","last_undegraded":"2024-01-11T08:28:17.975608+0000","last_fullsized":"2024-01-11T08:28:17.975608+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:52:19.210529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.963343+0000","last_change":"2024-01-11T08:28:16.969256+0000","last_active":"2024-01-11T08:28:17.963343+0000","last_peered":"2024-01-11T08:28:17.963343+0000","last_clean":"2024-01-11T08:28:17.963343+0000","last_became_active":"2024-01-11T08:28:16.967675+0000","last_became_peered":"2024-01-11T08:28:16.967675+0000","last_unstale":"2024-01-11T08:28:17.963343+0000","last_undegraded":"2024-01-11T08:28:17.963343+0000","last_fullsized":"2024-01-11T08:28:17.963343+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:34:22.883431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.963798+0000","last_change":"2024-01-11T08:28:17.957572+0000","last_active":"2024-01-11T08:28:17.963798+0000","last_peered":"2024-01-11T08:28:17.963798+0000","last_clean":"2024-01-11T08:28:17.963798+0000","last_became_active":"2024-01-11T08:28:17.957182+0000","last_became_peered":"2024-01-11T08:28:17.957182+0000","last_unstale":"2024-01-11T08:28:17.963798+0000","last_undegraded":"2024-01-11T08:28:17.963798+0000","last_fullsized":"2024-01-11T08:28:17.963798+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:01:23.824852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.956093+0000","last_change":"2024-01-11T08:28:16.965549+0000","last_active":"2024-01-11T08:28:17.956093+0000","last_peered":"2024-01-11T08:28:17.956093+0000","last_clean":"2024-01-11T08:28:17.956093+0000","last_became_active":"2024-01-11T08:28:16.964911+0000","last_became_peered":"2024-01-11T08:28:16.964911+0000","last_unstale":"2024-01-11T08:28:17.956093+0000","last_undegraded":"2024-01-11T08:28:17.956093+0000","last_fullsized":"2024-01-11T08:28:17.956093+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:28:36.423713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.976501+0000","last_change":"2024-01-11T08:28:17.968664+0000","last_active":"2024-01-11T08:28:17.976501+0000","last_peered":"2024-01-11T08:28:17.976501+0000","last_clean":"2024-01-11T08:28:17.976501+0000","last_became_active":"2024-01-11T08:28:17.968414+0000","last_became_peered":"2024-01-11T08:28:17.968414+0000","last_unstale":"2024-01-11T08:28:17.976501+0000","last_undegraded":"2024-01-11T08:28:17.976501+0000","last_fullsized":"2024-01-11T08:28:17.976501+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:56:54.199858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"115'8","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.994827+0000","last_change":"2024-01-11T08:28:16.965974+0000","last_active":"2024-01-11T08:28:17.994827+0000","last_peered":"2024-01-11T08:28:17.994827+0000","last_clean":"2024-01-11T08:28:17.994827+0000","last_became_active":"2024-01-11T08:28:16.962890+0000","last_became_peered":"2024-01-11T08:28:16.962890+0000","last_unstale":"2024-01-11T08:28:17.994827+0000","last_undegraded":"2024-01-11T08:28:17.994827+0000","last_fullsized":"2024-01-11T08:28:17.994827+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:56:53.193183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.978026+0000","last_change":"2024-01-11T08:28:17.975082+0000","last_active":"2024-01-11T08:28:17.978026+0000","last_peered":"2024-01-11T08:28:17.978026+0000","last_clean":"2024-01-11T08:28:17.978026+0000","last_became_active":"2024-01-11T08:28:17.974795+0000","last_became_peered":"2024-01-11T08:28:17.974795+0000","last_unstale":"2024-01-11T08:28:17.978026+0000","last_undegraded":"2024-01-11T08:28:17.978026+0000","last_fullsized":"2024-01-11T08:28:17.978026+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:01:23.824852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"115'2","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.992389+0000","last_change":"2024-01-11T08:28:16.970767+0000","last_active":"2024-01-11T08:28:17.992389+0000","last_peered":"2024-01-11T08:28:17.992389+0000","last_clean":"2024-01-11T08:28:17.992389+0000","last_became_active":"2024-01-11T08:28:16.970429+0000","last_became_peered":"2024-01-11T08:28:16.970429+0000","last_unstale":"2024-01-11T08:28:17.992389+0000","last_undegraded":"2024-01-11T08:28:17.992389+0000","last_fullsized":"2024-01-11T08:28:17.992389+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:01:17.535401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.984776+0000","last_change":"2024-01-11T08:28:17.981500+0000","last_active":"2024-01-11T08:28:17.984776+0000","last_peered":"2024-01-11T08:28:17.984776+0000","last_clean":"2024-01-11T08:28:17.984776+0000","last_became_active":"2024-01-11T08:28:17.979291+0000","last_became_peered":"2024-01-11T08:28:17.979291+0000","last_unstale":"2024-01-11T08:28:17.984776+0000","last_undegraded":"2024-01-11T08:28:17.984776+0000","last_fullsized":"2024-01-11T08:28:17.984776+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:10:08.279429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"115'1","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.990413+0000","last_change":"2024-01-11T08:28:16.966840+0000","last_active":"2024-01-11T08:28:17.990413+0000","last_peered":"2024-01-11T08:28:17.990413+0000","last_clean":"2024-01-11T08:28:17.990413+0000","last_became_active":"2024-01-11T08:28:16.966574+0000","last_became_peered":"2024-01-11T08:28:16.966574+0000","last_unstale":"2024-01-11T08:28:17.990413+0000","last_undegraded":"2024-01-11T08:28:17.990413+0000","last_fullsized":"2024-01-11T08:28:17.990413+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:23:38.768543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.971721+0000","last_change":"2024-01-11T08:28:17.968597+0000","last_active":"2024-01-11T08:28:17.971721+0000","last_peered":"2024-01-11T08:28:17.971721+0000","last_clean":"2024-01-11T08:28:17.971721+0000","last_became_active":"2024-01-11T08:28:17.968106+0000","last_became_peered":"2024-01-11T08:28:17.968106+0000","last_unstale":"2024-01-11T08:28:17.971721+0000","last_undegraded":"2024-01-11T08:28:17.971721+0000","last_fullsized":"2024-01-11T08:28:17.971721+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:21:26.125357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.958622+0000","last_change":"2024-01-11T08:28:16.972415+0000","last_active":"2024-01-11T08:28:17.958622+0000","last_peered":"2024-01-11T08:28:17.958622+0000","last_clean":"2024-01-11T08:28:17.958622+0000","last_became_active":"2024-01-11T08:28:16.971434+0000","last_became_peered":"2024-01-11T08:28:16.971434+0000","last_unstale":"2024-01-11T08:28:17.958622+0000","last_undegraded":"2024-01-11T08:28:17.958622+0000","last_fullsized":"2024-01-11T08:28:17.958622+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:28.686024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.977011+0000","last_change":"2024-01-11T08:28:17.969665+0000","last_active":"2024-01-11T08:28:17.977011+0000","last_peered":"2024-01-11T08:28:17.977011+0000","last_clean":"2024-01-11T08:28:17.977011+0000","last_became_active":"2024-01-11T08:28:17.969419+0000","last_became_peered":"2024-01-11T08:28:17.969419+0000","last_unstale":"2024-01-11T08:28:17.977011+0000","last_undegraded":"2024-01-11T08:28:17.977011+0000","last_fullsized":"2024-01-11T08:28:17.977011+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:10:31.911756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"115'2","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.989922+0000","last_change":"2024-01-11T08:28:16.974546+0000","last_active":"2024-01-11T08:28:17.989922+0000","last_peered":"2024-01-11T08:28:17.989922+0000","last_clean":"2024-01-11T08:28:17.989922+0000","last_became_active":"2024-01-11T08:28:16.974286+0000","last_became_peered":"2024-01-11T08:28:16.974286+0000","last_unstale":"2024-01-11T08:28:17.989922+0000","last_undegraded":"2024-01-11T08:28:17.989922+0000","last_fullsized":"2024-01-11T08:28:17.989922+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:28:36.423713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.984648+0000","last_change":"2024-01-11T08:28:17.978737+0000","last_active":"2024-01-11T08:28:17.984648+0000","last_peered":"2024-01-11T08:28:17.984648+0000","last_clean":"2024-01-11T08:28:17.984648+0000","last_became_active":"2024-01-11T08:28:17.978444+0000","last_became_peered":"2024-01-11T08:28:17.978444+0000","last_unstale":"2024-01-11T08:28:17.984648+0000","last_undegraded":"2024-01-11T08:28:17.984648+0000","last_fullsized":"2024-01-11T08:28:17.984648+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:49:42.272444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.953861+0000","last_change":"2024-01-11T08:28:16.963488+0000","last_active":"2024-01-11T08:28:17.953861+0000","last_peered":"2024-01-11T08:28:17.953861+0000","last_clean":"2024-01-11T08:28:17.953861+0000","last_became_active":"2024-01-11T08:28:16.962940+0000","last_became_peered":"2024-01-11T08:28:16.962940+0000","last_unstale":"2024-01-11T08:28:17.953861+0000","last_undegraded":"2024-01-11T08:28:17.953861+0000","last_fullsized":"2024-01-11T08:28:17.953861+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:20:04.676688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.981522+0000","last_change":"2024-01-11T08:28:17.975998+0000","last_active":"2024-01-11T08:28:17.981522+0000","last_peered":"2024-01-11T08:28:17.981522+0000","last_clean":"2024-01-11T08:28:17.981522+0000","last_became_active":"2024-01-11T08:28:17.972655+0000","last_became_peered":"2024-01-11T08:28:17.972655+0000","last_unstale":"2024-01-11T08:28:17.981522+0000","last_undegraded":"2024-01-11T08:28:17.981522+0000","last_fullsized":"2024-01-11T08:28:17.981522+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:55:18.176094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.953110+0000","last_change":"2024-01-11T08:28:16.973112+0000","last_active":"2024-01-11T08:28:17.953110+0000","last_peered":"2024-01-11T08:28:17.953110+0000","last_clean":"2024-01-11T08:28:17.953110+0000","last_became_active":"2024-01-11T08:28:16.972797+0000","last_became_peered":"2024-01-11T08:28:16.972797+0000","last_unstale":"2024-01-11T08:28:17.953110+0000","last_undegraded":"2024-01-11T08:28:17.953110+0000","last_fullsized":"2024-01-11T08:28:17.953110+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:47:11.453713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.978569+0000","last_change":"2024-01-11T08:28:17.975206+0000","last_active":"2024-01-11T08:28:17.978569+0000","last_peered":"2024-01-11T08:28:17.978569+0000","last_clean":"2024-01-11T08:28:17.978569+0000","last_became_active":"2024-01-11T08:28:17.973907+0000","last_became_peered":"2024-01-11T08:28:17.973907+0000","last_unstale":"2024-01-11T08:28:17.978569+0000","last_undegraded":"2024-01-11T08:28:17.978569+0000","last_fullsized":"2024-01-11T08:28:17.978569+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:51:46.671037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.963824+0000","last_change":"2024-01-11T08:28:16.972944+0000","last_active":"2024-01-11T08:28:17.963824+0000","last_peered":"2024-01-11T08:28:17.963824+0000","last_clean":"2024-01-11T08:28:17.963824+0000","last_became_active":"2024-01-11T08:28:16.972677+0000","last_became_peered":"2024-01-11T08:28:16.972677+0000","last_unstale":"2024-01-11T08:28:17.963824+0000","last_undegraded":"2024-01-11T08:28:17.963824+0000","last_fullsized":"2024-01-11T08:28:17.963824+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:41:30.011035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.987115+0000","last_change":"2024-01-11T08:28:17.985468+0000","last_active":"2024-01-11T08:28:17.987115+0000","last_peered":"2024-01-11T08:28:17.987115+0000","last_clean":"2024-01-11T08:28:17.987115+0000","last_became_active":"2024-01-11T08:28:17.984681+0000","last_became_peered":"2024-01-11T08:28:17.984681+0000","last_unstale":"2024-01-11T08:28:17.987115+0000","last_undegraded":"2024-01-11T08:28:17.987115+0000","last_fullsized":"2024-01-11T08:28:17.987115+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:42:56.802832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"115'2","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.985659+0000","last_change":"2024-01-11T08:28:16.966673+0000","last_active":"2024-01-11T08:28:17.985659+0000","last_peered":"2024-01-11T08:28:17.985659+0000","last_clean":"2024-01-11T08:28:17.985659+0000","last_became_active":"2024-01-11T08:28:16.966359+0000","last_became_peered":"2024-01-11T08:28:16.966359+0000","last_unstale":"2024-01-11T08:28:17.985659+0000","last_undegraded":"2024-01-11T08:28:17.985659+0000","last_fullsized":"2024-01-11T08:28:17.985659+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:07:23.638314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.975625+0000","last_change":"2024-01-11T08:28:17.973418+0000","last_active":"2024-01-11T08:28:17.975625+0000","last_peered":"2024-01-11T08:28:17.975625+0000","last_clean":"2024-01-11T08:28:17.975625+0000","last_became_active":"2024-01-11T08:28:17.973158+0000","last_became_peered":"2024-01-11T08:28:17.973158+0000","last_unstale":"2024-01-11T08:28:17.975625+0000","last_undegraded":"2024-01-11T08:28:17.975625+0000","last_fullsized":"2024-01-11T08:28:17.975625+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:26:18.820799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.966658+0000","last_change":"2024-01-11T08:28:16.974992+0000","last_active":"2024-01-11T08:28:17.966658+0000","last_peered":"2024-01-11T08:28:17.966658+0000","last_clean":"2024-01-11T08:28:17.966658+0000","last_became_active":"2024-01-11T08:28:16.973850+0000","last_became_peered":"2024-01-11T08:28:16.973850+0000","last_unstale":"2024-01-11T08:28:17.966658+0000","last_undegraded":"2024-01-11T08:28:17.966658+0000","last_fullsized":"2024-01-11T08:28:17.966658+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:23:38.768543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.981400+0000","last_change":"2024-01-11T08:28:17.979835+0000","last_active":"2024-01-11T08:28:17.981400+0000","last_peered":"2024-01-11T08:28:17.981400+0000","last_clean":"2024-01-11T08:28:17.981400+0000","last_became_active":"2024-01-11T08:28:17.979556+0000","last_became_peered":"2024-01-11T08:28:17.979556+0000","last_unstale":"2024-01-11T08:28:17.981400+0000","last_undegraded":"2024-01-11T08:28:17.981400+0000","last_fullsized":"2024-01-11T08:28:17.981400+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:58:37.483797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.960431+0000","last_change":"2024-01-11T08:28:16.968957+0000","last_active":"2024-01-11T08:28:17.960431+0000","last_peered":"2024-01-11T08:28:17.960431+0000","last_clean":"2024-01-11T08:28:17.960431+0000","last_became_active":"2024-01-11T08:28:16.967817+0000","last_became_peered":"2024-01-11T08:28:16.967817+0000","last_unstale":"2024-01-11T08:28:17.960431+0000","last_undegraded":"2024-01-11T08:28:17.960431+0000","last_fullsized":"2024-01-11T08:28:17.960431+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:10:07.272755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.978367+0000","last_change":"2024-01-11T08:28:17.976627+0000","last_active":"2024-01-11T08:28:17.978367+0000","last_peered":"2024-01-11T08:28:17.978367+0000","last_clean":"2024-01-11T08:28:17.978367+0000","last_became_active":"2024-01-11T08:28:17.976316+0000","last_became_peered":"2024-01-11T08:28:17.976316+0000","last_unstale":"2024-01-11T08:28:17.978367+0000","last_undegraded":"2024-01-11T08:28:17.978367+0000","last_fullsized":"2024-01-11T08:28:17.978367+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:11:37.602642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.963845+0000","last_change":"2024-01-11T08:28:16.968480+0000","last_active":"2024-01-11T08:28:17.963845+0000","last_peered":"2024-01-11T08:28:17.963845+0000","last_clean":"2024-01-11T08:28:17.963845+0000","last_became_active":"2024-01-11T08:28:16.968159+0000","last_became_peered":"2024-01-11T08:28:16.968159+0000","last_unstale":"2024-01-11T08:28:17.963845+0000","last_undegraded":"2024-01-11T08:28:17.963845+0000","last_fullsized":"2024-01-11T08:28:17.963845+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:01:17.535401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.977029+0000","last_change":"2024-01-11T08:28:17.969304+0000","last_active":"2024-01-11T08:28:17.977029+0000","last_peered":"2024-01-11T08:28:17.977029+0000","last_clean":"2024-01-11T08:28:17.977029+0000","last_became_active":"2024-01-11T08:28:17.968537+0000","last_became_peered":"2024-01-11T08:28:17.968537+0000","last_unstale":"2024-01-11T08:28:17.977029+0000","last_undegraded":"2024-01-11T08:28:17.977029+0000","last_fullsized":"2024-01-11T08:28:17.977029+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:07:56.270516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.967612+0000","last_change":"2024-01-11T08:28:16.967919+0000","last_active":"2024-01-11T08:28:17.967612+0000","last_peered":"2024-01-11T08:28:17.967612+0000","last_clean":"2024-01-11T08:28:17.967612+0000","last_became_active":"2024-01-11T08:28:16.967620+0000","last_became_peered":"2024-01-11T08:28:16.967620+0000","last_unstale":"2024-01-11T08:28:17.967612+0000","last_undegraded":"2024-01-11T08:28:17.967612+0000","last_fullsized":"2024-01-11T08:28:17.967612+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:47:11.453713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.984768+0000","last_change":"2024-01-11T08:28:17.979033+0000","last_active":"2024-01-11T08:28:17.984768+0000","last_peered":"2024-01-11T08:28:17.984768+0000","last_clean":"2024-01-11T08:28:17.984768+0000","last_became_active":"2024-01-11T08:28:17.978730+0000","last_became_peered":"2024-01-11T08:28:17.978730+0000","last_unstale":"2024-01-11T08:28:17.984768+0000","last_undegraded":"2024-01-11T08:28:17.984768+0000","last_fullsized":"2024-01-11T08:28:17.984768+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:11:37.602642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.964904+0000","last_change":"2024-01-11T08:28:16.969753+0000","last_active":"2024-01-11T08:28:17.964904+0000","last_peered":"2024-01-11T08:28:17.964904+0000","last_clean":"2024-01-11T08:28:17.964904+0000","last_became_active":"2024-01-11T08:28:16.969422+0000","last_became_peered":"2024-01-11T08:28:16.969422+0000","last_unstale":"2024-01-11T08:28:17.964904+0000","last_undegraded":"2024-01-11T08:28:17.964904+0000","last_fullsized":"2024-01-11T08:28:17.964904+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:28.686024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.981247+0000","last_change":"2024-01-11T08:28:17.974257+0000","last_active":"2024-01-11T08:28:17.981247+0000","last_peered":"2024-01-11T08:28:17.981247+0000","last_clean":"2024-01-11T08:28:17.981247+0000","last_became_active":"2024-01-11T08:28:17.972335+0000","last_became_peered":"2024-01-11T08:28:17.972335+0000","last_unstale":"2024-01-11T08:28:17.981247+0000","last_undegraded":"2024-01-11T08:28:17.981247+0000","last_fullsized":"2024-01-11T08:28:17.981247+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:25.552754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.952216+0000","last_change":"2024-01-11T08:28:16.965825+0000","last_active":"2024-01-11T08:28:17.952216+0000","last_peered":"2024-01-11T08:28:17.952216+0000","last_clean":"2024-01-11T08:28:17.952216+0000","last_became_active":"2024-01-11T08:28:16.965509+0000","last_became_peered":"2024-01-11T08:28:16.965509+0000","last_unstale":"2024-01-11T08:28:17.952216+0000","last_undegraded":"2024-01-11T08:28:17.952216+0000","last_fullsized":"2024-01-11T08:28:17.952216+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:20:04.676688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.975835+0000","last_change":"2024-01-11T08:28:17.970001+0000","last_active":"2024-01-11T08:28:17.975835+0000","last_peered":"2024-01-11T08:28:17.975835+0000","last_clean":"2024-01-11T08:28:17.975835+0000","last_became_active":"2024-01-11T08:28:17.969575+0000","last_became_peered":"2024-01-11T08:28:17.969575+0000","last_unstale":"2024-01-11T08:28:17.975835+0000","last_undegraded":"2024-01-11T08:28:17.975835+0000","last_fullsized":"2024-01-11T08:28:17.975835+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:42:56.802832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.963401+0000","last_change":"2024-01-11T08:28:16.969707+0000","last_active":"2024-01-11T08:28:17.963401+0000","last_peered":"2024-01-11T08:28:17.963401+0000","last_clean":"2024-01-11T08:28:17.963401+0000","last_became_active":"2024-01-11T08:28:16.968984+0000","last_became_peered":"2024-01-11T08:28:16.968984+0000","last_unstale":"2024-01-11T08:28:17.963401+0000","last_undegraded":"2024-01-11T08:28:17.963401+0000","last_fullsized":"2024-01-11T08:28:17.963401+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:08:09.413723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.976912+0000","last_change":"2024-01-11T08:28:17.971040+0000","last_active":"2024-01-11T08:28:17.976912+0000","last_peered":"2024-01-11T08:28:17.976912+0000","last_clean":"2024-01-11T08:28:17.976912+0000","last_became_active":"2024-01-11T08:28:17.970772+0000","last_became_peered":"2024-01-11T08:28:17.970772+0000","last_unstale":"2024-01-11T08:28:17.976912+0000","last_undegraded":"2024-01-11T08:28:17.976912+0000","last_fullsized":"2024-01-11T08:28:17.976912+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:17:30.830370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"115'1","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.993265+0000","last_change":"2024-01-11T08:28:16.971682+0000","last_active":"2024-01-11T08:28:17.993265+0000","last_peered":"2024-01-11T08:28:17.993265+0000","last_clean":"2024-01-11T08:28:17.993265+0000","last_became_active":"2024-01-11T08:28:16.971376+0000","last_became_peered":"2024-01-11T08:28:16.971376+0000","last_unstale":"2024-01-11T08:28:17.993265+0000","last_undegraded":"2024-01-11T08:28:17.993265+0000","last_fullsized":"2024-01-11T08:28:17.993265+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:34:24.876291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"107'223","reported_seq":614,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.958069+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:28:17.958069+0000","last_peered":"2024-01-11T08:28:17.958069+0000","last_clean":"2024-01-11T08:28:17.958069+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:28:17.958069+0000","last_undegraded":"2024-01-11T08:28:17.958069+0000","last_fullsized":"2024-01-11T08:28:17.958069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1040,"num_read_kb":2130,"num_write":351,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1040,"num_read_kb":2130,"num_write":351,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575060,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303372,"kb_used_data":1988,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068468,"statfs":{"total":96636764160,"available":96326111232,"internally_reserved":0,"allocated":2035712,"data_stored":1256826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.68200000000000005,"15min":0.69999999999999996},"min":{"1min":0.42199999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.93600000000000005,"5min":0.95899999999999996,"15min":0.996},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.66800000000000004,"15min":0.70899999999999996},"min":{"1min":0.44500000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89000000000000001,"5min":1.073,"15min":1.073},"last":0.60499999999999998}]},{"osd":1,"last update":"Thu Jan 11 08:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67900000000000005,"15min":0.70599999999999996},"min":{"1min":0.45800000000000002,"5min":0.432,"15min":0.41899999999999998},"max":{"1min":0.82799999999999996,"5min":0.99099999999999999,"15min":0.998},"last":0.622},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69099999999999995,"15min":0.71899999999999997},"min":{"1min":0.47799999999999998,"5min":0.41999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88400000000000001,"5min":0.94999999999999996,"15min":1.014},"last":0.65800000000000003}]},{"osd":2,"last update":"Thu Jan 11 08:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69499999999999995,"15min":0.73099999999999998},"min":{"1min":0.46700000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.034},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.68400000000000005,"15min":0.71499999999999997},"min":{"1min":0.497,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.89500000000000002,"5min":0.98299999999999998,"15min":1.0049999999999999},"last":0.67900000000000005}]},{"osd":3,"last update":"Thu Jan 11 08:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.69599999999999995,"15min":0.73499999999999999},"min":{"1min":0.53400000000000003,"5min":0.47899999999999998,"15min":0.45700000000000002},"max":{"1min":0.84699999999999998,"5min":0.91700000000000004,"15min":1.032},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.71399999999999997,"15min":0.73899999999999999},"min":{"1min":0.44,"5min":0.439,"15min":0.439},"max":{"1min":0.92600000000000005,"5min":1.008,"15min":1.008},"last":0.877}]},{"osd":4,"last update":"Thu Jan 11 08:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.40500000000000003,"5min":0.39900000000000002,"15min":0.34899999999999998},"max":{"1min":0.745,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.64100000000000001,"15min":0.625},"min":{"1min":0.46700000000000003,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.77100000000000002,"5min":1.002,"15min":1.002},"last":0.63400000000000001}]},{"osd":5,"last update":"Thu Jan 11 08:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.46500000000000002,"5min":0.41899999999999998,"15min":0.34599999999999997},"max":{"1min":0.83299999999999996,"5min":1.042,"15min":1.042},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.66100000000000003,"15min":0.66600000000000004},"min":{"1min":0.45300000000000001,"5min":0.41399999999999998,"15min":0.40400000000000003},"max":{"1min":0.85999999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.72999999999999998}]},{"osd":6,"last update":"Thu Jan 11 08:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.73699999999999999,"15min":0.751},"min":{"1min":0.57399999999999995,"5min":0.52000000000000002,"15min":0.46999999999999997},"max":{"1min":0.95899999999999996,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.54400000000000004,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.84299999999999997,"5min":1.028,"15min":1.028},"last":0.58299999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575059,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302792,"kb_used_data":1408,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069048,"statfs":{"total":96636764160,"available":96326705152,"internally_reserved":0,"allocated":1441792,"data_stored":666970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63900000000000001,"15min":0.66700000000000004},"min":{"1min":0.441,"5min":0.38300000000000001,"15min":0.29999999999999999},"max":{"1min":0.84299999999999997,"5min":0.89300000000000002,"15min":0.94799999999999995},"last":0.627},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.63200000000000001,"15min":0.68600000000000005},"min":{"1min":0.45100000000000001,"5min":0.40000000000000002,"15min":0.33300000000000002},"max":{"1min":0.88700000000000001,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.58999999999999997}]},{"osd":1,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.65400000000000003,"15min":0.69499999999999995},"min":{"1min":0.505,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86399999999999999,"5min":1.008,"15min":1.163},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64000000000000001,"15min":0.68500000000000005},"min":{"1min":0.48599999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.89200000000000002,"5min":0.92600000000000005,"15min":1.1950000000000001},"last":0.77000000000000002}]},{"osd":2,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66000000000000003,"15min":0.70599999999999996},"min":{"1min":0.42099999999999999,"5min":0.41499999999999998,"15min":0.39800000000000002},"max":{"1min":0.91400000000000003,"5min":0.95399999999999996,"15min":1.016},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.67000000000000004,"15min":0.71799999999999997},"min":{"1min":0.53100000000000003,"5min":0.41899999999999998,"15min":0.373},"max":{"1min":0.88600000000000001,"5min":0.91300000000000003,"15min":0.99299999999999999},"last":0.74099999999999999}]},{"osd":3,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.69799999999999995,"15min":0.73099999999999998},"min":{"1min":0.52600000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.89700000000000002,"5min":0.95099999999999996,"15min":1.0149999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.68899999999999995,"15min":0.73699999999999999},"min":{"1min":0.46000000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.876,"5min":0.90900000000000003,"15min":1.0049999999999999},"last":0.70699999999999996}]},{"osd":4,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.439,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85699999999999998,"5min":0.873,"15min":0.92000000000000004},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58899999999999997,"15min":0.59499999999999997},"min":{"1min":0.46300000000000002,"5min":0.38800000000000001,"15min":0.374},"max":{"1min":0.83299999999999996,"5min":0.83799999999999997,"15min":0.96399999999999997},"last":0.69199999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.48699999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.82899999999999996,"5min":1.03,"15min":1.03},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68300000000000005,"15min":0.69799999999999995},"min":{"1min":0.54200000000000004,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.877,"5min":0.91800000000000004,"15min":0.93700000000000006},"last":0.72799999999999998}]},{"osd":7,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68700000000000006,"15min":0.70499999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.42399999999999999},"max":{"1min":0.91500000000000004,"5min":0.93300000000000005,"15min":1.0289999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.72099999999999997,"15min":0.746},"min":{"1min":0.50800000000000001,"5min":0.45900000000000002,"15min":0.44700000000000001},"max":{"1min":0.90700000000000003,"5min":1.0600000000000001,"15min":1.0669999999999999},"last":0.75700000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575061,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302784,"kb_used_data":1400,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069056,"statfs":{"total":96636764160,"available":96326713344,"internally_reserved":0,"allocated":1433600,"data_stored":666346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:27:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.48199999999999998,"15min":0.48599999999999999},"min":{"1min":0.29999999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65200000000000002,"5min":0.85999999999999999,"15min":0.97699999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54300000000000004,"15min":0.57999999999999996},"min":{"1min":0.44600000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.76100000000000001,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.44900000000000001}]},{"osd":2,"last update":"Thu Jan 11 08:27:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45300000000000001,"15min":0.48199999999999998},"min":{"1min":0.35399999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.64800000000000002,"5min":1.0960000000000001,"15min":1.113},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54600000000000004,"15min":0.56399999999999995},"min":{"1min":0.38800000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":0.76600000000000001,"5min":0.76700000000000002,"15min":0.90100000000000002},"last":0.46400000000000002}]},{"osd":3,"last update":"Thu Jan 11 08:27:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.59499999999999997,"15min":0.61399999999999999},"min":{"1min":0.47499999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.80500000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.53700000000000003,"15min":0.56399999999999995},"min":{"1min":0.39900000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82999999999999996,"5min":0.88100000000000001,"15min":0.94899999999999995},"last":0.47799999999999998}]},{"osd":4,"last update":"Thu Jan 11 08:27:50 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.76100000000000001,"15min":0.79000000000000004},"min":{"1min":0.505,"5min":0.439,"15min":0.439},"max":{"1min":1.129,"5min":1.129,"15min":1.2170000000000001},"last":0.753},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.81299999999999994,"15min":0.83199999999999996},"min":{"1min":0.70599999999999996,"5min":0.501,"15min":0.501},"max":{"1min":1.109,"5min":1.109,"15min":1.246},"last":0.66000000000000003}]},{"osd":5,"last update":"Thu Jan 11 08:27:50 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.75800000000000001,"15min":0.79600000000000004},"min":{"1min":0.64500000000000002,"5min":0.498,"15min":0.498},"max":{"1min":1.0309999999999999,"5min":1.1299999999999999,"15min":1.2629999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.754,"15min":0.78800000000000003},"min":{"1min":0.57499999999999996,"5min":0.52900000000000003,"15min":0.51200000000000001},"max":{"1min":0.94899999999999995,"5min":1.1339999999999999,"15min":1.212},"last":0.67500000000000004}]},{"osd":6,"last update":"Thu Jan 11 08:27:50 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.78900000000000003,"15min":0.81899999999999995},"min":{"1min":0.622,"5min":0.56100000000000005,"15min":0.51300000000000001},"max":{"1min":1.042,"5min":1.115,"15min":1.1439999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.79400000000000004,"15min":0.82699999999999996},"min":{"1min":0.63600000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.228},"last":0.54700000000000004}]},{"osd":7,"last update":"Thu Jan 11 08:27:50 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.79900000000000004,"15min":0.84599999999999997},"min":{"1min":0.72899999999999998,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.226},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.876,"5min":0.82299999999999995,"15min":0.84199999999999997},"min":{"1min":0.63400000000000001,"5min":0.55900000000000005,"15min":0.51400000000000001},"max":{"1min":1.087,"5min":1.099,"15min":1.258},"last":0.72699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575060,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303368,"kb_used_data":1984,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068472,"statfs":{"total":96636764160,"available":96326115328,"internally_reserved":0,"allocated":2031616,"data_stored":1257268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 08:27:53 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.501,"15min":0.499},"min":{"1min":0.33700000000000002,"5min":0.28699999999999998,"15min":0.24299999999999999},"max":{"1min":0.65300000000000002,"5min":0.84399999999999997,"15min":0.92700000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.56999999999999995,"15min":0.57399999999999995},"min":{"1min":0.51600000000000001,"5min":0.40300000000000002,"15min":0.36799999999999999},"max":{"1min":0.78500000000000003,"5min":0.90500000000000003,"15min":0.92100000000000004},"last":0.43099999999999999}]},{"osd":2,"last update":"Thu Jan 11 08:27:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.56000000000000005,"15min":0.55000000000000004},"min":{"1min":0.42099999999999999,"5min":0.33000000000000002,"15min":0.29099999999999998},"max":{"1min":0.73899999999999999,"5min":1.335,"15min":1.335},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.55600000000000005,"15min":0.55800000000000005},"min":{"1min":0.44700000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.76500000000000001,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.39100000000000001}]},{"osd":3,"last update":"Thu Jan 11 08:27:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.39800000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.71799999999999997,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.627,"15min":0.629},"min":{"1min":0.48399999999999999,"5min":0.44700000000000001,"15min":0.39600000000000002},"max":{"1min":0.79100000000000004,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.59099999999999997}]},{"osd":4,"last update":"Thu Jan 11 08:27:53 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.79200000000000004,"15min":0.79300000000000004},"min":{"1min":0.64800000000000002,"5min":0.502,"15min":0.437},"max":{"1min":1.05,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.627},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79100000000000004,"15min":0.79300000000000004},"min":{"1min":0.60599999999999998,"5min":0.51600000000000001,"15min":0.46899999999999997},"max":{"1min":0.94299999999999995,"5min":1.1220000000000001,"15min":1.149},"last":0.56200000000000006}]},{"osd":5,"last update":"Thu Jan 11 08:27:53 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.79200000000000004,"15min":0.80700000000000005},"min":{"1min":0.69399999999999995,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.091,"5min":1.518,"15min":1.518},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.82299999999999995,"15min":0.82099999999999995},"min":{"1min":0.69399999999999995,"5min":0.55400000000000005,"15min":0.47799999999999998},"max":{"1min":1.0189999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.751}]},{"osd":6,"last update":"Thu Jan 11 08:27:53 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.81499999999999995,"15min":0.80800000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.50900000000000001},"max":{"1min":1.1120000000000001,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.82599999999999996,"15min":0.83599999999999997},"min":{"1min":0.66800000000000004,"5min":0.52000000000000002,"15min":0.44900000000000001},"max":{"1min":1.0820000000000001,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.64000000000000001}]},{"osd":7,"last update":"Thu Jan 11 08:27:53 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.84299999999999997,"15min":0.84799999999999998},"min":{"1min":0.755,"5min":0.61399999999999999,"15min":0.51400000000000001},"max":{"1min":1.085,"5min":1.371,"15min":1.371},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.872,"5min":0.83399999999999996,"15min":0.84499999999999997},"min":{"1min":0.59699999999999998,"5min":0.58099999999999996,"15min":0.57699999999999996},"max":{"1min":1.046,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.65800000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575059,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310980,"kb_used_data":1404,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060860,"statfs":{"total":96636764160,"available":96318320640,"internally_reserved":0,"allocated":1437696,"data_stored":666380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47199999999999998,"15min":0.47399999999999998},"min":{"1min":0.316,"5min":0.29899999999999999,"15min":0.26000000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51100000000000001,"15min":0.54100000000000004},"min":{"1min":0.36599999999999999,"5min":0.33000000000000002,"15min":0.28899999999999998},"max":{"1min":0.72899999999999998,"5min":0.81699999999999995,"15min":0.91400000000000003},"last":0.38500000000000001}]},{"osd":1,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53500000000000003,"15min":0.55000000000000004},"min":{"1min":0.44600000000000001,"5min":0.36399999999999999,"15min":0.33500000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.90000000000000002},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55200000000000005,"15min":0.58299999999999996},"min":{"1min":0.44700000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.94499999999999995},"last":0.65300000000000002}]},{"osd":3,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.36699999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.59899999999999998,"5min":0.82499999999999996,"15min":0.91600000000000004},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.59499999999999997,"15min":0.63300000000000001},"min":{"1min":0.48599999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":1.0009999999999999},"last":0.58199999999999996}]},{"osd":4,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.746,"15min":0.78600000000000003},"min":{"1min":0.61299999999999999,"5min":0.52400000000000002,"15min":0.47299999999999998},"max":{"1min":0.995,"5min":1.026,"15min":1.2889999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.72999999999999998,"15min":0.78100000000000003},"min":{"1min":0.625,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.3089999999999999},"last":0.56200000000000006}]},{"osd":5,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.79200000000000004,"15min":0.83199999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.3540000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.81699999999999995,"15min":0.84599999999999997},"min":{"1min":0.77600000000000002,"5min":0.61099999999999999,"15min":0.55500000000000005},"max":{"1min":1.0049999999999999,"5min":1.2,"15min":1.23},"last":0.60099999999999998}]},{"osd":6,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.75600000000000001,"15min":0.79800000000000004},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.024,"5min":1.131,"15min":1.2669999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.78300000000000003,"15min":0.82699999999999996},"min":{"1min":0.64500000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.351,"5min":1.351,"15min":1.47},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.82099999999999995,"15min":0.86499999999999999},"min":{"1min":0.69999999999999996,"5min":0.56599999999999995,"15min":0.51400000000000001},"max":{"1min":1.0600000000000001,"5min":1.2629999999999999,"15min":1.4099999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.79400000000000004,"15min":0.85799999999999998},"min":{"1min":0.71899999999999997,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.96099999999999997,"5min":1.2270000000000001,"15min":1.327},"last":0.66900000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575059,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306896,"kb_used_data":1416,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064944,"statfs":{"total":96636764160,"available":96322502656,"internally_reserved":0,"allocated":1449984,"data_stored":667904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.49099999999999999,"15min":0.48599999999999999},"min":{"1min":0.36499999999999999,"5min":0.31,"15min":0.27600000000000002},"max":{"1min":0.80400000000000005,"5min":0.80600000000000005,"15min":1.1930000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.52400000000000002,"15min":0.54800000000000004},"min":{"1min":0.44,"5min":0.39300000000000002,"15min":0.33400000000000002},"max":{"1min":0.75700000000000001,"5min":0.873,"15min":0.873},"last":0.46300000000000002}]},{"osd":1,"last update":"Thu Jan 11 08:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.46899999999999997,"15min":0.49099999999999999},"min":{"1min":0.34899999999999998,"5min":0.316,"15min":0.25900000000000001},"max":{"1min":0.69599999999999995,"5min":1.012,"15min":1.012},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56999999999999995,"15min":0.57699999999999996},"min":{"1min":0.433,"5min":0.36499999999999999,"15min":0.36299999999999999},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.1499999999999999},"last":0.39100000000000001}]},{"osd":2,"last update":"Thu Jan 11 08:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55900000000000005,"15min":0.58399999999999996},"min":{"1min":0.41499999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.873,"5min":0.873,"15min":0.97599999999999998},"last":0.434},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60899999999999999,"15min":0.63100000000000001},"min":{"1min":0.495,"5min":0.45100000000000001,"15min":0.32000000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":1.2270000000000001},"last":0.53300000000000003}]},{"osd":4,"last update":"Thu Jan 11 08:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.70799999999999996,"15min":0.73799999999999999},"min":{"1min":0.57199999999999995,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.94399999999999995,"5min":1.038,"15min":1.288},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.752,"15min":0.77900000000000003},"min":{"1min":0.65700000000000003,"5min":0.503,"15min":0.503},"max":{"1min":1.028,"5min":1.1459999999999999,"15min":1.1819999999999999},"last":0.70599999999999996}]},{"osd":5,"last update":"Thu Jan 11 08:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.754,"15min":0.78400000000000003},"min":{"1min":0.621,"5min":0.53000000000000003,"15min":0.51900000000000002},"max":{"1min":0.98299999999999998,"5min":1.0649999999999999,"15min":1.3260000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.79000000000000004,"15min":0.81000000000000005},"min":{"1min":0.70399999999999996,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.048,"5min":1.24,"15min":1.4239999999999999},"last":0.79400000000000004}]},{"osd":6,"last update":"Thu Jan 11 08:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.78800000000000003,"15min":0.82999999999999996},"min":{"1min":0.68100000000000005,"5min":0.54100000000000004,"15min":0.52600000000000002},"max":{"1min":1.0169999999999999,"5min":1.093,"15min":1.335},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.76900000000000002,"15min":0.79400000000000004},"min":{"1min":0.65000000000000002,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.45},"last":0.68500000000000005}]},{"osd":7,"last update":"Thu Jan 11 08:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.79300000000000004,"15min":0.83199999999999996},"min":{"1min":0.67500000000000004,"5min":0.52800000000000002,"15min":0.46600000000000003},"max":{"1min":1.0620000000000001,"5min":1.081,"15min":1.4019999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.83499999999999996,"15min":0.85899999999999999},"min":{"1min":0.73199999999999998,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.115,"5min":1.2050000000000001,"15min":1.21},"last":0.81100000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575060,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310996,"kb_used_data":1420,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060844,"statfs":{"total":96636764160,"available":96318304256,"internally_reserved":0,"allocated":1454080,"data_stored":668006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.70199999999999996,"15min":0.74099999999999999},"min":{"1min":0.54900000000000004,"5min":0.44500000000000001,"15min":0.28100000000000003},"max":{"1min":0.89000000000000001,"5min":0.94799999999999995,"15min":1.117},"last":0.504},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.70999999999999996,"15min":0.74199999999999999},"min":{"1min":0.61699999999999999,"5min":0.41599999999999998,"15min":0.38600000000000001},"max":{"1min":0.95299999999999996,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.67300000000000004}]},{"osd":1,"last update":"Thu Jan 11 08:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68100000000000005,"15min":0.70499999999999996},"min":{"1min":0.52400000000000002,"5min":0.434,"15min":0.33300000000000002},"max":{"1min":0.88200000000000001,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.67900000000000005,"15min":0.71299999999999997},"min":{"1min":0.60099999999999998,"5min":0.43099999999999999,"15min":0.36299999999999999},"max":{"1min":0.93899999999999995,"5min":0.99199999999999999,"15min":1.046},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 11 08:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.70799999999999996,"15min":0.72099999999999997},"min":{"1min":0.57099999999999995,"5min":0.46000000000000002,"15min":0.38900000000000001},"max":{"1min":0.92700000000000005,"5min":1.03,"15min":1.03},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69599999999999995,"15min":0.72999999999999998},"min":{"1min":0.58499999999999996,"5min":0.432,"15min":0.432},"max":{"1min":0.91500000000000004,"5min":0.97699999999999998,"15min":1.0409999999999999},"last":0.61899999999999999}]},{"osd":3,"last update":"Thu Jan 11 08:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.67000000000000004,"15min":0.70199999999999996},"min":{"1min":0.52100000000000002,"5min":0.46100000000000002,"15min":0.433},"max":{"1min":0.81599999999999995,"5min":0.96799999999999997,"15min":1.0189999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.72599999999999998,"15min":0.75},"min":{"1min":0.53700000000000003,"5min":0.502,"15min":0.47499999999999998},"max":{"1min":0.93200000000000005,"5min":1.026,"15min":1.069},"last":0.70299999999999996}]},{"osd":5,"last update":"Thu Jan 11 08:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.60099999999999998,"15min":0.60799999999999998},"min":{"1min":0.39700000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77800000000000002,"5min":0.96599999999999997,"15min":1.0700000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.61399999999999999,"15min":0.625},"min":{"1min":0.42499999999999999,"5min":0.38200000000000001,"15min":0.32600000000000001},"max":{"1min":0.89700000000000002,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.752}]},{"osd":6,"last update":"Thu Jan 11 08:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.64900000000000002,"15min":0.64700000000000002},"min":{"1min":0.44700000000000001,"5min":0.40500000000000003,"15min":0.39500000000000002},"max":{"1min":0.82699999999999996,"5min":1.133,"15min":1.133},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.46100000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88500000000000001,"5min":1.0269999999999999,"15min":1.119},"last":0.73399999999999999}]},{"osd":7,"last update":"Thu Jan 11 08:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.69099999999999995,"15min":0.68799999999999994},"min":{"1min":0.55400000000000005,"5min":0.47099999999999997,"15min":0.41299999999999998},"max":{"1min":0.98299999999999998,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.49099999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92100000000000004,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.90500000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575060,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302788,"kb_used_data":1404,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069052,"statfs":{"total":96636764160,"available":96326709248,"internally_reserved":0,"allocated":1437696,"data_stored":666416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.68100000000000005,"15min":0.71099999999999997},"min":{"1min":0.53700000000000003,"5min":0.39300000000000002,"15min":0.38700000000000001},"max":{"1min":0.86799999999999999,"5min":1.0249999999999999,"15min":1.032},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.68300000000000005,"15min":0.71099999999999997},"min":{"1min":0.498,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.93500000000000005,"5min":0.96199999999999997,"15min":1.696},"last":0.86099999999999999}]},{"osd":1,"last update":"Thu Jan 11 08:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.65500000000000003,"15min":0.69699999999999995},"min":{"1min":0.55600000000000005,"5min":0.39100000000000001,"15min":0.36399999999999999},"max":{"1min":0.997,"5min":0.997,"15min":1.4359999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.68999999999999995,"15min":0.73099999999999998},"min":{"1min":0.55400000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.86899999999999999,"5min":1.05,"15min":4.4969999999999999},"last":0.81699999999999995}]},{"osd":2,"last update":"Thu Jan 11 08:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.68799999999999994,"15min":0.71099999999999997},"min":{"1min":0.58499999999999996,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88,"5min":0.97299999999999998,"15min":1.238},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.68600000000000005,"15min":0.745},"min":{"1min":0.54300000000000004,"5min":0.45300000000000001,"15min":0.41599999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":8.0429999999999993},"last":0.84099999999999997}]},{"osd":3,"last update":"Thu Jan 11 08:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.69899999999999995,"15min":0.75},"min":{"1min":0.52600000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.90300000000000002,"5min":1.208,"15min":8.0839999999999996},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73199999999999998,"15min":0.745},"min":{"1min":0.57599999999999996,"5min":0.439,"15min":0.40899999999999997},"max":{"1min":0.89300000000000002,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.745}]},{"osd":4,"last update":"Thu Jan 11 08:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.41399999999999998,"5min":0.38300000000000001,"15min":0.34300000000000003},"max":{"1min":0.84299999999999997,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.64700000000000002,"15min":0.629},"min":{"1min":0.39400000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.88700000000000001,"5min":1.256,"15min":1.256},"last":0.60099999999999998}]},{"osd":6,"last update":"Thu Jan 11 08:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.64400000000000002},"min":{"1min":0.46500000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83599999999999997,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.66600000000000004,"15min":0.67900000000000005},"min":{"1min":0.505,"5min":0.443,"15min":0.443},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.93700000000000006}]},{"osd":7,"last update":"Thu Jan 11 08:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.70799999999999996,"15min":0.72299999999999998},"min":{"1min":0.45300000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.83699999999999997,"5min":1.341,"15min":1.341},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.76600000000000001,"15min":0.76500000000000001},"min":{"1min":0.57799999999999996,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.98399999999999999,"5min":1.119,"15min":1.119},"last":0.91400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:28:22.383 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:28:22.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.617+0000 7f4be09ff700 1 Processor -- start 2024-01-11T08:28:22.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.617+0000 7f4be09ff700 1 -- start start 2024-01-11T08:28:22.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.619+0000 7f4be09ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bdc0b2b90 0x7f4bdc159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:22.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.619+0000 7f4be09ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc0cd730 con 0x7f4bdc0b03e0 2024-01-11T08:28:22.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.619+0000 7f4be09ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc0ccb20 con 0x7f4bdc0b07b0 2024-01-11T08:28:22.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.619+0000 7f4be09ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc0cce00 con 0x7f4bdc0b2b90 2024-01-11T08:28:22.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.619+0000 7f4bda59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4bdc0b03e0 0x7f4bdc0b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51574/0 (socket says 172.21.15.27:51574) 2024-01-11T08:28:22.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.619+0000 7f4bda59c700 1 -- 172.21.15.27:0/2819485292 learned_addr learned my addr 172.21.15.27:0/2819485292 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:28:22.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.619+0000 7f4bd9d9b700 1 --2- 172.21.15.27:0/2819485292 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bdc0b2b90 0x7f4bdc159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:22.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.619+0000 7f4bd959a700 1 -- 172.21.15.27:0/2819485292 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199651903 0 0) 0x7f4bdc0cd730 con 0x7f4bdc0b03e0 2024-01-11T08:28:22.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.619+0000 7f4bd959a700 1 -- 172.21.15.27:0/2819485292 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc4003570 con 0x7f4bdc0b03e0 2024-01-11T08:28:22.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.620+0000 7f4bd9d9b700 1 -- 172.21.15.27:0/2819485292 >> v1:172.21.15.77:6789/0 conn(0x7f4bdc0b07b0 legacy=0x7f4bdc159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:22.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.620+0000 7f4bd959a700 1 -- 172.21.15.27:0/2819485292 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596106347 0 0) 0x7f4bdc0ccb20 con 0x7f4bdc0b07b0 2024-01-11T08:28:22.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.620+0000 7f4bd9d9b700 1 -- 172.21.15.27:0/2819485292 >> v1:172.21.15.27:6789/0 conn(0x7f4bdc0b03e0 legacy=0x7f4bdc0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:22.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.620+0000 7f4bd9d9b700 1 -- 172.21.15.27:0/2819485292 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc0cd0e0 con 0x7f4bdc0b2b90 2024-01-11T08:28:22.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.620+0000 7f4bd9d9b700 1 --2- 172.21.15.27:0/2819485292 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bdc0b2b90 0x7f4bdc159200 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0009910 tx=0x7f4bd00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ae1559cd3f2d614 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:22.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.620+0000 7f4bd959a700 1 -- 172.21.15.27:0/2819485292 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd000b680 con 0x7f4bdc0b2b90 2024-01-11T08:28:22.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.620+0000 7f4bd959a700 1 -- 172.21.15.27:0/2819485292 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd000b7e0 con 0x7f4bdc0b2b90 2024-01-11T08:28:22.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.620+0000 7f4bd959a700 1 -- 172.21.15.27:0/2819485292 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd000bba0 con 0x7f4bdc0b2b90 2024-01-11T08:28:22.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.621+0000 7f4be09ff700 1 -- 172.21.15.27:0/2819485292 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bdc0b2b90 msgr2=0x7f4bdc159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:22.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.621+0000 7f4be09ff700 1 --2- 172.21.15.27:0/2819485292 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bdc0b2b90 0x7f4bdc159200 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0009910 tx=0x7f4bd00091a0 comp rx=0 tx=0).stop 2024-01-11T08:28:22.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.621+0000 7f4be09ff700 1 -- 172.21.15.27:0/2819485292 shutdown_connections 2024-01-11T08:28:22.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.621+0000 7f4be09ff700 1 --2- 172.21.15.27:0/2819485292 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bdc0b2b90 0x7f4bdc159200 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:22.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.621+0000 7f4be09ff700 1 -- 172.21.15.27:0/2819485292 >> 172.21.15.27:0/2819485292 conn(0x7f4bdc06ed40 msgr2=0x7f4bdc06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:22.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.621+0000 7f4be09ff700 1 -- 172.21.15.27:0/2819485292 shutdown_connections 2024-01-11T08:28:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.621+0000 7f4be09ff700 1 -- 172.21.15.27:0/2819485292 wait complete. 2024-01-11T08:28:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.622+0000 7f4be09ff700 1 Processor -- start 2024-01-11T08:28:22.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.623+0000 7f4be09ff700 1 -- start start 2024-01-11T08:28:22.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.623+0000 7f4be09ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bdc0b03e0 0x7f4bdc0ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:22.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.624+0000 7f4be09ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bdc0b07b0 0x7f4bdc158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:22.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.624+0000 7f4be09ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bdc0b2b90 0x7f4bdc0c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:22.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.624+0000 7f4be09ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc169400 con 0x7f4bdc0b07b0 2024-01-11T08:28:22.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.624+0000 7f4be09ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc0cfbe0 con 0x7f4bdc0b03e0 2024-01-11T08:28:22.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.624+0000 7f4be09ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc0cf340 con 0x7f4bdc0b2b90 2024-01-11T08:28:22.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.624+0000 7f4bd9d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bdc0b07b0 0x7f4bdc158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:22.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.624+0000 7f4bd9d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bdc0b07b0 0x7f4bdc158d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34128/0 (socket says 172.21.15.27:34128) 2024-01-11T08:28:22.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.624+0000 7f4bd9d9b700 1 -- 172.21.15.27:0/325853820 learned_addr learned my addr 172.21.15.27:0/325853820 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:22.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.624+0000 7f4bda59c700 1 --2- 172.21.15.27:0/325853820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bdc0b03e0 0x7f4bdc0ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:22.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.624+0000 7f4bdad9d700 1 --2- 172.21.15.27:0/325853820 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bdc0b2b90 0x7f4bdc0c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:22.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.624+0000 7f4bd9d9b700 1 -- 172.21.15.27:0/325853820 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bdc0b2b90 msgr2=0x7f4bdc0c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:22.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.624+0000 7f4bd9d9b700 1 --2- 172.21.15.27:0/325853820 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bdc0b2b90 0x7f4bdc0c8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.624+0000 7f4bd9d9b700 1 -- 172.21.15.27:0/325853820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bdc0b03e0 msgr2=0x7f4bdc0ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.624+0000 7f4bd9d9b700 1 --2- 172.21.15.27:0/325853820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bdc0b03e0 0x7f4bdc0ceaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.624+0000 7f4bd9d9b700 1 -- 172.21.15.27:0/325853820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd0009420 con 0x7f4bdc0b07b0 2024-01-11T08:28:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.624+0000 7f4bdad9d700 1 --2- 172.21.15.27:0/325853820 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bdc0b2b90 0x7f4bdc0c8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.625+0000 7f4bd9d9b700 1 --2- 172.21.15.27:0/325853820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bdc0b07b0 0x7f4bdc158d90 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc069540 tx=0x7f4bd0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:22.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.625+0000 7f4bca7fc700 1 -- 172.21.15.27:0/325853820 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0000cc0 con 0x7f4bdc0b07b0 2024-01-11T08:28:22.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.625+0000 7f4bca7fc700 1 -- 172.21.15.27:0/325853820 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd000bb60 con 0x7f4bdc0b07b0 2024-01-11T08:28:22.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.625+0000 7f4be09ff700 1 -- 172.21.15.27:0/325853820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bdc0cf620 con 0x7f4bdc0b07b0 2024-01-11T08:28:22.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.625+0000 7f4bca7fc700 1 -- 172.21.15.27:0/325853820 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0027680 con 0x7f4bdc0b07b0 2024-01-11T08:28:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.625+0000 7f4be09ff700 1 -- 172.21.15.27:0/325853820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bdc169610 con 0x7f4bdc0b07b0 2024-01-11T08:28:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.626+0000 7f4be09ff700 1 -- 172.21.15.27:0/325853820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bdc169800 con 0x7f4bdc0b07b0 2024-01-11T08:28:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.627+0000 7f4bca7fc700 1 -- 172.21.15.27:0/325853820 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4bd00279f0 con 0x7f4bdc0b07b0 2024-01-11T08:28:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.629+0000 7f4bca7fc700 1 --2- 172.21.15.27:0/325853820 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4bbc066f30 0x7f4bbc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.629+0000 7f4bca7fc700 1 -- 172.21.15.27:0/325853820 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f4bd0086540 con 0x7f4bdc0b07b0 2024-01-11T08:28:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.629+0000 7f4bda59c700 1 --2- 172.21.15.27:0/325853820 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4bbc066f30 0x7f4bbc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.630+0000 7f4bda59c700 1 --2- 172.21.15.27:0/325853820 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4bbc066f30 0x7f4bbc0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc007fd0 tx=0x7f4bcc009040 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:22.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.633+0000 7f4bca7fc700 1 -- 172.21.15.27:0/325853820 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4bd0053790 con 0x7f4bdc0b07b0 2024-01-11T08:28:22.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.830+0000 7f4be09ff700 1 -- 172.21.15.27:0/325853820 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4bdc0718d0 con 0x7f4bbc066f30 2024-01-11T08:28:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.839+0000 7f4bca7fc700 1 -- 172.21.15.27:0/325853820 <== mgr.7177 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289727 (secure 0 0 0) 0x7f4bdc0718d0 con 0x7f4bbc066f30 2024-01-11T08:28:22.842 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:28:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.843+0000 7f4be09ff700 1 -- 172.21.15.27:0/325853820 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4bbc066f30 msgr2=0x7f4bbc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.843+0000 7f4be09ff700 1 --2- 172.21.15.27:0/325853820 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4bbc066f30 0x7f4bbc0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc007fd0 tx=0x7f4bcc009040 comp rx=0 tx=0).stop 2024-01-11T08:28:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.843+0000 7f4be09ff700 1 -- 172.21.15.27:0/325853820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bdc0b07b0 msgr2=0x7f4bdc158d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:22.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.843+0000 7f4be09ff700 1 --2- 172.21.15.27:0/325853820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bdc0b07b0 0x7f4bdc158d90 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc069540 tx=0x7f4bd0017000 comp rx=0 tx=0).stop 2024-01-11T08:28:22.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.843+0000 7f4be09ff700 1 -- 172.21.15.27:0/325853820 shutdown_connections 2024-01-11T08:28:22.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.843+0000 7f4be09ff700 1 --2- 172.21.15.27:0/325853820 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4bbc066f30 0x7f4bbc0693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:22.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.843+0000 7f4be09ff700 1 --2- 172.21.15.27:0/325853820 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bdc0b2b90 0x7f4bdc0c8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:22.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.843+0000 7f4be09ff700 1 --2- 172.21.15.27:0/325853820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bdc0b03e0 0x7f4bdc0ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:22.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.843+0000 7f4be09ff700 1 --2- 172.21.15.27:0/325853820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bdc0b07b0 0x7f4bdc158d90 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:22.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.843+0000 7f4be09ff700 1 -- 172.21.15.27:0/325853820 >> 172.21.15.27:0/325853820 conn(0x7f4bdc06ed40 msgr2=0x7f4bdc159740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:22.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.843+0000 7f4be09ff700 1 -- 172.21.15.27:0/325853820 shutdown_connections 2024-01-11T08:28:22.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:22.843+0000 7f4be09ff700 1 -- 172.21.15.27:0/325853820 wait complete. 2024-01-11T08:28:22.845 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:28:22.860 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-11T08:28:22.077222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1040,"num_read_kb":2130,"num_write":395,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":261,"ondisk_log_size":261,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":152,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2443976,"kb_used_data":12424,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752530744,"statfs":{"total":773094113280,"available":770591481856,"internally_reserved":0,"allocated":12722176,"data_stored":6516116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001285"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.990268+0000","last_change":"2024-01-11T08:28:17.984391+0000","last_active":"2024-01-11T08:28:17.990268+0000","last_peered":"2024-01-11T08:28:17.990268+0000","last_clean":"2024-01-11T08:28:17.990268+0000","last_became_active":"2024-01-11T08:28:17.984143+0000","last_became_peered":"2024-01-11T08:28:17.984143+0000","last_unstale":"2024-01-11T08:28:17.990268+0000","last_undegraded":"2024-01-11T08:28:17.990268+0000","last_fullsized":"2024-01-11T08:28:17.990268+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:52:47.412887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.987135+0000","last_change":"2024-01-11T08:28:17.985653+0000","last_active":"2024-01-11T08:28:17.987135+0000","last_peered":"2024-01-11T08:28:17.987135+0000","last_clean":"2024-01-11T08:28:17.987135+0000","last_became_active":"2024-01-11T08:28:17.985140+0000","last_became_peered":"2024-01-11T08:28:17.985140+0000","last_unstale":"2024-01-11T08:28:17.987135+0000","last_undegraded":"2024-01-11T08:28:17.987135+0000","last_fullsized":"2024-01-11T08:28:17.987135+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:01:19.046780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.992281+0000","last_change":"2024-01-11T08:28:17.988746+0000","last_active":"2024-01-11T08:28:17.992281+0000","last_peered":"2024-01-11T08:28:17.992281+0000","last_clean":"2024-01-11T08:28:17.992281+0000","last_became_active":"2024-01-11T08:28:17.987819+0000","last_became_peered":"2024-01-11T08:28:17.987819+0000","last_unstale":"2024-01-11T08:28:17.992281+0000","last_undegraded":"2024-01-11T08:28:17.992281+0000","last_fullsized":"2024-01-11T08:28:17.992281+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:17:37.552135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.986492+0000","last_change":"2024-01-11T08:28:17.982122+0000","last_active":"2024-01-11T08:28:17.986492+0000","last_peered":"2024-01-11T08:28:17.986492+0000","last_clean":"2024-01-11T08:28:17.986492+0000","last_became_active":"2024-01-11T08:28:17.981742+0000","last_became_peered":"2024-01-11T08:28:17.981742+0000","last_unstale":"2024-01-11T08:28:17.986492+0000","last_undegraded":"2024-01-11T08:28:17.986492+0000","last_fullsized":"2024-01-11T08:28:17.986492+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:12:51.077671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.977919+0000","last_change":"2024-01-11T08:28:17.975351+0000","last_active":"2024-01-11T08:28:17.977919+0000","last_peered":"2024-01-11T08:28:17.977919+0000","last_clean":"2024-01-11T08:28:17.977919+0000","last_became_active":"2024-01-11T08:28:17.974419+0000","last_became_peered":"2024-01-11T08:28:17.974419+0000","last_unstale":"2024-01-11T08:28:17.977919+0000","last_undegraded":"2024-01-11T08:28:17.977919+0000","last_fullsized":"2024-01-11T08:28:17.977919+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:40:34.768302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.986188+0000","last_change":"2024-01-11T08:28:17.984079+0000","last_active":"2024-01-11T08:28:17.986188+0000","last_peered":"2024-01-11T08:28:17.986188+0000","last_clean":"2024-01-11T08:28:17.986188+0000","last_became_active":"2024-01-11T08:28:17.983716+0000","last_became_peered":"2024-01-11T08:28:17.983716+0000","last_unstale":"2024-01-11T08:28:17.986188+0000","last_undegraded":"2024-01-11T08:28:17.986188+0000","last_fullsized":"2024-01-11T08:28:17.986188+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:35:44.632717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.985143+0000","last_change":"2024-01-11T08:28:17.982880+0000","last_active":"2024-01-11T08:28:17.985143+0000","last_peered":"2024-01-11T08:28:17.985143+0000","last_clean":"2024-01-11T08:28:17.985143+0000","last_became_active":"2024-01-11T08:28:17.982557+0000","last_became_peered":"2024-01-11T08:28:17.982557+0000","last_unstale":"2024-01-11T08:28:17.985143+0000","last_undegraded":"2024-01-11T08:28:17.985143+0000","last_fullsized":"2024-01-11T08:28:17.985143+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:36:53.350082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.976053+0000","last_change":"2024-01-11T08:28:17.968873+0000","last_active":"2024-01-11T08:28:17.976053+0000","last_peered":"2024-01-11T08:28:17.976053+0000","last_clean":"2024-01-11T08:28:17.976053+0000","last_became_active":"2024-01-11T08:28:17.968506+0000","last_became_peered":"2024-01-11T08:28:17.968506+0000","last_unstale":"2024-01-11T08:28:17.976053+0000","last_undegraded":"2024-01-11T08:28:17.976053+0000","last_fullsized":"2024-01-11T08:28:17.976053+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:08:30.335895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.970953+0000","last_change":"2024-01-11T08:28:17.968244+0000","last_active":"2024-01-11T08:28:17.970953+0000","last_peered":"2024-01-11T08:28:17.970953+0000","last_clean":"2024-01-11T08:28:17.970953+0000","last_became_active":"2024-01-11T08:28:17.966994+0000","last_became_peered":"2024-01-11T08:28:17.966994+0000","last_unstale":"2024-01-11T08:28:17.970953+0000","last_undegraded":"2024-01-11T08:28:17.970953+0000","last_fullsized":"2024-01-11T08:28:17.970953+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:51:42.562966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.978277+0000","last_change":"2024-01-11T08:28:17.976009+0000","last_active":"2024-01-11T08:28:17.978277+0000","last_peered":"2024-01-11T08:28:17.978277+0000","last_clean":"2024-01-11T08:28:17.978277+0000","last_became_active":"2024-01-11T08:28:17.975719+0000","last_became_peered":"2024-01-11T08:28:17.975719+0000","last_unstale":"2024-01-11T08:28:17.978277+0000","last_undegraded":"2024-01-11T08:28:17.978277+0000","last_fullsized":"2024-01-11T08:28:17.978277+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:02:38.249774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.970437+0000","last_change":"2024-01-11T08:28:17.967873+0000","last_active":"2024-01-11T08:28:17.970437+0000","last_peered":"2024-01-11T08:28:17.970437+0000","last_clean":"2024-01-11T08:28:17.970437+0000","last_became_active":"2024-01-11T08:28:17.966819+0000","last_became_peered":"2024-01-11T08:28:17.966819+0000","last_unstale":"2024-01-11T08:28:17.970437+0000","last_undegraded":"2024-01-11T08:28:17.970437+0000","last_fullsized":"2024-01-11T08:28:17.970437+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:01:23.824852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.975465+0000","last_change":"2024-01-11T08:28:17.972272+0000","last_active":"2024-01-11T08:28:17.975465+0000","last_peered":"2024-01-11T08:28:17.975465+0000","last_clean":"2024-01-11T08:28:17.975465+0000","last_became_active":"2024-01-11T08:28:17.971704+0000","last_became_peered":"2024-01-11T08:28:17.971704+0000","last_unstale":"2024-01-11T08:28:17.975465+0000","last_undegraded":"2024-01-11T08:28:17.975465+0000","last_fullsized":"2024-01-11T08:28:17.975465+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:01:19.046780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.968779+0000","last_change":"2024-01-11T08:28:17.967441+0000","last_active":"2024-01-11T08:28:17.968779+0000","last_peered":"2024-01-11T08:28:17.968779+0000","last_clean":"2024-01-11T08:28:17.968779+0000","last_became_active":"2024-01-11T08:28:17.966709+0000","last_became_peered":"2024-01-11T08:28:17.966709+0000","last_unstale":"2024-01-11T08:28:17.968779+0000","last_undegraded":"2024-01-11T08:28:17.968779+0000","last_fullsized":"2024-01-11T08:28:17.968779+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:19:53.198221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.984566+0000","last_change":"2024-01-11T08:28:17.977606+0000","last_active":"2024-01-11T08:28:17.984566+0000","last_peered":"2024-01-11T08:28:17.984566+0000","last_clean":"2024-01-11T08:28:17.984566+0000","last_became_active":"2024-01-11T08:28:17.977377+0000","last_became_peered":"2024-01-11T08:28:17.977377+0000","last_unstale":"2024-01-11T08:28:17.984566+0000","last_undegraded":"2024-01-11T08:28:17.984566+0000","last_fullsized":"2024-01-11T08:28:17.984566+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:07:27.285241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.977164+0000","last_change":"2024-01-11T08:28:17.972733+0000","last_active":"2024-01-11T08:28:17.977164+0000","last_peered":"2024-01-11T08:28:17.977164+0000","last_clean":"2024-01-11T08:28:17.977164+0000","last_became_active":"2024-01-11T08:28:17.972154+0000","last_became_peered":"2024-01-11T08:28:17.972154+0000","last_unstale":"2024-01-11T08:28:17.977164+0000","last_undegraded":"2024-01-11T08:28:17.977164+0000","last_fullsized":"2024-01-11T08:28:17.977164+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:28:37.430387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.973281+0000","last_change":"2024-01-11T08:28:17.970813+0000","last_active":"2024-01-11T08:28:17.973281+0000","last_peered":"2024-01-11T08:28:17.973281+0000","last_clean":"2024-01-11T08:28:17.973281+0000","last_became_active":"2024-01-11T08:28:17.969930+0000","last_became_peered":"2024-01-11T08:28:17.969930+0000","last_unstale":"2024-01-11T08:28:17.973281+0000","last_undegraded":"2024-01-11T08:28:17.973281+0000","last_fullsized":"2024-01-11T08:28:17.973281+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:53:55.216533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.977834+0000","last_change":"2024-01-11T08:28:17.970347+0000","last_active":"2024-01-11T08:28:17.977834+0000","last_peered":"2024-01-11T08:28:17.977834+0000","last_clean":"2024-01-11T08:28:17.977834+0000","last_became_active":"2024-01-11T08:28:17.970104+0000","last_became_peered":"2024-01-11T08:28:17.970104+0000","last_unstale":"2024-01-11T08:28:17.977834+0000","last_undegraded":"2024-01-11T08:28:17.977834+0000","last_fullsized":"2024-01-11T08:28:17.977834+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:39:18.705936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.992320+0000","last_change":"2024-01-11T08:28:17.986999+0000","last_active":"2024-01-11T08:28:17.992320+0000","last_peered":"2024-01-11T08:28:17.992320+0000","last_clean":"2024-01-11T08:28:17.992320+0000","last_became_active":"2024-01-11T08:28:17.986574+0000","last_became_peered":"2024-01-11T08:28:17.986574+0000","last_unstale":"2024-01-11T08:28:17.992320+0000","last_undegraded":"2024-01-11T08:28:17.992320+0000","last_fullsized":"2024-01-11T08:28:17.992320+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:35:44.632717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.977070+0000","last_change":"2024-01-11T08:28:17.971433+0000","last_active":"2024-01-11T08:28:17.977070+0000","last_peered":"2024-01-11T08:28:17.977070+0000","last_clean":"2024-01-11T08:28:17.977070+0000","last_became_active":"2024-01-11T08:28:17.971179+0000","last_became_peered":"2024-01-11T08:28:17.971179+0000","last_unstale":"2024-01-11T08:28:17.977070+0000","last_undegraded":"2024-01-11T08:28:17.977070+0000","last_fullsized":"2024-01-11T08:28:17.977070+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:37:35.976533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.975441+0000","last_change":"2024-01-11T08:28:17.972037+0000","last_active":"2024-01-11T08:28:17.975441+0000","last_peered":"2024-01-11T08:28:17.975441+0000","last_clean":"2024-01-11T08:28:17.975441+0000","last_became_active":"2024-01-11T08:28:17.971177+0000","last_became_peered":"2024-01-11T08:28:17.971177+0000","last_unstale":"2024-01-11T08:28:17.975441+0000","last_undegraded":"2024-01-11T08:28:17.975441+0000","last_fullsized":"2024-01-11T08:28:17.975441+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:40:34.768302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.984626+0000","last_change":"2024-01-11T08:28:17.976823+0000","last_active":"2024-01-11T08:28:17.984626+0000","last_peered":"2024-01-11T08:28:17.984626+0000","last_clean":"2024-01-11T08:28:17.984626+0000","last_became_active":"2024-01-11T08:28:17.976534+0000","last_became_peered":"2024-01-11T08:28:17.976534+0000","last_unstale":"2024-01-11T08:28:17.984626+0000","last_undegraded":"2024-01-11T08:28:17.984626+0000","last_fullsized":"2024-01-11T08:28:17.984626+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:48:49.108760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.984077+0000","last_change":"2024-01-11T08:28:17.982228+0000","last_active":"2024-01-11T08:28:17.984077+0000","last_peered":"2024-01-11T08:28:17.984077+0000","last_clean":"2024-01-11T08:28:17.984077+0000","last_became_active":"2024-01-11T08:28:17.981749+0000","last_became_peered":"2024-01-11T08:28:17.981749+0000","last_unstale":"2024-01-11T08:28:17.984077+0000","last_undegraded":"2024-01-11T08:28:17.984077+0000","last_fullsized":"2024-01-11T08:28:17.984077+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:20:14.569050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.983138+0000","last_change":"2024-01-11T08:28:17.982015+0000","last_active":"2024-01-11T08:28:17.983138+0000","last_peered":"2024-01-11T08:28:17.983138+0000","last_clean":"2024-01-11T08:28:17.983138+0000","last_became_active":"2024-01-11T08:28:17.981770+0000","last_became_peered":"2024-01-11T08:28:17.981770+0000","last_unstale":"2024-01-11T08:28:17.983138+0000","last_undegraded":"2024-01-11T08:28:17.983138+0000","last_fullsized":"2024-01-11T08:28:17.983138+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:11:37.602642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.973493+0000","last_change":"2024-01-11T08:28:17.972434+0000","last_active":"2024-01-11T08:28:17.973493+0000","last_peered":"2024-01-11T08:28:17.973493+0000","last_clean":"2024-01-11T08:28:17.973493+0000","last_became_active":"2024-01-11T08:28:17.972139+0000","last_became_peered":"2024-01-11T08:28:17.972139+0000","last_unstale":"2024-01-11T08:28:17.973493+0000","last_undegraded":"2024-01-11T08:28:17.973493+0000","last_fullsized":"2024-01-11T08:28:17.973493+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:33:46.515399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.981093+0000","last_change":"2024-01-11T08:28:17.978145+0000","last_active":"2024-01-11T08:28:17.981093+0000","last_peered":"2024-01-11T08:28:17.981093+0000","last_clean":"2024-01-11T08:28:17.981093+0000","last_became_active":"2024-01-11T08:28:17.977860+0000","last_became_peered":"2024-01-11T08:28:17.977860+0000","last_unstale":"2024-01-11T08:28:17.981093+0000","last_undegraded":"2024-01-11T08:28:17.981093+0000","last_fullsized":"2024-01-11T08:28:17.981093+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:51:42.562966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.979436+0000","last_change":"2024-01-11T08:28:17.976159+0000","last_active":"2024-01-11T08:28:17.979436+0000","last_peered":"2024-01-11T08:28:17.979436+0000","last_clean":"2024-01-11T08:28:17.979436+0000","last_became_active":"2024-01-11T08:28:17.975867+0000","last_became_peered":"2024-01-11T08:28:17.975867+0000","last_unstale":"2024-01-11T08:28:17.979436+0000","last_undegraded":"2024-01-11T08:28:17.979436+0000","last_fullsized":"2024-01-11T08:28:17.979436+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:47:38.042745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.971700+0000","last_change":"2024-01-11T08:28:17.968397+0000","last_active":"2024-01-11T08:28:17.971700+0000","last_peered":"2024-01-11T08:28:17.971700+0000","last_clean":"2024-01-11T08:28:17.971700+0000","last_became_active":"2024-01-11T08:28:17.967576+0000","last_became_peered":"2024-01-11T08:28:17.967576+0000","last_unstale":"2024-01-11T08:28:17.971700+0000","last_undegraded":"2024-01-11T08:28:17.971700+0000","last_fullsized":"2024-01-11T08:28:17.971700+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:09:13.812238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.980233+0000","last_change":"2024-01-11T08:28:17.977400+0000","last_active":"2024-01-11T08:28:17.980233+0000","last_peered":"2024-01-11T08:28:17.980233+0000","last_clean":"2024-01-11T08:28:17.980233+0000","last_became_active":"2024-01-11T08:28:17.977064+0000","last_became_peered":"2024-01-11T08:28:17.977064+0000","last_unstale":"2024-01-11T08:28:17.980233+0000","last_undegraded":"2024-01-11T08:28:17.980233+0000","last_fullsized":"2024-01-11T08:28:17.980233+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:01:19.046780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.977534+0000","last_change":"2024-01-11T08:28:17.974036+0000","last_active":"2024-01-11T08:28:17.977534+0000","last_peered":"2024-01-11T08:28:17.977534+0000","last_clean":"2024-01-11T08:28:17.977534+0000","last_became_active":"2024-01-11T08:28:17.973756+0000","last_became_peered":"2024-01-11T08:28:17.973756+0000","last_unstale":"2024-01-11T08:28:17.977534+0000","last_undegraded":"2024-01-11T08:28:17.977534+0000","last_fullsized":"2024-01-11T08:28:17.977534+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:40:34.768302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.981315+0000","last_change":"2024-01-11T08:28:17.973583+0000","last_active":"2024-01-11T08:28:17.981315+0000","last_peered":"2024-01-11T08:28:17.981315+0000","last_clean":"2024-01-11T08:28:17.981315+0000","last_became_active":"2024-01-11T08:28:17.973043+0000","last_became_peered":"2024-01-11T08:28:17.973043+0000","last_unstale":"2024-01-11T08:28:17.981315+0000","last_undegraded":"2024-01-11T08:28:17.981315+0000","last_fullsized":"2024-01-11T08:28:17.981315+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:53:27.932566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.980408+0000","last_change":"2024-01-11T08:28:17.978364+0000","last_active":"2024-01-11T08:28:17.980408+0000","last_peered":"2024-01-11T08:28:17.980408+0000","last_clean":"2024-01-11T08:28:17.980408+0000","last_became_active":"2024-01-11T08:28:17.977163+0000","last_became_peered":"2024-01-11T08:28:17.977163+0000","last_unstale":"2024-01-11T08:28:17.980408+0000","last_undegraded":"2024-01-11T08:28:17.980408+0000","last_fullsized":"2024-01-11T08:28:17.980408+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:36:53.350082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.974534+0000","last_change":"2024-01-11T08:28:17.972309+0000","last_active":"2024-01-11T08:28:17.974534+0000","last_peered":"2024-01-11T08:28:17.974534+0000","last_clean":"2024-01-11T08:28:17.974534+0000","last_became_active":"2024-01-11T08:28:17.972002+0000","last_became_peered":"2024-01-11T08:28:17.972002+0000","last_unstale":"2024-01-11T08:28:17.974534+0000","last_undegraded":"2024-01-11T08:28:17.974534+0000","last_fullsized":"2024-01-11T08:28:17.974534+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:57:08.672492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.975485+0000","last_change":"2024-01-11T08:28:17.968542+0000","last_active":"2024-01-11T08:28:17.975485+0000","last_peered":"2024-01-11T08:28:17.975485+0000","last_clean":"2024-01-11T08:28:17.975485+0000","last_became_active":"2024-01-11T08:28:17.967756+0000","last_became_peered":"2024-01-11T08:28:17.967756+0000","last_unstale":"2024-01-11T08:28:17.975485+0000","last_undegraded":"2024-01-11T08:28:17.975485+0000","last_fullsized":"2024-01-11T08:28:17.975485+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:22:27.717871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"115'4","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.991569+0000","last_change":"2024-01-11T08:28:16.970927+0000","last_active":"2024-01-11T08:28:17.991569+0000","last_peered":"2024-01-11T08:28:17.991569+0000","last_clean":"2024-01-11T08:28:17.991569+0000","last_became_active":"2024-01-11T08:28:16.970654+0000","last_became_peered":"2024-01-11T08:28:16.970654+0000","last_unstale":"2024-01-11T08:28:17.991569+0000","last_undegraded":"2024-01-11T08:28:17.991569+0000","last_fullsized":"2024-01-11T08:28:17.991569+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:32:21.280368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.980349+0000","last_change":"2024-01-11T08:28:17.977574+0000","last_active":"2024-01-11T08:28:17.980349+0000","last_peered":"2024-01-11T08:28:17.980349+0000","last_clean":"2024-01-11T08:28:17.980349+0000","last_became_active":"2024-01-11T08:28:17.977321+0000","last_became_peered":"2024-01-11T08:28:17.977321+0000","last_unstale":"2024-01-11T08:28:17.980349+0000","last_undegraded":"2024-01-11T08:28:17.980349+0000","last_fullsized":"2024-01-11T08:28:17.980349+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:09:13.812238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"115'1","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.987119+0000","last_change":"2024-01-11T08:28:16.969153+0000","last_active":"2024-01-11T08:28:17.987119+0000","last_peered":"2024-01-11T08:28:17.987119+0000","last_clean":"2024-01-11T08:28:17.987119+0000","last_became_active":"2024-01-11T08:28:16.968902+0000","last_became_peered":"2024-01-11T08:28:16.968902+0000","last_unstale":"2024-01-11T08:28:17.987119+0000","last_undegraded":"2024-01-11T08:28:17.987119+0000","last_fullsized":"2024-01-11T08:28:17.987119+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:41:30.011035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.972877+0000","last_change":"2024-01-11T08:28:17.969705+0000","last_active":"2024-01-11T08:28:17.972877+0000","last_peered":"2024-01-11T08:28:17.972877+0000","last_clean":"2024-01-11T08:28:17.972877+0000","last_became_active":"2024-01-11T08:28:17.969348+0000","last_became_peered":"2024-01-11T08:28:17.969348+0000","last_unstale":"2024-01-11T08:28:17.972877+0000","last_undegraded":"2024-01-11T08:28:17.972877+0000","last_fullsized":"2024-01-11T08:28:17.972877+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:40:29.251257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"115'2","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.990448+0000","last_change":"2024-01-11T08:28:16.975283+0000","last_active":"2024-01-11T08:28:17.990448+0000","last_peered":"2024-01-11T08:28:17.990448+0000","last_clean":"2024-01-11T08:28:17.990448+0000","last_became_active":"2024-01-11T08:28:16.973391+0000","last_became_peered":"2024-01-11T08:28:16.973391+0000","last_unstale":"2024-01-11T08:28:17.990448+0000","last_undegraded":"2024-01-11T08:28:17.990448+0000","last_fullsized":"2024-01-11T08:28:17.990448+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:01:17.535401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.968424+0000","last_change":"2024-01-11T08:28:17.965959+0000","last_active":"2024-01-11T08:28:17.968424+0000","last_peered":"2024-01-11T08:28:17.968424+0000","last_clean":"2024-01-11T08:28:17.968424+0000","last_became_active":"2024-01-11T08:28:17.965502+0000","last_became_peered":"2024-01-11T08:28:17.965502+0000","last_unstale":"2024-01-11T08:28:17.968424+0000","last_undegraded":"2024-01-11T08:28:17.968424+0000","last_fullsized":"2024-01-11T08:28:17.968424+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:42:56.802832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"115'2","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.987287+0000","last_change":"2024-01-11T08:28:16.970018+0000","last_active":"2024-01-11T08:28:17.987287+0000","last_peered":"2024-01-11T08:28:17.987287+0000","last_clean":"2024-01-11T08:28:17.987287+0000","last_became_active":"2024-01-11T08:28:16.968638+0000","last_became_peered":"2024-01-11T08:28:16.968638+0000","last_unstale":"2024-01-11T08:28:17.987287+0000","last_undegraded":"2024-01-11T08:28:17.987287+0000","last_fullsized":"2024-01-11T08:28:17.987287+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:18:42.004051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.980659+0000","last_change":"2024-01-11T08:28:17.967920+0000","last_active":"2024-01-11T08:28:17.980659+0000","last_peered":"2024-01-11T08:28:17.980659+0000","last_clean":"2024-01-11T08:28:17.980659+0000","last_became_active":"2024-01-11T08:28:17.967637+0000","last_became_peered":"2024-01-11T08:28:17.967637+0000","last_unstale":"2024-01-11T08:28:17.980659+0000","last_undegraded":"2024-01-11T08:28:17.980659+0000","last_fullsized":"2024-01-11T08:28:17.980659+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:23:39.775217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"115'1","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.992178+0000","last_change":"2024-01-11T08:28:16.965022+0000","last_active":"2024-01-11T08:28:17.992178+0000","last_peered":"2024-01-11T08:28:17.992178+0000","last_clean":"2024-01-11T08:28:17.992178+0000","last_became_active":"2024-01-11T08:28:16.964707+0000","last_became_peered":"2024-01-11T08:28:16.964707+0000","last_unstale":"2024-01-11T08:28:17.992178+0000","last_undegraded":"2024-01-11T08:28:17.992178+0000","last_fullsized":"2024-01-11T08:28:17.992178+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:17:29.823696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.980450+0000","last_change":"2024-01-11T08:28:17.977960+0000","last_active":"2024-01-11T08:28:17.980450+0000","last_peered":"2024-01-11T08:28:17.980450+0000","last_clean":"2024-01-11T08:28:17.980450+0000","last_became_active":"2024-01-11T08:28:17.977596+0000","last_became_peered":"2024-01-11T08:28:17.977596+0000","last_unstale":"2024-01-11T08:28:17.980450+0000","last_undegraded":"2024-01-11T08:28:17.980450+0000","last_fullsized":"2024-01-11T08:28:17.980450+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:51:46.671037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"115'1","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.990500+0000","last_change":"2024-01-11T08:28:16.975205+0000","last_active":"2024-01-11T08:28:17.990500+0000","last_peered":"2024-01-11T08:28:17.990500+0000","last_clean":"2024-01-11T08:28:17.990500+0000","last_became_active":"2024-01-11T08:28:16.974114+0000","last_became_peered":"2024-01-11T08:28:16.974114+0000","last_unstale":"2024-01-11T08:28:17.990500+0000","last_undegraded":"2024-01-11T08:28:17.990500+0000","last_fullsized":"2024-01-11T08:28:17.990500+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:26:59.400728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.986543+0000","last_change":"2024-01-11T08:28:17.982333+0000","last_active":"2024-01-11T08:28:17.986543+0000","last_peered":"2024-01-11T08:28:17.986543+0000","last_clean":"2024-01-11T08:28:17.986543+0000","last_became_active":"2024-01-11T08:28:17.976709+0000","last_became_peered":"2024-01-11T08:28:17.976709+0000","last_unstale":"2024-01-11T08:28:17.986543+0000","last_undegraded":"2024-01-11T08:28:17.986543+0000","last_fullsized":"2024-01-11T08:28:17.986543+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:18:43.010725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.961779+0000","last_change":"2024-01-11T08:28:16.968940+0000","last_active":"2024-01-11T08:28:17.961779+0000","last_peered":"2024-01-11T08:28:17.961779+0000","last_clean":"2024-01-11T08:28:17.961779+0000","last_became_active":"2024-01-11T08:28:16.968628+0000","last_became_peered":"2024-01-11T08:28:16.968628+0000","last_unstale":"2024-01-11T08:28:17.961779+0000","last_undegraded":"2024-01-11T08:28:17.961779+0000","last_fullsized":"2024-01-11T08:28:17.961779+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:10:07.272755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.965293+0000","last_change":"2024-01-11T08:28:17.961408+0000","last_active":"2024-01-11T08:28:17.965293+0000","last_peered":"2024-01-11T08:28:17.965293+0000","last_clean":"2024-01-11T08:28:17.965293+0000","last_became_active":"2024-01-11T08:28:17.961001+0000","last_became_peered":"2024-01-11T08:28:17.961001+0000","last_unstale":"2024-01-11T08:28:17.965293+0000","last_undegraded":"2024-01-11T08:28:17.965293+0000","last_fullsized":"2024-01-11T08:28:17.965293+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:02:38.249774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.961275+0000","last_change":"2024-01-11T08:28:16.970181+0000","last_active":"2024-01-11T08:28:17.961275+0000","last_peered":"2024-01-11T08:28:17.961275+0000","last_clean":"2024-01-11T08:28:17.961275+0000","last_became_active":"2024-01-11T08:28:16.969099+0000","last_became_peered":"2024-01-11T08:28:16.969099+0000","last_unstale":"2024-01-11T08:28:17.961275+0000","last_undegraded":"2024-01-11T08:28:17.961275+0000","last_fullsized":"2024-01-11T08:28:17.961275+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:07:55.263842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.979456+0000","last_change":"2024-01-11T08:28:17.976644+0000","last_active":"2024-01-11T08:28:17.979456+0000","last_peered":"2024-01-11T08:28:17.979456+0000","last_clean":"2024-01-11T08:28:17.979456+0000","last_became_active":"2024-01-11T08:28:17.976392+0000","last_became_peered":"2024-01-11T08:28:17.976392+0000","last_unstale":"2024-01-11T08:28:17.979456+0000","last_undegraded":"2024-01-11T08:28:17.979456+0000","last_fullsized":"2024-01-11T08:28:17.979456+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:32:23.513567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.959453+0000","last_change":"2024-01-11T08:28:16.966688+0000","last_active":"2024-01-11T08:28:17.959453+0000","last_peered":"2024-01-11T08:28:17.959453+0000","last_clean":"2024-01-11T08:28:17.959453+0000","last_became_active":"2024-01-11T08:28:16.966425+0000","last_became_peered":"2024-01-11T08:28:16.966425+0000","last_unstale":"2024-01-11T08:28:17.959453+0000","last_undegraded":"2024-01-11T08:28:17.959453+0000","last_fullsized":"2024-01-11T08:28:17.959453+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:55:17.169420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.963547+0000","last_change":"2024-01-11T08:28:17.958893+0000","last_active":"2024-01-11T08:28:17.963547+0000","last_peered":"2024-01-11T08:28:17.963547+0000","last_clean":"2024-01-11T08:28:17.963547+0000","last_became_active":"2024-01-11T08:28:17.958617+0000","last_became_peered":"2024-01-11T08:28:17.958617+0000","last_unstale":"2024-01-11T08:28:17.963547+0000","last_undegraded":"2024-01-11T08:28:17.963547+0000","last_fullsized":"2024-01-11T08:28:17.963547+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:53:55.216533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.952974+0000","last_change":"2024-01-11T08:28:16.966676+0000","last_active":"2024-01-11T08:28:17.952974+0000","last_peered":"2024-01-11T08:28:17.952974+0000","last_clean":"2024-01-11T08:28:17.952974+0000","last_became_active":"2024-01-11T08:28:16.966424+0000","last_became_peered":"2024-01-11T08:28:16.966424+0000","last_unstale":"2024-01-11T08:28:17.952974+0000","last_undegraded":"2024-01-11T08:28:17.952974+0000","last_fullsized":"2024-01-11T08:28:17.952974+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:47:11.453713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.981090+0000","last_change":"2024-01-11T08:28:17.977899+0000","last_active":"2024-01-11T08:28:17.981090+0000","last_peered":"2024-01-11T08:28:17.981090+0000","last_clean":"2024-01-11T08:28:17.981090+0000","last_became_active":"2024-01-11T08:28:17.977616+0000","last_became_peered":"2024-01-11T08:28:17.977616+0000","last_unstale":"2024-01-11T08:28:17.981090+0000","last_undegraded":"2024-01-11T08:28:17.981090+0000","last_fullsized":"2024-01-11T08:28:17.981090+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:53:55.216533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"115'3","reported_seq":13,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.988758+0000","last_change":"2024-01-11T08:28:16.968103+0000","last_active":"2024-01-11T08:28:17.988758+0000","last_peered":"2024-01-11T08:28:17.988758+0000","last_clean":"2024-01-11T08:28:17.988758+0000","last_became_active":"2024-01-11T08:28:16.967466+0000","last_became_peered":"2024-01-11T08:28:16.967466+0000","last_unstale":"2024-01-11T08:28:17.988758+0000","last_undegraded":"2024-01-11T08:28:17.988758+0000","last_fullsized":"2024-01-11T08:28:17.988758+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:47:20.068330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.983817+0000","last_change":"2024-01-11T08:28:17.981727+0000","last_active":"2024-01-11T08:28:17.983817+0000","last_peered":"2024-01-11T08:28:17.983817+0000","last_clean":"2024-01-11T08:28:17.983817+0000","last_became_active":"2024-01-11T08:28:17.981277+0000","last_became_peered":"2024-01-11T08:28:17.981277+0000","last_unstale":"2024-01-11T08:28:17.983817+0000","last_undegraded":"2024-01-11T08:28:17.983817+0000","last_fullsized":"2024-01-11T08:28:17.983817+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:02:38.249774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"115'6","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.995227+0000","last_change":"2024-01-11T08:28:16.970694+0000","last_active":"2024-01-11T08:28:17.995227+0000","last_peered":"2024-01-11T08:28:17.995227+0000","last_clean":"2024-01-11T08:28:17.995227+0000","last_became_active":"2024-01-11T08:28:16.970351+0000","last_became_peered":"2024-01-11T08:28:16.970351+0000","last_unstale":"2024-01-11T08:28:17.995227+0000","last_undegraded":"2024-01-11T08:28:17.995227+0000","last_fullsized":"2024-01-11T08:28:17.995227+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:18:42.004051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.967325+0000","last_change":"2024-01-11T08:28:17.959607+0000","last_active":"2024-01-11T08:28:17.967325+0000","last_peered":"2024-01-11T08:28:17.967325+0000","last_clean":"2024-01-11T08:28:17.967325+0000","last_became_active":"2024-01-11T08:28:17.959285+0000","last_became_peered":"2024-01-11T08:28:17.959285+0000","last_unstale":"2024-01-11T08:28:17.967325+0000","last_undegraded":"2024-01-11T08:28:17.967325+0000","last_fullsized":"2024-01-11T08:28:17.967325+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:32:22.287042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"115'2","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.985699+0000","last_change":"2024-01-11T08:28:16.967522+0000","last_active":"2024-01-11T08:28:17.985699+0000","last_peered":"2024-01-11T08:28:17.985699+0000","last_clean":"2024-01-11T08:28:17.985699+0000","last_became_active":"2024-01-11T08:28:16.967230+0000","last_became_peered":"2024-01-11T08:28:16.967230+0000","last_unstale":"2024-01-11T08:28:17.985699+0000","last_undegraded":"2024-01-11T08:28:17.985699+0000","last_fullsized":"2024-01-11T08:28:17.985699+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:02:54.383857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.975608+0000","last_change":"2024-01-11T08:28:17.972587+0000","last_active":"2024-01-11T08:28:17.975608+0000","last_peered":"2024-01-11T08:28:17.975608+0000","last_clean":"2024-01-11T08:28:17.975608+0000","last_became_active":"2024-01-11T08:28:17.972260+0000","last_became_peered":"2024-01-11T08:28:17.972260+0000","last_unstale":"2024-01-11T08:28:17.975608+0000","last_undegraded":"2024-01-11T08:28:17.975608+0000","last_fullsized":"2024-01-11T08:28:17.975608+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:52:19.210529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.963343+0000","last_change":"2024-01-11T08:28:16.969256+0000","last_active":"2024-01-11T08:28:17.963343+0000","last_peered":"2024-01-11T08:28:17.963343+0000","last_clean":"2024-01-11T08:28:17.963343+0000","last_became_active":"2024-01-11T08:28:16.967675+0000","last_became_peered":"2024-01-11T08:28:16.967675+0000","last_unstale":"2024-01-11T08:28:17.963343+0000","last_undegraded":"2024-01-11T08:28:17.963343+0000","last_fullsized":"2024-01-11T08:28:17.963343+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:34:22.883431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.963798+0000","last_change":"2024-01-11T08:28:17.957572+0000","last_active":"2024-01-11T08:28:17.963798+0000","last_peered":"2024-01-11T08:28:17.963798+0000","last_clean":"2024-01-11T08:28:17.963798+0000","last_became_active":"2024-01-11T08:28:17.957182+0000","last_became_peered":"2024-01-11T08:28:17.957182+0000","last_unstale":"2024-01-11T08:28:17.963798+0000","last_undegraded":"2024-01-11T08:28:17.963798+0000","last_fullsized":"2024-01-11T08:28:17.963798+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:01:23.824852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.956093+0000","last_change":"2024-01-11T08:28:16.965549+0000","last_active":"2024-01-11T08:28:17.956093+0000","last_peered":"2024-01-11T08:28:17.956093+0000","last_clean":"2024-01-11T08:28:17.956093+0000","last_became_active":"2024-01-11T08:28:16.964911+0000","last_became_peered":"2024-01-11T08:28:16.964911+0000","last_unstale":"2024-01-11T08:28:17.956093+0000","last_undegraded":"2024-01-11T08:28:17.956093+0000","last_fullsized":"2024-01-11T08:28:17.956093+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:28:36.423713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.976501+0000","last_change":"2024-01-11T08:28:17.968664+0000","last_active":"2024-01-11T08:28:17.976501+0000","last_peered":"2024-01-11T08:28:17.976501+0000","last_clean":"2024-01-11T08:28:17.976501+0000","last_became_active":"2024-01-11T08:28:17.968414+0000","last_became_peered":"2024-01-11T08:28:17.968414+0000","last_unstale":"2024-01-11T08:28:17.976501+0000","last_undegraded":"2024-01-11T08:28:17.976501+0000","last_fullsized":"2024-01-11T08:28:17.976501+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:56:54.199858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"115'8","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.994827+0000","last_change":"2024-01-11T08:28:16.965974+0000","last_active":"2024-01-11T08:28:17.994827+0000","last_peered":"2024-01-11T08:28:17.994827+0000","last_clean":"2024-01-11T08:28:17.994827+0000","last_became_active":"2024-01-11T08:28:16.962890+0000","last_became_peered":"2024-01-11T08:28:16.962890+0000","last_unstale":"2024-01-11T08:28:17.994827+0000","last_undegraded":"2024-01-11T08:28:17.994827+0000","last_fullsized":"2024-01-11T08:28:17.994827+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:56:53.193183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.978026+0000","last_change":"2024-01-11T08:28:17.975082+0000","last_active":"2024-01-11T08:28:17.978026+0000","last_peered":"2024-01-11T08:28:17.978026+0000","last_clean":"2024-01-11T08:28:17.978026+0000","last_became_active":"2024-01-11T08:28:17.974795+0000","last_became_peered":"2024-01-11T08:28:17.974795+0000","last_unstale":"2024-01-11T08:28:17.978026+0000","last_undegraded":"2024-01-11T08:28:17.978026+0000","last_fullsized":"2024-01-11T08:28:17.978026+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:01:23.824852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"115'2","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.992389+0000","last_change":"2024-01-11T08:28:16.970767+0000","last_active":"2024-01-11T08:28:17.992389+0000","last_peered":"2024-01-11T08:28:17.992389+0000","last_clean":"2024-01-11T08:28:17.992389+0000","last_became_active":"2024-01-11T08:28:16.970429+0000","last_became_peered":"2024-01-11T08:28:16.970429+0000","last_unstale":"2024-01-11T08:28:17.992389+0000","last_undegraded":"2024-01-11T08:28:17.992389+0000","last_fullsized":"2024-01-11T08:28:17.992389+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:01:17.535401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.984776+0000","last_change":"2024-01-11T08:28:17.981500+0000","last_active":"2024-01-11T08:28:17.984776+0000","last_peered":"2024-01-11T08:28:17.984776+0000","last_clean":"2024-01-11T08:28:17.984776+0000","last_became_active":"2024-01-11T08:28:17.979291+0000","last_became_peered":"2024-01-11T08:28:17.979291+0000","last_unstale":"2024-01-11T08:28:17.984776+0000","last_undegraded":"2024-01-11T08:28:17.984776+0000","last_fullsized":"2024-01-11T08:28:17.984776+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:10:08.279429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"115'1","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.990413+0000","last_change":"2024-01-11T08:28:16.966840+0000","last_active":"2024-01-11T08:28:17.990413+0000","last_peered":"2024-01-11T08:28:17.990413+0000","last_clean":"2024-01-11T08:28:17.990413+0000","last_became_active":"2024-01-11T08:28:16.966574+0000","last_became_peered":"2024-01-11T08:28:16.966574+0000","last_unstale":"2024-01-11T08:28:17.990413+0000","last_undegraded":"2024-01-11T08:28:17.990413+0000","last_fullsized":"2024-01-11T08:28:17.990413+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:23:38.768543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.971721+0000","last_change":"2024-01-11T08:28:17.968597+0000","last_active":"2024-01-11T08:28:17.971721+0000","last_peered":"2024-01-11T08:28:17.971721+0000","last_clean":"2024-01-11T08:28:17.971721+0000","last_became_active":"2024-01-11T08:28:17.968106+0000","last_became_peered":"2024-01-11T08:28:17.968106+0000","last_unstale":"2024-01-11T08:28:17.971721+0000","last_undegraded":"2024-01-11T08:28:17.971721+0000","last_fullsized":"2024-01-11T08:28:17.971721+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:21:26.125357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.958622+0000","last_change":"2024-01-11T08:28:16.972415+0000","last_active":"2024-01-11T08:28:17.958622+0000","last_peered":"2024-01-11T08:28:17.958622+0000","last_clean":"2024-01-11T08:28:17.958622+0000","last_became_active":"2024-01-11T08:28:16.971434+0000","last_became_peered":"2024-01-11T08:28:16.971434+0000","last_unstale":"2024-01-11T08:28:17.958622+0000","last_undegraded":"2024-01-11T08:28:17.958622+0000","last_fullsized":"2024-01-11T08:28:17.958622+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:28.686024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.977011+0000","last_change":"2024-01-11T08:28:17.969665+0000","last_active":"2024-01-11T08:28:17.977011+0000","last_peered":"2024-01-11T08:28:17.977011+0000","last_clean":"2024-01-11T08:28:17.977011+0000","last_became_active":"2024-01-11T08:28:17.969419+0000","last_became_peered":"2024-01-11T08:28:17.969419+0000","last_unstale":"2024-01-11T08:28:17.977011+0000","last_undegraded":"2024-01-11T08:28:17.977011+0000","last_fullsized":"2024-01-11T08:28:17.977011+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:10:31.911756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"115'2","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.989922+0000","last_change":"2024-01-11T08:28:16.974546+0000","last_active":"2024-01-11T08:28:17.989922+0000","last_peered":"2024-01-11T08:28:17.989922+0000","last_clean":"2024-01-11T08:28:17.989922+0000","last_became_active":"2024-01-11T08:28:16.974286+0000","last_became_peered":"2024-01-11T08:28:16.974286+0000","last_unstale":"2024-01-11T08:28:17.989922+0000","last_undegraded":"2024-01-11T08:28:17.989922+0000","last_fullsized":"2024-01-11T08:28:17.989922+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:28:36.423713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.984648+0000","last_change":"2024-01-11T08:28:17.978737+0000","last_active":"2024-01-11T08:28:17.984648+0000","last_peered":"2024-01-11T08:28:17.984648+0000","last_clean":"2024-01-11T08:28:17.984648+0000","last_became_active":"2024-01-11T08:28:17.978444+0000","last_became_peered":"2024-01-11T08:28:17.978444+0000","last_unstale":"2024-01-11T08:28:17.984648+0000","last_undegraded":"2024-01-11T08:28:17.984648+0000","last_fullsized":"2024-01-11T08:28:17.984648+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:49:42.272444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.953861+0000","last_change":"2024-01-11T08:28:16.963488+0000","last_active":"2024-01-11T08:28:17.953861+0000","last_peered":"2024-01-11T08:28:17.953861+0000","last_clean":"2024-01-11T08:28:17.953861+0000","last_became_active":"2024-01-11T08:28:16.962940+0000","last_became_peered":"2024-01-11T08:28:16.962940+0000","last_unstale":"2024-01-11T08:28:17.953861+0000","last_undegraded":"2024-01-11T08:28:17.953861+0000","last_fullsized":"2024-01-11T08:28:17.953861+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:20:04.676688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.981522+0000","last_change":"2024-01-11T08:28:17.975998+0000","last_active":"2024-01-11T08:28:17.981522+0000","last_peered":"2024-01-11T08:28:17.981522+0000","last_clean":"2024-01-11T08:28:17.981522+0000","last_became_active":"2024-01-11T08:28:17.972655+0000","last_became_peered":"2024-01-11T08:28:17.972655+0000","last_unstale":"2024-01-11T08:28:17.981522+0000","last_undegraded":"2024-01-11T08:28:17.981522+0000","last_fullsized":"2024-01-11T08:28:17.981522+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:55:18.176094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.953110+0000","last_change":"2024-01-11T08:28:16.973112+0000","last_active":"2024-01-11T08:28:17.953110+0000","last_peered":"2024-01-11T08:28:17.953110+0000","last_clean":"2024-01-11T08:28:17.953110+0000","last_became_active":"2024-01-11T08:28:16.972797+0000","last_became_peered":"2024-01-11T08:28:16.972797+0000","last_unstale":"2024-01-11T08:28:17.953110+0000","last_undegraded":"2024-01-11T08:28:17.953110+0000","last_fullsized":"2024-01-11T08:28:17.953110+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:47:11.453713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.978569+0000","last_change":"2024-01-11T08:28:17.975206+0000","last_active":"2024-01-11T08:28:17.978569+0000","last_peered":"2024-01-11T08:28:17.978569+0000","last_clean":"2024-01-11T08:28:17.978569+0000","last_became_active":"2024-01-11T08:28:17.973907+0000","last_became_peered":"2024-01-11T08:28:17.973907+0000","last_unstale":"2024-01-11T08:28:17.978569+0000","last_undegraded":"2024-01-11T08:28:17.978569+0000","last_fullsized":"2024-01-11T08:28:17.978569+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:51:46.671037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.963824+0000","last_change":"2024-01-11T08:28:16.972944+0000","last_active":"2024-01-11T08:28:17.963824+0000","last_peered":"2024-01-11T08:28:17.963824+0000","last_clean":"2024-01-11T08:28:17.963824+0000","last_became_active":"2024-01-11T08:28:16.972677+0000","last_became_peered":"2024-01-11T08:28:16.972677+0000","last_unstale":"2024-01-11T08:28:17.963824+0000","last_undegraded":"2024-01-11T08:28:17.963824+0000","last_fullsized":"2024-01-11T08:28:17.963824+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:41:30.011035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.987115+0000","last_change":"2024-01-11T08:28:17.985468+0000","last_active":"2024-01-11T08:28:17.987115+0000","last_peered":"2024-01-11T08:28:17.987115+0000","last_clean":"2024-01-11T08:28:17.987115+0000","last_became_active":"2024-01-11T08:28:17.984681+0000","last_became_peered":"2024-01-11T08:28:17.984681+0000","last_unstale":"2024-01-11T08:28:17.987115+0000","last_undegraded":"2024-01-11T08:28:17.987115+0000","last_fullsized":"2024-01-11T08:28:17.987115+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:42:56.802832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"115'2","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.985659+0000","last_change":"2024-01-11T08:28:16.966673+0000","last_active":"2024-01-11T08:28:17.985659+0000","last_peered":"2024-01-11T08:28:17.985659+0000","last_clean":"2024-01-11T08:28:17.985659+0000","last_became_active":"2024-01-11T08:28:16.966359+0000","last_became_peered":"2024-01-11T08:28:16.966359+0000","last_unstale":"2024-01-11T08:28:17.985659+0000","last_undegraded":"2024-01-11T08:28:17.985659+0000","last_fullsized":"2024-01-11T08:28:17.985659+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:07:23.638314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.975625+0000","last_change":"2024-01-11T08:28:17.973418+0000","last_active":"2024-01-11T08:28:17.975625+0000","last_peered":"2024-01-11T08:28:17.975625+0000","last_clean":"2024-01-11T08:28:17.975625+0000","last_became_active":"2024-01-11T08:28:17.973158+0000","last_became_peered":"2024-01-11T08:28:17.973158+0000","last_unstale":"2024-01-11T08:28:17.975625+0000","last_undegraded":"2024-01-11T08:28:17.975625+0000","last_fullsized":"2024-01-11T08:28:17.975625+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:26:18.820799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.966658+0000","last_change":"2024-01-11T08:28:16.974992+0000","last_active":"2024-01-11T08:28:17.966658+0000","last_peered":"2024-01-11T08:28:17.966658+0000","last_clean":"2024-01-11T08:28:17.966658+0000","last_became_active":"2024-01-11T08:28:16.973850+0000","last_became_peered":"2024-01-11T08:28:16.973850+0000","last_unstale":"2024-01-11T08:28:17.966658+0000","last_undegraded":"2024-01-11T08:28:17.966658+0000","last_fullsized":"2024-01-11T08:28:17.966658+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:23:38.768543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.981400+0000","last_change":"2024-01-11T08:28:17.979835+0000","last_active":"2024-01-11T08:28:17.981400+0000","last_peered":"2024-01-11T08:28:17.981400+0000","last_clean":"2024-01-11T08:28:17.981400+0000","last_became_active":"2024-01-11T08:28:17.979556+0000","last_became_peered":"2024-01-11T08:28:17.979556+0000","last_unstale":"2024-01-11T08:28:17.981400+0000","last_undegraded":"2024-01-11T08:28:17.981400+0000","last_fullsized":"2024-01-11T08:28:17.981400+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:58:37.483797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.960431+0000","last_change":"2024-01-11T08:28:16.968957+0000","last_active":"2024-01-11T08:28:17.960431+0000","last_peered":"2024-01-11T08:28:17.960431+0000","last_clean":"2024-01-11T08:28:17.960431+0000","last_became_active":"2024-01-11T08:28:16.967817+0000","last_became_peered":"2024-01-11T08:28:16.967817+0000","last_unstale":"2024-01-11T08:28:17.960431+0000","last_undegraded":"2024-01-11T08:28:17.960431+0000","last_fullsized":"2024-01-11T08:28:17.960431+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:10:07.272755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.978367+0000","last_change":"2024-01-11T08:28:17.976627+0000","last_active":"2024-01-11T08:28:17.978367+0000","last_peered":"2024-01-11T08:28:17.978367+0000","last_clean":"2024-01-11T08:28:17.978367+0000","last_became_active":"2024-01-11T08:28:17.976316+0000","last_became_peered":"2024-01-11T08:28:17.976316+0000","last_unstale":"2024-01-11T08:28:17.978367+0000","last_undegraded":"2024-01-11T08:28:17.978367+0000","last_fullsized":"2024-01-11T08:28:17.978367+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:11:37.602642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.963845+0000","last_change":"2024-01-11T08:28:16.968480+0000","last_active":"2024-01-11T08:28:17.963845+0000","last_peered":"2024-01-11T08:28:17.963845+0000","last_clean":"2024-01-11T08:28:17.963845+0000","last_became_active":"2024-01-11T08:28:16.968159+0000","last_became_peered":"2024-01-11T08:28:16.968159+0000","last_unstale":"2024-01-11T08:28:17.963845+0000","last_undegraded":"2024-01-11T08:28:17.963845+0000","last_fullsized":"2024-01-11T08:28:17.963845+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:01:17.535401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.977029+0000","last_change":"2024-01-11T08:28:17.969304+0000","last_active":"2024-01-11T08:28:17.977029+0000","last_peered":"2024-01-11T08:28:17.977029+0000","last_clean":"2024-01-11T08:28:17.977029+0000","last_became_active":"2024-01-11T08:28:17.968537+0000","last_became_peered":"2024-01-11T08:28:17.968537+0000","last_unstale":"2024-01-11T08:28:17.977029+0000","last_undegraded":"2024-01-11T08:28:17.977029+0000","last_fullsized":"2024-01-11T08:28:17.977029+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:07:56.270516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.967612+0000","last_change":"2024-01-11T08:28:16.967919+0000","last_active":"2024-01-11T08:28:17.967612+0000","last_peered":"2024-01-11T08:28:17.967612+0000","last_clean":"2024-01-11T08:28:17.967612+0000","last_became_active":"2024-01-11T08:28:16.967620+0000","last_became_peered":"2024-01-11T08:28:16.967620+0000","last_unstale":"2024-01-11T08:28:17.967612+0000","last_undegraded":"2024-01-11T08:28:17.967612+0000","last_fullsized":"2024-01-11T08:28:17.967612+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:47:11.453713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.984768+0000","last_change":"2024-01-11T08:28:17.979033+0000","last_active":"2024-01-11T08:28:17.984768+0000","last_peered":"2024-01-11T08:28:17.984768+0000","last_clean":"2024-01-11T08:28:17.984768+0000","last_became_active":"2024-01-11T08:28:17.978730+0000","last_became_peered":"2024-01-11T08:28:17.978730+0000","last_unstale":"2024-01-11T08:28:17.984768+0000","last_undegraded":"2024-01-11T08:28:17.984768+0000","last_fullsized":"2024-01-11T08:28:17.984768+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:11:37.602642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.964904+0000","last_change":"2024-01-11T08:28:16.969753+0000","last_active":"2024-01-11T08:28:17.964904+0000","last_peered":"2024-01-11T08:28:17.964904+0000","last_clean":"2024-01-11T08:28:17.964904+0000","last_became_active":"2024-01-11T08:28:16.969422+0000","last_became_peered":"2024-01-11T08:28:16.969422+0000","last_unstale":"2024-01-11T08:28:17.964904+0000","last_undegraded":"2024-01-11T08:28:17.964904+0000","last_fullsized":"2024-01-11T08:28:17.964904+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:28.686024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.981247+0000","last_change":"2024-01-11T08:28:17.974257+0000","last_active":"2024-01-11T08:28:17.981247+0000","last_peered":"2024-01-11T08:28:17.981247+0000","last_clean":"2024-01-11T08:28:17.981247+0000","last_became_active":"2024-01-11T08:28:17.972335+0000","last_became_peered":"2024-01-11T08:28:17.972335+0000","last_unstale":"2024-01-11T08:28:17.981247+0000","last_undegraded":"2024-01-11T08:28:17.981247+0000","last_fullsized":"2024-01-11T08:28:17.981247+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:25.552754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.952216+0000","last_change":"2024-01-11T08:28:16.965825+0000","last_active":"2024-01-11T08:28:17.952216+0000","last_peered":"2024-01-11T08:28:17.952216+0000","last_clean":"2024-01-11T08:28:17.952216+0000","last_became_active":"2024-01-11T08:28:16.965509+0000","last_became_peered":"2024-01-11T08:28:16.965509+0000","last_unstale":"2024-01-11T08:28:17.952216+0000","last_undegraded":"2024-01-11T08:28:17.952216+0000","last_fullsized":"2024-01-11T08:28:17.952216+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:20:04.676688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.975835+0000","last_change":"2024-01-11T08:28:17.970001+0000","last_active":"2024-01-11T08:28:17.975835+0000","last_peered":"2024-01-11T08:28:17.975835+0000","last_clean":"2024-01-11T08:28:17.975835+0000","last_became_active":"2024-01-11T08:28:17.969575+0000","last_became_peered":"2024-01-11T08:28:17.969575+0000","last_unstale":"2024-01-11T08:28:17.975835+0000","last_undegraded":"2024-01-11T08:28:17.975835+0000","last_fullsized":"2024-01-11T08:28:17.975835+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:42:56.802832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":10,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.963401+0000","last_change":"2024-01-11T08:28:16.969707+0000","last_active":"2024-01-11T08:28:17.963401+0000","last_peered":"2024-01-11T08:28:17.963401+0000","last_clean":"2024-01-11T08:28:17.963401+0000","last_became_active":"2024-01-11T08:28:16.968984+0000","last_became_peered":"2024-01-11T08:28:16.968984+0000","last_unstale":"2024-01-11T08:28:17.963401+0000","last_undegraded":"2024-01-11T08:28:17.963401+0000","last_fullsized":"2024-01-11T08:28:17.963401+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:08:09.413723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.976912+0000","last_change":"2024-01-11T08:28:17.971040+0000","last_active":"2024-01-11T08:28:17.976912+0000","last_peered":"2024-01-11T08:28:17.976912+0000","last_clean":"2024-01-11T08:28:17.976912+0000","last_became_active":"2024-01-11T08:28:17.970772+0000","last_became_peered":"2024-01-11T08:28:17.970772+0000","last_unstale":"2024-01-11T08:28:17.976912+0000","last_undegraded":"2024-01-11T08:28:17.976912+0000","last_fullsized":"2024-01-11T08:28:17.976912+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:16.938117+0000","last_clean_scrub_stamp":"2024-01-11T08:28:16.938117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:17:30.830370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"115'1","reported_seq":11,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.993265+0000","last_change":"2024-01-11T08:28:16.971682+0000","last_active":"2024-01-11T08:28:17.993265+0000","last_peered":"2024-01-11T08:28:17.993265+0000","last_clean":"2024-01-11T08:28:17.993265+0000","last_became_active":"2024-01-11T08:28:16.971376+0000","last_became_peered":"2024-01-11T08:28:16.971376+0000","last_unstale":"2024-01-11T08:28:17.993265+0000","last_undegraded":"2024-01-11T08:28:17.993265+0000","last_fullsized":"2024-01-11T08:28:17.993265+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:28:15.931443+0000","last_clean_scrub_stamp":"2024-01-11T08:28:15.931443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:34:24.876291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"107'223","reported_seq":614,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-01-11T08:28:17.958069+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:28:17.958069+0000","last_peered":"2024-01-11T08:28:17.958069+0000","last_clean":"2024-01-11T08:28:17.958069+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:28:17.958069+0000","last_undegraded":"2024-01-11T08:28:17.958069+0000","last_fullsized":"2024-01-11T08:28:17.958069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1040,"num_read_kb":2130,"num_write":351,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1040,"num_read_kb":2130,"num_write":351,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575060,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303372,"kb_used_data":1988,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068468,"statfs":{"total":96636764160,"available":96326111232,"internally_reserved":0,"allocated":2035712,"data_stored":1256826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.68200000000000005,"15min":0.69999999999999996},"min":{"1min":0.42199999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.93600000000000005,"5min":0.95899999999999996,"15min":0.996},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.66800000000000004,"15min":0.70899999999999996},"min":{"1min":0.44500000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89000000000000001,"5min":1.073,"15min":1.073},"last":0.60499999999999998}]},{"osd":1,"last update":"Thu Jan 11 08:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67900000000000005,"15min":0.70599999999999996},"min":{"1min":0.45800000000000002,"5min":0.432,"15min":0.41899999999999998},"max":{"1min":0.82799999999999996,"5min":0.99099999999999999,"15min":0.998},"last":0.622},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69099999999999995,"15min":0.71899999999999997},"min":{"1min":0.47799999999999998,"5min":0.41999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88400000000000001,"5min":0.94999999999999996,"15min":1.014},"last":0.65800000000000003}]},{"osd":2,"last update":"Thu Jan 11 08:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69499999999999995,"15min":0.73099999999999998},"min":{"1min":0.46700000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.034},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.68400000000000005,"15min":0.71499999999999997},"min":{"1min":0.497,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.89500000000000002,"5min":0.98299999999999998,"15min":1.0049999999999999},"last":0.67900000000000005}]},{"osd":3,"last update":"Thu Jan 11 08:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.69599999999999995,"15min":0.73499999999999999},"min":{"1min":0.53400000000000003,"5min":0.47899999999999998,"15min":0.45700000000000002},"max":{"1min":0.84699999999999998,"5min":0.91700000000000004,"15min":1.032},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.71399999999999997,"15min":0.73899999999999999},"min":{"1min":0.44,"5min":0.439,"15min":0.439},"max":{"1min":0.92600000000000005,"5min":1.008,"15min":1.008},"last":0.877}]},{"osd":4,"last update":"Thu Jan 11 08:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.40500000000000003,"5min":0.39900000000000002,"15min":0.34899999999999998},"max":{"1min":0.745,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.64100000000000001,"15min":0.625},"min":{"1min":0.46700000000000003,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.77100000000000002,"5min":1.002,"15min":1.002},"last":0.63400000000000001}]},{"osd":5,"last update":"Thu Jan 11 08:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.46500000000000002,"5min":0.41899999999999998,"15min":0.34599999999999997},"max":{"1min":0.83299999999999996,"5min":1.042,"15min":1.042},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.66100000000000003,"15min":0.66600000000000004},"min":{"1min":0.45300000000000001,"5min":0.41399999999999998,"15min":0.40400000000000003},"max":{"1min":0.85999999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.72999999999999998}]},{"osd":6,"last update":"Thu Jan 11 08:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.73699999999999999,"15min":0.751},"min":{"1min":0.57399999999999995,"5min":0.52000000000000002,"15min":0.46999999999999997},"max":{"1min":0.95899999999999996,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.54400000000000004,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.84299999999999997,"5min":1.028,"15min":1.028},"last":0.58299999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575059,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302792,"kb_used_data":1408,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069048,"statfs":{"total":96636764160,"available":96326705152,"internally_reserved":0,"allocated":1441792,"data_stored":666970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63900000000000001,"15min":0.66700000000000004},"min":{"1min":0.441,"5min":0.38300000000000001,"15min":0.29999999999999999},"max":{"1min":0.84299999999999997,"5min":0.89300000000000002,"15min":0.94799999999999995},"last":0.627},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.63200000000000001,"15min":0.68600000000000005},"min":{"1min":0.45100000000000001,"5min":0.40000000000000002,"15min":0.33300000000000002},"max":{"1min":0.88700000000000001,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.58999999999999997}]},{"osd":1,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.65400000000000003,"15min":0.69499999999999995},"min":{"1min":0.505,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86399999999999999,"5min":1.008,"15min":1.163},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64000000000000001,"15min":0.68500000000000005},"min":{"1min":0.48599999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.89200000000000002,"5min":0.92600000000000005,"15min":1.1950000000000001},"last":0.77000000000000002}]},{"osd":2,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66000000000000003,"15min":0.70599999999999996},"min":{"1min":0.42099999999999999,"5min":0.41499999999999998,"15min":0.39800000000000002},"max":{"1min":0.91400000000000003,"5min":0.95399999999999996,"15min":1.016},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.67000000000000004,"15min":0.71799999999999997},"min":{"1min":0.53100000000000003,"5min":0.41899999999999998,"15min":0.373},"max":{"1min":0.88600000000000001,"5min":0.91300000000000003,"15min":0.99299999999999999},"last":0.74099999999999999}]},{"osd":3,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.69799999999999995,"15min":0.73099999999999998},"min":{"1min":0.52600000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.89700000000000002,"5min":0.95099999999999996,"15min":1.0149999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.68899999999999995,"15min":0.73699999999999999},"min":{"1min":0.46000000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.876,"5min":0.90900000000000003,"15min":1.0049999999999999},"last":0.70699999999999996}]},{"osd":4,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.439,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85699999999999998,"5min":0.873,"15min":0.92000000000000004},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58899999999999997,"15min":0.59499999999999997},"min":{"1min":0.46300000000000002,"5min":0.38800000000000001,"15min":0.374},"max":{"1min":0.83299999999999996,"5min":0.83799999999999997,"15min":0.96399999999999997},"last":0.69199999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.48699999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.82899999999999996,"5min":1.03,"15min":1.03},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68300000000000005,"15min":0.69799999999999995},"min":{"1min":0.54200000000000004,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.877,"5min":0.91800000000000004,"15min":0.93700000000000006},"last":0.72799999999999998}]},{"osd":7,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68700000000000006,"15min":0.70499999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.42399999999999999},"max":{"1min":0.91500000000000004,"5min":0.93300000000000005,"15min":1.0289999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.72099999999999997,"15min":0.746},"min":{"1min":0.50800000000000001,"5min":0.45900000000000002,"15min":0.44700000000000001},"max":{"1min":0.90700000000000003,"5min":1.0600000000000001,"15min":1.0669999999999999},"last":0.75700000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575061,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302784,"kb_used_data":1400,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069056,"statfs":{"total":96636764160,"available":96326713344,"internally_reserved":0,"allocated":1433600,"data_stored":666346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:27:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.48199999999999998,"15min":0.48599999999999999},"min":{"1min":0.29999999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65200000000000002,"5min":0.85999999999999999,"15min":0.97699999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54300000000000004,"15min":0.57999999999999996},"min":{"1min":0.44600000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.76100000000000001,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.44900000000000001}]},{"osd":2,"last update":"Thu Jan 11 08:27:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45300000000000001,"15min":0.48199999999999998},"min":{"1min":0.35399999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.64800000000000002,"5min":1.0960000000000001,"15min":1.113},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54600000000000004,"15min":0.56399999999999995},"min":{"1min":0.38800000000000001,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":0.76600000000000001,"5min":0.76700000000000002,"15min":0.90100000000000002},"last":0.46400000000000002}]},{"osd":3,"last update":"Thu Jan 11 08:27:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.59499999999999997,"15min":0.61399999999999999},"min":{"1min":0.47499999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.80500000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.53700000000000003,"15min":0.56399999999999995},"min":{"1min":0.39900000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82999999999999996,"5min":0.88100000000000001,"15min":0.94899999999999995},"last":0.47799999999999998}]},{"osd":4,"last update":"Thu Jan 11 08:27:50 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.76100000000000001,"15min":0.79000000000000004},"min":{"1min":0.505,"5min":0.439,"15min":0.439},"max":{"1min":1.129,"5min":1.129,"15min":1.2170000000000001},"last":0.753},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.81299999999999994,"15min":0.83199999999999996},"min":{"1min":0.70599999999999996,"5min":0.501,"15min":0.501},"max":{"1min":1.109,"5min":1.109,"15min":1.246},"last":0.66000000000000003}]},{"osd":5,"last update":"Thu Jan 11 08:27:50 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.75800000000000001,"15min":0.79600000000000004},"min":{"1min":0.64500000000000002,"5min":0.498,"15min":0.498},"max":{"1min":1.0309999999999999,"5min":1.1299999999999999,"15min":1.2629999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.754,"15min":0.78800000000000003},"min":{"1min":0.57499999999999996,"5min":0.52900000000000003,"15min":0.51200000000000001},"max":{"1min":0.94899999999999995,"5min":1.1339999999999999,"15min":1.212},"last":0.67500000000000004}]},{"osd":6,"last update":"Thu Jan 11 08:27:50 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.78900000000000003,"15min":0.81899999999999995},"min":{"1min":0.622,"5min":0.56100000000000005,"15min":0.51300000000000001},"max":{"1min":1.042,"5min":1.115,"15min":1.1439999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.79400000000000004,"15min":0.82699999999999996},"min":{"1min":0.63600000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.228},"last":0.54700000000000004}]},{"osd":7,"last update":"Thu Jan 11 08:27:50 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.79900000000000004,"15min":0.84599999999999997},"min":{"1min":0.72899999999999998,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.226},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.876,"5min":0.82299999999999995,"15min":0.84199999999999997},"min":{"1min":0.63400000000000001,"5min":0.55900000000000005,"15min":0.51400000000000001},"max":{"1min":1.087,"5min":1.099,"15min":1.258},"last":0.72699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575060,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303368,"kb_used_data":1984,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068472,"statfs":{"total":96636764160,"available":96326115328,"internally_reserved":0,"allocated":2031616,"data_stored":1257268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 08:27:53 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.501,"15min":0.499},"min":{"1min":0.33700000000000002,"5min":0.28699999999999998,"15min":0.24299999999999999},"max":{"1min":0.65300000000000002,"5min":0.84399999999999997,"15min":0.92700000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.56999999999999995,"15min":0.57399999999999995},"min":{"1min":0.51600000000000001,"5min":0.40300000000000002,"15min":0.36799999999999999},"max":{"1min":0.78500000000000003,"5min":0.90500000000000003,"15min":0.92100000000000004},"last":0.43099999999999999}]},{"osd":2,"last update":"Thu Jan 11 08:27:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.56000000000000005,"15min":0.55000000000000004},"min":{"1min":0.42099999999999999,"5min":0.33000000000000002,"15min":0.29099999999999998},"max":{"1min":0.73899999999999999,"5min":1.335,"15min":1.335},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.55600000000000005,"15min":0.55800000000000005},"min":{"1min":0.44700000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.76500000000000001,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.39100000000000001}]},{"osd":3,"last update":"Thu Jan 11 08:27:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.39800000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.71799999999999997,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.627,"15min":0.629},"min":{"1min":0.48399999999999999,"5min":0.44700000000000001,"15min":0.39600000000000002},"max":{"1min":0.79100000000000004,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.59099999999999997}]},{"osd":4,"last update":"Thu Jan 11 08:27:53 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.79200000000000004,"15min":0.79300000000000004},"min":{"1min":0.64800000000000002,"5min":0.502,"15min":0.437},"max":{"1min":1.05,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.627},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79100000000000004,"15min":0.79300000000000004},"min":{"1min":0.60599999999999998,"5min":0.51600000000000001,"15min":0.46899999999999997},"max":{"1min":0.94299999999999995,"5min":1.1220000000000001,"15min":1.149},"last":0.56200000000000006}]},{"osd":5,"last update":"Thu Jan 11 08:27:53 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.79200000000000004,"15min":0.80700000000000005},"min":{"1min":0.69399999999999995,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.091,"5min":1.518,"15min":1.518},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.82299999999999995,"15min":0.82099999999999995},"min":{"1min":0.69399999999999995,"5min":0.55400000000000005,"15min":0.47799999999999998},"max":{"1min":1.0189999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.751}]},{"osd":6,"last update":"Thu Jan 11 08:27:53 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.81499999999999995,"15min":0.80800000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.50900000000000001},"max":{"1min":1.1120000000000001,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.82599999999999996,"15min":0.83599999999999997},"min":{"1min":0.66800000000000004,"5min":0.52000000000000002,"15min":0.44900000000000001},"max":{"1min":1.0820000000000001,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.64000000000000001}]},{"osd":7,"last update":"Thu Jan 11 08:27:53 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.84299999999999997,"15min":0.84799999999999998},"min":{"1min":0.755,"5min":0.61399999999999999,"15min":0.51400000000000001},"max":{"1min":1.085,"5min":1.371,"15min":1.371},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.872,"5min":0.83399999999999996,"15min":0.84499999999999997},"min":{"1min":0.59699999999999998,"5min":0.58099999999999996,"15min":0.57699999999999996},"max":{"1min":1.046,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.65800000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575059,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310980,"kb_used_data":1404,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060860,"statfs":{"total":96636764160,"available":96318320640,"internally_reserved":0,"allocated":1437696,"data_stored":666380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47199999999999998,"15min":0.47399999999999998},"min":{"1min":0.316,"5min":0.29899999999999999,"15min":0.26000000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51100000000000001,"15min":0.54100000000000004},"min":{"1min":0.36599999999999999,"5min":0.33000000000000002,"15min":0.28899999999999998},"max":{"1min":0.72899999999999998,"5min":0.81699999999999995,"15min":0.91400000000000003},"last":0.38500000000000001}]},{"osd":1,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53500000000000003,"15min":0.55000000000000004},"min":{"1min":0.44600000000000001,"5min":0.36399999999999999,"15min":0.33500000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.90000000000000002},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55200000000000005,"15min":0.58299999999999996},"min":{"1min":0.44700000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.94499999999999995},"last":0.65300000000000002}]},{"osd":3,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.36699999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.59899999999999998,"5min":0.82499999999999996,"15min":0.91600000000000004},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.59499999999999997,"15min":0.63300000000000001},"min":{"1min":0.48599999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":1.0009999999999999},"last":0.58199999999999996}]},{"osd":4,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.746,"15min":0.78600000000000003},"min":{"1min":0.61299999999999999,"5min":0.52400000000000002,"15min":0.47299999999999998},"max":{"1min":0.995,"5min":1.026,"15min":1.2889999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.72999999999999998,"15min":0.78100000000000003},"min":{"1min":0.625,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.3089999999999999},"last":0.56200000000000006}]},{"osd":5,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.79200000000000004,"15min":0.83199999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.3540000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.81699999999999995,"15min":0.84599999999999997},"min":{"1min":0.77600000000000002,"5min":0.61099999999999999,"15min":0.55500000000000005},"max":{"1min":1.0049999999999999,"5min":1.2,"15min":1.23},"last":0.60099999999999998}]},{"osd":6,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.75600000000000001,"15min":0.79800000000000004},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.024,"5min":1.131,"15min":1.2669999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.78300000000000003,"15min":0.82699999999999996},"min":{"1min":0.64500000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.351,"5min":1.351,"15min":1.47},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Jan 11 08:27:43 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.82099999999999995,"15min":0.86499999999999999},"min":{"1min":0.69999999999999996,"5min":0.56599999999999995,"15min":0.51400000000000001},"max":{"1min":1.0600000000000001,"5min":1.2629999999999999,"15min":1.4099999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.79400000000000004,"15min":0.85799999999999998},"min":{"1min":0.71899999999999997,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.96099999999999997,"5min":1.2270000000000001,"15min":1.327},"last":0.66900000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575059,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306896,"kb_used_data":1416,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064944,"statfs":{"total":96636764160,"available":96322502656,"internally_reserved":0,"allocated":1449984,"data_stored":667904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.49099999999999999,"15min":0.48599999999999999},"min":{"1min":0.36499999999999999,"5min":0.31,"15min":0.27600000000000002},"max":{"1min":0.80400000000000005,"5min":0.80600000000000005,"15min":1.1930000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.52400000000000002,"15min":0.54800000000000004},"min":{"1min":0.44,"5min":0.39300000000000002,"15min":0.33400000000000002},"max":{"1min":0.75700000000000001,"5min":0.873,"15min":0.873},"last":0.46300000000000002}]},{"osd":1,"last update":"Thu Jan 11 08:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.46899999999999997,"15min":0.49099999999999999},"min":{"1min":0.34899999999999998,"5min":0.316,"15min":0.25900000000000001},"max":{"1min":0.69599999999999995,"5min":1.012,"15min":1.012},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56999999999999995,"15min":0.57699999999999996},"min":{"1min":0.433,"5min":0.36499999999999999,"15min":0.36299999999999999},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.1499999999999999},"last":0.39100000000000001}]},{"osd":2,"last update":"Thu Jan 11 08:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55900000000000005,"15min":0.58399999999999996},"min":{"1min":0.41499999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.873,"5min":0.873,"15min":0.97599999999999998},"last":0.434},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60899999999999999,"15min":0.63100000000000001},"min":{"1min":0.495,"5min":0.45100000000000001,"15min":0.32000000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":1.2270000000000001},"last":0.53300000000000003}]},{"osd":4,"last update":"Thu Jan 11 08:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.70799999999999996,"15min":0.73799999999999999},"min":{"1min":0.57199999999999995,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.94399999999999995,"5min":1.038,"15min":1.288},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.752,"15min":0.77900000000000003},"min":{"1min":0.65700000000000003,"5min":0.503,"15min":0.503},"max":{"1min":1.028,"5min":1.1459999999999999,"15min":1.1819999999999999},"last":0.70599999999999996}]},{"osd":5,"last update":"Thu Jan 11 08:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.754,"15min":0.78400000000000003},"min":{"1min":0.621,"5min":0.53000000000000003,"15min":0.51900000000000002},"max":{"1min":0.98299999999999998,"5min":1.0649999999999999,"15min":1.3260000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.79000000000000004,"15min":0.81000000000000005},"min":{"1min":0.70399999999999996,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.048,"5min":1.24,"15min":1.4239999999999999},"last":0.79400000000000004}]},{"osd":6,"last update":"Thu Jan 11 08:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.78800000000000003,"15min":0.82999999999999996},"min":{"1min":0.68100000000000005,"5min":0.54100000000000004,"15min":0.52600000000000002},"max":{"1min":1.0169999999999999,"5min":1.093,"15min":1.335},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.76900000000000002,"15min":0.79400000000000004},"min":{"1min":0.65000000000000002,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.45},"last":0.68500000000000005}]},{"osd":7,"last update":"Thu Jan 11 08:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.79300000000000004,"15min":0.83199999999999996},"min":{"1min":0.67500000000000004,"5min":0.52800000000000002,"15min":0.46600000000000003},"max":{"1min":1.0620000000000001,"5min":1.081,"15min":1.4019999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.83499999999999996,"15min":0.85899999999999999},"min":{"1min":0.73199999999999998,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.115,"5min":1.2050000000000001,"15min":1.21},"last":0.81100000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575060,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310996,"kb_used_data":1420,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060844,"statfs":{"total":96636764160,"available":96318304256,"internally_reserved":0,"allocated":1454080,"data_stored":668006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.70199999999999996,"15min":0.74099999999999999},"min":{"1min":0.54900000000000004,"5min":0.44500000000000001,"15min":0.28100000000000003},"max":{"1min":0.89000000000000001,"5min":0.94799999999999995,"15min":1.117},"last":0.504},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.70999999999999996,"15min":0.74199999999999999},"min":{"1min":0.61699999999999999,"5min":0.41599999999999998,"15min":0.38600000000000001},"max":{"1min":0.95299999999999996,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.67300000000000004}]},{"osd":1,"last update":"Thu Jan 11 08:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68100000000000005,"15min":0.70499999999999996},"min":{"1min":0.52400000000000002,"5min":0.434,"15min":0.33300000000000002},"max":{"1min":0.88200000000000001,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.67900000000000005,"15min":0.71299999999999997},"min":{"1min":0.60099999999999998,"5min":0.43099999999999999,"15min":0.36299999999999999},"max":{"1min":0.93899999999999995,"5min":0.99199999999999999,"15min":1.046},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 11 08:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.70799999999999996,"15min":0.72099999999999997},"min":{"1min":0.57099999999999995,"5min":0.46000000000000002,"15min":0.38900000000000001},"max":{"1min":0.92700000000000005,"5min":1.03,"15min":1.03},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69599999999999995,"15min":0.72999999999999998},"min":{"1min":0.58499999999999996,"5min":0.432,"15min":0.432},"max":{"1min":0.91500000000000004,"5min":0.97699999999999998,"15min":1.0409999999999999},"last":0.61899999999999999}]},{"osd":3,"last update":"Thu Jan 11 08:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.67000000000000004,"15min":0.70199999999999996},"min":{"1min":0.52100000000000002,"5min":0.46100000000000002,"15min":0.433},"max":{"1min":0.81599999999999995,"5min":0.96799999999999997,"15min":1.0189999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.72599999999999998,"15min":0.75},"min":{"1min":0.53700000000000003,"5min":0.502,"15min":0.47499999999999998},"max":{"1min":0.93200000000000005,"5min":1.026,"15min":1.069},"last":0.70299999999999996}]},{"osd":5,"last update":"Thu Jan 11 08:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.60099999999999998,"15min":0.60799999999999998},"min":{"1min":0.39700000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77800000000000002,"5min":0.96599999999999997,"15min":1.0700000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.61399999999999999,"15min":0.625},"min":{"1min":0.42499999999999999,"5min":0.38200000000000001,"15min":0.32600000000000001},"max":{"1min":0.89700000000000002,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.752}]},{"osd":6,"last update":"Thu Jan 11 08:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.64900000000000002,"15min":0.64700000000000002},"min":{"1min":0.44700000000000001,"5min":0.40500000000000003,"15min":0.39500000000000002},"max":{"1min":0.82699999999999996,"5min":1.133,"15min":1.133},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.46100000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88500000000000001,"5min":1.0269999999999999,"15min":1.119},"last":0.73399999999999999}]},{"osd":7,"last update":"Thu Jan 11 08:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.69099999999999995,"15min":0.68799999999999994},"min":{"1min":0.55400000000000005,"5min":0.47099999999999997,"15min":0.41299999999999998},"max":{"1min":0.98299999999999998,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.49099999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92100000000000004,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.90500000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575060,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302788,"kb_used_data":1404,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069052,"statfs":{"total":96636764160,"available":96326709248,"internally_reserved":0,"allocated":1437696,"data_stored":666416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.68100000000000005,"15min":0.71099999999999997},"min":{"1min":0.53700000000000003,"5min":0.39300000000000002,"15min":0.38700000000000001},"max":{"1min":0.86799999999999999,"5min":1.0249999999999999,"15min":1.032},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.68300000000000005,"15min":0.71099999999999997},"min":{"1min":0.498,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.93500000000000005,"5min":0.96199999999999997,"15min":1.696},"last":0.86099999999999999}]},{"osd":1,"last update":"Thu Jan 11 08:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.65500000000000003,"15min":0.69699999999999995},"min":{"1min":0.55600000000000005,"5min":0.39100000000000001,"15min":0.36399999999999999},"max":{"1min":0.997,"5min":0.997,"15min":1.4359999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.68999999999999995,"15min":0.73099999999999998},"min":{"1min":0.55400000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.86899999999999999,"5min":1.05,"15min":4.4969999999999999},"last":0.81699999999999995}]},{"osd":2,"last update":"Thu Jan 11 08:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.68799999999999994,"15min":0.71099999999999997},"min":{"1min":0.58499999999999996,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88,"5min":0.97299999999999998,"15min":1.238},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.68600000000000005,"15min":0.745},"min":{"1min":0.54300000000000004,"5min":0.45300000000000001,"15min":0.41599999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":8.0429999999999993},"last":0.84099999999999997}]},{"osd":3,"last update":"Thu Jan 11 08:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.69899999999999995,"15min":0.75},"min":{"1min":0.52600000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.90300000000000002,"5min":1.208,"15min":8.0839999999999996},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73199999999999998,"15min":0.745},"min":{"1min":0.57599999999999996,"5min":0.439,"15min":0.40899999999999997},"max":{"1min":0.89300000000000002,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.745}]},{"osd":4,"last update":"Thu Jan 11 08:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.41399999999999998,"5min":0.38300000000000001,"15min":0.34300000000000003},"max":{"1min":0.84299999999999997,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.64700000000000002,"15min":0.629},"min":{"1min":0.39400000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.88700000000000001,"5min":1.256,"15min":1.256},"last":0.60099999999999998}]},{"osd":6,"last update":"Thu Jan 11 08:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.64400000000000002},"min":{"1min":0.46500000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83599999999999997,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.66600000000000004,"15min":0.67900000000000005},"min":{"1min":0.505,"5min":0.443,"15min":0.443},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.93700000000000006}]},{"osd":7,"last update":"Thu Jan 11 08:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.70799999999999996,"15min":0.72299999999999998},"min":{"1min":0.45300000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.83699999999999997,"5min":1.341,"15min":1.341},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.76600000000000001,"15min":0.76500000000000001},"min":{"1min":0.57799999999999996,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.98399999999999999,"5min":1.119,"15min":1.119},"last":0.91400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:28:22.865 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-11T08:28:22.866 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-11T08:28:23.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.109+0000 7effcbdff700 1 Processor -- start 2024-01-11T08:28:23.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.109+0000 7effcbdff700 1 -- start start 2024-01-11T08:28:23.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.111+0000 7effcbdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7effc4153850 0x7effc4159730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:23.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.111+0000 7effcbdff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7effc4159c70 0x7effc4158360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:23.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.111+0000 7effcbdff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc4157ed0 con 0x7effc4153480 2024-01-11T08:28:23.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.111+0000 7effcbdff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc4153d70 con 0x7effc4159c70 2024-01-11T08:28:23.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.111+0000 7effcbdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc415a0c0 con 0x7effc4153850 2024-01-11T08:28:23.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.111+0000 7effc9b9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7effc4153480 0x7effc40b01a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51580/0 (socket says 172.21.15.27:51580) 2024-01-11T08:28:23.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.111+0000 7effc9b9b700 1 -- 172.21.15.27:0/2863932141 learned_addr learned my addr 172.21.15.27:0/2863932141 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:28:23.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.111+0000 7effc939a700 1 --2- 172.21.15.27:0/2863932141 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7effc4153850 0x7effc4159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:23.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.111+0000 7effca39c700 1 --2- 172.21.15.27:0/2863932141 >> v2:172.21.15.77:3300/0 conn(0x7effc4159c70 0x7effc4158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:23.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.111+0000 7effc8b99700 1 -- 172.21.15.27:0/2863932141 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488119988 0 0) 0x7effc4157ed0 con 0x7effc4153480 2024-01-11T08:28:23.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.112+0000 7effc8b99700 1 -- 172.21.15.27:0/2863932141 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effb0003570 con 0x7effc4153480 2024-01-11T08:28:23.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.112+0000 7effc939a700 1 -- 172.21.15.27:0/2863932141 >> v2:172.21.15.77:3300/0 conn(0x7effc4159c70 msgr2=0x7effc4158360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:23.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.112+0000 7effc939a700 1 --2- 172.21.15.27:0/2863932141 >> v2:172.21.15.77:3300/0 conn(0x7effc4159c70 0x7effc4158360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.112+0000 7effc939a700 1 -- 172.21.15.27:0/2863932141 >> v1:172.21.15.27:6789/0 conn(0x7effc4153480 legacy=0x7effc40b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:23.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.112+0000 7effc939a700 1 -- 172.21.15.27:0/2863932141 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc40b08b0 con 0x7effc4153850 2024-01-11T08:28:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.112+0000 7effc939a700 1 --2- 172.21.15.27:0/2863932141 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7effc4153850 0x7effc4159730 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7effb40098e0 tx=0x7effb40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f105c921a1a7dd6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.112+0000 7effc8b99700 1 -- 172.21.15.27:0/2863932141 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb4011630 con 0x7effc4153850 2024-01-11T08:28:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.112+0000 7effc8b99700 1 -- 172.21.15.27:0/2863932141 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb4011790 con 0x7effc4153850 2024-01-11T08:28:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.112+0000 7effc8b99700 1 -- 172.21.15.27:0/2863932141 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb4011b30 con 0x7effc4153850 2024-01-11T08:28:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.113+0000 7effcbdff700 1 -- 172.21.15.27:0/2863932141 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7effc4153850 msgr2=0x7effc4159730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.113+0000 7effcbdff700 1 --2- 172.21.15.27:0/2863932141 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7effc4153850 0x7effc4159730 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7effb40098e0 tx=0x7effb40091a0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.113+0000 7effcbdff700 1 -- 172.21.15.27:0/2863932141 shutdown_connections 2024-01-11T08:28:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.113+0000 7effcbdff700 1 --2- 172.21.15.27:0/2863932141 >> v2:172.21.15.77:3300/0 conn(0x7effc4159c70 0x7effc4158360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.113+0000 7effcbdff700 1 --2- 172.21.15.27:0/2863932141 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7effc4153850 0x7effc4159730 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.113+0000 7effcbdff700 1 -- 172.21.15.27:0/2863932141 >> 172.21.15.27:0/2863932141 conn(0x7effc40680a0 msgr2=0x7effc4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.113+0000 7effcbdff700 1 -- 172.21.15.27:0/2863932141 shutdown_connections 2024-01-11T08:28:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.113+0000 7effcbdff700 1 -- 172.21.15.27:0/2863932141 wait complete. 2024-01-11T08:28:23.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.114+0000 7effcbdff700 1 Processor -- start 2024-01-11T08:28:23.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.115+0000 7effcbdff700 1 -- start start 2024-01-11T08:28:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.116+0000 7effcbdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc4153480 0x7effc4152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.116+0000 7effcbdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effc4153850 0x7effc41495e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.116+0000 7effcbdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7effc4159c70 0x7effc41420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.116+0000 7effcbdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc4149150 con 0x7effc4153480 2024-01-11T08:28:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.116+0000 7effcbdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc4149b20 con 0x7effc4153850 2024-01-11T08:28:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.116+0000 7effcbdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc4148820 con 0x7effc4159c70 2024-01-11T08:28:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.116+0000 7effc9b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc4153480 0x7effc4152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.116+0000 7effc9b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc4153480 0x7effc4152680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34142/0 (socket says 172.21.15.27:34142) 2024-01-11T08:28:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.116+0000 7effc9b9b700 1 -- 172.21.15.27:0/3888956529 learned_addr learned my addr 172.21.15.27:0/3888956529 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.116+0000 7effc9b9b700 1 -- 172.21.15.27:0/3888956529 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7effc4159c70 msgr2=0x7effc41420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.116+0000 7effc9b9b700 1 --2- 172.21.15.27:0/3888956529 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7effc4159c70 0x7effc41420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.116+0000 7effc9b9b700 1 -- 172.21.15.27:0/3888956529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effc4153850 msgr2=0x7effc41495e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.116+0000 7effc9b9b700 1 --2- 172.21.15.27:0/3888956529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effc4153850 0x7effc41495e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.116+0000 7effc9b9b700 1 -- 172.21.15.27:0/3888956529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb4009420 con 0x7effc4153480 2024-01-11T08:28:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.117+0000 7effc9b9b700 1 --2- 172.21.15.27:0/3888956529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc4153480 0x7effc4152680 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7effc000c590 tx=0x7effc0007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.117+0000 7effb9ffb700 1 -- 172.21.15.27:0/3888956529 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc0005b30 con 0x7effc4153480 2024-01-11T08:28:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.117+0000 7effb9ffb700 1 -- 172.21.15.27:0/3888956529 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc0005cb0 con 0x7effc4153480 2024-01-11T08:28:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.117+0000 7effcbdff700 1 -- 172.21.15.27:0/3888956529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc4148b00 con 0x7effc4153480 2024-01-11T08:28:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.117+0000 7effb9ffb700 1 -- 172.21.15.27:0/3888956529 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc0007320 con 0x7effc4153480 2024-01-11T08:28:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.117+0000 7effcbdff700 1 -- 172.21.15.27:0/3888956529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc4143c60 con 0x7effc4153480 2024-01-11T08:28:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.118+0000 7effcbdff700 1 -- 172.21.15.27:0/3888956529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc404c3c0 con 0x7effc4153480 2024-01-11T08:28:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.119+0000 7effb9ffb700 1 -- 172.21.15.27:0/3888956529 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7effc001b3b0 con 0x7effc4153480 2024-01-11T08:28:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.120+0000 7effb9ffb700 1 --2- 172.21.15.27:0/3888956529 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7effa8066f30 0x7effa80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.120+0000 7effb9ffb700 1 -- 172.21.15.27:0/3888956529 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5733+0+0 (secure 0 0 0) 0x7effc0083b70 con 0x7effc4153480 2024-01-11T08:28:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.120+0000 7effc939a700 1 --2- 172.21.15.27:0/3888956529 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7effa8066f30 0x7effa80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:23.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.121+0000 7effc939a700 1 --2- 172.21.15.27:0/3888956529 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7effa8066f30 0x7effa80693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7effc40cb1a0 tx=0x7effb4017000 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:23.121 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:28:23.126 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:28:23.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.124+0000 7effb9ffb700 1 -- 172.21.15.27:0/3888956529 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7effc0050dc0 con 0x7effc4153480 2024-01-11T08:28:23.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.405+0000 7effcbdff700 1 -- 172.21.15.27:0/3888956529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7effc406b700 con 0x7effc4153480 2024-01-11T08:28:23.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.410+0000 7effb9ffb700 1 -- 172.21.15.27:0/3888956529 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+0 (secure 0 0 0) 0x7effc0017020 con 0x7effc4153480 2024-01-11T08:28:23.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.412+0000 7effcbdff700 1 -- 172.21.15.27:0/3888956529 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7effa8066f30 msgr2=0x7effa80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:23.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.412+0000 7effcbdff700 1 --2- 172.21.15.27:0/3888956529 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7effa8066f30 0x7effa80693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7effc40cb1a0 tx=0x7effb4017000 comp rx=0 tx=0).stop 2024-01-11T08:28:23.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.412+0000 7effcbdff700 1 -- 172.21.15.27:0/3888956529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc4153480 msgr2=0x7effc4152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:23.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.412+0000 7effcbdff700 1 --2- 172.21.15.27:0/3888956529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc4153480 0x7effc4152680 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7effc000c590 tx=0x7effc0007e00 comp rx=0 tx=0).stop 2024-01-11T08:28:23.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.412+0000 7effcbdff700 1 -- 172.21.15.27:0/3888956529 shutdown_connections 2024-01-11T08:28:23.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.412+0000 7effcbdff700 1 --2- 172.21.15.27:0/3888956529 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7effa8066f30 0x7effa80693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.412+0000 7effcbdff700 1 --2- 172.21.15.27:0/3888956529 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7effc4159c70 0x7effc41420d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.412+0000 7effcbdff700 1 --2- 172.21.15.27:0/3888956529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc4153480 0x7effc4152680 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.412+0000 7effcbdff700 1 --2- 172.21.15.27:0/3888956529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effc4153850 0x7effc41495e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.412+0000 7effcbdff700 1 -- 172.21.15.27:0/3888956529 >> 172.21.15.27:0/3888956529 conn(0x7effc40680a0 msgr2=0x7effc4158200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:23.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.412+0000 7effcbdff700 1 -- 172.21.15.27:0/3888956529 shutdown_connections 2024-01-11T08:28:23.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.413+0000 7effcbdff700 1 -- 172.21.15.27:0/3888956529 wait complete. 2024-01-11T08:28:23.413 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-11T08:28:23.429 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:28:23.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.630+0000 7f347f3ff700 1 Processor -- start 2024-01-11T08:28:23.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.630+0000 7f347f3ff700 1 -- start start 2024-01-11T08:28:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.631+0000 7f347f3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34781575e0 0x7f3478158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.632+0000 7f347f3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3478158b60 con 0x7f347806b610 2024-01-11T08:28:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.632+0000 7f347f3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34781599b0 con 0x7f34780b19d0 2024-01-11T08:28:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.632+0000 7f347f3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34780cb5e0 con 0x7f34781575e0 2024-01-11T08:28:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.632+0000 7f347c99a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f347806b610 0x7f34780b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51588/0 (socket says 172.21.15.27:51588) 2024-01-11T08:28:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.632+0000 7f347c99a700 1 -- 172.21.15.27:0/1639253792 learned_addr learned my addr 172.21.15.27:0/1639253792 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:28:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.632+0000 7f347d19b700 1 --2- 172.21.15.27:0/1639253792 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34781575e0 0x7f3478158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.632+0000 7f346ffff700 1 -- 172.21.15.27:0/1639253792 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3014340035 0 0) 0x7f3468001110 con 0x7f347806b610 2024-01-11T08:28:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.632+0000 7f346ffff700 1 -- 172.21.15.27:0/1639253792 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3464003570 con 0x7f347806b610 2024-01-11T08:28:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.632+0000 7f346ffff700 1 -- 172.21.15.27:0/1639253792 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601145189 0 0) 0x7f34781599b0 con 0x7f34780b19d0 2024-01-11T08:28:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f346ffff700 1 -- 172.21.15.27:0/1639253792 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3468001110 con 0x7f34780b19d0 2024-01-11T08:28:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f346ffff700 1 -- 172.21.15.27:0/1639253792 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3544814216 0 0) 0x7f3464003570 con 0x7f347806b610 2024-01-11T08:28:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f346ffff700 1 -- 172.21.15.27:0/1639253792 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34781575e0 msgr2=0x7f3478158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f346ffff700 1 --2- 172.21.15.27:0/1639253792 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34781575e0 0x7f3478158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f346ffff700 1 -- 172.21.15.27:0/1639253792 >> v1:172.21.15.77:6789/0 conn(0x7f34780b19d0 legacy=0x7f3478159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f346ffff700 1 -- 172.21.15.27:0/1639253792 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34780c0f90 con 0x7f347806b610 2024-01-11T08:28:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f346ffff700 1 -- 172.21.15.27:0/1639253792 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f3468001600 con 0x7f347806b610 2024-01-11T08:28:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f346ffff700 1 -- 172.21.15.27:0/1639253792 >> v1:172.21.15.27:6789/0 conn(0x7f347806b610 legacy=0x7f34780b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f346ffff700 1 --2- 172.21.15.27:0/1639253792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3464003c20 0x7f3464004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f346ffff700 1 --2- 172.21.15.27:0/1639253792 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34781575e0 0x7f3464004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f346ffff700 1 --2- 172.21.15.27:0/1639253792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3464004d50 0x7f34640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f346ffff700 1 -- 172.21.15.27:0/1639253792 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34780cb5e0 con 0x7f3464003c20 2024-01-11T08:28:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f346ffff700 1 -- 172.21.15.27:0/1639253792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346400bb40 con 0x7f3464004d50 2024-01-11T08:28:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f346ffff700 1 -- 172.21.15.27:0/1639253792 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346400bd00 con 0x7f34781575e0 2024-01-11T08:28:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f347d19b700 1 --2- 172.21.15.27:0/1639253792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3464003c20 0x7f3464004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f347c99a700 1 --2- 172.21.15.27:0/1639253792 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34781575e0 0x7f3464004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f347d99c700 1 --2- 172.21.15.27:0/1639253792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3464004d50 0x7f34640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:23.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f347d19b700 1 -- 172.21.15.27:0/1639253792 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34781575e0 msgr2=0x7f3464004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:23.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f347d19b700 1 --2- 172.21.15.27:0/1639253792 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34781575e0 0x7f3464004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f347d19b700 1 -- 172.21.15.27:0/1639253792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3464004d50 msgr2=0x7f34640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:23.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f347d19b700 1 --2- 172.21.15.27:0/1639253792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3464004d50 0x7f34640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.633+0000 7f347d19b700 1 -- 172.21.15.27:0/1639253792 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3464003570 con 0x7f3464003c20 2024-01-11T08:28:23.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.634+0000 7f347c99a700 1 --2- 172.21.15.27:0/1639253792 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34781575e0 0x7f3464004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:23.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.634+0000 7f347d99c700 1 --2- 172.21.15.27:0/1639253792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3464004d50 0x7f34640051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:23.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.634+0000 7f347d19b700 1 --2- 172.21.15.27:0/1639253792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3464003c20 0x7f3464004010 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f3474009c00 tx=0x7f34740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814dc1a5aa93fa84 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:23.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.634+0000 7f346ffff700 1 -- 172.21.15.27:0/1639253792 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347400e710 con 0x7f3464003c20 2024-01-11T08:28:23.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.634+0000 7f346ffff700 1 -- 172.21.15.27:0/1639253792 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347400e870 con 0x7f3464003c20 2024-01-11T08:28:23.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.634+0000 7f346ffff700 1 -- 172.21.15.27:0/1639253792 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347400ec30 con 0x7f3464003c20 2024-01-11T08:28:23.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.634+0000 7f347f3ff700 1 -- 172.21.15.27:0/1639253792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3464003c20 msgr2=0x7f3464004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:23.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.634+0000 7f347f3ff700 1 --2- 172.21.15.27:0/1639253792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3464003c20 0x7f3464004010 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f3474009c00 tx=0x7f34740091a0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.634+0000 7f347d99c700 1 -- 172.21.15.27:0/1639253792 reap_dead start 2024-01-11T08:28:23.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.635+0000 7f347f3ff700 1 -- 172.21.15.27:0/1639253792 shutdown_connections 2024-01-11T08:28:23.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.635+0000 7f347f3ff700 1 -- 172.21.15.27:0/1639253792 >> 172.21.15.27:0/1639253792 conn(0x7f34780680e0 msgr2=0x7f3478069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:23.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.635+0000 7f347f3ff700 1 -- 172.21.15.27:0/1639253792 shutdown_connections 2024-01-11T08:28:23.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.635+0000 7f347f3ff700 1 -- 172.21.15.27:0/1639253792 wait complete. 2024-01-11T08:28:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.636+0000 7f347f3ff700 1 Processor -- start 2024-01-11T08:28:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.636+0000 7f347f3ff700 1 -- start start 2024-01-11T08:28:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.637+0000 7f347f3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3464003c20 0x7f3478169550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.637+0000 7f347f3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34780c5630 0x7f3478153d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.637+0000 7f347f3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34780ae330 0x7f34780b0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.637+0000 7f347f3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34780c8160 con 0x7f34780ae330 2024-01-11T08:28:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.637+0000 7f347f3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34780c8440 con 0x7f3464003c20 2024-01-11T08:28:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.637+0000 7f347f3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34780c8720 con 0x7f34780c5630 2024-01-11T08:28:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.637+0000 7f347d99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34780ae330 0x7f34780b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.637+0000 7f347d99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34780ae330 0x7f34780b0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34164/0 (socket says 172.21.15.27:34164) 2024-01-11T08:28:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.637+0000 7f347d99c700 1 -- 172.21.15.27:0/2378417086 learned_addr learned my addr 172.21.15.27:0/2378417086 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.638+0000 7f347d19b700 1 --2- 172.21.15.27:0/2378417086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3464003c20 0x7f3478169550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.638+0000 7f347c99a700 1 --2- 172.21.15.27:0/2378417086 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34780c5630 0x7f3478153d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.638+0000 7f347d99c700 1 -- 172.21.15.27:0/2378417086 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34780c5630 msgr2=0x7f3478153d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.638+0000 7f347d99c700 1 --2- 172.21.15.27:0/2378417086 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34780c5630 0x7f3478153d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.638+0000 7f347d99c700 1 -- 172.21.15.27:0/2378417086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3464003c20 msgr2=0x7f3478169550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.638+0000 7f347d99c700 1 --2- 172.21.15.27:0/2378417086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3464003c20 0x7f3478169550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.638+0000 7f347d99c700 1 -- 172.21.15.27:0/2378417086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34780c8a00 con 0x7f34780ae330 2024-01-11T08:28:23.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.638+0000 7f347d19b700 1 --2- 172.21.15.27:0/2378417086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3464003c20 0x7f3478169550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:23.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.638+0000 7f347d99c700 1 --2- 172.21.15.27:0/2378417086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34780ae330 0x7f34780b0080 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f3470001350 tx=0x7f3470009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:23.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.638+0000 7f346d7fa700 1 -- 172.21.15.27:0/2378417086 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3470017070 con 0x7f34780ae330 2024-01-11T08:28:23.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.638+0000 7f346d7fa700 1 -- 172.21.15.27:0/2378417086 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34700032e0 con 0x7f34780ae330 2024-01-11T08:28:23.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.638+0000 7f347f3ff700 1 -- 172.21.15.27:0/2378417086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3478154260 con 0x7f34780ae330 2024-01-11T08:28:23.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.638+0000 7f346d7fa700 1 -- 172.21.15.27:0/2378417086 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3470006ba0 con 0x7f34780ae330 2024-01-11T08:28:23.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.639+0000 7f347f3ff700 1 -- 172.21.15.27:0/2378417086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34780ae950 con 0x7f34780ae330 2024-01-11T08:28:23.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.640+0000 7f347f3ff700 1 -- 172.21.15.27:0/2378417086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34780c8b90 con 0x7f34780ae330 2024-01-11T08:28:23.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.641+0000 7f346d7fa700 1 -- 172.21.15.27:0/2378417086 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3470001680 con 0x7f34780ae330 2024-01-11T08:28:23.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.641+0000 7f346d7fa700 1 --2- 172.21.15.27:0/2378417086 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f345c066f30 0x7f345c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:23.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.642+0000 7f346d7fa700 1 -- 172.21.15.27:0/2378417086 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f3470083f90 con 0x7f34780ae330 2024-01-11T08:28:23.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.642+0000 7f347d19b700 1 --2- 172.21.15.27:0/2378417086 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f345c066f30 0x7f345c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:23.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.643+0000 7f347d19b700 1 --2- 172.21.15.27:0/2378417086 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f345c066f30 0x7f345c0693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f34740091e0 tx=0x7f347401a000 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:23.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.645+0000 7f346d7fa700 1 -- 172.21.15.27:0/2378417086 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f34780c8b90 con 0x7f34780ae330 2024-01-11T08:28:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.893+0000 7f347f3ff700 1 -- 172.21.15.27:0/2378417086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34780c8b90 con 0x7f34780ae330 2024-01-11T08:28:23.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.894+0000 7f346d7fa700 1 -- 172.21.15.27:0/2378417086 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4596 (secure 0 0 0) 0x7f34780c8b90 con 0x7f34780ae330 2024-01-11T08:28:23.893 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:28:23.893 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6608,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2001630215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2001630215},{"type":"v1","addr":"172.21.15.77:6834","nonce":2001630215}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7348,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/884469933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":884469933},{"type":"v1","addr":"172.21.15.77:6837","nonce":884469933}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":7584,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/4086481881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4086481881},{"type":"v1","addr":"172.21.15.27:6834","nonce":4086481881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:28:17.454224+0000","modified":"2024-01-11T08:28:20.975892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7587":{"gid":7587,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/2923524035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2923524035},{"type":"v1","addr":"172.21.15.27:6835","nonce":2923524035}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2024-01-11T08:28:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.896+0000 7f347f3ff700 1 -- 172.21.15.27:0/2378417086 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f345c066f30 msgr2=0x7f345c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.896+0000 7f347f3ff700 1 --2- 172.21.15.27:0/2378417086 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f345c066f30 0x7f345c0693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f34740091e0 tx=0x7f347401a000 comp rx=0 tx=0).stop 2024-01-11T08:28:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.896+0000 7f347f3ff700 1 -- 172.21.15.27:0/2378417086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34780ae330 msgr2=0x7f34780b0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.896+0000 7f347f3ff700 1 --2- 172.21.15.27:0/2378417086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34780ae330 0x7f34780b0080 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f3470001350 tx=0x7f3470009040 comp rx=0 tx=0).stop 2024-01-11T08:28:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.896+0000 7f347f3ff700 1 -- 172.21.15.27:0/2378417086 shutdown_connections 2024-01-11T08:28:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.896+0000 7f347f3ff700 1 --2- 172.21.15.27:0/2378417086 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f345c066f30 0x7f345c0693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.896+0000 7f347f3ff700 1 --2- 172.21.15.27:0/2378417086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34780ae330 0x7f34780b0080 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.896+0000 7f347f3ff700 1 --2- 172.21.15.27:0/2378417086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3464003c20 0x7f3478169550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.896+0000 7f347f3ff700 1 --2- 172.21.15.27:0/2378417086 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34780c5630 0x7f3478153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.896+0000 7f347f3ff700 1 -- 172.21.15.27:0/2378417086 >> 172.21.15.27:0/2378417086 conn(0x7f34780680e0 msgr2=0x7f347815a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:23.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.897+0000 7f347f3ff700 1 -- 172.21.15.27:0/2378417086 shutdown_connections 2024-01-11T08:28:23.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:23.897+0000 7f347f3ff700 1 -- 172.21.15.27:0/2378417086 wait complete. 2024-01-11T08:28:23.897 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 59 2024-01-11T08:28:23.929 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-01-11T08:28:23.930+0000 7f572d7f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T08:28:24.318 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 59, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-11T08:28:17.454224+0000', 'modified': '2024-01-11T08:28:20.975892+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7587}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7587': {'gid': 7587, 'name': 'a', 'rank': 0, 'incarnation': 55, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6835/2923524035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2923524035}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 2923524035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-11T08:28:24.318 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-11T08:28:24.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-11T08:28:24.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.520+0000 7f8f183ff700 1 Processor -- start 2024-01-11T08:28:24.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.520+0000 7f8f183ff700 1 -- start start 2024-01-11T08:28:24.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.522+0000 7f8f183ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8f100b0550 0x7f8f100ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.522+0000 7f8f183ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8f10156e80 0x7f8f101572c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.522+0000 7f8f183ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f100adc80 con 0x7f8f10156ab0 2024-01-11T08:28:24.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.522+0000 7f8f183ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f100adf60 con 0x7f8f10156e80 2024-01-11T08:28:24.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.522+0000 7f8f183ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f100ae240 con 0x7f8f100b0550 2024-01-11T08:28:24.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.522+0000 7f8f1699c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8f10156e80 0x7f8f101572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.522+0000 7f8f1699c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8f10156e80 0x7f8f101572c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41378/0 (socket says 172.21.15.27:41378) 2024-01-11T08:28:24.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.522+0000 7f8f1699c700 1 -- 172.21.15.27:0/1248310403 learned_addr learned my addr 172.21.15.27:0/1248310403 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:24.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.522+0000 7f8f1619b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8f10156ab0 0x7f8f100cc9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49368/0 (socket says 172.21.15.27:49368) 2024-01-11T08:28:24.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.522+0000 7f8f1599a700 1 --2- 172.21.15.27:0/1248310403 >> v2:172.21.15.77:3300/0 conn(0x7f8f100b0550 0x7f8f100ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.522+0000 7f8f1699c700 1 -- 172.21.15.27:0/1248310403 >> v2:172.21.15.77:3300/0 conn(0x7f8f100b0550 msgr2=0x7f8f100ad650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:24.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.522+0000 7f8f1699c700 1 --2- 172.21.15.27:0/1248310403 >> v2:172.21.15.77:3300/0 conn(0x7f8f100b0550 0x7f8f100ad650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.522+0000 7f8f1699c700 1 -- 172.21.15.27:0/1248310403 >> v1:172.21.15.27:6789/0 conn(0x7f8f10156ab0 legacy=0x7f8f100cc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:24.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.522+0000 7f8f1699c700 1 -- 172.21.15.27:0/1248310403 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f100ae520 con 0x7f8f10156e80 2024-01-11T08:28:24.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.522+0000 7f8f15199700 1 -- 172.21.15.27:0/1248310403 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319868134 0 0) 0x7f8f100adc80 con 0x7f8f10156ab0 2024-01-11T08:28:24.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.523+0000 7f8f1699c700 1 --2- 172.21.15.27:0/1248310403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f10156e80 0x7f8f101572c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f8f100cd290 tx=0x7f8f08009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=355db25e5f5ca4db server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:24.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.523+0000 7f8f15199700 1 -- 172.21.15.27:0/1248310403 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0800e070 con 0x7f8f10156e80 2024-01-11T08:28:24.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.523+0000 7f8f15199700 1 -- 172.21.15.27:0/1248310403 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0800ba50 con 0x7f8f10156e80 2024-01-11T08:28:24.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.523+0000 7f8f15199700 1 -- 172.21.15.27:0/1248310403 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f080092e0 con 0x7f8f10156e80 2024-01-11T08:28:24.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.524+0000 7f8f183ff700 1 -- 172.21.15.27:0/1248310403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f10156e80 msgr2=0x7f8f101572c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:24.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.524+0000 7f8f183ff700 1 --2- 172.21.15.27:0/1248310403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f10156e80 0x7f8f101572c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f8f100cd290 tx=0x7f8f08009840 comp rx=0 tx=0).stop 2024-01-11T08:28:24.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.524+0000 7f8f183ff700 1 -- 172.21.15.27:0/1248310403 shutdown_connections 2024-01-11T08:28:24.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.524+0000 7f8f183ff700 1 --2- 172.21.15.27:0/1248310403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f10156e80 0x7f8f101572c0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.524+0000 7f8f183ff700 1 --2- 172.21.15.27:0/1248310403 >> v2:172.21.15.77:3300/0 conn(0x7f8f100b0550 0x7f8f100ad650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.524+0000 7f8f183ff700 1 -- 172.21.15.27:0/1248310403 >> 172.21.15.27:0/1248310403 conn(0x7f8f100680b0 msgr2=0x7f8f100cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:24.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.524+0000 7f8f183ff700 1 -- 172.21.15.27:0/1248310403 shutdown_connections 2024-01-11T08:28:24.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.524+0000 7f8f183ff700 1 -- 172.21.15.27:0/1248310403 wait complete. 2024-01-11T08:28:24.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.525+0000 7f8f183ff700 1 Processor -- start 2024-01-11T08:28:24.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.526+0000 7f8f183ff700 1 -- start start 2024-01-11T08:28:24.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.526+0000 7f8f183ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f100b0550 0x7f8f10145cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.526+0000 7f8f183ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f10156ab0 0x7f8f10167960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.526+0000 7f8f183ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f10156e80 0x7f8f10144810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.527+0000 7f8f183ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f10146c40 con 0x7f8f10156ab0 2024-01-11T08:28:24.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.527+0000 7f8f183ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f100ca1d0 con 0x7f8f10156e80 2024-01-11T08:28:24.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.527+0000 7f8f183ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f100ca4b0 con 0x7f8f100b0550 2024-01-11T08:28:24.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.527+0000 7f8f1599a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f10156ab0 0x7f8f10167960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.527+0000 7f8f1599a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f10156ab0 0x7f8f10167960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41392/0 (socket says 172.21.15.27:41392) 2024-01-11T08:28:24.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.527+0000 7f8f1599a700 1 -- 172.21.15.27:0/1604204921 learned_addr learned my addr 172.21.15.27:0/1604204921 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:24.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.527+0000 7f8f1619b700 1 --2- 172.21.15.27:0/1604204921 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f100b0550 0x7f8f10145cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.527+0000 7f8f1699c700 1 --2- 172.21.15.27:0/1604204921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f10156e80 0x7f8f10144810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.527+0000 7f8f1599a700 1 -- 172.21.15.27:0/1604204921 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f100b0550 msgr2=0x7f8f10145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:24.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.527+0000 7f8f1599a700 1 --2- 172.21.15.27:0/1604204921 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f100b0550 0x7f8f10145cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.527+0000 7f8f1599a700 1 -- 172.21.15.27:0/1604204921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f10156e80 msgr2=0x7f8f10144810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:24.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.527+0000 7f8f1599a700 1 --2- 172.21.15.27:0/1604204921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f10156e80 0x7f8f10144810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.527+0000 7f8f1599a700 1 -- 172.21.15.27:0/1604204921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f08009ae0 con 0x7f8f10156ab0 2024-01-11T08:28:24.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.528+0000 7f8f1599a700 1 --2- 172.21.15.27:0/1604204921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f10156ab0 0x7f8f10167960 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f8f0000d860 tx=0x7f8f00008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:24.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.528+0000 7f8f067fc700 1 -- 172.21.15.27:0/1604204921 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f000125f0 con 0x7f8f10156ab0 2024-01-11T08:28:24.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.528+0000 7f8f067fc700 1 -- 172.21.15.27:0/1604204921 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f00012750 con 0x7f8f10156ab0 2024-01-11T08:28:24.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.528+0000 7f8f067fc700 1 -- 172.21.15.27:0/1604204921 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00012b10 con 0x7f8f10156ab0 2024-01-11T08:28:24.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.528+0000 7f8f183ff700 1 -- 172.21.15.27:0/1604204921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f101465f0 con 0x7f8f10156ab0 2024-01-11T08:28:24.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.528+0000 7f8f183ff700 1 -- 172.21.15.27:0/1604204921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f10144d50 con 0x7f8f10156ab0 2024-01-11T08:28:24.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.530+0000 7f8f067fc700 1 -- 172.21.15.27:0/1604204921 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8f0001e460 con 0x7f8f10156ab0 2024-01-11T08:28:24.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.531+0000 7f8f067fc700 1 --2- 172.21.15.27:0/1604204921 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8ef4066f30 0x7f8ef40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.531+0000 7f8f067fc700 1 -- 172.21.15.27:0/1604204921 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f8f00084b20 con 0x7f8f10156ab0 2024-01-11T08:28:24.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.532+0000 7f8f1619b700 1 --2- 172.21.15.27:0/1604204921 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8ef4066f30 0x7f8ef40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.532+0000 7f8f1619b700 1 --2- 172.21.15.27:0/1604204921 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8ef4066f30 0x7f8ef40693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c007fd0 tx=0x7f8f0c007d50 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:24.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.533+0000 7f8f26a90b80 1 Processor -- start 2024-01-11T08:28:24.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.533+0000 7f8f26a90b80 1 -- start start 2024-01-11T08:28:24.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.534+0000 7f8f26a90b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033f945e0 0x563033f949b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.534+0000 7f8f26a90b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033f99e40 0x563033fc3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.534+0000 7f8f26a90b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033fc3860 0x563033fc4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.534+0000 7f8f26a90b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563033eb1310 con 0x563033fc3860 2024-01-11T08:28:24.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.534+0000 7f8f26a90b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x563033eee130 con 0x563033f99e40 2024-01-11T08:28:24.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.534+0000 7f8f26a90b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x563033eebaf0 con 0x563033f945e0 2024-01-11T08:28:24.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.534+0000 7f8f1699c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033fc3860 0x563033fc4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.534+0000 7f8f1699c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033fc3860 0x563033fc4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41396/0 (socket says 172.21.15.27:41396) 2024-01-11T08:28:24.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.534+0000 7f8f1699c700 1 -- 172.21.15.27:0/2559765690 learned_addr learned my addr 172.21.15.27:0/2559765690 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.534+0000 7f8f1619b700 1 --2- 172.21.15.27:0/2559765690 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033f945e0 0x563033f949b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.534+0000 7f8f1599a700 1 --2- 172.21.15.27:0/2559765690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033f99e40 0x563033fc3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.534+0000 7f8f1699c700 1 -- 172.21.15.27:0/2559765690 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033f945e0 msgr2=0x563033f949b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.534+0000 7f8f1699c700 1 --2- 172.21.15.27:0/2559765690 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033f945e0 0x563033f949b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.534+0000 7f8f1699c700 1 -- 172.21.15.27:0/2559765690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033f99e40 msgr2=0x563033fc3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.534+0000 7f8f1699c700 1 --2- 172.21.15.27:0/2559765690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033f99e40 0x563033fc3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.534+0000 7f8f1699c700 1 -- 172.21.15.27:0/2559765690 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563033dda0a0 con 0x563033fc3860 2024-01-11T08:28:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.535+0000 7f8f1699c700 1 --2- 172.21.15.27:0/2559765690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033fc3860 0x563033fc4c60 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f8f1006fe90 tx=0x7f8f08018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=352c0eea6a5f0920 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.535+0000 7f8ef37fe700 1 -- 172.21.15.27:0/2559765690 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f08007200 con 0x563033fc3860 2024-01-11T08:28:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.535+0000 7f8ef37fe700 1 -- 172.21.15.27:0/2559765690 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0800bbc0 con 0x563033fc3860 2024-01-11T08:28:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.535+0000 7f8ef37fe700 1 -- 172.21.15.27:0/2559765690 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f08007820 con 0x563033fc3860 2024-01-11T08:28:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.536+0000 7f8f26a90b80 1 -- 172.21.15.27:0/2559765690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033fc3860 msgr2=0x563033fc4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.536+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/2559765690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033fc3860 0x563033fc4c60 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f8f1006fe90 tx=0x7f8f08018000 comp rx=0 tx=0).stop 2024-01-11T08:28:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.536+0000 7f8f26a90b80 1 -- 172.21.15.27:0/2559765690 shutdown_connections 2024-01-11T08:28:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.536+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/2559765690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033fc3860 0x563033fc4c60 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.536+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/2559765690 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033f945e0 0x563033f949b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.536+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/2559765690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033f99e40 0x563033fc3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.536+0000 7f8f26a90b80 1 -- 172.21.15.27:0/2559765690 >> 172.21.15.27:0/2559765690 conn(0x563033f96300 msgr2=0x563033f98bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.536+0000 7f8f26a90b80 1 -- 172.21.15.27:0/2559765690 shutdown_connections 2024-01-11T08:28:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.536+0000 7f8f26a90b80 1 -- 172.21.15.27:0/2559765690 wait complete. 2024-01-11T08:28:24.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.537+0000 7f8f26a90b80 1 -- start start 2024-01-11T08:28:24.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.537+0000 7f8f26a90b80 1 Processor -- start 2024-01-11T08:28:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.538+0000 7f8f26a90b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 0x563033ff2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.538+0000 7f8f26a90b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033f99e40 0x563033fc5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.538+0000 7f8f26a90b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033fc3860 0x563033fe1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.538+0000 7f8f26a90b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563033ee78e0 con 0x563033f945e0 2024-01-11T08:28:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.538+0000 7f8f26a90b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x563033e93980 con 0x563033fc3860 2024-01-11T08:28:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.538+0000 7f8f26a90b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x563033e93620 con 0x563033f99e40 2024-01-11T08:28:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.538+0000 7f8f26a90b80 10 client.0 _ll_register_callbacks cb 0x563033cf6a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T08:28:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.538+0000 7f8f1619b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 0x563033ff2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.538+0000 7f8f1619b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 0x563033ff2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41404/0 (socket says 172.21.15.27:41404) 2024-01-11T08:28:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.538+0000 7f8f1619b700 1 -- 172.21.15.27:0/3060230563 learned_addr learned my addr 172.21.15.27:0/3060230563 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.538+0000 7f8f1699c700 1 --2- 172.21.15.27:0/3060230563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033fc3860 0x563033fe1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.538+0000 7f8f1599a700 1 --2- 172.21.15.27:0/3060230563 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033f99e40 0x563033fc5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.538+0000 7f8f1619b700 1 -- 172.21.15.27:0/3060230563 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033f99e40 msgr2=0x563033fc5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.538+0000 7f8f1619b700 1 --2- 172.21.15.27:0/3060230563 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033f99e40 0x563033fc5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.538+0000 7f8f1619b700 1 -- 172.21.15.27:0/3060230563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033fc3860 msgr2=0x563033fe1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.538+0000 7f8f1619b700 1 --2- 172.21.15.27:0/3060230563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033fc3860 0x563033fe1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.538+0000 7f8f1619b700 1 -- 172.21.15.27:0/3060230563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f100ae520 con 0x563033f945e0 2024-01-11T08:28:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.539+0000 7f8f1599a700 1 --2- 172.21.15.27:0/3060230563 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033f99e40 0x563033fc5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:28:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.539+0000 7f8f1619b700 1 --2- 172.21.15.27:0/3060230563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 0x563033ff2b30 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c004860 tx=0x7f8f0c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f52cc324c6d2f78 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.539+0000 7f8ef0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T08:28:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.539+0000 7f8ef0ff9700 1 -- 172.21.15.27:0/3060230563 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c001d00 con 0x563033f945e0 2024-01-11T08:28:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.539+0000 7f8ef0ff9700 1 -- 172.21.15.27:0/3060230563 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(116..116 src has 1..116) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f8f0c01b070 con 0x563033f945e0 2024-01-11T08:28:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.539+0000 7f8f26a90b80 1 -- 172.21.15.27:0/3060230563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563033f8c560 con 0x563033f945e0 2024-01-11T08:28:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.539+0000 7f8ef0ff9700 1 -- 172.21.15.27:0/3060230563 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f0c022d30 con 0x563033f945e0 2024-01-11T08:28:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.539+0000 7f8f26a90b80 10 client.7728 fetch_fsmap learned FSMap version 59 2024-01-11T08:28:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.539+0000 7f8f26a90b80 1 -- 172.21.15.27:0/3060230563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=2+}) v3 -- 0x563033eb63c0 con 0x563033f945e0 2024-01-11T08:28:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.539+0000 7f8ef0ff9700 1 -- 172.21.15.27:0/3060230563 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563033f8c560 con 0x563033f945e0 2024-01-11T08:28:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.540+0000 7f8ef0ff9700 1 -- 172.21.15.27:0/3060230563 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 59) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f8f0c022020 con 0x563033f945e0 2024-01-11T08:28:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.540+0000 7f8f26a90b80 10 client.7728 fetch_fsmap finished waiting for FSMap version 59 2024-01-11T08:28:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.540+0000 7f8f26a90b80 10 client.7728 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-11T08:28:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.540+0000 7f8f26a90b80 20 client.7728 populate_metadata read hostname 'smithi027' 2024-01-11T08:28:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.540+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/3060230563 >> [v2:172.21.15.27:6832/2923524035,v1:172.21.15.27:6835/2923524035] conn(0x563033fe0420 0x563033fe07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.540+0000 7f8f26a90b80 4 client.7728 mds_command: new command op to 7587 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-11T08:28:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.540+0000 7f8f26a90b80 1 -- 172.21.15.27:0/3060230563 --> [v2:172.21.15.27:6832/2923524035,v1:172.21.15.27:6835/2923524035] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563033f690e0 con 0x563033fe0420 2024-01-11T08:28:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.540+0000 7f8f1599a700 1 --2- 172.21.15.27:0/3060230563 >> [v2:172.21.15.27:6832/2923524035,v1:172.21.15.27:6835/2923524035] conn(0x563033fe0420 0x563033fe07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.540+0000 7f8f1599a700 1 --2- 172.21.15.27:0/3060230563 >> [v2:172.21.15.27:6832/2923524035,v1:172.21.15.27:6835/2923524035] conn(0x563033fe0420 0x563033fe07f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f5fd12719ba0dc1 server_cookie=eead9d1bd598850e in_seq=0 out_seq=0 2024-01-11T08:28:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.540+0000 7f8ef0ff9700 10 client.7728 ms_handle_connect on v2:172.21.15.27:6832/2923524035 2024-01-11T08:28:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.541+0000 7f8ef0ff9700 1 -- 172.21.15.27:0/3060230563 <== mds.0 v2:172.21.15.27:6832/2923524035 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x563033f690e0 con 0x563033fe0420 2024-01-11T08:28:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.541+0000 7f8ef0ff9700 10 client.7728 handle_command_reply: tid=0 2024-01-11T08:28:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.541+0000 7f8f26a90b80 1 client.7728 shutdown 2024-01-11T08:28:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.541+0000 7f8f26a90b80 1 -- 172.21.15.27:0/3060230563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 msgr2=0x563033ff2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.541+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/3060230563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 0x563033ff2b30 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c004860 tx=0x7f8f0c014000 comp rx=0 tx=0).stop 2024-01-11T08:28:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.541+0000 7f8f26a90b80 1 -- 172.21.15.27:0/3060230563 shutdown_connections 2024-01-11T08:28:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.541+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/3060230563 >> [v2:172.21.15.27:6832/2923524035,v1:172.21.15.27:6835/2923524035] conn(0x563033fe0420 0x563033fe07f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.541+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/3060230563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033fc3860 0x563033fe1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.541+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/3060230563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 0x563033ff2b30 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.541+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/3060230563 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033f99e40 0x563033fc5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.541+0000 7f8ef0ff9700 0 client.7728 ms_handle_reset on v2:172.21.15.27:6832/2923524035 2024-01-11T08:28:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.541+0000 7f8f26a90b80 1 -- 172.21.15.27:0/3060230563 >> 172.21.15.27:0/3060230563 conn(0x563033f96300 msgr2=0x563033f993d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.542+0000 7f8f26a90b80 1 -- 172.21.15.27:0/3060230563 shutdown_connections 2024-01-11T08:28:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.542+0000 7f8f26a90b80 1 -- 172.21.15.27:0/3060230563 wait complete. 2024-01-11T08:28:24.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.542+0000 7f8f26a90b80 20 client.7728 trim_cache size 0 max 16384 2024-01-11T08:28:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.559+0000 7f8f26a90b80 1 Processor -- start 2024-01-11T08:28:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.559+0000 7f8f26a90b80 1 -- start start 2024-01-11T08:28:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.560+0000 7f8f26a90b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 0x563033ff42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.560+0000 7f8f26a90b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033f99e40 0x563033fb9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.560+0000 7f8f26a90b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033fc3860 0x56303403e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.560+0000 7f8f26a90b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563033d7f1e0 con 0x563033f945e0 2024-01-11T08:28:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.560+0000 7f8f26a90b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x563034003a00 con 0x563033f99e40 2024-01-11T08:28:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.560+0000 7f8f26a90b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x563033f936f0 con 0x563033fc3860 2024-01-11T08:28:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.560+0000 7f8f1619b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 0x563033ff42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.561+0000 7f8f1619b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 0x563033ff42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41408/0 (socket says 172.21.15.27:41408) 2024-01-11T08:28:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.561+0000 7f8f1619b700 1 -- 172.21.15.27:0/504753978 learned_addr learned my addr 172.21.15.27:0/504753978 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.561+0000 7f8f1599a700 1 --2- 172.21.15.27:0/504753978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033f99e40 0x563033fb9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.561+0000 7f8f1699c700 1 --2- 172.21.15.27:0/504753978 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033fc3860 0x56303403e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.561+0000 7f8f1619b700 1 -- 172.21.15.27:0/504753978 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033fc3860 msgr2=0x56303403e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.561+0000 7f8f1619b700 1 --2- 172.21.15.27:0/504753978 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033fc3860 0x56303403e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.561+0000 7f8f1619b700 1 -- 172.21.15.27:0/504753978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033f99e40 msgr2=0x563033fb9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.561+0000 7f8f1619b700 1 --2- 172.21.15.27:0/504753978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033f99e40 0x563033fb9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.561+0000 7f8f1619b700 1 -- 172.21.15.27:0/504753978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563033d4e1d0 con 0x563033f945e0 2024-01-11T08:28:24.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.561+0000 7f8f1599a700 1 --2- 172.21.15.27:0/504753978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033f99e40 0x563033fb9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:24.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.561+0000 7f8f1619b700 1 --2- 172.21.15.27:0/504753978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 0x563033ff42b0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c0011e0 tx=0x7f8f0c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b1fe490a930af94 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:24.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.561+0000 7f8ef3fff700 1 -- 172.21.15.27:0/504753978 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c022580 con 0x563033f945e0 2024-01-11T08:28:24.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.561+0000 7f8ef3fff700 1 -- 172.21.15.27:0/504753978 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0c00b030 con 0x563033f945e0 2024-01-11T08:28:24.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.561+0000 7f8ef3fff700 1 -- 172.21.15.27:0/504753978 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c001d00 con 0x563033f945e0 2024-01-11T08:28:24.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.562+0000 7f8f26a90b80 1 -- 172.21.15.27:0/504753978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 msgr2=0x563033ff42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:24.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.562+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/504753978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 0x563033ff42b0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c0011e0 tx=0x7f8f0c018000 comp rx=0 tx=0).stop 2024-01-11T08:28:24.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.562+0000 7f8f26a90b80 1 -- 172.21.15.27:0/504753978 shutdown_connections 2024-01-11T08:28:24.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.562+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/504753978 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033fc3860 0x56303403e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.562+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/504753978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 0x563033ff42b0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.562+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/504753978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033f99e40 0x563033fb9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.562+0000 7f8f26a90b80 1 -- 172.21.15.27:0/504753978 >> 172.21.15.27:0/504753978 conn(0x563033f96300 msgr2=0x563033fc7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:24.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.562+0000 7f8f26a90b80 1 -- 172.21.15.27:0/504753978 shutdown_connections 2024-01-11T08:28:24.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.562+0000 7f8f26a90b80 1 -- 172.21.15.27:0/504753978 wait complete. 2024-01-11T08:28:24.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.563+0000 7f8f26a90b80 1 -- start start 2024-01-11T08:28:24.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.563+0000 7f8f26a90b80 1 Processor -- start 2024-01-11T08:28:24.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.564+0000 7f8f26a90b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 0x563033fc2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.564+0000 7f8f26a90b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033f99e40 0x563033f991b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.564+0000 7f8f26a90b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033fc3860 0x563033f996f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.564+0000 7f8f26a90b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563033f97840 con 0x563033f945e0 2024-01-11T08:28:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.564+0000 7f8f26a90b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56303403b360 con 0x563033f99e40 2024-01-11T08:28:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.564+0000 7f8f26a90b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x563034049a30 con 0x563033fc3860 2024-01-11T08:28:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.564+0000 7f8f26a90b80 10 client.0 _ll_register_callbacks cb 0x563033cf6a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T08:28:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.564+0000 7f8f1619b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 0x563033fc2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.564+0000 7f8f1619b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 0x563033fc2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41416/0 (socket says 172.21.15.27:41416) 2024-01-11T08:28:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.564+0000 7f8f1619b700 1 -- 172.21.15.27:0/1379291768 learned_addr learned my addr 172.21.15.27:0/1379291768 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.564+0000 7f8f1699c700 1 --2- 172.21.15.27:0/1379291768 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033fc3860 0x563033f996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.564+0000 7f8f1599a700 1 --2- 172.21.15.27:0/1379291768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033f99e40 0x563033f991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.565+0000 7f8f1619b700 1 -- 172.21.15.27:0/1379291768 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033fc3860 msgr2=0x563033f996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.565+0000 7f8f1619b700 1 --2- 172.21.15.27:0/1379291768 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033fc3860 0x563033f996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.565+0000 7f8f1619b700 1 -- 172.21.15.27:0/1379291768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033f99e40 msgr2=0x563033f991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.565+0000 7f8f1619b700 1 --2- 172.21.15.27:0/1379291768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033f99e40 0x563033f991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.565+0000 7f8f1619b700 1 -- 172.21.15.27:0/1379291768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563033eb63c0 con 0x563033f945e0 2024-01-11T08:28:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.565+0000 7f8f1619b700 1 --2- 172.21.15.27:0/1379291768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 0x563033fc2be0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c037040 tx=0x7f8f0c02f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4caab52c840f799a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.565+0000 7f8ef0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T08:28:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.565+0000 7f8ef0ff9700 1 -- 172.21.15.27:0/1379291768 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c022490 con 0x563033f945e0 2024-01-11T08:28:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.565+0000 7f8ef0ff9700 1 -- 172.21.15.27:0/1379291768 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(116..116 src has 1..116) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f8f0c040470 con 0x563033f945e0 2024-01-11T08:28:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.565+0000 7f8f26a90b80 1 -- 172.21.15.27:0/1379291768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8f0800bbc0 con 0x563033f945e0 2024-01-11T08:28:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.566+0000 7f8ef0ff9700 1 -- 172.21.15.27:0/1379291768 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f0c0225f0 con 0x563033f945e0 2024-01-11T08:28:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.566+0000 7f8f26a90b80 10 client.7734 fetch_fsmap learned FSMap version 59 2024-01-11T08:28:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.566+0000 7f8f26a90b80 1 -- 172.21.15.27:0/1379291768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=2+}) v3 -- 0x563033d27d60 con 0x563033f945e0 2024-01-11T08:28:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.566+0000 7f8ef0ff9700 1 -- 172.21.15.27:0/1379291768 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0800bbc0 con 0x563033f945e0 2024-01-11T08:28:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.566+0000 7f8ef0ff9700 1 -- 172.21.15.27:0/1379291768 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 59) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f8f0c040840 con 0x563033f945e0 2024-01-11T08:28:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.566+0000 7f8f26a90b80 10 client.7734 fetch_fsmap finished waiting for FSMap version 59 2024-01-11T08:28:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.566+0000 7f8f26a90b80 10 client.7734 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-11T08:28:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.566+0000 7f8f26a90b80 20 client.7734 populate_metadata read hostname 'smithi027' 2024-01-11T08:28:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.566+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/1379291768 >> [v2:172.21.15.27:6832/2923524035,v1:172.21.15.27:6835/2923524035] conn(0x563033fc05c0 0x563033fc0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.566+0000 7f8f26a90b80 4 client.7734 mds_command: new command op to 7587 tid=0[{"prefix": "status"}] 2024-01-11T08:28:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.566+0000 7f8f26a90b80 1 -- 172.21.15.27:0/1379291768 --> [v2:172.21.15.27:6832/2923524035,v1:172.21.15.27:6835/2923524035] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563033e8a8d0 con 0x563033fc05c0 2024-01-11T08:28:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.566+0000 7f8f1599a700 1 --2- 172.21.15.27:0/1379291768 >> [v2:172.21.15.27:6832/2923524035,v1:172.21.15.27:6835/2923524035] conn(0x563033fc05c0 0x563033fc0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.567+0000 7f8ef0ff9700 10 client.7734 ms_handle_connect on v2:172.21.15.27:6832/2923524035 2024-01-11T08:28:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.567+0000 7f8f1599a700 1 --2- 172.21.15.27:0/1379291768 >> [v2:172.21.15.27:6832/2923524035,v1:172.21.15.27:6835/2923524035] conn(0x563033fc05c0 0x563033fc0990 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8c61b17a9249c32 server_cookie=4e07e7c219e29ab7 in_seq=0 out_seq=0 2024-01-11T08:28:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.567+0000 7f8ef0ff9700 1 -- 172.21.15.27:0/1379291768 <== mds.0 v2:172.21.15.27:6832/2923524035 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x563033e8a8d0 con 0x563033fc05c0 2024-01-11T08:28:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.567+0000 7f8ef0ff9700 10 client.7734 handle_command_reply: tid=0 2024-01-11T08:28:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.567+0000 7f8f26a90b80 1 client.7734 shutdown 2024-01-11T08:28:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.567+0000 7f8f26a90b80 1 -- 172.21.15.27:0/1379291768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 msgr2=0x563033fc2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.567+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/1379291768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 0x563033fc2be0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c037040 tx=0x7f8f0c02f070 comp rx=0 tx=0).stop 2024-01-11T08:28:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.567+0000 7f8f26a90b80 1 -- 172.21.15.27:0/1379291768 shutdown_connections 2024-01-11T08:28:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.567+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/1379291768 >> [v2:172.21.15.27:6832/2923524035,v1:172.21.15.27:6835/2923524035] conn(0x563033fc05c0 0x563033fc0990 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.567+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/1379291768 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563033fc3860 0x563033f996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.567+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/1379291768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563033f945e0 0x563033fc2be0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.567+0000 7f8ef0ff9700 0 client.7734 ms_handle_reset on v2:172.21.15.27:6832/2923524035 2024-01-11T08:28:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.567+0000 7f8f26a90b80 1 --2- 172.21.15.27:0/1379291768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563033f99e40 0x563033f991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.567+0000 7f8f26a90b80 1 -- 172.21.15.27:0/1379291768 >> 172.21.15.27:0/1379291768 conn(0x563033f96300 msgr2=0x563033fc74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.567+0000 7f8f26a90b80 1 -- 172.21.15.27:0/1379291768 shutdown_connections 2024-01-11T08:28:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.568+0000 7f8f26a90b80 1 -- 172.21.15.27:0/1379291768 wait complete. 2024-01-11T08:28:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.568+0000 7f8f26a90b80 20 client.7734 trim_cache size 0 max 16384 2024-01-11T08:28:24.571 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T08:28:24.571 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "8fddb4a7-9d66-40f1-892d-6fe86c728ee4", 2024-01-11T08:28:24.571 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-11T08:28:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 7587, 2024-01-11T08:28:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-11T08:28:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-11T08:28:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-11T08:28:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 6.6185059510000004, 2024-01-11T08:28:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 59, 2024-01-11T08:28:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 116, 2024-01-11T08:28:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 116, 2024-01-11T08:28:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 12.290104716 2024-01-11T08:28:24.572 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T08:28:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.568+0000 7f8f183ff700 1 -- 172.21.15.27:0/1604204921 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8ef4066f30 msgr2=0x7f8ef40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.568+0000 7f8f183ff700 1 --2- 172.21.15.27:0/1604204921 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8ef4066f30 0x7f8ef40693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c007fd0 tx=0x7f8f0c007d50 comp rx=0 tx=0).stop 2024-01-11T08:28:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.568+0000 7f8f183ff700 1 -- 172.21.15.27:0/1604204921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f10156ab0 msgr2=0x7f8f10167960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.568+0000 7f8f183ff700 1 --2- 172.21.15.27:0/1604204921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f10156ab0 0x7f8f10167960 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f8f0000d860 tx=0x7f8f00008040 comp rx=0 tx=0).stop 2024-01-11T08:28:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.568+0000 7f8f183ff700 1 -- 172.21.15.27:0/1604204921 shutdown_connections 2024-01-11T08:28:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.568+0000 7f8f183ff700 1 --2- 172.21.15.27:0/1604204921 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8ef4066f30 0x7f8ef40693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.568+0000 7f8f183ff700 1 --2- 172.21.15.27:0/1604204921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f10156e80 0x7f8f10144810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.568+0000 7f8f183ff700 1 --2- 172.21.15.27:0/1604204921 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f100b0550 0x7f8f10145cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.568+0000 7f8f183ff700 1 --2- 172.21.15.27:0/1604204921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f10156ab0 0x7f8f10167960 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.568+0000 7f8f183ff700 1 -- 172.21.15.27:0/1604204921 >> 172.21.15.27:0/1604204921 conn(0x7f8f100680b0 msgr2=0x7f8f1014ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:24.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.568+0000 7f8f183ff700 1 -- 172.21.15.27:0/1604204921 shutdown_connections 2024-01-11T08:28:24.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.568+0000 7f8f183ff700 1 -- 172.21.15.27:0/1604204921 wait complete. 2024-01-11T08:28:24.584 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-11T08:28:24.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.785+0000 7f84f93ff700 1 Processor -- start 2024-01-11T08:28:24.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.786+0000 7f84f93ff700 1 -- start start 2024-01-11T08:28:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.787+0000 7f84f93ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f84f40c8620 0x7f84f40af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.787+0000 7f84f93ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f4069d40 0x7f84f40b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.787+0000 7f84f93ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f84f40c89f0 0x7f84f40b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.787+0000 7f84f93ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84f4157390 con 0x7f84f40c8620 2024-01-11T08:28:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.787+0000 7f84f93ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84f40afeb0 con 0x7f84f40c89f0 2024-01-11T08:28:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.787+0000 7f84f93ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f406a200 con 0x7f84f4069d40 2024-01-11T08:28:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.787+0000 7f84f2ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f84f40c8620 0x7f84f40af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.787+0000 7f84f2ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f84f40c8620 0x7f84f40af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41424/0 (socket says 172.21.15.27:41424) 2024-01-11T08:28:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.787+0000 7f84f2ffd700 1 -- 172.21.15.27:0/909898645 learned_addr learned my addr 172.21.15.27:0/909898645 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.787+0000 7f84f37fe700 1 --2- 172.21.15.27:0/909898645 >> v2:172.21.15.77:3300/0 conn(0x7f84f40c89f0 0x7f84f40b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.787+0000 7f84f27fc700 1 --2- 172.21.15.27:0/909898645 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f4069d40 0x7f84f40b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.788+0000 7f84f2ffd700 1 -- 172.21.15.27:0/909898645 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f4069d40 msgr2=0x7f84f40b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.788+0000 7f84f2ffd700 1 --2- 172.21.15.27:0/909898645 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f4069d40 0x7f84f40b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.788+0000 7f84f2ffd700 1 -- 172.21.15.27:0/909898645 >> v2:172.21.15.77:3300/0 conn(0x7f84f40c89f0 msgr2=0x7f84f40b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.788+0000 7f84f2ffd700 1 --2- 172.21.15.27:0/909898645 >> v2:172.21.15.77:3300/0 conn(0x7f84f40c89f0 0x7f84f40b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.788+0000 7f84f2ffd700 1 -- 172.21.15.27:0/909898645 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f4156d40 con 0x7f84f40c8620 2024-01-11T08:28:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.788+0000 7f84f37fe700 1 --2- 172.21.15.27:0/909898645 >> v2:172.21.15.77:3300/0 conn(0x7f84f40c89f0 0x7f84f40b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:28:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.788+0000 7f84f2ffd700 1 --2- 172.21.15.27:0/909898645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f40c8620 0x7f84f40af970 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f84f41545c0 tx=0x7f84e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8583bfab032c3ced server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.788+0000 7f84f1ffb700 1 -- 172.21.15.27:0/909898645 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e400b860 con 0x7f84f40c8620 2024-01-11T08:28:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.788+0000 7f84f1ffb700 1 -- 172.21.15.27:0/909898645 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e400b9c0 con 0x7f84f40c8620 2024-01-11T08:28:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.788+0000 7f84f1ffb700 1 -- 172.21.15.27:0/909898645 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e400bd60 con 0x7f84f40c8620 2024-01-11T08:28:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.789+0000 7f84f93ff700 1 -- 172.21.15.27:0/909898645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f40c8620 msgr2=0x7f84f40af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.789+0000 7f84f93ff700 1 --2- 172.21.15.27:0/909898645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f40c8620 0x7f84f40af970 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f84f41545c0 tx=0x7f84e4009250 comp rx=0 tx=0).stop 2024-01-11T08:28:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.789+0000 7f84f93ff700 1 -- 172.21.15.27:0/909898645 shutdown_connections 2024-01-11T08:28:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.789+0000 7f84f93ff700 1 --2- 172.21.15.27:0/909898645 >> v2:172.21.15.77:3300/0 conn(0x7f84f40c89f0 0x7f84f40b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.789+0000 7f84f93ff700 1 --2- 172.21.15.27:0/909898645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f40c8620 0x7f84f40af970 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.789+0000 7f84f93ff700 1 --2- 172.21.15.27:0/909898645 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f4069d40 0x7f84f40b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.789+0000 7f84f93ff700 1 -- 172.21.15.27:0/909898645 >> 172.21.15.27:0/909898645 conn(0x7f84f40680e0 msgr2=0x7f84f4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.789+0000 7f84f93ff700 1 -- 172.21.15.27:0/909898645 shutdown_connections 2024-01-11T08:28:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.789+0000 7f84f93ff700 1 -- 172.21.15.27:0/909898645 wait complete. 2024-01-11T08:28:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.791+0000 7f84f93ff700 1 Processor -- start 2024-01-11T08:28:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.791+0000 7f84f93ff700 1 -- start start 2024-01-11T08:28:24.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.792+0000 7f84f93ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f4069d40 0x7f84f40abef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.792+0000 7f84f93ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84f40c8620 0x7f84f41413b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.793+0000 7f84f93ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f40c89f0 0x7f84f4160100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.793+0000 7f84f93ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f41452b0 con 0x7f84f4069d40 2024-01-11T08:28:24.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.793+0000 7f84f93ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f4154d70 con 0x7f84f40c8620 2024-01-11T08:28:24.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.793+0000 7f84f93ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f4142c60 con 0x7f84f40c89f0 2024-01-11T08:28:24.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.793+0000 7f84f2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f4069d40 0x7f84f40abef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.793+0000 7f84f2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f4069d40 0x7f84f40abef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41430/0 (socket says 172.21.15.27:41430) 2024-01-11T08:28:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.793+0000 7f84f2ffd700 1 -- 172.21.15.27:0/3191094831 learned_addr learned my addr 172.21.15.27:0/3191094831 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.793+0000 7f84f27fc700 1 --2- 172.21.15.27:0/3191094831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84f40c8620 0x7f84f41413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.793+0000 7f84f37fe700 1 --2- 172.21.15.27:0/3191094831 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f40c89f0 0x7f84f4160100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.793+0000 7f84f2ffd700 1 -- 172.21.15.27:0/3191094831 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f40c89f0 msgr2=0x7f84f4160100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.793+0000 7f84f2ffd700 1 --2- 172.21.15.27:0/3191094831 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f40c89f0 0x7f84f4160100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.793+0000 7f84f2ffd700 1 -- 172.21.15.27:0/3191094831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84f40c8620 msgr2=0x7f84f41413b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.793+0000 7f84f2ffd700 1 --2- 172.21.15.27:0/3191094831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84f40c8620 0x7f84f41413b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.793+0000 7f84f2ffd700 1 -- 172.21.15.27:0/3191094831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e40094f0 con 0x7f84f4069d40 2024-01-11T08:28:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.794+0000 7f84f2ffd700 1 --2- 172.21.15.27:0/3191094831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f4069d40 0x7f84f40abef0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f84e4009290 tx=0x7f84e4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.794+0000 7f84daffd700 1 -- 172.21.15.27:0/3191094831 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e4026480 con 0x7f84f4069d40 2024-01-11T08:28:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.794+0000 7f84daffd700 1 -- 172.21.15.27:0/3191094831 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e401f070 con 0x7f84f4069d40 2024-01-11T08:28:24.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.794+0000 7f84daffd700 1 -- 172.21.15.27:0/3191094831 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e4026830 con 0x7f84f4069d40 2024-01-11T08:28:24.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.794+0000 7f84f93ff700 1 -- 172.21.15.27:0/3191094831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f4144fd0 con 0x7f84f4069d40 2024-01-11T08:28:24.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.794+0000 7f84f93ff700 1 -- 172.21.15.27:0/3191094831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f4154a00 con 0x7f84f4069d40 2024-01-11T08:28:24.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.795+0000 7f84f93ff700 1 -- 172.21.15.27:0/3191094831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f40718d0 con 0x7f84f4069d40 2024-01-11T08:28:24.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.796+0000 7f84daffd700 1 -- 172.21.15.27:0/3191094831 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f84e4026ba0 con 0x7f84f4069d40 2024-01-11T08:28:24.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.797+0000 7f84daffd700 1 --2- 172.21.15.27:0/3191094831 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f84e0066f30 0x7f84e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:24.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.797+0000 7f84daffd700 1 -- 172.21.15.27:0/3191094831 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f84e4021070 con 0x7f84f4069d40 2024-01-11T08:28:24.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.797+0000 7f84f27fc700 1 --2- 172.21.15.27:0/3191094831 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f84e0066f30 0x7f84e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:24.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.798+0000 7f84f27fc700 1 --2- 172.21.15.27:0/3191094831 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f84e0066f30 0x7f84e00693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f84f406bfd0 tx=0x7f84e8008040 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:24.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:24.801+0000 7f84daffd700 1 -- 172.21.15.27:0/3191094831 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f84e40535d0 con 0x7f84f4069d40 2024-01-11T08:28:25.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:25.055+0000 7f84f93ff700 1 -- 172.21.15.27:0/3191094831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f84f4157020 con 0x7f84f4069d40 2024-01-11T08:28:25.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:25.055+0000 7f84daffd700 1 -- 172.21.15.27:0/3191094831 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v59) v1 ==== 53+0+73 (secure 0 0 0) 0x7f84e4056bf0 con 0x7f84f4069d40 2024-01-11T08:28:25.054 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-11T08:28:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:25.058+0000 7f84f93ff700 1 -- 172.21.15.27:0/3191094831 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f84e0066f30 msgr2=0x7f84e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:25.058+0000 7f84f93ff700 1 --2- 172.21.15.27:0/3191094831 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f84e0066f30 0x7f84e00693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f84f406bfd0 tx=0x7f84e8008040 comp rx=0 tx=0).stop 2024-01-11T08:28:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:25.058+0000 7f84f93ff700 1 -- 172.21.15.27:0/3191094831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f4069d40 msgr2=0x7f84f40abef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:25.058+0000 7f84f93ff700 1 --2- 172.21.15.27:0/3191094831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f4069d40 0x7f84f40abef0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f84e4009290 tx=0x7f84e4017000 comp rx=0 tx=0).stop 2024-01-11T08:28:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:25.058+0000 7f84f93ff700 1 -- 172.21.15.27:0/3191094831 shutdown_connections 2024-01-11T08:28:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:25.058+0000 7f84f93ff700 1 --2- 172.21.15.27:0/3191094831 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f84e0066f30 0x7f84e00693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:25.058+0000 7f84f93ff700 1 --2- 172.21.15.27:0/3191094831 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f84f40c89f0 0x7f84f4160100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:25.058+0000 7f84f93ff700 1 --2- 172.21.15.27:0/3191094831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84f4069d40 0x7f84f40abef0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:25.058+0000 7f84f93ff700 1 --2- 172.21.15.27:0/3191094831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84f40c8620 0x7f84f41413b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:25.058+0000 7f84f93ff700 1 -- 172.21.15.27:0/3191094831 >> 172.21.15.27:0/3191094831 conn(0x7f84f40680e0 msgr2=0x7f84f40b02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:25.058+0000 7f84f93ff700 1 -- 172.21.15.27:0/3191094831 shutdown_connections 2024-01-11T08:28:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:25.058+0000 7f84f93ff700 1 -- 172.21.15.27:0/3191094831 wait complete. 2024-01-11T08:28:25.074 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-11T08:28:25.075 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-11T08:28:25.075 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-11T08:28:25.075 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-11T08:28:25.075 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-11T08:28:25.075 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-11T08:28:25.075 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-11T08:28:25.075 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-11T08:28:25.075 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:28:25.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:28:25.075 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-11T08:28:25.125 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T08:28:25.125 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T08:28:25.125 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T08:28:25.125 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:28:25.125 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-11T08:28:25.125 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:28:25.125 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:28:25.125 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:28:25.125 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-11T08:28:25.126 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:28:25.126 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:28:25.126 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-11T08:28:25.126 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-11T08:28:25.126 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-11T08:28:25.126 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-11T08:28:25.126 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-11T08:28:25.126 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 41417sec preferred_lft 41417sec 2024-01-11T08:28:25.126 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-11T08:28:25.126 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:28:25.126 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:28:25.126 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-11T08:28:25.126 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-11T08:28:25.126 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T08:28:25.127 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff 2024-01-11T08:28:25.127 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-11T08:28:25.127 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:28:25.127 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::acd2:b3ff:fe6a:db4b/64 scope link 2024-01-11T08:28:25.127 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:28:25.127 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T08:28:25.127 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:fa:85:1b:3a:bc brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-11T08:28:25.127 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ecfa:85ff:fe1b:3abc/64 scope link 2024-01-11T08:28:25.127 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:28:25.127 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T08:28:25.127 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-11T08:28:25.127 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ec88:faff:fe17:41aa/64 scope link 2024-01-11T08:28:25.127 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:28:25.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:28:25.128 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-11T08:28:25.198 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-11T08:28:25.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:28:25.201 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-11T08:28:25.251 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-11T08:28:25.251 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-11T08:28:25.251 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-11T08:28:25.251 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-11T08:28:25.251 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-11T08:28:25.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:28:25.252 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T08:28:25.252 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T08:28:25.252 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-11T08:28:25.252 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-11T08:28:25.253 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-11T08:28:25.253 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T08:28:25.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:28:25.430 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T08:28:25.478 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-11T08:28:25.478 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-11T08:28:25.479 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:28:25.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:28:25.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-11T08:28:25.551 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:28:25.551 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-11T08:28:25.599 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:28:25.599 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:28:25.599 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-11T08:28:25.646 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-11T08:28:25.647 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-11T08:28:25.647 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:28:25.791 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:28:25.791 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:28:25.791 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:28:25.805 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:28:25.805 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:28:25.806 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:28:25.806 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:28:25.806 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:28:25.827 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:28:25.827 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:28:25.828 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:28:25.828 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:28:25.828 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:28:25.828 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790" 2024-01-11T08:28:25.828 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790:/ 2024-01-11T08:28:25.828 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=8fddb4a7-9d66-40f1-892d-6fe86c728ee4" will pass to kernel 2024-01-11T08:28:25.832 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-11T08:28:25.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:28:25.832 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-11T08:28:25.905 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-11T08:28:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.109+0000 7f7d48dff700 1 Processor -- start 2024-01-11T08:28:26.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.110+0000 7f7d48dff700 1 -- start start 2024-01-11T08:28:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.111+0000 7f7d48dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d440b0550 0x7f7d440ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.111+0000 7f7d48dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d440adc80 con 0x7f7d44156e80 2024-01-11T08:28:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.111+0000 7f7d48dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d440adf60 con 0x7f7d44156ab0 2024-01-11T08:28:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.111+0000 7f7d48dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d440ae240 con 0x7f7d440b0550 2024-01-11T08:28:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.111+0000 7f7d42d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7d44156e80 0x7f7d441572e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49376/0 (socket says 172.21.15.27:49376) 2024-01-11T08:28:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.111+0000 7f7d42d9d700 1 -- 172.21.15.27:0/2728920891 learned_addr learned my addr 172.21.15.27:0/2728920891 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:28:26.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.111+0000 7f7d41d9b700 1 --2- 172.21.15.27:0/2728920891 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d440b0550 0x7f7d440ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:26.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.112+0000 7f7d4159a700 1 -- 172.21.15.27:0/2728920891 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011771935 0 0) 0x7f7d440adc80 con 0x7f7d44156e80 2024-01-11T08:28:26.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.112+0000 7f7d4159a700 1 -- 172.21.15.27:0/2728920891 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d2c003570 con 0x7f7d44156e80 2024-01-11T08:28:26.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.112+0000 7f7d4159a700 1 -- 172.21.15.27:0/2728920891 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947535291 0 0) 0x7f7d440adf60 con 0x7f7d44156ab0 2024-01-11T08:28:26.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.112+0000 7f7d4159a700 1 -- 172.21.15.27:0/2728920891 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d440adc80 con 0x7f7d44156ab0 2024-01-11T08:28:26.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.112+0000 7f7d41d9b700 1 -- 172.21.15.27:0/2728920891 >> v1:172.21.15.77:6789/0 conn(0x7f7d44156ab0 legacy=0x7f7d440cc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:26.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.112+0000 7f7d41d9b700 1 -- 172.21.15.27:0/2728920891 >> v1:172.21.15.27:6789/0 conn(0x7f7d44156e80 legacy=0x7f7d441572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:26.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.112+0000 7f7d41d9b700 1 -- 172.21.15.27:0/2728920891 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d440ae520 con 0x7f7d440b0550 2024-01-11T08:28:26.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.112+0000 7f7d41d9b700 1 --2- 172.21.15.27:0/2728920891 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d440b0550 0x7f7d440ad650 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7d38009910 tx=0x7f7d380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=142f5dd612b5f837 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:26.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.113+0000 7f7d4159a700 1 -- 172.21.15.27:0/2728920891 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d38011630 con 0x7f7d440b0550 2024-01-11T08:28:26.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.113+0000 7f7d4159a700 1 -- 172.21.15.27:0/2728920891 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d38011790 con 0x7f7d440b0550 2024-01-11T08:28:26.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.113+0000 7f7d4159a700 1 -- 172.21.15.27:0/2728920891 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d38011b30 con 0x7f7d440b0550 2024-01-11T08:28:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.113+0000 7f7d48dff700 1 -- 172.21.15.27:0/2728920891 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d440b0550 msgr2=0x7f7d440ad650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.113+0000 7f7d48dff700 1 --2- 172.21.15.27:0/2728920891 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d440b0550 0x7f7d440ad650 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7d38009910 tx=0x7f7d380091a0 comp rx=0 tx=0).stop 2024-01-11T08:28:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.113+0000 7f7d48dff700 1 -- 172.21.15.27:0/2728920891 shutdown_connections 2024-01-11T08:28:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.113+0000 7f7d48dff700 1 --2- 172.21.15.27:0/2728920891 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d440b0550 0x7f7d440ad650 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.113+0000 7f7d48dff700 1 -- 172.21.15.27:0/2728920891 >> 172.21.15.27:0/2728920891 conn(0x7f7d440680b0 msgr2=0x7f7d440cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.113+0000 7f7d48dff700 1 -- 172.21.15.27:0/2728920891 shutdown_connections 2024-01-11T08:28:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.113+0000 7f7d48dff700 1 -- 172.21.15.27:0/2728920891 wait complete. 2024-01-11T08:28:26.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.115+0000 7f7d48dff700 1 Processor -- start 2024-01-11T08:28:26.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.115+0000 7f7d48dff700 1 -- start start 2024-01-11T08:28:26.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.116+0000 7f7d48dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d440b0550 0x7f7d440c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:26.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.116+0000 7f7d48dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d44156ab0 0x7f7d440cb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:26.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.116+0000 7f7d48dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d44156e80 0x7f7d44142370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:26.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.116+0000 7f7d48dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d440ca260 con 0x7f7d44156ab0 2024-01-11T08:28:26.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.116+0000 7f7d48dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d44169e00 con 0x7f7d44156e80 2024-01-11T08:28:26.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.116+0000 7f7d48dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d440cbc60 con 0x7f7d440b0550 2024-01-11T08:28:26.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.116+0000 7f7d41d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d44156ab0 0x7f7d440cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.116+0000 7f7d41d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d44156ab0 0x7f7d440cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41442/0 (socket says 172.21.15.27:41442) 2024-01-11T08:28:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.116+0000 7f7d41d9b700 1 -- 172.21.15.27:0/1192240446 learned_addr learned my addr 172.21.15.27:0/1192240446 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.116+0000 7f7d4259c700 1 --2- 172.21.15.27:0/1192240446 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d440b0550 0x7f7d440c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.116+0000 7f7d42d9d700 1 --2- 172.21.15.27:0/1192240446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d44156e80 0x7f7d44142370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:26.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.117+0000 7f7d41d9b700 1 -- 172.21.15.27:0/1192240446 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d440b0550 msgr2=0x7f7d440c9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:26.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.117+0000 7f7d41d9b700 1 --2- 172.21.15.27:0/1192240446 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d440b0550 0x7f7d440c9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:26.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.117+0000 7f7d41d9b700 1 -- 172.21.15.27:0/1192240446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d44156e80 msgr2=0x7f7d44142370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:26.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.117+0000 7f7d41d9b700 1 --2- 172.21.15.27:0/1192240446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d44156e80 0x7f7d44142370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:26.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.117+0000 7f7d41d9b700 1 -- 172.21.15.27:0/1192240446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d38009420 con 0x7f7d44156ab0 2024-01-11T08:28:26.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.117+0000 7f7d41d9b700 1 --2- 172.21.15.27:0/1192240446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d44156ab0 0x7f7d440cb720 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f7d38009fd0 tx=0x7f7d38017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:26.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.117+0000 7f7d327fc700 1 -- 172.21.15.27:0/1192240446 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3801f070 con 0x7f7d44156ab0 2024-01-11T08:28:26.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.117+0000 7f7d327fc700 1 -- 172.21.15.27:0/1192240446 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d38002f70 con 0x7f7d44156ab0 2024-01-11T08:28:26.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.117+0000 7f7d48dff700 1 -- 172.21.15.27:0/1192240446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4416ac30 con 0x7f7d44156ab0 2024-01-11T08:28:26.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.117+0000 7f7d327fc700 1 -- 172.21.15.27:0/1192240446 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d38011900 con 0x7f7d44156ab0 2024-01-11T08:28:26.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.117+0000 7f7d48dff700 1 -- 172.21.15.27:0/1192240446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4416a0a0 con 0x7f7d44156ab0 2024-01-11T08:28:26.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.118+0000 7f7d48dff700 1 -- 172.21.15.27:0/1192240446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4416adc0 con 0x7f7d44156ab0 2024-01-11T08:28:26.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.119+0000 7f7d327fc700 1 -- 172.21.15.27:0/1192240446 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7d38023070 con 0x7f7d44156ab0 2024-01-11T08:28:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.121+0000 7f7d327fc700 1 --2- 172.21.15.27:0/1192240446 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7d24066f60 0x7f7d24069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.121+0000 7f7d327fc700 1 -- 172.21.15.27:0/1192240446 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f7d38085740 con 0x7f7d44156ab0 2024-01-11T08:28:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.121+0000 7f7d4259c700 1 --2- 172.21.15.27:0/1192240446 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7d24066f60 0x7f7d24069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:26.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.122+0000 7f7d4259c700 1 --2- 172.21.15.27:0/1192240446 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7d24066f60 0x7f7d24069410 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7d34007fd0 tx=0x7f7d34007d50 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:26.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.125+0000 7f7d327fc700 1 -- 172.21.15.27:0/1192240446 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7d380528f0 con 0x7f7d44156ab0 2024-01-11T08:28:26.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.356+0000 7f7d48dff700 1 -- 172.21.15.27:0/1192240446 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d440ad150 con 0x7f7d24066f60 2024-01-11T08:28:26.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.358+0000 7f7d327fc700 1 -- 172.21.15.27:0/1192240446 <== mgr.7177 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f7d440ad150 con 0x7f7d24066f60 2024-01-11T08:28:26.357 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T08:28:26.357 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:28:26.357 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-11T08:28:26.357 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:28:26.357 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T08:28:26.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.360+0000 7f7d48dff700 1 -- 172.21.15.27:0/1192240446 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7d24066f60 msgr2=0x7f7d24069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:26.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.360+0000 7f7d48dff700 1 --2- 172.21.15.27:0/1192240446 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7d24066f60 0x7f7d24069410 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7d34007fd0 tx=0x7f7d34007d50 comp rx=0 tx=0).stop 2024-01-11T08:28:26.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.360+0000 7f7d48dff700 1 -- 172.21.15.27:0/1192240446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d44156ab0 msgr2=0x7f7d440cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:26.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.360+0000 7f7d48dff700 1 --2- 172.21.15.27:0/1192240446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d44156ab0 0x7f7d440cb720 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f7d38009fd0 tx=0x7f7d38017000 comp rx=0 tx=0).stop 2024-01-11T08:28:26.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.360+0000 7f7d48dff700 1 -- 172.21.15.27:0/1192240446 shutdown_connections 2024-01-11T08:28:26.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.360+0000 7f7d48dff700 1 --2- 172.21.15.27:0/1192240446 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7d24066f60 0x7f7d24069410 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:26.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.360+0000 7f7d48dff700 1 --2- 172.21.15.27:0/1192240446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d44156e80 0x7f7d44142370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:26.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.360+0000 7f7d48dff700 1 --2- 172.21.15.27:0/1192240446 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d440b0550 0x7f7d440c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:26.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.360+0000 7f7d48dff700 1 --2- 172.21.15.27:0/1192240446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d44156ab0 0x7f7d440cb720 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:26.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.360+0000 7f7d48dff700 1 -- 172.21.15.27:0/1192240446 >> 172.21.15.27:0/1192240446 conn(0x7f7d440680b0 msgr2=0x7f7d4414d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:26.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.360+0000 7f7d48dff700 1 -- 172.21.15.27:0/1192240446 shutdown_connections 2024-01-11T08:28:26.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.360+0000 7f7d48dff700 1 -- 172.21.15.27:0/1192240446 wait complete. 2024-01-11T08:28:26.377 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-11T08:28:26.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.581+0000 7f5fb1fff700 1 Processor -- start 2024-01-11T08:28:26.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.581+0000 7f5fb1fff700 1 -- start start 2024-01-11T08:28:26.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.583+0000 7f5fb1fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5fac15a1b0 0x7f5fac15a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:26.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.583+0000 7f5fb1fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5fac15ae70 0x7f5fac15b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:26.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.583+0000 7f5fb1fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fac0cf850 con 0x7f5fac0ccd70 2024-01-11T08:28:26.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.583+0000 7f5fb1fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fac0cec40 con 0x7f5fac15ae70 2024-01-11T08:28:26.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.583+0000 7f5fb1fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fac0cef20 con 0x7f5fac15a1b0 2024-01-11T08:28:26.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.583+0000 7f5fabfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5fac15ae70 0x7f5fac15b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:26.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.583+0000 7f5fabfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5fac15ae70 0x7f5fac15b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41458/0 (socket says 172.21.15.27:41458) 2024-01-11T08:28:26.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.583+0000 7f5fabfff700 1 -- 172.21.15.27:0/1768540384 learned_addr learned my addr 172.21.15.27:0/1768540384 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:26.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.583+0000 7f5fab7fe700 1 --2- 172.21.15.27:0/1768540384 >> v2:172.21.15.77:3300/0 conn(0x7f5fac15a1b0 0x7f5fac15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:26.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.583+0000 7f5fabfff700 1 -- 172.21.15.27:0/1768540384 >> v2:172.21.15.77:3300/0 conn(0x7f5fac15a1b0 msgr2=0x7f5fac15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:26.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.583+0000 7f5fabfff700 1 --2- 172.21.15.27:0/1768540384 >> v2:172.21.15.77:3300/0 conn(0x7f5fac15a1b0 0x7f5fac15a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:26.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.583+0000 7f5fabfff700 1 -- 172.21.15.27:0/1768540384 >> v1:172.21.15.77:6789/0 conn(0x7f5fac0ccd70 legacy=0x7f5fac0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:26.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.583+0000 7f5fabfff700 1 -- 172.21.15.27:0/1768540384 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fac0cf200 con 0x7f5fac15ae70 2024-01-11T08:28:26.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.583+0000 7f5fab7fe700 1 --2- 172.21.15.27:0/1768540384 >> v2:172.21.15.77:3300/0 conn(0x7f5fac15a1b0 0x7f5fac15a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:28:26.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.584+0000 7f5fabfff700 1 --2- 172.21.15.27:0/1768540384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fac15ae70 0x7f5fac15b2a0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f5fac0d16f0 tx=0x7f5fa4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e15dfd4bbdcc93a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:26.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.584+0000 7f5faa7fc700 1 -- 172.21.15.27:0/1768540384 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa400e070 con 0x7f5fac15ae70 2024-01-11T08:28:26.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.584+0000 7f5faa7fc700 1 -- 172.21.15.27:0/1768540384 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa400ba10 con 0x7f5fac15ae70 2024-01-11T08:28:26.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.584+0000 7f5faa7fc700 1 -- 172.21.15.27:0/1768540384 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa40092e0 con 0x7f5fac15ae70 2024-01-11T08:28:26.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.585+0000 7f5fb1fff700 1 -- 172.21.15.27:0/1768540384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fac15ae70 msgr2=0x7f5fac15b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:26.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.585+0000 7f5fb1fff700 1 --2- 172.21.15.27:0/1768540384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fac15ae70 0x7f5fac15b2a0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f5fac0d16f0 tx=0x7f5fa4009840 comp rx=0 tx=0).stop 2024-01-11T08:28:26.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.585+0000 7f5fb1fff700 1 -- 172.21.15.27:0/1768540384 shutdown_connections 2024-01-11T08:28:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.585+0000 7f5fb1fff700 1 --2- 172.21.15.27:0/1768540384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fac15ae70 0x7f5fac15b2a0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.585+0000 7f5fb1fff700 1 --2- 172.21.15.27:0/1768540384 >> v2:172.21.15.77:3300/0 conn(0x7f5fac15a1b0 0x7f5fac15a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.585+0000 7f5fb1fff700 1 -- 172.21.15.27:0/1768540384 >> 172.21.15.27:0/1768540384 conn(0x7f5fac06ed40 msgr2=0x7f5fac06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.585+0000 7f5fb1fff700 1 -- 172.21.15.27:0/1768540384 shutdown_connections 2024-01-11T08:28:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.585+0000 7f5fb1fff700 1 -- 172.21.15.27:0/1768540384 wait complete. 2024-01-11T08:28:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.586+0000 7f5fb1fff700 1 Processor -- start 2024-01-11T08:28:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.586+0000 7f5fb1fff700 1 -- start start 2024-01-11T08:28:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.587+0000 7f5fb1fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fac0ccd70 0x7f5fac148ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.587+0000 7f5fb1fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5fac15a1b0 0x7f5fac0c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.587+0000 7f5fb1fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fac15ae70 0x7f5fac147f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.587+0000 7f5fb1fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fac0c7bd0 con 0x7f5fac0ccd70 2024-01-11T08:28:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.587+0000 7f5fb1fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fac0b4380 con 0x7f5fac15ae70 2024-01-11T08:28:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.587+0000 7f5fb1fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fac0b5e40 con 0x7f5fac15a1b0 2024-01-11T08:28:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.588+0000 7f5fab7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fac0ccd70 0x7f5fac148ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.588+0000 7f5fab7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fac0ccd70 0x7f5fac148ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41464/0 (socket says 172.21.15.27:41464) 2024-01-11T08:28:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.588+0000 7f5fab7fe700 1 -- 172.21.15.27:0/736226593 learned_addr learned my addr 172.21.15.27:0/736226593 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.588+0000 7f5fabfff700 1 --2- 172.21.15.27:0/736226593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fac15ae70 0x7f5fac147f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.588+0000 7f5faaffd700 1 --2- 172.21.15.27:0/736226593 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5fac15a1b0 0x7f5fac0c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.588+0000 7f5fab7fe700 1 -- 172.21.15.27:0/736226593 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5fac15a1b0 msgr2=0x7f5fac0c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.588+0000 7f5fab7fe700 1 --2- 172.21.15.27:0/736226593 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5fac15a1b0 0x7f5fac0c7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.588+0000 7f5fab7fe700 1 -- 172.21.15.27:0/736226593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fac15ae70 msgr2=0x7f5fac147f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.588+0000 7f5fab7fe700 1 --2- 172.21.15.27:0/736226593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fac15ae70 0x7f5fac147f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.588+0000 7f5fab7fe700 1 -- 172.21.15.27:0/736226593 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa4009ae0 con 0x7f5fac0ccd70 2024-01-11T08:28:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.588+0000 7f5fab7fe700 1 --2- 172.21.15.27:0/736226593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fac0ccd70 0x7f5fac148ae0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c00c830 tx=0x7f5f9c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:26.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.589+0000 7f5f937fe700 1 -- 172.21.15.27:0/736226593 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c011600 con 0x7f5fac0ccd70 2024-01-11T08:28:26.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.589+0000 7f5f937fe700 1 -- 172.21.15.27:0/736226593 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9c011760 con 0x7f5fac0ccd70 2024-01-11T08:28:26.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.589+0000 7f5f937fe700 1 -- 172.21.15.27:0/736226593 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c011b20 con 0x7f5fac0ccd70 2024-01-11T08:28:26.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.589+0000 7f5fb1fff700 1 -- 172.21.15.27:0/736226593 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fac0b60e0 con 0x7f5fac0ccd70 2024-01-11T08:28:26.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.589+0000 7f5fb1fff700 1 -- 172.21.15.27:0/736226593 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fac0b48b0 con 0x7f5fac0ccd70 2024-01-11T08:28:26.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.590+0000 7f5fb1fff700 1 -- 172.21.15.27:0/736226593 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fac04c3c0 con 0x7f5fac0ccd70 2024-01-11T08:28:26.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.591+0000 7f5f937fe700 1 -- 172.21.15.27:0/736226593 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 57) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5f9c018360 con 0x7f5fac0ccd70 2024-01-11T08:28:26.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.592+0000 7f5f937fe700 1 --2- 172.21.15.27:0/736226593 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f98066f30 0x7f5f980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:26.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.592+0000 7f5f937fe700 1 -- 172.21.15.27:0/736226593 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f5f9c0845f0 con 0x7f5fac0ccd70 2024-01-11T08:28:26.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.593+0000 7f5faaffd700 1 --2- 172.21.15.27:0/736226593 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f98066f30 0x7f5f980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:26.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.593+0000 7f5faaffd700 1 --2- 172.21.15.27:0/736226593 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f98066f30 0x7f5f980693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5fac16cfa0 tx=0x7f5fa0009060 comp rx=0 tx=0).ready entity=mgr.7177 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:26.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.596+0000 7f5f937fe700 1 -- 172.21.15.27:0/736226593 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5f9c0517a0 con 0x7f5fac0ccd70 2024-01-11T08:28:26.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:26.856+0000 7f5fb1fff700 1 -- 172.21.15.27:0/736226593 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f5fac0cd6e0 con 0x7f5fac0ccd70 2024-01-11T08:28:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:27.766+0000 7f5f937fe700 1 -- 172.21.15.27:0/736226593 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 58) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f5f9c051b70 con 0x7f5fac0ccd70 2024-01-11T08:28:27.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:27.768+0000 7f5f937fe700 1 -- 172.21.15.27:0/736226593 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v58) v1 ==== 92+0+0 (secure 0 0 0) 0x7f5f9c07d190 con 0x7f5fac0ccd70 2024-01-11T08:28:27.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:27.771+0000 7f5fb1fff700 1 -- 172.21.15.27:0/736226593 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f98066f30 msgr2=0x7f5f980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:27.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:27.771+0000 7f5fb1fff700 1 --2- 172.21.15.27:0/736226593 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f98066f30 0x7f5f980693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5fac16cfa0 tx=0x7f5fa0009060 comp rx=0 tx=0).stop 2024-01-11T08:28:27.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:27.771+0000 7f5fb1fff700 1 -- 172.21.15.27:0/736226593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fac0ccd70 msgr2=0x7f5fac148ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:27.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:27.771+0000 7f5fb1fff700 1 --2- 172.21.15.27:0/736226593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fac0ccd70 0x7f5fac148ae0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c00c830 tx=0x7f5f9c008040 comp rx=0 tx=0).stop 2024-01-11T08:28:27.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:27.771+0000 7f5fb1fff700 1 -- 172.21.15.27:0/736226593 shutdown_connections 2024-01-11T08:28:27.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:27.771+0000 7f5fb1fff700 1 --2- 172.21.15.27:0/736226593 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f98066f30 0x7f5f980693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:27.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:27.771+0000 7f5fb1fff700 1 --2- 172.21.15.27:0/736226593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fac15ae70 0x7f5fac147f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:27.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:27.771+0000 7f5fb1fff700 1 --2- 172.21.15.27:0/736226593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fac0ccd70 0x7f5fac148ae0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:27.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:27.771+0000 7f5fb1fff700 1 --2- 172.21.15.27:0/736226593 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5fac15a1b0 0x7f5fac0c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:27.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:27.771+0000 7f5fb1fff700 1 -- 172.21.15.27:0/736226593 >> 172.21.15.27:0/736226593 conn(0x7f5fac06ed40 msgr2=0x7f5fac0c5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:27.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:27.771+0000 7f5fb1fff700 1 -- 172.21.15.27:0/736226593 shutdown_connections 2024-01-11T08:28:27.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:27.771+0000 7f5fb1fff700 1 -- 172.21.15.27:0/736226593 wait complete. 2024-01-11T08:28:27.791 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-11T08:28:27.979 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:27.980+0000 7fd3e3c411c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:28:27.986 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:27.987+0000 7ff57e8921c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:28:28.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.007+0000 7f46dafff700 1 Processor -- start 2024-01-11T08:28:28.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.007+0000 7f46dafff700 1 -- start start 2024-01-11T08:28:28.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.009+0000 7f46dafff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46d4069540 0x7f46d4162270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.009+0000 7f46dafff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d40699d0 con 0x7f46d4157410 2024-01-11T08:28:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.009+0000 7f46dafff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d4069cb0 con 0x7f46d41577e0 2024-01-11T08:28:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.009+0000 7f46dafff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d4069f90 con 0x7f46d4069540 2024-01-11T08:28:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.009+0000 7f46d8d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f46d4157410 0x7f46d40ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49388/0 (socket says 172.21.15.27:49388) 2024-01-11T08:28:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.009+0000 7f46d8d9b700 1 -- 172.21.15.27:0/3513005293 learned_addr learned my addr 172.21.15.27:0/3513005293 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:28:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.009+0000 7f46d959c700 1 --2- 172.21.15.27:0/3513005293 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46d4069540 0x7f46d4162270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.009+0000 7f46d37fe700 1 -- 172.21.15.27:0/3513005293 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384013962 0 0) 0x7f46d40699d0 con 0x7f46d4157410 2024-01-11T08:28:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.009+0000 7f46d37fe700 1 -- 172.21.15.27:0/3513005293 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c4003570 con 0x7f46d4157410 2024-01-11T08:28:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.009+0000 7f46d37fe700 1 -- 172.21.15.27:0/3513005293 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613958003 0 0) 0x7f46d4069cb0 con 0x7f46d41577e0 2024-01-11T08:28:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.009+0000 7f46d37fe700 1 -- 172.21.15.27:0/3513005293 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d40699d0 con 0x7f46d41577e0 2024-01-11T08:28:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.010+0000 7f46d959c700 1 -- 172.21.15.27:0/3513005293 >> v1:172.21.15.77:6789/0 conn(0x7f46d41577e0 legacy=0x7f46d40ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.010+0000 7f46d959c700 1 -- 172.21.15.27:0/3513005293 >> v1:172.21.15.27:6789/0 conn(0x7f46d4157410 legacy=0x7f46d40ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.010+0000 7f46d959c700 1 -- 172.21.15.27:0/3513005293 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d406a270 con 0x7f46d4069540 2024-01-11T08:28:28.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.010+0000 7f46d959c700 1 --2- 172.21.15.27:0/3513005293 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46d4069540 0x7f46d4162270 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f46cc009e90 tx=0x7f46cc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=705689d8418f5d8f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:28.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.010+0000 7f46d37fe700 1 -- 172.21.15.27:0/3513005293 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc00e070 con 0x7f46d4069540 2024-01-11T08:28:28.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.010+0000 7f46d37fe700 1 -- 172.21.15.27:0/3513005293 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46cc00bb80 con 0x7f46d4069540 2024-01-11T08:28:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.010+0000 7f46d37fe700 1 -- 172.21.15.27:0/3513005293 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc012430 con 0x7f46d4069540 2024-01-11T08:28:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.011+0000 7f46dafff700 1 -- 172.21.15.27:0/3513005293 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46d4069540 msgr2=0x7f46d4162270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.011+0000 7f46dafff700 1 --2- 172.21.15.27:0/3513005293 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46d4069540 0x7f46d4162270 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f46cc009e90 tx=0x7f46cc009780 comp rx=0 tx=0).stop 2024-01-11T08:28:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.011+0000 7f46dafff700 1 -- 172.21.15.27:0/3513005293 shutdown_connections 2024-01-11T08:28:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.011+0000 7f46dafff700 1 --2- 172.21.15.27:0/3513005293 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46d4069540 0x7f46d4162270 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.011+0000 7f46dafff700 1 -- 172.21.15.27:0/3513005293 >> 172.21.15.27:0/3513005293 conn(0x7f46d40680a0 msgr2=0x7f46d40cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.011+0000 7f46dafff700 1 -- 172.21.15.27:0/3513005293 shutdown_connections 2024-01-11T08:28:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.011+0000 7f46dafff700 1 -- 172.21.15.27:0/3513005293 wait complete. 2024-01-11T08:28:28.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.013+0000 7f46dafff700 1 Processor -- start 2024-01-11T08:28:28.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.013+0000 7f46dafff700 1 -- start start 2024-01-11T08:28:28.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.014+0000 7f46dafff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46d4069540 0x7f46d4155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:28.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.014+0000 7f46dafff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46d4157410 0x7f46d41511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:28.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.014+0000 7f46dafff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46d41577e0 0x7f46d4151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:28.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.014+0000 7f46dafff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d4167ba0 con 0x7f46d4157410 2024-01-11T08:28:28.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.014+0000 7f46dafff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d40c9100 con 0x7f46d4069540 2024-01-11T08:28:28.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.014+0000 7f46dafff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d40c93e0 con 0x7f46d41577e0 2024-01-11T08:28:28.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.014+0000 7f46d3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46d4157410 0x7f46d41511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:28.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.014+0000 7f46d3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46d4157410 0x7f46d41511e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41490/0 (socket says 172.21.15.27:41490) 2024-01-11T08:28:28.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.014+0000 7f46d3fff700 1 -- 172.21.15.27:0/574883131 learned_addr learned my addr 172.21.15.27:0/574883131 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:28.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.014+0000 7f46d8d9b700 1 --2- 172.21.15.27:0/574883131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46d4069540 0x7f46d4155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:28.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.014+0000 7f46d959c700 1 --2- 172.21.15.27:0/574883131 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46d41577e0 0x7f46d4151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:28.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.014+0000 7f46d3fff700 1 -- 172.21.15.27:0/574883131 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46d41577e0 msgr2=0x7f46d4151720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:28.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.014+0000 7f46d3fff700 1 --2- 172.21.15.27:0/574883131 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46d41577e0 0x7f46d4151720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.014+0000 7f46d3fff700 1 -- 172.21.15.27:0/574883131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46d4069540 msgr2=0x7f46d4155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:28.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.014+0000 7f46d3fff700 1 --2- 172.21.15.27:0/574883131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46d4069540 0x7f46d4155220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.014+0000 7f46d3fff700 1 -- 172.21.15.27:0/574883131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc009a20 con 0x7f46d4157410 2024-01-11T08:28:28.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.015+0000 7f46d3fff700 1 --2- 172.21.15.27:0/574883131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46d4157410 0x7f46d41511e0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f46c000c590 tx=0x7f46c0007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:28.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.015+0000 7f46d0ff9700 1 -- 172.21.15.27:0/574883131 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0010510 con 0x7f46d4157410 2024-01-11T08:28:28.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.015+0000 7f46d0ff9700 1 -- 172.21.15.27:0/574883131 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c0010690 con 0x7f46d4157410 2024-01-11T08:28:28.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.015+0000 7f46d0ff9700 1 -- 172.21.15.27:0/574883131 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0010a50 con 0x7f46d4157410 2024-01-11T08:28:28.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.015+0000 7f46dafff700 1 -- 172.21.15.27:0/574883131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d40c96c0 con 0x7f46d4157410 2024-01-11T08:28:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.015+0000 7f46dafff700 1 -- 172.21.15.27:0/574883131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d4155760 con 0x7f46d4157410 2024-01-11T08:28:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.016+0000 7f46dafff700 1 -- 172.21.15.27:0/574883131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d40ae730 con 0x7f46d4157410 2024-01-11T08:28:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.017+0000 7f46d0ff9700 1 -- 172.21.15.27:0/574883131 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 58) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f46c001b3b0 con 0x7f46d4157410 2024-01-11T08:28:28.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.019+0000 7f46d0ff9700 1 --2- 172.21.15.27:0/574883131 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f46b8066f80 0x7f46b8069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:28.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.019+0000 7f46d0ff9700 1 -- 172.21.15.27:0/574883131 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5733+0+0 (secure 0 0 0) 0x7f46c0083ce0 con 0x7f46d4157410 2024-01-11T08:28:28.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.019+0000 7f46d8d9b700 1 -- 172.21.15.27:0/574883131 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f46b8066f80 msgr2=0x7f46b8069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:28.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.019+0000 7f46d8d9b700 1 --2- 172.21.15.27:0/574883131 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f46b8066f80 0x7f46b8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:28:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.023+0000 7f46d0ff9700 1 -- 172.21.15.27:0/574883131 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f46c0050e90 con 0x7f46d4157410 2024-01-11T08:28:28.158 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:28.158+0000 7fd3e3c411c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:28:28.182 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:28.184+0000 7ff57e8921c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:28:28.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.220+0000 7f46d8d9b700 1 -- 172.21.15.27:0/574883131 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f46b8066f80 msgr2=0x7f46b8069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:28.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.220+0000 7f46d8d9b700 1 --2- 172.21.15.27:0/574883131 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f46b8066f80 0x7f46b8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:28:28.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.268+0000 7f46dafff700 1 -- 172.21.15.27:0/574883131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f46d404c3c0 con 0x7f46d4157410 2024-01-11T08:28:28.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.272+0000 7f46d0ff9700 1 -- 172.21.15.27:0/574883131 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v41)=0 v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7f46c0017070 con 0x7f46d4157410 2024-01-11T08:28:28.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.274+0000 7f46dafff700 1 -- 172.21.15.27:0/574883131 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f46b8066f80 msgr2=0x7f46b8069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:28:28.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.274+0000 7f46dafff700 1 --2- 172.21.15.27:0/574883131 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f46b8066f80 0x7f46b8069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.274+0000 7f46dafff700 1 -- 172.21.15.27:0/574883131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46d4157410 msgr2=0x7f46d41511e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:28.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.274+0000 7f46dafff700 1 --2- 172.21.15.27:0/574883131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46d4157410 0x7f46d41511e0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f46c000c590 tx=0x7f46c0007de0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.274+0000 7f46dafff700 1 -- 172.21.15.27:0/574883131 shutdown_connections 2024-01-11T08:28:28.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.274+0000 7f46dafff700 1 --2- 172.21.15.27:0/574883131 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f46b8066f80 0x7f46b8069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.274+0000 7f46dafff700 1 --2- 172.21.15.27:0/574883131 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f46d41577e0 0x7f46d4151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.274+0000 7f46dafff700 1 --2- 172.21.15.27:0/574883131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46d4069540 0x7f46d4155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.274+0000 7f46dafff700 1 --2- 172.21.15.27:0/574883131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46d4157410 0x7f46d41511e0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.274+0000 7f46dafff700 1 -- 172.21.15.27:0/574883131 >> 172.21.15.27:0/574883131 conn(0x7f46d40680a0 msgr2=0x7f46d4152920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:28.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.275+0000 7f46dafff700 1 -- 172.21.15.27:0/574883131 shutdown_connections 2024-01-11T08:28:28.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.275+0000 7f46dafff700 1 -- 172.21.15.27:0/574883131 wait complete. 2024-01-11T08:28:28.292 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-11T08:28:28.473 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:28.473+0000 7fd3e3c411c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:28:28.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.500+0000 7fde8ddff700 1 Processor -- start 2024-01-11T08:28:28.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.500+0000 7fde8ddff700 1 -- start start 2024-01-11T08:28:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.502+0000 7fde8ddff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fde8813ff80 0x7fde8815a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.502+0000 7fde8ddff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde8815a560 0x7fde880c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.502+0000 7fde8ddff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde880c6b30 con 0x7fde8813fbb0 2024-01-11T08:28:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.502+0000 7fde8ddff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde8815a9a0 con 0x7fde8813ff80 2024-01-11T08:28:28.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.502+0000 7fde8ddff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde880c6800 con 0x7fde8815a560 2024-01-11T08:28:28.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.502+0000 7fde877fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fde8813fbb0 0x7fde880ca7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49392/0 (socket says 172.21.15.27:49392) 2024-01-11T08:28:28.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.502+0000 7fde877fe700 1 -- 172.21.15.27:0/3053973882 learned_addr learned my addr 172.21.15.27:0/3053973882 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:28:28.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.502+0000 7fde86ffd700 1 --2- 172.21.15.27:0/3053973882 >> v2:172.21.15.27:3300/0 conn(0x7fde8813ff80 0x7fde8815a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:28.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.502+0000 7fde87fff700 1 --2- 172.21.15.27:0/3053973882 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde8815a560 0x7fde880c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:28.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.503+0000 7fde867fc700 1 -- 172.21.15.27:0/3053973882 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569137366 0 0) 0x7fde880c6b30 con 0x7fde8813fbb0 2024-01-11T08:28:28.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.503+0000 7fde867fc700 1 -- 172.21.15.27:0/3053973882 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde70003570 con 0x7fde8813fbb0 2024-01-11T08:28:28.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.503+0000 7fde86ffd700 1 -- 172.21.15.27:0/3053973882 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde8815a560 msgr2=0x7fde880c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:28.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.503+0000 7fde86ffd700 1 --2- 172.21.15.27:0/3053973882 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde8815a560 0x7fde880c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.503+0000 7fde86ffd700 1 -- 172.21.15.27:0/3053973882 >> v1:172.21.15.27:6789/0 conn(0x7fde8813fbb0 legacy=0x7fde880ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:28.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.503+0000 7fde86ffd700 1 -- 172.21.15.27:0/3053973882 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde880cb240 con 0x7fde8813ff80 2024-01-11T08:28:28.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.503+0000 7fde86ffd700 1 --2- 172.21.15.27:0/3053973882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde8813ff80 0x7fde8815a020 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fde7c0098e0 tx=0x7fde7c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e09d1c10d2ca46cd server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:28.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.503+0000 7fde867fc700 1 -- 172.21.15.27:0/3053973882 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c011630 con 0x7fde8813ff80 2024-01-11T08:28:28.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.503+0000 7fde867fc700 1 -- 172.21.15.27:0/3053973882 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7c011790 con 0x7fde8813ff80 2024-01-11T08:28:28.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.504+0000 7fde867fc700 1 -- 172.21.15.27:0/3053973882 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c011b30 con 0x7fde8813ff80 2024-01-11T08:28:28.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.504+0000 7fde8ddff700 1 -- 172.21.15.27:0/3053973882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde8813ff80 msgr2=0x7fde8815a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:28.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.504+0000 7fde8ddff700 1 --2- 172.21.15.27:0/3053973882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde8813ff80 0x7fde8815a020 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fde7c0098e0 tx=0x7fde7c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.504+0000 7fde8ddff700 1 -- 172.21.15.27:0/3053973882 shutdown_connections 2024-01-11T08:28:28.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.504+0000 7fde8ddff700 1 --2- 172.21.15.27:0/3053973882 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde8815a560 0x7fde880c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.504+0000 7fde8ddff700 1 --2- 172.21.15.27:0/3053973882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde8813ff80 0x7fde8815a020 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.504+0000 7fde8ddff700 1 -- 172.21.15.27:0/3053973882 >> 172.21.15.27:0/3053973882 conn(0x7fde880680a0 msgr2=0x7fde880cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:28.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.504+0000 7fde8ddff700 1 -- 172.21.15.27:0/3053973882 shutdown_connections 2024-01-11T08:28:28.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.504+0000 7fde8ddff700 1 -- 172.21.15.27:0/3053973882 wait complete. 2024-01-11T08:28:28.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.506+0000 7fde8ddff700 1 Processor -- start 2024-01-11T08:28:28.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.507+0000 7fde8ddff700 1 -- start start 2024-01-11T08:28:28.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.508+0000 7fde8ddff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde8813fbb0 0x7fde8813da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:28.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.508+0000 7fde8ddff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde8813ff80 0x7fde880c3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:28.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.508+0000 7fde8ddff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde8815a560 0x7fde88169120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:28.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.508+0000 7fde8ddff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde88138310 con 0x7fde8813fbb0 2024-01-11T08:28:28.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.508+0000 7fde8ddff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8816e260 con 0x7fde8815a560 2024-01-11T08:28:28.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.508+0000 7fde8ddff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde8816d650 con 0x7fde8813ff80 2024-01-11T08:28:28.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.508+0000 7fde877fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde8813fbb0 0x7fde8813da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:28.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.508+0000 7fde877fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde8813fbb0 0x7fde8813da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41510/0 (socket says 172.21.15.27:41510) 2024-01-11T08:28:28.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.508+0000 7fde877fe700 1 -- 172.21.15.27:0/238347324 learned_addr learned my addr 172.21.15.27:0/238347324 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:28.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.508+0000 7fde87fff700 1 --2- 172.21.15.27:0/238347324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde8815a560 0x7fde88169120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:28.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.508+0000 7fde86ffd700 1 --2- 172.21.15.27:0/238347324 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde8813ff80 0x7fde880c3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:28.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.509+0000 7fde877fe700 1 -- 172.21.15.27:0/238347324 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde8813ff80 msgr2=0x7fde880c3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:28.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.509+0000 7fde877fe700 1 --2- 172.21.15.27:0/238347324 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde8813ff80 0x7fde880c3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.509+0000 7fde877fe700 1 -- 172.21.15.27:0/238347324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde8815a560 msgr2=0x7fde88169120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:28.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.509+0000 7fde877fe700 1 --2- 172.21.15.27:0/238347324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde8815a560 0x7fde88169120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.509+0000 7fde877fe700 1 -- 172.21.15.27:0/238347324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7c009420 con 0x7fde8813fbb0 2024-01-11T08:28:28.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.509+0000 7fde877fe700 1 --2- 172.21.15.27:0/238347324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde8813fbb0 0x7fde8813da60 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fde7800c790 tx=0x7fde78007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:28.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.509+0000 7fde6f7fe700 1 -- 172.21.15.27:0/238347324 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde780072e0 con 0x7fde8813fbb0 2024-01-11T08:28:28.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.509+0000 7fde6f7fe700 1 -- 172.21.15.27:0/238347324 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde78007440 con 0x7fde8813fbb0 2024-01-11T08:28:28.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.509+0000 7fde6f7fe700 1 -- 172.21.15.27:0/238347324 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde780057f0 con 0x7fde8813fbb0 2024-01-11T08:28:28.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.509+0000 7fde8ddff700 1 -- 172.21.15.27:0/238347324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8816d930 con 0x7fde8813fbb0 2024-01-11T08:28:28.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.510+0000 7fde8ddff700 1 -- 172.21.15.27:0/238347324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8816d320 con 0x7fde8813fbb0 2024-01-11T08:28:28.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.511+0000 7fde8ddff700 1 -- 172.21.15.27:0/238347324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde8804c3c0 con 0x7fde8813fbb0 2024-01-11T08:28:28.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.512+0000 7fde6f7fe700 1 -- 172.21.15.27:0/238347324 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 58) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fde78005be0 con 0x7fde8813fbb0 2024-01-11T08:28:28.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.513+0000 7fde6f7fe700 1 --2- 172.21.15.27:0/238347324 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fde74066f80 0x7fde74069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:28.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.513+0000 7fde6f7fe700 1 -- 172.21.15.27:0/238347324 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5733+0+0 (secure 0 0 0) 0x7fde78083960 con 0x7fde8813fbb0 2024-01-11T08:28:28.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.513+0000 7fde86ffd700 1 -- 172.21.15.27:0/238347324 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fde74066f80 msgr2=0x7fde74069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:28.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.513+0000 7fde86ffd700 1 --2- 172.21.15.27:0/238347324 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fde74066f80 0x7fde74069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:28:28.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.521+0000 7fde6f7fe700 1 -- 172.21.15.27:0/238347324 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fde7804b5a0 con 0x7fde8813fbb0 2024-01-11T08:28:28.527 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:28.528+0000 7ff57e8921c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:28:28.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.714+0000 7fde86ffd700 1 -- 172.21.15.27:0/238347324 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fde74066f80 msgr2=0x7fde74069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:28.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.714+0000 7fde86ffd700 1 --2- 172.21.15.27:0/238347324 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fde74066f80 0x7fde74069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:28:28.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.726+0000 7fde8ddff700 1 -- 172.21.15.27:0/238347324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fde880cb520 con 0x7fde8813fbb0 2024-01-11T08:28:28.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.730+0000 7fde6f7fe700 1 -- 172.21.15.27:0/238347324 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v42)=0 v42) v1 ==== 133+0+0 (secure 0 0 0) 0x7fde78017070 con 0x7fde8813fbb0 2024-01-11T08:28:28.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.732+0000 7fde8ddff700 1 -- 172.21.15.27:0/238347324 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fde74066f80 msgr2=0x7fde74069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:28:28.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.732+0000 7fde8ddff700 1 --2- 172.21.15.27:0/238347324 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fde74066f80 0x7fde74069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.732+0000 7fde8ddff700 1 -- 172.21.15.27:0/238347324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde8813fbb0 msgr2=0x7fde8813da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:28.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.732+0000 7fde8ddff700 1 --2- 172.21.15.27:0/238347324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde8813fbb0 0x7fde8813da60 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fde7800c790 tx=0x7fde78007c80 comp rx=0 tx=0).stop 2024-01-11T08:28:28.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.732+0000 7fde8ddff700 1 -- 172.21.15.27:0/238347324 shutdown_connections 2024-01-11T08:28:28.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.732+0000 7fde8ddff700 1 --2- 172.21.15.27:0/238347324 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fde74066f80 0x7fde74069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.732+0000 7fde8ddff700 1 --2- 172.21.15.27:0/238347324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde8815a560 0x7fde88169120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.732+0000 7fde8ddff700 1 --2- 172.21.15.27:0/238347324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde8813fbb0 0x7fde8813da60 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.732+0000 7fde8ddff700 1 --2- 172.21.15.27:0/238347324 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde8813ff80 0x7fde880c3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:28.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.733+0000 7fde8ddff700 1 -- 172.21.15.27:0/238347324 >> 172.21.15.27:0/238347324 conn(0x7fde880680a0 msgr2=0x7fde880c8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:28.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.733+0000 7fde8ddff700 1 -- 172.21.15.27:0/238347324 shutdown_connections 2024-01-11T08:28:28.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:28.733+0000 7fde8ddff700 1 -- 172.21.15.27:0/238347324 wait complete. 2024-01-11T08:28:28.751 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:28.751 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-01-11T08:28:28.804 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-01-11T08:28:29.001 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:29.001+0000 7fd3e3c411c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:28:29.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.013+0000 7efe287ff700 1 Processor -- start 2024-01-11T08:28:29.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.013+0000 7efe287ff700 1 -- start start 2024-01-11T08:28:29.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.014+0000 7efe287ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efe201577e0 0x7efe200ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:29.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.015+0000 7efe287ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe200699d0 con 0x7efe20157410 2024-01-11T08:28:29.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.015+0000 7efe287ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe20069cb0 con 0x7efe20069540 2024-01-11T08:28:29.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.015+0000 7efe287ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe20069f90 con 0x7efe201577e0 2024-01-11T08:28:29.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.015+0000 7efe2659b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efe20157410 0x7efe200ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49404/0 (socket says 172.21.15.27:49404) 2024-01-11T08:28:29.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.015+0000 7efe2659b700 1 -- 172.21.15.27:0/3002907968 learned_addr learned my addr 172.21.15.27:0/3002907968 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:28:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.015+0000 7efe25d9a700 1 --2- 172.21.15.27:0/3002907968 >> v2:172.21.15.77:3300/0 conn(0x7efe201577e0 0x7efe200ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.015+0000 7efe25599700 1 -- 172.21.15.27:0/3002907968 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020668446 0 0) 0x7efe200699d0 con 0x7efe20157410 2024-01-11T08:28:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.015+0000 7efe25599700 1 -- 172.21.15.27:0/3002907968 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe0c003570 con 0x7efe20157410 2024-01-11T08:28:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.015+0000 7efe25599700 1 -- 172.21.15.27:0/3002907968 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41033050 0 0) 0x7efe20069cb0 con 0x7efe20069540 2024-01-11T08:28:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.015+0000 7efe25599700 1 -- 172.21.15.27:0/3002907968 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe200699d0 con 0x7efe20069540 2024-01-11T08:28:29.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.016+0000 7efe25d9a700 1 -- 172.21.15.27:0/3002907968 >> v1:172.21.15.77:6789/0 conn(0x7efe20069540 legacy=0x7efe200c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:29.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.016+0000 7efe25d9a700 1 -- 172.21.15.27:0/3002907968 >> v1:172.21.15.27:6789/0 conn(0x7efe20157410 legacy=0x7efe200ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:29.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.016+0000 7efe25d9a700 1 -- 172.21.15.27:0/3002907968 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2006a270 con 0x7efe201577e0 2024-01-11T08:28:29.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.016+0000 7efe25d9a700 1 --2- 172.21.15.27:0/3002907968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe201577e0 0x7efe200ae000 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7efe100098e0 tx=0x7efe100091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8617a18ada80e69f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:29.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.016+0000 7efe25599700 1 -- 172.21.15.27:0/3002907968 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe10011630 con 0x7efe201577e0 2024-01-11T08:28:29.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.016+0000 7efe25599700 1 -- 172.21.15.27:0/3002907968 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe10011790 con 0x7efe201577e0 2024-01-11T08:28:29.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.016+0000 7efe25599700 1 -- 172.21.15.27:0/3002907968 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe10011b30 con 0x7efe201577e0 2024-01-11T08:28:29.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.017+0000 7efe287ff700 1 -- 172.21.15.27:0/3002907968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe201577e0 msgr2=0x7efe200ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:29.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.017+0000 7efe287ff700 1 --2- 172.21.15.27:0/3002907968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe201577e0 0x7efe200ae000 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7efe100098e0 tx=0x7efe100091a0 comp rx=0 tx=0).stop 2024-01-11T08:28:29.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.017+0000 7efe287ff700 1 -- 172.21.15.27:0/3002907968 shutdown_connections 2024-01-11T08:28:29.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.017+0000 7efe287ff700 1 --2- 172.21.15.27:0/3002907968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe201577e0 0x7efe200ae000 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:29.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.017+0000 7efe287ff700 1 -- 172.21.15.27:0/3002907968 >> 172.21.15.27:0/3002907968 conn(0x7efe200680a0 msgr2=0x7efe200cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:29.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.017+0000 7efe287ff700 1 -- 172.21.15.27:0/3002907968 shutdown_connections 2024-01-11T08:28:29.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.017+0000 7efe287ff700 1 -- 172.21.15.27:0/3002907968 wait complete. 2024-01-11T08:28:29.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.019+0000 7efe287ff700 1 Processor -- start 2024-01-11T08:28:29.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.019+0000 7efe287ff700 1 -- start start 2024-01-11T08:28:29.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.020+0000 7efe287ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe20069540 0x7efe201696d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:29.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.020+0000 7efe287ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe20157410 0x7efe200c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:29.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.020+0000 7efe287ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe201577e0 0x7efe200c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:29.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.020+0000 7efe287ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe20152a60 con 0x7efe20157410 2024-01-11T08:28:29.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.020+0000 7efe287ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe20151e50 con 0x7efe201577e0 2024-01-11T08:28:29.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.020+0000 7efe287ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe20152130 con 0x7efe20069540 2024-01-11T08:28:29.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.020+0000 7efe25d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe20157410 0x7efe200c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:29.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.020+0000 7efe25d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe20157410 0x7efe200c3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41512/0 (socket says 172.21.15.27:41512) 2024-01-11T08:28:29.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.020+0000 7efe25d9a700 1 -- 172.21.15.27:0/2419396543 learned_addr learned my addr 172.21.15.27:0/2419396543 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:29.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.020+0000 7efe2659b700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe20069540 0x7efe201696d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:29.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.020+0000 7efe26d9c700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe201577e0 0x7efe200c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:29.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.020+0000 7efe25d9a700 1 -- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe20069540 msgr2=0x7efe201696d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:29.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.020+0000 7efe25d9a700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe20069540 0x7efe201696d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:29.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.020+0000 7efe25d9a700 1 -- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe201577e0 msgr2=0x7efe200c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:29.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.020+0000 7efe25d9a700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe201577e0 0x7efe200c41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:29.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.020+0000 7efe25d9a700 1 -- 172.21.15.27:0/2419396543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe10009420 con 0x7efe20157410 2024-01-11T08:28:29.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.021+0000 7efe25d9a700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe20157410 0x7efe200c3c80 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7efe10013000 tx=0x7efe10017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:29.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.021+0000 7efe167fc700 1 -- 172.21.15.27:0/2419396543 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1001f070 con 0x7efe20157410 2024-01-11T08:28:29.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.021+0000 7efe167fc700 1 -- 172.21.15.27:0/2419396543 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe10000bc0 con 0x7efe20157410 2024-01-11T08:28:29.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.021+0000 7efe287ff700 1 -- 172.21.15.27:0/2419396543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe20152410 con 0x7efe20157410 2024-01-11T08:28:29.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.021+0000 7efe167fc700 1 -- 172.21.15.27:0/2419396543 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe10025420 con 0x7efe20157410 2024-01-11T08:28:29.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.021+0000 7efe287ff700 1 -- 172.21.15.27:0/2419396543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe201518b0 con 0x7efe20157410 2024-01-11T08:28:29.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.022+0000 7efe287ff700 1 -- 172.21.15.27:0/2419396543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2006d190 con 0x7efe20157410 2024-01-11T08:28:29.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.023+0000 7efe167fc700 1 -- 172.21.15.27:0/2419396543 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 58) v1 ==== 84759+0+0 (secure 0 0 0) 0x7efe10023070 con 0x7efe20157410 2024-01-11T08:28:29.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.025+0000 7efe167fc700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe04066f80 0x7efe04069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:29.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.025+0000 7efe167fc700 1 -- 172.21.15.27:0/2419396543 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5733+0+0 (secure 0 0 0) 0x7efe10085890 con 0x7efe20157410 2024-01-11T08:28:29.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.025+0000 7efe2659b700 1 -- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe04066f80 msgr2=0x7efe04069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:29.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.025+0000 7efe2659b700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe04066f80 0x7efe04069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:28:29.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.029+0000 7efe167fc700 1 -- 172.21.15.27:0/2419396543 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efe10052a40 con 0x7efe20157410 2024-01-11T08:28:29.096 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:29.097+0000 7ff57e8921c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:28:29.172 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:29.173+0000 7fd3e3c411c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:28:29.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.226+0000 7efe2659b700 1 -- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe04066f80 msgr2=0x7efe04069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:29.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.226+0000 7efe2659b700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe04066f80 0x7efe04069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:28:29.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.259+0000 7efe287ff700 1 -- 172.21.15.27:0/2419396543 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7efe2004c3c0 con 0x7efe04066f80 2024-01-11T08:28:29.278 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:29.278+0000 7ff57e8921c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:28:29.586 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:29.586+0000 7fd3e3c411c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:28:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.627+0000 7efe2659b700 1 -- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe04066f80 msgr2=0x7efe04069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:29.627+0000 7efe2659b700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe04066f80 0x7efe04069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-11T08:28:29.713 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:29.714+0000 7ff57e8921c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:28:30.363 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:30.363+0000 7fd3e3c411c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:28:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:30.428+0000 7efe2659b700 1 -- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe04066f80 msgr2=0x7efe04069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:30.428+0000 7efe2659b700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe04066f80 0x7efe04069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-11T08:28:30.451 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:30.451+0000 7fd3e3c411c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:28:30.536 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:30.537+0000 7ff57e8921c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:28:30.606 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:30.606+0000 7fd3e3c411c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:28:30.629 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:30.630+0000 7ff57e8921c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:28:30.692 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:30.692+0000 7fd3e3c411c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:28:30.794 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:30.795+0000 7ff57e8921c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:28:30.887 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:30.888+0000 7ff57e8921c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:28:30.908 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:30.908+0000 7fd3e3c411c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:28:31.006 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:31.006+0000 7fd3e3c411c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:28:31.101 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:31.101+0000 7fd3e3c411c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:28:31.116 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:31.117+0000 7ff57e8921c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:28:31.196 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:31.196+0000 7fd3e3c411c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:28:31.217 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:31.218+0000 7ff57e8921c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:28:31.319 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:31.320+0000 7ff57e8921c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:28:31.386 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:31.386+0000 7fd3e3c411c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:28:31.421 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:31.420+0000 7ff57e8921c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:28:31.486 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:31.485+0000 7fd3e3c411c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:28:31.619 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:31.620+0000 7ff57e8921c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:28:31.723 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:31.724+0000 7ff57e8921c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:28:31.826 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:31.827+0000 7fd3e3c411c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:28:31.916 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:31.916+0000 7fd3e3c411c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:28:32.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:32.030+0000 7efe2659b700 1 -- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe04066f80 msgr2=0x7efe04069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:28:32.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:32.030+0000 7efe2659b700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe04066f80 0x7efe04069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-11T08:28:32.082 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:32.084+0000 7ff57e8921c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:28:32.177 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:32.178+0000 7ff57e8921c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:28:32.456 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:32.456+0000 7fd3e3c411c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:28:32.643 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:32.643+0000 7fd3e3c411c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:28:32.752 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:32.753+0000 7ff57e8921c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:28:32.949 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:32.950+0000 7ff57e8921c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:28:33.022 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:33.023+0000 7fd3e3c411c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:28:33.349 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:33.350+0000 7ff57e8921c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:28:33.440 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:33.440+0000 7fd3e3c411c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:28:33.621 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:33.621+0000 7fd3e3c411c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:28:33.800 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:33.801+0000 7ff57e8921c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:28:33.807 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:33.807+0000 7fd3e3c411c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:28:33.917 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:33.917+0000 7fd3e3c411c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:28:33.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:33.931+0000 7efe167fc700 1 -- 172.21.15.27:0/2419396543 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 59) v1 ==== 84470+0+0 (secure 0 0 0) 0x7efe10052e10 con 0x7efe20157410 2024-01-11T08:28:33.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:33.931+0000 7efe167fc700 1 -- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe04066f80 msgr2=0x7efe04069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:28:33.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:33.931+0000 7efe167fc700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe04066f80 0x7efe04069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:34.005 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:34.006+0000 7ff57e8921c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:28:34.050 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:34.051+0000 7fd35ba7e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:34.051 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:34.051+0000 7fd35ba7e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:34.051 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:34.051+0000 7fd35ba7e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:34.051 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:34.051+0000 7fd35ba7e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:34.051 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:34.051+0000 7fd35ba7e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:34.052 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:34.052+0000 7fd358a78700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:34.052 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:34.052+0000 7fd358a78700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:34.052 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:34.052+0000 7fd358a78700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:34.052 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:34.052+0000 7fd358a78700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:34.052 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:28:34.052+0000 7fd358a78700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:28:34.209 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:34.210+0000 7ff57e8921c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:28:34.324 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:28:34.325+0000 7ff57e8921c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:28:34.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:34.932+0000 7efe167fc700 1 -- 172.21.15.27:0/2419396543 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 60) v1 ==== 84597+0+0 (secure 0 0 0) 0x7efe1005d210 con 0x7efe20157410 2024-01-11T08:28:34.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:34.932+0000 7efe167fc700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe04066f80 0x7efe040693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:34.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:34.932+0000 7efe167fc700 1 -- 172.21.15.27:0/2419396543 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7efe2004c3c0 con 0x7efe04066f80 2024-01-11T08:28:34.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:34.935+0000 7efe2659b700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe04066f80 0x7efe040693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:34.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:34.940+0000 7efe2659b700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe04066f80 0x7efe040693b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efe1c0079e0 tx=0x7efe1c009040 comp rx=0 tx=0).ready entity=mgr.7498 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:35.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.011+0000 7efe167fc700 1 -- 172.21.15.27:0/2419396543 <== mgr.7498 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efe2004c3c0 con 0x7efe04066f80 2024-01-11T08:28:35.011 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path snap_test_dir1 2024-01-11T08:28:35.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.014+0000 7efe287ff700 1 -- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe04066f80 msgr2=0x7efe040693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:35.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.014+0000 7efe287ff700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe04066f80 0x7efe040693b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efe1c0079e0 tx=0x7efe1c009040 comp rx=0 tx=0).stop 2024-01-11T08:28:35.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.014+0000 7efe287ff700 1 -- 172.21.15.27:0/2419396543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe20157410 msgr2=0x7efe200c3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:35.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.014+0000 7efe287ff700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe20157410 0x7efe200c3c80 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7efe10013000 tx=0x7efe10017000 comp rx=0 tx=0).stop 2024-01-11T08:28:35.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.014+0000 7efe287ff700 1 -- 172.21.15.27:0/2419396543 shutdown_connections 2024-01-11T08:28:35.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.014+0000 7efe287ff700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe04066f80 0x7efe040693b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:35.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.014+0000 7efe287ff700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe201577e0 0x7efe200c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:35.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.014+0000 7efe287ff700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe20069540 0x7efe201696d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:35.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.014+0000 7efe287ff700 1 --2- 172.21.15.27:0/2419396543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe20157410 0x7efe200c3c80 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:35.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.014+0000 7efe287ff700 1 -- 172.21.15.27:0/2419396543 >> 172.21.15.27:0/2419396543 conn(0x7efe200680a0 msgr2=0x7efe20152fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:35.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.015+0000 7efe287ff700 1 -- 172.21.15.27:0/2419396543 shutdown_connections 2024-01-11T08:28:35.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.015+0000 7efe287ff700 1 -- 172.21.15.27:0/2419396543 wait complete. 2024-01-11T08:28:35.034 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-01-11T08:28:35.034 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-01-11-08_29_00 in ~25s... 2024-01-11T08:28:35.034 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2024-01-11T08:28:35.034 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-01-11T08:28:35.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.238+0000 7f55931ff700 1 Processor -- start 2024-01-11T08:28:35.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.239+0000 7f55931ff700 1 -- start start 2024-01-11T08:28:35.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.240+0000 7f55931ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f558c157410 0x7f558c0ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:35.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.240+0000 7f55931ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f558c1577e0 0x7f558c0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:35.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.240+0000 7f55931ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f558c069540 0x7f558c0ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:35.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.240+0000 7f55931ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f558c0699d0 con 0x7f558c157410 2024-01-11T08:28:35.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.240+0000 7f55931ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f558c069cb0 con 0x7f558c1577e0 2024-01-11T08:28:35.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.240+0000 7f55931ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558c069f90 con 0x7f558c069540 2024-01-11T08:28:35.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.240+0000 7f5590f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f558c157410 0x7f558c0ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:35.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.240+0000 7f5590f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f558c157410 0x7f558c0ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41984/0 (socket says 172.21.15.27:41984) 2024-01-11T08:28:35.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.240+0000 7f5590f9b700 1 -- 172.21.15.27:0/3379997963 learned_addr learned my addr 172.21.15.27:0/3379997963 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:35.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.240+0000 7f558bfff700 1 --2- 172.21.15.27:0/3379997963 >> v2:172.21.15.77:3300/0 conn(0x7f558c1577e0 0x7f558c0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.240+0000 7f559179c700 1 --2- 172.21.15.27:0/3379997963 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f558c069540 0x7f558c0ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.241+0000 7f5590f9b700 1 -- 172.21.15.27:0/3379997963 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f558c069540 msgr2=0x7f558c0ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.241+0000 7f5590f9b700 1 --2- 172.21.15.27:0/3379997963 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f558c069540 0x7f558c0ae540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.241+0000 7f5590f9b700 1 -- 172.21.15.27:0/3379997963 >> v2:172.21.15.77:3300/0 conn(0x7f558c1577e0 msgr2=0x7f558c0ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:28:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.241+0000 7f5590f9b700 1 --2- 172.21.15.27:0/3379997963 >> v2:172.21.15.77:3300/0 conn(0x7f558c1577e0 0x7f558c0ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.241+0000 7f5590f9b700 1 -- 172.21.15.27:0/3379997963 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558c06a270 con 0x7f558c157410 2024-01-11T08:28:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.241+0000 7f5590f9b700 1 --2- 172.21.15.27:0/3379997963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558c157410 0x7f558c0ccae0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f558c06d2a0 tx=0x7f5580009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe617b70ea5c58bb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:35.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.241+0000 7f558b7fe700 1 -- 172.21.15.27:0/3379997963 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5580005c10 con 0x7f558c157410 2024-01-11T08:28:35.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.241+0000 7f558b7fe700 1 -- 172.21.15.27:0/3379997963 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5580005d70 con 0x7f558c157410 2024-01-11T08:28:35.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.241+0000 7f558b7fe700 1 -- 172.21.15.27:0/3379997963 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5580006260 con 0x7f558c157410 2024-01-11T08:28:35.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.242+0000 7f55931ff700 1 -- 172.21.15.27:0/3379997963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558c157410 msgr2=0x7f558c0ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:35.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.242+0000 7f55931ff700 1 --2- 172.21.15.27:0/3379997963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558c157410 0x7f558c0ccae0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f558c06d2a0 tx=0x7f5580009220 comp rx=0 tx=0).stop 2024-01-11T08:28:35.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.242+0000 7f55931ff700 1 -- 172.21.15.27:0/3379997963 shutdown_connections 2024-01-11T08:28:35.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.242+0000 7f55931ff700 1 --2- 172.21.15.27:0/3379997963 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f558c069540 0x7f558c0ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:35.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.242+0000 7f55931ff700 1 --2- 172.21.15.27:0/3379997963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558c157410 0x7f558c0ccae0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:35.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.242+0000 7f55931ff700 1 --2- 172.21.15.27:0/3379997963 >> v2:172.21.15.77:3300/0 conn(0x7f558c1577e0 0x7f558c0ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:35.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.242+0000 7f55931ff700 1 -- 172.21.15.27:0/3379997963 >> 172.21.15.27:0/3379997963 conn(0x7f558c0680a0 msgr2=0x7f558c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:35.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.242+0000 7f55931ff700 1 -- 172.21.15.27:0/3379997963 shutdown_connections 2024-01-11T08:28:35.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.242+0000 7f55931ff700 1 -- 172.21.15.27:0/3379997963 wait complete. 2024-01-11T08:28:35.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.244+0000 7f55931ff700 1 Processor -- start 2024-01-11T08:28:35.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.244+0000 7f55931ff700 1 -- start start 2024-01-11T08:28:35.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.246+0000 7f55931ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f558c069540 0x7f558c0ca230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:35.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.246+0000 7f55931ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558c157410 0x7f558c155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:35.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.246+0000 7f55931ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f558c1577e0 0x7f558c144750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:35.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.246+0000 7f55931ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c0cac80 con 0x7f558c157410 2024-01-11T08:28:35.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.246+0000 7f55931ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c156c40 con 0x7f558c1577e0 2024-01-11T08:28:35.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.246+0000 7f55931ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558c0ca950 con 0x7f558c069540 2024-01-11T08:28:35.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.246+0000 7f558bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558c157410 0x7f558c155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:35.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.246+0000 7f558bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558c157410 0x7f558c155fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41992/0 (socket says 172.21.15.27:41992) 2024-01-11T08:28:35.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.246+0000 7f558bfff700 1 -- 172.21.15.27:0/3402376534 learned_addr learned my addr 172.21.15.27:0/3402376534 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:28:35.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.246+0000 7f5590f9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f558c069540 0x7f558c0ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:35.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.246+0000 7f559179c700 1 --2- 172.21.15.27:0/3402376534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f558c1577e0 0x7f558c144750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:35.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.246+0000 7f558bfff700 1 -- 172.21.15.27:0/3402376534 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f558c069540 msgr2=0x7f558c0ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:35.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.246+0000 7f558bfff700 1 --2- 172.21.15.27:0/3402376534 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f558c069540 0x7f558c0ca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:35.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.246+0000 7f558bfff700 1 -- 172.21.15.27:0/3402376534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f558c1577e0 msgr2=0x7f558c144750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:35.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.246+0000 7f558bfff700 1 --2- 172.21.15.27:0/3402376534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f558c1577e0 0x7f558c144750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:35.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.246+0000 7f558bfff700 1 -- 172.21.15.27:0/3402376534 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5580009480 con 0x7f558c157410 2024-01-11T08:28:35.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.247+0000 7f558bfff700 1 --2- 172.21.15.27:0/3402376534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558c157410 0x7f558c155fe0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f557800e900 tx=0x7f55780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:35.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.247+0000 7f5588ff9700 1 -- 172.21.15.27:0/3402376534 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5578013680 con 0x7f558c157410 2024-01-11T08:28:35.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.247+0000 7f5588ff9700 1 -- 172.21.15.27:0/3402376534 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55780137e0 con 0x7f558c157410 2024-01-11T08:28:35.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.247+0000 7f5588ff9700 1 -- 172.21.15.27:0/3402376534 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5578013b80 con 0x7f558c157410 2024-01-11T08:28:35.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.247+0000 7f55931ff700 1 -- 172.21.15.27:0/3402376534 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558c1539f0 con 0x7f558c157410 2024-01-11T08:28:35.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.247+0000 7f55931ff700 1 -- 172.21.15.27:0/3402376534 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558c156960 con 0x7f558c157410 2024-01-11T08:28:35.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.248+0000 7f55931ff700 1 -- 172.21.15.27:0/3402376534 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f558c04c3c0 con 0x7f558c157410 2024-01-11T08:28:35.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.249+0000 7f5588ff9700 1 -- 172.21.15.27:0/3402376534 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 60) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f55780123d0 con 0x7f558c157410 2024-01-11T08:28:35.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.250+0000 7f5588ff9700 1 --2- 172.21.15.27:0/3402376534 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5570066dd0 0x7f5570069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:28:35.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.250+0000 7f5588ff9700 1 -- 172.21.15.27:0/3402376534 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5991+0+0 (secure 0 0 0) 0x7f5578084480 con 0x7f558c157410 2024-01-11T08:28:35.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.250+0000 7f5590f9b700 1 --2- 172.21.15.27:0/3402376534 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5570066dd0 0x7f5570069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:28:35.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.251+0000 7f5590f9b700 1 --2- 172.21.15.27:0/3402376534 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5570066dd0 0x7f5570069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5580009260 tx=0x7f5580016000 comp rx=0 tx=0).ready entity=mgr.7498 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:28:35.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.254+0000 7f5588ff9700 1 -- 172.21.15.27:0/3402376534 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f55780517b0 con 0x7f558c157410 2024-01-11T08:28:35.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.482+0000 7f55931ff700 1 -- 172.21.15.27:0/3402376534 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f558c0cadc0 con 0x7f5570066dd0 2024-01-11T08:28:35.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.494+0000 7f5588ff9700 1 -- 172.21.15.27:0/3402376534 <== mgr.7498 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f558c0cadc0 con 0x7f5570066dd0 2024-01-11T08:28:35.494 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:28:35.494 INFO:teuthology.orchestra.run.smithi027.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2024-01-11T08:28:35.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.497+0000 7f55931ff700 1 -- 172.21.15.27:0/3402376534 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5570066dd0 msgr2=0x7f5570069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:35.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.497+0000 7f55931ff700 1 --2- 172.21.15.27:0/3402376534 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5570066dd0 0x7f5570069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5580009260 tx=0x7f5580016000 comp rx=0 tx=0).stop 2024-01-11T08:28:35.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.497+0000 7f55931ff700 1 -- 172.21.15.27:0/3402376534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558c157410 msgr2=0x7f558c155fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:28:35.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.497+0000 7f55931ff700 1 --2- 172.21.15.27:0/3402376534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558c157410 0x7f558c155fe0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f557800e900 tx=0x7f55780091a0 comp rx=0 tx=0).stop 2024-01-11T08:28:35.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.497+0000 7f55931ff700 1 -- 172.21.15.27:0/3402376534 shutdown_connections 2024-01-11T08:28:35.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.497+0000 7f55931ff700 1 --2- 172.21.15.27:0/3402376534 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5570066dd0 0x7f5570069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:35.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.497+0000 7f55931ff700 1 --2- 172.21.15.27:0/3402376534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f558c1577e0 0x7f558c144750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:35.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.497+0000 7f55931ff700 1 --2- 172.21.15.27:0/3402376534 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f558c069540 0x7f558c0ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:35.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.497+0000 7f55931ff700 1 --2- 172.21.15.27:0/3402376534 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f558c157410 0x7f558c155fe0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:28:35.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.497+0000 7f55931ff700 1 -- 172.21.15.27:0/3402376534 >> 172.21.15.27:0/3402376534 conn(0x7f558c0680a0 msgr2=0x7f558c152d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:28:35.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.497+0000 7f55931ff700 1 -- 172.21.15.27:0/3402376534 shutdown_connections 2024-01-11T08:28:35.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:28:35.497+0000 7f55931ff700 1 -- 172.21.15.27:0/3402376534 wait complete. 2024-01-11T08:28:35.514 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2024-01-11T08:28:35.515 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{}]} 2024-01-11T08:28:35.515 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:35.515 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:35.573 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:35.573 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:35.573 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:36.574 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:36.574 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:36.631 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:36.632 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:36.632 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:37.632 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:37.633 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:37.688 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:37.688 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:37.688 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:38.688 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:38.689 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:38.744 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:38.744 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:38.744 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:39.745 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:39.745 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:39.803 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:39.803 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:39.803 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:40.804 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:40.804 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:40.860 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:40.860 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:40.860 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:41.861 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:41.861 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:41.917 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:41.917 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:41.917 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:42.919 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:42.919 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:42.974 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:42.974 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:42.975 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:43.975 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:43.975 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:44.030 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:44.030 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:44.030 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:45.031 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:45.031 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:45.087 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:45.087 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:45.087 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:46.088 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:46.088 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:46.142 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:46.143 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:46.143 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:47.143 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:47.144 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:47.198 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:47.198 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:47.199 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:48.200 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:48.200 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:48.255 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:48.255 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:48.255 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:49.256 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:49.256 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:49.312 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:49.312 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:49.312 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:50.313 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:50.313 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:50.368 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:50.368 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:50.369 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:51.370 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:51.370 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:51.425 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:51.426 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:51.426 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:52.426 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:52.426 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:52.481 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:52.482 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:52.482 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:53.275 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:28:53.281 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:28:53.483 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:53.483 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:53.536 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:53.536 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:53.536 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:54.537 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:54.538 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:54.593 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:54.593 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:54.593 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:55.594 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:55.594 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:55.649 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:55.649 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:55.649 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:56.651 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:56.651 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:56.707 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:56.707 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:56.707 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:57.708 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:57.708 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:57.763 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:57.764 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:57.764 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:58.764 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:58.764 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:58.820 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:58.820 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:58.821 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:28:59.821 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:28:59.822 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:28:59.877 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T08:28:59.877 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T08:28:59.877 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:29:00.024 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:00.025+0000 7fd34ba5e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:00.024 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:00.025+0000 7fd34ba5e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:00.024 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:00.025+0000 7fd34ba5e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:00.024 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:00.025+0000 7fd34ba5e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:00.024 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:00.025+0000 7fd34ba5e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:00.878 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:00.878 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:29:00.932 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_29_00_UTC 2024-01-11T08:29:00.933 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-08_29_00_UTC'} 2024-01-11T08:29:00.933 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-01-11-08_29_00_UTC'} 2024-01-11T08:29:00.933 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T08:29:00.933 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-01-11-08_29_00_UTC'] 2024-01-11T08:29:00.933 DEBUG:tasks.cephfs.test_snap_schedules:exec=1704961715.0344892, now = 1704961740.9337947, timo = 25 2024-01-11T08:29:01.934 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-01-11T08:29:02.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.148+0000 7f23181ff700 1 Processor -- start 2024-01-11T08:29:02.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.149+0000 7f23181ff700 1 -- start start 2024-01-11T08:29:02.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.150+0000 7f23181ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2310143190 0x7f23100cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:02.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.150+0000 7f23181ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23100ccc70 con 0x7f2310142dc0 2024-01-11T08:29:02.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.150+0000 7f23181ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231015d320 con 0x7f23100cb4b0 2024-01-11T08:29:02.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.150+0000 7f23181ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f231015df30 con 0x7f2310143190 2024-01-11T08:29:02.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.150+0000 7f2315f9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2310142dc0 0x7f23100cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47264/0 (socket says 172.21.15.27:47264) 2024-01-11T08:29:02.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.150+0000 7f2315f9b700 1 -- 172.21.15.27:0/2216452111 learned_addr learned my addr 172.21.15.27:0/2216452111 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:02.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.151+0000 7f231579a700 1 --2- 172.21.15.27:0/2216452111 >> v2:172.21.15.77:3300/0 conn(0x7f2310143190 0x7f23100cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:02.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.151+0000 7f2314f99700 1 -- 172.21.15.27:0/2216452111 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183689405 0 0) 0x7f23100ccc70 con 0x7f2310142dc0 2024-01-11T08:29:02.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.151+0000 7f2314f99700 1 -- 172.21.15.27:0/2216452111 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22fc003570 con 0x7f2310142dc0 2024-01-11T08:29:02.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.151+0000 7f2314f99700 1 -- 172.21.15.27:0/2216452111 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443278831 0 0) 0x7f231015d320 con 0x7f23100cb4b0 2024-01-11T08:29:02.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.151+0000 7f2314f99700 1 -- 172.21.15.27:0/2216452111 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23100ccc70 con 0x7f23100cb4b0 2024-01-11T08:29:02.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.151+0000 7f231579a700 1 -- 172.21.15.27:0/2216452111 >> v1:172.21.15.77:6789/0 conn(0x7f23100cb4b0 legacy=0x7f23100cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:02.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.151+0000 7f231579a700 1 -- 172.21.15.27:0/2216452111 >> v1:172.21.15.27:6789/0 conn(0x7f2310142dc0 legacy=0x7f23100cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:02.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.151+0000 7f231579a700 1 -- 172.21.15.27:0/2216452111 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231015d590 con 0x7f2310143190 2024-01-11T08:29:02.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.151+0000 7f231579a700 1 --2- 172.21.15.27:0/2216452111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2310143190 0x7f23100cc020 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f23000098e0 tx=0x7f23000091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db7751327f0dc1b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:02.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.152+0000 7f2314f99700 1 -- 172.21.15.27:0/2216452111 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2300011630 con 0x7f2310143190 2024-01-11T08:29:02.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.152+0000 7f2314f99700 1 -- 172.21.15.27:0/2216452111 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2300011790 con 0x7f2310143190 2024-01-11T08:29:02.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.152+0000 7f2314f99700 1 -- 172.21.15.27:0/2216452111 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2300011b30 con 0x7f2310143190 2024-01-11T08:29:02.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.152+0000 7f23181ff700 1 -- 172.21.15.27:0/2216452111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2310143190 msgr2=0x7f23100cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:02.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.152+0000 7f23181ff700 1 --2- 172.21.15.27:0/2216452111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2310143190 0x7f23100cc020 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f23000098e0 tx=0x7f23000091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:02.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.152+0000 7f23181ff700 1 -- 172.21.15.27:0/2216452111 shutdown_connections 2024-01-11T08:29:02.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.152+0000 7f23181ff700 1 --2- 172.21.15.27:0/2216452111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2310143190 0x7f23100cc020 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:02.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.152+0000 7f23181ff700 1 -- 172.21.15.27:0/2216452111 >> 172.21.15.27:0/2216452111 conn(0x7f2310063ca0 msgr2=0x7f231006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:02.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.152+0000 7f23181ff700 1 -- 172.21.15.27:0/2216452111 shutdown_connections 2024-01-11T08:29:02.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.152+0000 7f23181ff700 1 -- 172.21.15.27:0/2216452111 wait complete. 2024-01-11T08:29:02.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.154+0000 7f23181ff700 1 Processor -- start 2024-01-11T08:29:02.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.154+0000 7f23181ff700 1 -- start start 2024-01-11T08:29:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.155+0000 7f23181ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23100cb4b0 0x7f23100ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.155+0000 7f23181ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2310142dc0 0x7f23100c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.155+0000 7f23181ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2310143190 0x7f231016d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.155+0000 7f23181ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23100ca8d0 con 0x7f2310142dc0 2024-01-11T08:29:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.155+0000 7f23181ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23100c8ac0 con 0x7f2310143190 2024-01-11T08:29:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.155+0000 7f23181ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23100c5800 con 0x7f23100cb4b0 2024-01-11T08:29:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.155+0000 7f231579a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2310142dc0 0x7f23100c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.155+0000 7f231579a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2310142dc0 0x7f23100c8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33606/0 (socket says 172.21.15.27:33606) 2024-01-11T08:29:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.155+0000 7f231579a700 1 -- 172.21.15.27:0/2041840148 learned_addr learned my addr 172.21.15.27:0/2041840148 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.156+0000 7f2315f9b700 1 --2- 172.21.15.27:0/2041840148 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23100cb4b0 0x7f23100ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.156+0000 7f231679c700 1 --2- 172.21.15.27:0/2041840148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2310143190 0x7f231016d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:02.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.156+0000 7f231579a700 1 -- 172.21.15.27:0/2041840148 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23100cb4b0 msgr2=0x7f23100ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:02.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.156+0000 7f231579a700 1 --2- 172.21.15.27:0/2041840148 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23100cb4b0 0x7f23100ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:02.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.156+0000 7f231579a700 1 -- 172.21.15.27:0/2041840148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2310143190 msgr2=0x7f231016d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:02.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.156+0000 7f231579a700 1 --2- 172.21.15.27:0/2041840148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2310143190 0x7f231016d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:02.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.156+0000 7f231579a700 1 -- 172.21.15.27:0/2041840148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2300009420 con 0x7f2310142dc0 2024-01-11T08:29:02.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.156+0000 7f231679c700 1 --2- 172.21.15.27:0/2041840148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2310143190 0x7f231016d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:29:02.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.156+0000 7f2315f9b700 1 --2- 172.21.15.27:0/2041840148 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23100cb4b0 0x7f23100ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:29:02.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.156+0000 7f231579a700 1 --2- 172.21.15.27:0/2041840148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2310142dc0 0x7f23100c8580 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f2310064a70 tx=0x7f2300017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:02.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.156+0000 7f2305ffb700 1 -- 172.21.15.27:0/2041840148 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230001f070 con 0x7f2310142dc0 2024-01-11T08:29:02.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.156+0000 7f2305ffb700 1 -- 172.21.15.27:0/2041840148 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2300000c90 con 0x7f2310142dc0 2024-01-11T08:29:02.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.156+0000 7f23181ff700 1 -- 172.21.15.27:0/2041840148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23100c8cd0 con 0x7f2310142dc0 2024-01-11T08:29:02.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.157+0000 7f2305ffb700 1 -- 172.21.15.27:0/2041840148 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2300025420 con 0x7f2310142dc0 2024-01-11T08:29:02.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.157+0000 7f23181ff700 1 -- 172.21.15.27:0/2041840148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23100c9290 con 0x7f2310142dc0 2024-01-11T08:29:02.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.158+0000 7f23181ff700 1 -- 172.21.15.27:0/2041840148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23100c5990 con 0x7f2310142dc0 2024-01-11T08:29:02.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.159+0000 7f2305ffb700 1 -- 172.21.15.27:0/2041840148 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 62) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f2300023070 con 0x7f2310142dc0 2024-01-11T08:29:02.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.159+0000 7f2305ffb700 1 --2- 172.21.15.27:0/2041840148 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f22f4067130 0x7f22f40695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:02.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.160+0000 7f2305ffb700 1 -- 172.21.15.27:0/2041840148 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5991+0+0 (secure 0 0 0) 0x7f2300085d20 con 0x7f2310142dc0 2024-01-11T08:29:02.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.160+0000 7f2315f9b700 1 --2- 172.21.15.27:0/2041840148 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f22f4067130 0x7f22f40695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:02.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.161+0000 7f2315f9b700 1 --2- 172.21.15.27:0/2041840148 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f22f4067130 0x7f22f40695e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f230c007fd0 tx=0x7f230c009040 comp rx=0 tx=0).ready entity=mgr.7498 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:02.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.165+0000 7f2305ffb700 1 -- 172.21.15.27:0/2041840148 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2300052b50 con 0x7f2310142dc0 2024-01-11T08:29:02.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.392+0000 7f23181ff700 1 -- 172.21.15.27:0/2041840148 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f231004c3c0 con 0x7f22f4067130 2024-01-11T08:29:02.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.467+0000 7f2305ffb700 1 -- 172.21.15.27:0/2041840148 <== mgr.7498 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f231004c3c0 con 0x7f22f4067130 2024-01-11T08:29:02.466 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule removed for path snap_test_dir1 2024-01-11T08:29:02.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.470+0000 7f23181ff700 1 -- 172.21.15.27:0/2041840148 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f22f4067130 msgr2=0x7f22f40695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:02.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.470+0000 7f23181ff700 1 --2- 172.21.15.27:0/2041840148 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f22f4067130 0x7f22f40695e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f230c007fd0 tx=0x7f230c009040 comp rx=0 tx=0).stop 2024-01-11T08:29:02.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.470+0000 7f23181ff700 1 -- 172.21.15.27:0/2041840148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2310142dc0 msgr2=0x7f23100c8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:02.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.470+0000 7f23181ff700 1 --2- 172.21.15.27:0/2041840148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2310142dc0 0x7f23100c8580 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f2310064a70 tx=0x7f2300017000 comp rx=0 tx=0).stop 2024-01-11T08:29:02.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.470+0000 7f23181ff700 1 -- 172.21.15.27:0/2041840148 shutdown_connections 2024-01-11T08:29:02.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.470+0000 7f23181ff700 1 --2- 172.21.15.27:0/2041840148 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f22f4067130 0x7f22f40695e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:02.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.470+0000 7f23181ff700 1 --2- 172.21.15.27:0/2041840148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2310143190 0x7f231016d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:02.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.470+0000 7f23181ff700 1 --2- 172.21.15.27:0/2041840148 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23100cb4b0 0x7f23100ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:02.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.470+0000 7f23181ff700 1 --2- 172.21.15.27:0/2041840148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2310142dc0 0x7f23100c8580 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:02.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.470+0000 7f23181ff700 1 -- 172.21.15.27:0/2041840148 >> 172.21.15.27:0/2041840148 conn(0x7f2310063ca0 msgr2=0x7f23100c4bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:02.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.470+0000 7f23181ff700 1 -- 172.21.15.27:0/2041840148 shutdown_connections 2024-01-11T08:29:02.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.470+0000 7f23181ff700 1 -- 172.21.15.27:0/2041840148 wait complete. 2024-01-11T08:29:02.487 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-01-11T08:29:02.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:02.487 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T08:29:02.535 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_29_00_UTC 2024-01-11T08:29:02.536 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_29_00_UTC 2024-01-11T08:29:02.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:02.537 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_29_00_UTC) 2024-01-11T08:29:02.590 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:02.590 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-01-11T08:29:02.641 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-11T08:29:02.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.845+0000 7fd2ff3ff700 1 Processor -- start 2024-01-11T08:29:02.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.845+0000 7fd2ff3ff700 1 -- start start 2024-01-11T08:29:02.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.846+0000 7fd2ff3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd2f813fbb0 0x7fd2f80ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:02.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.846+0000 7fd2ff3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2f813ff80 0x7fd2f815a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:02.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.846+0000 7fd2ff3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd2f80cace0 0x7fd2f815a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:02.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.846+0000 7fd2ff3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2f80c6b30 con 0x7fd2f813fbb0 2024-01-11T08:29:02.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.846+0000 7fd2ff3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2f80c6800 con 0x7fd2f80cace0 2024-01-11T08:29:02.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.846+0000 7fd2ff3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f80cb240 con 0x7fd2f813ff80 2024-01-11T08:29:02.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.847+0000 7fd2fd19b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd2f813fbb0 0x7fd2f80ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:02.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.847+0000 7fd2fd19b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd2f813fbb0 0x7fd2f80ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33620/0 (socket says 172.21.15.27:33620) 2024-01-11T08:29:02.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.847+0000 7fd2fd19b700 1 -- 172.21.15.27:0/2273296690 learned_addr learned my addr 172.21.15.27:0/2273296690 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:02.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.847+0000 7fd2fd19b700 1 -- 172.21.15.27:0/2273296690 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2f813ff80 msgr2=0x7fd2f815a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:02.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.847+0000 7fd2fd19b700 1 --2- 172.21.15.27:0/2273296690 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2f813ff80 0x7fd2f815a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:02.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.847+0000 7fd2fd99c700 1 --2- 172.21.15.27:0/2273296690 >> v2:172.21.15.77:3300/0 conn(0x7fd2f80cace0 0x7fd2f815a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:02.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.847+0000 7fd2fd19b700 1 -- 172.21.15.27:0/2273296690 >> v2:172.21.15.77:3300/0 conn(0x7fd2f80cace0 msgr2=0x7fd2f815a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:02.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.847+0000 7fd2fd19b700 1 --2- 172.21.15.27:0/2273296690 >> v2:172.21.15.77:3300/0 conn(0x7fd2f80cace0 0x7fd2f815a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.847+0000 7fd2fd19b700 1 -- 172.21.15.27:0/2273296690 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f80cb520 con 0x7fd2f813fbb0 2024-01-11T08:29:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.847+0000 7fd2fd99c700 1 --2- 172.21.15.27:0/2273296690 >> v2:172.21.15.77:3300/0 conn(0x7fd2f80cace0 0x7fd2f815a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:29:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.847+0000 7fd2fd19b700 1 --2- 172.21.15.27:0/2273296690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f813fbb0 0x7fd2f80ca7a0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80c8be0 tx=0x7fd2f4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cedc967a2cc751f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.848+0000 7fd2effff700 1 -- 172.21.15.27:0/2273296690 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4005c10 con 0x7fd2f813fbb0 2024-01-11T08:29:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.848+0000 7fd2effff700 1 -- 172.21.15.27:0/2273296690 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f4005d70 con 0x7fd2f813fbb0 2024-01-11T08:29:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.848+0000 7fd2effff700 1 -- 172.21.15.27:0/2273296690 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f400b880 con 0x7fd2f813fbb0 2024-01-11T08:29:02.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.848+0000 7fd2ff3ff700 1 -- 172.21.15.27:0/2273296690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f813fbb0 msgr2=0x7fd2f80ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:02.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.848+0000 7fd2ff3ff700 1 --2- 172.21.15.27:0/2273296690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f813fbb0 0x7fd2f80ca7a0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80c8be0 tx=0x7fd2f4009220 comp rx=0 tx=0).stop 2024-01-11T08:29:02.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.848+0000 7fd2ff3ff700 1 -- 172.21.15.27:0/2273296690 shutdown_connections 2024-01-11T08:29:02.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.848+0000 7fd2ff3ff700 1 --2- 172.21.15.27:0/2273296690 >> v2:172.21.15.77:3300/0 conn(0x7fd2f80cace0 0x7fd2f815a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:02.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.848+0000 7fd2ff3ff700 1 --2- 172.21.15.27:0/2273296690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f813fbb0 0x7fd2f80ca7a0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:02.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.848+0000 7fd2ff3ff700 1 --2- 172.21.15.27:0/2273296690 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2f813ff80 0x7fd2f815a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:02.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.848+0000 7fd2ff3ff700 1 -- 172.21.15.27:0/2273296690 >> 172.21.15.27:0/2273296690 conn(0x7fd2f80680a0 msgr2=0x7fd2f80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:02.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.848+0000 7fd2ff3ff700 1 -- 172.21.15.27:0/2273296690 shutdown_connections 2024-01-11T08:29:02.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.848+0000 7fd2ff3ff700 1 -- 172.21.15.27:0/2273296690 wait complete. 2024-01-11T08:29:02.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.850+0000 7fd2ff3ff700 1 Processor -- start 2024-01-11T08:29:02.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.850+0000 7fd2ff3ff700 1 -- start start 2024-01-11T08:29:02.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.851+0000 7fd2ff3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2f80cace0 0x7fd2f80afcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:02.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.851+0000 7fd2ff3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f813fbb0 0x7fd2f80c1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:02.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.851+0000 7fd2ff3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2f813ff80 0x7fd2f80c2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:02.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.851+0000 7fd2ff3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f80b1c30 con 0x7fd2f813fbb0 2024-01-11T08:29:02.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.851+0000 7fd2ff3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f80b1020 con 0x7fd2f80cace0 2024-01-11T08:29:02.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.851+0000 7fd2ff3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f80b1300 con 0x7fd2f813ff80 2024-01-11T08:29:02.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.851+0000 7fd2fc99a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f813fbb0 0x7fd2f80c1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:02.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.851+0000 7fd2fc99a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f813fbb0 0x7fd2f80c1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33630/0 (socket says 172.21.15.27:33630) 2024-01-11T08:29:02.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.851+0000 7fd2fc99a700 1 -- 172.21.15.27:0/1771864512 learned_addr learned my addr 172.21.15.27:0/1771864512 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:02.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.851+0000 7fd2fd19b700 1 --2- 172.21.15.27:0/1771864512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2f80cace0 0x7fd2f80afcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:02.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.851+0000 7fd2fd99c700 1 --2- 172.21.15.27:0/1771864512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2f813ff80 0x7fd2f80c2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:02.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.852+0000 7fd2fc99a700 1 -- 172.21.15.27:0/1771864512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2f813ff80 msgr2=0x7fd2f80c2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:02.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.852+0000 7fd2fc99a700 1 --2- 172.21.15.27:0/1771864512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2f813ff80 0x7fd2f80c2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:02.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.852+0000 7fd2fc99a700 1 -- 172.21.15.27:0/1771864512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2f80cace0 msgr2=0x7fd2f80afcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:02.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.852+0000 7fd2fc99a700 1 --2- 172.21.15.27:0/1771864512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2f80cace0 0x7fd2f80afcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:02.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.852+0000 7fd2fc99a700 1 -- 172.21.15.27:0/1771864512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f4009480 con 0x7fd2f813fbb0 2024-01-11T08:29:02.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.852+0000 7fd2fd99c700 1 --2- 172.21.15.27:0/1771864512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2f813ff80 0x7fd2f80c2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:02.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.852+0000 7fd2fc99a700 1 --2- 172.21.15.27:0/1771864512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f813fbb0 0x7fd2f80c1ae0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fd2e800b810 tx=0x7fd2e80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:02.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.852+0000 7fd2ed7fa700 1 -- 172.21.15.27:0/1771864512 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e800f510 con 0x7fd2f813fbb0 2024-01-11T08:29:02.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.852+0000 7fd2ed7fa700 1 -- 172.21.15.27:0/1771864512 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e800f690 con 0x7fd2f813fbb0 2024-01-11T08:29:02.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.852+0000 7fd2ff3ff700 1 -- 172.21.15.27:0/1771864512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f80b15e0 con 0x7fd2f813fbb0 2024-01-11T08:29:02.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.852+0000 7fd2ed7fa700 1 -- 172.21.15.27:0/1771864512 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e800fa50 con 0x7fd2f813fbb0 2024-01-11T08:29:02.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.852+0000 7fd2ff3ff700 1 -- 172.21.15.27:0/1771864512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f80b0230 con 0x7fd2f813fbb0 2024-01-11T08:29:02.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.853+0000 7fd2ff3ff700 1 -- 172.21.15.27:0/1771864512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f80b1de0 con 0x7fd2f813fbb0 2024-01-11T08:29:02.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.854+0000 7fd2ed7fa700 1 -- 172.21.15.27:0/1771864512 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 63) v1 ==== 84975+0+0 (secure 0 0 0) 0x7fd2e801a400 con 0x7fd2f813fbb0 2024-01-11T08:29:02.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.855+0000 7fd2ed7fa700 1 --2- 172.21.15.27:0/1771864512 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd2dc0671c0 0x7fd2dc069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:02.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.855+0000 7fd2ed7fa700 1 -- 172.21.15.27:0/1771864512 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5991+0+0 (secure 0 0 0) 0x7fd2e80845b0 con 0x7fd2f813fbb0 2024-01-11T08:29:02.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.856+0000 7fd2fd19b700 1 --2- 172.21.15.27:0/1771864512 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd2dc0671c0 0x7fd2dc069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:02.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.856+0000 7fd2fd19b700 1 --2- 172.21.15.27:0/1771864512 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd2dc0671c0 0x7fd2dc069670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4009260 tx=0x7fd2f4016000 comp rx=0 tx=0).ready entity=mgr.7498 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:02.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:02.859+0000 7fd2ed7fa700 1 -- 172.21.15.27:0/1771864512 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd2e8051450 con 0x7fd2f813fbb0 2024-01-11T08:29:03.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:03.120+0000 7fd2ff3ff700 1 -- 172.21.15.27:0/1771864512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fd2f804c3c0 con 0x7fd2f813fbb0 2024-01-11T08:29:03.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:03.812+0000 7fd2ed7fa700 1 -- 172.21.15.27:0/1771864512 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 64) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fd2e800fdc0 con 0x7fd2f813fbb0 2024-01-11T08:29:03.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:03.814+0000 7fd2ed7fa700 1 -- 172.21.15.27:0/1771864512 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v64) v1 ==== 93+0+0 (secure 0 0 0) 0x7fd2e8051450 con 0x7fd2f813fbb0 2024-01-11T08:29:03.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:03.817+0000 7fd2ff3ff700 1 -- 172.21.15.27:0/1771864512 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd2dc0671c0 msgr2=0x7fd2dc069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:03.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:03.817+0000 7fd2ff3ff700 1 --2- 172.21.15.27:0/1771864512 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd2dc0671c0 0x7fd2dc069670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4009260 tx=0x7fd2f4016000 comp rx=0 tx=0).stop 2024-01-11T08:29:03.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:03.817+0000 7fd2ff3ff700 1 -- 172.21.15.27:0/1771864512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f813fbb0 msgr2=0x7fd2f80c1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:03.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:03.817+0000 7fd2ff3ff700 1 --2- 172.21.15.27:0/1771864512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f813fbb0 0x7fd2f80c1ae0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fd2e800b810 tx=0x7fd2e80091d0 comp rx=0 tx=0).stop 2024-01-11T08:29:03.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:03.817+0000 7fd2ff3ff700 1 -- 172.21.15.27:0/1771864512 shutdown_connections 2024-01-11T08:29:03.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:03.817+0000 7fd2ff3ff700 1 --2- 172.21.15.27:0/1771864512 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd2dc0671c0 0x7fd2dc069670 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:03.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:03.817+0000 7fd2ff3ff700 1 --2- 172.21.15.27:0/1771864512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd2f813ff80 0x7fd2f80c2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:03.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:03.817+0000 7fd2ff3ff700 1 --2- 172.21.15.27:0/1771864512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2f80cace0 0x7fd2f80afcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:03.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:03.817+0000 7fd2ff3ff700 1 --2- 172.21.15.27:0/1771864512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2f813fbb0 0x7fd2f80c1ae0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:03.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:03.817+0000 7fd2ff3ff700 1 -- 172.21.15.27:0/1771864512 >> 172.21.15.27:0/1771864512 conn(0x7fd2f80680a0 msgr2=0x7fd2f80c0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:03.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:03.817+0000 7fd2ff3ff700 1 -- 172.21.15.27:0/1771864512 shutdown_connections 2024-01-11T08:29:03.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:03.818+0000 7fd2ff3ff700 1 -- 172.21.15.27:0/1771864512 wait complete. 2024-01-11T08:29:03.836 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T08:29:03.837 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:29:03.937 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:29:04.028 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T08:29:04.028 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-11T08:29:04.028 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:04.029 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-01-11T08:29:04.033 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:04.033+0000 7f502c3051c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:29:04.035 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:04.035+0000 7fa78193b1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:29:04.114 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T08:29:04.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:29:04.115 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-11T08:29:04.183 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:29:04.214 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:04.214+0000 7fa78193b1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:29:04.229 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:04.231+0000 7f502c3051c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:29:04.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.404+0000 7f39ed7ff700 1 Processor -- start 2024-01-11T08:29:04.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.404+0000 7f39ed7ff700 1 -- start start 2024-01-11T08:29:04.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.406+0000 7f39ed7ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39e81599e0 0x7f39e80b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:04.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.406+0000 7f39ed7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39e8159db0 0x7f39e815a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:04.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.406+0000 7f39ed7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f39e814fc20 0x7f39e8150080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:04.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.406+0000 7f39ed7ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39e8157e30 con 0x7f39e81599e0 2024-01-11T08:29:04.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.406+0000 7f39ed7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39e8157500 con 0x7f39e814fc20 2024-01-11T08:29:04.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.406+0000 7f39ed7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e81577e0 con 0x7f39e8159db0 2024-01-11T08:29:04.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.406+0000 7f39e6ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39e81599e0 0x7f39e80b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:04.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.406+0000 7f39e6ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39e81599e0 0x7f39e80b0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37354/0 (socket says 172.21.15.27:37354) 2024-01-11T08:29:04.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.406+0000 7f39e6ffd700 1 -- 172.21.15.27:0/1984075063 learned_addr learned my addr 172.21.15.27:0/1984075063 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:04.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.406+0000 7f39e77fe700 1 --2- 172.21.15.27:0/1984075063 >> v2:172.21.15.77:3300/0 conn(0x7f39e814fc20 0x7f39e8150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:04.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.406+0000 7f39e67fc700 1 --2- 172.21.15.27:0/1984075063 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39e8159db0 0x7f39e815a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:04.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.407+0000 7f39e6ffd700 1 -- 172.21.15.27:0/1984075063 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39e8159db0 msgr2=0x7f39e815a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:04.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.407+0000 7f39e6ffd700 1 --2- 172.21.15.27:0/1984075063 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39e8159db0 0x7f39e815a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:04.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.407+0000 7f39e6ffd700 1 -- 172.21.15.27:0/1984075063 >> v2:172.21.15.77:3300/0 conn(0x7f39e814fc20 msgr2=0x7f39e8150080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:04.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.407+0000 7f39e6ffd700 1 --2- 172.21.15.27:0/1984075063 >> v2:172.21.15.77:3300/0 conn(0x7f39e814fc20 0x7f39e8150080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:04.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.407+0000 7f39e6ffd700 1 -- 172.21.15.27:0/1984075063 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e8157ac0 con 0x7f39e81599e0 2024-01-11T08:29:04.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.407+0000 7f39e6ffd700 1 --2- 172.21.15.27:0/1984075063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39e81599e0 0x7f39e80b0130 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f39e80c89d0 tx=0x7f39d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adfb0a0b40b747d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:04.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.407+0000 7f39e5ffb700 1 -- 172.21.15.27:0/1984075063 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d800b640 con 0x7f39e81599e0 2024-01-11T08:29:04.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.407+0000 7f39e5ffb700 1 -- 172.21.15.27:0/1984075063 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d800b7a0 con 0x7f39e81599e0 2024-01-11T08:29:04.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.407+0000 7f39e5ffb700 1 -- 172.21.15.27:0/1984075063 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d800bb40 con 0x7f39e81599e0 2024-01-11T08:29:04.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.408+0000 7f39ed7ff700 1 -- 172.21.15.27:0/1984075063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39e81599e0 msgr2=0x7f39e80b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:04.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.408+0000 7f39ed7ff700 1 --2- 172.21.15.27:0/1984075063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39e81599e0 0x7f39e80b0130 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f39e80c89d0 tx=0x7f39d8009250 comp rx=0 tx=0).stop 2024-01-11T08:29:04.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.408+0000 7f39ed7ff700 1 -- 172.21.15.27:0/1984075063 shutdown_connections 2024-01-11T08:29:04.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.408+0000 7f39ed7ff700 1 --2- 172.21.15.27:0/1984075063 >> v2:172.21.15.77:3300/0 conn(0x7f39e814fc20 0x7f39e8150080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:04.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.408+0000 7f39ed7ff700 1 --2- 172.21.15.27:0/1984075063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39e81599e0 0x7f39e80b0130 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:04.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.408+0000 7f39ed7ff700 1 --2- 172.21.15.27:0/1984075063 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39e8159db0 0x7f39e815a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:04.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.408+0000 7f39ed7ff700 1 -- 172.21.15.27:0/1984075063 >> 172.21.15.27:0/1984075063 conn(0x7f39e80680e0 msgr2=0x7f39e8069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:04.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.408+0000 7f39ed7ff700 1 -- 172.21.15.27:0/1984075063 shutdown_connections 2024-01-11T08:29:04.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.408+0000 7f39ed7ff700 1 -- 172.21.15.27:0/1984075063 wait complete. 2024-01-11T08:29:04.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.410+0000 7f39ed7ff700 1 Processor -- start 2024-01-11T08:29:04.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.410+0000 7f39ed7ff700 1 -- start start 2024-01-11T08:29:04.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.411+0000 7f39ed7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39e814fc20 0x7f39e816a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:04.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.411+0000 7f39ed7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39e81599e0 0x7f39e8144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:04.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.411+0000 7f39ed7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39e8159db0 0x7f39e8144c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:04.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.411+0000 7f39ed7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e8169f70 con 0x7f39e81599e0 2024-01-11T08:29:04.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.411+0000 7f39ed7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e8145270 con 0x7f39e8159db0 2024-01-11T08:29:04.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.411+0000 7f39ed7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e8145550 con 0x7f39e814fc20 2024-01-11T08:29:04.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.411+0000 7f39e67fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39e81599e0 0x7f39e8144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:04.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.411+0000 7f39e67fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39e81599e0 0x7f39e8144700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37368/0 (socket says 172.21.15.27:37368) 2024-01-11T08:29:04.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.411+0000 7f39e67fc700 1 -- 172.21.15.27:0/2083982309 learned_addr learned my addr 172.21.15.27:0/2083982309 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:04.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.411+0000 7f39e6ffd700 1 --2- 172.21.15.27:0/2083982309 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39e814fc20 0x7f39e816a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:04.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.411+0000 7f39e77fe700 1 --2- 172.21.15.27:0/2083982309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39e8159db0 0x7f39e8144c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:04.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.412+0000 7f39e6ffd700 1 -- 172.21.15.27:0/2083982309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39e8159db0 msgr2=0x7f39e8144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:04.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.412+0000 7f39e6ffd700 1 --2- 172.21.15.27:0/2083982309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39e8159db0 0x7f39e8144c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:04.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.412+0000 7f39e6ffd700 1 -- 172.21.15.27:0/2083982309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39e81599e0 msgr2=0x7f39e8144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:04.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.412+0000 7f39e6ffd700 1 --2- 172.21.15.27:0/2083982309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39e81599e0 0x7f39e8144700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:04.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.412+0000 7f39e6ffd700 1 -- 172.21.15.27:0/2083982309 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d80094f0 con 0x7f39e814fc20 2024-01-11T08:29:04.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.412+0000 7f39e67fc700 1 --2- 172.21.15.27:0/2083982309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39e81599e0 0x7f39e8144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:04.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.412+0000 7f39e6ffd700 1 --2- 172.21.15.27:0/2083982309 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39e814fc20 0x7f39e816a660 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f39e8070d50 tx=0x7f39d8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:04.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.412+0000 7f39ceffd700 1 -- 172.21.15.27:0/2083982309 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d800bb20 con 0x7f39e814fc20 2024-01-11T08:29:04.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.412+0000 7f39ceffd700 1 -- 172.21.15.27:0/2083982309 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d8000c90 con 0x7f39e814fc20 2024-01-11T08:29:04.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.412+0000 7f39ed7ff700 1 -- 172.21.15.27:0/2083982309 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e8145830 con 0x7f39e814fc20 2024-01-11T08:29:04.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.412+0000 7f39ceffd700 1 -- 172.21.15.27:0/2083982309 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d8011690 con 0x7f39e814fc20 2024-01-11T08:29:04.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.413+0000 7f39ed7ff700 1 -- 172.21.15.27:0/2083982309 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e816aba0 con 0x7f39e814fc20 2024-01-11T08:29:04.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.414+0000 7f39ed7ff700 1 -- 172.21.15.27:0/2083982309 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e80718d0 con 0x7f39e814fc20 2024-01-11T08:29:04.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.415+0000 7f39ceffd700 1 -- 172.21.15.27:0/2083982309 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 64) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f39d8011a80 con 0x7f39e814fc20 2024-01-11T08:29:04.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.416+0000 7f39ceffd700 1 --2- 172.21.15.27:0/2083982309 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f39d4067170 0x7f39d4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:04.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.416+0000 7f39ceffd700 1 -- 172.21.15.27:0/2083982309 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5991+0+0 (secure 0 0 0) 0x7f39d8086150 con 0x7f39e814fc20 2024-01-11T08:29:04.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.416+0000 7f39e67fc700 1 -- 172.21.15.27:0/2083982309 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f39d4067170 msgr2=0x7f39d4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:04.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.416+0000 7f39e67fc700 1 --2- 172.21.15.27:0/2083982309 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f39d4067170 0x7f39d4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.421+0000 7f39ceffd700 1 -- 172.21.15.27:0/2083982309 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f39d8053010 con 0x7f39e814fc20 2024-01-11T08:29:04.532 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:04.533+0000 7fa78193b1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:29:04.573 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:04.574+0000 7f502c3051c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:29:04.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.617+0000 7f39e67fc700 1 -- 172.21.15.27:0/2083982309 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f39d4067170 msgr2=0x7f39d4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:04.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.617+0000 7f39e67fc700 1 --2- 172.21.15.27:0/2083982309 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f39d4067170 0x7f39d4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:04.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.671+0000 7f39ed7ff700 1 -- 172.21.15.27:0/2083982309 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39e80b0670 con 0x7f39e814fc20 2024-01-11T08:29:04.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.672+0000 7f39ceffd700 1 -- 172.21.15.27:0/2083982309 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4596 (secure 0 0 0) 0x7f39d8056630 con 0x7f39e814fc20 2024-01-11T08:29:04.671 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:04.671 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6608,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2001630215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2001630215},{"type":"v1","addr":"172.21.15.77:6834","nonce":2001630215}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7348,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/884469933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":884469933},{"type":"v1","addr":"172.21.15.77:6837","nonce":884469933}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":7584,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/4086481881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4086481881},{"type":"v1","addr":"172.21.15.27:6834","nonce":4086481881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:28:17.454224+0000","modified":"2024-01-11T08:28:20.975892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7587":{"gid":7587,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/2923524035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2923524035},{"type":"v1","addr":"172.21.15.27:6835","nonce":2923524035}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2024-01-11T08:29:04.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.675+0000 7f39ed7ff700 1 -- 172.21.15.27:0/2083982309 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f39d4067170 msgr2=0x7f39d4069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:04.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.675+0000 7f39ed7ff700 1 --2- 172.21.15.27:0/2083982309 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f39d4067170 0x7f39d4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:04.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.675+0000 7f39ed7ff700 1 -- 172.21.15.27:0/2083982309 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39e814fc20 msgr2=0x7f39e816a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:04.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.675+0000 7f39ed7ff700 1 --2- 172.21.15.27:0/2083982309 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39e814fc20 0x7f39e816a660 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f39e8070d50 tx=0x7f39d8017000 comp rx=0 tx=0).stop 2024-01-11T08:29:04.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.675+0000 7f39ed7ff700 1 -- 172.21.15.27:0/2083982309 shutdown_connections 2024-01-11T08:29:04.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.675+0000 7f39ed7ff700 1 --2- 172.21.15.27:0/2083982309 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f39d4067170 0x7f39d4069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:04.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.675+0000 7f39ed7ff700 1 --2- 172.21.15.27:0/2083982309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39e8159db0 0x7f39e8144c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:04.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.675+0000 7f39ed7ff700 1 --2- 172.21.15.27:0/2083982309 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39e814fc20 0x7f39e816a660 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:04.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.675+0000 7f39ed7ff700 1 --2- 172.21.15.27:0/2083982309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39e81599e0 0x7f39e8144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:04.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.675+0000 7f39ed7ff700 1 -- 172.21.15.27:0/2083982309 >> 172.21.15.27:0/2083982309 conn(0x7f39e80680e0 msgr2=0x7f39e80695b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:04.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.675+0000 7f39ed7ff700 1 -- 172.21.15.27:0/2083982309 shutdown_connections 2024-01-11T08:29:04.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.675+0000 7f39ed7ff700 1 -- 172.21.15.27:0/2083982309 wait complete. 2024-01-11T08:29:04.675 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 59 2024-01-11T08:29:04.693 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:29:04.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.949+0000 7fc8f65ff700 1 Processor -- start 2024-01-11T08:29:04.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.950+0000 7fc8f65ff700 1 -- start start 2024-01-11T08:29:04.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.951+0000 7fc8f65ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc8f00b19d0 0x7fc8f0158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:04.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.951+0000 7fc8f65ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8f006b610 0x7fc8f0159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:04.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.951+0000 7fc8f65ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f0158b60 con 0x7fc8f01575e0 2024-01-11T08:29:04.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.951+0000 7fc8f65ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f00cb5e0 con 0x7fc8f00b19d0 2024-01-11T08:29:04.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.951+0000 7fc8f65ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f01599b0 con 0x7fc8f006b610 2024-01-11T08:29:04.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.951+0000 7fc8ef7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc8f00b19d0 0x7fc8f0158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:04.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.951+0000 7fc8effff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc8f01575e0 0x7fc8f00b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:47850/0 (socket says 172.21.15.27:47850) 2024-01-11T08:29:04.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.951+0000 7fc8f4b9c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8f006b610 0x7fc8f0159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:04.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.951+0000 7fc8effff700 1 -- 172.21.15.27:0/1671107033 learned_addr learned my addr 172.21.15.27:0/1671107033 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.952+0000 7fc8eeffd700 1 -- 172.21.15.27:0/1671107033 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620093560 0 0) 0x7fc8e0001110 con 0x7fc8f01575e0 2024-01-11T08:29:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.952+0000 7fc8eeffd700 1 -- 172.21.15.27:0/1671107033 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8d8003570 con 0x7fc8f01575e0 2024-01-11T08:29:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.952+0000 7fc8f4b9c700 1 -- 172.21.15.27:0/1671107033 >> v2:172.21.15.77:3300/0 conn(0x7fc8f00b19d0 msgr2=0x7fc8f0158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.952+0000 7fc8f4b9c700 1 --2- 172.21.15.27:0/1671107033 >> v2:172.21.15.77:3300/0 conn(0x7fc8f00b19d0 0x7fc8f0158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.952+0000 7fc8f4b9c700 1 -- 172.21.15.27:0/1671107033 >> v1:172.21.15.77:6789/0 conn(0x7fc8f01575e0 legacy=0x7fc8f00b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.952+0000 7fc8f4b9c700 1 -- 172.21.15.27:0/1671107033 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f00c0f90 con 0x7fc8f006b610 2024-01-11T08:29:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.952+0000 7fc8ef7fe700 1 --2- 172.21.15.27:0/1671107033 >> v2:172.21.15.77:3300/0 conn(0x7fc8f00b19d0 0x7fc8f0158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.952+0000 7fc8f4b9c700 1 --2- 172.21.15.27:0/1671107033 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8f006b610 0x7fc8f0159e40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8009e60 tx=0x7fc8e8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0e24ffad05b0020 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.952+0000 7fc8eeffd700 1 -- 172.21.15.27:0/1671107033 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e800e070 con 0x7fc8f006b610 2024-01-11T08:29:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.953+0000 7fc8eeffd700 1 -- 172.21.15.27:0/1671107033 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e800bab0 con 0x7fc8f006b610 2024-01-11T08:29:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.953+0000 7fc8eeffd700 1 -- 172.21.15.27:0/1671107033 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e800be70 con 0x7fc8f006b610 2024-01-11T08:29:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.953+0000 7fc8f65ff700 1 -- 172.21.15.27:0/1671107033 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8f006b610 msgr2=0x7fc8f0159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.953+0000 7fc8f65ff700 1 --2- 172.21.15.27:0/1671107033 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8f006b610 0x7fc8f0159e40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8009e60 tx=0x7fc8e8009780 comp rx=0 tx=0).stop 2024-01-11T08:29:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.953+0000 7fc8f65ff700 1 -- 172.21.15.27:0/1671107033 shutdown_connections 2024-01-11T08:29:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.953+0000 7fc8f65ff700 1 --2- 172.21.15.27:0/1671107033 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8f006b610 0x7fc8f0159e40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.953+0000 7fc8f65ff700 1 --2- 172.21.15.27:0/1671107033 >> v2:172.21.15.77:3300/0 conn(0x7fc8f00b19d0 0x7fc8f0158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.953+0000 7fc8f65ff700 1 -- 172.21.15.27:0/1671107033 >> 172.21.15.27:0/1671107033 conn(0x7fc8f00680e0 msgr2=0x7fc8f0069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.953+0000 7fc8f65ff700 1 -- 172.21.15.27:0/1671107033 shutdown_connections 2024-01-11T08:29:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.953+0000 7fc8f65ff700 1 -- 172.21.15.27:0/1671107033 wait complete. 2024-01-11T08:29:04.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.955+0000 7fc8f65ff700 1 Processor -- start 2024-01-11T08:29:04.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.955+0000 7fc8f65ff700 1 -- start start 2024-01-11T08:29:04.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.956+0000 7fc8f65ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8f006b610 0x7fc8f00c6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:04.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.956+0000 7fc8f65ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8f00b19d0 0x7fc8f0169750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:04.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.956+0000 7fc8f65ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8f01575e0 0x7fc8f0153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:04.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.956+0000 7fc8f65ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f00c7490 con 0x7fc8f00b19d0 2024-01-11T08:29:04.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.956+0000 7fc8f65ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f00c8200 con 0x7fc8f01575e0 2024-01-11T08:29:04.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.956+0000 7fc8f65ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f015d330 con 0x7fc8f006b610 2024-01-11T08:29:04.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.956+0000 7fc8ef7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8f00b19d0 0x7fc8f0169750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:04.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.956+0000 7fc8effff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8f006b610 0x7fc8f00c6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:04.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.956+0000 7fc8effff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8f006b610 0x7fc8f00c6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:36840/0 (socket says 172.21.15.27:36840) 2024-01-11T08:29:04.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.956+0000 7fc8effff700 1 -- 172.21.15.27:0/1724673690 learned_addr learned my addr 172.21.15.27:0/1724673690 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:04.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.956+0000 7fc8f4b9c700 1 --2- 172.21.15.27:0/1724673690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8f01575e0 0x7fc8f0153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.957+0000 7fc8effff700 1 -- 172.21.15.27:0/1724673690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8f01575e0 msgr2=0x7fc8f0153c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.957+0000 7fc8effff700 1 --2- 172.21.15.27:0/1724673690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8f01575e0 0x7fc8f0153c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.957+0000 7fc8effff700 1 -- 172.21.15.27:0/1724673690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8f00b19d0 msgr2=0x7fc8f0169750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.957+0000 7fc8effff700 1 --2- 172.21.15.27:0/1724673690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8f00b19d0 0x7fc8f0169750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.957+0000 7fc8effff700 1 -- 172.21.15.27:0/1724673690 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e4009420 con 0x7fc8f006b610 2024-01-11T08:29:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.957+0000 7fc8effff700 1 --2- 172.21.15.27:0/1724673690 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8f006b610 0x7fc8f00c6f50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc8e000c5c0 tx=0x7fc8e0007c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.957+0000 7fc8d7fff700 1 -- 172.21.15.27:0/1724673690 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0005b30 con 0x7fc8f006b610 2024-01-11T08:29:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.957+0000 7fc8d7fff700 1 -- 172.21.15.27:0/1724673690 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e0005cb0 con 0x7fc8f006b610 2024-01-11T08:29:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.957+0000 7fc8d7fff700 1 -- 172.21.15.27:0/1724673690 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0007320 con 0x7fc8f006b610 2024-01-11T08:29:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.957+0000 7fc8f65ff700 1 -- 172.21.15.27:0/1724673690 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8e8009a20 con 0x7fc8f006b610 2024-01-11T08:29:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.958+0000 7fc8f65ff700 1 -- 172.21.15.27:0/1724673690 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f015dbd0 con 0x7fc8f006b610 2024-01-11T08:29:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.959+0000 7fc8f65ff700 1 -- 172.21.15.27:0/1724673690 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f0070af0 con 0x7fc8f006b610 2024-01-11T08:29:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.960+0000 7fc8d7fff700 1 -- 172.21.15.27:0/1724673690 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 64) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fc8e001b3b0 con 0x7fc8f006b610 2024-01-11T08:29:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.961+0000 7fc8d7fff700 1 --2- 172.21.15.27:0/1724673690 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc8dc067170 0x7fc8dc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.961+0000 7fc8d7fff700 1 -- 172.21.15.27:0/1724673690 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5991+0+0 (secure 0 0 0) 0x7fc8e0083cd0 con 0x7fc8f006b610 2024-01-11T08:29:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.961+0000 7fc8ef7fe700 1 -- 172.21.15.27:0/1724673690 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc8dc067170 msgr2=0x7fc8dc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.961+0000 7fc8ef7fe700 1 --2- 172.21.15.27:0/1724673690 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc8dc067170 0x7fc8dc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:04.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:04.970+0000 7fc8d7fff700 1 -- 172.21.15.27:0/1724673690 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc8e0050af0 con 0x7fc8f006b610 2024-01-11T08:29:05.065 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:05.065+0000 7fa78193b1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:29:05.154 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:05.155+0000 7f502c3051c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:29:05.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.162+0000 7fc8ef7fe700 1 -- 172.21.15.27:0/1724673690 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc8dc067170 msgr2=0x7fc8dc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:05.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.162+0000 7fc8ef7fe700 1 --2- 172.21.15.27:0/1724673690 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc8dc067170 0x7fc8dc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:05.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.175+0000 7fc8f65ff700 1 -- 172.21.15.27:0/1724673690 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8f0158b60 con 0x7fc8f006b610 2024-01-11T08:29:05.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.176+0000 7fc8d7fff700 1 -- 172.21.15.27:0/1724673690 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+15725 (secure 0 0 0) 0x7fc8f0158b60 con 0x7fc8f006b610 2024-01-11T08:29:05.176 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:05.176 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":117,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:28:33.921470+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:28:15.512106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-11T08:28:16.451115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6836/41381":"2024-01-12T08:28:33.921418+0000","172.21.15.77:0/709343718":"2024-01-12T08:28:33.921418+0000","172.21.15.77:6835/41381":"2024-01-12T08:28:33.921418+0000","172.21.15.77:0/2062244398":"2024-01-12T08:28:33.921418+0000","172.21.15.77:0/2474764321":"2024-01-12T08:28:33.921418+0000","172.21.15.77:0/2694649098":"2024-01-12T08:28:33.921418+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:29:05.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.179+0000 7fc8f65ff700 1 -- 172.21.15.27:0/1724673690 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc8dc067170 msgr2=0x7fc8dc069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:05.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.179+0000 7fc8f65ff700 1 --2- 172.21.15.27:0/1724673690 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc8dc067170 0x7fc8dc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.179+0000 7fc8f65ff700 1 -- 172.21.15.27:0/1724673690 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8f006b610 msgr2=0x7fc8f00c6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.179+0000 7fc8f65ff700 1 --2- 172.21.15.27:0/1724673690 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8f006b610 0x7fc8f00c6f50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc8e000c5c0 tx=0x7fc8e0007c70 comp rx=0 tx=0).stop 2024-01-11T08:29:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.179+0000 7fc8f65ff700 1 -- 172.21.15.27:0/1724673690 shutdown_connections 2024-01-11T08:29:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.179+0000 7fc8f65ff700 1 --2- 172.21.15.27:0/1724673690 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc8dc067170 0x7fc8dc069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.179+0000 7fc8f65ff700 1 --2- 172.21.15.27:0/1724673690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8f01575e0 0x7fc8f0153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.179+0000 7fc8f65ff700 1 --2- 172.21.15.27:0/1724673690 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8f006b610 0x7fc8f00c6f50 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.179+0000 7fc8f65ff700 1 --2- 172.21.15.27:0/1724673690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8f00b19d0 0x7fc8f0169750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.179+0000 7fc8f65ff700 1 -- 172.21.15.27:0/1724673690 >> 172.21.15.27:0/1724673690 conn(0x7fc8f00680e0 msgr2=0x7fc8f00b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.179+0000 7fc8f65ff700 1 -- 172.21.15.27:0/1724673690 shutdown_connections 2024-01-11T08:29:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.179+0000 7fc8f65ff700 1 -- 172.21.15.27:0/1724673690 wait complete. 2024-01-11T08:29:05.198 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T08:28:15.512106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '115', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T08:28:16.451115+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T08:29:05.199 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:29:05.235 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:05.235+0000 7fa78193b1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:29:05.341 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:05.342+0000 7f502c3051c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:29:05.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.407+0000 7fe9043ff700 1 Processor -- start 2024-01-11T08:29:05.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.407+0000 7fe9043ff700 1 -- start start 2024-01-11T08:29:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.409+0000 7fe9043ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8fc157410 0x7fe8fc0ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.409+0000 7fe9043ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe8fc1577e0 0x7fe8fc0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.409+0000 7fe9043ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe8fc069540 0x7fe8fc0ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.409+0000 7fe9043ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8fc0699d0 con 0x7fe8fc069540 2024-01-11T08:29:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.409+0000 7fe9043ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8fc069cb0 con 0x7fe8fc1577e0 2024-01-11T08:29:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.409+0000 7fe9043ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc069f90 con 0x7fe8fc157410 2024-01-11T08:29:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.409+0000 7fe90299c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe8fc069540 0x7fe8fc0ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.409+0000 7fe90299c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe8fc069540 0x7fe8fc0ae540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37384/0 (socket says 172.21.15.27:37384) 2024-01-11T08:29:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.409+0000 7fe90299c700 1 -- 172.21.15.27:0/3297738763 learned_addr learned my addr 172.21.15.27:0/3297738763 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.409+0000 7fe90199a700 1 --2- 172.21.15.27:0/3297738763 >> v2:172.21.15.77:3300/0 conn(0x7fe8fc1577e0 0x7fe8fc0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.409+0000 7fe90219b700 1 --2- 172.21.15.27:0/3297738763 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8fc157410 0x7fe8fc0ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.410+0000 7fe90299c700 1 -- 172.21.15.27:0/3297738763 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8fc157410 msgr2=0x7fe8fc0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.410+0000 7fe90299c700 1 --2- 172.21.15.27:0/3297738763 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8fc157410 0x7fe8fc0ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.410+0000 7fe90299c700 1 -- 172.21.15.27:0/3297738763 >> v2:172.21.15.77:3300/0 conn(0x7fe8fc1577e0 msgr2=0x7fe8fc0ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.410+0000 7fe90299c700 1 --2- 172.21.15.27:0/3297738763 >> v2:172.21.15.77:3300/0 conn(0x7fe8fc1577e0 0x7fe8fc0ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.410+0000 7fe90299c700 1 -- 172.21.15.27:0/3297738763 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8fc06a270 con 0x7fe8fc069540 2024-01-11T08:29:05.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.410+0000 7fe90299c700 1 --2- 172.21.15.27:0/3297738763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc069540 0x7fe8fc0ae540 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc070cc0 tx=0x7fe8f4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b7e21213d9486e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:05.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.410+0000 7fe901199700 1 -- 172.21.15.27:0/3297738763 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f400e070 con 0x7fe8fc069540 2024-01-11T08:29:05.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.410+0000 7fe901199700 1 -- 172.21.15.27:0/3297738763 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f400b970 con 0x7fe8fc069540 2024-01-11T08:29:05.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.410+0000 7fe901199700 1 -- 172.21.15.27:0/3297738763 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f40092e0 con 0x7fe8fc069540 2024-01-11T08:29:05.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.411+0000 7fe9043ff700 1 -- 172.21.15.27:0/3297738763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc069540 msgr2=0x7fe8fc0ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:05.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.411+0000 7fe9043ff700 1 --2- 172.21.15.27:0/3297738763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc069540 0x7fe8fc0ae540 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc070cc0 tx=0x7fe8f4009840 comp rx=0 tx=0).stop 2024-01-11T08:29:05.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.411+0000 7fe9043ff700 1 -- 172.21.15.27:0/3297738763 shutdown_connections 2024-01-11T08:29:05.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.411+0000 7fe9043ff700 1 --2- 172.21.15.27:0/3297738763 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8fc157410 0x7fe8fc0ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.411+0000 7fe9043ff700 1 --2- 172.21.15.27:0/3297738763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc069540 0x7fe8fc0ae540 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.411+0000 7fe9043ff700 1 --2- 172.21.15.27:0/3297738763 >> v2:172.21.15.77:3300/0 conn(0x7fe8fc1577e0 0x7fe8fc0ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.411+0000 7fe9043ff700 1 -- 172.21.15.27:0/3297738763 >> 172.21.15.27:0/3297738763 conn(0x7fe8fc0680a0 msgr2=0x7fe8fc0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.411+0000 7fe9043ff700 1 -- 172.21.15.27:0/3297738763 shutdown_connections 2024-01-11T08:29:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.411+0000 7fe9043ff700 1 -- 172.21.15.27:0/3297738763 wait complete. 2024-01-11T08:29:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.413+0000 7fe9043ff700 1 Processor -- start 2024-01-11T08:29:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.414+0000 7fe9043ff700 1 -- start start 2024-01-11T08:29:05.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.415+0000 7fe9043ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc069540 0x7fe8fc158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:05.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.415+0000 7fe9043ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8fc157410 0x7fe8fc166f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:05.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.415+0000 7fe9043ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8fc1577e0 0x7fe8fc0c8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:05.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.415+0000 7fe9043ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc16aa90 con 0x7fe8fc069540 2024-01-11T08:29:05.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.415+0000 7fe9043ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc1679d0 con 0x7fe8fc1577e0 2024-01-11T08:29:05.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.415+0000 7fe9043ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc167cb0 con 0x7fe8fc157410 2024-01-11T08:29:05.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.415+0000 7fe90219b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc069540 0x7fe8fc158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:05.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.415+0000 7fe90219b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc069540 0x7fe8fc158a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37396/0 (socket says 172.21.15.27:37396) 2024-01-11T08:29:05.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.415+0000 7fe90219b700 1 -- 172.21.15.27:0/1310168255 learned_addr learned my addr 172.21.15.27:0/1310168255 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:05.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.415+0000 7fe90199a700 1 --2- 172.21.15.27:0/1310168255 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8fc157410 0x7fe8fc166f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.415+0000 7fe90299c700 1 --2- 172.21.15.27:0/1310168255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8fc1577e0 0x7fe8fc0c8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.416+0000 7fe90219b700 1 -- 172.21.15.27:0/1310168255 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8fc157410 msgr2=0x7fe8fc166f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.416+0000 7fe90219b700 1 --2- 172.21.15.27:0/1310168255 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8fc157410 0x7fe8fc166f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.416+0000 7fe90219b700 1 -- 172.21.15.27:0/1310168255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8fc1577e0 msgr2=0x7fe8fc0c8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.416+0000 7fe90219b700 1 --2- 172.21.15.27:0/1310168255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8fc1577e0 0x7fe8fc0c8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.416+0000 7fe90219b700 1 -- 172.21.15.27:0/1310168255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f4009ae0 con 0x7fe8fc069540 2024-01-11T08:29:05.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.416+0000 7fe90219b700 1 --2- 172.21.15.27:0/1310168255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc069540 0x7fe8fc158a10 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8009e40 tx=0x7fe8f8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:05.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.417+0000 7fe8f27fc700 1 -- 172.21.15.27:0/1310168255 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f8013860 con 0x7fe8fc069540 2024-01-11T08:29:05.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.417+0000 7fe8f27fc700 1 -- 172.21.15.27:0/1310168255 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f80139c0 con 0x7fe8fc069540 2024-01-11T08:29:05.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.417+0000 7fe8f27fc700 1 -- 172.21.15.27:0/1310168255 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f8013d80 con 0x7fe8fc069540 2024-01-11T08:29:05.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.417+0000 7fe9043ff700 1 -- 172.21.15.27:0/1310168255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8fc16a720 con 0x7fe8fc069540 2024-01-11T08:29:05.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.417+0000 7fe9043ff700 1 -- 172.21.15.27:0/1310168255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8fc158f50 con 0x7fe8fc069540 2024-01-11T08:29:05.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.418+0000 7fe9043ff700 1 -- 172.21.15.27:0/1310168255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8fc16a580 con 0x7fe8fc069540 2024-01-11T08:29:05.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.419+0000 7fe8f27fc700 1 -- 172.21.15.27:0/1310168255 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 64) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fe8f800c850 con 0x7fe8fc069540 2024-01-11T08:29:05.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.420+0000 7fe8f27fc700 1 --2- 172.21.15.27:0/1310168255 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe8e0067170 0x7fe8e0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:05.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.420+0000 7fe8f27fc700 1 -- 172.21.15.27:0/1310168255 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5991+0+0 (secure 0 0 0) 0x7fe8f8088b10 con 0x7fe8fc069540 2024-01-11T08:29:05.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.420+0000 7fe90199a700 1 -- 172.21.15.27:0/1310168255 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe8e0067170 msgr2=0x7fe8e0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:05.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.420+0000 7fe90199a700 1 --2- 172.21.15.27:0/1310168255 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe8e0067170 0x7fe8e0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.425+0000 7fe8f27fc700 1 -- 172.21.15.27:0/1310168255 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe8fc16a580 con 0x7fe8fc069540 2024-01-11T08:29:05.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.621+0000 7fe90199a700 1 -- 172.21.15.27:0/1310168255 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe8e0067170 msgr2=0x7fe8e0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:05.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.621+0000 7fe90199a700 1 --2- 172.21.15.27:0/1310168255 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe8e0067170 0x7fe8e0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:05.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.628+0000 7fe9043ff700 1 -- 172.21.15.27:0/1310168255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe8fc16a580 con 0x7fe8fc069540 2024-01-11T08:29:05.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.628+0000 7fe8f27fc700 1 -- 172.21.15.27:0/1310168255 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe8fc16a580 con 0x7fe8fc069540 2024-01-11T08:29:05.627 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:29:05.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.631+0000 7fe9043ff700 1 -- 172.21.15.27:0/1310168255 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe8e0067170 msgr2=0x7fe8e0069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:05.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.631+0000 7fe9043ff700 1 --2- 172.21.15.27:0/1310168255 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe8e0067170 0x7fe8e0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.631+0000 7fe9043ff700 1 -- 172.21.15.27:0/1310168255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc069540 msgr2=0x7fe8fc158a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:05.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.631+0000 7fe9043ff700 1 --2- 172.21.15.27:0/1310168255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc069540 0x7fe8fc158a10 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8009e40 tx=0x7fe8f8009120 comp rx=0 tx=0).stop 2024-01-11T08:29:05.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.631+0000 7fe9043ff700 1 -- 172.21.15.27:0/1310168255 shutdown_connections 2024-01-11T08:29:05.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.631+0000 7fe9043ff700 1 --2- 172.21.15.27:0/1310168255 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe8e0067170 0x7fe8e0069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.631+0000 7fe9043ff700 1 --2- 172.21.15.27:0/1310168255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8fc1577e0 0x7fe8fc0c8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.631+0000 7fe9043ff700 1 --2- 172.21.15.27:0/1310168255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc069540 0x7fe8fc158a10 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.631+0000 7fe9043ff700 1 --2- 172.21.15.27:0/1310168255 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8fc157410 0x7fe8fc166f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.631+0000 7fe9043ff700 1 -- 172.21.15.27:0/1310168255 >> 172.21.15.27:0/1310168255 conn(0x7fe8fc0680a0 msgr2=0x7fe8fc0c6e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:05.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.631+0000 7fe9043ff700 1 -- 172.21.15.27:0/1310168255 shutdown_connections 2024-01-11T08:29:05.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.631+0000 7fe9043ff700 1 -- 172.21.15.27:0/1310168255 wait complete. 2024-01-11T08:29:05.648 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T08:29:05.649 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:05.649+0000 7fa78193b1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:29:05.796 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:05.795+0000 7f502c3051c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:29:05.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.856+0000 7ff9c9bff700 1 Processor -- start 2024-01-11T08:29:05.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.856+0000 7ff9c9bff700 1 -- start start 2024-01-11T08:29:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.857+0000 7ff9c9bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff9c41622d0 0x7ff9c41646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.857+0000 7ff9c9bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c40cf850 con 0x7ff9c415a1b0 2024-01-11T08:29:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.857+0000 7ff9c9bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c40cec40 con 0x7ff9c40ccd70 2024-01-11T08:29:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.857+0000 7ff9c9bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9c415ac90 con 0x7ff9c41622d0 2024-01-11T08:29:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.858+0000 7ff9c37fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff9c415a1b0 0x7ff9c415a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34638/0 (socket says 172.21.15.27:34638) 2024-01-11T08:29:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.858+0000 7ff9c37fe700 1 -- 172.21.15.27:0/3560599892 learned_addr learned my addr 172.21.15.27:0/3560599892 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.858+0000 7ff9c3fff700 1 --2- 172.21.15.27:0/3560599892 >> v2:172.21.15.77:3300/0 conn(0x7ff9c41622d0 0x7ff9c41646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:05.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.858+0000 7ff9c27fc700 1 -- 172.21.15.27:0/3560599892 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443886413 0 0) 0x7ff9c40cf850 con 0x7ff9c415a1b0 2024-01-11T08:29:05.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.858+0000 7ff9c27fc700 1 -- 172.21.15.27:0/3560599892 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9ac003570 con 0x7ff9c415a1b0 2024-01-11T08:29:05.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.858+0000 7ff9c27fc700 1 -- 172.21.15.27:0/3560599892 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265965607 0 0) 0x7ff9c40cec40 con 0x7ff9c40ccd70 2024-01-11T08:29:05.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.858+0000 7ff9c27fc700 1 -- 172.21.15.27:0/3560599892 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9c40cf850 con 0x7ff9c40ccd70 2024-01-11T08:29:05.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.858+0000 7ff9c3fff700 1 -- 172.21.15.27:0/3560599892 >> v1:172.21.15.77:6789/0 conn(0x7ff9c40ccd70 legacy=0x7ff9c40cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:05.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.858+0000 7ff9c3fff700 1 -- 172.21.15.27:0/3560599892 >> v1:172.21.15.27:6789/0 conn(0x7ff9c415a1b0 legacy=0x7ff9c415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:05.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.858+0000 7ff9c3fff700 1 -- 172.21.15.27:0/3560599892 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c40cef20 con 0x7ff9c41622d0 2024-01-11T08:29:05.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.859+0000 7ff9c3fff700 1 --2- 172.21.15.27:0/3560599892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9c41622d0 0x7ff9c41646b0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc009e60 tx=0x7ff9bc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9f68a8baa2d2089 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:05.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.859+0000 7ff9c27fc700 1 -- 172.21.15.27:0/3560599892 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc00e070 con 0x7ff9c41622d0 2024-01-11T08:29:05.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.859+0000 7ff9c27fc700 1 -- 172.21.15.27:0/3560599892 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9bc00bab0 con 0x7ff9c41622d0 2024-01-11T08:29:05.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.859+0000 7ff9c27fc700 1 -- 172.21.15.27:0/3560599892 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc00be70 con 0x7ff9c41622d0 2024-01-11T08:29:05.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.860+0000 7ff9c9bff700 1 -- 172.21.15.27:0/3560599892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9c41622d0 msgr2=0x7ff9c41646b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:05.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.860+0000 7ff9c9bff700 1 --2- 172.21.15.27:0/3560599892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9c41622d0 0x7ff9c41646b0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc009e60 tx=0x7ff9bc009780 comp rx=0 tx=0).stop 2024-01-11T08:29:05.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.860+0000 7ff9c9bff700 1 -- 172.21.15.27:0/3560599892 shutdown_connections 2024-01-11T08:29:05.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.860+0000 7ff9c9bff700 1 --2- 172.21.15.27:0/3560599892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9c41622d0 0x7ff9c41646b0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.860+0000 7ff9c9bff700 1 -- 172.21.15.27:0/3560599892 >> 172.21.15.27:0/3560599892 conn(0x7ff9c406ed40 msgr2=0x7ff9c406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:05.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.860+0000 7ff9c9bff700 1 -- 172.21.15.27:0/3560599892 shutdown_connections 2024-01-11T08:29:05.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.860+0000 7ff9c9bff700 1 -- 172.21.15.27:0/3560599892 wait complete. 2024-01-11T08:29:05.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.862+0000 7ff9c9bff700 1 Processor -- start 2024-01-11T08:29:05.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.862+0000 7ff9c9bff700 1 -- start start 2024-01-11T08:29:05.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.864+0000 7ff9c9bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9c40ccd70 0x7ff9c40b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:05.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.864+0000 7ff9c9bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9c415a1b0 0x7ff9c414e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:05.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.864+0000 7ff9c9bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9c41622d0 0x7ff9c414ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:05.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.864+0000 7ff9c9bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c416a070 con 0x7ff9c415a1b0 2024-01-11T08:29:05.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.864+0000 7ff9c9bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c40c9120 con 0x7ff9c40ccd70 2024-01-11T08:29:05.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.864+0000 7ff9c9bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c4169a20 con 0x7ff9c41622d0 2024-01-11T08:29:05.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.864+0000 7ff9c2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9c415a1b0 0x7ff9c414e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:05.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.864+0000 7ff9c37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9c40ccd70 0x7ff9c40b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:05.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.864+0000 7ff9c3fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9c41622d0 0x7ff9c414ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:05.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.864+0000 7ff9c2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9c415a1b0 0x7ff9c414e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37398/0 (socket says 172.21.15.27:37398) 2024-01-11T08:29:05.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.864+0000 7ff9c2ffd700 1 -- 172.21.15.27:0/672212333 learned_addr learned my addr 172.21.15.27:0/672212333 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:05.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.865+0000 7ff9c37fe700 1 -- 172.21.15.27:0/672212333 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9c41622d0 msgr2=0x7ff9c414ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:05.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.865+0000 7ff9c37fe700 1 --2- 172.21.15.27:0/672212333 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9c41622d0 0x7ff9c414ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.865+0000 7ff9c37fe700 1 -- 172.21.15.27:0/672212333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9c415a1b0 msgr2=0x7ff9c414e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:05.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.865+0000 7ff9c37fe700 1 --2- 172.21.15.27:0/672212333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9c415a1b0 0x7ff9c414e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:05.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.865+0000 7ff9c37fe700 1 -- 172.21.15.27:0/672212333 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9bc009a20 con 0x7ff9c40ccd70 2024-01-11T08:29:05.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.865+0000 7ff9c3fff700 1 --2- 172.21.15.27:0/672212333 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9c41622d0 0x7ff9c414ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:05.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.865+0000 7ff9c2ffd700 1 --2- 172.21.15.27:0/672212333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9c415a1b0 0x7ff9c414e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:05.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.865+0000 7ff9c37fe700 1 --2- 172.21.15.27:0/672212333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9c40ccd70 0x7ff9c40b35d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff9b400c580 tx=0x7ff9b4007d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:05.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.865+0000 7ff9ab7fe700 1 -- 172.21.15.27:0/672212333 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4005b30 con 0x7ff9c40ccd70 2024-01-11T08:29:05.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.865+0000 7ff9ab7fe700 1 -- 172.21.15.27:0/672212333 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b4005c90 con 0x7ff9c40ccd70 2024-01-11T08:29:05.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.865+0000 7ff9ab7fe700 1 -- 172.21.15.27:0/672212333 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4007320 con 0x7ff9c40ccd70 2024-01-11T08:29:05.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.865+0000 7ff9c9bff700 1 -- 172.21.15.27:0/672212333 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c4169d00 con 0x7ff9c40ccd70 2024-01-11T08:29:05.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.866+0000 7ff9c9bff700 1 -- 172.21.15.27:0/672212333 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c40b4090 con 0x7ff9c40ccd70 2024-01-11T08:29:05.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.867+0000 7ff9c9bff700 1 -- 172.21.15.27:0/672212333 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c404c3c0 con 0x7ff9c40ccd70 2024-01-11T08:29:05.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.868+0000 7ff9ab7fe700 1 -- 172.21.15.27:0/672212333 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 64) v1 ==== 84958+0+0 (secure 0 0 0) 0x7ff9b401b3b0 con 0x7ff9c40ccd70 2024-01-11T08:29:05.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.869+0000 7ff9ab7fe700 1 --2- 172.21.15.27:0/672212333 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff9b00671a0 0x7ff9b0069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:05.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.869+0000 7ff9ab7fe700 1 -- 172.21.15.27:0/672212333 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5991+0+0 (secure 0 0 0) 0x7ff9b4083e30 con 0x7ff9c40ccd70 2024-01-11T08:29:05.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.869+0000 7ff9c2ffd700 1 -- 172.21.15.27:0/672212333 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff9b00671a0 msgr2=0x7ff9b0069650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:05.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.869+0000 7ff9c2ffd700 1 --2- 172.21.15.27:0/672212333 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff9b00671a0 0x7ff9b0069650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:05.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:05.873+0000 7ff9ab7fe700 1 -- 172.21.15.27:0/672212333 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff9b4050c50 con 0x7ff9c40ccd70 2024-01-11T08:29:06.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.070+0000 7ff9c2ffd700 1 -- 172.21.15.27:0/672212333 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff9b00671a0 msgr2=0x7ff9b0069650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:06.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.070+0000 7ff9c2ffd700 1 --2- 172.21.15.27:0/672212333 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff9b00671a0 0x7ff9b0069650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:06.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.083+0000 7ff9c9bff700 1 -- 172.21.15.27:0/672212333 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff9c40cdac0 con 0x7ff9c40ccd70 2024-01-11T08:29:06.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.084+0000 7ff9ab7fe700 1 -- 172.21.15.27:0/672212333 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v117) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff9b40170a0 con 0x7ff9c40ccd70 2024-01-11T08:29:06.083 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T08:29:06.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.086+0000 7ff9c9bff700 1 -- 172.21.15.27:0/672212333 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff9b00671a0 msgr2=0x7ff9b0069650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:06.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.086+0000 7ff9c9bff700 1 --2- 172.21.15.27:0/672212333 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff9b00671a0 0x7ff9b0069650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.086+0000 7ff9c9bff700 1 -- 172.21.15.27:0/672212333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9c40ccd70 msgr2=0x7ff9c40b35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:06.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.086+0000 7ff9c9bff700 1 --2- 172.21.15.27:0/672212333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9c40ccd70 0x7ff9c40b35d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff9b400c580 tx=0x7ff9b4007d70 comp rx=0 tx=0).stop 2024-01-11T08:29:06.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.086+0000 7ff9c9bff700 1 -- 172.21.15.27:0/672212333 shutdown_connections 2024-01-11T08:29:06.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.086+0000 7ff9c9bff700 1 --2- 172.21.15.27:0/672212333 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff9b00671a0 0x7ff9b0069650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.086+0000 7ff9c9bff700 1 --2- 172.21.15.27:0/672212333 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9c41622d0 0x7ff9c414ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.086+0000 7ff9c9bff700 1 --2- 172.21.15.27:0/672212333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9c40ccd70 0x7ff9c40b35d0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.086+0000 7ff9c9bff700 1 --2- 172.21.15.27:0/672212333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9c415a1b0 0x7ff9c414e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.086+0000 7ff9c9bff700 1 -- 172.21.15.27:0/672212333 >> 172.21.15.27:0/672212333 conn(0x7ff9c406ed40 msgr2=0x7ff9c40b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:06.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.086+0000 7ff9c9bff700 1 -- 172.21.15.27:0/672212333 shutdown_connections 2024-01-11T08:29:06.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.086+0000 7ff9c9bff700 1 -- 172.21.15.27:0/672212333 wait complete. 2024-01-11T08:29:06.105 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T08:29:06.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.320+0000 7f2040fff700 1 Processor -- start 2024-01-11T08:29:06.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.320+0000 7f2040fff700 1 -- start start 2024-01-11T08:29:06.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.322+0000 7f2040fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f203c15a1b0 0x7f203c15a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:06.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.322+0000 7f2040fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f203c15ae70 0x7f203c15b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:06.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.322+0000 7f2040fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c0cf850 con 0x7f203c0ccd70 2024-01-11T08:29:06.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.322+0000 7f2040fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f203c0cec40 con 0x7f203c15ae70 2024-01-11T08:29:06.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.322+0000 7f2040fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203c0cef20 con 0x7f203c15a1b0 2024-01-11T08:29:06.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.322+0000 7f203a59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f203c15a1b0 0x7f203c15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:06.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.322+0000 7f203ad9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f203c15ae70 0x7f203c15b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:06.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.322+0000 7f203a59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f203c15a1b0 0x7f203c15a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:36860/0 (socket says 172.21.15.27:36860) 2024-01-11T08:29:06.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.322+0000 7f203ad9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f203c15ae70 0x7f203c15b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:56818/0 (socket says 172.21.15.27:56818) 2024-01-11T08:29:06.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.322+0000 7f203a59c700 1 -- 172.21.15.27:0/311504005 learned_addr learned my addr 172.21.15.27:0/311504005 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:06.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.322+0000 7f2039d9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f203c0ccd70 0x7f203c0cd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:47882/0 (socket says 172.21.15.27:47882) 2024-01-11T08:29:06.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.323+0000 7f203a59c700 1 -- 172.21.15.27:0/311504005 >> v2:172.21.15.77:3300/0 conn(0x7f203c15ae70 msgr2=0x7f203c15b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:06.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.323+0000 7f203a59c700 1 --2- 172.21.15.27:0/311504005 >> v2:172.21.15.77:3300/0 conn(0x7f203c15ae70 0x7f203c15b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.323+0000 7f203a59c700 1 -- 172.21.15.27:0/311504005 >> v1:172.21.15.77:6789/0 conn(0x7f203c0ccd70 legacy=0x7f203c0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:06.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.323+0000 7f203a59c700 1 -- 172.21.15.27:0/311504005 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203c0cf200 con 0x7f203c15a1b0 2024-01-11T08:29:06.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.323+0000 7f203ad9d700 1 --2- 172.21.15.27:0/311504005 >> v2:172.21.15.77:3300/0 conn(0x7f203c15ae70 0x7f203c15b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:06.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.323+0000 7f203959a700 1 -- 172.21.15.27:0/311504005 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933013897 0 0) 0x7f203c0cf850 con 0x7f203c0ccd70 2024-01-11T08:29:06.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.323+0000 7f203a59c700 1 --2- 172.21.15.27:0/311504005 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f203c15a1b0 0x7f203c15a580 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f202c0099b0 tx=0x7f202c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ae54de53d266916 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:06.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.323+0000 7f203959a700 1 -- 172.21.15.27:0/311504005 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c009ea0 con 0x7f203c15a1b0 2024-01-11T08:29:06.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.323+0000 7f203959a700 1 -- 172.21.15.27:0/311504005 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202c00b860 con 0x7f203c15a1b0 2024-01-11T08:29:06.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.324+0000 7f203959a700 1 -- 172.21.15.27:0/311504005 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c00bc00 con 0x7f203c15a1b0 2024-01-11T08:29:06.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.324+0000 7f2040fff700 1 -- 172.21.15.27:0/311504005 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f203c15a1b0 msgr2=0x7f203c15a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:06.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.324+0000 7f2040fff700 1 --2- 172.21.15.27:0/311504005 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f203c15a1b0 0x7f203c15a580 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f202c0099b0 tx=0x7f202c009250 comp rx=0 tx=0).stop 2024-01-11T08:29:06.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.324+0000 7f2040fff700 1 -- 172.21.15.27:0/311504005 shutdown_connections 2024-01-11T08:29:06.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.324+0000 7f2040fff700 1 --2- 172.21.15.27:0/311504005 >> v2:172.21.15.77:3300/0 conn(0x7f203c15ae70 0x7f203c15b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.324+0000 7f2040fff700 1 --2- 172.21.15.27:0/311504005 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f203c15a1b0 0x7f203c15a580 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.324+0000 7f2040fff700 1 -- 172.21.15.27:0/311504005 >> 172.21.15.27:0/311504005 conn(0x7f203c06ed40 msgr2=0x7f203c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:06.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.324+0000 7f2040fff700 1 -- 172.21.15.27:0/311504005 shutdown_connections 2024-01-11T08:29:06.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.324+0000 7f2040fff700 1 -- 172.21.15.27:0/311504005 wait complete. 2024-01-11T08:29:06.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.326+0000 7f2040fff700 1 Processor -- start 2024-01-11T08:29:06.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.326+0000 7f2040fff700 1 -- start start 2024-01-11T08:29:06.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.327+0000 7f2040fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f203c0ccd70 0x7f203c1581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:06.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.327+0000 7f2040fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f203c15a1b0 0x7f203c148f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:06.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.327+0000 7f2040fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f203c15ae70 0x7f203c15beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:06.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.327+0000 7f2040fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c157ba0 con 0x7f203c15ae70 2024-01-11T08:29:06.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.327+0000 7f2040fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c156f90 con 0x7f203c0ccd70 2024-01-11T08:29:06.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.327+0000 7f2040fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203c157270 con 0x7f203c15a1b0 2024-01-11T08:29:06.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.327+0000 7f203a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f203c0ccd70 0x7f203c1581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:06.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.327+0000 7f203ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f203c15ae70 0x7f203c15beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:06.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.327+0000 7f2039d9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f203c15a1b0 0x7f203c148f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:06.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.327+0000 7f203a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f203c0ccd70 0x7f203c1581b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:56824/0 (socket says 172.21.15.27:56824) 2024-01-11T08:29:06.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.327+0000 7f203a59c700 1 -- 172.21.15.27:0/4273649497 learned_addr learned my addr 172.21.15.27:0/4273649497 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:06.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.328+0000 7f203ad9d700 1 -- 172.21.15.27:0/4273649497 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f203c15a1b0 msgr2=0x7f203c148f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:06.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.328+0000 7f203ad9d700 1 --2- 172.21.15.27:0/4273649497 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f203c15a1b0 0x7f203c148f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.328+0000 7f203ad9d700 1 -- 172.21.15.27:0/4273649497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f203c0ccd70 msgr2=0x7f203c1581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:06.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.328+0000 7f203ad9d700 1 --2- 172.21.15.27:0/4273649497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f203c0ccd70 0x7f203c1581b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.328+0000 7f203ad9d700 1 -- 172.21.15.27:0/4273649497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2034009a20 con 0x7f203c15ae70 2024-01-11T08:29:06.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.328+0000 7f203a59c700 1 --2- 172.21.15.27:0/4273649497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f203c0ccd70 0x7f203c1581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.328+0000 7f203ad9d700 1 --2- 172.21.15.27:0/4273649497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f203c15ae70 0x7f203c15beb0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f2034009f10 tx=0x7f203400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.328+0000 7f202a7fc700 1 -- 172.21.15.27:0/4273649497 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203400cb70 con 0x7f203c15ae70 2024-01-11T08:29:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.328+0000 7f202a7fc700 1 -- 172.21.15.27:0/4273649497 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203400f040 con 0x7f203c15ae70 2024-01-11T08:29:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.328+0000 7f202a7fc700 1 -- 172.21.15.27:0/4273649497 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2034014480 con 0x7f203c15ae70 2024-01-11T08:29:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.328+0000 7f2040fff700 1 -- 172.21.15.27:0/4273649497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f202c0094f0 con 0x7f203c15ae70 2024-01-11T08:29:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.328+0000 7f2040fff700 1 -- 172.21.15.27:0/4273649497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f203c157830 con 0x7f203c15ae70 2024-01-11T08:29:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.330+0000 7f2040fff700 1 -- 172.21.15.27:0/4273649497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f203c157d10 con 0x7f203c15ae70 2024-01-11T08:29:06.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.330+0000 7f202a7fc700 1 -- 172.21.15.27:0/4273649497 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 64) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f2034003020 con 0x7f203c15ae70 2024-01-11T08:29:06.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.332+0000 7f202a7fc700 1 --2- 172.21.15.27:0/4273649497 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f201c067170 0x7f201c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:06.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.332+0000 7f202a7fc700 1 -- 172.21.15.27:0/4273649497 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5991+0+0 (secure 0 0 0) 0x7f20340853e0 con 0x7f203c15ae70 2024-01-11T08:29:06.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.332+0000 7f203a59c700 1 -- 172.21.15.27:0/4273649497 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f201c067170 msgr2=0x7f201c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:06.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.332+0000 7f203a59c700 1 --2- 172.21.15.27:0/4273649497 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f201c067170 0x7f201c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:06.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.336+0000 7f202a7fc700 1 -- 172.21.15.27:0/4273649497 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f20340520f0 con 0x7f203c15ae70 2024-01-11T08:29:06.434 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:06.434+0000 7fa78193b1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:29:06.521 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:06.522+0000 7fa78193b1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:29:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.533+0000 7f203a59c700 1 -- 172.21.15.27:0/4273649497 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f201c067170 msgr2=0x7f201c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:06.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.533+0000 7f203a59c700 1 --2- 172.21.15.27:0/4273649497 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f201c067170 0x7f201c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:06.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.540+0000 7f2040fff700 1 -- 172.21.15.27:0/4273649497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f203c04c3c0 con 0x7f203c15ae70 2024-01-11T08:29:06.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.541+0000 7f202a7fc700 1 -- 172.21.15.27:0/4273649497 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v117) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2034055710 con 0x7f203c15ae70 2024-01-11T08:29:06.540 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T08:29:06.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.543+0000 7f2040fff700 1 -- 172.21.15.27:0/4273649497 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f201c067170 msgr2=0x7f201c069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:06.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.543+0000 7f2040fff700 1 --2- 172.21.15.27:0/4273649497 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f201c067170 0x7f201c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.543+0000 7f2040fff700 1 -- 172.21.15.27:0/4273649497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f203c15ae70 msgr2=0x7f203c15beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:06.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.543+0000 7f2040fff700 1 --2- 172.21.15.27:0/4273649497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f203c15ae70 0x7f203c15beb0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f2034009f10 tx=0x7f203400b040 comp rx=0 tx=0).stop 2024-01-11T08:29:06.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.544+0000 7f2040fff700 1 -- 172.21.15.27:0/4273649497 shutdown_connections 2024-01-11T08:29:06.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.544+0000 7f2040fff700 1 --2- 172.21.15.27:0/4273649497 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f201c067170 0x7f201c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.544+0000 7f2040fff700 1 --2- 172.21.15.27:0/4273649497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f203c15ae70 0x7f203c15beb0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.544+0000 7f2040fff700 1 --2- 172.21.15.27:0/4273649497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f203c0ccd70 0x7f203c1581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.544+0000 7f2040fff700 1 --2- 172.21.15.27:0/4273649497 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f203c15a1b0 0x7f203c148f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.544+0000 7f2040fff700 1 -- 172.21.15.27:0/4273649497 >> 172.21.15.27:0/4273649497 conn(0x7f203c06ed40 msgr2=0x7f203c0c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:06.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.544+0000 7f2040fff700 1 -- 172.21.15.27:0/4273649497 shutdown_connections 2024-01-11T08:29:06.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.544+0000 7f2040fff700 1 -- 172.21.15.27:0/4273649497 wait complete. 2024-01-11T08:29:06.562 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:29:06.658 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:06.659+0000 7f502c3051c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:29:06.676 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:06.676+0000 7fa78193b1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:29:06.754 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:06.756+0000 7f502c3051c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:29:06.762 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:06.763+0000 7fa78193b1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:29:06.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.767+0000 7fa0343ff700 1 Processor -- start 2024-01-11T08:29:06.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.767+0000 7fa0343ff700 1 -- start start 2024-01-11T08:29:06.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.768+0000 7fa0343ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa02c0b03e0 0x7fa02c0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:06.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.768+0000 7fa0343ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa02c0b07b0 0x7fa02c0b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:06.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.768+0000 7fa0343ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02c0cd730 con 0x7fa02c159200 2024-01-11T08:29:06.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.768+0000 7fa0343ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa02c159d50 con 0x7fa02c0b07b0 2024-01-11T08:29:06.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.769+0000 7fa0343ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c0ccb20 con 0x7fa02c0b03e0 2024-01-11T08:29:06.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.769+0000 7fa03199a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa02c0b07b0 0x7fa02c0b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:06.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.769+0000 7fa03199a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa02c0b07b0 0x7fa02c0b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37410/0 (socket says 172.21.15.27:37410) 2024-01-11T08:29:06.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.769+0000 7fa03199a700 1 -- 172.21.15.27:0/3748736318 learned_addr learned my addr 172.21.15.27:0/3748736318 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:06.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.769+0000 7fa03219b700 1 --2- 172.21.15.27:0/3748736318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa02c0b03e0 0x7fa02c0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:06.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.769+0000 7fa03299c700 1 --1- 172.21.15.27:0/3748736318 >> v1:172.21.15.77:6789/0 conn(0x7fa02c159200 0x7fa02c159640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:47890/0 (socket says 172.21.15.27:47890) 2024-01-11T08:29:06.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.769+0000 7fa03199a700 1 -- 172.21.15.27:0/3748736318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa02c0b03e0 msgr2=0x7fa02c0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:06.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.769+0000 7fa03199a700 1 --2- 172.21.15.27:0/3748736318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa02c0b03e0 0x7fa02c0b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.769+0000 7fa03199a700 1 -- 172.21.15.27:0/3748736318 >> v1:172.21.15.77:6789/0 conn(0x7fa02c159200 legacy=0x7fa02c159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:06.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.769+0000 7fa03199a700 1 -- 172.21.15.27:0/3748736318 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02c0cce00 con 0x7fa02c0b07b0 2024-01-11T08:29:06.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.770+0000 7fa03199a700 1 --2- 172.21.15.27:0/3748736318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa02c0b07b0 0x7fa02c0b29c0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0099b0 tx=0x7fa01c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5353b2536a73a2b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:06.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.770+0000 7fa031199700 1 -- 172.21.15.27:0/3748736318 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01c009ea0 con 0x7fa02c0b07b0 2024-01-11T08:29:06.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.770+0000 7fa031199700 1 -- 172.21.15.27:0/3748736318 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa01c00b860 con 0x7fa02c0b07b0 2024-01-11T08:29:06.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.770+0000 7fa031199700 1 -- 172.21.15.27:0/3748736318 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01c00bc00 con 0x7fa02c0b07b0 2024-01-11T08:29:06.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.770+0000 7fa0343ff700 1 -- 172.21.15.27:0/3748736318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa02c0b07b0 msgr2=0x7fa02c0b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.770+0000 7fa0343ff700 1 --2- 172.21.15.27:0/3748736318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa02c0b07b0 0x7fa02c0b29c0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0099b0 tx=0x7fa01c009250 comp rx=0 tx=0).stop 2024-01-11T08:29:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.771+0000 7fa0343ff700 1 -- 172.21.15.27:0/3748736318 shutdown_connections 2024-01-11T08:29:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.771+0000 7fa0343ff700 1 --2- 172.21.15.27:0/3748736318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa02c0b03e0 0x7fa02c0b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.771+0000 7fa0343ff700 1 --2- 172.21.15.27:0/3748736318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa02c0b07b0 0x7fa02c0b29c0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.771+0000 7fa0343ff700 1 -- 172.21.15.27:0/3748736318 >> 172.21.15.27:0/3748736318 conn(0x7fa02c06ed40 msgr2=0x7fa02c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.771+0000 7fa0343ff700 1 -- 172.21.15.27:0/3748736318 shutdown_connections 2024-01-11T08:29:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.771+0000 7fa0343ff700 1 -- 172.21.15.27:0/3748736318 wait complete. 2024-01-11T08:29:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.772+0000 7fa0343ff700 1 Processor -- start 2024-01-11T08:29:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.772+0000 7fa0343ff700 1 -- start start 2024-01-11T08:29:06.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.773+0000 7fa0343ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa02c0b03e0 0x7fa02c0a6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:06.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.773+0000 7fa0343ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa02c0b07b0 0x7fa02c0af3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:06.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.773+0000 7fa0343ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa02c159200 0x7fa02c149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:06.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.773+0000 7fa0343ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c1583b0 con 0x7fa02c159200 2024-01-11T08:29:06.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.773+0000 7fa0343ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c0c7f80 con 0x7fa02c0b07b0 2024-01-11T08:29:06.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.773+0000 7fa0343ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c0a7480 con 0x7fa02c0b03e0 2024-01-11T08:29:06.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.774+0000 7fa03219b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa02c0b03e0 0x7fa02c0a6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:06.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.774+0000 7fa03219b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa02c0b03e0 0x7fa02c0a6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:36880/0 (socket says 172.21.15.27:36880) 2024-01-11T08:29:06.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.774+0000 7fa03299c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa02c159200 0x7fa02c149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:06.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.774+0000 7fa03219b700 1 -- 172.21.15.27:0/2308446133 learned_addr learned my addr 172.21.15.27:0/2308446133 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:06.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.774+0000 7fa03199a700 1 --2- 172.21.15.27:0/2308446133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa02c0b07b0 0x7fa02c0af3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:06.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.774+0000 7fa03219b700 1 -- 172.21.15.27:0/2308446133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa02c0b07b0 msgr2=0x7fa02c0af3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:06.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.774+0000 7fa03219b700 1 --2- 172.21.15.27:0/2308446133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa02c0b07b0 0x7fa02c0af3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.774+0000 7fa03219b700 1 -- 172.21.15.27:0/2308446133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa02c159200 msgr2=0x7fa02c149560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:06.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.774+0000 7fa03219b700 1 --2- 172.21.15.27:0/2308446133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa02c159200 0x7fa02c149560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:06.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.774+0000 7fa03219b700 1 -- 172.21.15.27:0/2308446133 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01c0094f0 con 0x7fa02c0b03e0 2024-01-11T08:29:06.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.774+0000 7fa03219b700 1 --2- 172.21.15.27:0/2308446133 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa02c0b03e0 0x7fa02c0a6f40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa02800ec80 tx=0x7fa0280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:06.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.775+0000 7fa0227fc700 1 -- 172.21.15.27:0/2308446133 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02800c850 con 0x7fa02c0b03e0 2024-01-11T08:29:06.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.775+0000 7fa0227fc700 1 -- 172.21.15.27:0/2308446133 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02800c9b0 con 0x7fa02c0b03e0 2024-01-11T08:29:06.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.775+0000 7fa0227fc700 1 -- 172.21.15.27:0/2308446133 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02800cd50 con 0x7fa02c0b03e0 2024-01-11T08:29:06.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.775+0000 7fa0343ff700 1 -- 172.21.15.27:0/2308446133 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa02c0d0e30 con 0x7fa02c0b03e0 2024-01-11T08:29:06.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.775+0000 7fa0343ff700 1 -- 172.21.15.27:0/2308446133 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa02c0c8500 con 0x7fa02c0b03e0 2024-01-11T08:29:06.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.776+0000 7fa0343ff700 1 -- 172.21.15.27:0/2308446133 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa02c0718d0 con 0x7fa02c0b03e0 2024-01-11T08:29:06.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.777+0000 7fa0227fc700 1 -- 172.21.15.27:0/2308446133 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 64) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fa028013680 con 0x7fa02c0b03e0 2024-01-11T08:29:06.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.778+0000 7fa0227fc700 1 --2- 172.21.15.27:0/2308446133 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa0100671a0 0x7fa010069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:06.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.778+0000 7fa0227fc700 1 -- 172.21.15.27:0/2308446133 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5991+0+0 (secure 0 0 0) 0x7fa028084c30 con 0x7fa02c0b03e0 2024-01-11T08:29:06.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.778+0000 7fa03199a700 1 -- 172.21.15.27:0/2308446133 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa0100671a0 msgr2=0x7fa010069650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:06.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.778+0000 7fa03199a700 1 --2- 172.21.15.27:0/2308446133 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa0100671a0 0x7fa010069650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:06.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.782+0000 7fa0227fc700 1 -- 172.21.15.27:0/2308446133 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa028051a50 con 0x7fa02c0b03e0 2024-01-11T08:29:06.926 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:06.927+0000 7f502c3051c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:29:06.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.979+0000 7fa03199a700 1 -- 172.21.15.27:0/2308446133 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa0100671a0 msgr2=0x7fa010069650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:06.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:06.979+0000 7fa03199a700 1 --2- 172.21.15.27:0/2308446133 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa0100671a0 0x7fa010069650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:06.983 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:06.984+0000 7fa78193b1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:29:07.019 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:07.020+0000 7f502c3051c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:29:07.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.032+0000 7fa0343ff700 1 -- 172.21.15.27:0/2308446133 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa02c0b46b0 con 0x7fa02c0b03e0 2024-01-11T08:29:07.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.033+0000 7fa0227fc700 1 -- 172.21.15.27:0/2308446133 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4596 (secure 0 0 0) 0x7fa028055070 con 0x7fa02c0b03e0 2024-01-11T08:29:07.032 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:07.032 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6608,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2001630215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2001630215},{"type":"v1","addr":"172.21.15.77:6834","nonce":2001630215}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7348,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/884469933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":884469933},{"type":"v1","addr":"172.21.15.77:6837","nonce":884469933}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":7584,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/4086481881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4086481881},{"type":"v1","addr":"172.21.15.27:6834","nonce":4086481881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:28:17.454224+0000","modified":"2024-01-11T08:28:20.975892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7587":{"gid":7587,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/2923524035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2923524035},{"type":"v1","addr":"172.21.15.27:6835","nonce":2923524035}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2024-01-11T08:29:07.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.035+0000 7fa0343ff700 1 -- 172.21.15.27:0/2308446133 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa0100671a0 msgr2=0x7fa010069650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:07.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.035+0000 7fa0343ff700 1 --2- 172.21.15.27:0/2308446133 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa0100671a0 0x7fa010069650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:07.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.035+0000 7fa0343ff700 1 -- 172.21.15.27:0/2308446133 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa02c0b03e0 msgr2=0x7fa02c0a6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:07.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.035+0000 7fa0343ff700 1 --2- 172.21.15.27:0/2308446133 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa02c0b03e0 0x7fa02c0a6f40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa02800ec80 tx=0x7fa0280091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:07.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.035+0000 7fa0343ff700 1 -- 172.21.15.27:0/2308446133 shutdown_connections 2024-01-11T08:29:07.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.035+0000 7fa0343ff700 1 --2- 172.21.15.27:0/2308446133 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa0100671a0 0x7fa010069650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:07.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.035+0000 7fa0343ff700 1 --2- 172.21.15.27:0/2308446133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa02c159200 0x7fa02c149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:07.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.035+0000 7fa0343ff700 1 --2- 172.21.15.27:0/2308446133 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa02c0b03e0 0x7fa02c0a6f40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:07.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.035+0000 7fa0343ff700 1 --2- 172.21.15.27:0/2308446133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa02c0b07b0 0x7fa02c0af3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:07.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.035+0000 7fa0343ff700 1 -- 172.21.15.27:0/2308446133 >> 172.21.15.27:0/2308446133 conn(0x7fa02c06ed40 msgr2=0x7fa02c0b48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:07.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.035+0000 7fa0343ff700 1 -- 172.21.15.27:0/2308446133 shutdown_connections 2024-01-11T08:29:07.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.035+0000 7fa0343ff700 1 -- 172.21.15.27:0/2308446133 wait complete. 2024-01-11T08:29:07.036 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 59 2024-01-11T08:29:07.052 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:29:07.079 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:07.080+0000 7fa78193b1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:29:07.175 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:07.175+0000 7fa78193b1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:29:07.251 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:07.252+0000 7f502c3051c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:29:07.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.259+0000 7f9c717ff700 1 Processor -- start 2024-01-11T08:29:07.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.260+0000 7f9c717ff700 1 -- start start 2024-01-11T08:29:07.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.261+0000 7f9c717ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9c6c0ad020 0x7f9c6c0af320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:07.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.261+0000 7f9c717ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6c0aa1d0 con 0x7f9c6c159890 2024-01-11T08:29:07.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.261+0000 7f9c717ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6c0af860 con 0x7f9c6c0ac370 2024-01-11T08:29:07.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.261+0000 7f9c717ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c6c0ac8d0 con 0x7f9c6c0ad020 2024-01-11T08:29:07.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.261+0000 7f9c6b7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9c6c159890 0x7f9c6c0aa660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34650/0 (socket says 172.21.15.27:34650) 2024-01-11T08:29:07.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.261+0000 7f9c6b7fe700 1 -- 172.21.15.27:0/3329003850 learned_addr learned my addr 172.21.15.27:0/3329003850 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:07.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.261+0000 7f9c6affd700 1 --2- 172.21.15.27:0/3329003850 >> v2:172.21.15.77:3300/0 conn(0x7f9c6c0ad020 0x7f9c6c0af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:07.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.262+0000 7f9c69ffb700 1 -- 172.21.15.27:0/3329003850 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435034472 0 0) 0x7f9c6c0aa1d0 con 0x7f9c6c159890 2024-01-11T08:29:07.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.262+0000 7f9c69ffb700 1 -- 172.21.15.27:0/3329003850 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c54003570 con 0x7f9c6c159890 2024-01-11T08:29:07.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.262+0000 7f9c6affd700 1 -- 172.21.15.27:0/3329003850 >> v1:172.21.15.77:6789/0 conn(0x7f9c6c0ac370 legacy=0x7f9c6c159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:07.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.262+0000 7f9c69ffb700 1 -- 172.21.15.27:0/3329003850 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389137853 0 0) 0x7f9c6c0af860 con 0x7f9c6c0ac370 2024-01-11T08:29:07.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.262+0000 7f9c6affd700 1 -- 172.21.15.27:0/3329003850 >> v1:172.21.15.27:6789/0 conn(0x7f9c6c159890 legacy=0x7f9c6c0aa660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:07.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.262+0000 7f9c6affd700 1 -- 172.21.15.27:0/3329003850 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6c0a98a0 con 0x7f9c6c0ad020 2024-01-11T08:29:07.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.262+0000 7f9c6affd700 1 --2- 172.21.15.27:0/3329003850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c6c0ad020 0x7f9c6c0af320 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c009910 tx=0x7f9c5c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67f81e490539cefe server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:07.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.263+0000 7f9c69ffb700 1 -- 172.21.15.27:0/3329003850 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c00b640 con 0x7f9c6c0ad020 2024-01-11T08:29:07.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.263+0000 7f9c69ffb700 1 -- 172.21.15.27:0/3329003850 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5c00b7a0 con 0x7f9c6c0ad020 2024-01-11T08:29:07.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.263+0000 7f9c69ffb700 1 -- 172.21.15.27:0/3329003850 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c00bb60 con 0x7f9c6c0ad020 2024-01-11T08:29:07.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.263+0000 7f9c717ff700 1 -- 172.21.15.27:0/3329003850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c6c0ad020 msgr2=0x7f9c6c0af320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:07.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.263+0000 7f9c717ff700 1 --2- 172.21.15.27:0/3329003850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c6c0ad020 0x7f9c6c0af320 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c009910 tx=0x7f9c5c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:07.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.263+0000 7f9c717ff700 1 -- 172.21.15.27:0/3329003850 shutdown_connections 2024-01-11T08:29:07.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.263+0000 7f9c717ff700 1 --2- 172.21.15.27:0/3329003850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c6c0ad020 0x7f9c6c0af320 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:07.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.263+0000 7f9c717ff700 1 -- 172.21.15.27:0/3329003850 >> 172.21.15.27:0/3329003850 conn(0x7f9c6c0680e0 msgr2=0x7f9c6c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:07.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.263+0000 7f9c717ff700 1 -- 172.21.15.27:0/3329003850 shutdown_connections 2024-01-11T08:29:07.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.263+0000 7f9c717ff700 1 -- 172.21.15.27:0/3329003850 wait complete. 2024-01-11T08:29:07.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.265+0000 7f9c717ff700 1 Processor -- start 2024-01-11T08:29:07.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.266+0000 7f9c717ff700 1 -- start start 2024-01-11T08:29:07.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.267+0000 7f9c717ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c6c0ac370 0x7f9c6c145c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:07.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.267+0000 7f9c717ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c6c0ad020 0x7f9c6c15d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:07.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.267+0000 7f9c717ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c6c159890 0x7f9c6c15dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:07.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.267+0000 7f9c717ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c15ef20 con 0x7f9c6c0ad020 2024-01-11T08:29:07.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.267+0000 7f9c717ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c15c960 con 0x7f9c6c0ac370 2024-01-11T08:29:07.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.267+0000 7f9c717ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c15e310 con 0x7f9c6c159890 2024-01-11T08:29:07.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.267+0000 7f9c6affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c6c0ac370 0x7f9c6c145c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:07.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.267+0000 7f9c6a7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c6c0ad020 0x7f9c6c15d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:07.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.267+0000 7f9c6affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c6c0ac370 0x7f9c6c145c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:56840/0 (socket says 172.21.15.27:56840) 2024-01-11T08:29:07.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.267+0000 7f9c6affd700 1 -- 172.21.15.27:0/239880040 learned_addr learned my addr 172.21.15.27:0/239880040 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:07.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.267+0000 7f9c6b7fe700 1 --2- 172.21.15.27:0/239880040 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c6c159890 0x7f9c6c15dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:07.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.268+0000 7f9c6affd700 1 -- 172.21.15.27:0/239880040 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c6c159890 msgr2=0x7f9c6c15dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:07.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.268+0000 7f9c6affd700 1 --2- 172.21.15.27:0/239880040 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c6c159890 0x7f9c6c15dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:07.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.268+0000 7f9c6affd700 1 -- 172.21.15.27:0/239880040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c6c0ad020 msgr2=0x7f9c6c15d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:07.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.268+0000 7f9c6affd700 1 --2- 172.21.15.27:0/239880040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c6c0ad020 0x7f9c6c15d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:07.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.268+0000 7f9c6affd700 1 -- 172.21.15.27:0/239880040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c009420 con 0x7f9c6c0ac370 2024-01-11T08:29:07.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.268+0000 7f9c6affd700 1 --2- 172.21.15.27:0/239880040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c6c0ac370 0x7f9c6c145c40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c066180 tx=0x7f9c5c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:07.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.268+0000 7f9c52ffd700 1 -- 172.21.15.27:0/239880040 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c01f070 con 0x7f9c6c0ac370 2024-01-11T08:29:07.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.268+0000 7f9c52ffd700 1 -- 172.21.15.27:0/239880040 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5c000bc0 con 0x7f9c6c0ac370 2024-01-11T08:29:07.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.268+0000 7f9c717ff700 1 -- 172.21.15.27:0/239880040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c6c15e5f0 con 0x7f9c6c0ac370 2024-01-11T08:29:07.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.268+0000 7f9c52ffd700 1 -- 172.21.15.27:0/239880040 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c00bb40 con 0x7f9c6c0ac370 2024-01-11T08:29:07.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.268+0000 7f9c717ff700 1 -- 172.21.15.27:0/239880040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c6c15ebb0 con 0x7f9c6c0ac370 2024-01-11T08:29:07.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.270+0000 7f9c717ff700 1 -- 172.21.15.27:0/239880040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c6c06b3f0 con 0x7f9c6c0ac370 2024-01-11T08:29:07.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.271+0000 7f9c52ffd700 1 -- 172.21.15.27:0/239880040 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 64) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f9c5c023070 con 0x7f9c6c0ac370 2024-01-11T08:29:07.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.271+0000 7f9c52ffd700 1 --2- 172.21.15.27:0/239880040 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9c58067170 0x7f9c58069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:07.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.272+0000 7f9c52ffd700 1 -- 172.21.15.27:0/239880040 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5991+0+0 (secure 0 0 0) 0x7f9c5c085ea0 con 0x7f9c6c0ac370 2024-01-11T08:29:07.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.272+0000 7f9c6a7fc700 1 -- 172.21.15.27:0/239880040 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9c58067170 msgr2=0x7f9c58069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:07.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.272+0000 7f9c6a7fc700 1 --2- 172.21.15.27:0/239880040 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9c58067170 0x7f9c58069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:07.272 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:07.272+0000 7fa78193b1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:29:07.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.276+0000 7f9c52ffd700 1 -- 172.21.15.27:0/239880040 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9c5c052bf0 con 0x7f9c6c0ac370 2024-01-11T08:29:07.357 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:07.358+0000 7f502c3051c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:29:07.459 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:07.460+0000 7f502c3051c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:29:07.461 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:07.462+0000 7fa78193b1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:29:07.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.473+0000 7f9c6a7fc700 1 -- 172.21.15.27:0/239880040 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9c58067170 msgr2=0x7f9c58069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:07.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.473+0000 7f9c6a7fc700 1 --2- 172.21.15.27:0/239880040 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9c58067170 0x7f9c58069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.485+0000 7f9c717ff700 1 -- 172.21.15.27:0/239880040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c6c0718d0 con 0x7f9c6c0ac370 2024-01-11T08:29:07.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.486+0000 7f9c52ffd700 1 -- 172.21.15.27:0/239880040 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+15725 (secure 0 0 0) 0x7f9c5c015090 con 0x7f9c6c0ac370 2024-01-11T08:29:07.486 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:07.486 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":117,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:28:33.921470+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:28:15.512106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-11T08:28:16.451115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6836/41381":"2024-01-12T08:28:33.921418+0000","172.21.15.77:0/709343718":"2024-01-12T08:28:33.921418+0000","172.21.15.77:6835/41381":"2024-01-12T08:28:33.921418+0000","172.21.15.77:0/2062244398":"2024-01-12T08:28:33.921418+0000","172.21.15.77:0/2474764321":"2024-01-12T08:28:33.921418+0000","172.21.15.77:0/2694649098":"2024-01-12T08:28:33.921418+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:29:07.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.489+0000 7f9c717ff700 1 -- 172.21.15.27:0/239880040 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9c58067170 msgr2=0x7f9c58069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.489+0000 7f9c717ff700 1 --2- 172.21.15.27:0/239880040 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9c58067170 0x7f9c58069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.489+0000 7f9c717ff700 1 -- 172.21.15.27:0/239880040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c6c0ac370 msgr2=0x7f9c6c145c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.489+0000 7f9c717ff700 1 --2- 172.21.15.27:0/239880040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c6c0ac370 0x7f9c6c145c40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c066180 tx=0x7f9c5c017000 comp rx=0 tx=0).stop 2024-01-11T08:29:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.489+0000 7f9c717ff700 1 -- 172.21.15.27:0/239880040 shutdown_connections 2024-01-11T08:29:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.489+0000 7f9c717ff700 1 --2- 172.21.15.27:0/239880040 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9c58067170 0x7f9c58069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.489+0000 7f9c717ff700 1 --2- 172.21.15.27:0/239880040 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9c6c159890 0x7f9c6c15dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.489+0000 7f9c717ff700 1 --2- 172.21.15.27:0/239880040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c6c0ac370 0x7f9c6c145c40 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.489+0000 7f9c717ff700 1 --2- 172.21.15.27:0/239880040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c6c0ad020 0x7f9c6c15d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.489+0000 7f9c717ff700 1 -- 172.21.15.27:0/239880040 >> 172.21.15.27:0/239880040 conn(0x7f9c6c0680e0 msgr2=0x7f9c6c06d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.489+0000 7f9c717ff700 1 -- 172.21.15.27:0/239880040 shutdown_connections 2024-01-11T08:29:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.489+0000 7f9c717ff700 1 -- 172.21.15.27:0/239880040 wait complete. 2024-01-11T08:29:07.506 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-11T08:29:07.507 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:29:07.560 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:07.561+0000 7fa78193b1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:29:07.570 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:07.571+0000 7f502c3051c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:29:07.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.719+0000 7f9e763ff700 1 Processor -- start 2024-01-11T08:29:07.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.720+0000 7f9e763ff700 1 -- start start 2024-01-11T08:29:07.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.721+0000 7f9e763ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e700c8620 0x7f9e700af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:07.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.721+0000 7f9e763ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e70157390 con 0x7f9e70069d40 2024-01-11T08:29:07.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.721+0000 7f9e763ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e700afeb0 con 0x7f9e700b09e0 2024-01-11T08:29:07.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.721+0000 7f9e763ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7006a200 con 0x7f9e700c8620 2024-01-11T08:29:07.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.721+0000 7f9e6f7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9e70069d40 0x7f9e700b02d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34666/0 (socket says 172.21.15.27:34666) 2024-01-11T08:29:07.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.721+0000 7f9e6f7fe700 1 -- 172.21.15.27:0/468288305 learned_addr learned my addr 172.21.15.27:0/468288305 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:07.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.721+0000 7f9e6ffff700 1 --2- 172.21.15.27:0/468288305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e700c8620 0x7f9e700af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.722+0000 7f9e6effd700 1 -- 172.21.15.27:0/468288305 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965775136 0 0) 0x7f9e70157390 con 0x7f9e70069d40 2024-01-11T08:29:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.722+0000 7f9e6effd700 1 -- 172.21.15.27:0/468288305 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e58003570 con 0x7f9e70069d40 2024-01-11T08:29:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.722+0000 7f9e6effd700 1 -- 172.21.15.27:0/468288305 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660219716 0 0) 0x7f9e700afeb0 con 0x7f9e700b09e0 2024-01-11T08:29:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.722+0000 7f9e6effd700 1 -- 172.21.15.27:0/468288305 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e70157390 con 0x7f9e700b09e0 2024-01-11T08:29:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.722+0000 7f9e6ffff700 1 -- 172.21.15.27:0/468288305 >> v1:172.21.15.77:6789/0 conn(0x7f9e700b09e0 legacy=0x7f9e700accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.722+0000 7f9e6ffff700 1 -- 172.21.15.27:0/468288305 >> v1:172.21.15.27:6789/0 conn(0x7f9e70069d40 legacy=0x7f9e700b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.722+0000 7f9e6ffff700 1 -- 172.21.15.27:0/468288305 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e70156d40 con 0x7f9e700c8620 2024-01-11T08:29:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.722+0000 7f9e6ffff700 1 --2- 172.21.15.27:0/468288305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e700c8620 0x7f9e700af970 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009910 tx=0x7f9e600091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc226ba47f0c1fd0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.723+0000 7f9e6effd700 1 -- 172.21.15.27:0/468288305 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60011630 con 0x7f9e700c8620 2024-01-11T08:29:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.723+0000 7f9e6effd700 1 -- 172.21.15.27:0/468288305 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e60011790 con 0x7f9e700c8620 2024-01-11T08:29:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.723+0000 7f9e6effd700 1 -- 172.21.15.27:0/468288305 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60011b30 con 0x7f9e700c8620 2024-01-11T08:29:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.723+0000 7f9e763ff700 1 -- 172.21.15.27:0/468288305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e700c8620 msgr2=0x7f9e700af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.723+0000 7f9e763ff700 1 --2- 172.21.15.27:0/468288305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e700c8620 0x7f9e700af970 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009910 tx=0x7f9e600091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.723+0000 7f9e763ff700 1 -- 172.21.15.27:0/468288305 shutdown_connections 2024-01-11T08:29:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.723+0000 7f9e763ff700 1 --2- 172.21.15.27:0/468288305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e700c8620 0x7f9e700af970 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.723+0000 7f9e763ff700 1 -- 172.21.15.27:0/468288305 >> 172.21.15.27:0/468288305 conn(0x7f9e700680e0 msgr2=0x7f9e70069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.723+0000 7f9e763ff700 1 -- 172.21.15.27:0/468288305 shutdown_connections 2024-01-11T08:29:07.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.723+0000 7f9e763ff700 1 -- 172.21.15.27:0/468288305 wait complete. 2024-01-11T08:29:07.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.725+0000 7f9e763ff700 1 Processor -- start 2024-01-11T08:29:07.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.725+0000 7f9e763ff700 1 -- start start 2024-01-11T08:29:07.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.726+0000 7f9e763ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e70069d40 0x7f9e70154220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:07.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.726+0000 7f9e763ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e700b09e0 0x7f9e7014d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:07.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.726+0000 7f9e763ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e700c8620 0x7f9e7014d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:07.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.726+0000 7f9e763ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7014c330 con 0x7f9e70069d40 2024-01-11T08:29:07.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.726+0000 7f9e763ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e70154760 con 0x7f9e700c8620 2024-01-11T08:29:07.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.726+0000 7f9e763ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e70153610 con 0x7f9e700b09e0 2024-01-11T08:29:07.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.726+0000 7f9e6ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e70069d40 0x7f9e70154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:07.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.726+0000 7f9e6ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e70069d40 0x7f9e70154220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37438/0 (socket says 172.21.15.27:37438) 2024-01-11T08:29:07.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.726+0000 7f9e6ffff700 1 -- 172.21.15.27:0/575595302 learned_addr learned my addr 172.21.15.27:0/575595302 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:07.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.726+0000 7f9e6f7fe700 1 --2- 172.21.15.27:0/575595302 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e700b09e0 0x7f9e7014d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.726+0000 7f9e7499c700 1 --2- 172.21.15.27:0/575595302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e700c8620 0x7f9e7014d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.727+0000 7f9e6ffff700 1 -- 172.21.15.27:0/575595302 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e700b09e0 msgr2=0x7f9e7014d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.727+0000 7f9e6ffff700 1 --2- 172.21.15.27:0/575595302 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e700b09e0 0x7f9e7014d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.727+0000 7f9e6ffff700 1 -- 172.21.15.27:0/575595302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e700c8620 msgr2=0x7f9e7014d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.727+0000 7f9e6ffff700 1 --2- 172.21.15.27:0/575595302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e700c8620 0x7f9e7014d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.727+0000 7f9e6ffff700 1 -- 172.21.15.27:0/575595302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e60009420 con 0x7f9e70069d40 2024-01-11T08:29:07.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.727+0000 7f9e6ffff700 1 --2- 172.21.15.27:0/575595302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e70069d40 0x7f9e70154220 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f9e70066180 tx=0x7f9e60017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.727+0000 7f9e57fff700 1 -- 172.21.15.27:0/575595302 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6001f070 con 0x7f9e70069d40 2024-01-11T08:29:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.727+0000 7f9e57fff700 1 -- 172.21.15.27:0/575595302 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e60002f70 con 0x7f9e70069d40 2024-01-11T08:29:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.727+0000 7f9e763ff700 1 -- 172.21.15.27:0/575595302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7014c560 con 0x7f9e70069d40 2024-01-11T08:29:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.727+0000 7f9e57fff700 1 -- 172.21.15.27:0/575595302 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60011900 con 0x7f9e70069d40 2024-01-11T08:29:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.728+0000 7f9e763ff700 1 -- 172.21.15.27:0/575595302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7014cb20 con 0x7f9e70069d40 2024-01-11T08:29:07.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.729+0000 7f9e763ff700 1 -- 172.21.15.27:0/575595302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e700718d0 con 0x7f9e70069d40 2024-01-11T08:29:07.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.730+0000 7f9e57fff700 1 -- 172.21.15.27:0/575595302 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 64) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f9e60023070 con 0x7f9e70069d40 2024-01-11T08:29:07.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.731+0000 7f9e57fff700 1 --2- 172.21.15.27:0/575595302 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e5c067170 0x7f9e5c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:07.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.731+0000 7f9e57fff700 1 -- 172.21.15.27:0/575595302 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5991+0+0 (secure 0 0 0) 0x7f9e60085d00 con 0x7f9e70069d40 2024-01-11T08:29:07.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.731+0000 7f9e6f7fe700 1 -- 172.21.15.27:0/575595302 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e5c067170 msgr2=0x7f9e5c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:07.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.731+0000 7f9e6f7fe700 1 --2- 172.21.15.27:0/575595302 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e5c067170 0x7f9e5c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:07.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.735+0000 7f9e57fff700 1 -- 172.21.15.27:0/575595302 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9e60052a50 con 0x7f9e70069d40 2024-01-11T08:29:07.782 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:07.784+0000 7f502c3051c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:29:07.894 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:07.895+0000 7f502c3051c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:29:07.912 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:07.913+0000 7fa78193b1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:29:07.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.932+0000 7f9e6f7fe700 1 -- 172.21.15.27:0/575595302 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e5c067170 msgr2=0x7f9e5c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:07.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:07.932+0000 7f9e6f7fe700 1 --2- 172.21.15.27:0/575595302 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e5c067170 0x7f9e5c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:07.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.001+0000 7f9e763ff700 1 -- 172.21.15.27:0/575595302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e70157020 con 0x7f9e70069d40 2024-01-11T08:29:08.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.001+0000 7f9e57fff700 1 -- 172.21.15.27:0/575595302 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4596 (secure 0 0 0) 0x7f9e6002a090 con 0x7f9e70069d40 2024-01-11T08:29:08.000 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:08.000 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6608,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2001630215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2001630215},{"type":"v1","addr":"172.21.15.77:6834","nonce":2001630215}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7348,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/884469933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":884469933},{"type":"v1","addr":"172.21.15.77:6837","nonce":884469933}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":7584,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/4086481881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4086481881},{"type":"v1","addr":"172.21.15.27:6834","nonce":4086481881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:28:17.454224+0000","modified":"2024-01-11T08:28:20.975892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7587":{"gid":7587,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/2923524035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2923524035},{"type":"v1","addr":"172.21.15.27:6835","nonce":2923524035}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2024-01-11T08:29:08.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.004+0000 7f9e763ff700 1 -- 172.21.15.27:0/575595302 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e5c067170 msgr2=0x7f9e5c069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:08.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.004+0000 7f9e763ff700 1 --2- 172.21.15.27:0/575595302 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e5c067170 0x7f9e5c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.004+0000 7f9e763ff700 1 -- 172.21.15.27:0/575595302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e70069d40 msgr2=0x7f9e70154220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:08.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.004+0000 7f9e763ff700 1 --2- 172.21.15.27:0/575595302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e70069d40 0x7f9e70154220 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f9e70066180 tx=0x7f9e60017000 comp rx=0 tx=0).stop 2024-01-11T08:29:08.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.004+0000 7f9e763ff700 1 -- 172.21.15.27:0/575595302 shutdown_connections 2024-01-11T08:29:08.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.004+0000 7f9e763ff700 1 --2- 172.21.15.27:0/575595302 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9e5c067170 0x7f9e5c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.004+0000 7f9e763ff700 1 --2- 172.21.15.27:0/575595302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e700c8620 0x7f9e7014d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.004+0000 7f9e763ff700 1 --2- 172.21.15.27:0/575595302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e70069d40 0x7f9e70154220 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.004+0000 7f9e763ff700 1 --2- 172.21.15.27:0/575595302 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e700b09e0 0x7f9e7014d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.004+0000 7f9e763ff700 1 -- 172.21.15.27:0/575595302 >> 172.21.15.27:0/575595302 conn(0x7f9e700680e0 msgr2=0x7f9e70158680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:08.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.004+0000 7f9e763ff700 1 -- 172.21.15.27:0/575595302 shutdown_connections 2024-01-11T08:29:08.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.004+0000 7f9e763ff700 1 -- 172.21.15.27:0/575595302 wait complete. 2024-01-11T08:29:08.004 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 59 2024-01-11T08:29:08.007 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:08.008+0000 7fa78193b1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:29:08.021 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:29:08.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.228+0000 7f06609ff700 1 Processor -- start 2024-01-11T08:29:08.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.229+0000 7f06609ff700 1 -- start start 2024-01-11T08:29:08.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.230+0000 7f06609ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c0c8620 0x7f065c0af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:08.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.230+0000 7f06609ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f065c0b09e0 0x7f065c0accc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:08.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.230+0000 7f06609ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065c157390 con 0x7f065c069d40 2024-01-11T08:29:08.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.230+0000 7f06609ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f065c0afeb0 con 0x7f065c0b09e0 2024-01-11T08:29:08.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.230+0000 7f06609ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065c06a200 con 0x7f065c0c8620 2024-01-11T08:29:08.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.230+0000 7f0659d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f065c069d40 0x7f065c0b02d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34668/0 (socket says 172.21.15.27:34668) 2024-01-11T08:29:08.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.230+0000 7f0659d9b700 1 -- 172.21.15.27:0/4031172600 learned_addr learned my addr 172.21.15.27:0/4031172600 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:08.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.230+0000 7f065a59c700 1 --2- 172.21.15.27:0/4031172600 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c0c8620 0x7f065c0af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:08.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.230+0000 7f065ad9d700 1 --2- 172.21.15.27:0/4031172600 >> v2:172.21.15.77:3300/0 conn(0x7f065c0b09e0 0x7f065c0accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:08.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.231+0000 7f065959a700 1 -- 172.21.15.27:0/4031172600 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021853117 0 0) 0x7f065c157390 con 0x7f065c069d40 2024-01-11T08:29:08.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.231+0000 7f065959a700 1 -- 172.21.15.27:0/4031172600 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0644003570 con 0x7f065c069d40 2024-01-11T08:29:08.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.231+0000 7f065a59c700 1 -- 172.21.15.27:0/4031172600 >> v2:172.21.15.77:3300/0 conn(0x7f065c0b09e0 msgr2=0x7f065c0accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.231+0000 7f065a59c700 1 --2- 172.21.15.27:0/4031172600 >> v2:172.21.15.77:3300/0 conn(0x7f065c0b09e0 0x7f065c0accc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.231+0000 7f065a59c700 1 -- 172.21.15.27:0/4031172600 >> v1:172.21.15.27:6789/0 conn(0x7f065c069d40 legacy=0x7f065c0b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.231+0000 7f065a59c700 1 -- 172.21.15.27:0/4031172600 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065c156d40 con 0x7f065c0c8620 2024-01-11T08:29:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.231+0000 7f065ad9d700 1 --2- 172.21.15.27:0/4031172600 >> v2:172.21.15.77:3300/0 conn(0x7f065c0b09e0 0x7f065c0accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.231+0000 7f065a59c700 1 --2- 172.21.15.27:0/4031172600 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c0c8620 0x7f065c0af970 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f064c0098e0 tx=0x7f064c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd91ebe277e5e69d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.232+0000 7f065959a700 1 -- 172.21.15.27:0/4031172600 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c011630 con 0x7f065c0c8620 2024-01-11T08:29:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.232+0000 7f065959a700 1 -- 172.21.15.27:0/4031172600 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064c011790 con 0x7f065c0c8620 2024-01-11T08:29:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.232+0000 7f065959a700 1 -- 172.21.15.27:0/4031172600 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c011b30 con 0x7f065c0c8620 2024-01-11T08:29:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.232+0000 7f06609ff700 1 -- 172.21.15.27:0/4031172600 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c0c8620 msgr2=0x7f065c0af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.232+0000 7f06609ff700 1 --2- 172.21.15.27:0/4031172600 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c0c8620 0x7f065c0af970 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f064c0098e0 tx=0x7f064c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.232+0000 7f06609ff700 1 -- 172.21.15.27:0/4031172600 shutdown_connections 2024-01-11T08:29:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.232+0000 7f06609ff700 1 --2- 172.21.15.27:0/4031172600 >> v2:172.21.15.77:3300/0 conn(0x7f065c0b09e0 0x7f065c0accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.232+0000 7f06609ff700 1 --2- 172.21.15.27:0/4031172600 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c0c8620 0x7f065c0af970 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.232+0000 7f06609ff700 1 -- 172.21.15.27:0/4031172600 >> 172.21.15.27:0/4031172600 conn(0x7f065c0680e0 msgr2=0x7f065c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:08.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.232+0000 7f06609ff700 1 -- 172.21.15.27:0/4031172600 shutdown_connections 2024-01-11T08:29:08.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.232+0000 7f06609ff700 1 -- 172.21.15.27:0/4031172600 wait complete. 2024-01-11T08:29:08.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.234+0000 7f06609ff700 1 Processor -- start 2024-01-11T08:29:08.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.234+0000 7f06609ff700 1 -- start start 2024-01-11T08:29:08.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.235+0000 7f06609ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f065c069d40 0x7f065c143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:08.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.235+0000 7f06609ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f065c0b09e0 0x7f065c1538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:08.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.235+0000 7f06609ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c0c8620 0x7f065c153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:08.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.235+0000 7f06609ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c0cb400 con 0x7f065c069d40 2024-01-11T08:29:08.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.235+0000 7f06609ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c169a30 con 0x7f065c0b09e0 2024-01-11T08:29:08.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.235+0000 7f06609ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065c0cbe10 con 0x7f065c0c8620 2024-01-11T08:29:08.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.235+0000 7f065a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f065c069d40 0x7f065c143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:08.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.235+0000 7f0659d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f065c0b09e0 0x7f065c1538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:08.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.235+0000 7f065a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f065c069d40 0x7f065c143600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37454/0 (socket says 172.21.15.27:37454) 2024-01-11T08:29:08.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.235+0000 7f065a59c700 1 -- 172.21.15.27:0/3526432102 learned_addr learned my addr 172.21.15.27:0/3526432102 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:08.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.235+0000 7f065ad9d700 1 --2- 172.21.15.27:0/3526432102 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c0c8620 0x7f065c153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.236+0000 7f065ad9d700 1 -- 172.21.15.27:0/3526432102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f065c0b09e0 msgr2=0x7f065c1538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.236+0000 7f065ad9d700 1 --2- 172.21.15.27:0/3526432102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f065c0b09e0 0x7f065c1538a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.236+0000 7f065ad9d700 1 -- 172.21.15.27:0/3526432102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f065c069d40 msgr2=0x7f065c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.236+0000 7f065ad9d700 1 --2- 172.21.15.27:0/3526432102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f065c069d40 0x7f065c143600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.236+0000 7f065ad9d700 1 -- 172.21.15.27:0/3526432102 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0654009a20 con 0x7f065c0c8620 2024-01-11T08:29:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.236+0000 7f0659d9b700 1 --2- 172.21.15.27:0/3526432102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f065c0b09e0 0x7f065c1538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.236+0000 7f065a59c700 1 --2- 172.21.15.27:0/3526432102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f065c069d40 0x7f065c143600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.236+0000 7f065ad9d700 1 --2- 172.21.15.27:0/3526432102 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c0c8620 0x7f065c153de0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0654009ee0 tx=0x7f065400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.236+0000 7f064a7fc700 1 -- 172.21.15.27:0/3526432102 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065400cb70 con 0x7f065c0c8620 2024-01-11T08:29:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.237+0000 7f064a7fc700 1 -- 172.21.15.27:0/3526432102 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065400f040 con 0x7f065c0c8620 2024-01-11T08:29:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.237+0000 7f064a7fc700 1 -- 172.21.15.27:0/3526432102 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0654014480 con 0x7f065c0c8620 2024-01-11T08:29:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.237+0000 7f06609ff700 1 -- 172.21.15.27:0/3526432102 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f064c009420 con 0x7f065c0c8620 2024-01-11T08:29:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.237+0000 7f06609ff700 1 -- 172.21.15.27:0/3526432102 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f065c0ca7f0 con 0x7f065c0c8620 2024-01-11T08:29:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.238+0000 7f06609ff700 1 -- 172.21.15.27:0/3526432102 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065c0cb5b0 con 0x7f065c0c8620 2024-01-11T08:29:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.239+0000 7f064a7fc700 1 -- 172.21.15.27:0/3526432102 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 64) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f0654003020 con 0x7f065c0c8620 2024-01-11T08:29:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.240+0000 7f064a7fc700 1 --2- 172.21.15.27:0/3526432102 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f063c067170 0x7f063c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.240+0000 7f064a7fc700 1 -- 172.21.15.27:0/3526432102 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f0654085840 con 0x7f065c0c8620 2024-01-11T08:29:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.240+0000 7f065a59c700 1 -- 172.21.15.27:0/3526432102 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f063c067170 msgr2=0x7f063c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.240+0000 7f065a59c700 1 --2- 172.21.15.27:0/3526432102 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f063c067170 0x7f063c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:08.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.246+0000 7f064a7fc700 1 -- 172.21.15.27:0/3526432102 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f06540525a0 con 0x7f065c0c8620 2024-01-11T08:29:08.276 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:08.277+0000 7f502c3051c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:29:08.372 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:08.373+0000 7f502c3051c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:29:08.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.441+0000 7f065a59c700 1 -- 172.21.15.27:0/3526432102 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f063c067170 msgr2=0x7f063c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:08.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.441+0000 7f065a59c700 1 --2- 172.21.15.27:0/3526432102 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f063c067170 0x7f063c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:08.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.496+0000 7f06609ff700 1 -- 172.21.15.27:0/3526432102 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f065c0cb5b0 con 0x7f065c0c8620 2024-01-11T08:29:08.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.497+0000 7f064a7fc700 1 -- 172.21.15.27:0/3526432102 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4596 (secure 0 0 0) 0x7f065c0cb5b0 con 0x7f065c0c8620 2024-01-11T08:29:08.496 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:08.496 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6608,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2001630215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2001630215},{"type":"v1","addr":"172.21.15.77:6834","nonce":2001630215}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7348,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/884469933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":884469933},{"type":"v1","addr":"172.21.15.77:6837","nonce":884469933}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":7584,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/4086481881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4086481881},{"type":"v1","addr":"172.21.15.27:6834","nonce":4086481881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:28:17.454224+0000","modified":"2024-01-11T08:28:20.975892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7587":{"gid":7587,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/2923524035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2923524035},{"type":"v1","addr":"172.21.15.27:6835","nonce":2923524035}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2024-01-11T08:29:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.499+0000 7f06609ff700 1 -- 172.21.15.27:0/3526432102 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f063c067170 msgr2=0x7f063c069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.499+0000 7f06609ff700 1 --2- 172.21.15.27:0/3526432102 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f063c067170 0x7f063c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.499+0000 7f06609ff700 1 -- 172.21.15.27:0/3526432102 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c0c8620 msgr2=0x7f065c153de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.499+0000 7f06609ff700 1 --2- 172.21.15.27:0/3526432102 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c0c8620 0x7f065c153de0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0654009ee0 tx=0x7f065400b040 comp rx=0 tx=0).stop 2024-01-11T08:29:08.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.499+0000 7f06609ff700 1 -- 172.21.15.27:0/3526432102 shutdown_connections 2024-01-11T08:29:08.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.499+0000 7f06609ff700 1 --2- 172.21.15.27:0/3526432102 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f063c067170 0x7f063c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.499+0000 7f06609ff700 1 --2- 172.21.15.27:0/3526432102 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f065c0c8620 0x7f065c153de0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.499+0000 7f06609ff700 1 --2- 172.21.15.27:0/3526432102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f065c069d40 0x7f065c143600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.499+0000 7f06609ff700 1 --2- 172.21.15.27:0/3526432102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f065c0b09e0 0x7f065c1538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.499+0000 7f06609ff700 1 -- 172.21.15.27:0/3526432102 >> 172.21.15.27:0/3526432102 conn(0x7f065c0680e0 msgr2=0x7f065c151a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:08.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.499+0000 7f06609ff700 1 -- 172.21.15.27:0/3526432102 shutdown_connections 2024-01-11T08:29:08.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.499+0000 7f06609ff700 1 -- 172.21.15.27:0/3526432102 wait complete. 2024-01-11T08:29:08.499 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 59 2024-01-11T08:29:08.519 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:29:08.556 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:08.556+0000 7fa78193b1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:29:08.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.721+0000 7fd7553ff700 1 Processor -- start 2024-01-11T08:29:08.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.722+0000 7fd7553ff700 1 -- start start 2024-01-11T08:29:08.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.723+0000 7fd7553ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd7500ca3e0 0x7fd7500cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:08.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.723+0000 7fd7553ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd7500ca7b0 0x7fd750159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:08.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.723+0000 7fd7553ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7500c8a20 0x7fd7500c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:08.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.723+0000 7fd7553ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd75015ae70 con 0x7fd7500ca3e0 2024-01-11T08:29:08.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.723+0000 7fd7553ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd75015a3c0 con 0x7fd7500ca7b0 2024-01-11T08:29:08.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.723+0000 7fd7553ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7500cc4f0 con 0x7fd7500c8a20 2024-01-11T08:29:08.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.723+0000 7fd74effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd7500ca3e0 0x7fd7500cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:08.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.723+0000 7fd74effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd7500ca3e0 0x7fd7500cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37470/0 (socket says 172.21.15.27:37470) 2024-01-11T08:29:08.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.723+0000 7fd74effd700 1 -- 172.21.15.27:0/906908972 learned_addr learned my addr 172.21.15.27:0/906908972 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:08.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.723+0000 7fd74e7fc700 1 --2- 172.21.15.27:0/906908972 >> v2:172.21.15.77:3300/0 conn(0x7fd7500ca7b0 0x7fd750159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:08.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.723+0000 7fd74f7fe700 1 --2- 172.21.15.27:0/906908972 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7500c8a20 0x7fd7500c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:08.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.724+0000 7fd74effd700 1 -- 172.21.15.27:0/906908972 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7500c8a20 msgr2=0x7fd7500c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:08.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.724+0000 7fd74effd700 1 --2- 172.21.15.27:0/906908972 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7500c8a20 0x7fd7500c8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.724+0000 7fd74effd700 1 -- 172.21.15.27:0/906908972 >> v2:172.21.15.77:3300/0 conn(0x7fd7500ca7b0 msgr2=0x7fd750159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:08.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.724+0000 7fd74effd700 1 --2- 172.21.15.27:0/906908972 >> v2:172.21.15.77:3300/0 conn(0x7fd7500ca7b0 0x7fd750159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.724+0000 7fd74effd700 1 -- 172.21.15.27:0/906908972 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75015d640 con 0x7fd7500ca3e0 2024-01-11T08:29:08.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.724+0000 7fd74effd700 1 --2- 172.21.15.27:0/906908972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7500ca3e0 0x7fd7500cca20 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fd7400099e0 tx=0x7fd740009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5e2b76897674c0f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:08.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.724+0000 7fd74dffb700 1 -- 172.21.15.27:0/906908972 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd740011630 con 0x7fd7500ca3e0 2024-01-11T08:29:08.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.724+0000 7fd74dffb700 1 -- 172.21.15.27:0/906908972 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd740011790 con 0x7fd7500ca3e0 2024-01-11T08:29:08.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.724+0000 7fd74dffb700 1 -- 172.21.15.27:0/906908972 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd740011b30 con 0x7fd7500ca3e0 2024-01-11T08:29:08.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.725+0000 7fd7553ff700 1 -- 172.21.15.27:0/906908972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7500ca3e0 msgr2=0x7fd7500cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:08.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.725+0000 7fd7553ff700 1 --2- 172.21.15.27:0/906908972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7500ca3e0 0x7fd7500cca20 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fd7400099e0 tx=0x7fd740009250 comp rx=0 tx=0).stop 2024-01-11T08:29:08.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.725+0000 7fd7553ff700 1 -- 172.21.15.27:0/906908972 shutdown_connections 2024-01-11T08:29:08.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.725+0000 7fd7553ff700 1 --2- 172.21.15.27:0/906908972 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7500c8a20 0x7fd7500c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.725+0000 7fd7553ff700 1 --2- 172.21.15.27:0/906908972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7500ca3e0 0x7fd7500cca20 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.725+0000 7fd7553ff700 1 --2- 172.21.15.27:0/906908972 >> v2:172.21.15.77:3300/0 conn(0x7fd7500ca7b0 0x7fd750159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.725+0000 7fd7553ff700 1 -- 172.21.15.27:0/906908972 >> 172.21.15.27:0/906908972 conn(0x7fd7500ce3e0 msgr2=0x7fd7500ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:08.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.725+0000 7fd7553ff700 1 -- 172.21.15.27:0/906908972 shutdown_connections 2024-01-11T08:29:08.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.725+0000 7fd7553ff700 1 -- 172.21.15.27:0/906908972 wait complete. 2024-01-11T08:29:08.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.727+0000 7fd7553ff700 1 Processor -- start 2024-01-11T08:29:08.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.727+0000 7fd7553ff700 1 -- start start 2024-01-11T08:29:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.728+0000 7fd7553ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7500c8a20 0x7fd7500b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.728+0000 7fd7553ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7500ca3e0 0x7fd750169f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.728+0000 7fd7553ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7500ca7b0 0x7fd75016a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.728+0000 7fd7553ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7500b16c0 con 0x7fd7500ca3e0 2024-01-11T08:29:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.728+0000 7fd7553ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7500b40b0 con 0x7fd7500c8a20 2024-01-11T08:29:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.728+0000 7fd7553ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd75015ef40 con 0x7fd7500ca7b0 2024-01-11T08:29:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.728+0000 7fd74e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7500ca3e0 0x7fd750169f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.728+0000 7fd74e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7500ca3e0 0x7fd750169f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37478/0 (socket says 172.21.15.27:37478) 2024-01-11T08:29:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.728+0000 7fd74e7fc700 1 -- 172.21.15.27:0/2980879737 learned_addr learned my addr 172.21.15.27:0/2980879737 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:08.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.728+0000 7fd74effd700 1 --2- 172.21.15.27:0/2980879737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7500c8a20 0x7fd7500b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:08.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.728+0000 7fd74f7fe700 1 --2- 172.21.15.27:0/2980879737 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7500ca7b0 0x7fd75016a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.729+0000 7fd74e7fc700 1 -- 172.21.15.27:0/2980879737 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7500ca7b0 msgr2=0x7fd75016a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.729+0000 7fd74e7fc700 1 --2- 172.21.15.27:0/2980879737 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7500ca7b0 0x7fd75016a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.729+0000 7fd74e7fc700 1 -- 172.21.15.27:0/2980879737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7500c8a20 msgr2=0x7fd7500b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.729+0000 7fd74e7fc700 1 --2- 172.21.15.27:0/2980879737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7500c8a20 0x7fd7500b1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.729+0000 7fd74e7fc700 1 -- 172.21.15.27:0/2980879737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7400094f0 con 0x7fd7500ca3e0 2024-01-11T08:29:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.729+0000 7fd74effd700 1 --2- 172.21.15.27:0/2980879737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7500c8a20 0x7fd7500b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.729+0000 7fd74e7fc700 1 --2- 172.21.15.27:0/2980879737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7500ca3e0 0x7fd750169f80 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fd74400e960 tx=0x7fd7440091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.729+0000 7fd736ffd700 1 -- 172.21.15.27:0/2980879737 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd744013680 con 0x7fd7500ca3e0 2024-01-11T08:29:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.729+0000 7fd736ffd700 1 -- 172.21.15.27:0/2980879737 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7440137e0 con 0x7fd7500ca3e0 2024-01-11T08:29:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.729+0000 7fd7553ff700 1 -- 172.21.15.27:0/2980879737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7500c8200 con 0x7fd7500ca3e0 2024-01-11T08:29:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.729+0000 7fd736ffd700 1 -- 172.21.15.27:0/2980879737 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd744013b80 con 0x7fd7500ca3e0 2024-01-11T08:29:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.729+0000 7fd7553ff700 1 -- 172.21.15.27:0/2980879737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7500b3af0 con 0x7fd7500ca3e0 2024-01-11T08:29:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.730+0000 7fd7553ff700 1 -- 172.21.15.27:0/2980879737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7500718d0 con 0x7fd7500ca3e0 2024-01-11T08:29:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.731+0000 7fd736ffd700 1 -- 172.21.15.27:0/2980879737 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 64) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fd7440123d0 con 0x7fd7500ca3e0 2024-01-11T08:29:08.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.733+0000 7fd736ffd700 1 --2- 172.21.15.27:0/2980879737 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd73c067170 0x7fd73c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:08.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.733+0000 7fd736ffd700 1 -- 172.21.15.27:0/2980879737 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6047+0+0 (secure 0 0 0) 0x7fd744084af0 con 0x7fd7500ca3e0 2024-01-11T08:29:08.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.733+0000 7fd74effd700 1 -- 172.21.15.27:0/2980879737 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd73c067170 msgr2=0x7fd73c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:08.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.733+0000 7fd74effd700 1 --2- 172.21.15.27:0/2980879737 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd73c067170 0x7fd73c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:08.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.737+0000 7fd736ffd700 1 -- 172.21.15.27:0/2980879737 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd744051980 con 0x7fd7500ca3e0 2024-01-11T08:29:08.742 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:08.743+0000 7fa78193b1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:29:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.934+0000 7fd74effd700 1 -- 172.21.15.27:0/2980879737 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd73c067170 msgr2=0x7fd73c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.934+0000 7fd74effd700 1 --2- 172.21.15.27:0/2980879737 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd73c067170 0x7fd73c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:08.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.943+0000 7fd7553ff700 1 -- 172.21.15.27:0/2980879737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd75015f080 con 0x7fd7500ca3e0 2024-01-11T08:29:08.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.944+0000 7fd736ffd700 1 -- 172.21.15.27:0/2980879737 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+15837 (secure 0 0 0) 0x7fd744054fa0 con 0x7fd7500ca3e0 2024-01-11T08:29:08.943 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:08.943 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":118,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:29:07.817107+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:28:15.512106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-11T08:28:16.451115+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":118,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6836/41381":"2024-01-12T08:28:33.921418+0000","172.21.15.77:0/709343718":"2024-01-12T08:28:33.921418+0000","172.21.15.77:6835/41381":"2024-01-12T08:28:33.921418+0000","172.21.15.77:0/2062244398":"2024-01-12T08:28:33.921418+0000","172.21.15.77:0/2474764321":"2024-01-12T08:28:33.921418+0000","172.21.15.77:0/2694649098":"2024-01-12T08:28:33.921418+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:29:08.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.947+0000 7fd7553ff700 1 -- 172.21.15.27:0/2980879737 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd73c067170 msgr2=0x7fd73c069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:08.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.947+0000 7fd7553ff700 1 --2- 172.21.15.27:0/2980879737 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd73c067170 0x7fd73c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.947+0000 7fd7553ff700 1 -- 172.21.15.27:0/2980879737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7500ca3e0 msgr2=0x7fd750169f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:08.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.947+0000 7fd7553ff700 1 --2- 172.21.15.27:0/2980879737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7500ca3e0 0x7fd750169f80 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fd74400e960 tx=0x7fd7440091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.947+0000 7fd7553ff700 1 -- 172.21.15.27:0/2980879737 shutdown_connections 2024-01-11T08:29:08.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.947+0000 7fd7553ff700 1 --2- 172.21.15.27:0/2980879737 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd73c067170 0x7fd73c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.947+0000 7fd7553ff700 1 --2- 172.21.15.27:0/2980879737 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7500ca7b0 0x7fd75016a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.947+0000 7fd7553ff700 1 --2- 172.21.15.27:0/2980879737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7500c8a20 0x7fd7500b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.947+0000 7fd7553ff700 1 --2- 172.21.15.27:0/2980879737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7500ca3e0 0x7fd750169f80 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:08.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.947+0000 7fd7553ff700 1 -- 172.21.15.27:0/2980879737 >> 172.21.15.27:0/2980879737 conn(0x7fd7500ce3e0 msgr2=0x7fd750068920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:08.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.947+0000 7fd7553ff700 1 -- 172.21.15.27:0/2980879737 shutdown_connections 2024-01-11T08:29:08.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:08.947+0000 7fd7553ff700 1 -- 172.21.15.27:0/2980879737 wait complete. 2024-01-11T08:29:08.965 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-11T08:29:08.970 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:08.971+0000 7f502c3051c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:29:09.121 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:09.121+0000 7fa78193b1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:29:09.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.179+0000 7fb6e45ff700 1 Processor -- start 2024-01-11T08:29:09.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.179+0000 7fb6e45ff700 1 -- start start 2024-01-11T08:29:09.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.180+0000 7fb6e45ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6dc157360 0x7fb6dc06f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:09.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.180+0000 7fb6e45ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb6dc157730 0x7fb6dc0cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.180+0000 7fb6e45ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc0cce10 con 0x7fb6dc0b1290 2024-01-11T08:29:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.180+0000 7fb6e45ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6dc0cb540 con 0x7fb6dc157360 2024-01-11T08:29:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.180+0000 7fb6e45ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc0ca960 con 0x7fb6dc157730 2024-01-11T08:29:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.181+0000 7fb6e239b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6dc157360 0x7fb6dc06f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.181+0000 7fb6e2b9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb6dc0b1290 0x7fb6dc15b140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34684/0 (socket says 172.21.15.27:34684) 2024-01-11T08:29:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.181+0000 7fb6e2b9c700 1 -- 172.21.15.27:0/2827567700 learned_addr learned my addr 172.21.15.27:0/2827567700 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.181+0000 7fb6e1b9a700 1 --2- 172.21.15.27:0/2827567700 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb6dc157730 0x7fb6dc0cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.181+0000 7fb6e239b700 1 -- 172.21.15.27:0/2827567700 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb6dc157730 msgr2=0x7fb6dc0cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.181+0000 7fb6e239b700 1 --2- 172.21.15.27:0/2827567700 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb6dc157730 0x7fb6dc0cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.181+0000 7fb6e239b700 1 -- 172.21.15.27:0/2827567700 >> v1:172.21.15.27:6789/0 conn(0x7fb6dc0b1290 legacy=0x7fb6dc15b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.181+0000 7fb6e239b700 1 -- 172.21.15.27:0/2827567700 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc0cac10 con 0x7fb6dc157360 2024-01-11T08:29:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.181+0000 7fb6e1399700 1 -- 172.21.15.27:0/2827567700 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433992507 0 0) 0x7fb6dc0cce10 con 0x7fb6dc0b1290 2024-01-11T08:29:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.181+0000 7fb6e239b700 1 --2- 172.21.15.27:0/2827567700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6dc157360 0x7fb6dc06f570 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80098e0 tx=0x7fb6d80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6808ca1a91f9381 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.182+0000 7fb6e1399700 1 -- 172.21.15.27:0/2827567700 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d800b640 con 0x7fb6dc157360 2024-01-11T08:29:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.182+0000 7fb6e1399700 1 -- 172.21.15.27:0/2827567700 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d800b7a0 con 0x7fb6dc157360 2024-01-11T08:29:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.182+0000 7fb6e1399700 1 -- 172.21.15.27:0/2827567700 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d800bb40 con 0x7fb6dc157360 2024-01-11T08:29:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.182+0000 7fb6e45ff700 1 -- 172.21.15.27:0/2827567700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6dc157360 msgr2=0x7fb6dc06f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.182+0000 7fb6e45ff700 1 --2- 172.21.15.27:0/2827567700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6dc157360 0x7fb6dc06f570 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80098e0 tx=0x7fb6d80091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.182+0000 7fb6e45ff700 1 -- 172.21.15.27:0/2827567700 shutdown_connections 2024-01-11T08:29:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.182+0000 7fb6e45ff700 1 --2- 172.21.15.27:0/2827567700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6dc157360 0x7fb6dc06f570 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.182+0000 7fb6e45ff700 1 --2- 172.21.15.27:0/2827567700 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb6dc157730 0x7fb6dc0cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.182+0000 7fb6e45ff700 1 -- 172.21.15.27:0/2827567700 >> 172.21.15.27:0/2827567700 conn(0x7fb6dc0680b0 msgr2=0x7fb6dc0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.182+0000 7fb6e45ff700 1 -- 172.21.15.27:0/2827567700 shutdown_connections 2024-01-11T08:29:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.183+0000 7fb6e45ff700 1 -- 172.21.15.27:0/2827567700 wait complete. 2024-01-11T08:29:09.184 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:09.183+0000 7f502c3051c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:29:09.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.185+0000 7fb6e45ff700 1 Processor -- start 2024-01-11T08:29:09.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.185+0000 7fb6e45ff700 1 -- start start 2024-01-11T08:29:09.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.186+0000 7fb6e45ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6dc0b1290 0x7fb6dc16a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.186+0000 7fb6e45ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6dc157360 0x7fb6dc159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.186+0000 7fb6e45ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb6dc157730 0x7fb6dc159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.186+0000 7fb6e45ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc15a810 con 0x7fb6dc157360 2024-01-11T08:29:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.186+0000 7fb6e45ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc153f80 con 0x7fb6dc0b1290 2024-01-11T08:29:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.186+0000 7fb6e45ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc1589e0 con 0x7fb6dc157730 2024-01-11T08:29:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.186+0000 7fb6e239b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6dc0b1290 0x7fb6dc16a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.186+0000 7fb6e1b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6dc157360 0x7fb6dc159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.186+0000 7fb6e239b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6dc0b1290 0x7fb6dc16a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:56906/0 (socket says 172.21.15.27:56906) 2024-01-11T08:29:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.187+0000 7fb6e239b700 1 -- 172.21.15.27:0/3597272430 learned_addr learned my addr 172.21.15.27:0/3597272430 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.187+0000 7fb6e2b9c700 1 --2- 172.21.15.27:0/3597272430 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb6dc157730 0x7fb6dc159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:09.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.187+0000 7fb6e1b9a700 1 -- 172.21.15.27:0/3597272430 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb6dc157730 msgr2=0x7fb6dc159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:09.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.187+0000 7fb6e1b9a700 1 --2- 172.21.15.27:0/3597272430 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb6dc157730 0x7fb6dc159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:09.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.187+0000 7fb6e1b9a700 1 -- 172.21.15.27:0/3597272430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6dc0b1290 msgr2=0x7fb6dc16a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:09.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.187+0000 7fb6e1b9a700 1 --2- 172.21.15.27:0/3597272430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6dc0b1290 0x7fb6dc16a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:09.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.187+0000 7fb6e1b9a700 1 -- 172.21.15.27:0/3597272430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d8009420 con 0x7fb6dc157360 2024-01-11T08:29:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.187+0000 7fb6e1b9a700 1 --2- 172.21.15.27:0/3597272430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6dc157360 0x7fb6dc159610 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc00ea20 tx=0x7fb6cc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.188+0000 7fb6d27fc700 1 -- 172.21.15.27:0/3597272430 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc00ee60 con 0x7fb6dc157360 2024-01-11T08:29:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.188+0000 7fb6d27fc700 1 -- 172.21.15.27:0/3597272430 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6cc00c7f0 con 0x7fb6dc157360 2024-01-11T08:29:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.188+0000 7fb6d27fc700 1 -- 172.21.15.27:0/3597272430 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc00cb90 con 0x7fb6dc157360 2024-01-11T08:29:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.188+0000 7fb6e45ff700 1 -- 172.21.15.27:0/3597272430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6dc153930 con 0x7fb6dc157360 2024-01-11T08:29:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.188+0000 7fb6e45ff700 1 -- 172.21.15.27:0/3597272430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6dc16aeb0 con 0x7fb6dc157360 2024-01-11T08:29:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.189+0000 7fb6e45ff700 1 -- 172.21.15.27:0/3597272430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6dc1540f0 con 0x7fb6dc157360 2024-01-11T08:29:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.191+0000 7fb6d27fc700 1 -- 172.21.15.27:0/3597272430 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 64) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fb6cc013470 con 0x7fb6dc157360 2024-01-11T08:29:09.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.191+0000 7fb6d27fc700 1 --2- 172.21.15.27:0/3597272430 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb6c0067170 0x7fb6c0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:09.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.192+0000 7fb6d27fc700 1 -- 172.21.15.27:0/3597272430 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6047+0+0 (secure 0 0 0) 0x7fb6cc084bd0 con 0x7fb6dc157360 2024-01-11T08:29:09.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.192+0000 7fb6e239b700 1 -- 172.21.15.27:0/3597272430 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb6c0067170 msgr2=0x7fb6c0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:09.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.192+0000 7fb6e239b700 1 --2- 172.21.15.27:0/3597272430 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb6c0067170 0x7fb6c0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.196+0000 7fb6d27fc700 1 -- 172.21.15.27:0/3597272430 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb6cc0519c0 con 0x7fb6dc157360 2024-01-11T08:29:09.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.397+0000 7fb6e239b700 1 -- 172.21.15.27:0/3597272430 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb6c0067170 msgr2=0x7fb6c0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:09.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.397+0000 7fb6e239b700 1 --2- 172.21.15.27:0/3597272430 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb6c0067170 0x7fb6c0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.445+0000 7fb6e45ff700 1 -- 172.21.15.27:0/3597272430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb6dc04c3c0 con 0x7fb6dc157360 2024-01-11T08:29:09.444 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-11T08:29:09.445+0000 7f27065f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-11T08:29:09.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.450+0000 7fb6d27fc700 1 -- 172.21.15.27:0/3597272430 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v60) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb6cc054fe0 con 0x7fb6dc157360 2024-01-11T08:29:09.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.453+0000 7fb6e45ff700 1 -- 172.21.15.27:0/3597272430 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb6c0067170 msgr2=0x7fb6c0069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:09.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.453+0000 7fb6e45ff700 1 --2- 172.21.15.27:0/3597272430 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb6c0067170 0x7fb6c0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:09.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.453+0000 7fb6e45ff700 1 -- 172.21.15.27:0/3597272430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6dc157360 msgr2=0x7fb6dc159610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:09.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.453+0000 7fb6e45ff700 1 --2- 172.21.15.27:0/3597272430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6dc157360 0x7fb6dc159610 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc00ea20 tx=0x7fb6cc0091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:09.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.454+0000 7fb6e45ff700 1 -- 172.21.15.27:0/3597272430 shutdown_connections 2024-01-11T08:29:09.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.454+0000 7fb6e45ff700 1 --2- 172.21.15.27:0/3597272430 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb6c0067170 0x7fb6c0069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:09.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.454+0000 7fb6e45ff700 1 --2- 172.21.15.27:0/3597272430 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb6dc157730 0x7fb6dc159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.454+0000 7fb6e45ff700 1 --2- 172.21.15.27:0/3597272430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6dc0b1290 0x7fb6dc16a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.454+0000 7fb6e45ff700 1 --2- 172.21.15.27:0/3597272430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6dc157360 0x7fb6dc159610 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.454+0000 7fb6e45ff700 1 -- 172.21.15.27:0/3597272430 >> 172.21.15.27:0/3597272430 conn(0x7fb6dc0680b0 msgr2=0x7fb6dc151c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.454+0000 7fb6e45ff700 1 -- 172.21.15.27:0/3597272430 shutdown_connections 2024-01-11T08:29:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.454+0000 7fb6e45ff700 1 -- 172.21.15.27:0/3597272430 wait complete. 2024-01-11T08:29:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-11T08:29:09.467 INFO:tasks.ceph.mds.a.smithi027.stderr: -6021> 2024-01-11T08:28:23.930+0000 7f572d7f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T08:29:09.481 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-11T08:29:09.521 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T08:29:09.544 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:09.544+0000 7fa78193b1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:29:09.594 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:09.595+0000 7f502c3051c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:29:09.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.691+0000 7f5ccd9ff700 1 Processor -- start 2024-01-11T08:29:09.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.691+0000 7f5ccd9ff700 1 -- start start 2024-01-11T08:29:09.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.692+0000 7f5ccd9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5cc8157410 0x7f5cc80ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:09.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.692+0000 7f5ccd9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5cc8069540 0x7f5cc80c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:09.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.692+0000 7f5ccd9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cc80699d0 con 0x7f5cc81577e0 2024-01-11T08:29:09.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.692+0000 7f5ccd9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc8069cb0 con 0x7f5cc8069540 2024-01-11T08:29:09.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.692+0000 7f5ccd9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc8069f90 con 0x7f5cc8157410 2024-01-11T08:29:09.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.693+0000 7f5cc77fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5cc8069540 0x7f5cc80c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:09.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.693+0000 7f5cc67fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5cc81577e0 0x7f5cc80ae000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34708/0 (socket says 172.21.15.27:34708) 2024-01-11T08:29:09.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.693+0000 7f5cc67fc700 1 -- 172.21.15.27:0/234501812 learned_addr learned my addr 172.21.15.27:0/234501812 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:09.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.693+0000 7f5cc6ffd700 1 --2- 172.21.15.27:0/234501812 >> v2:172.21.15.77:3300/0 conn(0x7f5cc8157410 0x7f5cc80ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:09.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.693+0000 7f5cc77fe700 1 -- 172.21.15.27:0/234501812 >> v2:172.21.15.77:3300/0 conn(0x7f5cc8157410 msgr2=0x7f5cc80ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:09.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.693+0000 7f5cc77fe700 1 --2- 172.21.15.27:0/234501812 >> v2:172.21.15.77:3300/0 conn(0x7f5cc8157410 0x7f5cc80ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:09.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.693+0000 7f5cc77fe700 1 -- 172.21.15.27:0/234501812 >> v1:172.21.15.27:6789/0 conn(0x7f5cc81577e0 legacy=0x7f5cc80ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:09.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.693+0000 7f5cc77fe700 1 -- 172.21.15.27:0/234501812 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc806a270 con 0x7f5cc8069540 2024-01-11T08:29:09.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.693+0000 7f5cc77fe700 1 --2- 172.21.15.27:0/234501812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc8069540 0x7f5cc80c6ba0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8070c90 tx=0x7f5cc0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d38196b4739cb9c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:09.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.694+0000 7f5cc5ffb700 1 -- 172.21.15.27:0/234501812 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc000e070 con 0x7f5cc8069540 2024-01-11T08:29:09.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.694+0000 7f5cc5ffb700 1 -- 172.21.15.27:0/234501812 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc000bb60 con 0x7f5cc8069540 2024-01-11T08:29:09.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.694+0000 7f5cc5ffb700 1 -- 172.21.15.27:0/234501812 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0012430 con 0x7f5cc8069540 2024-01-11T08:29:09.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.694+0000 7f5ccd9ff700 1 -- 172.21.15.27:0/234501812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc8069540 msgr2=0x7f5cc80c6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:09.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.694+0000 7f5ccd9ff700 1 --2- 172.21.15.27:0/234501812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc8069540 0x7f5cc80c6ba0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8070c90 tx=0x7f5cc0009780 comp rx=0 tx=0).stop 2024-01-11T08:29:09.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.694+0000 7f5ccd9ff700 1 -- 172.21.15.27:0/234501812 shutdown_connections 2024-01-11T08:29:09.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.694+0000 7f5ccd9ff700 1 --2- 172.21.15.27:0/234501812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc8069540 0x7f5cc80c6ba0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:09.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.694+0000 7f5ccd9ff700 1 --2- 172.21.15.27:0/234501812 >> v2:172.21.15.77:3300/0 conn(0x7f5cc8157410 0x7f5cc80ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:09.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.694+0000 7f5ccd9ff700 1 -- 172.21.15.27:0/234501812 >> 172.21.15.27:0/234501812 conn(0x7f5cc80680a0 msgr2=0x7f5cc80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:09.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.695+0000 7f5ccd9ff700 1 -- 172.21.15.27:0/234501812 shutdown_connections 2024-01-11T08:29:09.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.695+0000 7f5ccd9ff700 1 -- 172.21.15.27:0/234501812 wait complete. 2024-01-11T08:29:09.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.696+0000 7f5ccd9ff700 1 Processor -- start 2024-01-11T08:29:09.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.696+0000 7f5ccd9ff700 1 -- start start 2024-01-11T08:29:09.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.697+0000 7f5ccd9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cc8069540 0x7f5cc8155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:09.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.697+0000 7f5ccd9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc8157410 0x7f5cc80c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:09.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.697+0000 7f5ccd9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc81577e0 0x7f5cc80c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:09.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.697+0000 7f5ccd9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc81567e0 con 0x7f5cc8157410 2024-01-11T08:29:09.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.697+0000 7f5ccd9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc80c45b0 con 0x7f5cc81577e0 2024-01-11T08:29:09.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.697+0000 7f5ccd9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc80c3c80 con 0x7f5cc8069540 2024-01-11T08:29:09.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.697+0000 7f5cc67fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc8157410 0x7f5cc80c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:09.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.697+0000 7f5cc67fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc8157410 0x7f5cc80c7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37540/0 (socket says 172.21.15.27:37540) 2024-01-11T08:29:09.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.697+0000 7f5cc67fc700 1 -- 172.21.15.27:0/2619319103 learned_addr learned my addr 172.21.15.27:0/2619319103 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:09.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.697+0000 7f5cc6ffd700 1 --2- 172.21.15.27:0/2619319103 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cc8069540 0x7f5cc8155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:09.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.698+0000 7f5cc77fe700 1 --2- 172.21.15.27:0/2619319103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc81577e0 0x7f5cc80c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:09.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.698+0000 7f5cc67fc700 1 -- 172.21.15.27:0/2619319103 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cc8069540 msgr2=0x7f5cc8155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:09.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.698+0000 7f5cc67fc700 1 --2- 172.21.15.27:0/2619319103 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cc8069540 0x7f5cc8155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:09.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.698+0000 7f5cc67fc700 1 -- 172.21.15.27:0/2619319103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc81577e0 msgr2=0x7f5cc80c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:09.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.698+0000 7f5cc67fc700 1 --2- 172.21.15.27:0/2619319103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc81577e0 0x7f5cc80c7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:09.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.698+0000 7f5cc67fc700 1 -- 172.21.15.27:0/2619319103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc0009a20 con 0x7f5cc8157410 2024-01-11T08:29:09.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.698+0000 7f5cc67fc700 1 --2- 172.21.15.27:0/2619319103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc8157410 0x7f5cc80c7890 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00c790 tx=0x7f5cbc007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:09.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.698+0000 7f5caeffd700 1 -- 172.21.15.27:0/2619319103 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc005b30 con 0x7f5cc8157410 2024-01-11T08:29:09.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.698+0000 7f5caeffd700 1 -- 172.21.15.27:0/2619319103 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cbc005cb0 con 0x7f5cc8157410 2024-01-11T08:29:09.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.698+0000 7f5caeffd700 1 -- 172.21.15.27:0/2619319103 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc004a80 con 0x7f5cc8157410 2024-01-11T08:29:09.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.698+0000 7f5ccd9ff700 1 -- 172.21.15.27:0/2619319103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc80c3f60 con 0x7f5cc8157410 2024-01-11T08:29:09.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.699+0000 7f5ccd9ff700 1 -- 172.21.15.27:0/2619319103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc8153770 con 0x7f5cc8157410 2024-01-11T08:29:09.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.699+0000 7f5ccd9ff700 1 -- 172.21.15.27:0/2619319103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cc80c4720 con 0x7f5cc8157410 2024-01-11T08:29:09.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.701+0000 7f5caeffd700 1 -- 172.21.15.27:0/2619319103 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 64) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f5cbc0070d0 con 0x7f5cc8157410 2024-01-11T08:29:09.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.702+0000 7f5caeffd700 1 --2- 172.21.15.27:0/2619319103 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5cb4067170 0x7f5cb4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:09.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.702+0000 7f5caeffd700 1 -- 172.21.15.27:0/2619319103 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6105+0+0 (secure 0 0 0) 0x7f5cbc084960 con 0x7f5cc8157410 2024-01-11T08:29:09.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.702+0000 7f5cc6ffd700 1 -- 172.21.15.27:0/2619319103 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5cb4067170 msgr2=0x7f5cb4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:09.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.702+0000 7f5cc6ffd700 1 --2- 172.21.15.27:0/2619319103 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5cb4067170 0x7f5cb4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:09.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.705+0000 7f5caeffd700 1 -- 172.21.15.27:0/2619319103 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5cbc051790 con 0x7f5cc8157410 2024-01-11T08:29:09.727 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:09.727+0000 7fa78193b1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:29:09.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.903+0000 7f5cc6ffd700 1 -- 172.21.15.27:0/2619319103 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5cb4067170 msgr2=0x7f5cb4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:09.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.903+0000 7f5cc6ffd700 1 --2- 172.21.15.27:0/2619319103 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5cb4067170 0x7f5cb4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:09.919 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:09.919+0000 7fa78193b1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:29:09.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:09.956+0000 7f5ccd9ff700 1 -- 172.21.15.27:0/2619319103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5cc804c3c0 con 0x7f5cc8157410 2024-01-11T08:29:10.028 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:10.028+0000 7fa78193b1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:29:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.039+0000 7f5caeffd700 1 -- 172.21.15.27:0/2619319103 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 65) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f5cbc051b60 con 0x7f5cc8157410 2024-01-11T08:29:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.040+0000 7f5caeffd700 1 -- 172.21.15.27:0/2619319103 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5cb4067170 msgr2=0x7f5cb4069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.040+0000 7f5caeffd700 1 --2- 172.21.15.27:0/2619319103 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5cb4067170 0x7f5cb4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:10.057 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:10.058+0000 7f502c3051c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:29:10.106 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:10.107+0000 7fa6ff78a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:10.106 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:10.107+0000 7fa6ff78a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:10.106 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:10.107+0000 7fa6ff78a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:10.107 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:10.107+0000 7fa6ff78a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:10.107 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:10.107+0000 7fa6ff78a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:10.108 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:10.108+0000 7fa6fc784700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:10.108 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:10.108+0000 7fa6fc784700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:10.108 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:10.108+0000 7fa6fc784700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:10.108 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:10.108+0000 7fa6fc784700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:10.108 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:10.108+0000 7fa6fc784700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:10.251 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:10.252+0000 7f502c3051c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:29:10.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.451+0000 7f5caeffd700 1 -- 172.21.15.27:0/2619319103 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v61) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5cbc060860 con 0x7f5cc8157410 2024-01-11T08:29:10.450 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:10.452+0000 7f502c3051c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:29:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.454+0000 7f5ccd9ff700 1 -- 172.21.15.27:0/2619319103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc8157410 msgr2=0x7f5cc80c7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.454+0000 7f5ccd9ff700 1 --2- 172.21.15.27:0/2619319103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc8157410 0x7f5cc80c7890 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00c790 tx=0x7f5cbc007f40 comp rx=0 tx=0).stop 2024-01-11T08:29:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.454+0000 7f5ccd9ff700 1 -- 172.21.15.27:0/2619319103 shutdown_connections 2024-01-11T08:29:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.454+0000 7f5ccd9ff700 1 --2- 172.21.15.27:0/2619319103 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5cb4067170 0x7f5cb4069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.454+0000 7f5ccd9ff700 1 --2- 172.21.15.27:0/2619319103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc81577e0 0x7f5cc80c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.454+0000 7f5ccd9ff700 1 --2- 172.21.15.27:0/2619319103 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cc8069540 0x7f5cc8155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:10.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.454+0000 7f5ccd9ff700 1 --2- 172.21.15.27:0/2619319103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc8157410 0x7f5cc80c7890 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:10.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.454+0000 7f5ccd9ff700 1 -- 172.21.15.27:0/2619319103 >> 172.21.15.27:0/2619319103 conn(0x7f5cc80680a0 msgr2=0x7f5cc80c1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:10.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.454+0000 7f5ccd9ff700 1 -- 172.21.15.27:0/2619319103 shutdown_connections 2024-01-11T08:29:10.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.454+0000 7f5ccd9ff700 1 -- 172.21.15.27:0/2619319103 wait complete. 2024-01-11T08:29:10.472 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-11T08:29:10.571 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:10.572+0000 7f502c3051c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:29:10.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.719+0000 7fe0f29ff700 1 Processor -- start 2024-01-11T08:29:10.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.720+0000 7fe0f29ff700 1 -- start start 2024-01-11T08:29:10.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.721+0000 7fe0f29ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe0ec15a7a0 0x7fe0ec15c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:10.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.721+0000 7fe0f29ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0ec0685e0 0x7fe0ec068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:10.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.721+0000 7fe0f29ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ec068f80 con 0x7fe0ec159e60 2024-01-11T08:29:10.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.721+0000 7fe0f29ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0ec15c4b0 con 0x7fe0ec15a7a0 2024-01-11T08:29:10.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.721+0000 7fe0f29ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec15b5d0 con 0x7fe0ec0685e0 2024-01-11T08:29:10.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.721+0000 7fe0eb7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe0ec159e60 0x7fe0ec15aec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34714/0 (socket says 172.21.15.27:34714) 2024-01-11T08:29:10.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.721+0000 7fe0eb7fe700 1 -- 172.21.15.27:0/1159398042 learned_addr learned my addr 172.21.15.27:0/1159398042 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:10.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.721+0000 7fe0f0f9c700 1 --2- 172.21.15.27:0/1159398042 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0ec0685e0 0x7fe0ec068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:10.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.721+0000 7fe0ebfff700 1 --2- 172.21.15.27:0/1159398042 >> v2:172.21.15.77:3300/0 conn(0x7fe0ec15a7a0 0x7fe0ec15c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:10.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.722+0000 7fe0eaffd700 1 -- 172.21.15.27:0/1159398042 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143217710 0 0) 0x7fe0e0001110 con 0x7fe0ec159e60 2024-01-11T08:29:10.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.722+0000 7fe0eaffd700 1 -- 172.21.15.27:0/1159398042 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d4003570 con 0x7fe0ec159e60 2024-01-11T08:29:10.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.722+0000 7fe0f0f9c700 1 -- 172.21.15.27:0/1159398042 >> v2:172.21.15.77:3300/0 conn(0x7fe0ec15a7a0 msgr2=0x7fe0ec15c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:10.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.722+0000 7fe0f0f9c700 1 --2- 172.21.15.27:0/1159398042 >> v2:172.21.15.77:3300/0 conn(0x7fe0ec15a7a0 0x7fe0ec15c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:10.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.722+0000 7fe0f0f9c700 1 -- 172.21.15.27:0/1159398042 >> v1:172.21.15.27:6789/0 conn(0x7fe0ec159e60 legacy=0x7fe0ec15aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:10.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.722+0000 7fe0f0f9c700 1 -- 172.21.15.27:0/1159398042 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ec15b8a0 con 0x7fe0ec0685e0 2024-01-11T08:29:10.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.722+0000 7fe0ebfff700 1 --2- 172.21.15.27:0/1159398042 >> v2:172.21.15.77:3300/0 conn(0x7fe0ec15a7a0 0x7fe0ec15c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:10.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.722+0000 7fe0f0f9c700 1 --2- 172.21.15.27:0/1159398042 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0ec0685e0 0x7fe0ec068a40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4009e60 tx=0x7fe0e4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b8ba6204c61b956 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:10.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.723+0000 7fe0eaffd700 1 -- 172.21.15.27:0/1159398042 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e400e070 con 0x7fe0ec0685e0 2024-01-11T08:29:10.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.723+0000 7fe0eaffd700 1 -- 172.21.15.27:0/1159398042 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e400bab0 con 0x7fe0ec0685e0 2024-01-11T08:29:10.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.723+0000 7fe0eaffd700 1 -- 172.21.15.27:0/1159398042 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e400be70 con 0x7fe0ec0685e0 2024-01-11T08:29:10.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.723+0000 7fe0f29ff700 1 -- 172.21.15.27:0/1159398042 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0ec0685e0 msgr2=0x7fe0ec068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:10.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.723+0000 7fe0f29ff700 1 --2- 172.21.15.27:0/1159398042 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0ec0685e0 0x7fe0ec068a40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4009e60 tx=0x7fe0e4009780 comp rx=0 tx=0).stop 2024-01-11T08:29:10.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.723+0000 7fe0f29ff700 1 -- 172.21.15.27:0/1159398042 shutdown_connections 2024-01-11T08:29:10.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.723+0000 7fe0f29ff700 1 --2- 172.21.15.27:0/1159398042 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0ec0685e0 0x7fe0ec068a40 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:10.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.723+0000 7fe0f29ff700 1 --2- 172.21.15.27:0/1159398042 >> v2:172.21.15.77:3300/0 conn(0x7fe0ec15a7a0 0x7fe0ec15c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:10.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.723+0000 7fe0f29ff700 1 -- 172.21.15.27:0/1159398042 >> 172.21.15.27:0/1159398042 conn(0x7fe0ec06ed40 msgr2=0x7fe0ec06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:10.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.723+0000 7fe0f29ff700 1 -- 172.21.15.27:0/1159398042 shutdown_connections 2024-01-11T08:29:10.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.723+0000 7fe0f29ff700 1 -- 172.21.15.27:0/1159398042 wait complete. 2024-01-11T08:29:10.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.725+0000 7fe0f29ff700 1 Processor -- start 2024-01-11T08:29:10.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.725+0000 7fe0f29ff700 1 -- start start 2024-01-11T08:29:10.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.726+0000 7fe0f29ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0ec0685e0 0x7fe0ec0b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:10.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.726+0000 7fe0f29ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0ec15a7a0 0x7fe0ec1537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:10.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.726+0000 7fe0f29ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0ec153d00 0x7fe0ec16d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:10.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.726+0000 7fe0f29ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec154180 con 0x7fe0ec153d00 2024-01-11T08:29:10.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.726+0000 7fe0f29ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec0b28c0 con 0x7fe0ec15a7a0 2024-01-11T08:29:10.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.726+0000 7fe0f29ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec0b1840 con 0x7fe0ec0685e0 2024-01-11T08:29:10.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.726+0000 7fe0f0f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0ec153d00 0x7fe0ec16d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:10.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.726+0000 7fe0f0f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0ec153d00 0x7fe0ec16d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37556/0 (socket says 172.21.15.27:37556) 2024-01-11T08:29:10.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.726+0000 7fe0f0f9c700 1 -- 172.21.15.27:0/3652471806 learned_addr learned my addr 172.21.15.27:0/3652471806 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:10.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.726+0000 7fe0ebfff700 1 --2- 172.21.15.27:0/3652471806 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0ec0685e0 0x7fe0ec0b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:10.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.727+0000 7fe0eb7fe700 1 --2- 172.21.15.27:0/3652471806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0ec15a7a0 0x7fe0ec1537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:10.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.727+0000 7fe0f0f9c700 1 -- 172.21.15.27:0/3652471806 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0ec0685e0 msgr2=0x7fe0ec0b0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:10.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.727+0000 7fe0f0f9c700 1 --2- 172.21.15.27:0/3652471806 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0ec0685e0 0x7fe0ec0b0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:10.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.727+0000 7fe0f0f9c700 1 -- 172.21.15.27:0/3652471806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0ec15a7a0 msgr2=0x7fe0ec1537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:10.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.727+0000 7fe0f0f9c700 1 --2- 172.21.15.27:0/3652471806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0ec15a7a0 0x7fe0ec1537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:10.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.727+0000 7fe0f0f9c700 1 -- 172.21.15.27:0/3652471806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0dc009420 con 0x7fe0ec153d00 2024-01-11T08:29:10.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.727+0000 7fe0ebfff700 1 --2- 172.21.15.27:0/3652471806 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0ec0685e0 0x7fe0ec0b0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:10.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.727+0000 7fe0eb7fe700 1 --2- 172.21.15.27:0/3652471806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0ec15a7a0 0x7fe0ec1537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:29:10.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.727+0000 7fe0f0f9c700 1 --2- 172.21.15.27:0/3652471806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0ec153d00 0x7fe0ec16d360 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec15b710 tx=0x7fe0e4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:10.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.727+0000 7fe0d3fff700 1 -- 172.21.15.27:0/3652471806 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e4027470 con 0x7fe0ec153d00 2024-01-11T08:29:10.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.727+0000 7fe0d3fff700 1 -- 172.21.15.27:0/3652471806 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e40275f0 con 0x7fe0ec153d00 2024-01-11T08:29:10.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.727+0000 7fe0f29ff700 1 -- 172.21.15.27:0/3652471806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e4009a20 con 0x7fe0ec153d00 2024-01-11T08:29:10.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.727+0000 7fe0d3fff700 1 -- 172.21.15.27:0/3652471806 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e40279b0 con 0x7fe0ec153d00 2024-01-11T08:29:10.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.727+0000 7fe0f29ff700 1 -- 172.21.15.27:0/3652471806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ec0b1e80 con 0x7fe0ec153d00 2024-01-11T08:29:10.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.728+0000 7fe0f29ff700 1 -- 172.21.15.27:0/3652471806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ec04c3c0 con 0x7fe0ec153d00 2024-01-11T08:29:10.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.730+0000 7fe0d3fff700 1 -- 172.21.15.27:0/3652471806 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 65) v1 ==== 84453+0+0 (secure 0 0 0) 0x7fe0e4024070 con 0x7fe0ec153d00 2024-01-11T08:29:10.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.730+0000 7fe0d3fff700 1 -- 172.21.15.27:0/3652471806 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6449+0+0 (secure 0 0 0) 0x7fe0e4086ca0 con 0x7fe0ec153d00 2024-01-11T08:29:10.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.735+0000 7fe0d3fff700 1 -- 172.21.15.27:0/3652471806 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe0e408b070 con 0x7fe0ec153d00 2024-01-11T08:29:10.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:10.943+0000 7fe0f29ff700 1 -- 172.21.15.27:0/3652471806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe0ec15bb80 con 0x7fe0ec153d00 2024-01-11T08:29:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:11.041+0000 7fe0d3fff700 1 -- 172.21.15.27:0/3652471806 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 66) v1 ==== 84580+0+0 (secure 0 0 0) 0x7fe0e4003040 con 0x7fe0ec153d00 2024-01-11T08:29:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:11.042+0000 7fe0d3fff700 1 --2- 172.21.15.27:0/3652471806 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe0d806c7d0 0x7fe0d806ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:11.042+0000 7fe0ebfff700 1 --2- 172.21.15.27:0/3652471806 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe0d806c7d0 0x7fe0d806ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:11.044+0000 7fe0ebfff700 1 --2- 172.21.15.27:0/3652471806 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe0d806c7d0 0x7fe0d806ebb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0093f0 tx=0x7fe0dc008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:11.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:11.843+0000 7fe0d3fff700 1 -- 172.21.15.27:0/3652471806 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v121) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe0e405a930 con 0x7fe0ec153d00 2024-01-11T08:29:11.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:11.848+0000 7fe0f29ff700 1 -- 172.21.15.27:0/3652471806 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe0d806c7d0 msgr2=0x7fe0d806ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:11.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:11.848+0000 7fe0f29ff700 1 --2- 172.21.15.27:0/3652471806 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe0d806c7d0 0x7fe0d806ebb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0093f0 tx=0x7fe0dc008040 comp rx=0 tx=0).stop 2024-01-11T08:29:11.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:11.848+0000 7fe0f29ff700 1 -- 172.21.15.27:0/3652471806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0ec153d00 msgr2=0x7fe0ec16d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:11.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:11.848+0000 7fe0f29ff700 1 --2- 172.21.15.27:0/3652471806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0ec153d00 0x7fe0ec16d360 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec15b710 tx=0x7fe0e4019000 comp rx=0 tx=0).stop 2024-01-11T08:29:11.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:11.848+0000 7fe0f29ff700 1 -- 172.21.15.27:0/3652471806 shutdown_connections 2024-01-11T08:29:11.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:11.848+0000 7fe0f29ff700 1 --2- 172.21.15.27:0/3652471806 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe0d806c7d0 0x7fe0d806ebb0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:11.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:11.848+0000 7fe0f29ff700 1 --2- 172.21.15.27:0/3652471806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0ec153d00 0x7fe0ec16d360 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:11.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:11.848+0000 7fe0f29ff700 1 --2- 172.21.15.27:0/3652471806 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0ec0685e0 0x7fe0ec0b0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:11.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:11.848+0000 7fe0f29ff700 1 --2- 172.21.15.27:0/3652471806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0ec15a7a0 0x7fe0ec1537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:11.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:11.848+0000 7fe0f29ff700 1 -- 172.21.15.27:0/3652471806 >> 172.21.15.27:0/3652471806 conn(0x7fe0ec06ed40 msgr2=0x7fe0ec0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:11.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:11.848+0000 7fe0f29ff700 1 -- 172.21.15.27:0/3652471806 shutdown_connections 2024-01-11T08:29:11.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:11.848+0000 7fe0f29ff700 1 -- 172.21.15.27:0/3652471806 wait complete. 2024-01-11T08:29:11.849 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-01-11T08:29:11.870 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-11T08:29:12.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.086+0000 7f113dbff700 1 Processor -- start 2024-01-11T08:29:12.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.086+0000 7f113dbff700 1 -- start start 2024-01-11T08:29:12.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.087+0000 7f113dbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1138156440 0x7f11380c7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:12.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.087+0000 7f113dbff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1138156810 0x7f113806e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:12.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.087+0000 7f113dbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11380ccac0 0x7f113806ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:12.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.087+0000 7f113dbff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11380c8590 con 0x7f1138156810 2024-01-11T08:29:12.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.087+0000 7f113dbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11380c7980 con 0x7f11380ccac0 2024-01-11T08:29:12.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.087+0000 7f113dbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11380c7c60 con 0x7f1138156440 2024-01-11T08:29:12.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.087+0000 7f1136ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1138156810 0x7f113806e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:12.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.087+0000 7f1136ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1138156810 0x7f113806e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37564/0 (socket says 172.21.15.27:37564) 2024-01-11T08:29:12.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.087+0000 7f1136ffd700 1 -- 172.21.15.27:0/466198150 learned_addr learned my addr 172.21.15.27:0/466198150 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:12.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.087+0000 7f11377fe700 1 --2- 172.21.15.27:0/466198150 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1138156440 0x7f11380c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:12.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.087+0000 7f1137fff700 1 --2- 172.21.15.27:0/466198150 >> v2:172.21.15.77:3300/0 conn(0x7f11380ccac0 0x7f113806ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:12.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.088+0000 7f1136ffd700 1 -- 172.21.15.27:0/466198150 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1138156440 msgr2=0x7f11380c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:12.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.088+0000 7f1136ffd700 1 --2- 172.21.15.27:0/466198150 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1138156440 0x7f11380c7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:12.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.088+0000 7f1136ffd700 1 -- 172.21.15.27:0/466198150 >> v2:172.21.15.77:3300/0 conn(0x7f11380ccac0 msgr2=0x7f113806ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:12.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.088+0000 7f1136ffd700 1 --2- 172.21.15.27:0/466198150 >> v2:172.21.15.77:3300/0 conn(0x7f11380ccac0 0x7f113806ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:12.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.088+0000 7f1136ffd700 1 -- 172.21.15.27:0/466198150 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11380c7f40 con 0x7f1138156810 2024-01-11T08:29:12.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.088+0000 7f1136ffd700 1 --2- 172.21.15.27:0/466198150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1138156810 0x7f113806e720 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f11380ab8c0 tx=0x7f112c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edb637f0f55109a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:12.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.088+0000 7f11367fc700 1 -- 172.21.15.27:0/466198150 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c00b640 con 0x7f1138156810 2024-01-11T08:29:12.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.088+0000 7f11367fc700 1 -- 172.21.15.27:0/466198150 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c00b7a0 con 0x7f1138156810 2024-01-11T08:29:12.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.088+0000 7f11367fc700 1 -- 172.21.15.27:0/466198150 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c00bb60 con 0x7f1138156810 2024-01-11T08:29:12.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.089+0000 7f113dbff700 1 -- 172.21.15.27:0/466198150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1138156810 msgr2=0x7f113806e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:12.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.089+0000 7f113dbff700 1 --2- 172.21.15.27:0/466198150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1138156810 0x7f113806e720 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f11380ab8c0 tx=0x7f112c009250 comp rx=0 tx=0).stop 2024-01-11T08:29:12.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.089+0000 7f113dbff700 1 -- 172.21.15.27:0/466198150 shutdown_connections 2024-01-11T08:29:12.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.089+0000 7f113dbff700 1 --2- 172.21.15.27:0/466198150 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1138156440 0x7f11380c7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:12.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.089+0000 7f113dbff700 1 --2- 172.21.15.27:0/466198150 >> v2:172.21.15.77:3300/0 conn(0x7f11380ccac0 0x7f113806ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:12.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.089+0000 7f113dbff700 1 --2- 172.21.15.27:0/466198150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1138156810 0x7f113806e720 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:12.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.089+0000 7f113dbff700 1 -- 172.21.15.27:0/466198150 >> 172.21.15.27:0/466198150 conn(0x7f11380680e0 msgr2=0x7f11380cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:12.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.089+0000 7f113dbff700 1 -- 172.21.15.27:0/466198150 shutdown_connections 2024-01-11T08:29:12.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.089+0000 7f113dbff700 1 -- 172.21.15.27:0/466198150 wait complete. 2024-01-11T08:29:12.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.091+0000 7f113dbff700 1 Processor -- start 2024-01-11T08:29:12.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.091+0000 7f113dbff700 1 -- start start 2024-01-11T08:29:12.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.092+0000 7f113dbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11380ccac0 0x7f1138153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:12.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.092+0000 7f113dbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1138156440 0x7f113806e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:12.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.092+0000 7f113dbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1138156810 0x7f113806e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:12.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.092+0000 7f113dbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11380c3fc0 con 0x7f1138156440 2024-01-11T08:29:12.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.092+0000 7f113dbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11380c42a0 con 0x7f1138156810 2024-01-11T08:29:12.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.092+0000 7f113dbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11380c4580 con 0x7f11380ccac0 2024-01-11T08:29:12.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.092+0000 7f1136ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1138156440 0x7f113806e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:12.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.092+0000 7f1136ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1138156440 0x7f113806e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37580/0 (socket says 172.21.15.27:37580) 2024-01-11T08:29:12.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.092+0000 7f1136ffd700 1 -- 172.21.15.27:0/1105114944 learned_addr learned my addr 172.21.15.27:0/1105114944 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:12.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.092+0000 7f11377fe700 1 --2- 172.21.15.27:0/1105114944 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11380ccac0 0x7f1138153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:12.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.092+0000 7f1137fff700 1 --2- 172.21.15.27:0/1105114944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1138156810 0x7f113806e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:12.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.092+0000 7f1136ffd700 1 -- 172.21.15.27:0/1105114944 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11380ccac0 msgr2=0x7f1138153a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:12.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.092+0000 7f1136ffd700 1 --2- 172.21.15.27:0/1105114944 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11380ccac0 0x7f1138153a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:12.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.092+0000 7f1136ffd700 1 -- 172.21.15.27:0/1105114944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1138156810 msgr2=0x7f113806e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:12.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.092+0000 7f1136ffd700 1 --2- 172.21.15.27:0/1105114944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1138156810 0x7f113806e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:12.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.092+0000 7f1136ffd700 1 -- 172.21.15.27:0/1105114944 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c0094f0 con 0x7f1138156440 2024-01-11T08:29:12.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.093+0000 7f1136ffd700 1 --2- 172.21.15.27:0/1105114944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1138156440 0x7f113806e3c0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f1138069380 tx=0x7f112c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:12.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.093+0000 7f111f7fe700 1 -- 172.21.15.27:0/1105114944 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c00b910 con 0x7f1138156440 2024-01-11T08:29:12.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.093+0000 7f111f7fe700 1 -- 172.21.15.27:0/1105114944 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c00ba70 con 0x7f1138156440 2024-01-11T08:29:12.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.093+0000 7f111f7fe700 1 -- 172.21.15.27:0/1105114944 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c011630 con 0x7f1138156440 2024-01-11T08:29:12.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.093+0000 7f113dbff700 1 -- 172.21.15.27:0/1105114944 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11380c4860 con 0x7f1138156440 2024-01-11T08:29:12.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.093+0000 7f113dbff700 1 -- 172.21.15.27:0/1105114944 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1138153f90 con 0x7f1138156440 2024-01-11T08:29:12.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.094+0000 7f113dbff700 1 -- 172.21.15.27:0/1105114944 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113804c3c0 con 0x7f1138156440 2024-01-11T08:29:12.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.095+0000 7f111f7fe700 1 -- 172.21.15.27:0/1105114944 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 66) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f112c0119a0 con 0x7f1138156440 2024-01-11T08:29:12.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.096+0000 7f111f7fe700 1 --2- 172.21.15.27:0/1105114944 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1124066d80 0x7f1124069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:12.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.096+0000 7f111f7fe700 1 -- 172.21.15.27:0/1105114944 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6011+0+0 (secure 0 0 0) 0x7f112c0859e0 con 0x7f1138156440 2024-01-11T08:29:12.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.097+0000 7f11377fe700 1 --2- 172.21.15.27:0/1105114944 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1124066d80 0x7f1124069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:12.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.097+0000 7f11377fe700 1 --2- 172.21.15.27:0/1105114944 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1124066d80 0x7f1124069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1128009da0 tx=0x7f1128009120 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:12.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.101+0000 7f111f7fe700 1 -- 172.21.15.27:0/1105114944 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f112c052d00 con 0x7f1138156440 2024-01-11T08:29:12.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.307+0000 7f113dbff700 1 -- 172.21.15.27:0/1105114944 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f11380b09e0 con 0x7f1138156440 2024-01-11T08:29:12.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:12.848+0000 7f111f7fe700 1 -- 172.21.15.27:0/1105114944 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 67) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f112c0530d0 con 0x7f1138156440 2024-01-11T08:29:13.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.049+0000 7f111f7fe700 1 -- 172.21.15.27:0/1105114944 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v122) v1 ==== 169+0+0 (secure 0 0 0) 0x7f112c056790 con 0x7f1138156440 2024-01-11T08:29:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.054+0000 7f113dbff700 1 -- 172.21.15.27:0/1105114944 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1124066d80 msgr2=0x7f1124069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.054+0000 7f113dbff700 1 --2- 172.21.15.27:0/1105114944 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1124066d80 0x7f1124069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1128009da0 tx=0x7f1128009120 comp rx=0 tx=0).stop 2024-01-11T08:29:13.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.054+0000 7f113dbff700 1 -- 172.21.15.27:0/1105114944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1138156440 msgr2=0x7f113806e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:13.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.054+0000 7f113dbff700 1 --2- 172.21.15.27:0/1105114944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1138156440 0x7f113806e3c0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f1138069380 tx=0x7f112c017000 comp rx=0 tx=0).stop 2024-01-11T08:29:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.056+0000 7f113dbff700 1 -- 172.21.15.27:0/1105114944 shutdown_connections 2024-01-11T08:29:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.056+0000 7f113dbff700 1 --2- 172.21.15.27:0/1105114944 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1124066d80 0x7f1124069230 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.056+0000 7f113dbff700 1 --2- 172.21.15.27:0/1105114944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1138156810 0x7f113806e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.056+0000 7f113dbff700 1 --2- 172.21.15.27:0/1105114944 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11380ccac0 0x7f1138153a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.056+0000 7f113dbff700 1 --2- 172.21.15.27:0/1105114944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1138156440 0x7f113806e3c0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.056+0000 7f113dbff700 1 -- 172.21.15.27:0/1105114944 >> 172.21.15.27:0/1105114944 conn(0x7f11380680e0 msgr2=0x7f11380aacb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:13.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.057+0000 7f113dbff700 1 -- 172.21.15.27:0/1105114944 shutdown_connections 2024-01-11T08:29:13.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.057+0000 7f113dbff700 1 -- 172.21.15.27:0/1105114944 wait complete. 2024-01-11T08:29:13.059 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-01-11T08:29:13.080 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-11T08:29:13.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.293+0000 7f08dc5ff700 1 Processor -- start 2024-01-11T08:29:13.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.293+0000 7f08dc5ff700 1 -- start start 2024-01-11T08:29:13.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.295+0000 7f08dc5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f08d4157750 0x7f08d40af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:13.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.295+0000 7f08dc5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08d4068620 0x7f08d4068aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:13.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.295+0000 7f08dc5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d40cb3f0 con 0x7f08d4157380 2024-01-11T08:29:13.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.295+0000 7f08dc5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08d4068fe0 con 0x7f08d4157750 2024-01-11T08:29:13.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.295+0000 7f08dc5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d40afdf0 con 0x7f08d4068620 2024-01-11T08:29:13.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.295+0000 7f08d9b9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f08d4157750 0x7f08d40af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:13.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.295+0000 7f08d9b9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f08d4157750 0x7f08d40af8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37592/0 (socket says 172.21.15.27:37592) 2024-01-11T08:29:13.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.295+0000 7f08d9b9a700 1 -- 172.21.15.27:0/2871630274 learned_addr learned my addr 172.21.15.27:0/2871630274 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:13.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.295+0000 7f08dab9c700 1 --2- 172.21.15.27:0/2871630274 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08d4068620 0x7f08d4068aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:13.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.295+0000 7f08d9b9a700 1 -- 172.21.15.27:0/2871630274 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08d4068620 msgr2=0x7f08d4068aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:13.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.295+0000 7f08d9b9a700 1 --2- 172.21.15.27:0/2871630274 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08d4068620 0x7f08d4068aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.295+0000 7f08d9b9a700 1 -- 172.21.15.27:0/2871630274 >> v1:172.21.15.77:6789/0 conn(0x7f08d4157380 legacy=0x7f08d406e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:13.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.295+0000 7f08d9b9a700 1 -- 172.21.15.27:0/2871630274 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d406ef70 con 0x7f08d4157750 2024-01-11T08:29:13.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.295+0000 7f08dab9c700 1 --2- 172.21.15.27:0/2871630274 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08d4068620 0x7f08d4068aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:13.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.296+0000 7f08d9b9a700 1 --2- 172.21.15.27:0/2871630274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d4157750 0x7f08d40af8b0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f08c40099b0 tx=0x7f08c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c88acb34ffa0f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:13.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.296+0000 7f08d9399700 1 -- 172.21.15.27:0/2871630274 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c400b860 con 0x7f08d4157750 2024-01-11T08:29:13.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.296+0000 7f08d9399700 1 -- 172.21.15.27:0/2871630274 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c400b9c0 con 0x7f08d4157750 2024-01-11T08:29:13.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.296+0000 7f08d9399700 1 -- 172.21.15.27:0/2871630274 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c400bd60 con 0x7f08d4157750 2024-01-11T08:29:13.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.297+0000 7f08dc5ff700 1 -- 172.21.15.27:0/2871630274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d4157750 msgr2=0x7f08d40af8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:13.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.297+0000 7f08dc5ff700 1 --2- 172.21.15.27:0/2871630274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d4157750 0x7f08d40af8b0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f08c40099b0 tx=0x7f08c4009250 comp rx=0 tx=0).stop 2024-01-11T08:29:13.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.297+0000 7f08dc5ff700 1 -- 172.21.15.27:0/2871630274 shutdown_connections 2024-01-11T08:29:13.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.297+0000 7f08dc5ff700 1 --2- 172.21.15.27:0/2871630274 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08d4068620 0x7f08d4068aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.297+0000 7f08dc5ff700 1 --2- 172.21.15.27:0/2871630274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d4157750 0x7f08d40af8b0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.297+0000 7f08dc5ff700 1 -- 172.21.15.27:0/2871630274 >> 172.21.15.27:0/2871630274 conn(0x7f08d40c9fb0 msgr2=0x7f08d406ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:13.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.297+0000 7f08dc5ff700 1 -- 172.21.15.27:0/2871630274 shutdown_connections 2024-01-11T08:29:13.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.297+0000 7f08dc5ff700 1 -- 172.21.15.27:0/2871630274 wait complete. 2024-01-11T08:29:13.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.298+0000 7f08dc5ff700 1 Processor -- start 2024-01-11T08:29:13.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.299+0000 7f08dc5ff700 1 -- start start 2024-01-11T08:29:13.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.299+0000 7f08dc5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d4068620 0x7f08d40ab7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:13.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.299+0000 7f08dc5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08d4157380 0x7f08d41557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:13.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.299+0000 7f08dc5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08d4157750 0x7f08d40aee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:13.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.300+0000 7f08dc5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d40c7f30 con 0x7f08d4068620 2024-01-11T08:29:13.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.300+0000 7f08dc5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d4155d00 con 0x7f08d4157380 2024-01-11T08:29:13.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.300+0000 7f08dc5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d40af3c0 con 0x7f08d4157750 2024-01-11T08:29:13.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.300+0000 7f08da39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d4068620 0x7f08d40ab7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:13.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.300+0000 7f08da39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d4068620 0x7f08d40ab7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37594/0 (socket says 172.21.15.27:37594) 2024-01-11T08:29:13.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.300+0000 7f08da39b700 1 -- 172.21.15.27:0/1358796361 learned_addr learned my addr 172.21.15.27:0/1358796361 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:13.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.300+0000 7f08da39b700 1 -- 172.21.15.27:0/1358796361 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08d4157750 msgr2=0x7f08d40aee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:13.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.300+0000 7f08da39b700 1 --2- 172.21.15.27:0/1358796361 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08d4157750 0x7f08d40aee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.300+0000 7f08da39b700 1 -- 172.21.15.27:0/1358796361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08d4157380 msgr2=0x7f08d41557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:13.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.300+0000 7f08d9b9a700 1 --2- 172.21.15.27:0/1358796361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08d4157380 0x7f08d41557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:13.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.300+0000 7f08da39b700 1 --2- 172.21.15.27:0/1358796361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08d4157380 0x7f08d41557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.300+0000 7f08da39b700 1 -- 172.21.15.27:0/1358796361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c40094f0 con 0x7f08d4068620 2024-01-11T08:29:13.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.300+0000 7f08d9b9a700 1 --2- 172.21.15.27:0/1358796361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08d4157380 0x7f08d41557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:29:13.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.300+0000 7f08da39b700 1 --2- 172.21.15.27:0/1358796361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d4068620 0x7f08d40ab7a0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f08d000c590 tx=0x7f08d000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:13.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.301+0000 7f08ca7fc700 1 -- 172.21.15.27:0/1358796361 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d0011590 con 0x7f08d4068620 2024-01-11T08:29:13.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.301+0000 7f08ca7fc700 1 -- 172.21.15.27:0/1358796361 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d00116f0 con 0x7f08d4068620 2024-01-11T08:29:13.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.301+0000 7f08ca7fc700 1 -- 172.21.15.27:0/1358796361 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d0011ab0 con 0x7f08d4068620 2024-01-11T08:29:13.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.301+0000 7f08dc5ff700 1 -- 172.21.15.27:0/1358796361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d40c3cc0 con 0x7f08d4068620 2024-01-11T08:29:13.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.301+0000 7f08dc5ff700 1 -- 172.21.15.27:0/1358796361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d40c78e0 con 0x7f08d4068620 2024-01-11T08:29:13.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.302+0000 7f08dc5ff700 1 -- 172.21.15.27:0/1358796361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d404c3c0 con 0x7f08d4068620 2024-01-11T08:29:13.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.303+0000 7f08ca7fc700 1 -- 172.21.15.27:0/1358796361 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 67) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f08d001b3b0 con 0x7f08d4068620 2024-01-11T08:29:13.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.304+0000 7f08ca7fc700 1 --2- 172.21.15.27:0/1358796361 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f08b8066ea0 0x7f08b8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:13.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.304+0000 7f08ca7fc700 1 -- 172.21.15.27:0/1358796361 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5597+0+0 (secure 0 0 0) 0x7f08d00839f0 con 0x7f08d4068620 2024-01-11T08:29:13.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.304+0000 7f08d9b9a700 1 --2- 172.21.15.27:0/1358796361 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f08b8066ea0 0x7f08b8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:13.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.305+0000 7f08d9b9a700 1 --2- 172.21.15.27:0/1358796361 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f08b8066ea0 0x7f08b8069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f08c40099b0 tx=0x7f08c4017000 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.308+0000 7f08ca7fc700 1 -- 172.21.15.27:0/1358796361 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f08d0050d10 con 0x7f08d4068620 2024-01-11T08:29:13.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.512+0000 7f08dc5ff700 1 -- 172.21.15.27:0/1358796361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f08d40ca7e0 con 0x7f08d4068620 2024-01-11T08:29:13.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.516+0000 7f08ca7fc700 1 -- 172.21.15.27:0/1358796361 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v43) v1 ==== 120+0+0 (secure 0 0 0) 0x7f08d00170b0 con 0x7f08d4068620 2024-01-11T08:29:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.519+0000 7f08dc5ff700 1 -- 172.21.15.27:0/1358796361 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f08b8066ea0 msgr2=0x7f08b8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.519+0000 7f08dc5ff700 1 --2- 172.21.15.27:0/1358796361 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f08b8066ea0 0x7f08b8069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f08c40099b0 tx=0x7f08c4017000 comp rx=0 tx=0).stop 2024-01-11T08:29:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.519+0000 7f08dc5ff700 1 -- 172.21.15.27:0/1358796361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d4068620 msgr2=0x7f08d40ab7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.519+0000 7f08dc5ff700 1 --2- 172.21.15.27:0/1358796361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d4068620 0x7f08d40ab7a0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f08d000c590 tx=0x7f08d000c950 comp rx=0 tx=0).stop 2024-01-11T08:29:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.519+0000 7f08dc5ff700 1 -- 172.21.15.27:0/1358796361 shutdown_connections 2024-01-11T08:29:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.519+0000 7f08dc5ff700 1 --2- 172.21.15.27:0/1358796361 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f08b8066ea0 0x7f08b8069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.519+0000 7f08dc5ff700 1 --2- 172.21.15.27:0/1358796361 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08d4157750 0x7f08d40aee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.519+0000 7f08dc5ff700 1 --2- 172.21.15.27:0/1358796361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d4068620 0x7f08d40ab7a0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.519+0000 7f08dc5ff700 1 --2- 172.21.15.27:0/1358796361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08d4157380 0x7f08d41557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.519+0000 7f08dc5ff700 1 -- 172.21.15.27:0/1358796361 >> 172.21.15.27:0/1358796361 conn(0x7f08d40c9fb0 msgr2=0x7f08d40b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:13.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.519+0000 7f08dc5ff700 1 -- 172.21.15.27:0/1358796361 shutdown_connections 2024-01-11T08:29:13.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.519+0000 7f08dc5ff700 1 -- 172.21.15.27:0/1358796361 wait complete. 2024-01-11T08:29:13.536 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-11T08:29:13.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.740+0000 7f62367ff700 1 Processor -- start 2024-01-11T08:29:13.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.741+0000 7f62367ff700 1 -- start start 2024-01-11T08:29:13.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.742+0000 7f62367ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6230157410 0x7f62300ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:13.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.742+0000 7f62367ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6230069540 0x7f62300c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:13.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.742+0000 7f62367ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62300699d0 con 0x7f62301577e0 2024-01-11T08:29:13.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.742+0000 7f62367ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6230069cb0 con 0x7f6230157410 2024-01-11T08:29:13.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.742+0000 7f62367ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6230069f90 con 0x7f6230069540 2024-01-11T08:29:13.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.742+0000 7f622ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6230157410 0x7f62300ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:13.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.742+0000 7f622ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6230157410 0x7f62300ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37606/0 (socket says 172.21.15.27:37606) 2024-01-11T08:29:13.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.742+0000 7f622ffff700 1 -- 172.21.15.27:0/2556620986 learned_addr learned my addr 172.21.15.27:0/2556620986 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:13.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.742+0000 7f622f7fe700 1 --1- 172.21.15.27:0/2556620986 >> v1:172.21.15.27:6789/0 conn(0x7f62301577e0 0x7f62300ae000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34724/0 (socket says 172.21.15.27:34724) 2024-01-11T08:29:13.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.742+0000 7f6234d9c700 1 --2- 172.21.15.27:0/2556620986 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6230069540 0x7f62300c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:13.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.743+0000 7f622ffff700 1 -- 172.21.15.27:0/2556620986 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6230069540 msgr2=0x7f62300c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:13.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.743+0000 7f622ffff700 1 --2- 172.21.15.27:0/2556620986 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6230069540 0x7f62300c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.743+0000 7f622ffff700 1 -- 172.21.15.27:0/2556620986 >> v1:172.21.15.27:6789/0 conn(0x7f62301577e0 legacy=0x7f62300ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:13.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.743+0000 7f622ffff700 1 -- 172.21.15.27:0/2556620986 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623006a270 con 0x7f6230157410 2024-01-11T08:29:13.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.743+0000 7f6234d9c700 1 --2- 172.21.15.27:0/2556620986 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6230069540 0x7f62300c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:13.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.743+0000 7f622ffff700 1 --2- 172.21.15.27:0/2556620986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6230157410 0x7f62300ccae0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f6220009990 tx=0x7f6220009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d18ca815905c62 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:13.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.743+0000 7f622effd700 1 -- 172.21.15.27:0/2556620986 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6220005c10 con 0x7f6230157410 2024-01-11T08:29:13.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.743+0000 7f622effd700 1 -- 172.21.15.27:0/2556620986 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6220005d70 con 0x7f6230157410 2024-01-11T08:29:13.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.743+0000 7f622effd700 1 -- 172.21.15.27:0/2556620986 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622000b880 con 0x7f6230157410 2024-01-11T08:29:13.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.744+0000 7f62367ff700 1 -- 172.21.15.27:0/2556620986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6230157410 msgr2=0x7f62300ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:13.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.744+0000 7f62367ff700 1 --2- 172.21.15.27:0/2556620986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6230157410 0x7f62300ccae0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f6220009990 tx=0x7f6220009220 comp rx=0 tx=0).stop 2024-01-11T08:29:13.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.744+0000 7f62367ff700 1 -- 172.21.15.27:0/2556620986 shutdown_connections 2024-01-11T08:29:13.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.744+0000 7f62367ff700 1 --2- 172.21.15.27:0/2556620986 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6230069540 0x7f62300c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.744+0000 7f62367ff700 1 --2- 172.21.15.27:0/2556620986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6230157410 0x7f62300ccae0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.744+0000 7f62367ff700 1 -- 172.21.15.27:0/2556620986 >> 172.21.15.27:0/2556620986 conn(0x7f62300680a0 msgr2=0x7f62300cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:13.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.744+0000 7f62367ff700 1 -- 172.21.15.27:0/2556620986 shutdown_connections 2024-01-11T08:29:13.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.744+0000 7f62367ff700 1 -- 172.21.15.27:0/2556620986 wait complete. 2024-01-11T08:29:13.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.746+0000 7f62367ff700 1 Processor -- start 2024-01-11T08:29:13.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.746+0000 7f62367ff700 1 -- start start 2024-01-11T08:29:13.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.747+0000 7f62367ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6230069540 0x7f6230153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:13.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.747+0000 7f62367ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6230157410 0x7f6230158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:13.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.747+0000 7f62367ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62301577e0 0x7f6230158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:13.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.747+0000 7f62367ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62300c4160 con 0x7f6230069540 2024-01-11T08:29:13.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.747+0000 7f62367ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6230169990 con 0x7f62301577e0 2024-01-11T08:29:13.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.747+0000 7f62367ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f623016a4d0 con 0x7f6230157410 2024-01-11T08:29:13.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.747+0000 7f622ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6230069540 0x7f6230153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:13.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.747+0000 7f622ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6230069540 0x7f6230153c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37614/0 (socket says 172.21.15.27:37614) 2024-01-11T08:29:13.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.747+0000 7f622ffff700 1 -- 172.21.15.27:0/3886952735 learned_addr learned my addr 172.21.15.27:0/3886952735 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:13.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.747+0000 7f622ffff700 1 -- 172.21.15.27:0/3886952735 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6230157410 msgr2=0x7f6230158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:13.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.747+0000 7f622ffff700 1 --2- 172.21.15.27:0/3886952735 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6230157410 0x7f6230158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.747+0000 7f622ffff700 1 -- 172.21.15.27:0/3886952735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62301577e0 msgr2=0x7f6230158960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:13.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.747+0000 7f622ffff700 1 --2- 172.21.15.27:0/3886952735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62301577e0 0x7f6230158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.747+0000 7f622ffff700 1 -- 172.21.15.27:0/3886952735 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6220009480 con 0x7f6230069540 2024-01-11T08:29:13.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.748+0000 7f622ffff700 1 --2- 172.21.15.27:0/3886952735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6230069540 0x7f6230153c30 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f6220005d40 tx=0x7f6220016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:13.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.748+0000 7f6217fff700 1 -- 172.21.15.27:0/3886952735 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6220011a60 con 0x7f6230069540 2024-01-11T08:29:13.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.748+0000 7f6217fff700 1 -- 172.21.15.27:0/3886952735 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6220011bc0 con 0x7f6230069540 2024-01-11T08:29:13.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.748+0000 7f6217fff700 1 -- 172.21.15.27:0/3886952735 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622001b430 con 0x7f6230069540 2024-01-11T08:29:13.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.748+0000 7f62367ff700 1 -- 172.21.15.27:0/3886952735 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6230169ba0 con 0x7f6230069540 2024-01-11T08:29:13.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.748+0000 7f62367ff700 1 -- 172.21.15.27:0/3886952735 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f623016a160 con 0x7f6230069540 2024-01-11T08:29:13.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.749+0000 7f62367ff700 1 -- 172.21.15.27:0/3886952735 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f623004c3c0 con 0x7f6230069540 2024-01-11T08:29:13.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.750+0000 7f6217fff700 1 -- 172.21.15.27:0/3886952735 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 67) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f622001b860 con 0x7f6230069540 2024-01-11T08:29:13.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.751+0000 7f6217fff700 1 --2- 172.21.15.27:0/3886952735 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f621c066ea0 0x7f621c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:13.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.751+0000 7f6217fff700 1 -- 172.21.15.27:0/3886952735 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5597+0+0 (secure 0 0 0) 0x7f6220085e00 con 0x7f6230069540 2024-01-11T08:29:13.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.751+0000 7f622f7fe700 1 --2- 172.21.15.27:0/3886952735 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f621c066ea0 0x7f621c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:13.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.752+0000 7f622f7fe700 1 --2- 172.21.15.27:0/3886952735 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f621c066ea0 0x7f621c069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6230159200 tx=0x7f6224007b30 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:13.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.755+0000 7f6217fff700 1 -- 172.21.15.27:0/3886952735 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f622008a070 con 0x7f6230069540 2024-01-11T08:29:13.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.960+0000 7f62367ff700 1 -- 172.21.15.27:0/3886952735 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f623016a860 con 0x7f6230069540 2024-01-11T08:29:13.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.966+0000 7f6217fff700 1 -- 172.21.15.27:0/3886952735 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v44) v1 ==== 115+0+0 (secure 0 0 0) 0x7f622002a0b0 con 0x7f6230069540 2024-01-11T08:29:13.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.968+0000 7f62367ff700 1 -- 172.21.15.27:0/3886952735 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f621c066ea0 msgr2=0x7f621c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:13.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.968+0000 7f62367ff700 1 --2- 172.21.15.27:0/3886952735 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f621c066ea0 0x7f621c069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6230159200 tx=0x7f6224007b30 comp rx=0 tx=0).stop 2024-01-11T08:29:13.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.968+0000 7f62367ff700 1 -- 172.21.15.27:0/3886952735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6230069540 msgr2=0x7f6230153c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:13.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.968+0000 7f62367ff700 1 --2- 172.21.15.27:0/3886952735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6230069540 0x7f6230153c30 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f6220005d40 tx=0x7f6220016000 comp rx=0 tx=0).stop 2024-01-11T08:29:13.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.968+0000 7f62367ff700 1 -- 172.21.15.27:0/3886952735 shutdown_connections 2024-01-11T08:29:13.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.968+0000 7f62367ff700 1 --2- 172.21.15.27:0/3886952735 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f621c066ea0 0x7f621c069350 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6230159200 tx=0x7f6224007b30 comp rx=0 tx=0).stop 2024-01-11T08:29:13.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.968+0000 7f62367ff700 1 --2- 172.21.15.27:0/3886952735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62301577e0 0x7f6230158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.968+0000 7f62367ff700 1 --2- 172.21.15.27:0/3886952735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6230069540 0x7f6230153c30 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.968+0000 7f62367ff700 1 --2- 172.21.15.27:0/3886952735 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6230157410 0x7f6230158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:13.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.968+0000 7f62367ff700 1 -- 172.21.15.27:0/3886952735 >> 172.21.15.27:0/3886952735 conn(0x7f62300680a0 msgr2=0x7f62300b1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:13.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.968+0000 7f62367ff700 1 -- 172.21.15.27:0/3886952735 shutdown_connections 2024-01-11T08:29:13.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:13.968+0000 7f62367ff700 1 -- 172.21.15.27:0/3886952735 wait complete. 2024-01-11T08:29:13.985 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-11T08:29:14.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.187+0000 7fe79b9ff700 1 Processor -- start 2024-01-11T08:29:14.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.187+0000 7fe79b9ff700 1 -- start start 2024-01-11T08:29:14.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.189+0000 7fe79b9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe794155c50 0x7fe7940aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:14.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.189+0000 7fe79b9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe7940694c0 0x7fe794069910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:14.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.189+0000 7fe79b9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79406e480 con 0x7fe7940c6de0 2024-01-11T08:29:14.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.189+0000 7fe79b9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7940cb5a0 con 0x7fe7940694c0 2024-01-11T08:29:14.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.189+0000 7fe79b9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe79406df00 con 0x7fe794155c50 2024-01-11T08:29:14.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.189+0000 7fe799f9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe7940c6de0 0x7fe7940c7260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51148/0 (socket says 172.21.15.27:51148) 2024-01-11T08:29:14.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.189+0000 7fe799f9c700 1 -- 172.21.15.27:0/1672945305 learned_addr learned my addr 172.21.15.27:0/1672945305 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:14.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.189+0000 7fe798f9a700 1 --2- 172.21.15.27:0/1672945305 >> v2:172.21.15.77:3300/0 conn(0x7fe7940694c0 0x7fe794069910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:14.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.189+0000 7fe79979b700 1 --2- 172.21.15.27:0/1672945305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe794155c50 0x7fe7940aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:14.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.190+0000 7fe78bfff700 1 -- 172.21.15.27:0/1672945305 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198107350 0 0) 0x7fe79406e480 con 0x7fe7940c6de0 2024-01-11T08:29:14.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.190+0000 7fe78bfff700 1 -- 172.21.15.27:0/1672945305 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe780003570 con 0x7fe7940c6de0 2024-01-11T08:29:14.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.190+0000 7fe798f9a700 1 -- 172.21.15.27:0/1672945305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe794155c50 msgr2=0x7fe7940aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:14.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.190+0000 7fe798f9a700 1 --2- 172.21.15.27:0/1672945305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe794155c50 0x7fe7940aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:14.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.190+0000 7fe798f9a700 1 -- 172.21.15.27:0/1672945305 >> v1:172.21.15.27:6789/0 conn(0x7fe7940c6de0 legacy=0x7fe7940c7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:14.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.190+0000 7fe798f9a700 1 -- 172.21.15.27:0/1672945305 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe794069e50 con 0x7fe7940694c0 2024-01-11T08:29:14.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.190+0000 7fe798f9a700 1 --2- 172.21.15.27:0/1672945305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7940694c0 0x7fe794069910 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe7840098e0 tx=0x7fe7840091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9de2f967493e7bea server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:14.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.190+0000 7fe78bfff700 1 -- 172.21.15.27:0/1672945305 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe784011630 con 0x7fe7940694c0 2024-01-11T08:29:14.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.191+0000 7fe78bfff700 1 -- 172.21.15.27:0/1672945305 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe784011790 con 0x7fe7940694c0 2024-01-11T08:29:14.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.191+0000 7fe78bfff700 1 -- 172.21.15.27:0/1672945305 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe784011b30 con 0x7fe7940694c0 2024-01-11T08:29:14.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.191+0000 7fe79b9ff700 1 -- 172.21.15.27:0/1672945305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7940694c0 msgr2=0x7fe794069910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:14.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.191+0000 7fe79b9ff700 1 --2- 172.21.15.27:0/1672945305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7940694c0 0x7fe794069910 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe7840098e0 tx=0x7fe7840091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:14.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.191+0000 7fe79b9ff700 1 -- 172.21.15.27:0/1672945305 shutdown_connections 2024-01-11T08:29:14.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.191+0000 7fe79b9ff700 1 --2- 172.21.15.27:0/1672945305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe794155c50 0x7fe7940aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:14.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.191+0000 7fe79b9ff700 1 --2- 172.21.15.27:0/1672945305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7940694c0 0x7fe794069910 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:14.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.191+0000 7fe79b9ff700 1 -- 172.21.15.27:0/1672945305 >> 172.21.15.27:0/1672945305 conn(0x7fe7940680a0 msgr2=0x7fe7940cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:14.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.191+0000 7fe79b9ff700 1 -- 172.21.15.27:0/1672945305 shutdown_connections 2024-01-11T08:29:14.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.191+0000 7fe79b9ff700 1 -- 172.21.15.27:0/1672945305 wait complete. 2024-01-11T08:29:14.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.193+0000 7fe79b9ff700 1 Processor -- start 2024-01-11T08:29:14.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.193+0000 7fe79b9ff700 1 -- start start 2024-01-11T08:29:14.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.194+0000 7fe79b9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7940694c0 0x7fe794146c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:14.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.194+0000 7fe79b9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7940c6de0 0x7fe7941573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:14.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.194+0000 7fe79b9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe794155c50 0x7fe794158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:14.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.194+0000 7fe79b9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe794156f10 con 0x7fe7940c6de0 2024-01-11T08:29:14.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.194+0000 7fe79b9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7941471d0 con 0x7fe7940694c0 2024-01-11T08:29:14.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.194+0000 7fe79b9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe794156020 con 0x7fe794155c50 2024-01-11T08:29:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.194+0000 7fe798f9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7940c6de0 0x7fe7941573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.194+0000 7fe798f9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7940c6de0 0x7fe7941573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51218/0 (socket says 172.21.15.27:51218) 2024-01-11T08:29:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.194+0000 7fe798f9a700 1 -- 172.21.15.27:0/3480466746 learned_addr learned my addr 172.21.15.27:0/3480466746 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.194+0000 7fe799f9c700 1 --2- 172.21.15.27:0/3480466746 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe794155c50 0x7fe794158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.194+0000 7fe79979b700 1 --2- 172.21.15.27:0/3480466746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7940694c0 0x7fe794146c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.195+0000 7fe798f9a700 1 -- 172.21.15.27:0/3480466746 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe794155c50 msgr2=0x7fe794158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.195+0000 7fe798f9a700 1 --2- 172.21.15.27:0/3480466746 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe794155c50 0x7fe794158cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.195+0000 7fe798f9a700 1 -- 172.21.15.27:0/3480466746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7940694c0 msgr2=0x7fe794146c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.195+0000 7fe798f9a700 1 --2- 172.21.15.27:0/3480466746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7940694c0 0x7fe794146c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.195+0000 7fe798f9a700 1 -- 172.21.15.27:0/3480466746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe784009420 con 0x7fe7940c6de0 2024-01-11T08:29:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.195+0000 7fe79979b700 1 --2- 172.21.15.27:0/3480466746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7940694c0 0x7fe794146c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.195+0000 7fe799f9c700 1 --2- 172.21.15.27:0/3480466746 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe794155c50 0x7fe794158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.195+0000 7fe798f9a700 1 --2- 172.21.15.27:0/3480466746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7940c6de0 0x7fe7941573a0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fe794069490 tx=0x7fe784017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.195+0000 7fe7897fa700 1 -- 172.21.15.27:0/3480466746 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe784011b10 con 0x7fe7940c6de0 2024-01-11T08:29:14.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.195+0000 7fe7897fa700 1 -- 172.21.15.27:0/3480466746 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7840253f0 con 0x7fe7940c6de0 2024-01-11T08:29:14.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.195+0000 7fe79b9ff700 1 -- 172.21.15.27:0/3480466746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe794156300 con 0x7fe7940c6de0 2024-01-11T08:29:14.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.195+0000 7fe7897fa700 1 -- 172.21.15.27:0/3480466746 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7840257b0 con 0x7fe7940c6de0 2024-01-11T08:29:14.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.195+0000 7fe79b9ff700 1 -- 172.21.15.27:0/3480466746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7941568c0 con 0x7fe7940c6de0 2024-01-11T08:29:14.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.196+0000 7fe79b9ff700 1 -- 172.21.15.27:0/3480466746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe794157080 con 0x7fe7940c6de0 2024-01-11T08:29:14.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.197+0000 7fe7897fa700 1 -- 172.21.15.27:0/3480466746 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 67) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fe784025be0 con 0x7fe7940c6de0 2024-01-11T08:29:14.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.198+0000 7fe7897fa700 1 --2- 172.21.15.27:0/3480466746 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe778066ea0 0x7fe778069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:14.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.198+0000 7fe7897fa700 1 -- 172.21.15.27:0/3480466746 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5597+0+0 (secure 0 0 0) 0x7fe784085db0 con 0x7fe7940c6de0 2024-01-11T08:29:14.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.199+0000 7fe79979b700 1 --2- 172.21.15.27:0/3480466746 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe778066ea0 0x7fe778069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:14.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.199+0000 7fe79979b700 1 --2- 172.21.15.27:0/3480466746 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe778066ea0 0x7fe778069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe79000ad40 tx=0x7fe790008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.202+0000 7fe7897fa700 1 -- 172.21.15.27:0/3480466746 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe7840530d0 con 0x7fe7940c6de0 2024-01-11T08:29:14.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.406+0000 7fe79b9ff700 1 -- 172.21.15.27:0/3480466746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe79404c3c0 con 0x7fe7940c6de0 2024-01-11T08:29:14.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.412+0000 7fe7897fa700 1 -- 172.21.15.27:0/3480466746 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v45) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe7840566f0 con 0x7fe7940c6de0 2024-01-11T08:29:14.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.415+0000 7fe79b9ff700 1 -- 172.21.15.27:0/3480466746 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe778066ea0 msgr2=0x7fe778069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:14.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.415+0000 7fe79b9ff700 1 --2- 172.21.15.27:0/3480466746 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe778066ea0 0x7fe778069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe79000ad40 tx=0x7fe790008040 comp rx=0 tx=0).stop 2024-01-11T08:29:14.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.415+0000 7fe79b9ff700 1 -- 172.21.15.27:0/3480466746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7940c6de0 msgr2=0x7fe7941573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:14.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.415+0000 7fe79b9ff700 1 --2- 172.21.15.27:0/3480466746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7940c6de0 0x7fe7941573a0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fe794069490 tx=0x7fe784017000 comp rx=0 tx=0).stop 2024-01-11T08:29:14.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.415+0000 7fe79b9ff700 1 -- 172.21.15.27:0/3480466746 shutdown_connections 2024-01-11T08:29:14.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.415+0000 7fe79b9ff700 1 --2- 172.21.15.27:0/3480466746 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe778066ea0 0x7fe778069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:14.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.415+0000 7fe79b9ff700 1 --2- 172.21.15.27:0/3480466746 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe794155c50 0x7fe794158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:14.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.415+0000 7fe79b9ff700 1 --2- 172.21.15.27:0/3480466746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7940694c0 0x7fe794146c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:14.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.415+0000 7fe79b9ff700 1 --2- 172.21.15.27:0/3480466746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7940c6de0 0x7fe7941573a0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:14.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.415+0000 7fe79b9ff700 1 -- 172.21.15.27:0/3480466746 >> 172.21.15.27:0/3480466746 conn(0x7fe7940680a0 msgr2=0x7fe7940c8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:14.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.415+0000 7fe79b9ff700 1 -- 172.21.15.27:0/3480466746 shutdown_connections 2024-01-11T08:29:14.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.415+0000 7fe79b9ff700 1 -- 172.21.15.27:0/3480466746 wait complete. 2024-01-11T08:29:14.432 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2024-01-11T08:29:14.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.631+0000 7f369e5ff700 1 Processor -- start 2024-01-11T08:29:14.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.631+0000 7f369e5ff700 1 -- start start 2024-01-11T08:29:14.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.632+0000 7f369e5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f36980c8620 0x7f36980af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:14.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.632+0000 7f369e5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3698157390 con 0x7f36980b09e0 2024-01-11T08:29:14.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.632+0000 7f369e5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36980afeb0 con 0x7f3698069d40 2024-01-11T08:29:14.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.632+0000 7f369e5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f369806a200 con 0x7f36980c8620 2024-01-11T08:29:14.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.633+0000 7f369cb9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f36980b09e0 0x7f36980accc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51154/0 (socket says 172.21.15.27:51154) 2024-01-11T08:29:14.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.633+0000 7f369cb9c700 1 -- 172.21.15.27:0/965416649 learned_addr learned my addr 172.21.15.27:0/965416649 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:14.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.633+0000 7f3697fff700 1 --2- 172.21.15.27:0/965416649 >> v2:172.21.15.77:3300/0 conn(0x7f36980c8620 0x7f36980af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:14.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.633+0000 7f3696ffd700 1 -- 172.21.15.27:0/965416649 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109104478 0 0) 0x7f3698157390 con 0x7f36980b09e0 2024-01-11T08:29:14.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.633+0000 7f3696ffd700 1 -- 172.21.15.27:0/965416649 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3680003570 con 0x7f36980b09e0 2024-01-11T08:29:14.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.633+0000 7f3696ffd700 1 -- 172.21.15.27:0/965416649 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466349629 0 0) 0x7f36980afeb0 con 0x7f3698069d40 2024-01-11T08:29:14.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.633+0000 7f3696ffd700 1 -- 172.21.15.27:0/965416649 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3698157390 con 0x7f3698069d40 2024-01-11T08:29:14.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.633+0000 7f3696ffd700 1 -- 172.21.15.27:0/965416649 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1111218346 0 0) 0x7f3680003570 con 0x7f36980b09e0 2024-01-11T08:29:14.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.633+0000 7f3696ffd700 1 -- 172.21.15.27:0/965416649 >> v2:172.21.15.77:3300/0 conn(0x7f36980c8620 msgr2=0x7f36980af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:14.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.633+0000 7f3696ffd700 1 --2- 172.21.15.27:0/965416649 >> v2:172.21.15.77:3300/0 conn(0x7f36980c8620 0x7f36980af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:14.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.633+0000 7f3696ffd700 1 -- 172.21.15.27:0/965416649 >> v1:172.21.15.77:6789/0 conn(0x7f3698069d40 legacy=0x7f36980b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:14.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.633+0000 7f3696ffd700 1 -- 172.21.15.27:0/965416649 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3698156d40 con 0x7f36980b09e0 2024-01-11T08:29:14.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.633+0000 7f3696ffd700 1 -- 172.21.15.27:0/965416649 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f36900017f0 con 0x7f36980b09e0 2024-01-11T08:29:14.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.634+0000 7f3696ffd700 1 -- 172.21.15.27:0/965416649 >> v1:172.21.15.27:6789/0 conn(0x7f36980b09e0 legacy=0x7f36980accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:14.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.634+0000 7f3696ffd700 1 --2- 172.21.15.27:0/965416649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3680003bf0 0x7f3680003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:14.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.634+0000 7f3696ffd700 1 --2- 172.21.15.27:0/965416649 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3680004810 0x7f3680004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:14.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.634+0000 7f3696ffd700 1 --2- 172.21.15.27:0/965416649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800051c0 0x7f368000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:14.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.634+0000 7f3696ffd700 1 -- 172.21.15.27:0/965416649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369806a200 con 0x7f36800051c0 2024-01-11T08:29:14.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.634+0000 7f3696ffd700 1 -- 172.21.15.27:0/965416649 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368000c160 con 0x7f3680003bf0 2024-01-11T08:29:14.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.634+0000 7f3696ffd700 1 -- 172.21.15.27:0/965416649 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f368000c320 con 0x7f3680004810 2024-01-11T08:29:14.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.634+0000 7f369cb9c700 1 --2- 172.21.15.27:0/965416649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800051c0 0x7f368000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:14.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.634+0000 7f3697fff700 1 --2- 172.21.15.27:0/965416649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3680003bf0 0x7f3680003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:14.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.634+0000 7f36977fe700 1 --2- 172.21.15.27:0/965416649 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3680004810 0x7f3680004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:14.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.634+0000 7f369cb9c700 1 -- 172.21.15.27:0/965416649 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3680004810 msgr2=0x7f3680004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:14.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.634+0000 7f369cb9c700 1 --2- 172.21.15.27:0/965416649 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3680004810 0x7f3680004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:14.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.634+0000 7f369cb9c700 1 -- 172.21.15.27:0/965416649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3680003bf0 msgr2=0x7f3680003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:14.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.634+0000 7f369cb9c700 1 --2- 172.21.15.27:0/965416649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3680003bf0 0x7f3680003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:14.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.634+0000 7f369cb9c700 1 -- 172.21.15.27:0/965416649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3680003570 con 0x7f36800051c0 2024-01-11T08:29:14.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.634+0000 7f3697fff700 1 --2- 172.21.15.27:0/965416649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3680003bf0 0x7f3680003fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:14.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.635+0000 7f369cb9c700 1 -- 172.21.15.27:0/965416649 reap_dead start 2024-01-11T08:29:14.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.635+0000 7f369cb9c700 1 --2- 172.21.15.27:0/965416649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800051c0 0x7f368000ba60 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f36980b0de0 tx=0x7f3690002190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e274f7dad0a82f15 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:14.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.635+0000 7f3696ffd700 1 -- 172.21.15.27:0/965416649 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36900070d0 con 0x7f36800051c0 2024-01-11T08:29:14.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.635+0000 7f3696ffd700 1 -- 172.21.15.27:0/965416649 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3690007230 con 0x7f36800051c0 2024-01-11T08:29:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.635+0000 7f3696ffd700 1 -- 172.21.15.27:0/965416649 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36900075d0 con 0x7f36800051c0 2024-01-11T08:29:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.635+0000 7f369e5ff700 1 -- 172.21.15.27:0/965416649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800051c0 msgr2=0x7f368000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.635+0000 7f369e5ff700 1 --2- 172.21.15.27:0/965416649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800051c0 0x7f368000ba60 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f36980b0de0 tx=0x7f3690002190 comp rx=0 tx=0).stop 2024-01-11T08:29:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.635+0000 7f369e5ff700 1 -- 172.21.15.27:0/965416649 shutdown_connections 2024-01-11T08:29:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.635+0000 7f369e5ff700 1 --2- 172.21.15.27:0/965416649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800051c0 0x7f368000ba60 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.635+0000 7f369e5ff700 1 -- 172.21.15.27:0/965416649 >> 172.21.15.27:0/965416649 conn(0x7f36980680e0 msgr2=0x7f3698069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.636+0000 7f369e5ff700 1 -- 172.21.15.27:0/965416649 shutdown_connections 2024-01-11T08:29:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.636+0000 7f369e5ff700 1 -- 172.21.15.27:0/965416649 wait complete. 2024-01-11T08:29:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.637+0000 7f369e5ff700 1 Processor -- start 2024-01-11T08:29:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.637+0000 7f369e5ff700 1 -- start start 2024-01-11T08:29:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.638+0000 7f369e5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800051c0 0x7f36981516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.638+0000 7f369e5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36980add30 0x7f36980cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.638+0000 7f369e5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f36980ae1a0 0x7f36980ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.638+0000 7f369e5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369816ddf0 con 0x7f36800051c0 2024-01-11T08:29:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.638+0000 7f369e5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36980acc90 con 0x7f36980add30 2024-01-11T08:29:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.638+0000 7f369e5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36980acf70 con 0x7f36980ae1a0 2024-01-11T08:29:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.638+0000 7f3697fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800051c0 0x7f36981516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.638+0000 7f3697fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800051c0 0x7f36981516e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51224/0 (socket says 172.21.15.27:51224) 2024-01-11T08:29:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.638+0000 7f3697fff700 1 -- 172.21.15.27:0/2724800480 learned_addr learned my addr 172.21.15.27:0/2724800480 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.638+0000 7f36977fe700 1 --2- 172.21.15.27:0/2724800480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36980add30 0x7f36980cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.639+0000 7f369cb9c700 1 --2- 172.21.15.27:0/2724800480 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f36980ae1a0 0x7f36980ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.639+0000 7f3697fff700 1 -- 172.21.15.27:0/2724800480 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f36980ae1a0 msgr2=0x7f36980ccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.639+0000 7f3697fff700 1 --2- 172.21.15.27:0/2724800480 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f36980ae1a0 0x7f36980ccc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.639+0000 7f3697fff700 1 -- 172.21.15.27:0/2724800480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36980add30 msgr2=0x7f36980cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.639+0000 7f3697fff700 1 --2- 172.21.15.27:0/2724800480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36980add30 0x7f36980cc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.639+0000 7f3697fff700 1 -- 172.21.15.27:0/2724800480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36980ad250 con 0x7f36800051c0 2024-01-11T08:29:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.639+0000 7f369cb9c700 1 --2- 172.21.15.27:0/2724800480 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f36980ae1a0 0x7f36980ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.639+0000 7f3697fff700 1 --2- 172.21.15.27:0/2724800480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800051c0 0x7f36981516e0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f3688011d70 tx=0x7f36880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.639+0000 7f367ffff700 1 -- 172.21.15.27:0/2724800480 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3688011e70 con 0x7f36800051c0 2024-01-11T08:29:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.639+0000 7f367ffff700 1 -- 172.21.15.27:0/2724800480 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368800fcd0 con 0x7f36800051c0 2024-01-11T08:29:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.639+0000 7f369e5ff700 1 -- 172.21.15.27:0/2724800480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36980ad530 con 0x7f36800051c0 2024-01-11T08:29:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.639+0000 7f367ffff700 1 -- 172.21.15.27:0/2724800480 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3688014670 con 0x7f36800051c0 2024-01-11T08:29:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.640+0000 7f369e5ff700 1 -- 172.21.15.27:0/2724800480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3698151c20 con 0x7f36800051c0 2024-01-11T08:29:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.640+0000 7f369e5ff700 1 -- 172.21.15.27:0/2724800480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36980ada50 con 0x7f36800051c0 2024-01-11T08:29:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.641+0000 7f367ffff700 1 -- 172.21.15.27:0/2724800480 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 67) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f3688014a20 con 0x7f36800051c0 2024-01-11T08:29:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.642+0000 7f367ffff700 1 --2- 172.21.15.27:0/2724800480 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3684066ea0 0x7f3684069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.642+0000 7f367ffff700 1 -- 172.21.15.27:0/2724800480 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5597+0+0 (secure 0 0 0) 0x7f3688018070 con 0x7f36800051c0 2024-01-11T08:29:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.643+0000 7f36977fe700 1 --2- 172.21.15.27:0/2724800480 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3684066ea0 0x7f3684069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:14.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.643+0000 7f36977fe700 1 --2- 172.21.15.27:0/2724800480 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3684066ea0 0x7f3684069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f368c000c00 tx=0x7f368c007d30 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:14.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.646+0000 7f367ffff700 1 -- 172.21.15.27:0/2724800480 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f36980ada50 con 0x7f36800051c0 2024-01-11T08:29:14.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.934+0000 7f369e5ff700 1 -- 172.21.15.27:0/2724800480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f3698151ea0 con 0x7f36800051c0 2024-01-11T08:29:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:14.975+0000 7f367ffff700 1 -- 172.21.15.27:0/2724800480 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3688014cd0 con 0x7f36800051c0 2024-01-11T08:29:15.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:15.856+0000 7f367ffff700 1 -- 172.21.15.27:0/2724800480 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v708) v1 ==== 148+0+0 (secure 0 0 0) 0x7f3688055da0 con 0x7f36800051c0 2024-01-11T08:29:15.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:15.859+0000 7f369e5ff700 1 -- 172.21.15.27:0/2724800480 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3684066ea0 msgr2=0x7f3684069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:15.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:15.859+0000 7f369e5ff700 1 --2- 172.21.15.27:0/2724800480 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3684066ea0 0x7f3684069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f368c000c00 tx=0x7f368c007d30 comp rx=0 tx=0).stop 2024-01-11T08:29:15.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:15.859+0000 7f369e5ff700 1 -- 172.21.15.27:0/2724800480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800051c0 msgr2=0x7f36981516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:15.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:15.859+0000 7f369e5ff700 1 --2- 172.21.15.27:0/2724800480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800051c0 0x7f36981516e0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f3688011d70 tx=0x7f36880091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:15.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:15.859+0000 7f369e5ff700 1 -- 172.21.15.27:0/2724800480 shutdown_connections 2024-01-11T08:29:15.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:15.859+0000 7f369e5ff700 1 --2- 172.21.15.27:0/2724800480 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3684066ea0 0x7f3684069350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:15.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:15.859+0000 7f369e5ff700 1 --2- 172.21.15.27:0/2724800480 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f36980ae1a0 0x7f36980ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:15.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:15.859+0000 7f369e5ff700 1 --2- 172.21.15.27:0/2724800480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800051c0 0x7f36981516e0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:15.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:15.859+0000 7f369e5ff700 1 --2- 172.21.15.27:0/2724800480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36980add30 0x7f36980cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:15.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:15.859+0000 7f369e5ff700 1 -- 172.21.15.27:0/2724800480 >> 172.21.15.27:0/2724800480 conn(0x7f36980680e0 msgr2=0x7f3698153760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:15.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:15.859+0000 7f369e5ff700 1 -- 172.21.15.27:0/2724800480 shutdown_connections 2024-01-11T08:29:15.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:15.859+0000 7f369e5ff700 1 -- 172.21.15.27:0/2724800480 wait complete. 2024-01-11T08:29:15.880 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2024-01-11T08:29:15.880 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-11T08:29:15.881 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2024-01-11T08:29:16.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.085+0000 7fe3613ff700 1 Processor -- start 2024-01-11T08:29:16.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.085+0000 7fe3613ff700 1 -- start start 2024-01-11T08:29:16.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.086+0000 7fe3613ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe35c1575e0 0x7fe35c158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:16.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.086+0000 7fe3613ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35c158b60 con 0x7fe35c06b610 2024-01-11T08:29:16.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.086+0000 7fe3613ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35c1599b0 con 0x7fe35c0b19d0 2024-01-11T08:29:16.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.086+0000 7fe3613ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe35c0cb5e0 con 0x7fe35c1575e0 2024-01-11T08:29:16.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.087+0000 7fe35affd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe35c1575e0 0x7fe35c158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:16.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.087+0000 7fe35affd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe35c1575e0 0x7fe35c158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51228/0 (socket says 172.21.15.27:51228) 2024-01-11T08:29:16.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.087+0000 7fe35affd700 1 -- 172.21.15.27:0/387309686 learned_addr learned my addr 172.21.15.27:0/387309686 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:16.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.087+0000 7fe35a7fc700 1 --1- 172.21.15.27:0/387309686 >> v1:172.21.15.27:6789/0 conn(0x7fe35c06b610 0x7fe35c0b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51162/0 (socket says 172.21.15.27:51162) 2024-01-11T08:29:16.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.087+0000 7fe35affd700 1 -- 172.21.15.27:0/387309686 >> v1:172.21.15.77:6789/0 conn(0x7fe35c0b19d0 legacy=0x7fe35c159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:16.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.087+0000 7fe35affd700 1 -- 172.21.15.27:0/387309686 >> v1:172.21.15.27:6789/0 conn(0x7fe35c06b610 legacy=0x7fe35c0b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.087+0000 7fe35affd700 1 -- 172.21.15.27:0/387309686 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35c0c0f90 con 0x7fe35c1575e0 2024-01-11T08:29:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.087+0000 7fe35affd700 1 --2- 172.21.15.27:0/387309686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe35c1575e0 0x7fe35c158620 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0099b0 tx=0x7fe34c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1967115795ab2580 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.088+0000 7fe359ffb700 1 -- 172.21.15.27:0/387309686 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c00b860 con 0x7fe35c1575e0 2024-01-11T08:29:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.088+0000 7fe359ffb700 1 -- 172.21.15.27:0/387309686 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe34c00b9c0 con 0x7fe35c1575e0 2024-01-11T08:29:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.088+0000 7fe359ffb700 1 -- 172.21.15.27:0/387309686 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c00bd60 con 0x7fe35c1575e0 2024-01-11T08:29:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.088+0000 7fe3613ff700 1 -- 172.21.15.27:0/387309686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe35c1575e0 msgr2=0x7fe35c158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.088+0000 7fe3613ff700 1 --2- 172.21.15.27:0/387309686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe35c1575e0 0x7fe35c158620 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0099b0 tx=0x7fe34c009250 comp rx=0 tx=0).stop 2024-01-11T08:29:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.088+0000 7fe3613ff700 1 -- 172.21.15.27:0/387309686 shutdown_connections 2024-01-11T08:29:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.088+0000 7fe3613ff700 1 --2- 172.21.15.27:0/387309686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe35c1575e0 0x7fe35c158620 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.088+0000 7fe3613ff700 1 -- 172.21.15.27:0/387309686 >> 172.21.15.27:0/387309686 conn(0x7fe35c0680e0 msgr2=0x7fe35c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.088+0000 7fe3613ff700 1 -- 172.21.15.27:0/387309686 shutdown_connections 2024-01-11T08:29:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.088+0000 7fe3613ff700 1 -- 172.21.15.27:0/387309686 wait complete. 2024-01-11T08:29:16.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.090+0000 7fe3613ff700 1 Processor -- start 2024-01-11T08:29:16.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.090+0000 7fe3613ff700 1 -- start start 2024-01-11T08:29:16.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.091+0000 7fe3613ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe35c06b610 0x7fe35c143c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:16.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.091+0000 7fe3613ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe35c0b19d0 0x7fe35c16d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:16.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.091+0000 7fe3613ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe35c1575e0 0x7fe35c151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:16.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.091+0000 7fe3613ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c1441d0 con 0x7fe35c1575e0 2024-01-11T08:29:16.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.091+0000 7fe3613ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c16d980 con 0x7fe35c06b610 2024-01-11T08:29:16.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.091+0000 7fe3613ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c16dc60 con 0x7fe35c0b19d0 2024-01-11T08:29:16.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.091+0000 7fe35b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe35c1575e0 0x7fe35c151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:16.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.091+0000 7fe35b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe35c1575e0 0x7fe35c151620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51234/0 (socket says 172.21.15.27:51234) 2024-01-11T08:29:16.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.091+0000 7fe35b7fe700 1 -- 172.21.15.27:0/598802737 learned_addr learned my addr 172.21.15.27:0/598802737 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:16.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.091+0000 7fe35a7fc700 1 --2- 172.21.15.27:0/598802737 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe35c0b19d0 0x7fe35c16d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:16.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.091+0000 7fe35affd700 1 --2- 172.21.15.27:0/598802737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe35c06b610 0x7fe35c143c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:16.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.092+0000 7fe35b7fe700 1 -- 172.21.15.27:0/598802737 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe35c0b19d0 msgr2=0x7fe35c16d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:16.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.092+0000 7fe35b7fe700 1 --2- 172.21.15.27:0/598802737 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe35c0b19d0 0x7fe35c16d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:16.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.092+0000 7fe35b7fe700 1 -- 172.21.15.27:0/598802737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe35c06b610 msgr2=0x7fe35c143c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:16.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.092+0000 7fe35b7fe700 1 --2- 172.21.15.27:0/598802737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe35c06b610 0x7fe35c143c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:16.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.092+0000 7fe35b7fe700 1 -- 172.21.15.27:0/598802737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe34c0094f0 con 0x7fe35c1575e0 2024-01-11T08:29:16.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.092+0000 7fe35b7fe700 1 --2- 172.21.15.27:0/598802737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe35c1575e0 0x7fe35c151620 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fe35400c8e0 tx=0x7fe354001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:16.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.092+0000 7fe342ffd700 1 -- 172.21.15.27:0/598802737 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35400ce80 con 0x7fe35c1575e0 2024-01-11T08:29:16.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.092+0000 7fe342ffd700 1 -- 172.21.15.27:0/598802737 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35400a7e0 con 0x7fe35c1575e0 2024-01-11T08:29:16.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.092+0000 7fe3613ff700 1 -- 172.21.15.27:0/598802737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35c16df40 con 0x7fe35c1575e0 2024-01-11T08:29:16.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.092+0000 7fe342ffd700 1 -- 172.21.15.27:0/598802737 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35400ab80 con 0x7fe35c1575e0 2024-01-11T08:29:16.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.092+0000 7fe3613ff700 1 -- 172.21.15.27:0/598802737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe35c0af8d0 con 0x7fe35c1575e0 2024-01-11T08:29:16.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.093+0000 7fe3613ff700 1 -- 172.21.15.27:0/598802737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe35c0afac0 con 0x7fe35c1575e0 2024-01-11T08:29:16.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.094+0000 7fe342ffd700 1 -- 172.21.15.27:0/598802737 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe354012420 con 0x7fe35c1575e0 2024-01-11T08:29:16.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.095+0000 7fe342ffd700 1 --2- 172.21.15.27:0/598802737 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe348066f30 0x7fe3480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:16.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.095+0000 7fe342ffd700 1 -- 172.21.15.27:0/598802737 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5597+0+0 (secure 0 0 0) 0x7fe354084850 con 0x7fe35c1575e0 2024-01-11T08:29:16.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.096+0000 7fe35affd700 1 --2- 172.21.15.27:0/598802737 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe348066f30 0x7fe3480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:16.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.096+0000 7fe35affd700 1 --2- 172.21.15.27:0/598802737 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe348066f30 0x7fe3480693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0099b0 tx=0x7fe34c017000 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:16.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.100+0000 7fe342ffd700 1 -- 172.21.15.27:0/598802737 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe354051a90 con 0x7fe35c1575e0 2024-01-11T08:29:16.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.382+0000 7fe3613ff700 1 -- 172.21.15.27:0/598802737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7fe35c0718d0 con 0x7fe35c1575e0 2024-01-11T08:29:16.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.860+0000 7fe342ffd700 1 -- 172.21.15.27:0/598802737 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v709) v1 ==== 163+0+0 (secure 0 0 0) 0x7fe3540550b0 con 0x7fe35c1575e0 2024-01-11T08:29:16.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.863+0000 7fe3613ff700 1 -- 172.21.15.27:0/598802737 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe348066f30 msgr2=0x7fe3480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:16.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.863+0000 7fe3613ff700 1 --2- 172.21.15.27:0/598802737 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe348066f30 0x7fe3480693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0099b0 tx=0x7fe34c017000 comp rx=0 tx=0).stop 2024-01-11T08:29:16.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.863+0000 7fe3613ff700 1 -- 172.21.15.27:0/598802737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe35c1575e0 msgr2=0x7fe35c151620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:16.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.863+0000 7fe3613ff700 1 --2- 172.21.15.27:0/598802737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe35c1575e0 0x7fe35c151620 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fe35400c8e0 tx=0x7fe354001710 comp rx=0 tx=0).stop 2024-01-11T08:29:16.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.863+0000 7fe3613ff700 1 -- 172.21.15.27:0/598802737 shutdown_connections 2024-01-11T08:29:16.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.863+0000 7fe3613ff700 1 --2- 172.21.15.27:0/598802737 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe348066f30 0x7fe3480693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:16.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.863+0000 7fe3613ff700 1 --2- 172.21.15.27:0/598802737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe35c1575e0 0x7fe35c151620 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:16.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.863+0000 7fe3613ff700 1 --2- 172.21.15.27:0/598802737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe35c06b610 0x7fe35c143c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:16.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.863+0000 7fe3613ff700 1 --2- 172.21.15.27:0/598802737 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe35c0b19d0 0x7fe35c16d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:16.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.863+0000 7fe3613ff700 1 -- 172.21.15.27:0/598802737 >> 172.21.15.27:0/598802737 conn(0x7fe35c0680e0 msgr2=0x7fe35c152d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:16.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.863+0000 7fe3613ff700 1 -- 172.21.15.27:0/598802737 shutdown_connections 2024-01-11T08:29:16.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:16.863+0000 7fe3613ff700 1 -- 172.21.15.27:0/598802737 wait complete. 2024-01-11T08:29:16.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-11T08:29:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.085+0000 7f26695ff700 1 Processor -- start 2024-01-11T08:29:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.085+0000 7f26695ff700 1 -- start start 2024-01-11T08:29:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.086+0000 7f26695ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f26641598f0 0x7f26641646c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.086+0000 7f26695ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f266415aef0 con 0x7f26640cd5f0 2024-01-11T08:29:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.086+0000 7f26695ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26640cee00 con 0x7f26640cd220 2024-01-11T08:29:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.086+0000 7f26695ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f266415dd70 con 0x7f26641598f0 2024-01-11T08:29:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.086+0000 7f26627fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f26640cd5f0 0x7f26641591e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51176/0 (socket says 172.21.15.27:51176) 2024-01-11T08:29:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.086+0000 7f26627fc700 1 -- 172.21.15.27:0/3942361278 learned_addr learned my addr 172.21.15.27:0/3942361278 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.086+0000 7f26637fe700 1 --2- 172.21.15.27:0/3942361278 >> v2:172.21.15.77:3300/0 conn(0x7f26641598f0 0x7f26641646c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:17.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.087+0000 7f2661ffb700 1 -- 172.21.15.27:0/3942361278 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558401999 0 0) 0x7f266415aef0 con 0x7f26640cd5f0 2024-01-11T08:29:17.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.087+0000 7f2661ffb700 1 -- 172.21.15.27:0/3942361278 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f264c003570 con 0x7f26640cd5f0 2024-01-11T08:29:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.087+0000 7f26637fe700 1 -- 172.21.15.27:0/3942361278 >> v1:172.21.15.77:6789/0 conn(0x7f26640cd220 legacy=0x7f266415d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.087+0000 7f26637fe700 1 -- 172.21.15.27:0/3942361278 >> v1:172.21.15.27:6789/0 conn(0x7f26640cd5f0 legacy=0x7f26641591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.087+0000 7f26637fe700 1 -- 172.21.15.27:0/3942361278 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266415e050 con 0x7f26641598f0 2024-01-11T08:29:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.087+0000 7f26637fe700 1 --2- 172.21.15.27:0/3942361278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26641598f0 0x7f26641646c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f265c009e60 tx=0x7f265c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2f0032c8b047bb1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.088+0000 7f2661ffb700 1 -- 172.21.15.27:0/3942361278 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c00e070 con 0x7f26641598f0 2024-01-11T08:29:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.088+0000 7f2661ffb700 1 -- 172.21.15.27:0/3942361278 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f265c00bb60 con 0x7f26641598f0 2024-01-11T08:29:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.088+0000 7f2661ffb700 1 -- 172.21.15.27:0/3942361278 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c012430 con 0x7f26641598f0 2024-01-11T08:29:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.088+0000 7f26695ff700 1 -- 172.21.15.27:0/3942361278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26641598f0 msgr2=0x7f26641646c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.088+0000 7f26695ff700 1 --2- 172.21.15.27:0/3942361278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26641598f0 0x7f26641646c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f265c009e60 tx=0x7f265c009780 comp rx=0 tx=0).stop 2024-01-11T08:29:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.088+0000 7f26695ff700 1 -- 172.21.15.27:0/3942361278 shutdown_connections 2024-01-11T08:29:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.088+0000 7f26695ff700 1 --2- 172.21.15.27:0/3942361278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26641598f0 0x7f26641646c0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:17.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.088+0000 7f26695ff700 1 -- 172.21.15.27:0/3942361278 >> 172.21.15.27:0/3942361278 conn(0x7f266406ed40 msgr2=0x7f266406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:17.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.088+0000 7f26695ff700 1 -- 172.21.15.27:0/3942361278 shutdown_connections 2024-01-11T08:29:17.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.088+0000 7f26695ff700 1 -- 172.21.15.27:0/3942361278 wait complete. 2024-01-11T08:29:17.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.090+0000 7f26695ff700 1 Processor -- start 2024-01-11T08:29:17.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.090+0000 7f26695ff700 1 -- start start 2024-01-11T08:29:17.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.091+0000 7f26695ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26640cd220 0x7f266414f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:17.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.091+0000 7f26695ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26640cd5f0 0x7f266416d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:17.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.091+0000 7f26695ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26641598f0 0x7f266416d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:17.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.091+0000 7f26695ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26640c90b0 con 0x7f26641598f0 2024-01-11T08:29:17.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.091+0000 7f26695ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266414fa40 con 0x7f26640cd220 2024-01-11T08:29:17.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.091+0000 7f26695ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26640b1840 con 0x7f26640cd5f0 2024-01-11T08:29:17.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.091+0000 7f26637fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26641598f0 0x7f266416d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:17.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.091+0000 7f26637fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26641598f0 0x7f266416d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51236/0 (socket says 172.21.15.27:51236) 2024-01-11T08:29:17.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.091+0000 7f26637fe700 1 -- 172.21.15.27:0/2922197865 learned_addr learned my addr 172.21.15.27:0/2922197865 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:17.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.092+0000 7f2662ffd700 1 --2- 172.21.15.27:0/2922197865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26640cd220 0x7f266414f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:17.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.092+0000 7f26627fc700 1 --2- 172.21.15.27:0/2922197865 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26640cd5f0 0x7f266416d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:17.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.092+0000 7f26637fe700 1 -- 172.21.15.27:0/2922197865 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26640cd5f0 msgr2=0x7f266416d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:17.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.092+0000 7f26637fe700 1 --2- 172.21.15.27:0/2922197865 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26640cd5f0 0x7f266416d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:17.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.092+0000 7f26637fe700 1 -- 172.21.15.27:0/2922197865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26640cd220 msgr2=0x7f266414f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:17.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.092+0000 7f26637fe700 1 --2- 172.21.15.27:0/2922197865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26640cd220 0x7f266414f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:17.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.092+0000 7f26637fe700 1 -- 172.21.15.27:0/2922197865 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265c009a20 con 0x7f26641598f0 2024-01-11T08:29:17.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.092+0000 7f26627fc700 1 --2- 172.21.15.27:0/2922197865 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26640cd5f0 0x7f266416d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:29:17.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.092+0000 7f26637fe700 1 --2- 172.21.15.27:0/2922197865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26641598f0 0x7f266416d6f0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f265c012b20 tx=0x7f265c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:17.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.092+0000 7f264affd700 1 -- 172.21.15.27:0/2922197865 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c012e60 con 0x7f26641598f0 2024-01-11T08:29:17.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.092+0000 7f264affd700 1 -- 172.21.15.27:0/2922197865 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f265c021070 con 0x7f26641598f0 2024-01-11T08:29:17.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.092+0000 7f26695ff700 1 -- 172.21.15.27:0/2922197865 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26640c8810 con 0x7f26641598f0 2024-01-11T08:29:17.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.092+0000 7f264affd700 1 -- 172.21.15.27:0/2922197865 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c026680 con 0x7f26641598f0 2024-01-11T08:29:17.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.093+0000 7f26695ff700 1 -- 172.21.15.27:0/2922197865 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26640c8dd0 con 0x7f26641598f0 2024-01-11T08:29:17.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.094+0000 7f26695ff700 1 -- 172.21.15.27:0/2922197865 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f266404c3c0 con 0x7f26641598f0 2024-01-11T08:29:17.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.094+0000 7f264affd700 1 -- 172.21.15.27:0/2922197865 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f265c023030 con 0x7f26641598f0 2024-01-11T08:29:17.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.095+0000 7f264affd700 1 --2- 172.21.15.27:0/2922197865 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2650066f30 0x7f26500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:17.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.095+0000 7f264affd700 1 -- 172.21.15.27:0/2922197865 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5597+0+0 (secure 0 0 0) 0x7f265c086760 con 0x7f26641598f0 2024-01-11T08:29:17.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.096+0000 7f2662ffd700 1 --2- 172.21.15.27:0/2922197865 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2650066f30 0x7f26500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:17.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.096+0000 7f2662ffd700 1 --2- 172.21.15.27:0/2922197865 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2650066f30 0x7f26500693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2654003860 tx=0x7f2654009040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:17.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.099+0000 7f264affd700 1 -- 172.21.15.27:0/2922197865 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f265c0539a0 con 0x7f26641598f0 2024-01-11T08:29:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.304+0000 7f26695ff700 1 -- 172.21.15.27:0/2922197865 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f26640cea90 con 0x7f26641598f0 2024-01-11T08:29:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.308+0000 7f264affd700 1 -- 172.21.15.27:0/2922197865 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7f265c056fc0 con 0x7f26641598f0 2024-01-11T08:29:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.310+0000 7f26695ff700 1 -- 172.21.15.27:0/2922197865 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2650066f30 msgr2=0x7f26500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.310+0000 7f26695ff700 1 --2- 172.21.15.27:0/2922197865 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2650066f30 0x7f26500693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2654003860 tx=0x7f2654009040 comp rx=0 tx=0).stop 2024-01-11T08:29:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.310+0000 7f26695ff700 1 -- 172.21.15.27:0/2922197865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26641598f0 msgr2=0x7f266416d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.310+0000 7f26695ff700 1 --2- 172.21.15.27:0/2922197865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26641598f0 0x7f266416d6f0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f265c012b20 tx=0x7f265c019000 comp rx=0 tx=0).stop 2024-01-11T08:29:17.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.310+0000 7f26695ff700 1 -- 172.21.15.27:0/2922197865 shutdown_connections 2024-01-11T08:29:17.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.310+0000 7f26695ff700 1 --2- 172.21.15.27:0/2922197865 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2650066f30 0x7f26500693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:17.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.310+0000 7f26695ff700 1 --2- 172.21.15.27:0/2922197865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26641598f0 0x7f266416d6f0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:17.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.310+0000 7f26695ff700 1 --2- 172.21.15.27:0/2922197865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26640cd220 0x7f266414f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:17.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.310+0000 7f26695ff700 1 --2- 172.21.15.27:0/2922197865 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26640cd5f0 0x7f266416d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:17.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.310+0000 7f26695ff700 1 -- 172.21.15.27:0/2922197865 >> 172.21.15.27:0/2922197865 conn(0x7f266406ed40 msgr2=0x7f26640d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:17.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.311+0000 7f26695ff700 1 -- 172.21.15.27:0/2922197865 shutdown_connections 2024-01-11T08:29:17.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.311+0000 7f26695ff700 1 -- 172.21.15.27:0/2922197865 wait complete. 2024-01-11T08:29:17.328 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T08:29:17.328 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:29:17.417 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:29:17.511 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T08:29:17.511 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-11T08:29:17.511 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:29:17.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.712+0000 7f5c63bff700 1 Processor -- start 2024-01-11T08:29:17.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.712+0000 7f5c63bff700 1 -- start start 2024-01-11T08:29:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.713+0000 7f5c63bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c5c0c8620 0x7f5c5c0af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.713+0000 7f5c63bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5c5c069d40 0x7f5c5c0b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.713+0000 7f5c63bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5c157390 con 0x7f5c5c0c89f0 2024-01-11T08:29:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.713+0000 7f5c63bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c5c0adcd0 con 0x7f5c5c069d40 2024-01-11T08:29:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.713+0000 7f5c63bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c0afeb0 con 0x7f5c5c0c8620 2024-01-11T08:29:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.714+0000 7f5c6119a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5c5c069d40 0x7f5c5c0b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.714+0000 7f5c6199b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c5c0c8620 0x7f5c5c0af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.714+0000 7f5c6119a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5c5c069d40 0x7f5c5c0b02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:35948/0 (socket says 172.21.15.27:35948) 2024-01-11T08:29:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.714+0000 7f5c6199b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c5c0c8620 0x7f5c5c0af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:38592/0 (socket says 172.21.15.27:38592) 2024-01-11T08:29:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.714+0000 7f5c6119a700 1 -- 172.21.15.27:0/2065618686 learned_addr learned my addr 172.21.15.27:0/2065618686 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.714+0000 7f5c6219c700 1 --1- 172.21.15.27:0/2065618686 >> v1:172.21.15.77:6789/0 conn(0x7f5c5c0c89f0 0x7f5c5c159ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:35600/0 (socket says 172.21.15.27:35600) 2024-01-11T08:29:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.714+0000 7f5c6199b700 1 -- 172.21.15.27:0/2065618686 >> v2:172.21.15.77:3300/0 conn(0x7f5c5c069d40 msgr2=0x7f5c5c0b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.714+0000 7f5c6199b700 1 --2- 172.21.15.27:0/2065618686 >> v2:172.21.15.77:3300/0 conn(0x7f5c5c069d40 0x7f5c5c0b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.714+0000 7f5c6199b700 1 -- 172.21.15.27:0/2065618686 >> v1:172.21.15.77:6789/0 conn(0x7f5c5c0c89f0 legacy=0x7f5c5c159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.714+0000 7f5c6199b700 1 -- 172.21.15.27:0/2065618686 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5c06a200 con 0x7f5c5c0c8620 2024-01-11T08:29:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.714+0000 7f5c6119a700 1 --2- 172.21.15.27:0/2065618686 >> v2:172.21.15.77:3300/0 conn(0x7f5c5c069d40 0x7f5c5c0b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.715+0000 7f5c6199b700 1 --2- 172.21.15.27:0/2065618686 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c5c0c8620 0x7f5c5c0af970 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5c580098e0 tx=0x7f5c580091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c09c51c4ab0a865c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.715+0000 7f5c60999700 1 -- 172.21.15.27:0/2065618686 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5800b640 con 0x7f5c5c0c8620 2024-01-11T08:29:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.715+0000 7f5c60999700 1 -- 172.21.15.27:0/2065618686 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c5800b7a0 con 0x7f5c5c0c8620 2024-01-11T08:29:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.715+0000 7f5c60999700 1 -- 172.21.15.27:0/2065618686 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5800bb40 con 0x7f5c5c0c8620 2024-01-11T08:29:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.715+0000 7f5c63bff700 1 -- 172.21.15.27:0/2065618686 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c5c0c8620 msgr2=0x7f5c5c0af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.715+0000 7f5c63bff700 1 --2- 172.21.15.27:0/2065618686 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c5c0c8620 0x7f5c5c0af970 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5c580098e0 tx=0x7f5c580091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:17.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.715+0000 7f5c63bff700 1 -- 172.21.15.27:0/2065618686 shutdown_connections 2024-01-11T08:29:17.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.716+0000 7f5c63bff700 1 --2- 172.21.15.27:0/2065618686 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c5c0c8620 0x7f5c5c0af970 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:17.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.716+0000 7f5c63bff700 1 --2- 172.21.15.27:0/2065618686 >> v2:172.21.15.77:3300/0 conn(0x7f5c5c069d40 0x7f5c5c0b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:17.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.716+0000 7f5c63bff700 1 -- 172.21.15.27:0/2065618686 >> 172.21.15.27:0/2065618686 conn(0x7f5c5c0680e0 msgr2=0x7f5c5c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:17.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.716+0000 7f5c63bff700 1 -- 172.21.15.27:0/2065618686 shutdown_connections 2024-01-11T08:29:17.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.716+0000 7f5c63bff700 1 -- 172.21.15.27:0/2065618686 wait complete. 2024-01-11T08:29:17.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.718+0000 7f5c63bff700 1 Processor -- start 2024-01-11T08:29:17.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.718+0000 7f5c63bff700 1 -- start start 2024-01-11T08:29:17.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.719+0000 7f5c63bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c5c069d40 0x7f5c5c14c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:17.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.719+0000 7f5c63bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c5c0c8620 0x7f5c5c153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:17.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.719+0000 7f5c63bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c5c0c89f0 0x7f5c5c14d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:17.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.719+0000 7f5c63bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c14e6b0 con 0x7f5c5c0c89f0 2024-01-11T08:29:17.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.719+0000 7f5c63bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c14a5c0 con 0x7f5c5c069d40 2024-01-11T08:29:17.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.719+0000 7f5c63bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c0ccf20 con 0x7f5c5c0c8620 2024-01-11T08:29:17.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.720+0000 7f5c6119a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c5c0c8620 0x7f5c5c153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:17.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.720+0000 7f5c6119a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c5c0c8620 0x7f5c5c153170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:38598/0 (socket says 172.21.15.27:38598) 2024-01-11T08:29:17.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.720+0000 7f5c6119a700 1 -- 172.21.15.27:0/2552577796 learned_addr learned my addr 172.21.15.27:0/2552577796 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:17.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.720+0000 7f5c6219c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c5c0c89f0 0x7f5c5c14d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:17.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.720+0000 7f5c6199b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c5c069d40 0x7f5c5c14c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:17.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.720+0000 7f5c6119a700 1 -- 172.21.15.27:0/2552577796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c5c069d40 msgr2=0x7f5c5c14c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:17.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.720+0000 7f5c6119a700 1 --2- 172.21.15.27:0/2552577796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c5c069d40 0x7f5c5c14c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:17.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.720+0000 7f5c6119a700 1 -- 172.21.15.27:0/2552577796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c5c0c89f0 msgr2=0x7f5c5c14d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:17.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.720+0000 7f5c6119a700 1 --2- 172.21.15.27:0/2552577796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c5c0c89f0 0x7f5c5c14d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:17.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.720+0000 7f5c6119a700 1 -- 172.21.15.27:0/2552577796 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c0094f0 con 0x7f5c5c0c8620 2024-01-11T08:29:17.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.720+0000 7f5c6119a700 1 --2- 172.21.15.27:0/2552577796 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c5c0c8620 0x7f5c5c153170 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c00ec80 tx=0x7f5c4c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.720+0000 7f5c51ffb700 1 -- 172.21.15.27:0/2552577796 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c00c850 con 0x7f5c5c0c8620 2024-01-11T08:29:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.721+0000 7f5c51ffb700 1 -- 172.21.15.27:0/2552577796 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4c00c9b0 con 0x7f5c5c0c8620 2024-01-11T08:29:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.721+0000 7f5c51ffb700 1 -- 172.21.15.27:0/2552577796 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c00cd30 con 0x7f5c5c0c8620 2024-01-11T08:29:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.721+0000 7f5c63bff700 1 -- 172.21.15.27:0/2552577796 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c58009420 con 0x7f5c5c0c8620 2024-01-11T08:29:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.721+0000 7f5c63bff700 1 -- 172.21.15.27:0/2552577796 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5c14ce90 con 0x7f5c5c0c8620 2024-01-11T08:29:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.722+0000 7f5c63bff700 1 -- 172.21.15.27:0/2552577796 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c5c070d00 con 0x7f5c5c0c8620 2024-01-11T08:29:17.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.723+0000 7f5c51ffb700 1 -- 172.21.15.27:0/2552577796 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5c4c013680 con 0x7f5c5c0c8620 2024-01-11T08:29:17.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.724+0000 7f5c51ffb700 1 --2- 172.21.15.27:0/2552577796 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5c40066f30 0x7f5c400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.724+0000 7f5c51ffb700 1 -- 172.21.15.27:0/2552577796 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5597+0+0 (secure 0 0 0) 0x7f5c4c084760 con 0x7f5c5c0c8620 2024-01-11T08:29:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.724+0000 7f5c6199b700 1 --2- 172.21.15.27:0/2552577796 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5c40066f30 0x7f5c400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:17.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.725+0000 7f5c6199b700 1 --2- 172.21.15.27:0/2552577796 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5c40066f30 0x7f5c400693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5c580093f0 tx=0x7f5c58017000 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.729+0000 7f5c51ffb700 1 -- 172.21.15.27:0/2552577796 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5c4c051a40 con 0x7f5c5c0c8620 2024-01-11T08:29:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.987+0000 7f5c63bff700 1 -- 172.21.15.27:0/2552577796 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c5c0718d0 con 0x7f5c5c0c8620 2024-01-11T08:29:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.988+0000 7f5c51ffb700 1 -- 172.21.15.27:0/2552577796 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+3526 (secure 0 0 0) 0x7f5c4c017de0 con 0x7f5c5c0c8620 2024-01-11T08:29:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:17.988 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":61,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6608,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2001630215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2001630215},{"type":"v1","addr":"172.21.15.77:6834","nonce":2001630215}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6869,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1988301549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1988301549},{"type":"v1","addr":"172.21.15.27:6835","nonce":1988301549}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":7348,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/884469933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":884469933},{"type":"v1","addr":"172.21.15.77:6837","nonce":884469933}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":7584,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/4086481881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4086481881},{"type":"v1","addr":"172.21.15.27:6834","nonce":4086481881}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54}],"filesystems":[]} 2024-01-11T08:29:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.991+0000 7f5c63bff700 1 -- 172.21.15.27:0/2552577796 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5c40066f30 msgr2=0x7f5c400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.991+0000 7f5c63bff700 1 --2- 172.21.15.27:0/2552577796 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5c40066f30 0x7f5c400693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5c580093f0 tx=0x7f5c58017000 comp rx=0 tx=0).stop 2024-01-11T08:29:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.991+0000 7f5c63bff700 1 -- 172.21.15.27:0/2552577796 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c5c0c8620 msgr2=0x7f5c5c153170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.991+0000 7f5c63bff700 1 --2- 172.21.15.27:0/2552577796 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c5c0c8620 0x7f5c5c153170 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c00ec80 tx=0x7f5c4c009250 comp rx=0 tx=0).stop 2024-01-11T08:29:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.991+0000 7f5c63bff700 1 -- 172.21.15.27:0/2552577796 shutdown_connections 2024-01-11T08:29:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.991+0000 7f5c63bff700 1 --2- 172.21.15.27:0/2552577796 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5c40066f30 0x7f5c400693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.991+0000 7f5c63bff700 1 --2- 172.21.15.27:0/2552577796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c5c0c89f0 0x7f5c5c14d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.991+0000 7f5c63bff700 1 --2- 172.21.15.27:0/2552577796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c5c069d40 0x7f5c5c14c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.991+0000 7f5c63bff700 1 --2- 172.21.15.27:0/2552577796 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c5c0c8620 0x7f5c5c153170 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.991+0000 7f5c63bff700 1 -- 172.21.15.27:0/2552577796 >> 172.21.15.27:0/2552577796 conn(0x7f5c5c0680e0 msgr2=0x7f5c5c157890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.991+0000 7f5c63bff700 1 -- 172.21.15.27:0/2552577796 shutdown_connections 2024-01-11T08:29:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:17.991+0000 7f5c63bff700 1 -- 172.21.15.27:0/2552577796 wait complete. 2024-01-11T08:29:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 61 2024-01-11T08:29:18.009 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-11T08:29:18.009 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-11T08:29:18.009 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-11T08:29:18.009 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:29:18.010 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-11T08:29:18.010 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-11T08:29:18.010 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-11T08:29:18.010 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:29:18.010 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-11T08:29:18.010 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-11T08:29:18.011 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-11T08:29:18.011 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:29:18.011 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-11T08:29:18.011 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-11T08:29:18.011 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-11T08:29:18.011 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:29:18.023 INFO:tasks.ceph.mds.a:Stopped 2024-01-11T08:29:18.023 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-11T08:29:18.024 INFO:tasks.ceph.mds.c:Stopped 2024-01-11T08:29:18.025 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-11T08:29:18.030 INFO:tasks.ceph.mds.b:Stopped 2024-01-11T08:29:18.030 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-11T08:29:18.069 INFO:tasks.ceph.mds.a:Started 2024-01-11T08:29:18.069 DEBUG:teuthology.parallel:result is None 2024-01-11T08:29:18.072 INFO:tasks.ceph.mds.c:Started 2024-01-11T08:29:18.073 DEBUG:teuthology.parallel:result is None 2024-01-11T08:29:18.074 INFO:tasks.ceph.mds.b:Started 2024-01-11T08:29:18.075 DEBUG:teuthology.parallel:result is None 2024-01-11T08:29:18.215 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T08:29:18.215 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T08:29:18.217 INFO:tasks.ceph.mds.b.smithi077.stdout:starting mds.b at 2024-01-11T08:29:18.230 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-01-11T08:29:18.230 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-01-11T08:29:23.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:29:23.398 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:29:24.029 INFO:tasks.ceph.mds.d:Stopped 2024-01-11T08:29:24.029 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-11T08:29:24.034 INFO:tasks.ceph.mds.d:Started 2024-01-11T08:29:24.035 DEBUG:teuthology.parallel:result is None 2024-01-11T08:29:24.035 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-11T08:29:24.181 INFO:tasks.ceph.mds.d.smithi077.stdout:starting mds.d at 2024-01-11T08:29:24.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.248+0000 7f9f59bff700 1 Processor -- start 2024-01-11T08:29:24.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.248+0000 7f9f59bff700 1 -- start start 2024-01-11T08:29:24.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.249+0000 7f9f59bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9f541596a0 0x7f9f540abd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:24.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.249+0000 7f9f59bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f54068ba0 0x7f9f540af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:24.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.249+0000 7f9f59bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f540afdf0 0x7f9f54067f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:24.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.249+0000 7f9f59bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f540ac250 con 0x7f9f54068ba0 2024-01-11T08:29:24.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.249+0000 7f9f59bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f54067b00 con 0x7f9f541596a0 2024-01-11T08:29:24.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.249+0000 7f9f59bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5406bf60 con 0x7f9f540afdf0 2024-01-11T08:29:24.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.250+0000 7f9f52ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f54068ba0 0x7f9f540af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:24.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.250+0000 7f9f537fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9f541596a0 0x7f9f540abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:24.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.250+0000 7f9f53fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f540afdf0 0x7f9f54067f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:24.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.250+0000 7f9f52ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f54068ba0 0x7f9f540af8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59780/0 (socket says 172.21.15.27:59780) 2024-01-11T08:29:24.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.250+0000 7f9f52ffd700 1 -- 172.21.15.27:0/2471655833 learned_addr learned my addr 172.21.15.27:0/2471655833 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:24.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.250+0000 7f9f537fe700 1 -- 172.21.15.27:0/2471655833 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f540afdf0 msgr2=0x7f9f54067f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:24.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.250+0000 7f9f537fe700 1 --2- 172.21.15.27:0/2471655833 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f540afdf0 0x7f9f54067f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:24.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.250+0000 7f9f537fe700 1 -- 172.21.15.27:0/2471655833 >> v2:172.21.15.27:3300/0 conn(0x7f9f54068ba0 msgr2=0x7f9f540af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:24.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.250+0000 7f9f537fe700 1 --2- 172.21.15.27:0/2471655833 >> v2:172.21.15.27:3300/0 conn(0x7f9f54068ba0 0x7f9f540af8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:24.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.250+0000 7f9f537fe700 1 -- 172.21.15.27:0/2471655833 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5406c240 con 0x7f9f541596a0 2024-01-11T08:29:24.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.250+0000 7f9f52ffd700 1 --2- 172.21.15.27:0/2471655833 >> v2:172.21.15.27:3300/0 conn(0x7f9f54068ba0 0x7f9f540af8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:24.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.251+0000 7f9f537fe700 1 --2- 172.21.15.27:0/2471655833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f541596a0 0x7f9f540abd10 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9f540ac3e0 tx=0x7f9f440091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8bbc97c9e6651e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:24.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.251+0000 7f9f527fc700 1 -- 172.21.15.27:0/2471655833 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f44011630 con 0x7f9f541596a0 2024-01-11T08:29:24.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.251+0000 7f9f527fc700 1 -- 172.21.15.27:0/2471655833 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f440117b0 con 0x7f9f541596a0 2024-01-11T08:29:24.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.251+0000 7f9f527fc700 1 -- 172.21.15.27:0/2471655833 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f44011b70 con 0x7f9f541596a0 2024-01-11T08:29:24.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.251+0000 7f9f59bff700 1 -- 172.21.15.27:0/2471655833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f541596a0 msgr2=0x7f9f540abd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:24.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.251+0000 7f9f59bff700 1 --2- 172.21.15.27:0/2471655833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f541596a0 0x7f9f540abd10 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9f540ac3e0 tx=0x7f9f440091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:24.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.251+0000 7f9f59bff700 1 -- 172.21.15.27:0/2471655833 shutdown_connections 2024-01-11T08:29:24.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.251+0000 7f9f59bff700 1 --2- 172.21.15.27:0/2471655833 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f540afdf0 0x7f9f54067f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:24.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.251+0000 7f9f59bff700 1 --2- 172.21.15.27:0/2471655833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f541596a0 0x7f9f540abd10 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:24.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.251+0000 7f9f59bff700 1 --2- 172.21.15.27:0/2471655833 >> v2:172.21.15.27:3300/0 conn(0x7f9f54068ba0 0x7f9f540af8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:24.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.251+0000 7f9f59bff700 1 -- 172.21.15.27:0/2471655833 >> 172.21.15.27:0/2471655833 conn(0x7f9f5406a8b0 msgr2=0x7f9f540ae520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:24.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.251+0000 7f9f59bff700 1 -- 172.21.15.27:0/2471655833 shutdown_connections 2024-01-11T08:29:24.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.252+0000 7f9f59bff700 1 -- 172.21.15.27:0/2471655833 wait complete. 2024-01-11T08:29:24.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.253+0000 7f9f59bff700 1 Processor -- start 2024-01-11T08:29:24.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.253+0000 7f9f59bff700 1 -- start start 2024-01-11T08:29:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.254+0000 7f9f59bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f54068ba0 0x7f9f541558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.254+0000 7f9f59bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f540afdf0 0x7f9f54150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.254+0000 7f9f59bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f541596a0 0x7f9f54150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.254+0000 7f9f59bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f540acfc0 con 0x7f9f540afdf0 2024-01-11T08:29:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.254+0000 7f9f59bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5415b670 con 0x7f9f541596a0 2024-01-11T08:29:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.254+0000 7f9f59bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f540aec90 con 0x7f9f54068ba0 2024-01-11T08:29:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.254+0000 7f9f52ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f540afdf0 0x7f9f54150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.254+0000 7f9f52ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f540afdf0 0x7f9f54150220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59782/0 (socket says 172.21.15.27:59782) 2024-01-11T08:29:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.254+0000 7f9f52ffd700 1 -- 172.21.15.27:0/3832142605 learned_addr learned my addr 172.21.15.27:0/3832142605 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.255+0000 7f9f537fe700 1 --2- 172.21.15.27:0/3832142605 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f54068ba0 0x7f9f541558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.255+0000 7f9f53fff700 1 --2- 172.21.15.27:0/3832142605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f541596a0 0x7f9f54150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.255+0000 7f9f52ffd700 1 -- 172.21.15.27:0/3832142605 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f54068ba0 msgr2=0x7f9f541558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:24.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.255+0000 7f9f52ffd700 1 --2- 172.21.15.27:0/3832142605 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f54068ba0 0x7f9f541558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:24.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.255+0000 7f9f52ffd700 1 -- 172.21.15.27:0/3832142605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f541596a0 msgr2=0x7f9f54150760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:24.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.255+0000 7f9f52ffd700 1 --2- 172.21.15.27:0/3832142605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f541596a0 0x7f9f54150760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:24.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.255+0000 7f9f52ffd700 1 -- 172.21.15.27:0/3832142605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f480094f0 con 0x7f9f540afdf0 2024-01-11T08:29:24.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.255+0000 7f9f537fe700 1 --2- 172.21.15.27:0/3832142605 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f54068ba0 0x7f9f541558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:24.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.255+0000 7f9f52ffd700 1 --2- 172.21.15.27:0/3832142605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f540afdf0 0x7f9f54150220 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f9f4800ec50 tx=0x7f9f48009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:24.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.255+0000 7f9f3b7fe700 1 -- 172.21.15.27:0/3832142605 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4800c9d0 con 0x7f9f540afdf0 2024-01-11T08:29:24.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.255+0000 7f9f3b7fe700 1 -- 172.21.15.27:0/3832142605 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4800cb30 con 0x7f9f540afdf0 2024-01-11T08:29:24.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.255+0000 7f9f3b7fe700 1 -- 172.21.15.27:0/3832142605 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f48013470 con 0x7f9f540afdf0 2024-01-11T08:29:24.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.255+0000 7f9f59bff700 1 -- 172.21.15.27:0/3832142605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f44009420 con 0x7f9f540afdf0 2024-01-11T08:29:24.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.255+0000 7f9f59bff700 1 -- 172.21.15.27:0/3832142605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f540ac970 con 0x7f9f540afdf0 2024-01-11T08:29:24.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.256+0000 7f9f59bff700 1 -- 172.21.15.27:0/3832142605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f540ad170 con 0x7f9f540afdf0 2024-01-11T08:29:24.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.257+0000 7f9f3b7fe700 1 -- 172.21.15.27:0/3832142605 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9f48013860 con 0x7f9f540afdf0 2024-01-11T08:29:24.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.258+0000 7f9f3b7fe700 1 --2- 172.21.15.27:0/3832142605 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9f40066f30 0x7f9f400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:24.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.258+0000 7f9f3b7fe700 1 -- 172.21.15.27:0/3832142605 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5597+0+0 (secure 0 0 0) 0x7f9f48084cd0 con 0x7f9f540afdf0 2024-01-11T08:29:24.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.259+0000 7f9f537fe700 1 --2- 172.21.15.27:0/3832142605 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9f40066f30 0x7f9f400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:24.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.259+0000 7f9f537fe700 1 --2- 172.21.15.27:0/3832142605 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9f40066f30 0x7f9f400693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9f540ac3e0 tx=0x7f9f44017000 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:24.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.263+0000 7f9f3b7fe700 1 -- 172.21.15.27:0/3832142605 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9f540ad170 con 0x7f9f540afdf0 2024-01-11T08:29:24.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:24.480+0000 7f9f59bff700 1 -- 172.21.15.27:0/3832142605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f9f5404c3f0 con 0x7f9f540afdf0 2024-01-11T08:29:24.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-01-11T08:29:24.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-01-11T08:29:25.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:25.770+0000 7f9f3b7fe700 1 -- 172.21.15.27:0/3832142605 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v127) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9f48051f10 con 0x7f9f540afdf0 2024-01-11T08:29:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:25.773+0000 7f9f59bff700 1 -- 172.21.15.27:0/3832142605 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9f40066f30 msgr2=0x7f9f400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:25.773+0000 7f9f59bff700 1 --2- 172.21.15.27:0/3832142605 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9f40066f30 0x7f9f400693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9f540ac3e0 tx=0x7f9f44017000 comp rx=0 tx=0).stop 2024-01-11T08:29:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:25.773+0000 7f9f59bff700 1 -- 172.21.15.27:0/3832142605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f540afdf0 msgr2=0x7f9f54150220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:25.773+0000 7f9f59bff700 1 --2- 172.21.15.27:0/3832142605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f540afdf0 0x7f9f54150220 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f9f4800ec50 tx=0x7f9f48009250 comp rx=0 tx=0).stop 2024-01-11T08:29:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:25.773+0000 7f9f59bff700 1 -- 172.21.15.27:0/3832142605 shutdown_connections 2024-01-11T08:29:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:25.773+0000 7f9f59bff700 1 --2- 172.21.15.27:0/3832142605 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9f40066f30 0x7f9f400693e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:25.773+0000 7f9f59bff700 1 --2- 172.21.15.27:0/3832142605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f541596a0 0x7f9f54150760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:25.773+0000 7f9f59bff700 1 --2- 172.21.15.27:0/3832142605 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f54068ba0 0x7f9f541558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:25.773+0000 7f9f59bff700 1 --2- 172.21.15.27:0/3832142605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f540afdf0 0x7f9f54150220 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:25.773+0000 7f9f59bff700 1 -- 172.21.15.27:0/3832142605 >> 172.21.15.27:0/3832142605 conn(0x7f9f5406a8b0 msgr2=0x7f9f54067f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:25.774+0000 7f9f59bff700 1 -- 172.21.15.27:0/3832142605 shutdown_connections 2024-01-11T08:29:25.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:25.774+0000 7f9f59bff700 1 -- 172.21.15.27:0/3832142605 wait complete. 2024-01-11T08:29:25.774 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-11T08:29:25.793 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-11T08:29:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:25.999+0000 7fc967dff700 1 Processor -- start 2024-01-11T08:29:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:25.999+0000 7fc967dff700 1 -- start start 2024-01-11T08:29:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.000+0000 7fc967dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9601575e0 0x7fc960158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.001+0000 7fc967dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc96006b610 0x7fc9600b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.001+0000 7fc967dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9600b1800 0x7fc9600ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.001+0000 7fc967dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9600cb5e0 con 0x7fc9600b1800 2024-01-11T08:29:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.001+0000 7fc967dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9600b1c80 con 0x7fc96006b610 2024-01-11T08:29:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.001+0000 7fc967dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc960158b60 con 0x7fc9601575e0 2024-01-11T08:29:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.001+0000 7fc96639c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9600b1800 0x7fc9600ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.001+0000 7fc96639c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9600b1800 0x7fc9600ca1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59794/0 (socket says 172.21.15.27:59794) 2024-01-11T08:29:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.001+0000 7fc96639c700 1 -- 172.21.15.27:0/3569217351 learned_addr learned my addr 172.21.15.27:0/3569217351 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.001+0000 7fc965b9b700 1 --2- 172.21.15.27:0/3569217351 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9601575e0 0x7fc960158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.001+0000 7fc96539a700 1 --2- 172.21.15.27:0/3569217351 >> v2:172.21.15.77:3300/0 conn(0x7fc96006b610 0x7fc9600b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.001+0000 7fc96639c700 1 -- 172.21.15.27:0/3569217351 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9601575e0 msgr2=0x7fc960158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.001+0000 7fc96639c700 1 --2- 172.21.15.27:0/3569217351 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9601575e0 0x7fc960158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.001+0000 7fc96639c700 1 -- 172.21.15.27:0/3569217351 >> v2:172.21.15.77:3300/0 conn(0x7fc96006b610 msgr2=0x7fc9600b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.001+0000 7fc96639c700 1 --2- 172.21.15.27:0/3569217351 >> v2:172.21.15.77:3300/0 conn(0x7fc96006b610 0x7fc9600b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.001+0000 7fc96639c700 1 -- 172.21.15.27:0/3569217351 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9601599b0 con 0x7fc9600b1800 2024-01-11T08:29:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.001+0000 7fc96539a700 1 --2- 172.21.15.27:0/3569217351 >> v2:172.21.15.77:3300/0 conn(0x7fc96006b610 0x7fc9600b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.002+0000 7fc96639c700 1 --2- 172.21.15.27:0/3569217351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9600b1800 0x7fc9600ca1a0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fc9600694d0 tx=0x7fc958009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246c3b91e9c55537 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.002+0000 7fc964b99700 1 -- 172.21.15.27:0/3569217351 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95800e070 con 0x7fc9600b1800 2024-01-11T08:29:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.002+0000 7fc964b99700 1 -- 172.21.15.27:0/3569217351 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95800ba20 con 0x7fc9600b1800 2024-01-11T08:29:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.002+0000 7fc964b99700 1 -- 172.21.15.27:0/3569217351 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9580092e0 con 0x7fc9600b1800 2024-01-11T08:29:26.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.002+0000 7fc967dff700 1 -- 172.21.15.27:0/3569217351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9600b1800 msgr2=0x7fc9600ca1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:26.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.002+0000 7fc967dff700 1 --2- 172.21.15.27:0/3569217351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9600b1800 0x7fc9600ca1a0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fc9600694d0 tx=0x7fc958009840 comp rx=0 tx=0).stop 2024-01-11T08:29:26.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.003+0000 7fc967dff700 1 -- 172.21.15.27:0/3569217351 shutdown_connections 2024-01-11T08:29:26.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.003+0000 7fc967dff700 1 --2- 172.21.15.27:0/3569217351 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9601575e0 0x7fc960158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.003+0000 7fc967dff700 1 --2- 172.21.15.27:0/3569217351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9600b1800 0x7fc9600ca1a0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.003+0000 7fc967dff700 1 --2- 172.21.15.27:0/3569217351 >> v2:172.21.15.77:3300/0 conn(0x7fc96006b610 0x7fc9600b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.003+0000 7fc967dff700 1 -- 172.21.15.27:0/3569217351 >> 172.21.15.27:0/3569217351 conn(0x7fc9600680e0 msgr2=0x7fc960069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.003+0000 7fc967dff700 1 -- 172.21.15.27:0/3569217351 shutdown_connections 2024-01-11T08:29:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.003+0000 7fc967dff700 1 -- 172.21.15.27:0/3569217351 wait complete. 2024-01-11T08:29:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.004+0000 7fc967dff700 1 Processor -- start 2024-01-11T08:29:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.004+0000 7fc967dff700 1 -- start start 2024-01-11T08:29:26.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.005+0000 7fc967dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc96006b610 0x7fc9600c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:26.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.005+0000 7fc967dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9600b1800 0x7fc960143af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:26.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.005+0000 7fc967dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9601575e0 0x7fc960155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:26.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.005+0000 7fc967dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9600c5fe0 con 0x7fc9600b1800 2024-01-11T08:29:26.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.005+0000 7fc967dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9601568e0 con 0x7fc9601575e0 2024-01-11T08:29:26.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.005+0000 7fc967dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc960154350 con 0x7fc96006b610 2024-01-11T08:29:26.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.006+0000 7fc96539a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9600b1800 0x7fc960143af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:26.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.006+0000 7fc96539a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9600b1800 0x7fc960143af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59810/0 (socket says 172.21.15.27:59810) 2024-01-11T08:29:26.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.006+0000 7fc96539a700 1 -- 172.21.15.27:0/2999546858 learned_addr learned my addr 172.21.15.27:0/2999546858 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:26.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.006+0000 7fc965b9b700 1 --2- 172.21.15.27:0/2999546858 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc96006b610 0x7fc9600c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:26.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.006+0000 7fc96639c700 1 --2- 172.21.15.27:0/2999546858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9601575e0 0x7fc960155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:26.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.006+0000 7fc96539a700 1 -- 172.21.15.27:0/2999546858 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc96006b610 msgr2=0x7fc9600c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:26.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.006+0000 7fc96539a700 1 --2- 172.21.15.27:0/2999546858 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc96006b610 0x7fc9600c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.006+0000 7fc96539a700 1 -- 172.21.15.27:0/2999546858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9601575e0 msgr2=0x7fc960155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:26.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.006+0000 7fc96539a700 1 --2- 172.21.15.27:0/2999546858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9601575e0 0x7fc960155d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.006+0000 7fc96539a700 1 -- 172.21.15.27:0/2999546858 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc958009ae0 con 0x7fc9600b1800 2024-01-11T08:29:26.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.006+0000 7fc96539a700 1 --2- 172.21.15.27:0/2999546858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9600b1800 0x7fc960143af0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fc95000e8d0 tx=0x7fc950008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:26.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.006+0000 7fc955ffb700 1 -- 172.21.15.27:0/2999546858 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950013410 con 0x7fc9600b1800 2024-01-11T08:29:26.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.006+0000 7fc955ffb700 1 -- 172.21.15.27:0/2999546858 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc950013570 con 0x7fc9600b1800 2024-01-11T08:29:26.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.006+0000 7fc967dff700 1 -- 172.21.15.27:0/2999546858 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9600c56f0 con 0x7fc9600b1800 2024-01-11T08:29:26.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.007+0000 7fc955ffb700 1 -- 172.21.15.27:0/2999546858 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950013910 con 0x7fc9600b1800 2024-01-11T08:29:26.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.007+0000 7fc967dff700 1 -- 172.21.15.27:0/2999546858 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9600c5c70 con 0x7fc9600b1800 2024-01-11T08:29:26.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.008+0000 7fc967dff700 1 -- 172.21.15.27:0/2999546858 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc960158970 con 0x7fc9600b1800 2024-01-11T08:29:26.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.009+0000 7fc955ffb700 1 -- 172.21.15.27:0/2999546858 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc950013d40 con 0x7fc9600b1800 2024-01-11T08:29:26.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.009+0000 7fc955ffb700 1 --2- 172.21.15.27:0/2999546858 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc944066f30 0x7fc9440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:26.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.010+0000 7fc955ffb700 1 -- 172.21.15.27:0/2999546858 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 4909+0+0 (secure 0 0 0) 0x7fc950083fe0 con 0x7fc9600b1800 2024-01-11T08:29:26.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.010+0000 7fc965b9b700 1 --2- 172.21.15.27:0/2999546858 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc944066f30 0x7fc9440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:26.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.011+0000 7fc965b9b700 1 --2- 172.21.15.27:0/2999546858 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc944066f30 0x7fc9440693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc95c009d10 tx=0x7fc95c0091a0 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:26.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.014+0000 7fc955ffb700 1 -- 172.21.15.27:0/2999546858 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc9500514f0 con 0x7fc9600b1800 2024-01-11T08:29:26.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.300+0000 7fc967dff700 1 -- 172.21.15.27:0/2999546858 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc9600718d0 con 0x7fc9600b1800 2024-01-11T08:29:26.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.301+0000 7fc955ffb700 1 -- 172.21.15.27:0/2999546858 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fc950054b10 con 0x7fc9600b1800 2024-01-11T08:29:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:26.300 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T08:29:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-11T08:29:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-11T08:29:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lQm3yLhAAv6erz5hMfat8EUzRA5luMA==", 2024-01-11T08:29:26.300 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:29:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:29:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:29:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:29:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:29:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:29:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-11T08:29:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9l+FG8ChAACHndtVLx7qS3sidPniSycA==", 2024-01-11T08:29:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:29:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:29:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:29:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:29:26.301 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:29:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:29:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-11T08:29:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l1NnEOhAALHXsc5es7hufCthGdRzkyw==", 2024-01-11T08:29:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:29:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:29:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:29:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:29:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:29:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:29:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:26.302 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-11T08:29:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lruoWFhAAm8AmrZtTJhYI1+19F79RPQ==", 2024-01-11T08:29:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:29:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:29:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:29:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:29:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:29:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:29:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-11T08:29:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA2op9ltZygORAAZr1X1nCN0ce8rIVaGDpFyQ==", 2024-01-11T08:29:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:29:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:29:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:29:26.303 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:29:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:29:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-11T08:29:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA6op9lqSRPAhAAmtyCsyK5ZVGX2wfhOCMuFQ==", 2024-01-11T08:29:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:29:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:29:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:29:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:29:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:29:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:26.304 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-11T08:29:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA9op9lZcs8BhAA8j+U+jPOoy+fliVWZAPFYg==", 2024-01-11T08:29:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:29:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:29:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:29:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:29:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:29:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:26.305 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-11T08:29:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBAop9lFuCUChAAP6si/2W1mxH4ecQJvuumkw==", 2024-01-11T08:29:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:29:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:29:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:29:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:29:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:29:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-11T08:29:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBGop9lOpwBIRAAcpcsjwmX2IldcKHJWAgHuA==", 2024-01-11T08:29:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:29:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:29:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:29:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:29:26.306 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:29:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-11T08:29:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBJop9l4q8hJRAAK8mAYVRQMoWS6wBOU69HmA==", 2024-01-11T08:29:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:29:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:29:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:29:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:29:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:29:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-11T08:29:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBMop9lqMF9KBAAFgFFI2GWt9AGif4AoScThw==", 2024-01-11T08:29:26.307 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:29:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:29:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:29:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:29:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:29:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-11T08:29:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBPop9lN7eDLBAAKswPZFfs0g2AaSe6V+pbbQ==", 2024-01-11T08:29:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:29:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:29:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:29:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:29:26.308 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:29:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-11T08:29:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lT5okIBAABy8PbJJbA062VNmb1Y8qhg==", 2024-01-11T08:29:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:29:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:29:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-01-11T08:29:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-11T08:29:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:29:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-11T08:29:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lDf37LhAAqV+EabKXJEW1sS5xtu1tXw==", 2024-01-11T08:29:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:29:26.309 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:29:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:29:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:29:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:29:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:29:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-11T08:29:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lWayxJxAAcalM8QSOGtbCXxjYZGaNQA==", 2024-01-11T08:29:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:29:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:29:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:29:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:29:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:29:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.310 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:29:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-11T08:29:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lHD4tNhAAdmnFBmMrK+QhQYdpbzW0mg==", 2024-01-11T08:29:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:29:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:29:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:29:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:29:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:29:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:29:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-11T08:29:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAwop9lGPtDMxAAJ8FjrSEdCtQL8AhtAyqnTw==", 2024-01-11T08:29:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:29:26.311 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:29:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:29:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:29:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:29:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:29:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-11T08:29:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lXBZLJxAAXa1S8Vkm3MiV4igtNbg/1Q==", 2024-01-11T08:29:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:29:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:29:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T08:29:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:29:26.312 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:29:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-11T08:29:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l3ZGoHxAANqzytTTvWYSomG+RFsFDyg==", 2024-01-11T08:29:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:29:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:29:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T08:29:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:29:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:26.313 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-11T08:29:26.313 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T08:29:26.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.303+0000 7fc967dff700 1 -- 172.21.15.27:0/2999546858 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc944066f30 msgr2=0x7fc9440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:26.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.303+0000 7fc967dff700 1 --2- 172.21.15.27:0/2999546858 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc944066f30 0x7fc9440693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc95c009d10 tx=0x7fc95c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.303+0000 7fc967dff700 1 -- 172.21.15.27:0/2999546858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9600b1800 msgr2=0x7fc960143af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:26.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.303+0000 7fc967dff700 1 --2- 172.21.15.27:0/2999546858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9600b1800 0x7fc960143af0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fc95000e8d0 tx=0x7fc950008040 comp rx=0 tx=0).stop 2024-01-11T08:29:26.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.303+0000 7fc967dff700 1 -- 172.21.15.27:0/2999546858 shutdown_connections 2024-01-11T08:29:26.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.303+0000 7fc967dff700 1 --2- 172.21.15.27:0/2999546858 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc944066f30 0x7fc9440693e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.303+0000 7fc967dff700 1 --2- 172.21.15.27:0/2999546858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9601575e0 0x7fc960155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.303+0000 7fc967dff700 1 --2- 172.21.15.27:0/2999546858 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc96006b610 0x7fc9600c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.303+0000 7fc967dff700 1 --2- 172.21.15.27:0/2999546858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9600b1800 0x7fc960143af0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.303+0000 7fc967dff700 1 -- 172.21.15.27:0/2999546858 >> 172.21.15.27:0/2999546858 conn(0x7fc9600680e0 msgr2=0x7fc9600af0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:26.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.303+0000 7fc967dff700 1 -- 172.21.15.27:0/2999546858 shutdown_connections 2024-01-11T08:29:26.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.304+0000 7fc967dff700 1 -- 172.21.15.27:0/2999546858 wait complete. 2024-01-11T08:29:26.321 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:29:26.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.520+0000 7ff8d5bff700 1 Processor -- start 2024-01-11T08:29:26.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.521+0000 7ff8d5bff700 1 -- start start 2024-01-11T08:29:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.522+0000 7ff8d5bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff8d0159db0 0x7ff8d015a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.522+0000 7ff8d5bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d0157e30 con 0x7ff8d014fc20 2024-01-11T08:29:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.522+0000 7ff8d5bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d0157500 con 0x7ff8d01599e0 2024-01-11T08:29:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.523+0000 7ff8d5bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d01577e0 con 0x7ff8d0159db0 2024-01-11T08:29:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.523+0000 7ff8ceffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff8d0159db0 0x7ff8d015a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.523+0000 7ff8cffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff8d014fc20 0x7ff8d0150080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47148/0 (socket says 172.21.15.27:47148) 2024-01-11T08:29:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.523+0000 7ff8cffff700 1 -- 172.21.15.27:0/2952874269 learned_addr learned my addr 172.21.15.27:0/2952874269 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.523+0000 7ff8ceffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff8d0159db0 0x7ff8d015a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59826/0 (socket says 172.21.15.27:59826) 2024-01-11T08:29:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.523+0000 7ff8ceffd700 1 -- 172.21.15.27:0/2952874269 >> v1:172.21.15.77:6789/0 conn(0x7ff8d01599e0 legacy=0x7ff8d00b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.523+0000 7ff8ceffd700 1 -- 172.21.15.27:0/2952874269 >> v1:172.21.15.27:6789/0 conn(0x7ff8d014fc20 legacy=0x7ff8d0150080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.523+0000 7ff8ceffd700 1 -- 172.21.15.27:0/2952874269 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d0157ac0 con 0x7ff8d0159db0 2024-01-11T08:29:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.523+0000 7ff8ce7fc700 1 -- 172.21.15.27:0/2952874269 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077125244 0 0) 0x7ff8d0157e30 con 0x7ff8d014fc20 2024-01-11T08:29:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.524+0000 7ff8ceffd700 1 --2- 172.21.15.27:0/2952874269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d0159db0 0x7ff8d015a230 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400aa90 tx=0x7ff8c400a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3950a078d36d0c11 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.524+0000 7ff8ce7fc700 1 -- 172.21.15.27:0/2952874269 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c400ca40 con 0x7ff8d0159db0 2024-01-11T08:29:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.524+0000 7ff8ce7fc700 1 -- 172.21.15.27:0/2952874269 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c400cba0 con 0x7ff8d0159db0 2024-01-11T08:29:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.524+0000 7ff8ce7fc700 1 -- 172.21.15.27:0/2952874269 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4018400 con 0x7ff8d0159db0 2024-01-11T08:29:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.524+0000 7ff8d5bff700 1 -- 172.21.15.27:0/2952874269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d0159db0 msgr2=0x7ff8d015a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.524+0000 7ff8d5bff700 1 --2- 172.21.15.27:0/2952874269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d0159db0 0x7ff8d015a230 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400aa90 tx=0x7ff8c400a330 comp rx=0 tx=0).stop 2024-01-11T08:29:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.524+0000 7ff8d5bff700 1 -- 172.21.15.27:0/2952874269 shutdown_connections 2024-01-11T08:29:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.524+0000 7ff8d5bff700 1 --2- 172.21.15.27:0/2952874269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d0159db0 0x7ff8d015a230 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.524+0000 7ff8d5bff700 1 -- 172.21.15.27:0/2952874269 >> 172.21.15.27:0/2952874269 conn(0x7ff8d00680e0 msgr2=0x7ff8d0069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.525+0000 7ff8d5bff700 1 -- 172.21.15.27:0/2952874269 shutdown_connections 2024-01-11T08:29:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.525+0000 7ff8d5bff700 1 -- 172.21.15.27:0/2952874269 wait complete. 2024-01-11T08:29:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.527+0000 7ff8d5bff700 1 Processor -- start 2024-01-11T08:29:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.527+0000 7ff8d5bff700 1 -- start start 2024-01-11T08:29:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.528+0000 7ff8d5bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8d014fc20 0x7ff8d00ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.528+0000 7ff8d5bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d01599e0 0x7ff8d0147e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.528+0000 7ff8d5bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8d0159db0 0x7ff8d006de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.528+0000 7ff8d5bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d00aaa50 con 0x7ff8d01599e0 2024-01-11T08:29:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.528+0000 7ff8d5bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d00a9e40 con 0x7ff8d014fc20 2024-01-11T08:29:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.528+0000 7ff8d5bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d00aa120 con 0x7ff8d0159db0 2024-01-11T08:29:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.528+0000 7ff8ceffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d01599e0 0x7ff8d0147e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.528+0000 7ff8ceffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d01599e0 0x7ff8d0147e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59828/0 (socket says 172.21.15.27:59828) 2024-01-11T08:29:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.529+0000 7ff8ceffd700 1 -- 172.21.15.27:0/1875000178 learned_addr learned my addr 172.21.15.27:0/1875000178 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.529+0000 7ff8cf7fe700 1 --2- 172.21.15.27:0/1875000178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8d014fc20 0x7ff8d00ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.529+0000 7ff8cffff700 1 --2- 172.21.15.27:0/1875000178 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8d0159db0 0x7ff8d006de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.529+0000 7ff8ceffd700 1 -- 172.21.15.27:0/1875000178 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8d0159db0 msgr2=0x7ff8d006de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.529+0000 7ff8ceffd700 1 --2- 172.21.15.27:0/1875000178 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8d0159db0 0x7ff8d006de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.529+0000 7ff8ceffd700 1 -- 172.21.15.27:0/1875000178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8d014fc20 msgr2=0x7ff8d00ca7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.529+0000 7ff8ceffd700 1 --2- 172.21.15.27:0/1875000178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8d014fc20 0x7ff8d00ca7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.529+0000 7ff8ceffd700 1 -- 172.21.15.27:0/1875000178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c400a5d0 con 0x7ff8d01599e0 2024-01-11T08:29:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.529+0000 7ff8ceffd700 1 --2- 172.21.15.27:0/1875000178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d01599e0 0x7ff8d0147e70 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400a5a0 tx=0x7ff8c401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.530+0000 7ff8b77fe700 1 -- 172.21.15.27:0/1875000178 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4022400 con 0x7ff8d01599e0 2024-01-11T08:29:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.530+0000 7ff8b77fe700 1 -- 172.21.15.27:0/1875000178 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c4022560 con 0x7ff8d01599e0 2024-01-11T08:29:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.530+0000 7ff8d5bff700 1 -- 172.21.15.27:0/1875000178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d00aa400 con 0x7ff8d01599e0 2024-01-11T08:29:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.530+0000 7ff8b77fe700 1 -- 172.21.15.27:0/1875000178 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4022900 con 0x7ff8d01599e0 2024-01-11T08:29:26.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.530+0000 7ff8d5bff700 1 -- 172.21.15.27:0/1875000178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d00cad30 con 0x7ff8d01599e0 2024-01-11T08:29:26.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.531+0000 7ff8d5bff700 1 -- 172.21.15.27:0/1875000178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d00aabc0 con 0x7ff8d01599e0 2024-01-11T08:29:26.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.532+0000 7ff8b77fe700 1 -- 172.21.15.27:0/1875000178 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff8c4022cf0 con 0x7ff8d01599e0 2024-01-11T08:29:26.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.533+0000 7ff8b77fe700 1 --2- 172.21.15.27:0/1875000178 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff8bc066f30 0x7ff8bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:26.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.533+0000 7ff8b77fe700 1 -- 172.21.15.27:0/1875000178 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 4909+0+0 (secure 0 0 0) 0x7ff8c40857b0 con 0x7ff8d01599e0 2024-01-11T08:29:26.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.534+0000 7ff8cf7fe700 1 --2- 172.21.15.27:0/1875000178 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff8bc066f30 0x7ff8bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:26.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.534+0000 7ff8cf7fe700 1 --2- 172.21.15.27:0/1875000178 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff8bc066f30 0x7ff8bc0693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0003a80 tx=0x7ff8c0007ec0 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:26.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.538+0000 7ff8b77fe700 1 -- 172.21.15.27:0/1875000178 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff8c4052ca0 con 0x7ff8d01599e0 2024-01-11T08:29:26.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.741+0000 7ff8d5bff700 1 -- 172.21.15.27:0/1875000178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8d00718d0 con 0x7ff8d01599e0 2024-01-11T08:29:26.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.742+0000 7ff8b77fe700 1 -- 172.21.15.27:0/1875000178 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+12184 (secure 0 0 0) 0x7ff8c40562c0 con 0x7ff8d01599e0 2024-01-11T08:29:26.742 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:26.742 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":127,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:29:25.765823+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:29:26.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.744+0000 7ff8d5bff700 1 -- 172.21.15.27:0/1875000178 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff8bc066f30 msgr2=0x7ff8bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:26.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.744+0000 7ff8d5bff700 1 --2- 172.21.15.27:0/1875000178 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff8bc066f30 0x7ff8bc0693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0003a80 tx=0x7ff8c0007ec0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.744+0000 7ff8d5bff700 1 -- 172.21.15.27:0/1875000178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d01599e0 msgr2=0x7ff8d0147e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:26.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.744+0000 7ff8d5bff700 1 --2- 172.21.15.27:0/1875000178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d01599e0 0x7ff8d0147e70 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400a5a0 tx=0x7ff8c401a000 comp rx=0 tx=0).stop 2024-01-11T08:29:26.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.744+0000 7ff8d5bff700 1 -- 172.21.15.27:0/1875000178 shutdown_connections 2024-01-11T08:29:26.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.744+0000 7ff8d5bff700 1 --2- 172.21.15.27:0/1875000178 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff8bc066f30 0x7ff8bc0693e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.744+0000 7ff8d5bff700 1 --2- 172.21.15.27:0/1875000178 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8d0159db0 0x7ff8d006de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.744+0000 7ff8d5bff700 1 --2- 172.21.15.27:0/1875000178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8d014fc20 0x7ff8d00ca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.744+0000 7ff8d5bff700 1 --2- 172.21.15.27:0/1875000178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8d01599e0 0x7ff8d0147e70 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.744+0000 7ff8d5bff700 1 -- 172.21.15.27:0/1875000178 >> 172.21.15.27:0/1875000178 conn(0x7ff8d00680e0 msgr2=0x7ff8d014d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:26.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.745+0000 7ff8d5bff700 1 -- 172.21.15.27:0/1875000178 shutdown_connections 2024-01-11T08:29:26.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.745+0000 7ff8d5bff700 1 -- 172.21.15.27:0/1875000178 wait complete. 2024-01-11T08:29:26.762 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-11T08:29:26.763 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:29:26.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.968+0000 7f7e373ff700 1 Processor -- start 2024-01-11T08:29:26.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.968+0000 7f7e373ff700 1 -- start start 2024-01-11T08:29:26.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.970+0000 7f7e373ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7e3013fbb0 0x7f7e300ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:26.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.970+0000 7f7e373ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7e3013ff80 0x7f7e3015a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:26.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.970+0000 7f7e373ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e300c6b30 con 0x7f7e300cace0 2024-01-11T08:29:26.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.970+0000 7f7e373ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e300c6800 con 0x7f7e3013fbb0 2024-01-11T08:29:26.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.970+0000 7f7e373ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e300cb240 con 0x7f7e3013ff80 2024-01-11T08:29:26.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.970+0000 7f7e3519b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7e3013fbb0 0x7f7e300ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:26.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.970+0000 7f7e3599c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7e300cace0 0x7f7e3006e6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47152/0 (socket says 172.21.15.27:47152) 2024-01-11T08:29:26.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.970+0000 7f7e3519b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7e3013fbb0 0x7f7e300ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59836/0 (socket says 172.21.15.27:59836) 2024-01-11T08:29:26.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.970+0000 7f7e3599c700 1 -- 172.21.15.27:0/2401055494 learned_addr learned my addr 172.21.15.27:0/2401055494 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.970+0000 7f7e3499a700 1 --2- 172.21.15.27:0/2401055494 >> v2:172.21.15.77:3300/0 conn(0x7f7e3013ff80 0x7f7e3015a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.970+0000 7f7e3519b700 1 -- 172.21.15.27:0/2401055494 >> v2:172.21.15.77:3300/0 conn(0x7f7e3013ff80 msgr2=0x7f7e3015a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.970+0000 7f7e3519b700 1 --2- 172.21.15.27:0/2401055494 >> v2:172.21.15.77:3300/0 conn(0x7f7e3013ff80 0x7f7e3015a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.970+0000 7f7e3519b700 1 -- 172.21.15.27:0/2401055494 >> v1:172.21.15.27:6789/0 conn(0x7f7e300cace0 legacy=0x7f7e3006e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.971+0000 7f7e3519b700 1 -- 172.21.15.27:0/2401055494 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e300cb520 con 0x7f7e3013fbb0 2024-01-11T08:29:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.971+0000 7f7e27fff700 1 -- 172.21.15.27:0/2401055494 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458745048 0 0) 0x7f7e300c6b30 con 0x7f7e300cace0 2024-01-11T08:29:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.971+0000 7f7e3519b700 1 --2- 172.21.15.27:0/2401055494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e3013fbb0 0x7f7e300ca7a0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f7e3006bac0 tx=0x7f7e2c00b3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de2ff70445482933 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.971+0000 7f7e27fff700 1 -- 172.21.15.27:0/2401055494 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c005c10 con 0x7f7e3013fbb0 2024-01-11T08:29:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.971+0000 7f7e27fff700 1 -- 172.21.15.27:0/2401055494 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2c005d70 con 0x7f7e3013fbb0 2024-01-11T08:29:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.971+0000 7f7e27fff700 1 -- 172.21.15.27:0/2401055494 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c00db60 con 0x7f7e3013fbb0 2024-01-11T08:29:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.971+0000 7f7e373ff700 1 -- 172.21.15.27:0/2401055494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e3013fbb0 msgr2=0x7f7e300ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.972+0000 7f7e373ff700 1 --2- 172.21.15.27:0/2401055494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e3013fbb0 0x7f7e300ca7a0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f7e3006bac0 tx=0x7f7e2c00b3e0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.972+0000 7f7e373ff700 1 -- 172.21.15.27:0/2401055494 shutdown_connections 2024-01-11T08:29:26.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.972+0000 7f7e373ff700 1 --2- 172.21.15.27:0/2401055494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e3013fbb0 0x7f7e300ca7a0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.972+0000 7f7e373ff700 1 --2- 172.21.15.27:0/2401055494 >> v2:172.21.15.77:3300/0 conn(0x7f7e3013ff80 0x7f7e3015a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.972+0000 7f7e373ff700 1 -- 172.21.15.27:0/2401055494 >> 172.21.15.27:0/2401055494 conn(0x7f7e300680a0 msgr2=0x7f7e300cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:26.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.972+0000 7f7e373ff700 1 -- 172.21.15.27:0/2401055494 shutdown_connections 2024-01-11T08:29:26.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.972+0000 7f7e373ff700 1 -- 172.21.15.27:0/2401055494 wait complete. 2024-01-11T08:29:26.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.973+0000 7f7e373ff700 1 Processor -- start 2024-01-11T08:29:26.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.973+0000 7f7e373ff700 1 -- start start 2024-01-11T08:29:26.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.974+0000 7f7e373ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e300cace0 0x7f7e300aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:26.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.974+0000 7f7e373ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e3013fbb0 0x7f7e30166620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:26.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.974+0000 7f7e373ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e3013ff80 0x7f7e300c1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:26.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.974+0000 7f7e373ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e300aeb70 con 0x7f7e3013ff80 2024-01-11T08:29:26.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.974+0000 7f7e373ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e300afbd0 con 0x7f7e3013fbb0 2024-01-11T08:29:26.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.974+0000 7f7e373ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e300afeb0 con 0x7f7e300cace0 2024-01-11T08:29:26.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.975+0000 7f7e3599c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e3013ff80 0x7f7e300c1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:26.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.975+0000 7f7e3519b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e300cace0 0x7f7e300aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:26.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.975+0000 7f7e3499a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e3013fbb0 0x7f7e30166620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:26.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.975+0000 7f7e3519b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e300cace0 0x7f7e300aef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:40786/0 (socket says 172.21.15.27:40786) 2024-01-11T08:29:26.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.975+0000 7f7e3519b700 1 -- 172.21.15.27:0/3523621298 learned_addr learned my addr 172.21.15.27:0/3523621298 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:26.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.975+0000 7f7e3599c700 1 -- 172.21.15.27:0/3523621298 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e300cace0 msgr2=0x7f7e300aef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:26.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.975+0000 7f7e3599c700 1 --2- 172.21.15.27:0/3523621298 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e300cace0 0x7f7e300aef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.975+0000 7f7e3599c700 1 -- 172.21.15.27:0/3523621298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e3013fbb0 msgr2=0x7f7e30166620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:26.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.975+0000 7f7e3599c700 1 --2- 172.21.15.27:0/3523621298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e3013fbb0 0x7f7e30166620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:26.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.975+0000 7f7e3599c700 1 -- 172.21.15.27:0/3523621298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c00b660 con 0x7f7e3013ff80 2024-01-11T08:29:26.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.976+0000 7f7e3599c700 1 --2- 172.21.15.27:0/3523621298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e3013ff80 0x7f7e300c1c50 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800cbe0 tx=0x7f7e280019a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:26.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.976+0000 7f7e257fa700 1 -- 172.21.15.27:0/3523621298 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2800a8b0 con 0x7f7e3013ff80 2024-01-11T08:29:26.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.976+0000 7f7e257fa700 1 -- 172.21.15.27:0/3523621298 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2800aa30 con 0x7f7e3013ff80 2024-01-11T08:29:26.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.976+0000 7f7e257fa700 1 -- 172.21.15.27:0/3523621298 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2800adf0 con 0x7f7e3013ff80 2024-01-11T08:29:26.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.976+0000 7f7e373ff700 1 -- 172.21.15.27:0/3523621298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e300ae520 con 0x7f7e3013ff80 2024-01-11T08:29:26.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.976+0000 7f7e373ff700 1 -- 172.21.15.27:0/3523621298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e300af860 con 0x7f7e3013ff80 2024-01-11T08:29:26.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.977+0000 7f7e373ff700 1 -- 172.21.15.27:0/3523621298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e301654c0 con 0x7f7e3013ff80 2024-01-11T08:29:26.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.978+0000 7f7e257fa700 1 -- 172.21.15.27:0/3523621298 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7e28007300 con 0x7f7e3013ff80 2024-01-11T08:29:26.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.979+0000 7f7e257fa700 1 --2- 172.21.15.27:0/3523621298 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7e14066f30 0x7f7e140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.979+0000 7f7e257fa700 1 -- 172.21.15.27:0/3523621298 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 4909+0+0 (secure 0 0 0) 0x7f7e28084520 con 0x7f7e3013ff80 2024-01-11T08:29:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.979+0000 7f7e3519b700 1 --2- 172.21.15.27:0/3523621298 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7e14066f30 0x7f7e140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.980+0000 7f7e3519b700 1 --2- 172.21.15.27:0/3523621298 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7e14066f30 0x7f7e140693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c00b600 tx=0x7f7e2c01a000 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:26.983+0000 7f7e257fa700 1 -- 172.21.15.27:0/3523621298 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7e301654c0 con 0x7f7e3013ff80 2024-01-11T08:29:27.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.190+0000 7f7e373ff700 1 -- 172.21.15.27:0/3523621298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7e301654c0 con 0x7f7e3013ff80 2024-01-11T08:29:27.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.190+0000 7f7e257fa700 1 -- 172.21.15.27:0/3523621298 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v127) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7e301654c0 con 0x7f7e3013ff80 2024-01-11T08:29:27.190 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:29:27.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.193+0000 7f7e373ff700 1 -- 172.21.15.27:0/3523621298 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7e14066f30 msgr2=0x7f7e140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:27.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.193+0000 7f7e373ff700 1 --2- 172.21.15.27:0/3523621298 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7e14066f30 0x7f7e140693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c00b600 tx=0x7f7e2c01a000 comp rx=0 tx=0).stop 2024-01-11T08:29:27.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.193+0000 7f7e373ff700 1 -- 172.21.15.27:0/3523621298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e3013ff80 msgr2=0x7f7e300c1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:27.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.193+0000 7f7e373ff700 1 --2- 172.21.15.27:0/3523621298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e3013ff80 0x7f7e300c1c50 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800cbe0 tx=0x7f7e280019a0 comp rx=0 tx=0).stop 2024-01-11T08:29:27.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.193+0000 7f7e373ff700 1 -- 172.21.15.27:0/3523621298 shutdown_connections 2024-01-11T08:29:27.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.193+0000 7f7e373ff700 1 --2- 172.21.15.27:0/3523621298 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7e14066f30 0x7f7e140693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:27.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.193+0000 7f7e373ff700 1 --2- 172.21.15.27:0/3523621298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e3013ff80 0x7f7e300c1c50 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:27.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.193+0000 7f7e373ff700 1 --2- 172.21.15.27:0/3523621298 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e300cace0 0x7f7e300aef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:27.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.193+0000 7f7e373ff700 1 --2- 172.21.15.27:0/3523621298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e3013fbb0 0x7f7e30166620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:27.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.193+0000 7f7e373ff700 1 -- 172.21.15.27:0/3523621298 >> 172.21.15.27:0/3523621298 conn(0x7f7e300680a0 msgr2=0x7f7e3013e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:27.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.193+0000 7f7e373ff700 1 -- 172.21.15.27:0/3523621298 shutdown_connections 2024-01-11T08:29:27.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.193+0000 7f7e373ff700 1 -- 172.21.15.27:0/3523621298 wait complete. 2024-01-11T08:29:27.210 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-11T08:29:27.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.412+0000 7f9589dff700 1 Processor -- start 2024-01-11T08:29:27.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.412+0000 7f9589dff700 1 -- start start 2024-01-11T08:29:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.413+0000 7f9589dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f95840b03e0 0x7f95840b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.413+0000 7f9589dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f95840b07b0 0x7f95840b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.413+0000 7f9589dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95840cd730 con 0x7f9584159200 2024-01-11T08:29:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.413+0000 7f9589dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9584159d50 con 0x7f95840b07b0 2024-01-11T08:29:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.413+0000 7f9589dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95840ccb20 con 0x7f95840b03e0 2024-01-11T08:29:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.414+0000 7f9582ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f95840b07b0 0x7f95840b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.414+0000 7f9582ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f95840b07b0 0x7f95840b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59864/0 (socket says 172.21.15.27:59864) 2024-01-11T08:29:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.414+0000 7f9582ffd700 1 -- 172.21.15.27:0/3706220168 learned_addr learned my addr 172.21.15.27:0/3706220168 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.414+0000 7f95837fe700 1 --2- 172.21.15.27:0/3706220168 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f95840b03e0 0x7f95840b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:27.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.414+0000 7f9582ffd700 1 -- 172.21.15.27:0/3706220168 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f95840b03e0 msgr2=0x7f95840b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:27.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.414+0000 7f9582ffd700 1 --2- 172.21.15.27:0/3706220168 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f95840b03e0 0x7f95840b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:27.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.414+0000 7f9582ffd700 1 -- 172.21.15.27:0/3706220168 >> v1:172.21.15.77:6789/0 conn(0x7f9584159200 legacy=0x7f9584159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:27.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.414+0000 7f9582ffd700 1 -- 172.21.15.27:0/3706220168 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95840cce00 con 0x7f95840b07b0 2024-01-11T08:29:27.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.414+0000 7f95827fc700 1 -- 172.21.15.27:0/3706220168 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020021973 0 0) 0x7f95840cd730 con 0x7f9584159200 2024-01-11T08:29:27.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.415+0000 7f9582ffd700 1 --2- 172.21.15.27:0/3706220168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95840b07b0 0x7f95840b29c0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f95780099b0 tx=0x7f9578009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14978bcd8058e8ad server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:27.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.415+0000 7f95827fc700 1 -- 172.21.15.27:0/3706220168 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9578009ea0 con 0x7f95840b07b0 2024-01-11T08:29:27.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.415+0000 7f95827fc700 1 -- 172.21.15.27:0/3706220168 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957800b860 con 0x7f95840b07b0 2024-01-11T08:29:27.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.415+0000 7f95827fc700 1 -- 172.21.15.27:0/3706220168 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957800bc00 con 0x7f95840b07b0 2024-01-11T08:29:27.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.415+0000 7f9589dff700 1 -- 172.21.15.27:0/3706220168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95840b07b0 msgr2=0x7f95840b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:27.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.415+0000 7f9589dff700 1 --2- 172.21.15.27:0/3706220168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95840b07b0 0x7f95840b29c0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f95780099b0 tx=0x7f9578009250 comp rx=0 tx=0).stop 2024-01-11T08:29:27.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.415+0000 7f9589dff700 1 -- 172.21.15.27:0/3706220168 shutdown_connections 2024-01-11T08:29:27.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.415+0000 7f9589dff700 1 --2- 172.21.15.27:0/3706220168 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f95840b03e0 0x7f95840b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:27.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.415+0000 7f9589dff700 1 --2- 172.21.15.27:0/3706220168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95840b07b0 0x7f95840b29c0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:27.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.415+0000 7f9589dff700 1 -- 172.21.15.27:0/3706220168 >> 172.21.15.27:0/3706220168 conn(0x7f958406ed40 msgr2=0x7f958406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:27.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.415+0000 7f9589dff700 1 -- 172.21.15.27:0/3706220168 shutdown_connections 2024-01-11T08:29:27.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.416+0000 7f9589dff700 1 -- 172.21.15.27:0/3706220168 wait complete. 2024-01-11T08:29:27.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.417+0000 7f9589dff700 1 Processor -- start 2024-01-11T08:29:27.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.417+0000 7f9589dff700 1 -- start start 2024-01-11T08:29:27.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.418+0000 7f9589dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f95840b03e0 0x7f95840cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:27.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.418+0000 7f9589dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95840b07b0 0x7f95840a7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:27.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.418+0000 7f9589dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9584159200 0x7f9584157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:27.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.418+0000 7f9589dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95840c75c0 con 0x7f9584159200 2024-01-11T08:29:27.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.418+0000 7f9589dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95840c7810 con 0x7f95840b07b0 2024-01-11T08:29:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.418+0000 7f9589dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95840c7af0 con 0x7f95840b03e0 2024-01-11T08:29:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.418+0000 7f9583fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9584159200 0x7f9584157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.419+0000 7f9583fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9584159200 0x7f9584157740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59878/0 (socket says 172.21.15.27:59878) 2024-01-11T08:29:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.419+0000 7f9583fff700 1 -- 172.21.15.27:0/1789477400 learned_addr learned my addr 172.21.15.27:0/1789477400 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.419+0000 7f95837fe700 1 --2- 172.21.15.27:0/1789477400 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f95840b03e0 0x7f95840cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.419+0000 7f9582ffd700 1 --2- 172.21.15.27:0/1789477400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95840b07b0 0x7f95840a7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.419+0000 7f9583fff700 1 -- 172.21.15.27:0/1789477400 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f95840b03e0 msgr2=0x7f95840cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.419+0000 7f9583fff700 1 --2- 172.21.15.27:0/1789477400 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f95840b03e0 0x7f95840cbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:27.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.419+0000 7f9583fff700 1 -- 172.21.15.27:0/1789477400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95840b07b0 msgr2=0x7f95840a7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.419+0000 7f9583fff700 1 --2- 172.21.15.27:0/1789477400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95840b07b0 0x7f95840a7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.419+0000 7f9583fff700 1 -- 172.21.15.27:0/1789477400 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95780094f0 con 0x7f9584159200 2024-01-11T08:29:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.419+0000 7f9582ffd700 1 --2- 172.21.15.27:0/1789477400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95840b07b0 0x7f95840a7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.419+0000 7f9583fff700 1 --2- 172.21.15.27:0/1789477400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9584159200 0x7f9584157740 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f957c00c890 tx=0x7f957c001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.420+0000 7f956b7fe700 1 -- 172.21.15.27:0/1789477400 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c017070 con 0x7f9584159200 2024-01-11T08:29:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.420+0000 7f956b7fe700 1 -- 172.21.15.27:0/1789477400 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957c006ba0 con 0x7f9584159200 2024-01-11T08:29:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.420+0000 7f956b7fe700 1 -- 172.21.15.27:0/1789477400 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c012630 con 0x7f9584159200 2024-01-11T08:29:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.420+0000 7f9589dff700 1 -- 172.21.15.27:0/1789477400 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95840c7dd0 con 0x7f9584159200 2024-01-11T08:29:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.420+0000 7f9589dff700 1 -- 172.21.15.27:0/1789477400 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95840c8390 con 0x7f9584159200 2024-01-11T08:29:27.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.421+0000 7f9589dff700 1 -- 172.21.15.27:0/1789477400 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95840718d0 con 0x7f9584159200 2024-01-11T08:29:27.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.422+0000 7f956b7fe700 1 -- 172.21.15.27:0/1789477400 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f957c012a60 con 0x7f9584159200 2024-01-11T08:29:27.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.423+0000 7f956b7fe700 1 --2- 172.21.15.27:0/1789477400 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9570066f30 0x7f95700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:27.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.423+0000 7f956b7fe700 1 -- 172.21.15.27:0/1789477400 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 4909+0+0 (secure 0 0 0) 0x7f957c084090 con 0x7f9584159200 2024-01-11T08:29:27.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.423+0000 7f95837fe700 1 --2- 172.21.15.27:0/1789477400 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9570066f30 0x7f95700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:27.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.424+0000 7f95837fe700 1 --2- 172.21.15.27:0/1789477400 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9570066f30 0x7f95700693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f95840b5e00 tx=0x7f95740091a0 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.427+0000 7f956b7fe700 1 -- 172.21.15.27:0/1789477400 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f957c051580 con 0x7f9584159200 2024-01-11T08:29:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.633+0000 7f9589dff700 1 -- 172.21.15.27:0/1789477400 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f95840cd0e0 con 0x7f9584159200 2024-01-11T08:29:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.633+0000 7f956b7fe700 1 -- 172.21.15.27:0/1789477400 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v127) v1 ==== 84+0+65 (secure 0 0 0) 0x7f957c054ba0 con 0x7f9584159200 2024-01-11T08:29:27.632 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:27.632 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T08:29:27.632 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:27.632 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-11T08:29:27.633 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-11T08:29:27.633 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:27.633 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T08:29:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.635+0000 7f9589dff700 1 -- 172.21.15.27:0/1789477400 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9570066f30 msgr2=0x7f95700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.635+0000 7f9589dff700 1 --2- 172.21.15.27:0/1789477400 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9570066f30 0x7f95700693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f95840b5e00 tx=0x7f95740091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.635+0000 7f9589dff700 1 -- 172.21.15.27:0/1789477400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9584159200 msgr2=0x7f9584157740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.635+0000 7f9589dff700 1 --2- 172.21.15.27:0/1789477400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9584159200 0x7f9584157740 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f957c00c890 tx=0x7f957c001670 comp rx=0 tx=0).stop 2024-01-11T08:29:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.635+0000 7f9589dff700 1 -- 172.21.15.27:0/1789477400 shutdown_connections 2024-01-11T08:29:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.636+0000 7f9589dff700 1 --2- 172.21.15.27:0/1789477400 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9570066f30 0x7f95700693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.636+0000 7f9589dff700 1 --2- 172.21.15.27:0/1789477400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9584159200 0x7f9584157740 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.636+0000 7f9589dff700 1 --2- 172.21.15.27:0/1789477400 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f95840b03e0 0x7f95840cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.636+0000 7f9589dff700 1 --2- 172.21.15.27:0/1789477400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95840b07b0 0x7f95840a7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.636+0000 7f9589dff700 1 -- 172.21.15.27:0/1789477400 >> 172.21.15.27:0/1789477400 conn(0x7f958406ed40 msgr2=0x7f9584159640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.636+0000 7f9589dff700 1 -- 172.21.15.27:0/1789477400 shutdown_connections 2024-01-11T08:29:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.636+0000 7f9589dff700 1 -- 172.21.15.27:0/1789477400 wait complete. 2024-01-11T08:29:27.652 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-11T08:29:27.653 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-11T08:29:27.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.860+0000 7f99b9fff700 1 Processor -- start 2024-01-11T08:29:27.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.860+0000 7f99b9fff700 1 -- start start 2024-01-11T08:29:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.861+0000 7f99b9fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f99b4157440 0x7f99b40ae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.861+0000 7f99b9fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b406d420 con 0x7f99b4157810 2024-01-11T08:29:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.861+0000 7f99b9fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b406d700 con 0x7f99b4069d70 2024-01-11T08:29:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.861+0000 7f99b9fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b406d9e0 con 0x7f99b4157440 2024-01-11T08:29:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.861+0000 7f99b2ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f99b4157810 0x7f99b4069660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47154/0 (socket says 172.21.15.27:47154) 2024-01-11T08:29:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.861+0000 7f99b2ffd700 1 -- 172.21.15.27:0/2376805461 learned_addr learned my addr 172.21.15.27:0/2376805461 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.861+0000 7f99b37fe700 1 --2- 172.21.15.27:0/2376805461 >> v2:172.21.15.77:3300/0 conn(0x7f99b4157440 0x7f99b40ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:27.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.862+0000 7f99b27fc700 1 -- 172.21.15.27:0/2376805461 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1908436836 0 0) 0x7f99b406d420 con 0x7f99b4157810 2024-01-11T08:29:27.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.862+0000 7f99b27fc700 1 -- 172.21.15.27:0/2376805461 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f999c003570 con 0x7f99b4157810 2024-01-11T08:29:27.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.862+0000 7f99b27fc700 1 -- 172.21.15.27:0/2376805461 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832895447 0 0) 0x7f99b406d700 con 0x7f99b4069d70 2024-01-11T08:29:27.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.862+0000 7f99b37fe700 1 -- 172.21.15.27:0/2376805461 >> v1:172.21.15.77:6789/0 conn(0x7f99b4069d70 legacy=0x7f99b4161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:27.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.862+0000 7f99b37fe700 1 -- 172.21.15.27:0/2376805461 >> v1:172.21.15.27:6789/0 conn(0x7f99b4157810 legacy=0x7f99b4069660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:27.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.862+0000 7f99b37fe700 1 -- 172.21.15.27:0/2376805461 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b406dcc0 con 0x7f99b4157440 2024-01-11T08:29:27.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.862+0000 7f99b37fe700 1 --2- 172.21.15.27:0/2376805461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4157440 0x7f99b40ae430 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f99a40098d0 tx=0x7f99a4009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e23e872ee5439925 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:27.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.862+0000 7f99b27fc700 1 -- 172.21.15.27:0/2376805461 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a4005c10 con 0x7f99b4157440 2024-01-11T08:29:27.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.863+0000 7f99b27fc700 1 -- 172.21.15.27:0/2376805461 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a4005d70 con 0x7f99b4157440 2024-01-11T08:29:27.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.863+0000 7f99b27fc700 1 -- 172.21.15.27:0/2376805461 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a4006280 con 0x7f99b4157440 2024-01-11T08:29:27.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.863+0000 7f99b9fff700 1 -- 172.21.15.27:0/2376805461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4157440 msgr2=0x7f99b40ae430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:27.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.863+0000 7f99b9fff700 1 --2- 172.21.15.27:0/2376805461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4157440 0x7f99b40ae430 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f99a40098d0 tx=0x7f99a4009120 comp rx=0 tx=0).stop 2024-01-11T08:29:27.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.863+0000 7f99b9fff700 1 -- 172.21.15.27:0/2376805461 shutdown_connections 2024-01-11T08:29:27.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.863+0000 7f99b9fff700 1 --2- 172.21.15.27:0/2376805461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4157440 0x7f99b40ae430 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:27.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.863+0000 7f99b9fff700 1 -- 172.21.15.27:0/2376805461 >> 172.21.15.27:0/2376805461 conn(0x7f99b40680e0 msgr2=0x7f99b40cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:27.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.863+0000 7f99b9fff700 1 -- 172.21.15.27:0/2376805461 shutdown_connections 2024-01-11T08:29:27.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.863+0000 7f99b9fff700 1 -- 172.21.15.27:0/2376805461 wait complete. 2024-01-11T08:29:27.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.865+0000 7f99b9fff700 1 Processor -- start 2024-01-11T08:29:27.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.865+0000 7f99b9fff700 1 -- start start 2024-01-11T08:29:27.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.866+0000 7f99b9fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99b4069d70 0x7f99b4155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:27.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.866+0000 7f99b9fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99b4157440 0x7f99b414c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:27.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.866+0000 7f99b9fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4157810 0x7f99b40b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:27.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.866+0000 7f99b9fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b4155190 con 0x7f99b4069d70 2024-01-11T08:29:27.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.866+0000 7f99b9fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b4154580 con 0x7f99b4157810 2024-01-11T08:29:27.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.866+0000 7f99b9fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b4154860 con 0x7f99b4157440 2024-01-11T08:29:27.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.866+0000 7f99b37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99b4069d70 0x7f99b4155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:27.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.866+0000 7f99b37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99b4069d70 0x7f99b4155620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59882/0 (socket says 172.21.15.27:59882) 2024-01-11T08:29:27.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.866+0000 7f99b37fe700 1 -- 172.21.15.27:0/4004159850 learned_addr learned my addr 172.21.15.27:0/4004159850 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:27.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.866+0000 7f99b37fe700 1 -- 172.21.15.27:0/4004159850 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99b4157440 msgr2=0x7f99b414c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.866+0000 7f99b37fe700 1 --2- 172.21.15.27:0/4004159850 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99b4157440 0x7f99b414c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.866+0000 7f99b37fe700 1 -- 172.21.15.27:0/4004159850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4157810 msgr2=0x7f99b40b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.866+0000 7f99b3fff700 1 --2- 172.21.15.27:0/4004159850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4157810 0x7f99b40b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.867+0000 7f99b37fe700 1 --2- 172.21.15.27:0/4004159850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4157810 0x7f99b40b1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.867+0000 7f99b37fe700 1 -- 172.21.15.27:0/4004159850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a40093c0 con 0x7f99b4069d70 2024-01-11T08:29:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.867+0000 7f99b3fff700 1 --2- 172.21.15.27:0/4004159850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4157810 0x7f99b40b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:29:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.867+0000 7f99b37fe700 1 --2- 172.21.15.27:0/4004159850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99b4069d70 0x7f99b4155620 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f99a4009160 tx=0x7f99a4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.867+0000 7f999b7fe700 1 -- 172.21.15.27:0/4004159850 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a401d8b0 con 0x7f99b4069d70 2024-01-11T08:29:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.867+0000 7f999b7fe700 1 -- 172.21.15.27:0/4004159850 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a401da30 con 0x7f99b4069d70 2024-01-11T08:29:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.867+0000 7f999b7fe700 1 -- 172.21.15.27:0/4004159850 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a401ddf0 con 0x7f99b4069d70 2024-01-11T08:29:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.867+0000 7f99b9fff700 1 -- 172.21.15.27:0/4004159850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b4154b40 con 0x7f99b4069d70 2024-01-11T08:29:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.867+0000 7f99b9fff700 1 -- 172.21.15.27:0/4004159850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b414ca80 con 0x7f99b4069d70 2024-01-11T08:29:27.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.868+0000 7f99b9fff700 1 -- 172.21.15.27:0/4004159850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b4155470 con 0x7f99b4069d70 2024-01-11T08:29:27.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.869+0000 7f999b7fe700 1 -- 172.21.15.27:0/4004159850 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f99a401b690 con 0x7f99b4069d70 2024-01-11T08:29:27.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.870+0000 7f999b7fe700 1 --2- 172.21.15.27:0/4004159850 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f99a0066f30 0x7f99a00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:27.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.870+0000 7f999b7fe700 1 -- 172.21.15.27:0/4004159850 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 4909+0+0 (secure 0 0 0) 0x7f99a4085b60 con 0x7f99b4069d70 2024-01-11T08:29:27.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.871+0000 7f99b2ffd700 1 --2- 172.21.15.27:0/4004159850 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f99a0066f30 0x7f99a00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:27.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.872+0000 7f99b2ffd700 1 --2- 172.21.15.27:0/4004159850 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f99a0066f30 0x7f99a00693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f99a80058d0 tx=0x7f99a80079d0 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:27.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:27.874+0000 7f999b7fe700 1 -- 172.21.15.27:0/4004159850 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f99a40531a0 con 0x7f99b4069d70 2024-01-11T08:29:28.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:28.077+0000 7f99b9fff700 1 -- 172.21.15.27:0/4004159850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f99b414cc70 con 0x7f99b4069d70 2024-01-11T08:29:28.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:28.929+0000 7f999b7fe700 1 -- 172.21.15.27:0/4004159850 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v128) v1 ==== 138+0+0 (secure 0 0 0) 0x7f99a402a020 con 0x7f99b4069d70 2024-01-11T08:29:28.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:28.933+0000 7f99b9fff700 1 -- 172.21.15.27:0/4004159850 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f99a0066f30 msgr2=0x7f99a00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:28.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:28.933+0000 7f99b9fff700 1 --2- 172.21.15.27:0/4004159850 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f99a0066f30 0x7f99a00693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f99a80058d0 tx=0x7f99a80079d0 comp rx=0 tx=0).stop 2024-01-11T08:29:28.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:28.933+0000 7f99b9fff700 1 -- 172.21.15.27:0/4004159850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99b4069d70 msgr2=0x7f99b4155620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:28.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:28.933+0000 7f99b9fff700 1 --2- 172.21.15.27:0/4004159850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99b4069d70 0x7f99b4155620 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f99a4009160 tx=0x7f99a4016000 comp rx=0 tx=0).stop 2024-01-11T08:29:28.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:28.933+0000 7f99b9fff700 1 -- 172.21.15.27:0/4004159850 shutdown_connections 2024-01-11T08:29:28.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:28.933+0000 7f99b9fff700 1 --2- 172.21.15.27:0/4004159850 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f99a0066f30 0x7f99a00693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:28.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:28.933+0000 7f99b9fff700 1 --2- 172.21.15.27:0/4004159850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4157810 0x7f99b40b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:28.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:28.933+0000 7f99b9fff700 1 --2- 172.21.15.27:0/4004159850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99b4069d70 0x7f99b4155620 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:28.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:28.933+0000 7f99b9fff700 1 --2- 172.21.15.27:0/4004159850 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99b4157440 0x7f99b414c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:28.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:28.933+0000 7f99b9fff700 1 -- 172.21.15.27:0/4004159850 >> 172.21.15.27:0/4004159850 conn(0x7f99b40680e0 msgr2=0x7f99b414d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:28.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:28.934+0000 7f99b9fff700 1 -- 172.21.15.27:0/4004159850 shutdown_connections 2024-01-11T08:29:28.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:28.935+0000 7f99b9fff700 1 -- 172.21.15.27:0/4004159850 wait complete. 2024-01-11T08:29:28.935 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-11T08:29:28.955 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-11T08:29:29.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.166+0000 7f04dd7ff700 1 Processor -- start 2024-01-11T08:29:29.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.166+0000 7f04dd7ff700 1 -- start start 2024-01-11T08:29:29.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.167+0000 7f04dd7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f04d806bf70 0x7f04d80cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:29.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.167+0000 7f04dd7ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f04d815db00 0x7f04d815df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:29.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.167+0000 7f04dd7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d815d670 con 0x7f04d806bba0 2024-01-11T08:29:29.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.167+0000 7f04dd7ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04d8069fc0 con 0x7f04d815db00 2024-01-11T08:29:29.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.167+0000 7f04dd7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04d80cce10 con 0x7f04d806bf70 2024-01-11T08:29:29.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.168+0000 7f04d77fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f04d815db00 0x7f04d815df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:29.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.168+0000 7f04d6ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f04d806bba0 0x7f04d8159e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47166/0 (socket says 172.21.15.27:47166) 2024-01-11T08:29:29.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.168+0000 7f04d6ffd700 1 -- 172.21.15.27:0/691790893 learned_addr learned my addr 172.21.15.27:0/691790893 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:29.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.168+0000 7f04d67fc700 1 --2- 172.21.15.27:0/691790893 >> v2:172.21.15.77:3300/0 conn(0x7f04d806bf70 0x7f04d80cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:29.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.168+0000 7f04d77fe700 1 -- 172.21.15.27:0/691790893 >> v2:172.21.15.77:3300/0 conn(0x7f04d806bf70 msgr2=0x7f04d80cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:29.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.168+0000 7f04d77fe700 1 --2- 172.21.15.27:0/691790893 >> v2:172.21.15.77:3300/0 conn(0x7f04d806bf70 0x7f04d80cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:29.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.168+0000 7f04d77fe700 1 -- 172.21.15.27:0/691790893 >> v1:172.21.15.27:6789/0 conn(0x7f04d806bba0 legacy=0x7f04d8159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:29.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.168+0000 7f04d77fe700 1 -- 172.21.15.27:0/691790893 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d815a600 con 0x7f04d815db00 2024-01-11T08:29:29.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.168+0000 7f04d5ffb700 1 -- 172.21.15.27:0/691790893 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585929857 0 0) 0x7f04d815d670 con 0x7f04d806bba0 2024-01-11T08:29:29.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.168+0000 7f04d67fc700 1 --2- 172.21.15.27:0/691790893 >> v2:172.21.15.77:3300/0 conn(0x7f04d806bf70 0x7f04d80cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:29.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.169+0000 7f04d77fe700 1 --2- 172.21.15.27:0/691790893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04d815db00 0x7f04d815df20 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f04d806c770 tx=0x7f04d0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f8580566d44c515 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:29.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.169+0000 7f04d5ffb700 1 -- 172.21.15.27:0/691790893 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d000e070 con 0x7f04d815db00 2024-01-11T08:29:29.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.169+0000 7f04d5ffb700 1 -- 172.21.15.27:0/691790893 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d000bc30 con 0x7f04d815db00 2024-01-11T08:29:29.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.169+0000 7f04d5ffb700 1 -- 172.21.15.27:0/691790893 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0012430 con 0x7f04d815db00 2024-01-11T08:29:29.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.169+0000 7f04dd7ff700 1 -- 172.21.15.27:0/691790893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04d815db00 msgr2=0x7f04d815df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:29.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.169+0000 7f04dd7ff700 1 --2- 172.21.15.27:0/691790893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04d815db00 0x7f04d815df20 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f04d806c770 tx=0x7f04d0009780 comp rx=0 tx=0).stop 2024-01-11T08:29:29.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.169+0000 7f04dd7ff700 1 -- 172.21.15.27:0/691790893 shutdown_connections 2024-01-11T08:29:29.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.169+0000 7f04dd7ff700 1 --2- 172.21.15.27:0/691790893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04d815db00 0x7f04d815df20 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:29.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.169+0000 7f04dd7ff700 1 --2- 172.21.15.27:0/691790893 >> v2:172.21.15.77:3300/0 conn(0x7f04d806bf70 0x7f04d80cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:29.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.169+0000 7f04dd7ff700 1 -- 172.21.15.27:0/691790893 >> 172.21.15.27:0/691790893 conn(0x7f04d80ce3e0 msgr2=0x7f04d80ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:29.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.170+0000 7f04dd7ff700 1 -- 172.21.15.27:0/691790893 shutdown_connections 2024-01-11T08:29:29.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.170+0000 7f04dd7ff700 1 -- 172.21.15.27:0/691790893 wait complete. 2024-01-11T08:29:29.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.171+0000 7f04dd7ff700 1 Processor -- start 2024-01-11T08:29:29.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.171+0000 7f04dd7ff700 1 -- start start 2024-01-11T08:29:29.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.172+0000 7f04dd7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04d806bba0 0x7f04d8148f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.172+0000 7f04dd7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04d806bf70 0x7f04d81590e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.172+0000 7f04dd7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04d815db00 0x7f04d8159620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.172+0000 7f04dd7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d8149460 con 0x7f04d806bf70 2024-01-11T08:29:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.172+0000 7f04dd7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d80ae950 con 0x7f04d806bba0 2024-01-11T08:29:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.172+0000 7f04dd7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d80add40 con 0x7f04d815db00 2024-01-11T08:29:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.173+0000 7f04d67fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04d806bf70 0x7f04d81590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.173+0000 7f04d67fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04d806bf70 0x7f04d81590e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59908/0 (socket says 172.21.15.27:59908) 2024-01-11T08:29:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.173+0000 7f04d67fc700 1 -- 172.21.15.27:0/183919332 learned_addr learned my addr 172.21.15.27:0/183919332 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.173+0000 7f04d6ffd700 1 --2- 172.21.15.27:0/183919332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04d806bba0 0x7f04d8148f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.173+0000 7f04d77fe700 1 --2- 172.21.15.27:0/183919332 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04d815db00 0x7f04d8159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.173+0000 7f04d67fc700 1 -- 172.21.15.27:0/183919332 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04d815db00 msgr2=0x7f04d8159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.173+0000 7f04d67fc700 1 --2- 172.21.15.27:0/183919332 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04d815db00 0x7f04d8159620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:29.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.173+0000 7f04d67fc700 1 -- 172.21.15.27:0/183919332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04d806bba0 msgr2=0x7f04d8148f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:29.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.173+0000 7f04d67fc700 1 --2- 172.21.15.27:0/183919332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04d806bba0 0x7f04d8148f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:29.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.173+0000 7f04d67fc700 1 -- 172.21.15.27:0/183919332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d0009a20 con 0x7f04d806bf70 2024-01-11T08:29:29.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.173+0000 7f04d67fc700 1 --2- 172.21.15.27:0/183919332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04d806bf70 0x7f04d81590e0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f04cc00e8d0 tx=0x7f04cc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:29.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.173+0000 7f04beffd700 1 -- 172.21.15.27:0/183919332 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc013410 con 0x7f04d806bf70 2024-01-11T08:29:29.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.174+0000 7f04beffd700 1 -- 172.21.15.27:0/183919332 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04cc013570 con 0x7f04d806bf70 2024-01-11T08:29:29.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.174+0000 7f04dd7ff700 1 -- 172.21.15.27:0/183919332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d80ae020 con 0x7f04d806bf70 2024-01-11T08:29:29.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.174+0000 7f04beffd700 1 -- 172.21.15.27:0/183919332 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc013910 con 0x7f04d806bf70 2024-01-11T08:29:29.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.174+0000 7f04dd7ff700 1 -- 172.21.15.27:0/183919332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d80ae5e0 con 0x7f04d806bf70 2024-01-11T08:29:29.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.175+0000 7f04dd7ff700 1 -- 172.21.15.27:0/183919332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d80aeb00 con 0x7f04d806bf70 2024-01-11T08:29:29.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.176+0000 7f04beffd700 1 -- 172.21.15.27:0/183919332 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f04cc013d00 con 0x7f04d806bf70 2024-01-11T08:29:29.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.177+0000 7f04beffd700 1 --2- 172.21.15.27:0/183919332 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f04c4066f30 0x7f04c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:29.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.177+0000 7f04beffd700 1 -- 172.21.15.27:0/183919332 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5279+0+0 (secure 0 0 0) 0x7f04cc084390 con 0x7f04d806bf70 2024-01-11T08:29:29.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.177+0000 7f04d6ffd700 1 --2- 172.21.15.27:0/183919332 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f04c4066f30 0x7f04c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:29.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.178+0000 7f04d6ffd700 1 --2- 172.21.15.27:0/183919332 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f04c4066f30 0x7f04c40693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f04d80698c0 tx=0x7f04c8007d30 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:29.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.181+0000 7f04beffd700 1 -- 172.21.15.27:0/183919332 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f04cc051710 con 0x7f04d806bf70 2024-01-11T08:29:29.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.385+0000 7f04dd7ff700 1 -- 172.21.15.27:0/183919332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f04d80aeac0 con 0x7f04d806bf70 2024-01-11T08:29:29.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.932+0000 7f04beffd700 1 -- 172.21.15.27:0/183919332 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v129) v1 ==== 170+0+0 (secure 0 0 0) 0x7f04cc054d30 con 0x7f04d806bf70 2024-01-11T08:29:29.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.938+0000 7f04dd7ff700 1 -- 172.21.15.27:0/183919332 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f04c4066f30 msgr2=0x7f04c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:29.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.938+0000 7f04dd7ff700 1 --2- 172.21.15.27:0/183919332 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f04c4066f30 0x7f04c40693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f04d80698c0 tx=0x7f04c8007d30 comp rx=0 tx=0).stop 2024-01-11T08:29:29.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.938+0000 7f04dd7ff700 1 -- 172.21.15.27:0/183919332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04d806bf70 msgr2=0x7f04d81590e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:29.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.938+0000 7f04dd7ff700 1 --2- 172.21.15.27:0/183919332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04d806bf70 0x7f04d81590e0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f04cc00e8d0 tx=0x7f04cc008040 comp rx=0 tx=0).stop 2024-01-11T08:29:29.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.940+0000 7f04dd7ff700 1 -- 172.21.15.27:0/183919332 shutdown_connections 2024-01-11T08:29:29.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.940+0000 7f04dd7ff700 1 --2- 172.21.15.27:0/183919332 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f04c4066f30 0x7f04c40693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:29.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.940+0000 7f04dd7ff700 1 --2- 172.21.15.27:0/183919332 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04d815db00 0x7f04d8159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:29.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.940+0000 7f04dd7ff700 1 --2- 172.21.15.27:0/183919332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04d806bba0 0x7f04d8148f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:29.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.940+0000 7f04dd7ff700 1 --2- 172.21.15.27:0/183919332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04d806bf70 0x7f04d81590e0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:29.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.940+0000 7f04dd7ff700 1 -- 172.21.15.27:0/183919332 >> 172.21.15.27:0/183919332 conn(0x7f04d80ce3e0 msgr2=0x7f04d806d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:29.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.942+0000 7f04dd7ff700 1 -- 172.21.15.27:0/183919332 shutdown_connections 2024-01-11T08:29:29.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:29.942+0000 7f04dd7ff700 1 -- 172.21.15.27:0/183919332 wait complete. 2024-01-11T08:29:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-11T08:29:29.972 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-11T08:29:30.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.177+0000 7fc2d1dff700 1 Processor -- start 2024-01-11T08:29:30.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.178+0000 7fc2d1dff700 1 -- start start 2024-01-11T08:29:30.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.179+0000 7fc2d1dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc2cc15a1b0 0x7fc2cc15a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:30.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.179+0000 7fc2d1dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2cc0ccd70 0x7fc2cc0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:30.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.179+0000 7fc2d1dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2cc0cf850 con 0x7fc2cc0685c0 2024-01-11T08:29:30.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.179+0000 7fc2d1dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2cc0cec40 con 0x7fc2cc0ccd70 2024-01-11T08:29:30.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.179+0000 7fc2d1dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2cc0cef20 con 0x7fc2cc15a1b0 2024-01-11T08:29:30.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.179+0000 7fc2caffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2cc0ccd70 0x7fc2cc0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:30.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.179+0000 7fc2caffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2cc0ccd70 0x7fc2cc0cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59914/0 (socket says 172.21.15.27:59914) 2024-01-11T08:29:30.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.179+0000 7fc2caffd700 1 -- 172.21.15.27:0/1824345174 learned_addr learned my addr 172.21.15.27:0/1824345174 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:30.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.179+0000 7fc2cb7fe700 1 --2- 172.21.15.27:0/1824345174 >> v2:172.21.15.77:3300/0 conn(0x7fc2cc15a1b0 0x7fc2cc15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:30.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.180+0000 7fc2caffd700 1 -- 172.21.15.27:0/1824345174 >> v2:172.21.15.77:3300/0 conn(0x7fc2cc15a1b0 msgr2=0x7fc2cc15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:30.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.180+0000 7fc2caffd700 1 --2- 172.21.15.27:0/1824345174 >> v2:172.21.15.77:3300/0 conn(0x7fc2cc15a1b0 0x7fc2cc15a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:30.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.180+0000 7fc2caffd700 1 -- 172.21.15.27:0/1824345174 >> v1:172.21.15.27:6789/0 conn(0x7fc2cc0685c0 legacy=0x7fc2cc0689f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:30.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.180+0000 7fc2caffd700 1 -- 172.21.15.27:0/1824345174 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2cc0cf200 con 0x7fc2cc0ccd70 2024-01-11T08:29:30.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.180+0000 7fc2caffd700 1 --2- 172.21.15.27:0/1824345174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc0ccd70 0x7fc2cc0cd210 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc0d16f0 tx=0x7fc2c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab6e405978cf2368 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:30.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.180+0000 7fc2ca7fc700 1 -- 172.21.15.27:0/1824345174 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c0009e70 con 0x7fc2cc0ccd70 2024-01-11T08:29:30.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.180+0000 7fc2ca7fc700 1 -- 172.21.15.27:0/1824345174 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c000b860 con 0x7fc2cc0ccd70 2024-01-11T08:29:30.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.180+0000 7fc2ca7fc700 1 -- 172.21.15.27:0/1824345174 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c000bc00 con 0x7fc2cc0ccd70 2024-01-11T08:29:30.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.181+0000 7fc2d1dff700 1 -- 172.21.15.27:0/1824345174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc0ccd70 msgr2=0x7fc2cc0cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:30.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.181+0000 7fc2d1dff700 1 --2- 172.21.15.27:0/1824345174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc0ccd70 0x7fc2cc0cd210 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc0d16f0 tx=0x7fc2c0009250 comp rx=0 tx=0).stop 2024-01-11T08:29:30.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.181+0000 7fc2d1dff700 1 -- 172.21.15.27:0/1824345174 shutdown_connections 2024-01-11T08:29:30.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.181+0000 7fc2d1dff700 1 --2- 172.21.15.27:0/1824345174 >> v2:172.21.15.77:3300/0 conn(0x7fc2cc15a1b0 0x7fc2cc15a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:30.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.181+0000 7fc2d1dff700 1 --2- 172.21.15.27:0/1824345174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc0ccd70 0x7fc2cc0cd210 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:30.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.181+0000 7fc2d1dff700 1 -- 172.21.15.27:0/1824345174 >> 172.21.15.27:0/1824345174 conn(0x7fc2cc06ed40 msgr2=0x7fc2cc06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:30.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.181+0000 7fc2d1dff700 1 -- 172.21.15.27:0/1824345174 shutdown_connections 2024-01-11T08:29:30.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.181+0000 7fc2d1dff700 1 -- 172.21.15.27:0/1824345174 wait complete. 2024-01-11T08:29:30.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.182+0000 7fc2d1dff700 1 Processor -- start 2024-01-11T08:29:30.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.183+0000 7fc2d1dff700 1 -- start start 2024-01-11T08:29:30.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.183+0000 7fc2d1dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc0685c0 0x7fc2cc14fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:30.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.183+0000 7fc2d1dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2cc0ccd70 0x7fc2cc157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:30.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.183+0000 7fc2d1dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2cc15a1b0 0x7fc2cc0b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:30.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.183+0000 7fc2d1dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc150660 con 0x7fc2cc0685c0 2024-01-11T08:29:30.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.183+0000 7fc2d1dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc0b28e0 con 0x7fc2cc0ccd70 2024-01-11T08:29:30.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.183+0000 7fc2d1dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc158ec0 con 0x7fc2cc15a1b0 2024-01-11T08:29:30.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.184+0000 7fc2cb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc0685c0 0x7fc2cc14fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:30.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.184+0000 7fc2cb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc0685c0 0x7fc2cc14fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59930/0 (socket says 172.21.15.27:59930) 2024-01-11T08:29:30.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.184+0000 7fc2cb7fe700 1 -- 172.21.15.27:0/943111973 learned_addr learned my addr 172.21.15.27:0/943111973 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:30.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.184+0000 7fc2caffd700 1 --2- 172.21.15.27:0/943111973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2cc0ccd70 0x7fc2cc157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:30.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.184+0000 7fc2cbfff700 1 --2- 172.21.15.27:0/943111973 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2cc15a1b0 0x7fc2cc0b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:30.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.184+0000 7fc2cb7fe700 1 -- 172.21.15.27:0/943111973 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2cc15a1b0 msgr2=0x7fc2cc0b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:30.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.184+0000 7fc2cb7fe700 1 --2- 172.21.15.27:0/943111973 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2cc15a1b0 0x7fc2cc0b23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:30.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.184+0000 7fc2cb7fe700 1 -- 172.21.15.27:0/943111973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2cc0ccd70 msgr2=0x7fc2cc157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:30.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.184+0000 7fc2cb7fe700 1 --2- 172.21.15.27:0/943111973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2cc0ccd70 0x7fc2cc157f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:30.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.184+0000 7fc2cb7fe700 1 -- 172.21.15.27:0/943111973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c00094f0 con 0x7fc2cc0685c0 2024-01-11T08:29:30.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.184+0000 7fc2caffd700 1 --2- 172.21.15.27:0/943111973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2cc0ccd70 0x7fc2cc157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:30.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.184+0000 7fc2cb7fe700 1 --2- 172.21.15.27:0/943111973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc0685c0 0x7fc2cc14fdb0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00ecb0 tx=0x7fc2bc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:30.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.185+0000 7fc2b37fe700 1 -- 172.21.15.27:0/943111973 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc00c970 con 0x7fc2cc0685c0 2024-01-11T08:29:30.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.185+0000 7fc2b37fe700 1 -- 172.21.15.27:0/943111973 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2bc00cad0 con 0x7fc2cc0685c0 2024-01-11T08:29:30.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.185+0000 7fc2b37fe700 1 -- 172.21.15.27:0/943111973 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc00ce90 con 0x7fc2cc0685c0 2024-01-11T08:29:30.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.185+0000 7fc2d1dff700 1 -- 172.21.15.27:0/943111973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2cc158620 con 0x7fc2cc0685c0 2024-01-11T08:29:30.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.185+0000 7fc2d1dff700 1 -- 172.21.15.27:0/943111973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2cc158be0 con 0x7fc2cc0685c0 2024-01-11T08:29:30.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.186+0000 7fc2d1dff700 1 -- 172.21.15.27:0/943111973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2cc04c3c0 con 0x7fc2cc0685c0 2024-01-11T08:29:30.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.187+0000 7fc2b37fe700 1 -- 172.21.15.27:0/943111973 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc2bc013680 con 0x7fc2cc0685c0 2024-01-11T08:29:30.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.188+0000 7fc2b37fe700 1 --2- 172.21.15.27:0/943111973 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc2b8066f30 0x7fc2b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:30.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.188+0000 7fc2b37fe700 1 -- 172.21.15.27:0/943111973 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5661+0+0 (secure 0 0 0) 0x7fc2bc084fa0 con 0x7fc2cc0685c0 2024-01-11T08:29:30.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.189+0000 7fc2caffd700 1 --2- 172.21.15.27:0/943111973 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc2b8066f30 0x7fc2b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:30.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.189+0000 7fc2caffd700 1 --2- 172.21.15.27:0/943111973 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc2b8066f30 0x7fc2b80693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc0d16f0 tx=0x7fc2c0017000 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:30.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.192+0000 7fc2b37fe700 1 -- 172.21.15.27:0/943111973 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc2bc052220 con 0x7fc2cc0685c0 2024-01-11T08:29:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.440+0000 7fc2d1dff700 1 -- 172.21.15.27:0/943111973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc2cc069310 con 0x7fc2cc0685c0 2024-01-11T08:29:30.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.943+0000 7fc2b37fe700 1 -- 172.21.15.27:0/943111973 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v65) v1 ==== 174+0+0 (secure 0 0 0) 0x7fc2bc055840 con 0x7fc2cc0685c0 2024-01-11T08:29:30.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.947+0000 7fc2d1dff700 1 -- 172.21.15.27:0/943111973 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc2b8066f30 msgr2=0x7fc2b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:30.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.947+0000 7fc2d1dff700 1 --2- 172.21.15.27:0/943111973 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc2b8066f30 0x7fc2b80693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc0d16f0 tx=0x7fc2c0017000 comp rx=0 tx=0).stop 2024-01-11T08:29:30.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.947+0000 7fc2d1dff700 1 -- 172.21.15.27:0/943111973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc0685c0 msgr2=0x7fc2cc14fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:30.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.947+0000 7fc2d1dff700 1 --2- 172.21.15.27:0/943111973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc0685c0 0x7fc2cc14fdb0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00ecb0 tx=0x7fc2bc0091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:30.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.955+0000 7fc2d1dff700 1 -- 172.21.15.27:0/943111973 shutdown_connections 2024-01-11T08:29:30.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.955+0000 7fc2d1dff700 1 --2- 172.21.15.27:0/943111973 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc2b8066f30 0x7fc2b80693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:30.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.955+0000 7fc2d1dff700 1 --2- 172.21.15.27:0/943111973 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2cc15a1b0 0x7fc2cc0b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:30.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.955+0000 7fc2d1dff700 1 --2- 172.21.15.27:0/943111973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc0685c0 0x7fc2cc14fdb0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:30.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.955+0000 7fc2d1dff700 1 --2- 172.21.15.27:0/943111973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2cc0ccd70 0x7fc2cc157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:30.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.955+0000 7fc2d1dff700 1 -- 172.21.15.27:0/943111973 >> 172.21.15.27:0/943111973 conn(0x7fc2cc06ed40 msgr2=0x7fc2cc1563d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:30.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.956+0000 7fc2d1dff700 1 -- 172.21.15.27:0/943111973 shutdown_connections 2024-01-11T08:29:30.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:30.956+0000 7fc2d1dff700 1 -- 172.21.15.27:0/943111973 wait complete. 2024-01-11T08:29:30.959 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 16 and data pool 17 2024-01-11T08:29:30.985 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:29:31.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.186+0000 7fd9ecfff700 1 Processor -- start 2024-01-11T08:29:31.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.186+0000 7fd9ecfff700 1 -- start start 2024-01-11T08:29:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.187+0000 7fd9ecfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd9e80b03e0 0x7fd9e80b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.187+0000 7fd9ecfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd9e80b29c0 0x7fd9e8159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.187+0000 7fd9ecfff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e80cd730 con 0x7fd9e80b07b0 2024-01-11T08:29:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.187+0000 7fd9ecfff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e80b2de0 con 0x7fd9e80b29c0 2024-01-11T08:29:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.187+0000 7fd9ecfff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e80ccb20 con 0x7fd9e80b03e0 2024-01-11T08:29:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.188+0000 7fd9e6d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd9e80b29c0 0x7fd9e8159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.188+0000 7fd9e5d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd9e80b07b0 0x7fd9e8159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47192/0 (socket says 172.21.15.27:47192) 2024-01-11T08:29:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.188+0000 7fd9e5d9b700 1 -- 172.21.15.27:0/261291559 learned_addr learned my addr 172.21.15.27:0/261291559 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.188+0000 7fd9e659c700 1 --2- 172.21.15.27:0/261291559 >> v2:172.21.15.77:3300/0 conn(0x7fd9e80b03e0 0x7fd9e80b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.188+0000 7fd9e6d9d700 1 -- 172.21.15.27:0/261291559 >> v2:172.21.15.77:3300/0 conn(0x7fd9e80b03e0 msgr2=0x7fd9e80b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.188+0000 7fd9e6d9d700 1 --2- 172.21.15.27:0/261291559 >> v2:172.21.15.77:3300/0 conn(0x7fd9e80b03e0 0x7fd9e80b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.188+0000 7fd9e6d9d700 1 -- 172.21.15.27:0/261291559 >> v1:172.21.15.27:6789/0 conn(0x7fd9e80b07b0 legacy=0x7fd9e8159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.188+0000 7fd9e6d9d700 1 -- 172.21.15.27:0/261291559 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e80cce00 con 0x7fd9e80b29c0 2024-01-11T08:29:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.188+0000 7fd9e559a700 1 -- 172.21.15.27:0/261291559 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232358670 0 0) 0x7fd9e80cd730 con 0x7fd9e80b07b0 2024-01-11T08:29:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.189+0000 7fd9e6d9d700 1 --2- 172.21.15.27:0/261291559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e80b29c0 0x7fd9e8159910 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8089470 tx=0x7fd9e0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a6c6a53fdc211d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.189+0000 7fd9e559a700 1 -- 172.21.15.27:0/261291559 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e000e070 con 0x7fd9e80b29c0 2024-01-11T08:29:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.189+0000 7fd9e559a700 1 -- 172.21.15.27:0/261291559 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e000bc30 con 0x7fd9e80b29c0 2024-01-11T08:29:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.189+0000 7fd9e559a700 1 -- 172.21.15.27:0/261291559 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e0012430 con 0x7fd9e80b29c0 2024-01-11T08:29:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.189+0000 7fd9ecfff700 1 -- 172.21.15.27:0/261291559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e80b29c0 msgr2=0x7fd9e8159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.189+0000 7fd9ecfff700 1 --2- 172.21.15.27:0/261291559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e80b29c0 0x7fd9e8159910 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8089470 tx=0x7fd9e0009780 comp rx=0 tx=0).stop 2024-01-11T08:29:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.189+0000 7fd9ecfff700 1 -- 172.21.15.27:0/261291559 shutdown_connections 2024-01-11T08:29:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.189+0000 7fd9ecfff700 1 --2- 172.21.15.27:0/261291559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e80b29c0 0x7fd9e8159910 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.189+0000 7fd9ecfff700 1 --2- 172.21.15.27:0/261291559 >> v2:172.21.15.77:3300/0 conn(0x7fd9e80b03e0 0x7fd9e80b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.189+0000 7fd9ecfff700 1 -- 172.21.15.27:0/261291559 >> 172.21.15.27:0/261291559 conn(0x7fd9e806ed40 msgr2=0x7fd9e806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.190+0000 7fd9ecfff700 1 -- 172.21.15.27:0/261291559 shutdown_connections 2024-01-11T08:29:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.190+0000 7fd9ecfff700 1 -- 172.21.15.27:0/261291559 wait complete. 2024-01-11T08:29:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.191+0000 7fd9ecfff700 1 Processor -- start 2024-01-11T08:29:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.191+0000 7fd9ecfff700 1 -- start start 2024-01-11T08:29:31.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.192+0000 7fd9ecfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e80b03e0 0x7fd9e80ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.192+0000 7fd9ecfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e80b07b0 0x7fd9e8169300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.192+0000 7fd9ecfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd9e80b29c0 0x7fd9e8169840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.192+0000 7fd9ecfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e8159310 con 0x7fd9e80b07b0 2024-01-11T08:29:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.192+0000 7fd9ecfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e80a7070 con 0x7fd9e80b03e0 2024-01-11T08:29:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.192+0000 7fd9ecfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e80cefe0 con 0x7fd9e80b29c0 2024-01-11T08:29:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.192+0000 7fd9e5d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e80b07b0 0x7fd9e8169300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.192+0000 7fd9e5d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e80b07b0 0x7fd9e8169300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59950/0 (socket says 172.21.15.27:59950) 2024-01-11T08:29:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.192+0000 7fd9e5d9b700 1 -- 172.21.15.27:0/2108859764 learned_addr learned my addr 172.21.15.27:0/2108859764 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.192+0000 7fd9e659c700 1 --2- 172.21.15.27:0/2108859764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e80b03e0 0x7fd9e80ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.193+0000 7fd9e6d9d700 1 --2- 172.21.15.27:0/2108859764 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd9e80b29c0 0x7fd9e8169840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.193+0000 7fd9e5d9b700 1 -- 172.21.15.27:0/2108859764 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd9e80b29c0 msgr2=0x7fd9e8169840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.193+0000 7fd9e5d9b700 1 --2- 172.21.15.27:0/2108859764 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd9e80b29c0 0x7fd9e8169840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.193+0000 7fd9e5d9b700 1 -- 172.21.15.27:0/2108859764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e80b03e0 msgr2=0x7fd9e80ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.193+0000 7fd9e5d9b700 1 --2- 172.21.15.27:0/2108859764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e80b03e0 0x7fd9e80ceaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.193+0000 7fd9e5d9b700 1 -- 172.21.15.27:0/2108859764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e0009a20 con 0x7fd9e80b07b0 2024-01-11T08:29:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.193+0000 7fd9e5d9b700 1 --2- 172.21.15.27:0/2108859764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e80b07b0 0x7fd9e8169300 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00c7e0 tx=0x7fd9dc007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.193+0000 7fd9d67fc700 1 -- 172.21.15.27:0/2108859764 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc005b30 con 0x7fd9e80b07b0 2024-01-11T08:29:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.193+0000 7fd9d67fc700 1 -- 172.21.15.27:0/2108859764 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc005c90 con 0x7fd9e80b07b0 2024-01-11T08:29:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.193+0000 7fd9d67fc700 1 -- 172.21.15.27:0/2108859764 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc007300 con 0x7fd9e80b07b0 2024-01-11T08:29:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.194+0000 7fd9ecfff700 1 -- 172.21.15.27:0/2108859764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e8158de0 con 0x7fd9e80b07b0 2024-01-11T08:29:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.194+0000 7fd9ecfff700 1 -- 172.21.15.27:0/2108859764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e80cf540 con 0x7fd9e80b07b0 2024-01-11T08:29:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.195+0000 7fd9ecfff700 1 -- 172.21.15.27:0/2108859764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e80a71e0 con 0x7fd9e80b07b0 2024-01-11T08:29:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.196+0000 7fd9d67fc700 1 -- 172.21.15.27:0/2108859764 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd9dc01b3b0 con 0x7fd9e80b07b0 2024-01-11T08:29:31.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.197+0000 7fd9d67fc700 1 --2- 172.21.15.27:0/2108859764 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd9c8066f30 0x7fd9c80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:31.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.197+0000 7fd9d67fc700 1 -- 172.21.15.27:0/2108859764 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fd9dc083b30 con 0x7fd9e80b07b0 2024-01-11T08:29:31.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.198+0000 7fd9e659c700 1 --2- 172.21.15.27:0/2108859764 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd9c8066f30 0x7fd9c80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.198+0000 7fd9e659c700 1 --2- 172.21.15.27:0/2108859764 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd9c8066f30 0x7fd9c80693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8006fd0 tx=0x7fd9d8008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:31.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.201+0000 7fd9d67fc700 1 -- 172.21.15.27:0/2108859764 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd9e80a71e0 con 0x7fd9e80b07b0 2024-01-11T08:29:31.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.406+0000 7fd9ecfff700 1 -- 172.21.15.27:0/2108859764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9e80a71e0 con 0x7fd9e80b07b0 2024-01-11T08:29:31.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.407+0000 7fd9d67fc700 1 -- 172.21.15.27:0/2108859764 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+15417 (secure 0 0 0) 0x7fd9e80a71e0 con 0x7fd9e80b07b0 2024-01-11T08:29:31.406 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:31.406 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":130,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:29:30.441579+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:29:28.131181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-11T08:29:29.438397+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:29:31.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.409+0000 7fd9ecfff700 1 -- 172.21.15.27:0/2108859764 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd9c8066f30 msgr2=0x7fd9c80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:31.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.409+0000 7fd9ecfff700 1 --2- 172.21.15.27:0/2108859764 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd9c8066f30 0x7fd9c80693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8006fd0 tx=0x7fd9d8008040 comp rx=0 tx=0).stop 2024-01-11T08:29:31.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.409+0000 7fd9ecfff700 1 -- 172.21.15.27:0/2108859764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e80b07b0 msgr2=0x7fd9e8169300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:31.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.409+0000 7fd9ecfff700 1 --2- 172.21.15.27:0/2108859764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e80b07b0 0x7fd9e8169300 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00c7e0 tx=0x7fd9dc007df0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.409+0000 7fd9ecfff700 1 -- 172.21.15.27:0/2108859764 shutdown_connections 2024-01-11T08:29:31.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.409+0000 7fd9ecfff700 1 --2- 172.21.15.27:0/2108859764 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd9c8066f30 0x7fd9c80693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.409+0000 7fd9ecfff700 1 --2- 172.21.15.27:0/2108859764 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd9e80b29c0 0x7fd9e8169840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.409+0000 7fd9ecfff700 1 --2- 172.21.15.27:0/2108859764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e80b03e0 0x7fd9e80ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.409+0000 7fd9ecfff700 1 --2- 172.21.15.27:0/2108859764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e80b07b0 0x7fd9e8169300 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.409+0000 7fd9ecfff700 1 -- 172.21.15.27:0/2108859764 >> 172.21.15.27:0/2108859764 conn(0x7fd9e806ed40 msgr2=0x7fd9e81596d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:31.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.409+0000 7fd9ecfff700 1 -- 172.21.15.27:0/2108859764 shutdown_connections 2024-01-11T08:29:31.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.409+0000 7fd9ecfff700 1 -- 172.21.15.27:0/2108859764 wait complete. 2024-01-11T08:29:31.426 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:29:31.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.628+0000 7f08829ff700 1 Processor -- start 2024-01-11T08:29:31.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.628+0000 7f08829ff700 1 -- start start 2024-01-11T08:29:31.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.629+0000 7f08829ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f087c159650 0x7f087c06bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:31.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.629+0000 7f08829ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f087c159a20 0x7f087c0b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.629+0000 7f08829ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f087c0b2730 0x7f087c0b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.629+0000 7f08829ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f087c06e700 con 0x7f087c0b2730 2024-01-11T08:29:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.629+0000 7f08829ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f087c0b11a0 con 0x7f087c159a20 2024-01-11T08:29:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.629+0000 7f08829ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087c06c360 con 0x7f087c159650 2024-01-11T08:29:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.629+0000 7f0880f9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f087c0b2730 0x7f087c0b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.630+0000 7f0880f9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f087c0b2730 0x7f087c0b2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59966/0 (socket says 172.21.15.27:59966) 2024-01-11T08:29:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.630+0000 7f0880f9c700 1 -- 172.21.15.27:0/1174001344 learned_addr learned my addr 172.21.15.27:0/1174001344 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.630+0000 7f087bfff700 1 --2- 172.21.15.27:0/1174001344 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f087c159650 0x7f087c06bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.630+0000 7f087b7fe700 1 --2- 172.21.15.27:0/1174001344 >> v2:172.21.15.77:3300/0 conn(0x7f087c159a20 0x7f087c0b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.630+0000 7f0880f9c700 1 -- 172.21.15.27:0/1174001344 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f087c159650 msgr2=0x7f087c06bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.630+0000 7f0880f9c700 1 --2- 172.21.15.27:0/1174001344 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f087c159650 0x7f087c06bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.630+0000 7f0880f9c700 1 -- 172.21.15.27:0/1174001344 >> v2:172.21.15.77:3300/0 conn(0x7f087c159a20 msgr2=0x7f087c0b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.630+0000 7f0880f9c700 1 --2- 172.21.15.27:0/1174001344 >> v2:172.21.15.77:3300/0 conn(0x7f087c159a20 0x7f087c0b0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.630+0000 7f0880f9c700 1 -- 172.21.15.27:0/1174001344 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087c06c640 con 0x7f087c0b2730 2024-01-11T08:29:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.630+0000 7f0880f9c700 1 --2- 172.21.15.27:0/1174001344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f087c0b2730 0x7f087c0b2bb0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f087c06fe90 tx=0x7f0874009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=847db42e9f8c3463 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.630+0000 7f087affd700 1 -- 172.21.15.27:0/1174001344 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087400e070 con 0x7f087c0b2730 2024-01-11T08:29:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.631+0000 7f087affd700 1 -- 172.21.15.27:0/1174001344 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087400ba50 con 0x7f087c0b2730 2024-01-11T08:29:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.631+0000 7f087affd700 1 -- 172.21.15.27:0/1174001344 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08740092e0 con 0x7f087c0b2730 2024-01-11T08:29:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.631+0000 7f08829ff700 1 -- 172.21.15.27:0/1174001344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f087c0b2730 msgr2=0x7f087c0b2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.631+0000 7f08829ff700 1 --2- 172.21.15.27:0/1174001344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f087c0b2730 0x7f087c0b2bb0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f087c06fe90 tx=0x7f0874009840 comp rx=0 tx=0).stop 2024-01-11T08:29:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.631+0000 7f08829ff700 1 -- 172.21.15.27:0/1174001344 shutdown_connections 2024-01-11T08:29:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.631+0000 7f08829ff700 1 --2- 172.21.15.27:0/1174001344 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f087c159650 0x7f087c06bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.631+0000 7f08829ff700 1 --2- 172.21.15.27:0/1174001344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f087c0b2730 0x7f087c0b2bb0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.631+0000 7f08829ff700 1 --2- 172.21.15.27:0/1174001344 >> v2:172.21.15.77:3300/0 conn(0x7f087c159a20 0x7f087c0b0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.631+0000 7f08829ff700 1 -- 172.21.15.27:0/1174001344 >> 172.21.15.27:0/1174001344 conn(0x7f087c06b200 msgr2=0x7f087c0aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.631+0000 7f08829ff700 1 -- 172.21.15.27:0/1174001344 shutdown_connections 2024-01-11T08:29:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.631+0000 7f08829ff700 1 -- 172.21.15.27:0/1174001344 wait complete. 2024-01-11T08:29:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.633+0000 7f08829ff700 1 Processor -- start 2024-01-11T08:29:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.633+0000 7f08829ff700 1 -- start start 2024-01-11T08:29:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.634+0000 7f08829ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0b2730 0x7f087c0acf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.634+0000 7f08829ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f087c159650 0x7f087c14a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.634+0000 7f08829ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f087c159a20 0x7f087c0a9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.634+0000 7f08829ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c15ee70 con 0x7f087c159a20 2024-01-11T08:29:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.634+0000 7f08829ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c14a800 con 0x7f087c0b2730 2024-01-11T08:29:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.634+0000 7f08829ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087c15e260 con 0x7f087c159650 2024-01-11T08:29:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.634+0000 7f0880f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f087c159a20 0x7f087c0a9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.634+0000 7f0880f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f087c159a20 0x7f087c0a9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59980/0 (socket says 172.21.15.27:59980) 2024-01-11T08:29:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.634+0000 7f0880f9c700 1 -- 172.21.15.27:0/2602852248 learned_addr learned my addr 172.21.15.27:0/2602852248 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.634+0000 7f087bfff700 1 --2- 172.21.15.27:0/2602852248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0b2730 0x7f087c0acf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.634+0000 7f087b7fe700 1 --2- 172.21.15.27:0/2602852248 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f087c159650 0x7f087c14a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.635+0000 7f0880f9c700 1 -- 172.21.15.27:0/2602852248 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f087c159650 msgr2=0x7f087c14a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.635+0000 7f0880f9c700 1 --2- 172.21.15.27:0/2602852248 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f087c159650 0x7f087c14a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.635+0000 7f0880f9c700 1 -- 172.21.15.27:0/2602852248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0b2730 msgr2=0x7f087c0acf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.635+0000 7f0880f9c700 1 --2- 172.21.15.27:0/2602852248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0b2730 0x7f087c0acf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.635+0000 7f0880f9c700 1 -- 172.21.15.27:0/2602852248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0874009ae0 con 0x7f087c159a20 2024-01-11T08:29:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.635+0000 7f0880f9c700 1 --2- 172.21.15.27:0/2602852248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f087c159a20 0x7f087c0a9f00 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f087c063d40 tx=0x7f0874018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.635+0000 7f0863fff700 1 -- 172.21.15.27:0/2602852248 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08740032f0 con 0x7f087c159a20 2024-01-11T08:29:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.635+0000 7f0863fff700 1 -- 172.21.15.27:0/2602852248 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0874008d20 con 0x7f087c159a20 2024-01-11T08:29:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.635+0000 7f0863fff700 1 -- 172.21.15.27:0/2602852248 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0874026690 con 0x7f087c159a20 2024-01-11T08:29:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.635+0000 7f08829ff700 1 -- 172.21.15.27:0/2602852248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087c15e540 con 0x7f087c159a20 2024-01-11T08:29:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.636+0000 7f08829ff700 1 -- 172.21.15.27:0/2602852248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087c15eb00 con 0x7f087c159a20 2024-01-11T08:29:31.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.637+0000 7f08829ff700 1 -- 172.21.15.27:0/2602852248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f087c15efe0 con 0x7f087c159a20 2024-01-11T08:29:31.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.637+0000 7f0863fff700 1 -- 172.21.15.27:0/2602852248 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0874003040 con 0x7f087c159a20 2024-01-11T08:29:31.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.638+0000 7f0863fff700 1 --2- 172.21.15.27:0/2602852248 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0868066f30 0x7f08680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:31.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.638+0000 7f0863fff700 1 -- 172.21.15.27:0/2602852248 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f0874086490 con 0x7f087c159a20 2024-01-11T08:29:31.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.639+0000 7f087bfff700 1 --2- 172.21.15.27:0/2602852248 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0868066f30 0x7f08680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:31.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.639+0000 7f087bfff700 1 --2- 172.21.15.27:0/2602852248 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0868066f30 0x7f08680693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f086c006fd0 tx=0x7f086c008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:31.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.643+0000 7f0863fff700 1 -- 172.21.15.27:0/2602852248 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0874053620 con 0x7f087c159a20 2024-01-11T08:29:31.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.848+0000 7f08829ff700 1 -- 172.21.15.27:0/2602852248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f087c0718d0 con 0x7f087c159a20 2024-01-11T08:29:31.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.849+0000 7f0863fff700 1 -- 172.21.15.27:0/2602852248 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+15417 (secure 0 0 0) 0x7f0874056c40 con 0x7f087c159a20 2024-01-11T08:29:31.848 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:31.848 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":130,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:29:30.441579+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:29:28.131181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-11T08:29:29.438397+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:29:31.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.851+0000 7f08829ff700 1 -- 172.21.15.27:0/2602852248 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0868066f30 msgr2=0x7f08680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.851+0000 7f08829ff700 1 --2- 172.21.15.27:0/2602852248 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0868066f30 0x7f08680693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f086c006fd0 tx=0x7f086c008040 comp rx=0 tx=0).stop 2024-01-11T08:29:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.851+0000 7f08829ff700 1 -- 172.21.15.27:0/2602852248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f087c159a20 msgr2=0x7f087c0a9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.851+0000 7f08829ff700 1 --2- 172.21.15.27:0/2602852248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f087c159a20 0x7f087c0a9f00 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f087c063d40 tx=0x7f0874018000 comp rx=0 tx=0).stop 2024-01-11T08:29:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.851+0000 7f08829ff700 1 -- 172.21.15.27:0/2602852248 shutdown_connections 2024-01-11T08:29:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.851+0000 7f08829ff700 1 --2- 172.21.15.27:0/2602852248 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0868066f30 0x7f08680693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.851+0000 7f08829ff700 1 --2- 172.21.15.27:0/2602852248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f087c159a20 0x7f087c0a9f00 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.851+0000 7f08829ff700 1 --2- 172.21.15.27:0/2602852248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f087c0b2730 0x7f087c0acf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.851+0000 7f08829ff700 1 --2- 172.21.15.27:0/2602852248 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f087c159650 0x7f087c14a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.851+0000 7f08829ff700 1 -- 172.21.15.27:0/2602852248 >> 172.21.15.27:0/2602852248 conn(0x7f087c06b200 msgr2=0x7f087c155b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.852+0000 7f08829ff700 1 -- 172.21.15.27:0/2602852248 shutdown_connections 2024-01-11T08:29:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:31.852+0000 7f08829ff700 1 -- 172.21.15.27:0/2602852248 wait complete. 2024-01-11T08:29:31.869 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-11T08:29:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.076+0000 7f5fb17ff700 1 Processor -- start 2024-01-11T08:29:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.076+0000 7f5fb17ff700 1 -- start start 2024-01-11T08:29:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.077+0000 7f5fb17ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5fac13ff80 0x7f5fac15a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.077+0000 7f5fb17ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fac0c6b30 con 0x7f5fac15a560 2024-01-11T08:29:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.077+0000 7f5fb17ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fac0c6800 con 0x7f5fac13fbb0 2024-01-11T08:29:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.077+0000 7f5fb17ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fac15a960 con 0x7f5fac13ff80 2024-01-11T08:29:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.078+0000 7f5fab7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5fac15a560 0x7f5fac0c6fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47202/0 (socket says 172.21.15.27:47202) 2024-01-11T08:29:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.078+0000 7f5fab7fe700 1 -- 172.21.15.27:0/3358197873 learned_addr learned my addr 172.21.15.27:0/3358197873 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:32.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.078+0000 7f5faa7fc700 1 --2- 172.21.15.27:0/3358197873 >> v2:172.21.15.77:3300/0 conn(0x7f5fac13ff80 0x7f5fac15a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.078+0000 7f5fa9ffb700 1 -- 172.21.15.27:0/3358197873 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923008035 0 0) 0x7f5fac0c6b30 con 0x7f5fac15a560 2024-01-11T08:29:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.078+0000 7f5fa9ffb700 1 -- 172.21.15.27:0/3358197873 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f94003570 con 0x7f5fac15a560 2024-01-11T08:29:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.078+0000 7f5fa9ffb700 1 -- 172.21.15.27:0/3358197873 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2139116472 0 0) 0x7f5f94003570 con 0x7f5fac15a560 2024-01-11T08:29:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.078+0000 7f5fa9ffb700 1 -- 172.21.15.27:0/3358197873 >> v2:172.21.15.77:3300/0 conn(0x7f5fac13ff80 msgr2=0x7f5fac15a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.078+0000 7f5fa9ffb700 1 --2- 172.21.15.27:0/3358197873 >> v2:172.21.15.77:3300/0 conn(0x7f5fac13ff80 0x7f5fac15a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.078+0000 7f5fa9ffb700 1 -- 172.21.15.27:0/3358197873 >> v1:172.21.15.77:6789/0 conn(0x7f5fac13fbb0 legacy=0x7f5fac0ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.078+0000 7f5fa9ffb700 1 -- 172.21.15.27:0/3358197873 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fac0cb240 con 0x7f5fac15a560 2024-01-11T08:29:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.078+0000 7f5fa9ffb700 1 -- 172.21.15.27:0/3358197873 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f5fa40017f0 con 0x7f5fac15a560 2024-01-11T08:29:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.078+0000 7f5faa7fc700 1 --2- 172.21.15.27:0/3358197873 >> v2:172.21.15.77:3300/0 conn(0x7f5fac13ff80 0x7f5fac15a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.078+0000 7f5fa9ffb700 1 -- 172.21.15.27:0/3358197873 >> v1:172.21.15.27:6789/0 conn(0x7f5fac15a560 legacy=0x7f5fac0c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.078+0000 7f5fa9ffb700 1 --2- 172.21.15.27:0/3358197873 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f94003a60 0x7f5f94003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.078+0000 7f5fa9ffb700 1 --2- 172.21.15.27:0/3358197873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f94004810 0x7f5f94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.078+0000 7f5fa9ffb700 1 --2- 172.21.15.27:0/3358197873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f940051c0 0x7f5f9400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.079+0000 7f5fa9ffb700 1 -- 172.21.15.27:0/3358197873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fac15a960 con 0x7f5f94004810 2024-01-11T08:29:32.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.079+0000 7f5fa9ffb700 1 -- 172.21.15.27:0/3358197873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9400c0d0 con 0x7f5f940051c0 2024-01-11T08:29:32.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.079+0000 7f5fa9ffb700 1 -- 172.21.15.27:0/3358197873 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9400e4a0 con 0x7f5f94003a60 2024-01-11T08:29:32.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.079+0000 7f5faa7fc700 1 --2- 172.21.15.27:0/3358197873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f94004810 0x7f5f94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:32.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.079+0000 7f5faaffd700 1 --2- 172.21.15.27:0/3358197873 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f94003a60 0x7f5f94003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:32.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.079+0000 7f5fab7fe700 1 --2- 172.21.15.27:0/3358197873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f940051c0 0x7f5f9400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:32.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.079+0000 7f5faa7fc700 1 -- 172.21.15.27:0/3358197873 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f94003a60 msgr2=0x7f5f94003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:32.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.079+0000 7f5faa7fc700 1 --2- 172.21.15.27:0/3358197873 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f94003a60 0x7f5f94003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:32.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.079+0000 7f5faa7fc700 1 -- 172.21.15.27:0/3358197873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f940051c0 msgr2=0x7f5f9400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:32.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.079+0000 7f5faa7fc700 1 --2- 172.21.15.27:0/3358197873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f940051c0 0x7f5f9400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:32.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.079+0000 7f5faa7fc700 1 -- 172.21.15.27:0/3358197873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f94003570 con 0x7f5f94004810 2024-01-11T08:29:32.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.079+0000 7f5fab7fe700 1 -- 172.21.15.27:0/3358197873 reap_dead start 2024-01-11T08:29:32.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.079+0000 7f5faa7fc700 1 --2- 172.21.15.27:0/3358197873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f94004810 0x7f5f94004c80 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0009df0 tx=0x7f5fa00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c98e3c250be7b0ee server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:32.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.080+0000 7f5fa9ffb700 1 -- 172.21.15.27:0/3358197873 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0009740 con 0x7f5f94004810 2024-01-11T08:29:32.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.080+0000 7f5fa9ffb700 1 -- 172.21.15.27:0/3358197873 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa000e920 con 0x7f5f94004810 2024-01-11T08:29:32.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.080+0000 7f5fa9ffb700 1 -- 172.21.15.27:0/3358197873 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa000ece0 con 0x7f5f94004810 2024-01-11T08:29:32.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.080+0000 7f5fb17ff700 1 -- 172.21.15.27:0/3358197873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f94004810 msgr2=0x7f5f94004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.080+0000 7f5fb17ff700 1 --2- 172.21.15.27:0/3358197873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f94004810 0x7f5f94004c80 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0009df0 tx=0x7f5fa00091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.080+0000 7f5fb17ff700 1 -- 172.21.15.27:0/3358197873 shutdown_connections 2024-01-11T08:29:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.080+0000 7f5fb17ff700 1 --2- 172.21.15.27:0/3358197873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f94004810 0x7f5f94004c80 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.080+0000 7f5fb17ff700 1 -- 172.21.15.27:0/3358197873 >> 172.21.15.27:0/3358197873 conn(0x7f5fac0680a0 msgr2=0x7f5fac0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.080+0000 7f5fb17ff700 1 -- 172.21.15.27:0/3358197873 shutdown_connections 2024-01-11T08:29:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.080+0000 7f5fb17ff700 1 -- 172.21.15.27:0/3358197873 wait complete. 2024-01-11T08:29:32.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.082+0000 7f5fb17ff700 1 Processor -- start 2024-01-11T08:29:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.082+0000 7f5fb17ff700 1 -- start start 2024-01-11T08:29:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.083+0000 7f5fb17ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f94004810 0x7f5fac155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.083+0000 7f5fb17ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fac15ad70 0x7f5fac15b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.083+0000 7f5fb17ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fac0c4320 0x7f5fac0c4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.083+0000 7f5fb17ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fac0c6110 con 0x7f5fac15ad70 2024-01-11T08:29:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.083+0000 7f5fb17ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fac15b720 con 0x7f5fac0c4320 2024-01-11T08:29:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.083+0000 7f5fb17ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fac0c5070 con 0x7f5f94004810 2024-01-11T08:29:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.083+0000 7f5faa7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fac15ad70 0x7f5fac15b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.083+0000 7f5faa7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fac15ad70 0x7f5fac15b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59996/0 (socket says 172.21.15.27:59996) 2024-01-11T08:29:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.083+0000 7f5faa7fc700 1 -- 172.21.15.27:0/1145903768 learned_addr learned my addr 172.21.15.27:0/1145903768 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.083+0000 7f5faaffd700 1 --2- 172.21.15.27:0/1145903768 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f94004810 0x7f5fac155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.083+0000 7f5fab7fe700 1 --2- 172.21.15.27:0/1145903768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fac0c4320 0x7f5fac0c4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.084+0000 7f5faa7fc700 1 -- 172.21.15.27:0/1145903768 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f94004810 msgr2=0x7f5fac155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:32.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.084+0000 7f5faa7fc700 1 --2- 172.21.15.27:0/1145903768 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f94004810 0x7f5fac155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:32.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.084+0000 7f5faa7fc700 1 -- 172.21.15.27:0/1145903768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fac0c4320 msgr2=0x7f5fac0c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:32.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.084+0000 7f5faa7fc700 1 --2- 172.21.15.27:0/1145903768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fac0c4320 0x7f5fac0c4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:32.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.084+0000 7f5faa7fc700 1 -- 172.21.15.27:0/1145903768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fac0c5350 con 0x7f5fac15ad70 2024-01-11T08:29:32.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.084+0000 7f5faa7fc700 1 --2- 172.21.15.27:0/1145903768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fac15ad70 0x7f5fac15b1e0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f5f9400c0a0 tx=0x7f5fa001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:32.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.084+0000 7f5f92ffd700 1 -- 172.21.15.27:0/1145903768 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0022070 con 0x7f5fac15ad70 2024-01-11T08:29:32.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.084+0000 7f5f92ffd700 1 -- 172.21.15.27:0/1145903768 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa0013b30 con 0x7f5fac15ad70 2024-01-11T08:29:32.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.084+0000 7f5fb17ff700 1 -- 172.21.15.27:0/1145903768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fac0c5630 con 0x7f5fac15ad70 2024-01-11T08:29:32.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.084+0000 7f5f92ffd700 1 -- 172.21.15.27:0/1145903768 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa001c450 con 0x7f5fac15ad70 2024-01-11T08:29:32.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.085+0000 7f5fb17ff700 1 -- 172.21.15.27:0/1145903768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fac0c5bf0 con 0x7f5fac15ad70 2024-01-11T08:29:32.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.086+0000 7f5fb17ff700 1 -- 172.21.15.27:0/1145903768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fac0c5e10 con 0x7f5fac15ad70 2024-01-11T08:29:32.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.086+0000 7f5f92ffd700 1 -- 172.21.15.27:0/1145903768 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5fa0026070 con 0x7f5fac15ad70 2024-01-11T08:29:32.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.088+0000 7f5f92ffd700 1 --2- 172.21.15.27:0/1145903768 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f98066f60 0x7f5f98069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:32.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.088+0000 7f5f92ffd700 1 -- 172.21.15.27:0/1145903768 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f5fa0085c90 con 0x7f5fac15ad70 2024-01-11T08:29:32.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.088+0000 7f5faaffd700 1 --2- 172.21.15.27:0/1145903768 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f98066f60 0x7f5f98069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:32.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.089+0000 7f5faaffd700 1 --2- 172.21.15.27:0/1145903768 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f98066f60 0x7f5f98069410 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c000c00 tx=0x7f5f9c007d80 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:32.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.092+0000 7f5f92ffd700 1 -- 172.21.15.27:0/1145903768 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5fa00b75a0 con 0x7f5fac15ad70 2024-01-11T08:29:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.341+0000 7f5fb17ff700 1 -- 172.21.15.27:0/1145903768 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5fac04c3c0 con 0x7f5fac15ad70 2024-01-11T08:29:32.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.957+0000 7f5f92ffd700 1 -- 172.21.15.27:0/1145903768 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v68) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5fa002d0b0 con 0x7f5fac15ad70 2024-01-11T08:29:32.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.960+0000 7f5fb17ff700 1 -- 172.21.15.27:0/1145903768 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f98066f60 msgr2=0x7f5f98069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:32.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.960+0000 7f5fb17ff700 1 --2- 172.21.15.27:0/1145903768 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f98066f60 0x7f5f98069410 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c000c00 tx=0x7f5f9c007d80 comp rx=0 tx=0).stop 2024-01-11T08:29:32.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.960+0000 7f5fb17ff700 1 -- 172.21.15.27:0/1145903768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fac15ad70 msgr2=0x7f5fac15b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:32.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.960+0000 7f5fb17ff700 1 --2- 172.21.15.27:0/1145903768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fac15ad70 0x7f5fac15b1e0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f5f9400c0a0 tx=0x7f5fa001a000 comp rx=0 tx=0).stop 2024-01-11T08:29:32.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.960+0000 7f5fb17ff700 1 -- 172.21.15.27:0/1145903768 shutdown_connections 2024-01-11T08:29:32.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.960+0000 7f5fb17ff700 1 --2- 172.21.15.27:0/1145903768 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f98066f60 0x7f5f98069410 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:32.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.960+0000 7f5fb17ff700 1 --2- 172.21.15.27:0/1145903768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fac0c4320 0x7f5fac0c4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:32.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.960+0000 7f5fb17ff700 1 --2- 172.21.15.27:0/1145903768 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f94004810 0x7f5fac155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:32.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.960+0000 7f5fb17ff700 1 --2- 172.21.15.27:0/1145903768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fac15ad70 0x7f5fac15b1e0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:32.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.960+0000 7f5fb17ff700 1 -- 172.21.15.27:0/1145903768 >> 172.21.15.27:0/1145903768 conn(0x7f5fac0680a0 msgr2=0x7f5fac13ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:32.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.960+0000 7f5fb17ff700 1 -- 172.21.15.27:0/1145903768 shutdown_connections 2024-01-11T08:29:32.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:32.960+0000 7f5fb17ff700 1 -- 172.21.15.27:0/1145903768 wait complete. 2024-01-11T08:29:32.980 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-11T08:29:33.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.187+0000 7f7c69bff700 1 Processor -- start 2024-01-11T08:29:33.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.187+0000 7f7c69bff700 1 -- start start 2024-01-11T08:29:33.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.189+0000 7f7c69bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c641596f0 0x7f7c640a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:33.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.189+0000 7f7c69bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c64068d30 0x7f7c640c8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:33.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.189+0000 7f7c69bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c640c80a0 con 0x7f7c640a9de0 2024-01-11T08:29:33.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.189+0000 7f7c69bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c640cbac0 con 0x7f7c64068d30 2024-01-11T08:29:33.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.189+0000 7f7c69bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c640c7490 con 0x7f7c641596f0 2024-01-11T08:29:33.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.189+0000 7f7c63fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c64068d30 0x7f7c640c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:33.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.189+0000 7f7c63fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c64068d30 0x7f7c640c8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60004/0 (socket says 172.21.15.27:60004) 2024-01-11T08:29:33.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.189+0000 7f7c63fff700 1 -- 172.21.15.27:0/1295910973 learned_addr learned my addr 172.21.15.27:0/1295910973 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:33.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.189+0000 7f7c637fe700 1 --2- 172.21.15.27:0/1295910973 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c641596f0 0x7f7c640a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:33.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.189+0000 7f7c63fff700 1 -- 172.21.15.27:0/1295910973 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c641596f0 msgr2=0x7f7c640a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:33.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.189+0000 7f7c63fff700 1 --2- 172.21.15.27:0/1295910973 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c641596f0 0x7f7c640a98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:33.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.189+0000 7f7c63fff700 1 -- 172.21.15.27:0/1295910973 >> v1:172.21.15.77:6789/0 conn(0x7f7c640a9de0 legacy=0x7f7c64068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.189+0000 7f7c63fff700 1 -- 172.21.15.27:0/1295910973 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c640c7770 con 0x7f7c64068d30 2024-01-11T08:29:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.190+0000 7f7c63fff700 1 --2- 172.21.15.27:0/1295910973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c64068d30 0x7f7c640c8530 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c009f20 tx=0x7f7c5c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9765ece6ec0d1a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.190+0000 7f7c627fc700 1 -- 172.21.15.27:0/1295910973 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c00e070 con 0x7f7c64068d30 2024-01-11T08:29:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.190+0000 7f7c627fc700 1 -- 172.21.15.27:0/1295910973 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5c00b9d0 con 0x7f7c64068d30 2024-01-11T08:29:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.190+0000 7f7c627fc700 1 -- 172.21.15.27:0/1295910973 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0092e0 con 0x7f7c64068d30 2024-01-11T08:29:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.191+0000 7f7c69bff700 1 -- 172.21.15.27:0/1295910973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c64068d30 msgr2=0x7f7c640c8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.191+0000 7f7c69bff700 1 --2- 172.21.15.27:0/1295910973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c64068d30 0x7f7c640c8530 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c009f20 tx=0x7f7c5c009840 comp rx=0 tx=0).stop 2024-01-11T08:29:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.191+0000 7f7c69bff700 1 -- 172.21.15.27:0/1295910973 shutdown_connections 2024-01-11T08:29:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.191+0000 7f7c69bff700 1 --2- 172.21.15.27:0/1295910973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c64068d30 0x7f7c640c8530 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.191+0000 7f7c69bff700 1 --2- 172.21.15.27:0/1295910973 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c641596f0 0x7f7c640a98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.191+0000 7f7c69bff700 1 -- 172.21.15.27:0/1295910973 >> 172.21.15.27:0/1295910973 conn(0x7f7c640c9fb0 msgr2=0x7f7c6406ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.191+0000 7f7c69bff700 1 -- 172.21.15.27:0/1295910973 shutdown_connections 2024-01-11T08:29:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.191+0000 7f7c69bff700 1 -- 172.21.15.27:0/1295910973 wait complete. 2024-01-11T08:29:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.192+0000 7f7c69bff700 1 Processor -- start 2024-01-11T08:29:33.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.192+0000 7f7c69bff700 1 -- start start 2024-01-11T08:29:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.193+0000 7f7c69bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c64068d30 0x7f7c6414b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.193+0000 7f7c69bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c640a9de0 0x7f7c640ab960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:33.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.193+0000 7f7c69bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c641596f0 0x7f7c640b1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:33.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.193+0000 7f7c69bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c640c4860 con 0x7f7c641596f0 2024-01-11T08:29:33.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.193+0000 7f7c69bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c640c3c50 con 0x7f7c64068d30 2024-01-11T08:29:33.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.193+0000 7f7c69bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c640c3f30 con 0x7f7c640a9de0 2024-01-11T08:29:33.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.194+0000 7f7c63fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c641596f0 0x7f7c640b1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:33.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.194+0000 7f7c637fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c64068d30 0x7f7c6414b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:33.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.194+0000 7f7c62ffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c640a9de0 0x7f7c640ab960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:33.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.194+0000 7f7c63fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c641596f0 0x7f7c640b1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60014/0 (socket says 172.21.15.27:60014) 2024-01-11T08:29:33.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.194+0000 7f7c637fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c64068d30 0x7f7c6414b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:35208/0 (socket says 172.21.15.27:35208) 2024-01-11T08:29:33.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.194+0000 7f7c63fff700 1 -- 172.21.15.27:0/1671887664 learned_addr learned my addr 172.21.15.27:0/1671887664 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:33.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.194+0000 7f7c637fe700 1 -- 172.21.15.27:0/1671887664 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c640a9de0 msgr2=0x7f7c640ab960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:33.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.194+0000 7f7c637fe700 1 --2- 172.21.15.27:0/1671887664 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c640a9de0 0x7f7c640ab960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:33.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.194+0000 7f7c637fe700 1 -- 172.21.15.27:0/1671887664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c641596f0 msgr2=0x7f7c640b1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:33.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.194+0000 7f7c637fe700 1 --2- 172.21.15.27:0/1671887664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c641596f0 0x7f7c640b1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:33.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.194+0000 7f7c637fe700 1 -- 172.21.15.27:0/1671887664 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c009ae0 con 0x7f7c64068d30 2024-01-11T08:29:33.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.194+0000 7f7c63fff700 1 --2- 172.21.15.27:0/1671887664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c641596f0 0x7f7c640b1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:33.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.195+0000 7f7c637fe700 1 --2- 172.21.15.27:0/1671887664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c64068d30 0x7f7c6414b3e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7c5400e900 tx=0x7f7c540091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:33.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.195+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/1671887664 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c54013680 con 0x7f7c64068d30 2024-01-11T08:29:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.195+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/1671887664 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c540137e0 con 0x7f7c64068d30 2024-01-11T08:29:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.195+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/1671887664 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c54013b80 con 0x7f7c64068d30 2024-01-11T08:29:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.195+0000 7f7c69bff700 1 -- 172.21.15.27:0/1671887664 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c640c4210 con 0x7f7c64068d30 2024-01-11T08:29:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.195+0000 7f7c69bff700 1 -- 172.21.15.27:0/1671887664 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c640b2370 con 0x7f7c64068d30 2024-01-11T08:29:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.196+0000 7f7c69bff700 1 -- 172.21.15.27:0/1671887664 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c6404c3c0 con 0x7f7c64068d30 2024-01-11T08:29:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.197+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/1671887664 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7c540123d0 con 0x7f7c64068d30 2024-01-11T08:29:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.198+0000 7f7c4b7fe700 1 --2- 172.21.15.27:0/1671887664 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7c50066f30 0x7f7c500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.198+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/1671887664 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f7c540845e0 con 0x7f7c64068d30 2024-01-11T08:29:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.198+0000 7f7c62ffd700 1 --2- 172.21.15.27:0/1671887664 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7c50066f30 0x7f7c500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.199+0000 7f7c62ffd700 1 --2- 172.21.15.27:0/1671887664 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7c50066f30 0x7f7c500693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7c6406e430 tx=0x7f7c58007ec0 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:33.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.203+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/1671887664 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c540517f0 con 0x7f7c64068d30 2024-01-11T08:29:33.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.451+0000 7f7c69bff700 1 -- 172.21.15.27:0/1671887664 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7c640c7a50 con 0x7f7c64068d30 2024-01-11T08:29:33.880 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-01-11T08:29:33.881+0000 7fd51a3f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T08:29:33.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.962+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/1671887664 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v69) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7c54054e10 con 0x7f7c64068d30 2024-01-11T08:29:33.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.965+0000 7f7c69bff700 1 -- 172.21.15.27:0/1671887664 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7c50066f30 msgr2=0x7f7c500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:33.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.965+0000 7f7c69bff700 1 --2- 172.21.15.27:0/1671887664 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7c50066f30 0x7f7c500693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7c6406e430 tx=0x7f7c58007ec0 comp rx=0 tx=0).stop 2024-01-11T08:29:33.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.965+0000 7f7c69bff700 1 -- 172.21.15.27:0/1671887664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c64068d30 msgr2=0x7f7c6414b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:33.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.965+0000 7f7c69bff700 1 --2- 172.21.15.27:0/1671887664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c64068d30 0x7f7c6414b3e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7c5400e900 tx=0x7f7c540091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:33.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.965+0000 7f7c69bff700 1 -- 172.21.15.27:0/1671887664 shutdown_connections 2024-01-11T08:29:33.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.965+0000 7f7c69bff700 1 --2- 172.21.15.27:0/1671887664 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7c50066f30 0x7f7c500693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:33.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.965+0000 7f7c69bff700 1 --2- 172.21.15.27:0/1671887664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c641596f0 0x7f7c640b1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:33.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.965+0000 7f7c69bff700 1 --2- 172.21.15.27:0/1671887664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c64068d30 0x7f7c6414b3e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:33.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.965+0000 7f7c69bff700 1 --2- 172.21.15.27:0/1671887664 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c640a9de0 0x7f7c640ab960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:33.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.965+0000 7f7c69bff700 1 -- 172.21.15.27:0/1671887664 >> 172.21.15.27:0/1671887664 conn(0x7f7c640c9fb0 msgr2=0x7f7c640cac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:33.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.965+0000 7f7c69bff700 1 -- 172.21.15.27:0/1671887664 shutdown_connections 2024-01-11T08:29:33.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:33.965+0000 7f7c69bff700 1 -- 172.21.15.27:0/1671887664 wait complete. 2024-01-11T08:29:33.983 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:29:34.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.184+0000 7faa7b9ff700 1 Processor -- start 2024-01-11T08:29:34.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.184+0000 7faa7b9ff700 1 -- start start 2024-01-11T08:29:34.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.185+0000 7faa7b9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faa740b03e0 0x7faa740b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:34.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.185+0000 7faa7b9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa740cd730 con 0x7faa740b29c0 2024-01-11T08:29:34.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.185+0000 7faa7b9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa740b2de0 con 0x7faa740b07b0 2024-01-11T08:29:34.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.185+0000 7faa7b9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa740ccb20 con 0x7faa740b03e0 2024-01-11T08:29:34.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.186+0000 7faa79f9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7faa740b29c0 0x7faa740aefc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54776/0 (socket says 172.21.15.27:54776) 2024-01-11T08:29:34.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.186+0000 7faa79f9c700 1 -- 172.21.15.27:0/1417837821 learned_addr learned my addr 172.21.15.27:0/1417837821 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:34.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.186+0000 7faa7979b700 1 --2- 172.21.15.27:0/1417837821 >> v2:172.21.15.77:3300/0 conn(0x7faa740b03e0 0x7faa740b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.186+0000 7faa6bfff700 1 -- 172.21.15.27:0/1417837821 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387403448 0 0) 0x7faa740cd730 con 0x7faa740b29c0 2024-01-11T08:29:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.186+0000 7faa6bfff700 1 -- 172.21.15.27:0/1417837821 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa60003570 con 0x7faa740b29c0 2024-01-11T08:29:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.186+0000 7faa7979b700 1 -- 172.21.15.27:0/1417837821 >> v1:172.21.15.77:6789/0 conn(0x7faa740b07b0 legacy=0x7faa74159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.186+0000 7faa7979b700 1 -- 172.21.15.27:0/1417837821 >> v1:172.21.15.27:6789/0 conn(0x7faa740b29c0 legacy=0x7faa740aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.186+0000 7faa7979b700 1 -- 172.21.15.27:0/1417837821 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa740cce00 con 0x7faa740b03e0 2024-01-11T08:29:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.187+0000 7faa7979b700 1 --2- 172.21.15.27:0/1417837821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa740b03e0 0x7faa740b2480 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7faa700098e0 tx=0x7faa700091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6ebdf74138ac2f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.187+0000 7faa6bfff700 1 -- 172.21.15.27:0/1417837821 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7000b640 con 0x7faa740b03e0 2024-01-11T08:29:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.187+0000 7faa6bfff700 1 -- 172.21.15.27:0/1417837821 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7000b7a0 con 0x7faa740b03e0 2024-01-11T08:29:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.187+0000 7faa6bfff700 1 -- 172.21.15.27:0/1417837821 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7000bb40 con 0x7faa740b03e0 2024-01-11T08:29:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.187+0000 7faa7b9ff700 1 -- 172.21.15.27:0/1417837821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa740b03e0 msgr2=0x7faa740b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.187+0000 7faa7b9ff700 1 --2- 172.21.15.27:0/1417837821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa740b03e0 0x7faa740b2480 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7faa700098e0 tx=0x7faa700091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.187+0000 7faa7b9ff700 1 -- 172.21.15.27:0/1417837821 shutdown_connections 2024-01-11T08:29:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.187+0000 7faa7b9ff700 1 --2- 172.21.15.27:0/1417837821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa740b03e0 0x7faa740b2480 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.187+0000 7faa7b9ff700 1 -- 172.21.15.27:0/1417837821 >> 172.21.15.27:0/1417837821 conn(0x7faa7406ed40 msgr2=0x7faa7406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.188+0000 7faa7b9ff700 1 -- 172.21.15.27:0/1417837821 shutdown_connections 2024-01-11T08:29:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.188+0000 7faa7b9ff700 1 -- 172.21.15.27:0/1417837821 wait complete. 2024-01-11T08:29:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.189+0000 7faa7b9ff700 1 Processor -- start 2024-01-11T08:29:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.189+0000 7faa7b9ff700 1 -- start start 2024-01-11T08:29:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.190+0000 7faa7b9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa740b03e0 0x7faa7415b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.190+0000 7faa7b9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa740b07b0 0x7faa740c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.190+0000 7faa7b9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faa740b29c0 0x7faa740c6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.190+0000 7faa7b9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa740c7ec0 con 0x7faa740b07b0 2024-01-11T08:29:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.190+0000 7faa7b9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa740c81a0 con 0x7faa740b03e0 2024-01-11T08:29:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.190+0000 7faa7b9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa740c8480 con 0x7faa740b29c0 2024-01-11T08:29:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.190+0000 7faa78f9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa740b07b0 0x7faa740c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.190+0000 7faa78f9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa740b07b0 0x7faa740c75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36734/0 (socket says 172.21.15.27:36734) 2024-01-11T08:29:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.190+0000 7faa78f9a700 1 -- 172.21.15.27:0/2197685196 learned_addr learned my addr 172.21.15.27:0/2197685196 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.190+0000 7faa7979b700 1 --2- 172.21.15.27:0/2197685196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa740b03e0 0x7faa7415b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.190+0000 7faa79f9c700 1 --2- 172.21.15.27:0/2197685196 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faa740b29c0 0x7faa740c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.191+0000 7faa78f9a700 1 -- 172.21.15.27:0/2197685196 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faa740b29c0 msgr2=0x7faa740c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.191+0000 7faa78f9a700 1 --2- 172.21.15.27:0/2197685196 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faa740b29c0 0x7faa740c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.191+0000 7faa78f9a700 1 -- 172.21.15.27:0/2197685196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa740b03e0 msgr2=0x7faa7415b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.191+0000 7faa78f9a700 1 --2- 172.21.15.27:0/2197685196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa740b03e0 0x7faa7415b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.191+0000 7faa78f9a700 1 -- 172.21.15.27:0/2197685196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa70009420 con 0x7faa740b07b0 2024-01-11T08:29:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.191+0000 7faa7979b700 1 --2- 172.21.15.27:0/2197685196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa740b03e0 0x7faa7415b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.191+0000 7faa78f9a700 1 --2- 172.21.15.27:0/2197685196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa740b07b0 0x7faa740c75c0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7faa6400c590 tx=0x7faa6400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.191+0000 7faa697fa700 1 -- 172.21.15.27:0/2197685196 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64011590 con 0x7faa740b07b0 2024-01-11T08:29:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.191+0000 7faa697fa700 1 -- 172.21.15.27:0/2197685196 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa640116f0 con 0x7faa740b07b0 2024-01-11T08:29:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.191+0000 7faa7b9ff700 1 -- 172.21.15.27:0/2197685196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa740a6f40 con 0x7faa740b07b0 2024-01-11T08:29:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.191+0000 7faa697fa700 1 -- 172.21.15.27:0/2197685196 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64011ab0 con 0x7faa740b07b0 2024-01-11T08:29:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.191+0000 7faa7b9ff700 1 -- 172.21.15.27:0/2197685196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7415bf10 con 0x7faa740b07b0 2024-01-11T08:29:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.192+0000 7faa7b9ff700 1 -- 172.21.15.27:0/2197685196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa740718d0 con 0x7faa740b07b0 2024-01-11T08:29:34.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.193+0000 7faa697fa700 1 -- 172.21.15.27:0/2197685196 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7faa6401b3b0 con 0x7faa740b07b0 2024-01-11T08:29:34.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.194+0000 7faa697fa700 1 --2- 172.21.15.27:0/2197685196 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7faa58066f30 0x7faa580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:34.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.194+0000 7faa697fa700 1 -- 172.21.15.27:0/2197685196 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5761+0+0 (secure 0 0 0) 0x7faa64083b70 con 0x7faa740b07b0 2024-01-11T08:29:34.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.195+0000 7faa7979b700 1 --2- 172.21.15.27:0/2197685196 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7faa58066f30 0x7faa580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.195+0000 7faa7979b700 1 --2- 172.21.15.27:0/2197685196 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7faa58066f30 0x7faa580693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faa700093f0 tx=0x7faa70017000 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.199+0000 7faa697fa700 1 -- 172.21.15.27:0/2197685196 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faa64050d00 con 0x7faa740b07b0 2024-01-11T08:29:34.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.446+0000 7faa7b9ff700 1 -- 172.21.15.27:0/2197685196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa740cd0e0 con 0x7faa740b07b0 2024-01-11T08:29:34.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.447+0000 7faa697fa700 1 -- 172.21.15.27:0/2197685196 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4596 (secure 0 0 0) 0x7faa64017020 con 0x7faa740b07b0 2024-01-11T08:29:34.446 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:34.446 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":69,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6956,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1277153217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1277153217},{"type":"v1","addr":"172.21.15.77:6833","nonce":1277153217}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7777,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2435914089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2435914089},{"type":"v1","addr":"172.21.15.27:6835","nonce":2435914089}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7792,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/4128023892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4128023892},{"type":"v1","addr":"172.21.15.77:6837","nonce":4128023892}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:29:30.441936+0000","modified":"2024-01-11T08:29:33.957014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8058":{"gid":8058,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6834/716078331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":716078331},{"type":"v1","addr":"172.21.15.27:6834","nonce":716078331}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2024-01-11T08:29:34.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.449+0000 7faa7b9ff700 1 -- 172.21.15.27:0/2197685196 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7faa58066f30 msgr2=0x7faa580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:34.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.449+0000 7faa7b9ff700 1 --2- 172.21.15.27:0/2197685196 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7faa58066f30 0x7faa580693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faa700093f0 tx=0x7faa70017000 comp rx=0 tx=0).stop 2024-01-11T08:29:34.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.449+0000 7faa7b9ff700 1 -- 172.21.15.27:0/2197685196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa740b07b0 msgr2=0x7faa740c75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:34.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.449+0000 7faa7b9ff700 1 --2- 172.21.15.27:0/2197685196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa740b07b0 0x7faa740c75c0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7faa6400c590 tx=0x7faa6400c950 comp rx=0 tx=0).stop 2024-01-11T08:29:34.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.449+0000 7faa7b9ff700 1 -- 172.21.15.27:0/2197685196 shutdown_connections 2024-01-11T08:29:34.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.449+0000 7faa7b9ff700 1 --2- 172.21.15.27:0/2197685196 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7faa58066f30 0x7faa580693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:34.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.449+0000 7faa7b9ff700 1 --2- 172.21.15.27:0/2197685196 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faa740b29c0 0x7faa740c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:34.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.449+0000 7faa7b9ff700 1 --2- 172.21.15.27:0/2197685196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa740b03e0 0x7faa7415b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:34.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.449+0000 7faa7b9ff700 1 --2- 172.21.15.27:0/2197685196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa740b07b0 0x7faa740c75c0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:34.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.449+0000 7faa7b9ff700 1 -- 172.21.15.27:0/2197685196 >> 172.21.15.27:0/2197685196 conn(0x7faa7406ed40 msgr2=0x7faa7415c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:34.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.449+0000 7faa7b9ff700 1 -- 172.21.15.27:0/2197685196 shutdown_connections 2024-01-11T08:29:34.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.449+0000 7faa7b9ff700 1 -- 172.21.15.27:0/2197685196 wait complete. 2024-01-11T08:29:34.449 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 69 2024-01-11T08:29:34.466 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:29:34.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.666+0000 7f735c3ff700 1 Processor -- start 2024-01-11T08:29:34.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.666+0000 7f735c3ff700 1 -- start start 2024-01-11T08:29:34.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.668+0000 7f735c3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f73541596c0 0x7f735415d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:34.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.668+0000 7f735c3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7354068d30 0x7f73540aa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:34.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.668+0000 7f735c3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73540a9cb0 con 0x7f73540ca670 2024-01-11T08:29:34.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.668+0000 7f735c3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f735415e210 con 0x7f73541596c0 2024-01-11T08:29:34.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.668+0000 7f735c3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735415dc50 con 0x7f7354068d30 2024-01-11T08:29:34.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.668+0000 7f735a19b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f73541596c0 0x7f735415d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:34.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.668+0000 7f735999a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f73540ca670 0x7f7354068620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54782/0 (socket says 172.21.15.27:54782) 2024-01-11T08:29:34.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.668+0000 7f735a19b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f73541596c0 0x7f735415d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36742/0 (socket says 172.21.15.27:36742) 2024-01-11T08:29:34.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.668+0000 7f735999a700 1 -- 172.21.15.27:0/3860051960 learned_addr learned my addr 172.21.15.27:0/3860051960 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:34.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.668+0000 7f735a99c700 1 --2- 172.21.15.27:0/3860051960 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7354068d30 0x7f73540aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:34.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.668+0000 7f735a19b700 1 -- 172.21.15.27:0/3860051960 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7354068d30 msgr2=0x7f73540aa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:34.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.668+0000 7f735a19b700 1 --2- 172.21.15.27:0/3860051960 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7354068d30 0x7f73540aa140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:34.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.669+0000 7f735a19b700 1 -- 172.21.15.27:0/3860051960 >> v1:172.21.15.27:6789/0 conn(0x7f73540ca670 legacy=0x7f7354068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:34.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.669+0000 7f735a19b700 1 -- 172.21.15.27:0/3860051960 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735415df30 con 0x7f73541596c0 2024-01-11T08:29:34.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.669+0000 7f7359199700 1 -- 172.21.15.27:0/3860051960 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481973739 0 0) 0x7f73540a9cb0 con 0x7f73540ca670 2024-01-11T08:29:34.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.669+0000 7f735a19b700 1 --2- 172.21.15.27:0/3860051960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73541596c0 0x7f735415d300 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f73500098e0 tx=0x7f73500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c90178f1db5ad58 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:34.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.669+0000 7f7359199700 1 -- 172.21.15.27:0/3860051960 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7350009e80 con 0x7f73541596c0 2024-01-11T08:29:34.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.669+0000 7f7359199700 1 -- 172.21.15.27:0/3860051960 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735000b860 con 0x7f73541596c0 2024-01-11T08:29:34.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.669+0000 7f7359199700 1 -- 172.21.15.27:0/3860051960 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735000bbe0 con 0x7f73541596c0 2024-01-11T08:29:34.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.670+0000 7f735c3ff700 1 -- 172.21.15.27:0/3860051960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73541596c0 msgr2=0x7f735415d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:34.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.670+0000 7f735c3ff700 1 --2- 172.21.15.27:0/3860051960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73541596c0 0x7f735415d300 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f73500098e0 tx=0x7f73500091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:34.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.670+0000 7f735c3ff700 1 -- 172.21.15.27:0/3860051960 shutdown_connections 2024-01-11T08:29:34.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.670+0000 7f735c3ff700 1 --2- 172.21.15.27:0/3860051960 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7354068d30 0x7f73540aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:34.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.670+0000 7f735c3ff700 1 --2- 172.21.15.27:0/3860051960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73541596c0 0x7f735415d300 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:34.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.670+0000 7f735c3ff700 1 -- 172.21.15.27:0/3860051960 >> 172.21.15.27:0/3860051960 conn(0x7f73540c9f70 msgr2=0x7f735406deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:34.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.670+0000 7f735c3ff700 1 -- 172.21.15.27:0/3860051960 shutdown_connections 2024-01-11T08:29:34.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.670+0000 7f735c3ff700 1 -- 172.21.15.27:0/3860051960 wait complete. 2024-01-11T08:29:34.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.671+0000 7f735c3ff700 1 Processor -- start 2024-01-11T08:29:34.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.671+0000 7f735c3ff700 1 -- start start 2024-01-11T08:29:34.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.672+0000 7f735c3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7354068d30 0x7f73540b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:34.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.672+0000 7f735c3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73540ca670 0x7f735414a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:34.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.672+0000 7f735c3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73541596c0 0x7f73540c18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:34.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.672+0000 7f735c3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735415d510 con 0x7f7354068d30 2024-01-11T08:29:34.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.672+0000 7f735c3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735415c470 con 0x7f73541596c0 2024-01-11T08:29:34.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.672+0000 7f735c3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735415c750 con 0x7f73540ca670 2024-01-11T08:29:34.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.673+0000 7f735a19b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7354068d30 0x7f73540b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:34.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.673+0000 7f735a19b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7354068d30 0x7f73540b21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36756/0 (socket says 172.21.15.27:36756) 2024-01-11T08:29:34.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.673+0000 7f735a19b700 1 -- 172.21.15.27:0/1366083627 learned_addr learned my addr 172.21.15.27:0/1366083627 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:34.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.673+0000 7f735999a700 1 --2- 172.21.15.27:0/1366083627 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73540ca670 0x7f735414a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:34.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.673+0000 7f735a99c700 1 --2- 172.21.15.27:0/1366083627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73541596c0 0x7f73540c18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:34.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.673+0000 7f735a19b700 1 -- 172.21.15.27:0/1366083627 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73540ca670 msgr2=0x7f735414a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:34.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.673+0000 7f735a19b700 1 --2- 172.21.15.27:0/1366083627 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73540ca670 0x7f735414a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:34.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.673+0000 7f735a19b700 1 -- 172.21.15.27:0/1366083627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73541596c0 msgr2=0x7f73540c18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:34.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.673+0000 7f735a19b700 1 --2- 172.21.15.27:0/1366083627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73541596c0 0x7f73540c18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:34.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.673+0000 7f735a19b700 1 -- 172.21.15.27:0/1366083627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7350009420 con 0x7f7354068d30 2024-01-11T08:29:34.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.673+0000 7f735999a700 1 --2- 172.21.15.27:0/1366083627 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73540ca670 0x7f735414a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:34.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.673+0000 7f735a19b700 1 --2- 172.21.15.27:0/1366083627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7354068d30 0x7f73540b21b0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f735406d490 tx=0x7f7350017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:34.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.674+0000 7f734a7fc700 1 -- 172.21.15.27:0/1366083627 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7350011420 con 0x7f7354068d30 2024-01-11T08:29:34.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.674+0000 7f734a7fc700 1 -- 172.21.15.27:0/1366083627 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73500115c0 con 0x7f7354068d30 2024-01-11T08:29:34.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.674+0000 7f734a7fc700 1 -- 172.21.15.27:0/1366083627 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7350011980 con 0x7f7354068d30 2024-01-11T08:29:34.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.674+0000 7f735c3ff700 1 -- 172.21.15.27:0/1366083627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f735415ca30 con 0x7f7354068d30 2024-01-11T08:29:34.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.674+0000 7f735c3ff700 1 -- 172.21.15.27:0/1366083627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f735415cff0 con 0x7f7354068d30 2024-01-11T08:29:34.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.675+0000 7f735c3ff700 1 -- 172.21.15.27:0/1366083627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73540718d0 con 0x7f7354068d30 2024-01-11T08:29:34.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.676+0000 7f734a7fc700 1 -- 172.21.15.27:0/1366083627 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f73500253f0 con 0x7f7354068d30 2024-01-11T08:29:34.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.677+0000 7f734a7fc700 1 --2- 172.21.15.27:0/1366083627 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7338066f30 0x7f73380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:34.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.677+0000 7f734a7fc700 1 -- 172.21.15.27:0/1366083627 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f7350085fd0 con 0x7f7354068d30 2024-01-11T08:29:34.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.677+0000 7f735999a700 1 --2- 172.21.15.27:0/1366083627 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7338066f30 0x7f73380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:34.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.678+0000 7f735999a700 1 --2- 172.21.15.27:0/1366083627 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7338066f30 0x7f73380693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f73540c2c60 tx=0x7f7344009040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:34.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.681+0000 7f734a7fc700 1 -- 172.21.15.27:0/1366083627 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f73500531a0 con 0x7f7354068d30 2024-01-11T08:29:34.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.884+0000 7f735c3ff700 1 -- 172.21.15.27:0/1366083627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f735406ea80 con 0x7f7354068d30 2024-01-11T08:29:34.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.885+0000 7f734a7fc700 1 -- 172.21.15.27:0/1366083627 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+15404 (secure 0 0 0) 0x7f7350056780 con 0x7f7354068d30 2024-01-11T08:29:34.884 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:34.885 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":131,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:29:31.929826+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:29:28.131181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-11T08:29:29.438397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:29:34.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.888+0000 7f735c3ff700 1 -- 172.21.15.27:0/1366083627 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7338066f30 msgr2=0x7f73380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:34.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.888+0000 7f735c3ff700 1 --2- 172.21.15.27:0/1366083627 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7338066f30 0x7f73380693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f73540c2c60 tx=0x7f7344009040 comp rx=0 tx=0).stop 2024-01-11T08:29:34.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.888+0000 7f735c3ff700 1 -- 172.21.15.27:0/1366083627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7354068d30 msgr2=0x7f73540b21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:34.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.888+0000 7f735c3ff700 1 --2- 172.21.15.27:0/1366083627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7354068d30 0x7f73540b21b0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f735406d490 tx=0x7f7350017000 comp rx=0 tx=0).stop 2024-01-11T08:29:34.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.888+0000 7f735c3ff700 1 -- 172.21.15.27:0/1366083627 shutdown_connections 2024-01-11T08:29:34.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.888+0000 7f735c3ff700 1 --2- 172.21.15.27:0/1366083627 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7338066f30 0x7f73380693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:34.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.888+0000 7f735c3ff700 1 --2- 172.21.15.27:0/1366083627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73541596c0 0x7f73540c18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:34.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.888+0000 7f735c3ff700 1 --2- 172.21.15.27:0/1366083627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7354068d30 0x7f73540b21b0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:34.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.888+0000 7f735c3ff700 1 --2- 172.21.15.27:0/1366083627 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73540ca670 0x7f735414a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.888+0000 7f735c3ff700 1 -- 172.21.15.27:0/1366083627 >> 172.21.15.27:0/1366083627 conn(0x7f73540c9f70 msgr2=0x7f7354158820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.888+0000 7f735c3ff700 1 -- 172.21.15.27:0/1366083627 shutdown_connections 2024-01-11T08:29:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:34.888+0000 7f735c3ff700 1 -- 172.21.15.27:0/1366083627 wait complete. 2024-01-11T08:29:34.905 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-11T08:29:34.906 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:29:35.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.106+0000 7ff2e8bff700 1 Processor -- start 2024-01-11T08:29:35.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.107+0000 7ff2e8bff700 1 -- start start 2024-01-11T08:29:35.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.108+0000 7ff2e8bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff2e40c8620 0x7ff2e40af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.108+0000 7ff2e8bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff2e4069d40 0x7ff2e40b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.108+0000 7ff2e8bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e4157390 con 0x7ff2e40c89f0 2024-01-11T08:29:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.108+0000 7ff2e8bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2e40adcd0 con 0x7ff2e40c8620 2024-01-11T08:29:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.108+0000 7ff2e8bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2e40afeb0 con 0x7ff2e4069d40 2024-01-11T08:29:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.108+0000 7ff2e259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff2e40c8620 0x7ff2e40af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.108+0000 7ff2e259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff2e40c8620 0x7ff2e40af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36764/0 (socket says 172.21.15.27:36764) 2024-01-11T08:29:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.108+0000 7ff2e259c700 1 -- 172.21.15.27:0/1556950060 learned_addr learned my addr 172.21.15.27:0/1556950060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.108+0000 7ff2e2d9d700 1 --1- 172.21.15.27:0/1556950060 >> v1:172.21.15.27:6789/0 conn(0x7ff2e40c89f0 0x7ff2e4159ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54788/0 (socket says 172.21.15.27:54788) 2024-01-11T08:29:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.108+0000 7ff2e1d9b700 1 --2- 172.21.15.27:0/1556950060 >> v2:172.21.15.77:3300/0 conn(0x7ff2e4069d40 0x7ff2e40b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.109+0000 7ff2e259c700 1 -- 172.21.15.27:0/1556950060 >> v2:172.21.15.77:3300/0 conn(0x7ff2e4069d40 msgr2=0x7ff2e40b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.109+0000 7ff2e259c700 1 --2- 172.21.15.27:0/1556950060 >> v2:172.21.15.77:3300/0 conn(0x7ff2e4069d40 0x7ff2e40b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.109+0000 7ff2e259c700 1 -- 172.21.15.27:0/1556950060 >> v1:172.21.15.27:6789/0 conn(0x7ff2e40c89f0 legacy=0x7ff2e4159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:35.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.109+0000 7ff2e159a700 1 -- 172.21.15.27:0/1556950060 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921275751 0 0) 0x7ff2e4157390 con 0x7ff2e40c89f0 2024-01-11T08:29:35.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.109+0000 7ff2e259c700 1 -- 172.21.15.27:0/1556950060 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e406a200 con 0x7ff2e40c8620 2024-01-11T08:29:35.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.109+0000 7ff2e259c700 1 --2- 172.21.15.27:0/1556950060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e40c8620 0x7ff2e40af970 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7ff2d40099b0 tx=0x7ff2d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c4a490290d6b713 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:35.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.109+0000 7ff2e159a700 1 -- 172.21.15.27:0/1556950060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d4009ea0 con 0x7ff2e40c8620 2024-01-11T08:29:35.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.109+0000 7ff2e159a700 1 -- 172.21.15.27:0/1556950060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d400b820 con 0x7ff2e40c8620 2024-01-11T08:29:35.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.109+0000 7ff2e159a700 1 -- 172.21.15.27:0/1556950060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d400bbc0 con 0x7ff2e40c8620 2024-01-11T08:29:35.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.110+0000 7ff2e8bff700 1 -- 172.21.15.27:0/1556950060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e40c8620 msgr2=0x7ff2e40af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:35.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.110+0000 7ff2e8bff700 1 --2- 172.21.15.27:0/1556950060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e40c8620 0x7ff2e40af970 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7ff2d40099b0 tx=0x7ff2d4009250 comp rx=0 tx=0).stop 2024-01-11T08:29:35.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.110+0000 7ff2e8bff700 1 -- 172.21.15.27:0/1556950060 shutdown_connections 2024-01-11T08:29:35.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.110+0000 7ff2e8bff700 1 --2- 172.21.15.27:0/1556950060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e40c8620 0x7ff2e40af970 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.110+0000 7ff2e8bff700 1 --2- 172.21.15.27:0/1556950060 >> v2:172.21.15.77:3300/0 conn(0x7ff2e4069d40 0x7ff2e40b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.110+0000 7ff2e8bff700 1 -- 172.21.15.27:0/1556950060 >> 172.21.15.27:0/1556950060 conn(0x7ff2e40680e0 msgr2=0x7ff2e4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:35.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.110+0000 7ff2e8bff700 1 -- 172.21.15.27:0/1556950060 shutdown_connections 2024-01-11T08:29:35.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.110+0000 7ff2e8bff700 1 -- 172.21.15.27:0/1556950060 wait complete. 2024-01-11T08:29:35.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.112+0000 7ff2e8bff700 1 Processor -- start 2024-01-11T08:29:35.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.112+0000 7ff2e8bff700 1 -- start start 2024-01-11T08:29:35.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.113+0000 7ff2e8bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e4069d40 0x7ff2e4146cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:35.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.113+0000 7ff2e8bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2e40c8620 0x7ff2e40b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:35.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.113+0000 7ff2e8bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2e40c89f0 0x7ff2e40b0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:35.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.113+0000 7ff2e8bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e416d1b0 con 0x7ff2e4069d40 2024-01-11T08:29:35.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.113+0000 7ff2e8bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e4153110 con 0x7ff2e40c8620 2024-01-11T08:29:35.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.113+0000 7ff2e8bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e416d400 con 0x7ff2e40c89f0 2024-01-11T08:29:35.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.113+0000 7ff2e259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e4069d40 0x7ff2e4146cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:35.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.113+0000 7ff2e259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e4069d40 0x7ff2e4146cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36776/0 (socket says 172.21.15.27:36776) 2024-01-11T08:29:35.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.113+0000 7ff2e259c700 1 -- 172.21.15.27:0/3089453864 learned_addr learned my addr 172.21.15.27:0/3089453864 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:35.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.113+0000 7ff2e2d9d700 1 --2- 172.21.15.27:0/3089453864 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2e40c89f0 0x7ff2e40b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:35.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.113+0000 7ff2e1d9b700 1 --2- 172.21.15.27:0/3089453864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2e40c8620 0x7ff2e40b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.113+0000 7ff2e259c700 1 -- 172.21.15.27:0/3089453864 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2e40c89f0 msgr2=0x7ff2e40b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.113+0000 7ff2e259c700 1 --2- 172.21.15.27:0/3089453864 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2e40c89f0 0x7ff2e40b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.113+0000 7ff2e259c700 1 -- 172.21.15.27:0/3089453864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2e40c8620 msgr2=0x7ff2e40b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.113+0000 7ff2e259c700 1 --2- 172.21.15.27:0/3089453864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2e40c8620 0x7ff2e40b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.113+0000 7ff2e259c700 1 -- 172.21.15.27:0/3089453864 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d40094f0 con 0x7ff2e4069d40 2024-01-11T08:29:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.113+0000 7ff2e259c700 1 --2- 172.21.15.27:0/3089453864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e4069d40 0x7ff2e4146cc0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4066180 tx=0x7ff2d4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:35.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.114+0000 7ff2d27fc700 1 -- 172.21.15.27:0/3089453864 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d400bba0 con 0x7ff2e4069d40 2024-01-11T08:29:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.114+0000 7ff2d27fc700 1 -- 172.21.15.27:0/3089453864 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d4011420 con 0x7ff2e4069d40 2024-01-11T08:29:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.114+0000 7ff2e8bff700 1 -- 172.21.15.27:0/3089453864 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e416d6e0 con 0x7ff2e4069d40 2024-01-11T08:29:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.114+0000 7ff2d27fc700 1 -- 172.21.15.27:0/3089453864 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d40117c0 con 0x7ff2e4069d40 2024-01-11T08:29:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.114+0000 7ff2e8bff700 1 -- 172.21.15.27:0/3089453864 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e416dca0 con 0x7ff2e4069d40 2024-01-11T08:29:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.115+0000 7ff2e8bff700 1 -- 172.21.15.27:0/3089453864 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e406b940 con 0x7ff2e4069d40 2024-01-11T08:29:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.116+0000 7ff2d27fc700 1 -- 172.21.15.27:0/3089453864 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff2d4011b30 con 0x7ff2e4069d40 2024-01-11T08:29:35.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.117+0000 7ff2d27fc700 1 --2- 172.21.15.27:0/3089453864 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff2c4066f30 0x7ff2c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:35.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.117+0000 7ff2d27fc700 1 -- 172.21.15.27:0/3089453864 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5761+0+0 (secure 0 0 0) 0x7ff2d4085f90 con 0x7ff2e4069d40 2024-01-11T08:29:35.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.117+0000 7ff2e1d9b700 1 --2- 172.21.15.27:0/3089453864 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff2c4066f30 0x7ff2c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:35.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.118+0000 7ff2e1d9b700 1 --2- 172.21.15.27:0/3089453864 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff2c4066f30 0x7ff2c40693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4155430 tx=0x7ff2d8008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:35.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.121+0000 7ff2d27fc700 1 -- 172.21.15.27:0/3089453864 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff2d4053120 con 0x7ff2e4069d40 2024-01-11T08:29:35.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.317+0000 7ff2e8bff700 1 -- 172.21.15.27:0/3089453864 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff2e40718d0 con 0x7ff2c4066f30 2024-01-11T08:29:35.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.329+0000 7ff2d27fc700 1 -- 172.21.15.27:0/3089453864 <== mgr.7591 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289559 (secure 0 0 0) 0x7ff2e40718d0 con 0x7ff2c4066f30 2024-01-11T08:29:35.332 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:35.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.333+0000 7ff2e8bff700 1 -- 172.21.15.27:0/3089453864 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff2c4066f30 msgr2=0x7ff2c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:35.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.333+0000 7ff2e8bff700 1 --2- 172.21.15.27:0/3089453864 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff2c4066f30 0x7ff2c40693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4155430 tx=0x7ff2d8008040 comp rx=0 tx=0).stop 2024-01-11T08:29:35.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.333+0000 7ff2e8bff700 1 -- 172.21.15.27:0/3089453864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e4069d40 msgr2=0x7ff2e4146cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:35.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.333+0000 7ff2e8bff700 1 --2- 172.21.15.27:0/3089453864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e4069d40 0x7ff2e4146cc0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4066180 tx=0x7ff2d4017000 comp rx=0 tx=0).stop 2024-01-11T08:29:35.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.333+0000 7ff2e8bff700 1 -- 172.21.15.27:0/3089453864 shutdown_connections 2024-01-11T08:29:35.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.333+0000 7ff2e8bff700 1 --2- 172.21.15.27:0/3089453864 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff2c4066f30 0x7ff2c40693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.333+0000 7ff2e8bff700 1 --2- 172.21.15.27:0/3089453864 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2e40c89f0 0x7ff2e40b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.333+0000 7ff2e8bff700 1 --2- 172.21.15.27:0/3089453864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e4069d40 0x7ff2e4146cc0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.333+0000 7ff2e8bff700 1 --2- 172.21.15.27:0/3089453864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2e40c8620 0x7ff2e40b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.333+0000 7ff2e8bff700 1 -- 172.21.15.27:0/3089453864 >> 172.21.15.27:0/3089453864 conn(0x7ff2e40680e0 msgr2=0x7ff2e4155ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:35.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.334+0000 7ff2e8bff700 1 -- 172.21.15.27:0/3089453864 shutdown_connections 2024-01-11T08:29:35.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.334+0000 7ff2e8bff700 1 -- 172.21.15.27:0/3089453864 wait complete. 2024-01-11T08:29:35.336 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:29:35.351 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-01-11T08:29:34.057894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1232,"num_read_kb":2542,"num_write":435,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":283,"ondisk_log_size":283,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2449320,"kb_used_data":13672,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752525400,"statfs":{"total":773094113280,"available":770586009600,"internally_reserved":0,"allocated":14000128,"data_stored":7154784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.132190"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.963055+0000","last_change":"2024-01-11T08:29:30.963055+0000","last_active":"2024-01-11T08:29:30.963055+0000","last_peered":"2024-01-11T08:29:30.963055+0000","last_clean":"2024-01-11T08:29:30.963055+0000","last_became_active":"2024-01-11T08:29:30.961165+0000","last_became_peered":"2024-01-11T08:29:30.961165+0000","last_unstale":"2024-01-11T08:29:30.963055+0000","last_undegraded":"2024-01-11T08:29:30.963055+0000","last_fullsized":"2024-01-11T08:29:30.963055+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:47:37.059908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.952345+0000","last_change":"2024-01-11T08:29:30.952345+0000","last_active":"2024-01-11T08:29:30.952345+0000","last_peered":"2024-01-11T08:29:30.952345+0000","last_clean":"2024-01-11T08:29:30.952345+0000","last_became_active":"2024-01-11T08:29:30.952094+0000","last_became_peered":"2024-01-11T08:29:30.952094+0000","last_unstale":"2024-01-11T08:29:30.952345+0000","last_undegraded":"2024-01-11T08:29:30.952345+0000","last_fullsized":"2024-01-11T08:29:30.952345+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:11:42.267734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.955817+0000","last_change":"2024-01-11T08:29:30.955817+0000","last_active":"2024-01-11T08:29:30.955817+0000","last_peered":"2024-01-11T08:29:30.955817+0000","last_clean":"2024-01-11T08:29:30.955817+0000","last_became_active":"2024-01-11T08:29:30.954978+0000","last_became_peered":"2024-01-11T08:29:30.954978+0000","last_unstale":"2024-01-11T08:29:30.955817+0000","last_undegraded":"2024-01-11T08:29:30.955817+0000","last_fullsized":"2024-01-11T08:29:30.955817+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:15:59.730225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.965839+0000","last_change":"2024-01-11T08:29:30.965839+0000","last_active":"2024-01-11T08:29:30.965839+0000","last_peered":"2024-01-11T08:29:30.965839+0000","last_clean":"2024-01-11T08:29:30.965839+0000","last_became_active":"2024-01-11T08:29:30.965589+0000","last_became_peered":"2024-01-11T08:29:30.965589+0000","last_unstale":"2024-01-11T08:29:30.965839+0000","last_undegraded":"2024-01-11T08:29:30.965839+0000","last_fullsized":"2024-01-11T08:29:30.965839+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:09:19.577787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.959573+0000","last_change":"2024-01-11T08:29:30.959573+0000","last_active":"2024-01-11T08:29:30.959573+0000","last_peered":"2024-01-11T08:29:30.959573+0000","last_clean":"2024-01-11T08:29:30.959573+0000","last_became_active":"2024-01-11T08:29:30.958964+0000","last_became_peered":"2024-01-11T08:29:30.958964+0000","last_unstale":"2024-01-11T08:29:30.959573+0000","last_undegraded":"2024-01-11T08:29:30.959573+0000","last_fullsized":"2024-01-11T08:29:30.959573+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:42:02.127224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.943528+0000","last_change":"2024-01-11T08:29:30.943528+0000","last_active":"2024-01-11T08:29:30.943528+0000","last_peered":"2024-01-11T08:29:30.943528+0000","last_clean":"2024-01-11T08:29:30.943528+0000","last_became_active":"2024-01-11T08:29:30.943273+0000","last_became_peered":"2024-01-11T08:29:30.943273+0000","last_unstale":"2024-01-11T08:29:30.943528+0000","last_undegraded":"2024-01-11T08:29:30.943528+0000","last_fullsized":"2024-01-11T08:29:30.943528+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:49:50.965540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.950739+0000","last_change":"2024-01-11T08:29:30.950739+0000","last_active":"2024-01-11T08:29:30.950739+0000","last_peered":"2024-01-11T08:29:30.950739+0000","last_clean":"2024-01-11T08:29:30.950739+0000","last_became_active":"2024-01-11T08:29:30.949011+0000","last_became_peered":"2024-01-11T08:29:30.949011+0000","last_unstale":"2024-01-11T08:29:30.950739+0000","last_undegraded":"2024-01-11T08:29:30.950739+0000","last_fullsized":"2024-01-11T08:29:30.950739+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:38:51.548179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.965377+0000","last_change":"2024-01-11T08:29:30.965377+0000","last_active":"2024-01-11T08:29:30.965377+0000","last_peered":"2024-01-11T08:29:30.965377+0000","last_clean":"2024-01-11T08:29:30.965377+0000","last_became_active":"2024-01-11T08:29:30.965038+0000","last_became_peered":"2024-01-11T08:29:30.965038+0000","last_unstale":"2024-01-11T08:29:30.965377+0000","last_undegraded":"2024-01-11T08:29:30.965377+0000","last_fullsized":"2024-01-11T08:29:30.965377+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:39:29.986735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.961930+0000","last_change":"2024-01-11T08:29:30.961930+0000","last_active":"2024-01-11T08:29:30.961930+0000","last_peered":"2024-01-11T08:29:30.961930+0000","last_clean":"2024-01-11T08:29:30.961930+0000","last_became_active":"2024-01-11T08:29:30.959962+0000","last_became_peered":"2024-01-11T08:29:30.959962+0000","last_unstale":"2024-01-11T08:29:30.961930+0000","last_undegraded":"2024-01-11T08:29:30.961930+0000","last_fullsized":"2024-01-11T08:29:30.961930+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:33:06.520797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.961485+0000","last_change":"2024-01-11T08:29:30.961485+0000","last_active":"2024-01-11T08:29:30.961485+0000","last_peered":"2024-01-11T08:29:30.961485+0000","last_clean":"2024-01-11T08:29:30.961485+0000","last_became_active":"2024-01-11T08:29:30.961241+0000","last_became_peered":"2024-01-11T08:29:30.961241+0000","last_unstale":"2024-01-11T08:29:30.961485+0000","last_undegraded":"2024-01-11T08:29:30.961485+0000","last_fullsized":"2024-01-11T08:29:30.961485+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:08:31.589550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.949914+0000","last_change":"2024-01-11T08:29:30.949914+0000","last_active":"2024-01-11T08:29:30.949914+0000","last_peered":"2024-01-11T08:29:30.949914+0000","last_clean":"2024-01-11T08:29:30.949914+0000","last_became_active":"2024-01-11T08:29:30.949634+0000","last_became_peered":"2024-01-11T08:29:30.949634+0000","last_unstale":"2024-01-11T08:29:30.949914+0000","last_undegraded":"2024-01-11T08:29:30.949914+0000","last_fullsized":"2024-01-11T08:29:30.949914+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:53:28.537689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.974563+0000","last_change":"2024-01-11T08:29:30.974563+0000","last_active":"2024-01-11T08:29:30.974563+0000","last_peered":"2024-01-11T08:29:30.974563+0000","last_clean":"2024-01-11T08:29:30.974563+0000","last_became_active":"2024-01-11T08:29:30.974282+0000","last_became_peered":"2024-01-11T08:29:30.974282+0000","last_unstale":"2024-01-11T08:29:30.974563+0000","last_undegraded":"2024-01-11T08:29:30.974563+0000","last_fullsized":"2024-01-11T08:29:30.974563+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:43:06.583414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.961118+0000","last_change":"2024-01-11T08:29:30.961118+0000","last_active":"2024-01-11T08:29:30.961118+0000","last_peered":"2024-01-11T08:29:30.961118+0000","last_clean":"2024-01-11T08:29:30.961118+0000","last_became_active":"2024-01-11T08:29:30.960837+0000","last_became_peered":"2024-01-11T08:29:30.960837+0000","last_unstale":"2024-01-11T08:29:30.961118+0000","last_undegraded":"2024-01-11T08:29:30.961118+0000","last_fullsized":"2024-01-11T08:29:30.961118+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:30:31.171934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.957164+0000","last_change":"2024-01-11T08:29:30.957164+0000","last_active":"2024-01-11T08:29:30.957164+0000","last_peered":"2024-01-11T08:29:30.957164+0000","last_clean":"2024-01-11T08:29:30.957164+0000","last_became_active":"2024-01-11T08:29:30.956704+0000","last_became_peered":"2024-01-11T08:29:30.956704+0000","last_unstale":"2024-01-11T08:29:30.957164+0000","last_undegraded":"2024-01-11T08:29:30.957164+0000","last_fullsized":"2024-01-11T08:29:30.957164+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:34:38.966829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.943427+0000","last_change":"2024-01-11T08:29:30.943427+0000","last_active":"2024-01-11T08:29:30.943427+0000","last_peered":"2024-01-11T08:29:30.943427+0000","last_clean":"2024-01-11T08:29:30.943427+0000","last_became_active":"2024-01-11T08:29:30.943149+0000","last_became_peered":"2024-01-11T08:29:30.943149+0000","last_unstale":"2024-01-11T08:29:30.943427+0000","last_undegraded":"2024-01-11T08:29:30.943427+0000","last_fullsized":"2024-01-11T08:29:30.943427+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:37:56.694211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.957910+0000","last_change":"2024-01-11T08:29:30.957910+0000","last_active":"2024-01-11T08:29:30.957910+0000","last_peered":"2024-01-11T08:29:30.957910+0000","last_clean":"2024-01-11T08:29:30.957910+0000","last_became_active":"2024-01-11T08:29:30.957357+0000","last_became_peered":"2024-01-11T08:29:30.957357+0000","last_unstale":"2024-01-11T08:29:30.957910+0000","last_undegraded":"2024-01-11T08:29:30.957910+0000","last_fullsized":"2024-01-11T08:29:30.957910+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:31:27.156853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.958538+0000","last_change":"2024-01-11T08:29:30.958538+0000","last_active":"2024-01-11T08:29:30.958538+0000","last_peered":"2024-01-11T08:29:30.958538+0000","last_clean":"2024-01-11T08:29:30.958538+0000","last_became_active":"2024-01-11T08:29:30.958247+0000","last_became_peered":"2024-01-11T08:29:30.958247+0000","last_unstale":"2024-01-11T08:29:30.958538+0000","last_undegraded":"2024-01-11T08:29:30.958538+0000","last_fullsized":"2024-01-11T08:29:30.958538+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:49:51.388649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.955759+0000","last_change":"2024-01-11T08:29:30.955759+0000","last_active":"2024-01-11T08:29:30.955759+0000","last_peered":"2024-01-11T08:29:30.955759+0000","last_clean":"2024-01-11T08:29:30.955759+0000","last_became_active":"2024-01-11T08:29:30.955392+0000","last_became_peered":"2024-01-11T08:29:30.955392+0000","last_unstale":"2024-01-11T08:29:30.955759+0000","last_undegraded":"2024-01-11T08:29:30.955759+0000","last_fullsized":"2024-01-11T08:29:30.955759+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:44:11.291678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.959427+0000","last_change":"2024-01-11T08:29:30.959427+0000","last_active":"2024-01-11T08:29:30.959427+0000","last_peered":"2024-01-11T08:29:30.959427+0000","last_clean":"2024-01-11T08:29:30.959427+0000","last_became_active":"2024-01-11T08:29:30.959180+0000","last_became_peered":"2024-01-11T08:29:30.959180+0000","last_unstale":"2024-01-11T08:29:30.959427+0000","last_undegraded":"2024-01-11T08:29:30.959427+0000","last_fullsized":"2024-01-11T08:29:30.959427+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:51:30.126948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.952638+0000","last_change":"2024-01-11T08:29:30.952638+0000","last_active":"2024-01-11T08:29:30.952638+0000","last_peered":"2024-01-11T08:29:30.952638+0000","last_clean":"2024-01-11T08:29:30.952638+0000","last_became_active":"2024-01-11T08:29:30.951632+0000","last_became_peered":"2024-01-11T08:29:30.951632+0000","last_unstale":"2024-01-11T08:29:30.952638+0000","last_undegraded":"2024-01-11T08:29:30.952638+0000","last_fullsized":"2024-01-11T08:29:30.952638+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:00:15.233533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.951297+0000","last_change":"2024-01-11T08:29:30.951297+0000","last_active":"2024-01-11T08:29:30.951297+0000","last_peered":"2024-01-11T08:29:30.951297+0000","last_clean":"2024-01-11T08:29:30.951297+0000","last_became_active":"2024-01-11T08:29:30.950800+0000","last_became_peered":"2024-01-11T08:29:30.950800+0000","last_unstale":"2024-01-11T08:29:30.951297+0000","last_undegraded":"2024-01-11T08:29:30.951297+0000","last_fullsized":"2024-01-11T08:29:30.951297+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:39:29.986735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.948704+0000","last_change":"2024-01-11T08:29:30.948704+0000","last_active":"2024-01-11T08:29:30.948704+0000","last_peered":"2024-01-11T08:29:30.948704+0000","last_clean":"2024-01-11T08:29:30.948704+0000","last_became_active":"2024-01-11T08:29:30.948399+0000","last_became_peered":"2024-01-11T08:29:30.948399+0000","last_unstale":"2024-01-11T08:29:30.948704+0000","last_undegraded":"2024-01-11T08:29:30.948704+0000","last_fullsized":"2024-01-11T08:29:30.948704+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:41:16.769444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.945330+0000","last_change":"2024-01-11T08:29:30.945330+0000","last_active":"2024-01-11T08:29:30.945330+0000","last_peered":"2024-01-11T08:29:30.945330+0000","last_clean":"2024-01-11T08:29:30.945330+0000","last_became_active":"2024-01-11T08:29:30.945015+0000","last_became_peered":"2024-01-11T08:29:30.945015+0000","last_unstale":"2024-01-11T08:29:30.945330+0000","last_undegraded":"2024-01-11T08:29:30.945330+0000","last_fullsized":"2024-01-11T08:29:30.945330+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:35:48.167273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.948101+0000","last_change":"2024-01-11T08:29:30.948101+0000","last_active":"2024-01-11T08:29:30.948101+0000","last_peered":"2024-01-11T08:29:30.948101+0000","last_clean":"2024-01-11T08:29:30.948101+0000","last_became_active":"2024-01-11T08:29:30.947390+0000","last_became_peered":"2024-01-11T08:29:30.947390+0000","last_unstale":"2024-01-11T08:29:30.948101+0000","last_undegraded":"2024-01-11T08:29:30.948101+0000","last_fullsized":"2024-01-11T08:29:30.948101+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:43:31.364333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.953105+0000","last_change":"2024-01-11T08:29:30.953105+0000","last_active":"2024-01-11T08:29:30.953105+0000","last_peered":"2024-01-11T08:29:30.953105+0000","last_clean":"2024-01-11T08:29:30.953105+0000","last_became_active":"2024-01-11T08:29:30.952821+0000","last_became_peered":"2024-01-11T08:29:30.952821+0000","last_unstale":"2024-01-11T08:29:30.953105+0000","last_undegraded":"2024-01-11T08:29:30.953105+0000","last_fullsized":"2024-01-11T08:29:30.953105+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:34:09.126082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.953883+0000","last_change":"2024-01-11T08:29:30.953883+0000","last_active":"2024-01-11T08:29:30.953883+0000","last_peered":"2024-01-11T08:29:30.953883+0000","last_clean":"2024-01-11T08:29:30.953883+0000","last_became_active":"2024-01-11T08:29:30.953590+0000","last_became_peered":"2024-01-11T08:29:30.953590+0000","last_unstale":"2024-01-11T08:29:30.953883+0000","last_undegraded":"2024-01-11T08:29:30.953883+0000","last_fullsized":"2024-01-11T08:29:30.953883+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:33:35.619652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.957996+0000","last_change":"2024-01-11T08:29:30.957996+0000","last_active":"2024-01-11T08:29:30.957996+0000","last_peered":"2024-01-11T08:29:30.957996+0000","last_clean":"2024-01-11T08:29:30.957996+0000","last_became_active":"2024-01-11T08:29:30.957689+0000","last_became_peered":"2024-01-11T08:29:30.957689+0000","last_unstale":"2024-01-11T08:29:30.957996+0000","last_undegraded":"2024-01-11T08:29:30.957996+0000","last_fullsized":"2024-01-11T08:29:30.957996+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:13:49.966637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.955988+0000","last_change":"2024-01-11T08:29:30.955988+0000","last_active":"2024-01-11T08:29:30.955988+0000","last_peered":"2024-01-11T08:29:30.955988+0000","last_clean":"2024-01-11T08:29:30.955988+0000","last_became_active":"2024-01-11T08:29:30.954004+0000","last_became_peered":"2024-01-11T08:29:30.954004+0000","last_unstale":"2024-01-11T08:29:30.955988+0000","last_undegraded":"2024-01-11T08:29:30.955988+0000","last_fullsized":"2024-01-11T08:29:30.955988+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:45:57.853427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.945616+0000","last_change":"2024-01-11T08:29:30.945616+0000","last_active":"2024-01-11T08:29:30.945616+0000","last_peered":"2024-01-11T08:29:30.945616+0000","last_clean":"2024-01-11T08:29:30.945616+0000","last_became_active":"2024-01-11T08:29:30.945229+0000","last_became_peered":"2024-01-11T08:29:30.945229+0000","last_unstale":"2024-01-11T08:29:30.945616+0000","last_undegraded":"2024-01-11T08:29:30.945616+0000","last_fullsized":"2024-01-11T08:29:30.945616+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:09:45.932475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.940292+0000","last_change":"2024-01-11T08:29:30.940292+0000","last_active":"2024-01-11T08:29:30.940292+0000","last_peered":"2024-01-11T08:29:30.940292+0000","last_clean":"2024-01-11T08:29:30.940292+0000","last_became_active":"2024-01-11T08:29:30.940006+0000","last_became_peered":"2024-01-11T08:29:30.940006+0000","last_unstale":"2024-01-11T08:29:30.940292+0000","last_undegraded":"2024-01-11T08:29:30.940292+0000","last_fullsized":"2024-01-11T08:29:30.940292+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:43:33.145473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.954868+0000","last_change":"2024-01-11T08:29:30.954868+0000","last_active":"2024-01-11T08:29:30.954868+0000","last_peered":"2024-01-11T08:29:30.954868+0000","last_clean":"2024-01-11T08:29:30.954868+0000","last_became_active":"2024-01-11T08:29:30.954543+0000","last_became_peered":"2024-01-11T08:29:30.954543+0000","last_unstale":"2024-01-11T08:29:30.954868+0000","last_undegraded":"2024-01-11T08:29:30.954868+0000","last_fullsized":"2024-01-11T08:29:30.954868+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:40:39.476340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.946701+0000","last_change":"2024-01-11T08:29:30.946701+0000","last_active":"2024-01-11T08:29:30.946701+0000","last_peered":"2024-01-11T08:29:30.946701+0000","last_clean":"2024-01-11T08:29:30.946701+0000","last_became_active":"2024-01-11T08:29:30.945412+0000","last_became_peered":"2024-01-11T08:29:30.945412+0000","last_unstale":"2024-01-11T08:29:30.946701+0000","last_undegraded":"2024-01-11T08:29:30.946701+0000","last_fullsized":"2024-01-11T08:29:30.946701+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:45:23.561914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.952924+0000","last_change":"2024-01-11T08:29:30.952924+0000","last_active":"2024-01-11T08:29:30.952924+0000","last_peered":"2024-01-11T08:29:30.952924+0000","last_clean":"2024-01-11T08:29:30.952924+0000","last_became_active":"2024-01-11T08:29:30.952593+0000","last_became_peered":"2024-01-11T08:29:30.952593+0000","last_unstale":"2024-01-11T08:29:30.952924+0000","last_undegraded":"2024-01-11T08:29:30.952924+0000","last_fullsized":"2024-01-11T08:29:30.952924+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:10.043861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"130'4","reported_seq":13,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.967596+0000","last_change":"2024-01-11T08:29:29.947194+0000","last_active":"2024-01-11T08:29:30.967596+0000","last_peered":"2024-01-11T08:29:30.967596+0000","last_clean":"2024-01-11T08:29:30.967596+0000","last_became_active":"2024-01-11T08:29:29.946642+0000","last_became_peered":"2024-01-11T08:29:29.946642+0000","last_unstale":"2024-01-11T08:29:30.967596+0000","last_undegraded":"2024-01-11T08:29:30.967596+0000","last_fullsized":"2024-01-11T08:29:30.967596+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:07:11.026229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.951689+0000","last_change":"2024-01-11T08:29:30.951689+0000","last_active":"2024-01-11T08:29:30.951689+0000","last_peered":"2024-01-11T08:29:30.951689+0000","last_clean":"2024-01-11T08:29:30.951689+0000","last_became_active":"2024-01-11T08:29:30.950138+0000","last_became_peered":"2024-01-11T08:29:30.950138+0000","last_unstale":"2024-01-11T08:29:30.951689+0000","last_undegraded":"2024-01-11T08:29:30.951689+0000","last_fullsized":"2024-01-11T08:29:30.951689+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:43:31.364333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"130'1","reported_seq":10,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.946851+0000","last_change":"2024-01-11T08:29:29.951273+0000","last_active":"2024-01-11T08:29:30.946851+0000","last_peered":"2024-01-11T08:29:30.946851+0000","last_clean":"2024-01-11T08:29:30.946851+0000","last_became_active":"2024-01-11T08:29:29.950949+0000","last_became_peered":"2024-01-11T08:29:29.950949+0000","last_unstale":"2024-01-11T08:29:30.946851+0000","last_undegraded":"2024-01-11T08:29:30.946851+0000","last_fullsized":"2024-01-11T08:29:30.946851+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:37:00.082089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.944353+0000","last_change":"2024-01-11T08:29:30.944353+0000","last_active":"2024-01-11T08:29:30.944353+0000","last_peered":"2024-01-11T08:29:30.944353+0000","last_clean":"2024-01-11T08:29:30.944353+0000","last_became_active":"2024-01-11T08:29:30.943940+0000","last_became_peered":"2024-01-11T08:29:30.943940+0000","last_unstale":"2024-01-11T08:29:30.944353+0000","last_undegraded":"2024-01-11T08:29:30.944353+0000","last_fullsized":"2024-01-11T08:29:30.944353+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:01.558226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"130'2","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.971573+0000","last_change":"2024-01-11T08:29:29.950060+0000","last_active":"2024-01-11T08:29:30.971573+0000","last_peered":"2024-01-11T08:29:30.971573+0000","last_clean":"2024-01-11T08:29:30.971573+0000","last_became_active":"2024-01-11T08:29:29.949747+0000","last_became_peered":"2024-01-11T08:29:29.949747+0000","last_unstale":"2024-01-11T08:29:30.971573+0000","last_undegraded":"2024-01-11T08:29:30.971573+0000","last_fullsized":"2024-01-11T08:29:30.971573+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:09:52.953864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.961214+0000","last_change":"2024-01-11T08:29:30.961214+0000","last_active":"2024-01-11T08:29:30.961214+0000","last_peered":"2024-01-11T08:29:30.961214+0000","last_clean":"2024-01-11T08:29:30.961214+0000","last_became_active":"2024-01-11T08:29:30.960933+0000","last_became_peered":"2024-01-11T08:29:30.960933+0000","last_unstale":"2024-01-11T08:29:30.961214+0000","last_undegraded":"2024-01-11T08:29:30.961214+0000","last_fullsized":"2024-01-11T08:29:30.961214+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:44:11.291678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"130'2","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.974876+0000","last_change":"2024-01-11T08:29:29.950323+0000","last_active":"2024-01-11T08:29:30.974876+0000","last_peered":"2024-01-11T08:29:30.974876+0000","last_clean":"2024-01-11T08:29:30.974876+0000","last_became_active":"2024-01-11T08:29:29.950027+0000","last_became_peered":"2024-01-11T08:29:29.950027+0000","last_unstale":"2024-01-11T08:29:30.974876+0000","last_undegraded":"2024-01-11T08:29:30.974876+0000","last_fullsized":"2024-01-11T08:29:30.974876+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:47.073306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.956192+0000","last_change":"2024-01-11T08:29:30.956192+0000","last_active":"2024-01-11T08:29:30.956192+0000","last_peered":"2024-01-11T08:29:30.956192+0000","last_clean":"2024-01-11T08:29:30.956192+0000","last_became_active":"2024-01-11T08:29:30.955351+0000","last_became_peered":"2024-01-11T08:29:30.955351+0000","last_unstale":"2024-01-11T08:29:30.956192+0000","last_undegraded":"2024-01-11T08:29:30.956192+0000","last_fullsized":"2024-01-11T08:29:30.956192+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:07.405650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"130'1","reported_seq":10,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.935921+0000","last_change":"2024-01-11T08:29:29.944874+0000","last_active":"2024-01-11T08:29:30.935921+0000","last_peered":"2024-01-11T08:29:30.935921+0000","last_clean":"2024-01-11T08:29:30.935921+0000","last_became_active":"2024-01-11T08:29:29.944590+0000","last_became_peered":"2024-01-11T08:29:29.944590+0000","last_unstale":"2024-01-11T08:29:30.935921+0000","last_undegraded":"2024-01-11T08:29:30.935921+0000","last_fullsized":"2024-01-11T08:29:30.935921+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:43:27.465006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.945184+0000","last_change":"2024-01-11T08:29:30.945184+0000","last_active":"2024-01-11T08:29:30.945184+0000","last_peered":"2024-01-11T08:29:30.945184+0000","last_clean":"2024-01-11T08:29:30.945184+0000","last_became_active":"2024-01-11T08:29:30.944133+0000","last_became_peered":"2024-01-11T08:29:30.944133+0000","last_unstale":"2024-01-11T08:29:30.945184+0000","last_undegraded":"2024-01-11T08:29:30.945184+0000","last_fullsized":"2024-01-11T08:29:30.945184+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:29:12.239105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"130'1","reported_seq":10,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.933206+0000","last_change":"2024-01-11T08:29:29.943256+0000","last_active":"2024-01-11T08:29:30.933206+0000","last_peered":"2024-01-11T08:29:30.933206+0000","last_clean":"2024-01-11T08:29:30.933206+0000","last_became_active":"2024-01-11T08:29:29.942633+0000","last_became_peered":"2024-01-11T08:29:29.942633+0000","last_unstale":"2024-01-11T08:29:30.933206+0000","last_undegraded":"2024-01-11T08:29:30.933206+0000","last_fullsized":"2024-01-11T08:29:30.933206+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:03:39.111068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.968204+0000","last_change":"2024-01-11T08:29:30.968204+0000","last_active":"2024-01-11T08:29:30.968204+0000","last_peered":"2024-01-11T08:29:30.968204+0000","last_clean":"2024-01-11T08:29:30.968204+0000","last_became_active":"2024-01-11T08:29:30.967922+0000","last_became_peered":"2024-01-11T08:29:30.967922+0000","last_unstale":"2024-01-11T08:29:30.968204+0000","last_undegraded":"2024-01-11T08:29:30.968204+0000","last_fullsized":"2024-01-11T08:29:30.968204+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:56:44.744224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.945613+0000","last_change":"2024-01-11T08:29:29.943508+0000","last_active":"2024-01-11T08:29:29.945613+0000","last_peered":"2024-01-11T08:29:29.945613+0000","last_clean":"2024-01-11T08:29:29.945613+0000","last_became_active":"2024-01-11T08:29:29.943206+0000","last_became_peered":"2024-01-11T08:29:29.943206+0000","last_unstale":"2024-01-11T08:29:29.945613+0000","last_undegraded":"2024-01-11T08:29:29.945613+0000","last_fullsized":"2024-01-11T08:29:29.945613+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:29:34.325473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.951160+0000","last_change":"2024-01-11T08:29:30.951160+0000","last_active":"2024-01-11T08:29:30.951160+0000","last_peered":"2024-01-11T08:29:30.951160+0000","last_clean":"2024-01-11T08:29:30.951160+0000","last_became_active":"2024-01-11T08:29:30.949366+0000","last_became_peered":"2024-01-11T08:29:30.949366+0000","last_unstale":"2024-01-11T08:29:30.951160+0000","last_undegraded":"2024-01-11T08:29:30.951160+0000","last_fullsized":"2024-01-11T08:29:30.951160+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:41:26.264311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.963241+0000","last_change":"2024-01-11T08:29:29.959452+0000","last_active":"2024-01-11T08:29:29.963241+0000","last_peered":"2024-01-11T08:29:29.963241+0000","last_clean":"2024-01-11T08:29:29.963241+0000","last_became_active":"2024-01-11T08:29:29.959146+0000","last_became_peered":"2024-01-11T08:29:29.959146+0000","last_unstale":"2024-01-11T08:29:29.963241+0000","last_undegraded":"2024-01-11T08:29:29.963241+0000","last_fullsized":"2024-01-11T08:29:29.963241+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:39:28.976690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.951918+0000","last_change":"2024-01-11T08:29:30.951918+0000","last_active":"2024-01-11T08:29:30.951918+0000","last_peered":"2024-01-11T08:29:30.951918+0000","last_clean":"2024-01-11T08:29:30.951918+0000","last_became_active":"2024-01-11T08:29:30.951605+0000","last_became_peered":"2024-01-11T08:29:30.951605+0000","last_unstale":"2024-01-11T08:29:30.951918+0000","last_undegraded":"2024-01-11T08:29:30.951918+0000","last_fullsized":"2024-01-11T08:29:30.951918+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:53:58.467438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.955671+0000","last_change":"2024-01-11T08:29:29.952559+0000","last_active":"2024-01-11T08:29:29.955671+0000","last_peered":"2024-01-11T08:29:29.955671+0000","last_clean":"2024-01-11T08:29:29.955671+0000","last_became_active":"2024-01-11T08:29:29.952290+0000","last_became_peered":"2024-01-11T08:29:29.952290+0000","last_unstale":"2024-01-11T08:29:29.955671+0000","last_undegraded":"2024-01-11T08:29:29.955671+0000","last_fullsized":"2024-01-11T08:29:29.955671+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:03:50.661693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.966348+0000","last_change":"2024-01-11T08:29:30.966348+0000","last_active":"2024-01-11T08:29:30.966348+0000","last_peered":"2024-01-11T08:29:30.966348+0000","last_clean":"2024-01-11T08:29:30.966348+0000","last_became_active":"2024-01-11T08:29:30.966023+0000","last_became_peered":"2024-01-11T08:29:30.966023+0000","last_unstale":"2024-01-11T08:29:30.966348+0000","last_undegraded":"2024-01-11T08:29:30.966348+0000","last_fullsized":"2024-01-11T08:29:30.966348+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:39:29.986735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.948084+0000","last_change":"2024-01-11T08:29:29.944467+0000","last_active":"2024-01-11T08:29:29.948084+0000","last_peered":"2024-01-11T08:29:29.948084+0000","last_clean":"2024-01-11T08:29:29.948084+0000","last_became_active":"2024-01-11T08:29:29.944114+0000","last_became_peered":"2024-01-11T08:29:29.944114+0000","last_unstale":"2024-01-11T08:29:29.948084+0000","last_undegraded":"2024-01-11T08:29:29.948084+0000","last_fullsized":"2024-01-11T08:29:29.948084+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:40:55.948273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.944445+0000","last_change":"2024-01-11T08:29:30.944445+0000","last_active":"2024-01-11T08:29:30.944445+0000","last_peered":"2024-01-11T08:29:30.944445+0000","last_clean":"2024-01-11T08:29:30.944445+0000","last_became_active":"2024-01-11T08:29:30.944199+0000","last_became_peered":"2024-01-11T08:29:30.944199+0000","last_unstale":"2024-01-11T08:29:30.944445+0000","last_undegraded":"2024-01-11T08:29:30.944445+0000","last_fullsized":"2024-01-11T08:29:30.944445+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:14:37.534117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"130'3","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.973607+0000","last_change":"2024-01-11T08:29:29.956521+0000","last_active":"2024-01-11T08:29:30.973607+0000","last_peered":"2024-01-11T08:29:30.973607+0000","last_clean":"2024-01-11T08:29:30.973607+0000","last_became_active":"2024-01-11T08:29:29.956198+0000","last_became_peered":"2024-01-11T08:29:29.956198+0000","last_unstale":"2024-01-11T08:29:30.973607+0000","last_undegraded":"2024-01-11T08:29:30.973607+0000","last_fullsized":"2024-01-11T08:29:30.973607+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:01:36.960192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.947471+0000","last_change":"2024-01-11T08:29:30.947471+0000","last_active":"2024-01-11T08:29:30.947471+0000","last_peered":"2024-01-11T08:29:30.947471+0000","last_clean":"2024-01-11T08:29:30.947471+0000","last_became_active":"2024-01-11T08:29:30.947189+0000","last_became_peered":"2024-01-11T08:29:30.947189+0000","last_unstale":"2024-01-11T08:29:30.947471+0000","last_undegraded":"2024-01-11T08:29:30.947471+0000","last_fullsized":"2024-01-11T08:29:30.947471+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:01:14.432674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"130'6","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.978114+0000","last_change":"2024-01-11T08:29:29.961494+0000","last_active":"2024-01-11T08:29:30.978114+0000","last_peered":"2024-01-11T08:29:30.978114+0000","last_clean":"2024-01-11T08:29:30.978114+0000","last_became_active":"2024-01-11T08:29:29.961161+0000","last_became_peered":"2024-01-11T08:29:29.961161+0000","last_unstale":"2024-01-11T08:29:30.978114+0000","last_undegraded":"2024-01-11T08:29:30.978114+0000","last_fullsized":"2024-01-11T08:29:30.978114+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:51:29.116903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.939636+0000","last_change":"2024-01-11T08:29:30.939636+0000","last_active":"2024-01-11T08:29:30.939636+0000","last_peered":"2024-01-11T08:29:30.939636+0000","last_clean":"2024-01-11T08:29:30.939636+0000","last_became_active":"2024-01-11T08:29:30.938481+0000","last_became_peered":"2024-01-11T08:29:30.938481+0000","last_unstale":"2024-01-11T08:29:30.939636+0000","last_undegraded":"2024-01-11T08:29:30.939636+0000","last_fullsized":"2024-01-11T08:29:30.939636+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:01:37.810887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"130'2","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.975192+0000","last_change":"2024-01-11T08:29:29.946242+0000","last_active":"2024-01-11T08:29:30.975192+0000","last_peered":"2024-01-11T08:29:30.975192+0000","last_clean":"2024-01-11T08:29:30.975192+0000","last_became_active":"2024-01-11T08:29:29.945990+0000","last_became_peered":"2024-01-11T08:29:29.945990+0000","last_unstale":"2024-01-11T08:29:30.975192+0000","last_undegraded":"2024-01-11T08:29:30.975192+0000","last_fullsized":"2024-01-11T08:29:30.975192+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:31:19.370117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.947359+0000","last_change":"2024-01-11T08:29:30.947359+0000","last_active":"2024-01-11T08:29:30.947359+0000","last_peered":"2024-01-11T08:29:30.947359+0000","last_clean":"2024-01-11T08:29:30.947359+0000","last_became_active":"2024-01-11T08:29:30.947094+0000","last_became_peered":"2024-01-11T08:29:30.947094+0000","last_unstale":"2024-01-11T08:29:30.947359+0000","last_undegraded":"2024-01-11T08:29:30.947359+0000","last_fullsized":"2024-01-11T08:29:30.947359+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:30:31.171934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.955101+0000","last_change":"2024-01-11T08:29:29.951218+0000","last_active":"2024-01-11T08:29:29.955101+0000","last_peered":"2024-01-11T08:29:29.955101+0000","last_clean":"2024-01-11T08:29:29.955101+0000","last_became_active":"2024-01-11T08:29:29.950921+0000","last_became_peered":"2024-01-11T08:29:29.950921+0000","last_unstale":"2024-01-11T08:29:29.955101+0000","last_undegraded":"2024-01-11T08:29:29.955101+0000","last_fullsized":"2024-01-11T08:29:29.955101+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:34:37.956784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.942424+0000","last_change":"2024-01-11T08:29:30.942424+0000","last_active":"2024-01-11T08:29:30.942424+0000","last_peered":"2024-01-11T08:29:30.942424+0000","last_clean":"2024-01-11T08:29:30.942424+0000","last_became_active":"2024-01-11T08:29:30.941993+0000","last_became_peered":"2024-01-11T08:29:30.941993+0000","last_unstale":"2024-01-11T08:29:30.942424+0000","last_undegraded":"2024-01-11T08:29:30.942424+0000","last_fullsized":"2024-01-11T08:29:30.942424+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:45:08.580788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"130'8","reported_seq":17,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.980776+0000","last_change":"2024-01-11T08:29:29.957062+0000","last_active":"2024-01-11T08:29:30.980776+0000","last_peered":"2024-01-11T08:29:30.980776+0000","last_clean":"2024-01-11T08:29:30.980776+0000","last_became_active":"2024-01-11T08:29:29.956817+0000","last_became_peered":"2024-01-11T08:29:29.956817+0000","last_unstale":"2024-01-11T08:29:30.980776+0000","last_undegraded":"2024-01-11T08:29:30.980776+0000","last_fullsized":"2024-01-11T08:29:30.980776+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:57:32.872746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.966099+0000","last_change":"2024-01-11T08:29:30.966099+0000","last_active":"2024-01-11T08:29:30.966099+0000","last_peered":"2024-01-11T08:29:30.966099+0000","last_clean":"2024-01-11T08:29:30.966099+0000","last_became_active":"2024-01-11T08:29:30.965647+0000","last_became_peered":"2024-01-11T08:29:30.965647+0000","last_unstale":"2024-01-11T08:29:30.966099+0000","last_undegraded":"2024-01-11T08:29:30.966099+0000","last_fullsized":"2024-01-11T08:29:30.966099+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:02:55.663995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"130'1","reported_seq":10,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.953136+0000","last_change":"2024-01-11T08:29:29.955054+0000","last_active":"2024-01-11T08:29:30.953136+0000","last_peered":"2024-01-11T08:29:30.953136+0000","last_clean":"2024-01-11T08:29:30.953136+0000","last_became_active":"2024-01-11T08:29:29.954838+0000","last_became_peered":"2024-01-11T08:29:29.954838+0000","last_unstale":"2024-01-11T08:29:30.953136+0000","last_undegraded":"2024-01-11T08:29:30.953136+0000","last_fullsized":"2024-01-11T08:29:30.953136+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:45:56.843382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.978571+0000","last_change":"2024-01-11T08:29:30.978571+0000","last_active":"2024-01-11T08:29:30.978571+0000","last_peered":"2024-01-11T08:29:30.978571+0000","last_clean":"2024-01-11T08:29:30.978571+0000","last_became_active":"2024-01-11T08:29:30.977906+0000","last_became_peered":"2024-01-11T08:29:30.977906+0000","last_unstale":"2024-01-11T08:29:30.978571+0000","last_undegraded":"2024-01-11T08:29:30.978571+0000","last_fullsized":"2024-01-11T08:29:30.978571+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:28.686712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"130'2","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.973678+0000","last_change":"2024-01-11T08:29:29.947489+0000","last_active":"2024-01-11T08:29:30.973678+0000","last_peered":"2024-01-11T08:29:30.973678+0000","last_clean":"2024-01-11T08:29:30.973678+0000","last_became_active":"2024-01-11T08:29:29.947244+0000","last_became_peered":"2024-01-11T08:29:29.947244+0000","last_unstale":"2024-01-11T08:29:30.973678+0000","last_undegraded":"2024-01-11T08:29:30.973678+0000","last_fullsized":"2024-01-11T08:29:30.973678+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:22:49.796325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.962953+0000","last_change":"2024-01-11T08:29:30.962953+0000","last_active":"2024-01-11T08:29:30.962953+0000","last_peered":"2024-01-11T08:29:30.962953+0000","last_clean":"2024-01-11T08:29:30.962953+0000","last_became_active":"2024-01-11T08:29:30.962619+0000","last_became_peered":"2024-01-11T08:29:30.962619+0000","last_unstale":"2024-01-11T08:29:30.962953+0000","last_undegraded":"2024-01-11T08:29:30.962953+0000","last_fullsized":"2024-01-11T08:29:30.962953+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:09:45.932475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"130'1","reported_seq":10,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.948165+0000","last_change":"2024-01-11T08:29:29.946009+0000","last_active":"2024-01-11T08:29:30.948165+0000","last_peered":"2024-01-11T08:29:30.948165+0000","last_clean":"2024-01-11T08:29:30.948165+0000","last_became_active":"2024-01-11T08:29:29.945724+0000","last_became_peered":"2024-01-11T08:29:29.945724+0000","last_unstale":"2024-01-11T08:29:30.948165+0000","last_undegraded":"2024-01-11T08:29:30.948165+0000","last_fullsized":"2024-01-11T08:29:30.948165+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:53:34.148937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.958296+0000","last_change":"2024-01-11T08:29:30.958296+0000","last_active":"2024-01-11T08:29:30.958296+0000","last_peered":"2024-01-11T08:29:30.958296+0000","last_clean":"2024-01-11T08:29:30.958296+0000","last_became_active":"2024-01-11T08:29:30.957581+0000","last_became_peered":"2024-01-11T08:29:30.957581+0000","last_unstale":"2024-01-11T08:29:30.958296+0000","last_undegraded":"2024-01-11T08:29:30.958296+0000","last_fullsized":"2024-01-11T08:29:30.958296+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:01:37.810887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.960767+0000","last_change":"2024-01-11T08:29:29.959927+0000","last_active":"2024-01-11T08:29:29.960767+0000","last_peered":"2024-01-11T08:29:29.960767+0000","last_clean":"2024-01-11T08:29:29.960767+0000","last_became_active":"2024-01-11T08:29:29.959609+0000","last_became_peered":"2024-01-11T08:29:29.959609+0000","last_unstale":"2024-01-11T08:29:29.960767+0000","last_undegraded":"2024-01-11T08:29:29.960767+0000","last_fullsized":"2024-01-11T08:29:29.960767+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:24:47.609089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.956089+0000","last_change":"2024-01-11T08:29:30.956089+0000","last_active":"2024-01-11T08:29:30.956089+0000","last_peered":"2024-01-11T08:29:30.956089+0000","last_clean":"2024-01-11T08:29:30.956089+0000","last_became_active":"2024-01-11T08:29:30.955831+0000","last_became_peered":"2024-01-11T08:29:30.955831+0000","last_unstale":"2024-01-11T08:29:30.956089+0000","last_undegraded":"2024-01-11T08:29:30.956089+0000","last_fullsized":"2024-01-11T08:29:30.956089+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:41:16.769444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"130'2","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.978316+0000","last_change":"2024-01-11T08:29:29.956455+0000","last_active":"2024-01-11T08:29:30.978316+0000","last_peered":"2024-01-11T08:29:30.978316+0000","last_clean":"2024-01-11T08:29:30.978316+0000","last_became_active":"2024-01-11T08:29:29.956162+0000","last_became_peered":"2024-01-11T08:29:29.956162+0000","last_unstale":"2024-01-11T08:29:30.978316+0000","last_undegraded":"2024-01-11T08:29:30.978316+0000","last_fullsized":"2024-01-11T08:29:30.978316+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:47:10.201254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.974352+0000","last_change":"2024-01-11T08:29:30.974352+0000","last_active":"2024-01-11T08:29:30.974352+0000","last_peered":"2024-01-11T08:29:30.974352+0000","last_clean":"2024-01-11T08:29:30.974352+0000","last_became_active":"2024-01-11T08:29:30.972851+0000","last_became_peered":"2024-01-11T08:29:30.972851+0000","last_unstale":"2024-01-11T08:29:30.974352+0000","last_undegraded":"2024-01-11T08:29:30.974352+0000","last_fullsized":"2024-01-11T08:29:30.974352+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:48:12.562689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.958216+0000","last_change":"2024-01-11T08:29:29.954402+0000","last_active":"2024-01-11T08:29:29.958216+0000","last_peered":"2024-01-11T08:29:29.958216+0000","last_clean":"2024-01-11T08:29:29.958216+0000","last_became_active":"2024-01-11T08:29:29.954114+0000","last_became_peered":"2024-01-11T08:29:29.954114+0000","last_unstale":"2024-01-11T08:29:29.958216+0000","last_undegraded":"2024-01-11T08:29:29.958216+0000","last_fullsized":"2024-01-11T08:29:29.958216+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:48:39.783474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.961245+0000","last_change":"2024-01-11T08:29:30.961245+0000","last_active":"2024-01-11T08:29:30.961245+0000","last_peered":"2024-01-11T08:29:30.961245+0000","last_clean":"2024-01-11T08:29:30.961245+0000","last_became_active":"2024-01-11T08:29:30.960951+0000","last_became_peered":"2024-01-11T08:29:30.960951+0000","last_unstale":"2024-01-11T08:29:30.961245+0000","last_undegraded":"2024-01-11T08:29:30.961245+0000","last_fullsized":"2024-01-11T08:29:30.961245+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:45:57.853427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.953069+0000","last_change":"2024-01-11T08:29:29.949977+0000","last_active":"2024-01-11T08:29:29.953069+0000","last_peered":"2024-01-11T08:29:29.953069+0000","last_clean":"2024-01-11T08:29:29.953069+0000","last_became_active":"2024-01-11T08:29:29.949643+0000","last_became_peered":"2024-01-11T08:29:29.949643+0000","last_unstale":"2024-01-11T08:29:29.953069+0000","last_undegraded":"2024-01-11T08:29:29.953069+0000","last_fullsized":"2024-01-11T08:29:29.953069+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:42:01.117179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.956708+0000","last_change":"2024-01-11T08:29:30.956708+0000","last_active":"2024-01-11T08:29:30.956708+0000","last_peered":"2024-01-11T08:29:30.956708+0000","last_clean":"2024-01-11T08:29:30.956708+0000","last_became_active":"2024-01-11T08:29:30.956436+0000","last_became_peered":"2024-01-11T08:29:30.956436+0000","last_unstale":"2024-01-11T08:29:30.956708+0000","last_undegraded":"2024-01-11T08:29:30.956708+0000","last_fullsized":"2024-01-11T08:29:30.956708+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:44:11.291678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.960848+0000","last_change":"2024-01-11T08:29:29.957305+0000","last_active":"2024-01-11T08:29:29.960848+0000","last_peered":"2024-01-11T08:29:29.960848+0000","last_clean":"2024-01-11T08:29:29.960848+0000","last_became_active":"2024-01-11T08:29:29.955662+0000","last_became_peered":"2024-01-11T08:29:29.955662+0000","last_unstale":"2024-01-11T08:29:29.960848+0000","last_undegraded":"2024-01-11T08:29:29.960848+0000","last_fullsized":"2024-01-11T08:29:29.960848+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:29:11.229060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.956563+0000","last_change":"2024-01-11T08:29:30.956563+0000","last_active":"2024-01-11T08:29:30.956563+0000","last_peered":"2024-01-11T08:29:30.956563+0000","last_clean":"2024-01-11T08:29:30.956563+0000","last_became_active":"2024-01-11T08:29:30.955684+0000","last_became_peered":"2024-01-11T08:29:30.955684+0000","last_unstale":"2024-01-11T08:29:30.956563+0000","last_undegraded":"2024-01-11T08:29:30.956563+0000","last_fullsized":"2024-01-11T08:29:30.956563+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:51:30.126948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"130'2","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.980211+0000","last_change":"2024-01-11T08:29:29.955328+0000","last_active":"2024-01-11T08:29:30.980211+0000","last_peered":"2024-01-11T08:29:30.980211+0000","last_clean":"2024-01-11T08:29:30.980211+0000","last_became_active":"2024-01-11T08:29:29.955101+0000","last_became_peered":"2024-01-11T08:29:29.955101+0000","last_unstale":"2024-01-11T08:29:30.980211+0000","last_undegraded":"2024-01-11T08:29:30.980211+0000","last_fullsized":"2024-01-11T08:29:30.980211+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:02:38.334428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.967979+0000","last_change":"2024-01-11T08:29:30.967979+0000","last_active":"2024-01-11T08:29:30.967979+0000","last_peered":"2024-01-11T08:29:30.967979+0000","last_clean":"2024-01-11T08:29:30.967979+0000","last_became_active":"2024-01-11T08:29:30.967371+0000","last_became_peered":"2024-01-11T08:29:30.967371+0000","last_unstale":"2024-01-11T08:29:30.967979+0000","last_undegraded":"2024-01-11T08:29:30.967979+0000","last_fullsized":"2024-01-11T08:29:30.967979+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:37:23.414047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.950116+0000","last_change":"2024-01-11T08:29:29.947100+0000","last_active":"2024-01-11T08:29:29.950116+0000","last_peered":"2024-01-11T08:29:29.950116+0000","last_clean":"2024-01-11T08:29:29.950116+0000","last_became_active":"2024-01-11T08:29:29.946793+0000","last_became_peered":"2024-01-11T08:29:29.946793+0000","last_unstale":"2024-01-11T08:29:29.950116+0000","last_undegraded":"2024-01-11T08:29:29.950116+0000","last_fullsized":"2024-01-11T08:29:29.950116+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:38:01.095587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.967378+0000","last_change":"2024-01-11T08:29:30.967378+0000","last_active":"2024-01-11T08:29:30.967378+0000","last_peered":"2024-01-11T08:29:30.967378+0000","last_clean":"2024-01-11T08:29:30.967378+0000","last_became_active":"2024-01-11T08:29:30.967123+0000","last_became_peered":"2024-01-11T08:29:30.967123+0000","last_unstale":"2024-01-11T08:29:30.967378+0000","last_undegraded":"2024-01-11T08:29:30.967378+0000","last_fullsized":"2024-01-11T08:29:30.967378+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:09:37.262759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.960788+0000","last_change":"2024-01-11T08:29:29.956394+0000","last_active":"2024-01-11T08:29:29.960788+0000","last_peered":"2024-01-11T08:29:29.960788+0000","last_clean":"2024-01-11T08:29:29.960788+0000","last_became_active":"2024-01-11T08:29:29.956090+0000","last_became_peered":"2024-01-11T08:29:29.956090+0000","last_unstale":"2024-01-11T08:29:29.960788+0000","last_undegraded":"2024-01-11T08:29:29.960788+0000","last_fullsized":"2024-01-11T08:29:29.960788+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:53:27.527644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.955525+0000","last_change":"2024-01-11T08:29:30.955525+0000","last_active":"2024-01-11T08:29:30.955525+0000","last_peered":"2024-01-11T08:29:30.955525+0000","last_clean":"2024-01-11T08:29:30.955525+0000","last_became_active":"2024-01-11T08:29:30.955210+0000","last_became_peered":"2024-01-11T08:29:30.955210+0000","last_unstale":"2024-01-11T08:29:30.955525+0000","last_undegraded":"2024-01-11T08:29:30.955525+0000","last_fullsized":"2024-01-11T08:29:30.955525+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:45:57.853427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.950592+0000","last_change":"2024-01-11T08:29:29.945656+0000","last_active":"2024-01-11T08:29:29.950592+0000","last_peered":"2024-01-11T08:29:29.950592+0000","last_clean":"2024-01-11T08:29:29.950592+0000","last_became_active":"2024-01-11T08:29:29.945401+0000","last_became_peered":"2024-01-11T08:29:29.945401+0000","last_unstale":"2024-01-11T08:29:29.950592+0000","last_undegraded":"2024-01-11T08:29:29.950592+0000","last_fullsized":"2024-01-11T08:29:29.950592+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:37:00.082089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.944375+0000","last_change":"2024-01-11T08:29:30.944375+0000","last_active":"2024-01-11T08:29:30.944375+0000","last_peered":"2024-01-11T08:29:30.944375+0000","last_clean":"2024-01-11T08:29:30.944375+0000","last_became_active":"2024-01-11T08:29:30.944079+0000","last_became_peered":"2024-01-11T08:29:30.944079+0000","last_unstale":"2024-01-11T08:29:30.944375+0000","last_undegraded":"2024-01-11T08:29:30.944375+0000","last_fullsized":"2024-01-11T08:29:30.944375+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:01:37.970237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.952362+0000","last_change":"2024-01-11T08:29:29.947304+0000","last_active":"2024-01-11T08:29:29.952362+0000","last_peered":"2024-01-11T08:29:29.952362+0000","last_clean":"2024-01-11T08:29:29.952362+0000","last_became_active":"2024-01-11T08:29:29.946101+0000","last_became_peered":"2024-01-11T08:29:29.946101+0000","last_unstale":"2024-01-11T08:29:29.952362+0000","last_undegraded":"2024-01-11T08:29:29.952362+0000","last_fullsized":"2024-01-11T08:29:29.952362+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:53:16.993120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.951110+0000","last_change":"2024-01-11T08:29:30.951110+0000","last_active":"2024-01-11T08:29:30.951110+0000","last_peered":"2024-01-11T08:29:30.951110+0000","last_clean":"2024-01-11T08:29:30.951110+0000","last_became_active":"2024-01-11T08:29:30.947706+0000","last_became_peered":"2024-01-11T08:29:30.947706+0000","last_unstale":"2024-01-11T08:29:30.951110+0000","last_undegraded":"2024-01-11T08:29:30.951110+0000","last_fullsized":"2024-01-11T08:29:30.951110+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:26:37.958065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.945645+0000","last_change":"2024-01-11T08:29:29.943623+0000","last_active":"2024-01-11T08:29:29.945645+0000","last_peered":"2024-01-11T08:29:29.945645+0000","last_clean":"2024-01-11T08:29:29.945645+0000","last_became_active":"2024-01-11T08:29:29.943311+0000","last_became_peered":"2024-01-11T08:29:29.943311+0000","last_unstale":"2024-01-11T08:29:29.945645+0000","last_undegraded":"2024-01-11T08:29:29.945645+0000","last_fullsized":"2024-01-11T08:29:29.945645+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:34:37.956784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.958786+0000","last_change":"2024-01-11T08:29:30.958786+0000","last_active":"2024-01-11T08:29:30.958786+0000","last_peered":"2024-01-11T08:29:30.958786+0000","last_clean":"2024-01-11T08:29:30.958786+0000","last_became_active":"2024-01-11T08:29:30.958121+0000","last_became_peered":"2024-01-11T08:29:30.958121+0000","last_unstale":"2024-01-11T08:29:30.958786+0000","last_undegraded":"2024-01-11T08:29:30.958786+0000","last_fullsized":"2024-01-11T08:29:30.958786+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:43:31.364333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.955563+0000","last_change":"2024-01-11T08:29:29.951164+0000","last_active":"2024-01-11T08:29:29.955563+0000","last_peered":"2024-01-11T08:29:29.955563+0000","last_clean":"2024-01-11T08:29:29.955563+0000","last_became_active":"2024-01-11T08:29:29.950334+0000","last_became_peered":"2024-01-11T08:29:29.950334+0000","last_unstale":"2024-01-11T08:29:29.955563+0000","last_undegraded":"2024-01-11T08:29:29.955563+0000","last_fullsized":"2024-01-11T08:29:29.955563+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:09:21.938412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.962985+0000","last_change":"2024-01-11T08:29:30.962985+0000","last_active":"2024-01-11T08:29:30.962985+0000","last_peered":"2024-01-11T08:29:30.962985+0000","last_clean":"2024-01-11T08:29:30.962985+0000","last_became_active":"2024-01-11T08:29:30.962708+0000","last_became_peered":"2024-01-11T08:29:30.962708+0000","last_unstale":"2024-01-11T08:29:30.962985+0000","last_undegraded":"2024-01-11T08:29:30.962985+0000","last_fullsized":"2024-01-11T08:29:30.962985+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:47:05.813544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.960866+0000","last_change":"2024-01-11T08:29:29.956171+0000","last_active":"2024-01-11T08:29:29.960866+0000","last_peered":"2024-01-11T08:29:29.960866+0000","last_clean":"2024-01-11T08:29:29.960866+0000","last_became_active":"2024-01-11T08:29:29.955604+0000","last_became_peered":"2024-01-11T08:29:29.955604+0000","last_unstale":"2024-01-11T08:29:29.960866+0000","last_undegraded":"2024-01-11T08:29:29.960866+0000","last_fullsized":"2024-01-11T08:29:29.960866+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:09:44.922430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.947798+0000","last_change":"2024-01-11T08:29:30.947798+0000","last_active":"2024-01-11T08:29:30.947798+0000","last_peered":"2024-01-11T08:29:30.947798+0000","last_clean":"2024-01-11T08:29:30.947798+0000","last_became_active":"2024-01-11T08:29:30.947070+0000","last_became_peered":"2024-01-11T08:29:30.947070+0000","last_unstale":"2024-01-11T08:29:30.947798+0000","last_undegraded":"2024-01-11T08:29:30.947798+0000","last_fullsized":"2024-01-11T08:29:30.947798+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:15:59.730225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.957738+0000","last_change":"2024-01-11T08:29:29.954394+0000","last_active":"2024-01-11T08:29:29.957738+0000","last_peered":"2024-01-11T08:29:29.957738+0000","last_clean":"2024-01-11T08:29:29.957738+0000","last_became_active":"2024-01-11T08:29:29.954130+0000","last_became_peered":"2024-01-11T08:29:29.954130+0000","last_unstale":"2024-01-11T08:29:29.957738+0000","last_undegraded":"2024-01-11T08:29:29.957738+0000","last_fullsized":"2024-01-11T08:29:29.957738+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:27.676667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"121'245","reported_seq":704,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.943265+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:29:29.943265+0000","last_peered":"2024-01-11T08:29:29.943265+0000","last_clean":"2024-01-11T08:29:29.943265+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:29:29.943265+0000","last_undegraded":"2024-01-11T08:29:29.943265+0000","last_fullsized":"2024-01-11T08:29:29.943265+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1232,"num_read_kb":2542,"num_write":391,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1232,"num_read_kb":2542,"num_write":391,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":245,"ondisk_log_size":245,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575072,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303528,"kb_used_data":2144,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068312,"statfs":{"total":96636764160,"available":96325951488,"internally_reserved":0,"allocated":2195456,"data_stored":1337124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67900000000000005,"15min":0.69399999999999995},"min":{"1min":0.379,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.84699999999999998,"5min":0.95899999999999996,"15min":0.996},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.66500000000000004,"15min":0.70099999999999996},"min":{"1min":0.44900000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86099999999999999,"5min":1.073,"15min":1.073},"last":0.755}]},{"osd":1,"last update":"Thu Jan 11 08:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.68000000000000005,"15min":0.69899999999999995},"min":{"1min":0.504,"5min":0.432,"15min":0.41899999999999998},"max":{"1min":0.88300000000000001,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.68899999999999995,"15min":0.71099999999999997},"min":{"1min":0.42499999999999999,"5min":0.41999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90400000000000003,"5min":0.94999999999999996,"15min":1.014},"last":0.81200000000000006}]},{"osd":2,"last update":"Thu Jan 11 08:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.69599999999999995,"15min":0.72099999999999997},"min":{"1min":0.47099999999999997,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89500000000000002,"5min":0.99299999999999999,"15min":1.034},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.68500000000000005,"15min":0.70899999999999996},"min":{"1min":0.47299999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.76500000000000001}]},{"osd":3,"last update":"Thu Jan 11 08:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.69599999999999995,"15min":0.72899999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.89400000000000002,"5min":0.91700000000000004,"15min":1.032},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71499999999999997,"15min":0.73399999999999999},"min":{"1min":0.52000000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.92500000000000004,"5min":1.008,"15min":1.008},"last":0.84799999999999998}]},{"osd":4,"last update":"Thu Jan 11 08:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59299999999999997,"15min":0.58799999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":0.89700000000000002,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64500000000000002,"15min":0.629},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90100000000000002,"5min":1.002,"15min":1.002},"last":0.93300000000000005}]},{"osd":5,"last update":"Thu Jan 11 08:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.65900000000000003,"15min":0.64100000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.34599999999999997},"max":{"1min":0.97199999999999998,"5min":1.042,"15min":1.042},"last":0.621},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.66800000000000004,"15min":0.66700000000000004},"min":{"1min":0.433,"5min":0.41399999999999998,"15min":0.40400000000000003},"max":{"1min":0.99199999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.67300000000000004}]},{"osd":6,"last update":"Thu Jan 11 08:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.747,"15min":0.75},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.94899999999999995,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.75,"5min":0.71999999999999997,"15min":0.72899999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.96599999999999997,"5min":1.028,"15min":1.028},"last":0.80000000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575071,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302948,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068892,"statfs":{"total":96636764160,"available":96326545408,"internally_reserved":0,"allocated":1601536,"data_stored":746756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.63700000000000001,"15min":0.66400000000000003},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.94799999999999995},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.63,"5min":0.63700000000000001,"15min":0.68100000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.33800000000000002},"max":{"1min":0.82999999999999996,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.78400000000000003}]},{"osd":1,"last update":"Thu Jan 11 08:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.66300000000000003,"15min":0.69299999999999995},"min":{"1min":0.40100000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.93200000000000005,"5min":1.008,"15min":1.163},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.65400000000000003,"15min":0.68500000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88900000000000001,"5min":0.92600000000000005,"15min":1.1950000000000001},"last":0.82699999999999996}]},{"osd":2,"last update":"Thu Jan 11 08:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66900000000000004,"15min":0.70299999999999996},"min":{"1min":0.442,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.93400000000000005,"5min":0.95399999999999996,"15min":1.016},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.67800000000000005,"15min":0.71699999999999997},"min":{"1min":0.42399999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.90000000000000002,"5min":0.91100000000000003,"15min":0.97399999999999998},"last":0.69299999999999995}]},{"osd":3,"last update":"Thu Jan 11 08:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.69999999999999996,"15min":0.72699999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":1.0149999999999999},"last":0.749},{"interface":"front","average":{"1min":0.752,"5min":0.70699999999999996,"15min":0.73699999999999999},"min":{"1min":0.46999999999999997,"5min":0.46000000000000002,"15min":0.41299999999999998},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":1.0049999999999999},"last":0.93700000000000006}]},{"osd":4,"last update":"Thu Jan 11 08:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.57599999999999996,"15min":0.56599999999999995},"min":{"1min":0.45600000000000002,"5min":0.36699999999999999,"15min":0.30199999999999999},"max":{"1min":0.871,"5min":0.873,"15min":0.92000000000000004},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.48699999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.96399999999999997},"last":0.90000000000000002}]},{"osd":5,"last update":"Thu Jan 11 08:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.54700000000000004,"5min":0.375,"15min":0.375},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.751,"5min":0.70099999999999996,"15min":0.70199999999999996},"min":{"1min":0.53500000000000003,"5min":0.41799999999999998,"15min":0.35199999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.96099999999999997}]},{"osd":7,"last update":"Thu Jan 11 08:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.69999999999999996,"15min":0.70899999999999996},"min":{"1min":0.505,"5min":0.44700000000000001,"15min":0.42399999999999999},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.73999999999999999,"15min":0.754},"min":{"1min":0.61799999999999999,"5min":0.46700000000000003,"15min":0.44700000000000001},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":1.1140000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575073,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302948,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068892,"statfs":{"total":96636764160,"available":96326545408,"internally_reserved":0,"allocated":1601536,"data_stored":746756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:28:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47999999999999998,"15min":0.48599999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.85799999999999998,"5min":0.85999999999999999,"15min":0.97699999999999998},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54900000000000004,"15min":0.57499999999999996},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.91700000000000004,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.56499999999999995}]},{"osd":2,"last update":"Thu Jan 11 08:28:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45400000000000001,"15min":0.47899999999999998},"min":{"1min":0.34100000000000003,"5min":0.32300000000000001,"15min":0.312},"max":{"1min":0.70299999999999996,"5min":1.0960000000000001,"15min":1.113},"last":0.441},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54800000000000004,"15min":0.56000000000000005},"min":{"1min":0.35899999999999999,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.90100000000000002},"last":0.66200000000000003}]},{"osd":3,"last update":"Thu Jan 11 08:28:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.59099999999999997,"15min":0.61099999999999999},"min":{"1min":0.41599999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.68300000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55100000000000005,"15min":0.56000000000000005},"min":{"1min":0.34599999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.83799999999999997,"5min":0.88100000000000001,"15min":0.94899999999999995},"last":0.41899999999999998}]},{"osd":4,"last update":"Thu Jan 11 08:28:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.78500000000000003,"15min":0.79000000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.439},"max":{"1min":1.002,"5min":1.129,"15min":1.2170000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.82999999999999996,"15min":0.82899999999999996},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.501},"max":{"1min":1.085,"5min":1.109,"15min":1.246},"last":0.82899999999999996}]},{"osd":5,"last update":"Thu Jan 11 08:28:51 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.77800000000000002,"15min":0.79700000000000004},"min":{"1min":0.53400000000000003,"5min":0.498,"15min":0.498},"max":{"1min":1.028,"5min":1.1299999999999999,"15min":1.218},"last":0.754},{"interface":"front","average":{"1min":0.748,"5min":0.76100000000000001,"15min":0.78400000000000003},"min":{"1min":0.61099999999999999,"5min":0.52900000000000003,"15min":0.51200000000000001},"max":{"1min":0.93500000000000005,"5min":1.1339999999999999,"15min":1.1779999999999999},"last":0.63500000000000001}]},{"osd":6,"last update":"Thu Jan 11 08:28:51 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.79900000000000004,"15min":0.81599999999999995},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.51300000000000001},"max":{"1min":1.0009999999999999,"5min":1.115,"15min":1.1439999999999999},"last":0.875},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80200000000000005,"15min":0.82499999999999996},"min":{"1min":0.54700000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.034,"5min":1.091,"15min":1.228},"last":0.79700000000000004}]},{"osd":7,"last update":"Thu Jan 11 08:28:51 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.81999999999999995,"15min":0.84599999999999997},"min":{"1min":0.58299999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.83199999999999996,"15min":0.84099999999999997},"min":{"1min":0.68400000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1,"5min":1.099,"15min":1.258},"last":0.77200000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575072,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303528,"kb_used_data":2144,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068312,"statfs":{"total":96636764160,"available":96325951488,"internally_reserved":0,"allocated":2195456,"data_stored":1337124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.497,"15min":0.499},"min":{"1min":0.29199999999999998,"5min":0.28699999999999998,"15min":0.24299999999999999},"max":{"1min":0.61499999999999999,"5min":0.84399999999999997,"15min":0.92700000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.36799999999999999},"max":{"1min":0.66500000000000004,"5min":0.90500000000000003,"15min":0.92100000000000004},"last":0.44900000000000001}]},{"osd":2,"last update":"Thu Jan 11 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.55400000000000005,"15min":0.54700000000000004},"min":{"1min":0.41799999999999998,"5min":0.33000000000000002,"15min":0.29099999999999998},"max":{"1min":0.74299999999999999,"5min":1.335,"15min":1.335},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55800000000000005,"15min":0.55600000000000005},"min":{"1min":0.39100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68799999999999994,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.51000000000000001}]},{"osd":3,"last update":"Thu Jan 11 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.375,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.76600000000000001,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.627,"15min":0.626},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.745,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.72899999999999998}]},{"osd":4,"last update":"Thu Jan 11 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.57799999999999996,"5min":0.502,"15min":0.437},"max":{"1min":0.93400000000000005,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.623},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.77400000000000002,"15min":0.78900000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.98399999999999999,"5min":1.1220000000000001,"15min":1.149},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.78500000000000003,"15min":0.80300000000000005},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.52500000000000002},"max":{"1min":0.90900000000000003,"5min":1.518,"15min":1.518},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.81200000000000006,"15min":0.81899999999999995},"min":{"1min":0.61199999999999999,"5min":0.55400000000000005,"15min":0.47799999999999998},"max":{"1min":0.96699999999999997,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.82099999999999995}]},{"osd":6,"last update":"Thu Jan 11 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.80000000000000004,"15min":0.80600000000000005},"min":{"1min":0.54900000000000004,"5min":0.51700000000000002,"15min":0.50900000000000001},"max":{"1min":0.997,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.81699999999999995,"15min":0.83099999999999996},"min":{"1min":0.55000000000000004,"5min":0.52000000000000002,"15min":0.44900000000000001},"max":{"1min":0.97899999999999998,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.97299999999999998}]},{"osd":7,"last update":"Thu Jan 11 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.83299999999999996,"15min":0.84299999999999997},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.51400000000000001},"max":{"1min":0.90700000000000003,"5min":1.371,"15min":1.371},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.81599999999999995,"15min":0.83999999999999997},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.93600000000000005,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.68500000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575071,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311140,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060700,"statfs":{"total":96636764160,"available":96318156800,"internally_reserved":0,"allocated":1601536,"data_stored":746756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46400000000000002,"15min":0.46899999999999997},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.26000000000000001},"max":{"1min":0.64900000000000002,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.503,"15min":0.53600000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68700000000000006,"5min":0.81699999999999995,"15min":0.91400000000000003},"last":0.501}]},{"osd":1,"last update":"Thu Jan 11 08:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53600000000000003,"15min":0.54600000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70699999999999996,"5min":0.77200000000000002,"15min":0.90000000000000002},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55600000000000005,"15min":0.57699999999999996},"min":{"1min":0.40600000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.74399999999999999,"5min":0.79600000000000004,"15min":0.94499999999999995},"last":0.46500000000000002}]},{"osd":3,"last update":"Thu Jan 11 08:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51100000000000001,"15min":0.53800000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.74199999999999999,"5min":0.76500000000000001,"15min":0.91600000000000004},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59599999999999997,"15min":0.625},"min":{"1min":0.40000000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.74199999999999999,"5min":0.81799999999999995,"15min":1.0009999999999999},"last":0.52100000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.745,"15min":0.77900000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.109,"5min":1.109,"15min":1.2889999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.73799999999999999,"15min":0.77500000000000002},"min":{"1min":0.5,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.056,"5min":1.0760000000000001,"15min":1.3089999999999999},"last":0.68999999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.80300000000000005,"15min":0.82599999999999996},"min":{"1min":0.55900000000000005,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.091,"5min":1.2470000000000001,"15min":1.3540000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.80800000000000005,"15min":0.83499999999999996},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.147,"5min":1.2,"15min":1.23},"last":0.67400000000000004}]},{"osd":6,"last update":"Thu Jan 11 08:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.75,"15min":0.78900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.94999999999999996,"5min":1.131,"15min":1.2669999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78700000000000003,"15min":0.81699999999999995},"min":{"1min":0.55000000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.006,"5min":1.351,"15min":1.47},"last":0.60099999999999998}]},{"osd":7,"last update":"Thu Jan 11 08:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.81699999999999995,"15min":0.85299999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.51400000000000001},"max":{"1min":1.0029999999999999,"5min":1.2629999999999999,"15min":1.4099999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79700000000000004,"15min":0.84899999999999998},"min":{"1min":0.57199999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.0209999999999999,"5min":1.2270000000000001,"15min":1.327},"last":0.73799999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575071,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307044,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064796,"statfs":{"total":96636764160,"available":96322351104,"internally_reserved":0,"allocated":1601536,"data_stored":746756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49299999999999999,"15min":0.48599999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.27600000000000002},"max":{"1min":0.63600000000000001,"5min":0.80400000000000005,"15min":1.1930000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.52300000000000002,"15min":0.54300000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.33400000000000002},"max":{"1min":0.80700000000000005,"5min":0.873,"15min":0.873},"last":0.42199999999999999}]},{"osd":1,"last update":"Thu Jan 11 08:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47499999999999998,"15min":0.48599999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.25900000000000001},"max":{"1min":0.71699999999999997,"5min":1.012,"15min":1.012},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.56499999999999995,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.36299999999999999},"max":{"1min":0.78400000000000003,"5min":1.0960000000000001,"15min":1.1499999999999999},"last":0.46899999999999997}]},{"osd":2,"last update":"Thu Jan 11 08:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56699999999999995,"15min":0.57899999999999996},"min":{"1min":0.41999999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.97599999999999998},"last":0.378},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60499999999999998,"15min":0.626},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.32000000000000001},"max":{"1min":0.74299999999999999,"5min":0.89700000000000002,"15min":1.2270000000000001},"last":0.51300000000000001}]},{"osd":4,"last update":"Thu Jan 11 08:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.72399999999999998,"15min":0.73699999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.35999999999999999},"max":{"1min":2.2709999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.753},{"interface":"front","average":{"1min":0.748,"5min":0.76200000000000001,"15min":0.77500000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.503},"max":{"1min":0.97499999999999998,"5min":1.1459999999999999,"15min":1.1819999999999999},"last":0.64600000000000002}]},{"osd":5,"last update":"Thu Jan 11 08:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.76800000000000002,"15min":0.78100000000000003},"min":{"1min":0.55700000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.94899999999999995,"5min":1.0649999999999999,"15min":1.3260000000000001},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.79200000000000004,"15min":0.80400000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.49199999999999999},"max":{"1min":1.0329999999999999,"5min":1.24,"15min":1.4239999999999999},"last":0.88400000000000001}]},{"osd":6,"last update":"Thu Jan 11 08:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.79600000000000004,"15min":0.82699999999999996},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.101,"5min":1.101,"15min":1.335},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":2.3599999999999999,"5min":2.3599999999999999,"15min":2.3599999999999999},"last":0.82599999999999996}]},{"osd":7,"last update":"Thu Jan 11 08:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.81999999999999995,"15min":0.83199999999999996},"min":{"1min":0.54700000000000004,"5min":0.53100000000000003,"15min":0.46600000000000003},"max":{"1min":2.3180000000000001,"5min":2.3180000000000001,"15min":2.3180000000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.84599999999999997,"15min":0.85699999999999998},"min":{"1min":0.61499999999999999,"5min":0.59199999999999997,"15min":0.58399999999999996},"max":{"1min":1.0449999999999999,"5min":1.2050000000000001,"15min":1.21},"last":0.83799999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575072,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311140,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060700,"statfs":{"total":96636764160,"available":96318156800,"internally_reserved":0,"allocated":1601536,"data_stored":746756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69299999999999995,"15min":0.73699999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.28100000000000003},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.117},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70399999999999996,"15min":0.73599999999999999},"min":{"1min":0.51000000000000001,"5min":0.41599999999999998,"15min":0.38600000000000001},"max":{"1min":0.95699999999999996,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.48299999999999998}]},{"osd":1,"last update":"Thu Jan 11 08:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.66900000000000004,"15min":0.69599999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.33300000000000002},"max":{"1min":0.91000000000000003,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.67800000000000005,"15min":0.70799999999999996},"min":{"1min":0.49399999999999999,"5min":0.43099999999999999,"15min":0.36299999999999999},"max":{"1min":0.89100000000000001,"5min":0.99199999999999999,"15min":1.046},"last":0.84899999999999998}]},{"osd":2,"last update":"Thu Jan 11 08:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70999999999999996,"15min":0.71999999999999997},"min":{"1min":0.50700000000000001,"5min":0.46600000000000003,"15min":0.38900000000000001},"max":{"1min":0.92200000000000004,"5min":1.03,"15min":1.03},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.68600000000000005,"15min":0.72399999999999998},"min":{"1min":0.49299999999999999,"5min":0.432,"15min":0.432},"max":{"1min":0.84799999999999998,"5min":0.97699999999999998,"15min":1.0409999999999999},"last":0.751}]},{"osd":3,"last update":"Thu Jan 11 08:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.66800000000000004,"15min":0.69999999999999996},"min":{"1min":0.46300000000000002,"5min":0.46100000000000002,"15min":0.433},"max":{"1min":0.92500000000000004,"5min":0.96799999999999997,"15min":1.0189999999999999},"last":0.623},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.72499999999999998,"15min":0.745},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.97499999999999998,"5min":1.026,"15min":1.026},"last":0.72299999999999998}]},{"osd":5,"last update":"Thu Jan 11 08:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.60999999999999999,"15min":0.61199999999999999},"min":{"1min":0.43099999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.95699999999999996,"5min":0.96599999999999997,"15min":1.0700000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.624,"15min":0.628},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.32600000000000001},"max":{"1min":0.874,"5min":0.94299999999999995,"15min":0.97799999999999998},"last":0.77500000000000002}]},{"osd":6,"last update":"Thu Jan 11 08:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.42099999999999999,"5min":0.40500000000000003,"15min":0.39500000000000002},"max":{"1min":0.94199999999999995,"5min":1.0680000000000001,"15min":1.133},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.69399999999999995,"15min":0.68899999999999995},"min":{"1min":0.46300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.119},"last":0.89300000000000002}]},{"osd":7,"last update":"Thu Jan 11 08:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69299999999999995,"15min":0.68899999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.41299999999999998},"max":{"1min":0.91100000000000003,"5min":1.0129999999999999,"15min":1.0640000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.52400000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.999,"5min":1.127,"15min":1.2350000000000001},"last":0.80800000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575072,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307044,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064796,"statfs":{"total":96636764160,"available":96322351104,"internally_reserved":0,"allocated":1601536,"data_stored":746756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.67100000000000004,"15min":0.70299999999999996},"min":{"1min":0.41999999999999998,"5min":0.39300000000000002,"15min":0.38700000000000001},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.872},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.68300000000000005,"15min":0.70799999999999996},"min":{"1min":0.47999999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88,"5min":0.96199999999999997,"15min":1.696},"last":0.84199999999999997}]},{"osd":1,"last update":"Thu Jan 11 08:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.65100000000000002,"15min":0.69199999999999995},"min":{"1min":0.45500000000000002,"5min":0.39100000000000001,"15min":0.36399999999999999},"max":{"1min":0.88500000000000001,"5min":0.997,"15min":1.4359999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.68100000000000005,"15min":0.72499999999999998},"min":{"1min":0.42999999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.83199999999999996,"5min":1.05,"15min":4.4969999999999999},"last":0.91100000000000003}]},{"osd":2,"last update":"Thu Jan 11 08:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.68100000000000005,"15min":0.70599999999999996},"min":{"1min":0.46600000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81599999999999995,"5min":0.97299999999999998,"15min":1.238},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.68500000000000005,"15min":0.73899999999999999},"min":{"1min":0.49299999999999999,"5min":0.45300000000000001,"15min":0.41599999999999998},"max":{"1min":0.84399999999999997,"5min":0.95899999999999996,"15min":8.0429999999999993},"last":0.68500000000000005}]},{"osd":3,"last update":"Thu Jan 11 08:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.69799999999999995,"15min":0.745},"min":{"1min":0.44900000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0920000000000001,"5min":1.208,"15min":8.0839999999999996},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.73099999999999998,"15min":0.74099999999999999},"min":{"1min":0.61599999999999999,"5min":0.439,"15min":0.40899999999999997},"max":{"1min":0.83499999999999996,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.77100000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.58599999999999997,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.38300000000000001,"15min":0.34300000000000003},"max":{"1min":0.90400000000000003,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64000000000000001,"15min":0.63200000000000001},"min":{"1min":0.439,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85599999999999998,"5min":1.256,"15min":1.256},"last":0.64500000000000002}]},{"osd":6,"last update":"Thu Jan 11 08:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.64900000000000002,"15min":0.64700000000000002},"min":{"1min":0.41899999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.88,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.66700000000000004,"15min":0.68000000000000005},"min":{"1min":0.48699999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.93700000000000006,"5min":1.085,"15min":1.085},"last":0.624}]},{"osd":7,"last update":"Thu Jan 11 08:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.70699999999999996,"15min":0.72599999999999998},"min":{"1min":0.57499999999999996,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.95499999999999996,"5min":1.341,"15min":1.341},"last":0.746},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.76400000000000001,"15min":0.77000000000000002},"min":{"1min":0.57599999999999996,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.98799999999999999,"5min":1.119,"15min":1.119},"last":0.70299999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:29:35.356 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:29:35.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.590+0000 7f731a5ff700 1 Processor -- start 2024-01-11T08:29:35.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.591+0000 7f731a5ff700 1 -- start start 2024-01-11T08:29:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.592+0000 7f731a5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73141575e0 0x7f7314158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.592+0000 7f731a5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f731406b610 0x7f73140b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.592+0000 7f731a5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7314158b60 con 0x7f73140b1800 2024-01-11T08:29:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.592+0000 7f731a5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73140cb5e0 con 0x7f731406b610 2024-01-11T08:29:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.592+0000 7f731a5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73140b1c80 con 0x7f73141575e0 2024-01-11T08:29:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.592+0000 7f7318b9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f73140b1800 0x7f73140ca1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54800/0 (socket says 172.21.15.27:54800) 2024-01-11T08:29:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.592+0000 7f7318b9c700 1 -- 172.21.15.27:0/644847888 learned_addr learned my addr 172.21.15.27:0/644847888 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.592+0000 7f73137fe700 1 --2- 172.21.15.27:0/644847888 >> v2:172.21.15.77:3300/0 conn(0x7f731406b610 0x7f73140b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.592+0000 7f7313fff700 1 --2- 172.21.15.27:0/644847888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73141575e0 0x7f7314158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.593+0000 7f7312ffd700 1 -- 172.21.15.27:0/644847888 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994172904 0 0) 0x7f730c001490 con 0x7f73140b1800 2024-01-11T08:29:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.593+0000 7f7312ffd700 1 -- 172.21.15.27:0/644847888 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72fc003570 con 0x7f73140b1800 2024-01-11T08:29:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.593+0000 7f7312ffd700 1 -- 172.21.15.27:0/644847888 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 814424445 0 0) 0x7f72fc003570 con 0x7f73140b1800 2024-01-11T08:29:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.593+0000 7f7312ffd700 1 -- 172.21.15.27:0/644847888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73141575e0 msgr2=0x7f7314158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.593+0000 7f7312ffd700 1 --2- 172.21.15.27:0/644847888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73141575e0 0x7f7314158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.593+0000 7f7312ffd700 1 -- 172.21.15.27:0/644847888 >> v2:172.21.15.77:3300/0 conn(0x7f731406b610 msgr2=0x7f73140b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.593+0000 7f7312ffd700 1 --2- 172.21.15.27:0/644847888 >> v2:172.21.15.77:3300/0 conn(0x7f731406b610 0x7f73140b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.593+0000 7f7312ffd700 1 -- 172.21.15.27:0/644847888 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73141599b0 con 0x7f73140b1800 2024-01-11T08:29:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.593+0000 7f7312ffd700 1 -- 172.21.15.27:0/644847888 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f730c001980 con 0x7f73140b1800 2024-01-11T08:29:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.593+0000 7f7312ffd700 1 -- 172.21.15.27:0/644847888 >> v1:172.21.15.27:6789/0 conn(0x7f73140b1800 legacy=0x7f73140ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.593+0000 7f7312ffd700 1 --2- 172.21.15.27:0/644847888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72fc003ae0 0x7f72fc003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.593+0000 7f7312ffd700 1 --2- 172.21.15.27:0/644847888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72fc004810 0x7f72fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:35.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.593+0000 7f7312ffd700 1 --2- 172.21.15.27:0/644847888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73141575e0 0x7f72fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:35.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.593+0000 7f7312ffd700 1 -- 172.21.15.27:0/644847888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73140cb5e0 con 0x7f72fc003ae0 2024-01-11T08:29:35.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.593+0000 7f7312ffd700 1 -- 172.21.15.27:0/644847888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73140b1c80 con 0x7f72fc004810 2024-01-11T08:29:35.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.593+0000 7f7312ffd700 1 -- 172.21.15.27:0/644847888 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc00bb10 con 0x7f73141575e0 2024-01-11T08:29:35.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.593+0000 7f7313fff700 1 --2- 172.21.15.27:0/644847888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72fc003ae0 0x7f72fc003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:35.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.594+0000 7f73137fe700 1 --2- 172.21.15.27:0/644847888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72fc004810 0x7f72fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:35.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.594+0000 7f7318b9c700 1 --2- 172.21.15.27:0/644847888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73141575e0 0x7f72fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:35.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.594+0000 7f7313fff700 1 -- 172.21.15.27:0/644847888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73141575e0 msgr2=0x7f72fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:35.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.594+0000 7f7313fff700 1 --2- 172.21.15.27:0/644847888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73141575e0 0x7f72fc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.594+0000 7f7313fff700 1 -- 172.21.15.27:0/644847888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72fc004810 msgr2=0x7f72fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:35.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.594+0000 7f7313fff700 1 --2- 172.21.15.27:0/644847888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72fc004810 0x7f72fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.594+0000 7f7313fff700 1 -- 172.21.15.27:0/644847888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc003570 con 0x7f72fc003ae0 2024-01-11T08:29:35.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.594+0000 7f7318b9c700 1 --2- 172.21.15.27:0/644847888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73141575e0 0x7f72fc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:35.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.594+0000 7f7313fff700 1 --2- 172.21.15.27:0/644847888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72fc003ae0 0x7f72fc003ef0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f7304009c00 tx=0x7f73040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=717ccbbb2a5551b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:35.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.594+0000 7f7312ffd700 1 -- 172.21.15.27:0/644847888 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730400e710 con 0x7f72fc003ae0 2024-01-11T08:29:35.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.594+0000 7f7312ffd700 1 -- 172.21.15.27:0/644847888 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730400e870 con 0x7f72fc003ae0 2024-01-11T08:29:35.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.594+0000 7f7312ffd700 1 -- 172.21.15.27:0/644847888 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730400ec30 con 0x7f72fc003ae0 2024-01-11T08:29:35.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.595+0000 7f731a5ff700 1 -- 172.21.15.27:0/644847888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72fc003ae0 msgr2=0x7f72fc003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:35.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.595+0000 7f731a5ff700 1 --2- 172.21.15.27:0/644847888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72fc003ae0 0x7f72fc003ef0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f7304009c00 tx=0x7f73040091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.595+0000 7f7318b9c700 1 -- 172.21.15.27:0/644847888 reap_dead start 2024-01-11T08:29:35.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.595+0000 7f731a5ff700 1 -- 172.21.15.27:0/644847888 shutdown_connections 2024-01-11T08:29:35.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.595+0000 7f731a5ff700 1 -- 172.21.15.27:0/644847888 >> 172.21.15.27:0/644847888 conn(0x7f73140680e0 msgr2=0x7f7314069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:35.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.595+0000 7f731a5ff700 1 -- 172.21.15.27:0/644847888 shutdown_connections 2024-01-11T08:29:35.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.595+0000 7f731a5ff700 1 -- 172.21.15.27:0/644847888 wait complete. 2024-01-11T08:29:35.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.596+0000 7f731a5ff700 1 Processor -- start 2024-01-11T08:29:35.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.597+0000 7f731a5ff700 1 -- start start 2024-01-11T08:29:35.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.597+0000 7f731a5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72fc003ae0 0x7f73140c1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:35.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.598+0000 7f731a5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73140c1c40 0x7f73141654b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:35.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.598+0000 7f731a5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73141659f0 0x7f7314165e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:35.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.598+0000 7f731a5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73141663d0 con 0x7f73141659f0 2024-01-11T08:29:35.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.598+0000 7f731a5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73140c2470 con 0x7f72fc003ae0 2024-01-11T08:29:35.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.598+0000 7f731a5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73140c2750 con 0x7f73140c1c40 2024-01-11T08:29:35.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.598+0000 7f7318b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73141659f0 0x7f7314165e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:35.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.598+0000 7f7318b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73141659f0 0x7f7314165e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36784/0 (socket says 172.21.15.27:36784) 2024-01-11T08:29:35.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.598+0000 7f7318b9c700 1 -- 172.21.15.27:0/909290499 learned_addr learned my addr 172.21.15.27:0/909290499 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:35.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.598+0000 7f7313fff700 1 --2- 172.21.15.27:0/909290499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72fc003ae0 0x7f73140c1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:35.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.598+0000 7f73137fe700 1 --2- 172.21.15.27:0/909290499 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73140c1c40 0x7f73141654b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.598+0000 7f7318b9c700 1 -- 172.21.15.27:0/909290499 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73140c1c40 msgr2=0x7f73141654b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.598+0000 7f7318b9c700 1 --2- 172.21.15.27:0/909290499 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73140c1c40 0x7f73141654b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.598+0000 7f7318b9c700 1 -- 172.21.15.27:0/909290499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72fc003ae0 msgr2=0x7f73140c1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.598+0000 7f7318b9c700 1 --2- 172.21.15.27:0/909290499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72fc003ae0 0x7f73140c1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.598+0000 7f7318b9c700 1 -- 172.21.15.27:0/909290499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73140c2a30 con 0x7f73141659f0 2024-01-11T08:29:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.599+0000 7f7318b9c700 1 --2- 172.21.15.27:0/909290499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73141659f0 0x7f7314165e90 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f730c001350 tx=0x7f730c001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.599+0000 7f72fbfff700 1 -- 172.21.15.27:0/909290499 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c007b30 con 0x7f73141659f0 2024-01-11T08:29:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.599+0000 7f72fbfff700 1 -- 172.21.15.27:0/909290499 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730c00eb10 con 0x7f73141659f0 2024-01-11T08:29:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.599+0000 7f731a5ff700 1 -- 172.21.15.27:0/909290499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73140c2d10 con 0x7f73141659f0 2024-01-11T08:29:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.599+0000 7f72fbfff700 1 -- 172.21.15.27:0/909290499 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c00eed0 con 0x7f73141659f0 2024-01-11T08:29:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.599+0000 7f731a5ff700 1 -- 172.21.15.27:0/909290499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7314151e10 con 0x7f73141659f0 2024-01-11T08:29:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.600+0000 7f731a5ff700 1 -- 172.21.15.27:0/909290499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7314158da0 con 0x7f73141659f0 2024-01-11T08:29:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.601+0000 7f72fbfff700 1 -- 172.21.15.27:0/909290499 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f730c013630 con 0x7f73141659f0 2024-01-11T08:29:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.602+0000 7f72fbfff700 1 --2- 172.21.15.27:0/909290499 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7300066f20 0x7f73000693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:35.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.602+0000 7f72fbfff700 1 -- 172.21.15.27:0/909290499 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f730c0848e0 con 0x7f73141659f0 2024-01-11T08:29:35.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.602+0000 7f7313fff700 1 --2- 172.21.15.27:0/909290499 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7300066f20 0x7f73000693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:35.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.603+0000 7f7313fff700 1 --2- 172.21.15.27:0/909290499 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7300066f20 0x7f73000693d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f73040091e0 tx=0x7f730401a000 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:35.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.606+0000 7f72fbfff700 1 -- 172.21.15.27:0/909290499 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f730c051a70 con 0x7f73141659f0 2024-01-11T08:29:35.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.804+0000 7f731a5ff700 1 -- 172.21.15.27:0/909290499 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7314158b60 con 0x7f7300066f20 2024-01-11T08:29:35.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.814+0000 7f72fbfff700 1 -- 172.21.15.27:0/909290499 <== mgr.7591 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289559 (secure 0 0 0) 0x7f7314158b60 con 0x7f7300066f20 2024-01-11T08:29:35.819 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:35.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.818+0000 7f731a5ff700 1 -- 172.21.15.27:0/909290499 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7300066f20 msgr2=0x7f73000693d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:35.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.818+0000 7f731a5ff700 1 --2- 172.21.15.27:0/909290499 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7300066f20 0x7f73000693d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f73040091e0 tx=0x7f730401a000 comp rx=0 tx=0).stop 2024-01-11T08:29:35.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.818+0000 7f731a5ff700 1 -- 172.21.15.27:0/909290499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73141659f0 msgr2=0x7f7314165e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:35.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.818+0000 7f731a5ff700 1 --2- 172.21.15.27:0/909290499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73141659f0 0x7f7314165e90 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f730c001350 tx=0x7f730c001970 comp rx=0 tx=0).stop 2024-01-11T08:29:35.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.818+0000 7f731a5ff700 1 -- 172.21.15.27:0/909290499 shutdown_connections 2024-01-11T08:29:35.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.818+0000 7f731a5ff700 1 --2- 172.21.15.27:0/909290499 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7300066f20 0x7f73000693d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.818+0000 7f731a5ff700 1 --2- 172.21.15.27:0/909290499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73141659f0 0x7f7314165e90 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.818+0000 7f731a5ff700 1 --2- 172.21.15.27:0/909290499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72fc003ae0 0x7f73140c1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.818+0000 7f731a5ff700 1 --2- 172.21.15.27:0/909290499 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73140c1c40 0x7f73141654b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:35.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.818+0000 7f731a5ff700 1 -- 172.21.15.27:0/909290499 >> 172.21.15.27:0/909290499 conn(0x7f73140680e0 msgr2=0x7f73140cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:35.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.818+0000 7f731a5ff700 1 -- 172.21.15.27:0/909290499 shutdown_connections 2024-01-11T08:29:35.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:35.818+0000 7f731a5ff700 1 -- 172.21.15.27:0/909290499 wait complete. 2024-01-11T08:29:35.824 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:29:35.835 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-01-11T08:29:34.057894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1232,"num_read_kb":2542,"num_write":435,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":283,"ondisk_log_size":283,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2449320,"kb_used_data":13672,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752525400,"statfs":{"total":773094113280,"available":770586009600,"internally_reserved":0,"allocated":14000128,"data_stored":7154784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.132190"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.963055+0000","last_change":"2024-01-11T08:29:30.963055+0000","last_active":"2024-01-11T08:29:30.963055+0000","last_peered":"2024-01-11T08:29:30.963055+0000","last_clean":"2024-01-11T08:29:30.963055+0000","last_became_active":"2024-01-11T08:29:30.961165+0000","last_became_peered":"2024-01-11T08:29:30.961165+0000","last_unstale":"2024-01-11T08:29:30.963055+0000","last_undegraded":"2024-01-11T08:29:30.963055+0000","last_fullsized":"2024-01-11T08:29:30.963055+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:47:37.059908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.952345+0000","last_change":"2024-01-11T08:29:30.952345+0000","last_active":"2024-01-11T08:29:30.952345+0000","last_peered":"2024-01-11T08:29:30.952345+0000","last_clean":"2024-01-11T08:29:30.952345+0000","last_became_active":"2024-01-11T08:29:30.952094+0000","last_became_peered":"2024-01-11T08:29:30.952094+0000","last_unstale":"2024-01-11T08:29:30.952345+0000","last_undegraded":"2024-01-11T08:29:30.952345+0000","last_fullsized":"2024-01-11T08:29:30.952345+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:11:42.267734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.955817+0000","last_change":"2024-01-11T08:29:30.955817+0000","last_active":"2024-01-11T08:29:30.955817+0000","last_peered":"2024-01-11T08:29:30.955817+0000","last_clean":"2024-01-11T08:29:30.955817+0000","last_became_active":"2024-01-11T08:29:30.954978+0000","last_became_peered":"2024-01-11T08:29:30.954978+0000","last_unstale":"2024-01-11T08:29:30.955817+0000","last_undegraded":"2024-01-11T08:29:30.955817+0000","last_fullsized":"2024-01-11T08:29:30.955817+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:15:59.730225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.965839+0000","last_change":"2024-01-11T08:29:30.965839+0000","last_active":"2024-01-11T08:29:30.965839+0000","last_peered":"2024-01-11T08:29:30.965839+0000","last_clean":"2024-01-11T08:29:30.965839+0000","last_became_active":"2024-01-11T08:29:30.965589+0000","last_became_peered":"2024-01-11T08:29:30.965589+0000","last_unstale":"2024-01-11T08:29:30.965839+0000","last_undegraded":"2024-01-11T08:29:30.965839+0000","last_fullsized":"2024-01-11T08:29:30.965839+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:09:19.577787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.959573+0000","last_change":"2024-01-11T08:29:30.959573+0000","last_active":"2024-01-11T08:29:30.959573+0000","last_peered":"2024-01-11T08:29:30.959573+0000","last_clean":"2024-01-11T08:29:30.959573+0000","last_became_active":"2024-01-11T08:29:30.958964+0000","last_became_peered":"2024-01-11T08:29:30.958964+0000","last_unstale":"2024-01-11T08:29:30.959573+0000","last_undegraded":"2024-01-11T08:29:30.959573+0000","last_fullsized":"2024-01-11T08:29:30.959573+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:42:02.127224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.943528+0000","last_change":"2024-01-11T08:29:30.943528+0000","last_active":"2024-01-11T08:29:30.943528+0000","last_peered":"2024-01-11T08:29:30.943528+0000","last_clean":"2024-01-11T08:29:30.943528+0000","last_became_active":"2024-01-11T08:29:30.943273+0000","last_became_peered":"2024-01-11T08:29:30.943273+0000","last_unstale":"2024-01-11T08:29:30.943528+0000","last_undegraded":"2024-01-11T08:29:30.943528+0000","last_fullsized":"2024-01-11T08:29:30.943528+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:49:50.965540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.950739+0000","last_change":"2024-01-11T08:29:30.950739+0000","last_active":"2024-01-11T08:29:30.950739+0000","last_peered":"2024-01-11T08:29:30.950739+0000","last_clean":"2024-01-11T08:29:30.950739+0000","last_became_active":"2024-01-11T08:29:30.949011+0000","last_became_peered":"2024-01-11T08:29:30.949011+0000","last_unstale":"2024-01-11T08:29:30.950739+0000","last_undegraded":"2024-01-11T08:29:30.950739+0000","last_fullsized":"2024-01-11T08:29:30.950739+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:38:51.548179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.965377+0000","last_change":"2024-01-11T08:29:30.965377+0000","last_active":"2024-01-11T08:29:30.965377+0000","last_peered":"2024-01-11T08:29:30.965377+0000","last_clean":"2024-01-11T08:29:30.965377+0000","last_became_active":"2024-01-11T08:29:30.965038+0000","last_became_peered":"2024-01-11T08:29:30.965038+0000","last_unstale":"2024-01-11T08:29:30.965377+0000","last_undegraded":"2024-01-11T08:29:30.965377+0000","last_fullsized":"2024-01-11T08:29:30.965377+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:39:29.986735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.961930+0000","last_change":"2024-01-11T08:29:30.961930+0000","last_active":"2024-01-11T08:29:30.961930+0000","last_peered":"2024-01-11T08:29:30.961930+0000","last_clean":"2024-01-11T08:29:30.961930+0000","last_became_active":"2024-01-11T08:29:30.959962+0000","last_became_peered":"2024-01-11T08:29:30.959962+0000","last_unstale":"2024-01-11T08:29:30.961930+0000","last_undegraded":"2024-01-11T08:29:30.961930+0000","last_fullsized":"2024-01-11T08:29:30.961930+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:33:06.520797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.961485+0000","last_change":"2024-01-11T08:29:30.961485+0000","last_active":"2024-01-11T08:29:30.961485+0000","last_peered":"2024-01-11T08:29:30.961485+0000","last_clean":"2024-01-11T08:29:30.961485+0000","last_became_active":"2024-01-11T08:29:30.961241+0000","last_became_peered":"2024-01-11T08:29:30.961241+0000","last_unstale":"2024-01-11T08:29:30.961485+0000","last_undegraded":"2024-01-11T08:29:30.961485+0000","last_fullsized":"2024-01-11T08:29:30.961485+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:08:31.589550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.949914+0000","last_change":"2024-01-11T08:29:30.949914+0000","last_active":"2024-01-11T08:29:30.949914+0000","last_peered":"2024-01-11T08:29:30.949914+0000","last_clean":"2024-01-11T08:29:30.949914+0000","last_became_active":"2024-01-11T08:29:30.949634+0000","last_became_peered":"2024-01-11T08:29:30.949634+0000","last_unstale":"2024-01-11T08:29:30.949914+0000","last_undegraded":"2024-01-11T08:29:30.949914+0000","last_fullsized":"2024-01-11T08:29:30.949914+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:53:28.537689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.974563+0000","last_change":"2024-01-11T08:29:30.974563+0000","last_active":"2024-01-11T08:29:30.974563+0000","last_peered":"2024-01-11T08:29:30.974563+0000","last_clean":"2024-01-11T08:29:30.974563+0000","last_became_active":"2024-01-11T08:29:30.974282+0000","last_became_peered":"2024-01-11T08:29:30.974282+0000","last_unstale":"2024-01-11T08:29:30.974563+0000","last_undegraded":"2024-01-11T08:29:30.974563+0000","last_fullsized":"2024-01-11T08:29:30.974563+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:43:06.583414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.961118+0000","last_change":"2024-01-11T08:29:30.961118+0000","last_active":"2024-01-11T08:29:30.961118+0000","last_peered":"2024-01-11T08:29:30.961118+0000","last_clean":"2024-01-11T08:29:30.961118+0000","last_became_active":"2024-01-11T08:29:30.960837+0000","last_became_peered":"2024-01-11T08:29:30.960837+0000","last_unstale":"2024-01-11T08:29:30.961118+0000","last_undegraded":"2024-01-11T08:29:30.961118+0000","last_fullsized":"2024-01-11T08:29:30.961118+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:30:31.171934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.957164+0000","last_change":"2024-01-11T08:29:30.957164+0000","last_active":"2024-01-11T08:29:30.957164+0000","last_peered":"2024-01-11T08:29:30.957164+0000","last_clean":"2024-01-11T08:29:30.957164+0000","last_became_active":"2024-01-11T08:29:30.956704+0000","last_became_peered":"2024-01-11T08:29:30.956704+0000","last_unstale":"2024-01-11T08:29:30.957164+0000","last_undegraded":"2024-01-11T08:29:30.957164+0000","last_fullsized":"2024-01-11T08:29:30.957164+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:34:38.966829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.943427+0000","last_change":"2024-01-11T08:29:30.943427+0000","last_active":"2024-01-11T08:29:30.943427+0000","last_peered":"2024-01-11T08:29:30.943427+0000","last_clean":"2024-01-11T08:29:30.943427+0000","last_became_active":"2024-01-11T08:29:30.943149+0000","last_became_peered":"2024-01-11T08:29:30.943149+0000","last_unstale":"2024-01-11T08:29:30.943427+0000","last_undegraded":"2024-01-11T08:29:30.943427+0000","last_fullsized":"2024-01-11T08:29:30.943427+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:37:56.694211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.957910+0000","last_change":"2024-01-11T08:29:30.957910+0000","last_active":"2024-01-11T08:29:30.957910+0000","last_peered":"2024-01-11T08:29:30.957910+0000","last_clean":"2024-01-11T08:29:30.957910+0000","last_became_active":"2024-01-11T08:29:30.957357+0000","last_became_peered":"2024-01-11T08:29:30.957357+0000","last_unstale":"2024-01-11T08:29:30.957910+0000","last_undegraded":"2024-01-11T08:29:30.957910+0000","last_fullsized":"2024-01-11T08:29:30.957910+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:31:27.156853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.958538+0000","last_change":"2024-01-11T08:29:30.958538+0000","last_active":"2024-01-11T08:29:30.958538+0000","last_peered":"2024-01-11T08:29:30.958538+0000","last_clean":"2024-01-11T08:29:30.958538+0000","last_became_active":"2024-01-11T08:29:30.958247+0000","last_became_peered":"2024-01-11T08:29:30.958247+0000","last_unstale":"2024-01-11T08:29:30.958538+0000","last_undegraded":"2024-01-11T08:29:30.958538+0000","last_fullsized":"2024-01-11T08:29:30.958538+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:49:51.388649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.955759+0000","last_change":"2024-01-11T08:29:30.955759+0000","last_active":"2024-01-11T08:29:30.955759+0000","last_peered":"2024-01-11T08:29:30.955759+0000","last_clean":"2024-01-11T08:29:30.955759+0000","last_became_active":"2024-01-11T08:29:30.955392+0000","last_became_peered":"2024-01-11T08:29:30.955392+0000","last_unstale":"2024-01-11T08:29:30.955759+0000","last_undegraded":"2024-01-11T08:29:30.955759+0000","last_fullsized":"2024-01-11T08:29:30.955759+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:44:11.291678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.959427+0000","last_change":"2024-01-11T08:29:30.959427+0000","last_active":"2024-01-11T08:29:30.959427+0000","last_peered":"2024-01-11T08:29:30.959427+0000","last_clean":"2024-01-11T08:29:30.959427+0000","last_became_active":"2024-01-11T08:29:30.959180+0000","last_became_peered":"2024-01-11T08:29:30.959180+0000","last_unstale":"2024-01-11T08:29:30.959427+0000","last_undegraded":"2024-01-11T08:29:30.959427+0000","last_fullsized":"2024-01-11T08:29:30.959427+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:51:30.126948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.952638+0000","last_change":"2024-01-11T08:29:30.952638+0000","last_active":"2024-01-11T08:29:30.952638+0000","last_peered":"2024-01-11T08:29:30.952638+0000","last_clean":"2024-01-11T08:29:30.952638+0000","last_became_active":"2024-01-11T08:29:30.951632+0000","last_became_peered":"2024-01-11T08:29:30.951632+0000","last_unstale":"2024-01-11T08:29:30.952638+0000","last_undegraded":"2024-01-11T08:29:30.952638+0000","last_fullsized":"2024-01-11T08:29:30.952638+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:00:15.233533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.951297+0000","last_change":"2024-01-11T08:29:30.951297+0000","last_active":"2024-01-11T08:29:30.951297+0000","last_peered":"2024-01-11T08:29:30.951297+0000","last_clean":"2024-01-11T08:29:30.951297+0000","last_became_active":"2024-01-11T08:29:30.950800+0000","last_became_peered":"2024-01-11T08:29:30.950800+0000","last_unstale":"2024-01-11T08:29:30.951297+0000","last_undegraded":"2024-01-11T08:29:30.951297+0000","last_fullsized":"2024-01-11T08:29:30.951297+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:39:29.986735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.948704+0000","last_change":"2024-01-11T08:29:30.948704+0000","last_active":"2024-01-11T08:29:30.948704+0000","last_peered":"2024-01-11T08:29:30.948704+0000","last_clean":"2024-01-11T08:29:30.948704+0000","last_became_active":"2024-01-11T08:29:30.948399+0000","last_became_peered":"2024-01-11T08:29:30.948399+0000","last_unstale":"2024-01-11T08:29:30.948704+0000","last_undegraded":"2024-01-11T08:29:30.948704+0000","last_fullsized":"2024-01-11T08:29:30.948704+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:41:16.769444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.945330+0000","last_change":"2024-01-11T08:29:30.945330+0000","last_active":"2024-01-11T08:29:30.945330+0000","last_peered":"2024-01-11T08:29:30.945330+0000","last_clean":"2024-01-11T08:29:30.945330+0000","last_became_active":"2024-01-11T08:29:30.945015+0000","last_became_peered":"2024-01-11T08:29:30.945015+0000","last_unstale":"2024-01-11T08:29:30.945330+0000","last_undegraded":"2024-01-11T08:29:30.945330+0000","last_fullsized":"2024-01-11T08:29:30.945330+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:35:48.167273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.948101+0000","last_change":"2024-01-11T08:29:30.948101+0000","last_active":"2024-01-11T08:29:30.948101+0000","last_peered":"2024-01-11T08:29:30.948101+0000","last_clean":"2024-01-11T08:29:30.948101+0000","last_became_active":"2024-01-11T08:29:30.947390+0000","last_became_peered":"2024-01-11T08:29:30.947390+0000","last_unstale":"2024-01-11T08:29:30.948101+0000","last_undegraded":"2024-01-11T08:29:30.948101+0000","last_fullsized":"2024-01-11T08:29:30.948101+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:43:31.364333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.953105+0000","last_change":"2024-01-11T08:29:30.953105+0000","last_active":"2024-01-11T08:29:30.953105+0000","last_peered":"2024-01-11T08:29:30.953105+0000","last_clean":"2024-01-11T08:29:30.953105+0000","last_became_active":"2024-01-11T08:29:30.952821+0000","last_became_peered":"2024-01-11T08:29:30.952821+0000","last_unstale":"2024-01-11T08:29:30.953105+0000","last_undegraded":"2024-01-11T08:29:30.953105+0000","last_fullsized":"2024-01-11T08:29:30.953105+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:34:09.126082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.953883+0000","last_change":"2024-01-11T08:29:30.953883+0000","last_active":"2024-01-11T08:29:30.953883+0000","last_peered":"2024-01-11T08:29:30.953883+0000","last_clean":"2024-01-11T08:29:30.953883+0000","last_became_active":"2024-01-11T08:29:30.953590+0000","last_became_peered":"2024-01-11T08:29:30.953590+0000","last_unstale":"2024-01-11T08:29:30.953883+0000","last_undegraded":"2024-01-11T08:29:30.953883+0000","last_fullsized":"2024-01-11T08:29:30.953883+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:33:35.619652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.957996+0000","last_change":"2024-01-11T08:29:30.957996+0000","last_active":"2024-01-11T08:29:30.957996+0000","last_peered":"2024-01-11T08:29:30.957996+0000","last_clean":"2024-01-11T08:29:30.957996+0000","last_became_active":"2024-01-11T08:29:30.957689+0000","last_became_peered":"2024-01-11T08:29:30.957689+0000","last_unstale":"2024-01-11T08:29:30.957996+0000","last_undegraded":"2024-01-11T08:29:30.957996+0000","last_fullsized":"2024-01-11T08:29:30.957996+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:13:49.966637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.955988+0000","last_change":"2024-01-11T08:29:30.955988+0000","last_active":"2024-01-11T08:29:30.955988+0000","last_peered":"2024-01-11T08:29:30.955988+0000","last_clean":"2024-01-11T08:29:30.955988+0000","last_became_active":"2024-01-11T08:29:30.954004+0000","last_became_peered":"2024-01-11T08:29:30.954004+0000","last_unstale":"2024-01-11T08:29:30.955988+0000","last_undegraded":"2024-01-11T08:29:30.955988+0000","last_fullsized":"2024-01-11T08:29:30.955988+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:45:57.853427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.945616+0000","last_change":"2024-01-11T08:29:30.945616+0000","last_active":"2024-01-11T08:29:30.945616+0000","last_peered":"2024-01-11T08:29:30.945616+0000","last_clean":"2024-01-11T08:29:30.945616+0000","last_became_active":"2024-01-11T08:29:30.945229+0000","last_became_peered":"2024-01-11T08:29:30.945229+0000","last_unstale":"2024-01-11T08:29:30.945616+0000","last_undegraded":"2024-01-11T08:29:30.945616+0000","last_fullsized":"2024-01-11T08:29:30.945616+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:09:45.932475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.940292+0000","last_change":"2024-01-11T08:29:30.940292+0000","last_active":"2024-01-11T08:29:30.940292+0000","last_peered":"2024-01-11T08:29:30.940292+0000","last_clean":"2024-01-11T08:29:30.940292+0000","last_became_active":"2024-01-11T08:29:30.940006+0000","last_became_peered":"2024-01-11T08:29:30.940006+0000","last_unstale":"2024-01-11T08:29:30.940292+0000","last_undegraded":"2024-01-11T08:29:30.940292+0000","last_fullsized":"2024-01-11T08:29:30.940292+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:43:33.145473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.954868+0000","last_change":"2024-01-11T08:29:30.954868+0000","last_active":"2024-01-11T08:29:30.954868+0000","last_peered":"2024-01-11T08:29:30.954868+0000","last_clean":"2024-01-11T08:29:30.954868+0000","last_became_active":"2024-01-11T08:29:30.954543+0000","last_became_peered":"2024-01-11T08:29:30.954543+0000","last_unstale":"2024-01-11T08:29:30.954868+0000","last_undegraded":"2024-01-11T08:29:30.954868+0000","last_fullsized":"2024-01-11T08:29:30.954868+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:40:39.476340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.946701+0000","last_change":"2024-01-11T08:29:30.946701+0000","last_active":"2024-01-11T08:29:30.946701+0000","last_peered":"2024-01-11T08:29:30.946701+0000","last_clean":"2024-01-11T08:29:30.946701+0000","last_became_active":"2024-01-11T08:29:30.945412+0000","last_became_peered":"2024-01-11T08:29:30.945412+0000","last_unstale":"2024-01-11T08:29:30.946701+0000","last_undegraded":"2024-01-11T08:29:30.946701+0000","last_fullsized":"2024-01-11T08:29:30.946701+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:45:23.561914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.952924+0000","last_change":"2024-01-11T08:29:30.952924+0000","last_active":"2024-01-11T08:29:30.952924+0000","last_peered":"2024-01-11T08:29:30.952924+0000","last_clean":"2024-01-11T08:29:30.952924+0000","last_became_active":"2024-01-11T08:29:30.952593+0000","last_became_peered":"2024-01-11T08:29:30.952593+0000","last_unstale":"2024-01-11T08:29:30.952924+0000","last_undegraded":"2024-01-11T08:29:30.952924+0000","last_fullsized":"2024-01-11T08:29:30.952924+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:10.043861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"130'4","reported_seq":13,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.967596+0000","last_change":"2024-01-11T08:29:29.947194+0000","last_active":"2024-01-11T08:29:30.967596+0000","last_peered":"2024-01-11T08:29:30.967596+0000","last_clean":"2024-01-11T08:29:30.967596+0000","last_became_active":"2024-01-11T08:29:29.946642+0000","last_became_peered":"2024-01-11T08:29:29.946642+0000","last_unstale":"2024-01-11T08:29:30.967596+0000","last_undegraded":"2024-01-11T08:29:30.967596+0000","last_fullsized":"2024-01-11T08:29:30.967596+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:07:11.026229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.951689+0000","last_change":"2024-01-11T08:29:30.951689+0000","last_active":"2024-01-11T08:29:30.951689+0000","last_peered":"2024-01-11T08:29:30.951689+0000","last_clean":"2024-01-11T08:29:30.951689+0000","last_became_active":"2024-01-11T08:29:30.950138+0000","last_became_peered":"2024-01-11T08:29:30.950138+0000","last_unstale":"2024-01-11T08:29:30.951689+0000","last_undegraded":"2024-01-11T08:29:30.951689+0000","last_fullsized":"2024-01-11T08:29:30.951689+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:43:31.364333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"130'1","reported_seq":10,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.946851+0000","last_change":"2024-01-11T08:29:29.951273+0000","last_active":"2024-01-11T08:29:30.946851+0000","last_peered":"2024-01-11T08:29:30.946851+0000","last_clean":"2024-01-11T08:29:30.946851+0000","last_became_active":"2024-01-11T08:29:29.950949+0000","last_became_peered":"2024-01-11T08:29:29.950949+0000","last_unstale":"2024-01-11T08:29:30.946851+0000","last_undegraded":"2024-01-11T08:29:30.946851+0000","last_fullsized":"2024-01-11T08:29:30.946851+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:37:00.082089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.944353+0000","last_change":"2024-01-11T08:29:30.944353+0000","last_active":"2024-01-11T08:29:30.944353+0000","last_peered":"2024-01-11T08:29:30.944353+0000","last_clean":"2024-01-11T08:29:30.944353+0000","last_became_active":"2024-01-11T08:29:30.943940+0000","last_became_peered":"2024-01-11T08:29:30.943940+0000","last_unstale":"2024-01-11T08:29:30.944353+0000","last_undegraded":"2024-01-11T08:29:30.944353+0000","last_fullsized":"2024-01-11T08:29:30.944353+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:01.558226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"130'2","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.971573+0000","last_change":"2024-01-11T08:29:29.950060+0000","last_active":"2024-01-11T08:29:30.971573+0000","last_peered":"2024-01-11T08:29:30.971573+0000","last_clean":"2024-01-11T08:29:30.971573+0000","last_became_active":"2024-01-11T08:29:29.949747+0000","last_became_peered":"2024-01-11T08:29:29.949747+0000","last_unstale":"2024-01-11T08:29:30.971573+0000","last_undegraded":"2024-01-11T08:29:30.971573+0000","last_fullsized":"2024-01-11T08:29:30.971573+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:09:52.953864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.961214+0000","last_change":"2024-01-11T08:29:30.961214+0000","last_active":"2024-01-11T08:29:30.961214+0000","last_peered":"2024-01-11T08:29:30.961214+0000","last_clean":"2024-01-11T08:29:30.961214+0000","last_became_active":"2024-01-11T08:29:30.960933+0000","last_became_peered":"2024-01-11T08:29:30.960933+0000","last_unstale":"2024-01-11T08:29:30.961214+0000","last_undegraded":"2024-01-11T08:29:30.961214+0000","last_fullsized":"2024-01-11T08:29:30.961214+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:44:11.291678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"130'2","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.974876+0000","last_change":"2024-01-11T08:29:29.950323+0000","last_active":"2024-01-11T08:29:30.974876+0000","last_peered":"2024-01-11T08:29:30.974876+0000","last_clean":"2024-01-11T08:29:30.974876+0000","last_became_active":"2024-01-11T08:29:29.950027+0000","last_became_peered":"2024-01-11T08:29:29.950027+0000","last_unstale":"2024-01-11T08:29:30.974876+0000","last_undegraded":"2024-01-11T08:29:30.974876+0000","last_fullsized":"2024-01-11T08:29:30.974876+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:47.073306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.956192+0000","last_change":"2024-01-11T08:29:30.956192+0000","last_active":"2024-01-11T08:29:30.956192+0000","last_peered":"2024-01-11T08:29:30.956192+0000","last_clean":"2024-01-11T08:29:30.956192+0000","last_became_active":"2024-01-11T08:29:30.955351+0000","last_became_peered":"2024-01-11T08:29:30.955351+0000","last_unstale":"2024-01-11T08:29:30.956192+0000","last_undegraded":"2024-01-11T08:29:30.956192+0000","last_fullsized":"2024-01-11T08:29:30.956192+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:07.405650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"130'1","reported_seq":10,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.935921+0000","last_change":"2024-01-11T08:29:29.944874+0000","last_active":"2024-01-11T08:29:30.935921+0000","last_peered":"2024-01-11T08:29:30.935921+0000","last_clean":"2024-01-11T08:29:30.935921+0000","last_became_active":"2024-01-11T08:29:29.944590+0000","last_became_peered":"2024-01-11T08:29:29.944590+0000","last_unstale":"2024-01-11T08:29:30.935921+0000","last_undegraded":"2024-01-11T08:29:30.935921+0000","last_fullsized":"2024-01-11T08:29:30.935921+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:43:27.465006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.945184+0000","last_change":"2024-01-11T08:29:30.945184+0000","last_active":"2024-01-11T08:29:30.945184+0000","last_peered":"2024-01-11T08:29:30.945184+0000","last_clean":"2024-01-11T08:29:30.945184+0000","last_became_active":"2024-01-11T08:29:30.944133+0000","last_became_peered":"2024-01-11T08:29:30.944133+0000","last_unstale":"2024-01-11T08:29:30.945184+0000","last_undegraded":"2024-01-11T08:29:30.945184+0000","last_fullsized":"2024-01-11T08:29:30.945184+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:29:12.239105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"130'1","reported_seq":10,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.933206+0000","last_change":"2024-01-11T08:29:29.943256+0000","last_active":"2024-01-11T08:29:30.933206+0000","last_peered":"2024-01-11T08:29:30.933206+0000","last_clean":"2024-01-11T08:29:30.933206+0000","last_became_active":"2024-01-11T08:29:29.942633+0000","last_became_peered":"2024-01-11T08:29:29.942633+0000","last_unstale":"2024-01-11T08:29:30.933206+0000","last_undegraded":"2024-01-11T08:29:30.933206+0000","last_fullsized":"2024-01-11T08:29:30.933206+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:03:39.111068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.968204+0000","last_change":"2024-01-11T08:29:30.968204+0000","last_active":"2024-01-11T08:29:30.968204+0000","last_peered":"2024-01-11T08:29:30.968204+0000","last_clean":"2024-01-11T08:29:30.968204+0000","last_became_active":"2024-01-11T08:29:30.967922+0000","last_became_peered":"2024-01-11T08:29:30.967922+0000","last_unstale":"2024-01-11T08:29:30.968204+0000","last_undegraded":"2024-01-11T08:29:30.968204+0000","last_fullsized":"2024-01-11T08:29:30.968204+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:56:44.744224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.945613+0000","last_change":"2024-01-11T08:29:29.943508+0000","last_active":"2024-01-11T08:29:29.945613+0000","last_peered":"2024-01-11T08:29:29.945613+0000","last_clean":"2024-01-11T08:29:29.945613+0000","last_became_active":"2024-01-11T08:29:29.943206+0000","last_became_peered":"2024-01-11T08:29:29.943206+0000","last_unstale":"2024-01-11T08:29:29.945613+0000","last_undegraded":"2024-01-11T08:29:29.945613+0000","last_fullsized":"2024-01-11T08:29:29.945613+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:29:34.325473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.951160+0000","last_change":"2024-01-11T08:29:30.951160+0000","last_active":"2024-01-11T08:29:30.951160+0000","last_peered":"2024-01-11T08:29:30.951160+0000","last_clean":"2024-01-11T08:29:30.951160+0000","last_became_active":"2024-01-11T08:29:30.949366+0000","last_became_peered":"2024-01-11T08:29:30.949366+0000","last_unstale":"2024-01-11T08:29:30.951160+0000","last_undegraded":"2024-01-11T08:29:30.951160+0000","last_fullsized":"2024-01-11T08:29:30.951160+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:41:26.264311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.963241+0000","last_change":"2024-01-11T08:29:29.959452+0000","last_active":"2024-01-11T08:29:29.963241+0000","last_peered":"2024-01-11T08:29:29.963241+0000","last_clean":"2024-01-11T08:29:29.963241+0000","last_became_active":"2024-01-11T08:29:29.959146+0000","last_became_peered":"2024-01-11T08:29:29.959146+0000","last_unstale":"2024-01-11T08:29:29.963241+0000","last_undegraded":"2024-01-11T08:29:29.963241+0000","last_fullsized":"2024-01-11T08:29:29.963241+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:39:28.976690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.951918+0000","last_change":"2024-01-11T08:29:30.951918+0000","last_active":"2024-01-11T08:29:30.951918+0000","last_peered":"2024-01-11T08:29:30.951918+0000","last_clean":"2024-01-11T08:29:30.951918+0000","last_became_active":"2024-01-11T08:29:30.951605+0000","last_became_peered":"2024-01-11T08:29:30.951605+0000","last_unstale":"2024-01-11T08:29:30.951918+0000","last_undegraded":"2024-01-11T08:29:30.951918+0000","last_fullsized":"2024-01-11T08:29:30.951918+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:53:58.467438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.955671+0000","last_change":"2024-01-11T08:29:29.952559+0000","last_active":"2024-01-11T08:29:29.955671+0000","last_peered":"2024-01-11T08:29:29.955671+0000","last_clean":"2024-01-11T08:29:29.955671+0000","last_became_active":"2024-01-11T08:29:29.952290+0000","last_became_peered":"2024-01-11T08:29:29.952290+0000","last_unstale":"2024-01-11T08:29:29.955671+0000","last_undegraded":"2024-01-11T08:29:29.955671+0000","last_fullsized":"2024-01-11T08:29:29.955671+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:03:50.661693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.966348+0000","last_change":"2024-01-11T08:29:30.966348+0000","last_active":"2024-01-11T08:29:30.966348+0000","last_peered":"2024-01-11T08:29:30.966348+0000","last_clean":"2024-01-11T08:29:30.966348+0000","last_became_active":"2024-01-11T08:29:30.966023+0000","last_became_peered":"2024-01-11T08:29:30.966023+0000","last_unstale":"2024-01-11T08:29:30.966348+0000","last_undegraded":"2024-01-11T08:29:30.966348+0000","last_fullsized":"2024-01-11T08:29:30.966348+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:39:29.986735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.948084+0000","last_change":"2024-01-11T08:29:29.944467+0000","last_active":"2024-01-11T08:29:29.948084+0000","last_peered":"2024-01-11T08:29:29.948084+0000","last_clean":"2024-01-11T08:29:29.948084+0000","last_became_active":"2024-01-11T08:29:29.944114+0000","last_became_peered":"2024-01-11T08:29:29.944114+0000","last_unstale":"2024-01-11T08:29:29.948084+0000","last_undegraded":"2024-01-11T08:29:29.948084+0000","last_fullsized":"2024-01-11T08:29:29.948084+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:40:55.948273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.944445+0000","last_change":"2024-01-11T08:29:30.944445+0000","last_active":"2024-01-11T08:29:30.944445+0000","last_peered":"2024-01-11T08:29:30.944445+0000","last_clean":"2024-01-11T08:29:30.944445+0000","last_became_active":"2024-01-11T08:29:30.944199+0000","last_became_peered":"2024-01-11T08:29:30.944199+0000","last_unstale":"2024-01-11T08:29:30.944445+0000","last_undegraded":"2024-01-11T08:29:30.944445+0000","last_fullsized":"2024-01-11T08:29:30.944445+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:14:37.534117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"130'3","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.973607+0000","last_change":"2024-01-11T08:29:29.956521+0000","last_active":"2024-01-11T08:29:30.973607+0000","last_peered":"2024-01-11T08:29:30.973607+0000","last_clean":"2024-01-11T08:29:30.973607+0000","last_became_active":"2024-01-11T08:29:29.956198+0000","last_became_peered":"2024-01-11T08:29:29.956198+0000","last_unstale":"2024-01-11T08:29:30.973607+0000","last_undegraded":"2024-01-11T08:29:30.973607+0000","last_fullsized":"2024-01-11T08:29:30.973607+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:01:36.960192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.947471+0000","last_change":"2024-01-11T08:29:30.947471+0000","last_active":"2024-01-11T08:29:30.947471+0000","last_peered":"2024-01-11T08:29:30.947471+0000","last_clean":"2024-01-11T08:29:30.947471+0000","last_became_active":"2024-01-11T08:29:30.947189+0000","last_became_peered":"2024-01-11T08:29:30.947189+0000","last_unstale":"2024-01-11T08:29:30.947471+0000","last_undegraded":"2024-01-11T08:29:30.947471+0000","last_fullsized":"2024-01-11T08:29:30.947471+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:01:14.432674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"130'6","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.978114+0000","last_change":"2024-01-11T08:29:29.961494+0000","last_active":"2024-01-11T08:29:30.978114+0000","last_peered":"2024-01-11T08:29:30.978114+0000","last_clean":"2024-01-11T08:29:30.978114+0000","last_became_active":"2024-01-11T08:29:29.961161+0000","last_became_peered":"2024-01-11T08:29:29.961161+0000","last_unstale":"2024-01-11T08:29:30.978114+0000","last_undegraded":"2024-01-11T08:29:30.978114+0000","last_fullsized":"2024-01-11T08:29:30.978114+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:51:29.116903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.939636+0000","last_change":"2024-01-11T08:29:30.939636+0000","last_active":"2024-01-11T08:29:30.939636+0000","last_peered":"2024-01-11T08:29:30.939636+0000","last_clean":"2024-01-11T08:29:30.939636+0000","last_became_active":"2024-01-11T08:29:30.938481+0000","last_became_peered":"2024-01-11T08:29:30.938481+0000","last_unstale":"2024-01-11T08:29:30.939636+0000","last_undegraded":"2024-01-11T08:29:30.939636+0000","last_fullsized":"2024-01-11T08:29:30.939636+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:01:37.810887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"130'2","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.975192+0000","last_change":"2024-01-11T08:29:29.946242+0000","last_active":"2024-01-11T08:29:30.975192+0000","last_peered":"2024-01-11T08:29:30.975192+0000","last_clean":"2024-01-11T08:29:30.975192+0000","last_became_active":"2024-01-11T08:29:29.945990+0000","last_became_peered":"2024-01-11T08:29:29.945990+0000","last_unstale":"2024-01-11T08:29:30.975192+0000","last_undegraded":"2024-01-11T08:29:30.975192+0000","last_fullsized":"2024-01-11T08:29:30.975192+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:31:19.370117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.947359+0000","last_change":"2024-01-11T08:29:30.947359+0000","last_active":"2024-01-11T08:29:30.947359+0000","last_peered":"2024-01-11T08:29:30.947359+0000","last_clean":"2024-01-11T08:29:30.947359+0000","last_became_active":"2024-01-11T08:29:30.947094+0000","last_became_peered":"2024-01-11T08:29:30.947094+0000","last_unstale":"2024-01-11T08:29:30.947359+0000","last_undegraded":"2024-01-11T08:29:30.947359+0000","last_fullsized":"2024-01-11T08:29:30.947359+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:30:31.171934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.955101+0000","last_change":"2024-01-11T08:29:29.951218+0000","last_active":"2024-01-11T08:29:29.955101+0000","last_peered":"2024-01-11T08:29:29.955101+0000","last_clean":"2024-01-11T08:29:29.955101+0000","last_became_active":"2024-01-11T08:29:29.950921+0000","last_became_peered":"2024-01-11T08:29:29.950921+0000","last_unstale":"2024-01-11T08:29:29.955101+0000","last_undegraded":"2024-01-11T08:29:29.955101+0000","last_fullsized":"2024-01-11T08:29:29.955101+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:34:37.956784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.942424+0000","last_change":"2024-01-11T08:29:30.942424+0000","last_active":"2024-01-11T08:29:30.942424+0000","last_peered":"2024-01-11T08:29:30.942424+0000","last_clean":"2024-01-11T08:29:30.942424+0000","last_became_active":"2024-01-11T08:29:30.941993+0000","last_became_peered":"2024-01-11T08:29:30.941993+0000","last_unstale":"2024-01-11T08:29:30.942424+0000","last_undegraded":"2024-01-11T08:29:30.942424+0000","last_fullsized":"2024-01-11T08:29:30.942424+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:45:08.580788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"130'8","reported_seq":17,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.980776+0000","last_change":"2024-01-11T08:29:29.957062+0000","last_active":"2024-01-11T08:29:30.980776+0000","last_peered":"2024-01-11T08:29:30.980776+0000","last_clean":"2024-01-11T08:29:30.980776+0000","last_became_active":"2024-01-11T08:29:29.956817+0000","last_became_peered":"2024-01-11T08:29:29.956817+0000","last_unstale":"2024-01-11T08:29:30.980776+0000","last_undegraded":"2024-01-11T08:29:30.980776+0000","last_fullsized":"2024-01-11T08:29:30.980776+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:57:32.872746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.966099+0000","last_change":"2024-01-11T08:29:30.966099+0000","last_active":"2024-01-11T08:29:30.966099+0000","last_peered":"2024-01-11T08:29:30.966099+0000","last_clean":"2024-01-11T08:29:30.966099+0000","last_became_active":"2024-01-11T08:29:30.965647+0000","last_became_peered":"2024-01-11T08:29:30.965647+0000","last_unstale":"2024-01-11T08:29:30.966099+0000","last_undegraded":"2024-01-11T08:29:30.966099+0000","last_fullsized":"2024-01-11T08:29:30.966099+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:02:55.663995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"130'1","reported_seq":10,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.953136+0000","last_change":"2024-01-11T08:29:29.955054+0000","last_active":"2024-01-11T08:29:30.953136+0000","last_peered":"2024-01-11T08:29:30.953136+0000","last_clean":"2024-01-11T08:29:30.953136+0000","last_became_active":"2024-01-11T08:29:29.954838+0000","last_became_peered":"2024-01-11T08:29:29.954838+0000","last_unstale":"2024-01-11T08:29:30.953136+0000","last_undegraded":"2024-01-11T08:29:30.953136+0000","last_fullsized":"2024-01-11T08:29:30.953136+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:45:56.843382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.978571+0000","last_change":"2024-01-11T08:29:30.978571+0000","last_active":"2024-01-11T08:29:30.978571+0000","last_peered":"2024-01-11T08:29:30.978571+0000","last_clean":"2024-01-11T08:29:30.978571+0000","last_became_active":"2024-01-11T08:29:30.977906+0000","last_became_peered":"2024-01-11T08:29:30.977906+0000","last_unstale":"2024-01-11T08:29:30.978571+0000","last_undegraded":"2024-01-11T08:29:30.978571+0000","last_fullsized":"2024-01-11T08:29:30.978571+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:28.686712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"130'2","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.973678+0000","last_change":"2024-01-11T08:29:29.947489+0000","last_active":"2024-01-11T08:29:30.973678+0000","last_peered":"2024-01-11T08:29:30.973678+0000","last_clean":"2024-01-11T08:29:30.973678+0000","last_became_active":"2024-01-11T08:29:29.947244+0000","last_became_peered":"2024-01-11T08:29:29.947244+0000","last_unstale":"2024-01-11T08:29:30.973678+0000","last_undegraded":"2024-01-11T08:29:30.973678+0000","last_fullsized":"2024-01-11T08:29:30.973678+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:22:49.796325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.962953+0000","last_change":"2024-01-11T08:29:30.962953+0000","last_active":"2024-01-11T08:29:30.962953+0000","last_peered":"2024-01-11T08:29:30.962953+0000","last_clean":"2024-01-11T08:29:30.962953+0000","last_became_active":"2024-01-11T08:29:30.962619+0000","last_became_peered":"2024-01-11T08:29:30.962619+0000","last_unstale":"2024-01-11T08:29:30.962953+0000","last_undegraded":"2024-01-11T08:29:30.962953+0000","last_fullsized":"2024-01-11T08:29:30.962953+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:09:45.932475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"130'1","reported_seq":10,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.948165+0000","last_change":"2024-01-11T08:29:29.946009+0000","last_active":"2024-01-11T08:29:30.948165+0000","last_peered":"2024-01-11T08:29:30.948165+0000","last_clean":"2024-01-11T08:29:30.948165+0000","last_became_active":"2024-01-11T08:29:29.945724+0000","last_became_peered":"2024-01-11T08:29:29.945724+0000","last_unstale":"2024-01-11T08:29:30.948165+0000","last_undegraded":"2024-01-11T08:29:30.948165+0000","last_fullsized":"2024-01-11T08:29:30.948165+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:53:34.148937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.958296+0000","last_change":"2024-01-11T08:29:30.958296+0000","last_active":"2024-01-11T08:29:30.958296+0000","last_peered":"2024-01-11T08:29:30.958296+0000","last_clean":"2024-01-11T08:29:30.958296+0000","last_became_active":"2024-01-11T08:29:30.957581+0000","last_became_peered":"2024-01-11T08:29:30.957581+0000","last_unstale":"2024-01-11T08:29:30.958296+0000","last_undegraded":"2024-01-11T08:29:30.958296+0000","last_fullsized":"2024-01-11T08:29:30.958296+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:01:37.810887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.960767+0000","last_change":"2024-01-11T08:29:29.959927+0000","last_active":"2024-01-11T08:29:29.960767+0000","last_peered":"2024-01-11T08:29:29.960767+0000","last_clean":"2024-01-11T08:29:29.960767+0000","last_became_active":"2024-01-11T08:29:29.959609+0000","last_became_peered":"2024-01-11T08:29:29.959609+0000","last_unstale":"2024-01-11T08:29:29.960767+0000","last_undegraded":"2024-01-11T08:29:29.960767+0000","last_fullsized":"2024-01-11T08:29:29.960767+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:24:47.609089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.956089+0000","last_change":"2024-01-11T08:29:30.956089+0000","last_active":"2024-01-11T08:29:30.956089+0000","last_peered":"2024-01-11T08:29:30.956089+0000","last_clean":"2024-01-11T08:29:30.956089+0000","last_became_active":"2024-01-11T08:29:30.955831+0000","last_became_peered":"2024-01-11T08:29:30.955831+0000","last_unstale":"2024-01-11T08:29:30.956089+0000","last_undegraded":"2024-01-11T08:29:30.956089+0000","last_fullsized":"2024-01-11T08:29:30.956089+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:41:16.769444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"130'2","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.978316+0000","last_change":"2024-01-11T08:29:29.956455+0000","last_active":"2024-01-11T08:29:30.978316+0000","last_peered":"2024-01-11T08:29:30.978316+0000","last_clean":"2024-01-11T08:29:30.978316+0000","last_became_active":"2024-01-11T08:29:29.956162+0000","last_became_peered":"2024-01-11T08:29:29.956162+0000","last_unstale":"2024-01-11T08:29:30.978316+0000","last_undegraded":"2024-01-11T08:29:30.978316+0000","last_fullsized":"2024-01-11T08:29:30.978316+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:47:10.201254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.974352+0000","last_change":"2024-01-11T08:29:30.974352+0000","last_active":"2024-01-11T08:29:30.974352+0000","last_peered":"2024-01-11T08:29:30.974352+0000","last_clean":"2024-01-11T08:29:30.974352+0000","last_became_active":"2024-01-11T08:29:30.972851+0000","last_became_peered":"2024-01-11T08:29:30.972851+0000","last_unstale":"2024-01-11T08:29:30.974352+0000","last_undegraded":"2024-01-11T08:29:30.974352+0000","last_fullsized":"2024-01-11T08:29:30.974352+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:48:12.562689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.958216+0000","last_change":"2024-01-11T08:29:29.954402+0000","last_active":"2024-01-11T08:29:29.958216+0000","last_peered":"2024-01-11T08:29:29.958216+0000","last_clean":"2024-01-11T08:29:29.958216+0000","last_became_active":"2024-01-11T08:29:29.954114+0000","last_became_peered":"2024-01-11T08:29:29.954114+0000","last_unstale":"2024-01-11T08:29:29.958216+0000","last_undegraded":"2024-01-11T08:29:29.958216+0000","last_fullsized":"2024-01-11T08:29:29.958216+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:48:39.783474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.961245+0000","last_change":"2024-01-11T08:29:30.961245+0000","last_active":"2024-01-11T08:29:30.961245+0000","last_peered":"2024-01-11T08:29:30.961245+0000","last_clean":"2024-01-11T08:29:30.961245+0000","last_became_active":"2024-01-11T08:29:30.960951+0000","last_became_peered":"2024-01-11T08:29:30.960951+0000","last_unstale":"2024-01-11T08:29:30.961245+0000","last_undegraded":"2024-01-11T08:29:30.961245+0000","last_fullsized":"2024-01-11T08:29:30.961245+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:45:57.853427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.953069+0000","last_change":"2024-01-11T08:29:29.949977+0000","last_active":"2024-01-11T08:29:29.953069+0000","last_peered":"2024-01-11T08:29:29.953069+0000","last_clean":"2024-01-11T08:29:29.953069+0000","last_became_active":"2024-01-11T08:29:29.949643+0000","last_became_peered":"2024-01-11T08:29:29.949643+0000","last_unstale":"2024-01-11T08:29:29.953069+0000","last_undegraded":"2024-01-11T08:29:29.953069+0000","last_fullsized":"2024-01-11T08:29:29.953069+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:42:01.117179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.956708+0000","last_change":"2024-01-11T08:29:30.956708+0000","last_active":"2024-01-11T08:29:30.956708+0000","last_peered":"2024-01-11T08:29:30.956708+0000","last_clean":"2024-01-11T08:29:30.956708+0000","last_became_active":"2024-01-11T08:29:30.956436+0000","last_became_peered":"2024-01-11T08:29:30.956436+0000","last_unstale":"2024-01-11T08:29:30.956708+0000","last_undegraded":"2024-01-11T08:29:30.956708+0000","last_fullsized":"2024-01-11T08:29:30.956708+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:44:11.291678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.960848+0000","last_change":"2024-01-11T08:29:29.957305+0000","last_active":"2024-01-11T08:29:29.960848+0000","last_peered":"2024-01-11T08:29:29.960848+0000","last_clean":"2024-01-11T08:29:29.960848+0000","last_became_active":"2024-01-11T08:29:29.955662+0000","last_became_peered":"2024-01-11T08:29:29.955662+0000","last_unstale":"2024-01-11T08:29:29.960848+0000","last_undegraded":"2024-01-11T08:29:29.960848+0000","last_fullsized":"2024-01-11T08:29:29.960848+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:29:11.229060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.956563+0000","last_change":"2024-01-11T08:29:30.956563+0000","last_active":"2024-01-11T08:29:30.956563+0000","last_peered":"2024-01-11T08:29:30.956563+0000","last_clean":"2024-01-11T08:29:30.956563+0000","last_became_active":"2024-01-11T08:29:30.955684+0000","last_became_peered":"2024-01-11T08:29:30.955684+0000","last_unstale":"2024-01-11T08:29:30.956563+0000","last_undegraded":"2024-01-11T08:29:30.956563+0000","last_fullsized":"2024-01-11T08:29:30.956563+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:51:30.126948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"130'2","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.980211+0000","last_change":"2024-01-11T08:29:29.955328+0000","last_active":"2024-01-11T08:29:30.980211+0000","last_peered":"2024-01-11T08:29:30.980211+0000","last_clean":"2024-01-11T08:29:30.980211+0000","last_became_active":"2024-01-11T08:29:29.955101+0000","last_became_peered":"2024-01-11T08:29:29.955101+0000","last_unstale":"2024-01-11T08:29:30.980211+0000","last_undegraded":"2024-01-11T08:29:30.980211+0000","last_fullsized":"2024-01-11T08:29:30.980211+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:02:38.334428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.967979+0000","last_change":"2024-01-11T08:29:30.967979+0000","last_active":"2024-01-11T08:29:30.967979+0000","last_peered":"2024-01-11T08:29:30.967979+0000","last_clean":"2024-01-11T08:29:30.967979+0000","last_became_active":"2024-01-11T08:29:30.967371+0000","last_became_peered":"2024-01-11T08:29:30.967371+0000","last_unstale":"2024-01-11T08:29:30.967979+0000","last_undegraded":"2024-01-11T08:29:30.967979+0000","last_fullsized":"2024-01-11T08:29:30.967979+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:37:23.414047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.950116+0000","last_change":"2024-01-11T08:29:29.947100+0000","last_active":"2024-01-11T08:29:29.950116+0000","last_peered":"2024-01-11T08:29:29.950116+0000","last_clean":"2024-01-11T08:29:29.950116+0000","last_became_active":"2024-01-11T08:29:29.946793+0000","last_became_peered":"2024-01-11T08:29:29.946793+0000","last_unstale":"2024-01-11T08:29:29.950116+0000","last_undegraded":"2024-01-11T08:29:29.950116+0000","last_fullsized":"2024-01-11T08:29:29.950116+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:38:01.095587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.967378+0000","last_change":"2024-01-11T08:29:30.967378+0000","last_active":"2024-01-11T08:29:30.967378+0000","last_peered":"2024-01-11T08:29:30.967378+0000","last_clean":"2024-01-11T08:29:30.967378+0000","last_became_active":"2024-01-11T08:29:30.967123+0000","last_became_peered":"2024-01-11T08:29:30.967123+0000","last_unstale":"2024-01-11T08:29:30.967378+0000","last_undegraded":"2024-01-11T08:29:30.967378+0000","last_fullsized":"2024-01-11T08:29:30.967378+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:09:37.262759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.960788+0000","last_change":"2024-01-11T08:29:29.956394+0000","last_active":"2024-01-11T08:29:29.960788+0000","last_peered":"2024-01-11T08:29:29.960788+0000","last_clean":"2024-01-11T08:29:29.960788+0000","last_became_active":"2024-01-11T08:29:29.956090+0000","last_became_peered":"2024-01-11T08:29:29.956090+0000","last_unstale":"2024-01-11T08:29:29.960788+0000","last_undegraded":"2024-01-11T08:29:29.960788+0000","last_fullsized":"2024-01-11T08:29:29.960788+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:53:27.527644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.955525+0000","last_change":"2024-01-11T08:29:30.955525+0000","last_active":"2024-01-11T08:29:30.955525+0000","last_peered":"2024-01-11T08:29:30.955525+0000","last_clean":"2024-01-11T08:29:30.955525+0000","last_became_active":"2024-01-11T08:29:30.955210+0000","last_became_peered":"2024-01-11T08:29:30.955210+0000","last_unstale":"2024-01-11T08:29:30.955525+0000","last_undegraded":"2024-01-11T08:29:30.955525+0000","last_fullsized":"2024-01-11T08:29:30.955525+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:45:57.853427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.950592+0000","last_change":"2024-01-11T08:29:29.945656+0000","last_active":"2024-01-11T08:29:29.950592+0000","last_peered":"2024-01-11T08:29:29.950592+0000","last_clean":"2024-01-11T08:29:29.950592+0000","last_became_active":"2024-01-11T08:29:29.945401+0000","last_became_peered":"2024-01-11T08:29:29.945401+0000","last_unstale":"2024-01-11T08:29:29.950592+0000","last_undegraded":"2024-01-11T08:29:29.950592+0000","last_fullsized":"2024-01-11T08:29:29.950592+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:37:00.082089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.944375+0000","last_change":"2024-01-11T08:29:30.944375+0000","last_active":"2024-01-11T08:29:30.944375+0000","last_peered":"2024-01-11T08:29:30.944375+0000","last_clean":"2024-01-11T08:29:30.944375+0000","last_became_active":"2024-01-11T08:29:30.944079+0000","last_became_peered":"2024-01-11T08:29:30.944079+0000","last_unstale":"2024-01-11T08:29:30.944375+0000","last_undegraded":"2024-01-11T08:29:30.944375+0000","last_fullsized":"2024-01-11T08:29:30.944375+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:01:37.970237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.952362+0000","last_change":"2024-01-11T08:29:29.947304+0000","last_active":"2024-01-11T08:29:29.952362+0000","last_peered":"2024-01-11T08:29:29.952362+0000","last_clean":"2024-01-11T08:29:29.952362+0000","last_became_active":"2024-01-11T08:29:29.946101+0000","last_became_peered":"2024-01-11T08:29:29.946101+0000","last_unstale":"2024-01-11T08:29:29.952362+0000","last_undegraded":"2024-01-11T08:29:29.952362+0000","last_fullsized":"2024-01-11T08:29:29.952362+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:53:16.993120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.951110+0000","last_change":"2024-01-11T08:29:30.951110+0000","last_active":"2024-01-11T08:29:30.951110+0000","last_peered":"2024-01-11T08:29:30.951110+0000","last_clean":"2024-01-11T08:29:30.951110+0000","last_became_active":"2024-01-11T08:29:30.947706+0000","last_became_peered":"2024-01-11T08:29:30.947706+0000","last_unstale":"2024-01-11T08:29:30.951110+0000","last_undegraded":"2024-01-11T08:29:30.951110+0000","last_fullsized":"2024-01-11T08:29:30.951110+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:26:37.958065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.945645+0000","last_change":"2024-01-11T08:29:29.943623+0000","last_active":"2024-01-11T08:29:29.945645+0000","last_peered":"2024-01-11T08:29:29.945645+0000","last_clean":"2024-01-11T08:29:29.945645+0000","last_became_active":"2024-01-11T08:29:29.943311+0000","last_became_peered":"2024-01-11T08:29:29.943311+0000","last_unstale":"2024-01-11T08:29:29.945645+0000","last_undegraded":"2024-01-11T08:29:29.945645+0000","last_fullsized":"2024-01-11T08:29:29.945645+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:34:37.956784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.958786+0000","last_change":"2024-01-11T08:29:30.958786+0000","last_active":"2024-01-11T08:29:30.958786+0000","last_peered":"2024-01-11T08:29:30.958786+0000","last_clean":"2024-01-11T08:29:30.958786+0000","last_became_active":"2024-01-11T08:29:30.958121+0000","last_became_peered":"2024-01-11T08:29:30.958121+0000","last_unstale":"2024-01-11T08:29:30.958786+0000","last_undegraded":"2024-01-11T08:29:30.958786+0000","last_fullsized":"2024-01-11T08:29:30.958786+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:43:31.364333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.955563+0000","last_change":"2024-01-11T08:29:29.951164+0000","last_active":"2024-01-11T08:29:29.955563+0000","last_peered":"2024-01-11T08:29:29.955563+0000","last_clean":"2024-01-11T08:29:29.955563+0000","last_became_active":"2024-01-11T08:29:29.950334+0000","last_became_peered":"2024-01-11T08:29:29.950334+0000","last_unstale":"2024-01-11T08:29:29.955563+0000","last_undegraded":"2024-01-11T08:29:29.955563+0000","last_fullsized":"2024-01-11T08:29:29.955563+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:09:21.938412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.962985+0000","last_change":"2024-01-11T08:29:30.962985+0000","last_active":"2024-01-11T08:29:30.962985+0000","last_peered":"2024-01-11T08:29:30.962985+0000","last_clean":"2024-01-11T08:29:30.962985+0000","last_became_active":"2024-01-11T08:29:30.962708+0000","last_became_peered":"2024-01-11T08:29:30.962708+0000","last_unstale":"2024-01-11T08:29:30.962985+0000","last_undegraded":"2024-01-11T08:29:30.962985+0000","last_fullsized":"2024-01-11T08:29:30.962985+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:47:05.813544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.960866+0000","last_change":"2024-01-11T08:29:29.956171+0000","last_active":"2024-01-11T08:29:29.960866+0000","last_peered":"2024-01-11T08:29:29.960866+0000","last_clean":"2024-01-11T08:29:29.960866+0000","last_became_active":"2024-01-11T08:29:29.955604+0000","last_became_peered":"2024-01-11T08:29:29.955604+0000","last_unstale":"2024-01-11T08:29:29.960866+0000","last_undegraded":"2024-01-11T08:29:29.960866+0000","last_fullsized":"2024-01-11T08:29:29.960866+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:09:44.922430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":8,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-01-11T08:29:30.947798+0000","last_change":"2024-01-11T08:29:30.947798+0000","last_active":"2024-01-11T08:29:30.947798+0000","last_peered":"2024-01-11T08:29:30.947798+0000","last_clean":"2024-01-11T08:29:30.947798+0000","last_became_active":"2024-01-11T08:29:30.947070+0000","last_became_peered":"2024-01-11T08:29:30.947070+0000","last_unstale":"2024-01-11T08:29:30.947798+0000","last_undegraded":"2024-01-11T08:29:30.947798+0000","last_fullsized":"2024-01-11T08:29:30.947798+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:29.924097+0000","last_clean_scrub_stamp":"2024-01-11T08:29:29.924097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:15:59.730225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.957738+0000","last_change":"2024-01-11T08:29:29.954394+0000","last_active":"2024-01-11T08:29:29.957738+0000","last_peered":"2024-01-11T08:29:29.957738+0000","last_clean":"2024-01-11T08:29:29.957738+0000","last_became_active":"2024-01-11T08:29:29.954130+0000","last_became_peered":"2024-01-11T08:29:29.954130+0000","last_unstale":"2024-01-11T08:29:29.957738+0000","last_undegraded":"2024-01-11T08:29:29.957738+0000","last_fullsized":"2024-01-11T08:29:29.957738+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:29:28.914052+0000","last_clean_scrub_stamp":"2024-01-11T08:29:28.914052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:27.676667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"121'245","reported_seq":704,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-01-11T08:29:29.943265+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:29:29.943265+0000","last_peered":"2024-01-11T08:29:29.943265+0000","last_clean":"2024-01-11T08:29:29.943265+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:29:29.943265+0000","last_undegraded":"2024-01-11T08:29:29.943265+0000","last_fullsized":"2024-01-11T08:29:29.943265+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1232,"num_read_kb":2542,"num_write":391,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1232,"num_read_kb":2542,"num_write":391,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":245,"ondisk_log_size":245,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575072,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303528,"kb_used_data":2144,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068312,"statfs":{"total":96636764160,"available":96325951488,"internally_reserved":0,"allocated":2195456,"data_stored":1337124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67900000000000005,"15min":0.69399999999999995},"min":{"1min":0.379,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.84699999999999998,"5min":0.95899999999999996,"15min":0.996},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.66500000000000004,"15min":0.70099999999999996},"min":{"1min":0.44900000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86099999999999999,"5min":1.073,"15min":1.073},"last":0.755}]},{"osd":1,"last update":"Thu Jan 11 08:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.68000000000000005,"15min":0.69899999999999995},"min":{"1min":0.504,"5min":0.432,"15min":0.41899999999999998},"max":{"1min":0.88300000000000001,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.68899999999999995,"15min":0.71099999999999997},"min":{"1min":0.42499999999999999,"5min":0.41999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90400000000000003,"5min":0.94999999999999996,"15min":1.014},"last":0.81200000000000006}]},{"osd":2,"last update":"Thu Jan 11 08:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.69599999999999995,"15min":0.72099999999999997},"min":{"1min":0.47099999999999997,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89500000000000002,"5min":0.99299999999999999,"15min":1.034},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.68500000000000005,"15min":0.70899999999999996},"min":{"1min":0.47299999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.76500000000000001}]},{"osd":3,"last update":"Thu Jan 11 08:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.69599999999999995,"15min":0.72899999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.89400000000000002,"5min":0.91700000000000004,"15min":1.032},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71499999999999997,"15min":0.73399999999999999},"min":{"1min":0.52000000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.92500000000000004,"5min":1.008,"15min":1.008},"last":0.84799999999999998}]},{"osd":4,"last update":"Thu Jan 11 08:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59299999999999997,"15min":0.58799999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":0.89700000000000002,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64500000000000002,"15min":0.629},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90100000000000002,"5min":1.002,"15min":1.002},"last":0.93300000000000005}]},{"osd":5,"last update":"Thu Jan 11 08:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.65900000000000003,"15min":0.64100000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.34599999999999997},"max":{"1min":0.97199999999999998,"5min":1.042,"15min":1.042},"last":0.621},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.66800000000000004,"15min":0.66700000000000004},"min":{"1min":0.433,"5min":0.41399999999999998,"15min":0.40400000000000003},"max":{"1min":0.99199999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.67300000000000004}]},{"osd":6,"last update":"Thu Jan 11 08:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.747,"15min":0.75},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.94899999999999995,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.75,"5min":0.71999999999999997,"15min":0.72899999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.96599999999999997,"5min":1.028,"15min":1.028},"last":0.80000000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575071,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302948,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068892,"statfs":{"total":96636764160,"available":96326545408,"internally_reserved":0,"allocated":1601536,"data_stored":746756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.63700000000000001,"15min":0.66400000000000003},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.94799999999999995},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.63,"5min":0.63700000000000001,"15min":0.68100000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.33800000000000002},"max":{"1min":0.82999999999999996,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.78400000000000003}]},{"osd":1,"last update":"Thu Jan 11 08:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.66300000000000003,"15min":0.69299999999999995},"min":{"1min":0.40100000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.93200000000000005,"5min":1.008,"15min":1.163},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.65400000000000003,"15min":0.68500000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88900000000000001,"5min":0.92600000000000005,"15min":1.1950000000000001},"last":0.82699999999999996}]},{"osd":2,"last update":"Thu Jan 11 08:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66900000000000004,"15min":0.70299999999999996},"min":{"1min":0.442,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.93400000000000005,"5min":0.95399999999999996,"15min":1.016},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.67800000000000005,"15min":0.71699999999999997},"min":{"1min":0.42399999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.90000000000000002,"5min":0.91100000000000003,"15min":0.97399999999999998},"last":0.69299999999999995}]},{"osd":3,"last update":"Thu Jan 11 08:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.69999999999999996,"15min":0.72699999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":1.0149999999999999},"last":0.749},{"interface":"front","average":{"1min":0.752,"5min":0.70699999999999996,"15min":0.73699999999999999},"min":{"1min":0.46999999999999997,"5min":0.46000000000000002,"15min":0.41299999999999998},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":1.0049999999999999},"last":0.93700000000000006}]},{"osd":4,"last update":"Thu Jan 11 08:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.57599999999999996,"15min":0.56599999999999995},"min":{"1min":0.45600000000000002,"5min":0.36699999999999999,"15min":0.30199999999999999},"max":{"1min":0.871,"5min":0.873,"15min":0.92000000000000004},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.48699999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.96399999999999997},"last":0.90000000000000002}]},{"osd":5,"last update":"Thu Jan 11 08:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.54700000000000004,"5min":0.375,"15min":0.375},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.751,"5min":0.70099999999999996,"15min":0.70199999999999996},"min":{"1min":0.53500000000000003,"5min":0.41799999999999998,"15min":0.35199999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.96099999999999997}]},{"osd":7,"last update":"Thu Jan 11 08:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.69999999999999996,"15min":0.70899999999999996},"min":{"1min":0.505,"5min":0.44700000000000001,"15min":0.42399999999999999},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.73999999999999999,"15min":0.754},"min":{"1min":0.61799999999999999,"5min":0.46700000000000003,"15min":0.44700000000000001},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":1.1140000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575073,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302948,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068892,"statfs":{"total":96636764160,"available":96326545408,"internally_reserved":0,"allocated":1601536,"data_stored":746756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:28:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47999999999999998,"15min":0.48599999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.85799999999999998,"5min":0.85999999999999999,"15min":0.97699999999999998},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54900000000000004,"15min":0.57499999999999996},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.91700000000000004,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.56499999999999995}]},{"osd":2,"last update":"Thu Jan 11 08:28:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45400000000000001,"15min":0.47899999999999998},"min":{"1min":0.34100000000000003,"5min":0.32300000000000001,"15min":0.312},"max":{"1min":0.70299999999999996,"5min":1.0960000000000001,"15min":1.113},"last":0.441},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54800000000000004,"15min":0.56000000000000005},"min":{"1min":0.35899999999999999,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.90100000000000002},"last":0.66200000000000003}]},{"osd":3,"last update":"Thu Jan 11 08:28:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.59099999999999997,"15min":0.61099999999999999},"min":{"1min":0.41599999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.68300000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55100000000000005,"15min":0.56000000000000005},"min":{"1min":0.34599999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.83799999999999997,"5min":0.88100000000000001,"15min":0.94899999999999995},"last":0.41899999999999998}]},{"osd":4,"last update":"Thu Jan 11 08:28:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.78500000000000003,"15min":0.79000000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.439},"max":{"1min":1.002,"5min":1.129,"15min":1.2170000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.82999999999999996,"15min":0.82899999999999996},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.501},"max":{"1min":1.085,"5min":1.109,"15min":1.246},"last":0.82899999999999996}]},{"osd":5,"last update":"Thu Jan 11 08:28:51 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.77800000000000002,"15min":0.79700000000000004},"min":{"1min":0.53400000000000003,"5min":0.498,"15min":0.498},"max":{"1min":1.028,"5min":1.1299999999999999,"15min":1.218},"last":0.754},{"interface":"front","average":{"1min":0.748,"5min":0.76100000000000001,"15min":0.78400000000000003},"min":{"1min":0.61099999999999999,"5min":0.52900000000000003,"15min":0.51200000000000001},"max":{"1min":0.93500000000000005,"5min":1.1339999999999999,"15min":1.1779999999999999},"last":0.63500000000000001}]},{"osd":6,"last update":"Thu Jan 11 08:28:51 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.79900000000000004,"15min":0.81599999999999995},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.51300000000000001},"max":{"1min":1.0009999999999999,"5min":1.115,"15min":1.1439999999999999},"last":0.875},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80200000000000005,"15min":0.82499999999999996},"min":{"1min":0.54700000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.034,"5min":1.091,"15min":1.228},"last":0.79700000000000004}]},{"osd":7,"last update":"Thu Jan 11 08:28:51 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.81999999999999995,"15min":0.84599999999999997},"min":{"1min":0.58299999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.83199999999999996,"15min":0.84099999999999997},"min":{"1min":0.68400000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1,"5min":1.099,"15min":1.258},"last":0.77200000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575072,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303528,"kb_used_data":2144,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068312,"statfs":{"total":96636764160,"available":96325951488,"internally_reserved":0,"allocated":2195456,"data_stored":1337124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.497,"15min":0.499},"min":{"1min":0.29199999999999998,"5min":0.28699999999999998,"15min":0.24299999999999999},"max":{"1min":0.61499999999999999,"5min":0.84399999999999997,"15min":0.92700000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.36799999999999999},"max":{"1min":0.66500000000000004,"5min":0.90500000000000003,"15min":0.92100000000000004},"last":0.44900000000000001}]},{"osd":2,"last update":"Thu Jan 11 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.55400000000000005,"15min":0.54700000000000004},"min":{"1min":0.41799999999999998,"5min":0.33000000000000002,"15min":0.29099999999999998},"max":{"1min":0.74299999999999999,"5min":1.335,"15min":1.335},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55800000000000005,"15min":0.55600000000000005},"min":{"1min":0.39100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68799999999999994,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.51000000000000001}]},{"osd":3,"last update":"Thu Jan 11 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.375,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.76600000000000001,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.627,"15min":0.626},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.745,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.72899999999999998}]},{"osd":4,"last update":"Thu Jan 11 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.57799999999999996,"5min":0.502,"15min":0.437},"max":{"1min":0.93400000000000005,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.623},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.77400000000000002,"15min":0.78900000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.98399999999999999,"5min":1.1220000000000001,"15min":1.149},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.78500000000000003,"15min":0.80300000000000005},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.52500000000000002},"max":{"1min":0.90900000000000003,"5min":1.518,"15min":1.518},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.81200000000000006,"15min":0.81899999999999995},"min":{"1min":0.61199999999999999,"5min":0.55400000000000005,"15min":0.47799999999999998},"max":{"1min":0.96699999999999997,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.82099999999999995}]},{"osd":6,"last update":"Thu Jan 11 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.80000000000000004,"15min":0.80600000000000005},"min":{"1min":0.54900000000000004,"5min":0.51700000000000002,"15min":0.50900000000000001},"max":{"1min":0.997,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.81699999999999995,"15min":0.83099999999999996},"min":{"1min":0.55000000000000004,"5min":0.52000000000000002,"15min":0.44900000000000001},"max":{"1min":0.97899999999999998,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.97299999999999998}]},{"osd":7,"last update":"Thu Jan 11 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.83299999999999996,"15min":0.84299999999999997},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.51400000000000001},"max":{"1min":0.90700000000000003,"5min":1.371,"15min":1.371},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.81599999999999995,"15min":0.83999999999999997},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.93600000000000005,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.68500000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575071,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311140,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060700,"statfs":{"total":96636764160,"available":96318156800,"internally_reserved":0,"allocated":1601536,"data_stored":746756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46400000000000002,"15min":0.46899999999999997},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.26000000000000001},"max":{"1min":0.64900000000000002,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.503,"15min":0.53600000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68700000000000006,"5min":0.81699999999999995,"15min":0.91400000000000003},"last":0.501}]},{"osd":1,"last update":"Thu Jan 11 08:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53600000000000003,"15min":0.54600000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70699999999999996,"5min":0.77200000000000002,"15min":0.90000000000000002},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55600000000000005,"15min":0.57699999999999996},"min":{"1min":0.40600000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.74399999999999999,"5min":0.79600000000000004,"15min":0.94499999999999995},"last":0.46500000000000002}]},{"osd":3,"last update":"Thu Jan 11 08:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51100000000000001,"15min":0.53800000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.74199999999999999,"5min":0.76500000000000001,"15min":0.91600000000000004},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59599999999999997,"15min":0.625},"min":{"1min":0.40000000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.74199999999999999,"5min":0.81799999999999995,"15min":1.0009999999999999},"last":0.52100000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.745,"15min":0.77900000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.109,"5min":1.109,"15min":1.2889999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.73799999999999999,"15min":0.77500000000000002},"min":{"1min":0.5,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.056,"5min":1.0760000000000001,"15min":1.3089999999999999},"last":0.68999999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.80300000000000005,"15min":0.82599999999999996},"min":{"1min":0.55900000000000005,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.091,"5min":1.2470000000000001,"15min":1.3540000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.80800000000000005,"15min":0.83499999999999996},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.147,"5min":1.2,"15min":1.23},"last":0.67400000000000004}]},{"osd":6,"last update":"Thu Jan 11 08:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.75,"15min":0.78900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.94999999999999996,"5min":1.131,"15min":1.2669999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78700000000000003,"15min":0.81699999999999995},"min":{"1min":0.55000000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.006,"5min":1.351,"15min":1.47},"last":0.60099999999999998}]},{"osd":7,"last update":"Thu Jan 11 08:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.81699999999999995,"15min":0.85299999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.51400000000000001},"max":{"1min":1.0029999999999999,"5min":1.2629999999999999,"15min":1.4099999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79700000000000004,"15min":0.84899999999999998},"min":{"1min":0.57199999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.0209999999999999,"5min":1.2270000000000001,"15min":1.327},"last":0.73799999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575071,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307044,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064796,"statfs":{"total":96636764160,"available":96322351104,"internally_reserved":0,"allocated":1601536,"data_stored":746756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49299999999999999,"15min":0.48599999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.27600000000000002},"max":{"1min":0.63600000000000001,"5min":0.80400000000000005,"15min":1.1930000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.52300000000000002,"15min":0.54300000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.33400000000000002},"max":{"1min":0.80700000000000005,"5min":0.873,"15min":0.873},"last":0.42199999999999999}]},{"osd":1,"last update":"Thu Jan 11 08:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47499999999999998,"15min":0.48599999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.25900000000000001},"max":{"1min":0.71699999999999997,"5min":1.012,"15min":1.012},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.56499999999999995,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.36299999999999999},"max":{"1min":0.78400000000000003,"5min":1.0960000000000001,"15min":1.1499999999999999},"last":0.46899999999999997}]},{"osd":2,"last update":"Thu Jan 11 08:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56699999999999995,"15min":0.57899999999999996},"min":{"1min":0.41999999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.97599999999999998},"last":0.378},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60499999999999998,"15min":0.626},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.32000000000000001},"max":{"1min":0.74299999999999999,"5min":0.89700000000000002,"15min":1.2270000000000001},"last":0.51300000000000001}]},{"osd":4,"last update":"Thu Jan 11 08:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.72399999999999998,"15min":0.73699999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.35999999999999999},"max":{"1min":2.2709999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.753},{"interface":"front","average":{"1min":0.748,"5min":0.76200000000000001,"15min":0.77500000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.503},"max":{"1min":0.97499999999999998,"5min":1.1459999999999999,"15min":1.1819999999999999},"last":0.64600000000000002}]},{"osd":5,"last update":"Thu Jan 11 08:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.76800000000000002,"15min":0.78100000000000003},"min":{"1min":0.55700000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.94899999999999995,"5min":1.0649999999999999,"15min":1.3260000000000001},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.79200000000000004,"15min":0.80400000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.49199999999999999},"max":{"1min":1.0329999999999999,"5min":1.24,"15min":1.4239999999999999},"last":0.88400000000000001}]},{"osd":6,"last update":"Thu Jan 11 08:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.79600000000000004,"15min":0.82699999999999996},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.101,"5min":1.101,"15min":1.335},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":2.3599999999999999,"5min":2.3599999999999999,"15min":2.3599999999999999},"last":0.82599999999999996}]},{"osd":7,"last update":"Thu Jan 11 08:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.81999999999999995,"15min":0.83199999999999996},"min":{"1min":0.54700000000000004,"5min":0.53100000000000003,"15min":0.46600000000000003},"max":{"1min":2.3180000000000001,"5min":2.3180000000000001,"15min":2.3180000000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.84599999999999997,"15min":0.85699999999999998},"min":{"1min":0.61499999999999999,"5min":0.59199999999999997,"15min":0.58399999999999996},"max":{"1min":1.0449999999999999,"5min":1.2050000000000001,"15min":1.21},"last":0.83799999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575072,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311140,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060700,"statfs":{"total":96636764160,"available":96318156800,"internally_reserved":0,"allocated":1601536,"data_stored":746756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69299999999999995,"15min":0.73699999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.28100000000000003},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.117},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70399999999999996,"15min":0.73599999999999999},"min":{"1min":0.51000000000000001,"5min":0.41599999999999998,"15min":0.38600000000000001},"max":{"1min":0.95699999999999996,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.48299999999999998}]},{"osd":1,"last update":"Thu Jan 11 08:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.66900000000000004,"15min":0.69599999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.33300000000000002},"max":{"1min":0.91000000000000003,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.67800000000000005,"15min":0.70799999999999996},"min":{"1min":0.49399999999999999,"5min":0.43099999999999999,"15min":0.36299999999999999},"max":{"1min":0.89100000000000001,"5min":0.99199999999999999,"15min":1.046},"last":0.84899999999999998}]},{"osd":2,"last update":"Thu Jan 11 08:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70999999999999996,"15min":0.71999999999999997},"min":{"1min":0.50700000000000001,"5min":0.46600000000000003,"15min":0.38900000000000001},"max":{"1min":0.92200000000000004,"5min":1.03,"15min":1.03},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.68600000000000005,"15min":0.72399999999999998},"min":{"1min":0.49299999999999999,"5min":0.432,"15min":0.432},"max":{"1min":0.84799999999999998,"5min":0.97699999999999998,"15min":1.0409999999999999},"last":0.751}]},{"osd":3,"last update":"Thu Jan 11 08:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.66800000000000004,"15min":0.69999999999999996},"min":{"1min":0.46300000000000002,"5min":0.46100000000000002,"15min":0.433},"max":{"1min":0.92500000000000004,"5min":0.96799999999999997,"15min":1.0189999999999999},"last":0.623},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.72499999999999998,"15min":0.745},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.97499999999999998,"5min":1.026,"15min":1.026},"last":0.72299999999999998}]},{"osd":5,"last update":"Thu Jan 11 08:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.60999999999999999,"15min":0.61199999999999999},"min":{"1min":0.43099999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.95699999999999996,"5min":0.96599999999999997,"15min":1.0700000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.624,"15min":0.628},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.32600000000000001},"max":{"1min":0.874,"5min":0.94299999999999995,"15min":0.97799999999999998},"last":0.77500000000000002}]},{"osd":6,"last update":"Thu Jan 11 08:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.42099999999999999,"5min":0.40500000000000003,"15min":0.39500000000000002},"max":{"1min":0.94199999999999995,"5min":1.0680000000000001,"15min":1.133},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.69399999999999995,"15min":0.68899999999999995},"min":{"1min":0.46300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.119},"last":0.89300000000000002}]},{"osd":7,"last update":"Thu Jan 11 08:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69299999999999995,"15min":0.68899999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.41299999999999998},"max":{"1min":0.91100000000000003,"5min":1.0129999999999999,"15min":1.0640000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.52400000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.999,"5min":1.127,"15min":1.2350000000000001},"last":0.80800000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575072,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307044,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064796,"statfs":{"total":96636764160,"available":96322351104,"internally_reserved":0,"allocated":1601536,"data_stored":746756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.67100000000000004,"15min":0.70299999999999996},"min":{"1min":0.41999999999999998,"5min":0.39300000000000002,"15min":0.38700000000000001},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.872},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.68300000000000005,"15min":0.70799999999999996},"min":{"1min":0.47999999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88,"5min":0.96199999999999997,"15min":1.696},"last":0.84199999999999997}]},{"osd":1,"last update":"Thu Jan 11 08:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.65100000000000002,"15min":0.69199999999999995},"min":{"1min":0.45500000000000002,"5min":0.39100000000000001,"15min":0.36399999999999999},"max":{"1min":0.88500000000000001,"5min":0.997,"15min":1.4359999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.68100000000000005,"15min":0.72499999999999998},"min":{"1min":0.42999999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.83199999999999996,"5min":1.05,"15min":4.4969999999999999},"last":0.91100000000000003}]},{"osd":2,"last update":"Thu Jan 11 08:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.68100000000000005,"15min":0.70599999999999996},"min":{"1min":0.46600000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81599999999999995,"5min":0.97299999999999998,"15min":1.238},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.68500000000000005,"15min":0.73899999999999999},"min":{"1min":0.49299999999999999,"5min":0.45300000000000001,"15min":0.41599999999999998},"max":{"1min":0.84399999999999997,"5min":0.95899999999999996,"15min":8.0429999999999993},"last":0.68500000000000005}]},{"osd":3,"last update":"Thu Jan 11 08:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.69799999999999995,"15min":0.745},"min":{"1min":0.44900000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0920000000000001,"5min":1.208,"15min":8.0839999999999996},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.73099999999999998,"15min":0.74099999999999999},"min":{"1min":0.61599999999999999,"5min":0.439,"15min":0.40899999999999997},"max":{"1min":0.83499999999999996,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.77100000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.58599999999999997,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.38300000000000001,"15min":0.34300000000000003},"max":{"1min":0.90400000000000003,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64000000000000001,"15min":0.63200000000000001},"min":{"1min":0.439,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85599999999999998,"5min":1.256,"15min":1.256},"last":0.64500000000000002}]},{"osd":6,"last update":"Thu Jan 11 08:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.64900000000000002,"15min":0.64700000000000002},"min":{"1min":0.41899999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.88,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.66700000000000004,"15min":0.68000000000000005},"min":{"1min":0.48699999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.93700000000000006,"5min":1.085,"15min":1.085},"last":0.624}]},{"osd":7,"last update":"Thu Jan 11 08:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.70699999999999996,"15min":0.72599999999999998},"min":{"1min":0.57499999999999996,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.95499999999999996,"5min":1.341,"15min":1.341},"last":0.746},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.76400000000000001,"15min":0.77000000000000002},"min":{"1min":0.57599999999999996,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.98799999999999999,"5min":1.119,"15min":1.119},"last":0.70299999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:29:35.840 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-11T08:29:35.840 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-11T08:29:36.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.083+0000 7fa5277ff700 1 Processor -- start 2024-01-11T08:29:36.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.083+0000 7fa5277ff700 1 -- start start 2024-01-11T08:29:36.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.084+0000 7fa5277ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5200b3e20 0x7fa5200c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:36.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.084+0000 7fa5277ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5200c8f60 0x7fa52006b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:36.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.084+0000 7fa5277ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5200c93d0 con 0x7fa52006be90 2024-01-11T08:29:36.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.084+0000 7fa5277ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5200c9600 con 0x7fa5200c8f60 2024-01-11T08:29:36.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.084+0000 7fa5277ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52006c310 con 0x7fa5200b3e20 2024-01-11T08:29:36.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.085+0000 7fa524d9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5200c8f60 0x7fa52006b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:36.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.085+0000 7fa525d9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa52006be90 0x7fa52006ce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54808/0 (socket says 172.21.15.27:54808) 2024-01-11T08:29:36.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.085+0000 7fa524d9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5200c8f60 0x7fa52006b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36786/0 (socket says 172.21.15.27:36786) 2024-01-11T08:29:36.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.085+0000 7fa525d9c700 1 -- 172.21.15.27:0/2206416280 learned_addr learned my addr 172.21.15.27:0/2206416280 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:36.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.085+0000 7fa52559b700 1 --2- 172.21.15.27:0/2206416280 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5200b3e20 0x7fa5200c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.085+0000 7fa524d9a700 1 -- 172.21.15.27:0/2206416280 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5200b3e20 msgr2=0x7fa5200c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.085+0000 7fa524d9a700 1 --2- 172.21.15.27:0/2206416280 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5200b3e20 0x7fa5200c8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.085+0000 7fa524d9a700 1 -- 172.21.15.27:0/2206416280 >> v1:172.21.15.27:6789/0 conn(0x7fa52006be90 legacy=0x7fa52006ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.085+0000 7fa524d9a700 1 -- 172.21.15.27:0/2206416280 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5200ccc90 con 0x7fa5200c8f60 2024-01-11T08:29:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.085+0000 7fa524d9a700 1 --2- 172.21.15.27:0/2206416280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5200c8f60 0x7fa52006b950 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fa510009910 tx=0x7fa5100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c53cea972ff658 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.086+0000 7fa517fff700 1 -- 172.21.15.27:0/2206416280 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51000b820 con 0x7fa5200c8f60 2024-01-11T08:29:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.086+0000 7fa517fff700 1 -- 172.21.15.27:0/2206416280 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51000b980 con 0x7fa5200c8f60 2024-01-11T08:29:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.086+0000 7fa517fff700 1 -- 172.21.15.27:0/2206416280 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51000bd20 con 0x7fa5200c8f60 2024-01-11T08:29:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.086+0000 7fa5277ff700 1 -- 172.21.15.27:0/2206416280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5200c8f60 msgr2=0x7fa52006b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.086+0000 7fa5277ff700 1 --2- 172.21.15.27:0/2206416280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5200c8f60 0x7fa52006b950 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fa510009910 tx=0x7fa5100091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.086+0000 7fa5277ff700 1 -- 172.21.15.27:0/2206416280 shutdown_connections 2024-01-11T08:29:36.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.086+0000 7fa5277ff700 1 --2- 172.21.15.27:0/2206416280 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5200b3e20 0x7fa5200c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:36.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.086+0000 7fa5277ff700 1 --2- 172.21.15.27:0/2206416280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5200c8f60 0x7fa52006b950 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:36.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.086+0000 7fa5277ff700 1 -- 172.21.15.27:0/2206416280 >> 172.21.15.27:0/2206416280 conn(0x7fa5200ce3e0 msgr2=0x7fa5200ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:36.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.086+0000 7fa5277ff700 1 -- 172.21.15.27:0/2206416280 shutdown_connections 2024-01-11T08:29:36.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.086+0000 7fa5277ff700 1 -- 172.21.15.27:0/2206416280 wait complete. 2024-01-11T08:29:36.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.088+0000 7fa5277ff700 1 Processor -- start 2024-01-11T08:29:36.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.088+0000 7fa5277ff700 1 -- start start 2024-01-11T08:29:36.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.089+0000 7fa5277ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52006be90 0x7fa5201550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:36.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.089+0000 7fa5277ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5200b3e20 0x7fa52015a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:36.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.089+0000 7fa5277ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5200c8f60 0x7fa5200c4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:36.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.089+0000 7fa5277ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5200c6750 con 0x7fa52006be90 2024-01-11T08:29:36.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.089+0000 7fa5277ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5200c75b0 con 0x7fa5200c8f60 2024-01-11T08:29:36.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.089+0000 7fa5277ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5200c7a40 con 0x7fa5200b3e20 2024-01-11T08:29:36.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.089+0000 7fa52559b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52006be90 0x7fa5201550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:36.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.089+0000 7fa52559b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52006be90 0x7fa5201550a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36794/0 (socket says 172.21.15.27:36794) 2024-01-11T08:29:36.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.089+0000 7fa52559b700 1 -- 172.21.15.27:0/1314549304 learned_addr learned my addr 172.21.15.27:0/1314549304 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:36.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.090+0000 7fa524d9a700 1 --2- 172.21.15.27:0/1314549304 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5200b3e20 0x7fa52015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:36.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.090+0000 7fa525d9c700 1 --2- 172.21.15.27:0/1314549304 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5200c8f60 0x7fa5200c4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:36.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.090+0000 7fa52559b700 1 -- 172.21.15.27:0/1314549304 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5200b3e20 msgr2=0x7fa52015a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:36.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.090+0000 7fa52559b700 1 --2- 172.21.15.27:0/1314549304 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5200b3e20 0x7fa52015a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:36.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.090+0000 7fa52559b700 1 -- 172.21.15.27:0/1314549304 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5200c8f60 msgr2=0x7fa5200c4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:36.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.090+0000 7fa52559b700 1 --2- 172.21.15.27:0/1314549304 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5200c8f60 0x7fa5200c4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:36.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.090+0000 7fa52559b700 1 -- 172.21.15.27:0/1314549304 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa510009420 con 0x7fa52006be90 2024-01-11T08:29:36.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.090+0000 7fa524d9a700 1 --2- 172.21.15.27:0/1314549304 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5200b3e20 0x7fa52015a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:36.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.090+0000 7fa525d9c700 1 --2- 172.21.15.27:0/1314549304 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5200c8f60 0x7fa5200c4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:36.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.090+0000 7fa52559b700 1 --2- 172.21.15.27:0/1314549304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52006be90 0x7fa5201550a0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fa51c00d890 tx=0x7fa51c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:36.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.090+0000 7fa5157fa700 1 -- 172.21.15.27:0/1314549304 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51c009610 con 0x7fa52006be90 2024-01-11T08:29:36.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.090+0000 7fa5157fa700 1 -- 172.21.15.27:0/1314549304 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51c009770 con 0x7fa52006be90 2024-01-11T08:29:36.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.090+0000 7fa5277ff700 1 -- 172.21.15.27:0/1314549304 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5201555e0 con 0x7fa52006be90 2024-01-11T08:29:36.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.090+0000 7fa5157fa700 1 -- 172.21.15.27:0/1314549304 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51c009b30 con 0x7fa52006be90 2024-01-11T08:29:36.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.091+0000 7fa5277ff700 1 -- 172.21.15.27:0/1314549304 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5200c6c80 con 0x7fa52006be90 2024-01-11T08:29:36.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.092+0000 7fa5277ff700 1 -- 172.21.15.27:0/1314549304 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa52004c3c0 con 0x7fa52006be90 2024-01-11T08:29:36.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.092+0000 7fa5157fa700 1 -- 172.21.15.27:0/1314549304 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa51c01e460 con 0x7fa52006be90 2024-01-11T08:29:36.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.093+0000 7fa5157fa700 1 --2- 172.21.15.27:0/1314549304 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa504066f30 0x7fa5040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:36.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.094+0000 7fa5157fa700 1 -- 172.21.15.27:0/1314549304 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fa51c084c90 con 0x7fa52006be90 2024-01-11T08:29:36.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.094+0000 7fa524d9a700 1 --2- 172.21.15.27:0/1314549304 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa504066f30 0x7fa5040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:36.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.095+0000 7fa524d9a700 1 --2- 172.21.15.27:0/1314549304 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa504066f30 0x7fa5040693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa5100091e0 tx=0x7fa510017000 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:36.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.097+0000 7fa5157fa700 1 -- 172.21.15.27:0/1314549304 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa51c051e20 con 0x7fa52006be90 2024-01-11T08:29:36.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.379+0000 7fa5277ff700 1 -- 172.21.15.27:0/1314549304 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa5200c86f0 con 0x7fa52006be90 2024-01-11T08:29:36.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.385+0000 7fa5157fa700 1 -- 172.21.15.27:0/1314549304 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+0 (secure 0 0 0) 0x7fa51c055440 con 0x7fa52006be90 2024-01-11T08:29:36.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.388+0000 7fa5277ff700 1 -- 172.21.15.27:0/1314549304 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa504066f30 msgr2=0x7fa5040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:36.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.388+0000 7fa5277ff700 1 --2- 172.21.15.27:0/1314549304 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa504066f30 0x7fa5040693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa5100091e0 tx=0x7fa510017000 comp rx=0 tx=0).stop 2024-01-11T08:29:36.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.388+0000 7fa5277ff700 1 -- 172.21.15.27:0/1314549304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52006be90 msgr2=0x7fa5201550a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:36.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.388+0000 7fa5277ff700 1 --2- 172.21.15.27:0/1314549304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52006be90 0x7fa5201550a0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fa51c00d890 tx=0x7fa51c008040 comp rx=0 tx=0).stop 2024-01-11T08:29:36.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.388+0000 7fa5277ff700 1 -- 172.21.15.27:0/1314549304 shutdown_connections 2024-01-11T08:29:36.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.388+0000 7fa5277ff700 1 --2- 172.21.15.27:0/1314549304 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa504066f30 0x7fa5040693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:36.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.388+0000 7fa5277ff700 1 --2- 172.21.15.27:0/1314549304 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5200c8f60 0x7fa5200c4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:36.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.388+0000 7fa5277ff700 1 --2- 172.21.15.27:0/1314549304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52006be90 0x7fa5201550a0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:36.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.388+0000 7fa5277ff700 1 --2- 172.21.15.27:0/1314549304 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5200b3e20 0x7fa52015a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:36.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.388+0000 7fa5277ff700 1 -- 172.21.15.27:0/1314549304 >> 172.21.15.27:0/1314549304 conn(0x7fa5200ce3e0 msgr2=0x7fa520156ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:36.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.388+0000 7fa5277ff700 1 -- 172.21.15.27:0/1314549304 shutdown_connections 2024-01-11T08:29:36.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.388+0000 7fa5277ff700 1 -- 172.21.15.27:0/1314549304 wait complete. 2024-01-11T08:29:36.388 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-11T08:29:36.404 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:29:36.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.605+0000 7f045a7ff700 1 Processor -- start 2024-01-11T08:29:36.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.606+0000 7f045a7ff700 1 -- start start 2024-01-11T08:29:36.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.607+0000 7f045a7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04540b03e0 0x7f04540b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.607+0000 7f045a7ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f04540b07b0 0x7f04540b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.607+0000 7f045a7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04540cd730 con 0x7f0454159200 2024-01-11T08:29:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.607+0000 7f045a7ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0454159d50 con 0x7f04540b07b0 2024-01-11T08:29:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.607+0000 7f045a7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04540ccb20 con 0x7f04540b03e0 2024-01-11T08:29:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.607+0000 7f04537fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f04540b07b0 0x7f04540b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.607+0000 7f04537fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f04540b07b0 0x7f04540b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36802/0 (socket says 172.21.15.27:36802) 2024-01-11T08:29:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.607+0000 7f04537fe700 1 -- 172.21.15.27:0/3584589820 learned_addr learned my addr 172.21.15.27:0/3584589820 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.607+0000 7f0458d9c700 1 --1- 172.21.15.27:0/3584589820 >> v1:172.21.15.27:6789/0 conn(0x7f0454159200 0x7f0454159640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54814/0 (socket says 172.21.15.27:54814) 2024-01-11T08:29:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.607+0000 7f0453fff700 1 --2- 172.21.15.27:0/3584589820 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04540b03e0 0x7f04540b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.608+0000 7f04537fe700 1 -- 172.21.15.27:0/3584589820 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04540b03e0 msgr2=0x7f04540b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.608+0000 7f04537fe700 1 --2- 172.21.15.27:0/3584589820 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04540b03e0 0x7f04540b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.608+0000 7f04537fe700 1 -- 172.21.15.27:0/3584589820 >> v1:172.21.15.27:6789/0 conn(0x7f0454159200 legacy=0x7f0454159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.608+0000 7f04537fe700 1 -- 172.21.15.27:0/3584589820 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04540cce00 con 0x7f04540b07b0 2024-01-11T08:29:36.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.608+0000 7f04537fe700 1 --2- 172.21.15.27:0/3584589820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04540b07b0 0x7f04540b29c0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f04480099b0 tx=0x7f0448009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d3a5848ae3fc15 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.608+0000 7f0452ffd700 1 -- 172.21.15.27:0/3584589820 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044800b860 con 0x7f04540b07b0 2024-01-11T08:29:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.608+0000 7f0452ffd700 1 -- 172.21.15.27:0/3584589820 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044800b9c0 con 0x7f04540b07b0 2024-01-11T08:29:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.608+0000 7f0452ffd700 1 -- 172.21.15.27:0/3584589820 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044800bd60 con 0x7f04540b07b0 2024-01-11T08:29:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.609+0000 7f045a7ff700 1 -- 172.21.15.27:0/3584589820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04540b07b0 msgr2=0x7f04540b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.609+0000 7f045a7ff700 1 --2- 172.21.15.27:0/3584589820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04540b07b0 0x7f04540b29c0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f04480099b0 tx=0x7f0448009250 comp rx=0 tx=0).stop 2024-01-11T08:29:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.609+0000 7f045a7ff700 1 -- 172.21.15.27:0/3584589820 shutdown_connections 2024-01-11T08:29:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.609+0000 7f045a7ff700 1 --2- 172.21.15.27:0/3584589820 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04540b03e0 0x7f04540b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.609+0000 7f045a7ff700 1 --2- 172.21.15.27:0/3584589820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04540b07b0 0x7f04540b29c0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.609+0000 7f045a7ff700 1 -- 172.21.15.27:0/3584589820 >> 172.21.15.27:0/3584589820 conn(0x7f045406ed40 msgr2=0x7f045406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.609+0000 7f045a7ff700 1 -- 172.21.15.27:0/3584589820 shutdown_connections 2024-01-11T08:29:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.609+0000 7f045a7ff700 1 -- 172.21.15.27:0/3584589820 wait complete. 2024-01-11T08:29:36.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.610+0000 7f045a7ff700 1 Processor -- start 2024-01-11T08:29:36.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.611+0000 7f045a7ff700 1 -- start start 2024-01-11T08:29:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.612+0000 7f045a7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04540b03e0 0x7f04540a7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.612+0000 7f045a7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04540b07b0 0x7f0454169b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.612+0000 7f045a7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0454159200 0x7f0454157800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.612+0000 7f045a7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045416a450 con 0x7f04540b07b0 2024-01-11T08:29:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.612+0000 7f045a7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045416a730 con 0x7f04540b03e0 2024-01-11T08:29:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.612+0000 7f045a7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f045416aa10 con 0x7f0454159200 2024-01-11T08:29:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.612+0000 7f04537fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04540b07b0 0x7f0454169b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:36.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.612+0000 7f04537fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04540b07b0 0x7f0454169b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36810/0 (socket says 172.21.15.27:36810) 2024-01-11T08:29:36.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.612+0000 7f04537fe700 1 -- 172.21.15.27:0/1264650771 learned_addr learned my addr 172.21.15.27:0/1264650771 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:36.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.612+0000 7f0458d9c700 1 --2- 172.21.15.27:0/1264650771 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0454159200 0x7f0454157800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:36.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.612+0000 7f0453fff700 1 --2- 172.21.15.27:0/1264650771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04540b03e0 0x7f04540a7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:36.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.612+0000 7f04537fe700 1 -- 172.21.15.27:0/1264650771 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0454159200 msgr2=0x7f0454157800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:36.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.612+0000 7f04537fe700 1 --2- 172.21.15.27:0/1264650771 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0454159200 0x7f0454157800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:36.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.612+0000 7f04537fe700 1 -- 172.21.15.27:0/1264650771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04540b03e0 msgr2=0x7f04540a7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:36.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.612+0000 7f04537fe700 1 --2- 172.21.15.27:0/1264650771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04540b03e0 0x7f04540a7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:36.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.612+0000 7f04537fe700 1 -- 172.21.15.27:0/1264650771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04480094f0 con 0x7f04540b07b0 2024-01-11T08:29:36.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.612+0000 7f0453fff700 1 --2- 172.21.15.27:0/1264650771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04540b03e0 0x7f04540a7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:29:36.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.613+0000 7f04537fe700 1 --2- 172.21.15.27:0/1264650771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04540b07b0 0x7f0454169b40 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f0454069540 tx=0x7f0448017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:36.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.613+0000 7f043bfff700 1 -- 172.21.15.27:0/1264650771 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0448024440 con 0x7f04540b07b0 2024-01-11T08:29:36.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.613+0000 7f043bfff700 1 -- 172.21.15.27:0/1264650771 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04480245a0 con 0x7f04540b07b0 2024-01-11T08:29:36.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.613+0000 7f043bfff700 1 -- 172.21.15.27:0/1264650771 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0448024940 con 0x7f04540b07b0 2024-01-11T08:29:36.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.613+0000 7f045a7ff700 1 -- 172.21.15.27:0/1264650771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f045416ad60 con 0x7f04540b07b0 2024-01-11T08:29:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.613+0000 7f045a7ff700 1 -- 172.21.15.27:0/1264650771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f045416a080 con 0x7f04540b07b0 2024-01-11T08:29:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.614+0000 7f045a7ff700 1 -- 172.21.15.27:0/1264650771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04540718d0 con 0x7f04540b07b0 2024-01-11T08:29:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.615+0000 7f043bfff700 1 -- 172.21.15.27:0/1264650771 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f044801a420 con 0x7f04540b07b0 2024-01-11T08:29:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.616+0000 7f043bfff700 1 --2- 172.21.15.27:0/1264650771 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0440066f30 0x7f04400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.616+0000 7f043bfff700 1 -- 172.21.15.27:0/1264650771 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f0448086400 con 0x7f04540b07b0 2024-01-11T08:29:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.616+0000 7f0453fff700 1 --2- 172.21.15.27:0/1264650771 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0440066f30 0x7f04400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:36.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.617+0000 7f0453fff700 1 --2- 172.21.15.27:0/1264650771 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0440066f30 0x7f04400693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0444006fd0 tx=0x7f0444008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:36.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.620+0000 7f043bfff700 1 -- 172.21.15.27:0/1264650771 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0448053610 con 0x7f04540b07b0 2024-01-11T08:29:36.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.872+0000 7f045a7ff700 1 -- 172.21.15.27:0/1264650771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04540cd0e0 con 0x7f04540b07b0 2024-01-11T08:29:36.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.872+0000 7f043bfff700 1 -- 172.21.15.27:0/1264650771 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4596 (secure 0 0 0) 0x7f0448056c30 con 0x7f04540b07b0 2024-01-11T08:29:36.872 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:36.872 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":69,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6956,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1277153217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1277153217},{"type":"v1","addr":"172.21.15.77:6833","nonce":1277153217}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7777,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2435914089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2435914089},{"type":"v1","addr":"172.21.15.27:6835","nonce":2435914089}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7792,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/4128023892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4128023892},{"type":"v1","addr":"172.21.15.77:6837","nonce":4128023892}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:29:30.441936+0000","modified":"2024-01-11T08:29:33.957014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8058":{"gid":8058,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6834/716078331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":716078331},{"type":"v1","addr":"172.21.15.27:6834","nonce":716078331}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2024-01-11T08:29:36.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.875+0000 7f045a7ff700 1 -- 172.21.15.27:0/1264650771 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0440066f30 msgr2=0x7f04400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:36.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.875+0000 7f045a7ff700 1 --2- 172.21.15.27:0/1264650771 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0440066f30 0x7f04400693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0444006fd0 tx=0x7f0444008040 comp rx=0 tx=0).stop 2024-01-11T08:29:36.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.875+0000 7f045a7ff700 1 -- 172.21.15.27:0/1264650771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04540b07b0 msgr2=0x7f0454169b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:36.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.875+0000 7f045a7ff700 1 --2- 172.21.15.27:0/1264650771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04540b07b0 0x7f0454169b40 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f0454069540 tx=0x7f0448017000 comp rx=0 tx=0).stop 2024-01-11T08:29:36.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.875+0000 7f045a7ff700 1 -- 172.21.15.27:0/1264650771 shutdown_connections 2024-01-11T08:29:36.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.875+0000 7f045a7ff700 1 --2- 172.21.15.27:0/1264650771 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0440066f30 0x7f04400693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:36.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.875+0000 7f045a7ff700 1 --2- 172.21.15.27:0/1264650771 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0454159200 0x7f0454157800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:36.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.875+0000 7f045a7ff700 1 --2- 172.21.15.27:0/1264650771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04540b03e0 0x7f04540a7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:36.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.875+0000 7f045a7ff700 1 --2- 172.21.15.27:0/1264650771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04540b07b0 0x7f0454169b40 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:36.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.875+0000 7f045a7ff700 1 -- 172.21.15.27:0/1264650771 >> 172.21.15.27:0/1264650771 conn(0x7f045406ed40 msgr2=0x7f04540cc320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:36.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.875+0000 7f045a7ff700 1 -- 172.21.15.27:0/1264650771 shutdown_connections 2024-01-11T08:29:36.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:36.875+0000 7f045a7ff700 1 -- 172.21.15.27:0/1264650771 wait complete. 2024-01-11T08:29:36.876 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 69 2024-01-11T08:29:37.295 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-11T08:29:30.441936+0000', 'modified': '2024-01-11T08:29:33.957014+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8058}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8058': {'gid': 8058, 'name': 'a', 'rank': 0, 'incarnation': 65, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.27:6834/716078331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 716078331}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 716078331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-11T08:29:37.295 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-11T08:29:37.296 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-11T08:29:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.498+0000 7f49071ff700 1 Processor -- start 2024-01-11T08:29:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.499+0000 7f49071ff700 1 -- start start 2024-01-11T08:29:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.500+0000 7f49071ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4900068cd0 0x7f490015a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.500+0000 7f49071ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49000b4bd0 con 0x7f490015ca10 2024-01-11T08:29:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.500+0000 7f49071ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f490015ac70 con 0x7f490015cde0 2024-01-11T08:29:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.500+0000 7f49071ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49000b3fc0 con 0x7f4900068cd0 2024-01-11T08:29:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.500+0000 7f4904f9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f490015ca10 0x7f49000685c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54818/0 (socket says 172.21.15.27:54818) 2024-01-11T08:29:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.500+0000 7f4904f9b700 1 -- 172.21.15.27:0/2384351735 learned_addr learned my addr 172.21.15.27:0/2384351735 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:37.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.500+0000 7f48fffff700 1 --2- 172.21.15.27:0/2384351735 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4900068cd0 0x7f490015a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.501+0000 7f48ff7fe700 1 -- 172.21.15.27:0/2384351735 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861360030 0 0) 0x7f49000b4bd0 con 0x7f490015ca10 2024-01-11T08:29:37.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.501+0000 7f48ff7fe700 1 -- 172.21.15.27:0/2384351735 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48f0003570 con 0x7f490015ca10 2024-01-11T08:29:37.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.501+0000 7f48ff7fe700 1 -- 172.21.15.27:0/2384351735 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210930383 0 0) 0x7f490015ac70 con 0x7f490015cde0 2024-01-11T08:29:37.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.501+0000 7f48ff7fe700 1 -- 172.21.15.27:0/2384351735 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49000b4bd0 con 0x7f490015cde0 2024-01-11T08:29:37.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.501+0000 7f48fffff700 1 -- 172.21.15.27:0/2384351735 >> v1:172.21.15.77:6789/0 conn(0x7f490015cde0 legacy=0x7f49000b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:37.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.501+0000 7f48fffff700 1 -- 172.21.15.27:0/2384351735 >> v1:172.21.15.27:6789/0 conn(0x7f490015ca10 legacy=0x7f49000685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:37.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.501+0000 7f48fffff700 1 -- 172.21.15.27:0/2384351735 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49000b42a0 con 0x7f4900068cd0 2024-01-11T08:29:37.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.501+0000 7f48fffff700 1 --2- 172.21.15.27:0/2384351735 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4900068cd0 0x7f490015a730 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f49000b4b80 tx=0x7f48ec0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8431857b8dd2db3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:37.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.501+0000 7f48ff7fe700 1 -- 172.21.15.27:0/2384351735 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ec011630 con 0x7f4900068cd0 2024-01-11T08:29:37.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.502+0000 7f48ff7fe700 1 -- 172.21.15.27:0/2384351735 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48ec011790 con 0x7f4900068cd0 2024-01-11T08:29:37.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.502+0000 7f48ff7fe700 1 -- 172.21.15.27:0/2384351735 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ec011b50 con 0x7f4900068cd0 2024-01-11T08:29:37.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.502+0000 7f49071ff700 1 -- 172.21.15.27:0/2384351735 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4900068cd0 msgr2=0x7f490015a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:37.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.502+0000 7f49071ff700 1 --2- 172.21.15.27:0/2384351735 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4900068cd0 0x7f490015a730 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f49000b4b80 tx=0x7f48ec0091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.502+0000 7f49071ff700 1 -- 172.21.15.27:0/2384351735 shutdown_connections 2024-01-11T08:29:37.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.502+0000 7f49071ff700 1 --2- 172.21.15.27:0/2384351735 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4900068cd0 0x7f490015a730 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.502+0000 7f49071ff700 1 -- 172.21.15.27:0/2384351735 >> 172.21.15.27:0/2384351735 conn(0x7f490006ed40 msgr2=0x7f490006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:37.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.502+0000 7f49071ff700 1 -- 172.21.15.27:0/2384351735 shutdown_connections 2024-01-11T08:29:37.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.502+0000 7f49071ff700 1 -- 172.21.15.27:0/2384351735 wait complete. 2024-01-11T08:29:37.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.504+0000 7f49071ff700 1 Processor -- start 2024-01-11T08:29:37.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.504+0000 7f49071ff700 1 -- start start 2024-01-11T08:29:37.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.505+0000 7f49071ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4900068cd0 0x7f49000b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.505+0000 7f49071ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f490015ca10 0x7f490014d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.505+0000 7f49071ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f490015cde0 0x7f4900169ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.505+0000 7f49071ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490016a220 con 0x7f490015cde0 2024-01-11T08:29:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.505+0000 7f49071ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490014fb00 con 0x7f4900068cd0 2024-01-11T08:29:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.505+0000 7f49071ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4900155cd0 con 0x7f490015ca10 2024-01-11T08:29:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.505+0000 7f490579c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f490015cde0 0x7f4900169ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.505+0000 7f490579c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f490015cde0 0x7f4900169ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36826/0 (socket says 172.21.15.27:36826) 2024-01-11T08:29:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.505+0000 7f490579c700 1 -- 172.21.15.27:0/3520376270 learned_addr learned my addr 172.21.15.27:0/3520376270 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.505+0000 7f4904f9b700 1 --2- 172.21.15.27:0/3520376270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4900068cd0 0x7f49000b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.505+0000 7f48fffff700 1 --2- 172.21.15.27:0/3520376270 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f490015ca10 0x7f490014d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.506+0000 7f490579c700 1 -- 172.21.15.27:0/3520376270 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f490015ca10 msgr2=0x7f490014d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:37.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.506+0000 7f490579c700 1 --2- 172.21.15.27:0/3520376270 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f490015ca10 0x7f490014d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.506+0000 7f490579c700 1 -- 172.21.15.27:0/3520376270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4900068cd0 msgr2=0x7f49000b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:37.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.506+0000 7f490579c700 1 --2- 172.21.15.27:0/3520376270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4900068cd0 0x7f49000b2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.506+0000 7f490579c700 1 -- 172.21.15.27:0/3520376270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ec009420 con 0x7f490015cde0 2024-01-11T08:29:37.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.506+0000 7f490579c700 1 --2- 172.21.15.27:0/3520376270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f490015cde0 0x7f4900169ce0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f48f800c830 tx=0x7f48f8007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:37.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.506+0000 7f48fcff9700 1 -- 172.21.15.27:0/3520376270 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f8012630 con 0x7f490015cde0 2024-01-11T08:29:37.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.506+0000 7f48fcff9700 1 -- 172.21.15.27:0/3520376270 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f8012790 con 0x7f490015cde0 2024-01-11T08:29:37.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.506+0000 7f48fcff9700 1 -- 172.21.15.27:0/3520376270 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f8012b50 con 0x7f490015cde0 2024-01-11T08:29:37.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.506+0000 7f49071ff700 1 -- 172.21.15.27:0/3520376270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490016acc0 con 0x7f490015cde0 2024-01-11T08:29:37.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.506+0000 7f49071ff700 1 -- 172.21.15.27:0/3520376270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490016a700 con 0x7f490015cde0 2024-01-11T08:29:37.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.510+0000 7f48fcff9700 1 -- 172.21.15.27:0/3520376270 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f48f801a3b0 con 0x7f490015cde0 2024-01-11T08:29:37.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.510+0000 7f48fcff9700 1 --2- 172.21.15.27:0/3520376270 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48e4066f30 0x7f48e40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.511+0000 7f48fcff9700 1 -- 172.21.15.27:0/3520376270 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f48f80841d0 con 0x7f490015cde0 2024-01-11T08:29:37.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.511+0000 7f4904f9b700 1 --2- 172.21.15.27:0/3520376270 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48e4066f30 0x7f48e40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.512+0000 7f4904f9b700 1 --2- 172.21.15.27:0/3520376270 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48e4066f30 0x7f48e40693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f48f4007fd0 tx=0x7f48f4009040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:37.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.512+0000 7f4915913b80 1 Processor -- start 2024-01-11T08:29:37.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.512+0000 7f4915913b80 1 -- start start 2024-01-11T08:29:37.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.513+0000 7f4915913b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9db20 0x559a27a7edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.513+0000 7f4915913b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a793a0 0x559a27aa5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.513+0000 7f4915913b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a9def0 0x559a27aa5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.513+0000 7f4915913b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a27996310 con 0x559a27a9db20 2024-01-11T08:29:37.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.513+0000 7f4915913b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a279d3130 con 0x559a27a9def0 2024-01-11T08:29:37.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.513+0000 7f4915913b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x559a279d0af0 con 0x559a27a793a0 2024-01-11T08:29:37.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.513+0000 7f4904f9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9db20 0x559a27a7edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.513+0000 7f4904f9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9db20 0x559a27a7edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36830/0 (socket says 172.21.15.27:36830) 2024-01-11T08:29:37.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.513+0000 7f4904f9b700 1 -- 172.21.15.27:0/3208785678 learned_addr learned my addr 172.21.15.27:0/3208785678 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:37.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.514+0000 7f48fffff700 1 --2- 172.21.15.27:0/3208785678 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a793a0 0x559a27aa5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.514+0000 7f490579c700 1 --2- 172.21.15.27:0/3208785678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a9def0 0x559a27aa5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.514+0000 7f4904f9b700 1 -- 172.21.15.27:0/3208785678 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a793a0 msgr2=0x559a27aa5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:37.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.514+0000 7f4904f9b700 1 --2- 172.21.15.27:0/3208785678 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a793a0 0x559a27aa5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.514+0000 7f4904f9b700 1 -- 172.21.15.27:0/3208785678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a9def0 msgr2=0x559a27aa5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:37.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.514+0000 7f4904f9b700 1 --2- 172.21.15.27:0/3208785678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a9def0 0x559a27aa5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.514+0000 7f4904f9b700 1 -- 172.21.15.27:0/3208785678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559a278bf0a0 con 0x559a27a9db20 2024-01-11T08:29:37.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.514+0000 7f4904f9b700 1 --2- 172.21.15.27:0/3208785678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9db20 0x559a27a7edf0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f48f400f980 tx=0x7f48f4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddafd72c0ff36c40 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:37.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.514+0000 7f48e9ffb700 1 -- 172.21.15.27:0/3208785678 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f4004a40 con 0x559a27a9db20 2024-01-11T08:29:37.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.514+0000 7f48e9ffb700 1 -- 172.21.15.27:0/3208785678 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f4004ba0 con 0x559a27a9db20 2024-01-11T08:29:37.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.514+0000 7f48e9ffb700 1 -- 172.21.15.27:0/3208785678 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f4005050 con 0x559a27a9db20 2024-01-11T08:29:37.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.515+0000 7f4915913b80 1 -- 172.21.15.27:0/3208785678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9db20 msgr2=0x559a27a7edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:37.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.515+0000 7f4915913b80 1 --2- 172.21.15.27:0/3208785678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9db20 0x559a27a7edf0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f48f400f980 tx=0x7f48f4015000 comp rx=0 tx=0).stop 2024-01-11T08:29:37.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.515+0000 7f4915913b80 1 -- 172.21.15.27:0/3208785678 shutdown_connections 2024-01-11T08:29:37.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.515+0000 7f4915913b80 1 --2- 172.21.15.27:0/3208785678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a9def0 0x559a27aa5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.515+0000 7f4915913b80 1 --2- 172.21.15.27:0/3208785678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9db20 0x559a27a7edf0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.515+0000 7f4915913b80 1 --2- 172.21.15.27:0/3208785678 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a793a0 0x559a27aa5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.515+0000 7f4915913b80 1 -- 172.21.15.27:0/3208785678 >> 172.21.15.27:0/3208785678 conn(0x559a27a7b300 msgr2=0x559a27a7dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.515+0000 7f4915913b80 1 -- 172.21.15.27:0/3208785678 shutdown_connections 2024-01-11T08:29:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.515+0000 7f4915913b80 1 -- 172.21.15.27:0/3208785678 wait complete. 2024-01-11T08:29:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.516+0000 7f4915913b80 1 -- start start 2024-01-11T08:29:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.516+0000 7f4915913b80 1 Processor -- start 2024-01-11T08:29:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.517+0000 7f4915913b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a793a0 0x559a27ad2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.517+0000 7f4915913b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a9db20 0x559a27aab1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.517+0000 7f4915913b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9def0 0x559a27aa9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.517+0000 7f4915913b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a279cc8e0 con 0x559a27a9def0 2024-01-11T08:29:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.517+0000 7f4915913b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a27978980 con 0x559a27a793a0 2024-01-11T08:29:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.517+0000 7f4915913b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x559a27978620 con 0x559a27a9db20 2024-01-11T08:29:37.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.517+0000 7f4915913b80 10 client.0 _ll_register_callbacks cb 0x559a277dba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T08:29:37.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.517+0000 7f490579c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9def0 0x559a27aa9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.517+0000 7f490579c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9def0 0x559a27aa9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36832/0 (socket says 172.21.15.27:36832) 2024-01-11T08:29:37.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.517+0000 7f4904f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a793a0 0x559a27ad2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.517+0000 7f490579c700 1 -- 172.21.15.27:0/1746295434 learned_addr learned my addr 172.21.15.27:0/1746295434 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:37.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.517+0000 7f48fffff700 1 --2- 172.21.15.27:0/1746295434 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a9db20 0x559a27aab1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.518+0000 7f490579c700 1 -- 172.21.15.27:0/1746295434 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a9db20 msgr2=0x559a27aab1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:37.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.518+0000 7f490579c700 1 --2- 172.21.15.27:0/1746295434 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a9db20 0x559a27aab1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.518+0000 7f490579c700 1 -- 172.21.15.27:0/1746295434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a793a0 msgr2=0x559a27ad2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:37.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.518+0000 7f490579c700 1 --2- 172.21.15.27:0/1746295434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a793a0 0x559a27ad2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.518+0000 7f490579c700 1 -- 172.21.15.27:0/1746295434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f0003570 con 0x559a27a9def0 2024-01-11T08:29:37.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.518+0000 7f490579c700 1 --2- 172.21.15.27:0/1746295434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9def0 0x559a27aa9e30 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f48f800cfd0 tx=0x7f48f805a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aeebeb41499eb5e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:37.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.518+0000 7f48deffd700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T08:29:37.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.518+0000 7f48deffd700 1 -- 172.21.15.27:0/1746295434 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f8058270 con 0x559a27a9def0 2024-01-11T08:29:37.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.518+0000 7f48deffd700 1 -- 172.21.15.27:0/1746295434 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(131..131 src has 1..131) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f48f805bfc0 con 0x559a27a9def0 2024-01-11T08:29:37.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.518+0000 7f4915913b80 1 -- 172.21.15.27:0/1746295434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559a27a71560 con 0x559a27a9def0 2024-01-11T08:29:37.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.518+0000 7f48deffd700 1 -- 172.21.15.27:0/1746295434 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48f806b1c0 con 0x559a27a9def0 2024-01-11T08:29:37.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.518+0000 7f4915913b80 10 client.8217 fetch_fsmap learned FSMap version 69 2024-01-11T08:29:37.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.518+0000 7f4915913b80 1 -- 172.21.15.27:0/1746295434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=2+}) v3 -- 0x559a2799b3c0 con 0x559a27a9def0 2024-01-11T08:29:37.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.519+0000 7f48deffd700 1 -- 172.21.15.27:0/1746295434 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559a27a71560 con 0x559a27a9def0 2024-01-11T08:29:37.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.519+0000 7f48deffd700 1 -- 172.21.15.27:0/1746295434 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 69) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f48f8055280 con 0x559a27a9def0 2024-01-11T08:29:37.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.519+0000 7f4915913b80 10 client.8217 fetch_fsmap finished waiting for FSMap version 69 2024-01-11T08:29:37.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.519+0000 7f4915913b80 10 client.8217 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-11T08:29:37.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.519+0000 7f4915913b80 20 client.8217 populate_metadata read hostname 'smithi027' 2024-01-11T08:29:37.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.519+0000 7f4915913b80 1 --2- 172.21.15.27:0/1746295434 >> [v2:172.21.15.27:6832/716078331,v1:172.21.15.27:6834/716078331] conn(0x559a27aaa370 0x559a27aa8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.519+0000 7f4915913b80 4 client.8217 mds_command: new command op to 8058 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-11T08:29:37.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.519+0000 7f4915913b80 1 -- 172.21.15.27:0/1746295434 --> [v2:172.21.15.27:6832/716078331,v1:172.21.15.27:6834/716078331] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559a27a4e0e0 con 0x559a27aaa370 2024-01-11T08:29:37.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.519+0000 7f4904f9b700 1 --2- 172.21.15.27:0/1746295434 >> [v2:172.21.15.27:6832/716078331,v1:172.21.15.27:6834/716078331] conn(0x559a27aaa370 0x559a27aa8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.520+0000 7f48deffd700 10 client.8217 ms_handle_connect on v2:172.21.15.27:6832/716078331 2024-01-11T08:29:37.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.520+0000 7f4904f9b700 1 --2- 172.21.15.27:0/1746295434 >> [v2:172.21.15.27:6832/716078331,v1:172.21.15.27:6834/716078331] conn(0x559a27aaa370 0x559a27aa8d00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54539be4a8f25aa5 server_cookie=faf717d77ca3fe5b in_seq=0 out_seq=0 2024-01-11T08:29:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.520+0000 7f48deffd700 1 -- 172.21.15.27:0/1746295434 <== mds.0 v2:172.21.15.27:6832/716078331 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x559a27a4e0e0 con 0x559a27aaa370 2024-01-11T08:29:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.520+0000 7f48deffd700 10 client.8217 handle_command_reply: tid=0 2024-01-11T08:29:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.520+0000 7f4915913b80 1 client.8217 shutdown 2024-01-11T08:29:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.521+0000 7f4915913b80 1 -- 172.21.15.27:0/1746295434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9def0 msgr2=0x559a27aa9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.521+0000 7f4915913b80 1 --2- 172.21.15.27:0/1746295434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9def0 0x559a27aa9e30 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f48f800cfd0 tx=0x7f48f805a040 comp rx=0 tx=0).stop 2024-01-11T08:29:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.521+0000 7f4915913b80 1 -- 172.21.15.27:0/1746295434 shutdown_connections 2024-01-11T08:29:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.521+0000 7f4915913b80 1 --2- 172.21.15.27:0/1746295434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9def0 0x559a27aa9e30 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.521+0000 7f4915913b80 1 --2- 172.21.15.27:0/1746295434 >> [v2:172.21.15.27:6832/716078331,v1:172.21.15.27:6834/716078331] conn(0x559a27aaa370 0x559a27aa8d00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.521+0000 7f4915913b80 1 --2- 172.21.15.27:0/1746295434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a793a0 0x559a27ad2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.521+0000 7f4915913b80 1 --2- 172.21.15.27:0/1746295434 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a9db20 0x559a27aab1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.521+0000 7f48deffd700 0 client.8217 ms_handle_reset on v2:172.21.15.27:6832/716078331 2024-01-11T08:29:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.521+0000 7f4915913b80 1 -- 172.21.15.27:0/1746295434 >> 172.21.15.27:0/1746295434 conn(0x559a27a7b300 msgr2=0x559a27a9d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.521+0000 7f4915913b80 1 -- 172.21.15.27:0/1746295434 shutdown_connections 2024-01-11T08:29:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.521+0000 7f4915913b80 1 -- 172.21.15.27:0/1746295434 wait complete. 2024-01-11T08:29:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.521+0000 7f4915913b80 20 client.8217 trim_cache size 0 max 16384 2024-01-11T08:29:37.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.538+0000 7f4915913b80 1 Processor -- start 2024-01-11T08:29:37.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.538+0000 7f4915913b80 1 -- start start 2024-01-11T08:29:37.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.539+0000 7f4915913b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a793a0 0x559a27aabf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.539+0000 7f4915913b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a9db20 0x559a27a7fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.539+0000 7f4915913b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a9def0 0x559a27a80070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.539+0000 7f4915913b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a278641e0 con 0x559a27a793a0 2024-01-11T08:29:37.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.539+0000 7f4915913b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a27a9ee20 con 0x559a27a9db20 2024-01-11T08:29:37.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.539+0000 7f4915913b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x559a27a79220 con 0x559a27a9def0 2024-01-11T08:29:37.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.539+0000 7f4904f9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a793a0 0x559a27aabf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.540+0000 7f4904f9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a793a0 0x559a27aabf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36848/0 (socket says 172.21.15.27:36848) 2024-01-11T08:29:37.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.540+0000 7f4904f9b700 1 -- 172.21.15.27:0/1072092906 learned_addr learned my addr 172.21.15.27:0/1072092906 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:37.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.540+0000 7f490579c700 1 --2- 172.21.15.27:0/1072092906 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a9def0 0x559a27a80070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.540+0000 7f48fffff700 1 --2- 172.21.15.27:0/1072092906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a9db20 0x559a27a7fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.540+0000 7f4904f9b700 1 -- 172.21.15.27:0/1072092906 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a9def0 msgr2=0x559a27a80070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.540+0000 7f4904f9b700 1 --2- 172.21.15.27:0/1072092906 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a9def0 0x559a27a80070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.540+0000 7f4904f9b700 1 -- 172.21.15.27:0/1072092906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a9db20 msgr2=0x559a27a7fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.540+0000 7f4904f9b700 1 --2- 172.21.15.27:0/1072092906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a9db20 0x559a27a7fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.540+0000 7f4904f9b700 1 -- 172.21.15.27:0/1072092906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559a278331d0 con 0x559a27a793a0 2024-01-11T08:29:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.540+0000 7f490579c700 1 --2- 172.21.15.27:0/1072092906 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a9def0 0x559a27a80070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:29:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.540+0000 7f4904f9b700 1 --2- 172.21.15.27:0/1072092906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a793a0 0x559a27aabf80 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f48f400f950 tx=0x7f48f4022040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43f1e339eb336769 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.540+0000 7f48ea7fc700 1 -- 172.21.15.27:0/1072092906 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f4033070 con 0x559a27a793a0 2024-01-11T08:29:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.540+0000 7f48ea7fc700 1 -- 172.21.15.27:0/1072092906 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f4000c30 con 0x559a27a793a0 2024-01-11T08:29:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.540+0000 7f48ea7fc700 1 -- 172.21.15.27:0/1072092906 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f4004d10 con 0x559a27a793a0 2024-01-11T08:29:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.541+0000 7f4915913b80 1 -- 172.21.15.27:0/1072092906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a793a0 msgr2=0x559a27aabf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.541+0000 7f4915913b80 1 --2- 172.21.15.27:0/1072092906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a793a0 0x559a27aabf80 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f48f400f950 tx=0x7f48f4022040 comp rx=0 tx=0).stop 2024-01-11T08:29:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.541+0000 7f4915913b80 1 -- 172.21.15.27:0/1072092906 shutdown_connections 2024-01-11T08:29:37.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.541+0000 7f4915913b80 1 --2- 172.21.15.27:0/1072092906 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a9def0 0x559a27a80070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.541+0000 7f4915913b80 1 --2- 172.21.15.27:0/1072092906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a793a0 0x559a27aabf80 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.541+0000 7f4915913b80 1 --2- 172.21.15.27:0/1072092906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a9db20 0x559a27a7fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.541+0000 7f4915913b80 1 -- 172.21.15.27:0/1072092906 >> 172.21.15.27:0/1072092906 conn(0x559a27a7b300 msgr2=0x559a27a9d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:37.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.541+0000 7f4915913b80 1 -- 172.21.15.27:0/1072092906 shutdown_connections 2024-01-11T08:29:37.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.541+0000 7f4915913b80 1 -- 172.21.15.27:0/1072092906 wait complete. 2024-01-11T08:29:37.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.542+0000 7f4915913b80 1 -- start start 2024-01-11T08:29:37.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.542+0000 7f4915913b80 1 Processor -- start 2024-01-11T08:29:37.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.543+0000 7f4915913b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a793a0 0x559a27aac000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.543+0000 7f4915913b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9db20 0x559a27b281e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.543+0000 7f4915913b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a9def0 0x559a27b28720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.543+0000 7f4915913b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a27a99c70 con 0x559a27a9db20 2024-01-11T08:29:37.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.543+0000 7f4915913b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a27aa9190 con 0x559a27a793a0 2024-01-11T08:29:37.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.543+0000 7f4915913b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x559a27aa26e0 con 0x559a27a9def0 2024-01-11T08:29:37.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.543+0000 7f4915913b80 10 client.0 _ll_register_callbacks cb 0x559a277dba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T08:29:37.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.543+0000 7f48fffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9db20 0x559a27b281e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.543+0000 7f48fffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9db20 0x559a27b281e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36858/0 (socket says 172.21.15.27:36858) 2024-01-11T08:29:37.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.543+0000 7f48fffff700 1 -- 172.21.15.27:0/2421304014 learned_addr learned my addr 172.21.15.27:0/2421304014 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:37.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.543+0000 7f4904f9b700 1 --2- 172.21.15.27:0/2421304014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a793a0 0x559a27aac000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.543+0000 7f490579c700 1 --2- 172.21.15.27:0/2421304014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a9def0 0x559a27b28720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.544+0000 7f48fffff700 1 -- 172.21.15.27:0/2421304014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a9def0 msgr2=0x559a27b28720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:37.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.544+0000 7f48fffff700 1 --2- 172.21.15.27:0/2421304014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a9def0 0x559a27b28720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.544+0000 7f48fffff700 1 -- 172.21.15.27:0/2421304014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a793a0 msgr2=0x559a27aac000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:37.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.544+0000 7f48fffff700 1 --2- 172.21.15.27:0/2421304014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a793a0 0x559a27aac000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.544+0000 7f48fffff700 1 -- 172.21.15.27:0/2421304014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559a278bf0a0 con 0x559a27a9db20 2024-01-11T08:29:37.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.544+0000 7f48fffff700 1 --2- 172.21.15.27:0/2421304014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9db20 0x559a27b281e0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f48ec009ed0 tx=0x7f48ec016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31050c8b40e1b868 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:37.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.544+0000 7f48deffd700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T08:29:37.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.544+0000 7f48deffd700 1 -- 172.21.15.27:0/2421304014 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ec01b6e0 con 0x559a27a9db20 2024-01-11T08:29:37.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.544+0000 7f4915913b80 1 -- 172.21.15.27:0/2421304014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48f4004ba0 con 0x559a27a9db20 2024-01-11T08:29:37.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.544+0000 7f48deffd700 1 -- 172.21.15.27:0/2421304014 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(131..131 src has 1..131) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f48ec01b840 con 0x559a27a9db20 2024-01-11T08:29:37.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.544+0000 7f48deffd700 1 -- 172.21.15.27:0/2421304014 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48ec01bc50 con 0x559a27a9db20 2024-01-11T08:29:37.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.545+0000 7f4915913b80 10 client.8223 fetch_fsmap learned FSMap version 69 2024-01-11T08:29:37.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.545+0000 7f4915913b80 1 -- 172.21.15.27:0/2421304014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=2+}) v3 -- 0x559a2780cd60 con 0x559a27a9db20 2024-01-11T08:29:37.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.545+0000 7f48deffd700 1 -- 172.21.15.27:0/2421304014 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f4004ba0 con 0x559a27a9db20 2024-01-11T08:29:37.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.545+0000 7f48deffd700 1 -- 172.21.15.27:0/2421304014 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 69) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f48ec027480 con 0x559a27a9db20 2024-01-11T08:29:37.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.545+0000 7f4915913b80 10 client.8223 fetch_fsmap finished waiting for FSMap version 69 2024-01-11T08:29:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.545+0000 7f4915913b80 10 client.8223 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-11T08:29:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.545+0000 7f4915913b80 20 client.8223 populate_metadata read hostname 'smithi027' 2024-01-11T08:29:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.545+0000 7f4915913b80 1 --2- 172.21.15.27:0/2421304014 >> [v2:172.21.15.27:6832/716078331,v1:172.21.15.27:6834/716078331] conn(0x559a27aa2df0 0x559a27aa31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.545+0000 7f4915913b80 4 client.8223 mds_command: new command op to 8058 tid=0[{"prefix": "status"}] 2024-01-11T08:29:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.545+0000 7f4915913b80 1 -- 172.21.15.27:0/2421304014 --> [v2:172.21.15.27:6832/716078331,v1:172.21.15.27:6834/716078331] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559a2796f8d0 con 0x559a27aa2df0 2024-01-11T08:29:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.545+0000 7f4904f9b700 1 --2- 172.21.15.27:0/2421304014 >> [v2:172.21.15.27:6832/716078331,v1:172.21.15.27:6834/716078331] conn(0x559a27aa2df0 0x559a27aa31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.546+0000 7f4904f9b700 1 --2- 172.21.15.27:0/2421304014 >> [v2:172.21.15.27:6832/716078331,v1:172.21.15.27:6834/716078331] conn(0x559a27aa2df0 0x559a27aa31c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7661d8ecca4bbd2f server_cookie=ac1828609ba7cb1c in_seq=0 out_seq=0 2024-01-11T08:29:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.546+0000 7f48deffd700 10 client.8223 ms_handle_connect on v2:172.21.15.27:6832/716078331 2024-01-11T08:29:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.546+0000 7f48deffd700 1 -- 172.21.15.27:0/2421304014 <== mds.0 v2:172.21.15.27:6832/716078331 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x559a2796f8d0 con 0x559a27aa2df0 2024-01-11T08:29:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.546+0000 7f48deffd700 10 client.8223 handle_command_reply: tid=0 2024-01-11T08:29:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.546+0000 7f4915913b80 1 client.8223 shutdown 2024-01-11T08:29:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.546+0000 7f4915913b80 1 -- 172.21.15.27:0/2421304014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9db20 msgr2=0x559a27b281e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.546+0000 7f4915913b80 1 --2- 172.21.15.27:0/2421304014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9db20 0x559a27b281e0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f48ec009ed0 tx=0x7f48ec016000 comp rx=0 tx=0).stop 2024-01-11T08:29:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.546+0000 7f4915913b80 1 -- 172.21.15.27:0/2421304014 shutdown_connections 2024-01-11T08:29:37.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.546+0000 7f4915913b80 1 --2- 172.21.15.27:0/2421304014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x559a27a9def0 0x559a27b28720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.546+0000 7f4915913b80 1 --2- 172.21.15.27:0/2421304014 >> [v2:172.21.15.27:6832/716078331,v1:172.21.15.27:6834/716078331] conn(0x559a27aa2df0 0x559a27aa31c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.546+0000 7f4915913b80 1 --2- 172.21.15.27:0/2421304014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559a27a793a0 0x559a27aac000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.546+0000 7f4915913b80 1 --2- 172.21.15.27:0/2421304014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559a27a9db20 0x559a27b281e0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.546+0000 7f48deffd700 0 client.8223 ms_handle_reset on v2:172.21.15.27:6832/716078331 2024-01-11T08:29:37.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.546+0000 7f4915913b80 1 -- 172.21.15.27:0/2421304014 >> 172.21.15.27:0/2421304014 conn(0x559a27a7b300 msgr2=0x559a27a7fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:37.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.546+0000 7f4915913b80 1 -- 172.21.15.27:0/2421304014 shutdown_connections 2024-01-11T08:29:37.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.546+0000 7f4915913b80 1 -- 172.21.15.27:0/2421304014 wait complete. 2024-01-11T08:29:37.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.547+0000 7f4915913b80 20 client.8223 trim_cache size 0 max 16384 2024-01-11T08:29:37.551 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T08:29:37.551 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "8fddb4a7-9d66-40f1-892d-6fe86c728ee4", 2024-01-11T08:29:37.552 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-11T08:29:37.552 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 8058, 2024-01-11T08:29:37.552 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-11T08:29:37.552 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-11T08:29:37.552 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-11T08:29:37.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 6.6040539049999998, 2024-01-11T08:29:37.552 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 69, 2024-01-11T08:29:37.552 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 131, 2024-01-11T08:29:37.552 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 131, 2024-01-11T08:29:37.552 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 19.326139893000001 2024-01-11T08:29:37.552 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T08:29:37.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.547+0000 7f49071ff700 1 -- 172.21.15.27:0/3520376270 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48e4066f30 msgr2=0x7f48e40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:37.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.547+0000 7f49071ff700 1 --2- 172.21.15.27:0/3520376270 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48e4066f30 0x7f48e40693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f48f4007fd0 tx=0x7f48f4009040 comp rx=0 tx=0).stop 2024-01-11T08:29:37.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.547+0000 7f49071ff700 1 -- 172.21.15.27:0/3520376270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f490015cde0 msgr2=0x7f4900169ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:37.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.547+0000 7f49071ff700 1 --2- 172.21.15.27:0/3520376270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f490015cde0 0x7f4900169ce0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f48f800c830 tx=0x7f48f8007f90 comp rx=0 tx=0).stop 2024-01-11T08:29:37.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.547+0000 7f49071ff700 1 -- 172.21.15.27:0/3520376270 shutdown_connections 2024-01-11T08:29:37.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.547+0000 7f49071ff700 1 --2- 172.21.15.27:0/3520376270 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48e4066f30 0x7f48e40693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.547+0000 7f49071ff700 1 --2- 172.21.15.27:0/3520376270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f490015cde0 0x7f4900169ce0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.547+0000 7f49071ff700 1 --2- 172.21.15.27:0/3520376270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4900068cd0 0x7f49000b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.547+0000 7f49071ff700 1 --2- 172.21.15.27:0/3520376270 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f490015ca10 0x7f490014d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.547+0000 7f49071ff700 1 -- 172.21.15.27:0/3520376270 >> 172.21.15.27:0/3520376270 conn(0x7f490006ed40 msgr2=0x7f490014f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:37.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.547+0000 7f49071ff700 1 -- 172.21.15.27:0/3520376270 shutdown_connections 2024-01-11T08:29:37.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.547+0000 7f49071ff700 1 -- 172.21.15.27:0/3520376270 wait complete. 2024-01-11T08:29:37.563 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-11T08:29:37.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.763+0000 7f304bfff700 1 Processor -- start 2024-01-11T08:29:37.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.763+0000 7f304bfff700 1 -- start start 2024-01-11T08:29:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.765+0000 7f304bfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f30440b2b90 0x7f3044159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.765+0000 7f304bfff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30440cd730 con 0x7f30440b03e0 2024-01-11T08:29:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.765+0000 7f304bfff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30440ccb20 con 0x7f30440b07b0 2024-01-11T08:29:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.765+0000 7f304bfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30440cce00 con 0x7f30440b2b90 2024-01-11T08:29:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.765+0000 7f3049d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f30440b03e0 0x7f30440b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54834/0 (socket says 172.21.15.27:54834) 2024-01-11T08:29:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.765+0000 7f3049d9b700 1 -- 172.21.15.27:0/898179596 learned_addr learned my addr 172.21.15.27:0/898179596 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:37.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.765+0000 7f304959a700 1 --2- 172.21.15.27:0/898179596 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f30440b2b90 0x7f3044159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.765+0000 7f3048d99700 1 -- 172.21.15.27:0/898179596 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937001943 0 0) 0x7f30440cd730 con 0x7f30440b03e0 2024-01-11T08:29:37.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.765+0000 7f3048d99700 1 -- 172.21.15.27:0/898179596 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3030003570 con 0x7f30440b03e0 2024-01-11T08:29:37.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.766+0000 7f3048d99700 1 -- 172.21.15.27:0/898179596 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997355721 0 0) 0x7f30440ccb20 con 0x7f30440b07b0 2024-01-11T08:29:37.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.766+0000 7f3048d99700 1 -- 172.21.15.27:0/898179596 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30440cd730 con 0x7f30440b07b0 2024-01-11T08:29:37.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.766+0000 7f304959a700 1 -- 172.21.15.27:0/898179596 >> v1:172.21.15.77:6789/0 conn(0x7f30440b07b0 legacy=0x7f3044159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:37.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.766+0000 7f304959a700 1 -- 172.21.15.27:0/898179596 >> v1:172.21.15.27:6789/0 conn(0x7f30440b03e0 legacy=0x7f30440b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:37.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.766+0000 7f3048d99700 1 -- 172.21.15.27:0/898179596 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1679530949 0 0) 0x7f3030003570 con 0x7f30440b03e0 2024-01-11T08:29:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.766+0000 7f304959a700 1 -- 172.21.15.27:0/898179596 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30440cd0e0 con 0x7f30440b2b90 2024-01-11T08:29:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.766+0000 7f304959a700 1 --2- 172.21.15.27:0/898179596 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f30440b2b90 0x7f3044159200 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3034009910 tx=0x7f30340091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7b6ba619c348e51 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.766+0000 7f3048d99700 1 -- 172.21.15.27:0/898179596 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040001470 con 0x7f30440b2b90 2024-01-11T08:29:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.766+0000 7f3048d99700 1 -- 172.21.15.27:0/898179596 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3034011420 con 0x7f30440b2b90 2024-01-11T08:29:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.766+0000 7f3048d99700 1 -- 172.21.15.27:0/898179596 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30340117c0 con 0x7f30440b2b90 2024-01-11T08:29:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.767+0000 7f304bfff700 1 -- 172.21.15.27:0/898179596 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f30440b2b90 msgr2=0x7f3044159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.767+0000 7f304bfff700 1 --2- 172.21.15.27:0/898179596 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f30440b2b90 0x7f3044159200 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3034009910 tx=0x7f30340091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.767+0000 7f304bfff700 1 -- 172.21.15.27:0/898179596 shutdown_connections 2024-01-11T08:29:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.767+0000 7f304bfff700 1 --2- 172.21.15.27:0/898179596 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f30440b2b90 0x7f3044159200 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.767+0000 7f304bfff700 1 -- 172.21.15.27:0/898179596 >> 172.21.15.27:0/898179596 conn(0x7f304406ed40 msgr2=0x7f304406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.767+0000 7f304bfff700 1 -- 172.21.15.27:0/898179596 shutdown_connections 2024-01-11T08:29:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.767+0000 7f304bfff700 1 -- 172.21.15.27:0/898179596 wait complete. 2024-01-11T08:29:37.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.769+0000 7f304bfff700 1 Processor -- start 2024-01-11T08:29:37.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.769+0000 7f304bfff700 1 -- start start 2024-01-11T08:29:37.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.770+0000 7f304bfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f30440b03e0 0x7f304414da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.770+0000 7f304bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30440b07b0 0x7f30440d0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.770+0000 7f304bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30440b2b90 0x7f30440c92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.770+0000 7f304bfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30440c8b50 con 0x7f30440b07b0 2024-01-11T08:29:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.770+0000 7f304bfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304414df60 con 0x7f30440b2b90 2024-01-11T08:29:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.770+0000 7f304bfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30440c7fd0 con 0x7f30440b03e0 2024-01-11T08:29:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.770+0000 7f304959a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30440b07b0 0x7f30440d0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.770+0000 7f304959a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30440b07b0 0x7f30440d0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36872/0 (socket says 172.21.15.27:36872) 2024-01-11T08:29:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.770+0000 7f304959a700 1 -- 172.21.15.27:0/3249946205 learned_addr learned my addr 172.21.15.27:0/3249946205 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.770+0000 7f3049d9b700 1 --2- 172.21.15.27:0/3249946205 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f30440b03e0 0x7f304414da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.770+0000 7f304a59c700 1 --2- 172.21.15.27:0/3249946205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30440b2b90 0x7f30440c92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.771+0000 7f304959a700 1 -- 172.21.15.27:0/3249946205 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f30440b03e0 msgr2=0x7f304414da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.771+0000 7f304959a700 1 --2- 172.21.15.27:0/3249946205 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f30440b03e0 0x7f304414da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.771+0000 7f304959a700 1 -- 172.21.15.27:0/3249946205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30440b2b90 msgr2=0x7f30440c92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.771+0000 7f304959a700 1 --2- 172.21.15.27:0/3249946205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30440b2b90 0x7f30440c92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.771+0000 7f304959a700 1 -- 172.21.15.27:0/3249946205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3034009420 con 0x7f30440b07b0 2024-01-11T08:29:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.771+0000 7f304a59c700 1 --2- 172.21.15.27:0/3249946205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30440b2b90 0x7f30440c92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.771+0000 7f304959a700 1 --2- 172.21.15.27:0/3249946205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30440b07b0 0x7f30440d0600 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f30340091e0 tx=0x7f3034016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.771+0000 7f3039ffb700 1 -- 172.21.15.27:0/3249946205 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303401cbe0 con 0x7f30440b07b0 2024-01-11T08:29:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.771+0000 7f3039ffb700 1 -- 172.21.15.27:0/3249946205 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f303401cd40 con 0x7f30440b07b0 2024-01-11T08:29:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.771+0000 7f304bfff700 1 -- 172.21.15.27:0/3249946205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30440c82b0 con 0x7f30440b07b0 2024-01-11T08:29:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.771+0000 7f3039ffb700 1 -- 172.21.15.27:0/3249946205 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303401a6a0 con 0x7f30440b07b0 2024-01-11T08:29:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.771+0000 7f304bfff700 1 -- 172.21.15.27:0/3249946205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30440c8870 con 0x7f30440b07b0 2024-01-11T08:29:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.772+0000 7f304bfff700 1 -- 172.21.15.27:0/3249946205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30440718d0 con 0x7f30440b07b0 2024-01-11T08:29:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.773+0000 7f3039ffb700 1 -- 172.21.15.27:0/3249946205 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f303401aa90 con 0x7f30440b07b0 2024-01-11T08:29:37.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.774+0000 7f3039ffb700 1 --2- 172.21.15.27:0/3249946205 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3028066f30 0x7f30280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:37.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.774+0000 7f3039ffb700 1 -- 172.21.15.27:0/3249946205 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f3034086400 con 0x7f30440b07b0 2024-01-11T08:29:37.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.774+0000 7f3049d9b700 1 --2- 172.21.15.27:0/3249946205 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3028066f30 0x7f30280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:37.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.775+0000 7f3049d9b700 1 --2- 172.21.15.27:0/3249946205 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3028066f30 0x7f30280693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f30400015a0 tx=0x7f3040000bc0 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:37.778+0000 7f3039ffb700 1 -- 172.21.15.27:0/3249946205 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3034053630 con 0x7f30440b07b0 2024-01-11T08:29:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:38.027+0000 7f304bfff700 1 -- 172.21.15.27:0/3249946205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f30440cd3c0 con 0x7f30440b07b0 2024-01-11T08:29:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:38.027+0000 7f3039ffb700 1 -- 172.21.15.27:0/3249946205 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v69) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3034056c50 con 0x7f30440b07b0 2024-01-11T08:29:38.026 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-11T08:29:38.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:38.029+0000 7f304bfff700 1 -- 172.21.15.27:0/3249946205 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3028066f30 msgr2=0x7f30280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:38.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:38.029+0000 7f304bfff700 1 --2- 172.21.15.27:0/3249946205 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3028066f30 0x7f30280693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f30400015a0 tx=0x7f3040000bc0 comp rx=0 tx=0).stop 2024-01-11T08:29:38.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:38.029+0000 7f304bfff700 1 -- 172.21.15.27:0/3249946205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30440b07b0 msgr2=0x7f30440d0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:38.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:38.029+0000 7f304bfff700 1 --2- 172.21.15.27:0/3249946205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30440b07b0 0x7f30440d0600 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f30340091e0 tx=0x7f3034016000 comp rx=0 tx=0).stop 2024-01-11T08:29:38.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:38.029+0000 7f304bfff700 1 -- 172.21.15.27:0/3249946205 shutdown_connections 2024-01-11T08:29:38.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:38.029+0000 7f304bfff700 1 --2- 172.21.15.27:0/3249946205 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3028066f30 0x7f30280693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:38.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:38.029+0000 7f304bfff700 1 --2- 172.21.15.27:0/3249946205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30440b2b90 0x7f30440c92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:38.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:38.029+0000 7f304bfff700 1 --2- 172.21.15.27:0/3249946205 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f30440b03e0 0x7f304414da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:38.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:38.029+0000 7f304bfff700 1 --2- 172.21.15.27:0/3249946205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30440b07b0 0x7f30440d0600 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:38.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:38.029+0000 7f304bfff700 1 -- 172.21.15.27:0/3249946205 >> 172.21.15.27:0/3249946205 conn(0x7f304406ed40 msgr2=0x7f30440704a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:38.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:38.030+0000 7f304bfff700 1 -- 172.21.15.27:0/3249946205 shutdown_connections 2024-01-11T08:29:38.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:38.030+0000 7f304bfff700 1 -- 172.21.15.27:0/3249946205 wait complete. 2024-01-11T08:29:38.046 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-11T08:29:38.046 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-11T08:29:38.046 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-11T08:29:38.046 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-11T08:29:38.046 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-11T08:29:38.047 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-11T08:29:38.047 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-11T08:29:38.047 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-11T08:29:38.047 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:29:38.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:29:38.047 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-11T08:29:38.096 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T08:29:38.096 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T08:29:38.096 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T08:29:38.096 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:29:38.096 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-11T08:29:38.097 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:29:38.097 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:29:38.097 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:29:38.097 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-11T08:29:38.097 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:29:38.097 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:29:38.097 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-11T08:29:38.097 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-11T08:29:38.097 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-11T08:29:38.097 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-11T08:29:38.097 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-11T08:29:38.097 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 41344sec preferred_lft 41344sec 2024-01-11T08:29:38.097 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-11T08:29:38.097 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:29:38.097 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:29:38.098 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-11T08:29:38.098 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-11T08:29:38.098 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T08:29:38.098 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff 2024-01-11T08:29:38.098 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-11T08:29:38.098 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:29:38.098 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::acd2:b3ff:fe6a:db4b/64 scope link 2024-01-11T08:29:38.098 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:29:38.098 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T08:29:38.098 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:fa:85:1b:3a:bc brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-11T08:29:38.098 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ecfa:85ff:fe1b:3abc/64 scope link 2024-01-11T08:29:38.098 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:29:38.098 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T08:29:38.098 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-11T08:29:38.098 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ec88:faff:fe17:41aa/64 scope link 2024-01-11T08:29:38.099 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:29:38.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:29:38.099 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-11T08:29:38.169 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-11T08:29:38.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:29:38.173 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-11T08:29:38.223 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-11T08:29:38.223 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-11T08:29:38.223 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-11T08:29:38.223 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-11T08:29:38.224 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-11T08:29:38.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:29:38.225 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T08:29:38.225 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T08:29:38.225 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-11T08:29:38.225 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-11T08:29:38.225 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-11T08:29:38.225 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T08:29:38.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:29:38.402 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T08:29:38.449 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-11T08:29:38.449 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-11T08:29:38.450 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:29:38.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:29:38.451 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-11T08:29:38.524 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:29:38.524 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-11T08:29:38.571 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:29:38.571 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:29:38.571 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-11T08:29:38.618 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-11T08:29:38.618 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-11T08:29:38.619 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:29:38.722 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:29:38.722 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:29:38.722 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:29:38.737 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:29:38.737 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:29:38.737 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:29:38.737 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:29:38.737 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:29:38.757 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:29:38.757 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:29:38.758 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:29:38.758 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:29:38.758 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:29:38.758 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790" 2024-01-11T08:29:38.758 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790:/ 2024-01-11T08:29:38.758 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=8fddb4a7-9d66-40f1-892d-6fe86c728ee4" will pass to kernel 2024-01-11T08:29:38.763 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-11T08:29:38.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:29:38.763 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-11T08:29:38.836 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-11T08:29:39.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.041+0000 7fbd121ff700 1 Processor -- start 2024-01-11T08:29:39.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.041+0000 7fbd121ff700 1 -- start start 2024-01-11T08:29:39.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.042+0000 7fbd121ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbd0c0b0550 0x7fbd0c0ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:39.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.042+0000 7fbd121ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0c0adc80 con 0x7fbd0c156ab0 2024-01-11T08:29:39.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.042+0000 7fbd121ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0c0adf60 con 0x7fbd0c156e80 2024-01-11T08:29:39.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.042+0000 7fbd121ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c0ae240 con 0x7fbd0c0b0550 2024-01-11T08:29:39.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.042+0000 7fbd0b7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbd0c156ab0 0x7fbd0c0cc9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54840/0 (socket says 172.21.15.27:54840) 2024-01-11T08:29:39.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.043+0000 7fbd0b7fe700 1 -- 172.21.15.27:0/2884117175 learned_addr learned my addr 172.21.15.27:0/2884117175 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:39.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.043+0000 7fbd0affd700 1 --2- 172.21.15.27:0/2884117175 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbd0c0b0550 0x7fbd0c0ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:39.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.043+0000 7fbd0a7fc700 1 -- 172.21.15.27:0/2884117175 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868593596 0 0) 0x7fbd0c0adc80 con 0x7fbd0c156ab0 2024-01-11T08:29:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.043+0000 7fbd0a7fc700 1 -- 172.21.15.27:0/2884117175 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcf4003570 con 0x7fbd0c156ab0 2024-01-11T08:29:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.043+0000 7fbd0a7fc700 1 -- 172.21.15.27:0/2884117175 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572615970 0 0) 0x7fbd0c0adf60 con 0x7fbd0c156e80 2024-01-11T08:29:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.043+0000 7fbd0a7fc700 1 -- 172.21.15.27:0/2884117175 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd0c0adc80 con 0x7fbd0c156e80 2024-01-11T08:29:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.043+0000 7fbd0affd700 1 -- 172.21.15.27:0/2884117175 >> v1:172.21.15.77:6789/0 conn(0x7fbd0c156e80 legacy=0x7fbd0c1572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.043+0000 7fbd0affd700 1 -- 172.21.15.27:0/2884117175 >> v1:172.21.15.27:6789/0 conn(0x7fbd0c156ab0 legacy=0x7fbd0c0cc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.043+0000 7fbd0a7fc700 1 -- 172.21.15.27:0/2884117175 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4204633918 0 0) 0x7fbcf4003570 con 0x7fbd0c156ab0 2024-01-11T08:29:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.043+0000 7fbd0affd700 1 -- 172.21.15.27:0/2884117175 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0c0ae520 con 0x7fbd0c0b0550 2024-01-11T08:29:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.044+0000 7fbd0affd700 1 --2- 172.21.15.27:0/2884117175 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbd0c0b0550 0x7fbd0c0ad650 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbd00009910 tx=0x7fbd000091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60c76494021c7a4c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.044+0000 7fbd0a7fc700 1 -- 172.21.15.27:0/2884117175 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc001470 con 0x7fbd0c0b0550 2024-01-11T08:29:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.044+0000 7fbd0a7fc700 1 -- 172.21.15.27:0/2884117175 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd00011420 con 0x7fbd0c0b0550 2024-01-11T08:29:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.044+0000 7fbd0a7fc700 1 -- 172.21.15.27:0/2884117175 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd000117c0 con 0x7fbd0c0b0550 2024-01-11T08:29:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.044+0000 7fbd121ff700 1 -- 172.21.15.27:0/2884117175 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbd0c0b0550 msgr2=0x7fbd0c0ad650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:39.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.044+0000 7fbd121ff700 1 --2- 172.21.15.27:0/2884117175 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbd0c0b0550 0x7fbd0c0ad650 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbd00009910 tx=0x7fbd000091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:39.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.044+0000 7fbd121ff700 1 -- 172.21.15.27:0/2884117175 shutdown_connections 2024-01-11T08:29:39.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.044+0000 7fbd121ff700 1 --2- 172.21.15.27:0/2884117175 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbd0c0b0550 0x7fbd0c0ad650 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:39.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.044+0000 7fbd121ff700 1 -- 172.21.15.27:0/2884117175 >> 172.21.15.27:0/2884117175 conn(0x7fbd0c0680b0 msgr2=0x7fbd0c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:39.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.044+0000 7fbd121ff700 1 -- 172.21.15.27:0/2884117175 shutdown_connections 2024-01-11T08:29:39.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.045+0000 7fbd121ff700 1 -- 172.21.15.27:0/2884117175 wait complete. 2024-01-11T08:29:39.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.046+0000 7fbd121ff700 1 Processor -- start 2024-01-11T08:29:39.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.046+0000 7fbd121ff700 1 -- start start 2024-01-11T08:29:39.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.047+0000 7fbd121ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c0b0550 0x7fbd0c14b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:39.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.047+0000 7fbd121ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbd0c156ab0 0x7fbd0c16e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:39.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.047+0000 7fbd121ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd0c156e80 0x7fbd0c0cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:39.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.047+0000 7fbd121ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c16e790 con 0x7fbd0c0b0550 2024-01-11T08:29:39.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.047+0000 7fbd121ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c15cb10 con 0x7fbd0c156e80 2024-01-11T08:29:39.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.047+0000 7fbd121ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c15cd60 con 0x7fbd0c156ab0 2024-01-11T08:29:39.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.047+0000 7fbd0b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c0b0550 0x7fbd0c14b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:39.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.047+0000 7fbd0b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c0b0550 0x7fbd0c14b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36874/0 (socket says 172.21.15.27:36874) 2024-01-11T08:29:39.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.047+0000 7fbd0b7fe700 1 -- 172.21.15.27:0/4192844760 learned_addr learned my addr 172.21.15.27:0/4192844760 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:39.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.048+0000 7fbd0bfff700 1 --2- 172.21.15.27:0/4192844760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd0c156e80 0x7fbd0c0cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:39.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.048+0000 7fbd0affd700 1 --2- 172.21.15.27:0/4192844760 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbd0c156ab0 0x7fbd0c16e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:39.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.048+0000 7fbd0b7fe700 1 -- 172.21.15.27:0/4192844760 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbd0c156ab0 msgr2=0x7fbd0c16e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:39.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.048+0000 7fbd0b7fe700 1 --2- 172.21.15.27:0/4192844760 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbd0c156ab0 0x7fbd0c16e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:39.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.048+0000 7fbd0b7fe700 1 -- 172.21.15.27:0/4192844760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd0c156e80 msgr2=0x7fbd0c0cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:39.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.048+0000 7fbd0b7fe700 1 --2- 172.21.15.27:0/4192844760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd0c156e80 0x7fbd0c0cb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:39.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.048+0000 7fbd0b7fe700 1 -- 172.21.15.27:0/4192844760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd00009420 con 0x7fbd0c0b0550 2024-01-11T08:29:39.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.048+0000 7fbd0bfff700 1 --2- 172.21.15.27:0/4192844760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd0c156e80 0x7fbd0c0cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:39.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.048+0000 7fbd0b7fe700 1 --2- 172.21.15.27:0/4192844760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c0b0550 0x7fbd0c14b000 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00cb70 tx=0x7fbcfc001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:39.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.048+0000 7fbcf37fe700 1 -- 172.21.15.27:0/4192844760 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc001d20 con 0x7fbd0c0b0550 2024-01-11T08:29:39.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.048+0000 7fbcf37fe700 1 -- 172.21.15.27:0/4192844760 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcfc007360 con 0x7fbd0c0b0550 2024-01-11T08:29:39.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.048+0000 7fbcf37fe700 1 -- 172.21.15.27:0/4192844760 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc004830 con 0x7fbd0c0b0550 2024-01-11T08:29:39.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.048+0000 7fbd121ff700 1 -- 172.21.15.27:0/4192844760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd0c15d040 con 0x7fbd0c0b0550 2024-01-11T08:29:39.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.049+0000 7fbd121ff700 1 -- 172.21.15.27:0/4192844760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd0c0c8dd0 con 0x7fbd0c0b0550 2024-01-11T08:29:39.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.049+0000 7fbd121ff700 1 -- 172.21.15.27:0/4192844760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd0c14d110 con 0x7fbd0c0b0550 2024-01-11T08:29:39.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.050+0000 7fbcf37fe700 1 -- 172.21.15.27:0/4192844760 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fbcfc004ba0 con 0x7fbd0c0b0550 2024-01-11T08:29:39.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.051+0000 7fbcf37fe700 1 --2- 172.21.15.27:0/4192844760 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbcf8066f30 0x7fbcf80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:39.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.051+0000 7fbcf37fe700 1 -- 172.21.15.27:0/4192844760 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fbcfc083960 con 0x7fbd0c0b0550 2024-01-11T08:29:39.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.052+0000 7fbd0affd700 1 --2- 172.21.15.27:0/4192844760 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbcf8066f30 0x7fbcf80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:39.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.053+0000 7fbd0affd700 1 --2- 172.21.15.27:0/4192844760 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbcf8066f30 0x7fbcf80693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbd000030c0 tx=0x7fbd00016000 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:39.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.056+0000 7fbcf37fe700 1 -- 172.21.15.27:0/4192844760 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbcfc050c10 con 0x7fbd0c0b0550 2024-01-11T08:29:39.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.290+0000 7fbd121ff700 1 -- 172.21.15.27:0/4192844760 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd0c04c3c0 con 0x7fbcf8066f30 2024-01-11T08:29:39.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.292+0000 7fbcf37fe700 1 -- 172.21.15.27:0/4192844760 <== mgr.7591 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fbd0c04c3c0 con 0x7fbcf8066f30 2024-01-11T08:29:39.291 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T08:29:39.291 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:29:39.292 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-11T08:29:39.292 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:29:39.292 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T08:29:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.294+0000 7fbd121ff700 1 -- 172.21.15.27:0/4192844760 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbcf8066f30 msgr2=0x7fbcf80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.294+0000 7fbd121ff700 1 --2- 172.21.15.27:0/4192844760 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbcf8066f30 0x7fbcf80693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbd000030c0 tx=0x7fbd00016000 comp rx=0 tx=0).stop 2024-01-11T08:29:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.295+0000 7fbd121ff700 1 -- 172.21.15.27:0/4192844760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c0b0550 msgr2=0x7fbd0c14b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.295+0000 7fbd121ff700 1 --2- 172.21.15.27:0/4192844760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c0b0550 0x7fbd0c14b000 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00cb70 tx=0x7fbcfc001ab0 comp rx=0 tx=0).stop 2024-01-11T08:29:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.295+0000 7fbd121ff700 1 -- 172.21.15.27:0/4192844760 shutdown_connections 2024-01-11T08:29:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.295+0000 7fbd121ff700 1 --2- 172.21.15.27:0/4192844760 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fbcf8066f30 0x7fbcf80693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.295+0000 7fbd121ff700 1 --2- 172.21.15.27:0/4192844760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd0c156e80 0x7fbd0c0cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.295+0000 7fbd121ff700 1 --2- 172.21.15.27:0/4192844760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c0b0550 0x7fbd0c14b000 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.295+0000 7fbd121ff700 1 --2- 172.21.15.27:0/4192844760 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbd0c156ab0 0x7fbd0c16e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.295+0000 7fbd121ff700 1 -- 172.21.15.27:0/4192844760 >> 172.21.15.27:0/4192844760 conn(0x7fbd0c0680b0 msgr2=0x7fbd0c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.295+0000 7fbd121ff700 1 -- 172.21.15.27:0/4192844760 shutdown_connections 2024-01-11T08:29:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.295+0000 7fbd121ff700 1 -- 172.21.15.27:0/4192844760 wait complete. 2024-01-11T08:29:39.312 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-11T08:29:39.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.515+0000 7f6d5efff700 1 Processor -- start 2024-01-11T08:29:39.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.515+0000 7f6d5efff700 1 -- start start 2024-01-11T08:29:39.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.516+0000 7f6d5efff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6d580cd220 0x7f6d5815d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:39.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.516+0000 7f6d5efff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d5815d860 0x7f6d581591e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:39.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.516+0000 7f6d5efff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6d580cd5f0 0x7f6d58159720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:39.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.516+0000 7f6d5efff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d5815aef0 con 0x7f6d580cd220 2024-01-11T08:29:39.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.516+0000 7f6d5efff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d580cee00 con 0x7f6d580cd5f0 2024-01-11T08:29:39.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.516+0000 7f6d5efff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5815dd70 con 0x7f6d5815d860 2024-01-11T08:29:39.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.517+0000 7f6d5cd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6d580cd220 0x7f6d5815d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:39.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.517+0000 7f6d5cd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6d580cd220 0x7f6d5815d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36876/0 (socket says 172.21.15.27:36876) 2024-01-11T08:29:39.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.517+0000 7f6d5cd9b700 1 -- 172.21.15.27:0/488416467 learned_addr learned my addr 172.21.15.27:0/488416467 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:39.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.517+0000 7f6d57fff700 1 --2- 172.21.15.27:0/488416467 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d5815d860 0x7f6d581591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:39.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.517+0000 7f6d5d59c700 1 --2- 172.21.15.27:0/488416467 >> v2:172.21.15.77:3300/0 conn(0x7f6d580cd5f0 0x7f6d58159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:39.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.517+0000 7f6d5cd9b700 1 -- 172.21.15.27:0/488416467 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d5815d860 msgr2=0x7f6d581591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:39.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.517+0000 7f6d5cd9b700 1 --2- 172.21.15.27:0/488416467 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d5815d860 0x7f6d581591e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:39.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.517+0000 7f6d5cd9b700 1 -- 172.21.15.27:0/488416467 >> v2:172.21.15.77:3300/0 conn(0x7f6d580cd5f0 msgr2=0x7f6d58159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:39.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.517+0000 7f6d5cd9b700 1 --2- 172.21.15.27:0/488416467 >> v2:172.21.15.77:3300/0 conn(0x7f6d580cd5f0 0x7f6d58159720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:39.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.517+0000 7f6d5cd9b700 1 -- 172.21.15.27:0/488416467 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d5815e050 con 0x7f6d580cd220 2024-01-11T08:29:39.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.517+0000 7f6d57fff700 1 --2- 172.21.15.27:0/488416467 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d5815d860 0x7f6d581591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:39.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.518+0000 7f6d5cd9b700 1 --2- 172.21.15.27:0/488416467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d580cd220 0x7f6d5815d320 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f6d580895a0 tx=0x7f6d4c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c919ef83e71326ed server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:39.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.518+0000 7f6d577fe700 1 -- 172.21.15.27:0/488416467 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c009e70 con 0x7f6d580cd220 2024-01-11T08:29:39.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.518+0000 7f6d577fe700 1 -- 172.21.15.27:0/488416467 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4c00b860 con 0x7f6d580cd220 2024-01-11T08:29:39.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.518+0000 7f6d577fe700 1 -- 172.21.15.27:0/488416467 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c00bc00 con 0x7f6d580cd220 2024-01-11T08:29:39.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.518+0000 7f6d5efff700 1 -- 172.21.15.27:0/488416467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d580cd220 msgr2=0x7f6d5815d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:39.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.518+0000 7f6d5efff700 1 --2- 172.21.15.27:0/488416467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d580cd220 0x7f6d5815d320 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f6d580895a0 tx=0x7f6d4c009250 comp rx=0 tx=0).stop 2024-01-11T08:29:39.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.518+0000 7f6d5efff700 1 -- 172.21.15.27:0/488416467 shutdown_connections 2024-01-11T08:29:39.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.518+0000 7f6d5efff700 1 --2- 172.21.15.27:0/488416467 >> v2:172.21.15.77:3300/0 conn(0x7f6d580cd5f0 0x7f6d58159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:39.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.518+0000 7f6d5efff700 1 --2- 172.21.15.27:0/488416467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d580cd220 0x7f6d5815d320 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:39.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.518+0000 7f6d5efff700 1 --2- 172.21.15.27:0/488416467 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d5815d860 0x7f6d581591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:39.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.518+0000 7f6d5efff700 1 -- 172.21.15.27:0/488416467 >> 172.21.15.27:0/488416467 conn(0x7f6d5806ed40 msgr2=0x7f6d5806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:39.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.519+0000 7f6d5efff700 1 -- 172.21.15.27:0/488416467 shutdown_connections 2024-01-11T08:29:39.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.519+0000 7f6d5efff700 1 -- 172.21.15.27:0/488416467 wait complete. 2024-01-11T08:29:39.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.520+0000 7f6d5efff700 1 Processor -- start 2024-01-11T08:29:39.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.520+0000 7f6d5efff700 1 -- start start 2024-01-11T08:29:39.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.521+0000 7f6d5efff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d580cd220 0x7f6d58150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:39.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.521+0000 7f6d5efff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d580cd5f0 0x7f6d580c8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:39.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.521+0000 7f6d5efff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d5815d860 0x7f6d58151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:39.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.521+0000 7f6d5efff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d580ca570 con 0x7f6d580cd220 2024-01-11T08:29:39.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.521+0000 7f6d5efff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d58169c70 con 0x7f6d580cd5f0 2024-01-11T08:29:39.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.521+0000 7f6d5efff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d580b4b70 con 0x7f6d5815d860 2024-01-11T08:29:39.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.521+0000 7f6d5cd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d580cd220 0x7f6d58150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:39.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.522+0000 7f6d5cd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d580cd220 0x7f6d58150b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36882/0 (socket says 172.21.15.27:36882) 2024-01-11T08:29:39.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.522+0000 7f6d5cd9b700 1 -- 172.21.15.27:0/2388961567 learned_addr learned my addr 172.21.15.27:0/2388961567 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:39.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.522+0000 7f6d57fff700 1 --2- 172.21.15.27:0/2388961567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d580cd5f0 0x7f6d580c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:39.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.522+0000 7f6d5d59c700 1 --2- 172.21.15.27:0/2388961567 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d5815d860 0x7f6d58151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:39.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.522+0000 7f6d5cd9b700 1 -- 172.21.15.27:0/2388961567 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d5815d860 msgr2=0x7f6d58151a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:39.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.522+0000 7f6d5cd9b700 1 --2- 172.21.15.27:0/2388961567 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d5815d860 0x7f6d58151a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:39.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.522+0000 7f6d5cd9b700 1 -- 172.21.15.27:0/2388961567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d580cd5f0 msgr2=0x7f6d580c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:39.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.522+0000 7f6d5cd9b700 1 --2- 172.21.15.27:0/2388961567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d580cd5f0 0x7f6d580c8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:39.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.522+0000 7f6d5cd9b700 1 -- 172.21.15.27:0/2388961567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4c0094f0 con 0x7f6d580cd220 2024-01-11T08:29:39.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.522+0000 7f6d5cd9b700 1 --2- 172.21.15.27:0/2388961567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d580cd220 0x7f6d58150b20 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c009290 tx=0x7f6d4c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:39.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.523+0000 7f6d54ff9700 1 -- 172.21.15.27:0/2388961567 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c00bbe0 con 0x7f6d580cd220 2024-01-11T08:29:39.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.523+0000 7f6d54ff9700 1 -- 172.21.15.27:0/2388961567 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4c000c90 con 0x7f6d580cd220 2024-01-11T08:29:39.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.523+0000 7f6d5efff700 1 -- 172.21.15.27:0/2388961567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d580b4840 con 0x7f6d580cd220 2024-01-11T08:29:39.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.523+0000 7f6d54ff9700 1 -- 172.21.15.27:0/2388961567 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c01a680 con 0x7f6d580cd220 2024-01-11T08:29:39.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.523+0000 7f6d5efff700 1 -- 172.21.15.27:0/2388961567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d580c9fb0 con 0x7f6d580cd220 2024-01-11T08:29:39.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.524+0000 7f6d5efff700 1 -- 172.21.15.27:0/2388961567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d5804c3c0 con 0x7f6d580cd220 2024-01-11T08:29:39.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.525+0000 7f6d54ff9700 1 -- 172.21.15.27:0/2388961567 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 68) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6d4c01aa30 con 0x7f6d580cd220 2024-01-11T08:29:39.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.526+0000 7f6d54ff9700 1 --2- 172.21.15.27:0/2388961567 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6d3c066f30 0x7f6d3c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:39.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.526+0000 7f6d54ff9700 1 -- 172.21.15.27:0/2388961567 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f6d4c085b30 con 0x7f6d580cd220 2024-01-11T08:29:39.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.527+0000 7f6d57fff700 1 --2- 172.21.15.27:0/2388961567 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6d3c066f30 0x7f6d3c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:39.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.527+0000 7f6d57fff700 1 --2- 172.21.15.27:0/2388961567 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6d3c066f30 0x7f6d3c0693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6d580afff0 tx=0x7f6d44008040 comp rx=0 tx=0).ready entity=mgr.7591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:39.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.531+0000 7f6d54ff9700 1 -- 172.21.15.27:0/2388961567 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6d4c052cc0 con 0x7f6d580cd220 2024-01-11T08:29:39.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:39.792+0000 7f6d5efff700 1 -- 172.21.15.27:0/2388961567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f6d580ce380 con 0x7f6d580cd220 2024-01-11T08:29:40.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:40.780+0000 7f6d54ff9700 1 -- 172.21.15.27:0/2388961567 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 69) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f6d4c01ace0 con 0x7f6d580cd220 2024-01-11T08:29:40.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:40.782+0000 7f6d54ff9700 1 -- 172.21.15.27:0/2388961567 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v69) v1 ==== 92+0+0 (secure 0 0 0) 0x7f6d4c056750 con 0x7f6d580cd220 2024-01-11T08:29:40.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:40.786+0000 7f6d5efff700 1 -- 172.21.15.27:0/2388961567 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6d3c066f30 msgr2=0x7f6d3c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:40.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:40.786+0000 7f6d5efff700 1 --2- 172.21.15.27:0/2388961567 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6d3c066f30 0x7f6d3c0693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6d580afff0 tx=0x7f6d44008040 comp rx=0 tx=0).stop 2024-01-11T08:29:40.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:40.786+0000 7f6d5efff700 1 -- 172.21.15.27:0/2388961567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d580cd220 msgr2=0x7f6d58150b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:40.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:40.786+0000 7f6d5efff700 1 --2- 172.21.15.27:0/2388961567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d580cd220 0x7f6d58150b20 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c009290 tx=0x7f6d4c017000 comp rx=0 tx=0).stop 2024-01-11T08:29:40.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:40.786+0000 7f6d5efff700 1 -- 172.21.15.27:0/2388961567 shutdown_connections 2024-01-11T08:29:40.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:40.786+0000 7f6d5efff700 1 --2- 172.21.15.27:0/2388961567 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6d3c066f30 0x7f6d3c0693e0 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6d580afff0 tx=0x7f6d44008040 comp rx=0 tx=0).stop 2024-01-11T08:29:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:40.786+0000 7f6d5efff700 1 --2- 172.21.15.27:0/2388961567 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6d5815d860 0x7f6d58151a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:40.786+0000 7f6d5efff700 1 --2- 172.21.15.27:0/2388961567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d580cd220 0x7f6d58150b20 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:40.786+0000 7f6d5efff700 1 --2- 172.21.15.27:0/2388961567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d580cd5f0 0x7f6d580c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:40.786+0000 7f6d5efff700 1 -- 172.21.15.27:0/2388961567 >> 172.21.15.27:0/2388961567 conn(0x7f6d5806ed40 msgr2=0x7f6d5815a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:40.786+0000 7f6d5efff700 1 -- 172.21.15.27:0/2388961567 shutdown_connections 2024-01-11T08:29:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:40.786+0000 7f6d5efff700 1 -- 172.21.15.27:0/2388961567 wait complete. 2024-01-11T08:29:40.808 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-11T08:29:40.994 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:40.995+0000 7f5dc4c681c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:29:41.005 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:41.006+0000 7ff9fcd311c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:29:41.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.021+0000 7efdee1ff700 1 Processor -- start 2024-01-11T08:29:41.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.021+0000 7efdee1ff700 1 -- start start 2024-01-11T08:29:41.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.023+0000 7efdee1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efde813fbb0 0x7efde80ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:41.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.023+0000 7efdee1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde80c6b30 con 0x7efde815a730 2024-01-11T08:29:41.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.023+0000 7efdee1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde80c6800 con 0x7efde813ff80 2024-01-11T08:29:41.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.023+0000 7efdee1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde80cb240 con 0x7efde813fbb0 2024-01-11T08:29:41.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.024+0000 7efde77fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efde813fbb0 0x7efde80ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:41.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.024+0000 7efde6ffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7efde813ff80 0x7efde815a020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:50460/0 (socket says 172.21.15.27:50460) 2024-01-11T08:29:41.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.024+0000 7efde6ffd700 1 -- 172.21.15.27:0/3610990469 learned_addr learned my addr 172.21.15.27:0/3610990469 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:41.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.024+0000 7efde7fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efde815a730 0x7efde80c6fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54846/0 (socket says 172.21.15.27:54846) 2024-01-11T08:29:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.024+0000 7efde67fc700 1 -- 172.21.15.27:0/3610990469 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559207587 0 0) 0x7efde80c6800 con 0x7efde813ff80 2024-01-11T08:29:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.024+0000 7efde77fe700 1 -- 172.21.15.27:0/3610990469 >> v1:172.21.15.77:6789/0 conn(0x7efde813ff80 legacy=0x7efde815a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.024+0000 7efde77fe700 1 -- 172.21.15.27:0/3610990469 >> v1:172.21.15.27:6789/0 conn(0x7efde815a730 legacy=0x7efde80c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.024+0000 7efde77fe700 1 -- 172.21.15.27:0/3610990469 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde80cb520 con 0x7efde813fbb0 2024-01-11T08:29:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.025+0000 7efde77fe700 1 --2- 172.21.15.27:0/3610990469 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efde813fbb0 0x7efde80ca7a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efdd8009900 tx=0x7efdd8009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=559f61c8cf1d7d87 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.025+0000 7efde67fc700 1 -- 172.21.15.27:0/3610990469 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd8009e40 con 0x7efde813fbb0 2024-01-11T08:29:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.025+0000 7efde67fc700 1 -- 172.21.15.27:0/3610990469 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd800b760 con 0x7efde813fbb0 2024-01-11T08:29:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.025+0000 7efde67fc700 1 -- 172.21.15.27:0/3610990469 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd800bb00 con 0x7efde813fbb0 2024-01-11T08:29:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.025+0000 7efdee1ff700 1 -- 172.21.15.27:0/3610990469 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efde813fbb0 msgr2=0x7efde80ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.025+0000 7efdee1ff700 1 --2- 172.21.15.27:0/3610990469 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efde813fbb0 0x7efde80ca7a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efdd8009900 tx=0x7efdd8009120 comp rx=0 tx=0).stop 2024-01-11T08:29:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.025+0000 7efdee1ff700 1 -- 172.21.15.27:0/3610990469 shutdown_connections 2024-01-11T08:29:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.025+0000 7efdee1ff700 1 --2- 172.21.15.27:0/3610990469 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efde813fbb0 0x7efde80ca7a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.025+0000 7efdee1ff700 1 -- 172.21.15.27:0/3610990469 >> 172.21.15.27:0/3610990469 conn(0x7efde80680a0 msgr2=0x7efde80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.026+0000 7efdee1ff700 1 -- 172.21.15.27:0/3610990469 shutdown_connections 2024-01-11T08:29:41.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.026+0000 7efdee1ff700 1 -- 172.21.15.27:0/3610990469 wait complete. 2024-01-11T08:29:41.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.028+0000 7efdee1ff700 1 Processor -- start 2024-01-11T08:29:41.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.028+0000 7efdee1ff700 1 -- start start 2024-01-11T08:29:41.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.029+0000 7efdee1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efde813fbb0 0x7efde8156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:41.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.029+0000 7efdee1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde813ff80 0x7efde8159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:41.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.029+0000 7efdee1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efde815a730 0x7efde813c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:41.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.029+0000 7efdee1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde8157a20 con 0x7efde815a730 2024-01-11T08:29:41.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.029+0000 7efdee1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde80af400 con 0x7efde813ff80 2024-01-11T08:29:41.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.029+0000 7efdee1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde81574a0 con 0x7efde813fbb0 2024-01-11T08:29:41.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.029+0000 7efde77fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efde813fbb0 0x7efde8156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:41.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.030+0000 7efde77fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efde813fbb0 0x7efde8156b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:41270/0 (socket says 172.21.15.27:41270) 2024-01-11T08:29:41.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.030+0000 7efde77fe700 1 -- 172.21.15.27:0/3232227410 learned_addr learned my addr 172.21.15.27:0/3232227410 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:41.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.030+0000 7efde6ffd700 1 --2- 172.21.15.27:0/3232227410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde813ff80 0x7efde8159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:41.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.030+0000 7efde7fff700 1 --2- 172.21.15.27:0/3232227410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efde815a730 0x7efde813c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:41.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.030+0000 7efde77fe700 1 -- 172.21.15.27:0/3232227410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde813ff80 msgr2=0x7efde8159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:41.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.030+0000 7efde77fe700 1 --2- 172.21.15.27:0/3232227410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde813ff80 0x7efde8159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:41.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.030+0000 7efde77fe700 1 -- 172.21.15.27:0/3232227410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efde815a730 msgr2=0x7efde813c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:41.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.030+0000 7efde77fe700 1 --2- 172.21.15.27:0/3232227410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efde815a730 0x7efde813c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:41.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.030+0000 7efde77fe700 1 -- 172.21.15.27:0/3232227410 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd80093c0 con 0x7efde813fbb0 2024-01-11T08:29:41.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.030+0000 7efde77fe700 1 --2- 172.21.15.27:0/3232227410 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efde813fbb0 0x7efde8156b00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efdd8009160 tx=0x7efdd8016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:41.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.030+0000 7efdcf7fe700 1 -- 172.21.15.27:0/3232227410 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd801c480 con 0x7efde813fbb0 2024-01-11T08:29:41.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.030+0000 7efdcf7fe700 1 -- 172.21.15.27:0/3232227410 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd801c5e0 con 0x7efde813fbb0 2024-01-11T08:29:41.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.031+0000 7efdee1ff700 1 -- 172.21.15.27:0/3232227410 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde8157740 con 0x7efde813fbb0 2024-01-11T08:29:41.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.031+0000 7efdcf7fe700 1 -- 172.21.15.27:0/3232227410 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd801c980 con 0x7efde813fbb0 2024-01-11T08:29:41.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.031+0000 7efdee1ff700 1 -- 172.21.15.27:0/3232227410 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde813c680 con 0x7efde813fbb0 2024-01-11T08:29:41.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.032+0000 7efdee1ff700 1 -- 172.21.15.27:0/3232227410 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde804c3c0 con 0x7efde813fbb0 2024-01-11T08:29:41.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.033+0000 7efdcf7fe700 1 -- 172.21.15.27:0/3232227410 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 69) v1 ==== 84759+0+0 (secure 0 0 0) 0x7efdd80193f0 con 0x7efde813fbb0 2024-01-11T08:29:41.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.033+0000 7efdcf7fe700 1 --2- 172.21.15.27:0/3232227410 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efdd4066f80 0x7efdd4069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:41.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.034+0000 7efdcf7fe700 1 -- 172.21.15.27:0/3232227410 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5761+0+0 (secure 0 0 0) 0x7efdd8085fa0 con 0x7efde813fbb0 2024-01-11T08:29:41.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.034+0000 7efde6ffd700 1 -- 172.21.15.27:0/3232227410 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efdd4066f80 msgr2=0x7efdd4069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:41.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.034+0000 7efde6ffd700 1 --2- 172.21.15.27:0/3232227410 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efdd4066f80 0x7efdd4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:41.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.038+0000 7efdcf7fe700 1 -- 172.21.15.27:0/3232227410 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efdd80531b0 con 0x7efde813fbb0 2024-01-11T08:29:41.173 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:41.174+0000 7f5dc4c681c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:29:41.208 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:41.210+0000 7ff9fcd311c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:29:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.235+0000 7efde6ffd700 1 -- 172.21.15.27:0/3232227410 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efdd4066f80 msgr2=0x7efdd4069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.235+0000 7efde6ffd700 1 --2- 172.21.15.27:0/3232227410 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efdd4066f80 0x7efdd4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:41.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.283+0000 7efdee1ff700 1 -- 172.21.15.27:0/3232227410 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7efde80cb800 con 0x7efde813fbb0 2024-01-11T08:29:41.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.290+0000 7efdcf7fe700 1 -- 172.21.15.27:0/3232227410 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v47)=0 v47) v1 ==== 138+0+0 (secure 0 0 0) 0x7efdd80567d0 con 0x7efde813fbb0 2024-01-11T08:29:41.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.293+0000 7efdee1ff700 1 -- 172.21.15.27:0/3232227410 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efdd4066f80 msgr2=0x7efdd4069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:41.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.293+0000 7efdee1ff700 1 --2- 172.21.15.27:0/3232227410 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efdd4066f80 0x7efdd4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:41.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.293+0000 7efdee1ff700 1 -- 172.21.15.27:0/3232227410 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efde813fbb0 msgr2=0x7efde8156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:41.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.293+0000 7efdee1ff700 1 --2- 172.21.15.27:0/3232227410 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efde813fbb0 0x7efde8156b00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efdd8009160 tx=0x7efdd8016000 comp rx=0 tx=0).stop 2024-01-11T08:29:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.293+0000 7efdee1ff700 1 -- 172.21.15.27:0/3232227410 shutdown_connections 2024-01-11T08:29:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.293+0000 7efdee1ff700 1 --2- 172.21.15.27:0/3232227410 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efdd4066f80 0x7efdd4069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.293+0000 7efdee1ff700 1 --2- 172.21.15.27:0/3232227410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efde815a730 0x7efde813c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.293+0000 7efdee1ff700 1 --2- 172.21.15.27:0/3232227410 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efde813fbb0 0x7efde8156b00 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.293+0000 7efdee1ff700 1 --2- 172.21.15.27:0/3232227410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde813ff80 0x7efde8159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.293+0000 7efdee1ff700 1 -- 172.21.15.27:0/3232227410 >> 172.21.15.27:0/3232227410 conn(0x7efde80680a0 msgr2=0x7efde813df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.293+0000 7efdee1ff700 1 -- 172.21.15.27:0/3232227410 shutdown_connections 2024-01-11T08:29:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.293+0000 7efdee1ff700 1 -- 172.21.15.27:0/3232227410 wait complete. 2024-01-11T08:29:41.311 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-11T08:29:41.490 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:41.491+0000 7f5dc4c681c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:29:41.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.517+0000 7fe0f81ff700 1 Processor -- start 2024-01-11T08:29:41.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.517+0000 7fe0f81ff700 1 -- start start 2024-01-11T08:29:41.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.518+0000 7fe0f81ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0f013fbb0 0x7fe0f00ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:41.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.518+0000 7fe0f81ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe0f015a730 0x7fe0f00c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:41.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.518+0000 7fe0f81ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f00c6b30 con 0x7fe0f013ff80 2024-01-11T08:29:41.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.518+0000 7fe0f81ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0f00c6800 con 0x7fe0f015a730 2024-01-11T08:29:41.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.518+0000 7fe0f81ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f00cb240 con 0x7fe0f013fbb0 2024-01-11T08:29:41.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.519+0000 7fe0f5f9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0f013fbb0 0x7fe0f00ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:41.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.519+0000 7fe0f679c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe0f015a730 0x7fe0f00c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:41.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.519+0000 7fe0f5f9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0f013fbb0 0x7fe0f00ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:41284/0 (socket says 172.21.15.27:41284) 2024-01-11T08:29:41.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.519+0000 7fe0f679c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe0f015a730 0x7fe0f00c6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:60456/0 (socket says 172.21.15.27:60456) 2024-01-11T08:29:41.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.519+0000 7fe0f579a700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe0f013ff80 0x7fe0f015a020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:50470/0 (socket says 172.21.15.27:50470) 2024-01-11T08:29:41.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.519+0000 7fe0f679c700 1 -- 172.21.15.27:0/1865312936 learned_addr learned my addr 172.21.15.27:0/1865312936 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:41.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.519+0000 7fe0f5f9b700 1 -- 172.21.15.27:0/1865312936 >> v2:172.21.15.77:3300/0 conn(0x7fe0f015a730 msgr2=0x7fe0f00c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:41.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.519+0000 7fe0f5f9b700 1 --2- 172.21.15.27:0/1865312936 >> v2:172.21.15.77:3300/0 conn(0x7fe0f015a730 0x7fe0f00c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:41.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.519+0000 7fe0f5f9b700 1 -- 172.21.15.27:0/1865312936 >> v1:172.21.15.77:6789/0 conn(0x7fe0f013ff80 legacy=0x7fe0f015a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:41.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.519+0000 7fe0f5f9b700 1 -- 172.21.15.27:0/1865312936 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f00cb520 con 0x7fe0f013fbb0 2024-01-11T08:29:41.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.519+0000 7fe0f679c700 1 --2- 172.21.15.27:0/1865312936 >> v2:172.21.15.77:3300/0 conn(0x7fe0f015a730 0x7fe0f00c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:41.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.519+0000 7fe0f4f99700 1 -- 172.21.15.27:0/1865312936 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598412197 0 0) 0x7fe0f00c6b30 con 0x7fe0f013ff80 2024-01-11T08:29:41.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.520+0000 7fe0f5f9b700 1 --2- 172.21.15.27:0/1865312936 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0f013fbb0 0x7fe0f00ca7a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0098d0 tx=0x7fe0ec009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33a2f67b513d2a77 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.520+0000 7fe0f4f99700 1 -- 172.21.15.27:0/1865312936 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec005c10 con 0x7fe0f013fbb0 2024-01-11T08:29:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.520+0000 7fe0f4f99700 1 -- 172.21.15.27:0/1865312936 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0ec005d70 con 0x7fe0f013fbb0 2024-01-11T08:29:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.520+0000 7fe0f4f99700 1 -- 172.21.15.27:0/1865312936 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec006280 con 0x7fe0f013fbb0 2024-01-11T08:29:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.521+0000 7fe0f81ff700 1 -- 172.21.15.27:0/1865312936 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0f013fbb0 msgr2=0x7fe0f00ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.521+0000 7fe0f81ff700 1 --2- 172.21.15.27:0/1865312936 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0f013fbb0 0x7fe0f00ca7a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0098d0 tx=0x7fe0ec009120 comp rx=0 tx=0).stop 2024-01-11T08:29:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.521+0000 7fe0f81ff700 1 -- 172.21.15.27:0/1865312936 shutdown_connections 2024-01-11T08:29:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.521+0000 7fe0f81ff700 1 --2- 172.21.15.27:0/1865312936 >> v2:172.21.15.77:3300/0 conn(0x7fe0f015a730 0x7fe0f00c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.521+0000 7fe0f81ff700 1 --2- 172.21.15.27:0/1865312936 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0f013fbb0 0x7fe0f00ca7a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.521+0000 7fe0f81ff700 1 -- 172.21.15.27:0/1865312936 >> 172.21.15.27:0/1865312936 conn(0x7fe0f00680a0 msgr2=0x7fe0f00cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.521+0000 7fe0f81ff700 1 -- 172.21.15.27:0/1865312936 shutdown_connections 2024-01-11T08:29:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.521+0000 7fe0f81ff700 1 -- 172.21.15.27:0/1865312936 wait complete. 2024-01-11T08:29:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.523+0000 7fe0f81ff700 1 Processor -- start 2024-01-11T08:29:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.523+0000 7fe0f81ff700 1 -- start start 2024-01-11T08:29:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.525+0000 7fe0f81ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0f013fbb0 0x7fe0f00b1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.525+0000 7fe0f81ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0f013ff80 0x7fe0f013be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.525+0000 7fe0f81ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0f015a730 0x7fe0f013d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.525+0000 7fe0f81ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f00c4590 con 0x7fe0f013ff80 2024-01-11T08:29:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.525+0000 7fe0f81ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f00c2660 con 0x7fe0f013fbb0 2024-01-11T08:29:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.525+0000 7fe0f81ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f00c4720 con 0x7fe0f015a730 2024-01-11T08:29:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.525+0000 7fe0f579a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0f013ff80 0x7fe0f013be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.525+0000 7fe0f5f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0f013fbb0 0x7fe0f00b1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.525+0000 7fe0f579a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0f013ff80 0x7fe0f013be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36910/0 (socket says 172.21.15.27:36910) 2024-01-11T08:29:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.525+0000 7fe0f579a700 1 -- 172.21.15.27:0/1576311898 learned_addr learned my addr 172.21.15.27:0/1576311898 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.525+0000 7fe0f679c700 1 --2- 172.21.15.27:0/1576311898 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0f015a730 0x7fe0f013d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.525+0000 7fe0f579a700 1 -- 172.21.15.27:0/1576311898 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0f015a730 msgr2=0x7fe0f013d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.525+0000 7fe0f579a700 1 --2- 172.21.15.27:0/1576311898 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0f015a730 0x7fe0f013d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.525+0000 7fe0f579a700 1 -- 172.21.15.27:0/1576311898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0f013fbb0 msgr2=0x7fe0f00b1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.525+0000 7fe0f579a700 1 --2- 172.21.15.27:0/1576311898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0f013fbb0 0x7fe0f00b1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.525+0000 7fe0f579a700 1 -- 172.21.15.27:0/1576311898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e8009ae0 con 0x7fe0f013ff80 2024-01-11T08:29:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.526+0000 7fe0f579a700 1 --2- 172.21.15.27:0/1576311898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0f013ff80 0x7fe0f013be10 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00011e0 tx=0x7fe0e0007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.526+0000 7fe0e5ffb700 1 -- 172.21.15.27:0/1576311898 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e0005a70 con 0x7fe0f013ff80 2024-01-11T08:29:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.526+0000 7fe0e5ffb700 1 -- 172.21.15.27:0/1576311898 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e0005bd0 con 0x7fe0f013ff80 2024-01-11T08:29:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.526+0000 7fe0f81ff700 1 -- 172.21.15.27:0/1576311898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ec0093c0 con 0x7fe0f013ff80 2024-01-11T08:29:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.526+0000 7fe0e5ffb700 1 -- 172.21.15.27:0/1576311898 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e00070d0 con 0x7fe0f013ff80 2024-01-11T08:29:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.526+0000 7fe0f81ff700 1 -- 172.21.15.27:0/1576311898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f00c1800 con 0x7fe0f013ff80 2024-01-11T08:29:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.528+0000 7fe0f81ff700 1 -- 172.21.15.27:0/1576311898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f004c3c0 con 0x7fe0f013ff80 2024-01-11T08:29:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.528+0000 7fe0e5ffb700 1 -- 172.21.15.27:0/1576311898 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 69) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fe0e0007480 con 0x7fe0f013ff80 2024-01-11T08:29:41.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.530+0000 7fe0e5ffb700 1 --2- 172.21.15.27:0/1576311898 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe0d4066f80 0x7fe0d4069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:41.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.530+0000 7fe0e5ffb700 1 -- 172.21.15.27:0/1576311898 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fe0e0083b20 con 0x7fe0f013ff80 2024-01-11T08:29:41.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.530+0000 7fe0f5f9b700 1 -- 172.21.15.27:0/1576311898 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe0d4066f80 msgr2=0x7fe0d4069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:41.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.530+0000 7fe0f5f9b700 1 --2- 172.21.15.27:0/1576311898 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe0d4066f80 0x7fe0d4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:41.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.534+0000 7fe0e5ffb700 1 -- 172.21.15.27:0/1576311898 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe0e0050cb0 con 0x7fe0f013ff80 2024-01-11T08:29:41.564 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:41.565+0000 7ff9fcd311c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:29:41.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.731+0000 7fe0f5f9b700 1 -- 172.21.15.27:0/1576311898 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe0d4066f80 msgr2=0x7fe0d4069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:41.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.731+0000 7fe0f5f9b700 1 --2- 172.21.15.27:0/1576311898 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe0d4066f80 0x7fe0d4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:41.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.748+0000 7fe0f81ff700 1 -- 172.21.15.27:0/1576311898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fe0f00c4860 con 0x7fe0f013ff80 2024-01-11T08:29:41.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.754+0000 7fe0e5ffb700 1 -- 172.21.15.27:0/1576311898 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v48)=0 v48) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe0e0017070 con 0x7fe0f013ff80 2024-01-11T08:29:41.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.756+0000 7fe0f81ff700 1 -- 172.21.15.27:0/1576311898 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe0d4066f80 msgr2=0x7fe0d4069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:41.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.756+0000 7fe0f81ff700 1 --2- 172.21.15.27:0/1576311898 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe0d4066f80 0x7fe0d4069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:41.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.757+0000 7fe0f81ff700 1 -- 172.21.15.27:0/1576311898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0f013ff80 msgr2=0x7fe0f013be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:41.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.757+0000 7fe0f81ff700 1 --2- 172.21.15.27:0/1576311898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0f013ff80 0x7fe0f013be10 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00011e0 tx=0x7fe0e0007e10 comp rx=0 tx=0).stop 2024-01-11T08:29:41.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.757+0000 7fe0f81ff700 1 -- 172.21.15.27:0/1576311898 shutdown_connections 2024-01-11T08:29:41.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.757+0000 7fe0f81ff700 1 --2- 172.21.15.27:0/1576311898 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe0d4066f80 0x7fe0d4069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:41.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.757+0000 7fe0f81ff700 1 --2- 172.21.15.27:0/1576311898 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0f015a730 0x7fe0f013d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:41.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.757+0000 7fe0f81ff700 1 --2- 172.21.15.27:0/1576311898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0f013fbb0 0x7fe0f00b1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:41.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.757+0000 7fe0f81ff700 1 --2- 172.21.15.27:0/1576311898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0f013ff80 0x7fe0f013be10 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:41.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.757+0000 7fe0f81ff700 1 -- 172.21.15.27:0/1576311898 >> 172.21.15.27:0/1576311898 conn(0x7fe0f00680a0 msgr2=0x7fe0f013ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:41.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.757+0000 7fe0f81ff700 1 -- 172.21.15.27:0/1576311898 shutdown_connections 2024-01-11T08:29:41.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:41.757+0000 7fe0f81ff700 1 -- 172.21.15.27:0/1576311898 wait complete. 2024-01-11T08:29:41.774 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:41.774 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/minutes) 2024-01-11T08:29:41.830 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/minutes 1m 2024-01-11T08:29:42.018 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:42.019+0000 7f5dc4c681c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:29:42.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.038+0000 7f2a169ff700 1 Processor -- start 2024-01-11T08:29:42.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.038+0000 7f2a169ff700 1 -- start start 2024-01-11T08:29:42.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.039+0000 7f2a169ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2a10069b60 0x7f2a100cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:42.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.039+0000 7f2a169ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2a1015d6d0 0x7f2a1015daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:42.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.039+0000 7f2a169ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a100cb770 con 0x7f2a10069790 2024-01-11T08:29:42.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.039+0000 7f2a169ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a100cab60 con 0x7f2a1015d6d0 2024-01-11T08:29:42.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.039+0000 7f2a169ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a100cae40 con 0x7f2a10069b60 2024-01-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.040+0000 7f2a14f9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2a1015d6d0 0x7f2a1015daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.040+0000 7f2a0ffff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2a10069790 0x7f2a100c70a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:50472/0 (socket says 172.21.15.27:50472) 2024-01-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.040+0000 7f2a0ffff700 1 -- 172.21.15.27:0/3891562864 learned_addr learned my addr 172.21.15.27:0/3891562864 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.040+0000 7f2a0f7fe700 1 --2- 172.21.15.27:0/3891562864 >> v2:172.21.15.77:3300/0 conn(0x7f2a10069b60 0x7f2a100cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.040+0000 7f2a14f9c700 1 -- 172.21.15.27:0/3891562864 >> v2:172.21.15.77:3300/0 conn(0x7f2a10069b60 msgr2=0x7f2a100cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.040+0000 7f2a14f9c700 1 --2- 172.21.15.27:0/3891562864 >> v2:172.21.15.77:3300/0 conn(0x7f2a10069b60 0x7f2a100cbc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.040+0000 7f2a0effd700 1 -- 172.21.15.27:0/3891562864 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371861772 0 0) 0x7f2a100cb770 con 0x7f2a10069790 2024-01-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.040+0000 7f2a14f9c700 1 -- 172.21.15.27:0/3891562864 >> v1:172.21.15.77:6789/0 conn(0x7f2a10069790 legacy=0x7f2a100c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.040+0000 7f2a14f9c700 1 -- 172.21.15.27:0/3891562864 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a100cb120 con 0x7f2a1015d6d0 2024-01-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.040+0000 7f2a14f9c700 1 --2- 172.21.15.27:0/3891562864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a1015d6d0 0x7f2a1015daf0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f2a10071050 tx=0x7f2a08009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd67d37da1401005 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.041+0000 7f2a0effd700 1 -- 172.21.15.27:0/3891562864 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0800e070 con 0x7f2a1015d6d0 2024-01-11T08:29:42.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.041+0000 7f2a0effd700 1 -- 172.21.15.27:0/3891562864 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0800bbf0 con 0x7f2a1015d6d0 2024-01-11T08:29:42.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.041+0000 7f2a0effd700 1 -- 172.21.15.27:0/3891562864 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a08012430 con 0x7f2a1015d6d0 2024-01-11T08:29:42.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.041+0000 7f2a169ff700 1 -- 172.21.15.27:0/3891562864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a1015d6d0 msgr2=0x7f2a1015daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:42.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.041+0000 7f2a169ff700 1 --2- 172.21.15.27:0/3891562864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a1015d6d0 0x7f2a1015daf0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f2a10071050 tx=0x7f2a08009780 comp rx=0 tx=0).stop 2024-01-11T08:29:42.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.041+0000 7f2a169ff700 1 -- 172.21.15.27:0/3891562864 shutdown_connections 2024-01-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.041+0000 7f2a169ff700 1 --2- 172.21.15.27:0/3891562864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a1015d6d0 0x7f2a1015daf0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.041+0000 7f2a169ff700 1 --2- 172.21.15.27:0/3891562864 >> v2:172.21.15.77:3300/0 conn(0x7f2a10069b60 0x7f2a100cbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.041+0000 7f2a169ff700 1 -- 172.21.15.27:0/3891562864 >> 172.21.15.27:0/3891562864 conn(0x7f2a100680a0 msgr2=0x7f2a100cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.042+0000 7f2a169ff700 1 -- 172.21.15.27:0/3891562864 shutdown_connections 2024-01-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.042+0000 7f2a169ff700 1 -- 172.21.15.27:0/3891562864 wait complete. 2024-01-11T08:29:42.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.044+0000 7f2a169ff700 1 Processor -- start 2024-01-11T08:29:42.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.044+0000 7f2a169ff700 1 -- start start 2024-01-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.045+0000 7f2a169ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a10069790 0x7f2a10143d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.045+0000 7f2a169ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a10069b60 0x7f2a100c32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.045+0000 7f2a169ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a1015d6d0 0x7f2a100c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.045+0000 7f2a169ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a100af7d0 con 0x7f2a10069b60 2024-01-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.045+0000 7f2a169ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a100af4a0 con 0x7f2a10069790 2024-01-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.045+0000 7f2a169ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a100c5f30 con 0x7f2a1015d6d0 2024-01-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.045+0000 7f2a0f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a10069b60 0x7f2a100c32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:42.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.046+0000 7f2a0ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a10069790 0x7f2a10143d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.046+0000 7f2a0f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a10069b60 0x7f2a100c32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36930/0 (socket says 172.21.15.27:36930) 2024-01-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.046+0000 7f2a0f7fe700 1 -- 172.21.15.27:0/3447272014 learned_addr learned my addr 172.21.15.27:0/3447272014 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.046+0000 7f2a14f9c700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a1015d6d0 0x7f2a100c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.046+0000 7f2a0f7fe700 1 -- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a1015d6d0 msgr2=0x7f2a100c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.046+0000 7f2a0f7fe700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a1015d6d0 0x7f2a100c4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.046+0000 7f2a0f7fe700 1 -- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a10069790 msgr2=0x7f2a10143d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.046+0000 7f2a0f7fe700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a10069790 0x7f2a10143d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.046+0000 7f2a0f7fe700 1 -- 172.21.15.27:0/3447272014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a08009a20 con 0x7f2a10069b60 2024-01-11T08:29:42.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.046+0000 7f2a0f7fe700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a10069b60 0x7f2a100c32e0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f2a0400ec80 tx=0x7f2a040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.047+0000 7f29f7fff700 1 -- 172.21.15.27:0/3447272014 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0400c850 con 0x7f2a10069b60 2024-01-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.047+0000 7f29f7fff700 1 -- 172.21.15.27:0/3447272014 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0400c9b0 con 0x7f2a10069b60 2024-01-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.047+0000 7f29f7fff700 1 -- 172.21.15.27:0/3447272014 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0400cd50 con 0x7f2a10069b60 2024-01-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.047+0000 7f2a169ff700 1 -- 172.21.15.27:0/3447272014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a100c6210 con 0x7f2a10069b60 2024-01-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.047+0000 7f2a169ff700 1 -- 172.21.15.27:0/3447272014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a101442a0 con 0x7f2a10069b60 2024-01-11T08:29:42.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.048+0000 7f2a169ff700 1 -- 172.21.15.27:0/3447272014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a1004c3c0 con 0x7f2a10069b60 2024-01-11T08:29:42.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.049+0000 7f29f7fff700 1 -- 172.21.15.27:0/3447272014 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 69) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f2a04013680 con 0x7f2a10069b60 2024-01-11T08:29:42.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.050+0000 7f29f7fff700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f29fc066f80 0x7f29fc069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:42.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.050+0000 7f29f7fff700 1 -- 172.21.15.27:0/3447272014 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f2a040848d0 con 0x7f2a10069b60 2024-01-11T08:29:42.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.050+0000 7f2a0ffff700 1 -- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f29fc066f80 msgr2=0x7f29fc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:42.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.050+0000 7f2a0ffff700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f29fc066f80 0x7f29fc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:42.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.054+0000 7f29f7fff700 1 -- 172.21.15.27:0/3447272014 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2a04051a60 con 0x7f2a10069b60 2024-01-11T08:29:42.157 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:42.158+0000 7ff9fcd311c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:29:42.190 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:42.192+0000 7f5dc4c681c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:29:42.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.251+0000 7f2a0ffff700 1 -- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f29fc066f80 msgr2=0x7f29fc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:42.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.251+0000 7f2a0ffff700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f29fc066f80 0x7f29fc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:42.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.313+0000 7f2a169ff700 1 -- 172.21.15.27:0/3447272014 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a100af910 con 0x7f29fc066f80 2024-01-11T08:29:42.338 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:42.339+0000 7ff9fcd311c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:29:42.606 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:42.608+0000 7f5dc4c681c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:29:42.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.652+0000 7f2a0ffff700 1 -- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f29fc066f80 msgr2=0x7f29fc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:42.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:42.652+0000 7f2a0ffff700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f29fc066f80 0x7f29fc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-11T08:29:42.774 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:42.776+0000 7ff9fcd311c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:29:43.384 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:43.385+0000 7f5dc4c681c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:29:43.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:43.453+0000 7f2a0ffff700 1 -- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f29fc066f80 msgr2=0x7f29fc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:43.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:43.453+0000 7f2a0ffff700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f29fc066f80 0x7f29fc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-11T08:29:43.472 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:43.473+0000 7f5dc4c681c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:29:43.601 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:43.602+0000 7ff9fcd311c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:29:43.626 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:43.628+0000 7f5dc4c681c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:29:43.694 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:43.695+0000 7ff9fcd311c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:29:43.713 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:43.715+0000 7f5dc4c681c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:29:43.861 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:43.862+0000 7ff9fcd311c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:29:43.929 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:43.930+0000 7f5dc4c681c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:29:43.954 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:43.955+0000 7ff9fcd311c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:29:44.024 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:44.026+0000 7f5dc4c681c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:29:44.120 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:44.121+0000 7f5dc4c681c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:29:44.182 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:44.183+0000 7ff9fcd311c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:29:44.215 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:44.216+0000 7f5dc4c681c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:29:44.285 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:44.286+0000 7ff9fcd311c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:29:44.385 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:44.386+0000 7ff9fcd311c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:29:44.403 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:44.404+0000 7f5dc4c681c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:29:44.485 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:44.487+0000 7ff9fcd311c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:29:44.500 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:44.502+0000 7f5dc4c681c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:29:44.684 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:44.686+0000 7ff9fcd311c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:29:44.788 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:44.789+0000 7ff9fcd311c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:29:44.841 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:44.842+0000 7f5dc4c681c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:29:44.931 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:44.932+0000 7f5dc4c681c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:29:45.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:45.055+0000 7f2a0ffff700 1 -- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f29fc066f80 msgr2=0x7f29fc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:45.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:45.055+0000 7f2a0ffff700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f29fc066f80 0x7f29fc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-11T08:29:45.148 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:45.149+0000 7ff9fcd311c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:29:45.242 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:45.243+0000 7ff9fcd311c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:29:45.472 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:45.473+0000 7f5dc4c681c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:29:45.658 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:45.659+0000 7f5dc4c681c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:29:45.815 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:45.816+0000 7ff9fcd311c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:29:46.014 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:46.016+0000 7ff9fcd311c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:29:46.034 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:46.035+0000 7f5dc4c681c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:29:46.411 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:46.412+0000 7ff9fcd311c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:29:46.455 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:46.456+0000 7f5dc4c681c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:29:46.640 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:46.640+0000 7f5dc4c681c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:29:46.827 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:46.827+0000 7f5dc4c681c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:29:46.858 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:46.860+0000 7ff9fcd311c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:29:46.937 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:46.937+0000 7f5dc4c681c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:29:46.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:46.948+0000 7f29f7fff700 1 -- 172.21.15.27:0/3447272014 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 70) v1 ==== 84470+0+0 (secure 0 0 0) 0x7f2a04051e30 con 0x7f2a10069b60 2024-01-11T08:29:46.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:46.948+0000 7f29f7fff700 1 -- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f29fc066f80 msgr2=0x7f29fc069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:46.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:46.948+0000 7f29f7fff700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f29fc066f80 0x7f29fc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:47.065 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:47.066+0000 7ff9fcd311c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:29:47.066 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:47.067+0000 7f5d3d23f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:47.066 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:47.067+0000 7f5d3d23f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:47.066 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:47.067+0000 7f5d3d23f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:47.066 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:47.067+0000 7f5d3d23f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:47.067 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:47.067+0000 7f5d3d23f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:47.067 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:47.068+0000 7f5d3a239700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:47.067 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:47.068+0000 7f5d3a239700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:47.067 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:47.068+0000 7f5d3a239700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:47.068 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:47.068+0000 7f5d3a239700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:47.068 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:47.068+0000 7f5d3a239700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:29:47.266 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:47.267+0000 7ff9fcd311c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:29:47.382 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:47.383+0000 7ff9fcd311c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:29:47.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:47.951+0000 7f29f7fff700 1 -- 172.21.15.27:0/3447272014 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 71) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f2a0405c230 con 0x7f2a10069b60 2024-01-11T08:29:47.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:47.951+0000 7f29f7fff700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f29fc066f80 0x7f29fc0693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:47.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:47.951+0000 7f29f7fff700 1 -- 172.21.15.27:0/3447272014 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a100af910 con 0x7f29fc066f80 2024-01-11T08:29:47.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:47.951+0000 7f2a0ffff700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f29fc066f80 0x7f29fc0693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:47.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:47.953+0000 7f2a0ffff700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f29fc066f80 0x7f29fc0693b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2a00007fd0 tx=0x7f2a00007f40 comp rx=0 tx=0).ready entity=mgr.7076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:48.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.028+0000 7f29f7fff700 1 -- 172.21.15.27:0/3447272014 <== mgr.7076 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7f2a100af910 con 0x7f29fc066f80 2024-01-11T08:29:48.026 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path snap_test_dir1/minutes 2024-01-11T08:29:48.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.030+0000 7f2a169ff700 1 -- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f29fc066f80 msgr2=0x7f29fc0693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:48.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.030+0000 7f2a169ff700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f29fc066f80 0x7f29fc0693b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2a00007fd0 tx=0x7f2a00007f40 comp rx=0 tx=0).stop 2024-01-11T08:29:48.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.030+0000 7f2a169ff700 1 -- 172.21.15.27:0/3447272014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a10069b60 msgr2=0x7f2a100c32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:48.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.030+0000 7f2a169ff700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a10069b60 0x7f2a100c32e0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f2a0400ec80 tx=0x7f2a040091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:48.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.030+0000 7f2a169ff700 1 -- 172.21.15.27:0/3447272014 shutdown_connections 2024-01-11T08:29:48.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.030+0000 7f2a169ff700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f29fc066f80 0x7f29fc0693b0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:48.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.030+0000 7f2a169ff700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a1015d6d0 0x7f2a100c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:48.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.030+0000 7f2a169ff700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a10069790 0x7f2a10143d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:48.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.030+0000 7f2a169ff700 1 --2- 172.21.15.27:0/3447272014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a10069b60 0x7f2a100c32e0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:48.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.030+0000 7f2a169ff700 1 -- 172.21.15.27:0/3447272014 >> 172.21.15.27:0/3447272014 conn(0x7f2a100680a0 msgr2=0x7f2a100c26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:48.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.030+0000 7f2a169ff700 1 -- 172.21.15.27:0/3447272014 shutdown_connections 2024-01-11T08:29:48.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.030+0000 7f2a169ff700 1 -- 172.21.15.27:0/3447272014 wait complete. 2024-01-11T08:29:48.047 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/minutes 2024-01-11T08:29:48.048 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:48.048 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/hourly) 2024-01-11T08:29:48.098 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/hourly 1h 2024-01-11T08:29:48.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.302+0000 7f08d29ff700 1 Processor -- start 2024-01-11T08:29:48.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.302+0000 7f08d29ff700 1 -- start start 2024-01-11T08:29:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.303+0000 7f08d29ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08cc13ff80 0x7f08cc15a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.303+0000 7f08d29ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08cc15a9c0 con 0x7f08cc13fbb0 2024-01-11T08:29:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.303+0000 7f08d29ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08cc0c6b30 con 0x7f08cc15a560 2024-01-11T08:29:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.304+0000 7f08d29ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc0c6800 con 0x7f08cc13ff80 2024-01-11T08:29:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.304+0000 7f08cbfff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f08cc13fbb0 0x7f08cc0ca7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43758/0 (socket says 172.21.15.27:43758) 2024-01-11T08:29:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.304+0000 7f08cbfff700 1 -- 172.21.15.27:0/1217406802 learned_addr learned my addr 172.21.15.27:0/1217406802 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.304+0000 7f08cb7fe700 1 --2- 172.21.15.27:0/1217406802 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08cc13ff80 0x7f08cc15a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:48.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.304+0000 7f08caffd700 1 -- 172.21.15.27:0/1217406802 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638888377 0 0) 0x7f08bc001290 con 0x7f08cc13fbb0 2024-01-11T08:29:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.304+0000 7f08caffd700 1 -- 172.21.15.27:0/1217406802 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08b4003570 con 0x7f08cc13fbb0 2024-01-11T08:29:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.304+0000 7f08caffd700 1 -- 172.21.15.27:0/1217406802 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918599888 0 0) 0x7f08cc0c6b30 con 0x7f08cc15a560 2024-01-11T08:29:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.304+0000 7f08caffd700 1 -- 172.21.15.27:0/1217406802 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08bc001290 con 0x7f08cc15a560 2024-01-11T08:29:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.304+0000 7f08cb7fe700 1 -- 172.21.15.27:0/1217406802 >> v1:172.21.15.77:6789/0 conn(0x7f08cc15a560 legacy=0x7f08cc0c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.305+0000 7f08cb7fe700 1 -- 172.21.15.27:0/1217406802 >> v1:172.21.15.27:6789/0 conn(0x7f08cc13fbb0 legacy=0x7f08cc0ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.305+0000 7f08cb7fe700 1 -- 172.21.15.27:0/1217406802 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08cc0cb240 con 0x7f08cc13ff80 2024-01-11T08:29:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.305+0000 7f08cb7fe700 1 --2- 172.21.15.27:0/1217406802 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08cc13ff80 0x7f08cc15a020 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f08c0009910 tx=0x7f08c00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5f52096a88bca8c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.305+0000 7f08caffd700 1 -- 172.21.15.27:0/1217406802 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c0011630 con 0x7f08cc13ff80 2024-01-11T08:29:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.305+0000 7f08caffd700 1 -- 172.21.15.27:0/1217406802 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c0011790 con 0x7f08cc13ff80 2024-01-11T08:29:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.305+0000 7f08caffd700 1 -- 172.21.15.27:0/1217406802 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c0011b30 con 0x7f08cc13ff80 2024-01-11T08:29:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.306+0000 7f08d29ff700 1 -- 172.21.15.27:0/1217406802 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08cc13ff80 msgr2=0x7f08cc15a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.306+0000 7f08d29ff700 1 --2- 172.21.15.27:0/1217406802 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08cc13ff80 0x7f08cc15a020 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f08c0009910 tx=0x7f08c00091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.306+0000 7f08d29ff700 1 -- 172.21.15.27:0/1217406802 shutdown_connections 2024-01-11T08:29:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.306+0000 7f08d29ff700 1 --2- 172.21.15.27:0/1217406802 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08cc13ff80 0x7f08cc15a020 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.306+0000 7f08d29ff700 1 -- 172.21.15.27:0/1217406802 >> 172.21.15.27:0/1217406802 conn(0x7f08cc0680a0 msgr2=0x7f08cc0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:48.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.306+0000 7f08d29ff700 1 -- 172.21.15.27:0/1217406802 shutdown_connections 2024-01-11T08:29:48.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.306+0000 7f08d29ff700 1 -- 172.21.15.27:0/1217406802 wait complete. 2024-01-11T08:29:48.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.307+0000 7f08d29ff700 1 Processor -- start 2024-01-11T08:29:48.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.308+0000 7f08d29ff700 1 -- start start 2024-01-11T08:29:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.308+0000 7f08d29ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08cc13fbb0 0x7f08cc157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.308+0000 7f08d29ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08cc13ff80 0x7f08cc0c4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.309+0000 7f08d29ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08cc15a560 0x7f08cc0c4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.309+0000 7f08d29ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc157280 con 0x7f08cc13ff80 2024-01-11T08:29:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.309+0000 7f08d29ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc0c1f00 con 0x7f08cc13fbb0 2024-01-11T08:29:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.309+0000 7f08d29ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc156c30 con 0x7f08cc15a560 2024-01-11T08:29:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.309+0000 7f08cb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08cc13ff80 0x7f08cc0c4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.309+0000 7f08cb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08cc13ff80 0x7f08cc0c4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54406/0 (socket says 172.21.15.27:54406) 2024-01-11T08:29:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.309+0000 7f08cb7fe700 1 -- 172.21.15.27:0/192432598 learned_addr learned my addr 172.21.15.27:0/192432598 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.309+0000 7f08cbfff700 1 --2- 172.21.15.27:0/192432598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08cc13fbb0 0x7f08cc157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.309+0000 7f08d0f9c700 1 --2- 172.21.15.27:0/192432598 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08cc15a560 0x7f08cc0c4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.309+0000 7f08cb7fe700 1 -- 172.21.15.27:0/192432598 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08cc15a560 msgr2=0x7f08cc0c4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.309+0000 7f08cb7fe700 1 --2- 172.21.15.27:0/192432598 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08cc15a560 0x7f08cc0c4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.309+0000 7f08cb7fe700 1 -- 172.21.15.27:0/192432598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08cc13fbb0 msgr2=0x7f08cc157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.309+0000 7f08cb7fe700 1 --2- 172.21.15.27:0/192432598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08cc13fbb0 0x7f08cc157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.309+0000 7f08cb7fe700 1 -- 172.21.15.27:0/192432598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c0009420 con 0x7f08cc13ff80 2024-01-11T08:29:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.310+0000 7f08cb7fe700 1 --2- 172.21.15.27:0/192432598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08cc13ff80 0x7f08cc0c4970 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f08c0009f00 tx=0x7f08c0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.310+0000 7f08b3fff700 1 -- 172.21.15.27:0/192432598 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c001f070 con 0x7f08cc13ff80 2024-01-11T08:29:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.310+0000 7f08b3fff700 1 -- 172.21.15.27:0/192432598 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c0002f70 con 0x7f08cc13ff80 2024-01-11T08:29:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.310+0000 7f08d29ff700 1 -- 172.21.15.27:0/192432598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08cc156f10 con 0x7f08cc13ff80 2024-01-11T08:29:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.310+0000 7f08b3fff700 1 -- 172.21.15.27:0/192432598 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c0011900 con 0x7f08cc13ff80 2024-01-11T08:29:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.310+0000 7f08d29ff700 1 -- 172.21.15.27:0/192432598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08cc157e10 con 0x7f08cc13ff80 2024-01-11T08:29:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.311+0000 7f08d29ff700 1 -- 172.21.15.27:0/192432598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08cc156a90 con 0x7f08cc13ff80 2024-01-11T08:29:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.312+0000 7f08b3fff700 1 -- 172.21.15.27:0/192432598 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 71) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f08c0023070 con 0x7f08cc13ff80 2024-01-11T08:29:48.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.313+0000 7f08b3fff700 1 --2- 172.21.15.27:0/192432598 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f08b8066dd0 0x7f08b8069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:48.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.313+0000 7f08b3fff700 1 -- 172.21.15.27:0/192432598 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f08c00856d0 con 0x7f08cc13ff80 2024-01-11T08:29:48.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.314+0000 7f08cbfff700 1 --2- 172.21.15.27:0/192432598 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f08b8066dd0 0x7f08b8069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:48.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.314+0000 7f08cbfff700 1 --2- 172.21.15.27:0/192432598 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f08b8066dd0 0x7f08b8069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f08bc0014f0 tx=0x7f08bc009040 comp rx=0 tx=0).ready entity=mgr.7076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:48.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.318+0000 7f08b3fff700 1 -- 172.21.15.27:0/192432598 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f08cc156a90 con 0x7f08cc13ff80 2024-01-11T08:29:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.550+0000 7f08d29ff700 1 -- 172.21.15.27:0/192432598 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/hourly", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f08cc156a90 con 0x7f08b8066dd0 2024-01-11T08:29:48.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.617+0000 7f08b3fff700 1 -- 172.21.15.27:0/192432598 <== mgr.7076 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f08cc156a90 con 0x7f08b8066dd0 2024-01-11T08:29:48.616 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path snap_test_dir1/hourly 2024-01-11T08:29:48.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.619+0000 7f08d29ff700 1 -- 172.21.15.27:0/192432598 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f08b8066dd0 msgr2=0x7f08b8069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:48.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.619+0000 7f08d29ff700 1 --2- 172.21.15.27:0/192432598 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f08b8066dd0 0x7f08b8069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f08bc0014f0 tx=0x7f08bc009040 comp rx=0 tx=0).stop 2024-01-11T08:29:48.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.619+0000 7f08d29ff700 1 -- 172.21.15.27:0/192432598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08cc13ff80 msgr2=0x7f08cc0c4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:48.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.619+0000 7f08d29ff700 1 --2- 172.21.15.27:0/192432598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08cc13ff80 0x7f08cc0c4970 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f08c0009f00 tx=0x7f08c0017000 comp rx=0 tx=0).stop 2024-01-11T08:29:48.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.619+0000 7f08d29ff700 1 -- 172.21.15.27:0/192432598 shutdown_connections 2024-01-11T08:29:48.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.619+0000 7f08d29ff700 1 --2- 172.21.15.27:0/192432598 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f08b8066dd0 0x7f08b8069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:48.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.619+0000 7f08d29ff700 1 --2- 172.21.15.27:0/192432598 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08cc15a560 0x7f08cc0c4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:48.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.619+0000 7f08d29ff700 1 --2- 172.21.15.27:0/192432598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08cc13fbb0 0x7f08cc157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:48.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.619+0000 7f08d29ff700 1 --2- 172.21.15.27:0/192432598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08cc13ff80 0x7f08cc0c4970 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:48.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.619+0000 7f08d29ff700 1 -- 172.21.15.27:0/192432598 >> 172.21.15.27:0/192432598 conn(0x7f08cc0680a0 msgr2=0x7f08cc0cc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:48.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.619+0000 7f08d29ff700 1 -- 172.21.15.27:0/192432598 shutdown_connections 2024-01-11T08:29:48.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.619+0000 7f08d29ff700 1 -- 172.21.15.27:0/192432598 wait complete. 2024-01-11T08:29:48.637 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/hourly 2024-01-11T08:29:48.637 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:48.637 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/daily) 2024-01-11T08:29:48.687 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/daily 1d 2024-01-11T08:29:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.890+0000 7f5d181ff700 1 Processor -- start 2024-01-11T08:29:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.890+0000 7f5d181ff700 1 -- start start 2024-01-11T08:29:48.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.892+0000 7f5d181ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5d1013ff80 0x7f5d1015a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:48.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.892+0000 7f5d181ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d1015a560 0x7f5d100c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:48.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.892+0000 7f5d181ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d100c6b30 con 0x7f5d1013fbb0 2024-01-11T08:29:48.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.892+0000 7f5d181ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d1015a9a0 con 0x7f5d1013ff80 2024-01-11T08:29:48.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.892+0000 7f5d181ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d100c6800 con 0x7f5d1015a560 2024-01-11T08:29:48.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.892+0000 7f5d1679c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d1015a560 0x7f5d100c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:48.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.892+0000 7f5d1579a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5d1013ff80 0x7f5d1015a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:48.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.892+0000 7f5d15f9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5d1013fbb0 0x7f5d100ca7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:48736/0 (socket says 172.21.15.27:48736) 2024-01-11T08:29:48.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.892+0000 7f5d15f9b700 1 -- 172.21.15.27:0/3192345603 learned_addr learned my addr 172.21.15.27:0/3192345603 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:48.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.893+0000 7f5d1579a700 1 -- 172.21.15.27:0/3192345603 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d1015a560 msgr2=0x7f5d100c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:48.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.893+0000 7f5d1579a700 1 --2- 172.21.15.27:0/3192345603 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d1015a560 0x7f5d100c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:48.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.893+0000 7f5d1579a700 1 -- 172.21.15.27:0/3192345603 >> v1:172.21.15.77:6789/0 conn(0x7f5d1013fbb0 legacy=0x7f5d100ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:48.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.893+0000 7f5d1579a700 1 -- 172.21.15.27:0/3192345603 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d100cb240 con 0x7f5d1013ff80 2024-01-11T08:29:48.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.893+0000 7f5d1679c700 1 --2- 172.21.15.27:0/3192345603 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d1015a560 0x7f5d100c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:48.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.893+0000 7f5d14f99700 1 -- 172.21.15.27:0/3192345603 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922810342 0 0) 0x7f5d100c6b30 con 0x7f5d1013fbb0 2024-01-11T08:29:48.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.893+0000 7f5d1579a700 1 --2- 172.21.15.27:0/3192345603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d1013ff80 0x7f5d1015a020 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5d000098e0 tx=0x7f5d000091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2d526416433aa20 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:48.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.893+0000 7f5d14f99700 1 -- 172.21.15.27:0/3192345603 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0000b640 con 0x7f5d1013ff80 2024-01-11T08:29:48.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.893+0000 7f5d14f99700 1 -- 172.21.15.27:0/3192345603 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0000b7a0 con 0x7f5d1013ff80 2024-01-11T08:29:48.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.893+0000 7f5d14f99700 1 -- 172.21.15.27:0/3192345603 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0000bb40 con 0x7f5d1013ff80 2024-01-11T08:29:48.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.894+0000 7f5d181ff700 1 -- 172.21.15.27:0/3192345603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d1013ff80 msgr2=0x7f5d1015a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:48.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.894+0000 7f5d181ff700 1 --2- 172.21.15.27:0/3192345603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d1013ff80 0x7f5d1015a020 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5d000098e0 tx=0x7f5d000091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:48.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.894+0000 7f5d181ff700 1 -- 172.21.15.27:0/3192345603 shutdown_connections 2024-01-11T08:29:48.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.894+0000 7f5d181ff700 1 --2- 172.21.15.27:0/3192345603 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d1015a560 0x7f5d100c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:48.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.894+0000 7f5d181ff700 1 --2- 172.21.15.27:0/3192345603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d1013ff80 0x7f5d1015a020 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:48.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.894+0000 7f5d181ff700 1 -- 172.21.15.27:0/3192345603 >> 172.21.15.27:0/3192345603 conn(0x7f5d100680a0 msgr2=0x7f5d100cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:48.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.894+0000 7f5d181ff700 1 -- 172.21.15.27:0/3192345603 shutdown_connections 2024-01-11T08:29:48.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.894+0000 7f5d181ff700 1 -- 172.21.15.27:0/3192345603 wait complete. 2024-01-11T08:29:48.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.896+0000 7f5d181ff700 1 Processor -- start 2024-01-11T08:29:48.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.896+0000 7f5d181ff700 1 -- start start 2024-01-11T08:29:48.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.897+0000 7f5d181ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d1013fbb0 0x7f5d100c4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:48.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.897+0000 7f5d181ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d1013ff80 0x7f5d100c3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:48.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.897+0000 7f5d181ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d1015a560 0x7f5d10159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:48.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.897+0000 7f5d181ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d10167be0 con 0x7f5d1013ff80 2024-01-11T08:29:48.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.897+0000 7f5d181ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1013c900 con 0x7f5d1015a560 2024-01-11T08:29:48.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.897+0000 7f5d181ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1013cbe0 con 0x7f5d1013fbb0 2024-01-11T08:29:48.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.897+0000 7f5d1579a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d1013ff80 0x7f5d100c3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:48.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.897+0000 7f5d1579a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d1013ff80 0x7f5d100c3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54412/0 (socket says 172.21.15.27:54412) 2024-01-11T08:29:48.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.897+0000 7f5d1579a700 1 -- 172.21.15.27:0/4059369761 learned_addr learned my addr 172.21.15.27:0/4059369761 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:48.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.897+0000 7f5d15f9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d1013fbb0 0x7f5d100c4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:48.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.897+0000 7f5d1679c700 1 --2- 172.21.15.27:0/4059369761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d1015a560 0x7f5d10159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:48.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.897+0000 7f5d1579a700 1 -- 172.21.15.27:0/4059369761 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d1013fbb0 msgr2=0x7f5d100c4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:48.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.897+0000 7f5d1579a700 1 --2- 172.21.15.27:0/4059369761 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d1013fbb0 0x7f5d100c4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:48.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.897+0000 7f5d1579a700 1 -- 172.21.15.27:0/4059369761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d1015a560 msgr2=0x7f5d10159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:48.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.897+0000 7f5d1579a700 1 --2- 172.21.15.27:0/4059369761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d1015a560 0x7f5d10159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:48.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.897+0000 7f5d1579a700 1 -- 172.21.15.27:0/4059369761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d08009a20 con 0x7f5d1013ff80 2024-01-11T08:29:48.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.897+0000 7f5d1679c700 1 --2- 172.21.15.27:0/4059369761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d1015a560 0x7f5d10159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:48.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.898+0000 7f5d1579a700 1 --2- 172.21.15.27:0/4059369761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d1013ff80 0x7f5d100c3270 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f5d10069490 tx=0x7f5d00017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:48.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.898+0000 7f5d05ffb700 1 -- 172.21.15.27:0/4059369761 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d00000d10 con 0x7f5d1013ff80 2024-01-11T08:29:48.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.898+0000 7f5d05ffb700 1 -- 172.21.15.27:0/4059369761 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0000bb20 con 0x7f5d1013ff80 2024-01-11T08:29:48.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.898+0000 7f5d05ffb700 1 -- 172.21.15.27:0/4059369761 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d00011650 con 0x7f5d1013ff80 2024-01-11T08:29:48.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.898+0000 7f5d181ff700 1 -- 172.21.15.27:0/4059369761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d00009420 con 0x7f5d1013ff80 2024-01-11T08:29:48.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.898+0000 7f5d181ff700 1 -- 172.21.15.27:0/4059369761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d1013d1a0 con 0x7f5d1013ff80 2024-01-11T08:29:48.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.899+0000 7f5d181ff700 1 -- 172.21.15.27:0/4059369761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d10167d50 con 0x7f5d1013ff80 2024-01-11T08:29:48.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.900+0000 7f5d05ffb700 1 -- 172.21.15.27:0/4059369761 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 71) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f5d000119c0 con 0x7f5d1013ff80 2024-01-11T08:29:48.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.901+0000 7f5d05ffb700 1 --2- 172.21.15.27:0/4059369761 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5cf4066dd0 0x7f5cf4069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:48.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.901+0000 7f5d05ffb700 1 -- 172.21.15.27:0/4059369761 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f5d000856c0 con 0x7f5d1013ff80 2024-01-11T08:29:48.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.902+0000 7f5d15f9b700 1 --2- 172.21.15.27:0/4059369761 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5cf4066dd0 0x7f5cf4069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:48.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.902+0000 7f5d15f9b700 1 --2- 172.21.15.27:0/4059369761 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5cf4066dd0 0x7f5cf4069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c001230 tx=0x7f5d0c007d20 comp rx=0 tx=0).ready entity=mgr.7076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:48.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:48.905+0000 7f5d05ffb700 1 -- 172.21.15.27:0/4059369761 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5d00052b60 con 0x7f5d1013ff80 2024-01-11T08:29:49.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.003+0000 7f5d05ffb700 1 -- 172.21.15.27:0/4059369761 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 72) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f5d00052f30 con 0x7f5d1013ff80 2024-01-11T08:29:49.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.141+0000 7f5d181ff700 1 -- 172.21.15.27:0/4059369761 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/daily", "snap_schedule": "1d", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d1004c3c0 con 0x7f5cf4066dd0 2024-01-11T08:29:49.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.206+0000 7f5d05ffb700 1 -- 172.21.15.27:0/4059369761 <== mgr.7076 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f5d1004c3c0 con 0x7f5cf4066dd0 2024-01-11T08:29:49.205 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path snap_test_dir1/daily 2024-01-11T08:29:49.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.208+0000 7f5d181ff700 1 -- 172.21.15.27:0/4059369761 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5cf4066dd0 msgr2=0x7f5cf4069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:49.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.208+0000 7f5d181ff700 1 --2- 172.21.15.27:0/4059369761 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5cf4066dd0 0x7f5cf4069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c001230 tx=0x7f5d0c007d20 comp rx=0 tx=0).stop 2024-01-11T08:29:49.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.208+0000 7f5d181ff700 1 -- 172.21.15.27:0/4059369761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d1013ff80 msgr2=0x7f5d100c3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:49.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.208+0000 7f5d181ff700 1 --2- 172.21.15.27:0/4059369761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d1013ff80 0x7f5d100c3270 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f5d10069490 tx=0x7f5d00017000 comp rx=0 tx=0).stop 2024-01-11T08:29:49.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.208+0000 7f5d181ff700 1 -- 172.21.15.27:0/4059369761 shutdown_connections 2024-01-11T08:29:49.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.208+0000 7f5d181ff700 1 --2- 172.21.15.27:0/4059369761 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5cf4066dd0 0x7f5cf4069280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:49.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.208+0000 7f5d181ff700 1 --2- 172.21.15.27:0/4059369761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d1015a560 0x7f5d10159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:49.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.208+0000 7f5d181ff700 1 --2- 172.21.15.27:0/4059369761 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d1013fbb0 0x7f5d100c4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:49.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.208+0000 7f5d181ff700 1 --2- 172.21.15.27:0/4059369761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d1013ff80 0x7f5d100c3270 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:49.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.208+0000 7f5d181ff700 1 -- 172.21.15.27:0/4059369761 >> 172.21.15.27:0/4059369761 conn(0x7f5d100680a0 msgr2=0x7f5d1013da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:49.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.209+0000 7f5d181ff700 1 -- 172.21.15.27:0/4059369761 shutdown_connections 2024-01-11T08:29:49.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.209+0000 7f5d181ff700 1 -- 172.21.15.27:0/4059369761 wait complete. 2024-01-11T08:29:49.226 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/daily 2024-01-11T08:29:49.226 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:49.226 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/weekly) 2024-01-11T08:29:49.275 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/weekly 1w 2024-01-11T08:29:49.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.479+0000 7f69111ff700 1 Processor -- start 2024-01-11T08:29:49.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.480+0000 7f69111ff700 1 -- start start 2024-01-11T08:29:49.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.481+0000 7f69111ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f690c068d30 0x7f690c0c8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:49.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.481+0000 7f69111ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f690c0c80a0 con 0x7f690c159ac0 2024-01-11T08:29:49.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.481+0000 7f69111ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f690c0cbac0 con 0x7f690c1596f0 2024-01-11T08:29:49.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.481+0000 7f69111ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f690c0a9fb0 con 0x7f690c068d30 2024-01-11T08:29:49.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.481+0000 7f690a59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f690c159ac0 0x7f690c068620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43772/0 (socket says 172.21.15.27:43772) 2024-01-11T08:29:49.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.481+0000 7f690a59c700 1 -- 172.21.15.27:0/423457599 learned_addr learned my addr 172.21.15.27:0/423457599 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:49.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.481+0000 7f690b59e700 1 --2- 172.21.15.27:0/423457599 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f690c068d30 0x7f690c0c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.482+0000 7f6909d9b700 1 -- 172.21.15.27:0/423457599 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427862745 0 0) 0x7f690c0c80a0 con 0x7f690c159ac0 2024-01-11T08:29:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.482+0000 7f6909d9b700 1 -- 172.21.15.27:0/423457599 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68f4003570 con 0x7f690c159ac0 2024-01-11T08:29:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.482+0000 7f6909d9b700 1 -- 172.21.15.27:0/423457599 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808886363 0 0) 0x7f690c0cbac0 con 0x7f690c1596f0 2024-01-11T08:29:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.482+0000 7f6909d9b700 1 -- 172.21.15.27:0/423457599 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f690c0c80a0 con 0x7f690c1596f0 2024-01-11T08:29:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.482+0000 7f6909d9b700 1 -- 172.21.15.27:0/423457599 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4183705333 0 0) 0x7f68f4003570 con 0x7f690c159ac0 2024-01-11T08:29:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.482+0000 7f6909d9b700 1 -- 172.21.15.27:0/423457599 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f690c068d30 msgr2=0x7f690c0c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.482+0000 7f6909d9b700 1 --2- 172.21.15.27:0/423457599 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f690c068d30 0x7f690c0c8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.482+0000 7f6909d9b700 1 -- 172.21.15.27:0/423457599 >> v1:172.21.15.77:6789/0 conn(0x7f690c1596f0 legacy=0x7f690c0a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.482+0000 7f6909d9b700 1 -- 172.21.15.27:0/423457599 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f690c0c7490 con 0x7f690c159ac0 2024-01-11T08:29:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.482+0000 7f6909d9b700 1 -- 172.21.15.27:0/423457599 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f6900001470 con 0x7f690c159ac0 2024-01-11T08:29:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.482+0000 7f6909d9b700 1 -- 172.21.15.27:0/423457599 >> v1:172.21.15.27:6789/0 conn(0x7f690c159ac0 legacy=0x7f690c068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.482+0000 7f6909d9b700 1 --2- 172.21.15.27:0/423457599 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f690c068d30 0x7f68f4003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.482+0000 7f6909d9b700 1 --2- 172.21.15.27:0/423457599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68f4004180 0x7f68f4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.482+0000 7f6909d9b700 1 --2- 172.21.15.27:0/423457599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68f4004d50 0x7f68f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.482+0000 7f6909d9b700 1 -- 172.21.15.27:0/423457599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c0a9fb0 con 0x7f68f4004180 2024-01-11T08:29:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.482+0000 7f6909d9b700 1 -- 172.21.15.27:0/423457599 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f400bba0 con 0x7f68f4004d50 2024-01-11T08:29:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.482+0000 7f6909d9b700 1 -- 172.21.15.27:0/423457599 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f400bd60 con 0x7f690c068d30 2024-01-11T08:29:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.482+0000 7f690a59c700 1 --2- 172.21.15.27:0/423457599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68f4004180 0x7f68f4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.482+0000 7f690ad9d700 1 --2- 172.21.15.27:0/423457599 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f690c068d30 0x7f68f4003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.483+0000 7f690b59e700 1 --2- 172.21.15.27:0/423457599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68f4004d50 0x7f68f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.483+0000 7f690ad9d700 1 -- 172.21.15.27:0/423457599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68f4004d50 msgr2=0x7f68f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.483+0000 7f690ad9d700 1 --2- 172.21.15.27:0/423457599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68f4004d50 0x7f68f40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.483+0000 7f690ad9d700 1 -- 172.21.15.27:0/423457599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68f4004180 msgr2=0x7f68f4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:49.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.483+0000 7f690ad9d700 1 --2- 172.21.15.27:0/423457599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68f4004180 0x7f68f4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:49.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.483+0000 7f690ad9d700 1 -- 172.21.15.27:0/423457599 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f4003570 con 0x7f690c068d30 2024-01-11T08:29:49.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.483+0000 7f690a59c700 1 --2- 172.21.15.27:0/423457599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68f4004180 0x7f68f4004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:49.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.483+0000 7f690ad9d700 1 --2- 172.21.15.27:0/423457599 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f690c068d30 0x7f68f4003c20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f68fc001a50 tx=0x7f68fc007d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22f09b9fb93041be server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:49.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.483+0000 7f6909d9b700 1 -- 172.21.15.27:0/423457599 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc00c5d0 con 0x7f690c068d30 2024-01-11T08:29:49.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.483+0000 7f6909d9b700 1 -- 172.21.15.27:0/423457599 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68fc00c730 con 0x7f690c068d30 2024-01-11T08:29:49.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.483+0000 7f6909d9b700 1 -- 172.21.15.27:0/423457599 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc00cad0 con 0x7f690c068d30 2024-01-11T08:29:49.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.484+0000 7f69111ff700 1 -- 172.21.15.27:0/423457599 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f690c068d30 msgr2=0x7f68f4003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:49.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.484+0000 7f69111ff700 1 --2- 172.21.15.27:0/423457599 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f690c068d30 0x7f68f4003c20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f68fc001a50 tx=0x7f68fc007d30 comp rx=0 tx=0).stop 2024-01-11T08:29:49.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.484+0000 7f690b59e700 1 -- 172.21.15.27:0/423457599 reap_dead start 2024-01-11T08:29:49.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.484+0000 7f69111ff700 1 -- 172.21.15.27:0/423457599 shutdown_connections 2024-01-11T08:29:49.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.484+0000 7f69111ff700 1 -- 172.21.15.27:0/423457599 >> 172.21.15.27:0/423457599 conn(0x7f690c0c9fb0 msgr2=0x7f690c06ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:49.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.484+0000 7f69111ff700 1 -- 172.21.15.27:0/423457599 shutdown_connections 2024-01-11T08:29:49.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.484+0000 7f69111ff700 1 -- 172.21.15.27:0/423457599 wait complete. 2024-01-11T08:29:49.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.486+0000 7f69111ff700 1 Processor -- start 2024-01-11T08:29:49.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.486+0000 7f69111ff700 1 -- start start 2024-01-11T08:29:49.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.487+0000 7f69111ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f690c068d30 0x7f690c14a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:49.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.487+0000 7f69111ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f690c0b20e0 0x7f690c0c5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:49.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.487+0000 7f69111ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f690c0b2550 0x7f690c0c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:49.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.487+0000 7f69111ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c0c4e80 con 0x7f690c0b2550 2024-01-11T08:29:49.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.487+0000 7f69111ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c0b0630 con 0x7f690c068d30 2024-01-11T08:29:49.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.487+0000 7f69111ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f690c0b0910 con 0x7f690c0b20e0 2024-01-11T08:29:49.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.487+0000 7f690b59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f690c0b2550 0x7f690c0c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:49.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.487+0000 7f690b59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f690c0b2550 0x7f690c0c6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54424/0 (socket says 172.21.15.27:54424) 2024-01-11T08:29:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.487+0000 7f690b59e700 1 -- 172.21.15.27:0/2087948874 learned_addr learned my addr 172.21.15.27:0/2087948874 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.487+0000 7f690a59c700 1 --2- 172.21.15.27:0/2087948874 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f690c0b20e0 0x7f690c0c5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.487+0000 7f690ad9d700 1 --2- 172.21.15.27:0/2087948874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f690c068d30 0x7f690c14a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.487+0000 7f690b59e700 1 -- 172.21.15.27:0/2087948874 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f690c0b20e0 msgr2=0x7f690c0c5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.487+0000 7f690b59e700 1 --2- 172.21.15.27:0/2087948874 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f690c0b20e0 0x7f690c0c5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.487+0000 7f690b59e700 1 -- 172.21.15.27:0/2087948874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f690c068d30 msgr2=0x7f690c14a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.487+0000 7f690b59e700 1 --2- 172.21.15.27:0/2087948874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f690c068d30 0x7f690c14a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.487+0000 7f690b59e700 1 -- 172.21.15.27:0/2087948874 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f690c0b0bf0 con 0x7f690c0b2550 2024-01-11T08:29:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.488+0000 7f690b59e700 1 --2- 172.21.15.27:0/2087948874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f690c0b2550 0x7f690c0c6100 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f690c069130 tx=0x7f690400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.488+0000 7f68f2ffd700 1 -- 172.21.15.27:0/2087948874 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690400e040 con 0x7f690c0b2550 2024-01-11T08:29:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.488+0000 7f68f2ffd700 1 -- 172.21.15.27:0/2087948874 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f690400f040 con 0x7f690c0b2550 2024-01-11T08:29:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.488+0000 7f68f2ffd700 1 -- 172.21.15.27:0/2087948874 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690400fe20 con 0x7f690c0b2550 2024-01-11T08:29:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.488+0000 7f69111ff700 1 -- 172.21.15.27:0/2087948874 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f690c0b0ed0 con 0x7f690c0b2550 2024-01-11T08:29:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.488+0000 7f69111ff700 1 -- 172.21.15.27:0/2087948874 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f690c14a980 con 0x7f690c0b2550 2024-01-11T08:29:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.489+0000 7f69111ff700 1 -- 172.21.15.27:0/2087948874 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f690c0b13f0 con 0x7f690c0b2550 2024-01-11T08:29:49.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.490+0000 7f68f2ffd700 1 -- 172.21.15.27:0/2087948874 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 72) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f6904015640 con 0x7f690c0b2550 2024-01-11T08:29:49.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.492+0000 7f68f2ffd700 1 --2- 172.21.15.27:0/2087948874 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f68f80670a0 0x7f68f8069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:49.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.492+0000 7f68f2ffd700 1 -- 172.21.15.27:0/2087948874 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f69040850a0 con 0x7f690c0b2550 2024-01-11T08:29:49.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.492+0000 7f690ad9d700 1 --2- 172.21.15.27:0/2087948874 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f68f80670a0 0x7f68f8069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:49.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.493+0000 7f690ad9d700 1 --2- 172.21.15.27:0/2087948874 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f68f80670a0 0x7f68f8069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f68fc001ed0 tx=0x7f68fc015000 comp rx=0 tx=0).ready entity=mgr.7076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:49.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.496+0000 7f68f2ffd700 1 -- 172.21.15.27:0/2087948874 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f690c0b13f0 con 0x7f690c0b2550 2024-01-11T08:29:49.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.723+0000 7f69111ff700 1 -- 172.21.15.27:0/2087948874 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/weekly", "snap_schedule": "1w", "target": ["mon-mgr", ""]}) v1 -- 0x7f690c0b13f0 con 0x7f68f80670a0 2024-01-11T08:29:49.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.790+0000 7f68f2ffd700 1 -- 172.21.15.27:0/2087948874 <== mgr.7076 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f690c0b13f0 con 0x7f68f80670a0 2024-01-11T08:29:49.789 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path snap_test_dir1/weekly 2024-01-11T08:29:49.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.792+0000 7f69111ff700 1 -- 172.21.15.27:0/2087948874 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f68f80670a0 msgr2=0x7f68f8069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:49.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.792+0000 7f69111ff700 1 --2- 172.21.15.27:0/2087948874 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f68f80670a0 0x7f68f8069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f68fc001ed0 tx=0x7f68fc015000 comp rx=0 tx=0).stop 2024-01-11T08:29:49.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.792+0000 7f69111ff700 1 -- 172.21.15.27:0/2087948874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f690c0b2550 msgr2=0x7f690c0c6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:49.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.792+0000 7f69111ff700 1 --2- 172.21.15.27:0/2087948874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f690c0b2550 0x7f690c0c6100 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f690c069130 tx=0x7f690400b040 comp rx=0 tx=0).stop 2024-01-11T08:29:49.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.792+0000 7f69111ff700 1 -- 172.21.15.27:0/2087948874 shutdown_connections 2024-01-11T08:29:49.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.792+0000 7f69111ff700 1 --2- 172.21.15.27:0/2087948874 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f68f80670a0 0x7f68f8069550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:49.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.792+0000 7f69111ff700 1 --2- 172.21.15.27:0/2087948874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f690c0b2550 0x7f690c0c6100 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:49.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.792+0000 7f69111ff700 1 --2- 172.21.15.27:0/2087948874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f690c068d30 0x7f690c14a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:49.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.792+0000 7f69111ff700 1 --2- 172.21.15.27:0/2087948874 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f690c0b20e0 0x7f690c0c5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:49.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.792+0000 7f69111ff700 1 -- 172.21.15.27:0/2087948874 >> 172.21.15.27:0/2087948874 conn(0x7f690c0c9fb0 msgr2=0x7f690c158730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:49.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.792+0000 7f69111ff700 1 -- 172.21.15.27:0/2087948874 shutdown_connections 2024-01-11T08:29:49.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:49.792+0000 7f69111ff700 1 -- 172.21.15.27:0/2087948874 wait complete. 2024-01-11T08:29:49.809 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/weekly 2024-01-11T08:29:49.809 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:49.809 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/monthly) 2024-01-11T08:29:49.859 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/monthly 1M 2024-01-11T08:29:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.064+0000 7fd4455ff700 1 Processor -- start 2024-01-11T08:29:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.064+0000 7fd4455ff700 1 -- start start 2024-01-11T08:29:50.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.065+0000 7fd4455ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd44015a1b0 0x7fd44015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:50.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.065+0000 7fd4455ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd4400ccd70 0x7fd4400cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:50.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.065+0000 7fd4455ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4400cf850 con 0x7fd4400685c0 2024-01-11T08:29:50.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.066+0000 7fd4455ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4400cec40 con 0x7fd4400ccd70 2024-01-11T08:29:50.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.066+0000 7fd4455ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4400cef20 con 0x7fd44015a1b0 2024-01-11T08:29:50.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.066+0000 7fd43e7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd4400ccd70 0x7fd4400cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:50.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.066+0000 7fd43f7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd4400685c0 0x7fd440068a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43780/0 (socket says 172.21.15.27:43780) 2024-01-11T08:29:50.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.066+0000 7fd43f7fe700 1 -- 172.21.15.27:0/300639522 learned_addr learned my addr 172.21.15.27:0/300639522 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:50.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.066+0000 7fd43effd700 1 --2- 172.21.15.27:0/300639522 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd44015a1b0 0x7fd44015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:50.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.066+0000 7fd43e7fc700 1 -- 172.21.15.27:0/300639522 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd44015a1b0 msgr2=0x7fd44015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:50.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.066+0000 7fd43e7fc700 1 --2- 172.21.15.27:0/300639522 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd44015a1b0 0x7fd44015a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.066+0000 7fd43e7fc700 1 -- 172.21.15.27:0/300639522 >> v1:172.21.15.27:6789/0 conn(0x7fd4400685c0 legacy=0x7fd440068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:50.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.066+0000 7fd43e7fc700 1 -- 172.21.15.27:0/300639522 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4400cf200 con 0x7fd4400ccd70 2024-01-11T08:29:50.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.066+0000 7fd43dffb700 1 -- 172.21.15.27:0/300639522 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315558975 0 0) 0x7fd4400cf850 con 0x7fd4400685c0 2024-01-11T08:29:50.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.067+0000 7fd43e7fc700 1 --2- 172.21.15.27:0/300639522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4400ccd70 0x7fd4400cd1e0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fd4340098e0 tx=0x7fd4340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=268c108d50f59a98 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:50.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.067+0000 7fd43dffb700 1 -- 172.21.15.27:0/300639522 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43400b640 con 0x7fd4400ccd70 2024-01-11T08:29:50.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.067+0000 7fd43dffb700 1 -- 172.21.15.27:0/300639522 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd43400b7a0 con 0x7fd4400ccd70 2024-01-11T08:29:50.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.067+0000 7fd43dffb700 1 -- 172.21.15.27:0/300639522 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43400bb40 con 0x7fd4400ccd70 2024-01-11T08:29:50.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.067+0000 7fd4455ff700 1 -- 172.21.15.27:0/300639522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4400ccd70 msgr2=0x7fd4400cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:50.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.067+0000 7fd4455ff700 1 --2- 172.21.15.27:0/300639522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4400ccd70 0x7fd4400cd1e0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fd4340098e0 tx=0x7fd4340091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.067+0000 7fd4455ff700 1 -- 172.21.15.27:0/300639522 shutdown_connections 2024-01-11T08:29:50.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.067+0000 7fd4455ff700 1 --2- 172.21.15.27:0/300639522 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd44015a1b0 0x7fd44015a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.067+0000 7fd4455ff700 1 --2- 172.21.15.27:0/300639522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4400ccd70 0x7fd4400cd1e0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.067+0000 7fd4455ff700 1 -- 172.21.15.27:0/300639522 >> 172.21.15.27:0/300639522 conn(0x7fd44006ed40 msgr2=0x7fd44006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:50.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.068+0000 7fd4455ff700 1 -- 172.21.15.27:0/300639522 shutdown_connections 2024-01-11T08:29:50.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.068+0000 7fd4455ff700 1 -- 172.21.15.27:0/300639522 wait complete. 2024-01-11T08:29:50.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.069+0000 7fd4455ff700 1 Processor -- start 2024-01-11T08:29:50.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.069+0000 7fd4455ff700 1 -- start start 2024-01-11T08:29:50.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.070+0000 7fd4455ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4400685c0 0x7fd4401698e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:50.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.070+0000 7fd4455ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4400ccd70 0x7fd44014af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.070+0000 7fd4455ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd44015a1b0 0x7fd4400c9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.070+0000 7fd4455ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd440169e20 con 0x7fd4400ccd70 2024-01-11T08:29:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.070+0000 7fd4455ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd440157c50 con 0x7fd4400685c0 2024-01-11T08:29:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.070+0000 7fd4455ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd440157f30 con 0x7fd44015a1b0 2024-01-11T08:29:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.070+0000 7fd43e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4400ccd70 0x7fd44014af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.070+0000 7fd43e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4400ccd70 0x7fd44014af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54456/0 (socket says 172.21.15.27:54456) 2024-01-11T08:29:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.070+0000 7fd43e7fc700 1 -- 172.21.15.27:0/3861068098 learned_addr learned my addr 172.21.15.27:0/3861068098 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.071+0000 7fd43f7fe700 1 --2- 172.21.15.27:0/3861068098 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd44015a1b0 0x7fd4400c9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.071+0000 7fd43effd700 1 --2- 172.21.15.27:0/3861068098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4400685c0 0x7fd4401698e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.071+0000 7fd43e7fc700 1 -- 172.21.15.27:0/3861068098 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd44015a1b0 msgr2=0x7fd4400c9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.071+0000 7fd43e7fc700 1 --2- 172.21.15.27:0/3861068098 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd44015a1b0 0x7fd4400c9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.071+0000 7fd43e7fc700 1 -- 172.21.15.27:0/3861068098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4400685c0 msgr2=0x7fd4401698e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.071+0000 7fd43e7fc700 1 --2- 172.21.15.27:0/3861068098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4400685c0 0x7fd4401698e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.071+0000 7fd43e7fc700 1 -- 172.21.15.27:0/3861068098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd434009420 con 0x7fd4400ccd70 2024-01-11T08:29:50.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.071+0000 7fd43f7fe700 1 --2- 172.21.15.27:0/3861068098 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd44015a1b0 0x7fd4400c9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:50.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.071+0000 7fd43effd700 1 --2- 172.21.15.27:0/3861068098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4400685c0 0x7fd4401698e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:50.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.071+0000 7fd43e7fc700 1 --2- 172.21.15.27:0/3861068098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4400ccd70 0x7fd44014af80 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fd440069770 tx=0x7fd434017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:50.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.071+0000 7fd426ffd700 1 -- 172.21.15.27:0/3861068098 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43400bb20 con 0x7fd4400ccd70 2024-01-11T08:29:50.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.072+0000 7fd426ffd700 1 -- 172.21.15.27:0/3861068098 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd434011420 con 0x7fd4400ccd70 2024-01-11T08:29:50.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.072+0000 7fd4455ff700 1 -- 172.21.15.27:0/3861068098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd440158210 con 0x7fd4400ccd70 2024-01-11T08:29:50.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.072+0000 7fd426ffd700 1 -- 172.21.15.27:0/3861068098 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4340117c0 con 0x7fd4400ccd70 2024-01-11T08:29:50.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.072+0000 7fd4455ff700 1 -- 172.21.15.27:0/3861068098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd440158850 con 0x7fd4400ccd70 2024-01-11T08:29:50.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.073+0000 7fd4455ff700 1 -- 172.21.15.27:0/3861068098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4401589e0 con 0x7fd4400ccd70 2024-01-11T08:29:50.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.074+0000 7fd426ffd700 1 -- 172.21.15.27:0/3861068098 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 72) v1 ==== 84862+0+0 (secure 0 0 0) 0x7fd434011b70 con 0x7fd4400ccd70 2024-01-11T08:29:50.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.075+0000 7fd426ffd700 1 --2- 172.21.15.27:0/3861068098 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd42c0670a0 0x7fd42c069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:50.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.075+0000 7fd426ffd700 1 -- 172.21.15.27:0/3861068098 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7fd434085f20 con 0x7fd4400ccd70 2024-01-11T08:29:50.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.075+0000 7fd43effd700 1 --2- 172.21.15.27:0/3861068098 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd42c0670a0 0x7fd42c069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:50.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.076+0000 7fd43effd700 1 --2- 172.21.15.27:0/3861068098 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd42c0670a0 0x7fd42c069550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd4400b2ab0 tx=0x7fd430008040 comp rx=0 tx=0).ready entity=mgr.7076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:50.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.079+0000 7fd426ffd700 1 -- 172.21.15.27:0/3861068098 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd434052f40 con 0x7fd4400ccd70 2024-01-11T08:29:50.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.309+0000 7fd4455ff700 1 -- 172.21.15.27:0/3861068098 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/monthly", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd44004c3c0 con 0x7fd42c0670a0 2024-01-11T08:29:50.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.378+0000 7fd426ffd700 1 -- 172.21.15.27:0/3861068098 <== mgr.7076 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7fd44004c3c0 con 0x7fd42c0670a0 2024-01-11T08:29:50.377 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path snap_test_dir1/monthly 2024-01-11T08:29:50.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.380+0000 7fd4455ff700 1 -- 172.21.15.27:0/3861068098 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd42c0670a0 msgr2=0x7fd42c069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:50.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.380+0000 7fd4455ff700 1 --2- 172.21.15.27:0/3861068098 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd42c0670a0 0x7fd42c069550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd4400b2ab0 tx=0x7fd430008040 comp rx=0 tx=0).stop 2024-01-11T08:29:50.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.380+0000 7fd4455ff700 1 -- 172.21.15.27:0/3861068098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4400ccd70 msgr2=0x7fd44014af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:50.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.380+0000 7fd4455ff700 1 --2- 172.21.15.27:0/3861068098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4400ccd70 0x7fd44014af80 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fd440069770 tx=0x7fd434017000 comp rx=0 tx=0).stop 2024-01-11T08:29:50.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.380+0000 7fd4455ff700 1 -- 172.21.15.27:0/3861068098 shutdown_connections 2024-01-11T08:29:50.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.380+0000 7fd4455ff700 1 --2- 172.21.15.27:0/3861068098 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd42c0670a0 0x7fd42c069550 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.380+0000 7fd4455ff700 1 --2- 172.21.15.27:0/3861068098 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd44015a1b0 0x7fd4400c9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.380+0000 7fd4455ff700 1 --2- 172.21.15.27:0/3861068098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4400685c0 0x7fd4401698e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.380+0000 7fd4455ff700 1 --2- 172.21.15.27:0/3861068098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4400ccd70 0x7fd44014af80 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.380+0000 7fd4455ff700 1 -- 172.21.15.27:0/3861068098 >> 172.21.15.27:0/3861068098 conn(0x7fd44006ed40 msgr2=0x7fd440156120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:50.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.381+0000 7fd4455ff700 1 -- 172.21.15.27:0/3861068098 shutdown_connections 2024-01-11T08:29:50.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.381+0000 7fd4455ff700 1 -- 172.21.15.27:0/3861068098 wait complete. 2024-01-11T08:29:50.397 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/monthly 2024-01-11T08:29:50.397 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:50.397 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/yearly) 2024-01-11T08:29:50.447 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/yearly 1Y 2024-01-11T08:29:50.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.650+0000 7f3b5d3ff700 1 Processor -- start 2024-01-11T08:29:50.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.650+0000 7f3b5d3ff700 1 -- start start 2024-01-11T08:29:50.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.651+0000 7f3b5d3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b581596f0 0x7f3b580a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:50.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.651+0000 7f3b5d3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3b580a9de0 0x7f3b58068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:50.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.651+0000 7f3b5d3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b580c80a0 con 0x7f3b58159ac0 2024-01-11T08:29:50.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.651+0000 7f3b5d3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b580cbac0 con 0x7f3b580a9de0 2024-01-11T08:29:50.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.651+0000 7f3b5d3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b580c7490 con 0x7f3b581596f0 2024-01-11T08:29:50.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.652+0000 7f3b567fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3b580a9de0 0x7f3b58068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:50.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.652+0000 7f3b577fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3b58159ac0 0x7f3b580c8530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43782/0 (socket says 172.21.15.27:43782) 2024-01-11T08:29:50.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.652+0000 7f3b577fe700 1 -- 172.21.15.27:0/1213133455 learned_addr learned my addr 172.21.15.27:0/1213133455 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:50.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.652+0000 7f3b56ffd700 1 --2- 172.21.15.27:0/1213133455 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b581596f0 0x7f3b580a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:50.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.652+0000 7f3b567fc700 1 -- 172.21.15.27:0/1213133455 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b581596f0 msgr2=0x7f3b580a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:50.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.652+0000 7f3b567fc700 1 --2- 172.21.15.27:0/1213133455 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b581596f0 0x7f3b580a98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.652+0000 7f3b567fc700 1 -- 172.21.15.27:0/1213133455 >> v1:172.21.15.27:6789/0 conn(0x7f3b58159ac0 legacy=0x7f3b580c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:50.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.652+0000 7f3b567fc700 1 -- 172.21.15.27:0/1213133455 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b580c7770 con 0x7f3b580a9de0 2024-01-11T08:29:50.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.652+0000 7f3b56ffd700 1 --2- 172.21.15.27:0/1213133455 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b581596f0 0x7f3b580a98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:50.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.652+0000 7f3b55ffb700 1 -- 172.21.15.27:0/1213133455 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953659584 0 0) 0x7f3b580c80a0 con 0x7f3b58159ac0 2024-01-11T08:29:50.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.653+0000 7f3b567fc700 1 --2- 172.21.15.27:0/1213133455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b580a9de0 0x7f3b58068620 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c0098e0 tx=0x7f3b4c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c709792361cdb35 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:50.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.653+0000 7f3b55ffb700 1 -- 172.21.15.27:0/1213133455 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c009e80 con 0x7f3b580a9de0 2024-01-11T08:29:50.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.653+0000 7f3b55ffb700 1 -- 172.21.15.27:0/1213133455 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4c00b860 con 0x7f3b580a9de0 2024-01-11T08:29:50.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.653+0000 7f3b55ffb700 1 -- 172.21.15.27:0/1213133455 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c00bbe0 con 0x7f3b580a9de0 2024-01-11T08:29:50.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.653+0000 7f3b5d3ff700 1 -- 172.21.15.27:0/1213133455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b580a9de0 msgr2=0x7f3b58068620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:50.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.653+0000 7f3b5d3ff700 1 --2- 172.21.15.27:0/1213133455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b580a9de0 0x7f3b58068620 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c0098e0 tx=0x7f3b4c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.653+0000 7f3b5d3ff700 1 -- 172.21.15.27:0/1213133455 shutdown_connections 2024-01-11T08:29:50.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.653+0000 7f3b5d3ff700 1 --2- 172.21.15.27:0/1213133455 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b581596f0 0x7f3b580a98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.653+0000 7f3b5d3ff700 1 --2- 172.21.15.27:0/1213133455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b580a9de0 0x7f3b58068620 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.653+0000 7f3b5d3ff700 1 -- 172.21.15.27:0/1213133455 >> 172.21.15.27:0/1213133455 conn(0x7f3b580c9fb0 msgr2=0x7f3b5806ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:50.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.654+0000 7f3b5d3ff700 1 -- 172.21.15.27:0/1213133455 shutdown_connections 2024-01-11T08:29:50.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.654+0000 7f3b5d3ff700 1 -- 172.21.15.27:0/1213133455 wait complete. 2024-01-11T08:29:50.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.656+0000 7f3b5d3ff700 1 Processor -- start 2024-01-11T08:29:50.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.656+0000 7f3b5d3ff700 1 -- start start 2024-01-11T08:29:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.657+0000 7f3b5d3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b580a9de0 0x7f3b580b0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.657+0000 7f3b5d3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b581596f0 0x7f3b580ae690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.657+0000 7f3b5d3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b58159ac0 0x7f3b5806b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.657+0000 7f3b5d3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b580b1750 con 0x7f3b581596f0 2024-01-11T08:29:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.657+0000 7f3b5d3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b580afa50 con 0x7f3b580a9de0 2024-01-11T08:29:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.657+0000 7f3b5d3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b580afd30 con 0x7f3b58159ac0 2024-01-11T08:29:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.657+0000 7f3b567fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b581596f0 0x7f3b580ae690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.657+0000 7f3b567fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b581596f0 0x7f3b580ae690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54470/0 (socket says 172.21.15.27:54470) 2024-01-11T08:29:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.657+0000 7f3b567fc700 1 -- 172.21.15.27:0/2905163627 learned_addr learned my addr 172.21.15.27:0/2905163627 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.658+0000 7f3b567fc700 1 -- 172.21.15.27:0/2905163627 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b58159ac0 msgr2=0x7f3b5806b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.658+0000 7f3b567fc700 1 --2- 172.21.15.27:0/2905163627 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b58159ac0 0x7f3b5806b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.658+0000 7f3b567fc700 1 -- 172.21.15.27:0/2905163627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b580a9de0 msgr2=0x7f3b580b0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.658+0000 7f3b567fc700 1 --2- 172.21.15.27:0/2905163627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b580a9de0 0x7f3b580b0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.658+0000 7f3b567fc700 1 -- 172.21.15.27:0/2905163627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4c009420 con 0x7f3b581596f0 2024-01-11T08:29:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.658+0000 7f3b567fc700 1 --2- 172.21.15.27:0/2905163627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b581596f0 0x7f3b580ae690 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f3b5806d730 tx=0x7f3b4c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.658+0000 7f3b3effd700 1 -- 172.21.15.27:0/2905163627 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c00bbe0 con 0x7f3b581596f0 2024-01-11T08:29:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.658+0000 7f3b3effd700 1 -- 172.21.15.27:0/2905163627 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4c011420 con 0x7f3b581596f0 2024-01-11T08:29:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.658+0000 7f3b3effd700 1 -- 172.21.15.27:0/2905163627 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c0117a0 con 0x7f3b581596f0 2024-01-11T08:29:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.658+0000 7f3b5d3ff700 1 -- 172.21.15.27:0/2905163627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b580b0010 con 0x7f3b581596f0 2024-01-11T08:29:50.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.659+0000 7f3b5d3ff700 1 -- 172.21.15.27:0/2905163627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b580b1460 con 0x7f3b581596f0 2024-01-11T08:29:50.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.659+0000 7f3b5d3ff700 1 -- 172.21.15.27:0/2905163627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b580b0530 con 0x7f3b581596f0 2024-01-11T08:29:50.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.660+0000 7f3b3effd700 1 -- 172.21.15.27:0/2905163627 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 72) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f3b4c011bd0 con 0x7f3b581596f0 2024-01-11T08:29:50.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.662+0000 7f3b3effd700 1 --2- 172.21.15.27:0/2905163627 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3b440670a0 0x7f3b44069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:50.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.662+0000 7f3b3effd700 1 -- 172.21.15.27:0/2905163627 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f3b4c085d90 con 0x7f3b581596f0 2024-01-11T08:29:50.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.662+0000 7f3b56ffd700 1 --2- 172.21.15.27:0/2905163627 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3b440670a0 0x7f3b44069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:50.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.663+0000 7f3b56ffd700 1 --2- 172.21.15.27:0/2905163627 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3b440670a0 0x7f3b44069550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3b48006fd0 tx=0x7f3b48008040 comp rx=0 tx=0).ready entity=mgr.7076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:50.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.666+0000 7f3b3effd700 1 -- 172.21.15.27:0/2905163627 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3b580b0530 con 0x7f3b581596f0 2024-01-11T08:29:50.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.896+0000 7f3b5d3ff700 1 -- 172.21.15.27:0/2905163627 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/yearly", "snap_schedule": "1Y", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b580b0530 con 0x7f3b440670a0 2024-01-11T08:29:50.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.962+0000 7f3b3effd700 1 -- 172.21.15.27:0/2905163627 <== mgr.7076 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f3b580b0530 con 0x7f3b440670a0 2024-01-11T08:29:50.962 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path snap_test_dir1/yearly 2024-01-11T08:29:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.965+0000 7f3b5d3ff700 1 -- 172.21.15.27:0/2905163627 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3b440670a0 msgr2=0x7f3b44069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.965+0000 7f3b5d3ff700 1 --2- 172.21.15.27:0/2905163627 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3b440670a0 0x7f3b44069550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3b48006fd0 tx=0x7f3b48008040 comp rx=0 tx=0).stop 2024-01-11T08:29:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.965+0000 7f3b5d3ff700 1 -- 172.21.15.27:0/2905163627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b581596f0 msgr2=0x7f3b580ae690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.965+0000 7f3b5d3ff700 1 --2- 172.21.15.27:0/2905163627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b581596f0 0x7f3b580ae690 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f3b5806d730 tx=0x7f3b4c017000 comp rx=0 tx=0).stop 2024-01-11T08:29:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.965+0000 7f3b5d3ff700 1 -- 172.21.15.27:0/2905163627 shutdown_connections 2024-01-11T08:29:50.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.965+0000 7f3b5d3ff700 1 --2- 172.21.15.27:0/2905163627 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f3b440670a0 0x7f3b44069550 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.965+0000 7f3b5d3ff700 1 --2- 172.21.15.27:0/2905163627 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b58159ac0 0x7f3b5806b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.965+0000 7f3b5d3ff700 1 --2- 172.21.15.27:0/2905163627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b580a9de0 0x7f3b580b0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.965+0000 7f3b5d3ff700 1 --2- 172.21.15.27:0/2905163627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b581596f0 0x7f3b580ae690 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:50.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.965+0000 7f3b5d3ff700 1 -- 172.21.15.27:0/2905163627 >> 172.21.15.27:0/2905163627 conn(0x7f3b580c9fb0 msgr2=0x7f3b580c1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:50.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.965+0000 7f3b5d3ff700 1 -- 172.21.15.27:0/2905163627 shutdown_connections 2024-01-11T08:29:50.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:50.965+0000 7f3b5d3ff700 1 -- 172.21.15.27:0/2905163627 wait complete. 2024-01-11T08:29:50.982 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/yearly 2024-01-11T08:29:50.982 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:50.983 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/bad_period_spec) 2024-01-11T08:29:51.034 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1X 2024-01-11T08:29:51.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.237+0000 7f85d65ff700 1 Processor -- start 2024-01-11T08:29:51.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.237+0000 7f85d65ff700 1 -- start start 2024-01-11T08:29:51.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.238+0000 7f85d65ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f85d0157410 0x7f85d00ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:51.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.238+0000 7f85d65ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f85d01577e0 0x7f85d00ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:51.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.238+0000 7f85d65ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d00699d0 con 0x7f85d0069540 2024-01-11T08:29:51.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.238+0000 7f85d65ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d0069cb0 con 0x7f85d01577e0 2024-01-11T08:29:51.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.238+0000 7f85d65ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d0069f90 con 0x7f85d0157410 2024-01-11T08:29:51.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.239+0000 7f85cf7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f85d01577e0 0x7f85d00ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:51.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.239+0000 7f85d4b9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f85d0069540 0x7f85d015b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43792/0 (socket says 172.21.15.27:43792) 2024-01-11T08:29:51.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.239+0000 7f85d4b9c700 1 -- 172.21.15.27:0/2666650399 learned_addr learned my addr 172.21.15.27:0/2666650399 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:51.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.239+0000 7f85cffff700 1 --2- 172.21.15.27:0/2666650399 >> v2:172.21.15.77:3300/0 conn(0x7f85d0157410 0x7f85d00ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:51.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.239+0000 7f85ceffd700 1 -- 172.21.15.27:0/2666650399 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837767214 0 0) 0x7f85d00699d0 con 0x7f85d0069540 2024-01-11T08:29:51.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.239+0000 7f85ceffd700 1 -- 172.21.15.27:0/2666650399 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85b8003570 con 0x7f85d0069540 2024-01-11T08:29:51.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.239+0000 7f85cf7fe700 1 -- 172.21.15.27:0/2666650399 >> v2:172.21.15.77:3300/0 conn(0x7f85d0157410 msgr2=0x7f85d00ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:51.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.239+0000 7f85cf7fe700 1 --2- 172.21.15.27:0/2666650399 >> v2:172.21.15.77:3300/0 conn(0x7f85d0157410 0x7f85d00ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:51.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.239+0000 7f85cf7fe700 1 -- 172.21.15.27:0/2666650399 >> v1:172.21.15.27:6789/0 conn(0x7f85d0069540 legacy=0x7f85d015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:51.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.239+0000 7f85cf7fe700 1 -- 172.21.15.27:0/2666650399 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d006a270 con 0x7f85d01577e0 2024-01-11T08:29:51.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.240+0000 7f85cf7fe700 1 --2- 172.21.15.27:0/2666650399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85d01577e0 0x7f85d00ae000 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f85d0070c90 tx=0x7f85c40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f62d2e6e47df017 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:51.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.240+0000 7f85ceffd700 1 -- 172.21.15.27:0/2666650399 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c4011630 con 0x7f85d01577e0 2024-01-11T08:29:51.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.240+0000 7f85ceffd700 1 -- 172.21.15.27:0/2666650399 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c40117b0 con 0x7f85d01577e0 2024-01-11T08:29:51.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.240+0000 7f85ceffd700 1 -- 172.21.15.27:0/2666650399 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c4011b70 con 0x7f85d01577e0 2024-01-11T08:29:51.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.240+0000 7f85d65ff700 1 -- 172.21.15.27:0/2666650399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85d01577e0 msgr2=0x7f85d00ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:51.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.240+0000 7f85d65ff700 1 --2- 172.21.15.27:0/2666650399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85d01577e0 0x7f85d00ae000 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f85d0070c90 tx=0x7f85c40091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:51.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.240+0000 7f85d65ff700 1 -- 172.21.15.27:0/2666650399 shutdown_connections 2024-01-11T08:29:51.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.240+0000 7f85d65ff700 1 --2- 172.21.15.27:0/2666650399 >> v2:172.21.15.77:3300/0 conn(0x7f85d0157410 0x7f85d00ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:51.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.240+0000 7f85d65ff700 1 --2- 172.21.15.27:0/2666650399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85d01577e0 0x7f85d00ae000 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:51.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.240+0000 7f85d65ff700 1 -- 172.21.15.27:0/2666650399 >> 172.21.15.27:0/2666650399 conn(0x7f85d00680a0 msgr2=0x7f85d00cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:51.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.241+0000 7f85d65ff700 1 -- 172.21.15.27:0/2666650399 shutdown_connections 2024-01-11T08:29:51.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.241+0000 7f85d65ff700 1 -- 172.21.15.27:0/2666650399 wait complete. 2024-01-11T08:29:51.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.242+0000 7f85d65ff700 1 Processor -- start 2024-01-11T08:29:51.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.242+0000 7f85d65ff700 1 -- start start 2024-01-11T08:29:51.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.243+0000 7f85d65ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85d0069540 0x7f85d00c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:51.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.243+0000 7f85d65ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f85d0157410 0x7f85d0149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.243+0000 7f85d65ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85d01577e0 0x7f85d014b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.243+0000 7f85d65ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d015cbd0 con 0x7f85d0069540 2024-01-11T08:29:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.243+0000 7f85d65ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d00c1a80 con 0x7f85d01577e0 2024-01-11T08:29:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.243+0000 7f85d65ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d00c5c50 con 0x7f85d0157410 2024-01-11T08:29:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.243+0000 7f85cffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85d0069540 0x7f85d00c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.243+0000 7f85cffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85d0069540 0x7f85d00c53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54492/0 (socket says 172.21.15.27:54492) 2024-01-11T08:29:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.243+0000 7f85cffff700 1 -- 172.21.15.27:0/3481043221 learned_addr learned my addr 172.21.15.27:0/3481043221 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.244+0000 7f85cf7fe700 1 --2- 172.21.15.27:0/3481043221 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f85d0157410 0x7f85d0149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.244+0000 7f85d4b9c700 1 --2- 172.21.15.27:0/3481043221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85d01577e0 0x7f85d014b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.244+0000 7f85cffff700 1 -- 172.21.15.27:0/3481043221 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f85d0157410 msgr2=0x7f85d0149b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.244+0000 7f85cffff700 1 --2- 172.21.15.27:0/3481043221 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f85d0157410 0x7f85d0149b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.244+0000 7f85cffff700 1 -- 172.21.15.27:0/3481043221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85d01577e0 msgr2=0x7f85d014b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.244+0000 7f85cffff700 1 --2- 172.21.15.27:0/3481043221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85d01577e0 0x7f85d014b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:51.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.244+0000 7f85cffff700 1 -- 172.21.15.27:0/3481043221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c4009420 con 0x7f85d0069540 2024-01-11T08:29:51.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.244+0000 7f85d4b9c700 1 --2- 172.21.15.27:0/3481043221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85d01577e0 0x7f85d014b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:51.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.244+0000 7f85cffff700 1 --2- 172.21.15.27:0/3481043221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85d0069540 0x7f85d00c53b0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f85c000ec80 tx=0x7f85c0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:51.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.244+0000 7f85b7fff700 1 -- 172.21.15.27:0/3481043221 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c000c850 con 0x7f85d0069540 2024-01-11T08:29:51.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.244+0000 7f85b7fff700 1 -- 172.21.15.27:0/3481043221 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c000c9b0 con 0x7f85d0069540 2024-01-11T08:29:51.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.244+0000 7f85b7fff700 1 -- 172.21.15.27:0/3481043221 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c000cd30 con 0x7f85d0069540 2024-01-11T08:29:51.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.244+0000 7f85d65ff700 1 -- 172.21.15.27:0/3481043221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d00c5f30 con 0x7f85d0069540 2024-01-11T08:29:51.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.245+0000 7f85d65ff700 1 -- 172.21.15.27:0/3481043221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d00c1710 con 0x7f85d0069540 2024-01-11T08:29:51.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.246+0000 7f85d65ff700 1 -- 172.21.15.27:0/3481043221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d004c3c0 con 0x7f85d0069540 2024-01-11T08:29:51.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.246+0000 7f85b7fff700 1 -- 172.21.15.27:0/3481043221 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 73) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f85c0013680 con 0x7f85d0069540 2024-01-11T08:29:51.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.247+0000 7f85b7fff700 1 --2- 172.21.15.27:0/3481043221 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f85bc067130 0x7f85bc0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:51.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.247+0000 7f85b7fff700 1 -- 172.21.15.27:0/3481043221 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f85c0085110 con 0x7f85d0069540 2024-01-11T08:29:51.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.248+0000 7f85cf7fe700 1 --2- 172.21.15.27:0/3481043221 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f85bc067130 0x7f85bc0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:51.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.249+0000 7f85cf7fe700 1 --2- 172.21.15.27:0/3481043221 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f85bc067130 0x7f85bc0695e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f85d006d560 tx=0x7f85c4017000 comp rx=0 tx=0).ready entity=mgr.7076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:51.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.252+0000 7f85b7fff700 1 -- 172.21.15.27:0/3481043221 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f85c0051f20 con 0x7f85d0069540 2024-01-11T08:29:51.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.479+0000 7f85d65ff700 1 -- 172.21.15.27:0/3481043221 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1X", "target": ["mon-mgr", ""]}) v1 -- 0x7f85d00c1bc0 con 0x7f85bc067130 2024-01-11T08:29:51.480 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:51.480+0000 7f5d4424d700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - multiplier should be one of h,d,w,M,Y 2024-01-11T08:29:51.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.481+0000 7f85b7fff700 1 -- 172.21.15.27:0/3481043221 <== mgr.7076 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - multiplier should be one of h,d,w,M,Y) v1 ==== 74+0+0 (secure 0 0 0) 0x7f85c4020040 con 0x7f85bc067130 2024-01-11T08:29:51.482 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENOENT: invalid schedule specified - multiplier should be one of h,d,w,M,Y 2024-01-11T08:29:51.490 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-11T08:29:51.491 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1MM 2024-01-11T08:29:51.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.693+0000 7f7fa8bff700 1 Processor -- start 2024-01-11T08:29:51.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.693+0000 7f7fa8bff700 1 -- start start 2024-01-11T08:29:51.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.694+0000 7f7fa8bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7fa4157410 0x7f7fa40ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:51.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.694+0000 7f7fa8bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7fa4069540 0x7f7fa40c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:51.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.694+0000 7f7fa8bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa40699d0 con 0x7f7fa41577e0 2024-01-11T08:29:51.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.694+0000 7f7fa8bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fa4069cb0 con 0x7f7fa4069540 2024-01-11T08:29:51.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.694+0000 7f7fa8bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa4069f90 con 0x7f7fa4157410 2024-01-11T08:29:51.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.695+0000 7f7fa259c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7fa4157410 0x7f7fa40ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:51.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.695+0000 7f7fa2d9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7fa4069540 0x7f7fa40c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:51.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.695+0000 7f7fa1d9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7fa41577e0 0x7f7fa40ae000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:48756/0 (socket says 172.21.15.27:48756) 2024-01-11T08:29:51.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.695+0000 7f7fa1d9b700 1 -- 172.21.15.27:0/459152250 learned_addr learned my addr 172.21.15.27:0/459152250 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:51.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.695+0000 7f7fa2d9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7fa4069540 0x7f7fa40c6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:44674/0 (socket says 172.21.15.27:44674) 2024-01-11T08:29:51.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.695+0000 7f7fa259c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7fa4157410 0x7f7fa40ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:44202/0 (socket says 172.21.15.27:44202) 2024-01-11T08:29:51.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.695+0000 7f7fa2d9d700 1 -- 172.21.15.27:0/459152250 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7fa4157410 msgr2=0x7f7fa40ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:51.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.695+0000 7f7fa2d9d700 1 --2- 172.21.15.27:0/459152250 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7fa4157410 0x7f7fa40ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:51.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.695+0000 7f7fa2d9d700 1 -- 172.21.15.27:0/459152250 >> v1:172.21.15.77:6789/0 conn(0x7f7fa41577e0 legacy=0x7f7fa40ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:51.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.696+0000 7f7fa2d9d700 1 -- 172.21.15.27:0/459152250 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa406a270 con 0x7f7fa4069540 2024-01-11T08:29:51.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.695+0000 7f7fa159a700 1 -- 172.21.15.27:0/459152250 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494391519 0 0) 0x7f7fa40699d0 con 0x7f7fa41577e0 2024-01-11T08:29:51.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.696+0000 7f7fa2d9d700 1 --2- 172.21.15.27:0/459152250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fa4069540 0x7f7fa40c6ba0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c009e60 tx=0x7f7f9c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74470ad1f7145638 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.696+0000 7f7fa159a700 1 -- 172.21.15.27:0/459152250 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c00e070 con 0x7f7fa4069540 2024-01-11T08:29:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.696+0000 7f7fa159a700 1 -- 172.21.15.27:0/459152250 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9c00bb60 con 0x7f7fa4069540 2024-01-11T08:29:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.696+0000 7f7fa159a700 1 -- 172.21.15.27:0/459152250 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c012430 con 0x7f7fa4069540 2024-01-11T08:29:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.697+0000 7f7fa8bff700 1 -- 172.21.15.27:0/459152250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fa4069540 msgr2=0x7f7fa40c6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.697+0000 7f7fa8bff700 1 --2- 172.21.15.27:0/459152250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fa4069540 0x7f7fa40c6ba0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c009e60 tx=0x7f7f9c009780 comp rx=0 tx=0).stop 2024-01-11T08:29:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.697+0000 7f7fa8bff700 1 -- 172.21.15.27:0/459152250 shutdown_connections 2024-01-11T08:29:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.697+0000 7f7fa8bff700 1 --2- 172.21.15.27:0/459152250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fa4069540 0x7f7fa40c6ba0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.697+0000 7f7fa8bff700 1 --2- 172.21.15.27:0/459152250 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7fa4157410 0x7f7fa40ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.697+0000 7f7fa8bff700 1 -- 172.21.15.27:0/459152250 >> 172.21.15.27:0/459152250 conn(0x7f7fa40680a0 msgr2=0x7f7fa40cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.697+0000 7f7fa8bff700 1 -- 172.21.15.27:0/459152250 shutdown_connections 2024-01-11T08:29:51.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.697+0000 7f7fa8bff700 1 -- 172.21.15.27:0/459152250 wait complete. 2024-01-11T08:29:51.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.698+0000 7f7fa8bff700 1 Processor -- start 2024-01-11T08:29:51.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.699+0000 7f7fa8bff700 1 -- start start 2024-01-11T08:29:51.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.700+0000 7f7fa8bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fa4069540 0x7f7fa40c4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:51.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.700+0000 7f7fa8bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fa4157410 0x7f7fa40c5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:51.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.700+0000 7f7fa8bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7fa41577e0 0x7f7fa414e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:51.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.700+0000 7f7fa8bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa414ec30 con 0x7f7fa4069540 2024-01-11T08:29:51.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.700+0000 7f7fa8bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa414ee20 con 0x7f7fa4157410 2024-01-11T08:29:51.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.700+0000 7f7fa8bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa414f100 con 0x7f7fa41577e0 2024-01-11T08:29:51.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.700+0000 7f7fa259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fa4069540 0x7f7fa40c4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:51.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.700+0000 7f7fa259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fa4069540 0x7f7fa40c4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54498/0 (socket says 172.21.15.27:54498) 2024-01-11T08:29:51.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.700+0000 7f7fa259c700 1 -- 172.21.15.27:0/519199754 learned_addr learned my addr 172.21.15.27:0/519199754 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:51.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.700+0000 7f7fa1d9b700 1 --2- 172.21.15.27:0/519199754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fa4157410 0x7f7fa40c5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:51.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.700+0000 7f7fa2d9d700 1 --2- 172.21.15.27:0/519199754 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7fa41577e0 0x7f7fa414e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:51.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.700+0000 7f7fa259c700 1 -- 172.21.15.27:0/519199754 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7fa41577e0 msgr2=0x7f7fa414e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:51.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.700+0000 7f7fa259c700 1 --2- 172.21.15.27:0/519199754 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7fa41577e0 0x7f7fa414e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:51.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.700+0000 7f7fa259c700 1 -- 172.21.15.27:0/519199754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fa4157410 msgr2=0x7f7fa40c5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:51.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.700+0000 7f7fa259c700 1 --2- 172.21.15.27:0/519199754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fa4157410 0x7f7fa40c5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:51.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.700+0000 7f7fa259c700 1 -- 172.21.15.27:0/519199754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9c009a20 con 0x7f7fa4069540 2024-01-11T08:29:51.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.701+0000 7f7fa259c700 1 --2- 172.21.15.27:0/519199754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fa4069540 0x7f7fa40c4cb0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f7f9400ec80 tx=0x7f7f94009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:51.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.701+0000 7f7f927fc700 1 -- 172.21.15.27:0/519199754 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9400c9d0 con 0x7f7fa4069540 2024-01-11T08:29:51.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.701+0000 7f7f927fc700 1 -- 172.21.15.27:0/519199754 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9400cb30 con 0x7f7fa4069540 2024-01-11T08:29:51.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.701+0000 7f7fa8bff700 1 -- 172.21.15.27:0/519199754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa414abc0 con 0x7f7fa4069540 2024-01-11T08:29:51.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.701+0000 7f7f927fc700 1 -- 172.21.15.27:0/519199754 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f94013470 con 0x7f7fa4069540 2024-01-11T08:29:51.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.701+0000 7f7fa8bff700 1 -- 172.21.15.27:0/519199754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa414a7c0 con 0x7f7fa4069540 2024-01-11T08:29:51.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.702+0000 7f7fa8bff700 1 -- 172.21.15.27:0/519199754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa404c3c0 con 0x7f7fa4069540 2024-01-11T08:29:51.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.703+0000 7f7f927fc700 1 -- 172.21.15.27:0/519199754 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 73) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f7f94013860 con 0x7f7fa4069540 2024-01-11T08:29:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.704+0000 7f7f927fc700 1 --2- 172.21.15.27:0/519199754 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7f84067130 0x7f7f840695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.704+0000 7f7f927fc700 1 -- 172.21.15.27:0/519199754 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f7f94085140 con 0x7f7fa4069540 2024-01-11T08:29:51.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.705+0000 7f7fa1d9b700 1 --2- 172.21.15.27:0/519199754 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7f84067130 0x7f7f840695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.705+0000 7f7fa1d9b700 1 --2- 172.21.15.27:0/519199754 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7f84067130 0x7f7f840695e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7fa406d560 tx=0x7f7f98009040 comp rx=0 tx=0).ready entity=mgr.7076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:51.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.708+0000 7f7f927fc700 1 -- 172.21.15.27:0/519199754 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7f94051ff0 con 0x7f7fa4069540 2024-01-11T08:29:51.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.935+0000 7f7fa8bff700 1 -- 172.21.15.27:0/519199754 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1MM", "target": ["mon-mgr", ""]}) v1 -- 0x7f7fa406e630 con 0x7f7f84067130 2024-01-11T08:29:51.935 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:51.936+0000 7f5d4424d700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-01-11T08:29:51.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:51.938+0000 7f7f927fc700 1 -- 172.21.15.27:0/519199754 <== mgr.7076 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7fa406e630 con 0x7f7f84067130 2024-01-11T08:29:51.938 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-01-11T08:29:51.945 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-11T08:29:51.946 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1 2024-01-11T08:29:52.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.150+0000 7f2483fff700 1 Processor -- start 2024-01-11T08:29:52.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.151+0000 7f2483fff700 1 -- start start 2024-01-11T08:29:52.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.152+0000 7f2483fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f247c0b0ab0 0x7f247c159780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:52.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.152+0000 7f2483fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247c0b0620 con 0x7f247c1593b0 2024-01-11T08:29:52.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.152+0000 7f2483fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247c0b2cf0 con 0x7f247c068490 2024-01-11T08:29:52.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.152+0000 7f2483fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247c0b2b10 con 0x7f247c0b0ab0 2024-01-11T08:29:52.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.152+0000 7f2481d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f247c1593b0 0x7f247c0b2400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43800/0 (socket says 172.21.15.27:43800) 2024-01-11T08:29:52.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.152+0000 7f2481d9b700 1 -- 172.21.15.27:0/3226218804 learned_addr learned my addr 172.21.15.27:0/3226218804 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:52.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.152+0000 7f248159a700 1 --2- 172.21.15.27:0/3226218804 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f247c0b0ab0 0x7f247c159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:52.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.153+0000 7f2480d99700 1 -- 172.21.15.27:0/3226218804 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356862131 0 0) 0x7f247c0b0620 con 0x7f247c1593b0 2024-01-11T08:29:52.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.153+0000 7f2480d99700 1 -- 172.21.15.27:0/3226218804 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2468003570 con 0x7f247c1593b0 2024-01-11T08:29:52.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.153+0000 7f2480d99700 1 -- 172.21.15.27:0/3226218804 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981745163 0 0) 0x7f247c0b2cf0 con 0x7f247c068490 2024-01-11T08:29:52.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.153+0000 7f2480d99700 1 -- 172.21.15.27:0/3226218804 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f247c0b0620 con 0x7f247c068490 2024-01-11T08:29:52.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.153+0000 7f248159a700 1 -- 172.21.15.27:0/3226218804 >> v1:172.21.15.77:6789/0 conn(0x7f247c068490 legacy=0x7f247c0cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:52.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.153+0000 7f248159a700 1 -- 172.21.15.27:0/3226218804 >> v1:172.21.15.27:6789/0 conn(0x7f247c1593b0 legacy=0x7f247c0b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:52.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.153+0000 7f2480d99700 1 -- 172.21.15.27:0/3226218804 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3975815217 0 0) 0x7f2468003570 con 0x7f247c1593b0 2024-01-11T08:29:52.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.153+0000 7f248159a700 1 -- 172.21.15.27:0/3226218804 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247c0afcf0 con 0x7f247c0b0ab0 2024-01-11T08:29:52.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.153+0000 7f248159a700 1 --2- 172.21.15.27:0/3226218804 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f247c0b0ab0 0x7f247c159780 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f246c009910 tx=0x7f246c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c13a02d20a399d65 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:52.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.154+0000 7f2480d99700 1 -- 172.21.15.27:0/3226218804 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2478001470 con 0x7f247c0b0ab0 2024-01-11T08:29:52.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.154+0000 7f2480d99700 1 -- 172.21.15.27:0/3226218804 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246c011420 con 0x7f247c0b0ab0 2024-01-11T08:29:52.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.154+0000 7f2480d99700 1 -- 172.21.15.27:0/3226218804 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c0117c0 con 0x7f247c0b0ab0 2024-01-11T08:29:52.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.154+0000 7f2483fff700 1 -- 172.21.15.27:0/3226218804 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f247c0b0ab0 msgr2=0x7f247c159780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:52.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.154+0000 7f2483fff700 1 --2- 172.21.15.27:0/3226218804 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f247c0b0ab0 0x7f247c159780 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f246c009910 tx=0x7f246c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:52.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.154+0000 7f2483fff700 1 -- 172.21.15.27:0/3226218804 shutdown_connections 2024-01-11T08:29:52.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.154+0000 7f2483fff700 1 --2- 172.21.15.27:0/3226218804 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f247c0b0ab0 0x7f247c159780 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:52.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.154+0000 7f2483fff700 1 -- 172.21.15.27:0/3226218804 >> 172.21.15.27:0/3226218804 conn(0x7f247c06ed40 msgr2=0x7f247c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:52.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.154+0000 7f2483fff700 1 -- 172.21.15.27:0/3226218804 shutdown_connections 2024-01-11T08:29:52.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.154+0000 7f2483fff700 1 -- 172.21.15.27:0/3226218804 wait complete. 2024-01-11T08:29:52.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.156+0000 7f2483fff700 1 Processor -- start 2024-01-11T08:29:52.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.156+0000 7f2483fff700 1 -- start start 2024-01-11T08:29:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.157+0000 7f2483fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f247c068490 0x7f247c154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.157+0000 7f2483fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f247c0b0ab0 0x7f247c0c48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.157+0000 7f2483fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f247c1593b0 0x7f247c1577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.157+0000 7f2483fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247c0b42d0 con 0x7f247c068490 2024-01-11T08:29:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.157+0000 7f2483fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247c0b4760 con 0x7f247c1593b0 2024-01-11T08:29:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.157+0000 7f2483fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247c0b36c0 con 0x7f247c0b0ab0 2024-01-11T08:29:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.157+0000 7f2481d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f247c068490 0x7f247c154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.157+0000 7f2481d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f247c068490 0x7f247c154f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54514/0 (socket says 172.21.15.27:54514) 2024-01-11T08:29:52.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.157+0000 7f2481d9b700 1 -- 172.21.15.27:0/1291236363 learned_addr learned my addr 172.21.15.27:0/1291236363 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:52.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.157+0000 7f248159a700 1 --2- 172.21.15.27:0/1291236363 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f247c0b0ab0 0x7f247c0c48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:52.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.157+0000 7f248259c700 1 --2- 172.21.15.27:0/1291236363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f247c1593b0 0x7f247c1577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:52.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.158+0000 7f2481d9b700 1 -- 172.21.15.27:0/1291236363 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f247c0b0ab0 msgr2=0x7f247c0c48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:52.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.158+0000 7f2481d9b700 1 --2- 172.21.15.27:0/1291236363 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f247c0b0ab0 0x7f247c0c48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:52.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.158+0000 7f2481d9b700 1 -- 172.21.15.27:0/1291236363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f247c1593b0 msgr2=0x7f247c1577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:52.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.158+0000 7f2481d9b700 1 --2- 172.21.15.27:0/1291236363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f247c1593b0 0x7f247c1577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:52.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.158+0000 7f2481d9b700 1 -- 172.21.15.27:0/1291236363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246c009420 con 0x7f247c068490 2024-01-11T08:29:52.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.158+0000 7f248259c700 1 --2- 172.21.15.27:0/1291236363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f247c1593b0 0x7f247c1577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:29:52.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.158+0000 7f2481d9b700 1 --2- 172.21.15.27:0/1291236363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f247c068490 0x7f247c154f70 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f247800cb90 tx=0x7f2478001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:52.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.158+0000 7f2471ffb700 1 -- 172.21.15.27:0/1291236363 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2478001d00 con 0x7f247c068490 2024-01-11T08:29:52.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.158+0000 7f2471ffb700 1 -- 172.21.15.27:0/1291236363 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2478007360 con 0x7f247c068490 2024-01-11T08:29:52.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.158+0000 7f2471ffb700 1 -- 172.21.15.27:0/1291236363 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2478004830 con 0x7f247c068490 2024-01-11T08:29:52.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.158+0000 7f2483fff700 1 -- 172.21.15.27:0/1291236363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f247c0b39a0 con 0x7f247c068490 2024-01-11T08:29:52.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.158+0000 7f2483fff700 1 -- 172.21.15.27:0/1291236363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247c0b3f60 con 0x7f247c068490 2024-01-11T08:29:52.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.160+0000 7f2483fff700 1 -- 172.21.15.27:0/1291236363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247c04c3c0 con 0x7f247c068490 2024-01-11T08:29:52.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.160+0000 7f2471ffb700 1 -- 172.21.15.27:0/1291236363 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 73) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f2478004be0 con 0x7f247c068490 2024-01-11T08:29:52.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.161+0000 7f2471ffb700 1 --2- 172.21.15.27:0/1291236363 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2460067130 0x7f24600695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:52.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.161+0000 7f2471ffb700 1 -- 172.21.15.27:0/1291236363 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f24780842f0 con 0x7f247c068490 2024-01-11T08:29:52.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.162+0000 7f248159a700 1 --2- 172.21.15.27:0/1291236363 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2460067130 0x7f24600695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:52.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.163+0000 7f248159a700 1 --2- 172.21.15.27:0/1291236363 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2460067130 0x7f24600695e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f246c0030c0 tx=0x7f246c016000 comp rx=0 tx=0).ready entity=mgr.7076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:52.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.165+0000 7f2471ffb700 1 -- 172.21.15.27:0/1291236363 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f24780511a0 con 0x7f247c068490 2024-01-11T08:29:52.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.398+0000 7f2483fff700 1 -- 172.21.15.27:0/1291236363 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f247c0b4410 con 0x7f2460067130 2024-01-11T08:29:52.398 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:52.398+0000 7f5d4424d700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-01-11T08:29:52.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.400+0000 7f2471ffb700 1 -- 172.21.15.27:0/1291236363 <== mgr.7076 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f246c011760 con 0x7f2460067130 2024-01-11T08:29:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-01-11T08:29:52.407 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-11T08:29:52.408 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec M 2024-01-11T08:29:52.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.611+0000 7f4b16bff700 1 Processor -- start 2024-01-11T08:29:52.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.611+0000 7f4b16bff700 1 -- start start 2024-01-11T08:29:52.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.612+0000 7f4b16bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b1015a1b0 0x7f4b1015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:52.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.612+0000 7f4b16bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b100cf850 con 0x7f4b100ccd70 2024-01-11T08:29:52.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.612+0000 7f4b16bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b100cec40 con 0x7f4b1015ae70 2024-01-11T08:29:52.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.612+0000 7f4b16bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b100cef20 con 0x7f4b1015a1b0 2024-01-11T08:29:52.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.612+0000 7f4b1499b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b1015a1b0 0x7f4b1015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:52.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.612+0000 7f4b0ffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4b100ccd70 0x7f4b100cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43814/0 (socket says 172.21.15.27:43814) 2024-01-11T08:29:52.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.612+0000 7f4b0ffff700 1 -- 172.21.15.27:0/97191157 learned_addr learned my addr 172.21.15.27:0/97191157 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.613+0000 7f4b1499b700 1 -- 172.21.15.27:0/97191157 >> v1:172.21.15.77:6789/0 conn(0x7f4b1015ae70 legacy=0x7f4b1015b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.613+0000 7f4b1499b700 1 -- 172.21.15.27:0/97191157 >> v1:172.21.15.27:6789/0 conn(0x7f4b100ccd70 legacy=0x7f4b100cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.613+0000 7f4b1499b700 1 -- 172.21.15.27:0/97191157 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b100cf200 con 0x7f4b1015a1b0 2024-01-11T08:29:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.613+0000 7f4b1499b700 1 --2- 172.21.15.27:0/97191157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b1015a1b0 0x7f4b1015a580 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f4b040098e0 tx=0x7f4b040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f402cf8bf9a7af server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.613+0000 7f4b0f7fe700 1 -- 172.21.15.27:0/97191157 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b04009e80 con 0x7f4b1015a1b0 2024-01-11T08:29:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.613+0000 7f4b0f7fe700 1 -- 172.21.15.27:0/97191157 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0400b760 con 0x7f4b1015a1b0 2024-01-11T08:29:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.613+0000 7f4b0f7fe700 1 -- 172.21.15.27:0/97191157 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0400bae0 con 0x7f4b1015a1b0 2024-01-11T08:29:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.614+0000 7f4b16bff700 1 -- 172.21.15.27:0/97191157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b1015a1b0 msgr2=0x7f4b1015a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.614+0000 7f4b16bff700 1 --2- 172.21.15.27:0/97191157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b1015a1b0 0x7f4b1015a580 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f4b040098e0 tx=0x7f4b040091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:52.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.614+0000 7f4b16bff700 1 -- 172.21.15.27:0/97191157 shutdown_connections 2024-01-11T08:29:52.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.614+0000 7f4b16bff700 1 --2- 172.21.15.27:0/97191157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b1015a1b0 0x7f4b1015a580 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:52.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.614+0000 7f4b16bff700 1 -- 172.21.15.27:0/97191157 >> 172.21.15.27:0/97191157 conn(0x7f4b1006ed40 msgr2=0x7f4b1006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:52.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.614+0000 7f4b16bff700 1 -- 172.21.15.27:0/97191157 shutdown_connections 2024-01-11T08:29:52.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.614+0000 7f4b16bff700 1 -- 172.21.15.27:0/97191157 wait complete. 2024-01-11T08:29:52.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.616+0000 7f4b16bff700 1 Processor -- start 2024-01-11T08:29:52.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.616+0000 7f4b16bff700 1 -- start start 2024-01-11T08:29:52.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.617+0000 7f4b16bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b100ccd70 0x7f4b10155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:52.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.617+0000 7f4b16bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4b1015a1b0 0x7f4b100c8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:52.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.617+0000 7f4b16bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b1015ae70 0x7f4b100c8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:52.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.617+0000 7f4b16bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b100c8c10 con 0x7f4b1015ae70 2024-01-11T08:29:52.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.617+0000 7f4b16bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b100afa40 con 0x7f4b100ccd70 2024-01-11T08:29:52.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.617+0000 7f4b16bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b100b34b0 con 0x7f4b1015a1b0 2024-01-11T08:29:52.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.617+0000 7f4b1519c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b1015ae70 0x7f4b100c8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:52.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.617+0000 7f4b1519c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b1015ae70 0x7f4b100c8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54532/0 (socket says 172.21.15.27:54532) 2024-01-11T08:29:52.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.617+0000 7f4b1519c700 1 -- 172.21.15.27:0/3051310725 learned_addr learned my addr 172.21.15.27:0/3051310725 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:52.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.617+0000 7f4b1499b700 1 --2- 172.21.15.27:0/3051310725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b100ccd70 0x7f4b10155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:52.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.617+0000 7f4b0ffff700 1 --2- 172.21.15.27:0/3051310725 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4b1015a1b0 0x7f4b100c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:52.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.617+0000 7f4b1519c700 1 -- 172.21.15.27:0/3051310725 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4b1015a1b0 msgr2=0x7f4b100c8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:52.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.617+0000 7f4b1519c700 1 --2- 172.21.15.27:0/3051310725 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4b1015a1b0 0x7f4b100c8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:52.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.617+0000 7f4b1519c700 1 -- 172.21.15.27:0/3051310725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b100ccd70 msgr2=0x7f4b10155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:52.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.617+0000 7f4b1519c700 1 --2- 172.21.15.27:0/3051310725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b100ccd70 0x7f4b10155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:52.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.617+0000 7f4b1519c700 1 -- 172.21.15.27:0/3051310725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b04009420 con 0x7f4b1015ae70 2024-01-11T08:29:52.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.617+0000 7f4b1499b700 1 --2- 172.21.15.27:0/3051310725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b100ccd70 0x7f4b10155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:29:52.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.617+0000 7f4b0ffff700 1 --2- 172.21.15.27:0/3051310725 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4b1015a1b0 0x7f4b100c8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:29:52.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.618+0000 7f4b1519c700 1 --2- 172.21.15.27:0/3051310725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b1015ae70 0x7f4b100c8590 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800c8e0 tx=0x7f4b08001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:52.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.618+0000 7f4b0cff9700 1 -- 172.21.15.27:0/3051310725 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0800ce80 con 0x7f4b1015ae70 2024-01-11T08:29:52.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.618+0000 7f4b0cff9700 1 -- 172.21.15.27:0/3051310725 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0800a7e0 con 0x7f4b1015ae70 2024-01-11T08:29:52.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.618+0000 7f4b0cff9700 1 -- 172.21.15.27:0/3051310725 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0800ab80 con 0x7f4b1015ae70 2024-01-11T08:29:52.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.618+0000 7f4b16bff700 1 -- 172.21.15.27:0/3051310725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b100b3790 con 0x7f4b1015ae70 2024-01-11T08:29:52.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.618+0000 7f4b16bff700 1 -- 172.21.15.27:0/3051310725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b100b3d50 con 0x7f4b1015ae70 2024-01-11T08:29:52.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.619+0000 7f4b16bff700 1 -- 172.21.15.27:0/3051310725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b100b3f70 con 0x7f4b1015ae70 2024-01-11T08:29:52.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.620+0000 7f4b0cff9700 1 -- 172.21.15.27:0/3051310725 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 73) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f4b08012420 con 0x7f4b1015ae70 2024-01-11T08:29:52.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.621+0000 7f4b0cff9700 1 --2- 172.21.15.27:0/3051310725 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4af4067130 0x7f4af40695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:52.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.621+0000 7f4b0cff9700 1 -- 172.21.15.27:0/3051310725 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f4b08084ca0 con 0x7f4b1015ae70 2024-01-11T08:29:52.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.621+0000 7f4b1499b700 1 --2- 172.21.15.27:0/3051310725 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4af4067130 0x7f4af40695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:52.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.622+0000 7f4b1499b700 1 --2- 172.21.15.27:0/3051310725 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4af4067130 0x7f4af40695e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4b040091e0 tx=0x7f4b04016000 comp rx=0 tx=0).ready entity=mgr.7076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:52.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.625+0000 7f4b0cff9700 1 -- 172.21.15.27:0/3051310725 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4b08051b50 con 0x7f4b1015ae70 2024-01-11T08:29:52.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.855+0000 7f4b16bff700 1 -- 172.21.15.27:0/3051310725 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "M", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b1004c3c0 con 0x7f4af4067130 2024-01-11T08:29:52.855 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:52.855+0000 7f5d4424d700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-01-11T08:29:52.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:52.856+0000 7f4b0cff9700 1 -- 172.21.15.27:0/3051310725 <== mgr.7076 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4b1004c3c0 con 0x7f4af4067130 2024-01-11T08:29:52.857 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-01-11T08:29:52.864 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-11T08:29:52.865 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec -1m 2024-01-11T08:29:53.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.070+0000 7fb7f19ff700 1 Processor -- start 2024-01-11T08:29:53.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.070+0000 7fb7f19ff700 1 -- start start 2024-01-11T08:29:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.071+0000 7fb7f19ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7ec15a1b0 0x7fb7ec15a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.071+0000 7fb7f19ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb7ec15ae70 0x7fb7ec15b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.071+0000 7fb7f19ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ec0cf850 con 0x7fb7ec0ccd70 2024-01-11T08:29:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.071+0000 7fb7f19ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7ec0cec40 con 0x7fb7ec15a1b0 2024-01-11T08:29:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.071+0000 7fb7f19ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7ec0cef20 con 0x7fb7ec15ae70 2024-01-11T08:29:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.072+0000 7fb7eaffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7ec15a1b0 0x7fb7ec15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.072+0000 7fb7eaffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7ec15a1b0 0x7fb7ec15a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54548/0 (socket says 172.21.15.27:54548) 2024-01-11T08:29:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.072+0000 7fb7eaffd700 1 -- 172.21.15.27:0/1011015393 learned_addr learned my addr 172.21.15.27:0/1011015393 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.072+0000 7fb7eb7fe700 1 --2- 172.21.15.27:0/1011015393 >> v2:172.21.15.77:3300/0 conn(0x7fb7ec15ae70 0x7fb7ec15b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.072+0000 7fb7eaffd700 1 -- 172.21.15.27:0/1011015393 >> v2:172.21.15.77:3300/0 conn(0x7fb7ec15ae70 msgr2=0x7fb7ec15b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.072+0000 7fb7eaffd700 1 --2- 172.21.15.27:0/1011015393 >> v2:172.21.15.77:3300/0 conn(0x7fb7ec15ae70 0x7fb7ec15b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.072+0000 7fb7eaffd700 1 -- 172.21.15.27:0/1011015393 >> v1:172.21.15.77:6789/0 conn(0x7fb7ec0ccd70 legacy=0x7fb7ec0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.072+0000 7fb7eaffd700 1 -- 172.21.15.27:0/1011015393 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ec0cf200 con 0x7fb7ec15a1b0 2024-01-11T08:29:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.073+0000 7fb7eaffd700 1 --2- 172.21.15.27:0/1011015393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7ec15a1b0 0x7fb7ec15a580 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec089810 tx=0x7fb7dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0cdccb6ab41f6ec server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.073+0000 7fb7e9ffb700 1 -- 172.21.15.27:0/1011015393 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc009e70 con 0x7fb7ec15a1b0 2024-01-11T08:29:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.073+0000 7fb7e9ffb700 1 -- 172.21.15.27:0/1011015393 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7dc00b860 con 0x7fb7ec15a1b0 2024-01-11T08:29:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.073+0000 7fb7e9ffb700 1 -- 172.21.15.27:0/1011015393 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc00bc00 con 0x7fb7ec15a1b0 2024-01-11T08:29:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.073+0000 7fb7f19ff700 1 -- 172.21.15.27:0/1011015393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7ec15a1b0 msgr2=0x7fb7ec15a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.073+0000 7fb7f19ff700 1 --2- 172.21.15.27:0/1011015393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7ec15a1b0 0x7fb7ec15a580 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec089810 tx=0x7fb7dc009250 comp rx=0 tx=0).stop 2024-01-11T08:29:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.073+0000 7fb7f19ff700 1 -- 172.21.15.27:0/1011015393 shutdown_connections 2024-01-11T08:29:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.073+0000 7fb7f19ff700 1 --2- 172.21.15.27:0/1011015393 >> v2:172.21.15.77:3300/0 conn(0x7fb7ec15ae70 0x7fb7ec15b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.073+0000 7fb7f19ff700 1 --2- 172.21.15.27:0/1011015393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7ec15a1b0 0x7fb7ec15a580 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:53.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.073+0000 7fb7f19ff700 1 -- 172.21.15.27:0/1011015393 >> 172.21.15.27:0/1011015393 conn(0x7fb7ec06ed40 msgr2=0x7fb7ec06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:53.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.073+0000 7fb7f19ff700 1 -- 172.21.15.27:0/1011015393 shutdown_connections 2024-01-11T08:29:53.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.074+0000 7fb7f19ff700 1 -- 172.21.15.27:0/1011015393 wait complete. 2024-01-11T08:29:53.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.075+0000 7fb7f19ff700 1 Processor -- start 2024-01-11T08:29:53.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.075+0000 7fb7f19ff700 1 -- start start 2024-01-11T08:29:53.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.076+0000 7fb7f19ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7ec0ccd70 0x7fb7ec14af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:53.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.076+0000 7fb7f19ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7ec15a1b0 0x7fb7ec158360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:53.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.076+0000 7fb7f19ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7ec15ae70 0x7fb7ec0c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:53.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.076+0000 7fb7f19ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec158d40 con 0x7fb7ec15ae70 2024-01-11T08:29:53.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.076+0000 7fb7f19ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec159020 con 0x7fb7ec15a1b0 2024-01-11T08:29:53.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.076+0000 7fb7f19ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec159300 con 0x7fb7ec0ccd70 2024-01-11T08:29:53.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.076+0000 7fb7eb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7ec15ae70 0x7fb7ec0c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:53.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.076+0000 7fb7eb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7ec15ae70 0x7fb7ec0c6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54556/0 (socket says 172.21.15.27:54556) 2024-01-11T08:29:53.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.076+0000 7fb7eb7fe700 1 -- 172.21.15.27:0/3960505516 learned_addr learned my addr 172.21.15.27:0/3960505516 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:53.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.076+0000 7fb7ea7fc700 1 --2- 172.21.15.27:0/3960505516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7ec15a1b0 0x7fb7ec158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:53.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.076+0000 7fb7eaffd700 1 --2- 172.21.15.27:0/3960505516 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7ec0ccd70 0x7fb7ec14af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:53.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.077+0000 7fb7eb7fe700 1 -- 172.21.15.27:0/3960505516 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7ec0ccd70 msgr2=0x7fb7ec14af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:53.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.077+0000 7fb7eb7fe700 1 --2- 172.21.15.27:0/3960505516 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7ec0ccd70 0x7fb7ec14af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:53.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.077+0000 7fb7eb7fe700 1 -- 172.21.15.27:0/3960505516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7ec15a1b0 msgr2=0x7fb7ec158360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:53.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.077+0000 7fb7eb7fe700 1 --2- 172.21.15.27:0/3960505516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7ec15a1b0 0x7fb7ec158360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:53.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.077+0000 7fb7eb7fe700 1 -- 172.21.15.27:0/3960505516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7dc0094f0 con 0x7fb7ec15ae70 2024-01-11T08:29:53.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.077+0000 7fb7eb7fe700 1 --2- 172.21.15.27:0/3960505516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7ec15ae70 0x7fb7ec0c6ad0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4009e90 tx=0x7fb7e400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:53.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.077+0000 7fb7d2ffd700 1 -- 172.21.15.27:0/3960505516 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e400cb70 con 0x7fb7ec15ae70 2024-01-11T08:29:53.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.077+0000 7fb7d2ffd700 1 -- 172.21.15.27:0/3960505516 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e400f040 con 0x7fb7ec15ae70 2024-01-11T08:29:53.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.077+0000 7fb7f19ff700 1 -- 172.21.15.27:0/3960505516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ec1595e0 con 0x7fb7ec15ae70 2024-01-11T08:29:53.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.077+0000 7fb7d2ffd700 1 -- 172.21.15.27:0/3960505516 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4014480 con 0x7fb7ec15ae70 2024-01-11T08:29:53.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.077+0000 7fb7f19ff700 1 -- 172.21.15.27:0/3960505516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ec14b4c0 con 0x7fb7ec15ae70 2024-01-11T08:29:53.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.078+0000 7fb7f19ff700 1 -- 172.21.15.27:0/3960505516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ec04c3c0 con 0x7fb7ec15ae70 2024-01-11T08:29:53.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.079+0000 7fb7d2ffd700 1 -- 172.21.15.27:0/3960505516 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 73) v1 ==== 84918+0+0 (secure 0 0 0) 0x7fb7e4014830 con 0x7fb7ec15ae70 2024-01-11T08:29:53.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.080+0000 7fb7d2ffd700 1 --2- 172.21.15.27:0/3960505516 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb7d8067130 0x7fb7d80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:53.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.080+0000 7fb7d2ffd700 1 -- 172.21.15.27:0/3960505516 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7fb7e4085550 con 0x7fb7ec15ae70 2024-01-11T08:29:53.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.081+0000 7fb7eaffd700 1 --2- 172.21.15.27:0/3960505516 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb7d8067130 0x7fb7d80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:53.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.081+0000 7fb7eaffd700 1 --2- 172.21.15.27:0/3960505516 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb7d8067130 0x7fb7d80695e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec089810 tx=0x7fb7dc017000 comp rx=0 tx=0).ready entity=mgr.7076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:53.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.084+0000 7fb7d2ffd700 1 -- 172.21.15.27:0/3960505516 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb7e4052250 con 0x7fb7ec15ae70 2024-01-11T08:29:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.312+0000 7fb7f19ff700 1 -- 172.21.15.27:0/3960505516 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "-1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7ec15c2b0 con 0x7fb7d8067130 2024-01-11T08:29:53.313 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:53.314+0000 7f5d4424d700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2024-01-11T08:29:53.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.313+0000 7fb7d2ffd700 1 -- 172.21.15.27:0/3960505516 <== mgr.7076 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc.) v1 ==== 96+0+0 (secure 0 0 0) 0x7fb7ec15c2b0 con 0x7fb7d8067130 2024-01-11T08:29:53.315 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENOENT: invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2024-01-11T08:29:53.322 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-11T08:29:53.323 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec '' 2024-01-11T08:29:53.511 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:29:53.515 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:29:53.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.526+0000 7f8bf63ff700 1 Processor -- start 2024-01-11T08:29:53.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.526+0000 7f8bf63ff700 1 -- start start 2024-01-11T08:29:53.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.528+0000 7f8bf63ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8bf013ff80 0x7f8bf015a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:53.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.528+0000 7f8bf63ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8bf015a560 0x7f8bf00c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:53.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.528+0000 7f8bf63ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bf00c6b30 con 0x7f8bf013fbb0 2024-01-11T08:29:53.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.528+0000 7f8bf63ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf015a9a0 con 0x7f8bf013ff80 2024-01-11T08:29:53.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.528+0000 7f8bf63ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf00c6800 con 0x7f8bf015a560 2024-01-11T08:29:53.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.528+0000 7f8bef7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8bf013ff80 0x7f8bf015a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:53.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.528+0000 7f8beffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8bf013fbb0 0x7f8bf00ca7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43816/0 (socket says 172.21.15.27:43816) 2024-01-11T08:29:53.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.528+0000 7f8beffff700 1 -- 172.21.15.27:0/1701476618 learned_addr learned my addr 172.21.15.27:0/1701476618 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:53.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.528+0000 7f8bf499c700 1 --2- 172.21.15.27:0/1701476618 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8bf015a560 0x7f8bf00c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:53.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.529+0000 7f8bef7fe700 1 -- 172.21.15.27:0/1701476618 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8bf015a560 msgr2=0x7f8bf00c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:53.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.529+0000 7f8bef7fe700 1 --2- 172.21.15.27:0/1701476618 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8bf015a560 0x7f8bf00c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:53.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.529+0000 7f8bef7fe700 1 -- 172.21.15.27:0/1701476618 >> v1:172.21.15.27:6789/0 conn(0x7f8bf013fbb0 legacy=0x7f8bf00ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:53.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.529+0000 7f8bef7fe700 1 -- 172.21.15.27:0/1701476618 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf00cb240 con 0x7f8bf013ff80 2024-01-11T08:29:53.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.529+0000 7f8beeffd700 1 -- 172.21.15.27:0/1701476618 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517134960 0 0) 0x7f8bf00c6b30 con 0x7f8bf013fbb0 2024-01-11T08:29:53.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.529+0000 7f8bef7fe700 1 --2- 172.21.15.27:0/1701476618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bf013ff80 0x7f8bf015a020 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f8be40098e0 tx=0x7f8be40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f228fba8f8a4c96 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:53.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.529+0000 7f8beeffd700 1 -- 172.21.15.27:0/1701476618 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be400b640 con 0x7f8bf013ff80 2024-01-11T08:29:53.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.529+0000 7f8beeffd700 1 -- 172.21.15.27:0/1701476618 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be400b7a0 con 0x7f8bf013ff80 2024-01-11T08:29:53.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.529+0000 7f8beeffd700 1 -- 172.21.15.27:0/1701476618 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be400bb40 con 0x7f8bf013ff80 2024-01-11T08:29:53.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.530+0000 7f8bf63ff700 1 -- 172.21.15.27:0/1701476618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bf013ff80 msgr2=0x7f8bf015a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:53.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.530+0000 7f8bf63ff700 1 --2- 172.21.15.27:0/1701476618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bf013ff80 0x7f8bf015a020 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f8be40098e0 tx=0x7f8be40091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:53.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.530+0000 7f8bf63ff700 1 -- 172.21.15.27:0/1701476618 shutdown_connections 2024-01-11T08:29:53.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.530+0000 7f8bf63ff700 1 --2- 172.21.15.27:0/1701476618 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8bf015a560 0x7f8bf00c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:53.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.530+0000 7f8bf63ff700 1 --2- 172.21.15.27:0/1701476618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bf013ff80 0x7f8bf015a020 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:53.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.530+0000 7f8bf63ff700 1 -- 172.21.15.27:0/1701476618 >> 172.21.15.27:0/1701476618 conn(0x7f8bf00680a0 msgr2=0x7f8bf00cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:53.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.530+0000 7f8bf63ff700 1 -- 172.21.15.27:0/1701476618 shutdown_connections 2024-01-11T08:29:53.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.530+0000 7f8bf63ff700 1 -- 172.21.15.27:0/1701476618 wait complete. 2024-01-11T08:29:53.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.532+0000 7f8bf63ff700 1 Processor -- start 2024-01-11T08:29:53.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.532+0000 7f8bf63ff700 1 -- start start 2024-01-11T08:29:53.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.533+0000 7f8bf63ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8bf013fbb0 0x7f8bf00b1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:53.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.533+0000 7f8bf63ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bf013ff80 0x7f8bf0150fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:53.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.533+0000 7f8bf63ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bf015a560 0x7f8bf00aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:53.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.533+0000 7f8bf63ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf00b1ca0 con 0x7f8bf013ff80 2024-01-11T08:29:53.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.533+0000 7f8bf63ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf00c3d40 con 0x7f8bf015a560 2024-01-11T08:29:53.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.533+0000 7f8bf63ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf00c3410 con 0x7f8bf013fbb0 2024-01-11T08:29:53.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.533+0000 7f8bef7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bf013ff80 0x7f8bf0150fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:53.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.533+0000 7f8bef7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bf013ff80 0x7f8bf0150fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54578/0 (socket says 172.21.15.27:54578) 2024-01-11T08:29:53.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.533+0000 7f8bef7fe700 1 -- 172.21.15.27:0/3473712733 learned_addr learned my addr 172.21.15.27:0/3473712733 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:53.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.533+0000 7f8beffff700 1 --2- 172.21.15.27:0/3473712733 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8bf013fbb0 0x7f8bf00b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:53.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.533+0000 7f8bf499c700 1 --2- 172.21.15.27:0/3473712733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bf015a560 0x7f8bf00aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:53.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.534+0000 7f8bef7fe700 1 -- 172.21.15.27:0/3473712733 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8bf013fbb0 msgr2=0x7f8bf00b1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:53.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.534+0000 7f8bef7fe700 1 --2- 172.21.15.27:0/3473712733 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8bf013fbb0 0x7f8bf00b1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:53.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.534+0000 7f8bef7fe700 1 -- 172.21.15.27:0/3473712733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bf015a560 msgr2=0x7f8bf00aef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:53.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.534+0000 7f8bef7fe700 1 --2- 172.21.15.27:0/3473712733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8bf015a560 0x7f8bf00aef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:53.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.534+0000 7f8bef7fe700 1 -- 172.21.15.27:0/3473712733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be4009420 con 0x7f8bf013ff80 2024-01-11T08:29:53.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.534+0000 7f8bef7fe700 1 --2- 172.21.15.27:0/3473712733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bf013ff80 0x7f8bf0150fb0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0069490 tx=0x7f8be4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:53.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.534+0000 7f8bd7fff700 1 -- 172.21.15.27:0/3473712733 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be400bb20 con 0x7f8bf013ff80 2024-01-11T08:29:53.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.534+0000 7f8bd7fff700 1 -- 172.21.15.27:0/3473712733 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be4000c90 con 0x7f8bf013ff80 2024-01-11T08:29:53.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.534+0000 7f8bd7fff700 1 -- 172.21.15.27:0/3473712733 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be4011630 con 0x7f8bf013ff80 2024-01-11T08:29:53.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.534+0000 7f8bf63ff700 1 -- 172.21.15.27:0/3473712733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf00c36f0 con 0x7f8bf013ff80 2024-01-11T08:29:53.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.535+0000 7f8bf63ff700 1 -- 172.21.15.27:0/3473712733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf0134cc0 con 0x7f8bf013ff80 2024-01-11T08:29:53.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.536+0000 7f8bf63ff700 1 -- 172.21.15.27:0/3473712733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf00c3ed0 con 0x7f8bf013ff80 2024-01-11T08:29:53.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.537+0000 7f8bd7fff700 1 -- 172.21.15.27:0/3473712733 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 73) v1 ==== 84918+0+0 (secure 0 0 0) 0x7f8be40119a0 con 0x7f8bf013ff80 2024-01-11T08:29:53.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.538+0000 7f8bd7fff700 1 --2- 172.21.15.27:0/3473712733 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8bdc067130 0x7f8bdc0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:53.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.538+0000 7f8bd7fff700 1 -- 172.21.15.27:0/3473712733 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f8be4085de0 con 0x7f8bf013ff80 2024-01-11T08:29:53.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.538+0000 7f8beffff700 1 --2- 172.21.15.27:0/3473712733 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8bdc067130 0x7f8bdc0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:53.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.539+0000 7f8beffff700 1 --2- 172.21.15.27:0/3473712733 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8bdc067130 0x7f8bdc0695e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0162150 tx=0x7f8be0009040 comp rx=0 tx=0).ready entity=mgr.7076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:53.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.542+0000 7f8bd7fff700 1 -- 172.21.15.27:0/3473712733 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8be4052bf0 con 0x7f8bf013ff80 2024-01-11T08:29:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.774+0000 7f8bf63ff700 1 -- 172.21.15.27:0/3473712733 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f8bf004c3c0 con 0x7f8bdc067130 2024-01-11T08:29:53.775 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:53.775+0000 7f5d4424d700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-01-11T08:29:53.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:53.775+0000 7f8bd7fff700 1 -- 172.21.15.27:0/3473712733 <== mgr.7076 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8bf004c3c0 con 0x7f8bdc067130 2024-01-11T08:29:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-01-11T08:29:53.784 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-11T08:29:53.785 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:53.785 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/minutes) 2024-01-11T08:29:53.835 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:53.835 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/hourly) 2024-01-11T08:29:53.883 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:53.884 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/daily) 2024-01-11T08:29:53.932 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:53.933 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/weekly) 2024-01-11T08:29:53.983 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:53.983 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/monthly) 2024-01-11T08:29:54.032 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:54.033 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/yearly) 2024-01-11T08:29:54.082 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:54.082 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/bad_period_spec) 2024-01-11T08:29:54.131 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-11T08:29:54.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.332+0000 7fb7e15ff700 1 Processor -- start 2024-01-11T08:29:54.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.332+0000 7fb7e15ff700 1 -- start start 2024-01-11T08:29:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.334+0000 7fb7e15ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7dc155c50 0x7fb7dc0aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.334+0000 7fb7e15ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7dc06e480 con 0x7fb7dc0694c0 2024-01-11T08:29:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.334+0000 7fb7e15ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7dc0c7a60 con 0x7fb7dc0c74f0 2024-01-11T08:29:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.334+0000 7fb7e15ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc0cb5a0 con 0x7fb7dc155c50 2024-01-11T08:29:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.334+0000 7fb7da7fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb7dc0694c0 0x7fb7dc0c6de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58020/0 (socket says 172.21.15.27:58020) 2024-01-11T08:29:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.334+0000 7fb7da7fc700 1 -- 172.21.15.27:0/1411283689 learned_addr learned my addr 172.21.15.27:0/1411283689 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.334+0000 7fb7daffd700 1 --2- 172.21.15.27:0/1411283689 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7dc155c50 0x7fb7dc0aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:54.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.334+0000 7fb7d9ffb700 1 -- 172.21.15.27:0/1411283689 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954422246 0 0) 0x7fb7dc06e480 con 0x7fb7dc0694c0 2024-01-11T08:29:54.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.334+0000 7fb7d9ffb700 1 -- 172.21.15.27:0/1411283689 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c4003570 con 0x7fb7dc0694c0 2024-01-11T08:29:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.335+0000 7fb7d9ffb700 1 -- 172.21.15.27:0/1411283689 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 351687909 0 0) 0x7fb7dc0c7a60 con 0x7fb7dc0c74f0 2024-01-11T08:29:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.335+0000 7fb7d9ffb700 1 -- 172.21.15.27:0/1411283689 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7dc06e480 con 0x7fb7dc0c74f0 2024-01-11T08:29:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.335+0000 7fb7daffd700 1 -- 172.21.15.27:0/1411283689 >> v1:172.21.15.77:6789/0 conn(0x7fb7dc0c74f0 legacy=0x7fb7dc06e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.335+0000 7fb7d9ffb700 1 -- 172.21.15.27:0/1411283689 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3079544691 0 0) 0x7fb7c4003570 con 0x7fb7dc0694c0 2024-01-11T08:29:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.335+0000 7fb7daffd700 1 -- 172.21.15.27:0/1411283689 >> v1:172.21.15.27:6789/0 conn(0x7fb7dc0694c0 legacy=0x7fb7dc0c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.335+0000 7fb7daffd700 1 -- 172.21.15.27:0/1411283689 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7dc0c7d40 con 0x7fb7dc155c50 2024-01-11T08:29:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.335+0000 7fb7daffd700 1 --2- 172.21.15.27:0/1411283689 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7dc155c50 0x7fb7dc0aef80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00aaa0 tx=0x7fb7cc00a2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7df7372deaeb693 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.335+0000 7fb7d9ffb700 1 -- 172.21.15.27:0/1411283689 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0001470 con 0x7fb7dc155c50 2024-01-11T08:29:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.335+0000 7fb7d9ffb700 1 -- 172.21.15.27:0/1411283689 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7cc005a00 con 0x7fb7dc155c50 2024-01-11T08:29:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.335+0000 7fb7d9ffb700 1 -- 172.21.15.27:0/1411283689 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc005d80 con 0x7fb7dc155c50 2024-01-11T08:29:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.336+0000 7fb7e15ff700 1 -- 172.21.15.27:0/1411283689 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7dc155c50 msgr2=0x7fb7dc0aef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.336+0000 7fb7e15ff700 1 --2- 172.21.15.27:0/1411283689 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7dc155c50 0x7fb7dc0aef80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00aaa0 tx=0x7fb7cc00a2e0 comp rx=0 tx=0).stop 2024-01-11T08:29:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.336+0000 7fb7e15ff700 1 -- 172.21.15.27:0/1411283689 shutdown_connections 2024-01-11T08:29:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.336+0000 7fb7e15ff700 1 --2- 172.21.15.27:0/1411283689 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7dc155c50 0x7fb7dc0aef80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:54.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.336+0000 7fb7e15ff700 1 -- 172.21.15.27:0/1411283689 >> 172.21.15.27:0/1411283689 conn(0x7fb7dc0680a0 msgr2=0x7fb7dc0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:54.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.336+0000 7fb7e15ff700 1 -- 172.21.15.27:0/1411283689 shutdown_connections 2024-01-11T08:29:54.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.336+0000 7fb7e15ff700 1 -- 172.21.15.27:0/1411283689 wait complete. 2024-01-11T08:29:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.338+0000 7fb7e15ff700 1 Processor -- start 2024-01-11T08:29:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.338+0000 7fb7e15ff700 1 -- start start 2024-01-11T08:29:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.339+0000 7fb7e15ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7dc0694c0 0x7fb7dc0c4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.339+0000 7fb7e15ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7dc0c74f0 0x7fb7dc151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.339+0000 7fb7e15ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7dc155c50 0x7fb7dc0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.339+0000 7fb7e15ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc0c4570 con 0x7fb7dc0694c0 2024-01-11T08:29:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.339+0000 7fb7e15ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc1520c0 con 0x7fb7dc155c50 2024-01-11T08:29:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.339+0000 7fb7e15ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc0c53f0 con 0x7fb7dc0c74f0 2024-01-11T08:29:54.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.339+0000 7fb7daffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7dc0694c0 0x7fb7dc0c4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:54.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.339+0000 7fb7daffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7dc0694c0 0x7fb7dc0c4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57590/0 (socket says 172.21.15.27:57590) 2024-01-11T08:29:54.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.339+0000 7fb7daffd700 1 -- 172.21.15.27:0/2151366606 learned_addr learned my addr 172.21.15.27:0/2151366606 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:54.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.339+0000 7fb7da7fc700 1 --2- 172.21.15.27:0/2151366606 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7dc0c74f0 0x7fb7dc151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:54.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.339+0000 7fb7db7fe700 1 --2- 172.21.15.27:0/2151366606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7dc155c50 0x7fb7dc0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:54.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.339+0000 7fb7daffd700 1 -- 172.21.15.27:0/2151366606 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7dc0c74f0 msgr2=0x7fb7dc151b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:54.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.339+0000 7fb7daffd700 1 --2- 172.21.15.27:0/2151366606 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7dc0c74f0 0x7fb7dc151b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:54.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.339+0000 7fb7daffd700 1 -- 172.21.15.27:0/2151366606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7dc155c50 msgr2=0x7fb7dc0c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:54.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.339+0000 7fb7daffd700 1 --2- 172.21.15.27:0/2151366606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7dc155c50 0x7fb7dc0c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:54.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.339+0000 7fb7daffd700 1 -- 172.21.15.27:0/2151366606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7cc00a560 con 0x7fb7dc0694c0 2024-01-11T08:29:54.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.340+0000 7fb7daffd700 1 --2- 172.21.15.27:0/2151366606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7dc0694c0 0x7fb7dc0c4030 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc003080 tx=0x7fb7cc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:54.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.340+0000 7fb7c2ffd700 1 -- 172.21.15.27:0/2151366606 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc01f070 con 0x7fb7dc0694c0 2024-01-11T08:29:54.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.340+0000 7fb7c2ffd700 1 -- 172.21.15.27:0/2151366606 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7cc006240 con 0x7fb7dc0694c0 2024-01-11T08:29:54.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.340+0000 7fb7c2ffd700 1 -- 172.21.15.27:0/2151366606 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc006620 con 0x7fb7dc0694c0 2024-01-11T08:29:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.340+0000 7fb7e15ff700 1 -- 172.21.15.27:0/2151366606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7dc0c3270 con 0x7fb7dc0694c0 2024-01-11T08:29:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.340+0000 7fb7e15ff700 1 -- 172.21.15.27:0/2151366606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7dc0c3830 con 0x7fb7dc0694c0 2024-01-11T08:29:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.341+0000 7fb7e15ff700 1 -- 172.21.15.27:0/2151366606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7dc04c3c0 con 0x7fb7dc0694c0 2024-01-11T08:29:54.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.342+0000 7fb7c2ffd700 1 -- 172.21.15.27:0/2151366606 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 73) v1 ==== 84918+0+0 (secure 0 0 0) 0x7fb7cc023410 con 0x7fb7dc0694c0 2024-01-11T08:29:54.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.343+0000 7fb7c2ffd700 1 --2- 172.21.15.27:0/2151366606 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb7c8067130 0x7fb7c80695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:54.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.343+0000 7fb7c2ffd700 1 -- 172.21.15.27:0/2151366606 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7fb7cc0858a0 con 0x7fb7dc0694c0 2024-01-11T08:29:54.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.344+0000 7fb7da7fc700 1 --2- 172.21.15.27:0/2151366606 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb7c8067130 0x7fb7c80695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:54.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.344+0000 7fb7da7fc700 1 --2- 172.21.15.27:0/2151366606 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb7c8067130 0x7fb7c80695e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc0ca2d0 tx=0x7fb7d0000bc0 comp rx=0 tx=0).ready entity=mgr.7076 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:54.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.347+0000 7fb7c2ffd700 1 -- 172.21.15.27:0/2151366606 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb7cc052640 con 0x7fb7dc0694c0 2024-01-11T08:29:54.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:54.609+0000 7fb7e15ff700 1 -- 172.21.15.27:0/2151366606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fb7dc0b1ae0 con 0x7fb7dc0694c0 2024-01-11T08:29:55.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.037+0000 7fb7c2ffd700 1 -- 172.21.15.27:0/2151366606 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 74) v1 ==== 84901+0+0 (secure 0 0 0) 0x7fb7cc052a10 con 0x7fb7dc0694c0 2024-01-11T08:29:55.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.038+0000 7fb7c2ffd700 1 -- 172.21.15.27:0/2151366606 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v74) v1 ==== 93+0+0 (secure 0 0 0) 0x7fb7cc0560d0 con 0x7fb7dc0694c0 2024-01-11T08:29:55.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.041+0000 7fb7e15ff700 1 -- 172.21.15.27:0/2151366606 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb7c8067130 msgr2=0x7fb7c80695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:55.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.041+0000 7fb7e15ff700 1 --2- 172.21.15.27:0/2151366606 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb7c8067130 0x7fb7c80695e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc0ca2d0 tx=0x7fb7d0000bc0 comp rx=0 tx=0).stop 2024-01-11T08:29:55.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.041+0000 7fb7e15ff700 1 -- 172.21.15.27:0/2151366606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7dc0694c0 msgr2=0x7fb7dc0c4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:55.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.041+0000 7fb7e15ff700 1 --2- 172.21.15.27:0/2151366606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7dc0694c0 0x7fb7dc0c4030 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc003080 tx=0x7fb7cc018000 comp rx=0 tx=0).stop 2024-01-11T08:29:55.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.041+0000 7fb7e15ff700 1 -- 172.21.15.27:0/2151366606 shutdown_connections 2024-01-11T08:29:55.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.041+0000 7fb7e15ff700 1 --2- 172.21.15.27:0/2151366606 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb7c8067130 0x7fb7c80695e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:55.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.041+0000 7fb7e15ff700 1 --2- 172.21.15.27:0/2151366606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7dc155c50 0x7fb7dc0c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:55.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.041+0000 7fb7e15ff700 1 --2- 172.21.15.27:0/2151366606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7dc0694c0 0x7fb7dc0c4030 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:55.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.041+0000 7fb7e15ff700 1 --2- 172.21.15.27:0/2151366606 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb7dc0c74f0 0x7fb7dc151b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:55.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.041+0000 7fb7e15ff700 1 -- 172.21.15.27:0/2151366606 >> 172.21.15.27:0/2151366606 conn(0x7fb7dc0680a0 msgr2=0x7fb7dc06f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:55.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.041+0000 7fb7e15ff700 1 -- 172.21.15.27:0/2151366606 shutdown_connections 2024-01-11T08:29:55.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.041+0000 7fb7e15ff700 1 -- 172.21.15.27:0/2151366606 wait complete. 2024-01-11T08:29:55.061 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T08:29:55.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:29:55.154 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:29:55.254 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:55.254+0000 7fad63d071c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:29:55.257 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T08:29:55.257 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-11T08:29:55.257 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:29:55.257 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-01-11T08:29:55.259 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:55.260+0000 7ff71dca21c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:29:55.346 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T08:29:55.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:29:55.346 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-11T08:29:55.422 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:29:55.439 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:55.438+0000 7ff71dca21c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:29:55.453 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:55.454+0000 7fad63d071c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:29:55.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.633+0000 7ff3289ff700 1 Processor -- start 2024-01-11T08:29:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.633+0000 7ff3289ff700 1 -- start start 2024-01-11T08:29:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.635+0000 7ff3289ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff32406b610 0x7ff324158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.635+0000 7ff3289ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff324158b60 con 0x7ff3241575e0 2024-01-11T08:29:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.635+0000 7ff3289ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3241599b0 con 0x7ff3240b19d0 2024-01-11T08:29:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.635+0000 7ff3289ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3240cb5e0 con 0x7ff32406b610 2024-01-11T08:29:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.635+0000 7ff32259c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff3241575e0 0x7ff3240b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58030/0 (socket says 172.21.15.27:58030) 2024-01-11T08:29:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.635+0000 7ff32259c700 1 -- 172.21.15.27:0/1014757177 learned_addr learned my addr 172.21.15.27:0/1014757177 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:55.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.635+0000 7ff322d9d700 1 --2- 172.21.15.27:0/1014757177 >> v2:172.21.15.77:3300/0 conn(0x7ff32406b610 0x7ff324158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:55.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.636+0000 7ff32159a700 1 -- 172.21.15.27:0/1014757177 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660413144 0 0) 0x7ff314001110 con 0x7ff3241575e0 2024-01-11T08:29:55.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.636+0000 7ff32159a700 1 -- 172.21.15.27:0/1014757177 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff30c003570 con 0x7ff3241575e0 2024-01-11T08:29:55.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.636+0000 7ff32159a700 1 -- 172.21.15.27:0/1014757177 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066739851 0 0) 0x7ff3241599b0 con 0x7ff3240b19d0 2024-01-11T08:29:55.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.636+0000 7ff32159a700 1 -- 172.21.15.27:0/1014757177 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff314001110 con 0x7ff3240b19d0 2024-01-11T08:29:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.636+0000 7ff32159a700 1 -- 172.21.15.27:0/1014757177 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3431245817 0 0) 0x7ff30c003570 con 0x7ff3241575e0 2024-01-11T08:29:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.636+0000 7ff322d9d700 1 -- 172.21.15.27:0/1014757177 >> v1:172.21.15.77:6789/0 conn(0x7ff3240b19d0 legacy=0x7ff324159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.636+0000 7ff322d9d700 1 -- 172.21.15.27:0/1014757177 >> v1:172.21.15.27:6789/0 conn(0x7ff3241575e0 legacy=0x7ff3240b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.636+0000 7ff322d9d700 1 -- 172.21.15.27:0/1014757177 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3240c0f90 con 0x7ff32406b610 2024-01-11T08:29:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.636+0000 7ff322d9d700 1 --2- 172.21.15.27:0/1014757177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff32406b610 0x7ff324158620 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009e60 tx=0x7ff31c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78f685ad1dac30fa server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.637+0000 7ff32159a700 1 -- 172.21.15.27:0/1014757177 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c00e070 con 0x7ff32406b610 2024-01-11T08:29:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.637+0000 7ff32159a700 1 -- 172.21.15.27:0/1014757177 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31c00b7a0 con 0x7ff32406b610 2024-01-11T08:29:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.637+0000 7ff32159a700 1 -- 172.21.15.27:0/1014757177 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314001600 con 0x7ff32406b610 2024-01-11T08:29:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.637+0000 7ff3289ff700 1 -- 172.21.15.27:0/1014757177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff32406b610 msgr2=0x7ff324158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.637+0000 7ff3289ff700 1 --2- 172.21.15.27:0/1014757177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff32406b610 0x7ff324158620 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009e60 tx=0x7ff31c009780 comp rx=0 tx=0).stop 2024-01-11T08:29:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.637+0000 7ff3289ff700 1 -- 172.21.15.27:0/1014757177 shutdown_connections 2024-01-11T08:29:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.637+0000 7ff3289ff700 1 --2- 172.21.15.27:0/1014757177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff32406b610 0x7ff324158620 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.637+0000 7ff3289ff700 1 -- 172.21.15.27:0/1014757177 >> 172.21.15.27:0/1014757177 conn(0x7ff3240680e0 msgr2=0x7ff324069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.637+0000 7ff3289ff700 1 -- 172.21.15.27:0/1014757177 shutdown_connections 2024-01-11T08:29:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.637+0000 7ff3289ff700 1 -- 172.21.15.27:0/1014757177 wait complete. 2024-01-11T08:29:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.639+0000 7ff3289ff700 1 Processor -- start 2024-01-11T08:29:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.639+0000 7ff3289ff700 1 -- start start 2024-01-11T08:29:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.640+0000 7ff3289ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff32406b610 0x7ff324146de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.640+0000 7ff3289ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff3240b19d0 0x7ff3240c4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.640+0000 7ff3289ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3241575e0 0x7ff32415d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.640+0000 7ff3289ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3240c8930 con 0x7ff32406b610 2024-01-11T08:29:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.640+0000 7ff3289ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3240c5130 con 0x7ff3241575e0 2024-01-11T08:29:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.640+0000 7ff3289ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3240c4760 con 0x7ff3240b19d0 2024-01-11T08:29:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.640+0000 7ff32259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff32406b610 0x7ff324146de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.640+0000 7ff32259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff32406b610 0x7ff324146de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57606/0 (socket says 172.21.15.27:57606) 2024-01-11T08:29:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.640+0000 7ff32259c700 1 -- 172.21.15.27:0/2513898989 learned_addr learned my addr 172.21.15.27:0/2513898989 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.641+0000 7ff321d9b700 1 --2- 172.21.15.27:0/2513898989 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff3240b19d0 0x7ff3240c4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.641+0000 7ff322d9d700 1 --2- 172.21.15.27:0/2513898989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3241575e0 0x7ff32415d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.641+0000 7ff32259c700 1 -- 172.21.15.27:0/2513898989 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff3240b19d0 msgr2=0x7ff3240c4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.641+0000 7ff32259c700 1 --2- 172.21.15.27:0/2513898989 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff3240b19d0 0x7ff3240c4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.641+0000 7ff32259c700 1 -- 172.21.15.27:0/2513898989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3241575e0 msgr2=0x7ff32415d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.641+0000 7ff32259c700 1 --2- 172.21.15.27:0/2513898989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3241575e0 0x7ff32415d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.641+0000 7ff32259c700 1 -- 172.21.15.27:0/2513898989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c009a20 con 0x7ff32406b610 2024-01-11T08:29:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.641+0000 7ff32259c700 1 --2- 172.21.15.27:0/2513898989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff32406b610 0x7ff324146de0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7ff31400cb90 tx=0x7ff314001c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.641+0000 7ff3127fc700 1 -- 172.21.15.27:0/2513898989 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31400cea0 con 0x7ff32406b610 2024-01-11T08:29:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.641+0000 7ff3127fc700 1 -- 172.21.15.27:0/2513898989 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff314007360 con 0x7ff32406b610 2024-01-11T08:29:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.641+0000 7ff3127fc700 1 -- 172.21.15.27:0/2513898989 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314007700 con 0x7ff32406b610 2024-01-11T08:29:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.641+0000 7ff3289ff700 1 -- 172.21.15.27:0/2513898989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3240c3e70 con 0x7ff32406b610 2024-01-11T08:29:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.642+0000 7ff3289ff700 1 -- 172.21.15.27:0/2513898989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3240c4110 con 0x7ff32406b610 2024-01-11T08:29:55.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.643+0000 7ff3289ff700 1 -- 172.21.15.27:0/2513898989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3240c4a40 con 0x7ff32406b610 2024-01-11T08:29:55.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.644+0000 7ff3127fc700 1 -- 172.21.15.27:0/2513898989 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 74) v1 ==== 84901+0+0 (secure 0 0 0) 0x7ff314004a40 con 0x7ff32406b610 2024-01-11T08:29:55.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.645+0000 7ff3127fc700 1 --2- 172.21.15.27:0/2513898989 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff3040670e0 0x7ff304069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:55.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.645+0000 7ff3127fc700 1 -- 172.21.15.27:0/2513898989 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7ff314083b50 con 0x7ff32406b610 2024-01-11T08:29:55.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.645+0000 7ff321d9b700 1 -- 172.21.15.27:0/2513898989 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff3040670e0 msgr2=0x7ff304069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:55.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.645+0000 7ff321d9b700 1 --2- 172.21.15.27:0/2513898989 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff3040670e0 0x7ff304069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:55.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.649+0000 7ff3127fc700 1 -- 172.21.15.27:0/2513898989 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff314050a80 con 0x7ff32406b610 2024-01-11T08:29:55.753 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:55.754+0000 7ff71dca21c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:29:55.797 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:55.797+0000 7fad63d071c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:29:55.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.846+0000 7ff321d9b700 1 -- 172.21.15.27:0/2513898989 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff3040670e0 msgr2=0x7ff304069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:55.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.846+0000 7ff321d9b700 1 --2- 172.21.15.27:0/2513898989 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff3040670e0 0x7ff304069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:55.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.900+0000 7ff3289ff700 1 -- 172.21.15.27:0/2513898989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff324158b60 con 0x7ff32406b610 2024-01-11T08:29:55.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.900+0000 7ff3127fc700 1 -- 172.21.15.27:0/2513898989 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4596 (secure 0 0 0) 0x7ff324158b60 con 0x7ff32406b610 2024-01-11T08:29:55.900 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:55.901 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":69,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6956,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1277153217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1277153217},{"type":"v1","addr":"172.21.15.77:6833","nonce":1277153217}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7777,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2435914089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2435914089},{"type":"v1","addr":"172.21.15.27:6835","nonce":2435914089}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7792,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/4128023892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4128023892},{"type":"v1","addr":"172.21.15.77:6837","nonce":4128023892}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:29:30.441936+0000","modified":"2024-01-11T08:29:33.957014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8058":{"gid":8058,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6834/716078331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":716078331},{"type":"v1","addr":"172.21.15.27:6834","nonce":716078331}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2024-01-11T08:29:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.903+0000 7ff3289ff700 1 -- 172.21.15.27:0/2513898989 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff3040670e0 msgr2=0x7ff304069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.903+0000 7ff3289ff700 1 --2- 172.21.15.27:0/2513898989 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff3040670e0 0x7ff304069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.903+0000 7ff3289ff700 1 -- 172.21.15.27:0/2513898989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff32406b610 msgr2=0x7ff324146de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.903+0000 7ff3289ff700 1 --2- 172.21.15.27:0/2513898989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff32406b610 0x7ff324146de0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7ff31400cb90 tx=0x7ff314001c40 comp rx=0 tx=0).stop 2024-01-11T08:29:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.903+0000 7ff3289ff700 1 -- 172.21.15.27:0/2513898989 shutdown_connections 2024-01-11T08:29:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.903+0000 7ff3289ff700 1 --2- 172.21.15.27:0/2513898989 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff3040670e0 0x7ff304069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:55.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.903+0000 7ff3289ff700 1 --2- 172.21.15.27:0/2513898989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3241575e0 0x7ff32415d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:55.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.903+0000 7ff3289ff700 1 --2- 172.21.15.27:0/2513898989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff32406b610 0x7ff324146de0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:55.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.903+0000 7ff3289ff700 1 --2- 172.21.15.27:0/2513898989 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff3240b19d0 0x7ff3240c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:55.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.903+0000 7ff3289ff700 1 -- 172.21.15.27:0/2513898989 >> 172.21.15.27:0/2513898989 conn(0x7ff3240680e0 msgr2=0x7ff32406de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:55.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.903+0000 7ff3289ff700 1 -- 172.21.15.27:0/2513898989 shutdown_connections 2024-01-11T08:29:55.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:55.903+0000 7ff3289ff700 1 -- 172.21.15.27:0/2513898989 wait complete. 2024-01-11T08:29:55.904 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 69 2024-01-11T08:29:55.921 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:29:56.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.169+0000 7f626f5ff700 1 Processor -- start 2024-01-11T08:29:56.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.169+0000 7f626f5ff700 1 -- start start 2024-01-11T08:29:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.171+0000 7f626f5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f62681575e0 0x7f6268158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.171+0000 7f626f5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f626806b610 0x7f62680b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.171+0000 7f626f5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6268158b60 con 0x7f62680b1800 2024-01-11T08:29:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.171+0000 7f626f5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62680cb5e0 con 0x7f62681575e0 2024-01-11T08:29:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.171+0000 7f626f5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62680b1c80 con 0x7f626806b610 2024-01-11T08:29:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.171+0000 7f626cb9a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f626806b610 0x7f62680b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.171+0000 7f626d39b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f62681575e0 0x7f6268158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.171+0000 7f626db9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f62680b1800 0x7f62680ca1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58046/0 (socket says 172.21.15.27:58046) 2024-01-11T08:29:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.171+0000 7f626db9c700 1 -- 172.21.15.27:0/2287592958 learned_addr learned my addr 172.21.15.27:0/2287592958 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:56.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.172+0000 7f626cb9a700 1 -- 172.21.15.27:0/2287592958 >> v2:172.21.15.77:3300/0 conn(0x7f62681575e0 msgr2=0x7f6268158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:56.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.172+0000 7f626cb9a700 1 --2- 172.21.15.27:0/2287592958 >> v2:172.21.15.77:3300/0 conn(0x7f62681575e0 0x7f6268158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.172+0000 7f626cb9a700 1 -- 172.21.15.27:0/2287592958 >> v1:172.21.15.27:6789/0 conn(0x7f62680b1800 legacy=0x7f62680ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.172+0000 7f626cb9a700 1 -- 172.21.15.27:0/2287592958 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62681599b0 con 0x7f626806b610 2024-01-11T08:29:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.172+0000 7f626d39b700 1 --2- 172.21.15.27:0/2287592958 >> v2:172.21.15.77:3300/0 conn(0x7f62681575e0 0x7f6268158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.172+0000 7f625ffff700 1 -- 172.21.15.27:0/2287592958 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267517574 0 0) 0x7f6260001490 con 0x7f62680b1800 2024-01-11T08:29:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.172+0000 7f626cb9a700 1 --2- 172.21.15.27:0/2287592958 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f626806b610 0x7f62680b12c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f62580098e0 tx=0x7f62580091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b73524fb2757b68 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.172+0000 7f625ffff700 1 -- 172.21.15.27:0/2287592958 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625800b640 con 0x7f626806b610 2024-01-11T08:29:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.172+0000 7f625ffff700 1 -- 172.21.15.27:0/2287592958 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625800b7a0 con 0x7f626806b610 2024-01-11T08:29:56.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.172+0000 7f625ffff700 1 -- 172.21.15.27:0/2287592958 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625800bb40 con 0x7f626806b610 2024-01-11T08:29:56.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.173+0000 7f626f5ff700 1 -- 172.21.15.27:0/2287592958 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f626806b610 msgr2=0x7f62680b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:56.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.173+0000 7f626f5ff700 1 --2- 172.21.15.27:0/2287592958 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f626806b610 0x7f62680b12c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f62580098e0 tx=0x7f62580091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.173+0000 7f626f5ff700 1 -- 172.21.15.27:0/2287592958 shutdown_connections 2024-01-11T08:29:56.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.173+0000 7f626f5ff700 1 --2- 172.21.15.27:0/2287592958 >> v2:172.21.15.77:3300/0 conn(0x7f62681575e0 0x7f6268158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.173+0000 7f626f5ff700 1 --2- 172.21.15.27:0/2287592958 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f626806b610 0x7f62680b12c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.173+0000 7f626f5ff700 1 -- 172.21.15.27:0/2287592958 >> 172.21.15.27:0/2287592958 conn(0x7f62680680e0 msgr2=0x7f6268069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:56.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.173+0000 7f626f5ff700 1 -- 172.21.15.27:0/2287592958 shutdown_connections 2024-01-11T08:29:56.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.173+0000 7f626f5ff700 1 -- 172.21.15.27:0/2287592958 wait complete. 2024-01-11T08:29:56.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.175+0000 7f626f5ff700 1 Processor -- start 2024-01-11T08:29:56.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.175+0000 7f626f5ff700 1 -- start start 2024-01-11T08:29:56.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.176+0000 7f626f5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f626806b610 0x7f626815bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:56.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.176+0000 7f626f5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f62680b1800 0x7f6268142700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:56.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.176+0000 7f626f5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62681575e0 0x7f6268142c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:56.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.176+0000 7f626f5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62680c11b0 con 0x7f626806b610 2024-01-11T08:29:56.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.176+0000 7f626f5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6268143180 con 0x7f62681575e0 2024-01-11T08:29:56.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.176+0000 7f626f5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6268168990 con 0x7f62680b1800 2024-01-11T08:29:56.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.176+0000 7f626d39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f626806b610 0x7f626815bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:56.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.176+0000 7f626d39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f626806b610 0x7f626815bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57608/0 (socket says 172.21.15.27:57608) 2024-01-11T08:29:56.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.176+0000 7f626d39b700 1 -- 172.21.15.27:0/1174774887 learned_addr learned my addr 172.21.15.27:0/1174774887 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:56.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.176+0000 7f626db9c700 1 --2- 172.21.15.27:0/1174774887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62681575e0 0x7f6268142c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:56.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.177+0000 7f626cb9a700 1 --2- 172.21.15.27:0/1174774887 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f62680b1800 0x7f6268142700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:56.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.177+0000 7f626d39b700 1 -- 172.21.15.27:0/1174774887 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f62680b1800 msgr2=0x7f6268142700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:56.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.177+0000 7f626d39b700 1 --2- 172.21.15.27:0/1174774887 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f62680b1800 0x7f6268142700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.177+0000 7f626d39b700 1 -- 172.21.15.27:0/1174774887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62681575e0 msgr2=0x7f6268142c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:56.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.177+0000 7f626d39b700 1 --2- 172.21.15.27:0/1174774887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62681575e0 0x7f6268142c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.177+0000 7f626d39b700 1 -- 172.21.15.27:0/1174774887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6264009420 con 0x7f626806b610 2024-01-11T08:29:56.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.177+0000 7f626d39b700 1 --2- 172.21.15.27:0/1174774887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f626806b610 0x7f626815bb80 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f626400ec80 tx=0x7f62640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:56.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.177+0000 7f625d7fa700 1 -- 172.21.15.27:0/1174774887 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626400c920 con 0x7f626806b610 2024-01-11T08:29:56.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.177+0000 7f625d7fa700 1 -- 172.21.15.27:0/1174774887 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f626400ca80 con 0x7f626806b610 2024-01-11T08:29:56.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.177+0000 7f625d7fa700 1 -- 172.21.15.27:0/1174774887 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626400ce00 con 0x7f626806b610 2024-01-11T08:29:56.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.177+0000 7f626f5ff700 1 -- 172.21.15.27:0/1174774887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6258009420 con 0x7f626806b610 2024-01-11T08:29:56.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.178+0000 7f626f5ff700 1 -- 172.21.15.27:0/1174774887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62680c1720 con 0x7f626806b610 2024-01-11T08:29:56.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.179+0000 7f626f5ff700 1 -- 172.21.15.27:0/1174774887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62680718d0 con 0x7f626806b610 2024-01-11T08:29:56.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.180+0000 7f625d7fa700 1 -- 172.21.15.27:0/1174774887 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 74) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f6264013680 con 0x7f626806b610 2024-01-11T08:29:56.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.181+0000 7f625d7fa700 1 --2- 172.21.15.27:0/1174774887 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f624c0670b0 0x7f624c069560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:56.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.181+0000 7f625d7fa700 1 -- 172.21.15.27:0/1174774887 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f6264084dc0 con 0x7f626806b610 2024-01-11T08:29:56.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.181+0000 7f626cb9a700 1 -- 172.21.15.27:0/1174774887 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f624c0670b0 msgr2=0x7f624c069560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:56.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.181+0000 7f626cb9a700 1 --2- 172.21.15.27:0/1174774887 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f624c0670b0 0x7f624c069560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:56.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.185+0000 7f625d7fa700 1 -- 172.21.15.27:0/1174774887 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6264051bd0 con 0x7f626806b610 2024-01-11T08:29:56.277 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:56.278+0000 7ff71dca21c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:29:56.365 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:56.365+0000 7fad63d071c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:29:56.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.382+0000 7f626cb9a700 1 -- 172.21.15.27:0/1174774887 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f624c0670b0 msgr2=0x7f624c069560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:56.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.382+0000 7f626cb9a700 1 --2- 172.21.15.27:0/1174774887 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f624c0670b0 0x7f624c069560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:56.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.390+0000 7f626f5ff700 1 -- 172.21.15.27:0/1174774887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6268156e10 con 0x7f626806b610 2024-01-11T08:29:56.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.391+0000 7f625d7fa700 1 -- 172.21.15.27:0/1174774887 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+15770 (secure 0 0 0) 0x7f62640551f0 con 0x7f626806b610 2024-01-11T08:29:56.391 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:56.391 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":132,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:29:46.941384+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:29:28.131181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-11T08:29:29.438397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/259960245":"2024-01-12T08:29:46.941337+0000","172.21.15.77:6836/41381":"2024-01-12T08:29:46.941337+0000","172.21.15.77:6835/41381":"2024-01-12T08:29:46.941337+0000","172.21.15.77:0/3475507657":"2024-01-12T08:29:46.941337+0000","172.21.15.77:0/3338577024":"2024-01-12T08:29:46.941337+0000","172.21.15.77:0/3974991130":"2024-01-12T08:29:46.941337+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:29:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.393+0000 7f626f5ff700 1 -- 172.21.15.27:0/1174774887 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f624c0670b0 msgr2=0x7f624c069560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.393+0000 7f626f5ff700 1 --2- 172.21.15.27:0/1174774887 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f624c0670b0 0x7f624c069560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.394+0000 7f626f5ff700 1 -- 172.21.15.27:0/1174774887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f626806b610 msgr2=0x7f626815bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.394+0000 7f626f5ff700 1 --2- 172.21.15.27:0/1174774887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f626806b610 0x7f626815bb80 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f626400ec80 tx=0x7f62640091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.394+0000 7f626f5ff700 1 -- 172.21.15.27:0/1174774887 shutdown_connections 2024-01-11T08:29:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.394+0000 7f626f5ff700 1 --2- 172.21.15.27:0/1174774887 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f624c0670b0 0x7f624c069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.394+0000 7f626f5ff700 1 --2- 172.21.15.27:0/1174774887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62681575e0 0x7f6268142c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.394+0000 7f626f5ff700 1 --2- 172.21.15.27:0/1174774887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f626806b610 0x7f626815bb80 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.394+0000 7f626f5ff700 1 --2- 172.21.15.27:0/1174774887 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f62680b1800 0x7f6268142700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.394+0000 7f626f5ff700 1 -- 172.21.15.27:0/1174774887 >> 172.21.15.27:0/1174774887 conn(0x7f62680680e0 msgr2=0x7f6268156150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:56.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.394+0000 7f626f5ff700 1 -- 172.21.15.27:0/1174774887 shutdown_connections 2024-01-11T08:29:56.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.394+0000 7f626f5ff700 1 -- 172.21.15.27:0/1174774887 wait complete. 2024-01-11T08:29:56.412 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T08:29:28.131181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T08:29:29.438397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '131', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T08:29:56.413 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:29:56.448 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:56.448+0000 7ff71dca21c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:29:56.553 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:56.554+0000 7fad63d071c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:29:56.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.624+0000 7f8aea1ff700 1 Processor -- start 2024-01-11T08:29:56.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.624+0000 7f8aea1ff700 1 -- start start 2024-01-11T08:29:56.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.627+0000 7f8aea1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8ae415a1b0 0x7f8ae415a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:56.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.627+0000 7f8aea1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae40cf850 con 0x7f8ae40ccd70 2024-01-11T08:29:56.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.627+0000 7f8aea1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae40cec40 con 0x7f8ae415ae70 2024-01-11T08:29:56.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.627+0000 7f8aea1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae40cef20 con 0x7f8ae415a1b0 2024-01-11T08:29:56.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.627+0000 7f8ae2ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8ae40ccd70 0x7f8ae40cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58058/0 (socket says 172.21.15.27:58058) 2024-01-11T08:29:56.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.627+0000 7f8ae2ffd700 1 -- 172.21.15.27:0/3656961702 learned_addr learned my addr 172.21.15.27:0/3656961702 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:56.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.627+0000 7f8ae37fe700 1 --2- 172.21.15.27:0/3656961702 >> v2:172.21.15.77:3300/0 conn(0x7f8ae415a1b0 0x7f8ae415a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:56.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.628+0000 7f8ae27fc700 1 -- 172.21.15.27:0/3656961702 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960762131 0 0) 0x7f8ae40cf850 con 0x7f8ae40ccd70 2024-01-11T08:29:56.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.628+0000 7f8ae27fc700 1 -- 172.21.15.27:0/3656961702 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8acc003570 con 0x7f8ae40ccd70 2024-01-11T08:29:56.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.628+0000 7f8ae27fc700 1 -- 172.21.15.27:0/3656961702 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926191384 0 0) 0x7f8ae40cec40 con 0x7f8ae415ae70 2024-01-11T08:29:56.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.628+0000 7f8ae27fc700 1 -- 172.21.15.27:0/3656961702 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ae40cf850 con 0x7f8ae415ae70 2024-01-11T08:29:56.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.628+0000 7f8ae37fe700 1 -- 172.21.15.27:0/3656961702 >> v1:172.21.15.77:6789/0 conn(0x7f8ae415ae70 legacy=0x7f8ae415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:56.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.628+0000 7f8ae27fc700 1 -- 172.21.15.27:0/3656961702 <== mon.1 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4024516919 0 0) 0x7f8ae40cf850 con 0x7f8ae415ae70 2024-01-11T08:29:56.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.628+0000 7f8ae37fe700 1 -- 172.21.15.27:0/3656961702 >> v1:172.21.15.27:6789/0 conn(0x7f8ae40ccd70 legacy=0x7f8ae40cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:56.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.628+0000 7f8ae37fe700 1 -- 172.21.15.27:0/3656961702 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae40cf200 con 0x7f8ae415a1b0 2024-01-11T08:29:56.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.629+0000 7f8ae37fe700 1 --2- 172.21.15.27:0/3656961702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae415a1b0 0x7f8ae415a580 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4009860 tx=0x7f8ad40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afccad5a8de22d2d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:56.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.629+0000 7f8ae27fc700 1 -- 172.21.15.27:0/3656961702 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8001470 con 0x7f8ae415a1b0 2024-01-11T08:29:56.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.629+0000 7f8ae27fc700 1 -- 172.21.15.27:0/3656961702 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad4011440 con 0x7f8ae415a1b0 2024-01-11T08:29:56.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.629+0000 7f8ae27fc700 1 -- 172.21.15.27:0/3656961702 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc0017f0 con 0x7f8ae415a1b0 2024-01-11T08:29:56.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.630+0000 7f8aea1ff700 1 -- 172.21.15.27:0/3656961702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae415a1b0 msgr2=0x7f8ae415a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:56.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.630+0000 7f8aea1ff700 1 --2- 172.21.15.27:0/3656961702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae415a1b0 0x7f8ae415a580 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4009860 tx=0x7f8ad40091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.630+0000 7f8aea1ff700 1 -- 172.21.15.27:0/3656961702 shutdown_connections 2024-01-11T08:29:56.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.630+0000 7f8aea1ff700 1 --2- 172.21.15.27:0/3656961702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae415a1b0 0x7f8ae415a580 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.630+0000 7f8aea1ff700 1 -- 172.21.15.27:0/3656961702 >> 172.21.15.27:0/3656961702 conn(0x7f8ae406ed40 msgr2=0x7f8ae406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:56.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.630+0000 7f8aea1ff700 1 -- 172.21.15.27:0/3656961702 shutdown_connections 2024-01-11T08:29:56.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.630+0000 7f8aea1ff700 1 -- 172.21.15.27:0/3656961702 wait complete. 2024-01-11T08:29:56.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.632+0000 7f8aea1ff700 1 Processor -- start 2024-01-11T08:29:56.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.633+0000 7f8aea1ff700 1 -- start start 2024-01-11T08:29:56.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.634+0000 7f8aea1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ae40ccd70 0x7f8ae40b5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:56.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.634+0000 7f8aea1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ae415a1b0 0x7f8ae4158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:56.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.634+0000 7f8aea1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae415ae70 0x7f8ae415ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:56.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.634+0000 7f8aea1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae4158960 con 0x7f8ae415a1b0 2024-01-11T08:29:56.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.634+0000 7f8aea1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae40b4a00 con 0x7f8ae415ae70 2024-01-11T08:29:56.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.634+0000 7f8aea1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae40b4ce0 con 0x7f8ae40ccd70 2024-01-11T08:29:56.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.634+0000 7f8ae2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ae415a1b0 0x7f8ae4158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:56.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.634+0000 7f8ae37fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ae40ccd70 0x7f8ae40b5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:56.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.634+0000 7f8ae2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ae415a1b0 0x7f8ae4158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57618/0 (socket says 172.21.15.27:57618) 2024-01-11T08:29:56.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.634+0000 7f8ae2ffd700 1 -- 172.21.15.27:0/10569781 learned_addr learned my addr 172.21.15.27:0/10569781 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:56.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.634+0000 7f8ae3fff700 1 --2- 172.21.15.27:0/10569781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae415ae70 0x7f8ae415ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:56.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.635+0000 7f8ae37fe700 1 -- 172.21.15.27:0/10569781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae415ae70 msgr2=0x7f8ae415ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:56.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.635+0000 7f8ae37fe700 1 --2- 172.21.15.27:0/10569781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae415ae70 0x7f8ae415ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.635+0000 7f8ae37fe700 1 -- 172.21.15.27:0/10569781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ae415a1b0 msgr2=0x7f8ae4158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:56.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.635+0000 7f8ae37fe700 1 --2- 172.21.15.27:0/10569781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ae415a1b0 0x7f8ae4158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.635+0000 7f8ae37fe700 1 -- 172.21.15.27:0/10569781 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad4009420 con 0x7f8ae40ccd70 2024-01-11T08:29:56.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.635+0000 7f8ae2ffd700 1 --2- 172.21.15.27:0/10569781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ae415a1b0 0x7f8ae4158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:56.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.635+0000 7f8ae37fe700 1 --2- 172.21.15.27:0/10569781 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ae40ccd70 0x7f8ae40b5aa0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4013000 tx=0x7f8ad4016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:56.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.635+0000 7f8acb7fe700 1 -- 172.21.15.27:0/10569781 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad4021070 con 0x7f8ae40ccd70 2024-01-11T08:29:56.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.635+0000 7f8acb7fe700 1 -- 172.21.15.27:0/10569781 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad4010900 con 0x7f8ae40ccd70 2024-01-11T08:29:56.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.635+0000 7f8acb7fe700 1 -- 172.21.15.27:0/10569781 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad4010ca0 con 0x7f8ae40ccd70 2024-01-11T08:29:56.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.635+0000 7f8aea1ff700 1 -- 172.21.15.27:0/10569781 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae40b4fc0 con 0x7f8ae40ccd70 2024-01-11T08:29:56.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.636+0000 7f8aea1ff700 1 -- 172.21.15.27:0/10569781 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae40b5580 con 0x7f8ae40ccd70 2024-01-11T08:29:56.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.637+0000 7f8aea1ff700 1 -- 172.21.15.27:0/10569781 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae404c3c0 con 0x7f8ae40ccd70 2024-01-11T08:29:56.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.638+0000 7f8acb7fe700 1 -- 172.21.15.27:0/10569781 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 74) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f8ad4018030 con 0x7f8ae40ccd70 2024-01-11T08:29:56.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.639+0000 7f8acb7fe700 1 --2- 172.21.15.27:0/10569781 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8ad00670e0 0x7f8ad0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:56.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.639+0000 7f8acb7fe700 1 -- 172.21.15.27:0/10569781 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f8ad40851f0 con 0x7f8ae40ccd70 2024-01-11T08:29:56.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.639+0000 7f8ae2ffd700 1 -- 172.21.15.27:0/10569781 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8ad00670e0 msgr2=0x7f8ad0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:56.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.639+0000 7f8ae2ffd700 1 --2- 172.21.15.27:0/10569781 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8ad00670e0 0x7f8ad0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:56.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.643+0000 7f8acb7fe700 1 -- 172.21.15.27:0/10569781 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8ad4052000 con 0x7f8ae40ccd70 2024-01-11T08:29:56.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.840+0000 7f8ae2ffd700 1 -- 172.21.15.27:0/10569781 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8ad00670e0 msgr2=0x7f8ad0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:56.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.840+0000 7f8ae2ffd700 1 --2- 172.21.15.27:0/10569781 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8ad00670e0 0x7f8ad0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:56.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.849+0000 7f8aea1ff700 1 -- 172.21.15.27:0/10569781 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8ae40b24d0 con 0x7f8ae40ccd70 2024-01-11T08:29:56.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.850+0000 7f8acb7fe700 1 -- 172.21.15.27:0/10569781 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v132) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8ad4055620 con 0x7f8ae40ccd70 2024-01-11T08:29:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:29:56.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.852+0000 7f8aea1ff700 1 -- 172.21.15.27:0/10569781 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8ad00670e0 msgr2=0x7f8ad0069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:56.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.852+0000 7f8aea1ff700 1 --2- 172.21.15.27:0/10569781 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8ad00670e0 0x7f8ad0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.852+0000 7f8aea1ff700 1 -- 172.21.15.27:0/10569781 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ae40ccd70 msgr2=0x7f8ae40b5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:56.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.852+0000 7f8aea1ff700 1 --2- 172.21.15.27:0/10569781 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ae40ccd70 0x7f8ae40b5aa0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4013000 tx=0x7f8ad4016000 comp rx=0 tx=0).stop 2024-01-11T08:29:56.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.852+0000 7f8aea1ff700 1 -- 172.21.15.27:0/10569781 shutdown_connections 2024-01-11T08:29:56.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.852+0000 7f8aea1ff700 1 --2- 172.21.15.27:0/10569781 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8ad00670e0 0x7f8ad0069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.852+0000 7f8aea1ff700 1 --2- 172.21.15.27:0/10569781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ae415ae70 0x7f8ae415ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.852+0000 7f8aea1ff700 1 --2- 172.21.15.27:0/10569781 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ae40ccd70 0x7f8ae40b5aa0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.852+0000 7f8aea1ff700 1 --2- 172.21.15.27:0/10569781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ae415a1b0 0x7f8ae4158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:56.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.852+0000 7f8aea1ff700 1 -- 172.21.15.27:0/10569781 >> 172.21.15.27:0/10569781 conn(0x7f8ae406ed40 msgr2=0x7f8ae415c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:56.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.852+0000 7f8aea1ff700 1 -- 172.21.15.27:0/10569781 shutdown_connections 2024-01-11T08:29:56.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:56.852+0000 7f8aea1ff700 1 -- 172.21.15.27:0/10569781 wait complete. 2024-01-11T08:29:56.865 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:56.865+0000 7ff71dca21c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:29:56.870 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T08:29:57.004 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:57.004+0000 7fad63d071c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:29:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.077+0000 7f31f6fff700 1 Processor -- start 2024-01-11T08:29:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.077+0000 7f31f6fff700 1 -- start start 2024-01-11T08:29:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.078+0000 7f31f6fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f015a1b0 0x7f31f015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.078+0000 7f31f6fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f00cf850 con 0x7f31f00ccd70 2024-01-11T08:29:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.078+0000 7f31f6fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f00cec40 con 0x7f31f015ae70 2024-01-11T08:29:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.078+0000 7f31f6fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f00cef20 con 0x7f31f015a1b0 2024-01-11T08:29:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.079+0000 7f31f4d9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f015a1b0 0x7f31f015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.079+0000 7f31effff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f31f00ccd70 0x7f31f00cd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58060/0 (socket says 172.21.15.27:58060) 2024-01-11T08:29:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.079+0000 7f31effff700 1 -- 172.21.15.27:0/4122739784 learned_addr learned my addr 172.21.15.27:0/4122739784 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.079+0000 7f31ef7fe700 1 -- 172.21.15.27:0/4122739784 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209168493 0 0) 0x7f31f00cf850 con 0x7f31f00ccd70 2024-01-11T08:29:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.079+0000 7f31ef7fe700 1 -- 172.21.15.27:0/4122739784 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31e0003570 con 0x7f31f00ccd70 2024-01-11T08:29:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.079+0000 7f31f4d9b700 1 -- 172.21.15.27:0/4122739784 >> v1:172.21.15.77:6789/0 conn(0x7f31f015ae70 legacy=0x7f31f015b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.079+0000 7f31ef7fe700 1 -- 172.21.15.27:0/4122739784 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892000695 0 0) 0x7f31f00cec40 con 0x7f31f015ae70 2024-01-11T08:29:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.079+0000 7f31f4d9b700 1 -- 172.21.15.27:0/4122739784 >> v1:172.21.15.27:6789/0 conn(0x7f31f00ccd70 legacy=0x7f31f00cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.079+0000 7f31f4d9b700 1 -- 172.21.15.27:0/4122739784 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f00cf200 con 0x7f31f015a1b0 2024-01-11T08:29:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.080+0000 7f31f4d9b700 1 --2- 172.21.15.27:0/4122739784 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f015a1b0 0x7f31f015a580 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f31e4009910 tx=0x7f31e40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e3a5792d7af5337 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.080+0000 7f31ef7fe700 1 -- 172.21.15.27:0/4122739784 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e400b680 con 0x7f31f015a1b0 2024-01-11T08:29:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.080+0000 7f31ef7fe700 1 -- 172.21.15.27:0/4122739784 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e400b7e0 con 0x7f31f015a1b0 2024-01-11T08:29:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.080+0000 7f31ef7fe700 1 -- 172.21.15.27:0/4122739784 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e400bba0 con 0x7f31f015a1b0 2024-01-11T08:29:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.081+0000 7f31f6fff700 1 -- 172.21.15.27:0/4122739784 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f015a1b0 msgr2=0x7f31f015a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.081+0000 7f31f6fff700 1 --2- 172.21.15.27:0/4122739784 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f015a1b0 0x7f31f015a580 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f31e4009910 tx=0x7f31e40091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.081+0000 7f31f6fff700 1 -- 172.21.15.27:0/4122739784 shutdown_connections 2024-01-11T08:29:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.081+0000 7f31f6fff700 1 --2- 172.21.15.27:0/4122739784 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f015a1b0 0x7f31f015a580 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.081+0000 7f31f6fff700 1 -- 172.21.15.27:0/4122739784 >> 172.21.15.27:0/4122739784 conn(0x7f31f006ed40 msgr2=0x7f31f006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.081+0000 7f31f6fff700 1 -- 172.21.15.27:0/4122739784 shutdown_connections 2024-01-11T08:29:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.081+0000 7f31f6fff700 1 -- 172.21.15.27:0/4122739784 wait complete. 2024-01-11T08:29:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.083+0000 7f31f6fff700 1 Processor -- start 2024-01-11T08:29:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.083+0000 7f31f6fff700 1 -- start start 2024-01-11T08:29:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.085+0000 7f31f6fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f00ccd70 0x7f31f0169a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.085+0000 7f31f6fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f015a1b0 0x7f31f00b5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.085+0000 7f31f6fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31f015ae70 0x7f31f0146c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.085+0000 7f31f6fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f00c9490 con 0x7f31f015ae70 2024-01-11T08:29:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.085+0000 7f31f6fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f0169f60 con 0x7f31f00ccd70 2024-01-11T08:29:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.085+0000 7f31f6fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f00c8ba0 con 0x7f31f015a1b0 2024-01-11T08:29:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.085+0000 7f31f559c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31f015ae70 0x7f31f0146c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.085+0000 7f31effff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f015a1b0 0x7f31f00b5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.085+0000 7f31f559c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31f015ae70 0x7f31f0146c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57622/0 (socket says 172.21.15.27:57622) 2024-01-11T08:29:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.085+0000 7f31f4d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f00ccd70 0x7f31f0169a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.085+0000 7f31f559c700 1 -- 172.21.15.27:0/3802676246 learned_addr learned my addr 172.21.15.27:0/3802676246 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:57.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.085+0000 7f31effff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f015a1b0 0x7f31f00b5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:44732/0 (socket says 172.21.15.27:44732) 2024-01-11T08:29:57.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.085+0000 7f31effff700 1 -- 172.21.15.27:0/3802676246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f00ccd70 msgr2=0x7f31f0169a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:57.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.086+0000 7f31effff700 1 --2- 172.21.15.27:0/3802676246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f00ccd70 0x7f31f0169a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.086+0000 7f31effff700 1 -- 172.21.15.27:0/3802676246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31f015ae70 msgr2=0x7f31f0146c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:57.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.086+0000 7f31effff700 1 --2- 172.21.15.27:0/3802676246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31f015ae70 0x7f31f0146c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.086+0000 7f31effff700 1 -- 172.21.15.27:0/3802676246 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e4009420 con 0x7f31f015a1b0 2024-01-11T08:29:57.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.086+0000 7f31f559c700 1 --2- 172.21.15.27:0/3802676246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31f015ae70 0x7f31f0146c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:57.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.086+0000 7f31effff700 1 --2- 172.21.15.27:0/3802676246 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f015a1b0 0x7f31f00b5720 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f31dc0058d0 tx=0x7f31dc007d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:57.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.086+0000 7f31ecff9700 1 -- 172.21.15.27:0/3802676246 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31dc005b30 con 0x7f31f015a1b0 2024-01-11T08:29:57.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.086+0000 7f31ecff9700 1 -- 172.21.15.27:0/3802676246 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31dc005c90 con 0x7f31f015a1b0 2024-01-11T08:29:57.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.086+0000 7f31ecff9700 1 -- 172.21.15.27:0/3802676246 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31dc007320 con 0x7f31f015a1b0 2024-01-11T08:29:57.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.086+0000 7f31f6fff700 1 -- 172.21.15.27:0/3802676246 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f01604b0 con 0x7f31f015a1b0 2024-01-11T08:29:57.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.086+0000 7f31f6fff700 1 -- 172.21.15.27:0/3802676246 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f00c9120 con 0x7f31f015a1b0 2024-01-11T08:29:57.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.087+0000 7f31f6fff700 1 -- 172.21.15.27:0/3802676246 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f00ce4c0 con 0x7f31f015a1b0 2024-01-11T08:29:57.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.088+0000 7f31ecff9700 1 -- 172.21.15.27:0/3802676246 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 74) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f31dc01b3b0 con 0x7f31f015a1b0 2024-01-11T08:29:57.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.089+0000 7f31ecff9700 1 --2- 172.21.15.27:0/3802676246 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f31d4067110 0x7f31d40695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:57.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.089+0000 7f31ecff9700 1 -- 172.21.15.27:0/3802676246 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f31dc083cb0 con 0x7f31f015a1b0 2024-01-11T08:29:57.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.089+0000 7f31f4d9b700 1 -- 172.21.15.27:0/3802676246 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f31d4067110 msgr2=0x7f31d40695c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:57.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.089+0000 7f31f4d9b700 1 --2- 172.21.15.27:0/3802676246 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f31d4067110 0x7f31d40695c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:57.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.093+0000 7f31ecff9700 1 -- 172.21.15.27:0/3802676246 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f31dc050ac0 con 0x7f31f015a1b0 2024-01-11T08:29:57.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.290+0000 7f31f4d9b700 1 -- 172.21.15.27:0/3802676246 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f31d4067110 msgr2=0x7f31d40695c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:57.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.290+0000 7f31f4d9b700 1 --2- 172.21.15.27:0/3802676246 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f31d4067110 0x7f31d40695c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:57.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.298+0000 7f31f6fff700 1 -- 172.21.15.27:0/3802676246 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f31f00c9600 con 0x7f31f015a1b0 2024-01-11T08:29:57.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.299+0000 7f31ecff9700 1 -- 172.21.15.27:0/3802676246 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v132) v1 ==== 104+0+11 (secure 0 0 0) 0x7f31dc0170a0 con 0x7f31f015a1b0 2024-01-11T08:29:57.298 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T08:29:57.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.301+0000 7f31f6fff700 1 -- 172.21.15.27:0/3802676246 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f31d4067110 msgr2=0x7f31d40695c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:57.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.301+0000 7f31f6fff700 1 --2- 172.21.15.27:0/3802676246 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f31d4067110 0x7f31d40695c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.301+0000 7f31f6fff700 1 -- 172.21.15.27:0/3802676246 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f015a1b0 msgr2=0x7f31f00b5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:57.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.301+0000 7f31f6fff700 1 --2- 172.21.15.27:0/3802676246 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f015a1b0 0x7f31f00b5720 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f31dc0058d0 tx=0x7f31dc007d10 comp rx=0 tx=0).stop 2024-01-11T08:29:57.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.301+0000 7f31f6fff700 1 -- 172.21.15.27:0/3802676246 shutdown_connections 2024-01-11T08:29:57.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.301+0000 7f31f6fff700 1 --2- 172.21.15.27:0/3802676246 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f31d4067110 0x7f31d40695c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.301+0000 7f31f6fff700 1 --2- 172.21.15.27:0/3802676246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31f015ae70 0x7f31f0146c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.301+0000 7f31f6fff700 1 --2- 172.21.15.27:0/3802676246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f00ccd70 0x7f31f0169a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.301+0000 7f31f6fff700 1 --2- 172.21.15.27:0/3802676246 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31f015a1b0 0x7f31f00b5720 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.301+0000 7f31f6fff700 1 -- 172.21.15.27:0/3802676246 >> 172.21.15.27:0/3802676246 conn(0x7f31f006ed40 msgr2=0x7f31f006f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:57.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.301+0000 7f31f6fff700 1 -- 172.21.15.27:0/3802676246 shutdown_connections 2024-01-11T08:29:57.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.301+0000 7f31f6fff700 1 -- 172.21.15.27:0/3802676246 wait complete. 2024-01-11T08:29:57.319 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T08:29:57.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.522+0000 7f36973ff700 1 Processor -- start 2024-01-11T08:29:57.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.522+0000 7f36973ff700 1 -- start start 2024-01-11T08:29:57.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.524+0000 7f36973ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f369015a1b0 0x7f369015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:57.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.524+0000 7f36973ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f369015ae70 0x7f369015b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:57.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.524+0000 7f36973ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36900cf850 con 0x7f36900ccd70 2024-01-11T08:29:57.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.524+0000 7f36973ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36900cec40 con 0x7f369015a1b0 2024-01-11T08:29:57.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.524+0000 7f36973ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36900cef20 con 0x7f369015ae70 2024-01-11T08:29:57.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.524+0000 7f369519b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f369015a1b0 0x7f369015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:57.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.524+0000 7f369519b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f369015a1b0 0x7f369015a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57628/0 (socket says 172.21.15.27:57628) 2024-01-11T08:29:57.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.524+0000 7f369519b700 1 -- 172.21.15.27:0/1596473892 learned_addr learned my addr 172.21.15.27:0/1596473892 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.524+0000 7f369599c700 1 --2- 172.21.15.27:0/1596473892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f369015ae70 0x7f369015b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.524+0000 7f369519b700 1 -- 172.21.15.27:0/1596473892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f369015ae70 msgr2=0x7f369015b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.524+0000 7f369519b700 1 --2- 172.21.15.27:0/1596473892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f369015ae70 0x7f369015b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.524+0000 7f369519b700 1 -- 172.21.15.27:0/1596473892 >> v1:172.21.15.77:6789/0 conn(0x7f36900ccd70 legacy=0x7f36900cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.524+0000 7f369519b700 1 -- 172.21.15.27:0/1596473892 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36900cf200 con 0x7f369015a1b0 2024-01-11T08:29:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.524+0000 7f369599c700 1 --2- 172.21.15.27:0/1596473892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f369015ae70 0x7f369015b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.525+0000 7f369519b700 1 --2- 172.21.15.27:0/1596473892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f369015a1b0 0x7f369015a580 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f368c0099b0 tx=0x7f368c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0c4091bc956668b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.525+0000 7f3687fff700 1 -- 172.21.15.27:0/1596473892 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368c00b860 con 0x7f369015a1b0 2024-01-11T08:29:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.525+0000 7f3687fff700 1 -- 172.21.15.27:0/1596473892 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368c00b9c0 con 0x7f369015a1b0 2024-01-11T08:29:57.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.525+0000 7f3687fff700 1 -- 172.21.15.27:0/1596473892 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368c00bd60 con 0x7f369015a1b0 2024-01-11T08:29:57.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.525+0000 7f36973ff700 1 -- 172.21.15.27:0/1596473892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f369015a1b0 msgr2=0x7f369015a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:57.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.525+0000 7f36973ff700 1 --2- 172.21.15.27:0/1596473892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f369015a1b0 0x7f369015a580 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f368c0099b0 tx=0x7f368c009250 comp rx=0 tx=0).stop 2024-01-11T08:29:57.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.526+0000 7f36973ff700 1 -- 172.21.15.27:0/1596473892 shutdown_connections 2024-01-11T08:29:57.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.526+0000 7f36973ff700 1 --2- 172.21.15.27:0/1596473892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f369015a1b0 0x7f369015a580 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.526+0000 7f36973ff700 1 --2- 172.21.15.27:0/1596473892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f369015ae70 0x7f369015b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.526+0000 7f36973ff700 1 -- 172.21.15.27:0/1596473892 >> 172.21.15.27:0/1596473892 conn(0x7f369006ed40 msgr2=0x7f369006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:57.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.526+0000 7f36973ff700 1 -- 172.21.15.27:0/1596473892 shutdown_connections 2024-01-11T08:29:57.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.526+0000 7f36973ff700 1 -- 172.21.15.27:0/1596473892 wait complete. 2024-01-11T08:29:57.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.528+0000 7f36973ff700 1 Processor -- start 2024-01-11T08:29:57.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.528+0000 7f36973ff700 1 -- start start 2024-01-11T08:29:57.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.530+0000 7f36973ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36900ccd70 0x7f36900b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:57.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.530+0000 7f36973ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f369015a1b0 0x7f3690155290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:57.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.530+0000 7f36973ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f369015ae70 0x7f36900c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:57.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.530+0000 7f36973ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36900b47c0 con 0x7f36900ccd70 2024-01-11T08:29:57.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.530+0000 7f36973ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36900b3bb0 con 0x7f369015ae70 2024-01-11T08:29:57.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.530+0000 7f36973ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36900b3e90 con 0x7f369015a1b0 2024-01-11T08:29:57.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.530+0000 7f369519b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36900ccd70 0x7f36900b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:57.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.530+0000 7f369519b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36900ccd70 0x7f36900b3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57644/0 (socket says 172.21.15.27:57644) 2024-01-11T08:29:57.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.530+0000 7f369519b700 1 -- 172.21.15.27:0/2066188879 learned_addr learned my addr 172.21.15.27:0/2066188879 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:57.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.530+0000 7f369599c700 1 --2- 172.21.15.27:0/2066188879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f369015ae70 0x7f36900c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:57.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.530+0000 7f369499a700 1 --2- 172.21.15.27:0/2066188879 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f369015a1b0 0x7f3690155290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:57.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.531+0000 7f369519b700 1 -- 172.21.15.27:0/2066188879 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f369015a1b0 msgr2=0x7f3690155290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:57.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.531+0000 7f369519b700 1 --2- 172.21.15.27:0/2066188879 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f369015a1b0 0x7f3690155290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.531+0000 7f369519b700 1 -- 172.21.15.27:0/2066188879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f369015ae70 msgr2=0x7f36900c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:57.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.531+0000 7f369519b700 1 --2- 172.21.15.27:0/2066188879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f369015ae70 0x7f36900c6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.531+0000 7f369519b700 1 -- 172.21.15.27:0/2066188879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f368c0094f0 con 0x7f36900ccd70 2024-01-11T08:29:57.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.531+0000 7f369599c700 1 --2- 172.21.15.27:0/2066188879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f369015ae70 0x7f36900c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:29:57.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.531+0000 7f369519b700 1 --2- 172.21.15.27:0/2066188879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36900ccd70 0x7f36900b3380 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f368c009290 tx=0x7f368c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:57.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.531+0000 7f36857fa700 1 -- 172.21.15.27:0/2066188879 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368c026480 con 0x7f36900ccd70 2024-01-11T08:29:57.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.531+0000 7f36857fa700 1 -- 172.21.15.27:0/2066188879 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368c01f070 con 0x7f36900ccd70 2024-01-11T08:29:57.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.531+0000 7f36973ff700 1 -- 172.21.15.27:0/2066188879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36900b4170 con 0x7f36900ccd70 2024-01-11T08:29:57.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.531+0000 7f36857fa700 1 -- 172.21.15.27:0/2066188879 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368c026830 con 0x7f36900ccd70 2024-01-11T08:29:57.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.532+0000 7f36973ff700 1 -- 172.21.15.27:0/2066188879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36900b38c0 con 0x7f36900ccd70 2024-01-11T08:29:57.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.533+0000 7f36973ff700 1 -- 172.21.15.27:0/2066188879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f369004c3c0 con 0x7f36900ccd70 2024-01-11T08:29:57.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.534+0000 7f36857fa700 1 -- 172.21.15.27:0/2066188879 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 74) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f368c026ba0 con 0x7f36900ccd70 2024-01-11T08:29:57.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.535+0000 7f36857fa700 1 --2- 172.21.15.27:0/2066188879 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f36740670e0 0x7f3674069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:57.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.535+0000 7f36857fa700 1 -- 172.21.15.27:0/2066188879 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f368c021070 con 0x7f36900ccd70 2024-01-11T08:29:57.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.536+0000 7f369499a700 1 -- 172.21.15.27:0/2066188879 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f36740670e0 msgr2=0x7f3674069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:57.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.536+0000 7f369499a700 1 --2- 172.21.15.27:0/2066188879 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f36740670e0 0x7f3674069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:57.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.539+0000 7f36857fa700 1 -- 172.21.15.27:0/2066188879 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f368c053730 con 0x7f36900ccd70 2024-01-11T08:29:57.642 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:57.642+0000 7ff71dca21c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:29:57.731 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:57.730+0000 7ff71dca21c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:29:57.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.737+0000 7f369499a700 1 -- 172.21.15.27:0/2066188879 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f36740670e0 msgr2=0x7f3674069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:57.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.737+0000 7f369499a700 1 --2- 172.21.15.27:0/2066188879 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f36740670e0 0x7f3674069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:57.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.743+0000 7f36973ff700 1 -- 172.21.15.27:0/2066188879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3690068660 con 0x7f36900ccd70 2024-01-11T08:29:57.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.744+0000 7f36857fa700 1 -- 172.21.15.27:0/2066188879 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v132) v1 ==== 100+0+11 (secure 0 0 0) 0x7f368c056d50 con 0x7f36900ccd70 2024-01-11T08:29:57.746 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T08:29:57.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.746+0000 7f36973ff700 1 -- 172.21.15.27:0/2066188879 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f36740670e0 msgr2=0x7f3674069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:57.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.746+0000 7f36973ff700 1 --2- 172.21.15.27:0/2066188879 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f36740670e0 0x7f3674069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.746+0000 7f36973ff700 1 -- 172.21.15.27:0/2066188879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36900ccd70 msgr2=0x7f36900b3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:57.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.746+0000 7f36973ff700 1 --2- 172.21.15.27:0/2066188879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36900ccd70 0x7f36900b3380 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f368c009290 tx=0x7f368c017000 comp rx=0 tx=0).stop 2024-01-11T08:29:57.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.746+0000 7f36973ff700 1 -- 172.21.15.27:0/2066188879 shutdown_connections 2024-01-11T08:29:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.746+0000 7f36973ff700 1 --2- 172.21.15.27:0/2066188879 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f36740670e0 0x7f3674069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.746+0000 7f36973ff700 1 --2- 172.21.15.27:0/2066188879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f369015ae70 0x7f36900c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.746+0000 7f36973ff700 1 --2- 172.21.15.27:0/2066188879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36900ccd70 0x7f36900b3380 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.746+0000 7f36973ff700 1 --2- 172.21.15.27:0/2066188879 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f369015a1b0 0x7f3690155290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.746+0000 7f36973ff700 1 -- 172.21.15.27:0/2066188879 >> 172.21.15.27:0/2066188879 conn(0x7f369006ed40 msgr2=0x7f3690156110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.746+0000 7f36973ff700 1 -- 172.21.15.27:0/2066188879 shutdown_connections 2024-01-11T08:29:57.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.746+0000 7f36973ff700 1 -- 172.21.15.27:0/2066188879 wait complete. 2024-01-11T08:29:57.764 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:29:57.847 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:57.847+0000 7fad63d071c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:29:57.885 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:57.885+0000 7ff71dca21c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:29:57.943 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:57.943+0000 7fad63d071c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:29:57.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.969+0000 7f8b14fff700 1 Processor -- start 2024-01-11T08:29:57.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.970+0000 7f8b14fff700 1 -- start start 2024-01-11T08:29:57.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.971+0000 7f8b14fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b100ad020 0x7f8b100af320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:57.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.971+0000 7f8b14fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b100aa1d0 con 0x7f8b100ac370 2024-01-11T08:29:57.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.971+0000 7f8b14fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b100af860 con 0x7f8b10159890 2024-01-11T08:29:57.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.971+0000 7f8b14fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b100ac8d0 con 0x7f8b100ad020 2024-01-11T08:29:57.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.971+0000 7f8b0e59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b100ad020 0x7f8b100af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:57.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.971+0000 7f8b0dd9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8b100ac370 0x7f8b10159180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58066/0 (socket says 172.21.15.27:58066) 2024-01-11T08:29:57.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.971+0000 7f8b0dd9b700 1 -- 172.21.15.27:0/148487562 learned_addr learned my addr 172.21.15.27:0/148487562 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:57.973 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:57.973+0000 7ff71dca21c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:29:57.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.972+0000 7f8b0e59c700 1 -- 172.21.15.27:0/148487562 >> v1:172.21.15.77:6789/0 conn(0x7f8b10159890 legacy=0x7f8b100aa660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:57.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.972+0000 7f8b0e59c700 1 -- 172.21.15.27:0/148487562 >> v1:172.21.15.27:6789/0 conn(0x7f8b100ac370 legacy=0x7f8b10159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:57.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.972+0000 7f8b0e59c700 1 -- 172.21.15.27:0/148487562 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b100a98a0 con 0x7f8b100ad020 2024-01-11T08:29:57.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.972+0000 7f8b0d59a700 1 -- 172.21.15.27:0/148487562 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678174657 0 0) 0x7f8b100aa1d0 con 0x7f8b100ac370 2024-01-11T08:29:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.972+0000 7f8b0e59c700 1 --2- 172.21.15.27:0/148487562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b100ad020 0x7f8b100af320 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f8b00009910 tx=0x7f8b000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d168b51d9cf5da51 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.972+0000 7f8b0d59a700 1 -- 172.21.15.27:0/148487562 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0000b820 con 0x7f8b100ad020 2024-01-11T08:29:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.972+0000 7f8b0d59a700 1 -- 172.21.15.27:0/148487562 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0000b980 con 0x7f8b100ad020 2024-01-11T08:29:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.972+0000 7f8b0d59a700 1 -- 172.21.15.27:0/148487562 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0000bd20 con 0x7f8b100ad020 2024-01-11T08:29:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.973+0000 7f8b14fff700 1 -- 172.21.15.27:0/148487562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b100ad020 msgr2=0x7f8b100af320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.973+0000 7f8b14fff700 1 --2- 172.21.15.27:0/148487562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b100ad020 0x7f8b100af320 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f8b00009910 tx=0x7f8b000091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.973+0000 7f8b14fff700 1 -- 172.21.15.27:0/148487562 shutdown_connections 2024-01-11T08:29:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.973+0000 7f8b14fff700 1 --2- 172.21.15.27:0/148487562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b100ad020 0x7f8b100af320 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.973+0000 7f8b14fff700 1 -- 172.21.15.27:0/148487562 >> 172.21.15.27:0/148487562 conn(0x7f8b100680e0 msgr2=0x7f8b10069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.973+0000 7f8b14fff700 1 -- 172.21.15.27:0/148487562 shutdown_connections 2024-01-11T08:29:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.973+0000 7f8b14fff700 1 -- 172.21.15.27:0/148487562 wait complete. 2024-01-11T08:29:57.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.975+0000 7f8b14fff700 1 Processor -- start 2024-01-11T08:29:57.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.975+0000 7f8b14fff700 1 -- start start 2024-01-11T08:29:57.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.976+0000 7f8b14fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b100ac370 0x7f8b1006d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:57.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.976+0000 7f8b14fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b100ad020 0x7f8b1016d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:57.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.976+0000 7f8b14fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b10159890 0x7f8b10151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.976+0000 7f8b14fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1016dbb0 con 0x7f8b100ad020 2024-01-11T08:29:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.976+0000 7f8b14fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1016de90 con 0x7f8b10159890 2024-01-11T08:29:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.976+0000 7f8b14fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1015b150 con 0x7f8b100ac370 2024-01-11T08:29:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.976+0000 7f8b0dd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b100ad020 0x7f8b1016d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.976+0000 7f8b0dd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b100ad020 0x7f8b1016d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57660/0 (socket says 172.21.15.27:57660) 2024-01-11T08:29:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.976+0000 7f8b0dd9b700 1 -- 172.21.15.27:0/1177874300 learned_addr learned my addr 172.21.15.27:0/1177874300 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.976+0000 7f8b0e59c700 1 --2- 172.21.15.27:0/1177874300 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b100ac370 0x7f8b1006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.976+0000 7f8b0ed9d700 1 --2- 172.21.15.27:0/1177874300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b10159890 0x7f8b10151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.977+0000 7f8b0dd9b700 1 -- 172.21.15.27:0/1177874300 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b100ac370 msgr2=0x7f8b1006d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.977+0000 7f8b0dd9b700 1 --2- 172.21.15.27:0/1177874300 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b100ac370 0x7f8b1006d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.977+0000 7f8b0dd9b700 1 -- 172.21.15.27:0/1177874300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b10159890 msgr2=0x7f8b10151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.977+0000 7f8b0dd9b700 1 --2- 172.21.15.27:0/1177874300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b10159890 0x7f8b10151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.977+0000 7f8b0dd9b700 1 -- 172.21.15.27:0/1177874300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b00009420 con 0x7f8b100ad020 2024-01-11T08:29:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.977+0000 7f8b0e59c700 1 --2- 172.21.15.27:0/1177874300 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b100ac370 0x7f8b1006d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:57.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.977+0000 7f8b0dd9b700 1 --2- 172.21.15.27:0/1177874300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b100ad020 0x7f8b1016d160 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f8b0400c560 tx=0x7f8b04000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:57.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.977+0000 7f8afe7fc700 1 -- 172.21.15.27:0/1177874300 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b04005b30 con 0x7f8b100ad020 2024-01-11T08:29:57.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.977+0000 7f8afe7fc700 1 -- 172.21.15.27:0/1177874300 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b04005c90 con 0x7f8b100ad020 2024-01-11T08:29:57.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.977+0000 7f8afe7fc700 1 -- 172.21.15.27:0/1177874300 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b04007320 con 0x7f8b100ad020 2024-01-11T08:29:57.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.977+0000 7f8b14fff700 1 -- 172.21.15.27:0/1177874300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b1015b430 con 0x7f8b100ad020 2024-01-11T08:29:57.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.977+0000 7f8b14fff700 1 -- 172.21.15.27:0/1177874300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b1006dd80 con 0x7f8b100ad020 2024-01-11T08:29:57.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.978+0000 7f8b14fff700 1 -- 172.21.15.27:0/1177874300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b1015b650 con 0x7f8b100ad020 2024-01-11T08:29:57.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.980+0000 7f8afe7fc700 1 -- 172.21.15.27:0/1177874300 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 74) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f8b0401b3b0 con 0x7f8b100ad020 2024-01-11T08:29:57.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.981+0000 7f8afe7fc700 1 --2- 172.21.15.27:0/1177874300 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8af00670e0 0x7f8af0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:57.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.981+0000 7f8afe7fc700 1 -- 172.21.15.27:0/1177874300 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f8b04083e50 con 0x7f8b100ad020 2024-01-11T08:29:57.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.981+0000 7f8b0e59c700 1 -- 172.21.15.27:0/1177874300 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8af00670e0 msgr2=0x7f8af0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:57.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.981+0000 7f8b0e59c700 1 --2- 172.21.15.27:0/1177874300 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8af00670e0 0x7f8af0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:57.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:57.985+0000 7f8afe7fc700 1 -- 172.21.15.27:0/1177874300 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8b04050c60 con 0x7f8b100ad020 2024-01-11T08:29:58.113 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:58.113+0000 7fad63d071c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:29:58.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.182+0000 7f8b0e59c700 1 -- 172.21.15.27:0/1177874300 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8af00670e0 msgr2=0x7f8af0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:58.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.182+0000 7f8b0e59c700 1 --2- 172.21.15.27:0/1177874300 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8af00670e0 0x7f8af0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:58.192 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:58.192+0000 7ff71dca21c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:29:58.206 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:58.206+0000 7fad63d071c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:29:58.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.234+0000 7f8b14fff700 1 -- 172.21.15.27:0/1177874300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b100718d0 con 0x7f8b100ad020 2024-01-11T08:29:58.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.235+0000 7f8afe7fc700 1 -- 172.21.15.27:0/1177874300 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4596 (secure 0 0 0) 0x7f8b04017090 con 0x7f8b100ad020 2024-01-11T08:29:58.235 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:58.235 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":69,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6956,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1277153217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1277153217},{"type":"v1","addr":"172.21.15.77:6833","nonce":1277153217}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7777,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2435914089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2435914089},{"type":"v1","addr":"172.21.15.27:6835","nonce":2435914089}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7792,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/4128023892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4128023892},{"type":"v1","addr":"172.21.15.77:6837","nonce":4128023892}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:29:30.441936+0000","modified":"2024-01-11T08:29:33.957014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8058":{"gid":8058,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6834/716078331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":716078331},{"type":"v1","addr":"172.21.15.27:6834","nonce":716078331}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2024-01-11T08:29:58.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.237+0000 7f8b14fff700 1 -- 172.21.15.27:0/1177874300 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8af00670e0 msgr2=0x7f8af0069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:58.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.237+0000 7f8b14fff700 1 --2- 172.21.15.27:0/1177874300 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8af00670e0 0x7f8af0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.237+0000 7f8b14fff700 1 -- 172.21.15.27:0/1177874300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b100ad020 msgr2=0x7f8b1016d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:58.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.237+0000 7f8b14fff700 1 --2- 172.21.15.27:0/1177874300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b100ad020 0x7f8b1016d160 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f8b0400c560 tx=0x7f8b04000fe0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.237+0000 7f8b14fff700 1 -- 172.21.15.27:0/1177874300 shutdown_connections 2024-01-11T08:29:58.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.237+0000 7f8b14fff700 1 --2- 172.21.15.27:0/1177874300 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8af00670e0 0x7f8af0069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.237+0000 7f8b14fff700 1 --2- 172.21.15.27:0/1177874300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b10159890 0x7f8b10151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.237+0000 7f8b14fff700 1 --2- 172.21.15.27:0/1177874300 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b100ac370 0x7f8b1006d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.237+0000 7f8b14fff700 1 --2- 172.21.15.27:0/1177874300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b100ad020 0x7f8b1016d160 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.237+0000 7f8b14fff700 1 -- 172.21.15.27:0/1177874300 >> 172.21.15.27:0/1177874300 conn(0x7f8b100680e0 msgr2=0x7f8b10153a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:58.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.238+0000 7f8b14fff700 1 -- 172.21.15.27:0/1177874300 shutdown_connections 2024-01-11T08:29:58.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.238+0000 7f8b14fff700 1 -- 172.21.15.27:0/1177874300 wait complete. 2024-01-11T08:29:58.244 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 69 2024-01-11T08:29:58.257 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:29:58.288 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:58.288+0000 7ff71dca21c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:29:58.383 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:58.384+0000 7ff71dca21c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:29:58.443 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:58.443+0000 7fad63d071c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:29:58.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.468+0000 7f952ffff700 1 Processor -- start 2024-01-11T08:29:58.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.468+0000 7f952ffff700 1 -- start start 2024-01-11T08:29:58.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.470+0000 7f952ffff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f95281599e0 0x7f95280b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:58.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.470+0000 7f952ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9528159db0 0x7f952815a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:58.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.470+0000 7f952ffff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9528157e30 con 0x7f952814fc20 2024-01-11T08:29:58.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.470+0000 7f952ffff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9528157500 con 0x7f9528159db0 2024-01-11T08:29:58.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.470+0000 7f952ffff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95281577e0 con 0x7f95281599e0 2024-01-11T08:29:58.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.471+0000 7f952dd9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f95281599e0 0x7f95280b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:58.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.471+0000 7f952e59c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f952814fc20 0x7f9528150080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:50854/0 (socket says 172.21.15.27:50854) 2024-01-11T08:29:58.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.471+0000 7f952e59c700 1 -- 172.21.15.27:0/4118641460 learned_addr learned my addr 172.21.15.27:0/4118641460 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:58.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.471+0000 7f952d59a700 1 --2- 172.21.15.27:0/4118641460 >> v2:172.21.15.77:3300/0 conn(0x7f9528159db0 0x7f952815a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:58.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.471+0000 7f952cd99700 1 -- 172.21.15.27:0/4118641460 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521449313 0 0) 0x7f9528157e30 con 0x7f952814fc20 2024-01-11T08:29:58.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.471+0000 7f952cd99700 1 -- 172.21.15.27:0/4118641460 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9514003570 con 0x7f952814fc20 2024-01-11T08:29:58.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.471+0000 7f952dd9b700 1 -- 172.21.15.27:0/4118641460 >> v2:172.21.15.77:3300/0 conn(0x7f9528159db0 msgr2=0x7f952815a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:58.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.471+0000 7f952dd9b700 1 --2- 172.21.15.27:0/4118641460 >> v2:172.21.15.77:3300/0 conn(0x7f9528159db0 0x7f952815a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.471+0000 7f952dd9b700 1 -- 172.21.15.27:0/4118641460 >> v1:172.21.15.77:6789/0 conn(0x7f952814fc20 legacy=0x7f9528150080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:58.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.471+0000 7f952dd9b700 1 -- 172.21.15.27:0/4118641460 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9528157ac0 con 0x7f95281599e0 2024-01-11T08:29:58.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.471+0000 7f952d59a700 1 --2- 172.21.15.27:0/4118641460 >> v2:172.21.15.77:3300/0 conn(0x7f9528159db0 0x7f952815a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:58.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.472+0000 7f952dd9b700 1 --2- 172.21.15.27:0/4118641460 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f95281599e0 0x7f95280b0130 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f95240098e0 tx=0x7f95240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf43835555bb8898 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:58.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.472+0000 7f952cd99700 1 -- 172.21.15.27:0/4118641460 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524011630 con 0x7f95281599e0 2024-01-11T08:29:58.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.472+0000 7f952cd99700 1 -- 172.21.15.27:0/4118641460 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9524011790 con 0x7f95281599e0 2024-01-11T08:29:58.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.472+0000 7f952cd99700 1 -- 172.21.15.27:0/4118641460 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524011b30 con 0x7f95281599e0 2024-01-11T08:29:58.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.473+0000 7f952ffff700 1 -- 172.21.15.27:0/4118641460 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f95281599e0 msgr2=0x7f95280b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:58.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.473+0000 7f952ffff700 1 --2- 172.21.15.27:0/4118641460 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f95281599e0 0x7f95280b0130 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f95240098e0 tx=0x7f95240091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.473+0000 7f952ffff700 1 -- 172.21.15.27:0/4118641460 shutdown_connections 2024-01-11T08:29:58.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.473+0000 7f952ffff700 1 --2- 172.21.15.27:0/4118641460 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f95281599e0 0x7f95280b0130 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.473+0000 7f952ffff700 1 --2- 172.21.15.27:0/4118641460 >> v2:172.21.15.77:3300/0 conn(0x7f9528159db0 0x7f952815a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.473+0000 7f952ffff700 1 -- 172.21.15.27:0/4118641460 >> 172.21.15.27:0/4118641460 conn(0x7f95280680e0 msgr2=0x7f9528069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:58.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.473+0000 7f952ffff700 1 -- 172.21.15.27:0/4118641460 shutdown_connections 2024-01-11T08:29:58.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.473+0000 7f952ffff700 1 -- 172.21.15.27:0/4118641460 wait complete. 2024-01-11T08:29:58.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.475+0000 7f952ffff700 1 Processor -- start 2024-01-11T08:29:58.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.476+0000 7f952ffff700 1 -- start start 2024-01-11T08:29:58.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.477+0000 7f952ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f952814fc20 0x7f95281498c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:58.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.477+0000 7f952ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95281599e0 0x7f95280c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:58.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.477+0000 7f952ffff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9528159db0 0x7f95280ca360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:58.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.477+0000 7f952ffff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952814a4c0 con 0x7f95281599e0 2024-01-11T08:29:58.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.477+0000 7f952ffff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9528145910 con 0x7f952814fc20 2024-01-11T08:29:58.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.477+0000 7f952ffff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9528145270 con 0x7f9528159db0 2024-01-11T08:29:58.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.477+0000 7f952d59a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95281599e0 0x7f95280c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:58.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.477+0000 7f952d59a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95281599e0 0x7f95280c9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57672/0 (socket says 172.21.15.27:57672) 2024-01-11T08:29:58.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.477+0000 7f952d59a700 1 -- 172.21.15.27:0/214500985 learned_addr learned my addr 172.21.15.27:0/214500985 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:58.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.477+0000 7f952e59c700 1 --2- 172.21.15.27:0/214500985 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9528159db0 0x7f95280ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:58.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.477+0000 7f952dd9b700 1 --2- 172.21.15.27:0/214500985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f952814fc20 0x7f95281498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:58.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.478+0000 7f952d59a700 1 -- 172.21.15.27:0/214500985 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9528159db0 msgr2=0x7f95280ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:58.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.478+0000 7f952d59a700 1 --2- 172.21.15.27:0/214500985 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9528159db0 0x7f95280ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.478+0000 7f952d59a700 1 -- 172.21.15.27:0/214500985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f952814fc20 msgr2=0x7f95281498c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:58.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.478+0000 7f952d59a700 1 --2- 172.21.15.27:0/214500985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f952814fc20 0x7f95281498c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.478+0000 7f952d59a700 1 -- 172.21.15.27:0/214500985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951800a5d0 con 0x7f95281599e0 2024-01-11T08:29:58.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.478+0000 7f952e59c700 1 --2- 172.21.15.27:0/214500985 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9528159db0 0x7f95280ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.478+0000 7f952d59a700 1 --2- 172.21.15.27:0/214500985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95281599e0 0x7f95280c9e20 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f951800ea40 tx=0x7f951800a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.478+0000 7f951dffb700 1 -- 172.21.15.27:0/214500985 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951800ee80 con 0x7f95281599e0 2024-01-11T08:29:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.478+0000 7f951dffb700 1 -- 172.21.15.27:0/214500985 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951800c950 con 0x7f95281599e0 2024-01-11T08:29:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.479+0000 7f952ffff700 1 -- 172.21.15.27:0/214500985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9524009420 con 0x7f95281599e0 2024-01-11T08:29:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.479+0000 7f951dffb700 1 -- 172.21.15.27:0/214500985 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951800ccd0 con 0x7f95281599e0 2024-01-11T08:29:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.479+0000 7f952ffff700 1 -- 172.21.15.27:0/214500985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f952814a260 con 0x7f95281599e0 2024-01-11T08:29:58.480 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:58.479+0000 7ff71dca21c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:29:58.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.480+0000 7f952ffff700 1 -- 172.21.15.27:0/214500985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9528145a80 con 0x7f95281599e0 2024-01-11T08:29:58.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.481+0000 7f951dffb700 1 -- 172.21.15.27:0/214500985 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 74) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f95180113f0 con 0x7f95281599e0 2024-01-11T08:29:58.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.482+0000 7f951dffb700 1 --2- 172.21.15.27:0/214500985 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f950c0670e0 0x7f950c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.482+0000 7f951dffb700 1 -- 172.21.15.27:0/214500985 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f9518015030 con 0x7f95281599e0 2024-01-11T08:29:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.482+0000 7f952dd9b700 1 -- 172.21.15.27:0/214500985 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f950c0670e0 msgr2=0x7f950c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.482+0000 7f952dd9b700 1 --2- 172.21.15.27:0/214500985 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f950c0670e0 0x7f950c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:58.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.486+0000 7f951dffb700 1 -- 172.21.15.27:0/214500985 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9518051520 con 0x7f95281599e0 2024-01-11T08:29:58.550 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:58.550+0000 7fad63d071c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:29:58.651 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:58.652+0000 7fad63d071c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:29:58.667 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:58.668+0000 7ff71dca21c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:29:58.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.683+0000 7f952dd9b700 1 -- 172.21.15.27:0/214500985 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f950c0670e0 msgr2=0x7f950c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:58.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.683+0000 7f952dd9b700 1 --2- 172.21.15.27:0/214500985 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f950c0670e0 0x7f950c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:58.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.690+0000 7f952ffff700 1 -- 172.21.15.27:0/214500985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95280718d0 con 0x7f95281599e0 2024-01-11T08:29:58.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.691+0000 7f951dffb700 1 -- 172.21.15.27:0/214500985 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+15770 (secure 0 0 0) 0x7f951805a030 con 0x7f95281599e0 2024-01-11T08:29:58.691 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:58.692 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":132,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:29:46.941384+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:29:28.131181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-11T08:29:29.438397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/259960245":"2024-01-12T08:29:46.941337+0000","172.21.15.77:6836/41381":"2024-01-12T08:29:46.941337+0000","172.21.15.77:6835/41381":"2024-01-12T08:29:46.941337+0000","172.21.15.77:0/3475507657":"2024-01-12T08:29:46.941337+0000","172.21.15.77:0/3338577024":"2024-01-12T08:29:46.941337+0000","172.21.15.77:0/3974991130":"2024-01-12T08:29:46.941337+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:29:58.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.694+0000 7f952ffff700 1 -- 172.21.15.27:0/214500985 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f950c0670e0 msgr2=0x7f950c069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:58.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.694+0000 7f952ffff700 1 --2- 172.21.15.27:0/214500985 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f950c0670e0 0x7f950c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.694+0000 7f952ffff700 1 -- 172.21.15.27:0/214500985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95281599e0 msgr2=0x7f95280c9e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:58.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.694+0000 7f952ffff700 1 --2- 172.21.15.27:0/214500985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95281599e0 0x7f95280c9e20 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f951800ea40 tx=0x7f951800a330 comp rx=0 tx=0).stop 2024-01-11T08:29:58.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.694+0000 7f952ffff700 1 -- 172.21.15.27:0/214500985 shutdown_connections 2024-01-11T08:29:58.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.694+0000 7f952ffff700 1 --2- 172.21.15.27:0/214500985 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f950c0670e0 0x7f950c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.694+0000 7f952ffff700 1 --2- 172.21.15.27:0/214500985 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9528159db0 0x7f95280ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.694+0000 7f952ffff700 1 --2- 172.21.15.27:0/214500985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f952814fc20 0x7f95281498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.694+0000 7f952ffff700 1 --2- 172.21.15.27:0/214500985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95281599e0 0x7f95280c9e20 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.694+0000 7f952ffff700 1 -- 172.21.15.27:0/214500985 >> 172.21.15.27:0/214500985 conn(0x7f95280680e0 msgr2=0x7f952806d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:58.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.694+0000 7f952ffff700 1 -- 172.21.15.27:0/214500985 shutdown_connections 2024-01-11T08:29:58.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.694+0000 7f952ffff700 1 -- 172.21.15.27:0/214500985 wait complete. 2024-01-11T08:29:58.712 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-11T08:29:58.713 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:29:58.755 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:58.755+0000 7fad63d071c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:29:58.764 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:58.764+0000 7ff71dca21c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:29:58.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.915+0000 7f5f2bbff700 1 Processor -- start 2024-01-11T08:29:58.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.915+0000 7f5f2bbff700 1 -- start start 2024-01-11T08:29:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.916+0000 7f5f2bbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f24069d40 0x7f5f240b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.917+0000 7f5f2bbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f240c89f0 0x7f5f240b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.917+0000 7f5f2bbff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f24157390 con 0x7f5f240c8620 2024-01-11T08:29:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.917+0000 7f5f2bbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f240adcd0 con 0x7f5f24069d40 2024-01-11T08:29:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.917+0000 7f5f2bbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2406a200 con 0x7f5f240c89f0 2024-01-11T08:29:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.917+0000 7f5f2a19c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f240c89f0 0x7f5f240b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.917+0000 7f5f2919a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f24069d40 0x7f5f240b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.917+0000 7f5f2999b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5f240c8620 0x7f5f240af970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:50870/0 (socket says 172.21.15.27:50870) 2024-01-11T08:29:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.917+0000 7f5f2999b700 1 -- 172.21.15.27:0/3104399004 learned_addr learned my addr 172.21.15.27:0/3104399004 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.918+0000 7f5f2919a700 1 -- 172.21.15.27:0/3104399004 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f240c89f0 msgr2=0x7f5f240b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.917+0000 7f5f28999700 1 -- 172.21.15.27:0/3104399004 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644937160 0 0) 0x7f5f24157390 con 0x7f5f240c8620 2024-01-11T08:29:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.918+0000 7f5f2919a700 1 --2- 172.21.15.27:0/3104399004 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f240c89f0 0x7f5f240b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.918+0000 7f5f2919a700 1 -- 172.21.15.27:0/3104399004 >> v1:172.21.15.77:6789/0 conn(0x7f5f240c8620 legacy=0x7f5f240af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.918+0000 7f5f2919a700 1 -- 172.21.15.27:0/3104399004 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f24156d40 con 0x7f5f24069d40 2024-01-11T08:29:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.918+0000 7f5f2a19c700 1 --2- 172.21.15.27:0/3104399004 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f240c89f0 0x7f5f240b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.918+0000 7f5f2919a700 1 --2- 172.21.15.27:0/3104399004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f24069d40 0x7f5f240b02d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5f140098e0 tx=0x7f5f140091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6e13f78a099cb1c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.918+0000 7f5f28999700 1 -- 172.21.15.27:0/3104399004 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1400b640 con 0x7f5f24069d40 2024-01-11T08:29:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.918+0000 7f5f28999700 1 -- 172.21.15.27:0/3104399004 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1400b7a0 con 0x7f5f24069d40 2024-01-11T08:29:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.918+0000 7f5f28999700 1 -- 172.21.15.27:0/3104399004 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1400bb40 con 0x7f5f24069d40 2024-01-11T08:29:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.919+0000 7f5f2bbff700 1 -- 172.21.15.27:0/3104399004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f24069d40 msgr2=0x7f5f240b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.919+0000 7f5f2bbff700 1 --2- 172.21.15.27:0/3104399004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f24069d40 0x7f5f240b02d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5f140098e0 tx=0x7f5f140091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.919+0000 7f5f2bbff700 1 -- 172.21.15.27:0/3104399004 shutdown_connections 2024-01-11T08:29:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.919+0000 7f5f2bbff700 1 --2- 172.21.15.27:0/3104399004 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f240c89f0 0x7f5f240b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.919+0000 7f5f2bbff700 1 --2- 172.21.15.27:0/3104399004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f24069d40 0x7f5f240b02d0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.919+0000 7f5f2bbff700 1 -- 172.21.15.27:0/3104399004 >> 172.21.15.27:0/3104399004 conn(0x7f5f240680e0 msgr2=0x7f5f24069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.919+0000 7f5f2bbff700 1 -- 172.21.15.27:0/3104399004 shutdown_connections 2024-01-11T08:29:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.919+0000 7f5f2bbff700 1 -- 172.21.15.27:0/3104399004 wait complete. 2024-01-11T08:29:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.920+0000 7f5f2bbff700 1 Processor -- start 2024-01-11T08:29:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.921+0000 7f5f2bbff700 1 -- start start 2024-01-11T08:29:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.922+0000 7f5f2bbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f24069d40 0x7f5f240c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.922+0000 7f5f2bbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f240c8620 0x7f5f2416c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.922+0000 7f5f2bbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f240c89f0 0x7f5f24169780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.922+0000 7f5f2bbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f240c58e0 con 0x7f5f240c89f0 2024-01-11T08:29:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.922+0000 7f5f2bbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2416cab0 con 0x7f5f24069d40 2024-01-11T08:29:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.922+0000 7f5f2bbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f240c5b80 con 0x7f5f240c8620 2024-01-11T08:29:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.922+0000 7f5f2a19c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f240c89f0 0x7f5f24169780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.922+0000 7f5f2a19c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f240c89f0 0x7f5f24169780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57686/0 (socket says 172.21.15.27:57686) 2024-01-11T08:29:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.922+0000 7f5f2a19c700 1 -- 172.21.15.27:0/1141676434 learned_addr learned my addr 172.21.15.27:0/1141676434 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.922+0000 7f5f2999b700 1 --2- 172.21.15.27:0/1141676434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f24069d40 0x7f5f240c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.922+0000 7f5f2919a700 1 --2- 172.21.15.27:0/1141676434 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f240c8620 0x7f5f2416c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.922+0000 7f5f2a19c700 1 -- 172.21.15.27:0/1141676434 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f240c8620 msgr2=0x7f5f2416c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.922+0000 7f5f2a19c700 1 --2- 172.21.15.27:0/1141676434 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f240c8620 0x7f5f2416c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.922+0000 7f5f2a19c700 1 -- 172.21.15.27:0/1141676434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f24069d40 msgr2=0x7f5f240c53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.922+0000 7f5f2a19c700 1 --2- 172.21.15.27:0/1141676434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f24069d40 0x7f5f240c53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.923+0000 7f5f2a19c700 1 -- 172.21.15.27:0/1141676434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c009a20 con 0x7f5f240c89f0 2024-01-11T08:29:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.923+0000 7f5f2a19c700 1 --2- 172.21.15.27:0/1141676434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f240c89f0 0x7f5f24169780 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c009f50 tx=0x7f5f1c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.923+0000 7f5f19ffb700 1 -- 172.21.15.27:0/1141676434 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c00caf0 con 0x7f5f240c89f0 2024-01-11T08:29:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.923+0000 7f5f19ffb700 1 -- 172.21.15.27:0/1141676434 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1c00f040 con 0x7f5f240c89f0 2024-01-11T08:29:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.923+0000 7f5f2bbff700 1 -- 172.21.15.27:0/1141676434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f14009420 con 0x7f5f240c89f0 2024-01-11T08:29:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.923+0000 7f5f19ffb700 1 -- 172.21.15.27:0/1141676434 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c00ce80 con 0x7f5f240c89f0 2024-01-11T08:29:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.923+0000 7f5f2bbff700 1 -- 172.21.15.27:0/1141676434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f240c6140 con 0x7f5f240c89f0 2024-01-11T08:29:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.924+0000 7f5f2bbff700 1 -- 172.21.15.27:0/1141676434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f240718d0 con 0x7f5f240c89f0 2024-01-11T08:29:58.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.925+0000 7f5f19ffb700 1 -- 172.21.15.27:0/1141676434 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 74) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f5f1c014690 con 0x7f5f240c89f0 2024-01-11T08:29:58.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.926+0000 7f5f19ffb700 1 --2- 172.21.15.27:0/1141676434 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f080670e0 0x7f5f08069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:58.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.926+0000 7f5f19ffb700 1 -- 172.21.15.27:0/1141676434 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f5f1c085750 con 0x7f5f240c89f0 2024-01-11T08:29:58.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.926+0000 7f5f2999b700 1 -- 172.21.15.27:0/1141676434 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f080670e0 msgr2=0x7f5f08069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:58.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.926+0000 7f5f2999b700 1 --2- 172.21.15.27:0/1141676434 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f080670e0 0x7f5f08069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:58.930+0000 7f5f19ffb700 1 -- 172.21.15.27:0/1141676434 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5f1c055f80 con 0x7f5f240c89f0 2024-01-11T08:29:58.962 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:58.962+0000 7fad63d071c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:29:59.068 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:59.068+0000 7fad63d071c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:29:59.107 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:59.107+0000 7ff71dca21c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:29:59.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.127+0000 7f5f2999b700 1 -- 172.21.15.27:0/1141676434 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f080670e0 msgr2=0x7f5f08069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:59.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.127+0000 7f5f2999b700 1 --2- 172.21.15.27:0/1141676434 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f080670e0 0x7f5f08069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:59.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.178+0000 7f5f2bbff700 1 -- 172.21.15.27:0/1141676434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f24157020 con 0x7f5f240c89f0 2024-01-11T08:29:59.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.178+0000 7f5f19ffb700 1 -- 172.21.15.27:0/1141676434 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4596 (secure 0 0 0) 0x7f5f1c055b10 con 0x7f5f240c89f0 2024-01-11T08:29:59.178 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:59.179 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":69,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6956,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1277153217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1277153217},{"type":"v1","addr":"172.21.15.77:6833","nonce":1277153217}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7777,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2435914089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2435914089},{"type":"v1","addr":"172.21.15.27:6835","nonce":2435914089}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7792,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/4128023892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4128023892},{"type":"v1","addr":"172.21.15.77:6837","nonce":4128023892}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:29:30.441936+0000","modified":"2024-01-11T08:29:33.957014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8058":{"gid":8058,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6834/716078331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":716078331},{"type":"v1","addr":"172.21.15.27:6834","nonce":716078331}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2024-01-11T08:29:59.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.181+0000 7f5f2bbff700 1 -- 172.21.15.27:0/1141676434 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f080670e0 msgr2=0x7f5f08069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:59.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.181+0000 7f5f2bbff700 1 --2- 172.21.15.27:0/1141676434 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f080670e0 0x7f5f08069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.181+0000 7f5f2bbff700 1 -- 172.21.15.27:0/1141676434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f240c89f0 msgr2=0x7f5f24169780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:59.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.181+0000 7f5f2bbff700 1 --2- 172.21.15.27:0/1141676434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f240c89f0 0x7f5f24169780 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c009f50 tx=0x7f5f1c00b040 comp rx=0 tx=0).stop 2024-01-11T08:29:59.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.181+0000 7f5f2bbff700 1 -- 172.21.15.27:0/1141676434 shutdown_connections 2024-01-11T08:29:59.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.181+0000 7f5f2bbff700 1 --2- 172.21.15.27:0/1141676434 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5f080670e0 0x7f5f08069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.181+0000 7f5f2bbff700 1 --2- 172.21.15.27:0/1141676434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f240c89f0 0x7f5f24169780 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.181+0000 7f5f2bbff700 1 --2- 172.21.15.27:0/1141676434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f24069d40 0x7f5f240c53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.181+0000 7f5f2bbff700 1 --2- 172.21.15.27:0/1141676434 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5f240c8620 0x7f5f2416c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.181+0000 7f5f2bbff700 1 -- 172.21.15.27:0/1141676434 >> 172.21.15.27:0/1141676434 conn(0x7f5f240680e0 msgr2=0x7f5f241555f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:59.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.181+0000 7f5f2bbff700 1 -- 172.21.15.27:0/1141676434 shutdown_connections 2024-01-11T08:29:59.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.181+0000 7f5f2bbff700 1 -- 172.21.15.27:0/1141676434 wait complete. 2024-01-11T08:29:59.182 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 69 2024-01-11T08:29:59.195 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:59.196+0000 7ff71dca21c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:29:59.199 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:29:59.435 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:59.435+0000 7fad63d071c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:29:59.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.445+0000 7f527dfff700 1 Processor -- start 2024-01-11T08:29:59.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.445+0000 7f527dfff700 1 -- start start 2024-01-11T08:29:59.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.447+0000 7f527dfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52781575e0 0x7f5278158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:59.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.447+0000 7f527dfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f527806b610 0x7f52780b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:59.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.447+0000 7f527dfff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5278158b60 con 0x7f52780b1800 2024-01-11T08:29:59.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.447+0000 7f527dfff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52780cb5e0 con 0x7f527806b610 2024-01-11T08:29:59.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.447+0000 7f527dfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52780b1c80 con 0x7f52781575e0 2024-01-11T08:29:59.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.447+0000 7f5277fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f52780b1800 0x7f52780ca1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58082/0 (socket says 172.21.15.27:58082) 2024-01-11T08:29:59.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.447+0000 7f5277fff700 1 -- 172.21.15.27:0/2186298337 learned_addr learned my addr 172.21.15.27:0/2186298337 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:29:59.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.447+0000 7f52777fe700 1 --2- 172.21.15.27:0/2186298337 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52781575e0 0x7f5278158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:59.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.447+0000 7f5276ffd700 1 --2- 172.21.15.27:0/2186298337 >> v2:172.21.15.77:3300/0 conn(0x7f527806b610 0x7f52780b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:59.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.447+0000 7f52767fc700 1 -- 172.21.15.27:0/2186298337 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138301957 0 0) 0x7f5270001490 con 0x7f52780b1800 2024-01-11T08:29:59.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.448+0000 7f52767fc700 1 -- 172.21.15.27:0/2186298337 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5260003570 con 0x7f52780b1800 2024-01-11T08:29:59.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.448+0000 7f5276ffd700 1 -- 172.21.15.27:0/2186298337 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52781575e0 msgr2=0x7f5278158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:59.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.448+0000 7f5276ffd700 1 --2- 172.21.15.27:0/2186298337 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52781575e0 0x7f5278158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.448+0000 7f5276ffd700 1 -- 172.21.15.27:0/2186298337 >> v1:172.21.15.27:6789/0 conn(0x7f52780b1800 legacy=0x7f52780ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:59.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.448+0000 7f5276ffd700 1 -- 172.21.15.27:0/2186298337 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52781599b0 con 0x7f527806b610 2024-01-11T08:29:59.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.448+0000 7f52777fe700 1 --2- 172.21.15.27:0/2186298337 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52781575e0 0x7f5278158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:59.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.448+0000 7f5276ffd700 1 --2- 172.21.15.27:0/2186298337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f527806b610 0x7f52780b12c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f526c0098e0 tx=0x7f526c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2eeaca3bb584614 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:59.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.448+0000 7f52767fc700 1 -- 172.21.15.27:0/2186298337 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c011630 con 0x7f527806b610 2024-01-11T08:29:59.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.448+0000 7f52767fc700 1 -- 172.21.15.27:0/2186298337 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526c011790 con 0x7f527806b610 2024-01-11T08:29:59.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.449+0000 7f52767fc700 1 -- 172.21.15.27:0/2186298337 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c011b30 con 0x7f527806b610 2024-01-11T08:29:59.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.449+0000 7f527dfff700 1 -- 172.21.15.27:0/2186298337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f527806b610 msgr2=0x7f52780b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:59.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.449+0000 7f527dfff700 1 --2- 172.21.15.27:0/2186298337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f527806b610 0x7f52780b12c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f526c0098e0 tx=0x7f526c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.449+0000 7f527dfff700 1 -- 172.21.15.27:0/2186298337 shutdown_connections 2024-01-11T08:29:59.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.449+0000 7f527dfff700 1 --2- 172.21.15.27:0/2186298337 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52781575e0 0x7f5278158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.449+0000 7f527dfff700 1 --2- 172.21.15.27:0/2186298337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f527806b610 0x7f52780b12c0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.449+0000 7f527dfff700 1 -- 172.21.15.27:0/2186298337 >> 172.21.15.27:0/2186298337 conn(0x7f52780680e0 msgr2=0x7f5278069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:59.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.449+0000 7f527dfff700 1 -- 172.21.15.27:0/2186298337 shutdown_connections 2024-01-11T08:29:59.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.449+0000 7f527dfff700 1 -- 172.21.15.27:0/2186298337 wait complete. 2024-01-11T08:29:59.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.451+0000 7f527dfff700 1 Processor -- start 2024-01-11T08:29:59.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.451+0000 7f527dfff700 1 -- start start 2024-01-11T08:29:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.452+0000 7f527dfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f527806b610 0x7f5278144c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.452+0000 7f527dfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52780b1800 0x7f52780b0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.452+0000 7f527dfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52781575e0 0x7f5278154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.452+0000 7f527dfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52781524a0 con 0x7f527806b610 2024-01-11T08:29:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.452+0000 7f527dfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52780af490 con 0x7f52780b1800 2024-01-11T08:29:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.452+0000 7f527dfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52780af770 con 0x7f52781575e0 2024-01-11T08:29:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.452+0000 7f52777fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f527806b610 0x7f5278144c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.452+0000 7f52777fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f527806b610 0x7f5278144c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57694/0 (socket says 172.21.15.27:57694) 2024-01-11T08:29:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.452+0000 7f52777fe700 1 -- 172.21.15.27:0/1690707681 learned_addr learned my addr 172.21.15.27:0/1690707681 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.452+0000 7f5276ffd700 1 --2- 172.21.15.27:0/1690707681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52780b1800 0x7f52780b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.452+0000 7f5277fff700 1 --2- 172.21.15.27:0/1690707681 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52781575e0 0x7f5278154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.453+0000 7f52777fe700 1 -- 172.21.15.27:0/1690707681 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52781575e0 msgr2=0x7f5278154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:59.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.453+0000 7f52777fe700 1 --2- 172.21.15.27:0/1690707681 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52781575e0 0x7f5278154680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.453+0000 7f52777fe700 1 -- 172.21.15.27:0/1690707681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52780b1800 msgr2=0x7f52780b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:59.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.453+0000 7f52777fe700 1 --2- 172.21.15.27:0/1690707681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52780b1800 0x7f52780b0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.453+0000 7f52777fe700 1 -- 172.21.15.27:0/1690707681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5268009420 con 0x7f527806b610 2024-01-11T08:29:59.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.453+0000 7f5276ffd700 1 --2- 172.21.15.27:0/1690707681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52780b1800 0x7f52780b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:29:59.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.453+0000 7f52777fe700 1 --2- 172.21.15.27:0/1690707681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f527806b610 0x7f5278144c30 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f526800ec80 tx=0x7f52680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:59.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.453+0000 7f525f7fe700 1 -- 172.21.15.27:0/1690707681 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526800c850 con 0x7f527806b610 2024-01-11T08:29:59.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.453+0000 7f525f7fe700 1 -- 172.21.15.27:0/1690707681 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526800c9b0 con 0x7f527806b610 2024-01-11T08:29:59.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.453+0000 7f527dfff700 1 -- 172.21.15.27:0/1690707681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f526c009420 con 0x7f527806b610 2024-01-11T08:29:59.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.453+0000 7f525f7fe700 1 -- 172.21.15.27:0/1690707681 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526800cd30 con 0x7f527806b610 2024-01-11T08:29:59.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.454+0000 7f527dfff700 1 -- 172.21.15.27:0/1690707681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52780afd30 con 0x7f527806b610 2024-01-11T08:29:59.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.455+0000 7f527dfff700 1 -- 172.21.15.27:0/1690707681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52780718d0 con 0x7f527806b610 2024-01-11T08:29:59.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.456+0000 7f525f7fe700 1 -- 172.21.15.27:0/1690707681 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 74) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f5268013680 con 0x7f527806b610 2024-01-11T08:29:59.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.457+0000 7f525f7fe700 1 --2- 172.21.15.27:0/1690707681 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f52640670e0 0x7f5264069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:59.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.457+0000 7f525f7fe700 1 -- 172.21.15.27:0/1690707681 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f5268084c70 con 0x7f527806b610 2024-01-11T08:29:59.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.458+0000 7f5276ffd700 1 -- 172.21.15.27:0/1690707681 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f52640670e0 msgr2=0x7f5264069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:59.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.458+0000 7f5276ffd700 1 --2- 172.21.15.27:0/1690707681 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f52640670e0 0x7f5264069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:59.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.462+0000 7f525f7fe700 1 -- 172.21.15.27:0/1690707681 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5268051b20 con 0x7f527806b610 2024-01-11T08:29:59.535 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:29:59.535+0000 7fad63d071c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:29:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.659+0000 7f5276ffd700 1 -- 172.21.15.27:0/1690707681 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f52640670e0 msgr2=0x7f5264069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.659+0000 7f5276ffd700 1 --2- 172.21.15.27:0/1690707681 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f52640670e0 0x7f5264069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:29:59.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.712+0000 7f527dfff700 1 -- 172.21.15.27:0/1690707681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52781558e0 con 0x7f527806b610 2024-01-11T08:29:59.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.713+0000 7f525f7fe700 1 -- 172.21.15.27:0/1690707681 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4596 (secure 0 0 0) 0x7f5268017090 con 0x7f527806b610 2024-01-11T08:29:59.713 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:29:59.713 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":69,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6956,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1277153217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1277153217},{"type":"v1","addr":"172.21.15.77:6833","nonce":1277153217}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7777,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2435914089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2435914089},{"type":"v1","addr":"172.21.15.27:6835","nonce":2435914089}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7792,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/4128023892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4128023892},{"type":"v1","addr":"172.21.15.77:6837","nonce":4128023892}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:29:30.441936+0000","modified":"2024-01-11T08:29:33.957014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8058},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8058":{"gid":8058,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6834/716078331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":716078331},{"type":"v1","addr":"172.21.15.27:6834","nonce":716078331}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2024-01-11T08:29:59.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.715+0000 7f527dfff700 1 -- 172.21.15.27:0/1690707681 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f52640670e0 msgr2=0x7f5264069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:29:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.715+0000 7f527dfff700 1 --2- 172.21.15.27:0/1690707681 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f52640670e0 0x7f5264069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.715+0000 7f527dfff700 1 -- 172.21.15.27:0/1690707681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f527806b610 msgr2=0x7f5278144c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.715+0000 7f527dfff700 1 --2- 172.21.15.27:0/1690707681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f527806b610 0x7f5278144c30 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f526800ec80 tx=0x7f52680091a0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.715+0000 7f527dfff700 1 -- 172.21.15.27:0/1690707681 shutdown_connections 2024-01-11T08:29:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.715+0000 7f527dfff700 1 --2- 172.21.15.27:0/1690707681 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f52640670e0 0x7f5264069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.715+0000 7f527dfff700 1 --2- 172.21.15.27:0/1690707681 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52781575e0 0x7f5278154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.715+0000 7f527dfff700 1 --2- 172.21.15.27:0/1690707681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f527806b610 0x7f5278144c30 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.715+0000 7f527dfff700 1 --2- 172.21.15.27:0/1690707681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52780b1800 0x7f52780b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.715+0000 7f527dfff700 1 -- 172.21.15.27:0/1690707681 >> 172.21.15.27:0/1690707681 conn(0x7f52780680e0 msgr2=0x7f5278155b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.715+0000 7f527dfff700 1 -- 172.21.15.27:0/1690707681 shutdown_connections 2024-01-11T08:29:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.715+0000 7f527dfff700 1 -- 172.21.15.27:0/1690707681 wait complete. 2024-01-11T08:29:59.717 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 69 2024-01-11T08:29:59.733 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:59.734+0000 7ff71dca21c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:29:59.734 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:29:59.920 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:29:59.921+0000 7ff71dca21c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:29:59.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.942+0000 7fe085fff700 1 Processor -- start 2024-01-11T08:29:59.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.942+0000 7fe085fff700 1 -- start start 2024-01-11T08:29:59.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.943+0000 7fe085fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0801575e0 0x7fe080158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:59.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.943+0000 7fe085fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe08006b610 0x7fe0800b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:59.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.943+0000 7fe085fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe0800b1800 0x7fe0800ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:59.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.943+0000 7fe085fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0800cb5e0 con 0x7fe0800b1800 2024-01-11T08:29:59.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.943+0000 7fe085fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0800b1c80 con 0x7fe08006b610 2024-01-11T08:29:59.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.943+0000 7fe085fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe080158b60 con 0x7fe0801575e0 2024-01-11T08:29:59.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.944+0000 7fe07ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe0800b1800 0x7fe0800ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:59.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.944+0000 7fe07effd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe08006b610 0x7fe0800b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:59.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.944+0000 7fe07f7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0801575e0 0x7fe080158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:59.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.944+0000 7fe07ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe0800b1800 0x7fe0800ca1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57696/0 (socket says 172.21.15.27:57696) 2024-01-11T08:29:59.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.944+0000 7fe07ffff700 1 -- 172.21.15.27:0/1889551162 learned_addr learned my addr 172.21.15.27:0/1889551162 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:59.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.944+0000 7fe07ffff700 1 -- 172.21.15.27:0/1889551162 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0801575e0 msgr2=0x7fe080158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:59.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.944+0000 7fe07ffff700 1 --2- 172.21.15.27:0/1889551162 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0801575e0 0x7fe080158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.944+0000 7fe07ffff700 1 -- 172.21.15.27:0/1889551162 >> v2:172.21.15.77:3300/0 conn(0x7fe08006b610 msgr2=0x7fe0800b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:29:59.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.944+0000 7fe07ffff700 1 --2- 172.21.15.27:0/1889551162 >> v2:172.21.15.77:3300/0 conn(0x7fe08006b610 0x7fe0800b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.944+0000 7fe07ffff700 1 -- 172.21.15.27:0/1889551162 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0801599b0 con 0x7fe0800b1800 2024-01-11T08:29:59.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.944+0000 7fe07ffff700 1 --2- 172.21.15.27:0/1889551162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0800b1800 0x7fe0800ca1a0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fe0800694d0 tx=0x7fe078009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=285872146bf2241c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:59.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.945+0000 7fe07e7fc700 1 -- 172.21.15.27:0/1889551162 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07800e070 con 0x7fe0800b1800 2024-01-11T08:29:59.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.945+0000 7fe07e7fc700 1 -- 172.21.15.27:0/1889551162 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07800b970 con 0x7fe0800b1800 2024-01-11T08:29:59.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.945+0000 7fe07e7fc700 1 -- 172.21.15.27:0/1889551162 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0780092e0 con 0x7fe0800b1800 2024-01-11T08:29:59.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.945+0000 7fe085fff700 1 -- 172.21.15.27:0/1889551162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0800b1800 msgr2=0x7fe0800ca1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:59.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.945+0000 7fe085fff700 1 --2- 172.21.15.27:0/1889551162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0800b1800 0x7fe0800ca1a0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fe0800694d0 tx=0x7fe078009840 comp rx=0 tx=0).stop 2024-01-11T08:29:59.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.945+0000 7fe085fff700 1 -- 172.21.15.27:0/1889551162 shutdown_connections 2024-01-11T08:29:59.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.945+0000 7fe085fff700 1 --2- 172.21.15.27:0/1889551162 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe0801575e0 0x7fe080158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.945+0000 7fe085fff700 1 --2- 172.21.15.27:0/1889551162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0800b1800 0x7fe0800ca1a0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.945+0000 7fe085fff700 1 --2- 172.21.15.27:0/1889551162 >> v2:172.21.15.77:3300/0 conn(0x7fe08006b610 0x7fe0800b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.945+0000 7fe085fff700 1 -- 172.21.15.27:0/1889551162 >> 172.21.15.27:0/1889551162 conn(0x7fe0800680e0 msgr2=0x7fe080069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:29:59.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.945+0000 7fe085fff700 1 -- 172.21.15.27:0/1889551162 shutdown_connections 2024-01-11T08:29:59.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.946+0000 7fe085fff700 1 -- 172.21.15.27:0/1889551162 wait complete. 2024-01-11T08:29:59.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.947+0000 7fe085fff700 1 Processor -- start 2024-01-11T08:29:59.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.947+0000 7fe085fff700 1 -- start start 2024-01-11T08:29:59.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.948+0000 7fe085fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe08006b610 0x7fe080153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:59.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.948+0000 7fe085fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0800b1800 0x7fe0800a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:59.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.948+0000 7fe085fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0801575e0 0x7fe080147c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:59.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.948+0000 7fe085fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0800a59d0 con 0x7fe0801575e0 2024-01-11T08:29:59.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.948+0000 7fe085fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0800af0b0 con 0x7fe0800b1800 2024-01-11T08:29:59.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.948+0000 7fe085fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0800af910 con 0x7fe08006b610 2024-01-11T08:29:59.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.948+0000 7fe07ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0801575e0 0x7fe080147c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:59.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.948+0000 7fe07ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0801575e0 0x7fe080147c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57702/0 (socket says 172.21.15.27:57702) 2024-01-11T08:29:59.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.948+0000 7fe07ffff700 1 -- 172.21.15.27:0/71677631 learned_addr learned my addr 172.21.15.27:0/71677631 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:29:59.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.948+0000 7fe07effd700 1 --2- 172.21.15.27:0/71677631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0800b1800 0x7fe0800a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:59.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.948+0000 7fe07f7fe700 1 --2- 172.21.15.27:0/71677631 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe08006b610 0x7fe080153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:29:59.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.949+0000 7fe07effd700 1 -- 172.21.15.27:0/71677631 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe08006b610 msgr2=0x7fe080153650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:59.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.949+0000 7fe07effd700 1 --2- 172.21.15.27:0/71677631 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe08006b610 0x7fe080153650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.949+0000 7fe07effd700 1 -- 172.21.15.27:0/71677631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0801575e0 msgr2=0x7fe080147c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:29:59.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.949+0000 7fe07effd700 1 --2- 172.21.15.27:0/71677631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0801575e0 0x7fe080147c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:29:59.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.949+0000 7fe07effd700 1 -- 172.21.15.27:0/71677631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe078009ae0 con 0x7fe0800b1800 2024-01-11T08:29:59.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.949+0000 7fe07effd700 1 --2- 172.21.15.27:0/71677631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0800b1800 0x7fe0800a5490 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe07400ea50 tx=0x7fe0740091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:29:59.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.949+0000 7fe0677fe700 1 -- 172.21.15.27:0/71677631 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07400ee90 con 0x7fe0800b1800 2024-01-11T08:29:59.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.949+0000 7fe0677fe700 1 -- 172.21.15.27:0/71677631 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07400c7f0 con 0x7fe0800b1800 2024-01-11T08:29:59.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.949+0000 7fe085fff700 1 -- 172.21.15.27:0/71677631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe08016a0e0 con 0x7fe0800b1800 2024-01-11T08:29:59.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.949+0000 7fe0677fe700 1 -- 172.21.15.27:0/71677631 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07400cb90 con 0x7fe0800b1800 2024-01-11T08:29:59.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.950+0000 7fe085fff700 1 -- 172.21.15.27:0/71677631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0800af2c0 con 0x7fe0800b1800 2024-01-11T08:29:59.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.951+0000 7fe085fff700 1 -- 172.21.15.27:0/71677631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe080070d70 con 0x7fe0800b1800 2024-01-11T08:29:59.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.952+0000 7fe0677fe700 1 -- 172.21.15.27:0/71677631 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 74) v1 ==== 84901+0+0 (secure 0 0 0) 0x7fe074013470 con 0x7fe0800b1800 2024-01-11T08:29:59.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.952+0000 7fe0677fe700 1 --2- 172.21.15.27:0/71677631 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe06c0670e0 0x7fe06c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:29:59.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.953+0000 7fe0677fe700 1 -- 172.21.15.27:0/71677631 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7fe074084b20 con 0x7fe0800b1800 2024-01-11T08:29:59.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.953+0000 7fe07f7fe700 1 -- 172.21.15.27:0/71677631 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe06c0670e0 msgr2=0x7fe06c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:29:59.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.953+0000 7fe07f7fe700 1 --2- 172.21.15.27:0/71677631 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe06c0670e0 0x7fe06c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:29:59.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:29:59.957+0000 7fe0677fe700 1 -- 172.21.15.27:0/71677631 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe080070d70 con 0x7fe0800b1800 2024-01-11T08:30:00.129 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:00.129+0000 7fad63d071c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:30:00.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.154+0000 7fe07f7fe700 1 -- 172.21.15.27:0/71677631 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe06c0670e0 msgr2=0x7fe06c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:30:00.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.154+0000 7fe07f7fe700 1 --2- 172.21.15.27:0/71677631 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe06c0670e0 0x7fe06c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:30:00.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.161+0000 7fe085fff700 1 -- 172.21.15.27:0/71677631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0800afa50 con 0x7fe0800b1800 2024-01-11T08:30:00.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.162+0000 7fe0677fe700 1 -- 172.21.15.27:0/71677631 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+15770 (secure 0 0 0) 0x7fe0740519d0 con 0x7fe0800b1800 2024-01-11T08:30:00.162 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:30:00.162 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":132,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:29:46.941384+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:29:28.131181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-11T08:29:29.438397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/259960245":"2024-01-12T08:29:46.941337+0000","172.21.15.77:6836/41381":"2024-01-12T08:29:46.941337+0000","172.21.15.77:6835/41381":"2024-01-12T08:29:46.941337+0000","172.21.15.77:0/3475507657":"2024-01-12T08:29:46.941337+0000","172.21.15.77:0/3338577024":"2024-01-12T08:29:46.941337+0000","172.21.15.77:0/3974991130":"2024-01-12T08:29:46.941337+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:30:00.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.164+0000 7fe085fff700 1 -- 172.21.15.27:0/71677631 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe06c0670e0 msgr2=0x7fe06c069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:30:00.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.164+0000 7fe085fff700 1 --2- 172.21.15.27:0/71677631 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe06c0670e0 0x7fe06c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.164+0000 7fe085fff700 1 -- 172.21.15.27:0/71677631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0800b1800 msgr2=0x7fe0800a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:00.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.164+0000 7fe085fff700 1 --2- 172.21.15.27:0/71677631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0800b1800 0x7fe0800a5490 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe07400ea50 tx=0x7fe0740091a0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.165+0000 7fe085fff700 1 -- 172.21.15.27:0/71677631 shutdown_connections 2024-01-11T08:30:00.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.165+0000 7fe085fff700 1 --2- 172.21.15.27:0/71677631 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fe06c0670e0 0x7fe06c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.165+0000 7fe085fff700 1 --2- 172.21.15.27:0/71677631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0801575e0 0x7fe080147c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.165+0000 7fe085fff700 1 --2- 172.21.15.27:0/71677631 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe08006b610 0x7fe080153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.165+0000 7fe085fff700 1 --2- 172.21.15.27:0/71677631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0800b1800 0x7fe0800a5490 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.165+0000 7fe085fff700 1 -- 172.21.15.27:0/71677631 >> 172.21.15.27:0/71677631 conn(0x7fe0800680e0 msgr2=0x7fe08006df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:00.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.165+0000 7fe085fff700 1 -- 172.21.15.27:0/71677631 shutdown_connections 2024-01-11T08:30:00.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.165+0000 7fe085fff700 1 -- 172.21.15.27:0/71677631 wait complete. 2024-01-11T08:30:00.185 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-11T08:30:00.302 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:00.303+0000 7ff71dca21c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:30:00.334 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:00.334+0000 7fad63d071c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:30:00.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.400+0000 7f7c76fff700 1 Processor -- start 2024-01-11T08:30:00.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.401+0000 7f7c76fff700 1 -- start start 2024-01-11T08:30:00.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.402+0000 7f7c76fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c70156ab0 0x7f7c700ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:00.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.402+0000 7f7c76fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c700cc9b0 0x7f7c70156e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:00.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.402+0000 7f7c76fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c700adc80 con 0x7f7c701573c0 2024-01-11T08:30:00.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.402+0000 7f7c76fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c700adf60 con 0x7f7c70156ab0 2024-01-11T08:30:00.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.402+0000 7f7c76fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c700ae240 con 0x7f7c700cc9b0 2024-01-11T08:30:00.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.402+0000 7f7c74d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c70156ab0 0x7f7c700ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:00.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.402+0000 7f7c7559c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7c701573c0 0x7f7c700af540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58086/0 (socket says 172.21.15.27:58086) 2024-01-11T08:30:00.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.402+0000 7f7c7559c700 1 -- 172.21.15.27:0/3363445931 learned_addr learned my addr 172.21.15.27:0/3363445931 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:30:00.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.402+0000 7f7c6ffff700 1 --2- 172.21.15.27:0/3363445931 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c700cc9b0 0x7f7c70156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:00.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.403+0000 7f7c6f7fe700 1 -- 172.21.15.27:0/3363445931 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063553300 0 0) 0x7f7c700adc80 con 0x7f7c701573c0 2024-01-11T08:30:00.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.403+0000 7f7c6f7fe700 1 -- 172.21.15.27:0/3363445931 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c60003570 con 0x7f7c701573c0 2024-01-11T08:30:00.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.403+0000 7f7c74d9b700 1 -- 172.21.15.27:0/3363445931 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c700cc9b0 msgr2=0x7f7c70156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:00.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.403+0000 7f7c74d9b700 1 --2- 172.21.15.27:0/3363445931 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c700cc9b0 0x7f7c70156e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.403+0000 7f7c74d9b700 1 -- 172.21.15.27:0/3363445931 >> v1:172.21.15.27:6789/0 conn(0x7f7c701573c0 legacy=0x7f7c700af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:00.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.403+0000 7f7c74d9b700 1 -- 172.21.15.27:0/3363445931 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c700ae520 con 0x7f7c70156ab0 2024-01-11T08:30:00.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.403+0000 7f7c6ffff700 1 --2- 172.21.15.27:0/3363445931 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c700cc9b0 0x7f7c70156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:30:00.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.403+0000 7f7c74d9b700 1 --2- 172.21.15.27:0/3363445931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c70156ab0 0x7f7c700ad650 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f7c640098e0 tx=0x7f7c640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7edc890d05e350c7 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:00.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.403+0000 7f7c6f7fe700 1 -- 172.21.15.27:0/3363445931 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c64011630 con 0x7f7c70156ab0 2024-01-11T08:30:00.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.403+0000 7f7c6f7fe700 1 -- 172.21.15.27:0/3363445931 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c64011790 con 0x7f7c70156ab0 2024-01-11T08:30:00.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.403+0000 7f7c6f7fe700 1 -- 172.21.15.27:0/3363445931 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c64011b30 con 0x7f7c70156ab0 2024-01-11T08:30:00.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.404+0000 7f7c76fff700 1 -- 172.21.15.27:0/3363445931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c70156ab0 msgr2=0x7f7c700ad650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:00.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.404+0000 7f7c76fff700 1 --2- 172.21.15.27:0/3363445931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c70156ab0 0x7f7c700ad650 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f7c640098e0 tx=0x7f7c640091a0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.404+0000 7f7c76fff700 1 -- 172.21.15.27:0/3363445931 shutdown_connections 2024-01-11T08:30:00.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.404+0000 7f7c76fff700 1 --2- 172.21.15.27:0/3363445931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c70156ab0 0x7f7c700ad650 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.404+0000 7f7c76fff700 1 --2- 172.21.15.27:0/3363445931 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c700cc9b0 0x7f7c70156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.404+0000 7f7c76fff700 1 -- 172.21.15.27:0/3363445931 >> 172.21.15.27:0/3363445931 conn(0x7f7c700680b0 msgr2=0x7f7c700cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:00.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.404+0000 7f7c76fff700 1 -- 172.21.15.27:0/3363445931 shutdown_connections 2024-01-11T08:30:00.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.404+0000 7f7c76fff700 1 -- 172.21.15.27:0/3363445931 wait complete. 2024-01-11T08:30:00.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.406+0000 7f7c76fff700 1 Processor -- start 2024-01-11T08:30:00.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.406+0000 7f7c76fff700 1 -- start start 2024-01-11T08:30:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.407+0000 7f7c76fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c700cc9b0 0x7f7c7014c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.407+0000 7f7c76fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c70156ab0 0x7f7c7015c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.407+0000 7f7c76fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c701573c0 0x7f7c700cb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.407+0000 7f7c76fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7015af90 con 0x7f7c701573c0 2024-01-11T08:30:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.407+0000 7f7c76fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7015c7b0 con 0x7f7c700cc9b0 2024-01-11T08:30:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.407+0000 7f7c76fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c700cbdc0 con 0x7f7c70156ab0 2024-01-11T08:30:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.407+0000 7f7c7559c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c701573c0 0x7f7c700cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.407+0000 7f7c7559c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c701573c0 0x7f7c700cb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57722/0 (socket says 172.21.15.27:57722) 2024-01-11T08:30:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.407+0000 7f7c74d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c700cc9b0 0x7f7c7014c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.407+0000 7f7c7559c700 1 -- 172.21.15.27:0/2393801907 learned_addr learned my addr 172.21.15.27:0/2393801907 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.407+0000 7f7c6ffff700 1 --2- 172.21.15.27:0/2393801907 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c70156ab0 0x7f7c7015c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.408+0000 7f7c7559c700 1 -- 172.21.15.27:0/2393801907 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c70156ab0 msgr2=0x7f7c7015c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.408+0000 7f7c7559c700 1 --2- 172.21.15.27:0/2393801907 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c70156ab0 0x7f7c7015c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.408+0000 7f7c7559c700 1 -- 172.21.15.27:0/2393801907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c700cc9b0 msgr2=0x7f7c7014c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.408+0000 7f7c7559c700 1 --2- 172.21.15.27:0/2393801907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c700cc9b0 0x7f7c7014c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.408+0000 7f7c7559c700 1 -- 172.21.15.27:0/2393801907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c009420 con 0x7f7c701573c0 2024-01-11T08:30:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.408+0000 7f7c7559c700 1 --2- 172.21.15.27:0/2393801907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c701573c0 0x7f7c700cb880 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f7c68001460 tx=0x7f7c68001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.408+0000 7f7c6cff9700 1 -- 172.21.15.27:0/2393801907 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6800a890 con 0x7f7c701573c0 2024-01-11T08:30:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.408+0000 7f7c6cff9700 1 -- 172.21.15.27:0/2393801907 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6800a9f0 con 0x7f7c701573c0 2024-01-11T08:30:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.408+0000 7f7c76fff700 1 -- 172.21.15.27:0/2393801907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c64009420 con 0x7f7c701573c0 2024-01-11T08:30:00.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.408+0000 7f7c6cff9700 1 -- 172.21.15.27:0/2393801907 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6800ad70 con 0x7f7c701573c0 2024-01-11T08:30:00.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.408+0000 7f7c76fff700 1 -- 172.21.15.27:0/2393801907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7015a660 con 0x7f7c701573c0 2024-01-11T08:30:00.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.410+0000 7f7c76fff700 1 -- 172.21.15.27:0/2393801907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c70070ce0 con 0x7f7c701573c0 2024-01-11T08:30:00.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.410+0000 7f7c6cff9700 1 -- 172.21.15.27:0/2393801907 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 74) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f7c68007300 con 0x7f7c701573c0 2024-01-11T08:30:00.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.411+0000 7f7c6cff9700 1 --2- 172.21.15.27:0/2393801907 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7c540670e0 0x7f7c54069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:00.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.411+0000 7f7c6cff9700 1 -- 172.21.15.27:0/2393801907 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f7c68084eb0 con 0x7f7c701573c0 2024-01-11T08:30:00.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.411+0000 7f7c74d9b700 1 -- 172.21.15.27:0/2393801907 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7c540670e0 msgr2=0x7f7c54069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:30:00.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.411+0000 7f7c74d9b700 1 --2- 172.21.15.27:0/2393801907 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7c540670e0 0x7f7c54069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:30:00.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.416+0000 7f7c6cff9700 1 -- 172.21.15.27:0/2393801907 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c70070ce0 con 0x7f7c701573c0 2024-01-11T08:30:00.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.612+0000 7f7c74d9b700 1 -- 172.21.15.27:0/2393801907 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7c540670e0 msgr2=0x7f7c54069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:30:00.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.612+0000 7f7c74d9b700 1 --2- 172.21.15.27:0/2393801907 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7c540670e0 0x7f7c54069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:30:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.668+0000 7f7c76fff700 1 -- 172.21.15.27:0/2393801907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7c7004c3c0 con 0x7f7c701573c0 2024-01-11T08:30:00.668 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-11T08:30:00.669+0000 7f27065f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-11T08:30:00.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.674+0000 7f7c6cff9700 1 -- 172.21.15.27:0/2393801907 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v70) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7c68051cb0 con 0x7f7c701573c0 2024-01-11T08:30:00.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.678+0000 7f7c76fff700 1 -- 172.21.15.27:0/2393801907 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7c540670e0 msgr2=0x7f7c54069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:30:00.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.678+0000 7f7c76fff700 1 --2- 172.21.15.27:0/2393801907 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7c540670e0 0x7f7c54069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.678+0000 7f7c76fff700 1 -- 172.21.15.27:0/2393801907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c701573c0 msgr2=0x7f7c700cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:00.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.678+0000 7f7c76fff700 1 --2- 172.21.15.27:0/2393801907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c701573c0 0x7f7c700cb880 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f7c68001460 tx=0x7f7c68001ae0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.678+0000 7f7c76fff700 1 -- 172.21.15.27:0/2393801907 shutdown_connections 2024-01-11T08:30:00.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.678+0000 7f7c76fff700 1 --2- 172.21.15.27:0/2393801907 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7c540670e0 0x7f7c54069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.678+0000 7f7c76fff700 1 --2- 172.21.15.27:0/2393801907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c701573c0 0x7f7c700cb880 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.678+0000 7f7c76fff700 1 --2- 172.21.15.27:0/2393801907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c700cc9b0 0x7f7c7014c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.678+0000 7f7c76fff700 1 --2- 172.21.15.27:0/2393801907 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c70156ab0 0x7f7c7015c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.678+0000 7f7c76fff700 1 -- 172.21.15.27:0/2393801907 >> 172.21.15.27:0/2393801907 conn(0x7f7c700680b0 msgr2=0x7f7c7014de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:00.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.678+0000 7f7c76fff700 1 -- 172.21.15.27:0/2393801907 shutdown_connections 2024-01-11T08:30:00.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.678+0000 7f7c76fff700 1 -- 172.21.15.27:0/2393801907 wait complete. 2024-01-11T08:30:00.679 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-11T08:30:00.693 INFO:tasks.ceph.mds.a.smithi027.stderr: -7190> 2024-01-11T08:29:33.881+0000 7fd51a3f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T08:30:00.707 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-11T08:30:00.722 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:00.723+0000 7ff71dca21c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:30:00.741 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:00.741+0000 7fad63d071c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:30:00.748 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T08:30:00.905 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:00.906+0000 7ff71dca21c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:30:00.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.917+0000 7f73cd3ff700 1 Processor -- start 2024-01-11T08:30:00.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.917+0000 7f73cd3ff700 1 -- start start 2024-01-11T08:30:00.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.919+0000 7f73cd3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73c81577e0 0x7f73c80ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:00.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.919+0000 7f73cd3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f73c8069540 0x7f73c80ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:00.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.919+0000 7f73cd3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c80699d0 con 0x7f73c8157410 2024-01-11T08:30:00.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.919+0000 7f73cd3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c8069cb0 con 0x7f73c8069540 2024-01-11T08:30:00.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.919+0000 7f73cd3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c8069f90 con 0x7f73c81577e0 2024-01-11T08:30:00.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.919+0000 7f73c77fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f73c8069540 0x7f73c80ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:00.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.919+0000 7f73c67fc700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73c81577e0 0x7f73c80ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:00.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.919+0000 7f73c77fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f73c8069540 0x7f73c80ae540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57750/0 (socket says 172.21.15.27:57750) 2024-01-11T08:30:00.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.919+0000 7f73c77fe700 1 -- 172.21.15.27:0/3125394538 learned_addr learned my addr 172.21.15.27:0/3125394538 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:00.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.919+0000 7f73c6ffd700 1 --1- 172.21.15.27:0/3125394538 >> v1:172.21.15.77:6789/0 conn(0x7f73c8157410 0x7f73c80ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:50872/0 (socket says 172.21.15.27:50872) 2024-01-11T08:30:00.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.920+0000 7f73c77fe700 1 -- 172.21.15.27:0/3125394538 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73c81577e0 msgr2=0x7f73c80ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:00.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.920+0000 7f73c77fe700 1 --2- 172.21.15.27:0/3125394538 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73c81577e0 0x7f73c80ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.920+0000 7f73c5ffb700 1 -- 172.21.15.27:0/3125394538 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882536860 0 0) 0x7f73c80699d0 con 0x7f73c8157410 2024-01-11T08:30:00.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.920+0000 7f73c77fe700 1 -- 172.21.15.27:0/3125394538 >> v1:172.21.15.77:6789/0 conn(0x7f73c8157410 legacy=0x7f73c80ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:00.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.920+0000 7f73c77fe700 1 -- 172.21.15.27:0/3125394538 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c806a270 con 0x7f73c8069540 2024-01-11T08:30:00.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.920+0000 7f73c77fe700 1 --2- 172.21.15.27:0/3125394538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73c8069540 0x7f73c80ae540 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f73c0009f20 tx=0x7f73c0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b6aede3a4acc0d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:00.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.920+0000 7f73c5ffb700 1 -- 172.21.15.27:0/3125394538 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c000e070 con 0x7f73c8069540 2024-01-11T08:30:00.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.920+0000 7f73c5ffb700 1 -- 172.21.15.27:0/3125394538 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c000b9d0 con 0x7f73c8069540 2024-01-11T08:30:00.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.920+0000 7f73c5ffb700 1 -- 172.21.15.27:0/3125394538 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c00092e0 con 0x7f73c8069540 2024-01-11T08:30:00.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.921+0000 7f73cd3ff700 1 -- 172.21.15.27:0/3125394538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73c8069540 msgr2=0x7f73c80ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:00.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.921+0000 7f73cd3ff700 1 --2- 172.21.15.27:0/3125394538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73c8069540 0x7f73c80ae540 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f73c0009f20 tx=0x7f73c0009840 comp rx=0 tx=0).stop 2024-01-11T08:30:00.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.921+0000 7f73cd3ff700 1 -- 172.21.15.27:0/3125394538 shutdown_connections 2024-01-11T08:30:00.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.921+0000 7f73cd3ff700 1 --2- 172.21.15.27:0/3125394538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73c8069540 0x7f73c80ae540 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.921+0000 7f73cd3ff700 1 --2- 172.21.15.27:0/3125394538 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73c81577e0 0x7f73c80ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.921+0000 7f73cd3ff700 1 -- 172.21.15.27:0/3125394538 >> 172.21.15.27:0/3125394538 conn(0x7f73c80680a0 msgr2=0x7f73c80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:00.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.921+0000 7f73cd3ff700 1 -- 172.21.15.27:0/3125394538 shutdown_connections 2024-01-11T08:30:00.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.921+0000 7f73cd3ff700 1 -- 172.21.15.27:0/3125394538 wait complete. 2024-01-11T08:30:00.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.923+0000 7f73cd3ff700 1 Processor -- start 2024-01-11T08:30:00.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.923+0000 7f73cd3ff700 1 -- start start 2024-01-11T08:30:00.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.924+0000 7f73cd3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73c8069540 0x7f73c815d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:00.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.924+0000 7f73cd3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73c8157410 0x7f73c80b1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:00.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.924+0000 7f73cd3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73c81577e0 0x7f73c8154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:00.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.924+0000 7f73cd3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c815da60 con 0x7f73c8157410 2024-01-11T08:30:00.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.924+0000 7f73cd3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c815c730 con 0x7f73c8069540 2024-01-11T08:30:00.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.924+0000 7f73cd3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c815c450 con 0x7f73c81577e0 2024-01-11T08:30:00.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.924+0000 7f73c67fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73c8157410 0x7f73c80b1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:00.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.924+0000 7f73c6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73c8069540 0x7f73c815d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:00.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.924+0000 7f73c67fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73c8157410 0x7f73c80b1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57766/0 (socket says 172.21.15.27:57766) 2024-01-11T08:30:00.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.924+0000 7f73c67fc700 1 -- 172.21.15.27:0/3827678859 learned_addr learned my addr 172.21.15.27:0/3827678859 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:00.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.924+0000 7f73c77fe700 1 --2- 172.21.15.27:0/3827678859 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73c81577e0 0x7f73c8154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:00.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.925+0000 7f73c67fc700 1 -- 172.21.15.27:0/3827678859 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73c81577e0 msgr2=0x7f73c8154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:00.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.925+0000 7f73c67fc700 1 --2- 172.21.15.27:0/3827678859 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73c81577e0 0x7f73c8154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.925+0000 7f73c67fc700 1 -- 172.21.15.27:0/3827678859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73c8069540 msgr2=0x7f73c815d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:00.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.925+0000 7f73c67fc700 1 --2- 172.21.15.27:0/3827678859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73c8069540 0x7f73c815d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:00.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.925+0000 7f73c67fc700 1 -- 172.21.15.27:0/3827678859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c0009ae0 con 0x7f73c8157410 2024-01-11T08:30:00.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.925+0000 7f73c6ffd700 1 --2- 172.21.15.27:0/3827678859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73c8069540 0x7f73c815d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:30:00.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.925+0000 7f73c67fc700 1 --2- 172.21.15.27:0/3827678859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73c8157410 0x7f73c80b1720 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f73bc00e900 tx=0x7f73bc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:00.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.925+0000 7f73aeffd700 1 -- 172.21.15.27:0/3827678859 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc013680 con 0x7f73c8157410 2024-01-11T08:30:00.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.925+0000 7f73aeffd700 1 -- 172.21.15.27:0/3827678859 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73bc0137e0 con 0x7f73c8157410 2024-01-11T08:30:00.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.925+0000 7f73aeffd700 1 -- 172.21.15.27:0/3827678859 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc013ba0 con 0x7f73c8157410 2024-01-11T08:30:00.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.925+0000 7f73cd3ff700 1 -- 172.21.15.27:0/3827678859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c815ca40 con 0x7f73c8157410 2024-01-11T08:30:00.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.925+0000 7f73cd3ff700 1 -- 172.21.15.27:0/3827678859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c815d000 con 0x7f73c8157410 2024-01-11T08:30:00.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.927+0000 7f73cd3ff700 1 -- 172.21.15.27:0/3827678859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c804c3c0 con 0x7f73c8157410 2024-01-11T08:30:00.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.928+0000 7f73aeffd700 1 -- 172.21.15.27:0/3827678859 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 74) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f73bc0123d0 con 0x7f73c8157410 2024-01-11T08:30:00.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.929+0000 7f73aeffd700 1 --2- 172.21.15.27:0/3827678859 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f73b40670e0 0x7f73b4069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:00.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.929+0000 7f73aeffd700 1 -- 172.21.15.27:0/3827678859 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6105+0+0 (secure 0 0 0) 0x7f73bc084880 con 0x7f73c8157410 2024-01-11T08:30:00.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.929+0000 7f73c6ffd700 1 -- 172.21.15.27:0/3827678859 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f73b40670e0 msgr2=0x7f73b4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:30:00.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.929+0000 7f73c6ffd700 1 --2- 172.21.15.27:0/3827678859 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f73b40670e0 0x7f73b4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:30:00.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:00.933+0000 7f73aeffd700 1 -- 172.21.15.27:0/3827678859 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f73bc051640 con 0x7f73c8157410 2024-01-11T08:30:01.096 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:01.097+0000 7ff71dca21c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:30:01.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.130+0000 7f73c6ffd700 1 -- 172.21.15.27:0/3827678859 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f73b40670e0 msgr2=0x7f73b4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:30:01.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.130+0000 7f73c6ffd700 1 --2- 172.21.15.27:0/3827678859 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f73b40670e0 0x7f73b4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:30:01.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.187+0000 7f73cd3ff700 1 -- 172.21.15.27:0/3827678859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f73c80c9760 con 0x7f73c8157410 2024-01-11T08:30:01.203 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:01.203+0000 7fad63d071c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:30:01.206 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:01.207+0000 7ff71dca21c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:30:01.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.216+0000 7f73aeffd700 1 -- 172.21.15.27:0/3827678859 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 75) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f73bc051a10 con 0x7f73c8157410 2024-01-11T08:30:01.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.217+0000 7f73aeffd700 1 -- 172.21.15.27:0/3827678859 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f73b40670e0 msgr2=0x7f73b4069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:30:01.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.217+0000 7f73aeffd700 1 --2- 172.21.15.27:0/3827678859 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f73b40670e0 0x7f73b4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:01.283 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:01.282+0000 7ff698a84700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:01.283 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:01.282+0000 7ff698a84700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:01.283 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:01.282+0000 7ff698a84700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:01.283 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:01.282+0000 7ff698a84700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:01.283 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:01.282+0000 7ff698a84700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:01.283 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:01.283+0000 7ff69aa88700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:01.283 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:01.283+0000 7ff69aa88700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:01.284 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:01.283+0000 7ff69aa88700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:01.284 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:01.283+0000 7ff69aa88700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:01.284 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:01.283+0000 7ff69aa88700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:01.401 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:01.401+0000 7fad63d071c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:30:01.599 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:01.599+0000 7fad63d071c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:30:01.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.674+0000 7f73aeffd700 1 -- 172.21.15.27:0/3827678859 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v71) v1 ==== 104+0+0 (secure 0 0 0) 0x7f73bc06fef0 con 0x7f73c8157410 2024-01-11T08:30:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.677+0000 7f73cd3ff700 1 -- 172.21.15.27:0/3827678859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73c8157410 msgr2=0x7f73c80b1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.677+0000 7f73cd3ff700 1 --2- 172.21.15.27:0/3827678859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73c8157410 0x7f73c80b1720 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f73bc00e900 tx=0x7f73bc0091a0 comp rx=0 tx=0).stop 2024-01-11T08:30:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.677+0000 7f73cd3ff700 1 -- 172.21.15.27:0/3827678859 shutdown_connections 2024-01-11T08:30:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.677+0000 7f73cd3ff700 1 --2- 172.21.15.27:0/3827678859 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f73b40670e0 0x7f73b4069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.677+0000 7f73cd3ff700 1 --2- 172.21.15.27:0/3827678859 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73c81577e0 0x7f73c8154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.677+0000 7f73cd3ff700 1 --2- 172.21.15.27:0/3827678859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73c8069540 0x7f73c815d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:01.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.677+0000 7f73cd3ff700 1 --2- 172.21.15.27:0/3827678859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73c8157410 0x7f73c80b1720 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:01.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.677+0000 7f73cd3ff700 1 -- 172.21.15.27:0/3827678859 >> 172.21.15.27:0/3827678859 conn(0x7f73c80680a0 msgr2=0x7f73c80c8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:01.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.677+0000 7f73cd3ff700 1 -- 172.21.15.27:0/3827678859 shutdown_connections 2024-01-11T08:30:01.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.677+0000 7f73cd3ff700 1 -- 172.21.15.27:0/3827678859 wait complete. 2024-01-11T08:30:01.696 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-11T08:30:01.716 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:01.717+0000 7fad63d071c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:30:01.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.921+0000 7f06b25ff700 1 Processor -- start 2024-01-11T08:30:01.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.921+0000 7f06b25ff700 1 -- start start 2024-01-11T08:30:01.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.921+0000 7f06b25ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f06ac0b08a0 0x7f06ac0684d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:01.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.921+0000 7f06b25ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f06ac068a10 0x7f06ac0ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:01.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.921+0000 7f06b25ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06ac068e10 con 0x7f06ac159250 2024-01-11T08:30:01.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.921+0000 7f06b25ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06ac15ace0 con 0x7f06ac0b08a0 2024-01-11T08:30:01.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.921+0000 7f06b25ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06ac0cacf0 con 0x7f06ac068a10 2024-01-11T08:30:01.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.922+0000 7f06ab7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f06ac0b08a0 0x7f06ac0684d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:01.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.922+0000 7f06b0b9c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f06ac068a10 0x7f06ac0ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:01.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.922+0000 7f06ab7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f06ac0b08a0 0x7f06ac0684d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57776/0 (socket says 172.21.15.27:57776) 2024-01-11T08:30:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.922+0000 7f06ab7fe700 1 -- 172.21.15.27:0/535708627 learned_addr learned my addr 172.21.15.27:0/535708627 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.922+0000 7f06ab7fe700 1 -- 172.21.15.27:0/535708627 >> v2:172.21.15.77:3300/0 conn(0x7f06ac068a10 msgr2=0x7f06ac0ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.922+0000 7f06ab7fe700 1 --2- 172.21.15.27:0/535708627 >> v2:172.21.15.77:3300/0 conn(0x7f06ac068a10 0x7f06ac0ce2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.922+0000 7f06ab7fe700 1 -- 172.21.15.27:0/535708627 >> v1:172.21.15.77:6789/0 conn(0x7f06ac159250 legacy=0x7f06ac0cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.922+0000 7f06ab7fe700 1 -- 172.21.15.27:0/535708627 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ac0b0d20 con 0x7f06ac0b08a0 2024-01-11T08:30:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.922+0000 7f06ab7fe700 1 --2- 172.21.15.27:0/535708627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06ac0b08a0 0x7f06ac0684d0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f06ac06f230 tx=0x7f06a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc50241171aec0b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.922+0000 7f06aaffd700 1 -- 172.21.15.27:0/535708627 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a0009e70 con 0x7f06ac0b08a0 2024-01-11T08:30:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.923+0000 7f06aaffd700 1 -- 172.21.15.27:0/535708627 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a000b860 con 0x7f06ac0b08a0 2024-01-11T08:30:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.923+0000 7f06aaffd700 1 -- 172.21.15.27:0/535708627 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a000bc00 con 0x7f06ac0b08a0 2024-01-11T08:30:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.923+0000 7f06b25ff700 1 -- 172.21.15.27:0/535708627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06ac0b08a0 msgr2=0x7f06ac0684d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.923+0000 7f06b25ff700 1 --2- 172.21.15.27:0/535708627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06ac0b08a0 0x7f06ac0684d0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f06ac06f230 tx=0x7f06a0009250 comp rx=0 tx=0).stop 2024-01-11T08:30:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.923+0000 7f06b25ff700 1 -- 172.21.15.27:0/535708627 shutdown_connections 2024-01-11T08:30:01.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.923+0000 7f06b25ff700 1 --2- 172.21.15.27:0/535708627 >> v2:172.21.15.77:3300/0 conn(0x7f06ac068a10 0x7f06ac0ce2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:01.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.923+0000 7f06b25ff700 1 --2- 172.21.15.27:0/535708627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06ac0b08a0 0x7f06ac0684d0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:01.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.923+0000 7f06b25ff700 1 -- 172.21.15.27:0/535708627 >> 172.21.15.27:0/535708627 conn(0x7f06ac067610 msgr2=0x7f06ac067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:01.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.923+0000 7f06b25ff700 1 -- 172.21.15.27:0/535708627 shutdown_connections 2024-01-11T08:30:01.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.923+0000 7f06b25ff700 1 -- 172.21.15.27:0/535708627 wait complete. 2024-01-11T08:30:01.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.925+0000 7f06b25ff700 1 Processor -- start 2024-01-11T08:30:01.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.925+0000 7f06b25ff700 1 -- start start 2024-01-11T08:30:01.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.926+0000 7f06b25ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f06ac068a10 0x7f06ac0c2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:01.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.926+0000 7f06b25ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06ac0b08a0 0x7f06ac0b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.926+0000 7f06b25ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06ac159250 0x7f06ac0c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.926+0000 7f06b25ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac0c3290 con 0x7f06ac0b08a0 2024-01-11T08:30:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.926+0000 7f06b25ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac0b3fb0 con 0x7f06ac159250 2024-01-11T08:30:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.926+0000 7f06b25ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac0c81c0 con 0x7f06ac068a10 2024-01-11T08:30:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.926+0000 7f06ab7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06ac0b08a0 0x7f06ac0b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.926+0000 7f06ab7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06ac0b08a0 0x7f06ac0b3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57790/0 (socket says 172.21.15.27:57790) 2024-01-11T08:30:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.926+0000 7f06ab7fe700 1 -- 172.21.15.27:0/420441291 learned_addr learned my addr 172.21.15.27:0/420441291 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.926+0000 7f06b0b9c700 1 --2- 172.21.15.27:0/420441291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06ac159250 0x7f06ac0c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.926+0000 7f06abfff700 1 --2- 172.21.15.27:0/420441291 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f06ac068a10 0x7f06ac0c2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.926+0000 7f06ab7fe700 1 -- 172.21.15.27:0/420441291 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f06ac068a10 msgr2=0x7f06ac0c2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.926+0000 7f06ab7fe700 1 --2- 172.21.15.27:0/420441291 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f06ac068a10 0x7f06ac0c2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:01.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.926+0000 7f06ab7fe700 1 -- 172.21.15.27:0/420441291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06ac159250 msgr2=0x7f06ac0c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:01.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.926+0000 7f06ab7fe700 1 --2- 172.21.15.27:0/420441291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06ac159250 0x7f06ac0c7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:01.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.927+0000 7f06ab7fe700 1 -- 172.21.15.27:0/420441291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a00094f0 con 0x7f06ac0b08a0 2024-01-11T08:30:01.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.927+0000 7f06abfff700 1 --2- 172.21.15.27:0/420441291 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f06ac068a10 0x7f06ac0c2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:30:01.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.927+0000 7f06b0b9c700 1 --2- 172.21.15.27:0/420441291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06ac159250 0x7f06ac0c7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:30:01.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.927+0000 7f06ab7fe700 1 --2- 172.21.15.27:0/420441291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06ac0b08a0 0x7f06ac0b3a70 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f06ac0ceef0 tx=0x7f06a0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:01.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.927+0000 7f0693fff700 1 -- 172.21.15.27:0/420441291 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a0011420 con 0x7f06ac0b08a0 2024-01-11T08:30:01.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.927+0000 7f0693fff700 1 -- 172.21.15.27:0/420441291 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a0011580 con 0x7f06ac0b08a0 2024-01-11T08:30:01.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.927+0000 7f0693fff700 1 -- 172.21.15.27:0/420441291 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a0011940 con 0x7f06ac0b08a0 2024-01-11T08:30:01.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.927+0000 7f06b25ff700 1 -- 172.21.15.27:0/420441291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ac0c2cd0 con 0x7f06ac0b08a0 2024-01-11T08:30:01.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.927+0000 7f06b25ff700 1 -- 172.21.15.27:0/420441291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ac0b3270 con 0x7f06ac0b08a0 2024-01-11T08:30:01.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.928+0000 7f06b25ff700 1 -- 172.21.15.27:0/420441291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ac04c3c0 con 0x7f06ac0b08a0 2024-01-11T08:30:01.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.929+0000 7f0693fff700 1 -- 172.21.15.27:0/420441291 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 75) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f06a002e430 con 0x7f06ac0b08a0 2024-01-11T08:30:01.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.930+0000 7f0693fff700 1 -- 172.21.15.27:0/420441291 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6406+0+0 (secure 0 0 0) 0x7f06a0085c90 con 0x7f06ac0b08a0 2024-01-11T08:30:01.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:01.934+0000 7f0693fff700 1 -- 172.21.15.27:0/420441291 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f06a00530b0 con 0x7f06ac0b08a0 2024-01-11T08:30:02.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:02.143+0000 7f06b25ff700 1 -- 172.21.15.27:0/420441291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f06ac04c3c0 con 0x7f06ac0b08a0 2024-01-11T08:30:02.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:02.218+0000 7f0693fff700 1 -- 172.21.15.27:0/420441291 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 76) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f06a0052b60 con 0x7f06ac0b08a0 2024-01-11T08:30:02.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:02.218+0000 7f0693fff700 1 --2- 172.21.15.27:0/420441291 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f069806c780 0x7f069806eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:02.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:02.219+0000 7f06abfff700 1 --2- 172.21.15.27:0/420441291 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f069806c780 0x7f069806eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:02.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:02.220+0000 7f06abfff700 1 --2- 172.21.15.27:0/420441291 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f069806c780 0x7f069806eb60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f069c001840 tx=0x7f069c009040 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.063+0000 7f0693fff700 1 -- 172.21.15.27:0/420441291 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v135) v1 ==== 181+0+0 (secure 0 0 0) 0x7f06a0059760 con 0x7f06ac0b08a0 2024-01-11T08:30:03.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.070+0000 7f06b25ff700 1 -- 172.21.15.27:0/420441291 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f069806c780 msgr2=0x7f069806eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:03.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.070+0000 7f06b25ff700 1 --2- 172.21.15.27:0/420441291 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f069806c780 0x7f069806eb60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f069c001840 tx=0x7f069c009040 comp rx=0 tx=0).stop 2024-01-11T08:30:03.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.070+0000 7f06b25ff700 1 -- 172.21.15.27:0/420441291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06ac0b08a0 msgr2=0x7f06ac0b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:03.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.070+0000 7f06b25ff700 1 --2- 172.21.15.27:0/420441291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06ac0b08a0 0x7f06ac0b3a70 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f06ac0ceef0 tx=0x7f06a0017000 comp rx=0 tx=0).stop 2024-01-11T08:30:03.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.070+0000 7f06b25ff700 1 -- 172.21.15.27:0/420441291 shutdown_connections 2024-01-11T08:30:03.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.070+0000 7f06b25ff700 1 --2- 172.21.15.27:0/420441291 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f069806c780 0x7f069806eb60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:03.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.070+0000 7f06b25ff700 1 --2- 172.21.15.27:0/420441291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06ac159250 0x7f06ac0c7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:03.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.070+0000 7f06b25ff700 1 --2- 172.21.15.27:0/420441291 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f06ac068a10 0x7f06ac0c2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:03.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.070+0000 7f06b25ff700 1 --2- 172.21.15.27:0/420441291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06ac0b08a0 0x7f06ac0b3a70 secure :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f06ac0ceef0 tx=0x7f06a0017000 comp rx=0 tx=0).stop 2024-01-11T08:30:03.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.070+0000 7f06b25ff700 1 -- 172.21.15.27:0/420441291 >> 172.21.15.27:0/420441291 conn(0x7f06ac067610 msgr2=0x7f06ac0cc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:03.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.071+0000 7f06b25ff700 1 -- 172.21.15.27:0/420441291 shutdown_connections 2024-01-11T08:30:03.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.071+0000 7f06b25ff700 1 -- 172.21.15.27:0/420441291 wait complete. 2024-01-11T08:30:03.072 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-01-11T08:30:03.092 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-11T08:30:03.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.308+0000 7f081bfff700 1 Processor -- start 2024-01-11T08:30:03.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.309+0000 7f081bfff700 1 -- start start 2024-01-11T08:30:03.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.310+0000 7f081bfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0814159250 0x7f08140b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:03.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.310+0000 7f081bfff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f081415a440 con 0x7f0814068be0 2024-01-11T08:30:03.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.310+0000 7f081bfff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f081415a720 con 0x7f08140cb410 2024-01-11T08:30:03.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.310+0000 7f081bfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f081415ace0 con 0x7f0814159250 2024-01-11T08:30:03.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.310+0000 7f081a59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0814068be0 0x7f08140ce2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58102/0 (socket says 172.21.15.27:58102) 2024-01-11T08:30:03.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.310+0000 7f081a59c700 1 -- 172.21.15.27:0/3507848899 learned_addr learned my addr 172.21.15.27:0/3507848899 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:30:03.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.311+0000 7f0819d9b700 1 --2- 172.21.15.27:0/3507848899 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0814159250 0x7f08140b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:03.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.311+0000 7f0818d99700 1 -- 172.21.15.27:0/3507848899 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229767557 0 0) 0x7f081415a440 con 0x7f0814068be0 2024-01-11T08:30:03.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.311+0000 7f0818d99700 1 -- 172.21.15.27:0/3507848899 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0800003570 con 0x7f0814068be0 2024-01-11T08:30:03.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.311+0000 7f0818d99700 1 -- 172.21.15.27:0/3507848899 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548298552 0 0) 0x7f081415a720 con 0x7f08140cb410 2024-01-11T08:30:03.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.311+0000 7f0818d99700 1 -- 172.21.15.27:0/3507848899 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f081415a440 con 0x7f08140cb410 2024-01-11T08:30:03.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.311+0000 7f0818d99700 1 -- 172.21.15.27:0/3507848899 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3809326548 0 0) 0x7f0800003570 con 0x7f0814068be0 2024-01-11T08:30:03.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.311+0000 7f0818d99700 1 -- 172.21.15.27:0/3507848899 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0814159250 msgr2=0x7f08140b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:03.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.311+0000 7f0818d99700 1 --2- 172.21.15.27:0/3507848899 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0814159250 0x7f08140b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:03.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.311+0000 7f0818d99700 1 -- 172.21.15.27:0/3507848899 >> v1:172.21.15.77:6789/0 conn(0x7f08140cb410 legacy=0x7f08140684d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:03.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.311+0000 7f0818d99700 1 -- 172.21.15.27:0/3507848899 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081415aa00 con 0x7f0814068be0 2024-01-11T08:30:03.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.311+0000 7f0818d99700 1 -- 172.21.15.27:0/3507848899 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f080c0017f0 con 0x7f0814068be0 2024-01-11T08:30:03.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.311+0000 7f0818d99700 1 -- 172.21.15.27:0/3507848899 >> v1:172.21.15.27:6789/0 conn(0x7f0814068be0 legacy=0x7f08140ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:03.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.311+0000 7f0818d99700 1 --2- 172.21.15.27:0/3507848899 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0814159250 0x7f0800003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:03.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.311+0000 7f0818d99700 1 --2- 172.21.15.27:0/3507848899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0800004180 0x7f0800004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:03.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.311+0000 7f0818d99700 1 --2- 172.21.15.27:0/3507848899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0800004d50 0x7f08000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:03.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.311+0000 7f0818d99700 1 -- 172.21.15.27:0/3507848899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081415ace0 con 0x7f0800004180 2024-01-11T08:30:03.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.311+0000 7f0818d99700 1 -- 172.21.15.27:0/3507848899 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080000bba0 con 0x7f0800004d50 2024-01-11T08:30:03.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.311+0000 7f0818d99700 1 -- 172.21.15.27:0/3507848899 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080000bd60 con 0x7f0814159250 2024-01-11T08:30:03.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.311+0000 7f081959a700 1 --2- 172.21.15.27:0/3507848899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0800004180 0x7f0800004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:03.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.312+0000 7f0819d9b700 1 --2- 172.21.15.27:0/3507848899 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0814159250 0x7f0800003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:03.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.312+0000 7f081a59c700 1 --2- 172.21.15.27:0/3507848899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0800004d50 0x7f08000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:03.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.312+0000 7f081959a700 1 -- 172.21.15.27:0/3507848899 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0814159250 msgr2=0x7f0800003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:03.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.312+0000 7f081959a700 1 --2- 172.21.15.27:0/3507848899 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0814159250 0x7f0800003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:03.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.312+0000 7f081959a700 1 -- 172.21.15.27:0/3507848899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0800004d50 msgr2=0x7f08000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:03.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.312+0000 7f081959a700 1 --2- 172.21.15.27:0/3507848899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0800004d50 0x7f08000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:03.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.312+0000 7f081959a700 1 -- 172.21.15.27:0/3507848899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0800003570 con 0x7f0800004180 2024-01-11T08:30:03.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.312+0000 7f081959a700 1 --2- 172.21.15.27:0/3507848899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0800004180 0x7f0800004810 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f0804001a10 tx=0x7f0804007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3499ce6de950967 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:03.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.312+0000 7f0818d99700 1 -- 172.21.15.27:0/3507848899 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080400c5d0 con 0x7f0800004180 2024-01-11T08:30:03.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.312+0000 7f0818d99700 1 -- 172.21.15.27:0/3507848899 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f080400c730 con 0x7f0800004180 2024-01-11T08:30:03.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.312+0000 7f0818d99700 1 -- 172.21.15.27:0/3507848899 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080400cad0 con 0x7f0800004180 2024-01-11T08:30:03.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.313+0000 7f081bfff700 1 -- 172.21.15.27:0/3507848899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0800004180 msgr2=0x7f0800004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:03.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.313+0000 7f081bfff700 1 --2- 172.21.15.27:0/3507848899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0800004180 0x7f0800004810 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f0804001a10 tx=0x7f0804007d30 comp rx=0 tx=0).stop 2024-01-11T08:30:03.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.313+0000 7f081a59c700 1 -- 172.21.15.27:0/3507848899 reap_dead start 2024-01-11T08:30:03.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.313+0000 7f081bfff700 1 -- 172.21.15.27:0/3507848899 shutdown_connections 2024-01-11T08:30:03.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.313+0000 7f081bfff700 1 -- 172.21.15.27:0/3507848899 >> 172.21.15.27:0/3507848899 conn(0x7f0814067610 msgr2=0x7f0814067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:03.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.313+0000 7f081bfff700 1 -- 172.21.15.27:0/3507848899 shutdown_connections 2024-01-11T08:30:03.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.313+0000 7f081bfff700 1 -- 172.21.15.27:0/3507848899 wait complete. 2024-01-11T08:30:03.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.315+0000 7f081bfff700 1 Processor -- start 2024-01-11T08:30:03.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.315+0000 7f081bfff700 1 -- start start 2024-01-11T08:30:03.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.316+0000 7f081bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0800004180 0x7f08140c1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:03.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.316+0000 7f081bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08140c20b0 0x7f0814143980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:03.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.316+0000 7f081bfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0814143ec0 0x7f08140c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:03.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.316+0000 7f081bfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08140c5870 con 0x7f08140c20b0 2024-01-11T08:30:03.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.316+0000 7f081bfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08140c4800 con 0x7f0800004180 2024-01-11T08:30:03.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.316+0000 7f081bfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08140c4ae0 con 0x7f0814143ec0 2024-01-11T08:30:03.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.316+0000 7f081959a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08140c20b0 0x7f0814143980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:03.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.316+0000 7f081959a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08140c20b0 0x7f0814143980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57814/0 (socket says 172.21.15.27:57814) 2024-01-11T08:30:03.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.316+0000 7f081959a700 1 -- 172.21.15.27:0/443042504 learned_addr learned my addr 172.21.15.27:0/443042504 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:03.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.316+0000 7f0819d9b700 1 --2- 172.21.15.27:0/443042504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0800004180 0x7f08140c1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:03.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.316+0000 7f081a59c700 1 --2- 172.21.15.27:0/443042504 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0814143ec0 0x7f08140c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:03.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.316+0000 7f081959a700 1 -- 172.21.15.27:0/443042504 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0814143ec0 msgr2=0x7f08140c3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:03.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.316+0000 7f081959a700 1 --2- 172.21.15.27:0/443042504 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0814143ec0 0x7f08140c3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:03.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.316+0000 7f081959a700 1 -- 172.21.15.27:0/443042504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0800004180 msgr2=0x7f08140c1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:03.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.316+0000 7f081959a700 1 --2- 172.21.15.27:0/443042504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0800004180 0x7f08140c1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:03.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.316+0000 7f081959a700 1 -- 172.21.15.27:0/443042504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08140c4dc0 con 0x7f08140c20b0 2024-01-11T08:30:03.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.316+0000 7f0819d9b700 1 --2- 172.21.15.27:0/443042504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0800004180 0x7f08140c1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:30:03.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.316+0000 7f081959a700 1 --2- 172.21.15.27:0/443042504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08140c20b0 0x7f0814143980 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f080000bb70 tx=0x7f0804015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:03.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.317+0000 7f0809ffb700 1 -- 172.21.15.27:0/443042504 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080400c5d0 con 0x7f08140c20b0 2024-01-11T08:30:03.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.317+0000 7f0809ffb700 1 -- 172.21.15.27:0/443042504 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0804001360 con 0x7f08140c20b0 2024-01-11T08:30:03.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.317+0000 7f0809ffb700 1 -- 172.21.15.27:0/443042504 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0804012e00 con 0x7f08140c20b0 2024-01-11T08:30:03.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.317+0000 7f081bfff700 1 -- 172.21.15.27:0/443042504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0814144340 con 0x7f08140c20b0 2024-01-11T08:30:03.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.317+0000 7f081bfff700 1 -- 172.21.15.27:0/443042504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08140c3fe0 con 0x7f08140c20b0 2024-01-11T08:30:03.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.318+0000 7f081bfff700 1 -- 172.21.15.27:0/443042504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08140c5a20 con 0x7f08140c20b0 2024-01-11T08:30:03.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.319+0000 7f0809ffb700 1 -- 172.21.15.27:0/443042504 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 76) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f0804019070 con 0x7f08140c20b0 2024-01-11T08:30:03.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.320+0000 7f0809ffb700 1 --2- 172.21.15.27:0/443042504 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f07f8066d80 0x7f07f8069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:03.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.320+0000 7f0809ffb700 1 -- 172.21.15.27:0/443042504 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f08040854a0 con 0x7f08140c20b0 2024-01-11T08:30:03.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.320+0000 7f0819d9b700 1 --2- 172.21.15.27:0/443042504 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f07f8066d80 0x7f07f8069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:03.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.321+0000 7f0819d9b700 1 --2- 172.21.15.27:0/443042504 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f07f8066d80 0x7f07f8069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0810006dc0 tx=0x7f0810008040 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:03.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.324+0000 7f0809ffb700 1 -- 172.21.15.27:0/443042504 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f08140c5a20 con 0x7f08140c20b0 2024-01-11T08:30:03.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:03.529+0000 7f081bfff700 1 -- 172.21.15.27:0/443042504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f081404c3c0 con 0x7f08140c20b0 2024-01-11T08:30:04.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.067+0000 7f0809ffb700 1 -- 172.21.15.27:0/443042504 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 77) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f0804052be0 con 0x7f08140c20b0 2024-01-11T08:30:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.227+0000 7f0809ffb700 1 -- 172.21.15.27:0/443042504 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v136) v1 ==== 169+0+0 (secure 0 0 0) 0x7f080406caf0 con 0x7f08140c20b0 2024-01-11T08:30:04.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.232+0000 7f081bfff700 1 -- 172.21.15.27:0/443042504 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f07f8066d80 msgr2=0x7f07f8069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:04.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.232+0000 7f081bfff700 1 --2- 172.21.15.27:0/443042504 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f07f8066d80 0x7f07f8069230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0810006dc0 tx=0x7f0810008040 comp rx=0 tx=0).stop 2024-01-11T08:30:04.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.232+0000 7f081bfff700 1 -- 172.21.15.27:0/443042504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08140c20b0 msgr2=0x7f0814143980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:04.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.232+0000 7f081bfff700 1 --2- 172.21.15.27:0/443042504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08140c20b0 0x7f0814143980 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f080000bb70 tx=0x7f0804015000 comp rx=0 tx=0).stop 2024-01-11T08:30:04.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.232+0000 7f081bfff700 1 -- 172.21.15.27:0/443042504 shutdown_connections 2024-01-11T08:30:04.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.232+0000 7f081bfff700 1 --2- 172.21.15.27:0/443042504 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f07f8066d80 0x7f07f8069230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.232+0000 7f081bfff700 1 --2- 172.21.15.27:0/443042504 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0814143ec0 0x7f08140c3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.232+0000 7f081bfff700 1 --2- 172.21.15.27:0/443042504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0800004180 0x7f08140c1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.232+0000 7f081bfff700 1 --2- 172.21.15.27:0/443042504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08140c20b0 0x7f0814143980 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.232+0000 7f081bfff700 1 -- 172.21.15.27:0/443042504 >> 172.21.15.27:0/443042504 conn(0x7f0814067610 msgr2=0x7f0814158960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:04.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.233+0000 7f081bfff700 1 -- 172.21.15.27:0/443042504 shutdown_connections 2024-01-11T08:30:04.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.233+0000 7f081bfff700 1 -- 172.21.15.27:0/443042504 wait complete. 2024-01-11T08:30:04.235 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-01-11T08:30:04.255 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-11T08:30:04.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.460+0000 7f4589fff700 1 Processor -- start 2024-01-11T08:30:04.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.460+0000 7f4589fff700 1 -- start start 2024-01-11T08:30:04.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.461+0000 7f4589fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f458415a1b0 0x7f458415a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.461+0000 7f4589fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f45840ccd70 0x7f45840cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.461+0000 7f4589fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45840685c0 0x7f4584068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.461+0000 7f4589fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45840cf850 con 0x7f45840ccd70 2024-01-11T08:30:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.461+0000 7f4589fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45840cec40 con 0x7f458415a1b0 2024-01-11T08:30:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.461+0000 7f4589fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45840cef20 con 0x7f45840685c0 2024-01-11T08:30:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.461+0000 7f4582ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f45840ccd70 0x7f45840cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.461+0000 7f4582ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f45840ccd70 0x7f45840cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53648/0 (socket says 172.21.15.27:53648) 2024-01-11T08:30:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.461+0000 7f4582ffd700 1 -- 172.21.15.27:0/2003413374 learned_addr learned my addr 172.21.15.27:0/2003413374 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.462+0000 7f45837fe700 1 --2- 172.21.15.27:0/2003413374 >> v2:172.21.15.77:3300/0 conn(0x7f458415a1b0 0x7f458415a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.462+0000 7f4583fff700 1 --2- 172.21.15.27:0/2003413374 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45840685c0 0x7f4584068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.462+0000 7f4582ffd700 1 -- 172.21.15.27:0/2003413374 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45840685c0 msgr2=0x7f4584068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.462+0000 7f4582ffd700 1 --2- 172.21.15.27:0/2003413374 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45840685c0 0x7f4584068a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.462+0000 7f4582ffd700 1 -- 172.21.15.27:0/2003413374 >> v2:172.21.15.77:3300/0 conn(0x7f458415a1b0 msgr2=0x7f458415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:04.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.462+0000 7f4582ffd700 1 --2- 172.21.15.27:0/2003413374 >> v2:172.21.15.77:3300/0 conn(0x7f458415a1b0 0x7f458415a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.462+0000 7f4582ffd700 1 -- 172.21.15.27:0/2003413374 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45840cf200 con 0x7f45840ccd70 2024-01-11T08:30:04.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.462+0000 7f45837fe700 1 --2- 172.21.15.27:0/2003413374 >> v2:172.21.15.77:3300/0 conn(0x7f458415a1b0 0x7f458415a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:30:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.462+0000 7f4582ffd700 1 --2- 172.21.15.27:0/2003413374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45840ccd70 0x7f45840cd210 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f45840d16f0 tx=0x7f4578009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab0c7042cf131d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.462+0000 7f45827fc700 1 -- 172.21.15.27:0/2003413374 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457800b860 con 0x7f45840ccd70 2024-01-11T08:30:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.462+0000 7f45827fc700 1 -- 172.21.15.27:0/2003413374 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457800b9c0 con 0x7f45840ccd70 2024-01-11T08:30:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.462+0000 7f45827fc700 1 -- 172.21.15.27:0/2003413374 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457800bd60 con 0x7f45840ccd70 2024-01-11T08:30:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.463+0000 7f4589fff700 1 -- 172.21.15.27:0/2003413374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45840ccd70 msgr2=0x7f45840cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.463+0000 7f4589fff700 1 --2- 172.21.15.27:0/2003413374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45840ccd70 0x7f45840cd210 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f45840d16f0 tx=0x7f4578009250 comp rx=0 tx=0).stop 2024-01-11T08:30:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.463+0000 7f4589fff700 1 -- 172.21.15.27:0/2003413374 shutdown_connections 2024-01-11T08:30:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.463+0000 7f4589fff700 1 --2- 172.21.15.27:0/2003413374 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45840685c0 0x7f4584068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.463+0000 7f4589fff700 1 --2- 172.21.15.27:0/2003413374 >> v2:172.21.15.77:3300/0 conn(0x7f458415a1b0 0x7f458415a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.463+0000 7f4589fff700 1 --2- 172.21.15.27:0/2003413374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45840ccd70 0x7f45840cd210 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.463+0000 7f4589fff700 1 -- 172.21.15.27:0/2003413374 >> 172.21.15.27:0/2003413374 conn(0x7f458406ed40 msgr2=0x7f458406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:04.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.463+0000 7f4589fff700 1 -- 172.21.15.27:0/2003413374 shutdown_connections 2024-01-11T08:30:04.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.463+0000 7f4589fff700 1 -- 172.21.15.27:0/2003413374 wait complete. 2024-01-11T08:30:04.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.465+0000 7f4589fff700 1 Processor -- start 2024-01-11T08:30:04.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.466+0000 7f4589fff700 1 -- start start 2024-01-11T08:30:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.467+0000 7f4589fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45840685c0 0x7f45840b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.467+0000 7f4589fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45840ccd70 0x7f4584148f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.467+0000 7f4589fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f458415a1b0 0x7f458415b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.467+0000 7f4589fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45840b20c0 con 0x7f45840685c0 2024-01-11T08:30:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.467+0000 7f4589fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45840c7b30 con 0x7f458415a1b0 2024-01-11T08:30:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.467+0000 7f4589fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45840b1b00 con 0x7f45840ccd70 2024-01-11T08:30:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.467+0000 7f45837fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45840685c0 0x7f45840b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.467+0000 7f45837fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45840685c0 0x7f45840b0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53650/0 (socket says 172.21.15.27:53650) 2024-01-11T08:30:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.467+0000 7f45837fe700 1 -- 172.21.15.27:0/3092157324 learned_addr learned my addr 172.21.15.27:0/3092157324 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.467+0000 7f4582ffd700 1 --2- 172.21.15.27:0/3092157324 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45840ccd70 0x7f4584148f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.467+0000 7f4583fff700 1 --2- 172.21.15.27:0/3092157324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f458415a1b0 0x7f458415b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.467+0000 7f45837fe700 1 -- 172.21.15.27:0/3092157324 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45840ccd70 msgr2=0x7f4584148f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.467+0000 7f45837fe700 1 --2- 172.21.15.27:0/3092157324 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45840ccd70 0x7f4584148f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.468+0000 7f45837fe700 1 -- 172.21.15.27:0/3092157324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f458415a1b0 msgr2=0x7f458415b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.468+0000 7f45837fe700 1 --2- 172.21.15.27:0/3092157324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f458415a1b0 0x7f458415b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.468+0000 7f45837fe700 1 -- 172.21.15.27:0/3092157324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45780094f0 con 0x7f45840685c0 2024-01-11T08:30:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.468+0000 7f45837fe700 1 --2- 172.21.15.27:0/3092157324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45840685c0 0x7f45840b0c40 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f457400e8d0 tx=0x7f4574008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.468+0000 7f456b7fe700 1 -- 172.21.15.27:0/3092157324 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574013410 con 0x7f45840685c0 2024-01-11T08:30:04.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.468+0000 7f456b7fe700 1 -- 172.21.15.27:0/3092157324 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4574013570 con 0x7f45840685c0 2024-01-11T08:30:04.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.468+0000 7f4589fff700 1 -- 172.21.15.27:0/3092157324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45840b1de0 con 0x7f45840685c0 2024-01-11T08:30:04.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.468+0000 7f456b7fe700 1 -- 172.21.15.27:0/3092157324 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574013910 con 0x7f45840685c0 2024-01-11T08:30:04.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.468+0000 7f4589fff700 1 -- 172.21.15.27:0/3092157324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45840b1180 con 0x7f45840685c0 2024-01-11T08:30:04.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.469+0000 7f4589fff700 1 -- 172.21.15.27:0/3092157324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458404c3c0 con 0x7f45840685c0 2024-01-11T08:30:04.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.470+0000 7f456b7fe700 1 -- 172.21.15.27:0/3092157324 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 77) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f4574013d40 con 0x7f45840685c0 2024-01-11T08:30:04.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.471+0000 7f456b7fe700 1 --2- 172.21.15.27:0/3092157324 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4570066ea0 0x7f4570069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.472+0000 7f456b7fe700 1 -- 172.21.15.27:0/3092157324 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f4574084120 con 0x7f45840685c0 2024-01-11T08:30:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.472+0000 7f4582ffd700 1 --2- 172.21.15.27:0/3092157324 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4570066ea0 0x7f4570069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.473+0000 7f4582ffd700 1 --2- 172.21.15.27:0/3092157324 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4570066ea0 0x7f4570069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f45840d16f0 tx=0x7f4578017000 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:04.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.476+0000 7f456b7fe700 1 -- 172.21.15.27:0/3092157324 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4574051460 con 0x7f45840685c0 2024-01-11T08:30:04.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.680+0000 7f4589fff700 1 -- 172.21.15.27:0/3092157324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f45840b2d60 con 0x7f45840685c0 2024-01-11T08:30:04.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.684+0000 7f456b7fe700 1 -- 172.21.15.27:0/3092157324 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v49) v1 ==== 120+0+0 (secure 0 0 0) 0x7f4574054a80 con 0x7f45840685c0 2024-01-11T08:30:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.687+0000 7f4589fff700 1 -- 172.21.15.27:0/3092157324 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4570066ea0 msgr2=0x7f4570069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.687+0000 7f4589fff700 1 --2- 172.21.15.27:0/3092157324 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4570066ea0 0x7f4570069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f45840d16f0 tx=0x7f4578017000 comp rx=0 tx=0).stop 2024-01-11T08:30:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.687+0000 7f4589fff700 1 -- 172.21.15.27:0/3092157324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45840685c0 msgr2=0x7f45840b0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.687+0000 7f4589fff700 1 --2- 172.21.15.27:0/3092157324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45840685c0 0x7f45840b0c40 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f457400e8d0 tx=0x7f4574008040 comp rx=0 tx=0).stop 2024-01-11T08:30:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.687+0000 7f4589fff700 1 -- 172.21.15.27:0/3092157324 shutdown_connections 2024-01-11T08:30:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.687+0000 7f4589fff700 1 --2- 172.21.15.27:0/3092157324 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4570066ea0 0x7f4570069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.687+0000 7f4589fff700 1 --2- 172.21.15.27:0/3092157324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f458415a1b0 0x7f458415b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.687+0000 7f4589fff700 1 --2- 172.21.15.27:0/3092157324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45840685c0 0x7f45840b0c40 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.687+0000 7f4589fff700 1 --2- 172.21.15.27:0/3092157324 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f45840ccd70 0x7f4584148f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.687+0000 7f4589fff700 1 -- 172.21.15.27:0/3092157324 >> 172.21.15.27:0/3092157324 conn(0x7f458406ed40 msgr2=0x7f4584159240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:04.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.687+0000 7f4589fff700 1 -- 172.21.15.27:0/3092157324 shutdown_connections 2024-01-11T08:30:04.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.687+0000 7f4589fff700 1 -- 172.21.15.27:0/3092157324 wait complete. 2024-01-11T08:30:04.703 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-11T08:30:04.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.910+0000 7f14397ff700 1 Processor -- start 2024-01-11T08:30:04.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.911+0000 7f14397ff700 1 -- start start 2024-01-11T08:30:04.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.912+0000 7f14397ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1434142dc0 0x7f14340cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:04.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.912+0000 7f14397ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1434143190 0x7f14340cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:04.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.912+0000 7f14397ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f143415d320 con 0x7f14340cc020 2024-01-11T08:30:04.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.912+0000 7f14397ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f143415df30 con 0x7f1434142dc0 2024-01-11T08:30:04.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.912+0000 7f14397ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f143415d590 con 0x7f1434143190 2024-01-11T08:30:04.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.912+0000 7f1432ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1434142dc0 0x7f14340cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:04.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.912+0000 7f1432ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1434142dc0 0x7f14340cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53662/0 (socket says 172.21.15.27:53662) 2024-01-11T08:30:04.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.912+0000 7f14337fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f14340cc020 0x7f14340cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:32782/0 (socket says 172.21.15.27:32782) 2024-01-11T08:30:04.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.912+0000 7f1432ffd700 1 -- 172.21.15.27:0/1756826606 learned_addr learned my addr 172.21.15.27:0/1756826606 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.913+0000 7f14327fc700 1 --2- 172.21.15.27:0/1756826606 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1434143190 0x7f14340cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.913+0000 7f1432ffd700 1 -- 172.21.15.27:0/1756826606 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1434143190 msgr2=0x7f14340cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.913+0000 7f1432ffd700 1 --2- 172.21.15.27:0/1756826606 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1434143190 0x7f14340cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.913+0000 7f1432ffd700 1 -- 172.21.15.27:0/1756826606 >> v1:172.21.15.27:6789/0 conn(0x7f14340cc020 legacy=0x7f14340cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.913+0000 7f1432ffd700 1 -- 172.21.15.27:0/1756826606 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143415d870 con 0x7f1434142dc0 2024-01-11T08:30:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.913+0000 7f1431ffb700 1 -- 172.21.15.27:0/1756826606 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318533815 0 0) 0x7f143415d320 con 0x7f14340cc020 2024-01-11T08:30:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.913+0000 7f1432ffd700 1 --2- 172.21.15.27:0/1756826606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1434142dc0 0x7f14340cada0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f14240099b0 tx=0x7f1424009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d26f9c2bf140f75 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.913+0000 7f1431ffb700 1 -- 172.21.15.27:0/1756826606 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142400b860 con 0x7f1434142dc0 2024-01-11T08:30:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.913+0000 7f1431ffb700 1 -- 172.21.15.27:0/1756826606 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142400b9c0 con 0x7f1434142dc0 2024-01-11T08:30:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.913+0000 7f1431ffb700 1 -- 172.21.15.27:0/1756826606 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142400bd60 con 0x7f1434142dc0 2024-01-11T08:30:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.914+0000 7f14397ff700 1 -- 172.21.15.27:0/1756826606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1434142dc0 msgr2=0x7f14340cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.914+0000 7f14397ff700 1 --2- 172.21.15.27:0/1756826606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1434142dc0 0x7f14340cada0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f14240099b0 tx=0x7f1424009250 comp rx=0 tx=0).stop 2024-01-11T08:30:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.914+0000 7f14397ff700 1 -- 172.21.15.27:0/1756826606 shutdown_connections 2024-01-11T08:30:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.914+0000 7f14397ff700 1 --2- 172.21.15.27:0/1756826606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1434142dc0 0x7f14340cada0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.914+0000 7f14397ff700 1 --2- 172.21.15.27:0/1756826606 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1434143190 0x7f14340cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.914+0000 7f14397ff700 1 -- 172.21.15.27:0/1756826606 >> 172.21.15.27:0/1756826606 conn(0x7f1434063ca0 msgr2=0x7f143406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.914+0000 7f14397ff700 1 -- 172.21.15.27:0/1756826606 shutdown_connections 2024-01-11T08:30:04.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.914+0000 7f14397ff700 1 -- 172.21.15.27:0/1756826606 wait complete. 2024-01-11T08:30:04.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.916+0000 7f14397ff700 1 Processor -- start 2024-01-11T08:30:04.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.916+0000 7f14397ff700 1 -- start start 2024-01-11T08:30:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.918+0000 7f14397ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14340cc020 0x7f14340c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.918+0000 7f14397ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1434142dc0 0x7f1434152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.918+0000 7f14397ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1434143190 0x7f1434152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.918+0000 7f14397ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14340c6eb0 con 0x7f1434143190 2024-01-11T08:30:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.918+0000 7f14397ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14340c62a0 con 0x7f14340cc020 2024-01-11T08:30:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.918+0000 7f14397ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14340c6580 con 0x7f1434142dc0 2024-01-11T08:30:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.918+0000 7f14337fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1434143190 0x7f1434152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.918+0000 7f14337fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1434143190 0x7f1434152b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53664/0 (socket says 172.21.15.27:53664) 2024-01-11T08:30:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.918+0000 7f14337fe700 1 -- 172.21.15.27:0/2863362060 learned_addr learned my addr 172.21.15.27:0/2863362060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.918+0000 7f14337fe700 1 -- 172.21.15.27:0/2863362060 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1434142dc0 msgr2=0x7f1434152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.918+0000 7f14327fc700 1 --2- 172.21.15.27:0/2863362060 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1434142dc0 0x7f1434152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.918+0000 7f1432ffd700 1 --2- 172.21.15.27:0/2863362060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14340cc020 0x7f14340c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.918+0000 7f14337fe700 1 --2- 172.21.15.27:0/2863362060 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1434142dc0 0x7f1434152650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.919+0000 7f14337fe700 1 -- 172.21.15.27:0/2863362060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14340cc020 msgr2=0x7f14340c8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.919+0000 7f14337fe700 1 --2- 172.21.15.27:0/2863362060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14340cc020 0x7f14340c8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.919+0000 7f14337fe700 1 -- 172.21.15.27:0/2863362060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14240094f0 con 0x7f1434143190 2024-01-11T08:30:04.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.919+0000 7f14327fc700 1 --2- 172.21.15.27:0/2863362060 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1434142dc0 0x7f1434152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:30:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.919+0000 7f1432ffd700 1 --2- 172.21.15.27:0/2863362060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14340cc020 0x7f14340c8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:30:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.919+0000 7f14337fe700 1 --2- 172.21.15.27:0/2863362060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1434143190 0x7f1434152b90 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f142c00cd50 tx=0x7f142c001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.919+0000 7f141affd700 1 -- 172.21.15.27:0/2863362060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142c005df0 con 0x7f1434143190 2024-01-11T08:30:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.919+0000 7f141affd700 1 -- 172.21.15.27:0/2863362060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142c00a960 con 0x7f1434143190 2024-01-11T08:30:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.919+0000 7f14397ff700 1 -- 172.21.15.27:0/2863362060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14340c6860 con 0x7f1434143190 2024-01-11T08:30:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.919+0000 7f141affd700 1 -- 172.21.15.27:0/2863362060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142c00ad20 con 0x7f1434143190 2024-01-11T08:30:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.919+0000 7f14397ff700 1 -- 172.21.15.27:0/2863362060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14340c9360 con 0x7f1434143190 2024-01-11T08:30:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.920+0000 7f14397ff700 1 -- 172.21.15.27:0/2863362060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f143404c3c0 con 0x7f1434143190 2024-01-11T08:30:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.921+0000 7f141affd700 1 -- 172.21.15.27:0/2863362060 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 77) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f142c007300 con 0x7f1434143190 2024-01-11T08:30:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.922+0000 7f141affd700 1 --2- 172.21.15.27:0/2863362060 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1420066ea0 0x7f1420069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.922+0000 7f141affd700 1 -- 172.21.15.27:0/2863362060 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f142c0845d0 con 0x7f1434143190 2024-01-11T08:30:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.923+0000 7f1432ffd700 1 --2- 172.21.15.27:0/2863362060 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1420066ea0 0x7f1420069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:04.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.923+0000 7f1432ffd700 1 --2- 172.21.15.27:0/2863362060 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1420066ea0 0x7f1420069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f14240099b0 tx=0x7f1424017000 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:04.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:04.927+0000 7f141affd700 1 -- 172.21.15.27:0/2863362060 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f142c051990 con 0x7f1434143190 2024-01-11T08:30:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.134+0000 7f14397ff700 1 -- 172.21.15.27:0/2863362060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f143415db50 con 0x7f1434143190 2024-01-11T08:30:05.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.138+0000 7f141affd700 1 -- 172.21.15.27:0/2863362060 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v50) v1 ==== 115+0+0 (secure 0 0 0) 0x7f142c054fb0 con 0x7f1434143190 2024-01-11T08:30:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.141+0000 7f14397ff700 1 -- 172.21.15.27:0/2863362060 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1420066ea0 msgr2=0x7f1420069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.141+0000 7f14397ff700 1 --2- 172.21.15.27:0/2863362060 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1420066ea0 0x7f1420069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f14240099b0 tx=0x7f1424017000 comp rx=0 tx=0).stop 2024-01-11T08:30:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.141+0000 7f14397ff700 1 -- 172.21.15.27:0/2863362060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1434143190 msgr2=0x7f1434152b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.141+0000 7f14397ff700 1 --2- 172.21.15.27:0/2863362060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1434143190 0x7f1434152b90 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f142c00cd50 tx=0x7f142c001b10 comp rx=0 tx=0).stop 2024-01-11T08:30:05.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.141+0000 7f14397ff700 1 -- 172.21.15.27:0/2863362060 shutdown_connections 2024-01-11T08:30:05.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.141+0000 7f14397ff700 1 --2- 172.21.15.27:0/2863362060 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1420066ea0 0x7f1420069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:05.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.141+0000 7f14397ff700 1 --2- 172.21.15.27:0/2863362060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1434143190 0x7f1434152b90 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:05.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.141+0000 7f14397ff700 1 --2- 172.21.15.27:0/2863362060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14340cc020 0x7f14340c8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:05.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.141+0000 7f14397ff700 1 --2- 172.21.15.27:0/2863362060 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1434142dc0 0x7f1434152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:05.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.141+0000 7f14397ff700 1 -- 172.21.15.27:0/2863362060 >> 172.21.15.27:0/2863362060 conn(0x7f1434063ca0 msgr2=0x7f14340c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:05.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.141+0000 7f14397ff700 1 -- 172.21.15.27:0/2863362060 shutdown_connections 2024-01-11T08:30:05.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.141+0000 7f14397ff700 1 -- 172.21.15.27:0/2863362060 wait complete. 2024-01-11T08:30:05.159 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-11T08:30:05.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.362+0000 7f35b61ff700 1 Processor -- start 2024-01-11T08:30:05.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.362+0000 7f35b61ff700 1 -- start start 2024-01-11T08:30:05.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.363+0000 7f35b61ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35b013ff80 0x7f35b015a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:05.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.363+0000 7f35b61ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b00c6b30 con 0x7f35b015a560 2024-01-11T08:30:05.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.363+0000 7f35b61ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b00c6800 con 0x7f35b013fbb0 2024-01-11T08:30:05.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.363+0000 7f35b61ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b015a960 con 0x7f35b013ff80 2024-01-11T08:30:05.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.363+0000 7f35aeffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35b013ff80 0x7f35b015a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:05.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.363+0000 7f35affff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f35b015a560 0x7f35b00c6fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:32788/0 (socket says 172.21.15.27:32788) 2024-01-11T08:30:05.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.363+0000 7f35affff700 1 -- 172.21.15.27:0/860886073 learned_addr learned my addr 172.21.15.27:0/860886073 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:30:05.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.363+0000 7f35aeffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35b013ff80 0x7f35b015a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53668/0 (socket says 172.21.15.27:53668) 2024-01-11T08:30:05.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.364+0000 7f35aeffd700 1 -- 172.21.15.27:0/860886073 >> v1:172.21.15.77:6789/0 conn(0x7f35b013fbb0 legacy=0x7f35b00ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.364+0000 7f35aeffd700 1 -- 172.21.15.27:0/860886073 >> v1:172.21.15.27:6789/0 conn(0x7f35b015a560 legacy=0x7f35b00c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.364+0000 7f35aeffd700 1 -- 172.21.15.27:0/860886073 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b00cb240 con 0x7f35b013ff80 2024-01-11T08:30:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.364+0000 7f35ae7fc700 1 -- 172.21.15.27:0/860886073 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609348488 0 0) 0x7f35b00c6b30 con 0x7f35b015a560 2024-01-11T08:30:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.364+0000 7f35aeffd700 1 --2- 172.21.15.27:0/860886073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b013ff80 0x7f35b015a020 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f35a400bb90 tx=0x7f35a400b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11d62fd9c29083b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.364+0000 7f35ae7fc700 1 -- 172.21.15.27:0/860886073 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a400dba0 con 0x7f35b013ff80 2024-01-11T08:30:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.364+0000 7f35ae7fc700 1 -- 172.21.15.27:0/860886073 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a400dd00 con 0x7f35b013ff80 2024-01-11T08:30:05.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.364+0000 7f35ae7fc700 1 -- 172.21.15.27:0/860886073 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a40196b0 con 0x7f35b013ff80 2024-01-11T08:30:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.365+0000 7f35b61ff700 1 -- 172.21.15.27:0/860886073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b013ff80 msgr2=0x7f35b015a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.365+0000 7f35b61ff700 1 --2- 172.21.15.27:0/860886073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b013ff80 0x7f35b015a020 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f35a400bb90 tx=0x7f35a400b430 comp rx=0 tx=0).stop 2024-01-11T08:30:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.365+0000 7f35b61ff700 1 -- 172.21.15.27:0/860886073 shutdown_connections 2024-01-11T08:30:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.365+0000 7f35b61ff700 1 --2- 172.21.15.27:0/860886073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b013ff80 0x7f35b015a020 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.365+0000 7f35b61ff700 1 -- 172.21.15.27:0/860886073 >> 172.21.15.27:0/860886073 conn(0x7f35b00680a0 msgr2=0x7f35b00cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.365+0000 7f35b61ff700 1 -- 172.21.15.27:0/860886073 shutdown_connections 2024-01-11T08:30:05.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.365+0000 7f35b61ff700 1 -- 172.21.15.27:0/860886073 wait complete. 2024-01-11T08:30:05.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.367+0000 7f35b61ff700 1 Processor -- start 2024-01-11T08:30:05.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.367+0000 7f35b61ff700 1 -- start start 2024-01-11T08:30:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.368+0000 7f35b61ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b013fbb0 0x7f35b016e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.368+0000 7f35b61ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35b013ff80 0x7f35b0157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.368+0000 7f35b61ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35b015a560 0x7f35b0155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.368+0000 7f35b61ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b0159080 con 0x7f35b013fbb0 2024-01-11T08:30:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.368+0000 7f35b61ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b0158070 con 0x7f35b013ff80 2024-01-11T08:30:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.368+0000 7f35b61ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b0159510 con 0x7f35b015a560 2024-01-11T08:30:05.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.368+0000 7f35af7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b013fbb0 0x7f35b016e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.368+0000 7f35af7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b013fbb0 0x7f35b016e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53674/0 (socket says 172.21.15.27:53674) 2024-01-11T08:30:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.368+0000 7f35af7fe700 1 -- 172.21.15.27:0/4248541783 learned_addr learned my addr 172.21.15.27:0/4248541783 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.368+0000 7f35aeffd700 1 --2- 172.21.15.27:0/4248541783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35b013ff80 0x7f35b0157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.368+0000 7f35affff700 1 --2- 172.21.15.27:0/4248541783 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35b015a560 0x7f35b0155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.369+0000 7f35af7fe700 1 -- 172.21.15.27:0/4248541783 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35b015a560 msgr2=0x7f35b0155df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.369+0000 7f35af7fe700 1 --2- 172.21.15.27:0/4248541783 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35b015a560 0x7f35b0155df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.369+0000 7f35af7fe700 1 -- 172.21.15.27:0/4248541783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35b013ff80 msgr2=0x7f35b0157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.369+0000 7f35af7fe700 1 --2- 172.21.15.27:0/4248541783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35b013ff80 0x7f35b0157b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.369+0000 7f35af7fe700 1 -- 172.21.15.27:0/4248541783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a400b6d0 con 0x7f35b013fbb0 2024-01-11T08:30:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.369+0000 7f35affff700 1 --2- 172.21.15.27:0/4248541783 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35b015a560 0x7f35b0155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:30:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.369+0000 7f35aeffd700 1 --2- 172.21.15.27:0/4248541783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35b013ff80 0x7f35b0157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:30:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.369+0000 7f35af7fe700 1 --2- 172.21.15.27:0/4248541783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b013fbb0 0x7f35b016e170 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f35a000c7c0 tx=0x7f35a0007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.369+0000 7f35977fe700 1 -- 172.21.15.27:0/4248541783 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a0011590 con 0x7f35b013fbb0 2024-01-11T08:30:05.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.369+0000 7f35977fe700 1 -- 172.21.15.27:0/4248541783 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a0011710 con 0x7f35b013fbb0 2024-01-11T08:30:05.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.369+0000 7f35b61ff700 1 -- 172.21.15.27:0/4248541783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b0158470 con 0x7f35b013fbb0 2024-01-11T08:30:05.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.369+0000 7f35977fe700 1 -- 172.21.15.27:0/4248541783 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a0011ad0 con 0x7f35b013fbb0 2024-01-11T08:30:05.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.369+0000 7f35b61ff700 1 -- 172.21.15.27:0/4248541783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b0158a30 con 0x7f35b013fbb0 2024-01-11T08:30:05.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.370+0000 7f35b61ff700 1 -- 172.21.15.27:0/4248541783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b0159360 con 0x7f35b013fbb0 2024-01-11T08:30:05.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.371+0000 7f35977fe700 1 -- 172.21.15.27:0/4248541783 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 77) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f35a001b3b0 con 0x7f35b013fbb0 2024-01-11T08:30:05.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.372+0000 7f35977fe700 1 --2- 172.21.15.27:0/4248541783 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f359c066ea0 0x7f359c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:05.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.372+0000 7f35977fe700 1 -- 172.21.15.27:0/4248541783 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f35a0083880 con 0x7f35b013fbb0 2024-01-11T08:30:05.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.373+0000 7f35aeffd700 1 --2- 172.21.15.27:0/4248541783 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f359c066ea0 0x7f359c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:05.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.373+0000 7f35aeffd700 1 --2- 172.21.15.27:0/4248541783 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f359c066ea0 0x7f359c069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f35a400b6a0 tx=0x7f35a401b000 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:05.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.376+0000 7f35977fe700 1 -- 172.21.15.27:0/4248541783 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f35a0050d10 con 0x7f35b013fbb0 2024-01-11T08:30:05.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.581+0000 7f35b61ff700 1 -- 172.21.15.27:0/4248541783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f35b004c3c0 con 0x7f35b013fbb0 2024-01-11T08:30:05.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.586+0000 7f35977fe700 1 -- 172.21.15.27:0/4248541783 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v51) v1 ==== 104+0+0 (secure 0 0 0) 0x7f35a0017020 con 0x7f35b013fbb0 2024-01-11T08:30:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.588+0000 7f35b61ff700 1 -- 172.21.15.27:0/4248541783 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f359c066ea0 msgr2=0x7f359c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.588+0000 7f35b61ff700 1 --2- 172.21.15.27:0/4248541783 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f359c066ea0 0x7f359c069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f35a400b6a0 tx=0x7f35a401b000 comp rx=0 tx=0).stop 2024-01-11T08:30:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.588+0000 7f35b61ff700 1 -- 172.21.15.27:0/4248541783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b013fbb0 msgr2=0x7f35b016e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.588+0000 7f35b61ff700 1 --2- 172.21.15.27:0/4248541783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b013fbb0 0x7f35b016e170 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f35a000c7c0 tx=0x7f35a0007f70 comp rx=0 tx=0).stop 2024-01-11T08:30:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.588+0000 7f35b61ff700 1 -- 172.21.15.27:0/4248541783 shutdown_connections 2024-01-11T08:30:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.588+0000 7f35b61ff700 1 --2- 172.21.15.27:0/4248541783 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f359c066ea0 0x7f359c069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.588+0000 7f35b61ff700 1 --2- 172.21.15.27:0/4248541783 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35b015a560 0x7f35b0155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.588+0000 7f35b61ff700 1 --2- 172.21.15.27:0/4248541783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b013fbb0 0x7f35b016e170 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:05.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.588+0000 7f35b61ff700 1 --2- 172.21.15.27:0/4248541783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35b013ff80 0x7f35b0157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:05.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.588+0000 7f35b61ff700 1 -- 172.21.15.27:0/4248541783 >> 172.21.15.27:0/4248541783 conn(0x7f35b00680a0 msgr2=0x7f35b00c7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:05.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.588+0000 7f35b61ff700 1 -- 172.21.15.27:0/4248541783 shutdown_connections 2024-01-11T08:30:05.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.588+0000 7f35b61ff700 1 -- 172.21.15.27:0/4248541783 wait complete. 2024-01-11T08:30:05.605 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2024-01-11T08:30:05.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.806+0000 7f7a621ff700 1 Processor -- start 2024-01-11T08:30:05.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.807+0000 7f7a621ff700 1 -- start start 2024-01-11T08:30:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.808+0000 7f7a621ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7a5c0ca3e0 0x7f7a5c0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.808+0000 7f7a621ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a5c159e80 0x7f7a5c15dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.808+0000 7f7a621ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5c15d640 con 0x7f7a5c0ca7b0 2024-01-11T08:30:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.808+0000 7f7a621ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a5c15ae70 con 0x7f7a5c0ca3e0 2024-01-11T08:30:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.808+0000 7f7a621ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5c0cc4f0 con 0x7f7a5c159e80 2024-01-11T08:30:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.809+0000 7f7a5b7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7a5c0ca3e0 0x7f7a5c0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.809+0000 7f7a5affd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7a5c0ca7b0 0x7f7a5c0c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:55328/0 (socket says 172.21.15.27:55328) 2024-01-11T08:30:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.809+0000 7f7a5affd700 1 -- 172.21.15.27:0/3914316652 learned_addr learned my addr 172.21.15.27:0/3914316652 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:30:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.809+0000 7f7a5bfff700 1 --2- 172.21.15.27:0/3914316652 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a5c159e80 0x7f7a5c15dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.809+0000 7f7a5a7fc700 1 -- 172.21.15.27:0/3914316652 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606540334 0 0) 0x7f7a5c15d640 con 0x7f7a5c0ca7b0 2024-01-11T08:30:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.809+0000 7f7a5a7fc700 1 -- 172.21.15.27:0/3914316652 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a44003570 con 0x7f7a5c0ca7b0 2024-01-11T08:30:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.809+0000 7f7a5bfff700 1 -- 172.21.15.27:0/3914316652 >> v2:172.21.15.77:3300/0 conn(0x7f7a5c0ca3e0 msgr2=0x7f7a5c0cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.809+0000 7f7a5bfff700 1 --2- 172.21.15.27:0/3914316652 >> v2:172.21.15.77:3300/0 conn(0x7f7a5c0ca3e0 0x7f7a5c0cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.809+0000 7f7a5bfff700 1 -- 172.21.15.27:0/3914316652 >> v1:172.21.15.77:6789/0 conn(0x7f7a5c0ca7b0 legacy=0x7f7a5c0c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.809+0000 7f7a5bfff700 1 -- 172.21.15.27:0/3914316652 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5c15a5d0 con 0x7f7a5c159e80 2024-01-11T08:30:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.809+0000 7f7a5b7fe700 1 --2- 172.21.15.27:0/3914316652 >> v2:172.21.15.77:3300/0 conn(0x7f7a5c0ca3e0 0x7f7a5c0cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:30:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.810+0000 7f7a5bfff700 1 --2- 172.21.15.27:0/3914316652 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a5c159e80 0x7f7a5c15dad0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7a54009e90 tx=0x7f7a54009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd09bccf4071e3a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.810+0000 7f7a5a7fc700 1 -- 172.21.15.27:0/3914316652 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5400e070 con 0x7f7a5c159e80 2024-01-11T08:30:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.810+0000 7f7a5a7fc700 1 -- 172.21.15.27:0/3914316652 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a5400bb40 con 0x7f7a5c159e80 2024-01-11T08:30:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.810+0000 7f7a5a7fc700 1 -- 172.21.15.27:0/3914316652 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a54012430 con 0x7f7a5c159e80 2024-01-11T08:30:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.810+0000 7f7a621ff700 1 -- 172.21.15.27:0/3914316652 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a5c159e80 msgr2=0x7f7a5c15dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.810+0000 7f7a621ff700 1 --2- 172.21.15.27:0/3914316652 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a5c159e80 0x7f7a5c15dad0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7a54009e90 tx=0x7f7a54009780 comp rx=0 tx=0).stop 2024-01-11T08:30:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.811+0000 7f7a621ff700 1 -- 172.21.15.27:0/3914316652 shutdown_connections 2024-01-11T08:30:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.811+0000 7f7a621ff700 1 --2- 172.21.15.27:0/3914316652 >> v2:172.21.15.77:3300/0 conn(0x7f7a5c0ca3e0 0x7f7a5c0cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.811+0000 7f7a621ff700 1 --2- 172.21.15.27:0/3914316652 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a5c159e80 0x7f7a5c15dad0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.811+0000 7f7a621ff700 1 -- 172.21.15.27:0/3914316652 >> 172.21.15.27:0/3914316652 conn(0x7f7a5c0ce3e0 msgr2=0x7f7a5c0ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.811+0000 7f7a621ff700 1 -- 172.21.15.27:0/3914316652 shutdown_connections 2024-01-11T08:30:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.811+0000 7f7a621ff700 1 -- 172.21.15.27:0/3914316652 wait complete. 2024-01-11T08:30:05.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.812+0000 7f7a621ff700 1 Processor -- start 2024-01-11T08:30:05.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.812+0000 7f7a621ff700 1 -- start start 2024-01-11T08:30:05.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.813+0000 7f7a621ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a5c0ca3e0 0x7f7a5c0c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:05.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.813+0000 7f7a621ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a5c0ca7b0 0x7f7a5c0c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:05.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.813+0000 7f7a621ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a5c159e80 0x7f7a5c16cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:05.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.813+0000 7f7a621ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5c140650 con 0x7f7a5c0ca7b0 2024-01-11T08:30:05.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.813+0000 7f7a621ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5c0c6740 con 0x7f7a5c159e80 2024-01-11T08:30:05.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.813+0000 7f7a621ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5c0c6a20 con 0x7f7a5c0ca3e0 2024-01-11T08:30:05.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.814+0000 7f7a5affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a5c0ca7b0 0x7f7a5c0c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:05.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.814+0000 7f7a5affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a5c0ca7b0 0x7f7a5c0c8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53678/0 (socket says 172.21.15.27:53678) 2024-01-11T08:30:05.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.814+0000 7f7a5affd700 1 -- 172.21.15.27:0/3594380663 learned_addr learned my addr 172.21.15.27:0/3594380663 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:05.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.814+0000 7f7a5b7fe700 1 --2- 172.21.15.27:0/3594380663 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a5c0ca3e0 0x7f7a5c0c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:05.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.814+0000 7f7a5bfff700 1 --2- 172.21.15.27:0/3594380663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a5c159e80 0x7f7a5c16cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:05.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.814+0000 7f7a5b7fe700 1 -- 172.21.15.27:0/3594380663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a5c159e80 msgr2=0x7f7a5c16cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:05.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.814+0000 7f7a5b7fe700 1 --2- 172.21.15.27:0/3594380663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a5c159e80 0x7f7a5c16cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:05.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.814+0000 7f7a5b7fe700 1 -- 172.21.15.27:0/3594380663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a5c0ca7b0 msgr2=0x7f7a5c0c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:05.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.814+0000 7f7a5b7fe700 1 --2- 172.21.15.27:0/3594380663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a5c0ca7b0 0x7f7a5c0c8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:05.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.814+0000 7f7a5b7fe700 1 -- 172.21.15.27:0/3594380663 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c009420 con 0x7f7a5c0ca3e0 2024-01-11T08:30:05.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.814+0000 7f7a5affd700 1 --2- 172.21.15.27:0/3594380663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a5c0ca7b0 0x7f7a5c0c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:30:05.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.815+0000 7f7a5b7fe700 1 --2- 172.21.15.27:0/3594380663 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a5c0ca3e0 0x7f7a5c0c60b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c00ec80 tx=0x7f7a4c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:05.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.815+0000 7f7a437fe700 1 -- 172.21.15.27:0/3594380663 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c00c850 con 0x7f7a5c0ca3e0 2024-01-11T08:30:05.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.815+0000 7f7a437fe700 1 -- 172.21.15.27:0/3594380663 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a4c00c9b0 con 0x7f7a5c0ca3e0 2024-01-11T08:30:05.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.815+0000 7f7a437fe700 1 -- 172.21.15.27:0/3594380663 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c00cd70 con 0x7f7a5c0ca3e0 2024-01-11T08:30:05.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.815+0000 7f7a621ff700 1 -- 172.21.15.27:0/3594380663 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a54009a20 con 0x7f7a5c0ca3e0 2024-01-11T08:30:05.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.815+0000 7f7a621ff700 1 -- 172.21.15.27:0/3594380663 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a5c0c6fe0 con 0x7f7a5c0ca3e0 2024-01-11T08:30:05.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.816+0000 7f7a621ff700 1 -- 172.21.15.27:0/3594380663 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a5c0718d0 con 0x7f7a5c0ca3e0 2024-01-11T08:30:05.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.817+0000 7f7a437fe700 1 -- 172.21.15.27:0/3594380663 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 77) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f7a4c013680 con 0x7f7a5c0ca3e0 2024-01-11T08:30:05.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.818+0000 7f7a437fe700 1 --2- 172.21.15.27:0/3594380663 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7a48066ea0 0x7f7a48069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:05.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.818+0000 7f7a437fe700 1 -- 172.21.15.27:0/3594380663 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f7a4c084640 con 0x7f7a5c0ca3e0 2024-01-11T08:30:05.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.818+0000 7f7a5affd700 1 --2- 172.21.15.27:0/3594380663 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7a48066ea0 0x7f7a48069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:05.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.819+0000 7f7a5affd700 1 --2- 172.21.15.27:0/3594380663 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7a48066ea0 0x7f7a48069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c16ce60 tx=0x7f7a50009040 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:05.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:05.823+0000 7f7a437fe700 1 -- 172.21.15.27:0/3594380663 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7a4c051980 con 0x7f7a5c0ca3e0 2024-01-11T08:30:06.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:06.108+0000 7f7a621ff700 1 -- 172.21.15.27:0/3594380663 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7f7a5c15a8b0 con 0x7f7a5c0ca3e0 2024-01-11T08:30:06.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:06.155+0000 7f7a437fe700 1 -- 172.21.15.27:0/3594380663 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7a4c051d50 con 0x7f7a5c0ca3e0 2024-01-11T08:30:07.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.074+0000 7f7a437fe700 1 -- 172.21.15.27:0/3594380663 <== mon.2 v2:172.21.15.77:3301/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v753) v1 ==== 160+0+0 (secure 0 0 0) 0x7f7a4c055410 con 0x7f7a5c0ca3e0 2024-01-11T08:30:07.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.077+0000 7f7a621ff700 1 -- 172.21.15.27:0/3594380663 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7a48066ea0 msgr2=0x7f7a48069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:07.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.077+0000 7f7a621ff700 1 --2- 172.21.15.27:0/3594380663 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7a48066ea0 0x7f7a48069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c16ce60 tx=0x7f7a50009040 comp rx=0 tx=0).stop 2024-01-11T08:30:07.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.077+0000 7f7a621ff700 1 -- 172.21.15.27:0/3594380663 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a5c0ca3e0 msgr2=0x7f7a5c0c60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:07.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.077+0000 7f7a621ff700 1 --2- 172.21.15.27:0/3594380663 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a5c0ca3e0 0x7f7a5c0c60b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c00ec80 tx=0x7f7a4c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:30:07.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.077+0000 7f7a621ff700 1 -- 172.21.15.27:0/3594380663 shutdown_connections 2024-01-11T08:30:07.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.077+0000 7f7a621ff700 1 --2- 172.21.15.27:0/3594380663 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7a48066ea0 0x7f7a48069350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:07.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.077+0000 7f7a621ff700 1 --2- 172.21.15.27:0/3594380663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a5c159e80 0x7f7a5c16cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:07.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.077+0000 7f7a621ff700 1 --2- 172.21.15.27:0/3594380663 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a5c0ca3e0 0x7f7a5c0c60b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:07.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.077+0000 7f7a621ff700 1 --2- 172.21.15.27:0/3594380663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a5c0ca7b0 0x7f7a5c0c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:07.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.077+0000 7f7a621ff700 1 -- 172.21.15.27:0/3594380663 >> 172.21.15.27:0/3594380663 conn(0x7f7a5c0ce3e0 msgr2=0x7f7a5c06d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:07.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.077+0000 7f7a621ff700 1 -- 172.21.15.27:0/3594380663 shutdown_connections 2024-01-11T08:30:07.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.077+0000 7f7a621ff700 1 -- 172.21.15.27:0/3594380663 wait complete. 2024-01-11T08:30:07.097 INFO:tasks.cephfs_test_runner:test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) ... ok 2024-01-11T08:30:07.097 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that number of snaps retained are as per user spec. 2024-01-11T08:30:07.097 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-11T08:30:07.097 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2024-01-11T08:30:07.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.302+0000 7f277cfff700 1 Processor -- start 2024-01-11T08:30:07.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.302+0000 7f277cfff700 1 -- start start 2024-01-11T08:30:07.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.303+0000 7f277cfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f27780b19d0 0x7f2778158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:07.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.303+0000 7f277cfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f277806b610 0x7f2778159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:07.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.303+0000 7f277cfff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2778158b60 con 0x7f27781575e0 2024-01-11T08:30:07.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.303+0000 7f277cfff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27780cb5e0 con 0x7f27780b19d0 2024-01-11T08:30:07.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.303+0000 7f277cfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27781599b0 con 0x7f277806b610 2024-01-11T08:30:07.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.303+0000 7f2775d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f27780b19d0 0x7f2778158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:07.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.303+0000 7f2775d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f27780b19d0 0x7f2778158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53682/0 (socket says 172.21.15.27:53682) 2024-01-11T08:30:07.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.303+0000 7f2775d9b700 1 -- 172.21.15.27:0/1856771932 learned_addr learned my addr 172.21.15.27:0/1856771932 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:07.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.304+0000 7f2776d9d700 1 --2- 172.21.15.27:0/1856771932 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f277806b610 0x7f2778159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:07.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.304+0000 7f2775d9b700 1 -- 172.21.15.27:0/1856771932 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f277806b610 msgr2=0x7f2778159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:07.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.304+0000 7f2775d9b700 1 --2- 172.21.15.27:0/1856771932 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f277806b610 0x7f2778159e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:07.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.304+0000 7f2775d9b700 1 -- 172.21.15.27:0/1856771932 >> v1:172.21.15.77:6789/0 conn(0x7f27781575e0 legacy=0x7f27780b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:07.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.304+0000 7f2775d9b700 1 -- 172.21.15.27:0/1856771932 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27780c0f90 con 0x7f27780b19d0 2024-01-11T08:30:07.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.304+0000 7f2775d9b700 1 --2- 172.21.15.27:0/1856771932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27780b19d0 0x7f2778158620 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f276c0099b0 tx=0x7f276c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20cd35f9fdf12d36 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:07.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.304+0000 7f277559a700 1 -- 172.21.15.27:0/1856771932 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c00b860 con 0x7f27780b19d0 2024-01-11T08:30:07.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.305+0000 7f277559a700 1 -- 172.21.15.27:0/1856771932 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276c00b9c0 con 0x7f27780b19d0 2024-01-11T08:30:07.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.305+0000 7f277559a700 1 -- 172.21.15.27:0/1856771932 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c00bd60 con 0x7f27780b19d0 2024-01-11T08:30:07.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.305+0000 7f277cfff700 1 -- 172.21.15.27:0/1856771932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27780b19d0 msgr2=0x7f2778158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:07.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.305+0000 7f277cfff700 1 --2- 172.21.15.27:0/1856771932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27780b19d0 0x7f2778158620 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f276c0099b0 tx=0x7f276c009250 comp rx=0 tx=0).stop 2024-01-11T08:30:07.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.305+0000 7f277cfff700 1 -- 172.21.15.27:0/1856771932 shutdown_connections 2024-01-11T08:30:07.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.305+0000 7f277cfff700 1 --2- 172.21.15.27:0/1856771932 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f277806b610 0x7f2778159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:07.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.305+0000 7f277cfff700 1 --2- 172.21.15.27:0/1856771932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27780b19d0 0x7f2778158620 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:07.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.305+0000 7f277cfff700 1 -- 172.21.15.27:0/1856771932 >> 172.21.15.27:0/1856771932 conn(0x7f27780680e0 msgr2=0x7f2778069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:07.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.305+0000 7f277cfff700 1 -- 172.21.15.27:0/1856771932 shutdown_connections 2024-01-11T08:30:07.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.305+0000 7f277cfff700 1 -- 172.21.15.27:0/1856771932 wait complete. 2024-01-11T08:30:07.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.307+0000 7f277cfff700 1 Processor -- start 2024-01-11T08:30:07.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.307+0000 7f277cfff700 1 -- start start 2024-01-11T08:30:07.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.308+0000 7f277cfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f277806b610 0x7f277816af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:07.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.308+0000 7f277cfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27780b19d0 0x7f27781538b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:07.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.308+0000 7f277cfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27781575e0 0x7f27780c8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:07.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.308+0000 7f277cfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277816ad70 con 0x7f27780b19d0 2024-01-11T08:30:07.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.308+0000 7f277cfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277816a160 con 0x7f27781575e0 2024-01-11T08:30:07.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.308+0000 7f277cfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f277816a440 con 0x7f277806b610 2024-01-11T08:30:07.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.308+0000 7f2775d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27780b19d0 0x7f27781538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:07.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.308+0000 7f2775d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27780b19d0 0x7f27781538b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53692/0 (socket says 172.21.15.27:53692) 2024-01-11T08:30:07.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.308+0000 7f2775d9b700 1 -- 172.21.15.27:0/833057004 learned_addr learned my addr 172.21.15.27:0/833057004 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:07.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.308+0000 7f277659c700 1 --2- 172.21.15.27:0/833057004 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f277806b610 0x7f277816af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:07.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.308+0000 7f2776d9d700 1 --2- 172.21.15.27:0/833057004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27781575e0 0x7f27780c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:07.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.309+0000 7f2775d9b700 1 -- 172.21.15.27:0/833057004 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f277806b610 msgr2=0x7f277816af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:07.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.309+0000 7f2775d9b700 1 --2- 172.21.15.27:0/833057004 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f277806b610 0x7f277816af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:07.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.309+0000 7f2775d9b700 1 -- 172.21.15.27:0/833057004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27781575e0 msgr2=0x7f27780c8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:07.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.309+0000 7f2775d9b700 1 --2- 172.21.15.27:0/833057004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27781575e0 0x7f27780c8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:07.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.309+0000 7f2775d9b700 1 -- 172.21.15.27:0/833057004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f276c0094f0 con 0x7f27780b19d0 2024-01-11T08:30:07.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.309+0000 7f2775d9b700 1 --2- 172.21.15.27:0/833057004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27780b19d0 0x7f27781538b0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f2778066180 tx=0x7f276c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:07.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.309+0000 7f27667fc700 1 -- 172.21.15.27:0/833057004 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c011e40 con 0x7f27780b19d0 2024-01-11T08:30:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.309+0000 7f27667fc700 1 -- 172.21.15.27:0/833057004 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276c024440 con 0x7f27780b19d0 2024-01-11T08:30:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.309+0000 7f27667fc700 1 -- 172.21.15.27:0/833057004 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c0247e0 con 0x7f27780b19d0 2024-01-11T08:30:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.309+0000 7f277cfff700 1 -- 172.21.15.27:0/833057004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f277816a720 con 0x7f27780b19d0 2024-01-11T08:30:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.309+0000 7f277cfff700 1 -- 172.21.15.27:0/833057004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2778153df0 con 0x7f27780b19d0 2024-01-11T08:30:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.310+0000 7f277cfff700 1 -- 172.21.15.27:0/833057004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27780718d0 con 0x7f27780b19d0 2024-01-11T08:30:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.311+0000 7f27667fc700 1 -- 172.21.15.27:0/833057004 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f276c02d440 con 0x7f27780b19d0 2024-01-11T08:30:07.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.313+0000 7f27667fc700 1 --2- 172.21.15.27:0/833057004 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2758066f30 0x7f27580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:07.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.313+0000 7f27667fc700 1 -- 172.21.15.27:0/833057004 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f276c085e30 con 0x7f27780b19d0 2024-01-11T08:30:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.313+0000 7f277659c700 1 --2- 172.21.15.27:0/833057004 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2758066f30 0x7f27580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.314+0000 7f277659c700 1 --2- 172.21.15.27:0/833057004 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2758066f30 0x7f27580693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f277815e9c0 tx=0x7f2768009040 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:07.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.316+0000 7f27667fc700 1 -- 172.21.15.27:0/833057004 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f276c053110 con 0x7f27780b19d0 2024-01-11T08:30:07.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:07.599+0000 7f277cfff700 1 -- 172.21.15.27:0/833057004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f2778158da0 con 0x7f27780b19d0 2024-01-11T08:30:08.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.080+0000 7f27667fc700 1 -- 172.21.15.27:0/833057004 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v754) v1 ==== 181+0+0 (secure 0 0 0) 0x7f276c056730 con 0x7f27780b19d0 2024-01-11T08:30:08.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.083+0000 7f277cfff700 1 -- 172.21.15.27:0/833057004 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2758066f30 msgr2=0x7f27580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:08.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.083+0000 7f277cfff700 1 --2- 172.21.15.27:0/833057004 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2758066f30 0x7f27580693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f277815e9c0 tx=0x7f2768009040 comp rx=0 tx=0).stop 2024-01-11T08:30:08.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.083+0000 7f277cfff700 1 -- 172.21.15.27:0/833057004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27780b19d0 msgr2=0x7f27781538b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:08.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.083+0000 7f277cfff700 1 --2- 172.21.15.27:0/833057004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27780b19d0 0x7f27781538b0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f2778066180 tx=0x7f276c017000 comp rx=0 tx=0).stop 2024-01-11T08:30:08.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.083+0000 7f277cfff700 1 -- 172.21.15.27:0/833057004 shutdown_connections 2024-01-11T08:30:08.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.083+0000 7f277cfff700 1 --2- 172.21.15.27:0/833057004 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2758066f30 0x7f27580693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:08.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.083+0000 7f277cfff700 1 --2- 172.21.15.27:0/833057004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27781575e0 0x7f27780c8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:08.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.083+0000 7f277cfff700 1 --2- 172.21.15.27:0/833057004 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f277806b610 0x7f277816af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:08.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.083+0000 7f277cfff700 1 --2- 172.21.15.27:0/833057004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27780b19d0 0x7f27781538b0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:08.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.083+0000 7f277cfff700 1 -- 172.21.15.27:0/833057004 >> 172.21.15.27:0/833057004 conn(0x7f27780680e0 msgr2=0x7f277815a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:08.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.083+0000 7f277cfff700 1 -- 172.21.15.27:0/833057004 shutdown_connections 2024-01-11T08:30:08.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.084+0000 7f277cfff700 1 -- 172.21.15.27:0/833057004 wait complete. 2024-01-11T08:30:08.102 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-11T08:30:08.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.307+0000 7f7d789ff700 1 Processor -- start 2024-01-11T08:30:08.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.308+0000 7f7d789ff700 1 -- start start 2024-01-11T08:30:08.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.309+0000 7f7d789ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7d7413fbb0 0x7f7d740ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:08.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.309+0000 7f7d789ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d7415a730 0x7f7d740c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:08.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.309+0000 7f7d789ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d740c6b30 con 0x7f7d7413ff80 2024-01-11T08:30:08.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.309+0000 7f7d789ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d740c6800 con 0x7f7d7413fbb0 2024-01-11T08:30:08.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.309+0000 7f7d789ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d740cb240 con 0x7f7d7415a730 2024-01-11T08:30:08.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.309+0000 7f7d72d9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d7415a730 0x7f7d740c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:08.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.309+0000 7f7d71d9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7d7413ff80 0x7f7d7415a020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:55340/0 (socket says 172.21.15.27:55340) 2024-01-11T08:30:08.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.310+0000 7f7d71d9b700 1 -- 172.21.15.27:0/3855180561 learned_addr learned my addr 172.21.15.27:0/3855180561 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:30:08.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.310+0000 7f7d7259c700 1 --2- 172.21.15.27:0/3855180561 >> v2:172.21.15.77:3300/0 conn(0x7f7d7413fbb0 0x7f7d740ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:08.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.310+0000 7f7d7159a700 1 -- 172.21.15.27:0/3855180561 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550553035 0 0) 0x7f7d740c6b30 con 0x7f7d7413ff80 2024-01-11T08:30:08.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.310+0000 7f7d7159a700 1 -- 172.21.15.27:0/3855180561 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d5c003570 con 0x7f7d7413ff80 2024-01-11T08:30:08.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.310+0000 7f7d7259c700 1 -- 172.21.15.27:0/3855180561 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d7415a730 msgr2=0x7f7d740c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:08.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.310+0000 7f7d7259c700 1 --2- 172.21.15.27:0/3855180561 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d7415a730 0x7f7d740c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:08.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.310+0000 7f7d7259c700 1 -- 172.21.15.27:0/3855180561 >> v1:172.21.15.77:6789/0 conn(0x7f7d7413ff80 legacy=0x7f7d7415a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:08.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.310+0000 7f7d7259c700 1 -- 172.21.15.27:0/3855180561 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d740cb520 con 0x7f7d7413fbb0 2024-01-11T08:30:08.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.310+0000 7f7d72d9d700 1 --2- 172.21.15.27:0/3855180561 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d7415a730 0x7f7d740c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:30:08.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.310+0000 7f7d7259c700 1 --2- 172.21.15.27:0/3855180561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d7413fbb0 0x7f7d740ca7a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7d640098d0 tx=0x7f7d64009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6433e921874f3736 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:08.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.311+0000 7f7d7159a700 1 -- 172.21.15.27:0/3855180561 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d64005c10 con 0x7f7d7413fbb0 2024-01-11T08:30:08.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.311+0000 7f7d7159a700 1 -- 172.21.15.27:0/3855180561 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d64005d70 con 0x7f7d7413fbb0 2024-01-11T08:30:08.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.311+0000 7f7d7159a700 1 -- 172.21.15.27:0/3855180561 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d64006260 con 0x7f7d7413fbb0 2024-01-11T08:30:08.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.311+0000 7f7d789ff700 1 -- 172.21.15.27:0/3855180561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d7413fbb0 msgr2=0x7f7d740ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:08.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.311+0000 7f7d789ff700 1 --2- 172.21.15.27:0/3855180561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d7413fbb0 0x7f7d740ca7a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7d640098d0 tx=0x7f7d64009120 comp rx=0 tx=0).stop 2024-01-11T08:30:08.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.311+0000 7f7d789ff700 1 -- 172.21.15.27:0/3855180561 shutdown_connections 2024-01-11T08:30:08.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.311+0000 7f7d789ff700 1 --2- 172.21.15.27:0/3855180561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d7413fbb0 0x7f7d740ca7a0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:08.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.311+0000 7f7d789ff700 1 --2- 172.21.15.27:0/3855180561 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d7415a730 0x7f7d740c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:08.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.311+0000 7f7d789ff700 1 -- 172.21.15.27:0/3855180561 >> 172.21.15.27:0/3855180561 conn(0x7f7d740680a0 msgr2=0x7f7d740cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:08.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.312+0000 7f7d789ff700 1 -- 172.21.15.27:0/3855180561 shutdown_connections 2024-01-11T08:30:08.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.312+0000 7f7d789ff700 1 -- 172.21.15.27:0/3855180561 wait complete. 2024-01-11T08:30:08.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.313+0000 7f7d789ff700 1 Processor -- start 2024-01-11T08:30:08.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.313+0000 7f7d789ff700 1 -- start start 2024-01-11T08:30:08.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.314+0000 7f7d789ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d7413fbb0 0x7f7d740c5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:08.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.314+0000 7f7d789ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d7413ff80 0x7f7d7413cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:08.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.314+0000 7f7d789ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d7415a730 0x7f7d740b0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:08.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.314+0000 7f7d789ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7413d4d0 con 0x7f7d7415a730 2024-01-11T08:30:08.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.314+0000 7f7d789ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d740b1b40 con 0x7f7d7413fbb0 2024-01-11T08:30:08.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.314+0000 7f7d789ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d740c4060 con 0x7f7d7413ff80 2024-01-11T08:30:08.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.314+0000 7f7d72d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d7415a730 0x7f7d740b0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:08.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.314+0000 7f7d72d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d7415a730 0x7f7d740b0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53702/0 (socket says 172.21.15.27:53702) 2024-01-11T08:30:08.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.314+0000 7f7d72d9d700 1 -- 172.21.15.27:0/3441933981 learned_addr learned my addr 172.21.15.27:0/3441933981 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:08.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.315+0000 7f7d71d9b700 1 --2- 172.21.15.27:0/3441933981 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d7413ff80 0x7f7d7413cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:08.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.315+0000 7f7d7259c700 1 --2- 172.21.15.27:0/3441933981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d7413fbb0 0x7f7d740c5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:08.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.315+0000 7f7d72d9d700 1 -- 172.21.15.27:0/3441933981 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d7413ff80 msgr2=0x7f7d7413cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:08.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.315+0000 7f7d72d9d700 1 --2- 172.21.15.27:0/3441933981 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d7413ff80 0x7f7d7413cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:08.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.315+0000 7f7d72d9d700 1 -- 172.21.15.27:0/3441933981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d7413fbb0 msgr2=0x7f7d740c5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:08.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.315+0000 7f7d72d9d700 1 --2- 172.21.15.27:0/3441933981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d7413fbb0 0x7f7d740c5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:08.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.315+0000 7f7d72d9d700 1 -- 172.21.15.27:0/3441933981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6c00bc30 con 0x7f7d7415a730 2024-01-11T08:30:08.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.315+0000 7f7d7259c700 1 --2- 172.21.15.27:0/3441933981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d7413fbb0 0x7f7d740c5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:30:08.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.315+0000 7f7d71d9b700 1 --2- 172.21.15.27:0/3441933981 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d7413ff80 0x7f7d7413cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:30:08.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.315+0000 7f7d72d9d700 1 --2- 172.21.15.27:0/3441933981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d7415a730 0x7f7d740b0f10 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00e0f0 tx=0x7f7d6c00d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:08.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.315+0000 7f7d627fc700 1 -- 172.21.15.27:0/3441933981 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c010040 con 0x7f7d7415a730 2024-01-11T08:30:08.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.315+0000 7f7d627fc700 1 -- 172.21.15.27:0/3441933981 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d6c00b2e0 con 0x7f7d7415a730 2024-01-11T08:30:08.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.315+0000 7f7d789ff700 1 -- 172.21.15.27:0/3441933981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d640093c0 con 0x7f7d7415a730 2024-01-11T08:30:08.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.315+0000 7f7d627fc700 1 -- 172.21.15.27:0/3441933981 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c00edd0 con 0x7f7d7415a730 2024-01-11T08:30:08.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.315+0000 7f7d789ff700 1 -- 172.21.15.27:0/3441933981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d740c4620 con 0x7f7d7415a730 2024-01-11T08:30:08.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.316+0000 7f7d789ff700 1 -- 172.21.15.27:0/3441933981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d7404c3c0 con 0x7f7d7415a730 2024-01-11T08:30:08.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.317+0000 7f7d627fc700 1 -- 172.21.15.27:0/3441933981 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7d6c009050 con 0x7f7d7415a730 2024-01-11T08:30:08.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.319+0000 7f7d627fc700 1 --2- 172.21.15.27:0/3441933981 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7d54066f30 0x7f7d540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:08.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.319+0000 7f7d627fc700 1 -- 172.21.15.27:0/3441933981 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f7d6c0844f0 con 0x7f7d7415a730 2024-01-11T08:30:08.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.319+0000 7f7d7259c700 1 --2- 172.21.15.27:0/3441933981 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7d54066f30 0x7f7d540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.320+0000 7f7d7259c700 1 --2- 172.21.15.27:0/3441933981 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7d54066f30 0x7f7d540693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7d64009160 tx=0x7f7d64016000 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:08.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.323+0000 7f7d627fc700 1 -- 172.21.15.27:0/3441933981 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7d6c051750 con 0x7f7d7415a730 2024-01-11T08:30:08.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.526+0000 7f7d789ff700 1 -- 172.21.15.27:0/3441933981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f7d740cb800 con 0x7f7d7415a730 2024-01-11T08:30:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.530+0000 7f7d627fc700 1 -- 172.21.15.27:0/3441933981 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7d6c054d70 con 0x7f7d7415a730 2024-01-11T08:30:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.532+0000 7f7d789ff700 1 -- 172.21.15.27:0/3441933981 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7d54066f30 msgr2=0x7f7d540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.532+0000 7f7d789ff700 1 --2- 172.21.15.27:0/3441933981 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7d54066f30 0x7f7d540693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7d64009160 tx=0x7f7d64016000 comp rx=0 tx=0).stop 2024-01-11T08:30:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.532+0000 7f7d789ff700 1 -- 172.21.15.27:0/3441933981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d7415a730 msgr2=0x7f7d740b0f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.532+0000 7f7d789ff700 1 --2- 172.21.15.27:0/3441933981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d7415a730 0x7f7d740b0f10 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00e0f0 tx=0x7f7d6c00d040 comp rx=0 tx=0).stop 2024-01-11T08:30:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.532+0000 7f7d789ff700 1 -- 172.21.15.27:0/3441933981 shutdown_connections 2024-01-11T08:30:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.532+0000 7f7d789ff700 1 --2- 172.21.15.27:0/3441933981 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7d54066f30 0x7f7d540693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.532+0000 7f7d789ff700 1 --2- 172.21.15.27:0/3441933981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d7415a730 0x7f7d740b0f10 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.532+0000 7f7d789ff700 1 --2- 172.21.15.27:0/3441933981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d7413fbb0 0x7f7d740c5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.532+0000 7f7d789ff700 1 --2- 172.21.15.27:0/3441933981 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d7413ff80 0x7f7d7413cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.532+0000 7f7d789ff700 1 -- 172.21.15.27:0/3441933981 >> 172.21.15.27:0/3441933981 conn(0x7f7d740680a0 msgr2=0x7f7d740c81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.533+0000 7f7d789ff700 1 -- 172.21.15.27:0/3441933981 shutdown_connections 2024-01-11T08:30:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.533+0000 7f7d789ff700 1 -- 172.21.15.27:0/3441933981 wait complete. 2024-01-11T08:30:08.550 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T08:30:08.550 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:30:08.644 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T08:30:08.748 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T08:30:08.748 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-11T08:30:08.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:30:08.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.949+0000 7fc70f7ff700 1 Processor -- start 2024-01-11T08:30:08.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.949+0000 7fc70f7ff700 1 -- start start 2024-01-11T08:30:08.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.950+0000 7fc70f7ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7080b19d0 0x7fc708158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:08.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.950+0000 7fc70f7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc70806b610 0x7fc708159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:08.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.950+0000 7fc70f7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc708158b60 con 0x7fc7081575e0 2024-01-11T08:30:08.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.950+0000 7fc70f7ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7080cb5e0 con 0x7fc7080b19d0 2024-01-11T08:30:08.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.950+0000 7fc70f7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7081599b0 con 0x7fc70806b610 2024-01-11T08:30:08.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.950+0000 7fc70cd9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7080b19d0 0x7fc708158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:08.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.950+0000 7fc70cd9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7080b19d0 0x7fc708158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53714/0 (socket says 172.21.15.27:53714) 2024-01-11T08:30:08.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.950+0000 7fc70cd9a700 1 -- 172.21.15.27:0/29236354 learned_addr learned my addr 172.21.15.27:0/29236354 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:08.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.950+0000 7fc70d59b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc7081575e0 0x7fc7080b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:32790/0 (socket says 172.21.15.27:32790) 2024-01-11T08:30:08.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.951+0000 7fc70dd9c700 1 --2- 172.21.15.27:0/29236354 >> v2:172.21.15.77:3300/0 conn(0x7fc70806b610 0x7fc708159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:08.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.951+0000 7fc70cd9a700 1 -- 172.21.15.27:0/29236354 >> v2:172.21.15.77:3300/0 conn(0x7fc70806b610 msgr2=0x7fc708159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:08.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.951+0000 7fc70cd9a700 1 --2- 172.21.15.27:0/29236354 >> v2:172.21.15.77:3300/0 conn(0x7fc70806b610 0x7fc708159e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:08.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.951+0000 7fc70cd9a700 1 -- 172.21.15.27:0/29236354 >> v1:172.21.15.27:6789/0 conn(0x7fc7081575e0 legacy=0x7fc7080b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:08.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.951+0000 7fc70cd9a700 1 -- 172.21.15.27:0/29236354 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7080c0f90 con 0x7fc7080b19d0 2024-01-11T08:30:08.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.951+0000 7fc6fffff700 1 -- 172.21.15.27:0/29236354 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510423858 0 0) 0x7fc704001110 con 0x7fc7081575e0 2024-01-11T08:30:08.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.951+0000 7fc70cd9a700 1 --2- 172.21.15.27:0/29236354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7080b19d0 0x7fc708158620 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fc70806d0a0 tx=0x7fc6f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64f44b15310b22de server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:08.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.951+0000 7fc6fffff700 1 -- 172.21.15.27:0/29236354 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8009e70 con 0x7fc7080b19d0 2024-01-11T08:30:08.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.952+0000 7fc6fffff700 1 -- 172.21.15.27:0/29236354 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f800b860 con 0x7fc7080b19d0 2024-01-11T08:30:08.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.952+0000 7fc6fffff700 1 -- 172.21.15.27:0/29236354 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f800bc00 con 0x7fc7080b19d0 2024-01-11T08:30:08.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.952+0000 7fc70f7ff700 1 -- 172.21.15.27:0/29236354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7080b19d0 msgr2=0x7fc708158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:08.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.952+0000 7fc70f7ff700 1 --2- 172.21.15.27:0/29236354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7080b19d0 0x7fc708158620 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fc70806d0a0 tx=0x7fc6f8009250 comp rx=0 tx=0).stop 2024-01-11T08:30:08.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.952+0000 7fc70f7ff700 1 -- 172.21.15.27:0/29236354 shutdown_connections 2024-01-11T08:30:08.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.952+0000 7fc70f7ff700 1 --2- 172.21.15.27:0/29236354 >> v2:172.21.15.77:3300/0 conn(0x7fc70806b610 0x7fc708159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:08.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.952+0000 7fc70f7ff700 1 --2- 172.21.15.27:0/29236354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7080b19d0 0x7fc708158620 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:08.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.952+0000 7fc70f7ff700 1 -- 172.21.15.27:0/29236354 >> 172.21.15.27:0/29236354 conn(0x7fc7080680e0 msgr2=0x7fc708069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:08.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.952+0000 7fc70f7ff700 1 -- 172.21.15.27:0/29236354 shutdown_connections 2024-01-11T08:30:08.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.952+0000 7fc70f7ff700 1 -- 172.21.15.27:0/29236354 wait complete. 2024-01-11T08:30:08.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.954+0000 7fc70f7ff700 1 Processor -- start 2024-01-11T08:30:08.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.954+0000 7fc70f7ff700 1 -- start start 2024-01-11T08:30:08.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.955+0000 7fc70f7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc70806b610 0x7fc7080ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:08.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.955+0000 7fc70f7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7080b19d0 0x7fc7080c6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:08.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.955+0000 7fc70f7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7081575e0 0x7fc7080c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:08.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.955+0000 7fc70f7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7080c7040 con 0x7fc7081575e0 2024-01-11T08:30:08.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.955+0000 7fc70f7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7080ae3a0 con 0x7fc7080b19d0 2024-01-11T08:30:08.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.955+0000 7fc70f7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70816de30 con 0x7fc70806b610 2024-01-11T08:30:08.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.955+0000 7fc70dd9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7081575e0 0x7fc7080c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:08.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.955+0000 7fc70dd9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7081575e0 0x7fc7080c69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53726/0 (socket says 172.21.15.27:53726) 2024-01-11T08:30:08.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.955+0000 7fc70dd9c700 1 -- 172.21.15.27:0/3774587107 learned_addr learned my addr 172.21.15.27:0/3774587107 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:08.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.955+0000 7fc70cd9a700 1 --2- 172.21.15.27:0/3774587107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7080b19d0 0x7fc7080c6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:08.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.955+0000 7fc70d59b700 1 --2- 172.21.15.27:0/3774587107 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc70806b610 0x7fc7080ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:08.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.956+0000 7fc70dd9c700 1 -- 172.21.15.27:0/3774587107 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc70806b610 msgr2=0x7fc7080ade60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:08.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.956+0000 7fc70dd9c700 1 --2- 172.21.15.27:0/3774587107 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc70806b610 0x7fc7080ade60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:08.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.956+0000 7fc70dd9c700 1 -- 172.21.15.27:0/3774587107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7080b19d0 msgr2=0x7fc7080c6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:08.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.956+0000 7fc70dd9c700 1 --2- 172.21.15.27:0/3774587107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7080b19d0 0x7fc7080c6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:08.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.956+0000 7fc70dd9c700 1 -- 172.21.15.27:0/3774587107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f80094f0 con 0x7fc7081575e0 2024-01-11T08:30:08.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.956+0000 7fc70dd9c700 1 --2- 172.21.15.27:0/3774587107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7081575e0 0x7fc7080c69c0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fc700009e90 tx=0x7fc70000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:08.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.956+0000 7fc6fd7fa700 1 -- 172.21.15.27:0/3774587107 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70000cc20 con 0x7fc7081575e0 2024-01-11T08:30:08.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.956+0000 7fc6fd7fa700 1 -- 172.21.15.27:0/3774587107 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70000f040 con 0x7fc7081575e0 2024-01-11T08:30:08.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.956+0000 7fc70f7ff700 1 -- 172.21.15.27:0/3774587107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7081544e0 con 0x7fc7081575e0 2024-01-11T08:30:08.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.956+0000 7fc6fd7fa700 1 -- 172.21.15.27:0/3774587107 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc700014480 con 0x7fc7081575e0 2024-01-11T08:30:08.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.956+0000 7fc70f7ff700 1 -- 172.21.15.27:0/3774587107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc70816d7e0 con 0x7fc7081575e0 2024-01-11T08:30:08.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.957+0000 7fc70f7ff700 1 -- 172.21.15.27:0/3774587107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7080b1da0 con 0x7fc7081575e0 2024-01-11T08:30:08.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.958+0000 7fc6fd7fa700 1 -- 172.21.15.27:0/3774587107 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc700014830 con 0x7fc7081575e0 2024-01-11T08:30:08.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.959+0000 7fc6fd7fa700 1 --2- 172.21.15.27:0/3774587107 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc6ec066f30 0x7fc6ec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:08.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.959+0000 7fc6fd7fa700 1 -- 172.21.15.27:0/3774587107 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fc700084f60 con 0x7fc7081575e0 2024-01-11T08:30:08.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.960+0000 7fc70d59b700 1 --2- 172.21.15.27:0/3774587107 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc6ec066f30 0x7fc6ec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:08.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.960+0000 7fc70d59b700 1 --2- 172.21.15.27:0/3774587107 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc6ec066f30 0x7fc6ec0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc70816d2c0 tx=0x7fc704007d80 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:08.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:08.963+0000 7fc6fd7fa700 1 -- 172.21.15.27:0/3774587107 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc700055c50 con 0x7fc7081575e0 2024-01-11T08:30:09.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.212+0000 7fc70f7ff700 1 -- 172.21.15.27:0/3774587107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc70816dfa0 con 0x7fc7081575e0 2024-01-11T08:30:09.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.213+0000 7fc6fd7fa700 1 -- 172.21.15.27:0/3774587107 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+3526 (secure 0 0 0) 0x7fc700018de0 con 0x7fc7081575e0 2024-01-11T08:30:09.212 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:30:09.212 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":71,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6956,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1277153217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1277153217},{"type":"v1","addr":"172.21.15.77:6833","nonce":1277153217}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7777,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2435914089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2435914089},{"type":"v1","addr":"172.21.15.27:6835","nonce":2435914089}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63},{"gid":7792,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/4128023892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":4128023892},{"type":"v1","addr":"172.21.15.77:6837","nonce":4128023892}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":64},{"gid":8167,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/174898072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":174898072},{"type":"v1","addr":"172.21.15.27:6834","nonce":174898072}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":71}],"filesystems":[]} 2024-01-11T08:30:09.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.215+0000 7fc70f7ff700 1 -- 172.21.15.27:0/3774587107 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc6ec066f30 msgr2=0x7fc6ec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:09.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.215+0000 7fc70f7ff700 1 --2- 172.21.15.27:0/3774587107 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc6ec066f30 0x7fc6ec0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc70816d2c0 tx=0x7fc704007d80 comp rx=0 tx=0).stop 2024-01-11T08:30:09.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.215+0000 7fc70f7ff700 1 -- 172.21.15.27:0/3774587107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7081575e0 msgr2=0x7fc7080c69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:09.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.215+0000 7fc70f7ff700 1 --2- 172.21.15.27:0/3774587107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7081575e0 0x7fc7080c69c0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fc700009e90 tx=0x7fc70000b040 comp rx=0 tx=0).stop 2024-01-11T08:30:09.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.215+0000 7fc70f7ff700 1 -- 172.21.15.27:0/3774587107 shutdown_connections 2024-01-11T08:30:09.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.215+0000 7fc70f7ff700 1 --2- 172.21.15.27:0/3774587107 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc6ec066f30 0x7fc6ec0693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:09.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.215+0000 7fc70f7ff700 1 --2- 172.21.15.27:0/3774587107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7081575e0 0x7fc7080c69c0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:09.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.215+0000 7fc70f7ff700 1 --2- 172.21.15.27:0/3774587107 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc70806b610 0x7fc7080ade60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:09.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.215+0000 7fc70f7ff700 1 --2- 172.21.15.27:0/3774587107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7080b19d0 0x7fc7080c6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:09.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.215+0000 7fc70f7ff700 1 -- 172.21.15.27:0/3774587107 >> 172.21.15.27:0/3774587107 conn(0x7fc7080680e0 msgr2=0x7fc708154990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:09.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.215+0000 7fc70f7ff700 1 -- 172.21.15.27:0/3774587107 shutdown_connections 2024-01-11T08:30:09.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.215+0000 7fc70f7ff700 1 -- 172.21.15.27:0/3774587107 wait complete. 2024-01-11T08:30:09.215 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 71 2024-01-11T08:30:09.232 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-11T08:30:09.232 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-11T08:30:09.232 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-11T08:30:09.232 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:30:09.233 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-11T08:30:09.233 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-11T08:30:09.233 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-11T08:30:09.233 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:30:09.233 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-11T08:30:09.233 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-11T08:30:09.233 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-11T08:30:09.233 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:30:09.234 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-11T08:30:09.234 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-11T08:30:09.234 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-11T08:30:09.234 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T08:30:09.248 INFO:tasks.ceph.mds.a:Stopped 2024-01-11T08:30:09.248 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-11T08:30:09.249 INFO:tasks.ceph.mds.c:Stopped 2024-01-11T08:30:09.249 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-11T08:30:09.254 INFO:tasks.ceph.mds.b:Stopped 2024-01-11T08:30:09.254 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-11T08:30:09.254 INFO:tasks.ceph.mds.d:Stopped 2024-01-11T08:30:09.255 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-11T08:30:09.294 INFO:tasks.ceph.mds.a:Started 2024-01-11T08:30:09.294 DEBUG:teuthology.parallel:result is None 2024-01-11T08:30:09.297 INFO:tasks.ceph.mds.c:Started 2024-01-11T08:30:09.297 DEBUG:teuthology.parallel:result is None 2024-01-11T08:30:09.299 INFO:tasks.ceph.mds.b:Started 2024-01-11T08:30:09.299 DEBUG:teuthology.parallel:result is None 2024-01-11T08:30:09.302 INFO:tasks.ceph.mds.d:Started 2024-01-11T08:30:09.302 DEBUG:teuthology.parallel:result is None 2024-01-11T08:30:09.303 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-11T08:30:09.445 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T08:30:09.446 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T08:30:09.449 INFO:tasks.ceph.mds.b.smithi077.stdout:starting mds.b at 2024-01-11T08:30:09.450 INFO:tasks.ceph.mds.d.smithi077.stdout:starting mds.d at 2024-01-11T08:30:09.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.518+0000 7f4fb6fff700 1 Processor -- start 2024-01-11T08:30:09.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.518+0000 7f4fb6fff700 1 -- start start 2024-01-11T08:30:09.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.519+0000 7f4fb6fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4fb006bf10 0x7f4fb0159c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:09.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.519+0000 7f4fb6fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4fb0068720 0x7f4fb0068b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:09.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.520+0000 7f4fb6fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fb0069060 con 0x7f4fb00b0a40 2024-01-11T08:30:09.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.520+0000 7f4fb6fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb015d290 con 0x7f4fb006bf10 2024-01-11T08:30:09.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.520+0000 7f4fb6fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb015d570 con 0x7f4fb0068720 2024-01-11T08:30:09.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.520+0000 7f4fb4d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4fb006bf10 0x7f4fb0159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:09.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.520+0000 7f4faffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4fb00b0a40 0x7f4fb015da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:32800/0 (socket says 172.21.15.27:32800) 2024-01-11T08:30:09.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.520+0000 7f4faffff700 1 -- 172.21.15.27:0/337142409 learned_addr learned my addr 172.21.15.27:0/337142409 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:30:09.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.520+0000 7f4fb559c700 1 --2- 172.21.15.27:0/337142409 >> v2:172.21.15.77:3300/0 conn(0x7f4fb0068720 0x7f4fb0068b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:09.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.520+0000 7f4fb4d9b700 1 -- 172.21.15.27:0/337142409 >> v2:172.21.15.77:3300/0 conn(0x7f4fb0068720 msgr2=0x7f4fb0068b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:09.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.520+0000 7f4fb4d9b700 1 --2- 172.21.15.27:0/337142409 >> v2:172.21.15.77:3300/0 conn(0x7f4fb0068720 0x7f4fb0068b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:09.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.520+0000 7f4fb4d9b700 1 -- 172.21.15.27:0/337142409 >> v1:172.21.15.27:6789/0 conn(0x7f4fb00b0a40 legacy=0x7f4fb015da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:09.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.520+0000 7f4fb4d9b700 1 -- 172.21.15.27:0/337142409 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb0154180 con 0x7f4fb006bf10 2024-01-11T08:30:09.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.521+0000 7f4fb4d9b700 1 --2- 172.21.15.27:0/337142409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb006bf10 0x7f4fb0159c50 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00cf310 tx=0x7f4fa40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=282e5ab055cef363 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:09.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.521+0000 7f4faf7fe700 1 -- 172.21.15.27:0/337142409 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa400b6b0 con 0x7f4fb006bf10 2024-01-11T08:30:09.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.521+0000 7f4faf7fe700 1 -- 172.21.15.27:0/337142409 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa400b810 con 0x7f4fb006bf10 2024-01-11T08:30:09.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.521+0000 7f4faf7fe700 1 -- 172.21.15.27:0/337142409 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa400bbb0 con 0x7f4fb006bf10 2024-01-11T08:30:09.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.521+0000 7f4fb6fff700 1 -- 172.21.15.27:0/337142409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb006bf10 msgr2=0x7f4fb0159c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:09.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.521+0000 7f4fb6fff700 1 --2- 172.21.15.27:0/337142409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb006bf10 0x7f4fb0159c50 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00cf310 tx=0x7f4fa40091a0 comp rx=0 tx=0).stop 2024-01-11T08:30:09.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.522+0000 7f4fb6fff700 1 -- 172.21.15.27:0/337142409 shutdown_connections 2024-01-11T08:30:09.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.522+0000 7f4fb6fff700 1 --2- 172.21.15.27:0/337142409 >> v2:172.21.15.77:3300/0 conn(0x7f4fb0068720 0x7f4fb0068b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:09.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.522+0000 7f4fb6fff700 1 --2- 172.21.15.27:0/337142409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb006bf10 0x7f4fb0159c50 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:09.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.522+0000 7f4fb6fff700 1 -- 172.21.15.27:0/337142409 >> 172.21.15.27:0/337142409 conn(0x7f4fb00ce3c0 msgr2=0x7f4fb00ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:09.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.522+0000 7f4fb6fff700 1 -- 172.21.15.27:0/337142409 shutdown_connections 2024-01-11T08:30:09.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.522+0000 7f4fb6fff700 1 -- 172.21.15.27:0/337142409 wait complete. 2024-01-11T08:30:09.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.523+0000 7f4fb6fff700 1 Processor -- start 2024-01-11T08:30:09.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.523+0000 7f4fb6fff700 1 -- start start 2024-01-11T08:30:09.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.524+0000 7f4fb6fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb0068720 0x7f4fb016d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.524+0000 7f4fb6fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4fb006bf10 0x7f4fb00b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.524+0000 7f4fb6fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fb00b0a40 0x7f4fb00b16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.524+0000 7f4fb6fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb015e740 con 0x7f4fb0068720 2024-01-11T08:30:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.524+0000 7f4fb6fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb00b3b50 con 0x7f4fb00b0a40 2024-01-11T08:30:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.524+0000 7f4fb6fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb016de40 con 0x7f4fb006bf10 2024-01-11T08:30:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.525+0000 7f4fb4d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb0068720 0x7f4fb016d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.525+0000 7f4fb4d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb0068720 0x7f4fb016d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53766/0 (socket says 172.21.15.27:53766) 2024-01-11T08:30:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.525+0000 7f4fb4d9b700 1 -- 172.21.15.27:0/2291030102 learned_addr learned my addr 172.21.15.27:0/2291030102 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.525+0000 7f4faffff700 1 --2- 172.21.15.27:0/2291030102 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4fb006bf10 0x7f4fb00b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.525+0000 7f4fb559c700 1 --2- 172.21.15.27:0/2291030102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fb00b0a40 0x7f4fb00b16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.525+0000 7f4fb4d9b700 1 -- 172.21.15.27:0/2291030102 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4fb006bf10 msgr2=0x7f4fb00b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.525+0000 7f4fb4d9b700 1 --2- 172.21.15.27:0/2291030102 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4fb006bf10 0x7f4fb00b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.525+0000 7f4fb4d9b700 1 -- 172.21.15.27:0/2291030102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fb00b0a40 msgr2=0x7f4fb00b16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:09.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.525+0000 7f4fb4d9b700 1 --2- 172.21.15.27:0/2291030102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fb00b0a40 0x7f4fb00b16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:09.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.525+0000 7f4fb4d9b700 1 -- 172.21.15.27:0/2291030102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa4009420 con 0x7f4fb0068720 2024-01-11T08:30:09.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.525+0000 7f4fb4d9b700 1 --2- 172.21.15.27:0/2291030102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb0068720 0x7f4fb016d900 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400b0e0 tx=0x7f4fa4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:09.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.526+0000 7f4facff9700 1 -- 172.21.15.27:0/2291030102 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4011420 con 0x7f4fb0068720 2024-01-11T08:30:09.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.526+0000 7f4facff9700 1 -- 172.21.15.27:0/2291030102 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa4011580 con 0x7f4fb0068720 2024-01-11T08:30:09.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.526+0000 7f4fb6fff700 1 -- 172.21.15.27:0/2291030102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb00b3220 con 0x7f4fb0068720 2024-01-11T08:30:09.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.526+0000 7f4facff9700 1 -- 172.21.15.27:0/2291030102 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4011940 con 0x7f4fb0068720 2024-01-11T08:30:09.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.526+0000 7f4fb6fff700 1 -- 172.21.15.27:0/2291030102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb00b37e0 con 0x7f4fb0068720 2024-01-11T08:30:09.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.527+0000 7f4fb6fff700 1 -- 172.21.15.27:0/2291030102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb004c3c0 con 0x7f4fb0068720 2024-01-11T08:30:09.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.528+0000 7f4facff9700 1 -- 172.21.15.27:0/2291030102 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4fa4011d30 con 0x7f4fb0068720 2024-01-11T08:30:09.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.529+0000 7f4facff9700 1 --2- 172.21.15.27:0/2291030102 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4f94066f60 0x7f4f94069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:09.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.529+0000 7f4facff9700 1 -- 172.21.15.27:0/2291030102 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f4fa4085e50 con 0x7f4fb0068720 2024-01-11T08:30:09.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.529+0000 7f4faffff700 1 --2- 172.21.15.27:0/2291030102 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4f94066f60 0x7f4f94069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:09.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.530+0000 7f4faffff700 1 --2- 172.21.15.27:0/2291030102 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4f94066f60 0x7f4f94069410 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00b2e70 tx=0x7f4f9c009040 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:09.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.533+0000 7f4facff9700 1 -- 172.21.15.27:0/2291030102 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4fa4053130 con 0x7f4fb0068720 2024-01-11T08:30:09.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:09.749+0000 7f4fb6fff700 1 -- 172.21.15.27:0/2291030102 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f4fb015a510 con 0x7f4fb0068720 2024-01-11T08:30:10.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:10.803+0000 7f4facff9700 1 -- 172.21.15.27:0/2291030102 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v141) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4fa4056750 con 0x7f4fb0068720 2024-01-11T08:30:10.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:10.806+0000 7f4fb6fff700 1 -- 172.21.15.27:0/2291030102 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4f94066f60 msgr2=0x7f4f94069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:10.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:10.806+0000 7f4fb6fff700 1 --2- 172.21.15.27:0/2291030102 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4f94066f60 0x7f4f94069410 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00b2e70 tx=0x7f4f9c009040 comp rx=0 tx=0).stop 2024-01-11T08:30:10.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:10.806+0000 7f4fb6fff700 1 -- 172.21.15.27:0/2291030102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb0068720 msgr2=0x7f4fb016d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:10.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:10.806+0000 7f4fb6fff700 1 --2- 172.21.15.27:0/2291030102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb0068720 0x7f4fb016d900 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400b0e0 tx=0x7f4fa4017000 comp rx=0 tx=0).stop 2024-01-11T08:30:10.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:10.806+0000 7f4fb6fff700 1 -- 172.21.15.27:0/2291030102 shutdown_connections 2024-01-11T08:30:10.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:10.806+0000 7f4fb6fff700 1 --2- 172.21.15.27:0/2291030102 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4f94066f60 0x7f4f94069410 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:10.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:10.806+0000 7f4fb6fff700 1 --2- 172.21.15.27:0/2291030102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4fb00b0a40 0x7f4fb00b16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:10.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:10.806+0000 7f4fb6fff700 1 --2- 172.21.15.27:0/2291030102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb0068720 0x7f4fb016d900 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:10.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:10.806+0000 7f4fb6fff700 1 --2- 172.21.15.27:0/2291030102 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4fb006bf10 0x7f4fb00b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:10.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:10.806+0000 7f4fb6fff700 1 -- 172.21.15.27:0/2291030102 >> 172.21.15.27:0/2291030102 conn(0x7f4fb00ce3c0 msgr2=0x7f4fb006d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:10.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:10.807+0000 7f4fb6fff700 1 -- 172.21.15.27:0/2291030102 shutdown_connections 2024-01-11T08:30:10.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:10.807+0000 7f4fb6fff700 1 -- 172.21.15.27:0/2291030102 wait complete. 2024-01-11T08:30:10.807 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-11T08:30:10.829 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-11T08:30:11.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.035+0000 7fa0f87ff700 1 Processor -- start 2024-01-11T08:30:11.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.035+0000 7fa0f87ff700 1 -- start start 2024-01-11T08:30:11.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.037+0000 7fa0f87ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa0f01575e0 0x7fa0f0158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:11.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.037+0000 7fa0f87ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0f006b610 0x7fa0f00b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:11.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.037+0000 7fa0f87ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f0158b60 con 0x7fa0f00b1800 2024-01-11T08:30:11.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.037+0000 7fa0f87ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f00cb5e0 con 0x7fa0f01575e0 2024-01-11T08:30:11.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.037+0000 7fa0f87ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f00b1c80 con 0x7fa0f006b610 2024-01-11T08:30:11.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.037+0000 7fa0f5d9a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0f006b610 0x7fa0f00b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:11.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.037+0000 7fa0f659b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa0f01575e0 0x7fa0f0158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:11.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.037+0000 7fa0f659b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa0f01575e0 0x7fa0f0158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:51120/0 (socket says 172.21.15.27:51120) 2024-01-11T08:30:11.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.037+0000 7fa0f659b700 1 -- 172.21.15.27:0/2227656484 learned_addr learned my addr 172.21.15.27:0/2227656484 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:11.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.037+0000 7fa0f5d9a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0f006b610 0x7fa0f00b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:36870/0 (socket says 172.21.15.27:36870) 2024-01-11T08:30:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.037+0000 7fa0f6d9c700 1 --1- 172.21.15.27:0/2227656484 >> v1:172.21.15.77:6789/0 conn(0x7fa0f00b1800 0x7fa0f00ca1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:55364/0 (socket says 172.21.15.27:55364) 2024-01-11T08:30:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.038+0000 7fa0f659b700 1 -- 172.21.15.27:0/2227656484 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0f006b610 msgr2=0x7fa0f00b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.038+0000 7fa0f659b700 1 --2- 172.21.15.27:0/2227656484 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0f006b610 0x7fa0f00b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.038+0000 7fa0f659b700 1 -- 172.21.15.27:0/2227656484 >> v1:172.21.15.77:6789/0 conn(0x7fa0f00b1800 legacy=0x7fa0f00ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.038+0000 7fa0f659b700 1 -- 172.21.15.27:0/2227656484 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f01599b0 con 0x7fa0f01575e0 2024-01-11T08:30:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.038+0000 7fa0f5599700 1 -- 172.21.15.27:0/2227656484 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626926044 0 0) 0x7fa0e8001490 con 0x7fa0f00b1800 2024-01-11T08:30:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.038+0000 7fa0f5d9a700 1 --2- 172.21.15.27:0/2227656484 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0f006b610 0x7fa0f00b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:30:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.038+0000 7fa0f659b700 1 --2- 172.21.15.27:0/2227656484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0f01575e0 0x7fa0f0158620 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec0099b0 tx=0x7fa0ec009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cf099863c1dfcc8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.039+0000 7fa0f5599700 1 -- 172.21.15.27:0/2227656484 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec009ea0 con 0x7fa0f01575e0 2024-01-11T08:30:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.039+0000 7fa0f5599700 1 -- 172.21.15.27:0/2227656484 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ec00b860 con 0x7fa0f01575e0 2024-01-11T08:30:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.039+0000 7fa0f5599700 1 -- 172.21.15.27:0/2227656484 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec00bc00 con 0x7fa0f01575e0 2024-01-11T08:30:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.039+0000 7fa0f87ff700 1 -- 172.21.15.27:0/2227656484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0f01575e0 msgr2=0x7fa0f0158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.039+0000 7fa0f87ff700 1 --2- 172.21.15.27:0/2227656484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0f01575e0 0x7fa0f0158620 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec0099b0 tx=0x7fa0ec009250 comp rx=0 tx=0).stop 2024-01-11T08:30:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.039+0000 7fa0f87ff700 1 -- 172.21.15.27:0/2227656484 shutdown_connections 2024-01-11T08:30:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.039+0000 7fa0f87ff700 1 --2- 172.21.15.27:0/2227656484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0f01575e0 0x7fa0f0158620 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.039+0000 7fa0f87ff700 1 --2- 172.21.15.27:0/2227656484 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0f006b610 0x7fa0f00b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.039+0000 7fa0f87ff700 1 -- 172.21.15.27:0/2227656484 >> 172.21.15.27:0/2227656484 conn(0x7fa0f00680e0 msgr2=0x7fa0f0069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:11.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.039+0000 7fa0f87ff700 1 -- 172.21.15.27:0/2227656484 shutdown_connections 2024-01-11T08:30:11.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.039+0000 7fa0f87ff700 1 -- 172.21.15.27:0/2227656484 wait complete. 2024-01-11T08:30:11.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.041+0000 7fa0f87ff700 1 Processor -- start 2024-01-11T08:30:11.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.041+0000 7fa0f87ff700 1 -- start start 2024-01-11T08:30:11.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.042+0000 7fa0f87ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0f006b610 0x7fa0f015de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.042+0000 7fa0f87ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0f00b1800 0x7fa0f0153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.042+0000 7fa0f87ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0f01575e0 0x7fa0f015c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.042+0000 7fa0f87ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f015d350 con 0x7fa0f006b610 2024-01-11T08:30:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.042+0000 7fa0f87ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f015cb20 con 0x7fa0f01575e0 2024-01-11T08:30:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.042+0000 7fa0f87ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f015d610 con 0x7fa0f00b1800 2024-01-11T08:30:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.042+0000 7fa0f659b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0f006b610 0x7fa0f015de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.042+0000 7fa0f659b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0f006b610 0x7fa0f015de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53770/0 (socket says 172.21.15.27:53770) 2024-01-11T08:30:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.042+0000 7fa0f659b700 1 -- 172.21.15.27:0/666264525 learned_addr learned my addr 172.21.15.27:0/666264525 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.042+0000 7fa0f6d9c700 1 --2- 172.21.15.27:0/666264525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0f01575e0 0x7fa0f015c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.042+0000 7fa0f5d9a700 1 --2- 172.21.15.27:0/666264525 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0f00b1800 0x7fa0f0153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.043+0000 7fa0f659b700 1 -- 172.21.15.27:0/666264525 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0f00b1800 msgr2=0x7fa0f0153bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.043+0000 7fa0f659b700 1 --2- 172.21.15.27:0/666264525 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0f00b1800 0x7fa0f0153bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.043+0000 7fa0f659b700 1 -- 172.21.15.27:0/666264525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0f01575e0 msgr2=0x7fa0f015c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.043+0000 7fa0f659b700 1 --2- 172.21.15.27:0/666264525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0f01575e0 0x7fa0f015c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.043+0000 7fa0f659b700 1 -- 172.21.15.27:0/666264525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e0009420 con 0x7fa0f006b610 2024-01-11T08:30:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.043+0000 7fa0f659b700 1 --2- 172.21.15.27:0/666264525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0f006b610 0x7fa0f015de10 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0066180 tx=0x7fa0ec017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.043+0000 7fa0e67fc700 1 -- 172.21.15.27:0/666264525 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec000d10 con 0x7fa0f006b610 2024-01-11T08:30:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.043+0000 7fa0e67fc700 1 -- 172.21.15.27:0/666264525 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ec00bbe0 con 0x7fa0f006b610 2024-01-11T08:30:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.043+0000 7fa0e67fc700 1 -- 172.21.15.27:0/666264525 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec011690 con 0x7fa0f006b610 2024-01-11T08:30:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.043+0000 7fa0f87ff700 1 -- 172.21.15.27:0/666264525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0ec0094f0 con 0x7fa0f006b610 2024-01-11T08:30:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.044+0000 7fa0f87ff700 1 -- 172.21.15.27:0/666264525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f00ae860 con 0x7fa0f006b610 2024-01-11T08:30:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.045+0000 7fa0f87ff700 1 -- 172.21.15.27:0/666264525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f00718d0 con 0x7fa0f006b610 2024-01-11T08:30:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.046+0000 7fa0e67fc700 1 -- 172.21.15.27:0/666264525 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa0ec011ac0 con 0x7fa0f006b610 2024-01-11T08:30:11.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.046+0000 7fa0e67fc700 1 --2- 172.21.15.27:0/666264525 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa0d4066f30 0x7fa0d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:11.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.046+0000 7fa0e67fc700 1 -- 172.21.15.27:0/666264525 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 4909+0+0 (secure 0 0 0) 0x7fa0ec085a60 con 0x7fa0f006b610 2024-01-11T08:30:11.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.047+0000 7fa0f5d9a700 1 --2- 172.21.15.27:0/666264525 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa0d4066f30 0x7fa0d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:11.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.047+0000 7fa0f5d9a700 1 --2- 172.21.15.27:0/666264525 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa0d4066f30 0x7fa0d40693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0156aa0 tx=0x7fa0e00091a0 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:11.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.051+0000 7fa0e67fc700 1 -- 172.21.15.27:0/666264525 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa0ec052f50 con 0x7fa0f006b610 2024-01-11T08:30:11.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.331+0000 7fa0f87ff700 1 -- 172.21.15.27:0/666264525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa0f00b2070 con 0x7fa0f006b610 2024-01-11T08:30:11.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.331+0000 7fa0e67fc700 1 -- 172.21.15.27:0/666264525 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fa0ec015090 con 0x7fa0f006b610 2024-01-11T08:30:11.330 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:30:11.330 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T08:30:11.331 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-11T08:30:11.331 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:11.331 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-11T08:30:11.331 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lQm3yLhAAv6erz5hMfat8EUzRA5luMA==", 2024-01-11T08:30:11.331 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:30:11.331 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:30:11.331 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:30:11.331 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:30:11.331 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:30:11.331 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.331 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:30:11.331 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:11.331 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-11T08:30:11.331 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9l+FG8ChAACHndtVLx7qS3sidPniSycA==", 2024-01-11T08:30:11.332 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:30:11.332 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:30:11.332 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:30:11.332 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:30:11.332 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:30:11.332 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.332 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:30:11.332 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:11.332 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-11T08:30:11.332 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l1NnEOhAALHXsc5es7hufCthGdRzkyw==", 2024-01-11T08:30:11.332 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:30:11.332 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:30:11.332 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:30:11.332 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:30:11.332 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:30:11.333 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.333 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:30:11.333 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:11.333 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-11T08:30:11.333 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lruoWFhAAm8AmrZtTJhYI1+19F79RPQ==", 2024-01-11T08:30:11.333 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:30:11.333 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:30:11.333 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:30:11.333 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:30:11.333 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:30:11.333 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.333 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:30:11.333 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:11.333 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-11T08:30:11.333 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA2op9ltZygORAAZr1X1nCN0ce8rIVaGDpFyQ==", 2024-01-11T08:30:11.334 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:30:11.334 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:30:11.334 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:30:11.334 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:30:11.334 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.334 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:30:11.334 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:11.334 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-11T08:30:11.334 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA6op9lqSRPAhAAmtyCsyK5ZVGX2wfhOCMuFQ==", 2024-01-11T08:30:11.334 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:30:11.334 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:30:11.334 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:30:11.334 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:30:11.334 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.334 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:30:11.334 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:11.335 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-11T08:30:11.335 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA9op9lZcs8BhAA8j+U+jPOoy+fliVWZAPFYg==", 2024-01-11T08:30:11.335 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:30:11.335 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:30:11.335 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:30:11.335 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:30:11.335 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.335 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:30:11.335 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:11.335 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-11T08:30:11.335 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBAop9lFuCUChAAP6si/2W1mxH4ecQJvuumkw==", 2024-01-11T08:30:11.335 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:30:11.335 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:30:11.335 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:30:11.335 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:30:11.336 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.336 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:30:11.336 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:11.336 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-11T08:30:11.336 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBGop9lOpwBIRAAcpcsjwmX2IldcKHJWAgHuA==", 2024-01-11T08:30:11.336 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:30:11.336 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:30:11.336 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:30:11.336 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:30:11.336 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.336 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:30:11.336 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:11.336 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-11T08:30:11.336 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBJop9l4q8hJRAAK8mAYVRQMoWS6wBOU69HmA==", 2024-01-11T08:30:11.336 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:30:11.337 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:30:11.337 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:30:11.337 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:30:11.337 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.337 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:30:11.337 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:11.337 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-11T08:30:11.337 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBMop9lqMF9KBAAFgFFI2GWt9AGif4AoScThw==", 2024-01-11T08:30:11.337 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:30:11.337 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:30:11.337 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:30:11.337 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:30:11.337 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.337 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:30:11.337 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:11.338 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-11T08:30:11.338 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBPop9lN7eDLBAAKswPZFfs0g2AaSe6V+pbbQ==", 2024-01-11T08:30:11.338 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:30:11.338 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T08:30:11.338 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T08:30:11.338 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:30:11.338 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.338 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:30:11.338 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:11.338 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-11T08:30:11.338 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lT5okIBAABy8PbJJbA062VNmb1Y8qhg==", 2024-01-11T08:30:11.338 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:30:11.338 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:30:11.338 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-01-11T08:30:11.338 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-11T08:30:11.338 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.339 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:30:11.339 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:11.339 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-11T08:30:11.339 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lDf37LhAAqV+EabKXJEW1sS5xtu1tXw==", 2024-01-11T08:30:11.339 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:30:11.339 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:30:11.339 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:30:11.339 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:30:11.339 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:30:11.339 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.339 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:30:11.339 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:11.339 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-11T08:30:11.339 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lWayxJxAAcalM8QSOGtbCXxjYZGaNQA==", 2024-01-11T08:30:11.339 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:30:11.340 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:30:11.340 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:30:11.340 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:30:11.340 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:30:11.340 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.340 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:30:11.340 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:11.340 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-11T08:30:11.340 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lHD4tNhAAdmnFBmMrK+QhQYdpbzW0mg==", 2024-01-11T08:30:11.340 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:30:11.340 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T08:30:11.340 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T08:30:11.340 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T08:30:11.340 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T08:30:11.340 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:30:11.341 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:11.341 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-11T08:30:11.341 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAwop9lGPtDMxAAJ8FjrSEdCtQL8AhtAyqnTw==", 2024-01-11T08:30:11.341 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:30:11.341 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:30:11.341 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T08:30:11.341 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T08:30:11.341 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:30:11.341 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:30:11.341 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:11.341 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-11T08:30:11.341 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lXBZLJxAAXa1S8Vkm3MiV4igtNbg/1Q==", 2024-01-11T08:30:11.341 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:30:11.342 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:30:11.342 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T08:30:11.342 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:30:11.342 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.342 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T08:30:11.342 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:11.342 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-11T08:30:11.342 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l3ZGoHxAANqzytTTvWYSomG+RFsFDyg==", 2024-01-11T08:30:11.342 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T08:30:11.342 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T08:30:11.342 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T08:30:11.342 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T08:30:11.342 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.342 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:11.342 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-11T08:30:11.343 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T08:30:11.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.334+0000 7fa0f87ff700 1 -- 172.21.15.27:0/666264525 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa0d4066f30 msgr2=0x7fa0d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:11.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.334+0000 7fa0f87ff700 1 --2- 172.21.15.27:0/666264525 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa0d4066f30 0x7fa0d40693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0156aa0 tx=0x7fa0e00091a0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.334+0000 7fa0f87ff700 1 -- 172.21.15.27:0/666264525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0f006b610 msgr2=0x7fa0f015de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:11.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.334+0000 7fa0f87ff700 1 --2- 172.21.15.27:0/666264525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0f006b610 0x7fa0f015de10 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0066180 tx=0x7fa0ec017000 comp rx=0 tx=0).stop 2024-01-11T08:30:11.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.334+0000 7fa0f87ff700 1 -- 172.21.15.27:0/666264525 shutdown_connections 2024-01-11T08:30:11.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.334+0000 7fa0f87ff700 1 --2- 172.21.15.27:0/666264525 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fa0d4066f30 0x7fa0d40693e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.334+0000 7fa0f87ff700 1 --2- 172.21.15.27:0/666264525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0f01575e0 0x7fa0f015c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.334+0000 7fa0f87ff700 1 --2- 172.21.15.27:0/666264525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0f006b610 0x7fa0f015de10 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.334+0000 7fa0f87ff700 1 --2- 172.21.15.27:0/666264525 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0f00b1800 0x7fa0f0153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.334+0000 7fa0f87ff700 1 -- 172.21.15.27:0/666264525 >> 172.21.15.27:0/666264525 conn(0x7fa0f00680e0 msgr2=0x7fa0f00cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:11.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.334+0000 7fa0f87ff700 1 -- 172.21.15.27:0/666264525 shutdown_connections 2024-01-11T08:30:11.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.334+0000 7fa0f87ff700 1 -- 172.21.15.27:0/666264525 wait complete. 2024-01-11T08:30:11.351 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:30:11.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.552+0000 7f940ebff700 1 Processor -- start 2024-01-11T08:30:11.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.552+0000 7f940ebff700 1 -- start start 2024-01-11T08:30:11.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.553+0000 7f940ebff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94080b2b90 0x7f9408159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:11.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.553+0000 7f940ebff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94080b07b0 0x7f9408159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:11.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.553+0000 7f940ebff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94080cd730 con 0x7f94080b03e0 2024-01-11T08:30:11.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.553+0000 7f940ebff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94080ccb20 con 0x7f94080b07b0 2024-01-11T08:30:11.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.553+0000 7f940ebff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94080cce00 con 0x7f94080b2b90 2024-01-11T08:30:11.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.554+0000 7f940d19c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94080b07b0 0x7f9408159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:11.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.554+0000 7f940d19c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94080b07b0 0x7f9408159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53778/0 (socket says 172.21.15.27:53778) 2024-01-11T08:30:11.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.554+0000 7f940d19c700 1 -- 172.21.15.27:0/95096794 learned_addr learned my addr 172.21.15.27:0/95096794 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:11.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.554+0000 7f9407fff700 1 --2- 172.21.15.27:0/95096794 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94080b2b90 0x7f9408159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:11.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.554+0000 7f940d19c700 1 -- 172.21.15.27:0/95096794 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94080b2b90 msgr2=0x7f9408159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:11.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.554+0000 7f940d19c700 1 --2- 172.21.15.27:0/95096794 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94080b2b90 0x7f9408159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.554+0000 7f940d19c700 1 -- 172.21.15.27:0/95096794 >> v1:172.21.15.77:6789/0 conn(0x7f94080b03e0 legacy=0x7f94080b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:11.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.554+0000 7f940d19c700 1 -- 172.21.15.27:0/95096794 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94080cd0e0 con 0x7f94080b07b0 2024-01-11T08:30:11.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.554+0000 7f940d19c700 1 --2- 172.21.15.27:0/95096794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94080b07b0 0x7f9408159740 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f9400009f20 tx=0x7f9400009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f40a738de4dc94 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:11.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.555+0000 7f94077fe700 1 -- 172.21.15.27:0/95096794 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940000e070 con 0x7f94080b07b0 2024-01-11T08:30:11.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.555+0000 7f94077fe700 1 -- 172.21.15.27:0/95096794 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940000b9d0 con 0x7f94080b07b0 2024-01-11T08:30:11.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.555+0000 7f94077fe700 1 -- 172.21.15.27:0/95096794 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94000092e0 con 0x7f94080b07b0 2024-01-11T08:30:11.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.555+0000 7f940ebff700 1 -- 172.21.15.27:0/95096794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94080b07b0 msgr2=0x7f9408159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:11.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.555+0000 7f940ebff700 1 --2- 172.21.15.27:0/95096794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94080b07b0 0x7f9408159740 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f9400009f20 tx=0x7f9400009840 comp rx=0 tx=0).stop 2024-01-11T08:30:11.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.555+0000 7f940ebff700 1 -- 172.21.15.27:0/95096794 shutdown_connections 2024-01-11T08:30:11.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.555+0000 7f940ebff700 1 --2- 172.21.15.27:0/95096794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94080b07b0 0x7f9408159740 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.555+0000 7f940ebff700 1 --2- 172.21.15.27:0/95096794 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94080b2b90 0x7f9408159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.555+0000 7f940ebff700 1 -- 172.21.15.27:0/95096794 >> 172.21.15.27:0/95096794 conn(0x7f940806ed40 msgr2=0x7f940806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:11.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.555+0000 7f940ebff700 1 -- 172.21.15.27:0/95096794 shutdown_connections 2024-01-11T08:30:11.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.556+0000 7f940ebff700 1 -- 172.21.15.27:0/95096794 wait complete. 2024-01-11T08:30:11.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.557+0000 7f940ebff700 1 Processor -- start 2024-01-11T08:30:11.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.557+0000 7f940ebff700 1 -- start start 2024-01-11T08:30:11.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.558+0000 7f940ebff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94080b03e0 0x7f94080c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:11.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.558+0000 7f940ebff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94080b07b0 0x7f94080c7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:11.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.558+0000 7f940ebff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94080b2b90 0x7f9408169b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:11.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.558+0000 7f940ebff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94080c99a0 con 0x7f94080b03e0 2024-01-11T08:30:11.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.558+0000 7f940ebff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94080c8a40 con 0x7f94080b07b0 2024-01-11T08:30:11.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.558+0000 7f940ebff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94080c51e0 con 0x7f94080b2b90 2024-01-11T08:30:11.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.559+0000 7f940c99b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94080b03e0 0x7f94080c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:11.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.559+0000 7f940c99b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94080b03e0 0x7f94080c75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53786/0 (socket says 172.21.15.27:53786) 2024-01-11T08:30:11.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.559+0000 7f940c99b700 1 -- 172.21.15.27:0/1963829526 learned_addr learned my addr 172.21.15.27:0/1963829526 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:11.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.559+0000 7f9407fff700 1 --2- 172.21.15.27:0/1963829526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94080b07b0 0x7f94080c7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:11.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.559+0000 7f940d19c700 1 --2- 172.21.15.27:0/1963829526 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94080b2b90 0x7f9408169b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:11.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.559+0000 7f940c99b700 1 -- 172.21.15.27:0/1963829526 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94080b2b90 msgr2=0x7f9408169b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:11.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.559+0000 7f940c99b700 1 --2- 172.21.15.27:0/1963829526 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94080b2b90 0x7f9408169b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.559+0000 7f940c99b700 1 -- 172.21.15.27:0/1963829526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94080b07b0 msgr2=0x7f94080c7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:11.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.559+0000 7f940c99b700 1 --2- 172.21.15.27:0/1963829526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94080b07b0 0x7f94080c7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.559+0000 7f940c99b700 1 -- 172.21.15.27:0/1963829526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9400009ae0 con 0x7f94080b03e0 2024-01-11T08:30:11.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.559+0000 7f940c99b700 1 --2- 172.21.15.27:0/1963829526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94080b03e0 0x7f94080c75c0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f93fc00c590 tx=0x7f93fc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:11.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.560+0000 7f9404ff9700 1 -- 172.21.15.27:0/1963829526 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc011590 con 0x7f94080b03e0 2024-01-11T08:30:11.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.560+0000 7f9404ff9700 1 -- 172.21.15.27:0/1963829526 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc011710 con 0x7f94080b03e0 2024-01-11T08:30:11.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.560+0000 7f9404ff9700 1 -- 172.21.15.27:0/1963829526 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc011ad0 con 0x7f94080b03e0 2024-01-11T08:30:11.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.560+0000 7f940ebff700 1 -- 172.21.15.27:0/1963829526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94080c81a0 con 0x7f94080b03e0 2024-01-11T08:30:11.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.560+0000 7f940ebff700 1 -- 172.21.15.27:0/1963829526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94080c8760 con 0x7f94080b03e0 2024-01-11T08:30:11.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.561+0000 7f940ebff700 1 -- 172.21.15.27:0/1963829526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94080718d0 con 0x7f94080b03e0 2024-01-11T08:30:11.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.562+0000 7f9404ff9700 1 -- 172.21.15.27:0/1963829526 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f93fc01b3b0 con 0x7f94080b03e0 2024-01-11T08:30:11.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.563+0000 7f9404ff9700 1 --2- 172.21.15.27:0/1963829526 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f93ec066f30 0x7f93ec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:11.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.563+0000 7f9404ff9700 1 -- 172.21.15.27:0/1963829526 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 4909+0+0 (secure 0 0 0) 0x7f93fc083620 con 0x7f94080b03e0 2024-01-11T08:30:11.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.563+0000 7f9407fff700 1 --2- 172.21.15.27:0/1963829526 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f93ec066f30 0x7f93ec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:11.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.564+0000 7f9407fff700 1 --2- 172.21.15.27:0/1963829526 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f93ec066f30 0x7f93ec0693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f94080c9370 tx=0x7f93f40091a0 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:11.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.567+0000 7f9404ff9700 1 -- 172.21.15.27:0/1963829526 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f93fc050bb0 con 0x7f94080b03e0 2024-01-11T08:30:11.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.772+0000 7f940ebff700 1 -- 172.21.15.27:0/1963829526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94080cd3c0 con 0x7f94080b03e0 2024-01-11T08:30:11.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.772+0000 7f9404ff9700 1 -- 172.21.15.27:0/1963829526 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+12184 (secure 0 0 0) 0x7f93fc017020 con 0x7f94080b03e0 2024-01-11T08:30:11.772 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:30:11.772 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":141,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:30:10.799495+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:30:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.775+0000 7f940ebff700 1 -- 172.21.15.27:0/1963829526 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f93ec066f30 msgr2=0x7f93ec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.775+0000 7f940ebff700 1 --2- 172.21.15.27:0/1963829526 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f93ec066f30 0x7f93ec0693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f94080c9370 tx=0x7f93f40091a0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.775+0000 7f940ebff700 1 -- 172.21.15.27:0/1963829526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94080b03e0 msgr2=0x7f94080c75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.775+0000 7f940ebff700 1 --2- 172.21.15.27:0/1963829526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94080b03e0 0x7f94080c75c0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f93fc00c590 tx=0x7f93fc00c8a0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.775+0000 7f940ebff700 1 -- 172.21.15.27:0/1963829526 shutdown_connections 2024-01-11T08:30:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.775+0000 7f940ebff700 1 --2- 172.21.15.27:0/1963829526 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f93ec066f30 0x7f93ec0693e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.775+0000 7f940ebff700 1 --2- 172.21.15.27:0/1963829526 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94080b2b90 0x7f9408169b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.775+0000 7f940ebff700 1 --2- 172.21.15.27:0/1963829526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94080b03e0 0x7f94080c75c0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.775+0000 7f940ebff700 1 --2- 172.21.15.27:0/1963829526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94080b07b0 0x7f94080c7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.775+0000 7f940ebff700 1 -- 172.21.15.27:0/1963829526 >> 172.21.15.27:0/1963829526 conn(0x7f940806ed40 msgr2=0x7f94080ce4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.775+0000 7f940ebff700 1 -- 172.21.15.27:0/1963829526 shutdown_connections 2024-01-11T08:30:11.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.775+0000 7f940ebff700 1 -- 172.21.15.27:0/1963829526 wait complete. 2024-01-11T08:30:11.792 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-11T08:30:11.792 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T08:30:11.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.996+0000 7f461d9ff700 1 Processor -- start 2024-01-11T08:30:11.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.997+0000 7f461d9ff700 1 -- start start 2024-01-11T08:30:11.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.998+0000 7f461d9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f46180ccd70 0x7f46180cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:11.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.998+0000 7f461d9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f461815ae70 0x7f461815b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:11.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.998+0000 7f461d9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46180cf850 con 0x7f461815a1b0 2024-01-11T08:30:11.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.998+0000 7f461d9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f461815ac90 con 0x7f46180ccd70 2024-01-11T08:30:11.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.998+0000 7f461d9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46180cec40 con 0x7f461815ae70 2024-01-11T08:30:11.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.998+0000 7f46167fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f46180ccd70 0x7f46180cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:11.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.998+0000 7f4616ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f461815a1b0 0x7f461815a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:32802/0 (socket says 172.21.15.27:32802) 2024-01-11T08:30:11.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.998+0000 7f4616ffd700 1 -- 172.21.15.27:0/1553363410 learned_addr learned my addr 172.21.15.27:0/1553363410 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:30:11.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.999+0000 7f46177fe700 1 --2- 172.21.15.27:0/1553363410 >> v2:172.21.15.77:3300/0 conn(0x7f461815ae70 0x7f461815b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:11.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.999+0000 7f46167fc700 1 -- 172.21.15.27:0/1553363410 >> v2:172.21.15.77:3300/0 conn(0x7f461815ae70 msgr2=0x7f461815b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:11.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.999+0000 7f46167fc700 1 --2- 172.21.15.27:0/1553363410 >> v2:172.21.15.77:3300/0 conn(0x7f461815ae70 0x7f461815b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:11.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.999+0000 7f46167fc700 1 -- 172.21.15.27:0/1553363410 >> v1:172.21.15.27:6789/0 conn(0x7f461815a1b0 legacy=0x7f461815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:11.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.999+0000 7f46167fc700 1 -- 172.21.15.27:0/1553363410 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46180cef20 con 0x7f46180ccd70 2024-01-11T08:30:11.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.999+0000 7f4615ffb700 1 -- 172.21.15.27:0/1553363410 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3795111836 0 0) 0x7f46180cf850 con 0x7f461815a1b0 2024-01-11T08:30:11.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.999+0000 7f46167fc700 1 --2- 172.21.15.27:0/1553363410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46180ccd70 0x7f46180cd210 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f4618089810 tx=0x7f460c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2ee74afbea68f9b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:11.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.999+0000 7f4615ffb700 1 -- 172.21.15.27:0/1553363410 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c00b6b0 con 0x7f46180ccd70 2024-01-11T08:30:11.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.999+0000 7f4615ffb700 1 -- 172.21.15.27:0/1553363410 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460c00b810 con 0x7f46180ccd70 2024-01-11T08:30:11.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:11.999+0000 7f4615ffb700 1 -- 172.21.15.27:0/1553363410 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c00bbb0 con 0x7f46180ccd70 2024-01-11T08:30:12.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.000+0000 7f461d9ff700 1 -- 172.21.15.27:0/1553363410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46180ccd70 msgr2=0x7f46180cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:12.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.000+0000 7f461d9ff700 1 --2- 172.21.15.27:0/1553363410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46180ccd70 0x7f46180cd210 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f4618089810 tx=0x7f460c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.000+0000 7f461d9ff700 1 -- 172.21.15.27:0/1553363410 shutdown_connections 2024-01-11T08:30:12.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.000+0000 7f461d9ff700 1 --2- 172.21.15.27:0/1553363410 >> v2:172.21.15.77:3300/0 conn(0x7f461815ae70 0x7f461815b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.000+0000 7f461d9ff700 1 --2- 172.21.15.27:0/1553363410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46180ccd70 0x7f46180cd210 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.000+0000 7f461d9ff700 1 -- 172.21.15.27:0/1553363410 >> 172.21.15.27:0/1553363410 conn(0x7f461806ed40 msgr2=0x7f461806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:12.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.000+0000 7f461d9ff700 1 -- 172.21.15.27:0/1553363410 shutdown_connections 2024-01-11T08:30:12.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.000+0000 7f461d9ff700 1 -- 172.21.15.27:0/1553363410 wait complete. 2024-01-11T08:30:12.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.002+0000 7f461d9ff700 1 Processor -- start 2024-01-11T08:30:12.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.002+0000 7f461d9ff700 1 -- start start 2024-01-11T08:30:12.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.003+0000 7f461d9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46180ccd70 0x7f46180a6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:12.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.003+0000 7f461d9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f461815a1b0 0x7f461815c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:12.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.003+0000 7f461d9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f461815ae70 0x7f46180c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:12.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.003+0000 7f461d9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46180b50e0 con 0x7f46180ccd70 2024-01-11T08:30:12.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.003+0000 7f461d9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46180b44d0 con 0x7f461815ae70 2024-01-11T08:30:12.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.003+0000 7f461d9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46180b47b0 con 0x7f461815a1b0 2024-01-11T08:30:12.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.003+0000 7f4616ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46180ccd70 0x7f46180a6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:12.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.003+0000 7f4616ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46180ccd70 0x7f46180a6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53800/0 (socket says 172.21.15.27:53800) 2024-01-11T08:30:12.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.003+0000 7f4616ffd700 1 -- 172.21.15.27:0/3408844584 learned_addr learned my addr 172.21.15.27:0/3408844584 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:12.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.003+0000 7f46177fe700 1 --2- 172.21.15.27:0/3408844584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f461815ae70 0x7f46180c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:12.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.003+0000 7f46167fc700 1 --2- 172.21.15.27:0/3408844584 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f461815a1b0 0x7f461815c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:12.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.003+0000 7f4616ffd700 1 -- 172.21.15.27:0/3408844584 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f461815a1b0 msgr2=0x7f461815c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:12.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.003+0000 7f4616ffd700 1 --2- 172.21.15.27:0/3408844584 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f461815a1b0 0x7f461815c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.003+0000 7f4616ffd700 1 -- 172.21.15.27:0/3408844584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f461815ae70 msgr2=0x7f46180c8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:12.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.003+0000 7f4616ffd700 1 --2- 172.21.15.27:0/3408844584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f461815ae70 0x7f46180c8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.003+0000 7f4616ffd700 1 -- 172.21.15.27:0/3408844584 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460c009420 con 0x7f46180ccd70 2024-01-11T08:30:12.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.004+0000 7f4616ffd700 1 --2- 172.21.15.27:0/3408844584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46180ccd70 0x7f46180a6f10 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f460800c7e0 tx=0x7f4608007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:12.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.004+0000 7f45feffd700 1 -- 172.21.15.27:0/3408844584 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4608005b30 con 0x7f46180ccd70 2024-01-11T08:30:12.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.004+0000 7f45feffd700 1 -- 172.21.15.27:0/3408844584 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4608005c90 con 0x7f46180ccd70 2024-01-11T08:30:12.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.004+0000 7f461d9ff700 1 -- 172.21.15.27:0/3408844584 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46180b4a90 con 0x7f46180ccd70 2024-01-11T08:30:12.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.004+0000 7f45feffd700 1 -- 172.21.15.27:0/3408844584 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4608007300 con 0x7f46180ccd70 2024-01-11T08:30:12.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.004+0000 7f461d9ff700 1 -- 172.21.15.27:0/3408844584 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46180b4130 con 0x7f46180ccd70 2024-01-11T08:30:12.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.005+0000 7f461d9ff700 1 -- 172.21.15.27:0/3408844584 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f461804c3c0 con 0x7f46180ccd70 2024-01-11T08:30:12.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.006+0000 7f45feffd700 1 -- 172.21.15.27:0/3408844584 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f46080076b0 con 0x7f46180ccd70 2024-01-11T08:30:12.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.007+0000 7f45feffd700 1 --2- 172.21.15.27:0/3408844584 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4604066f30 0x7f46040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:12.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.007+0000 7f45feffd700 1 -- 172.21.15.27:0/3408844584 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 4909+0+0 (secure 0 0 0) 0x7f4608083820 con 0x7f46180ccd70 2024-01-11T08:30:12.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.008+0000 7f46167fc700 1 --2- 172.21.15.27:0/3408844584 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4604066f30 0x7f46040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:12.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.008+0000 7f46167fc700 1 --2- 172.21.15.27:0/3408844584 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4604066f30 0x7f46040693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f460c0093f0 tx=0x7f460c017000 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:12.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.011+0000 7f45feffd700 1 -- 172.21.15.27:0/3408844584 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4608050d10 con 0x7f46180ccd70 2024-01-11T08:30:12.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.215+0000 7f461d9ff700 1 -- 172.21.15.27:0/3408844584 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f461815ba70 con 0x7f46180ccd70 2024-01-11T08:30:12.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.216+0000 7f45feffd700 1 -- 172.21.15.27:0/3408844584 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v141) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4608054330 con 0x7f46180ccd70 2024-01-11T08:30:12.215 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T08:30:12.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.218+0000 7f461d9ff700 1 -- 172.21.15.27:0/3408844584 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4604066f30 msgr2=0x7f46040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:12.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.218+0000 7f461d9ff700 1 --2- 172.21.15.27:0/3408844584 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4604066f30 0x7f46040693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f460c0093f0 tx=0x7f460c017000 comp rx=0 tx=0).stop 2024-01-11T08:30:12.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.218+0000 7f461d9ff700 1 -- 172.21.15.27:0/3408844584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46180ccd70 msgr2=0x7f46180a6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:12.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.218+0000 7f461d9ff700 1 --2- 172.21.15.27:0/3408844584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46180ccd70 0x7f46180a6f10 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f460800c7e0 tx=0x7f4608007df0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.218+0000 7f461d9ff700 1 -- 172.21.15.27:0/3408844584 shutdown_connections 2024-01-11T08:30:12.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.218+0000 7f461d9ff700 1 --2- 172.21.15.27:0/3408844584 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4604066f30 0x7f46040693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.218+0000 7f461d9ff700 1 --2- 172.21.15.27:0/3408844584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f461815ae70 0x7f46180c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.218+0000 7f461d9ff700 1 --2- 172.21.15.27:0/3408844584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46180ccd70 0x7f46180a6f10 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.218+0000 7f461d9ff700 1 --2- 172.21.15.27:0/3408844584 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f461815a1b0 0x7f461815c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.218+0000 7f461d9ff700 1 -- 172.21.15.27:0/3408844584 >> 172.21.15.27:0/3408844584 conn(0x7f461806ed40 msgr2=0x7f46180c5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:12.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.218+0000 7f461d9ff700 1 -- 172.21.15.27:0/3408844584 shutdown_connections 2024-01-11T08:30:12.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.218+0000 7f461d9ff700 1 -- 172.21.15.27:0/3408844584 wait complete. 2024-01-11T08:30:12.235 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-11T08:30:12.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.436+0000 7f23fabff700 1 Processor -- start 2024-01-11T08:30:12.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.437+0000 7f23fabff700 1 -- start start 2024-01-11T08:30:12.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.438+0000 7f23fabff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f23f40c8620 0x7f23f40af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:12.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.438+0000 7f23fabff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f23f4069d40 0x7f23f40b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:12.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.438+0000 7f23fabff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23f4157390 con 0x7f23f40c89f0 2024-01-11T08:30:12.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.438+0000 7f23fabff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23f40adcd0 con 0x7f23f40c8620 2024-01-11T08:30:12.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.438+0000 7f23fabff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23f40afeb0 con 0x7f23f4069d40 2024-01-11T08:30:12.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.438+0000 7f23f899b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f23f40c8620 0x7f23f40af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:12.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.438+0000 7f23f899b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f23f40c8620 0x7f23f40af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53816/0 (socket says 172.21.15.27:53816) 2024-01-11T08:30:12.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.438+0000 7f23f899b700 1 -- 172.21.15.27:0/3800745396 learned_addr learned my addr 172.21.15.27:0/3800745396 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:12.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.438+0000 7f23f3fff700 1 --2- 172.21.15.27:0/3800745396 >> v2:172.21.15.77:3300/0 conn(0x7f23f4069d40 0x7f23f40b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:12.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.439+0000 7f23f899b700 1 -- 172.21.15.27:0/3800745396 >> v2:172.21.15.77:3300/0 conn(0x7f23f4069d40 msgr2=0x7f23f40b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:12.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.439+0000 7f23f899b700 1 --2- 172.21.15.27:0/3800745396 >> v2:172.21.15.77:3300/0 conn(0x7f23f4069d40 0x7f23f40b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.439+0000 7f23f899b700 1 -- 172.21.15.27:0/3800745396 >> v1:172.21.15.77:6789/0 conn(0x7f23f40c89f0 legacy=0x7f23f4159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:12.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.439+0000 7f23f899b700 1 -- 172.21.15.27:0/3800745396 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f406a200 con 0x7f23f40c8620 2024-01-11T08:30:12.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.439+0000 7f23f3fff700 1 --2- 172.21.15.27:0/3800745396 >> v2:172.21.15.77:3300/0 conn(0x7f23f4069d40 0x7f23f40b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:30:12.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.439+0000 7f23f899b700 1 --2- 172.21.15.27:0/3800745396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f40c8620 0x7f23f40af970 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f23e80099b0 tx=0x7f23e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67fbaf65d98356be server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:12.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.439+0000 7f23f37fe700 1 -- 172.21.15.27:0/3800745396 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e800b820 con 0x7f23f40c8620 2024-01-11T08:30:12.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.439+0000 7f23f37fe700 1 -- 172.21.15.27:0/3800745396 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e800b980 con 0x7f23f40c8620 2024-01-11T08:30:12.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.439+0000 7f23f37fe700 1 -- 172.21.15.27:0/3800745396 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e800bd20 con 0x7f23f40c8620 2024-01-11T08:30:12.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.440+0000 7f23fabff700 1 -- 172.21.15.27:0/3800745396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f40c8620 msgr2=0x7f23f40af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:12.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.440+0000 7f23fabff700 1 --2- 172.21.15.27:0/3800745396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f40c8620 0x7f23f40af970 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f23e80099b0 tx=0x7f23e8009250 comp rx=0 tx=0).stop 2024-01-11T08:30:12.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.440+0000 7f23fabff700 1 -- 172.21.15.27:0/3800745396 shutdown_connections 2024-01-11T08:30:12.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.440+0000 7f23fabff700 1 --2- 172.21.15.27:0/3800745396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f40c8620 0x7f23f40af970 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.440+0000 7f23fabff700 1 --2- 172.21.15.27:0/3800745396 >> v2:172.21.15.77:3300/0 conn(0x7f23f4069d40 0x7f23f40b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.440+0000 7f23fabff700 1 -- 172.21.15.27:0/3800745396 >> 172.21.15.27:0/3800745396 conn(0x7f23f40680e0 msgr2=0x7f23f4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:12.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.440+0000 7f23fabff700 1 -- 172.21.15.27:0/3800745396 shutdown_connections 2024-01-11T08:30:12.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.440+0000 7f23fabff700 1 -- 172.21.15.27:0/3800745396 wait complete. 2024-01-11T08:30:12.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.441+0000 7f23fabff700 1 Processor -- start 2024-01-11T08:30:12.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.442+0000 7f23fabff700 1 -- start start 2024-01-11T08:30:12.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.442+0000 7f23fabff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f4069d40 0x7f23f415f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:12.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.443+0000 7f23fabff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f40c8620 0x7f23f415f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:12.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.443+0000 7f23fabff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23f40c89f0 0x7f23f414dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:12.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.443+0000 7f23fabff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f41550d0 con 0x7f23f40c8620 2024-01-11T08:30:12.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.443+0000 7f23fabff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f41544c0 con 0x7f23f4069d40 2024-01-11T08:30:12.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.443+0000 7f23fabff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f41547a0 con 0x7f23f40c89f0 2024-01-11T08:30:12.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.443+0000 7f23f3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f40c8620 0x7f23f415f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:12.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.443+0000 7f23f3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f40c8620 0x7f23f415f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53828/0 (socket says 172.21.15.27:53828) 2024-01-11T08:30:12.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.443+0000 7f23f3fff700 1 -- 172.21.15.27:0/411864430 learned_addr learned my addr 172.21.15.27:0/411864430 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:12.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.443+0000 7f23f899b700 1 --2- 172.21.15.27:0/411864430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f4069d40 0x7f23f415f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:12.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.443+0000 7f23f919c700 1 --2- 172.21.15.27:0/411864430 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23f40c89f0 0x7f23f414dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:12.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.443+0000 7f23f3fff700 1 -- 172.21.15.27:0/411864430 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23f40c89f0 msgr2=0x7f23f414dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:12.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.443+0000 7f23f3fff700 1 --2- 172.21.15.27:0/411864430 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23f40c89f0 0x7f23f414dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.443+0000 7f23f3fff700 1 -- 172.21.15.27:0/411864430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f4069d40 msgr2=0x7f23f415f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:12.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.443+0000 7f23f3fff700 1 --2- 172.21.15.27:0/411864430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f4069d40 0x7f23f415f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.443+0000 7f23f3fff700 1 -- 172.21.15.27:0/411864430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e80094f0 con 0x7f23f40c8620 2024-01-11T08:30:12.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.443+0000 7f23f919c700 1 --2- 172.21.15.27:0/411864430 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23f40c89f0 0x7f23f414dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:30:12.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.444+0000 7f23f3fff700 1 --2- 172.21.15.27:0/411864430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f40c8620 0x7f23f415f800 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f23e000e8d0 tx=0x7f23e0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:12.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.444+0000 7f23f0ff9700 1 -- 172.21.15.27:0/411864430 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e0013410 con 0x7f23f40c8620 2024-01-11T08:30:12.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.444+0000 7f23f0ff9700 1 -- 172.21.15.27:0/411864430 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e0013570 con 0x7f23f40c8620 2024-01-11T08:30:12.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.444+0000 7f23f0ff9700 1 -- 172.21.15.27:0/411864430 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e0013910 con 0x7f23f40c8620 2024-01-11T08:30:12.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.444+0000 7f23fabff700 1 -- 172.21.15.27:0/411864430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f4154a80 con 0x7f23f40c8620 2024-01-11T08:30:12.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.444+0000 7f23fabff700 1 -- 172.21.15.27:0/411864430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f4137810 con 0x7f23f40c8620 2024-01-11T08:30:12.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.445+0000 7f23fabff700 1 -- 172.21.15.27:0/411864430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f4137a90 con 0x7f23f40c8620 2024-01-11T08:30:12.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.446+0000 7f23f0ff9700 1 -- 172.21.15.27:0/411864430 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f23e0013d00 con 0x7f23f40c8620 2024-01-11T08:30:12.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.447+0000 7f23f0ff9700 1 --2- 172.21.15.27:0/411864430 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f23d8066f30 0x7f23d80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:12.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.447+0000 7f23f0ff9700 1 -- 172.21.15.27:0/411864430 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 4909+0+0 (secure 0 0 0) 0x7f23e0084070 con 0x7f23f40c8620 2024-01-11T08:30:12.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.448+0000 7f23f899b700 1 --2- 172.21.15.27:0/411864430 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f23d8066f30 0x7f23d80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:12.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.448+0000 7f23f899b700 1 --2- 172.21.15.27:0/411864430 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f23d8066f30 0x7f23d80693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f23e80099b0 tx=0x7f23e8019000 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:12.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.451+0000 7f23f0ff9700 1 -- 172.21.15.27:0/411864430 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f23f4137a90 con 0x7f23f40c8620 2024-01-11T08:30:12.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.662+0000 7f23fabff700 1 -- 172.21.15.27:0/411864430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f23f4137a90 con 0x7f23f40c8620 2024-01-11T08:30:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.662+0000 7f23f0ff9700 1 -- 172.21.15.27:0/411864430 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v141) v1 ==== 84+0+65 (secure 0 0 0) 0x7f23e0020060 con 0x7f23f40c8620 2024-01-11T08:30:12.665 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:30:12.665 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T08:30:12.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:12.665 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-11T08:30:12.665 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-11T08:30:12.665 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:12.665 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T08:30:12.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.665+0000 7f23fabff700 1 -- 172.21.15.27:0/411864430 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f23d8066f30 msgr2=0x7f23d80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:12.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.665+0000 7f23fabff700 1 --2- 172.21.15.27:0/411864430 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f23d8066f30 0x7f23d80693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f23e80099b0 tx=0x7f23e8019000 comp rx=0 tx=0).stop 2024-01-11T08:30:12.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.665+0000 7f23fabff700 1 -- 172.21.15.27:0/411864430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f40c8620 msgr2=0x7f23f415f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:12.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.665+0000 7f23fabff700 1 --2- 172.21.15.27:0/411864430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f40c8620 0x7f23f415f800 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f23e000e8d0 tx=0x7f23e0008040 comp rx=0 tx=0).stop 2024-01-11T08:30:12.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.665+0000 7f23fabff700 1 -- 172.21.15.27:0/411864430 shutdown_connections 2024-01-11T08:30:12.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.665+0000 7f23fabff700 1 --2- 172.21.15.27:0/411864430 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f23d8066f30 0x7f23d80693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.665+0000 7f23fabff700 1 --2- 172.21.15.27:0/411864430 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23f40c89f0 0x7f23f414dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.665+0000 7f23fabff700 1 --2- 172.21.15.27:0/411864430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23f4069d40 0x7f23f415f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.665+0000 7f23fabff700 1 --2- 172.21.15.27:0/411864430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23f40c8620 0x7f23f415f800 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.665+0000 7f23fabff700 1 -- 172.21.15.27:0/411864430 >> 172.21.15.27:0/411864430 conn(0x7f23f40680e0 msgr2=0x7f23f41534f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:12.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.665+0000 7f23fabff700 1 -- 172.21.15.27:0/411864430 shutdown_connections 2024-01-11T08:30:12.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.665+0000 7f23fabff700 1 -- 172.21.15.27:0/411864430 wait complete. 2024-01-11T08:30:12.682 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-11T08:30:12.682 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-11T08:30:12.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.888+0000 7ff64f3ff700 1 Processor -- start 2024-01-11T08:30:12.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.888+0000 7ff64f3ff700 1 -- start start 2024-01-11T08:30:12.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.889+0000 7ff64f3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff648159250 0x7ff6480b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:12.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.889+0000 7ff64f3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff648068be0 0x7ff6480cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:12.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.889+0000 7ff64f3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64815a440 con 0x7ff6480cb410 2024-01-11T08:30:12.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.889+0000 7ff64f3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff64815ace0 con 0x7ff648068be0 2024-01-11T08:30:12.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.889+0000 7ff64f3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6480b0de0 con 0x7ff648159250 2024-01-11T08:30:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.889+0000 7ff64d99c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff648068be0 0x7ff6480cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.889+0000 7ff64d99c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff648068be0 0x7ff6480cb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53838/0 (socket says 172.21.15.27:53838) 2024-01-11T08:30:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.889+0000 7ff64d99c700 1 -- 172.21.15.27:0/1204070370 learned_addr learned my addr 172.21.15.27:0/1204070370 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.890+0000 7ff64d19b700 1 --2- 172.21.15.27:0/1204070370 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff648159250 0x7ff6480b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.890+0000 7ff63ffff700 1 -- 172.21.15.27:0/1204070370 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917631753 0 0) 0x7ff64815a440 con 0x7ff6480cb410 2024-01-11T08:30:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.890+0000 7ff64d99c700 1 -- 172.21.15.27:0/1204070370 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff648159250 msgr2=0x7ff6480b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.890+0000 7ff64d99c700 1 --2- 172.21.15.27:0/1204070370 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff648159250 0x7ff6480b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.890+0000 7ff64d99c700 1 -- 172.21.15.27:0/1204070370 >> v1:172.21.15.27:6789/0 conn(0x7ff6480cb410 legacy=0x7ff6480684d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.890+0000 7ff64d99c700 1 -- 172.21.15.27:0/1204070370 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64815a720 con 0x7ff648068be0 2024-01-11T08:30:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.890+0000 7ff64d99c700 1 --2- 172.21.15.27:0/1204070370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff648068be0 0x7ff6480cb880 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7ff640009f20 tx=0x7ff640009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f3478fc1f15f937 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.890+0000 7ff63ffff700 1 -- 172.21.15.27:0/1204070370 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64000e070 con 0x7ff648068be0 2024-01-11T08:30:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.891+0000 7ff63ffff700 1 -- 172.21.15.27:0/1204070370 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff64000ba80 con 0x7ff648068be0 2024-01-11T08:30:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.891+0000 7ff63ffff700 1 -- 172.21.15.27:0/1204070370 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6400092e0 con 0x7ff648068be0 2024-01-11T08:30:12.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.891+0000 7ff64f3ff700 1 -- 172.21.15.27:0/1204070370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff648068be0 msgr2=0x7ff6480cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:12.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.891+0000 7ff64f3ff700 1 --2- 172.21.15.27:0/1204070370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff648068be0 0x7ff6480cb880 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7ff640009f20 tx=0x7ff640009840 comp rx=0 tx=0).stop 2024-01-11T08:30:12.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.891+0000 7ff64f3ff700 1 -- 172.21.15.27:0/1204070370 shutdown_connections 2024-01-11T08:30:12.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.891+0000 7ff64f3ff700 1 --2- 172.21.15.27:0/1204070370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff648068be0 0x7ff6480cb880 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.891+0000 7ff64f3ff700 1 --2- 172.21.15.27:0/1204070370 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff648159250 0x7ff6480b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.891+0000 7ff64f3ff700 1 -- 172.21.15.27:0/1204070370 >> 172.21.15.27:0/1204070370 conn(0x7ff648067610 msgr2=0x7ff648067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.891+0000 7ff64f3ff700 1 -- 172.21.15.27:0/1204070370 shutdown_connections 2024-01-11T08:30:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.891+0000 7ff64f3ff700 1 -- 172.21.15.27:0/1204070370 wait complete. 2024-01-11T08:30:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.893+0000 7ff64f3ff700 1 Processor -- start 2024-01-11T08:30:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.893+0000 7ff64f3ff700 1 -- start start 2024-01-11T08:30:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.894+0000 7ff64f3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff648068be0 0x7ff6480b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.894+0000 7ff64f3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6480cb410 0x7ff6480c3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.894+0000 7ff64f3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff648159250 0x7ff6480c4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.894+0000 7ff64f3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff648156fd0 con 0x7ff6480cb410 2024-01-11T08:30:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.894+0000 7ff64f3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6480b3780 con 0x7ff648159250 2024-01-11T08:30:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.894+0000 7ff64f3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6480b3c10 con 0x7ff648068be0 2024-01-11T08:30:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.894+0000 7ff64c99a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6480cb410 0x7ff6480c3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.894+0000 7ff64c99a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6480cb410 0x7ff6480c3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53844/0 (socket says 172.21.15.27:53844) 2024-01-11T08:30:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.894+0000 7ff64c99a700 1 -- 172.21.15.27:0/2391733186 learned_addr learned my addr 172.21.15.27:0/2391733186 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.894+0000 7ff64d99c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff648159250 0x7ff6480c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.894+0000 7ff64d19b700 1 --2- 172.21.15.27:0/2391733186 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff648068be0 0x7ff6480b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.895+0000 7ff64c99a700 1 -- 172.21.15.27:0/2391733186 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff648068be0 msgr2=0x7ff6480b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.895+0000 7ff64c99a700 1 --2- 172.21.15.27:0/2391733186 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff648068be0 0x7ff6480b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.895+0000 7ff64c99a700 1 -- 172.21.15.27:0/2391733186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff648159250 msgr2=0x7ff6480c4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.895+0000 7ff64c99a700 1 --2- 172.21.15.27:0/2391733186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff648159250 0x7ff6480c4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.895+0000 7ff64c99a700 1 -- 172.21.15.27:0/2391733186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff640009ae0 con 0x7ff6480cb410 2024-01-11T08:30:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.895+0000 7ff64c99a700 1 --2- 172.21.15.27:0/2391733186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6480cb410 0x7ff6480c3f00 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7ff63800c7e0 tx=0x7ff638007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.895+0000 7ff63d7fa700 1 -- 172.21.15.27:0/2391733186 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff638011590 con 0x7ff6480cb410 2024-01-11T08:30:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.895+0000 7ff63d7fa700 1 -- 172.21.15.27:0/2391733186 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff638011710 con 0x7ff6480cb410 2024-01-11T08:30:12.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.895+0000 7ff64f3ff700 1 -- 172.21.15.27:0/2391733186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6480b2b70 con 0x7ff6480cb410 2024-01-11T08:30:12.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.895+0000 7ff63d7fa700 1 -- 172.21.15.27:0/2391733186 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff638011ad0 con 0x7ff6480cb410 2024-01-11T08:30:12.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.895+0000 7ff64f3ff700 1 -- 172.21.15.27:0/2391733186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6480b3130 con 0x7ff6480cb410 2024-01-11T08:30:12.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.896+0000 7ff64f3ff700 1 -- 172.21.15.27:0/2391733186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6480b3930 con 0x7ff6480cb410 2024-01-11T08:30:12.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.897+0000 7ff63d7fa700 1 -- 172.21.15.27:0/2391733186 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff63801b3b0 con 0x7ff6480cb410 2024-01-11T08:30:12.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.898+0000 7ff63d7fa700 1 --2- 172.21.15.27:0/2391733186 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff62c066f30 0x7ff62c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:12.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.898+0000 7ff63d7fa700 1 -- 172.21.15.27:0/2391733186 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 4909+0+0 (secure 0 0 0) 0x7ff6380836e0 con 0x7ff6480cb410 2024-01-11T08:30:12.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.898+0000 7ff64d19b700 1 --2- 172.21.15.27:0/2391733186 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff62c066f30 0x7ff62c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:12.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.899+0000 7ff64d19b700 1 --2- 172.21.15.27:0/2391733186 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff62c066f30 0x7ff62c0693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff644006fd0 tx=0x7ff644008040 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:12.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:12.902+0000 7ff63d7fa700 1 -- 172.21.15.27:0/2391733186 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff6480b3930 con 0x7ff6480cb410 2024-01-11T08:30:13.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:13.108+0000 7ff64f3ff700 1 -- 172.21.15.27:0/2391733186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff6480b38f0 con 0x7ff6480cb410 2024-01-11T08:30:14.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.106+0000 7ff63d7fa700 1 -- 172.21.15.27:0/2391733186 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v142) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff638017020 con 0x7ff6480cb410 2024-01-11T08:30:14.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.112+0000 7ff64f3ff700 1 -- 172.21.15.27:0/2391733186 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff62c066f30 msgr2=0x7ff62c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:14.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.112+0000 7ff64f3ff700 1 --2- 172.21.15.27:0/2391733186 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff62c066f30 0x7ff62c0693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff644006fd0 tx=0x7ff644008040 comp rx=0 tx=0).stop 2024-01-11T08:30:14.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.112+0000 7ff64f3ff700 1 -- 172.21.15.27:0/2391733186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6480cb410 msgr2=0x7ff6480c3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:14.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.112+0000 7ff64f3ff700 1 --2- 172.21.15.27:0/2391733186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6480cb410 0x7ff6480c3f00 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7ff63800c7e0 tx=0x7ff638007df0 comp rx=0 tx=0).stop 2024-01-11T08:30:14.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.113+0000 7ff64f3ff700 1 -- 172.21.15.27:0/2391733186 shutdown_connections 2024-01-11T08:30:14.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.113+0000 7ff64f3ff700 1 --2- 172.21.15.27:0/2391733186 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff62c066f30 0x7ff62c0693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:14.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.113+0000 7ff64f3ff700 1 --2- 172.21.15.27:0/2391733186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff648159250 0x7ff6480c4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:14.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.113+0000 7ff64f3ff700 1 --2- 172.21.15.27:0/2391733186 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff648068be0 0x7ff6480b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:14.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.113+0000 7ff64f3ff700 1 --2- 172.21.15.27:0/2391733186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6480cb410 0x7ff6480c3f00 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:14.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.113+0000 7ff64f3ff700 1 -- 172.21.15.27:0/2391733186 >> 172.21.15.27:0/2391733186 conn(0x7ff648067610 msgr2=0x7ff648158810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:14.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.114+0000 7ff64f3ff700 1 -- 172.21.15.27:0/2391733186 shutdown_connections 2024-01-11T08:30:14.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.114+0000 7ff64f3ff700 1 -- 172.21.15.27:0/2391733186 wait complete. 2024-01-11T08:30:14.117 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-11T08:30:14.136 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-11T08:30:14.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.347+0000 7f8d169ff700 1 Processor -- start 2024-01-11T08:30:14.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.348+0000 7f8d169ff700 1 -- start start 2024-01-11T08:30:14.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.349+0000 7f8d169ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8d1006bba0 0x7f8d100cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:14.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.349+0000 7f8d169ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d1015db00 0x7f8d1015df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:14.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.349+0000 7f8d169ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1015d670 con 0x7f8d1006bf70 2024-01-11T08:30:14.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.349+0000 7f8d169ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d10069fc0 con 0x7f8d1015db00 2024-01-11T08:30:14.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.349+0000 7f8d169ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d100cce10 con 0x7f8d1006bba0 2024-01-11T08:30:14.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.349+0000 7f8d14f9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d1015db00 0x7f8d1015df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:14.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.349+0000 7f8d14f9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d1015db00 0x7f8d1015df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34806/0 (socket says 172.21.15.27:34806) 2024-01-11T08:30:14.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.349+0000 7f8d14f9c700 1 -- 172.21.15.27:0/1775510058 learned_addr learned my addr 172.21.15.27:0/1775510058 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:14.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.349+0000 7f8d0f7fe700 1 --1- 172.21.15.27:0/1775510058 >> v1:172.21.15.27:6789/0 conn(0x7f8d1006bf70 0x7f8d10159e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38656/0 (socket says 172.21.15.27:38656) 2024-01-11T08:30:14.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.349+0000 7f8d0ffff700 1 --2- 172.21.15.27:0/1775510058 >> v2:172.21.15.77:3300/0 conn(0x7f8d1006bba0 0x7f8d100cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:14.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.350+0000 7f8d14f9c700 1 -- 172.21.15.27:0/1775510058 >> v2:172.21.15.77:3300/0 conn(0x7f8d1006bba0 msgr2=0x7f8d100cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:14.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.350+0000 7f8d14f9c700 1 --2- 172.21.15.27:0/1775510058 >> v2:172.21.15.77:3300/0 conn(0x7f8d1006bba0 0x7f8d100cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:14.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.350+0000 7f8d14f9c700 1 -- 172.21.15.27:0/1775510058 >> v1:172.21.15.27:6789/0 conn(0x7f8d1006bf70 legacy=0x7f8d10159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:14.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.350+0000 7f8d14f9c700 1 -- 172.21.15.27:0/1775510058 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1015a600 con 0x7f8d1015db00 2024-01-11T08:30:14.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.350+0000 7f8d0effd700 1 -- 172.21.15.27:0/1775510058 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303169745 0 0) 0x7f8d1015d670 con 0x7f8d1006bf70 2024-01-11T08:30:14.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.350+0000 7f8d14f9c700 1 --2- 172.21.15.27:0/1775510058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d1015db00 0x7f8d1015df20 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f8d100adc70 tx=0x7f8d08009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ba1e8d396fa7ce server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:14.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.350+0000 7f8d0effd700 1 -- 172.21.15.27:0/1775510058 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0800e070 con 0x7f8d1015db00 2024-01-11T08:30:14.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.350+0000 7f8d0effd700 1 -- 172.21.15.27:0/1775510058 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0800ba20 con 0x7f8d1015db00 2024-01-11T08:30:14.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.350+0000 7f8d0effd700 1 -- 172.21.15.27:0/1775510058 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d080092e0 con 0x7f8d1015db00 2024-01-11T08:30:14.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.351+0000 7f8d169ff700 1 -- 172.21.15.27:0/1775510058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d1015db00 msgr2=0x7f8d1015df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:14.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.351+0000 7f8d169ff700 1 --2- 172.21.15.27:0/1775510058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d1015db00 0x7f8d1015df20 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f8d100adc70 tx=0x7f8d08009840 comp rx=0 tx=0).stop 2024-01-11T08:30:14.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.351+0000 7f8d169ff700 1 -- 172.21.15.27:0/1775510058 shutdown_connections 2024-01-11T08:30:14.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.351+0000 7f8d169ff700 1 --2- 172.21.15.27:0/1775510058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d1015db00 0x7f8d1015df20 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:14.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.351+0000 7f8d169ff700 1 --2- 172.21.15.27:0/1775510058 >> v2:172.21.15.77:3300/0 conn(0x7f8d1006bba0 0x7f8d100cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:14.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.351+0000 7f8d169ff700 1 -- 172.21.15.27:0/1775510058 >> 172.21.15.27:0/1775510058 conn(0x7f8d100ce3e0 msgr2=0x7f8d100ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:14.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.351+0000 7f8d169ff700 1 -- 172.21.15.27:0/1775510058 shutdown_connections 2024-01-11T08:30:14.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.351+0000 7f8d169ff700 1 -- 172.21.15.27:0/1775510058 wait complete. 2024-01-11T08:30:14.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.353+0000 7f8d169ff700 1 Processor -- start 2024-01-11T08:30:14.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.353+0000 7f8d169ff700 1 -- start start 2024-01-11T08:30:14.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.354+0000 7f8d169ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d1006bba0 0x7f8d100c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:14.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.354+0000 7f8d169ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d1006bf70 0x7f8d10148db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:14.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.354+0000 7f8d169ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d1015db00 0x7f8d10159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:14.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.354+0000 7f8d169ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d100b1b60 con 0x7f8d1006bba0 2024-01-11T08:30:14.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.354+0000 7f8d169ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d100c5a50 con 0x7f8d1006bf70 2024-01-11T08:30:14.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.354+0000 7f8d169ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d100c5d30 con 0x7f8d1015db00 2024-01-11T08:30:14.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.354+0000 7f8d0ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d1006bba0 0x7f8d100c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:14.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.354+0000 7f8d0ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d1006bba0 0x7f8d100c76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34812/0 (socket says 172.21.15.27:34812) 2024-01-11T08:30:14.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.354+0000 7f8d0ffff700 1 -- 172.21.15.27:0/3511488462 learned_addr learned my addr 172.21.15.27:0/3511488462 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:14.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.354+0000 7f8d0f7fe700 1 --2- 172.21.15.27:0/3511488462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d1006bf70 0x7f8d10148db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:14.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.354+0000 7f8d14f9c700 1 --2- 172.21.15.27:0/3511488462 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d1015db00 0x7f8d10159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:14.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.354+0000 7f8d0ffff700 1 -- 172.21.15.27:0/3511488462 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d1015db00 msgr2=0x7f8d10159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:14.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.354+0000 7f8d0ffff700 1 --2- 172.21.15.27:0/3511488462 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d1015db00 0x7f8d10159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:14.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.354+0000 7f8d0ffff700 1 -- 172.21.15.27:0/3511488462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d1006bf70 msgr2=0x7f8d10148db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:14.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.354+0000 7f8d0ffff700 1 --2- 172.21.15.27:0/3511488462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d1006bf70 0x7f8d10148db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:14.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.354+0000 7f8d0ffff700 1 -- 172.21.15.27:0/3511488462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d08009ae0 con 0x7f8d1006bba0 2024-01-11T08:30:14.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.354+0000 7f8d14f9c700 1 --2- 172.21.15.27:0/3511488462 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d1015db00 0x7f8d10159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:30:14.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.354+0000 7f8d0f7fe700 1 --2- 172.21.15.27:0/3511488462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d1006bf70 0x7f8d10148db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:30:14.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.355+0000 7f8d0ffff700 1 --2- 172.21.15.27:0/3511488462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d1006bba0 0x7f8d100c76a0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000ec80 tx=0x7f8d000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:14.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.355+0000 7f8cf7fff700 1 -- 172.21.15.27:0/3511488462 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0000c9d0 con 0x7f8d1006bba0 2024-01-11T08:30:14.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.355+0000 7f8cf7fff700 1 -- 172.21.15.27:0/3511488462 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0000cb30 con 0x7f8d1006bba0 2024-01-11T08:30:14.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.355+0000 7f8cf7fff700 1 -- 172.21.15.27:0/3511488462 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d00013470 con 0x7f8d1006bba0 2024-01-11T08:30:14.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.355+0000 7f8d169ff700 1 -- 172.21.15.27:0/3511488462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d100c6010 con 0x7f8d1006bba0 2024-01-11T08:30:14.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.355+0000 7f8d169ff700 1 -- 172.21.15.27:0/3511488462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d100c71f0 con 0x7f8d1006bba0 2024-01-11T08:30:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.356+0000 7f8d169ff700 1 -- 172.21.15.27:0/3511488462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d100cd720 con 0x7f8d1006bba0 2024-01-11T08:30:14.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.357+0000 7f8cf7fff700 1 -- 172.21.15.27:0/3511488462 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8d00013820 con 0x7f8d1006bba0 2024-01-11T08:30:14.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.358+0000 7f8cf7fff700 1 --2- 172.21.15.27:0/3511488462 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8cfc066f30 0x7f8cfc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:14.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.358+0000 7f8cf7fff700 1 -- 172.21.15.27:0/3511488462 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5279+0+0 (secure 0 0 0) 0x7f8d000847b0 con 0x7f8d1006bba0 2024-01-11T08:30:14.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.359+0000 7f8d0f7fe700 1 --2- 172.21.15.27:0/3511488462 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8cfc066f30 0x7f8cfc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:14.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.359+0000 7f8d0f7fe700 1 --2- 172.21.15.27:0/3511488462 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8cfc066f30 0x7f8cfc0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8d1016cf70 tx=0x7f8d04009040 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:14.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.363+0000 7f8cf7fff700 1 -- 172.21.15.27:0/3511488462 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8d00051b30 con 0x7f8d1006bba0 2024-01-11T08:30:14.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:14.567+0000 7f8d169ff700 1 -- 172.21.15.27:0/3511488462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8d100b1ca0 con 0x7f8d1006bba0 2024-01-11T08:30:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.113+0000 7f8cf7fff700 1 -- 172.21.15.27:0/3511488462 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v143) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8d00055150 con 0x7f8d1006bba0 2024-01-11T08:30:15.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.125+0000 7f8d169ff700 1 -- 172.21.15.27:0/3511488462 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8cfc066f30 msgr2=0x7f8cfc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:15.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.125+0000 7f8d169ff700 1 --2- 172.21.15.27:0/3511488462 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8cfc066f30 0x7f8cfc0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8d1016cf70 tx=0x7f8d04009040 comp rx=0 tx=0).stop 2024-01-11T08:30:15.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.125+0000 7f8d169ff700 1 -- 172.21.15.27:0/3511488462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d1006bba0 msgr2=0x7f8d100c76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:15.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.125+0000 7f8d169ff700 1 --2- 172.21.15.27:0/3511488462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d1006bba0 0x7f8d100c76a0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000ec80 tx=0x7f8d000091a0 comp rx=0 tx=0).stop 2024-01-11T08:30:15.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.125+0000 7f8d169ff700 1 -- 172.21.15.27:0/3511488462 shutdown_connections 2024-01-11T08:30:15.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.125+0000 7f8d169ff700 1 --2- 172.21.15.27:0/3511488462 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f8cfc066f30 0x7f8cfc0693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:15.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.125+0000 7f8d169ff700 1 --2- 172.21.15.27:0/3511488462 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d1015db00 0x7f8d10159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:15.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.125+0000 7f8d169ff700 1 --2- 172.21.15.27:0/3511488462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d1006bba0 0x7f8d100c76a0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:15.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.125+0000 7f8d169ff700 1 --2- 172.21.15.27:0/3511488462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d1006bf70 0x7f8d10148db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:15.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.125+0000 7f8d169ff700 1 -- 172.21.15.27:0/3511488462 >> 172.21.15.27:0/3511488462 conn(0x7f8d100ce3e0 msgr2=0x7f8d1015bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:15.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.127+0000 7f8d169ff700 1 -- 172.21.15.27:0/3511488462 shutdown_connections 2024-01-11T08:30:15.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.127+0000 7f8d169ff700 1 -- 172.21.15.27:0/3511488462 wait complete. 2024-01-11T08:30:15.132 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-11T08:30:15.154 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-11T08:30:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.359+0000 7f72247ff700 1 Processor -- start 2024-01-11T08:30:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.360+0000 7f72247ff700 1 -- start start 2024-01-11T08:30:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.361+0000 7f72247ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f721c1577e0 0x7f721c0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.361+0000 7f72247ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f721c0699d0 con 0x7f721c069540 2024-01-11T08:30:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.361+0000 7f72247ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f721c069cb0 con 0x7f721c157410 2024-01-11T08:30:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.361+0000 7f72247ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f721c069f90 con 0x7f721c1577e0 2024-01-11T08:30:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.361+0000 7f7222d9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f721c069540 0x7f721c0c6ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38664/0 (socket says 172.21.15.27:38664) 2024-01-11T08:30:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.361+0000 7f7222d9c700 1 -- 172.21.15.27:0/3526398570 learned_addr learned my addr 172.21.15.27:0/3526398570 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:30:15.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.361+0000 7f7221d9a700 1 --2- 172.21.15.27:0/3526398570 >> v2:172.21.15.77:3300/0 conn(0x7f721c1577e0 0x7f721c0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:15.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.362+0000 7f7221599700 1 -- 172.21.15.27:0/3526398570 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273613611 0 0) 0x7f721c0699d0 con 0x7f721c069540 2024-01-11T08:30:15.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.362+0000 7f7221599700 1 -- 172.21.15.27:0/3526398570 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7208003570 con 0x7f721c069540 2024-01-11T08:30:15.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.362+0000 7f7221599700 1 -- 172.21.15.27:0/3526398570 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934364496 0 0) 0x7f721c069cb0 con 0x7f721c157410 2024-01-11T08:30:15.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.362+0000 7f7221599700 1 -- 172.21.15.27:0/3526398570 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f721c0699d0 con 0x7f721c157410 2024-01-11T08:30:15.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.362+0000 7f7221d9a700 1 -- 172.21.15.27:0/3526398570 >> v1:172.21.15.77:6789/0 conn(0x7f721c157410 legacy=0x7f721c0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:15.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.362+0000 7f7221d9a700 1 -- 172.21.15.27:0/3526398570 >> v1:172.21.15.27:6789/0 conn(0x7f721c069540 legacy=0x7f721c0c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:15.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.362+0000 7f7221d9a700 1 -- 172.21.15.27:0/3526398570 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c06a270 con 0x7f721c1577e0 2024-01-11T08:30:15.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.362+0000 7f7221d9a700 1 --2- 172.21.15.27:0/3526398570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f721c1577e0 0x7f721c0ae000 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f720c0098e0 tx=0x7f720c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d621c7092ea1966 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.362+0000 7f7221599700 1 -- 172.21.15.27:0/3526398570 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c011630 con 0x7f721c1577e0 2024-01-11T08:30:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.363+0000 7f7221599700 1 -- 172.21.15.27:0/3526398570 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f720c011790 con 0x7f721c1577e0 2024-01-11T08:30:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.363+0000 7f7221599700 1 -- 172.21.15.27:0/3526398570 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c011b30 con 0x7f721c1577e0 2024-01-11T08:30:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.363+0000 7f72247ff700 1 -- 172.21.15.27:0/3526398570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f721c1577e0 msgr2=0x7f721c0ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.363+0000 7f72247ff700 1 --2- 172.21.15.27:0/3526398570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f721c1577e0 0x7f721c0ae000 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f720c0098e0 tx=0x7f720c0091a0 comp rx=0 tx=0).stop 2024-01-11T08:30:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.363+0000 7f72247ff700 1 -- 172.21.15.27:0/3526398570 shutdown_connections 2024-01-11T08:30:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.363+0000 7f72247ff700 1 --2- 172.21.15.27:0/3526398570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f721c1577e0 0x7f721c0ae000 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.363+0000 7f72247ff700 1 -- 172.21.15.27:0/3526398570 >> 172.21.15.27:0/3526398570 conn(0x7f721c0680a0 msgr2=0x7f721c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.363+0000 7f72247ff700 1 -- 172.21.15.27:0/3526398570 shutdown_connections 2024-01-11T08:30:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.363+0000 7f72247ff700 1 -- 172.21.15.27:0/3526398570 wait complete. 2024-01-11T08:30:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.365+0000 7f72247ff700 1 Processor -- start 2024-01-11T08:30:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.365+0000 7f72247ff700 1 -- start start 2024-01-11T08:30:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.366+0000 7f72247ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f721c069540 0x7f721c0cb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.366+0000 7f72247ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f721c157410 0x7f721c0caad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.366+0000 7f72247ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f721c1577e0 0x7f721c0ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.366+0000 7f72247ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c0cbe60 con 0x7f721c069540 2024-01-11T08:30:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.366+0000 7f72247ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c156810 con 0x7f721c157410 2024-01-11T08:30:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.366+0000 7f72247ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721c0cb010 con 0x7f721c1577e0 2024-01-11T08:30:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.366+0000 7f722259b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f721c069540 0x7f721c0cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.366+0000 7f722259b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f721c069540 0x7f721c0cb920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34822/0 (socket says 172.21.15.27:34822) 2024-01-11T08:30:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.366+0000 7f722259b700 1 -- 172.21.15.27:0/3496343589 learned_addr learned my addr 172.21.15.27:0/3496343589 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.366+0000 7f7221d9a700 1 --2- 172.21.15.27:0/3496343589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f721c157410 0x7f721c0caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.366+0000 7f7222d9c700 1 --2- 172.21.15.27:0/3496343589 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f721c1577e0 0x7f721c0ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.367+0000 7f722259b700 1 -- 172.21.15.27:0/3496343589 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f721c1577e0 msgr2=0x7f721c0ca140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.367+0000 7f722259b700 1 --2- 172.21.15.27:0/3496343589 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f721c1577e0 0x7f721c0ca140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.367+0000 7f722259b700 1 -- 172.21.15.27:0/3496343589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f721c157410 msgr2=0x7f721c0caad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.367+0000 7f722259b700 1 --2- 172.21.15.27:0/3496343589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f721c157410 0x7f721c0caad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:15.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.367+0000 7f722259b700 1 -- 172.21.15.27:0/3496343589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720c009420 con 0x7f721c069540 2024-01-11T08:30:15.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.367+0000 7f722259b700 1 --2- 172.21.15.27:0/3496343589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f721c069540 0x7f721c0cb920 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f721800c7f0 tx=0x7f7218007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:15.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.367+0000 7f72127fc700 1 -- 172.21.15.27:0/3496343589 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7218001530 con 0x7f721c069540 2024-01-11T08:30:15.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.367+0000 7f72127fc700 1 -- 172.21.15.27:0/3496343589 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7218001690 con 0x7f721c069540 2024-01-11T08:30:15.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.367+0000 7f72127fc700 1 -- 172.21.15.27:0/3496343589 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7218001a50 con 0x7f721c069540 2024-01-11T08:30:15.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.367+0000 7f72247ff700 1 -- 172.21.15.27:0/3496343589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f721c169e80 con 0x7f721c069540 2024-01-11T08:30:15.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.367+0000 7f72247ff700 1 -- 172.21.15.27:0/3496343589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721c16a440 con 0x7f721c069540 2024-01-11T08:30:15.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.368+0000 7f72247ff700 1 -- 172.21.15.27:0/3496343589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f721c04c3c0 con 0x7f721c069540 2024-01-11T08:30:15.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.370+0000 7f72127fc700 1 -- 172.21.15.27:0/3496343589 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f721801b3b0 con 0x7f721c069540 2024-01-11T08:30:15.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.371+0000 7f72127fc700 1 --2- 172.21.15.27:0/3496343589 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7200066f30 0x7f72000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:15.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.371+0000 7f72127fc700 1 -- 172.21.15.27:0/3496343589 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5661+0+0 (secure 0 0 0) 0x7f7218083db0 con 0x7f721c069540 2024-01-11T08:30:15.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.371+0000 7f7221d9a700 1 --2- 172.21.15.27:0/3496343589 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7200066f30 0x7f72000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:15.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.372+0000 7f7221d9a700 1 --2- 172.21.15.27:0/3496343589 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7200066f30 0x7f72000693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f720c0098e0 tx=0x7f720c017000 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:15.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.376+0000 7f72127fc700 1 -- 172.21.15.27:0/3496343589 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7218088070 con 0x7f721c069540 2024-01-11T08:30:15.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:15.622+0000 7f72247ff700 1 -- 172.21.15.27:0/3496343589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f721c154100 con 0x7f721c069540 2024-01-11T08:30:16.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.142+0000 7f72127fc700 1 -- 172.21.15.27:0/3496343589 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v73) v1 ==== 174+0+0 (secure 0 0 0) 0x7f7218017020 con 0x7f721c069540 2024-01-11T08:30:16.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.149+0000 7f72247ff700 1 -- 172.21.15.27:0/3496343589 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7200066f30 msgr2=0x7f72000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:16.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.149+0000 7f72247ff700 1 --2- 172.21.15.27:0/3496343589 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7200066f30 0x7f72000693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f720c0098e0 tx=0x7f720c017000 comp rx=0 tx=0).stop 2024-01-11T08:30:16.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.149+0000 7f72247ff700 1 -- 172.21.15.27:0/3496343589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f721c069540 msgr2=0x7f721c0cb920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:16.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.149+0000 7f72247ff700 1 --2- 172.21.15.27:0/3496343589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f721c069540 0x7f721c0cb920 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f721800c7f0 tx=0x7f7218007e00 comp rx=0 tx=0).stop 2024-01-11T08:30:16.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.149+0000 7f72247ff700 1 -- 172.21.15.27:0/3496343589 shutdown_connections 2024-01-11T08:30:16.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.149+0000 7f72247ff700 1 --2- 172.21.15.27:0/3496343589 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7200066f30 0x7f72000693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:16.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.149+0000 7f72247ff700 1 --2- 172.21.15.27:0/3496343589 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f721c1577e0 0x7f721c0ca140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:16.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.149+0000 7f72247ff700 1 --2- 172.21.15.27:0/3496343589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f721c069540 0x7f721c0cb920 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:16.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.149+0000 7f72247ff700 1 --2- 172.21.15.27:0/3496343589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f721c157410 0x7f721c0caad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:16.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.149+0000 7f72247ff700 1 -- 172.21.15.27:0/3496343589 >> 172.21.15.27:0/3496343589 conn(0x7f721c0680a0 msgr2=0x7f721c152d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:16.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.149+0000 7f72247ff700 1 -- 172.21.15.27:0/3496343589 shutdown_connections 2024-01-11T08:30:16.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.149+0000 7f72247ff700 1 -- 172.21.15.27:0/3496343589 wait complete. 2024-01-11T08:30:16.150 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 18 and data pool 19 2024-01-11T08:30:16.172 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:30:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.372+0000 7f14a95ff700 1 Processor -- start 2024-01-11T08:30:16.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.373+0000 7f14a95ff700 1 -- start start 2024-01-11T08:30:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.374+0000 7f14a95ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f14a40cca20 0x7f14a4159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.374+0000 7f14a95ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a415d640 con 0x7f14a40ca3e0 2024-01-11T08:30:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.374+0000 7f14a95ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a40cce60 con 0x7f14a40ca7b0 2024-01-11T08:30:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.374+0000 7f14a95ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a415ae70 con 0x7f14a40cca20 2024-01-11T08:30:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.374+0000 7f14a2ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f14a40ca3e0 0x7f14a40c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38666/0 (socket says 172.21.15.27:38666) 2024-01-11T08:30:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.374+0000 7f14a2ffd700 1 -- 172.21.15.27:0/194191853 learned_addr learned my addr 172.21.15.27:0/194191853 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:30:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.374+0000 7f14a37fe700 1 --2- 172.21.15.27:0/194191853 >> v2:172.21.15.77:3300/0 conn(0x7f14a40cca20 0x7f14a4159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.375+0000 7f14a1ffb700 1 -- 172.21.15.27:0/194191853 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348247293 0 0) 0x7f14a415d640 con 0x7f14a40ca3e0 2024-01-11T08:30:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.375+0000 7f14a1ffb700 1 -- 172.21.15.27:0/194191853 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f148c003570 con 0x7f14a40ca3e0 2024-01-11T08:30:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.375+0000 7f14a1ffb700 1 -- 172.21.15.27:0/194191853 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584482538 0 0) 0x7f14a40cce60 con 0x7f14a40ca7b0 2024-01-11T08:30:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.375+0000 7f14a1ffb700 1 -- 172.21.15.27:0/194191853 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14a415d640 con 0x7f14a40ca7b0 2024-01-11T08:30:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.375+0000 7f14a37fe700 1 -- 172.21.15.27:0/194191853 >> v1:172.21.15.77:6789/0 conn(0x7f14a40ca7b0 legacy=0x7f14a415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.375+0000 7f14a37fe700 1 -- 172.21.15.27:0/194191853 >> v1:172.21.15.27:6789/0 conn(0x7f14a40ca3e0 legacy=0x7f14a40c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.375+0000 7f14a37fe700 1 -- 172.21.15.27:0/194191853 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a415a5d0 con 0x7f14a40cca20 2024-01-11T08:30:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.375+0000 7f14a37fe700 1 --2- 172.21.15.27:0/194191853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14a40cca20 0x7f14a4159e80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f149c009e90 tx=0x7f149c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9d21e5d9c4a6181 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.376+0000 7f14a1ffb700 1 -- 172.21.15.27:0/194191853 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c00e070 con 0x7f14a40cca20 2024-01-11T08:30:16.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.376+0000 7f14a1ffb700 1 -- 172.21.15.27:0/194191853 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149c00bb90 con 0x7f14a40cca20 2024-01-11T08:30:16.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.376+0000 7f14a1ffb700 1 -- 172.21.15.27:0/194191853 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c012430 con 0x7f14a40cca20 2024-01-11T08:30:16.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.376+0000 7f14a95ff700 1 -- 172.21.15.27:0/194191853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14a40cca20 msgr2=0x7f14a4159e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:16.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.376+0000 7f14a95ff700 1 --2- 172.21.15.27:0/194191853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14a40cca20 0x7f14a4159e80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f149c009e90 tx=0x7f149c009780 comp rx=0 tx=0).stop 2024-01-11T08:30:16.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.376+0000 7f14a95ff700 1 -- 172.21.15.27:0/194191853 shutdown_connections 2024-01-11T08:30:16.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.376+0000 7f14a95ff700 1 --2- 172.21.15.27:0/194191853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14a40cca20 0x7f14a4159e80 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:16.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.376+0000 7f14a95ff700 1 -- 172.21.15.27:0/194191853 >> 172.21.15.27:0/194191853 conn(0x7f14a40ce3e0 msgr2=0x7f14a40ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:16.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.376+0000 7f14a95ff700 1 -- 172.21.15.27:0/194191853 shutdown_connections 2024-01-11T08:30:16.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.376+0000 7f14a95ff700 1 -- 172.21.15.27:0/194191853 wait complete. 2024-01-11T08:30:16.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.378+0000 7f14a95ff700 1 Processor -- start 2024-01-11T08:30:16.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.378+0000 7f14a95ff700 1 -- start start 2024-01-11T08:30:16.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.379+0000 7f14a95ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14a40ca3e0 0x7f14a414c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:16.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.379+0000 7f14a95ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14a40ca7b0 0x7f14a414b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:16.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.379+0000 7f14a95ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14a40cca20 0x7f14a40b3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:16.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.379+0000 7f14a95ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a414cc00 con 0x7f14a40cca20 2024-01-11T08:30:16.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.379+0000 7f14a95ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a414b970 con 0x7f14a40ca3e0 2024-01-11T08:30:16.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.379+0000 7f14a95ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a40b3d70 con 0x7f14a40ca7b0 2024-01-11T08:30:16.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.379+0000 7f14a37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14a40cca20 0x7f14a40b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:16.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.379+0000 7f14a37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14a40cca20 0x7f14a40b3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34826/0 (socket says 172.21.15.27:34826) 2024-01-11T08:30:16.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.379+0000 7f14a37fe700 1 -- 172.21.15.27:0/209728211 learned_addr learned my addr 172.21.15.27:0/209728211 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:16.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.379+0000 7f14a2ffd700 1 --2- 172.21.15.27:0/209728211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14a40ca3e0 0x7f14a414c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:16.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.379+0000 7f14a27fc700 1 --2- 172.21.15.27:0/209728211 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14a40ca7b0 0x7f14a414b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:16.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.379+0000 7f14a37fe700 1 -- 172.21.15.27:0/209728211 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14a40ca7b0 msgr2=0x7f14a414b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:16.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.379+0000 7f14a37fe700 1 --2- 172.21.15.27:0/209728211 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14a40ca7b0 0x7f14a414b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:16.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.380+0000 7f14a37fe700 1 -- 172.21.15.27:0/209728211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14a40ca3e0 msgr2=0x7f14a414c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:16.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.380+0000 7f14a37fe700 1 --2- 172.21.15.27:0/209728211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14a40ca3e0 0x7f14a414c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:16.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.380+0000 7f14a37fe700 1 -- 172.21.15.27:0/209728211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f149c009a20 con 0x7f14a40cca20 2024-01-11T08:30:16.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.380+0000 7f14a37fe700 1 --2- 172.21.15.27:0/209728211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14a40cca20 0x7f14a40b3390 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f149c013000 tx=0x7f149c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:16.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.380+0000 7f148affd700 1 -- 172.21.15.27:0/209728211 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c0263f0 con 0x7f14a40cca20 2024-01-11T08:30:16.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.380+0000 7f148affd700 1 -- 172.21.15.27:0/209728211 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149c026550 con 0x7f14a40cca20 2024-01-11T08:30:16.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.380+0000 7f14a95ff700 1 -- 172.21.15.27:0/209728211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a40b4050 con 0x7f14a40cca20 2024-01-11T08:30:16.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.380+0000 7f148affd700 1 -- 172.21.15.27:0/209728211 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c026910 con 0x7f14a40cca20 2024-01-11T08:30:16.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.380+0000 7f14a95ff700 1 -- 172.21.15.27:0/209728211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a414bec0 con 0x7f14a40cca20 2024-01-11T08:30:16.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.381+0000 7f14a95ff700 1 -- 172.21.15.27:0/209728211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a40718d0 con 0x7f14a40cca20 2024-01-11T08:30:16.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.382+0000 7f148affd700 1 -- 172.21.15.27:0/209728211 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f149c003020 con 0x7f14a40cca20 2024-01-11T08:30:16.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.383+0000 7f148affd700 1 --2- 172.21.15.27:0/209728211 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1490066f30 0x7f14900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:16.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.383+0000 7f148affd700 1 -- 172.21.15.27:0/209728211 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f149c086b50 con 0x7f14a40cca20 2024-01-11T08:30:16.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.384+0000 7f14a2ffd700 1 --2- 172.21.15.27:0/209728211 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1490066f30 0x7f14900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:16.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.384+0000 7f14a2ffd700 1 --2- 172.21.15.27:0/209728211 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1490066f30 0x7f14900693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1494007fd0 tx=0x7f1494009040 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:16.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.387+0000 7f148affd700 1 -- 172.21.15.27:0/209728211 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f149c053d60 con 0x7f14a40cca20 2024-01-11T08:30:16.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.587+0000 7f14a95ff700 1 -- 172.21.15.27:0/209728211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14a406c840 con 0x7f14a40cca20 2024-01-11T08:30:16.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.588+0000 7f148affd700 1 -- 172.21.15.27:0/209728211 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+15417 (secure 0 0 0) 0x7f149c057380 con 0x7f14a40cca20 2024-01-11T08:30:16.588 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:30:16.588 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":144,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:30:15.623446+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:30:13.161778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-01-11T08:30:14.620026+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:30:16.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.591+0000 7f14a95ff700 1 -- 172.21.15.27:0/209728211 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1490066f30 msgr2=0x7f14900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:16.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.591+0000 7f14a95ff700 1 --2- 172.21.15.27:0/209728211 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1490066f30 0x7f14900693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1494007fd0 tx=0x7f1494009040 comp rx=0 tx=0).stop 2024-01-11T08:30:16.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.591+0000 7f14a95ff700 1 -- 172.21.15.27:0/209728211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14a40cca20 msgr2=0x7f14a40b3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:16.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.591+0000 7f14a95ff700 1 --2- 172.21.15.27:0/209728211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14a40cca20 0x7f14a40b3390 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f149c013000 tx=0x7f149c019000 comp rx=0 tx=0).stop 2024-01-11T08:30:16.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.591+0000 7f14a95ff700 1 -- 172.21.15.27:0/209728211 shutdown_connections 2024-01-11T08:30:16.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.591+0000 7f14a95ff700 1 --2- 172.21.15.27:0/209728211 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1490066f30 0x7f14900693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:16.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.591+0000 7f14a95ff700 1 --2- 172.21.15.27:0/209728211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14a40cca20 0x7f14a40b3390 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:16.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.591+0000 7f14a95ff700 1 --2- 172.21.15.27:0/209728211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14a40ca3e0 0x7f14a414c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:16.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.591+0000 7f14a95ff700 1 --2- 172.21.15.27:0/209728211 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14a40ca7b0 0x7f14a414b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:16.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.591+0000 7f14a95ff700 1 -- 172.21.15.27:0/209728211 >> 172.21.15.27:0/209728211 conn(0x7f14a40ce3e0 msgr2=0x7f14a4068f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:16.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.591+0000 7f14a95ff700 1 -- 172.21.15.27:0/209728211 shutdown_connections 2024-01-11T08:30:16.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.591+0000 7f14a95ff700 1 -- 172.21.15.27:0/209728211 wait complete. 2024-01-11T08:30:16.608 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:30:16.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.809+0000 7f56cc3ff700 1 Processor -- start 2024-01-11T08:30:16.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.810+0000 7f56cc3ff700 1 -- start start 2024-01-11T08:30:16.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.811+0000 7f56cc3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56c41575e0 0x7f56c4158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:16.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.811+0000 7f56cc3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56c406b610 0x7f56c40b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:16.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.811+0000 7f56cc3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c4158b60 con 0x7f56c40b1800 2024-01-11T08:30:16.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.811+0000 7f56cc3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c40cb5e0 con 0x7f56c406b610 2024-01-11T08:30:16.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.811+0000 7f56cc3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c40b1c80 con 0x7f56c41575e0 2024-01-11T08:30:16.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.811+0000 7f56c999a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56c406b610 0x7f56c40b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:16.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.811+0000 7f56ca99c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f56c40b1800 0x7f56c40ca1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38682/0 (socket says 172.21.15.27:38682) 2024-01-11T08:30:16.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.811+0000 7f56ca99c700 1 -- 172.21.15.27:0/180554119 learned_addr learned my addr 172.21.15.27:0/180554119 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:30:16.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.812+0000 7f56ca19b700 1 --2- 172.21.15.27:0/180554119 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56c41575e0 0x7f56c4158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:16.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.812+0000 7f56c999a700 1 -- 172.21.15.27:0/180554119 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56c41575e0 msgr2=0x7f56c4158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:16.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.812+0000 7f56c999a700 1 --2- 172.21.15.27:0/180554119 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56c41575e0 0x7f56c4158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:16.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.812+0000 7f56c999a700 1 -- 172.21.15.27:0/180554119 >> v1:172.21.15.27:6789/0 conn(0x7f56c40b1800 legacy=0x7f56c40ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:16.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.812+0000 7f56c999a700 1 -- 172.21.15.27:0/180554119 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c41599b0 con 0x7f56c406b610 2024-01-11T08:30:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.812+0000 7f56c9199700 1 -- 172.21.15.27:0/180554119 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248185621 0 0) 0x7f56bc001490 con 0x7f56c40b1800 2024-01-11T08:30:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.812+0000 7f56c999a700 1 --2- 172.21.15.27:0/180554119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c406b610 0x7f56c40b12c0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f56b40098e0 tx=0x7f56b40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a01ef80422980b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.812+0000 7f56c9199700 1 -- 172.21.15.27:0/180554119 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b4009e80 con 0x7f56c406b610 2024-01-11T08:30:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.812+0000 7f56c9199700 1 -- 172.21.15.27:0/180554119 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b400b860 con 0x7f56c406b610 2024-01-11T08:30:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.812+0000 7f56c9199700 1 -- 172.21.15.27:0/180554119 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b400bbe0 con 0x7f56c406b610 2024-01-11T08:30:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.813+0000 7f56cc3ff700 1 -- 172.21.15.27:0/180554119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c406b610 msgr2=0x7f56c40b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.813+0000 7f56cc3ff700 1 --2- 172.21.15.27:0/180554119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c406b610 0x7f56c40b12c0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f56b40098e0 tx=0x7f56b40091a0 comp rx=0 tx=0).stop 2024-01-11T08:30:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.813+0000 7f56cc3ff700 1 -- 172.21.15.27:0/180554119 shutdown_connections 2024-01-11T08:30:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.813+0000 7f56cc3ff700 1 --2- 172.21.15.27:0/180554119 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56c41575e0 0x7f56c4158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.813+0000 7f56cc3ff700 1 --2- 172.21.15.27:0/180554119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c406b610 0x7f56c40b12c0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.813+0000 7f56cc3ff700 1 -- 172.21.15.27:0/180554119 >> 172.21.15.27:0/180554119 conn(0x7f56c40680e0 msgr2=0x7f56c4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:16.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.813+0000 7f56cc3ff700 1 -- 172.21.15.27:0/180554119 shutdown_connections 2024-01-11T08:30:16.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.813+0000 7f56cc3ff700 1 -- 172.21.15.27:0/180554119 wait complete. 2024-01-11T08:30:16.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.815+0000 7f56cc3ff700 1 Processor -- start 2024-01-11T08:30:16.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.815+0000 7f56cc3ff700 1 -- start start 2024-01-11T08:30:16.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.816+0000 7f56cc3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56c406b610 0x7f56c40c3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:16.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.816+0000 7f56cc3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56c40b1800 0x7f56c4142b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.816+0000 7f56cc3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c41575e0 0x7f56c40c3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.816+0000 7f56cc3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c416ae20 con 0x7f56c41575e0 2024-01-11T08:30:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.816+0000 7f56cc3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c40c80d0 con 0x7f56c40b1800 2024-01-11T08:30:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.816+0000 7f56cc3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c40c83b0 con 0x7f56c406b610 2024-01-11T08:30:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.816+0000 7f56ca99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c41575e0 0x7f56c40c3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.816+0000 7f56ca99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c41575e0 0x7f56c40c3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34832/0 (socket says 172.21.15.27:34832) 2024-01-11T08:30:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.816+0000 7f56ca99c700 1 -- 172.21.15.27:0/1464929296 learned_addr learned my addr 172.21.15.27:0/1464929296 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.816+0000 7f56ca19b700 1 --2- 172.21.15.27:0/1464929296 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56c406b610 0x7f56c40c3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.816+0000 7f56c999a700 1 --2- 172.21.15.27:0/1464929296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56c40b1800 0x7f56c4142b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.816+0000 7f56ca99c700 1 -- 172.21.15.27:0/1464929296 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56c406b610 msgr2=0x7f56c40c3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.816+0000 7f56ca99c700 1 --2- 172.21.15.27:0/1464929296 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56c406b610 0x7f56c40c3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.816+0000 7f56ca99c700 1 -- 172.21.15.27:0/1464929296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56c40b1800 msgr2=0x7f56c4142b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.816+0000 7f56ca99c700 1 --2- 172.21.15.27:0/1464929296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56c40b1800 0x7f56c4142b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.816+0000 7f56ca99c700 1 -- 172.21.15.27:0/1464929296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b4009420 con 0x7f56c41575e0 2024-01-11T08:30:16.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.816+0000 7f56ca19b700 1 --2- 172.21.15.27:0/1464929296 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56c406b610 0x7f56c40c3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:30:16.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.817+0000 7f56ca99c700 1 --2- 172.21.15.27:0/1464929296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c41575e0 0x7f56c40c3300 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f56bc00cc90 tx=0x7f56bc001a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:16.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.817+0000 7f56ba7fc700 1 -- 172.21.15.27:0/1464929296 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc00cea0 con 0x7f56c41575e0 2024-01-11T08:30:16.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.817+0000 7f56ba7fc700 1 -- 172.21.15.27:0/1464929296 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56bc004bb0 con 0x7f56c41575e0 2024-01-11T08:30:16.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.817+0000 7f56ba7fc700 1 -- 172.21.15.27:0/1464929296 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc00ab70 con 0x7f56c41575e0 2024-01-11T08:30:16.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.817+0000 7f56cc3ff700 1 -- 172.21.15.27:0/1464929296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c40c8690 con 0x7f56c41575e0 2024-01-11T08:30:16.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.817+0000 7f56cc3ff700 1 -- 172.21.15.27:0/1464929296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c40c41d0 con 0x7f56c41575e0 2024-01-11T08:30:16.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.818+0000 7f56cc3ff700 1 -- 172.21.15.27:0/1464929296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c40718d0 con 0x7f56c41575e0 2024-01-11T08:30:16.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.819+0000 7f56ba7fc700 1 -- 172.21.15.27:0/1464929296 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f56bc0070f0 con 0x7f56c41575e0 2024-01-11T08:30:16.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.820+0000 7f56ba7fc700 1 --2- 172.21.15.27:0/1464929296 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f56a8066f30 0x7f56a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:16.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.820+0000 7f56ba7fc700 1 -- 172.21.15.27:0/1464929296 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f56bc0842f0 con 0x7f56c41575e0 2024-01-11T08:30:16.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.821+0000 7f56ca19b700 1 --2- 172.21.15.27:0/1464929296 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f56a8066f30 0x7f56a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:16.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.821+0000 7f56ca19b700 1 --2- 172.21.15.27:0/1464929296 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f56a8066f30 0x7f56a80693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f56c415bb70 tx=0x7f56c0008040 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:16.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:16.824+0000 7f56ba7fc700 1 -- 172.21.15.27:0/1464929296 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f56bc0515a0 con 0x7f56c41575e0 2024-01-11T08:30:17.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.029+0000 7f56cc3ff700 1 -- 172.21.15.27:0/1464929296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56c4158b60 con 0x7f56c41575e0 2024-01-11T08:30:17.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.030+0000 7f56ba7fc700 1 -- 172.21.15.27:0/1464929296 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+15417 (secure 0 0 0) 0x7f56c4158b60 con 0x7f56c41575e0 2024-01-11T08:30:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:30:17.029 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":144,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:30:15.623446+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:30:13.161778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-01-11T08:30:14.620026+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:30:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.032+0000 7f56cc3ff700 1 -- 172.21.15.27:0/1464929296 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f56a8066f30 msgr2=0x7f56a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:17.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.032+0000 7f56cc3ff700 1 --2- 172.21.15.27:0/1464929296 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f56a8066f30 0x7f56a80693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f56c415bb70 tx=0x7f56c0008040 comp rx=0 tx=0).stop 2024-01-11T08:30:17.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.032+0000 7f56cc3ff700 1 -- 172.21.15.27:0/1464929296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c41575e0 msgr2=0x7f56c40c3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:17.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.032+0000 7f56cc3ff700 1 --2- 172.21.15.27:0/1464929296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c41575e0 0x7f56c40c3300 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f56bc00cc90 tx=0x7f56bc001a50 comp rx=0 tx=0).stop 2024-01-11T08:30:17.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.032+0000 7f56cc3ff700 1 -- 172.21.15.27:0/1464929296 shutdown_connections 2024-01-11T08:30:17.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.032+0000 7f56cc3ff700 1 --2- 172.21.15.27:0/1464929296 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f56a8066f30 0x7f56a80693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:17.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.032+0000 7f56cc3ff700 1 --2- 172.21.15.27:0/1464929296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c41575e0 0x7f56c40c3300 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:17.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.032+0000 7f56cc3ff700 1 --2- 172.21.15.27:0/1464929296 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56c406b610 0x7f56c40c3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:17.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.032+0000 7f56cc3ff700 1 --2- 172.21.15.27:0/1464929296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56c40b1800 0x7f56c4142b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:17.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.032+0000 7f56cc3ff700 1 -- 172.21.15.27:0/1464929296 >> 172.21.15.27:0/1464929296 conn(0x7f56c40680e0 msgr2=0x7f56c415ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:17.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.032+0000 7f56cc3ff700 1 -- 172.21.15.27:0/1464929296 shutdown_connections 2024-01-11T08:30:17.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.032+0000 7f56cc3ff700 1 -- 172.21.15.27:0/1464929296 wait complete. 2024-01-11T08:30:17.050 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-11T08:30:17.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.254+0000 7f4c57bff700 1 Processor -- start 2024-01-11T08:30:17.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.254+0000 7f4c57bff700 1 -- start start 2024-01-11T08:30:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.255+0000 7f4c57bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4c50157410 0x7f4c500ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.255+0000 7f4c57bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4c501577e0 0x7f4c500ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.255+0000 7f4c57bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c500699d0 con 0x7f4c50069540 2024-01-11T08:30:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.255+0000 7f4c57bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c50069cb0 con 0x7f4c501577e0 2024-01-11T08:30:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.255+0000 7f4c57bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c50069f90 con 0x7f4c50157410 2024-01-11T08:30:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.255+0000 7f4c5519a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4c501577e0 0x7f4c500ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.255+0000 7f4c5619c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4c50069540 0x7f4c5015b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38688/0 (socket says 172.21.15.27:38688) 2024-01-11T08:30:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.256+0000 7f4c5619c700 1 -- 172.21.15.27:0/298226853 learned_addr learned my addr 172.21.15.27:0/298226853 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:30:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.256+0000 7f4c5599b700 1 --2- 172.21.15.27:0/298226853 >> v2:172.21.15.77:3300/0 conn(0x7f4c50157410 0x7f4c500ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.256+0000 7f4c5519a700 1 -- 172.21.15.27:0/298226853 >> v2:172.21.15.77:3300/0 conn(0x7f4c50157410 msgr2=0x7f4c500ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.256+0000 7f4c5519a700 1 --2- 172.21.15.27:0/298226853 >> v2:172.21.15.77:3300/0 conn(0x7f4c50157410 0x7f4c500ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.256+0000 7f4c5519a700 1 -- 172.21.15.27:0/298226853 >> v1:172.21.15.27:6789/0 conn(0x7f4c50069540 legacy=0x7f4c5015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.256+0000 7f4c5519a700 1 -- 172.21.15.27:0/298226853 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5006a270 con 0x7f4c501577e0 2024-01-11T08:30:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.256+0000 7f4c54999700 1 -- 172.21.15.27:0/298226853 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776526902 0 0) 0x7f4c500699d0 con 0x7f4c50069540 2024-01-11T08:30:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.256+0000 7f4c5519a700 1 --2- 172.21.15.27:0/298226853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c501577e0 0x7f4c500ae000 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f4c50070c90 tx=0x7f4c400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d5fd15c05d5fef8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.256+0000 7f4c54999700 1 -- 172.21.15.27:0/298226853 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4000b6b0 con 0x7f4c501577e0 2024-01-11T08:30:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.257+0000 7f4c54999700 1 -- 172.21.15.27:0/298226853 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4000b810 con 0x7f4c501577e0 2024-01-11T08:30:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.257+0000 7f4c54999700 1 -- 172.21.15.27:0/298226853 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4000bbb0 con 0x7f4c501577e0 2024-01-11T08:30:17.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.257+0000 7f4c57bff700 1 -- 172.21.15.27:0/298226853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c501577e0 msgr2=0x7f4c500ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:17.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.257+0000 7f4c57bff700 1 --2- 172.21.15.27:0/298226853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c501577e0 0x7f4c500ae000 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f4c50070c90 tx=0x7f4c400091a0 comp rx=0 tx=0).stop 2024-01-11T08:30:17.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.257+0000 7f4c57bff700 1 -- 172.21.15.27:0/298226853 shutdown_connections 2024-01-11T08:30:17.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.257+0000 7f4c57bff700 1 --2- 172.21.15.27:0/298226853 >> v2:172.21.15.77:3300/0 conn(0x7f4c50157410 0x7f4c500ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:17.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.257+0000 7f4c57bff700 1 --2- 172.21.15.27:0/298226853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c501577e0 0x7f4c500ae000 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:17.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.257+0000 7f4c57bff700 1 -- 172.21.15.27:0/298226853 >> 172.21.15.27:0/298226853 conn(0x7f4c500680a0 msgr2=0x7f4c500cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:17.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.257+0000 7f4c57bff700 1 -- 172.21.15.27:0/298226853 shutdown_connections 2024-01-11T08:30:17.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.257+0000 7f4c57bff700 1 -- 172.21.15.27:0/298226853 wait complete. 2024-01-11T08:30:17.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.259+0000 7f4c57bff700 1 Processor -- start 2024-01-11T08:30:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.259+0000 7f4c57bff700 1 -- start start 2024-01-11T08:30:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.260+0000 7f4c57bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c50069540 0x7f4c500c57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.260+0000 7f4c57bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c50157410 0x7f4c50158be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.260+0000 7f4c57bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c501577e0 0x7f4c50159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.260+0000 7f4c57bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c500c5350 con 0x7f4c501577e0 2024-01-11T08:30:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.260+0000 7f4c57bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c500c4740 con 0x7f4c50069540 2024-01-11T08:30:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.260+0000 7f4c57bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c500c4a20 con 0x7f4c50157410 2024-01-11T08:30:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.260+0000 7f4c5619c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c501577e0 0x7f4c50159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:17.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.260+0000 7f4c5619c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c501577e0 0x7f4c50159120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34848/0 (socket says 172.21.15.27:34848) 2024-01-11T08:30:17.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.260+0000 7f4c5619c700 1 -- 172.21.15.27:0/1136879167 learned_addr learned my addr 172.21.15.27:0/1136879167 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:17.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.260+0000 7f4c5599b700 1 --2- 172.21.15.27:0/1136879167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c50069540 0x7f4c500c57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:17.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.260+0000 7f4c5519a700 1 --2- 172.21.15.27:0/1136879167 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c50157410 0x7f4c50158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.261+0000 7f4c5619c700 1 -- 172.21.15.27:0/1136879167 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c50157410 msgr2=0x7f4c50158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.261+0000 7f4c5619c700 1 --2- 172.21.15.27:0/1136879167 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c50157410 0x7f4c50158be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.261+0000 7f4c5619c700 1 -- 172.21.15.27:0/1136879167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c50069540 msgr2=0x7f4c500c57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.261+0000 7f4c5619c700 1 --2- 172.21.15.27:0/1136879167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c50069540 0x7f4c500c57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.261+0000 7f4c5619c700 1 -- 172.21.15.27:0/1136879167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c40009420 con 0x7f4c501577e0 2024-01-11T08:30:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.261+0000 7f4c5619c700 1 --2- 172.21.15.27:0/1136879167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c501577e0 0x7f4c50159120 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f4c4800cbf0 tx=0x7f4c480019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.261+0000 7f4c45ffb700 1 -- 172.21.15.27:0/1136879167 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4800a8b0 con 0x7f4c501577e0 2024-01-11T08:30:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.261+0000 7f4c45ffb700 1 -- 172.21.15.27:0/1136879167 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4800aa10 con 0x7f4c501577e0 2024-01-11T08:30:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.261+0000 7f4c45ffb700 1 -- 172.21.15.27:0/1136879167 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4800add0 con 0x7f4c501577e0 2024-01-11T08:30:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.261+0000 7f4c57bff700 1 -- 172.21.15.27:0/1136879167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c500c4d00 con 0x7f4c501577e0 2024-01-11T08:30:17.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.261+0000 7f4c57bff700 1 -- 172.21.15.27:0/1136879167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c5013f860 con 0x7f4c501577e0 2024-01-11T08:30:17.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.262+0000 7f4c57bff700 1 -- 172.21.15.27:0/1136879167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c5004c3c0 con 0x7f4c501577e0 2024-01-11T08:30:17.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.264+0000 7f4c45ffb700 1 -- 172.21.15.27:0/1136879167 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4c48007300 con 0x7f4c501577e0 2024-01-11T08:30:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.265+0000 7f4c45ffb700 1 --2- 172.21.15.27:0/1136879167 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4c34066f30 0x7f4c340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.265+0000 7f4c45ffb700 1 -- 172.21.15.27:0/1136879167 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f4c480848e0 con 0x7f4c501577e0 2024-01-11T08:30:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.266+0000 7f4c5599b700 1 --2- 172.21.15.27:0/1136879167 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4c34066f30 0x7f4c340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.266+0000 7f4c5599b700 1 --2- 172.21.15.27:0/1136879167 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4c34066f30 0x7f4c340693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4c50152770 tx=0x7f4c4c008040 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:17.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.269+0000 7f4c45ffb700 1 -- 172.21.15.27:0/1136879167 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4c48051af0 con 0x7f4c501577e0 2024-01-11T08:30:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:17.521+0000 7f4c57bff700 1 -- 172.21.15.27:0/1136879167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4c500c5490 con 0x7f4c501577e0 2024-01-11T08:30:18.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.151+0000 7f4c45ffb700 1 -- 172.21.15.27:0/1136879167 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v76) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4c48055110 con 0x7f4c501577e0 2024-01-11T08:30:18.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.154+0000 7f4c57bff700 1 -- 172.21.15.27:0/1136879167 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4c34066f30 msgr2=0x7f4c340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:18.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.154+0000 7f4c57bff700 1 --2- 172.21.15.27:0/1136879167 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4c34066f30 0x7f4c340693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4c50152770 tx=0x7f4c4c008040 comp rx=0 tx=0).stop 2024-01-11T08:30:18.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.154+0000 7f4c57bff700 1 -- 172.21.15.27:0/1136879167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c501577e0 msgr2=0x7f4c50159120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:18.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.154+0000 7f4c57bff700 1 --2- 172.21.15.27:0/1136879167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c501577e0 0x7f4c50159120 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f4c4800cbf0 tx=0x7f4c480019b0 comp rx=0 tx=0).stop 2024-01-11T08:30:18.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.154+0000 7f4c57bff700 1 -- 172.21.15.27:0/1136879167 shutdown_connections 2024-01-11T08:30:18.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.154+0000 7f4c57bff700 1 --2- 172.21.15.27:0/1136879167 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4c34066f30 0x7f4c340693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:18.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.154+0000 7f4c57bff700 1 --2- 172.21.15.27:0/1136879167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c501577e0 0x7f4c50159120 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:18.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.154+0000 7f4c57bff700 1 --2- 172.21.15.27:0/1136879167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c50069540 0x7f4c500c57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:18.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.154+0000 7f4c57bff700 1 --2- 172.21.15.27:0/1136879167 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c50157410 0x7f4c50158be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:18.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.154+0000 7f4c57bff700 1 -- 172.21.15.27:0/1136879167 >> 172.21.15.27:0/1136879167 conn(0x7f4c500680a0 msgr2=0x7f4c500cb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:18.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.154+0000 7f4c57bff700 1 -- 172.21.15.27:0/1136879167 shutdown_connections 2024-01-11T08:30:18.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.154+0000 7f4c57bff700 1 -- 172.21.15.27:0/1136879167 wait complete. 2024-01-11T08:30:18.173 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-11T08:30:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.379+0000 7f73127ff700 1 Processor -- start 2024-01-11T08:30:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.379+0000 7f73127ff700 1 -- start start 2024-01-11T08:30:18.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.381+0000 7f73127ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f730c15a1b0 0x7f730c15a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:18.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.381+0000 7f73127ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f730c15ae70 0x7f730c15b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:18.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.381+0000 7f73127ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730c0cf850 con 0x7f730c0ccd70 2024-01-11T08:30:18.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.381+0000 7f73127ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f730c0cec40 con 0x7f730c15ae70 2024-01-11T08:30:18.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.381+0000 7f73127ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f730c0cef20 con 0x7f730c15a1b0 2024-01-11T08:30:18.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.381+0000 7f7310d9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f730c15ae70 0x7f730c15b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:18.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.381+0000 7f7310d9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f730c15ae70 0x7f730c15b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34858/0 (socket says 172.21.15.27:34858) 2024-01-11T08:30:18.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.381+0000 7f7310d9c700 1 -- 172.21.15.27:0/3431727319 learned_addr learned my addr 172.21.15.27:0/3431727319 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:18.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.381+0000 7f730bfff700 1 --2- 172.21.15.27:0/3431727319 >> v2:172.21.15.77:3300/0 conn(0x7f730c15a1b0 0x7f730c15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:18.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.382+0000 7f7310d9c700 1 -- 172.21.15.27:0/3431727319 >> v2:172.21.15.77:3300/0 conn(0x7f730c15a1b0 msgr2=0x7f730c15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:18.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.382+0000 7f7310d9c700 1 --2- 172.21.15.27:0/3431727319 >> v2:172.21.15.77:3300/0 conn(0x7f730c15a1b0 0x7f730c15a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:18.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.382+0000 7f7310d9c700 1 -- 172.21.15.27:0/3431727319 >> v1:172.21.15.27:6789/0 conn(0x7f730c0ccd70 legacy=0x7f730c0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:18.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.382+0000 7f7310d9c700 1 -- 172.21.15.27:0/3431727319 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c0cf200 con 0x7f730c15ae70 2024-01-11T08:30:18.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.382+0000 7f730bfff700 1 --2- 172.21.15.27:0/3431727319 >> v2:172.21.15.77:3300/0 conn(0x7f730c15a1b0 0x7f730c15a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:30:18.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.382+0000 7f7310d9c700 1 --2- 172.21.15.27:0/3431727319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c15ae70 0x7f730c15b2a0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f730c0d16f0 tx=0x7f7304009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc7568f1cead42e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:18.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.382+0000 7f730affd700 1 -- 172.21.15.27:0/3431727319 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730400e070 con 0x7f730c15ae70 2024-01-11T08:30:18.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.382+0000 7f730affd700 1 -- 172.21.15.27:0/3431727319 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730400ba50 con 0x7f730c15ae70 2024-01-11T08:30:18.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.382+0000 7f730affd700 1 -- 172.21.15.27:0/3431727319 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73040092e0 con 0x7f730c15ae70 2024-01-11T08:30:18.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.383+0000 7f73127ff700 1 -- 172.21.15.27:0/3431727319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c15ae70 msgr2=0x7f730c15b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:18.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.383+0000 7f73127ff700 1 --2- 172.21.15.27:0/3431727319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c15ae70 0x7f730c15b2a0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f730c0d16f0 tx=0x7f7304009840 comp rx=0 tx=0).stop 2024-01-11T08:30:18.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.383+0000 7f73127ff700 1 -- 172.21.15.27:0/3431727319 shutdown_connections 2024-01-11T08:30:18.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.383+0000 7f73127ff700 1 --2- 172.21.15.27:0/3431727319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c15ae70 0x7f730c15b2a0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:18.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.383+0000 7f73127ff700 1 --2- 172.21.15.27:0/3431727319 >> v2:172.21.15.77:3300/0 conn(0x7f730c15a1b0 0x7f730c15a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:18.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.383+0000 7f73127ff700 1 -- 172.21.15.27:0/3431727319 >> 172.21.15.27:0/3431727319 conn(0x7f730c06ed40 msgr2=0x7f730c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:18.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.383+0000 7f73127ff700 1 -- 172.21.15.27:0/3431727319 shutdown_connections 2024-01-11T08:30:18.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.383+0000 7f73127ff700 1 -- 172.21.15.27:0/3431727319 wait complete. 2024-01-11T08:30:18.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.384+0000 7f73127ff700 1 Processor -- start 2024-01-11T08:30:18.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.385+0000 7f73127ff700 1 -- start start 2024-01-11T08:30:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.385+0000 7f73127ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f730c0ccd70 0x7f730c0ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.386+0000 7f73127ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c15a1b0 0x7f730c151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.386+0000 7f73127ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f730c15ae70 0x7f730c0b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.386+0000 7f73127ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c0c7d50 con 0x7f730c15a1b0 2024-01-11T08:30:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.386+0000 7f73127ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c1692f0 con 0x7f730c15ae70 2024-01-11T08:30:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.386+0000 7f73127ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730c169b50 con 0x7f730c0ccd70 2024-01-11T08:30:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.386+0000 7f730b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c15a1b0 0x7f730c151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.386+0000 7f730bfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f730c0ccd70 0x7f730c0ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.386+0000 7f730b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c15a1b0 0x7f730c151910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34862/0 (socket says 172.21.15.27:34862) 2024-01-11T08:30:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.386+0000 7f730b7fe700 1 -- 172.21.15.27:0/1006756264 learned_addr learned my addr 172.21.15.27:0/1006756264 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:18.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.386+0000 7f7310d9c700 1 --2- 172.21.15.27:0/1006756264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f730c15ae70 0x7f730c0b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.386+0000 7f730b7fe700 1 -- 172.21.15.27:0/1006756264 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f730c0ccd70 msgr2=0x7f730c0ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.386+0000 7f730b7fe700 1 --2- 172.21.15.27:0/1006756264 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f730c0ccd70 0x7f730c0ca4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.386+0000 7f730b7fe700 1 -- 172.21.15.27:0/1006756264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f730c15ae70 msgr2=0x7f730c0b54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.386+0000 7f730b7fe700 1 --2- 172.21.15.27:0/1006756264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f730c15ae70 0x7f730c0b54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.386+0000 7f730b7fe700 1 -- 172.21.15.27:0/1006756264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7304009ae0 con 0x7f730c15a1b0 2024-01-11T08:30:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.387+0000 7f730b7fe700 1 --2- 172.21.15.27:0/1006756264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c15a1b0 0x7f730c151910 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f730000c4e0 tx=0x7f730000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.387+0000 7f72f3fff700 1 -- 172.21.15.27:0/1006756264 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7300011590 con 0x7f730c15a1b0 2024-01-11T08:30:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.387+0000 7f72f3fff700 1 -- 172.21.15.27:0/1006756264 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7300011710 con 0x7f730c15a1b0 2024-01-11T08:30:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.387+0000 7f73127ff700 1 -- 172.21.15.27:0/1006756264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730c0c7a70 con 0x7f730c15a1b0 2024-01-11T08:30:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.387+0000 7f72f3fff700 1 -- 172.21.15.27:0/1006756264 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7300011ad0 con 0x7f730c15a1b0 2024-01-11T08:30:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.387+0000 7f73127ff700 1 -- 172.21.15.27:0/1006756264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730c1697e0 con 0x7f730c15a1b0 2024-01-11T08:30:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.388+0000 7f73127ff700 1 -- 172.21.15.27:0/1006756264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f730c04c3c0 con 0x7f730c15a1b0 2024-01-11T08:30:18.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.389+0000 7f72f3fff700 1 -- 172.21.15.27:0/1006756264 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f730001b3b0 con 0x7f730c15a1b0 2024-01-11T08:30:18.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.390+0000 7f72f3fff700 1 --2- 172.21.15.27:0/1006756264 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f72f8066f30 0x7f72f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:18.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.390+0000 7f72f3fff700 1 -- 172.21.15.27:0/1006756264 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f73000839a0 con 0x7f730c15a1b0 2024-01-11T08:30:18.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.390+0000 7f730bfff700 1 --2- 172.21.15.27:0/1006756264 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f72f8066f30 0x7f72f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:18.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.391+0000 7f730bfff700 1 --2- 172.21.15.27:0/1006756264 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f72f8066f30 0x7f72f80693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f72fc006f00 tx=0x7f72fc008040 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:18.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.394+0000 7f72f3fff700 1 -- 172.21.15.27:0/1006756264 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7300050b30 con 0x7f730c15a1b0 2024-01-11T08:30:18.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:18.641+0000 7f73127ff700 1 -- 172.21.15.27:0/1006756264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f730c15c3f0 con 0x7f730c15a1b0 2024-01-11T08:30:19.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.153+0000 7f72f3fff700 1 -- 172.21.15.27:0/1006756264 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v77) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7300017070 con 0x7f730c15a1b0 2024-01-11T08:30:19.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.155+0000 7f73127ff700 1 -- 172.21.15.27:0/1006756264 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f72f8066f30 msgr2=0x7f72f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:19.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.155+0000 7f73127ff700 1 --2- 172.21.15.27:0/1006756264 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f72f8066f30 0x7f72f80693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f72fc006f00 tx=0x7f72fc008040 comp rx=0 tx=0).stop 2024-01-11T08:30:19.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.155+0000 7f73127ff700 1 -- 172.21.15.27:0/1006756264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c15a1b0 msgr2=0x7f730c151910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:19.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.155+0000 7f73127ff700 1 --2- 172.21.15.27:0/1006756264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c15a1b0 0x7f730c151910 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f730000c4e0 tx=0x7f730000c7f0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.155+0000 7f73127ff700 1 -- 172.21.15.27:0/1006756264 shutdown_connections 2024-01-11T08:30:19.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.155+0000 7f73127ff700 1 --2- 172.21.15.27:0/1006756264 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f72f8066f30 0x7f72f80693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.155+0000 7f73127ff700 1 --2- 172.21.15.27:0/1006756264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f730c15ae70 0x7f730c0b54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.155+0000 7f73127ff700 1 --2- 172.21.15.27:0/1006756264 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f730c0ccd70 0x7f730c0ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.155+0000 7f73127ff700 1 --2- 172.21.15.27:0/1006756264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c15a1b0 0x7f730c151910 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.155+0000 7f73127ff700 1 -- 172.21.15.27:0/1006756264 >> 172.21.15.27:0/1006756264 conn(0x7f730c06ed40 msgr2=0x7f730c06f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:19.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.155+0000 7f73127ff700 1 -- 172.21.15.27:0/1006756264 shutdown_connections 2024-01-11T08:30:19.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.155+0000 7f73127ff700 1 -- 172.21.15.27:0/1006756264 wait complete. 2024-01-11T08:30:19.173 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:30:19.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.375+0000 7fde105ff700 1 Processor -- start 2024-01-11T08:30:19.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.375+0000 7fde105ff700 1 -- start start 2024-01-11T08:30:19.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.376+0000 7fde105ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fde080b03e0 0x7fde080b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:19.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.376+0000 7fde105ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fde080b29c0 0x7fde08159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:19.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.376+0000 7fde105ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde080cd730 con 0x7fde080b07b0 2024-01-11T08:30:19.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.376+0000 7fde105ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde080b2de0 con 0x7fde080b03e0 2024-01-11T08:30:19.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.376+0000 7fde105ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde080ccb20 con 0x7fde080b29c0 2024-01-11T08:30:19.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.377+0000 7fde0e39b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fde080b03e0 0x7fde080b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:19.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.377+0000 7fde0e39b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fde080b03e0 0x7fde080b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34866/0 (socket says 172.21.15.27:34866) 2024-01-11T08:30:19.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.377+0000 7fde0db9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fde080b07b0 0x7fde08159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38712/0 (socket says 172.21.15.27:38712) 2024-01-11T08:30:19.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.377+0000 7fde0e39b700 1 -- 172.21.15.27:0/3404654377 learned_addr learned my addr 172.21.15.27:0/3404654377 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:19.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.377+0000 7fde0eb9c700 1 --2- 172.21.15.27:0/3404654377 >> v2:172.21.15.77:3300/0 conn(0x7fde080b29c0 0x7fde08159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:19.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.377+0000 7fde0e39b700 1 -- 172.21.15.27:0/3404654377 >> v2:172.21.15.77:3300/0 conn(0x7fde080b29c0 msgr2=0x7fde08159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:19.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.377+0000 7fde0e39b700 1 --2- 172.21.15.27:0/3404654377 >> v2:172.21.15.77:3300/0 conn(0x7fde080b29c0 0x7fde08159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.377+0000 7fde0e39b700 1 -- 172.21.15.27:0/3404654377 >> v1:172.21.15.27:6789/0 conn(0x7fde080b07b0 legacy=0x7fde08159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:19.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.377+0000 7fde0e39b700 1 -- 172.21.15.27:0/3404654377 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde080cce00 con 0x7fde080b03e0 2024-01-11T08:30:19.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.377+0000 7fde0d399700 1 -- 172.21.15.27:0/3404654377 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415165997 0 0) 0x7fde080cd730 con 0x7fde080b07b0 2024-01-11T08:30:19.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.377+0000 7fde0e39b700 1 --2- 172.21.15.27:0/3404654377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde080b03e0 0x7fde080b2480 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fde080b1d10 tx=0x7fde04009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c33c7652602ff9c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:19.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.378+0000 7fde0d399700 1 -- 172.21.15.27:0/3404654377 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0400b860 con 0x7fde080b03e0 2024-01-11T08:30:19.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.378+0000 7fde0d399700 1 -- 172.21.15.27:0/3404654377 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0400b9c0 con 0x7fde080b03e0 2024-01-11T08:30:19.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.378+0000 7fde0d399700 1 -- 172.21.15.27:0/3404654377 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0400bd60 con 0x7fde080b03e0 2024-01-11T08:30:19.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.378+0000 7fde105ff700 1 -- 172.21.15.27:0/3404654377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde080b03e0 msgr2=0x7fde080b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:19.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.378+0000 7fde105ff700 1 --2- 172.21.15.27:0/3404654377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde080b03e0 0x7fde080b2480 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fde080b1d10 tx=0x7fde04009250 comp rx=0 tx=0).stop 2024-01-11T08:30:19.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.378+0000 7fde105ff700 1 -- 172.21.15.27:0/3404654377 shutdown_connections 2024-01-11T08:30:19.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.378+0000 7fde105ff700 1 --2- 172.21.15.27:0/3404654377 >> v2:172.21.15.77:3300/0 conn(0x7fde080b29c0 0x7fde08159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.378+0000 7fde105ff700 1 --2- 172.21.15.27:0/3404654377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde080b03e0 0x7fde080b2480 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.378+0000 7fde105ff700 1 -- 172.21.15.27:0/3404654377 >> 172.21.15.27:0/3404654377 conn(0x7fde0806ed40 msgr2=0x7fde0806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:19.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.378+0000 7fde105ff700 1 -- 172.21.15.27:0/3404654377 shutdown_connections 2024-01-11T08:30:19.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.378+0000 7fde105ff700 1 -- 172.21.15.27:0/3404654377 wait complete. 2024-01-11T08:30:19.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.381+0000 7fde105ff700 1 Processor -- start 2024-01-11T08:30:19.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.381+0000 7fde105ff700 1 -- start start 2024-01-11T08:30:19.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.382+0000 7fde105ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde080b03e0 0x7fde0816d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:19.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.382+0000 7fde105ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde080b07b0 0x7fde0816d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:19.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.382+0000 7fde105ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde080b29c0 0x7fde080cfa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:19.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.382+0000 7fde105ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde080d0ad0 con 0x7fde080b03e0 2024-01-11T08:30:19.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.382+0000 7fde105ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0816b5a0 con 0x7fde080b07b0 2024-01-11T08:30:19.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.382+0000 7fde105ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde080d07f0 con 0x7fde080b29c0 2024-01-11T08:30:19.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.382+0000 7fde0e39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde080b03e0 0x7fde0816d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:19.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.382+0000 7fde0e39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde080b03e0 0x7fde0816d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34874/0 (socket says 172.21.15.27:34874) 2024-01-11T08:30:19.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.382+0000 7fde0e39b700 1 -- 172.21.15.27:0/1969973973 learned_addr learned my addr 172.21.15.27:0/1969973973 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:19.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.382+0000 7fde0db9a700 1 --2- 172.21.15.27:0/1969973973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde080b07b0 0x7fde0816d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:19.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.382+0000 7fde0eb9c700 1 --2- 172.21.15.27:0/1969973973 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde080b29c0 0x7fde080cfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:19.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.383+0000 7fde0e39b700 1 -- 172.21.15.27:0/1969973973 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde080b29c0 msgr2=0x7fde080cfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:19.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.383+0000 7fde0e39b700 1 --2- 172.21.15.27:0/1969973973 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde080b29c0 0x7fde080cfa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.383+0000 7fde0e39b700 1 -- 172.21.15.27:0/1969973973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde080b07b0 msgr2=0x7fde0816d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:19.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.383+0000 7fde0e39b700 1 --2- 172.21.15.27:0/1969973973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde080b07b0 0x7fde0816d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.383+0000 7fde0e39b700 1 -- 172.21.15.27:0/1969973973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde040094f0 con 0x7fde080b03e0 2024-01-11T08:30:19.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.383+0000 7fde0e39b700 1 --2- 172.21.15.27:0/1969973973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde080b03e0 0x7fde0816d1e0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fde04006f50 tx=0x7fde04017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:19.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.383+0000 7fddfe7fc700 1 -- 172.21.15.27:0/1969973973 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04011e40 con 0x7fde080b03e0 2024-01-11T08:30:19.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.383+0000 7fddfe7fc700 1 -- 172.21.15.27:0/1969973973 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde04024440 con 0x7fde080b03e0 2024-01-11T08:30:19.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.383+0000 7fddfe7fc700 1 -- 172.21.15.27:0/1969973973 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde040247e0 con 0x7fde080b03e0 2024-01-11T08:30:19.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.383+0000 7fde105ff700 1 -- 172.21.15.27:0/1969973973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde0816af50 con 0x7fde080b03e0 2024-01-11T08:30:19.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.384+0000 7fde105ff700 1 -- 172.21.15.27:0/1969973973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde080af090 con 0x7fde080b03e0 2024-01-11T08:30:19.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.385+0000 7fde105ff700 1 -- 172.21.15.27:0/1969973973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0816adb0 con 0x7fde080b03e0 2024-01-11T08:30:19.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.385+0000 7fddfe7fc700 1 -- 172.21.15.27:0/1969973973 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fde0402d440 con 0x7fde080b03e0 2024-01-11T08:30:19.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.386+0000 7fddfe7fc700 1 --2- 172.21.15.27:0/1969973973 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fddec066f30 0x7fddec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:19.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.386+0000 7fddfe7fc700 1 -- 172.21.15.27:0/1969973973 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fde040860a0 con 0x7fde080b03e0 2024-01-11T08:30:19.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.387+0000 7fde0db9a700 1 --2- 172.21.15.27:0/1969973973 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fddec066f30 0x7fddec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:19.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.388+0000 7fde0db9a700 1 --2- 172.21.15.27:0/1969973973 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fddec066f30 0x7fddec0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fde0815cf40 tx=0x7fddf8007d50 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:19.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.390+0000 7fddfe7fc700 1 -- 172.21.15.27:0/1969973973 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fde0816adb0 con 0x7fde080b03e0 2024-01-11T08:30:19.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.636+0000 7fde105ff700 1 -- 172.21.15.27:0/1969973973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde0816adb0 con 0x7fde080b03e0 2024-01-11T08:30:19.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.637+0000 7fddfe7fc700 1 -- 172.21.15.27:0/1969973973 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4596 (secure 0 0 0) 0x7fde0816adb0 con 0x7fde080b03e0 2024-01-11T08:30:19.636 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:30:19.636 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7352,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3818350333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3818350333},{"type":"v1","addr":"172.21.15.27:6834","nonce":3818350333}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/217390254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":217390254},{"type":"v1","addr":"172.21.15.27:6835","nonce":217390254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2824437717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2824437717},{"type":"v1","addr":"172.21.15.77:6834","nonce":2824437717}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:30:15.623796+0000","modified":"2024-01-11T08:30:19.150622+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8601":{"gid":8601,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/1641163931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1641163931},{"type":"v1","addr":"172.21.15.77:6837","nonce":1641163931}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2024-01-11T08:30:19.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.639+0000 7fde105ff700 1 -- 172.21.15.27:0/1969973973 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fddec066f30 msgr2=0x7fddec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.639+0000 7fde105ff700 1 --2- 172.21.15.27:0/1969973973 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fddec066f30 0x7fddec0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fde0815cf40 tx=0x7fddf8007d50 comp rx=0 tx=0).stop 2024-01-11T08:30:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.639+0000 7fde105ff700 1 -- 172.21.15.27:0/1969973973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde080b03e0 msgr2=0x7fde0816d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.639+0000 7fde105ff700 1 --2- 172.21.15.27:0/1969973973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde080b03e0 0x7fde0816d1e0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fde04006f50 tx=0x7fde04017000 comp rx=0 tx=0).stop 2024-01-11T08:30:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.639+0000 7fde105ff700 1 -- 172.21.15.27:0/1969973973 shutdown_connections 2024-01-11T08:30:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.639+0000 7fde105ff700 1 --2- 172.21.15.27:0/1969973973 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fddec066f30 0x7fddec0693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.639+0000 7fde105ff700 1 --2- 172.21.15.27:0/1969973973 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde080b29c0 0x7fde080cfa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.639+0000 7fde105ff700 1 --2- 172.21.15.27:0/1969973973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde080b03e0 0x7fde0816d1e0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.639+0000 7fde105ff700 1 --2- 172.21.15.27:0/1969973973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde080b07b0 0x7fde0816d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.639+0000 7fde105ff700 1 -- 172.21.15.27:0/1969973973 >> 172.21.15.27:0/1969973973 conn(0x7fde0806ed40 msgr2=0x7fde08159530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.639+0000 7fde105ff700 1 -- 172.21.15.27:0/1969973973 shutdown_connections 2024-01-11T08:30:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.640+0000 7fde105ff700 1 -- 172.21.15.27:0/1969973973 wait complete. 2024-01-11T08:30:19.640 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 77 2024-01-11T08:30:19.656 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T08:30:19.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.857+0000 7f9ee93ff700 1 Processor -- start 2024-01-11T08:30:19.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.858+0000 7f9ee93ff700 1 -- start start 2024-01-11T08:30:19.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.859+0000 7f9ee93ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9ee40cc410 0x7f9ee4159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:19.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.859+0000 7f9ee93ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee40cd730 con 0x7f9ee40b2b90 2024-01-11T08:30:19.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.859+0000 7f9ee93ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee40ccb20 con 0x7f9ee40b03e0 2024-01-11T08:30:19.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.859+0000 7f9ee93ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee40cce00 con 0x7f9ee40cc410 2024-01-11T08:30:19.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.859+0000 7f9ee27fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9ee40b2b90 0x7f9ee4159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38728/0 (socket says 172.21.15.27:38728) 2024-01-11T08:30:19.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.859+0000 7f9ee27fc700 1 -- 172.21.15.27:0/1935224143 learned_addr learned my addr 172.21.15.27:0/1935224143 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:30:19.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.859+0000 7f9ee37fe700 1 --2- 172.21.15.27:0/1935224143 >> v2:172.21.15.77:3300/0 conn(0x7f9ee40cc410 0x7f9ee4159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:19.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.860+0000 7f9ee1ffb700 1 -- 172.21.15.27:0/1935224143 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589358560 0 0) 0x7f9ee40cd730 con 0x7f9ee40b2b90 2024-01-11T08:30:19.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.860+0000 7f9ee1ffb700 1 -- 172.21.15.27:0/1935224143 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ecc003570 con 0x7f9ee40b2b90 2024-01-11T08:30:19.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.860+0000 7f9ee1ffb700 1 -- 172.21.15.27:0/1935224143 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268559700 0 0) 0x7f9ee40ccb20 con 0x7f9ee40b03e0 2024-01-11T08:30:19.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.860+0000 7f9ee1ffb700 1 -- 172.21.15.27:0/1935224143 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee40cd730 con 0x7f9ee40b03e0 2024-01-11T08:30:19.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.860+0000 7f9ee1ffb700 1 -- 172.21.15.27:0/1935224143 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 447589337 0 0) 0x7f9ecc003570 con 0x7f9ee40b2b90 2024-01-11T08:30:19.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.860+0000 7f9ee1ffb700 1 -- 172.21.15.27:0/1935224143 >> v2:172.21.15.77:3300/0 conn(0x7f9ee40cc410 msgr2=0x7f9ee4159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:19.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.860+0000 7f9ee1ffb700 1 --2- 172.21.15.27:0/1935224143 >> v2:172.21.15.77:3300/0 conn(0x7f9ee40cc410 0x7f9ee4159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.860+0000 7f9ee1ffb700 1 -- 172.21.15.27:0/1935224143 >> v1:172.21.15.77:6789/0 conn(0x7f9ee40b03e0 legacy=0x7f9ee40b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:19.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.860+0000 7f9ee1ffb700 1 -- 172.21.15.27:0/1935224143 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee40cd0e0 con 0x7f9ee40b2b90 2024-01-11T08:30:19.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.860+0000 7f9ee1ffb700 1 -- 172.21.15.27:0/1935224143 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f9ed8001470 con 0x7f9ee40b2b90 2024-01-11T08:30:19.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.860+0000 7f9ee1ffb700 1 -- 172.21.15.27:0/1935224143 >> v1:172.21.15.27:6789/0 conn(0x7f9ee40b2b90 legacy=0x7f9ee4159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:19.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.860+0000 7f9ee1ffb700 1 --2- 172.21.15.27:0/1935224143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ecc003c70 0x7f9ecc004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:19.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.860+0000 7f9ee1ffb700 1 --2- 172.21.15.27:0/1935224143 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ecc004810 0x7f9ecc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:19.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.860+0000 7f9ee1ffb700 1 --2- 172.21.15.27:0/1935224143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ecc005210 0x7f9ecc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:19.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.860+0000 7f9ee1ffb700 1 -- 172.21.15.27:0/1935224143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee40cce00 con 0x7f9ecc005210 2024-01-11T08:30:19.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.860+0000 7f9ee1ffb700 1 -- 172.21.15.27:0/1935224143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc00c130 con 0x7f9ecc003c70 2024-01-11T08:30:19.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.860+0000 7f9ee1ffb700 1 -- 172.21.15.27:0/1935224143 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc00c2f0 con 0x7f9ecc004810 2024-01-11T08:30:19.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.860+0000 7f9ee37fe700 1 --2- 172.21.15.27:0/1935224143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ecc005210 0x7f9ecc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:19.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.861+0000 7f9ee2ffd700 1 --2- 172.21.15.27:0/1935224143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ecc003c70 0x7f9ecc004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:19.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.861+0000 7f9ee27fc700 1 --2- 172.21.15.27:0/1935224143 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ecc004810 0x7f9ecc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:19.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.861+0000 7f9ee37fe700 1 -- 172.21.15.27:0/1935224143 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ecc004810 msgr2=0x7f9ecc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:19.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.861+0000 7f9ee37fe700 1 --2- 172.21.15.27:0/1935224143 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ecc004810 0x7f9ecc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.861+0000 7f9ee37fe700 1 -- 172.21.15.27:0/1935224143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ecc003c70 msgr2=0x7f9ecc004060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:19.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.861+0000 7f9ee37fe700 1 --2- 172.21.15.27:0/1935224143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ecc003c70 0x7f9ecc004060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.861+0000 7f9ee37fe700 1 -- 172.21.15.27:0/1935224143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ecc003570 con 0x7f9ecc005210 2024-01-11T08:30:19.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.861+0000 7f9ee37fe700 1 -- 172.21.15.27:0/1935224143 reap_dead start 2024-01-11T08:30:19.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.861+0000 7f9ee37fe700 1 --2- 172.21.15.27:0/1935224143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ecc005210 0x7f9ecc00ba60 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0094d0 tx=0x7f9edc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b381953732d34ddd server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:19.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.861+0000 7f9ee1ffb700 1 -- 172.21.15.27:0/1935224143 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc00ea90 con 0x7f9ecc005210 2024-01-11T08:30:19.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.861+0000 7f9ee1ffb700 1 -- 172.21.15.27:0/1935224143 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9edc00f040 con 0x7f9ecc005210 2024-01-11T08:30:19.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.861+0000 7f9ee1ffb700 1 -- 172.21.15.27:0/1935224143 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc00ee40 con 0x7f9ecc005210 2024-01-11T08:30:19.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.862+0000 7f9ee93ff700 1 -- 172.21.15.27:0/1935224143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ecc005210 msgr2=0x7f9ecc00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:19.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.862+0000 7f9ee93ff700 1 --2- 172.21.15.27:0/1935224143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ecc005210 0x7f9ecc00ba60 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0094d0 tx=0x7f9edc00b040 comp rx=0 tx=0).stop 2024-01-11T08:30:19.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.862+0000 7f9ee93ff700 1 -- 172.21.15.27:0/1935224143 shutdown_connections 2024-01-11T08:30:19.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.862+0000 7f9ee93ff700 1 --2- 172.21.15.27:0/1935224143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ecc005210 0x7f9ecc00ba60 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.862+0000 7f9ee93ff700 1 -- 172.21.15.27:0/1935224143 >> 172.21.15.27:0/1935224143 conn(0x7f9ee406ed40 msgr2=0x7f9ee406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:19.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.862+0000 7f9ee93ff700 1 -- 172.21.15.27:0/1935224143 shutdown_connections 2024-01-11T08:30:19.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.862+0000 7f9ee93ff700 1 -- 172.21.15.27:0/1935224143 wait complete. 2024-01-11T08:30:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.864+0000 7f9ee93ff700 1 Processor -- start 2024-01-11T08:30:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.864+0000 7f9ee93ff700 1 -- start start 2024-01-11T08:30:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.865+0000 7f9ee93ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ecc005210 0x7f9ee415bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.865+0000 7f9ee93ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ee40a7d10 0x7f9ee40a8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.865+0000 7f9ee93ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ee416b690 0x7f9ee416bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.865+0000 7f9ee93ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee40a86c0 con 0x7f9ee40a7d10 2024-01-11T08:30:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.865+0000 7f9ee93ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee416c350 con 0x7f9ecc005210 2024-01-11T08:30:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.865+0000 7f9ee93ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee416c630 con 0x7f9ee416b690 2024-01-11T08:30:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.865+0000 7f9ee27fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ee40a7d10 0x7f9ee40a8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.865+0000 7f9ee27fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ee40a7d10 0x7f9ee40a8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34892/0 (socket says 172.21.15.27:34892) 2024-01-11T08:30:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.865+0000 7f9ee27fc700 1 -- 172.21.15.27:0/2819851009 learned_addr learned my addr 172.21.15.27:0/2819851009 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.865+0000 7f9ee2ffd700 1 --2- 172.21.15.27:0/2819851009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ecc005210 0x7f9ee415bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.865+0000 7f9ee37fe700 1 --2- 172.21.15.27:0/2819851009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ee416b690 0x7f9ee416bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.865+0000 7f9ee27fc700 1 -- 172.21.15.27:0/2819851009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ee416b690 msgr2=0x7f9ee416bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.865+0000 7f9ee27fc700 1 --2- 172.21.15.27:0/2819851009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ee416b690 0x7f9ee416bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.865+0000 7f9ee27fc700 1 -- 172.21.15.27:0/2819851009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ecc005210 msgr2=0x7f9ee415bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.865+0000 7f9ee27fc700 1 --2- 172.21.15.27:0/2819851009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ecc005210 0x7f9ee415bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.865+0000 7f9ee27fc700 1 -- 172.21.15.27:0/2819851009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee416c910 con 0x7f9ee40a7d10 2024-01-11T08:30:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.865+0000 7f9ee2ffd700 1 --2- 172.21.15.27:0/2819851009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ecc005210 0x7f9ee415bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:30:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.866+0000 7f9ee27fc700 1 --2- 172.21.15.27:0/2819851009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ee40a7d10 0x7f9ee40a8180 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8001e30 tx=0x7f9ed8001d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.866+0000 7f9ecaffd700 1 -- 172.21.15.27:0/2819851009 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed80077d0 con 0x7f9ee40a7d10 2024-01-11T08:30:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.866+0000 7f9ecaffd700 1 -- 172.21.15.27:0/2819851009 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed8000cb0 con 0x7f9ee40a7d10 2024-01-11T08:30:19.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.866+0000 7f9ecaffd700 1 -- 172.21.15.27:0/2819851009 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed800fb00 con 0x7f9ee40a7d10 2024-01-11T08:30:19.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.866+0000 7f9ee93ff700 1 -- 172.21.15.27:0/2819851009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee416cbf0 con 0x7f9ee40a7d10 2024-01-11T08:30:19.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.866+0000 7f9ee93ff700 1 -- 172.21.15.27:0/2819851009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee415c090 con 0x7f9ee40a7d10 2024-01-11T08:30:19.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.867+0000 7f9ee93ff700 1 -- 172.21.15.27:0/2819851009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee40718d0 con 0x7f9ee40a7d10 2024-01-11T08:30:19.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.868+0000 7f9ecaffd700 1 -- 172.21.15.27:0/2819851009 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9ed801d460 con 0x7f9ee40a7d10 2024-01-11T08:30:19.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.869+0000 7f9ecaffd700 1 --2- 172.21.15.27:0/2819851009 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9ed0066f30 0x7f9ed00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:19.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.870+0000 7f9ecaffd700 1 -- 172.21.15.27:0/2819851009 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f9ed8084680 con 0x7f9ee40a7d10 2024-01-11T08:30:19.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.870+0000 7f9ee2ffd700 1 --2- 172.21.15.27:0/2819851009 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9ed0066f30 0x7f9ed00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:19.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.871+0000 7f9ee2ffd700 1 --2- 172.21.15.27:0/2819851009 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9ed0066f30 0x7f9ed00693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4000c00 tx=0x7f9ed4009040 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:19.873+0000 7f9ecaffd700 1 -- 172.21.15.27:0/2819851009 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9ed80518c0 con 0x7f9ee40a7d10 2024-01-11T08:30:20.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.077+0000 7f9ee93ff700 1 -- 172.21.15.27:0/2819851009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ee40cd3c0 con 0x7f9ee40a7d10 2024-01-11T08:30:20.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.078+0000 7f9ecaffd700 1 -- 172.21.15.27:0/2819851009 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+15404 (secure 0 0 0) 0x7f9ed8054ee0 con 0x7f9ee40a7d10 2024-01-11T08:30:20.077 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:30:20.077 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":145,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T08:30:17.130787+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:30:13.161778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-01-11T08:30:14.620026+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T08:30:20.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.080+0000 7f9ee93ff700 1 -- 172.21.15.27:0/2819851009 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9ed0066f30 msgr2=0x7f9ed00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:20.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.080+0000 7f9ee93ff700 1 --2- 172.21.15.27:0/2819851009 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9ed0066f30 0x7f9ed00693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4000c00 tx=0x7f9ed4009040 comp rx=0 tx=0).stop 2024-01-11T08:30:20.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.080+0000 7f9ee93ff700 1 -- 172.21.15.27:0/2819851009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ee40a7d10 msgr2=0x7f9ee40a8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:20.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.080+0000 7f9ee93ff700 1 --2- 172.21.15.27:0/2819851009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ee40a7d10 0x7f9ee40a8180 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8001e30 tx=0x7f9ed8001d20 comp rx=0 tx=0).stop 2024-01-11T08:30:20.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.081+0000 7f9ee93ff700 1 -- 172.21.15.27:0/2819851009 shutdown_connections 2024-01-11T08:30:20.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.081+0000 7f9ee93ff700 1 --2- 172.21.15.27:0/2819851009 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9ed0066f30 0x7f9ed00693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:20.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.081+0000 7f9ee93ff700 1 --2- 172.21.15.27:0/2819851009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ee416b690 0x7f9ee416bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:20.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.081+0000 7f9ee93ff700 1 --2- 172.21.15.27:0/2819851009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ecc005210 0x7f9ee415bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:20.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.081+0000 7f9ee93ff700 1 --2- 172.21.15.27:0/2819851009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ee40a7d10 0x7f9ee40a8180 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:20.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.081+0000 7f9ee93ff700 1 -- 172.21.15.27:0/2819851009 >> 172.21.15.27:0/2819851009 conn(0x7f9ee406ed40 msgr2=0x7f9ee40b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:20.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.081+0000 7f9ee93ff700 1 -- 172.21.15.27:0/2819851009 shutdown_connections 2024-01-11T08:30:20.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.081+0000 7f9ee93ff700 1 -- 172.21.15.27:0/2819851009 wait complete. 2024-01-11T08:30:20.095 INFO:tasks.ceph.mds.d.smithi077.stderr:2024-01-11T08:30:20.096+0000 7fd10cdf1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T08:30:20.098 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-11T08:30:20.098 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:30:20.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.300+0000 7f5467bff700 1 Processor -- start 2024-01-11T08:30:20.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.300+0000 7f5467bff700 1 -- start start 2024-01-11T08:30:20.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.301+0000 7f5467bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f54600b2b90 0x7f5460159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:20.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.301+0000 7f5467bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f54600b07b0 0x7f5460159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:20.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.301+0000 7f5467bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54600cd730 con 0x7f54600b03e0 2024-01-11T08:30:20.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.301+0000 7f5467bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54600ccb20 con 0x7f54600b2b90 2024-01-11T08:30:20.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.301+0000 7f5467bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54600cce00 con 0x7f54600b07b0 2024-01-11T08:30:20.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.302+0000 7f546519a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f54600b2b90 0x7f5460159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:20.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.302+0000 7f546599b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f54600b03e0 0x7f54600b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38736/0 (socket says 172.21.15.27:38736) 2024-01-11T08:30:20.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.302+0000 7f546599b700 1 -- 172.21.15.27:0/2182001656 learned_addr learned my addr 172.21.15.27:0/2182001656 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:30:20.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.302+0000 7f546519a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f54600b2b90 0x7f5460159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34900/0 (socket says 172.21.15.27:34900) 2024-01-11T08:30:20.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.302+0000 7f546619c700 1 --2- 172.21.15.27:0/2182001656 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f54600b07b0 0x7f5460159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:20.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.302+0000 7f546519a700 1 -- 172.21.15.27:0/2182001656 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f54600b07b0 msgr2=0x7f5460159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:20.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.302+0000 7f546519a700 1 --2- 172.21.15.27:0/2182001656 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f54600b07b0 0x7f5460159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:20.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.302+0000 7f546519a700 1 -- 172.21.15.27:0/2182001656 >> v1:172.21.15.27:6789/0 conn(0x7f54600b03e0 legacy=0x7f54600b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:20.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.302+0000 7f546519a700 1 -- 172.21.15.27:0/2182001656 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54600cd0e0 con 0x7f54600b2b90 2024-01-11T08:30:20.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.302+0000 7f546519a700 1 --2- 172.21.15.27:0/2182001656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54600b2b90 0x7f5460159200 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f54500098e0 tx=0x7f54500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d6298b732d1a12 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:20.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.303+0000 7f5464999700 1 -- 172.21.15.27:0/2182001656 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5450009e80 con 0x7f54600b2b90 2024-01-11T08:30:20.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.303+0000 7f5464999700 1 -- 172.21.15.27:0/2182001656 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545000b860 con 0x7f54600b2b90 2024-01-11T08:30:20.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.303+0000 7f5464999700 1 -- 172.21.15.27:0/2182001656 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545000bbe0 con 0x7f54600b2b90 2024-01-11T08:30:20.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.303+0000 7f5467bff700 1 -- 172.21.15.27:0/2182001656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54600b2b90 msgr2=0x7f5460159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:20.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.303+0000 7f5467bff700 1 --2- 172.21.15.27:0/2182001656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54600b2b90 0x7f5460159200 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f54500098e0 tx=0x7f54500091a0 comp rx=0 tx=0).stop 2024-01-11T08:30:20.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.303+0000 7f5467bff700 1 -- 172.21.15.27:0/2182001656 shutdown_connections 2024-01-11T08:30:20.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.303+0000 7f5467bff700 1 --2- 172.21.15.27:0/2182001656 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f54600b07b0 0x7f5460159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:20.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.303+0000 7f5467bff700 1 --2- 172.21.15.27:0/2182001656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54600b2b90 0x7f5460159200 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:20.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.303+0000 7f5467bff700 1 -- 172.21.15.27:0/2182001656 >> 172.21.15.27:0/2182001656 conn(0x7f546006ed40 msgr2=0x7f546006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:20.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.303+0000 7f5467bff700 1 -- 172.21.15.27:0/2182001656 shutdown_connections 2024-01-11T08:30:20.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.303+0000 7f5467bff700 1 -- 172.21.15.27:0/2182001656 wait complete. 2024-01-11T08:30:20.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.305+0000 7f5467bff700 1 Processor -- start 2024-01-11T08:30:20.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.305+0000 7f5467bff700 1 -- start start 2024-01-11T08:30:20.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.306+0000 7f5467bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54600b03e0 0x7f54600c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:20.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.306+0000 7f5467bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54600b07b0 0x7f5460158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:20.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.306+0000 7f5467bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f54600b2b90 0x7f54600c8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:20.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.306+0000 7f5467bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54600aed60 con 0x7f54600b03e0 2024-01-11T08:30:20.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.306+0000 7f5467bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5460159010 con 0x7f54600b07b0 2024-01-11T08:30:20.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.306+0000 7f5467bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54600c7b60 con 0x7f54600b2b90 2024-01-11T08:30:20.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.306+0000 7f546599b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54600b03e0 0x7f54600c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:20.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.306+0000 7f546599b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54600b03e0 0x7f54600c6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34912/0 (socket says 172.21.15.27:34912) 2024-01-11T08:30:20.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.306+0000 7f546599b700 1 -- 172.21.15.27:0/1675518147 learned_addr learned my addr 172.21.15.27:0/1675518147 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:20.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.306+0000 7f546519a700 1 --2- 172.21.15.27:0/1675518147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54600b07b0 0x7f5460158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:20.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.306+0000 7f546619c700 1 --2- 172.21.15.27:0/1675518147 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f54600b2b90 0x7f54600c8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:20.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.307+0000 7f546599b700 1 -- 172.21.15.27:0/1675518147 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f54600b2b90 msgr2=0x7f54600c8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:20.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.307+0000 7f546599b700 1 --2- 172.21.15.27:0/1675518147 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f54600b2b90 0x7f54600c8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:20.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.307+0000 7f546599b700 1 -- 172.21.15.27:0/1675518147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54600b07b0 msgr2=0x7f5460158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:20.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.307+0000 7f546599b700 1 --2- 172.21.15.27:0/1675518147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54600b07b0 0x7f5460158ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:20.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.307+0000 7f546599b700 1 -- 172.21.15.27:0/1675518147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5450009420 con 0x7f54600b03e0 2024-01-11T08:30:20.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.307+0000 7f546599b700 1 --2- 172.21.15.27:0/1675518147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54600b03e0 0x7f54600c6f70 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f545c00c7c0 tx=0x7f545c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:20.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.307+0000 7f5455ffb700 1 -- 172.21.15.27:0/1675518147 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c005b30 con 0x7f54600b03e0 2024-01-11T08:30:20.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.307+0000 7f5455ffb700 1 -- 172.21.15.27:0/1675518147 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545c005c90 con 0x7f54600b03e0 2024-01-11T08:30:20.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.307+0000 7f5455ffb700 1 -- 172.21.15.27:0/1675518147 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c004a80 con 0x7f54600b03e0 2024-01-11T08:30:20.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.307+0000 7f5467bff700 1 -- 172.21.15.27:0/1675518147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54600c7e40 con 0x7f54600b03e0 2024-01-11T08:30:20.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.307+0000 7f5467bff700 1 -- 172.21.15.27:0/1675518147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54600c8400 con 0x7f54600b03e0 2024-01-11T08:30:20.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.308+0000 7f5467bff700 1 -- 172.21.15.27:0/1675518147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f546015db30 con 0x7f54600b03e0 2024-01-11T08:30:20.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.309+0000 7f5455ffb700 1 -- 172.21.15.27:0/1675518147 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f545c0070d0 con 0x7f54600b03e0 2024-01-11T08:30:20.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.310+0000 7f5455ffb700 1 --2- 172.21.15.27:0/1675518147 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5444066f30 0x7f54440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:20.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.310+0000 7f5455ffb700 1 -- 172.21.15.27:0/1675518147 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f545c083940 con 0x7f54600b03e0 2024-01-11T08:30:20.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.311+0000 7f546519a700 1 --2- 172.21.15.27:0/1675518147 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5444066f30 0x7f54440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:20.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.311+0000 7f546519a700 1 --2- 172.21.15.27:0/1675518147 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5444066f30 0x7f54440693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f54500091e0 tx=0x7f5450017000 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:20.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.314+0000 7f5455ffb700 1 -- 172.21.15.27:0/1675518147 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f545c04b5a0 con 0x7f54600b03e0 2024-01-11T08:30:20.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.511+0000 7f5467bff700 1 -- 172.21.15.27:0/1675518147 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f54600718d0 con 0x7f5444066f30 2024-01-11T08:30:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.523+0000 7f5455ffb700 1 -- 172.21.15.27:0/1675518147 <== mgr.8086 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289644 (secure 0 0 0) 0x7f54600718d0 con 0x7f5444066f30 2024-01-11T08:30:20.526 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:30:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.527+0000 7f5467bff700 1 -- 172.21.15.27:0/1675518147 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5444066f30 msgr2=0x7f54440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.527+0000 7f5467bff700 1 --2- 172.21.15.27:0/1675518147 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5444066f30 0x7f54440693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f54500091e0 tx=0x7f5450017000 comp rx=0 tx=0).stop 2024-01-11T08:30:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.527+0000 7f5467bff700 1 -- 172.21.15.27:0/1675518147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54600b03e0 msgr2=0x7f54600c6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.527+0000 7f5467bff700 1 --2- 172.21.15.27:0/1675518147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54600b03e0 0x7f54600c6f70 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f545c00c7c0 tx=0x7f545c007f70 comp rx=0 tx=0).stop 2024-01-11T08:30:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.527+0000 7f5467bff700 1 -- 172.21.15.27:0/1675518147 shutdown_connections 2024-01-11T08:30:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.527+0000 7f5467bff700 1 --2- 172.21.15.27:0/1675518147 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f5444066f30 0x7f54440693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.527+0000 7f5467bff700 1 --2- 172.21.15.27:0/1675518147 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f54600b2b90 0x7f54600c8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.527+0000 7f5467bff700 1 --2- 172.21.15.27:0/1675518147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54600b03e0 0x7f54600c6f70 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.527+0000 7f5467bff700 1 --2- 172.21.15.27:0/1675518147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54600b07b0 0x7f5460158ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.527+0000 7f5467bff700 1 -- 172.21.15.27:0/1675518147 >> 172.21.15.27:0/1675518147 conn(0x7f546006ed40 msgr2=0x7f54600d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.527+0000 7f5467bff700 1 -- 172.21.15.27:0/1675518147 shutdown_connections 2024-01-11T08:30:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.527+0000 7f5467bff700 1 -- 172.21.15.27:0/1675518147 wait complete. 2024-01-11T08:30:20.529 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:30:20.544 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-11T08:30:19.234186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1424,"num_read_kb":2954,"num_write":475,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":86,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2459092,"kb_used_data":15252,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752515628,"statfs":{"total":773094113280,"available":770576003072,"internally_reserved":0,"allocated":15618048,"data_stored":7912804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.122993"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.143946+0000","last_change":"2024-01-11T08:30:16.143946+0000","last_active":"2024-01-11T08:30:16.143946+0000","last_peered":"2024-01-11T08:30:16.143946+0000","last_clean":"2024-01-11T08:30:16.143946+0000","last_became_active":"2024-01-11T08:30:16.143546+0000","last_became_peered":"2024-01-11T08:30:16.143546+0000","last_unstale":"2024-01-11T08:30:16.143946+0000","last_undegraded":"2024-01-11T08:30:16.143946+0000","last_fullsized":"2024-01-11T08:30:16.143946+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:10:04.758738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.142248+0000","last_change":"2024-01-11T08:30:16.142248+0000","last_active":"2024-01-11T08:30:16.142248+0000","last_peered":"2024-01-11T08:30:16.142248+0000","last_clean":"2024-01-11T08:30:16.142248+0000","last_became_active":"2024-01-11T08:30:16.141936+0000","last_became_peered":"2024-01-11T08:30:16.141936+0000","last_unstale":"2024-01-11T08:30:16.142248+0000","last_undegraded":"2024-01-11T08:30:16.142248+0000","last_fullsized":"2024-01-11T08:30:16.142248+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:39:59.752162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.150380+0000","last_change":"2024-01-11T08:30:16.150380+0000","last_active":"2024-01-11T08:30:16.150380+0000","last_peered":"2024-01-11T08:30:16.150380+0000","last_clean":"2024-01-11T08:30:16.150380+0000","last_became_active":"2024-01-11T08:30:16.149429+0000","last_became_peered":"2024-01-11T08:30:16.149429+0000","last_unstale":"2024-01-11T08:30:16.150380+0000","last_undegraded":"2024-01-11T08:30:16.150380+0000","last_fullsized":"2024-01-11T08:30:16.150380+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:42:16.910360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.130424+0000","last_change":"2024-01-11T08:30:16.130424+0000","last_active":"2024-01-11T08:30:16.130424+0000","last_peered":"2024-01-11T08:30:16.130424+0000","last_clean":"2024-01-11T08:30:16.130424+0000","last_became_active":"2024-01-11T08:30:16.130147+0000","last_became_peered":"2024-01-11T08:30:16.130147+0000","last_unstale":"2024-01-11T08:30:16.130424+0000","last_undegraded":"2024-01-11T08:30:16.130424+0000","last_fullsized":"2024-01-11T08:30:16.130424+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:52.586601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.146250+0000","last_change":"2024-01-11T08:30:16.146250+0000","last_active":"2024-01-11T08:30:16.146250+0000","last_peered":"2024-01-11T08:30:16.146250+0000","last_clean":"2024-01-11T08:30:16.146250+0000","last_became_active":"2024-01-11T08:30:16.145994+0000","last_became_peered":"2024-01-11T08:30:16.145994+0000","last_unstale":"2024-01-11T08:30:16.146250+0000","last_undegraded":"2024-01-11T08:30:16.146250+0000","last_fullsized":"2024-01-11T08:30:16.146250+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:04:19.928740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.152342+0000","last_change":"2024-01-11T08:30:16.141328+0000","last_active":"2024-01-11T08:30:16.152342+0000","last_peered":"2024-01-11T08:30:16.152342+0000","last_clean":"2024-01-11T08:30:16.152342+0000","last_became_active":"2024-01-11T08:30:16.140856+0000","last_became_peered":"2024-01-11T08:30:16.140856+0000","last_unstale":"2024-01-11T08:30:16.152342+0000","last_undegraded":"2024-01-11T08:30:16.152342+0000","last_fullsized":"2024-01-11T08:30:16.152342+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:37:00.824614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.152761+0000","last_change":"2024-01-11T08:30:16.148890+0000","last_active":"2024-01-11T08:30:16.152761+0000","last_peered":"2024-01-11T08:30:16.152761+0000","last_clean":"2024-01-11T08:30:16.152761+0000","last_became_active":"2024-01-11T08:30:16.148585+0000","last_became_peered":"2024-01-11T08:30:16.148585+0000","last_unstale":"2024-01-11T08:30:16.152761+0000","last_undegraded":"2024-01-11T08:30:16.152761+0000","last_fullsized":"2024-01-11T08:30:16.152761+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:05:09.530419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.143282+0000","last_change":"2024-01-11T08:30:16.143282+0000","last_active":"2024-01-11T08:30:16.143282+0000","last_peered":"2024-01-11T08:30:16.143282+0000","last_clean":"2024-01-11T08:30:16.143282+0000","last_became_active":"2024-01-11T08:30:16.143027+0000","last_became_peered":"2024-01-11T08:30:16.143027+0000","last_unstale":"2024-01-11T08:30:16.143282+0000","last_undegraded":"2024-01-11T08:30:16.143282+0000","last_fullsized":"2024-01-11T08:30:16.143282+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:19:17.746788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.147452+0000","last_change":"2024-01-11T08:30:16.147452+0000","last_active":"2024-01-11T08:30:16.147452+0000","last_peered":"2024-01-11T08:30:16.147452+0000","last_clean":"2024-01-11T08:30:16.147452+0000","last_became_active":"2024-01-11T08:30:16.143243+0000","last_became_peered":"2024-01-11T08:30:16.143243+0000","last_unstale":"2024-01-11T08:30:16.147452+0000","last_undegraded":"2024-01-11T08:30:16.147452+0000","last_fullsized":"2024-01-11T08:30:16.147452+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:56:46.100091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.143040+0000","last_change":"2024-01-11T08:30:16.143040+0000","last_active":"2024-01-11T08:30:16.143040+0000","last_peered":"2024-01-11T08:30:16.143040+0000","last_clean":"2024-01-11T08:30:16.143040+0000","last_became_active":"2024-01-11T08:30:16.142747+0000","last_became_peered":"2024-01-11T08:30:16.142747+0000","last_unstale":"2024-01-11T08:30:16.143040+0000","last_undegraded":"2024-01-11T08:30:16.143040+0000","last_fullsized":"2024-01-11T08:30:16.143040+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:21:45.781511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141412+0000","last_change":"2024-01-11T08:30:16.141412+0000","last_active":"2024-01-11T08:30:16.141412+0000","last_peered":"2024-01-11T08:30:16.141412+0000","last_clean":"2024-01-11T08:30:16.141412+0000","last_became_active":"2024-01-11T08:30:16.141134+0000","last_became_peered":"2024-01-11T08:30:16.141134+0000","last_unstale":"2024-01-11T08:30:16.141412+0000","last_undegraded":"2024-01-11T08:30:16.141412+0000","last_fullsized":"2024-01-11T08:30:16.141412+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:00:59.316133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.142637+0000","last_change":"2024-01-11T08:30:16.142637+0000","last_active":"2024-01-11T08:30:16.142637+0000","last_peered":"2024-01-11T08:30:16.142637+0000","last_clean":"2024-01-11T08:30:16.142637+0000","last_became_active":"2024-01-11T08:30:16.142313+0000","last_became_peered":"2024-01-11T08:30:16.142313+0000","last_unstale":"2024-01-11T08:30:16.142637+0000","last_undegraded":"2024-01-11T08:30:16.142637+0000","last_fullsized":"2024-01-11T08:30:16.142637+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:25:31.845686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.137877+0000","last_change":"2024-01-11T08:30:16.137877+0000","last_active":"2024-01-11T08:30:16.137877+0000","last_peered":"2024-01-11T08:30:16.137877+0000","last_clean":"2024-01-11T08:30:16.137877+0000","last_became_active":"2024-01-11T08:30:16.136442+0000","last_became_peered":"2024-01-11T08:30:16.136442+0000","last_unstale":"2024-01-11T08:30:16.137877+0000","last_undegraded":"2024-01-11T08:30:16.137877+0000","last_fullsized":"2024-01-11T08:30:16.137877+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:04:58.411905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.136312+0000","last_change":"2024-01-11T08:30:16.136312+0000","last_active":"2024-01-11T08:30:16.136312+0000","last_peered":"2024-01-11T08:30:16.136312+0000","last_clean":"2024-01-11T08:30:16.136312+0000","last_became_active":"2024-01-11T08:30:16.136042+0000","last_became_peered":"2024-01-11T08:30:16.136042+0000","last_unstale":"2024-01-11T08:30:16.136312+0000","last_undegraded":"2024-01-11T08:30:16.136312+0000","last_fullsized":"2024-01-11T08:30:16.136312+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:44:05.447717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.143639+0000","last_change":"2024-01-11T08:30:16.143639+0000","last_active":"2024-01-11T08:30:16.143639+0000","last_peered":"2024-01-11T08:30:16.143639+0000","last_clean":"2024-01-11T08:30:16.143639+0000","last_became_active":"2024-01-11T08:30:16.143354+0000","last_became_peered":"2024-01-11T08:30:16.143354+0000","last_unstale":"2024-01-11T08:30:16.143639+0000","last_undegraded":"2024-01-11T08:30:16.143639+0000","last_fullsized":"2024-01-11T08:30:16.143639+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:10:16.641138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.140827+0000","last_change":"2024-01-11T08:30:16.135719+0000","last_active":"2024-01-11T08:30:16.140827+0000","last_peered":"2024-01-11T08:30:16.140827+0000","last_clean":"2024-01-11T08:30:16.140827+0000","last_became_active":"2024-01-11T08:30:16.135461+0000","last_became_peered":"2024-01-11T08:30:16.135461+0000","last_unstale":"2024-01-11T08:30:16.140827+0000","last_undegraded":"2024-01-11T08:30:16.140827+0000","last_fullsized":"2024-01-11T08:30:16.140827+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:33:47.586824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.151159+0000","last_change":"2024-01-11T08:30:16.139205+0000","last_active":"2024-01-11T08:30:16.151159+0000","last_peered":"2024-01-11T08:30:16.151159+0000","last_clean":"2024-01-11T08:30:16.151159+0000","last_became_active":"2024-01-11T08:30:16.138960+0000","last_became_peered":"2024-01-11T08:30:16.138960+0000","last_unstale":"2024-01-11T08:30:16.151159+0000","last_undegraded":"2024-01-11T08:30:16.151159+0000","last_fullsized":"2024-01-11T08:30:16.151159+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:04:19.928740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.144967+0000","last_change":"2024-01-11T08:30:16.139363+0000","last_active":"2024-01-11T08:30:16.144967+0000","last_peered":"2024-01-11T08:30:16.144967+0000","last_clean":"2024-01-11T08:30:16.144967+0000","last_became_active":"2024-01-11T08:30:16.138462+0000","last_became_peered":"2024-01-11T08:30:16.138462+0000","last_unstale":"2024-01-11T08:30:16.144967+0000","last_undegraded":"2024-01-11T08:30:16.144967+0000","last_fullsized":"2024-01-11T08:30:16.144967+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:56:46.100091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.142741+0000","last_change":"2024-01-11T08:30:16.142741+0000","last_active":"2024-01-11T08:30:16.142741+0000","last_peered":"2024-01-11T08:30:16.142741+0000","last_clean":"2024-01-11T08:30:16.142741+0000","last_became_active":"2024-01-11T08:30:16.142463+0000","last_became_peered":"2024-01-11T08:30:16.142463+0000","last_unstale":"2024-01-11T08:30:16.142741+0000","last_undegraded":"2024-01-11T08:30:16.142741+0000","last_fullsized":"2024-01-11T08:30:16.142741+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:59:13.612911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141099+0000","last_change":"2024-01-11T08:30:16.138465+0000","last_active":"2024-01-11T08:30:16.141099+0000","last_peered":"2024-01-11T08:30:16.141099+0000","last_clean":"2024-01-11T08:30:16.141099+0000","last_became_active":"2024-01-11T08:30:16.138200+0000","last_became_peered":"2024-01-11T08:30:16.138200+0000","last_unstale":"2024-01-11T08:30:16.141099+0000","last_undegraded":"2024-01-11T08:30:16.141099+0000","last_fullsized":"2024-01-11T08:30:16.141099+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:31:57.002018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.145162+0000","last_change":"2024-01-11T08:30:16.145162+0000","last_active":"2024-01-11T08:30:16.145162+0000","last_peered":"2024-01-11T08:30:16.145162+0000","last_clean":"2024-01-11T08:30:16.145162+0000","last_became_active":"2024-01-11T08:30:16.144960+0000","last_became_peered":"2024-01-11T08:30:16.144960+0000","last_unstale":"2024-01-11T08:30:16.145162+0000","last_undegraded":"2024-01-11T08:30:16.145162+0000","last_fullsized":"2024-01-11T08:30:16.145162+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:05:09.530419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.143068+0000","last_change":"2024-01-11T08:30:16.143068+0000","last_active":"2024-01-11T08:30:16.143068+0000","last_peered":"2024-01-11T08:30:16.143068+0000","last_clean":"2024-01-11T08:30:16.143068+0000","last_became_active":"2024-01-11T08:30:16.142095+0000","last_became_peered":"2024-01-11T08:30:16.142095+0000","last_unstale":"2024-01-11T08:30:16.143068+0000","last_undegraded":"2024-01-11T08:30:16.143068+0000","last_fullsized":"2024-01-11T08:30:16.143068+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:32:40.713675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141584+0000","last_change":"2024-01-11T08:30:16.141584+0000","last_active":"2024-01-11T08:30:16.141584+0000","last_peered":"2024-01-11T08:30:16.141584+0000","last_clean":"2024-01-11T08:30:16.141584+0000","last_became_active":"2024-01-11T08:30:16.141284+0000","last_became_peered":"2024-01-11T08:30:16.141284+0000","last_unstale":"2024-01-11T08:30:16.141584+0000","last_undegraded":"2024-01-11T08:30:16.141584+0000","last_fullsized":"2024-01-11T08:30:16.141584+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:39:48.244351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141234+0000","last_change":"2024-01-11T08:30:16.141234+0000","last_active":"2024-01-11T08:30:16.141234+0000","last_peered":"2024-01-11T08:30:16.141234+0000","last_clean":"2024-01-11T08:30:16.141234+0000","last_became_active":"2024-01-11T08:30:16.140926+0000","last_became_peered":"2024-01-11T08:30:16.140926+0000","last_unstale":"2024-01-11T08:30:16.141234+0000","last_undegraded":"2024-01-11T08:30:16.141234+0000","last_fullsized":"2024-01-11T08:30:16.141234+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:11:27.663936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.144423+0000","last_change":"2024-01-11T08:30:16.144423+0000","last_active":"2024-01-11T08:30:16.144423+0000","last_peered":"2024-01-11T08:30:16.144423+0000","last_clean":"2024-01-11T08:30:16.144423+0000","last_became_active":"2024-01-11T08:30:16.144182+0000","last_became_peered":"2024-01-11T08:30:16.144182+0000","last_unstale":"2024-01-11T08:30:16.144423+0000","last_undegraded":"2024-01-11T08:30:16.144423+0000","last_fullsized":"2024-01-11T08:30:16.144423+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:10:29.829592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141123+0000","last_change":"2024-01-11T08:30:16.136708+0000","last_active":"2024-01-11T08:30:16.141123+0000","last_peered":"2024-01-11T08:30:16.141123+0000","last_clean":"2024-01-11T08:30:16.141123+0000","last_became_active":"2024-01-11T08:30:16.136420+0000","last_became_peered":"2024-01-11T08:30:16.136420+0000","last_unstale":"2024-01-11T08:30:16.141123+0000","last_undegraded":"2024-01-11T08:30:16.141123+0000","last_fullsized":"2024-01-11T08:30:16.141123+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:42:16.910360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.137170+0000","last_change":"2024-01-11T08:30:16.137170+0000","last_active":"2024-01-11T08:30:16.137170+0000","last_peered":"2024-01-11T08:30:16.137170+0000","last_clean":"2024-01-11T08:30:16.137170+0000","last_became_active":"2024-01-11T08:30:16.136878+0000","last_became_peered":"2024-01-11T08:30:16.136878+0000","last_unstale":"2024-01-11T08:30:16.137170+0000","last_undegraded":"2024-01-11T08:30:16.137170+0000","last_fullsized":"2024-01-11T08:30:16.137170+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:43:51.764365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.131623+0000","last_change":"2024-01-11T08:30:16.128144+0000","last_active":"2024-01-11T08:30:16.131623+0000","last_peered":"2024-01-11T08:30:16.131623+0000","last_clean":"2024-01-11T08:30:16.131623+0000","last_became_active":"2024-01-11T08:30:16.127859+0000","last_became_peered":"2024-01-11T08:30:16.127859+0000","last_unstale":"2024-01-11T08:30:16.131623+0000","last_undegraded":"2024-01-11T08:30:16.131623+0000","last_fullsized":"2024-01-11T08:30:16.131623+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:39:34.480780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.143046+0000","last_change":"2024-01-11T08:30:16.143046+0000","last_active":"2024-01-11T08:30:16.143046+0000","last_peered":"2024-01-11T08:30:16.143046+0000","last_clean":"2024-01-11T08:30:16.143046+0000","last_became_active":"2024-01-11T08:30:16.141984+0000","last_became_peered":"2024-01-11T08:30:16.141984+0000","last_unstale":"2024-01-11T08:30:16.143046+0000","last_undegraded":"2024-01-11T08:30:16.143046+0000","last_fullsized":"2024-01-11T08:30:16.143046+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:16:34.884314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.145372+0000","last_change":"2024-01-11T08:30:16.145372+0000","last_active":"2024-01-11T08:30:16.145372+0000","last_peered":"2024-01-11T08:30:16.145372+0000","last_clean":"2024-01-11T08:30:16.145372+0000","last_became_active":"2024-01-11T08:30:16.145083+0000","last_became_peered":"2024-01-11T08:30:16.145083+0000","last_unstale":"2024-01-11T08:30:16.145372+0000","last_undegraded":"2024-01-11T08:30:16.145372+0000","last_fullsized":"2024-01-11T08:30:16.145372+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:07:50.377957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.140588+0000","last_change":"2024-01-11T08:30:16.140588+0000","last_active":"2024-01-11T08:30:16.140588+0000","last_peered":"2024-01-11T08:30:16.140588+0000","last_clean":"2024-01-11T08:30:16.140588+0000","last_became_active":"2024-01-11T08:30:16.140299+0000","last_became_peered":"2024-01-11T08:30:16.140299+0000","last_unstale":"2024-01-11T08:30:16.140588+0000","last_undegraded":"2024-01-11T08:30:16.140588+0000","last_fullsized":"2024-01-11T08:30:16.140588+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:42:53.996924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.152335+0000","last_change":"2024-01-11T08:30:16.141342+0000","last_active":"2024-01-11T08:30:16.152335+0000","last_peered":"2024-01-11T08:30:16.152335+0000","last_clean":"2024-01-11T08:30:16.152335+0000","last_became_active":"2024-01-11T08:30:16.139754+0000","last_became_peered":"2024-01-11T08:30:16.139754+0000","last_unstale":"2024-01-11T08:30:16.152335+0000","last_undegraded":"2024-01-11T08:30:16.152335+0000","last_fullsized":"2024-01-11T08:30:16.152335+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:46:01.822939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.137111+0000","last_change":"2024-01-11T08:30:16.132398+0000","last_active":"2024-01-11T08:30:16.137111+0000","last_peered":"2024-01-11T08:30:16.137111+0000","last_clean":"2024-01-11T08:30:16.137111+0000","last_became_active":"2024-01-11T08:30:16.132147+0000","last_became_peered":"2024-01-11T08:30:16.132147+0000","last_unstale":"2024-01-11T08:30:16.137111+0000","last_undegraded":"2024-01-11T08:30:16.137111+0000","last_fullsized":"2024-01-11T08:30:16.137111+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:39:48.244351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"144'4","reported_seq":13,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.160702+0000","last_change":"2024-01-11T08:30:15.138028+0000","last_active":"2024-01-11T08:30:16.160702+0000","last_peered":"2024-01-11T08:30:16.160702+0000","last_clean":"2024-01-11T08:30:16.160702+0000","last_became_active":"2024-01-11T08:30:15.137414+0000","last_became_peered":"2024-01-11T08:30:15.137414+0000","last_unstale":"2024-01-11T08:30:16.160702+0000","last_undegraded":"2024-01-11T08:30:16.160702+0000","last_fullsized":"2024-01-11T08:30:16.160702+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:42:06.560411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.147228+0000","last_change":"2024-01-11T08:30:16.147228+0000","last_active":"2024-01-11T08:30:16.147228+0000","last_peered":"2024-01-11T08:30:16.147228+0000","last_clean":"2024-01-11T08:30:16.147228+0000","last_became_active":"2024-01-11T08:30:16.142890+0000","last_became_peered":"2024-01-11T08:30:16.142890+0000","last_unstale":"2024-01-11T08:30:16.147228+0000","last_undegraded":"2024-01-11T08:30:16.147228+0000","last_fullsized":"2024-01-11T08:30:16.147228+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:37:00.824614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.159816+0000","last_change":"2024-01-11T08:30:15.130924+0000","last_active":"2024-01-11T08:30:16.159816+0000","last_peered":"2024-01-11T08:30:16.159816+0000","last_clean":"2024-01-11T08:30:16.159816+0000","last_became_active":"2024-01-11T08:30:15.130220+0000","last_became_peered":"2024-01-11T08:30:15.130220+0000","last_unstale":"2024-01-11T08:30:16.159816+0000","last_undegraded":"2024-01-11T08:30:16.159816+0000","last_fullsized":"2024-01-11T08:30:16.159816+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:39:58.743737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141748+0000","last_change":"2024-01-11T08:30:16.141748+0000","last_active":"2024-01-11T08:30:16.141748+0000","last_peered":"2024-01-11T08:30:16.141748+0000","last_clean":"2024-01-11T08:30:16.141748+0000","last_became_active":"2024-01-11T08:30:16.141458+0000","last_became_peered":"2024-01-11T08:30:16.141458+0000","last_unstale":"2024-01-11T08:30:16.141748+0000","last_undegraded":"2024-01-11T08:30:16.141748+0000","last_fullsized":"2024-01-11T08:30:16.141748+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:41:41.649132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"144'2","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.160444+0000","last_change":"2024-01-11T08:30:15.137966+0000","last_active":"2024-01-11T08:30:16.160444+0000","last_peered":"2024-01-11T08:30:16.160444+0000","last_clean":"2024-01-11T08:30:16.160444+0000","last_became_active":"2024-01-11T08:30:15.137678+0000","last_became_peered":"2024-01-11T08:30:15.137678+0000","last_unstale":"2024-01-11T08:30:16.160444+0000","last_undegraded":"2024-01-11T08:30:16.160444+0000","last_fullsized":"2024-01-11T08:30:16.160444+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:31:27.749785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.146971+0000","last_change":"2024-01-11T08:30:16.146971+0000","last_active":"2024-01-11T08:30:16.146971+0000","last_peered":"2024-01-11T08:30:16.146971+0000","last_clean":"2024-01-11T08:30:16.146971+0000","last_became_active":"2024-01-11T08:30:16.146700+0000","last_became_peered":"2024-01-11T08:30:16.146700+0000","last_unstale":"2024-01-11T08:30:16.146971+0000","last_undegraded":"2024-01-11T08:30:16.146971+0000","last_fullsized":"2024-01-11T08:30:16.146971+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:37:00.824614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"144'2","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.160385+0000","last_change":"2024-01-11T08:30:15.134992+0000","last_active":"2024-01-11T08:30:16.160385+0000","last_peered":"2024-01-11T08:30:16.160385+0000","last_clean":"2024-01-11T08:30:16.160385+0000","last_became_active":"2024-01-11T08:30:15.133331+0000","last_became_peered":"2024-01-11T08:30:15.133331+0000","last_unstale":"2024-01-11T08:30:16.160385+0000","last_undegraded":"2024-01-11T08:30:16.160385+0000","last_fullsized":"2024-01-11T08:30:16.160385+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:04:57.403479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141445+0000","last_change":"2024-01-11T08:30:16.136649+0000","last_active":"2024-01-11T08:30:16.141445+0000","last_peered":"2024-01-11T08:30:16.141445+0000","last_clean":"2024-01-11T08:30:16.141445+0000","last_became_active":"2024-01-11T08:30:16.135558+0000","last_became_peered":"2024-01-11T08:30:16.135558+0000","last_unstale":"2024-01-11T08:30:16.141445+0000","last_undegraded":"2024-01-11T08:30:16.141445+0000","last_fullsized":"2024-01-11T08:30:16.141445+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:37:00.824614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"144'1","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.131533+0000","last_change":"2024-01-11T08:30:15.131790+0000","last_active":"2024-01-11T08:30:16.131533+0000","last_peered":"2024-01-11T08:30:16.131533+0000","last_clean":"2024-01-11T08:30:16.131533+0000","last_became_active":"2024-01-11T08:30:15.131319+0000","last_became_peered":"2024-01-11T08:30:15.131319+0000","last_unstale":"2024-01-11T08:30:16.131533+0000","last_undegraded":"2024-01-11T08:30:16.131533+0000","last_fullsized":"2024-01-11T08:30:16.131533+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:40:49.059728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.150741+0000","last_change":"2024-01-11T08:30:16.138446+0000","last_active":"2024-01-11T08:30:16.150741+0000","last_peered":"2024-01-11T08:30:16.150741+0000","last_clean":"2024-01-11T08:30:16.150741+0000","last_became_active":"2024-01-11T08:30:16.138018+0000","last_became_peered":"2024-01-11T08:30:16.138018+0000","last_unstale":"2024-01-11T08:30:16.150741+0000","last_undegraded":"2024-01-11T08:30:16.150741+0000","last_fullsized":"2024-01-11T08:30:16.150741+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:41:41.649132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.162104+0000","last_change":"2024-01-11T08:30:15.123357+0000","last_active":"2024-01-11T08:30:16.162104+0000","last_peered":"2024-01-11T08:30:16.162104+0000","last_clean":"2024-01-11T08:30:16.162104+0000","last_became_active":"2024-01-11T08:30:15.123031+0000","last_became_peered":"2024-01-11T08:30:15.123031+0000","last_unstale":"2024-01-11T08:30:16.162104+0000","last_undegraded":"2024-01-11T08:30:16.162104+0000","last_fullsized":"2024-01-11T08:30:16.162104+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:42:18.913764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.140121+0000","last_change":"2024-01-11T08:30:16.140121+0000","last_active":"2024-01-11T08:30:16.140121+0000","last_peered":"2024-01-11T08:30:16.140121+0000","last_clean":"2024-01-11T08:30:16.140121+0000","last_became_active":"2024-01-11T08:30:16.139438+0000","last_became_peered":"2024-01-11T08:30:16.139438+0000","last_unstale":"2024-01-11T08:30:16.140121+0000","last_undegraded":"2024-01-11T08:30:16.140121+0000","last_fullsized":"2024-01-11T08:30:16.140121+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:18:01.054004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.131265+0000","last_change":"2024-01-11T08:30:15.133630+0000","last_active":"2024-01-11T08:30:16.131265+0000","last_peered":"2024-01-11T08:30:16.131265+0000","last_clean":"2024-01-11T08:30:16.131265+0000","last_became_active":"2024-01-11T08:30:15.133115+0000","last_became_peered":"2024-01-11T08:30:15.133115+0000","last_unstale":"2024-01-11T08:30:16.131265+0000","last_undegraded":"2024-01-11T08:30:16.131265+0000","last_fullsized":"2024-01-11T08:30:16.131265+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:13:48.277660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.138510+0000","last_change":"2024-01-11T08:30:16.138510+0000","last_active":"2024-01-11T08:30:16.138510+0000","last_peered":"2024-01-11T08:30:16.138510+0000","last_clean":"2024-01-11T08:30:16.138510+0000","last_became_active":"2024-01-11T08:30:16.138256+0000","last_became_peered":"2024-01-11T08:30:16.138256+0000","last_unstale":"2024-01-11T08:30:16.138510+0000","last_undegraded":"2024-01-11T08:30:16.138510+0000","last_fullsized":"2024-01-11T08:30:16.138510+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:47:19.039951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.136547+0000","last_change":"2024-01-11T08:30:15.132789+0000","last_active":"2024-01-11T08:30:15.136547+0000","last_peered":"2024-01-11T08:30:15.136547+0000","last_clean":"2024-01-11T08:30:15.136547+0000","last_became_active":"2024-01-11T08:30:15.132550+0000","last_became_peered":"2024-01-11T08:30:15.132550+0000","last_unstale":"2024-01-11T08:30:15.136547+0000","last_undegraded":"2024-01-11T08:30:15.136547+0000","last_fullsized":"2024-01-11T08:30:15.136547+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:43:50.755940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141745+0000","last_change":"2024-01-11T08:30:16.141745+0000","last_active":"2024-01-11T08:30:16.141745+0000","last_peered":"2024-01-11T08:30:16.141745+0000","last_clean":"2024-01-11T08:30:16.141745+0000","last_became_active":"2024-01-11T08:30:16.141487+0000","last_became_peered":"2024-01-11T08:30:16.141487+0000","last_unstale":"2024-01-11T08:30:16.141745+0000","last_undegraded":"2024-01-11T08:30:16.141745+0000","last_fullsized":"2024-01-11T08:30:16.141745+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:38:08.594999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.124542+0000","last_change":"2024-01-11T08:30:15.132780+0000","last_active":"2024-01-11T08:30:16.124542+0000","last_peered":"2024-01-11T08:30:16.124542+0000","last_clean":"2024-01-11T08:30:16.124542+0000","last_became_active":"2024-01-11T08:30:15.132135+0000","last_became_peered":"2024-01-11T08:30:15.132135+0000","last_unstale":"2024-01-11T08:30:16.124542+0000","last_undegraded":"2024-01-11T08:30:16.124542+0000","last_fullsized":"2024-01-11T08:30:16.124542+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:10:28.821166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.135703+0000","last_change":"2024-01-11T08:30:16.129747+0000","last_active":"2024-01-11T08:30:16.135703+0000","last_peered":"2024-01-11T08:30:16.135703+0000","last_clean":"2024-01-11T08:30:16.135703+0000","last_became_active":"2024-01-11T08:30:16.128576+0000","last_became_peered":"2024-01-11T08:30:16.128576+0000","last_unstale":"2024-01-11T08:30:16.135703+0000","last_undegraded":"2024-01-11T08:30:16.135703+0000","last_fullsized":"2024-01-11T08:30:16.135703+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:50:55.636631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.127216+0000","last_change":"2024-01-11T08:30:15.138007+0000","last_active":"2024-01-11T08:30:16.127216+0000","last_peered":"2024-01-11T08:30:16.127216+0000","last_clean":"2024-01-11T08:30:16.127216+0000","last_became_active":"2024-01-11T08:30:15.137149+0000","last_became_peered":"2024-01-11T08:30:15.137149+0000","last_unstale":"2024-01-11T08:30:16.127216+0000","last_undegraded":"2024-01-11T08:30:16.127216+0000","last_fullsized":"2024-01-11T08:30:16.127216+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:22:08.567433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.144564+0000","last_change":"2024-01-11T08:30:16.144564+0000","last_active":"2024-01-11T08:30:16.144564+0000","last_peered":"2024-01-11T08:30:16.144564+0000","last_clean":"2024-01-11T08:30:16.144564+0000","last_became_active":"2024-01-11T08:30:16.144153+0000","last_became_peered":"2024-01-11T08:30:16.144153+0000","last_unstale":"2024-01-11T08:30:16.144564+0000","last_undegraded":"2024-01-11T08:30:16.144564+0000","last_fullsized":"2024-01-11T08:30:16.144564+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:55:31.991780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"144'3","reported_seq":13,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.162585+0000","last_change":"2024-01-11T08:30:15.125204+0000","last_active":"2024-01-11T08:30:16.162585+0000","last_peered":"2024-01-11T08:30:16.162585+0000","last_clean":"2024-01-11T08:30:16.162585+0000","last_became_active":"2024-01-11T08:30:15.124648+0000","last_became_peered":"2024-01-11T08:30:15.124648+0000","last_unstale":"2024-01-11T08:30:16.162585+0000","last_undegraded":"2024-01-11T08:30:16.162585+0000","last_fullsized":"2024-01-11T08:30:16.162585+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:54:29.952959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.147354+0000","last_change":"2024-01-11T08:30:16.147354+0000","last_active":"2024-01-11T08:30:16.147354+0000","last_peered":"2024-01-11T08:30:16.147354+0000","last_clean":"2024-01-11T08:30:16.147354+0000","last_became_active":"2024-01-11T08:30:16.147065+0000","last_became_peered":"2024-01-11T08:30:16.147065+0000","last_unstale":"2024-01-11T08:30:16.147354+0000","last_undegraded":"2024-01-11T08:30:16.147354+0000","last_fullsized":"2024-01-11T08:30:16.147354+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:27:29.470280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"144'6","reported_seq":15,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.162601+0000","last_change":"2024-01-11T08:30:15.134571+0000","last_active":"2024-01-11T08:30:16.162601+0000","last_peered":"2024-01-11T08:30:16.162601+0000","last_clean":"2024-01-11T08:30:16.162601+0000","last_became_active":"2024-01-11T08:30:15.133172+0000","last_became_peered":"2024-01-11T08:30:15.133172+0000","last_unstale":"2024-01-11T08:30:16.162601+0000","last_undegraded":"2024-01-11T08:30:16.162601+0000","last_fullsized":"2024-01-11T08:30:16.162601+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:04:31.287522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.138480+0000","last_change":"2024-01-11T08:30:16.138480+0000","last_active":"2024-01-11T08:30:16.138480+0000","last_peered":"2024-01-11T08:30:16.138480+0000","last_clean":"2024-01-11T08:30:16.138480+0000","last_became_active":"2024-01-11T08:30:16.138200+0000","last_became_peered":"2024-01-11T08:30:16.138200+0000","last_unstale":"2024-01-11T08:30:16.138480+0000","last_undegraded":"2024-01-11T08:30:16.138480+0000","last_fullsized":"2024-01-11T08:30:16.138480+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:40:50.068154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.140076+0000","last_change":"2024-01-11T08:30:15.135258+0000","last_active":"2024-01-11T08:30:15.140076+0000","last_peered":"2024-01-11T08:30:15.140076+0000","last_clean":"2024-01-11T08:30:15.140076+0000","last_became_active":"2024-01-11T08:30:15.133480+0000","last_became_peered":"2024-01-11T08:30:15.133480+0000","last_unstale":"2024-01-11T08:30:15.140076+0000","last_undegraded":"2024-01-11T08:30:15.140076+0000","last_fullsized":"2024-01-11T08:30:15.140076+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:27:02.286647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.145951+0000","last_change":"2024-01-11T08:30:16.145951+0000","last_active":"2024-01-11T08:30:16.145951+0000","last_peered":"2024-01-11T08:30:16.145951+0000","last_clean":"2024-01-11T08:30:16.145951+0000","last_became_active":"2024-01-11T08:30:16.145671+0000","last_became_peered":"2024-01-11T08:30:16.145671+0000","last_unstale":"2024-01-11T08:30:16.145951+0000","last_undegraded":"2024-01-11T08:30:16.145951+0000","last_fullsized":"2024-01-11T08:30:16.145951+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:58:23.810853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.130899+0000","last_change":"2024-01-11T08:30:15.125107+0000","last_active":"2024-01-11T08:30:15.130899+0000","last_peered":"2024-01-11T08:30:15.130899+0000","last_clean":"2024-01-11T08:30:15.130899+0000","last_became_active":"2024-01-11T08:30:15.124715+0000","last_became_peered":"2024-01-11T08:30:15.124715+0000","last_unstale":"2024-01-11T08:30:15.130899+0000","last_undegraded":"2024-01-11T08:30:15.130899+0000","last_fullsized":"2024-01-11T08:30:15.130899+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:10:15.632713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141647+0000","last_change":"2024-01-11T08:30:16.141647+0000","last_active":"2024-01-11T08:30:16.141647+0000","last_peered":"2024-01-11T08:30:16.141647+0000","last_clean":"2024-01-11T08:30:16.141647+0000","last_became_active":"2024-01-11T08:30:16.141334+0000","last_became_peered":"2024-01-11T08:30:16.141334+0000","last_unstale":"2024-01-11T08:30:16.141647+0000","last_undegraded":"2024-01-11T08:30:16.141647+0000","last_fullsized":"2024-01-11T08:30:16.141647+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:13:49.286086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"144'8","reported_seq":17,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.164350+0000","last_change":"2024-01-11T08:30:15.131144+0000","last_active":"2024-01-11T08:30:16.164350+0000","last_peered":"2024-01-11T08:30:16.164350+0000","last_clean":"2024-01-11T08:30:16.164350+0000","last_became_active":"2024-01-11T08:30:15.130831+0000","last_became_peered":"2024-01-11T08:30:15.130831+0000","last_unstale":"2024-01-11T08:30:16.164350+0000","last_undegraded":"2024-01-11T08:30:16.164350+0000","last_fullsized":"2024-01-11T08:30:16.164350+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:42:18.913764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.147131+0000","last_change":"2024-01-11T08:30:16.147131+0000","last_active":"2024-01-11T08:30:16.147131+0000","last_peered":"2024-01-11T08:30:16.147131+0000","last_clean":"2024-01-11T08:30:16.147131+0000","last_became_active":"2024-01-11T08:30:16.146876+0000","last_became_peered":"2024-01-11T08:30:16.146876+0000","last_unstale":"2024-01-11T08:30:16.147131+0000","last_undegraded":"2024-01-11T08:30:16.147131+0000","last_fullsized":"2024-01-11T08:30:16.147131+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:31:57.002018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.162255+0000","last_change":"2024-01-11T08:30:15.130796+0000","last_active":"2024-01-11T08:30:16.162255+0000","last_peered":"2024-01-11T08:30:16.162255+0000","last_clean":"2024-01-11T08:30:16.162255+0000","last_became_active":"2024-01-11T08:30:15.130180+0000","last_became_peered":"2024-01-11T08:30:15.130180+0000","last_unstale":"2024-01-11T08:30:16.162255+0000","last_undegraded":"2024-01-11T08:30:16.162255+0000","last_fullsized":"2024-01-11T08:30:16.162255+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:42.449905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.144809+0000","last_change":"2024-01-11T08:30:16.142801+0000","last_active":"2024-01-11T08:30:16.144809+0000","last_peered":"2024-01-11T08:30:16.144809+0000","last_clean":"2024-01-11T08:30:16.144809+0000","last_became_active":"2024-01-11T08:30:16.142452+0000","last_became_peered":"2024-01-11T08:30:16.142452+0000","last_unstale":"2024-01-11T08:30:16.144809+0000","last_undegraded":"2024-01-11T08:30:16.144809+0000","last_fullsized":"2024-01-11T08:30:16.144809+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:02:12.583528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"144'2","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.161523+0000","last_change":"2024-01-11T08:30:15.135821+0000","last_active":"2024-01-11T08:30:16.161523+0000","last_peered":"2024-01-11T08:30:16.161523+0000","last_clean":"2024-01-11T08:30:16.161523+0000","last_became_active":"2024-01-11T08:30:15.135463+0000","last_became_peered":"2024-01-11T08:30:15.135463+0000","last_unstale":"2024-01-11T08:30:16.161523+0000","last_undegraded":"2024-01-11T08:30:16.161523+0000","last_fullsized":"2024-01-11T08:30:16.161523+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:09:47.457657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141080+0000","last_change":"2024-01-11T08:30:16.136859+0000","last_active":"2024-01-11T08:30:16.141080+0000","last_peered":"2024-01-11T08:30:16.141080+0000","last_clean":"2024-01-11T08:30:16.141080+0000","last_became_active":"2024-01-11T08:30:16.136570+0000","last_became_peered":"2024-01-11T08:30:16.136570+0000","last_unstale":"2024-01-11T08:30:16.141080+0000","last_undegraded":"2024-01-11T08:30:16.141080+0000","last_fullsized":"2024-01-11T08:30:16.141080+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:18:01.054004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.161771+0000","last_change":"2024-01-11T08:30:15.135328+0000","last_active":"2024-01-11T08:30:16.161771+0000","last_peered":"2024-01-11T08:30:16.161771+0000","last_clean":"2024-01-11T08:30:16.161771+0000","last_became_active":"2024-01-11T08:30:15.135078+0000","last_became_peered":"2024-01-11T08:30:15.135078+0000","last_unstale":"2024-01-11T08:30:16.161771+0000","last_undegraded":"2024-01-11T08:30:16.161771+0000","last_fullsized":"2024-01-11T08:30:16.161771+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:25:30.837261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.139966+0000","last_change":"2024-01-11T08:30:16.139966+0000","last_active":"2024-01-11T08:30:16.139966+0000","last_peered":"2024-01-11T08:30:16.139966+0000","last_clean":"2024-01-11T08:30:16.139966+0000","last_became_active":"2024-01-11T08:30:16.138694+0000","last_became_peered":"2024-01-11T08:30:16.138694+0000","last_unstale":"2024-01-11T08:30:16.139966+0000","last_undegraded":"2024-01-11T08:30:16.139966+0000","last_fullsized":"2024-01-11T08:30:16.139966+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:48.322813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.124300+0000","last_change":"2024-01-11T08:30:15.136191+0000","last_active":"2024-01-11T08:30:16.124300+0000","last_peered":"2024-01-11T08:30:16.124300+0000","last_clean":"2024-01-11T08:30:16.124300+0000","last_became_active":"2024-01-11T08:30:15.135840+0000","last_became_peered":"2024-01-11T08:30:15.135840+0000","last_unstale":"2024-01-11T08:30:16.124300+0000","last_undegraded":"2024-01-11T08:30:16.124300+0000","last_fullsized":"2024-01-11T08:30:16.124300+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:21:44.773086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.137233+0000","last_change":"2024-01-11T08:30:16.137233+0000","last_active":"2024-01-11T08:30:16.137233+0000","last_peered":"2024-01-11T08:30:16.137233+0000","last_clean":"2024-01-11T08:30:16.137233+0000","last_became_active":"2024-01-11T08:30:16.136671+0000","last_became_peered":"2024-01-11T08:30:16.136671+0000","last_unstale":"2024-01-11T08:30:16.137233+0000","last_undegraded":"2024-01-11T08:30:16.137233+0000","last_fullsized":"2024-01-11T08:30:16.137233+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:41:24.657291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.160091+0000","last_change":"2024-01-11T08:30:15.132771+0000","last_active":"2024-01-11T08:30:16.160091+0000","last_peered":"2024-01-11T08:30:16.160091+0000","last_clean":"2024-01-11T08:30:16.160091+0000","last_became_active":"2024-01-11T08:30:15.132014+0000","last_became_peered":"2024-01-11T08:30:15.132014+0000","last_unstale":"2024-01-11T08:30:16.160091+0000","last_undegraded":"2024-01-11T08:30:16.160091+0000","last_fullsized":"2024-01-11T08:30:16.160091+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:22:22.854310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.152813+0000","last_change":"2024-01-11T08:30:16.149298+0000","last_active":"2024-01-11T08:30:16.152813+0000","last_peered":"2024-01-11T08:30:16.152813+0000","last_clean":"2024-01-11T08:30:16.152813+0000","last_became_active":"2024-01-11T08:30:16.148629+0000","last_became_peered":"2024-01-11T08:30:16.148629+0000","last_unstale":"2024-01-11T08:30:16.152813+0000","last_undegraded":"2024-01-11T08:30:16.152813+0000","last_fullsized":"2024-01-11T08:30:16.152813+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:33:47.586824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.126525+0000","last_change":"2024-01-11T08:30:15.130187+0000","last_active":"2024-01-11T08:30:16.126525+0000","last_peered":"2024-01-11T08:30:16.126525+0000","last_clean":"2024-01-11T08:30:16.126525+0000","last_became_active":"2024-01-11T08:30:15.129793+0000","last_became_peered":"2024-01-11T08:30:15.129793+0000","last_unstale":"2024-01-11T08:30:16.126525+0000","last_undegraded":"2024-01-11T08:30:16.126525+0000","last_fullsized":"2024-01-11T08:30:16.126525+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:04:31.287522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.137861+0000","last_change":"2024-01-11T08:30:16.137861+0000","last_active":"2024-01-11T08:30:16.137861+0000","last_peered":"2024-01-11T08:30:16.137861+0000","last_clean":"2024-01-11T08:30:16.137861+0000","last_became_active":"2024-01-11T08:30:16.137534+0000","last_became_peered":"2024-01-11T08:30:16.137534+0000","last_unstale":"2024-01-11T08:30:16.137861+0000","last_undegraded":"2024-01-11T08:30:16.137861+0000","last_fullsized":"2024-01-11T08:30:16.137861+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:00:47.446985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.125438+0000","last_change":"2024-01-11T08:30:15.126662+0000","last_active":"2024-01-11T08:30:16.125438+0000","last_peered":"2024-01-11T08:30:16.125438+0000","last_clean":"2024-01-11T08:30:16.125438+0000","last_became_active":"2024-01-11T08:30:15.125810+0000","last_became_peered":"2024-01-11T08:30:15.125810+0000","last_unstale":"2024-01-11T08:30:16.125438+0000","last_undegraded":"2024-01-11T08:30:16.125438+0000","last_fullsized":"2024-01-11T08:30:16.125438+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:42:06.560411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.145680+0000","last_change":"2024-01-11T08:30:16.145680+0000","last_active":"2024-01-11T08:30:16.145680+0000","last_peered":"2024-01-11T08:30:16.145680+0000","last_clean":"2024-01-11T08:30:16.145680+0000","last_became_active":"2024-01-11T08:30:16.145384+0000","last_became_peered":"2024-01-11T08:30:16.145384+0000","last_unstale":"2024-01-11T08:30:16.145680+0000","last_undegraded":"2024-01-11T08:30:16.145680+0000","last_fullsized":"2024-01-11T08:30:16.145680+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:46:01.822939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.139787+0000","last_change":"2024-01-11T08:30:15.137127+0000","last_active":"2024-01-11T08:30:15.139787+0000","last_peered":"2024-01-11T08:30:15.139787+0000","last_clean":"2024-01-11T08:30:15.139787+0000","last_became_active":"2024-01-11T08:30:15.136803+0000","last_became_peered":"2024-01-11T08:30:15.136803+0000","last_unstale":"2024-01-11T08:30:15.139787+0000","last_undegraded":"2024-01-11T08:30:15.139787+0000","last_fullsized":"2024-01-11T08:30:15.139787+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:29:02.643186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.152478+0000","last_change":"2024-01-11T08:30:16.141789+0000","last_active":"2024-01-11T08:30:16.152478+0000","last_peered":"2024-01-11T08:30:16.152478+0000","last_clean":"2024-01-11T08:30:16.152478+0000","last_became_active":"2024-01-11T08:30:16.141540+0000","last_became_peered":"2024-01-11T08:30:16.141540+0000","last_unstale":"2024-01-11T08:30:16.152478+0000","last_undegraded":"2024-01-11T08:30:16.152478+0000","last_fullsized":"2024-01-11T08:30:16.152478+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:22:13.465322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"144'2","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.160977+0000","last_change":"2024-01-11T08:30:15.134093+0000","last_active":"2024-01-11T08:30:16.160977+0000","last_peered":"2024-01-11T08:30:16.160977+0000","last_clean":"2024-01-11T08:30:16.160977+0000","last_became_active":"2024-01-11T08:30:15.133551+0000","last_became_peered":"2024-01-11T08:30:15.133551+0000","last_unstale":"2024-01-11T08:30:16.160977+0000","last_undegraded":"2024-01-11T08:30:16.160977+0000","last_fullsized":"2024-01-11T08:30:16.160977+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:48:51.323814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.144503+0000","last_change":"2024-01-11T08:30:16.144503+0000","last_active":"2024-01-11T08:30:16.144503+0000","last_peered":"2024-01-11T08:30:16.144503+0000","last_clean":"2024-01-11T08:30:16.144503+0000","last_became_active":"2024-01-11T08:30:16.144222+0000","last_became_peered":"2024-01-11T08:30:16.144222+0000","last_unstale":"2024-01-11T08:30:16.144503+0000","last_undegraded":"2024-01-11T08:30:16.144503+0000","last_fullsized":"2024-01-11T08:30:16.144503+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:10:22.443710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.136936+0000","last_change":"2024-01-11T08:30:15.135166+0000","last_active":"2024-01-11T08:30:15.136936+0000","last_peered":"2024-01-11T08:30:15.136936+0000","last_clean":"2024-01-11T08:30:15.136936+0000","last_became_active":"2024-01-11T08:30:15.134222+0000","last_became_peered":"2024-01-11T08:30:15.134222+0000","last_unstale":"2024-01-11T08:30:15.136936+0000","last_undegraded":"2024-01-11T08:30:15.136936+0000","last_fullsized":"2024-01-11T08:30:15.136936+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:16:49.867287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.147632+0000","last_change":"2024-01-11T08:30:16.147632+0000","last_active":"2024-01-11T08:30:16.147632+0000","last_peered":"2024-01-11T08:30:16.147632+0000","last_clean":"2024-01-11T08:30:16.147632+0000","last_became_active":"2024-01-11T08:30:16.147389+0000","last_became_peered":"2024-01-11T08:30:16.147389+0000","last_unstale":"2024-01-11T08:30:16.147632+0000","last_undegraded":"2024-01-11T08:30:16.147632+0000","last_fullsized":"2024-01-11T08:30:16.147632+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:11:56.999227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.138475+0000","last_change":"2024-01-11T08:30:15.135652+0000","last_active":"2024-01-11T08:30:15.138475+0000","last_peered":"2024-01-11T08:30:15.138475+0000","last_clean":"2024-01-11T08:30:15.138475+0000","last_became_active":"2024-01-11T08:30:15.135391+0000","last_became_peered":"2024-01-11T08:30:15.135391+0000","last_unstale":"2024-01-11T08:30:15.138475+0000","last_undegraded":"2024-01-11T08:30:15.138475+0000","last_fullsized":"2024-01-11T08:30:15.138475+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:10:03.750312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.150794+0000","last_change":"2024-01-11T08:30:16.137616+0000","last_active":"2024-01-11T08:30:16.150794+0000","last_peered":"2024-01-11T08:30:16.150794+0000","last_clean":"2024-01-11T08:30:16.150794+0000","last_became_active":"2024-01-11T08:30:16.137349+0000","last_became_peered":"2024-01-11T08:30:16.137349+0000","last_unstale":"2024-01-11T08:30:16.150794+0000","last_undegraded":"2024-01-11T08:30:16.150794+0000","last_fullsized":"2024-01-11T08:30:16.150794+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:55:31.991780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.139332+0000","last_change":"2024-01-11T08:30:15.135331+0000","last_active":"2024-01-11T08:30:15.139332+0000","last_peered":"2024-01-11T08:30:15.139332+0000","last_clean":"2024-01-11T08:30:15.139332+0000","last_became_active":"2024-01-11T08:30:15.134851+0000","last_became_peered":"2024-01-11T08:30:15.134851+0000","last_unstale":"2024-01-11T08:30:15.139332+0000","last_undegraded":"2024-01-11T08:30:15.139332+0000","last_fullsized":"2024-01-11T08:30:15.139332+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:33:57.183661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.146652+0000","last_change":"2024-01-11T08:30:16.146652+0000","last_active":"2024-01-11T08:30:16.146652+0000","last_peered":"2024-01-11T08:30:16.146652+0000","last_clean":"2024-01-11T08:30:16.146652+0000","last_became_active":"2024-01-11T08:30:16.144158+0000","last_became_peered":"2024-01-11T08:30:16.144158+0000","last_unstale":"2024-01-11T08:30:16.146652+0000","last_undegraded":"2024-01-11T08:30:16.146652+0000","last_fullsized":"2024-01-11T08:30:16.146652+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:33:47.586824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.136538+0000","last_change":"2024-01-11T08:30:15.130938+0000","last_active":"2024-01-11T08:30:15.136538+0000","last_peered":"2024-01-11T08:30:15.136538+0000","last_clean":"2024-01-11T08:30:15.136538+0000","last_became_active":"2024-01-11T08:30:15.130636+0000","last_became_peered":"2024-01-11T08:30:15.130636+0000","last_unstale":"2024-01-11T08:30:15.136538+0000","last_undegraded":"2024-01-11T08:30:15.136538+0000","last_fullsized":"2024-01-11T08:30:15.136538+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:07:49.369531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.135741+0000","last_change":"2024-01-11T08:30:16.131802+0000","last_active":"2024-01-11T08:30:16.135741+0000","last_peered":"2024-01-11T08:30:16.135741+0000","last_clean":"2024-01-11T08:30:16.135741+0000","last_became_active":"2024-01-11T08:30:16.131536+0000","last_became_peered":"2024-01-11T08:30:16.131536+0000","last_unstale":"2024-01-11T08:30:16.135741+0000","last_undegraded":"2024-01-11T08:30:16.135741+0000","last_fullsized":"2024-01-11T08:30:16.135741+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:42:19.922189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.139400+0000","last_change":"2024-01-11T08:30:15.137866+0000","last_active":"2024-01-11T08:30:15.139400+0000","last_peered":"2024-01-11T08:30:15.139400+0000","last_clean":"2024-01-11T08:30:15.139400+0000","last_became_active":"2024-01-11T08:30:15.137562+0000","last_became_peered":"2024-01-11T08:30:15.137562+0000","last_unstale":"2024-01-11T08:30:15.139400+0000","last_undegraded":"2024-01-11T08:30:15.139400+0000","last_fullsized":"2024-01-11T08:30:15.139400+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:32:43.530518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.146544+0000","last_change":"2024-01-11T08:30:16.139974+0000","last_active":"2024-01-11T08:30:16.146544+0000","last_peered":"2024-01-11T08:30:16.146544+0000","last_clean":"2024-01-11T08:30:16.146544+0000","last_became_active":"2024-01-11T08:30:16.139432+0000","last_became_peered":"2024-01-11T08:30:16.139432+0000","last_unstale":"2024-01-11T08:30:16.146544+0000","last_undegraded":"2024-01-11T08:30:16.146544+0000","last_fullsized":"2024-01-11T08:30:16.146544+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:19:17.746788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.125883+0000","last_change":"2024-01-11T08:30:15.128567+0000","last_active":"2024-01-11T08:30:16.125883+0000","last_peered":"2024-01-11T08:30:16.125883+0000","last_clean":"2024-01-11T08:30:16.125883+0000","last_became_active":"2024-01-11T08:30:15.128307+0000","last_became_peered":"2024-01-11T08:30:15.128307+0000","last_unstale":"2024-01-11T08:30:16.125883+0000","last_undegraded":"2024-01-11T08:30:16.125883+0000","last_fullsized":"2024-01-11T08:30:16.125883+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:47:26.962304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.147528+0000","last_change":"2024-01-11T08:30:16.147528+0000","last_active":"2024-01-11T08:30:16.147528+0000","last_peered":"2024-01-11T08:30:16.147528+0000","last_clean":"2024-01-11T08:30:16.147528+0000","last_became_active":"2024-01-11T08:30:16.147270+0000","last_became_peered":"2024-01-11T08:30:16.147270+0000","last_unstale":"2024-01-11T08:30:16.147528+0000","last_undegraded":"2024-01-11T08:30:16.147528+0000","last_fullsized":"2024-01-11T08:30:16.147528+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:02:12.583528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.137017+0000","last_change":"2024-01-11T08:30:15.132288+0000","last_active":"2024-01-11T08:30:15.137017+0000","last_peered":"2024-01-11T08:30:15.137017+0000","last_clean":"2024-01-11T08:30:15.137017+0000","last_became_active":"2024-01-11T08:30:15.131944+0000","last_became_peered":"2024-01-11T08:30:15.131944+0000","last_unstale":"2024-01-11T08:30:15.137017+0000","last_undegraded":"2024-01-11T08:30:15.137017+0000","last_fullsized":"2024-01-11T08:30:15.137017+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:33.055906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.144810+0000","last_change":"2024-01-11T08:30:16.144810+0000","last_active":"2024-01-11T08:30:16.144810+0000","last_peered":"2024-01-11T08:30:16.144810+0000","last_clean":"2024-01-11T08:30:16.144810+0000","last_became_active":"2024-01-11T08:30:16.143818+0000","last_became_peered":"2024-01-11T08:30:16.143818+0000","last_unstale":"2024-01-11T08:30:16.144810+0000","last_undegraded":"2024-01-11T08:30:16.144810+0000","last_fullsized":"2024-01-11T08:30:16.144810+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:22:13.465322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"144'2","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.160442+0000","last_change":"2024-01-11T08:30:15.138266+0000","last_active":"2024-01-11T08:30:16.160442+0000","last_peered":"2024-01-11T08:30:16.160442+0000","last_clean":"2024-01-11T08:30:16.160442+0000","last_became_active":"2024-01-11T08:30:15.137599+0000","last_became_peered":"2024-01-11T08:30:15.137599+0000","last_unstale":"2024-01-11T08:30:16.160442+0000","last_undegraded":"2024-01-11T08:30:16.160442+0000","last_fullsized":"2024-01-11T08:30:16.160442+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:22:08.567433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"135'267","reported_seq":792,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.125966+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:30:15.125966+0000","last_peered":"2024-01-11T08:30:15.125966+0000","last_clean":"2024-01-11T08:30:15.125966+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:30:15.125966+0000","last_undegraded":"2024-01-11T08:30:15.125966+0000","last_fullsized":"2024-01-11T08:30:15.125966+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1424,"num_read_kb":2954,"num_write":431,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1424,"num_read_kb":2954,"num_write":431,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267,"ondisk_log_size":267,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575079,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303740,"kb_used_data":2356,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068100,"statfs":{"total":96636764160,"available":96325734400,"internally_reserved":0,"allocated":2412544,"data_stored":1432653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64600000000000002,"15min":0.68799999999999994},"min":{"1min":0.42399999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.745},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64200000000000002,"15min":0.69399999999999995},"min":{"1min":0.46999999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":1.073},"last":0.94499999999999995}]},{"osd":1,"last update":"Thu Jan 11 08:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65200000000000002,"15min":0.69099999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81699999999999995,"5min":0.93400000000000005,"15min":0.99099999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67400000000000004,"15min":0.70699999999999996},"min":{"1min":0.45100000000000001,"5min":0.41999999999999998,"15min":0.40999999999999998},"max":{"1min":0.86499999999999999,"5min":0.94999999999999996,"15min":1.014},"last":0.86499999999999999}]},{"osd":2,"last update":"Thu Jan 11 08:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.65900000000000003,"15min":0.71299999999999997},"min":{"1min":0.42099999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66900000000000004,"15min":0.70699999999999996},"min":{"1min":0.44700000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.95299999999999996,"5min":1.006,"15min":1.006},"last":0.84599999999999997}]},{"osd":3,"last update":"Thu Jan 11 08:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.68700000000000006,"15min":0.72699999999999998},"min":{"1min":0.50600000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.997,"5min":0.997,"15min":1.032},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.69799999999999995,"15min":0.72699999999999998},"min":{"1min":0.55600000000000005,"5min":0.439,"15min":0.439},"max":{"1min":0.95699999999999996,"5min":1.008,"15min":1.008},"last":0.80000000000000004}]},{"osd":4,"last update":"Thu Jan 11 08:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.496,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":0.80600000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64300000000000002,"15min":0.63200000000000001},"min":{"1min":0.38600000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.96399999999999997,"5min":1.002,"15min":1.002},"last":0.90100000000000002}]},{"osd":5,"last update":"Thu Jan 11 08:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.66200000000000003,"15min":0.64100000000000001},"min":{"1min":0.43099999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92600000000000005,"5min":1.042,"15min":1.042},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.66900000000000004,"15min":0.67200000000000004},"min":{"1min":0.503,"5min":0.41399999999999998,"15min":0.40400000000000003},"max":{"1min":0.97399999999999998,"5min":0.99199999999999999,"15min":1.0089999999999999},"last":0.98899999999999999}]},{"osd":6,"last update":"Thu Jan 11 08:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.73899999999999999,"15min":0.749},"min":{"1min":0.60499999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.028,"5min":1.028,"15min":1.0469999999999999},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.71699999999999997,"15min":0.72799999999999998},"min":{"1min":0.57199999999999995,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.91200000000000003,"5min":0.97599999999999998,"15min":1.028},"last":0.96099999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575078,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307244,"kb_used_data":1764,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064596,"statfs":{"total":96636764160,"available":96322146304,"internally_reserved":0,"allocated":1806336,"data_stored":841675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:29:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.61599999999999999,"15min":0.65700000000000003},"min":{"1min":0.34799999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.89500000000000002,"5min":0.91800000000000004,"15min":0.94799999999999995},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.628,"15min":0.67500000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.33800000000000002},"max":{"1min":0.78400000000000003,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.77300000000000002}]},{"osd":1,"last update":"Thu Jan 11 08:29:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65300000000000002,"15min":0.68799999999999994},"min":{"1min":0.41899999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.91900000000000004,"5min":1.008,"15min":1.163},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64800000000000002,"15min":0.67700000000000005},"min":{"1min":0.42599999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82699999999999996,"5min":0.92600000000000005,"15min":1.1950000000000001},"last":0.62}]},{"osd":2,"last update":"Thu Jan 11 08:29:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66300000000000003,"15min":0.69699999999999995},"min":{"1min":0.46899999999999997,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84799999999999998,"5min":0.94399999999999995,"15min":1.016},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66800000000000004,"15min":0.70999999999999996},"min":{"1min":0.47899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.88200000000000001,"5min":0.90000000000000002,"15min":0.97399999999999998},"last":0.64900000000000002}]},{"osd":3,"last update":"Thu Jan 11 08:29:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.69199999999999995,"15min":0.72099999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.876,"5min":0.98199999999999998,"15min":1.0149999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70399999999999996,"15min":0.73299999999999998},"min":{"1min":0.499,"5min":0.46000000000000002,"15min":0.41299999999999998},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":1.0049999999999999},"last":0.68899999999999995}]},{"osd":4,"last update":"Thu Jan 11 08:29:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.59199999999999997,"15min":0.56799999999999995},"min":{"1min":0.44900000000000001,"5min":0.36699999999999999,"15min":0.30199999999999999},"max":{"1min":0.84099999999999997,"5min":0.873,"15min":0.92000000000000004},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.61099999999999999,"15min":0.60199999999999998},"min":{"1min":0.442,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90000000000000002,"5min":0.90300000000000002,"15min":0.96399999999999997},"last":0.53100000000000003}]},{"osd":5,"last update":"Thu Jan 11 08:29:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65800000000000003,"15min":0.65500000000000003},"min":{"1min":0.42299999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.98599999999999999,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.755},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.71099999999999997,"15min":0.70299999999999996},"min":{"1min":0.52900000000000003,"5min":0.41799999999999998,"15min":0.35199999999999998},"max":{"1min":0.96099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.49399999999999999}]},{"osd":7,"last update":"Thu Jan 11 08:29:46 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.70799999999999996,"15min":0.71099999999999997},"min":{"1min":0.47899999999999998,"5min":0.44700000000000001,"15min":0.42399999999999999},"max":{"1min":0.97799999999999998,"5min":1.085,"15min":1.085},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.748,"15min":0.752},"min":{"1min":0.57699999999999996,"5min":0.46700000000000003,"15min":0.44700000000000001},"max":{"1min":1.1140000000000001,"5min":1.159,"15min":1.159},"last":0.81499999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575080,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303136,"kb_used_data":1752,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068704,"statfs":{"total":96636764160,"available":96326352896,"internally_reserved":0,"allocated":1794048,"data_stored":840139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46400000000000002,"15min":0.48299999999999998},"min":{"1min":0.30199999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.70099999999999996,"5min":0.85799999999999998,"15min":0.97699999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53500000000000003,"15min":0.57199999999999995},"min":{"1min":0.41499999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67500000000000004,"5min":0.91700000000000004,"15min":0.93999999999999995},"last":0.66400000000000003}]},{"osd":2,"last update":"Thu Jan 11 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.443,"15min":0.47499999999999998},"min":{"1min":0.34799999999999998,"5min":0.32300000000000001,"15min":0.312},"max":{"1min":0.63900000000000001,"5min":0.70299999999999996,"15min":1.113},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53200000000000003,"15min":0.55700000000000005},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.90100000000000002},"last":0.373}]},{"osd":3,"last update":"Thu Jan 11 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57899999999999996,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.92800000000000005},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.55600000000000005},"min":{"1min":0.38600000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.81000000000000005,"5min":0.88100000000000001,"15min":0.94899999999999995},"last":0.51900000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.76800000000000002,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.45900000000000002,"15min":0.439},"max":{"1min":1.097,"5min":1.129,"15min":1.2170000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.81699999999999995,"15min":0.82499999999999996},"min":{"1min":0.60599999999999998,"5min":0.59199999999999997,"15min":0.501},"max":{"1min":1.0249999999999999,"5min":1.109,"15min":1.246},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Jan 11 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.76800000000000002,"15min":0.79100000000000004},"min":{"1min":0.52000000000000002,"5min":0.498,"15min":0.498},"max":{"1min":1.0029999999999999,"5min":1.1299999999999999,"15min":1.218},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.746,"15min":0.77800000000000002},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.51200000000000001},"max":{"1min":0.98599999999999999,"5min":1.052,"15min":1.1779999999999999},"last":0.63500000000000001}]},{"osd":6,"last update":"Thu Jan 11 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78500000000000003,"15min":0.81299999999999994},"min":{"1min":0.57699999999999996,"5min":0.56000000000000005,"15min":0.51300000000000001},"max":{"1min":1.0660000000000001,"5min":1.115,"15min":1.1439999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79200000000000004,"15min":0.81899999999999995},"min":{"1min":0.56499999999999995,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.97199999999999998,"5min":1.091,"15min":1.228},"last":0.623}]},{"osd":7,"last update":"Thu Jan 11 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81299999999999994,"15min":0.84199999999999997},"min":{"1min":0.57699999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.1080000000000001,"5min":1.248,"15min":1.248},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.81399999999999995,"15min":0.83499999999999996},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0349999999999999,"5min":1.099,"15min":1.258},"last":0.69899999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575079,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303716,"kb_used_data":2332,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068124,"statfs":{"total":96636764160,"available":96325758976,"internally_reserved":0,"allocated":2387968,"data_stored":1430507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.47999999999999998,"15min":0.497},"min":{"1min":0.36399999999999999,"5min":0.28699999999999998,"15min":0.24299999999999999},"max":{"1min":0.54700000000000004,"5min":0.84399999999999997,"15min":0.92700000000000005},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54800000000000004,"15min":0.56799999999999995},"min":{"1min":0.40999999999999998,"5min":0.39900000000000002,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.89600000000000002,"15min":0.92100000000000004},"last":0.60099999999999998}]},{"osd":2,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.53400000000000003,"15min":0.54400000000000004},"min":{"1min":0.33300000000000002,"5min":0.33000000000000002,"15min":0.29099999999999998},"max":{"1min":0.68400000000000005,"5min":1.335,"15min":1.335},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54600000000000004,"15min":0.55500000000000005},"min":{"1min":0.35999999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.626,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.63700000000000001}]},{"osd":3,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.50800000000000001,"15min":0.52200000000000002},"min":{"1min":0.34999999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.628,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.60799999999999998,"15min":0.621},"min":{"1min":0.45500000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.75800000000000001,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.72699999999999998}]},{"osd":4,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.76400000000000001,"15min":0.78800000000000003},"min":{"1min":0.51900000000000002,"5min":0.502,"15min":0.437},"max":{"1min":1.0429999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.76000000000000001,"15min":0.78800000000000003},"min":{"1min":0.56799999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.99099999999999999,"5min":1.1220000000000001,"15min":1.149},"last":0.82299999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.76800000000000002,"15min":0.80100000000000005},"min":{"1min":0.56399999999999995,"5min":0.54700000000000004,"15min":0.52500000000000002},"max":{"1min":0.95299999999999996,"5min":1.518,"15min":1.518},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.78900000000000003,"15min":0.81499999999999995},"min":{"1min":0.58999999999999997,"5min":0.55400000000000005,"15min":0.47799999999999998},"max":{"1min":0.94999999999999996,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.84799999999999998}]},{"osd":6,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.78000000000000003,"15min":0.80200000000000005},"min":{"1min":0.54100000000000004,"5min":0.51700000000000002,"15min":0.50900000000000001},"max":{"1min":0.96099999999999997,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.80100000000000005,"15min":0.82799999999999996},"min":{"1min":0.52800000000000002,"5min":0.52000000000000002,"15min":0.44900000000000001},"max":{"1min":0.97299999999999998,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.93600000000000005}]},{"osd":7,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.81599999999999995,"15min":0.84199999999999997},"min":{"1min":0.63900000000000001,"5min":0.59199999999999997,"15min":0.51400000000000001},"max":{"1min":0.98499999999999999,"5min":1.371,"15min":1.371},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.79600000000000004,"15min":0.83399999999999996},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.056,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":1.02}]}]},{"osd":2,"up_from":13,"seq":55834575078,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311336,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060504,"statfs":{"total":96636764160,"available":96317956096,"internally_reserved":0,"allocated":1802240,"data_stored":840727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45000000000000001,"15min":0.46800000000000003},"min":{"1min":0.307,"5min":0.27800000000000002,"15min":0.26000000000000001},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.625},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.50800000000000001,"15min":0.53700000000000003},"min":{"1min":0.313,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.47699999999999998}]},{"osd":1,"last update":"Thu Jan 11 08:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53500000000000003,"15min":0.54600000000000004},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55200000000000005,"15min":0.57299999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.70999999999999996,"5min":0.79600000000000004,"15min":0.94499999999999995},"last":0.56299999999999994}]},{"osd":3,"last update":"Thu Jan 11 08:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.52200000000000002,"15min":0.54100000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.901,"5min":1.901,"15min":1.901},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.58399999999999996,"15min":0.623},"min":{"1min":0.436,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":1.0009999999999999},"last":0.51900000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.73799999999999999,"15min":0.77700000000000002},"min":{"1min":0.5,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.2889999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.73599999999999999,"15min":0.77200000000000002},"min":{"1min":0.51700000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.61199999999999999}]},{"osd":5,"last update":"Thu Jan 11 08:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.79600000000000004,"15min":0.82399999999999995},"min":{"1min":0.61899999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0229999999999999,"5min":1.2470000000000001,"15min":1.3540000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79200000000000004,"15min":0.83099999999999996},"min":{"1min":0.57299999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.749}]},{"osd":6,"last update":"Thu Jan 11 08:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.746,"15min":0.78800000000000003},"min":{"1min":0.50700000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.77500000000000002,"15min":0.81299999999999994},"min":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.073,"5min":1.351,"15min":1.47},"last":0.64900000000000002}]},{"osd":7,"last update":"Thu Jan 11 08:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.79000000000000004,"15min":0.84399999999999997},"min":{"1min":0.54600000000000004,"5min":0.54000000000000004,"15min":0.51400000000000001},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.4099999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.80100000000000005,"15min":0.84599999999999997},"min":{"1min":0.55700000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.992,"5min":1.992,"15min":1.992},"last":0.72899999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575078,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311336,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060504,"statfs":{"total":96636764160,"available":96317956096,"internally_reserved":0,"allocated":1802240,"data_stored":840299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.49199999999999999,"15min":0.49099999999999999},"min":{"1min":0.309,"5min":0.29799999999999999,"15min":0.27600000000000002},"max":{"1min":0.79900000000000004,"5min":0.80400000000000005,"15min":1.1930000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.54200000000000004},"min":{"1min":0.378,"5min":0.36699999999999999,"15min":0.33400000000000002},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.441}]},{"osd":1,"last update":"Thu Jan 11 08:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.48799999999999999,"15min":0.48899999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.25900000000000001},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55500000000000005,"15min":0.57099999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.36299999999999999},"max":{"1min":1.0069999999999999,"5min":1.0960000000000001,"15min":1.1499999999999999},"last":0.47099999999999997}]},{"osd":2,"last update":"Thu Jan 11 08:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56200000000000006,"15min":0.57699999999999996},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78700000000000003,"5min":0.94699999999999995,"15min":0.97599999999999998},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.61099999999999999,"15min":0.628},"min":{"1min":0.433,"5min":0.42899999999999999,"15min":0.32000000000000001},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":0.61599999999999999}]},{"osd":4,"last update":"Thu Jan 11 08:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72399999999999998,"15min":0.73299999999999998},"min":{"1min":0.53100000000000003,"5min":0.495,"15min":0.35999999999999999},"max":{"1min":1.4690000000000001,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.75,"15min":0.76900000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.503},"max":{"1min":1.1259999999999999,"5min":1.1459999999999999,"15min":1.1819999999999999},"last":0.72399999999999998}]},{"osd":5,"last update":"Thu Jan 11 08:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76300000000000001,"15min":0.78000000000000003},"min":{"1min":0.56599999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.96199999999999997,"5min":1.0649999999999999,"15min":1.3260000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79200000000000004,"15min":0.80200000000000005},"min":{"1min":0.52800000000000002,"5min":0.5,"15min":0.49199999999999999},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.80700000000000005}]},{"osd":6,"last update":"Thu Jan 11 08:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.80700000000000005,"15min":0.82799999999999996},"min":{"1min":0.57199999999999995,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.77400000000000002,"15min":0.78900000000000003},"min":{"1min":0.52300000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.425,"5min":2.3599999999999999,"15min":2.3599999999999999},"last":0.745}]},{"osd":7,"last update":"Thu Jan 11 08:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82299999999999995,"15min":0.83099999999999996},"min":{"1min":0.60999999999999999,"5min":0.53100000000000003,"15min":0.46600000000000003},"max":{"1min":1.516,"5min":2.3180000000000001,"15min":2.3180000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83699999999999997,"15min":0.85399999999999998},"min":{"1min":0.626,"5min":0.59199999999999997,"15min":0.58399999999999996},"max":{"1min":1.1639999999999999,"5min":1.2050000000000001,"15min":1.21},"last":0.83299999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575079,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311332,"kb_used_data":1756,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060508,"statfs":{"total":96636764160,"available":96317960192,"internally_reserved":0,"allocated":1798144,"data_stored":840209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.67300000000000004,"15min":0.72899999999999998},"min":{"1min":0.51300000000000001,"5min":0.41999999999999998,"15min":0.28100000000000003},"max":{"1min":0.76500000000000001,"5min":1.0149999999999999,"15min":1.117},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.67900000000000005,"15min":0.72699999999999998},"min":{"1min":0.48299999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80800000000000005,"5min":0.95699999999999996,"15min":1.0049999999999999},"last":0.58499999999999996}]},{"osd":1,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.65500000000000003,"15min":0.69399999999999995},"min":{"1min":0.48899999999999999,"5min":0.39100000000000001,"15min":0.33300000000000002},"max":{"1min":0.89300000000000002,"5min":0.91000000000000003,"15min":1.0069999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.66300000000000003,"15min":0.70199999999999996},"min":{"1min":0.48499999999999999,"5min":0.43099999999999999,"15min":0.36299999999999999},"max":{"1min":0.84899999999999998,"5min":0.94899999999999995,"15min":1.046},"last":0.61799999999999999}]},{"osd":2,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.68799999999999994,"15min":0.71599999999999997},"min":{"1min":0.505,"5min":0.46600000000000003,"15min":0.38900000000000001},"max":{"1min":0.82099999999999995,"5min":1.03,"15min":1.03},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.66000000000000003,"15min":0.71499999999999997},"min":{"1min":0.45900000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.751,"5min":0.91500000000000004,"15min":1.0409999999999999},"last":0.751}]},{"osd":3,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.64900000000000002,"15min":0.69099999999999995},"min":{"1min":0.48399999999999999,"5min":0.46100000000000002,"15min":0.433},"max":{"1min":0.753,"5min":0.96799999999999997,"15min":1.0189999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.69699999999999995,"15min":0.73699999999999999},"min":{"1min":0.47299999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.92400000000000004,"5min":0.97499999999999998,"15min":1.026},"last":0.69999999999999996}]},{"osd":5,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.60599999999999998,"15min":0.60699999999999998},"min":{"1min":0.39800000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79400000000000004,"5min":0.96599999999999997,"15min":1.0700000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.61799999999999999,"15min":0.62},"min":{"1min":0.44700000000000001,"5min":0.39400000000000002,"15min":0.32600000000000001},"max":{"1min":0.77500000000000002,"5min":0.94299999999999995,"15min":0.97799999999999998},"last":0.89400000000000002}]},{"osd":6,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.64800000000000002,"15min":0.64500000000000002},"min":{"1min":0.47099999999999997,"5min":0.40500000000000003,"15min":0.39500000000000002},"max":{"1min":0.79400000000000004,"5min":1.0680000000000001,"15min":1.133},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.69899999999999995,"15min":0.68799999999999994},"min":{"1min":0.45900000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.89300000000000002,"5min":1.0129999999999999,"15min":1.119},"last":0.83899999999999997}]},{"osd":7,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.68100000000000005,"15min":0.68200000000000005},"min":{"1min":0.51800000000000002,"5min":0.44600000000000001,"15min":0.41299999999999998},"max":{"1min":0.83199999999999996,"5min":1.0129999999999999,"15min":1.0640000000000001},"last":1.01},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.71999999999999997,"15min":0.72999999999999998},"min":{"1min":0.55000000000000004,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.84599999999999997,"5min":1.028,"15min":1.2350000000000001},"last":0.82499999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575079,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307252,"kb_used_data":1772,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064588,"statfs":{"total":96636764160,"available":96322138112,"internally_reserved":0,"allocated":1814528,"data_stored":846595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.65700000000000003,"15min":0.69999999999999996},"min":{"1min":0.44500000000000001,"5min":0.39300000000000002,"15min":0.38700000000000001},"max":{"1min":0.98499999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.66800000000000004,"15min":0.70599999999999996},"min":{"1min":0.498,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.878,"5min":0.93500000000000005,"15min":1.696},"last":0.80600000000000005}]},{"osd":1,"last update":"Thu Jan 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.64000000000000001,"15min":0.68799999999999994},"min":{"1min":0.41699999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.94399999999999995,"5min":0.997,"15min":1.4359999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.67000000000000004,"15min":0.72599999999999998},"min":{"1min":0.42899999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.98099999999999998,"5min":1.05,"15min":4.4969999999999999},"last":0.60399999999999998}]},{"osd":2,"last update":"Thu Jan 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66500000000000004,"15min":0.70299999999999996},"min":{"1min":0.39800000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.01,"5min":1.01,"15min":1.238},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67600000000000005,"15min":0.73499999999999999},"min":{"1min":0.47099999999999997,"5min":0.45300000000000001,"15min":0.41599999999999998},"max":{"1min":0.91900000000000004,"5min":0.95899999999999996,"15min":8.0429999999999993},"last":0.63}]},{"osd":3,"last update":"Thu Jan 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.69099999999999995,"15min":0.745},"min":{"1min":0.48799999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.052,"5min":1.208,"15min":8.0839999999999996},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.71399999999999997,"15min":0.73799999999999999},"min":{"1min":0.50600000000000001,"5min":0.49099999999999999,"15min":0.40899999999999997},"max":{"1min":1.0409999999999999,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.65100000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.59399999999999997,"15min":0.59899999999999998},"min":{"1min":0.45400000000000001,"5min":0.38300000000000001,"15min":0.34300000000000003},"max":{"1min":0.78100000000000003,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.63500000000000001,"15min":0.63100000000000001},"min":{"1min":0.432,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.91400000000000003,"5min":1.256,"15min":1.256},"last":0.81999999999999995}]},{"osd":6,"last update":"Thu Jan 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.39000000000000001},"max":{"1min":0.96499999999999997,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.67000000000000004,"15min":0.67900000000000005},"min":{"1min":0.46400000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.90300000000000002,"5min":1.085,"15min":1.085},"last":0.72599999999999998}]},{"osd":7,"last update":"Thu Jan 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.70599999999999996,"15min":0.72499999999999998},"min":{"1min":0.49199999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.038,"5min":1.341,"15min":1.341},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.76300000000000001,"15min":0.76900000000000002},"min":{"1min":0.53900000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.1120000000000001,"5min":1.119,"15min":1.119},"last":0.84499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:30:20.550 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T08:30:20.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.785+0000 7fc478bff700 1 Processor -- start 2024-01-11T08:30:20.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.785+0000 7fc478bff700 1 -- start start 2024-01-11T08:30:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.787+0000 7fc478bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4740b2b90 0x7fc474159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.787+0000 7fc478bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4740cd730 con 0x7fc4740b03e0 2024-01-11T08:30:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.787+0000 7fc478bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4740ccb20 con 0x7fc4740b07b0 2024-01-11T08:30:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.787+0000 7fc478bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4740cce00 con 0x7fc4740b2b90 2024-01-11T08:30:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.787+0000 7fc47259c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc4740b03e0 0x7fc4740b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38738/0 (socket says 172.21.15.27:38738) 2024-01-11T08:30:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.787+0000 7fc47259c700 1 -- 172.21.15.27:0/4241628148 learned_addr learned my addr 172.21.15.27:0/4241628148 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:30:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.787+0000 7fc46bfff700 1 --2- 172.21.15.27:0/4241628148 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4740b2b90 0x7fc474159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.787+0000 7fc471d9b700 1 -- 172.21.15.27:0/4241628148 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948524848 0 0) 0x7fc4740cd730 con 0x7fc4740b03e0 2024-01-11T08:30:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.788+0000 7fc471d9b700 1 -- 172.21.15.27:0/4241628148 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc460003570 con 0x7fc4740b03e0 2024-01-11T08:30:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.788+0000 7fc471d9b700 1 -- 172.21.15.27:0/4241628148 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359358585 0 0) 0x7fc4740ccb20 con 0x7fc4740b07b0 2024-01-11T08:30:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.788+0000 7fc471d9b700 1 -- 172.21.15.27:0/4241628148 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4740cd730 con 0x7fc4740b07b0 2024-01-11T08:30:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.788+0000 7fc471d9b700 1 -- 172.21.15.27:0/4241628148 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4097407783 0 0) 0x7fc460003570 con 0x7fc4740b03e0 2024-01-11T08:30:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.788+0000 7fc471d9b700 1 -- 172.21.15.27:0/4241628148 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4740b2b90 msgr2=0x7fc474159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.788+0000 7fc471d9b700 1 --2- 172.21.15.27:0/4241628148 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4740b2b90 0x7fc474159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.788+0000 7fc471d9b700 1 -- 172.21.15.27:0/4241628148 >> v1:172.21.15.77:6789/0 conn(0x7fc4740b07b0 legacy=0x7fc474159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.788+0000 7fc471d9b700 1 -- 172.21.15.27:0/4241628148 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4740cd0e0 con 0x7fc4740b03e0 2024-01-11T08:30:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.788+0000 7fc471d9b700 1 -- 172.21.15.27:0/4241628148 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fc464001470 con 0x7fc4740b03e0 2024-01-11T08:30:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.788+0000 7fc471d9b700 1 -- 172.21.15.27:0/4241628148 >> v1:172.21.15.27:6789/0 conn(0x7fc4740b03e0 legacy=0x7fc4740b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.788+0000 7fc471d9b700 1 --2- 172.21.15.27:0/4241628148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc460003c20 0x7fc460004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.788+0000 7fc471d9b700 1 --2- 172.21.15.27:0/4241628148 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4740b2b90 0x7fc460004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.788+0000 7fc471d9b700 1 --2- 172.21.15.27:0/4241628148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc460004d50 0x7fc4600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.788+0000 7fc471d9b700 1 -- 172.21.15.27:0/4241628148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4740cce00 con 0x7fc460003c20 2024-01-11T08:30:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.788+0000 7fc471d9b700 1 -- 172.21.15.27:0/4241628148 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46000bbf0 con 0x7fc460004d50 2024-01-11T08:30:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.788+0000 7fc471d9b700 1 -- 172.21.15.27:0/4241628148 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc46000bd60 con 0x7fc4740b2b90 2024-01-11T08:30:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.788+0000 7fc47259c700 1 --2- 172.21.15.27:0/4241628148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc460003c20 0x7fc460004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.788+0000 7fc46bfff700 1 --2- 172.21.15.27:0/4241628148 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4740b2b90 0x7fc460004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.788+0000 7fc472d9d700 1 --2- 172.21.15.27:0/4241628148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc460004d50 0x7fc4600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:20.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.789+0000 7fc47259c700 1 -- 172.21.15.27:0/4241628148 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4740b2b90 msgr2=0x7fc460004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:20.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.789+0000 7fc47259c700 1 --2- 172.21.15.27:0/4241628148 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4740b2b90 0x7fc460004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:20.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.789+0000 7fc47259c700 1 -- 172.21.15.27:0/4241628148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc460004d50 msgr2=0x7fc4600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:20.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.789+0000 7fc47259c700 1 --2- 172.21.15.27:0/4241628148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc460004d50 0x7fc4600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:20.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.789+0000 7fc47259c700 1 -- 172.21.15.27:0/4241628148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc460003570 con 0x7fc460003c20 2024-01-11T08:30:20.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.789+0000 7fc47259c700 1 --2- 172.21.15.27:0/4241628148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc460003c20 0x7fc460004010 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fc46400c1e0 tx=0x7fc464001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd6bb3ab1e1625b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:20.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.789+0000 7fc471d9b700 1 -- 172.21.15.27:0/4241628148 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46400cbe0 con 0x7fc460003c20 2024-01-11T08:30:20.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.789+0000 7fc471d9b700 1 -- 172.21.15.27:0/4241628148 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46400cd40 con 0x7fc460003c20 2024-01-11T08:30:20.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.789+0000 7fc471d9b700 1 -- 172.21.15.27:0/4241628148 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc464005b70 con 0x7fc460003c20 2024-01-11T08:30:20.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.790+0000 7fc478bff700 1 -- 172.21.15.27:0/4241628148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc460003c20 msgr2=0x7fc460004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:20.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.790+0000 7fc478bff700 1 --2- 172.21.15.27:0/4241628148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc460003c20 0x7fc460004010 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fc46400c1e0 tx=0x7fc464001d40 comp rx=0 tx=0).stop 2024-01-11T08:30:20.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.790+0000 7fc472d9d700 1 -- 172.21.15.27:0/4241628148 reap_dead start 2024-01-11T08:30:20.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.790+0000 7fc478bff700 1 -- 172.21.15.27:0/4241628148 shutdown_connections 2024-01-11T08:30:20.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.790+0000 7fc478bff700 1 -- 172.21.15.27:0/4241628148 >> 172.21.15.27:0/4241628148 conn(0x7fc47406ed40 msgr2=0x7fc47406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:20.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.790+0000 7fc478bff700 1 -- 172.21.15.27:0/4241628148 shutdown_connections 2024-01-11T08:30:20.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.790+0000 7fc478bff700 1 -- 172.21.15.27:0/4241628148 wait complete. 2024-01-11T08:30:20.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.792+0000 7fc478bff700 1 Processor -- start 2024-01-11T08:30:20.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.792+0000 7fc478bff700 1 -- start start 2024-01-11T08:30:20.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.793+0000 7fc478bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc460003c20 0x7fc4740c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:20.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.793+0000 7fc478bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc47415cae0 0x7fc47414a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:20.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.793+0000 7fc478bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4740c9350 0x7fc47414b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:20.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.793+0000 7fc478bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4740c8ec0 con 0x7fc4740c9350 2024-01-11T08:30:20.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.793+0000 7fc478bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4740c7fd0 con 0x7fc460003c20 2024-01-11T08:30:20.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.793+0000 7fc478bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4740c82b0 con 0x7fc47415cae0 2024-01-11T08:30:20.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.793+0000 7fc472d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4740c9350 0x7fc47414b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:20.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.793+0000 7fc472d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4740c9350 0x7fc47414b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34936/0 (socket says 172.21.15.27:34936) 2024-01-11T08:30:20.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.793+0000 7fc472d9d700 1 -- 172.21.15.27:0/4010275719 learned_addr learned my addr 172.21.15.27:0/4010275719 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:20.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.793+0000 7fc46bfff700 1 --2- 172.21.15.27:0/4010275719 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc47415cae0 0x7fc47414a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:20.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.793+0000 7fc47259c700 1 --2- 172.21.15.27:0/4010275719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc460003c20 0x7fc4740c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:20.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.793+0000 7fc472d9d700 1 -- 172.21.15.27:0/4010275719 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc47415cae0 msgr2=0x7fc47414a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:20.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.793+0000 7fc472d9d700 1 --2- 172.21.15.27:0/4010275719 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc47415cae0 0x7fc47414a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:20.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.793+0000 7fc472d9d700 1 -- 172.21.15.27:0/4010275719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc460003c20 msgr2=0x7fc4740c6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:20.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.793+0000 7fc472d9d700 1 --2- 172.21.15.27:0/4010275719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc460003c20 0x7fc4740c6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:20.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.793+0000 7fc472d9d700 1 -- 172.21.15.27:0/4010275719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4740c8590 con 0x7fc4740c9350 2024-01-11T08:30:20.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.793+0000 7fc47259c700 1 --2- 172.21.15.27:0/4010275719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc460003c20 0x7fc4740c6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:30:20.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.794+0000 7fc472d9d700 1 --2- 172.21.15.27:0/4010275719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4740c9350 0x7fc47414b4f0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fc47415d3b0 tx=0x7fc46c0017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.794+0000 7fc46a7fc700 1 -- 172.21.15.27:0/4010275719 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c006db0 con 0x7fc4740c9350 2024-01-11T08:30:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.794+0000 7fc46a7fc700 1 -- 172.21.15.27:0/4010275719 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46c00c670 con 0x7fc4740c9350 2024-01-11T08:30:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.794+0000 7fc46a7fc700 1 -- 172.21.15.27:0/4010275719 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c00ca30 con 0x7fc4740c9350 2024-01-11T08:30:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.794+0000 7fc478bff700 1 -- 172.21.15.27:0/4010275719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4740c8870 con 0x7fc4740c9350 2024-01-11T08:30:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.794+0000 7fc478bff700 1 -- 172.21.15.27:0/4010275719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47414aea0 con 0x7fc4740c9350 2024-01-11T08:30:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.795+0000 7fc478bff700 1 -- 172.21.15.27:0/4010275719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4740c8d10 con 0x7fc4740c9350 2024-01-11T08:30:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.796+0000 7fc46a7fc700 1 -- 172.21.15.27:0/4010275719 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc46c01d460 con 0x7fc4740c9350 2024-01-11T08:30:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.797+0000 7fc46a7fc700 1 --2- 172.21.15.27:0/4010275719 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc454066f30 0x7fc4540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.797+0000 7fc46a7fc700 1 -- 172.21.15.27:0/4010275719 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fc46c0845f0 con 0x7fc4740c9350 2024-01-11T08:30:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.798+0000 7fc47259c700 1 --2- 172.21.15.27:0/4010275719 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc454066f30 0x7fc4540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:20.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.798+0000 7fc47259c700 1 --2- 172.21.15.27:0/4010275719 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc454066f30 0x7fc4540693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc464007ec0 tx=0x7fc464015000 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:20.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.801+0000 7fc46a7fc700 1 -- 172.21.15.27:0/4010275719 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc46c051780 con 0x7fc4740c9350 2024-01-11T08:30:20.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:20.999+0000 7fc478bff700 1 -- 172.21.15.27:0/4010275719 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc4740b0c90 con 0x7fc454066f30 2024-01-11T08:30:21.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.008+0000 7fc46a7fc700 1 -- 172.21.15.27:0/4010275719 <== mgr.8086 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289644 (secure 0 0 0) 0x7fc4740b0c90 con 0x7fc454066f30 2024-01-11T08:30:21.012 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:30:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.012+0000 7fc478bff700 1 -- 172.21.15.27:0/4010275719 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc454066f30 msgr2=0x7fc4540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.012+0000 7fc478bff700 1 --2- 172.21.15.27:0/4010275719 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc454066f30 0x7fc4540693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc464007ec0 tx=0x7fc464015000 comp rx=0 tx=0).stop 2024-01-11T08:30:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.012+0000 7fc478bff700 1 -- 172.21.15.27:0/4010275719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4740c9350 msgr2=0x7fc47414b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:21.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.012+0000 7fc478bff700 1 --2- 172.21.15.27:0/4010275719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4740c9350 0x7fc47414b4f0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fc47415d3b0 tx=0x7fc46c0017a0 comp rx=0 tx=0).stop 2024-01-11T08:30:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.013+0000 7fc478bff700 1 -- 172.21.15.27:0/4010275719 shutdown_connections 2024-01-11T08:30:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.013+0000 7fc478bff700 1 --2- 172.21.15.27:0/4010275719 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc454066f30 0x7fc4540693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.013+0000 7fc478bff700 1 --2- 172.21.15.27:0/4010275719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4740c9350 0x7fc47414b4f0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.013+0000 7fc478bff700 1 --2- 172.21.15.27:0/4010275719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc460003c20 0x7fc4740c6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.013+0000 7fc478bff700 1 --2- 172.21.15.27:0/4010275719 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc47415cae0 0x7fc47414a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.013+0000 7fc478bff700 1 -- 172.21.15.27:0/4010275719 >> 172.21.15.27:0/4010275719 conn(0x7fc47406ed40 msgr2=0x7fc4740d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.013+0000 7fc478bff700 1 -- 172.21.15.27:0/4010275719 shutdown_connections 2024-01-11T08:30:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.013+0000 7fc478bff700 1 -- 172.21.15.27:0/4010275719 wait complete. 2024-01-11T08:30:21.015 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T08:30:21.030 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-11T08:30:19.234186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1424,"num_read_kb":2954,"num_write":475,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":86,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2459092,"kb_used_data":15252,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752515628,"statfs":{"total":773094113280,"available":770576003072,"internally_reserved":0,"allocated":15618048,"data_stored":7912804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.122993"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.143946+0000","last_change":"2024-01-11T08:30:16.143946+0000","last_active":"2024-01-11T08:30:16.143946+0000","last_peered":"2024-01-11T08:30:16.143946+0000","last_clean":"2024-01-11T08:30:16.143946+0000","last_became_active":"2024-01-11T08:30:16.143546+0000","last_became_peered":"2024-01-11T08:30:16.143546+0000","last_unstale":"2024-01-11T08:30:16.143946+0000","last_undegraded":"2024-01-11T08:30:16.143946+0000","last_fullsized":"2024-01-11T08:30:16.143946+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:10:04.758738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.142248+0000","last_change":"2024-01-11T08:30:16.142248+0000","last_active":"2024-01-11T08:30:16.142248+0000","last_peered":"2024-01-11T08:30:16.142248+0000","last_clean":"2024-01-11T08:30:16.142248+0000","last_became_active":"2024-01-11T08:30:16.141936+0000","last_became_peered":"2024-01-11T08:30:16.141936+0000","last_unstale":"2024-01-11T08:30:16.142248+0000","last_undegraded":"2024-01-11T08:30:16.142248+0000","last_fullsized":"2024-01-11T08:30:16.142248+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:39:59.752162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.150380+0000","last_change":"2024-01-11T08:30:16.150380+0000","last_active":"2024-01-11T08:30:16.150380+0000","last_peered":"2024-01-11T08:30:16.150380+0000","last_clean":"2024-01-11T08:30:16.150380+0000","last_became_active":"2024-01-11T08:30:16.149429+0000","last_became_peered":"2024-01-11T08:30:16.149429+0000","last_unstale":"2024-01-11T08:30:16.150380+0000","last_undegraded":"2024-01-11T08:30:16.150380+0000","last_fullsized":"2024-01-11T08:30:16.150380+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:42:16.910360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.130424+0000","last_change":"2024-01-11T08:30:16.130424+0000","last_active":"2024-01-11T08:30:16.130424+0000","last_peered":"2024-01-11T08:30:16.130424+0000","last_clean":"2024-01-11T08:30:16.130424+0000","last_became_active":"2024-01-11T08:30:16.130147+0000","last_became_peered":"2024-01-11T08:30:16.130147+0000","last_unstale":"2024-01-11T08:30:16.130424+0000","last_undegraded":"2024-01-11T08:30:16.130424+0000","last_fullsized":"2024-01-11T08:30:16.130424+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:52.586601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.146250+0000","last_change":"2024-01-11T08:30:16.146250+0000","last_active":"2024-01-11T08:30:16.146250+0000","last_peered":"2024-01-11T08:30:16.146250+0000","last_clean":"2024-01-11T08:30:16.146250+0000","last_became_active":"2024-01-11T08:30:16.145994+0000","last_became_peered":"2024-01-11T08:30:16.145994+0000","last_unstale":"2024-01-11T08:30:16.146250+0000","last_undegraded":"2024-01-11T08:30:16.146250+0000","last_fullsized":"2024-01-11T08:30:16.146250+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:04:19.928740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.152342+0000","last_change":"2024-01-11T08:30:16.141328+0000","last_active":"2024-01-11T08:30:16.152342+0000","last_peered":"2024-01-11T08:30:16.152342+0000","last_clean":"2024-01-11T08:30:16.152342+0000","last_became_active":"2024-01-11T08:30:16.140856+0000","last_became_peered":"2024-01-11T08:30:16.140856+0000","last_unstale":"2024-01-11T08:30:16.152342+0000","last_undegraded":"2024-01-11T08:30:16.152342+0000","last_fullsized":"2024-01-11T08:30:16.152342+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:37:00.824614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.152761+0000","last_change":"2024-01-11T08:30:16.148890+0000","last_active":"2024-01-11T08:30:16.152761+0000","last_peered":"2024-01-11T08:30:16.152761+0000","last_clean":"2024-01-11T08:30:16.152761+0000","last_became_active":"2024-01-11T08:30:16.148585+0000","last_became_peered":"2024-01-11T08:30:16.148585+0000","last_unstale":"2024-01-11T08:30:16.152761+0000","last_undegraded":"2024-01-11T08:30:16.152761+0000","last_fullsized":"2024-01-11T08:30:16.152761+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:05:09.530419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.143282+0000","last_change":"2024-01-11T08:30:16.143282+0000","last_active":"2024-01-11T08:30:16.143282+0000","last_peered":"2024-01-11T08:30:16.143282+0000","last_clean":"2024-01-11T08:30:16.143282+0000","last_became_active":"2024-01-11T08:30:16.143027+0000","last_became_peered":"2024-01-11T08:30:16.143027+0000","last_unstale":"2024-01-11T08:30:16.143282+0000","last_undegraded":"2024-01-11T08:30:16.143282+0000","last_fullsized":"2024-01-11T08:30:16.143282+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:19:17.746788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.147452+0000","last_change":"2024-01-11T08:30:16.147452+0000","last_active":"2024-01-11T08:30:16.147452+0000","last_peered":"2024-01-11T08:30:16.147452+0000","last_clean":"2024-01-11T08:30:16.147452+0000","last_became_active":"2024-01-11T08:30:16.143243+0000","last_became_peered":"2024-01-11T08:30:16.143243+0000","last_unstale":"2024-01-11T08:30:16.147452+0000","last_undegraded":"2024-01-11T08:30:16.147452+0000","last_fullsized":"2024-01-11T08:30:16.147452+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:56:46.100091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.143040+0000","last_change":"2024-01-11T08:30:16.143040+0000","last_active":"2024-01-11T08:30:16.143040+0000","last_peered":"2024-01-11T08:30:16.143040+0000","last_clean":"2024-01-11T08:30:16.143040+0000","last_became_active":"2024-01-11T08:30:16.142747+0000","last_became_peered":"2024-01-11T08:30:16.142747+0000","last_unstale":"2024-01-11T08:30:16.143040+0000","last_undegraded":"2024-01-11T08:30:16.143040+0000","last_fullsized":"2024-01-11T08:30:16.143040+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:21:45.781511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141412+0000","last_change":"2024-01-11T08:30:16.141412+0000","last_active":"2024-01-11T08:30:16.141412+0000","last_peered":"2024-01-11T08:30:16.141412+0000","last_clean":"2024-01-11T08:30:16.141412+0000","last_became_active":"2024-01-11T08:30:16.141134+0000","last_became_peered":"2024-01-11T08:30:16.141134+0000","last_unstale":"2024-01-11T08:30:16.141412+0000","last_undegraded":"2024-01-11T08:30:16.141412+0000","last_fullsized":"2024-01-11T08:30:16.141412+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:00:59.316133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.142637+0000","last_change":"2024-01-11T08:30:16.142637+0000","last_active":"2024-01-11T08:30:16.142637+0000","last_peered":"2024-01-11T08:30:16.142637+0000","last_clean":"2024-01-11T08:30:16.142637+0000","last_became_active":"2024-01-11T08:30:16.142313+0000","last_became_peered":"2024-01-11T08:30:16.142313+0000","last_unstale":"2024-01-11T08:30:16.142637+0000","last_undegraded":"2024-01-11T08:30:16.142637+0000","last_fullsized":"2024-01-11T08:30:16.142637+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:25:31.845686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.137877+0000","last_change":"2024-01-11T08:30:16.137877+0000","last_active":"2024-01-11T08:30:16.137877+0000","last_peered":"2024-01-11T08:30:16.137877+0000","last_clean":"2024-01-11T08:30:16.137877+0000","last_became_active":"2024-01-11T08:30:16.136442+0000","last_became_peered":"2024-01-11T08:30:16.136442+0000","last_unstale":"2024-01-11T08:30:16.137877+0000","last_undegraded":"2024-01-11T08:30:16.137877+0000","last_fullsized":"2024-01-11T08:30:16.137877+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:04:58.411905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.136312+0000","last_change":"2024-01-11T08:30:16.136312+0000","last_active":"2024-01-11T08:30:16.136312+0000","last_peered":"2024-01-11T08:30:16.136312+0000","last_clean":"2024-01-11T08:30:16.136312+0000","last_became_active":"2024-01-11T08:30:16.136042+0000","last_became_peered":"2024-01-11T08:30:16.136042+0000","last_unstale":"2024-01-11T08:30:16.136312+0000","last_undegraded":"2024-01-11T08:30:16.136312+0000","last_fullsized":"2024-01-11T08:30:16.136312+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:44:05.447717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.143639+0000","last_change":"2024-01-11T08:30:16.143639+0000","last_active":"2024-01-11T08:30:16.143639+0000","last_peered":"2024-01-11T08:30:16.143639+0000","last_clean":"2024-01-11T08:30:16.143639+0000","last_became_active":"2024-01-11T08:30:16.143354+0000","last_became_peered":"2024-01-11T08:30:16.143354+0000","last_unstale":"2024-01-11T08:30:16.143639+0000","last_undegraded":"2024-01-11T08:30:16.143639+0000","last_fullsized":"2024-01-11T08:30:16.143639+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:10:16.641138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.140827+0000","last_change":"2024-01-11T08:30:16.135719+0000","last_active":"2024-01-11T08:30:16.140827+0000","last_peered":"2024-01-11T08:30:16.140827+0000","last_clean":"2024-01-11T08:30:16.140827+0000","last_became_active":"2024-01-11T08:30:16.135461+0000","last_became_peered":"2024-01-11T08:30:16.135461+0000","last_unstale":"2024-01-11T08:30:16.140827+0000","last_undegraded":"2024-01-11T08:30:16.140827+0000","last_fullsized":"2024-01-11T08:30:16.140827+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:33:47.586824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.151159+0000","last_change":"2024-01-11T08:30:16.139205+0000","last_active":"2024-01-11T08:30:16.151159+0000","last_peered":"2024-01-11T08:30:16.151159+0000","last_clean":"2024-01-11T08:30:16.151159+0000","last_became_active":"2024-01-11T08:30:16.138960+0000","last_became_peered":"2024-01-11T08:30:16.138960+0000","last_unstale":"2024-01-11T08:30:16.151159+0000","last_undegraded":"2024-01-11T08:30:16.151159+0000","last_fullsized":"2024-01-11T08:30:16.151159+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:04:19.928740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.144967+0000","last_change":"2024-01-11T08:30:16.139363+0000","last_active":"2024-01-11T08:30:16.144967+0000","last_peered":"2024-01-11T08:30:16.144967+0000","last_clean":"2024-01-11T08:30:16.144967+0000","last_became_active":"2024-01-11T08:30:16.138462+0000","last_became_peered":"2024-01-11T08:30:16.138462+0000","last_unstale":"2024-01-11T08:30:16.144967+0000","last_undegraded":"2024-01-11T08:30:16.144967+0000","last_fullsized":"2024-01-11T08:30:16.144967+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:56:46.100091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.142741+0000","last_change":"2024-01-11T08:30:16.142741+0000","last_active":"2024-01-11T08:30:16.142741+0000","last_peered":"2024-01-11T08:30:16.142741+0000","last_clean":"2024-01-11T08:30:16.142741+0000","last_became_active":"2024-01-11T08:30:16.142463+0000","last_became_peered":"2024-01-11T08:30:16.142463+0000","last_unstale":"2024-01-11T08:30:16.142741+0000","last_undegraded":"2024-01-11T08:30:16.142741+0000","last_fullsized":"2024-01-11T08:30:16.142741+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:59:13.612911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141099+0000","last_change":"2024-01-11T08:30:16.138465+0000","last_active":"2024-01-11T08:30:16.141099+0000","last_peered":"2024-01-11T08:30:16.141099+0000","last_clean":"2024-01-11T08:30:16.141099+0000","last_became_active":"2024-01-11T08:30:16.138200+0000","last_became_peered":"2024-01-11T08:30:16.138200+0000","last_unstale":"2024-01-11T08:30:16.141099+0000","last_undegraded":"2024-01-11T08:30:16.141099+0000","last_fullsized":"2024-01-11T08:30:16.141099+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:31:57.002018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.145162+0000","last_change":"2024-01-11T08:30:16.145162+0000","last_active":"2024-01-11T08:30:16.145162+0000","last_peered":"2024-01-11T08:30:16.145162+0000","last_clean":"2024-01-11T08:30:16.145162+0000","last_became_active":"2024-01-11T08:30:16.144960+0000","last_became_peered":"2024-01-11T08:30:16.144960+0000","last_unstale":"2024-01-11T08:30:16.145162+0000","last_undegraded":"2024-01-11T08:30:16.145162+0000","last_fullsized":"2024-01-11T08:30:16.145162+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:05:09.530419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.143068+0000","last_change":"2024-01-11T08:30:16.143068+0000","last_active":"2024-01-11T08:30:16.143068+0000","last_peered":"2024-01-11T08:30:16.143068+0000","last_clean":"2024-01-11T08:30:16.143068+0000","last_became_active":"2024-01-11T08:30:16.142095+0000","last_became_peered":"2024-01-11T08:30:16.142095+0000","last_unstale":"2024-01-11T08:30:16.143068+0000","last_undegraded":"2024-01-11T08:30:16.143068+0000","last_fullsized":"2024-01-11T08:30:16.143068+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:32:40.713675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141584+0000","last_change":"2024-01-11T08:30:16.141584+0000","last_active":"2024-01-11T08:30:16.141584+0000","last_peered":"2024-01-11T08:30:16.141584+0000","last_clean":"2024-01-11T08:30:16.141584+0000","last_became_active":"2024-01-11T08:30:16.141284+0000","last_became_peered":"2024-01-11T08:30:16.141284+0000","last_unstale":"2024-01-11T08:30:16.141584+0000","last_undegraded":"2024-01-11T08:30:16.141584+0000","last_fullsized":"2024-01-11T08:30:16.141584+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:39:48.244351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141234+0000","last_change":"2024-01-11T08:30:16.141234+0000","last_active":"2024-01-11T08:30:16.141234+0000","last_peered":"2024-01-11T08:30:16.141234+0000","last_clean":"2024-01-11T08:30:16.141234+0000","last_became_active":"2024-01-11T08:30:16.140926+0000","last_became_peered":"2024-01-11T08:30:16.140926+0000","last_unstale":"2024-01-11T08:30:16.141234+0000","last_undegraded":"2024-01-11T08:30:16.141234+0000","last_fullsized":"2024-01-11T08:30:16.141234+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:11:27.663936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.144423+0000","last_change":"2024-01-11T08:30:16.144423+0000","last_active":"2024-01-11T08:30:16.144423+0000","last_peered":"2024-01-11T08:30:16.144423+0000","last_clean":"2024-01-11T08:30:16.144423+0000","last_became_active":"2024-01-11T08:30:16.144182+0000","last_became_peered":"2024-01-11T08:30:16.144182+0000","last_unstale":"2024-01-11T08:30:16.144423+0000","last_undegraded":"2024-01-11T08:30:16.144423+0000","last_fullsized":"2024-01-11T08:30:16.144423+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:10:29.829592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141123+0000","last_change":"2024-01-11T08:30:16.136708+0000","last_active":"2024-01-11T08:30:16.141123+0000","last_peered":"2024-01-11T08:30:16.141123+0000","last_clean":"2024-01-11T08:30:16.141123+0000","last_became_active":"2024-01-11T08:30:16.136420+0000","last_became_peered":"2024-01-11T08:30:16.136420+0000","last_unstale":"2024-01-11T08:30:16.141123+0000","last_undegraded":"2024-01-11T08:30:16.141123+0000","last_fullsized":"2024-01-11T08:30:16.141123+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:42:16.910360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.137170+0000","last_change":"2024-01-11T08:30:16.137170+0000","last_active":"2024-01-11T08:30:16.137170+0000","last_peered":"2024-01-11T08:30:16.137170+0000","last_clean":"2024-01-11T08:30:16.137170+0000","last_became_active":"2024-01-11T08:30:16.136878+0000","last_became_peered":"2024-01-11T08:30:16.136878+0000","last_unstale":"2024-01-11T08:30:16.137170+0000","last_undegraded":"2024-01-11T08:30:16.137170+0000","last_fullsized":"2024-01-11T08:30:16.137170+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:43:51.764365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.131623+0000","last_change":"2024-01-11T08:30:16.128144+0000","last_active":"2024-01-11T08:30:16.131623+0000","last_peered":"2024-01-11T08:30:16.131623+0000","last_clean":"2024-01-11T08:30:16.131623+0000","last_became_active":"2024-01-11T08:30:16.127859+0000","last_became_peered":"2024-01-11T08:30:16.127859+0000","last_unstale":"2024-01-11T08:30:16.131623+0000","last_undegraded":"2024-01-11T08:30:16.131623+0000","last_fullsized":"2024-01-11T08:30:16.131623+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:39:34.480780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.143046+0000","last_change":"2024-01-11T08:30:16.143046+0000","last_active":"2024-01-11T08:30:16.143046+0000","last_peered":"2024-01-11T08:30:16.143046+0000","last_clean":"2024-01-11T08:30:16.143046+0000","last_became_active":"2024-01-11T08:30:16.141984+0000","last_became_peered":"2024-01-11T08:30:16.141984+0000","last_unstale":"2024-01-11T08:30:16.143046+0000","last_undegraded":"2024-01-11T08:30:16.143046+0000","last_fullsized":"2024-01-11T08:30:16.143046+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:16:34.884314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.145372+0000","last_change":"2024-01-11T08:30:16.145372+0000","last_active":"2024-01-11T08:30:16.145372+0000","last_peered":"2024-01-11T08:30:16.145372+0000","last_clean":"2024-01-11T08:30:16.145372+0000","last_became_active":"2024-01-11T08:30:16.145083+0000","last_became_peered":"2024-01-11T08:30:16.145083+0000","last_unstale":"2024-01-11T08:30:16.145372+0000","last_undegraded":"2024-01-11T08:30:16.145372+0000","last_fullsized":"2024-01-11T08:30:16.145372+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:07:50.377957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.140588+0000","last_change":"2024-01-11T08:30:16.140588+0000","last_active":"2024-01-11T08:30:16.140588+0000","last_peered":"2024-01-11T08:30:16.140588+0000","last_clean":"2024-01-11T08:30:16.140588+0000","last_became_active":"2024-01-11T08:30:16.140299+0000","last_became_peered":"2024-01-11T08:30:16.140299+0000","last_unstale":"2024-01-11T08:30:16.140588+0000","last_undegraded":"2024-01-11T08:30:16.140588+0000","last_fullsized":"2024-01-11T08:30:16.140588+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:42:53.996924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.152335+0000","last_change":"2024-01-11T08:30:16.141342+0000","last_active":"2024-01-11T08:30:16.152335+0000","last_peered":"2024-01-11T08:30:16.152335+0000","last_clean":"2024-01-11T08:30:16.152335+0000","last_became_active":"2024-01-11T08:30:16.139754+0000","last_became_peered":"2024-01-11T08:30:16.139754+0000","last_unstale":"2024-01-11T08:30:16.152335+0000","last_undegraded":"2024-01-11T08:30:16.152335+0000","last_fullsized":"2024-01-11T08:30:16.152335+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:46:01.822939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.137111+0000","last_change":"2024-01-11T08:30:16.132398+0000","last_active":"2024-01-11T08:30:16.137111+0000","last_peered":"2024-01-11T08:30:16.137111+0000","last_clean":"2024-01-11T08:30:16.137111+0000","last_became_active":"2024-01-11T08:30:16.132147+0000","last_became_peered":"2024-01-11T08:30:16.132147+0000","last_unstale":"2024-01-11T08:30:16.137111+0000","last_undegraded":"2024-01-11T08:30:16.137111+0000","last_fullsized":"2024-01-11T08:30:16.137111+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:39:48.244351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"144'4","reported_seq":13,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.160702+0000","last_change":"2024-01-11T08:30:15.138028+0000","last_active":"2024-01-11T08:30:16.160702+0000","last_peered":"2024-01-11T08:30:16.160702+0000","last_clean":"2024-01-11T08:30:16.160702+0000","last_became_active":"2024-01-11T08:30:15.137414+0000","last_became_peered":"2024-01-11T08:30:15.137414+0000","last_unstale":"2024-01-11T08:30:16.160702+0000","last_undegraded":"2024-01-11T08:30:16.160702+0000","last_fullsized":"2024-01-11T08:30:16.160702+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:42:06.560411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.147228+0000","last_change":"2024-01-11T08:30:16.147228+0000","last_active":"2024-01-11T08:30:16.147228+0000","last_peered":"2024-01-11T08:30:16.147228+0000","last_clean":"2024-01-11T08:30:16.147228+0000","last_became_active":"2024-01-11T08:30:16.142890+0000","last_became_peered":"2024-01-11T08:30:16.142890+0000","last_unstale":"2024-01-11T08:30:16.147228+0000","last_undegraded":"2024-01-11T08:30:16.147228+0000","last_fullsized":"2024-01-11T08:30:16.147228+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:37:00.824614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.159816+0000","last_change":"2024-01-11T08:30:15.130924+0000","last_active":"2024-01-11T08:30:16.159816+0000","last_peered":"2024-01-11T08:30:16.159816+0000","last_clean":"2024-01-11T08:30:16.159816+0000","last_became_active":"2024-01-11T08:30:15.130220+0000","last_became_peered":"2024-01-11T08:30:15.130220+0000","last_unstale":"2024-01-11T08:30:16.159816+0000","last_undegraded":"2024-01-11T08:30:16.159816+0000","last_fullsized":"2024-01-11T08:30:16.159816+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:39:58.743737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141748+0000","last_change":"2024-01-11T08:30:16.141748+0000","last_active":"2024-01-11T08:30:16.141748+0000","last_peered":"2024-01-11T08:30:16.141748+0000","last_clean":"2024-01-11T08:30:16.141748+0000","last_became_active":"2024-01-11T08:30:16.141458+0000","last_became_peered":"2024-01-11T08:30:16.141458+0000","last_unstale":"2024-01-11T08:30:16.141748+0000","last_undegraded":"2024-01-11T08:30:16.141748+0000","last_fullsized":"2024-01-11T08:30:16.141748+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:41:41.649132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"144'2","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.160444+0000","last_change":"2024-01-11T08:30:15.137966+0000","last_active":"2024-01-11T08:30:16.160444+0000","last_peered":"2024-01-11T08:30:16.160444+0000","last_clean":"2024-01-11T08:30:16.160444+0000","last_became_active":"2024-01-11T08:30:15.137678+0000","last_became_peered":"2024-01-11T08:30:15.137678+0000","last_unstale":"2024-01-11T08:30:16.160444+0000","last_undegraded":"2024-01-11T08:30:16.160444+0000","last_fullsized":"2024-01-11T08:30:16.160444+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:31:27.749785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.146971+0000","last_change":"2024-01-11T08:30:16.146971+0000","last_active":"2024-01-11T08:30:16.146971+0000","last_peered":"2024-01-11T08:30:16.146971+0000","last_clean":"2024-01-11T08:30:16.146971+0000","last_became_active":"2024-01-11T08:30:16.146700+0000","last_became_peered":"2024-01-11T08:30:16.146700+0000","last_unstale":"2024-01-11T08:30:16.146971+0000","last_undegraded":"2024-01-11T08:30:16.146971+0000","last_fullsized":"2024-01-11T08:30:16.146971+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:37:00.824614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"144'2","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.160385+0000","last_change":"2024-01-11T08:30:15.134992+0000","last_active":"2024-01-11T08:30:16.160385+0000","last_peered":"2024-01-11T08:30:16.160385+0000","last_clean":"2024-01-11T08:30:16.160385+0000","last_became_active":"2024-01-11T08:30:15.133331+0000","last_became_peered":"2024-01-11T08:30:15.133331+0000","last_unstale":"2024-01-11T08:30:16.160385+0000","last_undegraded":"2024-01-11T08:30:16.160385+0000","last_fullsized":"2024-01-11T08:30:16.160385+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:04:57.403479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141445+0000","last_change":"2024-01-11T08:30:16.136649+0000","last_active":"2024-01-11T08:30:16.141445+0000","last_peered":"2024-01-11T08:30:16.141445+0000","last_clean":"2024-01-11T08:30:16.141445+0000","last_became_active":"2024-01-11T08:30:16.135558+0000","last_became_peered":"2024-01-11T08:30:16.135558+0000","last_unstale":"2024-01-11T08:30:16.141445+0000","last_undegraded":"2024-01-11T08:30:16.141445+0000","last_fullsized":"2024-01-11T08:30:16.141445+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:37:00.824614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"144'1","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.131533+0000","last_change":"2024-01-11T08:30:15.131790+0000","last_active":"2024-01-11T08:30:16.131533+0000","last_peered":"2024-01-11T08:30:16.131533+0000","last_clean":"2024-01-11T08:30:16.131533+0000","last_became_active":"2024-01-11T08:30:15.131319+0000","last_became_peered":"2024-01-11T08:30:15.131319+0000","last_unstale":"2024-01-11T08:30:16.131533+0000","last_undegraded":"2024-01-11T08:30:16.131533+0000","last_fullsized":"2024-01-11T08:30:16.131533+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:40:49.059728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.150741+0000","last_change":"2024-01-11T08:30:16.138446+0000","last_active":"2024-01-11T08:30:16.150741+0000","last_peered":"2024-01-11T08:30:16.150741+0000","last_clean":"2024-01-11T08:30:16.150741+0000","last_became_active":"2024-01-11T08:30:16.138018+0000","last_became_peered":"2024-01-11T08:30:16.138018+0000","last_unstale":"2024-01-11T08:30:16.150741+0000","last_undegraded":"2024-01-11T08:30:16.150741+0000","last_fullsized":"2024-01-11T08:30:16.150741+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:41:41.649132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.162104+0000","last_change":"2024-01-11T08:30:15.123357+0000","last_active":"2024-01-11T08:30:16.162104+0000","last_peered":"2024-01-11T08:30:16.162104+0000","last_clean":"2024-01-11T08:30:16.162104+0000","last_became_active":"2024-01-11T08:30:15.123031+0000","last_became_peered":"2024-01-11T08:30:15.123031+0000","last_unstale":"2024-01-11T08:30:16.162104+0000","last_undegraded":"2024-01-11T08:30:16.162104+0000","last_fullsized":"2024-01-11T08:30:16.162104+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:42:18.913764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.140121+0000","last_change":"2024-01-11T08:30:16.140121+0000","last_active":"2024-01-11T08:30:16.140121+0000","last_peered":"2024-01-11T08:30:16.140121+0000","last_clean":"2024-01-11T08:30:16.140121+0000","last_became_active":"2024-01-11T08:30:16.139438+0000","last_became_peered":"2024-01-11T08:30:16.139438+0000","last_unstale":"2024-01-11T08:30:16.140121+0000","last_undegraded":"2024-01-11T08:30:16.140121+0000","last_fullsized":"2024-01-11T08:30:16.140121+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:18:01.054004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.131265+0000","last_change":"2024-01-11T08:30:15.133630+0000","last_active":"2024-01-11T08:30:16.131265+0000","last_peered":"2024-01-11T08:30:16.131265+0000","last_clean":"2024-01-11T08:30:16.131265+0000","last_became_active":"2024-01-11T08:30:15.133115+0000","last_became_peered":"2024-01-11T08:30:15.133115+0000","last_unstale":"2024-01-11T08:30:16.131265+0000","last_undegraded":"2024-01-11T08:30:16.131265+0000","last_fullsized":"2024-01-11T08:30:16.131265+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:13:48.277660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.138510+0000","last_change":"2024-01-11T08:30:16.138510+0000","last_active":"2024-01-11T08:30:16.138510+0000","last_peered":"2024-01-11T08:30:16.138510+0000","last_clean":"2024-01-11T08:30:16.138510+0000","last_became_active":"2024-01-11T08:30:16.138256+0000","last_became_peered":"2024-01-11T08:30:16.138256+0000","last_unstale":"2024-01-11T08:30:16.138510+0000","last_undegraded":"2024-01-11T08:30:16.138510+0000","last_fullsized":"2024-01-11T08:30:16.138510+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:47:19.039951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.136547+0000","last_change":"2024-01-11T08:30:15.132789+0000","last_active":"2024-01-11T08:30:15.136547+0000","last_peered":"2024-01-11T08:30:15.136547+0000","last_clean":"2024-01-11T08:30:15.136547+0000","last_became_active":"2024-01-11T08:30:15.132550+0000","last_became_peered":"2024-01-11T08:30:15.132550+0000","last_unstale":"2024-01-11T08:30:15.136547+0000","last_undegraded":"2024-01-11T08:30:15.136547+0000","last_fullsized":"2024-01-11T08:30:15.136547+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:43:50.755940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141745+0000","last_change":"2024-01-11T08:30:16.141745+0000","last_active":"2024-01-11T08:30:16.141745+0000","last_peered":"2024-01-11T08:30:16.141745+0000","last_clean":"2024-01-11T08:30:16.141745+0000","last_became_active":"2024-01-11T08:30:16.141487+0000","last_became_peered":"2024-01-11T08:30:16.141487+0000","last_unstale":"2024-01-11T08:30:16.141745+0000","last_undegraded":"2024-01-11T08:30:16.141745+0000","last_fullsized":"2024-01-11T08:30:16.141745+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:38:08.594999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.124542+0000","last_change":"2024-01-11T08:30:15.132780+0000","last_active":"2024-01-11T08:30:16.124542+0000","last_peered":"2024-01-11T08:30:16.124542+0000","last_clean":"2024-01-11T08:30:16.124542+0000","last_became_active":"2024-01-11T08:30:15.132135+0000","last_became_peered":"2024-01-11T08:30:15.132135+0000","last_unstale":"2024-01-11T08:30:16.124542+0000","last_undegraded":"2024-01-11T08:30:16.124542+0000","last_fullsized":"2024-01-11T08:30:16.124542+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:10:28.821166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.135703+0000","last_change":"2024-01-11T08:30:16.129747+0000","last_active":"2024-01-11T08:30:16.135703+0000","last_peered":"2024-01-11T08:30:16.135703+0000","last_clean":"2024-01-11T08:30:16.135703+0000","last_became_active":"2024-01-11T08:30:16.128576+0000","last_became_peered":"2024-01-11T08:30:16.128576+0000","last_unstale":"2024-01-11T08:30:16.135703+0000","last_undegraded":"2024-01-11T08:30:16.135703+0000","last_fullsized":"2024-01-11T08:30:16.135703+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:50:55.636631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.127216+0000","last_change":"2024-01-11T08:30:15.138007+0000","last_active":"2024-01-11T08:30:16.127216+0000","last_peered":"2024-01-11T08:30:16.127216+0000","last_clean":"2024-01-11T08:30:16.127216+0000","last_became_active":"2024-01-11T08:30:15.137149+0000","last_became_peered":"2024-01-11T08:30:15.137149+0000","last_unstale":"2024-01-11T08:30:16.127216+0000","last_undegraded":"2024-01-11T08:30:16.127216+0000","last_fullsized":"2024-01-11T08:30:16.127216+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:22:08.567433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.144564+0000","last_change":"2024-01-11T08:30:16.144564+0000","last_active":"2024-01-11T08:30:16.144564+0000","last_peered":"2024-01-11T08:30:16.144564+0000","last_clean":"2024-01-11T08:30:16.144564+0000","last_became_active":"2024-01-11T08:30:16.144153+0000","last_became_peered":"2024-01-11T08:30:16.144153+0000","last_unstale":"2024-01-11T08:30:16.144564+0000","last_undegraded":"2024-01-11T08:30:16.144564+0000","last_fullsized":"2024-01-11T08:30:16.144564+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:55:31.991780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"144'3","reported_seq":13,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.162585+0000","last_change":"2024-01-11T08:30:15.125204+0000","last_active":"2024-01-11T08:30:16.162585+0000","last_peered":"2024-01-11T08:30:16.162585+0000","last_clean":"2024-01-11T08:30:16.162585+0000","last_became_active":"2024-01-11T08:30:15.124648+0000","last_became_peered":"2024-01-11T08:30:15.124648+0000","last_unstale":"2024-01-11T08:30:16.162585+0000","last_undegraded":"2024-01-11T08:30:16.162585+0000","last_fullsized":"2024-01-11T08:30:16.162585+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:54:29.952959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.147354+0000","last_change":"2024-01-11T08:30:16.147354+0000","last_active":"2024-01-11T08:30:16.147354+0000","last_peered":"2024-01-11T08:30:16.147354+0000","last_clean":"2024-01-11T08:30:16.147354+0000","last_became_active":"2024-01-11T08:30:16.147065+0000","last_became_peered":"2024-01-11T08:30:16.147065+0000","last_unstale":"2024-01-11T08:30:16.147354+0000","last_undegraded":"2024-01-11T08:30:16.147354+0000","last_fullsized":"2024-01-11T08:30:16.147354+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:27:29.470280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"144'6","reported_seq":15,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.162601+0000","last_change":"2024-01-11T08:30:15.134571+0000","last_active":"2024-01-11T08:30:16.162601+0000","last_peered":"2024-01-11T08:30:16.162601+0000","last_clean":"2024-01-11T08:30:16.162601+0000","last_became_active":"2024-01-11T08:30:15.133172+0000","last_became_peered":"2024-01-11T08:30:15.133172+0000","last_unstale":"2024-01-11T08:30:16.162601+0000","last_undegraded":"2024-01-11T08:30:16.162601+0000","last_fullsized":"2024-01-11T08:30:16.162601+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:04:31.287522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.138480+0000","last_change":"2024-01-11T08:30:16.138480+0000","last_active":"2024-01-11T08:30:16.138480+0000","last_peered":"2024-01-11T08:30:16.138480+0000","last_clean":"2024-01-11T08:30:16.138480+0000","last_became_active":"2024-01-11T08:30:16.138200+0000","last_became_peered":"2024-01-11T08:30:16.138200+0000","last_unstale":"2024-01-11T08:30:16.138480+0000","last_undegraded":"2024-01-11T08:30:16.138480+0000","last_fullsized":"2024-01-11T08:30:16.138480+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:40:50.068154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.140076+0000","last_change":"2024-01-11T08:30:15.135258+0000","last_active":"2024-01-11T08:30:15.140076+0000","last_peered":"2024-01-11T08:30:15.140076+0000","last_clean":"2024-01-11T08:30:15.140076+0000","last_became_active":"2024-01-11T08:30:15.133480+0000","last_became_peered":"2024-01-11T08:30:15.133480+0000","last_unstale":"2024-01-11T08:30:15.140076+0000","last_undegraded":"2024-01-11T08:30:15.140076+0000","last_fullsized":"2024-01-11T08:30:15.140076+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:27:02.286647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.145951+0000","last_change":"2024-01-11T08:30:16.145951+0000","last_active":"2024-01-11T08:30:16.145951+0000","last_peered":"2024-01-11T08:30:16.145951+0000","last_clean":"2024-01-11T08:30:16.145951+0000","last_became_active":"2024-01-11T08:30:16.145671+0000","last_became_peered":"2024-01-11T08:30:16.145671+0000","last_unstale":"2024-01-11T08:30:16.145951+0000","last_undegraded":"2024-01-11T08:30:16.145951+0000","last_fullsized":"2024-01-11T08:30:16.145951+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:58:23.810853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.130899+0000","last_change":"2024-01-11T08:30:15.125107+0000","last_active":"2024-01-11T08:30:15.130899+0000","last_peered":"2024-01-11T08:30:15.130899+0000","last_clean":"2024-01-11T08:30:15.130899+0000","last_became_active":"2024-01-11T08:30:15.124715+0000","last_became_peered":"2024-01-11T08:30:15.124715+0000","last_unstale":"2024-01-11T08:30:15.130899+0000","last_undegraded":"2024-01-11T08:30:15.130899+0000","last_fullsized":"2024-01-11T08:30:15.130899+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:10:15.632713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141647+0000","last_change":"2024-01-11T08:30:16.141647+0000","last_active":"2024-01-11T08:30:16.141647+0000","last_peered":"2024-01-11T08:30:16.141647+0000","last_clean":"2024-01-11T08:30:16.141647+0000","last_became_active":"2024-01-11T08:30:16.141334+0000","last_became_peered":"2024-01-11T08:30:16.141334+0000","last_unstale":"2024-01-11T08:30:16.141647+0000","last_undegraded":"2024-01-11T08:30:16.141647+0000","last_fullsized":"2024-01-11T08:30:16.141647+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:13:49.286086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"144'8","reported_seq":17,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.164350+0000","last_change":"2024-01-11T08:30:15.131144+0000","last_active":"2024-01-11T08:30:16.164350+0000","last_peered":"2024-01-11T08:30:16.164350+0000","last_clean":"2024-01-11T08:30:16.164350+0000","last_became_active":"2024-01-11T08:30:15.130831+0000","last_became_peered":"2024-01-11T08:30:15.130831+0000","last_unstale":"2024-01-11T08:30:16.164350+0000","last_undegraded":"2024-01-11T08:30:16.164350+0000","last_fullsized":"2024-01-11T08:30:16.164350+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:42:18.913764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.147131+0000","last_change":"2024-01-11T08:30:16.147131+0000","last_active":"2024-01-11T08:30:16.147131+0000","last_peered":"2024-01-11T08:30:16.147131+0000","last_clean":"2024-01-11T08:30:16.147131+0000","last_became_active":"2024-01-11T08:30:16.146876+0000","last_became_peered":"2024-01-11T08:30:16.146876+0000","last_unstale":"2024-01-11T08:30:16.147131+0000","last_undegraded":"2024-01-11T08:30:16.147131+0000","last_fullsized":"2024-01-11T08:30:16.147131+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:31:57.002018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.162255+0000","last_change":"2024-01-11T08:30:15.130796+0000","last_active":"2024-01-11T08:30:16.162255+0000","last_peered":"2024-01-11T08:30:16.162255+0000","last_clean":"2024-01-11T08:30:16.162255+0000","last_became_active":"2024-01-11T08:30:15.130180+0000","last_became_peered":"2024-01-11T08:30:15.130180+0000","last_unstale":"2024-01-11T08:30:16.162255+0000","last_undegraded":"2024-01-11T08:30:16.162255+0000","last_fullsized":"2024-01-11T08:30:16.162255+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:42.449905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.144809+0000","last_change":"2024-01-11T08:30:16.142801+0000","last_active":"2024-01-11T08:30:16.144809+0000","last_peered":"2024-01-11T08:30:16.144809+0000","last_clean":"2024-01-11T08:30:16.144809+0000","last_became_active":"2024-01-11T08:30:16.142452+0000","last_became_peered":"2024-01-11T08:30:16.142452+0000","last_unstale":"2024-01-11T08:30:16.144809+0000","last_undegraded":"2024-01-11T08:30:16.144809+0000","last_fullsized":"2024-01-11T08:30:16.144809+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:02:12.583528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"144'2","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.161523+0000","last_change":"2024-01-11T08:30:15.135821+0000","last_active":"2024-01-11T08:30:16.161523+0000","last_peered":"2024-01-11T08:30:16.161523+0000","last_clean":"2024-01-11T08:30:16.161523+0000","last_became_active":"2024-01-11T08:30:15.135463+0000","last_became_peered":"2024-01-11T08:30:15.135463+0000","last_unstale":"2024-01-11T08:30:16.161523+0000","last_undegraded":"2024-01-11T08:30:16.161523+0000","last_fullsized":"2024-01-11T08:30:16.161523+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:09:47.457657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.141080+0000","last_change":"2024-01-11T08:30:16.136859+0000","last_active":"2024-01-11T08:30:16.141080+0000","last_peered":"2024-01-11T08:30:16.141080+0000","last_clean":"2024-01-11T08:30:16.141080+0000","last_became_active":"2024-01-11T08:30:16.136570+0000","last_became_peered":"2024-01-11T08:30:16.136570+0000","last_unstale":"2024-01-11T08:30:16.141080+0000","last_undegraded":"2024-01-11T08:30:16.141080+0000","last_fullsized":"2024-01-11T08:30:16.141080+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:18:01.054004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"144'1","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.161771+0000","last_change":"2024-01-11T08:30:15.135328+0000","last_active":"2024-01-11T08:30:16.161771+0000","last_peered":"2024-01-11T08:30:16.161771+0000","last_clean":"2024-01-11T08:30:16.161771+0000","last_became_active":"2024-01-11T08:30:15.135078+0000","last_became_peered":"2024-01-11T08:30:15.135078+0000","last_unstale":"2024-01-11T08:30:16.161771+0000","last_undegraded":"2024-01-11T08:30:16.161771+0000","last_fullsized":"2024-01-11T08:30:16.161771+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:25:30.837261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.139966+0000","last_change":"2024-01-11T08:30:16.139966+0000","last_active":"2024-01-11T08:30:16.139966+0000","last_peered":"2024-01-11T08:30:16.139966+0000","last_clean":"2024-01-11T08:30:16.139966+0000","last_became_active":"2024-01-11T08:30:16.138694+0000","last_became_peered":"2024-01-11T08:30:16.138694+0000","last_unstale":"2024-01-11T08:30:16.139966+0000","last_undegraded":"2024-01-11T08:30:16.139966+0000","last_fullsized":"2024-01-11T08:30:16.139966+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:48.322813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.124300+0000","last_change":"2024-01-11T08:30:15.136191+0000","last_active":"2024-01-11T08:30:16.124300+0000","last_peered":"2024-01-11T08:30:16.124300+0000","last_clean":"2024-01-11T08:30:16.124300+0000","last_became_active":"2024-01-11T08:30:15.135840+0000","last_became_peered":"2024-01-11T08:30:15.135840+0000","last_unstale":"2024-01-11T08:30:16.124300+0000","last_undegraded":"2024-01-11T08:30:16.124300+0000","last_fullsized":"2024-01-11T08:30:16.124300+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:21:44.773086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.137233+0000","last_change":"2024-01-11T08:30:16.137233+0000","last_active":"2024-01-11T08:30:16.137233+0000","last_peered":"2024-01-11T08:30:16.137233+0000","last_clean":"2024-01-11T08:30:16.137233+0000","last_became_active":"2024-01-11T08:30:16.136671+0000","last_became_peered":"2024-01-11T08:30:16.136671+0000","last_unstale":"2024-01-11T08:30:16.137233+0000","last_undegraded":"2024-01-11T08:30:16.137233+0000","last_fullsized":"2024-01-11T08:30:16.137233+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:41:24.657291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"144'2","reported_seq":12,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.160091+0000","last_change":"2024-01-11T08:30:15.132771+0000","last_active":"2024-01-11T08:30:16.160091+0000","last_peered":"2024-01-11T08:30:16.160091+0000","last_clean":"2024-01-11T08:30:16.160091+0000","last_became_active":"2024-01-11T08:30:15.132014+0000","last_became_peered":"2024-01-11T08:30:15.132014+0000","last_unstale":"2024-01-11T08:30:16.160091+0000","last_undegraded":"2024-01-11T08:30:16.160091+0000","last_fullsized":"2024-01-11T08:30:16.160091+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:22:22.854310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.152813+0000","last_change":"2024-01-11T08:30:16.149298+0000","last_active":"2024-01-11T08:30:16.152813+0000","last_peered":"2024-01-11T08:30:16.152813+0000","last_clean":"2024-01-11T08:30:16.152813+0000","last_became_active":"2024-01-11T08:30:16.148629+0000","last_became_peered":"2024-01-11T08:30:16.148629+0000","last_unstale":"2024-01-11T08:30:16.152813+0000","last_undegraded":"2024-01-11T08:30:16.152813+0000","last_fullsized":"2024-01-11T08:30:16.152813+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:33:47.586824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.126525+0000","last_change":"2024-01-11T08:30:15.130187+0000","last_active":"2024-01-11T08:30:16.126525+0000","last_peered":"2024-01-11T08:30:16.126525+0000","last_clean":"2024-01-11T08:30:16.126525+0000","last_became_active":"2024-01-11T08:30:15.129793+0000","last_became_peered":"2024-01-11T08:30:15.129793+0000","last_unstale":"2024-01-11T08:30:16.126525+0000","last_undegraded":"2024-01-11T08:30:16.126525+0000","last_fullsized":"2024-01-11T08:30:16.126525+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:04:31.287522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.137861+0000","last_change":"2024-01-11T08:30:16.137861+0000","last_active":"2024-01-11T08:30:16.137861+0000","last_peered":"2024-01-11T08:30:16.137861+0000","last_clean":"2024-01-11T08:30:16.137861+0000","last_became_active":"2024-01-11T08:30:16.137534+0000","last_became_peered":"2024-01-11T08:30:16.137534+0000","last_unstale":"2024-01-11T08:30:16.137861+0000","last_undegraded":"2024-01-11T08:30:16.137861+0000","last_fullsized":"2024-01-11T08:30:16.137861+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:00:47.446985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.125438+0000","last_change":"2024-01-11T08:30:15.126662+0000","last_active":"2024-01-11T08:30:16.125438+0000","last_peered":"2024-01-11T08:30:16.125438+0000","last_clean":"2024-01-11T08:30:16.125438+0000","last_became_active":"2024-01-11T08:30:15.125810+0000","last_became_peered":"2024-01-11T08:30:15.125810+0000","last_unstale":"2024-01-11T08:30:16.125438+0000","last_undegraded":"2024-01-11T08:30:16.125438+0000","last_fullsized":"2024-01-11T08:30:16.125438+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T08:42:06.560411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.145680+0000","last_change":"2024-01-11T08:30:16.145680+0000","last_active":"2024-01-11T08:30:16.145680+0000","last_peered":"2024-01-11T08:30:16.145680+0000","last_clean":"2024-01-11T08:30:16.145680+0000","last_became_active":"2024-01-11T08:30:16.145384+0000","last_became_peered":"2024-01-11T08:30:16.145384+0000","last_unstale":"2024-01-11T08:30:16.145680+0000","last_undegraded":"2024-01-11T08:30:16.145680+0000","last_fullsized":"2024-01-11T08:30:16.145680+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:46:01.822939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.139787+0000","last_change":"2024-01-11T08:30:15.137127+0000","last_active":"2024-01-11T08:30:15.139787+0000","last_peered":"2024-01-11T08:30:15.139787+0000","last_clean":"2024-01-11T08:30:15.139787+0000","last_became_active":"2024-01-11T08:30:15.136803+0000","last_became_peered":"2024-01-11T08:30:15.136803+0000","last_unstale":"2024-01-11T08:30:15.139787+0000","last_undegraded":"2024-01-11T08:30:15.139787+0000","last_fullsized":"2024-01-11T08:30:15.139787+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:29:02.643186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.152478+0000","last_change":"2024-01-11T08:30:16.141789+0000","last_active":"2024-01-11T08:30:16.152478+0000","last_peered":"2024-01-11T08:30:16.152478+0000","last_clean":"2024-01-11T08:30:16.152478+0000","last_became_active":"2024-01-11T08:30:16.141540+0000","last_became_peered":"2024-01-11T08:30:16.141540+0000","last_unstale":"2024-01-11T08:30:16.152478+0000","last_undegraded":"2024-01-11T08:30:16.152478+0000","last_fullsized":"2024-01-11T08:30:16.152478+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:22:13.465322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"144'2","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.160977+0000","last_change":"2024-01-11T08:30:15.134093+0000","last_active":"2024-01-11T08:30:16.160977+0000","last_peered":"2024-01-11T08:30:16.160977+0000","last_clean":"2024-01-11T08:30:16.160977+0000","last_became_active":"2024-01-11T08:30:15.133551+0000","last_became_peered":"2024-01-11T08:30:15.133551+0000","last_unstale":"2024-01-11T08:30:16.160977+0000","last_undegraded":"2024-01-11T08:30:16.160977+0000","last_fullsized":"2024-01-11T08:30:16.160977+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:48:51.323814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.144503+0000","last_change":"2024-01-11T08:30:16.144503+0000","last_active":"2024-01-11T08:30:16.144503+0000","last_peered":"2024-01-11T08:30:16.144503+0000","last_clean":"2024-01-11T08:30:16.144503+0000","last_became_active":"2024-01-11T08:30:16.144222+0000","last_became_peered":"2024-01-11T08:30:16.144222+0000","last_unstale":"2024-01-11T08:30:16.144503+0000","last_undegraded":"2024-01-11T08:30:16.144503+0000","last_fullsized":"2024-01-11T08:30:16.144503+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:10:22.443710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.136936+0000","last_change":"2024-01-11T08:30:15.135166+0000","last_active":"2024-01-11T08:30:15.136936+0000","last_peered":"2024-01-11T08:30:15.136936+0000","last_clean":"2024-01-11T08:30:15.136936+0000","last_became_active":"2024-01-11T08:30:15.134222+0000","last_became_peered":"2024-01-11T08:30:15.134222+0000","last_unstale":"2024-01-11T08:30:15.136936+0000","last_undegraded":"2024-01-11T08:30:15.136936+0000","last_fullsized":"2024-01-11T08:30:15.136936+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:16:49.867287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.147632+0000","last_change":"2024-01-11T08:30:16.147632+0000","last_active":"2024-01-11T08:30:16.147632+0000","last_peered":"2024-01-11T08:30:16.147632+0000","last_clean":"2024-01-11T08:30:16.147632+0000","last_became_active":"2024-01-11T08:30:16.147389+0000","last_became_peered":"2024-01-11T08:30:16.147389+0000","last_unstale":"2024-01-11T08:30:16.147632+0000","last_undegraded":"2024-01-11T08:30:16.147632+0000","last_fullsized":"2024-01-11T08:30:16.147632+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:11:56.999227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.138475+0000","last_change":"2024-01-11T08:30:15.135652+0000","last_active":"2024-01-11T08:30:15.138475+0000","last_peered":"2024-01-11T08:30:15.138475+0000","last_clean":"2024-01-11T08:30:15.138475+0000","last_became_active":"2024-01-11T08:30:15.135391+0000","last_became_peered":"2024-01-11T08:30:15.135391+0000","last_unstale":"2024-01-11T08:30:15.138475+0000","last_undegraded":"2024-01-11T08:30:15.138475+0000","last_fullsized":"2024-01-11T08:30:15.138475+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:10:03.750312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.150794+0000","last_change":"2024-01-11T08:30:16.137616+0000","last_active":"2024-01-11T08:30:16.150794+0000","last_peered":"2024-01-11T08:30:16.150794+0000","last_clean":"2024-01-11T08:30:16.150794+0000","last_became_active":"2024-01-11T08:30:16.137349+0000","last_became_peered":"2024-01-11T08:30:16.137349+0000","last_unstale":"2024-01-11T08:30:16.150794+0000","last_undegraded":"2024-01-11T08:30:16.150794+0000","last_fullsized":"2024-01-11T08:30:16.150794+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:55:31.991780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.139332+0000","last_change":"2024-01-11T08:30:15.135331+0000","last_active":"2024-01-11T08:30:15.139332+0000","last_peered":"2024-01-11T08:30:15.139332+0000","last_clean":"2024-01-11T08:30:15.139332+0000","last_became_active":"2024-01-11T08:30:15.134851+0000","last_became_peered":"2024-01-11T08:30:15.134851+0000","last_unstale":"2024-01-11T08:30:15.139332+0000","last_undegraded":"2024-01-11T08:30:15.139332+0000","last_fullsized":"2024-01-11T08:30:15.139332+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:33:57.183661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.146652+0000","last_change":"2024-01-11T08:30:16.146652+0000","last_active":"2024-01-11T08:30:16.146652+0000","last_peered":"2024-01-11T08:30:16.146652+0000","last_clean":"2024-01-11T08:30:16.146652+0000","last_became_active":"2024-01-11T08:30:16.144158+0000","last_became_peered":"2024-01-11T08:30:16.144158+0000","last_unstale":"2024-01-11T08:30:16.146652+0000","last_undegraded":"2024-01-11T08:30:16.146652+0000","last_fullsized":"2024-01-11T08:30:16.146652+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:33:47.586824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.136538+0000","last_change":"2024-01-11T08:30:15.130938+0000","last_active":"2024-01-11T08:30:15.136538+0000","last_peered":"2024-01-11T08:30:15.136538+0000","last_clean":"2024-01-11T08:30:15.136538+0000","last_became_active":"2024-01-11T08:30:15.130636+0000","last_became_peered":"2024-01-11T08:30:15.130636+0000","last_unstale":"2024-01-11T08:30:15.136538+0000","last_undegraded":"2024-01-11T08:30:15.136538+0000","last_fullsized":"2024-01-11T08:30:15.136538+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:07:49.369531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.135741+0000","last_change":"2024-01-11T08:30:16.131802+0000","last_active":"2024-01-11T08:30:16.135741+0000","last_peered":"2024-01-11T08:30:16.135741+0000","last_clean":"2024-01-11T08:30:16.135741+0000","last_became_active":"2024-01-11T08:30:16.131536+0000","last_became_peered":"2024-01-11T08:30:16.131536+0000","last_unstale":"2024-01-11T08:30:16.135741+0000","last_undegraded":"2024-01-11T08:30:16.135741+0000","last_fullsized":"2024-01-11T08:30:16.135741+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:42:19.922189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.139400+0000","last_change":"2024-01-11T08:30:15.137866+0000","last_active":"2024-01-11T08:30:15.139400+0000","last_peered":"2024-01-11T08:30:15.139400+0000","last_clean":"2024-01-11T08:30:15.139400+0000","last_became_active":"2024-01-11T08:30:15.137562+0000","last_became_peered":"2024-01-11T08:30:15.137562+0000","last_unstale":"2024-01-11T08:30:15.139400+0000","last_undegraded":"2024-01-11T08:30:15.139400+0000","last_fullsized":"2024-01-11T08:30:15.139400+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:32:43.530518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.146544+0000","last_change":"2024-01-11T08:30:16.139974+0000","last_active":"2024-01-11T08:30:16.146544+0000","last_peered":"2024-01-11T08:30:16.146544+0000","last_clean":"2024-01-11T08:30:16.146544+0000","last_became_active":"2024-01-11T08:30:16.139432+0000","last_became_peered":"2024-01-11T08:30:16.139432+0000","last_unstale":"2024-01-11T08:30:16.146544+0000","last_undegraded":"2024-01-11T08:30:16.146544+0000","last_fullsized":"2024-01-11T08:30:16.146544+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:19:17.746788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.125883+0000","last_change":"2024-01-11T08:30:15.128567+0000","last_active":"2024-01-11T08:30:16.125883+0000","last_peered":"2024-01-11T08:30:16.125883+0000","last_clean":"2024-01-11T08:30:16.125883+0000","last_became_active":"2024-01-11T08:30:15.128307+0000","last_became_peered":"2024-01-11T08:30:15.128307+0000","last_unstale":"2024-01-11T08:30:16.125883+0000","last_undegraded":"2024-01-11T08:30:16.125883+0000","last_fullsized":"2024-01-11T08:30:16.125883+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:47:26.962304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.147528+0000","last_change":"2024-01-11T08:30:16.147528+0000","last_active":"2024-01-11T08:30:16.147528+0000","last_peered":"2024-01-11T08:30:16.147528+0000","last_clean":"2024-01-11T08:30:16.147528+0000","last_became_active":"2024-01-11T08:30:16.147270+0000","last_became_peered":"2024-01-11T08:30:16.147270+0000","last_unstale":"2024-01-11T08:30:16.147528+0000","last_undegraded":"2024-01-11T08:30:16.147528+0000","last_fullsized":"2024-01-11T08:30:16.147528+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:02:12.583528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.137017+0000","last_change":"2024-01-11T08:30:15.132288+0000","last_active":"2024-01-11T08:30:15.137017+0000","last_peered":"2024-01-11T08:30:15.137017+0000","last_clean":"2024-01-11T08:30:15.137017+0000","last_became_active":"2024-01-11T08:30:15.131944+0000","last_became_peered":"2024-01-11T08:30:15.131944+0000","last_unstale":"2024-01-11T08:30:15.137017+0000","last_undegraded":"2024-01-11T08:30:15.137017+0000","last_fullsized":"2024-01-11T08:30:15.137017+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:33.055906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":8,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.144810+0000","last_change":"2024-01-11T08:30:16.144810+0000","last_active":"2024-01-11T08:30:16.144810+0000","last_peered":"2024-01-11T08:30:16.144810+0000","last_clean":"2024-01-11T08:30:16.144810+0000","last_became_active":"2024-01-11T08:30:16.143818+0000","last_became_peered":"2024-01-11T08:30:16.143818+0000","last_unstale":"2024-01-11T08:30:16.144810+0000","last_undegraded":"2024-01-11T08:30:16.144810+0000","last_fullsized":"2024-01-11T08:30:16.144810+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:15.105048+0000","last_clean_scrub_stamp":"2024-01-11T08:30:15.105048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:22:13.465322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"144'2","reported_seq":11,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-01-11T08:30:16.160442+0000","last_change":"2024-01-11T08:30:15.138266+0000","last_active":"2024-01-11T08:30:16.160442+0000","last_peered":"2024-01-11T08:30:16.160442+0000","last_clean":"2024-01-11T08:30:16.160442+0000","last_became_active":"2024-01-11T08:30:15.137599+0000","last_became_peered":"2024-01-11T08:30:15.137599+0000","last_unstale":"2024-01-11T08:30:16.160442+0000","last_undegraded":"2024-01-11T08:30:16.160442+0000","last_fullsized":"2024-01-11T08:30:16.160442+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:30:14.096623+0000","last_clean_scrub_stamp":"2024-01-11T08:30:14.096623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:22:08.567433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"135'267","reported_seq":792,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-01-11T08:30:15.125966+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T08:30:15.125966+0000","last_peered":"2024-01-11T08:30:15.125966+0000","last_clean":"2024-01-11T08:30:15.125966+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T08:30:15.125966+0000","last_undegraded":"2024-01-11T08:30:15.125966+0000","last_fullsized":"2024-01-11T08:30:15.125966+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1424,"num_read_kb":2954,"num_write":431,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1424,"num_read_kb":2954,"num_write":431,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267,"ondisk_log_size":267,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575079,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303740,"kb_used_data":2356,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068100,"statfs":{"total":96636764160,"available":96325734400,"internally_reserved":0,"allocated":2412544,"data_stored":1432653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64600000000000002,"15min":0.68799999999999994},"min":{"1min":0.42399999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.745},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64200000000000002,"15min":0.69399999999999995},"min":{"1min":0.46999999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":1.073},"last":0.94499999999999995}]},{"osd":1,"last update":"Thu Jan 11 08:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65200000000000002,"15min":0.69099999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81699999999999995,"5min":0.93400000000000005,"15min":0.99099999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67400000000000004,"15min":0.70699999999999996},"min":{"1min":0.45100000000000001,"5min":0.41999999999999998,"15min":0.40999999999999998},"max":{"1min":0.86499999999999999,"5min":0.94999999999999996,"15min":1.014},"last":0.86499999999999999}]},{"osd":2,"last update":"Thu Jan 11 08:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.65900000000000003,"15min":0.71299999999999997},"min":{"1min":0.42099999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66900000000000004,"15min":0.70699999999999996},"min":{"1min":0.44700000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.95299999999999996,"5min":1.006,"15min":1.006},"last":0.84599999999999997}]},{"osd":3,"last update":"Thu Jan 11 08:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.68700000000000006,"15min":0.72699999999999998},"min":{"1min":0.50600000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.997,"5min":0.997,"15min":1.032},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.69799999999999995,"15min":0.72699999999999998},"min":{"1min":0.55600000000000005,"5min":0.439,"15min":0.439},"max":{"1min":0.95699999999999996,"5min":1.008,"15min":1.008},"last":0.80000000000000004}]},{"osd":4,"last update":"Thu Jan 11 08:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.496,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":0.80600000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64300000000000002,"15min":0.63200000000000001},"min":{"1min":0.38600000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.96399999999999997,"5min":1.002,"15min":1.002},"last":0.90100000000000002}]},{"osd":5,"last update":"Thu Jan 11 08:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.66200000000000003,"15min":0.64100000000000001},"min":{"1min":0.43099999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92600000000000005,"5min":1.042,"15min":1.042},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.66900000000000004,"15min":0.67200000000000004},"min":{"1min":0.503,"5min":0.41399999999999998,"15min":0.40400000000000003},"max":{"1min":0.97399999999999998,"5min":0.99199999999999999,"15min":1.0089999999999999},"last":0.98899999999999999}]},{"osd":6,"last update":"Thu Jan 11 08:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.73899999999999999,"15min":0.749},"min":{"1min":0.60499999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.028,"5min":1.028,"15min":1.0469999999999999},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.71699999999999997,"15min":0.72799999999999998},"min":{"1min":0.57199999999999995,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.91200000000000003,"5min":0.97599999999999998,"15min":1.028},"last":0.96099999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575078,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307244,"kb_used_data":1764,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064596,"statfs":{"total":96636764160,"available":96322146304,"internally_reserved":0,"allocated":1806336,"data_stored":841675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:29:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.61599999999999999,"15min":0.65700000000000003},"min":{"1min":0.34799999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.89500000000000002,"5min":0.91800000000000004,"15min":0.94799999999999995},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.628,"15min":0.67500000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.33800000000000002},"max":{"1min":0.78400000000000003,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.77300000000000002}]},{"osd":1,"last update":"Thu Jan 11 08:29:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65300000000000002,"15min":0.68799999999999994},"min":{"1min":0.41899999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.91900000000000004,"5min":1.008,"15min":1.163},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64800000000000002,"15min":0.67700000000000005},"min":{"1min":0.42599999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82699999999999996,"5min":0.92600000000000005,"15min":1.1950000000000001},"last":0.62}]},{"osd":2,"last update":"Thu Jan 11 08:29:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66300000000000003,"15min":0.69699999999999995},"min":{"1min":0.46899999999999997,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84799999999999998,"5min":0.94399999999999995,"15min":1.016},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66800000000000004,"15min":0.70999999999999996},"min":{"1min":0.47899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.88200000000000001,"5min":0.90000000000000002,"15min":0.97399999999999998},"last":0.64900000000000002}]},{"osd":3,"last update":"Thu Jan 11 08:29:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.69199999999999995,"15min":0.72099999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.876,"5min":0.98199999999999998,"15min":1.0149999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70399999999999996,"15min":0.73299999999999998},"min":{"1min":0.499,"5min":0.46000000000000002,"15min":0.41299999999999998},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":1.0049999999999999},"last":0.68899999999999995}]},{"osd":4,"last update":"Thu Jan 11 08:29:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.59199999999999997,"15min":0.56799999999999995},"min":{"1min":0.44900000000000001,"5min":0.36699999999999999,"15min":0.30199999999999999},"max":{"1min":0.84099999999999997,"5min":0.873,"15min":0.92000000000000004},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.61099999999999999,"15min":0.60199999999999998},"min":{"1min":0.442,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90000000000000002,"5min":0.90300000000000002,"15min":0.96399999999999997},"last":0.53100000000000003}]},{"osd":5,"last update":"Thu Jan 11 08:29:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65800000000000003,"15min":0.65500000000000003},"min":{"1min":0.42299999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.98599999999999999,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.755},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.71099999999999997,"15min":0.70299999999999996},"min":{"1min":0.52900000000000003,"5min":0.41799999999999998,"15min":0.35199999999999998},"max":{"1min":0.96099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.49399999999999999}]},{"osd":7,"last update":"Thu Jan 11 08:29:46 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.70799999999999996,"15min":0.71099999999999997},"min":{"1min":0.47899999999999998,"5min":0.44700000000000001,"15min":0.42399999999999999},"max":{"1min":0.97799999999999998,"5min":1.085,"15min":1.085},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.748,"15min":0.752},"min":{"1min":0.57699999999999996,"5min":0.46700000000000003,"15min":0.44700000000000001},"max":{"1min":1.1140000000000001,"5min":1.159,"15min":1.159},"last":0.81499999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575080,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303136,"kb_used_data":1752,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068704,"statfs":{"total":96636764160,"available":96326352896,"internally_reserved":0,"allocated":1794048,"data_stored":840139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46400000000000002,"15min":0.48299999999999998},"min":{"1min":0.30199999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.70099999999999996,"5min":0.85799999999999998,"15min":0.97699999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53500000000000003,"15min":0.57199999999999995},"min":{"1min":0.41499999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67500000000000004,"5min":0.91700000000000004,"15min":0.93999999999999995},"last":0.66400000000000003}]},{"osd":2,"last update":"Thu Jan 11 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.443,"15min":0.47499999999999998},"min":{"1min":0.34799999999999998,"5min":0.32300000000000001,"15min":0.312},"max":{"1min":0.63900000000000001,"5min":0.70299999999999996,"15min":1.113},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53200000000000003,"15min":0.55700000000000005},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.90100000000000002},"last":0.373}]},{"osd":3,"last update":"Thu Jan 11 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57899999999999996,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.92800000000000005},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.55600000000000005},"min":{"1min":0.38600000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.81000000000000005,"5min":0.88100000000000001,"15min":0.94899999999999995},"last":0.51900000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.76800000000000002,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.45900000000000002,"15min":0.439},"max":{"1min":1.097,"5min":1.129,"15min":1.2170000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.81699999999999995,"15min":0.82499999999999996},"min":{"1min":0.60599999999999998,"5min":0.59199999999999997,"15min":0.501},"max":{"1min":1.0249999999999999,"5min":1.109,"15min":1.246},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Jan 11 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.76800000000000002,"15min":0.79100000000000004},"min":{"1min":0.52000000000000002,"5min":0.498,"15min":0.498},"max":{"1min":1.0029999999999999,"5min":1.1299999999999999,"15min":1.218},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.746,"15min":0.77800000000000002},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.51200000000000001},"max":{"1min":0.98599999999999999,"5min":1.052,"15min":1.1779999999999999},"last":0.63500000000000001}]},{"osd":6,"last update":"Thu Jan 11 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78500000000000003,"15min":0.81299999999999994},"min":{"1min":0.57699999999999996,"5min":0.56000000000000005,"15min":0.51300000000000001},"max":{"1min":1.0660000000000001,"5min":1.115,"15min":1.1439999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79200000000000004,"15min":0.81899999999999995},"min":{"1min":0.56499999999999995,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.97199999999999998,"5min":1.091,"15min":1.228},"last":0.623}]},{"osd":7,"last update":"Thu Jan 11 08:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81299999999999994,"15min":0.84199999999999997},"min":{"1min":0.57699999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.1080000000000001,"5min":1.248,"15min":1.248},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.81399999999999995,"15min":0.83499999999999996},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0349999999999999,"5min":1.099,"15min":1.258},"last":0.69899999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575079,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303716,"kb_used_data":2332,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068124,"statfs":{"total":96636764160,"available":96325758976,"internally_reserved":0,"allocated":2387968,"data_stored":1430507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.47999999999999998,"15min":0.497},"min":{"1min":0.36399999999999999,"5min":0.28699999999999998,"15min":0.24299999999999999},"max":{"1min":0.54700000000000004,"5min":0.84399999999999997,"15min":0.92700000000000005},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54800000000000004,"15min":0.56799999999999995},"min":{"1min":0.40999999999999998,"5min":0.39900000000000002,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.89600000000000002,"15min":0.92100000000000004},"last":0.60099999999999998}]},{"osd":2,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.53400000000000003,"15min":0.54400000000000004},"min":{"1min":0.33300000000000002,"5min":0.33000000000000002,"15min":0.29099999999999998},"max":{"1min":0.68400000000000005,"5min":1.335,"15min":1.335},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54600000000000004,"15min":0.55500000000000005},"min":{"1min":0.35999999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.626,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.63700000000000001}]},{"osd":3,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.50800000000000001,"15min":0.52200000000000002},"min":{"1min":0.34999999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.628,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.60799999999999998,"15min":0.621},"min":{"1min":0.45500000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.75800000000000001,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.72699999999999998}]},{"osd":4,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.76400000000000001,"15min":0.78800000000000003},"min":{"1min":0.51900000000000002,"5min":0.502,"15min":0.437},"max":{"1min":1.0429999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.76000000000000001,"15min":0.78800000000000003},"min":{"1min":0.56799999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.99099999999999999,"5min":1.1220000000000001,"15min":1.149},"last":0.82299999999999995}]},{"osd":5,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.76800000000000002,"15min":0.80100000000000005},"min":{"1min":0.56399999999999995,"5min":0.54700000000000004,"15min":0.52500000000000002},"max":{"1min":0.95299999999999996,"5min":1.518,"15min":1.518},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.78900000000000003,"15min":0.81499999999999995},"min":{"1min":0.58999999999999997,"5min":0.55400000000000005,"15min":0.47799999999999998},"max":{"1min":0.94999999999999996,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.84799999999999998}]},{"osd":6,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.78000000000000003,"15min":0.80200000000000005},"min":{"1min":0.54100000000000004,"5min":0.51700000000000002,"15min":0.50900000000000001},"max":{"1min":0.96099999999999997,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.80100000000000005,"15min":0.82799999999999996},"min":{"1min":0.52800000000000002,"5min":0.52000000000000002,"15min":0.44900000000000001},"max":{"1min":0.97299999999999998,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.93600000000000005}]},{"osd":7,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.81599999999999995,"15min":0.84199999999999997},"min":{"1min":0.63900000000000001,"5min":0.59199999999999997,"15min":0.51400000000000001},"max":{"1min":0.98499999999999999,"5min":1.371,"15min":1.371},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.79600000000000004,"15min":0.83399999999999996},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.056,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":1.02}]}]},{"osd":2,"up_from":13,"seq":55834575078,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311336,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060504,"statfs":{"total":96636764160,"available":96317956096,"internally_reserved":0,"allocated":1802240,"data_stored":840727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45000000000000001,"15min":0.46800000000000003},"min":{"1min":0.307,"5min":0.27800000000000002,"15min":0.26000000000000001},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.625},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.50800000000000001,"15min":0.53700000000000003},"min":{"1min":0.313,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.47699999999999998}]},{"osd":1,"last update":"Thu Jan 11 08:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53500000000000003,"15min":0.54600000000000004},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55200000000000005,"15min":0.57299999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.70999999999999996,"5min":0.79600000000000004,"15min":0.94499999999999995},"last":0.56299999999999994}]},{"osd":3,"last update":"Thu Jan 11 08:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.52200000000000002,"15min":0.54100000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.901,"5min":1.901,"15min":1.901},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.58399999999999996,"15min":0.623},"min":{"1min":0.436,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":1.0009999999999999},"last":0.51900000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.73799999999999999,"15min":0.77700000000000002},"min":{"1min":0.5,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.2889999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.73599999999999999,"15min":0.77200000000000002},"min":{"1min":0.51700000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.61199999999999999}]},{"osd":5,"last update":"Thu Jan 11 08:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.79600000000000004,"15min":0.82399999999999995},"min":{"1min":0.61899999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0229999999999999,"5min":1.2470000000000001,"15min":1.3540000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79200000000000004,"15min":0.83099999999999996},"min":{"1min":0.57299999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.749}]},{"osd":6,"last update":"Thu Jan 11 08:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.746,"15min":0.78800000000000003},"min":{"1min":0.50700000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.77500000000000002,"15min":0.81299999999999994},"min":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.073,"5min":1.351,"15min":1.47},"last":0.64900000000000002}]},{"osd":7,"last update":"Thu Jan 11 08:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.79000000000000004,"15min":0.84399999999999997},"min":{"1min":0.54600000000000004,"5min":0.54000000000000004,"15min":0.51400000000000001},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.4099999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.80100000000000005,"15min":0.84599999999999997},"min":{"1min":0.55700000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.992,"5min":1.992,"15min":1.992},"last":0.72899999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575078,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311336,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060504,"statfs":{"total":96636764160,"available":96317956096,"internally_reserved":0,"allocated":1802240,"data_stored":840299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.49199999999999999,"15min":0.49099999999999999},"min":{"1min":0.309,"5min":0.29799999999999999,"15min":0.27600000000000002},"max":{"1min":0.79900000000000004,"5min":0.80400000000000005,"15min":1.1930000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.54200000000000004},"min":{"1min":0.378,"5min":0.36699999999999999,"15min":0.33400000000000002},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.441}]},{"osd":1,"last update":"Thu Jan 11 08:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.48799999999999999,"15min":0.48899999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.25900000000000001},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55500000000000005,"15min":0.57099999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.36299999999999999},"max":{"1min":1.0069999999999999,"5min":1.0960000000000001,"15min":1.1499999999999999},"last":0.47099999999999997}]},{"osd":2,"last update":"Thu Jan 11 08:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56200000000000006,"15min":0.57699999999999996},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78700000000000003,"5min":0.94699999999999995,"15min":0.97599999999999998},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.61099999999999999,"15min":0.628},"min":{"1min":0.433,"5min":0.42899999999999999,"15min":0.32000000000000001},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":0.61599999999999999}]},{"osd":4,"last update":"Thu Jan 11 08:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72399999999999998,"15min":0.73299999999999998},"min":{"1min":0.53100000000000003,"5min":0.495,"15min":0.35999999999999999},"max":{"1min":1.4690000000000001,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.75,"15min":0.76900000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.503},"max":{"1min":1.1259999999999999,"5min":1.1459999999999999,"15min":1.1819999999999999},"last":0.72399999999999998}]},{"osd":5,"last update":"Thu Jan 11 08:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76300000000000001,"15min":0.78000000000000003},"min":{"1min":0.56599999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.96199999999999997,"5min":1.0649999999999999,"15min":1.3260000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79200000000000004,"15min":0.80200000000000005},"min":{"1min":0.52800000000000002,"5min":0.5,"15min":0.49199999999999999},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.80700000000000005}]},{"osd":6,"last update":"Thu Jan 11 08:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.80700000000000005,"15min":0.82799999999999996},"min":{"1min":0.57199999999999995,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.77400000000000002,"15min":0.78900000000000003},"min":{"1min":0.52300000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.425,"5min":2.3599999999999999,"15min":2.3599999999999999},"last":0.745}]},{"osd":7,"last update":"Thu Jan 11 08:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82299999999999995,"15min":0.83099999999999996},"min":{"1min":0.60999999999999999,"5min":0.53100000000000003,"15min":0.46600000000000003},"max":{"1min":1.516,"5min":2.3180000000000001,"15min":2.3180000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83699999999999997,"15min":0.85399999999999998},"min":{"1min":0.626,"5min":0.59199999999999997,"15min":0.58399999999999996},"max":{"1min":1.1639999999999999,"5min":1.2050000000000001,"15min":1.21},"last":0.83299999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575079,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311332,"kb_used_data":1756,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060508,"statfs":{"total":96636764160,"available":96317960192,"internally_reserved":0,"allocated":1798144,"data_stored":840209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.67300000000000004,"15min":0.72899999999999998},"min":{"1min":0.51300000000000001,"5min":0.41999999999999998,"15min":0.28100000000000003},"max":{"1min":0.76500000000000001,"5min":1.0149999999999999,"15min":1.117},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.67900000000000005,"15min":0.72699999999999998},"min":{"1min":0.48299999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80800000000000005,"5min":0.95699999999999996,"15min":1.0049999999999999},"last":0.58499999999999996}]},{"osd":1,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.65500000000000003,"15min":0.69399999999999995},"min":{"1min":0.48899999999999999,"5min":0.39100000000000001,"15min":0.33300000000000002},"max":{"1min":0.89300000000000002,"5min":0.91000000000000003,"15min":1.0069999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.66300000000000003,"15min":0.70199999999999996},"min":{"1min":0.48499999999999999,"5min":0.43099999999999999,"15min":0.36299999999999999},"max":{"1min":0.84899999999999998,"5min":0.94899999999999995,"15min":1.046},"last":0.61799999999999999}]},{"osd":2,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.68799999999999994,"15min":0.71599999999999997},"min":{"1min":0.505,"5min":0.46600000000000003,"15min":0.38900000000000001},"max":{"1min":0.82099999999999995,"5min":1.03,"15min":1.03},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.66000000000000003,"15min":0.71499999999999997},"min":{"1min":0.45900000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.751,"5min":0.91500000000000004,"15min":1.0409999999999999},"last":0.751}]},{"osd":3,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.64900000000000002,"15min":0.69099999999999995},"min":{"1min":0.48399999999999999,"5min":0.46100000000000002,"15min":0.433},"max":{"1min":0.753,"5min":0.96799999999999997,"15min":1.0189999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.69699999999999995,"15min":0.73699999999999999},"min":{"1min":0.47299999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.92400000000000004,"5min":0.97499999999999998,"15min":1.026},"last":0.69999999999999996}]},{"osd":5,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.60599999999999998,"15min":0.60699999999999998},"min":{"1min":0.39800000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79400000000000004,"5min":0.96599999999999997,"15min":1.0700000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.61799999999999999,"15min":0.62},"min":{"1min":0.44700000000000001,"5min":0.39400000000000002,"15min":0.32600000000000001},"max":{"1min":0.77500000000000002,"5min":0.94299999999999995,"15min":0.97799999999999998},"last":0.89400000000000002}]},{"osd":6,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.64800000000000002,"15min":0.64500000000000002},"min":{"1min":0.47099999999999997,"5min":0.40500000000000003,"15min":0.39500000000000002},"max":{"1min":0.79400000000000004,"5min":1.0680000000000001,"15min":1.133},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.69899999999999995,"15min":0.68799999999999994},"min":{"1min":0.45900000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.89300000000000002,"5min":1.0129999999999999,"15min":1.119},"last":0.83899999999999997}]},{"osd":7,"last update":"Thu Jan 11 08:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.68100000000000005,"15min":0.68200000000000005},"min":{"1min":0.51800000000000002,"5min":0.44600000000000001,"15min":0.41299999999999998},"max":{"1min":0.83199999999999996,"5min":1.0129999999999999,"15min":1.0640000000000001},"last":1.01},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.71999999999999997,"15min":0.72999999999999998},"min":{"1min":0.55000000000000004,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.84599999999999997,"5min":1.028,"15min":1.2350000000000001},"last":0.82499999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575079,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307252,"kb_used_data":1772,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064588,"statfs":{"total":96636764160,"available":96322138112,"internally_reserved":0,"allocated":1814528,"data_stored":846595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.65700000000000003,"15min":0.69999999999999996},"min":{"1min":0.44500000000000001,"5min":0.39300000000000002,"15min":0.38700000000000001},"max":{"1min":0.98499999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.66800000000000004,"15min":0.70599999999999996},"min":{"1min":0.498,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.878,"5min":0.93500000000000005,"15min":1.696},"last":0.80600000000000005}]},{"osd":1,"last update":"Thu Jan 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.64000000000000001,"15min":0.68799999999999994},"min":{"1min":0.41699999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.94399999999999995,"5min":0.997,"15min":1.4359999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.67000000000000004,"15min":0.72599999999999998},"min":{"1min":0.42899999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.98099999999999998,"5min":1.05,"15min":4.4969999999999999},"last":0.60399999999999998}]},{"osd":2,"last update":"Thu Jan 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66500000000000004,"15min":0.70299999999999996},"min":{"1min":0.39800000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.01,"5min":1.01,"15min":1.238},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67600000000000005,"15min":0.73499999999999999},"min":{"1min":0.47099999999999997,"5min":0.45300000000000001,"15min":0.41599999999999998},"max":{"1min":0.91900000000000004,"5min":0.95899999999999996,"15min":8.0429999999999993},"last":0.63}]},{"osd":3,"last update":"Thu Jan 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.69099999999999995,"15min":0.745},"min":{"1min":0.48799999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.052,"5min":1.208,"15min":8.0839999999999996},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.71399999999999997,"15min":0.73799999999999999},"min":{"1min":0.50600000000000001,"5min":0.49099999999999999,"15min":0.40899999999999997},"max":{"1min":1.0409999999999999,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.65100000000000002}]},{"osd":4,"last update":"Thu Jan 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.59399999999999997,"15min":0.59899999999999998},"min":{"1min":0.45400000000000001,"5min":0.38300000000000001,"15min":0.34300000000000003},"max":{"1min":0.78100000000000003,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.63500000000000001,"15min":0.63100000000000001},"min":{"1min":0.432,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.91400000000000003,"5min":1.256,"15min":1.256},"last":0.81999999999999995}]},{"osd":6,"last update":"Thu Jan 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.39000000000000001},"max":{"1min":0.96499999999999997,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.67000000000000004,"15min":0.67900000000000005},"min":{"1min":0.46400000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.90300000000000002,"5min":1.085,"15min":1.085},"last":0.72599999999999998}]},{"osd":7,"last update":"Thu Jan 11 08:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.70599999999999996,"15min":0.72499999999999998},"min":{"1min":0.49199999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.038,"5min":1.341,"15min":1.341},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.76300000000000001,"15min":0.76900000000000002},"min":{"1min":0.53900000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.1120000000000001,"5min":1.119,"15min":1.119},"last":0.84499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T08:30:21.035 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-11T08:30:21.035 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-11T08:30:21.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.279+0000 7f2552dff700 1 Processor -- start 2024-01-11T08:30:21.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.279+0000 7f2552dff700 1 -- start start 2024-01-11T08:30:21.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.280+0000 7f2552dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f254c06bba0 0x7f254c0cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.280+0000 7f2552dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f254c15db00 0x7f254c15df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.281+0000 7f2552dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254c15d670 con 0x7f254c06bf70 2024-01-11T08:30:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.281+0000 7f2552dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f254c069fc0 con 0x7f254c15db00 2024-01-11T08:30:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.281+0000 7f2552dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f254c0cce10 con 0x7f254c06bba0 2024-01-11T08:30:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.281+0000 7f255139c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f254c15db00 0x7f254c15df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.281+0000 7f255139c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f254c15db00 0x7f254c15df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34952/0 (socket says 172.21.15.27:34952) 2024-01-11T08:30:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.281+0000 7f255139c700 1 -- 172.21.15.27:0/2669504422 learned_addr learned my addr 172.21.15.27:0/2669504422 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.281+0000 7f2550b9b700 1 --2- 172.21.15.27:0/2669504422 >> v2:172.21.15.77:3300/0 conn(0x7f254c06bba0 0x7f254c0cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.281+0000 7f255139c700 1 -- 172.21.15.27:0/2669504422 >> v2:172.21.15.77:3300/0 conn(0x7f254c06bba0 msgr2=0x7f254c0cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.281+0000 7f255139c700 1 --2- 172.21.15.27:0/2669504422 >> v2:172.21.15.77:3300/0 conn(0x7f254c06bba0 0x7f254c0cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.281+0000 7f255139c700 1 -- 172.21.15.27:0/2669504422 >> v1:172.21.15.77:6789/0 conn(0x7f254c06bf70 legacy=0x7f254c159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.281+0000 7f255139c700 1 -- 172.21.15.27:0/2669504422 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254c15a600 con 0x7f254c15db00 2024-01-11T08:30:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.281+0000 7f2550b9b700 1 --2- 172.21.15.27:0/2669504422 >> v2:172.21.15.77:3300/0 conn(0x7f254c06bba0 0x7f254c0cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:30:21.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.282+0000 7f255139c700 1 --2- 172.21.15.27:0/2669504422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f254c15db00 0x7f254c15df20 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f254c0adc70 tx=0x7f2544009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b43accb82fe803 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:21.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.282+0000 7f254b7fe700 1 -- 172.21.15.27:0/2669504422 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254400e070 con 0x7f254c15db00 2024-01-11T08:30:21.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.282+0000 7f254b7fe700 1 -- 172.21.15.27:0/2669504422 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f254400ba10 con 0x7f254c15db00 2024-01-11T08:30:21.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.282+0000 7f254b7fe700 1 -- 172.21.15.27:0/2669504422 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25440092e0 con 0x7f254c15db00 2024-01-11T08:30:21.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.282+0000 7f2552dff700 1 -- 172.21.15.27:0/2669504422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f254c15db00 msgr2=0x7f254c15df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:21.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.282+0000 7f2552dff700 1 --2- 172.21.15.27:0/2669504422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f254c15db00 0x7f254c15df20 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f254c0adc70 tx=0x7f2544009840 comp rx=0 tx=0).stop 2024-01-11T08:30:21.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.282+0000 7f2552dff700 1 -- 172.21.15.27:0/2669504422 shutdown_connections 2024-01-11T08:30:21.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.282+0000 7f2552dff700 1 --2- 172.21.15.27:0/2669504422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f254c15db00 0x7f254c15df20 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:21.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.282+0000 7f2552dff700 1 --2- 172.21.15.27:0/2669504422 >> v2:172.21.15.77:3300/0 conn(0x7f254c06bba0 0x7f254c0cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:21.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.282+0000 7f2552dff700 1 -- 172.21.15.27:0/2669504422 >> 172.21.15.27:0/2669504422 conn(0x7f254c0ce3e0 msgr2=0x7f254c0ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:21.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.283+0000 7f2552dff700 1 -- 172.21.15.27:0/2669504422 shutdown_connections 2024-01-11T08:30:21.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.283+0000 7f2552dff700 1 -- 172.21.15.27:0/2669504422 wait complete. 2024-01-11T08:30:21.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.285+0000 7f2552dff700 1 Processor -- start 2024-01-11T08:30:21.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.285+0000 7f2552dff700 1 -- start start 2024-01-11T08:30:21.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.286+0000 7f2552dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f254c06bba0 0x7f254c14da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:21.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.286+0000 7f2552dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f254c06bf70 0x7f254c14ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:21.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.286+0000 7f2552dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f254c15db00 0x7f254c0adca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:21.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.286+0000 7f2552dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254c14d3c0 con 0x7f254c06bf70 2024-01-11T08:30:21.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.286+0000 7f2552dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254c0b2850 con 0x7f254c06bba0 2024-01-11T08:30:21.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.286+0000 7f2552dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f254c0aff40 con 0x7f254c15db00 2024-01-11T08:30:21.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.286+0000 7f254bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f254c06bf70 0x7f254c14ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.286+0000 7f254bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f254c06bf70 0x7f254c14ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34954/0 (socket says 172.21.15.27:34954) 2024-01-11T08:30:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.286+0000 7f254bfff700 1 -- 172.21.15.27:0/805661352 learned_addr learned my addr 172.21.15.27:0/805661352 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.286+0000 7f255139c700 1 --2- 172.21.15.27:0/805661352 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f254c15db00 0x7f254c0adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.286+0000 7f2550b9b700 1 --2- 172.21.15.27:0/805661352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f254c06bba0 0x7f254c14da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.286+0000 7f254bfff700 1 -- 172.21.15.27:0/805661352 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f254c15db00 msgr2=0x7f254c0adca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.286+0000 7f254bfff700 1 --2- 172.21.15.27:0/805661352 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f254c15db00 0x7f254c0adca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.286+0000 7f254bfff700 1 -- 172.21.15.27:0/805661352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f254c06bba0 msgr2=0x7f254c14da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.286+0000 7f254bfff700 1 --2- 172.21.15.27:0/805661352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f254c06bba0 0x7f254c14da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:21.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.286+0000 7f254bfff700 1 -- 172.21.15.27:0/805661352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2544009ae0 con 0x7f254c06bf70 2024-01-11T08:30:21.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.287+0000 7f254bfff700 1 --2- 172.21.15.27:0/805661352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f254c06bf70 0x7f254c14ce80 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f253800c5d0 tx=0x7f2538007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:21.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.287+0000 7f2548ff9700 1 -- 172.21.15.27:0/805661352 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538011590 con 0x7f254c06bf70 2024-01-11T08:30:21.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.287+0000 7f2548ff9700 1 -- 172.21.15.27:0/805661352 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2538011710 con 0x7f254c06bf70 2024-01-11T08:30:21.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.287+0000 7f2552dff700 1 -- 172.21.15.27:0/805661352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f254c0b0220 con 0x7f254c06bf70 2024-01-11T08:30:21.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.287+0000 7f2548ff9700 1 -- 172.21.15.27:0/805661352 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538011ad0 con 0x7f254c06bf70 2024-01-11T08:30:21.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.287+0000 7f2552dff700 1 -- 172.21.15.27:0/805661352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f254c0b24e0 con 0x7f254c06bf70 2024-01-11T08:30:21.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.288+0000 7f2552dff700 1 -- 172.21.15.27:0/805661352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f254c0b2a00 con 0x7f254c06bf70 2024-01-11T08:30:21.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.289+0000 7f2548ff9700 1 -- 172.21.15.27:0/805661352 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f253801c460 con 0x7f254c06bf70 2024-01-11T08:30:21.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.291+0000 7f2548ff9700 1 --2- 172.21.15.27:0/805661352 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2530066f60 0x7f2530069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:21.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.291+0000 7f2548ff9700 1 -- 172.21.15.27:0/805661352 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f2538084260 con 0x7f254c06bf70 2024-01-11T08:30:21.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.291+0000 7f2550b9b700 1 --2- 172.21.15.27:0/805661352 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2530066f60 0x7f2530069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:21.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.292+0000 7f2550b9b700 1 --2- 172.21.15.27:0/805661352 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2530066f60 0x7f2530069410 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f254000adc0 tx=0x7f2540006f90 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:21.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.295+0000 7f2548ff9700 1 -- 172.21.15.27:0/805661352 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f254c0b2a00 con 0x7f254c06bf70 2024-01-11T08:30:21.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.580+0000 7f2552dff700 1 -- 172.21.15.27:0/805661352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f254c0b29c0 con 0x7f254c06bf70 2024-01-11T08:30:21.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.585+0000 7f2548ff9700 1 -- 172.21.15.27:0/805661352 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+0 (secure 0 0 0) 0x7f25380513f0 con 0x7f254c06bf70 2024-01-11T08:30:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.587+0000 7f2552dff700 1 -- 172.21.15.27:0/805661352 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2530066f60 msgr2=0x7f2530069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.587+0000 7f2552dff700 1 --2- 172.21.15.27:0/805661352 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2530066f60 0x7f2530069410 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f254000adc0 tx=0x7f2540006f90 comp rx=0 tx=0).stop 2024-01-11T08:30:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.587+0000 7f2552dff700 1 -- 172.21.15.27:0/805661352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f254c06bf70 msgr2=0x7f254c14ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.587+0000 7f2552dff700 1 --2- 172.21.15.27:0/805661352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f254c06bf70 0x7f254c14ce80 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f253800c5d0 tx=0x7f2538007f60 comp rx=0 tx=0).stop 2024-01-11T08:30:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.587+0000 7f2552dff700 1 -- 172.21.15.27:0/805661352 shutdown_connections 2024-01-11T08:30:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.587+0000 7f2552dff700 1 --2- 172.21.15.27:0/805661352 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f2530066f60 0x7f2530069410 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.587+0000 7f2552dff700 1 --2- 172.21.15.27:0/805661352 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f254c15db00 0x7f254c0adca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.587+0000 7f2552dff700 1 --2- 172.21.15.27:0/805661352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f254c06bba0 0x7f254c14da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.587+0000 7f2552dff700 1 --2- 172.21.15.27:0/805661352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f254c06bf70 0x7f254c14ce80 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.587+0000 7f2552dff700 1 -- 172.21.15.27:0/805661352 >> 172.21.15.27:0/805661352 conn(0x7f254c0ce3e0 msgr2=0x7f254c15c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.588+0000 7f2552dff700 1 -- 172.21.15.27:0/805661352 shutdown_connections 2024-01-11T08:30:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.588+0000 7f2552dff700 1 -- 172.21.15.27:0/805661352 wait complete. 2024-01-11T08:30:21.588 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-11T08:30:21.604 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T08:30:21.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.806+0000 7f001a1ff700 1 Processor -- start 2024-01-11T08:30:21.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.806+0000 7f001a1ff700 1 -- start start 2024-01-11T08:30:21.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.807+0000 7f001a1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00141575e0 0x7f0014158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:21.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.807+0000 7f001a1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f00140b19d0 0x7f0014159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.807+0000 7f001a1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0014158b60 con 0x7f001406b610 2024-01-11T08:30:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.807+0000 7f001a1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00140cb5e0 con 0x7f00140b19d0 2024-01-11T08:30:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.808+0000 7f001a1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00141599b0 con 0x7f00141575e0 2024-01-11T08:30:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.808+0000 7f0013fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f00140b19d0 0x7f0014159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.808+0000 7f0013fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f00140b19d0 0x7f0014159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34962/0 (socket says 172.21.15.27:34962) 2024-01-11T08:30:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.808+0000 7f0013fff700 1 -- 172.21.15.27:0/2773363533 learned_addr learned my addr 172.21.15.27:0/2773363533 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.808+0000 7f00137fe700 1 --2- 172.21.15.27:0/2773363533 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00141575e0 0x7f0014158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.808+0000 7f0013fff700 1 -- 172.21.15.27:0/2773363533 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00141575e0 msgr2=0x7f0014158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.808+0000 7f0013fff700 1 --2- 172.21.15.27:0/2773363533 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00141575e0 0x7f0014158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.808+0000 7f0013fff700 1 -- 172.21.15.27:0/2773363533 >> v1:172.21.15.77:6789/0 conn(0x7f001406b610 legacy=0x7f00140b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.808+0000 7f0013fff700 1 -- 172.21.15.27:0/2773363533 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00140c0f90 con 0x7f00140b19d0 2024-01-11T08:30:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.808+0000 7f00137fe700 1 --2- 172.21.15.27:0/2773363533 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00141575e0 0x7f0014158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:30:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.809+0000 7f0013fff700 1 --2- 172.21.15.27:0/2773363533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00140b19d0 0x7f0014159e40 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f000c009f20 tx=0x7f000c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1038e87fcfb9b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.809+0000 7f00127fc700 1 -- 172.21.15.27:0/2773363533 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000c00e070 con 0x7f00140b19d0 2024-01-11T08:30:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.809+0000 7f00127fc700 1 -- 172.21.15.27:0/2773363533 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000c00ba40 con 0x7f00140b19d0 2024-01-11T08:30:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.809+0000 7f00127fc700 1 -- 172.21.15.27:0/2773363533 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000c0092e0 con 0x7f00140b19d0 2024-01-11T08:30:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.809+0000 7f001a1ff700 1 -- 172.21.15.27:0/2773363533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00140b19d0 msgr2=0x7f0014159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.809+0000 7f001a1ff700 1 --2- 172.21.15.27:0/2773363533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00140b19d0 0x7f0014159e40 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f000c009f20 tx=0x7f000c009840 comp rx=0 tx=0).stop 2024-01-11T08:30:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.809+0000 7f001a1ff700 1 -- 172.21.15.27:0/2773363533 shutdown_connections 2024-01-11T08:30:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.809+0000 7f001a1ff700 1 --2- 172.21.15.27:0/2773363533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00140b19d0 0x7f0014159e40 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.809+0000 7f001a1ff700 1 --2- 172.21.15.27:0/2773363533 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00141575e0 0x7f0014158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.809+0000 7f001a1ff700 1 -- 172.21.15.27:0/2773363533 >> 172.21.15.27:0/2773363533 conn(0x7f00140680e0 msgr2=0x7f0014069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.809+0000 7f001a1ff700 1 -- 172.21.15.27:0/2773363533 shutdown_connections 2024-01-11T08:30:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.810+0000 7f001a1ff700 1 -- 172.21.15.27:0/2773363533 wait complete. 2024-01-11T08:30:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.812+0000 7f001a1ff700 1 Processor -- start 2024-01-11T08:30:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.812+0000 7f001a1ff700 1 -- start start 2024-01-11T08:30:21.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.813+0000 7f001a1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001406b610 0x7f00140adbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:21.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.813+0000 7f001a1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00140b19d0 0x7f001415d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:21.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.813+0000 7f001a1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00141575e0 0x7f001415d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:21.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.813+0000 7f001a1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00140c8a50 con 0x7f001406b610 2024-01-11T08:30:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.813+0000 7f001a1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00140c3740 con 0x7f00140b19d0 2024-01-11T08:30:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.813+0000 7f001a1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00140c8770 con 0x7f00141575e0 2024-01-11T08:30:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.813+0000 7f00137fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001406b610 0x7f00140adbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.813+0000 7f00137fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001406b610 0x7f00140adbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34968/0 (socket says 172.21.15.27:34968) 2024-01-11T08:30:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.813+0000 7f00137fe700 1 -- 172.21.15.27:0/3494349780 learned_addr learned my addr 172.21.15.27:0/3494349780 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.814+0000 7f0012ffd700 1 --2- 172.21.15.27:0/3494349780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00140b19d0 0x7f001415d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.814+0000 7f0013fff700 1 --2- 172.21.15.27:0/3494349780 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00141575e0 0x7f001415d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.814+0000 7f00137fe700 1 -- 172.21.15.27:0/3494349780 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00141575e0 msgr2=0x7f001415d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.814+0000 7f00137fe700 1 --2- 172.21.15.27:0/3494349780 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00141575e0 0x7f001415d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.814+0000 7f00137fe700 1 -- 172.21.15.27:0/3494349780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00140b19d0 msgr2=0x7f001415d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.814+0000 7f00137fe700 1 --2- 172.21.15.27:0/3494349780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00140b19d0 0x7f001415d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.814+0000 7f00137fe700 1 -- 172.21.15.27:0/3494349780 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000c009ae0 con 0x7f001406b610 2024-01-11T08:30:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.814+0000 7f0013fff700 1 --2- 172.21.15.27:0/3494349780 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00141575e0 0x7f001415d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:30:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.814+0000 7f00137fe700 1 --2- 172.21.15.27:0/3494349780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001406b610 0x7f00140adbc0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f000400c850 tx=0x7f0004008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:21.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.814+0000 7efffb7fe700 1 -- 172.21.15.27:0/3494349780 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0004007650 con 0x7f001406b610 2024-01-11T08:30:21.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.814+0000 7efffb7fe700 1 -- 172.21.15.27:0/3494349780 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00040077b0 con 0x7f001406b610 2024-01-11T08:30:21.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.814+0000 7efffb7fe700 1 -- 172.21.15.27:0/3494349780 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0004007b50 con 0x7f001406b610 2024-01-11T08:30:21.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.814+0000 7f001a1ff700 1 -- 172.21.15.27:0/3494349780 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00140c30f0 con 0x7f001406b610 2024-01-11T08:30:21.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.815+0000 7f001a1ff700 1 -- 172.21.15.27:0/3494349780 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00140ae100 con 0x7f001406b610 2024-01-11T08:30:21.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.816+0000 7f001a1ff700 1 -- 172.21.15.27:0/3494349780 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00140c2f50 con 0x7f001406b610 2024-01-11T08:30:21.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.816+0000 7efffb7fe700 1 -- 172.21.15.27:0/3494349780 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f00040113f0 con 0x7f001406b610 2024-01-11T08:30:21.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.817+0000 7efffb7fe700 1 --2- 172.21.15.27:0/3494349780 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0000066f30 0x7f00000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:21.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.817+0000 7efffb7fe700 1 -- 172.21.15.27:0/3494349780 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f0004084600 con 0x7f001406b610 2024-01-11T08:30:21.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.818+0000 7f0012ffd700 1 --2- 172.21.15.27:0/3494349780 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0000066f30 0x7f00000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:21.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.819+0000 7f0012ffd700 1 --2- 172.21.15.27:0/3494349780 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0000066f30 0x7f00000693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f001416a990 tx=0x7f0008009040 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:21.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:21.822+0000 7efffb7fe700 1 -- 172.21.15.27:0/3494349780 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f00140c2f50 con 0x7f001406b610 2024-01-11T08:30:22.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.073+0000 7f001a1ff700 1 -- 172.21.15.27:0/3494349780 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00140c2f50 con 0x7f001406b610 2024-01-11T08:30:22.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.074+0000 7efffb7fe700 1 -- 172.21.15.27:0/3494349780 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4596 (secure 0 0 0) 0x7f00140c2f50 con 0x7f001406b610 2024-01-11T08:30:22.073 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:30:22.073 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7352,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3818350333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3818350333},{"type":"v1","addr":"172.21.15.27:6834","nonce":3818350333}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/217390254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":217390254},{"type":"v1","addr":"172.21.15.27:6835","nonce":217390254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2824437717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2824437717},{"type":"v1","addr":"172.21.15.77:6834","nonce":2824437717}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:30:15.623796+0000","modified":"2024-01-11T08:30:19.150622+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8601":{"gid":8601,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/1641163931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1641163931},{"type":"v1","addr":"172.21.15.77:6837","nonce":1641163931}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2024-01-11T08:30:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.076+0000 7f001a1ff700 1 -- 172.21.15.27:0/3494349780 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0000066f30 msgr2=0x7f00000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.076+0000 7f001a1ff700 1 --2- 172.21.15.27:0/3494349780 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0000066f30 0x7f00000693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f001416a990 tx=0x7f0008009040 comp rx=0 tx=0).stop 2024-01-11T08:30:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.076+0000 7f001a1ff700 1 -- 172.21.15.27:0/3494349780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001406b610 msgr2=0x7f00140adbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.076+0000 7f001a1ff700 1 --2- 172.21.15.27:0/3494349780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001406b610 0x7f00140adbc0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f000400c850 tx=0x7f0004008040 comp rx=0 tx=0).stop 2024-01-11T08:30:22.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.076+0000 7f001a1ff700 1 -- 172.21.15.27:0/3494349780 shutdown_connections 2024-01-11T08:30:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.076+0000 7f001a1ff700 1 --2- 172.21.15.27:0/3494349780 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0000066f30 0x7f00000693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.076+0000 7f001a1ff700 1 --2- 172.21.15.27:0/3494349780 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00141575e0 0x7f001415d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.076+0000 7f001a1ff700 1 --2- 172.21.15.27:0/3494349780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001406b610 0x7f00140adbc0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.076+0000 7f001a1ff700 1 --2- 172.21.15.27:0/3494349780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00140b19d0 0x7f001415d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.076+0000 7f001a1ff700 1 -- 172.21.15.27:0/3494349780 >> 172.21.15.27:0/3494349780 conn(0x7f00140680e0 msgr2=0x7f0014159e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.076+0000 7f001a1ff700 1 -- 172.21.15.27:0/3494349780 shutdown_connections 2024-01-11T08:30:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.076+0000 7f001a1ff700 1 -- 172.21.15.27:0/3494349780 wait complete. 2024-01-11T08:30:22.076 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 77 2024-01-11T08:30:22.495 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-11T08:30:15.623796+0000', 'modified': '2024-01-11T08:30:19.150622+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8601}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8601': {'gid': 8601, 'name': 'd', 'rank': 0, 'incarnation': 73, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.77:6837/1641163931', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6833', 'nonce': 1641163931}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 1641163931}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-11T08:30:22.496 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-11T08:30:22.496 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-01-11T08:30:22.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.700+0000 7f6bef7ff700 1 Processor -- start 2024-01-11T08:30:22.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.701+0000 7f6bef7ff700 1 -- start start 2024-01-11T08:30:22.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.702+0000 7f6bef7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6be806bf10 0x7f6be8159c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.702+0000 7f6bef7ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6be80b0a40 0x7f6be815da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.702+0000 7f6bef7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6be815df40 0x7f6be8068720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.702+0000 7f6bef7ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be815d290 con 0x7f6be80b0a40 2024-01-11T08:30:22.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.702+0000 7f6bef7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be815d570 con 0x7f6be806bf10 2024-01-11T08:30:22.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.702+0000 7f6bef7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be8154180 con 0x7f6be815df40 2024-01-11T08:30:22.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.702+0000 7f6becd9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6be80b0a40 0x7f6be815da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.702+0000 7f6becd9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6be80b0a40 0x7f6be815da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34978/0 (socket says 172.21.15.27:34978) 2024-01-11T08:30:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.702+0000 7f6becd9a700 1 -- 172.21.15.27:0/3720601229 learned_addr learned my addr 172.21.15.27:0/3720601229 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.702+0000 7f6bed59b700 1 --2- 172.21.15.27:0/3720601229 >> v2:172.21.15.77:3300/0 conn(0x7f6be806bf10 0x7f6be8159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.702+0000 7f6bedd9c700 1 --2- 172.21.15.27:0/3720601229 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6be815df40 0x7f6be8068720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.702+0000 7f6becd9a700 1 -- 172.21.15.27:0/3720601229 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6be815df40 msgr2=0x7f6be8068720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.703+0000 7f6becd9a700 1 --2- 172.21.15.27:0/3720601229 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6be815df40 0x7f6be8068720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.703+0000 7f6becd9a700 1 -- 172.21.15.27:0/3720601229 >> v2:172.21.15.77:3300/0 conn(0x7f6be806bf10 msgr2=0x7f6be8159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.703+0000 7f6becd9a700 1 --2- 172.21.15.27:0/3720601229 >> v2:172.21.15.77:3300/0 conn(0x7f6be806bf10 0x7f6be8159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.703+0000 7f6becd9a700 1 -- 172.21.15.27:0/3720601229 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be815a510 con 0x7f6be80b0a40 2024-01-11T08:30:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.703+0000 7f6bed59b700 1 --2- 172.21.15.27:0/3720601229 >> v2:172.21.15.77:3300/0 conn(0x7f6be806bf10 0x7f6be8159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:30:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.703+0000 7f6bedd9c700 1 --2- 172.21.15.27:0/3720601229 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6be815df40 0x7f6be8068720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:30:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.703+0000 7f6becd9a700 1 --2- 172.21.15.27:0/3720601229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6be80b0a40 0x7f6be815da00 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f6be806f450 tx=0x7f6bd8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b267e1e2baa138cf server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.703+0000 7f6bdffff700 1 -- 172.21.15.27:0/3720601229 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd800b860 con 0x7f6be80b0a40 2024-01-11T08:30:22.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.703+0000 7f6bdffff700 1 -- 172.21.15.27:0/3720601229 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd800b9c0 con 0x7f6be80b0a40 2024-01-11T08:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.703+0000 7f6bdffff700 1 -- 172.21.15.27:0/3720601229 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd800bd80 con 0x7f6be80b0a40 2024-01-11T08:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.704+0000 7f6bef7ff700 1 -- 172.21.15.27:0/3720601229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6be80b0a40 msgr2=0x7f6be815da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.704+0000 7f6bef7ff700 1 --2- 172.21.15.27:0/3720601229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6be80b0a40 0x7f6be815da00 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f6be806f450 tx=0x7f6bd8009250 comp rx=0 tx=0).stop 2024-01-11T08:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.704+0000 7f6bef7ff700 1 -- 172.21.15.27:0/3720601229 shutdown_connections 2024-01-11T08:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.704+0000 7f6bef7ff700 1 --2- 172.21.15.27:0/3720601229 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6be815df40 0x7f6be8068720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.704+0000 7f6bef7ff700 1 --2- 172.21.15.27:0/3720601229 >> v2:172.21.15.77:3300/0 conn(0x7f6be806bf10 0x7f6be8159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.704+0000 7f6bef7ff700 1 --2- 172.21.15.27:0/3720601229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6be80b0a40 0x7f6be815da00 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.704+0000 7f6bef7ff700 1 -- 172.21.15.27:0/3720601229 >> 172.21.15.27:0/3720601229 conn(0x7f6be80ce3c0 msgr2=0x7f6be80ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.704+0000 7f6bef7ff700 1 -- 172.21.15.27:0/3720601229 shutdown_connections 2024-01-11T08:30:22.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.704+0000 7f6bef7ff700 1 -- 172.21.15.27:0/3720601229 wait complete. 2024-01-11T08:30:22.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.706+0000 7f6bef7ff700 1 Processor -- start 2024-01-11T08:30:22.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.706+0000 7f6bef7ff700 1 -- start start 2024-01-11T08:30:22.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.707+0000 7f6bef7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6be806bf10 0x7f6be8152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.707+0000 7f6bef7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be80b0a40 0x7f6be80c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.707+0000 7f6bef7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6be815df40 0x7f6be80c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.707+0000 7f6bef7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be80c5f90 con 0x7f6be806bf10 2024-01-11T08:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.707+0000 7f6bef7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be8153180 con 0x7f6be80b0a40 2024-01-11T08:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.707+0000 7f6bef7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be80c3e80 con 0x7f6be815df40 2024-01-11T08:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.708+0000 7f6bed59b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6be806bf10 0x7f6be8152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.708+0000 7f6bed59b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6be806bf10 0x7f6be8152c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34980/0 (socket says 172.21.15.27:34980) 2024-01-11T08:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.708+0000 7f6bed59b700 1 -- 172.21.15.27:0/2380580861 learned_addr learned my addr 172.21.15.27:0/2380580861 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:22.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.708+0000 7f6becd9a700 1 --2- 172.21.15.27:0/2380580861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be80b0a40 0x7f6be80c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.708+0000 7f6bedd9c700 1 --2- 172.21.15.27:0/2380580861 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6be815df40 0x7f6be80c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.708+0000 7f6bed59b700 1 -- 172.21.15.27:0/2380580861 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6be815df40 msgr2=0x7f6be80c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.708+0000 7f6bed59b700 1 --2- 172.21.15.27:0/2380580861 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6be815df40 0x7f6be80c8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.708+0000 7f6bed59b700 1 -- 172.21.15.27:0/2380580861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be80b0a40 msgr2=0x7f6be80c34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.708+0000 7f6bed59b700 1 --2- 172.21.15.27:0/2380580861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be80b0a40 0x7f6be80c34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.708+0000 7f6bed59b700 1 -- 172.21.15.27:0/2380580861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd80094f0 con 0x7f6be806bf10 2024-01-11T08:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.708+0000 7f6becd9a700 1 --2- 172.21.15.27:0/2380580861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be80b0a40 0x7f6be80c34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.709+0000 7f6bed59b700 1 --2- 172.21.15.27:0/2380580861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6be806bf10 0x7f6be8152c40 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f6be400c880 tx=0x7f6be4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:22.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.709+0000 7f6bdd7fa700 1 -- 172.21.15.27:0/2380580861 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be4011600 con 0x7f6be806bf10 2024-01-11T08:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.709+0000 7f6bdd7fa700 1 -- 172.21.15.27:0/2380580861 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be4011760 con 0x7f6be806bf10 2024-01-11T08:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.709+0000 7f6bef7ff700 1 -- 172.21.15.27:0/2380580861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be80c4160 con 0x7f6be806bf10 2024-01-11T08:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.709+0000 7f6bdd7fa700 1 -- 172.21.15.27:0/2380580861 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be4011b20 con 0x7f6be806bf10 2024-01-11T08:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.709+0000 7f6bef7ff700 1 -- 172.21.15.27:0/2380580861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be80c61c0 con 0x7f6be806bf10 2024-01-11T08:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.711+0000 7f6bdd7fa700 1 -- 172.21.15.27:0/2380580861 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6be40193b0 con 0x7f6be806bf10 2024-01-11T08:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.712+0000 7f6bdd7fa700 1 --2- 172.21.15.27:0/2380580861 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6bcc066f30 0x7f6bcc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.712+0000 7f6bdd7fa700 1 -- 172.21.15.27:0/2380580861 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f6be4084580 con 0x7f6be806bf10 2024-01-11T08:30:22.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.712+0000 7f6becd9a700 1 --2- 172.21.15.27:0/2380580861 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6bcc066f30 0x7f6bcc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.713+0000 7f6becd9a700 1 --2- 172.21.15.27:0/2380580861 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6bcc066f30 0x7f6bcc0693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6be806f450 tx=0x7f6bd8017000 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:22.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.714+0000 7f6bfdefeb80 1 Processor -- start 2024-01-11T08:30:22.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.714+0000 7f6bfdefeb80 1 -- start start 2024-01-11T08:30:22.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.715+0000 7f6bfdefeb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa0833b20 0x558fa0814df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.715+0000 7f6bfdefeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa080f3a0 0x558fa083b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.715+0000 7f6bfdefeb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa0833ef0 0x558fa083bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.715+0000 7f6bfdefeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa072c310 con 0x558fa080f3a0 2024-01-11T08:30:22.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.715+0000 7f6bfdefeb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa0769130 con 0x558fa0833b20 2024-01-11T08:30:22.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.715+0000 7f6bfdefeb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x558fa0766af0 con 0x558fa0833ef0 2024-01-11T08:30:22.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.715+0000 7f6becd9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa080f3a0 0x558fa083b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.715+0000 7f6becd9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa080f3a0 0x558fa083b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34994/0 (socket says 172.21.15.27:34994) 2024-01-11T08:30:22.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.715+0000 7f6becd9a700 1 -- 172.21.15.27:0/2315307706 learned_addr learned my addr 172.21.15.27:0/2315307706 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:22.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.715+0000 7f6bed59b700 1 --2- 172.21.15.27:0/2315307706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa0833b20 0x558fa0814df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.715+0000 7f6bedd9c700 1 --2- 172.21.15.27:0/2315307706 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa0833ef0 0x558fa083bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.715+0000 7f6becd9a700 1 -- 172.21.15.27:0/2315307706 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa0833ef0 msgr2=0x558fa083bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:22.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.715+0000 7f6becd9a700 1 --2- 172.21.15.27:0/2315307706 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa0833ef0 0x558fa083bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.715+0000 7f6becd9a700 1 -- 172.21.15.27:0/2315307706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa0833b20 msgr2=0x558fa0814df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:22.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.715+0000 7f6becd9a700 1 --2- 172.21.15.27:0/2315307706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa0833b20 0x558fa0814df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.715+0000 7f6becd9a700 1 -- 172.21.15.27:0/2315307706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fa06550a0 con 0x558fa080f3a0 2024-01-11T08:30:22.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.716+0000 7f6becd9a700 1 --2- 172.21.15.27:0/2315307706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa080f3a0 0x558fa083b860 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f6bd800b3c0 tx=0x7f6bd8020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e0179ae4db2ce8e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:22.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.716+0000 7f6bd27fc700 1 -- 172.21.15.27:0/2315307706 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd802c070 con 0x558fa080f3a0 2024-01-11T08:30:22.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.716+0000 7f6bd27fc700 1 -- 172.21.15.27:0/2315307706 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd8011b90 con 0x558fa080f3a0 2024-01-11T08:30:22.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.716+0000 7f6bd27fc700 1 -- 172.21.15.27:0/2315307706 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8020070 con 0x558fa080f3a0 2024-01-11T08:30:22.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.716+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/2315307706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa080f3a0 msgr2=0x558fa083b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:22.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.717+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/2315307706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa080f3a0 0x558fa083b860 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f6bd800b3c0 tx=0x7f6bd8020000 comp rx=0 tx=0).stop 2024-01-11T08:30:22.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.717+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/2315307706 shutdown_connections 2024-01-11T08:30:22.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.717+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/2315307706 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa0833ef0 0x558fa083bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.717+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/2315307706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa0833b20 0x558fa0814df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.717+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/2315307706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa080f3a0 0x558fa083b860 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.717+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/2315307706 >> 172.21.15.27:0/2315307706 conn(0x558fa0811300 msgr2=0x558fa0813bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:22.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.717+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/2315307706 shutdown_connections 2024-01-11T08:30:22.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.717+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/2315307706 wait complete. 2024-01-11T08:30:22.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.717+0000 7f6bfdefeb80 1 -- start start 2024-01-11T08:30:22.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.718+0000 7f6bfdefeb80 1 Processor -- start 2024-01-11T08:30:22.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.719+0000 7f6bfdefeb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa080f3a0 0x558fa0832b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.719+0000 7f6bfdefeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa0833b20 0x558fa0835690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.719+0000 7f6bfdefeb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa0833ef0 0x558fa0868a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.719+0000 7f6bfdefeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa07628e0 con 0x558fa0833b20 2024-01-11T08:30:22.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.719+0000 7f6bfdefeb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa070e980 con 0x558fa080f3a0 2024-01-11T08:30:22.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.719+0000 7f6bfdefeb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x558fa070e620 con 0x558fa0833ef0 2024-01-11T08:30:22.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.719+0000 7f6bfdefeb80 10 client.0 _ll_register_callbacks cb 0x558fa0571a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T08:30:22.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.719+0000 7f6becd9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa0833b20 0x558fa0835690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.719+0000 7f6becd9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa0833b20 0x558fa0835690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34996/0 (socket says 172.21.15.27:34996) 2024-01-11T08:30:22.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.719+0000 7f6becd9a700 1 -- 172.21.15.27:0/322878421 learned_addr learned my addr 172.21.15.27:0/322878421 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:22.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.719+0000 7f6bedd9c700 1 --2- 172.21.15.27:0/322878421 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa0833ef0 0x558fa0868a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.719+0000 7f6becd9a700 1 -- 172.21.15.27:0/322878421 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa0833ef0 msgr2=0x558fa0868a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:22.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.719+0000 7f6bed59b700 1 --2- 172.21.15.27:0/322878421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa080f3a0 0x558fa0832b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.719+0000 7f6becd9a700 1 --2- 172.21.15.27:0/322878421 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa0833ef0 0x558fa0868a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.719+0000 7f6becd9a700 1 -- 172.21.15.27:0/322878421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa080f3a0 msgr2=0x558fa0832b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:22.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.719+0000 7f6becd9a700 1 --2- 172.21.15.27:0/322878421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa080f3a0 0x558fa0832b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.719+0000 7f6becd9a700 1 -- 172.21.15.27:0/322878421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be815a510 con 0x558fa0833b20 2024-01-11T08:30:22.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.719+0000 7f6bed59b700 1 --2- 172.21.15.27:0/322878421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa080f3a0 0x558fa0832b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:30:22.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.719+0000 7f6bedd9c700 1 --2- 172.21.15.27:0/322878421 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa0833ef0 0x558fa0868a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:30:22.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.720+0000 7f6becd9a700 1 --2- 172.21.15.27:0/322878421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa0833b20 0x558fa0835690 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f6bd800c010 tx=0x7f6bd8029070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bfedee7a3bd95d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:22.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.720+0000 7f6bc77fe700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T08:30:22.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.720+0000 7f6bc77fe700 1 -- 172.21.15.27:0/322878421 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd802c070 con 0x558fa0833b20 2024-01-11T08:30:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.720+0000 7f6bc77fe700 1 -- 172.21.15.27:0/322878421 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(145..145 src has 1..145) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f6bd8031c70 con 0x558fa0833b20 2024-01-11T08:30:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.720+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/322878421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558fa0807560 con 0x558fa0833b20 2024-01-11T08:30:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.720+0000 7f6bc77fe700 1 -- 172.21.15.27:0/322878421 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6bd8020040 con 0x558fa0833b20 2024-01-11T08:30:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.720+0000 7f6bfdefeb80 10 client.8745 fetch_fsmap learned FSMap version 77 2024-01-11T08:30:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.720+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/322878421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x558fa07313c0 con 0x558fa0833b20 2024-01-11T08:30:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.720+0000 7f6bc77fe700 1 -- 172.21.15.27:0/322878421 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558fa0807560 con 0x558fa0833b20 2024-01-11T08:30:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.720+0000 7f6bc77fe700 1 -- 172.21.15.27:0/322878421 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 77) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f6bd803c670 con 0x558fa0833b20 2024-01-11T08:30:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.720+0000 7f6bfdefeb80 10 client.8745 fetch_fsmap finished waiting for FSMap version 77 2024-01-11T08:30:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.721+0000 7f6bfdefeb80 10 client.8745 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-11T08:30:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.721+0000 7f6bfdefeb80 20 client.8745 populate_metadata read hostname 'smithi027' 2024-01-11T08:30:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.721+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/322878421 >> [v2:172.21.15.77:6833/1641163931,v1:172.21.15.77:6837/1641163931] conn(0x558fa0842600 0x558fa083ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.721+0000 7f6bfdefeb80 4 client.8745 mds_command: new command op to 8601 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-11T08:30:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.721+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/322878421 --> [v2:172.21.15.77:6833/1641163931,v1:172.21.15.77:6837/1641163931] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558fa07e40e0 con 0x558fa0842600 2024-01-11T08:30:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.721+0000 7f6bed59b700 1 --2- 172.21.15.27:0/322878421 >> [v2:172.21.15.77:6833/1641163931,v1:172.21.15.77:6837/1641163931] conn(0x558fa0842600 0x558fa083ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.721+0000 7f6bed59b700 1 --2- 172.21.15.27:0/322878421 >> [v2:172.21.15.77:6833/1641163931,v1:172.21.15.77:6837/1641163931] conn(0x558fa0842600 0x558fa083ea10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89eae65491c62af0 server_cookie=52843a42840a39f6 in_seq=0 out_seq=0 2024-01-11T08:30:22.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.721+0000 7f6bc77fe700 10 client.8745 ms_handle_connect on v2:172.21.15.77:6833/1641163931 2024-01-11T08:30:22.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.722+0000 7f6bc77fe700 1 -- 172.21.15.27:0/322878421 <== mds.0 v2:172.21.15.77:6833/1641163931 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x558fa07e40e0 con 0x558fa0842600 2024-01-11T08:30:22.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.722+0000 7f6bc77fe700 10 client.8745 handle_command_reply: tid=0 2024-01-11T08:30:22.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.723+0000 7f6bfdefeb80 1 client.8745 shutdown 2024-01-11T08:30:22.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.723+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/322878421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa0833b20 msgr2=0x558fa0835690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.723+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/322878421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa0833b20 0x558fa0835690 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f6bd800c010 tx=0x7f6bd8029070 comp rx=0 tx=0).stop 2024-01-11T08:30:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.723+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/322878421 shutdown_connections 2024-01-11T08:30:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.723+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/322878421 >> [v2:172.21.15.77:6833/1641163931,v1:172.21.15.77:6837/1641163931] conn(0x558fa0842600 0x558fa083ea10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.723+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/322878421 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa0833ef0 0x558fa0868a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.723+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/322878421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa080f3a0 0x558fa0832b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.723+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/322878421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa0833b20 0x558fa0835690 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.723+0000 7f6bc77fe700 0 client.8745 ms_handle_reset on v2:172.21.15.77:6833/1641163931 2024-01-11T08:30:22.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.723+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/322878421 >> 172.21.15.27:0/322878421 conn(0x558fa0811300 msgr2=0x558fa0833200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:22.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.723+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/322878421 shutdown_connections 2024-01-11T08:30:22.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.723+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/322878421 wait complete. 2024-01-11T08:30:22.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.723+0000 7f6bfdefeb80 20 client.8745 trim_cache size 0 max 16384 2024-01-11T08:30:22.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.741+0000 7f6bfdefeb80 1 Processor -- start 2024-01-11T08:30:22.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.741+0000 7f6bfdefeb80 1 -- start start 2024-01-11T08:30:22.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.742+0000 7f6bfdefeb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa080f3a0 0x558fa083e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.742+0000 7f6bfdefeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa0833b20 0x558fa083ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.742+0000 7f6bfdefeb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa0833ef0 0x558fa0867fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.742+0000 7f6bfdefeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa05fa1e0 con 0x558fa0833b20 2024-01-11T08:30:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.742+0000 7f6bfdefeb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa0831c10 con 0x558fa0833ef0 2024-01-11T08:30:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.742+0000 7f6bfdefeb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x558fa08368e0 con 0x558fa080f3a0 2024-01-11T08:30:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.742+0000 7f6becd9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa0833b20 0x558fa083ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.742+0000 7f6becd9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa0833b20 0x558fa083ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34998/0 (socket says 172.21.15.27:34998) 2024-01-11T08:30:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.742+0000 7f6becd9a700 1 -- 172.21.15.27:0/3283675776 learned_addr learned my addr 172.21.15.27:0/3283675776 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.742+0000 7f6bedd9c700 1 --2- 172.21.15.27:0/3283675776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa0833ef0 0x558fa0867fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.742+0000 7f6bed59b700 1 --2- 172.21.15.27:0/3283675776 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa080f3a0 0x558fa083e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.742+0000 7f6becd9a700 1 -- 172.21.15.27:0/3283675776 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa080f3a0 msgr2=0x558fa083e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:22.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.742+0000 7f6becd9a700 1 --2- 172.21.15.27:0/3283675776 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa080f3a0 0x558fa083e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.742+0000 7f6becd9a700 1 -- 172.21.15.27:0/3283675776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa0833ef0 msgr2=0x558fa0867fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:22.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.742+0000 7f6becd9a700 1 --2- 172.21.15.27:0/3283675776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa0833ef0 0x558fa0867fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.742+0000 7f6becd9a700 1 -- 172.21.15.27:0/3283675776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fa05c91d0 con 0x558fa0833b20 2024-01-11T08:30:22.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.742+0000 7f6bed59b700 1 --2- 172.21.15.27:0/3283675776 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa080f3a0 0x558fa083e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T08:30:22.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.743+0000 7f6becd9a700 1 --2- 172.21.15.27:0/3283675776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa0833b20 0x558fa083ef30 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8011f90 tx=0x7f6bd802c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a053e8b3485d32b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:22.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.743+0000 7f6bd2ffd700 1 -- 172.21.15.27:0/3283675776 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8020040 con 0x558fa0833b20 2024-01-11T08:30:22.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.743+0000 7f6bd2ffd700 1 -- 172.21.15.27:0/3283675776 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd80318e0 con 0x558fa0833b20 2024-01-11T08:30:22.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.743+0000 7f6bd2ffd700 1 -- 172.21.15.27:0/3283675776 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd80119c0 con 0x558fa0833b20 2024-01-11T08:30:22.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.743+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/3283675776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa0833b20 msgr2=0x558fa083ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:22.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.743+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/3283675776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa0833b20 0x558fa083ef30 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8011f90 tx=0x7f6bd802c000 comp rx=0 tx=0).stop 2024-01-11T08:30:22.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.743+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/3283675776 shutdown_connections 2024-01-11T08:30:22.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.743+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/3283675776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa0833ef0 0x558fa0867fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.743+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/3283675776 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa080f3a0 0x558fa083e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.743+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/3283675776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa0833b20 0x558fa083ef30 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.743+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/3283675776 >> 172.21.15.27:0/3283675776 conn(0x558fa0811300 msgr2=0x558fa083b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:22.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.743+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/3283675776 shutdown_connections 2024-01-11T08:30:22.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.743+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/3283675776 wait complete. 2024-01-11T08:30:22.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.744+0000 7f6bfdefeb80 1 -- start start 2024-01-11T08:30:22.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.744+0000 7f6bfdefeb80 1 Processor -- start 2024-01-11T08:30:22.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.746+0000 7f6bfdefeb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa080f3a0 0x558fa083d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.746+0000 7f6bfdefeb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa0833b20 0x558fa0839cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.746+0000 7f6bfdefeb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa0833ef0 0x558fa08bdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.746+0000 7f6bfdefeb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa08be760 con 0x558fa080f3a0 2024-01-11T08:30:22.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.746+0000 7f6bfdefeb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa0812930 con 0x558fa0833b20 2024-01-11T08:30:22.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.746+0000 7f6bfdefeb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x558fa08c1380 con 0x558fa0833ef0 2024-01-11T08:30:22.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.746+0000 7f6bfdefeb80 10 client.0 _ll_register_callbacks cb 0x558fa0571a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T08:30:22.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.746+0000 7f6bed59b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa080f3a0 0x558fa083d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.746+0000 7f6bed59b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa080f3a0 0x558fa083d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35014/0 (socket says 172.21.15.27:35014) 2024-01-11T08:30:22.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.746+0000 7f6bed59b700 1 -- 172.21.15.27:0/2951557196 learned_addr learned my addr 172.21.15.27:0/2951557196 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:22.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.747+0000 7f6bedd9c700 1 --2- 172.21.15.27:0/2951557196 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa0833ef0 0x558fa08bdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.747+0000 7f6becd9a700 1 --2- 172.21.15.27:0/2951557196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa0833b20 0x558fa0839cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.747+0000 7f6bed59b700 1 -- 172.21.15.27:0/2951557196 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa0833ef0 msgr2=0x558fa08bdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:22.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.747+0000 7f6bed59b700 1 --2- 172.21.15.27:0/2951557196 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa0833ef0 0x558fa08bdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.747+0000 7f6bed59b700 1 -- 172.21.15.27:0/2951557196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa0833b20 msgr2=0x558fa0839cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:22.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.747+0000 7f6bed59b700 1 --2- 172.21.15.27:0/2951557196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa0833b20 0x558fa0839cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.747+0000 7f6bed59b700 1 -- 172.21.15.27:0/2951557196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558fa07313c0 con 0x558fa080f3a0 2024-01-11T08:30:22.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.747+0000 7f6bed59b700 1 --2- 172.21.15.27:0/2951557196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa080f3a0 0x558fa083d6e0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f6be4051970 tx=0x7f6be4084040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13823f1749ef98f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:22.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.747+0000 7f6bc77fe700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T08:30:22.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.747+0000 7f6bc77fe700 1 -- 172.21.15.27:0/2951557196 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be4055b70 con 0x558fa080f3a0 2024-01-11T08:30:22.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.747+0000 7f6bc77fe700 1 -- 172.21.15.27:0/2951557196 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(145..145 src has 1..145) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f6be4053810 con 0x558fa080f3a0 2024-01-11T08:30:22.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.747+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/2951557196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558fa07058d0 con 0x558fa080f3a0 2024-01-11T08:30:22.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.747+0000 7f6bc77fe700 1 -- 172.21.15.27:0/2951557196 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6be4054b70 con 0x558fa080f3a0 2024-01-11T08:30:22.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.747+0000 7f6bfdefeb80 10 client.8751 fetch_fsmap learned FSMap version 77 2024-01-11T08:30:22.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.747+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/2951557196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x558fa05a2d60 con 0x558fa080f3a0 2024-01-11T08:30:22.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.748+0000 7f6bc77fe700 1 -- 172.21.15.27:0/2951557196 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558fa07058d0 con 0x558fa080f3a0 2024-01-11T08:30:22.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.748+0000 7f6bc77fe700 1 -- 172.21.15.27:0/2951557196 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 77) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f6be405bf80 con 0x558fa080f3a0 2024-01-11T08:30:22.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.748+0000 7f6bfdefeb80 10 client.8751 fetch_fsmap finished waiting for FSMap version 77 2024-01-11T08:30:22.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.748+0000 7f6bfdefeb80 10 client.8751 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-11T08:30:22.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.748+0000 7f6bfdefeb80 20 client.8751 populate_metadata read hostname 'smithi027' 2024-01-11T08:30:22.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.748+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/2951557196 >> [v2:172.21.15.77:6833/1641163931,v1:172.21.15.77:6837/1641163931] conn(0x558fa0867fe0 0x558fa08683b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.748+0000 7f6bfdefeb80 4 client.8751 mds_command: new command op to 8601 tid=0[{"prefix": "status"}] 2024-01-11T08:30:22.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.748+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/2951557196 --> [v2:172.21.15.77:6833/1641163931,v1:172.21.15.77:6837/1641163931] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558fa05c3ef0 con 0x558fa0867fe0 2024-01-11T08:30:22.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.748+0000 7f6becd9a700 1 --2- 172.21.15.27:0/2951557196 >> [v2:172.21.15.77:6833/1641163931,v1:172.21.15.77:6837/1641163931] conn(0x558fa0867fe0 0x558fa08683b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.749+0000 7f6becd9a700 1 --2- 172.21.15.27:0/2951557196 >> [v2:172.21.15.77:6833/1641163931,v1:172.21.15.77:6837/1641163931] conn(0x558fa0867fe0 0x558fa08683b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d4d1053c7a3cf1e server_cookie=de8c7e363935d406 in_seq=0 out_seq=0 2024-01-11T08:30:22.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.749+0000 7f6bc77fe700 10 client.8751 ms_handle_connect on v2:172.21.15.77:6833/1641163931 2024-01-11T08:30:22.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.749+0000 7f6bc77fe700 1 -- 172.21.15.27:0/2951557196 <== mds.0 v2:172.21.15.77:6833/1641163931 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x558fa05c3ef0 con 0x558fa0867fe0 2024-01-11T08:30:22.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.749+0000 7f6bc77fe700 10 client.8751 handle_command_reply: tid=0 2024-01-11T08:30:22.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.749+0000 7f6bfdefeb80 1 client.8751 shutdown 2024-01-11T08:30:22.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.749+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/2951557196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa080f3a0 msgr2=0x558fa083d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:22.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.749+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/2951557196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa080f3a0 0x558fa083d6e0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f6be4051970 tx=0x7f6be4084040 comp rx=0 tx=0).stop 2024-01-11T08:30:22.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.749+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/2951557196 shutdown_connections 2024-01-11T08:30:22.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.749+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/2951557196 >> [v2:172.21.15.77:6833/1641163931,v1:172.21.15.77:6837/1641163931] conn(0x558fa0867fe0 0x558fa08683b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.749+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/2951557196 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x558fa0833ef0 0x558fa08bdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.749+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/2951557196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558fa080f3a0 0x558fa083d6e0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.749+0000 7f6bc77fe700 0 client.8751 ms_handle_reset on v2:172.21.15.77:6833/1641163931 2024-01-11T08:30:22.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.749+0000 7f6bfdefeb80 1 --2- 172.21.15.27:0/2951557196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558fa0833b20 0x558fa0839cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.749+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/2951557196 >> 172.21.15.27:0/2951557196 conn(0x558fa0811300 msgr2=0x558fa0832b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:22.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.749+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/2951557196 shutdown_connections 2024-01-11T08:30:22.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.749+0000 7f6bfdefeb80 1 -- 172.21.15.27:0/2951557196 wait complete. 2024-01-11T08:30:22.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.750+0000 7f6bfdefeb80 20 client.8751 trim_cache size 0 max 16384 2024-01-11T08:30:22.767 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T08:30:22.767 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "8fddb4a7-9d66-40f1-892d-6fe86c728ee4", 2024-01-11T08:30:22.768 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-11T08:30:22.768 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 8601, 2024-01-11T08:30:22.768 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-11T08:30:22.768 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-11T08:30:22.768 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-11T08:30:22.768 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 6.6153645220000001, 2024-01-11T08:30:22.769 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 77, 2024-01-11T08:30:22.769 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 145, 2024-01-11T08:30:22.769 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 145, 2024-01-11T08:30:22.769 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 13.292853023999999 2024-01-11T08:30:22.769 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T08:30:22.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.750+0000 7f6bef7ff700 1 -- 172.21.15.27:0/2380580861 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6bcc066f30 msgr2=0x7f6bcc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:22.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.750+0000 7f6bef7ff700 1 --2- 172.21.15.27:0/2380580861 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6bcc066f30 0x7f6bcc0693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6be806f450 tx=0x7f6bd8017000 comp rx=0 tx=0).stop 2024-01-11T08:30:22.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.750+0000 7f6bef7ff700 1 -- 172.21.15.27:0/2380580861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6be806bf10 msgr2=0x7f6be8152c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:22.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.750+0000 7f6bef7ff700 1 --2- 172.21.15.27:0/2380580861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6be806bf10 0x7f6be8152c40 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f6be400c880 tx=0x7f6be4008040 comp rx=0 tx=0).stop 2024-01-11T08:30:22.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.750+0000 7f6bef7ff700 1 -- 172.21.15.27:0/2380580861 shutdown_connections 2024-01-11T08:30:22.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.750+0000 7f6bef7ff700 1 --2- 172.21.15.27:0/2380580861 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6bcc066f30 0x7f6bcc0693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.750+0000 7f6bef7ff700 1 --2- 172.21.15.27:0/2380580861 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6be815df40 0x7f6be80c8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.750+0000 7f6bef7ff700 1 --2- 172.21.15.27:0/2380580861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6be806bf10 0x7f6be8152c40 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.750+0000 7f6bef7ff700 1 --2- 172.21.15.27:0/2380580861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be80b0a40 0x7f6be80c34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.750+0000 7f6bef7ff700 1 -- 172.21.15.27:0/2380580861 >> 172.21.15.27:0/2380580861 conn(0x7f6be80ce3c0 msgr2=0x7f6be815ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:22.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.750+0000 7f6bef7ff700 1 -- 172.21.15.27:0/2380580861 shutdown_connections 2024-01-11T08:30:22.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.750+0000 7f6bef7ff700 1 -- 172.21.15.27:0/2380580861 wait complete. 2024-01-11T08:30:22.774 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-11T08:30:22.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.976+0000 7fd0de7ff700 1 Processor -- start 2024-01-11T08:30:22.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.976+0000 7fd0de7ff700 1 -- start start 2024-01-11T08:30:22.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.977+0000 7fd0de7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd0d81575e0 0x7fd0d8158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.977+0000 7fd0de7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d8158b60 con 0x7fd0d80b19d0 2024-01-11T08:30:22.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.977+0000 7fd0de7ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d81599b0 con 0x7fd0d806b610 2024-01-11T08:30:22.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.977+0000 7fd0de7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d80cb5e0 con 0x7fd0d81575e0 2024-01-11T08:30:22.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.978+0000 7fd0dcd9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd0d80b19d0 0x7fd0d8159e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38744/0 (socket says 172.21.15.27:38744) 2024-01-11T08:30:22.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.978+0000 7fd0dcd9c700 1 -- 172.21.15.27:0/3470600544 learned_addr learned my addr 172.21.15.27:0/3470600544 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:30:22.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.978+0000 7fd0d7fff700 1 --2- 172.21.15.27:0/3470600544 >> v2:172.21.15.77:3300/0 conn(0x7fd0d81575e0 0x7fd0d8158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.978+0000 7fd0d6ffd700 1 -- 172.21.15.27:0/3470600544 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081689380 0 0) 0x7fd0d0001490 con 0x7fd0d80b19d0 2024-01-11T08:30:22.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.978+0000 7fd0d6ffd700 1 -- 172.21.15.27:0/3470600544 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0c0003570 con 0x7fd0d80b19d0 2024-01-11T08:30:22.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.978+0000 7fd0d7fff700 1 -- 172.21.15.27:0/3470600544 >> v1:172.21.15.77:6789/0 conn(0x7fd0d806b610 legacy=0x7fd0d80b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:22.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.978+0000 7fd0d7fff700 1 -- 172.21.15.27:0/3470600544 >> v1:172.21.15.27:6789/0 conn(0x7fd0d80b19d0 legacy=0x7fd0d8159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:22.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.978+0000 7fd0d7fff700 1 -- 172.21.15.27:0/3470600544 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d80c0f90 con 0x7fd0d81575e0 2024-01-11T08:30:22.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.979+0000 7fd0d7fff700 1 --2- 172.21.15.27:0/3470600544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0d81575e0 0x7fd0d8158620 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd0c80098e0 tx=0x7fd0c80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a0709175fb0661d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:22.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.979+0000 7fd0d6ffd700 1 -- 172.21.15.27:0/3470600544 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c800b640 con 0x7fd0d81575e0 2024-01-11T08:30:22.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.979+0000 7fd0d6ffd700 1 -- 172.21.15.27:0/3470600544 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c800b7a0 con 0x7fd0d81575e0 2024-01-11T08:30:22.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.979+0000 7fd0d6ffd700 1 -- 172.21.15.27:0/3470600544 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c800bb40 con 0x7fd0d81575e0 2024-01-11T08:30:22.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.980+0000 7fd0de7ff700 1 -- 172.21.15.27:0/3470600544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0d81575e0 msgr2=0x7fd0d8158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:22.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.980+0000 7fd0de7ff700 1 --2- 172.21.15.27:0/3470600544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0d81575e0 0x7fd0d8158620 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd0c80098e0 tx=0x7fd0c80091a0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.980+0000 7fd0de7ff700 1 -- 172.21.15.27:0/3470600544 shutdown_connections 2024-01-11T08:30:22.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.980+0000 7fd0de7ff700 1 --2- 172.21.15.27:0/3470600544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0d81575e0 0x7fd0d8158620 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.980+0000 7fd0de7ff700 1 -- 172.21.15.27:0/3470600544 >> 172.21.15.27:0/3470600544 conn(0x7fd0d80680e0 msgr2=0x7fd0d8069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:22.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.980+0000 7fd0de7ff700 1 -- 172.21.15.27:0/3470600544 shutdown_connections 2024-01-11T08:30:22.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.980+0000 7fd0de7ff700 1 -- 172.21.15.27:0/3470600544 wait complete. 2024-01-11T08:30:22.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.981+0000 7fd0de7ff700 1 Processor -- start 2024-01-11T08:30:22.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.981+0000 7fd0de7ff700 1 -- start start 2024-01-11T08:30:22.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.982+0000 7fd0de7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d806b610 0x7fd0d8146dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.982+0000 7fd0de7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0d80b19d0 0x7fd0d80c84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.982+0000 7fd0de7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0d81575e0 0x7fd0d806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.982+0000 7fd0de7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d8154340 con 0x7fd0d806b610 2024-01-11T08:30:22.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.982+0000 7fd0de7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d80c5c80 con 0x7fd0d80b19d0 2024-01-11T08:30:22.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.982+0000 7fd0de7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d8147300 con 0x7fd0d81575e0 2024-01-11T08:30:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.982+0000 7fd0d7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d806b610 0x7fd0d8146dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.983+0000 7fd0d7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d806b610 0x7fd0d8146dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35028/0 (socket says 172.21.15.27:35028) 2024-01-11T08:30:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.983+0000 7fd0d7fff700 1 -- 172.21.15.27:0/2035211169 learned_addr learned my addr 172.21.15.27:0/2035211169 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.983+0000 7fd0dcd9c700 1 --2- 172.21.15.27:0/2035211169 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0d81575e0 0x7fd0d806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.983+0000 7fd0d77fe700 1 --2- 172.21.15.27:0/2035211169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0d80b19d0 0x7fd0d80c84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.983+0000 7fd0d7fff700 1 -- 172.21.15.27:0/2035211169 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0d81575e0 msgr2=0x7fd0d806db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.983+0000 7fd0d7fff700 1 --2- 172.21.15.27:0/2035211169 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0d81575e0 0x7fd0d806db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.983+0000 7fd0d7fff700 1 -- 172.21.15.27:0/2035211169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0d80b19d0 msgr2=0x7fd0d80c84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.983+0000 7fd0d7fff700 1 --2- 172.21.15.27:0/2035211169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0d80b19d0 0x7fd0d80c84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.983+0000 7fd0d7fff700 1 -- 172.21.15.27:0/2035211169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c8009420 con 0x7fd0d806b610 2024-01-11T08:30:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.983+0000 7fd0d77fe700 1 --2- 172.21.15.27:0/2035211169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0d80b19d0 0x7fd0d80c84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:30:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.983+0000 7fd0dcd9c700 1 --2- 172.21.15.27:0/2035211169 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0d81575e0 0x7fd0d806db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:30:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.983+0000 7fd0d7fff700 1 --2- 172.21.15.27:0/2035211169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d806b610 0x7fd0d8146dc0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8066180 tx=0x7fd0c8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:22.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.983+0000 7fd0bffff700 1 -- 172.21.15.27:0/2035211169 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c800bb20 con 0x7fd0d806b610 2024-01-11T08:30:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.984+0000 7fd0bffff700 1 -- 172.21.15.27:0/2035211169 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c8011420 con 0x7fd0d806b610 2024-01-11T08:30:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.984+0000 7fd0bffff700 1 -- 172.21.15.27:0/2035211169 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c80117a0 con 0x7fd0d806b610 2024-01-11T08:30:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.984+0000 7fd0de7ff700 1 -- 172.21.15.27:0/2035211169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d80c59a0 con 0x7fd0d806b610 2024-01-11T08:30:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.984+0000 7fd0de7ff700 1 -- 172.21.15.27:0/2035211169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d816d700 con 0x7fd0d806b610 2024-01-11T08:30:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.985+0000 7fd0de7ff700 1 -- 172.21.15.27:0/2035211169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d80718d0 con 0x7fd0d806b610 2024-01-11T08:30:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.986+0000 7fd0bffff700 1 -- 172.21.15.27:0/2035211169 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd0c8011b10 con 0x7fd0d806b610 2024-01-11T08:30:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.987+0000 7fd0bffff700 1 --2- 172.21.15.27:0/2035211169 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0c4066f30 0x7fd0c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.987+0000 7fd0bffff700 1 -- 172.21.15.27:0/2035211169 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fd0c8085960 con 0x7fd0d806b610 2024-01-11T08:30:22.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.988+0000 7fd0d77fe700 1 --2- 172.21.15.27:0/2035211169 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0c4066f30 0x7fd0c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:22.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.988+0000 7fd0d77fe700 1 --2- 172.21.15.27:0/2035211169 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0c4066f30 0x7fd0c40693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc007fd0 tx=0x7fd0cc009040 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:22.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:22.992+0000 7fd0bffff700 1 -- 172.21.15.27:0/2035211169 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd0c8052c10 con 0x7fd0d806b610 2024-01-11T08:30:23.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:23.244+0000 7fd0de7ff700 1 -- 172.21.15.27:0/2035211169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd0d8158da0 con 0x7fd0d806b610 2024-01-11T08:30:23.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:23.245+0000 7fd0bffff700 1 -- 172.21.15.27:0/2035211169 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v77) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd0c8015060 con 0x7fd0d806b610 2024-01-11T08:30:23.244 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-11T08:30:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:23.247+0000 7fd0de7ff700 1 -- 172.21.15.27:0/2035211169 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0c4066f30 msgr2=0x7fd0c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:23.247+0000 7fd0de7ff700 1 --2- 172.21.15.27:0/2035211169 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0c4066f30 0x7fd0c40693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc007fd0 tx=0x7fd0cc009040 comp rx=0 tx=0).stop 2024-01-11T08:30:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:23.247+0000 7fd0de7ff700 1 -- 172.21.15.27:0/2035211169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d806b610 msgr2=0x7fd0d8146dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:23.247+0000 7fd0de7ff700 1 --2- 172.21.15.27:0/2035211169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d806b610 0x7fd0d8146dc0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8066180 tx=0x7fd0c8017000 comp rx=0 tx=0).stop 2024-01-11T08:30:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:23.247+0000 7fd0de7ff700 1 -- 172.21.15.27:0/2035211169 shutdown_connections 2024-01-11T08:30:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:23.247+0000 7fd0de7ff700 1 --2- 172.21.15.27:0/2035211169 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fd0c4066f30 0x7fd0c40693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:23.247+0000 7fd0de7ff700 1 --2- 172.21.15.27:0/2035211169 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0d81575e0 0x7fd0d806db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:23.247+0000 7fd0de7ff700 1 --2- 172.21.15.27:0/2035211169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d806b610 0x7fd0d8146dc0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:23.247+0000 7fd0de7ff700 1 --2- 172.21.15.27:0/2035211169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0d80b19d0 0x7fd0d80c84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:23.247+0000 7fd0de7ff700 1 -- 172.21.15.27:0/2035211169 >> 172.21.15.27:0/2035211169 conn(0x7fd0d80680e0 msgr2=0x7fd0d8156140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:23.247+0000 7fd0de7ff700 1 -- 172.21.15.27:0/2035211169 shutdown_connections 2024-01-11T08:30:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:23.247+0000 7fd0de7ff700 1 -- 172.21.15.27:0/2035211169 wait complete. 2024-01-11T08:30:23.264 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-11T08:30:23.264 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-11T08:30:23.264 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-11T08:30:23.264 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-11T08:30:23.264 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-11T08:30:23.264 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-11T08:30:23.264 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-11T08:30:23.265 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-11T08:30:23.265 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:30:23.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:30:23.265 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-11T08:30:23.315 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T08:30:23.315 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T08:30:23.315 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T08:30:23.315 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:30:23.315 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-11T08:30:23.315 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:30:23.315 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:30:23.315 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:30:23.315 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-11T08:30:23.315 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:30:23.315 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-11T08:30:23.315 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-11T08:30:23.316 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-11T08:30:23.316 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-11T08:30:23.316 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-11T08:30:23.316 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-11T08:30:23.316 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 41299sec preferred_lft 41299sec 2024-01-11T08:30:23.316 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-11T08:30:23.316 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:30:23.316 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T08:30:23.316 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-11T08:30:23.316 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-11T08:30:23.316 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T08:30:23.316 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff 2024-01-11T08:30:23.317 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-11T08:30:23.317 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:30:23.317 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::acd2:b3ff:fe6a:db4b/64 scope link 2024-01-11T08:30:23.317 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:30:23.317 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T08:30:23.317 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:fa:85:1b:3a:bc brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-11T08:30:23.317 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ecfa:85ff:fe1b:3abc/64 scope link 2024-01-11T08:30:23.317 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:30:23.317 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T08:30:23.317 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-11T08:30:23.317 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ec88:faff:fe17:41aa/64 scope link 2024-01-11T08:30:23.317 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T08:30:23.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:30:23.318 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-11T08:30:23.388 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-11T08:30:23.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:30:23.393 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-11T08:30:23.442 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-11T08:30:23.443 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-11T08:30:23.443 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-11T08:30:23.443 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-11T08:30:23.443 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-11T08:30:23.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:30:23.444 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T08:30:23.444 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T08:30:23.444 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-11T08:30:23.444 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-11T08:30:23.444 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-11T08:30:23.444 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T08:30:23.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:30:23.621 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T08:30:23.638 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:30:23.671 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:30:23.673 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-11T08:30:23.673 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-11T08:30:23.673 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:30:23.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:30:23.673 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-11T08:30:23.788 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:30:23.788 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-11T08:30:23.875 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-11T08:30:23.876 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T08:30:23.876 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-11T08:30:23.923 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-11T08:30:23.923 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-11T08:30:23.923 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:30:24.027 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:30:24.027 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:30:24.028 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:30:24.043 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:30:24.043 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:30:24.043 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:30:24.043 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:30:24.043 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:30:24.065 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T08:30:24.065 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T08:30:24.065 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T08:30:24.065 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T08:30:24.065 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T08:30:24.065 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790" 2024-01-11T08:30:24.065 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790:/ 2024-01-11T08:30:24.066 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=8fddb4a7-9d66-40f1-892d-6fe86c728ee4" will pass to kernel 2024-01-11T08:30:24.071 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-11T08:30:24.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T08:30:24.071 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-11T08:30:24.146 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-11T08:30:24.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.350+0000 7f0eab5ff700 1 Processor -- start 2024-01-11T08:30:24.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.350+0000 7f0eab5ff700 1 -- start start 2024-01-11T08:30:24.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.351+0000 7f0eab5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0ea4068720 0x7f0ea4068b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:24.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.351+0000 7f0eab5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea4069060 con 0x7f0ea40b0a40 2024-01-11T08:30:24.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.351+0000 7f0eab5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea415d290 con 0x7f0ea406bf10 2024-01-11T08:30:24.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.351+0000 7f0eab5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea415d570 con 0x7f0ea4068720 2024-01-11T08:30:24.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.352+0000 7f0ea8b9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0ea40b0a40 0x7f0ea415da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45876/0 (socket says 172.21.15.27:45876) 2024-01-11T08:30:24.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.352+0000 7f0ea8b9a700 1 -- 172.21.15.27:0/2974494258 learned_addr learned my addr 172.21.15.27:0/2974494258 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:30:24.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.352+0000 7f0ea9b9c700 1 --2- 172.21.15.27:0/2974494258 >> v2:172.21.15.77:3300/0 conn(0x7f0ea4068720 0x7f0ea4068b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:24.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.352+0000 7f0e9bfff700 1 -- 172.21.15.27:0/2974494258 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848336991 0 0) 0x7f0e94001110 con 0x7f0ea40b0a40 2024-01-11T08:30:24.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.352+0000 7f0e9bfff700 1 -- 172.21.15.27:0/2974494258 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e90003570 con 0x7f0ea40b0a40 2024-01-11T08:30:24.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.352+0000 7f0ea9b9c700 1 -- 172.21.15.27:0/2974494258 >> v1:172.21.15.77:6789/0 conn(0x7f0ea406bf10 legacy=0x7f0ea4159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:24.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.352+0000 7f0ea9b9c700 1 -- 172.21.15.27:0/2974494258 >> v1:172.21.15.27:6789/0 conn(0x7f0ea40b0a40 legacy=0x7f0ea415da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:24.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.352+0000 7f0e9bfff700 1 -- 172.21.15.27:0/2974494258 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2023891467 0 0) 0x7f0e90003570 con 0x7f0ea40b0a40 2024-01-11T08:30:24.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.352+0000 7f0ea9b9c700 1 -- 172.21.15.27:0/2974494258 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea4154180 con 0x7f0ea4068720 2024-01-11T08:30:24.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.353+0000 7f0ea9b9c700 1 --2- 172.21.15.27:0/2974494258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea4068720 0x7f0ea4068b20 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c009e60 tx=0x7f0e9c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d89bb7104bc6835 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:24.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.353+0000 7f0e9bfff700 1 -- 172.21.15.27:0/2974494258 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c00e070 con 0x7f0ea4068720 2024-01-11T08:30:24.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.353+0000 7f0e9bfff700 1 -- 172.21.15.27:0/2974494258 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9c00b850 con 0x7f0ea4068720 2024-01-11T08:30:24.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.353+0000 7f0e9bfff700 1 -- 172.21.15.27:0/2974494258 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e94001600 con 0x7f0ea4068720 2024-01-11T08:30:24.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.354+0000 7f0eab5ff700 1 -- 172.21.15.27:0/2974494258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea4068720 msgr2=0x7f0ea4068b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:24.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.354+0000 7f0eab5ff700 1 --2- 172.21.15.27:0/2974494258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea4068720 0x7f0ea4068b20 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c009e60 tx=0x7f0e9c009780 comp rx=0 tx=0).stop 2024-01-11T08:30:24.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.354+0000 7f0eab5ff700 1 -- 172.21.15.27:0/2974494258 shutdown_connections 2024-01-11T08:30:24.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.354+0000 7f0eab5ff700 1 --2- 172.21.15.27:0/2974494258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea4068720 0x7f0ea4068b20 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:24.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.354+0000 7f0eab5ff700 1 -- 172.21.15.27:0/2974494258 >> 172.21.15.27:0/2974494258 conn(0x7f0ea40ce3c0 msgr2=0x7f0ea40ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:24.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.354+0000 7f0eab5ff700 1 -- 172.21.15.27:0/2974494258 shutdown_connections 2024-01-11T08:30:24.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.354+0000 7f0eab5ff700 1 -- 172.21.15.27:0/2974494258 wait complete. 2024-01-11T08:30:24.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.355+0000 7f0eab5ff700 1 Processor -- start 2024-01-11T08:30:24.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.355+0000 7f0eab5ff700 1 -- start start 2024-01-11T08:30:24.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.356+0000 7f0eab5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0ea4068720 0x7f0ea415d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:24.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.356+0000 7f0eab5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea406bf10 0x7f0ea40c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:24.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.356+0000 7f0eab5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea40b0a40 0x7f0ea4153840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:24.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.356+0000 7f0eab5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea40c7d50 con 0x7f0ea40b0a40 2024-01-11T08:30:24.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.356+0000 7f0eab5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea415ebb0 con 0x7f0ea406bf10 2024-01-11T08:30:24.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.356+0000 7f0eab5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea415dfa0 con 0x7f0ea4068720 2024-01-11T08:30:24.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.357+0000 7f0ea9b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea40b0a40 0x7f0ea4153840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:24.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.357+0000 7f0ea9b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea40b0a40 0x7f0ea4153840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42732/0 (socket says 172.21.15.27:42732) 2024-01-11T08:30:24.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.357+0000 7f0ea9b9c700 1 -- 172.21.15.27:0/3590926911 learned_addr learned my addr 172.21.15.27:0/3590926911 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:24.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.357+0000 7f0ea8b9a700 1 --2- 172.21.15.27:0/3590926911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea406bf10 0x7f0ea40c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:24.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.357+0000 7f0ea939b700 1 --2- 172.21.15.27:0/3590926911 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0ea4068720 0x7f0ea415d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:24.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.357+0000 7f0ea9b9c700 1 -- 172.21.15.27:0/3590926911 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0ea4068720 msgr2=0x7f0ea415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:24.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.357+0000 7f0ea9b9c700 1 --2- 172.21.15.27:0/3590926911 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0ea4068720 0x7f0ea415d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:24.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.357+0000 7f0ea9b9c700 1 -- 172.21.15.27:0/3590926911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea406bf10 msgr2=0x7f0ea40c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:24.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.357+0000 7f0ea9b9c700 1 --2- 172.21.15.27:0/3590926911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea406bf10 0x7f0ea40c7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:24.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.357+0000 7f0ea9b9c700 1 -- 172.21.15.27:0/3590926911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9c009a20 con 0x7f0ea40b0a40 2024-01-11T08:30:24.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.357+0000 7f0ea939b700 1 --2- 172.21.15.27:0/3590926911 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0ea4068720 0x7f0ea415d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T08:30:24.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.357+0000 7f0ea9b9c700 1 --2- 172.21.15.27:0/3590926911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea40b0a40 0x7f0ea4153840 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f0ea406b3c0 tx=0x7f0e9c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:24.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.357+0000 7f0e997fa700 1 -- 172.21.15.27:0/3590926911 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c012cd0 con 0x7f0ea40b0a40 2024-01-11T08:30:24.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.357+0000 7f0e997fa700 1 -- 172.21.15.27:0/3590926911 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9c012e30 con 0x7f0ea40b0a40 2024-01-11T08:30:24.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.357+0000 7f0e997fa700 1 -- 172.21.15.27:0/3590926911 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c01e680 con 0x7f0ea40b0a40 2024-01-11T08:30:24.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.357+0000 7f0eab5ff700 1 -- 172.21.15.27:0/3590926911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea415e280 con 0x7f0ea40b0a40 2024-01-11T08:30:24.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.358+0000 7f0eab5ff700 1 -- 172.21.15.27:0/3590926911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea415e840 con 0x7f0ea40b0a40 2024-01-11T08:30:24.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.359+0000 7f0eab5ff700 1 -- 172.21.15.27:0/3590926911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea406a6d0 con 0x7f0ea40b0a40 2024-01-11T08:30:24.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.360+0000 7f0e997fa700 1 -- 172.21.15.27:0/3590926911 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0e9c01e9f0 con 0x7f0ea40b0a40 2024-01-11T08:30:24.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.360+0000 7f0e997fa700 1 --2- 172.21.15.27:0/3590926911 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0e88066f30 0x7f0e880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:24.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.361+0000 7f0e997fa700 1 -- 172.21.15.27:0/3590926911 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f0e9c086800 con 0x7f0ea40b0a40 2024-01-11T08:30:24.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.361+0000 7f0ea939b700 1 --2- 172.21.15.27:0/3590926911 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0e88066f30 0x7f0e880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:24.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.362+0000 7f0ea939b700 1 --2- 172.21.15.27:0/3590926911 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0e88066f30 0x7f0e880693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0007fd0 tx=0x7f0ea0009040 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:24.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.365+0000 7f0e997fa700 1 -- 172.21.15.27:0/3590926911 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0e9c053990 con 0x7f0ea40b0a40 2024-01-11T08:30:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.601+0000 7f0eab5ff700 1 -- 172.21.15.27:0/3590926911 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ea404c3c0 con 0x7f0e88066f30 2024-01-11T08:30:24.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.604+0000 7f0e997fa700 1 -- 172.21.15.27:0/3590926911 <== mgr.8086 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f0ea404c3c0 con 0x7f0e88066f30 2024-01-11T08:30:24.602 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T08:30:24.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T08:30:24.603 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-11T08:30:24.603 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T08:30:24.603 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T08:30:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.606+0000 7f0eab5ff700 1 -- 172.21.15.27:0/3590926911 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0e88066f30 msgr2=0x7f0e880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.606+0000 7f0eab5ff700 1 --2- 172.21.15.27:0/3590926911 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0e88066f30 0x7f0e880693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0007fd0 tx=0x7f0ea0009040 comp rx=0 tx=0).stop 2024-01-11T08:30:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.606+0000 7f0eab5ff700 1 -- 172.21.15.27:0/3590926911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea40b0a40 msgr2=0x7f0ea4153840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.606+0000 7f0eab5ff700 1 --2- 172.21.15.27:0/3590926911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea40b0a40 0x7f0ea4153840 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f0ea406b3c0 tx=0x7f0e9c017000 comp rx=0 tx=0).stop 2024-01-11T08:30:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.606+0000 7f0eab5ff700 1 -- 172.21.15.27:0/3590926911 shutdown_connections 2024-01-11T08:30:24.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.606+0000 7f0eab5ff700 1 --2- 172.21.15.27:0/3590926911 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f0e88066f30 0x7f0e880693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:24.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.606+0000 7f0eab5ff700 1 --2- 172.21.15.27:0/3590926911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea40b0a40 0x7f0ea4153840 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:24.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.606+0000 7f0eab5ff700 1 --2- 172.21.15.27:0/3590926911 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0ea4068720 0x7f0ea415d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:24.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.606+0000 7f0eab5ff700 1 --2- 172.21.15.27:0/3590926911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ea406bf10 0x7f0ea40c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:24.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.606+0000 7f0eab5ff700 1 -- 172.21.15.27:0/3590926911 >> 172.21.15.27:0/3590926911 conn(0x7f0ea40ce3c0 msgr2=0x7f0ea406d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:24.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.606+0000 7f0eab5ff700 1 -- 172.21.15.27:0/3590926911 shutdown_connections 2024-01-11T08:30:24.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.606+0000 7f0eab5ff700 1 -- 172.21.15.27:0/3590926911 wait complete. 2024-01-11T08:30:24.623 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-11T08:30:24.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.828+0000 7f31b01ff700 1 Processor -- start 2024-01-11T08:30:24.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.829+0000 7f31b01ff700 1 -- start start 2024-01-11T08:30:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.830+0000 7f31b01ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80ccd70 0x7f31a80cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.830+0000 7f31b01ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a80cf850 con 0x7f31a815a1b0 2024-01-11T08:30:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.830+0000 7f31b01ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a80cec40 con 0x7f31a815ae70 2024-01-11T08:30:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.830+0000 7f31b01ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a815ac90 con 0x7f31a80ccd70 2024-01-11T08:30:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.830+0000 7f31adf9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f31a815a1b0 0x7f31a815a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45880/0 (socket says 172.21.15.27:45880) 2024-01-11T08:30:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.830+0000 7f31adf9b700 1 -- 172.21.15.27:0/735638927 learned_addr learned my addr 172.21.15.27:0/735638927 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:30:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.831+0000 7f31acf99700 1 -- 172.21.15.27:0/735638927 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364031526 0 0) 0x7f31a80cf850 con 0x7f31a815a1b0 2024-01-11T08:30:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.831+0000 7f31ad79a700 1 --2- 172.21.15.27:0/735638927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80ccd70 0x7f31a80cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.831+0000 7f31acf99700 1 -- 172.21.15.27:0/735638927 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3194003570 con 0x7f31a815a1b0 2024-01-11T08:30:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.831+0000 7f31acf99700 1 -- 172.21.15.27:0/735638927 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2050336509 0 0) 0x7f3194003570 con 0x7f31a815a1b0 2024-01-11T08:30:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.831+0000 7f31acf99700 1 -- 172.21.15.27:0/735638927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80ccd70 msgr2=0x7f31a80cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.831+0000 7f31acf99700 1 --2- 172.21.15.27:0/735638927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80ccd70 0x7f31a80cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.831+0000 7f31acf99700 1 -- 172.21.15.27:0/735638927 >> v1:172.21.15.77:6789/0 conn(0x7f31a815ae70 legacy=0x7f31a815b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.831+0000 7f31acf99700 1 -- 172.21.15.27:0/735638927 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a80cef20 con 0x7f31a815a1b0 2024-01-11T08:30:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.831+0000 7f31acf99700 1 -- 172.21.15.27:0/735638927 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f31a4001470 con 0x7f31a815a1b0 2024-01-11T08:30:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.831+0000 7f31acf99700 1 -- 172.21.15.27:0/735638927 >> v1:172.21.15.27:6789/0 conn(0x7f31a815a1b0 legacy=0x7f31a815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.831+0000 7f31acf99700 1 --2- 172.21.15.27:0/735638927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80ccd70 0x7f3194003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.831+0000 7f31acf99700 1 --2- 172.21.15.27:0/735638927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3194003fd0 0x7f3194004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.831+0000 7f31acf99700 1 --2- 172.21.15.27:0/735638927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3194004d50 0x7f31940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.831+0000 7f31acf99700 1 -- 172.21.15.27:0/735638927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a815ac90 con 0x7f3194004d50 2024-01-11T08:30:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.831+0000 7f31acf99700 1 -- 172.21.15.27:0/735638927 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319400bb40 con 0x7f3194003fd0 2024-01-11T08:30:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.831+0000 7f31acf99700 1 -- 172.21.15.27:0/735638927 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f319400bd00 con 0x7f31a80ccd70 2024-01-11T08:30:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.832+0000 7f31adf9b700 1 --2- 172.21.15.27:0/735638927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80ccd70 0x7f3194003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.832+0000 7f31ad79a700 1 --2- 172.21.15.27:0/735638927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3194003fd0 0x7f3194004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.832+0000 7f31ae79c700 1 --2- 172.21.15.27:0/735638927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3194004d50 0x7f31940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.832+0000 7f31adf9b700 1 -- 172.21.15.27:0/735638927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3194003fd0 msgr2=0x7f3194004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.832+0000 7f31adf9b700 1 --2- 172.21.15.27:0/735638927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3194003fd0 0x7f3194004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.832+0000 7f31adf9b700 1 -- 172.21.15.27:0/735638927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3194004d50 msgr2=0x7f31940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.832+0000 7f31adf9b700 1 --2- 172.21.15.27:0/735638927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3194004d50 0x7f31940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.832+0000 7f31adf9b700 1 -- 172.21.15.27:0/735638927 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3194003570 con 0x7f31a80ccd70 2024-01-11T08:30:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.832+0000 7f31adf9b700 1 --2- 172.21.15.27:0/735638927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80ccd70 0x7f3194003a90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f31a400c3c0 tx=0x7f31a4001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c42cb003932096a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.833+0000 7f31acf99700 1 -- 172.21.15.27:0/735638927 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a400cc90 con 0x7f31a80ccd70 2024-01-11T08:30:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.833+0000 7f31acf99700 1 -- 172.21.15.27:0/735638927 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a400cdf0 con 0x7f31a80ccd70 2024-01-11T08:30:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.833+0000 7f31acf99700 1 -- 172.21.15.27:0/735638927 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a4005b70 con 0x7f31a80ccd70 2024-01-11T08:30:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.833+0000 7f31b01ff700 1 -- 172.21.15.27:0/735638927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80ccd70 msgr2=0x7f3194003a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.833+0000 7f31b01ff700 1 --2- 172.21.15.27:0/735638927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80ccd70 0x7f3194003a90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f31a400c3c0 tx=0x7f31a4001e70 comp rx=0 tx=0).stop 2024-01-11T08:30:24.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.833+0000 7f31ae79c700 1 -- 172.21.15.27:0/735638927 reap_dead start 2024-01-11T08:30:24.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.833+0000 7f31b01ff700 1 -- 172.21.15.27:0/735638927 shutdown_connections 2024-01-11T08:30:24.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.833+0000 7f31b01ff700 1 -- 172.21.15.27:0/735638927 >> 172.21.15.27:0/735638927 conn(0x7f31a806ed40 msgr2=0x7f31a806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:24.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.833+0000 7f31b01ff700 1 -- 172.21.15.27:0/735638927 shutdown_connections 2024-01-11T08:30:24.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.833+0000 7f31b01ff700 1 -- 172.21.15.27:0/735638927 wait complete. 2024-01-11T08:30:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.835+0000 7f31b01ff700 1 Processor -- start 2024-01-11T08:30:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.835+0000 7f31b01ff700 1 -- start start 2024-01-11T08:30:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.836+0000 7f31b01ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80ccd70 0x7f31a8147f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.836+0000 7f31b01ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a815dd60 0x7f31a815e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.836+0000 7f31b01ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31a80c8f80 0x7f31a80c9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.836+0000 7f31b01ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a815e710 con 0x7f31a80c8f80 2024-01-11T08:30:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.836+0000 7f31b01ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a80c9cf0 con 0x7f31a815dd60 2024-01-11T08:30:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.836+0000 7f31b01ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a80c9fd0 con 0x7f31a80ccd70 2024-01-11T08:30:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.836+0000 7f31ae79c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31a80c8f80 0x7f31a80c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.836+0000 7f31ae79c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31a80c8f80 0x7f31a80c9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42742/0 (socket says 172.21.15.27:42742) 2024-01-11T08:30:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.836+0000 7f31ae79c700 1 -- 172.21.15.27:0/3553217065 learned_addr learned my addr 172.21.15.27:0/3553217065 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.836+0000 7f31ae79c700 1 -- 172.21.15.27:0/3553217065 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80ccd70 msgr2=0x7f31a8147f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.836+0000 7f31ae79c700 1 --2- 172.21.15.27:0/3553217065 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80ccd70 0x7f31a8147f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.836+0000 7f31ae79c700 1 -- 172.21.15.27:0/3553217065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a815dd60 msgr2=0x7f31a815e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.836+0000 7f31ae79c700 1 --2- 172.21.15.27:0/3553217065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a815dd60 0x7f31a815e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.836+0000 7f31ae79c700 1 -- 172.21.15.27:0/3553217065 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a80ca2b0 con 0x7f31a80c8f80 2024-01-11T08:30:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.837+0000 7f31ae79c700 1 --2- 172.21.15.27:0/3553217065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31a80c8f80 0x7f31a80c9420 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f31a8089840 tx=0x7f31a00016d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.837+0000 7f319dffb700 1 -- 172.21.15.27:0/3553217065 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a000c820 con 0x7f31a80c8f80 2024-01-11T08:30:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.837+0000 7f319dffb700 1 -- 172.21.15.27:0/3553217065 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a000c980 con 0x7f31a80c8f80 2024-01-11T08:30:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.837+0000 7f319dffb700 1 -- 172.21.15.27:0/3553217065 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a000cd40 con 0x7f31a80c8f80 2024-01-11T08:30:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.837+0000 7f31b01ff700 1 -- 172.21.15.27:0/3553217065 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a80ca590 con 0x7f31a80c8f80 2024-01-11T08:30:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.837+0000 7f31b01ff700 1 -- 172.21.15.27:0/3553217065 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a80c9960 con 0x7f31a80c8f80 2024-01-11T08:30:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.838+0000 7f31b01ff700 1 -- 172.21.15.27:0/3553217065 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a804c3c0 con 0x7f31a80c8f80 2024-01-11T08:30:24.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.839+0000 7f319dffb700 1 -- 172.21.15.27:0/3553217065 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 78) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f31a0015640 con 0x7f31a80c8f80 2024-01-11T08:30:24.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.840+0000 7f319dffb700 1 --2- 172.21.15.27:0/3553217065 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f318c066f30 0x7f318c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:24.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.840+0000 7f319dffb700 1 -- 172.21.15.27:0/3553217065 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f31a0084b80 con 0x7f31a80c8f80 2024-01-11T08:30:24.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.840+0000 7f31adf9b700 1 --2- 172.21.15.27:0/3553217065 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f318c066f30 0x7f318c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:24.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.841+0000 7f31adf9b700 1 --2- 172.21.15.27:0/3553217065 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f318c066f30 0x7f318c0693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f31a4001eb0 tx=0x7f31a4015000 comp rx=0 tx=0).ready entity=mgr.8086 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:24.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:24.845+0000 7f319dffb700 1 -- 172.21.15.27:0/3553217065 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f31a0051db0 con 0x7f31a80c8f80 2024-01-11T08:30:25.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:25.101+0000 7f31b01ff700 1 -- 172.21.15.27:0/3553217065 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f31a80c6ad0 con 0x7f31a80c8f80 2024-01-11T08:30:25.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:25.793+0000 7f319dffb700 1 -- 172.21.15.27:0/3553217065 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 79) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f31a0052180 con 0x7f31a80c8f80 2024-01-11T08:30:25.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:25.795+0000 7f319dffb700 1 -- 172.21.15.27:0/3553217065 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v79) v1 ==== 92+0+0 (secure 0 0 0) 0x7f31a0055840 con 0x7f31a80c8f80 2024-01-11T08:30:25.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:25.798+0000 7f31b01ff700 1 -- 172.21.15.27:0/3553217065 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f318c066f30 msgr2=0x7f318c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:25.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:25.798+0000 7f31b01ff700 1 --2- 172.21.15.27:0/3553217065 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f318c066f30 0x7f318c0693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f31a4001eb0 tx=0x7f31a4015000 comp rx=0 tx=0).stop 2024-01-11T08:30:25.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:25.798+0000 7f31b01ff700 1 -- 172.21.15.27:0/3553217065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31a80c8f80 msgr2=0x7f31a80c9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:25.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:25.798+0000 7f31b01ff700 1 --2- 172.21.15.27:0/3553217065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31a80c8f80 0x7f31a80c9420 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f31a8089840 tx=0x7f31a00016d0 comp rx=0 tx=0).stop 2024-01-11T08:30:25.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:25.799+0000 7f31b01ff700 1 -- 172.21.15.27:0/3553217065 shutdown_connections 2024-01-11T08:30:25.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:25.799+0000 7f31b01ff700 1 --2- 172.21.15.27:0/3553217065 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f318c066f30 0x7f318c0693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:25.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:25.799+0000 7f31b01ff700 1 --2- 172.21.15.27:0/3553217065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31a80c8f80 0x7f31a80c9420 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:25.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:25.799+0000 7f31b01ff700 1 --2- 172.21.15.27:0/3553217065 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80ccd70 0x7f31a8147f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:25.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:25.799+0000 7f31b01ff700 1 --2- 172.21.15.27:0/3553217065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a815dd60 0x7f31a815e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:25.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:25.799+0000 7f31b01ff700 1 -- 172.21.15.27:0/3553217065 >> 172.21.15.27:0/3553217065 conn(0x7f31a806ed40 msgr2=0x7f31a815c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:25.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:25.799+0000 7f31b01ff700 1 -- 172.21.15.27:0/3553217065 shutdown_connections 2024-01-11T08:30:25.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:25.799+0000 7f31b01ff700 1 -- 172.21.15.27:0/3553217065 wait complete. 2024-01-11T08:30:25.821 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-11T08:30:26.006 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:26.008+0000 7f94485631c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:30:26.020 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:26.021+0000 7f4a2056c1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T08:30:26.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.031+0000 7efe9f1ff700 1 Processor -- start 2024-01-11T08:30:26.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.031+0000 7efe9f1ff700 1 -- start start 2024-01-11T08:30:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.033+0000 7efe9f1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe98155c50 0x7efe980aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.033+0000 7efe9f1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe980c74f0 0x7efe98069910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.033+0000 7efe9f1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9806e480 con 0x7efe980694c0 2024-01-11T08:30:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.033+0000 7efe9f1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe980cb5a0 con 0x7efe98155c50 2024-01-11T08:30:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.033+0000 7efe9f1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe9806df00 con 0x7efe980c74f0 2024-01-11T08:30:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.033+0000 7efe9cf9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe98155c50 0x7efe980aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.033+0000 7efe9cf9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe98155c50 0x7efe980aef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42770/0 (socket says 172.21.15.27:42770) 2024-01-11T08:30:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.033+0000 7efe9cf9b700 1 -- 172.21.15.27:0/557258485 learned_addr learned my addr 172.21.15.27:0/557258485 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.033+0000 7efe97fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efe980694c0 0x7efe980c6de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45890/0 (socket says 172.21.15.27:45890) 2024-01-11T08:30:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.033+0000 7efe9d79c700 1 --2- 172.21.15.27:0/557258485 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe980c74f0 0x7efe98069910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.034+0000 7efe9cf9b700 1 -- 172.21.15.27:0/557258485 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe980c74f0 msgr2=0x7efe98069910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.034+0000 7efe9cf9b700 1 --2- 172.21.15.27:0/557258485 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe980c74f0 0x7efe98069910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.034+0000 7efe9cf9b700 1 -- 172.21.15.27:0/557258485 >> v1:172.21.15.27:6789/0 conn(0x7efe980694c0 legacy=0x7efe980c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.034+0000 7efe9cf9b700 1 -- 172.21.15.27:0/557258485 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe98069e50 con 0x7efe98155c50 2024-01-11T08:30:26.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.034+0000 7efe9cf9b700 1 --2- 172.21.15.27:0/557258485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe98155c50 0x7efe980aef80 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009990 tx=0x7efe8c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6426408106125743 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.034+0000 7efe977fe700 1 -- 172.21.15.27:0/557258485 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c009e80 con 0x7efe98155c50 2024-01-11T08:30:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.034+0000 7efe977fe700 1 -- 172.21.15.27:0/557258485 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8c005c10 con 0x7efe98155c50 2024-01-11T08:30:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.034+0000 7efe977fe700 1 -- 172.21.15.27:0/557258485 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c00b6e0 con 0x7efe98155c50 2024-01-11T08:30:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.035+0000 7efe9f1ff700 1 -- 172.21.15.27:0/557258485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe98155c50 msgr2=0x7efe980aef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.035+0000 7efe9f1ff700 1 --2- 172.21.15.27:0/557258485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe98155c50 0x7efe980aef80 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009990 tx=0x7efe8c009220 comp rx=0 tx=0).stop 2024-01-11T08:30:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.035+0000 7efe9f1ff700 1 -- 172.21.15.27:0/557258485 shutdown_connections 2024-01-11T08:30:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.035+0000 7efe9f1ff700 1 --2- 172.21.15.27:0/557258485 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe980c74f0 0x7efe98069910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.035+0000 7efe9f1ff700 1 --2- 172.21.15.27:0/557258485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe98155c50 0x7efe980aef80 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.035+0000 7efe9f1ff700 1 -- 172.21.15.27:0/557258485 >> 172.21.15.27:0/557258485 conn(0x7efe980680a0 msgr2=0x7efe980cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.035+0000 7efe9f1ff700 1 -- 172.21.15.27:0/557258485 shutdown_connections 2024-01-11T08:30:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.035+0000 7efe9f1ff700 1 -- 172.21.15.27:0/557258485 wait complete. 2024-01-11T08:30:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.037+0000 7efe9f1ff700 1 Processor -- start 2024-01-11T08:30:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.037+0000 7efe9f1ff700 1 -- start start 2024-01-11T08:30:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.038+0000 7efe9f1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe980694c0 0x7efe9815a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.038+0000 7efe9f1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe980c74f0 0x7efe98164f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.038+0000 7efe9f1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe98155c50 0x7efe9814faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.038+0000 7efe9f1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe98156020 con 0x7efe98155c50 2024-01-11T08:30:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.038+0000 7efe9f1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe98156300 con 0x7efe980694c0 2024-01-11T08:30:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.038+0000 7efe9f1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe981657d0 con 0x7efe980c74f0 2024-01-11T08:30:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.038+0000 7efe9d79c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe98155c50 0x7efe9814faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.038+0000 7efe97fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe980c74f0 0x7efe98164f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.038+0000 7efe9d79c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe98155c50 0x7efe9814faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42784/0 (socket says 172.21.15.27:42784) 2024-01-11T08:30:26.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.038+0000 7efe9d79c700 1 -- 172.21.15.27:0/1702272820 learned_addr learned my addr 172.21.15.27:0/1702272820 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:26.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.038+0000 7efe9cf9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe980694c0 0x7efe9815a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:26.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.039+0000 7efe9d79c700 1 -- 172.21.15.27:0/1702272820 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe980c74f0 msgr2=0x7efe98164f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:26.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.039+0000 7efe9d79c700 1 --2- 172.21.15.27:0/1702272820 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe980c74f0 0x7efe98164f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.039+0000 7efe9d79c700 1 -- 172.21.15.27:0/1702272820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe980694c0 msgr2=0x7efe9815a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:26.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.039+0000 7efe9d79c700 1 --2- 172.21.15.27:0/1702272820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe980694c0 0x7efe9815a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.039+0000 7efe9d79c700 1 -- 172.21.15.27:0/1702272820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8c009480 con 0x7efe98155c50 2024-01-11T08:30:26.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.039+0000 7efe9d79c700 1 --2- 172.21.15.27:0/1702272820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe98155c50 0x7efe9814faf0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7efe90009e90 tx=0x7efe9000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:26.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.039+0000 7efe94ff9700 1 -- 172.21.15.27:0/1702272820 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9000cb70 con 0x7efe98155c50 2024-01-11T08:30:26.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.039+0000 7efe94ff9700 1 -- 172.21.15.27:0/1702272820 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9000f040 con 0x7efe98155c50 2024-01-11T08:30:26.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.039+0000 7efe9f1ff700 1 -- 172.21.15.27:0/1702272820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe98165ab0 con 0x7efe98155c50 2024-01-11T08:30:26.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.039+0000 7efe94ff9700 1 -- 172.21.15.27:0/1702272820 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe90014480 con 0x7efe98155c50 2024-01-11T08:30:26.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.039+0000 7efe9f1ff700 1 -- 172.21.15.27:0/1702272820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe981654d0 con 0x7efe98155c50 2024-01-11T08:30:26.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.040+0000 7efe9f1ff700 1 -- 172.21.15.27:0/1702272820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe98156470 con 0x7efe98155c50 2024-01-11T08:30:26.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.041+0000 7efe94ff9700 1 -- 172.21.15.27:0/1702272820 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 79) v1 ==== 84759+0+0 (secure 0 0 0) 0x7efe90014830 con 0x7efe98155c50 2024-01-11T08:30:26.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.042+0000 7efe94ff9700 1 --2- 172.21.15.27:0/1702272820 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe7c066f80 0x7efe7c069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:26.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.042+0000 7efe94ff9700 1 -- 172.21.15.27:0/1702272820 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5761+0+0 (secure 0 0 0) 0x7efe90085320 con 0x7efe98155c50 2024-01-11T08:30:26.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.042+0000 7efe9cf9b700 1 -- 172.21.15.27:0/1702272820 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe7c066f80 msgr2=0x7efe7c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:30:26.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.042+0000 7efe9cf9b700 1 --2- 172.21.15.27:0/1702272820 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe7c066f80 0x7efe7c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:30:26.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.046+0000 7efe94ff9700 1 -- 172.21.15.27:0/1702272820 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efe900524b0 con 0x7efe98155c50 2024-01-11T08:30:26.186 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:26.187+0000 7f94485631c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:30:26.220 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:26.221+0000 7f4a2056c1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T08:30:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.243+0000 7efe9cf9b700 1 -- 172.21.15.27:0/1702272820 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe7c066f80 msgr2=0x7efe7c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:30:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.243+0000 7efe9cf9b700 1 --2- 172.21.15.27:0/1702272820 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe7c066f80 0x7efe7c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:30:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.291+0000 7efe9f1ff700 1 -- 172.21.15.27:0/1702272820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7efe9804c3c0 con 0x7efe98155c50 2024-01-11T08:30:26.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.295+0000 7efe94ff9700 1 -- 172.21.15.27:0/1702272820 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v53)=0 v53) v1 ==== 138+0+0 (secure 0 0 0) 0x7efe90055ad0 con 0x7efe98155c50 2024-01-11T08:30:26.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.298+0000 7efe9f1ff700 1 -- 172.21.15.27:0/1702272820 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe7c066f80 msgr2=0x7efe7c069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:30:26.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.298+0000 7efe9f1ff700 1 --2- 172.21.15.27:0/1702272820 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe7c066f80 0x7efe7c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.298+0000 7efe9f1ff700 1 -- 172.21.15.27:0/1702272820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe98155c50 msgr2=0x7efe9814faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:26.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.298+0000 7efe9f1ff700 1 --2- 172.21.15.27:0/1702272820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe98155c50 0x7efe9814faf0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7efe90009e90 tx=0x7efe9000b040 comp rx=0 tx=0).stop 2024-01-11T08:30:26.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.298+0000 7efe9f1ff700 1 -- 172.21.15.27:0/1702272820 shutdown_connections 2024-01-11T08:30:26.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.298+0000 7efe9f1ff700 1 --2- 172.21.15.27:0/1702272820 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7efe7c066f80 0x7efe7c069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.298+0000 7efe9f1ff700 1 --2- 172.21.15.27:0/1702272820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe98155c50 0x7efe9814faf0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.298+0000 7efe9f1ff700 1 --2- 172.21.15.27:0/1702272820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe980694c0 0x7efe9815a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.298+0000 7efe9f1ff700 1 --2- 172.21.15.27:0/1702272820 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe980c74f0 0x7efe98164f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.298+0000 7efe9f1ff700 1 -- 172.21.15.27:0/1702272820 >> 172.21.15.27:0/1702272820 conn(0x7efe980680a0 msgr2=0x7efe980c0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:26.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.298+0000 7efe9f1ff700 1 -- 172.21.15.27:0/1702272820 shutdown_connections 2024-01-11T08:30:26.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.298+0000 7efe9f1ff700 1 -- 172.21.15.27:0/1702272820 wait complete. 2024-01-11T08:30:26.316 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-11T08:30:26.500 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:26.502+0000 7f94485631c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:30:26.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.522+0000 7ff722dff700 1 Processor -- start 2024-01-11T08:30:26.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.523+0000 7ff722dff700 1 -- start start 2024-01-11T08:30:26.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.524+0000 7ff722dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff71c1596f0 0x7ff71c0a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:26.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.524+0000 7ff722dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff71c068d30 0x7ff71c0c8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:26.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.524+0000 7ff722dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71c0c80a0 con 0x7ff71c0a9de0 2024-01-11T08:30:26.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.524+0000 7ff722dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff71c0cbac0 con 0x7ff71c068d30 2024-01-11T08:30:26.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.524+0000 7ff722dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff71c0c7490 con 0x7ff71c1596f0 2024-01-11T08:30:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.524+0000 7ff72139c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff71c068d30 0x7ff71c0c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.524+0000 7ff71bfff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff71c0a9de0 0x7ff71c068620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45904/0 (socket says 172.21.15.27:45904) 2024-01-11T08:30:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.524+0000 7ff71bfff700 1 -- 172.21.15.27:0/834906698 learned_addr learned my addr 172.21.15.27:0/834906698 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:30:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.524+0000 7ff720b9b700 1 --2- 172.21.15.27:0/834906698 >> v2:172.21.15.77:3300/0 conn(0x7ff71c1596f0 0x7ff71c0a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.525+0000 7ff72139c700 1 -- 172.21.15.27:0/834906698 >> v2:172.21.15.77:3300/0 conn(0x7ff71c1596f0 msgr2=0x7ff71c0a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.525+0000 7ff72139c700 1 --2- 172.21.15.27:0/834906698 >> v2:172.21.15.77:3300/0 conn(0x7ff71c1596f0 0x7ff71c0a98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.525+0000 7ff72139c700 1 -- 172.21.15.27:0/834906698 >> v1:172.21.15.27:6789/0 conn(0x7ff71c0a9de0 legacy=0x7ff71c068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.525+0000 7ff72139c700 1 -- 172.21.15.27:0/834906698 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71c0c7770 con 0x7ff71c068d30 2024-01-11T08:30:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.525+0000 7ff72139c700 1 --2- 172.21.15.27:0/834906698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c068d30 0x7ff71c0c8530 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0cc9b0 tx=0x7ff714009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e7df24ca3678d51 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.525+0000 7ff71b7fe700 1 -- 172.21.15.27:0/834906698 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff71400e070 con 0x7ff71c068d30 2024-01-11T08:30:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.525+0000 7ff71b7fe700 1 -- 172.21.15.27:0/834906698 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71400bb60 con 0x7ff71c068d30 2024-01-11T08:30:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.525+0000 7ff71b7fe700 1 -- 172.21.15.27:0/834906698 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff714012430 con 0x7ff71c068d30 2024-01-11T08:30:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.526+0000 7ff722dff700 1 -- 172.21.15.27:0/834906698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c068d30 msgr2=0x7ff71c0c8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.526+0000 7ff722dff700 1 --2- 172.21.15.27:0/834906698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c068d30 0x7ff71c0c8530 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0cc9b0 tx=0x7ff714009780 comp rx=0 tx=0).stop 2024-01-11T08:30:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.526+0000 7ff722dff700 1 -- 172.21.15.27:0/834906698 shutdown_connections 2024-01-11T08:30:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.526+0000 7ff722dff700 1 --2- 172.21.15.27:0/834906698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c068d30 0x7ff71c0c8530 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.526+0000 7ff722dff700 1 --2- 172.21.15.27:0/834906698 >> v2:172.21.15.77:3300/0 conn(0x7ff71c1596f0 0x7ff71c0a98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.526+0000 7ff722dff700 1 -- 172.21.15.27:0/834906698 >> 172.21.15.27:0/834906698 conn(0x7ff71c0c9fb0 msgr2=0x7ff71c06ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.526+0000 7ff722dff700 1 -- 172.21.15.27:0/834906698 shutdown_connections 2024-01-11T08:30:26.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.526+0000 7ff722dff700 1 -- 172.21.15.27:0/834906698 wait complete. 2024-01-11T08:30:26.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.528+0000 7ff722dff700 1 Processor -- start 2024-01-11T08:30:26.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.528+0000 7ff722dff700 1 -- start start 2024-01-11T08:30:26.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.529+0000 7ff722dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff71c068d30 0x7ff71c156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:26.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.529+0000 7ff722dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff71c0a9de0 0x7ff71c0abec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.529+0000 7ff722dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c1596f0 0x7ff71c0ac400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.529+0000 7ff722dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c0adad0 con 0x7ff71c1596f0 2024-01-11T08:30:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.529+0000 7ff722dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c157370 con 0x7ff71c0a9de0 2024-01-11T08:30:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.529+0000 7ff722dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c0ad640 con 0x7ff71c068d30 2024-01-11T08:30:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.529+0000 7ff72139c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c1596f0 0x7ff71c0ac400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.529+0000 7ff72139c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c1596f0 0x7ff71c0ac400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42794/0 (socket says 172.21.15.27:42794) 2024-01-11T08:30:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.529+0000 7ff72139c700 1 -- 172.21.15.27:0/2172070989 learned_addr learned my addr 172.21.15.27:0/2172070989 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.529+0000 7ff71bfff700 1 --2- 172.21.15.27:0/2172070989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff71c0a9de0 0x7ff71c0abec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.529+0000 7ff720b9b700 1 --2- 172.21.15.27:0/2172070989 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff71c068d30 0x7ff71c156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.530+0000 7ff72139c700 1 -- 172.21.15.27:0/2172070989 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff71c068d30 msgr2=0x7ff71c156e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.530+0000 7ff72139c700 1 --2- 172.21.15.27:0/2172070989 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff71c068d30 0x7ff71c156e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.530+0000 7ff72139c700 1 -- 172.21.15.27:0/2172070989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff71c0a9de0 msgr2=0x7ff71c0abec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.530+0000 7ff72139c700 1 --2- 172.21.15.27:0/2172070989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff71c0a9de0 0x7ff71c0abec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.530+0000 7ff72139c700 1 -- 172.21.15.27:0/2172070989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff714009a20 con 0x7ff71c1596f0 2024-01-11T08:30:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.530+0000 7ff71bfff700 1 --2- 172.21.15.27:0/2172070989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff71c0a9de0 0x7ff71c0abec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:30:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.530+0000 7ff72139c700 1 --2- 172.21.15.27:0/2172070989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c1596f0 0x7ff71c0ac400 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7ff714009fd0 tx=0x7ff714019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.530+0000 7ff718ff9700 1 -- 172.21.15.27:0/2172070989 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff714027470 con 0x7ff71c1596f0 2024-01-11T08:30:26.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.530+0000 7ff718ff9700 1 -- 172.21.15.27:0/2172070989 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7140275f0 con 0x7ff71c1596f0 2024-01-11T08:30:26.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.530+0000 7ff722dff700 1 -- 172.21.15.27:0/2172070989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff71c0b1500 con 0x7ff71c1596f0 2024-01-11T08:30:26.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.530+0000 7ff718ff9700 1 -- 172.21.15.27:0/2172070989 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7140279b0 con 0x7ff71c1596f0 2024-01-11T08:30:26.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.530+0000 7ff722dff700 1 -- 172.21.15.27:0/2172070989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff71c0acd10 con 0x7ff71c1596f0 2024-01-11T08:30:26.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.531+0000 7ff722dff700 1 -- 172.21.15.27:0/2172070989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff71c0ad7f0 con 0x7ff71c1596f0 2024-01-11T08:30:26.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.532+0000 7ff718ff9700 1 -- 172.21.15.27:0/2172070989 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 79) v1 ==== 84759+0+0 (secure 0 0 0) 0x7ff714024070 con 0x7ff71c1596f0 2024-01-11T08:30:26.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.534+0000 7ff718ff9700 1 --2- 172.21.15.27:0/2172070989 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff700066f80 0x7ff700069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:26.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.534+0000 7ff718ff9700 1 -- 172.21.15.27:0/2172070989 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5761+0+0 (secure 0 0 0) 0x7ff714086fe0 con 0x7ff71c1596f0 2024-01-11T08:30:26.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.534+0000 7ff720b9b700 1 -- 172.21.15.27:0/2172070989 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff700066f80 msgr2=0x7ff700069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:30:26.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.534+0000 7ff720b9b700 1 --2- 172.21.15.27:0/2172070989 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff700066f80 0x7ff700069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:30:26.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.538+0000 7ff718ff9700 1 -- 172.21.15.27:0/2172070989 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff71c0ad7f0 con 0x7ff71c1596f0 2024-01-11T08:30:26.575 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:26.576+0000 7f4a2056c1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T08:30:26.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.735+0000 7ff720b9b700 1 -- 172.21.15.27:0/2172070989 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff700066f80 msgr2=0x7ff700069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:30:26.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.735+0000 7ff720b9b700 1 --2- 172.21.15.27:0/2172070989 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff700066f80 0x7ff700069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:30:26.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.744+0000 7ff722dff700 1 -- 172.21.15.27:0/2172070989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7ff71c0ad7b0 con 0x7ff71c1596f0 2024-01-11T08:30:26.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.750+0000 7ff718ff9700 1 -- 172.21.15.27:0/2172070989 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v54)=0 v54) v1 ==== 133+0+0 (secure 0 0 0) 0x7ff7140541d0 con 0x7ff71c1596f0 2024-01-11T08:30:26.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.752+0000 7ff722dff700 1 -- 172.21.15.27:0/2172070989 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff700066f80 msgr2=0x7ff700069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:30:26.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.752+0000 7ff722dff700 1 --2- 172.21.15.27:0/2172070989 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff700066f80 0x7ff700069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.752+0000 7ff722dff700 1 -- 172.21.15.27:0/2172070989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c1596f0 msgr2=0x7ff71c0ac400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:26.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.752+0000 7ff722dff700 1 --2- 172.21.15.27:0/2172070989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c1596f0 0x7ff71c0ac400 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7ff714009fd0 tx=0x7ff714019000 comp rx=0 tx=0).stop 2024-01-11T08:30:26.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.752+0000 7ff722dff700 1 -- 172.21.15.27:0/2172070989 shutdown_connections 2024-01-11T08:30:26.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.752+0000 7ff722dff700 1 --2- 172.21.15.27:0/2172070989 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff700066f80 0x7ff700069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.752+0000 7ff722dff700 1 --2- 172.21.15.27:0/2172070989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c1596f0 0x7ff71c0ac400 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.752+0000 7ff722dff700 1 --2- 172.21.15.27:0/2172070989 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff71c068d30 0x7ff71c156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.752+0000 7ff722dff700 1 --2- 172.21.15.27:0/2172070989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff71c0a9de0 0x7ff71c0abec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:26.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.752+0000 7ff722dff700 1 -- 172.21.15.27:0/2172070989 >> 172.21.15.27:0/2172070989 conn(0x7ff71c0c9fb0 msgr2=0x7ff71c0cac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:26.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.752+0000 7ff722dff700 1 -- 172.21.15.27:0/2172070989 shutdown_connections 2024-01-11T08:30:26.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:26.753+0000 7ff722dff700 1 -- 172.21.15.27:0/2172070989 wait complete. 2024-01-11T08:30:26.770 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T08:30:26.770 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-01-11T08:30:26.823 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1 1m 2024-01-11T08:30:27.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.032+0000 7f482a9ff700 1 Processor -- start 2024-01-11T08:30:27.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.033+0000 7f482a9ff700 1 -- start start 2024-01-11T08:30:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.034+0000 7f482a9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4824143190 0x7f48240cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.034+0000 7f482a9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48240ccc70 con 0x7f4824142dc0 2024-01-11T08:30:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.034+0000 7f482a9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482415d320 con 0x7f48240cb4b0 2024-01-11T08:30:27.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.034+0000 7f482a9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482415df30 con 0x7f4824143190 2024-01-11T08:30:27.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.034+0000 7f4823fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4824142dc0 0x7f48240cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45914/0 (socket says 172.21.15.27:45914) 2024-01-11T08:30:27.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.034+0000 7f4823fff700 1 -- 172.21.15.27:0/625161266 learned_addr learned my addr 172.21.15.27:0/625161266 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T08:30:27.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.035+0000 7f48237fe700 1 --2- 172.21.15.27:0/625161266 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4824143190 0x7f48240cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:27.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.035+0000 7f4822ffd700 1 -- 172.21.15.27:0/625161266 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420364497 0 0) 0x7f48240ccc70 con 0x7f4824142dc0 2024-01-11T08:30:27.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.035+0000 7f4822ffd700 1 -- 172.21.15.27:0/625161266 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f480c003570 con 0x7f4824142dc0 2024-01-11T08:30:27.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.035+0000 7f4822ffd700 1 -- 172.21.15.27:0/625161266 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342665033 0 0) 0x7f482415d320 con 0x7f48240cb4b0 2024-01-11T08:30:27.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.035+0000 7f4822ffd700 1 -- 172.21.15.27:0/625161266 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48240ccc70 con 0x7f48240cb4b0 2024-01-11T08:30:27.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.035+0000 7f48237fe700 1 -- 172.21.15.27:0/625161266 >> v1:172.21.15.77:6789/0 conn(0x7f48240cb4b0 legacy=0x7f48240cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:27.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.035+0000 7f48237fe700 1 -- 172.21.15.27:0/625161266 >> v1:172.21.15.27:6789/0 conn(0x7f4824142dc0 legacy=0x7f48240cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:27.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.035+0000 7f48237fe700 1 -- 172.21.15.27:0/625161266 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482415d590 con 0x7f4824143190 2024-01-11T08:30:27.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.035+0000 7f48237fe700 1 --2- 172.21.15.27:0/625161266 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4824143190 0x7f48240cc020 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4818009910 tx=0x7f48180091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d3ca34cfb17fae6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:27.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.036+0000 7f4822ffd700 1 -- 172.21.15.27:0/625161266 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4818011630 con 0x7f4824143190 2024-01-11T08:30:27.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.036+0000 7f4822ffd700 1 -- 172.21.15.27:0/625161266 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4818011790 con 0x7f4824143190 2024-01-11T08:30:27.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.036+0000 7f4822ffd700 1 -- 172.21.15.27:0/625161266 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4818011b30 con 0x7f4824143190 2024-01-11T08:30:27.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.036+0000 7f482a9ff700 1 -- 172.21.15.27:0/625161266 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4824143190 msgr2=0x7f48240cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:27.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.036+0000 7f482a9ff700 1 --2- 172.21.15.27:0/625161266 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4824143190 0x7f48240cc020 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4818009910 tx=0x7f48180091a0 comp rx=0 tx=0).stop 2024-01-11T08:30:27.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.036+0000 7f482a9ff700 1 -- 172.21.15.27:0/625161266 shutdown_connections 2024-01-11T08:30:27.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.036+0000 7f482a9ff700 1 --2- 172.21.15.27:0/625161266 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4824143190 0x7f48240cc020 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:27.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.036+0000 7f482a9ff700 1 -- 172.21.15.27:0/625161266 >> 172.21.15.27:0/625161266 conn(0x7f4824063ca0 msgr2=0x7f482406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:27.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.036+0000 7f482a9ff700 1 -- 172.21.15.27:0/625161266 shutdown_connections 2024-01-11T08:30:27.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.037+0000 7f482a9ff700 1 -- 172.21.15.27:0/625161266 wait complete. 2024-01-11T08:30:27.038 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:27.036+0000 7f94485631c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:30:27.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.038+0000 7f482a9ff700 1 Processor -- start 2024-01-11T08:30:27.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.038+0000 7f482a9ff700 1 -- start start 2024-01-11T08:30:27.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.039+0000 7f482a9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48240cb4b0 0x7f482413f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:27.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.039+0000 7f482a9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4824142dc0 0x7f48240b13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:27.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.039+0000 7f482a9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4824143190 0x7f48240b1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:27.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.039+0000 7f482a9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48240c57e0 con 0x7f4824142dc0 2024-01-11T08:30:27.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.039+0000 7f482a9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48240b2b90 con 0x7f4824143190 2024-01-11T08:30:27.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.039+0000 7f482a9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48240b1f80 con 0x7f48240cb4b0 2024-01-11T08:30:27.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.039+0000 7f48237fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4824142dc0 0x7f48240b13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:27.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.039+0000 7f48237fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4824142dc0 0x7f48240b13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42808/0 (socket says 172.21.15.27:42808) 2024-01-11T08:30:27.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.039+0000 7f48237fe700 1 -- 172.21.15.27:0/3334975334 learned_addr learned my addr 172.21.15.27:0/3334975334 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:27.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.039+0000 7f4828f9c700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4824143190 0x7f48240b1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:27.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.039+0000 7f4823fff700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48240cb4b0 0x7f482413f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:27.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.040+0000 7f4828f9c700 1 -- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48240cb4b0 msgr2=0x7f482413f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:27.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.040+0000 7f4828f9c700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48240cb4b0 0x7f482413f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:27.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.040+0000 7f4828f9c700 1 -- 172.21.15.27:0/3334975334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4824142dc0 msgr2=0x7f48240b13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:27.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.040+0000 7f4828f9c700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4824142dc0 0x7f48240b13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:27.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.040+0000 7f4828f9c700 1 -- 172.21.15.27:0/3334975334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4818009420 con 0x7f4824143190 2024-01-11T08:30:27.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.040+0000 7f48237fe700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4824142dc0 0x7f48240b13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:30:27.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.040+0000 7f4828f9c700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4824143190 0x7f48240b1900 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f481c00c8e0 tx=0x7f481c0026f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:27.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.040+0000 7f480bfff700 1 -- 172.21.15.27:0/3334975334 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481c012630 con 0x7f4824143190 2024-01-11T08:30:27.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.040+0000 7f480bfff700 1 -- 172.21.15.27:0/3334975334 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481c012790 con 0x7f4824143190 2024-01-11T08:30:27.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.040+0000 7f482a9ff700 1 -- 172.21.15.27:0/3334975334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48240b2260 con 0x7f4824143190 2024-01-11T08:30:27.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.040+0000 7f480bfff700 1 -- 172.21.15.27:0/3334975334 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481c012b50 con 0x7f4824143190 2024-01-11T08:30:27.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.041+0000 7f482a9ff700 1 -- 172.21.15.27:0/3334975334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48240b2820 con 0x7f4824143190 2024-01-11T08:30:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.042+0000 7f482a9ff700 1 -- 172.21.15.27:0/3334975334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f482404c3c0 con 0x7f4824143190 2024-01-11T08:30:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.042+0000 7f480bfff700 1 -- 172.21.15.27:0/3334975334 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 79) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f481c01a3b0 con 0x7f4824143190 2024-01-11T08:30:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.043+0000 7f480bfff700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4810066f80 0x7f4810069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.043+0000 7f480bfff700 1 -- 172.21.15.27:0/3334975334 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f481c084630 con 0x7f4824143190 2024-01-11T08:30:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.044+0000 7f4823fff700 1 -- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4810066f80 msgr2=0x7f4810069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:30:27.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.044+0000 7f4823fff700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4810066f80 0x7f4810069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T08:30:27.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.048+0000 7f480bfff700 1 -- 172.21.15.27:0/3334975334 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f481c0517c0 con 0x7f4824143190 2024-01-11T08:30:27.160 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:27.162+0000 7f4a2056c1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T08:30:27.206 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:27.208+0000 7f94485631c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:30:27.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.245+0000 7f4823fff700 1 -- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4810066f80 msgr2=0x7f4810069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:30:27.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.245+0000 7f4823fff700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4810066f80 0x7f4810069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T08:30:27.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.280+0000 7f482a9ff700 1 -- 172.21.15.27:0/3334975334 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f482415d870 con 0x7f4810066f80 2024-01-11T08:30:27.344 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:27.345+0000 7f4a2056c1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T08:30:27.620 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:27.622+0000 7f94485631c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:30:27.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.646+0000 7f4823fff700 1 -- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4810066f80 msgr2=0x7f4810069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:30:27.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:27.646+0000 7f4823fff700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4810066f80 0x7f4810069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-11T08:30:27.786 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:27.788+0000 7f4a2056c1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T08:30:28.398 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:28.400+0000 7f94485631c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:30:28.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:28.447+0000 7f4823fff700 1 -- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4810066f80 msgr2=0x7f4810069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:30:28.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:28.447+0000 7f4823fff700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4810066f80 0x7f4810069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-11T08:30:28.487 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:28.488+0000 7f94485631c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:30:28.623 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:28.624+0000 7f4a2056c1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T08:30:28.642 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:28.642+0000 7f94485631c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:30:28.717 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:28.719+0000 7f4a2056c1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T08:30:28.729 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:28.729+0000 7f94485631c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:30:28.886 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:28.887+0000 7f4a2056c1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T08:30:28.945 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:28.946+0000 7f94485631c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:30:28.979 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:28.980+0000 7f4a2056c1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T08:30:29.043 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:29.044+0000 7f94485631c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:30:29.138 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:29.139+0000 7f94485631c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:30:29.208 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:29.209+0000 7f4a2056c1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T08:30:29.235 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:29.235+0000 7f94485631c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:30:29.311 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:29.312+0000 7f4a2056c1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T08:30:29.415 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:29.416+0000 7f4a2056c1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T08:30:29.425 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:29.425+0000 7f94485631c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:30:29.517 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:29.518+0000 7f4a2056c1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T08:30:29.524 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:29.525+0000 7f94485631c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:30:29.721 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:29.722+0000 7f4a2056c1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T08:30:29.827 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:29.828+0000 7f4a2056c1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T08:30:29.868 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:29.868+0000 7f94485631c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:30:29.958 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:29.958+0000 7f94485631c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:30:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:30.049+0000 7f4823fff700 1 -- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4810066f80 msgr2=0x7f4810069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T08:30:30.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:30.049+0000 7f4823fff700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4810066f80 0x7f4810069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-11T08:30:30.191 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:30.192+0000 7f4a2056c1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T08:30:30.288 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:30.289+0000 7f4a2056c1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T08:30:30.505 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:30.505+0000 7f94485631c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:30:30.691 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:30.691+0000 7f94485631c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:30:30.873 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:30.874+0000 7f4a2056c1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T08:30:31.068 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:31.068+0000 7f94485631c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:30:31.073 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:31.074+0000 7f4a2056c1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T08:30:31.474 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:31.476+0000 7f4a2056c1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:30:31.494 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:31.494+0000 7f94485631c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:30:31.675 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:31.676+0000 7f94485631c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:30:31.865 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:31.866+0000 7f94485631c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:30:31.932 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:31.933+0000 7f4a2056c1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T08:30:31.976 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:31.976+0000 7f94485631c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:30:31.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:31.989+0000 7f480bfff700 1 -- 172.21.15.27:0/3334975334 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 80) v1 ==== 84470+0+0 (secure 0 0 0) 0x7f481c051b90 con 0x7f4824143190 2024-01-11T08:30:31.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:31.989+0000 7f480bfff700 1 -- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4810066f80 msgr2=0x7f4810069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T08:30:31.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:31.989+0000 7f480bfff700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4810066f80 0x7f4810069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:32.107 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:32.107+0000 7f93bdbb9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:32.107 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:32.107+0000 7f93bdbb9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:32.107 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:32.107+0000 7f93bdbb9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:32.107 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:32.107+0000 7f93bdbb9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:32.107 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:32.107+0000 7f93bdbb9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:32.107 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:32.107+0000 7f93c0bbf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:32.107 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:32.107+0000 7f93c0bbf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:32.108 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:32.107+0000 7f93c0bbf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:32.108 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:32.107+0000 7f93c0bbf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:32.108 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:30:32.107+0000 7f93c0bbf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:30:32.138 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:32.138+0000 7f4a2056c1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T08:30:32.344 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:32.344+0000 7f4a2056c1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T08:30:32.461 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T08:30:32.461+0000 7f4a2056c1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T08:30:32.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:32.994+0000 7f480bfff700 1 -- 172.21.15.27:0/3334975334 <== mon.1 v2:172.21.15.77:3300/0 8 ==== mgrmap(e 81) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f481c05bf90 con 0x7f4824143190 2024-01-11T08:30:32.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:32.995+0000 7f480bfff700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4810066f80 0x7f48100693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:32.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:32.995+0000 7f480bfff700 1 -- 172.21.15.27:0/3334975334 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f482415d870 con 0x7f4810066f80 2024-01-11T08:30:32.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:32.999+0000 7f4823fff700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4810066f80 0x7f48100693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:33.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.001+0000 7f4823fff700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4810066f80 0x7f48100693b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4814007fd0 tx=0x7f4814009040 comp rx=0 tx=0).ready entity=mgr.8434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:33.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.075+0000 7f480bfff700 1 -- 172.21.15.27:0/3334975334 <== mgr.8434 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f482415d870 con 0x7f4810066f80 2024-01-11T08:30:33.075 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path /snap_test_dir1 2024-01-11T08:30:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.078+0000 7f482a9ff700 1 -- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4810066f80 msgr2=0x7f48100693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.078+0000 7f482a9ff700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4810066f80 0x7f48100693b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4814007fd0 tx=0x7f4814009040 comp rx=0 tx=0).stop 2024-01-11T08:30:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.078+0000 7f482a9ff700 1 -- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4824143190 msgr2=0x7f48240b1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.078+0000 7f482a9ff700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4824143190 0x7f48240b1900 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f481c00c8e0 tx=0x7f481c0026f0 comp rx=0 tx=0).stop 2024-01-11T08:30:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.078+0000 7f482a9ff700 1 -- 172.21.15.27:0/3334975334 shutdown_connections 2024-01-11T08:30:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.078+0000 7f482a9ff700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4810066f80 0x7f48100693b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.078+0000 7f482a9ff700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4824143190 0x7f48240b1900 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.078+0000 7f482a9ff700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48240cb4b0 0x7f482413f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.078+0000 7f482a9ff700 1 --2- 172.21.15.27:0/3334975334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4824142dc0 0x7f48240b13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.078+0000 7f482a9ff700 1 -- 172.21.15.27:0/3334975334 >> 172.21.15.27:0/3334975334 conn(0x7f4824063ca0 msgr2=0x7f48240c4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.078+0000 7f482a9ff700 1 -- 172.21.15.27:0/3334975334 shutdown_connections 2024-01-11T08:30:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.078+0000 7f482a9ff700 1 -- 172.21.15.27:0/3334975334 wait complete. 2024-01-11T08:30:33.098 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1 2024-01-11T08:30:33.098 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add /snap_test_dir1 55n 2024-01-11T08:30:33.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.302+0000 7f4b081ff700 1 Processor -- start 2024-01-11T08:30:33.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.302+0000 7f4b081ff700 1 -- start start 2024-01-11T08:30:33.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.304+0000 7f4b081ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b00156440 0x7f4b000c7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:33.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.304+0000 7f4b081ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4b00156810 0x7f4b0006e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:33.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.304+0000 7f4b081ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b000c8590 con 0x7f4b000ccac0 2024-01-11T08:30:33.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.304+0000 7f4b081ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b000c7980 con 0x7f4b00156440 2024-01-11T08:30:33.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.304+0000 7f4b081ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b000c7c60 con 0x7f4b00156810 2024-01-11T08:30:33.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.304+0000 7f4b05f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b00156440 0x7f4b000c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:33.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.304+0000 7f4b05f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b00156440 0x7f4b000c7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42816/0 (socket says 172.21.15.27:42816) 2024-01-11T08:30:33.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.304+0000 7f4b05f9b700 1 -- 172.21.15.27:0/2952731977 learned_addr learned my addr 172.21.15.27:0/2952731977 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:33.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.304+0000 7f4b0579a700 1 --2- 172.21.15.27:0/2952731977 >> v2:172.21.15.77:3300/0 conn(0x7f4b00156810 0x7f4b0006e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:33.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.304+0000 7f4b05f9b700 1 -- 172.21.15.27:0/2952731977 >> v2:172.21.15.77:3300/0 conn(0x7f4b00156810 msgr2=0x7f4b0006e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:33.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.304+0000 7f4b05f9b700 1 --2- 172.21.15.27:0/2952731977 >> v2:172.21.15.77:3300/0 conn(0x7f4b00156810 0x7f4b0006e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:33.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.305+0000 7f4b05f9b700 1 -- 172.21.15.27:0/2952731977 >> v1:172.21.15.77:6789/0 conn(0x7f4b000ccac0 legacy=0x7f4b0006d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:33.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.305+0000 7f4b05f9b700 1 -- 172.21.15.27:0/2952731977 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b000c7f40 con 0x7f4b00156440 2024-01-11T08:30:33.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.305+0000 7f4b05f9b700 1 --2- 172.21.15.27:0/2952731977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b00156440 0x7f4b000c7120 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f4b000cbe70 tx=0x7f4afc009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6873b931e939d1d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:33.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.305+0000 7f4b04f99700 1 -- 172.21.15.27:0/2952731977 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc005c10 con 0x7f4b00156440 2024-01-11T08:30:33.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.305+0000 7f4b04f99700 1 -- 172.21.15.27:0/2952731977 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4afc005d70 con 0x7f4b00156440 2024-01-11T08:30:33.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.305+0000 7f4b04f99700 1 -- 172.21.15.27:0/2952731977 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc00b840 con 0x7f4b00156440 2024-01-11T08:30:33.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.306+0000 7f4b081ff700 1 -- 172.21.15.27:0/2952731977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b00156440 msgr2=0x7f4b000c7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:33.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.306+0000 7f4b081ff700 1 --2- 172.21.15.27:0/2952731977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b00156440 0x7f4b000c7120 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f4b000cbe70 tx=0x7f4afc009220 comp rx=0 tx=0).stop 2024-01-11T08:30:33.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.306+0000 7f4b081ff700 1 -- 172.21.15.27:0/2952731977 shutdown_connections 2024-01-11T08:30:33.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.306+0000 7f4b081ff700 1 --2- 172.21.15.27:0/2952731977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b00156440 0x7f4b000c7120 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:33.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.306+0000 7f4b081ff700 1 --2- 172.21.15.27:0/2952731977 >> v2:172.21.15.77:3300/0 conn(0x7f4b00156810 0x7f4b0006e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:33.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.306+0000 7f4b081ff700 1 -- 172.21.15.27:0/2952731977 >> 172.21.15.27:0/2952731977 conn(0x7f4b000680e0 msgr2=0x7f4b000cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:33.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.306+0000 7f4b081ff700 1 -- 172.21.15.27:0/2952731977 shutdown_connections 2024-01-11T08:30:33.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.306+0000 7f4b081ff700 1 -- 172.21.15.27:0/2952731977 wait complete. 2024-01-11T08:30:33.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.307+0000 7f4b081ff700 1 Processor -- start 2024-01-11T08:30:33.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.307+0000 7f4b081ff700 1 -- start start 2024-01-11T08:30:33.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.308+0000 7f4b081ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b000ccac0 0x7f4b0014d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:33.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.308+0000 7f4b081ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b00156440 0x7f4b000af090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:33.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.308+0000 7f4b081ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4b00156810 0x7f4b000699a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.308+0000 7f4b081ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0014dc90 con 0x7f4b000ccac0 2024-01-11T08:30:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.308+0000 7f4b081ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b000c4110 con 0x7f4b00156440 2024-01-11T08:30:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.308+0000 7f4b081ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b000af5d0 con 0x7f4b00156810 2024-01-11T08:30:33.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.309+0000 7f4b05f9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b000ccac0 0x7f4b0014d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:33.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.309+0000 7f4b05f9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b000ccac0 0x7f4b0014d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42818/0 (socket says 172.21.15.27:42818) 2024-01-11T08:30:33.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.309+0000 7f4b05f9b700 1 -- 172.21.15.27:0/2305954719 learned_addr learned my addr 172.21.15.27:0/2305954719 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:33.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.309+0000 7f4b0579a700 1 --2- 172.21.15.27:0/2305954719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b00156440 0x7f4b000af090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:33.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.309+0000 7f4b0679c700 1 --2- 172.21.15.27:0/2305954719 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4b00156810 0x7f4b000699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:33.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.309+0000 7f4b05f9b700 1 -- 172.21.15.27:0/2305954719 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4b00156810 msgr2=0x7f4b000699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:33.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.309+0000 7f4b05f9b700 1 --2- 172.21.15.27:0/2305954719 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4b00156810 0x7f4b000699a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:33.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.309+0000 7f4b05f9b700 1 -- 172.21.15.27:0/2305954719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b00156440 msgr2=0x7f4b000af090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:33.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.309+0000 7f4b05f9b700 1 --2- 172.21.15.27:0/2305954719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b00156440 0x7f4b000af090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:33.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.309+0000 7f4b05f9b700 1 -- 172.21.15.27:0/2305954719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4afc009480 con 0x7f4b000ccac0 2024-01-11T08:30:33.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.309+0000 7f4b0579a700 1 --2- 172.21.15.27:0/2305954719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b00156440 0x7f4b000af090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:30:33.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.310+0000 7f4b05f9b700 1 --2- 172.21.15.27:0/2305954719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b000ccac0 0x7f4b0014d750 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f4afc0030c0 tx=0x7f4afc015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.310+0000 7f4af5ffb700 1 -- 172.21.15.27:0/2305954719 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc011a10 con 0x7f4b000ccac0 2024-01-11T08:30:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.310+0000 7f4af5ffb700 1 -- 172.21.15.27:0/2305954719 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4afc011b90 con 0x7f4b000ccac0 2024-01-11T08:30:33.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.310+0000 7f4af5ffb700 1 -- 172.21.15.27:0/2305954719 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc01d430 con 0x7f4b000ccac0 2024-01-11T08:30:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.310+0000 7f4b081ff700 1 -- 172.21.15.27:0/2305954719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b000c3500 con 0x7f4b000ccac0 2024-01-11T08:30:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.310+0000 7f4b081ff700 1 -- 172.21.15.27:0/2305954719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b000c3ac0 con 0x7f4b000ccac0 2024-01-11T08:30:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.311+0000 7f4b081ff700 1 -- 172.21.15.27:0/2305954719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b0004c3c0 con 0x7f4b000ccac0 2024-01-11T08:30:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.312+0000 7f4af5ffb700 1 -- 172.21.15.27:0/2305954719 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 81) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f4afc01d820 con 0x7f4b000ccac0 2024-01-11T08:30:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.313+0000 7f4af5ffb700 1 --2- 172.21.15.27:0/2305954719 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4ae4066e00 0x7f4ae40692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.313+0000 7f4af5ffb700 1 -- 172.21.15.27:0/2305954719 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f4afc0865d0 con 0x7f4b000ccac0 2024-01-11T08:30:33.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.314+0000 7f4b0579a700 1 --2- 172.21.15.27:0/2305954719 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4ae4066e00 0x7f4ae40692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.314+0000 7f4b0579a700 1 --2- 172.21.15.27:0/2305954719 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4ae4066e00 0x7f4ae40692b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4b000b2050 tx=0x7f4af0008040 comp rx=0 tx=0).ready entity=mgr.8434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:33.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.317+0000 7f4af5ffb700 1 -- 172.21.15.27:0/2305954719 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4afc053960 con 0x7f4b000ccac0 2024-01-11T08:30:33.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.553+0000 7f4b081ff700 1 -- 172.21.15.27:0/2305954719 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "/snap_test_dir1", "retention_spec_or_period": "55n", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b000c8220 con 0x7f4ae4066e00 2024-01-11T08:30:33.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.613+0000 7f4af5ffb700 1 -- 172.21.15.27:0/2305954719 <== mgr.8434 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+40 (secure 0 0 0) 0x7f4b000c8220 con 0x7f4ae4066e00 2024-01-11T08:30:33.613 INFO:teuthology.orchestra.run.smithi027.stdout:Retention added to path /snap_test_dir1 2024-01-11T08:30:33.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.615+0000 7f4b081ff700 1 -- 172.21.15.27:0/2305954719 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4ae4066e00 msgr2=0x7f4ae40692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:33.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.615+0000 7f4b081ff700 1 --2- 172.21.15.27:0/2305954719 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4ae4066e00 0x7f4ae40692b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4b000b2050 tx=0x7f4af0008040 comp rx=0 tx=0).stop 2024-01-11T08:30:33.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.615+0000 7f4b081ff700 1 -- 172.21.15.27:0/2305954719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b000ccac0 msgr2=0x7f4b0014d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:33.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.615+0000 7f4b081ff700 1 --2- 172.21.15.27:0/2305954719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b000ccac0 0x7f4b0014d750 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f4afc0030c0 tx=0x7f4afc015000 comp rx=0 tx=0).stop 2024-01-11T08:30:33.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.615+0000 7f4b081ff700 1 -- 172.21.15.27:0/2305954719 shutdown_connections 2024-01-11T08:30:33.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.615+0000 7f4b081ff700 1 --2- 172.21.15.27:0/2305954719 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f4ae4066e00 0x7f4ae40692b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:33.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.615+0000 7f4b081ff700 1 --2- 172.21.15.27:0/2305954719 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4b00156810 0x7f4b000699a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:33.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.615+0000 7f4b081ff700 1 --2- 172.21.15.27:0/2305954719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b000ccac0 0x7f4b0014d750 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:33.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.615+0000 7f4b081ff700 1 --2- 172.21.15.27:0/2305954719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b00156440 0x7f4b000af090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:33.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.615+0000 7f4b081ff700 1 -- 172.21.15.27:0/2305954719 >> 172.21.15.27:0/2305954719 conn(0x7f4b000680e0 msgr2=0x7f4b00154310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:33.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.615+0000 7f4b081ff700 1 -- 172.21.15.27:0/2305954719 shutdown_connections 2024-01-11T08:30:33.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.615+0000 7f4b081ff700 1 -- 172.21.15.27:0/2305954719 wait complete. 2024-01-11T08:30:33.634 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path /snap_test_dir1 2024-01-11T08:30:33.634 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2024-01-11T08:30:33.634 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json /snap_test_dir1 2024-01-11T08:30:33.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.838+0000 7ff0c21ff700 1 Processor -- start 2024-01-11T08:30:33.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.838+0000 7ff0c21ff700 1 -- start start 2024-01-11T08:30:33.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.840+0000 7ff0c21ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff0bc143190 0x7ff0bc0cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:33.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.840+0000 7ff0c21ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0bc0cb4b0 0x7ff0bc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:33.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.840+0000 7ff0c21ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0bc15d320 con 0x7ff0bc142dc0 2024-01-11T08:30:33.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.840+0000 7ff0c21ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0bc15df30 con 0x7ff0bc143190 2024-01-11T08:30:33.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.840+0000 7ff0c21ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc15d590 con 0x7ff0bc0cb4b0 2024-01-11T08:30:33.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.840+0000 7ff0baffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff0bc143190 0x7ff0bc0cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:33.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.840+0000 7ff0baffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff0bc143190 0x7ff0bc0cc020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:47994/0 (socket says 172.21.15.27:47994) 2024-01-11T08:30:33.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.840+0000 7ff0baffd700 1 -- 172.21.15.27:0/4072502083 learned_addr learned my addr 172.21.15.27:0/4072502083 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:33.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.840+0000 7ff0bbfff700 1 --2- 172.21.15.27:0/4072502083 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0bc0cb4b0 0x7ff0bc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:33.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.841+0000 7ff0ba7fc700 1 -- 172.21.15.27:0/4072502083 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011113106 0 0) 0x7ff0bc15d320 con 0x7ff0bc142dc0 2024-01-11T08:30:33.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.841+0000 7ff0bbfff700 1 -- 172.21.15.27:0/4072502083 >> v2:172.21.15.77:3300/0 conn(0x7ff0bc143190 msgr2=0x7ff0bc0cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T08:30:33.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.841+0000 7ff0bbfff700 1 --2- 172.21.15.27:0/4072502083 >> v2:172.21.15.77:3300/0 conn(0x7ff0bc143190 0x7ff0bc0cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:33.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.841+0000 7ff0bbfff700 1 -- 172.21.15.27:0/4072502083 >> v1:172.21.15.77:6789/0 conn(0x7ff0bc142dc0 legacy=0x7ff0bc0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:33.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.841+0000 7ff0bbfff700 1 -- 172.21.15.27:0/4072502083 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0bc15d870 con 0x7ff0bc0cb4b0 2024-01-11T08:30:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.841+0000 7ff0bbfff700 1 --2- 172.21.15.27:0/4072502083 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0bc0cb4b0 0x7ff0bc0cc560 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4009e60 tx=0x7ff0b4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=483ebfb77e3fa23f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.841+0000 7ff0ba7fc700 1 -- 172.21.15.27:0/4072502083 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b400e070 con 0x7ff0bc0cb4b0 2024-01-11T08:30:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.841+0000 7ff0ba7fc700 1 -- 172.21.15.27:0/4072502083 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b400bb60 con 0x7ff0bc0cb4b0 2024-01-11T08:30:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.841+0000 7ff0ba7fc700 1 -- 172.21.15.27:0/4072502083 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b4012430 con 0x7ff0bc0cb4b0 2024-01-11T08:30:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.842+0000 7ff0c21ff700 1 -- 172.21.15.27:0/4072502083 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0bc0cb4b0 msgr2=0x7ff0bc0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.842+0000 7ff0c21ff700 1 --2- 172.21.15.27:0/4072502083 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0bc0cb4b0 0x7ff0bc0cc560 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4009e60 tx=0x7ff0b4009780 comp rx=0 tx=0).stop 2024-01-11T08:30:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.842+0000 7ff0c21ff700 1 -- 172.21.15.27:0/4072502083 shutdown_connections 2024-01-11T08:30:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.842+0000 7ff0c21ff700 1 --2- 172.21.15.27:0/4072502083 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0bc0cb4b0 0x7ff0bc0cc560 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.842+0000 7ff0c21ff700 1 --2- 172.21.15.27:0/4072502083 >> v2:172.21.15.77:3300/0 conn(0x7ff0bc143190 0x7ff0bc0cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.842+0000 7ff0c21ff700 1 -- 172.21.15.27:0/4072502083 >> 172.21.15.27:0/4072502083 conn(0x7ff0bc063ca0 msgr2=0x7ff0bc06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.842+0000 7ff0c21ff700 1 -- 172.21.15.27:0/4072502083 shutdown_connections 2024-01-11T08:30:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.842+0000 7ff0c21ff700 1 -- 172.21.15.27:0/4072502083 wait complete. 2024-01-11T08:30:33.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.843+0000 7ff0c21ff700 1 Processor -- start 2024-01-11T08:30:33.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.844+0000 7ff0c21ff700 1 -- start start 2024-01-11T08:30:33.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.844+0000 7ff0c21ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0bc0cb4b0 0x7ff0bc0b1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:33.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.844+0000 7ff0c21ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0bc142dc0 0x7ff0bc0ca300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:33.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.844+0000 7ff0c21ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0bc143190 0x7ff0bc0b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:33.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.845+0000 7ff0c21ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc0b2450 con 0x7ff0bc142dc0 2024-01-11T08:30:33.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.845+0000 7ff0c21ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc0ca840 con 0x7ff0bc0cb4b0 2024-01-11T08:30:33.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.845+0000 7ff0c21ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc0b1840 con 0x7ff0bc143190 2024-01-11T08:30:33.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.845+0000 7ff0baffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0bc142dc0 0x7ff0bc0ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:33.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.845+0000 7ff0baffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0bc142dc0 0x7ff0bc0ca300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57578/0 (socket says 172.21.15.27:57578) 2024-01-11T08:30:33.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.845+0000 7ff0bb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0bc0cb4b0 0x7ff0bc0b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:33.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.845+0000 7ff0baffd700 1 -- 172.21.15.27:0/1069221556 learned_addr learned my addr 172.21.15.27:0/1069221556 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T08:30:33.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.845+0000 7ff0bbfff700 1 --2- 172.21.15.27:0/1069221556 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0bc143190 0x7ff0bc0b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.845+0000 7ff0baffd700 1 -- 172.21.15.27:0/1069221556 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0bc143190 msgr2=0x7ff0bc0b28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.845+0000 7ff0baffd700 1 --2- 172.21.15.27:0/1069221556 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0bc143190 0x7ff0bc0b28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.845+0000 7ff0baffd700 1 -- 172.21.15.27:0/1069221556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0bc0cb4b0 msgr2=0x7ff0bc0b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.845+0000 7ff0baffd700 1 --2- 172.21.15.27:0/1069221556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0bc0cb4b0 0x7ff0bc0b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.845+0000 7ff0baffd700 1 -- 172.21.15.27:0/1069221556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b4009a20 con 0x7ff0bc142dc0 2024-01-11T08:30:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.845+0000 7ff0bb7fe700 1 --2- 172.21.15.27:0/1069221556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0bc0cb4b0 0x7ff0bc0b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T08:30:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.846+0000 7ff0baffd700 1 --2- 172.21.15.27:0/1069221556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0bc142dc0 0x7ff0bc0ca300 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7ff0b000e980 tx=0x7ff0b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.846+0000 7ff0a37fe700 1 -- 172.21.15.27:0/1069221556 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0013680 con 0x7ff0bc142dc0 2024-01-11T08:30:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.846+0000 7ff0a37fe700 1 -- 172.21.15.27:0/1069221556 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b00137e0 con 0x7ff0bc142dc0 2024-01-11T08:30:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.846+0000 7ff0c21ff700 1 -- 172.21.15.27:0/1069221556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0bc0b1b20 con 0x7ff0bc142dc0 2024-01-11T08:30:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.846+0000 7ff0a37fe700 1 -- 172.21.15.27:0/1069221556 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0013b60 con 0x7ff0bc142dc0 2024-01-11T08:30:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.846+0000 7ff0c21ff700 1 -- 172.21.15.27:0/1069221556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0bc0b20e0 con 0x7ff0bc142dc0 2024-01-11T08:30:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.847+0000 7ff0c21ff700 1 -- 172.21.15.27:0/1069221556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0bc04c3c0 con 0x7ff0bc142dc0 2024-01-11T08:30:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.848+0000 7ff0a37fe700 1 -- 172.21.15.27:0/1069221556 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 81) v1 ==== 84597+0+0 (secure 0 0 0) 0x7ff0b000c400 con 0x7ff0bc142dc0 2024-01-11T08:30:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.849+0000 7ff0a37fe700 1 --2- 172.21.15.27:0/1069221556 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff0a8066dd0 0x7ff0a8069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T08:30:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.849+0000 7ff0a37fe700 1 -- 172.21.15.27:0/1069221556 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6019+0+0 (secure 0 0 0) 0x7ff0b00844d0 con 0x7ff0bc142dc0 2024-01-11T08:30:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.849+0000 7ff0bb7fe700 1 --2- 172.21.15.27:0/1069221556 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff0a8066dd0 0x7ff0a8069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T08:30:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.850+0000 7ff0bb7fe700 1 --2- 172.21.15.27:0/1069221556 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff0a8066dd0 0x7ff0a8069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac007fd0 tx=0x7ff0ac009040 comp rx=0 tx=0).ready entity=mgr.8434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T08:30:33.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:33.853+0000 7ff0a37fe700 1 -- 172.21.15.27:0/1069221556 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff0b0055270 con 0x7ff0bc142dc0 2024-01-11T08:30:34.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:34.040+0000 7ff0a37fe700 1 -- 172.21.15.27:0/1069221556 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 82) v1 ==== 84862+0+0 (secure 0 0 0) 0x7ff0b0051bb0 con 0x7ff0bc142dc0 2024-01-11T08:30:34.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:34.088+0000 7ff0c21ff700 1 -- 172.21.15.27:0/1069221556 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff0bc0b2590 con 0x7ff0a8066dd0 2024-01-11T08:30:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:34.096+0000 7ff0a37fe700 1 -- 172.21.15.27:0/1069221556 <== mgr.8434 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+74 (secure 0 0 0) 0x7ff0bc0b2590 con 0x7ff0a8066dd0 2024-01-11T08:30:34.098 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T08:30:34.098 INFO:teuthology.orchestra.run.smithi027.stdout:{"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2024-01-11T08:30:34.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:34.098+0000 7ff0c21ff700 1 -- 172.21.15.27:0/1069221556 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff0a8066dd0 msgr2=0x7ff0a8069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:34.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:34.098+0000 7ff0c21ff700 1 --2- 172.21.15.27:0/1069221556 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff0a8066dd0 0x7ff0a8069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac007fd0 tx=0x7ff0ac009040 comp rx=0 tx=0).stop 2024-01-11T08:30:34.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:34.098+0000 7ff0c21ff700 1 -- 172.21.15.27:0/1069221556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0bc142dc0 msgr2=0x7ff0bc0ca300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T08:30:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:34.098+0000 7ff0c21ff700 1 --2- 172.21.15.27:0/1069221556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0bc142dc0 0x7ff0bc0ca300 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7ff0b000e980 tx=0x7ff0b0009250 comp rx=0 tx=0).stop 2024-01-11T08:30:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:34.098+0000 7ff0c21ff700 1 -- 172.21.15.27:0/1069221556 shutdown_connections 2024-01-11T08:30:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:34.098+0000 7ff0c21ff700 1 --2- 172.21.15.27:0/1069221556 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7ff0a8066dd0 0x7ff0a8069280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:34.098+0000 7ff0c21ff700 1 --2- 172.21.15.27:0/1069221556 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0bc143190 0x7ff0bc0b28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:34.098+0000 7ff0c21ff700 1 --2- 172.21.15.27:0/1069221556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0bc0cb4b0 0x7ff0bc0b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:34.098+0000 7ff0c21ff700 1 --2- 172.21.15.27:0/1069221556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0bc142dc0 0x7ff0bc0ca300 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T08:30:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:34.098+0000 7ff0c21ff700 1 -- 172.21.15.27:0/1069221556 >> 172.21.15.27:0/1069221556 conn(0x7ff0bc063ca0 msgr2=0x7ff0bc06cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T08:30:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:34.098+0000 7ff0c21ff700 1 -- 172.21.15.27:0/1069221556 shutdown_connections 2024-01-11T08:30:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T08:30:34.098+0000 7ff0c21ff700 1 -- 172.21.15.27:0/1069221556 wait complete. 2024-01-11T08:30:34.115 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2024-01-11T08:30:34.116 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': '/snap_test_dir1', 'schedule': ['1m'], 'retention': [{'n': 55}]} 2024-01-11T08:30:53.778 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:30:53.784 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:31:00.625 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:31:00.626+0000 7f93afb9d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:31:00.625 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:31:00.626+0000 7f93afb9d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:31:00.625 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:31:00.626+0000 7f93afb9d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:31:00.626 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:31:00.626+0000 7f93afb9d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:31:00.626 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T08:31:00.626+0000 7f93afb9d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T08:31:23.907 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:31:23.913 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:31:54.029 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:31:54.035 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:32:24.144 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:32:24.149 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:32:54.261 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:32:54.267 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:33:24.412 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:33:24.418 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:33:54.551 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:33:54.578 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:34:24.728 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:34:24.734 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:34:54.846 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:34:54.851 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:35:24.956 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:35:24.962 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:35:55.070 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:35:55.075 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:36:25.185 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:36:25.191 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:36:55.320 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:36:55.325 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:37:25.444 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:37:25.450 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:37:55.562 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:37:55.568 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:38:25.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:38:25.687 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:38:55.801 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:38:55.807 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:39:25.942 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:39:25.948 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:39:56.083 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:39:56.089 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:40:26.205 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:40:26.211 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:40:56.329 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:40:56.335 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:41:26.451 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:41:26.457 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:41:56.574 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:41:56.579 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:42:26.688 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:42:26.694 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:42:56.872 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:42:56.878 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:43:26.994 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:43:27.000 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:43:57.113 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:43:57.119 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:44:27.228 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:44:27.234 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:44:57.345 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:44:57.351 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:45:27.503 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:45:27.508 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:45:57.643 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:45:57.649 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:46:27.799 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:46:27.805 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:46:57.917 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:46:57.923 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:47:28.136 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:47:28.142 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:47:58.258 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:47:58.264 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:48:28.440 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:48:28.446 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:48:58.621 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:48:58.627 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:49:28.802 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:49:28.808 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:49:58.991 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:49:58.997 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:50:29.155 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:50:29.160 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:50:59.278 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:50:59.284 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:51:29.400 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:51:29.406 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:51:59.573 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:51:59.580 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:52:29.738 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:52:29.743 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:52:59.918 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:52:59.924 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:53:30.067 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:53:30.073 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:54:00.255 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:54:00.261 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:54:30.411 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:54:30.417 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:55:00.617 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:55:00.622 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:55:30.755 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:55:30.761 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:56:00.874 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:56:00.880 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:56:31.110 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:56:31.115 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:57:01.238 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:57:01.244 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:57:31.354 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:57:31.360 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:58:01.535 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:58:01.542 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:58:31.725 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:58:31.730 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:59:01.923 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:59:01.929 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:59:32.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T08:59:32.126 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:00:02.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:00:02.324 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:00:32.458 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:00:32.464 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:01:02.633 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:01:02.640 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:01:32.836 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:01:32.842 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:02:03.009 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:02:03.015 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:02:33.119 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:02:33.125 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:03:03.321 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:03:03.327 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:03:33.502 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:03:33.508 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:04:03.683 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:04:03.690 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:04:33.881 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:04:33.886 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:05:04.010 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:05:04.016 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:05:34.192 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:05:34.198 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:06:04.373 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:06:04.379 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:06:34.487 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:06:34.493 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:07:04.678 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:07:04.683 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:07:34.858 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:07:34.864 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:08:05.098 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:08:05.104 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:08:35.278 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:08:35.284 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:09:05.459 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:09:05.465 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:09:35.665 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:09:35.671 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:10:05.781 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:10:05.787 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:10:36.036 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:10:36.042 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:11:06.217 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:11:06.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:11:36.333 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:11:36.339 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:12:06.522 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:12:06.528 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:12:36.719 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:12:36.724 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:13:06.926 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:13:06.931 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:13:37.089 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:13:37.095 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:14:07.221 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:14:07.227 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:14:37.339 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:14:37.345 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:15:07.525 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:15:07.531 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:15:37.714 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:15:37.720 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:16:07.878 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:16:07.883 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:16:38.083 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:16:38.089 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:17:08.347 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:17:08.353 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:17:38.546 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:17:38.555 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:18:08.686 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:18:08.692 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:18:39.074 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:18:39.080 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:19:09.247 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:19:09.253 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:19:39.401 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:19:39.407 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:20:09.625 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:20:09.630 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:20:39.814 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:20:39.819 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:21:10.012 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:21:10.018 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:21:40.210 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:21:40.217 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:22:10.399 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:22:10.405 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:22:40.521 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:22:40.526 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:23:10.669 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:23:10.675 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:23:40.810 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:23:40.815 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:24:10.933 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:24:10.939 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:24:41.053 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:24:41.059 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:25:11.219 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:25:11.224 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:25:41.408 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:25:41.413 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:26:11.524 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:26:11.530 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:26:16.116 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:16.117 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:26:16.177 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_32_00_UTC 2024-01-11T09:26:16.177 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_33_00_UTC 2024-01-11T09:26:16.178 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_34_00_UTC 2024-01-11T09:26:16.178 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_35_00_UTC 2024-01-11T09:26:16.178 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_36_00_UTC 2024-01-11T09:26:16.178 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_37_00_UTC 2024-01-11T09:26:16.178 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_38_00_UTC 2024-01-11T09:26:16.178 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_39_00_UTC 2024-01-11T09:26:16.178 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_40_00_UTC 2024-01-11T09:26:16.178 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_41_00_UTC 2024-01-11T09:26:16.178 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_42_00_UTC 2024-01-11T09:26:16.178 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_43_00_UTC 2024-01-11T09:26:16.178 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_44_00_UTC 2024-01-11T09:26:16.178 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_45_00_UTC 2024-01-11T09:26:16.178 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_46_00_UTC 2024-01-11T09:26:16.179 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_47_00_UTC 2024-01-11T09:26:16.179 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_48_00_UTC 2024-01-11T09:26:16.179 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_49_00_UTC 2024-01-11T09:26:16.179 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_50_00_UTC 2024-01-11T09:26:16.179 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_51_00_UTC 2024-01-11T09:26:16.179 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_52_00_UTC 2024-01-11T09:26:16.179 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_53_00_UTC 2024-01-11T09:26:16.179 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_54_00_UTC 2024-01-11T09:26:16.179 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_55_00_UTC 2024-01-11T09:26:16.179 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_56_00_UTC 2024-01-11T09:26:16.179 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_57_00_UTC 2024-01-11T09:26:16.179 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_58_00_UTC 2024-01-11T09:26:16.179 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_59_00_UTC 2024-01-11T09:26:16.179 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_00_00_UTC 2024-01-11T09:26:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_01_00_UTC 2024-01-11T09:26:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_02_00_UTC 2024-01-11T09:26:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_03_00_UTC 2024-01-11T09:26:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_04_00_UTC 2024-01-11T09:26:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_05_00_UTC 2024-01-11T09:26:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_06_00_UTC 2024-01-11T09:26:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_07_00_UTC 2024-01-11T09:26:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_08_00_UTC 2024-01-11T09:26:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_09_00_UTC 2024-01-11T09:26:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_10_00_UTC 2024-01-11T09:26:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_11_00_UTC 2024-01-11T09:26:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_12_00_UTC 2024-01-11T09:26:16.180 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_13_00_UTC 2024-01-11T09:26:16.181 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_14_00_UTC 2024-01-11T09:26:16.181 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_15_00_UTC 2024-01-11T09:26:16.181 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_16_00_UTC 2024-01-11T09:26:16.181 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_17_00_UTC 2024-01-11T09:26:16.181 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_18_00_UTC 2024-01-11T09:26:16.181 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_19_00_UTC 2024-01-11T09:26:16.181 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_20_00_UTC 2024-01-11T09:26:16.181 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_21_00_UTC 2024-01-11T09:26:16.181 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_22_00_UTC 2024-01-11T09:26:16.181 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_23_00_UTC 2024-01-11T09:26:16.181 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_24_00_UTC 2024-01-11T09:26:16.181 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_25_00_UTC 2024-01-11T09:26:16.181 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_26_00_UTC 2024-01-11T09:26:16.183 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-11-08_32_00_UTC', 'scheduled-2024-01-11-08_33_00_UTC', 'scheduled-2024-01-11-08_34_00_UTC', 'scheduled-2024-01-11-08_35_00_UTC', 'scheduled-2024-01-11-08_36_00_UTC', 'scheduled-2024-01-11-08_37_00_UTC', 'scheduled-2024-01-11-08_38_00_UTC', 'scheduled-2024-01-11-08_39_00_UTC', 'scheduled-2024-01-11-08_40_00_UTC', 'scheduled-2024-01-11-08_41_00_UTC', 'scheduled-2024-01-11-08_42_00_UTC', 'scheduled-2024-01-11-08_43_00_UTC', 'scheduled-2024-01-11-08_44_00_UTC', 'scheduled-2024-01-11-08_45_00_UTC', 'scheduled-2024-01-11-08_46_00_UTC', 'scheduled-2024-01-11-08_47_00_UTC', 'scheduled-2024-01-11-08_48_00_UTC', 'scheduled-2024-01-11-08_49_00_UTC', 'scheduled-2024-01-11-08_50_00_UTC', 'scheduled-2024-01-11-08_51_00_UTC', 'scheduled-2024-01-11-08_52_00_UTC', 'scheduled-2024-01-11-08_53_00_UTC', 'scheduled-2024-01-11-08_54_00_UTC', 'scheduled-2024-01-11-08_55_00_UTC', 'scheduled-2024-01-11-08_56_00_UTC', 'scheduled-2024-01-11-08_57_00_UTC', 'scheduled-2024-01-11-08_58_00_UTC', 'scheduled-2024-01-11-08_59_00_UTC', 'scheduled-2024-01-11-09_00_00_UTC', 'scheduled-2024-01-11-09_01_00_UTC', 'scheduled-2024-01-11-09_02_00_UTC', 'scheduled-2024-01-11-09_03_00_UTC', 'scheduled-2024-01-11-09_04_00_UTC', 'scheduled-2024-01-11-09_05_00_UTC', 'scheduled-2024-01-11-09_06_00_UTC', 'scheduled-2024-01-11-09_07_00_UTC', 'scheduled-2024-01-11-09_08_00_UTC', 'scheduled-2024-01-11-09_09_00_UTC', 'scheduled-2024-01-11-09_10_00_UTC', 'scheduled-2024-01-11-09_11_00_UTC', 'scheduled-2024-01-11-09_12_00_UTC', 'scheduled-2024-01-11-09_13_00_UTC', 'scheduled-2024-01-11-09_14_00_UTC', 'scheduled-2024-01-11-09_15_00_UTC', 'scheduled-2024-01-11-09_16_00_UTC', 'scheduled-2024-01-11-09_17_00_UTC', 'scheduled-2024-01-11-09_18_00_UTC', 'scheduled-2024-01-11-09_19_00_UTC', 'scheduled-2024-01-11-09_20_00_UTC', 'scheduled-2024-01-11-09_21_00_UTC', 'scheduled-2024-01-11-09_22_00_UTC', 'scheduled-2024-01-11-09_23_00_UTC', 'scheduled-2024-01-11-09_24_00_UTC', 'scheduled-2024-01-11-09_25_00_UTC', 'scheduled-2024-01-11-09_26_00_UTC'] 2024-01-11T09:26:16.183 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1 2024-01-11T09:26:16.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.393+0000 7f1bbd7ff700 1 Processor -- start 2024-01-11T09:26:16.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.393+0000 7f1bbd7ff700 1 -- start start 2024-01-11T09:26:16.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.395+0000 7f1bbd7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1bb813ff80 0x7f1bb815a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:16.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.395+0000 7f1bbd7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1bb815a560 0x7f1bb80c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:16.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.395+0000 7f1bbd7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb815a9c0 con 0x7f1bb813fbb0 2024-01-11T09:26:16.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.395+0000 7f1bbd7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bb80c6b30 con 0x7f1bb815a560 2024-01-11T09:26:16.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.395+0000 7f1bbd7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb80c6800 con 0x7f1bb813ff80 2024-01-11T09:26:16.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.395+0000 7f1bb6ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1bb813fbb0 0x7f1bb80ca7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51508/0 (socket says 172.21.15.27:51508) 2024-01-11T09:26:16.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.395+0000 7f1bb6ffd700 1 -- 172.21.15.27:0/1723482766 learned_addr learned my addr 172.21.15.27:0/1723482766 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:16.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.395+0000 7f1bb67fc700 1 --2- 172.21.15.27:0/1723482766 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1bb813ff80 0x7f1bb815a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:16.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.395+0000 7f1bb77fe700 1 --2- 172.21.15.27:0/1723482766 >> v2:172.21.15.77:3300/0 conn(0x7f1bb815a560 0x7f1bb80c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:16.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.395+0000 7f1bb5ffb700 1 -- 172.21.15.27:0/1723482766 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144606541 0 0) 0x7f1ba8001290 con 0x7f1bb813fbb0 2024-01-11T09:26:16.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.396+0000 7f1bb5ffb700 1 -- 172.21.15.27:0/1723482766 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ba0003570 con 0x7f1bb813fbb0 2024-01-11T09:26:16.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.396+0000 7f1bb5ffb700 1 -- 172.21.15.27:0/1723482766 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 749422578 0 0) 0x7f1ba0003570 con 0x7f1bb813fbb0 2024-01-11T09:26:16.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.396+0000 7f1bb5ffb700 1 -- 172.21.15.27:0/1723482766 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1bb813ff80 msgr2=0x7f1bb815a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:16.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.396+0000 7f1bb5ffb700 1 --2- 172.21.15.27:0/1723482766 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1bb813ff80 0x7f1bb815a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:16.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.396+0000 7f1bb5ffb700 1 -- 172.21.15.27:0/1723482766 >> v2:172.21.15.77:3300/0 conn(0x7f1bb815a560 msgr2=0x7f1bb80c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:16.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.396+0000 7f1bb5ffb700 1 --2- 172.21.15.27:0/1723482766 >> v2:172.21.15.77:3300/0 conn(0x7f1bb815a560 0x7f1bb80c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:16.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.396+0000 7f1bb5ffb700 1 -- 172.21.15.27:0/1723482766 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb80cb240 con 0x7f1bb813fbb0 2024-01-11T09:26:16.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.396+0000 7f1bb5ffb700 1 -- 172.21.15.27:0/1723482766 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f1ba8001780 con 0x7f1bb813fbb0 2024-01-11T09:26:16.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.396+0000 7f1bb5ffb700 1 -- 172.21.15.27:0/1723482766 >> v1:172.21.15.27:6789/0 conn(0x7f1bb813fbb0 legacy=0x7f1bb80ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:16.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.396+0000 7f1bb5ffb700 1 --2- 172.21.15.27:0/1723482766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ba0003a60 0x7f1ba0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:16.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.396+0000 7f1bb5ffb700 1 --2- 172.21.15.27:0/1723482766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ba0004810 0x7f1ba0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:16.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.396+0000 7f1bb5ffb700 1 --2- 172.21.15.27:0/1723482766 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1bb813ff80 0x7f1ba00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:16.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.396+0000 7f1bb5ffb700 1 -- 172.21.15.27:0/1723482766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb80c6b30 con 0x7f1ba0003a60 2024-01-11T09:26:16.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.396+0000 7f1bb5ffb700 1 -- 172.21.15.27:0/1723482766 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb80c6800 con 0x7f1ba0004810 2024-01-11T09:26:16.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.396+0000 7f1bb5ffb700 1 -- 172.21.15.27:0/1723482766 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba000dc70 con 0x7f1bb813ff80 2024-01-11T09:26:16.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.396+0000 7f1bb6ffd700 1 --2- 172.21.15.27:0/1723482766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ba0003a60 0x7f1ba0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:16.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.396+0000 7f1bb67fc700 1 --2- 172.21.15.27:0/1723482766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ba0004810 0x7f1ba0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:16.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.397+0000 7f1bb77fe700 1 --2- 172.21.15.27:0/1723482766 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1bb813ff80 0x7f1ba00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:16.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.397+0000 7f1bb6ffd700 1 -- 172.21.15.27:0/1723482766 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1bb813ff80 msgr2=0x7f1ba00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:16.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.397+0000 7f1bb6ffd700 1 --2- 172.21.15.27:0/1723482766 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1bb813ff80 0x7f1ba00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:16.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.397+0000 7f1bb6ffd700 1 -- 172.21.15.27:0/1723482766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ba0004810 msgr2=0x7f1ba0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:16.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.397+0000 7f1bb6ffd700 1 --2- 172.21.15.27:0/1723482766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ba0004810 0x7f1ba0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:16.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.397+0000 7f1bb6ffd700 1 -- 172.21.15.27:0/1723482766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba0003570 con 0x7f1ba0003a60 2024-01-11T09:26:16.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.397+0000 7f1bb6ffd700 1 --2- 172.21.15.27:0/1723482766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ba0003a60 0x7f1ba0003e70 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800c620 tx=0x7f1ba8007ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3980a987f5f25798 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:16.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.397+0000 7f1bb5ffb700 1 -- 172.21.15.27:0/1723482766 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba800cde0 con 0x7f1ba0003a60 2024-01-11T09:26:16.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.397+0000 7f1bb5ffb700 1 -- 172.21.15.27:0/1723482766 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba80070d0 con 0x7f1ba0003a60 2024-01-11T09:26:16.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.397+0000 7f1bb5ffb700 1 -- 172.21.15.27:0/1723482766 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba8007470 con 0x7f1ba0003a60 2024-01-11T09:26:16.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.398+0000 7f1bbd7ff700 1 -- 172.21.15.27:0/1723482766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ba0003a60 msgr2=0x7f1ba0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:16.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.398+0000 7f1bbd7ff700 1 --2- 172.21.15.27:0/1723482766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ba0003a60 0x7f1ba0003e70 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800c620 tx=0x7f1ba8007ab0 comp rx=0 tx=0).stop 2024-01-11T09:26:16.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.398+0000 7f1bb77fe700 1 -- 172.21.15.27:0/1723482766 reap_dead start 2024-01-11T09:26:16.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.398+0000 7f1bbd7ff700 1 -- 172.21.15.27:0/1723482766 shutdown_connections 2024-01-11T09:26:16.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.398+0000 7f1bbd7ff700 1 -- 172.21.15.27:0/1723482766 >> 172.21.15.27:0/1723482766 conn(0x7f1bb80680a0 msgr2=0x7f1bb80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:16.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.398+0000 7f1bbd7ff700 1 -- 172.21.15.27:0/1723482766 shutdown_connections 2024-01-11T09:26:16.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.398+0000 7f1bbd7ff700 1 -- 172.21.15.27:0/1723482766 wait complete. 2024-01-11T09:26:16.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.400+0000 7f1bbd7ff700 1 Processor -- start 2024-01-11T09:26:16.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.400+0000 7f1bbd7ff700 1 -- start start 2024-01-11T09:26:16.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.401+0000 7f1bbd7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ba0003a60 0x7f1bb8158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:16.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.401+0000 7f1bbd7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb8155c50 0x7f1bb8158f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:16.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.401+0000 7f1bbd7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bb80c8a90 0x7f1bb80c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:16.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.401+0000 7f1bbd7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb80c9450 con 0x7f1bb80c8a90 2024-01-11T09:26:16.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.401+0000 7f1bbd7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb816cff0 con 0x7f1bb8155c50 2024-01-11T09:26:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.401+0000 7f1bbd7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb8157d00 con 0x7f1ba0003a60 2024-01-11T09:26:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.401+0000 7f1bb77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bb80c8a90 0x7f1bb80c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.401+0000 7f1bb77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bb80c8a90 0x7f1bb80c8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55520/0 (socket says 172.21.15.27:55520) 2024-01-11T09:26:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.401+0000 7f1bb77fe700 1 -- 172.21.15.27:0/3243908845 learned_addr learned my addr 172.21.15.27:0/3243908845 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.402+0000 7f1bb77fe700 1 -- 172.21.15.27:0/3243908845 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ba0003a60 msgr2=0x7f1bb8158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.402+0000 7f1bb77fe700 1 --2- 172.21.15.27:0/3243908845 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ba0003a60 0x7f1bb8158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.402+0000 7f1bb77fe700 1 -- 172.21.15.27:0/3243908845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb8155c50 msgr2=0x7f1bb8158f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.402+0000 7f1bb67fc700 1 --2- 172.21.15.27:0/3243908845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb8155c50 0x7f1bb8158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.402+0000 7f1bb77fe700 1 --2- 172.21.15.27:0/3243908845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb8155c50 0x7f1bb8158f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.402+0000 7f1bb77fe700 1 -- 172.21.15.27:0/3243908845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb816d260 con 0x7f1bb80c8a90 2024-01-11T09:26:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.402+0000 7f1bb67fc700 1 --2- 172.21.15.27:0/3243908845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb8155c50 0x7f1bb8158f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:26:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.402+0000 7f1bb77fe700 1 --2- 172.21.15.27:0/3243908845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bb80c8a90 0x7f1bb80c8f10 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f1bb00094d0 tx=0x7f1bb000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.402+0000 7f1b9effd700 1 -- 172.21.15.27:0/3243908845 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb000e040 con 0x7f1bb80c8a90 2024-01-11T09:26:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.402+0000 7f1b9effd700 1 -- 172.21.15.27:0/3243908845 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb000f040 con 0x7f1bb80c8a90 2024-01-11T09:26:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.402+0000 7f1bbd7ff700 1 -- 172.21.15.27:0/3243908845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb816d540 con 0x7f1bb80c8a90 2024-01-11T09:26:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.402+0000 7f1b9effd700 1 -- 172.21.15.27:0/3243908845 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb000fe50 con 0x7f1bb80c8a90 2024-01-11T09:26:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.403+0000 7f1bbd7ff700 1 -- 172.21.15.27:0/3243908845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb8159460 con 0x7f1bb80c8a90 2024-01-11T09:26:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.403+0000 7f1bbd7ff700 1 -- 172.21.15.27:0/3243908845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb816da60 con 0x7f1bb80c8a90 2024-01-11T09:26:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.404+0000 7f1b9effd700 1 -- 172.21.15.27:0/3243908845 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 85) v1 ==== 84873+0+0 (secure 0 0 0) 0x7f1bb0015640 con 0x7f1bb80c8a90 2024-01-11T09:26:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.405+0000 7f1b9effd700 1 --2- 172.21.15.27:0/3243908845 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1ba40670a0 0x7f1ba4069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:16.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.406+0000 7f1b9effd700 1 -- 172.21.15.27:0/3243908845 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f1bb0085170 con 0x7f1bb80c8a90 2024-01-11T09:26:16.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.406+0000 7f1bb6ffd700 1 --2- 172.21.15.27:0/3243908845 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1ba40670a0 0x7f1ba4069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:16.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.407+0000 7f1bb6ffd700 1 --2- 172.21.15.27:0/3243908845 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1ba40670a0 0x7f1ba4069550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800c540 tx=0x7f1ba8015000 comp rx=0 tx=0).ready entity=mgr.8434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.410+0000 7f1b9effd700 1 -- 172.21.15.27:0/3243908845 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1bb816da60 con 0x7f1bb80c8a90 2024-01-11T09:26:16.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.640+0000 7f1bbd7ff700 1 -- 172.21.15.27:0/3243908845 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1bb816da60 con 0x7f1ba40670a0 2024-01-11T09:26:16.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.651+0000 7f1b9effd700 1 -- 172.21.15.27:0/3243908845 <== mgr.8434 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f1bb816da60 con 0x7f1ba40670a0 2024-01-11T09:26:16.648 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:26:16.648 INFO:teuthology.orchestra.run.smithi027.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:30:33", "first": "2024-01-11T08:31:00", "last": "2024-01-11T09:26:00", "last_pruned": "2024-01-11T09:26:00", "created_count": 56, "pruned_count": 1, "active": true}] 2024-01-11T09:26:16.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.652+0000 7f1bbd7ff700 1 -- 172.21.15.27:0/3243908845 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1ba40670a0 msgr2=0x7f1ba4069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.652+0000 7f1bbd7ff700 1 --2- 172.21.15.27:0/3243908845 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1ba40670a0 0x7f1ba4069550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800c540 tx=0x7f1ba8015000 comp rx=0 tx=0).stop 2024-01-11T09:26:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.652+0000 7f1bbd7ff700 1 -- 172.21.15.27:0/3243908845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bb80c8a90 msgr2=0x7f1bb80c8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.652+0000 7f1bbd7ff700 1 --2- 172.21.15.27:0/3243908845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bb80c8a90 0x7f1bb80c8f10 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f1bb00094d0 tx=0x7f1bb000b040 comp rx=0 tx=0).stop 2024-01-11T09:26:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.653+0000 7f1bbd7ff700 1 -- 172.21.15.27:0/3243908845 shutdown_connections 2024-01-11T09:26:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.653+0000 7f1bbd7ff700 1 --2- 172.21.15.27:0/3243908845 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1ba40670a0 0x7f1ba4069550 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.653+0000 7f1bbd7ff700 1 --2- 172.21.15.27:0/3243908845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bb80c8a90 0x7f1bb80c8f10 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.653+0000 7f1bbd7ff700 1 --2- 172.21.15.27:0/3243908845 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1ba0003a60 0x7f1bb8158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.653+0000 7f1bbd7ff700 1 --2- 172.21.15.27:0/3243908845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb8155c50 0x7f1bb8158f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.653+0000 7f1bbd7ff700 1 -- 172.21.15.27:0/3243908845 >> 172.21.15.27:0/3243908845 conn(0x7f1bb80680a0 msgr2=0x7f1bb80c0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.653+0000 7f1bbd7ff700 1 -- 172.21.15.27:0/3243908845 shutdown_connections 2024-01-11T09:26:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.653+0000 7f1bbd7ff700 1 -- 172.21.15.27:0/3243908845 wait complete. 2024-01-11T09:26:16.668 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T08:30:33", "first": "2024-01-11T08:31:00", "last": "2024-01-11T09:26:00", "last_pruned": "2024-01-11T09:26:00", "created_count": 56, "pruned_count": 1, "active": true}] 2024-01-11T09:26:16.668 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1', 'rel_path': '/snap_test_dir1', 'schedule': '1m', 'retention': {'n': 55}, 'start': '2024-01-11T00:00:00', 'created': '2024-01-11T08:30:33', 'first': '2024-01-11T08:31:00', 'last': '2024-01-11T09:26:00', 'last_pruned': '2024-01-11T09:26:00', 'created_count': 56, 'pruned_count': 1, 'active': True} 2024-01-11T09:26:16.668 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 55 2024-01-11T09:26:16.668 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 56 2024-01-11T09:26:16.668 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1 2024-01-11T09:26:16.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.872+0000 7f7b4e3ff700 1 Processor -- start 2024-01-11T09:26:16.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.872+0000 7f7b4e3ff700 1 -- start start 2024-01-11T09:26:16.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.874+0000 7f7b4e3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b48142dc0 0x7f7b480cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:16.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.874+0000 7f7b4e3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7b480cb2e0 0x7f7b480cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:16.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.874+0000 7f7b4e3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b480ccc70 con 0x7f7b48143190 2024-01-11T09:26:16.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.874+0000 7f7b4e3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b4815df30 con 0x7f7b480cb2e0 2024-01-11T09:26:16.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.874+0000 7f7b4e3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4815d320 con 0x7f7b48142dc0 2024-01-11T09:26:16.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.874+0000 7f7b477fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7b48143190 0x7f7b480cc020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51524/0 (socket says 172.21.15.27:51524) 2024-01-11T09:26:16.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.874+0000 7f7b477fe700 1 -- 172.21.15.27:0/3135246139 learned_addr learned my addr 172.21.15.27:0/3135246139 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:16.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.874+0000 7f7b4c99c700 1 --2- 172.21.15.27:0/3135246139 >> v2:172.21.15.77:3300/0 conn(0x7f7b480cb2e0 0x7f7b480cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:16.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.874+0000 7f7b47fff700 1 --2- 172.21.15.27:0/3135246139 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b48142dc0 0x7f7b480cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:16.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.874+0000 7f7b46ffd700 1 -- 172.21.15.27:0/3135246139 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945407797 0 0) 0x7f7b480ccc70 con 0x7f7b48143190 2024-01-11T09:26:16.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.875+0000 7f7b46ffd700 1 -- 172.21.15.27:0/3135246139 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b30003570 con 0x7f7b48143190 2024-01-11T09:26:16.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.875+0000 7f7b4c99c700 1 -- 172.21.15.27:0/3135246139 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b48142dc0 msgr2=0x7f7b480cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:16.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.875+0000 7f7b4c99c700 1 --2- 172.21.15.27:0/3135246139 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b48142dc0 0x7f7b480cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:16.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.875+0000 7f7b4c99c700 1 -- 172.21.15.27:0/3135246139 >> v1:172.21.15.27:6789/0 conn(0x7f7b48143190 legacy=0x7f7b480cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:16.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.875+0000 7f7b46ffd700 1 -- 172.21.15.27:0/3135246139 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1911534703 0 0) 0x7f7b30003570 con 0x7f7b48143190 2024-01-11T09:26:16.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.875+0000 7f7b4c99c700 1 -- 172.21.15.27:0/3135246139 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b480cb700 con 0x7f7b480cb2e0 2024-01-11T09:26:16.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.875+0000 7f7b47fff700 1 --2- 172.21.15.27:0/3135246139 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b48142dc0 0x7f7b480cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:26:16.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.875+0000 7f7b4c99c700 1 --2- 172.21.15.27:0/3135246139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b480cb2e0 0x7f7b480cc730 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f7b40009e60 tx=0x7f7b40009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bff36db60d84ec57 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:16.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.875+0000 7f7b46ffd700 1 -- 172.21.15.27:0/3135246139 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4000e070 con 0x7f7b480cb2e0 2024-01-11T09:26:16.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.876+0000 7f7b46ffd700 1 -- 172.21.15.27:0/3135246139 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4000b7a0 con 0x7f7b480cb2e0 2024-01-11T09:26:16.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.876+0000 7f7b46ffd700 1 -- 172.21.15.27:0/3135246139 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c001470 con 0x7f7b480cb2e0 2024-01-11T09:26:16.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.876+0000 7f7b4e3ff700 1 -- 172.21.15.27:0/3135246139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b480cb2e0 msgr2=0x7f7b480cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:16.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.876+0000 7f7b4e3ff700 1 --2- 172.21.15.27:0/3135246139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b480cb2e0 0x7f7b480cc730 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f7b40009e60 tx=0x7f7b40009780 comp rx=0 tx=0).stop 2024-01-11T09:26:16.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.876+0000 7f7b4e3ff700 1 -- 172.21.15.27:0/3135246139 shutdown_connections 2024-01-11T09:26:16.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.876+0000 7f7b4e3ff700 1 --2- 172.21.15.27:0/3135246139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b480cb2e0 0x7f7b480cc730 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:16.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.876+0000 7f7b4e3ff700 1 --2- 172.21.15.27:0/3135246139 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b48142dc0 0x7f7b480cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:16.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.876+0000 7f7b4e3ff700 1 -- 172.21.15.27:0/3135246139 >> 172.21.15.27:0/3135246139 conn(0x7f7b48063ca0 msgr2=0x7f7b4806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:16.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.876+0000 7f7b4e3ff700 1 -- 172.21.15.27:0/3135246139 shutdown_connections 2024-01-11T09:26:16.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.876+0000 7f7b4e3ff700 1 -- 172.21.15.27:0/3135246139 wait complete. 2024-01-11T09:26:16.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.878+0000 7f7b4e3ff700 1 Processor -- start 2024-01-11T09:26:16.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.878+0000 7f7b4e3ff700 1 -- start start 2024-01-11T09:26:16.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.879+0000 7f7b4e3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b480cb2e0 0x7f7b480c7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:16.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.879+0000 7f7b4e3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b48142dc0 0x7f7b4815ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:16.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.879+0000 7f7b4e3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b48143190 0x7f7b4815c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:16.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.879+0000 7f7b4e3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4813c7f0 con 0x7f7b48142dc0 2024-01-11T09:26:16.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.879+0000 7f7b4e3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4813bbe0 con 0x7f7b480cb2e0 2024-01-11T09:26:16.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.879+0000 7f7b4e3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4813bec0 con 0x7f7b48143190 2024-01-11T09:26:16.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.879+0000 7f7b477fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b48142dc0 0x7f7b4815ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:16.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.879+0000 7f7b477fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b48142dc0 0x7f7b4815ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55532/0 (socket says 172.21.15.27:55532) 2024-01-11T09:26:16.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.879+0000 7f7b477fe700 1 -- 172.21.15.27:0/1568933681 learned_addr learned my addr 172.21.15.27:0/1568933681 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:16.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.879+0000 7f7b4c99c700 1 --2- 172.21.15.27:0/1568933681 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b48143190 0x7f7b4815c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:16.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.879+0000 7f7b47fff700 1 --2- 172.21.15.27:0/1568933681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b480cb2e0 0x7f7b480c7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:16.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.879+0000 7f7b477fe700 1 -- 172.21.15.27:0/1568933681 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b48143190 msgr2=0x7f7b4815c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:16.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.879+0000 7f7b477fe700 1 --2- 172.21.15.27:0/1568933681 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b48143190 0x7f7b4815c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:16.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.879+0000 7f7b477fe700 1 -- 172.21.15.27:0/1568933681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b480cb2e0 msgr2=0x7f7b480c7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:16.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.879+0000 7f7b477fe700 1 --2- 172.21.15.27:0/1568933681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b480cb2e0 0x7f7b480c7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:16.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.879+0000 7f7b477fe700 1 -- 172.21.15.27:0/1568933681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b38009420 con 0x7f7b48142dc0 2024-01-11T09:26:16.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.879+0000 7f7b4c99c700 1 --2- 172.21.15.27:0/1568933681 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b48143190 0x7f7b4815c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:26:16.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.879+0000 7f7b47fff700 1 --2- 172.21.15.27:0/1568933681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b480cb2e0 0x7f7b480c7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:26:16.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.880+0000 7f7b477fe700 1 --2- 172.21.15.27:0/1568933681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b48142dc0 0x7f7b4815ba70 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00cb90 tx=0x7f7b3c001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:16.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.880+0000 7f7b2ffff700 1 -- 172.21.15.27:0/1568933681 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c001d40 con 0x7f7b48142dc0 2024-01-11T09:26:16.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.880+0000 7f7b2ffff700 1 -- 172.21.15.27:0/1568933681 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b3c007360 con 0x7f7b48142dc0 2024-01-11T09:26:16.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.880+0000 7f7b4e3ff700 1 -- 172.21.15.27:0/1568933681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b40009a20 con 0x7f7b48142dc0 2024-01-11T09:26:16.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.880+0000 7f7b2ffff700 1 -- 172.21.15.27:0/1568933681 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c004830 con 0x7f7b48142dc0 2024-01-11T09:26:16.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.880+0000 7f7b4e3ff700 1 -- 172.21.15.27:0/1568933681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4813c480 con 0x7f7b48142dc0 2024-01-11T09:26:16.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.881+0000 7f7b4e3ff700 1 -- 172.21.15.27:0/1568933681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b4813c960 con 0x7f7b48142dc0 2024-01-11T09:26:16.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.882+0000 7f7b2ffff700 1 -- 172.21.15.27:0/1568933681 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 85) v1 ==== 84873+0+0 (secure 0 0 0) 0x7f7b3c004ba0 con 0x7f7b48142dc0 2024-01-11T09:26:16.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.883+0000 7f7b2ffff700 1 --2- 172.21.15.27:0/1568933681 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7b340670a0 0x7f7b34069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:16.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.883+0000 7f7b2ffff700 1 -- 172.21.15.27:0/1568933681 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6121+0+0 (secure 0 0 0) 0x7f7b3c084200 con 0x7f7b48142dc0 2024-01-11T09:26:16.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.883+0000 7f7b47fff700 1 --2- 172.21.15.27:0/1568933681 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7b340670a0 0x7f7b34069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:16.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.884+0000 7f7b47fff700 1 --2- 172.21.15.27:0/1568933681 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7b340670a0 0x7f7b34069550 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7b380093f0 tx=0x7f7b38008040 comp rx=0 tx=0).ready entity=mgr.8434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:16.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:16.887+0000 7f7b2ffff700 1 -- 172.21.15.27:0/1568933681 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7b3c0511a0 con 0x7f7b48142dc0 2024-01-11T09:26:17.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:17.121+0000 7f7b4e3ff700 1 -- 172.21.15.27:0/1568933681 --> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b4804c3c0 con 0x7f7b340670a0 2024-01-11T09:26:17.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:17.208+0000 7f7b2ffff700 1 -- 172.21.15.27:0/1568933681 <== mgr.8434 v2:172.21.15.77:6835/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+42 (secure 0 0 0) 0x7f7b4804c3c0 con 0x7f7b340670a0 2024-01-11T09:26:17.206 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule removed for path /snap_test_dir1 2024-01-11T09:26:17.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:17.210+0000 7f7b4e3ff700 1 -- 172.21.15.27:0/1568933681 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7b340670a0 msgr2=0x7f7b34069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:17.210+0000 7f7b4e3ff700 1 --2- 172.21.15.27:0/1568933681 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7b340670a0 0x7f7b34069550 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7b380093f0 tx=0x7f7b38008040 comp rx=0 tx=0).stop 2024-01-11T09:26:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:17.210+0000 7f7b4e3ff700 1 -- 172.21.15.27:0/1568933681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b48142dc0 msgr2=0x7f7b4815ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:17.210+0000 7f7b4e3ff700 1 --2- 172.21.15.27:0/1568933681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b48142dc0 0x7f7b4815ba70 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00cb90 tx=0x7f7b3c001ab0 comp rx=0 tx=0).stop 2024-01-11T09:26:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:17.210+0000 7f7b4e3ff700 1 -- 172.21.15.27:0/1568933681 shutdown_connections 2024-01-11T09:26:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:17.211+0000 7f7b4e3ff700 1 --2- 172.21.15.27:0/1568933681 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f7b340670a0 0x7f7b34069550 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:17.211+0000 7f7b4e3ff700 1 --2- 172.21.15.27:0/1568933681 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b48143190 0x7f7b4815c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:17.211+0000 7f7b4e3ff700 1 --2- 172.21.15.27:0/1568933681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b480cb2e0 0x7f7b480c7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:17.211+0000 7f7b4e3ff700 1 --2- 172.21.15.27:0/1568933681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b48142dc0 0x7f7b4815ba70 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:17.211+0000 7f7b4e3ff700 1 -- 172.21.15.27:0/1568933681 >> 172.21.15.27:0/1568933681 conn(0x7f7b48063ca0 msgr2=0x7f7b48141d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:17.211+0000 7f7b4e3ff700 1 -- 172.21.15.27:0/1568933681 shutdown_connections 2024-01-11T09:26:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:17.211+0000 7f7b4e3ff700 1 -- 172.21.15.27:0/1568933681 wait complete. 2024-01-11T09:26:17.226 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1 2024-01-11T09:26:17.226 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:17.226 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:26:17.279 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_32_00_UTC 2024-01-11T09:26:17.279 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_33_00_UTC 2024-01-11T09:26:17.279 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_34_00_UTC 2024-01-11T09:26:17.279 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_35_00_UTC 2024-01-11T09:26:17.279 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_36_00_UTC 2024-01-11T09:26:17.279 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_37_00_UTC 2024-01-11T09:26:17.279 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_38_00_UTC 2024-01-11T09:26:17.279 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_39_00_UTC 2024-01-11T09:26:17.279 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_40_00_UTC 2024-01-11T09:26:17.279 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_41_00_UTC 2024-01-11T09:26:17.279 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_42_00_UTC 2024-01-11T09:26:17.280 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_43_00_UTC 2024-01-11T09:26:17.280 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_44_00_UTC 2024-01-11T09:26:17.280 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_45_00_UTC 2024-01-11T09:26:17.280 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_46_00_UTC 2024-01-11T09:26:17.280 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_47_00_UTC 2024-01-11T09:26:17.280 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_48_00_UTC 2024-01-11T09:26:17.280 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_49_00_UTC 2024-01-11T09:26:17.280 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_50_00_UTC 2024-01-11T09:26:17.280 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_51_00_UTC 2024-01-11T09:26:17.280 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_52_00_UTC 2024-01-11T09:26:17.280 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_53_00_UTC 2024-01-11T09:26:17.280 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_54_00_UTC 2024-01-11T09:26:17.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_55_00_UTC 2024-01-11T09:26:17.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_56_00_UTC 2024-01-11T09:26:17.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_57_00_UTC 2024-01-11T09:26:17.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_58_00_UTC 2024-01-11T09:26:17.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-08_59_00_UTC 2024-01-11T09:26:17.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_00_00_UTC 2024-01-11T09:26:17.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_01_00_UTC 2024-01-11T09:26:17.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_02_00_UTC 2024-01-11T09:26:17.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_03_00_UTC 2024-01-11T09:26:17.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_04_00_UTC 2024-01-11T09:26:17.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_05_00_UTC 2024-01-11T09:26:17.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_06_00_UTC 2024-01-11T09:26:17.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_07_00_UTC 2024-01-11T09:26:17.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_08_00_UTC 2024-01-11T09:26:17.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_09_00_UTC 2024-01-11T09:26:17.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_10_00_UTC 2024-01-11T09:26:17.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_11_00_UTC 2024-01-11T09:26:17.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_12_00_UTC 2024-01-11T09:26:17.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_13_00_UTC 2024-01-11T09:26:17.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_14_00_UTC 2024-01-11T09:26:17.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_15_00_UTC 2024-01-11T09:26:17.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_16_00_UTC 2024-01-11T09:26:17.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_17_00_UTC 2024-01-11T09:26:17.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_18_00_UTC 2024-01-11T09:26:17.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_19_00_UTC 2024-01-11T09:26:17.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_20_00_UTC 2024-01-11T09:26:17.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_21_00_UTC 2024-01-11T09:26:17.283 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_22_00_UTC 2024-01-11T09:26:17.283 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_23_00_UTC 2024-01-11T09:26:17.283 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_24_00_UTC 2024-01-11T09:26:17.283 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_25_00_UTC 2024-01-11T09:26:17.283 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_26_00_UTC 2024-01-11T09:26:17.284 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_32_00_UTC 2024-01-11T09:26:17.284 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:17.284 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_32_00_UTC) 2024-01-11T09:26:17.338 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_33_00_UTC 2024-01-11T09:26:17.339 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:17.339 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_33_00_UTC) 2024-01-11T09:26:17.393 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_34_00_UTC 2024-01-11T09:26:17.393 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:17.393 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_34_00_UTC) 2024-01-11T09:26:17.448 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_35_00_UTC 2024-01-11T09:26:17.448 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:17.448 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_35_00_UTC) 2024-01-11T09:26:17.503 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_36_00_UTC 2024-01-11T09:26:17.503 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:17.503 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_36_00_UTC) 2024-01-11T09:26:17.557 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_37_00_UTC 2024-01-11T09:26:17.558 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:17.558 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_37_00_UTC) 2024-01-11T09:26:17.613 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_38_00_UTC 2024-01-11T09:26:17.614 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:17.614 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_38_00_UTC) 2024-01-11T09:26:17.669 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_39_00_UTC 2024-01-11T09:26:17.669 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:17.669 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_39_00_UTC) 2024-01-11T09:26:17.725 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_40_00_UTC 2024-01-11T09:26:17.725 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:17.725 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_40_00_UTC) 2024-01-11T09:26:17.779 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_41_00_UTC 2024-01-11T09:26:17.780 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:17.780 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_41_00_UTC) 2024-01-11T09:26:17.834 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_42_00_UTC 2024-01-11T09:26:17.834 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:17.834 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_42_00_UTC) 2024-01-11T09:26:17.889 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_43_00_UTC 2024-01-11T09:26:17.889 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:17.889 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_43_00_UTC) 2024-01-11T09:26:17.944 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_44_00_UTC 2024-01-11T09:26:17.944 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:17.944 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_44_00_UTC) 2024-01-11T09:26:17.999 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_45_00_UTC 2024-01-11T09:26:17.999 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:17.999 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_45_00_UTC) 2024-01-11T09:26:18.053 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_46_00_UTC 2024-01-11T09:26:18.054 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:18.054 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_46_00_UTC) 2024-01-11T09:26:18.109 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_47_00_UTC 2024-01-11T09:26:18.109 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:18.109 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_47_00_UTC) 2024-01-11T09:26:18.163 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_48_00_UTC 2024-01-11T09:26:18.164 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:18.164 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_48_00_UTC) 2024-01-11T09:26:18.218 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_49_00_UTC 2024-01-11T09:26:18.218 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:18.219 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_49_00_UTC) 2024-01-11T09:26:18.273 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_50_00_UTC 2024-01-11T09:26:18.274 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:18.274 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_50_00_UTC) 2024-01-11T09:26:18.329 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_51_00_UTC 2024-01-11T09:26:18.329 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:18.329 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_51_00_UTC) 2024-01-11T09:26:18.383 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_52_00_UTC 2024-01-11T09:26:18.383 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:18.383 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_52_00_UTC) 2024-01-11T09:26:18.437 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_53_00_UTC 2024-01-11T09:26:18.437 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:18.437 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_53_00_UTC) 2024-01-11T09:26:18.491 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_54_00_UTC 2024-01-11T09:26:18.492 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:18.492 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_54_00_UTC) 2024-01-11T09:26:18.546 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_55_00_UTC 2024-01-11T09:26:18.546 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:18.546 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_55_00_UTC) 2024-01-11T09:26:18.600 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_56_00_UTC 2024-01-11T09:26:18.600 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:18.601 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_56_00_UTC) 2024-01-11T09:26:18.655 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_57_00_UTC 2024-01-11T09:26:18.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:18.655 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_57_00_UTC) 2024-01-11T09:26:18.710 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_58_00_UTC 2024-01-11T09:26:18.710 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:18.710 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_58_00_UTC) 2024-01-11T09:26:18.765 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-08_59_00_UTC 2024-01-11T09:26:18.765 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:18.765 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-08_59_00_UTC) 2024-01-11T09:26:18.819 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_00_00_UTC 2024-01-11T09:26:18.820 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:18.820 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_00_00_UTC) 2024-01-11T09:26:18.874 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_01_00_UTC 2024-01-11T09:26:18.874 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:18.874 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_01_00_UTC) 2024-01-11T09:26:18.929 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_02_00_UTC 2024-01-11T09:26:18.929 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:18.929 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_02_00_UTC) 2024-01-11T09:26:18.983 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_03_00_UTC 2024-01-11T09:26:18.984 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:18.984 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_03_00_UTC) 2024-01-11T09:26:19.039 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_04_00_UTC 2024-01-11T09:26:19.039 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:19.039 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_04_00_UTC) 2024-01-11T09:26:19.093 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_05_00_UTC 2024-01-11T09:26:19.093 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:19.093 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_05_00_UTC) 2024-01-11T09:26:19.147 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_06_00_UTC 2024-01-11T09:26:19.147 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:19.148 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_06_00_UTC) 2024-01-11T09:26:19.203 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_07_00_UTC 2024-01-11T09:26:19.203 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:19.203 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_07_00_UTC) 2024-01-11T09:26:19.258 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_08_00_UTC 2024-01-11T09:26:19.258 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:19.258 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_08_00_UTC) 2024-01-11T09:26:19.313 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_09_00_UTC 2024-01-11T09:26:19.313 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:19.313 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_09_00_UTC) 2024-01-11T09:26:19.368 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_10_00_UTC 2024-01-11T09:26:19.368 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:19.368 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_10_00_UTC) 2024-01-11T09:26:19.421 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_11_00_UTC 2024-01-11T09:26:19.421 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:19.421 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_11_00_UTC) 2024-01-11T09:26:19.476 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_12_00_UTC 2024-01-11T09:26:19.476 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:19.476 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_12_00_UTC) 2024-01-11T09:26:19.530 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_13_00_UTC 2024-01-11T09:26:19.530 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:19.530 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_13_00_UTC) 2024-01-11T09:26:19.586 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_14_00_UTC 2024-01-11T09:26:19.586 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:19.586 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_14_00_UTC) 2024-01-11T09:26:19.641 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_15_00_UTC 2024-01-11T09:26:19.641 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:19.641 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_15_00_UTC) 2024-01-11T09:26:19.695 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_16_00_UTC 2024-01-11T09:26:19.695 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:19.695 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_16_00_UTC) 2024-01-11T09:26:19.750 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_17_00_UTC 2024-01-11T09:26:19.750 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:19.750 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_17_00_UTC) 2024-01-11T09:26:19.804 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_18_00_UTC 2024-01-11T09:26:19.804 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:19.804 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_18_00_UTC) 2024-01-11T09:26:19.860 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_19_00_UTC 2024-01-11T09:26:19.861 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:19.861 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_19_00_UTC) 2024-01-11T09:26:19.917 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_20_00_UTC 2024-01-11T09:26:19.917 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:19.917 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_20_00_UTC) 2024-01-11T09:26:19.972 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_21_00_UTC 2024-01-11T09:26:19.973 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:19.973 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_21_00_UTC) 2024-01-11T09:26:20.070 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_22_00_UTC 2024-01-11T09:26:20.070 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:20.070 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_22_00_UTC) 2024-01-11T09:26:20.180 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_23_00_UTC 2024-01-11T09:26:20.180 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:20.181 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_23_00_UTC) 2024-01-11T09:26:20.259 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_24_00_UTC 2024-01-11T09:26:20.259 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:20.259 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_24_00_UTC) 2024-01-11T09:26:20.313 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_25_00_UTC 2024-01-11T09:26:20.313 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:20.313 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_25_00_UTC) 2024-01-11T09:26:20.419 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_26_00_UTC 2024-01-11T09:26:20.419 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:20.419 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_26_00_UTC) 2024-01-11T09:26:20.523 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:20.523 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-01-11T09:26:20.598 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-11T09:26:20.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.803+0000 7f51fe7ff700 1 Processor -- start 2024-01-11T09:26:20.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.803+0000 7f51fe7ff700 1 -- start start 2024-01-11T09:26:20.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.804+0000 7f51fe7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f51f8157410 0x7f51f80ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:20.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.804+0000 7f51fe7ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f51f8069540 0x7f51f80c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:20.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.804+0000 7f51fe7ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f80699d0 con 0x7f51f81577e0 2024-01-11T09:26:20.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.804+0000 7f51fe7ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f8069cb0 con 0x7f51f8069540 2024-01-11T09:26:20.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.804+0000 7f51fe7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f8069f90 con 0x7f51f8157410 2024-01-11T09:26:20.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.804+0000 7f51fcd9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f51f8069540 0x7f51f80c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:20.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.804+0000 7f51fcd9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f51f8069540 0x7f51f80c6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55534/0 (socket says 172.21.15.27:55534) 2024-01-11T09:26:20.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.804+0000 7f51fcd9c700 1 -- 172.21.15.27:0/867654768 learned_addr learned my addr 172.21.15.27:0/867654768 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:20.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.805+0000 7f51f7fff700 1 --2- 172.21.15.27:0/867654768 >> v2:172.21.15.77:3300/0 conn(0x7f51f8157410 0x7f51f80ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.805+0000 7f51fcd9c700 1 -- 172.21.15.27:0/867654768 >> v2:172.21.15.77:3300/0 conn(0x7f51f8157410 msgr2=0x7f51f80ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.805+0000 7f51fcd9c700 1 --2- 172.21.15.27:0/867654768 >> v2:172.21.15.77:3300/0 conn(0x7f51f8157410 0x7f51f80ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.805+0000 7f51fcd9c700 1 -- 172.21.15.27:0/867654768 >> v1:172.21.15.77:6789/0 conn(0x7f51f81577e0 legacy=0x7f51f80ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.805+0000 7f51fcd9c700 1 -- 172.21.15.27:0/867654768 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f806a270 con 0x7f51f8069540 2024-01-11T09:26:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.805+0000 7f51fcd9c700 1 --2- 172.21.15.27:0/867654768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f8069540 0x7f51f80c6ba0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f51f8070c90 tx=0x7f51f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c7b1dc47c056e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.805+0000 7f51f6ffd700 1 -- 172.21.15.27:0/867654768 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f000e070 con 0x7f51f8069540 2024-01-11T09:26:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.805+0000 7f51f6ffd700 1 -- 172.21.15.27:0/867654768 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f000ba20 con 0x7f51f8069540 2024-01-11T09:26:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.805+0000 7f51f6ffd700 1 -- 172.21.15.27:0/867654768 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f00092e0 con 0x7f51f8069540 2024-01-11T09:26:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.806+0000 7f51fe7ff700 1 -- 172.21.15.27:0/867654768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f8069540 msgr2=0x7f51f80c6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.806+0000 7f51fe7ff700 1 --2- 172.21.15.27:0/867654768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f8069540 0x7f51f80c6ba0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f51f8070c90 tx=0x7f51f0009840 comp rx=0 tx=0).stop 2024-01-11T09:26:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.806+0000 7f51fe7ff700 1 -- 172.21.15.27:0/867654768 shutdown_connections 2024-01-11T09:26:20.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.806+0000 7f51fe7ff700 1 --2- 172.21.15.27:0/867654768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f8069540 0x7f51f80c6ba0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:20.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.806+0000 7f51fe7ff700 1 --2- 172.21.15.27:0/867654768 >> v2:172.21.15.77:3300/0 conn(0x7f51f8157410 0x7f51f80ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:20.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.806+0000 7f51fe7ff700 1 -- 172.21.15.27:0/867654768 >> 172.21.15.27:0/867654768 conn(0x7f51f80680a0 msgr2=0x7f51f80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:20.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.806+0000 7f51fe7ff700 1 -- 172.21.15.27:0/867654768 shutdown_connections 2024-01-11T09:26:20.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.806+0000 7f51fe7ff700 1 -- 172.21.15.27:0/867654768 wait complete. 2024-01-11T09:26:20.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.808+0000 7f51fe7ff700 1 Processor -- start 2024-01-11T09:26:20.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.808+0000 7f51fe7ff700 1 -- start start 2024-01-11T09:26:20.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.809+0000 7f51fe7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f51f8069540 0x7f51f81551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:20.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.809+0000 7f51fe7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51f8157410 0x7f51f81556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:20.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.809+0000 7f51fe7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f81577e0 0x7f51f80c4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:20.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.809+0000 7f51fe7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f8156940 con 0x7f51f81577e0 2024-01-11T09:26:20.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.809+0000 7f51fe7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f8155d30 con 0x7f51f8157410 2024-01-11T09:26:20.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.809+0000 7f51fe7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f8156010 con 0x7f51f8069540 2024-01-11T09:26:20.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.809+0000 7f51fcd9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f81577e0 0x7f51f80c4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:20.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.809+0000 7f51f7fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f51f8069540 0x7f51f81551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:20.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.809+0000 7f51fcd9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f81577e0 0x7f51f80c4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55538/0 (socket says 172.21.15.27:55538) 2024-01-11T09:26:20.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.809+0000 7f51f7fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f51f8069540 0x7f51f81551b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:42836/0 (socket says 172.21.15.27:42836) 2024-01-11T09:26:20.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.809+0000 7f51fcd9c700 1 -- 172.21.15.27:0/3107804474 learned_addr learned my addr 172.21.15.27:0/3107804474 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:20.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.809+0000 7f51f77fe700 1 --2- 172.21.15.27:0/3107804474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51f8157410 0x7f51f81556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:20.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.809+0000 7f51fcd9c700 1 -- 172.21.15.27:0/3107804474 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f51f8069540 msgr2=0x7f51f81551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:20.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.809+0000 7f51fcd9c700 1 --2- 172.21.15.27:0/3107804474 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f51f8069540 0x7f51f81551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:20.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.809+0000 7f51fcd9c700 1 -- 172.21.15.27:0/3107804474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51f8157410 msgr2=0x7f51f81556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:20.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.809+0000 7f51fcd9c700 1 --2- 172.21.15.27:0/3107804474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51f8157410 0x7f51f81556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:20.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.809+0000 7f51fcd9c700 1 -- 172.21.15.27:0/3107804474 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f0009ae0 con 0x7f51f81577e0 2024-01-11T09:26:20.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.810+0000 7f51fcd9c700 1 --2- 172.21.15.27:0/3107804474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f81577e0 0x7f51f80c4240 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f51f0009880 tx=0x7f51f0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:20.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.810+0000 7f51dffff700 1 -- 172.21.15.27:0/3107804474 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f0008a80 con 0x7f51f81577e0 2024-01-11T09:26:20.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.810+0000 7f51dffff700 1 -- 172.21.15.27:0/3107804474 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f0008be0 con 0x7f51f81577e0 2024-01-11T09:26:20.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.810+0000 7f51dffff700 1 -- 172.21.15.27:0/3107804474 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f0025460 con 0x7f51f81577e0 2024-01-11T09:26:20.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.810+0000 7f51fe7ff700 1 -- 172.21.15.27:0/3107804474 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f81562f0 con 0x7f51f81577e0 2024-01-11T09:26:20.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.810+0000 7f51fe7ff700 1 -- 172.21.15.27:0/3107804474 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f813f590 con 0x7f51f81577e0 2024-01-11T09:26:20.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.811+0000 7f51fe7ff700 1 -- 172.21.15.27:0/3107804474 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f8156af0 con 0x7f51f81577e0 2024-01-11T09:26:20.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.812+0000 7f51dffff700 1 -- 172.21.15.27:0/3107804474 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 85) v1 ==== 84873+0+0 (secure 0 0 0) 0x7f51f0003040 con 0x7f51f81577e0 2024-01-11T09:26:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.814+0000 7f51dffff700 1 --2- 172.21.15.27:0/3107804474 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f51e40670a0 0x7f51e4069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.814+0000 7f51dffff700 1 -- 172.21.15.27:0/3107804474 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f51f0085f70 con 0x7f51f81577e0 2024-01-11T09:26:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.814+0000 7f51f7fff700 1 --2- 172.21.15.27:0/3107804474 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f51e40670a0 0x7f51e4069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:20.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.815+0000 7f51f7fff700 1 --2- 172.21.15.27:0/3107804474 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f51e40670a0 0x7f51e4069550 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009da0 tx=0x7f51e8009120 comp rx=0 tx=0).ready entity=mgr.8434 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:20.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:20.818+0000 7f51dffff700 1 -- 172.21.15.27:0/3107804474 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f51f8156af0 con 0x7f51f81577e0 2024-01-11T09:26:21.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:21.094+0000 7f51fe7ff700 1 -- 172.21.15.27:0/3107804474 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f51f804c3c0 con 0x7f51f81577e0 2024-01-11T09:26:22.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.017+0000 7f51dffff700 1 -- 172.21.15.27:0/3107804474 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 86) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f51f00531d0 con 0x7f51f81577e0 2024-01-11T09:26:22.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.018+0000 7f51dffff700 1 -- 172.21.15.27:0/3107804474 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v86) v1 ==== 93+0+0 (secure 0 0 0) 0x7f51f0052e00 con 0x7f51f81577e0 2024-01-11T09:26:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.021+0000 7f51fe7ff700 1 -- 172.21.15.27:0/3107804474 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f51e40670a0 msgr2=0x7f51e4069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.021+0000 7f51fe7ff700 1 --2- 172.21.15.27:0/3107804474 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f51e40670a0 0x7f51e4069550 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009da0 tx=0x7f51e8009120 comp rx=0 tx=0).stop 2024-01-11T09:26:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.021+0000 7f51fe7ff700 1 -- 172.21.15.27:0/3107804474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f81577e0 msgr2=0x7f51f80c4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.021+0000 7f51fe7ff700 1 --2- 172.21.15.27:0/3107804474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f81577e0 0x7f51f80c4240 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f51f0009880 tx=0x7f51f0018000 comp rx=0 tx=0).stop 2024-01-11T09:26:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.021+0000 7f51fe7ff700 1 -- 172.21.15.27:0/3107804474 shutdown_connections 2024-01-11T09:26:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.021+0000 7f51fe7ff700 1 --2- 172.21.15.27:0/3107804474 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f51e40670a0 0x7f51e4069550 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.021+0000 7f51fe7ff700 1 --2- 172.21.15.27:0/3107804474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51f81577e0 0x7f51f80c4240 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:22.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.021+0000 7f51fe7ff700 1 --2- 172.21.15.27:0/3107804474 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f51f8069540 0x7f51f81551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.021+0000 7f51fe7ff700 1 --2- 172.21.15.27:0/3107804474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51f8157410 0x7f51f81556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.021+0000 7f51fe7ff700 1 -- 172.21.15.27:0/3107804474 >> 172.21.15.27:0/3107804474 conn(0x7f51f80680a0 msgr2=0x7f51f80af320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.021+0000 7f51fe7ff700 1 -- 172.21.15.27:0/3107804474 shutdown_connections 2024-01-11T09:26:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.021+0000 7f51fe7ff700 1 -- 172.21.15.27:0/3107804474 wait complete. 2024-01-11T09:26:22.050 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T09:26:22.050 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:26:22.164 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:26:22.257 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:22.260+0000 7f5cd60ea1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:26:22.258 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T09:26:22.258 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-11T09:26:22.258 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:22.258 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-01-11T09:26:22.265 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:22.268+0000 7f96802e81c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:26:22.350 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T09:26:22.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:26:22.351 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-11T09:26:22.421 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:26:22.441 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:22.445+0000 7f5cd60ea1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:26:22.477 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:22.480+0000 7f96802e81c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:26:22.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.661+0000 7f13b29ff700 1 Processor -- start 2024-01-11T09:26:22.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.661+0000 7f13b29ff700 1 -- start start 2024-01-11T09:26:22.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.662+0000 7f13b29ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f13ac0b19d0 0x7f13ac158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:22.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.663+0000 7f13b29ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13ac06b610 0x7f13ac159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:22.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.663+0000 7f13b29ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ac158b60 con 0x7f13ac1575e0 2024-01-11T09:26:22.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.663+0000 7f13b29ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13ac0cb5e0 con 0x7f13ac06b610 2024-01-11T09:26:22.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.663+0000 7f13b29ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13ac1599b0 con 0x7f13ac0b19d0 2024-01-11T09:26:22.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.663+0000 7f13abfff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f13ac1575e0 0x7f13ac0b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51530/0 (socket says 172.21.15.27:51530) 2024-01-11T09:26:22.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.663+0000 7f13b0f9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13ac06b610 0x7f13ac159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:22.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.663+0000 7f13abfff700 1 -- 172.21.15.27:0/1458171729 learned_addr learned my addr 172.21.15.27:0/1458171729 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:22.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.663+0000 7f13ab7fe700 1 --2- 172.21.15.27:0/1458171729 >> v2:172.21.15.77:3300/0 conn(0x7f13ac0b19d0 0x7f13ac158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:22.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.663+0000 7f13aaffd700 1 -- 172.21.15.27:0/1458171729 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180482449 0 0) 0x7f139c001110 con 0x7f13ac1575e0 2024-01-11T09:26:22.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.663+0000 7f13aaffd700 1 -- 172.21.15.27:0/1458171729 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1394003570 con 0x7f13ac1575e0 2024-01-11T09:26:22.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.663+0000 7f13b0f9c700 1 -- 172.21.15.27:0/1458171729 >> v2:172.21.15.77:3300/0 conn(0x7f13ac0b19d0 msgr2=0x7f13ac158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:22.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.663+0000 7f13b0f9c700 1 --2- 172.21.15.27:0/1458171729 >> v2:172.21.15.77:3300/0 conn(0x7f13ac0b19d0 0x7f13ac158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:22.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.663+0000 7f13b0f9c700 1 -- 172.21.15.27:0/1458171729 >> v1:172.21.15.27:6789/0 conn(0x7f13ac1575e0 legacy=0x7f13ac0b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:22.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.664+0000 7f13b0f9c700 1 -- 172.21.15.27:0/1458171729 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ac0c0f90 con 0x7f13ac06b610 2024-01-11T09:26:22.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.664+0000 7f13ab7fe700 1 --2- 172.21.15.27:0/1458171729 >> v2:172.21.15.77:3300/0 conn(0x7f13ac0b19d0 0x7f13ac158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:26:22.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.664+0000 7f13b0f9c700 1 --2- 172.21.15.27:0/1458171729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13ac06b610 0x7f13ac159e40 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f13ac064fe0 tx=0x7f13a4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11db0b20c0036e82 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:22.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.664+0000 7f13aaffd700 1 -- 172.21.15.27:0/1458171729 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a400e070 con 0x7f13ac06b610 2024-01-11T09:26:22.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.664+0000 7f13aaffd700 1 -- 172.21.15.27:0/1458171729 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a400bab0 con 0x7f13ac06b610 2024-01-11T09:26:22.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.664+0000 7f13aaffd700 1 -- 172.21.15.27:0/1458171729 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a400be70 con 0x7f13ac06b610 2024-01-11T09:26:22.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.665+0000 7f13b29ff700 1 -- 172.21.15.27:0/1458171729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13ac06b610 msgr2=0x7f13ac159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:22.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.665+0000 7f13b29ff700 1 --2- 172.21.15.27:0/1458171729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13ac06b610 0x7f13ac159e40 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f13ac064fe0 tx=0x7f13a4009780 comp rx=0 tx=0).stop 2024-01-11T09:26:22.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.665+0000 7f13b29ff700 1 -- 172.21.15.27:0/1458171729 shutdown_connections 2024-01-11T09:26:22.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.665+0000 7f13b29ff700 1 --2- 172.21.15.27:0/1458171729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13ac06b610 0x7f13ac159e40 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:22.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.665+0000 7f13b29ff700 1 --2- 172.21.15.27:0/1458171729 >> v2:172.21.15.77:3300/0 conn(0x7f13ac0b19d0 0x7f13ac158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:22.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.665+0000 7f13b29ff700 1 -- 172.21.15.27:0/1458171729 >> 172.21.15.27:0/1458171729 conn(0x7f13ac0680e0 msgr2=0x7f13ac069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:22.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.665+0000 7f13b29ff700 1 -- 172.21.15.27:0/1458171729 shutdown_connections 2024-01-11T09:26:22.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.665+0000 7f13b29ff700 1 -- 172.21.15.27:0/1458171729 wait complete. 2024-01-11T09:26:22.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.666+0000 7f13b29ff700 1 Processor -- start 2024-01-11T09:26:22.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.667+0000 7f13b29ff700 1 -- start start 2024-01-11T09:26:22.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.667+0000 7f13b29ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac06b610 0x7f13ac15fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:22.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.668+0000 7f13b29ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13ac0b19d0 0x7f13ac0c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:22.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.668+0000 7f13b29ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13ac1575e0 0x7f13ac155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:22.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.668+0000 7f13b29ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac1602c0 con 0x7f13ac1575e0 2024-01-11T09:26:22.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.668+0000 7f13b29ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac15f8f0 con 0x7f13ac06b610 2024-01-11T09:26:22.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.668+0000 7f13b29ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac15ece0 con 0x7f13ac0b19d0 2024-01-11T09:26:22.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.668+0000 7f13b0f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13ac1575e0 0x7f13ac155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:22.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.668+0000 7f13abfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac06b610 0x7f13ac15fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:22.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.668+0000 7f13abfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac06b610 0x7f13ac15fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:42206/0 (socket says 172.21.15.27:42206) 2024-01-11T09:26:22.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.668+0000 7f13abfff700 1 -- 172.21.15.27:0/1134427926 learned_addr learned my addr 172.21.15.27:0/1134427926 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:22.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.668+0000 7f13ab7fe700 1 --2- 172.21.15.27:0/1134427926 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13ac0b19d0 0x7f13ac0c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:22.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.668+0000 7f13abfff700 1 -- 172.21.15.27:0/1134427926 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13ac0b19d0 msgr2=0x7f13ac0c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:22.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.668+0000 7f13abfff700 1 --2- 172.21.15.27:0/1134427926 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13ac0b19d0 0x7f13ac0c85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:22.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.668+0000 7f13abfff700 1 -- 172.21.15.27:0/1134427926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13ac1575e0 msgr2=0x7f13ac155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:22.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.668+0000 7f13abfff700 1 --2- 172.21.15.27:0/1134427926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13ac1575e0 0x7f13ac155920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:22.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.668+0000 7f13abfff700 1 -- 172.21.15.27:0/1134427926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a0009420 con 0x7f13ac06b610 2024-01-11T09:26:22.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.669+0000 7f13abfff700 1 --2- 172.21.15.27:0/1134427926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac06b610 0x7f13ac15fd80 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f139c00c590 tx=0x7f139c007c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:22.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.669+0000 7f1393fff700 1 -- 172.21.15.27:0/1134427926 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c005b30 con 0x7f13ac06b610 2024-01-11T09:26:22.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.669+0000 7f1393fff700 1 -- 172.21.15.27:0/1134427926 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f139c005cb0 con 0x7f13ac06b610 2024-01-11T09:26:22.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.669+0000 7f13b29ff700 1 -- 172.21.15.27:0/1134427926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a4009a20 con 0x7f13ac06b610 2024-01-11T09:26:22.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.669+0000 7f1393fff700 1 -- 172.21.15.27:0/1134427926 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c007320 con 0x7f13ac06b610 2024-01-11T09:26:22.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.669+0000 7f13b29ff700 1 -- 172.21.15.27:0/1134427926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ac15f320 con 0x7f13ac06b610 2024-01-11T09:26:22.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.670+0000 7f13b29ff700 1 -- 172.21.15.27:0/1134427926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ac15fbd0 con 0x7f13ac06b610 2024-01-11T09:26:22.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.671+0000 7f1393fff700 1 -- 172.21.15.27:0/1134427926 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 86) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f139c01b3b0 con 0x7f13ac06b610 2024-01-11T09:26:22.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.672+0000 7f1393fff700 1 --2- 172.21.15.27:0/1134427926 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1398067050 0x7f1398069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:22.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.672+0000 7f1393fff700 1 -- 172.21.15.27:0/1134427926 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f139c083c10 con 0x7f13ac06b610 2024-01-11T09:26:22.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.672+0000 7f13ab7fe700 1 -- 172.21.15.27:0/1134427926 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1398067050 msgr2=0x7f1398069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:22.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.672+0000 7f13ab7fe700 1 --2- 172.21.15.27:0/1134427926 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1398067050 0x7f1398069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:26:22.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.677+0000 7f1393fff700 1 -- 172.21.15.27:0/1134427926 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f139c050b00 con 0x7f13ac06b610 2024-01-11T09:26:22.770 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:22.774+0000 7f5cd60ea1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:26:22.846 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:22.849+0000 7f96802e81c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:26:22.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.873+0000 7f13ab7fe700 1 -- 172.21.15.27:0/1134427926 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1398067050 msgr2=0x7f1398069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:22.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.873+0000 7f13ab7fe700 1 --2- 172.21.15.27:0/1134427926 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1398067050 0x7f1398069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:26:22.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.942+0000 7f13b29ff700 1 -- 172.21.15.27:0/1134427926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13ac158b60 con 0x7f13ac06b610 2024-01-11T09:26:22.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.943+0000 7f1393fff700 1 -- 172.21.15.27:0/1134427926 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4596 (secure 0 0 0) 0x7f13ac158b60 con 0x7f13ac06b610 2024-01-11T09:26:22.940 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:26:22.940 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7352,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3818350333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3818350333},{"type":"v1","addr":"172.21.15.27:6834","nonce":3818350333}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/217390254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":217390254},{"type":"v1","addr":"172.21.15.27:6835","nonce":217390254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2824437717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2824437717},{"type":"v1","addr":"172.21.15.77:6834","nonce":2824437717}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:30:15.623796+0000","modified":"2024-01-11T08:30:19.150622+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8601":{"gid":8601,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/1641163931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1641163931},{"type":"v1","addr":"172.21.15.77:6837","nonce":1641163931}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2024-01-11T09:26:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.946+0000 7f13b29ff700 1 -- 172.21.15.27:0/1134427926 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1398067050 msgr2=0x7f1398069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:26:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.946+0000 7f13b29ff700 1 --2- 172.21.15.27:0/1134427926 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1398067050 0x7f1398069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.946+0000 7f13b29ff700 1 -- 172.21.15.27:0/1134427926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac06b610 msgr2=0x7f13ac15fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.946+0000 7f13b29ff700 1 --2- 172.21.15.27:0/1134427926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac06b610 0x7f13ac15fd80 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f139c00c590 tx=0x7f139c007c70 comp rx=0 tx=0).stop 2024-01-11T09:26:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.946+0000 7f13b29ff700 1 -- 172.21.15.27:0/1134427926 shutdown_connections 2024-01-11T09:26:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.946+0000 7f13b29ff700 1 --2- 172.21.15.27:0/1134427926 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f1398067050 0x7f1398069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.946+0000 7f13b29ff700 1 --2- 172.21.15.27:0/1134427926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13ac1575e0 0x7f13ac155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.946+0000 7f13b29ff700 1 --2- 172.21.15.27:0/1134427926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13ac06b610 0x7f13ac15fd80 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.946+0000 7f13b29ff700 1 --2- 172.21.15.27:0/1134427926 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13ac0b19d0 0x7f13ac0c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.946+0000 7f13b29ff700 1 -- 172.21.15.27:0/1134427926 >> 172.21.15.27:0/1134427926 conn(0x7f13ac0680e0 msgr2=0x7f13ac0695b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.946+0000 7f13b29ff700 1 -- 172.21.15.27:0/1134427926 shutdown_connections 2024-01-11T09:26:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:22.946+0000 7f13b29ff700 1 -- 172.21.15.27:0/1134427926 wait complete. 2024-01-11T09:26:22.944 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 77 2024-01-11T09:26:22.961 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:26:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.168+0000 7fb1e89ff700 1 Processor -- start 2024-01-11T09:26:23.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.169+0000 7fb1e89ff700 1 -- start start 2024-01-11T09:26:23.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.170+0000 7fb1e89ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb1e406b610 0x7fb1e4158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:23.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.170+0000 7fb1e89ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e4158b60 con 0x7fb1e40b19d0 2024-01-11T09:26:23.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.170+0000 7fb1e89ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e41599b0 con 0x7fb1e41575e0 2024-01-11T09:26:23.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.170+0000 7fb1e89ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e40cb5e0 con 0x7fb1e406b610 2024-01-11T09:26:23.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.170+0000 7fb1e1d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb1e40b19d0 0x7fb1e4159e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51546/0 (socket says 172.21.15.27:51546) 2024-01-11T09:26:23.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.170+0000 7fb1e1d9b700 1 -- 172.21.15.27:0/2720923429 learned_addr learned my addr 172.21.15.27:0/2720923429 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:23.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.170+0000 7fb1e2d9d700 1 --2- 172.21.15.27:0/2720923429 >> v2:172.21.15.77:3300/0 conn(0x7fb1e406b610 0x7fb1e4158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:23.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.171+0000 7fb1e159a700 1 -- 172.21.15.27:0/2720923429 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135448926 0 0) 0x7fb1d8001110 con 0x7fb1e40b19d0 2024-01-11T09:26:23.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.171+0000 7fb1e159a700 1 -- 172.21.15.27:0/2720923429 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1cc003570 con 0x7fb1e40b19d0 2024-01-11T09:26:23.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.171+0000 7fb1e2d9d700 1 -- 172.21.15.27:0/2720923429 >> v1:172.21.15.77:6789/0 conn(0x7fb1e41575e0 legacy=0x7fb1e40b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:23.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.171+0000 7fb1e159a700 1 -- 172.21.15.27:0/2720923429 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567943764 0 0) 0x7fb1e41599b0 con 0x7fb1e41575e0 2024-01-11T09:26:23.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.171+0000 7fb1e2d9d700 1 -- 172.21.15.27:0/2720923429 >> v1:172.21.15.27:6789/0 conn(0x7fb1e40b19d0 legacy=0x7fb1e4159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:23.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.171+0000 7fb1e2d9d700 1 -- 172.21.15.27:0/2720923429 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e40c0f90 con 0x7fb1e406b610 2024-01-11T09:26:23.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.171+0000 7fb1e2d9d700 1 --2- 172.21.15.27:0/2720923429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e406b610 0x7fb1e4158620 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc009e60 tx=0x7fb1dc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbc811eb4903d072 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:23.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.172+0000 7fb1e159a700 1 -- 172.21.15.27:0/2720923429 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc00e070 con 0x7fb1e406b610 2024-01-11T09:26:23.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.172+0000 7fb1e159a700 1 -- 172.21.15.27:0/2720923429 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1dc00bb60 con 0x7fb1e406b610 2024-01-11T09:26:23.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.172+0000 7fb1e159a700 1 -- 172.21.15.27:0/2720923429 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc012430 con 0x7fb1e406b610 2024-01-11T09:26:23.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.172+0000 7fb1e89ff700 1 -- 172.21.15.27:0/2720923429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e406b610 msgr2=0x7fb1e4158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:23.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.172+0000 7fb1e89ff700 1 --2- 172.21.15.27:0/2720923429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e406b610 0x7fb1e4158620 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc009e60 tx=0x7fb1dc009780 comp rx=0 tx=0).stop 2024-01-11T09:26:23.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.172+0000 7fb1e89ff700 1 -- 172.21.15.27:0/2720923429 shutdown_connections 2024-01-11T09:26:23.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.172+0000 7fb1e89ff700 1 --2- 172.21.15.27:0/2720923429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e406b610 0x7fb1e4158620 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:23.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.172+0000 7fb1e89ff700 1 -- 172.21.15.27:0/2720923429 >> 172.21.15.27:0/2720923429 conn(0x7fb1e40680e0 msgr2=0x7fb1e4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:23.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.172+0000 7fb1e89ff700 1 -- 172.21.15.27:0/2720923429 shutdown_connections 2024-01-11T09:26:23.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.172+0000 7fb1e89ff700 1 -- 172.21.15.27:0/2720923429 wait complete. 2024-01-11T09:26:23.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.174+0000 7fb1e89ff700 1 Processor -- start 2024-01-11T09:26:23.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.174+0000 7fb1e89ff700 1 -- start start 2024-01-11T09:26:23.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.175+0000 7fb1e89ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb1e406b610 0x7fb1e4153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:23.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.175+0000 7fb1e89ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1e40b19d0 0x7fb1e40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:23.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.175+0000 7fb1e89ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e41575e0 0x7fb1e41694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:23.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.175+0000 7fb1e89ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e40a5490 con 0x7fb1e40b19d0 2024-01-11T09:26:23.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.175+0000 7fb1e89ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e40a5cf0 con 0x7fb1e41575e0 2024-01-11T09:26:23.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.175+0000 7fb1e89ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e4156410 con 0x7fb1e406b610 2024-01-11T09:26:23.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.175+0000 7fb1e1d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1e40b19d0 0x7fb1e40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:23.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.175+0000 7fb1e1d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1e40b19d0 0x7fb1e40a6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55582/0 (socket says 172.21.15.27:55582) 2024-01-11T09:26:23.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.175+0000 7fb1e1d9b700 1 -- 172.21.15.27:0/2084656121 learned_addr learned my addr 172.21.15.27:0/2084656121 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:23.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.176+0000 7fb1e259c700 1 --2- 172.21.15.27:0/2084656121 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb1e406b610 0x7fb1e4153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:23.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.176+0000 7fb1e2d9d700 1 --2- 172.21.15.27:0/2084656121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e41575e0 0x7fb1e41694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:23.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.176+0000 7fb1e1d9b700 1 -- 172.21.15.27:0/2084656121 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb1e406b610 msgr2=0x7fb1e4153d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:23.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.176+0000 7fb1e1d9b700 1 --2- 172.21.15.27:0/2084656121 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb1e406b610 0x7fb1e4153d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:23.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.176+0000 7fb1e1d9b700 1 -- 172.21.15.27:0/2084656121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e41575e0 msgr2=0x7fb1e41694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:23.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.176+0000 7fb1e1d9b700 1 --2- 172.21.15.27:0/2084656121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e41575e0 0x7fb1e41694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:23.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.176+0000 7fb1e1d9b700 1 -- 172.21.15.27:0/2084656121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1dc009a20 con 0x7fb1e40b19d0 2024-01-11T09:26:23.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.176+0000 7fb1e1d9b700 1 --2- 172.21.15.27:0/2084656121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1e40b19d0 0x7fb1e40a6180 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fb1d800c7c0 tx=0x7fb1d8007c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:23.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.176+0000 7fb1d27fc700 1 -- 172.21.15.27:0/2084656121 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8005b30 con 0x7fb1e40b19d0 2024-01-11T09:26:23.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.177+0000 7fb1d27fc700 1 -- 172.21.15.27:0/2084656121 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d8005c90 con 0x7fb1e40b19d0 2024-01-11T09:26:23.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.177+0000 7fb1e89ff700 1 -- 172.21.15.27:0/2084656121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e40a56a0 con 0x7fb1e40b19d0 2024-01-11T09:26:23.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.177+0000 7fb1d27fc700 1 -- 172.21.15.27:0/2084656121 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8007300 con 0x7fb1e40b19d0 2024-01-11T09:26:23.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.177+0000 7fb1e89ff700 1 -- 172.21.15.27:0/2084656121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e4154240 con 0x7fb1e40b19d0 2024-01-11T09:26:23.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.178+0000 7fb1e89ff700 1 -- 172.21.15.27:0/2084656121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e40718d0 con 0x7fb1e40b19d0 2024-01-11T09:26:23.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.179+0000 7fb1d27fc700 1 -- 172.21.15.27:0/2084656121 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 86) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fb1d80076b0 con 0x7fb1e40b19d0 2024-01-11T09:26:23.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.181+0000 7fb1d27fc700 1 --2- 172.21.15.27:0/2084656121 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb1c4067050 0x7fb1c4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:23.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.181+0000 7fb1d27fc700 1 -- 172.21.15.27:0/2084656121 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6165+0+0 (secure 0 0 0) 0x7fb1d8083b90 con 0x7fb1e40b19d0 2024-01-11T09:26:23.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.181+0000 7fb1e259c700 1 -- 172.21.15.27:0/2084656121 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb1c4067050 msgr2=0x7fb1c4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:23.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.181+0000 7fb1e259c700 1 --2- 172.21.15.27:0/2084656121 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb1c4067050 0x7fb1c4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:26:23.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.185+0000 7fb1d27fc700 1 -- 172.21.15.27:0/2084656121 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb1d8050a80 con 0x7fb1e40b19d0 2024-01-11T09:26:23.306 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:23.309+0000 7f5cd60ea1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:26:23.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.382+0000 7fb1e259c700 1 -- 172.21.15.27:0/2084656121 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb1c4067050 msgr2=0x7fb1c4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:23.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.382+0000 7fb1e259c700 1 --2- 172.21.15.27:0/2084656121 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb1c4067050 0x7fb1c4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:26:23.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.390+0000 7fb1e89ff700 1 -- 172.21.15.27:0/2084656121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1e4158b60 con 0x7fb1e40b19d0 2024-01-11T09:26:23.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.391+0000 7fb1d27fc700 1 -- 172.21.15.27:0/2084656121 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v153) v1 ==== 74+0+16033 (secure 0 0 0) 0x7fb1e4158b60 con 0x7fb1e40b19d0 2024-01-11T09:26:23.388 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:26:23.388 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":153,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:26:22.013258+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:30:13.161778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-01-11T08:30:14.620026+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":112,"snap_epoch":153,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.3","mappings":[{"from":2,"to":0}]},{"pgid":"19.1f","mappings":[{"from":2,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1967107807":"2024-01-12T08:30:31.980475+0000","172.21.15.77:6836/41381":"2024-01-12T08:30:31.980475+0000","172.21.15.77:6835/41381":"2024-01-12T08:30:31.980475+0000","172.21.15.77:0/864224333":"2024-01-12T08:30:31.980475+0000","172.21.15.77:0/4073869382":"2024-01-12T08:30:31.980475+0000","172.21.15.77:0/1400063768":"2024-01-12T08:30:31.980475+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":54},{"begin":59,"length":54}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":51,"length":6},{"begin":107,"length":6}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:26:23.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.393+0000 7fb1e89ff700 1 -- 172.21.15.27:0/2084656121 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb1c4067050 msgr2=0x7fb1c4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:26:23.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.393+0000 7fb1e89ff700 1 --2- 172.21.15.27:0/2084656121 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb1c4067050 0x7fb1c4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:23.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.393+0000 7fb1e89ff700 1 -- 172.21.15.27:0/2084656121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1e40b19d0 msgr2=0x7fb1e40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:23.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.393+0000 7fb1e89ff700 1 --2- 172.21.15.27:0/2084656121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1e40b19d0 0x7fb1e40a6180 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fb1d800c7c0 tx=0x7fb1d8007c30 comp rx=0 tx=0).stop 2024-01-11T09:26:23.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.393+0000 7fb1e89ff700 1 -- 172.21.15.27:0/2084656121 shutdown_connections 2024-01-11T09:26:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.393+0000 7fb1e89ff700 1 --2- 172.21.15.27:0/2084656121 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb1c4067050 0x7fb1c4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.393+0000 7fb1e89ff700 1 --2- 172.21.15.27:0/2084656121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1e41575e0 0x7fb1e41694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.393+0000 7fb1e89ff700 1 --2- 172.21.15.27:0/2084656121 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb1e406b610 0x7fb1e4153d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.393+0000 7fb1e89ff700 1 --2- 172.21.15.27:0/2084656121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1e40b19d0 0x7fb1e40a6180 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.393+0000 7fb1e89ff700 1 -- 172.21.15.27:0/2084656121 >> 172.21.15.27:0/2084656121 conn(0x7fb1e40680e0 msgr2=0x7fb1e406e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.394+0000 7fb1e89ff700 1 -- 172.21.15.27:0/2084656121 shutdown_connections 2024-01-11T09:26:23.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.394+0000 7fb1e89ff700 1 -- 172.21.15.27:0/2084656121 wait complete. 2024-01-11T09:26:23.409 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T08:30:13.161778+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T08:30:14.620026+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 112, 'snap_epoch': 153, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T09:26:23.409 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T09:26:23.425 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:23.429+0000 7f96802e81c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:26:23.476 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:23.479+0000 7f5cd60ea1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:26:23.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.617+0000 7f06161ff700 1 Processor -- start 2024-01-11T09:26:23.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.618+0000 7f06161ff700 1 -- start start 2024-01-11T09:26:23.614 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:23.618+0000 7f96802e81c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:26:23.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.619+0000 7f06161ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f061015a1b0 0x7f061015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:23.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.619+0000 7f06161ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f061015ae70 0x7f061015b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.619+0000 7f06161ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06100cf850 con 0x7f06100ccd70 2024-01-11T09:26:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.619+0000 7f06161ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06100cec40 con 0x7f061015a1b0 2024-01-11T09:26:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.619+0000 7f06161ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06100cef20 con 0x7f061015ae70 2024-01-11T09:26:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.619+0000 7f060f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f061015a1b0 0x7f061015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.619+0000 7f060f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f061015a1b0 0x7f061015a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55588/0 (socket says 172.21.15.27:55588) 2024-01-11T09:26:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.619+0000 7f060f7fe700 1 -- 172.21.15.27:0/1111592597 learned_addr learned my addr 172.21.15.27:0/1111592597 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.619+0000 7f060ffff700 1 --2- 172.21.15.27:0/1111592597 >> v2:172.21.15.77:3300/0 conn(0x7f061015ae70 0x7f061015b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.620+0000 7f060f7fe700 1 -- 172.21.15.27:0/1111592597 >> v2:172.21.15.77:3300/0 conn(0x7f061015ae70 msgr2=0x7f061015b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.620+0000 7f060f7fe700 1 --2- 172.21.15.27:0/1111592597 >> v2:172.21.15.77:3300/0 conn(0x7f061015ae70 0x7f061015b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.620+0000 7f060f7fe700 1 -- 172.21.15.27:0/1111592597 >> v1:172.21.15.77:6789/0 conn(0x7f06100ccd70 legacy=0x7f06100cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.620+0000 7f060f7fe700 1 -- 172.21.15.27:0/1111592597 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06100cf200 con 0x7f061015a1b0 2024-01-11T09:26:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.620+0000 7f060ffff700 1 --2- 172.21.15.27:0/1111592597 >> v2:172.21.15.77:3300/0 conn(0x7f061015ae70 0x7f061015b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.620+0000 7f060f7fe700 1 --2- 172.21.15.27:0/1111592597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f061015a1b0 0x7f061015a580 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f0610089810 tx=0x7f0600009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec8fe3f3834143ad server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.620+0000 7f060e7fc700 1 -- 172.21.15.27:0/1111592597 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060000b860 con 0x7f061015a1b0 2024-01-11T09:26:23.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.620+0000 7f060e7fc700 1 -- 172.21.15.27:0/1111592597 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f060000b9c0 con 0x7f061015a1b0 2024-01-11T09:26:23.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.620+0000 7f060e7fc700 1 -- 172.21.15.27:0/1111592597 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060000bd60 con 0x7f061015a1b0 2024-01-11T09:26:23.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.621+0000 7f06161ff700 1 -- 172.21.15.27:0/1111592597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f061015a1b0 msgr2=0x7f061015a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:23.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.621+0000 7f06161ff700 1 --2- 172.21.15.27:0/1111592597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f061015a1b0 0x7f061015a580 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f0610089810 tx=0x7f0600009250 comp rx=0 tx=0).stop 2024-01-11T09:26:23.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.621+0000 7f06161ff700 1 -- 172.21.15.27:0/1111592597 shutdown_connections 2024-01-11T09:26:23.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.621+0000 7f06161ff700 1 --2- 172.21.15.27:0/1111592597 >> v2:172.21.15.77:3300/0 conn(0x7f061015ae70 0x7f061015b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:23.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.621+0000 7f06161ff700 1 --2- 172.21.15.27:0/1111592597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f061015a1b0 0x7f061015a580 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:23.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.621+0000 7f06161ff700 1 -- 172.21.15.27:0/1111592597 >> 172.21.15.27:0/1111592597 conn(0x7f061006ed40 msgr2=0x7f061006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:23.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.621+0000 7f06161ff700 1 -- 172.21.15.27:0/1111592597 shutdown_connections 2024-01-11T09:26:23.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.621+0000 7f06161ff700 1 -- 172.21.15.27:0/1111592597 wait complete. 2024-01-11T09:26:23.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.622+0000 7f06161ff700 1 Processor -- start 2024-01-11T09:26:23.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.623+0000 7f06161ff700 1 -- start start 2024-01-11T09:26:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.623+0000 7f06161ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06100ccd70 0x7f061014d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.623+0000 7f06161ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f061015a1b0 0x7f06100b5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.624+0000 7f06161ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f061015ae70 0x7f06100c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.624+0000 7f06161ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06100c8940 con 0x7f061015ae70 2024-01-11T09:26:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.624+0000 7f06161ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06100c8050 con 0x7f06100ccd70 2024-01-11T09:26:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.624+0000 7f06161ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061014d680 con 0x7f061015a1b0 2024-01-11T09:26:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.624+0000 7f060ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f061015ae70 0x7f06100c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.624+0000 7f060ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f061015ae70 0x7f06100c90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55598/0 (socket says 172.21.15.27:55598) 2024-01-11T09:26:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.624+0000 7f060ffff700 1 -- 172.21.15.27:0/3447464661 learned_addr learned my addr 172.21.15.27:0/3447464661 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.624+0000 7f060f7fe700 1 --2- 172.21.15.27:0/3447464661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06100ccd70 0x7f061014d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.624+0000 7f060effd700 1 --2- 172.21.15.27:0/3447464661 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f061015a1b0 0x7f06100b5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.624+0000 7f060ffff700 1 -- 172.21.15.27:0/3447464661 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f061015a1b0 msgr2=0x7f06100b5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.624+0000 7f060ffff700 1 --2- 172.21.15.27:0/3447464661 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f061015a1b0 0x7f06100b5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.624+0000 7f060ffff700 1 -- 172.21.15.27:0/3447464661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06100ccd70 msgr2=0x7f061014d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.624+0000 7f060ffff700 1 --2- 172.21.15.27:0/3447464661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06100ccd70 0x7f061014d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.624+0000 7f060ffff700 1 -- 172.21.15.27:0/3447464661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06000094f0 con 0x7f061015ae70 2024-01-11T09:26:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.625+0000 7f060ffff700 1 --2- 172.21.15.27:0/3447464661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f061015ae70 0x7f06100c90a0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f060800d180 tx=0x7f0608009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.625+0000 7f05f77fe700 1 -- 172.21.15.27:0/3447464661 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060800dcc0 con 0x7f061015ae70 2024-01-11T09:26:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.625+0000 7f05f77fe700 1 -- 172.21.15.27:0/3447464661 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f060800f040 con 0x7f061015ae70 2024-01-11T09:26:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.625+0000 7f06161ff700 1 -- 172.21.15.27:0/3447464661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06100c8dd0 con 0x7f061015ae70 2024-01-11T09:26:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.625+0000 7f05f77fe700 1 -- 172.21.15.27:0/3447464661 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0608015690 con 0x7f061015ae70 2024-01-11T09:26:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.625+0000 7f06161ff700 1 -- 172.21.15.27:0/3447464661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06100c82f0 con 0x7f061015ae70 2024-01-11T09:26:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.626+0000 7f06161ff700 1 -- 172.21.15.27:0/3447464661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06100c8af0 con 0x7f061015ae70 2024-01-11T09:26:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.627+0000 7f05f77fe700 1 -- 172.21.15.27:0/3447464661 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 86) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f060801e3b0 con 0x7f061015ae70 2024-01-11T09:26:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.628+0000 7f05f77fe700 1 --2- 172.21.15.27:0/3447464661 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f05fc067050 0x7f05fc069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.628+0000 7f05f77fe700 1 -- 172.21.15.27:0/3447464661 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f0608084ec0 con 0x7f061015ae70 2024-01-11T09:26:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.628+0000 7f060f7fe700 1 -- 172.21.15.27:0/3447464661 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f05fc067050 msgr2=0x7f05fc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.628+0000 7f060f7fe700 1 --2- 172.21.15.27:0/3447464661 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f05fc067050 0x7f05fc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:26:23.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.632+0000 7f05f77fe700 1 -- 172.21.15.27:0/3447464661 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f06100c8af0 con 0x7f061015ae70 2024-01-11T09:26:23.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.829+0000 7f060f7fe700 1 -- 172.21.15.27:0/3447464661 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f05fc067050 msgr2=0x7f05fc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:23.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.829+0000 7f060f7fe700 1 --2- 172.21.15.27:0/3447464661 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f05fc067050 0x7f05fc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:26:23.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.836+0000 7f06161ff700 1 -- 172.21.15.27:0/3447464661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f061004c3c0 con 0x7f061015ae70 2024-01-11T09:26:23.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.836+0000 7f05f77fe700 1 -- 172.21.15.27:0/3447464661 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v153) v1 ==== 93+0+10 (secure 0 0 0) 0x7f060801a0a0 con 0x7f061015ae70 2024-01-11T09:26:23.833 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T09:26:23.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.839+0000 7f06161ff700 1 -- 172.21.15.27:0/3447464661 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f05fc067050 msgr2=0x7f05fc069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:26:23.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.839+0000 7f06161ff700 1 --2- 172.21.15.27:0/3447464661 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f05fc067050 0x7f05fc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:23.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.839+0000 7f06161ff700 1 -- 172.21.15.27:0/3447464661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f061015ae70 msgr2=0x7f06100c90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:23.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.839+0000 7f06161ff700 1 --2- 172.21.15.27:0/3447464661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f061015ae70 0x7f06100c90a0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f060800d180 tx=0x7f0608009a50 comp rx=0 tx=0).stop 2024-01-11T09:26:23.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.839+0000 7f06161ff700 1 -- 172.21.15.27:0/3447464661 shutdown_connections 2024-01-11T09:26:23.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.839+0000 7f06161ff700 1 --2- 172.21.15.27:0/3447464661 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f05fc067050 0x7f05fc069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:23.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.839+0000 7f06161ff700 1 --2- 172.21.15.27:0/3447464661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f061015ae70 0x7f06100c90a0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:23.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.839+0000 7f06161ff700 1 --2- 172.21.15.27:0/3447464661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06100ccd70 0x7f061014d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:23.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.839+0000 7f06161ff700 1 --2- 172.21.15.27:0/3447464661 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f061015a1b0 0x7f06100b5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:23.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.839+0000 7f06161ff700 1 -- 172.21.15.27:0/3447464661 >> 172.21.15.27:0/3447464661 conn(0x7f061006ed40 msgr2=0x7f06100c6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:23.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.839+0000 7f06161ff700 1 -- 172.21.15.27:0/3447464661 shutdown_connections 2024-01-11T09:26:23.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:23.839+0000 7f06161ff700 1 -- 172.21.15.27:0/3447464661 wait complete. 2024-01-11T09:26:23.853 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T09:26:23.889 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:23.893+0000 7f5cd60ea1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:26:24.067 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:24.070+0000 7f96802e81c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:26:24.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.106+0000 7fc4cc3ff700 1 Processor -- start 2024-01-11T09:26:24.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.106+0000 7fc4cc3ff700 1 -- start start 2024-01-11T09:26:24.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.107+0000 7fc4cc3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4c415a1b0 0x7fc4c415a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:24.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.107+0000 7fc4cc3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc4c40ccd70 0x7fc4c40cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:24.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.107+0000 7fc4cc3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c40cf850 con 0x7fc4c40685c0 2024-01-11T09:26:24.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.107+0000 7fc4cc3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4c40cec40 con 0x7fc4c40ccd70 2024-01-11T09:26:24.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.107+0000 7fc4cc3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c40cef20 con 0x7fc4c415a1b0 2024-01-11T09:26:24.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.107+0000 7fc4c999a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc4c40ccd70 0x7fc4c40cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:24.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.107+0000 7fc4c999a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc4c40ccd70 0x7fc4c40cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35118/0 (socket says 172.21.15.27:35118) 2024-01-11T09:26:24.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.107+0000 7fc4c999a700 1 -- 172.21.15.27:0/501496108 learned_addr learned my addr 172.21.15.27:0/501496108 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:24.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.108+0000 7fc4ca19b700 1 --2- 172.21.15.27:0/501496108 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4c415a1b0 0x7fc4c415a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:24.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.108+0000 7fc4c999a700 1 -- 172.21.15.27:0/501496108 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4c415a1b0 msgr2=0x7fc4c415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:24.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.108+0000 7fc4c999a700 1 --2- 172.21.15.27:0/501496108 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4c415a1b0 0x7fc4c415a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.108+0000 7fc4c999a700 1 -- 172.21.15.27:0/501496108 >> v1:172.21.15.77:6789/0 conn(0x7fc4c40685c0 legacy=0x7fc4c4068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:24.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.108+0000 7fc4c999a700 1 -- 172.21.15.27:0/501496108 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c40cf200 con 0x7fc4c40ccd70 2024-01-11T09:26:24.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.108+0000 7fc4c9199700 1 -- 172.21.15.27:0/501496108 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589339840 0 0) 0x7fc4c40cf850 con 0x7fc4c40685c0 2024-01-11T09:26:24.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.108+0000 7fc4c999a700 1 --2- 172.21.15.27:0/501496108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c40ccd70 0x7fc4c40cd1e0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fc4b40099b0 tx=0x7fc4b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4cc0195a7cabc5b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:24.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.108+0000 7fc4c9199700 1 -- 172.21.15.27:0/501496108 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4009ea0 con 0x7fc4c40ccd70 2024-01-11T09:26:24.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.109+0000 7fc4c9199700 1 -- 172.21.15.27:0/501496108 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b400b860 con 0x7fc4c40ccd70 2024-01-11T09:26:24.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.109+0000 7fc4c9199700 1 -- 172.21.15.27:0/501496108 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b400bc00 con 0x7fc4c40ccd70 2024-01-11T09:26:24.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.109+0000 7fc4cc3ff700 1 -- 172.21.15.27:0/501496108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c40ccd70 msgr2=0x7fc4c40cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:24.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.109+0000 7fc4cc3ff700 1 --2- 172.21.15.27:0/501496108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c40ccd70 0x7fc4c40cd1e0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fc4b40099b0 tx=0x7fc4b4009250 comp rx=0 tx=0).stop 2024-01-11T09:26:24.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.109+0000 7fc4cc3ff700 1 -- 172.21.15.27:0/501496108 shutdown_connections 2024-01-11T09:26:24.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.109+0000 7fc4cc3ff700 1 --2- 172.21.15.27:0/501496108 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4c415a1b0 0x7fc4c415a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.109+0000 7fc4cc3ff700 1 --2- 172.21.15.27:0/501496108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c40ccd70 0x7fc4c40cd1e0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.109+0000 7fc4cc3ff700 1 -- 172.21.15.27:0/501496108 >> 172.21.15.27:0/501496108 conn(0x7fc4c406ed40 msgr2=0x7fc4c406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:24.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.110+0000 7fc4cc3ff700 1 -- 172.21.15.27:0/501496108 shutdown_connections 2024-01-11T09:26:24.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.110+0000 7fc4cc3ff700 1 -- 172.21.15.27:0/501496108 wait complete. 2024-01-11T09:26:24.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.112+0000 7fc4cc3ff700 1 Processor -- start 2024-01-11T09:26:24.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.112+0000 7fc4cc3ff700 1 -- start start 2024-01-11T09:26:24.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.113+0000 7fc4cc3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4c40685c0 0x7fc4c40b5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:24.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.113+0000 7fc4cc3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c40ccd70 0x7fc4c4156370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.113+0000 7fc4cc3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4c415a1b0 0x7fc4c416c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.113+0000 7fc4cc3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c415d970 con 0x7fc4c40ccd70 2024-01-11T09:26:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.113+0000 7fc4cc3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c40b6160 con 0x7fc4c40685c0 2024-01-11T09:26:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.113+0000 7fc4cc3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c415d360 con 0x7fc4c415a1b0 2024-01-11T09:26:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.113+0000 7fc4c999a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c40ccd70 0x7fc4c4156370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.114+0000 7fc4ca19b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4c40685c0 0x7fc4c40b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.114+0000 7fc4c999a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c40ccd70 0x7fc4c4156370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35128/0 (socket says 172.21.15.27:35128) 2024-01-11T09:26:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.114+0000 7fc4ca99c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4c415a1b0 0x7fc4c416c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.114+0000 7fc4c999a700 1 -- 172.21.15.27:0/1045542305 learned_addr learned my addr 172.21.15.27:0/1045542305 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.114+0000 7fc4c999a700 1 -- 172.21.15.27:0/1045542305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4c415a1b0 msgr2=0x7fc4c416c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.114+0000 7fc4c999a700 1 --2- 172.21.15.27:0/1045542305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4c415a1b0 0x7fc4c416c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.114+0000 7fc4c999a700 1 -- 172.21.15.27:0/1045542305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4c40685c0 msgr2=0x7fc4c40b5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.114+0000 7fc4c999a700 1 --2- 172.21.15.27:0/1045542305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4c40685c0 0x7fc4c40b5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.114+0000 7fc4c999a700 1 -- 172.21.15.27:0/1045542305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b40094f0 con 0x7fc4c40ccd70 2024-01-11T09:26:24.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.114+0000 7fc4c999a700 1 --2- 172.21.15.27:0/1045542305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c40ccd70 0x7fc4c4156370 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4069770 tx=0x7fc4b4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:24.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.115+0000 7fc4ba7fc700 1 -- 172.21.15.27:0/1045542305 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4000d10 con 0x7fc4c40ccd70 2024-01-11T09:26:24.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.115+0000 7fc4ba7fc700 1 -- 172.21.15.27:0/1045542305 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b400bbe0 con 0x7fc4c40ccd70 2024-01-11T09:26:24.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.115+0000 7fc4ba7fc700 1 -- 172.21.15.27:0/1045542305 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4011670 con 0x7fc4c40ccd70 2024-01-11T09:26:24.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.115+0000 7fc4cc3ff700 1 -- 172.21.15.27:0/1045542305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c40b5e80 con 0x7fc4c40ccd70 2024-01-11T09:26:24.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.115+0000 7fc4cc3ff700 1 -- 172.21.15.27:0/1045542305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c40b56b0 con 0x7fc4c40ccd70 2024-01-11T09:26:24.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.116+0000 7fc4cc3ff700 1 -- 172.21.15.27:0/1045542305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c415dae0 con 0x7fc4c40ccd70 2024-01-11T09:26:24.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.117+0000 7fc4ba7fc700 1 -- 172.21.15.27:0/1045542305 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 86) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fc4b4011aa0 con 0x7fc4c40ccd70 2024-01-11T09:26:24.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.118+0000 7fc4ba7fc700 1 --2- 172.21.15.27:0/1045542305 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc4a8067050 0x7fc4a8069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:24.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.118+0000 7fc4ba7fc700 1 -- 172.21.15.27:0/1045542305 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6165+0+0 (secure 0 0 0) 0x7fc4b4086110 con 0x7fc4c40ccd70 2024-01-11T09:26:24.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.118+0000 7fc4ca19b700 1 -- 172.21.15.27:0/1045542305 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc4a8067050 msgr2=0x7fc4a8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:24.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.118+0000 7fc4ca19b700 1 --2- 172.21.15.27:0/1045542305 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc4a8067050 0x7fc4a8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:26:24.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.122+0000 7fc4ba7fc700 1 -- 172.21.15.27:0/1045542305 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc4b4053000 con 0x7fc4c40ccd70 2024-01-11T09:26:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.319+0000 7fc4ca19b700 1 -- 172.21.15.27:0/1045542305 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc4a8067050 msgr2=0x7fc4a8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:24.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.319+0000 7fc4ca19b700 1 --2- 172.21.15.27:0/1045542305 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc4a8067050 0x7fc4a8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:26:24.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.366+0000 7fc4cc3ff700 1 -- 172.21.15.27:0/1045542305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc4c404c3c0 con 0x7fc4c40ccd70 2024-01-11T09:26:24.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.367+0000 7fc4ba7fc700 1 -- 172.21.15.27:0/1045542305 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v153) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc4b4056620 con 0x7fc4c40ccd70 2024-01-11T09:26:24.364 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T09:26:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.371+0000 7fc4cc3ff700 1 -- 172.21.15.27:0/1045542305 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc4a8067050 msgr2=0x7fc4a8069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:26:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.371+0000 7fc4cc3ff700 1 --2- 172.21.15.27:0/1045542305 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc4a8067050 0x7fc4a8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.371+0000 7fc4cc3ff700 1 -- 172.21.15.27:0/1045542305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c40ccd70 msgr2=0x7fc4c4156370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.371+0000 7fc4cc3ff700 1 --2- 172.21.15.27:0/1045542305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c40ccd70 0x7fc4c4156370 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4069770 tx=0x7fc4b4017000 comp rx=0 tx=0).stop 2024-01-11T09:26:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.371+0000 7fc4cc3ff700 1 -- 172.21.15.27:0/1045542305 shutdown_connections 2024-01-11T09:26:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.371+0000 7fc4cc3ff700 1 --2- 172.21.15.27:0/1045542305 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc4a8067050 0x7fc4a8069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.371+0000 7fc4cc3ff700 1 --2- 172.21.15.27:0/1045542305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4c415a1b0 0x7fc4c416c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.371+0000 7fc4cc3ff700 1 --2- 172.21.15.27:0/1045542305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4c40685c0 0x7fc4c40b5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.371+0000 7fc4cc3ff700 1 --2- 172.21.15.27:0/1045542305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4c40ccd70 0x7fc4c4156370 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.371+0000 7fc4cc3ff700 1 -- 172.21.15.27:0/1045542305 >> 172.21.15.27:0/1045542305 conn(0x7fc4c406ed40 msgr2=0x7fc4c40c5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:24.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.371+0000 7fc4cc3ff700 1 -- 172.21.15.27:0/1045542305 shutdown_connections 2024-01-11T09:26:24.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.371+0000 7fc4cc3ff700 1 -- 172.21.15.27:0/1045542305 wait complete. 2024-01-11T09:26:24.395 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T09:26:24.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.602+0000 7f62741ff700 1 Processor -- start 2024-01-11T09:26:24.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.602+0000 7f62741ff700 1 -- start start 2024-01-11T09:26:24.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.603+0000 7f62741ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f626c141190 0x7f626c0c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:24.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.603+0000 7f62741ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f626c0c8f60 0x7f626c15b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:24.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.604+0000 7f62741ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626c15c060 con 0x7f626c140dc0 2024-01-11T09:26:24.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.604+0000 7f62741ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f626c0cc9b0 con 0x7f626c0c8f60 2024-01-11T09:26:24.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.604+0000 7f62741ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0ae390 con 0x7f626c141190 2024-01-11T09:26:24.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.604+0000 7f6271f9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f626c140dc0 0x7f626c0adc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57466/0 (socket says 172.21.15.27:57466) 2024-01-11T09:26:24.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.604+0000 7f6271f9b700 1 -- 172.21.15.27:0/895645851 learned_addr learned my addr 172.21.15.27:0/895645851 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:24.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.604+0000 7f627279c700 1 --2- 172.21.15.27:0/895645851 >> v2:172.21.15.77:3300/0 conn(0x7f626c0c8f60 0x7f626c15b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:24.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.604+0000 7f627179a700 1 --2- 172.21.15.27:0/895645851 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f626c141190 0x7f626c0c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:24.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.604+0000 7f6270f99700 1 -- 172.21.15.27:0/895645851 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651648267 0 0) 0x7f6268001110 con 0x7f626c140dc0 2024-01-11T09:26:24.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.604+0000 7f6270f99700 1 -- 172.21.15.27:0/895645851 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6258003570 con 0x7f626c140dc0 2024-01-11T09:26:24.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.604+0000 7f627179a700 1 -- 172.21.15.27:0/895645851 >> v2:172.21.15.77:3300/0 conn(0x7f626c0c8f60 msgr2=0x7f626c15b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:24.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.604+0000 7f627179a700 1 --2- 172.21.15.27:0/895645851 >> v2:172.21.15.77:3300/0 conn(0x7f626c0c8f60 0x7f626c15b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.605+0000 7f627179a700 1 -- 172.21.15.27:0/895645851 >> v1:172.21.15.27:6789/0 conn(0x7f626c140dc0 legacy=0x7f626c0adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:24.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.605+0000 7f627179a700 1 -- 172.21.15.27:0/895645851 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626c0c93a0 con 0x7f626c141190 2024-01-11T09:26:24.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.605+0000 7f627279c700 1 --2- 172.21.15.27:0/895645851 >> v2:172.21.15.77:3300/0 conn(0x7f626c0c8f60 0x7f626c15b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:26:24.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.605+0000 7f627179a700 1 --2- 172.21.15.27:0/895645851 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f626c141190 0x7f626c0c8a20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f625c0098e0 tx=0x7f625c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90541290bdcfaedb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:24.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.605+0000 7f6270f99700 1 -- 172.21.15.27:0/895645851 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625c011630 con 0x7f626c141190 2024-01-11T09:26:24.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.605+0000 7f6270f99700 1 -- 172.21.15.27:0/895645851 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625c011790 con 0x7f626c141190 2024-01-11T09:26:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.605+0000 7f6270f99700 1 -- 172.21.15.27:0/895645851 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625c011b30 con 0x7f626c141190 2024-01-11T09:26:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.606+0000 7f62741ff700 1 -- 172.21.15.27:0/895645851 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f626c141190 msgr2=0x7f626c0c8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.606+0000 7f62741ff700 1 --2- 172.21.15.27:0/895645851 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f626c141190 0x7f626c0c8a20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f625c0098e0 tx=0x7f625c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.606+0000 7f62741ff700 1 -- 172.21.15.27:0/895645851 shutdown_connections 2024-01-11T09:26:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.606+0000 7f62741ff700 1 --2- 172.21.15.27:0/895645851 >> v2:172.21.15.77:3300/0 conn(0x7f626c0c8f60 0x7f626c15b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.606+0000 7f62741ff700 1 --2- 172.21.15.27:0/895645851 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f626c141190 0x7f626c0c8a20 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.606+0000 7f62741ff700 1 -- 172.21.15.27:0/895645851 >> 172.21.15.27:0/895645851 conn(0x7f626c063ca0 msgr2=0x7f626c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.606+0000 7f62741ff700 1 -- 172.21.15.27:0/895645851 shutdown_connections 2024-01-11T09:26:24.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.606+0000 7f62741ff700 1 -- 172.21.15.27:0/895645851 wait complete. 2024-01-11T09:26:24.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.608+0000 7f62741ff700 1 Processor -- start 2024-01-11T09:26:24.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.608+0000 7f62741ff700 1 -- start start 2024-01-11T09:26:24.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.609+0000 7f62741ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f626c0c8f60 0x7f626c168930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:24.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.610+0000 7f62741ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f626c140dc0 0x7f626c15a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:24.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.610+0000 7f62741ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f626c141190 0x7f626c0c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:24.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.610+0000 7f62741ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0c8080 con 0x7f626c141190 2024-01-11T09:26:24.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.610+0000 7f62741ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c15a7a0 con 0x7f626c140dc0 2024-01-11T09:26:24.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.610+0000 7f62741ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0c71a0 con 0x7f626c0c8f60 2024-01-11T09:26:24.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.610+0000 7f627279c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f626c141190 0x7f626c0c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:24.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.610+0000 7f6271f9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f626c0c8f60 0x7f626c168930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:24.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.610+0000 7f627279c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f626c141190 0x7f626c0c6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35142/0 (socket says 172.21.15.27:35142) 2024-01-11T09:26:24.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.610+0000 7f627279c700 1 -- 172.21.15.27:0/84767876 learned_addr learned my addr 172.21.15.27:0/84767876 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:24.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.610+0000 7f627179a700 1 --2- 172.21.15.27:0/84767876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f626c140dc0 0x7f626c15a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:24.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.610+0000 7f627279c700 1 -- 172.21.15.27:0/84767876 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f626c0c8f60 msgr2=0x7f626c168930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:24.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.610+0000 7f627279c700 1 --2- 172.21.15.27:0/84767876 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f626c0c8f60 0x7f626c168930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.610+0000 7f627279c700 1 -- 172.21.15.27:0/84767876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f626c140dc0 msgr2=0x7f626c15a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:24.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.610+0000 7f627279c700 1 --2- 172.21.15.27:0/84767876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f626c140dc0 0x7f626c15a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.610+0000 7f627279c700 1 -- 172.21.15.27:0/84767876 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6264009a20 con 0x7f626c141190 2024-01-11T09:26:24.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.611+0000 7f627279c700 1 --2- 172.21.15.27:0/84767876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f626c141190 0x7f626c0c6c60 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f6264009fc0 tx=0x7f626400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:24.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.611+0000 7f6261ffb700 1 -- 172.21.15.27:0/84767876 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626400cbd0 con 0x7f626c141190 2024-01-11T09:26:24.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.611+0000 7f6261ffb700 1 -- 172.21.15.27:0/84767876 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f626400f040 con 0x7f626c141190 2024-01-11T09:26:24.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.611+0000 7f6261ffb700 1 -- 172.21.15.27:0/84767876 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6264014480 con 0x7f626c141190 2024-01-11T09:26:24.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.611+0000 7f62741ff700 1 -- 172.21.15.27:0/84767876 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f625c009420 con 0x7f626c141190 2024-01-11T09:26:24.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.611+0000 7f62741ff700 1 -- 172.21.15.27:0/84767876 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626c0c7750 con 0x7f626c141190 2024-01-11T09:26:24.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.612+0000 7f62741ff700 1 -- 172.21.15.27:0/84767876 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f626c04c3c0 con 0x7f626c141190 2024-01-11T09:26:24.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.613+0000 7f6261ffb700 1 -- 172.21.15.27:0/84767876 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 86) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f6264003020 con 0x7f626c141190 2024-01-11T09:26:24.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.614+0000 7f6261ffb700 1 --2- 172.21.15.27:0/84767876 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6250067050 0x7f6250069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:24.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.614+0000 7f6261ffb700 1 -- 172.21.15.27:0/84767876 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f6264085310 con 0x7f626c141190 2024-01-11T09:26:24.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.615+0000 7f6271f9b700 1 -- 172.21.15.27:0/84767876 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6250067050 msgr2=0x7f6250069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:24.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.615+0000 7f6271f9b700 1 --2- 172.21.15.27:0/84767876 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6250067050 0x7f6250069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:26:24.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.618+0000 7f6261ffb700 1 -- 172.21.15.27:0/84767876 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6264052190 con 0x7f626c141190 2024-01-11T09:26:24.675 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:24.679+0000 7f5cd60ea1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:26:24.763 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:24.767+0000 7f5cd60ea1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:26:24.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.816+0000 7f6271f9b700 1 -- 172.21.15.27:0/84767876 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6250067050 msgr2=0x7f6250069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:24.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.816+0000 7f6271f9b700 1 --2- 172.21.15.27:0/84767876 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6250067050 0x7f6250069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:26:24.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.823+0000 7f62741ff700 1 -- 172.21.15.27:0/84767876 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f626c06c4b0 con 0x7f626c141190 2024-01-11T09:26:24.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.823+0000 7f6261ffb700 1 -- 172.21.15.27:0/84767876 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v153) v1 ==== 100+0+11 (secure 0 0 0) 0x7f62640557b0 con 0x7f626c141190 2024-01-11T09:26:24.820 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T09:26:24.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.826+0000 7f62741ff700 1 -- 172.21.15.27:0/84767876 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6250067050 msgr2=0x7f6250069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:26:24.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.826+0000 7f62741ff700 1 --2- 172.21.15.27:0/84767876 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6250067050 0x7f6250069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.826+0000 7f62741ff700 1 -- 172.21.15.27:0/84767876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f626c141190 msgr2=0x7f626c0c6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:24.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.826+0000 7f62741ff700 1 --2- 172.21.15.27:0/84767876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f626c141190 0x7f626c0c6c60 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f6264009fc0 tx=0x7f626400b040 comp rx=0 tx=0).stop 2024-01-11T09:26:24.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.826+0000 7f62741ff700 1 -- 172.21.15.27:0/84767876 shutdown_connections 2024-01-11T09:26:24.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.826+0000 7f62741ff700 1 --2- 172.21.15.27:0/84767876 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f6250067050 0x7f6250069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.826+0000 7f62741ff700 1 --2- 172.21.15.27:0/84767876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f626c141190 0x7f626c0c6c60 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.826+0000 7f62741ff700 1 --2- 172.21.15.27:0/84767876 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f626c0c8f60 0x7f626c168930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.826+0000 7f62741ff700 1 --2- 172.21.15.27:0/84767876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f626c140dc0 0x7f626c15a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:24.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.826+0000 7f62741ff700 1 -- 172.21.15.27:0/84767876 >> 172.21.15.27:0/84767876 conn(0x7f626c063ca0 msgr2=0x7f626c0ceba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:24.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.826+0000 7f62741ff700 1 -- 172.21.15.27:0/84767876 shutdown_connections 2024-01-11T09:26:24.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:24.826+0000 7f62741ff700 1 -- 172.21.15.27:0/84767876 wait complete. 2024-01-11T09:26:24.843 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:26:24.919 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:24.923+0000 7f5cd60ea1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:26:24.950 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:24.953+0000 7f96802e81c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:26:25.006 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:25.010+0000 7f5cd60ea1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:26:25.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.053+0000 7f9aa9bff700 1 Processor -- start 2024-01-11T09:26:25.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.053+0000 7f9aa9bff700 1 -- start start 2024-01-11T09:26:25.050 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:25.054+0000 7f96802e81c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:26:25.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.055+0000 7f9aa9bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9aa40ad020 0x7f9aa40af320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.055+0000 7f9aa9bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9aa40ac370 0x7f9aa4159180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.055+0000 7f9aa9bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9aa406b460 0x7f9aa41596c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.055+0000 7f9aa9bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa4159c00 con 0x7f9aa40ac370 2024-01-11T09:26:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.055+0000 7f9aa9bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa40aa1d0 con 0x7f9aa40ad020 2024-01-11T09:26:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.055+0000 7f9aa9bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa40af860 con 0x7f9aa406b460 2024-01-11T09:26:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.055+0000 7f9aa2ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9aa40ac370 0x7f9aa4159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.055+0000 7f9aa2ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9aa40ac370 0x7f9aa4159180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35152/0 (socket says 172.21.15.27:35152) 2024-01-11T09:26:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.055+0000 7f9aa2ffd700 1 -- 172.21.15.27:0/2283480335 learned_addr learned my addr 172.21.15.27:0/2283480335 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.055+0000 7f9aa37fe700 1 --2- 172.21.15.27:0/2283480335 >> v2:172.21.15.77:3300/0 conn(0x7f9aa40ad020 0x7f9aa40af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.055+0000 7f9aa3fff700 1 --2- 172.21.15.27:0/2283480335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9aa406b460 0x7f9aa41596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.056+0000 7f9aa2ffd700 1 -- 172.21.15.27:0/2283480335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9aa406b460 msgr2=0x7f9aa41596c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.056+0000 7f9aa2ffd700 1 --2- 172.21.15.27:0/2283480335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9aa406b460 0x7f9aa41596c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.056+0000 7f9aa2ffd700 1 -- 172.21.15.27:0/2283480335 >> v2:172.21.15.77:3300/0 conn(0x7f9aa40ad020 msgr2=0x7f9aa40af320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:25.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.056+0000 7f9aa2ffd700 1 --2- 172.21.15.27:0/2283480335 >> v2:172.21.15.77:3300/0 conn(0x7f9aa40ad020 0x7f9aa40af320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.056+0000 7f9aa2ffd700 1 -- 172.21.15.27:0/2283480335 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa40ac8d0 con 0x7f9aa40ac370 2024-01-11T09:26:25.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.056+0000 7f9aa2ffd700 1 --2- 172.21.15.27:0/2283480335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa40ac370 0x7f9aa4159180 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4064b00 tx=0x7f9a98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84273c1cc71c48d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:25.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.056+0000 7f9aa27fc700 1 -- 172.21.15.27:0/2283480335 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9800b640 con 0x7f9aa40ac370 2024-01-11T09:26:25.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.056+0000 7f9aa27fc700 1 -- 172.21.15.27:0/2283480335 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9800b7a0 con 0x7f9aa40ac370 2024-01-11T09:26:25.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.056+0000 7f9aa27fc700 1 -- 172.21.15.27:0/2283480335 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9800bb60 con 0x7f9aa40ac370 2024-01-11T09:26:25.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.057+0000 7f9aa9bff700 1 -- 172.21.15.27:0/2283480335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa40ac370 msgr2=0x7f9aa4159180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:25.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.057+0000 7f9aa9bff700 1 --2- 172.21.15.27:0/2283480335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa40ac370 0x7f9aa4159180 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4064b00 tx=0x7f9a98009250 comp rx=0 tx=0).stop 2024-01-11T09:26:25.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.057+0000 7f9aa9bff700 1 -- 172.21.15.27:0/2283480335 shutdown_connections 2024-01-11T09:26:25.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.057+0000 7f9aa9bff700 1 --2- 172.21.15.27:0/2283480335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9aa406b460 0x7f9aa41596c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.057+0000 7f9aa9bff700 1 --2- 172.21.15.27:0/2283480335 >> v2:172.21.15.77:3300/0 conn(0x7f9aa40ad020 0x7f9aa40af320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.057+0000 7f9aa9bff700 1 --2- 172.21.15.27:0/2283480335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa40ac370 0x7f9aa4159180 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.057+0000 7f9aa9bff700 1 -- 172.21.15.27:0/2283480335 >> 172.21.15.27:0/2283480335 conn(0x7f9aa40680e0 msgr2=0x7f9aa4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:25.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.057+0000 7f9aa9bff700 1 -- 172.21.15.27:0/2283480335 shutdown_connections 2024-01-11T09:26:25.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.057+0000 7f9aa9bff700 1 -- 172.21.15.27:0/2283480335 wait complete. 2024-01-11T09:26:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.059+0000 7f9aa9bff700 1 Processor -- start 2024-01-11T09:26:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.059+0000 7f9aa9bff700 1 -- start start 2024-01-11T09:26:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.060+0000 7f9aa9bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9aa406b460 0x7f9aa416a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.060+0000 7f9aa9bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa40ac370 0x7f9aa4146bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.060+0000 7f9aa9bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9aa40ad020 0x7f9aa416cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.060+0000 7f9aa9bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa416da10 con 0x7f9aa40ac370 2024-01-11T09:26:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.060+0000 7f9aa9bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa416dcf0 con 0x7f9aa406b460 2024-01-11T09:26:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.060+0000 7f9aa9bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa4169ce0 con 0x7f9aa40ad020 2024-01-11T09:26:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.060+0000 7f9aa2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa40ac370 0x7f9aa4146bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:25.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.060+0000 7f9aa37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9aa406b460 0x7f9aa416a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.060+0000 7f9aa3fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9aa40ad020 0x7f9aa416cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.060+0000 7f9aa3fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9aa40ad020 0x7f9aa416cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:44760/0 (socket says 172.21.15.27:44760) 2024-01-11T09:26:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.060+0000 7f9aa2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa40ac370 0x7f9aa4146bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35164/0 (socket says 172.21.15.27:35164) 2024-01-11T09:26:25.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.060+0000 7f9aa3fff700 1 -- 172.21.15.27:0/3597513794 learned_addr learned my addr 172.21.15.27:0/3597513794 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.061+0000 7f9aa37fe700 1 -- 172.21.15.27:0/3597513794 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9aa40ad020 msgr2=0x7f9aa416cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.061+0000 7f9aa37fe700 1 --2- 172.21.15.27:0/3597513794 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9aa40ad020 0x7f9aa416cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.061+0000 7f9aa37fe700 1 -- 172.21.15.27:0/3597513794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa40ac370 msgr2=0x7f9aa4146bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.061+0000 7f9aa37fe700 1 --2- 172.21.15.27:0/3597513794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa40ac370 0x7f9aa4146bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.061+0000 7f9aa37fe700 1 -- 172.21.15.27:0/3597513794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a980094f0 con 0x7f9aa406b460 2024-01-11T09:26:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.061+0000 7f9aa2ffd700 1 --2- 172.21.15.27:0/3597513794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa40ac370 0x7f9aa4146bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:26:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.061+0000 7f9aa37fe700 1 --2- 172.21.15.27:0/3597513794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9aa406b460 0x7f9aa416a4e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9a9400e900 tx=0x7f9a940091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.061+0000 7f9a8b7fe700 1 -- 172.21.15.27:0/3597513794 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a94013680 con 0x7f9aa406b460 2024-01-11T09:26:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.062+0000 7f9a8b7fe700 1 -- 172.21.15.27:0/3597513794 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a940137e0 con 0x7f9aa406b460 2024-01-11T09:26:25.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.062+0000 7f9aa9bff700 1 -- 172.21.15.27:0/3597513794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa4169fc0 con 0x7f9aa406b460 2024-01-11T09:26:25.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.062+0000 7f9a8b7fe700 1 -- 172.21.15.27:0/3597513794 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a94013ba0 con 0x7f9aa406b460 2024-01-11T09:26:25.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.062+0000 7f9aa9bff700 1 -- 172.21.15.27:0/3597513794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa4169990 con 0x7f9aa406b460 2024-01-11T09:26:25.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.063+0000 7f9aa9bff700 1 -- 172.21.15.27:0/3597513794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa416a330 con 0x7f9aa406b460 2024-01-11T09:26:25.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.065+0000 7f9a8b7fe700 1 -- 172.21.15.27:0/3597513794 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 86) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f9a940123d0 con 0x7f9aa406b460 2024-01-11T09:26:25.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.065+0000 7f9a8b7fe700 1 --2- 172.21.15.27:0/3597513794 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9a90067050 0x7f9a90069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:25.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.066+0000 7f9a8b7fe700 1 -- 172.21.15.27:0/3597513794 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f9a940848e0 con 0x7f9aa406b460 2024-01-11T09:26:25.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.066+0000 7f9aa2ffd700 1 -- 172.21.15.27:0/3597513794 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9a90067050 msgr2=0x7f9a90069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:25.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.066+0000 7f9aa2ffd700 1 --2- 172.21.15.27:0/3597513794 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9a90067050 0x7f9a90069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:26:25.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.071+0000 7f9a8b7fe700 1 -- 172.21.15.27:0/3597513794 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9a94051870 con 0x7f9aa406b460 2024-01-11T09:26:25.228 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:25.232+0000 7f5cd60ea1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:26:25.256 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:25.260+0000 7f96802e81c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:26:25.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.267+0000 7f9aa2ffd700 1 -- 172.21.15.27:0/3597513794 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9a90067050 msgr2=0x7f9a90069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:25.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.267+0000 7f9aa2ffd700 1 --2- 172.21.15.27:0/3597513794 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9a90067050 0x7f9a90069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:26:25.324 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:25.327+0000 7f5cd60ea1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:26:25.352 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:25.355+0000 7f96802e81c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:26:25.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.362+0000 7f9aa9bff700 1 -- 172.21.15.27:0/3597513794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9aa416de30 con 0x7f9aa406b460 2024-01-11T09:26:25.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.362+0000 7f9a8b7fe700 1 -- 172.21.15.27:0/3597513794 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4596 (secure 0 0 0) 0x7f9aa416de30 con 0x7f9aa406b460 2024-01-11T09:26:25.359 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:26:25.359 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7352,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3818350333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3818350333},{"type":"v1","addr":"172.21.15.27:6834","nonce":3818350333}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/217390254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":217390254},{"type":"v1","addr":"172.21.15.27:6835","nonce":217390254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2824437717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2824437717},{"type":"v1","addr":"172.21.15.77:6834","nonce":2824437717}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:30:15.623796+0000","modified":"2024-01-11T08:30:19.150622+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8601":{"gid":8601,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/1641163931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1641163931},{"type":"v1","addr":"172.21.15.77:6837","nonce":1641163931}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2024-01-11T09:26:25.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.365+0000 7f9aa9bff700 1 -- 172.21.15.27:0/3597513794 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9a90067050 msgr2=0x7f9a90069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:26:25.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.365+0000 7f9aa9bff700 1 --2- 172.21.15.27:0/3597513794 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9a90067050 0x7f9a90069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.365+0000 7f9aa9bff700 1 -- 172.21.15.27:0/3597513794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9aa406b460 msgr2=0x7f9aa416a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:25.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.365+0000 7f9aa9bff700 1 --2- 172.21.15.27:0/3597513794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9aa406b460 0x7f9aa416a4e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9a9400e900 tx=0x7f9a940091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.366+0000 7f9aa9bff700 1 -- 172.21.15.27:0/3597513794 shutdown_connections 2024-01-11T09:26:25.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.366+0000 7f9aa9bff700 1 --2- 172.21.15.27:0/3597513794 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f9a90067050 0x7f9a90069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.366+0000 7f9aa9bff700 1 --2- 172.21.15.27:0/3597513794 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9aa40ad020 0x7f9aa416cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.366+0000 7f9aa9bff700 1 --2- 172.21.15.27:0/3597513794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9aa406b460 0x7f9aa416a4e0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.366+0000 7f9aa9bff700 1 --2- 172.21.15.27:0/3597513794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa40ac370 0x7f9aa4146bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.366+0000 7f9aa9bff700 1 -- 172.21.15.27:0/3597513794 >> 172.21.15.27:0/3597513794 conn(0x7f9aa40680e0 msgr2=0x7f9aa40abbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:25.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.366+0000 7f9aa9bff700 1 -- 172.21.15.27:0/3597513794 shutdown_connections 2024-01-11T09:26:25.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.366+0000 7f9aa9bff700 1 -- 172.21.15.27:0/3597513794 wait complete. 2024-01-11T09:26:25.365 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 77 2024-01-11T09:26:25.387 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:26:25.428 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:25.432+0000 7f5cd60ea1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:26:25.523 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:25.526+0000 7f5cd60ea1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:26:25.590 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:25.594+0000 7f96802e81c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:26:25.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.597+0000 7fc9569ff700 1 Processor -- start 2024-01-11T09:26:25.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.597+0000 7fc9569ff700 1 -- start start 2024-01-11T09:26:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.598+0000 7fc9569ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9500b2b90 0x7fc950159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.598+0000 7fc9569ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9500b07b0 0x7fc950159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.598+0000 7fc9569ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9500cd730 con 0x7fc9500b03e0 2024-01-11T09:26:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.598+0000 7fc9569ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9500ccb20 con 0x7fc9500b07b0 2024-01-11T09:26:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.598+0000 7fc9569ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9500cce00 con 0x7fc9500b2b90 2024-01-11T09:26:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.598+0000 7fc954f9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9500b07b0 0x7fc950159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.598+0000 7fc954f9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9500b07b0 0x7fc950159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35168/0 (socket says 172.21.15.27:35168) 2024-01-11T09:26:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.598+0000 7fc954f9c700 1 -- 172.21.15.27:0/944247002 learned_addr learned my addr 172.21.15.27:0/944247002 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.598+0000 7fc94ffff700 1 --1- 172.21.15.27:0/944247002 >> v1:172.21.15.27:6789/0 conn(0x7fc9500b03e0 0x7fc9500b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57474/0 (socket says 172.21.15.27:57474) 2024-01-11T09:26:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.598+0000 7fc94f7fe700 1 --2- 172.21.15.27:0/944247002 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9500b2b90 0x7fc950159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:25.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.599+0000 7fc954f9c700 1 -- 172.21.15.27:0/944247002 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9500b2b90 msgr2=0x7fc950159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.599+0000 7fc954f9c700 1 --2- 172.21.15.27:0/944247002 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9500b2b90 0x7fc950159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.599+0000 7fc954f9c700 1 -- 172.21.15.27:0/944247002 >> v1:172.21.15.27:6789/0 conn(0x7fc9500b03e0 legacy=0x7fc9500b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.599+0000 7fc954f9c700 1 -- 172.21.15.27:0/944247002 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9500cd0e0 con 0x7fc9500b07b0 2024-01-11T09:26:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.599+0000 7fc954f9c700 1 --2- 172.21.15.27:0/944247002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9500b07b0 0x7fc950159740 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fc948009f20 tx=0x7fc948009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd89f728e35e3c76 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.599+0000 7fc94effd700 1 -- 172.21.15.27:0/944247002 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94800e070 con 0x7fc9500b07b0 2024-01-11T09:26:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.599+0000 7fc94effd700 1 -- 172.21.15.27:0/944247002 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94800b9d0 con 0x7fc9500b07b0 2024-01-11T09:26:25.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.599+0000 7fc94effd700 1 -- 172.21.15.27:0/944247002 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9480092e0 con 0x7fc9500b07b0 2024-01-11T09:26:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.600+0000 7fc9569ff700 1 -- 172.21.15.27:0/944247002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9500b07b0 msgr2=0x7fc950159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.600+0000 7fc9569ff700 1 --2- 172.21.15.27:0/944247002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9500b07b0 0x7fc950159740 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fc948009f20 tx=0x7fc948009840 comp rx=0 tx=0).stop 2024-01-11T09:26:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.600+0000 7fc9569ff700 1 -- 172.21.15.27:0/944247002 shutdown_connections 2024-01-11T09:26:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.600+0000 7fc9569ff700 1 --2- 172.21.15.27:0/944247002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9500b07b0 0x7fc950159740 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.600+0000 7fc9569ff700 1 --2- 172.21.15.27:0/944247002 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9500b2b90 0x7fc950159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.600+0000 7fc9569ff700 1 -- 172.21.15.27:0/944247002 >> 172.21.15.27:0/944247002 conn(0x7fc95006ed40 msgr2=0x7fc95006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.600+0000 7fc9569ff700 1 -- 172.21.15.27:0/944247002 shutdown_connections 2024-01-11T09:26:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.600+0000 7fc9569ff700 1 -- 172.21.15.27:0/944247002 wait complete. 2024-01-11T09:26:25.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.602+0000 7fc9569ff700 1 Processor -- start 2024-01-11T09:26:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.602+0000 7fc9569ff700 1 -- start start 2024-01-11T09:26:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.603+0000 7fc9569ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9500b03e0 0x7fc9500c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.603+0000 7fc9569ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9500b07b0 0x7fc9500c3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.603+0000 7fc9569ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9500b2b90 0x7fc9500af6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.603+0000 7fc9569ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9500c43e0 con 0x7fc9500b2b90 2024-01-11T09:26:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.603+0000 7fc9569ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9500c4660 con 0x7fc9500b07b0 2024-01-11T09:26:25.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.603+0000 7fc9569ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9500c4940 con 0x7fc9500b03e0 2024-01-11T09:26:25.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.603+0000 7fc954f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9500b2b90 0x7fc9500af6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:25.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.603+0000 7fc954f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9500b2b90 0x7fc9500af6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35180/0 (socket says 172.21.15.27:35180) 2024-01-11T09:26:25.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.603+0000 7fc954f9c700 1 -- 172.21.15.27:0/2926141156 learned_addr learned my addr 172.21.15.27:0/2926141156 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:25.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.603+0000 7fc94ffff700 1 --2- 172.21.15.27:0/2926141156 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9500b03e0 0x7fc9500c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:25.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.603+0000 7fc94f7fe700 1 --2- 172.21.15.27:0/2926141156 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9500b07b0 0x7fc9500c3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:25.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.603+0000 7fc954f9c700 1 -- 172.21.15.27:0/2926141156 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9500b03e0 msgr2=0x7fc9500c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:25.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.603+0000 7fc954f9c700 1 --2- 172.21.15.27:0/2926141156 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9500b03e0 0x7fc9500c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.603+0000 7fc954f9c700 1 -- 172.21.15.27:0/2926141156 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9500b07b0 msgr2=0x7fc9500c3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:25.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.603+0000 7fc954f9c700 1 --2- 172.21.15.27:0/2926141156 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9500b07b0 0x7fc9500c3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.603+0000 7fc954f9c700 1 -- 172.21.15.27:0/2926141156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc948009ae0 con 0x7fc9500b2b90 2024-01-11T09:26:25.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.604+0000 7fc954f9c700 1 --2- 172.21.15.27:0/2926141156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9500b2b90 0x7fc9500af6a0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fc94800b610 tx=0x7fc948018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.604+0000 7fc937fff700 1 -- 172.21.15.27:0/2926141156 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc948008ad0 con 0x7fc9500b2b90 2024-01-11T09:26:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.604+0000 7fc937fff700 1 -- 172.21.15.27:0/2926141156 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc948008c30 con 0x7fc9500b2b90 2024-01-11T09:26:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.604+0000 7fc9569ff700 1 -- 172.21.15.27:0/2926141156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9500c4c20 con 0x7fc9500b2b90 2024-01-11T09:26:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.604+0000 7fc937fff700 1 -- 172.21.15.27:0/2926141156 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc948026460 con 0x7fc9500b2b90 2024-01-11T09:26:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.604+0000 7fc9569ff700 1 -- 172.21.15.27:0/2926141156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9500c51e0 con 0x7fc9500b2b90 2024-01-11T09:26:25.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.606+0000 7fc9569ff700 1 -- 172.21.15.27:0/2926141156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9500718d0 con 0x7fc9500b2b90 2024-01-11T09:26:25.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.606+0000 7fc937fff700 1 -- 172.21.15.27:0/2926141156 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 86) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fc948003040 con 0x7fc9500b2b90 2024-01-11T09:26:25.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.607+0000 7fc937fff700 1 --2- 172.21.15.27:0/2926141156 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc93c067050 0x7fc93c069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:25.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.607+0000 7fc937fff700 1 -- 172.21.15.27:0/2926141156 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6165+0+0 (secure 0 0 0) 0x7fc948086ba0 con 0x7fc9500b2b90 2024-01-11T09:26:25.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.607+0000 7fc94ffff700 1 -- 172.21.15.27:0/2926141156 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc93c067050 msgr2=0x7fc93c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:25.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.607+0000 7fc94ffff700 1 --2- 172.21.15.27:0/2926141156 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc93c067050 0x7fc93c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:26:25.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.612+0000 7fc937fff700 1 -- 172.21.15.27:0/2926141156 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc948053980 con 0x7fc9500b2b90 2024-01-11T09:26:25.712 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:25.716+0000 7f5cd60ea1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:26:25.735 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:25.738+0000 7f96802e81c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:26:25.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.808+0000 7fc94ffff700 1 -- 172.21.15.27:0/2926141156 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc93c067050 msgr2=0x7fc93c069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:25.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.808+0000 7fc94ffff700 1 --2- 172.21.15.27:0/2926141156 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc93c067050 0x7fc93c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:26:25.811 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:25.814+0000 7f5cd60ea1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:26:25.837 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:25.840+0000 7f96802e81c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:26:25.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.865+0000 7fc9569ff700 1 -- 172.21.15.27:0/2926141156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9500cd3c0 con 0x7fc9500b2b90 2024-01-11T09:26:25.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.866+0000 7fc937fff700 1 -- 172.21.15.27:0/2926141156 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) v1 ==== 74+0+16033 (secure 0 0 0) 0x7fc948056fa0 con 0x7fc9500b2b90 2024-01-11T09:26:25.863 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:26:25.863 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":154,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:26:25.029686+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:30:13.161778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-01-11T08:30:14.620026+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":154,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.3","mappings":[{"from":2,"to":0}]},{"pgid":"19.1f","mappings":[{"from":2,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1967107807":"2024-01-12T08:30:31.980475+0000","172.21.15.77:6836/41381":"2024-01-12T08:30:31.980475+0000","172.21.15.77:6835/41381":"2024-01-12T08:30:31.980475+0000","172.21.15.77:0/864224333":"2024-01-12T08:30:31.980475+0000","172.21.15.77:0/4073869382":"2024-01-12T08:30:31.980475+0000","172.21.15.77:0/1400063768":"2024-01-12T08:30:31.980475+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:26:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.869+0000 7fc9569ff700 1 -- 172.21.15.27:0/2926141156 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc93c067050 msgr2=0x7fc93c069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:26:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.869+0000 7fc9569ff700 1 --2- 172.21.15.27:0/2926141156 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc93c067050 0x7fc93c069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.869+0000 7fc9569ff700 1 -- 172.21.15.27:0/2926141156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9500b2b90 msgr2=0x7fc9500af6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.869+0000 7fc9569ff700 1 --2- 172.21.15.27:0/2926141156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9500b2b90 0x7fc9500af6a0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fc94800b610 tx=0x7fc948018000 comp rx=0 tx=0).stop 2024-01-11T09:26:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.869+0000 7fc9569ff700 1 -- 172.21.15.27:0/2926141156 shutdown_connections 2024-01-11T09:26:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.869+0000 7fc9569ff700 1 --2- 172.21.15.27:0/2926141156 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fc93c067050 0x7fc93c069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.869+0000 7fc9569ff700 1 --2- 172.21.15.27:0/2926141156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9500b2b90 0x7fc9500af6a0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.869+0000 7fc9569ff700 1 --2- 172.21.15.27:0/2926141156 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9500b03e0 0x7fc9500c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.869+0000 7fc9569ff700 1 --2- 172.21.15.27:0/2926141156 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9500b07b0 0x7fc9500c3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.869+0000 7fc9569ff700 1 -- 172.21.15.27:0/2926141156 >> 172.21.15.27:0/2926141156 conn(0x7fc95006ed40 msgr2=0x7fc9500cb400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.869+0000 7fc9569ff700 1 -- 172.21.15.27:0/2926141156 shutdown_connections 2024-01-11T09:26:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:25.869+0000 7fc9569ff700 1 -- 172.21.15.27:0/2926141156 wait complete. 2024-01-11T09:26:25.884 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-11T09:26:25.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:26:25.941 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:25.945+0000 7f96802e81c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:26:26.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.091+0000 7fca439ff700 1 Processor -- start 2024-01-11T09:26:26.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.091+0000 7fca439ff700 1 -- start start 2024-01-11T09:26:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.092+0000 7fca439ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fca3c0ad020 0x7fca3c0af320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.092+0000 7fca439ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fca3c159890 0x7fca3c0aa660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.092+0000 7fca439ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3c0aa1d0 con 0x7fca3c0ac370 2024-01-11T09:26:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.092+0000 7fca439ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca3c0af860 con 0x7fca3c159890 2024-01-11T09:26:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.092+0000 7fca439ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c0ac8d0 con 0x7fca3c0ad020 2024-01-11T09:26:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.093+0000 7fca41f9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fca3c159890 0x7fca3c0aa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.093+0000 7fca4179b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fca3c0ad020 0x7fca3c0af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.093+0000 7fca41f9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fca3c159890 0x7fca3c0aa660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35188/0 (socket says 172.21.15.27:35188) 2024-01-11T09:26:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.093+0000 7fca41f9c700 1 -- 172.21.15.27:0/3736907513 learned_addr learned my addr 172.21.15.27:0/3736907513 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.093+0000 7fca40f9a700 1 --1- 172.21.15.27:0/3736907513 >> v1:172.21.15.27:6789/0 conn(0x7fca3c0ac370 0x7fca3c159180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57476/0 (socket says 172.21.15.27:57476) 2024-01-11T09:26:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.093+0000 7fca41f9c700 1 -- 172.21.15.27:0/3736907513 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fca3c0ad020 msgr2=0x7fca3c0af320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:26.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.093+0000 7fca41f9c700 1 --2- 172.21.15.27:0/3736907513 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fca3c0ad020 0x7fca3c0af320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:26.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.093+0000 7fca41f9c700 1 -- 172.21.15.27:0/3736907513 >> v1:172.21.15.27:6789/0 conn(0x7fca3c0ac370 legacy=0x7fca3c159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:26.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.093+0000 7fca41f9c700 1 -- 172.21.15.27:0/3736907513 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c0a98a0 con 0x7fca3c159890 2024-01-11T09:26:26.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.093+0000 7fca33fff700 1 -- 172.21.15.27:0/3736907513 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723051144 0 0) 0x7fca3c0aa1d0 con 0x7fca3c0ac370 2024-01-11T09:26:26.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.093+0000 7fca4179b700 1 --2- 172.21.15.27:0/3736907513 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fca3c0ad020 0x7fca3c0af320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:26:26.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.094+0000 7fca41f9c700 1 --2- 172.21.15.27:0/3736907513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca3c159890 0x7fca3c0aa660 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fca34009f20 tx=0x7fca34009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1546766b352be56 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:26.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.094+0000 7fca33fff700 1 -- 172.21.15.27:0/3736907513 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3400e070 con 0x7fca3c159890 2024-01-11T09:26:26.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.094+0000 7fca33fff700 1 -- 172.21.15.27:0/3736907513 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3400b9d0 con 0x7fca3c159890 2024-01-11T09:26:26.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.094+0000 7fca33fff700 1 -- 172.21.15.27:0/3736907513 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca340092e0 con 0x7fca3c159890 2024-01-11T09:26:26.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.094+0000 7fca439ff700 1 -- 172.21.15.27:0/3736907513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca3c159890 msgr2=0x7fca3c0aa660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:26.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.094+0000 7fca439ff700 1 --2- 172.21.15.27:0/3736907513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca3c159890 0x7fca3c0aa660 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fca34009f20 tx=0x7fca34009840 comp rx=0 tx=0).stop 2024-01-11T09:26:26.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.095+0000 7fca439ff700 1 -- 172.21.15.27:0/3736907513 shutdown_connections 2024-01-11T09:26:26.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.095+0000 7fca439ff700 1 --2- 172.21.15.27:0/3736907513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca3c159890 0x7fca3c0aa660 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:26.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.095+0000 7fca439ff700 1 --2- 172.21.15.27:0/3736907513 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fca3c0ad020 0x7fca3c0af320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:26.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.095+0000 7fca439ff700 1 -- 172.21.15.27:0/3736907513 >> 172.21.15.27:0/3736907513 conn(0x7fca3c0680e0 msgr2=0x7fca3c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:26.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.095+0000 7fca439ff700 1 -- 172.21.15.27:0/3736907513 shutdown_connections 2024-01-11T09:26:26.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.095+0000 7fca439ff700 1 -- 172.21.15.27:0/3736907513 wait complete. 2024-01-11T09:26:26.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.097+0000 7fca439ff700 1 Processor -- start 2024-01-11T09:26:26.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.097+0000 7fca439ff700 1 -- start start 2024-01-11T09:26:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.098+0000 7fca439ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fca3c0ac370 0x7fca3c155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.098+0000 7fca439ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca3c0ad020 0x7fca3c14c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.098+0000 7fca439ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca3c159890 0x7fca3c149ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.098+0000 7fca439ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c14e6b0 con 0x7fca3c0ad020 2024-01-11T09:26:26.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.098+0000 7fca439ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c14b790 con 0x7fca3c159890 2024-01-11T09:26:26.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.098+0000 7fca439ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c14dd80 con 0x7fca3c0ac370 2024-01-11T09:26:26.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.099+0000 7fca40f9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca3c0ad020 0x7fca3c14c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:26.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.099+0000 7fca4179b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fca3c0ac370 0x7fca3c155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:26.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.099+0000 7fca40f9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca3c0ad020 0x7fca3c14c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35198/0 (socket says 172.21.15.27:35198) 2024-01-11T09:26:26.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.099+0000 7fca40f9a700 1 -- 172.21.15.27:0/2521749846 learned_addr learned my addr 172.21.15.27:0/2521749846 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:26.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.099+0000 7fca41f9c700 1 --2- 172.21.15.27:0/2521749846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca3c159890 0x7fca3c149ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:26.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.099+0000 7fca40f9a700 1 -- 172.21.15.27:0/2521749846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fca3c0ac370 msgr2=0x7fca3c155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:26.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.099+0000 7fca40f9a700 1 --2- 172.21.15.27:0/2521749846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fca3c0ac370 0x7fca3c155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:26.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.099+0000 7fca40f9a700 1 -- 172.21.15.27:0/2521749846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca3c159890 msgr2=0x7fca3c149ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:26.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.099+0000 7fca40f9a700 1 --2- 172.21.15.27:0/2521749846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca3c159890 0x7fca3c149ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:26.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.099+0000 7fca40f9a700 1 -- 172.21.15.27:0/2521749846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca38009420 con 0x7fca3c0ad020 2024-01-11T09:26:26.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.100+0000 7fca40f9a700 1 --2- 172.21.15.27:0/2521749846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca3c0ad020 0x7fca3c14c480 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fca2c0011e0 tx=0x7fca2c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:26.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.100+0000 7fca317fa700 1 -- 172.21.15.27:0/2521749846 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c005a70 con 0x7fca3c0ad020 2024-01-11T09:26:26.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.100+0000 7fca317fa700 1 -- 172.21.15.27:0/2521749846 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2c005bd0 con 0x7fca3c0ad020 2024-01-11T09:26:26.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.100+0000 7fca317fa700 1 -- 172.21.15.27:0/2521749846 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c0070d0 con 0x7fca3c0ad020 2024-01-11T09:26:26.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.100+0000 7fca439ff700 1 -- 172.21.15.27:0/2521749846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca34009ae0 con 0x7fca3c0ad020 2024-01-11T09:26:26.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.100+0000 7fca439ff700 1 -- 172.21.15.27:0/2521749846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca3c14e340 con 0x7fca3c0ad020 2024-01-11T09:26:26.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.101+0000 7fca439ff700 1 -- 172.21.15.27:0/2521749846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca3c14e860 con 0x7fca3c0ad020 2024-01-11T09:26:26.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.102+0000 7fca317fa700 1 -- 172.21.15.27:0/2521749846 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 86) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fca2c007480 con 0x7fca3c0ad020 2024-01-11T09:26:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.103+0000 7fca317fa700 1 --2- 172.21.15.27:0/2521749846 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fca20067050 0x7fca20069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.103+0000 7fca317fa700 1 -- 172.21.15.27:0/2521749846 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6165+0+0 (secure 0 0 0) 0x7fca2c083dd0 con 0x7fca3c0ad020 2024-01-11T09:26:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.103+0000 7fca4179b700 1 -- 172.21.15.27:0/2521749846 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fca20067050 msgr2=0x7fca20069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.103+0000 7fca4179b700 1 --2- 172.21.15.27:0/2521749846 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fca20067050 0x7fca20069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:26:26.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.107+0000 7fca317fa700 1 -- 172.21.15.27:0/2521749846 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fca3c14e860 con 0x7fca3c0ad020 2024-01-11T09:26:26.157 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:26.161+0000 7f5cd60ea1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:26:26.179 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:26.183+0000 7f96802e81c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:26:26.246 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:26.249+0000 7f5cd60ea1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:26:26.288 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:26.292+0000 7f96802e81c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:26:26.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.304+0000 7fca4179b700 1 -- 172.21.15.27:0/2521749846 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fca20067050 msgr2=0x7fca20069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:26.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.304+0000 7fca4179b700 1 --2- 172.21.15.27:0/2521749846 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fca20067050 0x7fca20069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:26:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.404+0000 7fca439ff700 1 -- 172.21.15.27:0/2521749846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca3c0718d0 con 0x7fca3c0ad020 2024-01-11T09:26:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.405+0000 7fca317fa700 1 -- 172.21.15.27:0/2521749846 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4596 (secure 0 0 0) 0x7fca2c017070 con 0x7fca3c0ad020 2024-01-11T09:26:26.402 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:26:26.402 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7352,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3818350333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3818350333},{"type":"v1","addr":"172.21.15.27:6834","nonce":3818350333}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/217390254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":217390254},{"type":"v1","addr":"172.21.15.27:6835","nonce":217390254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2824437717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2824437717},{"type":"v1","addr":"172.21.15.77:6834","nonce":2824437717}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:30:15.623796+0000","modified":"2024-01-11T08:30:19.150622+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8601":{"gid":8601,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/1641163931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1641163931},{"type":"v1","addr":"172.21.15.77:6837","nonce":1641163931}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2024-01-11T09:26:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.407+0000 7fca439ff700 1 -- 172.21.15.27:0/2521749846 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fca20067050 msgr2=0x7fca20069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:26:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.407+0000 7fca439ff700 1 --2- 172.21.15.27:0/2521749846 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fca20067050 0x7fca20069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.407+0000 7fca439ff700 1 -- 172.21.15.27:0/2521749846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca3c0ad020 msgr2=0x7fca3c14c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.407+0000 7fca439ff700 1 --2- 172.21.15.27:0/2521749846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca3c0ad020 0x7fca3c14c480 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fca2c0011e0 tx=0x7fca2c007e10 comp rx=0 tx=0).stop 2024-01-11T09:26:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.408+0000 7fca439ff700 1 -- 172.21.15.27:0/2521749846 shutdown_connections 2024-01-11T09:26:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.408+0000 7fca439ff700 1 --2- 172.21.15.27:0/2521749846 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fca20067050 0x7fca20069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.408+0000 7fca439ff700 1 --2- 172.21.15.27:0/2521749846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca3c159890 0x7fca3c149ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.408+0000 7fca439ff700 1 --2- 172.21.15.27:0/2521749846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fca3c0ac370 0x7fca3c155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.408+0000 7fca439ff700 1 --2- 172.21.15.27:0/2521749846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca3c0ad020 0x7fca3c14c480 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.408+0000 7fca439ff700 1 -- 172.21.15.27:0/2521749846 >> 172.21.15.27:0/2521749846 conn(0x7fca3c0680e0 msgr2=0x7fca3c154d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.408+0000 7fca439ff700 1 -- 172.21.15.27:0/2521749846 shutdown_connections 2024-01-11T09:26:26.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.408+0000 7fca439ff700 1 -- 172.21.15.27:0/2521749846 wait complete. 2024-01-11T09:26:26.406 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 77 2024-01-11T09:26:26.423 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:26:26.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.633+0000 7fdb1b9ff700 1 Processor -- start 2024-01-11T09:26:26.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.633+0000 7fdb1b9ff700 1 -- start start 2024-01-11T09:26:26.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.634+0000 7fdb1b9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdb140ca3e0 0x7fdb140cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:26.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.634+0000 7fdb1b9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb1415d640 con 0x7fdb140ca7b0 2024-01-11T09:26:26.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.634+0000 7fdb1b9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb1415a5d0 con 0x7fdb14159e80 2024-01-11T09:26:26.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.634+0000 7fdb1b9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb1415ae70 con 0x7fdb140ca3e0 2024-01-11T09:26:26.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.635+0000 7fdb1979b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdb140ca3e0 0x7fdb140cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:26.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.635+0000 7fdb1979b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdb140ca3e0 0x7fdb140cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35200/0 (socket says 172.21.15.27:35200) 2024-01-11T09:26:26.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.635+0000 7fdb1979b700 1 -- 172.21.15.27:0/471971633 learned_addr learned my addr 172.21.15.27:0/471971633 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:26.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.635+0000 7fdb19f9c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdb14159e80 0x7fdb1415dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:36694/0 (socket says 172.21.15.27:36694) 2024-01-11T09:26:26.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.635+0000 7fdb18f9a700 1 --1- 172.21.15.27:0/471971633 >> v1:172.21.15.27:6789/0 conn(0x7fdb140ca7b0 0x7fdb140c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57486/0 (socket says 172.21.15.27:57486) 2024-01-11T09:26:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.635+0000 7fdb1979b700 1 -- 172.21.15.27:0/471971633 >> v1:172.21.15.77:6789/0 conn(0x7fdb14159e80 legacy=0x7fdb1415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.635+0000 7fdb1979b700 1 -- 172.21.15.27:0/471971633 >> v1:172.21.15.27:6789/0 conn(0x7fdb140ca7b0 legacy=0x7fdb140c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.635+0000 7fdb1979b700 1 -- 172.21.15.27:0/471971633 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1415a8b0 con 0x7fdb140ca3e0 2024-01-11T09:26:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.635+0000 7fdb0bfff700 1 -- 172.21.15.27:0/471971633 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936081744 0 0) 0x7fdb1415a5d0 con 0x7fdb14159e80 2024-01-11T09:26:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.636+0000 7fdb1979b700 1 --2- 172.21.15.27:0/471971633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb140ca3e0 0x7fdb140cca20 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fdb100099e0 tx=0x7fdb10009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9740d83fe793f93 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.636+0000 7fdb0bfff700 1 -- 172.21.15.27:0/471971633 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1000b820 con 0x7fdb140ca3e0 2024-01-11T09:26:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.636+0000 7fdb0bfff700 1 -- 172.21.15.27:0/471971633 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1000b980 con 0x7fdb140ca3e0 2024-01-11T09:26:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.636+0000 7fdb0bfff700 1 -- 172.21.15.27:0/471971633 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1000bd20 con 0x7fdb140ca3e0 2024-01-11T09:26:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.636+0000 7fdb1b9ff700 1 -- 172.21.15.27:0/471971633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb140ca3e0 msgr2=0x7fdb140cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.636+0000 7fdb1b9ff700 1 --2- 172.21.15.27:0/471971633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb140ca3e0 0x7fdb140cca20 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fdb100099e0 tx=0x7fdb10009250 comp rx=0 tx=0).stop 2024-01-11T09:26:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.637+0000 7fdb1b9ff700 1 -- 172.21.15.27:0/471971633 shutdown_connections 2024-01-11T09:26:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.637+0000 7fdb1b9ff700 1 --2- 172.21.15.27:0/471971633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb140ca3e0 0x7fdb140cca20 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.637+0000 7fdb1b9ff700 1 -- 172.21.15.27:0/471971633 >> 172.21.15.27:0/471971633 conn(0x7fdb140ce3e0 msgr2=0x7fdb140ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.637+0000 7fdb1b9ff700 1 -- 172.21.15.27:0/471971633 shutdown_connections 2024-01-11T09:26:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.637+0000 7fdb1b9ff700 1 -- 172.21.15.27:0/471971633 wait complete. 2024-01-11T09:26:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.638+0000 7fdb1b9ff700 1 Processor -- start 2024-01-11T09:26:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.638+0000 7fdb1b9ff700 1 -- start start 2024-01-11T09:26:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.639+0000 7fdb1b9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb140ca3e0 0x7fdb140b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.639+0000 7fdb1b9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb140ca7b0 0x7fdb140c6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.639+0000 7fdb1b9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb14159e80 0x7fdb14143a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.639+0000 7fdb1b9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1416a0b0 con 0x7fdb140ca3e0 2024-01-11T09:26:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.639+0000 7fdb1b9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb140c9100 con 0x7fdb14159e80 2024-01-11T09:26:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.639+0000 7fdb1b9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb140c93e0 con 0x7fdb140ca7b0 2024-01-11T09:26:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.640+0000 7fdb1979b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb140ca3e0 0x7fdb140b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.640+0000 7fdb1979b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb140ca3e0 0x7fdb140b2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35210/0 (socket says 172.21.15.27:35210) 2024-01-11T09:26:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.640+0000 7fdb1979b700 1 -- 172.21.15.27:0/2665900087 learned_addr learned my addr 172.21.15.27:0/2665900087 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.640+0000 7fdb19f9c700 1 --2- 172.21.15.27:0/2665900087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb14159e80 0x7fdb14143a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.640+0000 7fdb18f9a700 1 --2- 172.21.15.27:0/2665900087 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb140ca7b0 0x7fdb140c6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.640+0000 7fdb1979b700 1 -- 172.21.15.27:0/2665900087 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb140ca7b0 msgr2=0x7fdb140c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.640+0000 7fdb1979b700 1 --2- 172.21.15.27:0/2665900087 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb140ca7b0 0x7fdb140c6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.640+0000 7fdb1979b700 1 -- 172.21.15.27:0/2665900087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb14159e80 msgr2=0x7fdb14143a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.640+0000 7fdb1979b700 1 --2- 172.21.15.27:0/2665900087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb14159e80 0x7fdb14143a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.640+0000 7fdb1979b700 1 -- 172.21.15.27:0/2665900087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb100094f0 con 0x7fdb140ca3e0 2024-01-11T09:26:26.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.640+0000 7fdb18f9a700 1 --2- 172.21.15.27:0/2665900087 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb140ca7b0 0x7fdb140c6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:26.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.640+0000 7fdb1979b700 1 --2- 172.21.15.27:0/2665900087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb140ca3e0 0x7fdb140b2d70 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fdb14064980 tx=0x7fdb10017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:26.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.641+0000 7fdb097fa700 1 -- 172.21.15.27:0/2665900087 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1001f070 con 0x7fdb140ca3e0 2024-01-11T09:26:26.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.641+0000 7fdb097fa700 1 -- 172.21.15.27:0/2665900087 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1000baf0 con 0x7fdb140ca3e0 2024-01-11T09:26:26.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.641+0000 7fdb1b9ff700 1 -- 172.21.15.27:0/2665900087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb140c96c0 con 0x7fdb140ca3e0 2024-01-11T09:26:26.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.641+0000 7fdb097fa700 1 -- 172.21.15.27:0/2665900087 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb10025420 con 0x7fdb140ca3e0 2024-01-11T09:26:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.641+0000 7fdb1b9ff700 1 -- 172.21.15.27:0/2665900087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb140c8d40 con 0x7fdb140ca3e0 2024-01-11T09:26:26.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.642+0000 7fdb1b9ff700 1 -- 172.21.15.27:0/2665900087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb140718d0 con 0x7fdb140ca3e0 2024-01-11T09:26:26.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.644+0000 7fdb097fa700 1 -- 172.21.15.27:0/2665900087 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 86) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fdb10023070 con 0x7fdb140ca3e0 2024-01-11T09:26:26.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.645+0000 7fdb097fa700 1 --2- 172.21.15.27:0/2665900087 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fdaf8067050 0x7fdaf8069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:26.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.645+0000 7fdb097fa700 1 -- 172.21.15.27:0/2665900087 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6165+0+0 (secure 0 0 0) 0x7fdb10085d10 con 0x7fdb140ca3e0 2024-01-11T09:26:26.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.645+0000 7fdb18f9a700 1 -- 172.21.15.27:0/2665900087 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fdaf8067050 msgr2=0x7fdaf8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:26.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.645+0000 7fdb18f9a700 1 --2- 172.21.15.27:0/2665900087 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fdaf8067050 0x7fdaf8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:26:26.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.649+0000 7fdb097fa700 1 -- 172.21.15.27:0/2665900087 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdb10052ca0 con 0x7fdb140ca3e0 2024-01-11T09:26:26.660 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:26.664+0000 7f96802e81c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:26:26.756 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:26.760+0000 7f96802e81c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:26:26.784 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:26.788+0000 7f5cd60ea1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:26:26.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.846+0000 7fdb18f9a700 1 -- 172.21.15.27:0/2665900087 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fdaf8067050 msgr2=0x7fdaf8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:26.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.846+0000 7fdb18f9a700 1 --2- 172.21.15.27:0/2665900087 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fdaf8067050 0x7fdaf8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:26:26.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.901+0000 7fdb1b9ff700 1 -- 172.21.15.27:0/2665900087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb1406c840 con 0x7fdb140ca3e0 2024-01-11T09:26:26.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.902+0000 7fdb097fa700 1 -- 172.21.15.27:0/2665900087 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4596 (secure 0 0 0) 0x7fdb10015090 con 0x7fdb140ca3e0 2024-01-11T09:26:26.899 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:26:26.899 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":77,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7352,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3818350333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3818350333},{"type":"v1","addr":"172.21.15.27:6834","nonce":3818350333}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/217390254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":217390254},{"type":"v1","addr":"172.21.15.27:6835","nonce":217390254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2824437717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2824437717},{"type":"v1","addr":"172.21.15.77:6834","nonce":2824437717}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T08:30:15.623796+0000","modified":"2024-01-11T08:30:19.150622+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8601":{"gid":8601,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.77:6837/1641163931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1641163931},{"type":"v1","addr":"172.21.15.77:6837","nonce":1641163931}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2024-01-11T09:26:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.905+0000 7fdb1b9ff700 1 -- 172.21.15.27:0/2665900087 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fdaf8067050 msgr2=0x7fdaf8069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:26:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.905+0000 7fdb1b9ff700 1 --2- 172.21.15.27:0/2665900087 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fdaf8067050 0x7fdaf8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.905+0000 7fdb1b9ff700 1 -- 172.21.15.27:0/2665900087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb140ca3e0 msgr2=0x7fdb140b2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.905+0000 7fdb1b9ff700 1 --2- 172.21.15.27:0/2665900087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb140ca3e0 0x7fdb140b2d70 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fdb14064980 tx=0x7fdb10017000 comp rx=0 tx=0).stop 2024-01-11T09:26:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.905+0000 7fdb1b9ff700 1 -- 172.21.15.27:0/2665900087 shutdown_connections 2024-01-11T09:26:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.905+0000 7fdb1b9ff700 1 --2- 172.21.15.27:0/2665900087 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fdaf8067050 0x7fdaf8069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.905+0000 7fdb1b9ff700 1 --2- 172.21.15.27:0/2665900087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb14159e80 0x7fdb14143a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.905+0000 7fdb1b9ff700 1 --2- 172.21.15.27:0/2665900087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb140ca3e0 0x7fdb140b2d70 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.905+0000 7fdb1b9ff700 1 --2- 172.21.15.27:0/2665900087 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdb140ca7b0 0x7fdb140c6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.905+0000 7fdb1b9ff700 1 -- 172.21.15.27:0/2665900087 >> 172.21.15.27:0/2665900087 conn(0x7fdb140ce3e0 msgr2=0x7fdb14068700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.905+0000 7fdb1b9ff700 1 -- 172.21.15.27:0/2665900087 shutdown_connections 2024-01-11T09:26:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:26.905+0000 7fdb1b9ff700 1 -- 172.21.15.27:0/2665900087 wait complete. 2024-01-11T09:26:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 77 2024-01-11T09:26:26.920 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:26:26.971 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:26.975+0000 7f5cd60ea1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:26:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.125+0000 7f48e35ff700 1 Processor -- start 2024-01-11T09:26:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.125+0000 7f48e35ff700 1 -- start start 2024-01-11T09:26:27.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.127+0000 7f48e35ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48dc0ca3e0 0x7f48dc0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:27.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.127+0000 7f48e35ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f48dc159e80 0x7f48dc15dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:27.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.127+0000 7f48e35ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48dc15d640 con 0x7f48dc0ca7b0 2024-01-11T09:26:27.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.127+0000 7f48e35ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48dc15ae70 con 0x7f48dc159e80 2024-01-11T09:26:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.127+0000 7f48e35ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc0cc4f0 con 0x7f48dc0ca3e0 2024-01-11T09:26:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.128+0000 7f48e1b9c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f48dc159e80 0x7f48dc15dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.128+0000 7f48e139b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48dc0ca3e0 0x7f48dc0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.128+0000 7f48e0b9a700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f48dc0ca7b0 0x7f48dc0c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:36710/0 (socket says 172.21.15.27:36710) 2024-01-11T09:26:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.128+0000 7f48e139b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48dc0ca3e0 0x7f48dc0cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:44802/0 (socket says 172.21.15.27:44802) 2024-01-11T09:26:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.128+0000 7f48e1b9c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f48dc159e80 0x7f48dc15dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:47820/0 (socket says 172.21.15.27:47820) 2024-01-11T09:26:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.128+0000 7f48e0b9a700 1 -- 172.21.15.27:0/944667286 learned_addr learned my addr 172.21.15.27:0/944667286 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.128+0000 7f48e1b9c700 1 -- 172.21.15.27:0/944667286 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48dc0ca3e0 msgr2=0x7f48dc0cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.128+0000 7f48e1b9c700 1 --2- 172.21.15.27:0/944667286 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48dc0ca3e0 0x7f48dc0cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.128+0000 7f48e1b9c700 1 -- 172.21.15.27:0/944667286 >> v1:172.21.15.77:6789/0 conn(0x7f48dc0ca7b0 legacy=0x7f48dc0c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.128+0000 7f48e1b9c700 1 -- 172.21.15.27:0/944667286 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48dc15a5d0 con 0x7f48dc159e80 2024-01-11T09:26:27.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.128+0000 7f48e139b700 1 --2- 172.21.15.27:0/944667286 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48dc0ca3e0 0x7f48dc0cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:26:27.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.129+0000 7f48e1b9c700 1 --2- 172.21.15.27:0/944667286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48dc159e80 0x7f48dc15dad0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009e60 tx=0x7f48d4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8d6c2cb13c59e97 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:27.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.129+0000 7f48d3fff700 1 -- 172.21.15.27:0/944667286 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d400e070 con 0x7f48dc159e80 2024-01-11T09:26:27.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.129+0000 7f48d3fff700 1 -- 172.21.15.27:0/944667286 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d400bb60 con 0x7f48dc159e80 2024-01-11T09:26:27.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.129+0000 7f48d3fff700 1 -- 172.21.15.27:0/944667286 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d4012430 con 0x7f48dc159e80 2024-01-11T09:26:27.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.129+0000 7f48e35ff700 1 -- 172.21.15.27:0/944667286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48dc159e80 msgr2=0x7f48dc15dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:27.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.129+0000 7f48e35ff700 1 --2- 172.21.15.27:0/944667286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48dc159e80 0x7f48dc15dad0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009e60 tx=0x7f48d4009780 comp rx=0 tx=0).stop 2024-01-11T09:26:27.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.130+0000 7f48e35ff700 1 -- 172.21.15.27:0/944667286 shutdown_connections 2024-01-11T09:26:27.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.130+0000 7f48e35ff700 1 --2- 172.21.15.27:0/944667286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48dc159e80 0x7f48dc15dad0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.130+0000 7f48e35ff700 1 --2- 172.21.15.27:0/944667286 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48dc0ca3e0 0x7f48dc0cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.130+0000 7f48e35ff700 1 -- 172.21.15.27:0/944667286 >> 172.21.15.27:0/944667286 conn(0x7f48dc0ce3e0 msgr2=0x7f48dc0ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:27.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.130+0000 7f48e35ff700 1 -- 172.21.15.27:0/944667286 shutdown_connections 2024-01-11T09:26:27.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.130+0000 7f48e35ff700 1 -- 172.21.15.27:0/944667286 wait complete. 2024-01-11T09:26:27.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.132+0000 7f48e35ff700 1 Processor -- start 2024-01-11T09:26:27.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.132+0000 7f48e35ff700 1 -- start start 2024-01-11T09:26:27.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.133+0000 7f48e35ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48dc0ca3e0 0x7f48dc147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:27.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.133+0000 7f48e35ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48dc0ca7b0 0x7f48dc14ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:27.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.133+0000 7f48e35ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48dc159e80 0x7f48dc14b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:27.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.133+0000 7f48e35ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc0b2030 con 0x7f48dc159e80 2024-01-11T09:26:27.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.133+0000 7f48e35ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc0b1180 con 0x7f48dc0ca3e0 2024-01-11T09:26:27.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.133+0000 7f48e35ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc0b1420 con 0x7f48dc0ca7b0 2024-01-11T09:26:27.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.134+0000 7f48e139b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48dc0ca3e0 0x7f48dc147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:27.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.134+0000 7f48e0b9a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48dc0ca7b0 0x7f48dc14ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:27.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.134+0000 7f48e1b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48dc159e80 0x7f48dc14b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:27.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.134+0000 7f48e139b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48dc0ca3e0 0x7f48dc147e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:47824/0 (socket says 172.21.15.27:47824) 2024-01-11T09:26:27.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.134+0000 7f48e139b700 1 -- 172.21.15.27:0/3816358130 learned_addr learned my addr 172.21.15.27:0/3816358130 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:27.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.134+0000 7f48e0b9a700 1 -- 172.21.15.27:0/3816358130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48dc0ca3e0 msgr2=0x7f48dc147e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:27.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.134+0000 7f48e0b9a700 1 --2- 172.21.15.27:0/3816358130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48dc0ca3e0 0x7f48dc147e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.134+0000 7f48e0b9a700 1 -- 172.21.15.27:0/3816358130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48dc159e80 msgr2=0x7f48dc14b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:27.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.134+0000 7f48e0b9a700 1 --2- 172.21.15.27:0/3816358130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48dc159e80 0x7f48dc14b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.134+0000 7f48e0b9a700 1 -- 172.21.15.27:0/3816358130 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d8009420 con 0x7f48dc0ca7b0 2024-01-11T09:26:27.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.134+0000 7f48e0b9a700 1 --2- 172.21.15.27:0/3816358130 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48dc0ca7b0 0x7f48dc14ad10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00c7c0 tx=0x7f48cc007eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.135+0000 7f48d17fa700 1 -- 172.21.15.27:0/3816358130 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc005b30 con 0x7f48dc0ca7b0 2024-01-11T09:26:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.135+0000 7f48d17fa700 1 -- 172.21.15.27:0/3816358130 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48cc005c90 con 0x7f48dc0ca7b0 2024-01-11T09:26:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.135+0000 7f48d17fa700 1 -- 172.21.15.27:0/3816358130 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc004a80 con 0x7f48dc0ca7b0 2024-01-11T09:26:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.135+0000 7f48e35ff700 1 -- 172.21.15.27:0/3816358130 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d4009a20 con 0x7f48dc0ca7b0 2024-01-11T09:26:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.135+0000 7f48e35ff700 1 -- 172.21.15.27:0/3816358130 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48dc0b19e0 con 0x7f48dc0ca7b0 2024-01-11T09:26:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.136+0000 7f48e35ff700 1 -- 172.21.15.27:0/3816358130 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48dc0b21a0 con 0x7f48dc0ca7b0 2024-01-11T09:26:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.137+0000 7f48d17fa700 1 -- 172.21.15.27:0/3816358130 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 86) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f48cc0070d0 con 0x7f48dc0ca7b0 2024-01-11T09:26:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.138+0000 7f48d17fa700 1 --2- 172.21.15.27:0/3816358130 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48c0067050 0x7f48c0069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.138+0000 7f48d17fa700 1 -- 172.21.15.27:0/3816358130 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f48cc083c10 con 0x7f48dc0ca7b0 2024-01-11T09:26:27.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.138+0000 7f48e139b700 1 -- 172.21.15.27:0/3816358130 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48c0067050 msgr2=0x7f48c0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.138+0000 7f48e139b700 1 --2- 172.21.15.27:0/3816358130 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48c0067050 0x7f48c0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:26:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.143+0000 7f48d17fa700 1 -- 172.21.15.27:0/3816358130 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f48cc04b5a0 con 0x7f48dc0ca7b0 2024-01-11T09:26:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.339+0000 7f48e139b700 1 -- 172.21.15.27:0/3816358130 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48c0067050 msgr2=0x7f48c0069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:27.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.339+0000 7f48e139b700 1 --2- 172.21.15.27:0/3816358130 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48c0067050 0x7f48c0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:26:27.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.349+0000 7f48e35ff700 1 -- 172.21.15.27:0/3816358130 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48dc15a8b0 con 0x7f48dc0ca7b0 2024-01-11T09:26:27.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.350+0000 7f48d17fa700 1 -- 172.21.15.27:0/3816358130 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) v1 ==== 74+0+16033 (secure 0 0 0) 0x7f48cc050b00 con 0x7f48dc0ca7b0 2024-01-11T09:26:27.347 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:26:27.347 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":154,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:26:25.029686+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-01-11T08:30:13.161778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-01-11T08:30:14.620026+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":154,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.3","mappings":[{"from":2,"to":0}]},{"pgid":"19.1f","mappings":[{"from":2,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1967107807":"2024-01-12T08:30:31.980475+0000","172.21.15.77:6836/41381":"2024-01-12T08:30:31.980475+0000","172.21.15.77:6835/41381":"2024-01-12T08:30:31.980475+0000","172.21.15.77:0/864224333":"2024-01-12T08:30:31.980475+0000","172.21.15.77:0/4073869382":"2024-01-12T08:30:31.980475+0000","172.21.15.77:0/1400063768":"2024-01-12T08:30:31.980475+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:26:27.349 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:27.352+0000 7f5cd60ea1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:26:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.352+0000 7f48e35ff700 1 -- 172.21.15.27:0/3816358130 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48c0067050 msgr2=0x7f48c0069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:26:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.352+0000 7f48e35ff700 1 --2- 172.21.15.27:0/3816358130 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48c0067050 0x7f48c0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.352+0000 7f48e35ff700 1 -- 172.21.15.27:0/3816358130 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48dc0ca7b0 msgr2=0x7f48dc14ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.352+0000 7f48e35ff700 1 --2- 172.21.15.27:0/3816358130 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48dc0ca7b0 0x7f48dc14ad10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00c7c0 tx=0x7f48cc007eb0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.352+0000 7f48e35ff700 1 -- 172.21.15.27:0/3816358130 shutdown_connections 2024-01-11T09:26:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.352+0000 7f48e35ff700 1 --2- 172.21.15.27:0/3816358130 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f48c0067050 0x7f48c0069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.352+0000 7f48e35ff700 1 --2- 172.21.15.27:0/3816358130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48dc159e80 0x7f48dc14b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.352+0000 7f48e35ff700 1 --2- 172.21.15.27:0/3816358130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48dc0ca3e0 0x7f48dc147e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.352+0000 7f48e35ff700 1 --2- 172.21.15.27:0/3816358130 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f48dc0ca7b0 0x7f48dc14ad10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.352+0000 7f48e35ff700 1 -- 172.21.15.27:0/3816358130 >> 172.21.15.27:0/3816358130 conn(0x7f48dc0ce3e0 msgr2=0x7f48dc069a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:27.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.353+0000 7f48e35ff700 1 -- 172.21.15.27:0/3816358130 shutdown_connections 2024-01-11T09:26:27.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.353+0000 7f48e35ff700 1 -- 172.21.15.27:0/3816358130 wait complete. 2024-01-11T09:26:27.351 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:27.353+0000 7f96802e81c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:26:27.368 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-11T09:26:27.552 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:27.556+0000 7f96802e81c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:26:27.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.573+0000 7fb42c5ff700 1 Processor -- start 2024-01-11T09:26:27.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.573+0000 7fb42c5ff700 1 -- start start 2024-01-11T09:26:27.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.574+0000 7fb42c5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb42415ca10 0x7fb42415a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:27.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.574+0000 7fb42c5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb42415cde0 0x7fb4240685c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:27.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.574+0000 7fb42c5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4240b4bd0 con 0x7fb42415aed0 2024-01-11T09:26:27.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.574+0000 7fb42c5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb42415ac70 con 0x7fb42415ca10 2024-01-11T09:26:27.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.574+0000 7fb42c5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4240b3fc0 con 0x7fb42415cde0 2024-01-11T09:26:27.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.574+0000 7fb42a39b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb42415ca10 0x7fb42415a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:27.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.574+0000 7fb42a39b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb42415ca10 0x7fb42415a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35232/0 (socket says 172.21.15.27:35232) 2024-01-11T09:26:27.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.575+0000 7fb42a39b700 1 -- 172.21.15.27:0/933466681 learned_addr learned my addr 172.21.15.27:0/933466681 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:27.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.575+0000 7fb429b9a700 1 --2- 172.21.15.27:0/933466681 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb42415cde0 0x7fb4240685c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:27.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.575+0000 7fb42a39b700 1 -- 172.21.15.27:0/933466681 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb42415cde0 msgr2=0x7fb4240685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:27.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.575+0000 7fb42a39b700 1 --2- 172.21.15.27:0/933466681 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb42415cde0 0x7fb4240685c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.575+0000 7fb42a39b700 1 -- 172.21.15.27:0/933466681 >> v1:172.21.15.77:6789/0 conn(0x7fb42415aed0 legacy=0x7fb42415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:27.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.575+0000 7fb42a39b700 1 -- 172.21.15.27:0/933466681 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4240b42a0 con 0x7fb42415ca10 2024-01-11T09:26:27.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.575+0000 7fb429399700 1 -- 172.21.15.27:0/933466681 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717777816 0 0) 0x7fb4240b4bd0 con 0x7fb42415aed0 2024-01-11T09:26:27.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.575+0000 7fb42a39b700 1 --2- 172.21.15.27:0/933466681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb42415ca10 0x7fb42415a730 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fb4200099b0 tx=0x7fb420009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70bfae53ad9eda88 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:27.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.575+0000 7fb429399700 1 -- 172.21.15.27:0/933466681 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb420009ea0 con 0x7fb42415ca10 2024-01-11T09:26:27.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.576+0000 7fb429399700 1 -- 172.21.15.27:0/933466681 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42000b860 con 0x7fb42415ca10 2024-01-11T09:26:27.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.576+0000 7fb429399700 1 -- 172.21.15.27:0/933466681 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42000bc00 con 0x7fb42415ca10 2024-01-11T09:26:27.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.576+0000 7fb42c5ff700 1 -- 172.21.15.27:0/933466681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb42415ca10 msgr2=0x7fb42415a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.576+0000 7fb42c5ff700 1 --2- 172.21.15.27:0/933466681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb42415ca10 0x7fb42415a730 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fb4200099b0 tx=0x7fb420009250 comp rx=0 tx=0).stop 2024-01-11T09:26:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.576+0000 7fb42c5ff700 1 -- 172.21.15.27:0/933466681 shutdown_connections 2024-01-11T09:26:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.576+0000 7fb42c5ff700 1 --2- 172.21.15.27:0/933466681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb42415ca10 0x7fb42415a730 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.576+0000 7fb42c5ff700 1 --2- 172.21.15.27:0/933466681 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb42415cde0 0x7fb4240685c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.576+0000 7fb42c5ff700 1 -- 172.21.15.27:0/933466681 >> 172.21.15.27:0/933466681 conn(0x7fb42406ed40 msgr2=0x7fb42406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.576+0000 7fb42c5ff700 1 -- 172.21.15.27:0/933466681 shutdown_connections 2024-01-11T09:26:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.576+0000 7fb42c5ff700 1 -- 172.21.15.27:0/933466681 wait complete. 2024-01-11T09:26:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.578+0000 7fb42c5ff700 1 Processor -- start 2024-01-11T09:26:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.578+0000 7fb42c5ff700 1 -- start start 2024-01-11T09:26:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.579+0000 7fb42c5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb42415aed0 0x7fb424149020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.579+0000 7fb42c5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb42415ca10 0x7fb42414a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.579+0000 7fb42c5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb42415cde0 0x7fb42416d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.579+0000 7fb42c5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42414eec0 con 0x7fb42415ca10 2024-01-11T09:26:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.579+0000 7fb42c5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42414b340 con 0x7fb42415aed0 2024-01-11T09:26:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.579+0000 7fb42c5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42414b620 con 0x7fb42415cde0 2024-01-11T09:26:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.579+0000 7fb429b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb42415ca10 0x7fb42414a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.579+0000 7fb429b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb42415ca10 0x7fb42414a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35246/0 (socket says 172.21.15.27:35246) 2024-01-11T09:26:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.579+0000 7fb42a39b700 1 --2- 172.21.15.27:0/1549841897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb42415aed0 0x7fb424149020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.579+0000 7fb429b9a700 1 -- 172.21.15.27:0/1549841897 learned_addr learned my addr 172.21.15.27:0/1549841897 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.579+0000 7fb42ab9c700 1 --2- 172.21.15.27:0/1549841897 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb42415cde0 0x7fb42416d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.580+0000 7fb42a39b700 1 -- 172.21.15.27:0/1549841897 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb42415cde0 msgr2=0x7fb42416d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.580+0000 7fb42a39b700 1 --2- 172.21.15.27:0/1549841897 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb42415cde0 0x7fb42416d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.580+0000 7fb42a39b700 1 -- 172.21.15.27:0/1549841897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb42415ca10 msgr2=0x7fb42414a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.580+0000 7fb42a39b700 1 --2- 172.21.15.27:0/1549841897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb42415ca10 0x7fb42414a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.580+0000 7fb42a39b700 1 -- 172.21.15.27:0/1549841897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4200094f0 con 0x7fb42415aed0 2024-01-11T09:26:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.580+0000 7fb429b9a700 1 --2- 172.21.15.27:0/1549841897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb42415ca10 0x7fb42414a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:26:27.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.580+0000 7fb42a39b700 1 --2- 172.21.15.27:0/1549841897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb42415aed0 0x7fb424149020 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb42415d230 tx=0x7fb420017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:27.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.580+0000 7fb41a7fc700 1 -- 172.21.15.27:0/1549841897 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb420000d10 con 0x7fb42415aed0 2024-01-11T09:26:27.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.580+0000 7fb41a7fc700 1 -- 172.21.15.27:0/1549841897 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42000bbe0 con 0x7fb42415aed0 2024-01-11T09:26:27.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.580+0000 7fb41a7fc700 1 -- 172.21.15.27:0/1549841897 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb420011630 con 0x7fb42415aed0 2024-01-11T09:26:27.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.581+0000 7fb42c5ff700 1 -- 172.21.15.27:0/1549841897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42414eb50 con 0x7fb42415aed0 2024-01-11T09:26:27.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.581+0000 7fb42c5ff700 1 -- 172.21.15.27:0/1549841897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb424149560 con 0x7fb42415aed0 2024-01-11T09:26:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.582+0000 7fb42c5ff700 1 -- 172.21.15.27:0/1549841897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42404c3c0 con 0x7fb42415aed0 2024-01-11T09:26:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.582+0000 7fb41a7fc700 1 -- 172.21.15.27:0/1549841897 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 86) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fb420011a60 con 0x7fb42415aed0 2024-01-11T09:26:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.583+0000 7fb41a7fc700 1 --2- 172.21.15.27:0/1549841897 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb408067050 0x7fb408069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.583+0000 7fb41a7fc700 1 -- 172.21.15.27:0/1549841897 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6165+0+0 (secure 0 0 0) 0x7fb420085a10 con 0x7fb42415aed0 2024-01-11T09:26:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.584+0000 7fb429b9a700 1 -- 172.21.15.27:0/1549841897 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb408067050 msgr2=0x7fb408069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.584+0000 7fb429b9a700 1 --2- 172.21.15.27:0/1549841897 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb408067050 0x7fb408069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:26:27.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.588+0000 7fb41a7fc700 1 -- 172.21.15.27:0/1549841897 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb420052980 con 0x7fb42415aed0 2024-01-11T09:26:27.772 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:27.776+0000 7f5cd60ea1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:26:27.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.785+0000 7fb429b9a700 1 -- 172.21.15.27:0/1549841897 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb408067050 msgr2=0x7fb408069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6835/41381 2024-01-11T09:26:27.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.785+0000 7fb429b9a700 1 --2- 172.21.15.27:0/1549841897 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb408067050 0x7fb408069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:26:27.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.836+0000 7fb42c5ff700 1 -- 172.21.15.27:0/1549841897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb4240b4580 con 0x7fb42415aed0 2024-01-11T09:26:27.834 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-11T09:26:27.838+0000 7f27065f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-11T09:26:27.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.844+0000 7fb41a7fc700 1 -- 172.21.15.27:0/1549841897 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v78) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb420028090 con 0x7fb42415aed0 2024-01-11T09:26:27.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.847+0000 7fb42c5ff700 1 -- 172.21.15.27:0/1549841897 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb408067050 msgr2=0x7fb408069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:26:27.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.847+0000 7fb42c5ff700 1 --2- 172.21.15.27:0/1549841897 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb408067050 0x7fb408069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.847+0000 7fb42c5ff700 1 -- 172.21.15.27:0/1549841897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb42415aed0 msgr2=0x7fb424149020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:27.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.847+0000 7fb42c5ff700 1 --2- 172.21.15.27:0/1549841897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb42415aed0 0x7fb424149020 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb42415d230 tx=0x7fb420017000 comp rx=0 tx=0).stop 2024-01-11T09:26:27.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.847+0000 7fb42c5ff700 1 -- 172.21.15.27:0/1549841897 shutdown_connections 2024-01-11T09:26:27.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.847+0000 7fb42c5ff700 1 --2- 172.21.15.27:0/1549841897 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7fb408067050 0x7fb408069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.847+0000 7fb42c5ff700 1 --2- 172.21.15.27:0/1549841897 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb42415cde0 0x7fb42416d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.847+0000 7fb42c5ff700 1 --2- 172.21.15.27:0/1549841897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb42415aed0 0x7fb424149020 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.847+0000 7fb42c5ff700 1 --2- 172.21.15.27:0/1549841897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb42415ca10 0x7fb42414a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:27.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.847+0000 7fb42c5ff700 1 -- 172.21.15.27:0/1549841897 >> 172.21.15.27:0/1549841897 conn(0x7fb42406ed40 msgr2=0x7fb4240b58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:27.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.847+0000 7fb42c5ff700 1 -- 172.21.15.27:0/1549841897 shutdown_connections 2024-01-11T09:26:27.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:27.847+0000 7fb42c5ff700 1 -- 172.21.15.27:0/1549841897 wait complete. 2024-01-11T09:26:27.845 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-11T09:26:27.862 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-11T09:26:27.899 INFO:tasks.ceph.mds.d.smithi077.stdout:starting mds.d at 2024-01-11T09:26:27.963 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:27.966+0000 7f96802e81c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:26:27.964 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:27.968+0000 7f5cd60ea1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:26:28.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.070+0000 7f19c35ff700 1 Processor -- start 2024-01-11T09:26:28.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.070+0000 7f19c35ff700 1 -- start start 2024-01-11T09:26:28.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.071+0000 7f19c35ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19bc143190 0x7f19bc0cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:28.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.072+0000 7f19c35ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19bc0cb4b0 0x7f19bc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:28.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.072+0000 7f19c35ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19bc15d320 con 0x7f19bc142dc0 2024-01-11T09:26:28.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.072+0000 7f19c35ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19bc15df30 con 0x7f19bc0cb4b0 2024-01-11T09:26:28.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.072+0000 7f19c35ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc15d590 con 0x7f19bc143190 2024-01-11T09:26:28.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.072+0000 7f19c1b9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19bc0cb4b0 0x7f19bc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:28.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.072+0000 7f19c0b9a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19bc143190 0x7f19bc0cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:28.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.072+0000 7f19c1b9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19bc0cb4b0 0x7f19bc0cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35258/0 (socket says 172.21.15.27:35258) 2024-01-11T09:26:28.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.072+0000 7f19c139b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f19bc142dc0 0x7f19bc0cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:36734/0 (socket says 172.21.15.27:36734) 2024-01-11T09:26:28.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.072+0000 7f19c1b9c700 1 -- 172.21.15.27:0/2240874124 learned_addr learned my addr 172.21.15.27:0/2240874124 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:28.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.072+0000 7f19c0b9a700 1 -- 172.21.15.27:0/2240874124 >> v2:172.21.15.27:3300/0 conn(0x7f19bc0cb4b0 msgr2=0x7f19bc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:28.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.072+0000 7f19c0b9a700 1 --2- 172.21.15.27:0/2240874124 >> v2:172.21.15.27:3300/0 conn(0x7f19bc0cb4b0 0x7f19bc0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:28.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.072+0000 7f19c0b9a700 1 -- 172.21.15.27:0/2240874124 >> v1:172.21.15.77:6789/0 conn(0x7f19bc142dc0 legacy=0x7f19bc0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:28.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.073+0000 7f19c0b9a700 1 -- 172.21.15.27:0/2240874124 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc15d870 con 0x7f19bc143190 2024-01-11T09:26:28.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.073+0000 7f19c1b9c700 1 --2- 172.21.15.27:0/2240874124 >> v2:172.21.15.27:3300/0 conn(0x7f19bc0cb4b0 0x7f19bc0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:26:28.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.072+0000 7f19b3fff700 1 -- 172.21.15.27:0/2240874124 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794396811 0 0) 0x7f19bc15d320 con 0x7f19bc142dc0 2024-01-11T09:26:28.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.073+0000 7f19c0b9a700 1 --2- 172.21.15.27:0/2240874124 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19bc143190 0x7f19bc0cc020 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f19ac0098e0 tx=0x7f19ac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a52530f0c4cded9b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:28.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.073+0000 7f19b3fff700 1 -- 172.21.15.27:0/2240874124 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac00b640 con 0x7f19bc143190 2024-01-11T09:26:28.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.073+0000 7f19b3fff700 1 -- 172.21.15.27:0/2240874124 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ac00b7a0 con 0x7f19bc143190 2024-01-11T09:26:28.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.073+0000 7f19b3fff700 1 -- 172.21.15.27:0/2240874124 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac00bb40 con 0x7f19bc143190 2024-01-11T09:26:28.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.074+0000 7f19c35ff700 1 -- 172.21.15.27:0/2240874124 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19bc143190 msgr2=0x7f19bc0cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:28.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.074+0000 7f19c35ff700 1 --2- 172.21.15.27:0/2240874124 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19bc143190 0x7f19bc0cc020 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f19ac0098e0 tx=0x7f19ac0091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:28.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.074+0000 7f19c35ff700 1 -- 172.21.15.27:0/2240874124 shutdown_connections 2024-01-11T09:26:28.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.074+0000 7f19c35ff700 1 --2- 172.21.15.27:0/2240874124 >> v2:172.21.15.27:3300/0 conn(0x7f19bc0cb4b0 0x7f19bc0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:28.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.074+0000 7f19c35ff700 1 --2- 172.21.15.27:0/2240874124 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19bc143190 0x7f19bc0cc020 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:28.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.074+0000 7f19c35ff700 1 -- 172.21.15.27:0/2240874124 >> 172.21.15.27:0/2240874124 conn(0x7f19bc063ca0 msgr2=0x7f19bc06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:28.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.074+0000 7f19c35ff700 1 -- 172.21.15.27:0/2240874124 shutdown_connections 2024-01-11T09:26:28.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.074+0000 7f19c35ff700 1 -- 172.21.15.27:0/2240874124 wait complete. 2024-01-11T09:26:28.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.075+0000 7f19c35ff700 1 Processor -- start 2024-01-11T09:26:28.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.075+0000 7f19c35ff700 1 -- start start 2024-01-11T09:26:28.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.076+0000 7f19c35ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19bc0cb4b0 0x7f19bc0c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:28.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.076+0000 7f19c35ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19bc142dc0 0x7f19bc0b1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:28.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.076+0000 7f19c35ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19bc143190 0x7f19bc0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:28.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.076+0000 7f19c35ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc0c9f50 con 0x7f19bc143190 2024-01-11T09:26:28.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.076+0000 7f19c35ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc0c5fe0 con 0x7f19bc0cb4b0 2024-01-11T09:26:28.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.076+0000 7f19c35ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc0ca790 con 0x7f19bc142dc0 2024-01-11T09:26:28.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.077+0000 7f19c1b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19bc143190 0x7f19bc0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:28.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.077+0000 7f19c139b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19bc0cb4b0 0x7f19bc0c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:28.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.077+0000 7f19c1b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19bc143190 0x7f19bc0b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35272/0 (socket says 172.21.15.27:35272) 2024-01-11T09:26:28.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.077+0000 7f19c1b9c700 1 -- 172.21.15.27:0/2045096349 learned_addr learned my addr 172.21.15.27:0/2045096349 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:28.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.077+0000 7f19c0b9a700 1 --2- 172.21.15.27:0/2045096349 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19bc142dc0 0x7f19bc0b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:28.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.077+0000 7f19c1b9c700 1 -- 172.21.15.27:0/2045096349 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19bc142dc0 msgr2=0x7f19bc0b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:28.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.077+0000 7f19c1b9c700 1 --2- 172.21.15.27:0/2045096349 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19bc142dc0 0x7f19bc0b1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:28.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.077+0000 7f19c1b9c700 1 -- 172.21.15.27:0/2045096349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19bc0cb4b0 msgr2=0x7f19bc0c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:28.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.077+0000 7f19c1b9c700 1 --2- 172.21.15.27:0/2045096349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19bc0cb4b0 0x7f19bc0c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:28.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.077+0000 7f19c1b9c700 1 -- 172.21.15.27:0/2045096349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b4009ae0 con 0x7f19bc143190 2024-01-11T09:26:28.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.077+0000 7f19c139b700 1 --2- 172.21.15.27:0/2045096349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19bc0cb4b0 0x7f19bc0c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:26:28.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.077+0000 7f19c1b9c700 1 --2- 172.21.15.27:0/2045096349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19bc143190 0x7f19bc0b1b40 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009fa0 tx=0x7f19b400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:28.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.078+0000 7f19b17fa700 1 -- 172.21.15.27:0/2045096349 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b4018070 con 0x7f19bc143190 2024-01-11T09:26:28.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.078+0000 7f19b17fa700 1 -- 172.21.15.27:0/2045096349 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b400f040 con 0x7f19bc143190 2024-01-11T09:26:28.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.078+0000 7f19c35ff700 1 -- 172.21.15.27:0/2045096349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ac009420 con 0x7f19bc143190 2024-01-11T09:26:28.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.078+0000 7f19b17fa700 1 -- 172.21.15.27:0/2045096349 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b40092e0 con 0x7f19bc143190 2024-01-11T09:26:28.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.078+0000 7f19c35ff700 1 -- 172.21.15.27:0/2045096349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19bc0ca530 con 0x7f19bc143190 2024-01-11T09:26:28.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.079+0000 7f19c35ff700 1 -- 172.21.15.27:0/2045096349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19bc04c3c0 con 0x7f19bc143190 2024-01-11T09:26:28.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.080+0000 7f19b17fa700 1 -- 172.21.15.27:0/2045096349 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 86) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f19b4003020 con 0x7f19bc143190 2024-01-11T09:26:28.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.081+0000 7f19b17fa700 1 --2- 172.21.15.27:0/2045096349 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f19a0067050 0x7f19a0069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:28.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.081+0000 7f19b17fa700 1 -- 172.21.15.27:0/2045096349 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6207+0+0 (secure 0 0 0) 0x7f19b4084910 con 0x7f19bc143190 2024-01-11T09:26:28.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.081+0000 7f19c139b700 -1 --2- 172.21.15.27:0/2045096349 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f19a0067050 0x7f19a0069500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] is using msgr V1 protocol 2024-01-11T09:26:28.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.081+0000 7f19c139b700 1 --2- 172.21.15.27:0/2045096349 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f19a0067050 0x7f19a0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:26:28.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.085+0000 7f19b17fa700 1 -- 172.21.15.27:0/2045096349 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f19b404d5a0 con 0x7f19bc143190 2024-01-11T09:26:28.154 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:28.157+0000 7f5cd60ea1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:26:28.264 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:28.268+0000 7f5cd60ea1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:26:28.267 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:28.271+0000 7f5ccc1fc700 -1 --2- 172.21.15.77:0/41381 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x55f46931a000 0x55f469320000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] is using msgr V1 protocol 2024-01-11T09:26:28.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.275+0000 7f19b17fa700 1 -- 172.21.15.27:0/2045096349 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 87) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f19b4051aa0 con 0x7f19bc143190 2024-01-11T09:26:28.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.276+0000 7f19b17fa700 1 -- 172.21.15.27:0/2045096349 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f19a0067050 msgr2=0x7f19a0069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:26:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.276+0000 7f19b17fa700 1 --2- 172.21.15.27:0/2045096349 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f19a0067050 0x7f19a0069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:28.340 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:28.343+0000 7f5c53f4a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:28.340 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:28.343+0000 7f5c53f4a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:28.343 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:28.343+0000 7f5c53f4a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:28.343 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:28.343+0000 7f5c53f4a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:28.344 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:28.343+0000 7f5c53f4a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:28.344 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:28.343+0000 7f5c50f44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:28.344 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:28.343+0000 7f5c50f44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:28.344 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:28.343+0000 7f5c50f44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:28.344 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:28.343+0000 7f5c50f44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:28.344 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:28.343+0000 7f5c50f44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:28.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.344+0000 7f19c35ff700 1 -- 172.21.15.27:0/2045096349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f19bc15db50 con 0x7f19bc143190 2024-01-11T09:26:28.426 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:28.430+0000 7f96802e81c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:26:28.619 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:28.622+0000 7f96802e81c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:26:28.818 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:28.822+0000 7f96802e81c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:26:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.843+0000 7f19b17fa700 1 -- 172.21.15.27:0/2045096349 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v79) v1 ==== 104+0+0 (secure 0 0 0) 0x7f19b407f970 con 0x7f19bc143190 2024-01-11T09:26:28.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.845+0000 7f19c35ff700 1 -- 172.21.15.27:0/2045096349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19bc143190 msgr2=0x7f19bc0b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:28.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.845+0000 7f19c35ff700 1 --2- 172.21.15.27:0/2045096349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19bc143190 0x7f19bc0b1b40 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009fa0 tx=0x7f19b400b040 comp rx=0 tx=0).stop 2024-01-11T09:26:28.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.845+0000 7f19c35ff700 1 -- 172.21.15.27:0/2045096349 shutdown_connections 2024-01-11T09:26:28.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.845+0000 7f19c35ff700 1 --2- 172.21.15.27:0/2045096349 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x7f19a0067050 0x7f19a0069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:28.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.845+0000 7f19c35ff700 1 --2- 172.21.15.27:0/2045096349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19bc143190 0x7f19bc0b1b40 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:28.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.845+0000 7f19c35ff700 1 --2- 172.21.15.27:0/2045096349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19bc0cb4b0 0x7f19bc0c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:28.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.845+0000 7f19c35ff700 1 --2- 172.21.15.27:0/2045096349 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19bc142dc0 0x7f19bc0b1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:28.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.845+0000 7f19c35ff700 1 -- 172.21.15.27:0/2045096349 >> 172.21.15.27:0/2045096349 conn(0x7f19bc063ca0 msgr2=0x7f19bc13fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:28.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.845+0000 7f19c35ff700 1 -- 172.21.15.27:0/2045096349 shutdown_connections 2024-01-11T09:26:28.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:28.845+0000 7f19c35ff700 1 -- 172.21.15.27:0/2045096349 wait complete. 2024-01-11T09:26:28.860 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-11T09:26:28.938 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:28.942+0000 7f96802e81c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:26:28.972 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:28.945+0000 7f96763fc700 -1 --2- 172.21.15.27:0/41514 >> [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] conn(0x555f6066c000 0x555f60672000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:172.21.15.77:6835/41381,v1:172.21.15.77:6836/41381] is using msgr V1 protocol 2024-01-11T09:26:29.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.072+0000 7f6af1bff700 1 Processor -- start 2024-01-11T09:26:29.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.072+0000 7f6af1bff700 1 -- start start 2024-01-11T09:26:29.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.073+0000 7f6af1bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6aec157440 0x7f6aec0ae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:29.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.073+0000 7f6af1bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec06d420 con 0x7f6aec069d70 2024-01-11T09:26:29.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.073+0000 7f6af1bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec06d700 con 0x7f6aec157810 2024-01-11T09:26:29.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.073+0000 7f6af1bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec06d9e0 con 0x7f6aec157440 2024-01-11T09:26:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.074+0000 7f6aebfff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6aec069d70 0x7f6aec161180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57490/0 (socket says 172.21.15.27:57490) 2024-01-11T09:26:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.074+0000 7f6aebfff700 1 -- 172.21.15.27:0/515991116 learned_addr learned my addr 172.21.15.27:0/515991116 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.074+0000 7f6aeaffd700 1 --1- 172.21.15.27:0/515991116 >> v1:172.21.15.77:6789/0 conn(0x7f6aec157810 0x7f6aec069660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:36748/0 (socket says 172.21.15.27:36748) 2024-01-11T09:26:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.074+0000 7f6aeb7fe700 1 --2- 172.21.15.27:0/515991116 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6aec157440 0x7f6aec0ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.074+0000 7f6aea7fc700 1 -- 172.21.15.27:0/515991116 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999259802 0 0) 0x7f6aec06d700 con 0x7f6aec157810 2024-01-11T09:26:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.074+0000 7f6aea7fc700 1 -- 172.21.15.27:0/515991116 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ad4003570 con 0x7f6aec157810 2024-01-11T09:26:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.074+0000 7f6aea7fc700 1 -- 172.21.15.27:0/515991116 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636398555 0 0) 0x7f6aec06d420 con 0x7f6aec069d70 2024-01-11T09:26:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.074+0000 7f6aeb7fe700 1 -- 172.21.15.27:0/515991116 >> v1:172.21.15.77:6789/0 conn(0x7f6aec157810 legacy=0x7f6aec069660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.074+0000 7f6aeb7fe700 1 -- 172.21.15.27:0/515991116 >> v1:172.21.15.27:6789/0 conn(0x7f6aec069d70 legacy=0x7f6aec161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.074+0000 7f6aeb7fe700 1 -- 172.21.15.27:0/515991116 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec06dcc0 con 0x7f6aec157440 2024-01-11T09:26:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.075+0000 7f6aeb7fe700 1 --2- 172.21.15.27:0/515991116 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6aec157440 0x7f6aec0ae430 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6adc009900 tx=0x7f6adc009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aea06ca691fe2e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.075+0000 7f6aea7fc700 1 -- 172.21.15.27:0/515991116 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc005c10 con 0x7f6aec157440 2024-01-11T09:26:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.075+0000 7f6aea7fc700 1 -- 172.21.15.27:0/515991116 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6adc005d70 con 0x7f6aec157440 2024-01-11T09:26:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.075+0000 7f6aea7fc700 1 -- 172.21.15.27:0/515991116 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc006260 con 0x7f6aec157440 2024-01-11T09:26:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.075+0000 7f6af1bff700 1 -- 172.21.15.27:0/515991116 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6aec157440 msgr2=0x7f6aec0ae430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.075+0000 7f6af1bff700 1 --2- 172.21.15.27:0/515991116 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6aec157440 0x7f6aec0ae430 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6adc009900 tx=0x7f6adc009120 comp rx=0 tx=0).stop 2024-01-11T09:26:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.075+0000 7f6af1bff700 1 -- 172.21.15.27:0/515991116 shutdown_connections 2024-01-11T09:26:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.075+0000 7f6af1bff700 1 --2- 172.21.15.27:0/515991116 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6aec157440 0x7f6aec0ae430 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.075+0000 7f6af1bff700 1 -- 172.21.15.27:0/515991116 >> 172.21.15.27:0/515991116 conn(0x7f6aec0680e0 msgr2=0x7f6aec0cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.076+0000 7f6af1bff700 1 -- 172.21.15.27:0/515991116 shutdown_connections 2024-01-11T09:26:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.076+0000 7f6af1bff700 1 -- 172.21.15.27:0/515991116 wait complete. 2024-01-11T09:26:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.077+0000 7f6af1bff700 1 Processor -- start 2024-01-11T09:26:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.077+0000 7f6af1bff700 1 -- start start 2024-01-11T09:26:29.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.078+0000 7f6af1bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6aec069d70 0x7f6aec0c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:29.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.078+0000 7f6af1bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6aec157440 0x7f6aec14b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:29.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.078+0000 7f6af1bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6aec157810 0x7f6aec16a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.078+0000 7f6af1bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec16db80 con 0x7f6aec069d70 2024-01-11T09:26:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.078+0000 7f6af1bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec16abb0 con 0x7f6aec157810 2024-01-11T09:26:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.078+0000 7f6af1bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec0c55c0 con 0x7f6aec157440 2024-01-11T09:26:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.079+0000 7f6aeb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6aec069d70 0x7f6aec0c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.079+0000 7f6aeb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6aec069d70 0x7f6aec0c4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35280/0 (socket says 172.21.15.27:35280) 2024-01-11T09:26:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.079+0000 7f6aeb7fe700 1 -- 172.21.15.27:0/1978525724 learned_addr learned my addr 172.21.15.27:0/1978525724 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.079+0000 7f6aeaffd700 1 --2- 172.21.15.27:0/1978525724 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6aec157440 0x7f6aec14b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.079+0000 7f6aebfff700 1 --2- 172.21.15.27:0/1978525724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6aec157810 0x7f6aec16a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.079+0000 7f6aeb7fe700 1 -- 172.21.15.27:0/1978525724 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6aec157440 msgr2=0x7f6aec14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.079+0000 7f6aeb7fe700 1 --2- 172.21.15.27:0/1978525724 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6aec157440 0x7f6aec14b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.079+0000 7f6aeb7fe700 1 -- 172.21.15.27:0/1978525724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6aec157810 msgr2=0x7f6aec16a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.079+0000 7f6aeb7fe700 1 --2- 172.21.15.27:0/1978525724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6aec157810 0x7f6aec16a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.079+0000 7f6aeb7fe700 1 -- 172.21.15.27:0/1978525724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6adc0093c0 con 0x7f6aec069d70 2024-01-11T09:26:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.079+0000 7f6aeb7fe700 1 --2- 172.21.15.27:0/1978525724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6aec069d70 0x7f6aec0c4f30 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f6adc009390 tx=0x7f6adc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.080+0000 7f6ad37fe700 1 -- 172.21.15.27:0/1978525724 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc00b7d0 con 0x7f6aec069d70 2024-01-11T09:26:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.080+0000 7f6ad37fe700 1 -- 172.21.15.27:0/1978525724 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6adc00b970 con 0x7f6aec069d70 2024-01-11T09:26:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.080+0000 7f6ad37fe700 1 -- 172.21.15.27:0/1978525724 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc00bd30 con 0x7f6aec069d70 2024-01-11T09:26:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.080+0000 7f6af1bff700 1 -- 172.21.15.27:0/1978525724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aec16d250 con 0x7f6aec069d70 2024-01-11T09:26:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.080+0000 7f6af1bff700 1 -- 172.21.15.27:0/1978525724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aec16d810 con 0x7f6aec069d70 2024-01-11T09:26:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.081+0000 7f6af1bff700 1 -- 172.21.15.27:0/1978525724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aec16de60 con 0x7f6aec069d70 2024-01-11T09:26:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.082+0000 7f6ad37fe700 1 -- 172.21.15.27:0/1978525724 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 87) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f6adc01b690 con 0x7f6aec069d70 2024-01-11T09:26:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.083+0000 7f6ad37fe700 1 -- 172.21.15.27:0/1978525724 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6465+0+0 (secure 0 0 0) 0x7f6adc086600 con 0x7f6aec069d70 2024-01-11T09:26:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.088+0000 7f6ad37fe700 1 -- 172.21.15.27:0/1978525724 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6adc053840 con 0x7f6aec069d70 2024-01-11T09:26:29.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.280+0000 7f6ad37fe700 1 -- 172.21.15.27:0/1978525724 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 88) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f6adc0532f0 con 0x7f6aec069d70 2024-01-11T09:26:29.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.280+0000 7f6ad37fe700 1 --2- 172.21.15.27:0/1978525724 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6ad806c8a0 0x7f6ad806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.281+0000 7f6aeaffd700 1 --2- 172.21.15.27:0/1978525724 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6ad806c8a0 0x7f6ad806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.281+0000 7f6aeaffd700 1 --2- 172.21.15.27:0/1978525724 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6ad806c8a0 0x7f6ad806ec80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6aec0c9c10 tx=0x7f6ae0007dc0 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:29.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:29.297+0000 7f6af1bff700 1 -- 172.21.15.27:0/1978525724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6aec04c3c0 con 0x7f6aec069d70 2024-01-11T09:26:30.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.294+0000 7f6ad37fe700 1 -- 172.21.15.27:0/1978525724 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 89) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f6adc0586e0 con 0x7f6aec069d70 2024-01-11T09:26:30.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.295+0000 7f6ad37fe700 1 -- 172.21.15.27:0/1978525724 <== mon.0 v2:172.21.15.27:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v157) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6adc054850 con 0x7f6aec069d70 2024-01-11T09:26:30.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.300+0000 7f6af1bff700 1 -- 172.21.15.27:0/1978525724 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6ad806c8a0 msgr2=0x7f6ad806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:30.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.300+0000 7f6af1bff700 1 --2- 172.21.15.27:0/1978525724 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6ad806c8a0 0x7f6ad806ec80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6aec0c9c10 tx=0x7f6ae0007dc0 comp rx=0 tx=0).stop 2024-01-11T09:26:30.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.300+0000 7f6af1bff700 1 -- 172.21.15.27:0/1978525724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6aec069d70 msgr2=0x7f6aec0c4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:30.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.300+0000 7f6af1bff700 1 --2- 172.21.15.27:0/1978525724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6aec069d70 0x7f6aec0c4f30 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f6adc009390 tx=0x7f6adc016000 comp rx=0 tx=0).stop 2024-01-11T09:26:30.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.300+0000 7f6af1bff700 1 -- 172.21.15.27:0/1978525724 shutdown_connections 2024-01-11T09:26:30.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.300+0000 7f6af1bff700 1 --2- 172.21.15.27:0/1978525724 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6ad806c8a0 0x7f6ad806ec80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:30.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.300+0000 7f6af1bff700 1 --2- 172.21.15.27:0/1978525724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6aec157810 0x7f6aec16a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:30.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.300+0000 7f6af1bff700 1 --2- 172.21.15.27:0/1978525724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6aec069d70 0x7f6aec0c4f30 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:30.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.300+0000 7f6af1bff700 1 --2- 172.21.15.27:0/1978525724 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6aec157440 0x7f6aec14b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:30.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.300+0000 7f6af1bff700 1 -- 172.21.15.27:0/1978525724 >> 172.21.15.27:0/1978525724 conn(0x7f6aec0680e0 msgr2=0x7f6aec0aef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:30.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.300+0000 7f6af1bff700 1 -- 172.21.15.27:0/1978525724 shutdown_connections 2024-01-11T09:26:30.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.301+0000 7f6af1bff700 1 -- 172.21.15.27:0/1978525724 wait complete. 2024-01-11T09:26:30.299 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-01-11T09:26:30.319 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-11T09:26:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.533+0000 7f5e46bff700 1 Processor -- start 2024-01-11T09:26:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.533+0000 7f5e46bff700 1 -- start start 2024-01-11T09:26:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.534+0000 7f5e46bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5e40159e60 0x7f5e4015aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.534+0000 7f5e46bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5e4015b400 0x7f5e400685e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.534+0000 7f5e46bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e4015c4b0 con 0x7f5e4015a7a0 2024-01-11T09:26:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.534+0000 7f5e46bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e4015d050 con 0x7f5e4015b400 2024-01-11T09:26:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.534+0000 7f5e46bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e4015b8a0 con 0x7f5e40159e60 2024-01-11T09:26:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.534+0000 7f5e4519c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5e4015b400 0x7f5e400685e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.535+0000 7f5e4519c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5e4015b400 0x7f5e400685e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35288/0 (socket says 172.21.15.27:35288) 2024-01-11T09:26:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.535+0000 7f5e4519c700 1 -- 172.21.15.27:0/1255542244 learned_addr learned my addr 172.21.15.27:0/1255542244 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.535+0000 7f5e3ffff700 1 --2- 172.21.15.27:0/1255542244 >> v2:172.21.15.77:3300/0 conn(0x7f5e40159e60 0x7f5e4015aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.535+0000 7f5e4519c700 1 -- 172.21.15.27:0/1255542244 >> v2:172.21.15.77:3300/0 conn(0x7f5e40159e60 msgr2=0x7f5e4015aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:30.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.535+0000 7f5e4519c700 1 --2- 172.21.15.27:0/1255542244 >> v2:172.21.15.77:3300/0 conn(0x7f5e40159e60 0x7f5e4015aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:30.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.535+0000 7f5e4519c700 1 -- 172.21.15.27:0/1255542244 >> v1:172.21.15.77:6789/0 conn(0x7f5e4015a7a0 legacy=0x7f5e4015c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:30.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.535+0000 7f5e4519c700 1 -- 172.21.15.27:0/1255542244 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4015bb80 con 0x7f5e4015b400 2024-01-11T09:26:30.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.535+0000 7f5e3ffff700 1 --2- 172.21.15.27:0/1255542244 >> v2:172.21.15.77:3300/0 conn(0x7f5e40159e60 0x7f5e4015aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:30.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.535+0000 7f5e4519c700 1 --2- 172.21.15.27:0/1255542244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e4015b400 0x7f5e400685e0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f5e400ce530 tx=0x7f5e38009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ba4204bd9fa94c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:30.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.535+0000 7f5e3f7fe700 1 -- 172.21.15.27:0/1255542244 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3800e070 con 0x7f5e4015b400 2024-01-11T09:26:30.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.536+0000 7f5e3f7fe700 1 -- 172.21.15.27:0/1255542244 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3800ba50 con 0x7f5e4015b400 2024-01-11T09:26:30.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.536+0000 7f5e3f7fe700 1 -- 172.21.15.27:0/1255542244 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e380092e0 con 0x7f5e4015b400 2024-01-11T09:26:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.536+0000 7f5e46bff700 1 -- 172.21.15.27:0/1255542244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e4015b400 msgr2=0x7f5e400685e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.536+0000 7f5e46bff700 1 --2- 172.21.15.27:0/1255542244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e4015b400 0x7f5e400685e0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f5e400ce530 tx=0x7f5e38009840 comp rx=0 tx=0).stop 2024-01-11T09:26:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.536+0000 7f5e46bff700 1 -- 172.21.15.27:0/1255542244 shutdown_connections 2024-01-11T09:26:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.536+0000 7f5e46bff700 1 --2- 172.21.15.27:0/1255542244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e4015b400 0x7f5e400685e0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.536+0000 7f5e46bff700 1 --2- 172.21.15.27:0/1255542244 >> v2:172.21.15.77:3300/0 conn(0x7f5e40159e60 0x7f5e4015aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.536+0000 7f5e46bff700 1 -- 172.21.15.27:0/1255542244 >> 172.21.15.27:0/1255542244 conn(0x7f5e4006ed40 msgr2=0x7f5e4006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.536+0000 7f5e46bff700 1 -- 172.21.15.27:0/1255542244 shutdown_connections 2024-01-11T09:26:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.536+0000 7f5e46bff700 1 -- 172.21.15.27:0/1255542244 wait complete. 2024-01-11T09:26:30.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.538+0000 7f5e46bff700 1 Processor -- start 2024-01-11T09:26:30.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.539+0000 7f5e46bff700 1 -- start start 2024-01-11T09:26:30.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.540+0000 7f5e46bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e4015a7a0 0x7f5e4014b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:30.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.540+0000 7f5e46bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e4015b400 0x7f5e401585f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:30.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.540+0000 7f5e46bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e400b1350 0x7f5e40159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:30.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.540+0000 7f5e46bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e401698c0 con 0x7f5e400b1350 2024-01-11T09:26:30.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.540+0000 7f5e46bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4016a720 con 0x7f5e4015a7a0 2024-01-11T09:26:30.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.540+0000 7f5e46bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4014ba50 con 0x7f5e4015b400 2024-01-11T09:26:30.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.540+0000 7f5e4519c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e400b1350 0x7f5e40159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:30.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.540+0000 7f5e4519c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e400b1350 0x7f5e40159080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35304/0 (socket says 172.21.15.27:35304) 2024-01-11T09:26:30.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.540+0000 7f5e4519c700 1 -- 172.21.15.27:0/1112091095 learned_addr learned my addr 172.21.15.27:0/1112091095 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:30.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.540+0000 7f5e3ffff700 1 --2- 172.21.15.27:0/1112091095 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e4015b400 0x7f5e401585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:30.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.540+0000 7f5e4499b700 1 --2- 172.21.15.27:0/1112091095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e4015a7a0 0x7f5e4014b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.541+0000 7f5e4519c700 1 -- 172.21.15.27:0/1112091095 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e4015b400 msgr2=0x7f5e401585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.541+0000 7f5e4519c700 1 --2- 172.21.15.27:0/1112091095 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e4015b400 0x7f5e401585f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.541+0000 7f5e4519c700 1 -- 172.21.15.27:0/1112091095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e4015a7a0 msgr2=0x7f5e4014b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.541+0000 7f5e4519c700 1 --2- 172.21.15.27:0/1112091095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e4015a7a0 0x7f5e4014b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.541+0000 7f5e4519c700 1 -- 172.21.15.27:0/1112091095 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e38009ae0 con 0x7f5e400b1350 2024-01-11T09:26:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.541+0000 7f5e4519c700 1 --2- 172.21.15.27:0/1112091095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e400b1350 0x7f5e40159080 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f5e40088e90 tx=0x7f5e38018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.541+0000 7f5e3cff9700 1 -- 172.21.15.27:0/1112091095 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3800be60 con 0x7f5e400b1350 2024-01-11T09:26:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.541+0000 7f5e3cff9700 1 -- 172.21.15.27:0/1112091095 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e38008ce0 con 0x7f5e400b1350 2024-01-11T09:26:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.541+0000 7f5e46bff700 1 -- 172.21.15.27:0/1112091095 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e40169b10 con 0x7f5e400b1350 2024-01-11T09:26:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.541+0000 7f5e3cff9700 1 -- 172.21.15.27:0/1112091095 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e380266b0 con 0x7f5e400b1350 2024-01-11T09:26:30.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.542+0000 7f5e46bff700 1 -- 172.21.15.27:0/1112091095 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e4016a0d0 con 0x7f5e400b1350 2024-01-11T09:26:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.542+0000 7f5e46bff700 1 -- 172.21.15.27:0/1112091095 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e4016a8d0 con 0x7f5e400b1350 2024-01-11T09:26:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.544+0000 7f5e3cff9700 1 -- 172.21.15.27:0/1112091095 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 89) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f5e38003040 con 0x7f5e400b1350 2024-01-11T09:26:30.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.545+0000 7f5e3cff9700 1 --2- 172.21.15.27:0/1112091095 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5e24066ea0 0x7f5e24069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:30.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.545+0000 7f5e3cff9700 1 -- 172.21.15.27:0/1112091095 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6027+0+0 (secure 0 0 0) 0x7f5e380864e0 con 0x7f5e400b1350 2024-01-11T09:26:30.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.547+0000 7f5e4499b700 1 --2- 172.21.15.27:0/1112091095 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5e24066ea0 0x7f5e24069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:30.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.548+0000 7f5e4499b700 1 --2- 172.21.15.27:0/1112091095 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5e24066ea0 0x7f5e24069350 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5e400b6150 tx=0x7f5e34009040 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:30.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.553+0000 7f5e3cff9700 1 -- 172.21.15.27:0/1112091095 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5e38053650 con 0x7f5e400b1350 2024-01-11T09:26:30.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:30.763+0000 7f5e46bff700 1 -- 172.21.15.27:0/1112091095 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5e4004c3c0 con 0x7f5e400b1350 2024-01-11T09:26:31.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.629+0000 7f5e3cff9700 1 -- 172.21.15.27:0/1112091095 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v159) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5e38056c70 con 0x7f5e400b1350 2024-01-11T09:26:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.635+0000 7f5e46bff700 1 -- 172.21.15.27:0/1112091095 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5e24066ea0 msgr2=0x7f5e24069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.635+0000 7f5e46bff700 1 --2- 172.21.15.27:0/1112091095 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5e24066ea0 0x7f5e24069350 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5e400b6150 tx=0x7f5e34009040 comp rx=0 tx=0).stop 2024-01-11T09:26:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.635+0000 7f5e46bff700 1 -- 172.21.15.27:0/1112091095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e400b1350 msgr2=0x7f5e40159080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.635+0000 7f5e46bff700 1 --2- 172.21.15.27:0/1112091095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e400b1350 0x7f5e40159080 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f5e40088e90 tx=0x7f5e38018000 comp rx=0 tx=0).stop 2024-01-11T09:26:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.635+0000 7f5e46bff700 1 -- 172.21.15.27:0/1112091095 shutdown_connections 2024-01-11T09:26:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.635+0000 7f5e46bff700 1 --2- 172.21.15.27:0/1112091095 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5e24066ea0 0x7f5e24069350 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.635+0000 7f5e46bff700 1 --2- 172.21.15.27:0/1112091095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e400b1350 0x7f5e40159080 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.635+0000 7f5e46bff700 1 --2- 172.21.15.27:0/1112091095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e4015a7a0 0x7f5e4014b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.635+0000 7f5e46bff700 1 --2- 172.21.15.27:0/1112091095 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e4015b400 0x7f5e401585f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.635+0000 7f5e46bff700 1 -- 172.21.15.27:0/1112091095 >> 172.21.15.27:0/1112091095 conn(0x7f5e4006ed40 msgr2=0x7f5e400b4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.636+0000 7f5e46bff700 1 -- 172.21.15.27:0/1112091095 shutdown_connections 2024-01-11T09:26:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.636+0000 7f5e46bff700 1 -- 172.21.15.27:0/1112091095 wait complete. 2024-01-11T09:26:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-01-11T09:26:31.654 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-11T09:26:31.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.863+0000 7f8dde5ff700 1 Processor -- start 2024-01-11T09:26:31.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.864+0000 7f8dde5ff700 1 -- start start 2024-01-11T09:26:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.865+0000 7f8dde5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8dd8069540 0x7f8dd8162270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.865+0000 7f8dde5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd80699d0 con 0x7f8dd8157410 2024-01-11T09:26:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.865+0000 7f8dde5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd8069cb0 con 0x7f8dd81577e0 2024-01-11T09:26:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.865+0000 7f8dde5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd8069f90 con 0x7f8dd8069540 2024-01-11T09:26:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.865+0000 7f8ddcb9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8dd8069540 0x7f8dd8162270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.865+0000 7f8dd7fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8dd8157410 0x7f8dd80ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57504/0 (socket says 172.21.15.27:57504) 2024-01-11T09:26:31.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.865+0000 7f8dd7fff700 1 -- 172.21.15.27:0/1597072560 learned_addr learned my addr 172.21.15.27:0/1597072560 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:31.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.866+0000 7f8ddcb9c700 1 -- 172.21.15.27:0/1597072560 >> v1:172.21.15.77:6789/0 conn(0x7f8dd81577e0 legacy=0x7f8dd80ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:31.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.866+0000 7f8ddcb9c700 1 -- 172.21.15.27:0/1597072560 >> v1:172.21.15.27:6789/0 conn(0x7f8dd8157410 legacy=0x7f8dd80ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:31.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.866+0000 7f8ddcb9c700 1 -- 172.21.15.27:0/1597072560 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd806a270 con 0x7f8dd8069540 2024-01-11T09:26:31.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.866+0000 7f8dd6ffd700 1 -- 172.21.15.27:0/1597072560 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554365359 0 0) 0x7f8dd80699d0 con 0x7f8dd8157410 2024-01-11T09:26:31.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.866+0000 7f8ddcb9c700 1 --2- 172.21.15.27:0/1597072560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd8069540 0x7f8dd8162270 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0009e60 tx=0x7f8dd0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36143283f7b0e0ed server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:31.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.866+0000 7f8dd6ffd700 1 -- 172.21.15.27:0/1597072560 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd000e070 con 0x7f8dd8069540 2024-01-11T09:26:31.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.866+0000 7f8dd6ffd700 1 -- 172.21.15.27:0/1597072560 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd000bbf0 con 0x7f8dd8069540 2024-01-11T09:26:31.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.866+0000 7f8dd6ffd700 1 -- 172.21.15.27:0/1597072560 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd0012430 con 0x7f8dd8069540 2024-01-11T09:26:31.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.867+0000 7f8dde5ff700 1 -- 172.21.15.27:0/1597072560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd8069540 msgr2=0x7f8dd8162270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:31.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.867+0000 7f8dde5ff700 1 --2- 172.21.15.27:0/1597072560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd8069540 0x7f8dd8162270 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0009e60 tx=0x7f8dd0009780 comp rx=0 tx=0).stop 2024-01-11T09:26:31.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.867+0000 7f8dde5ff700 1 -- 172.21.15.27:0/1597072560 shutdown_connections 2024-01-11T09:26:31.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.867+0000 7f8dde5ff700 1 --2- 172.21.15.27:0/1597072560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd8069540 0x7f8dd8162270 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:31.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.867+0000 7f8dde5ff700 1 -- 172.21.15.27:0/1597072560 >> 172.21.15.27:0/1597072560 conn(0x7f8dd80680a0 msgr2=0x7f8dd80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:31.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.867+0000 7f8dde5ff700 1 -- 172.21.15.27:0/1597072560 shutdown_connections 2024-01-11T09:26:31.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.867+0000 7f8dde5ff700 1 -- 172.21.15.27:0/1597072560 wait complete. 2024-01-11T09:26:31.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.868+0000 7f8dde5ff700 1 Processor -- start 2024-01-11T09:26:31.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.868+0000 7f8dde5ff700 1 -- start start 2024-01-11T09:26:31.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.869+0000 7f8dde5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8dd8069540 0x7f8dd80b1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:31.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.869+0000 7f8dde5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd8157410 0x7f8dd8158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:31.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.869+0000 7f8dde5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dd81577e0 0x7f8dd8159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:31.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.869+0000 7f8dde5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd80b1f70 con 0x7f8dd8157410 2024-01-11T09:26:31.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.869+0000 7f8dde5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd80aff30 con 0x7f8dd81577e0 2024-01-11T09:26:31.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.869+0000 7f8dde5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd80b0210 con 0x7f8dd8069540 2024-01-11T09:26:31.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.870+0000 7f8dd77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd8157410 0x7f8dd8158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:31.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.870+0000 7f8dd77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd8157410 0x7f8dd8158cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35326/0 (socket says 172.21.15.27:35326) 2024-01-11T09:26:31.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.870+0000 7f8dd77fe700 1 -- 172.21.15.27:0/2149267295 learned_addr learned my addr 172.21.15.27:0/2149267295 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:31.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.870+0000 7f8dd7fff700 1 --2- 172.21.15.27:0/2149267295 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8dd8069540 0x7f8dd80b1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:31.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.870+0000 7f8ddcb9c700 1 --2- 172.21.15.27:0/2149267295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dd81577e0 0x7f8dd8159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:31.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.870+0000 7f8dd77fe700 1 -- 172.21.15.27:0/2149267295 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8dd8069540 msgr2=0x7f8dd80b1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:31.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.870+0000 7f8dd77fe700 1 --2- 172.21.15.27:0/2149267295 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8dd8069540 0x7f8dd80b1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:31.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.870+0000 7f8dd77fe700 1 -- 172.21.15.27:0/2149267295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dd81577e0 msgr2=0x7f8dd8159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:31.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.870+0000 7f8dd77fe700 1 --2- 172.21.15.27:0/2149267295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dd81577e0 0x7f8dd8159230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:31.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.870+0000 7f8dd77fe700 1 -- 172.21.15.27:0/2149267295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd0009a20 con 0x7f8dd8157410 2024-01-11T09:26:31.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.870+0000 7f8dd77fe700 1 --2- 172.21.15.27:0/2149267295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd8157410 0x7f8dd8158cf0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc00b5e0 tx=0x7f8dcc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:31.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.871+0000 7f8dbffff700 1 -- 172.21.15.27:0/2149267295 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc00f510 con 0x7f8dd8157410 2024-01-11T09:26:31.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.871+0000 7f8dbffff700 1 -- 172.21.15.27:0/2149267295 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dcc00f6b0 con 0x7f8dd8157410 2024-01-11T09:26:31.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.871+0000 7f8dbffff700 1 -- 172.21.15.27:0/2149267295 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc00fa70 con 0x7f8dd8157410 2024-01-11T09:26:31.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.871+0000 7f8dde5ff700 1 -- 172.21.15.27:0/2149267295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd80b04f0 con 0x7f8dd8157410 2024-01-11T09:26:31.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.871+0000 7f8dde5ff700 1 -- 172.21.15.27:0/2149267295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd814b630 con 0x7f8dd8157410 2024-01-11T09:26:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.872+0000 7f8dde5ff700 1 -- 172.21.15.27:0/2149267295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd814b850 con 0x7f8dd8157410 2024-01-11T09:26:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.873+0000 7f8dbffff700 1 -- 172.21.15.27:0/2149267295 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 89) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f8dcc01a400 con 0x7f8dd8157410 2024-01-11T09:26:31.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.874+0000 7f8dbffff700 1 --2- 172.21.15.27:0/2149267295 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8dc4066ed0 0x7f8dc4069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:31.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.875+0000 7f8dbffff700 1 -- 172.21.15.27:0/2149267295 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5511+0+0 (secure 0 0 0) 0x7f8dcc083cd0 con 0x7f8dd8157410 2024-01-11T09:26:31.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.875+0000 7f8dd7fff700 1 --2- 172.21.15.27:0/2149267295 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8dc4066ed0 0x7f8dc4069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:31.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.876+0000 7f8dd7fff700 1 --2- 172.21.15.27:0/2149267295 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8dc4066ed0 0x7f8dc4069380 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8001360 tx=0x7f8dc8009040 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:31.879+0000 7f8dbffff700 1 -- 172.21.15.27:0/2149267295 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8dcc051160 con 0x7f8dd8157410 2024-01-11T09:26:32.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.081+0000 7f8dde5ff700 1 -- 172.21.15.27:0/2149267295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f8dd804c3c0 con 0x7f8dd8157410 2024-01-11T09:26:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.085+0000 7f8dbffff700 1 -- 172.21.15.27:0/2149267295 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v55) v1 ==== 120+0+0 (secure 0 0 0) 0x7f8dcc0091f0 con 0x7f8dd8157410 2024-01-11T09:26:32.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.088+0000 7f8dde5ff700 1 -- 172.21.15.27:0/2149267295 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8dc4066ed0 msgr2=0x7f8dc4069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.088+0000 7f8dde5ff700 1 --2- 172.21.15.27:0/2149267295 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8dc4066ed0 0x7f8dc4069380 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8001360 tx=0x7f8dc8009040 comp rx=0 tx=0).stop 2024-01-11T09:26:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.088+0000 7f8dde5ff700 1 -- 172.21.15.27:0/2149267295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd8157410 msgr2=0x7f8dd8158cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.088+0000 7f8dde5ff700 1 --2- 172.21.15.27:0/2149267295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd8157410 0x7f8dd8158cf0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc00b5e0 tx=0x7f8dcc0090f0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.088+0000 7f8dde5ff700 1 -- 172.21.15.27:0/2149267295 shutdown_connections 2024-01-11T09:26:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.088+0000 7f8dde5ff700 1 --2- 172.21.15.27:0/2149267295 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8dc4066ed0 0x7f8dc4069380 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.088+0000 7f8dde5ff700 1 --2- 172.21.15.27:0/2149267295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dd81577e0 0x7f8dd8159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.088+0000 7f8dde5ff700 1 --2- 172.21.15.27:0/2149267295 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8dd8069540 0x7f8dd80b1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.088+0000 7f8dde5ff700 1 --2- 172.21.15.27:0/2149267295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd8157410 0x7f8dd8158cf0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.088+0000 7f8dde5ff700 1 -- 172.21.15.27:0/2149267295 >> 172.21.15.27:0/2149267295 conn(0x7f8dd80680a0 msgr2=0x7f8dd80c1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.088+0000 7f8dde5ff700 1 -- 172.21.15.27:0/2149267295 shutdown_connections 2024-01-11T09:26:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.088+0000 7f8dde5ff700 1 -- 172.21.15.27:0/2149267295 wait complete. 2024-01-11T09:26:32.102 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-11T09:26:32.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.310+0000 7fa5bdbff700 1 Processor -- start 2024-01-11T09:26:32.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.311+0000 7fa5bdbff700 1 -- start start 2024-01-11T09:26:32.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.312+0000 7fa5bdbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa5b81596f0 0x7fa5b80a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:32.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.312+0000 7fa5bdbff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5b80a9de0 0x7fa5b8068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.312+0000 7fa5bdbff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b80c80a0 con 0x7fa5b8159ac0 2024-01-11T09:26:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.312+0000 7fa5bdbff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b80cbac0 con 0x7fa5b80a9de0 2024-01-11T09:26:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.312+0000 7fa5bdbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b80c7490 con 0x7fa5b81596f0 2024-01-11T09:26:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.312+0000 7fa5b6ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5b80a9de0 0x7fa5b8068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.312+0000 7fa5b6ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5b80a9de0 0x7fa5b8068620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35336/0 (socket says 172.21.15.27:35336) 2024-01-11T09:26:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.312+0000 7fa5b6ffd700 1 -- 172.21.15.27:0/144402226 learned_addr learned my addr 172.21.15.27:0/144402226 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.312+0000 7fa5b7fff700 1 --1- 172.21.15.27:0/144402226 >> v1:172.21.15.27:6789/0 conn(0x7fa5b8159ac0 0x7fa5b80c8530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57512/0 (socket says 172.21.15.27:57512) 2024-01-11T09:26:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.313+0000 7fa5b77fe700 1 --2- 172.21.15.27:0/144402226 >> v2:172.21.15.77:3300/0 conn(0x7fa5b81596f0 0x7fa5b80a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.313+0000 7fa5b6ffd700 1 -- 172.21.15.27:0/144402226 >> v2:172.21.15.77:3300/0 conn(0x7fa5b81596f0 msgr2=0x7fa5b80a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.313+0000 7fa5b6ffd700 1 --2- 172.21.15.27:0/144402226 >> v2:172.21.15.77:3300/0 conn(0x7fa5b81596f0 0x7fa5b80a98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.313+0000 7fa5b6ffd700 1 -- 172.21.15.27:0/144402226 >> v1:172.21.15.27:6789/0 conn(0x7fa5b8159ac0 legacy=0x7fa5b80c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.313+0000 7fa5b67fc700 1 -- 172.21.15.27:0/144402226 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47545647 0 0) 0x7fa5b80c80a0 con 0x7fa5b8159ac0 2024-01-11T09:26:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.313+0000 7fa5b6ffd700 1 -- 172.21.15.27:0/144402226 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b80c7770 con 0x7fa5b80a9de0 2024-01-11T09:26:32.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.313+0000 7fa5b77fe700 1 --2- 172.21.15.27:0/144402226 >> v2:172.21.15.77:3300/0 conn(0x7fa5b81596f0 0x7fa5b80a98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:32.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.313+0000 7fa5b6ffd700 1 --2- 172.21.15.27:0/144402226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b80a9de0 0x7fa5b8068620 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fa5b80cc9b0 tx=0x7fa5ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82538da86da63bc9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.313+0000 7fa5b67fc700 1 -- 172.21.15.27:0/144402226 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac00b860 con 0x7fa5b80a9de0 2024-01-11T09:26:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.313+0000 7fa5b67fc700 1 -- 172.21.15.27:0/144402226 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ac00b9c0 con 0x7fa5b80a9de0 2024-01-11T09:26:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.313+0000 7fa5b67fc700 1 -- 172.21.15.27:0/144402226 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac00bd60 con 0x7fa5b80a9de0 2024-01-11T09:26:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.314+0000 7fa5bdbff700 1 -- 172.21.15.27:0/144402226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b80a9de0 msgr2=0x7fa5b8068620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.314+0000 7fa5bdbff700 1 --2- 172.21.15.27:0/144402226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b80a9de0 0x7fa5b8068620 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fa5b80cc9b0 tx=0x7fa5ac009250 comp rx=0 tx=0).stop 2024-01-11T09:26:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.314+0000 7fa5bdbff700 1 -- 172.21.15.27:0/144402226 shutdown_connections 2024-01-11T09:26:32.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.314+0000 7fa5bdbff700 1 --2- 172.21.15.27:0/144402226 >> v2:172.21.15.77:3300/0 conn(0x7fa5b81596f0 0x7fa5b80a98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.314+0000 7fa5bdbff700 1 --2- 172.21.15.27:0/144402226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b80a9de0 0x7fa5b8068620 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.314+0000 7fa5bdbff700 1 -- 172.21.15.27:0/144402226 >> 172.21.15.27:0/144402226 conn(0x7fa5b80c9fb0 msgr2=0x7fa5b806ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.314+0000 7fa5bdbff700 1 -- 172.21.15.27:0/144402226 shutdown_connections 2024-01-11T09:26:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.314+0000 7fa5bdbff700 1 -- 172.21.15.27:0/144402226 wait complete. 2024-01-11T09:26:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.316+0000 7fa5bdbff700 1 Processor -- start 2024-01-11T09:26:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.316+0000 7fa5bdbff700 1 -- start start 2024-01-11T09:26:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.317+0000 7fa5bdbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5b80a9de0 0x7fa5b80c4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.317+0000 7fa5bdbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5b81596f0 0x7fa5b80cae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.317+0000 7fa5bdbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b8159ac0 0x7fa5b8169820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.317+0000 7fa5bdbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b80c4c40 con 0x7fa5b8159ac0 2024-01-11T09:26:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.317+0000 7fa5bdbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b80add20 con 0x7fa5b80a9de0 2024-01-11T09:26:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.317+0000 7fa5bdbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b80ad110 con 0x7fa5b81596f0 2024-01-11T09:26:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.317+0000 7fa5b7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b8159ac0 0x7fa5b8169820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.317+0000 7fa5b7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b8159ac0 0x7fa5b8169820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35342/0 (socket says 172.21.15.27:35342) 2024-01-11T09:26:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.317+0000 7fa5b7fff700 1 -- 172.21.15.27:0/4227918719 learned_addr learned my addr 172.21.15.27:0/4227918719 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.317+0000 7fa5b77fe700 1 --2- 172.21.15.27:0/4227918719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5b80a9de0 0x7fa5b80c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.317+0000 7fa5b7fff700 1 -- 172.21.15.27:0/4227918719 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5b81596f0 msgr2=0x7fa5b80cae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.317+0000 7fa5b7fff700 1 --2- 172.21.15.27:0/4227918719 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5b81596f0 0x7fa5b80cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.317+0000 7fa5b7fff700 1 -- 172.21.15.27:0/4227918719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5b80a9de0 msgr2=0x7fa5b80c4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.317+0000 7fa5b7fff700 1 --2- 172.21.15.27:0/4227918719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5b80a9de0 0x7fa5b80c4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.317+0000 7fa5b7fff700 1 -- 172.21.15.27:0/4227918719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ac0094f0 con 0x7fa5b8159ac0 2024-01-11T09:26:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.317+0000 7fa5b77fe700 1 --2- 172.21.15.27:0/4227918719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5b80a9de0 0x7fa5b80c4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:26:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.318+0000 7fa5b7fff700 1 --2- 172.21.15.27:0/4227918719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b8159ac0 0x7fa5b8169820 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000ccc0 tx=0x7fa5b0001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.318+0000 7fa59f7fe700 1 -- 172.21.15.27:0/4227918719 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b000ce00 con 0x7fa5b8159ac0 2024-01-11T09:26:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.318+0000 7fa59f7fe700 1 -- 172.21.15.27:0/4227918719 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b0005df0 con 0x7fa5b8159ac0 2024-01-11T09:26:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.318+0000 7fa59f7fe700 1 -- 172.21.15.27:0/4227918719 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b000abb0 con 0x7fa5b8159ac0 2024-01-11T09:26:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.318+0000 7fa5bdbff700 1 -- 172.21.15.27:0/4227918719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b80ad3f0 con 0x7fa5b8159ac0 2024-01-11T09:26:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.318+0000 7fa5bdbff700 1 -- 172.21.15.27:0/4227918719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b80ada30 con 0x7fa5b8159ac0 2024-01-11T09:26:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.319+0000 7fa5bdbff700 1 -- 172.21.15.27:0/4227918719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b80adeb0 con 0x7fa5b8159ac0 2024-01-11T09:26:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.320+0000 7fa59f7fe700 1 -- 172.21.15.27:0/4227918719 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 89) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fa5b00070f0 con 0x7fa5b8159ac0 2024-01-11T09:26:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.321+0000 7fa59f7fe700 1 --2- 172.21.15.27:0/4227918719 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa5a4066ea0 0x7fa5a4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.321+0000 7fa59f7fe700 1 -- 172.21.15.27:0/4227918719 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5511+0+0 (secure 0 0 0) 0x7fa5b0084530 con 0x7fa5b8159ac0 2024-01-11T09:26:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.321+0000 7fa5b77fe700 1 --2- 172.21.15.27:0/4227918719 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa5a4066ea0 0x7fa5a4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.322+0000 7fa5b77fe700 1 --2- 172.21.15.27:0/4227918719 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa5a4066ea0 0x7fa5a4069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8006f00 tx=0x7fa5a8008040 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:32.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.325+0000 7fa59f7fe700 1 -- 172.21.15.27:0/4227918719 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa5b0051920 con 0x7fa5b8159ac0 2024-01-11T09:26:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.528+0000 7fa5bdbff700 1 -- 172.21.15.27:0/4227918719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fa5b804c3c0 con 0x7fa5b8159ac0 2024-01-11T09:26:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.533+0000 7fa59f7fe700 1 -- 172.21.15.27:0/4227918719 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v56) v1 ==== 115+0+0 (secure 0 0 0) 0x7fa5b0054f40 con 0x7fa5b8159ac0 2024-01-11T09:26:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.535+0000 7fa5bdbff700 1 -- 172.21.15.27:0/4227918719 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa5a4066ea0 msgr2=0x7fa5a4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.535+0000 7fa5bdbff700 1 --2- 172.21.15.27:0/4227918719 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa5a4066ea0 0x7fa5a4069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8006f00 tx=0x7fa5a8008040 comp rx=0 tx=0).stop 2024-01-11T09:26:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.535+0000 7fa5bdbff700 1 -- 172.21.15.27:0/4227918719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b8159ac0 msgr2=0x7fa5b8169820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.535+0000 7fa5bdbff700 1 --2- 172.21.15.27:0/4227918719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b8159ac0 0x7fa5b8169820 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000ccc0 tx=0x7fa5b0001a70 comp rx=0 tx=0).stop 2024-01-11T09:26:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.535+0000 7fa5bdbff700 1 -- 172.21.15.27:0/4227918719 shutdown_connections 2024-01-11T09:26:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.535+0000 7fa5bdbff700 1 --2- 172.21.15.27:0/4227918719 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa5a4066ea0 0x7fa5a4069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.535+0000 7fa5bdbff700 1 --2- 172.21.15.27:0/4227918719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b8159ac0 0x7fa5b8169820 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.535+0000 7fa5bdbff700 1 --2- 172.21.15.27:0/4227918719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5b80a9de0 0x7fa5b80c4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.535+0000 7fa5bdbff700 1 --2- 172.21.15.27:0/4227918719 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5b81596f0 0x7fa5b80cae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.535+0000 7fa5bdbff700 1 -- 172.21.15.27:0/4227918719 >> 172.21.15.27:0/4227918719 conn(0x7fa5b80c9fb0 msgr2=0x7fa5b80c2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:32.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.535+0000 7fa5bdbff700 1 -- 172.21.15.27:0/4227918719 shutdown_connections 2024-01-11T09:26:32.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.535+0000 7fa5bdbff700 1 -- 172.21.15.27:0/4227918719 wait complete. 2024-01-11T09:26:32.549 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-11T09:26:32.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.753+0000 7f122efff700 1 Processor -- start 2024-01-11T09:26:32.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.754+0000 7f122efff700 1 -- start start 2024-01-11T09:26:32.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.755+0000 7f122efff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1228157410 0x7f12280ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:32.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.755+0000 7f122efff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12281577e0 0x7f12280ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:32.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.755+0000 7f122efff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12280699d0 con 0x7f1228069540 2024-01-11T09:26:32.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.755+0000 7f122efff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1228069cb0 con 0x7f1228157410 2024-01-11T09:26:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.755+0000 7f122efff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1228069f90 con 0x7f12281577e0 2024-01-11T09:26:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.755+0000 7f122cd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1228157410 0x7f12280ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.755+0000 7f122cd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1228157410 0x7f12280ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35346/0 (socket says 172.21.15.27:35346) 2024-01-11T09:26:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.755+0000 7f122cd9b700 1 -- 172.21.15.27:0/3083304589 learned_addr learned my addr 172.21.15.27:0/3083304589 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.756+0000 7f1227fff700 1 --2- 172.21.15.27:0/3083304589 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12281577e0 0x7f12280ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.756+0000 7f122cd9b700 1 -- 172.21.15.27:0/3083304589 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12281577e0 msgr2=0x7f12280ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.756+0000 7f122cd9b700 1 --2- 172.21.15.27:0/3083304589 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12281577e0 0x7f12280ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.756+0000 7f122cd9b700 1 -- 172.21.15.27:0/3083304589 >> v1:172.21.15.77:6789/0 conn(0x7f1228069540 legacy=0x7f122815b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.756+0000 7f122cd9b700 1 -- 172.21.15.27:0/3083304589 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122806a270 con 0x7f1228157410 2024-01-11T09:26:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.756+0000 7f122cd9b700 1 --2- 172.21.15.27:0/3083304589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1228157410 0x7f12280ccae0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f121c0099a0 tx=0x7f121c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2355f3dab2a1f434 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:32.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.756+0000 7f12277fe700 1 -- 172.21.15.27:0/3083304589 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c00b860 con 0x7f1228157410 2024-01-11T09:26:32.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.756+0000 7f12277fe700 1 -- 172.21.15.27:0/3083304589 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121c00b9c0 con 0x7f1228157410 2024-01-11T09:26:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.756+0000 7f12277fe700 1 -- 172.21.15.27:0/3083304589 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c00bd80 con 0x7f1228157410 2024-01-11T09:26:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.757+0000 7f122efff700 1 -- 172.21.15.27:0/3083304589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1228157410 msgr2=0x7f12280ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.757+0000 7f122efff700 1 --2- 172.21.15.27:0/3083304589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1228157410 0x7f12280ccae0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f121c0099a0 tx=0x7f121c009250 comp rx=0 tx=0).stop 2024-01-11T09:26:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.757+0000 7f122efff700 1 -- 172.21.15.27:0/3083304589 shutdown_connections 2024-01-11T09:26:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.757+0000 7f122efff700 1 --2- 172.21.15.27:0/3083304589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1228157410 0x7f12280ccae0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.757+0000 7f122efff700 1 --2- 172.21.15.27:0/3083304589 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12281577e0 0x7f12280ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.757+0000 7f122efff700 1 -- 172.21.15.27:0/3083304589 >> 172.21.15.27:0/3083304589 conn(0x7f12280680a0 msgr2=0x7f12280cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.757+0000 7f122efff700 1 -- 172.21.15.27:0/3083304589 shutdown_connections 2024-01-11T09:26:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.757+0000 7f122efff700 1 -- 172.21.15.27:0/3083304589 wait complete. 2024-01-11T09:26:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.759+0000 7f122efff700 1 Processor -- start 2024-01-11T09:26:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.759+0000 7f122efff700 1 -- start start 2024-01-11T09:26:32.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.761+0000 7f122efff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1228069540 0x7f122815ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:32.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.761+0000 7f122efff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1228157410 0x7f1228158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:32.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.761+0000 7f122efff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12281577e0 0x7f1228151db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:32.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.761+0000 7f122efff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1228153ef0 con 0x7f1228157410 2024-01-11T09:26:32.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.761+0000 7f122efff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1228153630 con 0x7f1228069540 2024-01-11T09:26:32.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.761+0000 7f122efff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12281522f0 con 0x7f12281577e0 2024-01-11T09:26:32.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.761+0000 7f1227fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1228157410 0x7f1228158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.761+0000 7f1227fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1228157410 0x7f1228158450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35350/0 (socket says 172.21.15.27:35350) 2024-01-11T09:26:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.761+0000 7f1227fff700 1 -- 172.21.15.27:0/3952908127 learned_addr learned my addr 172.21.15.27:0/3952908127 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.761+0000 7f122cd9b700 1 --2- 172.21.15.27:0/3952908127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1228069540 0x7f122815ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.761+0000 7f122d59c700 1 --2- 172.21.15.27:0/3952908127 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12281577e0 0x7f1228151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.761+0000 7f1227fff700 1 -- 172.21.15.27:0/3952908127 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12281577e0 msgr2=0x7f1228151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.761+0000 7f1227fff700 1 --2- 172.21.15.27:0/3952908127 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12281577e0 0x7f1228151db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.762+0000 7f1227fff700 1 -- 172.21.15.27:0/3952908127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1228069540 msgr2=0x7f122815ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.762+0000 7f1227fff700 1 --2- 172.21.15.27:0/3952908127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1228069540 0x7f122815ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.762+0000 7f1227fff700 1 -- 172.21.15.27:0/3952908127 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121c009490 con 0x7f1228157410 2024-01-11T09:26:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.762+0000 7f122d59c700 1 --2- 172.21.15.27:0/3952908127 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12281577e0 0x7f1228151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.762+0000 7f1227fff700 1 --2- 172.21.15.27:0/3952908127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1228157410 0x7f1228158450 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f1214006b00 tx=0x7f1214008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:32.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.762+0000 7f1224ff9700 1 -- 172.21.15.27:0/3952908127 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12140125f0 con 0x7f1228157410 2024-01-11T09:26:32.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.762+0000 7f1224ff9700 1 -- 172.21.15.27:0/3952908127 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1214012770 con 0x7f1228157410 2024-01-11T09:26:32.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.762+0000 7f122efff700 1 -- 172.21.15.27:0/3952908127 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12281538a0 con 0x7f1228157410 2024-01-11T09:26:32.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.762+0000 7f1224ff9700 1 -- 172.21.15.27:0/3952908127 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1214012b30 con 0x7f1228157410 2024-01-11T09:26:32.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.762+0000 7f122efff700 1 -- 172.21.15.27:0/3952908127 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1228158b50 con 0x7f1228157410 2024-01-11T09:26:32.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.763+0000 7f122efff700 1 -- 172.21.15.27:0/3952908127 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1228154060 con 0x7f1228157410 2024-01-11T09:26:32.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.764+0000 7f1224ff9700 1 -- 172.21.15.27:0/3952908127 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 89) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f121401e460 con 0x7f1228157410 2024-01-11T09:26:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.766+0000 7f1224ff9700 1 --2- 172.21.15.27:0/3952908127 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f120c066ea0 0x7f120c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.766+0000 7f1224ff9700 1 -- 172.21.15.27:0/3952908127 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5511+0+0 (secure 0 0 0) 0x7f12140849b0 con 0x7f1228157410 2024-01-11T09:26:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.766+0000 7f122cd9b700 1 --2- 172.21.15.27:0/3952908127 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f120c066ea0 0x7f120c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:32.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.767+0000 7f122cd9b700 1 --2- 172.21.15.27:0/3952908127 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f120c066ea0 0x7f120c069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f121c0099a0 tx=0x7f121c017000 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:32.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.770+0000 7f1224ff9700 1 -- 172.21.15.27:0/3952908127 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1214051d20 con 0x7f1228157410 2024-01-11T09:26:32.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.976+0000 7f122efff700 1 -- 172.21.15.27:0/3952908127 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f122804c3c0 con 0x7f1228157410 2024-01-11T09:26:32.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.984+0000 7f1224ff9700 1 -- 172.21.15.27:0/3952908127 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v57) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1214055340 con 0x7f1228157410 2024-01-11T09:26:32.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.987+0000 7f122efff700 1 -- 172.21.15.27:0/3952908127 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f120c066ea0 msgr2=0x7f120c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:32.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.987+0000 7f122efff700 1 --2- 172.21.15.27:0/3952908127 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f120c066ea0 0x7f120c069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f121c0099a0 tx=0x7f121c017000 comp rx=0 tx=0).stop 2024-01-11T09:26:32.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.987+0000 7f122efff700 1 -- 172.21.15.27:0/3952908127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1228157410 msgr2=0x7f1228158450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:32.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.987+0000 7f122efff700 1 --2- 172.21.15.27:0/3952908127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1228157410 0x7f1228158450 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f1214006b00 tx=0x7f1214008040 comp rx=0 tx=0).stop 2024-01-11T09:26:32.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.987+0000 7f122efff700 1 -- 172.21.15.27:0/3952908127 shutdown_connections 2024-01-11T09:26:32.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.987+0000 7f122efff700 1 --2- 172.21.15.27:0/3952908127 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f120c066ea0 0x7f120c069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.987+0000 7f122efff700 1 --2- 172.21.15.27:0/3952908127 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12281577e0 0x7f1228151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.987+0000 7f122efff700 1 --2- 172.21.15.27:0/3952908127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1228069540 0x7f122815ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.987+0000 7f122efff700 1 --2- 172.21.15.27:0/3952908127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1228157410 0x7f1228158450 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:32.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.987+0000 7f122efff700 1 -- 172.21.15.27:0/3952908127 >> 172.21.15.27:0/3952908127 conn(0x7f12280680a0 msgr2=0x7f1228152a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:32.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.988+0000 7f122efff700 1 -- 172.21.15.27:0/3952908127 shutdown_connections 2024-01-11T09:26:32.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:32.988+0000 7f122efff700 1 -- 172.21.15.27:0/3952908127 wait complete. 2024-01-11T09:26:33.004 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2024-01-11T09:26:33.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.210+0000 7f7596fff700 1 Processor -- start 2024-01-11T09:26:33.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.210+0000 7f7596fff700 1 -- start start 2024-01-11T09:26:33.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.211+0000 7f7596fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f759006b610 0x7f7590158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:33.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.211+0000 7f7596fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7590158b60 con 0x7f75900b19d0 2024-01-11T09:26:33.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.211+0000 7f7596fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75901599b0 con 0x7f75901575e0 2024-01-11T09:26:33.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.211+0000 7f7596fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75900cb5e0 con 0x7f759006b610 2024-01-11T09:26:33.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.211+0000 7f758ffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f75900b19d0 0x7f7590159e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57514/0 (socket says 172.21.15.27:57514) 2024-01-11T09:26:33.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.211+0000 7f758ffff700 1 -- 172.21.15.27:0/3429400923 learned_addr learned my addr 172.21.15.27:0/3429400923 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:33.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.212+0000 7f759559c700 1 --2- 172.21.15.27:0/3429400923 >> v2:172.21.15.77:3300/0 conn(0x7f759006b610 0x7f7590158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:33.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.212+0000 7f758f7fe700 1 -- 172.21.15.27:0/3429400923 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282463198 0 0) 0x7f757c001110 con 0x7f75900b19d0 2024-01-11T09:26:33.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.212+0000 7f758f7fe700 1 -- 172.21.15.27:0/3429400923 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7580003570 con 0x7f75900b19d0 2024-01-11T09:26:33.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.212+0000 7f758f7fe700 1 -- 172.21.15.27:0/3429400923 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3530616372 0 0) 0x7f75901599b0 con 0x7f75901575e0 2024-01-11T09:26:33.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.212+0000 7f758f7fe700 1 -- 172.21.15.27:0/3429400923 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f757c001110 con 0x7f75901575e0 2024-01-11T09:26:33.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.212+0000 7f759559c700 1 -- 172.21.15.27:0/3429400923 >> v1:172.21.15.77:6789/0 conn(0x7f75901575e0 legacy=0x7f75900b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:33.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.212+0000 7f759559c700 1 -- 172.21.15.27:0/3429400923 >> v1:172.21.15.27:6789/0 conn(0x7f75900b19d0 legacy=0x7f7590159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:33.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.212+0000 7f759559c700 1 -- 172.21.15.27:0/3429400923 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75900c0f90 con 0x7f759006b610 2024-01-11T09:26:33.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.213+0000 7f759559c700 1 --2- 172.21.15.27:0/3429400923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f759006b610 0x7f7590158620 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f7588009e60 tx=0x7f7588009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e61595058f0e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:33.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.213+0000 7f758f7fe700 1 -- 172.21.15.27:0/3429400923 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758800e070 con 0x7f759006b610 2024-01-11T09:26:33.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.213+0000 7f758f7fe700 1 -- 172.21.15.27:0/3429400923 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758800bab0 con 0x7f759006b610 2024-01-11T09:26:33.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.213+0000 7f758f7fe700 1 -- 172.21.15.27:0/3429400923 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758800be70 con 0x7f759006b610 2024-01-11T09:26:33.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.213+0000 7f7596fff700 1 -- 172.21.15.27:0/3429400923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f759006b610 msgr2=0x7f7590158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:33.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.213+0000 7f7596fff700 1 --2- 172.21.15.27:0/3429400923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f759006b610 0x7f7590158620 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f7588009e60 tx=0x7f7588009780 comp rx=0 tx=0).stop 2024-01-11T09:26:33.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.213+0000 7f7596fff700 1 -- 172.21.15.27:0/3429400923 shutdown_connections 2024-01-11T09:26:33.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.213+0000 7f7596fff700 1 --2- 172.21.15.27:0/3429400923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f759006b610 0x7f7590158620 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:33.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.213+0000 7f7596fff700 1 -- 172.21.15.27:0/3429400923 >> 172.21.15.27:0/3429400923 conn(0x7f75900680e0 msgr2=0x7f7590069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:33.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.214+0000 7f7596fff700 1 -- 172.21.15.27:0/3429400923 shutdown_connections 2024-01-11T09:26:33.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.214+0000 7f7596fff700 1 -- 172.21.15.27:0/3429400923 wait complete. 2024-01-11T09:26:33.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.215+0000 7f7596fff700 1 Processor -- start 2024-01-11T09:26:33.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.215+0000 7f7596fff700 1 -- start start 2024-01-11T09:26:33.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.216+0000 7f7596fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f759006b610 0x7f7590153e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:33.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.216+0000 7f7596fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75900b19d0 0x7f75900c6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:33.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.216+0000 7f7596fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f75901575e0 0x7f759015db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:33.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.216+0000 7f7596fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7590169c30 con 0x7f75900b19d0 2024-01-11T09:26:33.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.216+0000 7f7596fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7590154340 con 0x7f759006b610 2024-01-11T09:26:33.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.216+0000 7f7596fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75901698e0 con 0x7f75901575e0 2024-01-11T09:26:33.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.216+0000 7f758ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75900b19d0 0x7f75900c6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:33.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.217+0000 7f758ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75900b19d0 0x7f75900c6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35360/0 (socket says 172.21.15.27:35360) 2024-01-11T09:26:33.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.217+0000 7f758ffff700 1 -- 172.21.15.27:0/2346219024 learned_addr learned my addr 172.21.15.27:0/2346219024 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:33.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.217+0000 7f7594d9b700 1 --2- 172.21.15.27:0/2346219024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f759006b610 0x7f7590153e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:33.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.217+0000 7f759559c700 1 --2- 172.21.15.27:0/2346219024 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f75901575e0 0x7f759015db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:33.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.217+0000 7f758ffff700 1 -- 172.21.15.27:0/2346219024 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f75901575e0 msgr2=0x7f759015db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:33.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.217+0000 7f758ffff700 1 --2- 172.21.15.27:0/2346219024 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f75901575e0 0x7f759015db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:33.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.217+0000 7f758ffff700 1 -- 172.21.15.27:0/2346219024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f759006b610 msgr2=0x7f7590153e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:33.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.217+0000 7f758ffff700 1 --2- 172.21.15.27:0/2346219024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f759006b610 0x7f7590153e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:33.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.217+0000 7f758ffff700 1 -- 172.21.15.27:0/2346219024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7588009a20 con 0x7f75900b19d0 2024-01-11T09:26:33.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.217+0000 7f758ffff700 1 --2- 172.21.15.27:0/2346219024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75900b19d0 0x7f75900c6f50 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f757c0058d0 tx=0x7f757c007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:33.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.217+0000 7f758cff9700 1 -- 172.21.15.27:0/2346219024 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c005b30 con 0x7f75900b19d0 2024-01-11T09:26:33.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.218+0000 7f758cff9700 1 -- 172.21.15.27:0/2346219024 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757c005c90 con 0x7f75900b19d0 2024-01-11T09:26:33.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.218+0000 7f758cff9700 1 -- 172.21.15.27:0/2346219024 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c007300 con 0x7f75900b19d0 2024-01-11T09:26:33.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.218+0000 7f7596fff700 1 -- 172.21.15.27:0/2346219024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75900c7780 con 0x7f75900b19d0 2024-01-11T09:26:33.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.218+0000 7f7596fff700 1 -- 172.21.15.27:0/2346219024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75900c7d40 con 0x7f75900b19d0 2024-01-11T09:26:33.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.219+0000 7f7596fff700 1 -- 172.21.15.27:0/2346219024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75900718d0 con 0x7f75900b19d0 2024-01-11T09:26:33.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.220+0000 7f758cff9700 1 -- 172.21.15.27:0/2346219024 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f757c01b400 con 0x7f75900b19d0 2024-01-11T09:26:33.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.220+0000 7f758cff9700 1 --2- 172.21.15.27:0/2346219024 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7574066f30 0x7f75740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:33.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.221+0000 7f758cff9700 1 -- 172.21.15.27:0/2346219024 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5511+0+0 (secure 0 0 0) 0x7f757c083ee0 con 0x7f75900b19d0 2024-01-11T09:26:33.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.221+0000 7f7594d9b700 1 --2- 172.21.15.27:0/2346219024 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7574066f30 0x7f75740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:33.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.221+0000 7f7594d9b700 1 --2- 172.21.15.27:0/2346219024 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7574066f30 0x7f75740693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7584007fd0 tx=0x7f7584007d50 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:33.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.225+0000 7f758cff9700 1 -- 172.21.15.27:0/2346219024 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f757c0511f0 con 0x7f75900b19d0 2024-01-11T09:26:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.513+0000 7f7596fff700 1 -- 172.21.15.27:0/2346219024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f7590158b60 con 0x7f75900b19d0 2024-01-11T09:26:33.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.632+0000 7f758cff9700 1 -- 172.21.15.27:0/2346219024 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v2468) v1 ==== 178+0+0 (secure 0 0 0) 0x7f7590158b60 con 0x7f75900b19d0 2024-01-11T09:26:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.635+0000 7f7596fff700 1 -- 172.21.15.27:0/2346219024 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7574066f30 msgr2=0x7f75740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.635+0000 7f7596fff700 1 --2- 172.21.15.27:0/2346219024 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7574066f30 0x7f75740693e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7584007fd0 tx=0x7f7584007d50 comp rx=0 tx=0).stop 2024-01-11T09:26:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.635+0000 7f7596fff700 1 -- 172.21.15.27:0/2346219024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75900b19d0 msgr2=0x7f75900c6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.635+0000 7f7596fff700 1 --2- 172.21.15.27:0/2346219024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75900b19d0 0x7f75900c6f50 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f757c0058d0 tx=0x7f757c007e80 comp rx=0 tx=0).stop 2024-01-11T09:26:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.635+0000 7f7596fff700 1 -- 172.21.15.27:0/2346219024 shutdown_connections 2024-01-11T09:26:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.635+0000 7f7596fff700 1 --2- 172.21.15.27:0/2346219024 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7574066f30 0x7f75740693e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.635+0000 7f7596fff700 1 --2- 172.21.15.27:0/2346219024 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f75901575e0 0x7f759015db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.635+0000 7f7596fff700 1 --2- 172.21.15.27:0/2346219024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f759006b610 0x7f7590153e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.635+0000 7f7596fff700 1 --2- 172.21.15.27:0/2346219024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75900b19d0 0x7f75900c6f50 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.635+0000 7f7596fff700 1 -- 172.21.15.27:0/2346219024 >> 172.21.15.27:0/2346219024 conn(0x7f75900680e0 msgr2=0x7f759006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.635+0000 7f7596fff700 1 -- 172.21.15.27:0/2346219024 shutdown_connections 2024-01-11T09:26:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.635+0000 7f7596fff700 1 -- 172.21.15.27:0/2346219024 wait complete. 2024-01-11T09:26:33.650 INFO:tasks.cephfs_test_runner:Test that number of snaps retained are as per user spec. ... ok 2024-01-11T09:26:33.650 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2024-01-11T09:26:33.650 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-11T09:26:33.650 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2024-01-11T09:26:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.854+0000 7fe2ffdff700 1 Processor -- start 2024-01-11T09:26:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.854+0000 7fe2ffdff700 1 -- start start 2024-01-11T09:26:33.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.856+0000 7fe2ffdff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe2f81596c0 0x7fe2f815d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:33.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.856+0000 7fe2ffdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2f80ca670 0x7fe2f8068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.856+0000 7fe2ffdff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2f8068b60 0x7fe2f8159a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.856+0000 7fe2ffdff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2f8068fe0 con 0x7fe2f8068b60 2024-01-11T09:26:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.856+0000 7fe2ffdff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2f80a9cb0 con 0x7fe2f81596c0 2024-01-11T09:26:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.856+0000 7fe2ffdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f815e210 con 0x7fe2f80ca670 2024-01-11T09:26:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.856+0000 7fe2fe39c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2f8068b60 0x7fe2f8159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.856+0000 7fe2fe39c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2f8068b60 0x7fe2f8159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60620/0 (socket says 172.21.15.27:60620) 2024-01-11T09:26:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.856+0000 7fe2fe39c700 1 -- 172.21.15.27:0/1027547980 learned_addr learned my addr 172.21.15.27:0/1027547980 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.856+0000 7fe2fd39a700 1 --2- 172.21.15.27:0/1027547980 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2f80ca670 0x7fe2f8068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.856+0000 7fe2fdb9b700 1 --2- 172.21.15.27:0/1027547980 >> v2:172.21.15.77:3300/0 conn(0x7fe2f81596c0 0x7fe2f815d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.856+0000 7fe2fe39c700 1 -- 172.21.15.27:0/1027547980 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2f80ca670 msgr2=0x7fe2f8068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.856+0000 7fe2fe39c700 1 --2- 172.21.15.27:0/1027547980 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2f80ca670 0x7fe2f8068620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.856+0000 7fe2fe39c700 1 -- 172.21.15.27:0/1027547980 >> v2:172.21.15.77:3300/0 conn(0x7fe2f81596c0 msgr2=0x7fe2f815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.857+0000 7fe2fe39c700 1 --2- 172.21.15.27:0/1027547980 >> v2:172.21.15.77:3300/0 conn(0x7fe2f81596c0 0x7fe2f815d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.857+0000 7fe2fe39c700 1 -- 172.21.15.27:0/1027547980 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f815dc50 con 0x7fe2f8068b60 2024-01-11T09:26:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.857+0000 7fe2fdb9b700 1 --2- 172.21.15.27:0/1027547980 >> v2:172.21.15.77:3300/0 conn(0x7fe2f81596c0 0x7fe2f815d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.857+0000 7fe2fe39c700 1 --2- 172.21.15.27:0/1027547980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2f8068b60 0x7fe2f8159a90 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80c3a20 tx=0x7fe2f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf473bd55e5c968 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.857+0000 7fe2fcb99700 1 -- 172.21.15.27:0/1027547980 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f000e070 con 0x7fe2f8068b60 2024-01-11T09:26:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.857+0000 7fe2fcb99700 1 -- 172.21.15.27:0/1027547980 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f000ba50 con 0x7fe2f8068b60 2024-01-11T09:26:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.857+0000 7fe2fcb99700 1 -- 172.21.15.27:0/1027547980 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f00092e0 con 0x7fe2f8068b60 2024-01-11T09:26:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.858+0000 7fe2ffdff700 1 -- 172.21.15.27:0/1027547980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2f8068b60 msgr2=0x7fe2f8159a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.858+0000 7fe2ffdff700 1 --2- 172.21.15.27:0/1027547980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2f8068b60 0x7fe2f8159a90 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80c3a20 tx=0x7fe2f0009840 comp rx=0 tx=0).stop 2024-01-11T09:26:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.858+0000 7fe2ffdff700 1 -- 172.21.15.27:0/1027547980 shutdown_connections 2024-01-11T09:26:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.858+0000 7fe2ffdff700 1 --2- 172.21.15.27:0/1027547980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2f8068b60 0x7fe2f8159a90 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.858+0000 7fe2ffdff700 1 --2- 172.21.15.27:0/1027547980 >> v2:172.21.15.77:3300/0 conn(0x7fe2f81596c0 0x7fe2f815d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.858+0000 7fe2ffdff700 1 --2- 172.21.15.27:0/1027547980 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2f80ca670 0x7fe2f8068620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.858+0000 7fe2ffdff700 1 -- 172.21.15.27:0/1027547980 >> 172.21.15.27:0/1027547980 conn(0x7fe2f80c9f70 msgr2=0x7fe2f806deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.858+0000 7fe2ffdff700 1 -- 172.21.15.27:0/1027547980 shutdown_connections 2024-01-11T09:26:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.858+0000 7fe2ffdff700 1 -- 172.21.15.27:0/1027547980 wait complete. 2024-01-11T09:26:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.859+0000 7fe2ffdff700 1 Processor -- start 2024-01-11T09:26:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.859+0000 7fe2ffdff700 1 -- start start 2024-01-11T09:26:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.860+0000 7fe2ffdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2f8068b60 0x7fe2f8156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.860+0000 7fe2ffdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2f80ca670 0x7fe2f80afe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.860+0000 7fe2ffdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2f81596c0 0x7fe2f80b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.860+0000 7fe2ffdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f815bc00 con 0x7fe2f81596c0 2024-01-11T09:26:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.860+0000 7fe2ffdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f815d410 con 0x7fe2f80ca670 2024-01-11T09:26:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.860+0000 7fe2ffdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f80b2760 con 0x7fe2f8068b60 2024-01-11T09:26:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.861+0000 7fe2fe39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2f81596c0 0x7fe2f80b03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.861+0000 7fe2fd39a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2f80ca670 0x7fe2f80afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.861+0000 7fe2fdb9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2f8068b60 0x7fe2f8156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.861+0000 7fe2fe39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2f81596c0 0x7fe2f80b03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60628/0 (socket says 172.21.15.27:60628) 2024-01-11T09:26:33.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.861+0000 7fe2fe39c700 1 -- 172.21.15.27:0/2597420788 learned_addr learned my addr 172.21.15.27:0/2597420788 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:33.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.861+0000 7fe2fe39c700 1 -- 172.21.15.27:0/2597420788 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2f8068b60 msgr2=0x7fe2f8156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:33.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.861+0000 7fe2fe39c700 1 --2- 172.21.15.27:0/2597420788 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2f8068b60 0x7fe2f8156f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:33.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.861+0000 7fe2fe39c700 1 -- 172.21.15.27:0/2597420788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2f80ca670 msgr2=0x7fe2f80afe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:33.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.861+0000 7fe2fe39c700 1 --2- 172.21.15.27:0/2597420788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2f80ca670 0x7fe2f80afe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:33.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.861+0000 7fe2fe39c700 1 -- 172.21.15.27:0/2597420788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f0009ae0 con 0x7fe2f81596c0 2024-01-11T09:26:33.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.862+0000 7fe2fe39c700 1 --2- 172.21.15.27:0/2597420788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2f81596c0 0x7fe2f80b03b0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0009880 tx=0x7fe2f0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:33.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.862+0000 7fe2edffb700 1 -- 172.21.15.27:0/2597420788 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f00032f0 con 0x7fe2f81596c0 2024-01-11T09:26:33.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.862+0000 7fe2edffb700 1 -- 172.21.15.27:0/2597420788 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f0008d20 con 0x7fe2f81596c0 2024-01-11T09:26:33.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.862+0000 7fe2edffb700 1 -- 172.21.15.27:0/2597420788 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0026690 con 0x7fe2f81596c0 2024-01-11T09:26:33.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.862+0000 7fe2ffdff700 1 -- 172.21.15.27:0/2597420788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f815be10 con 0x7fe2f81596c0 2024-01-11T09:26:33.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.862+0000 7fe2ffdff700 1 -- 172.21.15.27:0/2597420788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f815d0a0 con 0x7fe2f81596c0 2024-01-11T09:26:33.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.863+0000 7fe2ffdff700 1 -- 172.21.15.27:0/2597420788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f815cef0 con 0x7fe2f81596c0 2024-01-11T09:26:33.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.864+0000 7fe2edffb700 1 -- 172.21.15.27:0/2597420788 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe2f0003040 con 0x7fe2f81596c0 2024-01-11T09:26:33.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.866+0000 7fe2edffb700 1 --2- 172.21.15.27:0/2597420788 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe2dc066f30 0x7fe2dc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:33.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.866+0000 7fe2edffb700 1 -- 172.21.15.27:0/2597420788 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5511+0+0 (secure 0 0 0) 0x7fe2f0085de0 con 0x7fe2f81596c0 2024-01-11T09:26:33.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.866+0000 7fe2fdb9b700 1 --2- 172.21.15.27:0/2597420788 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe2dc066f30 0x7fe2dc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.867+0000 7fe2fdb9b700 1 --2- 172.21.15.27:0/2597420788 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe2dc066f30 0x7fe2dc0693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80b2120 tx=0x7fe2f4008040 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:33.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:33.870+0000 7fe2edffb700 1 -- 172.21.15.27:0/2597420788 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe2f0053120 con 0x7fe2f81596c0 2024-01-11T09:26:34.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.152+0000 7fe2ffdff700 1 -- 172.21.15.27:0/2597420788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7fe2f815c310 con 0x7fe2f81596c0 2024-01-11T09:26:34.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.639+0000 7fe2edffb700 1 -- 172.21.15.27:0/2597420788 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2469) v1 ==== 177+0+0 (secure 0 0 0) 0x7fe2f0056740 con 0x7fe2f81596c0 2024-01-11T09:26:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.642+0000 7fe2ffdff700 1 -- 172.21.15.27:0/2597420788 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe2dc066f30 msgr2=0x7fe2dc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.642+0000 7fe2ffdff700 1 --2- 172.21.15.27:0/2597420788 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe2dc066f30 0x7fe2dc0693e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80b2120 tx=0x7fe2f4008040 comp rx=0 tx=0).stop 2024-01-11T09:26:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.642+0000 7fe2ffdff700 1 -- 172.21.15.27:0/2597420788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2f81596c0 msgr2=0x7fe2f80b03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.642+0000 7fe2ffdff700 1 --2- 172.21.15.27:0/2597420788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2f81596c0 0x7fe2f80b03b0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0009880 tx=0x7fe2f0018000 comp rx=0 tx=0).stop 2024-01-11T09:26:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.642+0000 7fe2ffdff700 1 -- 172.21.15.27:0/2597420788 shutdown_connections 2024-01-11T09:26:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.642+0000 7fe2ffdff700 1 --2- 172.21.15.27:0/2597420788 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe2dc066f30 0x7fe2dc0693e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.642+0000 7fe2ffdff700 1 --2- 172.21.15.27:0/2597420788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2f81596c0 0x7fe2f80b03b0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.642+0000 7fe2ffdff700 1 --2- 172.21.15.27:0/2597420788 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2f8068b60 0x7fe2f8156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.642+0000 7fe2ffdff700 1 --2- 172.21.15.27:0/2597420788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2f80ca670 0x7fe2f80afe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.642+0000 7fe2ffdff700 1 -- 172.21.15.27:0/2597420788 >> 172.21.15.27:0/2597420788 conn(0x7fe2f80c9f70 msgr2=0x7fe2f8159a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:34.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.642+0000 7fe2ffdff700 1 -- 172.21.15.27:0/2597420788 shutdown_connections 2024-01-11T09:26:34.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.642+0000 7fe2ffdff700 1 -- 172.21.15.27:0/2597420788 wait complete. 2024-01-11T09:26:34.660 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-11T09:26:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.870+0000 7f10d49ff700 1 Processor -- start 2024-01-11T09:26:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.870+0000 7f10d49ff700 1 -- start start 2024-01-11T09:26:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.872+0000 7f10d49ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f10d015a1b0 0x7f10d015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.872+0000 7f10d49ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10d015ae70 0x7f10d015b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.872+0000 7f10d49ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d00cf850 con 0x7f10d00ccd70 2024-01-11T09:26:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.872+0000 7f10d49ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d00cec40 con 0x7f10d015a1b0 2024-01-11T09:26:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.872+0000 7f10d49ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d00cef20 con 0x7f10d015ae70 2024-01-11T09:26:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.872+0000 7f10cdd9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f10d00ccd70 0x7f10d00cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36122/0 (socket says 172.21.15.27:36122) 2024-01-11T09:26:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.872+0000 7f10cdd9b700 1 -- 172.21.15.27:0/450472046 learned_addr learned my addr 172.21.15.27:0/450472046 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.872+0000 7f10ced9d700 1 --2- 172.21.15.27:0/450472046 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10d015ae70 0x7f10d015b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.872+0000 7f10ce59c700 1 --2- 172.21.15.27:0/450472046 >> v2:172.21.15.77:3300/0 conn(0x7f10d015a1b0 0x7f10d015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.872+0000 7f10cd59a700 1 -- 172.21.15.27:0/450472046 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522846868 0 0) 0x7f10d00cf850 con 0x7f10d00ccd70 2024-01-11T09:26:34.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.873+0000 7f10cd59a700 1 -- 172.21.15.27:0/450472046 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b8003570 con 0x7f10d00ccd70 2024-01-11T09:26:34.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.873+0000 7f10cd59a700 1 -- 172.21.15.27:0/450472046 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2440815929 0 0) 0x7f10b8003570 con 0x7f10d00ccd70 2024-01-11T09:26:34.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.873+0000 7f10cd59a700 1 -- 172.21.15.27:0/450472046 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10d015ae70 msgr2=0x7f10d015b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:34.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.873+0000 7f10cd59a700 1 --2- 172.21.15.27:0/450472046 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10d015ae70 0x7f10d015b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:34.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.873+0000 7f10cd59a700 1 -- 172.21.15.27:0/450472046 >> v2:172.21.15.77:3300/0 conn(0x7f10d015a1b0 msgr2=0x7f10d015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:34.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.873+0000 7f10cd59a700 1 --2- 172.21.15.27:0/450472046 >> v2:172.21.15.77:3300/0 conn(0x7f10d015a1b0 0x7f10d015a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:34.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.873+0000 7f10cd59a700 1 -- 172.21.15.27:0/450472046 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d00cf200 con 0x7f10d00ccd70 2024-01-11T09:26:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.873+0000 7f10cd59a700 1 -- 172.21.15.27:0/450472046 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f10c4001470 con 0x7f10d00ccd70 2024-01-11T09:26:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.873+0000 7f10ce59c700 1 --2- 172.21.15.27:0/450472046 >> v2:172.21.15.77:3300/0 conn(0x7f10d015a1b0 0x7f10d015a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:26:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.873+0000 7f10cd59a700 1 -- 172.21.15.27:0/450472046 >> v1:172.21.15.27:6789/0 conn(0x7f10d00ccd70 legacy=0x7f10d00cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.873+0000 7f10cd59a700 1 --2- 172.21.15.27:0/450472046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10b8003ae0 0x7f10b8003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.873+0000 7f10cd59a700 1 --2- 172.21.15.27:0/450472046 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10d015ae70 0x7f10b8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.873+0000 7f10cd59a700 1 --2- 172.21.15.27:0/450472046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10b8004d50 0x7f10b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.873+0000 7f10cd59a700 1 -- 172.21.15.27:0/450472046 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d00cec40 con 0x7f10b8003ae0 2024-01-11T09:26:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.873+0000 7f10cd59a700 1 -- 172.21.15.27:0/450472046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d00cef20 con 0x7f10b8004d50 2024-01-11T09:26:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.873+0000 7f10cd59a700 1 -- 172.21.15.27:0/450472046 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b800bb10 con 0x7f10d015ae70 2024-01-11T09:26:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.873+0000 7f10ce59c700 1 --2- 172.21.15.27:0/450472046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10b8003ae0 0x7f10b8003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.874+0000 7f10cdd9b700 1 --2- 172.21.15.27:0/450472046 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10d015ae70 0x7f10b8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.874+0000 7f10ced9d700 1 --2- 172.21.15.27:0/450472046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10b8004d50 0x7f10b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.874+0000 7f10ce59c700 1 -- 172.21.15.27:0/450472046 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10d015ae70 msgr2=0x7f10b8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.874+0000 7f10ce59c700 1 --2- 172.21.15.27:0/450472046 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10d015ae70 0x7f10b8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.874+0000 7f10ce59c700 1 -- 172.21.15.27:0/450472046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10b8004d50 msgr2=0x7f10b80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.874+0000 7f10ce59c700 1 --2- 172.21.15.27:0/450472046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10b8004d50 0x7f10b80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.874+0000 7f10ce59c700 1 -- 172.21.15.27:0/450472046 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b8003570 con 0x7f10b8003ae0 2024-01-11T09:26:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.874+0000 7f10ce59c700 1 --2- 172.21.15.27:0/450472046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10b8003ae0 0x7f10b8003ef0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f10c0009e20 tx=0x7f10c00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30a6708bbbb4a006 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.874+0000 7f10cd59a700 1 -- 172.21.15.27:0/450472046 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c000e920 con 0x7f10b8003ae0 2024-01-11T09:26:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.874+0000 7f10cd59a700 1 -- 172.21.15.27:0/450472046 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c000ea80 con 0x7f10b8003ae0 2024-01-11T09:26:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.874+0000 7f10cd59a700 1 -- 172.21.15.27:0/450472046 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c000ee20 con 0x7f10b8003ae0 2024-01-11T09:26:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.875+0000 7f10d49ff700 1 -- 172.21.15.27:0/450472046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10b8003ae0 msgr2=0x7f10b8003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.875+0000 7f10d49ff700 1 --2- 172.21.15.27:0/450472046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10b8003ae0 0x7f10b8003ef0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f10c0009e20 tx=0x7f10c00091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.875+0000 7f10ced9d700 1 -- 172.21.15.27:0/450472046 reap_dead start 2024-01-11T09:26:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.875+0000 7f10d49ff700 1 -- 172.21.15.27:0/450472046 shutdown_connections 2024-01-11T09:26:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.875+0000 7f10d49ff700 1 -- 172.21.15.27:0/450472046 >> 172.21.15.27:0/450472046 conn(0x7f10d006ed40 msgr2=0x7f10d006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.875+0000 7f10d49ff700 1 -- 172.21.15.27:0/450472046 shutdown_connections 2024-01-11T09:26:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.875+0000 7f10d49ff700 1 -- 172.21.15.27:0/450472046 wait complete. 2024-01-11T09:26:34.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.876+0000 7f10d49ff700 1 Processor -- start 2024-01-11T09:26:34.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.877+0000 7f10d49ff700 1 -- start start 2024-01-11T09:26:34.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.877+0000 7f10d49ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10b8003ae0 0x7f10d0158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:34.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.877+0000 7f10d49ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10d0158210 0x7f10d00a6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:34.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.878+0000 7f10d49ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10d00b54d0 0x7f10d00b5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:34.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.878+0000 7f10d49ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d0158680 con 0x7f10b8003ae0 2024-01-11T09:26:34.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.878+0000 7f10d49ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d00b5e80 con 0x7f10d0158210 2024-01-11T09:26:34.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.878+0000 7f10d49ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d0169930 con 0x7f10d00b54d0 2024-01-11T09:26:34.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.878+0000 7f10ce59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10b8003ae0 0x7f10d0158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:34.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.878+0000 7f10ce59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10b8003ae0 0x7f10d0158830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60642/0 (socket says 172.21.15.27:60642) 2024-01-11T09:26:34.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.878+0000 7f10ce59c700 1 -- 172.21.15.27:0/4035575589 learned_addr learned my addr 172.21.15.27:0/4035575589 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:34.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.878+0000 7f10cdd9b700 1 --2- 172.21.15.27:0/4035575589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10d0158210 0x7f10d00a6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:34.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.878+0000 7f10ced9d700 1 --2- 172.21.15.27:0/4035575589 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10d00b54d0 0x7f10d00b5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:34.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.878+0000 7f10ce59c700 1 -- 172.21.15.27:0/4035575589 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10d00b54d0 msgr2=0x7f10d00b5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:34.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.878+0000 7f10ce59c700 1 --2- 172.21.15.27:0/4035575589 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10d00b54d0 0x7f10d00b5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:34.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.878+0000 7f10ce59c700 1 -- 172.21.15.27:0/4035575589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10d0158210 msgr2=0x7f10d00a6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:34.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.878+0000 7f10ce59c700 1 --2- 172.21.15.27:0/4035575589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10d0158210 0x7f10d00a6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:34.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.878+0000 7f10ce59c700 1 -- 172.21.15.27:0/4035575589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d016a480 con 0x7f10b8003ae0 2024-01-11T09:26:34.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.878+0000 7f10cdd9b700 1 --2- 172.21.15.27:0/4035575589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10d0158210 0x7f10d00a6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:34.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.879+0000 7f10ce59c700 1 --2- 172.21.15.27:0/4035575589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10b8003ae0 0x7f10d0158830 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f10c00139b0 tx=0x7f10c001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:34.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.879+0000 7f10be7fc700 1 -- 172.21.15.27:0/4035575589 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c0022070 con 0x7f10b8003ae0 2024-01-11T09:26:34.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.879+0000 7f10be7fc700 1 -- 172.21.15.27:0/4035575589 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c0013cf0 con 0x7f10b8003ae0 2024-01-11T09:26:34.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.879+0000 7f10d49ff700 1 -- 172.21.15.27:0/4035575589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d016a910 con 0x7f10b8003ae0 2024-01-11T09:26:34.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.879+0000 7f10be7fc700 1 -- 172.21.15.27:0/4035575589 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c0006e50 con 0x7f10b8003ae0 2024-01-11T09:26:34.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.879+0000 7f10d49ff700 1 -- 172.21.15.27:0/4035575589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d0169b70 con 0x7f10b8003ae0 2024-01-11T09:26:34.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.880+0000 7f10d49ff700 1 -- 172.21.15.27:0/4035575589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d00cb010 con 0x7f10b8003ae0 2024-01-11T09:26:34.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.881+0000 7f10be7fc700 1 -- 172.21.15.27:0/4035575589 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f10c0026070 con 0x7f10b8003ae0 2024-01-11T09:26:34.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.882+0000 7f10be7fc700 1 --2- 172.21.15.27:0/4035575589 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f10b0066f30 0x7f10b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:34.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.882+0000 7f10be7fc700 1 -- 172.21.15.27:0/4035575589 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5511+0+0 (secure 0 0 0) 0x7f10c0085d20 con 0x7f10b8003ae0 2024-01-11T09:26:34.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.883+0000 7f10cdd9b700 1 --2- 172.21.15.27:0/4035575589 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f10b0066f30 0x7f10b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:34.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.883+0000 7f10cdd9b700 1 --2- 172.21.15.27:0/4035575589 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f10b0066f30 0x7f10b00693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f10c4001ed0 tx=0x7f10c4009040 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:34.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:34.887+0000 7f10be7fc700 1 -- 172.21.15.27:0/4035575589 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f10c0053050 con 0x7f10b8003ae0 2024-01-11T09:26:35.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.090+0000 7f10d49ff700 1 -- 172.21.15.27:0/4035575589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f10d004c3c0 con 0x7f10b8003ae0 2024-01-11T09:26:35.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.095+0000 7f10be7fc700 1 -- 172.21.15.27:0/4035575589 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v58)=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7f10c00170c0 con 0x7f10b8003ae0 2024-01-11T09:26:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.097+0000 7f10d49ff700 1 -- 172.21.15.27:0/4035575589 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f10b0066f30 msgr2=0x7f10b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.097+0000 7f10d49ff700 1 --2- 172.21.15.27:0/4035575589 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f10b0066f30 0x7f10b00693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f10c4001ed0 tx=0x7f10c4009040 comp rx=0 tx=0).stop 2024-01-11T09:26:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.097+0000 7f10d49ff700 1 -- 172.21.15.27:0/4035575589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10b8003ae0 msgr2=0x7f10d0158830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.097+0000 7f10d49ff700 1 --2- 172.21.15.27:0/4035575589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10b8003ae0 0x7f10d0158830 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f10c00139b0 tx=0x7f10c001a000 comp rx=0 tx=0).stop 2024-01-11T09:26:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.097+0000 7f10d49ff700 1 -- 172.21.15.27:0/4035575589 shutdown_connections 2024-01-11T09:26:35.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.097+0000 7f10d49ff700 1 --2- 172.21.15.27:0/4035575589 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f10b0066f30 0x7f10b00693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:35.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.097+0000 7f10d49ff700 1 --2- 172.21.15.27:0/4035575589 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f10d00b54d0 0x7f10d00b5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:35.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.097+0000 7f10d49ff700 1 --2- 172.21.15.27:0/4035575589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10b8003ae0 0x7f10d0158830 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:35.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.097+0000 7f10d49ff700 1 --2- 172.21.15.27:0/4035575589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10d0158210 0x7f10d00a6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:35.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.097+0000 7f10d49ff700 1 -- 172.21.15.27:0/4035575589 >> 172.21.15.27:0/4035575589 conn(0x7f10d006ed40 msgr2=0x7f10d00c5540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:35.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.098+0000 7f10d49ff700 1 -- 172.21.15.27:0/4035575589 shutdown_connections 2024-01-11T09:26:35.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.098+0000 7f10d49ff700 1 -- 172.21.15.27:0/4035575589 wait complete. 2024-01-11T09:26:35.112 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T09:26:35.112 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:26:35.202 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:26:35.303 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T09:26:35.303 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-11T09:26:35.303 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:26:35.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.505+0000 7f68957ff700 1 Processor -- start 2024-01-11T09:26:35.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.505+0000 7f68957ff700 1 -- start start 2024-01-11T09:26:35.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.506+0000 7f68957ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f68900ca7b0 0x7f68900cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:35.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.506+0000 7f68957ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6890159e80 0x7f689015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:35.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.506+0000 7f68957ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689015d640 con 0x7f68900ca3e0 2024-01-11T09:26:35.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.506+0000 7f68957ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f689015ae70 con 0x7f68900ca7b0 2024-01-11T09:26:35.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.506+0000 7f68957ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68900cc4f0 con 0x7f6890159e80 2024-01-11T09:26:35.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.507+0000 7f688effd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f68900ca3e0 0x7f68900c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36136/0 (socket says 172.21.15.27:36136) 2024-01-11T09:26:35.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.507+0000 7f688effd700 1 -- 172.21.15.27:0/3998808128 learned_addr learned my addr 172.21.15.27:0/3998808128 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:35.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.507+0000 7f688e7fc700 1 --2- 172.21.15.27:0/3998808128 >> v2:172.21.15.27:3300/0 conn(0x7f68900ca7b0 0x7f68900cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:35.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.507+0000 7f688f7fe700 1 --2- 172.21.15.27:0/3998808128 >> v2:172.21.15.77:3300/0 conn(0x7f6890159e80 0x7f689015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:35.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.507+0000 7f688e7fc700 1 -- 172.21.15.27:0/3998808128 >> v2:172.21.15.77:3300/0 conn(0x7f6890159e80 msgr2=0x7f689015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:35.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.507+0000 7f688dffb700 1 -- 172.21.15.27:0/3998808128 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238878150 0 0) 0x7f689015d640 con 0x7f68900ca3e0 2024-01-11T09:26:35.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.507+0000 7f688e7fc700 1 --2- 172.21.15.27:0/3998808128 >> v2:172.21.15.77:3300/0 conn(0x7f6890159e80 0x7f689015dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:35.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.507+0000 7f688e7fc700 1 -- 172.21.15.27:0/3998808128 >> v1:172.21.15.27:6789/0 conn(0x7f68900ca3e0 legacy=0x7f68900c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:35.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.507+0000 7f688e7fc700 1 -- 172.21.15.27:0/3998808128 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689015a5d0 con 0x7f68900ca7b0 2024-01-11T09:26:35.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.507+0000 7f688e7fc700 1 --2- 172.21.15.27:0/3998808128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68900ca7b0 0x7f68900cca20 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f6884009910 tx=0x7f68840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f3267d9afc01fa server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:35.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.508+0000 7f688dffb700 1 -- 172.21.15.27:0/3998808128 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688400b7e0 con 0x7f68900ca7b0 2024-01-11T09:26:35.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.508+0000 7f688dffb700 1 -- 172.21.15.27:0/3998808128 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f688400b940 con 0x7f68900ca7b0 2024-01-11T09:26:35.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.508+0000 7f688dffb700 1 -- 172.21.15.27:0/3998808128 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688400bce0 con 0x7f68900ca7b0 2024-01-11T09:26:35.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.508+0000 7f68957ff700 1 -- 172.21.15.27:0/3998808128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68900ca7b0 msgr2=0x7f68900cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:35.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.508+0000 7f68957ff700 1 --2- 172.21.15.27:0/3998808128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68900ca7b0 0x7f68900cca20 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f6884009910 tx=0x7f68840091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:35.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.508+0000 7f68957ff700 1 -- 172.21.15.27:0/3998808128 shutdown_connections 2024-01-11T09:26:35.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.508+0000 7f68957ff700 1 --2- 172.21.15.27:0/3998808128 >> v2:172.21.15.77:3300/0 conn(0x7f6890159e80 0x7f689015dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:35.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.508+0000 7f68957ff700 1 --2- 172.21.15.27:0/3998808128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68900ca7b0 0x7f68900cca20 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:35.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.508+0000 7f68957ff700 1 -- 172.21.15.27:0/3998808128 >> 172.21.15.27:0/3998808128 conn(0x7f68900ce3e0 msgr2=0x7f68900ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:35.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.508+0000 7f68957ff700 1 -- 172.21.15.27:0/3998808128 shutdown_connections 2024-01-11T09:26:35.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.508+0000 7f68957ff700 1 -- 172.21.15.27:0/3998808128 wait complete. 2024-01-11T09:26:35.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.510+0000 7f68957ff700 1 Processor -- start 2024-01-11T09:26:35.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.510+0000 7f68957ff700 1 -- start start 2024-01-11T09:26:35.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.511+0000 7f68957ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68900ca3e0 0x7f689013acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:35.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.511+0000 7f68957ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68900ca7b0 0x7f689014f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:35.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.511+0000 7f68957ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6890159e80 0x7f689014fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:35.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.511+0000 7f68957ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689015e9a0 con 0x7f68900ca7b0 2024-01-11T09:26:35.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.511+0000 7f68957ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68900b1440 con 0x7f68900ca3e0 2024-01-11T09:26:35.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.511+0000 7f68957ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f689015dd90 con 0x7f6890159e80 2024-01-11T09:26:35.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.511+0000 7f688e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68900ca7b0 0x7f689014f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:35.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.511+0000 7f688e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68900ca7b0 0x7f689014f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60650/0 (socket says 172.21.15.27:60650) 2024-01-11T09:26:35.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.511+0000 7f688e7fc700 1 -- 172.21.15.27:0/1855152114 learned_addr learned my addr 172.21.15.27:0/1855152114 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:35.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.511+0000 7f688effd700 1 --2- 172.21.15.27:0/1855152114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68900ca3e0 0x7f689013acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:35.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.511+0000 7f688f7fe700 1 --2- 172.21.15.27:0/1855152114 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6890159e80 0x7f689014fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:35.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.512+0000 7f688e7fc700 1 -- 172.21.15.27:0/1855152114 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6890159e80 msgr2=0x7f689014fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:35.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.512+0000 7f688e7fc700 1 --2- 172.21.15.27:0/1855152114 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6890159e80 0x7f689014fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:35.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.512+0000 7f688e7fc700 1 -- 172.21.15.27:0/1855152114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68900ca3e0 msgr2=0x7f689013acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:35.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.512+0000 7f688e7fc700 1 --2- 172.21.15.27:0/1855152114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68900ca3e0 0x7f689013acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:35.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.512+0000 7f688e7fc700 1 -- 172.21.15.27:0/1855152114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6884009420 con 0x7f68900ca7b0 2024-01-11T09:26:35.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.512+0000 7f688effd700 1 --2- 172.21.15.27:0/1855152114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68900ca3e0 0x7f689013acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:26:35.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.512+0000 7f688e7fc700 1 --2- 172.21.15.27:0/1855152114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68900ca7b0 0x7f689014f790 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f6890064980 tx=0x7f6884017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:35.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.512+0000 7f6876ffd700 1 -- 172.21.15.27:0/1855152114 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6884024440 con 0x7f68900ca7b0 2024-01-11T09:26:35.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.512+0000 7f6876ffd700 1 -- 172.21.15.27:0/1855152114 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68840245a0 con 0x7f68900ca7b0 2024-01-11T09:26:35.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.512+0000 7f6876ffd700 1 -- 172.21.15.27:0/1855152114 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6884024960 con 0x7f68900ca7b0 2024-01-11T09:26:35.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.512+0000 7f68957ff700 1 -- 172.21.15.27:0/1855152114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f689015e070 con 0x7f68900ca7b0 2024-01-11T09:26:35.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.512+0000 7f68957ff700 1 -- 172.21.15.27:0/1855152114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f689015e630 con 0x7f68900ca7b0 2024-01-11T09:26:35.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.513+0000 7f68957ff700 1 -- 172.21.15.27:0/1855152114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f689015eb10 con 0x7f68900ca7b0 2024-01-11T09:26:35.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.514+0000 7f6876ffd700 1 -- 172.21.15.27:0/1855152114 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6884024d50 con 0x7f68900ca7b0 2024-01-11T09:26:35.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.516+0000 7f6876ffd700 1 --2- 172.21.15.27:0/1855152114 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f687c066f30 0x7f687c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:35.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.516+0000 7f6876ffd700 1 -- 172.21.15.27:0/1855152114 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5511+0+0 (secure 0 0 0) 0x7f68840860b0 con 0x7f68900ca7b0 2024-01-11T09:26:35.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.516+0000 7f688effd700 1 --2- 172.21.15.27:0/1855152114 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f687c066f30 0x7f687c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:35.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.517+0000 7f688effd700 1 --2- 172.21.15.27:0/1855152114 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f687c066f30 0x7f687c0693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f68800018d0 tx=0x7f6880009040 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:35.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.520+0000 7f6876ffd700 1 -- 172.21.15.27:0/1855152114 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6884053490 con 0x7f68900ca7b0 2024-01-11T09:26:35.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.767+0000 7f68957ff700 1 -- 172.21.15.27:0/1855152114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68900718d0 con 0x7f68900ca7b0 2024-01-11T09:26:35.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.768+0000 7f6876ffd700 1 -- 172.21.15.27:0/1855152114 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+3526 (secure 0 0 0) 0x7f6884015de0 con 0x7f68900ca7b0 2024-01-11T09:26:35.765 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:26:35.765 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":79,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7352,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3818350333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3818350333},{"type":"v1","addr":"172.21.15.27:6834","nonce":3818350333}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7355,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/217390254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":217390254},{"type":"v1","addr":"172.21.15.27:6835","nonce":217390254}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2824437717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2824437717},{"type":"v1","addr":"172.21.15.77:6834","nonce":2824437717}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8623,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3796487243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":3796487243},{"type":"v1","addr":"172.21.15.77:6835","nonce":3796487243}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":79}],"filesystems":[]} 2024-01-11T09:26:35.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.770+0000 7f68957ff700 1 -- 172.21.15.27:0/1855152114 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f687c066f30 msgr2=0x7f687c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:35.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.770+0000 7f68957ff700 1 --2- 172.21.15.27:0/1855152114 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f687c066f30 0x7f687c0693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f68800018d0 tx=0x7f6880009040 comp rx=0 tx=0).stop 2024-01-11T09:26:35.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.770+0000 7f68957ff700 1 -- 172.21.15.27:0/1855152114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68900ca7b0 msgr2=0x7f689014f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:35.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.770+0000 7f68957ff700 1 --2- 172.21.15.27:0/1855152114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68900ca7b0 0x7f689014f790 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f6890064980 tx=0x7f6884017000 comp rx=0 tx=0).stop 2024-01-11T09:26:35.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.770+0000 7f68957ff700 1 -- 172.21.15.27:0/1855152114 shutdown_connections 2024-01-11T09:26:35.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.770+0000 7f68957ff700 1 --2- 172.21.15.27:0/1855152114 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f687c066f30 0x7f687c0693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:35.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.770+0000 7f68957ff700 1 --2- 172.21.15.27:0/1855152114 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6890159e80 0x7f689014fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:35.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.770+0000 7f68957ff700 1 --2- 172.21.15.27:0/1855152114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68900ca3e0 0x7f689013acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:35.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.770+0000 7f68957ff700 1 --2- 172.21.15.27:0/1855152114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68900ca7b0 0x7f689014f790 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:35.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.770+0000 7f68957ff700 1 -- 172.21.15.27:0/1855152114 >> 172.21.15.27:0/1855152114 conn(0x7f68900ce3e0 msgr2=0x7f6890069920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:35.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.770+0000 7f68957ff700 1 -- 172.21.15.27:0/1855152114 shutdown_connections 2024-01-11T09:26:35.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:35.770+0000 7f68957ff700 1 -- 172.21.15.27:0/1855152114 wait complete. 2024-01-11T09:26:35.768 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 79 2024-01-11T09:26:35.785 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-11T09:26:35.785 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-11T09:26:35.785 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-11T09:26:35.786 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:26:35.786 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-11T09:26:35.786 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-11T09:26:35.786 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-11T09:26:35.786 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:26:35.786 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-11T09:26:35.786 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-11T09:26:35.787 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-11T09:26:35.787 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:26:35.787 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-11T09:26:35.787 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-11T09:26:35.787 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-11T09:26:35.787 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:26:35.800 INFO:tasks.ceph.mds.a:Stopped 2024-01-11T09:26:35.801 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-11T09:26:35.801 INFO:tasks.ceph.mds.c:Stopped 2024-01-11T09:26:35.802 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-11T09:26:35.803 INFO:tasks.ceph.mds.d:Stopped 2024-01-11T09:26:35.803 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-11T09:26:35.805 INFO:tasks.ceph.mds.b:Stopped 2024-01-11T09:26:35.805 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-11T09:26:35.845 INFO:tasks.ceph.mds.a:Started 2024-01-11T09:26:35.845 DEBUG:teuthology.parallel:result is None 2024-01-11T09:26:35.848 INFO:tasks.ceph.mds.c:Started 2024-01-11T09:26:35.848 DEBUG:teuthology.parallel:result is None 2024-01-11T09:26:35.849 INFO:tasks.ceph.mds.d:Started 2024-01-11T09:26:35.849 DEBUG:teuthology.parallel:result is None 2024-01-11T09:26:35.852 INFO:tasks.ceph.mds.b:Started 2024-01-11T09:26:35.853 DEBUG:teuthology.parallel:result is None 2024-01-11T09:26:35.853 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-11T09:26:35.997 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T09:26:35.997 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T09:26:35.998 INFO:tasks.ceph.mds.b.smithi077.stdout:starting mds.b at 2024-01-11T09:26:35.998 INFO:tasks.ceph.mds.d.smithi077.stdout:starting mds.d at 2024-01-11T09:26:36.000 INFO:tasks.ceph.mds.d.smithi077.stderr:2024-01-11T09:26:36.003+0000 7f94303fd700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-01-11T09:26:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.071+0000 7f7f63dff700 1 Processor -- start 2024-01-11T09:26:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.071+0000 7f7f63dff700 1 -- start start 2024-01-11T09:26:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.072+0000 7f7f63dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f5c156ab0 0x7f7f5c0ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.072+0000 7f7f63dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7f5c0cc9b0 0x7f7f5c156e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.072+0000 7f7f63dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f5c0adc80 con 0x7f7f5c1573c0 2024-01-11T09:26:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.072+0000 7f7f63dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f5c0adf60 con 0x7f7f5c0cc9b0 2024-01-11T09:26:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.072+0000 7f7f63dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c0ae240 con 0x7f7f5c156ab0 2024-01-11T09:26:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.073+0000 7f7f6139a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7f5c0cc9b0 0x7f7f5c156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.073+0000 7f7f61b9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f5c156ab0 0x7f7f5c0ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.073+0000 7f7f6239c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7f5c1573c0 0x7f7f5c0af540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:46060/0 (socket says 172.21.15.27:46060) 2024-01-11T09:26:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.073+0000 7f7f6239c700 1 -- 172.21.15.27:0/4207940550 learned_addr learned my addr 172.21.15.27:0/4207940550 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.073+0000 7f7f6139a700 1 -- 172.21.15.27:0/4207940550 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f5c156ab0 msgr2=0x7f7f5c0ad650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.073+0000 7f7f6139a700 1 --2- 172.21.15.27:0/4207940550 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f5c156ab0 0x7f7f5c0ad650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.073+0000 7f7f6139a700 1 -- 172.21.15.27:0/4207940550 >> v1:172.21.15.77:6789/0 conn(0x7f7f5c1573c0 legacy=0x7f7f5c0af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.073+0000 7f7f6139a700 1 -- 172.21.15.27:0/4207940550 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c0ae520 con 0x7f7f5c0cc9b0 2024-01-11T09:26:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.073+0000 7f7f60b99700 1 -- 172.21.15.27:0/4207940550 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229112563 0 0) 0x7f7f5c0adc80 con 0x7f7f5c1573c0 2024-01-11T09:26:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.073+0000 7f7f6139a700 1 --2- 172.21.15.27:0/4207940550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c0cc9b0 0x7f7f5c156e80 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0098e0 tx=0x7f7f4c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11abd03f073ac2fd server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.074+0000 7f7f60b99700 1 -- 172.21.15.27:0/4207940550 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c00b640 con 0x7f7f5c0cc9b0 2024-01-11T09:26:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.074+0000 7f7f60b99700 1 -- 172.21.15.27:0/4207940550 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4c00b7a0 con 0x7f7f5c0cc9b0 2024-01-11T09:26:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.074+0000 7f7f60b99700 1 -- 172.21.15.27:0/4207940550 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c00bb40 con 0x7f7f5c0cc9b0 2024-01-11T09:26:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.074+0000 7f7f63dff700 1 -- 172.21.15.27:0/4207940550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c0cc9b0 msgr2=0x7f7f5c156e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.074+0000 7f7f63dff700 1 --2- 172.21.15.27:0/4207940550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c0cc9b0 0x7f7f5c156e80 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0098e0 tx=0x7f7f4c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.074+0000 7f7f63dff700 1 -- 172.21.15.27:0/4207940550 shutdown_connections 2024-01-11T09:26:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.074+0000 7f7f63dff700 1 --2- 172.21.15.27:0/4207940550 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f5c156ab0 0x7f7f5c0ad650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.074+0000 7f7f63dff700 1 --2- 172.21.15.27:0/4207940550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c0cc9b0 0x7f7f5c156e80 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.074+0000 7f7f63dff700 1 -- 172.21.15.27:0/4207940550 >> 172.21.15.27:0/4207940550 conn(0x7f7f5c0680b0 msgr2=0x7f7f5c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.074+0000 7f7f63dff700 1 -- 172.21.15.27:0/4207940550 shutdown_connections 2024-01-11T09:26:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.074+0000 7f7f63dff700 1 -- 172.21.15.27:0/4207940550 wait complete. 2024-01-11T09:26:36.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.076+0000 7f7f63dff700 1 Processor -- start 2024-01-11T09:26:36.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.076+0000 7f7f63dff700 1 -- start start 2024-01-11T09:26:36.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.077+0000 7f7f63dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f5c0cc9b0 0x7f7f5c155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:36.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.077+0000 7f7f63dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f5c156ab0 0x7f7f5c0c9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:36.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.077+0000 7f7f63dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c1573c0 0x7f7f5c158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:36.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.077+0000 7f7f63dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c0cb670 con 0x7f7f5c0cc9b0 2024-01-11T09:26:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.077+0000 7f7f63dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c0cb950 con 0x7f7f5c1573c0 2024-01-11T09:26:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.077+0000 7f7f63dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c0cbc30 con 0x7f7f5c156ab0 2024-01-11T09:26:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.077+0000 7f7f61b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f5c0cc9b0 0x7f7f5c155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.077+0000 7f7f61b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f5c0cc9b0 0x7f7f5c155b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60678/0 (socket says 172.21.15.27:60678) 2024-01-11T09:26:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.077+0000 7f7f61b9b700 1 -- 172.21.15.27:0/2084369483 learned_addr learned my addr 172.21.15.27:0/2084369483 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.078+0000 7f7f6239c700 1 --2- 172.21.15.27:0/2084369483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c1573c0 0x7f7f5c158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.078+0000 7f7f6139a700 1 --2- 172.21.15.27:0/2084369483 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f5c156ab0 0x7f7f5c0c9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.078+0000 7f7f61b9b700 1 -- 172.21.15.27:0/2084369483 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f5c156ab0 msgr2=0x7f7f5c0c9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.078+0000 7f7f61b9b700 1 --2- 172.21.15.27:0/2084369483 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f5c156ab0 0x7f7f5c0c9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.078+0000 7f7f61b9b700 1 -- 172.21.15.27:0/2084369483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c1573c0 msgr2=0x7f7f5c158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.078+0000 7f7f61b9b700 1 --2- 172.21.15.27:0/2084369483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c1573c0 0x7f7f5c158e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.078+0000 7f7f61b9b700 1 -- 172.21.15.27:0/2084369483 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c009420 con 0x7f7f5c0cc9b0 2024-01-11T09:26:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.078+0000 7f7f61b9b700 1 --2- 172.21.15.27:0/2084369483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f5c0cc9b0 0x7f7f5c155b90 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f7f5800ec80 tx=0x7f7f580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:36.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.078+0000 7f7f51ffb700 1 -- 172.21.15.27:0/2084369483 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5800c9d0 con 0x7f7f5c0cc9b0 2024-01-11T09:26:36.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.078+0000 7f7f51ffb700 1 -- 172.21.15.27:0/2084369483 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5800cb30 con 0x7f7f5c0cc9b0 2024-01-11T09:26:36.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.078+0000 7f7f51ffb700 1 -- 172.21.15.27:0/2084369483 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f58013470 con 0x7f7f5c0cc9b0 2024-01-11T09:26:36.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.078+0000 7f7f63dff700 1 -- 172.21.15.27:0/2084369483 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f5c158950 con 0x7f7f5c0cc9b0 2024-01-11T09:26:36.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.078+0000 7f7f63dff700 1 -- 172.21.15.27:0/2084369483 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f5c0cb1b0 con 0x7f7f5c0cc9b0 2024-01-11T09:26:36.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.079+0000 7f7f63dff700 1 -- 172.21.15.27:0/2084369483 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f5c158ae0 con 0x7f7f5c0cc9b0 2024-01-11T09:26:36.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.080+0000 7f7f51ffb700 1 -- 172.21.15.27:0/2084369483 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7f58013820 con 0x7f7f5c0cc9b0 2024-01-11T09:26:36.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.081+0000 7f7f51ffb700 1 --2- 172.21.15.27:0/2084369483 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7f40066f30 0x7f7f400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:36.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.081+0000 7f7f51ffb700 1 -- 172.21.15.27:0/2084369483 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5511+0+0 (secure 0 0 0) 0x7f7f58084900 con 0x7f7f5c0cc9b0 2024-01-11T09:26:36.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.082+0000 7f7f6139a700 1 --2- 172.21.15.27:0/2084369483 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7f40066f30 0x7f7f400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:36.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.082+0000 7f7f6139a700 1 --2- 172.21.15.27:0/2084369483 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7f40066f30 0x7f7f400693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0093f0 tx=0x7f7f4c017000 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:36.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.086+0000 7f7f51ffb700 1 -- 172.21.15.27:0/2084369483 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7f58051b90 con 0x7f7f5c0cc9b0 2024-01-11T09:26:36.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.306+0000 7f7f63dff700 1 -- 172.21.15.27:0/2084369483 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f7f5c04c3c0 con 0x7f7f5c0cc9b0 2024-01-11T09:26:36.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.658+0000 7f7f51ffb700 1 -- 172.21.15.27:0/2084369483 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v161) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7f580551b0 con 0x7f7f5c0cc9b0 2024-01-11T09:26:36.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.661+0000 7f7f63dff700 1 -- 172.21.15.27:0/2084369483 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7f40066f30 msgr2=0x7f7f400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:36.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.661+0000 7f7f63dff700 1 --2- 172.21.15.27:0/2084369483 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7f40066f30 0x7f7f400693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0093f0 tx=0x7f7f4c017000 comp rx=0 tx=0).stop 2024-01-11T09:26:36.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.661+0000 7f7f63dff700 1 -- 172.21.15.27:0/2084369483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f5c0cc9b0 msgr2=0x7f7f5c155b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:36.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.661+0000 7f7f63dff700 1 --2- 172.21.15.27:0/2084369483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f5c0cc9b0 0x7f7f5c155b90 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f7f5800ec80 tx=0x7f7f580091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:36.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.661+0000 7f7f63dff700 1 -- 172.21.15.27:0/2084369483 shutdown_connections 2024-01-11T09:26:36.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.661+0000 7f7f63dff700 1 --2- 172.21.15.27:0/2084369483 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7f40066f30 0x7f7f400693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:36.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.661+0000 7f7f63dff700 1 --2- 172.21.15.27:0/2084369483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f5c1573c0 0x7f7f5c158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:36.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.661+0000 7f7f63dff700 1 --2- 172.21.15.27:0/2084369483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f5c0cc9b0 0x7f7f5c155b90 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:36.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.661+0000 7f7f63dff700 1 --2- 172.21.15.27:0/2084369483 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f5c156ab0 0x7f7f5c0c9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:36.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.661+0000 7f7f63dff700 1 -- 172.21.15.27:0/2084369483 >> 172.21.15.27:0/2084369483 conn(0x7f7f5c0680b0 msgr2=0x7f7f5c14f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:36.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.661+0000 7f7f63dff700 1 -- 172.21.15.27:0/2084369483 shutdown_connections 2024-01-11T09:26:36.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.661+0000 7f7f63dff700 1 -- 172.21.15.27:0/2084369483 wait complete. 2024-01-11T09:26:36.659 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-11T09:26:36.678 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-11T09:26:36.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.882+0000 7fcd75bff700 1 Processor -- start 2024-01-11T09:26:36.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.882+0000 7fcd75bff700 1 -- start start 2024-01-11T09:26:36.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.883+0000 7fcd75bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd701575e0 0x7fcd70158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:36.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.883+0000 7fcd75bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd70158b60 con 0x7fcd7006b610 2024-01-11T09:26:36.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.883+0000 7fcd75bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd701599b0 con 0x7fcd700b19d0 2024-01-11T09:26:36.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.883+0000 7fcd75bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd700cb5e0 con 0x7fcd701575e0 2024-01-11T09:26:36.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.884+0000 7fcd6effd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcd7006b610 0x7fcd700b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36162/0 (socket says 172.21.15.27:36162) 2024-01-11T09:26:36.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.884+0000 7fcd6effd700 1 -- 172.21.15.27:0/438489206 learned_addr learned my addr 172.21.15.27:0/438489206 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:36.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.884+0000 7fcd6f7fe700 1 --2- 172.21.15.27:0/438489206 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd701575e0 0x7fcd70158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:36.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.884+0000 7fcd6e7fc700 1 -- 172.21.15.27:0/438489206 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736867812 0 0) 0x7fcd64001110 con 0x7fcd7006b610 2024-01-11T09:26:36.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.884+0000 7fcd6e7fc700 1 -- 172.21.15.27:0/438489206 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd58003570 con 0x7fcd7006b610 2024-01-11T09:26:36.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.884+0000 7fcd6e7fc700 1 -- 172.21.15.27:0/438489206 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530135199 0 0) 0x7fcd701599b0 con 0x7fcd700b19d0 2024-01-11T09:26:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.884+0000 7fcd6e7fc700 1 -- 172.21.15.27:0/438489206 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd64001110 con 0x7fcd700b19d0 2024-01-11T09:26:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.884+0000 7fcd6f7fe700 1 -- 172.21.15.27:0/438489206 >> v1:172.21.15.77:6789/0 conn(0x7fcd700b19d0 legacy=0x7fcd70159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.885+0000 7fcd6f7fe700 1 -- 172.21.15.27:0/438489206 >> v1:172.21.15.27:6789/0 conn(0x7fcd7006b610 legacy=0x7fcd700b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.885+0000 7fcd6f7fe700 1 -- 172.21.15.27:0/438489206 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd700c0f90 con 0x7fcd701575e0 2024-01-11T09:26:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.885+0000 7fcd6f7fe700 1 --2- 172.21.15.27:0/438489206 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd701575e0 0x7fcd70158620 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fcd60009910 tx=0x7fcd600091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f42b74e3516400db server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.885+0000 7fcd6e7fc700 1 -- 172.21.15.27:0/438489206 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd60011630 con 0x7fcd701575e0 2024-01-11T09:26:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.885+0000 7fcd6e7fc700 1 -- 172.21.15.27:0/438489206 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd60011790 con 0x7fcd701575e0 2024-01-11T09:26:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.885+0000 7fcd6e7fc700 1 -- 172.21.15.27:0/438489206 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd60011b30 con 0x7fcd701575e0 2024-01-11T09:26:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.886+0000 7fcd75bff700 1 -- 172.21.15.27:0/438489206 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd701575e0 msgr2=0x7fcd70158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.886+0000 7fcd75bff700 1 --2- 172.21.15.27:0/438489206 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd701575e0 0x7fcd70158620 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fcd60009910 tx=0x7fcd600091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.886+0000 7fcd75bff700 1 -- 172.21.15.27:0/438489206 shutdown_connections 2024-01-11T09:26:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.886+0000 7fcd75bff700 1 --2- 172.21.15.27:0/438489206 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd701575e0 0x7fcd70158620 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.886+0000 7fcd75bff700 1 -- 172.21.15.27:0/438489206 >> 172.21.15.27:0/438489206 conn(0x7fcd700680e0 msgr2=0x7fcd70069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.886+0000 7fcd75bff700 1 -- 172.21.15.27:0/438489206 shutdown_connections 2024-01-11T09:26:36.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.886+0000 7fcd75bff700 1 -- 172.21.15.27:0/438489206 wait complete. 2024-01-11T09:26:36.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.887+0000 7fcd75bff700 1 Processor -- start 2024-01-11T09:26:36.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.887+0000 7fcd75bff700 1 -- start start 2024-01-11T09:26:36.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.888+0000 7fcd75bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd7006b610 0x7fcd70169f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.888+0000 7fcd75bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd700b19d0 0x7fcd70153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.888+0000 7fcd75bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd701575e0 0x7fcd7006ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.888+0000 7fcd75bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd700c5650 con 0x7fcd701575e0 2024-01-11T09:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.888+0000 7fcd75bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7016a4d0 con 0x7fcd700b19d0 2024-01-11T09:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.888+0000 7fcd75bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd700c7d30 con 0x7fcd7006b610 2024-01-11T09:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.889+0000 7fcd6f7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd7006b610 0x7fcd70169f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.889+0000 7fcd6f7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd7006b610 0x7fcd70169f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:52902/0 (socket says 172.21.15.27:52902) 2024-01-11T09:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.889+0000 7fcd6ffff700 1 --2- 172.21.15.27:0/478075129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd701575e0 0x7fcd7006ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.889+0000 7fcd6effd700 1 --2- 172.21.15.27:0/478075129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd700b19d0 0x7fcd70153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.889+0000 7fcd6f7fe700 1 -- 172.21.15.27:0/478075129 learned_addr learned my addr 172.21.15.27:0/478075129 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.889+0000 7fcd6f7fe700 1 -- 172.21.15.27:0/478075129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd700b19d0 msgr2=0x7fcd70153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.889+0000 7fcd6f7fe700 1 --2- 172.21.15.27:0/478075129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd700b19d0 0x7fcd70153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.889+0000 7fcd6f7fe700 1 -- 172.21.15.27:0/478075129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd701575e0 msgr2=0x7fcd7006ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:36.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.889+0000 7fcd6f7fe700 1 --2- 172.21.15.27:0/478075129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd701575e0 0x7fcd7006ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:36.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.889+0000 7fcd6f7fe700 1 -- 172.21.15.27:0/478075129 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd60009420 con 0x7fcd7006b610 2024-01-11T09:26:36.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.889+0000 7fcd6ffff700 1 --2- 172.21.15.27:0/478075129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd701575e0 0x7fcd7006ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:26:36.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.889+0000 7fcd6effd700 1 --2- 172.21.15.27:0/478075129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd700b19d0 0x7fcd70153580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:26:36.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.890+0000 7fcd6f7fe700 1 --2- 172.21.15.27:0/478075129 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd7006b610 0x7fcd70169f90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fcd600091e0 tx=0x7fcd60017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:36.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.890+0000 7fcd577fe700 1 -- 172.21.15.27:0/478075129 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6001f070 con 0x7fcd7006b610 2024-01-11T09:26:36.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.890+0000 7fcd577fe700 1 -- 172.21.15.27:0/478075129 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd60002f70 con 0x7fcd7006b610 2024-01-11T09:26:36.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.890+0000 7fcd75bff700 1 -- 172.21.15.27:0/478075129 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd700c8010 con 0x7fcd7006b610 2024-01-11T09:26:36.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.890+0000 7fcd577fe700 1 -- 172.21.15.27:0/478075129 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd60011900 con 0x7fcd7006b610 2024-01-11T09:26:36.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.890+0000 7fcd75bff700 1 -- 172.21.15.27:0/478075129 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd700c85d0 con 0x7fcd7006b610 2024-01-11T09:26:36.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.891+0000 7fcd75bff700 1 -- 172.21.15.27:0/478075129 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd700718d0 con 0x7fcd7006b610 2024-01-11T09:26:36.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.892+0000 7fcd577fe700 1 -- 172.21.15.27:0/478075129 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fcd60023070 con 0x7fcd7006b610 2024-01-11T09:26:36.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.893+0000 7fcd577fe700 1 --2- 172.21.15.27:0/478075129 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fcd5c066f30 0x7fcd5c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:36.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.893+0000 7fcd577fe700 1 -- 172.21.15.27:0/478075129 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 4909+0+0 (secure 0 0 0) 0x7fcd600854b0 con 0x7fcd7006b610 2024-01-11T09:26:36.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.893+0000 7fcd6effd700 1 --2- 172.21.15.27:0/478075129 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fcd5c066f30 0x7fcd5c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:36.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.894+0000 7fcd6effd700 1 --2- 172.21.15.27:0/478075129 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fcd5c066f30 0x7fcd5c0693e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcd7006fb10 tx=0x7fcd64009040 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:36.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:36.898+0000 7fcd577fe700 1 -- 172.21.15.27:0/478075129 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcd600529a0 con 0x7fcd7006b610 2024-01-11T09:26:37.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.179+0000 7fcd75bff700 1 -- 172.21.15.27:0/478075129 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fcd70158da0 con 0x7fcd7006b610 2024-01-11T09:26:37.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.180+0000 7fcd577fe700 1 -- 172.21.15.27:0/478075129 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v39) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fcd6002a090 con 0x7fcd7006b610 2024-01-11T09:26:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:26:37.177 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:26:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-11T09:26:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-11T09:26:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lQm3yLhAAv6erz5hMfat8EUzRA5luMA==", 2024-01-11T09:26:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:26:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:26:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:26:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:26:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:26:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:26:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-11T09:26:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9l+FG8ChAACHndtVLx7qS3sidPniSycA==", 2024-01-11T09:26:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:26:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:26:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:26:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:26:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:26:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:26:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-11T09:26:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l1NnEOhAALHXsc5es7hufCthGdRzkyw==", 2024-01-11T09:26:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:26:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:26:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:26:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:26:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:26:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:26:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-11T09:26:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lruoWFhAAm8AmrZtTJhYI1+19F79RPQ==", 2024-01-11T09:26:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:26:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:26:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:26:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:26:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:26:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:26:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-11T09:26:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA2op9ltZygORAAZr1X1nCN0ce8rIVaGDpFyQ==", 2024-01-11T09:26:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:26:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:26:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:26:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:26:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:26:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-11T09:26:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA6op9lqSRPAhAAmtyCsyK5ZVGX2wfhOCMuFQ==", 2024-01-11T09:26:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:26:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:26:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:26:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:26:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:26:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-11T09:26:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA9op9lZcs8BhAA8j+U+jPOoy+fliVWZAPFYg==", 2024-01-11T09:26:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:26:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:26:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:26:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:26:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:26:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-11T09:26:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBAop9lFuCUChAAP6si/2W1mxH4ecQJvuumkw==", 2024-01-11T09:26:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:26:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:26:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:26:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:26:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:26:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-11T09:26:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBGop9lOpwBIRAAcpcsjwmX2IldcKHJWAgHuA==", 2024-01-11T09:26:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:26:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:26:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:26:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:26:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:26:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-11T09:26:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBJop9l4q8hJRAAK8mAYVRQMoWS6wBOU69HmA==", 2024-01-11T09:26:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:26:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:26:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:26:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:26:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:26:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-11T09:26:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBMop9lqMF9KBAAFgFFI2GWt9AGif4AoScThw==", 2024-01-11T09:26:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:26:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:26:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:26:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:26:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:26:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-11T09:26:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBPop9lN7eDLBAAKswPZFfs0g2AaSe6V+pbbQ==", 2024-01-11T09:26:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:26:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:26:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:26:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:26:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:26:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-11T09:26:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lT5okIBAABy8PbJJbA062VNmb1Y8qhg==", 2024-01-11T09:26:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:26:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:26:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-01-11T09:26:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-11T09:26:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:26:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-11T09:26:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lDf37LhAAqV+EabKXJEW1sS5xtu1tXw==", 2024-01-11T09:26:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:26:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:26:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T09:26:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T09:26:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T09:26:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:26:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-11T09:26:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lWayxJxAAcalM8QSOGtbCXxjYZGaNQA==", 2024-01-11T09:26:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:26:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:26:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T09:26:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T09:26:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T09:26:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:26:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-11T09:26:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lHD4tNhAAdmnFBmMrK+QhQYdpbzW0mg==", 2024-01-11T09:26:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:26:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:26:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T09:26:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T09:26:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T09:26:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:26:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-11T09:26:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAwop9lGPtDMxAAJ8FjrSEdCtQL8AhtAyqnTw==", 2024-01-11T09:26:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:26:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T09:26:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:26:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:26:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:26:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:26:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-11T09:26:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lXBZLJxAAXa1S8Vkm3MiV4igtNbg/1Q==", 2024-01-11T09:26:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:26:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T09:26:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T09:26:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:26:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:26:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-11T09:26:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l3ZGoHxAANqzytTTvWYSomG+RFsFDyg==", 2024-01-11T09:26:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:26:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T09:26:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T09:26:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:26:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-11T09:26:37.189 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:26:37.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.182+0000 7fcd75bff700 1 -- 172.21.15.27:0/478075129 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fcd5c066f30 msgr2=0x7fcd5c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:37.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.182+0000 7fcd75bff700 1 --2- 172.21.15.27:0/478075129 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fcd5c066f30 0x7fcd5c0693e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcd7006fb10 tx=0x7fcd64009040 comp rx=0 tx=0).stop 2024-01-11T09:26:37.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.182+0000 7fcd75bff700 1 -- 172.21.15.27:0/478075129 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd7006b610 msgr2=0x7fcd70169f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:37.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.182+0000 7fcd75bff700 1 --2- 172.21.15.27:0/478075129 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd7006b610 0x7fcd70169f90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fcd600091e0 tx=0x7fcd60017000 comp rx=0 tx=0).stop 2024-01-11T09:26:37.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.183+0000 7fcd75bff700 1 -- 172.21.15.27:0/478075129 shutdown_connections 2024-01-11T09:26:37.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.183+0000 7fcd75bff700 1 --2- 172.21.15.27:0/478075129 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fcd5c066f30 0x7fcd5c0693e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.183+0000 7fcd75bff700 1 --2- 172.21.15.27:0/478075129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd701575e0 0x7fcd7006ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.183+0000 7fcd75bff700 1 --2- 172.21.15.27:0/478075129 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd7006b610 0x7fcd70169f90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.183+0000 7fcd75bff700 1 --2- 172.21.15.27:0/478075129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd700b19d0 0x7fcd70153580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.183+0000 7fcd75bff700 1 -- 172.21.15.27:0/478075129 >> 172.21.15.27:0/478075129 conn(0x7fcd700680e0 msgr2=0x7fcd70159e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:37.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.183+0000 7fcd75bff700 1 -- 172.21.15.27:0/478075129 shutdown_connections 2024-01-11T09:26:37.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.183+0000 7fcd75bff700 1 -- 172.21.15.27:0/478075129 wait complete. 2024-01-11T09:26:37.197 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:26:37.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.400+0000 7f31adbff700 1 Processor -- start 2024-01-11T09:26:37.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.400+0000 7f31adbff700 1 -- start start 2024-01-11T09:26:37.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.401+0000 7f31adbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80b03e0 0x7f31a80b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:37.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.401+0000 7f31adbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f31a80b07b0 0x7f31a80b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:37.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.401+0000 7f31adbff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a80cd730 con 0x7f31a8159200 2024-01-11T09:26:37.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.401+0000 7f31adbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31a8159d50 con 0x7f31a80b07b0 2024-01-11T09:26:37.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.401+0000 7f31adbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a80ccb20 con 0x7f31a80b03e0 2024-01-11T09:26:37.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.402+0000 7f31a7fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f31a8159200 0x7f31a8159640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36166/0 (socket says 172.21.15.27:36166) 2024-01-11T09:26:37.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.402+0000 7f31a7fff700 1 -- 172.21.15.27:0/3777867781 learned_addr learned my addr 172.21.15.27:0/3777867781 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:37.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.402+0000 7f31a77fe700 1 --2- 172.21.15.27:0/3777867781 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80b03e0 0x7f31a80b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:37.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.402+0000 7f31a6ffd700 1 --2- 172.21.15.27:0/3777867781 >> v2:172.21.15.77:3300/0 conn(0x7f31a80b07b0 0x7f31a80b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:37.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.402+0000 7f31a67fc700 1 -- 172.21.15.27:0/3777867781 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214832044 0 0) 0x7f31a80cd730 con 0x7f31a8159200 2024-01-11T09:26:37.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.402+0000 7f31a67fc700 1 -- 172.21.15.27:0/3777867781 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3190003570 con 0x7f31a8159200 2024-01-11T09:26:37.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.402+0000 7f31a77fe700 1 -- 172.21.15.27:0/3777867781 >> v2:172.21.15.77:3300/0 conn(0x7f31a80b07b0 msgr2=0x7f31a80b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:37.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.402+0000 7f31a77fe700 1 --2- 172.21.15.27:0/3777867781 >> v2:172.21.15.77:3300/0 conn(0x7f31a80b07b0 0x7f31a80b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.402+0000 7f31a77fe700 1 -- 172.21.15.27:0/3777867781 >> v1:172.21.15.27:6789/0 conn(0x7f31a8159200 legacy=0x7f31a8159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.402+0000 7f31a77fe700 1 -- 172.21.15.27:0/3777867781 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a80cce00 con 0x7f31a80b03e0 2024-01-11T09:26:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.402+0000 7f31a6ffd700 1 --2- 172.21.15.27:0/3777867781 >> v2:172.21.15.77:3300/0 conn(0x7f31a80b07b0 0x7f31a80b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:26:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.403+0000 7f31a77fe700 1 --2- 172.21.15.27:0/3777867781 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80b03e0 0x7f31a80b2480 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f31980098e0 tx=0x7f31980091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b6d993343156a3d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.403+0000 7f31a67fc700 1 -- 172.21.15.27:0/3777867781 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3198011630 con 0x7f31a80b03e0 2024-01-11T09:26:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.403+0000 7f31a67fc700 1 -- 172.21.15.27:0/3777867781 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3198011790 con 0x7f31a80b03e0 2024-01-11T09:26:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.403+0000 7f31a67fc700 1 -- 172.21.15.27:0/3777867781 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3198011b30 con 0x7f31a80b03e0 2024-01-11T09:26:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.404+0000 7f31adbff700 1 -- 172.21.15.27:0/3777867781 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80b03e0 msgr2=0x7f31a80b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.404+0000 7f31adbff700 1 --2- 172.21.15.27:0/3777867781 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80b03e0 0x7f31a80b2480 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f31980098e0 tx=0x7f31980091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.404+0000 7f31adbff700 1 -- 172.21.15.27:0/3777867781 shutdown_connections 2024-01-11T09:26:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.404+0000 7f31adbff700 1 --2- 172.21.15.27:0/3777867781 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80b03e0 0x7f31a80b2480 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.404+0000 7f31adbff700 1 --2- 172.21.15.27:0/3777867781 >> v2:172.21.15.77:3300/0 conn(0x7f31a80b07b0 0x7f31a80b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.404+0000 7f31adbff700 1 -- 172.21.15.27:0/3777867781 >> 172.21.15.27:0/3777867781 conn(0x7f31a806ed40 msgr2=0x7f31a806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.404+0000 7f31adbff700 1 -- 172.21.15.27:0/3777867781 shutdown_connections 2024-01-11T09:26:37.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.404+0000 7f31adbff700 1 -- 172.21.15.27:0/3777867781 wait complete. 2024-01-11T09:26:37.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.405+0000 7f31adbff700 1 Processor -- start 2024-01-11T09:26:37.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.405+0000 7f31adbff700 1 -- start start 2024-01-11T09:26:37.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.406+0000 7f31adbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31a80b03e0 0x7f31a816a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:37.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.406+0000 7f31adbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80b07b0 0x7f31a8149950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:37.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.406+0000 7f31adbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a8159200 0x7f31a80af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:37.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.406+0000 7f31adbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a816ac50 con 0x7f31a80b03e0 2024-01-11T09:26:37.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.406+0000 7f31adbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a816b530 con 0x7f31a8159200 2024-01-11T09:26:37.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.406+0000 7f31adbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a816af00 con 0x7f31a80b07b0 2024-01-11T09:26:37.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.407+0000 7f31a77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31a80b03e0 0x7f31a816a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.407+0000 7f31a77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31a80b03e0 0x7f31a816a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60706/0 (socket says 172.21.15.27:60706) 2024-01-11T09:26:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.407+0000 7f31a77fe700 1 -- 172.21.15.27:0/3352397397 learned_addr learned my addr 172.21.15.27:0/3352397397 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.407+0000 7f31a6ffd700 1 --2- 172.21.15.27:0/3352397397 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80b07b0 0x7f31a8149950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.407+0000 7f31a7fff700 1 --2- 172.21.15.27:0/3352397397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a8159200 0x7f31a80af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.407+0000 7f31a77fe700 1 -- 172.21.15.27:0/3352397397 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80b07b0 msgr2=0x7f31a8149950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.407+0000 7f31a77fe700 1 --2- 172.21.15.27:0/3352397397 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80b07b0 0x7f31a8149950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.407+0000 7f31a77fe700 1 -- 172.21.15.27:0/3352397397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a8159200 msgr2=0x7f31a80af0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.407+0000 7f31a77fe700 1 --2- 172.21.15.27:0/3352397397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a8159200 0x7f31a80af0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.407+0000 7f31a77fe700 1 -- 172.21.15.27:0/3352397397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319c009420 con 0x7f31a80b03e0 2024-01-11T09:26:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.407+0000 7f31a7fff700 1 --2- 172.21.15.27:0/3352397397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a8159200 0x7f31a80af0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.407+0000 7f31a77fe700 1 --2- 172.21.15.27:0/3352397397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31a80b03e0 0x7f31a816a710 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f31a806fd20 tx=0x7f3198017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.407+0000 7f318f7fe700 1 -- 172.21.15.27:0/3352397397 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3198011b10 con 0x7f31a80b03e0 2024-01-11T09:26:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.408+0000 7f318f7fe700 1 -- 172.21.15.27:0/3352397397 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31980253f0 con 0x7f31a80b03e0 2024-01-11T09:26:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.408+0000 7f318f7fe700 1 -- 172.21.15.27:0/3352397397 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31980257b0 con 0x7f31a80b03e0 2024-01-11T09:26:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.408+0000 7f31adbff700 1 -- 172.21.15.27:0/3352397397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3198009420 con 0x7f31a80b03e0 2024-01-11T09:26:37.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.408+0000 7f31adbff700 1 -- 172.21.15.27:0/3352397397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a816b1c0 con 0x7f31a80b03e0 2024-01-11T09:26:37.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.409+0000 7f31adbff700 1 -- 172.21.15.27:0/3352397397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a80718d0 con 0x7f31a80b03e0 2024-01-11T09:26:37.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.410+0000 7f318f7fe700 1 -- 172.21.15.27:0/3352397397 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3198025ba0 con 0x7f31a80b03e0 2024-01-11T09:26:37.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.410+0000 7f318f7fe700 1 --2- 172.21.15.27:0/3352397397 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f3194066f30 0x7f31940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:37.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.411+0000 7f318f7fe700 1 -- 172.21.15.27:0/3352397397 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 4909+0+0 (secure 0 0 0) 0x7f3198085b20 con 0x7f31a80b03e0 2024-01-11T09:26:37.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.411+0000 7f31a6ffd700 1 --2- 172.21.15.27:0/3352397397 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f3194066f30 0x7f31940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:37.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.412+0000 7f31a6ffd700 1 --2- 172.21.15.27:0/3352397397 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f3194066f30 0x7f31940693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f31a80c6a90 tx=0x7f319c0091a0 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:37.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.415+0000 7f318f7fe700 1 -- 172.21.15.27:0/3352397397 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3198053010 con 0x7f31a80b03e0 2024-01-11T09:26:37.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.619+0000 7f31adbff700 1 -- 172.21.15.27:0/3352397397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31a80cd0e0 con 0x7f31a80b03e0 2024-01-11T09:26:37.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.620+0000 7f318f7fe700 1 -- 172.21.15.27:0/3352397397 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v163) v1 ==== 74+0+12184 (secure 0 0 0) 0x7f3198056630 con 0x7f31a80b03e0 2024-01-11T09:26:37.617 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:26:37.617 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":163,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:26:36.668372+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:26:37.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.623+0000 7f31adbff700 1 -- 172.21.15.27:0/3352397397 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f3194066f30 msgr2=0x7f31940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.623+0000 7f31adbff700 1 --2- 172.21.15.27:0/3352397397 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f3194066f30 0x7f31940693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f31a80c6a90 tx=0x7f319c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.623+0000 7f31adbff700 1 -- 172.21.15.27:0/3352397397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31a80b03e0 msgr2=0x7f31a816a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.623+0000 7f31adbff700 1 --2- 172.21.15.27:0/3352397397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31a80b03e0 0x7f31a816a710 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f31a806fd20 tx=0x7f3198017000 comp rx=0 tx=0).stop 2024-01-11T09:26:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.623+0000 7f31adbff700 1 -- 172.21.15.27:0/3352397397 shutdown_connections 2024-01-11T09:26:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.623+0000 7f31adbff700 1 --2- 172.21.15.27:0/3352397397 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f3194066f30 0x7f31940693e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.623+0000 7f31adbff700 1 --2- 172.21.15.27:0/3352397397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a8159200 0x7f31a80af0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.623+0000 7f31adbff700 1 --2- 172.21.15.27:0/3352397397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31a80b03e0 0x7f31a816a710 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.623+0000 7f31adbff700 1 --2- 172.21.15.27:0/3352397397 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31a80b07b0 0x7f31a8149950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.623+0000 7f31adbff700 1 -- 172.21.15.27:0/3352397397 >> 172.21.15.27:0/3352397397 conn(0x7f31a806ed40 msgr2=0x7f31a8158920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.623+0000 7f31adbff700 1 -- 172.21.15.27:0/3352397397 shutdown_connections 2024-01-11T09:26:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.623+0000 7f31adbff700 1 -- 172.21.15.27:0/3352397397 wait complete. 2024-01-11T09:26:37.638 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-11T09:26:37.639 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T09:26:37.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.846+0000 7fee0cdff700 1 Processor -- start 2024-01-11T09:26:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.846+0000 7fee0cdff700 1 -- start start 2024-01-11T09:26:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.847+0000 7fee0cdff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fee0813fbb0 0x7fee080ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.847+0000 7fee0cdff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fee0815a730 0x7fee080c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.847+0000 7fee0cdff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee080c6b30 con 0x7fee0813ff80 2024-01-11T09:26:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.847+0000 7fee0cdff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee080c6800 con 0x7fee0815a730 2024-01-11T09:26:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.847+0000 7fee0cdff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee080cb240 con 0x7fee0813fbb0 2024-01-11T09:26:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.848+0000 7fee06d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fee0815a730 0x7fee080c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.848+0000 7fee05d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fee0813ff80 0x7fee0815a020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36178/0 (socket says 172.21.15.27:36178) 2024-01-11T09:26:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.848+0000 7fee06d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fee0815a730 0x7fee080c6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60708/0 (socket says 172.21.15.27:60708) 2024-01-11T09:26:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.848+0000 7fee05d9b700 1 -- 172.21.15.27:0/2673337834 learned_addr learned my addr 172.21.15.27:0/2673337834 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.848+0000 7fee0659c700 1 --2- 172.21.15.27:0/2673337834 >> v2:172.21.15.77:3300/0 conn(0x7fee0813fbb0 0x7fee080ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.848+0000 7fee06d9d700 1 -- 172.21.15.27:0/2673337834 >> v2:172.21.15.77:3300/0 conn(0x7fee0813fbb0 msgr2=0x7fee080ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.848+0000 7fee06d9d700 1 --2- 172.21.15.27:0/2673337834 >> v2:172.21.15.77:3300/0 conn(0x7fee0813fbb0 0x7fee080ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.848+0000 7fee06d9d700 1 -- 172.21.15.27:0/2673337834 >> v1:172.21.15.27:6789/0 conn(0x7fee0813ff80 legacy=0x7fee0815a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:37.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.848+0000 7fee06d9d700 1 -- 172.21.15.27:0/2673337834 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee080cb520 con 0x7fee0815a730 2024-01-11T09:26:37.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.848+0000 7fee0559a700 1 -- 172.21.15.27:0/2673337834 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627008743 0 0) 0x7fee080c6b30 con 0x7fee0813ff80 2024-01-11T09:26:37.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.848+0000 7fee06d9d700 1 --2- 172.21.15.27:0/2673337834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0815a730 0x7fee080c6fc0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fee08064df0 tx=0x7fee00009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8475d226949dd9e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.849+0000 7fee0559a700 1 -- 172.21.15.27:0/2673337834 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0000e070 con 0x7fee0815a730 2024-01-11T09:26:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.849+0000 7fee0559a700 1 -- 172.21.15.27:0/2673337834 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0000bb60 con 0x7fee0815a730 2024-01-11T09:26:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.849+0000 7fee0559a700 1 -- 172.21.15.27:0/2673337834 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee00012430 con 0x7fee0815a730 2024-01-11T09:26:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.849+0000 7fee0cdff700 1 -- 172.21.15.27:0/2673337834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0815a730 msgr2=0x7fee080c6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.849+0000 7fee0cdff700 1 --2- 172.21.15.27:0/2673337834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0815a730 0x7fee080c6fc0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fee08064df0 tx=0x7fee00009780 comp rx=0 tx=0).stop 2024-01-11T09:26:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.849+0000 7fee0cdff700 1 -- 172.21.15.27:0/2673337834 shutdown_connections 2024-01-11T09:26:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.849+0000 7fee0cdff700 1 --2- 172.21.15.27:0/2673337834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0815a730 0x7fee080c6fc0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.849+0000 7fee0cdff700 1 --2- 172.21.15.27:0/2673337834 >> v2:172.21.15.77:3300/0 conn(0x7fee0813fbb0 0x7fee080ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.849+0000 7fee0cdff700 1 -- 172.21.15.27:0/2673337834 >> 172.21.15.27:0/2673337834 conn(0x7fee080680a0 msgr2=0x7fee080cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.849+0000 7fee0cdff700 1 -- 172.21.15.27:0/2673337834 shutdown_connections 2024-01-11T09:26:37.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.849+0000 7fee0cdff700 1 -- 172.21.15.27:0/2673337834 wait complete. 2024-01-11T09:26:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.851+0000 7fee0cdff700 1 Processor -- start 2024-01-11T09:26:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.851+0000 7fee0cdff700 1 -- start start 2024-01-11T09:26:37.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.852+0000 7fee0cdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0813fbb0 0x7fee080b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:37.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.852+0000 7fee0cdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee0813ff80 0x7fee0816a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:37.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.852+0000 7fee0cdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee0815a730 0x7fee0813cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.852+0000 7fee0cdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0816ab10 con 0x7fee0813fbb0 2024-01-11T09:26:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.852+0000 7fee0cdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0813d0a0 con 0x7fee0815a730 2024-01-11T09:26:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.852+0000 7fee0cdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee080b1b10 con 0x7fee0813ff80 2024-01-11T09:26:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.852+0000 7fee0659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0813fbb0 0x7fee080b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.852+0000 7fee0659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0813fbb0 0x7fee080b0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60710/0 (socket says 172.21.15.27:60710) 2024-01-11T09:26:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.852+0000 7fee0659c700 1 -- 172.21.15.27:0/1851367828 learned_addr learned my addr 172.21.15.27:0/1851367828 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.852+0000 7fee05d9b700 1 --2- 172.21.15.27:0/1851367828 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee0813ff80 0x7fee0816a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.853+0000 7fee06d9d700 1 --2- 172.21.15.27:0/1851367828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee0815a730 0x7fee0813cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.853+0000 7fee0659c700 1 -- 172.21.15.27:0/1851367828 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee0813ff80 msgr2=0x7fee0816a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.853+0000 7fee0659c700 1 --2- 172.21.15.27:0/1851367828 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee0813ff80 0x7fee0816a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.853+0000 7fee0659c700 1 -- 172.21.15.27:0/1851367828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee0815a730 msgr2=0x7fee0813cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.853+0000 7fee0659c700 1 --2- 172.21.15.27:0/1851367828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee0815a730 0x7fee0813cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.853+0000 7fee0659c700 1 -- 172.21.15.27:0/1851367828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee00009a20 con 0x7fee0813fbb0 2024-01-11T09:26:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.853+0000 7fee05d9b700 1 --2- 172.21.15.27:0/1851367828 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee0813ff80 0x7fee0816a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:37.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.853+0000 7fee0659c700 1 --2- 172.21.15.27:0/1851367828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0813fbb0 0x7fee080b0820 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fedf800ec80 tx=0x7fedf8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:37.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.853+0000 7fedf67fc700 1 -- 172.21.15.27:0/1851367828 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf800c9d0 con 0x7fee0813fbb0 2024-01-11T09:26:37.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.853+0000 7fedf67fc700 1 -- 172.21.15.27:0/1851367828 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf800cb30 con 0x7fee0813fbb0 2024-01-11T09:26:37.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.853+0000 7fedf67fc700 1 -- 172.21.15.27:0/1851367828 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8013470 con 0x7fee0813fbb0 2024-01-11T09:26:37.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.853+0000 7fee0cdff700 1 -- 172.21.15.27:0/1851367828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee080b0f00 con 0x7fee0813fbb0 2024-01-11T09:26:37.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.854+0000 7fee0cdff700 1 -- 172.21.15.27:0/1851367828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee080b14c0 con 0x7fee0813fbb0 2024-01-11T09:26:37.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.855+0000 7fee0cdff700 1 -- 172.21.15.27:0/1851367828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee0804c3c0 con 0x7fee0813fbb0 2024-01-11T09:26:37.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.855+0000 7fedf67fc700 1 -- 172.21.15.27:0/1851367828 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fedf8013820 con 0x7fee0813fbb0 2024-01-11T09:26:37.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.857+0000 7fedf67fc700 1 --2- 172.21.15.27:0/1851367828 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fede8066f30 0x7fede80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:37.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.857+0000 7fedf67fc700 1 -- 172.21.15.27:0/1851367828 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 4909+0+0 (secure 0 0 0) 0x7fedf8084980 con 0x7fee0813fbb0 2024-01-11T09:26:37.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.857+0000 7fee05d9b700 1 --2- 172.21.15.27:0/1851367828 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fede8066f30 0x7fede80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:37.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.858+0000 7fee05d9b700 1 --2- 172.21.15.27:0/1851367828 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fede8066f30 0x7fede80693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fee080b2170 tx=0x7fedfc009040 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:37.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:37.860+0000 7fedf67fc700 1 -- 172.21.15.27:0/1851367828 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fedf8051e70 con 0x7fee0813fbb0 2024-01-11T09:26:38.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.064+0000 7fee0cdff700 1 -- 172.21.15.27:0/1851367828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fee080cb800 con 0x7fee0813fbb0 2024-01-11T09:26:38.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.064+0000 7fedf67fc700 1 -- 172.21.15.27:0/1851367828 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v163) v1 ==== 93+0+10 (secure 0 0 0) 0x7fedf80210a0 con 0x7fee0813fbb0 2024-01-11T09:26:38.061 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T09:26:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.066+0000 7fee0cdff700 1 -- 172.21.15.27:0/1851367828 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fede8066f30 msgr2=0x7fede80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.066+0000 7fee0cdff700 1 --2- 172.21.15.27:0/1851367828 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fede8066f30 0x7fede80693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fee080b2170 tx=0x7fedfc009040 comp rx=0 tx=0).stop 2024-01-11T09:26:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.066+0000 7fee0cdff700 1 -- 172.21.15.27:0/1851367828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0813fbb0 msgr2=0x7fee080b0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.066+0000 7fee0cdff700 1 --2- 172.21.15.27:0/1851367828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0813fbb0 0x7fee080b0820 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fedf800ec80 tx=0x7fedf8009120 comp rx=0 tx=0).stop 2024-01-11T09:26:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.066+0000 7fee0cdff700 1 -- 172.21.15.27:0/1851367828 shutdown_connections 2024-01-11T09:26:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.066+0000 7fee0cdff700 1 --2- 172.21.15.27:0/1851367828 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fede8066f30 0x7fede80693e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.066+0000 7fee0cdff700 1 --2- 172.21.15.27:0/1851367828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee0815a730 0x7fee0813cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.066+0000 7fee0cdff700 1 --2- 172.21.15.27:0/1851367828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee0813fbb0 0x7fee080b0820 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.066+0000 7fee0cdff700 1 --2- 172.21.15.27:0/1851367828 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee0813ff80 0x7fee0816a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.066+0000 7fee0cdff700 1 -- 172.21.15.27:0/1851367828 >> 172.21.15.27:0/1851367828 conn(0x7fee080680a0 msgr2=0x7fee0813ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:38.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.066+0000 7fee0cdff700 1 -- 172.21.15.27:0/1851367828 shutdown_connections 2024-01-11T09:26:38.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.066+0000 7fee0cdff700 1 -- 172.21.15.27:0/1851367828 wait complete. 2024-01-11T09:26:38.081 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-11T09:26:38.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.284+0000 7fb1907ff700 1 Processor -- start 2024-01-11T09:26:38.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.285+0000 7fb1907ff700 1 -- start start 2024-01-11T09:26:38.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.286+0000 7fb1907ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb188159650 0x7fb18806bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:38.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.286+0000 7fb1907ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb188159a20 0x7fb1880b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:38.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.286+0000 7fb1907ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb18806fba0 con 0x7fb1880b2730 2024-01-11T09:26:38.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.286+0000 7fb1907ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb18806e700 con 0x7fb188159a20 2024-01-11T09:26:38.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.286+0000 7fb1907ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1880b11a0 con 0x7fb188159650 2024-01-11T09:26:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.287+0000 7fb18e59b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb188159650 0x7fb18806bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.287+0000 7fb18e59b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb188159650 0x7fb18806bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:52928/0 (socket says 172.21.15.27:52928) 2024-01-11T09:26:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.287+0000 7fb18e59b700 1 -- 172.21.15.27:0/3904539304 learned_addr learned my addr 172.21.15.27:0/3904539304 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.287+0000 7fb18dd9a700 1 --2- 172.21.15.27:0/3904539304 >> v2:172.21.15.77:3300/0 conn(0x7fb188159a20 0x7fb1880b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.287+0000 7fb18ed9c700 1 --1- 172.21.15.27:0/3904539304 >> v1:172.21.15.77:6789/0 conn(0x7fb1880b2730 0x7fb1880b2bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:46072/0 (socket says 172.21.15.27:46072) 2024-01-11T09:26:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.287+0000 7fb18dd9a700 1 -- 172.21.15.27:0/3904539304 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb188159650 msgr2=0x7fb18806bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.287+0000 7fb18dd9a700 1 --2- 172.21.15.27:0/3904539304 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb188159650 0x7fb18806bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.287+0000 7fb18dd9a700 1 -- 172.21.15.27:0/3904539304 >> v1:172.21.15.77:6789/0 conn(0x7fb1880b2730 legacy=0x7fb1880b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.287+0000 7fb18dd9a700 1 -- 172.21.15.27:0/3904539304 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18806c360 con 0x7fb188159a20 2024-01-11T09:26:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.287+0000 7fb18e59b700 1 --2- 172.21.15.27:0/3904539304 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb188159650 0x7fb18806bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:26:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.288+0000 7fb18dd9a700 1 --2- 172.21.15.27:0/3904539304 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb188159a20 0x7fb1880b0c60 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fb1780098e0 tx=0x7fb1780091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb8af6515239b59f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.288+0000 7fb18d599700 1 -- 172.21.15.27:0/3904539304 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17800b640 con 0x7fb188159a20 2024-01-11T09:26:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.288+0000 7fb18d599700 1 -- 172.21.15.27:0/3904539304 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb17800b7a0 con 0x7fb188159a20 2024-01-11T09:26:38.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.288+0000 7fb18d599700 1 -- 172.21.15.27:0/3904539304 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17800bb40 con 0x7fb188159a20 2024-01-11T09:26:38.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.288+0000 7fb1907ff700 1 -- 172.21.15.27:0/3904539304 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb188159a20 msgr2=0x7fb1880b0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:38.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.288+0000 7fb1907ff700 1 --2- 172.21.15.27:0/3904539304 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb188159a20 0x7fb1880b0c60 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fb1780098e0 tx=0x7fb1780091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.288+0000 7fb1907ff700 1 -- 172.21.15.27:0/3904539304 shutdown_connections 2024-01-11T09:26:38.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.288+0000 7fb1907ff700 1 --2- 172.21.15.27:0/3904539304 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb188159650 0x7fb18806bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.288+0000 7fb1907ff700 1 --2- 172.21.15.27:0/3904539304 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb188159a20 0x7fb1880b0c60 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.288+0000 7fb1907ff700 1 -- 172.21.15.27:0/3904539304 >> 172.21.15.27:0/3904539304 conn(0x7fb18806b200 msgr2=0x7fb1880aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:38.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.289+0000 7fb1907ff700 1 -- 172.21.15.27:0/3904539304 shutdown_connections 2024-01-11T09:26:38.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.289+0000 7fb1907ff700 1 -- 172.21.15.27:0/3904539304 wait complete. 2024-01-11T09:26:38.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.291+0000 7fb1907ff700 1 Processor -- start 2024-01-11T09:26:38.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.291+0000 7fb1907ff700 1 -- start start 2024-01-11T09:26:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.292+0000 7fb1907ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1880b2730 0x7fb1881696e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.292+0000 7fb1907ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb188159650 0x7fb188152820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.292+0000 7fb1907ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb188159a20 0x7fb188152d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.292+0000 7fb1907ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1880aa550 con 0x7fb1880b2730 2024-01-11T09:26:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.292+0000 7fb1907ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1880a9940 con 0x7fb188159650 2024-01-11T09:26:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.292+0000 7fb1907ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1880a9c20 con 0x7fb188159a20 2024-01-11T09:26:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.292+0000 7fb18e59b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1880b2730 0x7fb1881696e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.293+0000 7fb18e59b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1880b2730 0x7fb1881696e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60726/0 (socket says 172.21.15.27:60726) 2024-01-11T09:26:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.293+0000 7fb18e59b700 1 -- 172.21.15.27:0/1144294256 learned_addr learned my addr 172.21.15.27:0/1144294256 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.293+0000 7fb18dd9a700 1 --2- 172.21.15.27:0/1144294256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb188159650 0x7fb188152820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.293+0000 7fb18ed9c700 1 --2- 172.21.15.27:0/1144294256 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb188159a20 0x7fb188152d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.293+0000 7fb18e59b700 1 -- 172.21.15.27:0/1144294256 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb188159a20 msgr2=0x7fb188152d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.293+0000 7fb18e59b700 1 --2- 172.21.15.27:0/1144294256 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb188159a20 0x7fb188152d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.293+0000 7fb18e59b700 1 -- 172.21.15.27:0/1144294256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb188159650 msgr2=0x7fb188152820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.293+0000 7fb18e59b700 1 --2- 172.21.15.27:0/1144294256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb188159650 0x7fb188152820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.293+0000 7fb18e59b700 1 -- 172.21.15.27:0/1144294256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1840094f0 con 0x7fb1880b2730 2024-01-11T09:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.293+0000 7fb18e59b700 1 --2- 172.21.15.27:0/1144294256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1880b2730 0x7fb1881696e0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fb18400ec50 tx=0x7fb184009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.294+0000 7fb17e7fc700 1 -- 172.21.15.27:0/1144294256 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18400c850 con 0x7fb1880b2730 2024-01-11T09:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.294+0000 7fb17e7fc700 1 -- 172.21.15.27:0/1144294256 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18400c9b0 con 0x7fb1880b2730 2024-01-11T09:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.294+0000 7fb17e7fc700 1 -- 172.21.15.27:0/1144294256 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18400cd50 con 0x7fb1880b2730 2024-01-11T09:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.294+0000 7fb1907ff700 1 -- 172.21.15.27:0/1144294256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb178009420 con 0x7fb1880b2730 2024-01-11T09:26:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.294+0000 7fb1907ff700 1 -- 172.21.15.27:0/1144294256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1880aa1e0 con 0x7fb1880b2730 2024-01-11T09:26:38.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.295+0000 7fb1907ff700 1 -- 172.21.15.27:0/1144294256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1880aa6c0 con 0x7fb1880b2730 2024-01-11T09:26:38.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.296+0000 7fb17e7fc700 1 -- 172.21.15.27:0/1144294256 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb184013680 con 0x7fb1880b2730 2024-01-11T09:26:38.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.297+0000 7fb17e7fc700 1 --2- 172.21.15.27:0/1144294256 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb16c066f30 0x7fb16c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:38.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.298+0000 7fb17e7fc700 1 -- 172.21.15.27:0/1144294256 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 4909+0+0 (secure 0 0 0) 0x7fb184084290 con 0x7fb1880b2730 2024-01-11T09:26:38.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.298+0000 7fb18dd9a700 1 --2- 172.21.15.27:0/1144294256 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb16c066f30 0x7fb16c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:38.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.299+0000 7fb18dd9a700 1 --2- 172.21.15.27:0/1144294256 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb16c066f30 0x7fb16c0693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb1780093f0 tx=0x7fb178017000 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.302+0000 7fb17e7fc700 1 -- 172.21.15.27:0/1144294256 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb184051820 con 0x7fb1880b2730 2024-01-11T09:26:38.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.510+0000 7fb1907ff700 1 -- 172.21.15.27:0/1144294256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb1880aa850 con 0x7fb1880b2730 2024-01-11T09:26:38.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.510+0000 7fb17e7fc700 1 -- 172.21.15.27:0/1144294256 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v163) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb184054e40 con 0x7fb1880b2730 2024-01-11T09:26:38.507 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:26:38.507 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T09:26:38.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:38.507 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-11T09:26:38.507 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-11T09:26:38.508 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:38.508 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T09:26:38.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.513+0000 7fb1907ff700 1 -- 172.21.15.27:0/1144294256 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb16c066f30 msgr2=0x7fb16c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:38.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.513+0000 7fb1907ff700 1 --2- 172.21.15.27:0/1144294256 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb16c066f30 0x7fb16c0693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb1780093f0 tx=0x7fb178017000 comp rx=0 tx=0).stop 2024-01-11T09:26:38.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.513+0000 7fb1907ff700 1 -- 172.21.15.27:0/1144294256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1880b2730 msgr2=0x7fb1881696e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:38.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.513+0000 7fb1907ff700 1 --2- 172.21.15.27:0/1144294256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1880b2730 0x7fb1881696e0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fb18400ec50 tx=0x7fb184009250 comp rx=0 tx=0).stop 2024-01-11T09:26:38.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.513+0000 7fb1907ff700 1 -- 172.21.15.27:0/1144294256 shutdown_connections 2024-01-11T09:26:38.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.513+0000 7fb1907ff700 1 --2- 172.21.15.27:0/1144294256 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb16c066f30 0x7fb16c0693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.513+0000 7fb1907ff700 1 --2- 172.21.15.27:0/1144294256 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb188159a20 0x7fb188152d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.513+0000 7fb1907ff700 1 --2- 172.21.15.27:0/1144294256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1880b2730 0x7fb1881696e0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.513+0000 7fb1907ff700 1 --2- 172.21.15.27:0/1144294256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb188159650 0x7fb188152820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.513+0000 7fb1907ff700 1 -- 172.21.15.27:0/1144294256 >> 172.21.15.27:0/1144294256 conn(0x7fb18806b200 msgr2=0x7fb188155c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:38.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.513+0000 7fb1907ff700 1 -- 172.21.15.27:0/1144294256 shutdown_connections 2024-01-11T09:26:38.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.513+0000 7fb1907ff700 1 -- 172.21.15.27:0/1144294256 wait complete. 2024-01-11T09:26:38.527 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-11T09:26:38.528 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-11T09:26:38.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.737+0000 7fa504fff700 1 Processor -- start 2024-01-11T09:26:38.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.737+0000 7fa504fff700 1 -- start start 2024-01-11T09:26:38.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.738+0000 7fa504fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa50015a7a0 0x7fa50015c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:38.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.738+0000 7fa504fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa500159e60 0x7fa50015aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:38.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.738+0000 7fa504fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa50015b400 0x7fa5000685e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:38.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.738+0000 7fa504fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa50015c4b0 con 0x7fa50015b400 2024-01-11T09:26:38.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.738+0000 7fa504fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa50015b8a0 con 0x7fa500159e60 2024-01-11T09:26:38.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.738+0000 7fa504fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa50015bb80 con 0x7fa50015a7a0 2024-01-11T09:26:38.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.738+0000 7fa4fed9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa50015b400 0x7fa5000685e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:38.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.738+0000 7fa4fed9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa50015b400 0x7fa5000685e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60742/0 (socket says 172.21.15.27:60742) 2024-01-11T09:26:38.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.738+0000 7fa4fed9d700 1 -- 172.21.15.27:0/86055797 learned_addr learned my addr 172.21.15.27:0/86055797 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:38.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.739+0000 7fa4fe59c700 1 --2- 172.21.15.27:0/86055797 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa50015a7a0 0x7fa50015c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:38.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.739+0000 7fa4fdd9b700 1 --2- 172.21.15.27:0/86055797 >> v2:172.21.15.77:3300/0 conn(0x7fa500159e60 0x7fa50015aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:38.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.739+0000 7fa4fed9d700 1 -- 172.21.15.27:0/86055797 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa50015a7a0 msgr2=0x7fa50015c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:38.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.739+0000 7fa4fed9d700 1 --2- 172.21.15.27:0/86055797 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa50015a7a0 0x7fa50015c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.739+0000 7fa4fed9d700 1 -- 172.21.15.27:0/86055797 >> v2:172.21.15.77:3300/0 conn(0x7fa500159e60 msgr2=0x7fa50015aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:38.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.739+0000 7fa4fed9d700 1 --2- 172.21.15.27:0/86055797 >> v2:172.21.15.77:3300/0 conn(0x7fa500159e60 0x7fa50015aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.739+0000 7fa4fed9d700 1 -- 172.21.15.27:0/86055797 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50015be60 con 0x7fa50015b400 2024-01-11T09:26:38.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.739+0000 7fa4fdd9b700 1 --2- 172.21.15.27:0/86055797 >> v2:172.21.15.77:3300/0 conn(0x7fa500159e60 0x7fa50015aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:38.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.739+0000 7fa4fed9d700 1 --2- 172.21.15.27:0/86055797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa50015b400 0x7fa5000685e0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fa500089450 tx=0x7fa4f8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32c0ed57653cf1f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:38.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.739+0000 7fa4fd59a700 1 -- 172.21.15.27:0/86055797 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f800e070 con 0x7fa50015b400 2024-01-11T09:26:38.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.739+0000 7fa4fd59a700 1 -- 172.21.15.27:0/86055797 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f800ba20 con 0x7fa50015b400 2024-01-11T09:26:38.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.739+0000 7fa4fd59a700 1 -- 172.21.15.27:0/86055797 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f80092e0 con 0x7fa50015b400 2024-01-11T09:26:38.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.740+0000 7fa504fff700 1 -- 172.21.15.27:0/86055797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa50015b400 msgr2=0x7fa5000685e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:38.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.740+0000 7fa504fff700 1 --2- 172.21.15.27:0/86055797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa50015b400 0x7fa5000685e0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fa500089450 tx=0x7fa4f8009840 comp rx=0 tx=0).stop 2024-01-11T09:26:38.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.740+0000 7fa504fff700 1 -- 172.21.15.27:0/86055797 shutdown_connections 2024-01-11T09:26:38.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.740+0000 7fa504fff700 1 --2- 172.21.15.27:0/86055797 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa50015a7a0 0x7fa50015c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.740+0000 7fa504fff700 1 --2- 172.21.15.27:0/86055797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa50015b400 0x7fa5000685e0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.740+0000 7fa504fff700 1 --2- 172.21.15.27:0/86055797 >> v2:172.21.15.77:3300/0 conn(0x7fa500159e60 0x7fa50015aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.740+0000 7fa504fff700 1 -- 172.21.15.27:0/86055797 >> 172.21.15.27:0/86055797 conn(0x7fa50006ed40 msgr2=0x7fa50006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:38.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.740+0000 7fa504fff700 1 -- 172.21.15.27:0/86055797 shutdown_connections 2024-01-11T09:26:38.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.740+0000 7fa504fff700 1 -- 172.21.15.27:0/86055797 wait complete. 2024-01-11T09:26:38.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.742+0000 7fa504fff700 1 Processor -- start 2024-01-11T09:26:38.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.742+0000 7fa504fff700 1 -- start start 2024-01-11T09:26:38.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.743+0000 7fa504fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa50015a7a0 0x7fa500151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:38.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.743+0000 7fa504fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa50015b400 0x7fa50016aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:38.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.743+0000 7fa504fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500150df0 0x7fa50014cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:38.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.743+0000 7fa504fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5000b0660 con 0x7fa50015a7a0 2024-01-11T09:26:38.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.743+0000 7fa504fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5001522f0 con 0x7fa500150df0 2024-01-11T09:26:38.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.743+0000 7fa504fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5001525d0 con 0x7fa50015b400 2024-01-11T09:26:38.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.743+0000 7fa4fe59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa50015a7a0 0x7fa500151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:38.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.743+0000 7fa4fe59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa50015a7a0 0x7fa500151ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60748/0 (socket says 172.21.15.27:60748) 2024-01-11T09:26:38.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.743+0000 7fa4fe59c700 1 -- 172.21.15.27:0/994993716 learned_addr learned my addr 172.21.15.27:0/994993716 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:38.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.743+0000 7fa4fed9d700 1 --2- 172.21.15.27:0/994993716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500150df0 0x7fa50014cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.743+0000 7fa4fdd9b700 1 --2- 172.21.15.27:0/994993716 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa50015b400 0x7fa50016aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.743+0000 7fa4fe59c700 1 -- 172.21.15.27:0/994993716 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa50015b400 msgr2=0x7fa50016aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.743+0000 7fa4fe59c700 1 --2- 172.21.15.27:0/994993716 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa50015b400 0x7fa50016aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.743+0000 7fa4fe59c700 1 -- 172.21.15.27:0/994993716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500150df0 msgr2=0x7fa50014cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.743+0000 7fa4fe59c700 1 --2- 172.21.15.27:0/994993716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500150df0 0x7fa50014cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.743+0000 7fa4fe59c700 1 -- 172.21.15.27:0/994993716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f8009ae0 con 0x7fa50015a7a0 2024-01-11T09:26:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.744+0000 7fa4fe59c700 1 --2- 172.21.15.27:0/994993716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa50015a7a0 0x7fa500151ce0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fa4f000ec80 tx=0x7fa4f00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.744+0000 7fa4ee7fc700 1 -- 172.21.15.27:0/994993716 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f000c9d0 con 0x7fa50015a7a0 2024-01-11T09:26:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.744+0000 7fa4ee7fc700 1 -- 172.21.15.27:0/994993716 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f000cb50 con 0x7fa50015a7a0 2024-01-11T09:26:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.744+0000 7fa4ee7fc700 1 -- 172.21.15.27:0/994993716 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0013470 con 0x7fa50015a7a0 2024-01-11T09:26:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.744+0000 7fa504fff700 1 -- 172.21.15.27:0/994993716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5000b0010 con 0x7fa50015a7a0 2024-01-11T09:26:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.744+0000 7fa504fff700 1 -- 172.21.15.27:0/994993716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5000b2df0 con 0x7fa50015a7a0 2024-01-11T09:26:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.745+0000 7fa504fff700 1 -- 172.21.15.27:0/994993716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa50004c3c0 con 0x7fa50015a7a0 2024-01-11T09:26:38.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.746+0000 7fa4ee7fc700 1 -- 172.21.15.27:0/994993716 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa4f0013860 con 0x7fa50015a7a0 2024-01-11T09:26:38.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.747+0000 7fa4ee7fc700 1 --2- 172.21.15.27:0/994993716 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa4e0066f30 0x7fa4e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:38.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.747+0000 7fa4ee7fc700 1 -- 172.21.15.27:0/994993716 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 4909+0+0 (secure 0 0 0) 0x7fa4f0084690 con 0x7fa50015a7a0 2024-01-11T09:26:38.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.748+0000 7fa4fdd9b700 1 --2- 172.21.15.27:0/994993716 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa4e0066f30 0x7fa4e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:38.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.748+0000 7fa4fdd9b700 1 --2- 172.21.15.27:0/994993716 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa4e0066f30 0x7fa4e00693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4006f00 tx=0x7fa4f4008040 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:38.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.751+0000 7fa4ee7fc700 1 -- 172.21.15.27:0/994993716 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa4f0051b80 con 0x7fa50015a7a0 2024-01-11T09:26:38.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:38.953+0000 7fa504fff700 1 -- 172.21.15.27:0/994993716 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa50015c140 con 0x7fa50015a7a0 2024-01-11T09:26:39.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.672+0000 7fa4ee7fc700 1 -- 172.21.15.27:0/994993716 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v164) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa4f00551a0 con 0x7fa50015a7a0 2024-01-11T09:26:39.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.677+0000 7fa504fff700 1 -- 172.21.15.27:0/994993716 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa4e0066f30 msgr2=0x7fa4e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:39.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.677+0000 7fa504fff700 1 --2- 172.21.15.27:0/994993716 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa4e0066f30 0x7fa4e00693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4006f00 tx=0x7fa4f4008040 comp rx=0 tx=0).stop 2024-01-11T09:26:39.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.677+0000 7fa504fff700 1 -- 172.21.15.27:0/994993716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa50015a7a0 msgr2=0x7fa500151ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:39.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.677+0000 7fa504fff700 1 --2- 172.21.15.27:0/994993716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa50015a7a0 0x7fa500151ce0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fa4f000ec80 tx=0x7fa4f00091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:39.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.678+0000 7fa504fff700 1 -- 172.21.15.27:0/994993716 shutdown_connections 2024-01-11T09:26:39.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.678+0000 7fa504fff700 1 --2- 172.21.15.27:0/994993716 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa4e0066f30 0x7fa4e00693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:39.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.678+0000 7fa504fff700 1 --2- 172.21.15.27:0/994993716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500150df0 0x7fa50014cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:39.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.678+0000 7fa504fff700 1 --2- 172.21.15.27:0/994993716 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa50015a7a0 0x7fa500151ce0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:39.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.678+0000 7fa504fff700 1 --2- 172.21.15.27:0/994993716 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa50015b400 0x7fa50016aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:39.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.678+0000 7fa504fff700 1 -- 172.21.15.27:0/994993716 >> 172.21.15.27:0/994993716 conn(0x7fa50006ed40 msgr2=0x7fa50014d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:39.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.679+0000 7fa504fff700 1 -- 172.21.15.27:0/994993716 shutdown_connections 2024-01-11T09:26:39.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.680+0000 7fa504fff700 1 -- 172.21.15.27:0/994993716 wait complete. 2024-01-11T09:26:39.678 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-11T09:26:39.698 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-11T09:26:39.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.910+0000 7f25f2dff700 1 Processor -- start 2024-01-11T09:26:39.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.910+0000 7f25f2dff700 1 -- start start 2024-01-11T09:26:39.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.912+0000 7f25f2dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f25ec153e90 0x7f25ec0accc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:39.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.912+0000 7f25f2dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f25ec0b0e30 0x7f25ec0af950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:39.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.912+0000 7f25f2dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ec0af4c0 con 0x7f25ec069d40 2024-01-11T09:26:39.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.912+0000 7f25f2dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ec1574d0 con 0x7f25ec153e90 2024-01-11T09:26:39.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.912+0000 7f25f2dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ec1577b0 con 0x7f25ec0b0e30 2024-01-11T09:26:39.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.912+0000 7f25f0b9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f25ec153e90 0x7f25ec0accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:39.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.912+0000 7f25f0b9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f25ec153e90 0x7f25ec0accc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60764/0 (socket says 172.21.15.27:60764) 2024-01-11T09:26:39.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.912+0000 7f25f0b9b700 1 -- 172.21.15.27:0/1623329304 learned_addr learned my addr 172.21.15.27:0/1623329304 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:39.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.912+0000 7f25f139c700 1 --2- 172.21.15.27:0/1623329304 >> v2:172.21.15.77:3300/0 conn(0x7f25ec0b0e30 0x7f25ec0af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:39.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.913+0000 7f25f0b9b700 1 -- 172.21.15.27:0/1623329304 >> v2:172.21.15.77:3300/0 conn(0x7f25ec0b0e30 msgr2=0x7f25ec0af950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:39.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.913+0000 7f25f0b9b700 1 --2- 172.21.15.27:0/1623329304 >> v2:172.21.15.77:3300/0 conn(0x7f25ec0b0e30 0x7f25ec0af950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:39.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.913+0000 7f25f0b9b700 1 -- 172.21.15.27:0/1623329304 >> v1:172.21.15.77:6789/0 conn(0x7f25ec069d40 legacy=0x7f25ec0b0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:39.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.913+0000 7f25f0b9b700 1 -- 172.21.15.27:0/1623329304 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec0af150 con 0x7f25ec153e90 2024-01-11T09:26:39.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.913+0000 7f25f0b9b700 1 --2- 172.21.15.27:0/1623329304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec153e90 0x7f25ec0accc0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f25e00099b0 tx=0x7f25e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d0ef7d5af4d4eb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:39.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.913+0000 7f25eb7fe700 1 -- 172.21.15.27:0/1623329304 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e000b820 con 0x7f25ec153e90 2024-01-11T09:26:39.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.913+0000 7f25eb7fe700 1 -- 172.21.15.27:0/1623329304 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e000b980 con 0x7f25ec153e90 2024-01-11T09:26:39.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.913+0000 7f25eb7fe700 1 -- 172.21.15.27:0/1623329304 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e000bd20 con 0x7f25ec153e90 2024-01-11T09:26:39.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.914+0000 7f25f2dff700 1 -- 172.21.15.27:0/1623329304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec153e90 msgr2=0x7f25ec0accc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:39.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.914+0000 7f25f2dff700 1 --2- 172.21.15.27:0/1623329304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec153e90 0x7f25ec0accc0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f25e00099b0 tx=0x7f25e0009250 comp rx=0 tx=0).stop 2024-01-11T09:26:39.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.914+0000 7f25f2dff700 1 -- 172.21.15.27:0/1623329304 shutdown_connections 2024-01-11T09:26:39.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.914+0000 7f25f2dff700 1 --2- 172.21.15.27:0/1623329304 >> v2:172.21.15.77:3300/0 conn(0x7f25ec0b0e30 0x7f25ec0af950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:39.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.914+0000 7f25f2dff700 1 --2- 172.21.15.27:0/1623329304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec153e90 0x7f25ec0accc0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:39.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.914+0000 7f25f2dff700 1 -- 172.21.15.27:0/1623329304 >> 172.21.15.27:0/1623329304 conn(0x7f25ec0680a0 msgr2=0x7f25ec069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:39.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.914+0000 7f25f2dff700 1 -- 172.21.15.27:0/1623329304 shutdown_connections 2024-01-11T09:26:39.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.914+0000 7f25f2dff700 1 -- 172.21.15.27:0/1623329304 wait complete. 2024-01-11T09:26:39.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.916+0000 7f25f2dff700 1 Processor -- start 2024-01-11T09:26:39.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.916+0000 7f25f2dff700 1 -- start start 2024-01-11T09:26:39.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.917+0000 7f25f2dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec069d40 0x7f25ec1521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:39.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.917+0000 7f25f2dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f25ec0b0e30 0x7f25ec15d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:39.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.917+0000 7f25f2dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25ec153e90 0x7f25ec15dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:39.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.917+0000 7f25f2dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0c9960 con 0x7f25ec069d40 2024-01-11T09:26:39.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.917+0000 7f25f2dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0c9330 con 0x7f25ec153e90 2024-01-11T09:26:39.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.917+0000 7f25f2dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec1603c0 con 0x7f25ec0b0e30 2024-01-11T09:26:39.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.917+0000 7f25f0b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec069d40 0x7f25ec1521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:39.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.917+0000 7f25f0b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec069d40 0x7f25ec1521b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60768/0 (socket says 172.21.15.27:60768) 2024-01-11T09:26:39.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.917+0000 7f25f0b9b700 1 -- 172.21.15.27:0/2951989113 learned_addr learned my addr 172.21.15.27:0/2951989113 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:39.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.917+0000 7f25f139c700 1 --2- 172.21.15.27:0/2951989113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25ec153e90 0x7f25ec15dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:39.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.917+0000 7f25ebfff700 1 --2- 172.21.15.27:0/2951989113 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f25ec0b0e30 0x7f25ec15d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:39.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.918+0000 7f25f0b9b700 1 -- 172.21.15.27:0/2951989113 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f25ec0b0e30 msgr2=0x7f25ec15d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:39.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.918+0000 7f25f0b9b700 1 --2- 172.21.15.27:0/2951989113 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f25ec0b0e30 0x7f25ec15d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:39.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.918+0000 7f25f0b9b700 1 -- 172.21.15.27:0/2951989113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25ec153e90 msgr2=0x7f25ec15dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:39.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.918+0000 7f25f0b9b700 1 --2- 172.21.15.27:0/2951989113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25ec153e90 0x7f25ec15dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:39.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.918+0000 7f25f0b9b700 1 -- 172.21.15.27:0/2951989113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e00094f0 con 0x7f25ec069d40 2024-01-11T09:26:39.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.918+0000 7f25f0b9b700 1 --2- 172.21.15.27:0/2951989113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec069d40 0x7f25ec1521b0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f25e00094c0 tx=0x7f25e0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:39.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.918+0000 7f25e8ff9700 1 -- 172.21.15.27:0/2951989113 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e000bd00 con 0x7f25ec069d40 2024-01-11T09:26:39.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.918+0000 7f25e8ff9700 1 -- 172.21.15.27:0/2951989113 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e0024440 con 0x7f25ec069d40 2024-01-11T09:26:39.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.918+0000 7f25e8ff9700 1 -- 172.21.15.27:0/2951989113 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e0024800 con 0x7f25ec069d40 2024-01-11T09:26:39.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.918+0000 7f25f2dff700 1 -- 172.21.15.27:0/2951989113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ec0c95f0 con 0x7f25ec069d40 2024-01-11T09:26:39.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.918+0000 7f25f2dff700 1 -- 172.21.15.27:0/2951989113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ec157d40 con 0x7f25ec069d40 2024-01-11T09:26:39.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.919+0000 7f25f2dff700 1 -- 172.21.15.27:0/2951989113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ec1606a0 con 0x7f25ec069d40 2024-01-11T09:26:39.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.921+0000 7f25e8ff9700 1 -- 172.21.15.27:0/2951989113 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f25e002d440 con 0x7f25ec069d40 2024-01-11T09:26:39.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.922+0000 7f25e8ff9700 1 --2- 172.21.15.27:0/2951989113 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f25d0066f30 0x7f25d00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:39.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.922+0000 7f25e8ff9700 1 -- 172.21.15.27:0/2951989113 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5279+0+0 (secure 0 0 0) 0x7f25e0085d10 con 0x7f25ec069d40 2024-01-11T09:26:39.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.922+0000 7f25ebfff700 1 --2- 172.21.15.27:0/2951989113 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f25d0066f30 0x7f25d00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:39.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.923+0000 7f25ebfff700 1 --2- 172.21.15.27:0/2951989113 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f25d0066f30 0x7f25d00693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f25ec06a1a0 tx=0x7f25d8007ec0 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:39.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:39.927+0000 7f25e8ff9700 1 -- 172.21.15.27:0/2951989113 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f25e008b020 con 0x7f25ec069d40 2024-01-11T09:26:40.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.131+0000 7f25f2dff700 1 -- 172.21.15.27:0/2951989113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f25ec04c3c0 con 0x7f25ec069d40 2024-01-11T09:26:40.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.685+0000 7f25e8ff9700 1 -- 172.21.15.27:0/2951989113 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v165) v1 ==== 170+0+0 (secure 0 0 0) 0x7f25e0053090 con 0x7f25ec069d40 2024-01-11T09:26:40.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.689+0000 7f25f2dff700 1 -- 172.21.15.27:0/2951989113 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f25d0066f30 msgr2=0x7f25d00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:40.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.689+0000 7f25f2dff700 1 --2- 172.21.15.27:0/2951989113 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f25d0066f30 0x7f25d00693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f25ec06a1a0 tx=0x7f25d8007ec0 comp rx=0 tx=0).stop 2024-01-11T09:26:40.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.689+0000 7f25f2dff700 1 -- 172.21.15.27:0/2951989113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec069d40 msgr2=0x7f25ec1521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:40.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.689+0000 7f25f2dff700 1 --2- 172.21.15.27:0/2951989113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec069d40 0x7f25ec1521b0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f25e00094c0 tx=0x7f25e0017000 comp rx=0 tx=0).stop 2024-01-11T09:26:40.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.689+0000 7f25f2dff700 1 -- 172.21.15.27:0/2951989113 shutdown_connections 2024-01-11T09:26:40.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.689+0000 7f25f2dff700 1 --2- 172.21.15.27:0/2951989113 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f25d0066f30 0x7f25d00693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:40.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.689+0000 7f25f2dff700 1 --2- 172.21.15.27:0/2951989113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25ec153e90 0x7f25ec15dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:40.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.689+0000 7f25f2dff700 1 --2- 172.21.15.27:0/2951989113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25ec069d40 0x7f25ec1521b0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:40.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.689+0000 7f25f2dff700 1 --2- 172.21.15.27:0/2951989113 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f25ec0b0e30 0x7f25ec15d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:40.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.689+0000 7f25f2dff700 1 -- 172.21.15.27:0/2951989113 >> 172.21.15.27:0/2951989113 conn(0x7f25ec0680a0 msgr2=0x7f25ec0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:40.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.689+0000 7f25f2dff700 1 -- 172.21.15.27:0/2951989113 shutdown_connections 2024-01-11T09:26:40.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.689+0000 7f25f2dff700 1 -- 172.21.15.27:0/2951989113 wait complete. 2024-01-11T09:26:40.689 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-11T09:26:40.719 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-11T09:26:40.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.926+0000 7f26e5dff700 1 Processor -- start 2024-01-11T09:26:40.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.926+0000 7f26e5dff700 1 -- start start 2024-01-11T09:26:40.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.927+0000 7f26e5dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f26e015a1b0 0x7f26e015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:40.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.927+0000 7f26e5dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26e00ccd70 0x7f26e00cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:40.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.927+0000 7f26e5dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e00cf850 con 0x7f26e00685c0 2024-01-11T09:26:40.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.927+0000 7f26e5dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e00cec40 con 0x7f26e015a1b0 2024-01-11T09:26:40.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.927+0000 7f26e5dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e00cef20 con 0x7f26e00ccd70 2024-01-11T09:26:40.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.927+0000 7f26deffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26e00ccd70 0x7f26e00cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:40.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.928+0000 7f26df7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f26e015a1b0 0x7f26e015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:40.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.928+0000 7f26dffff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f26e00685c0 0x7f26e0068a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:46078/0 (socket says 172.21.15.27:46078) 2024-01-11T09:26:40.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.928+0000 7f26dffff700 1 -- 172.21.15.27:0/2976182726 learned_addr learned my addr 172.21.15.27:0/2976182726 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:40.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.928+0000 7f26de7fc700 1 -- 172.21.15.27:0/2976182726 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954916314 0 0) 0x7f26e00cf850 con 0x7f26e00685c0 2024-01-11T09:26:40.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.928+0000 7f26de7fc700 1 -- 172.21.15.27:0/2976182726 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c8003570 con 0x7f26e00685c0 2024-01-11T09:26:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.928+0000 7f26df7fe700 1 -- 172.21.15.27:0/2976182726 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26e00ccd70 msgr2=0x7f26e00cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.928+0000 7f26df7fe700 1 --2- 172.21.15.27:0/2976182726 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26e00ccd70 0x7f26e00cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.928+0000 7f26df7fe700 1 -- 172.21.15.27:0/2976182726 >> v1:172.21.15.77:6789/0 conn(0x7f26e00685c0 legacy=0x7f26e0068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.928+0000 7f26df7fe700 1 -- 172.21.15.27:0/2976182726 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e00cf200 con 0x7f26e015a1b0 2024-01-11T09:26:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.928+0000 7f26deffd700 1 --2- 172.21.15.27:0/2976182726 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26e00ccd70 0x7f26e00cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:26:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.928+0000 7f26df7fe700 1 --2- 172.21.15.27:0/2976182726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26e015a1b0 0x7f26e015a580 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f26d00098e0 tx=0x7f26d00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccc48746c20b9068 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.929+0000 7f26de7fc700 1 -- 172.21.15.27:0/2976182726 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0011630 con 0x7f26e015a1b0 2024-01-11T09:26:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.929+0000 7f26de7fc700 1 -- 172.21.15.27:0/2976182726 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d0011790 con 0x7f26e015a1b0 2024-01-11T09:26:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.929+0000 7f26de7fc700 1 -- 172.21.15.27:0/2976182726 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0011b30 con 0x7f26e015a1b0 2024-01-11T09:26:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.929+0000 7f26e5dff700 1 -- 172.21.15.27:0/2976182726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26e015a1b0 msgr2=0x7f26e015a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.929+0000 7f26e5dff700 1 --2- 172.21.15.27:0/2976182726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26e015a1b0 0x7f26e015a580 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f26d00098e0 tx=0x7f26d00091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.929+0000 7f26e5dff700 1 -- 172.21.15.27:0/2976182726 shutdown_connections 2024-01-11T09:26:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.929+0000 7f26e5dff700 1 --2- 172.21.15.27:0/2976182726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26e015a1b0 0x7f26e015a580 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:40.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.929+0000 7f26e5dff700 1 --2- 172.21.15.27:0/2976182726 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26e00ccd70 0x7f26e00cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:40.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.929+0000 7f26e5dff700 1 -- 172.21.15.27:0/2976182726 >> 172.21.15.27:0/2976182726 conn(0x7f26e006ed40 msgr2=0x7f26e006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:40.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.929+0000 7f26e5dff700 1 -- 172.21.15.27:0/2976182726 shutdown_connections 2024-01-11T09:26:40.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.929+0000 7f26e5dff700 1 -- 172.21.15.27:0/2976182726 wait complete. 2024-01-11T09:26:40.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.931+0000 7f26e5dff700 1 Processor -- start 2024-01-11T09:26:40.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.931+0000 7f26e5dff700 1 -- start start 2024-01-11T09:26:40.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.932+0000 7f26e5dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e00685c0 0x7f26e00af6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:40.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.932+0000 7f26e5dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26e00ccd70 0x7f26e0169920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:40.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.932+0000 7f26e5dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26e015a1b0 0x7f26e00c6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:40.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.932+0000 7f26e5dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e016af00 con 0x7f26e00685c0 2024-01-11T09:26:40.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.932+0000 7f26e5dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e016a250 con 0x7f26e015a1b0 2024-01-11T09:26:40.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.932+0000 7f26e5dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e016a530 con 0x7f26e00ccd70 2024-01-11T09:26:40.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.932+0000 7f26df7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e00685c0 0x7f26e00af6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:40.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.932+0000 7f26df7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e00685c0 0x7f26e00af6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60778/0 (socket says 172.21.15.27:60778) 2024-01-11T09:26:40.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.932+0000 7f26df7fe700 1 -- 172.21.15.27:0/2842637701 learned_addr learned my addr 172.21.15.27:0/2842637701 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:40.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.933+0000 7f26dffff700 1 --2- 172.21.15.27:0/2842637701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26e015a1b0 0x7f26e00c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:40.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.933+0000 7f26deffd700 1 --2- 172.21.15.27:0/2842637701 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26e00ccd70 0x7f26e0169920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:40.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.933+0000 7f26df7fe700 1 -- 172.21.15.27:0/2842637701 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26e00ccd70 msgr2=0x7f26e0169920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.933+0000 7f26df7fe700 1 --2- 172.21.15.27:0/2842637701 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26e00ccd70 0x7f26e0169920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.933+0000 7f26df7fe700 1 -- 172.21.15.27:0/2842637701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26e015a1b0 msgr2=0x7f26e00c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.933+0000 7f26df7fe700 1 --2- 172.21.15.27:0/2842637701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26e015a1b0 0x7f26e00c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.933+0000 7f26df7fe700 1 -- 172.21.15.27:0/2842637701 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d4009420 con 0x7f26e00685c0 2024-01-11T09:26:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.933+0000 7f26df7fe700 1 --2- 172.21.15.27:0/2842637701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e00685c0 0x7f26e00af6d0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f26e0069770 tx=0x7f26d0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.933+0000 7f26c77fe700 1 -- 172.21.15.27:0/2842637701 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0011b10 con 0x7f26e00685c0 2024-01-11T09:26:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.933+0000 7f26c77fe700 1 -- 172.21.15.27:0/2842637701 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d00253f0 con 0x7f26e00685c0 2024-01-11T09:26:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.934+0000 7f26e5dff700 1 -- 172.21.15.27:0/2842637701 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d0009420 con 0x7f26e00685c0 2024-01-11T09:26:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.934+0000 7f26c77fe700 1 -- 172.21.15.27:0/2842637701 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d00257b0 con 0x7f26e00685c0 2024-01-11T09:26:40.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.934+0000 7f26e5dff700 1 -- 172.21.15.27:0/2842637701 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e016ab90 con 0x7f26e00685c0 2024-01-11T09:26:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.935+0000 7f26e5dff700 1 -- 172.21.15.27:0/2842637701 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e004c3c0 con 0x7f26e00685c0 2024-01-11T09:26:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.936+0000 7f26c77fe700 1 -- 172.21.15.27:0/2842637701 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f26d0025ba0 con 0x7f26e00685c0 2024-01-11T09:26:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.937+0000 7f26c77fe700 1 --2- 172.21.15.27:0/2842637701 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f26cc066f30 0x7f26cc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.937+0000 7f26c77fe700 1 -- 172.21.15.27:0/2842637701 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5661+0+0 (secure 0 0 0) 0x7f26d0085e10 con 0x7f26e00685c0 2024-01-11T09:26:40.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.937+0000 7f26deffd700 1 --2- 172.21.15.27:0/2842637701 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f26cc066f30 0x7f26cc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:40.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.938+0000 7f26deffd700 1 --2- 172.21.15.27:0/2842637701 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f26cc066f30 0x7f26cc0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f26e00c9b00 tx=0x7f26d4008040 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:40.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:40.942+0000 7f26c77fe700 1 -- 172.21.15.27:0/2842637701 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f26d0089030 con 0x7f26e00685c0 2024-01-11T09:26:41.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.188+0000 7f26e5dff700 1 -- 172.21.15.27:0/2842637701 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f26e00ce4a0 con 0x7f26e00685c0 2024-01-11T09:26:41.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.686+0000 7f26c77fe700 1 -- 172.21.15.27:0/2842637701 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v82) v1 ==== 174+0+0 (secure 0 0 0) 0x7f26d0053010 con 0x7f26e00685c0 2024-01-11T09:26:41.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.691+0000 7f26e5dff700 1 -- 172.21.15.27:0/2842637701 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f26cc066f30 msgr2=0x7f26cc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:41.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.691+0000 7f26e5dff700 1 --2- 172.21.15.27:0/2842637701 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f26cc066f30 0x7f26cc0693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f26e00c9b00 tx=0x7f26d4008040 comp rx=0 tx=0).stop 2024-01-11T09:26:41.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.691+0000 7f26e5dff700 1 -- 172.21.15.27:0/2842637701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e00685c0 msgr2=0x7f26e00af6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:41.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.691+0000 7f26e5dff700 1 --2- 172.21.15.27:0/2842637701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e00685c0 0x7f26e00af6d0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f26e0069770 tx=0x7f26d0017000 comp rx=0 tx=0).stop 2024-01-11T09:26:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.692+0000 7f26e5dff700 1 -- 172.21.15.27:0/2842637701 shutdown_connections 2024-01-11T09:26:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.692+0000 7f26e5dff700 1 --2- 172.21.15.27:0/2842637701 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f26cc066f30 0x7f26cc0693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.692+0000 7f26e5dff700 1 --2- 172.21.15.27:0/2842637701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26e015a1b0 0x7f26e00c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.692+0000 7f26e5dff700 1 --2- 172.21.15.27:0/2842637701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e00685c0 0x7f26e00af6d0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.692+0000 7f26e5dff700 1 --2- 172.21.15.27:0/2842637701 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26e00ccd70 0x7f26e0169920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.692+0000 7f26e5dff700 1 -- 172.21.15.27:0/2842637701 >> 172.21.15.27:0/2842637701 conn(0x7f26e006ed40 msgr2=0x7f26e00c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:41.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.693+0000 7f26e5dff700 1 -- 172.21.15.27:0/2842637701 shutdown_connections 2024-01-11T09:26:41.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.694+0000 7f26e5dff700 1 -- 172.21.15.27:0/2842637701 wait complete. 2024-01-11T09:26:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 20 and data pool 21 2024-01-11T09:26:41.720 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:26:41.725 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:26:41.731 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:26:41.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.976+0000 7f3f30dff700 1 Processor -- start 2024-01-11T09:26:41.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.976+0000 7f3f30dff700 1 -- start start 2024-01-11T09:26:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.977+0000 7f3f30dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f2c0b2b90 0x7f3f2c159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.977+0000 7f3f30dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f2c0b07b0 0x7f3f2c159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.977+0000 7f3f30dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2c0cd730 con 0x7f3f2c0b03e0 2024-01-11T09:26:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.977+0000 7f3f30dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f2c0ccb20 con 0x7f3f2c0b2b90 2024-01-11T09:26:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.977+0000 7f3f30dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c0cce00 con 0x7f3f2c0b07b0 2024-01-11T09:26:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.978+0000 7f3f29d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f2c0b2b90 0x7f3f2c159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.978+0000 7f3f29d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f2c0b2b90 0x7f3f2c159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60794/0 (socket says 172.21.15.27:60794) 2024-01-11T09:26:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.978+0000 7f3f29d9b700 1 -- 172.21.15.27:0/3344533106 learned_addr learned my addr 172.21.15.27:0/3344533106 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.978+0000 7f3f2ad9d700 1 --2- 172.21.15.27:0/3344533106 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f2c0b07b0 0x7f3f2c159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.978+0000 7f3f29d9b700 1 -- 172.21.15.27:0/3344533106 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f2c0b07b0 msgr2=0x7f3f2c159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.978+0000 7f3f29d9b700 1 --2- 172.21.15.27:0/3344533106 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f2c0b07b0 0x7f3f2c159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:41.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.978+0000 7f3f29d9b700 1 -- 172.21.15.27:0/3344533106 >> v1:172.21.15.77:6789/0 conn(0x7f3f2c0b03e0 legacy=0x7f3f2c0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.978+0000 7f3f29d9b700 1 -- 172.21.15.27:0/3344533106 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2c0cd0e0 con 0x7f3f2c0b2b90 2024-01-11T09:26:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.978+0000 7f3f2ad9d700 1 --2- 172.21.15.27:0/3344533106 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f2c0b07b0 0x7f3f2c159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:41.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.978+0000 7f3f29d9b700 1 --2- 172.21.15.27:0/3344533106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f2c0b2b90 0x7f3f2c159200 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f3f200099b0 tx=0x7f3f20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc07e346e398dc2e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:41.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.979+0000 7f3f2959a700 1 -- 172.21.15.27:0/3344533106 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2000b860 con 0x7f3f2c0b2b90 2024-01-11T09:26:41.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.979+0000 7f3f2959a700 1 -- 172.21.15.27:0/3344533106 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2000b9c0 con 0x7f3f2c0b2b90 2024-01-11T09:26:41.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.979+0000 7f3f2959a700 1 -- 172.21.15.27:0/3344533106 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2000bd60 con 0x7f3f2c0b2b90 2024-01-11T09:26:41.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.979+0000 7f3f30dff700 1 -- 172.21.15.27:0/3344533106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f2c0b2b90 msgr2=0x7f3f2c159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:41.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.979+0000 7f3f30dff700 1 --2- 172.21.15.27:0/3344533106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f2c0b2b90 0x7f3f2c159200 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f3f200099b0 tx=0x7f3f20009250 comp rx=0 tx=0).stop 2024-01-11T09:26:41.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.979+0000 7f3f30dff700 1 -- 172.21.15.27:0/3344533106 shutdown_connections 2024-01-11T09:26:41.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.979+0000 7f3f30dff700 1 --2- 172.21.15.27:0/3344533106 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f2c0b07b0 0x7f3f2c159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:41.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.979+0000 7f3f30dff700 1 --2- 172.21.15.27:0/3344533106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f2c0b2b90 0x7f3f2c159200 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:41.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.979+0000 7f3f30dff700 1 -- 172.21.15.27:0/3344533106 >> 172.21.15.27:0/3344533106 conn(0x7f3f2c06ed40 msgr2=0x7f3f2c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:41.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.980+0000 7f3f30dff700 1 -- 172.21.15.27:0/3344533106 shutdown_connections 2024-01-11T09:26:41.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.980+0000 7f3f30dff700 1 -- 172.21.15.27:0/3344533106 wait complete. 2024-01-11T09:26:41.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.981+0000 7f3f30dff700 1 Processor -- start 2024-01-11T09:26:41.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.981+0000 7f3f30dff700 1 -- start start 2024-01-11T09:26:41.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.982+0000 7f3f30dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f2c0b03e0 0x7f3f2c15bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:41.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.982+0000 7f3f30dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f2c0b07b0 0x7f3f2c0c9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:41.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.982+0000 7f3f30dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f2c0b2b90 0x7f3f2c0c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:41.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.982+0000 7f3f30dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c0c9d50 con 0x7f3f2c0b07b0 2024-01-11T09:26:41.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.982+0000 7f3f30dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c159020 con 0x7f3f2c0b03e0 2024-01-11T09:26:41.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.982+0000 7f3f30dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c16cd30 con 0x7f3f2c0b2b90 2024-01-11T09:26:41.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.982+0000 7f3f29d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f2c0b07b0 0x7f3f2c0c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:41.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.983+0000 7f3f2ad9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f2c0b2b90 0x7f3f2c0c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:41.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.983+0000 7f3f29d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f2c0b07b0 0x7f3f2c0c9140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60804/0 (socket says 172.21.15.27:60804) 2024-01-11T09:26:41.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.983+0000 7f3f29d9b700 1 -- 172.21.15.27:0/3521603835 learned_addr learned my addr 172.21.15.27:0/3521603835 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:41.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.983+0000 7f3f2a59c700 1 --2- 172.21.15.27:0/3521603835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f2c0b03e0 0x7f3f2c15bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:41.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.983+0000 7f3f29d9b700 1 -- 172.21.15.27:0/3521603835 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f2c0b2b90 msgr2=0x7f3f2c0c9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:41.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.983+0000 7f3f29d9b700 1 --2- 172.21.15.27:0/3521603835 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f2c0b2b90 0x7f3f2c0c9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:41.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.983+0000 7f3f29d9b700 1 -- 172.21.15.27:0/3521603835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f2c0b03e0 msgr2=0x7f3f2c15bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:41.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.983+0000 7f3f29d9b700 1 --2- 172.21.15.27:0/3521603835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f2c0b03e0 0x7f3f2c15bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:41.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.983+0000 7f3f29d9b700 1 -- 172.21.15.27:0/3521603835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f200094f0 con 0x7f3f2c0b07b0 2024-01-11T09:26:41.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.983+0000 7f3f29d9b700 1 --2- 172.21.15.27:0/3521603835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f2c0b07b0 0x7f3f2c0c9140 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f3f200090b0 tx=0x7f3f20017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:41.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.983+0000 7f3f1a7fc700 1 -- 172.21.15.27:0/3521603835 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f20011dc0 con 0x7f3f2c0b07b0 2024-01-11T09:26:41.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.984+0000 7f3f1a7fc700 1 -- 172.21.15.27:0/3521603835 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2001f070 con 0x7f3f2c0b07b0 2024-01-11T09:26:41.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.984+0000 7f3f1a7fc700 1 -- 172.21.15.27:0/3521603835 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f200266d0 con 0x7f3f2c0b07b0 2024-01-11T09:26:41.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.984+0000 7f3f30dff700 1 -- 172.21.15.27:0/3521603835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2c16d010 con 0x7f3f2c0b07b0 2024-01-11T09:26:41.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.984+0000 7f3f30dff700 1 -- 172.21.15.27:0/3521603835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2c16d5d0 con 0x7f3f2c0b07b0 2024-01-11T09:26:41.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.985+0000 7f3f30dff700 1 -- 172.21.15.27:0/3521603835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f2c16d810 con 0x7f3f2c0b07b0 2024-01-11T09:26:41.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.986+0000 7f3f1a7fc700 1 -- 172.21.15.27:0/3521603835 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3f20026a40 con 0x7f3f2c0b07b0 2024-01-11T09:26:41.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.987+0000 7f3f1a7fc700 1 --2- 172.21.15.27:0/3521603835 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f3f0c066f30 0x7f3f0c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:41.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.987+0000 7f3f1a7fc700 1 -- 172.21.15.27:0/3521603835 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f3f20021070 con 0x7f3f2c0b07b0 2024-01-11T09:26:41.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.988+0000 7f3f2a59c700 1 --2- 172.21.15.27:0/3521603835 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f3f0c066f30 0x7f3f0c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:41.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.988+0000 7f3f2a59c700 1 --2- 172.21.15.27:0/3521603835 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f3f0c066f30 0x7f3f0c0693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c003a80 tx=0x7f3f1c007ec0 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:41.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:41.991+0000 7f3f1a7fc700 1 -- 172.21.15.27:0/3521603835 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3f2c16d810 con 0x7f3f2c0b07b0 2024-01-11T09:26:42.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.196+0000 7f3f30dff700 1 -- 172.21.15.27:0/3521603835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f2c16d810 con 0x7f3f2c0b07b0 2024-01-11T09:26:42.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.197+0000 7f3f1a7fc700 1 -- 172.21.15.27:0/3521603835 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v166) v1 ==== 74+0+15417 (secure 0 0 0) 0x7f3f2c16d810 con 0x7f3f2c0b07b0 2024-01-11T09:26:42.194 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:26:42.194 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":166,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:26:41.190128+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:26:39.008224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-01-11T09:26:40.186962+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:26:42.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.200+0000 7f3f30dff700 1 -- 172.21.15.27:0/3521603835 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f3f0c066f30 msgr2=0x7f3f0c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:42.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.200+0000 7f3f30dff700 1 --2- 172.21.15.27:0/3521603835 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f3f0c066f30 0x7f3f0c0693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c003a80 tx=0x7f3f1c007ec0 comp rx=0 tx=0).stop 2024-01-11T09:26:42.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.200+0000 7f3f30dff700 1 -- 172.21.15.27:0/3521603835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f2c0b07b0 msgr2=0x7f3f2c0c9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:42.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.200+0000 7f3f30dff700 1 --2- 172.21.15.27:0/3521603835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f2c0b07b0 0x7f3f2c0c9140 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f3f200090b0 tx=0x7f3f20017000 comp rx=0 tx=0).stop 2024-01-11T09:26:42.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.200+0000 7f3f30dff700 1 -- 172.21.15.27:0/3521603835 shutdown_connections 2024-01-11T09:26:42.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.200+0000 7f3f30dff700 1 --2- 172.21.15.27:0/3521603835 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f3f0c066f30 0x7f3f0c0693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:42.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.200+0000 7f3f30dff700 1 --2- 172.21.15.27:0/3521603835 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3f2c0b2b90 0x7f3f2c0c9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:42.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.200+0000 7f3f30dff700 1 --2- 172.21.15.27:0/3521603835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f2c0b03e0 0x7f3f2c15bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:42.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.200+0000 7f3f30dff700 1 --2- 172.21.15.27:0/3521603835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f2c0b07b0 0x7f3f2c0c9140 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:42.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.200+0000 7f3f30dff700 1 -- 172.21.15.27:0/3521603835 >> 172.21.15.27:0/3521603835 conn(0x7f3f2c06ed40 msgr2=0x7f3f2c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:42.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.200+0000 7f3f30dff700 1 -- 172.21.15.27:0/3521603835 shutdown_connections 2024-01-11T09:26:42.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.200+0000 7f3f30dff700 1 -- 172.21.15.27:0/3521603835 wait complete. 2024-01-11T09:26:42.215 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:26:42.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.417+0000 7fa6019ff700 1 Processor -- start 2024-01-11T09:26:42.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.417+0000 7fa6019ff700 1 -- start start 2024-01-11T09:26:42.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.418+0000 7fa6019ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa5fc0ca3e0 0x7fa5fc0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:42.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.418+0000 7fa6019ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5fc0ca7b0 0x7fa5fc159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:42.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.418+0000 7fa6019ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc15d640 con 0x7fa5fc0c8a20 2024-01-11T09:26:42.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.418+0000 7fa6019ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5fc15ae70 con 0x7fa5fc0ca7b0 2024-01-11T09:26:42.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.418+0000 7fa6019ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5fc15a3c0 con 0x7fa5fc0ca3e0 2024-01-11T09:26:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.419+0000 7fa5fa7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5fc0ca7b0 0x7fa5fc159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.419+0000 7fa5fb7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa5fc0c8a20 0x7fa5fc15dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36192/0 (socket says 172.21.15.27:36192) 2024-01-11T09:26:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.419+0000 7fa5fb7fe700 1 -- 172.21.15.27:0/993532461 learned_addr learned my addr 172.21.15.27:0/993532461 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.419+0000 7fa5faffd700 1 --2- 172.21.15.27:0/993532461 >> v2:172.21.15.77:3300/0 conn(0x7fa5fc0ca3e0 0x7fa5fc0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.419+0000 7fa5fa7fc700 1 -- 172.21.15.27:0/993532461 >> v2:172.21.15.77:3300/0 conn(0x7fa5fc0ca3e0 msgr2=0x7fa5fc0cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.419+0000 7fa5fa7fc700 1 --2- 172.21.15.27:0/993532461 >> v2:172.21.15.77:3300/0 conn(0x7fa5fc0ca3e0 0x7fa5fc0cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.419+0000 7fa5fa7fc700 1 -- 172.21.15.27:0/993532461 >> v1:172.21.15.27:6789/0 conn(0x7fa5fc0c8a20 legacy=0x7fa5fc15dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.419+0000 7fa5fa7fc700 1 -- 172.21.15.27:0/993532461 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc15a5d0 con 0x7fa5fc0ca7b0 2024-01-11T09:26:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.419+0000 7fa5fa7fc700 1 --2- 172.21.15.27:0/993532461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc0ca7b0 0x7fa5fc159e80 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00098e0 tx=0x7fa5f00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c1459d9e8007a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.420+0000 7fa5f9ffb700 1 -- 172.21.15.27:0/993532461 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0011630 con 0x7fa5fc0ca7b0 2024-01-11T09:26:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.420+0000 7fa5f9ffb700 1 -- 172.21.15.27:0/993532461 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f0011790 con 0x7fa5fc0ca7b0 2024-01-11T09:26:42.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.420+0000 7fa5f9ffb700 1 -- 172.21.15.27:0/993532461 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0011b30 con 0x7fa5fc0ca7b0 2024-01-11T09:26:42.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.420+0000 7fa6019ff700 1 -- 172.21.15.27:0/993532461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc0ca7b0 msgr2=0x7fa5fc159e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:42.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.420+0000 7fa6019ff700 1 --2- 172.21.15.27:0/993532461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc0ca7b0 0x7fa5fc159e80 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00098e0 tx=0x7fa5f00091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:42.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.420+0000 7fa6019ff700 1 -- 172.21.15.27:0/993532461 shutdown_connections 2024-01-11T09:26:42.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.420+0000 7fa6019ff700 1 --2- 172.21.15.27:0/993532461 >> v2:172.21.15.77:3300/0 conn(0x7fa5fc0ca3e0 0x7fa5fc0cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:42.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.420+0000 7fa6019ff700 1 --2- 172.21.15.27:0/993532461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc0ca7b0 0x7fa5fc159e80 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:42.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.420+0000 7fa6019ff700 1 -- 172.21.15.27:0/993532461 >> 172.21.15.27:0/993532461 conn(0x7fa5fc0ce3e0 msgr2=0x7fa5fc0ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:42.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.420+0000 7fa6019ff700 1 -- 172.21.15.27:0/993532461 shutdown_connections 2024-01-11T09:26:42.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.420+0000 7fa6019ff700 1 -- 172.21.15.27:0/993532461 wait complete. 2024-01-11T09:26:42.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.422+0000 7fa6019ff700 1 Processor -- start 2024-01-11T09:26:42.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.422+0000 7fa6019ff700 1 -- start start 2024-01-11T09:26:42.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.423+0000 7fa6019ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5fc0c8a20 0x7fa5fc0b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:42.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.423+0000 7fa6019ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5fc0ca3e0 0x7fa5fc14ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:42.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.423+0000 7fa6019ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc0ca7b0 0x7fa5fc14f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:42.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.423+0000 7fa6019ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc14f6c0 con 0x7fa5fc0ca7b0 2024-01-11T09:26:42.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.423+0000 7fa6019ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc151800 con 0x7fa5fc0c8a20 2024-01-11T09:26:42.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.423+0000 7fa6019ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc0c5c10 con 0x7fa5fc0ca3e0 2024-01-11T09:26:42.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.423+0000 7fa5fb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc0ca7b0 0x7fa5fc14f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:42.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.423+0000 7fa5fb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc0ca7b0 0x7fa5fc14f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60822/0 (socket says 172.21.15.27:60822) 2024-01-11T09:26:42.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.423+0000 7fa5fb7fe700 1 -- 172.21.15.27:0/1083795030 learned_addr learned my addr 172.21.15.27:0/1083795030 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:42.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.424+0000 7fa5fa7fc700 1 --2- 172.21.15.27:0/1083795030 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5fc0ca3e0 0x7fa5fc14ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:42.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.424+0000 7fa5faffd700 1 --2- 172.21.15.27:0/1083795030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5fc0c8a20 0x7fa5fc0b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:42.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.424+0000 7fa5fb7fe700 1 -- 172.21.15.27:0/1083795030 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5fc0ca3e0 msgr2=0x7fa5fc14ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:42.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.424+0000 7fa5fb7fe700 1 --2- 172.21.15.27:0/1083795030 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5fc0ca3e0 0x7fa5fc14ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:42.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.424+0000 7fa5fb7fe700 1 -- 172.21.15.27:0/1083795030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5fc0c8a20 msgr2=0x7fa5fc0b1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:42.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.424+0000 7fa5fb7fe700 1 --2- 172.21.15.27:0/1083795030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5fc0c8a20 0x7fa5fc0b1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:42.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.424+0000 7fa5fb7fe700 1 -- 172.21.15.27:0/1083795030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f0009420 con 0x7fa5fc0ca7b0 2024-01-11T09:26:42.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.424+0000 7fa5fa7fc700 1 --2- 172.21.15.27:0/1083795030 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5fc0ca3e0 0x7fa5fc14ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:26:42.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.424+0000 7fa5fb7fe700 1 --2- 172.21.15.27:0/1083795030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc0ca7b0 0x7fa5fc14f180 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fa5f400cc70 tx=0x7fa5f4001a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:42.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.424+0000 7fa5e2ffd700 1 -- 172.21.15.27:0/1083795030 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f400ce80 con 0x7fa5fc0ca7b0 2024-01-11T09:26:42.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.424+0000 7fa5e2ffd700 1 -- 172.21.15.27:0/1083795030 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f400ab30 con 0x7fa5fc0ca7b0 2024-01-11T09:26:42.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.424+0000 7fa6019ff700 1 -- 172.21.15.27:0/1083795030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5fc151520 con 0x7fa5fc0ca7b0 2024-01-11T09:26:42.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.424+0000 7fa5e2ffd700 1 -- 172.21.15.27:0/1083795030 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f40070f0 con 0x7fa5fc0ca7b0 2024-01-11T09:26:42.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.424+0000 7fa6019ff700 1 -- 172.21.15.27:0/1083795030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5fc0c58a0 con 0x7fa5fc0ca7b0 2024-01-11T09:26:42.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.425+0000 7fa6019ff700 1 -- 172.21.15.27:0/1083795030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5fc0c5da0 con 0x7fa5fc0ca7b0 2024-01-11T09:26:42.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.427+0000 7fa5e2ffd700 1 -- 172.21.15.27:0/1083795030 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa5f40074e0 con 0x7fa5fc0ca7b0 2024-01-11T09:26:42.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.427+0000 7fa5e2ffd700 1 --2- 172.21.15.27:0/1083795030 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa5e8066f30 0x7fa5e80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:42.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.428+0000 7fa5e2ffd700 1 -- 172.21.15.27:0/1083795030 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fa5f40849e0 con 0x7fa5fc0ca7b0 2024-01-11T09:26:42.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.428+0000 7fa5faffd700 1 --2- 172.21.15.27:0/1083795030 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa5e8066f30 0x7fa5e80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:42.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.429+0000 7fa5faffd700 1 --2- 172.21.15.27:0/1083795030 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa5e8066f30 0x7fa5e80693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec009d40 tx=0x7fa5ec008040 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:42.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.431+0000 7fa5e2ffd700 1 -- 172.21.15.27:0/1083795030 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa5f4051bf0 con 0x7fa5fc0ca7b0 2024-01-11T09:26:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.634+0000 7fa6019ff700 1 -- 172.21.15.27:0/1083795030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5fc0718d0 con 0x7fa5fc0ca7b0 2024-01-11T09:26:42.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.635+0000 7fa5e2ffd700 1 -- 172.21.15.27:0/1083795030 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v166) v1 ==== 74+0+15417 (secure 0 0 0) 0x7fa5f4055210 con 0x7fa5fc0ca7b0 2024-01-11T09:26:42.631 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:26:42.631 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":166,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:26:41.190128+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:26:39.008224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-01-11T09:26:40.186962+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:26:42.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.637+0000 7fa6019ff700 1 -- 172.21.15.27:0/1083795030 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa5e8066f30 msgr2=0x7fa5e80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:42.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.637+0000 7fa6019ff700 1 --2- 172.21.15.27:0/1083795030 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa5e8066f30 0x7fa5e80693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec009d40 tx=0x7fa5ec008040 comp rx=0 tx=0).stop 2024-01-11T09:26:42.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.637+0000 7fa6019ff700 1 -- 172.21.15.27:0/1083795030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc0ca7b0 msgr2=0x7fa5fc14f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:42.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.637+0000 7fa6019ff700 1 --2- 172.21.15.27:0/1083795030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc0ca7b0 0x7fa5fc14f180 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fa5f400cc70 tx=0x7fa5f4001a50 comp rx=0 tx=0).stop 2024-01-11T09:26:42.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.637+0000 7fa6019ff700 1 -- 172.21.15.27:0/1083795030 shutdown_connections 2024-01-11T09:26:42.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.637+0000 7fa6019ff700 1 --2- 172.21.15.27:0/1083795030 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa5e8066f30 0x7fa5e80693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:42.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.637+0000 7fa6019ff700 1 --2- 172.21.15.27:0/1083795030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc0ca7b0 0x7fa5fc14f180 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:42.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.637+0000 7fa6019ff700 1 --2- 172.21.15.27:0/1083795030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5fc0c8a20 0x7fa5fc0b1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:42.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.637+0000 7fa6019ff700 1 --2- 172.21.15.27:0/1083795030 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5fc0ca3e0 0x7fa5fc14ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:42.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.637+0000 7fa6019ff700 1 -- 172.21.15.27:0/1083795030 >> 172.21.15.27:0/1083795030 conn(0x7fa5fc0ce3e0 msgr2=0x7fa5fc151e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:42.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.637+0000 7fa6019ff700 1 -- 172.21.15.27:0/1083795030 shutdown_connections 2024-01-11T09:26:42.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.637+0000 7fa6019ff700 1 -- 172.21.15.27:0/1083795030 wait complete. 2024-01-11T09:26:42.652 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-11T09:26:42.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.861+0000 7f4b487ff700 1 Processor -- start 2024-01-11T09:26:42.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.862+0000 7f4b487ff700 1 -- start start 2024-01-11T09:26:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.863+0000 7f4b487ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4b401622d0 0x7f4b401646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.863+0000 7f4b487ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b400cf850 con 0x7f4b4015a1b0 2024-01-11T09:26:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.863+0000 7f4b487ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b400cec40 con 0x7f4b400ccd70 2024-01-11T09:26:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.863+0000 7f4b487ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b4015ac90 con 0x7f4b401622d0 2024-01-11T09:26:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.863+0000 7f4b4659b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4b4015a1b0 0x7f4b4015a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36206/0 (socket says 172.21.15.27:36206) 2024-01-11T09:26:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.863+0000 7f4b4659b700 1 -- 172.21.15.27:0/4186550336 learned_addr learned my addr 172.21.15.27:0/4186550336 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.863+0000 7f4b46d9c700 1 --2- 172.21.15.27:0/4186550336 >> v2:172.21.15.77:3300/0 conn(0x7f4b401622d0 0x7f4b401646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:42.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.864+0000 7f4b45599700 1 -- 172.21.15.27:0/4186550336 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649929007 0 0) 0x7f4b400cf850 con 0x7f4b4015a1b0 2024-01-11T09:26:42.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.864+0000 7f4b45599700 1 -- 172.21.15.27:0/4186550336 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b2c003570 con 0x7f4b4015a1b0 2024-01-11T09:26:42.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.864+0000 7f4b46d9c700 1 -- 172.21.15.27:0/4186550336 >> v1:172.21.15.77:6789/0 conn(0x7f4b400ccd70 legacy=0x7f4b400cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:42.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.864+0000 7f4b46d9c700 1 -- 172.21.15.27:0/4186550336 >> v1:172.21.15.27:6789/0 conn(0x7f4b4015a1b0 legacy=0x7f4b4015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:42.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.864+0000 7f4b46d9c700 1 -- 172.21.15.27:0/4186550336 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b400cef20 con 0x7f4b401622d0 2024-01-11T09:26:42.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.864+0000 7f4b46d9c700 1 --2- 172.21.15.27:0/4186550336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b401622d0 0x7f4b401646b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4b38009e60 tx=0x7f4b38009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c9507cc06141120 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:42.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.865+0000 7f4b45599700 1 -- 172.21.15.27:0/4186550336 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3800e070 con 0x7f4b401622d0 2024-01-11T09:26:42.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.865+0000 7f4b45599700 1 -- 172.21.15.27:0/4186550336 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3800bb60 con 0x7f4b401622d0 2024-01-11T09:26:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.865+0000 7f4b45599700 1 -- 172.21.15.27:0/4186550336 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b38012430 con 0x7f4b401622d0 2024-01-11T09:26:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.865+0000 7f4b487ff700 1 -- 172.21.15.27:0/4186550336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b401622d0 msgr2=0x7f4b401646b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.865+0000 7f4b487ff700 1 --2- 172.21.15.27:0/4186550336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b401622d0 0x7f4b401646b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4b38009e60 tx=0x7f4b38009780 comp rx=0 tx=0).stop 2024-01-11T09:26:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.865+0000 7f4b487ff700 1 -- 172.21.15.27:0/4186550336 shutdown_connections 2024-01-11T09:26:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.865+0000 7f4b487ff700 1 --2- 172.21.15.27:0/4186550336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b401622d0 0x7f4b401646b0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.865+0000 7f4b487ff700 1 -- 172.21.15.27:0/4186550336 >> 172.21.15.27:0/4186550336 conn(0x7f4b4006ed40 msgr2=0x7f4b4006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.865+0000 7f4b487ff700 1 -- 172.21.15.27:0/4186550336 shutdown_connections 2024-01-11T09:26:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.865+0000 7f4b487ff700 1 -- 172.21.15.27:0/4186550336 wait complete. 2024-01-11T09:26:42.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.867+0000 7f4b487ff700 1 Processor -- start 2024-01-11T09:26:42.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.868+0000 7f4b487ff700 1 -- start start 2024-01-11T09:26:42.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.869+0000 7f4b487ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4b400ccd70 0x7f4b400b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:42.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.869+0000 7f4b487ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b4015a1b0 0x7f4b4014f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:42.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.869+0000 7f4b487ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b401622d0 0x7f4b4014c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:42.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.869+0000 7f4b487ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4014e280 con 0x7f4b4015a1b0 2024-01-11T09:26:42.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.869+0000 7f4b487ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b400b1af0 con 0x7f4b401622d0 2024-01-11T09:26:42.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.869+0000 7f4b487ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b401519f0 con 0x7f4b400ccd70 2024-01-11T09:26:42.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.869+0000 7f4b45d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b4015a1b0 0x7f4b4014f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:42.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.869+0000 7f4b45d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b4015a1b0 0x7f4b4014f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60832/0 (socket says 172.21.15.27:60832) 2024-01-11T09:26:42.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.869+0000 7f4b45d9a700 1 -- 172.21.15.27:0/397122954 learned_addr learned my addr 172.21.15.27:0/397122954 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:42.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.869+0000 7f4b46d9c700 1 --2- 172.21.15.27:0/397122954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b401622d0 0x7f4b4014c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:42.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.869+0000 7f4b4659b700 1 --2- 172.21.15.27:0/397122954 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4b400ccd70 0x7f4b400b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:42.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.870+0000 7f4b45d9a700 1 -- 172.21.15.27:0/397122954 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4b400ccd70 msgr2=0x7f4b400b1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:42.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.870+0000 7f4b45d9a700 1 --2- 172.21.15.27:0/397122954 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4b400ccd70 0x7f4b400b1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:42.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.870+0000 7f4b45d9a700 1 -- 172.21.15.27:0/397122954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b401622d0 msgr2=0x7f4b4014c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:42.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.870+0000 7f4b45d9a700 1 --2- 172.21.15.27:0/397122954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b401622d0 0x7f4b4014c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:42.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.870+0000 7f4b45d9a700 1 -- 172.21.15.27:0/397122954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b38009a20 con 0x7f4b4015a1b0 2024-01-11T09:26:42.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.870+0000 7f4b45d9a700 1 --2- 172.21.15.27:0/397122954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b4015a1b0 0x7f4b4014f3a0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f4b3000c4e0 tx=0x7f4b3000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:42.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.870+0000 7f4b367fc700 1 -- 172.21.15.27:0/397122954 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b30011590 con 0x7f4b4015a1b0 2024-01-11T09:26:42.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.870+0000 7f4b367fc700 1 -- 172.21.15.27:0/397122954 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b30011710 con 0x7f4b4015a1b0 2024-01-11T09:26:42.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.870+0000 7f4b367fc700 1 -- 172.21.15.27:0/397122954 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b30011ad0 con 0x7f4b4015a1b0 2024-01-11T09:26:42.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.870+0000 7f4b487ff700 1 -- 172.21.15.27:0/397122954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b40151710 con 0x7f4b4015a1b0 2024-01-11T09:26:42.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.870+0000 7f4b487ff700 1 -- 172.21.15.27:0/397122954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b400b1780 con 0x7f4b4015a1b0 2024-01-11T09:26:42.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.871+0000 7f4b487ff700 1 -- 172.21.15.27:0/397122954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b400b1c60 con 0x7f4b4015a1b0 2024-01-11T09:26:42.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.872+0000 7f4b367fc700 1 -- 172.21.15.27:0/397122954 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4b3001b3b0 con 0x7f4b4015a1b0 2024-01-11T09:26:42.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.873+0000 7f4b367fc700 1 --2- 172.21.15.27:0/397122954 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4b24066f30 0x7f4b240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:42.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.873+0000 7f4b367fc700 1 -- 172.21.15.27:0/397122954 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f4b30083930 con 0x7f4b4015a1b0 2024-01-11T09:26:42.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.874+0000 7f4b4659b700 1 --2- 172.21.15.27:0/397122954 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4b24066f30 0x7f4b240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:42.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.874+0000 7f4b4659b700 1 --2- 172.21.15.27:0/397122954 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4b24066f30 0x7f4b240693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c007fd0 tx=0x7f4b3c009040 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:42.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:42.878+0000 7f4b367fc700 1 -- 172.21.15.27:0/397122954 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4b30050b60 con 0x7f4b4015a1b0 2024-01-11T09:26:43.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.126+0000 7f4b487ff700 1 -- 172.21.15.27:0/397122954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4b4004c3c0 con 0x7f4b4015a1b0 2024-01-11T09:26:43.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.708+0000 7f4b367fc700 1 -- 172.21.15.27:0/397122954 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v85) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4b30017070 con 0x7f4b4015a1b0 2024-01-11T09:26:43.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.710+0000 7f4b487ff700 1 -- 172.21.15.27:0/397122954 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4b24066f30 msgr2=0x7f4b240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:43.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.710+0000 7f4b487ff700 1 --2- 172.21.15.27:0/397122954 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4b24066f30 0x7f4b240693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c007fd0 tx=0x7f4b3c009040 comp rx=0 tx=0).stop 2024-01-11T09:26:43.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.710+0000 7f4b487ff700 1 -- 172.21.15.27:0/397122954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b4015a1b0 msgr2=0x7f4b4014f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.710+0000 7f4b487ff700 1 --2- 172.21.15.27:0/397122954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b4015a1b0 0x7f4b4014f3a0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f4b3000c4e0 tx=0x7f4b3000c7f0 comp rx=0 tx=0).stop 2024-01-11T09:26:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.710+0000 7f4b487ff700 1 -- 172.21.15.27:0/397122954 shutdown_connections 2024-01-11T09:26:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.710+0000 7f4b487ff700 1 --2- 172.21.15.27:0/397122954 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4b24066f30 0x7f4b240693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.710+0000 7f4b487ff700 1 --2- 172.21.15.27:0/397122954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b401622d0 0x7f4b4014c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.710+0000 7f4b487ff700 1 --2- 172.21.15.27:0/397122954 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4b400ccd70 0x7f4b400b1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.710+0000 7f4b487ff700 1 --2- 172.21.15.27:0/397122954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b4015a1b0 0x7f4b4014f3a0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.710+0000 7f4b487ff700 1 -- 172.21.15.27:0/397122954 >> 172.21.15.27:0/397122954 conn(0x7f4b4006ed40 msgr2=0x7f4b40154080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.711+0000 7f4b487ff700 1 -- 172.21.15.27:0/397122954 shutdown_connections 2024-01-11T09:26:43.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.711+0000 7f4b487ff700 1 -- 172.21.15.27:0/397122954 wait complete. 2024-01-11T09:26:43.726 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-11T09:26:43.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.934+0000 7f77849ff700 1 Processor -- start 2024-01-11T09:26:43.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.934+0000 7f77849ff700 1 -- start start 2024-01-11T09:26:43.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.935+0000 7f77849ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f778015a1b0 0x7f778015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:43.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.935+0000 7f77849ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77800ccd70 0x7f77800cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:43.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.935+0000 7f77849ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77800cf850 con 0x7f77800685c0 2024-01-11T09:26:43.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.935+0000 7f77849ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77800cec40 con 0x7f77800ccd70 2024-01-11T09:26:43.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.935+0000 7f77849ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77800cef20 con 0x7f778015a1b0 2024-01-11T09:26:43.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.935+0000 7f777dd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77800ccd70 0x7f77800cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:43.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.935+0000 7f777dd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77800ccd70 0x7f77800cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32930/0 (socket says 172.21.15.27:32930) 2024-01-11T09:26:43.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.935+0000 7f777dd9b700 1 -- 172.21.15.27:0/1733081345 learned_addr learned my addr 172.21.15.27:0/1733081345 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:43.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.936+0000 7f777e59c700 1 --2- 172.21.15.27:0/1733081345 >> v2:172.21.15.77:3300/0 conn(0x7f778015a1b0 0x7f778015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:43.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.936+0000 7f777dd9b700 1 -- 172.21.15.27:0/1733081345 >> v2:172.21.15.77:3300/0 conn(0x7f778015a1b0 msgr2=0x7f778015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:43.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.936+0000 7f777dd9b700 1 --2- 172.21.15.27:0/1733081345 >> v2:172.21.15.77:3300/0 conn(0x7f778015a1b0 0x7f778015a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:43.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.936+0000 7f777dd9b700 1 -- 172.21.15.27:0/1733081345 >> v1:172.21.15.77:6789/0 conn(0x7f77800685c0 legacy=0x7f77800689f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:43.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.936+0000 7f777dd9b700 1 -- 172.21.15.27:0/1733081345 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77800cf200 con 0x7f77800ccd70 2024-01-11T09:26:43.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.936+0000 7f777dd9b700 1 --2- 172.21.15.27:0/1733081345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77800ccd70 0x7f77800cd210 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f77800d16f0 tx=0x7f7774009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf6571937caae0a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:43.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.936+0000 7f777d59a700 1 -- 172.21.15.27:0/1733081345 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777400b860 con 0x7f77800ccd70 2024-01-11T09:26:43.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.937+0000 7f777d59a700 1 -- 172.21.15.27:0/1733081345 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777400b9c0 con 0x7f77800ccd70 2024-01-11T09:26:43.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.937+0000 7f777d59a700 1 -- 172.21.15.27:0/1733081345 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777400bd60 con 0x7f77800ccd70 2024-01-11T09:26:43.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.937+0000 7f77849ff700 1 -- 172.21.15.27:0/1733081345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77800ccd70 msgr2=0x7f77800cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:43.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.937+0000 7f77849ff700 1 --2- 172.21.15.27:0/1733081345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77800ccd70 0x7f77800cd210 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f77800d16f0 tx=0x7f7774009250 comp rx=0 tx=0).stop 2024-01-11T09:26:43.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.937+0000 7f77849ff700 1 -- 172.21.15.27:0/1733081345 shutdown_connections 2024-01-11T09:26:43.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.937+0000 7f77849ff700 1 --2- 172.21.15.27:0/1733081345 >> v2:172.21.15.77:3300/0 conn(0x7f778015a1b0 0x7f778015a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:43.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.937+0000 7f77849ff700 1 --2- 172.21.15.27:0/1733081345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77800ccd70 0x7f77800cd210 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:43.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.937+0000 7f77849ff700 1 -- 172.21.15.27:0/1733081345 >> 172.21.15.27:0/1733081345 conn(0x7f778006ed40 msgr2=0x7f778006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:43.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.937+0000 7f77849ff700 1 -- 172.21.15.27:0/1733081345 shutdown_connections 2024-01-11T09:26:43.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.937+0000 7f77849ff700 1 -- 172.21.15.27:0/1733081345 wait complete. 2024-01-11T09:26:43.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.939+0000 7f77849ff700 1 Processor -- start 2024-01-11T09:26:43.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.940+0000 7f77849ff700 1 -- start start 2024-01-11T09:26:43.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.941+0000 7f77849ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77800685c0 0x7f778014fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:43.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.941+0000 7f77849ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77800ccd70 0x7f7780157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:43.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.941+0000 7f77849ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f778015a1b0 0x7f77800b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:43.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.941+0000 7f77849ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7780150660 con 0x7f778015a1b0 2024-01-11T09:26:43.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.941+0000 7f77849ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77800b28e0 con 0x7f77800685c0 2024-01-11T09:26:43.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.941+0000 7f77849ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7780158ec0 con 0x7f77800ccd70 2024-01-11T09:26:43.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.941+0000 7f777ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f778015a1b0 0x7f77800b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:43.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.941+0000 7f777ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f778015a1b0 0x7f77800b23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32938/0 (socket says 172.21.15.27:32938) 2024-01-11T09:26:43.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.941+0000 7f777ed9d700 1 -- 172.21.15.27:0/3366644247 learned_addr learned my addr 172.21.15.27:0/3366644247 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:43.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.941+0000 7f777e59c700 1 --2- 172.21.15.27:0/3366644247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77800685c0 0x7f778014fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:43.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.941+0000 7f777dd9b700 1 --2- 172.21.15.27:0/3366644247 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77800ccd70 0x7f7780157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:43.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.942+0000 7f777ed9d700 1 -- 172.21.15.27:0/3366644247 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77800ccd70 msgr2=0x7f7780157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:43.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.942+0000 7f777ed9d700 1 --2- 172.21.15.27:0/3366644247 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77800ccd70 0x7f7780157f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:43.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.942+0000 7f777ed9d700 1 -- 172.21.15.27:0/3366644247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77800685c0 msgr2=0x7f778014fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:43.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.942+0000 7f777ed9d700 1 --2- 172.21.15.27:0/3366644247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77800685c0 0x7f778014fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:43.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.942+0000 7f777ed9d700 1 -- 172.21.15.27:0/3366644247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77740094f0 con 0x7f778015a1b0 2024-01-11T09:26:43.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.942+0000 7f777ed9d700 1 --2- 172.21.15.27:0/3366644247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f778015a1b0 0x7f77800b23a0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f777800c870 tx=0x7f7778001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:43.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.942+0000 7f776e7fc700 1 -- 172.21.15.27:0/3366644247 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778012630 con 0x7f778015a1b0 2024-01-11T09:26:43.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.942+0000 7f776e7fc700 1 -- 172.21.15.27:0/3366644247 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77780127b0 con 0x7f778015a1b0 2024-01-11T09:26:43.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.942+0000 7f776e7fc700 1 -- 172.21.15.27:0/3366644247 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778012b70 con 0x7f778015a1b0 2024-01-11T09:26:43.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.942+0000 7f77849ff700 1 -- 172.21.15.27:0/3366644247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7780158620 con 0x7f778015a1b0 2024-01-11T09:26:43.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.942+0000 7f77849ff700 1 -- 172.21.15.27:0/3366644247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7780158be0 con 0x7f778015a1b0 2024-01-11T09:26:43.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.943+0000 7f77849ff700 1 -- 172.21.15.27:0/3366644247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f778004c3c0 con 0x7f778015a1b0 2024-01-11T09:26:43.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.944+0000 7f776e7fc700 1 -- 172.21.15.27:0/3366644247 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7778003030 con 0x7f778015a1b0 2024-01-11T09:26:43.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.946+0000 7f776e7fc700 1 --2- 172.21.15.27:0/3366644247 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7760066f30 0x7f77600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:43.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.946+0000 7f776e7fc700 1 -- 172.21.15.27:0/3366644247 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f77780843b0 con 0x7f778015a1b0 2024-01-11T09:26:43.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.946+0000 7f777e59c700 1 --2- 172.21.15.27:0/3366644247 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7760066f30 0x7f77600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:43.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.947+0000 7f777e59c700 1 --2- 172.21.15.27:0/3366644247 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7760066f30 0x7f77600693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7770006fd0 tx=0x7f7770008040 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:43.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:43.950+0000 7f776e7fc700 1 -- 172.21.15.27:0/3366644247 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7778051540 con 0x7f778015a1b0 2024-01-11T09:26:44.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.196+0000 7f77849ff700 1 -- 172.21.15.27:0/3366644247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7780069310 con 0x7f778015a1b0 2024-01-11T09:26:44.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.719+0000 7f776e7fc700 1 -- 172.21.15.27:0/3366644247 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v86) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7778054b60 con 0x7f778015a1b0 2024-01-11T09:26:44.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.721+0000 7f77849ff700 1 -- 172.21.15.27:0/3366644247 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7760066f30 msgr2=0x7f77600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:44.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.721+0000 7f77849ff700 1 --2- 172.21.15.27:0/3366644247 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7760066f30 0x7f77600693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7770006fd0 tx=0x7f7770008040 comp rx=0 tx=0).stop 2024-01-11T09:26:44.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.721+0000 7f77849ff700 1 -- 172.21.15.27:0/3366644247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f778015a1b0 msgr2=0x7f77800b23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:44.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.721+0000 7f77849ff700 1 --2- 172.21.15.27:0/3366644247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f778015a1b0 0x7f77800b23a0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f777800c870 tx=0x7f7778001670 comp rx=0 tx=0).stop 2024-01-11T09:26:44.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.721+0000 7f77849ff700 1 -- 172.21.15.27:0/3366644247 shutdown_connections 2024-01-11T09:26:44.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.721+0000 7f77849ff700 1 --2- 172.21.15.27:0/3366644247 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7760066f30 0x7f77600693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:44.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.721+0000 7f77849ff700 1 --2- 172.21.15.27:0/3366644247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f778015a1b0 0x7f77800b23a0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:44.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.721+0000 7f77849ff700 1 --2- 172.21.15.27:0/3366644247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77800685c0 0x7f778014fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:44.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.721+0000 7f77849ff700 1 --2- 172.21.15.27:0/3366644247 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77800ccd70 0x7f7780157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:44.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.721+0000 7f77849ff700 1 -- 172.21.15.27:0/3366644247 >> 172.21.15.27:0/3366644247 conn(0x7f778006ed40 msgr2=0x7f77801563d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:44.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.722+0000 7f77849ff700 1 -- 172.21.15.27:0/3366644247 shutdown_connections 2024-01-11T09:26:44.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.722+0000 7f77849ff700 1 -- 172.21.15.27:0/3366644247 wait complete. 2024-01-11T09:26:44.739 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:26:44.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.944+0000 7f1f031ff700 1 Processor -- start 2024-01-11T09:26:44.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.944+0000 7f1f031ff700 1 -- start start 2024-01-11T09:26:44.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.945+0000 7f1f031ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1efc0b2e40 0x7f1efc06bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:44.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.945+0000 7f1f031ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1efc06fba0 con 0x7f1efc159650 2024-01-11T09:26:44.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.945+0000 7f1f031ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1efc06e700 con 0x7f1efc159a20 2024-01-11T09:26:44.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.945+0000 7f1f031ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1efc06c360 con 0x7f1efc0b2e40 2024-01-11T09:26:44.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.945+0000 7f1f00f9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1efc159650 0x7f1efc0b0c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40394/0 (socket says 172.21.15.27:40394) 2024-01-11T09:26:44.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.945+0000 7f1f00f9b700 1 -- 172.21.15.27:0/2931775225 learned_addr learned my addr 172.21.15.27:0/2931775225 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:44.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.946+0000 7f1f0179c700 1 --2- 172.21.15.27:0/2931775225 >> v2:172.21.15.77:3300/0 conn(0x7f1efc0b2e40 0x7f1efc06bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:44.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.946+0000 7f1efb7fe700 1 -- 172.21.15.27:0/2931775225 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040149881 0 0) 0x7f1efc06fba0 con 0x7f1efc159650 2024-01-11T09:26:44.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.946+0000 7f1efb7fe700 1 -- 172.21.15.27:0/2931775225 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eec003570 con 0x7f1efc159650 2024-01-11T09:26:44.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.946+0000 7f1efb7fe700 1 -- 172.21.15.27:0/2931775225 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990139600 0 0) 0x7f1efc06e700 con 0x7f1efc159a20 2024-01-11T09:26:44.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.946+0000 7f1efb7fe700 1 -- 172.21.15.27:0/2931775225 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1efc06fba0 con 0x7f1efc159a20 2024-01-11T09:26:44.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.946+0000 7f1efb7fe700 1 -- 172.21.15.27:0/2931775225 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2564949376 0 0) 0x7f1eec003570 con 0x7f1efc159650 2024-01-11T09:26:44.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.946+0000 7f1efb7fe700 1 -- 172.21.15.27:0/2931775225 >> v2:172.21.15.77:3300/0 conn(0x7f1efc0b2e40 msgr2=0x7f1efc06bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:44.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.946+0000 7f1efb7fe700 1 --2- 172.21.15.27:0/2931775225 >> v2:172.21.15.77:3300/0 conn(0x7f1efc0b2e40 0x7f1efc06bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:44.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.946+0000 7f1efb7fe700 1 -- 172.21.15.27:0/2931775225 >> v1:172.21.15.77:6789/0 conn(0x7f1efc159a20 legacy=0x7f1efc0b2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:44.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.946+0000 7f1efb7fe700 1 -- 172.21.15.27:0/2931775225 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1efc06c640 con 0x7f1efc159650 2024-01-11T09:26:44.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.946+0000 7f1efb7fe700 1 -- 172.21.15.27:0/2931775225 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f1ef0001470 con 0x7f1efc159650 2024-01-11T09:26:44.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.946+0000 7f1efb7fe700 1 -- 172.21.15.27:0/2931775225 >> v1:172.21.15.27:6789/0 conn(0x7f1efc159650 legacy=0x7f1efc0b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:44.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.946+0000 7f1efb7fe700 1 --2- 172.21.15.27:0/2931775225 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1eec003bf0 0x7f1eec003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:44.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.946+0000 7f1efb7fe700 1 --2- 172.21.15.27:0/2931775225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1eec004810 0x7f1eec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:44.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.946+0000 7f1efb7fe700 1 --2- 172.21.15.27:0/2931775225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eec0051c0 0x7f1eec00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:44.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.946+0000 7f1efb7fe700 1 -- 172.21.15.27:0/2931775225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1efc06c360 con 0x7f1eec0051c0 2024-01-11T09:26:44.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.946+0000 7f1efb7fe700 1 -- 172.21.15.27:0/2931775225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec00c0b0 con 0x7f1eec004810 2024-01-11T09:26:44.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.946+0000 7f1efb7fe700 1 -- 172.21.15.27:0/2931775225 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec00c270 con 0x7f1eec003bf0 2024-01-11T09:26:44.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.947+0000 7f1f0179c700 1 --2- 172.21.15.27:0/2931775225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eec0051c0 0x7f1eec00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:44.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.947+0000 7f1f00f9b700 1 --2- 172.21.15.27:0/2931775225 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1eec003bf0 0x7f1eec003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:44.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.947+0000 7f1efbfff700 1 --2- 172.21.15.27:0/2931775225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1eec004810 0x7f1eec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:44.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.947+0000 7f1f0179c700 1 -- 172.21.15.27:0/2931775225 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1eec003bf0 msgr2=0x7f1eec003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:44.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.947+0000 7f1f0179c700 1 --2- 172.21.15.27:0/2931775225 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1eec003bf0 0x7f1eec003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:44.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.947+0000 7f1f0179c700 1 -- 172.21.15.27:0/2931775225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1eec004810 msgr2=0x7f1eec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:44.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.947+0000 7f1f0179c700 1 --2- 172.21.15.27:0/2931775225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1eec004810 0x7f1eec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:44.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.947+0000 7f1f0179c700 1 -- 172.21.15.27:0/2931775225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eec003570 con 0x7f1eec0051c0 2024-01-11T09:26:44.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.947+0000 7f1f0179c700 1 -- 172.21.15.27:0/2931775225 reap_dead start 2024-01-11T09:26:44.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.947+0000 7f1f0179c700 1 --2- 172.21.15.27:0/2931775225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eec0051c0 0x7f1eec00ba60 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f1ef400e210 tx=0x7f1ef400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=536aea0fd2b248a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:44.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.948+0000 7f1efb7fe700 1 -- 172.21.15.27:0/2931775225 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef400eac0 con 0x7f1eec0051c0 2024-01-11T09:26:44.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.948+0000 7f1efb7fe700 1 -- 172.21.15.27:0/2931775225 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef400f040 con 0x7f1eec0051c0 2024-01-11T09:26:44.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.948+0000 7f1efb7fe700 1 -- 172.21.15.27:0/2931775225 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef400ee70 con 0x7f1eec0051c0 2024-01-11T09:26:44.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.948+0000 7f1f031ff700 1 -- 172.21.15.27:0/2931775225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eec0051c0 msgr2=0x7f1eec00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:44.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.948+0000 7f1f031ff700 1 --2- 172.21.15.27:0/2931775225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eec0051c0 0x7f1eec00ba60 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f1ef400e210 tx=0x7f1ef400b040 comp rx=0 tx=0).stop 2024-01-11T09:26:44.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.948+0000 7f1f031ff700 1 -- 172.21.15.27:0/2931775225 shutdown_connections 2024-01-11T09:26:44.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.948+0000 7f1f031ff700 1 --2- 172.21.15.27:0/2931775225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eec0051c0 0x7f1eec00ba60 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:44.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.948+0000 7f1f031ff700 1 -- 172.21.15.27:0/2931775225 >> 172.21.15.27:0/2931775225 conn(0x7f1efc06b200 msgr2=0x7f1efc0aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:44.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.949+0000 7f1f031ff700 1 -- 172.21.15.27:0/2931775225 shutdown_connections 2024-01-11T09:26:44.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.949+0000 7f1f031ff700 1 -- 172.21.15.27:0/2931775225 wait complete. 2024-01-11T09:26:44.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.950+0000 7f1f031ff700 1 Processor -- start 2024-01-11T09:26:44.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.950+0000 7f1f031ff700 1 -- start start 2024-01-11T09:26:44.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.951+0000 7f1f031ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eec0051c0 0x7f1efc151cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:44.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.951+0000 7f1f031ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1efc0aad00 0x7f1efc0ab170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:44.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.951+0000 7f1f031ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1efc155cc0 0x7f1efc156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:44.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.951+0000 7f1f031ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1efc0ac060 con 0x7f1eec0051c0 2024-01-11T09:26:44.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.951+0000 7f1f031ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1efc169d40 con 0x7f1efc0aad00 2024-01-11T09:26:44.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.951+0000 7f1f031ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1efc156650 con 0x7f1efc155cc0 2024-01-11T09:26:44.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.952+0000 7f1f00f9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eec0051c0 0x7f1efc151cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:44.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.952+0000 7f1f00f9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eec0051c0 0x7f1efc151cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32958/0 (socket says 172.21.15.27:32958) 2024-01-11T09:26:44.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.952+0000 7f1f00f9b700 1 -- 172.21.15.27:0/2529483394 learned_addr learned my addr 172.21.15.27:0/2529483394 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:44.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.952+0000 7f1efbfff700 1 --2- 172.21.15.27:0/2529483394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1efc0aad00 0x7f1efc0ab170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:44.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.952+0000 7f1f0179c700 1 --2- 172.21.15.27:0/2529483394 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1efc155cc0 0x7f1efc156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:44.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.952+0000 7f1f00f9b700 1 -- 172.21.15.27:0/2529483394 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1efc155cc0 msgr2=0x7f1efc156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:44.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.952+0000 7f1f00f9b700 1 --2- 172.21.15.27:0/2529483394 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1efc155cc0 0x7f1efc156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:44.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.952+0000 7f1f00f9b700 1 -- 172.21.15.27:0/2529483394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1efc0aad00 msgr2=0x7f1efc0ab170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:44.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.952+0000 7f1f00f9b700 1 --2- 172.21.15.27:0/2529483394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1efc0aad00 0x7f1efc0ab170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:44.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.952+0000 7f1f00f9b700 1 -- 172.21.15.27:0/2529483394 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1efc169a50 con 0x7f1eec0051c0 2024-01-11T09:26:44.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.953+0000 7f1f00f9b700 1 --2- 172.21.15.27:0/2529483394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eec0051c0 0x7f1efc151cd0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f1ef00073f0 tx=0x7f1ef0001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:44.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.953+0000 7f1ef8ff9700 1 -- 172.21.15.27:0/2529483394 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0011590 con 0x7f1eec0051c0 2024-01-11T09:26:44.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.953+0000 7f1ef8ff9700 1 -- 172.21.15.27:0/2529483394 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef0011710 con 0x7f1eec0051c0 2024-01-11T09:26:44.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.953+0000 7f1f031ff700 1 -- 172.21.15.27:0/2529483394 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1efc152210 con 0x7f1eec0051c0 2024-01-11T09:26:44.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.953+0000 7f1ef8ff9700 1 -- 172.21.15.27:0/2529483394 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0011ad0 con 0x7f1eec0051c0 2024-01-11T09:26:44.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.953+0000 7f1f031ff700 1 -- 172.21.15.27:0/2529483394 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1efc0abb20 con 0x7f1eec0051c0 2024-01-11T09:26:44.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.954+0000 7f1f031ff700 1 -- 172.21.15.27:0/2529483394 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1efc0718d0 con 0x7f1eec0051c0 2024-01-11T09:26:44.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.955+0000 7f1ef8ff9700 1 -- 172.21.15.27:0/2529483394 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1ef001d460 con 0x7f1eec0051c0 2024-01-11T09:26:44.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.956+0000 7f1ef8ff9700 1 --2- 172.21.15.27:0/2529483394 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1ee0066f30 0x7f1ee00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:44.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.957+0000 7f1ef8ff9700 1 -- 172.21.15.27:0/2529483394 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f1ef00846e0 con 0x7f1eec0051c0 2024-01-11T09:26:44.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.957+0000 7f1efbfff700 1 --2- 172.21.15.27:0/2529483394 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1ee0066f30 0x7f1ee00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:44.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.958+0000 7f1efbfff700 1 --2- 172.21.15.27:0/2529483394 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1ee0066f30 0x7f1ee00693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1efc0aab80 tx=0x7f1ee8007d30 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:44.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:44.961+0000 7f1ef8ff9700 1 -- 172.21.15.27:0/2529483394 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1ef0051870 con 0x7f1eec0051c0 2024-01-11T09:26:45.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.207+0000 7f1f031ff700 1 -- 172.21.15.27:0/2529483394 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1efc06c920 con 0x7f1eec0051c0 2024-01-11T09:26:45.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.208+0000 7f1ef8ff9700 1 -- 172.21.15.27:0/2529483394 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4598 (secure 0 0 0) 0x7f1ef0054e90 con 0x7f1eec0051c0 2024-01-11T09:26:45.204 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:26:45.204 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":86,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7700,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/191611944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":191611944},{"type":"v1","addr":"172.21.15.27:6835","nonce":191611944}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":8734,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/3566978772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3566978772},{"type":"v1","addr":"172.21.15.77:6834","nonce":3566978772}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":8737,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1168169536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1168169536},{"type":"v1","addr":"172.21.15.77:6835","nonce":1168169536}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:26:41.190629+0000","modified":"2024-01-11T09:26:44.707534+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9093},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9093":{"gid":9093,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/3544429684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3544429684},{"type":"v1","addr":"172.21.15.27:6834","nonce":3544429684}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2024-01-11T09:26:45.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.210+0000 7f1f031ff700 1 -- 172.21.15.27:0/2529483394 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1ee0066f30 msgr2=0x7f1ee00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:45.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.210+0000 7f1f031ff700 1 --2- 172.21.15.27:0/2529483394 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1ee0066f30 0x7f1ee00693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1efc0aab80 tx=0x7f1ee8007d30 comp rx=0 tx=0).stop 2024-01-11T09:26:45.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.210+0000 7f1f031ff700 1 -- 172.21.15.27:0/2529483394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eec0051c0 msgr2=0x7f1efc151cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:45.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.210+0000 7f1f031ff700 1 --2- 172.21.15.27:0/2529483394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eec0051c0 0x7f1efc151cd0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f1ef00073f0 tx=0x7f1ef0001e10 comp rx=0 tx=0).stop 2024-01-11T09:26:45.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.210+0000 7f1f031ff700 1 -- 172.21.15.27:0/2529483394 shutdown_connections 2024-01-11T09:26:45.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.210+0000 7f1f031ff700 1 --2- 172.21.15.27:0/2529483394 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1ee0066f30 0x7f1ee00693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:45.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.210+0000 7f1f031ff700 1 --2- 172.21.15.27:0/2529483394 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1efc155cc0 0x7f1efc156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:45.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.210+0000 7f1f031ff700 1 --2- 172.21.15.27:0/2529483394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1eec0051c0 0x7f1efc151cd0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:45.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.210+0000 7f1f031ff700 1 --2- 172.21.15.27:0/2529483394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1efc0aad00 0x7f1efc0ab170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:45.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.210+0000 7f1f031ff700 1 -- 172.21.15.27:0/2529483394 >> 172.21.15.27:0/2529483394 conn(0x7f1efc06b200 msgr2=0x7f1efc15d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:45.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.210+0000 7f1f031ff700 1 -- 172.21.15.27:0/2529483394 shutdown_connections 2024-01-11T09:26:45.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.210+0000 7f1f031ff700 1 -- 172.21.15.27:0/2529483394 wait complete. 2024-01-11T09:26:45.208 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 86 2024-01-11T09:26:45.225 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:26:45.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.428+0000 7ff20c9ff700 1 Processor -- start 2024-01-11T09:26:45.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.428+0000 7ff20c9ff700 1 -- start start 2024-01-11T09:26:45.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.429+0000 7ff20c9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2080ca670 0x7ff208068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:45.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.429+0000 7ff20c9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff208068b60 0x7ff2080aa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.429+0000 7ff20c9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20815da10 con 0x7ff2081596c0 2024-01-11T09:26:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.429+0000 7ff20c9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff208068fe0 con 0x7ff208068b60 2024-01-11T09:26:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.429+0000 7ff20c9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080a9cb0 con 0x7ff2080ca670 2024-01-11T09:26:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.429+0000 7ff20659c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff2081596c0 0x7ff20815d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40404/0 (socket says 172.21.15.27:40404) 2024-01-11T09:26:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.430+0000 7ff20659c700 1 -- 172.21.15.27:0/377281006 learned_addr learned my addr 172.21.15.27:0/377281006 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.430+0000 7ff205d9b700 1 --2- 172.21.15.27:0/377281006 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2080ca670 0x7ff208068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.430+0000 7ff206d9d700 1 --2- 172.21.15.27:0/377281006 >> v2:172.21.15.77:3300/0 conn(0x7ff208068b60 0x7ff2080aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.430+0000 7ff20559a700 1 -- 172.21.15.27:0/377281006 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612591597 0 0) 0x7ff1f8001110 con 0x7ff2081596c0 2024-01-11T09:26:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.430+0000 7ff20559a700 1 -- 172.21.15.27:0/377281006 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f0003570 con 0x7ff2081596c0 2024-01-11T09:26:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.430+0000 7ff205d9b700 1 -- 172.21.15.27:0/377281006 >> v2:172.21.15.77:3300/0 conn(0x7ff208068b60 msgr2=0x7ff2080aa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.430+0000 7ff205d9b700 1 --2- 172.21.15.27:0/377281006 >> v2:172.21.15.77:3300/0 conn(0x7ff208068b60 0x7ff2080aa140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.430+0000 7ff205d9b700 1 -- 172.21.15.27:0/377281006 >> v1:172.21.15.27:6789/0 conn(0x7ff2081596c0 legacy=0x7ff20815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:45.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.430+0000 7ff205d9b700 1 -- 172.21.15.27:0/377281006 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20815e210 con 0x7ff2080ca670 2024-01-11T09:26:45.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.430+0000 7ff206d9d700 1 --2- 172.21.15.27:0/377281006 >> v2:172.21.15.77:3300/0 conn(0x7ff208068b60 0x7ff2080aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:26:45.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.431+0000 7ff205d9b700 1 --2- 172.21.15.27:0/377281006 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2080ca670 0x7ff208068620 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff20815e400 tx=0x7ff1fc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f2b92620e8282aa server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:45.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.431+0000 7ff20559a700 1 -- 172.21.15.27:0/377281006 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc011630 con 0x7ff2080ca670 2024-01-11T09:26:45.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.431+0000 7ff20559a700 1 -- 172.21.15.27:0/377281006 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1fc011790 con 0x7ff2080ca670 2024-01-11T09:26:45.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.431+0000 7ff20559a700 1 -- 172.21.15.27:0/377281006 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc011b50 con 0x7ff2080ca670 2024-01-11T09:26:45.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.431+0000 7ff20c9ff700 1 -- 172.21.15.27:0/377281006 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2080ca670 msgr2=0x7ff208068620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:45.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.432+0000 7ff20c9ff700 1 --2- 172.21.15.27:0/377281006 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2080ca670 0x7ff208068620 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff20815e400 tx=0x7ff1fc0091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:45.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.432+0000 7ff20c9ff700 1 -- 172.21.15.27:0/377281006 shutdown_connections 2024-01-11T09:26:45.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.432+0000 7ff20c9ff700 1 --2- 172.21.15.27:0/377281006 >> v2:172.21.15.77:3300/0 conn(0x7ff208068b60 0x7ff2080aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:45.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.432+0000 7ff20c9ff700 1 --2- 172.21.15.27:0/377281006 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2080ca670 0x7ff208068620 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:45.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.432+0000 7ff20c9ff700 1 -- 172.21.15.27:0/377281006 >> 172.21.15.27:0/377281006 conn(0x7ff2080c9f70 msgr2=0x7ff20806deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:45.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.432+0000 7ff20c9ff700 1 -- 172.21.15.27:0/377281006 shutdown_connections 2024-01-11T09:26:45.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.432+0000 7ff20c9ff700 1 -- 172.21.15.27:0/377281006 wait complete. 2024-01-11T09:26:45.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.434+0000 7ff20c9ff700 1 Processor -- start 2024-01-11T09:26:45.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.434+0000 7ff20c9ff700 1 -- start start 2024-01-11T09:26:45.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.435+0000 7ff20c9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff208068b60 0x7ff2080c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:45.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.435+0000 7ff20c9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2080ca670 0x7ff208157de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.435+0000 7ff20c9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2081596c0 0x7ff208156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.435+0000 7ff20c9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080c80c0 con 0x7ff2081596c0 2024-01-11T09:26:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.435+0000 7ff20c9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080c83a0 con 0x7ff208068b60 2024-01-11T09:26:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.435+0000 7ff20c9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080c8680 con 0x7ff2080ca670 2024-01-11T09:26:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.436+0000 7ff206d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2081596c0 0x7ff208156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.436+0000 7ff206d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2081596c0 0x7ff208156600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32960/0 (socket says 172.21.15.27:32960) 2024-01-11T09:26:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.436+0000 7ff206d9d700 1 -- 172.21.15.27:0/2879970869 learned_addr learned my addr 172.21.15.27:0/2879970869 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.436+0000 7ff20659c700 1 --2- 172.21.15.27:0/2879970869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff208068b60 0x7ff2080c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.436+0000 7ff205d9b700 1 --2- 172.21.15.27:0/2879970869 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2080ca670 0x7ff208157de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.436+0000 7ff206d9d700 1 -- 172.21.15.27:0/2879970869 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2080ca670 msgr2=0x7ff208157de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.436+0000 7ff206d9d700 1 --2- 172.21.15.27:0/2879970869 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2080ca670 0x7ff208157de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.436+0000 7ff206d9d700 1 -- 172.21.15.27:0/2879970869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff208068b60 msgr2=0x7ff2080c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.436+0000 7ff206d9d700 1 --2- 172.21.15.27:0/2879970869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff208068b60 0x7ff2080c79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.436+0000 7ff206d9d700 1 -- 172.21.15.27:0/2879970869 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff200009a20 con 0x7ff2081596c0 2024-01-11T09:26:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.436+0000 7ff206d9d700 1 --2- 172.21.15.27:0/2879970869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2081596c0 0x7ff208156600 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7ff200009f20 tx=0x7ff20000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.437+0000 7ff1f67fc700 1 -- 172.21.15.27:0/2879970869 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20000cc00 con 0x7ff2081596c0 2024-01-11T09:26:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.437+0000 7ff1f67fc700 1 -- 172.21.15.27:0/2879970869 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20000f040 con 0x7ff2081596c0 2024-01-11T09:26:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.437+0000 7ff1f67fc700 1 -- 172.21.15.27:0/2879970869 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff200014480 con 0x7ff2081596c0 2024-01-11T09:26:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.437+0000 7ff20c9ff700 1 -- 172.21.15.27:0/2879970869 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1fc009420 con 0x7ff2081596c0 2024-01-11T09:26:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.437+0000 7ff20c9ff700 1 -- 172.21.15.27:0/2879970869 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff208156b40 con 0x7ff2081596c0 2024-01-11T09:26:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.438+0000 7ff20c9ff700 1 -- 172.21.15.27:0/2879970869 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2080718d0 con 0x7ff2081596c0 2024-01-11T09:26:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.439+0000 7ff1f67fc700 1 -- 172.21.15.27:0/2879970869 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff200003020 con 0x7ff2081596c0 2024-01-11T09:26:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.440+0000 7ff1f67fc700 1 --2- 172.21.15.27:0/2879970869 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7ff1e8066f30 0x7ff1e80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.440+0000 7ff1f67fc700 1 -- 172.21.15.27:0/2879970869 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5761+0+0 (secure 0 0 0) 0x7ff200084f90 con 0x7ff2081596c0 2024-01-11T09:26:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.441+0000 7ff20659c700 1 --2- 172.21.15.27:0/2879970869 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7ff1e8066f30 0x7ff1e80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:45.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.441+0000 7ff20659c700 1 --2- 172.21.15.27:0/2879970869 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7ff1e8066f30 0x7ff1e80693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8001270 tx=0x7ff1f8007bf0 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.444+0000 7ff1f67fc700 1 -- 172.21.15.27:0/2879970869 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff200052120 con 0x7ff2081596c0 2024-01-11T09:26:45.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.647+0000 7ff20c9ff700 1 -- 172.21.15.27:0/2879970869 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff20806ea80 con 0x7ff2081596c0 2024-01-11T09:26:45.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.647+0000 7ff1f67fc700 1 -- 172.21.15.27:0/2879970869 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v167) v1 ==== 74+0+15404 (secure 0 0 0) 0x7ff200055740 con 0x7ff2081596c0 2024-01-11T09:26:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:26:45.644 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":167,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:26:42.692628+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:26:39.008224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-01-11T09:26:40.186962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:26:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.650+0000 7ff20c9ff700 1 -- 172.21.15.27:0/2879970869 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7ff1e8066f30 msgr2=0x7ff1e80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.650+0000 7ff20c9ff700 1 --2- 172.21.15.27:0/2879970869 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7ff1e8066f30 0x7ff1e80693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8001270 tx=0x7ff1f8007bf0 comp rx=0 tx=0).stop 2024-01-11T09:26:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.650+0000 7ff20c9ff700 1 -- 172.21.15.27:0/2879970869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2081596c0 msgr2=0x7ff208156600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.650+0000 7ff20c9ff700 1 --2- 172.21.15.27:0/2879970869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2081596c0 0x7ff208156600 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7ff200009f20 tx=0x7ff20000b040 comp rx=0 tx=0).stop 2024-01-11T09:26:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.650+0000 7ff20c9ff700 1 -- 172.21.15.27:0/2879970869 shutdown_connections 2024-01-11T09:26:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.650+0000 7ff20c9ff700 1 --2- 172.21.15.27:0/2879970869 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7ff1e8066f30 0x7ff1e80693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.650+0000 7ff20c9ff700 1 --2- 172.21.15.27:0/2879970869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2081596c0 0x7ff208156600 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.650+0000 7ff20c9ff700 1 --2- 172.21.15.27:0/2879970869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff208068b60 0x7ff2080c79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:45.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.650+0000 7ff20c9ff700 1 --2- 172.21.15.27:0/2879970869 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2080ca670 0x7ff208157de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:45.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.650+0000 7ff20c9ff700 1 -- 172.21.15.27:0/2879970869 >> 172.21.15.27:0/2879970869 conn(0x7ff2080c9f70 msgr2=0x7ff2080c3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:45.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.650+0000 7ff20c9ff700 1 -- 172.21.15.27:0/2879970869 shutdown_connections 2024-01-11T09:26:45.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.650+0000 7ff20c9ff700 1 -- 172.21.15.27:0/2879970869 wait complete. 2024-01-11T09:26:45.665 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-11T09:26:45.665 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:26:45.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.868+0000 7f96d3dff700 1 Processor -- start 2024-01-11T09:26:45.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.868+0000 7f96d3dff700 1 -- start start 2024-01-11T09:26:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.869+0000 7f96d3dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f96cc0ca7b0 0x7f96cc0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.869+0000 7f96d3dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96cc15d640 con 0x7f96cc159e80 2024-01-11T09:26:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.870+0000 7f96d3dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96cc15a5d0 con 0x7f96cc0ca3e0 2024-01-11T09:26:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.870+0000 7f96d3dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96cc15ae70 con 0x7f96cc0ca7b0 2024-01-11T09:26:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.870+0000 7f96d139a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f96cc0ca7b0 0x7f96cc0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.870+0000 7f96d239c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f96cc159e80 0x7f96cc15dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40410/0 (socket says 172.21.15.27:40410) 2024-01-11T09:26:45.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.870+0000 7f96d239c700 1 -- 172.21.15.27:0/340434909 learned_addr learned my addr 172.21.15.27:0/340434909 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.870+0000 7f96d139a700 1 -- 172.21.15.27:0/340434909 >> v1:172.21.15.77:6789/0 conn(0x7f96cc0ca3e0 legacy=0x7f96cc0c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.870+0000 7f96d139a700 1 -- 172.21.15.27:0/340434909 >> v1:172.21.15.27:6789/0 conn(0x7f96cc159e80 legacy=0x7f96cc15dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.870+0000 7f96d139a700 1 -- 172.21.15.27:0/340434909 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96cc15a8b0 con 0x7f96cc0ca7b0 2024-01-11T09:26:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.871+0000 7f96d139a700 1 --2- 172.21.15.27:0/340434909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96cc0ca7b0 0x7f96cc0cca20 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009910 tx=0x7f96bc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad8dbb83b20509e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.871+0000 7f96d0b99700 1 -- 172.21.15.27:0/340434909 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc00b820 con 0x7f96cc0ca7b0 2024-01-11T09:26:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.871+0000 7f96d0b99700 1 -- 172.21.15.27:0/340434909 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96bc00b980 con 0x7f96cc0ca7b0 2024-01-11T09:26:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.871+0000 7f96d0b99700 1 -- 172.21.15.27:0/340434909 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc00bd20 con 0x7f96cc0ca7b0 2024-01-11T09:26:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.871+0000 7f96d3dff700 1 -- 172.21.15.27:0/340434909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96cc0ca7b0 msgr2=0x7f96cc0cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.871+0000 7f96d3dff700 1 --2- 172.21.15.27:0/340434909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96cc0ca7b0 0x7f96cc0cca20 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009910 tx=0x7f96bc0091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.871+0000 7f96d3dff700 1 -- 172.21.15.27:0/340434909 shutdown_connections 2024-01-11T09:26:45.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.871+0000 7f96d3dff700 1 --2- 172.21.15.27:0/340434909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96cc0ca7b0 0x7f96cc0cca20 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:45.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.871+0000 7f96d3dff700 1 -- 172.21.15.27:0/340434909 >> 172.21.15.27:0/340434909 conn(0x7f96cc0ce3e0 msgr2=0x7f96cc0ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:45.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.872+0000 7f96d3dff700 1 -- 172.21.15.27:0/340434909 shutdown_connections 2024-01-11T09:26:45.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.872+0000 7f96d3dff700 1 -- 172.21.15.27:0/340434909 wait complete. 2024-01-11T09:26:45.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.873+0000 7f96d3dff700 1 Processor -- start 2024-01-11T09:26:45.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.873+0000 7f96d3dff700 1 -- start start 2024-01-11T09:26:45.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.874+0000 7f96d3dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96cc0ca3e0 0x7f96cc1584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:45.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.874+0000 7f96d3dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96cc0ca7b0 0x7f96cc15bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:45.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.874+0000 7f96d3dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96cc159e80 0x7f96cc14fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:45.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.874+0000 7f96d3dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc158d90 con 0x7f96cc0ca7b0 2024-01-11T09:26:45.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.874+0000 7f96d3dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc0b25e0 con 0x7f96cc159e80 2024-01-11T09:26:45.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.874+0000 7f96d3dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc0b2880 con 0x7f96cc0ca3e0 2024-01-11T09:26:45.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.874+0000 7f96d139a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96cc0ca7b0 0x7f96cc15bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:45.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.874+0000 7f96d139a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96cc0ca7b0 0x7f96cc15bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32982/0 (socket says 172.21.15.27:32982) 2024-01-11T09:26:45.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.875+0000 7f96d139a700 1 -- 172.21.15.27:0/1267382955 learned_addr learned my addr 172.21.15.27:0/1267382955 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:45.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.875+0000 7f96d239c700 1 --2- 172.21.15.27:0/1267382955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96cc159e80 0x7f96cc14fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:45.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.875+0000 7f96d1b9b700 1 --2- 172.21.15.27:0/1267382955 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96cc0ca3e0 0x7f96cc1584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:45.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.875+0000 7f96d139a700 1 -- 172.21.15.27:0/1267382955 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96cc0ca3e0 msgr2=0x7f96cc1584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:45.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.875+0000 7f96d139a700 1 --2- 172.21.15.27:0/1267382955 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96cc0ca3e0 0x7f96cc1584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:45.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.875+0000 7f96d139a700 1 -- 172.21.15.27:0/1267382955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96cc159e80 msgr2=0x7f96cc14fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:45.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.875+0000 7f96d139a700 1 --2- 172.21.15.27:0/1267382955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96cc159e80 0x7f96cc14fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:45.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.875+0000 7f96d139a700 1 -- 172.21.15.27:0/1267382955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96bc009420 con 0x7f96cc0ca7b0 2024-01-11T09:26:45.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.875+0000 7f96d1b9b700 1 --2- 172.21.15.27:0/1267382955 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96cc0ca3e0 0x7f96cc1584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:26:45.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.875+0000 7f96d239c700 1 --2- 172.21.15.27:0/1267382955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96cc159e80 0x7f96cc14fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:45.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.875+0000 7f96d139a700 1 --2- 172.21.15.27:0/1267382955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96cc0ca7b0 0x7f96cc15bcb0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f96cc064980 tx=0x7f96bc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:45.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.875+0000 7f96c1ffb700 1 -- 172.21.15.27:0/1267382955 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc024440 con 0x7f96cc0ca7b0 2024-01-11T09:26:45.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.876+0000 7f96c1ffb700 1 -- 172.21.15.27:0/1267382955 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96bc0245a0 con 0x7f96cc0ca7b0 2024-01-11T09:26:45.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.876+0000 7f96c1ffb700 1 -- 172.21.15.27:0/1267382955 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc024960 con 0x7f96cc0ca7b0 2024-01-11T09:26:45.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.876+0000 7f96d3dff700 1 -- 172.21.15.27:0/1267382955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96cc0b2b60 con 0x7f96cc0ca7b0 2024-01-11T09:26:45.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.876+0000 7f96d3dff700 1 -- 172.21.15.27:0/1267382955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96cc0b3120 con 0x7f96cc0ca7b0 2024-01-11T09:26:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.877+0000 7f96d3dff700 1 -- 172.21.15.27:0/1267382955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96cc0b3310 con 0x7f96cc0ca7b0 2024-01-11T09:26:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.878+0000 7f96c1ffb700 1 -- 172.21.15.27:0/1267382955 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f96bc01a420 con 0x7f96cc0ca7b0 2024-01-11T09:26:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.878+0000 7f96c1ffb700 1 --2- 172.21.15.27:0/1267382955 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f96b0066f30 0x7f96b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.879+0000 7f96c1ffb700 1 -- 172.21.15.27:0/1267382955 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f96bc0865a0 con 0x7f96cc0ca7b0 2024-01-11T09:26:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.879+0000 7f96d1b9b700 1 --2- 172.21.15.27:0/1267382955 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f96b0066f30 0x7f96b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:45.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.880+0000 7f96d1b9b700 1 --2- 172.21.15.27:0/1267382955 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f96b0066f30 0x7f96b00693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f96c8001700 tx=0x7f96c8009040 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:45.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:45.883+0000 7f96c1ffb700 1 -- 172.21.15.27:0/1267382955 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f96bc053730 con 0x7f96cc0ca7b0 2024-01-11T09:26:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.086+0000 7f96d3dff700 1 -- 172.21.15.27:0/1267382955 --> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f96cc0718d0 con 0x7f96b0066f30 2024-01-11T09:26:46.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.097+0000 7f96c1ffb700 1 -- 172.21.15.27:0/1267382955 <== mgr.8542 v2:172.21.15.77:6836/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+279785 (secure 0 0 0) 0x7f96cc0718d0 con 0x7f96b0066f30 2024-01-11T09:26:46.098 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:26:46.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.101+0000 7f96d3dff700 1 -- 172.21.15.27:0/1267382955 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f96b0066f30 msgr2=0x7f96b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:46.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.101+0000 7f96d3dff700 1 --2- 172.21.15.27:0/1267382955 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f96b0066f30 0x7f96b00693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f96c8001700 tx=0x7f96c8009040 comp rx=0 tx=0).stop 2024-01-11T09:26:46.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.101+0000 7f96d3dff700 1 -- 172.21.15.27:0/1267382955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96cc0ca7b0 msgr2=0x7f96cc15bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:46.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.101+0000 7f96d3dff700 1 --2- 172.21.15.27:0/1267382955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96cc0ca7b0 0x7f96cc15bcb0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f96cc064980 tx=0x7f96bc017000 comp rx=0 tx=0).stop 2024-01-11T09:26:46.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.101+0000 7f96d3dff700 1 -- 172.21.15.27:0/1267382955 shutdown_connections 2024-01-11T09:26:46.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.102+0000 7f96d3dff700 1 --2- 172.21.15.27:0/1267382955 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f96b0066f30 0x7f96b00693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.102+0000 7f96d3dff700 1 --2- 172.21.15.27:0/1267382955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96cc159e80 0x7f96cc14fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.102+0000 7f96d3dff700 1 --2- 172.21.15.27:0/1267382955 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96cc0ca3e0 0x7f96cc1584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.102+0000 7f96d3dff700 1 --2- 172.21.15.27:0/1267382955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96cc0ca7b0 0x7f96cc15bcb0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.102+0000 7f96d3dff700 1 -- 172.21.15.27:0/1267382955 >> 172.21.15.27:0/1267382955 conn(0x7f96cc0ce3e0 msgr2=0x7f96cc069790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:46.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.102+0000 7f96d3dff700 1 -- 172.21.15.27:0/1267382955 shutdown_connections 2024-01-11T09:26:46.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.102+0000 7f96d3dff700 1 -- 172.21.15.27:0/1267382955 wait complete. 2024-01-11T09:26:46.101 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:26:46.116 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-11T09:26:44.293610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590514,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1616,"num_read_kb":3366,"num_write":482,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":299,"ondisk_log_size":299,"up":70,"acting":70,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":2469260,"kb_used_data":17228,"kb_used_omap":0,"kb_used_meta":2451968,"kb_avail":752505460,"statfs":{"total":773094113280,"available":770565591040,"internally_reserved":0,"allocated":17641472,"data_stored":8936623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2510815232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":146,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.621462"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.710680+0000","last_change":"2024-01-11T09:26:41.703650+0000","last_active":"2024-01-11T09:26:41.710680+0000","last_peered":"2024-01-11T09:26:41.710680+0000","last_clean":"2024-01-11T09:26:41.710680+0000","last_became_active":"2024-01-11T09:26:41.699944+0000","last_became_peered":"2024-01-11T09:26:41.699944+0000","last_unstale":"2024-01-11T09:26:41.710680+0000","last_undegraded":"2024-01-11T09:26:41.710680+0000","last_fullsized":"2024-01-11T09:26:41.710680+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:46:54.067657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.707638+0000","last_change":"2024-01-11T09:26:41.704825+0000","last_active":"2024-01-11T09:26:41.707638+0000","last_peered":"2024-01-11T09:26:41.707638+0000","last_clean":"2024-01-11T09:26:41.707638+0000","last_became_active":"2024-01-11T09:26:41.704571+0000","last_became_peered":"2024-01-11T09:26:41.704571+0000","last_unstale":"2024-01-11T09:26:41.707638+0000","last_undegraded":"2024-01-11T09:26:41.707638+0000","last_fullsized":"2024-01-11T09:26:41.707638+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:55:59.520132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.715547+0000","last_change":"2024-01-11T09:26:41.710134+0000","last_active":"2024-01-11T09:26:41.715547+0000","last_peered":"2024-01-11T09:26:41.715547+0000","last_clean":"2024-01-11T09:26:41.715547+0000","last_became_active":"2024-01-11T09:26:41.709506+0000","last_became_peered":"2024-01-11T09:26:41.709506+0000","last_unstale":"2024-01-11T09:26:41.715547+0000","last_undegraded":"2024-01-11T09:26:41.715547+0000","last_fullsized":"2024-01-11T09:26:41.715547+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:50:46.665371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.713561+0000","last_change":"2024-01-11T09:26:41.704659+0000","last_active":"2024-01-11T09:26:41.713561+0000","last_peered":"2024-01-11T09:26:41.713561+0000","last_clean":"2024-01-11T09:26:41.713561+0000","last_became_active":"2024-01-11T09:26:41.704407+0000","last_became_peered":"2024-01-11T09:26:41.704407+0000","last_unstale":"2024-01-11T09:26:41.713561+0000","last_undegraded":"2024-01-11T09:26:41.713561+0000","last_fullsized":"2024-01-11T09:26:41.713561+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:02:58.472920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.715569+0000","last_change":"2024-01-11T09:26:41.710297+0000","last_active":"2024-01-11T09:26:41.715569+0000","last_peered":"2024-01-11T09:26:41.715569+0000","last_clean":"2024-01-11T09:26:41.715569+0000","last_became_active":"2024-01-11T09:26:41.709890+0000","last_became_peered":"2024-01-11T09:26:41.709890+0000","last_unstale":"2024-01-11T09:26:41.715569+0000","last_undegraded":"2024-01-11T09:26:41.715569+0000","last_fullsized":"2024-01-11T09:26:41.715569+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:46:18.959911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.721226+0000","last_change":"2024-01-11T09:26:41.718708+0000","last_active":"2024-01-11T09:26:41.721226+0000","last_peered":"2024-01-11T09:26:41.721226+0000","last_clean":"2024-01-11T09:26:41.721226+0000","last_became_active":"2024-01-11T09:26:41.717412+0000","last_became_peered":"2024-01-11T09:26:41.717412+0000","last_unstale":"2024-01-11T09:26:41.721226+0000","last_undegraded":"2024-01-11T09:26:41.721226+0000","last_fullsized":"2024-01-11T09:26:41.721226+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:42:50.270447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.712408+0000","last_change":"2024-01-11T09:26:41.703862+0000","last_active":"2024-01-11T09:26:41.712408+0000","last_peered":"2024-01-11T09:26:41.712408+0000","last_clean":"2024-01-11T09:26:41.712408+0000","last_became_active":"2024-01-11T09:26:41.703569+0000","last_became_peered":"2024-01-11T09:26:41.703569+0000","last_unstale":"2024-01-11T09:26:41.712408+0000","last_undegraded":"2024-01-11T09:26:41.712408+0000","last_fullsized":"2024-01-11T09:26:41.712408+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:23:30.649071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.710732+0000","last_change":"2024-01-11T09:26:41.703657+0000","last_active":"2024-01-11T09:26:41.710732+0000","last_peered":"2024-01-11T09:26:41.710732+0000","last_clean":"2024-01-11T09:26:41.710732+0000","last_became_active":"2024-01-11T09:26:41.701059+0000","last_became_peered":"2024-01-11T09:26:41.701059+0000","last_unstale":"2024-01-11T09:26:41.710732+0000","last_undegraded":"2024-01-11T09:26:41.710732+0000","last_fullsized":"2024-01-11T09:26:41.710732+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:02:54.487205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"166'1","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.713099+0000","last_change":"2024-01-11T09:26:40.711040+0000","last_active":"2024-01-11T09:26:41.713099+0000","last_peered":"2024-01-11T09:26:41.713099+0000","last_clean":"2024-01-11T09:26:41.713099+0000","last_became_active":"2024-01-11T09:26:40.710725+0000","last_became_peered":"2024-01-11T09:26:40.710725+0000","last_unstale":"2024-01-11T09:26:41.713099+0000","last_undegraded":"2024-01-11T09:26:41.713099+0000","last_fullsized":"2024-01-11T09:26:41.713099+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:22:02.081826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.719009+0000","last_change":"2024-01-11T09:26:41.715363+0000","last_active":"2024-01-11T09:26:41.719009+0000","last_peered":"2024-01-11T09:26:41.719009+0000","last_clean":"2024-01-11T09:26:41.719009+0000","last_became_active":"2024-01-11T09:26:41.714216+0000","last_became_peered":"2024-01-11T09:26:41.714216+0000","last_unstale":"2024-01-11T09:26:41.719009+0000","last_undegraded":"2024-01-11T09:26:41.719009+0000","last_fullsized":"2024-01-11T09:26:41.719009+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:20.319578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1a","version":"166'1","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.710617+0000","last_change":"2024-01-11T09:26:40.706966+0000","last_active":"2024-01-11T09:26:41.710617+0000","last_peered":"2024-01-11T09:26:41.710617+0000","last_clean":"2024-01-11T09:26:41.710617+0000","last_became_active":"2024-01-11T09:26:40.706678+0000","last_became_peered":"2024-01-11T09:26:40.706678+0000","last_unstale":"2024-01-11T09:26:41.710617+0000","last_undegraded":"2024-01-11T09:26:41.710617+0000","last_fullsized":"2024-01-11T09:26:41.710617+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:30:57.214674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.705890+0000","last_change":"2024-01-11T09:26:41.702886+0000","last_active":"2024-01-11T09:26:41.705890+0000","last_peered":"2024-01-11T09:26:41.705890+0000","last_clean":"2024-01-11T09:26:41.705890+0000","last_became_active":"2024-01-11T09:26:41.700541+0000","last_became_peered":"2024-01-11T09:26:41.700541+0000","last_unstale":"2024-01-11T09:26:41.705890+0000","last_undegraded":"2024-01-11T09:26:41.705890+0000","last_fullsized":"2024-01-11T09:26:41.705890+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:22:48.073902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.709303+0000","last_change":"2024-01-11T09:26:41.701311+0000","last_active":"2024-01-11T09:26:41.709303+0000","last_peered":"2024-01-11T09:26:41.709303+0000","last_clean":"2024-01-11T09:26:41.709303+0000","last_became_active":"2024-01-11T09:26:41.701030+0000","last_became_peered":"2024-01-11T09:26:41.701030+0000","last_unstale":"2024-01-11T09:26:41.709303+0000","last_undegraded":"2024-01-11T09:26:41.709303+0000","last_fullsized":"2024-01-11T09:26:41.709303+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:28.437611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.715606+0000","last_change":"2024-01-11T09:26:41.712115+0000","last_active":"2024-01-11T09:26:41.715606+0000","last_peered":"2024-01-11T09:26:41.715606+0000","last_clean":"2024-01-11T09:26:41.715606+0000","last_became_active":"2024-01-11T09:26:41.711851+0000","last_became_peered":"2024-01-11T09:26:41.711851+0000","last_unstale":"2024-01-11T09:26:41.715606+0000","last_undegraded":"2024-01-11T09:26:41.715606+0000","last_fullsized":"2024-01-11T09:26:41.715606+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:04:09.551826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.686627+0000","last_change":"2024-01-11T09:26:40.704595+0000","last_active":"2024-01-11T09:26:41.686627+0000","last_peered":"2024-01-11T09:26:41.686627+0000","last_clean":"2024-01-11T09:26:41.686627+0000","last_became_active":"2024-01-11T09:26:40.704338+0000","last_became_peered":"2024-01-11T09:26:40.704338+0000","last_unstale":"2024-01-11T09:26:41.686627+0000","last_undegraded":"2024-01-11T09:26:41.686627+0000","last_fullsized":"2024-01-11T09:26:41.686627+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:07:41.508678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.715623+0000","last_change":"2024-01-11T09:26:41.713716+0000","last_active":"2024-01-11T09:26:41.715623+0000","last_peered":"2024-01-11T09:26:41.715623+0000","last_clean":"2024-01-11T09:26:41.715623+0000","last_became_active":"2024-01-11T09:26:41.713443+0000","last_became_peered":"2024-01-11T09:26:41.713443+0000","last_unstale":"2024-01-11T09:26:41.715623+0000","last_undegraded":"2024-01-11T09:26:41.715623+0000","last_fullsized":"2024-01-11T09:26:41.715623+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:00:31.275152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"166'2","reported_seq":12,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.716765+0000","last_change":"2024-01-11T09:26:40.703202+0000","last_active":"2024-01-11T09:26:41.716765+0000","last_peered":"2024-01-11T09:26:41.716765+0000","last_clean":"2024-01-11T09:26:41.716765+0000","last_became_active":"2024-01-11T09:26:40.702745+0000","last_became_peered":"2024-01-11T09:26:40.702745+0000","last_unstale":"2024-01-11T09:26:41.716765+0000","last_undegraded":"2024-01-11T09:26:41.716765+0000","last_fullsized":"2024-01-11T09:26:41.716765+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:34:05.586284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.716797+0000","last_change":"2024-01-11T09:26:41.708152+0000","last_active":"2024-01-11T09:26:41.716797+0000","last_peered":"2024-01-11T09:26:41.716797+0000","last_clean":"2024-01-11T09:26:41.716797+0000","last_became_active":"2024-01-11T09:26:41.707904+0000","last_became_peered":"2024-01-11T09:26:41.707904+0000","last_unstale":"2024-01-11T09:26:41.716797+0000","last_undegraded":"2024-01-11T09:26:41.716797+0000","last_fullsized":"2024-01-11T09:26:41.716797+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:18:08.104121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.698278+0000","last_change":"2024-01-11T09:26:40.706553+0000","last_active":"2024-01-11T09:26:41.698278+0000","last_peered":"2024-01-11T09:26:41.698278+0000","last_clean":"2024-01-11T09:26:41.698278+0000","last_became_active":"2024-01-11T09:26:40.705001+0000","last_became_peered":"2024-01-11T09:26:40.705001+0000","last_unstale":"2024-01-11T09:26:41.698278+0000","last_undegraded":"2024-01-11T09:26:41.698278+0000","last_fullsized":"2024-01-11T09:26:41.698278+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:42:58.382613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.715433+0000","last_change":"2024-01-11T09:26:41.709029+0000","last_active":"2024-01-11T09:26:41.715433+0000","last_peered":"2024-01-11T09:26:41.715433+0000","last_clean":"2024-01-11T09:26:41.715433+0000","last_became_active":"2024-01-11T09:26:41.708765+0000","last_became_peered":"2024-01-11T09:26:41.708765+0000","last_unstale":"2024-01-11T09:26:41.715433+0000","last_undegraded":"2024-01-11T09:26:41.715433+0000","last_fullsized":"2024-01-11T09:26:41.715433+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:05:18.067025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.721869+0000","last_change":"2024-01-11T09:26:41.719864+0000","last_active":"2024-01-11T09:26:41.721869+0000","last_peered":"2024-01-11T09:26:41.721869+0000","last_clean":"2024-01-11T09:26:41.721869+0000","last_became_active":"2024-01-11T09:26:41.719612+0000","last_became_peered":"2024-01-11T09:26:41.719612+0000","last_unstale":"2024-01-11T09:26:41.721869+0000","last_undegraded":"2024-01-11T09:26:41.721869+0000","last_fullsized":"2024-01-11T09:26:41.721869+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:17:30.608645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.f","version":"166'1","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.708488+0000","last_change":"2024-01-11T09:26:40.703362+0000","last_active":"2024-01-11T09:26:41.708488+0000","last_peered":"2024-01-11T09:26:41.708488+0000","last_clean":"2024-01-11T09:26:41.708488+0000","last_became_active":"2024-01-11T09:26:40.701980+0000","last_became_peered":"2024-01-11T09:26:40.701980+0000","last_unstale":"2024-01-11T09:26:41.708488+0000","last_undegraded":"2024-01-11T09:26:41.708488+0000","last_fullsized":"2024-01-11T09:26:41.708488+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:29:28.407605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.e","version":"166'2","reported_seq":12,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.724285+0000","last_change":"2024-01-11T09:26:40.702279+0000","last_active":"2024-01-11T09:26:41.724285+0000","last_peered":"2024-01-11T09:26:41.724285+0000","last_clean":"2024-01-11T09:26:41.724285+0000","last_became_active":"2024-01-11T09:26:40.701971+0000","last_became_peered":"2024-01-11T09:26:40.701971+0000","last_unstale":"2024-01-11T09:26:41.724285+0000","last_undegraded":"2024-01-11T09:26:41.724285+0000","last_fullsized":"2024-01-11T09:26:41.724285+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:33:28.856947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.721140+0000","last_change":"2024-01-11T09:26:41.716816+0000","last_active":"2024-01-11T09:26:41.721140+0000","last_peered":"2024-01-11T09:26:41.721140+0000","last_clean":"2024-01-11T09:26:41.721140+0000","last_became_active":"2024-01-11T09:26:41.716550+0000","last_became_peered":"2024-01-11T09:26:41.716550+0000","last_unstale":"2024-01-11T09:26:41.721140+0000","last_undegraded":"2024-01-11T09:26:41.721140+0000","last_fullsized":"2024-01-11T09:26:41.721140+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:48:24.611840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.686331+0000","last_change":"2024-01-11T09:26:40.709183+0000","last_active":"2024-01-11T09:26:41.686331+0000","last_peered":"2024-01-11T09:26:41.686331+0000","last_clean":"2024-01-11T09:26:41.686331+0000","last_became_active":"2024-01-11T09:26:40.708875+0000","last_became_peered":"2024-01-11T09:26:40.708875+0000","last_unstale":"2024-01-11T09:26:41.686331+0000","last_undegraded":"2024-01-11T09:26:41.686331+0000","last_fullsized":"2024-01-11T09:26:41.686331+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:21:05.322778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.695709+0000","last_change":"2024-01-11T09:26:40.702598+0000","last_active":"2024-01-11T09:26:41.695709+0000","last_peered":"2024-01-11T09:26:41.695709+0000","last_clean":"2024-01-11T09:26:41.695709+0000","last_became_active":"2024-01-11T09:26:40.702301+0000","last_became_peered":"2024-01-11T09:26:40.702301+0000","last_unstale":"2024-01-11T09:26:41.695709+0000","last_undegraded":"2024-01-11T09:26:41.695709+0000","last_fullsized":"2024-01-11T09:26:41.695709+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:23:29.641605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.714063+0000","last_change":"2024-01-11T09:26:41.706765+0000","last_active":"2024-01-11T09:26:41.714063+0000","last_peered":"2024-01-11T09:26:41.714063+0000","last_clean":"2024-01-11T09:26:41.714063+0000","last_became_active":"2024-01-11T09:26:41.706458+0000","last_became_peered":"2024-01-11T09:26:41.706458+0000","last_unstale":"2024-01-11T09:26:41.714063+0000","last_undegraded":"2024-01-11T09:26:41.714063+0000","last_fullsized":"2024-01-11T09:26:41.714063+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:26:44.657352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.715627+0000","last_change":"2024-01-11T09:26:41.713497+0000","last_active":"2024-01-11T09:26:41.715627+0000","last_peered":"2024-01-11T09:26:41.715627+0000","last_clean":"2024-01-11T09:26:41.715627+0000","last_became_active":"2024-01-11T09:26:41.713226+0000","last_became_peered":"2024-01-11T09:26:41.713226+0000","last_unstale":"2024-01-11T09:26:41.715627+0000","last_undegraded":"2024-01-11T09:26:41.715627+0000","last_fullsized":"2024-01-11T09:26:41.715627+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:44:23.252777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.712593+0000","last_change":"2024-01-11T09:26:41.706364+0000","last_active":"2024-01-11T09:26:41.712593+0000","last_peered":"2024-01-11T09:26:41.712593+0000","last_clean":"2024-01-11T09:26:41.712593+0000","last_became_active":"2024-01-11T09:26:41.706042+0000","last_became_peered":"2024-01-11T09:26:41.706042+0000","last_unstale":"2024-01-11T09:26:41.712593+0000","last_undegraded":"2024-01-11T09:26:41.712593+0000","last_fullsized":"2024-01-11T09:26:41.712593+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:58:29.815846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.714404+0000","last_change":"2024-01-11T09:26:41.707143+0000","last_active":"2024-01-11T09:26:41.714404+0000","last_peered":"2024-01-11T09:26:41.714404+0000","last_clean":"2024-01-11T09:26:41.714404+0000","last_became_active":"2024-01-11T09:26:41.706836+0000","last_became_peered":"2024-01-11T09:26:41.706836+0000","last_unstale":"2024-01-11T09:26:41.714404+0000","last_undegraded":"2024-01-11T09:26:41.714404+0000","last_fullsized":"2024-01-11T09:26:41.714404+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:31:19.426090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:39.664170+0000","last_change":"2024-01-11T09:26:39.664170+0000","last_active":"2024-01-11T09:26:39.664170+0000","last_peered":"2024-01-11T09:26:39.664170+0000","last_clean":"2024-01-11T09:26:39.664170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:39.664170+0000","last_undegraded":"2024-01-11T09:26:39.664170+0000","last_fullsized":"2024-01-11T09:26:39.664170+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.697911+0000","last_change":"2024-01-11T09:26:40.703957+0000","last_active":"2024-01-11T09:26:41.697911+0000","last_peered":"2024-01-11T09:26:41.697911+0000","last_clean":"2024-01-11T09:26:41.697911+0000","last_became_active":"2024-01-11T09:26:40.703671+0000","last_became_peered":"2024-01-11T09:26:40.703671+0000","last_unstale":"2024-01-11T09:26:41.697911+0000","last_undegraded":"2024-01-11T09:26:41.697911+0000","last_fullsized":"2024-01-11T09:26:41.697911+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:38:52.866733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.715636+0000","last_change":"2024-01-11T09:26:41.712143+0000","last_active":"2024-01-11T09:26:41.715636+0000","last_peered":"2024-01-11T09:26:41.715636+0000","last_clean":"2024-01-11T09:26:41.715636+0000","last_became_active":"2024-01-11T09:26:41.711868+0000","last_became_peered":"2024-01-11T09:26:41.711868+0000","last_unstale":"2024-01-11T09:26:41.715636+0000","last_undegraded":"2024-01-11T09:26:41.715636+0000","last_fullsized":"2024-01-11T09:26:41.715636+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:42:39.118184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.686144+0000","last_change":"2024-01-11T09:26:40.706610+0000","last_active":"2024-01-11T09:26:41.686144+0000","last_peered":"2024-01-11T09:26:41.686144+0000","last_clean":"2024-01-11T09:26:41.686144+0000","last_became_active":"2024-01-11T09:26:40.706341+0000","last_became_peered":"2024-01-11T09:26:40.706341+0000","last_unstale":"2024-01-11T09:26:41.686144+0000","last_undegraded":"2024-01-11T09:26:41.686144+0000","last_fullsized":"2024-01-11T09:26:41.686144+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:34:27.129163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-11T09:26:40.671636+0000","last_change":"2024-01-11T09:26:40.671636+0000","last_active":"2024-01-11T09:26:40.671636+0000","last_peered":"2024-01-11T09:26:40.671636+0000","last_clean":"2024-01-11T09:26:40.671636+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T09:26:40.671636+0000","last_undegraded":"2024-01-11T09:26:40.671636+0000","last_fullsized":"2024-01-11T09:26:40.671636+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.15","version":"166'3","reported_seq":13,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.724477+0000","last_change":"2024-01-11T09:26:40.702628+0000","last_active":"2024-01-11T09:26:41.724477+0000","last_peered":"2024-01-11T09:26:41.724477+0000","last_clean":"2024-01-11T09:26:41.724477+0000","last_became_active":"2024-01-11T09:26:40.702323+0000","last_became_peered":"2024-01-11T09:26:40.702323+0000","last_unstale":"2024-01-11T09:26:41.724477+0000","last_undegraded":"2024-01-11T09:26:41.724477+0000","last_fullsized":"2024-01-11T09:26:41.724477+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:19.312112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"157'289","reported_seq":884,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-01-11T09:26:33.347326+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:26:33.347326+0000","last_peered":"2024-01-11T09:26:33.347326+0000","last_clean":"2024-01-11T09:26:33.347326+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:26:33.347326+0000","last_undegraded":"2024-01-11T09:26:33.347326+0000","last_fullsized":"2024-01-11T09:26:33.347326+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1616,"num_read_kb":3366,"num_write":471,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":44,"acting":44,"num_store_stats":3},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":146,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":24,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1616,"num_read_kb":3366,"num_write":471,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":289,"ondisk_log_size":289,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312144,"kb_used_data":2568,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059696,"statfs":{"total":96636764160,"available":96317128704,"internally_reserved":0,"allocated":2629632,"data_stored":1550318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.75800000000000001,"15min":0.73399999999999999},"min":{"1min":0.52400000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.97999999999999998,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.74299999999999999,"15min":0.73299999999999998},"min":{"1min":0.55500000000000005,"5min":0.52000000000000002,"15min":0.45600000000000002},"max":{"1min":0.91500000000000004,"5min":0.99299999999999999,"15min":1.028},"last":0.498}]},{"osd":1,"last update":"Thu Jan 11 09:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.73799999999999999,"15min":0.72699999999999998},"min":{"1min":0.502,"5min":0.497,"15min":0.42399999999999999},"max":{"1min":0.91300000000000003,"5min":1.034,"15min":1.034},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76500000000000001,"15min":0.754},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.49299999999999999},"max":{"1min":0.96099999999999997,"5min":1.0069999999999999,"15min":1.0409999999999999},"last":0.54900000000000004}]},{"osd":2,"last update":"Thu Jan 11 09:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.75900000000000001,"15min":0.754},"min":{"1min":0.53900000000000003,"5min":0.52100000000000002,"15min":0.46899999999999997},"max":{"1min":0.93400000000000005,"5min":1.0089999999999999,"15min":1.0880000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.74299999999999999,"15min":0.73899999999999999},"min":{"1min":0.53700000000000003,"5min":0.51000000000000001,"15min":0.50800000000000001},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.57299999999999995}]},{"osd":3,"last update":"Thu Jan 11 09:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.77000000000000002,"15min":0.76500000000000001},"min":{"1min":0.57499999999999996,"5min":0.53700000000000003,"15min":0.52700000000000002},"max":{"1min":0.94999999999999996,"5min":1.046,"15min":1.056},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.76200000000000001,"15min":0.76700000000000002},"min":{"1min":0.54400000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.997,"5min":1.006,"15min":1.012},"last":0.71699999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59699999999999998,"15min":0.60199999999999998},"min":{"1min":0.46400000000000002,"5min":0.45300000000000001,"15min":0.40600000000000003},"max":{"1min":0.746,"5min":0.94599999999999995,"15min":1.004},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.625,"5min":0.63900000000000001,"15min":0.64700000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.82099999999999995,"5min":1.016,"15min":1.0940000000000001},"last":0.83899999999999997}]},{"osd":5,"last update":"Thu Jan 11 09:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63100000000000001,"15min":0.64500000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81200000000000006,"5min":0.96499999999999997,"15min":1.0309999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.67400000000000004,"15min":0.68300000000000005},"min":{"1min":0.53800000000000003,"5min":0.41999999999999998,"15min":0.41299999999999998},"max":{"1min":0.879,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.77000000000000002}]},{"osd":6,"last update":"Thu Jan 11 09:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.76800000000000002,"15min":0.77800000000000002},"min":{"1min":0.61499999999999999,"5min":0.50900000000000001,"15min":0.47299999999999998},"max":{"1min":0.90000000000000002,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.748,"5min":0.749,"15min":0.75900000000000001},"min":{"1min":0.44900000000000001,"5min":0.40999999999999998,"15min":0.40100000000000002},"max":{"1min":1.014,"5min":1.073,"15min":1.073},"last":0.94299999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575753,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307528,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064312,"statfs":{"total":96636764160,"available":96321855488,"internally_reserved":0,"allocated":2097152,"data_stored":985439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.70599999999999996,"15min":0.70499999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.40500000000000003},"max":{"1min":0.91200000000000003,"5min":0.98099999999999998,"15min":1.095},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.71799999999999997,"15min":0.71999999999999997},"min":{"1min":0.53500000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.92700000000000005,"5min":0.97899999999999998,"15min":1.0489999999999999},"last":0.745}]},{"osd":1,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.72599999999999998,"15min":0.72699999999999998},"min":{"1min":0.54100000000000004,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93999999999999995,"5min":0.96499999999999997,"15min":1.077},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71399999999999997,"15min":0.71899999999999997},"min":{"1min":0.49099999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.94299999999999995,"5min":0.96999999999999997,"15min":1.01},"last":0.85899999999999999}]},{"osd":2,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.74399999999999999,"15min":0.73599999999999999},"min":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.49399999999999999},"max":{"1min":1.006,"5min":1.0129999999999999,"15min":1.0309999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72999999999999998,"15min":0.75},"min":{"1min":0.54500000000000004,"5min":0.51600000000000001,"15min":0.505},"max":{"1min":0.89300000000000002,"5min":0.94599999999999995,"15min":1.161},"last":0.95899999999999996}]},{"osd":3,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76200000000000001,"15min":0.76000000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.47499999999999998},"max":{"1min":0.91300000000000003,"5min":0.999,"15min":1.0329999999999999},"last":1.026},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76300000000000001,"15min":0.76700000000000002},"min":{"1min":0.57199999999999995,"5min":0.53700000000000003,"15min":0.503},"max":{"1min":0.91300000000000003,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.97599999999999998}]},{"osd":4,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.45900000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76100000000000001,"5min":0.82999999999999996,"15min":1.0469999999999999},"last":1.006},{"interface":"front","average":{"1min":0.623,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85799999999999998,"5min":0.97199999999999998,"15min":1.0569999999999999},"last":1.081}]},{"osd":5,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.67600000000000005,"15min":0.66700000000000004},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.39200000000000002},"max":{"1min":0.89700000000000002,"5min":1.032,"15min":1.032},"last":1.05},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.69999999999999996,"15min":0.71199999999999997},"min":{"1min":0.441,"5min":0.441,"15min":0.41699999999999998},"max":{"1min":0.93300000000000005,"5min":1.052,"15min":1.1299999999999999},"last":1.0660000000000001}]},{"osd":7,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.71099999999999997,"15min":0.71899999999999997},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.435},"max":{"1min":0.96799999999999997,"5min":1.034,"15min":1.1080000000000001},"last":1.143},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.75800000000000001,"15min":0.75900000000000001},"min":{"1min":0.501,"5min":0.5,"15min":0.47299999999999998},"max":{"1min":0.95499999999999996,"5min":1.081,"15min":1.0840000000000001},"last":1.1699999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":307468,"kb_used_data":1988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064372,"statfs":{"total":96636764160,"available":96321916928,"internally_reserved":0,"allocated":2035712,"data_stored":959950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51400000000000001,"15min":0.52300000000000002},"min":{"1min":0.36899999999999999,"5min":0.313,"15min":0.30199999999999999},"max":{"1min":0.76900000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.5},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.498,"5min":0.46100000000000002,"15min":0.39900000000000002},"max":{"1min":0.79400000000000004,"5min":0.81399999999999995,"15min":0.91500000000000004},"last":0.45100000000000001}]},{"osd":2,"last update":"Thu Jan 11 09:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.50700000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.60999999999999999,"5min":0.76000000000000001,"15min":0.83699999999999997},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58299999999999996,"15min":0.58999999999999997},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.36899999999999999},"max":{"1min":0.83699999999999997,"5min":0.89800000000000002,"15min":0.91900000000000004},"last":0.56799999999999995}]},{"osd":3,"last update":"Thu Jan 11 09:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65200000000000002,"15min":0.65300000000000002},"min":{"1min":0.51900000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.77300000000000002,"5min":1.002,"15min":1.002},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58599999999999997,"15min":0.58499999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.40500000000000003},"max":{"1min":0.748,"5min":0.80700000000000005,"15min":1.075},"last":0.41099999999999998}]},{"osd":4,"last update":"Thu Jan 11 09:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.81899999999999995,"15min":0.81399999999999995},"min":{"1min":0.621,"5min":0.53100000000000003,"15min":0.46999999999999997},"max":{"1min":1.129,"5min":1.1539999999999999,"15min":1.1839999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.878,"5min":0.873,"15min":0.87},"min":{"1min":0.69599999999999995,"5min":0.63300000000000001,"15min":0.60599999999999998},"max":{"1min":1.04,"5min":1.1779999999999999,"15min":1.198},"last":0.82299999999999995}]},{"osd":5,"last update":"Thu Jan 11 09:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.82899999999999996,"15min":0.82299999999999995},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.98799999999999999,"5min":1.212,"15min":1.212},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.81200000000000006,"15min":0.81299999999999994},"min":{"1min":0.64800000000000002,"5min":0.623,"15min":0.55400000000000005},"max":{"1min":0.93999999999999995,"5min":1.0800000000000001,"15min":1.248},"last":0.67400000000000004}]},{"osd":6,"last update":"Thu Jan 11 09:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.85599999999999998,"15min":0.84999999999999998},"min":{"1min":0.68500000000000005,"5min":0.61599999999999999,"15min":0.57799999999999996},"max":{"1min":1.0169999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.874,"5min":0.85199999999999998,"15min":0.84399999999999997},"min":{"1min":0.68400000000000005,"5min":0.59199999999999997,"15min":0.54400000000000004},"max":{"1min":1.016,"5min":1.1499999999999999,"15min":1.2529999999999999},"last":0.83999999999999997}]},{"osd":7,"last update":"Thu Jan 11 09:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.879,"15min":0.86899999999999999},"min":{"1min":0.78400000000000003,"5min":0.64400000000000002,"15min":0.61699999999999999},"max":{"1min":1.083,"5min":1.2130000000000001,"15min":1.218},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.877,"5min":0.84899999999999998,"15min":0.85299999999999998},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.1180000000000001,"5min":1.1950000000000001,"15min":1.244},"last":0.76400000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575753,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303952,"kb_used_data":2568,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067888,"statfs":{"total":96636764160,"available":96325517312,"internally_reserved":0,"allocated":2629632,"data_stored":1550318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.52400000000000002,"15min":0.52000000000000002},"min":{"1min":0.311,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.70199999999999996,"5min":0.751,"15min":0.79600000000000004},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60299999999999998,"15min":0.59699999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.42499999999999999},"max":{"1min":0.72399999999999998,"5min":0.83699999999999997,"15min":0.94599999999999995},"last":0.46899999999999997}]},{"osd":2,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55300000000000005,"15min":0.55400000000000005},"min":{"1min":0.41399999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.80600000000000005,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55200000000000005,"15min":0.57499999999999996},"min":{"1min":0.41399999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89900000000000002},"last":0.42799999999999999}]},{"osd":3,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70599999999999996,"5min":0.79400000000000004,"15min":0.90600000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63400000000000001,"15min":0.64300000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.79900000000000004,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.69999999999999996}]},{"osd":4,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79200000000000004,"15min":0.81699999999999995},"min":{"1min":0.59599999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.075,"5min":1.075,"15min":1.4239999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.80800000000000005,"15min":0.81200000000000006},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.1659999999999999,"5min":1.2050000000000001,"15min":1.2749999999999999},"last":0.79100000000000004}]},{"osd":5,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.81599999999999995,"15min":0.83299999999999996},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.57399999999999995},"max":{"1min":1.0249999999999999,"5min":1.1140000000000001,"15min":1.391},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.84199999999999997,"15min":0.84099999999999997},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.151,"5min":1.151,"15min":1.484},"last":0.84199999999999997}]},{"osd":6,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.82599999999999996,"15min":0.82899999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.353},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.85399999999999998,"15min":0.86399999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.3069999999999999},"last":0.753}]},{"osd":7,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.86799999999999999,"15min":0.875},"min":{"1min":0.60899999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.125,"5min":1.2010000000000001,"15min":1.246},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.872,"15min":0.86399999999999999},"min":{"1min":0.67300000000000004,"5min":0.63600000000000001,"15min":0.59999999999999998},"max":{"1min":1.048,"5min":1.157,"15min":1.272},"last":0.68100000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575752,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":311564,"kb_used_data":1988,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060276,"statfs":{"total":96636764160,"available":96317722624,"internally_reserved":0,"allocated":2035712,"data_stored":959950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.497,"15min":0.5},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55100000000000005,"15min":0.55400000000000005},"min":{"1min":0.34599999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.47399999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.45100000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79700000000000004,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60899999999999999,"15min":0.60699999999999998},"min":{"1min":0.41199999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.82699999999999996,"5min":0.93600000000000005,"15min":1},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54900000000000004,"15min":0.55700000000000005},"min":{"1min":0.40400000000000003,"5min":0.37,"15min":0.34799999999999998},"max":{"1min":0.79500000000000004,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65600000000000003,"15min":0.66100000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.624}]},{"osd":4,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.80700000000000005,"15min":0.81399999999999995},"min":{"1min":0.68200000000000005,"5min":0.55300000000000005,"15min":0.54000000000000004},"max":{"1min":1.1220000000000001,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81299999999999994,"15min":0.79800000000000004},"min":{"1min":0.69199999999999995,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.036,"5min":1.391,"15min":1.391},"last":0.66900000000000004}]},{"osd":5,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.878,"15min":0.86899999999999999},"min":{"1min":0.73599999999999999,"5min":0.621,"15min":0.61699999999999999},"max":{"1min":1.1240000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.88800000000000001,"15min":0.86799999999999999},"min":{"1min":0.72099999999999997,"5min":0.627,"15min":0.54700000000000004},"max":{"1min":1.238,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.79900000000000004}]},{"osd":6,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.84099999999999997,"15min":0.82999999999999996},"min":{"1min":0.71199999999999997,"5min":0.57399999999999995,"15min":0.48999999999999999},"max":{"1min":1.1180000000000001,"5min":1.534,"15min":1.534},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.72099999999999997,"5min":0.59199999999999997,"15min":0.56999999999999995},"max":{"1min":1.127,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.84699999999999998}]},{"osd":7,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.66500000000000004,"5min":0.63100000000000001,"15min":0.58799999999999997},"max":{"1min":1.2010000000000001,"5min":1.55,"15min":1.55},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.90400000000000003,"15min":0.89400000000000002},"min":{"1min":0.67900000000000005,"5min":0.54900000000000004,"15min":0.53600000000000003},"max":{"1min":1.087,"5min":1.593,"15min":1.593},"last":0.753}]}]},{"osd":3,"up_from":13,"seq":55834575753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315712,"kb_used_data":2040,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94056128,"statfs":{"total":96636764160,"available":96313475072,"internally_reserved":0,"allocated":2088960,"data_stored":985383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52800000000000002,"15min":0.503},"min":{"1min":0.35899999999999999,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.81399999999999995,"15min":0.82999999999999996},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57999999999999996,"15min":0.58099999999999996},"min":{"1min":0.48499999999999999,"5min":0.45600000000000002,"15min":0.42099999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.378}]},{"osd":1,"last update":"Thu Jan 11 09:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53200000000000003,"15min":0.51800000000000002},"min":{"1min":0.39600000000000002,"5min":0.36699999999999999,"15min":0.32500000000000001},"max":{"1min":0.84299999999999997,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60399999999999998,"15min":0.59299999999999997},"min":{"1min":0.44500000000000001,"5min":0.42999999999999999,"15min":0.36299999999999999},"max":{"1min":0.80200000000000005,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.60799999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.621,"15min":0.61299999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.97799999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65600000000000003,"15min":0.64900000000000002},"min":{"1min":0.5,"5min":0.45800000000000002,"15min":0.443},"max":{"1min":0.85799999999999998,"5min":0.96299999999999997,"15min":1.0449999999999999},"last":0.67700000000000005}]},{"osd":4,"last update":"Thu Jan 11 09:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.81399999999999995,"15min":0.79000000000000004},"min":{"1min":0.63100000000000001,"5min":0.48199999999999998,"15min":0.44900000000000001},"max":{"1min":1.0289999999999999,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81999999999999995,"15min":0.80400000000000005},"min":{"1min":0.63,"5min":0.56799999999999995,"15min":0.52300000000000002},"max":{"1min":0.96099999999999997,"5min":1.0880000000000001,"15min":1.133},"last":0.70999999999999996}]},{"osd":5,"last update":"Thu Jan 11 09:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83299999999999996,"15min":0.81899999999999995},"min":{"1min":0.66000000000000003,"5min":0.63900000000000001,"15min":0.55200000000000005},"max":{"1min":1.002,"5min":1.0940000000000001,"15min":1.1499999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.85299999999999998,"15min":0.83399999999999996},"min":{"1min":0.64500000000000002,"5min":0.60999999999999999,"15min":0.55500000000000005},"max":{"1min":0.97299999999999998,"5min":1.167,"15min":1.167},"last":0.84599999999999997}]},{"osd":6,"last update":"Thu Jan 11 09:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.874,"15min":0.84999999999999998},"min":{"1min":0.73699999999999999,"5min":0.65400000000000003,"15min":0.58299999999999996},"max":{"1min":1.073,"5min":1.1699999999999999,"15min":1.1819999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83499999999999996,"15min":0.81599999999999995},"min":{"1min":0.66500000000000004,"5min":0.55900000000000005,"15min":0.49099999999999999},"max":{"1min":1.1140000000000001,"5min":1.1299999999999999,"15min":1.143},"last":0.69099999999999995}]},{"osd":7,"last update":"Thu Jan 11 09:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.88100000000000001,"15min":0.86499999999999999},"min":{"1min":0.76600000000000001,"5min":0.623,"15min":0.51800000000000002},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.196},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90700000000000003,"15min":0.88200000000000001},"min":{"1min":0.72299999999999998,"5min":0.70999999999999996,"15min":0.56499999999999995},"max":{"1min":1.0680000000000001,"5min":1.1659999999999999,"15min":1.2230000000000001},"last":0.90100000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":303372,"kb_used_data":1988,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068468,"statfs":{"total":96636764160,"available":96326111232,"internally_reserved":0,"allocated":2035712,"data_stored":959950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.81100000000000005,"15min":0.78700000000000003},"min":{"1min":0.65800000000000003,"5min":0.437,"15min":0.35099999999999998},"max":{"1min":1.006,"5min":1.0369999999999999,"15min":1.048},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.78600000000000003,"15min":0.77800000000000002},"min":{"1min":0.59199999999999997,"5min":0.55200000000000005,"15min":0.48999999999999999},"max":{"1min":0.94099999999999995,"5min":1.018,"15min":1.018},"last":0.70899999999999996}]},{"osd":1,"last update":"Thu Jan 11 09:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.747,"15min":0.73899999999999999},"min":{"1min":0.52200000000000002,"5min":0.46600000000000003,"15min":0.45300000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":1.03},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.745,"15min":0.73999999999999999},"min":{"1min":0.54400000000000004,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.89800000000000002,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.73099999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.77200000000000002,"15min":0.75800000000000001},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.42599999999999999},"max":{"1min":0.91100000000000003,"5min":1,"15min":1},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.75800000000000001},"min":{"1min":0.60099999999999998,"5min":0.49199999999999999,"15min":0.46100000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.0760000000000001},"last":0.67700000000000005}]},{"osd":3,"last update":"Thu Jan 11 09:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.73799999999999999,"15min":0.73499999999999999},"min":{"1min":0.496,"5min":0.442,"15min":0.442},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":1.0109999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.79000000000000004,"15min":0.78200000000000003},"min":{"1min":0.60899999999999999,"5min":0.54400000000000004,"15min":0.53100000000000003},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.751}]},{"osd":5,"last update":"Thu Jan 11 09:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61599999999999999,"15min":0.61699999999999999},"min":{"1min":0.45100000000000001,"5min":0.43099999999999999,"15min":0.39400000000000002},"max":{"1min":0.83799999999999997,"5min":0.89300000000000002,"15min":0.94999999999999996},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63500000000000001,"15min":0.625},"min":{"1min":0.437,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78700000000000003,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.46700000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.65200000000000002,"15min":0.65100000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.96999999999999997,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.432},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.69899999999999995,"15min":0.69699999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.96399999999999997},"last":0.498}]},{"osd":7,"last update":"Thu Jan 11 09:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.68899999999999995,"15min":0.69399999999999995},"min":{"1min":0.50600000000000001,"5min":0.47699999999999998,"15min":0.47299999999999998},"max":{"1min":0.97699999999999998,"5min":1.139,"15min":1.139},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.73399999999999999,"15min":0.73999999999999999},"min":{"1min":0.55900000000000005,"5min":0.51400000000000001,"15min":0.48199999999999998},"max":{"1min":0.91700000000000004,"5min":1.016,"15min":1.0389999999999999},"last":0.623}]}]},{"osd":5,"up_from":13,"seq":55834575754,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307520,"kb_used_data":2040,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064320,"statfs":{"total":96636764160,"available":96321863680,"internally_reserved":0,"allocated":2088960,"data_stored":985315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.73499999999999999,"15min":0.73699999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.77700000000000002,"5min":1.006,"15min":1.0229999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.76100000000000001,"15min":0.75600000000000001},"min":{"1min":0.46500000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.84799999999999998,"5min":0.97599999999999998,"15min":1.1020000000000001},"last":0.84699999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.745,"15min":0.74099999999999999},"min":{"1min":0.50700000000000001,"5min":0.438,"15min":0.41399999999999998},"max":{"1min":0.997,"5min":0.997,"15min":1.046},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.74399999999999999,"15min":0.75800000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82399999999999995,"5min":0.997,"15min":1.0249999999999999},"last":0.91300000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.72599999999999998,"15min":0.747},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.76600000000000001,"5min":1.014,"15min":1.0569999999999999},"last":0.874},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.76400000000000001,"15min":0.75600000000000001},"min":{"1min":0.51700000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94599999999999995,"5min":0.97999999999999998,"15min":1.012},"last":0.93600000000000005}]},{"osd":3,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.76600000000000001,"15min":0.76900000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.85799999999999998,"5min":1.016,"15min":1.016},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.76900000000000002,"15min":0.77300000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.45800000000000002},"max":{"1min":0.91400000000000003,"5min":0.99199999999999999,"15min":1.0609999999999999},"last":0.752}]},{"osd":4,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.622,"15min":0.61199999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.64200000000000002,"15min":0.63200000000000001},"min":{"1min":0.48299999999999998,"5min":0.45800000000000002,"15min":0.38900000000000001},"max":{"1min":0.88900000000000001,"5min":0.91300000000000003,"15min":0.94899999999999995},"last":0.72499999999999998}]},{"osd":6,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.64700000000000002,"15min":0.64300000000000002},"min":{"1min":0.443,"5min":0.433,"15min":0.39400000000000002},"max":{"1min":0.874,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.62},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.44500000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.64600000000000002}]},{"osd":7,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.75,"15min":0.73599999999999999},"min":{"1min":0.53700000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.89300000000000002,"5min":0.997,"15min":1.0920000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.78900000000000003,"15min":0.78100000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96899999999999997,"5min":1.01,"15min":1.137},"last":0.80300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:26:46.122 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:26:46.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.359+0000 7f273d3ff700 1 Processor -- start 2024-01-11T09:26:46.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.359+0000 7f273d3ff700 1 -- start start 2024-01-11T09:26:46.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.361+0000 7f273d3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2738069d40 0x7f27380b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:46.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.361+0000 7f273d3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27380c89f0 0x7f27380b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:46.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.361+0000 7f273d3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2738157390 con 0x7f27380c8620 2024-01-11T09:26:46.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.361+0000 7f273d3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27380adcd0 con 0x7f2738069d40 2024-01-11T09:26:46.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.361+0000 7f273d3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f273806a200 con 0x7f27380c89f0 2024-01-11T09:26:46.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.361+0000 7f27367fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2738069d40 0x7f27380b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:46.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.361+0000 7f2736ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f27380c8620 0x7f27380af970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40426/0 (socket says 172.21.15.27:40426) 2024-01-11T09:26:46.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.361+0000 7f2736ffd700 1 -- 172.21.15.27:0/2216139852 learned_addr learned my addr 172.21.15.27:0/2216139852 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:46.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.361+0000 7f27377fe700 1 --2- 172.21.15.27:0/2216139852 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27380c89f0 0x7f27380b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:46.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.361+0000 7f27367fc700 1 -- 172.21.15.27:0/2216139852 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27380c89f0 msgr2=0x7f27380b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:46.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.361+0000 7f27367fc700 1 --2- 172.21.15.27:0/2216139852 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27380c89f0 0x7f27380b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.361+0000 7f27367fc700 1 -- 172.21.15.27:0/2216139852 >> v1:172.21.15.27:6789/0 conn(0x7f27380c8620 legacy=0x7f27380af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:46.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.361+0000 7f27367fc700 1 -- 172.21.15.27:0/2216139852 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2738156d40 con 0x7f2738069d40 2024-01-11T09:26:46.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.361+0000 7f2735ffb700 1 -- 172.21.15.27:0/2216139852 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764293126 0 0) 0x7f2738157390 con 0x7f27380c8620 2024-01-11T09:26:46.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.362+0000 7f27367fc700 1 --2- 172.21.15.27:0/2216139852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738069d40 0x7f27380b02d0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f272c0098e0 tx=0x7f272c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c9bc0f4f8ec4af server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:46.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.362+0000 7f2735ffb700 1 -- 172.21.15.27:0/2216139852 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272c009e80 con 0x7f2738069d40 2024-01-11T09:26:46.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.362+0000 7f2735ffb700 1 -- 172.21.15.27:0/2216139852 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f272c00b860 con 0x7f2738069d40 2024-01-11T09:26:46.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.362+0000 7f2735ffb700 1 -- 172.21.15.27:0/2216139852 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272c00bbe0 con 0x7f2738069d40 2024-01-11T09:26:46.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.362+0000 7f273d3ff700 1 -- 172.21.15.27:0/2216139852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738069d40 msgr2=0x7f27380b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:46.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.362+0000 7f273d3ff700 1 --2- 172.21.15.27:0/2216139852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738069d40 0x7f27380b02d0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f272c0098e0 tx=0x7f272c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.363+0000 7f273d3ff700 1 -- 172.21.15.27:0/2216139852 shutdown_connections 2024-01-11T09:26:46.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.363+0000 7f273d3ff700 1 --2- 172.21.15.27:0/2216139852 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27380c89f0 0x7f27380b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.363+0000 7f273d3ff700 1 --2- 172.21.15.27:0/2216139852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738069d40 0x7f27380b02d0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.363+0000 7f273d3ff700 1 -- 172.21.15.27:0/2216139852 >> 172.21.15.27:0/2216139852 conn(0x7f27380680e0 msgr2=0x7f2738069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:46.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.363+0000 7f273d3ff700 1 -- 172.21.15.27:0/2216139852 shutdown_connections 2024-01-11T09:26:46.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.363+0000 7f273d3ff700 1 -- 172.21.15.27:0/2216139852 wait complete. 2024-01-11T09:26:46.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.364+0000 7f273d3ff700 1 Processor -- start 2024-01-11T09:26:46.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.364+0000 7f273d3ff700 1 -- start start 2024-01-11T09:26:46.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.365+0000 7f273d3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2738069d40 0x7f27381531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:46.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.365+0000 7f273d3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27380c8620 0x7f273816a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:46.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.365+0000 7f273d3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27380c89f0 0x7f27380a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:46.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.365+0000 7f273d3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273816a290 con 0x7f27380c89f0 2024-01-11T09:26:46.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.365+0000 7f273d3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2738169cb0 con 0x7f27380c8620 2024-01-11T09:26:46.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.365+0000 7f273d3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27380acf80 con 0x7f2738069d40 2024-01-11T09:26:46.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.366+0000 7f27377fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27380c89f0 0x7f27380a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:46.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.366+0000 7f27377fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27380c89f0 0x7f27380a5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32998/0 (socket says 172.21.15.27:32998) 2024-01-11T09:26:46.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.366+0000 7f27377fe700 1 -- 172.21.15.27:0/2981786644 learned_addr learned my addr 172.21.15.27:0/2981786644 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:46.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.366+0000 7f2736ffd700 1 --2- 172.21.15.27:0/2981786644 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2738069d40 0x7f27381531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:46.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.366+0000 7f27367fc700 1 --2- 172.21.15.27:0/2981786644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27380c8620 0x7f273816a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:46.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.366+0000 7f27377fe700 1 -- 172.21.15.27:0/2981786644 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2738069d40 msgr2=0x7f27381531b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:46.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.366+0000 7f27377fe700 1 --2- 172.21.15.27:0/2981786644 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2738069d40 0x7f27381531b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.366+0000 7f27377fe700 1 -- 172.21.15.27:0/2981786644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27380c8620 msgr2=0x7f273816a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:46.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.366+0000 7f27377fe700 1 --2- 172.21.15.27:0/2981786644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27380c8620 0x7f273816a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.366+0000 7f27377fe700 1 -- 172.21.15.27:0/2981786644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272c009420 con 0x7f27380c89f0 2024-01-11T09:26:46.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.366+0000 7f27377fe700 1 --2- 172.21.15.27:0/2981786644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27380c89f0 0x7f27380a5490 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f2730009c80 tx=0x7f2730009400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:46.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.366+0000 7f271effd700 1 -- 172.21.15.27:0/2981786644 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2730010070 con 0x7f27380c89f0 2024-01-11T09:26:46.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.367+0000 7f271effd700 1 -- 172.21.15.27:0/2981786644 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2730008770 con 0x7f27380c89f0 2024-01-11T09:26:46.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.367+0000 7f273d3ff700 1 -- 172.21.15.27:0/2981786644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2738169f20 con 0x7f27380c89f0 2024-01-11T09:26:46.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.367+0000 7f271effd700 1 -- 172.21.15.27:0/2981786644 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2730008b30 con 0x7f27380c89f0 2024-01-11T09:26:46.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.367+0000 7f273d3ff700 1 -- 172.21.15.27:0/2981786644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27380acc50 con 0x7f27380c89f0 2024-01-11T09:26:46.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.368+0000 7f273d3ff700 1 -- 172.21.15.27:0/2981786644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27380718d0 con 0x7f27380c89f0 2024-01-11T09:26:46.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.369+0000 7f271effd700 1 -- 172.21.15.27:0/2981786644 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2730003020 con 0x7f27380c89f0 2024-01-11T09:26:46.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.369+0000 7f271effd700 1 --2- 172.21.15.27:0/2981786644 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2724066f30 0x7f27240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:46.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.370+0000 7f271effd700 1 -- 172.21.15.27:0/2981786644 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f2730084120 con 0x7f27380c89f0 2024-01-11T09:26:46.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.370+0000 7f2736ffd700 1 --2- 172.21.15.27:0/2981786644 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2724066f30 0x7f27240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:46.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.371+0000 7f2736ffd700 1 --2- 172.21.15.27:0/2981786644 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2724066f30 0x7f27240693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2728007fd0 tx=0x7f2728007d30 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:46.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.374+0000 7f271effd700 1 -- 172.21.15.27:0/2981786644 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f273004e5a0 con 0x7f27380c89f0 2024-01-11T09:26:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.571+0000 7f273d3ff700 1 -- 172.21.15.27:0/2981786644 --> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f273816a3d0 con 0x7f2724066f30 2024-01-11T09:26:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.580+0000 7f271effd700 1 -- 172.21.15.27:0/2981786644 <== mgr.8542 v2:172.21.15.77:6836/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289472 (secure 0 0 0) 0x7f273816a3d0 con 0x7f2724066f30 2024-01-11T09:26:46.583 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:26:46.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.584+0000 7f273d3ff700 1 -- 172.21.15.27:0/2981786644 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2724066f30 msgr2=0x7f27240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.584+0000 7f273d3ff700 1 --2- 172.21.15.27:0/2981786644 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2724066f30 0x7f27240693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2728007fd0 tx=0x7f2728007d30 comp rx=0 tx=0).stop 2024-01-11T09:26:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.584+0000 7f273d3ff700 1 -- 172.21.15.27:0/2981786644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27380c89f0 msgr2=0x7f27380a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.584+0000 7f273d3ff700 1 --2- 172.21.15.27:0/2981786644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27380c89f0 0x7f27380a5490 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f2730009c80 tx=0x7f2730009400 comp rx=0 tx=0).stop 2024-01-11T09:26:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.585+0000 7f273d3ff700 1 -- 172.21.15.27:0/2981786644 shutdown_connections 2024-01-11T09:26:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.585+0000 7f273d3ff700 1 --2- 172.21.15.27:0/2981786644 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2724066f30 0x7f27240693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.585+0000 7f273d3ff700 1 --2- 172.21.15.27:0/2981786644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27380c89f0 0x7f27380a5490 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.585+0000 7f273d3ff700 1 --2- 172.21.15.27:0/2981786644 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2738069d40 0x7f27381531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.585+0000 7f273d3ff700 1 --2- 172.21.15.27:0/2981786644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27380c8620 0x7f273816a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.585+0000 7f273d3ff700 1 -- 172.21.15.27:0/2981786644 >> 172.21.15.27:0/2981786644 conn(0x7f27380680e0 msgr2=0x7f273806b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.585+0000 7f273d3ff700 1 -- 172.21.15.27:0/2981786644 shutdown_connections 2024-01-11T09:26:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.585+0000 7f273d3ff700 1 -- 172.21.15.27:0/2981786644 wait complete. 2024-01-11T09:26:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:26:46.599 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-11T09:26:46.294260+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1616,"num_read_kb":3366,"num_write":515,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":327,"ondisk_log_size":327,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":167,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2469544,"kb_used_data":17512,"kb_used_omap":0,"kb_used_meta":2451968,"kb_avail":752505176,"statfs":{"total":773094113280,"available":770565300224,"internally_reserved":0,"allocated":17932288,"data_stored":9067692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2510815232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001819"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.715684+0000","last_change":"2024-01-11T09:26:41.709773+0000","last_active":"2024-01-11T09:26:41.715684+0000","last_peered":"2024-01-11T09:26:41.715684+0000","last_clean":"2024-01-11T09:26:41.715684+0000","last_became_active":"2024-01-11T09:26:41.709482+0000","last_became_peered":"2024-01-11T09:26:41.709482+0000","last_unstale":"2024-01-11T09:26:41.715684+0000","last_undegraded":"2024-01-11T09:26:41.715684+0000","last_fullsized":"2024-01-11T09:26:41.715684+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:42:03.896763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.710680+0000","last_change":"2024-01-11T09:26:41.703650+0000","last_active":"2024-01-11T09:26:41.710680+0000","last_peered":"2024-01-11T09:26:41.710680+0000","last_clean":"2024-01-11T09:26:41.710680+0000","last_became_active":"2024-01-11T09:26:41.699944+0000","last_became_peered":"2024-01-11T09:26:41.699944+0000","last_unstale":"2024-01-11T09:26:41.710680+0000","last_undegraded":"2024-01-11T09:26:41.710680+0000","last_fullsized":"2024-01-11T09:26:41.710680+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:46:54.067657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.707638+0000","last_change":"2024-01-11T09:26:41.704825+0000","last_active":"2024-01-11T09:26:41.707638+0000","last_peered":"2024-01-11T09:26:41.707638+0000","last_clean":"2024-01-11T09:26:41.707638+0000","last_became_active":"2024-01-11T09:26:41.704571+0000","last_became_peered":"2024-01-11T09:26:41.704571+0000","last_unstale":"2024-01-11T09:26:41.707638+0000","last_undegraded":"2024-01-11T09:26:41.707638+0000","last_fullsized":"2024-01-11T09:26:41.707638+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:55:59.520132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.718716+0000","last_change":"2024-01-11T09:26:41.713809+0000","last_active":"2024-01-11T09:26:41.718716+0000","last_peered":"2024-01-11T09:26:41.718716+0000","last_clean":"2024-01-11T09:26:41.718716+0000","last_became_active":"2024-01-11T09:26:41.708899+0000","last_became_peered":"2024-01-11T09:26:41.708899+0000","last_unstale":"2024-01-11T09:26:41.718716+0000","last_undegraded":"2024-01-11T09:26:41.718716+0000","last_fullsized":"2024-01-11T09:26:41.718716+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:56:49.467904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.715547+0000","last_change":"2024-01-11T09:26:41.710134+0000","last_active":"2024-01-11T09:26:41.715547+0000","last_peered":"2024-01-11T09:26:41.715547+0000","last_clean":"2024-01-11T09:26:41.715547+0000","last_became_active":"2024-01-11T09:26:41.709506+0000","last_became_peered":"2024-01-11T09:26:41.709506+0000","last_unstale":"2024-01-11T09:26:41.715547+0000","last_undegraded":"2024-01-11T09:26:41.715547+0000","last_fullsized":"2024-01-11T09:26:41.715547+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:50:46.665371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.716840+0000","last_change":"2024-01-11T09:26:41.712672+0000","last_active":"2024-01-11T09:26:41.716840+0000","last_peered":"2024-01-11T09:26:41.716840+0000","last_clean":"2024-01-11T09:26:41.716840+0000","last_became_active":"2024-01-11T09:26:41.710861+0000","last_became_peered":"2024-01-11T09:26:41.710861+0000","last_unstale":"2024-01-11T09:26:41.716840+0000","last_undegraded":"2024-01-11T09:26:41.716840+0000","last_fullsized":"2024-01-11T09:26:41.716840+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:42:39.118184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.720388+0000","last_change":"2024-01-11T09:26:41.715291+0000","last_active":"2024-01-11T09:26:41.720388+0000","last_peered":"2024-01-11T09:26:41.720388+0000","last_clean":"2024-01-11T09:26:41.720388+0000","last_became_active":"2024-01-11T09:26:41.714864+0000","last_became_peered":"2024-01-11T09:26:41.714864+0000","last_unstale":"2024-01-11T09:26:41.720388+0000","last_undegraded":"2024-01-11T09:26:41.720388+0000","last_fullsized":"2024-01-11T09:26:41.720388+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:48:24.611840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.705094+0000","last_change":"2024-01-11T09:26:41.701181+0000","last_active":"2024-01-11T09:26:41.705094+0000","last_peered":"2024-01-11T09:26:41.705094+0000","last_clean":"2024-01-11T09:26:41.705094+0000","last_became_active":"2024-01-11T09:26:41.700739+0000","last_became_peered":"2024-01-11T09:26:41.700739+0000","last_unstale":"2024-01-11T09:26:41.705094+0000","last_undegraded":"2024-01-11T09:26:41.705094+0000","last_fullsized":"2024-01-11T09:26:41.705094+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:12:22.573069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.713561+0000","last_change":"2024-01-11T09:26:41.704659+0000","last_active":"2024-01-11T09:26:41.713561+0000","last_peered":"2024-01-11T09:26:41.713561+0000","last_clean":"2024-01-11T09:26:41.713561+0000","last_became_active":"2024-01-11T09:26:41.704407+0000","last_became_peered":"2024-01-11T09:26:41.704407+0000","last_unstale":"2024-01-11T09:26:41.713561+0000","last_undegraded":"2024-01-11T09:26:41.713561+0000","last_fullsized":"2024-01-11T09:26:41.713561+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:02:58.472920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.723504+0000","last_change":"2024-01-11T09:26:41.718863+0000","last_active":"2024-01-11T09:26:41.723504+0000","last_peered":"2024-01-11T09:26:41.723504+0000","last_clean":"2024-01-11T09:26:41.723504+0000","last_became_active":"2024-01-11T09:26:41.718582+0000","last_became_peered":"2024-01-11T09:26:41.718582+0000","last_unstale":"2024-01-11T09:26:41.723504+0000","last_undegraded":"2024-01-11T09:26:41.723504+0000","last_fullsized":"2024-01-11T09:26:41.723504+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:33:05.873335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.711050+0000","last_change":"2024-01-11T09:26:41.707682+0000","last_active":"2024-01-11T09:26:41.711050+0000","last_peered":"2024-01-11T09:26:41.711050+0000","last_clean":"2024-01-11T09:26:41.711050+0000","last_became_active":"2024-01-11T09:26:41.707412+0000","last_became_peered":"2024-01-11T09:26:41.707412+0000","last_unstale":"2024-01-11T09:26:41.711050+0000","last_undegraded":"2024-01-11T09:26:41.711050+0000","last_fullsized":"2024-01-11T09:26:41.711050+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:06:41.588774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.722411+0000","last_change":"2024-01-11T09:26:41.715126+0000","last_active":"2024-01-11T09:26:41.722411+0000","last_peered":"2024-01-11T09:26:41.722411+0000","last_clean":"2024-01-11T09:26:41.722411+0000","last_became_active":"2024-01-11T09:26:41.714887+0000","last_became_peered":"2024-01-11T09:26:41.714887+0000","last_unstale":"2024-01-11T09:26:41.722411+0000","last_undegraded":"2024-01-11T09:26:41.722411+0000","last_fullsized":"2024-01-11T09:26:41.722411+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:38:53.874199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.715569+0000","last_change":"2024-01-11T09:26:41.710297+0000","last_active":"2024-01-11T09:26:41.715569+0000","last_peered":"2024-01-11T09:26:41.715569+0000","last_clean":"2024-01-11T09:26:41.715569+0000","last_became_active":"2024-01-11T09:26:41.709890+0000","last_became_peered":"2024-01-11T09:26:41.709890+0000","last_unstale":"2024-01-11T09:26:41.715569+0000","last_undegraded":"2024-01-11T09:26:41.715569+0000","last_fullsized":"2024-01-11T09:26:41.715569+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:46:18.959911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.718864+0000","last_change":"2024-01-11T09:26:41.712738+0000","last_active":"2024-01-11T09:26:41.718864+0000","last_peered":"2024-01-11T09:26:41.718864+0000","last_clean":"2024-01-11T09:26:41.718864+0000","last_became_active":"2024-01-11T09:26:41.708964+0000","last_became_peered":"2024-01-11T09:26:41.708964+0000","last_unstale":"2024-01-11T09:26:41.718864+0000","last_undegraded":"2024-01-11T09:26:41.718864+0000","last_fullsized":"2024-01-11T09:26:41.718864+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:45:19.193731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.724489+0000","last_change":"2024-01-11T09:26:41.720110+0000","last_active":"2024-01-11T09:26:41.724489+0000","last_peered":"2024-01-11T09:26:41.724489+0000","last_clean":"2024-01-11T09:26:41.724489+0000","last_became_active":"2024-01-11T09:26:41.718216+0000","last_became_peered":"2024-01-11T09:26:41.718216+0000","last_unstale":"2024-01-11T09:26:41.724489+0000","last_undegraded":"2024-01-11T09:26:41.724489+0000","last_fullsized":"2024-01-11T09:26:41.724489+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:45:19.193731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.720043+0000","last_change":"2024-01-11T09:26:41.715065+0000","last_active":"2024-01-11T09:26:41.720043+0000","last_peered":"2024-01-11T09:26:41.720043+0000","last_clean":"2024-01-11T09:26:41.720043+0000","last_became_active":"2024-01-11T09:26:41.714517+0000","last_became_peered":"2024-01-11T09:26:41.714517+0000","last_unstale":"2024-01-11T09:26:41.720043+0000","last_undegraded":"2024-01-11T09:26:41.720043+0000","last_fullsized":"2024-01-11T09:26:41.720043+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:05:18.067025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.723519+0000","last_change":"2024-01-11T09:26:41.718769+0000","last_active":"2024-01-11T09:26:41.723519+0000","last_peered":"2024-01-11T09:26:41.723519+0000","last_clean":"2024-01-11T09:26:41.723519+0000","last_became_active":"2024-01-11T09:26:41.718175+0000","last_became_peered":"2024-01-11T09:26:41.718175+0000","last_unstale":"2024-01-11T09:26:41.723519+0000","last_undegraded":"2024-01-11T09:26:41.723519+0000","last_fullsized":"2024-01-11T09:26:41.723519+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:20.319578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.721226+0000","last_change":"2024-01-11T09:26:41.718708+0000","last_active":"2024-01-11T09:26:41.721226+0000","last_peered":"2024-01-11T09:26:41.721226+0000","last_clean":"2024-01-11T09:26:41.721226+0000","last_became_active":"2024-01-11T09:26:41.717412+0000","last_became_peered":"2024-01-11T09:26:41.717412+0000","last_unstale":"2024-01-11T09:26:41.721226+0000","last_undegraded":"2024-01-11T09:26:41.721226+0000","last_fullsized":"2024-01-11T09:26:41.721226+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:42:50.270447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.716985+0000","last_change":"2024-01-11T09:26:41.712766+0000","last_active":"2024-01-11T09:26:41.716985+0000","last_peered":"2024-01-11T09:26:41.716985+0000","last_clean":"2024-01-11T09:26:41.716985+0000","last_became_active":"2024-01-11T09:26:41.712461+0000","last_became_peered":"2024-01-11T09:26:41.712461+0000","last_unstale":"2024-01-11T09:26:41.716985+0000","last_undegraded":"2024-01-11T09:26:41.716985+0000","last_fullsized":"2024-01-11T09:26:41.716985+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:27:06.923577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.723552+0000","last_change":"2024-01-11T09:26:41.717424+0000","last_active":"2024-01-11T09:26:41.723552+0000","last_peered":"2024-01-11T09:26:41.723552+0000","last_clean":"2024-01-11T09:26:41.723552+0000","last_became_active":"2024-01-11T09:26:41.717109+0000","last_became_peered":"2024-01-11T09:26:41.717109+0000","last_unstale":"2024-01-11T09:26:41.723552+0000","last_undegraded":"2024-01-11T09:26:41.723552+0000","last_fullsized":"2024-01-11T09:26:41.723552+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:27:06.923577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.724634+0000","last_change":"2024-01-11T09:26:41.722325+0000","last_active":"2024-01-11T09:26:41.724634+0000","last_peered":"2024-01-11T09:26:41.724634+0000","last_clean":"2024-01-11T09:26:41.724634+0000","last_became_active":"2024-01-11T09:26:41.721971+0000","last_became_peered":"2024-01-11T09:26:41.721971+0000","last_unstale":"2024-01-11T09:26:41.724634+0000","last_undegraded":"2024-01-11T09:26:41.724634+0000","last_fullsized":"2024-01-11T09:26:41.724634+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:42:59.390079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.709190+0000","last_change":"2024-01-11T09:26:41.696350+0000","last_active":"2024-01-11T09:26:41.709190+0000","last_peered":"2024-01-11T09:26:41.709190+0000","last_clean":"2024-01-11T09:26:41.709190+0000","last_became_active":"2024-01-11T09:26:41.696035+0000","last_became_peered":"2024-01-11T09:26:41.696035+0000","last_unstale":"2024-01-11T09:26:41.709190+0000","last_undegraded":"2024-01-11T09:26:41.709190+0000","last_fullsized":"2024-01-11T09:26:41.709190+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:40:40.439277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.715210+0000","last_change":"2024-01-11T09:26:41.706818+0000","last_active":"2024-01-11T09:26:41.715210+0000","last_peered":"2024-01-11T09:26:41.715210+0000","last_clean":"2024-01-11T09:26:41.715210+0000","last_became_active":"2024-01-11T09:26:41.702005+0000","last_became_peered":"2024-01-11T09:26:41.702005+0000","last_unstale":"2024-01-11T09:26:41.715210+0000","last_undegraded":"2024-01-11T09:26:41.715210+0000","last_fullsized":"2024-01-11T09:26:41.715210+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:43:40.005149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.722501+0000","last_change":"2024-01-11T09:26:41.716844+0000","last_active":"2024-01-11T09:26:41.722501+0000","last_peered":"2024-01-11T09:26:41.722501+0000","last_clean":"2024-01-11T09:26:41.722501+0000","last_became_active":"2024-01-11T09:26:41.716451+0000","last_became_peered":"2024-01-11T09:26:41.716451+0000","last_unstale":"2024-01-11T09:26:41.722501+0000","last_undegraded":"2024-01-11T09:26:41.722501+0000","last_fullsized":"2024-01-11T09:26:41.722501+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:22.767311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.715638+0000","last_change":"2024-01-11T09:26:41.709719+0000","last_active":"2024-01-11T09:26:41.715638+0000","last_peered":"2024-01-11T09:26:41.715638+0000","last_clean":"2024-01-11T09:26:41.715638+0000","last_became_active":"2024-01-11T09:26:41.709463+0000","last_became_peered":"2024-01-11T09:26:41.709463+0000","last_unstale":"2024-01-11T09:26:41.715638+0000","last_undegraded":"2024-01-11T09:26:41.715638+0000","last_fullsized":"2024-01-11T09:26:41.715638+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:42:39.118184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.725159+0000","last_change":"2024-01-11T09:26:41.720885+0000","last_active":"2024-01-11T09:26:41.725159+0000","last_peered":"2024-01-11T09:26:41.725159+0000","last_clean":"2024-01-11T09:26:41.725159+0000","last_became_active":"2024-01-11T09:26:41.720587+0000","last_became_peered":"2024-01-11T09:26:41.720587+0000","last_unstale":"2024-01-11T09:26:41.725159+0000","last_undegraded":"2024-01-11T09:26:41.725159+0000","last_fullsized":"2024-01-11T09:26:41.725159+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:31:19.426090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.721280+0000","last_change":"2024-01-11T09:26:41.709848+0000","last_active":"2024-01-11T09:26:41.721280+0000","last_peered":"2024-01-11T09:26:41.721280+0000","last_clean":"2024-01-11T09:26:41.721280+0000","last_became_active":"2024-01-11T09:26:41.709584+0000","last_became_peered":"2024-01-11T09:26:41.709584+0000","last_unstale":"2024-01-11T09:26:41.721280+0000","last_undegraded":"2024-01-11T09:26:41.721280+0000","last_fullsized":"2024-01-11T09:26:41.721280+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:26:44.657352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.718794+0000","last_change":"2024-01-11T09:26:41.709048+0000","last_active":"2024-01-11T09:26:41.718794+0000","last_peered":"2024-01-11T09:26:41.718794+0000","last_clean":"2024-01-11T09:26:41.718794+0000","last_became_active":"2024-01-11T09:26:41.708677+0000","last_became_peered":"2024-01-11T09:26:41.708677+0000","last_unstale":"2024-01-11T09:26:41.718794+0000","last_undegraded":"2024-01-11T09:26:41.718794+0000","last_fullsized":"2024-01-11T09:26:41.718794+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:44:23.252777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.717675+0000","last_change":"2024-01-11T09:26:41.713535+0000","last_active":"2024-01-11T09:26:41.717675+0000","last_peered":"2024-01-11T09:26:41.717675+0000","last_clean":"2024-01-11T09:26:41.717675+0000","last_became_active":"2024-01-11T09:26:41.712738+0000","last_became_peered":"2024-01-11T09:26:41.712738+0000","last_unstale":"2024-01-11T09:26:41.717675+0000","last_undegraded":"2024-01-11T09:26:41.717675+0000","last_fullsized":"2024-01-11T09:26:41.717675+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:04:09.551826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.718847+0000","last_change":"2024-01-11T09:26:41.712696+0000","last_active":"2024-01-11T09:26:41.718847+0000","last_peered":"2024-01-11T09:26:41.718847+0000","last_clean":"2024-01-11T09:26:41.718847+0000","last_became_active":"2024-01-11T09:26:41.708993+0000","last_became_peered":"2024-01-11T09:26:41.708993+0000","last_unstale":"2024-01-11T09:26:41.718847+0000","last_undegraded":"2024-01-11T09:26:41.718847+0000","last_fullsized":"2024-01-11T09:26:41.718847+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:13:29.325881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.712408+0000","last_change":"2024-01-11T09:26:41.703862+0000","last_active":"2024-01-11T09:26:41.712408+0000","last_peered":"2024-01-11T09:26:41.712408+0000","last_clean":"2024-01-11T09:26:41.712408+0000","last_became_active":"2024-01-11T09:26:41.703569+0000","last_became_peered":"2024-01-11T09:26:41.703569+0000","last_unstale":"2024-01-11T09:26:41.712408+0000","last_undegraded":"2024-01-11T09:26:41.712408+0000","last_fullsized":"2024-01-11T09:26:41.712408+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:23:30.649071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.723422+0000","last_change":"2024-01-11T09:26:41.716885+0000","last_active":"2024-01-11T09:26:41.723422+0000","last_peered":"2024-01-11T09:26:41.723422+0000","last_clean":"2024-01-11T09:26:41.723422+0000","last_became_active":"2024-01-11T09:26:41.714815+0000","last_became_peered":"2024-01-11T09:26:41.714815+0000","last_unstale":"2024-01-11T09:26:41.723422+0000","last_undegraded":"2024-01-11T09:26:41.723422+0000","last_fullsized":"2024-01-11T09:26:41.723422+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:46:18.959911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.722717+0000","last_change":"2024-01-11T09:26:41.716663+0000","last_active":"2024-01-11T09:26:41.722717+0000","last_peered":"2024-01-11T09:26:41.722717+0000","last_clean":"2024-01-11T09:26:41.722717+0000","last_became_active":"2024-01-11T09:26:41.715465+0000","last_became_peered":"2024-01-11T09:26:41.715465+0000","last_unstale":"2024-01-11T09:26:41.722717+0000","last_undegraded":"2024-01-11T09:26:41.722717+0000","last_fullsized":"2024-01-11T09:26:41.722717+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:58:19.664719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"166'4","reported_seq":14,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.710657+0000","last_change":"2024-01-11T09:26:40.711057+0000","last_active":"2024-01-11T09:26:41.710657+0000","last_peered":"2024-01-11T09:26:41.710657+0000","last_clean":"2024-01-11T09:26:41.710657+0000","last_became_active":"2024-01-11T09:26:40.710729+0000","last_became_peered":"2024-01-11T09:26:40.710729+0000","last_unstale":"2024-01-11T09:26:41.710657+0000","last_undegraded":"2024-01-11T09:26:41.710657+0000","last_fullsized":"2024-01-11T09:26:41.710657+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:43:38.997683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.710732+0000","last_change":"2024-01-11T09:26:41.703657+0000","last_active":"2024-01-11T09:26:41.710732+0000","last_peered":"2024-01-11T09:26:41.710732+0000","last_clean":"2024-01-11T09:26:41.710732+0000","last_became_active":"2024-01-11T09:26:41.701059+0000","last_became_peered":"2024-01-11T09:26:41.701059+0000","last_unstale":"2024-01-11T09:26:41.710732+0000","last_undegraded":"2024-01-11T09:26:41.710732+0000","last_fullsized":"2024-01-11T09:26:41.710732+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:02:54.487205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"166'1","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.713099+0000","last_change":"2024-01-11T09:26:40.711040+0000","last_active":"2024-01-11T09:26:41.713099+0000","last_peered":"2024-01-11T09:26:41.713099+0000","last_clean":"2024-01-11T09:26:41.713099+0000","last_became_active":"2024-01-11T09:26:40.710725+0000","last_became_peered":"2024-01-11T09:26:40.710725+0000","last_unstale":"2024-01-11T09:26:41.713099+0000","last_undegraded":"2024-01-11T09:26:41.713099+0000","last_fullsized":"2024-01-11T09:26:41.713099+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:22:02.081826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.725230+0000","last_change":"2024-01-11T09:26:41.721892+0000","last_active":"2024-01-11T09:26:41.725230+0000","last_peered":"2024-01-11T09:26:41.725230+0000","last_clean":"2024-01-11T09:26:41.725230+0000","last_became_active":"2024-01-11T09:26:41.721607+0000","last_became_peered":"2024-01-11T09:26:41.721607+0000","last_unstale":"2024-01-11T09:26:41.725230+0000","last_undegraded":"2024-01-11T09:26:41.725230+0000","last_fullsized":"2024-01-11T09:26:41.725230+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:28.437611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"166'2","reported_seq":12,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.721701+0000","last_change":"2024-01-11T09:26:40.699899+0000","last_active":"2024-01-11T09:26:41.721701+0000","last_peered":"2024-01-11T09:26:41.721701+0000","last_clean":"2024-01-11T09:26:41.721701+0000","last_became_active":"2024-01-11T09:26:40.699625+0000","last_became_peered":"2024-01-11T09:26:40.699625+0000","last_unstale":"2024-01-11T09:26:41.721701+0000","last_undegraded":"2024-01-11T09:26:41.721701+0000","last_fullsized":"2024-01-11T09:26:41.721701+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:13:25.938342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.722143+0000","last_change":"2024-01-11T09:26:41.715020+0000","last_active":"2024-01-11T09:26:41.722143+0000","last_peered":"2024-01-11T09:26:41.722143+0000","last_clean":"2024-01-11T09:26:41.722143+0000","last_became_active":"2024-01-11T09:26:41.714776+0000","last_became_peered":"2024-01-11T09:26:41.714776+0000","last_unstale":"2024-01-11T09:26:41.722143+0000","last_undegraded":"2024-01-11T09:26:41.722143+0000","last_fullsized":"2024-01-11T09:26:41.722143+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:00:46.715271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"166'2","reported_seq":12,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.718649+0000","last_change":"2024-01-11T09:26:40.705416+0000","last_active":"2024-01-11T09:26:41.718649+0000","last_peered":"2024-01-11T09:26:41.718649+0000","last_clean":"2024-01-11T09:26:41.718649+0000","last_became_active":"2024-01-11T09:26:40.704727+0000","last_became_peered":"2024-01-11T09:26:40.704727+0000","last_unstale":"2024-01-11T09:26:41.718649+0000","last_undegraded":"2024-01-11T09:26:41.718649+0000","last_fullsized":"2024-01-11T09:26:41.718649+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:16:55.456209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.719009+0000","last_change":"2024-01-11T09:26:41.715363+0000","last_active":"2024-01-11T09:26:41.719009+0000","last_peered":"2024-01-11T09:26:41.719009+0000","last_clean":"2024-01-11T09:26:41.719009+0000","last_became_active":"2024-01-11T09:26:41.714216+0000","last_became_peered":"2024-01-11T09:26:41.714216+0000","last_unstale":"2024-01-11T09:26:41.719009+0000","last_undegraded":"2024-01-11T09:26:41.719009+0000","last_fullsized":"2024-01-11T09:26:41.719009+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:20.319578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"166'1","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.724960+0000","last_change":"2024-01-11T09:26:40.711625+0000","last_active":"2024-01-11T09:26:41.724960+0000","last_peered":"2024-01-11T09:26:41.724960+0000","last_clean":"2024-01-11T09:26:41.724960+0000","last_became_active":"2024-01-11T09:26:40.709181+0000","last_became_peered":"2024-01-11T09:26:40.709181+0000","last_unstale":"2024-01-11T09:26:41.724960+0000","last_undegraded":"2024-01-11T09:26:41.724960+0000","last_fullsized":"2024-01-11T09:26:41.724960+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:40:03.478289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.718184+0000","last_change":"2024-01-11T09:26:41.712308+0000","last_active":"2024-01-11T09:26:41.718184+0000","last_peered":"2024-01-11T09:26:41.718184+0000","last_clean":"2024-01-11T09:26:41.718184+0000","last_became_active":"2024-01-11T09:26:41.711817+0000","last_became_peered":"2024-01-11T09:26:41.711817+0000","last_unstale":"2024-01-11T09:26:41.718184+0000","last_undegraded":"2024-01-11T09:26:41.718184+0000","last_fullsized":"2024-01-11T09:26:41.718184+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:19:48.620667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"166'1","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.710617+0000","last_change":"2024-01-11T09:26:40.706966+0000","last_active":"2024-01-11T09:26:41.710617+0000","last_peered":"2024-01-11T09:26:41.710617+0000","last_clean":"2024-01-11T09:26:41.710617+0000","last_became_active":"2024-01-11T09:26:40.706678+0000","last_became_peered":"2024-01-11T09:26:40.706678+0000","last_unstale":"2024-01-11T09:26:41.710617+0000","last_undegraded":"2024-01-11T09:26:41.710617+0000","last_fullsized":"2024-01-11T09:26:41.710617+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:30:57.214674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.705890+0000","last_change":"2024-01-11T09:26:41.702886+0000","last_active":"2024-01-11T09:26:41.705890+0000","last_peered":"2024-01-11T09:26:41.705890+0000","last_clean":"2024-01-11T09:26:41.705890+0000","last_became_active":"2024-01-11T09:26:41.700541+0000","last_became_peered":"2024-01-11T09:26:41.700541+0000","last_unstale":"2024-01-11T09:26:41.705890+0000","last_undegraded":"2024-01-11T09:26:41.705890+0000","last_fullsized":"2024-01-11T09:26:41.705890+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:22:48.073902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.700969+0000","last_change":"2024-01-11T09:26:40.704556+0000","last_active":"2024-01-11T09:26:41.700969+0000","last_peered":"2024-01-11T09:26:41.700969+0000","last_clean":"2024-01-11T09:26:41.700969+0000","last_became_active":"2024-01-11T09:26:40.704283+0000","last_became_peered":"2024-01-11T09:26:40.704283+0000","last_unstale":"2024-01-11T09:26:41.700969+0000","last_undegraded":"2024-01-11T09:26:41.700969+0000","last_fullsized":"2024-01-11T09:26:41.700969+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:30:57.214674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.709303+0000","last_change":"2024-01-11T09:26:41.701311+0000","last_active":"2024-01-11T09:26:41.709303+0000","last_peered":"2024-01-11T09:26:41.709303+0000","last_clean":"2024-01-11T09:26:41.709303+0000","last_became_active":"2024-01-11T09:26:41.701030+0000","last_became_peered":"2024-01-11T09:26:41.701030+0000","last_unstale":"2024-01-11T09:26:41.709303+0000","last_undegraded":"2024-01-11T09:26:41.709303+0000","last_fullsized":"2024-01-11T09:26:41.709303+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:28.437611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.689857+0000","last_change":"2024-01-11T09:26:40.709216+0000","last_active":"2024-01-11T09:26:41.689857+0000","last_peered":"2024-01-11T09:26:41.689857+0000","last_clean":"2024-01-11T09:26:41.689857+0000","last_became_active":"2024-01-11T09:26:40.708904+0000","last_became_peered":"2024-01-11T09:26:40.708904+0000","last_unstale":"2024-01-11T09:26:41.689857+0000","last_undegraded":"2024-01-11T09:26:41.689857+0000","last_fullsized":"2024-01-11T09:26:41.689857+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:44:26.989363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.715606+0000","last_change":"2024-01-11T09:26:41.712115+0000","last_active":"2024-01-11T09:26:41.715606+0000","last_peered":"2024-01-11T09:26:41.715606+0000","last_clean":"2024-01-11T09:26:41.715606+0000","last_became_active":"2024-01-11T09:26:41.711851+0000","last_became_peered":"2024-01-11T09:26:41.711851+0000","last_unstale":"2024-01-11T09:26:41.715606+0000","last_undegraded":"2024-01-11T09:26:41.715606+0000","last_fullsized":"2024-01-11T09:26:41.715606+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:04:09.551826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.686627+0000","last_change":"2024-01-11T09:26:40.704595+0000","last_active":"2024-01-11T09:26:41.686627+0000","last_peered":"2024-01-11T09:26:41.686627+0000","last_clean":"2024-01-11T09:26:41.686627+0000","last_became_active":"2024-01-11T09:26:40.704338+0000","last_became_peered":"2024-01-11T09:26:40.704338+0000","last_unstale":"2024-01-11T09:26:41.686627+0000","last_undegraded":"2024-01-11T09:26:41.686627+0000","last_fullsized":"2024-01-11T09:26:41.686627+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:07:41.508678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.724226+0000","last_change":"2024-01-11T09:26:41.719810+0000","last_active":"2024-01-11T09:26:41.724226+0000","last_peered":"2024-01-11T09:26:41.724226+0000","last_clean":"2024-01-11T09:26:41.724226+0000","last_became_active":"2024-01-11T09:26:41.719546+0000","last_became_peered":"2024-01-11T09:26:41.719546+0000","last_unstale":"2024-01-11T09:26:41.724226+0000","last_undegraded":"2024-01-11T09:26:41.724226+0000","last_fullsized":"2024-01-11T09:26:41.724226+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:23:30.649071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.700525+0000","last_change":"2024-01-11T09:26:40.712506+0000","last_active":"2024-01-11T09:26:41.700525+0000","last_peered":"2024-01-11T09:26:41.700525+0000","last_clean":"2024-01-11T09:26:41.700525+0000","last_became_active":"2024-01-11T09:26:40.712169+0000","last_became_peered":"2024-01-11T09:26:40.712169+0000","last_unstale":"2024-01-11T09:26:41.700525+0000","last_undegraded":"2024-01-11T09:26:41.700525+0000","last_fullsized":"2024-01-11T09:26:41.700525+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:07:41.508678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.718773+0000","last_change":"2024-01-11T09:26:41.707451+0000","last_active":"2024-01-11T09:26:41.718773+0000","last_peered":"2024-01-11T09:26:41.718773+0000","last_clean":"2024-01-11T09:26:41.718773+0000","last_became_active":"2024-01-11T09:26:41.707173+0000","last_became_peered":"2024-01-11T09:26:41.707173+0000","last_unstale":"2024-01-11T09:26:41.718773+0000","last_undegraded":"2024-01-11T09:26:41.718773+0000","last_fullsized":"2024-01-11T09:26:41.718773+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:37:28.437611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"166'6","reported_seq":16,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.726670+0000","last_change":"2024-01-11T09:26:40.710232+0000","last_active":"2024-01-11T09:26:41.726670+0000","last_peered":"2024-01-11T09:26:41.726670+0000","last_clean":"2024-01-11T09:26:41.726670+0000","last_became_active":"2024-01-11T09:26:40.709882+0000","last_became_peered":"2024-01-11T09:26:40.709882+0000","last_unstale":"2024-01-11T09:26:41.726670+0000","last_undegraded":"2024-01-11T09:26:41.726670+0000","last_fullsized":"2024-01-11T09:26:41.726670+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:02:57.465454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.715623+0000","last_change":"2024-01-11T09:26:41.713716+0000","last_active":"2024-01-11T09:26:41.715623+0000","last_peered":"2024-01-11T09:26:41.715623+0000","last_clean":"2024-01-11T09:26:41.715623+0000","last_became_active":"2024-01-11T09:26:41.713443+0000","last_became_peered":"2024-01-11T09:26:41.713443+0000","last_unstale":"2024-01-11T09:26:41.715623+0000","last_undegraded":"2024-01-11T09:26:41.715623+0000","last_fullsized":"2024-01-11T09:26:41.715623+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:00:31.275152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"166'2","reported_seq":12,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.716765+0000","last_change":"2024-01-11T09:26:40.703202+0000","last_active":"2024-01-11T09:26:41.716765+0000","last_peered":"2024-01-11T09:26:41.716765+0000","last_clean":"2024-01-11T09:26:41.716765+0000","last_became_active":"2024-01-11T09:26:40.702745+0000","last_became_peered":"2024-01-11T09:26:40.702745+0000","last_unstale":"2024-01-11T09:26:41.716765+0000","last_undegraded":"2024-01-11T09:26:41.716765+0000","last_fullsized":"2024-01-11T09:26:41.716765+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:34:05.586284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.716797+0000","last_change":"2024-01-11T09:26:41.708152+0000","last_active":"2024-01-11T09:26:41.716797+0000","last_peered":"2024-01-11T09:26:41.716797+0000","last_clean":"2024-01-11T09:26:41.716797+0000","last_became_active":"2024-01-11T09:26:41.707904+0000","last_became_peered":"2024-01-11T09:26:41.707904+0000","last_unstale":"2024-01-11T09:26:41.716797+0000","last_undegraded":"2024-01-11T09:26:41.716797+0000","last_fullsized":"2024-01-11T09:26:41.716797+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:18:08.104121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.698278+0000","last_change":"2024-01-11T09:26:40.706553+0000","last_active":"2024-01-11T09:26:41.698278+0000","last_peered":"2024-01-11T09:26:41.698278+0000","last_clean":"2024-01-11T09:26:41.698278+0000","last_became_active":"2024-01-11T09:26:40.705001+0000","last_became_peered":"2024-01-11T09:26:40.705001+0000","last_unstale":"2024-01-11T09:26:41.698278+0000","last_undegraded":"2024-01-11T09:26:41.698278+0000","last_fullsized":"2024-01-11T09:26:41.698278+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:42:58.382613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.715433+0000","last_change":"2024-01-11T09:26:41.709029+0000","last_active":"2024-01-11T09:26:41.715433+0000","last_peered":"2024-01-11T09:26:41.715433+0000","last_clean":"2024-01-11T09:26:41.715433+0000","last_became_active":"2024-01-11T09:26:41.708765+0000","last_became_peered":"2024-01-11T09:26:41.708765+0000","last_unstale":"2024-01-11T09:26:41.715433+0000","last_undegraded":"2024-01-11T09:26:41.715433+0000","last_fullsized":"2024-01-11T09:26:41.715433+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:05:18.067025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.697612+0000","last_change":"2024-01-11T09:26:40.703415+0000","last_active":"2024-01-11T09:26:41.697612+0000","last_peered":"2024-01-11T09:26:41.697612+0000","last_clean":"2024-01-11T09:26:41.697612+0000","last_became_active":"2024-01-11T09:26:40.702339+0000","last_became_peered":"2024-01-11T09:26:40.702339+0000","last_unstale":"2024-01-11T09:26:41.697612+0000","last_undegraded":"2024-01-11T09:26:41.697612+0000","last_fullsized":"2024-01-11T09:26:41.697612+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:30:57.214674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.721869+0000","last_change":"2024-01-11T09:26:41.719864+0000","last_active":"2024-01-11T09:26:41.721869+0000","last_peered":"2024-01-11T09:26:41.721869+0000","last_clean":"2024-01-11T09:26:41.721869+0000","last_became_active":"2024-01-11T09:26:41.719612+0000","last_became_peered":"2024-01-11T09:26:41.719612+0000","last_unstale":"2024-01-11T09:26:41.721869+0000","last_undegraded":"2024-01-11T09:26:41.721869+0000","last_fullsized":"2024-01-11T09:26:41.721869+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:17:30.608645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"166'8","reported_seq":18,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.725131+0000","last_change":"2024-01-11T09:26:40.710060+0000","last_active":"2024-01-11T09:26:41.725131+0000","last_peered":"2024-01-11T09:26:41.725131+0000","last_clean":"2024-01-11T09:26:41.725131+0000","last_became_active":"2024-01-11T09:26:40.709397+0000","last_became_peered":"2024-01-11T09:26:40.709397+0000","last_unstale":"2024-01-11T09:26:41.725131+0000","last_undegraded":"2024-01-11T09:26:41.725131+0000","last_fullsized":"2024-01-11T09:26:41.725131+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:15:12.889069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.714266+0000","last_change":"2024-01-11T09:26:41.710217+0000","last_active":"2024-01-11T09:26:41.714266+0000","last_peered":"2024-01-11T09:26:41.714266+0000","last_clean":"2024-01-11T09:26:41.714266+0000","last_became_active":"2024-01-11T09:26:41.709884+0000","last_became_peered":"2024-01-11T09:26:41.709884+0000","last_unstale":"2024-01-11T09:26:41.714266+0000","last_undegraded":"2024-01-11T09:26:41.714266+0000","last_fullsized":"2024-01-11T09:26:41.714266+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:29:09.666238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"166'1","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.708488+0000","last_change":"2024-01-11T09:26:40.703362+0000","last_active":"2024-01-11T09:26:41.708488+0000","last_peered":"2024-01-11T09:26:41.708488+0000","last_clean":"2024-01-11T09:26:41.708488+0000","last_became_active":"2024-01-11T09:26:40.701980+0000","last_became_peered":"2024-01-11T09:26:40.701980+0000","last_unstale":"2024-01-11T09:26:41.708488+0000","last_undegraded":"2024-01-11T09:26:41.708488+0000","last_fullsized":"2024-01-11T09:26:41.708488+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:29:28.407605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.718051+0000","last_change":"2024-01-11T09:26:41.715730+0000","last_active":"2024-01-11T09:26:41.718051+0000","last_peered":"2024-01-11T09:26:41.718051+0000","last_clean":"2024-01-11T09:26:41.718051+0000","last_became_active":"2024-01-11T09:26:41.715302+0000","last_became_peered":"2024-01-11T09:26:41.715302+0000","last_unstale":"2024-01-11T09:26:41.718051+0000","last_undegraded":"2024-01-11T09:26:41.718051+0000","last_fullsized":"2024-01-11T09:26:41.718051+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:50:46.665371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"166'2","reported_seq":12,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.724285+0000","last_change":"2024-01-11T09:26:40.702279+0000","last_active":"2024-01-11T09:26:41.724285+0000","last_peered":"2024-01-11T09:26:41.724285+0000","last_clean":"2024-01-11T09:26:41.724285+0000","last_became_active":"2024-01-11T09:26:40.701971+0000","last_became_peered":"2024-01-11T09:26:40.701971+0000","last_unstale":"2024-01-11T09:26:41.724285+0000","last_undegraded":"2024-01-11T09:26:41.724285+0000","last_fullsized":"2024-01-11T09:26:41.724285+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:33:28.856947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.721140+0000","last_change":"2024-01-11T09:26:41.716816+0000","last_active":"2024-01-11T09:26:41.721140+0000","last_peered":"2024-01-11T09:26:41.721140+0000","last_clean":"2024-01-11T09:26:41.721140+0000","last_became_active":"2024-01-11T09:26:41.716550+0000","last_became_peered":"2024-01-11T09:26:41.716550+0000","last_unstale":"2024-01-11T09:26:41.721140+0000","last_undegraded":"2024-01-11T09:26:41.721140+0000","last_fullsized":"2024-01-11T09:26:41.721140+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:48:24.611840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"166'1","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.725073+0000","last_change":"2024-01-11T09:26:40.708917+0000","last_active":"2024-01-11T09:26:41.725073+0000","last_peered":"2024-01-11T09:26:41.725073+0000","last_clean":"2024-01-11T09:26:41.725073+0000","last_became_active":"2024-01-11T09:26:40.708566+0000","last_became_peered":"2024-01-11T09:26:40.708566+0000","last_unstale":"2024-01-11T09:26:41.725073+0000","last_undegraded":"2024-01-11T09:26:41.725073+0000","last_fullsized":"2024-01-11T09:26:41.725073+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:29:28.407605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.715489+0000","last_change":"2024-01-11T09:26:41.710944+0000","last_active":"2024-01-11T09:26:41.715489+0000","last_peered":"2024-01-11T09:26:41.715489+0000","last_clean":"2024-01-11T09:26:41.715489+0000","last_became_active":"2024-01-11T09:26:41.710616+0000","last_became_peered":"2024-01-11T09:26:41.710616+0000","last_unstale":"2024-01-11T09:26:41.715489+0000","last_undegraded":"2024-01-11T09:26:41.715489+0000","last_fullsized":"2024-01-11T09:26:41.715489+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:32:03.216586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.686331+0000","last_change":"2024-01-11T09:26:40.709183+0000","last_active":"2024-01-11T09:26:41.686331+0000","last_peered":"2024-01-11T09:26:41.686331+0000","last_clean":"2024-01-11T09:26:41.686331+0000","last_became_active":"2024-01-11T09:26:40.708875+0000","last_became_peered":"2024-01-11T09:26:40.708875+0000","last_unstale":"2024-01-11T09:26:41.686331+0000","last_undegraded":"2024-01-11T09:26:41.686331+0000","last_fullsized":"2024-01-11T09:26:41.686331+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:21:05.322778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.718828+0000","last_change":"2024-01-11T09:26:41.712495+0000","last_active":"2024-01-11T09:26:41.718828+0000","last_peered":"2024-01-11T09:26:41.718828+0000","last_clean":"2024-01-11T09:26:41.718828+0000","last_became_active":"2024-01-11T09:26:41.708251+0000","last_became_peered":"2024-01-11T09:26:41.708251+0000","last_unstale":"2024-01-11T09:26:41.718828+0000","last_undegraded":"2024-01-11T09:26:41.718828+0000","last_fullsized":"2024-01-11T09:26:41.718828+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:42:31.369318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"166'2","reported_seq":12,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.716196+0000","last_change":"2024-01-11T09:26:40.702474+0000","last_active":"2024-01-11T09:26:41.716196+0000","last_peered":"2024-01-11T09:26:41.716196+0000","last_clean":"2024-01-11T09:26:41.716196+0000","last_became_active":"2024-01-11T09:26:40.701275+0000","last_became_peered":"2024-01-11T09:26:40.701275+0000","last_unstale":"2024-01-11T09:26:41.716196+0000","last_undegraded":"2024-01-11T09:26:41.716196+0000","last_fullsized":"2024-01-11T09:26:41.716196+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:23:29.641605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.723814+0000","last_change":"2024-01-11T09:26:41.722435+0000","last_active":"2024-01-11T09:26:41.723814+0000","last_peered":"2024-01-11T09:26:41.723814+0000","last_clean":"2024-01-11T09:26:41.723814+0000","last_became_active":"2024-01-11T09:26:41.722177+0000","last_became_peered":"2024-01-11T09:26:41.722177+0000","last_unstale":"2024-01-11T09:26:41.723814+0000","last_undegraded":"2024-01-11T09:26:41.723814+0000","last_fullsized":"2024-01-11T09:26:41.723814+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:05:08.857476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.700261+0000","last_change":"2024-01-11T09:26:40.710813+0000","last_active":"2024-01-11T09:26:41.700261+0000","last_peered":"2024-01-11T09:26:41.700261+0000","last_clean":"2024-01-11T09:26:41.700261+0000","last_became_active":"2024-01-11T09:26:40.710549+0000","last_became_peered":"2024-01-11T09:26:40.710549+0000","last_unstale":"2024-01-11T09:26:41.700261+0000","last_undegraded":"2024-01-11T09:26:41.700261+0000","last_fullsized":"2024-01-11T09:26:41.700261+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:22:02.081826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.708786+0000","last_change":"2024-01-11T09:26:41.699834+0000","last_active":"2024-01-11T09:26:41.708786+0000","last_peered":"2024-01-11T09:26:41.708786+0000","last_clean":"2024-01-11T09:26:41.708786+0000","last_became_active":"2024-01-11T09:26:41.699406+0000","last_became_peered":"2024-01-11T09:26:41.699406+0000","last_unstale":"2024-01-11T09:26:41.708786+0000","last_undegraded":"2024-01-11T09:26:41.708786+0000","last_fullsized":"2024-01-11T09:26:41.708786+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:38:16.938970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.700131+0000","last_change":"2024-01-11T09:26:40.712437+0000","last_active":"2024-01-11T09:26:41.700131+0000","last_peered":"2024-01-11T09:26:41.700131+0000","last_clean":"2024-01-11T09:26:41.700131+0000","last_became_active":"2024-01-11T09:26:40.712094+0000","last_became_peered":"2024-01-11T09:26:40.712094+0000","last_unstale":"2024-01-11T09:26:41.700131+0000","last_undegraded":"2024-01-11T09:26:41.700131+0000","last_fullsized":"2024-01-11T09:26:41.700131+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:21:05.322778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.720366+0000","last_change":"2024-01-11T09:26:41.715119+0000","last_active":"2024-01-11T09:26:41.720366+0000","last_peered":"2024-01-11T09:26:41.720366+0000","last_clean":"2024-01-11T09:26:41.720366+0000","last_became_active":"2024-01-11T09:26:41.714539+0000","last_became_peered":"2024-01-11T09:26:41.714539+0000","last_unstale":"2024-01-11T09:26:41.720366+0000","last_undegraded":"2024-01-11T09:26:41.720366+0000","last_fullsized":"2024-01-11T09:26:41.720366+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:36:35.787804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.695709+0000","last_change":"2024-01-11T09:26:40.702598+0000","last_active":"2024-01-11T09:26:41.695709+0000","last_peered":"2024-01-11T09:26:41.695709+0000","last_clean":"2024-01-11T09:26:41.695709+0000","last_became_active":"2024-01-11T09:26:40.702301+0000","last_became_peered":"2024-01-11T09:26:40.702301+0000","last_unstale":"2024-01-11T09:26:41.695709+0000","last_undegraded":"2024-01-11T09:26:41.695709+0000","last_fullsized":"2024-01-11T09:26:41.695709+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:23:29.641605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.714063+0000","last_change":"2024-01-11T09:26:41.706765+0000","last_active":"2024-01-11T09:26:41.714063+0000","last_peered":"2024-01-11T09:26:41.714063+0000","last_clean":"2024-01-11T09:26:41.714063+0000","last_became_active":"2024-01-11T09:26:41.706458+0000","last_became_peered":"2024-01-11T09:26:41.706458+0000","last_unstale":"2024-01-11T09:26:41.714063+0000","last_undegraded":"2024-01-11T09:26:41.714063+0000","last_fullsized":"2024-01-11T09:26:41.714063+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:26:44.657352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"166'2","reported_seq":12,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.713534+0000","last_change":"2024-01-11T09:26:40.713344+0000","last_active":"2024-01-11T09:26:41.713534+0000","last_peered":"2024-01-11T09:26:41.713534+0000","last_clean":"2024-01-11T09:26:41.713534+0000","last_became_active":"2024-01-11T09:26:40.713063+0000","last_became_peered":"2024-01-11T09:26:40.713063+0000","last_unstale":"2024-01-11T09:26:41.713534+0000","last_undegraded":"2024-01-11T09:26:41.713534+0000","last_fullsized":"2024-01-11T09:26:41.713534+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:38:15.931504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.715627+0000","last_change":"2024-01-11T09:26:41.713497+0000","last_active":"2024-01-11T09:26:41.715627+0000","last_peered":"2024-01-11T09:26:41.715627+0000","last_clean":"2024-01-11T09:26:41.715627+0000","last_became_active":"2024-01-11T09:26:41.713226+0000","last_became_peered":"2024-01-11T09:26:41.713226+0000","last_unstale":"2024-01-11T09:26:41.715627+0000","last_undegraded":"2024-01-11T09:26:41.715627+0000","last_fullsized":"2024-01-11T09:26:41.715627+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:44:23.252777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.699729+0000","last_change":"2024-01-11T09:26:40.709098+0000","last_active":"2024-01-11T09:26:41.699729+0000","last_peered":"2024-01-11T09:26:41.699729+0000","last_clean":"2024-01-11T09:26:41.699729+0000","last_became_active":"2024-01-11T09:26:40.708861+0000","last_became_peered":"2024-01-11T09:26:40.708861+0000","last_unstale":"2024-01-11T09:26:41.699729+0000","last_undegraded":"2024-01-11T09:26:41.699729+0000","last_fullsized":"2024-01-11T09:26:41.699729+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:35:55.907519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.712593+0000","last_change":"2024-01-11T09:26:41.706364+0000","last_active":"2024-01-11T09:26:41.712593+0000","last_peered":"2024-01-11T09:26:41.712593+0000","last_clean":"2024-01-11T09:26:41.712593+0000","last_became_active":"2024-01-11T09:26:41.706042+0000","last_became_peered":"2024-01-11T09:26:41.706042+0000","last_unstale":"2024-01-11T09:26:41.712593+0000","last_undegraded":"2024-01-11T09:26:41.712593+0000","last_fullsized":"2024-01-11T09:26:41.712593+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:58:29.815846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.698965+0000","last_change":"2024-01-11T09:26:40.708265+0000","last_active":"2024-01-11T09:26:41.698965+0000","last_peered":"2024-01-11T09:26:41.698965+0000","last_clean":"2024-01-11T09:26:41.698965+0000","last_became_active":"2024-01-11T09:26:40.707960+0000","last_became_peered":"2024-01-11T09:26:40.707960+0000","last_unstale":"2024-01-11T09:26:41.698965+0000","last_undegraded":"2024-01-11T09:26:41.698965+0000","last_fullsized":"2024-01-11T09:26:41.698965+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:29:28.407605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.722432+0000","last_change":"2024-01-11T09:26:41.719441+0000","last_active":"2024-01-11T09:26:41.722432+0000","last_peered":"2024-01-11T09:26:41.722432+0000","last_clean":"2024-01-11T09:26:41.722432+0000","last_became_active":"2024-01-11T09:26:41.719162+0000","last_became_peered":"2024-01-11T09:26:41.719162+0000","last_unstale":"2024-01-11T09:26:41.722432+0000","last_undegraded":"2024-01-11T09:26:41.722432+0000","last_fullsized":"2024-01-11T09:26:41.722432+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:26:44.657352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.699391+0000","last_change":"2024-01-11T09:26:40.710161+0000","last_active":"2024-01-11T09:26:41.699391+0000","last_peered":"2024-01-11T09:26:41.699391+0000","last_clean":"2024-01-11T09:26:41.699391+0000","last_became_active":"2024-01-11T09:26:40.709552+0000","last_became_peered":"2024-01-11T09:26:40.709552+0000","last_unstale":"2024-01-11T09:26:41.699391+0000","last_undegraded":"2024-01-11T09:26:41.699391+0000","last_fullsized":"2024-01-11T09:26:41.699391+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:31:10.299073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.724574+0000","last_change":"2024-01-11T09:26:41.720112+0000","last_active":"2024-01-11T09:26:41.724574+0000","last_peered":"2024-01-11T09:26:41.724574+0000","last_clean":"2024-01-11T09:26:41.724574+0000","last_became_active":"2024-01-11T09:26:41.717446+0000","last_became_peered":"2024-01-11T09:26:41.717446+0000","last_unstale":"2024-01-11T09:26:41.724574+0000","last_undegraded":"2024-01-11T09:26:41.724574+0000","last_fullsized":"2024-01-11T09:26:41.724574+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:32:03.216586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.701715+0000","last_change":"2024-01-11T09:26:40.704917+0000","last_active":"2024-01-11T09:26:41.701715+0000","last_peered":"2024-01-11T09:26:41.701715+0000","last_clean":"2024-01-11T09:26:41.701715+0000","last_became_active":"2024-01-11T09:26:40.703795+0000","last_became_peered":"2024-01-11T09:26:40.703795+0000","last_unstale":"2024-01-11T09:26:41.701715+0000","last_undegraded":"2024-01-11T09:26:41.701715+0000","last_fullsized":"2024-01-11T09:26:41.701715+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:19.312112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.714404+0000","last_change":"2024-01-11T09:26:41.707143+0000","last_active":"2024-01-11T09:26:41.714404+0000","last_peered":"2024-01-11T09:26:41.714404+0000","last_clean":"2024-01-11T09:26:41.714404+0000","last_became_active":"2024-01-11T09:26:41.706836+0000","last_became_peered":"2024-01-11T09:26:41.706836+0000","last_unstale":"2024-01-11T09:26:41.714404+0000","last_undegraded":"2024-01-11T09:26:41.714404+0000","last_fullsized":"2024-01-11T09:26:41.714404+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:31:19.426090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.700368+0000","last_change":"2024-01-11T09:26:40.713489+0000","last_active":"2024-01-11T09:26:41.700368+0000","last_peered":"2024-01-11T09:26:41.700368+0000","last_clean":"2024-01-11T09:26:41.700368+0000","last_became_active":"2024-01-11T09:26:40.713030+0000","last_became_peered":"2024-01-11T09:26:40.713030+0000","last_unstale":"2024-01-11T09:26:41.700368+0000","last_undegraded":"2024-01-11T09:26:41.700368+0000","last_fullsized":"2024-01-11T09:26:41.700368+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:34:27.129163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.724031+0000","last_change":"2024-01-11T09:26:41.718986+0000","last_active":"2024-01-11T09:26:41.724031+0000","last_peered":"2024-01-11T09:26:41.724031+0000","last_clean":"2024-01-11T09:26:41.724031+0000","last_became_active":"2024-01-11T09:26:41.718563+0000","last_became_peered":"2024-01-11T09:26:41.718563+0000","last_unstale":"2024-01-11T09:26:41.724031+0000","last_undegraded":"2024-01-11T09:26:41.724031+0000","last_fullsized":"2024-01-11T09:26:41.724031+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:46:54.067657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.697911+0000","last_change":"2024-01-11T09:26:40.703957+0000","last_active":"2024-01-11T09:26:41.697911+0000","last_peered":"2024-01-11T09:26:41.697911+0000","last_clean":"2024-01-11T09:26:41.697911+0000","last_became_active":"2024-01-11T09:26:40.703671+0000","last_became_peered":"2024-01-11T09:26:40.703671+0000","last_unstale":"2024-01-11T09:26:41.697911+0000","last_undegraded":"2024-01-11T09:26:41.697911+0000","last_fullsized":"2024-01-11T09:26:41.697911+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:38:52.866733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.715636+0000","last_change":"2024-01-11T09:26:41.712143+0000","last_active":"2024-01-11T09:26:41.715636+0000","last_peered":"2024-01-11T09:26:41.715636+0000","last_clean":"2024-01-11T09:26:41.715636+0000","last_became_active":"2024-01-11T09:26:41.711868+0000","last_became_peered":"2024-01-11T09:26:41.711868+0000","last_unstale":"2024-01-11T09:26:41.715636+0000","last_undegraded":"2024-01-11T09:26:41.715636+0000","last_fullsized":"2024-01-11T09:26:41.715636+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:42:39.118184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.686144+0000","last_change":"2024-01-11T09:26:40.706610+0000","last_active":"2024-01-11T09:26:41.686144+0000","last_peered":"2024-01-11T09:26:41.686144+0000","last_clean":"2024-01-11T09:26:41.686144+0000","last_became_active":"2024-01-11T09:26:40.706341+0000","last_became_peered":"2024-01-11T09:26:40.706341+0000","last_unstale":"2024-01-11T09:26:41.686144+0000","last_undegraded":"2024-01-11T09:26:41.686144+0000","last_fullsized":"2024-01-11T09:26:41.686144+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:34:27.129163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.722945+0000","last_change":"2024-01-11T09:26:41.719213+0000","last_active":"2024-01-11T09:26:41.722945+0000","last_peered":"2024-01-11T09:26:41.722945+0000","last_clean":"2024-01-11T09:26:41.722945+0000","last_became_active":"2024-01-11T09:26:41.718614+0000","last_became_peered":"2024-01-11T09:26:41.718614+0000","last_unstale":"2024-01-11T09:26:41.722945+0000","last_undegraded":"2024-01-11T09:26:41.722945+0000","last_fullsized":"2024-01-11T09:26:41.722945+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:40.671636+0000","last_clean_scrub_stamp":"2024-01-11T09:26:40.671636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:22.767311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"166'3","reported_seq":13,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.724477+0000","last_change":"2024-01-11T09:26:40.702628+0000","last_active":"2024-01-11T09:26:41.724477+0000","last_peered":"2024-01-11T09:26:41.724477+0000","last_clean":"2024-01-11T09:26:41.724477+0000","last_became_active":"2024-01-11T09:26:40.702323+0000","last_became_peered":"2024-01-11T09:26:40.702323+0000","last_unstale":"2024-01-11T09:26:41.724477+0000","last_undegraded":"2024-01-11T09:26:41.724477+0000","last_fullsized":"2024-01-11T09:26:41.724477+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:26:39.664170+0000","last_clean_scrub_stamp":"2024-01-11T09:26:39.664170+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:19.312112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"157'289","reported_seq":888,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-01-11T09:26:41.699920+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:26:41.699920+0000","last_peered":"2024-01-11T09:26:41.699920+0000","last_clean":"2024-01-11T09:26:41.699920+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:26:41.699920+0000","last_undegraded":"2024-01-11T09:26:41.699920+0000","last_fullsized":"2024-01-11T09:26:41.699920+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1616,"num_read_kb":3366,"num_write":471,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1616,"num_read_kb":3366,"num_write":471,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":289,"ondisk_log_size":289,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575754,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312200,"kb_used_data":2624,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059640,"statfs":{"total":96636764160,"available":96317071360,"internally_reserved":0,"allocated":2686976,"data_stored":1576285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.75800000000000001,"15min":0.73399999999999999},"min":{"1min":0.52400000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.97999999999999998,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.74299999999999999,"15min":0.73299999999999998},"min":{"1min":0.55500000000000005,"5min":0.52000000000000002,"15min":0.45600000000000002},"max":{"1min":0.91500000000000004,"5min":0.99299999999999999,"15min":1.028},"last":0.58299999999999996}]},{"osd":1,"last update":"Thu Jan 11 09:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.73799999999999999,"15min":0.72699999999999998},"min":{"1min":0.502,"5min":0.497,"15min":0.42399999999999999},"max":{"1min":0.91300000000000003,"5min":1.034,"15min":1.034},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76500000000000001,"15min":0.754},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.49299999999999999},"max":{"1min":0.96099999999999997,"5min":1.0069999999999999,"15min":1.0409999999999999},"last":0.78600000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.75900000000000001,"15min":0.754},"min":{"1min":0.53900000000000003,"5min":0.52100000000000002,"15min":0.46899999999999997},"max":{"1min":0.93400000000000005,"5min":1.0089999999999999,"15min":1.0880000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.74299999999999999,"15min":0.73899999999999999},"min":{"1min":0.53700000000000003,"5min":0.51000000000000001,"15min":0.50800000000000001},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.70199999999999996}]},{"osd":3,"last update":"Thu Jan 11 09:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.77000000000000002,"15min":0.76500000000000001},"min":{"1min":0.57499999999999996,"5min":0.53700000000000003,"15min":0.52700000000000002},"max":{"1min":0.94999999999999996,"5min":1.046,"15min":1.056},"last":0.752},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.76200000000000001,"15min":0.76700000000000002},"min":{"1min":0.54400000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.997,"5min":1.006,"15min":1.012},"last":0.76800000000000002}]},{"osd":4,"last update":"Thu Jan 11 09:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59699999999999998,"15min":0.60199999999999998},"min":{"1min":0.46400000000000002,"5min":0.45300000000000001,"15min":0.40600000000000003},"max":{"1min":0.746,"5min":0.94599999999999995,"15min":1.004},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.63900000000000001,"15min":0.64700000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.82099999999999995,"5min":1.016,"15min":1.0940000000000001},"last":0.499}]},{"osd":5,"last update":"Thu Jan 11 09:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63100000000000001,"15min":0.64500000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81200000000000006,"5min":0.96499999999999997,"15min":1.0309999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.67400000000000004,"15min":0.68300000000000005},"min":{"1min":0.53800000000000003,"5min":0.41999999999999998,"15min":0.41299999999999998},"max":{"1min":0.879,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.83399999999999996}]},{"osd":6,"last update":"Thu Jan 11 09:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.76800000000000002,"15min":0.77800000000000002},"min":{"1min":0.61499999999999999,"5min":0.50900000000000001,"15min":0.47299999999999998},"max":{"1min":0.90000000000000002,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.749,"15min":0.75900000000000001},"min":{"1min":0.44900000000000001,"5min":0.40999999999999998,"15min":0.40100000000000002},"max":{"1min":1.014,"5min":1.073,"15min":1.073},"last":0.59899999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575753,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307528,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064312,"statfs":{"total":96636764160,"available":96321855488,"internally_reserved":0,"allocated":2097152,"data_stored":985439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.70599999999999996,"15min":0.70499999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.40500000000000003},"max":{"1min":0.91200000000000003,"5min":0.98099999999999998,"15min":1.095},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.71799999999999997,"15min":0.71999999999999997},"min":{"1min":0.53500000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.92700000000000005,"5min":0.97899999999999998,"15min":1.0489999999999999},"last":0.745}]},{"osd":1,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.72599999999999998,"15min":0.72699999999999998},"min":{"1min":0.54100000000000004,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93999999999999995,"5min":0.96499999999999997,"15min":1.077},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71399999999999997,"15min":0.71899999999999997},"min":{"1min":0.49099999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.94299999999999995,"5min":0.96999999999999997,"15min":1.01},"last":0.85899999999999999}]},{"osd":2,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.74399999999999999,"15min":0.73599999999999999},"min":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.49399999999999999},"max":{"1min":1.006,"5min":1.0129999999999999,"15min":1.0309999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72999999999999998,"15min":0.75},"min":{"1min":0.54500000000000004,"5min":0.51600000000000001,"15min":0.505},"max":{"1min":0.89300000000000002,"5min":0.94599999999999995,"15min":1.161},"last":0.95899999999999996}]},{"osd":3,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76200000000000001,"15min":0.76000000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.47499999999999998},"max":{"1min":0.91300000000000003,"5min":0.999,"15min":1.0329999999999999},"last":1.026},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76300000000000001,"15min":0.76700000000000002},"min":{"1min":0.57199999999999995,"5min":0.53700000000000003,"15min":0.503},"max":{"1min":0.91300000000000003,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.97599999999999998}]},{"osd":4,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.45900000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76100000000000001,"5min":0.82999999999999996,"15min":1.0469999999999999},"last":1.006},{"interface":"front","average":{"1min":0.623,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85799999999999998,"5min":0.97199999999999998,"15min":1.0569999999999999},"last":1.081}]},{"osd":5,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.67600000000000005,"15min":0.66700000000000004},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.39200000000000002},"max":{"1min":0.89700000000000002,"5min":1.032,"15min":1.032},"last":1.05},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.69999999999999996,"15min":0.71199999999999997},"min":{"1min":0.441,"5min":0.441,"15min":0.41699999999999998},"max":{"1min":0.93300000000000005,"5min":1.052,"15min":1.1299999999999999},"last":1.0660000000000001}]},{"osd":7,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.71099999999999997,"15min":0.71899999999999997},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.435},"max":{"1min":0.96799999999999997,"5min":1.034,"15min":1.1080000000000001},"last":1.143},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.75800000000000001,"15min":0.75900000000000001},"min":{"1min":0.501,"5min":0.5,"15min":0.47299999999999998},"max":{"1min":0.95499999999999996,"5min":1.081,"15min":1.0840000000000001},"last":1.1699999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575755,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307520,"kb_used_data":2040,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064320,"statfs":{"total":96636764160,"available":96321863680,"internally_reserved":0,"allocated":2088960,"data_stored":985847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51400000000000001,"15min":0.52300000000000002},"min":{"1min":0.36899999999999999,"5min":0.313,"15min":0.30199999999999999},"max":{"1min":0.76900000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.498,"5min":0.46100000000000002,"15min":0.39900000000000002},"max":{"1min":0.79400000000000004,"5min":0.81399999999999995,"15min":0.91500000000000004},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 11 09:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.50700000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.60999999999999999,"5min":0.76000000000000001,"15min":0.83699999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58299999999999996,"15min":0.58999999999999997},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.36899999999999999},"max":{"1min":0.83699999999999997,"5min":0.89800000000000002,"15min":0.91900000000000004},"last":0.65400000000000003}]},{"osd":3,"last update":"Thu Jan 11 09:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65200000000000002,"15min":0.65300000000000002},"min":{"1min":0.51900000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.77300000000000002,"5min":1.002,"15min":1.002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58599999999999997,"15min":0.58499999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.40500000000000003},"max":{"1min":0.748,"5min":0.80700000000000005,"15min":1.075},"last":0.67600000000000005}]},{"osd":4,"last update":"Thu Jan 11 09:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.81899999999999995,"15min":0.81399999999999995},"min":{"1min":0.621,"5min":0.53100000000000003,"15min":0.46999999999999997},"max":{"1min":1.129,"5min":1.1539999999999999,"15min":1.1839999999999999},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.878,"5min":0.873,"15min":0.87},"min":{"1min":0.69599999999999995,"5min":0.63300000000000001,"15min":0.60599999999999998},"max":{"1min":1.04,"5min":1.1779999999999999,"15min":1.198},"last":0.85399999999999998}]},{"osd":5,"last update":"Thu Jan 11 09:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.82899999999999996,"15min":0.82299999999999995},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.98799999999999999,"5min":1.212,"15min":1.212},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.81200000000000006,"15min":0.81299999999999994},"min":{"1min":0.64800000000000002,"5min":0.623,"15min":0.55400000000000005},"max":{"1min":0.93999999999999995,"5min":1.0800000000000001,"15min":1.248},"last":0.78500000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.85599999999999998,"15min":0.84999999999999998},"min":{"1min":0.68500000000000005,"5min":0.61599999999999999,"15min":0.57799999999999996},"max":{"1min":1.0169999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.874,"5min":0.85199999999999998,"15min":0.84399999999999997},"min":{"1min":0.68400000000000005,"5min":0.59199999999999997,"15min":0.54400000000000004},"max":{"1min":1.016,"5min":1.1499999999999999,"15min":1.2529999999999999},"last":0.95299999999999996}]},{"osd":7,"last update":"Thu Jan 11 09:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.879,"15min":0.86899999999999999},"min":{"1min":0.78400000000000003,"5min":0.64400000000000002,"15min":0.61699999999999999},"max":{"1min":1.083,"5min":1.2130000000000001,"15min":1.218},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.877,"5min":0.84899999999999998,"15min":0.85299999999999998},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.1180000000000001,"5min":1.1950000000000001,"15min":1.244},"last":0.83099999999999996}]}]},{"osd":0,"up_from":13,"seq":55834575754,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304016,"kb_used_data":2632,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067824,"statfs":{"total":96636764160,"available":96325451776,"internally_reserved":0,"allocated":2695168,"data_stored":1577231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.52400000000000002,"15min":0.52000000000000002},"min":{"1min":0.311,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.70199999999999996,"5min":0.751,"15min":0.79600000000000004},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60299999999999998,"15min":0.59699999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.42499999999999999},"max":{"1min":0.72399999999999998,"5min":0.83699999999999997,"15min":0.94599999999999995},"last":0.46600000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55300000000000005,"15min":0.55400000000000005},"min":{"1min":0.41399999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.80600000000000005,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55200000000000005,"15min":0.57499999999999996},"min":{"1min":0.41399999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89900000000000002},"last":0.50900000000000001}]},{"osd":3,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70599999999999996,"5min":0.79400000000000004,"15min":0.90600000000000003},"last":0.379},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63400000000000001,"15min":0.64300000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.79900000000000004,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.48699999999999999}]},{"osd":4,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79200000000000004,"15min":0.81699999999999995},"min":{"1min":0.59599999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.075,"5min":1.075,"15min":1.4239999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.80800000000000005,"15min":0.81200000000000006},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.1659999999999999,"5min":1.2050000000000001,"15min":1.2749999999999999},"last":0.92500000000000004}]},{"osd":5,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.81599999999999995,"15min":0.83299999999999996},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.57399999999999995},"max":{"1min":1.0249999999999999,"5min":1.1140000000000001,"15min":1.391},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.84199999999999997,"15min":0.84099999999999997},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.151,"5min":1.151,"15min":1.484},"last":0.91000000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.82599999999999996,"15min":0.82899999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.353},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.85399999999999998,"15min":0.86399999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.3069999999999999},"last":0.79300000000000004}]},{"osd":7,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.86799999999999999,"15min":0.875},"min":{"1min":0.60899999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.125,"5min":1.2010000000000001,"15min":1.246},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.872,"15min":0.86399999999999999},"min":{"1min":0.67300000000000004,"5min":0.63600000000000001,"15min":0.59999999999999998},"max":{"1min":1.048,"5min":1.157,"15min":1.272},"last":0.86799999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311628,"kb_used_data":2052,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060212,"statfs":{"total":96636764160,"available":96317657088,"internally_reserved":0,"allocated":2101248,"data_stored":986899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.497,"15min":0.5},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55100000000000005,"15min":0.55400000000000005},"min":{"1min":0.34599999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.64500000000000002}]},{"osd":1,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.45100000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79700000000000004,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60899999999999999,"15min":0.60699999999999998},"min":{"1min":0.41199999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.82699999999999996,"5min":0.93600000000000005,"15min":1},"last":0.70099999999999996}]},{"osd":3,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54900000000000004,"15min":0.55700000000000005},"min":{"1min":0.40400000000000003,"5min":0.37,"15min":0.34799999999999998},"max":{"1min":0.79500000000000004,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65600000000000003,"15min":0.66100000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.623}]},{"osd":4,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.80700000000000005,"15min":0.81399999999999995},"min":{"1min":0.68200000000000005,"5min":0.55300000000000005,"15min":0.54000000000000004},"max":{"1min":1.1220000000000001,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.871},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81299999999999994,"15min":0.79800000000000004},"min":{"1min":0.69199999999999995,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.036,"5min":1.391,"15min":1.391},"last":0.97899999999999998}]},{"osd":5,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.878,"15min":0.86899999999999999},"min":{"1min":0.73599999999999999,"5min":0.621,"15min":0.61699999999999999},"max":{"1min":1.1240000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.88800000000000001,"15min":0.86799999999999999},"min":{"1min":0.72099999999999997,"5min":0.627,"15min":0.54700000000000004},"max":{"1min":1.238,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.96499999999999997}]},{"osd":6,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.84099999999999997,"15min":0.82999999999999996},"min":{"1min":0.71199999999999997,"5min":0.57399999999999995,"15min":0.48999999999999999},"max":{"1min":1.1180000000000001,"5min":1.534,"15min":1.534},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.72099999999999997,"5min":0.59199999999999997,"15min":0.56999999999999995},"max":{"1min":1.127,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.78900000000000003}]},{"osd":7,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.66500000000000004,"5min":0.63100000000000001,"15min":0.58799999999999997},"max":{"1min":1.2010000000000001,"5min":1.55,"15min":1.55},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.90400000000000003,"15min":0.89400000000000002},"min":{"1min":0.67900000000000005,"5min":0.54900000000000004,"15min":0.53600000000000003},"max":{"1min":1.087,"5min":1.593,"15min":1.593},"last":0.88500000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315712,"kb_used_data":2040,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94056128,"statfs":{"total":96636764160,"available":96313475072,"internally_reserved":0,"allocated":2088960,"data_stored":985383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52800000000000002,"15min":0.503},"min":{"1min":0.35899999999999999,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.81399999999999995,"15min":0.82999999999999996},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57999999999999996,"15min":0.58099999999999996},"min":{"1min":0.48499999999999999,"5min":0.45600000000000002,"15min":0.42099999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.378}]},{"osd":1,"last update":"Thu Jan 11 09:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53200000000000003,"15min":0.51800000000000002},"min":{"1min":0.39600000000000002,"5min":0.36699999999999999,"15min":0.32500000000000001},"max":{"1min":0.84299999999999997,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60399999999999998,"15min":0.59299999999999997},"min":{"1min":0.44500000000000001,"5min":0.42999999999999999,"15min":0.36299999999999999},"max":{"1min":0.80200000000000005,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.60799999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.621,"15min":0.61299999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.97799999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65600000000000003,"15min":0.64900000000000002},"min":{"1min":0.5,"5min":0.45800000000000002,"15min":0.443},"max":{"1min":0.85799999999999998,"5min":0.96299999999999997,"15min":1.0449999999999999},"last":0.67700000000000005}]},{"osd":4,"last update":"Thu Jan 11 09:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.81399999999999995,"15min":0.79000000000000004},"min":{"1min":0.63100000000000001,"5min":0.48199999999999998,"15min":0.44900000000000001},"max":{"1min":1.0289999999999999,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81999999999999995,"15min":0.80400000000000005},"min":{"1min":0.63,"5min":0.56799999999999995,"15min":0.52300000000000002},"max":{"1min":0.96099999999999997,"5min":1.0880000000000001,"15min":1.133},"last":0.70999999999999996}]},{"osd":5,"last update":"Thu Jan 11 09:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83299999999999996,"15min":0.81899999999999995},"min":{"1min":0.66000000000000003,"5min":0.63900000000000001,"15min":0.55200000000000005},"max":{"1min":1.002,"5min":1.0940000000000001,"15min":1.1499999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.85299999999999998,"15min":0.83399999999999996},"min":{"1min":0.64500000000000002,"5min":0.60999999999999999,"15min":0.55500000000000005},"max":{"1min":0.97299999999999998,"5min":1.167,"15min":1.167},"last":0.84599999999999997}]},{"osd":6,"last update":"Thu Jan 11 09:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.874,"15min":0.84999999999999998},"min":{"1min":0.73699999999999999,"5min":0.65400000000000003,"15min":0.58299999999999996},"max":{"1min":1.073,"5min":1.1699999999999999,"15min":1.1819999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83499999999999996,"15min":0.81599999999999995},"min":{"1min":0.66500000000000004,"5min":0.55900000000000005,"15min":0.49099999999999999},"max":{"1min":1.1140000000000001,"5min":1.1299999999999999,"15min":1.143},"last":0.69099999999999995}]},{"osd":7,"last update":"Thu Jan 11 09:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.88100000000000001,"15min":0.86499999999999999},"min":{"1min":0.76600000000000001,"5min":0.623,"15min":0.51800000000000002},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.196},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90700000000000003,"15min":0.88200000000000001},"min":{"1min":0.72299999999999998,"5min":0.70999999999999996,"15min":0.56499999999999995},"max":{"1min":1.0680000000000001,"5min":1.1659999999999999,"15min":1.2230000000000001},"last":0.90100000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575754,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303420,"kb_used_data":2036,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068420,"statfs":{"total":96636764160,"available":96326062080,"internally_reserved":0,"allocated":2084864,"data_stored":985293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.81100000000000005,"15min":0.78700000000000003},"min":{"1min":0.65800000000000003,"5min":0.437,"15min":0.35099999999999998},"max":{"1min":1.006,"5min":1.0369999999999999,"15min":1.048},"last":0.495},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.78600000000000003,"15min":0.77800000000000002},"min":{"1min":0.59199999999999997,"5min":0.55200000000000005,"15min":0.48999999999999999},"max":{"1min":0.94099999999999995,"5min":1.018,"15min":1.018},"last":0.65000000000000002}]},{"osd":1,"last update":"Thu Jan 11 09:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.747,"15min":0.73899999999999999},"min":{"1min":0.52200000000000002,"5min":0.46600000000000003,"15min":0.45300000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":1.03},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.745,"15min":0.73999999999999999},"min":{"1min":0.54400000000000004,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.89800000000000002,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.745}]},{"osd":2,"last update":"Thu Jan 11 09:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.77200000000000002,"15min":0.75800000000000001},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.42599999999999999},"max":{"1min":0.91100000000000003,"5min":1,"15min":1},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.75800000000000001},"min":{"1min":0.60099999999999998,"5min":0.49199999999999999,"15min":0.46100000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.0760000000000001},"last":0.68300000000000005}]},{"osd":3,"last update":"Thu Jan 11 09:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.73799999999999999,"15min":0.73499999999999999},"min":{"1min":0.496,"5min":0.442,"15min":0.442},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":1.0109999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.79000000000000004,"15min":0.78200000000000003},"min":{"1min":0.60899999999999999,"5min":0.54400000000000004,"15min":0.53100000000000003},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.80200000000000005}]},{"osd":5,"last update":"Thu Jan 11 09:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61599999999999999,"15min":0.61699999999999999},"min":{"1min":0.45100000000000001,"5min":0.43099999999999999,"15min":0.39400000000000002},"max":{"1min":0.83799999999999997,"5min":0.89300000000000002,"15min":0.94999999999999996},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63500000000000001,"15min":0.625},"min":{"1min":0.437,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78700000000000003,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.93500000000000005}]},{"osd":6,"last update":"Thu Jan 11 09:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.65200000000000002,"15min":0.65100000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.96999999999999997,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.877},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.69899999999999995,"15min":0.69699999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.96399999999999997},"last":0.86199999999999999}]},{"osd":7,"last update":"Thu Jan 11 09:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.68899999999999995,"15min":0.69399999999999995},"min":{"1min":0.50600000000000001,"5min":0.47699999999999998,"15min":0.47299999999999998},"max":{"1min":0.97699999999999998,"5min":1.139,"15min":1.139},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.73399999999999999,"15min":0.73999999999999999},"min":{"1min":0.55900000000000005,"5min":0.51400000000000001,"15min":0.48199999999999998},"max":{"1min":0.91700000000000004,"5min":1.016,"15min":1.0389999999999999},"last":0.91500000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575754,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307520,"kb_used_data":2040,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064320,"statfs":{"total":96636764160,"available":96321863680,"internally_reserved":0,"allocated":2088960,"data_stored":985315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.73499999999999999,"15min":0.73699999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.77700000000000002,"5min":1.006,"15min":1.0229999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.76100000000000001,"15min":0.75600000000000001},"min":{"1min":0.46500000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.84799999999999998,"5min":0.97599999999999998,"15min":1.1020000000000001},"last":0.84699999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.745,"15min":0.74099999999999999},"min":{"1min":0.50700000000000001,"5min":0.438,"15min":0.41399999999999998},"max":{"1min":0.997,"5min":0.997,"15min":1.046},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.74399999999999999,"15min":0.75800000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82399999999999995,"5min":0.997,"15min":1.0249999999999999},"last":0.91300000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.72599999999999998,"15min":0.747},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.76600000000000001,"5min":1.014,"15min":1.0569999999999999},"last":0.874},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.76400000000000001,"15min":0.75600000000000001},"min":{"1min":0.51700000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94599999999999995,"5min":0.97999999999999998,"15min":1.012},"last":0.93600000000000005}]},{"osd":3,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.76600000000000001,"15min":0.76900000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.85799999999999998,"5min":1.016,"15min":1.016},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.76900000000000002,"15min":0.77300000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.45800000000000002},"max":{"1min":0.91400000000000003,"5min":0.99199999999999999,"15min":1.0609999999999999},"last":0.752}]},{"osd":4,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.622,"15min":0.61199999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.64200000000000002,"15min":0.63200000000000001},"min":{"1min":0.48299999999999998,"5min":0.45800000000000002,"15min":0.38900000000000001},"max":{"1min":0.88900000000000001,"5min":0.91300000000000003,"15min":0.94899999999999995},"last":0.72499999999999998}]},{"osd":6,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.64700000000000002,"15min":0.64300000000000002},"min":{"1min":0.443,"5min":0.433,"15min":0.39400000000000002},"max":{"1min":0.874,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.62},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.44500000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.64600000000000002}]},{"osd":7,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.75,"15min":0.73599999999999999},"min":{"1min":0.53700000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.89300000000000002,"5min":0.997,"15min":1.0920000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.78900000000000003,"15min":0.78100000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96899999999999997,"5min":1.01,"15min":1.137},"last":0.80300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:26:46.604 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-11T09:26:46.605 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-11T09:26:46.654 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-01-11T09:26:46.658+0000 7fa7757f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T09:26:46.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.851+0000 7fa76bbff700 1 Processor -- start 2024-01-11T09:26:46.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.851+0000 7fa76bbff700 1 -- start start 2024-01-11T09:26:46.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.852+0000 7fa76bbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa7640b03e0 0x7fa7640b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:46.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.852+0000 7fa76bbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7640b29c0 0x7fa764159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:46.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.852+0000 7fa76bbff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7640cd730 con 0x7fa7640b07b0 2024-01-11T09:26:46.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.852+0000 7fa76bbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7640b2de0 con 0x7fa7640b03e0 2024-01-11T09:26:46.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.852+0000 7fa76bbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7640ccb20 con 0x7fa7640b29c0 2024-01-11T09:26:46.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.853+0000 7fa76919a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa7640b07b0 0x7fa764159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40432/0 (socket says 172.21.15.27:40432) 2024-01-11T09:26:46.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.853+0000 7fa76919a700 1 -- 172.21.15.27:0/2563274828 learned_addr learned my addr 172.21.15.27:0/2563274828 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:46.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.853+0000 7fa76999b700 1 --2- 172.21.15.27:0/2563274828 >> v2:172.21.15.77:3300/0 conn(0x7fa7640b03e0 0x7fa7640b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:46.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.853+0000 7fa76a19c700 1 --2- 172.21.15.27:0/2563274828 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7640b29c0 0x7fa764159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:46.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.853+0000 7fa768999700 1 -- 172.21.15.27:0/2563274828 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40766955 0 0) 0x7fa7640cd730 con 0x7fa7640b07b0 2024-01-11T09:26:46.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.853+0000 7fa768999700 1 -- 172.21.15.27:0/2563274828 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa750003570 con 0x7fa7640b07b0 2024-01-11T09:26:46.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.853+0000 7fa76999b700 1 -- 172.21.15.27:0/2563274828 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7640b29c0 msgr2=0x7fa764159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:46.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.853+0000 7fa76999b700 1 --2- 172.21.15.27:0/2563274828 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7640b29c0 0x7fa764159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.853+0000 7fa76999b700 1 -- 172.21.15.27:0/2563274828 >> v1:172.21.15.27:6789/0 conn(0x7fa7640b07b0 legacy=0x7fa764159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:46.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.853+0000 7fa76999b700 1 -- 172.21.15.27:0/2563274828 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7640cce00 con 0x7fa7640b03e0 2024-01-11T09:26:46.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.853+0000 7fa76a19c700 1 --2- 172.21.15.27:0/2563274828 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7640b29c0 0x7fa764159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:26:46.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.854+0000 7fa76999b700 1 --2- 172.21.15.27:0/2563274828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7640b03e0 0x7fa7640b2480 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa7600098e0 tx=0x7fa7600091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c6728ccbfb47a1c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:46.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.854+0000 7fa768999700 1 -- 172.21.15.27:0/2563274828 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760011630 con 0x7fa7640b03e0 2024-01-11T09:26:46.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.854+0000 7fa768999700 1 -- 172.21.15.27:0/2563274828 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa760011790 con 0x7fa7640b03e0 2024-01-11T09:26:46.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.854+0000 7fa768999700 1 -- 172.21.15.27:0/2563274828 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760011b30 con 0x7fa7640b03e0 2024-01-11T09:26:46.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.854+0000 7fa76bbff700 1 -- 172.21.15.27:0/2563274828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7640b03e0 msgr2=0x7fa7640b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:46.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.854+0000 7fa76bbff700 1 --2- 172.21.15.27:0/2563274828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7640b03e0 0x7fa7640b2480 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa7600098e0 tx=0x7fa7600091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.855+0000 7fa76bbff700 1 -- 172.21.15.27:0/2563274828 shutdown_connections 2024-01-11T09:26:46.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.855+0000 7fa76bbff700 1 --2- 172.21.15.27:0/2563274828 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7640b29c0 0x7fa764159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.855+0000 7fa76bbff700 1 --2- 172.21.15.27:0/2563274828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7640b03e0 0x7fa7640b2480 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.855+0000 7fa76bbff700 1 -- 172.21.15.27:0/2563274828 >> 172.21.15.27:0/2563274828 conn(0x7fa76406ed40 msgr2=0x7fa76406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:46.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.855+0000 7fa76bbff700 1 -- 172.21.15.27:0/2563274828 shutdown_connections 2024-01-11T09:26:46.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.855+0000 7fa76bbff700 1 -- 172.21.15.27:0/2563274828 wait complete. 2024-01-11T09:26:46.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.856+0000 7fa76bbff700 1 Processor -- start 2024-01-11T09:26:46.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.856+0000 7fa76bbff700 1 -- start start 2024-01-11T09:26:46.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.857+0000 7fa76bbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7640b03e0 0x7fa7641586d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:46.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.857+0000 7fa76bbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7640b07b0 0x7fa7640ae960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:46.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.857+0000 7fa76bbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7640b29c0 0x7fa7640c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:46.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.857+0000 7fa76bbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7640ce0e0 con 0x7fa7640b03e0 2024-01-11T09:26:46.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.857+0000 7fa76bbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa764169cb0 con 0x7fa7640b29c0 2024-01-11T09:26:46.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.857+0000 7fa76bbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa764169f90 con 0x7fa7640b07b0 2024-01-11T09:26:46.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.857+0000 7fa76999b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7640b03e0 0x7fa7641586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:46.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.858+0000 7fa76999b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7640b03e0 0x7fa7641586d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33006/0 (socket says 172.21.15.27:33006) 2024-01-11T09:26:46.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.858+0000 7fa76999b700 1 -- 172.21.15.27:0/2187303175 learned_addr learned my addr 172.21.15.27:0/2187303175 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:46.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.858+0000 7fa76919a700 1 --2- 172.21.15.27:0/2187303175 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7640b07b0 0x7fa7640ae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:46.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.858+0000 7fa76a19c700 1 --2- 172.21.15.27:0/2187303175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7640b29c0 0x7fa7640c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:46.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.858+0000 7fa76999b700 1 -- 172.21.15.27:0/2187303175 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7640b07b0 msgr2=0x7fa7640ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:46.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.858+0000 7fa76999b700 1 --2- 172.21.15.27:0/2187303175 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7640b07b0 0x7fa7640ae960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.858+0000 7fa76999b700 1 -- 172.21.15.27:0/2187303175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7640b29c0 msgr2=0x7fa7640c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:46.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.858+0000 7fa76999b700 1 --2- 172.21.15.27:0/2187303175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7640b29c0 0x7fa7640c8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:46.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.858+0000 7fa76999b700 1 -- 172.21.15.27:0/2187303175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa75c009a20 con 0x7fa7640b03e0 2024-01-11T09:26:46.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.858+0000 7fa76999b700 1 --2- 172.21.15.27:0/2187303175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7640b03e0 0x7fa7641586d0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fa764069540 tx=0x7fa760017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:46.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.859+0000 7fa759ffb700 1 -- 172.21.15.27:0/2187303175 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760011b10 con 0x7fa7640b03e0 2024-01-11T09:26:46.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.859+0000 7fa759ffb700 1 -- 172.21.15.27:0/2187303175 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7600253f0 con 0x7fa7640b03e0 2024-01-11T09:26:46.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.859+0000 7fa76bbff700 1 -- 172.21.15.27:0/2187303175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa760009420 con 0x7fa7640b03e0 2024-01-11T09:26:46.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.859+0000 7fa759ffb700 1 -- 172.21.15.27:0/2187303175 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7600257b0 con 0x7fa7640b03e0 2024-01-11T09:26:46.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.859+0000 7fa76bbff700 1 -- 172.21.15.27:0/2187303175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76416a550 con 0x7fa7640b03e0 2024-01-11T09:26:46.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.860+0000 7fa76bbff700 1 -- 172.21.15.27:0/2187303175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa76404c3c0 con 0x7fa7640b03e0 2024-01-11T09:26:46.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.861+0000 7fa759ffb700 1 -- 172.21.15.27:0/2187303175 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa760025ba0 con 0x7fa7640b03e0 2024-01-11T09:26:46.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.862+0000 7fa759ffb700 1 --2- 172.21.15.27:0/2187303175 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa748066f30 0x7fa7480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:46.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.862+0000 7fa759ffb700 1 -- 172.21.15.27:0/2187303175 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fa760085dc0 con 0x7fa7640b03e0 2024-01-11T09:26:46.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.862+0000 7fa76919a700 1 --2- 172.21.15.27:0/2187303175 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa748066f30 0x7fa7480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:46.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.863+0000 7fa76919a700 1 --2- 172.21.15.27:0/2187303175 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa748066f30 0x7fa7480693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa7640b5880 tx=0x7fa754009040 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:46.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:46.866+0000 7fa759ffb700 1 -- 172.21.15.27:0/2187303175 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa760053010 con 0x7fa7640b03e0 2024-01-11T09:26:47.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.148+0000 7fa76bbff700 1 -- 172.21.15.27:0/2187303175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa7640cd0e0 con 0x7fa7640b03e0 2024-01-11T09:26:47.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.153+0000 7fa759ffb700 1 -- 172.21.15.27:0/2187303175 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v41) v1 ==== 190+0+0 (secure 0 0 0) 0x7fa760056630 con 0x7fa7640b03e0 2024-01-11T09:26:47.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.155+0000 7fa76bbff700 1 -- 172.21.15.27:0/2187303175 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa748066f30 msgr2=0x7fa7480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:47.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.155+0000 7fa76bbff700 1 --2- 172.21.15.27:0/2187303175 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa748066f30 0x7fa7480693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa7640b5880 tx=0x7fa754009040 comp rx=0 tx=0).stop 2024-01-11T09:26:47.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.155+0000 7fa76bbff700 1 -- 172.21.15.27:0/2187303175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7640b03e0 msgr2=0x7fa7641586d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:47.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.155+0000 7fa76bbff700 1 --2- 172.21.15.27:0/2187303175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7640b03e0 0x7fa7641586d0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fa764069540 tx=0x7fa760017000 comp rx=0 tx=0).stop 2024-01-11T09:26:47.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.155+0000 7fa76bbff700 1 -- 172.21.15.27:0/2187303175 shutdown_connections 2024-01-11T09:26:47.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.155+0000 7fa76bbff700 1 --2- 172.21.15.27:0/2187303175 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa748066f30 0x7fa7480693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:47.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.155+0000 7fa76bbff700 1 --2- 172.21.15.27:0/2187303175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7640b29c0 0x7fa7640c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:47.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.155+0000 7fa76bbff700 1 --2- 172.21.15.27:0/2187303175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7640b03e0 0x7fa7641586d0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:47.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.155+0000 7fa76bbff700 1 --2- 172.21.15.27:0/2187303175 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7640b07b0 0x7fa7640ae960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:47.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.155+0000 7fa76bbff700 1 -- 172.21.15.27:0/2187303175 >> 172.21.15.27:0/2187303175 conn(0x7fa76406ed40 msgr2=0x7fa7640cb400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:47.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.155+0000 7fa76bbff700 1 -- 172.21.15.27:0/2187303175 shutdown_connections 2024-01-11T09:26:47.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.155+0000 7fa76bbff700 1 -- 172.21.15.27:0/2187303175 wait complete. 2024-01-11T09:26:47.153 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-11T09:26:47.169 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:26:47.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.413+0000 7f2162bff700 1 Processor -- start 2024-01-11T09:26:47.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.413+0000 7f2162bff700 1 -- start start 2024-01-11T09:26:47.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.414+0000 7f2162bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f215c0b03e0 0x7f215c0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:47.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.414+0000 7f2162bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f215c0b2e00 con 0x7f215c0b29c0 2024-01-11T09:26:47.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.414+0000 7f2162bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f215c0cd730 con 0x7f215c0b07b0 2024-01-11T09:26:47.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.414+0000 7f2162bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f215c0ccb20 con 0x7f215c0b03e0 2024-01-11T09:26:47.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.415+0000 7f216119c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f215c0b29c0 0x7f215c0aefc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40440/0 (socket says 172.21.15.27:40440) 2024-01-11T09:26:47.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.415+0000 7f216119c700 1 -- 172.21.15.27:0/2311368934 learned_addr learned my addr 172.21.15.27:0/2311368934 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:47.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.415+0000 7f216099b700 1 --2- 172.21.15.27:0/2311368934 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f215c0b03e0 0x7f215c0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:47.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.415+0000 7f215b7fe700 1 -- 172.21.15.27:0/2311368934 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709292078 0 0) 0x7f2154001490 con 0x7f215c0b29c0 2024-01-11T09:26:47.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.415+0000 7f215b7fe700 1 -- 172.21.15.27:0/2311368934 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f214c003570 con 0x7f215c0b29c0 2024-01-11T09:26:47.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.415+0000 7f215b7fe700 1 -- 172.21.15.27:0/2311368934 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413573599 0 0) 0x7f215c0cd730 con 0x7f215c0b07b0 2024-01-11T09:26:47.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.415+0000 7f215b7fe700 1 -- 172.21.15.27:0/2311368934 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2154001490 con 0x7f215c0b07b0 2024-01-11T09:26:47.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.415+0000 7f216099b700 1 -- 172.21.15.27:0/2311368934 >> v1:172.21.15.77:6789/0 conn(0x7f215c0b07b0 legacy=0x7f215c159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:47.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.415+0000 7f216099b700 1 -- 172.21.15.27:0/2311368934 >> v1:172.21.15.27:6789/0 conn(0x7f215c0b29c0 legacy=0x7f215c0aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:47.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.415+0000 7f215b7fe700 1 -- 172.21.15.27:0/2311368934 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 145958966 0 0) 0x7f214c003570 con 0x7f215c0b29c0 2024-01-11T09:26:47.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.416+0000 7f216099b700 1 -- 172.21.15.27:0/2311368934 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215c0cce00 con 0x7f215c0b03e0 2024-01-11T09:26:47.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.416+0000 7f216099b700 1 --2- 172.21.15.27:0/2311368934 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f215c0b03e0 0x7f215c0b2480 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2150009910 tx=0x7f21500091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9417b753f1ad5dbb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:47.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.416+0000 7f215b7fe700 1 -- 172.21.15.27:0/2311368934 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2154001980 con 0x7f215c0b03e0 2024-01-11T09:26:47.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.416+0000 7f215b7fe700 1 -- 172.21.15.27:0/2311368934 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2150011420 con 0x7f215c0b03e0 2024-01-11T09:26:47.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.416+0000 7f215b7fe700 1 -- 172.21.15.27:0/2311368934 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21500117c0 con 0x7f215c0b03e0 2024-01-11T09:26:47.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.417+0000 7f2162bff700 1 -- 172.21.15.27:0/2311368934 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f215c0b03e0 msgr2=0x7f215c0b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:47.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.417+0000 7f2162bff700 1 --2- 172.21.15.27:0/2311368934 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f215c0b03e0 0x7f215c0b2480 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2150009910 tx=0x7f21500091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:47.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.417+0000 7f2162bff700 1 -- 172.21.15.27:0/2311368934 shutdown_connections 2024-01-11T09:26:47.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.417+0000 7f2162bff700 1 --2- 172.21.15.27:0/2311368934 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f215c0b03e0 0x7f215c0b2480 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:47.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.417+0000 7f2162bff700 1 -- 172.21.15.27:0/2311368934 >> 172.21.15.27:0/2311368934 conn(0x7f215c06ed40 msgr2=0x7f215c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:47.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.417+0000 7f2162bff700 1 -- 172.21.15.27:0/2311368934 shutdown_connections 2024-01-11T09:26:47.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.417+0000 7f2162bff700 1 -- 172.21.15.27:0/2311368934 wait complete. 2024-01-11T09:26:47.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.418+0000 7f2162bff700 1 Processor -- start 2024-01-11T09:26:47.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.418+0000 7f2162bff700 1 -- start start 2024-01-11T09:26:47.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.419+0000 7f2162bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f215c0b03e0 0x7f215c15bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:47.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.419+0000 7f2162bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f215c0b07b0 0x7f215c0a8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:47.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.419+0000 7f2162bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f215c0b29c0 0x7f215c0a9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:47.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.419+0000 7f2162bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215c0c75c0 con 0x7f215c0b29c0 2024-01-11T09:26:47.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.419+0000 7f2162bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215c0c7b40 con 0x7f215c0b03e0 2024-01-11T09:26:47.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.419+0000 7f2162bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f215c0a7df0 con 0x7f215c0b07b0 2024-01-11T09:26:47.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.420+0000 7f216119c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f215c0b29c0 0x7f215c0a9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:47.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.420+0000 7f216119c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f215c0b29c0 0x7f215c0a9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33014/0 (socket says 172.21.15.27:33014) 2024-01-11T09:26:47.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.420+0000 7f216119c700 1 -- 172.21.15.27:0/3238804731 learned_addr learned my addr 172.21.15.27:0/3238804731 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:47.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.420+0000 7f216099b700 1 --2- 172.21.15.27:0/3238804731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f215c0b03e0 0x7f215c15bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:47.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.420+0000 7f215bfff700 1 --2- 172.21.15.27:0/3238804731 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f215c0b07b0 0x7f215c0a8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:47.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.420+0000 7f216119c700 1 -- 172.21.15.27:0/3238804731 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f215c0b07b0 msgr2=0x7f215c0a8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:47.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.420+0000 7f216119c700 1 --2- 172.21.15.27:0/3238804731 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f215c0b07b0 0x7f215c0a8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:47.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.420+0000 7f216119c700 1 -- 172.21.15.27:0/3238804731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f215c0b03e0 msgr2=0x7f215c15bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:47.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.420+0000 7f216119c700 1 --2- 172.21.15.27:0/3238804731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f215c0b03e0 0x7f215c15bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:47.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.420+0000 7f216119c700 1 -- 172.21.15.27:0/3238804731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2150009420 con 0x7f215c0b29c0 2024-01-11T09:26:47.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.420+0000 7f215bfff700 1 --2- 172.21.15.27:0/3238804731 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f215c0b07b0 0x7f215c0a8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:26:47.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.420+0000 7f216119c700 1 --2- 172.21.15.27:0/3238804731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f215c0b29c0 0x7f215c0a9110 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f215400cd80 tx=0x7f2154007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:47.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.420+0000 7f2158ff9700 1 -- 172.21.15.27:0/3238804731 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2154004dc0 con 0x7f215c0b29c0 2024-01-11T09:26:47.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.421+0000 7f2158ff9700 1 -- 172.21.15.27:0/3238804731 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f215400a8e0 con 0x7f215c0b29c0 2024-01-11T09:26:47.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.421+0000 7f2162bff700 1 -- 172.21.15.27:0/3238804731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f215c0a80d0 con 0x7f215c0b29c0 2024-01-11T09:26:47.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.421+0000 7f2158ff9700 1 -- 172.21.15.27:0/3238804731 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215400aca0 con 0x7f215c0b29c0 2024-01-11T09:26:47.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.421+0000 7f2162bff700 1 -- 172.21.15.27:0/3238804731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f215c0ca400 con 0x7f215c0b29c0 2024-01-11T09:26:47.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.422+0000 7f2162bff700 1 -- 172.21.15.27:0/3238804731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f215c0718d0 con 0x7f215c0b29c0 2024-01-11T09:26:47.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.423+0000 7f2158ff9700 1 -- 172.21.15.27:0/3238804731 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2154007360 con 0x7f215c0b29c0 2024-01-11T09:26:47.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.424+0000 7f2158ff9700 1 --2- 172.21.15.27:0/3238804731 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2140066f30 0x7f21400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:47.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.424+0000 7f2158ff9700 1 -- 172.21.15.27:0/3238804731 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f2154083fe0 con 0x7f215c0b29c0 2024-01-11T09:26:47.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.425+0000 7f216099b700 1 --2- 172.21.15.27:0/3238804731 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2140066f30 0x7f21400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:47.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.425+0000 7f216099b700 1 --2- 172.21.15.27:0/3238804731 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2140066f30 0x7f21400693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21500030c0 tx=0x7f2150016000 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:47.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.429+0000 7f2158ff9700 1 -- 172.21.15.27:0/3238804731 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2154051210 con 0x7f215c0b29c0 2024-01-11T09:26:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.676+0000 7f2162bff700 1 -- 172.21.15.27:0/3238804731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f215c0b2e00 con 0x7f215c0b29c0 2024-01-11T09:26:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.677+0000 7f2158ff9700 1 -- 172.21.15.27:0/3238804731 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4598 (secure 0 0 0) 0x7f215c0b2e00 con 0x7f215c0b29c0 2024-01-11T09:26:47.674 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:26:47.674 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":86,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7700,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/191611944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":191611944},{"type":"v1","addr":"172.21.15.27:6835","nonce":191611944}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":8734,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/3566978772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3566978772},{"type":"v1","addr":"172.21.15.77:6834","nonce":3566978772}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":8737,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1168169536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1168169536},{"type":"v1","addr":"172.21.15.77:6835","nonce":1168169536}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:26:41.190629+0000","modified":"2024-01-11T09:26:44.707534+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9093},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9093":{"gid":9093,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/3544429684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3544429684},{"type":"v1","addr":"172.21.15.27:6834","nonce":3544429684}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2024-01-11T09:26:47.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.679+0000 7f2162bff700 1 -- 172.21.15.27:0/3238804731 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2140066f30 msgr2=0x7f21400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:47.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.679+0000 7f2162bff700 1 --2- 172.21.15.27:0/3238804731 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2140066f30 0x7f21400693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21500030c0 tx=0x7f2150016000 comp rx=0 tx=0).stop 2024-01-11T09:26:47.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.679+0000 7f2162bff700 1 -- 172.21.15.27:0/3238804731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f215c0b29c0 msgr2=0x7f215c0a9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:47.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.679+0000 7f2162bff700 1 --2- 172.21.15.27:0/3238804731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f215c0b29c0 0x7f215c0a9110 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f215400cd80 tx=0x7f2154007d20 comp rx=0 tx=0).stop 2024-01-11T09:26:47.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.679+0000 7f2162bff700 1 -- 172.21.15.27:0/3238804731 shutdown_connections 2024-01-11T09:26:47.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.679+0000 7f2162bff700 1 --2- 172.21.15.27:0/3238804731 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2140066f30 0x7f21400693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:47.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.679+0000 7f2162bff700 1 --2- 172.21.15.27:0/3238804731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f215c0b29c0 0x7f215c0a9110 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:47.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.679+0000 7f2162bff700 1 --2- 172.21.15.27:0/3238804731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f215c0b03e0 0x7f215c15bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:47.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.679+0000 7f2162bff700 1 --2- 172.21.15.27:0/3238804731 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f215c0b07b0 0x7f215c0a8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:47.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.679+0000 7f2162bff700 1 -- 172.21.15.27:0/3238804731 >> 172.21.15.27:0/3238804731 conn(0x7f215c06ed40 msgr2=0x7f215c15c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:47.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.679+0000 7f2162bff700 1 -- 172.21.15.27:0/3238804731 shutdown_connections 2024-01-11T09:26:47.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:47.679+0000 7f2162bff700 1 -- 172.21.15.27:0/3238804731 wait complete. 2024-01-11T09:26:47.678 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 86 2024-01-11T09:26:48.098 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-11T09:26:41.190629+0000', 'modified': '2024-01-11T09:26:44.707534+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9093}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9093': {'gid': 9093, 'name': 'a', 'rank': 0, 'incarnation': 82, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/3544429684', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 3544429684}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 3544429684}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-11T09:26:48.098 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-11T09:26:48.098 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-11T09:26:48.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.302+0000 7f1561dff700 1 Processor -- start 2024-01-11T09:26:48.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.302+0000 7f1561dff700 1 -- start start 2024-01-11T09:26:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.304+0000 7f1561dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f155c155e30 0x7f155c0afc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.304+0000 7f1561dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f155c0b0190 0x7f155c0ac3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.304+0000 7f1561dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155c0aeb50 con 0x7f155c155a60 2024-01-11T09:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.304+0000 7f1561dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f155c0b0610 con 0x7f155c155e30 2024-01-11T09:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.304+0000 7f1561dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f155c0abf20 con 0x7f155c0b0190 2024-01-11T09:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.304+0000 7f155affd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f155c155e30 0x7f155c0afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.304+0000 7f155affd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f155c155e30 0x7f155c0afc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33030/0 (socket says 172.21.15.27:33030) 2024-01-11T09:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.304+0000 7f155affd700 1 -- 172.21.15.27:0/3106860738 learned_addr learned my addr 172.21.15.27:0/3106860738 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.304+0000 7f155b7fe700 1 --1- 172.21.15.27:0/3106860738 >> v1:172.21.15.27:6789/0 conn(0x7f155c155a60 0x7f155c0aef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40450/0 (socket says 172.21.15.27:40450) 2024-01-11T09:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.304+0000 7f155affd700 1 -- 172.21.15.27:0/3106860738 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f155c0b0190 msgr2=0x7f155c0ac3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.304+0000 7f155bfff700 1 --2- 172.21.15.27:0/3106860738 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f155c0b0190 0x7f155c0ac3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.305+0000 7f155affd700 1 --2- 172.21.15.27:0/3106860738 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f155c0b0190 0x7f155c0ac3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.305+0000 7f155affd700 1 -- 172.21.15.27:0/3106860738 >> v1:172.21.15.27:6789/0 conn(0x7f155c155a60 legacy=0x7f155c0aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.305+0000 7f155affd700 1 -- 172.21.15.27:0/3106860738 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c0ae590 con 0x7f155c155e30 2024-01-11T09:26:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.305+0000 7f155bfff700 1 --2- 172.21.15.27:0/3106860738 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f155c0b0190 0x7f155c0ac3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:26:48.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.305+0000 7f155a7fc700 1 -- 172.21.15.27:0/3106860738 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 351867285 0 0) 0x7f155c0aeb50 con 0x7f155c155a60 2024-01-11T09:26:48.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.305+0000 7f155affd700 1 --2- 172.21.15.27:0/3106860738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f155c155e30 0x7f155c0afc50 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f15500099b0 tx=0x7f1550009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=464c55db62013102 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:48.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.305+0000 7f155a7fc700 1 -- 172.21.15.27:0/3106860738 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155000b860 con 0x7f155c155e30 2024-01-11T09:26:48.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.305+0000 7f155a7fc700 1 -- 172.21.15.27:0/3106860738 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155000b9c0 con 0x7f155c155e30 2024-01-11T09:26:48.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.305+0000 7f155a7fc700 1 -- 172.21.15.27:0/3106860738 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155000bd60 con 0x7f155c155e30 2024-01-11T09:26:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.306+0000 7f1561dff700 1 -- 172.21.15.27:0/3106860738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f155c155e30 msgr2=0x7f155c0afc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.306+0000 7f1561dff700 1 --2- 172.21.15.27:0/3106860738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f155c155e30 0x7f155c0afc50 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f15500099b0 tx=0x7f1550009250 comp rx=0 tx=0).stop 2024-01-11T09:26:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.306+0000 7f1561dff700 1 -- 172.21.15.27:0/3106860738 shutdown_connections 2024-01-11T09:26:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.306+0000 7f1561dff700 1 --2- 172.21.15.27:0/3106860738 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f155c0b0190 0x7f155c0ac3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.306+0000 7f1561dff700 1 --2- 172.21.15.27:0/3106860738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f155c155e30 0x7f155c0afc50 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.306+0000 7f1561dff700 1 -- 172.21.15.27:0/3106860738 >> 172.21.15.27:0/3106860738 conn(0x7f155c0680b0 msgr2=0x7f155c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.306+0000 7f1561dff700 1 -- 172.21.15.27:0/3106860738 shutdown_connections 2024-01-11T09:26:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.306+0000 7f1561dff700 1 -- 172.21.15.27:0/3106860738 wait complete. 2024-01-11T09:26:48.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.308+0000 7f1561dff700 1 Processor -- start 2024-01-11T09:26:48.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.308+0000 7f1561dff700 1 -- start start 2024-01-11T09:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.309+0000 7f1561dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f155c0b0190 0x7f155c14deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.309+0000 7f1561dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f155c155a60 0x7f155c14fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.310+0000 7f1561dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f155c155e30 0x7f155c06e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.310+0000 7f1561dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c14e3f0 con 0x7f155c155e30 2024-01-11T09:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.310+0000 7f1561dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c1563c0 con 0x7f155c0b0190 2024-01-11T09:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.310+0000 7f1561dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f155c0c8c70 con 0x7f155c155a60 2024-01-11T09:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.310+0000 7f155bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f155c155e30 0x7f155c06e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.310+0000 7f155bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f155c155e30 0x7f155c06e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33032/0 (socket says 172.21.15.27:33032) 2024-01-11T09:26:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.310+0000 7f155bfff700 1 -- 172.21.15.27:0/1947222511 learned_addr learned my addr 172.21.15.27:0/1947222511 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.310+0000 7f155b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f155c0b0190 0x7f155c14deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.310+0000 7f155affd700 1 --2- 172.21.15.27:0/1947222511 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f155c155a60 0x7f155c14fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.310+0000 7f155bfff700 1 -- 172.21.15.27:0/1947222511 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f155c155a60 msgr2=0x7f155c14fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.310+0000 7f155bfff700 1 --2- 172.21.15.27:0/1947222511 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f155c155a60 0x7f155c14fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.310+0000 7f155bfff700 1 -- 172.21.15.27:0/1947222511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f155c0b0190 msgr2=0x7f155c14deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.310+0000 7f155bfff700 1 --2- 172.21.15.27:0/1947222511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f155c0b0190 0x7f155c14deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.310+0000 7f155bfff700 1 -- 172.21.15.27:0/1947222511 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15500094f0 con 0x7f155c155e30 2024-01-11T09:26:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.311+0000 7f155bfff700 1 --2- 172.21.15.27:0/1947222511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f155c155e30 0x7f155c06e530 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f155400a380 tx=0x7f1554007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.311+0000 7f15437fe700 1 -- 172.21.15.27:0/1947222511 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1554015440 con 0x7f155c155e30 2024-01-11T09:26:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.311+0000 7f15437fe700 1 -- 172.21.15.27:0/1947222511 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15540155a0 con 0x7f155c155e30 2024-01-11T09:26:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.311+0000 7f15437fe700 1 -- 172.21.15.27:0/1947222511 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1554015960 con 0x7f155c155e30 2024-01-11T09:26:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.311+0000 7f1561dff700 1 -- 172.21.15.27:0/1947222511 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f155c150180 con 0x7f155c155e30 2024-01-11T09:26:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.311+0000 7f1561dff700 1 -- 172.21.15.27:0/1947222511 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f155c14edc0 con 0x7f155c155e30 2024-01-11T09:26:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.313+0000 7f15437fe700 1 -- 172.21.15.27:0/1947222511 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1554015d50 con 0x7f155c155e30 2024-01-11T09:26:48.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.314+0000 7f15437fe700 1 --2- 172.21.15.27:0/1947222511 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1548066f30 0x7f15480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.314+0000 7f15437fe700 1 -- 172.21.15.27:0/1947222511 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f1554084e00 con 0x7f155c155e30 2024-01-11T09:26:48.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.315+0000 7f155b7fe700 1 --2- 172.21.15.27:0/1947222511 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1548066f30 0x7f15480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.315+0000 7f155b7fe700 1 --2- 172.21.15.27:0/1947222511 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1548066f30 0x7f15480693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f154c007fd0 tx=0x7f154c007d50 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:48.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.316+0000 7f15705b6b80 1 Processor -- start 2024-01-11T09:26:48.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.317+0000 7f15705b6b80 1 -- start start 2024-01-11T09:26:48.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.317+0000 7f15705b6b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b46db20 0x56529b44edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.317+0000 7f15705b6b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b4493a0 0x56529b475860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.318+0000 7f15705b6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 0x56529b475da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.318+0000 7f15705b6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529b366310 con 0x56529b46def0 2024-01-11T09:26:48.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.318+0000 7f15705b6b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529b3a3130 con 0x56529b4493a0 2024-01-11T09:26:48.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.318+0000 7f15705b6b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x56529b3a0af0 con 0x56529b46db20 2024-01-11T09:26:48.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.318+0000 7f155bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 0x56529b475da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.318+0000 7f155bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 0x56529b475da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33036/0 (socket says 172.21.15.27:33036) 2024-01-11T09:26:48.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.318+0000 7f155bfff700 1 -- 172.21.15.27:0/3360044819 learned_addr learned my addr 172.21.15.27:0/3360044819 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:48.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.318+0000 7f155affd700 1 --2- 172.21.15.27:0/3360044819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b4493a0 0x56529b475860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.318+0000 7f155b7fe700 1 --2- 172.21.15.27:0/3360044819 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b46db20 0x56529b44edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.318+0000 7f155bfff700 1 -- 172.21.15.27:0/3360044819 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b46db20 msgr2=0x56529b44edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:48.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.318+0000 7f155bfff700 1 --2- 172.21.15.27:0/3360044819 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b46db20 0x56529b44edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.318+0000 7f155bfff700 1 -- 172.21.15.27:0/3360044819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b4493a0 msgr2=0x56529b475860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.318+0000 7f155bfff700 1 --2- 172.21.15.27:0/3360044819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b4493a0 0x56529b475860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.318+0000 7f155bfff700 1 -- 172.21.15.27:0/3360044819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56529b28f0a0 con 0x56529b46def0 2024-01-11T09:26:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.318+0000 7f155affd700 1 --2- 172.21.15.27:0/3360044819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b4493a0 0x56529b475860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.319+0000 7f155bfff700 1 --2- 172.21.15.27:0/3360044819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 0x56529b475da0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f15540521f0 tx=0x7f1554083240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f48e09d1643f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.319+0000 7f15417fa700 1 -- 172.21.15.27:0/3360044819 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1554055ac0 con 0x56529b46def0 2024-01-11T09:26:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.319+0000 7f15417fa700 1 -- 172.21.15.27:0/3360044819 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1554058f40 con 0x56529b46def0 2024-01-11T09:26:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.319+0000 7f15417fa700 1 -- 172.21.15.27:0/3360044819 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155405cc90 con 0x56529b46def0 2024-01-11T09:26:48.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.319+0000 7f15705b6b80 1 -- 172.21.15.27:0/3360044819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 msgr2=0x56529b475da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:48.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.319+0000 7f15705b6b80 1 --2- 172.21.15.27:0/3360044819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 0x56529b475da0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f15540521f0 tx=0x7f1554083240 comp rx=0 tx=0).stop 2024-01-11T09:26:48.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.320+0000 7f15705b6b80 1 -- 172.21.15.27:0/3360044819 shutdown_connections 2024-01-11T09:26:48.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.320+0000 7f15705b6b80 1 --2- 172.21.15.27:0/3360044819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 0x56529b475da0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.320+0000 7f15705b6b80 1 --2- 172.21.15.27:0/3360044819 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b46db20 0x56529b44edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.320+0000 7f15705b6b80 1 --2- 172.21.15.27:0/3360044819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b4493a0 0x56529b475860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.320+0000 7f15705b6b80 1 -- 172.21.15.27:0/3360044819 >> 172.21.15.27:0/3360044819 conn(0x56529b44b300 msgr2=0x56529b44dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:48.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.320+0000 7f15705b6b80 1 -- 172.21.15.27:0/3360044819 shutdown_connections 2024-01-11T09:26:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.320+0000 7f15705b6b80 1 -- 172.21.15.27:0/3360044819 wait complete. 2024-01-11T09:26:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.320+0000 7f15705b6b80 1 -- start start 2024-01-11T09:26:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.321+0000 7f15705b6b80 1 Processor -- start 2024-01-11T09:26:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.322+0000 7f15705b6b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b4493a0 0x56529b494630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.322+0000 7f15705b6b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b46db20 0x56529b459910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.322+0000 7f15705b6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 0x56529b47c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.322+0000 7f15705b6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529b39c8e0 con 0x56529b46def0 2024-01-11T09:26:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.322+0000 7f15705b6b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529b348980 con 0x56529b46db20 2024-01-11T09:26:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.322+0000 7f15705b6b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x56529b348620 con 0x56529b4493a0 2024-01-11T09:26:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.322+0000 7f15705b6b80 10 client.0 _ll_register_callbacks cb 0x56529b1aba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T09:26:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.322+0000 7f155bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 0x56529b47c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.322+0000 7f155bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 0x56529b47c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33044/0 (socket says 172.21.15.27:33044) 2024-01-11T09:26:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.322+0000 7f155bfff700 1 -- 172.21.15.27:0/2062791085 learned_addr learned my addr 172.21.15.27:0/2062791085 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.322+0000 7f155b7fe700 1 --2- 172.21.15.27:0/2062791085 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b4493a0 0x56529b494630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.322+0000 7f155affd700 1 --2- 172.21.15.27:0/2062791085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b46db20 0x56529b459910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.322+0000 7f155bfff700 1 -- 172.21.15.27:0/2062791085 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b4493a0 msgr2=0x56529b494630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.322+0000 7f155bfff700 1 --2- 172.21.15.27:0/2062791085 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b4493a0 0x56529b494630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.322+0000 7f155bfff700 1 -- 172.21.15.27:0/2062791085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b46db20 msgr2=0x56529b459910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.322+0000 7f155bfff700 1 --2- 172.21.15.27:0/2062791085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b46db20 0x56529b459910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.322+0000 7f155bfff700 1 -- 172.21.15.27:0/2062791085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f155c14edc0 con 0x56529b46def0 2024-01-11T09:26:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.322+0000 7f155affd700 1 --2- 172.21.15.27:0/2062791085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b46db20 0x56529b459910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.322+0000 7f155b7fe700 1 --2- 172.21.15.27:0/2062791085 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b4493a0 0x56529b494630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.323+0000 7f155bfff700 1 --2- 172.21.15.27:0/2062791085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 0x56529b47c3a0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f1554017010 tx=0x7f1554021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=942b3628f1370661 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.323+0000 7f1539ffb700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T09:26:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.323+0000 7f1539ffb700 1 -- 172.21.15.27:0/2062791085 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155405cc90 con 0x56529b46def0 2024-01-11T09:26:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.323+0000 7f1539ffb700 1 -- 172.21.15.27:0/2062791085 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(167..167 src has 1..167) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f155405c800 con 0x56529b46def0 2024-01-11T09:26:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.323+0000 7f15705b6b80 1 -- 172.21.15.27:0/2062791085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56529b441560 con 0x56529b46def0 2024-01-11T09:26:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.323+0000 7f1539ffb700 1 -- 172.21.15.27:0/2062791085 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1554051ae0 con 0x56529b46def0 2024-01-11T09:26:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.323+0000 7f15705b6b80 10 client.9228 fetch_fsmap learned FSMap version 86 2024-01-11T09:26:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.323+0000 7f15705b6b80 1 -- 172.21.15.27:0/2062791085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=2+}) v3 -- 0x56529b36b3c0 con 0x56529b46def0 2024-01-11T09:26:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.323+0000 7f1539ffb700 1 -- 172.21.15.27:0/2062791085 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56529b441560 con 0x56529b46def0 2024-01-11T09:26:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.323+0000 7f1539ffb700 1 -- 172.21.15.27:0/2062791085 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 86) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f1554051c40 con 0x56529b46def0 2024-01-11T09:26:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.323+0000 7f15705b6b80 10 client.9228 fetch_fsmap finished waiting for FSMap version 86 2024-01-11T09:26:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.323+0000 7f15705b6b80 10 client.9228 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-11T09:26:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.324+0000 7f15705b6b80 20 client.9228 populate_metadata read hostname 'smithi027' 2024-01-11T09:26:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.324+0000 7f15705b6b80 1 --2- 172.21.15.27:0/2062791085 >> [v2:172.21.15.27:6832/3544429684,v1:172.21.15.27:6834/3544429684] conn(0x56529b459e50 0x56529b46e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.324+0000 7f15705b6b80 4 client.9228 mds_command: new command op to 9093 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-11T09:26:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.324+0000 7f15705b6b80 1 -- 172.21.15.27:0/2062791085 --> [v2:172.21.15.27:6832/3544429684,v1:172.21.15.27:6834/3544429684] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56529b41e0e0 con 0x56529b459e50 2024-01-11T09:26:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.324+0000 7f155b7fe700 1 --2- 172.21.15.27:0/2062791085 >> [v2:172.21.15.27:6832/3544429684,v1:172.21.15.27:6834/3544429684] conn(0x56529b459e50 0x56529b46e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.324+0000 7f155b7fe700 1 --2- 172.21.15.27:0/2062791085 >> [v2:172.21.15.27:6832/3544429684,v1:172.21.15.27:6834/3544429684] conn(0x56529b459e50 0x56529b46e3f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22a9d31e6c200786 server_cookie=cc279116c2b1d5e7 in_seq=0 out_seq=0 2024-01-11T09:26:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.324+0000 7f1539ffb700 10 client.9228 ms_handle_connect on v2:172.21.15.27:6832/3544429684 2024-01-11T09:26:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.325+0000 7f1539ffb700 1 -- 172.21.15.27:0/2062791085 <== mds.0 v2:172.21.15.27:6832/3544429684 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x56529b41e0e0 con 0x56529b459e50 2024-01-11T09:26:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.325+0000 7f1539ffb700 10 client.9228 handle_command_reply: tid=0 2024-01-11T09:26:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.325+0000 7f15705b6b80 1 client.9228 shutdown 2024-01-11T09:26:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.325+0000 7f15705b6b80 1 -- 172.21.15.27:0/2062791085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 msgr2=0x56529b47c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.325+0000 7f15705b6b80 1 --2- 172.21.15.27:0/2062791085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 0x56529b47c3a0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f1554017010 tx=0x7f1554021000 comp rx=0 tx=0).stop 2024-01-11T09:26:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.326+0000 7f15705b6b80 1 -- 172.21.15.27:0/2062791085 shutdown_connections 2024-01-11T09:26:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.326+0000 7f15705b6b80 1 --2- 172.21.15.27:0/2062791085 >> [v2:172.21.15.27:6832/3544429684,v1:172.21.15.27:6834/3544429684] conn(0x56529b459e50 0x56529b46e3f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.326+0000 7f15705b6b80 1 --2- 172.21.15.27:0/2062791085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 0x56529b47c3a0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.326+0000 7f15705b6b80 1 --2- 172.21.15.27:0/2062791085 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b4493a0 0x56529b494630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.326+0000 7f15705b6b80 1 --2- 172.21.15.27:0/2062791085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b46db20 0x56529b459910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.326+0000 7f1539ffb700 0 client.9228 ms_handle_reset on v2:172.21.15.27:6832/3544429684 2024-01-11T09:26:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.326+0000 7f15705b6b80 1 -- 172.21.15.27:0/2062791085 >> 172.21.15.27:0/2062791085 conn(0x56529b44b300 msgr2=0x56529b449b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.326+0000 7f15705b6b80 1 -- 172.21.15.27:0/2062791085 shutdown_connections 2024-01-11T09:26:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.326+0000 7f15705b6b80 1 -- 172.21.15.27:0/2062791085 wait complete. 2024-01-11T09:26:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.326+0000 7f15705b6b80 20 client.9228 trim_cache size 0 max 16384 2024-01-11T09:26:48.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.343+0000 7f15705b6b80 1 Processor -- start 2024-01-11T09:26:48.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.343+0000 7f15705b6b80 1 -- start start 2024-01-11T09:26:48.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.344+0000 7f15705b6b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b4493a0 0x56529b44a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.344+0000 7f15705b6b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b46db20 0x56529b4748c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.344+0000 7f15705b6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 0x56529b475830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.344+0000 7f15705b6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529b2341e0 con 0x56529b46def0 2024-01-11T09:26:48.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.344+0000 7f15705b6b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529b4770e0 con 0x56529b4493a0 2024-01-11T09:26:48.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.344+0000 7f15705b6b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x56529b47c380 con 0x56529b46db20 2024-01-11T09:26:48.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.344+0000 7f155bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 0x56529b475830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.344+0000 7f155bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 0x56529b475830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33056/0 (socket says 172.21.15.27:33056) 2024-01-11T09:26:48.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.344+0000 7f155bfff700 1 -- 172.21.15.27:0/2342493701 learned_addr learned my addr 172.21.15.27:0/2342493701 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:48.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.345+0000 7f155b7fe700 1 --2- 172.21.15.27:0/2342493701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b4493a0 0x56529b44a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.345+0000 7f155affd700 1 --2- 172.21.15.27:0/2342493701 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b46db20 0x56529b4748c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.345+0000 7f155bfff700 1 -- 172.21.15.27:0/2342493701 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b46db20 msgr2=0x56529b4748c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:48.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.345+0000 7f155bfff700 1 --2- 172.21.15.27:0/2342493701 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b46db20 0x56529b4748c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.345+0000 7f155bfff700 1 -- 172.21.15.27:0/2342493701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b4493a0 msgr2=0x56529b44a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:48.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.345+0000 7f155bfff700 1 --2- 172.21.15.27:0/2342493701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b4493a0 0x56529b44a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.345+0000 7f155bfff700 1 -- 172.21.15.27:0/2342493701 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56529b2031d0 con 0x56529b46def0 2024-01-11T09:26:48.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.345+0000 7f155b7fe700 1 --2- 172.21.15.27:0/2342493701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b4493a0 0x56529b44a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:48.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.345+0000 7f155bfff700 1 --2- 172.21.15.27:0/2342493701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 0x56529b475830 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f1554007b90 tx=0x7f1554070040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91609ae0f53048b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:48.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.345+0000 7f1541ffb700 1 -- 172.21.15.27:0/2342493701 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155405b070 con 0x56529b46def0 2024-01-11T09:26:48.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.345+0000 7f1541ffb700 1 -- 172.21.15.27:0/2342493701 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155400adf0 con 0x56529b46def0 2024-01-11T09:26:48.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.345+0000 7f1541ffb700 1 -- 172.21.15.27:0/2342493701 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1554055830 con 0x56529b46def0 2024-01-11T09:26:48.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.346+0000 7f15705b6b80 1 -- 172.21.15.27:0/2342493701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 msgr2=0x56529b475830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:48.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.346+0000 7f15705b6b80 1 --2- 172.21.15.27:0/2342493701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 0x56529b475830 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f1554007b90 tx=0x7f1554070040 comp rx=0 tx=0).stop 2024-01-11T09:26:48.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.346+0000 7f15705b6b80 1 -- 172.21.15.27:0/2342493701 shutdown_connections 2024-01-11T09:26:48.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.346+0000 7f15705b6b80 1 --2- 172.21.15.27:0/2342493701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46def0 0x56529b475830 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.346+0000 7f15705b6b80 1 --2- 172.21.15.27:0/2342493701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b4493a0 0x56529b44a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.346+0000 7f15705b6b80 1 --2- 172.21.15.27:0/2342493701 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b46db20 0x56529b4748c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.346+0000 7f15705b6b80 1 -- 172.21.15.27:0/2342493701 >> 172.21.15.27:0/2342493701 conn(0x56529b44b300 msgr2=0x56529b4486f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:48.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.346+0000 7f15705b6b80 1 -- 172.21.15.27:0/2342493701 shutdown_connections 2024-01-11T09:26:48.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.346+0000 7f15705b6b80 1 -- 172.21.15.27:0/2342493701 wait complete. 2024-01-11T09:26:48.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.347+0000 7f15705b6b80 1 -- start start 2024-01-11T09:26:48.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.347+0000 7f15705b6b80 1 Processor -- start 2024-01-11T09:26:48.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.348+0000 7f15705b6b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b4493a0 0x56529b443190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.348+0000 7f15705b6b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46db20 0x56529b4a5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.348+0000 7f15705b6b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b46def0 0x56529b492ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.348+0000 7f15705b6b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529b4936f0 con 0x56529b46db20 2024-01-11T09:26:48.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.348+0000 7f15705b6b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529b4a50e0 con 0x56529b46def0 2024-01-11T09:26:48.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.348+0000 7f15705b6b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x56529b4a5ab0 con 0x56529b4493a0 2024-01-11T09:26:48.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.348+0000 7f15705b6b80 10 client.0 _ll_register_callbacks cb 0x56529b1aba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T09:26:48.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.348+0000 7f155affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46db20 0x56529b4a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.348+0000 7f155b7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b4493a0 0x56529b443190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.348+0000 7f155affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46db20 0x56529b4a5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33060/0 (socket says 172.21.15.27:33060) 2024-01-11T09:26:48.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.348+0000 7f155affd700 1 -- 172.21.15.27:0/274636556 learned_addr learned my addr 172.21.15.27:0/274636556 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:48.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.348+0000 7f155bfff700 1 --2- 172.21.15.27:0/274636556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b46def0 0x56529b492ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.348+0000 7f155affd700 1 -- 172.21.15.27:0/274636556 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b4493a0 msgr2=0x56529b443190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:48.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.348+0000 7f155affd700 1 --2- 172.21.15.27:0/274636556 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b4493a0 0x56529b443190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.348+0000 7f155affd700 1 -- 172.21.15.27:0/274636556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b46def0 msgr2=0x56529b492ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:48.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.348+0000 7f155affd700 1 --2- 172.21.15.27:0/274636556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b46def0 0x56529b492ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.348+0000 7f155affd700 1 -- 172.21.15.27:0/274636556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56529b36b3c0 con 0x56529b46db20 2024-01-11T09:26:48.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.349+0000 7f155affd700 1 --2- 172.21.15.27:0/274636556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46db20 0x56529b4a5570 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f155c06fe90 tx=0x7f1550017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d845e460c815155f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:48.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.349+0000 7f1539ffb700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T09:26:48.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.349+0000 7f1539ffb700 1 -- 172.21.15.27:0/274636556 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1550025070 con 0x56529b46db20 2024-01-11T09:26:48.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.349+0000 7f15705b6b80 1 -- 172.21.15.27:0/274636556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1554058f40 con 0x56529b46db20 2024-01-11T09:26:48.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.349+0000 7f1539ffb700 1 -- 172.21.15.27:0/274636556 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(167..167 src has 1..167) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f155000be60 con 0x56529b46db20 2024-01-11T09:26:48.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.349+0000 7f1539ffb700 1 -- 172.21.15.27:0/274636556 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1550011a50 con 0x56529b46db20 2024-01-11T09:26:48.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.349+0000 7f15705b6b80 10 client.9234 fetch_fsmap learned FSMap version 86 2024-01-11T09:26:48.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.349+0000 7f15705b6b80 1 -- 172.21.15.27:0/274636556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=2+}) v3 -- 0x56529b1dcd60 con 0x56529b46db20 2024-01-11T09:26:48.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.349+0000 7f1539ffb700 1 -- 172.21.15.27:0/274636556 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1554058f40 con 0x56529b46db20 2024-01-11T09:26:48.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.350+0000 7f1539ffb700 1 -- 172.21.15.27:0/274636556 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 86) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f1550011bb0 con 0x56529b46db20 2024-01-11T09:26:48.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.350+0000 7f15705b6b80 10 client.9234 fetch_fsmap finished waiting for FSMap version 86 2024-01-11T09:26:48.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.350+0000 7f15705b6b80 10 client.9234 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-11T09:26:48.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.350+0000 7f15705b6b80 20 client.9234 populate_metadata read hostname 'smithi027' 2024-01-11T09:26:48.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.350+0000 7f15705b6b80 1 --2- 172.21.15.27:0/274636556 >> [v2:172.21.15.27:6832/3544429684,v1:172.21.15.27:6834/3544429684] conn(0x56529b449e30 0x56529b478be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.350+0000 7f15705b6b80 4 client.9234 mds_command: new command op to 9093 tid=0[{"prefix": "status"}] 2024-01-11T09:26:48.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.350+0000 7f15705b6b80 1 -- 172.21.15.27:0/274636556 --> [v2:172.21.15.27:6832/3544429684,v1:172.21.15.27:6834/3544429684] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56529b33f8d0 con 0x56529b449e30 2024-01-11T09:26:48.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.350+0000 7f155b7fe700 1 --2- 172.21.15.27:0/274636556 >> [v2:172.21.15.27:6832/3544429684,v1:172.21.15.27:6834/3544429684] conn(0x56529b449e30 0x56529b478be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.350+0000 7f155b7fe700 1 --2- 172.21.15.27:0/274636556 >> [v2:172.21.15.27:6832/3544429684,v1:172.21.15.27:6834/3544429684] conn(0x56529b449e30 0x56529b478be0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=792eb01feba10dcc server_cookie=7f39b6574f025a6 in_seq=0 out_seq=0 2024-01-11T09:26:48.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.350+0000 7f1539ffb700 10 client.9234 ms_handle_connect on v2:172.21.15.27:6832/3544429684 2024-01-11T09:26:48.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.351+0000 7f1539ffb700 1 -- 172.21.15.27:0/274636556 <== mds.0 v2:172.21.15.27:6832/3544429684 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x56529b33f8d0 con 0x56529b449e30 2024-01-11T09:26:48.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.351+0000 7f1539ffb700 10 client.9234 handle_command_reply: tid=0 2024-01-11T09:26:48.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.351+0000 7f15705b6b80 1 client.9234 shutdown 2024-01-11T09:26:48.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.351+0000 7f15705b6b80 1 -- 172.21.15.27:0/274636556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46db20 msgr2=0x56529b4a5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:48.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.351+0000 7f15705b6b80 1 --2- 172.21.15.27:0/274636556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46db20 0x56529b4a5570 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f155c06fe90 tx=0x7f1550017000 comp rx=0 tx=0).stop 2024-01-11T09:26:48.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.351+0000 7f15705b6b80 1 -- 172.21.15.27:0/274636556 shutdown_connections 2024-01-11T09:26:48.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.351+0000 7f15705b6b80 1 --2- 172.21.15.27:0/274636556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56529b46def0 0x56529b492ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.351+0000 7f15705b6b80 1 --2- 172.21.15.27:0/274636556 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x56529b4493a0 0x56529b443190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.351+0000 7f15705b6b80 1 --2- 172.21.15.27:0/274636556 >> [v2:172.21.15.27:6832/3544429684,v1:172.21.15.27:6834/3544429684] conn(0x56529b449e30 0x56529b478be0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.351+0000 7f15705b6b80 1 --2- 172.21.15.27:0/274636556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56529b46db20 0x56529b4a5570 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.351+0000 7f1539ffb700 0 client.9234 ms_handle_reset on v2:172.21.15.27:6832/3544429684 2024-01-11T09:26:48.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.351+0000 7f15705b6b80 1 -- 172.21.15.27:0/274636556 >> 172.21.15.27:0/274636556 conn(0x56529b44b300 msgr2=0x56529b44d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:48.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.351+0000 7f15705b6b80 1 -- 172.21.15.27:0/274636556 shutdown_connections 2024-01-11T09:26:48.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.351+0000 7f15705b6b80 1 -- 172.21.15.27:0/274636556 wait complete. 2024-01-11T09:26:48.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.351+0000 7f15705b6b80 20 client.9234 trim_cache size 0 max 16384 2024-01-11T09:26:48.357 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:26:48.358 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "8fddb4a7-9d66-40f1-892d-6fe86c728ee4", 2024-01-11T09:26:48.358 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-11T09:26:48.358 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 9093, 2024-01-11T09:26:48.358 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-11T09:26:48.358 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-11T09:26:48.358 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-11T09:26:48.358 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 6.6683111940000002, 2024-01-11T09:26:48.358 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 86, 2024-01-11T09:26:48.358 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 167, 2024-01-11T09:26:48.358 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 167, 2024-01-11T09:26:48.358 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 12.347131201 2024-01-11T09:26:48.358 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:26:48.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.352+0000 7f1561dff700 1 -- 172.21.15.27:0/1947222511 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1548066f30 msgr2=0x7f15480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:48.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.352+0000 7f1561dff700 1 --2- 172.21.15.27:0/1947222511 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1548066f30 0x7f15480693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f154c007fd0 tx=0x7f154c007d50 comp rx=0 tx=0).stop 2024-01-11T09:26:48.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.352+0000 7f1561dff700 1 -- 172.21.15.27:0/1947222511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f155c155e30 msgr2=0x7f155c06e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:48.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.352+0000 7f1561dff700 1 --2- 172.21.15.27:0/1947222511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f155c155e30 0x7f155c06e530 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f155400a380 tx=0x7f1554007900 comp rx=0 tx=0).stop 2024-01-11T09:26:48.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.352+0000 7f1561dff700 1 -- 172.21.15.27:0/1947222511 shutdown_connections 2024-01-11T09:26:48.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.352+0000 7f1561dff700 1 --2- 172.21.15.27:0/1947222511 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1548066f30 0x7f15480693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.352+0000 7f1561dff700 1 --2- 172.21.15.27:0/1947222511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f155c155e30 0x7f155c06e530 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.352+0000 7f1561dff700 1 --2- 172.21.15.27:0/1947222511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f155c0b0190 0x7f155c14deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.352+0000 7f1561dff700 1 --2- 172.21.15.27:0/1947222511 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f155c155a60 0x7f155c14fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.352+0000 7f1561dff700 1 -- 172.21.15.27:0/1947222511 >> 172.21.15.27:0/1947222511 conn(0x7f155c0680b0 msgr2=0x7f155c152980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:48.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.352+0000 7f1561dff700 1 -- 172.21.15.27:0/1947222511 shutdown_connections 2024-01-11T09:26:48.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.352+0000 7f1561dff700 1 -- 172.21.15.27:0/1947222511 wait complete. 2024-01-11T09:26:48.367 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-11T09:26:48.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.569+0000 7f5e561ff700 1 Processor -- start 2024-01-11T09:26:48.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.569+0000 7f5e561ff700 1 -- start start 2024-01-11T09:26:48.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.570+0000 7f5e561ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e500b03e0 0x7f5e500b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.570+0000 7f5e561ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5e500b07b0 0x7f5e500b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.571+0000 7f5e561ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5e50159200 0x7f5e50159640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.571+0000 7f5e561ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e500cd730 con 0x7f5e50159200 2024-01-11T09:26:48.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.571+0000 7f5e561ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e500ccb20 con 0x7f5e500b07b0 2024-01-11T09:26:48.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.571+0000 7f5e561ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e500cce00 con 0x7f5e500b03e0 2024-01-11T09:26:48.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.571+0000 7f5e4ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5e50159200 0x7f5e50159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.571+0000 7f5e4ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5e50159200 0x7f5e50159640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33070/0 (socket says 172.21.15.27:33070) 2024-01-11T09:26:48.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.571+0000 7f5e4ffff700 1 -- 172.21.15.27:0/496807243 learned_addr learned my addr 172.21.15.27:0/496807243 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:48.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.571+0000 7f5e4effd700 1 --2- 172.21.15.27:0/496807243 >> v2:172.21.15.77:3300/0 conn(0x7f5e500b07b0 0x7f5e500b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.571+0000 7f5e4f7fe700 1 --2- 172.21.15.27:0/496807243 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e500b03e0 0x7f5e500b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.571+0000 7f5e4ffff700 1 -- 172.21.15.27:0/496807243 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e500b03e0 msgr2=0x7f5e500b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:48.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.571+0000 7f5e4ffff700 1 --2- 172.21.15.27:0/496807243 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e500b03e0 0x7f5e500b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.571+0000 7f5e4ffff700 1 -- 172.21.15.27:0/496807243 >> v2:172.21.15.77:3300/0 conn(0x7f5e500b07b0 msgr2=0x7f5e500b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:48.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.571+0000 7f5e4ffff700 1 --2- 172.21.15.27:0/496807243 >> v2:172.21.15.77:3300/0 conn(0x7f5e500b07b0 0x7f5e500b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.571+0000 7f5e4ffff700 1 -- 172.21.15.27:0/496807243 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e500cd0e0 con 0x7f5e50159200 2024-01-11T09:26:48.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.571+0000 7f5e4effd700 1 --2- 172.21.15.27:0/496807243 >> v2:172.21.15.77:3300/0 conn(0x7f5e500b07b0 0x7f5e500b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:48.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.572+0000 7f5e4ffff700 1 --2- 172.21.15.27:0/496807243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e50159200 0x7f5e50159640 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f5e500893d0 tx=0x7f5e48009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=348673e7fc31896 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:48.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.572+0000 7f5e4e7fc700 1 -- 172.21.15.27:0/496807243 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4800e070 con 0x7f5e50159200 2024-01-11T09:26:48.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.572+0000 7f5e4e7fc700 1 -- 172.21.15.27:0/496807243 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e4800ba50 con 0x7f5e50159200 2024-01-11T09:26:48.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.572+0000 7f5e4e7fc700 1 -- 172.21.15.27:0/496807243 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e480092e0 con 0x7f5e50159200 2024-01-11T09:26:48.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.573+0000 7f5e561ff700 1 -- 172.21.15.27:0/496807243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e50159200 msgr2=0x7f5e50159640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:48.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.573+0000 7f5e561ff700 1 --2- 172.21.15.27:0/496807243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e50159200 0x7f5e50159640 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f5e500893d0 tx=0x7f5e48009840 comp rx=0 tx=0).stop 2024-01-11T09:26:48.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.573+0000 7f5e561ff700 1 -- 172.21.15.27:0/496807243 shutdown_connections 2024-01-11T09:26:48.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.573+0000 7f5e561ff700 1 --2- 172.21.15.27:0/496807243 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e500b03e0 0x7f5e500b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.573+0000 7f5e561ff700 1 --2- 172.21.15.27:0/496807243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e50159200 0x7f5e50159640 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.573+0000 7f5e561ff700 1 --2- 172.21.15.27:0/496807243 >> v2:172.21.15.77:3300/0 conn(0x7f5e500b07b0 0x7f5e500b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.573+0000 7f5e561ff700 1 -- 172.21.15.27:0/496807243 >> 172.21.15.27:0/496807243 conn(0x7f5e5006ed40 msgr2=0x7f5e5006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:48.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.573+0000 7f5e561ff700 1 -- 172.21.15.27:0/496807243 shutdown_connections 2024-01-11T09:26:48.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.573+0000 7f5e561ff700 1 -- 172.21.15.27:0/496807243 wait complete. 2024-01-11T09:26:48.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.574+0000 7f5e561ff700 1 Processor -- start 2024-01-11T09:26:48.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.574+0000 7f5e561ff700 1 -- start start 2024-01-11T09:26:48.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.575+0000 7f5e561ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e500b03e0 0x7f5e500c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.575+0000 7f5e561ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e500b07b0 0x7f5e5016a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.575+0000 7f5e561ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e50159200 0x7f5e50158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.575+0000 7f5e561ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5016ad60 con 0x7f5e500b07b0 2024-01-11T09:26:48.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.575+0000 7f5e561ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e500c88b0 con 0x7f5e500b03e0 2024-01-11T09:26:48.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.575+0000 7f5e561ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e50158dc0 con 0x7f5e50159200 2024-01-11T09:26:48.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.575+0000 7f5e4effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e500b07b0 0x7f5e5016a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.575+0000 7f5e4effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e500b07b0 0x7f5e5016a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33082/0 (socket says 172.21.15.27:33082) 2024-01-11T09:26:48.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.575+0000 7f5e4effd700 1 -- 172.21.15.27:0/3541254665 learned_addr learned my addr 172.21.15.27:0/3541254665 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:48.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.576+0000 7f5e4f7fe700 1 --2- 172.21.15.27:0/3541254665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e500b03e0 0x7f5e500c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.576+0000 7f5e4effd700 1 -- 172.21.15.27:0/3541254665 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e50159200 msgr2=0x7f5e50158880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:48.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.576+0000 7f5e4ffff700 1 --2- 172.21.15.27:0/3541254665 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e50159200 0x7f5e50158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.576+0000 7f5e4effd700 1 --2- 172.21.15.27:0/3541254665 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e50159200 0x7f5e50158880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.576+0000 7f5e4effd700 1 -- 172.21.15.27:0/3541254665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e500b03e0 msgr2=0x7f5e500c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:48.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.576+0000 7f5e4effd700 1 --2- 172.21.15.27:0/3541254665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e500b03e0 0x7f5e500c75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.576+0000 7f5e4effd700 1 -- 172.21.15.27:0/3541254665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e48009ae0 con 0x7f5e500b07b0 2024-01-11T09:26:48.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.576+0000 7f5e4f7fe700 1 --2- 172.21.15.27:0/3541254665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e500b03e0 0x7f5e500c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:26:48.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.576+0000 7f5e4ffff700 1 --2- 172.21.15.27:0/3541254665 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e50159200 0x7f5e50158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:26:48.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.576+0000 7f5e4effd700 1 --2- 172.21.15.27:0/3541254665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e500b07b0 0x7f5e5016a820 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400e8d0 tx=0x7f5e44008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:48.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.576+0000 7f5e377fe700 1 -- 172.21.15.27:0/3541254665 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4400ee70 con 0x7f5e500b07b0 2024-01-11T09:26:48.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.576+0000 7f5e377fe700 1 -- 172.21.15.27:0/3541254665 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e4400c5e0 con 0x7f5e500b07b0 2024-01-11T09:26:48.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.576+0000 7f5e561ff700 1 -- 172.21.15.27:0/3541254665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e500c7ca0 con 0x7f5e500b07b0 2024-01-11T09:26:48.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.576+0000 7f5e377fe700 1 -- 172.21.15.27:0/3541254665 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4400c980 con 0x7f5e500b07b0 2024-01-11T09:26:48.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.577+0000 7f5e561ff700 1 -- 172.21.15.27:0/3541254665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e500c8260 con 0x7f5e500b07b0 2024-01-11T09:26:48.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.577+0000 7f5e561ff700 1 -- 172.21.15.27:0/3541254665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e500c5c10 con 0x7f5e500b07b0 2024-01-11T09:26:48.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.578+0000 7f5e377fe700 1 -- 172.21.15.27:0/3541254665 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5e44013410 con 0x7f5e500b07b0 2024-01-11T09:26:48.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.579+0000 7f5e377fe700 1 --2- 172.21.15.27:0/3541254665 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5e3c066f30 0x7f5e3c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:48.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.579+0000 7f5e377fe700 1 -- 172.21.15.27:0/3541254665 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f5e44084470 con 0x7f5e500b07b0 2024-01-11T09:26:48.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.580+0000 7f5e4f7fe700 1 --2- 172.21.15.27:0/3541254665 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5e3c066f30 0x7f5e3c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:48.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.580+0000 7f5e4f7fe700 1 --2- 172.21.15.27:0/3541254665 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5e3c066f30 0x7f5e3c0693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e40009990 tx=0x7f5e40008040 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:48.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.584+0000 7f5e377fe700 1 -- 172.21.15.27:0/3541254665 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5e44051600 con 0x7f5e500b07b0 2024-01-11T09:26:48.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.833+0000 7f5e561ff700 1 -- 172.21.15.27:0/3541254665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5e500c8a20 con 0x7f5e500b07b0 2024-01-11T09:26:48.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.834+0000 7f5e377fe700 1 -- 172.21.15.27:0/3541254665 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v86) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5e44054c20 con 0x7f5e500b07b0 2024-01-11T09:26:48.833 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-11T09:26:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.836+0000 7f5e561ff700 1 -- 172.21.15.27:0/3541254665 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5e3c066f30 msgr2=0x7f5e3c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.836+0000 7f5e561ff700 1 --2- 172.21.15.27:0/3541254665 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5e3c066f30 0x7f5e3c0693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e40009990 tx=0x7f5e40008040 comp rx=0 tx=0).stop 2024-01-11T09:26:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.836+0000 7f5e561ff700 1 -- 172.21.15.27:0/3541254665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e500b07b0 msgr2=0x7f5e5016a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.836+0000 7f5e561ff700 1 --2- 172.21.15.27:0/3541254665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e500b07b0 0x7f5e5016a820 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400e8d0 tx=0x7f5e44008040 comp rx=0 tx=0).stop 2024-01-11T09:26:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.836+0000 7f5e561ff700 1 -- 172.21.15.27:0/3541254665 shutdown_connections 2024-01-11T09:26:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.836+0000 7f5e561ff700 1 --2- 172.21.15.27:0/3541254665 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5e3c066f30 0x7f5e3c0693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.836+0000 7f5e561ff700 1 --2- 172.21.15.27:0/3541254665 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e50159200 0x7f5e50158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.836+0000 7f5e561ff700 1 --2- 172.21.15.27:0/3541254665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e500b03e0 0x7f5e500c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.836+0000 7f5e561ff700 1 --2- 172.21.15.27:0/3541254665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e500b07b0 0x7f5e5016a820 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.836+0000 7f5e561ff700 1 -- 172.21.15.27:0/3541254665 >> 172.21.15.27:0/3541254665 conn(0x7f5e5006ed40 msgr2=0x7f5e500b2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.836+0000 7f5e561ff700 1 -- 172.21.15.27:0/3541254665 shutdown_connections 2024-01-11T09:26:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:48.836+0000 7f5e561ff700 1 -- 172.21.15.27:0/3541254665 wait complete. 2024-01-11T09:26:48.852 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-11T09:26:48.853 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-11T09:26:48.853 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-11T09:26:48.853 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-11T09:26:48.853 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-11T09:26:48.853 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-11T09:26:48.853 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-11T09:26:48.853 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-11T09:26:48.853 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T09:26:48.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:26:48.853 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-11T09:26:48.902 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T09:26:48.903 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T09:26:48.903 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T09:26:48.903 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:26:48.903 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-11T09:26:48.903 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:26:48.903 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T09:26:48.903 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-11T09:26:48.903 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-11T09:26:48.903 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T09:26:48.903 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-11T09:26:48.903 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-11T09:26:48.903 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-11T09:26:48.903 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-11T09:26:48.904 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-11T09:26:48.904 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-11T09:26:48.904 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 37913sec preferred_lft 37913sec 2024-01-11T09:26:48.904 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-11T09:26:48.904 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:26:48.904 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T09:26:48.904 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-11T09:26:48.904 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-11T09:26:48.904 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T09:26:48.904 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff 2024-01-11T09:26:48.904 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-11T09:26:48.904 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:26:48.905 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::acd2:b3ff:fe6a:db4b/64 scope link 2024-01-11T09:26:48.905 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:26:48.905 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T09:26:48.905 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:fa:85:1b:3a:bc brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-11T09:26:48.905 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ecfa:85ff:fe1b:3abc/64 scope link 2024-01-11T09:26:48.905 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:26:48.905 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T09:26:48.905 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-11T09:26:48.905 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ec88:faff:fe17:41aa/64 scope link 2024-01-11T09:26:48.905 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:26:48.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:26:48.906 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-11T09:26:48.977 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-11T09:26:48.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:26:48.980 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-11T09:26:49.032 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-11T09:26:49.032 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-11T09:26:49.032 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-11T09:26:49.032 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-11T09:26:49.032 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-11T09:26:49.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:26:49.033 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T09:26:49.033 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T09:26:49.033 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-11T09:26:49.033 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-11T09:26:49.033 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-11T09:26:49.033 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T09:26:49.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:26:49.209 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T09:26:49.257 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-11T09:26:49.257 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-11T09:26:49.258 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T09:26:49.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:26:49.258 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-11T09:26:49.331 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T09:26:49.332 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-11T09:26:49.379 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-11T09:26:49.379 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T09:26:49.380 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-11T09:26:49.426 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-11T09:26:49.427 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-11T09:26:49.427 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:26:49.569 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:26:49.569 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T09:26:49.569 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T09:26:49.584 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:26:49.584 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T09:26:49.584 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T09:26:49.584 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T09:26:49.584 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T09:26:49.605 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:26:49.605 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T09:26:49.605 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T09:26:49.605 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T09:26:49.605 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T09:26:49.605 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790" 2024-01-11T09:26:49.605 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790:/ 2024-01-11T09:26:49.605 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=8fddb4a7-9d66-40f1-892d-6fe86c728ee4" will pass to kernel 2024-01-11T09:26:49.609 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-11T09:26:49.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:26:49.610 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-11T09:26:49.683 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-11T09:26:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.890+0000 7f99e1bff700 1 Processor -- start 2024-01-11T09:26:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.890+0000 7f99e1bff700 1 -- start start 2024-01-11T09:26:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.892+0000 7f99e1bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99dc0b0a40 0x7f99dc15da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.892+0000 7f99e1bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f99dc15df40 0x7f99dc068720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.892+0000 7f99e1bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99dc15d290 con 0x7f99dc06bf10 2024-01-11T09:26:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.892+0000 7f99e1bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99dc15d570 con 0x7f99dc15df40 2024-01-11T09:26:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.892+0000 7f99e1bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc154180 con 0x7f99dc0b0a40 2024-01-11T09:26:49.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.893+0000 7f99dbfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f99dc15df40 0x7f99dc068720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:49.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.893+0000 7f99daffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99dc0b0a40 0x7f99dc15da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:49.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.893+0000 7f99daffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99dc0b0a40 0x7f99dc15da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:56496/0 (socket says 172.21.15.27:56496) 2024-01-11T09:26:49.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.893+0000 7f99dbfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f99dc15df40 0x7f99dc068720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:38418/0 (socket says 172.21.15.27:38418) 2024-01-11T09:26:49.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.893+0000 7f99daffd700 1 -- 172.21.15.27:0/3331934862 learned_addr learned my addr 172.21.15.27:0/3331934862 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:49.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.893+0000 7f99db7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f99dc06bf10 0x7f99dc159c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:44676/0 (socket says 172.21.15.27:44676) 2024-01-11T09:26:49.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.893+0000 7f99da7fc700 1 -- 172.21.15.27:0/3331934862 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107627227 0 0) 0x7f99dc15d290 con 0x7f99dc06bf10 2024-01-11T09:26:49.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.893+0000 7f99daffd700 1 -- 172.21.15.27:0/3331934862 >> v2:172.21.15.77:3300/0 conn(0x7f99dc15df40 msgr2=0x7f99dc068720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.893+0000 7f99daffd700 1 --2- 172.21.15.27:0/3331934862 >> v2:172.21.15.77:3300/0 conn(0x7f99dc15df40 0x7f99dc068720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.893+0000 7f99daffd700 1 -- 172.21.15.27:0/3331934862 >> v1:172.21.15.77:6789/0 conn(0x7f99dc06bf10 legacy=0x7f99dc159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.893+0000 7f99daffd700 1 -- 172.21.15.27:0/3331934862 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99dc15a510 con 0x7f99dc0b0a40 2024-01-11T09:26:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.894+0000 7f99daffd700 1 --2- 172.21.15.27:0/3331934862 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99dc0b0a40 0x7f99dc15da00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f99d00099b0 tx=0x7f99d0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37f645cbb2e84dac server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.894+0000 7f99da7fc700 1 -- 172.21.15.27:0/3331934862 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d0009ea0 con 0x7f99dc0b0a40 2024-01-11T09:26:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.894+0000 7f99da7fc700 1 -- 172.21.15.27:0/3331934862 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d000b860 con 0x7f99dc0b0a40 2024-01-11T09:26:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.894+0000 7f99da7fc700 1 -- 172.21.15.27:0/3331934862 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d000bc00 con 0x7f99dc0b0a40 2024-01-11T09:26:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.894+0000 7f99e1bff700 1 -- 172.21.15.27:0/3331934862 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99dc0b0a40 msgr2=0x7f99dc15da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.894+0000 7f99e1bff700 1 --2- 172.21.15.27:0/3331934862 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99dc0b0a40 0x7f99dc15da00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f99d00099b0 tx=0x7f99d0009250 comp rx=0 tx=0).stop 2024-01-11T09:26:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.894+0000 7f99e1bff700 1 -- 172.21.15.27:0/3331934862 shutdown_connections 2024-01-11T09:26:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.894+0000 7f99e1bff700 1 --2- 172.21.15.27:0/3331934862 >> v2:172.21.15.77:3300/0 conn(0x7f99dc15df40 0x7f99dc068720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.894+0000 7f99e1bff700 1 --2- 172.21.15.27:0/3331934862 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99dc0b0a40 0x7f99dc15da00 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.894+0000 7f99e1bff700 1 -- 172.21.15.27:0/3331934862 >> 172.21.15.27:0/3331934862 conn(0x7f99dc0ce3c0 msgr2=0x7f99dc0ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.894+0000 7f99e1bff700 1 -- 172.21.15.27:0/3331934862 shutdown_connections 2024-01-11T09:26:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.894+0000 7f99e1bff700 1 -- 172.21.15.27:0/3331934862 wait complete. 2024-01-11T09:26:49.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.896+0000 7f99e1bff700 1 Processor -- start 2024-01-11T09:26:49.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.896+0000 7f99e1bff700 1 -- start start 2024-01-11T09:26:49.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.897+0000 7f99e1bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99dc06bf10 0x7f99dc152b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:49.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.897+0000 7f99e1bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99dc0b0a40 0x7f99dc0c3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:49.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.897+0000 7f99e1bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99dc15df40 0x7f99dc1566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:49.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.897+0000 7f99e1bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc0b2690 con 0x7f99dc0b0a40 2024-01-11T09:26:49.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.897+0000 7f99e1bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc0b3210 con 0x7f99dc06bf10 2024-01-11T09:26:49.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.897+0000 7f99e1bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc0b28e0 con 0x7f99dc15df40 2024-01-11T09:26:49.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.897+0000 7f99daffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99dc0b0a40 0x7f99dc0c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:49.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.897+0000 7f99daffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99dc0b0a40 0x7f99dc0c3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33090/0 (socket says 172.21.15.27:33090) 2024-01-11T09:26:49.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.897+0000 7f99daffd700 1 -- 172.21.15.27:0/3436604644 learned_addr learned my addr 172.21.15.27:0/3436604644 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:49.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.897+0000 7f99dbfff700 1 --2- 172.21.15.27:0/3436604644 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99dc15df40 0x7f99dc1566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:49.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.897+0000 7f99db7fe700 1 --2- 172.21.15.27:0/3436604644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99dc06bf10 0x7f99dc152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:49.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.898+0000 7f99daffd700 1 -- 172.21.15.27:0/3436604644 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99dc15df40 msgr2=0x7f99dc1566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:49.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.898+0000 7f99daffd700 1 --2- 172.21.15.27:0/3436604644 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99dc15df40 0x7f99dc1566c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:49.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.898+0000 7f99daffd700 1 -- 172.21.15.27:0/3436604644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99dc06bf10 msgr2=0x7f99dc152b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:49.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.898+0000 7f99daffd700 1 --2- 172.21.15.27:0/3436604644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99dc06bf10 0x7f99dc152b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:49.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.898+0000 7f99daffd700 1 -- 172.21.15.27:0/3436604644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d00094f0 con 0x7f99dc0b0a40 2024-01-11T09:26:49.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.898+0000 7f99daffd700 1 --2- 172.21.15.27:0/3436604644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99dc0b0a40 0x7f99dc0c3180 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f99dc1542c0 tx=0x7f99d0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:49.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.898+0000 7f99c37fe700 1 -- 172.21.15.27:0/3436604644 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d0000d10 con 0x7f99dc0b0a40 2024-01-11T09:26:49.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.898+0000 7f99c37fe700 1 -- 172.21.15.27:0/3436604644 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d000bbe0 con 0x7f99dc0b0a40 2024-01-11T09:26:49.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.898+0000 7f99e1bff700 1 -- 172.21.15.27:0/3436604644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99dc0b2bc0 con 0x7f99dc0b0a40 2024-01-11T09:26:49.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.898+0000 7f99c37fe700 1 -- 172.21.15.27:0/3436604644 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d0011670 con 0x7f99dc0b0a40 2024-01-11T09:26:49.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.899+0000 7f99e1bff700 1 -- 172.21.15.27:0/3436604644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99dc157390 con 0x7f99dc0b0a40 2024-01-11T09:26:49.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.899+0000 7f99e1bff700 1 -- 172.21.15.27:0/3436604644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99dc0b3380 con 0x7f99dc0b0a40 2024-01-11T09:26:49.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.901+0000 7f99c37fe700 1 -- 172.21.15.27:0/3436604644 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f99d0011aa0 con 0x7f99dc0b0a40 2024-01-11T09:26:49.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.902+0000 7f99c37fe700 1 --2- 172.21.15.27:0/3436604644 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f99c8066f30 0x7f99c80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:49.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.902+0000 7f99c37fe700 1 -- 172.21.15.27:0/3436604644 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f99d0085a90 con 0x7f99dc0b0a40 2024-01-11T09:26:49.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.902+0000 7f99db7fe700 1 --2- 172.21.15.27:0/3436604644 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f99c8066f30 0x7f99c80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:49.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.903+0000 7f99db7fe700 1 --2- 172.21.15.27:0/3436604644 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f99c8066f30 0x7f99c80693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f99cc007180 tx=0x7f99cc007ef0 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:49.906+0000 7f99c37fe700 1 -- 172.21.15.27:0/3436604644 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f99d0052ca0 con 0x7f99dc0b0a40 2024-01-11T09:26:50.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.135+0000 7f99e1bff700 1 -- 172.21.15.27:0/3436604644 --> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f99dc04c3c0 con 0x7f99c8066f30 2024-01-11T09:26:50.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.138+0000 7f99c37fe700 1 -- 172.21.15.27:0/3436604644 <== mgr.8542 v2:172.21.15.77:6836/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f99dc04c3c0 con 0x7f99c8066f30 2024-01-11T09:26:50.135 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T09:26:50.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:26:50.136 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-11T09:26:50.136 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:26:50.136 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T09:26:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.140+0000 7f99e1bff700 1 -- 172.21.15.27:0/3436604644 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f99c8066f30 msgr2=0x7f99c80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.140+0000 7f99e1bff700 1 --2- 172.21.15.27:0/3436604644 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f99c8066f30 0x7f99c80693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f99cc007180 tx=0x7f99cc007ef0 comp rx=0 tx=0).stop 2024-01-11T09:26:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.140+0000 7f99e1bff700 1 -- 172.21.15.27:0/3436604644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99dc0b0a40 msgr2=0x7f99dc0c3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.140+0000 7f99e1bff700 1 --2- 172.21.15.27:0/3436604644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99dc0b0a40 0x7f99dc0c3180 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f99dc1542c0 tx=0x7f99d0017000 comp rx=0 tx=0).stop 2024-01-11T09:26:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.140+0000 7f99e1bff700 1 -- 172.21.15.27:0/3436604644 shutdown_connections 2024-01-11T09:26:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.140+0000 7f99e1bff700 1 --2- 172.21.15.27:0/3436604644 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f99c8066f30 0x7f99c80693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.140+0000 7f99e1bff700 1 --2- 172.21.15.27:0/3436604644 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99dc15df40 0x7f99dc1566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.140+0000 7f99e1bff700 1 --2- 172.21.15.27:0/3436604644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99dc06bf10 0x7f99dc152b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.140+0000 7f99e1bff700 1 --2- 172.21.15.27:0/3436604644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99dc0b0a40 0x7f99dc0c3180 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.140+0000 7f99e1bff700 1 -- 172.21.15.27:0/3436604644 >> 172.21.15.27:0/3436604644 conn(0x7f99dc0ce3c0 msgr2=0x7f99dc15b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:50.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.140+0000 7f99e1bff700 1 -- 172.21.15.27:0/3436604644 shutdown_connections 2024-01-11T09:26:50.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.140+0000 7f99e1bff700 1 -- 172.21.15.27:0/3436604644 wait complete. 2024-01-11T09:26:50.155 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-11T09:26:50.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.361+0000 7f76e0dff700 1 Processor -- start 2024-01-11T09:26:50.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.361+0000 7f76e0dff700 1 -- start start 2024-01-11T09:26:50.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.363+0000 7f76e0dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f76dc15a730 0x7f76dc1613a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:50.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.363+0000 7f76e0dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76dc0c6b30 con 0x7f76dc13ff80 2024-01-11T09:26:50.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.363+0000 7f76e0dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76dc0c6800 con 0x7f76dc13fbb0 2024-01-11T09:26:50.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.363+0000 7f76e0dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76dc0cb240 con 0x7f76dc15a730 2024-01-11T09:26:50.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.363+0000 7f76d9d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f76dc13ff80 0x7f76dc15a020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40460/0 (socket says 172.21.15.27:40460) 2024-01-11T09:26:50.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.363+0000 7f76d9d9b700 1 -- 172.21.15.27:0/1606944345 learned_addr learned my addr 172.21.15.27:0/1606944345 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:50.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.363+0000 7f76dad9d700 1 --2- 172.21.15.27:0/1606944345 >> v2:172.21.15.77:3300/0 conn(0x7f76dc15a730 0x7f76dc1613a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:50.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.363+0000 7f76d959a700 1 -- 172.21.15.27:0/1606944345 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468568693 0 0) 0x7f76dc0c6b30 con 0x7f76dc13ff80 2024-01-11T09:26:50.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.364+0000 7f76d959a700 1 -- 172.21.15.27:0/1606944345 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76c4003570 con 0x7f76dc13ff80 2024-01-11T09:26:50.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.364+0000 7f76d959a700 1 -- 172.21.15.27:0/1606944345 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556541007 0 0) 0x7f76dc0c6800 con 0x7f76dc13fbb0 2024-01-11T09:26:50.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.364+0000 7f76d959a700 1 -- 172.21.15.27:0/1606944345 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76dc0c6b30 con 0x7f76dc13fbb0 2024-01-11T09:26:50.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.364+0000 7f76dad9d700 1 -- 172.21.15.27:0/1606944345 >> v1:172.21.15.77:6789/0 conn(0x7f76dc13fbb0 legacy=0x7f76dc0ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:50.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.364+0000 7f76dad9d700 1 -- 172.21.15.27:0/1606944345 >> v1:172.21.15.27:6789/0 conn(0x7f76dc13ff80 legacy=0x7f76dc15a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:50.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.364+0000 7f76dad9d700 1 -- 172.21.15.27:0/1606944345 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76dc0cb520 con 0x7f76dc15a730 2024-01-11T09:26:50.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.364+0000 7f76dad9d700 1 --2- 172.21.15.27:0/1606944345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76dc15a730 0x7f76dc1613a0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f76d4009e60 tx=0x7f76d4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56f484d0c2a6b735 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:50.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.364+0000 7f76d959a700 1 -- 172.21.15.27:0/1606944345 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d400e070 con 0x7f76dc15a730 2024-01-11T09:26:50.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.364+0000 7f76d959a700 1 -- 172.21.15.27:0/1606944345 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d400bab0 con 0x7f76dc15a730 2024-01-11T09:26:50.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.364+0000 7f76d959a700 1 -- 172.21.15.27:0/1606944345 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d400be70 con 0x7f76dc15a730 2024-01-11T09:26:50.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.365+0000 7f76e0dff700 1 -- 172.21.15.27:0/1606944345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76dc15a730 msgr2=0x7f76dc1613a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:50.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.365+0000 7f76e0dff700 1 --2- 172.21.15.27:0/1606944345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76dc15a730 0x7f76dc1613a0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f76d4009e60 tx=0x7f76d4009780 comp rx=0 tx=0).stop 2024-01-11T09:26:50.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.365+0000 7f76e0dff700 1 -- 172.21.15.27:0/1606944345 shutdown_connections 2024-01-11T09:26:50.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.365+0000 7f76e0dff700 1 --2- 172.21.15.27:0/1606944345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76dc15a730 0x7f76dc1613a0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.365+0000 7f76e0dff700 1 -- 172.21.15.27:0/1606944345 >> 172.21.15.27:0/1606944345 conn(0x7f76dc0680a0 msgr2=0x7f76dc0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.365+0000 7f76e0dff700 1 -- 172.21.15.27:0/1606944345 shutdown_connections 2024-01-11T09:26:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.365+0000 7f76e0dff700 1 -- 172.21.15.27:0/1606944345 wait complete. 2024-01-11T09:26:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.366+0000 7f76e0dff700 1 Processor -- start 2024-01-11T09:26:50.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.367+0000 7f76e0dff700 1 -- start start 2024-01-11T09:26:50.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.368+0000 7f76e0dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76dc13fbb0 0x7f76dc16d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:50.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.368+0000 7f76e0dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76dc13ff80 0x7f76dc0aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:50.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.368+0000 7f76e0dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76dc15a730 0x7f76dc0c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:50.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.368+0000 7f76e0dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc0afe90 con 0x7f76dc15a730 2024-01-11T09:26:50.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.368+0000 7f76e0dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc16d480 con 0x7f76dc13fbb0 2024-01-11T09:26:50.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.368+0000 7f76e0dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc16de50 con 0x7f76dc13ff80 2024-01-11T09:26:50.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.368+0000 7f76dad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76dc15a730 0x7f76dc0c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:50.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.368+0000 7f76dad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76dc15a730 0x7f76dc0c7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33100/0 (socket says 172.21.15.27:33100) 2024-01-11T09:26:50.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.368+0000 7f76dad9d700 1 -- 172.21.15.27:0/3860330160 learned_addr learned my addr 172.21.15.27:0/3860330160 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:50.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.368+0000 7f76d9d9b700 1 --2- 172.21.15.27:0/3860330160 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76dc13ff80 0x7f76dc0aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:50.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.368+0000 7f76da59c700 1 --2- 172.21.15.27:0/3860330160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76dc13fbb0 0x7f76dc16d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:50.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.368+0000 7f76dad9d700 1 -- 172.21.15.27:0/3860330160 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76dc13ff80 msgr2=0x7f76dc0aef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:50.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.368+0000 7f76dad9d700 1 --2- 172.21.15.27:0/3860330160 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76dc13ff80 0x7f76dc0aef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:50.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.368+0000 7f76dad9d700 1 -- 172.21.15.27:0/3860330160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76dc13fbb0 msgr2=0x7f76dc16d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:50.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.368+0000 7f76dad9d700 1 --2- 172.21.15.27:0/3860330160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76dc13fbb0 0x7f76dc16d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:50.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.368+0000 7f76dad9d700 1 -- 172.21.15.27:0/3860330160 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d4009a20 con 0x7f76dc15a730 2024-01-11T09:26:50.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.368+0000 7f76d9d9b700 1 --2- 172.21.15.27:0/3860330160 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76dc13ff80 0x7f76dc0aef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:50.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.369+0000 7f76dad9d700 1 --2- 172.21.15.27:0/3860330160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76dc15a730 0x7f76dc0c7c00 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f76d4009460 tx=0x7f76d4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:50.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.369+0000 7f76ca7fc700 1 -- 172.21.15.27:0/3860330160 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d4012e80 con 0x7f76dc15a730 2024-01-11T09:26:50.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.369+0000 7f76ca7fc700 1 -- 172.21.15.27:0/3860330160 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d4026480 con 0x7f76dc15a730 2024-01-11T09:26:50.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.369+0000 7f76e0dff700 1 -- 172.21.15.27:0/3860330160 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76dc0af8d0 con 0x7f76dc15a730 2024-01-11T09:26:50.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.369+0000 7f76ca7fc700 1 -- 172.21.15.27:0/3860330160 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d4026820 con 0x7f76dc15a730 2024-01-11T09:26:50.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.369+0000 7f76e0dff700 1 -- 172.21.15.27:0/3860330160 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76dc16d110 con 0x7f76dc15a730 2024-01-11T09:26:50.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.370+0000 7f76e0dff700 1 -- 172.21.15.27:0/3860330160 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76dc16d5f0 con 0x7f76dc15a730 2024-01-11T09:26:50.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.371+0000 7f76ca7fc700 1 -- 172.21.15.27:0/3860330160 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 90) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f76d401c400 con 0x7f76dc15a730 2024-01-11T09:26:50.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.372+0000 7f76ca7fc700 1 --2- 172.21.15.27:0/3860330160 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f76bc066f30 0x7f76bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:50.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.372+0000 7f76ca7fc700 1 -- 172.21.15.27:0/3860330160 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f76d4086600 con 0x7f76dc15a730 2024-01-11T09:26:50.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.373+0000 7f76da59c700 1 --2- 172.21.15.27:0/3860330160 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f76bc066f30 0x7f76bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:50.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.373+0000 7f76da59c700 1 --2- 172.21.15.27:0/3860330160 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f76bc066f30 0x7f76bc0693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f76cc003ae0 tx=0x7f76cc007d40 comp rx=0 tx=0).ready entity=mgr.8542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:50.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.376+0000 7f76ca7fc700 1 -- 172.21.15.27:0/3860330160 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f76d4053790 con 0x7f76dc15a730 2024-01-11T09:26:50.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:50.632+0000 7f76e0dff700 1 -- 172.21.15.27:0/3860330160 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f76dc04c3c0 con 0x7f76dc15a730 2024-01-11T09:26:51.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.631+0000 7f76ca7fc700 1 -- 172.21.15.27:0/3860330160 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 91) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f76d4053b60 con 0x7f76dc15a730 2024-01-11T09:26:51.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.634+0000 7f76ca7fc700 1 -- 172.21.15.27:0/3860330160 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v91) v1 ==== 92+0+0 (secure 0 0 0) 0x7f76d4069ca0 con 0x7f76dc15a730 2024-01-11T09:26:51.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.637+0000 7f76e0dff700 1 -- 172.21.15.27:0/3860330160 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f76bc066f30 msgr2=0x7f76bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:51.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.637+0000 7f76e0dff700 1 --2- 172.21.15.27:0/3860330160 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f76bc066f30 0x7f76bc0693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f76cc003ae0 tx=0x7f76cc007d40 comp rx=0 tx=0).stop 2024-01-11T09:26:51.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.637+0000 7f76e0dff700 1 -- 172.21.15.27:0/3860330160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76dc15a730 msgr2=0x7f76dc0c7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:51.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.637+0000 7f76e0dff700 1 --2- 172.21.15.27:0/3860330160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76dc15a730 0x7f76dc0c7c00 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f76d4009460 tx=0x7f76d4019000 comp rx=0 tx=0).stop 2024-01-11T09:26:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.638+0000 7f76e0dff700 1 -- 172.21.15.27:0/3860330160 shutdown_connections 2024-01-11T09:26:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.638+0000 7f76e0dff700 1 --2- 172.21.15.27:0/3860330160 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f76bc066f30 0x7f76bc0693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.638+0000 7f76e0dff700 1 --2- 172.21.15.27:0/3860330160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76dc15a730 0x7f76dc0c7c00 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.638+0000 7f76e0dff700 1 --2- 172.21.15.27:0/3860330160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76dc13fbb0 0x7f76dc16d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.638+0000 7f76e0dff700 1 --2- 172.21.15.27:0/3860330160 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f76dc13ff80 0x7f76dc0aef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.638+0000 7f76e0dff700 1 -- 172.21.15.27:0/3860330160 >> 172.21.15.27:0/3860330160 conn(0x7f76dc0680a0 msgr2=0x7f76dc0c2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.638+0000 7f76e0dff700 1 -- 172.21.15.27:0/3860330160 shutdown_connections 2024-01-11T09:26:51.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.638+0000 7f76e0dff700 1 -- 172.21.15.27:0/3860330160 wait complete. 2024-01-11T09:26:51.658 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-11T09:26:51.846 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:51.849+0000 7f2f29b9c1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:26:51.854 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:51.856+0000 7f1fd9efe1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:26:51.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.874+0000 7fe1495ff700 1 Processor -- start 2024-01-11T09:26:51.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.874+0000 7fe1495ff700 1 -- start start 2024-01-11T09:26:51.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.876+0000 7fe1495ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe144142dc0 0x7fe1440cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:51.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.876+0000 7fe1495ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe1440cb2e0 0x7fe1440cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:51.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.876+0000 7fe1495ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1440ccc70 con 0x7fe144143190 2024-01-11T09:26:51.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.876+0000 7fe1495ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe14415df30 con 0x7fe144142dc0 2024-01-11T09:26:51.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.876+0000 7fe1495ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe14415d320 con 0x7fe1440cb2e0 2024-01-11T09:26:51.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.877+0000 7fe142ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe144142dc0 0x7fe1440cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:51.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.877+0000 7fe142ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe144142dc0 0x7fe1440cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33132/0 (socket says 172.21.15.27:33132) 2024-01-11T09:26:51.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.877+0000 7fe142ffd700 1 -- 172.21.15.27:0/2124279830 learned_addr learned my addr 172.21.15.27:0/2124279830 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:51.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.877+0000 7fe1437fe700 1 --2- 172.21.15.27:0/2124279830 >> v2:172.21.15.77:3300/0 conn(0x7fe1440cb2e0 0x7fe1440cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:51.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.877+0000 7fe142ffd700 1 -- 172.21.15.27:0/2124279830 >> v2:172.21.15.77:3300/0 conn(0x7fe1440cb2e0 msgr2=0x7fe1440cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:51.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.877+0000 7fe142ffd700 1 --2- 172.21.15.27:0/2124279830 >> v2:172.21.15.77:3300/0 conn(0x7fe1440cb2e0 0x7fe1440cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:51.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.877+0000 7fe142ffd700 1 -- 172.21.15.27:0/2124279830 >> v1:172.21.15.77:6789/0 conn(0x7fe144143190 legacy=0x7fe1440cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:51.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.877+0000 7fe142ffd700 1 -- 172.21.15.27:0/2124279830 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1440cb700 con 0x7fe144142dc0 2024-01-11T09:26:51.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.878+0000 7fe142ffd700 1 --2- 172.21.15.27:0/2124279830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144142dc0 0x7fe1440cada0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fe1441577b0 tx=0x7fe134009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f538aed2e3ec779 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:51.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.878+0000 7fe141ffb700 1 -- 172.21.15.27:0/2124279830 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe134009e70 con 0x7fe144142dc0 2024-01-11T09:26:51.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.878+0000 7fe141ffb700 1 -- 172.21.15.27:0/2124279830 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe13400b860 con 0x7fe144142dc0 2024-01-11T09:26:51.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.878+0000 7fe141ffb700 1 -- 172.21.15.27:0/2124279830 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13400bc00 con 0x7fe144142dc0 2024-01-11T09:26:51.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.879+0000 7fe1495ff700 1 -- 172.21.15.27:0/2124279830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144142dc0 msgr2=0x7fe1440cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:51.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.879+0000 7fe1495ff700 1 --2- 172.21.15.27:0/2124279830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144142dc0 0x7fe1440cada0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fe1441577b0 tx=0x7fe134009250 comp rx=0 tx=0).stop 2024-01-11T09:26:51.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.879+0000 7fe1495ff700 1 -- 172.21.15.27:0/2124279830 shutdown_connections 2024-01-11T09:26:51.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.879+0000 7fe1495ff700 1 --2- 172.21.15.27:0/2124279830 >> v2:172.21.15.77:3300/0 conn(0x7fe1440cb2e0 0x7fe1440cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:51.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.879+0000 7fe1495ff700 1 --2- 172.21.15.27:0/2124279830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144142dc0 0x7fe1440cada0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:51.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.879+0000 7fe1495ff700 1 -- 172.21.15.27:0/2124279830 >> 172.21.15.27:0/2124279830 conn(0x7fe144063ca0 msgr2=0x7fe14406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:51.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.879+0000 7fe1495ff700 1 -- 172.21.15.27:0/2124279830 shutdown_connections 2024-01-11T09:26:51.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.879+0000 7fe1495ff700 1 -- 172.21.15.27:0/2124279830 wait complete. 2024-01-11T09:26:51.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.880+0000 7fe1495ff700 1 Processor -- start 2024-01-11T09:26:51.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.881+0000 7fe1495ff700 1 -- start start 2024-01-11T09:26:51.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.881+0000 7fe1495ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1440cb2e0 0x7fe14413e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:51.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.882+0000 7fe1495ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe144142dc0 0x7fe144140610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:51.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.882+0000 7fe1495ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144143190 0x7fe144140b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:51.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.882+0000 7fe1495ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14415a560 con 0x7fe144143190 2024-01-11T09:26:51.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.882+0000 7fe1495ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1440b3c10 con 0x7fe144142dc0 2024-01-11T09:26:51.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.882+0000 7fe1495ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1440b3ef0 con 0x7fe1440cb2e0 2024-01-11T09:26:51.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.882+0000 7fe1437fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144143190 0x7fe144140b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:51.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.882+0000 7fe142ffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1440cb2e0 0x7fe14413e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:51.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.882+0000 7fe1427fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe144142dc0 0x7fe144140610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:51.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.882+0000 7fe1437fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144143190 0x7fe144140b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33140/0 (socket says 172.21.15.27:33140) 2024-01-11T09:26:51.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.882+0000 7fe142ffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1440cb2e0 0x7fe14413e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:56532/0 (socket says 172.21.15.27:56532) 2024-01-11T09:26:51.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.882+0000 7fe1437fe700 1 -- 172.21.15.27:0/1754565131 learned_addr learned my addr 172.21.15.27:0/1754565131 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:51.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.882+0000 7fe1437fe700 1 -- 172.21.15.27:0/1754565131 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1440cb2e0 msgr2=0x7fe14413e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:51.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.882+0000 7fe1437fe700 1 --2- 172.21.15.27:0/1754565131 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1440cb2e0 0x7fe14413e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:51.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.882+0000 7fe1437fe700 1 -- 172.21.15.27:0/1754565131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe144142dc0 msgr2=0x7fe144140610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:51.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.882+0000 7fe1437fe700 1 --2- 172.21.15.27:0/1754565131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe144142dc0 0x7fe144140610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:51.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.882+0000 7fe1437fe700 1 -- 172.21.15.27:0/1754565131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1340094f0 con 0x7fe144143190 2024-01-11T09:26:51.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.882+0000 7fe1427fc700 1 --2- 172.21.15.27:0/1754565131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe144142dc0 0x7fe144140610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:26:51.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.883+0000 7fe1437fe700 1 --2- 172.21.15.27:0/1754565131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144143190 0x7fe144140b50 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fe13c009e90 tx=0x7fe13c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:51.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.883+0000 7fe12affd700 1 -- 172.21.15.27:0/1754565131 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13c00cb70 con 0x7fe144143190 2024-01-11T09:26:51.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.883+0000 7fe12affd700 1 -- 172.21.15.27:0/1754565131 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe13c00f040 con 0x7fe144143190 2024-01-11T09:26:51.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.883+0000 7fe1495ff700 1 -- 172.21.15.27:0/1754565131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1440b3920 con 0x7fe144143190 2024-01-11T09:26:51.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.883+0000 7fe12affd700 1 -- 172.21.15.27:0/1754565131 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13c014480 con 0x7fe144143190 2024-01-11T09:26:51.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.883+0000 7fe1495ff700 1 -- 172.21.15.27:0/1754565131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe14413e7a0 con 0x7fe144143190 2024-01-11T09:26:51.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.885+0000 7fe1495ff700 1 -- 172.21.15.27:0/1754565131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe14404c3c0 con 0x7fe144143190 2024-01-11T09:26:51.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.886+0000 7fe12affd700 1 -- 172.21.15.27:0/1754565131 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 91) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fe13c014830 con 0x7fe144143190 2024-01-11T09:26:51.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.887+0000 7fe12affd700 1 --2- 172.21.15.27:0/1754565131 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe130066f80 0x7fe130069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:51.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.887+0000 7fe12affd700 1 -- 172.21.15.27:0/1754565131 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fe13c084fe0 con 0x7fe144143190 2024-01-11T09:26:51.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.887+0000 7fe142ffd700 1 -- 172.21.15.27:0/1754565131 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe130066f80 msgr2=0x7fe130069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:26:51.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.887+0000 7fe142ffd700 1 --2- 172.21.15.27:0/1754565131 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe130066f80 0x7fe130069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:26:51.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:51.891+0000 7fe12affd700 1 -- 172.21.15.27:0/1754565131 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe13c0521d0 con 0x7fe144143190 2024-01-11T09:26:52.025 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:52.028+0000 7f2f29b9c1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:26:52.057 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:52.060+0000 7f1fd9efe1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:26:52.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.088+0000 7fe142ffd700 1 -- 172.21.15.27:0/1754565131 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe130066f80 msgr2=0x7fe130069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:26:52.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.088+0000 7fe142ffd700 1 --2- 172.21.15.27:0/1754565131 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe130066f80 0x7fe130069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:26:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.138+0000 7fe1495ff700 1 -- 172.21.15.27:0/1754565131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fe14415d590 con 0x7fe144143190 2024-01-11T09:26:52.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.142+0000 7fe12affd700 1 -- 172.21.15.27:0/1754565131 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v59)=0 v59) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe13c0557f0 con 0x7fe144143190 2024-01-11T09:26:52.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.145+0000 7fe1495ff700 1 -- 172.21.15.27:0/1754565131 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe130066f80 msgr2=0x7fe130069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:26:52.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.145+0000 7fe1495ff700 1 --2- 172.21.15.27:0/1754565131 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe130066f80 0x7fe130069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:52.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.145+0000 7fe1495ff700 1 -- 172.21.15.27:0/1754565131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144143190 msgr2=0x7fe144140b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:52.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.145+0000 7fe1495ff700 1 --2- 172.21.15.27:0/1754565131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144143190 0x7fe144140b50 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fe13c009e90 tx=0x7fe13c00b040 comp rx=0 tx=0).stop 2024-01-11T09:26:52.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.145+0000 7fe1495ff700 1 -- 172.21.15.27:0/1754565131 shutdown_connections 2024-01-11T09:26:52.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.145+0000 7fe1495ff700 1 --2- 172.21.15.27:0/1754565131 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe130066f80 0x7fe130069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:52.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.145+0000 7fe1495ff700 1 --2- 172.21.15.27:0/1754565131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe144143190 0x7fe144140b50 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:52.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.145+0000 7fe1495ff700 1 --2- 172.21.15.27:0/1754565131 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1440cb2e0 0x7fe14413e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:52.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.145+0000 7fe1495ff700 1 --2- 172.21.15.27:0/1754565131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe144142dc0 0x7fe144140610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:52.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.145+0000 7fe1495ff700 1 -- 172.21.15.27:0/1754565131 >> 172.21.15.27:0/1754565131 conn(0x7fe144063ca0 msgr2=0x7fe14415f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:52.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.145+0000 7fe1495ff700 1 -- 172.21.15.27:0/1754565131 shutdown_connections 2024-01-11T09:26:52.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.145+0000 7fe1495ff700 1 -- 172.21.15.27:0/1754565131 wait complete. 2024-01-11T09:26:52.162 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-11T09:26:52.341 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:52.344+0000 7f2f29b9c1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:26:52.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.371+0000 7f40519ff700 1 Processor -- start 2024-01-11T09:26:52.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.372+0000 7f40519ff700 1 -- start start 2024-01-11T09:26:52.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.373+0000 7f40519ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f404c0c8720 0x7f404c1591a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:52.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.373+0000 7f40519ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404c0cd0a0 con 0x7f404c069b60 2024-01-11T09:26:52.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.373+0000 7f40519ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404c0c7680 con 0x7f404c069790 2024-01-11T09:26:52.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.373+0000 7f40519ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f404c0c7960 con 0x7f404c0c8720 2024-01-11T09:26:52.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.373+0000 7f404a7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f404c0c8720 0x7f404c1591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:52.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.373+0000 7f404b7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f404c069b60 0x7f404c1596e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40464/0 (socket says 172.21.15.27:40464) 2024-01-11T09:26:52.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.373+0000 7f404b7fe700 1 -- 172.21.15.27:0/3572149970 learned_addr learned my addr 172.21.15.27:0/3572149970 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:52.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.374+0000 7f404a7fc700 1 -- 172.21.15.27:0/3572149970 >> v1:172.21.15.77:6789/0 conn(0x7f404c069790 legacy=0x7f404c0c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:52.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.374+0000 7f404a7fc700 1 -- 172.21.15.27:0/3572149970 >> v1:172.21.15.27:6789/0 conn(0x7f404c069b60 legacy=0x7f404c1596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:52.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.374+0000 7f404a7fc700 1 -- 172.21.15.27:0/3572149970 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404c0c7c40 con 0x7f404c0c8720 2024-01-11T09:26:52.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.374+0000 7f4049ffb700 1 -- 172.21.15.27:0/3572149970 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2104796436 0 0) 0x7f404c0cd0a0 con 0x7f404c069b60 2024-01-11T09:26:52.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.374+0000 7f404a7fc700 1 --2- 172.21.15.27:0/3572149970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f404c0c8720 0x7f404c1591a0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f40400098e0 tx=0x7f40400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb504684994df34f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:52.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.375+0000 7f4049ffb700 1 -- 172.21.15.27:0/3572149970 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4040009e80 con 0x7f404c0c8720 2024-01-11T09:26:52.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.375+0000 7f4049ffb700 1 -- 172.21.15.27:0/3572149970 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404000b8f0 con 0x7f404c0c8720 2024-01-11T09:26:52.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.375+0000 7f4049ffb700 1 -- 172.21.15.27:0/3572149970 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404000bc70 con 0x7f404c0c8720 2024-01-11T09:26:52.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.375+0000 7f40519ff700 1 -- 172.21.15.27:0/3572149970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f404c0c8720 msgr2=0x7f404c1591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:52.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.375+0000 7f40519ff700 1 --2- 172.21.15.27:0/3572149970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f404c0c8720 0x7f404c1591a0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f40400098e0 tx=0x7f40400091a0 comp rx=0 tx=0).stop 2024-01-11T09:26:52.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.376+0000 7f40519ff700 1 -- 172.21.15.27:0/3572149970 shutdown_connections 2024-01-11T09:26:52.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.376+0000 7f40519ff700 1 --2- 172.21.15.27:0/3572149970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f404c0c8720 0x7f404c1591a0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:52.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.376+0000 7f40519ff700 1 -- 172.21.15.27:0/3572149970 >> 172.21.15.27:0/3572149970 conn(0x7f404c0680a0 msgr2=0x7f404c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:52.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.376+0000 7f40519ff700 1 -- 172.21.15.27:0/3572149970 shutdown_connections 2024-01-11T09:26:52.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.376+0000 7f40519ff700 1 -- 172.21.15.27:0/3572149970 wait complete. 2024-01-11T09:26:52.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.377+0000 7f40519ff700 1 Processor -- start 2024-01-11T09:26:52.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.377+0000 7f40519ff700 1 -- start start 2024-01-11T09:26:52.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.378+0000 7f40519ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f404c069790 0x7f404c0c2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:52.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.378+0000 7f40519ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f404c069b60 0x7f404c154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:52.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.378+0000 7f40519ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f404c0c8720 0x7f404c0aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:52.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.378+0000 7f40519ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404c15cdd0 con 0x7f404c069b60 2024-01-11T09:26:52.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.378+0000 7f40519ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404c155750 con 0x7f404c0c8720 2024-01-11T09:26:52.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.378+0000 7f40519ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f404c153da0 con 0x7f404c069790 2024-01-11T09:26:52.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.379+0000 7f404a7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f404c069b60 0x7f404c154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:52.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.379+0000 7f404a7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f404c069b60 0x7f404c154230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33152/0 (socket says 172.21.15.27:33152) 2024-01-11T09:26:52.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.379+0000 7f404a7fc700 1 -- 172.21.15.27:0/3648939230 learned_addr learned my addr 172.21.15.27:0/3648939230 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:52.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.379+0000 7f404affd700 1 --2- 172.21.15.27:0/3648939230 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f404c069790 0x7f404c0c2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:52.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.379+0000 7f404b7fe700 1 --2- 172.21.15.27:0/3648939230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f404c0c8720 0x7f404c0aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:52.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.379+0000 7f404a7fc700 1 -- 172.21.15.27:0/3648939230 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f404c069790 msgr2=0x7f404c0c2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:52.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.379+0000 7f404a7fc700 1 --2- 172.21.15.27:0/3648939230 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f404c069790 0x7f404c0c2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:52.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.379+0000 7f404a7fc700 1 -- 172.21.15.27:0/3648939230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f404c0c8720 msgr2=0x7f404c0aef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:52.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.379+0000 7f404a7fc700 1 --2- 172.21.15.27:0/3648939230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f404c0c8720 0x7f404c0aef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:52.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.379+0000 7f404a7fc700 1 -- 172.21.15.27:0/3648939230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4040009420 con 0x7f404c069b60 2024-01-11T09:26:52.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.380+0000 7f404a7fc700 1 --2- 172.21.15.27:0/3648939230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f404c069b60 0x7f404c154230 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f4040013000 tx=0x7f4040017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:52.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.380+0000 7f4032ffd700 1 -- 172.21.15.27:0/3648939230 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404001f070 con 0x7f404c069b60 2024-01-11T09:26:52.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.380+0000 7f4032ffd700 1 -- 172.21.15.27:0/3648939230 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4040000d10 con 0x7f404c069b60 2024-01-11T09:26:52.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.380+0000 7f4032ffd700 1 -- 172.21.15.27:0/3648939230 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4040025420 con 0x7f404c069b60 2024-01-11T09:26:52.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.380+0000 7f40519ff700 1 -- 172.21.15.27:0/3648939230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f404c153470 con 0x7f404c069b60 2024-01-11T09:26:52.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.380+0000 7f40519ff700 1 -- 172.21.15.27:0/3648939230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f404c153a30 con 0x7f404c069b60 2024-01-11T09:26:52.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.381+0000 7f40519ff700 1 -- 172.21.15.27:0/3648939230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f404c0c1ac0 con 0x7f404c069b60 2024-01-11T09:26:52.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.382+0000 7f4032ffd700 1 -- 172.21.15.27:0/3648939230 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 91) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f4040023070 con 0x7f404c069b60 2024-01-11T09:26:52.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.383+0000 7f4032ffd700 1 --2- 172.21.15.27:0/3648939230 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4038066fb0 0x7f4038069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:52.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.383+0000 7f4032ffd700 1 -- 172.21.15.27:0/3648939230 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f4040085910 con 0x7f404c069b60 2024-01-11T09:26:52.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.383+0000 7f404affd700 1 -- 172.21.15.27:0/3648939230 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4038066fb0 msgr2=0x7f4038069460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:26:52.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.383+0000 7f404affd700 1 --2- 172.21.15.27:0/3648939230 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4038066fb0 0x7f4038069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:26:52.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.387+0000 7f4032ffd700 1 -- 172.21.15.27:0/3648939230 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4040052aa0 con 0x7f404c069b60 2024-01-11T09:26:52.400 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:52.402+0000 7f1fd9efe1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:26:52.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.584+0000 7f404affd700 1 -- 172.21.15.27:0/3648939230 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4038066fb0 msgr2=0x7f4038069460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:26:52.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.584+0000 7f404affd700 1 --2- 172.21.15.27:0/3648939230 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4038066fb0 0x7f4038069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:26:52.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.596+0000 7f40519ff700 1 -- 172.21.15.27:0/3648939230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f404c04c3c0 con 0x7f404c069b60 2024-01-11T09:26:52.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.600+0000 7f4032ffd700 1 -- 172.21.15.27:0/3648939230 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v60)=0 v60) v1 ==== 133+0+0 (secure 0 0 0) 0x7f404002a0c0 con 0x7f404c069b60 2024-01-11T09:26:52.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.603+0000 7f40519ff700 1 -- 172.21.15.27:0/3648939230 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4038066fb0 msgr2=0x7f4038069460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:26:52.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.603+0000 7f40519ff700 1 --2- 172.21.15.27:0/3648939230 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4038066fb0 0x7f4038069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:52.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.603+0000 7f40519ff700 1 -- 172.21.15.27:0/3648939230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f404c069b60 msgr2=0x7f404c154230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:52.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.603+0000 7f40519ff700 1 --2- 172.21.15.27:0/3648939230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f404c069b60 0x7f404c154230 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f4040013000 tx=0x7f4040017000 comp rx=0 tx=0).stop 2024-01-11T09:26:52.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.603+0000 7f40519ff700 1 -- 172.21.15.27:0/3648939230 shutdown_connections 2024-01-11T09:26:52.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.603+0000 7f40519ff700 1 --2- 172.21.15.27:0/3648939230 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4038066fb0 0x7f4038069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:52.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.603+0000 7f40519ff700 1 --2- 172.21.15.27:0/3648939230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f404c0c8720 0x7f404c0aef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:52.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.603+0000 7f40519ff700 1 --2- 172.21.15.27:0/3648939230 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f404c069790 0x7f404c0c2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:52.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.603+0000 7f40519ff700 1 --2- 172.21.15.27:0/3648939230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f404c069b60 0x7f404c154230 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:52.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.603+0000 7f40519ff700 1 -- 172.21.15.27:0/3648939230 >> 172.21.15.27:0/3648939230 conn(0x7f404c0680a0 msgr2=0x7f404c152720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:52.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.603+0000 7f40519ff700 1 -- 172.21.15.27:0/3648939230 shutdown_connections 2024-01-11T09:26:52.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.603+0000 7f40519ff700 1 -- 172.21.15.27:0/3648939230 wait complete. 2024-01-11T09:26:52.619 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:52.619 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-01-11T09:26:52.673 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1h 2024-01-11T09:26:52.871 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:52.874+0000 7f2f29b9c1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:26:52.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.882+0000 7f27fedff700 1 Processor -- start 2024-01-11T09:26:52.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.882+0000 7f27fedff700 1 -- start start 2024-01-11T09:26:52.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.883+0000 7f27fedff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f27f8157410 0x7f27f80ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.883+0000 7f27fedff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27f8069540 0x7f27f80c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.884+0000 7f27fedff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f80699d0 con 0x7f27f81577e0 2024-01-11T09:26:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.884+0000 7f27fedff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f8069cb0 con 0x7f27f8157410 2024-01-11T09:26:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.884+0000 7f27fedff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f8069f90 con 0x7f27f8069540 2024-01-11T09:26:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.884+0000 7f27f7fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f27f81577e0 0x7f27f80ae000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40478/0 (socket says 172.21.15.27:40478) 2024-01-11T09:26:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.884+0000 7f27f7fff700 1 -- 172.21.15.27:0/2718428668 learned_addr learned my addr 172.21.15.27:0/2718428668 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:26:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.884+0000 7f27fcb9b700 1 --2- 172.21.15.27:0/2718428668 >> v2:172.21.15.77:3300/0 conn(0x7f27f8157410 0x7f27f80ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.884+0000 7f27fd39c700 1 --2- 172.21.15.27:0/2718428668 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27f8069540 0x7f27f80c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.884+0000 7f27f77fe700 1 -- 172.21.15.27:0/2718428668 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411265280 0 0) 0x7f27f80699d0 con 0x7f27f81577e0 2024-01-11T09:26:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.884+0000 7f27f77fe700 1 -- 172.21.15.27:0/2718428668 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27e8003570 con 0x7f27f81577e0 2024-01-11T09:26:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.884+0000 7f27fcb9b700 1 -- 172.21.15.27:0/2718428668 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27f8069540 msgr2=0x7f27f80c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:52.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.884+0000 7f27fcb9b700 1 --2- 172.21.15.27:0/2718428668 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27f8069540 0x7f27f80c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.885+0000 7f27fcb9b700 1 -- 172.21.15.27:0/2718428668 >> v1:172.21.15.27:6789/0 conn(0x7f27f81577e0 legacy=0x7f27f80ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.885+0000 7f27fcb9b700 1 -- 172.21.15.27:0/2718428668 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f806a270 con 0x7f27f8157410 2024-01-11T09:26:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.885+0000 7f27fd39c700 1 --2- 172.21.15.27:0/2718428668 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27f8069540 0x7f27f80c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:26:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.885+0000 7f27fcb9b700 1 --2- 172.21.15.27:0/2718428668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27f8157410 0x7f27f80ccae0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f27ec0098d0 tx=0x7f27ec009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adaa8da1f9c99628 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.885+0000 7f27f77fe700 1 -- 172.21.15.27:0/2718428668 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec005c10 con 0x7f27f8157410 2024-01-11T09:26:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.885+0000 7f27f77fe700 1 -- 172.21.15.27:0/2718428668 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ec005d70 con 0x7f27f8157410 2024-01-11T09:26:52.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.885+0000 7f27f77fe700 1 -- 172.21.15.27:0/2718428668 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec006260 con 0x7f27f8157410 2024-01-11T09:26:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.886+0000 7f27fedff700 1 -- 172.21.15.27:0/2718428668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27f8157410 msgr2=0x7f27f80ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.886+0000 7f27fedff700 1 --2- 172.21.15.27:0/2718428668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27f8157410 0x7f27f80ccae0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f27ec0098d0 tx=0x7f27ec009120 comp rx=0 tx=0).stop 2024-01-11T09:26:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.886+0000 7f27fedff700 1 -- 172.21.15.27:0/2718428668 shutdown_connections 2024-01-11T09:26:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.886+0000 7f27fedff700 1 --2- 172.21.15.27:0/2718428668 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27f8069540 0x7f27f80c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.886+0000 7f27fedff700 1 --2- 172.21.15.27:0/2718428668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27f8157410 0x7f27f80ccae0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.886+0000 7f27fedff700 1 -- 172.21.15.27:0/2718428668 >> 172.21.15.27:0/2718428668 conn(0x7f27f80680a0 msgr2=0x7f27f80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.886+0000 7f27fedff700 1 -- 172.21.15.27:0/2718428668 shutdown_connections 2024-01-11T09:26:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.886+0000 7f27fedff700 1 -- 172.21.15.27:0/2718428668 wait complete. 2024-01-11T09:26:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.887+0000 7f27fedff700 1 Processor -- start 2024-01-11T09:26:52.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.888+0000 7f27fedff700 1 -- start start 2024-01-11T09:26:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.888+0000 7f27fedff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27f8069540 0x7f27f80cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.889+0000 7f27fedff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27f8157410 0x7f27f8154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.889+0000 7f27fedff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27f81577e0 0x7f27f8169aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:52.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.889+0000 7f27fedff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f8154690 con 0x7f27f8157410 2024-01-11T09:26:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.889+0000 7f27fedff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f8155e70 con 0x7f27f81577e0 2024-01-11T09:26:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.889+0000 7f27fedff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f816a2f0 con 0x7f27f8069540 2024-01-11T09:26:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.889+0000 7f27f7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27f8157410 0x7f27f8154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.889+0000 7f27f7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27f8157410 0x7f27f8154150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33164/0 (socket says 172.21.15.27:33164) 2024-01-11T09:26:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.889+0000 7f27fcb9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27f8069540 0x7f27f80cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.889+0000 7f27f7fff700 1 -- 172.21.15.27:0/460152457 learned_addr learned my addr 172.21.15.27:0/460152457 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.889+0000 7f27fd39c700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27f81577e0 0x7f27f8169aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.889+0000 7f27f7fff700 1 -- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27f8069540 msgr2=0x7f27f80cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.889+0000 7f27f7fff700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27f8069540 0x7f27f80cb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.889+0000 7f27f7fff700 1 -- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27f81577e0 msgr2=0x7f27f8169aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.889+0000 7f27f7fff700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27f81577e0 0x7f27f8169aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:52.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.889+0000 7f27f7fff700 1 -- 172.21.15.27:0/460152457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f0009a20 con 0x7f27f8157410 2024-01-11T09:26:52.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.890+0000 7f27f7fff700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27f8157410 0x7f27f8154150 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f27e400c590 tx=0x7f27e4007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:52.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.890+0000 7f27f4ff9700 1 -- 172.21.15.27:0/460152457 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e4005c00 con 0x7f27f8157410 2024-01-11T09:26:52.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.890+0000 7f27f4ff9700 1 -- 172.21.15.27:0/460152457 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e4005d60 con 0x7f27f8157410 2024-01-11T09:26:52.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.890+0000 7f27fedff700 1 -- 172.21.15.27:0/460152457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27ec0093c0 con 0x7f27f8157410 2024-01-11T09:26:52.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.890+0000 7f27f4ff9700 1 -- 172.21.15.27:0/460152457 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e4007300 con 0x7f27f8157410 2024-01-11T09:26:52.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.890+0000 7f27fedff700 1 -- 172.21.15.27:0/460152457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f816a8b0 con 0x7f27f8157410 2024-01-11T09:26:52.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.891+0000 7f27fedff700 1 -- 172.21.15.27:0/460152457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f804c3c0 con 0x7f27f8157410 2024-01-11T09:26:52.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.892+0000 7f27f4ff9700 1 -- 172.21.15.27:0/460152457 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 91) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f27e401b3b0 con 0x7f27f8157410 2024-01-11T09:26:52.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.893+0000 7f27f4ff9700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f27dc066f80 0x7f27dc069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.894+0000 7f27f4ff9700 1 -- 172.21.15.27:0/460152457 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f27e4083ad0 con 0x7f27f8157410 2024-01-11T09:26:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.894+0000 7f27fcb9b700 1 -- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f27dc066f80 msgr2=0x7f27dc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:26:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.894+0000 7f27fcb9b700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f27dc066f80 0x7f27dc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:26:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:52.898+0000 7f27f4ff9700 1 -- 172.21.15.27:0/460152457 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f27e4050d20 con 0x7f27f8157410 2024-01-11T09:26:52.965 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:52.968+0000 7f1fd9efe1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:26:53.042 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:53.045+0000 7f2f29b9c1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:26:53.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:53.095+0000 7f27fcb9b700 1 -- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f27dc066f80 msgr2=0x7f27dc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:26:53.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:53.095+0000 7f27fcb9b700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f27dc066f80 0x7f27dc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:26:53.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:53.129+0000 7f27fedff700 1 -- 172.21.15.27:0/460152457 --> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f27f8155fb0 con 0x7f27dc066f80 2024-01-11T09:26:53.145 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:53.148+0000 7f1fd9efe1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:26:53.461 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:53.463+0000 7f2f29b9c1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:26:53.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:53.496+0000 7f27fcb9b700 1 -- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f27dc066f80 msgr2=0x7f27dc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:26:53.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:53.496+0000 7f27fcb9b700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f27dc066f80 0x7f27dc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-11T09:26:53.581 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:53.583+0000 7f1fd9efe1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:26:54.233 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:54.236+0000 7f2f29b9c1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:26:54.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:54.297+0000 7f27fcb9b700 1 -- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f27dc066f80 msgr2=0x7f27dc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:26:54.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:54.297+0000 7f27fcb9b700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f27dc066f80 0x7f27dc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-11T09:26:54.321 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:54.324+0000 7f2f29b9c1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:26:54.401 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:54.404+0000 7f1fd9efe1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:26:54.476 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:54.479+0000 7f2f29b9c1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:26:54.494 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:54.496+0000 7f1fd9efe1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:26:54.564 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:54.566+0000 7f2f29b9c1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:26:54.657 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:54.660+0000 7f1fd9efe1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:26:54.750 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:54.752+0000 7f1fd9efe1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:26:54.779 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:54.782+0000 7f2f29b9c1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:26:54.876 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:54.879+0000 7f2f29b9c1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:26:54.972 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:54.975+0000 7f2f29b9c1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:26:54.977 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:54.979+0000 7f1fd9efe1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:26:55.067 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:55.070+0000 7f2f29b9c1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:26:55.077 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:55.080+0000 7f1fd9efe1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:26:55.178 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:55.181+0000 7f1fd9efe1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:26:55.256 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:55.259+0000 7f2f29b9c1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:26:55.278 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:55.281+0000 7f1fd9efe1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:26:55.353 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:55.356+0000 7f2f29b9c1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:26:55.478 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:55.481+0000 7f1fd9efe1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:26:55.581 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:55.584+0000 7f1fd9efe1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:26:55.699 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:55.703+0000 7f2f29b9c1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:26:55.789 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:55.791+0000 7f2f29b9c1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:26:55.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:55.898+0000 7f27fcb9b700 1 -- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f27dc066f80 msgr2=0x7f27dc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:26:55.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:55.898+0000 7f27fcb9b700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f27dc066f80 0x7f27dc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-11T09:26:55.944 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:55.947+0000 7f1fd9efe1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:26:56.040 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:56.042+0000 7f1fd9efe1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:26:56.330 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:56.333+0000 7f2f29b9c1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:26:56.515 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:56.519+0000 7f2f29b9c1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:26:56.614 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:56.617+0000 7f1fd9efe1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:26:56.814 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:56.816+0000 7f1fd9efe1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:26:56.893 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:56.896+0000 7f2f29b9c1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:26:57.210 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:57.214+0000 7f1fd9efe1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:26:57.314 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:57.318+0000 7f2f29b9c1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:26:57.497 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:57.501+0000 7f2f29b9c1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:26:57.658 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:57.661+0000 7f1fd9efe1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:26:57.685 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:57.689+0000 7f2f29b9c1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:26:57.800 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:57.804+0000 7f2f29b9c1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:26:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:57.812+0000 7f27f4ff9700 1 -- 172.21.15.27:0/460152457 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 92) v1 ==== 84470+0+0 (secure 0 0 0) 0x7f27e40510f0 con 0x7f27f8157410 2024-01-11T09:26:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:57.813+0000 7f27f4ff9700 1 -- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f27dc066f80 msgr2=0x7f27dc069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:26:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:57.813+0000 7f27f4ff9700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f27dc066f80 0x7f27dc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:57.857 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:57.861+0000 7f1fd9efe1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:26:57.944 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:57.948+0000 7f2ea21bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:57.944 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:57.948+0000 7f2ea21bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:57.944 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:57.948+0000 7f2ea21bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:57.944 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:57.948+0000 7f2ea21bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:57.945 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:57.948+0000 7f2ea21bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:57.945 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:57.948+0000 7f2e9f1b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:57.945 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:57.948+0000 7f2e9f1b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:57.945 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:57.948+0000 7f2e9f1b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:57.945 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:57.948+0000 7f2e9f1b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:57.945 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:57.948+0000 7f2e9f1b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:26:58.069 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:58.073+0000 7f1fd9efe1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:26:58.184 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:26:58.188+0000 7f1fd9efe1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:26:58.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:58.816+0000 7f27f4ff9700 1 -- 172.21.15.27:0/460152457 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 93) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f27e4054c40 con 0x7f27f8157410 2024-01-11T09:26:58.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:58.816+0000 7f27f4ff9700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f27dc066f80 0x7f27dc0693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:58.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:58.816+0000 7f27f4ff9700 1 -- 172.21.15.27:0/460152457 --> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f27f8155fb0 con 0x7f27dc066f80 2024-01-11T09:26:58.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:58.817+0000 7f27fcb9b700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f27dc066f80 0x7f27dc0693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:58.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:58.819+0000 7f27fcb9b700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f27dc066f80 0x7f27dc0693b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f27f8152bd0 tx=0x7f27ec016000 comp rx=0 tx=0).ready entity=mgr.8914 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:58.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:58.894+0000 7f27f4ff9700 1 -- 172.21.15.27:0/460152457 <== mgr.8914 v2:172.21.15.77:6836/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f27f8155fb0 con 0x7f27dc066f80 2024-01-11T09:26:58.891 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path snap_test_dir1 2024-01-11T09:26:58.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:58.896+0000 7f27fedff700 1 -- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f27dc066f80 msgr2=0x7f27dc0693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:58.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:58.896+0000 7f27fedff700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f27dc066f80 0x7f27dc0693b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f27f8152bd0 tx=0x7f27ec016000 comp rx=0 tx=0).stop 2024-01-11T09:26:58.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:58.896+0000 7f27fedff700 1 -- 172.21.15.27:0/460152457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27f8157410 msgr2=0x7f27f8154150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:58.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:58.896+0000 7f27fedff700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27f8157410 0x7f27f8154150 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f27e400c590 tx=0x7f27e4007dd0 comp rx=0 tx=0).stop 2024-01-11T09:26:58.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:58.896+0000 7f27fedff700 1 -- 172.21.15.27:0/460152457 shutdown_connections 2024-01-11T09:26:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:58.896+0000 7f27fedff700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f27dc066f80 0x7f27dc0693b0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:58.896+0000 7f27fedff700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27f81577e0 0x7f27f8169aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:58.896+0000 7f27fedff700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27f8069540 0x7f27f80cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:58.896+0000 7f27fedff700 1 --2- 172.21.15.27:0/460152457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27f8157410 0x7f27f8154150 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:58.896+0000 7f27fedff700 1 -- 172.21.15.27:0/460152457 >> 172.21.15.27:0/460152457 conn(0x7f27f80680a0 msgr2=0x7f27f8152dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:58.897+0000 7f27fedff700 1 -- 172.21.15.27:0/460152457 shutdown_connections 2024-01-11T09:26:58.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:58.897+0000 7f27fedff700 1 -- 172.21.15.27:0/460152457 wait complete. 2024-01-11T09:26:58.912 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-01-11T09:26:58.912 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-01-11T09:26:59.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.119+0000 7f2da5fff700 1 Processor -- start 2024-01-11T09:26:59.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.119+0000 7f2da5fff700 1 -- start start 2024-01-11T09:26:59.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.120+0000 7f2da5fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2da01577e0 0x7f2da00ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:59.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.120+0000 7f2da5fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2da0069540 0x7f2da00ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:59.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.120+0000 7f2da5fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da00699d0 con 0x7f2da0157410 2024-01-11T09:26:59.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.120+0000 7f2da5fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2da0069cb0 con 0x7f2da01577e0 2024-01-11T09:26:59.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.120+0000 7f2da5fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2da0069f90 con 0x7f2da0069540 2024-01-11T09:26:59.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.121+0000 7f2d9effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2da01577e0 0x7f2da00ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:59.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.121+0000 7f2d9effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2da01577e0 0x7f2da00ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39584/0 (socket says 172.21.15.27:39584) 2024-01-11T09:26:59.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.121+0000 7f2d9effd700 1 -- 172.21.15.27:0/445290351 learned_addr learned my addr 172.21.15.27:0/445290351 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:59.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.121+0000 7f2d9ffff700 1 --2- 172.21.15.27:0/445290351 >> v2:172.21.15.77:3300/0 conn(0x7f2da0069540 0x7f2da00ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:59.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.121+0000 7f2d9effd700 1 -- 172.21.15.27:0/445290351 >> v2:172.21.15.77:3300/0 conn(0x7f2da0069540 msgr2=0x7f2da00ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:59.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.121+0000 7f2d9effd700 1 --2- 172.21.15.27:0/445290351 >> v2:172.21.15.77:3300/0 conn(0x7f2da0069540 0x7f2da00ae540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:59.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.121+0000 7f2d9effd700 1 -- 172.21.15.27:0/445290351 >> v1:172.21.15.77:6789/0 conn(0x7f2da0157410 legacy=0x7f2da00ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:59.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.121+0000 7f2d9effd700 1 -- 172.21.15.27:0/445290351 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da006a270 con 0x7f2da01577e0 2024-01-11T09:26:59.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.121+0000 7f2d9ffff700 1 --2- 172.21.15.27:0/445290351 >> v2:172.21.15.77:3300/0 conn(0x7f2da0069540 0x7f2da00ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:26:59.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.121+0000 7f2d9effd700 1 --2- 172.21.15.27:0/445290351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2da01577e0 0x7f2da00ae000 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f2da006d030 tx=0x7f2d94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e05dc802d88e5b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:59.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.122+0000 7f2d9e7fc700 1 -- 172.21.15.27:0/445290351 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9400b860 con 0x7f2da01577e0 2024-01-11T09:26:59.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.122+0000 7f2d9e7fc700 1 -- 172.21.15.27:0/445290351 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d9400b9c0 con 0x7f2da01577e0 2024-01-11T09:26:59.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.122+0000 7f2d9e7fc700 1 -- 172.21.15.27:0/445290351 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9400bd60 con 0x7f2da01577e0 2024-01-11T09:26:59.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.122+0000 7f2da5fff700 1 -- 172.21.15.27:0/445290351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2da01577e0 msgr2=0x7f2da00ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:59.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.122+0000 7f2da5fff700 1 --2- 172.21.15.27:0/445290351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2da01577e0 0x7f2da00ae000 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f2da006d030 tx=0x7f2d94009250 comp rx=0 tx=0).stop 2024-01-11T09:26:59.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.122+0000 7f2da5fff700 1 -- 172.21.15.27:0/445290351 shutdown_connections 2024-01-11T09:26:59.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.122+0000 7f2da5fff700 1 --2- 172.21.15.27:0/445290351 >> v2:172.21.15.77:3300/0 conn(0x7f2da0069540 0x7f2da00ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:59.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.122+0000 7f2da5fff700 1 --2- 172.21.15.27:0/445290351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2da01577e0 0x7f2da00ae000 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:59.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.122+0000 7f2da5fff700 1 -- 172.21.15.27:0/445290351 >> 172.21.15.27:0/445290351 conn(0x7f2da00680a0 msgr2=0x7f2da00cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:59.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.122+0000 7f2da5fff700 1 -- 172.21.15.27:0/445290351 shutdown_connections 2024-01-11T09:26:59.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.122+0000 7f2da5fff700 1 -- 172.21.15.27:0/445290351 wait complete. 2024-01-11T09:26:59.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.124+0000 7f2da5fff700 1 Processor -- start 2024-01-11T09:26:59.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.124+0000 7f2da5fff700 1 -- start start 2024-01-11T09:26:59.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.125+0000 7f2da5fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2da0069540 0x7f2da015a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:59.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.125+0000 7f2da5fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2da0157410 0x7f2da014a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:59.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.125+0000 7f2da5fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2da01577e0 0x7f2da00c48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:59.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.125+0000 7f2da5fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da014a660 con 0x7f2da01577e0 2024-01-11T09:26:59.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.125+0000 7f2da5fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da00b17a0 con 0x7f2da0157410 2024-01-11T09:26:59.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.125+0000 7f2da5fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da00b1a80 con 0x7f2da0069540 2024-01-11T09:26:59.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.125+0000 7f2d9ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2da01577e0 0x7f2da00c48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:59.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.125+0000 7f2d9ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2da01577e0 0x7f2da00c48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39596/0 (socket says 172.21.15.27:39596) 2024-01-11T09:26:59.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.125+0000 7f2d9f7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2da0069540 0x7f2da015a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:59.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.125+0000 7f2d9effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2da0157410 0x7f2da014a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:59.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.125+0000 7f2d9ffff700 1 -- 172.21.15.27:0/1322415983 learned_addr learned my addr 172.21.15.27:0/1322415983 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:59.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.126+0000 7f2d9ffff700 1 -- 172.21.15.27:0/1322415983 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2da0069540 msgr2=0x7f2da015a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:59.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.126+0000 7f2d9ffff700 1 --2- 172.21.15.27:0/1322415983 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2da0069540 0x7f2da015a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:59.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.126+0000 7f2d9ffff700 1 -- 172.21.15.27:0/1322415983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2da0157410 msgr2=0x7f2da014a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:59.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.126+0000 7f2d9ffff700 1 --2- 172.21.15.27:0/1322415983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2da0157410 0x7f2da014a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:59.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.126+0000 7f2d9ffff700 1 -- 172.21.15.27:0/1322415983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d940094f0 con 0x7f2da01577e0 2024-01-11T09:26:59.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.126+0000 7f2d9ffff700 1 --2- 172.21.15.27:0/1322415983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2da01577e0 0x7f2da00c48a0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800a380 tx=0x7f2d98007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:59.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.126+0000 7f2d877fe700 1 -- 172.21.15.27:0/1322415983 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d98015440 con 0x7f2da01577e0 2024-01-11T09:26:59.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.126+0000 7f2d877fe700 1 -- 172.21.15.27:0/1322415983 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d980155a0 con 0x7f2da01577e0 2024-01-11T09:26:59.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.126+0000 7f2d877fe700 1 -- 172.21.15.27:0/1322415983 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d98015960 con 0x7f2da01577e0 2024-01-11T09:26:59.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.126+0000 7f2da5fff700 1 -- 172.21.15.27:0/1322415983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da0157d50 con 0x7f2da01577e0 2024-01-11T09:26:59.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.126+0000 7f2da5fff700 1 -- 172.21.15.27:0/1322415983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da014e670 con 0x7f2da01577e0 2024-01-11T09:26:59.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.127+0000 7f2da5fff700 1 -- 172.21.15.27:0/1322415983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da0157bb0 con 0x7f2da01577e0 2024-01-11T09:26:59.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.128+0000 7f2d877fe700 1 -- 172.21.15.27:0/1322415983 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 93) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f2d9800d440 con 0x7f2da01577e0 2024-01-11T09:26:59.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.129+0000 7f2d877fe700 1 --2- 172.21.15.27:0/1322415983 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2d8c066dd0 0x7f2d8c069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:59.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.129+0000 7f2d877fe700 1 -- 172.21.15.27:0/1322415983 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f2d98084e80 con 0x7f2da01577e0 2024-01-11T09:26:59.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.130+0000 7f2d9f7fe700 1 --2- 172.21.15.27:0/1322415983 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2d8c066dd0 0x7f2d8c069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:59.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.130+0000 7f2d9f7fe700 1 --2- 172.21.15.27:0/1322415983 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2d8c066dd0 0x7f2d8c069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2da016ccb0 tx=0x7f2d90009040 comp rx=0 tx=0).ready entity=mgr.8914 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:59.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.133+0000 7f2d877fe700 1 -- 172.21.15.27:0/1322415983 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2da0157bb0 con 0x7f2da01577e0 2024-01-11T09:26:59.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.365+0000 7f2da5fff700 1 -- 172.21.15.27:0/1322415983 --> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2da0157bb0 con 0x7f2d8c066dd0 2024-01-11T09:26:59.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.433+0000 7f2d877fe700 1 -- 172.21.15.27:0/1322415983 <== mgr.8914 v2:172.21.15.77:6836/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f2da0157bb0 con 0x7f2d8c066dd0 2024-01-11T09:26:59.430 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule removed for path snap_test_dir1 2024-01-11T09:26:59.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.435+0000 7f2da5fff700 1 -- 172.21.15.27:0/1322415983 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2d8c066dd0 msgr2=0x7f2d8c069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:59.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.435+0000 7f2da5fff700 1 --2- 172.21.15.27:0/1322415983 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2d8c066dd0 0x7f2d8c069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2da016ccb0 tx=0x7f2d90009040 comp rx=0 tx=0).stop 2024-01-11T09:26:59.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.435+0000 7f2da5fff700 1 -- 172.21.15.27:0/1322415983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2da01577e0 msgr2=0x7f2da00c48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:59.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.435+0000 7f2da5fff700 1 --2- 172.21.15.27:0/1322415983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2da01577e0 0x7f2da00c48a0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800a380 tx=0x7f2d98007900 comp rx=0 tx=0).stop 2024-01-11T09:26:59.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.436+0000 7f2da5fff700 1 -- 172.21.15.27:0/1322415983 shutdown_connections 2024-01-11T09:26:59.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.436+0000 7f2da5fff700 1 --2- 172.21.15.27:0/1322415983 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2d8c066dd0 0x7f2d8c069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:59.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.436+0000 7f2da5fff700 1 --2- 172.21.15.27:0/1322415983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2da01577e0 0x7f2da00c48a0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:59.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.436+0000 7f2da5fff700 1 --2- 172.21.15.27:0/1322415983 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2da0069540 0x7f2da015a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:59.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.436+0000 7f2da5fff700 1 --2- 172.21.15.27:0/1322415983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2da0157410 0x7f2da014a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:59.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.436+0000 7f2da5fff700 1 -- 172.21.15.27:0/1322415983 >> 172.21.15.27:0/1322415983 conn(0x7f2da00680a0 msgr2=0x7f2da00c0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:59.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.436+0000 7f2da5fff700 1 -- 172.21.15.27:0/1322415983 shutdown_connections 2024-01-11T09:26:59.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.436+0000 7f2da5fff700 1 -- 172.21.15.27:0/1322415983 wait complete. 2024-01-11T09:26:59.450 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-01-11T09:26:59.451 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-01-11T09:26:59.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.654+0000 7f792e3ff700 1 Processor -- start 2024-01-11T09:26:59.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.655+0000 7f792e3ff700 1 -- start start 2024-01-11T09:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.656+0000 7f792e3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f792815a1b0 0x7f792815a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.656+0000 7f792e3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f79280ccd70 0x7f79280cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.656+0000 7f792e3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79280685c0 0x7f7928068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.656+0000 7f792e3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79280cf850 con 0x7f792815a1b0 2024-01-11T09:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.656+0000 7f792e3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79280cec40 con 0x7f79280ccd70 2024-01-11T09:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.656+0000 7f792e3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79280cef20 con 0x7f79280685c0 2024-01-11T09:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.656+0000 7f7927fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f792815a1b0 0x7f792815a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.656+0000 7f7927fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f792815a1b0 0x7f792815a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39600/0 (socket says 172.21.15.27:39600) 2024-01-11T09:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.656+0000 7f7927fff700 1 -- 172.21.15.27:0/1867759878 learned_addr learned my addr 172.21.15.27:0/1867759878 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.657+0000 7f79277fe700 1 --2- 172.21.15.27:0/1867759878 >> v2:172.21.15.77:3300/0 conn(0x7f79280ccd70 0x7f79280cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.657+0000 7f792c99c700 1 --2- 172.21.15.27:0/1867759878 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79280685c0 0x7f7928068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.657+0000 7f7927fff700 1 -- 172.21.15.27:0/1867759878 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79280685c0 msgr2=0x7f7928068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:59.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.657+0000 7f7927fff700 1 --2- 172.21.15.27:0/1867759878 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79280685c0 0x7f7928068a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:59.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.657+0000 7f7927fff700 1 -- 172.21.15.27:0/1867759878 >> v2:172.21.15.77:3300/0 conn(0x7f79280ccd70 msgr2=0x7f79280cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:26:59.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.657+0000 7f7927fff700 1 --2- 172.21.15.27:0/1867759878 >> v2:172.21.15.77:3300/0 conn(0x7f79280ccd70 0x7f79280cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:59.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.657+0000 7f7927fff700 1 -- 172.21.15.27:0/1867759878 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79280cf200 con 0x7f792815a1b0 2024-01-11T09:26:59.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.657+0000 7f7927fff700 1 --2- 172.21.15.27:0/1867759878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f792815a1b0 0x7f792815a580 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f79280696f0 tx=0x7f7918009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba73c7fda4d195de server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:59.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.657+0000 7f7926ffd700 1 -- 172.21.15.27:0/1867759878 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791800b860 con 0x7f792815a1b0 2024-01-11T09:26:59.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.657+0000 7f7926ffd700 1 -- 172.21.15.27:0/1867759878 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f791800b9c0 con 0x7f792815a1b0 2024-01-11T09:26:59.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.657+0000 7f7926ffd700 1 -- 172.21.15.27:0/1867759878 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791800bd60 con 0x7f792815a1b0 2024-01-11T09:26:59.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.658+0000 7f792e3ff700 1 -- 172.21.15.27:0/1867759878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f792815a1b0 msgr2=0x7f792815a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:59.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.658+0000 7f792e3ff700 1 --2- 172.21.15.27:0/1867759878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f792815a1b0 0x7f792815a580 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f79280696f0 tx=0x7f7918009250 comp rx=0 tx=0).stop 2024-01-11T09:26:59.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.658+0000 7f792e3ff700 1 -- 172.21.15.27:0/1867759878 shutdown_connections 2024-01-11T09:26:59.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.658+0000 7f792e3ff700 1 --2- 172.21.15.27:0/1867759878 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79280685c0 0x7f7928068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:59.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.658+0000 7f792e3ff700 1 --2- 172.21.15.27:0/1867759878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f792815a1b0 0x7f792815a580 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.658+0000 7f792e3ff700 1 --2- 172.21.15.27:0/1867759878 >> v2:172.21.15.77:3300/0 conn(0x7f79280ccd70 0x7f79280cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.658+0000 7f792e3ff700 1 -- 172.21.15.27:0/1867759878 >> 172.21.15.27:0/1867759878 conn(0x7f792806ed40 msgr2=0x7f792806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:26:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.658+0000 7f792e3ff700 1 -- 172.21.15.27:0/1867759878 shutdown_connections 2024-01-11T09:26:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.658+0000 7f792e3ff700 1 -- 172.21.15.27:0/1867759878 wait complete. 2024-01-11T09:26:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.659+0000 7f792e3ff700 1 Processor -- start 2024-01-11T09:26:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.660+0000 7f792e3ff700 1 -- start start 2024-01-11T09:26:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.660+0000 7f792e3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79280685c0 0x7f792815af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.661+0000 7f792e3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79280ccd70 0x7f792815c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.661+0000 7f792e3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f792815a1b0 0x7f79281558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.661+0000 7f792e3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792815cdd0 con 0x7f79280685c0 2024-01-11T09:26:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.661+0000 7f792e3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792815b600 con 0x7f79280ccd70 2024-01-11T09:26:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.661+0000 7f792e3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f792815b8e0 con 0x7f792815a1b0 2024-01-11T09:26:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.661+0000 7f7927fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79280685c0 0x7f792815af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.661+0000 7f7927fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79280685c0 0x7f792815af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39616/0 (socket says 172.21.15.27:39616) 2024-01-11T09:26:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.661+0000 7f7927fff700 1 -- 172.21.15.27:0/50847795 learned_addr learned my addr 172.21.15.27:0/50847795 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:26:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.661+0000 7f792c99c700 1 --2- 172.21.15.27:0/50847795 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f792815a1b0 0x7f79281558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.661+0000 7f79277fe700 1 --2- 172.21.15.27:0/50847795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79280ccd70 0x7f792815c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.661+0000 7f7927fff700 1 -- 172.21.15.27:0/50847795 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f792815a1b0 msgr2=0x7f79281558e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.661+0000 7f7927fff700 1 --2- 172.21.15.27:0/50847795 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f792815a1b0 0x7f79281558e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.661+0000 7f7927fff700 1 -- 172.21.15.27:0/50847795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79280ccd70 msgr2=0x7f792815c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:26:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.661+0000 7f7927fff700 1 --2- 172.21.15.27:0/50847795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79280ccd70 0x7f792815c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:26:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.661+0000 7f7927fff700 1 -- 172.21.15.27:0/50847795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79180094f0 con 0x7f79280685c0 2024-01-11T09:26:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.661+0000 7f79277fe700 1 --2- 172.21.15.27:0/50847795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79280ccd70 0x7f792815c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.661+0000 7f792c99c700 1 --2- 172.21.15.27:0/50847795 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f792815a1b0 0x7f79281558e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:26:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.662+0000 7f7927fff700 1 --2- 172.21.15.27:0/50847795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79280685c0 0x7f792815af40 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f7928069770 tx=0x7f7918017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.662+0000 7f790ffff700 1 -- 172.21.15.27:0/50847795 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7918024440 con 0x7f79280685c0 2024-01-11T09:26:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.662+0000 7f790ffff700 1 -- 172.21.15.27:0/50847795 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79180245a0 con 0x7f79280685c0 2024-01-11T09:26:59.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.662+0000 7f790ffff700 1 -- 172.21.15.27:0/50847795 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7918024960 con 0x7f79280685c0 2024-01-11T09:26:59.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.662+0000 7f792e3ff700 1 -- 172.21.15.27:0/50847795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f792815bbc0 con 0x7f79280685c0 2024-01-11T09:26:59.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.662+0000 7f792e3ff700 1 -- 172.21.15.27:0/50847795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79280c8880 con 0x7f79280685c0 2024-01-11T09:26:59.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.663+0000 7f792e3ff700 1 -- 172.21.15.27:0/50847795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f792804c3c0 con 0x7f79280685c0 2024-01-11T09:26:59.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.664+0000 7f790ffff700 1 -- 172.21.15.27:0/50847795 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 93) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f791801a420 con 0x7f79280685c0 2024-01-11T09:26:59.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.665+0000 7f790ffff700 1 --2- 172.21.15.27:0/50847795 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7914066dd0 0x7f7914069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:26:59.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.665+0000 7f790ffff700 1 -- 172.21.15.27:0/50847795 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f7918086270 con 0x7f79280685c0 2024-01-11T09:26:59.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.665+0000 7f79277fe700 1 --2- 172.21.15.27:0/50847795 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7914066dd0 0x7f7914069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:26:59.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.666+0000 7f79277fe700 1 --2- 172.21.15.27:0/50847795 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f7914066dd0 0x7f7914069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f79280b48e0 tx=0x7f791c008040 comp rx=0 tx=0).ready entity=mgr.8914 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:26:59.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.669+0000 7f790ffff700 1 -- 172.21.15.27:0/50847795 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7918053580 con 0x7f79280685c0 2024-01-11T09:26:59.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.874+0000 7f790ffff700 1 -- 172.21.15.27:0/50847795 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 94) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f7918053950 con 0x7f79280685c0 2024-01-11T09:26:59.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.902+0000 7f792e3ff700 1 -- 172.21.15.27:0/50847795 --> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f79280b2290 con 0x7f7914066dd0 2024-01-11T09:26:59.905 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:26:59.909+0000 7f2ea91cd700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-01-11T09:26:59.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:26:59.909+0000 7f790ffff700 1 -- 172.21.15.27:0/50847795 <== mgr.8914 v2:172.21.15.77:6836/41381 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f79280b2290 con 0x7f7914066dd0 2024-01-11T09:26:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-01-11T09:26:59.915 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-11T09:26:59.916 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:26:59.916 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-01-11T09:26:59.967 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-11T09:27:00.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.171+0000 7f609b9ff700 1 Processor -- start 2024-01-11T09:27:00.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.172+0000 7f609b9ff700 1 -- start start 2024-01-11T09:27:00.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.173+0000 7f609b9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f609413ff80 0x7f609415a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:00.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.173+0000 7f609b9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60940c6b30 con 0x7f609415a560 2024-01-11T09:27:00.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.173+0000 7f609b9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60940c6800 con 0x7f609413fbb0 2024-01-11T09:27:00.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.173+0000 7f609b9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f609415a960 con 0x7f609413ff80 2024-01-11T09:27:00.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.173+0000 7f6098f9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f609413ff80 0x7f609415a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:00.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.173+0000 7f6098f9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f609413ff80 0x7f609415a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39628/0 (socket says 172.21.15.27:39628) 2024-01-11T09:27:00.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.173+0000 7f6098f9a700 1 -- 172.21.15.27:0/1051788389 learned_addr learned my addr 172.21.15.27:0/1051788389 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:00.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.174+0000 7f6098f9a700 1 -- 172.21.15.27:0/1051788389 >> v1:172.21.15.77:6789/0 conn(0x7f609413fbb0 legacy=0x7f60940ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:00.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.174+0000 7f6098f9a700 1 -- 172.21.15.27:0/1051788389 >> v1:172.21.15.27:6789/0 conn(0x7f609415a560 legacy=0x7f60940c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:00.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.174+0000 7f6098f9a700 1 -- 172.21.15.27:0/1051788389 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60940cb240 con 0x7f609413ff80 2024-01-11T09:27:00.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.174+0000 7f608bfff700 1 -- 172.21.15.27:0/1051788389 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564088483 0 0) 0x7f60940c6b30 con 0x7f609415a560 2024-01-11T09:27:00.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.174+0000 7f6098f9a700 1 --2- 172.21.15.27:0/1051788389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f609413ff80 0x7f609415a020 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f60840099b0 tx=0x7f6084009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dce1c296dfd8f7b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:00.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.174+0000 7f608bfff700 1 -- 172.21.15.27:0/1051788389 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608400b860 con 0x7f609413ff80 2024-01-11T09:27:00.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.174+0000 7f608bfff700 1 -- 172.21.15.27:0/1051788389 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f608400b9c0 con 0x7f609413ff80 2024-01-11T09:27:00.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.174+0000 7f608bfff700 1 -- 172.21.15.27:0/1051788389 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608400bd60 con 0x7f609413ff80 2024-01-11T09:27:00.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.175+0000 7f609b9ff700 1 -- 172.21.15.27:0/1051788389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f609413ff80 msgr2=0x7f609415a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:00.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.175+0000 7f609b9ff700 1 --2- 172.21.15.27:0/1051788389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f609413ff80 0x7f609415a020 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f60840099b0 tx=0x7f6084009250 comp rx=0 tx=0).stop 2024-01-11T09:27:00.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.175+0000 7f609b9ff700 1 -- 172.21.15.27:0/1051788389 shutdown_connections 2024-01-11T09:27:00.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.175+0000 7f609b9ff700 1 --2- 172.21.15.27:0/1051788389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f609413ff80 0x7f609415a020 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:00.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.175+0000 7f609b9ff700 1 -- 172.21.15.27:0/1051788389 >> 172.21.15.27:0/1051788389 conn(0x7f60940680a0 msgr2=0x7f60940cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:00.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.175+0000 7f609b9ff700 1 -- 172.21.15.27:0/1051788389 shutdown_connections 2024-01-11T09:27:00.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.175+0000 7f609b9ff700 1 -- 172.21.15.27:0/1051788389 wait complete. 2024-01-11T09:27:00.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.177+0000 7f609b9ff700 1 Processor -- start 2024-01-11T09:27:00.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.177+0000 7f609b9ff700 1 -- start start 2024-01-11T09:27:00.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.178+0000 7f609b9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f609413fbb0 0x7f60941561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:00.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.179+0000 7f609b9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f609413ff80 0x7f6094156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:00.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.179+0000 7f609b9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f609415a560 0x7f6094158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:00.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.179+0000 7f609b9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60941580d0 con 0x7f609413fbb0 2024-01-11T09:27:00.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.179+0000 7f609b9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6094156700 con 0x7f609413ff80 2024-01-11T09:27:00.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.179+0000 7f609b9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60941574c0 con 0x7f609415a560 2024-01-11T09:27:00.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.179+0000 7f609979b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f609413fbb0 0x7f60941561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:00.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.179+0000 7f609979b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f609413fbb0 0x7f60941561c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39642/0 (socket says 172.21.15.27:39642) 2024-01-11T09:27:00.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.179+0000 7f609979b700 1 -- 172.21.15.27:0/4236696267 learned_addr learned my addr 172.21.15.27:0/4236696267 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:00.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.179+0000 7f6098f9a700 1 --2- 172.21.15.27:0/4236696267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f609413ff80 0x7f6094156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:00.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.179+0000 7f6099f9c700 1 --2- 172.21.15.27:0/4236696267 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f609415a560 0x7f6094158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:00.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.179+0000 7f609979b700 1 -- 172.21.15.27:0/4236696267 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f609415a560 msgr2=0x7f6094158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:00.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.179+0000 7f609979b700 1 --2- 172.21.15.27:0/4236696267 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f609415a560 0x7f6094158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:00.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.179+0000 7f609979b700 1 -- 172.21.15.27:0/4236696267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f609413ff80 msgr2=0x7f6094156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:00.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.179+0000 7f609979b700 1 --2- 172.21.15.27:0/4236696267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f609413ff80 0x7f6094156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:00.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.179+0000 7f609979b700 1 -- 172.21.15.27:0/4236696267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60840094f0 con 0x7f609413fbb0 2024-01-11T09:27:00.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.180+0000 7f609979b700 1 --2- 172.21.15.27:0/4236696267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f609413fbb0 0x7f60941561c0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f609000c7f0 tx=0x7f6090007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:00.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.180+0000 7f60897fa700 1 -- 172.21.15.27:0/4236696267 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6090011590 con 0x7f609413fbb0 2024-01-11T09:27:00.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.180+0000 7f60897fa700 1 -- 172.21.15.27:0/4236696267 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6090011710 con 0x7f609413fbb0 2024-01-11T09:27:00.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.180+0000 7f60897fa700 1 -- 172.21.15.27:0/4236696267 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6090011ad0 con 0x7f609413fbb0 2024-01-11T09:27:00.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.180+0000 7f609b9ff700 1 -- 172.21.15.27:0/4236696267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60941577a0 con 0x7f609413fbb0 2024-01-11T09:27:00.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.180+0000 7f609b9ff700 1 -- 172.21.15.27:0/4236696267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6094157d60 con 0x7f609413fbb0 2024-01-11T09:27:00.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.181+0000 7f609b9ff700 1 -- 172.21.15.27:0/4236696267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f609404c3c0 con 0x7f609413fbb0 2024-01-11T09:27:00.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.182+0000 7f60897fa700 1 -- 172.21.15.27:0/4236696267 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 94) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f609001b3b0 con 0x7f609413fbb0 2024-01-11T09:27:00.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.183+0000 7f60897fa700 1 --2- 172.21.15.27:0/4236696267 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f60780670a0 0x7f6078069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:00.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.183+0000 7f60897fa700 1 -- 172.21.15.27:0/4236696267 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f6090083d20 con 0x7f609413fbb0 2024-01-11T09:27:00.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.183+0000 7f6098f9a700 1 --2- 172.21.15.27:0/4236696267 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f60780670a0 0x7f6078069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:00.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.184+0000 7f6098f9a700 1 --2- 172.21.15.27:0/4236696267 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f60780670a0 0x7f6078069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f60840099b0 tx=0x7f6084017000 comp rx=0 tx=0).ready entity=mgr.8914 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:00.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.188+0000 7f60897fa700 1 -- 172.21.15.27:0/4236696267 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6090050d50 con 0x7f609413fbb0 2024-01-11T09:27:00.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.453+0000 7f609b9ff700 1 -- 172.21.15.27:0/4236696267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f60940cb520 con 0x7f609413fbb0 2024-01-11T09:27:00.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.883+0000 7f60897fa700 1 -- 172.21.15.27:0/4236696267 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 95) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f60900b55a0 con 0x7f609413fbb0 2024-01-11T09:27:00.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.885+0000 7f60897fa700 1 -- 172.21.15.27:0/4236696267 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v95) v1 ==== 93+0+0 (secure 0 0 0) 0x7f6090050d50 con 0x7f609413fbb0 2024-01-11T09:27:00.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.887+0000 7f609b9ff700 1 -- 172.21.15.27:0/4236696267 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f60780670a0 msgr2=0x7f6078069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:00.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.887+0000 7f609b9ff700 1 --2- 172.21.15.27:0/4236696267 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f60780670a0 0x7f6078069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f60840099b0 tx=0x7f6084017000 comp rx=0 tx=0).stop 2024-01-11T09:27:00.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.887+0000 7f609b9ff700 1 -- 172.21.15.27:0/4236696267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f609413fbb0 msgr2=0x7f60941561c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:00.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.887+0000 7f609b9ff700 1 --2- 172.21.15.27:0/4236696267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f609413fbb0 0x7f60941561c0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f609000c7f0 tx=0x7f6090007f70 comp rx=0 tx=0).stop 2024-01-11T09:27:00.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.888+0000 7f609b9ff700 1 -- 172.21.15.27:0/4236696267 shutdown_connections 2024-01-11T09:27:00.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.888+0000 7f609b9ff700 1 --2- 172.21.15.27:0/4236696267 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f60780670a0 0x7f6078069550 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f60840099b0 tx=0x7f6084017000 comp rx=0 tx=0).stop 2024-01-11T09:27:00.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.888+0000 7f609b9ff700 1 --2- 172.21.15.27:0/4236696267 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f609415a560 0x7f6094158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:00.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.888+0000 7f609b9ff700 1 --2- 172.21.15.27:0/4236696267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f609413fbb0 0x7f60941561c0 secure :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f609000c7f0 tx=0x7f6090007f70 comp rx=0 tx=0).stop 2024-01-11T09:27:00.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.888+0000 7f609b9ff700 1 --2- 172.21.15.27:0/4236696267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f609413ff80 0x7f6094156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:00.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.888+0000 7f609b9ff700 1 -- 172.21.15.27:0/4236696267 >> 172.21.15.27:0/4236696267 conn(0x7f60940680a0 msgr2=0x7f609413e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:00.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.888+0000 7f609b9ff700 1 -- 172.21.15.27:0/4236696267 shutdown_connections 2024-01-11T09:27:00.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:00.888+0000 7f609b9ff700 1 -- 172.21.15.27:0/4236696267 wait complete. 2024-01-11T09:27:00.905 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T09:27:00.905 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:27:01.007 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:27:01.096 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T09:27:01.096 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-11T09:27:01.096 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:27:01.096 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-01-11T09:27:01.102 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:01.106+0000 7f8450c391c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:27:01.103 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:01.106+0000 7f6e5d36f1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:27:01.178 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T09:27:01.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:27:01.179 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-11T09:27:01.282 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:01.286+0000 7f8450c391c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:27:01.302 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:01.306+0000 7f6e5d36f1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:27:01.305 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:27:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.512+0000 7f63bb7ff700 1 Processor -- start 2024-01-11T09:27:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.512+0000 7f63bb7ff700 1 -- start start 2024-01-11T09:27:01.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.514+0000 7f63bb7ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f63b40ac370 0x7f63b40af320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:01.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.514+0000 7f63bb7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b40aa1d0 con 0x7f63b40ad020 2024-01-11T09:27:01.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.514+0000 7f63bb7ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b40af860 con 0x7f63b4159890 2024-01-11T09:27:01.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.514+0000 7f63bb7ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b40ac8d0 con 0x7f63b40ac370 2024-01-11T09:27:01.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.514+0000 7f63b959b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f63b40ad020 0x7f63b4159180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46798/0 (socket says 172.21.15.27:46798) 2024-01-11T09:27:01.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.514+0000 7f63b959b700 1 -- 172.21.15.27:0/3648041924 learned_addr learned my addr 172.21.15.27:0/3648041924 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:01.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.514+0000 7f63b9d9c700 1 --2- 172.21.15.27:0/3648041924 >> v2:172.21.15.27:3300/0 conn(0x7f63b40ac370 0x7f63b40af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.514+0000 7f63abfff700 1 -- 172.21.15.27:0/3648041924 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905161578 0 0) 0x7f63b40aa1d0 con 0x7f63b40ad020 2024-01-11T09:27:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.515+0000 7f63abfff700 1 -- 172.21.15.27:0/3648041924 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a0003570 con 0x7f63b40ad020 2024-01-11T09:27:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.515+0000 7f63abfff700 1 -- 172.21.15.27:0/3648041924 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191060909 0 0) 0x7f63b40af860 con 0x7f63b4159890 2024-01-11T09:27:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.515+0000 7f63abfff700 1 -- 172.21.15.27:0/3648041924 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b40aa1d0 con 0x7f63b4159890 2024-01-11T09:27:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.515+0000 7f63b9d9c700 1 -- 172.21.15.27:0/3648041924 >> v1:172.21.15.77:6789/0 conn(0x7f63b4159890 legacy=0x7f63b40aa660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.515+0000 7f63b9d9c700 1 -- 172.21.15.27:0/3648041924 >> v1:172.21.15.27:6789/0 conn(0x7f63b40ad020 legacy=0x7f63b4159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.515+0000 7f63b9d9c700 1 -- 172.21.15.27:0/3648041924 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b40a98a0 con 0x7f63b40ac370 2024-01-11T09:27:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.515+0000 7f63b9d9c700 1 --2- 172.21.15.27:0/3648041924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b40ac370 0x7f63b40af320 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f63ac009e60 tx=0x7f63ac009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999d13284837ac52 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.515+0000 7f63abfff700 1 -- 172.21.15.27:0/3648041924 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac00e070 con 0x7f63b40ac370 2024-01-11T09:27:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.515+0000 7f63abfff700 1 -- 172.21.15.27:0/3648041924 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63ac00bab0 con 0x7f63b40ac370 2024-01-11T09:27:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.515+0000 7f63abfff700 1 -- 172.21.15.27:0/3648041924 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac00be70 con 0x7f63b40ac370 2024-01-11T09:27:01.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.516+0000 7f63bb7ff700 1 -- 172.21.15.27:0/3648041924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b40ac370 msgr2=0x7f63b40af320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:01.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.516+0000 7f63bb7ff700 1 --2- 172.21.15.27:0/3648041924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b40ac370 0x7f63b40af320 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f63ac009e60 tx=0x7f63ac009780 comp rx=0 tx=0).stop 2024-01-11T09:27:01.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.516+0000 7f63bb7ff700 1 -- 172.21.15.27:0/3648041924 shutdown_connections 2024-01-11T09:27:01.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.516+0000 7f63bb7ff700 1 --2- 172.21.15.27:0/3648041924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b40ac370 0x7f63b40af320 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:01.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.516+0000 7f63bb7ff700 1 -- 172.21.15.27:0/3648041924 >> 172.21.15.27:0/3648041924 conn(0x7f63b40680e0 msgr2=0x7f63b4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:01.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.516+0000 7f63bb7ff700 1 -- 172.21.15.27:0/3648041924 shutdown_connections 2024-01-11T09:27:01.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.516+0000 7f63bb7ff700 1 -- 172.21.15.27:0/3648041924 wait complete. 2024-01-11T09:27:01.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.518+0000 7f63bb7ff700 1 Processor -- start 2024-01-11T09:27:01.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.518+0000 7f63bb7ff700 1 -- start start 2024-01-11T09:27:01.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.519+0000 7f63bb7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b40ac370 0x7f63b414a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:01.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.519+0000 7f63bb7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63b40ad020 0x7f63b41465a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:01.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.519+0000 7f63bb7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63b4159890 0x7f63b4146da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:01.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.519+0000 7f63bb7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b414b5b0 con 0x7f63b40ac370 2024-01-11T09:27:01.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.519+0000 7f63bb7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b4146ae0 con 0x7f63b4159890 2024-01-11T09:27:01.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.519+0000 7f63bb7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b41472e0 con 0x7f63b40ad020 2024-01-11T09:27:01.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.519+0000 7f63b959b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b40ac370 0x7f63b414a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:01.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.519+0000 7f63b959b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b40ac370 0x7f63b414a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39676/0 (socket says 172.21.15.27:39676) 2024-01-11T09:27:01.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.519+0000 7f63b959b700 1 -- 172.21.15.27:0/3471731936 learned_addr learned my addr 172.21.15.27:0/3471731936 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:01.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.519+0000 7f63b8d9a700 1 --2- 172.21.15.27:0/3471731936 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63b40ad020 0x7f63b41465a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:01.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.519+0000 7f63b9d9c700 1 --2- 172.21.15.27:0/3471731936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63b4159890 0x7f63b4146da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:01.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.520+0000 7f63b959b700 1 -- 172.21.15.27:0/3471731936 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63b40ad020 msgr2=0x7f63b41465a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:01.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.520+0000 7f63b959b700 1 --2- 172.21.15.27:0/3471731936 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63b40ad020 0x7f63b41465a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:01.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.520+0000 7f63b959b700 1 -- 172.21.15.27:0/3471731936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63b4159890 msgr2=0x7f63b4146da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:01.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.520+0000 7f63b959b700 1 --2- 172.21.15.27:0/3471731936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63b4159890 0x7f63b4146da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:01.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.520+0000 7f63b959b700 1 -- 172.21.15.27:0/3471731936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ac009a20 con 0x7f63b40ac370 2024-01-11T09:27:01.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.520+0000 7f63b9d9c700 1 --2- 172.21.15.27:0/3471731936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63b4159890 0x7f63b4146da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:27:01.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.520+0000 7f63b959b700 1 --2- 172.21.15.27:0/3471731936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b40ac370 0x7f63b414a2a0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f63b000c7c0 tx=0x7f63b0007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:01.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.520+0000 7f63a97fa700 1 -- 172.21.15.27:0/3471731936 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b0005b30 con 0x7f63b40ac370 2024-01-11T09:27:01.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.520+0000 7f63a97fa700 1 -- 172.21.15.27:0/3471731936 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b0005cd0 con 0x7f63b40ac370 2024-01-11T09:27:01.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.520+0000 7f63a97fa700 1 -- 172.21.15.27:0/3471731936 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b0007320 con 0x7f63b40ac370 2024-01-11T09:27:01.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.520+0000 7f63bb7ff700 1 -- 172.21.15.27:0/3471731936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b414a9a0 con 0x7f63b40ac370 2024-01-11T09:27:01.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.520+0000 7f63bb7ff700 1 -- 172.21.15.27:0/3471731936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b414af60 con 0x7f63b40ac370 2024-01-11T09:27:01.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.521+0000 7f63bb7ff700 1 -- 172.21.15.27:0/3471731936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b40718d0 con 0x7f63b40ac370 2024-01-11T09:27:01.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.523+0000 7f63a97fa700 1 -- 172.21.15.27:0/3471731936 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 95) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f63b0012400 con 0x7f63b40ac370 2024-01-11T09:27:01.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.525+0000 7f63a97fa700 1 --2- 172.21.15.27:0/3471731936 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6398067050 0x7f6398069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:01.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.525+0000 7f63a97fa700 1 -- 172.21.15.27:0/3471731936 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f63b0083bd0 con 0x7f63b40ac370 2024-01-11T09:27:01.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.525+0000 7f63b8d9a700 1 -- 172.21.15.27:0/3471731936 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6398067050 msgr2=0x7f6398069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:01.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.525+0000 7f63b8d9a700 1 --2- 172.21.15.27:0/3471731936 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6398067050 0x7f6398069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:27:01.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.529+0000 7f63a97fa700 1 -- 172.21.15.27:0/3471731936 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f63b004b5a0 con 0x7f63b40ac370 2024-01-11T09:27:01.596 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:01.600+0000 7f8450c391c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:27:01.647 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:01.651+0000 7f6e5d36f1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:27:01.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.726+0000 7f63b8d9a700 1 -- 172.21.15.27:0/3471731936 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6398067050 msgr2=0x7f6398069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:01.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.726+0000 7f63b8d9a700 1 --2- 172.21.15.27:0/3471731936 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6398067050 0x7f6398069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:27:01.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.781+0000 7f63bb7ff700 1 -- 172.21.15.27:0/3471731936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63b40a9b80 con 0x7f63b40ac370 2024-01-11T09:27:01.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.782+0000 7f63a97fa700 1 -- 172.21.15.27:0/3471731936 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4598 (secure 0 0 0) 0x7f63b0017020 con 0x7f63b40ac370 2024-01-11T09:27:01.778 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:27:01.778 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":86,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7700,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/191611944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":191611944},{"type":"v1","addr":"172.21.15.27:6835","nonce":191611944}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":8734,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/3566978772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3566978772},{"type":"v1","addr":"172.21.15.77:6834","nonce":3566978772}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":8737,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1168169536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1168169536},{"type":"v1","addr":"172.21.15.77:6835","nonce":1168169536}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:26:41.190629+0000","modified":"2024-01-11T09:26:44.707534+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9093},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9093":{"gid":9093,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/3544429684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3544429684},{"type":"v1","addr":"172.21.15.27:6834","nonce":3544429684}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2024-01-11T09:27:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.784+0000 7f63bb7ff700 1 -- 172.21.15.27:0/3471731936 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6398067050 msgr2=0x7f6398069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:27:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.784+0000 7f63bb7ff700 1 --2- 172.21.15.27:0/3471731936 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6398067050 0x7f6398069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.784+0000 7f63bb7ff700 1 -- 172.21.15.27:0/3471731936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b40ac370 msgr2=0x7f63b414a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.784+0000 7f63bb7ff700 1 --2- 172.21.15.27:0/3471731936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b40ac370 0x7f63b414a2a0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f63b000c7c0 tx=0x7f63b0007e80 comp rx=0 tx=0).stop 2024-01-11T09:27:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.784+0000 7f63bb7ff700 1 -- 172.21.15.27:0/3471731936 shutdown_connections 2024-01-11T09:27:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.784+0000 7f63bb7ff700 1 --2- 172.21.15.27:0/3471731936 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6398067050 0x7f6398069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.784+0000 7f63bb7ff700 1 --2- 172.21.15.27:0/3471731936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63b4159890 0x7f63b4146da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.784+0000 7f63bb7ff700 1 --2- 172.21.15.27:0/3471731936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b40ac370 0x7f63b414a2a0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.784+0000 7f63bb7ff700 1 --2- 172.21.15.27:0/3471731936 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63b40ad020 0x7f63b41465a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.784+0000 7f63bb7ff700 1 -- 172.21.15.27:0/3471731936 >> 172.21.15.27:0/3471731936 conn(0x7f63b40680e0 msgr2=0x7f63b41581f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.784+0000 7f63bb7ff700 1 -- 172.21.15.27:0/3471731936 shutdown_connections 2024-01-11T09:27:01.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:01.784+0000 7f63bb7ff700 1 -- 172.21.15.27:0/3471731936 wait complete. 2024-01-11T09:27:01.782 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 86 2024-01-11T09:27:01.800 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:27:02.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.007+0000 7f1bcbfff700 1 Processor -- start 2024-01-11T09:27:02.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.007+0000 7f1bcbfff700 1 -- start start 2024-01-11T09:27:02.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.009+0000 7f1bcbfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1bc40cca20 0x7f1bc4159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:02.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.009+0000 7f1bcbfff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc415d640 con 0x7f1bc40ca7b0 2024-01-11T09:27:02.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.009+0000 7f1bcbfff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc40cce60 con 0x7f1bc40ca3e0 2024-01-11T09:27:02.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.009+0000 7f1bcbfff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bc415ae70 con 0x7f1bc40cca20 2024-01-11T09:27:02.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.010+0000 7f1bc959a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1bc40ca7b0 0x7f1bc415dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46808/0 (socket says 172.21.15.27:46808) 2024-01-11T09:27:02.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.010+0000 7f1bc959a700 1 -- 172.21.15.27:0/1058414281 learned_addr learned my addr 172.21.15.27:0/1058414281 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:02.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.010+0000 7f1bca59c700 1 --2- 172.21.15.27:0/1058414281 >> v2:172.21.15.27:3300/0 conn(0x7f1bc40cca20 0x7f1bc4159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:02.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.010+0000 7f1bc8d99700 1 -- 172.21.15.27:0/1058414281 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1878189547 0 0) 0x7f1bc40cce60 con 0x7f1bc40ca3e0 2024-01-11T09:27:02.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.010+0000 7f1bc8d99700 1 -- 172.21.15.27:0/1058414281 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb0003570 con 0x7f1bc40ca3e0 2024-01-11T09:27:02.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.010+0000 7f1bc8d99700 1 -- 172.21.15.27:0/1058414281 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694863935 0 0) 0x7f1bc415d640 con 0x7f1bc40ca7b0 2024-01-11T09:27:02.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.010+0000 7f1bc8d99700 1 -- 172.21.15.27:0/1058414281 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc40cce60 con 0x7f1bc40ca7b0 2024-01-11T09:27:02.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.010+0000 7f1bca59c700 1 -- 172.21.15.27:0/1058414281 >> v1:172.21.15.77:6789/0 conn(0x7f1bc40ca3e0 legacy=0x7f1bc40c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:02.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.010+0000 7f1bca59c700 1 -- 172.21.15.27:0/1058414281 >> v1:172.21.15.27:6789/0 conn(0x7f1bc40ca7b0 legacy=0x7f1bc415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:02.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.010+0000 7f1bca59c700 1 -- 172.21.15.27:0/1058414281 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc415a5d0 con 0x7f1bc40cca20 2024-01-11T09:27:02.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.011+0000 7f1bca59c700 1 --2- 172.21.15.27:0/1058414281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bc40cca20 0x7f1bc4159e80 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc009e90 tx=0x7f1bbc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86c4af6b9aa10d5b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:02.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.011+0000 7f1bc8d99700 1 -- 172.21.15.27:0/1058414281 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc00e070 con 0x7f1bc40cca20 2024-01-11T09:27:02.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.011+0000 7f1bc8d99700 1 -- 172.21.15.27:0/1058414281 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bbc00bb90 con 0x7f1bc40cca20 2024-01-11T09:27:02.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.011+0000 7f1bc8d99700 1 -- 172.21.15.27:0/1058414281 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc012430 con 0x7f1bc40cca20 2024-01-11T09:27:02.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.012+0000 7f1bcbfff700 1 -- 172.21.15.27:0/1058414281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bc40cca20 msgr2=0x7f1bc4159e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:02.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.012+0000 7f1bcbfff700 1 --2- 172.21.15.27:0/1058414281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bc40cca20 0x7f1bc4159e80 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc009e90 tx=0x7f1bbc009780 comp rx=0 tx=0).stop 2024-01-11T09:27:02.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.012+0000 7f1bcbfff700 1 -- 172.21.15.27:0/1058414281 shutdown_connections 2024-01-11T09:27:02.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.012+0000 7f1bcbfff700 1 --2- 172.21.15.27:0/1058414281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bc40cca20 0x7f1bc4159e80 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.012+0000 7f1bcbfff700 1 -- 172.21.15.27:0/1058414281 >> 172.21.15.27:0/1058414281 conn(0x7f1bc40ce3e0 msgr2=0x7f1bc40ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:02.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.012+0000 7f1bcbfff700 1 -- 172.21.15.27:0/1058414281 shutdown_connections 2024-01-11T09:27:02.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.012+0000 7f1bcbfff700 1 -- 172.21.15.27:0/1058414281 wait complete. 2024-01-11T09:27:02.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.014+0000 7f1bcbfff700 1 Processor -- start 2024-01-11T09:27:02.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.015+0000 7f1bcbfff700 1 -- start start 2024-01-11T09:27:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.016+0000 7f1bcbfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1bc40ca3e0 0x7f1bc415fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.016+0000 7f1bcbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bc40ca7b0 0x7f1bc40b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.016+0000 7f1bcbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bc40cca20 0x7f1bc4147f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.016+0000 7f1bcbfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc4149190 con 0x7f1bc40ca7b0 2024-01-11T09:27:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.016+0000 7f1bcbfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc40b1880 con 0x7f1bc40cca20 2024-01-11T09:27:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.016+0000 7f1bcbfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc40b1b60 con 0x7f1bc40ca3e0 2024-01-11T09:27:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.016+0000 7f1bc959a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bc40ca7b0 0x7f1bc40b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.016+0000 7f1bc9d9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1bc40ca3e0 0x7f1bc415fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.016+0000 7f1bca59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bc40cca20 0x7f1bc4147f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.016+0000 7f1bc959a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bc40ca7b0 0x7f1bc40b1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39686/0 (socket says 172.21.15.27:39686) 2024-01-11T09:27:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.016+0000 7f1bc9d9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1bc40ca3e0 0x7f1bc415fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:56572/0 (socket says 172.21.15.27:56572) 2024-01-11T09:27:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.016+0000 7f1bc959a700 1 -- 172.21.15.27:0/3136234327 learned_addr learned my addr 172.21.15.27:0/3136234327 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.017+0000 7f1bc959a700 1 -- 172.21.15.27:0/3136234327 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1bc40ca3e0 msgr2=0x7f1bc415fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.017+0000 7f1bc959a700 1 --2- 172.21.15.27:0/3136234327 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1bc40ca3e0 0x7f1bc415fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.017+0000 7f1bc959a700 1 -- 172.21.15.27:0/3136234327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bc40cca20 msgr2=0x7f1bc4147f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:02.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.017+0000 7f1bc959a700 1 --2- 172.21.15.27:0/3136234327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bc40cca20 0x7f1bc4147f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.017+0000 7f1bc959a700 1 -- 172.21.15.27:0/3136234327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bbc009a20 con 0x7f1bc40ca7b0 2024-01-11T09:27:02.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.017+0000 7f1bca59c700 1 --2- 172.21.15.27:0/3136234327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bc40cca20 0x7f1bc4147f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:27:02.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.017+0000 7f1bc959a700 1 --2- 172.21.15.27:0/3136234327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bc40ca7b0 0x7f1bc40b1180 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f1bb400c7c0 tx=0x7f1bb4007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:02.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.017+0000 7f1bb9ffb700 1 -- 172.21.15.27:0/3136234327 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb4005cd0 con 0x7f1bc40ca7b0 2024-01-11T09:27:02.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.017+0000 7f1bb9ffb700 1 -- 172.21.15.27:0/3136234327 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb4005e30 con 0x7f1bc40ca7b0 2024-01-11T09:27:02.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.017+0000 7f1bb9ffb700 1 -- 172.21.15.27:0/3136234327 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb4007320 con 0x7f1bc40ca7b0 2024-01-11T09:27:02.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.017+0000 7f1bcbfff700 1 -- 172.21.15.27:0/3136234327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc40b1e40 con 0x7f1bc40ca7b0 2024-01-11T09:27:02.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.018+0000 7f1bcbfff700 1 -- 172.21.15.27:0/3136234327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc4148470 con 0x7f1bc40ca7b0 2024-01-11T09:27:02.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.019+0000 7f1bcbfff700 1 -- 172.21.15.27:0/3136234327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc40b2080 con 0x7f1bc40ca7b0 2024-01-11T09:27:02.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.019+0000 7f1bb9ffb700 1 -- 172.21.15.27:0/3136234327 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 95) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f1bb401b3b0 con 0x7f1bc40ca7b0 2024-01-11T09:27:02.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.020+0000 7f1bb9ffb700 1 --2- 172.21.15.27:0/3136234327 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1ba8067050 0x7f1ba8069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:02.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.020+0000 7f1bb9ffb700 1 -- 172.21.15.27:0/3136234327 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f1bb4083f20 con 0x7f1bc40ca7b0 2024-01-11T09:27:02.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.021+0000 7f1bc9d9b700 1 -- 172.21.15.27:0/3136234327 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1ba8067050 msgr2=0x7f1ba8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:02.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.021+0000 7f1bc9d9b700 1 --2- 172.21.15.27:0/3136234327 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1ba8067050 0x7f1ba8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:27:02.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.024+0000 7f1bb9ffb700 1 -- 172.21.15.27:0/3136234327 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1bc40b2080 con 0x7f1bc40ca7b0 2024-01-11T09:27:02.119 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:02.123+0000 7f8450c391c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:27:02.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.222+0000 7f1bc9d9b700 1 -- 172.21.15.27:0/3136234327 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1ba8067050 msgr2=0x7f1ba8069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:02.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.222+0000 7f1bc9d9b700 1 --2- 172.21.15.27:0/3136234327 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1ba8067050 0x7f1ba8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:27:02.272 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:02.275+0000 7f6e5d36f1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:27:02.289 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:02.293+0000 7f8450c391c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:27:02.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.297+0000 7f1bcbfff700 1 -- 172.21.15.27:0/3136234327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bc40b2080 con 0x7f1bc40ca7b0 2024-01-11T09:27:02.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.298+0000 7f1bb9ffb700 1 -- 172.21.15.27:0/3136234327 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) v1 ==== 74+0+15771 (secure 0 0 0) 0x7f1bc40b2080 con 0x7f1bc40ca7b0 2024-01-11T09:27:02.295 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:27:02.295 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":168,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:26:57.806860+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:26:39.008224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-01-11T09:26:40.186962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/2366220214":"2024-01-12T09:26:57.806813+0000","172.21.15.77:6837/41381":"2024-01-12T09:26:57.806813+0000","172.21.15.77:6836/41381":"2024-01-12T09:26:57.806813+0000","172.21.15.77:0/4172230033":"2024-01-12T09:26:57.806813+0000","172.21.15.77:0/1155148094":"2024-01-12T09:26:57.806813+0000","172.21.15.77:0/3578968849":"2024-01-12T09:26:57.806813+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:27:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.301+0000 7f1bcbfff700 1 -- 172.21.15.27:0/3136234327 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1ba8067050 msgr2=0x7f1ba8069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:27:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.301+0000 7f1bcbfff700 1 --2- 172.21.15.27:0/3136234327 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1ba8067050 0x7f1ba8069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.301+0000 7f1bcbfff700 1 -- 172.21.15.27:0/3136234327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bc40ca7b0 msgr2=0x7f1bc40b1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.301+0000 7f1bcbfff700 1 --2- 172.21.15.27:0/3136234327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bc40ca7b0 0x7f1bc40b1180 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f1bb400c7c0 tx=0x7f1bb4007e00 comp rx=0 tx=0).stop 2024-01-11T09:27:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.301+0000 7f1bcbfff700 1 -- 172.21.15.27:0/3136234327 shutdown_connections 2024-01-11T09:27:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.301+0000 7f1bcbfff700 1 --2- 172.21.15.27:0/3136234327 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f1ba8067050 0x7f1ba8069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.301+0000 7f1bcbfff700 1 --2- 172.21.15.27:0/3136234327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bc40cca20 0x7f1bc4147f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.301+0000 7f1bcbfff700 1 --2- 172.21.15.27:0/3136234327 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1bc40ca3e0 0x7f1bc415fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.301+0000 7f1bcbfff700 1 --2- 172.21.15.27:0/3136234327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bc40ca7b0 0x7f1bc40b1180 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.301+0000 7f1bcbfff700 1 -- 172.21.15.27:0/3136234327 >> 172.21.15.27:0/3136234327 conn(0x7f1bc40ce3e0 msgr2=0x7f1bc406ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.301+0000 7f1bcbfff700 1 -- 172.21.15.27:0/3136234327 shutdown_connections 2024-01-11T09:27:02.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.301+0000 7f1bcbfff700 1 -- 172.21.15.27:0/3136234327 wait complete. 2024-01-11T09:27:02.316 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T09:26:39.008224+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '166', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T09:26:40.186962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '167', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T09:27:02.317 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T09:27:02.464 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:02.464+0000 7f6e5d36f1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:27:02.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.530+0000 7f8fd91ff700 1 Processor -- start 2024-01-11T09:27:02.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.530+0000 7f8fd91ff700 1 -- start start 2024-01-11T09:27:02.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.532+0000 7f8fd91ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8fd40ccd70 0x7f8fd40cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:02.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.532+0000 7f8fd91ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd40cf850 con 0x7f8fd415a1b0 2024-01-11T09:27:02.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.532+0000 7f8fd91ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd40cec40 con 0x7f8fd415ae70 2024-01-11T09:27:02.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.532+0000 7f8fd91ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd415ac90 con 0x7f8fd40ccd70 2024-01-11T09:27:02.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.532+0000 7f8fd259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8fd40ccd70 0x7f8fd40cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:02.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.532+0000 7f8fd2d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8fd415a1b0 0x7f8fd415a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46820/0 (socket says 172.21.15.27:46820) 2024-01-11T09:27:02.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.532+0000 7f8fd2d9d700 1 -- 172.21.15.27:0/462141989 learned_addr learned my addr 172.21.15.27:0/462141989 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.533+0000 7f8fd259c700 1 -- 172.21.15.27:0/462141989 >> v1:172.21.15.77:6789/0 conn(0x7f8fd415ae70 legacy=0x7f8fd415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.533+0000 7f8fd259c700 1 -- 172.21.15.27:0/462141989 >> v1:172.21.15.27:6789/0 conn(0x7f8fd415a1b0 legacy=0x7f8fd415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.533+0000 7f8fd259c700 1 -- 172.21.15.27:0/462141989 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd40cef20 con 0x7f8fd40ccd70 2024-01-11T09:27:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.533+0000 7f8fd1d9b700 1 -- 172.21.15.27:0/462141989 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161472133 0 0) 0x7f8fd40cf850 con 0x7f8fd415a1b0 2024-01-11T09:27:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.533+0000 7f8fd259c700 1 --2- 172.21.15.27:0/462141989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd40ccd70 0x7f8fd40cd210 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80098e0 tx=0x7f8fc80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bde374c048a2aa server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.533+0000 7f8fd1d9b700 1 -- 172.21.15.27:0/462141989 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8009e80 con 0x7f8fd40ccd70 2024-01-11T09:27:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.533+0000 7f8fd1d9b700 1 -- 172.21.15.27:0/462141989 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc800b860 con 0x7f8fd40ccd70 2024-01-11T09:27:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.533+0000 7f8fd1d9b700 1 -- 172.21.15.27:0/462141989 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc800bbe0 con 0x7f8fd40ccd70 2024-01-11T09:27:02.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.534+0000 7f8fd91ff700 1 -- 172.21.15.27:0/462141989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd40ccd70 msgr2=0x7f8fd40cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:02.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.534+0000 7f8fd91ff700 1 --2- 172.21.15.27:0/462141989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd40ccd70 0x7f8fd40cd210 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80098e0 tx=0x7f8fc80091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.534+0000 7f8fd91ff700 1 -- 172.21.15.27:0/462141989 shutdown_connections 2024-01-11T09:27:02.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.534+0000 7f8fd91ff700 1 --2- 172.21.15.27:0/462141989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd40ccd70 0x7f8fd40cd210 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.534+0000 7f8fd91ff700 1 -- 172.21.15.27:0/462141989 >> 172.21.15.27:0/462141989 conn(0x7f8fd406ed40 msgr2=0x7f8fd406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:02.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.534+0000 7f8fd91ff700 1 -- 172.21.15.27:0/462141989 shutdown_connections 2024-01-11T09:27:02.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.534+0000 7f8fd91ff700 1 -- 172.21.15.27:0/462141989 wait complete. 2024-01-11T09:27:02.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.536+0000 7f8fd91ff700 1 Processor -- start 2024-01-11T09:27:02.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.536+0000 7f8fd91ff700 1 -- start start 2024-01-11T09:27:02.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.537+0000 7f8fd91ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd40ccd70 0x7f8fd4147f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:02.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.537+0000 7f8fd91ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fd415a1b0 0x7f8fd40c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:02.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.537+0000 7f8fd91ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd415ae70 0x7f8fd40c6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.537+0000 7f8fd91ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40b3380 con 0x7f8fd415ae70 2024-01-11T09:27:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.537+0000 7f8fd91ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40b3590 con 0x7f8fd40ccd70 2024-01-11T09:27:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.537+0000 7f8fd91ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40b3870 con 0x7f8fd415a1b0 2024-01-11T09:27:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.537+0000 7f8fd359e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd415ae70 0x7f8fd40c6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.537+0000 7f8fd359e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd415ae70 0x7f8fd40c6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39698/0 (socket says 172.21.15.27:39698) 2024-01-11T09:27:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.537+0000 7f8fd359e700 1 -- 172.21.15.27:0/1546557373 learned_addr learned my addr 172.21.15.27:0/1546557373 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.537+0000 7f8fd259c700 1 --2- 172.21.15.27:0/1546557373 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fd415a1b0 0x7f8fd40c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:02.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.537+0000 7f8fd2d9d700 1 --2- 172.21.15.27:0/1546557373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd40ccd70 0x7f8fd4147f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:02.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.538+0000 7f8fd359e700 1 -- 172.21.15.27:0/1546557373 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fd415a1b0 msgr2=0x7f8fd40c9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:02.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.538+0000 7f8fd359e700 1 --2- 172.21.15.27:0/1546557373 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fd415a1b0 0x7f8fd40c9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.538+0000 7f8fd359e700 1 -- 172.21.15.27:0/1546557373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd40ccd70 msgr2=0x7f8fd4147f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:02.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.538+0000 7f8fd359e700 1 --2- 172.21.15.27:0/1546557373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd40ccd70 0x7f8fd4147f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.538+0000 7f8fd359e700 1 -- 172.21.15.27:0/1546557373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc8009420 con 0x7f8fd415ae70 2024-01-11T09:27:02.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.538+0000 7f8fd359e700 1 --2- 172.21.15.27:0/1546557373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd415ae70 0x7f8fd40c6a40 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc00c8e0 tx=0x7f8fcc001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:02.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.538+0000 7f8fbaffd700 1 -- 172.21.15.27:0/1546557373 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc012630 con 0x7f8fd415ae70 2024-01-11T09:27:02.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.538+0000 7f8fbaffd700 1 -- 172.21.15.27:0/1546557373 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fcc012790 con 0x7f8fd415ae70 2024-01-11T09:27:02.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.538+0000 7f8fd91ff700 1 -- 172.21.15.27:0/1546557373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd40b3b50 con 0x7f8fd415ae70 2024-01-11T09:27:02.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.538+0000 7f8fbaffd700 1 -- 172.21.15.27:0/1546557373 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc012b50 con 0x7f8fd415ae70 2024-01-11T09:27:02.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.538+0000 7f8fd91ff700 1 -- 172.21.15.27:0/1546557373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd40b4110 con 0x7f8fd415ae70 2024-01-11T09:27:02.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.539+0000 7f8fd91ff700 1 -- 172.21.15.27:0/1546557373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd40b4300 con 0x7f8fd415ae70 2024-01-11T09:27:02.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.540+0000 7f8fbaffd700 1 -- 172.21.15.27:0/1546557373 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 95) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f8fcc01a3a0 con 0x7f8fd415ae70 2024-01-11T09:27:02.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.541+0000 7f8fbaffd700 1 --2- 172.21.15.27:0/1546557373 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8fc0067020 0x7f8fc00694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:02.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.541+0000 7f8fbaffd700 1 -- 172.21.15.27:0/1546557373 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f8fcc0849f0 con 0x7f8fd415ae70 2024-01-11T09:27:02.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.541+0000 7f8fd2d9d700 1 -- 172.21.15.27:0/1546557373 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8fc0067020 msgr2=0x7f8fc00694d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:02.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.541+0000 7f8fd2d9d700 1 --2- 172.21.15.27:0/1546557373 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8fc0067020 0x7f8fc00694d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:27:02.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.545+0000 7f8fbaffd700 1 -- 172.21.15.27:0/1546557373 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8fcc051970 con 0x7f8fd415ae70 2024-01-11T09:27:02.706 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:02.709+0000 7f8450c391c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:27:02.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.742+0000 7f8fd2d9d700 1 -- 172.21.15.27:0/1546557373 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8fc0067020 msgr2=0x7f8fc00694d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:02.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.742+0000 7f8fd2d9d700 1 --2- 172.21.15.27:0/1546557373 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8fc0067020 0x7f8fc00694d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:27:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.749+0000 7f8fd91ff700 1 -- 172.21.15.27:0/1546557373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8fd404c3c0 con 0x7f8fd415ae70 2024-01-11T09:27:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.750+0000 7f8fbaffd700 1 -- 172.21.15.27:0/1546557373 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v168) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8fcc054f90 con 0x7f8fd415ae70 2024-01-11T09:27:02.746 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T09:27:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.752+0000 7f8fd91ff700 1 -- 172.21.15.27:0/1546557373 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8fc0067020 msgr2=0x7f8fc00694d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:27:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.752+0000 7f8fd91ff700 1 --2- 172.21.15.27:0/1546557373 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8fc0067020 0x7f8fc00694d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.752+0000 7f8fd91ff700 1 -- 172.21.15.27:0/1546557373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd415ae70 msgr2=0x7f8fd40c6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.752+0000 7f8fd91ff700 1 --2- 172.21.15.27:0/1546557373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd415ae70 0x7f8fd40c6a40 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc00c8e0 tx=0x7f8fcc001710 comp rx=0 tx=0).stop 2024-01-11T09:27:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.752+0000 7f8fd91ff700 1 -- 172.21.15.27:0/1546557373 shutdown_connections 2024-01-11T09:27:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.752+0000 7f8fd91ff700 1 --2- 172.21.15.27:0/1546557373 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8fc0067020 0x7f8fc00694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.752+0000 7f8fd91ff700 1 --2- 172.21.15.27:0/1546557373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd415ae70 0x7f8fd40c6a40 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.752+0000 7f8fd91ff700 1 --2- 172.21.15.27:0/1546557373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd40ccd70 0x7f8fd4147f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.752+0000 7f8fd91ff700 1 --2- 172.21.15.27:0/1546557373 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fd415a1b0 0x7f8fd40c9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.752+0000 7f8fd91ff700 1 -- 172.21.15.27:0/1546557373 >> 172.21.15.27:0/1546557373 conn(0x7f8fd406ed40 msgr2=0x7f8fd4156240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.753+0000 7f8fd91ff700 1 -- 172.21.15.27:0/1546557373 shutdown_connections 2024-01-11T09:27:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.753+0000 7f8fd91ff700 1 -- 172.21.15.27:0/1546557373 wait complete. 2024-01-11T09:27:02.768 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T09:27:02.917 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:02.920+0000 7f6e5d36f1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:27:02.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.986+0000 7f12de7ff700 1 Processor -- start 2024-01-11T09:27:02.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.986+0000 7f12de7ff700 1 -- start start 2024-01-11T09:27:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.987+0000 7f12de7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f12d815a1b0 0x7f12d815a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.987+0000 7f12de7ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f12d815ae70 0x7f12d815b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.988+0000 7f12de7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12d80cf850 con 0x7f12d80ccd70 2024-01-11T09:27:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.988+0000 7f12de7ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12d80cec40 con 0x7f12d815ae70 2024-01-11T09:27:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.988+0000 7f12de7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12d80cef20 con 0x7f12d815a1b0 2024-01-11T09:27:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.988+0000 7f12dcd9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f12d815ae70 0x7f12d815b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.988+0000 7f12d77fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f12d80ccd70 0x7f12d80cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46832/0 (socket says 172.21.15.27:46832) 2024-01-11T09:27:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.988+0000 7f12d77fe700 1 -- 172.21.15.27:0/1381833258 learned_addr learned my addr 172.21.15.27:0/1381833258 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.988+0000 7f12d7fff700 1 --2- 172.21.15.27:0/1381833258 >> v2:172.21.15.77:3300/0 conn(0x7f12d815a1b0 0x7f12d815a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.988+0000 7f12dcd9c700 1 -- 172.21.15.27:0/1381833258 >> v2:172.21.15.77:3300/0 conn(0x7f12d815a1b0 msgr2=0x7f12d815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.988+0000 7f12dcd9c700 1 --2- 172.21.15.27:0/1381833258 >> v2:172.21.15.77:3300/0 conn(0x7f12d815a1b0 0x7f12d815a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.988+0000 7f12dcd9c700 1 -- 172.21.15.27:0/1381833258 >> v1:172.21.15.27:6789/0 conn(0x7f12d80ccd70 legacy=0x7f12d80cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.988+0000 7f12dcd9c700 1 -- 172.21.15.27:0/1381833258 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d80cf200 con 0x7f12d815ae70 2024-01-11T09:27:02.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.988+0000 7f12d6ffd700 1 -- 172.21.15.27:0/1381833258 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156894708 0 0) 0x7f12d80cf850 con 0x7f12d80ccd70 2024-01-11T09:27:02.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.989+0000 7f12dcd9c700 1 --2- 172.21.15.27:0/1381833258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12d815ae70 0x7f12d815b2a0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f12d80d16f0 tx=0x7f12d0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2cf38f99a10d8d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:02.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.989+0000 7f12d6ffd700 1 -- 172.21.15.27:0/1381833258 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d000e070 con 0x7f12d815ae70 2024-01-11T09:27:02.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.989+0000 7f12d6ffd700 1 -- 172.21.15.27:0/1381833258 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d000bb60 con 0x7f12d815ae70 2024-01-11T09:27:02.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.989+0000 7f12d6ffd700 1 -- 172.21.15.27:0/1381833258 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d0012430 con 0x7f12d815ae70 2024-01-11T09:27:02.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.990+0000 7f12de7ff700 1 -- 172.21.15.27:0/1381833258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12d815ae70 msgr2=0x7f12d815b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:02.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.990+0000 7f12de7ff700 1 --2- 172.21.15.27:0/1381833258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12d815ae70 0x7f12d815b2a0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f12d80d16f0 tx=0x7f12d0009780 comp rx=0 tx=0).stop 2024-01-11T09:27:02.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.990+0000 7f12de7ff700 1 -- 172.21.15.27:0/1381833258 shutdown_connections 2024-01-11T09:27:02.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.990+0000 7f12de7ff700 1 --2- 172.21.15.27:0/1381833258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12d815ae70 0x7f12d815b2a0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.990+0000 7f12de7ff700 1 --2- 172.21.15.27:0/1381833258 >> v2:172.21.15.77:3300/0 conn(0x7f12d815a1b0 0x7f12d815a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.990+0000 7f12de7ff700 1 -- 172.21.15.27:0/1381833258 >> 172.21.15.27:0/1381833258 conn(0x7f12d806ed40 msgr2=0x7f12d806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:02.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.990+0000 7f12de7ff700 1 -- 172.21.15.27:0/1381833258 shutdown_connections 2024-01-11T09:27:02.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.990+0000 7f12de7ff700 1 -- 172.21.15.27:0/1381833258 wait complete. 2024-01-11T09:27:02.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.992+0000 7f12de7ff700 1 Processor -- start 2024-01-11T09:27:02.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.992+0000 7f12de7ff700 1 -- start start 2024-01-11T09:27:02.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.993+0000 7f12de7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12d80ccd70 0x7f12d814afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:02.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.993+0000 7f12de7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12d815a1b0 0x7f12d80ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:02.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.993+0000 7f12de7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12d815ae70 0x7f12d80b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:02.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.993+0000 7f12de7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d81697c0 con 0x7f12d80ccd70 2024-01-11T09:27:02.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.993+0000 7f12de7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d816a5e0 con 0x7f12d815a1b0 2024-01-11T09:27:02.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.993+0000 7f12de7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12d81699d0 con 0x7f12d815ae70 2024-01-11T09:27:02.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.993+0000 7f12d7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12d80ccd70 0x7f12d814afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:02.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.993+0000 7f12d7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12d80ccd70 0x7f12d814afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39720/0 (socket says 172.21.15.27:39720) 2024-01-11T09:27:02.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.993+0000 7f12d7fff700 1 -- 172.21.15.27:0/3858647258 learned_addr learned my addr 172.21.15.27:0/3858647258 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:02.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.993+0000 7f12d77fe700 1 --2- 172.21.15.27:0/3858647258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12d815a1b0 0x7f12d80ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:02.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.993+0000 7f12dcd9c700 1 --2- 172.21.15.27:0/3858647258 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12d815ae70 0x7f12d80b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:02.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.993+0000 7f12d7fff700 1 -- 172.21.15.27:0/3858647258 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12d815ae70 msgr2=0x7f12d80b54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:02.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.993+0000 7f12d7fff700 1 --2- 172.21.15.27:0/3858647258 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12d815ae70 0x7f12d80b54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.993+0000 7f12d7fff700 1 -- 172.21.15.27:0/3858647258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12d815a1b0 msgr2=0x7f12d80ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:02.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.993+0000 7f12d7fff700 1 --2- 172.21.15.27:0/3858647258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12d815a1b0 0x7f12d80ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:02.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.994+0000 7f12d7fff700 1 -- 172.21.15.27:0/3858647258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d0009a20 con 0x7f12d80ccd70 2024-01-11T09:27:02.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.994+0000 7f12d7fff700 1 --2- 172.21.15.27:0/3858647258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12d80ccd70 0x7f12d814afb0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f12c800ec80 tx=0x7f12c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:02.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.994+0000 7f12bffff700 1 -- 172.21.15.27:0/3858647258 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c800c850 con 0x7f12d80ccd70 2024-01-11T09:27:02.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.994+0000 7f12bffff700 1 -- 172.21.15.27:0/3858647258 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c800c9b0 con 0x7f12d80ccd70 2024-01-11T09:27:02.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.994+0000 7f12de7ff700 1 -- 172.21.15.27:0/3858647258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d8169cb0 con 0x7f12d80ccd70 2024-01-11T09:27:02.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.994+0000 7f12bffff700 1 -- 172.21.15.27:0/3858647258 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c800cd50 con 0x7f12d80ccd70 2024-01-11T09:27:02.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.994+0000 7f12de7ff700 1 -- 172.21.15.27:0/3858647258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d816a270 con 0x7f12d80ccd70 2024-01-11T09:27:02.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.995+0000 7f12de7ff700 1 -- 172.21.15.27:0/3858647258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d804c3c0 con 0x7f12d80ccd70 2024-01-11T09:27:02.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.996+0000 7f12bffff700 1 -- 172.21.15.27:0/3858647258 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 95) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f12c8013680 con 0x7f12d80ccd70 2024-01-11T09:27:02.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.997+0000 7f12bffff700 1 --2- 172.21.15.27:0/3858647258 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f12c4067050 0x7f12c4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:02.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.997+0000 7f12bffff700 1 -- 172.21.15.27:0/3858647258 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f12c8084a70 con 0x7f12d80ccd70 2024-01-11T09:27:02.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.997+0000 7f12d77fe700 1 -- 172.21.15.27:0/3858647258 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f12c4067050 msgr2=0x7f12c4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:02.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:02.998+0000 7f12d77fe700 1 --2- 172.21.15.27:0/3858647258 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f12c4067050 0x7f12c4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:27:03.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.004+0000 7f12bffff700 1 -- 172.21.15.27:0/3858647258 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f12c80519f0 con 0x7f12d80ccd70 2024-01-11T09:27:03.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.199+0000 7f12d77fe700 1 -- 172.21.15.27:0/3858647258 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f12c4067050 msgr2=0x7f12c4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:03.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.199+0000 7f12d77fe700 1 --2- 172.21.15.27:0/3858647258 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f12c4067050 0x7f12c4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:27:03.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.208+0000 7f12de7ff700 1 -- 172.21.15.27:0/3858647258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f12d8159970 con 0x7f12d80ccd70 2024-01-11T09:27:03.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.208+0000 7f12bffff700 1 -- 172.21.15.27:0/3858647258 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v168) v1 ==== 104+0+11 (secure 0 0 0) 0x7f12c8055010 con 0x7f12d80ccd70 2024-01-11T09:27:03.205 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T09:27:03.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.210+0000 7f12de7ff700 1 -- 172.21.15.27:0/3858647258 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f12c4067050 msgr2=0x7f12c4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:27:03.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.210+0000 7f12de7ff700 1 --2- 172.21.15.27:0/3858647258 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f12c4067050 0x7f12c4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.210+0000 7f12de7ff700 1 -- 172.21.15.27:0/3858647258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12d80ccd70 msgr2=0x7f12d814afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:03.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.210+0000 7f12de7ff700 1 --2- 172.21.15.27:0/3858647258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12d80ccd70 0x7f12d814afb0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f12c800ec80 tx=0x7f12c80091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.210+0000 7f12de7ff700 1 -- 172.21.15.27:0/3858647258 shutdown_connections 2024-01-11T09:27:03.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.210+0000 7f12de7ff700 1 --2- 172.21.15.27:0/3858647258 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f12c4067050 0x7f12c4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.210+0000 7f12de7ff700 1 --2- 172.21.15.27:0/3858647258 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f12d815ae70 0x7f12d80b54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.210+0000 7f12de7ff700 1 --2- 172.21.15.27:0/3858647258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12d80ccd70 0x7f12d814afb0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.210+0000 7f12de7ff700 1 --2- 172.21.15.27:0/3858647258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12d815a1b0 0x7f12d80ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.210+0000 7f12de7ff700 1 -- 172.21.15.27:0/3858647258 >> 172.21.15.27:0/3858647258 conn(0x7f12d806ed40 msgr2=0x7f12d80c6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:03.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.211+0000 7f12de7ff700 1 -- 172.21.15.27:0/3858647258 shutdown_connections 2024-01-11T09:27:03.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.211+0000 7f12de7ff700 1 -- 172.21.15.27:0/3858647258 wait complete. 2024-01-11T09:27:03.226 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T09:27:03.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.447+0000 7fefc6bff700 1 Processor -- start 2024-01-11T09:27:03.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.448+0000 7fefc6bff700 1 -- start start 2024-01-11T09:27:03.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.449+0000 7fefc6bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fefc0142dc0 0x7fefc00cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:03.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.449+0000 7fefc6bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefc0143190 0x7fefc00cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:03.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.449+0000 7fefc6bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc015d320 con 0x7fefc00cc020 2024-01-11T09:27:03.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.449+0000 7fefc6bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefc015df30 con 0x7fefc0142dc0 2024-01-11T09:27:03.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.449+0000 7fefc6bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc015d590 con 0x7fefc0143190 2024-01-11T09:27:03.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.449+0000 7fefbffff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefc0143190 0x7fefc00cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:03.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.449+0000 7fefc519c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fefc00cc020 0x7fefc00cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:38982/0 (socket says 172.21.15.27:38982) 2024-01-11T09:27:03.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.449+0000 7fefc519c700 1 -- 172.21.15.27:0/3247612506 learned_addr learned my addr 172.21.15.27:0/3247612506 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:03.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.449+0000 7fefc499b700 1 --2- 172.21.15.27:0/3247612506 >> v2:172.21.15.77:3300/0 conn(0x7fefc0142dc0 0x7fefc00cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:03.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.450+0000 7fefbf7fe700 1 -- 172.21.15.27:0/3247612506 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267429620 0 0) 0x7fefc015d320 con 0x7fefc00cc020 2024-01-11T09:27:03.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.450+0000 7fefbf7fe700 1 -- 172.21.15.27:0/3247612506 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefb0003570 con 0x7fefc00cc020 2024-01-11T09:27:03.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.450+0000 7fefbffff700 1 -- 172.21.15.27:0/3247612506 >> v2:172.21.15.77:3300/0 conn(0x7fefc0142dc0 msgr2=0x7fefc00cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:03.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.450+0000 7fefbffff700 1 --2- 172.21.15.27:0/3247612506 >> v2:172.21.15.77:3300/0 conn(0x7fefc0142dc0 0x7fefc00cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.450+0000 7fefbffff700 1 -- 172.21.15.27:0/3247612506 >> v1:172.21.15.77:6789/0 conn(0x7fefc00cc020 legacy=0x7fefc00cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:03.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.450+0000 7fefbffff700 1 -- 172.21.15.27:0/3247612506 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc015d870 con 0x7fefc0143190 2024-01-11T09:27:03.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.450+0000 7fefc499b700 1 --2- 172.21.15.27:0/3247612506 >> v2:172.21.15.77:3300/0 conn(0x7fefc0142dc0 0x7fefc00cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:27:03.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.450+0000 7fefbffff700 1 --2- 172.21.15.27:0/3247612506 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefc0143190 0x7fefc00cb2e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fefac0098e0 tx=0x7fefac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c88329f414655ab2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:03.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.450+0000 7fefbf7fe700 1 -- 172.21.15.27:0/3247612506 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac011630 con 0x7fefc0143190 2024-01-11T09:27:03.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.450+0000 7fefbf7fe700 1 -- 172.21.15.27:0/3247612506 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefac011790 con 0x7fefc0143190 2024-01-11T09:27:03.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.450+0000 7fefbf7fe700 1 -- 172.21.15.27:0/3247612506 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac011b30 con 0x7fefc0143190 2024-01-11T09:27:03.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.451+0000 7fefc6bff700 1 -- 172.21.15.27:0/3247612506 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefc0143190 msgr2=0x7fefc00cb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:03.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.451+0000 7fefc6bff700 1 --2- 172.21.15.27:0/3247612506 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefc0143190 0x7fefc00cb2e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fefac0098e0 tx=0x7fefac0091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.451+0000 7fefc6bff700 1 -- 172.21.15.27:0/3247612506 shutdown_connections 2024-01-11T09:27:03.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.451+0000 7fefc6bff700 1 --2- 172.21.15.27:0/3247612506 >> v2:172.21.15.77:3300/0 conn(0x7fefc0142dc0 0x7fefc00cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.451+0000 7fefc6bff700 1 --2- 172.21.15.27:0/3247612506 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefc0143190 0x7fefc00cb2e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.451+0000 7fefc6bff700 1 -- 172.21.15.27:0/3247612506 >> 172.21.15.27:0/3247612506 conn(0x7fefc0063ca0 msgr2=0x7fefc006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:03.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.451+0000 7fefc6bff700 1 -- 172.21.15.27:0/3247612506 shutdown_connections 2024-01-11T09:27:03.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.451+0000 7fefc6bff700 1 -- 172.21.15.27:0/3247612506 wait complete. 2024-01-11T09:27:03.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.453+0000 7fefc6bff700 1 Processor -- start 2024-01-11T09:27:03.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.453+0000 7fefc6bff700 1 -- start start 2024-01-11T09:27:03.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.454+0000 7fefc6bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefc00cc020 0x7fefc016cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:03.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.454+0000 7fefc6bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefc0142dc0 0x7fefc00c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:03.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.454+0000 7fefc6bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefc0143190 0x7fefc015a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:03.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.454+0000 7fefc6bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc016b1b0 con 0x7fefc0142dc0 2024-01-11T09:27:03.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.454+0000 7fefc6bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc00b2150 con 0x7fefc00cc020 2024-01-11T09:27:03.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.454+0000 7fefc6bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc016a7f0 con 0x7fefc0143190 2024-01-11T09:27:03.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.454+0000 7fefbffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefc0142dc0 0x7fefc00c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:03.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.454+0000 7fefc499b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefc00cc020 0x7fefc016cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.454+0000 7fefc499b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefc00cc020 0x7fefc016cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:33080/0 (socket says 172.21.15.27:33080) 2024-01-11T09:27:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.454+0000 7fefc499b700 1 -- 172.21.15.27:0/1345762390 learned_addr learned my addr 172.21.15.27:0/1345762390 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.454+0000 7fefbffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefc0142dc0 0x7fefc00c5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39724/0 (socket says 172.21.15.27:39724) 2024-01-11T09:27:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.454+0000 7fefc519c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefc0143190 0x7fefc015a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.455+0000 7fefc499b700 1 -- 172.21.15.27:0/1345762390 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefc0143190 msgr2=0x7fefc015a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.455+0000 7fefc499b700 1 --2- 172.21.15.27:0/1345762390 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefc0143190 0x7fefc015a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.455+0000 7fefc499b700 1 -- 172.21.15.27:0/1345762390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefc0142dc0 msgr2=0x7fefc00c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.455+0000 7fefc499b700 1 --2- 172.21.15.27:0/1345762390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefc0142dc0 0x7fefc00c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.455+0000 7fefc499b700 1 -- 172.21.15.27:0/1345762390 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb4009420 con 0x7fefc00cc020 2024-01-11T09:27:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.455+0000 7fefc499b700 1 --2- 172.21.15.27:0/1345762390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefc00cc020 0x7fefc016cca0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fefb400ec80 tx=0x7fefb40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.455+0000 7fefbcff9700 1 -- 172.21.15.27:0/1345762390 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb400c920 con 0x7fefc00cc020 2024-01-11T09:27:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.455+0000 7fefbcff9700 1 -- 172.21.15.27:0/1345762390 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb400ca80 con 0x7fefc00cc020 2024-01-11T09:27:03.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.455+0000 7fefbcff9700 1 -- 172.21.15.27:0/1345762390 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb400ce00 con 0x7fefc00cc020 2024-01-11T09:27:03.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.455+0000 7fefc6bff700 1 -- 172.21.15.27:0/1345762390 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefac009420 con 0x7fefc00cc020 2024-01-11T09:27:03.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.455+0000 7fefc6bff700 1 -- 172.21.15.27:0/1345762390 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc016adb0 con 0x7fefc00cc020 2024-01-11T09:27:03.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.456+0000 7fefc6bff700 1 -- 172.21.15.27:0/1345762390 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc004c3c0 con 0x7fefc00cc020 2024-01-11T09:27:03.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.457+0000 7fefbcff9700 1 -- 172.21.15.27:0/1345762390 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 95) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fefb4013680 con 0x7fefc00cc020 2024-01-11T09:27:03.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.458+0000 7fefbcff9700 1 --2- 172.21.15.27:0/1345762390 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fefa4067050 0x7fefa4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:03.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.458+0000 7fefbcff9700 1 -- 172.21.15.27:0/1345762390 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6019+0+0 (secure 0 0 0) 0x7fefb4084be0 con 0x7fefc00cc020 2024-01-11T09:27:03.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.458+0000 7fefbffff700 1 -- 172.21.15.27:0/1345762390 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fefa4067050 msgr2=0x7fefa4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:03.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.458+0000 7fefbffff700 1 --2- 172.21.15.27:0/1345762390 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fefa4067050 0x7fefa4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:27:03.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.463+0000 7fefbcff9700 1 -- 172.21.15.27:0/1345762390 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fefb4051be0 con 0x7fefc00cc020 2024-01-11T09:27:03.488 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:03.492+0000 7f8450c391c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:27:03.577 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:03.581+0000 7f8450c391c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:27:03.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.659+0000 7fefbffff700 1 -- 172.21.15.27:0/1345762390 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fefa4067050 msgr2=0x7fefa4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:03.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.659+0000 7fefbffff700 1 --2- 172.21.15.27:0/1345762390 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fefa4067050 0x7fefa4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:27:03.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.670+0000 7fefc6bff700 1 -- 172.21.15.27:0/1345762390 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fefc015db50 con 0x7fefc00cc020 2024-01-11T09:27:03.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.671+0000 7fefbcff9700 1 -- 172.21.15.27:0/1345762390 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v168) v1 ==== 100+0+11 (secure 0 0 0) 0x7fefb4055200 con 0x7fefc00cc020 2024-01-11T09:27:03.667 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T09:27:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.673+0000 7fefc6bff700 1 -- 172.21.15.27:0/1345762390 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fefa4067050 msgr2=0x7fefa4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:27:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.673+0000 7fefc6bff700 1 --2- 172.21.15.27:0/1345762390 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fefa4067050 0x7fefa4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.674+0000 7fefc6bff700 1 -- 172.21.15.27:0/1345762390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefc00cc020 msgr2=0x7fefc016cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.674+0000 7fefc6bff700 1 --2- 172.21.15.27:0/1345762390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefc00cc020 0x7fefc016cca0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fefb400ec80 tx=0x7fefb40091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.674+0000 7fefc6bff700 1 -- 172.21.15.27:0/1345762390 shutdown_connections 2024-01-11T09:27:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.674+0000 7fefc6bff700 1 --2- 172.21.15.27:0/1345762390 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fefa4067050 0x7fefa4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.674+0000 7fefc6bff700 1 --2- 172.21.15.27:0/1345762390 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefc0143190 0x7fefc015a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.674+0000 7fefc6bff700 1 --2- 172.21.15.27:0/1345762390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefc00cc020 0x7fefc016cca0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.674+0000 7fefc6bff700 1 --2- 172.21.15.27:0/1345762390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefc0142dc0 0x7fefc00c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.674+0000 7fefc6bff700 1 -- 172.21.15.27:0/1345762390 >> 172.21.15.27:0/1345762390 conn(0x7fefc0063ca0 msgr2=0x7fefc015b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.674+0000 7fefc6bff700 1 -- 172.21.15.27:0/1345762390 shutdown_connections 2024-01-11T09:27:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.674+0000 7fefc6bff700 1 -- 172.21.15.27:0/1345762390 wait complete. 2024-01-11T09:27:03.691 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:27:03.733 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:03.737+0000 7f8450c391c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:27:03.780 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:03.783+0000 7f6e5d36f1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:27:03.821 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:03.825+0000 7f8450c391c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:27:03.876 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:03.880+0000 7f6e5d36f1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:27:03.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.897+0000 7f01a81ff700 1 Processor -- start 2024-01-11T09:27:03.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.897+0000 7f01a81ff700 1 -- start start 2024-01-11T09:27:03.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.899+0000 7f01a81ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01a00b2b90 0x7f01a0159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:03.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.899+0000 7f01a81ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f01a00b07b0 0x7f01a0159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:03.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.899+0000 7f01a81ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a00cd730 con 0x7f01a00b03e0 2024-01-11T09:27:03.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.899+0000 7f01a81ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a00ccb20 con 0x7f01a00b07b0 2024-01-11T09:27:03.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.899+0000 7f01a81ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a00cce00 con 0x7f01a00b2b90 2024-01-11T09:27:03.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.899+0000 7f01a679c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f01a00b07b0 0x7f01a0159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:03.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.899+0000 7f01a579a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01a00b2b90 0x7f01a0159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:03.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.899+0000 7f01a5f9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f01a00b03e0 0x7f01a00b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:37036/0 (socket says 172.21.15.27:37036) 2024-01-11T09:27:03.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.899+0000 7f01a5f9b700 1 -- 172.21.15.27:0/839232342 learned_addr learned my addr 172.21.15.27:0/839232342 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:03.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.899+0000 7f01a4f99700 1 -- 172.21.15.27:0/839232342 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3398251925 0 0) 0x7f01a00cd730 con 0x7f01a00b03e0 2024-01-11T09:27:03.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.900+0000 7f01a4f99700 1 -- 172.21.15.27:0/839232342 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f018c003570 con 0x7f01a00b03e0 2024-01-11T09:27:03.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.900+0000 7f01a679c700 1 -- 172.21.15.27:0/839232342 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01a00b2b90 msgr2=0x7f01a0159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:03.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.900+0000 7f01a679c700 1 --2- 172.21.15.27:0/839232342 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01a00b2b90 0x7f01a0159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.900+0000 7f01a679c700 1 -- 172.21.15.27:0/839232342 >> v1:172.21.15.77:6789/0 conn(0x7f01a00b03e0 legacy=0x7f01a00b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:03.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.900+0000 7f01a679c700 1 -- 172.21.15.27:0/839232342 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a00cd0e0 con 0x7f01a00b07b0 2024-01-11T09:27:03.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.900+0000 7f01a579a700 1 --2- 172.21.15.27:0/839232342 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01a00b2b90 0x7f01a0159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:27:03.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.900+0000 7f01a679c700 1 --2- 172.21.15.27:0/839232342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a00b07b0 0x7f01a0159740 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f0198009e60 tx=0x7f0198009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a62340f2906f9534 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:03.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.900+0000 7f01a4f99700 1 -- 172.21.15.27:0/839232342 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019800e070 con 0x7f01a00b07b0 2024-01-11T09:27:03.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.900+0000 7f01a4f99700 1 -- 172.21.15.27:0/839232342 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019800bab0 con 0x7f01a00b07b0 2024-01-11T09:27:03.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.900+0000 7f01a4f99700 1 -- 172.21.15.27:0/839232342 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019800be70 con 0x7f01a00b07b0 2024-01-11T09:27:03.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.901+0000 7f01a81ff700 1 -- 172.21.15.27:0/839232342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a00b07b0 msgr2=0x7f01a0159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:03.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.901+0000 7f01a81ff700 1 --2- 172.21.15.27:0/839232342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a00b07b0 0x7f01a0159740 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f0198009e60 tx=0x7f0198009780 comp rx=0 tx=0).stop 2024-01-11T09:27:03.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.901+0000 7f01a81ff700 1 -- 172.21.15.27:0/839232342 shutdown_connections 2024-01-11T09:27:03.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.901+0000 7f01a81ff700 1 --2- 172.21.15.27:0/839232342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a00b07b0 0x7f01a0159740 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.901+0000 7f01a81ff700 1 --2- 172.21.15.27:0/839232342 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01a00b2b90 0x7f01a0159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.901+0000 7f01a81ff700 1 -- 172.21.15.27:0/839232342 >> 172.21.15.27:0/839232342 conn(0x7f01a006ed40 msgr2=0x7f01a006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:03.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.901+0000 7f01a81ff700 1 -- 172.21.15.27:0/839232342 shutdown_connections 2024-01-11T09:27:03.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.901+0000 7f01a81ff700 1 -- 172.21.15.27:0/839232342 wait complete. 2024-01-11T09:27:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.902+0000 7f01a81ff700 1 Processor -- start 2024-01-11T09:27:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.902+0000 7f01a81ff700 1 -- start start 2024-01-11T09:27:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.903+0000 7f01a81ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01a00b03e0 0x7f01a0158c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.903+0000 7f01a81ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a00b07b0 0x7f01a015bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.903+0000 7f01a81ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01a00b2b90 0x7f01a015c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.903+0000 7f01a81ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a00c7350 con 0x7f01a00b07b0 2024-01-11T09:27:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.903+0000 7f01a81ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a00c5af0 con 0x7f01a00b03e0 2024-01-11T09:27:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.903+0000 7f01a81ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a01591b0 con 0x7f01a00b2b90 2024-01-11T09:27:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.904+0000 7f01a5f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01a00b03e0 0x7f01a0158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.904+0000 7f01a5f9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01a00b03e0 0x7f01a0158c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:46126/0 (socket says 172.21.15.27:46126) 2024-01-11T09:27:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.904+0000 7f01a579a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a00b07b0 0x7f01a015bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:03.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.904+0000 7f01a5f9b700 1 -- 172.21.15.27:0/1277597742 learned_addr learned my addr 172.21.15.27:0/1277597742 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:03.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.904+0000 7f01a679c700 1 --2- 172.21.15.27:0/1277597742 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01a00b2b90 0x7f01a015c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:03.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.904+0000 7f01a5f9b700 1 -- 172.21.15.27:0/1277597742 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01a00b2b90 msgr2=0x7f01a015c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:03.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.904+0000 7f01a5f9b700 1 --2- 172.21.15.27:0/1277597742 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01a00b2b90 0x7f01a015c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.904+0000 7f01a5f9b700 1 -- 172.21.15.27:0/1277597742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a00b07b0 msgr2=0x7f01a015bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:03.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.904+0000 7f01a5f9b700 1 --2- 172.21.15.27:0/1277597742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a00b07b0 0x7f01a015bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:03.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.904+0000 7f01a5f9b700 1 -- 172.21.15.27:0/1277597742 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0190009420 con 0x7f01a00b03e0 2024-01-11T09:27:03.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.905+0000 7f01a5f9b700 1 --2- 172.21.15.27:0/1277597742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01a00b03e0 0x7f01a0158c70 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f019c00c7c0 tx=0x7f019c007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:03.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.905+0000 7f0195ffb700 1 -- 172.21.15.27:0/1277597742 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c005b30 con 0x7f01a00b03e0 2024-01-11T09:27:03.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.905+0000 7f0195ffb700 1 -- 172.21.15.27:0/1277597742 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c005c90 con 0x7f01a00b03e0 2024-01-11T09:27:03.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.905+0000 7f0195ffb700 1 -- 172.21.15.27:0/1277597742 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c007320 con 0x7f01a00b03e0 2024-01-11T09:27:03.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.905+0000 7f01a81ff700 1 -- 172.21.15.27:0/1277597742 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0198009a20 con 0x7f01a00b03e0 2024-01-11T09:27:03.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.905+0000 7f01a81ff700 1 -- 172.21.15.27:0/1277597742 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a00c54a0 con 0x7f01a00b03e0 2024-01-11T09:27:03.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.906+0000 7f01a81ff700 1 -- 172.21.15.27:0/1277597742 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a00718d0 con 0x7f01a00b03e0 2024-01-11T09:27:03.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.907+0000 7f0195ffb700 1 -- 172.21.15.27:0/1277597742 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 95) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f019c01b3b0 con 0x7f01a00b03e0 2024-01-11T09:27:03.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.909+0000 7f0195ffb700 1 --2- 172.21.15.27:0/1277597742 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0184067080 0x7f0184069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:03.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.909+0000 7f0195ffb700 1 -- 172.21.15.27:0/1277597742 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f019c083db0 con 0x7f01a00b03e0 2024-01-11T09:27:03.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.909+0000 7f01a579a700 1 -- 172.21.15.27:0/1277597742 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0184067080 msgr2=0x7f0184069530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:03.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.909+0000 7f01a579a700 1 --2- 172.21.15.27:0/1277597742 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0184067080 0x7f0184069530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:27:03.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:03.912+0000 7f0195ffb700 1 -- 172.21.15.27:0/1277597742 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f019c088070 con 0x7f01a00b03e0 2024-01-11T09:27:04.041 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:04.045+0000 7f8450c391c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:27:04.049 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:04.053+0000 7f6e5d36f1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:27:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.110+0000 7f01a579a700 1 -- 172.21.15.27:0/1277597742 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0184067080 msgr2=0x7f0184069530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.110+0000 7f01a579a700 1 --2- 172.21.15.27:0/1277597742 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0184067080 0x7f0184069530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:27:04.138 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:04.142+0000 7f8450c391c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:27:04.143 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:04.146+0000 7f6e5d36f1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:27:04.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.162+0000 7f01a81ff700 1 -- 172.21.15.27:0/1277597742 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01a00cd3c0 con 0x7f01a00b03e0 2024-01-11T09:27:04.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.163+0000 7f0195ffb700 1 -- 172.21.15.27:0/1277597742 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4598 (secure 0 0 0) 0x7f019c017070 con 0x7f01a00b03e0 2024-01-11T09:27:04.159 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:27:04.159 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":86,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7700,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/191611944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":191611944},{"type":"v1","addr":"172.21.15.27:6835","nonce":191611944}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":8734,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/3566978772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3566978772},{"type":"v1","addr":"172.21.15.77:6834","nonce":3566978772}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":8737,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1168169536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1168169536},{"type":"v1","addr":"172.21.15.77:6835","nonce":1168169536}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:26:41.190629+0000","modified":"2024-01-11T09:26:44.707534+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9093},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9093":{"gid":9093,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/3544429684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3544429684},{"type":"v1","addr":"172.21.15.27:6834","nonce":3544429684}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2024-01-11T09:27:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.165+0000 7f01a81ff700 1 -- 172.21.15.27:0/1277597742 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0184067080 msgr2=0x7f0184069530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:27:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.165+0000 7f01a81ff700 1 --2- 172.21.15.27:0/1277597742 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0184067080 0x7f0184069530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.165+0000 7f01a81ff700 1 -- 172.21.15.27:0/1277597742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01a00b03e0 msgr2=0x7f01a0158c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.165+0000 7f01a81ff700 1 --2- 172.21.15.27:0/1277597742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01a00b03e0 0x7f01a0158c70 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f019c00c7c0 tx=0x7f019c007e00 comp rx=0 tx=0).stop 2024-01-11T09:27:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.165+0000 7f01a81ff700 1 -- 172.21.15.27:0/1277597742 shutdown_connections 2024-01-11T09:27:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.165+0000 7f01a81ff700 1 --2- 172.21.15.27:0/1277597742 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0184067080 0x7f0184069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.165+0000 7f01a81ff700 1 --2- 172.21.15.27:0/1277597742 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01a00b2b90 0x7f01a015c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.165+0000 7f01a81ff700 1 --2- 172.21.15.27:0/1277597742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01a00b03e0 0x7f01a0158c70 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.165+0000 7f01a81ff700 1 --2- 172.21.15.27:0/1277597742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a00b07b0 0x7f01a015bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.165+0000 7f01a81ff700 1 -- 172.21.15.27:0/1277597742 >> 172.21.15.27:0/1277597742 conn(0x7f01a006ed40 msgr2=0x7f01a01594a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:04.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.165+0000 7f01a81ff700 1 -- 172.21.15.27:0/1277597742 shutdown_connections 2024-01-11T09:27:04.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.166+0000 7f01a81ff700 1 -- 172.21.15.27:0/1277597742 wait complete. 2024-01-11T09:27:04.163 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 86 2024-01-11T09:27:04.180 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:27:04.234 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:04.238+0000 7f8450c391c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:27:04.329 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:04.333+0000 7f8450c391c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:27:04.376 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:04.379+0000 7f6e5d36f1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:27:04.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.388+0000 7f4e0e5ff700 1 Processor -- start 2024-01-11T09:27:04.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.388+0000 7f4e0e5ff700 1 -- start start 2024-01-11T09:27:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.390+0000 7f4e0e5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e08159650 0x7f4e0806bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.390+0000 7f4e0e5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4e080b2730 0x7f4e080b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.390+0000 7f4e0e5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0806fba0 con 0x7f4e08159a20 2024-01-11T09:27:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.390+0000 7f4e0e5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e0806e700 con 0x7f4e080b2730 2024-01-11T09:27:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.390+0000 7f4e0e5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0806c360 con 0x7f4e08159650 2024-01-11T09:27:04.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.390+0000 7f4e07fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e08159650 0x7f4e0806bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:04.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.390+0000 7f4e0cb9c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4e080b2730 0x7f4e080b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.390+0000 7f4e077fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4e08159a20 0x7f4e080b0c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:37052/0 (socket says 172.21.15.27:37052) 2024-01-11T09:27:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.390+0000 7f4e077fe700 1 -- 172.21.15.27:0/3274791480 learned_addr learned my addr 172.21.15.27:0/3274791480 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.391+0000 7f4e0cb9c700 1 -- 172.21.15.27:0/3274791480 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e08159650 msgr2=0x7f4e0806bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.391+0000 7f4e0cb9c700 1 --2- 172.21.15.27:0/3274791480 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e08159650 0x7f4e0806bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.391+0000 7f4e0cb9c700 1 -- 172.21.15.27:0/3274791480 >> v1:172.21.15.77:6789/0 conn(0x7f4e08159a20 legacy=0x7f4e080b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.391+0000 7f4e0cb9c700 1 -- 172.21.15.27:0/3274791480 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0806c640 con 0x7f4e080b2730 2024-01-11T09:27:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.391+0000 7f4e06ffd700 1 -- 172.21.15.27:0/3274791480 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940111550 0 0) 0x7f4e0806fba0 con 0x7f4e08159a20 2024-01-11T09:27:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.391+0000 7f4e0cb9c700 1 --2- 172.21.15.27:0/3274791480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e080b2730 0x7f4e080b2bd0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f4e00009e60 tx=0x7f4e00009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38838bee26a23bb6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.391+0000 7f4e06ffd700 1 -- 172.21.15.27:0/3274791480 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0000e070 con 0x7f4e080b2730 2024-01-11T09:27:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.391+0000 7f4e06ffd700 1 -- 172.21.15.27:0/3274791480 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e0000bb60 con 0x7f4e080b2730 2024-01-11T09:27:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.391+0000 7f4e06ffd700 1 -- 172.21.15.27:0/3274791480 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e00012430 con 0x7f4e080b2730 2024-01-11T09:27:04.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.392+0000 7f4e0e5ff700 1 -- 172.21.15.27:0/3274791480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e080b2730 msgr2=0x7f4e080b2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:04.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.392+0000 7f4e0e5ff700 1 --2- 172.21.15.27:0/3274791480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e080b2730 0x7f4e080b2bd0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f4e00009e60 tx=0x7f4e00009780 comp rx=0 tx=0).stop 2024-01-11T09:27:04.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.392+0000 7f4e0e5ff700 1 -- 172.21.15.27:0/3274791480 shutdown_connections 2024-01-11T09:27:04.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.392+0000 7f4e0e5ff700 1 --2- 172.21.15.27:0/3274791480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e080b2730 0x7f4e080b2bd0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.392+0000 7f4e0e5ff700 1 --2- 172.21.15.27:0/3274791480 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e08159650 0x7f4e0806bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.392+0000 7f4e0e5ff700 1 -- 172.21.15.27:0/3274791480 >> 172.21.15.27:0/3274791480 conn(0x7f4e0806b200 msgr2=0x7f4e080aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.392+0000 7f4e0e5ff700 1 -- 172.21.15.27:0/3274791480 shutdown_connections 2024-01-11T09:27:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.392+0000 7f4e0e5ff700 1 -- 172.21.15.27:0/3274791480 wait complete. 2024-01-11T09:27:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.394+0000 7f4e0e5ff700 1 Processor -- start 2024-01-11T09:27:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.394+0000 7f4e0e5ff700 1 -- start start 2024-01-11T09:27:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.395+0000 7f4e0e5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e080b2730 0x7f4e080add60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.395+0000 7f4e0e5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e08159650 0x7f4e0815f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.395+0000 7f4e0e5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e08159a20 0x7f4e0815f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:04.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.395+0000 7f4e0e5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e080aa890 con 0x7f4e08159a20 2024-01-11T09:27:04.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.395+0000 7f4e0e5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e08153f80 con 0x7f4e080b2730 2024-01-11T09:27:04.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.395+0000 7f4e0e5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e080a9c80 con 0x7f4e08159650 2024-01-11T09:27:04.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.395+0000 7f4e07fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e080b2730 0x7f4e080add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:04.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.395+0000 7f4e07fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e080b2730 0x7f4e080add60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:46144/0 (socket says 172.21.15.27:46144) 2024-01-11T09:27:04.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.395+0000 7f4e07fff700 1 -- 172.21.15.27:0/2218071186 learned_addr learned my addr 172.21.15.27:0/2218071186 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:04.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.395+0000 7f4e077fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e08159650 0x7f4e0815f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:04.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.395+0000 7f4e0cb9c700 1 --2- 172.21.15.27:0/2218071186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e08159a20 0x7f4e0815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:04.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.395+0000 7f4e07fff700 1 -- 172.21.15.27:0/2218071186 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e08159650 msgr2=0x7f4e0815f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:04.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.395+0000 7f4e07fff700 1 --2- 172.21.15.27:0/2218071186 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e08159650 0x7f4e0815f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.395+0000 7f4e07fff700 1 -- 172.21.15.27:0/2218071186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e08159a20 msgr2=0x7f4e0815f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:04.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.395+0000 7f4e07fff700 1 --2- 172.21.15.27:0/2218071186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e08159a20 0x7f4e0815f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.395+0000 7f4e07fff700 1 -- 172.21.15.27:0/2218071186 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e00009a20 con 0x7f4e080b2730 2024-01-11T09:27:04.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.396+0000 7f4e07fff700 1 --2- 172.21.15.27:0/2218071186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e080b2730 0x7f4e080add60 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f4df800ec80 tx=0x7f4df80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:04.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.396+0000 7f4deffff700 1 -- 172.21.15.27:0/2218071186 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df800c850 con 0x7f4e080b2730 2024-01-11T09:27:04.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.396+0000 7f4deffff700 1 -- 172.21.15.27:0/2218071186 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df800c9b0 con 0x7f4e080b2730 2024-01-11T09:27:04.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.396+0000 7f4deffff700 1 -- 172.21.15.27:0/2218071186 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df800cd50 con 0x7f4e080b2730 2024-01-11T09:27:04.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.396+0000 7f4e0e5ff700 1 -- 172.21.15.27:0/2218071186 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e080a9f60 con 0x7f4e080b2730 2024-01-11T09:27:04.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.396+0000 7f4e0e5ff700 1 -- 172.21.15.27:0/2218071186 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e080aa520 con 0x7f4e080b2730 2024-01-11T09:27:04.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.397+0000 7f4e0e5ff700 1 -- 172.21.15.27:0/2218071186 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e080ab9a0 con 0x7f4e080b2730 2024-01-11T09:27:04.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.398+0000 7f4deffff700 1 -- 172.21.15.27:0/2218071186 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 95) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f4df8013680 con 0x7f4e080b2730 2024-01-11T09:27:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.399+0000 7f4deffff700 1 --2- 172.21.15.27:0/2218071186 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4df4067050 0x7f4df4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.399+0000 7f4deffff700 1 -- 172.21.15.27:0/2218071186 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f4df8084a60 con 0x7f4e080b2730 2024-01-11T09:27:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.400+0000 7f4e077fe700 1 -- 172.21.15.27:0/2218071186 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4df4067050 msgr2=0x7f4df4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.400+0000 7f4e077fe700 1 --2- 172.21.15.27:0/2218071186 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4df4067050 0x7f4df4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:27:04.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.404+0000 7f4deffff700 1 -- 172.21.15.27:0/2218071186 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4df8051a80 con 0x7f4e080b2730 2024-01-11T09:27:04.482 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:04.485+0000 7f6e5d36f1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:27:04.519 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:04.522+0000 7f8450c391c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:27:04.584 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:04.588+0000 7f6e5d36f1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:27:04.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.601+0000 7f4e077fe700 1 -- 172.21.15.27:0/2218071186 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4df4067050 msgr2=0x7f4df4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:04.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.601+0000 7f4e077fe700 1 --2- 172.21.15.27:0/2218071186 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4df4067050 0x7f4df4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:27:04.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.611+0000 7f4e0e5ff700 1 -- 172.21.15.27:0/2218071186 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e080b3260 con 0x7f4e080b2730 2024-01-11T09:27:04.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.612+0000 7f4deffff700 1 -- 172.21.15.27:0/2218071186 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) v1 ==== 74+0+15771 (secure 0 0 0) 0x7f4df80550a0 con 0x7f4e080b2730 2024-01-11T09:27:04.611 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:27:04.611 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":168,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:26:57.806860+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:26:39.008224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-01-11T09:26:40.186962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/2366220214":"2024-01-12T09:26:57.806813+0000","172.21.15.77:6837/41381":"2024-01-12T09:26:57.806813+0000","172.21.15.77:6836/41381":"2024-01-12T09:26:57.806813+0000","172.21.15.77:0/4172230033":"2024-01-12T09:26:57.806813+0000","172.21.15.77:0/1155148094":"2024-01-12T09:26:57.806813+0000","172.21.15.77:0/3578968849":"2024-01-12T09:26:57.806813+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:27:04.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.615+0000 7f4e0e5ff700 1 -- 172.21.15.27:0/2218071186 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4df4067050 msgr2=0x7f4df4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:27:04.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.615+0000 7f4e0e5ff700 1 --2- 172.21.15.27:0/2218071186 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4df4067050 0x7f4df4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.615+0000 7f4e0e5ff700 1 -- 172.21.15.27:0/2218071186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e080b2730 msgr2=0x7f4e080add60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:04.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.615+0000 7f4e0e5ff700 1 --2- 172.21.15.27:0/2218071186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e080b2730 0x7f4e080add60 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f4df800ec80 tx=0x7f4df80091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.615+0000 7f4e0e5ff700 1 -- 172.21.15.27:0/2218071186 shutdown_connections 2024-01-11T09:27:04.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.615+0000 7f4e0e5ff700 1 --2- 172.21.15.27:0/2218071186 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4df4067050 0x7f4df4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.615+0000 7f4e0e5ff700 1 --2- 172.21.15.27:0/2218071186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e08159a20 0x7f4e0815f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.615+0000 7f4e0e5ff700 1 --2- 172.21.15.27:0/2218071186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e080b2730 0x7f4e080add60 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.615+0000 7f4e0e5ff700 1 --2- 172.21.15.27:0/2218071186 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e08159650 0x7f4e0815f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.615+0000 7f4e0e5ff700 1 -- 172.21.15.27:0/2218071186 >> 172.21.15.27:0/2218071186 conn(0x7f4e0806b200 msgr2=0x7f4e08067610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:04.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.615+0000 7f4e0e5ff700 1 -- 172.21.15.27:0/2218071186 shutdown_connections 2024-01-11T09:27:04.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.615+0000 7f4e0e5ff700 1 -- 172.21.15.27:0/2218071186 wait complete. 2024-01-11T09:27:04.616 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:04.620+0000 7f8450c391c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:27:04.630 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-11T09:27:04.630 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:27:04.691 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:04.695+0000 7f6e5d36f1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:27:04.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.839+0000 7f254a5ff700 1 Processor -- start 2024-01-11T09:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.839+0000 7f254a5ff700 1 -- start start 2024-01-11T09:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.841+0000 7f254a5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f25440ca7b0 0x7f25440cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.841+0000 7f254a5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254415d640 con 0x7f2544159e80 2024-01-11T09:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.841+0000 7f254a5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254415a5d0 con 0x7f25440ca3e0 2024-01-11T09:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.841+0000 7f254a5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f254415ae70 con 0x7f25440ca7b0 2024-01-11T09:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.842+0000 7f25437fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f25440ca7b0 0x7f25440cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.842+0000 7f2548b9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2544159e80 0x7f254415dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40442/0 (socket says 172.21.15.27:40442) 2024-01-11T09:27:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.842+0000 7f2548b9c700 1 -- 172.21.15.27:0/3480980683 learned_addr learned my addr 172.21.15.27:0/3480980683 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.842+0000 7f25437fe700 1 -- 172.21.15.27:0/3480980683 >> v1:172.21.15.77:6789/0 conn(0x7f25440ca3e0 legacy=0x7f25440c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.842+0000 7f25437fe700 1 -- 172.21.15.27:0/3480980683 >> v1:172.21.15.27:6789/0 conn(0x7f2544159e80 legacy=0x7f254415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.842+0000 7f25437fe700 1 -- 172.21.15.27:0/3480980683 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254415a8b0 con 0x7f25440ca7b0 2024-01-11T09:27:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.842+0000 7f25437fe700 1 --2- 172.21.15.27:0/3480980683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25440ca7b0 0x7f25440cca20 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f2538009910 tx=0x7f25380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0dfe34b720abd11 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.843+0000 7f2542ffd700 1 -- 172.21.15.27:0/3480980683 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253800b820 con 0x7f25440ca7b0 2024-01-11T09:27:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.843+0000 7f2542ffd700 1 -- 172.21.15.27:0/3480980683 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253800b980 con 0x7f25440ca7b0 2024-01-11T09:27:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.843+0000 7f2542ffd700 1 -- 172.21.15.27:0/3480980683 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253800bd20 con 0x7f25440ca7b0 2024-01-11T09:27:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.843+0000 7f254a5ff700 1 -- 172.21.15.27:0/3480980683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25440ca7b0 msgr2=0x7f25440cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.843+0000 7f254a5ff700 1 --2- 172.21.15.27:0/3480980683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25440ca7b0 0x7f25440cca20 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f2538009910 tx=0x7f25380091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.844+0000 7f254a5ff700 1 -- 172.21.15.27:0/3480980683 shutdown_connections 2024-01-11T09:27:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.844+0000 7f254a5ff700 1 --2- 172.21.15.27:0/3480980683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25440ca7b0 0x7f25440cca20 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.844+0000 7f254a5ff700 1 -- 172.21.15.27:0/3480980683 >> 172.21.15.27:0/3480980683 conn(0x7f25440ce3e0 msgr2=0x7f25440ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.844+0000 7f254a5ff700 1 -- 172.21.15.27:0/3480980683 shutdown_connections 2024-01-11T09:27:04.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.844+0000 7f254a5ff700 1 -- 172.21.15.27:0/3480980683 wait complete. 2024-01-11T09:27:04.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.845+0000 7f254a5ff700 1 Processor -- start 2024-01-11T09:27:04.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.845+0000 7f254a5ff700 1 -- start start 2024-01-11T09:27:04.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.846+0000 7f254a5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25440ca3e0 0x7f25441584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:04.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.846+0000 7f254a5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25440ca7b0 0x7f254415bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:04.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.846+0000 7f254a5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2544159e80 0x7f254414fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:04.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.846+0000 7f254a5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2544158d90 con 0x7f25440ca3e0 2024-01-11T09:27:04.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.846+0000 7f254a5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25440b25e0 con 0x7f25440ca7b0 2024-01-11T09:27:04.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.846+0000 7f254a5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25440b2880 con 0x7f2544159e80 2024-01-11T09:27:04.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.847+0000 7f2543fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25440ca3e0 0x7f25441584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:04.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.847+0000 7f2543fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25440ca3e0 0x7f25441584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36042/0 (socket says 172.21.15.27:36042) 2024-01-11T09:27:04.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.847+0000 7f2543fff700 1 -- 172.21.15.27:0/4245250045 learned_addr learned my addr 172.21.15.27:0/4245250045 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:04.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.847+0000 7f2548b9c700 1 --2- 172.21.15.27:0/4245250045 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2544159e80 0x7f254414fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:04.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.847+0000 7f25437fe700 1 --2- 172.21.15.27:0/4245250045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25440ca7b0 0x7f254415bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:04.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.847+0000 7f2543fff700 1 -- 172.21.15.27:0/4245250045 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2544159e80 msgr2=0x7f254414fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:04.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.847+0000 7f2543fff700 1 --2- 172.21.15.27:0/4245250045 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2544159e80 0x7f254414fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.847+0000 7f2543fff700 1 -- 172.21.15.27:0/4245250045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25440ca7b0 msgr2=0x7f254415bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:04.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.847+0000 7f2543fff700 1 --2- 172.21.15.27:0/4245250045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25440ca7b0 0x7f254415bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:04.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.847+0000 7f2543fff700 1 -- 172.21.15.27:0/4245250045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2538009420 con 0x7f25440ca3e0 2024-01-11T09:27:04.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.847+0000 7f2543fff700 1 --2- 172.21.15.27:0/4245250045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25440ca3e0 0x7f25441584b0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f253400c4e0 tx=0x7f253400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:04.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.847+0000 7f252bfff700 1 -- 172.21.15.27:0/4245250045 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2534011590 con 0x7f25440ca3e0 2024-01-11T09:27:04.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.848+0000 7f252bfff700 1 -- 172.21.15.27:0/4245250045 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2534011710 con 0x7f25440ca3e0 2024-01-11T09:27:04.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.848+0000 7f252bfff700 1 -- 172.21.15.27:0/4245250045 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2534011ad0 con 0x7f25440ca3e0 2024-01-11T09:27:04.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.848+0000 7f254a5ff700 1 -- 172.21.15.27:0/4245250045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25440b2b60 con 0x7f25440ca3e0 2024-01-11T09:27:04.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.848+0000 7f254a5ff700 1 -- 172.21.15.27:0/4245250045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25440b3120 con 0x7f25440ca3e0 2024-01-11T09:27:04.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.849+0000 7f254a5ff700 1 -- 172.21.15.27:0/4245250045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25440718d0 con 0x7f25440ca3e0 2024-01-11T09:27:04.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.850+0000 7f252bfff700 1 -- 172.21.15.27:0/4245250045 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 95) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f253401b3b0 con 0x7f25440ca3e0 2024-01-11T09:27:04.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.851+0000 7f252bfff700 1 --2- 172.21.15.27:0/4245250045 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2530067050 0x7f2530069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:04.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.851+0000 7f252bfff700 1 -- 172.21.15.27:0/4245250045 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f2534083c40 con 0x7f25440ca3e0 2024-01-11T09:27:04.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.851+0000 7f25437fe700 1 -- 172.21.15.27:0/4245250045 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2530067050 msgr2=0x7f2530069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:04.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.851+0000 7f25437fe700 1 --2- 172.21.15.27:0/4245250045 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2530067050 0x7f2530069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:27:04.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:04.854+0000 7f252bfff700 1 -- 172.21.15.27:0/4245250045 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2534050bc0 con 0x7f25440ca3e0 2024-01-11T09:27:04.931 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:04.908+0000 7f6e5d36f1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:27:04.963 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:04.967+0000 7f8450c391c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:27:05.010 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:05.014+0000 7f6e5d36f1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:27:05.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.052+0000 7f25437fe700 1 -- 172.21.15.27:0/4245250045 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2530067050 msgr2=0x7f2530069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:05.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.052+0000 7f25437fe700 1 --2- 172.21.15.27:0/4245250045 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2530067050 0x7f2530069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:27:05.053 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:05.057+0000 7f8450c391c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:27:05.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.104+0000 7f254a5ff700 1 -- 172.21.15.27:0/4245250045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f254406c840 con 0x7f25440ca3e0 2024-01-11T09:27:05.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.105+0000 7f252bfff700 1 -- 172.21.15.27:0/4245250045 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4598 (secure 0 0 0) 0x7f2534017070 con 0x7f25440ca3e0 2024-01-11T09:27:05.102 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:27:05.102 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":86,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7700,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/191611944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":191611944},{"type":"v1","addr":"172.21.15.27:6835","nonce":191611944}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":8734,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/3566978772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3566978772},{"type":"v1","addr":"172.21.15.77:6834","nonce":3566978772}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":8737,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1168169536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1168169536},{"type":"v1","addr":"172.21.15.77:6835","nonce":1168169536}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:26:41.190629+0000","modified":"2024-01-11T09:26:44.707534+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9093},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9093":{"gid":9093,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/3544429684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3544429684},{"type":"v1","addr":"172.21.15.27:6834","nonce":3544429684}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2024-01-11T09:27:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.108+0000 7f254a5ff700 1 -- 172.21.15.27:0/4245250045 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2530067050 msgr2=0x7f2530069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:27:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.108+0000 7f254a5ff700 1 --2- 172.21.15.27:0/4245250045 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2530067050 0x7f2530069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.108+0000 7f254a5ff700 1 -- 172.21.15.27:0/4245250045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25440ca3e0 msgr2=0x7f25441584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.108+0000 7f254a5ff700 1 --2- 172.21.15.27:0/4245250045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25440ca3e0 0x7f25441584b0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f253400c4e0 tx=0x7f253400c7f0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.108+0000 7f254a5ff700 1 -- 172.21.15.27:0/4245250045 shutdown_connections 2024-01-11T09:27:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.108+0000 7f254a5ff700 1 --2- 172.21.15.27:0/4245250045 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2530067050 0x7f2530069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.108+0000 7f254a5ff700 1 --2- 172.21.15.27:0/4245250045 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2544159e80 0x7f254414fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.108+0000 7f254a5ff700 1 --2- 172.21.15.27:0/4245250045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25440ca3e0 0x7f25441584b0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.108+0000 7f254a5ff700 1 --2- 172.21.15.27:0/4245250045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25440ca7b0 0x7f254415bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.108+0000 7f254a5ff700 1 -- 172.21.15.27:0/4245250045 >> 172.21.15.27:0/4245250045 conn(0x7f25440ce3e0 msgr2=0x7f2544069790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.108+0000 7f254a5ff700 1 -- 172.21.15.27:0/4245250045 shutdown_connections 2024-01-11T09:27:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.108+0000 7f254a5ff700 1 -- 172.21.15.27:0/4245250045 wait complete. 2024-01-11T09:27:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 86 2024-01-11T09:27:05.123 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:27:05.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.334+0000 7f0c129ff700 1 Processor -- start 2024-01-11T09:27:05.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.334+0000 7f0c129ff700 1 -- start start 2024-01-11T09:27:05.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.335+0000 7f0c129ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0c0c069d40 0x7f0c0c0b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:05.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.335+0000 7f0c129ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0c157390 con 0x7f0c0c0c8620 2024-01-11T09:27:05.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.335+0000 7f0c129ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0c06a200 con 0x7f0c0c0c89f0 2024-01-11T09:27:05.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.335+0000 7f0c129ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c0c156d40 con 0x7f0c0c069d40 2024-01-11T09:27:05.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.335+0000 7f0c0b7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0c0c069d40 0x7f0c0c0b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:05.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.335+0000 7f0c0bfff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0c0c0c8620 0x7f0c0c0af970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40454/0 (socket says 172.21.15.27:40454) 2024-01-11T09:27:05.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.336+0000 7f0c0bfff700 1 -- 172.21.15.27:0/741993331 learned_addr learned my addr 172.21.15.27:0/741993331 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:05.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.336+0000 7f0c0affd700 1 -- 172.21.15.27:0/741993331 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817483735 0 0) 0x7f0c0c157390 con 0x7f0c0c0c8620 2024-01-11T09:27:05.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.336+0000 7f0c0affd700 1 -- 172.21.15.27:0/741993331 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf4003570 con 0x7f0c0c0c8620 2024-01-11T09:27:05.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.336+0000 7f0c0affd700 1 -- 172.21.15.27:0/741993331 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232402528 0 0) 0x7f0c0c06a200 con 0x7f0c0c0c89f0 2024-01-11T09:27:05.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.336+0000 7f0c0b7fe700 1 -- 172.21.15.27:0/741993331 >> v1:172.21.15.77:6789/0 conn(0x7f0c0c0c89f0 legacy=0x7f0c0c0accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:05.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.336+0000 7f0c0b7fe700 1 -- 172.21.15.27:0/741993331 >> v1:172.21.15.27:6789/0 conn(0x7f0c0c0c8620 legacy=0x7f0c0c0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:05.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.336+0000 7f0c0b7fe700 1 -- 172.21.15.27:0/741993331 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c157020 con 0x7f0c0c069d40 2024-01-11T09:27:05.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.337+0000 7f0c0b7fe700 1 --2- 172.21.15.27:0/741993331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c0c069d40 0x7f0c0c0b02d0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f0c000098e0 tx=0x7f0c000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c5bfdd1257aa656 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:05.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.337+0000 7f0c0affd700 1 -- 172.21.15.27:0/741993331 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0000b640 con 0x7f0c0c069d40 2024-01-11T09:27:05.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.337+0000 7f0c0affd700 1 -- 172.21.15.27:0/741993331 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0000b7a0 con 0x7f0c0c069d40 2024-01-11T09:27:05.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.337+0000 7f0c0affd700 1 -- 172.21.15.27:0/741993331 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0000bb40 con 0x7f0c0c069d40 2024-01-11T09:27:05.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.337+0000 7f0c129ff700 1 -- 172.21.15.27:0/741993331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c0c069d40 msgr2=0x7f0c0c0b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:05.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.337+0000 7f0c129ff700 1 --2- 172.21.15.27:0/741993331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c0c069d40 0x7f0c0c0b02d0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f0c000098e0 tx=0x7f0c000091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.338+0000 7f0c129ff700 1 -- 172.21.15.27:0/741993331 shutdown_connections 2024-01-11T09:27:05.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.338+0000 7f0c129ff700 1 --2- 172.21.15.27:0/741993331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c0c069d40 0x7f0c0c0b02d0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.338+0000 7f0c129ff700 1 -- 172.21.15.27:0/741993331 >> 172.21.15.27:0/741993331 conn(0x7f0c0c0680e0 msgr2=0x7f0c0c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:05.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.338+0000 7f0c129ff700 1 -- 172.21.15.27:0/741993331 shutdown_connections 2024-01-11T09:27:05.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.338+0000 7f0c129ff700 1 -- 172.21.15.27:0/741993331 wait complete. 2024-01-11T09:27:05.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.339+0000 7f0c129ff700 1 Processor -- start 2024-01-11T09:27:05.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.339+0000 7f0c129ff700 1 -- start start 2024-01-11T09:27:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.340+0000 7f0c129ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0c0c069d40 0x7f0c0c16d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.340+0000 7f0c129ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c0c0c8620 0x7f0c0c143af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.340+0000 7f0c129ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c0c0c89f0 0x7f0c0c0c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.340+0000 7f0c129ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c0c6780 con 0x7f0c0c0c89f0 2024-01-11T09:27:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.340+0000 7f0c129ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c16d910 con 0x7f0c0c0c8620 2024-01-11T09:27:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.340+0000 7f0c129ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c16dbf0 con 0x7f0c0c069d40 2024-01-11T09:27:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.341+0000 7f0c0b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c0c0c8620 0x7f0c0c143af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.341+0000 7f0c10f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c0c0c89f0 0x7f0c0c0c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.341+0000 7f0c0bfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0c0c069d40 0x7f0c0c16d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.341+0000 7f0c0b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c0c0c8620 0x7f0c0c143af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:46164/0 (socket says 172.21.15.27:46164) 2024-01-11T09:27:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.341+0000 7f0c0b7fe700 1 -- 172.21.15.27:0/3013325449 learned_addr learned my addr 172.21.15.27:0/3013325449 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.341+0000 7f0c0b7fe700 1 -- 172.21.15.27:0/3013325449 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0c0c069d40 msgr2=0x7f0c0c16d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.341+0000 7f0c0b7fe700 1 --2- 172.21.15.27:0/3013325449 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0c0c069d40 0x7f0c0c16d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.341+0000 7f0c0b7fe700 1 -- 172.21.15.27:0/3013325449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c0c0c89f0 msgr2=0x7f0c0c0c6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:05.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.341+0000 7f0c0b7fe700 1 --2- 172.21.15.27:0/3013325449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c0c0c89f0 0x7f0c0c0c6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.341+0000 7f0c0b7fe700 1 -- 172.21.15.27:0/3013325449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c00009420 con 0x7f0c0c0c8620 2024-01-11T09:27:05.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.341+0000 7f0c10f9c700 1 --2- 172.21.15.27:0/3013325449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c0c0c89f0 0x7f0c0c0c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:27:05.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.341+0000 7f0c0b7fe700 1 --2- 172.21.15.27:0/3013325449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c0c0c8620 0x7f0c0c143af0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c066180 tx=0x7f0c00017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:05.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.342+0000 7f0bf3fff700 1 -- 172.21.15.27:0/3013325449 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0001f070 con 0x7f0c0c0c8620 2024-01-11T09:27:05.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.342+0000 7f0bf3fff700 1 -- 172.21.15.27:0/3013325449 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c00002f70 con 0x7f0c0c0c8620 2024-01-11T09:27:05.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.342+0000 7f0c129ff700 1 -- 172.21.15.27:0/3013325449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c0c0c6410 con 0x7f0c0c0c8620 2024-01-11T09:27:05.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.342+0000 7f0bf3fff700 1 -- 172.21.15.27:0/3013325449 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c00011420 con 0x7f0c0c0c8620 2024-01-11T09:27:05.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.342+0000 7f0c129ff700 1 -- 172.21.15.27:0/3013325449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c0c144030 con 0x7f0c0c0c8620 2024-01-11T09:27:05.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.343+0000 7f0c129ff700 1 -- 172.21.15.27:0/3013325449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0c144220 con 0x7f0c0c0c8620 2024-01-11T09:27:05.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.344+0000 7f0bf3fff700 1 -- 172.21.15.27:0/3013325449 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 95) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f0c00011810 con 0x7f0c0c0c8620 2024-01-11T09:27:05.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.345+0000 7f0bf3fff700 1 --2- 172.21.15.27:0/3013325449 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0bf8067080 0x7f0bf8069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:05.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.345+0000 7f0bf3fff700 1 -- 172.21.15.27:0/3013325449 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f0c000857c0 con 0x7f0c0c0c8620 2024-01-11T09:27:05.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.345+0000 7f0c0bfff700 1 -- 172.21.15.27:0/3013325449 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0bf8067080 msgr2=0x7f0bf8069530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:05.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.345+0000 7f0c0bfff700 1 --2- 172.21.15.27:0/3013325449 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0bf8067080 0x7f0bf8069530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:27:05.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.349+0000 7f0bf3fff700 1 -- 172.21.15.27:0/3013325449 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0c0c144220 con 0x7f0c0c0c8620 2024-01-11T09:27:05.388 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:05.392+0000 7f6e5d36f1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:27:05.486 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:05.489+0000 7f6e5d36f1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:27:05.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.546+0000 7f0c0bfff700 1 -- 172.21.15.27:0/3013325449 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0bf8067080 msgr2=0x7f0bf8069530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:05.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.546+0000 7f0c0bfff700 1 --2- 172.21.15.27:0/3013325449 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0bf8067080 0x7f0bf8069530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:27:05.595 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:05.599+0000 7f8450c391c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:27:05.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.603+0000 7f0c129ff700 1 -- 172.21.15.27:0/3013325449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c0c144220 con 0x7f0c0c0c8620 2024-01-11T09:27:05.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.604+0000 7f0bf3fff700 1 -- 172.21.15.27:0/3013325449 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4598 (secure 0 0 0) 0x7f0c0c144220 con 0x7f0c0c0c8620 2024-01-11T09:27:05.601 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:27:05.601 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":86,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7700,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/191611944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":191611944},{"type":"v1","addr":"172.21.15.27:6835","nonce":191611944}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":8734,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/3566978772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3566978772},{"type":"v1","addr":"172.21.15.77:6834","nonce":3566978772}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":8737,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1168169536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1168169536},{"type":"v1","addr":"172.21.15.77:6835","nonce":1168169536}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:26:41.190629+0000","modified":"2024-01-11T09:26:44.707534+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9093},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9093":{"gid":9093,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/3544429684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3544429684},{"type":"v1","addr":"172.21.15.27:6834","nonce":3544429684}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2024-01-11T09:27:05.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.607+0000 7f0c129ff700 1 -- 172.21.15.27:0/3013325449 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0bf8067080 msgr2=0x7f0bf8069530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:27:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.607+0000 7f0c129ff700 1 --2- 172.21.15.27:0/3013325449 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0bf8067080 0x7f0bf8069530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.607+0000 7f0c129ff700 1 -- 172.21.15.27:0/3013325449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c0c0c8620 msgr2=0x7f0c0c143af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.607+0000 7f0c129ff700 1 --2- 172.21.15.27:0/3013325449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c0c0c8620 0x7f0c0c143af0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c066180 tx=0x7f0c00017000 comp rx=0 tx=0).stop 2024-01-11T09:27:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.607+0000 7f0c129ff700 1 -- 172.21.15.27:0/3013325449 shutdown_connections 2024-01-11T09:27:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.607+0000 7f0c129ff700 1 --2- 172.21.15.27:0/3013325449 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0bf8067080 0x7f0bf8069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.607+0000 7f0c129ff700 1 --2- 172.21.15.27:0/3013325449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c0c0c89f0 0x7f0c0c0c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.607+0000 7f0c129ff700 1 --2- 172.21.15.27:0/3013325449 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0c0c069d40 0x7f0c0c16d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.607+0000 7f0c129ff700 1 --2- 172.21.15.27:0/3013325449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c0c0c8620 0x7f0c0c143af0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.607+0000 7f0c129ff700 1 -- 172.21.15.27:0/3013325449 >> 172.21.15.27:0/3013325449 conn(0x7f0c0c0680e0 msgr2=0x7f0c0c1538a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.607+0000 7f0c129ff700 1 -- 172.21.15.27:0/3013325449 shutdown_connections 2024-01-11T09:27:05.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.607+0000 7f0c129ff700 1 -- 172.21.15.27:0/3013325449 wait complete. 2024-01-11T09:27:05.605 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 86 2024-01-11T09:27:05.621 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:27:05.791 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:05.795+0000 7f8450c391c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:27:05.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.834+0000 7fb0207ff700 1 Processor -- start 2024-01-11T09:27:05.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.834+0000 7fb0207ff700 1 -- start start 2024-01-11T09:27:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.836+0000 7fb0207ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0180cc410 0x7fb018159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.836+0000 7fb0207ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0180cd730 con 0x7fb0180b2b90 2024-01-11T09:27:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.836+0000 7fb0207ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0180ccb20 con 0x7fb0180b03e0 2024-01-11T09:27:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.836+0000 7fb0207ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180cce00 con 0x7fb0180cc410 2024-01-11T09:27:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.836+0000 7fb01dd9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb0180b2b90 0x7fb018159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40458/0 (socket says 172.21.15.27:40458) 2024-01-11T09:27:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.836+0000 7fb01dd9a700 1 -- 172.21.15.27:0/3767897729 learned_addr learned my addr 172.21.15.27:0/3767897729 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.836+0000 7fb01ed9c700 1 --2- 172.21.15.27:0/3767897729 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0180cc410 0x7fb018159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.836+0000 7fb01d599700 1 -- 172.21.15.27:0/3767897729 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127485477 0 0) 0x7fb0180cd730 con 0x7fb0180b2b90 2024-01-11T09:27:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.836+0000 7fb01d599700 1 -- 172.21.15.27:0/3767897729 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb004003570 con 0x7fb0180b2b90 2024-01-11T09:27:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.836+0000 7fb01d599700 1 -- 172.21.15.27:0/3767897729 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478103033 0 0) 0x7fb0180ccb20 con 0x7fb0180b03e0 2024-01-11T09:27:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.836+0000 7fb01d599700 1 -- 172.21.15.27:0/3767897729 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0180cd730 con 0x7fb0180b03e0 2024-01-11T09:27:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.837+0000 7fb01d599700 1 -- 172.21.15.27:0/3767897729 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 272096092 0 0) 0x7fb004003570 con 0x7fb0180b2b90 2024-01-11T09:27:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.837+0000 7fb01d599700 1 -- 172.21.15.27:0/3767897729 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0180cc410 msgr2=0x7fb018159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.837+0000 7fb01d599700 1 --2- 172.21.15.27:0/3767897729 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0180cc410 0x7fb018159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.837+0000 7fb01d599700 1 -- 172.21.15.27:0/3767897729 >> v1:172.21.15.77:6789/0 conn(0x7fb0180b03e0 legacy=0x7fb0180b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.837+0000 7fb01d599700 1 -- 172.21.15.27:0/3767897729 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0180cd0e0 con 0x7fb0180b2b90 2024-01-11T09:27:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.837+0000 7fb01d599700 1 -- 172.21.15.27:0/3767897729 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fb008001470 con 0x7fb0180b2b90 2024-01-11T09:27:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.837+0000 7fb01d599700 1 -- 172.21.15.27:0/3767897729 >> v1:172.21.15.27:6789/0 conn(0x7fb0180b2b90 legacy=0x7fb018159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.837+0000 7fb01d599700 1 --2- 172.21.15.27:0/3767897729 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0180cc410 0x7fb004004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.837+0000 7fb01d599700 1 --2- 172.21.15.27:0/3767897729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb004005230 0x7fb0040037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.837+0000 7fb01d599700 1 --2- 172.21.15.27:0/3767897729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb004003cf0 0x7fb004004160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.837+0000 7fb01d599700 1 -- 172.21.15.27:0/3767897729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180cce00 con 0x7fb004003cf0 2024-01-11T09:27:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.837+0000 7fb01d599700 1 -- 172.21.15.27:0/3767897729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00400bbf0 con 0x7fb004005230 2024-01-11T09:27:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.837+0000 7fb01d599700 1 -- 172.21.15.27:0/3767897729 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb00400bdb0 con 0x7fb0180cc410 2024-01-11T09:27:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.837+0000 7fb01e59b700 1 --2- 172.21.15.27:0/3767897729 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0180cc410 0x7fb004004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.837+0000 7fb01ed9c700 1 --2- 172.21.15.27:0/3767897729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb004003cf0 0x7fb004004160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.837+0000 7fb01dd9a700 1 --2- 172.21.15.27:0/3767897729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb004005230 0x7fb0040037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.838+0000 7fb01ed9c700 1 -- 172.21.15.27:0/3767897729 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0180cc410 msgr2=0x7fb004004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.838+0000 7fb01ed9c700 1 --2- 172.21.15.27:0/3767897729 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0180cc410 0x7fb004004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.838+0000 7fb01ed9c700 1 -- 172.21.15.27:0/3767897729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb004005230 msgr2=0x7fb0040037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.838+0000 7fb01ed9c700 1 --2- 172.21.15.27:0/3767897729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb004005230 0x7fb0040037b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.838+0000 7fb01ed9c700 1 -- 172.21.15.27:0/3767897729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb004003570 con 0x7fb004003cf0 2024-01-11T09:27:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.838+0000 7fb01e59b700 1 --2- 172.21.15.27:0/3767897729 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0180cc410 0x7fb004004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:27:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.838+0000 7fb01ed9c700 1 --2- 172.21.15.27:0/3767897729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb004003cf0 0x7fb004004160 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fb01000e210 tx=0x7fb01000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=366de35177172530 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.838+0000 7fb01d599700 1 -- 172.21.15.27:0/3767897729 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01000eac0 con 0x7fb004003cf0 2024-01-11T09:27:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.838+0000 7fb01d599700 1 -- 172.21.15.27:0/3767897729 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01000f040 con 0x7fb004003cf0 2024-01-11T09:27:05.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.838+0000 7fb01d599700 1 -- 172.21.15.27:0/3767897729 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01000ee70 con 0x7fb004003cf0 2024-01-11T09:27:05.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.839+0000 7fb0207ff700 1 -- 172.21.15.27:0/3767897729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb004003cf0 msgr2=0x7fb004004160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:05.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.839+0000 7fb0207ff700 1 --2- 172.21.15.27:0/3767897729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb004003cf0 0x7fb004004160 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fb01000e210 tx=0x7fb01000b040 comp rx=0 tx=0).stop 2024-01-11T09:27:05.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.839+0000 7fb01ed9c700 1 -- 172.21.15.27:0/3767897729 reap_dead start 2024-01-11T09:27:05.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.839+0000 7fb0207ff700 1 -- 172.21.15.27:0/3767897729 shutdown_connections 2024-01-11T09:27:05.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.839+0000 7fb0207ff700 1 -- 172.21.15.27:0/3767897729 >> 172.21.15.27:0/3767897729 conn(0x7fb01806ed40 msgr2=0x7fb01806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:05.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.839+0000 7fb0207ff700 1 -- 172.21.15.27:0/3767897729 shutdown_connections 2024-01-11T09:27:05.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.839+0000 7fb0207ff700 1 -- 172.21.15.27:0/3767897729 wait complete. 2024-01-11T09:27:05.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.841+0000 7fb0207ff700 1 Processor -- start 2024-01-11T09:27:05.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.841+0000 7fb0207ff700 1 -- start start 2024-01-11T09:27:05.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.842+0000 7fb0207ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb004003cf0 0x7fb0180a7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:05.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.842+0000 7fb0207ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0180a7a80 0x7fb0180b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:05.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.842+0000 7fb0207ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0180b5f70 0x7fb0180c3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.842+0000 7fb0207ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180b63c0 con 0x7fb0180a7a80 2024-01-11T09:27:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.842+0000 7fb0207ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180c4bb0 con 0x7fb004003cf0 2024-01-11T09:27:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.842+0000 7fb0207ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180c4e90 con 0x7fb0180b5f70 2024-01-11T09:27:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.842+0000 7fb01dd9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0180a7a80 0x7fb0180b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.842+0000 7fb01dd9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0180a7a80 0x7fb0180b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36082/0 (socket says 172.21.15.27:36082) 2024-01-11T09:27:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.842+0000 7fb01dd9a700 1 -- 172.21.15.27:0/3165683846 learned_addr learned my addr 172.21.15.27:0/3165683846 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.842+0000 7fb01e59b700 1 --2- 172.21.15.27:0/3165683846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb004003cf0 0x7fb0180a7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.842+0000 7fb01ed9c700 1 --2- 172.21.15.27:0/3165683846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0180b5f70 0x7fb0180c3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.842+0000 7fb01dd9a700 1 -- 172.21.15.27:0/3165683846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0180b5f70 msgr2=0x7fb0180c3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.842+0000 7fb01dd9a700 1 --2- 172.21.15.27:0/3165683846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0180b5f70 0x7fb0180c3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.842+0000 7fb01dd9a700 1 -- 172.21.15.27:0/3165683846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb004003cf0 msgr2=0x7fb0180a7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.842+0000 7fb01dd9a700 1 --2- 172.21.15.27:0/3165683846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb004003cf0 0x7fb0180a7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.842+0000 7fb01dd9a700 1 -- 172.21.15.27:0/3165683846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0180c5170 con 0x7fb0180a7a80 2024-01-11T09:27:05.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.842+0000 7fb01e59b700 1 --2- 172.21.15.27:0/3165683846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb004003cf0 0x7fb0180a7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:27:05.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.843+0000 7fb01dd9a700 1 --2- 172.21.15.27:0/3165683846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0180a7a80 0x7fb0180b5a30 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fb0080070e0 tx=0x7fb008001d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:05.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.843+0000 7fb00e7fc700 1 -- 172.21.15.27:0/3165683846 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008000c50 con 0x7fb0180a7a80 2024-01-11T09:27:05.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.843+0000 7fb00e7fc700 1 -- 172.21.15.27:0/3165683846 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb008011590 con 0x7fb0180a7a80 2024-01-11T09:27:05.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.843+0000 7fb0207ff700 1 -- 172.21.15.27:0/3165683846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0180c5450 con 0x7fb0180a7a80 2024-01-11T09:27:05.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.843+0000 7fb00e7fc700 1 -- 172.21.15.27:0/3165683846 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008011950 con 0x7fb0180a7a80 2024-01-11T09:27:05.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.843+0000 7fb0207ff700 1 -- 172.21.15.27:0/3165683846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb01814a5f0 con 0x7fb0180a7a80 2024-01-11T09:27:05.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.844+0000 7fb0207ff700 1 -- 172.21.15.27:0/3165683846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0180c5970 con 0x7fb0180a7a80 2024-01-11T09:27:05.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.845+0000 7fb00e7fc700 1 -- 172.21.15.27:0/3165683846 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 95) v1 ==== 84845+0+0 (secure 0 0 0) 0x7fb00801d460 con 0x7fb0180a7a80 2024-01-11T09:27:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.847+0000 7fb00e7fc700 1 --2- 172.21.15.27:0/3165683846 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7faffc067050 0x7faffc069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.847+0000 7fb00e7fc700 1 -- 172.21.15.27:0/3165683846 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6019+0+0 (secure 0 0 0) 0x7fb008084850 con 0x7fb0180a7a80 2024-01-11T09:27:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.847+0000 7fb01e59b700 1 -- 172.21.15.27:0/3165683846 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7faffc067050 msgr2=0x7faffc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:05.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.847+0000 7fb01e59b700 1 --2- 172.21.15.27:0/3165683846 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7faffc067050 0x7faffc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:27:05.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:05.851+0000 7fb00e7fc700 1 -- 172.21.15.27:0/3165683846 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb0180c5970 con 0x7fb0180a7a80 2024-01-11T09:27:06.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.048+0000 7fb01e59b700 1 -- 172.21.15.27:0/3165683846 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7faffc067050 msgr2=0x7faffc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:06.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.048+0000 7fb01e59b700 1 --2- 172.21.15.27:0/3165683846 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7faffc067050 0x7faffc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:27:06.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.058+0000 7fb0207ff700 1 -- 172.21.15.27:0/3165683846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0180c5970 con 0x7fb0180a7a80 2024-01-11T09:27:06.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.059+0000 7fb00e7fc700 1 -- 172.21.15.27:0/3165683846 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) v1 ==== 74+0+15771 (secure 0 0 0) 0x7fb0180c5970 con 0x7fb0180a7a80 2024-01-11T09:27:06.056 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:27:06.057 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":168,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:26:57.806860+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:26:39.008224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-01-11T09:26:40.186962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/2366220214":"2024-01-12T09:26:57.806813+0000","172.21.15.77:6837/41381":"2024-01-12T09:26:57.806813+0000","172.21.15.77:6836/41381":"2024-01-12T09:26:57.806813+0000","172.21.15.77:0/4172230033":"2024-01-12T09:26:57.806813+0000","172.21.15.77:0/1155148094":"2024-01-12T09:26:57.806813+0000","172.21.15.77:0/3578968849":"2024-01-12T09:26:57.806813+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:27:06.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.062+0000 7fb0207ff700 1 -- 172.21.15.27:0/3165683846 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7faffc067050 msgr2=0x7faffc069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:27:06.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.062+0000 7fb0207ff700 1 --2- 172.21.15.27:0/3165683846 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7faffc067050 0x7faffc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:06.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.062+0000 7fb0207ff700 1 -- 172.21.15.27:0/3165683846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0180a7a80 msgr2=0x7fb0180b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:06.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.062+0000 7fb0207ff700 1 --2- 172.21.15.27:0/3165683846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0180a7a80 0x7fb0180b5a30 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fb0080070e0 tx=0x7fb008001d20 comp rx=0 tx=0).stop 2024-01-11T09:27:06.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.062+0000 7fb0207ff700 1 -- 172.21.15.27:0/3165683846 shutdown_connections 2024-01-11T09:27:06.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.062+0000 7fb0207ff700 1 --2- 172.21.15.27:0/3165683846 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7faffc067050 0x7faffc069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:06.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.062+0000 7fb0207ff700 1 --2- 172.21.15.27:0/3165683846 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0180b5f70 0x7fb0180c3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:06.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.062+0000 7fb0207ff700 1 --2- 172.21.15.27:0/3165683846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb004003cf0 0x7fb0180a7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:06.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.062+0000 7fb0207ff700 1 --2- 172.21.15.27:0/3165683846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0180a7a80 0x7fb0180b5a30 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:06.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.062+0000 7fb0207ff700 1 -- 172.21.15.27:0/3165683846 >> 172.21.15.27:0/3165683846 conn(0x7fb01806ed40 msgr2=0x7fb01815c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:06.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.062+0000 7fb0207ff700 1 -- 172.21.15.27:0/3165683846 shutdown_connections 2024-01-11T09:27:06.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.062+0000 7fb0207ff700 1 -- 172.21.15.27:0/3165683846 wait complete. 2024-01-11T09:27:06.078 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-11T09:27:06.092 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:06.095+0000 7f6e5d36f1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:27:06.174 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:06.177+0000 7f8450c391c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:27:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.287+0000 7f02793ff700 1 Processor -- start 2024-01-11T09:27:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.287+0000 7f02793ff700 1 -- start start 2024-01-11T09:27:06.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.288+0000 7f02793ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0274157360 0x7f027406f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:06.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.289+0000 7f02793ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02740b19a0 0x7f02740cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:06.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.289+0000 7f02793ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02740cce10 con 0x7f0274157730 2024-01-11T09:27:06.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.289+0000 7f02793ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02740cb540 con 0x7f0274157360 2024-01-11T09:27:06.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.289+0000 7f02793ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02740ca960 con 0x7f02740b19a0 2024-01-11T09:27:06.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.289+0000 7f02727fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0274157730 0x7f02740b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40462/0 (socket says 172.21.15.27:40462) 2024-01-11T09:27:06.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.289+0000 7f02727fc700 1 -- 172.21.15.27:0/2948207089 learned_addr learned my addr 172.21.15.27:0/2948207089 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.289+0000 7f02737fe700 1 --2- 172.21.15.27:0/2948207089 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02740b19a0 0x7f02740cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.289+0000 7f0272ffd700 1 --2- 172.21.15.27:0/2948207089 >> v2:172.21.15.77:3300/0 conn(0x7f0274157360 0x7f027406f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.289+0000 7f0271ffb700 1 -- 172.21.15.27:0/2948207089 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292956975 0 0) 0x7f02740cce10 con 0x7f0274157730 2024-01-11T09:27:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.289+0000 7f0271ffb700 1 -- 172.21.15.27:0/2948207089 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f025c003570 con 0x7f0274157730 2024-01-11T09:27:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.289+0000 7f02737fe700 1 -- 172.21.15.27:0/2948207089 >> v2:172.21.15.77:3300/0 conn(0x7f0274157360 msgr2=0x7f027406f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.289+0000 7f02737fe700 1 --2- 172.21.15.27:0/2948207089 >> v2:172.21.15.77:3300/0 conn(0x7f0274157360 0x7f027406f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.289+0000 7f02737fe700 1 -- 172.21.15.27:0/2948207089 >> v1:172.21.15.27:6789/0 conn(0x7f0274157730 legacy=0x7f02740b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.290+0000 7f02737fe700 1 -- 172.21.15.27:0/2948207089 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02740cac10 con 0x7f02740b19a0 2024-01-11T09:27:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.290+0000 7f0272ffd700 1 --2- 172.21.15.27:0/2948207089 >> v2:172.21.15.77:3300/0 conn(0x7f0274157360 0x7f027406f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:27:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.290+0000 7f02737fe700 1 --2- 172.21.15.27:0/2948207089 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02740b19a0 0x7f02740cc8d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f026c009e60 tx=0x7f026c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ade87c443ccd773b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:06.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.290+0000 7f0271ffb700 1 -- 172.21.15.27:0/2948207089 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c00e070 con 0x7f02740b19a0 2024-01-11T09:27:06.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.290+0000 7f0271ffb700 1 -- 172.21.15.27:0/2948207089 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026c00bab0 con 0x7f02740b19a0 2024-01-11T09:27:06.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.290+0000 7f0271ffb700 1 -- 172.21.15.27:0/2948207089 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c00be70 con 0x7f02740b19a0 2024-01-11T09:27:06.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.291+0000 7f02793ff700 1 -- 172.21.15.27:0/2948207089 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02740b19a0 msgr2=0x7f02740cc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:06.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.291+0000 7f02793ff700 1 --2- 172.21.15.27:0/2948207089 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02740b19a0 0x7f02740cc8d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f026c009e60 tx=0x7f026c009780 comp rx=0 tx=0).stop 2024-01-11T09:27:06.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.291+0000 7f02793ff700 1 -- 172.21.15.27:0/2948207089 shutdown_connections 2024-01-11T09:27:06.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.291+0000 7f02793ff700 1 --2- 172.21.15.27:0/2948207089 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02740b19a0 0x7f02740cc8d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:06.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.291+0000 7f02793ff700 1 --2- 172.21.15.27:0/2948207089 >> v2:172.21.15.77:3300/0 conn(0x7f0274157360 0x7f027406f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:06.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.291+0000 7f02793ff700 1 -- 172.21.15.27:0/2948207089 >> 172.21.15.27:0/2948207089 conn(0x7f02740680b0 msgr2=0x7f02740cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:06.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.291+0000 7f02793ff700 1 -- 172.21.15.27:0/2948207089 shutdown_connections 2024-01-11T09:27:06.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.291+0000 7f02793ff700 1 -- 172.21.15.27:0/2948207089 wait complete. 2024-01-11T09:27:06.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.293+0000 7f02793ff700 1 Processor -- start 2024-01-11T09:27:06.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.293+0000 7f02793ff700 1 -- start start 2024-01-11T09:27:06.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.294+0000 7f02793ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02740b19a0 0x7f02740c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.295+0000 7f02793ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0274157360 0x7f02740ac910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.295+0000 7f02793ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0274157730 0x7f02740ace50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.295+0000 7f02793ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0274145560 con 0x7f0274157360 2024-01-11T09:27:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.295+0000 7f02793ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027415aa60 con 0x7f02740b19a0 2024-01-11T09:27:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.295+0000 7f02793ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027415ad40 con 0x7f0274157730 2024-01-11T09:27:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.295+0000 7f02727fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0274157360 0x7f02740ac910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.295+0000 7f02737fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0274157730 0x7f02740ace50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.295+0000 7f0272ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02740b19a0 0x7f02740c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.295+0000 7f02737fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0274157730 0x7f02740ace50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:35354/0 (socket says 172.21.15.27:35354) 2024-01-11T09:27:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.295+0000 7f02737fe700 1 -- 172.21.15.27:0/150039234 learned_addr learned my addr 172.21.15.27:0/150039234 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.295+0000 7f02737fe700 1 -- 172.21.15.27:0/150039234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02740b19a0 msgr2=0x7f02740c7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.295+0000 7f02737fe700 1 --2- 172.21.15.27:0/150039234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02740b19a0 0x7f02740c7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:06.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.295+0000 7f02737fe700 1 -- 172.21.15.27:0/150039234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0274157360 msgr2=0x7f02740ac910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:06.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.295+0000 7f02737fe700 1 --2- 172.21.15.27:0/150039234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0274157360 0x7f02740ac910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:06.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.295+0000 7f02737fe700 1 -- 172.21.15.27:0/150039234 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0264009420 con 0x7f0274157730 2024-01-11T09:27:06.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.296+0000 7f02737fe700 1 --2- 172.21.15.27:0/150039234 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0274157730 0x7f02740ace50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f026c013000 tx=0x7f026c019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:06.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.296+0000 7f025affd700 1 -- 172.21.15.27:0/150039234 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c012e10 con 0x7f0274157730 2024-01-11T09:27:06.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.296+0000 7f025affd700 1 -- 172.21.15.27:0/150039234 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026c027470 con 0x7f0274157730 2024-01-11T09:27:06.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.296+0000 7f025affd700 1 -- 172.21.15.27:0/150039234 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c027830 con 0x7f0274157730 2024-01-11T09:27:06.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.296+0000 7f02793ff700 1 -- 172.21.15.27:0/150039234 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f026c009a20 con 0x7f0274157730 2024-01-11T09:27:06.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.296+0000 7f02793ff700 1 -- 172.21.15.27:0/150039234 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027415b300 con 0x7f0274157730 2024-01-11T09:27:06.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.297+0000 7f02793ff700 1 -- 172.21.15.27:0/150039234 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f027404c3c0 con 0x7f0274157730 2024-01-11T09:27:06.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.298+0000 7f025affd700 1 -- 172.21.15.27:0/150039234 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 95) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f026c024070 con 0x7f0274157730 2024-01-11T09:27:06.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.299+0000 7f025affd700 1 --2- 172.21.15.27:0/150039234 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0260067050 0x7f0260069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:06.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.300+0000 7f025affd700 1 -- 172.21.15.27:0/150039234 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f026c087200 con 0x7f0274157730 2024-01-11T09:27:06.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.300+0000 7f0272ffd700 1 -- 172.21.15.27:0/150039234 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0260067050 msgr2=0x7f0260069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:06.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.300+0000 7f0272ffd700 1 --2- 172.21.15.27:0/150039234 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0260067050 0x7f0260069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:27:06.302 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:06.305+0000 7f6e5d36f1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:27:06.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.307+0000 7f025affd700 1 -- 172.21.15.27:0/150039234 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f026c054180 con 0x7f0274157730 2024-01-11T09:27:06.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.501+0000 7f0272ffd700 1 -- 172.21.15.27:0/150039234 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0260067050 msgr2=0x7f0260069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:06.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.501+0000 7f0272ffd700 1 --2- 172.21.15.27:0/150039234 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0260067050 0x7f0260069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:27:06.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.557+0000 7f02793ff700 1 -- 172.21.15.27:0/150039234 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f027406cdf0 con 0x7f0274157730 2024-01-11T09:27:06.555 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-11T09:27:06.559+0000 7f27065f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-11T09:27:06.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.564+0000 7f025affd700 1 -- 172.21.15.27:0/150039234 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v87) v1 ==== 161+0+0 (secure 0 0 0) 0x7f026c0577a0 con 0x7f0274157730 2024-01-11T09:27:06.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.568+0000 7f02793ff700 1 -- 172.21.15.27:0/150039234 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0260067050 msgr2=0x7f0260069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:27:06.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.568+0000 7f02793ff700 1 --2- 172.21.15.27:0/150039234 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0260067050 0x7f0260069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:06.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.568+0000 7f02793ff700 1 -- 172.21.15.27:0/150039234 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0274157730 msgr2=0x7f02740ace50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:06.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.568+0000 7f02793ff700 1 --2- 172.21.15.27:0/150039234 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0274157730 0x7f02740ace50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f026c013000 tx=0x7f026c019000 comp rx=0 tx=0).stop 2024-01-11T09:27:06.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.569+0000 7f02793ff700 1 -- 172.21.15.27:0/150039234 shutdown_connections 2024-01-11T09:27:06.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.569+0000 7f02793ff700 1 --2- 172.21.15.27:0/150039234 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0260067050 0x7f0260069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:06.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.569+0000 7f02793ff700 1 --2- 172.21.15.27:0/150039234 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0274157730 0x7f02740ace50 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:06.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.569+0000 7f02793ff700 1 --2- 172.21.15.27:0/150039234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02740b19a0 0x7f02740c7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:06.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.569+0000 7f02793ff700 1 --2- 172.21.15.27:0/150039234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0274157360 0x7f02740ac910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:06.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.569+0000 7f02793ff700 1 -- 172.21.15.27:0/150039234 >> 172.21.15.27:0/150039234 conn(0x7f02740680b0 msgr2=0x7f02741564c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:06.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.569+0000 7f02793ff700 1 -- 172.21.15.27:0/150039234 shutdown_connections 2024-01-11T09:27:06.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.569+0000 7f02793ff700 1 -- 172.21.15.27:0/150039234 wait complete. 2024-01-11T09:27:06.567 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-11T09:27:06.567 INFO:tasks.ceph.mds.a.smithi027.stderr: -2330> 2024-01-11T09:26:46.658+0000 7fa7757f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T09:27:06.594 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-11T09:27:06.604 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:06.608+0000 7f8450c391c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:27:06.610 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T09:27:06.723 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:06.727+0000 7f6e5d36f1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:27:06.789 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:06.793+0000 7f8450c391c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:27:06.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.819+0000 7f21de5ff700 1 Processor -- start 2024-01-11T09:27:06.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.819+0000 7f21de5ff700 1 -- start start 2024-01-11T09:27:06.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.820+0000 7f21de5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f21d815a1b0 0x7f21d815a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:06.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.821+0000 7f21de5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d80cf850 con 0x7f21d80ccd70 2024-01-11T09:27:06.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.821+0000 7f21de5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d80cec40 con 0x7f21d815ae70 2024-01-11T09:27:06.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.821+0000 7f21de5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d80cef20 con 0x7f21d815a1b0 2024-01-11T09:27:06.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.821+0000 7f21d7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f21d815a1b0 0x7f21d815a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:06.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.821+0000 7f21d77fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f21d80ccd70 0x7f21d80cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40468/0 (socket says 172.21.15.27:40468) 2024-01-11T09:27:06.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.821+0000 7f21d77fe700 1 -- 172.21.15.27:0/2727070013 learned_addr learned my addr 172.21.15.27:0/2727070013 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:06.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.821+0000 7f21d7fff700 1 -- 172.21.15.27:0/2727070013 >> v1:172.21.15.77:6789/0 conn(0x7f21d815ae70 legacy=0x7f21d815b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:06.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.821+0000 7f21d7fff700 1 -- 172.21.15.27:0/2727070013 >> v1:172.21.15.27:6789/0 conn(0x7f21d80ccd70 legacy=0x7f21d80cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:06.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.821+0000 7f21d7fff700 1 -- 172.21.15.27:0/2727070013 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d80cf200 con 0x7f21d815a1b0 2024-01-11T09:27:06.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.822+0000 7f21d7fff700 1 --2- 172.21.15.27:0/2727070013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d815a1b0 0x7f21d815a580 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f21c80098e0 tx=0x7f21c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df2364219c81718 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:06.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.822+0000 7f21d6ffd700 1 -- 172.21.15.27:0/2727070013 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c8009e80 con 0x7f21d815a1b0 2024-01-11T09:27:06.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.822+0000 7f21d6ffd700 1 -- 172.21.15.27:0/2727070013 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c800b860 con 0x7f21d815a1b0 2024-01-11T09:27:06.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.822+0000 7f21d6ffd700 1 -- 172.21.15.27:0/2727070013 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c800bbe0 con 0x7f21d815a1b0 2024-01-11T09:27:06.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.822+0000 7f21de5ff700 1 -- 172.21.15.27:0/2727070013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d815a1b0 msgr2=0x7f21d815a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:06.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.822+0000 7f21de5ff700 1 --2- 172.21.15.27:0/2727070013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d815a1b0 0x7f21d815a580 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f21c80098e0 tx=0x7f21c80091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:06.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.823+0000 7f21de5ff700 1 -- 172.21.15.27:0/2727070013 shutdown_connections 2024-01-11T09:27:06.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.823+0000 7f21de5ff700 1 --2- 172.21.15.27:0/2727070013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d815a1b0 0x7f21d815a580 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:06.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.823+0000 7f21de5ff700 1 -- 172.21.15.27:0/2727070013 >> 172.21.15.27:0/2727070013 conn(0x7f21d806ed40 msgr2=0x7f21d806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:06.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.823+0000 7f21de5ff700 1 -- 172.21.15.27:0/2727070013 shutdown_connections 2024-01-11T09:27:06.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.823+0000 7f21de5ff700 1 -- 172.21.15.27:0/2727070013 wait complete. 2024-01-11T09:27:06.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.824+0000 7f21de5ff700 1 Processor -- start 2024-01-11T09:27:06.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.824+0000 7f21de5ff700 1 -- start start 2024-01-11T09:27:06.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.825+0000 7f21de5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d80ccd70 0x7f21d814d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:06.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.825+0000 7f21de5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d815a1b0 0x7f21d80ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.825+0000 7f21de5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f21d815ae70 0x7f21d8157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.825+0000 7f21de5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d80c9cb0 con 0x7f21d80ccd70 2024-01-11T09:27:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.825+0000 7f21de5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d80b5c00 con 0x7f21d815a1b0 2024-01-11T09:27:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.825+0000 7f21de5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d80c7b50 con 0x7f21d815ae70 2024-01-11T09:27:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.826+0000 7f21d7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d80ccd70 0x7f21d814d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.826+0000 7f21d7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d80ccd70 0x7f21d814d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36116/0 (socket says 172.21.15.27:36116) 2024-01-11T09:27:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.826+0000 7f21d7fff700 1 -- 172.21.15.27:0/1765889500 learned_addr learned my addr 172.21.15.27:0/1765889500 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.826+0000 7f21d77fe700 1 --2- 172.21.15.27:0/1765889500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d815a1b0 0x7f21d80ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.826+0000 7f21dcb9c700 1 --2- 172.21.15.27:0/1765889500 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f21d815ae70 0x7f21d8157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.826+0000 7f21d7fff700 1 -- 172.21.15.27:0/1765889500 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f21d815ae70 msgr2=0x7f21d8157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.826+0000 7f21d7fff700 1 --2- 172.21.15.27:0/1765889500 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f21d815ae70 0x7f21d8157900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.826+0000 7f21d7fff700 1 -- 172.21.15.27:0/1765889500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d815a1b0 msgr2=0x7f21d80ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.826+0000 7f21d7fff700 1 --2- 172.21.15.27:0/1765889500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d815a1b0 0x7f21d80ca0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.826+0000 7f21d7fff700 1 -- 172.21.15.27:0/1765889500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c8009420 con 0x7f21d80ccd70 2024-01-11T09:27:06.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.826+0000 7f21d7fff700 1 --2- 172.21.15.27:0/1765889500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d80ccd70 0x7f21d814d2f0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f21d8069770 tx=0x7f21c8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:06.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.827+0000 7f21bffff700 1 -- 172.21.15.27:0/1765889500 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c800b9d0 con 0x7f21d80ccd70 2024-01-11T09:27:06.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.827+0000 7f21bffff700 1 -- 172.21.15.27:0/1765889500 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c800bb30 con 0x7f21d80ccd70 2024-01-11T09:27:06.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.827+0000 7f21bffff700 1 -- 172.21.15.27:0/1765889500 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c8025670 con 0x7f21d80ccd70 2024-01-11T09:27:06.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.827+0000 7f21de5ff700 1 -- 172.21.15.27:0/1765889500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d80b5e50 con 0x7f21d80ccd70 2024-01-11T09:27:06.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.827+0000 7f21de5ff700 1 -- 172.21.15.27:0/1765889500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d80b3a10 con 0x7f21d80ccd70 2024-01-11T09:27:06.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.828+0000 7f21de5ff700 1 -- 172.21.15.27:0/1765889500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d80c9f30 con 0x7f21d80ccd70 2024-01-11T09:27:06.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.829+0000 7f21bffff700 1 -- 172.21.15.27:0/1765889500 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 95) v1 ==== 84845+0+0 (secure 0 0 0) 0x7f21c8023070 con 0x7f21d80ccd70 2024-01-11T09:27:06.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.830+0000 7f21bffff700 1 --2- 172.21.15.27:0/1765889500 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f21c4067050 0x7f21c4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:06.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.830+0000 7f21bffff700 1 -- 172.21.15.27:0/1765889500 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6105+0+0 (secure 0 0 0) 0x7f21c8085e20 con 0x7f21d80ccd70 2024-01-11T09:27:06.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.830+0000 7f21d77fe700 1 -- 172.21.15.27:0/1765889500 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f21c4067050 msgr2=0x7f21c4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:06.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.830+0000 7f21d77fe700 1 --2- 172.21.15.27:0/1765889500 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f21c4067050 0x7f21c4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:27:06.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:06.838+0000 7f21bffff700 1 -- 172.21.15.27:0/1765889500 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f21c8052ea0 con 0x7f21d80ccd70 2024-01-11T09:27:06.981 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:06.985+0000 7f8450c391c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:27:07.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.031+0000 7f21d77fe700 1 -- 172.21.15.27:0/1765889500 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f21c4067050 msgr2=0x7f21c4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:07.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.031+0000 7f21d77fe700 1 --2- 172.21.15.27:0/1765889500 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f21c4067050 0x7f21c4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:27:07.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.085+0000 7f21de5ff700 1 -- 172.21.15.27:0/1765889500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f21d804c3c0 con 0x7f21d80ccd70 2024-01-11T09:27:07.089 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:07.093+0000 7f8450c391c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:27:07.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.101+0000 7f21bffff700 1 -- 172.21.15.27:0/1765889500 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 96) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f21c8053270 con 0x7f21d80ccd70 2024-01-11T09:27:07.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.101+0000 7f21bffff700 1 -- 172.21.15.27:0/1765889500 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f21c4067050 msgr2=0x7f21c4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:27:07.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.101+0000 7f21bffff700 1 --2- 172.21.15.27:0/1765889500 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f21c4067050 0x7f21c4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:07.161 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:07.165+0000 7f83cba44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:07.162 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:07.165+0000 7f83cba44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:07.162 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:07.165+0000 7f83cba44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:07.162 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:07.165+0000 7f83cba44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:07.162 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:07.165+0000 7f83cba44700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:07.162 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:07.165+0000 7f83ce249700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:07.162 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:07.165+0000 7f83ce249700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:07.162 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:07.165+0000 7f83ce249700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:07.162 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:07.165+0000 7f83ce249700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:07.162 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:07.165+0000 7f83ce249700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:07.193 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:07.196+0000 7f6e5d36f1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:27:07.388 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:07.391+0000 7f6e5d36f1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:27:07.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.564+0000 7f21bffff700 1 -- 172.21.15.27:0/1765889500 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v88) v1 ==== 104+0+0 (secure 0 0 0) 0x7f21c80270d0 con 0x7f21d80ccd70 2024-01-11T09:27:07.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.567+0000 7f21de5ff700 1 -- 172.21.15.27:0/1765889500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d80ccd70 msgr2=0x7f21d814d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:07.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.567+0000 7f21de5ff700 1 --2- 172.21.15.27:0/1765889500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d80ccd70 0x7f21d814d2f0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f21d8069770 tx=0x7f21c8017000 comp rx=0 tx=0).stop 2024-01-11T09:27:07.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.567+0000 7f21de5ff700 1 -- 172.21.15.27:0/1765889500 shutdown_connections 2024-01-11T09:27:07.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.567+0000 7f21de5ff700 1 --2- 172.21.15.27:0/1765889500 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f21c4067050 0x7f21c4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:07.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.567+0000 7f21de5ff700 1 --2- 172.21.15.27:0/1765889500 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f21d815ae70 0x7f21d8157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:07.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.567+0000 7f21de5ff700 1 --2- 172.21.15.27:0/1765889500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d80ccd70 0x7f21d814d2f0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:07.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.567+0000 7f21de5ff700 1 --2- 172.21.15.27:0/1765889500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d815a1b0 0x7f21d80ca0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:07.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.567+0000 7f21de5ff700 1 -- 172.21.15.27:0/1765889500 >> 172.21.15.27:0/1765889500 conn(0x7f21d806ed40 msgr2=0x7f21d80c5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:07.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.567+0000 7f21de5ff700 1 -- 172.21.15.27:0/1765889500 shutdown_connections 2024-01-11T09:27:07.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.567+0000 7f21de5ff700 1 -- 172.21.15.27:0/1765889500 wait complete. 2024-01-11T09:27:07.582 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-11T09:27:07.591 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:07.595+0000 7f6e5d36f1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:27:07.715 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:07.719+0000 7f6e5d36f1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:27:07.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.799+0000 7f0f609ff700 1 Processor -- start 2024-01-11T09:27:07.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.799+0000 7f0f609ff700 1 -- start start 2024-01-11T09:27:07.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.800+0000 7f0f609ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0f5c159250 0x7f0f5c0b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:07.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.801+0000 7f0f609ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0f5c0cb410 0x7f0f5c0cb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:07.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.801+0000 7f0f609ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0f5c0684d0 0x7f0f5c068910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:07.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.801+0000 7f0f609ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f5c15ace0 con 0x7f0f5c0cb410 2024-01-11T09:27:07.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.801+0000 7f0f609ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f5c0b0de0 con 0x7f0f5c159250 2024-01-11T09:27:07.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.801+0000 7f0f609ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c0cacf0 con 0x7f0f5c0684d0 2024-01-11T09:27:07.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.801+0000 7f0f59d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0f5c0cb410 0x7f0f5c0cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:07.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.801+0000 7f0f5a59c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0f5c159250 0x7f0f5c0b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:07.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.801+0000 7f0f59d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0f5c0cb410 0x7f0f5c0cb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36128/0 (socket says 172.21.15.27:36128) 2024-01-11T09:27:07.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.801+0000 7f0f59d9b700 1 -- 172.21.15.27:0/233696241 learned_addr learned my addr 172.21.15.27:0/233696241 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:07.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.801+0000 7f0f5ad9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0f5c0684d0 0x7f0f5c068910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:07.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.801+0000 7f0f59d9b700 1 -- 172.21.15.27:0/233696241 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0f5c0684d0 msgr2=0x7f0f5c068910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:07.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.801+0000 7f0f59d9b700 1 --2- 172.21.15.27:0/233696241 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0f5c0684d0 0x7f0f5c068910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:07.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.801+0000 7f0f59d9b700 1 -- 172.21.15.27:0/233696241 >> v2:172.21.15.77:3300/0 conn(0x7f0f5c159250 msgr2=0x7f0f5c0b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:07.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.802+0000 7f0f59d9b700 1 --2- 172.21.15.27:0/233696241 >> v2:172.21.15.77:3300/0 conn(0x7f0f5c159250 0x7f0f5c0b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:07.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.802+0000 7f0f59d9b700 1 -- 172.21.15.27:0/233696241 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5c15a440 con 0x7f0f5c0cb410 2024-01-11T09:27:07.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.802+0000 7f0f5a59c700 1 --2- 172.21.15.27:0/233696241 >> v2:172.21.15.77:3300/0 conn(0x7f0f5c159250 0x7f0f5c0b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:27:07.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.802+0000 7f0f59d9b700 1 --2- 172.21.15.27:0/233696241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f5c0cb410 0x7f0f5c0cb890 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c0ce0c0 tx=0x7f0f50009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4546dceec98b3619 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:07.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.802+0000 7f0f5959a700 1 -- 172.21.15.27:0/233696241 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5000b640 con 0x7f0f5c0cb410 2024-01-11T09:27:07.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.802+0000 7f0f5959a700 1 -- 172.21.15.27:0/233696241 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f5000b7a0 con 0x7f0f5c0cb410 2024-01-11T09:27:07.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.802+0000 7f0f5959a700 1 -- 172.21.15.27:0/233696241 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5000bb60 con 0x7f0f5c0cb410 2024-01-11T09:27:07.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.803+0000 7f0f609ff700 1 -- 172.21.15.27:0/233696241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f5c0cb410 msgr2=0x7f0f5c0cb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:07.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.803+0000 7f0f609ff700 1 --2- 172.21.15.27:0/233696241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f5c0cb410 0x7f0f5c0cb890 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c0ce0c0 tx=0x7f0f50009250 comp rx=0 tx=0).stop 2024-01-11T09:27:07.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.803+0000 7f0f609ff700 1 -- 172.21.15.27:0/233696241 shutdown_connections 2024-01-11T09:27:07.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.803+0000 7f0f609ff700 1 --2- 172.21.15.27:0/233696241 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0f5c0684d0 0x7f0f5c068910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:07.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.803+0000 7f0f609ff700 1 --2- 172.21.15.27:0/233696241 >> v2:172.21.15.77:3300/0 conn(0x7f0f5c159250 0x7f0f5c0b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:07.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.803+0000 7f0f609ff700 1 --2- 172.21.15.27:0/233696241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f5c0cb410 0x7f0f5c0cb890 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:07.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.803+0000 7f0f609ff700 1 -- 172.21.15.27:0/233696241 >> 172.21.15.27:0/233696241 conn(0x7f0f5c067610 msgr2=0x7f0f5c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:07.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.803+0000 7f0f609ff700 1 -- 172.21.15.27:0/233696241 shutdown_connections 2024-01-11T09:27:07.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.803+0000 7f0f609ff700 1 -- 172.21.15.27:0/233696241 wait complete. 2024-01-11T09:27:07.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.804+0000 7f0f609ff700 1 Processor -- start 2024-01-11T09:27:07.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.805+0000 7f0f609ff700 1 -- start start 2024-01-11T09:27:07.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.806+0000 7f0f609ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0f5c0684d0 0x7f0f5c0c6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:07.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.806+0000 7f0f609ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f5c0cb410 0x7f0f5c0c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:07.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.806+0000 7f0f609ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f5c159250 0x7f0f5c15f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:07.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.806+0000 7f0f609ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c0c75a0 con 0x7f0f5c0cb410 2024-01-11T09:27:07.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.806+0000 7f0f609ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c0c5540 con 0x7f0f5c159250 2024-01-11T09:27:07.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.806+0000 7f0f609ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c0c5820 con 0x7f0f5c0684d0 2024-01-11T09:27:07.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.806+0000 7f0f59d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f5c0cb410 0x7f0f5c0c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:07.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.806+0000 7f0f59d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f5c0cb410 0x7f0f5c0c6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36136/0 (socket says 172.21.15.27:36136) 2024-01-11T09:27:07.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.806+0000 7f0f5ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f5c159250 0x7f0f5c15f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:07.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.806+0000 7f0f5a59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0f5c0684d0 0x7f0f5c0c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:07.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.806+0000 7f0f59d9b700 1 -- 172.21.15.27:0/4172434459 learned_addr learned my addr 172.21.15.27:0/4172434459 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:07.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.806+0000 7f0f5ad9d700 1 -- 172.21.15.27:0/4172434459 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0f5c0684d0 msgr2=0x7f0f5c0c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:07.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.806+0000 7f0f5ad9d700 1 --2- 172.21.15.27:0/4172434459 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0f5c0684d0 0x7f0f5c0c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:07.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.806+0000 7f0f5ad9d700 1 -- 172.21.15.27:0/4172434459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f5c0cb410 msgr2=0x7f0f5c0c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:07.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.806+0000 7f0f5ad9d700 1 --2- 172.21.15.27:0/4172434459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f5c0cb410 0x7f0f5c0c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:07.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.806+0000 7f0f5ad9d700 1 -- 172.21.15.27:0/4172434459 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4c009420 con 0x7f0f5c159250 2024-01-11T09:27:07.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.807+0000 7f0f5ad9d700 1 --2- 172.21.15.27:0/4172434459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f5c159250 0x7f0f5c15f850 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f0f5400ecc0 tx=0x7f0f5400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:07.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.807+0000 7f0f4a7fc700 1 -- 172.21.15.27:0/4172434459 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5400c8c0 con 0x7f0f5c159250 2024-01-11T09:27:07.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.807+0000 7f0f4a7fc700 1 -- 172.21.15.27:0/4172434459 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f5400f040 con 0x7f0f5c159250 2024-01-11T09:27:07.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.807+0000 7f0f4a7fc700 1 -- 172.21.15.27:0/4172434459 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5400cc90 con 0x7f0f5c159250 2024-01-11T09:27:07.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.807+0000 7f0f609ff700 1 -- 172.21.15.27:0/4172434459 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f500094f0 con 0x7f0f5c159250 2024-01-11T09:27:07.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.807+0000 7f0f609ff700 1 -- 172.21.15.27:0/4172434459 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5c0c6560 con 0x7f0f5c159250 2024-01-11T09:27:07.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.808+0000 7f0f609ff700 1 -- 172.21.15.27:0/4172434459 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f5c0c5d40 con 0x7f0f5c159250 2024-01-11T09:27:07.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.809+0000 7f0f4a7fc700 1 -- 172.21.15.27:0/4172434459 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 96) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f0f54014690 con 0x7f0f5c159250 2024-01-11T09:27:07.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.810+0000 7f0f4a7fc700 1 -- 172.21.15.27:0/4172434459 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f0f54084db0 con 0x7f0f5c159250 2024-01-11T09:27:07.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:07.814+0000 7f0f4a7fc700 1 -- 172.21.15.27:0/4172434459 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0f5c0c5d40 con 0x7f0f5c159250 2024-01-11T09:27:08.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:08.019+0000 7f0f609ff700 1 -- 172.21.15.27:0/4172434459 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0f5c0c5d40 con 0x7f0f5c159250 2024-01-11T09:27:08.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:08.111+0000 7f0f4a7fc700 1 -- 172.21.15.27:0/4172434459 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 97) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f0f54051b10 con 0x7f0f5c159250 2024-01-11T09:27:08.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:08.111+0000 7f0f4a7fc700 1 --2- 172.21.15.27:0/4172434459 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0f3c06c730 0x7f0f3c06eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:08.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:08.112+0000 7f0f5a59c700 1 --2- 172.21.15.27:0/4172434459 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0f3c06c730 0x7f0f3c06eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:08.114+0000 7f0f5a59c700 1 --2- 172.21.15.27:0/4172434459 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0f3c06c730 0x7f0f3c06eb10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c009f40 tx=0x7f0f4c0091a0 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:08.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:08.900+0000 7f0f4a7fc700 1 -- 172.21.15.27:0/4172434459 <== mon.1 v2:172.21.15.77:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v171) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0f5406fe70 con 0x7f0f5c159250 2024-01-11T09:27:08.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:08.905+0000 7f0f609ff700 1 -- 172.21.15.27:0/4172434459 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0f3c06c730 msgr2=0x7f0f3c06eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:08.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:08.905+0000 7f0f609ff700 1 --2- 172.21.15.27:0/4172434459 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0f3c06c730 0x7f0f3c06eb10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c009f40 tx=0x7f0f4c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:08.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:08.905+0000 7f0f609ff700 1 -- 172.21.15.27:0/4172434459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f5c159250 msgr2=0x7f0f5c15f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:08.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:08.905+0000 7f0f609ff700 1 --2- 172.21.15.27:0/4172434459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f5c159250 0x7f0f5c15f850 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f0f5400ecc0 tx=0x7f0f5400b040 comp rx=0 tx=0).stop 2024-01-11T09:27:08.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:08.906+0000 7f0f609ff700 1 -- 172.21.15.27:0/4172434459 shutdown_connections 2024-01-11T09:27:08.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:08.906+0000 7f0f609ff700 1 --2- 172.21.15.27:0/4172434459 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0f3c06c730 0x7f0f3c06eb10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:08.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:08.906+0000 7f0f609ff700 1 --2- 172.21.15.27:0/4172434459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f5c159250 0x7f0f5c15f850 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:08.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:08.906+0000 7f0f609ff700 1 --2- 172.21.15.27:0/4172434459 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0f5c0684d0 0x7f0f5c0c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:08.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:08.906+0000 7f0f609ff700 1 --2- 172.21.15.27:0/4172434459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f5c0cb410 0x7f0f5c0c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:08.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:08.906+0000 7f0f609ff700 1 -- 172.21.15.27:0/4172434459 >> 172.21.15.27:0/4172434459 conn(0x7f0f5c067610 msgr2=0x7f0f5c063230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:08.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:08.906+0000 7f0f609ff700 1 -- 172.21.15.27:0/4172434459 shutdown_connections 2024-01-11T09:27:08.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:08.906+0000 7f0f609ff700 1 -- 172.21.15.27:0/4172434459 wait complete. 2024-01-11T09:27:08.904 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-01-11T09:27:08.926 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-11T09:27:09.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.184+0000 7f2a611ff700 1 Processor -- start 2024-01-11T09:27:09.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.184+0000 7f2a611ff700 1 -- start start 2024-01-11T09:27:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.186+0000 7f2a611ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a5c156810 0x7f2a5c06e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.186+0000 7f2a611ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2a5c0ccac0 0x7f2a5c06ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.186+0000 7f2a611ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a5c0c8590 con 0x7f2a5c156440 2024-01-11T09:27:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.186+0000 7f2a611ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a5c0c7980 con 0x7f2a5c0ccac0 2024-01-11T09:27:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.186+0000 7f2a611ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5c0c7c60 con 0x7f2a5c156810 2024-01-11T09:27:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.186+0000 7f2a5b59e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2a5c0ccac0 0x7f2a5c06ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.186+0000 7f2a5b59e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2a5c0ccac0 0x7f2a5c06ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36138/0 (socket says 172.21.15.27:36138) 2024-01-11T09:27:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.186+0000 7f2a5b59e700 1 -- 172.21.15.27:0/277872170 learned_addr learned my addr 172.21.15.27:0/277872170 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.186+0000 7f2a5b59e700 1 -- 172.21.15.27:0/277872170 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a5c156810 msgr2=0x7f2a5c06e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.186+0000 7f2a5a59c700 1 --2- 172.21.15.27:0/277872170 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a5c156810 0x7f2a5c06e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.186+0000 7f2a5b59e700 1 --2- 172.21.15.27:0/277872170 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a5c156810 0x7f2a5c06e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.186+0000 7f2a5b59e700 1 -- 172.21.15.27:0/277872170 >> v1:172.21.15.77:6789/0 conn(0x7f2a5c156440 legacy=0x7f2a5c0c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.186+0000 7f2a5b59e700 1 -- 172.21.15.27:0/277872170 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5c0c7f40 con 0x7f2a5c0ccac0 2024-01-11T09:27:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.186+0000 7f2a5a59c700 1 --2- 172.21.15.27:0/277872170 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a5c156810 0x7f2a5c06e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:27:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.187+0000 7f2a5b59e700 1 --2- 172.21.15.27:0/277872170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a5c0ccac0 0x7f2a5c06ec60 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f2a5400d1a0 tx=0x7f2a5400ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ff92c0effa397d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.187+0000 7f2a59d9b700 1 -- 172.21.15.27:0/277872170 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a54010070 con 0x7f2a5c0ccac0 2024-01-11T09:27:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.187+0000 7f2a59d9b700 1 -- 172.21.15.27:0/277872170 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a5400de80 con 0x7f2a5c0ccac0 2024-01-11T09:27:09.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.187+0000 7f2a59d9b700 1 -- 172.21.15.27:0/277872170 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5400b320 con 0x7f2a5c0ccac0 2024-01-11T09:27:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.187+0000 7f2a611ff700 1 -- 172.21.15.27:0/277872170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a5c0ccac0 msgr2=0x7f2a5c06ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.187+0000 7f2a611ff700 1 --2- 172.21.15.27:0/277872170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a5c0ccac0 0x7f2a5c06ec60 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f2a5400d1a0 tx=0x7f2a5400ba50 comp rx=0 tx=0).stop 2024-01-11T09:27:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.188+0000 7f2a611ff700 1 -- 172.21.15.27:0/277872170 shutdown_connections 2024-01-11T09:27:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.188+0000 7f2a611ff700 1 --2- 172.21.15.27:0/277872170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a5c0ccac0 0x7f2a5c06ec60 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.188+0000 7f2a611ff700 1 --2- 172.21.15.27:0/277872170 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a5c156810 0x7f2a5c06e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.188+0000 7f2a611ff700 1 -- 172.21.15.27:0/277872170 >> 172.21.15.27:0/277872170 conn(0x7f2a5c0680e0 msgr2=0x7f2a5c0cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.188+0000 7f2a611ff700 1 -- 172.21.15.27:0/277872170 shutdown_connections 2024-01-11T09:27:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.188+0000 7f2a611ff700 1 -- 172.21.15.27:0/277872170 wait complete. 2024-01-11T09:27:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.189+0000 7f2a611ff700 1 Processor -- start 2024-01-11T09:27:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.189+0000 7f2a611ff700 1 -- start start 2024-01-11T09:27:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.190+0000 7f2a611ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a5c0ccac0 0x7f2a5c0b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.190+0000 7f2a611ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a5c156440 0x7f2a5c0a98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.190+0000 7f2a611ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a5c156810 0x7f2a5c0af900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.190+0000 7f2a611ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5c0b12d0 con 0x7f2a5c0ccac0 2024-01-11T09:27:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.190+0000 7f2a611ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5c0b03b0 con 0x7f2a5c156810 2024-01-11T09:27:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.190+0000 7f2a611ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5c159660 con 0x7f2a5c156440 2024-01-11T09:27:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.191+0000 7f2a5ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a5c0ccac0 0x7f2a5c0b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.191+0000 7f2a5ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a5c0ccac0 0x7f2a5c0b1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36148/0 (socket says 172.21.15.27:36148) 2024-01-11T09:27:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.191+0000 7f2a5ad9d700 1 -- 172.21.15.27:0/1688518560 learned_addr learned my addr 172.21.15.27:0/1688518560 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.191+0000 7f2a5a59c700 1 --2- 172.21.15.27:0/1688518560 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a5c156440 0x7f2a5c0a98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.191+0000 7f2a5ad9d700 1 -- 172.21.15.27:0/1688518560 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a5c156440 msgr2=0x7f2a5c0a98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.191+0000 7f2a5ad9d700 1 --2- 172.21.15.27:0/1688518560 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a5c156440 0x7f2a5c0a98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.191+0000 7f2a5ad9d700 1 -- 172.21.15.27:0/1688518560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a5c156810 msgr2=0x7f2a5c0af900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.191+0000 7f2a5ad9d700 1 --2- 172.21.15.27:0/1688518560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a5c156810 0x7f2a5c0af900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.191+0000 7f2a5ad9d700 1 -- 172.21.15.27:0/1688518560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5400bcf0 con 0x7f2a5c0ccac0 2024-01-11T09:27:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.191+0000 7f2a5ad9d700 1 --2- 172.21.15.27:0/1688518560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a5c0ccac0 0x7f2a5c0b1c30 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c00c5b0 tx=0x7f2a4c0020e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.192+0000 7f2a42ffd700 1 -- 172.21.15.27:0/1688518560 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c011590 con 0x7f2a5c0ccac0 2024-01-11T09:27:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.192+0000 7f2a42ffd700 1 -- 172.21.15.27:0/1688518560 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a4c0116f0 con 0x7f2a5c0ccac0 2024-01-11T09:27:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.192+0000 7f2a42ffd700 1 -- 172.21.15.27:0/1688518560 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c011a90 con 0x7f2a5c0ccac0 2024-01-11T09:27:09.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.192+0000 7f2a611ff700 1 -- 172.21.15.27:0/1688518560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a5c0c4cc0 con 0x7f2a5c0ccac0 2024-01-11T09:27:09.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.192+0000 7f2a611ff700 1 -- 172.21.15.27:0/1688518560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a5c0afe40 con 0x7f2a5c0ccac0 2024-01-11T09:27:09.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.193+0000 7f2a611ff700 1 -- 172.21.15.27:0/1688518560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a5c04c3c0 con 0x7f2a5c0ccac0 2024-01-11T09:27:09.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.194+0000 7f2a42ffd700 1 -- 172.21.15.27:0/1688518560 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 97) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f2a4c01c460 con 0x7f2a5c0ccac0 2024-01-11T09:27:09.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.195+0000 7f2a42ffd700 1 --2- 172.21.15.27:0/1688518560 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2a48066db0 0x7f2a48069260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:09.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.195+0000 7f2a42ffd700 1 -- 172.21.15.27:0/1688518560 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5925+0+0 (secure 0 0 0) 0x7f2a4c0840d0 con 0x7f2a5c0ccac0 2024-01-11T09:27:09.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.195+0000 7f2a5a59c700 1 --2- 172.21.15.27:0/1688518560 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2a48066db0 0x7f2a48069260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:09.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.196+0000 7f2a5a59c700 1 --2- 172.21.15.27:0/1688518560 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2a48066db0 0x7f2a48069260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2a50006fd0 tx=0x7f2a50008040 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:09.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.199+0000 7f2a42ffd700 1 -- 172.21.15.27:0/1688518560 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2a4c051440 con 0x7f2a5c0ccac0 2024-01-11T09:27:09.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.409+0000 7f2a611ff700 1 -- 172.21.15.27:0/1688518560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2a5c06dcb0 con 0x7f2a5c0ccac0 2024-01-11T09:27:09.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:09.914+0000 7f2a42ffd700 1 -- 172.21.15.27:0/1688518560 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 98) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f2a4c051810 con 0x7f2a5c0ccac0 2024-01-11T09:27:10.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.112+0000 7f2a42ffd700 1 -- 172.21.15.27:0/1688518560 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v172) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2a4c054ed0 con 0x7f2a5c0ccac0 2024-01-11T09:27:10.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.115+0000 7f2a611ff700 1 -- 172.21.15.27:0/1688518560 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2a48066db0 msgr2=0x7f2a48069260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:10.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.115+0000 7f2a611ff700 1 --2- 172.21.15.27:0/1688518560 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2a48066db0 0x7f2a48069260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2a50006fd0 tx=0x7f2a50008040 comp rx=0 tx=0).stop 2024-01-11T09:27:10.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.115+0000 7f2a611ff700 1 -- 172.21.15.27:0/1688518560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a5c0ccac0 msgr2=0x7f2a5c0b1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:10.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.115+0000 7f2a611ff700 1 --2- 172.21.15.27:0/1688518560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a5c0ccac0 0x7f2a5c0b1c30 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c00c5b0 tx=0x7f2a4c0020e0 comp rx=0 tx=0).stop 2024-01-11T09:27:10.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.115+0000 7f2a611ff700 1 -- 172.21.15.27:0/1688518560 shutdown_connections 2024-01-11T09:27:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.115+0000 7f2a611ff700 1 --2- 172.21.15.27:0/1688518560 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2a48066db0 0x7f2a48069260 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.115+0000 7f2a611ff700 1 --2- 172.21.15.27:0/1688518560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a5c156810 0x7f2a5c0af900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.115+0000 7f2a611ff700 1 --2- 172.21.15.27:0/1688518560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a5c0ccac0 0x7f2a5c0b1c30 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.115+0000 7f2a611ff700 1 --2- 172.21.15.27:0/1688518560 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a5c156440 0x7f2a5c0a98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.115+0000 7f2a611ff700 1 -- 172.21.15.27:0/1688518560 >> 172.21.15.27:0/1688518560 conn(0x7f2a5c0680e0 msgr2=0x7f2a5c0ad170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.116+0000 7f2a611ff700 1 -- 172.21.15.27:0/1688518560 shutdown_connections 2024-01-11T09:27:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.116+0000 7f2a611ff700 1 -- 172.21.15.27:0/1688518560 wait complete. 2024-01-11T09:27:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-01-11T09:27:10.132 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-11T09:27:10.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.340+0000 7fef5c1ff700 1 Processor -- start 2024-01-11T09:27:10.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.340+0000 7fef5c1ff700 1 -- start start 2024-01-11T09:27:10.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.341+0000 7fef5c1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fef5413fbb0 0x7fef540ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:10.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.341+0000 7fef5c1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fef5415a730 0x7fef540c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:10.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.341+0000 7fef5c1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef540c6b30 con 0x7fef5413ff80 2024-01-11T09:27:10.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.341+0000 7fef5c1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef540c6800 con 0x7fef5415a730 2024-01-11T09:27:10.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.341+0000 7fef5c1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef540cb240 con 0x7fef5413fbb0 2024-01-11T09:27:10.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.341+0000 7fef5a79c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fef5415a730 0x7fef540c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:10.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.341+0000 7fef5a79c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fef5415a730 0x7fef540c6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36154/0 (socket says 172.21.15.27:36154) 2024-01-11T09:27:10.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.341+0000 7fef5a79c700 1 -- 172.21.15.27:0/3813837714 learned_addr learned my addr 172.21.15.27:0/3813837714 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:10.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.342+0000 7fef59f9b700 1 --2- 172.21.15.27:0/3813837714 >> v2:172.21.15.77:3300/0 conn(0x7fef5413fbb0 0x7fef540ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:10.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.342+0000 7fef5a79c700 1 -- 172.21.15.27:0/3813837714 >> v2:172.21.15.77:3300/0 conn(0x7fef5413fbb0 msgr2=0x7fef540ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.342+0000 7fef5a79c700 1 --2- 172.21.15.27:0/3813837714 >> v2:172.21.15.77:3300/0 conn(0x7fef5413fbb0 0x7fef540ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.342+0000 7fef5a79c700 1 -- 172.21.15.27:0/3813837714 >> v1:172.21.15.77:6789/0 conn(0x7fef5413ff80 legacy=0x7fef5415a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.342+0000 7fef5a79c700 1 -- 172.21.15.27:0/3813837714 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef540cb520 con 0x7fef5415a730 2024-01-11T09:27:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.342+0000 7fef5a79c700 1 --2- 172.21.15.27:0/3813837714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5415a730 0x7fef540c6fc0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fef54064df0 tx=0x7fef4c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6f3a75db1169617 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.342+0000 7fef58f99700 1 -- 172.21.15.27:0/3813837714 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c00e070 con 0x7fef5415a730 2024-01-11T09:27:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.342+0000 7fef58f99700 1 -- 172.21.15.27:0/3813837714 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4c00ba10 con 0x7fef5415a730 2024-01-11T09:27:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.342+0000 7fef58f99700 1 -- 172.21.15.27:0/3813837714 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c0092e0 con 0x7fef5415a730 2024-01-11T09:27:10.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.343+0000 7fef5c1ff700 1 -- 172.21.15.27:0/3813837714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5415a730 msgr2=0x7fef540c6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:10.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.343+0000 7fef5c1ff700 1 --2- 172.21.15.27:0/3813837714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5415a730 0x7fef540c6fc0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fef54064df0 tx=0x7fef4c009840 comp rx=0 tx=0).stop 2024-01-11T09:27:10.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.343+0000 7fef5c1ff700 1 -- 172.21.15.27:0/3813837714 shutdown_connections 2024-01-11T09:27:10.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.343+0000 7fef5c1ff700 1 --2- 172.21.15.27:0/3813837714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5415a730 0x7fef540c6fc0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:10.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.343+0000 7fef5c1ff700 1 --2- 172.21.15.27:0/3813837714 >> v2:172.21.15.77:3300/0 conn(0x7fef5413fbb0 0x7fef540ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:10.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.343+0000 7fef5c1ff700 1 -- 172.21.15.27:0/3813837714 >> 172.21.15.27:0/3813837714 conn(0x7fef540680a0 msgr2=0x7fef540cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:10.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.343+0000 7fef5c1ff700 1 -- 172.21.15.27:0/3813837714 shutdown_connections 2024-01-11T09:27:10.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.343+0000 7fef5c1ff700 1 -- 172.21.15.27:0/3813837714 wait complete. 2024-01-11T09:27:10.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.345+0000 7fef5c1ff700 1 Processor -- start 2024-01-11T09:27:10.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.345+0000 7fef5c1ff700 1 -- start start 2024-01-11T09:27:10.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.346+0000 7fef5c1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5413fbb0 0x7fef5412f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:10.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.346+0000 7fef5c1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef5413ff80 0x7fef5415be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:10.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.346+0000 7fef5c1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef5415a730 0x7fef5413d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:10.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.346+0000 7fef5c1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5415e460 con 0x7fef5413fbb0 2024-01-11T09:27:10.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.346+0000 7fef5c1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5415e670 con 0x7fef5415a730 2024-01-11T09:27:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.346+0000 7fef5c1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5415e950 con 0x7fef5413ff80 2024-01-11T09:27:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.346+0000 7fef59f9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5413fbb0 0x7fef5412f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.346+0000 7fef59f9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5413fbb0 0x7fef5412f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36170/0 (socket says 172.21.15.27:36170) 2024-01-11T09:27:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.346+0000 7fef59f9b700 1 -- 172.21.15.27:0/3048831981 learned_addr learned my addr 172.21.15.27:0/3048831981 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.346+0000 7fef5a79c700 1 --2- 172.21.15.27:0/3048831981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef5415a730 0x7fef5413d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.346+0000 7fef5979a700 1 --2- 172.21.15.27:0/3048831981 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef5413ff80 0x7fef5415be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.346+0000 7fef59f9b700 1 -- 172.21.15.27:0/3048831981 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef5413ff80 msgr2=0x7fef5415be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.346+0000 7fef59f9b700 1 --2- 172.21.15.27:0/3048831981 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef5413ff80 0x7fef5415be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.346+0000 7fef59f9b700 1 -- 172.21.15.27:0/3048831981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef5415a730 msgr2=0x7fef5413d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.346+0000 7fef59f9b700 1 --2- 172.21.15.27:0/3048831981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef5415a730 0x7fef5413d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.346+0000 7fef59f9b700 1 -- 172.21.15.27:0/3048831981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4c009ae0 con 0x7fef5413fbb0 2024-01-11T09:27:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.346+0000 7fef5a79c700 1 --2- 172.21.15.27:0/3048831981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef5415a730 0x7fef5413d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:27:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.347+0000 7fef59f9b700 1 --2- 172.21.15.27:0/3048831981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5413fbb0 0x7fef5412f870 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fef5000d890 tx=0x7fef50008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:10.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.347+0000 7fef49ffb700 1 -- 172.21.15.27:0/3048831981 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef50018070 con 0x7fef5413fbb0 2024-01-11T09:27:10.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.347+0000 7fef49ffb700 1 -- 172.21.15.27:0/3048831981 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef50005a00 con 0x7fef5413fbb0 2024-01-11T09:27:10.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.347+0000 7fef5c1ff700 1 -- 172.21.15.27:0/3048831981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef5415ec30 con 0x7fef5413fbb0 2024-01-11T09:27:10.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.347+0000 7fef49ffb700 1 -- 172.21.15.27:0/3048831981 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef50005dc0 con 0x7fef5413fbb0 2024-01-11T09:27:10.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.347+0000 7fef5c1ff700 1 -- 172.21.15.27:0/3048831981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5412fdb0 con 0x7fef5413fbb0 2024-01-11T09:27:10.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.348+0000 7fef5c1ff700 1 -- 172.21.15.27:0/3048831981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef5412db10 con 0x7fef5413fbb0 2024-01-11T09:27:10.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.349+0000 7fef49ffb700 1 -- 172.21.15.27:0/3048831981 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 98) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fef5000b810 con 0x7fef5413fbb0 2024-01-11T09:27:10.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.350+0000 7fef49ffb700 1 --2- 172.21.15.27:0/3048831981 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fef38066ea0 0x7fef38069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:10.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.350+0000 7fef49ffb700 1 -- 172.21.15.27:0/3048831981 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5511+0+0 (secure 0 0 0) 0x7fef50084510 con 0x7fef5413fbb0 2024-01-11T09:27:10.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.351+0000 7fef5979a700 1 --2- 172.21.15.27:0/3048831981 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fef38066ea0 0x7fef38069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:10.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.351+0000 7fef5979a700 1 --2- 172.21.15.27:0/3048831981 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fef38066ea0 0x7fef38069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fef44003870 tx=0x7fef44009040 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:10.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.354+0000 7fef49ffb700 1 -- 172.21.15.27:0/3048831981 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fef5412db10 con 0x7fef5413fbb0 2024-01-11T09:27:10.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.557+0000 7fef5c1ff700 1 -- 172.21.15.27:0/3048831981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fef5412db10 con 0x7fef5413fbb0 2024-01-11T09:27:10.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.561+0000 7fef49ffb700 1 -- 172.21.15.27:0/3048831981 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v61) v1 ==== 120+0+0 (secure 0 0 0) 0x7fef5412db10 con 0x7fef5413fbb0 2024-01-11T09:27:10.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.563+0000 7fef5c1ff700 1 -- 172.21.15.27:0/3048831981 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fef38066ea0 msgr2=0x7fef38069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:10.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.563+0000 7fef5c1ff700 1 --2- 172.21.15.27:0/3048831981 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fef38066ea0 0x7fef38069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fef44003870 tx=0x7fef44009040 comp rx=0 tx=0).stop 2024-01-11T09:27:10.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.563+0000 7fef5c1ff700 1 -- 172.21.15.27:0/3048831981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5413fbb0 msgr2=0x7fef5412f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:10.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.563+0000 7fef5c1ff700 1 --2- 172.21.15.27:0/3048831981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5413fbb0 0x7fef5412f870 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fef5000d890 tx=0x7fef50008040 comp rx=0 tx=0).stop 2024-01-11T09:27:10.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.563+0000 7fef5c1ff700 1 -- 172.21.15.27:0/3048831981 shutdown_connections 2024-01-11T09:27:10.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.563+0000 7fef5c1ff700 1 --2- 172.21.15.27:0/3048831981 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fef38066ea0 0x7fef38069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:10.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.563+0000 7fef5c1ff700 1 --2- 172.21.15.27:0/3048831981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef5415a730 0x7fef5413d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:10.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.563+0000 7fef5c1ff700 1 --2- 172.21.15.27:0/3048831981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef5413fbb0 0x7fef5412f870 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:10.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.563+0000 7fef5c1ff700 1 --2- 172.21.15.27:0/3048831981 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef5413ff80 0x7fef5415be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:10.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.563+0000 7fef5c1ff700 1 -- 172.21.15.27:0/3048831981 >> 172.21.15.27:0/3048831981 conn(0x7fef540680a0 msgr2=0x7fef5413ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:10.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.564+0000 7fef5c1ff700 1 -- 172.21.15.27:0/3048831981 shutdown_connections 2024-01-11T09:27:10.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.564+0000 7fef5c1ff700 1 -- 172.21.15.27:0/3048831981 wait complete. 2024-01-11T09:27:10.578 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-11T09:27:10.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.783+0000 7fd5b51ff700 1 Processor -- start 2024-01-11T09:27:10.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.783+0000 7fd5b51ff700 1 -- start start 2024-01-11T09:27:10.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.784+0000 7fd5b51ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd5b01622d0 0x7fd5b01646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:10.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.784+0000 7fd5b51ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b00cf850 con 0x7fd5b015a1b0 2024-01-11T09:27:10.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.784+0000 7fd5b51ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b00cec40 con 0x7fd5b00ccd70 2024-01-11T09:27:10.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.784+0000 7fd5b51ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5b015ac90 con 0x7fd5b01622d0 2024-01-11T09:27:10.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.785+0000 7fd5af59e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd5b01622d0 0x7fd5b01646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:10.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.785+0000 7fd5af59e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd5b01622d0 0x7fd5b01646b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36186/0 (socket says 172.21.15.27:36186) 2024-01-11T09:27:10.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.785+0000 7fd5af59e700 1 -- 172.21.15.27:0/1769642137 learned_addr learned my addr 172.21.15.27:0/1769642137 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:10.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.785+0000 7fd5aed9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd5b015a1b0 0x7fd5b015a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40482/0 (socket says 172.21.15.27:40482) 2024-01-11T09:27:10.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.785+0000 7fd5af59e700 1 -- 172.21.15.27:0/1769642137 >> v1:172.21.15.77:6789/0 conn(0x7fd5b00ccd70 legacy=0x7fd5b00cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:10.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.785+0000 7fd5af59e700 1 -- 172.21.15.27:0/1769642137 >> v1:172.21.15.27:6789/0 conn(0x7fd5b015a1b0 legacy=0x7fd5b015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:10.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.785+0000 7fd5af59e700 1 -- 172.21.15.27:0/1769642137 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b00cef20 con 0x7fd5b01622d0 2024-01-11T09:27:10.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.785+0000 7fd5add9b700 1 -- 172.21.15.27:0/1769642137 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872741350 0 0) 0x7fd5b00cf850 con 0x7fd5b015a1b0 2024-01-11T09:27:10.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.785+0000 7fd5af59e700 1 --2- 172.21.15.27:0/1769642137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5b01622d0 0x7fd5b01646b0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009f20 tx=0x7fd5a8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9407901228eb3e96 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:10.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.786+0000 7fd5add9b700 1 -- 172.21.15.27:0/1769642137 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a800e070 con 0x7fd5b01622d0 2024-01-11T09:27:10.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.786+0000 7fd5add9b700 1 -- 172.21.15.27:0/1769642137 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a800ba80 con 0x7fd5b01622d0 2024-01-11T09:27:10.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.786+0000 7fd5add9b700 1 -- 172.21.15.27:0/1769642137 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a80092e0 con 0x7fd5b01622d0 2024-01-11T09:27:10.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.786+0000 7fd5b51ff700 1 -- 172.21.15.27:0/1769642137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5b01622d0 msgr2=0x7fd5b01646b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:10.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.786+0000 7fd5b51ff700 1 --2- 172.21.15.27:0/1769642137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5b01622d0 0x7fd5b01646b0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009f20 tx=0x7fd5a8009840 comp rx=0 tx=0).stop 2024-01-11T09:27:10.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.786+0000 7fd5b51ff700 1 -- 172.21.15.27:0/1769642137 shutdown_connections 2024-01-11T09:27:10.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.786+0000 7fd5b51ff700 1 --2- 172.21.15.27:0/1769642137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5b01622d0 0x7fd5b01646b0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:10.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.786+0000 7fd5b51ff700 1 -- 172.21.15.27:0/1769642137 >> 172.21.15.27:0/1769642137 conn(0x7fd5b006ed40 msgr2=0x7fd5b006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:10.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.786+0000 7fd5b51ff700 1 -- 172.21.15.27:0/1769642137 shutdown_connections 2024-01-11T09:27:10.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.786+0000 7fd5b51ff700 1 -- 172.21.15.27:0/1769642137 wait complete. 2024-01-11T09:27:10.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.788+0000 7fd5b51ff700 1 Processor -- start 2024-01-11T09:27:10.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.788+0000 7fd5b51ff700 1 -- start start 2024-01-11T09:27:10.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.789+0000 7fd5b51ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd5b00ccd70 0x7fd5b014ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:10.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.789+0000 7fd5b51ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5b015a1b0 0x7fd5b00c9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:10.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.789+0000 7fd5b51ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5b01622d0 0x7fd5b00c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:10.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.789+0000 7fd5b51ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b0150b80 con 0x7fd5b015a1b0 2024-01-11T09:27:10.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.789+0000 7fd5b51ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b0151980 con 0x7fd5b01622d0 2024-01-11T09:27:10.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.789+0000 7fd5b51ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b0150db0 con 0x7fd5b00ccd70 2024-01-11T09:27:10.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.789+0000 7fd5ae59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5b015a1b0 0x7fd5b00c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:10.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.789+0000 7fd5ae59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5b015a1b0 0x7fd5b00c9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36198/0 (socket says 172.21.15.27:36198) 2024-01-11T09:27:10.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.789+0000 7fd5ae59c700 1 -- 172.21.15.27:0/1259674160 learned_addr learned my addr 172.21.15.27:0/1259674160 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:10.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.790+0000 7fd5ae59c700 1 -- 172.21.15.27:0/1259674160 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd5b00ccd70 msgr2=0x7fd5b014ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:10.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.790+0000 7fd5ae59c700 1 --2- 172.21.15.27:0/1259674160 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd5b00ccd70 0x7fd5b014ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:10.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.790+0000 7fd5ae59c700 1 -- 172.21.15.27:0/1259674160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5b01622d0 msgr2=0x7fd5b00c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:10.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.790+0000 7fd5ae59c700 1 --2- 172.21.15.27:0/1259674160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5b01622d0 0x7fd5b00c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:10.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.790+0000 7fd5ae59c700 1 -- 172.21.15.27:0/1259674160 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a8009ae0 con 0x7fd5b015a1b0 2024-01-11T09:27:10.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.790+0000 7fd5ae59c700 1 --2- 172.21.15.27:0/1259674160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5b015a1b0 0x7fd5b00c9070 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fd5a400c4e0 tx=0x7fd5a400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:10.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.790+0000 7fd596ffd700 1 -- 172.21.15.27:0/1259674160 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a4011590 con 0x7fd5b015a1b0 2024-01-11T09:27:10.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.790+0000 7fd596ffd700 1 -- 172.21.15.27:0/1259674160 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a4011710 con 0x7fd5b015a1b0 2024-01-11T09:27:10.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.790+0000 7fd5b51ff700 1 -- 172.21.15.27:0/1259674160 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b0151050 con 0x7fd5b015a1b0 2024-01-11T09:27:10.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.790+0000 7fd596ffd700 1 -- 172.21.15.27:0/1259674160 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a4011ad0 con 0x7fd5b015a1b0 2024-01-11T09:27:10.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.790+0000 7fd5b51ff700 1 -- 172.21.15.27:0/1259674160 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b0151610 con 0x7fd5b015a1b0 2024-01-11T09:27:10.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.791+0000 7fd5b51ff700 1 -- 172.21.15.27:0/1259674160 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b004c3c0 con 0x7fd5b015a1b0 2024-01-11T09:27:10.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.792+0000 7fd596ffd700 1 -- 172.21.15.27:0/1259674160 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 98) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fd5a401b3b0 con 0x7fd5b015a1b0 2024-01-11T09:27:10.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.793+0000 7fd596ffd700 1 --2- 172.21.15.27:0/1259674160 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd59c066ea0 0x7fd59c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:10.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.793+0000 7fd596ffd700 1 -- 172.21.15.27:0/1259674160 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5511+0+0 (secure 0 0 0) 0x7fd5a4083930 con 0x7fd5b015a1b0 2024-01-11T09:27:10.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.793+0000 7fd5aed9d700 1 --2- 172.21.15.27:0/1259674160 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd59c066ea0 0x7fd59c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:10.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.794+0000 7fd5aed9d700 1 --2- 172.21.15.27:0/1259674160 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd59c066ea0 0x7fd59c069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd5a00011e0 tx=0x7fd5a0007990 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:10.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:10.797+0000 7fd596ffd700 1 -- 172.21.15.27:0/1259674160 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd5a4050ca0 con 0x7fd5b015a1b0 2024-01-11T09:27:10.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.003+0000 7fd5b51ff700 1 -- 172.21.15.27:0/1259674160 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fd5b0151af0 con 0x7fd5b015a1b0 2024-01-11T09:27:11.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.007+0000 7fd596ffd700 1 -- 172.21.15.27:0/1259674160 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v62) v1 ==== 115+0+0 (secure 0 0 0) 0x7fd5a4017070 con 0x7fd5b015a1b0 2024-01-11T09:27:11.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.009+0000 7fd5b51ff700 1 -- 172.21.15.27:0/1259674160 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd59c066ea0 msgr2=0x7fd59c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:11.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.009+0000 7fd5b51ff700 1 --2- 172.21.15.27:0/1259674160 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd59c066ea0 0x7fd59c069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd5a00011e0 tx=0x7fd5a0007990 comp rx=0 tx=0).stop 2024-01-11T09:27:11.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.009+0000 7fd5b51ff700 1 -- 172.21.15.27:0/1259674160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5b015a1b0 msgr2=0x7fd5b00c9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:11.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.009+0000 7fd5b51ff700 1 --2- 172.21.15.27:0/1259674160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5b015a1b0 0x7fd5b00c9070 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fd5a400c4e0 tx=0x7fd5a400c8a0 comp rx=0 tx=0).stop 2024-01-11T09:27:11.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.009+0000 7fd5b51ff700 1 -- 172.21.15.27:0/1259674160 shutdown_connections 2024-01-11T09:27:11.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.009+0000 7fd5b51ff700 1 --2- 172.21.15.27:0/1259674160 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd59c066ea0 0x7fd59c069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.009+0000 7fd5b51ff700 1 --2- 172.21.15.27:0/1259674160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5b01622d0 0x7fd5b00c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.009+0000 7fd5b51ff700 1 --2- 172.21.15.27:0/1259674160 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd5b00ccd70 0x7fd5b014ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.009+0000 7fd5b51ff700 1 --2- 172.21.15.27:0/1259674160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5b015a1b0 0x7fd5b00c9070 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.009+0000 7fd5b51ff700 1 -- 172.21.15.27:0/1259674160 >> 172.21.15.27:0/1259674160 conn(0x7fd5b006ed40 msgr2=0x7fd5b00b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.009+0000 7fd5b51ff700 1 -- 172.21.15.27:0/1259674160 shutdown_connections 2024-01-11T09:27:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.009+0000 7fd5b51ff700 1 -- 172.21.15.27:0/1259674160 wait complete. 2024-01-11T09:27:11.024 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-11T09:27:11.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.234+0000 7f14e97ff700 1 Processor -- start 2024-01-11T09:27:11.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.235+0000 7f14e97ff700 1 -- start start 2024-01-11T09:27:11.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.236+0000 7f14e97ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14e415a1b0 0x7f14e415a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:11.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.236+0000 7f14e97ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e40cf850 con 0x7f14e40ccd70 2024-01-11T09:27:11.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.236+0000 7f14e97ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e40cec40 con 0x7f14e415ae70 2024-01-11T09:27:11.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.236+0000 7f14e97ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e40cef20 con 0x7f14e415a1b0 2024-01-11T09:27:11.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.236+0000 7f14e2ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14e415a1b0 0x7f14e415a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:11.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.236+0000 7f14e27fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f14e40ccd70 0x7f14e40cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40484/0 (socket says 172.21.15.27:40484) 2024-01-11T09:27:11.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.236+0000 7f14e27fc700 1 -- 172.21.15.27:0/153862552 learned_addr learned my addr 172.21.15.27:0/153862552 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.237+0000 7f14e1ffb700 1 -- 172.21.15.27:0/153862552 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172831317 0 0) 0x7f14e40cf850 con 0x7f14e40ccd70 2024-01-11T09:27:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.237+0000 7f14e1ffb700 1 -- 172.21.15.27:0/153862552 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14cc003570 con 0x7f14e40ccd70 2024-01-11T09:27:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.237+0000 7f14e2ffd700 1 -- 172.21.15.27:0/153862552 >> v1:172.21.15.77:6789/0 conn(0x7f14e415ae70 legacy=0x7f14e415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.237+0000 7f14e2ffd700 1 -- 172.21.15.27:0/153862552 >> v1:172.21.15.27:6789/0 conn(0x7f14e40ccd70 legacy=0x7f14e40cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.237+0000 7f14e2ffd700 1 -- 172.21.15.27:0/153862552 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e40cf200 con 0x7f14e415a1b0 2024-01-11T09:27:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.237+0000 7f14e2ffd700 1 --2- 172.21.15.27:0/153862552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e415a1b0 0x7f14e415a580 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f14d40098e0 tx=0x7f14d40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dceb7c67dcbea129 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.237+0000 7f14e1ffb700 1 -- 172.21.15.27:0/153862552 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d400b640 con 0x7f14e415a1b0 2024-01-11T09:27:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.237+0000 7f14e1ffb700 1 -- 172.21.15.27:0/153862552 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d400b7a0 con 0x7f14e415a1b0 2024-01-11T09:27:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.237+0000 7f14e1ffb700 1 -- 172.21.15.27:0/153862552 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d400bb40 con 0x7f14e415a1b0 2024-01-11T09:27:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.238+0000 7f14e97ff700 1 -- 172.21.15.27:0/153862552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e415a1b0 msgr2=0x7f14e415a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.238+0000 7f14e97ff700 1 --2- 172.21.15.27:0/153862552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e415a1b0 0x7f14e415a580 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f14d40098e0 tx=0x7f14d40091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.238+0000 7f14e97ff700 1 -- 172.21.15.27:0/153862552 shutdown_connections 2024-01-11T09:27:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.238+0000 7f14e97ff700 1 --2- 172.21.15.27:0/153862552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e415a1b0 0x7f14e415a580 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:11.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.238+0000 7f14e97ff700 1 -- 172.21.15.27:0/153862552 >> 172.21.15.27:0/153862552 conn(0x7f14e406ed40 msgr2=0x7f14e406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:11.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.238+0000 7f14e97ff700 1 -- 172.21.15.27:0/153862552 shutdown_connections 2024-01-11T09:27:11.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.238+0000 7f14e97ff700 1 -- 172.21.15.27:0/153862552 wait complete. 2024-01-11T09:27:11.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.240+0000 7f14e97ff700 1 Processor -- start 2024-01-11T09:27:11.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.240+0000 7f14e97ff700 1 -- start start 2024-01-11T09:27:11.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.241+0000 7f14e97ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14e40ccd70 0x7f14e415d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:11.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.241+0000 7f14e97ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e415a1b0 0x7f14e40c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.241+0000 7f14e97ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14e415ae70 0x7f14e40ca410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.241+0000 7f14e97ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e40ca950 con 0x7f14e415a1b0 2024-01-11T09:27:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.241+0000 7f14e97ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e4158d90 con 0x7f14e415ae70 2024-01-11T09:27:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.241+0000 7f14e97ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e40c9f80 con 0x7f14e40ccd70 2024-01-11T09:27:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.241+0000 7f14e27fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e415a1b0 0x7f14e40c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.241+0000 7f14e27fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e415a1b0 0x7f14e40c8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36214/0 (socket says 172.21.15.27:36214) 2024-01-11T09:27:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.241+0000 7f14e27fc700 1 -- 172.21.15.27:0/2094486106 learned_addr learned my addr 172.21.15.27:0/2094486106 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.241+0000 7f14e2ffd700 1 --2- 172.21.15.27:0/2094486106 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14e40ccd70 0x7f14e415d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.241+0000 7f14e37fe700 1 --2- 172.21.15.27:0/2094486106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14e415ae70 0x7f14e40ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.241+0000 7f14e27fc700 1 -- 172.21.15.27:0/2094486106 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14e40ccd70 msgr2=0x7f14e415d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.241+0000 7f14e27fc700 1 --2- 172.21.15.27:0/2094486106 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14e40ccd70 0x7f14e415d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.241+0000 7f14e27fc700 1 -- 172.21.15.27:0/2094486106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14e415ae70 msgr2=0x7f14e40ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.241+0000 7f14e27fc700 1 --2- 172.21.15.27:0/2094486106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14e415ae70 0x7f14e40ca410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.241+0000 7f14e27fc700 1 -- 172.21.15.27:0/2094486106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d4009420 con 0x7f14e415a1b0 2024-01-11T09:27:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.241+0000 7f14e37fe700 1 --2- 172.21.15.27:0/2094486106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14e415ae70 0x7f14e40ca410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:27:11.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.242+0000 7f14e27fc700 1 --2- 172.21.15.27:0/2094486106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e415a1b0 0x7f14e40c8ce0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f14d800c590 tx=0x7f14d8007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:11.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.242+0000 7f14caffd700 1 -- 172.21.15.27:0/2094486106 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d8005b30 con 0x7f14e415a1b0 2024-01-11T09:27:11.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.242+0000 7f14caffd700 1 -- 172.21.15.27:0/2094486106 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d8005cb0 con 0x7f14e415a1b0 2024-01-11T09:27:11.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.242+0000 7f14caffd700 1 -- 172.21.15.27:0/2094486106 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d8007320 con 0x7f14e415a1b0 2024-01-11T09:27:11.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.242+0000 7f14e97ff700 1 -- 172.21.15.27:0/2094486106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e40c9370 con 0x7f14e415a1b0 2024-01-11T09:27:11.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.242+0000 7f14e97ff700 1 -- 172.21.15.27:0/2094486106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e40c9930 con 0x7f14e415a1b0 2024-01-11T09:27:11.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.243+0000 7f14e97ff700 1 -- 172.21.15.27:0/2094486106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e40ca130 con 0x7f14e415a1b0 2024-01-11T09:27:11.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.244+0000 7f14caffd700 1 -- 172.21.15.27:0/2094486106 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 98) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f14d801b3b0 con 0x7f14e415a1b0 2024-01-11T09:27:11.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.245+0000 7f14caffd700 1 --2- 172.21.15.27:0/2094486106 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f14d0066ea0 0x7f14d0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:11.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.245+0000 7f14caffd700 1 -- 172.21.15.27:0/2094486106 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5511+0+0 (secure 0 0 0) 0x7f14d8083970 con 0x7f14e415a1b0 2024-01-11T09:27:11.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.246+0000 7f14e2ffd700 1 --2- 172.21.15.27:0/2094486106 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f14d0066ea0 0x7f14d0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:11.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.247+0000 7f14e2ffd700 1 --2- 172.21.15.27:0/2094486106 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f14d0066ea0 0x7f14d0069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f14d4003080 tx=0x7f14d4017000 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:11.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.249+0000 7f14caffd700 1 -- 172.21.15.27:0/2094486106 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f14e40ca130 con 0x7f14e415a1b0 2024-01-11T09:27:11.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.452+0000 7f14e97ff700 1 -- 172.21.15.27:0/2094486106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f14e40b2db0 con 0x7f14e415a1b0 2024-01-11T09:27:11.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.456+0000 7f14caffd700 1 -- 172.21.15.27:0/2094486106 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v63) v1 ==== 104+0+0 (secure 0 0 0) 0x7f14d80170a0 con 0x7f14e415a1b0 2024-01-11T09:27:11.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.458+0000 7f14e97ff700 1 -- 172.21.15.27:0/2094486106 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f14d0066ea0 msgr2=0x7f14d0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.458+0000 7f14e97ff700 1 --2- 172.21.15.27:0/2094486106 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f14d0066ea0 0x7f14d0069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f14d4003080 tx=0x7f14d4017000 comp rx=0 tx=0).stop 2024-01-11T09:27:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.459+0000 7f14e97ff700 1 -- 172.21.15.27:0/2094486106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e415a1b0 msgr2=0x7f14e40c8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.459+0000 7f14e97ff700 1 --2- 172.21.15.27:0/2094486106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e415a1b0 0x7f14e40c8ce0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f14d800c590 tx=0x7f14d8007e00 comp rx=0 tx=0).stop 2024-01-11T09:27:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.459+0000 7f14e97ff700 1 -- 172.21.15.27:0/2094486106 shutdown_connections 2024-01-11T09:27:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.459+0000 7f14e97ff700 1 --2- 172.21.15.27:0/2094486106 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f14d0066ea0 0x7f14d0069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.459+0000 7f14e97ff700 1 --2- 172.21.15.27:0/2094486106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14e415ae70 0x7f14e40ca410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.459+0000 7f14e97ff700 1 --2- 172.21.15.27:0/2094486106 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14e40ccd70 0x7f14e415d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.459+0000 7f14e97ff700 1 --2- 172.21.15.27:0/2094486106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e415a1b0 0x7f14e40c8ce0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.459+0000 7f14e97ff700 1 -- 172.21.15.27:0/2094486106 >> 172.21.15.27:0/2094486106 conn(0x7f14e406ed40 msgr2=0x7f14e415b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.459+0000 7f14e97ff700 1 -- 172.21.15.27:0/2094486106 shutdown_connections 2024-01-11T09:27:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.459+0000 7f14e97ff700 1 -- 172.21.15.27:0/2094486106 wait complete. 2024-01-11T09:27:11.473 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2024-01-11T09:27:11.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.676+0000 7f88497ff700 1 Processor -- start 2024-01-11T09:27:11.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.676+0000 7f88497ff700 1 -- start start 2024-01-11T09:27:11.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.677+0000 7f88497ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b2b90 0x7f8844159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:11.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.677+0000 7f88497ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f88440b07b0 0x7f8844159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:11.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.677+0000 7f88497ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88440cd730 con 0x7f88440b03e0 2024-01-11T09:27:11.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.677+0000 7f88497ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88440ccb20 con 0x7f88440b07b0 2024-01-11T09:27:11.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.677+0000 7f88497ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88440cce00 con 0x7f88440b2b90 2024-01-11T09:27:11.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.678+0000 7f88427fc700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b2b90 0x7f8844159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.678+0000 7f88427fc700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b2b90 0x7f8844159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:35454/0 (socket says 172.21.15.27:35454) 2024-01-11T09:27:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.678+0000 7f88427fc700 1 -- 172.21.15.27:0/1106021939 learned_addr learned my addr 172.21.15.27:0/1106021939 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.678+0000 7f88437fe700 1 --2- 172.21.15.27:0/1106021939 >> v2:172.21.15.77:3300/0 conn(0x7f88440b07b0 0x7f8844159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.678+0000 7f8842ffd700 1 --1- 172.21.15.27:0/1106021939 >> v1:172.21.15.77:6789/0 conn(0x7f88440b03e0 0x7f88440b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:37126/0 (socket says 172.21.15.27:37126) 2024-01-11T09:27:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.679+0000 7f88437fe700 1 -- 172.21.15.27:0/1106021939 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b2b90 msgr2=0x7f8844159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.679+0000 7f88437fe700 1 --2- 172.21.15.27:0/1106021939 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b2b90 0x7f8844159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.679+0000 7f88437fe700 1 -- 172.21.15.27:0/1106021939 >> v1:172.21.15.77:6789/0 conn(0x7f88440b03e0 legacy=0x7f88440b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.679+0000 7f88437fe700 1 -- 172.21.15.27:0/1106021939 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88440cd0e0 con 0x7f88440b07b0 2024-01-11T09:27:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.679+0000 7f88427fc700 1 --2- 172.21.15.27:0/1106021939 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b2b90 0x7f8844159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:27:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.679+0000 7f88437fe700 1 --2- 172.21.15.27:0/1106021939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88440b07b0 0x7f8844159740 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f883c009e60 tx=0x7f883c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba3de71903296474 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.679+0000 7f8841ffb700 1 -- 172.21.15.27:0/1106021939 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883c00e070 con 0x7f88440b07b0 2024-01-11T09:27:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.679+0000 7f8841ffb700 1 -- 172.21.15.27:0/1106021939 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f883c00bb60 con 0x7f88440b07b0 2024-01-11T09:27:11.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.679+0000 7f8841ffb700 1 -- 172.21.15.27:0/1106021939 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883c012430 con 0x7f88440b07b0 2024-01-11T09:27:11.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.680+0000 7f88497ff700 1 -- 172.21.15.27:0/1106021939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88440b07b0 msgr2=0x7f8844159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:11.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.680+0000 7f88497ff700 1 --2- 172.21.15.27:0/1106021939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88440b07b0 0x7f8844159740 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f883c009e60 tx=0x7f883c009780 comp rx=0 tx=0).stop 2024-01-11T09:27:11.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.680+0000 7f88497ff700 1 -- 172.21.15.27:0/1106021939 shutdown_connections 2024-01-11T09:27:11.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.680+0000 7f88497ff700 1 --2- 172.21.15.27:0/1106021939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88440b07b0 0x7f8844159740 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:11.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.680+0000 7f88497ff700 1 --2- 172.21.15.27:0/1106021939 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b2b90 0x7f8844159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:11.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.680+0000 7f88497ff700 1 -- 172.21.15.27:0/1106021939 >> 172.21.15.27:0/1106021939 conn(0x7f884406ed40 msgr2=0x7f884406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:11.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.680+0000 7f88497ff700 1 -- 172.21.15.27:0/1106021939 shutdown_connections 2024-01-11T09:27:11.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.680+0000 7f88497ff700 1 -- 172.21.15.27:0/1106021939 wait complete. 2024-01-11T09:27:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.681+0000 7f88497ff700 1 Processor -- start 2024-01-11T09:27:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.682+0000 7f88497ff700 1 -- start start 2024-01-11T09:27:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.683+0000 7f88497ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88440b03e0 0x7f88440c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.683+0000 7f88497ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88440b07b0 0x7f88440c3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.683+0000 7f88497ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b2b90 0x7f88440af6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.683+0000 7f88497ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88440c43e0 con 0x7f88440b03e0 2024-01-11T09:27:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.683+0000 7f88497ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88440c4660 con 0x7f88440b07b0 2024-01-11T09:27:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.683+0000 7f88497ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88440c4940 con 0x7f88440b2b90 2024-01-11T09:27:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.683+0000 7f8842ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88440b03e0 0x7f88440c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.683+0000 7f8842ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88440b03e0 0x7f88440c6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36230/0 (socket says 172.21.15.27:36230) 2024-01-11T09:27:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.683+0000 7f8842ffd700 1 -- 172.21.15.27:0/84263942 learned_addr learned my addr 172.21.15.27:0/84263942 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.683+0000 7f88427fc700 1 --2- 172.21.15.27:0/84263942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88440b07b0 0x7f88440c3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.683+0000 7f88437fe700 1 --2- 172.21.15.27:0/84263942 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b2b90 0x7f88440af6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:11.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.683+0000 7f8842ffd700 1 -- 172.21.15.27:0/84263942 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b2b90 msgr2=0x7f88440af6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:11.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.683+0000 7f8842ffd700 1 --2- 172.21.15.27:0/84263942 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b2b90 0x7f88440af6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:11.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.683+0000 7f8842ffd700 1 -- 172.21.15.27:0/84263942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88440b07b0 msgr2=0x7f88440c3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:11.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.683+0000 7f8842ffd700 1 --2- 172.21.15.27:0/84263942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88440b07b0 0x7f88440c3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:11.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.683+0000 7f8842ffd700 1 -- 172.21.15.27:0/84263942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88380094f0 con 0x7f88440b03e0 2024-01-11T09:27:11.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.683+0000 7f88427fc700 1 --2- 172.21.15.27:0/84263942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88440b07b0 0x7f88440c3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:27:11.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.684+0000 7f8842ffd700 1 --2- 172.21.15.27:0/84263942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88440b03e0 0x7f88440c6e30 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f883400c7c0 tx=0x7f8834007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:11.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.684+0000 7f882affd700 1 -- 172.21.15.27:0/84263942 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8834005a70 con 0x7f88440b03e0 2024-01-11T09:27:11.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.684+0000 7f882affd700 1 -- 172.21.15.27:0/84263942 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8834005bf0 con 0x7f88440b03e0 2024-01-11T09:27:11.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.684+0000 7f88497ff700 1 -- 172.21.15.27:0/84263942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f883c009a20 con 0x7f88440b03e0 2024-01-11T09:27:11.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.684+0000 7f882affd700 1 -- 172.21.15.27:0/84263942 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88340070f0 con 0x7f88440b03e0 2024-01-11T09:27:11.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.684+0000 7f88497ff700 1 -- 172.21.15.27:0/84263942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88440c4f00 con 0x7f88440b03e0 2024-01-11T09:27:11.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.685+0000 7f88497ff700 1 -- 172.21.15.27:0/84263942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88440718d0 con 0x7f88440b03e0 2024-01-11T09:27:11.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.686+0000 7f882affd700 1 -- 172.21.15.27:0/84263942 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 98) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f8834007460 con 0x7f88440b03e0 2024-01-11T09:27:11.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.687+0000 7f882affd700 1 --2- 172.21.15.27:0/84263942 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8830066ea0 0x7f8830069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:11.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.687+0000 7f882affd700 1 -- 172.21.15.27:0/84263942 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5511+0+0 (secure 0 0 0) 0x7f8834083e90 con 0x7f88440b03e0 2024-01-11T09:27:11.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.688+0000 7f88427fc700 1 --2- 172.21.15.27:0/84263942 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8830066ea0 0x7f8830069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:11.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.688+0000 7f88427fc700 1 --2- 172.21.15.27:0/84263942 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8830066ea0 0x7f8830069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8838006d40 tx=0x7f8838008040 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:11.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.691+0000 7f882affd700 1 -- 172.21.15.27:0/84263942 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8834051280 con 0x7f88440b03e0 2024-01-11T09:27:11.834 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:27:11.839 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:27:11.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:11.975+0000 7f88497ff700 1 -- 172.21.15.27:0/84263942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f88440cd3c0 con 0x7f88440b03e0 2024-01-11T09:27:12.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:12.017+0000 7f882affd700 1 -- 172.21.15.27:0/84263942 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8834051650 con 0x7f88440b03e0 2024-01-11T09:27:12.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:12.920+0000 7f882affd700 1 -- 172.21.15.27:0/84263942 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2502) v1 ==== 174+0+0 (secure 0 0 0) 0x7f8834051280 con 0x7f88440b03e0 2024-01-11T09:27:12.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:12.924+0000 7f88497ff700 1 -- 172.21.15.27:0/84263942 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8830066ea0 msgr2=0x7f8830069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:12.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:12.924+0000 7f88497ff700 1 --2- 172.21.15.27:0/84263942 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8830066ea0 0x7f8830069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8838006d40 tx=0x7f8838008040 comp rx=0 tx=0).stop 2024-01-11T09:27:12.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:12.924+0000 7f88497ff700 1 -- 172.21.15.27:0/84263942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88440b03e0 msgr2=0x7f88440c6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:12.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:12.924+0000 7f88497ff700 1 --2- 172.21.15.27:0/84263942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88440b03e0 0x7f88440c6e30 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f883400c7c0 tx=0x7f8834007f70 comp rx=0 tx=0).stop 2024-01-11T09:27:12.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:12.924+0000 7f88497ff700 1 -- 172.21.15.27:0/84263942 shutdown_connections 2024-01-11T09:27:12.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:12.924+0000 7f88497ff700 1 --2- 172.21.15.27:0/84263942 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f8830066ea0 0x7f8830069350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:12.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:12.924+0000 7f88497ff700 1 --2- 172.21.15.27:0/84263942 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b2b90 0x7f88440af6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:12.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:12.924+0000 7f88497ff700 1 --2- 172.21.15.27:0/84263942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88440b03e0 0x7f88440c6e30 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:12.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:12.924+0000 7f88497ff700 1 --2- 172.21.15.27:0/84263942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88440b07b0 0x7f88440c3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:12.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:12.924+0000 7f88497ff700 1 -- 172.21.15.27:0/84263942 >> 172.21.15.27:0/84263942 conn(0x7f884406ed40 msgr2=0x7f88440cb400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:12.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:12.924+0000 7f88497ff700 1 -- 172.21.15.27:0/84263942 shutdown_connections 2024-01-11T09:27:12.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:12.924+0000 7f88497ff700 1 -- 172.21.15.27:0/84263942 wait complete. 2024-01-11T09:27:12.941 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2024-01-11T09:27:12.942 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2024-01-11T09:27:12.942 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-11T09:27:12.942 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2024-01-11T09:27:13.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.191+0000 7fa24d1ff700 1 Processor -- start 2024-01-11T09:27:13.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.191+0000 7fa24d1ff700 1 -- start start 2024-01-11T09:27:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.193+0000 7fa24d1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa248159650 0x7fa24806bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.193+0000 7fa24d1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa248159a20 0x7fa2480b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.193+0000 7fa24d1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa2480b2730 0x7fa2480b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.193+0000 7fa24d1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa24806e700 con 0x7fa2480b2730 2024-01-11T09:27:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.193+0000 7fa24d1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2480b11a0 con 0x7fa248159a20 2024-01-11T09:27:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.193+0000 7fa24d1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24806c360 con 0x7fa248159650 2024-01-11T09:27:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.193+0000 7fa24759e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa2480b2730 0x7fa2480b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.193+0000 7fa24759e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa2480b2730 0x7fa2480b2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36238/0 (socket says 172.21.15.27:36238) 2024-01-11T09:27:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.193+0000 7fa24759e700 1 -- 172.21.15.27:0/1287343357 learned_addr learned my addr 172.21.15.27:0/1287343357 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.193+0000 7fa24759e700 1 -- 172.21.15.27:0/1287343357 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa248159650 msgr2=0x7fa24806bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.193+0000 7fa246d9d700 1 --2- 172.21.15.27:0/1287343357 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa248159650 0x7fa24806bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.193+0000 7fa24659c700 1 --2- 172.21.15.27:0/1287343357 >> v2:172.21.15.77:3300/0 conn(0x7fa248159a20 0x7fa2480b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.194+0000 7fa24759e700 1 --2- 172.21.15.27:0/1287343357 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa248159650 0x7fa24806bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.194+0000 7fa24759e700 1 -- 172.21.15.27:0/1287343357 >> v2:172.21.15.77:3300/0 conn(0x7fa248159a20 msgr2=0x7fa2480b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.194+0000 7fa24759e700 1 --2- 172.21.15.27:0/1287343357 >> v2:172.21.15.77:3300/0 conn(0x7fa248159a20 0x7fa2480b0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.194+0000 7fa24759e700 1 -- 172.21.15.27:0/1287343357 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24806c640 con 0x7fa2480b2730 2024-01-11T09:27:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.194+0000 7fa246d9d700 1 --2- 172.21.15.27:0/1287343357 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa248159650 0x7fa24806bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:27:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.194+0000 7fa24659c700 1 --2- 172.21.15.27:0/1287343357 >> v2:172.21.15.77:3300/0 conn(0x7fa248159a20 0x7fa2480b0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:27:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.194+0000 7fa24759e700 1 --2- 172.21.15.27:0/1287343357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b2730 0x7fa2480b2bb0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fa24806fe90 tx=0x7fa240009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e224ff369025 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.194+0000 7fa245d9b700 1 -- 172.21.15.27:0/1287343357 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24000e070 con 0x7fa2480b2730 2024-01-11T09:27:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.194+0000 7fa245d9b700 1 -- 172.21.15.27:0/1287343357 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24000ba50 con 0x7fa2480b2730 2024-01-11T09:27:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.194+0000 7fa245d9b700 1 -- 172.21.15.27:0/1287343357 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2400092e0 con 0x7fa2480b2730 2024-01-11T09:27:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.195+0000 7fa24d1ff700 1 -- 172.21.15.27:0/1287343357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b2730 msgr2=0x7fa2480b2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.195+0000 7fa24d1ff700 1 --2- 172.21.15.27:0/1287343357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b2730 0x7fa2480b2bb0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fa24806fe90 tx=0x7fa240009840 comp rx=0 tx=0).stop 2024-01-11T09:27:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.195+0000 7fa24d1ff700 1 -- 172.21.15.27:0/1287343357 shutdown_connections 2024-01-11T09:27:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.195+0000 7fa24d1ff700 1 --2- 172.21.15.27:0/1287343357 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa248159650 0x7fa24806bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.195+0000 7fa24d1ff700 1 --2- 172.21.15.27:0/1287343357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b2730 0x7fa2480b2bb0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.195+0000 7fa24d1ff700 1 --2- 172.21.15.27:0/1287343357 >> v2:172.21.15.77:3300/0 conn(0x7fa248159a20 0x7fa2480b0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.195+0000 7fa24d1ff700 1 -- 172.21.15.27:0/1287343357 >> 172.21.15.27:0/1287343357 conn(0x7fa24806b200 msgr2=0x7fa2480aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.195+0000 7fa24d1ff700 1 -- 172.21.15.27:0/1287343357 shutdown_connections 2024-01-11T09:27:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.195+0000 7fa24d1ff700 1 -- 172.21.15.27:0/1287343357 wait complete. 2024-01-11T09:27:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.196+0000 7fa24d1ff700 1 Processor -- start 2024-01-11T09:27:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.196+0000 7fa24d1ff700 1 -- start start 2024-01-11T09:27:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.197+0000 7fa24d1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b2730 0x7fa248169060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.197+0000 7fa24d1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa248159650 0x7fa248169fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.197+0000 7fa24d1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa248159a20 0x7fa24814fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.197+0000 7fa24d1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24814e700 con 0x7fa2480b2730 2024-01-11T09:27:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.197+0000 7fa24d1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24814f660 con 0x7fa248159a20 2024-01-11T09:27:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.197+0000 7fa24d1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24816a4e0 con 0x7fa248159650 2024-01-11T09:27:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.198+0000 7fa246d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b2730 0x7fa248169060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.198+0000 7fa246d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b2730 0x7fa248169060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36242/0 (socket says 172.21.15.27:36242) 2024-01-11T09:27:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.198+0000 7fa246d9d700 1 -- 172.21.15.27:0/1663880466 learned_addr learned my addr 172.21.15.27:0/1663880466 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.198+0000 7fa24759e700 1 --2- 172.21.15.27:0/1663880466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa248159a20 0x7fa24814fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.198+0000 7fa24659c700 1 --2- 172.21.15.27:0/1663880466 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa248159650 0x7fa248169fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.198+0000 7fa246d9d700 1 -- 172.21.15.27:0/1663880466 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa248159650 msgr2=0x7fa248169fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.198+0000 7fa246d9d700 1 --2- 172.21.15.27:0/1663880466 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa248159650 0x7fa248169fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.198+0000 7fa246d9d700 1 -- 172.21.15.27:0/1663880466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa248159a20 msgr2=0x7fa24814fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.198+0000 7fa246d9d700 1 --2- 172.21.15.27:0/1663880466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa248159a20 0x7fa24814fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.198+0000 7fa246d9d700 1 -- 172.21.15.27:0/1663880466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa240009ae0 con 0x7fa2480b2730 2024-01-11T09:27:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.198+0000 7fa246d9d700 1 --2- 172.21.15.27:0/1663880466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b2730 0x7fa248169060 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fa23800c880 tx=0x7fa238008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.199+0000 7fa22effd700 1 -- 172.21.15.27:0/1663880466 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa238011600 con 0x7fa2480b2730 2024-01-11T09:27:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.199+0000 7fa22effd700 1 -- 172.21.15.27:0/1663880466 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa238011760 con 0x7fa2480b2730 2024-01-11T09:27:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.199+0000 7fa22effd700 1 -- 172.21.15.27:0/1663880466 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa238011ae0 con 0x7fa2480b2730 2024-01-11T09:27:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.199+0000 7fa24d1ff700 1 -- 172.21.15.27:0/1663880466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa24814ea50 con 0x7fa2480b2730 2024-01-11T09:27:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.199+0000 7fa24d1ff700 1 -- 172.21.15.27:0/1663880466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa24814f010 con 0x7fa2480b2730 2024-01-11T09:27:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.200+0000 7fa24d1ff700 1 -- 172.21.15.27:0/1663880466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2480718d0 con 0x7fa2480b2730 2024-01-11T09:27:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.201+0000 7fa22effd700 1 -- 172.21.15.27:0/1663880466 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa238018350 con 0x7fa2480b2730 2024-01-11T09:27:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.202+0000 7fa22effd700 1 --2- 172.21.15.27:0/1663880466 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa234066f30 0x7fa2340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.202+0000 7fa22effd700 1 -- 172.21.15.27:0/1663880466 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5511+0+0 (secure 0 0 0) 0x7fa2380843b0 con 0x7fa2480b2730 2024-01-11T09:27:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.202+0000 7fa24659c700 1 --2- 172.21.15.27:0/1663880466 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa234066f30 0x7fa2340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.203+0000 7fa24659c700 1 --2- 172.21.15.27:0/1663880466 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa234066f30 0x7fa2340693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa2481586d0 tx=0x7fa23c008040 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.206+0000 7fa22effd700 1 -- 172.21.15.27:0/1663880466 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa2380550d0 con 0x7fa2480b2730 2024-01-11T09:27:13.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.489+0000 7fa24d1ff700 1 -- 172.21.15.27:0/1663880466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7fa24806c920 con 0x7fa2480b2730 2024-01-11T09:27:13.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.927+0000 7fa22effd700 1 -- 172.21.15.27:0/1663880466 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2503) v1 ==== 168+0+0 (secure 0 0 0) 0x7fa238054c60 con 0x7fa2480b2730 2024-01-11T09:27:13.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.929+0000 7fa24d1ff700 1 -- 172.21.15.27:0/1663880466 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa234066f30 msgr2=0x7fa2340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:13.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.929+0000 7fa24d1ff700 1 --2- 172.21.15.27:0/1663880466 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa234066f30 0x7fa2340693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa2481586d0 tx=0x7fa23c008040 comp rx=0 tx=0).stop 2024-01-11T09:27:13.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.929+0000 7fa24d1ff700 1 -- 172.21.15.27:0/1663880466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b2730 msgr2=0x7fa248169060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:13.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.929+0000 7fa24d1ff700 1 --2- 172.21.15.27:0/1663880466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b2730 0x7fa248169060 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fa23800c880 tx=0x7fa238008040 comp rx=0 tx=0).stop 2024-01-11T09:27:13.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.929+0000 7fa24d1ff700 1 -- 172.21.15.27:0/1663880466 shutdown_connections 2024-01-11T09:27:13.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.929+0000 7fa24d1ff700 1 --2- 172.21.15.27:0/1663880466 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa234066f30 0x7fa2340693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:13.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.929+0000 7fa24d1ff700 1 --2- 172.21.15.27:0/1663880466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa248159a20 0x7fa24814fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:13.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.929+0000 7fa24d1ff700 1 --2- 172.21.15.27:0/1663880466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2480b2730 0x7fa248169060 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:13.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.929+0000 7fa24d1ff700 1 --2- 172.21.15.27:0/1663880466 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa248159650 0x7fa248169fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:13.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.929+0000 7fa24d1ff700 1 -- 172.21.15.27:0/1663880466 >> 172.21.15.27:0/1663880466 conn(0x7fa24806b200 msgr2=0x7fa248154290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:13.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.929+0000 7fa24d1ff700 1 -- 172.21.15.27:0/1663880466 shutdown_connections 2024-01-11T09:27:13.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:13.930+0000 7fa24d1ff700 1 -- 172.21.15.27:0/1663880466 wait complete. 2024-01-11T09:27:13.945 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-11T09:27:14.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.152+0000 7fc3381ff700 1 Processor -- start 2024-01-11T09:27:14.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.152+0000 7fc3381ff700 1 -- start start 2024-01-11T09:27:14.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.153+0000 7fc3381ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc3300ccd70 0x7fc3300cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:14.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.153+0000 7fc3381ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3300cf850 con 0x7fc33015a1b0 2024-01-11T09:27:14.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.153+0000 7fc3381ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3300cec40 con 0x7fc33015ae70 2024-01-11T09:27:14.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.153+0000 7fc3381ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc33015ac90 con 0x7fc3300ccd70 2024-01-11T09:27:14.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.153+0000 7fc335f9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc33015a1b0 0x7fc33015a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57302/0 (socket says 172.21.15.27:57302) 2024-01-11T09:27:14.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.153+0000 7fc335f9b700 1 -- 172.21.15.27:0/1221229614 learned_addr learned my addr 172.21.15.27:0/1221229614 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:14.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.154+0000 7fc33579a700 1 --2- 172.21.15.27:0/1221229614 >> v2:172.21.15.77:3300/0 conn(0x7fc3300ccd70 0x7fc3300cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:14.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.154+0000 7fc334f99700 1 -- 172.21.15.27:0/1221229614 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254112867 0 0) 0x7fc3300cf850 con 0x7fc33015a1b0 2024-01-11T09:27:14.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.154+0000 7fc334f99700 1 -- 172.21.15.27:0/1221229614 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc31c003570 con 0x7fc33015a1b0 2024-01-11T09:27:14.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.154+0000 7fc33579a700 1 -- 172.21.15.27:0/1221229614 >> v1:172.21.15.77:6789/0 conn(0x7fc33015ae70 legacy=0x7fc33015b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:14.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.154+0000 7fc33579a700 1 -- 172.21.15.27:0/1221229614 >> v1:172.21.15.27:6789/0 conn(0x7fc33015a1b0 legacy=0x7fc33015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:14.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.154+0000 7fc33579a700 1 -- 172.21.15.27:0/1221229614 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3300cef20 con 0x7fc3300ccd70 2024-01-11T09:27:14.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.154+0000 7fc33579a700 1 --2- 172.21.15.27:0/1221229614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3300ccd70 0x7fc3300cd210 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc3200098e0 tx=0x7fc3200091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df8cdc81b391b960 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:14.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.155+0000 7fc334f99700 1 -- 172.21.15.27:0/1221229614 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32000b640 con 0x7fc3300ccd70 2024-01-11T09:27:14.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.155+0000 7fc334f99700 1 -- 172.21.15.27:0/1221229614 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32000b7a0 con 0x7fc3300ccd70 2024-01-11T09:27:14.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.155+0000 7fc334f99700 1 -- 172.21.15.27:0/1221229614 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32000bb40 con 0x7fc3300ccd70 2024-01-11T09:27:14.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.155+0000 7fc3381ff700 1 -- 172.21.15.27:0/1221229614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3300ccd70 msgr2=0x7fc3300cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.155+0000 7fc3381ff700 1 --2- 172.21.15.27:0/1221229614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3300ccd70 0x7fc3300cd210 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc3200098e0 tx=0x7fc3200091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.155+0000 7fc3381ff700 1 -- 172.21.15.27:0/1221229614 shutdown_connections 2024-01-11T09:27:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.155+0000 7fc3381ff700 1 --2- 172.21.15.27:0/1221229614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3300ccd70 0x7fc3300cd210 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.155+0000 7fc3381ff700 1 -- 172.21.15.27:0/1221229614 >> 172.21.15.27:0/1221229614 conn(0x7fc33006ed40 msgr2=0x7fc33006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.155+0000 7fc3381ff700 1 -- 172.21.15.27:0/1221229614 shutdown_connections 2024-01-11T09:27:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.155+0000 7fc3381ff700 1 -- 172.21.15.27:0/1221229614 wait complete. 2024-01-11T09:27:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.157+0000 7fc3381ff700 1 Processor -- start 2024-01-11T09:27:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.157+0000 7fc3381ff700 1 -- start start 2024-01-11T09:27:14.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.158+0000 7fc3381ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3300ccd70 0x7fc3301481c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:14.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.158+0000 7fc3381ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc33015a1b0 0x7fc3301488b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:14.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.158+0000 7fc3381ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33015ae70 0x7fc3300c5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:14.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.158+0000 7fc3381ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3300c8cc0 con 0x7fc33015ae70 2024-01-11T09:27:14.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.158+0000 7fc3381ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3300b6110 con 0x7fc3300ccd70 2024-01-11T09:27:14.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.158+0000 7fc3381ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3300c8ed0 con 0x7fc33015a1b0 2024-01-11T09:27:14.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.158+0000 7fc33679c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33015ae70 0x7fc3300c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:14.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.158+0000 7fc33679c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33015ae70 0x7fc3300c5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36930/0 (socket says 172.21.15.27:36930) 2024-01-11T09:27:14.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.158+0000 7fc33679c700 1 -- 172.21.15.27:0/2587676445 learned_addr learned my addr 172.21.15.27:0/2587676445 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:14.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.159+0000 7fc33579a700 1 --2- 172.21.15.27:0/2587676445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc33015a1b0 0x7fc3301488b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:14.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.159+0000 7fc335f9b700 1 --2- 172.21.15.27:0/2587676445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3300ccd70 0x7fc3301481c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:14.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.159+0000 7fc33679c700 1 -- 172.21.15.27:0/2587676445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc33015a1b0 msgr2=0x7fc3301488b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:14.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.159+0000 7fc33679c700 1 --2- 172.21.15.27:0/2587676445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc33015a1b0 0x7fc3301488b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:14.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.159+0000 7fc33679c700 1 -- 172.21.15.27:0/2587676445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3300ccd70 msgr2=0x7fc3301481c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:14.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.159+0000 7fc33679c700 1 --2- 172.21.15.27:0/2587676445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3300ccd70 0x7fc3301481c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:14.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.159+0000 7fc33679c700 1 -- 172.21.15.27:0/2587676445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc320009420 con 0x7fc33015ae70 2024-01-11T09:27:14.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.159+0000 7fc335f9b700 1 --2- 172.21.15.27:0/2587676445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3300ccd70 0x7fc3301481c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:27:14.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.159+0000 7fc33579a700 1 --2- 172.21.15.27:0/2587676445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc33015a1b0 0x7fc3301488b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:27:14.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.159+0000 7fc33679c700 1 --2- 172.21.15.27:0/2587676445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33015ae70 0x7fc3300c5750 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fc32800c8e0 tx=0x7fc328001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:14.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.159+0000 7fc325ffb700 1 -- 172.21.15.27:0/2587676445 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32800ce80 con 0x7fc33015ae70 2024-01-11T09:27:14.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.159+0000 7fc325ffb700 1 -- 172.21.15.27:0/2587676445 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32800a7e0 con 0x7fc33015ae70 2024-01-11T09:27:14.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.159+0000 7fc325ffb700 1 -- 172.21.15.27:0/2587676445 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32800ab80 con 0x7fc33015ae70 2024-01-11T09:27:14.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.159+0000 7fc3381ff700 1 -- 172.21.15.27:0/2587676445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3300c91b0 con 0x7fc33015ae70 2024-01-11T09:27:14.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.159+0000 7fc3381ff700 1 -- 172.21.15.27:0/2587676445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3301556a0 con 0x7fc33015ae70 2024-01-11T09:27:14.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.160+0000 7fc3381ff700 1 -- 172.21.15.27:0/2587676445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33004c3c0 con 0x7fc33015ae70 2024-01-11T09:27:14.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.161+0000 7fc325ffb700 1 -- 172.21.15.27:0/2587676445 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc328012420 con 0x7fc33015ae70 2024-01-11T09:27:14.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.162+0000 7fc325ffb700 1 --2- 172.21.15.27:0/2587676445 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc314066f30 0x7fc3140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:14.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.162+0000 7fc325ffb700 1 -- 172.21.15.27:0/2587676445 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5511+0+0 (secure 0 0 0) 0x7fc3280847c0 con 0x7fc33015ae70 2024-01-11T09:27:14.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.163+0000 7fc335f9b700 1 --2- 172.21.15.27:0/2587676445 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc314066f30 0x7fc3140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:14.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.163+0000 7fc335f9b700 1 --2- 172.21.15.27:0/2587676445 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc314066f30 0x7fc3140693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc32c007fd0 tx=0x7fc32c009040 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:14.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.166+0000 7fc325ffb700 1 -- 172.21.15.27:0/2587676445 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc328051a50 con 0x7fc33015ae70 2024-01-11T09:27:14.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.372+0000 7fc3381ff700 1 -- 172.21.15.27:0/2587676445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc33015aa40 con 0x7fc33015ae70 2024-01-11T09:27:14.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.376+0000 7fc325ffb700 1 -- 172.21.15.27:0/2587676445 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc328055070 con 0x7fc33015ae70 2024-01-11T09:27:14.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.378+0000 7fc3381ff700 1 -- 172.21.15.27:0/2587676445 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc314066f30 msgr2=0x7fc3140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:14.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.378+0000 7fc3381ff700 1 --2- 172.21.15.27:0/2587676445 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc314066f30 0x7fc3140693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc32c007fd0 tx=0x7fc32c009040 comp rx=0 tx=0).stop 2024-01-11T09:27:14.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.378+0000 7fc3381ff700 1 -- 172.21.15.27:0/2587676445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33015ae70 msgr2=0x7fc3300c5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:14.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.378+0000 7fc3381ff700 1 --2- 172.21.15.27:0/2587676445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33015ae70 0x7fc3300c5750 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fc32800c8e0 tx=0x7fc328001710 comp rx=0 tx=0).stop 2024-01-11T09:27:14.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.379+0000 7fc3381ff700 1 -- 172.21.15.27:0/2587676445 shutdown_connections 2024-01-11T09:27:14.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.379+0000 7fc3381ff700 1 --2- 172.21.15.27:0/2587676445 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc314066f30 0x7fc3140693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:14.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.379+0000 7fc3381ff700 1 --2- 172.21.15.27:0/2587676445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc33015ae70 0x7fc3300c5750 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:14.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.379+0000 7fc3381ff700 1 --2- 172.21.15.27:0/2587676445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3300ccd70 0x7fc3301481c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:14.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.379+0000 7fc3381ff700 1 --2- 172.21.15.27:0/2587676445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc33015a1b0 0x7fc3301488b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:14.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.379+0000 7fc3381ff700 1 -- 172.21.15.27:0/2587676445 >> 172.21.15.27:0/2587676445 conn(0x7fc33006ed40 msgr2=0x7fc3300d0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:14.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.379+0000 7fc3381ff700 1 -- 172.21.15.27:0/2587676445 shutdown_connections 2024-01-11T09:27:14.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.379+0000 7fc3381ff700 1 -- 172.21.15.27:0/2587676445 wait complete. 2024-01-11T09:27:14.393 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T09:27:14.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:27:14.491 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:27:14.588 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T09:27:14.588 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-11T09:27:14.588 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:27:14.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.792+0000 7f02751ff700 1 Processor -- start 2024-01-11T09:27:14.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.792+0000 7f02751ff700 1 -- start start 2024-01-11T09:27:14.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.794+0000 7f02751ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f02700c8620 0x7f02700af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:14.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.794+0000 7f02751ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f02700b09e0 0x7f02700accc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:14.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.794+0000 7f02751ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0270157390 con 0x7f0270069d40 2024-01-11T09:27:14.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.794+0000 7f02751ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02700afeb0 con 0x7f02700c8620 2024-01-11T09:27:14.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.794+0000 7f02751ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027006a200 con 0x7f02700b09e0 2024-01-11T09:27:14.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.794+0000 7f026ed9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f02700c8620 0x7f02700af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:14.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.794+0000 7f026ed9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f02700c8620 0x7f02700af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36944/0 (socket says 172.21.15.27:36944) 2024-01-11T09:27:14.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.794+0000 7f026ed9d700 1 -- 172.21.15.27:0/3322223585 learned_addr learned my addr 172.21.15.27:0/3322223585 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:14.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.794+0000 7f026e59c700 1 --1- 172.21.15.27:0/3322223585 >> v1:172.21.15.27:6789/0 conn(0x7f0270069d40 0x7f02700b02d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57306/0 (socket says 172.21.15.27:57306) 2024-01-11T09:27:14.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.794+0000 7f026f59e700 1 --2- 172.21.15.27:0/3322223585 >> v2:172.21.15.77:3300/0 conn(0x7f02700b09e0 0x7f02700accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:14.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.795+0000 7f026ed9d700 1 -- 172.21.15.27:0/3322223585 >> v2:172.21.15.77:3300/0 conn(0x7f02700b09e0 msgr2=0x7f02700accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:14.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.795+0000 7f026ed9d700 1 --2- 172.21.15.27:0/3322223585 >> v2:172.21.15.77:3300/0 conn(0x7f02700b09e0 0x7f02700accc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:14.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.795+0000 7f026ed9d700 1 -- 172.21.15.27:0/3322223585 >> v1:172.21.15.27:6789/0 conn(0x7f0270069d40 legacy=0x7f02700b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:14.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.795+0000 7f026ed9d700 1 -- 172.21.15.27:0/3322223585 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0270156d40 con 0x7f02700c8620 2024-01-11T09:27:14.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.795+0000 7f026f59e700 1 --2- 172.21.15.27:0/3322223585 >> v2:172.21.15.77:3300/0 conn(0x7f02700b09e0 0x7f02700accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:27:14.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.795+0000 7f026dd9b700 1 -- 172.21.15.27:0/3322223585 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437009326 0 0) 0x7f0270157390 con 0x7f0270069d40 2024-01-11T09:27:14.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.795+0000 7f026ed9d700 1 --2- 172.21.15.27:0/3322223585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02700c8620 0x7f02700af970 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f026000bc50 tx=0x7f026000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87842ab14e0f40b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:14.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.795+0000 7f026dd9b700 1 -- 172.21.15.27:0/3322223585 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026000be60 con 0x7f02700c8620 2024-01-11T09:27:14.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.795+0000 7f026dd9b700 1 -- 172.21.15.27:0/3322223585 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026000ddc0 con 0x7f02700c8620 2024-01-11T09:27:14.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.795+0000 7f026dd9b700 1 -- 172.21.15.27:0/3322223585 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02600196b0 con 0x7f02700c8620 2024-01-11T09:27:14.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.796+0000 7f02751ff700 1 -- 172.21.15.27:0/3322223585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02700c8620 msgr2=0x7f02700af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:14.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.796+0000 7f02751ff700 1 --2- 172.21.15.27:0/3322223585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02700c8620 0x7f02700af970 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f026000bc50 tx=0x7f026000b4f0 comp rx=0 tx=0).stop 2024-01-11T09:27:14.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.796+0000 7f02751ff700 1 -- 172.21.15.27:0/3322223585 shutdown_connections 2024-01-11T09:27:14.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.796+0000 7f02751ff700 1 --2- 172.21.15.27:0/3322223585 >> v2:172.21.15.77:3300/0 conn(0x7f02700b09e0 0x7f02700accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:14.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.796+0000 7f02751ff700 1 --2- 172.21.15.27:0/3322223585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02700c8620 0x7f02700af970 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:14.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.796+0000 7f02751ff700 1 -- 172.21.15.27:0/3322223585 >> 172.21.15.27:0/3322223585 conn(0x7f02700680e0 msgr2=0x7f0270069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:14.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.796+0000 7f02751ff700 1 -- 172.21.15.27:0/3322223585 shutdown_connections 2024-01-11T09:27:14.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.796+0000 7f02751ff700 1 -- 172.21.15.27:0/3322223585 wait complete. 2024-01-11T09:27:14.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.798+0000 7f02751ff700 1 Processor -- start 2024-01-11T09:27:14.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.798+0000 7f02751ff700 1 -- start start 2024-01-11T09:27:14.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.800+0000 7f02751ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0270069d40 0x7f027016aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:14.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.800+0000 7f02751ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02700b09e0 0x7f02700c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:14.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.800+0000 7f02751ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02700c8620 0x7f0270169870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:14.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.800+0000 7f02751ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02700c53a0 con 0x7f02700b09e0 2024-01-11T09:27:14.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.800+0000 7f02751ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02700c5960 con 0x7f02700c8620 2024-01-11T09:27:14.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.800+0000 7f02751ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027016a440 con 0x7f0270069d40 2024-01-11T09:27:14.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.800+0000 7f026e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02700b09e0 0x7f02700c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.800+0000 7f026e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02700b09e0 0x7f02700c7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36948/0 (socket says 172.21.15.27:36948) 2024-01-11T09:27:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.800+0000 7f026e59c700 1 -- 172.21.15.27:0/2846365181 learned_addr learned my addr 172.21.15.27:0/2846365181 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.800+0000 7f026ed9d700 1 --2- 172.21.15.27:0/2846365181 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0270069d40 0x7f027016aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.800+0000 7f026f59e700 1 --2- 172.21.15.27:0/2846365181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02700c8620 0x7f0270169870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.800+0000 7f026e59c700 1 -- 172.21.15.27:0/2846365181 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0270069d40 msgr2=0x7f027016aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.800+0000 7f026e59c700 1 --2- 172.21.15.27:0/2846365181 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0270069d40 0x7f027016aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.800+0000 7f026e59c700 1 -- 172.21.15.27:0/2846365181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02700c8620 msgr2=0x7f0270169870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.800+0000 7f026e59c700 1 --2- 172.21.15.27:0/2846365181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02700c8620 0x7f0270169870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.800+0000 7f026e59c700 1 -- 172.21.15.27:0/2846365181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026000b790 con 0x7f02700b09e0 2024-01-11T09:27:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.801+0000 7f026e59c700 1 --2- 172.21.15.27:0/2846365181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02700b09e0 0x7f02700c7980 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f02640011e0 tx=0x7f0264007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.801+0000 7f0256ffd700 1 -- 172.21.15.27:0/2846365181 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0264005a70 con 0x7f02700b09e0 2024-01-11T09:27:14.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.801+0000 7f0256ffd700 1 -- 172.21.15.27:0/2846365181 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0264005bd0 con 0x7f02700b09e0 2024-01-11T09:27:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.801+0000 7f02751ff700 1 -- 172.21.15.27:0/2846365181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027016a720 con 0x7f02700b09e0 2024-01-11T09:27:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.801+0000 7f0256ffd700 1 -- 172.21.15.27:0/2846365181 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02640070d0 con 0x7f02700b09e0 2024-01-11T09:27:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.801+0000 7f02751ff700 1 -- 172.21.15.27:0/2846365181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027016afe0 con 0x7f02700b09e0 2024-01-11T09:27:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.802+0000 7f02751ff700 1 -- 172.21.15.27:0/2846365181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02700718d0 con 0x7f02700b09e0 2024-01-11T09:27:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.803+0000 7f0256ffd700 1 -- 172.21.15.27:0/2846365181 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0264007480 con 0x7f02700b09e0 2024-01-11T09:27:14.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.805+0000 7f0256ffd700 1 --2- 172.21.15.27:0/2846365181 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f025c066f30 0x7f025c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:14.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.805+0000 7f0256ffd700 1 -- 172.21.15.27:0/2846365181 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5511+0+0 (secure 0 0 0) 0x7f0264083920 con 0x7f02700b09e0 2024-01-11T09:27:14.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.805+0000 7f026ed9d700 1 --2- 172.21.15.27:0/2846365181 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f025c066f30 0x7f025c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:14.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.806+0000 7f026ed9d700 1 --2- 172.21.15.27:0/2846365181 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f025c066f30 0x7f025c0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f026000bc50 tx=0x7f026001b000 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:14.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:14.809+0000 7f0256ffd700 1 -- 172.21.15.27:0/2846365181 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0264050c50 con 0x7f02700b09e0 2024-01-11T09:27:15.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.058+0000 7f02751ff700 1 -- 172.21.15.27:0/2846365181 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0270157020 con 0x7f02700b09e0 2024-01-11T09:27:15.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.059+0000 7f0256ffd700 1 -- 172.21.15.27:0/2846365181 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3526 (secure 0 0 0) 0x7f0264017070 con 0x7f02700b09e0 2024-01-11T09:27:15.056 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:27:15.056 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":88,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7700,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/191611944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":191611944},{"type":"v1","addr":"172.21.15.27:6835","nonce":191611944}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":7946,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3524745284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3524745284},{"type":"v1","addr":"172.21.15.27:6834","nonce":3524745284}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88},{"gid":8734,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/3566978772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3566978772},{"type":"v1","addr":"172.21.15.77:6834","nonce":3566978772}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":8737,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1168169536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1168169536},{"type":"v1","addr":"172.21.15.77:6835","nonce":1168169536}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[]} 2024-01-11T09:27:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.061+0000 7f02751ff700 1 -- 172.21.15.27:0/2846365181 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f025c066f30 msgr2=0x7f025c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.061+0000 7f02751ff700 1 --2- 172.21.15.27:0/2846365181 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f025c066f30 0x7f025c0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f026000bc50 tx=0x7f026001b000 comp rx=0 tx=0).stop 2024-01-11T09:27:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.061+0000 7f02751ff700 1 -- 172.21.15.27:0/2846365181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02700b09e0 msgr2=0x7f02700c7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.061+0000 7f02751ff700 1 --2- 172.21.15.27:0/2846365181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02700b09e0 0x7f02700c7980 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f02640011e0 tx=0x7f0264007e10 comp rx=0 tx=0).stop 2024-01-11T09:27:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.061+0000 7f02751ff700 1 -- 172.21.15.27:0/2846365181 shutdown_connections 2024-01-11T09:27:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.061+0000 7f02751ff700 1 --2- 172.21.15.27:0/2846365181 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f025c066f30 0x7f025c0693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:15.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.061+0000 7f02751ff700 1 --2- 172.21.15.27:0/2846365181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02700c8620 0x7f0270169870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:15.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.061+0000 7f02751ff700 1 --2- 172.21.15.27:0/2846365181 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0270069d40 0x7f027016aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:15.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.061+0000 7f02751ff700 1 --2- 172.21.15.27:0/2846365181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02700b09e0 0x7f02700c7980 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:15.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.061+0000 7f02751ff700 1 -- 172.21.15.27:0/2846365181 >> 172.21.15.27:0/2846365181 conn(0x7f02700680e0 msgr2=0x7f0270152ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:15.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.062+0000 7f02751ff700 1 -- 172.21.15.27:0/2846365181 shutdown_connections 2024-01-11T09:27:15.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.062+0000 7f02751ff700 1 -- 172.21.15.27:0/2846365181 wait complete. 2024-01-11T09:27:15.059 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 88 2024-01-11T09:27:15.079 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-11T09:27:15.080 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-11T09:27:15.080 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-11T09:27:15.080 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:27:15.080 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-11T09:27:15.080 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-11T09:27:15.080 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-11T09:27:15.080 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:27:15.081 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-11T09:27:15.081 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-11T09:27:15.081 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-11T09:27:15.081 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:27:15.081 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-11T09:27:15.081 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-11T09:27:15.081 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-11T09:27:15.081 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:27:15.095 INFO:tasks.ceph.mds.a:Stopped 2024-01-11T09:27:15.095 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-11T09:27:15.096 INFO:tasks.ceph.mds.c:Stopped 2024-01-11T09:27:15.096 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-11T09:27:15.097 INFO:tasks.ceph.mds.b:Stopped 2024-01-11T09:27:15.097 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-11T09:27:15.098 INFO:tasks.ceph.mds.d:Stopped 2024-01-11T09:27:15.098 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-11T09:27:15.140 INFO:tasks.ceph.mds.a:Started 2024-01-11T09:27:15.140 DEBUG:teuthology.parallel:result is None 2024-01-11T09:27:15.142 INFO:tasks.ceph.mds.b:Started 2024-01-11T09:27:15.142 DEBUG:teuthology.parallel:result is None 2024-01-11T09:27:15.143 INFO:tasks.ceph.mds.c:Started 2024-01-11T09:27:15.143 DEBUG:teuthology.parallel:result is None 2024-01-11T09:27:15.144 INFO:tasks.ceph.mds.d:Started 2024-01-11T09:27:15.144 DEBUG:teuthology.parallel:result is None 2024-01-11T09:27:15.145 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-11T09:27:15.286 INFO:tasks.ceph.mds.b.smithi077.stdout:starting mds.b at 2024-01-11T09:27:15.286 INFO:tasks.ceph.mds.d.smithi077.stdout:starting mds.d at 2024-01-11T09:27:15.292 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T09:27:15.293 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T09:27:15.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.358+0000 7fe965fff700 1 Processor -- start 2024-01-11T09:27:15.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.358+0000 7fe965fff700 1 -- start start 2024-01-11T09:27:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.359+0000 7fe965fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe960157360 0x7fe96006f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.359+0000 7fe965fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe960157730 0x7fe9600cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.359+0000 7fe965fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9600b1290 0x7fe9600b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.359+0000 7fe965fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9600cce10 con 0x7fe9600b1290 2024-01-11T09:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.359+0000 7fe965fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9600b1c30 con 0x7fe960157360 2024-01-11T09:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.359+0000 7fe965fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9600cb540 con 0x7fe960157730 2024-01-11T09:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.360+0000 7fe95ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9600b1290 0x7fe9600b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.360+0000 7fe95ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9600b1290 0x7fe9600b16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36992/0 (socket says 172.21.15.27:36992) 2024-01-11T09:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.360+0000 7fe95ffff700 1 -- 172.21.15.27:0/1772323232 learned_addr learned my addr 172.21.15.27:0/1772323232 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.360+0000 7fe95f7fe700 1 --2- 172.21.15.27:0/1772323232 >> v2:172.21.15.77:3300/0 conn(0x7fe960157360 0x7fe96006f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.360+0000 7fe95effd700 1 --2- 172.21.15.27:0/1772323232 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe960157730 0x7fe9600cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.360+0000 7fe95ffff700 1 -- 172.21.15.27:0/1772323232 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe960157730 msgr2=0x7fe9600cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.360+0000 7fe95ffff700 1 --2- 172.21.15.27:0/1772323232 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe960157730 0x7fe9600cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.360+0000 7fe95ffff700 1 -- 172.21.15.27:0/1772323232 >> v2:172.21.15.77:3300/0 conn(0x7fe960157360 msgr2=0x7fe96006f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.360+0000 7fe95ffff700 1 --2- 172.21.15.27:0/1772323232 >> v2:172.21.15.77:3300/0 conn(0x7fe960157360 0x7fe96006f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.360+0000 7fe95ffff700 1 -- 172.21.15.27:0/1772323232 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9600ca960 con 0x7fe9600b1290 2024-01-11T09:27:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.360+0000 7fe95effd700 1 --2- 172.21.15.27:0/1772323232 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe960157730 0x7fe9600cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:27:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.361+0000 7fe95ffff700 1 --2- 172.21.15.27:0/1772323232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9600b1290 0x7fe9600b16f0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fe960076d60 tx=0x7fe958009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d821803f9ed817 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.361+0000 7fe95e7fc700 1 -- 172.21.15.27:0/1772323232 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95800e070 con 0x7fe9600b1290 2024-01-11T09:27:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.361+0000 7fe95e7fc700 1 -- 172.21.15.27:0/1772323232 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95800ba20 con 0x7fe9600b1290 2024-01-11T09:27:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.361+0000 7fe95e7fc700 1 -- 172.21.15.27:0/1772323232 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9580092e0 con 0x7fe9600b1290 2024-01-11T09:27:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.361+0000 7fe965fff700 1 -- 172.21.15.27:0/1772323232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9600b1290 msgr2=0x7fe9600b16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.361+0000 7fe965fff700 1 --2- 172.21.15.27:0/1772323232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9600b1290 0x7fe9600b16f0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fe960076d60 tx=0x7fe958009840 comp rx=0 tx=0).stop 2024-01-11T09:27:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.361+0000 7fe965fff700 1 -- 172.21.15.27:0/1772323232 shutdown_connections 2024-01-11T09:27:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.361+0000 7fe965fff700 1 --2- 172.21.15.27:0/1772323232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9600b1290 0x7fe9600b16f0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.361+0000 7fe965fff700 1 --2- 172.21.15.27:0/1772323232 >> v2:172.21.15.77:3300/0 conn(0x7fe960157360 0x7fe96006f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.361+0000 7fe965fff700 1 --2- 172.21.15.27:0/1772323232 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe960157730 0x7fe9600cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.362+0000 7fe965fff700 1 -- 172.21.15.27:0/1772323232 >> 172.21.15.27:0/1772323232 conn(0x7fe9600680b0 msgr2=0x7fe9600cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:15.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.362+0000 7fe965fff700 1 -- 172.21.15.27:0/1772323232 shutdown_connections 2024-01-11T09:27:15.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.362+0000 7fe965fff700 1 -- 172.21.15.27:0/1772323232 wait complete. 2024-01-11T09:27:15.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.363+0000 7fe965fff700 1 Processor -- start 2024-01-11T09:27:15.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.363+0000 7fe965fff700 1 -- start start 2024-01-11T09:27:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.364+0000 7fe965fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9600b1290 0x7fe9601591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.364+0000 7fe965fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe960157360 0x7fe9600aba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.364+0000 7fe965fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe960157730 0x7fe96014afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.364+0000 7fe965fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9600abfc0 con 0x7fe960157730 2024-01-11T09:27:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.364+0000 7fe965fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9600ac260 con 0x7fe9600b1290 2024-01-11T09:27:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.364+0000 7fe965fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9600ac540 con 0x7fe960157360 2024-01-11T09:27:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.364+0000 7fe95ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe960157730 0x7fe96014afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.364+0000 7fe95ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe960157730 0x7fe96014afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36998/0 (socket says 172.21.15.27:36998) 2024-01-11T09:27:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.364+0000 7fe95ffff700 1 -- 172.21.15.27:0/510869149 learned_addr learned my addr 172.21.15.27:0/510869149 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.364+0000 7fe95f7fe700 1 --2- 172.21.15.27:0/510869149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9600b1290 0x7fe9601591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.365+0000 7fe95effd700 1 --2- 172.21.15.27:0/510869149 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe960157360 0x7fe9600aba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.365+0000 7fe95ffff700 1 -- 172.21.15.27:0/510869149 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe960157360 msgr2=0x7fe9600aba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.365+0000 7fe95ffff700 1 --2- 172.21.15.27:0/510869149 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe960157360 0x7fe9600aba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.365+0000 7fe95ffff700 1 -- 172.21.15.27:0/510869149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9600b1290 msgr2=0x7fe9601591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.365+0000 7fe95ffff700 1 --2- 172.21.15.27:0/510869149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9600b1290 0x7fe9601591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.365+0000 7fe95ffff700 1 -- 172.21.15.27:0/510869149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe958009ae0 con 0x7fe960157730 2024-01-11T09:27:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.365+0000 7fe95f7fe700 1 --2- 172.21.15.27:0/510869149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9600b1290 0x7fe9601591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:27:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.365+0000 7fe95ffff700 1 --2- 172.21.15.27:0/510869149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe960157730 0x7fe96014afe0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fe96006fe90 tx=0x7fe958018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.365+0000 7fe9477fe700 1 -- 172.21.15.27:0/510869149 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9580032f0 con 0x7fe960157730 2024-01-11T09:27:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.365+0000 7fe9477fe700 1 -- 172.21.15.27:0/510869149 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe958008db0 con 0x7fe960157730 2024-01-11T09:27:15.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.365+0000 7fe9477fe700 1 -- 172.21.15.27:0/510869149 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe958026690 con 0x7fe960157730 2024-01-11T09:27:15.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.365+0000 7fe965fff700 1 -- 172.21.15.27:0/510869149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9600ac820 con 0x7fe960157730 2024-01-11T09:27:15.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.366+0000 7fe965fff700 1 -- 172.21.15.27:0/510869149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9600acde0 con 0x7fe960157730 2024-01-11T09:27:15.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.367+0000 7fe965fff700 1 -- 172.21.15.27:0/510869149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9600acfd0 con 0x7fe960157730 2024-01-11T09:27:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.368+0000 7fe9477fe700 1 -- 172.21.15.27:0/510869149 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe958003040 con 0x7fe960157730 2024-01-11T09:27:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.368+0000 7fe9477fe700 1 --2- 172.21.15.27:0/510869149 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe94c066f30 0x7fe94c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.369+0000 7fe9477fe700 1 -- 172.21.15.27:0/510869149 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5511+0+0 (secure 0 0 0) 0x7fe958086480 con 0x7fe960157730 2024-01-11T09:27:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.369+0000 7fe95f7fe700 1 --2- 172.21.15.27:0/510869149 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe94c066f30 0x7fe94c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.369+0000 7fe95f7fe700 1 --2- 172.21.15.27:0/510869149 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe94c066f30 0x7fe94c0693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe950011480 tx=0x7fe9500091a0 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:15.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.373+0000 7fe9477fe700 1 -- 172.21.15.27:0/510869149 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe958053860 con 0x7fe960157730 2024-01-11T09:27:15.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:15.590+0000 7fe965fff700 1 -- 172.21.15.27:0/510869149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe96004c3c0 con 0x7fe960157730 2024-01-11T09:27:16.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.661+0000 7fe9477fe700 1 -- 172.21.15.27:0/510869149 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v177) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe958056e80 con 0x7fe960157730 2024-01-11T09:27:16.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.665+0000 7fe965fff700 1 -- 172.21.15.27:0/510869149 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe94c066f30 msgr2=0x7fe94c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:16.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.665+0000 7fe965fff700 1 --2- 172.21.15.27:0/510869149 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe94c066f30 0x7fe94c0693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe950011480 tx=0x7fe9500091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:16.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.665+0000 7fe965fff700 1 -- 172.21.15.27:0/510869149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe960157730 msgr2=0x7fe96014afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:16.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.665+0000 7fe965fff700 1 --2- 172.21.15.27:0/510869149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe960157730 0x7fe96014afe0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fe96006fe90 tx=0x7fe958018000 comp rx=0 tx=0).stop 2024-01-11T09:27:16.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.665+0000 7fe965fff700 1 -- 172.21.15.27:0/510869149 shutdown_connections 2024-01-11T09:27:16.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.665+0000 7fe965fff700 1 --2- 172.21.15.27:0/510869149 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe94c066f30 0x7fe94c0693e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:16.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.665+0000 7fe965fff700 1 --2- 172.21.15.27:0/510869149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe960157730 0x7fe96014afe0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:16.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.665+0000 7fe965fff700 1 --2- 172.21.15.27:0/510869149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9600b1290 0x7fe9601591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:16.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.665+0000 7fe965fff700 1 --2- 172.21.15.27:0/510869149 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe960157360 0x7fe9600aba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:16.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.665+0000 7fe965fff700 1 -- 172.21.15.27:0/510869149 >> 172.21.15.27:0/510869149 conn(0x7fe9600680b0 msgr2=0x7fe9601528e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:16.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.665+0000 7fe965fff700 1 -- 172.21.15.27:0/510869149 shutdown_connections 2024-01-11T09:27:16.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.665+0000 7fe965fff700 1 -- 172.21.15.27:0/510869149 wait complete. 2024-01-11T09:27:16.664 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-11T09:27:16.683 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-11T09:27:16.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.890+0000 7f5c9cdff700 1 Processor -- start 2024-01-11T09:27:16.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.890+0000 7f5c9cdff700 1 -- start start 2024-01-11T09:27:16.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.892+0000 7f5c9cdff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c98069d40 0x7f5c980b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:16.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.892+0000 7f5c9cdff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5c980c89f0 0x7f5c980b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:16.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.892+0000 7f5c9cdff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c98157390 con 0x7f5c980c8620 2024-01-11T09:27:16.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.892+0000 7f5c9cdff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c980adcd0 con 0x7f5c98069d40 2024-01-11T09:27:16.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.892+0000 7f5c9cdff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c9806a200 con 0x7f5c980c89f0 2024-01-11T09:27:16.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.892+0000 7f5c95d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c98069d40 0x7f5c980b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:16.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.892+0000 7f5c95d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c98069d40 0x7f5c980b02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37014/0 (socket says 172.21.15.27:37014) 2024-01-11T09:27:16.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.892+0000 7f5c95d9b700 1 -- 172.21.15.27:0/1853122132 learned_addr learned my addr 172.21.15.27:0/1853122132 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:16.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.892+0000 7f5c96d9d700 1 --2- 172.21.15.27:0/1853122132 >> v2:172.21.15.77:3300/0 conn(0x7f5c980c89f0 0x7f5c980b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:16.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.892+0000 7f5c95d9b700 1 -- 172.21.15.27:0/1853122132 >> v2:172.21.15.77:3300/0 conn(0x7f5c980c89f0 msgr2=0x7f5c980b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:16.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.892+0000 7f5c95d9b700 1 --2- 172.21.15.27:0/1853122132 >> v2:172.21.15.77:3300/0 conn(0x7f5c980c89f0 0x7f5c980b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:16.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.892+0000 7f5c95d9b700 1 -- 172.21.15.27:0/1853122132 >> v1:172.21.15.77:6789/0 conn(0x7f5c980c8620 legacy=0x7f5c980af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:16.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.893+0000 7f5c95d9b700 1 -- 172.21.15.27:0/1853122132 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c98156d40 con 0x7f5c98069d40 2024-01-11T09:27:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.893+0000 7f5c95d9b700 1 --2- 172.21.15.27:0/1853122132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c98069d40 0x7f5c980b02d0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f5c980b1bd0 tx=0x7f5c8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f83463370deda3a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.893+0000 7f5c9559a700 1 -- 172.21.15.27:0/1853122132 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c009e70 con 0x7f5c98069d40 2024-01-11T09:27:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.893+0000 7f5c9559a700 1 -- 172.21.15.27:0/1853122132 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8c00b860 con 0x7f5c98069d40 2024-01-11T09:27:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.893+0000 7f5c9559a700 1 -- 172.21.15.27:0/1853122132 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c00bc00 con 0x7f5c98069d40 2024-01-11T09:27:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.894+0000 7f5c9cdff700 1 -- 172.21.15.27:0/1853122132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c98069d40 msgr2=0x7f5c980b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.894+0000 7f5c9cdff700 1 --2- 172.21.15.27:0/1853122132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c98069d40 0x7f5c980b02d0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f5c980b1bd0 tx=0x7f5c8c009250 comp rx=0 tx=0).stop 2024-01-11T09:27:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.894+0000 7f5c9cdff700 1 -- 172.21.15.27:0/1853122132 shutdown_connections 2024-01-11T09:27:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.894+0000 7f5c9cdff700 1 --2- 172.21.15.27:0/1853122132 >> v2:172.21.15.77:3300/0 conn(0x7f5c980c89f0 0x7f5c980b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.894+0000 7f5c9cdff700 1 --2- 172.21.15.27:0/1853122132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c98069d40 0x7f5c980b02d0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.894+0000 7f5c9cdff700 1 -- 172.21.15.27:0/1853122132 >> 172.21.15.27:0/1853122132 conn(0x7f5c980680e0 msgr2=0x7f5c98069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:16.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.894+0000 7f5c9cdff700 1 -- 172.21.15.27:0/1853122132 shutdown_connections 2024-01-11T09:27:16.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.894+0000 7f5c9cdff700 1 -- 172.21.15.27:0/1853122132 wait complete. 2024-01-11T09:27:16.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.895+0000 7f5c9cdff700 1 Processor -- start 2024-01-11T09:27:16.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.895+0000 7f5c9cdff700 1 -- start start 2024-01-11T09:27:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.896+0000 7f5c9cdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c98069d40 0x7f5c9816bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.896+0000 7f5c9cdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c980c8620 0x7f5c980abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.896+0000 7f5c9cdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980c89f0 0x7f5c98167ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.896+0000 7f5c9cdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9816c090 con 0x7f5c980c89f0 2024-01-11T09:27:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.896+0000 7f5c9cdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c980a6480 con 0x7f5c980c8620 2024-01-11T09:27:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.896+0000 7f5c9cdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c980ac300 con 0x7f5c98069d40 2024-01-11T09:27:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.897+0000 7f5c96d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980c89f0 0x7f5c98167ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.897+0000 7f5c96d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980c89f0 0x7f5c98167ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37020/0 (socket says 172.21.15.27:37020) 2024-01-11T09:27:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.897+0000 7f5c96d9d700 1 -- 172.21.15.27:0/751170297 learned_addr learned my addr 172.21.15.27:0/751170297 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.897+0000 7f5c95d9b700 1 --2- 172.21.15.27:0/751170297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c980c8620 0x7f5c980abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.897+0000 7f5c9659c700 1 --2- 172.21.15.27:0/751170297 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c98069d40 0x7f5c9816bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.897+0000 7f5c96d9d700 1 -- 172.21.15.27:0/751170297 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c98069d40 msgr2=0x7f5c9816bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.897+0000 7f5c96d9d700 1 --2- 172.21.15.27:0/751170297 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c98069d40 0x7f5c9816bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.897+0000 7f5c96d9d700 1 -- 172.21.15.27:0/751170297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c980c8620 msgr2=0x7f5c980abdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.897+0000 7f5c96d9d700 1 --2- 172.21.15.27:0/751170297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c980c8620 0x7f5c980abdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.897+0000 7f5c96d9d700 1 -- 172.21.15.27:0/751170297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c0094f0 con 0x7f5c980c89f0 2024-01-11T09:27:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.897+0000 7f5c96d9d700 1 --2- 172.21.15.27:0/751170297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980c89f0 0x7f5c98167ba0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f5c90009d70 tx=0x7f5c9000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.898+0000 7f5c867fc700 1 -- 172.21.15.27:0/751170297 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9000caa0 con 0x7f5c980c89f0 2024-01-11T09:27:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.898+0000 7f5c867fc700 1 -- 172.21.15.27:0/751170297 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c9000f040 con 0x7f5c980c89f0 2024-01-11T09:27:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.898+0000 7f5c867fc700 1 -- 172.21.15.27:0/751170297 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9000ce50 con 0x7f5c980c89f0 2024-01-11T09:27:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.898+0000 7f5c9cdff700 1 -- 172.21.15.27:0/751170297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c980a5870 con 0x7f5c980c89f0 2024-01-11T09:27:16.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.898+0000 7f5c9cdff700 1 -- 172.21.15.27:0/751170297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c980a5e30 con 0x7f5c980c89f0 2024-01-11T09:27:16.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.899+0000 7f5c9cdff700 1 -- 172.21.15.27:0/751170297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c980718d0 con 0x7f5c980c89f0 2024-01-11T09:27:16.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.900+0000 7f5c867fc700 1 -- 172.21.15.27:0/751170297 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5c90014690 con 0x7f5c980c89f0 2024-01-11T09:27:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.901+0000 7f5c867fc700 1 --2- 172.21.15.27:0/751170297 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5c78066f30 0x7f5c780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.901+0000 7f5c867fc700 1 -- 172.21.15.27:0/751170297 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 4909+0+0 (secure 0 0 0) 0x7f5c90084d80 con 0x7f5c980c89f0 2024-01-11T09:27:16.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.901+0000 7f5c9659c700 1 --2- 172.21.15.27:0/751170297 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5c78066f30 0x7f5c780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:16.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.902+0000 7f5c9659c700 1 --2- 172.21.15.27:0/751170297 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5c78066f30 0x7f5c780693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5c8800ed50 tx=0x7f5c88007f50 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:16.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:16.905+0000 7f5c867fc700 1 -- 172.21.15.27:0/751170297 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5c90052310 con 0x7f5c980c89f0 2024-01-11T09:27:17.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.189+0000 7f5c9cdff700 1 -- 172.21.15.27:0/751170297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5c980a65c0 con 0x7f5c980c89f0 2024-01-11T09:27:17.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.189+0000 7f5c867fc700 1 -- 172.21.15.27:0/751170297 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v42) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f5c90055930 con 0x7f5c980c89f0 2024-01-11T09:27:17.186 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:27:17.186 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:27:17.186 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-11T09:27:17.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-11T09:27:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lQm3yLhAAv6erz5hMfat8EUzRA5luMA==", 2024-01-11T09:27:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:27:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:27:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:27:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:27:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:27:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:27:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-11T09:27:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9l+FG8ChAACHndtVLx7qS3sidPniSycA==", 2024-01-11T09:27:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:27:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:27:17.187 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:27:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:27:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:27:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:27:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-11T09:27:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l1NnEOhAALHXsc5es7hufCthGdRzkyw==", 2024-01-11T09:27:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:27:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:27:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:27:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:27:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:27:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:27:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-11T09:27:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lruoWFhAAm8AmrZtTJhYI1+19F79RPQ==", 2024-01-11T09:27:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:27:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:27:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:27:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:27:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:27:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:27:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-11T09:27:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA2op9ltZygORAAZr1X1nCN0ce8rIVaGDpFyQ==", 2024-01-11T09:27:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:27:17.189 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:27:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:27:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:27:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:27:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-11T09:27:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA6op9lqSRPAhAAmtyCsyK5ZVGX2wfhOCMuFQ==", 2024-01-11T09:27:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:27:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:27:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:27:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:27:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:27:17.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-11T09:27:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA9op9lZcs8BhAA8j+U+jPOoy+fliVWZAPFYg==", 2024-01-11T09:27:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:27:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:27:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:27:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:27:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:27:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-11T09:27:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBAop9lFuCUChAAP6si/2W1mxH4ecQJvuumkw==", 2024-01-11T09:27:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:27:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:27:17.191 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:27:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:27:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:27:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-11T09:27:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBGop9lOpwBIRAAcpcsjwmX2IldcKHJWAgHuA==", 2024-01-11T09:27:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:27:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:27:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:27:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:27:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:27:17.192 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-11T09:27:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBJop9l4q8hJRAAK8mAYVRQMoWS6wBOU69HmA==", 2024-01-11T09:27:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:27:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:27:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:27:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:27:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:27:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-11T09:27:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBMop9lqMF9KBAAFgFFI2GWt9AGif4AoScThw==", 2024-01-11T09:27:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:27:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:27:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:27:17.193 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:27:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:27:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-11T09:27:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBPop9lN7eDLBAAKswPZFfs0g2AaSe6V+pbbQ==", 2024-01-11T09:27:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:27:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:27:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:27:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:27:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:27:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-11T09:27:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lT5okIBAABy8PbJJbA062VNmb1Y8qhg==", 2024-01-11T09:27:17.194 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:27:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:27:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-01-11T09:27:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-11T09:27:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:27:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-11T09:27:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lDf37LhAAqV+EabKXJEW1sS5xtu1tXw==", 2024-01-11T09:27:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:27:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:27:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T09:27:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T09:27:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T09:27:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:27:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-11T09:27:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lWayxJxAAcalM8QSOGtbCXxjYZGaNQA==", 2024-01-11T09:27:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:27:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:27:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T09:27:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T09:27:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T09:27:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:27:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-11T09:27:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lHD4tNhAAdmnFBmMrK+QhQYdpbzW0mg==", 2024-01-11T09:27:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:27:17.196 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:27:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T09:27:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T09:27:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T09:27:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:27:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-11T09:27:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAwop9lGPtDMxAAJ8FjrSEdCtQL8AhtAyqnTw==", 2024-01-11T09:27:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:27:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T09:27:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:27:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:27:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:27:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:27:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-11T09:27:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lXBZLJxAAXa1S8Vkm3MiV4igtNbg/1Q==", 2024-01-11T09:27:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:27:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T09:27:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T09:27:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:27:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:27:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-11T09:27:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l3ZGoHxAANqzytTTvWYSomG+RFsFDyg==", 2024-01-11T09:27:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:27:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T09:27:17.198 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T09:27:17.199 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:27:17.199 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.199 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:17.199 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-11T09:27:17.199 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:27:17.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.192+0000 7f5c9cdff700 1 -- 172.21.15.27:0/751170297 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5c78066f30 msgr2=0x7f5c780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:17.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.192+0000 7f5c9cdff700 1 --2- 172.21.15.27:0/751170297 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5c78066f30 0x7f5c780693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5c8800ed50 tx=0x7f5c88007f50 comp rx=0 tx=0).stop 2024-01-11T09:27:17.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.192+0000 7f5c9cdff700 1 -- 172.21.15.27:0/751170297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980c89f0 msgr2=0x7f5c98167ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:17.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.192+0000 7f5c9cdff700 1 --2- 172.21.15.27:0/751170297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980c89f0 0x7f5c98167ba0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f5c90009d70 tx=0x7f5c9000b040 comp rx=0 tx=0).stop 2024-01-11T09:27:17.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.192+0000 7f5c9cdff700 1 -- 172.21.15.27:0/751170297 shutdown_connections 2024-01-11T09:27:17.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.192+0000 7f5c9cdff700 1 --2- 172.21.15.27:0/751170297 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5c78066f30 0x7f5c780693e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:17.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.192+0000 7f5c9cdff700 1 --2- 172.21.15.27:0/751170297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980c89f0 0x7f5c98167ba0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:17.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.192+0000 7f5c9cdff700 1 --2- 172.21.15.27:0/751170297 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5c98069d40 0x7f5c9816bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:17.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.192+0000 7f5c9cdff700 1 --2- 172.21.15.27:0/751170297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c980c8620 0x7f5c980abdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:17.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.192+0000 7f5c9cdff700 1 -- 172.21.15.27:0/751170297 >> 172.21.15.27:0/751170297 conn(0x7f5c980680e0 msgr2=0x7f5c980cb130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:17.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.192+0000 7f5c9cdff700 1 -- 172.21.15.27:0/751170297 shutdown_connections 2024-01-11T09:27:17.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.192+0000 7f5c9cdff700 1 -- 172.21.15.27:0/751170297 wait complete. 2024-01-11T09:27:17.207 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:27:17.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.410+0000 7fb44d5ff700 1 Processor -- start 2024-01-11T09:27:17.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.411+0000 7fb44d5ff700 1 -- start start 2024-01-11T09:27:17.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.412+0000 7fb44d5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4480b03e0 0x7fb4480b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:17.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.412+0000 7fb44d5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4480b2e00 con 0x7fb4480b07b0 2024-01-11T09:27:17.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.412+0000 7fb44d5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4480cd730 con 0x7fb4480b29c0 2024-01-11T09:27:17.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.412+0000 7fb44d5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480ccb20 con 0x7fb4480b03e0 2024-01-11T09:27:17.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.412+0000 7fb4467fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb4480b07b0 0x7fb448159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57346/0 (socket says 172.21.15.27:57346) 2024-01-11T09:27:17.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.412+0000 7fb4467fc700 1 -- 172.21.15.27:0/44976698 learned_addr learned my addr 172.21.15.27:0/44976698 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:17.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.412+0000 7fb446ffd700 1 --2- 172.21.15.27:0/44976698 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4480b03e0 0x7fb4480b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:17.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.412+0000 7fb445ffb700 1 -- 172.21.15.27:0/44976698 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132477628 0 0) 0x7fb43c001110 con 0x7fb4480b07b0 2024-01-11T09:27:17.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.413+0000 7fb445ffb700 1 -- 172.21.15.27:0/44976698 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb430003570 con 0x7fb4480b07b0 2024-01-11T09:27:17.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.413+0000 7fb445ffb700 1 -- 172.21.15.27:0/44976698 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686200114 0 0) 0x7fb4480cd730 con 0x7fb4480b29c0 2024-01-11T09:27:17.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.413+0000 7fb445ffb700 1 -- 172.21.15.27:0/44976698 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb43c001110 con 0x7fb4480b29c0 2024-01-11T09:27:17.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.413+0000 7fb446ffd700 1 -- 172.21.15.27:0/44976698 >> v1:172.21.15.77:6789/0 conn(0x7fb4480b29c0 legacy=0x7fb4480aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:17.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.413+0000 7fb446ffd700 1 -- 172.21.15.27:0/44976698 >> v1:172.21.15.27:6789/0 conn(0x7fb4480b07b0 legacy=0x7fb448159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:17.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.413+0000 7fb446ffd700 1 -- 172.21.15.27:0/44976698 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4480cce00 con 0x7fb4480b03e0 2024-01-11T09:27:17.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.413+0000 7fb446ffd700 1 --2- 172.21.15.27:0/44976698 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4480b03e0 0x7fb4480b2480 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb438009910 tx=0x7fb4380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e41bb1f21c140212 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:17.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.413+0000 7fb445ffb700 1 -- 172.21.15.27:0/44976698 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb438011630 con 0x7fb4480b03e0 2024-01-11T09:27:17.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.413+0000 7fb445ffb700 1 -- 172.21.15.27:0/44976698 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb438011790 con 0x7fb4480b03e0 2024-01-11T09:27:17.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.413+0000 7fb445ffb700 1 -- 172.21.15.27:0/44976698 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb438011b30 con 0x7fb4480b03e0 2024-01-11T09:27:17.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.414+0000 7fb44d5ff700 1 -- 172.21.15.27:0/44976698 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4480b03e0 msgr2=0x7fb4480b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:17.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.414+0000 7fb44d5ff700 1 --2- 172.21.15.27:0/44976698 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4480b03e0 0x7fb4480b2480 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb438009910 tx=0x7fb4380091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:17.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.414+0000 7fb44d5ff700 1 -- 172.21.15.27:0/44976698 shutdown_connections 2024-01-11T09:27:17.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.414+0000 7fb44d5ff700 1 --2- 172.21.15.27:0/44976698 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4480b03e0 0x7fb4480b2480 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:17.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.414+0000 7fb44d5ff700 1 -- 172.21.15.27:0/44976698 >> 172.21.15.27:0/44976698 conn(0x7fb44806ed40 msgr2=0x7fb44806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:17.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.414+0000 7fb44d5ff700 1 -- 172.21.15.27:0/44976698 shutdown_connections 2024-01-11T09:27:17.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.414+0000 7fb44d5ff700 1 -- 172.21.15.27:0/44976698 wait complete. 2024-01-11T09:27:17.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.416+0000 7fb44d5ff700 1 Processor -- start 2024-01-11T09:27:17.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.416+0000 7fb44d5ff700 1 -- start start 2024-01-11T09:27:17.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.417+0000 7fb44d5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4480b03e0 0x7fb4480a8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:17.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.417+0000 7fb44d5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4480b07b0 0x7fb4480a6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:17.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.417+0000 7fb44d5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4480b29c0 0x7fb448169b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:17.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.417+0000 7fb44d5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480a8c90 con 0x7fb4480b07b0 2024-01-11T09:27:17.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.417+0000 7fb44d5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480a7480 con 0x7fb4480b03e0 2024-01-11T09:27:17.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.417+0000 7fb44d5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480aec80 con 0x7fb4480b29c0 2024-01-11T09:27:17.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.417+0000 7fb4467fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4480b07b0 0x7fb4480a6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:17.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.417+0000 7fb4467fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4480b07b0 0x7fb4480a6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37026/0 (socket says 172.21.15.27:37026) 2024-01-11T09:27:17.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.417+0000 7fb4467fc700 1 -- 172.21.15.27:0/1728905148 learned_addr learned my addr 172.21.15.27:0/1728905148 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:17.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.417+0000 7fb446ffd700 1 --2- 172.21.15.27:0/1728905148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4480b03e0 0x7fb4480a8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:17.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.417+0000 7fb4477fe700 1 --2- 172.21.15.27:0/1728905148 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4480b29c0 0x7fb448169b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:17.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.417+0000 7fb4467fc700 1 -- 172.21.15.27:0/1728905148 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4480b29c0 msgr2=0x7fb448169b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:17.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.417+0000 7fb4467fc700 1 --2- 172.21.15.27:0/1728905148 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4480b29c0 0x7fb448169b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:17.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.417+0000 7fb4467fc700 1 -- 172.21.15.27:0/1728905148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4480b03e0 msgr2=0x7fb4480a8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:17.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.418+0000 7fb4467fc700 1 --2- 172.21.15.27:0/1728905148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4480b03e0 0x7fb4480a8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:17.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.418+0000 7fb4467fc700 1 -- 172.21.15.27:0/1728905148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb438009420 con 0x7fb4480b07b0 2024-01-11T09:27:17.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.418+0000 7fb446ffd700 1 --2- 172.21.15.27:0/1728905148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4480b03e0 0x7fb4480a8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:27:17.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.418+0000 7fb4467fc700 1 --2- 172.21.15.27:0/1728905148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4480b07b0 0x7fb4480a6f40 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fb43c0058d0 tx=0x7fb43c007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:17.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.418+0000 7fb42effd700 1 -- 172.21.15.27:0/1728905148 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43c005b30 con 0x7fb4480b07b0 2024-01-11T09:27:17.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.418+0000 7fb42effd700 1 -- 172.21.15.27:0/1728905148 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43c005c90 con 0x7fb4480b07b0 2024-01-11T09:27:17.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.418+0000 7fb42effd700 1 -- 172.21.15.27:0/1728905148 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43c007300 con 0x7fb4480b07b0 2024-01-11T09:27:17.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.418+0000 7fb44d5ff700 1 -- 172.21.15.27:0/1728905148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4480a82c0 con 0x7fb4480b07b0 2024-01-11T09:27:17.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.418+0000 7fb44d5ff700 1 -- 172.21.15.27:0/1728905148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4480a7990 con 0x7fb4480b07b0 2024-01-11T09:27:17.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.419+0000 7fb44d5ff700 1 -- 172.21.15.27:0/1728905148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4480718d0 con 0x7fb4480b07b0 2024-01-11T09:27:17.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.420+0000 7fb42effd700 1 -- 172.21.15.27:0/1728905148 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb43c01b400 con 0x7fb4480b07b0 2024-01-11T09:27:17.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.421+0000 7fb42effd700 1 --2- 172.21.15.27:0/1728905148 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb434066f30 0x7fb4340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:17.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.422+0000 7fb42effd700 1 -- 172.21.15.27:0/1728905148 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 4909+0+0 (secure 0 0 0) 0x7fb43c083d70 con 0x7fb4480b07b0 2024-01-11T09:27:17.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.422+0000 7fb446ffd700 1 --2- 172.21.15.27:0/1728905148 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb434066f30 0x7fb4340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:17.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.423+0000 7fb446ffd700 1 --2- 172.21.15.27:0/1728905148 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb434066f30 0x7fb4340693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb4380091e0 tx=0x7fb438017000 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:17.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.425+0000 7fb42effd700 1 -- 172.21.15.27:0/1728905148 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb43c054d70 con 0x7fb4480b07b0 2024-01-11T09:27:17.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.628+0000 7fb44d5ff700 1 -- 172.21.15.27:0/1728905148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4480b2e00 con 0x7fb4480b07b0 2024-01-11T09:27:17.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.629+0000 7fb42effd700 1 -- 172.21.15.27:0/1728905148 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v177) v1 ==== 74+0+12184 (secure 0 0 0) 0x7fb4480b2e00 con 0x7fb4480b07b0 2024-01-11T09:27:17.626 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:27:17.626 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":177,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:27:16.657466+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:27:17.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.632+0000 7fb44d5ff700 1 -- 172.21.15.27:0/1728905148 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb434066f30 msgr2=0x7fb4340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:17.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.632+0000 7fb44d5ff700 1 --2- 172.21.15.27:0/1728905148 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb434066f30 0x7fb4340693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb4380091e0 tx=0x7fb438017000 comp rx=0 tx=0).stop 2024-01-11T09:27:17.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.632+0000 7fb44d5ff700 1 -- 172.21.15.27:0/1728905148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4480b07b0 msgr2=0x7fb4480a6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.632+0000 7fb44d5ff700 1 --2- 172.21.15.27:0/1728905148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4480b07b0 0x7fb4480a6f40 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fb43c0058d0 tx=0x7fb43c007ea0 comp rx=0 tx=0).stop 2024-01-11T09:27:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.632+0000 7fb44d5ff700 1 -- 172.21.15.27:0/1728905148 shutdown_connections 2024-01-11T09:27:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.632+0000 7fb44d5ff700 1 --2- 172.21.15.27:0/1728905148 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb434066f30 0x7fb4340693e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.632+0000 7fb44d5ff700 1 --2- 172.21.15.27:0/1728905148 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4480b29c0 0x7fb448169b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.632+0000 7fb44d5ff700 1 --2- 172.21.15.27:0/1728905148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4480b03e0 0x7fb4480a8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.632+0000 7fb44d5ff700 1 --2- 172.21.15.27:0/1728905148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4480b07b0 0x7fb4480a6f40 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.632+0000 7fb44d5ff700 1 -- 172.21.15.27:0/1728905148 >> 172.21.15.27:0/1728905148 conn(0x7fb44806ed40 msgr2=0x7fb4480ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.632+0000 7fb44d5ff700 1 -- 172.21.15.27:0/1728905148 shutdown_connections 2024-01-11T09:27:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.632+0000 7fb44d5ff700 1 -- 172.21.15.27:0/1728905148 wait complete. 2024-01-11T09:27:17.647 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-11T09:27:17.647 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T09:27:17.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.852+0000 7fe8169ff700 1 Processor -- start 2024-01-11T09:27:17.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.852+0000 7fe8169ff700 1 -- start start 2024-01-11T09:27:17.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.853+0000 7fe8169ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe810142dc0 0x7fe8100cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:17.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.853+0000 7fe8169ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8100cb2e0 0x7fe8100cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:17.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.853+0000 7fe8169ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8100ccc70 con 0x7fe810143190 2024-01-11T09:27:17.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.853+0000 7fe8169ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe81015df30 con 0x7fe810142dc0 2024-01-11T09:27:17.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.853+0000 7fe8169ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8100cb740 con 0x7fe8100cb2e0 2024-01-11T09:27:17.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.854+0000 7fe814f9c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8100cb2e0 0x7fe8100cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:17.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.854+0000 7fe80ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe810142dc0 0x7fe8100cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.854+0000 7fe814f9c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8100cb2e0 0x7fe8100cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:58016/0 (socket says 172.21.15.27:58016) 2024-01-11T09:27:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.854+0000 7fe814f9c700 1 -- 172.21.15.27:0/3711355042 learned_addr learned my addr 172.21.15.27:0/3711355042 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.854+0000 7fe80ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe810142dc0 0x7fe8100cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:59122/0 (socket says 172.21.15.27:59122) 2024-01-11T09:27:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.854+0000 7fe80f7fe700 1 --1- 172.21.15.27:0/3711355042 >> v1:172.21.15.77:6789/0 conn(0x7fe810143190 0x7fe8100cc020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:37172/0 (socket says 172.21.15.27:37172) 2024-01-11T09:27:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.854+0000 7fe814f9c700 1 -- 172.21.15.27:0/3711355042 >> v2:172.21.15.77:3300/0 conn(0x7fe810142dc0 msgr2=0x7fe8100cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.855+0000 7fe814f9c700 1 --2- 172.21.15.27:0/3711355042 >> v2:172.21.15.77:3300/0 conn(0x7fe810142dc0 0x7fe8100cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.855+0000 7fe814f9c700 1 -- 172.21.15.27:0/3711355042 >> v1:172.21.15.77:6789/0 conn(0x7fe810143190 legacy=0x7fe8100cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.855+0000 7fe814f9c700 1 -- 172.21.15.27:0/3711355042 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81015d320 con 0x7fe8100cb2e0 2024-01-11T09:27:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.855+0000 7fe80ffff700 1 --2- 172.21.15.27:0/3711355042 >> v2:172.21.15.77:3300/0 conn(0x7fe810142dc0 0x7fe8100cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:27:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.855+0000 7fe80effd700 1 -- 172.21.15.27:0/3711355042 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395977964 0 0) 0x7fe8100ccc70 con 0x7fe810143190 2024-01-11T09:27:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.855+0000 7fe814f9c700 1 --2- 172.21.15.27:0/3711355042 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8100cb2e0 0x7fe8100cc730 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe808009f20 tx=0x7fe808009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dc8e9068b53683c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.855+0000 7fe80effd700 1 -- 172.21.15.27:0/3711355042 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80800e070 con 0x7fe8100cb2e0 2024-01-11T09:27:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.855+0000 7fe80effd700 1 -- 172.21.15.27:0/3711355042 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80800b9d0 con 0x7fe8100cb2e0 2024-01-11T09:27:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.855+0000 7fe80effd700 1 -- 172.21.15.27:0/3711355042 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8080092e0 con 0x7fe8100cb2e0 2024-01-11T09:27:17.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.856+0000 7fe8169ff700 1 -- 172.21.15.27:0/3711355042 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8100cb2e0 msgr2=0x7fe8100cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:17.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.856+0000 7fe8169ff700 1 --2- 172.21.15.27:0/3711355042 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8100cb2e0 0x7fe8100cc730 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe808009f20 tx=0x7fe808009840 comp rx=0 tx=0).stop 2024-01-11T09:27:17.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.856+0000 7fe8169ff700 1 -- 172.21.15.27:0/3711355042 shutdown_connections 2024-01-11T09:27:17.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.856+0000 7fe8169ff700 1 --2- 172.21.15.27:0/3711355042 >> v2:172.21.15.77:3300/0 conn(0x7fe810142dc0 0x7fe8100cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:17.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.856+0000 7fe8169ff700 1 --2- 172.21.15.27:0/3711355042 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8100cb2e0 0x7fe8100cc730 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:17.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.856+0000 7fe8169ff700 1 -- 172.21.15.27:0/3711355042 >> 172.21.15.27:0/3711355042 conn(0x7fe810063ca0 msgr2=0x7fe81006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:17.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.856+0000 7fe8169ff700 1 -- 172.21.15.27:0/3711355042 shutdown_connections 2024-01-11T09:27:17.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.856+0000 7fe8169ff700 1 -- 172.21.15.27:0/3711355042 wait complete. 2024-01-11T09:27:17.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.858+0000 7fe8169ff700 1 Processor -- start 2024-01-11T09:27:17.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.858+0000 7fe8169ff700 1 -- start start 2024-01-11T09:27:17.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.859+0000 7fe8169ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8100cb2e0 0x7fe8100b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:17.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.859+0000 7fe8169ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe810142dc0 0x7fe81013ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:17.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.859+0000 7fe8169ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe810143190 0x7fe8100b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:17.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.859+0000 7fe8169ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8100b3520 con 0x7fe810143190 2024-01-11T09:27:17.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.859+0000 7fe8169ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81015a6c0 con 0x7fe810142dc0 2024-01-11T09:27:17.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.859+0000 7fe8169ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8100b20d0 con 0x7fe8100cb2e0 2024-01-11T09:27:17.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.860+0000 7fe814f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe810143190 0x7fe8100b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:17.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.860+0000 7fe814f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe810143190 0x7fe8100b39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37030/0 (socket says 172.21.15.27:37030) 2024-01-11T09:27:17.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.860+0000 7fe814f9c700 1 -- 172.21.15.27:0/2102198666 learned_addr learned my addr 172.21.15.27:0/2102198666 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:17.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.860+0000 7fe80ffff700 1 --2- 172.21.15.27:0/2102198666 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8100cb2e0 0x7fe8100b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:17.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.860+0000 7fe80f7fe700 1 --2- 172.21.15.27:0/2102198666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe810142dc0 0x7fe81013ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:17.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.860+0000 7fe814f9c700 1 -- 172.21.15.27:0/2102198666 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8100cb2e0 msgr2=0x7fe8100b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:17.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.860+0000 7fe814f9c700 1 --2- 172.21.15.27:0/2102198666 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8100cb2e0 0x7fe8100b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:17.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.860+0000 7fe814f9c700 1 -- 172.21.15.27:0/2102198666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe810142dc0 msgr2=0x7fe81013ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:17.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.860+0000 7fe814f9c700 1 --2- 172.21.15.27:0/2102198666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe810142dc0 0x7fe81013ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:17.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.860+0000 7fe814f9c700 1 -- 172.21.15.27:0/2102198666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe800009420 con 0x7fe810143190 2024-01-11T09:27:17.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.861+0000 7fe814f9c700 1 --2- 172.21.15.27:0/2102198666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe810143190 0x7fe8100b39b0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fe808009430 tx=0x7fe808018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:17.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.861+0000 7fe7f7fff700 1 -- 172.21.15.27:0/2102198666 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe808008b00 con 0x7fe810143190 2024-01-11T09:27:17.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.861+0000 7fe7f7fff700 1 -- 172.21.15.27:0/2102198666 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe808008ca0 con 0x7fe810143190 2024-01-11T09:27:17.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.861+0000 7fe8169ff700 1 -- 172.21.15.27:0/2102198666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe808009ae0 con 0x7fe810143190 2024-01-11T09:27:17.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.861+0000 7fe7f7fff700 1 -- 172.21.15.27:0/2102198666 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8080266b0 con 0x7fe810143190 2024-01-11T09:27:17.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.861+0000 7fe8169ff700 1 -- 172.21.15.27:0/2102198666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8100b2bf0 con 0x7fe810143190 2024-01-11T09:27:17.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.862+0000 7fe8169ff700 1 -- 172.21.15.27:0/2102198666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe81004c3c0 con 0x7fe810143190 2024-01-11T09:27:17.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.863+0000 7fe7f7fff700 1 -- 172.21.15.27:0/2102198666 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe808003040 con 0x7fe810143190 2024-01-11T09:27:17.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.864+0000 7fe7f7fff700 1 --2- 172.21.15.27:0/2102198666 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe7fc066f30 0x7fe7fc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:17.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.864+0000 7fe7f7fff700 1 -- 172.21.15.27:0/2102198666 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 4909+0+0 (secure 0 0 0) 0x7fe8080861c0 con 0x7fe810143190 2024-01-11T09:27:17.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.865+0000 7fe80ffff700 1 --2- 172.21.15.27:0/2102198666 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe7fc066f30 0x7fe7fc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:17.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.865+0000 7fe80ffff700 1 --2- 172.21.15.27:0/2102198666 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe7fc066f30 0x7fe7fc0693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe810142410 tx=0x7fe8000091a0 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:17.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:17.868+0000 7fe7f7fff700 1 -- 172.21.15.27:0/2102198666 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe8080536b0 con 0x7fe810143190 2024-01-11T09:27:18.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.069+0000 7fe8169ff700 1 -- 172.21.15.27:0/2102198666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe81015d620 con 0x7fe810143190 2024-01-11T09:27:18.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.070+0000 7fe7f7fff700 1 -- 172.21.15.27:0/2102198666 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v177) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe808056cd0 con 0x7fe810143190 2024-01-11T09:27:18.067 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T09:27:18.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.072+0000 7fe8169ff700 1 -- 172.21.15.27:0/2102198666 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe7fc066f30 msgr2=0x7fe7fc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:18.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.072+0000 7fe8169ff700 1 --2- 172.21.15.27:0/2102198666 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe7fc066f30 0x7fe7fc0693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe810142410 tx=0x7fe8000091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.072+0000 7fe8169ff700 1 -- 172.21.15.27:0/2102198666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe810143190 msgr2=0x7fe8100b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:18.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.072+0000 7fe8169ff700 1 --2- 172.21.15.27:0/2102198666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe810143190 0x7fe8100b39b0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fe808009430 tx=0x7fe808018000 comp rx=0 tx=0).stop 2024-01-11T09:27:18.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.072+0000 7fe8169ff700 1 -- 172.21.15.27:0/2102198666 shutdown_connections 2024-01-11T09:27:18.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.072+0000 7fe8169ff700 1 --2- 172.21.15.27:0/2102198666 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe7fc066f30 0x7fe7fc0693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.072+0000 7fe8169ff700 1 --2- 172.21.15.27:0/2102198666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe810143190 0x7fe8100b39b0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.072+0000 7fe8169ff700 1 --2- 172.21.15.27:0/2102198666 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe8100cb2e0 0x7fe8100b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.072+0000 7fe8169ff700 1 --2- 172.21.15.27:0/2102198666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe810142dc0 0x7fe81013ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.072+0000 7fe8169ff700 1 -- 172.21.15.27:0/2102198666 >> 172.21.15.27:0/2102198666 conn(0x7fe810063ca0 msgr2=0x7fe81015afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:18.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.072+0000 7fe8169ff700 1 -- 172.21.15.27:0/2102198666 shutdown_connections 2024-01-11T09:27:18.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.072+0000 7fe8169ff700 1 -- 172.21.15.27:0/2102198666 wait complete. 2024-01-11T09:27:18.087 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-11T09:27:18.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.289+0000 7f6f0a1ff700 1 Processor -- start 2024-01-11T09:27:18.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.289+0000 7f6f0a1ff700 1 -- start start 2024-01-11T09:27:18.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.291+0000 7f6f0a1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f04069d40 0x7f6f040b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:18.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.291+0000 7f6f0a1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6f040c89f0 0x7f6f040b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:18.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.291+0000 7f6f0a1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f04157390 con 0x7f6f040c8620 2024-01-11T09:27:18.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.291+0000 7f6f0a1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f040adcd0 con 0x7f6f040c89f0 2024-01-11T09:27:18.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.291+0000 7f6f0a1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0406a200 con 0x7f6f04069d40 2024-01-11T09:27:18.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.291+0000 7f6f02ffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f04069d40 0x7f6f040b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:18.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.291+0000 7f6f03fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6f040c89f0 0x7f6f040b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:18.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.291+0000 7f6f02ffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f04069d40 0x7f6f040b02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:58040/0 (socket says 172.21.15.27:58040) 2024-01-11T09:27:18.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.291+0000 7f6f03fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6f040c89f0 0x7f6f040b0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:59134/0 (socket says 172.21.15.27:59134) 2024-01-11T09:27:18.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.291+0000 7f6f02ffd700 1 -- 172.21.15.27:0/1461827707 learned_addr learned my addr 172.21.15.27:0/1461827707 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:18.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.291+0000 7f6f037fe700 1 --1- 172.21.15.27:0/1461827707 >> v1:172.21.15.77:6789/0 conn(0x7f6f040c8620 0x7f6f040af970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:37180/0 (socket says 172.21.15.27:37180) 2024-01-11T09:27:18.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.292+0000 7f6f02ffd700 1 -- 172.21.15.27:0/1461827707 >> v2:172.21.15.77:3300/0 conn(0x7f6f040c89f0 msgr2=0x7f6f040b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:18.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.292+0000 7f6f02ffd700 1 --2- 172.21.15.27:0/1461827707 >> v2:172.21.15.77:3300/0 conn(0x7f6f040c89f0 0x7f6f040b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.292+0000 7f6f02ffd700 1 -- 172.21.15.27:0/1461827707 >> v1:172.21.15.77:6789/0 conn(0x7f6f040c8620 legacy=0x7f6f040af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:18.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.292+0000 7f6f02ffd700 1 -- 172.21.15.27:0/1461827707 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f04156d40 con 0x7f6f04069d40 2024-01-11T09:27:18.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.292+0000 7f6f02ffd700 1 --2- 172.21.15.27:0/1461827707 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f04069d40 0x7f6f040b02d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6ef80099e0 tx=0x7f6ef8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1dfb16482730be8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.292+0000 7f6f027fc700 1 -- 172.21.15.27:0/1461827707 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef800b820 con 0x7f6f04069d40 2024-01-11T09:27:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.292+0000 7f6f027fc700 1 -- 172.21.15.27:0/1461827707 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef800b980 con 0x7f6f04069d40 2024-01-11T09:27:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.292+0000 7f6f027fc700 1 -- 172.21.15.27:0/1461827707 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef800bd20 con 0x7f6f04069d40 2024-01-11T09:27:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.293+0000 7f6f0a1ff700 1 -- 172.21.15.27:0/1461827707 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f04069d40 msgr2=0x7f6f040b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.293+0000 7f6f0a1ff700 1 --2- 172.21.15.27:0/1461827707 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f04069d40 0x7f6f040b02d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6ef80099e0 tx=0x7f6ef8009250 comp rx=0 tx=0).stop 2024-01-11T09:27:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.293+0000 7f6f0a1ff700 1 -- 172.21.15.27:0/1461827707 shutdown_connections 2024-01-11T09:27:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.293+0000 7f6f0a1ff700 1 --2- 172.21.15.27:0/1461827707 >> v2:172.21.15.77:3300/0 conn(0x7f6f040c89f0 0x7f6f040b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.293+0000 7f6f0a1ff700 1 --2- 172.21.15.27:0/1461827707 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f04069d40 0x7f6f040b02d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.293+0000 7f6f0a1ff700 1 -- 172.21.15.27:0/1461827707 >> 172.21.15.27:0/1461827707 conn(0x7f6f040680e0 msgr2=0x7f6f04069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.293+0000 7f6f0a1ff700 1 -- 172.21.15.27:0/1461827707 shutdown_connections 2024-01-11T09:27:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.293+0000 7f6f0a1ff700 1 -- 172.21.15.27:0/1461827707 wait complete. 2024-01-11T09:27:18.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.295+0000 7f6f0a1ff700 1 Processor -- start 2024-01-11T09:27:18.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.295+0000 7f6f0a1ff700 1 -- start start 2024-01-11T09:27:18.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.296+0000 7f6f0a1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f04069d40 0x7f6f04145a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:18.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.296+0000 7f6f0a1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f040c8620 0x7f6f040cb100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:18.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.296+0000 7f6f0a1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f040c89f0 0x7f6f04146300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:18.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.296+0000 7f6f0a1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f04145fa0 con 0x7f6f040c8620 2024-01-11T09:27:18.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.296+0000 7f6f0a1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f041583c0 con 0x7f6f040c89f0 2024-01-11T09:27:18.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.296+0000 7f6f0a1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f040ca2d0 con 0x7f6f04069d40 2024-01-11T09:27:18.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.296+0000 7f6f02ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f040c8620 0x7f6f040cb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:18.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.296+0000 7f6f02ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f040c8620 0x7f6f040cb100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37032/0 (socket says 172.21.15.27:37032) 2024-01-11T09:27:18.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.296+0000 7f6f02ffd700 1 -- 172.21.15.27:0/2232196262 learned_addr learned my addr 172.21.15.27:0/2232196262 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:18.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.296+0000 7f6f037fe700 1 --2- 172.21.15.27:0/2232196262 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f04069d40 0x7f6f04145a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:18.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.296+0000 7f6f03fff700 1 --2- 172.21.15.27:0/2232196262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f040c89f0 0x7f6f04146300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:18.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.297+0000 7f6f02ffd700 1 -- 172.21.15.27:0/2232196262 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f04069d40 msgr2=0x7f6f04145a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:18.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.297+0000 7f6f02ffd700 1 --2- 172.21.15.27:0/2232196262 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f04069d40 0x7f6f04145a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.297+0000 7f6f02ffd700 1 -- 172.21.15.27:0/2232196262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f040c89f0 msgr2=0x7f6f04146300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:18.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.297+0000 7f6f02ffd700 1 --2- 172.21.15.27:0/2232196262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f040c89f0 0x7f6f04146300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.297+0000 7f6f02ffd700 1 -- 172.21.15.27:0/2232196262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef80094f0 con 0x7f6f040c8620 2024-01-11T09:27:18.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.297+0000 7f6f02ffd700 1 --2- 172.21.15.27:0/2232196262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f040c8620 0x7f6f040cb100 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8009290 tx=0x7f6ef8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:18.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.297+0000 7f6eeb7fe700 1 -- 172.21.15.27:0/2232196262 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8011d70 con 0x7f6f040c8620 2024-01-11T09:27:18.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.297+0000 7f6eeb7fe700 1 -- 172.21.15.27:0/2232196262 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef8024440 con 0x7f6f040c8620 2024-01-11T09:27:18.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.297+0000 7f6eeb7fe700 1 -- 172.21.15.27:0/2232196262 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8024800 con 0x7f6f040c8620 2024-01-11T09:27:18.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.297+0000 7f6f0a1ff700 1 -- 172.21.15.27:0/2232196262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f040ca5b0 con 0x7f6f040c8620 2024-01-11T09:27:18.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.297+0000 7f6f0a1ff700 1 -- 172.21.15.27:0/2232196262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f04147380 con 0x7f6f040c8620 2024-01-11T09:27:18.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.298+0000 7f6f0a1ff700 1 -- 172.21.15.27:0/2232196262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f04158190 con 0x7f6f040c8620 2024-01-11T09:27:18.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.300+0000 7f6eeb7fe700 1 -- 172.21.15.27:0/2232196262 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6ef802d440 con 0x7f6f040c8620 2024-01-11T09:27:18.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.301+0000 7f6eeb7fe700 1 --2- 172.21.15.27:0/2232196262 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6ef0066f30 0x7f6ef00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:18.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.301+0000 7f6eeb7fe700 1 -- 172.21.15.27:0/2232196262 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 4909+0+0 (secure 0 0 0) 0x7f6ef8085b90 con 0x7f6f040c8620 2024-01-11T09:27:18.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.301+0000 7f6f037fe700 1 --2- 172.21.15.27:0/2232196262 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6ef0066f30 0x7f6ef00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:18.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.302+0000 7f6f037fe700 1 --2- 172.21.15.27:0/2232196262 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6ef0066f30 0x7f6ef00693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4004fd0 tx=0x7f6ef4007eb0 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:18.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.305+0000 7f6eeb7fe700 1 -- 172.21.15.27:0/2232196262 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6ef8053080 con 0x7f6f040c8620 2024-01-11T09:27:18.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.510+0000 7f6f0a1ff700 1 -- 172.21.15.27:0/2232196262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6f04158530 con 0x7f6f040c8620 2024-01-11T09:27:18.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.510+0000 7f6eeb7fe700 1 -- 172.21.15.27:0/2232196262 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v177) v1 ==== 84+0+65 (secure 0 0 0) 0x7f6ef8015060 con 0x7f6f040c8620 2024-01-11T09:27:18.507 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:27:18.507 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T09:27:18.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:18.507 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-11T09:27:18.507 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-11T09:27:18.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:18.507 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T09:27:18.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.512+0000 7f6f0a1ff700 1 -- 172.21.15.27:0/2232196262 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6ef0066f30 msgr2=0x7f6ef00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:18.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.512+0000 7f6f0a1ff700 1 --2- 172.21.15.27:0/2232196262 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6ef0066f30 0x7f6ef00693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4004fd0 tx=0x7f6ef4007eb0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.512+0000 7f6f0a1ff700 1 -- 172.21.15.27:0/2232196262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f040c8620 msgr2=0x7f6f040cb100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:18.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.512+0000 7f6f0a1ff700 1 --2- 172.21.15.27:0/2232196262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f040c8620 0x7f6f040cb100 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8009290 tx=0x7f6ef8017000 comp rx=0 tx=0).stop 2024-01-11T09:27:18.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.512+0000 7f6f0a1ff700 1 -- 172.21.15.27:0/2232196262 shutdown_connections 2024-01-11T09:27:18.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.512+0000 7f6f0a1ff700 1 --2- 172.21.15.27:0/2232196262 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6ef0066f30 0x7f6ef00693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.513+0000 7f6f0a1ff700 1 --2- 172.21.15.27:0/2232196262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f040c89f0 0x7f6f04146300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.513+0000 7f6f0a1ff700 1 --2- 172.21.15.27:0/2232196262 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f04069d40 0x7f6f04145a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.513+0000 7f6f0a1ff700 1 --2- 172.21.15.27:0/2232196262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f040c8620 0x7f6f040cb100 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.513+0000 7f6f0a1ff700 1 -- 172.21.15.27:0/2232196262 >> 172.21.15.27:0/2232196262 conn(0x7f6f040680e0 msgr2=0x7f6f0406d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:18.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.513+0000 7f6f0a1ff700 1 -- 172.21.15.27:0/2232196262 shutdown_connections 2024-01-11T09:27:18.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.513+0000 7f6f0a1ff700 1 -- 172.21.15.27:0/2232196262 wait complete. 2024-01-11T09:27:18.527 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-11T09:27:18.527 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-11T09:27:18.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.734+0000 7f79d69ff700 1 Processor -- start 2024-01-11T09:27:18.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.734+0000 7f79d69ff700 1 -- start start 2024-01-11T09:27:18.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.736+0000 7f79d69ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79d0159e60 0x7f79d015aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:18.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.736+0000 7f79d69ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f79d015b400 0x7f79d00685e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:18.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.736+0000 7f79d69ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d015c4b0 con 0x7f79d015a7a0 2024-01-11T09:27:18.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.736+0000 7f79d69ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79d015d050 con 0x7f79d015b400 2024-01-11T09:27:18.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.736+0000 7f79d69ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d015b8a0 con 0x7f79d0159e60 2024-01-11T09:27:18.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.736+0000 7f79d4f9c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f79d015b400 0x7f79d00685e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:18.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.736+0000 7f79cf7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79d0159e60 0x7f79d015aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:18.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.736+0000 7f79d4f9c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f79d015b400 0x7f79d00685e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:59148/0 (socket says 172.21.15.27:59148) 2024-01-11T09:27:18.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.736+0000 7f79d4f9c700 1 -- 172.21.15.27:0/4105887658 learned_addr learned my addr 172.21.15.27:0/4105887658 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:18.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.736+0000 7f79cffff700 1 --1- 172.21.15.27:0/4105887658 >> v1:172.21.15.77:6789/0 conn(0x7f79d015a7a0 0x7f79d015c940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:37182/0 (socket says 172.21.15.27:37182) 2024-01-11T09:27:18.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.737+0000 7f79d4f9c700 1 -- 172.21.15.27:0/4105887658 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79d0159e60 msgr2=0x7f79d015aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:18.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.737+0000 7f79d4f9c700 1 --2- 172.21.15.27:0/4105887658 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79d0159e60 0x7f79d015aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.737+0000 7f79d4f9c700 1 -- 172.21.15.27:0/4105887658 >> v1:172.21.15.77:6789/0 conn(0x7f79d015a7a0 legacy=0x7f79d015c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:18.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.737+0000 7f79d4f9c700 1 -- 172.21.15.27:0/4105887658 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d015bb80 con 0x7f79d015b400 2024-01-11T09:27:18.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.737+0000 7f79ceffd700 1 -- 172.21.15.27:0/4105887658 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547871314 0 0) 0x7f79d015c4b0 con 0x7f79d015a7a0 2024-01-11T09:27:18.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.737+0000 7f79d4f9c700 1 --2- 172.21.15.27:0/4105887658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79d015b400 0x7f79d00685e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f79c8009f20 tx=0x7f79c8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc87fb5994f08498 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:18.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.737+0000 7f79ceffd700 1 -- 172.21.15.27:0/4105887658 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c800e070 con 0x7f79d015b400 2024-01-11T09:27:18.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.738+0000 7f79ceffd700 1 -- 172.21.15.27:0/4105887658 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c800b9d0 con 0x7f79d015b400 2024-01-11T09:27:18.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.738+0000 7f79ceffd700 1 -- 172.21.15.27:0/4105887658 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c80092e0 con 0x7f79d015b400 2024-01-11T09:27:18.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.738+0000 7f79d69ff700 1 -- 172.21.15.27:0/4105887658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79d015b400 msgr2=0x7f79d00685e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:18.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.738+0000 7f79d69ff700 1 --2- 172.21.15.27:0/4105887658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79d015b400 0x7f79d00685e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f79c8009f20 tx=0x7f79c8009840 comp rx=0 tx=0).stop 2024-01-11T09:27:18.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.738+0000 7f79d69ff700 1 -- 172.21.15.27:0/4105887658 shutdown_connections 2024-01-11T09:27:18.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.738+0000 7f79d69ff700 1 --2- 172.21.15.27:0/4105887658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79d015b400 0x7f79d00685e0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.738+0000 7f79d69ff700 1 --2- 172.21.15.27:0/4105887658 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79d0159e60 0x7f79d015aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.738+0000 7f79d69ff700 1 -- 172.21.15.27:0/4105887658 >> 172.21.15.27:0/4105887658 conn(0x7f79d006ed40 msgr2=0x7f79d006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:18.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.738+0000 7f79d69ff700 1 -- 172.21.15.27:0/4105887658 shutdown_connections 2024-01-11T09:27:18.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.738+0000 7f79d69ff700 1 -- 172.21.15.27:0/4105887658 wait complete. 2024-01-11T09:27:18.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.740+0000 7f79d69ff700 1 Processor -- start 2024-01-11T09:27:18.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.740+0000 7f79d69ff700 1 -- start start 2024-01-11T09:27:18.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.741+0000 7f79d69ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79d015a7a0 0x7f79d01366b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:18.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.741+0000 7f79d69ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79d015b400 0x7f79d00aff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:18.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.741+0000 7f79d69ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79d00b04b0 0x7f79d00b2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:18.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.741+0000 7f79d69ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d00b0950 con 0x7f79d015b400 2024-01-11T09:27:18.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.741+0000 7f79d69ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d014a480 con 0x7f79d00b04b0 2024-01-11T09:27:18.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.741+0000 7f79d69ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d014c2e0 con 0x7f79d015a7a0 2024-01-11T09:27:18.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.741+0000 7f79cf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79d015b400 0x7f79d00aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:18.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.741+0000 7f79cf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79d015b400 0x7f79d00aff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37042/0 (socket says 172.21.15.27:37042) 2024-01-11T09:27:18.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.741+0000 7f79cf7fe700 1 -- 172.21.15.27:0/924412567 learned_addr learned my addr 172.21.15.27:0/924412567 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:18.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.742+0000 7f79cffff700 1 --2- 172.21.15.27:0/924412567 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79d015a7a0 0x7f79d01366b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:18.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.742+0000 7f79d4f9c700 1 --2- 172.21.15.27:0/924412567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79d00b04b0 0x7f79d00b2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:18.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.742+0000 7f79cf7fe700 1 -- 172.21.15.27:0/924412567 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79d015a7a0 msgr2=0x7f79d01366b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:18.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.742+0000 7f79cf7fe700 1 --2- 172.21.15.27:0/924412567 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79d015a7a0 0x7f79d01366b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.742+0000 7f79cf7fe700 1 -- 172.21.15.27:0/924412567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79d00b04b0 msgr2=0x7f79d00b2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:18.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.742+0000 7f79cf7fe700 1 --2- 172.21.15.27:0/924412567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79d00b04b0 0x7f79d00b2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:18.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.742+0000 7f79cf7fe700 1 -- 172.21.15.27:0/924412567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c8009ae0 con 0x7f79d015b400 2024-01-11T09:27:18.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.742+0000 7f79cf7fe700 1 --2- 172.21.15.27:0/924412567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79d015b400 0x7f79d00aff70 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f79c400e960 tx=0x7f79c40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:18.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.742+0000 7f79b7fff700 1 -- 172.21.15.27:0/924412567 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c4013680 con 0x7f79d015b400 2024-01-11T09:27:18.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.742+0000 7f79b7fff700 1 -- 172.21.15.27:0/924412567 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c40137e0 con 0x7f79d015b400 2024-01-11T09:27:18.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.743+0000 7f79d69ff700 1 -- 172.21.15.27:0/924412567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d014b6d0 con 0x7f79d015b400 2024-01-11T09:27:18.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.743+0000 7f79b7fff700 1 -- 172.21.15.27:0/924412567 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c4013b80 con 0x7f79d015b400 2024-01-11T09:27:18.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.743+0000 7f79d69ff700 1 -- 172.21.15.27:0/924412567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d014bc90 con 0x7f79d015b400 2024-01-11T09:27:18.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.744+0000 7f79d69ff700 1 -- 172.21.15.27:0/924412567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d014c470 con 0x7f79d015b400 2024-01-11T09:27:18.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.745+0000 7f79b7fff700 1 -- 172.21.15.27:0/924412567 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f79c40253d0 con 0x7f79d015b400 2024-01-11T09:27:18.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.746+0000 7f79b7fff700 1 --2- 172.21.15.27:0/924412567 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f79bc066f30 0x7f79bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:18.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.746+0000 7f79b7fff700 1 -- 172.21.15.27:0/924412567 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 4909+0+0 (secure 0 0 0) 0x7f79c4084b00 con 0x7f79d015b400 2024-01-11T09:27:18.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.746+0000 7f79cffff700 1 --2- 172.21.15.27:0/924412567 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f79bc066f30 0x7f79bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:18.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.747+0000 7f79cffff700 1 --2- 172.21.15.27:0/924412567 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f79bc066f30 0x7f79bc0693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f79d014c870 tx=0x7f79c0007d30 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:18.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.749+0000 7f79b7fff700 1 -- 172.21.15.27:0/924412567 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f79c4052070 con 0x7f79d015b400 2024-01-11T09:27:18.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:18.954+0000 7f79d69ff700 1 -- 172.21.15.27:0/924412567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f79d004c420 con 0x7f79d015b400 2024-01-11T09:27:19.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:19.953+0000 7f79b7fff700 1 -- 172.21.15.27:0/924412567 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v178) v1 ==== 138+0+0 (secure 0 0 0) 0x7f79c4055690 con 0x7f79d015b400 2024-01-11T09:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:19.957+0000 7f79d69ff700 1 -- 172.21.15.27:0/924412567 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f79bc066f30 msgr2=0x7f79bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:19.957+0000 7f79d69ff700 1 --2- 172.21.15.27:0/924412567 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f79bc066f30 0x7f79bc0693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f79d014c870 tx=0x7f79c0007d30 comp rx=0 tx=0).stop 2024-01-11T09:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:19.957+0000 7f79d69ff700 1 -- 172.21.15.27:0/924412567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79d015b400 msgr2=0x7f79d00aff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:19.957+0000 7f79d69ff700 1 --2- 172.21.15.27:0/924412567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79d015b400 0x7f79d00aff70 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f79c400e960 tx=0x7f79c40091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:19.958+0000 7f79d69ff700 1 -- 172.21.15.27:0/924412567 shutdown_connections 2024-01-11T09:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:19.958+0000 7f79d69ff700 1 --2- 172.21.15.27:0/924412567 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f79bc066f30 0x7f79bc0693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:19.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:19.958+0000 7f79d69ff700 1 --2- 172.21.15.27:0/924412567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79d00b04b0 0x7f79d00b2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:19.958+0000 7f79d69ff700 1 --2- 172.21.15.27:0/924412567 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79d015a7a0 0x7f79d01366b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:19.958+0000 7f79d69ff700 1 --2- 172.21.15.27:0/924412567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79d015b400 0x7f79d00aff70 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:19.958+0000 7f79d69ff700 1 -- 172.21.15.27:0/924412567 >> 172.21.15.27:0/924412567 conn(0x7f79d006ed40 msgr2=0x7f79d00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:19.958+0000 7f79d69ff700 1 -- 172.21.15.27:0/924412567 shutdown_connections 2024-01-11T09:27:19.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:19.959+0000 7f79d69ff700 1 -- 172.21.15.27:0/924412567 wait complete. 2024-01-11T09:27:19.958 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-11T09:27:19.978 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-11T09:27:20.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.192+0000 7f5cf43ff700 1 Processor -- start 2024-01-11T09:27:20.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.193+0000 7f5cf43ff700 1 -- start start 2024-01-11T09:27:20.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.194+0000 7f5cf43ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cec0b03e0 0x7f5cec0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:20.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.194+0000 7f5cf43ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5cec0b29c0 0x7f5cec159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:20.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.194+0000 7f5cf43ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cec0b2e00 con 0x7f5cec0b07b0 2024-01-11T09:27:20.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.194+0000 7f5cf43ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cec0cd730 con 0x7f5cec0b29c0 2024-01-11T09:27:20.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.194+0000 7f5cf43ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cec0ccb20 con 0x7f5cec0b03e0 2024-01-11T09:27:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.194+0000 7f5cf299c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5cec0b29c0 0x7f5cec159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.194+0000 7f5cf219b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cec0b03e0 0x7f5cec0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.194+0000 7f5cf299c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5cec0b29c0 0x7f5cec159910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:59168/0 (socket says 172.21.15.27:59168) 2024-01-11T09:27:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.194+0000 7f5cf219b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cec0b03e0 0x7f5cec0b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:58078/0 (socket says 172.21.15.27:58078) 2024-01-11T09:27:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.194+0000 7f5cf299c700 1 -- 172.21.15.27:0/4045905360 learned_addr learned my addr 172.21.15.27:0/4045905360 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.194+0000 7f5cf199a700 1 --1- 172.21.15.27:0/4045905360 >> v1:172.21.15.77:6789/0 conn(0x7f5cec0b07b0 0x7f5cec159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:37184/0 (socket says 172.21.15.27:37184) 2024-01-11T09:27:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.195+0000 7f5cf219b700 1 -- 172.21.15.27:0/4045905360 >> v2:172.21.15.77:3300/0 conn(0x7f5cec0b29c0 msgr2=0x7f5cec159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.195+0000 7f5cf219b700 1 --2- 172.21.15.27:0/4045905360 >> v2:172.21.15.77:3300/0 conn(0x7f5cec0b29c0 0x7f5cec159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.195+0000 7f5cf219b700 1 -- 172.21.15.27:0/4045905360 >> v1:172.21.15.77:6789/0 conn(0x7f5cec0b07b0 legacy=0x7f5cec159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.195+0000 7f5cf219b700 1 -- 172.21.15.27:0/4045905360 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cec0cce00 con 0x7f5cec0b03e0 2024-01-11T09:27:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.195+0000 7f5cf299c700 1 --2- 172.21.15.27:0/4045905360 >> v2:172.21.15.77:3300/0 conn(0x7f5cec0b29c0 0x7f5cec159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:27:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.195+0000 7f5cf1199700 1 -- 172.21.15.27:0/4045905360 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275644342 0 0) 0x7f5cdc001110 con 0x7f5cec0b07b0 2024-01-11T09:27:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.195+0000 7f5cf219b700 1 --2- 172.21.15.27:0/4045905360 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cec0b03e0 0x7f5cec0b2480 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5ce80098e0 tx=0x7f5ce80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b9d653b264d2417 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.196+0000 7f5cf1199700 1 -- 172.21.15.27:0/4045905360 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce800b640 con 0x7f5cec0b03e0 2024-01-11T09:27:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.196+0000 7f5cf1199700 1 -- 172.21.15.27:0/4045905360 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce800b7a0 con 0x7f5cec0b03e0 2024-01-11T09:27:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.196+0000 7f5cf1199700 1 -- 172.21.15.27:0/4045905360 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce800bb40 con 0x7f5cec0b03e0 2024-01-11T09:27:20.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.196+0000 7f5cf43ff700 1 -- 172.21.15.27:0/4045905360 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cec0b03e0 msgr2=0x7f5cec0b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:20.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.196+0000 7f5cf43ff700 1 --2- 172.21.15.27:0/4045905360 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cec0b03e0 0x7f5cec0b2480 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5ce80098e0 tx=0x7f5ce80091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:20.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.196+0000 7f5cf43ff700 1 -- 172.21.15.27:0/4045905360 shutdown_connections 2024-01-11T09:27:20.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.196+0000 7f5cf43ff700 1 --2- 172.21.15.27:0/4045905360 >> v2:172.21.15.77:3300/0 conn(0x7f5cec0b29c0 0x7f5cec159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:20.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.196+0000 7f5cf43ff700 1 --2- 172.21.15.27:0/4045905360 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cec0b03e0 0x7f5cec0b2480 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:20.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.196+0000 7f5cf43ff700 1 -- 172.21.15.27:0/4045905360 >> 172.21.15.27:0/4045905360 conn(0x7f5cec06ed40 msgr2=0x7f5cec06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:20.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.196+0000 7f5cf43ff700 1 -- 172.21.15.27:0/4045905360 shutdown_connections 2024-01-11T09:27:20.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.196+0000 7f5cf43ff700 1 -- 172.21.15.27:0/4045905360 wait complete. 2024-01-11T09:27:20.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.198+0000 7f5cf43ff700 1 Processor -- start 2024-01-11T09:27:20.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.198+0000 7f5cf43ff700 1 -- start start 2024-01-11T09:27:20.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.199+0000 7f5cf43ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cec0b03e0 0x7f5cec16a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:20.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.199+0000 7f5cf43ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cec0b07b0 0x7f5cec0ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:20.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.199+0000 7f5cf43ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cec0b29c0 0x7f5cec0c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:20.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.199+0000 7f5cf43ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cec16aa60 con 0x7f5cec0b03e0 2024-01-11T09:27:20.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.199+0000 7f5cf43ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cec0af730 con 0x7f5cec0b29c0 2024-01-11T09:27:20.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.199+0000 7f5cf43ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cec0afa10 con 0x7f5cec0b07b0 2024-01-11T09:27:20.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.199+0000 7f5cf219b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cec0b03e0 0x7f5cec16a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:20.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.199+0000 7f5cf219b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cec0b03e0 0x7f5cec16a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37048/0 (socket says 172.21.15.27:37048) 2024-01-11T09:27:20.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.199+0000 7f5cf219b700 1 -- 172.21.15.27:0/2048725562 learned_addr learned my addr 172.21.15.27:0/2048725562 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:20.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.199+0000 7f5cf199a700 1 --2- 172.21.15.27:0/2048725562 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cec0b07b0 0x7f5cec0ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:20.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.199+0000 7f5cf299c700 1 --2- 172.21.15.27:0/2048725562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cec0b29c0 0x7f5cec0c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:20.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.200+0000 7f5cf219b700 1 -- 172.21.15.27:0/2048725562 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cec0b07b0 msgr2=0x7f5cec0ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:20.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.200+0000 7f5cf219b700 1 --2- 172.21.15.27:0/2048725562 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cec0b07b0 0x7f5cec0ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:20.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.200+0000 7f5cf219b700 1 -- 172.21.15.27:0/2048725562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cec0b29c0 msgr2=0x7f5cec0c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:20.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.200+0000 7f5cf219b700 1 --2- 172.21.15.27:0/2048725562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cec0b29c0 0x7f5cec0c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:20.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.200+0000 7f5cf219b700 1 -- 172.21.15.27:0/2048725562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce4009ae0 con 0x7f5cec0b03e0 2024-01-11T09:27:20.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.200+0000 7f5cf219b700 1 --2- 172.21.15.27:0/2048725562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cec0b03e0 0x7f5cec16a520 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8009f80 tx=0x7f5ce8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:20.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.200+0000 7f5ce27fc700 1 -- 172.21.15.27:0/2048725562 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce800bb20 con 0x7f5cec0b03e0 2024-01-11T09:27:20.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.200+0000 7f5ce27fc700 1 -- 172.21.15.27:0/2048725562 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce8011420 con 0x7f5cec0b03e0 2024-01-11T09:27:20.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.200+0000 7f5cf43ff700 1 -- 172.21.15.27:0/2048725562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce8009420 con 0x7f5cec0b03e0 2024-01-11T09:27:20.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.200+0000 7f5ce27fc700 1 -- 172.21.15.27:0/2048725562 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8011800 con 0x7f5cec0b03e0 2024-01-11T09:27:20.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.200+0000 7f5cf43ff700 1 -- 172.21.15.27:0/2048725562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cec0c7d70 con 0x7f5cec0b03e0 2024-01-11T09:27:20.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.202+0000 7f5cf43ff700 1 -- 172.21.15.27:0/2048725562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cec04c3c0 con 0x7f5cec0b03e0 2024-01-11T09:27:20.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.202+0000 7f5ce27fc700 1 -- 172.21.15.27:0/2048725562 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5ce8011c30 con 0x7f5cec0b03e0 2024-01-11T09:27:20.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.203+0000 7f5ce27fc700 1 --2- 172.21.15.27:0/2048725562 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5cd0066f30 0x7f5cd00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:20.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.203+0000 7f5ce27fc700 1 -- 172.21.15.27:0/2048725562 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5279+0+0 (secure 0 0 0) 0x7f5ce8085f90 con 0x7f5cec0b03e0 2024-01-11T09:27:20.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.204+0000 7f5cf199a700 1 --2- 172.21.15.27:0/2048725562 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5cd0066f30 0x7f5cd00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:20.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.204+0000 7f5cf199a700 1 --2- 172.21.15.27:0/2048725562 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5cd0066f30 0x7f5cd00693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc005d80 tx=0x7f5cdc007a30 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:20.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.208+0000 7f5ce27fc700 1 -- 172.21.15.27:0/2048725562 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5ce8053310 con 0x7f5cec0b03e0 2024-01-11T09:27:20.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.413+0000 7f5cf43ff700 1 -- 172.21.15.27:0/2048725562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5cec0cd0e0 con 0x7f5cec0b03e0 2024-01-11T09:27:20.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.957+0000 7f5ce27fc700 1 -- 172.21.15.27:0/2048725562 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v179) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5ce8056930 con 0x7f5cec0b03e0 2024-01-11T09:27:20.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.963+0000 7f5cf43ff700 1 -- 172.21.15.27:0/2048725562 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5cd0066f30 msgr2=0x7f5cd00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:20.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.963+0000 7f5cf43ff700 1 --2- 172.21.15.27:0/2048725562 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5cd0066f30 0x7f5cd00693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc005d80 tx=0x7f5cdc007a30 comp rx=0 tx=0).stop 2024-01-11T09:27:20.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.963+0000 7f5cf43ff700 1 -- 172.21.15.27:0/2048725562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cec0b03e0 msgr2=0x7f5cec16a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:20.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.963+0000 7f5cf43ff700 1 --2- 172.21.15.27:0/2048725562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cec0b03e0 0x7f5cec16a520 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8009f80 tx=0x7f5ce8017000 comp rx=0 tx=0).stop 2024-01-11T09:27:20.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.963+0000 7f5cf43ff700 1 -- 172.21.15.27:0/2048725562 shutdown_connections 2024-01-11T09:27:20.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.963+0000 7f5cf43ff700 1 --2- 172.21.15.27:0/2048725562 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5cd0066f30 0x7f5cd00693e0 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc005d80 tx=0x7f5cdc007a30 comp rx=0 tx=0).stop 2024-01-11T09:27:20.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.963+0000 7f5cf43ff700 1 --2- 172.21.15.27:0/2048725562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cec0b29c0 0x7f5cec0c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:20.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.963+0000 7f5cf43ff700 1 --2- 172.21.15.27:0/2048725562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cec0b03e0 0x7f5cec16a520 secure :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8009f80 tx=0x7f5ce8017000 comp rx=0 tx=0).stop 2024-01-11T09:27:20.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.963+0000 7f5cf43ff700 1 --2- 172.21.15.27:0/2048725562 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cec0b07b0 0x7f5cec0ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:20.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.963+0000 7f5cf43ff700 1 -- 172.21.15.27:0/2048725562 >> 172.21.15.27:0/2048725562 conn(0x7f5cec06ed40 msgr2=0x7f5cec15a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:20.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.966+0000 7f5cf43ff700 1 -- 172.21.15.27:0/2048725562 shutdown_connections 2024-01-11T09:27:20.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:20.967+0000 7f5cf43ff700 1 -- 172.21.15.27:0/2048725562 wait complete. 2024-01-11T09:27:20.971 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-11T09:27:20.991 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-11T09:27:21.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.200+0000 7fe92b5ff700 1 Processor -- start 2024-01-11T09:27:21.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.200+0000 7fe92b5ff700 1 -- start start 2024-01-11T09:27:21.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.201+0000 7fe92b5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe9240ccd70 0x7fe9240cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:21.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.202+0000 7fe92b5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe92415ae70 0x7fe92415b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:21.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.202+0000 7fe92b5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9240cf850 con 0x7fe92415a1b0 2024-01-11T09:27:21.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.202+0000 7fe92b5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe92415ac90 con 0x7fe92415ae70 2024-01-11T09:27:21.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.202+0000 7fe92b5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9240cec40 con 0x7fe9240ccd70 2024-01-11T09:27:21.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.202+0000 7fe929b9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe92415ae70 0x7fe92415b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:21.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.202+0000 7fe92939b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe92415a1b0 0x7fe92415a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57348/0 (socket says 172.21.15.27:57348) 2024-01-11T09:27:21.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.202+0000 7fe92939b700 1 -- 172.21.15.27:0/980641512 learned_addr learned my addr 172.21.15.27:0/980641512 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:21.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.202+0000 7fe928b9a700 1 --2- 172.21.15.27:0/980641512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe9240ccd70 0x7fe9240cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:21.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.202+0000 7fe929b9c700 1 -- 172.21.15.27:0/980641512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe9240ccd70 msgr2=0x7fe9240cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:21.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.202+0000 7fe929b9c700 1 --2- 172.21.15.27:0/980641512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe9240ccd70 0x7fe9240cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:21.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.202+0000 7fe929b9c700 1 -- 172.21.15.27:0/980641512 >> v1:172.21.15.27:6789/0 conn(0x7fe92415a1b0 legacy=0x7fe92415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:21.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.202+0000 7fe929b9c700 1 -- 172.21.15.27:0/980641512 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9240cef20 con 0x7fe92415ae70 2024-01-11T09:27:21.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.202+0000 7fe928b9a700 1 --2- 172.21.15.27:0/980641512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe9240ccd70 0x7fe9240cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:27:21.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.202+0000 7fe91bfff700 1 -- 172.21.15.27:0/980641512 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952903379 0 0) 0x7fe9240cf850 con 0x7fe92415a1b0 2024-01-11T09:27:21.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.203+0000 7fe929b9c700 1 --2- 172.21.15.27:0/980641512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92415ae70 0x7fe92415b2f0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009e60 tx=0x7fe91c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952ea5a87e4380c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:21.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.203+0000 7fe91bfff700 1 -- 172.21.15.27:0/980641512 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c00e070 con 0x7fe92415ae70 2024-01-11T09:27:21.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.203+0000 7fe91bfff700 1 -- 172.21.15.27:0/980641512 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91c00bc30 con 0x7fe92415ae70 2024-01-11T09:27:21.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.203+0000 7fe91bfff700 1 -- 172.21.15.27:0/980641512 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c012430 con 0x7fe92415ae70 2024-01-11T09:27:21.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.203+0000 7fe92b5ff700 1 -- 172.21.15.27:0/980641512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92415ae70 msgr2=0x7fe92415b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:21.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.203+0000 7fe92b5ff700 1 --2- 172.21.15.27:0/980641512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92415ae70 0x7fe92415b2f0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009e60 tx=0x7fe91c009780 comp rx=0 tx=0).stop 2024-01-11T09:27:21.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.204+0000 7fe92b5ff700 1 -- 172.21.15.27:0/980641512 shutdown_connections 2024-01-11T09:27:21.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.204+0000 7fe92b5ff700 1 --2- 172.21.15.27:0/980641512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92415ae70 0x7fe92415b2f0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:21.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.204+0000 7fe92b5ff700 1 --2- 172.21.15.27:0/980641512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe9240ccd70 0x7fe9240cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:21.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.204+0000 7fe92b5ff700 1 -- 172.21.15.27:0/980641512 >> 172.21.15.27:0/980641512 conn(0x7fe92406ed40 msgr2=0x7fe92406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:21.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.204+0000 7fe92b5ff700 1 -- 172.21.15.27:0/980641512 shutdown_connections 2024-01-11T09:27:21.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.204+0000 7fe92b5ff700 1 -- 172.21.15.27:0/980641512 wait complete. 2024-01-11T09:27:21.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.205+0000 7fe92b5ff700 1 Processor -- start 2024-01-11T09:27:21.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.205+0000 7fe92b5ff700 1 -- start start 2024-01-11T09:27:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.206+0000 7fe92b5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe9240ccd70 0x7fe924154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.206+0000 7fe92b5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92415a1b0 0x7fe924152d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.206+0000 7fe92b5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92415ae70 0x7fe924153280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.206+0000 7fe92b5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92415e880 con 0x7fe92415ae70 2024-01-11T09:27:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.206+0000 7fe92b5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92415eb60 con 0x7fe92415a1b0 2024-01-11T09:27:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.206+0000 7fe92b5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe92415ee40 con 0x7fe9240ccd70 2024-01-11T09:27:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.207+0000 7fe929b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92415ae70 0x7fe924153280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.207+0000 7fe929b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92415ae70 0x7fe924153280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37076/0 (socket says 172.21.15.27:37076) 2024-01-11T09:27:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.207+0000 7fe929b9c700 1 -- 172.21.15.27:0/164349766 learned_addr learned my addr 172.21.15.27:0/164349766 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.207+0000 7fe928b9a700 1 --2- 172.21.15.27:0/164349766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92415a1b0 0x7fe924152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.207+0000 7fe92939b700 1 --2- 172.21.15.27:0/164349766 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe9240ccd70 0x7fe924154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.207+0000 7fe929b9c700 1 -- 172.21.15.27:0/164349766 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe9240ccd70 msgr2=0x7fe924154bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.207+0000 7fe929b9c700 1 --2- 172.21.15.27:0/164349766 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe9240ccd70 0x7fe924154bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.207+0000 7fe929b9c700 1 -- 172.21.15.27:0/164349766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92415a1b0 msgr2=0x7fe924152d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.207+0000 7fe929b9c700 1 --2- 172.21.15.27:0/164349766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92415a1b0 0x7fe924152d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.207+0000 7fe929b9c700 1 -- 172.21.15.27:0/164349766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c009a20 con 0x7fe92415ae70 2024-01-11T09:27:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.207+0000 7fe928b9a700 1 --2- 172.21.15.27:0/164349766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92415a1b0 0x7fe924152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:27:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.207+0000 7fe929b9c700 1 --2- 172.21.15.27:0/164349766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92415ae70 0x7fe924153280 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fe924069770 tx=0x7fe91c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.208+0000 7fe9197fa700 1 -- 172.21.15.27:0/164349766 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c012e50 con 0x7fe92415ae70 2024-01-11T09:27:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.208+0000 7fe9197fa700 1 -- 172.21.15.27:0/164349766 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91c01d630 con 0x7fe92415ae70 2024-01-11T09:27:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.208+0000 7fe92b5ff700 1 -- 172.21.15.27:0/164349766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92415f120 con 0x7fe92415ae70 2024-01-11T09:27:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.208+0000 7fe9197fa700 1 -- 172.21.15.27:0/164349766 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c01d9f0 con 0x7fe92415ae70 2024-01-11T09:27:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.208+0000 7fe92b5ff700 1 -- 172.21.15.27:0/164349766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9241550f0 con 0x7fe92415ae70 2024-01-11T09:27:21.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.209+0000 7fe92b5ff700 1 -- 172.21.15.27:0/164349766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9240b3af0 con 0x7fe92415ae70 2024-01-11T09:27:21.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.211+0000 7fe9197fa700 1 -- 172.21.15.27:0/164349766 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe91c024070 con 0x7fe92415ae70 2024-01-11T09:27:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.212+0000 7fe9197fa700 1 --2- 172.21.15.27:0/164349766 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe908066f30 0x7fe9080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.212+0000 7fe9197fa700 1 -- 172.21.15.27:0/164349766 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5661+0+0 (secure 0 0 0) 0x7fe91c086e40 con 0x7fe92415ae70 2024-01-11T09:27:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.212+0000 7fe92939b700 1 --2- 172.21.15.27:0/164349766 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe908066f30 0x7fe9080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.213+0000 7fe92939b700 1 --2- 172.21.15.27:0/164349766 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe908066f30 0x7fe9080693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe9200011e0 tx=0x7fe920009040 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:21.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.216+0000 7fe9197fa700 1 -- 172.21.15.27:0/164349766 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe91c0540c0 con 0x7fe92415ae70 2024-01-11T09:27:21.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.468+0000 7fe92b5ff700 1 -- 172.21.15.27:0/164349766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe9240b3af0 con 0x7fe92415ae70 2024-01-11T09:27:21.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.972+0000 7fe9197fa700 1 -- 172.21.15.27:0/164349766 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 22 and data pool 23 v90) v1 ==== 174+0+0 (secure 0 0 0) 0x7fe9240b3af0 con 0x7fe92415ae70 2024-01-11T09:27:21.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.978+0000 7fe92b5ff700 1 -- 172.21.15.27:0/164349766 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe908066f30 msgr2=0x7fe9080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:21.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.978+0000 7fe92b5ff700 1 --2- 172.21.15.27:0/164349766 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe908066f30 0x7fe9080693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe9200011e0 tx=0x7fe920009040 comp rx=0 tx=0).stop 2024-01-11T09:27:21.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.978+0000 7fe92b5ff700 1 -- 172.21.15.27:0/164349766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92415ae70 msgr2=0x7fe924153280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:21.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.978+0000 7fe92b5ff700 1 --2- 172.21.15.27:0/164349766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92415ae70 0x7fe924153280 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fe924069770 tx=0x7fe91c019000 comp rx=0 tx=0).stop 2024-01-11T09:27:21.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.981+0000 7fe92b5ff700 1 -- 172.21.15.27:0/164349766 shutdown_connections 2024-01-11T09:27:21.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.981+0000 7fe92b5ff700 1 --2- 172.21.15.27:0/164349766 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe908066f30 0x7fe9080693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:21.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.981+0000 7fe92b5ff700 1 --2- 172.21.15.27:0/164349766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe92415ae70 0x7fe924153280 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:21.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.981+0000 7fe92b5ff700 1 --2- 172.21.15.27:0/164349766 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe9240ccd70 0x7fe924154bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:21.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.981+0000 7fe92b5ff700 1 --2- 172.21.15.27:0/164349766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92415a1b0 0x7fe924152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:21.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.981+0000 7fe92b5ff700 1 -- 172.21.15.27:0/164349766 >> 172.21.15.27:0/164349766 conn(0x7fe92406ed40 msgr2=0x7fe9240c5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:21.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.983+0000 7fe92b5ff700 1 -- 172.21.15.27:0/164349766 shutdown_connections 2024-01-11T09:27:21.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:21.983+0000 7fe92b5ff700 1 -- 172.21.15.27:0/164349766 wait complete. 2024-01-11T09:27:21.993 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 22 and data pool 23 2024-01-11T09:27:22.019 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:27:22.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.223+0000 7f0b1c9ff700 1 Processor -- start 2024-01-11T09:27:22.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.224+0000 7f0b1c9ff700 1 -- start start 2024-01-11T09:27:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.225+0000 7f0b1c9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0b180b03e0 0x7f0b180b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.225+0000 7f0b1c9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b180b29c0 0x7f0b18159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.225+0000 7f0b1c9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b180cd730 con 0x7f0b180b07b0 2024-01-11T09:27:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.225+0000 7f0b1c9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b180b2de0 con 0x7f0b180b29c0 2024-01-11T09:27:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.225+0000 7f0b1c9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b180ccb20 con 0x7f0b180b03e0 2024-01-11T09:27:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.225+0000 7f0b16d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b180b29c0 0x7f0b18159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.225+0000 7f0b15d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0b180b07b0 0x7f0b18159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57362/0 (socket says 172.21.15.27:57362) 2024-01-11T09:27:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.225+0000 7f0b15d9b700 1 -- 172.21.15.27:0/2164176259 learned_addr learned my addr 172.21.15.27:0/2164176259 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.225+0000 7f0b1659c700 1 --2- 172.21.15.27:0/2164176259 >> v2:172.21.15.77:3300/0 conn(0x7f0b180b03e0 0x7f0b180b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.226+0000 7f0b1559a700 1 -- 172.21.15.27:0/2164176259 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488551337 0 0) 0x7f0b180cd730 con 0x7f0b180b07b0 2024-01-11T09:27:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.226+0000 7f0b1559a700 1 -- 172.21.15.27:0/2164176259 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b00003570 con 0x7f0b180b07b0 2024-01-11T09:27:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.226+0000 7f0b16d9d700 1 -- 172.21.15.27:0/2164176259 >> v2:172.21.15.77:3300/0 conn(0x7f0b180b03e0 msgr2=0x7f0b180b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.226+0000 7f0b16d9d700 1 --2- 172.21.15.27:0/2164176259 >> v2:172.21.15.77:3300/0 conn(0x7f0b180b03e0 0x7f0b180b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.226+0000 7f0b16d9d700 1 -- 172.21.15.27:0/2164176259 >> v1:172.21.15.27:6789/0 conn(0x7f0b180b07b0 legacy=0x7f0b18159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.226+0000 7f0b16d9d700 1 -- 172.21.15.27:0/2164176259 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b180cce00 con 0x7f0b180b29c0 2024-01-11T09:27:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.226+0000 7f0b16d9d700 1 --2- 172.21.15.27:0/2164176259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b180b29c0 0x7f0b18159910 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f0b18089470 tx=0x7f0b10009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=612c8e64e3c04278 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.226+0000 7f0b1559a700 1 -- 172.21.15.27:0/2164176259 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1000e070 con 0x7f0b180b29c0 2024-01-11T09:27:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.227+0000 7f0b1559a700 1 -- 172.21.15.27:0/2164176259 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1000bab0 con 0x7f0b180b29c0 2024-01-11T09:27:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.227+0000 7f0b1559a700 1 -- 172.21.15.27:0/2164176259 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1000be30 con 0x7f0b180b29c0 2024-01-11T09:27:22.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.227+0000 7f0b1c9ff700 1 -- 172.21.15.27:0/2164176259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b180b29c0 msgr2=0x7f0b18159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:22.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.227+0000 7f0b1c9ff700 1 --2- 172.21.15.27:0/2164176259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b180b29c0 0x7f0b18159910 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f0b18089470 tx=0x7f0b10009780 comp rx=0 tx=0).stop 2024-01-11T09:27:22.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.227+0000 7f0b1c9ff700 1 -- 172.21.15.27:0/2164176259 shutdown_connections 2024-01-11T09:27:22.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.227+0000 7f0b1c9ff700 1 --2- 172.21.15.27:0/2164176259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b180b29c0 0x7f0b18159910 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.227+0000 7f0b1c9ff700 1 --2- 172.21.15.27:0/2164176259 >> v2:172.21.15.77:3300/0 conn(0x7f0b180b03e0 0x7f0b180b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.227+0000 7f0b1c9ff700 1 -- 172.21.15.27:0/2164176259 >> 172.21.15.27:0/2164176259 conn(0x7f0b1806ed40 msgr2=0x7f0b1806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:22.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.227+0000 7f0b1c9ff700 1 -- 172.21.15.27:0/2164176259 shutdown_connections 2024-01-11T09:27:22.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.227+0000 7f0b1c9ff700 1 -- 172.21.15.27:0/2164176259 wait complete. 2024-01-11T09:27:22.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.229+0000 7f0b1c9ff700 1 Processor -- start 2024-01-11T09:27:22.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.229+0000 7f0b1c9ff700 1 -- start start 2024-01-11T09:27:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.230+0000 7f0b1c9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b180b03e0 0x7f0b18157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.230+0000 7f0b1c9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b180b07b0 0x7f0b180cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.230+0000 7f0b1c9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b180b29c0 0x7f0b180cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.230+0000 7f0b1c9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b180a7e60 con 0x7f0b180b07b0 2024-01-11T09:27:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.230+0000 7f0b1c9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b180d0050 con 0x7f0b180b29c0 2024-01-11T09:27:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.230+0000 7f0b1c9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b180d0330 con 0x7f0b180b03e0 2024-01-11T09:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.230+0000 7f0b15d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b180b07b0 0x7f0b180cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.230+0000 7f0b15d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b180b07b0 0x7f0b180cb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37088/0 (socket says 172.21.15.27:37088) 2024-01-11T09:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.230+0000 7f0b15d9b700 1 -- 172.21.15.27:0/288743885 learned_addr learned my addr 172.21.15.27:0/288743885 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.230+0000 7f0b1659c700 1 --2- 172.21.15.27:0/288743885 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b180b03e0 0x7f0b18157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.231+0000 7f0b16d9d700 1 --2- 172.21.15.27:0/288743885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b180b29c0 0x7f0b180cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.231+0000 7f0b15d9b700 1 -- 172.21.15.27:0/288743885 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b180b03e0 msgr2=0x7f0b18157d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.231+0000 7f0b15d9b700 1 --2- 172.21.15.27:0/288743885 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b180b03e0 0x7f0b18157d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.231+0000 7f0b15d9b700 1 -- 172.21.15.27:0/288743885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b180b29c0 msgr2=0x7f0b180cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.231+0000 7f0b15d9b700 1 --2- 172.21.15.27:0/288743885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b180b29c0 0x7f0b180cb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.231+0000 7f0b15d9b700 1 -- 172.21.15.27:0/288743885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b10009a20 con 0x7f0b180b07b0 2024-01-11T09:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.231+0000 7f0b16d9d700 1 --2- 172.21.15.27:0/288743885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b180b29c0 0x7f0b180cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.231+0000 7f0b15d9b700 1 --2- 172.21.15.27:0/288743885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b180b07b0 0x7f0b180cb400 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c0011e0 tx=0x7f0b0c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.231+0000 7f0b067fc700 1 -- 172.21.15.27:0/288743885 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c005b30 con 0x7f0b180b07b0 2024-01-11T09:27:22.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.231+0000 7f0b067fc700 1 -- 172.21.15.27:0/288743885 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0c005c90 con 0x7f0b180b07b0 2024-01-11T09:27:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.231+0000 7f0b067fc700 1 -- 172.21.15.27:0/288743885 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c0072e0 con 0x7f0b180b07b0 2024-01-11T09:27:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.231+0000 7f0b1c9ff700 1 -- 172.21.15.27:0/288743885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b180d0610 con 0x7f0b180b07b0 2024-01-11T09:27:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.231+0000 7f0b1c9ff700 1 -- 172.21.15.27:0/288743885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b180d0e10 con 0x7f0b180b07b0 2024-01-11T09:27:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.232+0000 7f0b1c9ff700 1 -- 172.21.15.27:0/288743885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b180718d0 con 0x7f0b180b07b0 2024-01-11T09:27:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.233+0000 7f0b067fc700 1 -- 172.21.15.27:0/288743885 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0b0c007650 con 0x7f0b180b07b0 2024-01-11T09:27:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.235+0000 7f0b067fc700 1 --2- 172.21.15.27:0/288743885 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0af8066f30 0x7f0af80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.236+0000 7f0b067fc700 1 -- 172.21.15.27:0/288743885 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f0b0c083a30 con 0x7f0b180b07b0 2024-01-11T09:27:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.236+0000 7f0b1659c700 1 --2- 172.21.15.27:0/288743885 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0af8066f30 0x7f0af80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.237+0000 7f0b1659c700 1 --2- 172.21.15.27:0/288743885 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0af8066f30 0x7f0af80693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0b08009d40 tx=0x7f0b08008040 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.240+0000 7f0b067fc700 1 -- 172.21.15.27:0/288743885 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0b0c04b5a0 con 0x7f0b180b07b0 2024-01-11T09:27:22.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.443+0000 7f0b1c9ff700 1 -- 172.21.15.27:0/288743885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b180cd0e0 con 0x7f0b180b07b0 2024-01-11T09:27:22.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.444+0000 7f0b067fc700 1 -- 172.21.15.27:0/288743885 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+15417 (secure 0 0 0) 0x7f0b0c017020 con 0x7f0b180b07b0 2024-01-11T09:27:22.441 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:27:22.441 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":180,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:27:21.470153+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:27:19.009515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-01-11T09:27:20.468964+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:27:22.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.446+0000 7f0b1c9ff700 1 -- 172.21.15.27:0/288743885 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0af8066f30 msgr2=0x7f0af80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:22.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.446+0000 7f0b1c9ff700 1 --2- 172.21.15.27:0/288743885 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0af8066f30 0x7f0af80693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0b08009d40 tx=0x7f0b08008040 comp rx=0 tx=0).stop 2024-01-11T09:27:22.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.446+0000 7f0b1c9ff700 1 -- 172.21.15.27:0/288743885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b180b07b0 msgr2=0x7f0b180cb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:22.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.446+0000 7f0b1c9ff700 1 --2- 172.21.15.27:0/288743885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b180b07b0 0x7f0b180cb400 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c0011e0 tx=0x7f0b0c007df0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.446+0000 7f0b1c9ff700 1 -- 172.21.15.27:0/288743885 shutdown_connections 2024-01-11T09:27:22.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.446+0000 7f0b1c9ff700 1 --2- 172.21.15.27:0/288743885 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f0af8066f30 0x7f0af80693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.446+0000 7f0b1c9ff700 1 --2- 172.21.15.27:0/288743885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b180b29c0 0x7f0b180cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.446+0000 7f0b1c9ff700 1 --2- 172.21.15.27:0/288743885 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b180b03e0 0x7f0b18157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.446+0000 7f0b1c9ff700 1 --2- 172.21.15.27:0/288743885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b180b07b0 0x7f0b180cb400 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.446+0000 7f0b1c9ff700 1 -- 172.21.15.27:0/288743885 >> 172.21.15.27:0/288743885 conn(0x7f0b1806ed40 msgr2=0x7f0b18159ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:22.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.446+0000 7f0b1c9ff700 1 -- 172.21.15.27:0/288743885 shutdown_connections 2024-01-11T09:27:22.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.446+0000 7f0b1c9ff700 1 -- 172.21.15.27:0/288743885 wait complete. 2024-01-11T09:27:22.462 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:27:22.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.666+0000 7f32dc7ff700 1 Processor -- start 2024-01-11T09:27:22.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.667+0000 7f32dc7ff700 1 -- start start 2024-01-11T09:27:22.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.668+0000 7f32dc7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32d40b19d0 0x7f32d4158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:22.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.668+0000 7f32dc7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d4158b60 con 0x7f32d41575e0 2024-01-11T09:27:22.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.668+0000 7f32dc7ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d41599b0 con 0x7f32d406b610 2024-01-11T09:27:22.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.668+0000 7f32dc7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d40cb5e0 con 0x7f32d40b19d0 2024-01-11T09:27:22.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.668+0000 7f32da59b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f32d41575e0 0x7f32d40b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57374/0 (socket says 172.21.15.27:57374) 2024-01-11T09:27:22.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.668+0000 7f32da59b700 1 -- 172.21.15.27:0/3038971302 learned_addr learned my addr 172.21.15.27:0/3038971302 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:22.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.669+0000 7f32d9599700 1 -- 172.21.15.27:0/3038971302 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867129625 0 0) 0x7f32d0001110 con 0x7f32d41575e0 2024-01-11T09:27:22.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.669+0000 7f32d9d9a700 1 --2- 172.21.15.27:0/3038971302 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32d40b19d0 0x7f32d4158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:22.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.669+0000 7f32d9599700 1 -- 172.21.15.27:0/3038971302 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32c0003570 con 0x7f32d41575e0 2024-01-11T09:27:22.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.669+0000 7f32d9599700 1 -- 172.21.15.27:0/3038971302 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3718857490 0 0) 0x7f32c0003570 con 0x7f32d41575e0 2024-01-11T09:27:22.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.669+0000 7f32d9599700 1 -- 172.21.15.27:0/3038971302 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32d40b19d0 msgr2=0x7f32d4158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:22.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.669+0000 7f32d9599700 1 --2- 172.21.15.27:0/3038971302 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32d40b19d0 0x7f32d4158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.669+0000 7f32d9599700 1 -- 172.21.15.27:0/3038971302 >> v1:172.21.15.77:6789/0 conn(0x7f32d406b610 legacy=0x7f32d4159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:22.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.669+0000 7f32d9599700 1 -- 172.21.15.27:0/3038971302 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d40c0f90 con 0x7f32d41575e0 2024-01-11T09:27:22.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.669+0000 7f32d9599700 1 -- 172.21.15.27:0/3038971302 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f32d0001600 con 0x7f32d41575e0 2024-01-11T09:27:22.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.669+0000 7f32d9599700 1 -- 172.21.15.27:0/3038971302 >> v1:172.21.15.27:6789/0 conn(0x7f32d41575e0 legacy=0x7f32d40b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:22.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.669+0000 7f32d9599700 1 --2- 172.21.15.27:0/3038971302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32c0003a90 0x7f32c0003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:22.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.669+0000 7f32d9599700 1 --2- 172.21.15.27:0/3038971302 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32d40b19d0 0x7f32c0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:22.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.669+0000 7f32d9599700 1 --2- 172.21.15.27:0/3038971302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32c0004d50 0x7f32c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:22.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.669+0000 7f32d9599700 1 -- 172.21.15.27:0/3038971302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d40cb5e0 con 0x7f32c0004d50 2024-01-11T09:27:22.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.669+0000 7f32d9599700 1 -- 172.21.15.27:0/3038971302 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c000bb70 con 0x7f32c0003a90 2024-01-11T09:27:22.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.669+0000 7f32d9599700 1 -- 172.21.15.27:0/3038971302 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c000bd30 con 0x7f32d40b19d0 2024-01-11T09:27:22.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.669+0000 7f32da59b700 1 --2- 172.21.15.27:0/3038971302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32c0003a90 0x7f32c0003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:22.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.669+0000 7f32dad9c700 1 --2- 172.21.15.27:0/3038971302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32c0004d50 0x7f32c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:22.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.669+0000 7f32d9d9a700 1 --2- 172.21.15.27:0/3038971302 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32d40b19d0 0x7f32c0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:22.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.670+0000 7f32da59b700 1 -- 172.21.15.27:0/3038971302 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32d40b19d0 msgr2=0x7f32c0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:22.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.670+0000 7f32da59b700 1 --2- 172.21.15.27:0/3038971302 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32d40b19d0 0x7f32c0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.670+0000 7f32da59b700 1 -- 172.21.15.27:0/3038971302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32c0004d50 msgr2=0x7f32c00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:22.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.670+0000 7f32da59b700 1 --2- 172.21.15.27:0/3038971302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32c0004d50 0x7f32c00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.670+0000 7f32da59b700 1 -- 172.21.15.27:0/3038971302 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c0003570 con 0x7f32c0003a90 2024-01-11T09:27:22.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.670+0000 7f32d9d9a700 1 --2- 172.21.15.27:0/3038971302 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32d40b19d0 0x7f32c0004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:27:22.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.670+0000 7f32dad9c700 1 --2- 172.21.15.27:0/3038971302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32c0004d50 0x7f32c00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:27:22.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.670+0000 7f32da59b700 1 --2- 172.21.15.27:0/3038971302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32c0003a90 0x7f32c0003ea0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f32d000c5c0 tx=0x7f32d0007a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bf45ad3ff1f1862 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:22.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.670+0000 7f32d9599700 1 -- 172.21.15.27:0/3038971302 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d000cdc0 con 0x7f32c0003a90 2024-01-11T09:27:22.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.670+0000 7f32d9599700 1 -- 172.21.15.27:0/3038971302 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d0005920 con 0x7f32c0003a90 2024-01-11T09:27:22.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.670+0000 7f32d9599700 1 -- 172.21.15.27:0/3038971302 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d0005ce0 con 0x7f32c0003a90 2024-01-11T09:27:22.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.671+0000 7f32dc7ff700 1 -- 172.21.15.27:0/3038971302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32c0003a90 msgr2=0x7f32c0003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:22.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.671+0000 7f32dc7ff700 1 --2- 172.21.15.27:0/3038971302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32c0003a90 0x7f32c0003ea0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f32d000c5c0 tx=0x7f32d0007a50 comp rx=0 tx=0).stop 2024-01-11T09:27:22.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.671+0000 7f32dad9c700 1 -- 172.21.15.27:0/3038971302 reap_dead start 2024-01-11T09:27:22.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.671+0000 7f32dc7ff700 1 -- 172.21.15.27:0/3038971302 shutdown_connections 2024-01-11T09:27:22.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.671+0000 7f32dc7ff700 1 -- 172.21.15.27:0/3038971302 >> 172.21.15.27:0/3038971302 conn(0x7f32d40680e0 msgr2=0x7f32d4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:22.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.671+0000 7f32dc7ff700 1 -- 172.21.15.27:0/3038971302 shutdown_connections 2024-01-11T09:27:22.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.671+0000 7f32dc7ff700 1 -- 172.21.15.27:0/3038971302 wait complete. 2024-01-11T09:27:22.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.673+0000 7f32dc7ff700 1 Processor -- start 2024-01-11T09:27:22.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.673+0000 7f32dc7ff700 1 -- start start 2024-01-11T09:27:22.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.674+0000 7f32dc7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32c0003a90 0x7f32d4141d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:22.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.674+0000 7f32dc7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40c1260 0x7f32d4151100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:22.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.674+0000 7f32dc7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32d415c4e0 0x7f32d415c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:22.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.674+0000 7f32dc7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d415cec0 con 0x7f32d40c1260 2024-01-11T09:27:22.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.674+0000 7f32dc7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d415be40 con 0x7f32d415c4e0 2024-01-11T09:27:22.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.674+0000 7f32dc7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d41422b0 con 0x7f32c0003a90 2024-01-11T09:27:22.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.674+0000 7f32d9d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40c1260 0x7f32d4151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:22.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.674+0000 7f32da59b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32c0003a90 0x7f32d4141d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:22.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.674+0000 7f32d9d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40c1260 0x7f32d4151100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37098/0 (socket says 172.21.15.27:37098) 2024-01-11T09:27:22.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.674+0000 7f32d9d9a700 1 -- 172.21.15.27:0/3124781547 learned_addr learned my addr 172.21.15.27:0/3124781547 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:22.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.674+0000 7f32dad9c700 1 --2- 172.21.15.27:0/3124781547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32d415c4e0 0x7f32d415c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:22.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.675+0000 7f32d9d9a700 1 -- 172.21.15.27:0/3124781547 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32c0003a90 msgr2=0x7f32d4141d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:22.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.675+0000 7f32d9d9a700 1 --2- 172.21.15.27:0/3124781547 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32c0003a90 0x7f32d4141d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.675+0000 7f32d9d9a700 1 -- 172.21.15.27:0/3124781547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32d415c4e0 msgr2=0x7f32d415c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:22.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.675+0000 7f32d9d9a700 1 --2- 172.21.15.27:0/3124781547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32d415c4e0 0x7f32d415c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.675+0000 7f32d9d9a700 1 -- 172.21.15.27:0/3124781547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d415b230 con 0x7f32d40c1260 2024-01-11T09:27:22.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.675+0000 7f32d9d9a700 1 --2- 172.21.15.27:0/3124781547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40c1260 0x7f32d4151100 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f32c4011d70 tx=0x7f32c40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:22.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.675+0000 7f32ca7fc700 1 -- 172.21.15.27:0/3124781547 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c4011e70 con 0x7f32d40c1260 2024-01-11T09:27:22.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.675+0000 7f32ca7fc700 1 -- 172.21.15.27:0/3124781547 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c400fc70 con 0x7f32d40c1260 2024-01-11T09:27:22.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.675+0000 7f32ca7fc700 1 -- 172.21.15.27:0/3124781547 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c401d480 con 0x7f32d40c1260 2024-01-11T09:27:22.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.675+0000 7f32dc7ff700 1 -- 172.21.15.27:0/3124781547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d415b510 con 0x7f32d40c1260 2024-01-11T09:27:22.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.675+0000 7f32dc7ff700 1 -- 172.21.15.27:0/3124781547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d415bad0 con 0x7f32d40c1260 2024-01-11T09:27:22.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.677+0000 7f32dc7ff700 1 -- 172.21.15.27:0/3124781547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d40718d0 con 0x7f32d40c1260 2024-01-11T09:27:22.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.678+0000 7f32ca7fc700 1 -- 172.21.15.27:0/3124781547 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f32c401d8b0 con 0x7f32d40c1260 2024-01-11T09:27:22.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.679+0000 7f32ca7fc700 1 --2- 172.21.15.27:0/3124781547 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f32b8066f30 0x7f32b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:22.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.679+0000 7f32ca7fc700 1 -- 172.21.15.27:0/3124781547 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f32c4018070 con 0x7f32d40c1260 2024-01-11T09:27:22.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.679+0000 7f32da59b700 1 --2- 172.21.15.27:0/3124781547 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f32b8066f30 0x7f32b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:22.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.680+0000 7f32da59b700 1 --2- 172.21.15.27:0/3124781547 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f32b8066f30 0x7f32b80693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f32d0007e70 tx=0x7f32d0015000 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:22.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.683+0000 7f32ca7fc700 1 -- 172.21.15.27:0/3124781547 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f32c4052620 con 0x7f32d40c1260 2024-01-11T09:27:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.888+0000 7f32dc7ff700 1 -- 172.21.15.27:0/3124781547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32d415bf80 con 0x7f32d40c1260 2024-01-11T09:27:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.889+0000 7f32ca7fc700 1 -- 172.21.15.27:0/3124781547 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+15417 (secure 0 0 0) 0x7f32c4055c40 con 0x7f32d40c1260 2024-01-11T09:27:22.886 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:27:22.886 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":180,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:27:21.470153+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:27:19.009515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-01-11T09:27:20.468964+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:27:22.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.891+0000 7f32dc7ff700 1 -- 172.21.15.27:0/3124781547 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f32b8066f30 msgr2=0x7f32b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:22.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.891+0000 7f32dc7ff700 1 --2- 172.21.15.27:0/3124781547 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f32b8066f30 0x7f32b80693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f32d0007e70 tx=0x7f32d0015000 comp rx=0 tx=0).stop 2024-01-11T09:27:22.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.891+0000 7f32dc7ff700 1 -- 172.21.15.27:0/3124781547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40c1260 msgr2=0x7f32d4151100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.891+0000 7f32dc7ff700 1 --2- 172.21.15.27:0/3124781547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40c1260 0x7f32d4151100 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f32c4011d70 tx=0x7f32c40091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.891+0000 7f32dc7ff700 1 -- 172.21.15.27:0/3124781547 shutdown_connections 2024-01-11T09:27:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.891+0000 7f32dc7ff700 1 --2- 172.21.15.27:0/3124781547 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f32b8066f30 0x7f32b80693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.891+0000 7f32dc7ff700 1 --2- 172.21.15.27:0/3124781547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32d415c4e0 0x7f32d415c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.891+0000 7f32dc7ff700 1 --2- 172.21.15.27:0/3124781547 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f32c0003a90 0x7f32d4141d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.891+0000 7f32dc7ff700 1 --2- 172.21.15.27:0/3124781547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40c1260 0x7f32d4151100 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.891+0000 7f32dc7ff700 1 -- 172.21.15.27:0/3124781547 >> 172.21.15.27:0/3124781547 conn(0x7f32d40680e0 msgr2=0x7f32d41540a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.891+0000 7f32dc7ff700 1 -- 172.21.15.27:0/3124781547 shutdown_connections 2024-01-11T09:27:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:22.891+0000 7f32dc7ff700 1 -- 172.21.15.27:0/3124781547 wait complete. 2024-01-11T09:27:22.908 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-11T09:27:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.116+0000 7f6a4759d700 1 Processor -- start 2024-01-11T09:27:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.116+0000 7f6a4759d700 1 -- start start 2024-01-11T09:27:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.118+0000 7f6a4759d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6a400b0740 0x7f6a400b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.118+0000 7f6a4759d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6a401598f0 0x7f6a400b2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.118+0000 7f6a4759d700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a400cb340 con 0x7f6a40159520 2024-01-11T09:27:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.118+0000 7f6a4759d700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a400cb620 con 0x7f6a401598f0 2024-01-11T09:27:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.118+0000 7f6a4759d700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a400cb900 con 0x7f6a400b0740 2024-01-11T09:27:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.118+0000 7f6a4659b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6a40159520 0x7f6a400b0030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57376/0 (socket says 172.21.15.27:57376) 2024-01-11T09:27:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a4659b700 1 -- 172.21.15.27:0/1294947368 learned_addr learned my addr 172.21.15.27:0/1294947368 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a45d9a700 1 --2- 172.21.15.27:0/1294947368 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6a400b0740 0x7f6a400b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a46d9c700 1 --2- 172.21.15.27:0/1294947368 >> v2:172.21.15.77:3300/0 conn(0x7f6a401598f0 0x7f6a400b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a45599700 1 -- 172.21.15.27:0/1294947368 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934252615 0 0) 0x7f6a400cb340 con 0x7f6a40159520 2024-01-11T09:27:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a45599700 1 -- 172.21.15.27:0/1294947368 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a2c003570 con 0x7f6a40159520 2024-01-11T09:27:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a45599700 1 -- 172.21.15.27:0/1294947368 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2784056015 0 0) 0x7f6a2c003570 con 0x7f6a40159520 2024-01-11T09:27:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a45599700 1 -- 172.21.15.27:0/1294947368 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6a400b0740 msgr2=0x7f6a400b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a45599700 1 --2- 172.21.15.27:0/1294947368 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6a400b0740 0x7f6a400b2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a45599700 1 -- 172.21.15.27:0/1294947368 >> v2:172.21.15.77:3300/0 conn(0x7f6a401598f0 msgr2=0x7f6a400b2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a45599700 1 --2- 172.21.15.27:0/1294947368 >> v2:172.21.15.77:3300/0 conn(0x7f6a401598f0 0x7f6a400b2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a45599700 1 -- 172.21.15.27:0/1294947368 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a400cbbe0 con 0x7f6a40159520 2024-01-11T09:27:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a45599700 1 -- 172.21.15.27:0/1294947368 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f6a3c001470 con 0x7f6a40159520 2024-01-11T09:27:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a45d9a700 1 --2- 172.21.15.27:0/1294947368 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6a400b0740 0x7f6a400b2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:27:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a45599700 1 -- 172.21.15.27:0/1294947368 >> v1:172.21.15.27:6789/0 conn(0x7f6a40159520 legacy=0x7f6a400b0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a45599700 1 --2- 172.21.15.27:0/1294947368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a2c003a60 0x7f6a2c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a45599700 1 --2- 172.21.15.27:0/1294947368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a2c004810 0x7f6a2c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a45599700 1 --2- 172.21.15.27:0/1294947368 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6a400b0740 0x7f6a2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a45599700 1 -- 172.21.15.27:0/1294947368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a400cb620 con 0x7f6a2c003a60 2024-01-11T09:27:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a45599700 1 -- 172.21.15.27:0/1294947368 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a400cb900 con 0x7f6a2c004810 2024-01-11T09:27:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.119+0000 7f6a45599700 1 -- 172.21.15.27:0/1294947368 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c00ba60 con 0x7f6a400b0740 2024-01-11T09:27:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.120+0000 7f6a4659b700 1 --2- 172.21.15.27:0/1294947368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a2c003a60 0x7f6a2c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.120+0000 7f6a45d9a700 1 --2- 172.21.15.27:0/1294947368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a2c004810 0x7f6a2c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.120+0000 7f6a46d9c700 1 --2- 172.21.15.27:0/1294947368 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6a400b0740 0x7f6a2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.120+0000 7f6a4659b700 1 -- 172.21.15.27:0/1294947368 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6a400b0740 msgr2=0x7f6a2c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.120+0000 7f6a4659b700 1 --2- 172.21.15.27:0/1294947368 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6a400b0740 0x7f6a2c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.120+0000 7f6a4659b700 1 -- 172.21.15.27:0/1294947368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a2c004810 msgr2=0x7f6a2c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.120+0000 7f6a4659b700 1 --2- 172.21.15.27:0/1294947368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a2c004810 0x7f6a2c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.120+0000 7f6a4659b700 1 -- 172.21.15.27:0/1294947368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a30009420 con 0x7f6a2c003a60 2024-01-11T09:27:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.120+0000 7f6a46d9c700 1 --2- 172.21.15.27:0/1294947368 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6a400b0740 0x7f6a2c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:27:23.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.120+0000 7f6a4659b700 1 --2- 172.21.15.27:0/1294947368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a2c003a60 0x7f6a2c003e70 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c00c3c0 tx=0x7f6a3c001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bc74191fe270859 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:23.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.121+0000 7f6a45599700 1 -- 172.21.15.27:0/1294947368 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c00cc90 con 0x7f6a2c003a60 2024-01-11T09:27:23.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.121+0000 7f6a45599700 1 -- 172.21.15.27:0/1294947368 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a3c00ce30 con 0x7f6a2c003a60 2024-01-11T09:27:23.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.121+0000 7f6a45599700 1 -- 172.21.15.27:0/1294947368 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c005b70 con 0x7f6a2c003a60 2024-01-11T09:27:23.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.121+0000 7f6a4759d700 1 -- 172.21.15.27:0/1294947368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a2c003a60 msgr2=0x7f6a2c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:23.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.121+0000 7f6a4759d700 1 --2- 172.21.15.27:0/1294947368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a2c003a60 0x7f6a2c003e70 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c00c3c0 tx=0x7f6a3c001e90 comp rx=0 tx=0).stop 2024-01-11T09:27:23.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.121+0000 7f6a46d9c700 1 -- 172.21.15.27:0/1294947368 reap_dead start 2024-01-11T09:27:23.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.121+0000 7f6a4759d700 1 -- 172.21.15.27:0/1294947368 shutdown_connections 2024-01-11T09:27:23.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.121+0000 7f6a4759d700 1 -- 172.21.15.27:0/1294947368 >> 172.21.15.27:0/1294947368 conn(0x7f6a4006ed40 msgr2=0x7f6a4006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:23.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.121+0000 7f6a4759d700 1 -- 172.21.15.27:0/1294947368 shutdown_connections 2024-01-11T09:27:23.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.121+0000 7f6a4759d700 1 -- 172.21.15.27:0/1294947368 wait complete. 2024-01-11T09:27:23.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.123+0000 7f6a4759d700 1 Processor -- start 2024-01-11T09:27:23.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.123+0000 7f6a4759d700 1 -- start start 2024-01-11T09:27:23.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.124+0000 7f6a4759d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a2c003a60 0x7f6a400af8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:23.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.124+0000 7f6a4759d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4015a700 0x7f6a400c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:23.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.124+0000 7f6a4759d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6a401556f0 0x7f6a40155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:23.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.124+0000 7f6a4759d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a400c74f0 con 0x7f6a4015a700 2024-01-11T09:27:23.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.124+0000 7f6a4759d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a401560b0 con 0x7f6a2c003a60 2024-01-11T09:27:23.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.124+0000 7f6a4759d700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4015ac70 con 0x7f6a401556f0 2024-01-11T09:27:23.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.124+0000 7f6a45d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4015a700 0x7f6a400c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:23.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.124+0000 7f6a45d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4015a700 0x7f6a400c98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37124/0 (socket says 172.21.15.27:37124) 2024-01-11T09:27:23.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.124+0000 7f6a45d9a700 1 -- 172.21.15.27:0/4153261312 learned_addr learned my addr 172.21.15.27:0/4153261312 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:23.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.124+0000 7f6a4659b700 1 --2- 172.21.15.27:0/4153261312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a2c003a60 0x7f6a400af8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:23.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.124+0000 7f6a46d9c700 1 --2- 172.21.15.27:0/4153261312 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6a401556f0 0x7f6a40155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:23.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.125+0000 7f6a45d9a700 1 -- 172.21.15.27:0/4153261312 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6a401556f0 msgr2=0x7f6a40155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:23.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.125+0000 7f6a45d9a700 1 --2- 172.21.15.27:0/4153261312 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6a401556f0 0x7f6a40155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:23.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.125+0000 7f6a45d9a700 1 -- 172.21.15.27:0/4153261312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a2c003a60 msgr2=0x7f6a400af8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:23.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.125+0000 7f6a45d9a700 1 --2- 172.21.15.27:0/4153261312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a2c003a60 0x7f6a400af8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:23.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.125+0000 7f6a45d9a700 1 -- 172.21.15.27:0/4153261312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4015af50 con 0x7f6a4015a700 2024-01-11T09:27:23.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.125+0000 7f6a45d9a700 1 --2- 172.21.15.27:0/4153261312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4015a700 0x7f6a400c98d0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f6a30011d70 tx=0x7f6a300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:23.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.125+0000 7f6a367fc700 1 -- 172.21.15.27:0/4153261312 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30011e70 con 0x7f6a4015a700 2024-01-11T09:27:23.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.125+0000 7f6a367fc700 1 -- 172.21.15.27:0/4153261312 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a3000fba0 con 0x7f6a4015a700 2024-01-11T09:27:23.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.125+0000 7f6a367fc700 1 -- 172.21.15.27:0/4153261312 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30014440 con 0x7f6a4015a700 2024-01-11T09:27:23.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.125+0000 7f6a4759d700 1 -- 172.21.15.27:0/4153261312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a4015b230 con 0x7f6a4015a700 2024-01-11T09:27:23.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.126+0000 7f6a4759d700 1 -- 172.21.15.27:0/4153261312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a400afe00 con 0x7f6a4015a700 2024-01-11T09:27:23.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.127+0000 7f6a4759d700 1 -- 172.21.15.27:0/4153261312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a4004bb80 con 0x7f6a4015a700 2024-01-11T09:27:23.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.128+0000 7f6a367fc700 1 -- 172.21.15.27:0/4153261312 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6a30014870 con 0x7f6a4015a700 2024-01-11T09:27:23.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.129+0000 7f6a367fc700 1 --2- 172.21.15.27:0/4153261312 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6a24066f30 0x7f6a240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:23.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.129+0000 7f6a367fc700 1 -- 172.21.15.27:0/4153261312 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f6a30018030 con 0x7f6a4015a700 2024-01-11T09:27:23.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.129+0000 7f6a4659b700 1 --2- 172.21.15.27:0/4153261312 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6a24066f30 0x7f6a240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:23.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.130+0000 7f6a4659b700 1 --2- 172.21.15.27:0/4153261312 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6a24066f30 0x7f6a240693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c00c3c0 tx=0x7f6a3c015000 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:23.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.133+0000 7f6a367fc700 1 -- 172.21.15.27:0/4153261312 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6a300526c0 con 0x7f6a4015a700 2024-01-11T09:27:23.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.382+0000 7f6a4759d700 1 -- 172.21.15.27:0/4153261312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6a400cac10 con 0x7f6a4015a700 2024-01-11T09:27:23.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.994+0000 7f6a367fc700 1 -- 172.21.15.27:0/4153261312 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v93) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6a30055ce0 con 0x7f6a4015a700 2024-01-11T09:27:23.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.996+0000 7f6a4759d700 1 -- 172.21.15.27:0/4153261312 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6a24066f30 msgr2=0x7f6a240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:23.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.996+0000 7f6a4759d700 1 --2- 172.21.15.27:0/4153261312 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6a24066f30 0x7f6a240693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c00c3c0 tx=0x7f6a3c015000 comp rx=0 tx=0).stop 2024-01-11T09:27:23.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.996+0000 7f6a4759d700 1 -- 172.21.15.27:0/4153261312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4015a700 msgr2=0x7f6a400c98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:23.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.996+0000 7f6a4759d700 1 --2- 172.21.15.27:0/4153261312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4015a700 0x7f6a400c98d0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f6a30011d70 tx=0x7f6a300091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:23.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.997+0000 7f6a4759d700 1 -- 172.21.15.27:0/4153261312 shutdown_connections 2024-01-11T09:27:23.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.997+0000 7f6a4759d700 1 --2- 172.21.15.27:0/4153261312 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f6a24066f30 0x7f6a240693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:23.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.997+0000 7f6a4759d700 1 --2- 172.21.15.27:0/4153261312 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6a401556f0 0x7f6a40155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:23.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.997+0000 7f6a4759d700 1 --2- 172.21.15.27:0/4153261312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a2c003a60 0x7f6a400af8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:23.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.997+0000 7f6a4759d700 1 --2- 172.21.15.27:0/4153261312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a4015a700 0x7f6a400c98d0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:23.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.997+0000 7f6a4759d700 1 -- 172.21.15.27:0/4153261312 >> 172.21.15.27:0/4153261312 conn(0x7f6a4006ed40 msgr2=0x7f6a400ccbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:23.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.997+0000 7f6a4759d700 1 -- 172.21.15.27:0/4153261312 shutdown_connections 2024-01-11T09:27:23.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:23.997+0000 7f6a4759d700 1 -- 172.21.15.27:0/4153261312 wait complete. 2024-01-11T09:27:24.013 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-11T09:27:24.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.221+0000 7fefa8dff700 1 Processor -- start 2024-01-11T09:27:24.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.222+0000 7fefa8dff700 1 -- start start 2024-01-11T09:27:24.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.223+0000 7fefa8dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefa4157410 0x7fefa40ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:24.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.223+0000 7fefa8dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa40699d0 con 0x7fefa4069540 2024-01-11T09:27:24.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.223+0000 7fefa8dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa4069cb0 con 0x7fefa41577e0 2024-01-11T09:27:24.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.223+0000 7fefa8dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa4069f90 con 0x7fefa4157410 2024-01-11T09:27:24.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.223+0000 7fefa2d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fefa4069540 0x7fefa40c6ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53540/0 (socket says 172.21.15.27:53540) 2024-01-11T09:27:24.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.223+0000 7fefa2d9d700 1 -- 172.21.15.27:0/3576502164 learned_addr learned my addr 172.21.15.27:0/3576502164 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:24.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.224+0000 7fefa259c700 1 --2- 172.21.15.27:0/3576502164 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefa4157410 0x7fefa40ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:24.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.224+0000 7fefa159a700 1 -- 172.21.15.27:0/3576502164 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295028486 0 0) 0x7fefa40699d0 con 0x7fefa4069540 2024-01-11T09:27:24.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.224+0000 7fefa159a700 1 -- 172.21.15.27:0/3576502164 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef8c003570 con 0x7fefa4069540 2024-01-11T09:27:24.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.224+0000 7fefa159a700 1 -- 172.21.15.27:0/3576502164 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130327885 0 0) 0x7fefa4069cb0 con 0x7fefa41577e0 2024-01-11T09:27:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.224+0000 7fefa159a700 1 -- 172.21.15.27:0/3576502164 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefa40699d0 con 0x7fefa41577e0 2024-01-11T09:27:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.224+0000 7fefa159a700 1 -- 172.21.15.27:0/3576502164 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1610844727 0 0) 0x7fef8c003570 con 0x7fefa4069540 2024-01-11T09:27:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.224+0000 7fefa159a700 1 -- 172.21.15.27:0/3576502164 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefa4157410 msgr2=0x7fefa40ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.224+0000 7fefa159a700 1 --2- 172.21.15.27:0/3576502164 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefa4157410 0x7fefa40ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.224+0000 7fefa159a700 1 -- 172.21.15.27:0/3576502164 >> v1:172.21.15.77:6789/0 conn(0x7fefa41577e0 legacy=0x7fefa40ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.224+0000 7fefa159a700 1 -- 172.21.15.27:0/3576502164 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa406a270 con 0x7fefa4069540 2024-01-11T09:27:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.224+0000 7fefa159a700 1 -- 172.21.15.27:0/3576502164 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fef9c0017f0 con 0x7fefa4069540 2024-01-11T09:27:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.224+0000 7fefa159a700 1 -- 172.21.15.27:0/3576502164 >> v1:172.21.15.27:6789/0 conn(0x7fefa4069540 legacy=0x7fefa40c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.224+0000 7fefa159a700 1 --2- 172.21.15.27:0/3576502164 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefa4157410 0x7fef8c003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.224+0000 7fefa159a700 1 --2- 172.21.15.27:0/3576502164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef8c0041d0 0x7fef8c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.224+0000 7fefa159a700 1 --2- 172.21.15.27:0/3576502164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef8c004d50 0x7fef8c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.224+0000 7fefa159a700 1 -- 172.21.15.27:0/3576502164 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa4069f90 con 0x7fef8c0041d0 2024-01-11T09:27:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.224+0000 7fefa159a700 1 -- 172.21.15.27:0/3576502164 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8c00bbf0 con 0x7fef8c004d50 2024-01-11T09:27:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.224+0000 7fefa159a700 1 -- 172.21.15.27:0/3576502164 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef8c00bdb0 con 0x7fefa4157410 2024-01-11T09:27:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.225+0000 7fefa1d9b700 1 --2- 172.21.15.27:0/3576502164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef8c0041d0 0x7fef8c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.225+0000 7fefa259c700 1 --2- 172.21.15.27:0/3576502164 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefa4157410 0x7fef8c003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.225+0000 7fefa2d9d700 1 --2- 172.21.15.27:0/3576502164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef8c004d50 0x7fef8c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.225+0000 7fefa1d9b700 1 -- 172.21.15.27:0/3576502164 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefa4157410 msgr2=0x7fef8c003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.225+0000 7fefa1d9b700 1 --2- 172.21.15.27:0/3576502164 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fefa4157410 0x7fef8c003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.225+0000 7fefa1d9b700 1 -- 172.21.15.27:0/3576502164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef8c004d50 msgr2=0x7fef8c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.225+0000 7fefa1d9b700 1 --2- 172.21.15.27:0/3576502164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef8c004d50 0x7fef8c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.225+0000 7fefa1d9b700 1 -- 172.21.15.27:0/3576502164 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8c003570 con 0x7fef8c0041d0 2024-01-11T09:27:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.225+0000 7fefa1d9b700 1 --2- 172.21.15.27:0/3576502164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef8c0041d0 0x7fef8c004810 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fef980019e0 tx=0x7fef98007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21929e413b5da786 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.225+0000 7fefa159a700 1 -- 172.21.15.27:0/3576502164 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9800c5d0 con 0x7fef8c0041d0 2024-01-11T09:27:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.226+0000 7fefa159a700 1 -- 172.21.15.27:0/3576502164 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9800c730 con 0x7fef8c0041d0 2024-01-11T09:27:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.226+0000 7fefa159a700 1 -- 172.21.15.27:0/3576502164 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9800cad0 con 0x7fef8c0041d0 2024-01-11T09:27:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.226+0000 7fefa8dff700 1 -- 172.21.15.27:0/3576502164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef8c0041d0 msgr2=0x7fef8c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.226+0000 7fefa8dff700 1 --2- 172.21.15.27:0/3576502164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef8c0041d0 0x7fef8c004810 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fef980019e0 tx=0x7fef98007d30 comp rx=0 tx=0).stop 2024-01-11T09:27:24.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.226+0000 7fefa2d9d700 1 -- 172.21.15.27:0/3576502164 reap_dead start 2024-01-11T09:27:24.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.226+0000 7fefa8dff700 1 -- 172.21.15.27:0/3576502164 shutdown_connections 2024-01-11T09:27:24.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.226+0000 7fefa8dff700 1 -- 172.21.15.27:0/3576502164 >> 172.21.15.27:0/3576502164 conn(0x7fefa40680a0 msgr2=0x7fefa40cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:24.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.226+0000 7fefa8dff700 1 -- 172.21.15.27:0/3576502164 shutdown_connections 2024-01-11T09:27:24.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.226+0000 7fefa8dff700 1 -- 172.21.15.27:0/3576502164 wait complete. 2024-01-11T09:27:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.228+0000 7fefa8dff700 1 Processor -- start 2024-01-11T09:27:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.228+0000 7fefa8dff700 1 -- start start 2024-01-11T09:27:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.229+0000 7fefa8dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef8c0041d0 0x7fefa40c5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.229+0000 7fefa8dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefa40c58c0 0x7fefa40ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.229+0000 7fefa8dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefa40ca8f0 0x7fefa4158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.229+0000 7fefa8dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa40c4ef0 con 0x7fefa40c58c0 2024-01-11T09:27:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.229+0000 7fefa8dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa40c42e0 con 0x7fefa40ca8f0 2024-01-11T09:27:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.229+0000 7fefa8dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa40c45c0 con 0x7fef8c0041d0 2024-01-11T09:27:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.229+0000 7fefa1d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefa40c58c0 0x7fefa40ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.229+0000 7fefa1d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefa40c58c0 0x7fefa40ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37614/0 (socket says 172.21.15.27:37614) 2024-01-11T09:27:24.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.229+0000 7fefa1d9b700 1 -- 172.21.15.27:0/4065593927 learned_addr learned my addr 172.21.15.27:0/4065593927 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.229+0000 7fefa259c700 1 --2- 172.21.15.27:0/4065593927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef8c0041d0 0x7fefa40c5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.229+0000 7fefa2d9d700 1 --2- 172.21.15.27:0/4065593927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefa40ca8f0 0x7fefa4158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.229+0000 7fefa1d9b700 1 -- 172.21.15.27:0/4065593927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef8c0041d0 msgr2=0x7fefa40c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.229+0000 7fefa1d9b700 1 --2- 172.21.15.27:0/4065593927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef8c0041d0 0x7fefa40c5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.229+0000 7fefa1d9b700 1 -- 172.21.15.27:0/4065593927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefa40ca8f0 msgr2=0x7fefa4158920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.229+0000 7fefa1d9b700 1 --2- 172.21.15.27:0/4065593927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefa40ca8f0 0x7fefa4158920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.230+0000 7fefa1d9b700 1 -- 172.21.15.27:0/4065593927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa40c48a0 con 0x7fefa40c58c0 2024-01-11T09:27:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.230+0000 7fefa1d9b700 1 --2- 172.21.15.27:0/4065593927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefa40c58c0 0x7fefa40ca3b0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fef9800c3c0 tx=0x7fef98015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.230+0000 7fef927fc700 1 -- 172.21.15.27:0/4065593927 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9800c5d0 con 0x7fefa40c58c0 2024-01-11T09:27:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.230+0000 7fef927fc700 1 -- 172.21.15.27:0/4065593927 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef98001360 con 0x7fefa40c58c0 2024-01-11T09:27:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.230+0000 7fefa8dff700 1 -- 172.21.15.27:0/4065593927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa40c4b80 con 0x7fefa40c58c0 2024-01-11T09:27:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.230+0000 7fef927fc700 1 -- 172.21.15.27:0/4065593927 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9800ce80 con 0x7fefa40c58c0 2024-01-11T09:27:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.230+0000 7fefa8dff700 1 -- 172.21.15.27:0/4065593927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa4158e60 con 0x7fefa40c58c0 2024-01-11T09:27:24.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.231+0000 7fefa8dff700 1 -- 172.21.15.27:0/4065593927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa40cacc0 con 0x7fefa40c58c0 2024-01-11T09:27:24.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.232+0000 7fef927fc700 1 -- 172.21.15.27:0/4065593927 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fef98019070 con 0x7fefa40c58c0 2024-01-11T09:27:24.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.233+0000 7fef927fc700 1 --2- 172.21.15.27:0/4065593927 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fef84066f30 0x7fef840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:24.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.233+0000 7fef927fc700 1 -- 172.21.15.27:0/4065593927 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fef980859b0 con 0x7fefa40c58c0 2024-01-11T09:27:24.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.234+0000 7fefa259c700 1 --2- 172.21.15.27:0/4065593927 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fef84066f30 0x7fef840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:24.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.235+0000 7fefa259c700 1 --2- 172.21.15.27:0/4065593927 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fef84066f30 0x7fef840693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fef94006ea0 tx=0x7fef94009120 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:24.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.237+0000 7fef927fc700 1 -- 172.21.15.27:0/4065593927 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fef98052b40 con 0x7fefa40c58c0 2024-01-11T09:27:24.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:24.487+0000 7fefa8dff700 1 -- 172.21.15.27:0/4065593927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fefa404c3c0 con 0x7fefa40c58c0 2024-01-11T09:27:24.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.002+0000 7fef927fc700 1 -- 172.21.15.27:0/4065593927 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v94) v1 ==== 122+0+0 (secure 0 0 0) 0x7fef980230c0 con 0x7fefa40c58c0 2024-01-11T09:27:25.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.005+0000 7fefa8dff700 1 -- 172.21.15.27:0/4065593927 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fef84066f30 msgr2=0x7fef840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:25.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.005+0000 7fefa8dff700 1 --2- 172.21.15.27:0/4065593927 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fef84066f30 0x7fef840693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fef94006ea0 tx=0x7fef94009120 comp rx=0 tx=0).stop 2024-01-11T09:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.005+0000 7fefa8dff700 1 -- 172.21.15.27:0/4065593927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefa40c58c0 msgr2=0x7fefa40ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.005+0000 7fefa8dff700 1 --2- 172.21.15.27:0/4065593927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefa40c58c0 0x7fefa40ca3b0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fef9800c3c0 tx=0x7fef98015000 comp rx=0 tx=0).stop 2024-01-11T09:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.005+0000 7fefa8dff700 1 -- 172.21.15.27:0/4065593927 shutdown_connections 2024-01-11T09:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.005+0000 7fefa8dff700 1 --2- 172.21.15.27:0/4065593927 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fef84066f30 0x7fef840693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.005+0000 7fefa8dff700 1 --2- 172.21.15.27:0/4065593927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefa40ca8f0 0x7fefa4158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.005+0000 7fefa8dff700 1 --2- 172.21.15.27:0/4065593927 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fef8c0041d0 0x7fefa40c5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.005+0000 7fefa8dff700 1 --2- 172.21.15.27:0/4065593927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefa40c58c0 0x7fefa40ca3b0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.005+0000 7fefa8dff700 1 -- 172.21.15.27:0/4065593927 >> 172.21.15.27:0/4065593927 conn(0x7fefa40680a0 msgr2=0x7fefa40b11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.005+0000 7fefa8dff700 1 -- 172.21.15.27:0/4065593927 shutdown_connections 2024-01-11T09:27:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.005+0000 7fefa8dff700 1 -- 172.21.15.27:0/4065593927 wait complete. 2024-01-11T09:27:25.020 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:27:25.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.228+0000 7f830d9ff700 1 Processor -- start 2024-01-11T09:27:25.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.228+0000 7f830d9ff700 1 -- start start 2024-01-11T09:27:25.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.230+0000 7f830d9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f83080ccf40 0x7f83080cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:25.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.230+0000 7f830d9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83080cc510 con 0x7f830815cef0 2024-01-11T09:27:25.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.230+0000 7f830d9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83080c6350 con 0x7f8308161520 2024-01-11T09:27:25.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.230+0000 7f830d9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83080cd920 con 0x7f83080ccf40 2024-01-11T09:27:25.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.230+0000 7f8306ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f830815cef0 0x7f83080ce430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53544/0 (socket says 172.21.15.27:53544) 2024-01-11T09:27:25.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.230+0000 7f8306ffd700 1 -- 172.21.15.27:0/4087099525 learned_addr learned my addr 172.21.15.27:0/4087099525 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:25.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.230+0000 7f83067fc700 1 --2- 172.21.15.27:0/4087099525 >> v2:172.21.15.27:3300/0 conn(0x7f83080ccf40 0x7f83080cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:25.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.230+0000 7f8305ffb700 1 -- 172.21.15.27:0/4087099525 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329132958 0 0) 0x7f83080cc510 con 0x7f830815cef0 2024-01-11T09:27:25.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.230+0000 7f8305ffb700 1 -- 172.21.15.27:0/4087099525 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82f0003570 con 0x7f830815cef0 2024-01-11T09:27:25.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.231+0000 7f83067fc700 1 -- 172.21.15.27:0/4087099525 >> v1:172.21.15.77:6789/0 conn(0x7f8308161520 legacy=0x7f8308163900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:25.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.231+0000 7f83067fc700 1 -- 172.21.15.27:0/4087099525 >> v1:172.21.15.27:6789/0 conn(0x7f830815cef0 legacy=0x7f83080ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:25.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.231+0000 7f83067fc700 1 -- 172.21.15.27:0/4087099525 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830815d2c0 con 0x7f83080ccf40 2024-01-11T09:27:25.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.231+0000 7f83067fc700 1 --2- 172.21.15.27:0/4087099525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83080ccf40 0x7f83080cd3e0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f82fc0098e0 tx=0x7f82fc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cbd798ba3f9e0bb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:25.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.231+0000 7f8305ffb700 1 -- 172.21.15.27:0/4087099525 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc00b640 con 0x7f83080ccf40 2024-01-11T09:27:25.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.231+0000 7f8305ffb700 1 -- 172.21.15.27:0/4087099525 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82fc00b7a0 con 0x7f83080ccf40 2024-01-11T09:27:25.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.231+0000 7f8305ffb700 1 -- 172.21.15.27:0/4087099525 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc00bb40 con 0x7f83080ccf40 2024-01-11T09:27:25.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.232+0000 7f830d9ff700 1 -- 172.21.15.27:0/4087099525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83080ccf40 msgr2=0x7f83080cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:25.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.232+0000 7f830d9ff700 1 --2- 172.21.15.27:0/4087099525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83080ccf40 0x7f83080cd3e0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f82fc0098e0 tx=0x7f82fc0091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.232+0000 7f830d9ff700 1 -- 172.21.15.27:0/4087099525 shutdown_connections 2024-01-11T09:27:25.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.232+0000 7f830d9ff700 1 --2- 172.21.15.27:0/4087099525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83080ccf40 0x7f83080cd3e0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.232+0000 7f830d9ff700 1 -- 172.21.15.27:0/4087099525 >> 172.21.15.27:0/4087099525 conn(0x7f8308099880 msgr2=0x7f830806f520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:25.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.232+0000 7f830d9ff700 1 -- 172.21.15.27:0/4087099525 shutdown_connections 2024-01-11T09:27:25.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.232+0000 7f830d9ff700 1 -- 172.21.15.27:0/4087099525 wait complete. 2024-01-11T09:27:25.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.233+0000 7f830d9ff700 1 Processor -- start 2024-01-11T09:27:25.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.234+0000 7f830d9ff700 1 -- start start 2024-01-11T09:27:25.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.234+0000 7f830d9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83080ccf40 0x7f83080c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:25.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.234+0000 7f830d9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f830815cef0 0x7f83080c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:25.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.235+0000 7f830d9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8308161520 0x7f830816d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:25.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.235+0000 7f830d9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83080ca500 con 0x7f83080ccf40 2024-01-11T09:27:25.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.235+0000 7f830d9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83080c8760 con 0x7f8308161520 2024-01-11T09:27:25.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.235+0000 7f830d9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83080c8a40 con 0x7f830815cef0 2024-01-11T09:27:25.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.235+0000 7f8306ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83080ccf40 0x7f83080c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:25.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.235+0000 7f8306ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83080ccf40 0x7f83080c8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37634/0 (socket says 172.21.15.27:37634) 2024-01-11T09:27:25.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.235+0000 7f8306ffd700 1 -- 172.21.15.27:0/2976304285 learned_addr learned my addr 172.21.15.27:0/2976304285 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:25.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.235+0000 7f83067fc700 1 --2- 172.21.15.27:0/2976304285 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f830815cef0 0x7f83080c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:25.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.235+0000 7f8306ffd700 1 -- 172.21.15.27:0/2976304285 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f830815cef0 msgr2=0x7f83080c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:25.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.235+0000 7f83077fe700 1 --2- 172.21.15.27:0/2976304285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8308161520 0x7f830816d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:25.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.235+0000 7f8306ffd700 1 --2- 172.21.15.27:0/2976304285 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f830815cef0 0x7f83080c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.235+0000 7f8306ffd700 1 -- 172.21.15.27:0/2976304285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8308161520 msgr2=0x7f830816d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:25.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.235+0000 7f8306ffd700 1 --2- 172.21.15.27:0/2976304285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8308161520 0x7f830816d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.235+0000 7f8306ffd700 1 -- 172.21.15.27:0/2976304285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82fc009420 con 0x7f83080ccf40 2024-01-11T09:27:25.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.235+0000 7f83067fc700 1 --2- 172.21.15.27:0/2976304285 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f830815cef0 0x7f83080c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:27:25.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.235+0000 7f83077fe700 1 --2- 172.21.15.27:0/2976304285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8308161520 0x7f830816d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:27:25.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.236+0000 7f8306ffd700 1 --2- 172.21.15.27:0/2976304285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83080ccf40 0x7f83080c8180 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f82f800c790 tx=0x7f82f8007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:25.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.236+0000 7f82eeffd700 1 -- 172.21.15.27:0/2976304285 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f8007300 con 0x7f83080ccf40 2024-01-11T09:27:25.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.236+0000 7f82eeffd700 1 -- 172.21.15.27:0/2976304285 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f8007460 con 0x7f83080ccf40 2024-01-11T09:27:25.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.236+0000 7f830d9ff700 1 -- 172.21.15.27:0/2976304285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83080c8d20 con 0x7f83080ccf40 2024-01-11T09:27:25.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.236+0000 7f82eeffd700 1 -- 172.21.15.27:0/2976304285 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f80057f0 con 0x7f83080ccf40 2024-01-11T09:27:25.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.236+0000 7f830d9ff700 1 -- 172.21.15.27:0/2976304285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f830815eea0 con 0x7f83080ccf40 2024-01-11T09:27:25.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.237+0000 7f830d9ff700 1 -- 172.21.15.27:0/2976304285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8308088720 con 0x7f83080ccf40 2024-01-11T09:27:25.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.238+0000 7f82eeffd700 1 -- 172.21.15.27:0/2976304285 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f82f8005c20 con 0x7f83080ccf40 2024-01-11T09:27:25.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.239+0000 7f82eeffd700 1 --2- 172.21.15.27:0/2976304285 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f82f4066f30 0x7f82f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:25.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.239+0000 7f82eeffd700 1 -- 172.21.15.27:0/2976304285 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f82f80840d0 con 0x7f83080ccf40 2024-01-11T09:27:25.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.239+0000 7f83067fc700 1 --2- 172.21.15.27:0/2976304285 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f82f4066f30 0x7f82f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:25.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.240+0000 7f83067fc700 1 --2- 172.21.15.27:0/2976304285 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f82f4066f30 0x7f82f40693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f83080c94e0 tx=0x7f82fc017000 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:25.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.243+0000 7f82eeffd700 1 -- 172.21.15.27:0/2976304285 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f82f80512e0 con 0x7f83080ccf40 2024-01-11T09:27:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.489+0000 7f830d9ff700 1 -- 172.21.15.27:0/2976304285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f830804c3c0 con 0x7f83080ccf40 2024-01-11T09:27:25.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.489+0000 7f82eeffd700 1 -- 172.21.15.27:0/2976304285 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4601 (secure 0 0 0) 0x7f82f8054900 con 0x7f83080ccf40 2024-01-11T09:27:25.486 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:27:25.486 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8015,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1003297202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1003297202},{"type":"v1","addr":"172.21.15.77:6834","nonce":1003297202}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":8018,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3321059147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":3321059147},{"type":"v1","addr":"172.21.15.77:6835","nonce":3321059147}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":9202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2606639222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2606639222},{"type":"v1","addr":"172.21.15.27:6835","nonce":2606639222}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:27:21.470488+0000","modified":"2024-01-11T09:27:24.992917+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9564":{"gid":9564,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/3897592862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3897592862},{"type":"v1","addr":"172.21.15.27:6834","nonce":3897592862}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2024-01-11T09:27:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.492+0000 7f830d9ff700 1 -- 172.21.15.27:0/2976304285 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f82f4066f30 msgr2=0x7f82f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.492+0000 7f830d9ff700 1 --2- 172.21.15.27:0/2976304285 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f82f4066f30 0x7f82f40693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f83080c94e0 tx=0x7f82fc017000 comp rx=0 tx=0).stop 2024-01-11T09:27:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.492+0000 7f830d9ff700 1 -- 172.21.15.27:0/2976304285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83080ccf40 msgr2=0x7f83080c8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.492+0000 7f830d9ff700 1 --2- 172.21.15.27:0/2976304285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83080ccf40 0x7f83080c8180 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f82f800c790 tx=0x7f82f8007c80 comp rx=0 tx=0).stop 2024-01-11T09:27:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.492+0000 7f830d9ff700 1 -- 172.21.15.27:0/2976304285 shutdown_connections 2024-01-11T09:27:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.492+0000 7f830d9ff700 1 --2- 172.21.15.27:0/2976304285 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f82f4066f30 0x7f82f40693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.492+0000 7f830d9ff700 1 --2- 172.21.15.27:0/2976304285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8308161520 0x7f830816d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.492+0000 7f830d9ff700 1 --2- 172.21.15.27:0/2976304285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83080ccf40 0x7f83080c8180 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.492+0000 7f830d9ff700 1 --2- 172.21.15.27:0/2976304285 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f830815cef0 0x7f83080c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.492+0000 7f830d9ff700 1 -- 172.21.15.27:0/2976304285 >> 172.21.15.27:0/2976304285 conn(0x7f8308099880 msgr2=0x7f8308156ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.492+0000 7f830d9ff700 1 -- 172.21.15.27:0/2976304285 shutdown_connections 2024-01-11T09:27:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.492+0000 7f830d9ff700 1 -- 172.21.15.27:0/2976304285 wait complete. 2024-01-11T09:27:25.490 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 94 2024-01-11T09:27:25.507 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:27:25.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.713+0000 7f9446dff700 1 Processor -- start 2024-01-11T09:27:25.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.714+0000 7f9446dff700 1 -- start start 2024-01-11T09:27:25.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.715+0000 7f9446dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94400b03e0 0x7f94400b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:25.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.715+0000 7f9446dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94400b29c0 0x7f9440159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:25.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.715+0000 7f9446dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94400cd730 con 0x7f94400b07b0 2024-01-11T09:27:25.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.715+0000 7f9446dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94400b2de0 con 0x7f94400b03e0 2024-01-11T09:27:25.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.715+0000 7f9446dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94400ccb20 con 0x7f94400b29c0 2024-01-11T09:27:25.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.715+0000 7f943ffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f94400b07b0 0x7f9440159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53548/0 (socket says 172.21.15.27:53548) 2024-01-11T09:27:25.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.715+0000 7f9444b9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94400b03e0 0x7f94400b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:25.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.715+0000 7f943ffff700 1 -- 172.21.15.27:0/4290417782 learned_addr learned my addr 172.21.15.27:0/4290417782 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:25.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.715+0000 7f944539c700 1 --2- 172.21.15.27:0/4290417782 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94400b29c0 0x7f9440159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:25.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.716+0000 7f943f7fe700 1 -- 172.21.15.27:0/4290417782 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97710440 0 0) 0x7f94400cd730 con 0x7f94400b07b0 2024-01-11T09:27:25.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.716+0000 7f943f7fe700 1 -- 172.21.15.27:0/4290417782 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9430003570 con 0x7f94400b07b0 2024-01-11T09:27:25.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.716+0000 7f9444b9b700 1 -- 172.21.15.27:0/4290417782 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94400b29c0 msgr2=0x7f9440159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:25.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.716+0000 7f9444b9b700 1 --2- 172.21.15.27:0/4290417782 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94400b29c0 0x7f9440159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.716+0000 7f9444b9b700 1 -- 172.21.15.27:0/4290417782 >> v1:172.21.15.27:6789/0 conn(0x7f94400b07b0 legacy=0x7f9440159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:25.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.716+0000 7f9444b9b700 1 -- 172.21.15.27:0/4290417782 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94400cce00 con 0x7f94400b03e0 2024-01-11T09:27:25.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.716+0000 7f9444b9b700 1 --2- 172.21.15.27:0/4290417782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94400b03e0 0x7f94400b2480 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f94340098e0 tx=0x7f94340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671a255eaeafdad1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:25.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.716+0000 7f943f7fe700 1 -- 172.21.15.27:0/4290417782 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9434011630 con 0x7f94400b03e0 2024-01-11T09:27:25.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.716+0000 7f943f7fe700 1 -- 172.21.15.27:0/4290417782 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9434011790 con 0x7f94400b03e0 2024-01-11T09:27:25.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.716+0000 7f943f7fe700 1 -- 172.21.15.27:0/4290417782 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9434011b30 con 0x7f94400b03e0 2024-01-11T09:27:25.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.717+0000 7f9446dff700 1 -- 172.21.15.27:0/4290417782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94400b03e0 msgr2=0x7f94400b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:25.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.717+0000 7f9446dff700 1 --2- 172.21.15.27:0/4290417782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94400b03e0 0x7f94400b2480 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f94340098e0 tx=0x7f94340091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.717+0000 7f9446dff700 1 -- 172.21.15.27:0/4290417782 shutdown_connections 2024-01-11T09:27:25.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.717+0000 7f9446dff700 1 --2- 172.21.15.27:0/4290417782 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94400b29c0 0x7f9440159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.717+0000 7f9446dff700 1 --2- 172.21.15.27:0/4290417782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94400b03e0 0x7f94400b2480 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.717+0000 7f9446dff700 1 -- 172.21.15.27:0/4290417782 >> 172.21.15.27:0/4290417782 conn(0x7f944006ed40 msgr2=0x7f944006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:25.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.717+0000 7f9446dff700 1 -- 172.21.15.27:0/4290417782 shutdown_connections 2024-01-11T09:27:25.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.717+0000 7f9446dff700 1 -- 172.21.15.27:0/4290417782 wait complete. 2024-01-11T09:27:25.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.718+0000 7f9446dff700 1 Processor -- start 2024-01-11T09:27:25.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.719+0000 7f9446dff700 1 -- start start 2024-01-11T09:27:25.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.720+0000 7f9446dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94400b03e0 0x7f944014d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:25.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.720+0000 7f9446dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94400b07b0 0x7f94400c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:25.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.720+0000 7f9446dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94400b29c0 0x7f94400d0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:25.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.720+0000 7f9446dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94400c6c30 con 0x7f94400b29c0 2024-01-11T09:27:25.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.720+0000 7f9446dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94400c8fc0 con 0x7f94400b03e0 2024-01-11T09:27:25.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.720+0000 7f9446dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94400c83b0 con 0x7f94400b07b0 2024-01-11T09:27:25.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.720+0000 7f944539c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94400b29c0 0x7f94400d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:25.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.720+0000 7f944539c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94400b29c0 0x7f94400d0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37650/0 (socket says 172.21.15.27:37650) 2024-01-11T09:27:25.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.720+0000 7f944539c700 1 -- 172.21.15.27:0/147484734 learned_addr learned my addr 172.21.15.27:0/147484734 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:25.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.720+0000 7f943ffff700 1 --2- 172.21.15.27:0/147484734 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94400b07b0 0x7f94400c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:25.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.720+0000 7f9444b9b700 1 --2- 172.21.15.27:0/147484734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94400b03e0 0x7f944014d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:25.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.720+0000 7f944539c700 1 -- 172.21.15.27:0/147484734 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94400b07b0 msgr2=0x7f94400c5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:25.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.720+0000 7f944539c700 1 --2- 172.21.15.27:0/147484734 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94400b07b0 0x7f94400c5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.720+0000 7f944539c700 1 -- 172.21.15.27:0/147484734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94400b03e0 msgr2=0x7f944014d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:25.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.720+0000 7f944539c700 1 --2- 172.21.15.27:0/147484734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94400b03e0 0x7f944014d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.720+0000 7f944539c700 1 -- 172.21.15.27:0/147484734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9434009420 con 0x7f94400b29c0 2024-01-11T09:27:25.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.721+0000 7f944539c700 1 --2- 172.21.15.27:0/147484734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94400b29c0 0x7f94400d0630 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f9438009df0 tx=0x7f943800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:25.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.721+0000 7f943cff9700 1 -- 172.21.15.27:0/147484734 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943800caa0 con 0x7f94400b29c0 2024-01-11T09:27:25.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.721+0000 7f943cff9700 1 -- 172.21.15.27:0/147484734 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f943800f040 con 0x7f94400b29c0 2024-01-11T09:27:25.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.721+0000 7f9446dff700 1 -- 172.21.15.27:0/147484734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94400c8690 con 0x7f94400b29c0 2024-01-11T09:27:25.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.721+0000 7f943cff9700 1 -- 172.21.15.27:0/147484734 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943800ce70 con 0x7f94400b29c0 2024-01-11T09:27:25.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.721+0000 7f9446dff700 1 -- 172.21.15.27:0/147484734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94400c8c50 con 0x7f94400b29c0 2024-01-11T09:27:25.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.722+0000 7f9446dff700 1 -- 172.21.15.27:0/147484734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94400718d0 con 0x7f94400b29c0 2024-01-11T09:27:25.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.723+0000 7f943cff9700 1 -- 172.21.15.27:0/147484734 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9438014690 con 0x7f94400b29c0 2024-01-11T09:27:25.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.724+0000 7f943cff9700 1 --2- 172.21.15.27:0/147484734 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f9424066f30 0x7f94240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:25.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.724+0000 7f943cff9700 1 -- 172.21.15.27:0/147484734 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f9438085090 con 0x7f94400b29c0 2024-01-11T09:27:25.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.724+0000 7f9444b9b700 1 --2- 172.21.15.27:0/147484734 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f9424066f30 0x7f94240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:25.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.725+0000 7f9444b9b700 1 --2- 172.21.15.27:0/147484734 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f9424066f30 0x7f94240693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f94340091e0 tx=0x7f9434017000 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:25.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.728+0000 7f943cff9700 1 -- 172.21.15.27:0/147484734 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9438055d50 con 0x7f94400b29c0 2024-01-11T09:27:25.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.932+0000 7f9446dff700 1 -- 172.21.15.27:0/147484734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94400cd0e0 con 0x7f94400b29c0 2024-01-11T09:27:25.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.933+0000 7f943cff9700 1 -- 172.21.15.27:0/147484734 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) v1 ==== 74+0+15404 (secure 0 0 0) 0x7f94380558e0 con 0x7f94400b29c0 2024-01-11T09:27:25.933 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:27:25.933 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":181,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:27:22.977917+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:27:19.009515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-01-11T09:27:20.468964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:27:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.935+0000 7f9446dff700 1 -- 172.21.15.27:0/147484734 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f9424066f30 msgr2=0x7f94240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.935+0000 7f9446dff700 1 --2- 172.21.15.27:0/147484734 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f9424066f30 0x7f94240693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f94340091e0 tx=0x7f9434017000 comp rx=0 tx=0).stop 2024-01-11T09:27:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.936+0000 7f9446dff700 1 -- 172.21.15.27:0/147484734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94400b29c0 msgr2=0x7f94400d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.936+0000 7f9446dff700 1 --2- 172.21.15.27:0/147484734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94400b29c0 0x7f94400d0630 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f9438009df0 tx=0x7f943800b040 comp rx=0 tx=0).stop 2024-01-11T09:27:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.936+0000 7f9446dff700 1 -- 172.21.15.27:0/147484734 shutdown_connections 2024-01-11T09:27:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.936+0000 7f9446dff700 1 --2- 172.21.15.27:0/147484734 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f9424066f30 0x7f94240693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.936+0000 7f9446dff700 1 --2- 172.21.15.27:0/147484734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94400b29c0 0x7f94400d0630 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.936+0000 7f9446dff700 1 --2- 172.21.15.27:0/147484734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94400b03e0 0x7f944014d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.936+0000 7f9446dff700 1 --2- 172.21.15.27:0/147484734 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94400b07b0 0x7f94400c5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.936+0000 7f9446dff700 1 -- 172.21.15.27:0/147484734 >> 172.21.15.27:0/147484734 conn(0x7f944006ed40 msgr2=0x7f94400cb400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.936+0000 7f9446dff700 1 -- 172.21.15.27:0/147484734 shutdown_connections 2024-01-11T09:27:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:25.936+0000 7f9446dff700 1 -- 172.21.15.27:0/147484734 wait complete. 2024-01-11T09:27:25.940 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-01-11T09:27:25.943+0000 7f5f459f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T09:27:25.951 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-11T09:27:25.951 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:27:26.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.154+0000 7fe47cfff700 1 Processor -- start 2024-01-11T09:27:26.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.154+0000 7fe47cfff700 1 -- start start 2024-01-11T09:27:26.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.156+0000 7fe47cfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe4780cc410 0x7fe478159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:26.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.156+0000 7fe47cfff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4780cd730 con 0x7fe4780b03e0 2024-01-11T09:27:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.156+0000 7fe47cfff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4780ccb20 con 0x7fe4780b2b90 2024-01-11T09:27:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.156+0000 7fe47cfff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4780cce00 con 0x7fe4780cc410 2024-01-11T09:27:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.156+0000 7fe47659c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe4780b03e0 0x7fe4780b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53550/0 (socket says 172.21.15.27:53550) 2024-01-11T09:27:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.156+0000 7fe47659c700 1 -- 172.21.15.27:0/326130198 learned_addr learned my addr 172.21.15.27:0/326130198 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.156+0000 7fe476d9d700 1 --2- 172.21.15.27:0/326130198 >> v2:172.21.15.27:3300/0 conn(0x7fe4780cc410 0x7fe478159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:26.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.156+0000 7fe47559a700 1 -- 172.21.15.27:0/326130198 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744376337 0 0) 0x7fe4780cd730 con 0x7fe4780b03e0 2024-01-11T09:27:26.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.156+0000 7fe47559a700 1 -- 172.21.15.27:0/326130198 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe460003570 con 0x7fe4780b03e0 2024-01-11T09:27:26.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.156+0000 7fe47559a700 1 -- 172.21.15.27:0/326130198 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152220591 0 0) 0x7fe4780ccb20 con 0x7fe4780b2b90 2024-01-11T09:27:26.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.156+0000 7fe47559a700 1 -- 172.21.15.27:0/326130198 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4780cd730 con 0x7fe4780b2b90 2024-01-11T09:27:26.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.157+0000 7fe476d9d700 1 -- 172.21.15.27:0/326130198 >> v1:172.21.15.77:6789/0 conn(0x7fe4780b2b90 legacy=0x7fe478159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:26.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.157+0000 7fe476d9d700 1 -- 172.21.15.27:0/326130198 >> v1:172.21.15.27:6789/0 conn(0x7fe4780b03e0 legacy=0x7fe4780b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:26.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.157+0000 7fe476d9d700 1 -- 172.21.15.27:0/326130198 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4780cd0e0 con 0x7fe4780cc410 2024-01-11T09:27:26.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.157+0000 7fe476d9d700 1 --2- 172.21.15.27:0/326130198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4780cc410 0x7fe478159910 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fe470009e90 tx=0x7fe470009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d79b49cac9ba0ce server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:26.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.157+0000 7fe47559a700 1 -- 172.21.15.27:0/326130198 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47000e070 con 0x7fe4780cc410 2024-01-11T09:27:26.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.157+0000 7fe47559a700 1 -- 172.21.15.27:0/326130198 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47000bb90 con 0x7fe4780cc410 2024-01-11T09:27:26.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.157+0000 7fe47559a700 1 -- 172.21.15.27:0/326130198 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470012430 con 0x7fe4780cc410 2024-01-11T09:27:26.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.158+0000 7fe47cfff700 1 -- 172.21.15.27:0/326130198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4780cc410 msgr2=0x7fe478159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:26.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.158+0000 7fe47cfff700 1 --2- 172.21.15.27:0/326130198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4780cc410 0x7fe478159910 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fe470009e90 tx=0x7fe470009780 comp rx=0 tx=0).stop 2024-01-11T09:27:26.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.158+0000 7fe47cfff700 1 -- 172.21.15.27:0/326130198 shutdown_connections 2024-01-11T09:27:26.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.158+0000 7fe47cfff700 1 --2- 172.21.15.27:0/326130198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4780cc410 0x7fe478159910 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:26.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.158+0000 7fe47cfff700 1 -- 172.21.15.27:0/326130198 >> 172.21.15.27:0/326130198 conn(0x7fe47806ed40 msgr2=0x7fe47806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:26.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.158+0000 7fe47cfff700 1 -- 172.21.15.27:0/326130198 shutdown_connections 2024-01-11T09:27:26.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.158+0000 7fe47cfff700 1 -- 172.21.15.27:0/326130198 wait complete. 2024-01-11T09:27:26.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.159+0000 7fe47cfff700 1 Processor -- start 2024-01-11T09:27:26.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.160+0000 7fe47cfff700 1 -- start start 2024-01-11T09:27:26.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.160+0000 7fe47cfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4780b03e0 0x7fe478158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:26.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.160+0000 7fe47cfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4780b2b90 0x7fe47816a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:26.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.161+0000 7fe47cfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4780cc410 0x7fe4780b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:26.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.161+0000 7fe47cfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4780d0ea0 con 0x7fe4780cc410 2024-01-11T09:27:26.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.161+0000 7fe47cfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4780c8e20 con 0x7fe4780b03e0 2024-01-11T09:27:26.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.161+0000 7fe47cfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe478159240 con 0x7fe4780b2b90 2024-01-11T09:27:26.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.161+0000 7fe476d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4780cc410 0x7fe4780b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:26.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.161+0000 7fe476d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4780cc410 0x7fe4780b3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37668/0 (socket says 172.21.15.27:37668) 2024-01-11T09:27:26.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.161+0000 7fe476d9d700 1 -- 172.21.15.27:0/3873390595 learned_addr learned my addr 172.21.15.27:0/3873390595 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:26.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.161+0000 7fe47659c700 1 --2- 172.21.15.27:0/3873390595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4780b03e0 0x7fe478158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:26.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.161+0000 7fe475d9b700 1 --2- 172.21.15.27:0/3873390595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4780b2b90 0x7fe47816a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:26.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.161+0000 7fe476d9d700 1 -- 172.21.15.27:0/3873390595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4780b2b90 msgr2=0x7fe47816a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:26.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.161+0000 7fe476d9d700 1 --2- 172.21.15.27:0/3873390595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4780b2b90 0x7fe47816a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:26.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.161+0000 7fe476d9d700 1 -- 172.21.15.27:0/3873390595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4780b03e0 msgr2=0x7fe478158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:26.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.161+0000 7fe476d9d700 1 --2- 172.21.15.27:0/3873390595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4780b03e0 0x7fe478158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:26.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.161+0000 7fe476d9d700 1 -- 172.21.15.27:0/3873390595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe470009a20 con 0x7fe4780cc410 2024-01-11T09:27:26.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.162+0000 7fe476d9d700 1 --2- 172.21.15.27:0/3873390595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4780cc410 0x7fe4780b3360 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fe47000b610 tx=0x7fe470019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.162+0000 7fe4667fc700 1 -- 172.21.15.27:0/3873390595 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470026480 con 0x7fe4780cc410 2024-01-11T09:27:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.162+0000 7fe4667fc700 1 -- 172.21.15.27:0/3873390595 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4700265e0 con 0x7fe4780cc410 2024-01-11T09:27:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.162+0000 7fe4667fc700 1 -- 172.21.15.27:0/3873390595 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4700269a0 con 0x7fe4780cc410 2024-01-11T09:27:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.162+0000 7fe47cfff700 1 -- 172.21.15.27:0/3873390595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe47816ac90 con 0x7fe4780cc410 2024-01-11T09:27:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.162+0000 7fe47cfff700 1 -- 172.21.15.27:0/3873390595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4780cf980 con 0x7fe4780cc410 2024-01-11T09:27:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.163+0000 7fe47cfff700 1 -- 172.21.15.27:0/3873390595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4780718d0 con 0x7fe4780cc410 2024-01-11T09:27:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.164+0000 7fe4667fc700 1 -- 172.21.15.27:0/3873390595 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe470026d10 con 0x7fe4780cc410 2024-01-11T09:27:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.165+0000 7fe4667fc700 1 --2- 172.21.15.27:0/3873390595 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe458066f30 0x7fe4580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.165+0000 7fe4667fc700 1 -- 172.21.15.27:0/3873390595 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fe4700864c0 con 0x7fe4780cc410 2024-01-11T09:27:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.166+0000 7fe47659c700 1 --2- 172.21.15.27:0/3873390595 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe458066f30 0x7fe4580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.166+0000 7fe47659c700 1 --2- 172.21.15.27:0/3873390595 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe458066f30 0x7fe4580693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe468007fd0 tx=0x7fe468009040 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.169+0000 7fe4667fc700 1 -- 172.21.15.27:0/3873390595 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe4700536f0 con 0x7fe4780cc410 2024-01-11T09:27:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.368+0000 7fe47cfff700 1 -- 172.21.15.27:0/3873390595 --> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe4780cd3c0 con 0x7fe458066f30 2024-01-11T09:27:26.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.379+0000 7fe4667fc700 1 -- 172.21.15.27:0/3873390595 <== mgr.9010 v2:172.21.15.77:6836/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289427 (secure 0 0 0) 0x7fe4780cd3c0 con 0x7fe458066f30 2024-01-11T09:27:26.381 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:27:26.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.384+0000 7fe47cfff700 1 -- 172.21.15.27:0/3873390595 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe458066f30 msgr2=0x7fe4580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:26.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.384+0000 7fe47cfff700 1 --2- 172.21.15.27:0/3873390595 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe458066f30 0x7fe4580693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe468007fd0 tx=0x7fe468009040 comp rx=0 tx=0).stop 2024-01-11T09:27:26.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.384+0000 7fe47cfff700 1 -- 172.21.15.27:0/3873390595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4780cc410 msgr2=0x7fe4780b3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:26.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.384+0000 7fe47cfff700 1 --2- 172.21.15.27:0/3873390595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4780cc410 0x7fe4780b3360 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fe47000b610 tx=0x7fe470019000 comp rx=0 tx=0).stop 2024-01-11T09:27:26.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.384+0000 7fe47cfff700 1 -- 172.21.15.27:0/3873390595 shutdown_connections 2024-01-11T09:27:26.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.384+0000 7fe47cfff700 1 --2- 172.21.15.27:0/3873390595 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fe458066f30 0x7fe4580693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:26.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.384+0000 7fe47cfff700 1 --2- 172.21.15.27:0/3873390595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4780cc410 0x7fe4780b3360 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:26.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.384+0000 7fe47cfff700 1 --2- 172.21.15.27:0/3873390595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4780b03e0 0x7fe478158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:26.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.384+0000 7fe47cfff700 1 --2- 172.21.15.27:0/3873390595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4780b2b90 0x7fe47816a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:26.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.384+0000 7fe47cfff700 1 -- 172.21.15.27:0/3873390595 >> 172.21.15.27:0/3873390595 conn(0x7fe47806ed40 msgr2=0x7fe4780b4c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:26.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.384+0000 7fe47cfff700 1 -- 172.21.15.27:0/3873390595 shutdown_connections 2024-01-11T09:27:26.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.384+0000 7fe47cfff700 1 -- 172.21.15.27:0/3873390595 wait complete. 2024-01-11T09:27:26.385 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:27:26.399 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-11T09:27:25.118717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1808,"num_read_kb":3778,"num_write":555,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":349,"ondisk_log_size":349,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":99,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2458568,"kb_used_data":18824,"kb_used_omap":0,"kb_used_meta":2439680,"kb_avail":752516152,"statfs":{"total":773094113280,"available":770576539648,"internally_reserved":0,"allocated":19275776,"data_stored":9706420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2498232320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.164745"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.003239+0000","last_change":"2024-01-11T09:27:22.003239+0000","last_active":"2024-01-11T09:27:22.003239+0000","last_peered":"2024-01-11T09:27:22.003239+0000","last_clean":"2024-01-11T09:27:22.003239+0000","last_became_active":"2024-01-11T09:27:22.002923+0000","last_became_peered":"2024-01-11T09:27:22.002923+0000","last_unstale":"2024-01-11T09:27:22.003239+0000","last_undegraded":"2024-01-11T09:27:22.003239+0000","last_fullsized":"2024-01-11T09:27:22.003239+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:46:13.646100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.994427+0000","last_change":"2024-01-11T09:27:21.994427+0000","last_active":"2024-01-11T09:27:21.994427+0000","last_peered":"2024-01-11T09:27:21.994427+0000","last_clean":"2024-01-11T09:27:21.994427+0000","last_became_active":"2024-01-11T09:27:21.993111+0000","last_became_peered":"2024-01-11T09:27:21.993111+0000","last_unstale":"2024-01-11T09:27:21.994427+0000","last_undegraded":"2024-01-11T09:27:21.994427+0000","last_fullsized":"2024-01-11T09:27:21.994427+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:12:11.031349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.992428+0000","last_change":"2024-01-11T09:27:21.984326+0000","last_active":"2024-01-11T09:27:21.992428+0000","last_peered":"2024-01-11T09:27:21.992428+0000","last_clean":"2024-01-11T09:27:21.992428+0000","last_became_active":"2024-01-11T09:27:21.984040+0000","last_became_peered":"2024-01-11T09:27:21.984040+0000","last_unstale":"2024-01-11T09:27:21.992428+0000","last_undegraded":"2024-01-11T09:27:21.992428+0000","last_fullsized":"2024-01-11T09:27:21.992428+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:07:57.735385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.996402+0000","last_change":"2024-01-11T09:27:21.996402+0000","last_active":"2024-01-11T09:27:21.996402+0000","last_peered":"2024-01-11T09:27:21.996402+0000","last_clean":"2024-01-11T09:27:21.996402+0000","last_became_active":"2024-01-11T09:27:21.994336+0000","last_became_peered":"2024-01-11T09:27:21.994336+0000","last_unstale":"2024-01-11T09:27:21.996402+0000","last_undegraded":"2024-01-11T09:27:21.996402+0000","last_fullsized":"2024-01-11T09:27:21.996402+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:40:30.768083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.995482+0000","last_change":"2024-01-11T09:27:21.995482+0000","last_active":"2024-01-11T09:27:21.995482+0000","last_peered":"2024-01-11T09:27:21.995482+0000","last_clean":"2024-01-11T09:27:21.995482+0000","last_became_active":"2024-01-11T09:27:21.994886+0000","last_became_peered":"2024-01-11T09:27:21.994886+0000","last_unstale":"2024-01-11T09:27:21.995482+0000","last_undegraded":"2024-01-11T09:27:21.995482+0000","last_fullsized":"2024-01-11T09:27:21.995482+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:15:32.573052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.995479+0000","last_change":"2024-01-11T09:27:21.995479+0000","last_active":"2024-01-11T09:27:21.995479+0000","last_peered":"2024-01-11T09:27:21.995479+0000","last_clean":"2024-01-11T09:27:21.995479+0000","last_became_active":"2024-01-11T09:27:21.995246+0000","last_became_peered":"2024-01-11T09:27:21.995246+0000","last_unstale":"2024-01-11T09:27:21.995479+0000","last_undegraded":"2024-01-11T09:27:21.995479+0000","last_fullsized":"2024-01-11T09:27:21.995479+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:17:28.045448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.993243+0000","last_change":"2024-01-11T09:27:21.993243+0000","last_active":"2024-01-11T09:27:21.993243+0000","last_peered":"2024-01-11T09:27:21.993243+0000","last_clean":"2024-01-11T09:27:21.993243+0000","last_became_active":"2024-01-11T09:27:21.992394+0000","last_became_peered":"2024-01-11T09:27:21.992394+0000","last_unstale":"2024-01-11T09:27:21.993243+0000","last_undegraded":"2024-01-11T09:27:21.993243+0000","last_fullsized":"2024-01-11T09:27:21.993243+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:44:16.678864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.995807+0000","last_change":"2024-01-11T09:27:21.995807+0000","last_active":"2024-01-11T09:27:21.995807+0000","last_peered":"2024-01-11T09:27:21.995807+0000","last_clean":"2024-01-11T09:27:21.995807+0000","last_became_active":"2024-01-11T09:27:21.995506+0000","last_became_peered":"2024-01-11T09:27:21.995506+0000","last_unstale":"2024-01-11T09:27:21.995807+0000","last_undegraded":"2024-01-11T09:27:21.995807+0000","last_fullsized":"2024-01-11T09:27:21.995807+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:22:02.217251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.994924+0000","last_change":"2024-01-11T09:27:21.994924+0000","last_active":"2024-01-11T09:27:21.994924+0000","last_peered":"2024-01-11T09:27:21.994924+0000","last_clean":"2024-01-11T09:27:21.994924+0000","last_became_active":"2024-01-11T09:27:21.994626+0000","last_became_peered":"2024-01-11T09:27:21.994626+0000","last_unstale":"2024-01-11T09:27:21.994924+0000","last_undegraded":"2024-01-11T09:27:21.994924+0000","last_fullsized":"2024-01-11T09:27:21.994924+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:48:22.941096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.004903+0000","last_change":"2024-01-11T09:27:22.004903+0000","last_active":"2024-01-11T09:27:22.004903+0000","last_peered":"2024-01-11T09:27:22.004903+0000","last_clean":"2024-01-11T09:27:22.004903+0000","last_became_active":"2024-01-11T09:27:22.004622+0000","last_became_peered":"2024-01-11T09:27:22.004622+0000","last_unstale":"2024-01-11T09:27:22.004903+0000","last_undegraded":"2024-01-11T09:27:22.004903+0000","last_fullsized":"2024-01-11T09:27:22.004903+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:49.010448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.996372+0000","last_change":"2024-01-11T09:27:21.996372+0000","last_active":"2024-01-11T09:27:21.996372+0000","last_peered":"2024-01-11T09:27:21.996372+0000","last_clean":"2024-01-11T09:27:21.996372+0000","last_became_active":"2024-01-11T09:27:21.995148+0000","last_became_peered":"2024-01-11T09:27:21.995148+0000","last_unstale":"2024-01-11T09:27:21.996372+0000","last_undegraded":"2024-01-11T09:27:21.996372+0000","last_fullsized":"2024-01-11T09:27:21.996372+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:56:38.205676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.990728+0000","last_change":"2024-01-11T09:27:21.990728+0000","last_active":"2024-01-11T09:27:21.990728+0000","last_peered":"2024-01-11T09:27:21.990728+0000","last_clean":"2024-01-11T09:27:21.990728+0000","last_became_active":"2024-01-11T09:27:21.990412+0000","last_became_peered":"2024-01-11T09:27:21.990412+0000","last_unstale":"2024-01-11T09:27:21.990728+0000","last_undegraded":"2024-01-11T09:27:21.990728+0000","last_fullsized":"2024-01-11T09:27:21.990728+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:06:01.994798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.989114+0000","last_change":"2024-01-11T09:27:21.977786+0000","last_active":"2024-01-11T09:27:21.989114+0000","last_peered":"2024-01-11T09:27:21.989114+0000","last_clean":"2024-01-11T09:27:21.989114+0000","last_became_active":"2024-01-11T09:27:21.977494+0000","last_became_peered":"2024-01-11T09:27:21.977494+0000","last_unstale":"2024-01-11T09:27:21.989114+0000","last_undegraded":"2024-01-11T09:27:21.989114+0000","last_fullsized":"2024-01-11T09:27:21.989114+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:38:37.294337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.996904+0000","last_change":"2024-01-11T09:27:21.989672+0000","last_active":"2024-01-11T09:27:21.996904+0000","last_peered":"2024-01-11T09:27:21.996904+0000","last_clean":"2024-01-11T09:27:21.996904+0000","last_became_active":"2024-01-11T09:27:21.988116+0000","last_became_peered":"2024-01-11T09:27:21.988116+0000","last_unstale":"2024-01-11T09:27:21.996904+0000","last_undegraded":"2024-01-11T09:27:21.996904+0000","last_fullsized":"2024-01-11T09:27:21.996904+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:38:04.939061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.003938+0000","last_change":"2024-01-11T09:27:22.003938+0000","last_active":"2024-01-11T09:27:22.003938+0000","last_peered":"2024-01-11T09:27:22.003938+0000","last_clean":"2024-01-11T09:27:22.003938+0000","last_became_active":"2024-01-11T09:27:22.003635+0000","last_became_peered":"2024-01-11T09:27:22.003635+0000","last_unstale":"2024-01-11T09:27:22.003938+0000","last_undegraded":"2024-01-11T09:27:22.003938+0000","last_fullsized":"2024-01-11T09:27:22.003938+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:04:20.823816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.989789+0000","last_change":"2024-01-11T09:27:21.989789+0000","last_active":"2024-01-11T09:27:21.989789+0000","last_peered":"2024-01-11T09:27:21.989789+0000","last_clean":"2024-01-11T09:27:21.989789+0000","last_became_active":"2024-01-11T09:27:21.989524+0000","last_became_peered":"2024-01-11T09:27:21.989524+0000","last_unstale":"2024-01-11T09:27:21.989789+0000","last_undegraded":"2024-01-11T09:27:21.989789+0000","last_fullsized":"2024-01-11T09:27:21.989789+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:17:26.059638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.006565+0000","last_change":"2024-01-11T09:27:22.006565+0000","last_active":"2024-01-11T09:27:22.006565+0000","last_peered":"2024-01-11T09:27:22.006565+0000","last_clean":"2024-01-11T09:27:22.006565+0000","last_became_active":"2024-01-11T09:27:22.006246+0000","last_became_peered":"2024-01-11T09:27:22.006246+0000","last_unstale":"2024-01-11T09:27:22.006565+0000","last_undegraded":"2024-01-11T09:27:22.006565+0000","last_fullsized":"2024-01-11T09:27:22.006565+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:09:18.445842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.008212+0000","last_change":"2024-01-11T09:27:22.003524+0000","last_active":"2024-01-11T09:27:22.008212+0000","last_peered":"2024-01-11T09:27:22.008212+0000","last_clean":"2024-01-11T09:27:22.008212+0000","last_became_active":"2024-01-11T09:27:22.003223+0000","last_became_peered":"2024-01-11T09:27:22.003223+0000","last_unstale":"2024-01-11T09:27:22.008212+0000","last_undegraded":"2024-01-11T09:27:22.008212+0000","last_fullsized":"2024-01-11T09:27:22.008212+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:55:38.095443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.994165+0000","last_change":"2024-01-11T09:27:21.994165+0000","last_active":"2024-01-11T09:27:21.994165+0000","last_peered":"2024-01-11T09:27:21.994165+0000","last_clean":"2024-01-11T09:27:21.994165+0000","last_became_active":"2024-01-11T09:27:21.993878+0000","last_became_peered":"2024-01-11T09:27:21.993878+0000","last_unstale":"2024-01-11T09:27:21.994165+0000","last_undegraded":"2024-01-11T09:27:21.994165+0000","last_fullsized":"2024-01-11T09:27:21.994165+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:07:57.735385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.996854+0000","last_change":"2024-01-11T09:27:21.989178+0000","last_active":"2024-01-11T09:27:21.996854+0000","last_peered":"2024-01-11T09:27:21.996854+0000","last_clean":"2024-01-11T09:27:21.996854+0000","last_became_active":"2024-01-11T09:27:21.988899+0000","last_became_peered":"2024-01-11T09:27:21.988899+0000","last_unstale":"2024-01-11T09:27:21.996854+0000","last_undegraded":"2024-01-11T09:27:21.996854+0000","last_fullsized":"2024-01-11T09:27:21.996854+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:34:07.131869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.004962+0000","last_change":"2024-01-11T09:27:22.004962+0000","last_active":"2024-01-11T09:27:22.004962+0000","last_peered":"2024-01-11T09:27:22.004962+0000","last_clean":"2024-01-11T09:27:22.004962+0000","last_became_active":"2024-01-11T09:27:22.004673+0000","last_became_peered":"2024-01-11T09:27:22.004673+0000","last_unstale":"2024-01-11T09:27:22.004962+0000","last_undegraded":"2024-01-11T09:27:22.004962+0000","last_fullsized":"2024-01-11T09:27:22.004962+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:34:07.131869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.997561+0000","last_change":"2024-01-11T09:27:21.997561+0000","last_active":"2024-01-11T09:27:21.997561+0000","last_peered":"2024-01-11T09:27:21.997561+0000","last_clean":"2024-01-11T09:27:21.997561+0000","last_became_active":"2024-01-11T09:27:21.997188+0000","last_became_peered":"2024-01-11T09:27:21.997188+0000","last_unstale":"2024-01-11T09:27:21.997561+0000","last_undegraded":"2024-01-11T09:27:21.997561+0000","last_fullsized":"2024-01-11T09:27:21.997561+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:44:16.678864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.991642+0000","last_change":"2024-01-11T09:27:21.991642+0000","last_active":"2024-01-11T09:27:21.991642+0000","last_peered":"2024-01-11T09:27:21.991642+0000","last_clean":"2024-01-11T09:27:21.991642+0000","last_became_active":"2024-01-11T09:27:21.991349+0000","last_became_peered":"2024-01-11T09:27:21.991349+0000","last_unstale":"2024-01-11T09:27:21.991642+0000","last_undegraded":"2024-01-11T09:27:21.991642+0000","last_fullsized":"2024-01-11T09:27:21.991642+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:55:38.946028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.996969+0000","last_change":"2024-01-11T09:27:21.996969+0000","last_active":"2024-01-11T09:27:21.996969+0000","last_peered":"2024-01-11T09:27:21.996969+0000","last_clean":"2024-01-11T09:27:21.996969+0000","last_became_active":"2024-01-11T09:27:21.996413+0000","last_became_peered":"2024-01-11T09:27:21.996413+0000","last_unstale":"2024-01-11T09:27:21.996969+0000","last_undegraded":"2024-01-11T09:27:21.996969+0000","last_fullsized":"2024-01-11T09:27:21.996969+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:13:01.725223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.992336+0000","last_change":"2024-01-11T09:27:21.982776+0000","last_active":"2024-01-11T09:27:21.992336+0000","last_peered":"2024-01-11T09:27:21.992336+0000","last_clean":"2024-01-11T09:27:21.992336+0000","last_became_active":"2024-01-11T09:27:21.980022+0000","last_became_peered":"2024-01-11T09:27:21.980022+0000","last_unstale":"2024-01-11T09:27:21.992336+0000","last_undegraded":"2024-01-11T09:27:21.992336+0000","last_fullsized":"2024-01-11T09:27:21.992336+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:58:47.672394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.008410+0000","last_change":"2024-01-11T09:27:22.008410+0000","last_active":"2024-01-11T09:27:22.008410+0000","last_peered":"2024-01-11T09:27:22.008410+0000","last_clean":"2024-01-11T09:27:22.008410+0000","last_became_active":"2024-01-11T09:27:22.007205+0000","last_became_peered":"2024-01-11T09:27:22.007205+0000","last_unstale":"2024-01-11T09:27:22.008410+0000","last_undegraded":"2024-01-11T09:27:22.008410+0000","last_fullsized":"2024-01-11T09:27:22.008410+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:13:01.725223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.000447+0000","last_change":"2024-01-11T09:27:21.995882+0000","last_active":"2024-01-11T09:27:22.000447+0000","last_peered":"2024-01-11T09:27:22.000447+0000","last_clean":"2024-01-11T09:27:22.000447+0000","last_became_active":"2024-01-11T09:27:21.994795+0000","last_became_peered":"2024-01-11T09:27:21.994795+0000","last_unstale":"2024-01-11T09:27:22.000447+0000","last_undegraded":"2024-01-11T09:27:22.000447+0000","last_fullsized":"2024-01-11T09:27:22.000447+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:41:27.430029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.002635+0000","last_change":"2024-01-11T09:27:22.002635+0000","last_active":"2024-01-11T09:27:22.002635+0000","last_peered":"2024-01-11T09:27:22.002635+0000","last_clean":"2024-01-11T09:27:22.002635+0000","last_became_active":"2024-01-11T09:27:22.002296+0000","last_became_peered":"2024-01-11T09:27:22.002296+0000","last_unstale":"2024-01-11T09:27:22.002635+0000","last_undegraded":"2024-01-11T09:27:22.002635+0000","last_fullsized":"2024-01-11T09:27:22.002635+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:35:10.052296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.001070+0000","last_change":"2024-01-11T09:27:21.995530+0000","last_active":"2024-01-11T09:27:22.001070+0000","last_peered":"2024-01-11T09:27:22.001070+0000","last_clean":"2024-01-11T09:27:22.001070+0000","last_became_active":"2024-01-11T09:27:21.993025+0000","last_became_peered":"2024-01-11T09:27:21.993025+0000","last_unstale":"2024-01-11T09:27:22.001070+0000","last_undegraded":"2024-01-11T09:27:22.001070+0000","last_fullsized":"2024-01-11T09:27:22.001070+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:03:37.377064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.993445+0000","last_change":"2024-01-11T09:27:21.990335+0000","last_active":"2024-01-11T09:27:21.993445+0000","last_peered":"2024-01-11T09:27:21.993445+0000","last_clean":"2024-01-11T09:27:21.993445+0000","last_became_active":"2024-01-11T09:27:21.990059+0000","last_became_peered":"2024-01-11T09:27:21.990059+0000","last_unstale":"2024-01-11T09:27:21.993445+0000","last_undegraded":"2024-01-11T09:27:21.993445+0000","last_fullsized":"2024-01-11T09:27:21.993445+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:54:30.583702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.991582+0000","last_change":"2024-01-11T09:27:21.987052+0000","last_active":"2024-01-11T09:27:21.991582+0000","last_peered":"2024-01-11T09:27:21.991582+0000","last_clean":"2024-01-11T09:27:21.991582+0000","last_became_active":"2024-01-11T09:27:21.986668+0000","last_became_peered":"2024-01-11T09:27:21.986668+0000","last_unstale":"2024-01-11T09:27:21.991582+0000","last_undegraded":"2024-01-11T09:27:21.991582+0000","last_fullsized":"2024-01-11T09:27:21.991582+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:42:08.682802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.992288+0000","last_change":"2024-01-11T09:27:21.989188+0000","last_active":"2024-01-11T09:27:21.992288+0000","last_peered":"2024-01-11T09:27:21.992288+0000","last_clean":"2024-01-11T09:27:21.992288+0000","last_became_active":"2024-01-11T09:27:21.988794+0000","last_became_peered":"2024-01-11T09:27:21.988794+0000","last_unstale":"2024-01-11T09:27:21.992288+0000","last_undegraded":"2024-01-11T09:27:21.992288+0000","last_fullsized":"2024-01-11T09:27:21.992288+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:05:36.862461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.005931+0000","last_change":"2024-01-11T09:27:22.005931+0000","last_active":"2024-01-11T09:27:22.005931+0000","last_peered":"2024-01-11T09:27:22.005931+0000","last_clean":"2024-01-11T09:27:22.005931+0000","last_became_active":"2024-01-11T09:27:22.005645+0000","last_became_peered":"2024-01-11T09:27:22.005645+0000","last_unstale":"2024-01-11T09:27:22.005931+0000","last_undegraded":"2024-01-11T09:27:22.005931+0000","last_fullsized":"2024-01-11T09:27:22.005931+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:56:38.205676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"180'4","reported_seq":13,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.008494+0000","last_change":"2024-01-11T09:27:20.979608+0000","last_active":"2024-01-11T09:27:22.008494+0000","last_peered":"2024-01-11T09:27:22.008494+0000","last_clean":"2024-01-11T09:27:22.008494+0000","last_became_active":"2024-01-11T09:27:20.979301+0000","last_became_peered":"2024-01-11T09:27:20.979301+0000","last_unstale":"2024-01-11T09:27:22.008494+0000","last_undegraded":"2024-01-11T09:27:22.008494+0000","last_fullsized":"2024-01-11T09:27:22.008494+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:49:12.315817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.992111+0000","last_change":"2024-01-11T09:27:21.985098+0000","last_active":"2024-01-11T09:27:21.992111+0000","last_peered":"2024-01-11T09:27:21.992111+0000","last_clean":"2024-01-11T09:27:21.992111+0000","last_became_active":"2024-01-11T09:27:21.983408+0000","last_became_peered":"2024-01-11T09:27:21.983408+0000","last_unstale":"2024-01-11T09:27:21.992111+0000","last_undegraded":"2024-01-11T09:27:21.992111+0000","last_fullsized":"2024-01-11T09:27:21.992111+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:26:58.930845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"180'1","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.999149+0000","last_change":"2024-01-11T09:27:20.980885+0000","last_active":"2024-01-11T09:27:21.999149+0000","last_peered":"2024-01-11T09:27:21.999149+0000","last_clean":"2024-01-11T09:27:21.999149+0000","last_became_active":"2024-01-11T09:27:20.980585+0000","last_became_peered":"2024-01-11T09:27:20.980585+0000","last_unstale":"2024-01-11T09:27:21.999149+0000","last_undegraded":"2024-01-11T09:27:21.999149+0000","last_fullsized":"2024-01-11T09:27:21.999149+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:03:36.370161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.993974+0000","last_change":"2024-01-11T09:27:21.993974+0000","last_active":"2024-01-11T09:27:21.993974+0000","last_peered":"2024-01-11T09:27:21.993974+0000","last_clean":"2024-01-11T09:27:21.993974+0000","last_became_active":"2024-01-11T09:27:21.991505+0000","last_became_peered":"2024-01-11T09:27:21.991505+0000","last_unstale":"2024-01-11T09:27:21.993974+0000","last_undegraded":"2024-01-11T09:27:21.993974+0000","last_fullsized":"2024-01-11T09:27:21.993974+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:19:54.081327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"180'2","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.001400+0000","last_change":"2024-01-11T09:27:20.980657+0000","last_active":"2024-01-11T09:27:22.001400+0000","last_peered":"2024-01-11T09:27:22.001400+0000","last_clean":"2024-01-11T09:27:22.001400+0000","last_became_active":"2024-01-11T09:27:20.980394+0000","last_became_peered":"2024-01-11T09:27:20.980394+0000","last_unstale":"2024-01-11T09:27:22.001400+0000","last_undegraded":"2024-01-11T09:27:22.001400+0000","last_fullsized":"2024-01-11T09:27:22.001400+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:35:14.539951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.997971+0000","last_change":"2024-01-11T09:27:21.987885+0000","last_active":"2024-01-11T09:27:21.997971+0000","last_peered":"2024-01-11T09:27:21.997971+0000","last_clean":"2024-01-11T09:27:21.997971+0000","last_became_active":"2024-01-11T09:27:21.987574+0000","last_became_peered":"2024-01-11T09:27:21.987574+0000","last_unstale":"2024-01-11T09:27:21.997971+0000","last_undegraded":"2024-01-11T09:27:21.997971+0000","last_fullsized":"2024-01-11T09:27:21.997971+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:09:18.445842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"180'2","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.002786+0000","last_change":"2024-01-11T09:27:20.982829+0000","last_active":"2024-01-11T09:27:22.002786+0000","last_peered":"2024-01-11T09:27:22.002786+0000","last_clean":"2024-01-11T09:27:22.002786+0000","last_became_active":"2024-01-11T09:27:20.982535+0000","last_became_peered":"2024-01-11T09:27:20.982535+0000","last_unstale":"2024-01-11T09:27:22.002786+0000","last_undegraded":"2024-01-11T09:27:22.002786+0000","last_fullsized":"2024-01-11T09:27:22.002786+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:17:27.038545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.992905+0000","last_change":"2024-01-11T09:27:21.992905+0000","last_active":"2024-01-11T09:27:21.992905+0000","last_peered":"2024-01-11T09:27:21.992905+0000","last_clean":"2024-01-11T09:27:21.992905+0000","last_became_active":"2024-01-11T09:27:21.992610+0000","last_became_peered":"2024-01-11T09:27:21.992610+0000","last_unstale":"2024-01-11T09:27:21.992905+0000","last_undegraded":"2024-01-11T09:27:21.992905+0000","last_fullsized":"2024-01-11T09:27:21.992905+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:36:27.578314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"180'1","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.000995+0000","last_change":"2024-01-11T09:27:20.980015+0000","last_active":"2024-01-11T09:27:22.000995+0000","last_peered":"2024-01-11T09:27:22.000995+0000","last_clean":"2024-01-11T09:27:22.000995+0000","last_became_active":"2024-01-11T09:27:20.979717+0000","last_became_peered":"2024-01-11T09:27:20.979717+0000","last_unstale":"2024-01-11T09:27:22.000995+0000","last_undegraded":"2024-01-11T09:27:22.000995+0000","last_fullsized":"2024-01-11T09:27:22.000995+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:57:16.382699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.997641+0000","last_change":"2024-01-11T09:27:21.997641+0000","last_active":"2024-01-11T09:27:21.997641+0000","last_peered":"2024-01-11T09:27:21.997641+0000","last_clean":"2024-01-11T09:27:21.997641+0000","last_became_active":"2024-01-11T09:27:21.997370+0000","last_became_peered":"2024-01-11T09:27:21.997370+0000","last_unstale":"2024-01-11T09:27:21.997641+0000","last_undegraded":"2024-01-11T09:27:21.997641+0000","last_fullsized":"2024-01-11T09:27:21.997641+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:03:37.377064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"180'1","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.978667+0000","last_change":"2024-01-11T09:27:20.973740+0000","last_active":"2024-01-11T09:27:21.978667+0000","last_peered":"2024-01-11T09:27:21.978667+0000","last_clean":"2024-01-11T09:27:21.978667+0000","last_became_active":"2024-01-11T09:27:20.973327+0000","last_became_peered":"2024-01-11T09:27:20.973327+0000","last_unstale":"2024-01-11T09:27:21.978667+0000","last_undegraded":"2024-01-11T09:27:21.978667+0000","last_fullsized":"2024-01-11T09:27:21.978667+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:55:37.939126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.997856+0000","last_change":"2024-01-11T09:27:21.997856+0000","last_active":"2024-01-11T09:27:21.997856+0000","last_peered":"2024-01-11T09:27:21.997856+0000","last_clean":"2024-01-11T09:27:21.997856+0000","last_became_active":"2024-01-11T09:27:21.997546+0000","last_became_peered":"2024-01-11T09:27:21.997546+0000","last_unstale":"2024-01-11T09:27:21.997856+0000","last_undegraded":"2024-01-11T09:27:21.997856+0000","last_fullsized":"2024-01-11T09:27:21.997856+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:46:41.232840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.982264+0000","last_change":"2024-01-11T09:27:20.978135+0000","last_active":"2024-01-11T09:27:20.982264+0000","last_peered":"2024-01-11T09:27:20.982264+0000","last_clean":"2024-01-11T09:27:20.982264+0000","last_became_active":"2024-01-11T09:27:20.977589+0000","last_became_peered":"2024-01-11T09:27:20.977589+0000","last_unstale":"2024-01-11T09:27:20.982264+0000","last_undegraded":"2024-01-11T09:27:20.982264+0000","last_fullsized":"2024-01-11T09:27:20.982264+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:05:35.855559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.998182+0000","last_change":"2024-01-11T09:27:21.998182+0000","last_active":"2024-01-11T09:27:21.998182+0000","last_peered":"2024-01-11T09:27:21.998182+0000","last_clean":"2024-01-11T09:27:21.998182+0000","last_became_active":"2024-01-11T09:27:21.997719+0000","last_became_peered":"2024-01-11T09:27:21.997719+0000","last_unstale":"2024-01-11T09:27:21.998182+0000","last_undegraded":"2024-01-11T09:27:21.998182+0000","last_fullsized":"2024-01-11T09:27:21.998182+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:58:10.141123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.972025+0000","last_change":"2024-01-11T09:27:20.970086+0000","last_active":"2024-01-11T09:27:20.972025+0000","last_peered":"2024-01-11T09:27:20.972025+0000","last_clean":"2024-01-11T09:27:20.972025+0000","last_became_active":"2024-01-11T09:27:20.969789+0000","last_became_peered":"2024-01-11T09:27:20.969789+0000","last_unstale":"2024-01-11T09:27:20.972025+0000","last_undegraded":"2024-01-11T09:27:20.972025+0000","last_fullsized":"2024-01-11T09:27:20.972025+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:30:31.681173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.006251+0000","last_change":"2024-01-11T09:27:22.001624+0000","last_active":"2024-01-11T09:27:22.006251+0000","last_peered":"2024-01-11T09:27:22.006251+0000","last_clean":"2024-01-11T09:27:22.006251+0000","last_became_active":"2024-01-11T09:27:22.000534+0000","last_became_peered":"2024-01-11T09:27:22.000534+0000","last_unstale":"2024-01-11T09:27:22.006251+0000","last_undegraded":"2024-01-11T09:27:22.006251+0000","last_fullsized":"2024-01-11T09:27:22.006251+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:55:25.729750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.976946+0000","last_change":"2024-01-11T09:27:20.972671+0000","last_active":"2024-01-11T09:27:20.976946+0000","last_peered":"2024-01-11T09:27:20.976946+0000","last_clean":"2024-01-11T09:27:20.976946+0000","last_became_active":"2024-01-11T09:27:20.972420+0000","last_became_peered":"2024-01-11T09:27:20.972420+0000","last_unstale":"2024-01-11T09:27:20.976946+0000","last_undegraded":"2024-01-11T09:27:20.976946+0000","last_fullsized":"2024-01-11T09:27:20.976946+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:46:55.573874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.994255+0000","last_change":"2024-01-11T09:27:21.991257+0000","last_active":"2024-01-11T09:27:21.994255+0000","last_peered":"2024-01-11T09:27:21.994255+0000","last_clean":"2024-01-11T09:27:21.994255+0000","last_became_active":"2024-01-11T09:27:21.990995+0000","last_became_peered":"2024-01-11T09:27:21.990995+0000","last_unstale":"2024-01-11T09:27:21.994255+0000","last_undegraded":"2024-01-11T09:27:21.994255+0000","last_fullsized":"2024-01-11T09:27:21.994255+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:46:13.646100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"180'3","reported_seq":12,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.008689+0000","last_change":"2024-01-11T09:27:20.982186+0000","last_active":"2024-01-11T09:27:22.008689+0000","last_peered":"2024-01-11T09:27:22.008689+0000","last_clean":"2024-01-11T09:27:22.008689+0000","last_became_active":"2024-01-11T09:27:20.981893+0000","last_became_peered":"2024-01-11T09:27:20.981893+0000","last_unstale":"2024-01-11T09:27:22.008689+0000","last_undegraded":"2024-01-11T09:27:22.008689+0000","last_fullsized":"2024-01-11T09:27:22.008689+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:57:16.382699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.989443+0000","last_change":"2024-01-11T09:27:21.980001+0000","last_active":"2024-01-11T09:27:21.989443+0000","last_peered":"2024-01-11T09:27:21.989443+0000","last_clean":"2024-01-11T09:27:21.989443+0000","last_became_active":"2024-01-11T09:27:21.979253+0000","last_became_peered":"2024-01-11T09:27:21.979253+0000","last_unstale":"2024-01-11T09:27:21.989443+0000","last_undegraded":"2024-01-11T09:27:21.989443+0000","last_fullsized":"2024-01-11T09:27:21.989443+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:33:53.686857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"180'6","reported_seq":16,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.013607+0000","last_change":"2024-01-11T09:27:20.981882+0000","last_active":"2024-01-11T09:27:22.013607+0000","last_peered":"2024-01-11T09:27:22.013607+0000","last_clean":"2024-01-11T09:27:22.013607+0000","last_became_active":"2024-01-11T09:27:20.980779+0000","last_became_peered":"2024-01-11T09:27:20.980779+0000","last_unstale":"2024-01-11T09:27:22.013607+0000","last_undegraded":"2024-01-11T09:27:22.013607+0000","last_fullsized":"2024-01-11T09:27:22.013607+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:35:09.045394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.006217+0000","last_change":"2024-01-11T09:27:21.999904+0000","last_active":"2024-01-11T09:27:22.006217+0000","last_peered":"2024-01-11T09:27:22.006217+0000","last_clean":"2024-01-11T09:27:22.006217+0000","last_became_active":"2024-01-11T09:27:21.998990+0000","last_became_peered":"2024-01-11T09:27:21.998990+0000","last_unstale":"2024-01-11T09:27:22.006217+0000","last_undegraded":"2024-01-11T09:27:22.006217+0000","last_fullsized":"2024-01-11T09:27:22.006217+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:03:04.965882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"180'2","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.001234+0000","last_change":"2024-01-11T09:27:20.982874+0000","last_active":"2024-01-11T09:27:22.001234+0000","last_peered":"2024-01-11T09:27:22.001234+0000","last_clean":"2024-01-11T09:27:22.001234+0000","last_became_active":"2024-01-11T09:27:20.982034+0000","last_became_peered":"2024-01-11T09:27:20.982034+0000","last_unstale":"2024-01-11T09:27:22.001234+0000","last_undegraded":"2024-01-11T09:27:22.001234+0000","last_fullsized":"2024-01-11T09:27:22.001234+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:51:34.857778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.999271+0000","last_change":"2024-01-11T09:27:21.999271+0000","last_active":"2024-01-11T09:27:21.999271+0000","last_peered":"2024-01-11T09:27:21.999271+0000","last_clean":"2024-01-11T09:27:21.999271+0000","last_became_active":"2024-01-11T09:27:21.998817+0000","last_became_peered":"2024-01-11T09:27:21.998817+0000","last_unstale":"2024-01-11T09:27:21.999271+0000","last_undegraded":"2024-01-11T09:27:21.999271+0000","last_fullsized":"2024-01-11T09:27:21.999271+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:17:40.366748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.970709+0000","last_change":"2024-01-11T09:27:20.979292+0000","last_active":"2024-01-11T09:27:21.970709+0000","last_peered":"2024-01-11T09:27:21.970709+0000","last_clean":"2024-01-11T09:27:21.970709+0000","last_became_active":"2024-01-11T09:27:20.978722+0000","last_became_peered":"2024-01-11T09:27:20.978722+0000","last_unstale":"2024-01-11T09:27:21.970709+0000","last_undegraded":"2024-01-11T09:27:21.970709+0000","last_fullsized":"2024-01-11T09:27:21.970709+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:46:40.225937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.001264+0000","last_change":"2024-01-11T09:27:22.001264+0000","last_active":"2024-01-11T09:27:22.001264+0000","last_peered":"2024-01-11T09:27:22.001264+0000","last_clean":"2024-01-11T09:27:22.001264+0000","last_became_active":"2024-01-11T09:27:22.000934+0000","last_became_peered":"2024-01-11T09:27:22.000934+0000","last_unstale":"2024-01-11T09:27:22.001264+0000","last_undegraded":"2024-01-11T09:27:22.001264+0000","last_fullsized":"2024-01-11T09:27:22.001264+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:53:23.281467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.981699+0000","last_change":"2024-01-11T09:27:20.978413+0000","last_active":"2024-01-11T09:27:20.981699+0000","last_peered":"2024-01-11T09:27:20.981699+0000","last_clean":"2024-01-11T09:27:20.981699+0000","last_became_active":"2024-01-11T09:27:20.977865+0000","last_became_peered":"2024-01-11T09:27:20.977865+0000","last_unstale":"2024-01-11T09:27:20.981699+0000","last_undegraded":"2024-01-11T09:27:20.981699+0000","last_fullsized":"2024-01-11T09:27:20.981699+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:58:09.134220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.008025+0000","last_change":"2024-01-11T09:27:22.002406+0000","last_active":"2024-01-11T09:27:22.008025+0000","last_peered":"2024-01-11T09:27:22.008025+0000","last_clean":"2024-01-11T09:27:22.008025+0000","last_became_active":"2024-01-11T09:27:22.001428+0000","last_became_peered":"2024-01-11T09:27:22.001428+0000","last_unstale":"2024-01-11T09:27:22.008025+0000","last_undegraded":"2024-01-11T09:27:22.008025+0000","last_fullsized":"2024-01-11T09:27:22.008025+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:45:21.227961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"180'8","reported_seq":18,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.006760+0000","last_change":"2024-01-11T09:27:20.981746+0000","last_active":"2024-01-11T09:27:22.006760+0000","last_peered":"2024-01-11T09:27:22.006760+0000","last_clean":"2024-01-11T09:27:22.006760+0000","last_became_active":"2024-01-11T09:27:20.981051+0000","last_became_peered":"2024-01-11T09:27:20.981051+0000","last_unstale":"2024-01-11T09:27:22.006760+0000","last_undegraded":"2024-01-11T09:27:22.006760+0000","last_fullsized":"2024-01-11T09:27:22.006760+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:53:22.274565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.006799+0000","last_change":"2024-01-11T09:27:22.006799+0000","last_active":"2024-01-11T09:27:22.006799+0000","last_peered":"2024-01-11T09:27:22.006799+0000","last_clean":"2024-01-11T09:27:22.006799+0000","last_became_active":"2024-01-11T09:27:22.006523+0000","last_became_peered":"2024-01-11T09:27:22.006523+0000","last_unstale":"2024-01-11T09:27:22.006799+0000","last_undegraded":"2024-01-11T09:27:22.006799+0000","last_fullsized":"2024-01-11T09:27:22.006799+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:05.603829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"180'1","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.001157+0000","last_change":"2024-01-11T09:27:20.973383+0000","last_active":"2024-01-11T09:27:22.001157+0000","last_peered":"2024-01-11T09:27:22.001157+0000","last_clean":"2024-01-11T09:27:22.001157+0000","last_became_active":"2024-01-11T09:27:20.971987+0000","last_became_peered":"2024-01-11T09:27:20.971987+0000","last_unstale":"2024-01-11T09:27:22.001157+0000","last_undegraded":"2024-01-11T09:27:22.001157+0000","last_fullsized":"2024-01-11T09:27:22.001157+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:08.969875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.991461+0000","last_change":"2024-01-11T09:27:21.982420+0000","last_active":"2024-01-11T09:27:21.991461+0000","last_peered":"2024-01-11T09:27:21.991461+0000","last_clean":"2024-01-11T09:27:21.991461+0000","last_became_active":"2024-01-11T09:27:21.982156+0000","last_became_peered":"2024-01-11T09:27:21.982156+0000","last_unstale":"2024-01-11T09:27:21.991461+0000","last_undegraded":"2024-01-11T09:27:21.991461+0000","last_fullsized":"2024-01-11T09:27:21.991461+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:06:42.072322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"180'2","reported_seq":12,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.002903+0000","last_change":"2024-01-11T09:27:20.980188+0000","last_active":"2024-01-11T09:27:22.002903+0000","last_peered":"2024-01-11T09:27:22.002903+0000","last_clean":"2024-01-11T09:27:22.002903+0000","last_became_active":"2024-01-11T09:27:20.979436+0000","last_became_peered":"2024-01-11T09:27:20.979436+0000","last_unstale":"2024-01-11T09:27:22.002903+0000","last_undegraded":"2024-01-11T09:27:22.002903+0000","last_fullsized":"2024-01-11T09:27:22.002903+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:46:55.573874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.996278+0000","last_change":"2024-01-11T09:27:21.996278+0000","last_active":"2024-01-11T09:27:21.996278+0000","last_peered":"2024-01-11T09:27:21.996278+0000","last_clean":"2024-01-11T09:27:21.996278+0000","last_became_active":"2024-01-11T09:27:21.994477+0000","last_became_peered":"2024-01-11T09:27:21.994477+0000","last_unstale":"2024-01-11T09:27:21.996278+0000","last_undegraded":"2024-01-11T09:27:21.996278+0000","last_fullsized":"2024-01-11T09:27:21.996278+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:41:27.430029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"180'1","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.983228+0000","last_change":"2024-01-11T09:27:20.979609+0000","last_active":"2024-01-11T09:27:21.983228+0000","last_peered":"2024-01-11T09:27:21.983228+0000","last_clean":"2024-01-11T09:27:21.983228+0000","last_became_active":"2024-01-11T09:27:20.979259+0000","last_became_peered":"2024-01-11T09:27:20.979259+0000","last_unstale":"2024-01-11T09:27:21.983228+0000","last_undegraded":"2024-01-11T09:27:21.983228+0000","last_fullsized":"2024-01-11T09:27:21.983228+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:42:07.675900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.996977+0000","last_change":"2024-01-11T09:27:21.996977+0000","last_active":"2024-01-11T09:27:21.996977+0000","last_peered":"2024-01-11T09:27:21.996977+0000","last_clean":"2024-01-11T09:27:21.996977+0000","last_became_active":"2024-01-11T09:27:21.996652+0000","last_became_peered":"2024-01-11T09:27:21.996652+0000","last_unstale":"2024-01-11T09:27:21.996977+0000","last_undegraded":"2024-01-11T09:27:21.996977+0000","last_fullsized":"2024-01-11T09:27:21.996977+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:43:28.389359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.981876+0000","last_change":"2024-01-11T09:27:20.975587+0000","last_active":"2024-01-11T09:27:20.981876+0000","last_peered":"2024-01-11T09:27:20.981876+0000","last_clean":"2024-01-11T09:27:20.981876+0000","last_became_active":"2024-01-11T09:27:20.975315+0000","last_became_peered":"2024-01-11T09:27:20.975315+0000","last_unstale":"2024-01-11T09:27:20.981876+0000","last_undegraded":"2024-01-11T09:27:20.981876+0000","last_fullsized":"2024-01-11T09:27:20.981876+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:30:31.681173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.981865+0000","last_change":"2024-01-11T09:27:21.977667+0000","last_active":"2024-01-11T09:27:21.981865+0000","last_peered":"2024-01-11T09:27:21.981865+0000","last_clean":"2024-01-11T09:27:21.981865+0000","last_became_active":"2024-01-11T09:27:21.977361+0000","last_became_peered":"2024-01-11T09:27:21.977361+0000","last_unstale":"2024-01-11T09:27:21.981865+0000","last_undegraded":"2024-01-11T09:27:21.981865+0000","last_fullsized":"2024-01-11T09:27:21.981865+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:36:03.680138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"180'2","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.001978+0000","last_change":"2024-01-11T09:27:20.978118+0000","last_active":"2024-01-11T09:27:22.001978+0000","last_peered":"2024-01-11T09:27:22.001978+0000","last_clean":"2024-01-11T09:27:22.001978+0000","last_became_active":"2024-01-11T09:27:20.977783+0000","last_became_peered":"2024-01-11T09:27:20.977783+0000","last_unstale":"2024-01-11T09:27:22.001978+0000","last_undegraded":"2024-01-11T09:27:22.001978+0000","last_fullsized":"2024-01-11T09:27:22.001978+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:41:39.741630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.992296+0000","last_change":"2024-01-11T09:27:21.992296+0000","last_active":"2024-01-11T09:27:21.992296+0000","last_peered":"2024-01-11T09:27:21.992296+0000","last_clean":"2024-01-11T09:27:21.992296+0000","last_became_active":"2024-01-11T09:27:21.992013+0000","last_became_peered":"2024-01-11T09:27:21.992013+0000","last_unstale":"2024-01-11T09:27:21.992296+0000","last_undegraded":"2024-01-11T09:27:21.992296+0000","last_fullsized":"2024-01-11T09:27:21.992296+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:53:56.792066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.982312+0000","last_change":"2024-01-11T09:27:20.977522+0000","last_active":"2024-01-11T09:27:20.982312+0000","last_peered":"2024-01-11T09:27:20.982312+0000","last_clean":"2024-01-11T09:27:20.982312+0000","last_became_active":"2024-01-11T09:27:20.977258+0000","last_became_peered":"2024-01-11T09:27:20.977258+0000","last_unstale":"2024-01-11T09:27:20.982312+0000","last_undegraded":"2024-01-11T09:27:20.982312+0000","last_fullsized":"2024-01-11T09:27:20.982312+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:32:14.769582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.000639+0000","last_change":"2024-01-11T09:27:22.000639+0000","last_active":"2024-01-11T09:27:22.000639+0000","last_peered":"2024-01-11T09:27:22.000639+0000","last_clean":"2024-01-11T09:27:22.000639+0000","last_became_active":"2024-01-11T09:27:22.000059+0000","last_became_peered":"2024-01-11T09:27:22.000059+0000","last_unstale":"2024-01-11T09:27:22.000639+0000","last_undegraded":"2024-01-11T09:27:22.000639+0000","last_fullsized":"2024-01-11T09:27:22.000639+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:55:25.729750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.981679+0000","last_change":"2024-01-11T09:27:20.978225+0000","last_active":"2024-01-11T09:27:20.981679+0000","last_peered":"2024-01-11T09:27:20.981679+0000","last_clean":"2024-01-11T09:27:20.981679+0000","last_became_active":"2024-01-11T09:27:20.977486+0000","last_became_peered":"2024-01-11T09:27:20.977486+0000","last_unstale":"2024-01-11T09:27:20.981679+0000","last_undegraded":"2024-01-11T09:27:20.981679+0000","last_fullsized":"2024-01-11T09:27:20.981679+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:14:58.043943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.998008+0000","last_change":"2024-01-11T09:27:21.998008+0000","last_active":"2024-01-11T09:27:21.998008+0000","last_peered":"2024-01-11T09:27:21.998008+0000","last_clean":"2024-01-11T09:27:21.998008+0000","last_became_active":"2024-01-11T09:27:21.997325+0000","last_became_peered":"2024-01-11T09:27:21.997325+0000","last_unstale":"2024-01-11T09:27:21.998008+0000","last_undegraded":"2024-01-11T09:27:21.998008+0000","last_fullsized":"2024-01-11T09:27:21.998008+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:41:40.748533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.982108+0000","last_change":"2024-01-11T09:27:20.976842+0000","last_active":"2024-01-11T09:27:20.982108+0000","last_peered":"2024-01-11T09:27:20.982108+0000","last_clean":"2024-01-11T09:27:20.982108+0000","last_became_active":"2024-01-11T09:27:20.976581+0000","last_became_peered":"2024-01-11T09:27:20.976581+0000","last_unstale":"2024-01-11T09:27:20.982108+0000","last_undegraded":"2024-01-11T09:27:20.982108+0000","last_fullsized":"2024-01-11T09:27:20.982108+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:34:52.519030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.999428+0000","last_change":"2024-01-11T09:27:21.998616+0000","last_active":"2024-01-11T09:27:21.999428+0000","last_peered":"2024-01-11T09:27:21.999428+0000","last_clean":"2024-01-11T09:27:21.999428+0000","last_became_active":"2024-01-11T09:27:21.995870+0000","last_became_peered":"2024-01-11T09:27:21.995870+0000","last_unstale":"2024-01-11T09:27:21.999428+0000","last_undegraded":"2024-01-11T09:27:21.999428+0000","last_fullsized":"2024-01-11T09:27:21.999428+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:04:20.823816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"180'2","reported_seq":12,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.005486+0000","last_change":"2024-01-11T09:27:20.969327+0000","last_active":"2024-01-11T09:27:22.005486+0000","last_peered":"2024-01-11T09:27:22.005486+0000","last_clean":"2024-01-11T09:27:22.005486+0000","last_became_active":"2024-01-11T09:27:20.969056+0000","last_became_peered":"2024-01-11T09:27:20.969056+0000","last_unstale":"2024-01-11T09:27:22.005486+0000","last_undegraded":"2024-01-11T09:27:22.005486+0000","last_fullsized":"2024-01-11T09:27:22.005486+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:05:05.483230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.996149+0000","last_change":"2024-01-11T09:27:21.996149+0000","last_active":"2024-01-11T09:27:21.996149+0000","last_peered":"2024-01-11T09:27:21.996149+0000","last_clean":"2024-01-11T09:27:21.996149+0000","last_became_active":"2024-01-11T09:27:21.995900+0000","last_became_peered":"2024-01-11T09:27:21.995900+0000","last_unstale":"2024-01-11T09:27:21.996149+0000","last_undegraded":"2024-01-11T09:27:21.996149+0000","last_fullsized":"2024-01-11T09:27:21.996149+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:53:41.444129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.974295+0000","last_change":"2024-01-11T09:27:20.977152+0000","last_active":"2024-01-11T09:27:21.974295+0000","last_peered":"2024-01-11T09:27:21.974295+0000","last_clean":"2024-01-11T09:27:21.974295+0000","last_became_active":"2024-01-11T09:27:20.976589+0000","last_became_peered":"2024-01-11T09:27:20.976589+0000","last_unstale":"2024-01-11T09:27:21.974295+0000","last_undegraded":"2024-01-11T09:27:21.974295+0000","last_fullsized":"2024-01-11T09:27:21.974295+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:05:54.863945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.001009+0000","last_change":"2024-01-11T09:27:22.001009+0000","last_active":"2024-01-11T09:27:22.001009+0000","last_peered":"2024-01-11T09:27:22.001009+0000","last_clean":"2024-01-11T09:27:22.001009+0000","last_became_active":"2024-01-11T09:27:22.000478+0000","last_became_peered":"2024-01-11T09:27:22.000478+0000","last_unstale":"2024-01-11T09:27:22.001009+0000","last_undegraded":"2024-01-11T09:27:22.001009+0000","last_fullsized":"2024-01-11T09:27:22.001009+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:36:03.680138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.977794+0000","last_change":"2024-01-11T09:27:20.972947+0000","last_active":"2024-01-11T09:27:20.977794+0000","last_peered":"2024-01-11T09:27:20.977794+0000","last_clean":"2024-01-11T09:27:20.977794+0000","last_became_active":"2024-01-11T09:27:20.972716+0000","last_became_peered":"2024-01-11T09:27:20.972716+0000","last_unstale":"2024-01-11T09:27:20.977794+0000","last_undegraded":"2024-01-11T09:27:20.977794+0000","last_fullsized":"2024-01-11T09:27:20.977794+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:03:03.958979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.992469+0000","last_change":"2024-01-11T09:27:21.988207+0000","last_active":"2024-01-11T09:27:21.992469+0000","last_peered":"2024-01-11T09:27:21.992469+0000","last_clean":"2024-01-11T09:27:21.992469+0000","last_became_active":"2024-01-11T09:27:21.987891+0000","last_became_peered":"2024-01-11T09:27:21.987891+0000","last_unstale":"2024-01-11T09:27:21.992469+0000","last_undegraded":"2024-01-11T09:27:21.992469+0000","last_fullsized":"2024-01-11T09:27:21.992469+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:49:13.322719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.970796+0000","last_change":"2024-01-11T09:27:20.969788+0000","last_active":"2024-01-11T09:27:21.970796+0000","last_peered":"2024-01-11T09:27:21.970796+0000","last_clean":"2024-01-11T09:27:21.970796+0000","last_became_active":"2024-01-11T09:27:20.969551+0000","last_became_peered":"2024-01-11T09:27:20.969551+0000","last_unstale":"2024-01-11T09:27:21.970796+0000","last_undegraded":"2024-01-11T09:27:21.970796+0000","last_fullsized":"2024-01-11T09:27:21.970796+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:40:29.761181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.006238+0000","last_change":"2024-01-11T09:27:22.001242+0000","last_active":"2024-01-11T09:27:22.006238+0000","last_peered":"2024-01-11T09:27:22.006238+0000","last_clean":"2024-01-11T09:27:22.006238+0000","last_became_active":"2024-01-11T09:27:21.996076+0000","last_became_peered":"2024-01-11T09:27:21.996076+0000","last_unstale":"2024-01-11T09:27:22.006238+0000","last_undegraded":"2024-01-11T09:27:22.006238+0000","last_fullsized":"2024-01-11T09:27:22.006238+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:58:50.682338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.973327+0000","last_change":"2024-01-11T09:27:20.979567+0000","last_active":"2024-01-11T09:27:21.973327+0000","last_peered":"2024-01-11T09:27:21.973327+0000","last_clean":"2024-01-11T09:27:21.973327+0000","last_became_active":"2024-01-11T09:27:20.979231+0000","last_became_peered":"2024-01-11T09:27:20.979231+0000","last_unstale":"2024-01-11T09:27:21.973327+0000","last_undegraded":"2024-01-11T09:27:21.973327+0000","last_fullsized":"2024-01-11T09:27:21.973327+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:38:03.932159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.002798+0000","last_change":"2024-01-11T09:27:21.999756+0000","last_active":"2024-01-11T09:27:22.002798+0000","last_peered":"2024-01-11T09:27:22.002798+0000","last_clean":"2024-01-11T09:27:22.002798+0000","last_became_active":"2024-01-11T09:27:21.999246+0000","last_became_peered":"2024-01-11T09:27:21.999246+0000","last_unstale":"2024-01-11T09:27:22.002798+0000","last_undegraded":"2024-01-11T09:27:22.002798+0000","last_fullsized":"2024-01-11T09:27:22.002798+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:34:53.525932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.981266+0000","last_change":"2024-01-11T09:27:20.976708+0000","last_active":"2024-01-11T09:27:20.981266+0000","last_peered":"2024-01-11T09:27:20.981266+0000","last_clean":"2024-01-11T09:27:20.981266+0000","last_became_active":"2024-01-11T09:27:20.976346+0000","last_became_peered":"2024-01-11T09:27:20.976346+0000","last_unstale":"2024-01-11T09:27:20.981266+0000","last_undegraded":"2024-01-11T09:27:20.981266+0000","last_fullsized":"2024-01-11T09:27:20.981266+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:23:58.946042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.989268+0000","last_change":"2024-01-11T09:27:21.989268+0000","last_active":"2024-01-11T09:27:21.989268+0000","last_peered":"2024-01-11T09:27:21.989268+0000","last_clean":"2024-01-11T09:27:21.989268+0000","last_became_active":"2024-01-11T09:27:21.988991+0000","last_became_peered":"2024-01-11T09:27:21.988991+0000","last_unstale":"2024-01-11T09:27:21.989268+0000","last_undegraded":"2024-01-11T09:27:21.989268+0000","last_fullsized":"2024-01-11T09:27:21.989268+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:22:21.100031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.977813+0000","last_change":"2024-01-11T09:27:20.975000+0000","last_active":"2024-01-11T09:27:20.977813+0000","last_peered":"2024-01-11T09:27:20.977813+0000","last_clean":"2024-01-11T09:27:20.977813+0000","last_became_active":"2024-01-11T09:27:20.974711+0000","last_became_peered":"2024-01-11T09:27:20.974711+0000","last_unstale":"2024-01-11T09:27:20.977813+0000","last_undegraded":"2024-01-11T09:27:20.977813+0000","last_fullsized":"2024-01-11T09:27:20.977813+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:13:27.228772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.008096+0000","last_change":"2024-01-11T09:27:22.003519+0000","last_active":"2024-01-11T09:27:22.008096+0000","last_peered":"2024-01-11T09:27:22.008096+0000","last_clean":"2024-01-11T09:27:22.008096+0000","last_became_active":"2024-01-11T09:27:22.003257+0000","last_became_peered":"2024-01-11T09:27:22.003257+0000","last_unstale":"2024-01-11T09:27:22.008096+0000","last_undegraded":"2024-01-11T09:27:22.008096+0000","last_fullsized":"2024-01-11T09:27:22.008096+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:19:54.081327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.973087+0000","last_change":"2024-01-11T09:27:20.977340+0000","last_active":"2024-01-11T09:27:21.973087+0000","last_peered":"2024-01-11T09:27:21.973087+0000","last_clean":"2024-01-11T09:27:21.973087+0000","last_became_active":"2024-01-11T09:27:20.977053+0000","last_became_peered":"2024-01-11T09:27:20.977053+0000","last_unstale":"2024-01-11T09:27:21.973087+0000","last_undegraded":"2024-01-11T09:27:21.973087+0000","last_fullsized":"2024-01-11T09:27:21.973087+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:41:26.423126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.008240+0000","last_change":"2024-01-11T09:27:22.005066+0000","last_active":"2024-01-11T09:27:22.008240+0000","last_peered":"2024-01-11T09:27:22.008240+0000","last_clean":"2024-01-11T09:27:22.008240+0000","last_became_active":"2024-01-11T09:27:22.004011+0000","last_became_peered":"2024-01-11T09:27:22.004011+0000","last_unstale":"2024-01-11T09:27:22.008240+0000","last_undegraded":"2024-01-11T09:27:22.008240+0000","last_fullsized":"2024-01-11T09:27:22.008240+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:17:40.366748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.975052+0000","last_change":"2024-01-11T09:27:20.982648+0000","last_active":"2024-01-11T09:27:21.975052+0000","last_peered":"2024-01-11T09:27:21.975052+0000","last_clean":"2024-01-11T09:27:21.975052+0000","last_became_active":"2024-01-11T09:27:20.982380+0000","last_became_peered":"2024-01-11T09:27:20.982380+0000","last_unstale":"2024-01-11T09:27:21.975052+0000","last_undegraded":"2024-01-11T09:27:21.975052+0000","last_fullsized":"2024-01-11T09:27:21.975052+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:04.596926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"171'311","reported_seq":976,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.965872+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:27:20.965872+0000","last_peered":"2024-01-11T09:27:20.965872+0000","last_clean":"2024-01-11T09:27:20.965872+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:27:20.965872+0000","last_undegraded":"2024-01-11T09:27:20.965872+0000","last_fullsized":"2024-01-11T09:27:20.965872+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1808,"num_read_kb":3778,"num_write":511,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1808,"num_read_kb":3778,"num_write":511,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":311,"ondisk_log_size":311,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575760,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312380,"kb_used_data":2804,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059460,"statfs":{"total":96636764160,"available":96316887040,"internally_reserved":0,"allocated":2871296,"data_stored":1657752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.72399999999999998,"15min":0.72599999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.91500000000000004,"5min":0.98199999999999998,"15min":0.99299999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.72199999999999998,"15min":0.72999999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.89200000000000002,"5min":0.99299999999999999,"15min":1.028},"last":0.76000000000000001}]},{"osd":1,"last update":"Thu Jan 11 09:26:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.71299999999999997,"15min":0.71799999999999997},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.42399999999999999},"max":{"1min":0.81200000000000006,"5min":1.0009999999999999,"15min":1.034},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.748,"15min":0.749},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.90900000000000003,"5min":0.96099999999999997,"15min":1.0409999999999999},"last":0.80800000000000005}]},{"osd":2,"last update":"Thu Jan 11 09:26:51 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.73399999999999999,"15min":0.748},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.46899999999999997},"max":{"1min":0.89100000000000001,"5min":1.0089999999999999,"15min":1.0880000000000001},"last":0.63},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.72699999999999998,"15min":0.73399999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50800000000000001},"max":{"1min":0.84599999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.68799999999999994}]},{"osd":3,"last update":"Thu Jan 11 09:26:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.99399999999999999,"5min":1.046,"15min":1.056},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.75,"15min":0.76100000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.92400000000000004,"5min":1.006,"15min":1.012},"last":0.752}]},{"osd":4,"last update":"Thu Jan 11 09:26:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59899999999999998,"15min":0.60199999999999998},"min":{"1min":0.47299999999999998,"5min":0.45300000000000001,"15min":0.40600000000000003},"max":{"1min":0.871,"5min":0.871,"15min":1.004},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64100000000000001,"15min":0.64300000000000002},"min":{"1min":0.433,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0009999999999999,"5min":1.016,"15min":1.0940000000000001},"last":0.78800000000000003}]},{"osd":5,"last update":"Thu Jan 11 09:26:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.63700000000000001,"15min":0.64200000000000002},"min":{"1min":0.503,"5min":0.433,"15min":0.433},"max":{"1min":0.84999999999999998,"5min":0.93400000000000005,"15min":1.0309999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.68600000000000005,"15min":0.67900000000000005},"min":{"1min":0.56000000000000005,"5min":0.41999999999999998,"15min":0.41299999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.0680000000000001},"last":0.71199999999999997}]},{"osd":6,"last update":"Thu Jan 11 09:26:51 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.76800000000000002,"15min":0.77400000000000002},"min":{"1min":0.59999999999999998,"5min":0.50900000000000001,"15min":0.47899999999999998},"max":{"1min":0.88100000000000001,"5min":0.97399999999999998,"15min":1.1200000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.749,"15min":0.754},"min":{"1min":0.46400000000000002,"5min":0.44600000000000001,"15min":0.40100000000000002},"max":{"1min":1.0389999999999999,"5min":1.073,"15min":1.073},"last":0.82699999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575759,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307684,"kb_used_data":2204,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064156,"statfs":{"total":96636764160,"available":96321695744,"internally_reserved":0,"allocated":2256896,"data_stored":1065156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.70599999999999996,"15min":0.70499999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.40500000000000003},"max":{"1min":0.91200000000000003,"5min":0.98099999999999998,"15min":1.095},"last":0.748},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.71799999999999997,"15min":0.71999999999999997},"min":{"1min":0.53500000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.92700000000000005,"5min":0.97899999999999998,"15min":1.0489999999999999},"last":0.93200000000000005}]},{"osd":1,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.72599999999999998,"15min":0.72699999999999998},"min":{"1min":0.54100000000000004,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93999999999999995,"5min":0.96499999999999997,"15min":1.077},"last":0.87},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71399999999999997,"15min":0.71899999999999997},"min":{"1min":0.49099999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.94299999999999995,"5min":0.96999999999999997,"15min":1.01},"last":0.80000000000000004}]},{"osd":2,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.74399999999999999,"15min":0.73599999999999999},"min":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.49399999999999999},"max":{"1min":1.006,"5min":1.0129999999999999,"15min":1.0309999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72999999999999998,"15min":0.75},"min":{"1min":0.54500000000000004,"5min":0.51600000000000001,"15min":0.505},"max":{"1min":0.89300000000000002,"5min":0.94599999999999995,"15min":1.161},"last":0.83499999999999996}]},{"osd":3,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76200000000000001,"15min":0.76000000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.47499999999999998},"max":{"1min":0.91300000000000003,"5min":0.999,"15min":1.0329999999999999},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76300000000000001,"15min":0.76700000000000002},"min":{"1min":0.57199999999999995,"5min":0.53700000000000003,"15min":0.503},"max":{"1min":0.91300000000000003,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.95699999999999996}]},{"osd":4,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.45900000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76100000000000001,"5min":0.82999999999999996,"15min":1.0469999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85799999999999998,"5min":0.97199999999999998,"15min":1.0569999999999999},"last":0.70399999999999996}]},{"osd":5,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.67600000000000005,"15min":0.66700000000000004},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.39200000000000002},"max":{"1min":0.89700000000000002,"5min":1.032,"15min":1.032},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.69999999999999996,"15min":0.71199999999999997},"min":{"1min":0.441,"5min":0.441,"15min":0.41699999999999998},"max":{"1min":0.93300000000000005,"5min":1.052,"15min":1.1299999999999999},"last":0.55100000000000005}]},{"osd":7,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.71099999999999997,"15min":0.71899999999999997},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.435},"max":{"1min":0.96799999999999997,"5min":1.034,"15min":1.1080000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.75800000000000001,"15min":0.75900000000000001},"min":{"1min":0.501,"5min":0.5,"15min":0.47299999999999998},"max":{"1min":0.95499999999999996,"5min":1.081,"15min":1.0840000000000001},"last":0.73599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575761,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307684,"kb_used_data":2204,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064156,"statfs":{"total":96636764160,"available":96321695744,"internally_reserved":0,"allocated":2256896,"data_stored":1065168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.51100000000000001,"15min":0.51800000000000002},"min":{"1min":0.33200000000000002,"5min":0.317,"15min":0.30199999999999999},"max":{"1min":0.71499999999999997,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.442},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39900000000000002},"max":{"1min":0.74099999999999999,"5min":0.81399999999999995,"15min":0.90200000000000002},"last":0.58499999999999996}]},{"osd":2,"last update":"Thu Jan 11 09:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.495,"15min":0.505},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.83699999999999997},"last":0.622},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.36899999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.91900000000000004},"last":0.495}]},{"osd":3,"last update":"Thu Jan 11 09:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.64400000000000002,"15min":0.64600000000000002},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84699999999999998,"5min":1.002,"15min":1.002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58499999999999996,"15min":0.58399999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.40500000000000003},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.55500000000000005}]},{"osd":4,"last update":"Thu Jan 11 09:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.81799999999999995,"15min":0.81000000000000005},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.46999999999999997},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.85999999999999999,"15min":0.86299999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.085,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.81899999999999995}]},{"osd":5,"last update":"Thu Jan 11 09:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.81299999999999994,"15min":0.81100000000000005},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.55400000000000005},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.72699999999999998}]},{"osd":6,"last update":"Thu Jan 11 09:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.84999999999999998,"15min":0.84199999999999997},"min":{"1min":0.629,"5min":0.61599999999999999,"15min":0.57799999999999996},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.85299999999999998,"15min":0.84099999999999997},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.54400000000000004},"max":{"1min":1.0329999999999999,"5min":1.1499999999999999,"15min":1.2529999999999999},"last":0.88900000000000001}]},{"osd":7,"last update":"Thu Jan 11 09:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.875,"15min":0.86399999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.153,"5min":1.2130000000000001,"15min":1.218},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.84999999999999998,"15min":0.84699999999999998},"min":{"1min":0.626,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.1839999999999999,"5min":1.1950000000000001,"15min":1.244},"last":0.68799999999999994}]}]},{"osd":0,"up_from":13,"seq":55834575760,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308260,"kb_used_data":2780,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063580,"statfs":{"total":96636764160,"available":96321105920,"internally_reserved":0,"allocated":2846720,"data_stored":1655502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.52400000000000002,"15min":0.52000000000000002},"min":{"1min":0.311,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.70199999999999996,"5min":0.751,"15min":0.79600000000000004},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60299999999999998,"15min":0.59699999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.42499999999999999},"max":{"1min":0.72399999999999998,"5min":0.83699999999999997,"15min":0.94599999999999995},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55300000000000005,"15min":0.55400000000000005},"min":{"1min":0.41399999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.80600000000000005,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55200000000000005,"15min":0.57499999999999996},"min":{"1min":0.41399999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89900000000000002},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70599999999999996,"5min":0.79400000000000004,"15min":0.90600000000000003},"last":0.496},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63400000000000001,"15min":0.64300000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.79900000000000004,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.55300000000000005}]},{"osd":4,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79200000000000004,"15min":0.81699999999999995},"min":{"1min":0.59599999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.075,"5min":1.075,"15min":1.4239999999999999},"last":1.036},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.80800000000000005,"15min":0.81200000000000006},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.1659999999999999,"5min":1.2050000000000001,"15min":1.2749999999999999},"last":1.0740000000000001}]},{"osd":5,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.81599999999999995,"15min":0.83299999999999996},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.57399999999999995},"max":{"1min":1.0249999999999999,"5min":1.1140000000000001,"15min":1.391},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.84199999999999997,"15min":0.84099999999999997},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.151,"5min":1.151,"15min":1.484},"last":0.93200000000000005}]},{"osd":6,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.82599999999999996,"15min":0.82899999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.353},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.85399999999999998,"15min":0.86399999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.3069999999999999},"last":0.89400000000000002}]},{"osd":7,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.86799999999999999,"15min":0.875},"min":{"1min":0.60899999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.125,"5min":1.2010000000000001,"15min":1.246},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.872,"15min":0.86399999999999999},"min":{"1min":0.67300000000000004,"5min":0.63600000000000001,"15min":0.59999999999999998},"max":{"1min":1.048,"5min":1.157,"15min":1.272},"last":0.86899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575759,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311780,"kb_used_data":2204,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060060,"statfs":{"total":96636764160,"available":96317501440,"internally_reserved":0,"allocated":2256896,"data_stored":1065224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.497,"15min":0.5},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55100000000000005,"15min":0.55400000000000005},"min":{"1min":0.34599999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.49299999999999999}]},{"osd":1,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.45100000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79700000000000004,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60899999999999999,"15min":0.60699999999999998},"min":{"1min":0.41199999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.82699999999999996,"5min":0.93600000000000005,"15min":1},"last":0.45300000000000001}]},{"osd":3,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54900000000000004,"15min":0.55700000000000005},"min":{"1min":0.40400000000000003,"5min":0.37,"15min":0.34799999999999998},"max":{"1min":0.79500000000000004,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.439},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65600000000000003,"15min":0.66100000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.57099999999999995}]},{"osd":4,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.80700000000000005,"15min":0.81399999999999995},"min":{"1min":0.68200000000000005,"5min":0.55300000000000005,"15min":0.54000000000000004},"max":{"1min":1.1220000000000001,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81299999999999994,"15min":0.79800000000000004},"min":{"1min":0.69199999999999995,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.036,"5min":1.391,"15min":1.391},"last":0.78800000000000003}]},{"osd":5,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.878,"15min":0.86899999999999999},"min":{"1min":0.73599999999999999,"5min":0.621,"15min":0.61699999999999999},"max":{"1min":1.1240000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.88800000000000001,"15min":0.86799999999999999},"min":{"1min":0.72099999999999997,"5min":0.627,"15min":0.54700000000000004},"max":{"1min":1.238,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.83899999999999997}]},{"osd":6,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.84099999999999997,"15min":0.82999999999999996},"min":{"1min":0.71199999999999997,"5min":0.57399999999999995,"15min":0.48999999999999999},"max":{"1min":1.1180000000000001,"5min":1.534,"15min":1.534},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.72099999999999997,"5min":0.59199999999999997,"15min":0.56999999999999995},"max":{"1min":1.127,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.86699999999999999}]},{"osd":7,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.66500000000000004,"5min":0.63100000000000001,"15min":0.58799999999999997},"max":{"1min":1.2010000000000001,"5min":1.55,"15min":1.55},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.90400000000000003,"15min":0.89400000000000002},"min":{"1min":0.67900000000000005,"5min":0.54900000000000004,"15min":0.53600000000000003},"max":{"1min":1.087,"5min":1.593,"15min":1.593},"last":0.84899999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575759,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299492,"kb_used_data":2204,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072348,"statfs":{"total":96636764160,"available":96330084352,"internally_reserved":0,"allocated":2256896,"data_stored":1065168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52000000000000002,"15min":0.504},"min":{"1min":0.36499999999999999,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56799999999999995,"15min":0.57899999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.42699999999999999}]},{"osd":1,"last update":"Thu Jan 11 09:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.51300000000000001,"15min":0.51600000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78200000000000003,"5min":0.88100000000000001,"15min":0.93100000000000005},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60199999999999998,"15min":0.59099999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.36299999999999999},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 11 09:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":0.373},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.58899999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.80300000000000005,"15min":0.78800000000000003},"min":{"1min":0.56399999999999995,"5min":0.48199999999999998,"15min":0.44900000000000001},"max":{"1min":1.075,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.80700000000000005,"15min":0.80200000000000005},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.52300000000000002},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.749}]},{"osd":5,"last update":"Thu Jan 11 09:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.81999999999999995,"15min":0.81799999999999995},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.84999999999999998,"15min":0.83599999999999997},"min":{"1min":0.67300000000000004,"5min":0.63800000000000001,"15min":0.58799999999999997},"max":{"1min":2.093,"5min":2.093,"15min":2.093},"last":0.83599999999999997}]},{"osd":6,"last update":"Thu Jan 11 09:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.872,"15min":0.85199999999999998},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.58299999999999996},"max":{"1min":2.2290000000000001,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.83199999999999996,"15min":0.81599999999999995},"min":{"1min":0.626,"5min":0.60099999999999998,"15min":0.49099999999999999},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.71399999999999997}]},{"osd":7,"last update":"Thu Jan 11 09:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.871,"15min":0.86199999999999999},"min":{"1min":0.68000000000000005,"5min":0.623,"15min":0.51800000000000002},"max":{"1min":1.038,"5min":1.1659999999999999,"15min":1.196},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.89700000000000002,"15min":0.88100000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.56499999999999995},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.79000000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303600,"kb_used_data":2216,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068240,"statfs":{"total":96636764160,"available":96325877760,"internally_reserved":0,"allocated":2269184,"data_stored":1066740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.77300000000000002,"15min":0.77700000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.35099999999999998},"max":{"1min":0.86899999999999999,"5min":1.0369999999999999,"15min":1.048},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.76200000000000001,"15min":0.77200000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.48999999999999999},"max":{"1min":0.98299999999999998,"5min":1.018,"15min":1.018},"last":0.72999999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.72099999999999997,"15min":0.73099999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.72499999999999998,"15min":0.73599999999999999},"min":{"1min":0.47299999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0009999999999999,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.66200000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.753,"15min":0.75600000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.96799999999999997,"5min":1,"15min":1},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.751,"15min":0.752},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.46100000000000002},"max":{"1min":1.0349999999999999,"5min":1.054,"15min":1.0760000000000001},"last":0.80500000000000005}]},{"osd":3,"last update":"Thu Jan 11 09:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.71699999999999997,"15min":0.72899999999999998},"min":{"1min":0.48199999999999998,"5min":0.442,"15min":0.442},"max":{"1min":0.93200000000000005,"5min":0.95399999999999996,"15min":1.0109999999999999},"last":0.621},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.78400000000000003,"15min":0.78100000000000003},"min":{"1min":0.55400000000000005,"5min":0.54400000000000004,"15min":0.53700000000000003},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.63700000000000001}]},{"osd":5,"last update":"Thu Jan 11 09:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61499999999999999,"15min":0.61699999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.86599999999999999,"5min":0.89300000000000002,"15min":0.94999999999999996},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.64200000000000002,"15min":0.629},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.53600000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.65900000000000003,"15min":0.65300000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.505},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.69999999999999996,"15min":0.69799999999999995},"min":{"1min":0.45000000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.92600000000000005,"5min":0.95899999999999996,"15min":0.96399999999999997},"last":0.77700000000000002}]},{"osd":7,"last update":"Thu Jan 11 09:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.69299999999999995,"15min":0.69399999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.92300000000000004,"5min":1.139,"15min":1.139},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.746,"15min":0.74299999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.67900000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575760,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307688,"kb_used_data":2208,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064152,"statfs":{"total":96636764160,"available":96321691648,"internally_reserved":0,"allocated":2260992,"data_stored":1065710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.73499999999999999,"15min":0.73699999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.77700000000000002,"5min":1.006,"15min":1.0229999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.76100000000000001,"15min":0.75600000000000001},"min":{"1min":0.46500000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.84799999999999998,"5min":0.97599999999999998,"15min":1.1020000000000001},"last":0.73299999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.745,"15min":0.74099999999999999},"min":{"1min":0.50700000000000001,"5min":0.438,"15min":0.41399999999999998},"max":{"1min":0.997,"5min":0.997,"15min":1.046},"last":0.753},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.74399999999999999,"15min":0.75800000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82399999999999995,"5min":0.997,"15min":1.0249999999999999},"last":0.66200000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.72599999999999998,"15min":0.747},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.76600000000000001,"5min":1.014,"15min":1.0569999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.76400000000000001,"15min":0.75600000000000001},"min":{"1min":0.51700000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94599999999999995,"5min":0.97999999999999998,"15min":1.012},"last":0.64500000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.76600000000000001,"15min":0.76900000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.85799999999999998,"5min":1.016,"15min":1.016},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.76900000000000002,"15min":0.77300000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.45800000000000002},"max":{"1min":0.91400000000000003,"5min":0.99199999999999999,"15min":1.0609999999999999},"last":0.68600000000000005}]},{"osd":4,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.622,"15min":0.61199999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.64200000000000002,"15min":0.63200000000000001},"min":{"1min":0.48299999999999998,"5min":0.45800000000000002,"15min":0.38900000000000001},"max":{"1min":0.88900000000000001,"5min":0.91300000000000003,"15min":0.94899999999999995},"last":0.78400000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.64700000000000002,"15min":0.64300000000000002},"min":{"1min":0.443,"5min":0.433,"15min":0.39400000000000002},"max":{"1min":0.874,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.44500000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.80700000000000005}]},{"osd":7,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.75,"15min":0.73599999999999999},"min":{"1min":0.53700000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.89300000000000002,"5min":0.997,"15min":1.0920000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.78900000000000003,"15min":0.78100000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96899999999999997,"5min":1.01,"15min":1.137},"last":0.82799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:27:26.405 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:27:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.642+0000 7fafc87ff700 1 Processor -- start 2024-01-11T09:27:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.643+0000 7fafc87ff700 1 -- start start 2024-01-11T09:27:26.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.644+0000 7fafc87ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafc00b03e0 0x7fafc00b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.644+0000 7fafc87ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc00b2e00 con 0x7fafc00b29c0 2024-01-11T09:27:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.644+0000 7fafc87ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc00cd730 con 0x7fafc00b07b0 2024-01-11T09:27:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.644+0000 7fafc87ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc00ccb20 con 0x7fafc00b03e0 2024-01-11T09:27:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.644+0000 7fafc6d9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fafc00b29c0 0x7fafc00aefc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53566/0 (socket says 172.21.15.27:53566) 2024-01-11T09:27:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.644+0000 7fafc6d9c700 1 -- 172.21.15.27:0/599600097 learned_addr learned my addr 172.21.15.27:0/599600097 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.644+0000 7fafc659b700 1 --2- 172.21.15.27:0/599600097 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafc00b03e0 0x7fafc00b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:26.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.645+0000 7fafc5599700 1 -- 172.21.15.27:0/599600097 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821093701 0 0) 0x7fafb8001490 con 0x7fafc00b29c0 2024-01-11T09:27:26.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.645+0000 7fafc5599700 1 -- 172.21.15.27:0/599600097 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafac003570 con 0x7fafc00b29c0 2024-01-11T09:27:26.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.645+0000 7fafc5599700 1 -- 172.21.15.27:0/599600097 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408947994 0 0) 0x7fafc00cd730 con 0x7fafc00b07b0 2024-01-11T09:27:26.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.645+0000 7fafc5599700 1 -- 172.21.15.27:0/599600097 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafb8001490 con 0x7fafc00b07b0 2024-01-11T09:27:26.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.645+0000 7fafc659b700 1 -- 172.21.15.27:0/599600097 >> v1:172.21.15.77:6789/0 conn(0x7fafc00b07b0 legacy=0x7fafc0159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:26.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.645+0000 7fafc659b700 1 -- 172.21.15.27:0/599600097 >> v1:172.21.15.27:6789/0 conn(0x7fafc00b29c0 legacy=0x7fafc00aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:26.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.645+0000 7fafc659b700 1 -- 172.21.15.27:0/599600097 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc00cce00 con 0x7fafc00b03e0 2024-01-11T09:27:26.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.645+0000 7fafc659b700 1 --2- 172.21.15.27:0/599600097 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafc00b03e0 0x7fafc00b2480 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fafbc009910 tx=0x7fafbc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0da187c19953a07 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:26.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.646+0000 7fafc5599700 1 -- 172.21.15.27:0/599600097 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc011630 con 0x7fafc00b03e0 2024-01-11T09:27:26.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.646+0000 7fafc5599700 1 -- 172.21.15.27:0/599600097 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafbc011790 con 0x7fafc00b03e0 2024-01-11T09:27:26.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.646+0000 7fafc5599700 1 -- 172.21.15.27:0/599600097 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc011b30 con 0x7fafc00b03e0 2024-01-11T09:27:26.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.646+0000 7fafc87ff700 1 -- 172.21.15.27:0/599600097 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafc00b03e0 msgr2=0x7fafc00b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:26.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.646+0000 7fafc87ff700 1 --2- 172.21.15.27:0/599600097 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafc00b03e0 0x7fafc00b2480 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fafbc009910 tx=0x7fafbc0091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:26.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.646+0000 7fafc87ff700 1 -- 172.21.15.27:0/599600097 shutdown_connections 2024-01-11T09:27:26.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.646+0000 7fafc87ff700 1 --2- 172.21.15.27:0/599600097 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafc00b03e0 0x7fafc00b2480 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:26.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.646+0000 7fafc87ff700 1 -- 172.21.15.27:0/599600097 >> 172.21.15.27:0/599600097 conn(0x7fafc006ed40 msgr2=0x7fafc006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:26.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.647+0000 7fafc87ff700 1 -- 172.21.15.27:0/599600097 shutdown_connections 2024-01-11T09:27:26.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.647+0000 7fafc87ff700 1 -- 172.21.15.27:0/599600097 wait complete. 2024-01-11T09:27:26.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.648+0000 7fafc87ff700 1 Processor -- start 2024-01-11T09:27:26.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.648+0000 7fafc87ff700 1 -- start start 2024-01-11T09:27:26.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.649+0000 7fafc87ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafc00b03e0 0x7fafc00c4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:26.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.649+0000 7fafc87ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafc00b07b0 0x7fafc014c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:26.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.649+0000 7fafc87ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafc00b29c0 0x7fafc00c65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:26.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.649+0000 7fafc87ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc014c2d0 con 0x7fafc00b29c0 2024-01-11T09:27:26.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.649+0000 7fafc87ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc00c6130 con 0x7fafc00b07b0 2024-01-11T09:27:26.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.649+0000 7fafc87ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc00c5520 con 0x7fafc00b03e0 2024-01-11T09:27:26.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.649+0000 7fafc6d9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafc00b29c0 0x7fafc00c65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:26.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.649+0000 7fafc659b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafc00b03e0 0x7fafc00c4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:26.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.649+0000 7fafc6d9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafc00b29c0 0x7fafc00c65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37674/0 (socket says 172.21.15.27:37674) 2024-01-11T09:27:26.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.649+0000 7fafc6d9c700 1 -- 172.21.15.27:0/4135634654 learned_addr learned my addr 172.21.15.27:0/4135634654 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:26.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.649+0000 7fafc5d9a700 1 --2- 172.21.15.27:0/4135634654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafc00b07b0 0x7fafc014c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:26.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.650+0000 7fafc6d9c700 1 -- 172.21.15.27:0/4135634654 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafc00b03e0 msgr2=0x7fafc00c4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:26.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.650+0000 7fafc6d9c700 1 --2- 172.21.15.27:0/4135634654 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafc00b03e0 0x7fafc00c4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:26.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.650+0000 7fafc6d9c700 1 -- 172.21.15.27:0/4135634654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafc00b07b0 msgr2=0x7fafc014c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:26.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.650+0000 7fafc6d9c700 1 --2- 172.21.15.27:0/4135634654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafc00b07b0 0x7fafc014c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:26.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.650+0000 7fafc6d9c700 1 -- 172.21.15.27:0/4135634654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafbc009420 con 0x7fafc00b29c0 2024-01-11T09:27:26.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.650+0000 7fafc6d9c700 1 --2- 172.21.15.27:0/4135634654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafc00b29c0 0x7fafc00c65c0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fafb8002250 tx=0x7fafb8001bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:26.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.650+0000 7fafb67fc700 1 -- 172.21.15.27:0/4135634654 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb800aac0 con 0x7fafc00b29c0 2024-01-11T09:27:26.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.650+0000 7fafb67fc700 1 -- 172.21.15.27:0/4135634654 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb800ac20 con 0x7fafc00b29c0 2024-01-11T09:27:26.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.650+0000 7fafb67fc700 1 -- 172.21.15.27:0/4135634654 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb80070f0 con 0x7fafc00b29c0 2024-01-11T09:27:26.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.650+0000 7fafc87ff700 1 -- 172.21.15.27:0/4135634654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc00c5800 con 0x7fafc00b29c0 2024-01-11T09:27:26.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.651+0000 7fafc87ff700 1 -- 172.21.15.27:0/4135634654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc00c5dc0 con 0x7fafc00b29c0 2024-01-11T09:27:26.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.651+0000 7fafc87ff700 1 -- 172.21.15.27:0/4135634654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc00c62e0 con 0x7fafc00b29c0 2024-01-11T09:27:26.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.653+0000 7fafb67fc700 1 -- 172.21.15.27:0/4135634654 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fafb80074e0 con 0x7fafc00b29c0 2024-01-11T09:27:26.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.653+0000 7fafb67fc700 1 --2- 172.21.15.27:0/4135634654 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fafa4066f30 0x7fafa40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:26.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.654+0000 7fafb67fc700 1 -- 172.21.15.27:0/4135634654 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fafb8084880 con 0x7fafc00b29c0 2024-01-11T09:27:26.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.654+0000 7fafc659b700 1 --2- 172.21.15.27:0/4135634654 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fafa4066f30 0x7fafa40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:26.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.655+0000 7fafc659b700 1 --2- 172.21.15.27:0/4135634654 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fafa4066f30 0x7fafa40693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fafbc009860 tx=0x7fafbc017000 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:26.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.657+0000 7fafb67fc700 1 -- 172.21.15.27:0/4135634654 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fafc00c62e0 con 0x7fafc00b29c0 2024-01-11T09:27:26.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.854+0000 7fafc87ff700 1 -- 172.21.15.27:0/4135634654 --> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fafc00b2e00 con 0x7fafa4066f30 2024-01-11T09:27:26.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.864+0000 7fafb67fc700 1 -- 172.21.15.27:0/4135634654 <== mgr.9010 v2:172.21.15.77:6836/41381 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289427 (secure 0 0 0) 0x7fafc00b2e00 con 0x7fafa4066f30 2024-01-11T09:27:26.866 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:27:26.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.869+0000 7fafc87ff700 1 -- 172.21.15.27:0/4135634654 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fafa4066f30 msgr2=0x7fafa40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:26.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.869+0000 7fafc87ff700 1 --2- 172.21.15.27:0/4135634654 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fafa4066f30 0x7fafa40693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fafbc009860 tx=0x7fafbc017000 comp rx=0 tx=0).stop 2024-01-11T09:27:26.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.869+0000 7fafc87ff700 1 -- 172.21.15.27:0/4135634654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafc00b29c0 msgr2=0x7fafc00c65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:26.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.869+0000 7fafc87ff700 1 --2- 172.21.15.27:0/4135634654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafc00b29c0 0x7fafc00c65c0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fafb8002250 tx=0x7fafb8001bb0 comp rx=0 tx=0).stop 2024-01-11T09:27:26.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.869+0000 7fafc87ff700 1 -- 172.21.15.27:0/4135634654 shutdown_connections 2024-01-11T09:27:26.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.869+0000 7fafc87ff700 1 --2- 172.21.15.27:0/4135634654 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fafa4066f30 0x7fafa40693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:26.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.869+0000 7fafc87ff700 1 --2- 172.21.15.27:0/4135634654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafc00b29c0 0x7fafc00c65c0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:26.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.869+0000 7fafc87ff700 1 --2- 172.21.15.27:0/4135634654 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafc00b03e0 0x7fafc00c4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:26.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.869+0000 7fafc87ff700 1 --2- 172.21.15.27:0/4135634654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafc00b07b0 0x7fafc014c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:26.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.869+0000 7fafc87ff700 1 -- 172.21.15.27:0/4135634654 >> 172.21.15.27:0/4135634654 conn(0x7fafc006ed40 msgr2=0x7fafc00ce780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:26.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.869+0000 7fafc87ff700 1 -- 172.21.15.27:0/4135634654 shutdown_connections 2024-01-11T09:27:26.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:26.869+0000 7fafc87ff700 1 -- 172.21.15.27:0/4135634654 wait complete. 2024-01-11T09:27:26.869 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:27:26.884 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-11T09:27:25.118717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1808,"num_read_kb":3778,"num_write":555,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":349,"ondisk_log_size":349,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":99,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2458568,"kb_used_data":18824,"kb_used_omap":0,"kb_used_meta":2439680,"kb_avail":752516152,"statfs":{"total":773094113280,"available":770576539648,"internally_reserved":0,"allocated":19275776,"data_stored":9706420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2498232320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.164745"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.003239+0000","last_change":"2024-01-11T09:27:22.003239+0000","last_active":"2024-01-11T09:27:22.003239+0000","last_peered":"2024-01-11T09:27:22.003239+0000","last_clean":"2024-01-11T09:27:22.003239+0000","last_became_active":"2024-01-11T09:27:22.002923+0000","last_became_peered":"2024-01-11T09:27:22.002923+0000","last_unstale":"2024-01-11T09:27:22.003239+0000","last_undegraded":"2024-01-11T09:27:22.003239+0000","last_fullsized":"2024-01-11T09:27:22.003239+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:46:13.646100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.994427+0000","last_change":"2024-01-11T09:27:21.994427+0000","last_active":"2024-01-11T09:27:21.994427+0000","last_peered":"2024-01-11T09:27:21.994427+0000","last_clean":"2024-01-11T09:27:21.994427+0000","last_became_active":"2024-01-11T09:27:21.993111+0000","last_became_peered":"2024-01-11T09:27:21.993111+0000","last_unstale":"2024-01-11T09:27:21.994427+0000","last_undegraded":"2024-01-11T09:27:21.994427+0000","last_fullsized":"2024-01-11T09:27:21.994427+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:12:11.031349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.992428+0000","last_change":"2024-01-11T09:27:21.984326+0000","last_active":"2024-01-11T09:27:21.992428+0000","last_peered":"2024-01-11T09:27:21.992428+0000","last_clean":"2024-01-11T09:27:21.992428+0000","last_became_active":"2024-01-11T09:27:21.984040+0000","last_became_peered":"2024-01-11T09:27:21.984040+0000","last_unstale":"2024-01-11T09:27:21.992428+0000","last_undegraded":"2024-01-11T09:27:21.992428+0000","last_fullsized":"2024-01-11T09:27:21.992428+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:07:57.735385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.996402+0000","last_change":"2024-01-11T09:27:21.996402+0000","last_active":"2024-01-11T09:27:21.996402+0000","last_peered":"2024-01-11T09:27:21.996402+0000","last_clean":"2024-01-11T09:27:21.996402+0000","last_became_active":"2024-01-11T09:27:21.994336+0000","last_became_peered":"2024-01-11T09:27:21.994336+0000","last_unstale":"2024-01-11T09:27:21.996402+0000","last_undegraded":"2024-01-11T09:27:21.996402+0000","last_fullsized":"2024-01-11T09:27:21.996402+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:40:30.768083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.995482+0000","last_change":"2024-01-11T09:27:21.995482+0000","last_active":"2024-01-11T09:27:21.995482+0000","last_peered":"2024-01-11T09:27:21.995482+0000","last_clean":"2024-01-11T09:27:21.995482+0000","last_became_active":"2024-01-11T09:27:21.994886+0000","last_became_peered":"2024-01-11T09:27:21.994886+0000","last_unstale":"2024-01-11T09:27:21.995482+0000","last_undegraded":"2024-01-11T09:27:21.995482+0000","last_fullsized":"2024-01-11T09:27:21.995482+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:15:32.573052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.995479+0000","last_change":"2024-01-11T09:27:21.995479+0000","last_active":"2024-01-11T09:27:21.995479+0000","last_peered":"2024-01-11T09:27:21.995479+0000","last_clean":"2024-01-11T09:27:21.995479+0000","last_became_active":"2024-01-11T09:27:21.995246+0000","last_became_peered":"2024-01-11T09:27:21.995246+0000","last_unstale":"2024-01-11T09:27:21.995479+0000","last_undegraded":"2024-01-11T09:27:21.995479+0000","last_fullsized":"2024-01-11T09:27:21.995479+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:17:28.045448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.993243+0000","last_change":"2024-01-11T09:27:21.993243+0000","last_active":"2024-01-11T09:27:21.993243+0000","last_peered":"2024-01-11T09:27:21.993243+0000","last_clean":"2024-01-11T09:27:21.993243+0000","last_became_active":"2024-01-11T09:27:21.992394+0000","last_became_peered":"2024-01-11T09:27:21.992394+0000","last_unstale":"2024-01-11T09:27:21.993243+0000","last_undegraded":"2024-01-11T09:27:21.993243+0000","last_fullsized":"2024-01-11T09:27:21.993243+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:44:16.678864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.995807+0000","last_change":"2024-01-11T09:27:21.995807+0000","last_active":"2024-01-11T09:27:21.995807+0000","last_peered":"2024-01-11T09:27:21.995807+0000","last_clean":"2024-01-11T09:27:21.995807+0000","last_became_active":"2024-01-11T09:27:21.995506+0000","last_became_peered":"2024-01-11T09:27:21.995506+0000","last_unstale":"2024-01-11T09:27:21.995807+0000","last_undegraded":"2024-01-11T09:27:21.995807+0000","last_fullsized":"2024-01-11T09:27:21.995807+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:22:02.217251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.994924+0000","last_change":"2024-01-11T09:27:21.994924+0000","last_active":"2024-01-11T09:27:21.994924+0000","last_peered":"2024-01-11T09:27:21.994924+0000","last_clean":"2024-01-11T09:27:21.994924+0000","last_became_active":"2024-01-11T09:27:21.994626+0000","last_became_peered":"2024-01-11T09:27:21.994626+0000","last_unstale":"2024-01-11T09:27:21.994924+0000","last_undegraded":"2024-01-11T09:27:21.994924+0000","last_fullsized":"2024-01-11T09:27:21.994924+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:48:22.941096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.004903+0000","last_change":"2024-01-11T09:27:22.004903+0000","last_active":"2024-01-11T09:27:22.004903+0000","last_peered":"2024-01-11T09:27:22.004903+0000","last_clean":"2024-01-11T09:27:22.004903+0000","last_became_active":"2024-01-11T09:27:22.004622+0000","last_became_peered":"2024-01-11T09:27:22.004622+0000","last_unstale":"2024-01-11T09:27:22.004903+0000","last_undegraded":"2024-01-11T09:27:22.004903+0000","last_fullsized":"2024-01-11T09:27:22.004903+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:49.010448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.996372+0000","last_change":"2024-01-11T09:27:21.996372+0000","last_active":"2024-01-11T09:27:21.996372+0000","last_peered":"2024-01-11T09:27:21.996372+0000","last_clean":"2024-01-11T09:27:21.996372+0000","last_became_active":"2024-01-11T09:27:21.995148+0000","last_became_peered":"2024-01-11T09:27:21.995148+0000","last_unstale":"2024-01-11T09:27:21.996372+0000","last_undegraded":"2024-01-11T09:27:21.996372+0000","last_fullsized":"2024-01-11T09:27:21.996372+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:56:38.205676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.990728+0000","last_change":"2024-01-11T09:27:21.990728+0000","last_active":"2024-01-11T09:27:21.990728+0000","last_peered":"2024-01-11T09:27:21.990728+0000","last_clean":"2024-01-11T09:27:21.990728+0000","last_became_active":"2024-01-11T09:27:21.990412+0000","last_became_peered":"2024-01-11T09:27:21.990412+0000","last_unstale":"2024-01-11T09:27:21.990728+0000","last_undegraded":"2024-01-11T09:27:21.990728+0000","last_fullsized":"2024-01-11T09:27:21.990728+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:06:01.994798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.989114+0000","last_change":"2024-01-11T09:27:21.977786+0000","last_active":"2024-01-11T09:27:21.989114+0000","last_peered":"2024-01-11T09:27:21.989114+0000","last_clean":"2024-01-11T09:27:21.989114+0000","last_became_active":"2024-01-11T09:27:21.977494+0000","last_became_peered":"2024-01-11T09:27:21.977494+0000","last_unstale":"2024-01-11T09:27:21.989114+0000","last_undegraded":"2024-01-11T09:27:21.989114+0000","last_fullsized":"2024-01-11T09:27:21.989114+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:38:37.294337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.996904+0000","last_change":"2024-01-11T09:27:21.989672+0000","last_active":"2024-01-11T09:27:21.996904+0000","last_peered":"2024-01-11T09:27:21.996904+0000","last_clean":"2024-01-11T09:27:21.996904+0000","last_became_active":"2024-01-11T09:27:21.988116+0000","last_became_peered":"2024-01-11T09:27:21.988116+0000","last_unstale":"2024-01-11T09:27:21.996904+0000","last_undegraded":"2024-01-11T09:27:21.996904+0000","last_fullsized":"2024-01-11T09:27:21.996904+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:38:04.939061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.003938+0000","last_change":"2024-01-11T09:27:22.003938+0000","last_active":"2024-01-11T09:27:22.003938+0000","last_peered":"2024-01-11T09:27:22.003938+0000","last_clean":"2024-01-11T09:27:22.003938+0000","last_became_active":"2024-01-11T09:27:22.003635+0000","last_became_peered":"2024-01-11T09:27:22.003635+0000","last_unstale":"2024-01-11T09:27:22.003938+0000","last_undegraded":"2024-01-11T09:27:22.003938+0000","last_fullsized":"2024-01-11T09:27:22.003938+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:04:20.823816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.989789+0000","last_change":"2024-01-11T09:27:21.989789+0000","last_active":"2024-01-11T09:27:21.989789+0000","last_peered":"2024-01-11T09:27:21.989789+0000","last_clean":"2024-01-11T09:27:21.989789+0000","last_became_active":"2024-01-11T09:27:21.989524+0000","last_became_peered":"2024-01-11T09:27:21.989524+0000","last_unstale":"2024-01-11T09:27:21.989789+0000","last_undegraded":"2024-01-11T09:27:21.989789+0000","last_fullsized":"2024-01-11T09:27:21.989789+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:17:26.059638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.006565+0000","last_change":"2024-01-11T09:27:22.006565+0000","last_active":"2024-01-11T09:27:22.006565+0000","last_peered":"2024-01-11T09:27:22.006565+0000","last_clean":"2024-01-11T09:27:22.006565+0000","last_became_active":"2024-01-11T09:27:22.006246+0000","last_became_peered":"2024-01-11T09:27:22.006246+0000","last_unstale":"2024-01-11T09:27:22.006565+0000","last_undegraded":"2024-01-11T09:27:22.006565+0000","last_fullsized":"2024-01-11T09:27:22.006565+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:09:18.445842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.008212+0000","last_change":"2024-01-11T09:27:22.003524+0000","last_active":"2024-01-11T09:27:22.008212+0000","last_peered":"2024-01-11T09:27:22.008212+0000","last_clean":"2024-01-11T09:27:22.008212+0000","last_became_active":"2024-01-11T09:27:22.003223+0000","last_became_peered":"2024-01-11T09:27:22.003223+0000","last_unstale":"2024-01-11T09:27:22.008212+0000","last_undegraded":"2024-01-11T09:27:22.008212+0000","last_fullsized":"2024-01-11T09:27:22.008212+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:55:38.095443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.994165+0000","last_change":"2024-01-11T09:27:21.994165+0000","last_active":"2024-01-11T09:27:21.994165+0000","last_peered":"2024-01-11T09:27:21.994165+0000","last_clean":"2024-01-11T09:27:21.994165+0000","last_became_active":"2024-01-11T09:27:21.993878+0000","last_became_peered":"2024-01-11T09:27:21.993878+0000","last_unstale":"2024-01-11T09:27:21.994165+0000","last_undegraded":"2024-01-11T09:27:21.994165+0000","last_fullsized":"2024-01-11T09:27:21.994165+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:07:57.735385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.996854+0000","last_change":"2024-01-11T09:27:21.989178+0000","last_active":"2024-01-11T09:27:21.996854+0000","last_peered":"2024-01-11T09:27:21.996854+0000","last_clean":"2024-01-11T09:27:21.996854+0000","last_became_active":"2024-01-11T09:27:21.988899+0000","last_became_peered":"2024-01-11T09:27:21.988899+0000","last_unstale":"2024-01-11T09:27:21.996854+0000","last_undegraded":"2024-01-11T09:27:21.996854+0000","last_fullsized":"2024-01-11T09:27:21.996854+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:34:07.131869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.004962+0000","last_change":"2024-01-11T09:27:22.004962+0000","last_active":"2024-01-11T09:27:22.004962+0000","last_peered":"2024-01-11T09:27:22.004962+0000","last_clean":"2024-01-11T09:27:22.004962+0000","last_became_active":"2024-01-11T09:27:22.004673+0000","last_became_peered":"2024-01-11T09:27:22.004673+0000","last_unstale":"2024-01-11T09:27:22.004962+0000","last_undegraded":"2024-01-11T09:27:22.004962+0000","last_fullsized":"2024-01-11T09:27:22.004962+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:34:07.131869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.997561+0000","last_change":"2024-01-11T09:27:21.997561+0000","last_active":"2024-01-11T09:27:21.997561+0000","last_peered":"2024-01-11T09:27:21.997561+0000","last_clean":"2024-01-11T09:27:21.997561+0000","last_became_active":"2024-01-11T09:27:21.997188+0000","last_became_peered":"2024-01-11T09:27:21.997188+0000","last_unstale":"2024-01-11T09:27:21.997561+0000","last_undegraded":"2024-01-11T09:27:21.997561+0000","last_fullsized":"2024-01-11T09:27:21.997561+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:44:16.678864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.991642+0000","last_change":"2024-01-11T09:27:21.991642+0000","last_active":"2024-01-11T09:27:21.991642+0000","last_peered":"2024-01-11T09:27:21.991642+0000","last_clean":"2024-01-11T09:27:21.991642+0000","last_became_active":"2024-01-11T09:27:21.991349+0000","last_became_peered":"2024-01-11T09:27:21.991349+0000","last_unstale":"2024-01-11T09:27:21.991642+0000","last_undegraded":"2024-01-11T09:27:21.991642+0000","last_fullsized":"2024-01-11T09:27:21.991642+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:55:38.946028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.996969+0000","last_change":"2024-01-11T09:27:21.996969+0000","last_active":"2024-01-11T09:27:21.996969+0000","last_peered":"2024-01-11T09:27:21.996969+0000","last_clean":"2024-01-11T09:27:21.996969+0000","last_became_active":"2024-01-11T09:27:21.996413+0000","last_became_peered":"2024-01-11T09:27:21.996413+0000","last_unstale":"2024-01-11T09:27:21.996969+0000","last_undegraded":"2024-01-11T09:27:21.996969+0000","last_fullsized":"2024-01-11T09:27:21.996969+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:13:01.725223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.992336+0000","last_change":"2024-01-11T09:27:21.982776+0000","last_active":"2024-01-11T09:27:21.992336+0000","last_peered":"2024-01-11T09:27:21.992336+0000","last_clean":"2024-01-11T09:27:21.992336+0000","last_became_active":"2024-01-11T09:27:21.980022+0000","last_became_peered":"2024-01-11T09:27:21.980022+0000","last_unstale":"2024-01-11T09:27:21.992336+0000","last_undegraded":"2024-01-11T09:27:21.992336+0000","last_fullsized":"2024-01-11T09:27:21.992336+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:58:47.672394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.008410+0000","last_change":"2024-01-11T09:27:22.008410+0000","last_active":"2024-01-11T09:27:22.008410+0000","last_peered":"2024-01-11T09:27:22.008410+0000","last_clean":"2024-01-11T09:27:22.008410+0000","last_became_active":"2024-01-11T09:27:22.007205+0000","last_became_peered":"2024-01-11T09:27:22.007205+0000","last_unstale":"2024-01-11T09:27:22.008410+0000","last_undegraded":"2024-01-11T09:27:22.008410+0000","last_fullsized":"2024-01-11T09:27:22.008410+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:13:01.725223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.000447+0000","last_change":"2024-01-11T09:27:21.995882+0000","last_active":"2024-01-11T09:27:22.000447+0000","last_peered":"2024-01-11T09:27:22.000447+0000","last_clean":"2024-01-11T09:27:22.000447+0000","last_became_active":"2024-01-11T09:27:21.994795+0000","last_became_peered":"2024-01-11T09:27:21.994795+0000","last_unstale":"2024-01-11T09:27:22.000447+0000","last_undegraded":"2024-01-11T09:27:22.000447+0000","last_fullsized":"2024-01-11T09:27:22.000447+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:41:27.430029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.002635+0000","last_change":"2024-01-11T09:27:22.002635+0000","last_active":"2024-01-11T09:27:22.002635+0000","last_peered":"2024-01-11T09:27:22.002635+0000","last_clean":"2024-01-11T09:27:22.002635+0000","last_became_active":"2024-01-11T09:27:22.002296+0000","last_became_peered":"2024-01-11T09:27:22.002296+0000","last_unstale":"2024-01-11T09:27:22.002635+0000","last_undegraded":"2024-01-11T09:27:22.002635+0000","last_fullsized":"2024-01-11T09:27:22.002635+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:35:10.052296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.001070+0000","last_change":"2024-01-11T09:27:21.995530+0000","last_active":"2024-01-11T09:27:22.001070+0000","last_peered":"2024-01-11T09:27:22.001070+0000","last_clean":"2024-01-11T09:27:22.001070+0000","last_became_active":"2024-01-11T09:27:21.993025+0000","last_became_peered":"2024-01-11T09:27:21.993025+0000","last_unstale":"2024-01-11T09:27:22.001070+0000","last_undegraded":"2024-01-11T09:27:22.001070+0000","last_fullsized":"2024-01-11T09:27:22.001070+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:03:37.377064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.993445+0000","last_change":"2024-01-11T09:27:21.990335+0000","last_active":"2024-01-11T09:27:21.993445+0000","last_peered":"2024-01-11T09:27:21.993445+0000","last_clean":"2024-01-11T09:27:21.993445+0000","last_became_active":"2024-01-11T09:27:21.990059+0000","last_became_peered":"2024-01-11T09:27:21.990059+0000","last_unstale":"2024-01-11T09:27:21.993445+0000","last_undegraded":"2024-01-11T09:27:21.993445+0000","last_fullsized":"2024-01-11T09:27:21.993445+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:54:30.583702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.991582+0000","last_change":"2024-01-11T09:27:21.987052+0000","last_active":"2024-01-11T09:27:21.991582+0000","last_peered":"2024-01-11T09:27:21.991582+0000","last_clean":"2024-01-11T09:27:21.991582+0000","last_became_active":"2024-01-11T09:27:21.986668+0000","last_became_peered":"2024-01-11T09:27:21.986668+0000","last_unstale":"2024-01-11T09:27:21.991582+0000","last_undegraded":"2024-01-11T09:27:21.991582+0000","last_fullsized":"2024-01-11T09:27:21.991582+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:42:08.682802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.992288+0000","last_change":"2024-01-11T09:27:21.989188+0000","last_active":"2024-01-11T09:27:21.992288+0000","last_peered":"2024-01-11T09:27:21.992288+0000","last_clean":"2024-01-11T09:27:21.992288+0000","last_became_active":"2024-01-11T09:27:21.988794+0000","last_became_peered":"2024-01-11T09:27:21.988794+0000","last_unstale":"2024-01-11T09:27:21.992288+0000","last_undegraded":"2024-01-11T09:27:21.992288+0000","last_fullsized":"2024-01-11T09:27:21.992288+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:05:36.862461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.005931+0000","last_change":"2024-01-11T09:27:22.005931+0000","last_active":"2024-01-11T09:27:22.005931+0000","last_peered":"2024-01-11T09:27:22.005931+0000","last_clean":"2024-01-11T09:27:22.005931+0000","last_became_active":"2024-01-11T09:27:22.005645+0000","last_became_peered":"2024-01-11T09:27:22.005645+0000","last_unstale":"2024-01-11T09:27:22.005931+0000","last_undegraded":"2024-01-11T09:27:22.005931+0000","last_fullsized":"2024-01-11T09:27:22.005931+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:56:38.205676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"180'4","reported_seq":13,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.008494+0000","last_change":"2024-01-11T09:27:20.979608+0000","last_active":"2024-01-11T09:27:22.008494+0000","last_peered":"2024-01-11T09:27:22.008494+0000","last_clean":"2024-01-11T09:27:22.008494+0000","last_became_active":"2024-01-11T09:27:20.979301+0000","last_became_peered":"2024-01-11T09:27:20.979301+0000","last_unstale":"2024-01-11T09:27:22.008494+0000","last_undegraded":"2024-01-11T09:27:22.008494+0000","last_fullsized":"2024-01-11T09:27:22.008494+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:49:12.315817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.992111+0000","last_change":"2024-01-11T09:27:21.985098+0000","last_active":"2024-01-11T09:27:21.992111+0000","last_peered":"2024-01-11T09:27:21.992111+0000","last_clean":"2024-01-11T09:27:21.992111+0000","last_became_active":"2024-01-11T09:27:21.983408+0000","last_became_peered":"2024-01-11T09:27:21.983408+0000","last_unstale":"2024-01-11T09:27:21.992111+0000","last_undegraded":"2024-01-11T09:27:21.992111+0000","last_fullsized":"2024-01-11T09:27:21.992111+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:26:58.930845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"180'1","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.999149+0000","last_change":"2024-01-11T09:27:20.980885+0000","last_active":"2024-01-11T09:27:21.999149+0000","last_peered":"2024-01-11T09:27:21.999149+0000","last_clean":"2024-01-11T09:27:21.999149+0000","last_became_active":"2024-01-11T09:27:20.980585+0000","last_became_peered":"2024-01-11T09:27:20.980585+0000","last_unstale":"2024-01-11T09:27:21.999149+0000","last_undegraded":"2024-01-11T09:27:21.999149+0000","last_fullsized":"2024-01-11T09:27:21.999149+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:03:36.370161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.993974+0000","last_change":"2024-01-11T09:27:21.993974+0000","last_active":"2024-01-11T09:27:21.993974+0000","last_peered":"2024-01-11T09:27:21.993974+0000","last_clean":"2024-01-11T09:27:21.993974+0000","last_became_active":"2024-01-11T09:27:21.991505+0000","last_became_peered":"2024-01-11T09:27:21.991505+0000","last_unstale":"2024-01-11T09:27:21.993974+0000","last_undegraded":"2024-01-11T09:27:21.993974+0000","last_fullsized":"2024-01-11T09:27:21.993974+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:19:54.081327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"180'2","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.001400+0000","last_change":"2024-01-11T09:27:20.980657+0000","last_active":"2024-01-11T09:27:22.001400+0000","last_peered":"2024-01-11T09:27:22.001400+0000","last_clean":"2024-01-11T09:27:22.001400+0000","last_became_active":"2024-01-11T09:27:20.980394+0000","last_became_peered":"2024-01-11T09:27:20.980394+0000","last_unstale":"2024-01-11T09:27:22.001400+0000","last_undegraded":"2024-01-11T09:27:22.001400+0000","last_fullsized":"2024-01-11T09:27:22.001400+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:35:14.539951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.997971+0000","last_change":"2024-01-11T09:27:21.987885+0000","last_active":"2024-01-11T09:27:21.997971+0000","last_peered":"2024-01-11T09:27:21.997971+0000","last_clean":"2024-01-11T09:27:21.997971+0000","last_became_active":"2024-01-11T09:27:21.987574+0000","last_became_peered":"2024-01-11T09:27:21.987574+0000","last_unstale":"2024-01-11T09:27:21.997971+0000","last_undegraded":"2024-01-11T09:27:21.997971+0000","last_fullsized":"2024-01-11T09:27:21.997971+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:09:18.445842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"180'2","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.002786+0000","last_change":"2024-01-11T09:27:20.982829+0000","last_active":"2024-01-11T09:27:22.002786+0000","last_peered":"2024-01-11T09:27:22.002786+0000","last_clean":"2024-01-11T09:27:22.002786+0000","last_became_active":"2024-01-11T09:27:20.982535+0000","last_became_peered":"2024-01-11T09:27:20.982535+0000","last_unstale":"2024-01-11T09:27:22.002786+0000","last_undegraded":"2024-01-11T09:27:22.002786+0000","last_fullsized":"2024-01-11T09:27:22.002786+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:17:27.038545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.992905+0000","last_change":"2024-01-11T09:27:21.992905+0000","last_active":"2024-01-11T09:27:21.992905+0000","last_peered":"2024-01-11T09:27:21.992905+0000","last_clean":"2024-01-11T09:27:21.992905+0000","last_became_active":"2024-01-11T09:27:21.992610+0000","last_became_peered":"2024-01-11T09:27:21.992610+0000","last_unstale":"2024-01-11T09:27:21.992905+0000","last_undegraded":"2024-01-11T09:27:21.992905+0000","last_fullsized":"2024-01-11T09:27:21.992905+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:36:27.578314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"180'1","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.000995+0000","last_change":"2024-01-11T09:27:20.980015+0000","last_active":"2024-01-11T09:27:22.000995+0000","last_peered":"2024-01-11T09:27:22.000995+0000","last_clean":"2024-01-11T09:27:22.000995+0000","last_became_active":"2024-01-11T09:27:20.979717+0000","last_became_peered":"2024-01-11T09:27:20.979717+0000","last_unstale":"2024-01-11T09:27:22.000995+0000","last_undegraded":"2024-01-11T09:27:22.000995+0000","last_fullsized":"2024-01-11T09:27:22.000995+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:57:16.382699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.997641+0000","last_change":"2024-01-11T09:27:21.997641+0000","last_active":"2024-01-11T09:27:21.997641+0000","last_peered":"2024-01-11T09:27:21.997641+0000","last_clean":"2024-01-11T09:27:21.997641+0000","last_became_active":"2024-01-11T09:27:21.997370+0000","last_became_peered":"2024-01-11T09:27:21.997370+0000","last_unstale":"2024-01-11T09:27:21.997641+0000","last_undegraded":"2024-01-11T09:27:21.997641+0000","last_fullsized":"2024-01-11T09:27:21.997641+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:03:37.377064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"180'1","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.978667+0000","last_change":"2024-01-11T09:27:20.973740+0000","last_active":"2024-01-11T09:27:21.978667+0000","last_peered":"2024-01-11T09:27:21.978667+0000","last_clean":"2024-01-11T09:27:21.978667+0000","last_became_active":"2024-01-11T09:27:20.973327+0000","last_became_peered":"2024-01-11T09:27:20.973327+0000","last_unstale":"2024-01-11T09:27:21.978667+0000","last_undegraded":"2024-01-11T09:27:21.978667+0000","last_fullsized":"2024-01-11T09:27:21.978667+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:55:37.939126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.997856+0000","last_change":"2024-01-11T09:27:21.997856+0000","last_active":"2024-01-11T09:27:21.997856+0000","last_peered":"2024-01-11T09:27:21.997856+0000","last_clean":"2024-01-11T09:27:21.997856+0000","last_became_active":"2024-01-11T09:27:21.997546+0000","last_became_peered":"2024-01-11T09:27:21.997546+0000","last_unstale":"2024-01-11T09:27:21.997856+0000","last_undegraded":"2024-01-11T09:27:21.997856+0000","last_fullsized":"2024-01-11T09:27:21.997856+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:46:41.232840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.982264+0000","last_change":"2024-01-11T09:27:20.978135+0000","last_active":"2024-01-11T09:27:20.982264+0000","last_peered":"2024-01-11T09:27:20.982264+0000","last_clean":"2024-01-11T09:27:20.982264+0000","last_became_active":"2024-01-11T09:27:20.977589+0000","last_became_peered":"2024-01-11T09:27:20.977589+0000","last_unstale":"2024-01-11T09:27:20.982264+0000","last_undegraded":"2024-01-11T09:27:20.982264+0000","last_fullsized":"2024-01-11T09:27:20.982264+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:05:35.855559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.998182+0000","last_change":"2024-01-11T09:27:21.998182+0000","last_active":"2024-01-11T09:27:21.998182+0000","last_peered":"2024-01-11T09:27:21.998182+0000","last_clean":"2024-01-11T09:27:21.998182+0000","last_became_active":"2024-01-11T09:27:21.997719+0000","last_became_peered":"2024-01-11T09:27:21.997719+0000","last_unstale":"2024-01-11T09:27:21.998182+0000","last_undegraded":"2024-01-11T09:27:21.998182+0000","last_fullsized":"2024-01-11T09:27:21.998182+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:58:10.141123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.972025+0000","last_change":"2024-01-11T09:27:20.970086+0000","last_active":"2024-01-11T09:27:20.972025+0000","last_peered":"2024-01-11T09:27:20.972025+0000","last_clean":"2024-01-11T09:27:20.972025+0000","last_became_active":"2024-01-11T09:27:20.969789+0000","last_became_peered":"2024-01-11T09:27:20.969789+0000","last_unstale":"2024-01-11T09:27:20.972025+0000","last_undegraded":"2024-01-11T09:27:20.972025+0000","last_fullsized":"2024-01-11T09:27:20.972025+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:30:31.681173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.006251+0000","last_change":"2024-01-11T09:27:22.001624+0000","last_active":"2024-01-11T09:27:22.006251+0000","last_peered":"2024-01-11T09:27:22.006251+0000","last_clean":"2024-01-11T09:27:22.006251+0000","last_became_active":"2024-01-11T09:27:22.000534+0000","last_became_peered":"2024-01-11T09:27:22.000534+0000","last_unstale":"2024-01-11T09:27:22.006251+0000","last_undegraded":"2024-01-11T09:27:22.006251+0000","last_fullsized":"2024-01-11T09:27:22.006251+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:55:25.729750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.976946+0000","last_change":"2024-01-11T09:27:20.972671+0000","last_active":"2024-01-11T09:27:20.976946+0000","last_peered":"2024-01-11T09:27:20.976946+0000","last_clean":"2024-01-11T09:27:20.976946+0000","last_became_active":"2024-01-11T09:27:20.972420+0000","last_became_peered":"2024-01-11T09:27:20.972420+0000","last_unstale":"2024-01-11T09:27:20.976946+0000","last_undegraded":"2024-01-11T09:27:20.976946+0000","last_fullsized":"2024-01-11T09:27:20.976946+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:46:55.573874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.994255+0000","last_change":"2024-01-11T09:27:21.991257+0000","last_active":"2024-01-11T09:27:21.994255+0000","last_peered":"2024-01-11T09:27:21.994255+0000","last_clean":"2024-01-11T09:27:21.994255+0000","last_became_active":"2024-01-11T09:27:21.990995+0000","last_became_peered":"2024-01-11T09:27:21.990995+0000","last_unstale":"2024-01-11T09:27:21.994255+0000","last_undegraded":"2024-01-11T09:27:21.994255+0000","last_fullsized":"2024-01-11T09:27:21.994255+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:46:13.646100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"180'3","reported_seq":12,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.008689+0000","last_change":"2024-01-11T09:27:20.982186+0000","last_active":"2024-01-11T09:27:22.008689+0000","last_peered":"2024-01-11T09:27:22.008689+0000","last_clean":"2024-01-11T09:27:22.008689+0000","last_became_active":"2024-01-11T09:27:20.981893+0000","last_became_peered":"2024-01-11T09:27:20.981893+0000","last_unstale":"2024-01-11T09:27:22.008689+0000","last_undegraded":"2024-01-11T09:27:22.008689+0000","last_fullsized":"2024-01-11T09:27:22.008689+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:57:16.382699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.989443+0000","last_change":"2024-01-11T09:27:21.980001+0000","last_active":"2024-01-11T09:27:21.989443+0000","last_peered":"2024-01-11T09:27:21.989443+0000","last_clean":"2024-01-11T09:27:21.989443+0000","last_became_active":"2024-01-11T09:27:21.979253+0000","last_became_peered":"2024-01-11T09:27:21.979253+0000","last_unstale":"2024-01-11T09:27:21.989443+0000","last_undegraded":"2024-01-11T09:27:21.989443+0000","last_fullsized":"2024-01-11T09:27:21.989443+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:33:53.686857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"180'6","reported_seq":16,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.013607+0000","last_change":"2024-01-11T09:27:20.981882+0000","last_active":"2024-01-11T09:27:22.013607+0000","last_peered":"2024-01-11T09:27:22.013607+0000","last_clean":"2024-01-11T09:27:22.013607+0000","last_became_active":"2024-01-11T09:27:20.980779+0000","last_became_peered":"2024-01-11T09:27:20.980779+0000","last_unstale":"2024-01-11T09:27:22.013607+0000","last_undegraded":"2024-01-11T09:27:22.013607+0000","last_fullsized":"2024-01-11T09:27:22.013607+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:35:09.045394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.006217+0000","last_change":"2024-01-11T09:27:21.999904+0000","last_active":"2024-01-11T09:27:22.006217+0000","last_peered":"2024-01-11T09:27:22.006217+0000","last_clean":"2024-01-11T09:27:22.006217+0000","last_became_active":"2024-01-11T09:27:21.998990+0000","last_became_peered":"2024-01-11T09:27:21.998990+0000","last_unstale":"2024-01-11T09:27:22.006217+0000","last_undegraded":"2024-01-11T09:27:22.006217+0000","last_fullsized":"2024-01-11T09:27:22.006217+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:03:04.965882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"180'2","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.001234+0000","last_change":"2024-01-11T09:27:20.982874+0000","last_active":"2024-01-11T09:27:22.001234+0000","last_peered":"2024-01-11T09:27:22.001234+0000","last_clean":"2024-01-11T09:27:22.001234+0000","last_became_active":"2024-01-11T09:27:20.982034+0000","last_became_peered":"2024-01-11T09:27:20.982034+0000","last_unstale":"2024-01-11T09:27:22.001234+0000","last_undegraded":"2024-01-11T09:27:22.001234+0000","last_fullsized":"2024-01-11T09:27:22.001234+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:51:34.857778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.999271+0000","last_change":"2024-01-11T09:27:21.999271+0000","last_active":"2024-01-11T09:27:21.999271+0000","last_peered":"2024-01-11T09:27:21.999271+0000","last_clean":"2024-01-11T09:27:21.999271+0000","last_became_active":"2024-01-11T09:27:21.998817+0000","last_became_peered":"2024-01-11T09:27:21.998817+0000","last_unstale":"2024-01-11T09:27:21.999271+0000","last_undegraded":"2024-01-11T09:27:21.999271+0000","last_fullsized":"2024-01-11T09:27:21.999271+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:17:40.366748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.970709+0000","last_change":"2024-01-11T09:27:20.979292+0000","last_active":"2024-01-11T09:27:21.970709+0000","last_peered":"2024-01-11T09:27:21.970709+0000","last_clean":"2024-01-11T09:27:21.970709+0000","last_became_active":"2024-01-11T09:27:20.978722+0000","last_became_peered":"2024-01-11T09:27:20.978722+0000","last_unstale":"2024-01-11T09:27:21.970709+0000","last_undegraded":"2024-01-11T09:27:21.970709+0000","last_fullsized":"2024-01-11T09:27:21.970709+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:46:40.225937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.001264+0000","last_change":"2024-01-11T09:27:22.001264+0000","last_active":"2024-01-11T09:27:22.001264+0000","last_peered":"2024-01-11T09:27:22.001264+0000","last_clean":"2024-01-11T09:27:22.001264+0000","last_became_active":"2024-01-11T09:27:22.000934+0000","last_became_peered":"2024-01-11T09:27:22.000934+0000","last_unstale":"2024-01-11T09:27:22.001264+0000","last_undegraded":"2024-01-11T09:27:22.001264+0000","last_fullsized":"2024-01-11T09:27:22.001264+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:53:23.281467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.981699+0000","last_change":"2024-01-11T09:27:20.978413+0000","last_active":"2024-01-11T09:27:20.981699+0000","last_peered":"2024-01-11T09:27:20.981699+0000","last_clean":"2024-01-11T09:27:20.981699+0000","last_became_active":"2024-01-11T09:27:20.977865+0000","last_became_peered":"2024-01-11T09:27:20.977865+0000","last_unstale":"2024-01-11T09:27:20.981699+0000","last_undegraded":"2024-01-11T09:27:20.981699+0000","last_fullsized":"2024-01-11T09:27:20.981699+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:58:09.134220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.008025+0000","last_change":"2024-01-11T09:27:22.002406+0000","last_active":"2024-01-11T09:27:22.008025+0000","last_peered":"2024-01-11T09:27:22.008025+0000","last_clean":"2024-01-11T09:27:22.008025+0000","last_became_active":"2024-01-11T09:27:22.001428+0000","last_became_peered":"2024-01-11T09:27:22.001428+0000","last_unstale":"2024-01-11T09:27:22.008025+0000","last_undegraded":"2024-01-11T09:27:22.008025+0000","last_fullsized":"2024-01-11T09:27:22.008025+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:45:21.227961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"180'8","reported_seq":18,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.006760+0000","last_change":"2024-01-11T09:27:20.981746+0000","last_active":"2024-01-11T09:27:22.006760+0000","last_peered":"2024-01-11T09:27:22.006760+0000","last_clean":"2024-01-11T09:27:22.006760+0000","last_became_active":"2024-01-11T09:27:20.981051+0000","last_became_peered":"2024-01-11T09:27:20.981051+0000","last_unstale":"2024-01-11T09:27:22.006760+0000","last_undegraded":"2024-01-11T09:27:22.006760+0000","last_fullsized":"2024-01-11T09:27:22.006760+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:53:22.274565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.006799+0000","last_change":"2024-01-11T09:27:22.006799+0000","last_active":"2024-01-11T09:27:22.006799+0000","last_peered":"2024-01-11T09:27:22.006799+0000","last_clean":"2024-01-11T09:27:22.006799+0000","last_became_active":"2024-01-11T09:27:22.006523+0000","last_became_peered":"2024-01-11T09:27:22.006523+0000","last_unstale":"2024-01-11T09:27:22.006799+0000","last_undegraded":"2024-01-11T09:27:22.006799+0000","last_fullsized":"2024-01-11T09:27:22.006799+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:05.603829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"180'1","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.001157+0000","last_change":"2024-01-11T09:27:20.973383+0000","last_active":"2024-01-11T09:27:22.001157+0000","last_peered":"2024-01-11T09:27:22.001157+0000","last_clean":"2024-01-11T09:27:22.001157+0000","last_became_active":"2024-01-11T09:27:20.971987+0000","last_became_peered":"2024-01-11T09:27:20.971987+0000","last_unstale":"2024-01-11T09:27:22.001157+0000","last_undegraded":"2024-01-11T09:27:22.001157+0000","last_fullsized":"2024-01-11T09:27:22.001157+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:08.969875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.991461+0000","last_change":"2024-01-11T09:27:21.982420+0000","last_active":"2024-01-11T09:27:21.991461+0000","last_peered":"2024-01-11T09:27:21.991461+0000","last_clean":"2024-01-11T09:27:21.991461+0000","last_became_active":"2024-01-11T09:27:21.982156+0000","last_became_peered":"2024-01-11T09:27:21.982156+0000","last_unstale":"2024-01-11T09:27:21.991461+0000","last_undegraded":"2024-01-11T09:27:21.991461+0000","last_fullsized":"2024-01-11T09:27:21.991461+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:06:42.072322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"180'2","reported_seq":12,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.002903+0000","last_change":"2024-01-11T09:27:20.980188+0000","last_active":"2024-01-11T09:27:22.002903+0000","last_peered":"2024-01-11T09:27:22.002903+0000","last_clean":"2024-01-11T09:27:22.002903+0000","last_became_active":"2024-01-11T09:27:20.979436+0000","last_became_peered":"2024-01-11T09:27:20.979436+0000","last_unstale":"2024-01-11T09:27:22.002903+0000","last_undegraded":"2024-01-11T09:27:22.002903+0000","last_fullsized":"2024-01-11T09:27:22.002903+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:46:55.573874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.996278+0000","last_change":"2024-01-11T09:27:21.996278+0000","last_active":"2024-01-11T09:27:21.996278+0000","last_peered":"2024-01-11T09:27:21.996278+0000","last_clean":"2024-01-11T09:27:21.996278+0000","last_became_active":"2024-01-11T09:27:21.994477+0000","last_became_peered":"2024-01-11T09:27:21.994477+0000","last_unstale":"2024-01-11T09:27:21.996278+0000","last_undegraded":"2024-01-11T09:27:21.996278+0000","last_fullsized":"2024-01-11T09:27:21.996278+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:41:27.430029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"180'1","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.983228+0000","last_change":"2024-01-11T09:27:20.979609+0000","last_active":"2024-01-11T09:27:21.983228+0000","last_peered":"2024-01-11T09:27:21.983228+0000","last_clean":"2024-01-11T09:27:21.983228+0000","last_became_active":"2024-01-11T09:27:20.979259+0000","last_became_peered":"2024-01-11T09:27:20.979259+0000","last_unstale":"2024-01-11T09:27:21.983228+0000","last_undegraded":"2024-01-11T09:27:21.983228+0000","last_fullsized":"2024-01-11T09:27:21.983228+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:42:07.675900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.996977+0000","last_change":"2024-01-11T09:27:21.996977+0000","last_active":"2024-01-11T09:27:21.996977+0000","last_peered":"2024-01-11T09:27:21.996977+0000","last_clean":"2024-01-11T09:27:21.996977+0000","last_became_active":"2024-01-11T09:27:21.996652+0000","last_became_peered":"2024-01-11T09:27:21.996652+0000","last_unstale":"2024-01-11T09:27:21.996977+0000","last_undegraded":"2024-01-11T09:27:21.996977+0000","last_fullsized":"2024-01-11T09:27:21.996977+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:43:28.389359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.981876+0000","last_change":"2024-01-11T09:27:20.975587+0000","last_active":"2024-01-11T09:27:20.981876+0000","last_peered":"2024-01-11T09:27:20.981876+0000","last_clean":"2024-01-11T09:27:20.981876+0000","last_became_active":"2024-01-11T09:27:20.975315+0000","last_became_peered":"2024-01-11T09:27:20.975315+0000","last_unstale":"2024-01-11T09:27:20.981876+0000","last_undegraded":"2024-01-11T09:27:20.981876+0000","last_fullsized":"2024-01-11T09:27:20.981876+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:30:31.681173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.981865+0000","last_change":"2024-01-11T09:27:21.977667+0000","last_active":"2024-01-11T09:27:21.981865+0000","last_peered":"2024-01-11T09:27:21.981865+0000","last_clean":"2024-01-11T09:27:21.981865+0000","last_became_active":"2024-01-11T09:27:21.977361+0000","last_became_peered":"2024-01-11T09:27:21.977361+0000","last_unstale":"2024-01-11T09:27:21.981865+0000","last_undegraded":"2024-01-11T09:27:21.981865+0000","last_fullsized":"2024-01-11T09:27:21.981865+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:36:03.680138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"180'2","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.001978+0000","last_change":"2024-01-11T09:27:20.978118+0000","last_active":"2024-01-11T09:27:22.001978+0000","last_peered":"2024-01-11T09:27:22.001978+0000","last_clean":"2024-01-11T09:27:22.001978+0000","last_became_active":"2024-01-11T09:27:20.977783+0000","last_became_peered":"2024-01-11T09:27:20.977783+0000","last_unstale":"2024-01-11T09:27:22.001978+0000","last_undegraded":"2024-01-11T09:27:22.001978+0000","last_fullsized":"2024-01-11T09:27:22.001978+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:41:39.741630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.992296+0000","last_change":"2024-01-11T09:27:21.992296+0000","last_active":"2024-01-11T09:27:21.992296+0000","last_peered":"2024-01-11T09:27:21.992296+0000","last_clean":"2024-01-11T09:27:21.992296+0000","last_became_active":"2024-01-11T09:27:21.992013+0000","last_became_peered":"2024-01-11T09:27:21.992013+0000","last_unstale":"2024-01-11T09:27:21.992296+0000","last_undegraded":"2024-01-11T09:27:21.992296+0000","last_fullsized":"2024-01-11T09:27:21.992296+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:53:56.792066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.982312+0000","last_change":"2024-01-11T09:27:20.977522+0000","last_active":"2024-01-11T09:27:20.982312+0000","last_peered":"2024-01-11T09:27:20.982312+0000","last_clean":"2024-01-11T09:27:20.982312+0000","last_became_active":"2024-01-11T09:27:20.977258+0000","last_became_peered":"2024-01-11T09:27:20.977258+0000","last_unstale":"2024-01-11T09:27:20.982312+0000","last_undegraded":"2024-01-11T09:27:20.982312+0000","last_fullsized":"2024-01-11T09:27:20.982312+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:32:14.769582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.000639+0000","last_change":"2024-01-11T09:27:22.000639+0000","last_active":"2024-01-11T09:27:22.000639+0000","last_peered":"2024-01-11T09:27:22.000639+0000","last_clean":"2024-01-11T09:27:22.000639+0000","last_became_active":"2024-01-11T09:27:22.000059+0000","last_became_peered":"2024-01-11T09:27:22.000059+0000","last_unstale":"2024-01-11T09:27:22.000639+0000","last_undegraded":"2024-01-11T09:27:22.000639+0000","last_fullsized":"2024-01-11T09:27:22.000639+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:55:25.729750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.981679+0000","last_change":"2024-01-11T09:27:20.978225+0000","last_active":"2024-01-11T09:27:20.981679+0000","last_peered":"2024-01-11T09:27:20.981679+0000","last_clean":"2024-01-11T09:27:20.981679+0000","last_became_active":"2024-01-11T09:27:20.977486+0000","last_became_peered":"2024-01-11T09:27:20.977486+0000","last_unstale":"2024-01-11T09:27:20.981679+0000","last_undegraded":"2024-01-11T09:27:20.981679+0000","last_fullsized":"2024-01-11T09:27:20.981679+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:14:58.043943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.998008+0000","last_change":"2024-01-11T09:27:21.998008+0000","last_active":"2024-01-11T09:27:21.998008+0000","last_peered":"2024-01-11T09:27:21.998008+0000","last_clean":"2024-01-11T09:27:21.998008+0000","last_became_active":"2024-01-11T09:27:21.997325+0000","last_became_peered":"2024-01-11T09:27:21.997325+0000","last_unstale":"2024-01-11T09:27:21.998008+0000","last_undegraded":"2024-01-11T09:27:21.998008+0000","last_fullsized":"2024-01-11T09:27:21.998008+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:41:40.748533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.982108+0000","last_change":"2024-01-11T09:27:20.976842+0000","last_active":"2024-01-11T09:27:20.982108+0000","last_peered":"2024-01-11T09:27:20.982108+0000","last_clean":"2024-01-11T09:27:20.982108+0000","last_became_active":"2024-01-11T09:27:20.976581+0000","last_became_peered":"2024-01-11T09:27:20.976581+0000","last_unstale":"2024-01-11T09:27:20.982108+0000","last_undegraded":"2024-01-11T09:27:20.982108+0000","last_fullsized":"2024-01-11T09:27:20.982108+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:34:52.519030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.999428+0000","last_change":"2024-01-11T09:27:21.998616+0000","last_active":"2024-01-11T09:27:21.999428+0000","last_peered":"2024-01-11T09:27:21.999428+0000","last_clean":"2024-01-11T09:27:21.999428+0000","last_became_active":"2024-01-11T09:27:21.995870+0000","last_became_peered":"2024-01-11T09:27:21.995870+0000","last_unstale":"2024-01-11T09:27:21.999428+0000","last_undegraded":"2024-01-11T09:27:21.999428+0000","last_fullsized":"2024-01-11T09:27:21.999428+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:04:20.823816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"180'2","reported_seq":12,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.005486+0000","last_change":"2024-01-11T09:27:20.969327+0000","last_active":"2024-01-11T09:27:22.005486+0000","last_peered":"2024-01-11T09:27:22.005486+0000","last_clean":"2024-01-11T09:27:22.005486+0000","last_became_active":"2024-01-11T09:27:20.969056+0000","last_became_peered":"2024-01-11T09:27:20.969056+0000","last_unstale":"2024-01-11T09:27:22.005486+0000","last_undegraded":"2024-01-11T09:27:22.005486+0000","last_fullsized":"2024-01-11T09:27:22.005486+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:05:05.483230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.996149+0000","last_change":"2024-01-11T09:27:21.996149+0000","last_active":"2024-01-11T09:27:21.996149+0000","last_peered":"2024-01-11T09:27:21.996149+0000","last_clean":"2024-01-11T09:27:21.996149+0000","last_became_active":"2024-01-11T09:27:21.995900+0000","last_became_peered":"2024-01-11T09:27:21.995900+0000","last_unstale":"2024-01-11T09:27:21.996149+0000","last_undegraded":"2024-01-11T09:27:21.996149+0000","last_fullsized":"2024-01-11T09:27:21.996149+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:53:41.444129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.974295+0000","last_change":"2024-01-11T09:27:20.977152+0000","last_active":"2024-01-11T09:27:21.974295+0000","last_peered":"2024-01-11T09:27:21.974295+0000","last_clean":"2024-01-11T09:27:21.974295+0000","last_became_active":"2024-01-11T09:27:20.976589+0000","last_became_peered":"2024-01-11T09:27:20.976589+0000","last_unstale":"2024-01-11T09:27:21.974295+0000","last_undegraded":"2024-01-11T09:27:21.974295+0000","last_fullsized":"2024-01-11T09:27:21.974295+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:05:54.863945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.001009+0000","last_change":"2024-01-11T09:27:22.001009+0000","last_active":"2024-01-11T09:27:22.001009+0000","last_peered":"2024-01-11T09:27:22.001009+0000","last_clean":"2024-01-11T09:27:22.001009+0000","last_became_active":"2024-01-11T09:27:22.000478+0000","last_became_peered":"2024-01-11T09:27:22.000478+0000","last_unstale":"2024-01-11T09:27:22.001009+0000","last_undegraded":"2024-01-11T09:27:22.001009+0000","last_fullsized":"2024-01-11T09:27:22.001009+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:36:03.680138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.977794+0000","last_change":"2024-01-11T09:27:20.972947+0000","last_active":"2024-01-11T09:27:20.977794+0000","last_peered":"2024-01-11T09:27:20.977794+0000","last_clean":"2024-01-11T09:27:20.977794+0000","last_became_active":"2024-01-11T09:27:20.972716+0000","last_became_peered":"2024-01-11T09:27:20.972716+0000","last_unstale":"2024-01-11T09:27:20.977794+0000","last_undegraded":"2024-01-11T09:27:20.977794+0000","last_fullsized":"2024-01-11T09:27:20.977794+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:03:03.958979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.992469+0000","last_change":"2024-01-11T09:27:21.988207+0000","last_active":"2024-01-11T09:27:21.992469+0000","last_peered":"2024-01-11T09:27:21.992469+0000","last_clean":"2024-01-11T09:27:21.992469+0000","last_became_active":"2024-01-11T09:27:21.987891+0000","last_became_peered":"2024-01-11T09:27:21.987891+0000","last_unstale":"2024-01-11T09:27:21.992469+0000","last_undegraded":"2024-01-11T09:27:21.992469+0000","last_fullsized":"2024-01-11T09:27:21.992469+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:49:13.322719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.970796+0000","last_change":"2024-01-11T09:27:20.969788+0000","last_active":"2024-01-11T09:27:21.970796+0000","last_peered":"2024-01-11T09:27:21.970796+0000","last_clean":"2024-01-11T09:27:21.970796+0000","last_became_active":"2024-01-11T09:27:20.969551+0000","last_became_peered":"2024-01-11T09:27:20.969551+0000","last_unstale":"2024-01-11T09:27:21.970796+0000","last_undegraded":"2024-01-11T09:27:21.970796+0000","last_fullsized":"2024-01-11T09:27:21.970796+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:40:29.761181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.006238+0000","last_change":"2024-01-11T09:27:22.001242+0000","last_active":"2024-01-11T09:27:22.006238+0000","last_peered":"2024-01-11T09:27:22.006238+0000","last_clean":"2024-01-11T09:27:22.006238+0000","last_became_active":"2024-01-11T09:27:21.996076+0000","last_became_peered":"2024-01-11T09:27:21.996076+0000","last_unstale":"2024-01-11T09:27:22.006238+0000","last_undegraded":"2024-01-11T09:27:22.006238+0000","last_fullsized":"2024-01-11T09:27:22.006238+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:58:50.682338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.973327+0000","last_change":"2024-01-11T09:27:20.979567+0000","last_active":"2024-01-11T09:27:21.973327+0000","last_peered":"2024-01-11T09:27:21.973327+0000","last_clean":"2024-01-11T09:27:21.973327+0000","last_became_active":"2024-01-11T09:27:20.979231+0000","last_became_peered":"2024-01-11T09:27:20.979231+0000","last_unstale":"2024-01-11T09:27:21.973327+0000","last_undegraded":"2024-01-11T09:27:21.973327+0000","last_fullsized":"2024-01-11T09:27:21.973327+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:38:03.932159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.002798+0000","last_change":"2024-01-11T09:27:21.999756+0000","last_active":"2024-01-11T09:27:22.002798+0000","last_peered":"2024-01-11T09:27:22.002798+0000","last_clean":"2024-01-11T09:27:22.002798+0000","last_became_active":"2024-01-11T09:27:21.999246+0000","last_became_peered":"2024-01-11T09:27:21.999246+0000","last_unstale":"2024-01-11T09:27:22.002798+0000","last_undegraded":"2024-01-11T09:27:22.002798+0000","last_fullsized":"2024-01-11T09:27:22.002798+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:34:53.525932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.981266+0000","last_change":"2024-01-11T09:27:20.976708+0000","last_active":"2024-01-11T09:27:20.981266+0000","last_peered":"2024-01-11T09:27:20.981266+0000","last_clean":"2024-01-11T09:27:20.981266+0000","last_became_active":"2024-01-11T09:27:20.976346+0000","last_became_peered":"2024-01-11T09:27:20.976346+0000","last_unstale":"2024-01-11T09:27:20.981266+0000","last_undegraded":"2024-01-11T09:27:20.981266+0000","last_fullsized":"2024-01-11T09:27:20.981266+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:23:58.946042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.989268+0000","last_change":"2024-01-11T09:27:21.989268+0000","last_active":"2024-01-11T09:27:21.989268+0000","last_peered":"2024-01-11T09:27:21.989268+0000","last_clean":"2024-01-11T09:27:21.989268+0000","last_became_active":"2024-01-11T09:27:21.988991+0000","last_became_peered":"2024-01-11T09:27:21.988991+0000","last_unstale":"2024-01-11T09:27:21.989268+0000","last_undegraded":"2024-01-11T09:27:21.989268+0000","last_fullsized":"2024-01-11T09:27:21.989268+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:22:21.100031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.977813+0000","last_change":"2024-01-11T09:27:20.975000+0000","last_active":"2024-01-11T09:27:20.977813+0000","last_peered":"2024-01-11T09:27:20.977813+0000","last_clean":"2024-01-11T09:27:20.977813+0000","last_became_active":"2024-01-11T09:27:20.974711+0000","last_became_peered":"2024-01-11T09:27:20.974711+0000","last_unstale":"2024-01-11T09:27:20.977813+0000","last_undegraded":"2024-01-11T09:27:20.977813+0000","last_fullsized":"2024-01-11T09:27:20.977813+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:13:27.228772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.008096+0000","last_change":"2024-01-11T09:27:22.003519+0000","last_active":"2024-01-11T09:27:22.008096+0000","last_peered":"2024-01-11T09:27:22.008096+0000","last_clean":"2024-01-11T09:27:22.008096+0000","last_became_active":"2024-01-11T09:27:22.003257+0000","last_became_peered":"2024-01-11T09:27:22.003257+0000","last_unstale":"2024-01-11T09:27:22.008096+0000","last_undegraded":"2024-01-11T09:27:22.008096+0000","last_fullsized":"2024-01-11T09:27:22.008096+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:19:54.081327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.973087+0000","last_change":"2024-01-11T09:27:20.977340+0000","last_active":"2024-01-11T09:27:21.973087+0000","last_peered":"2024-01-11T09:27:21.973087+0000","last_clean":"2024-01-11T09:27:21.973087+0000","last_became_active":"2024-01-11T09:27:20.977053+0000","last_became_peered":"2024-01-11T09:27:20.977053+0000","last_unstale":"2024-01-11T09:27:21.973087+0000","last_undegraded":"2024-01-11T09:27:21.973087+0000","last_fullsized":"2024-01-11T09:27:21.973087+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:41:26.423126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:22.008240+0000","last_change":"2024-01-11T09:27:22.005066+0000","last_active":"2024-01-11T09:27:22.008240+0000","last_peered":"2024-01-11T09:27:22.008240+0000","last_clean":"2024-01-11T09:27:22.008240+0000","last_became_active":"2024-01-11T09:27:22.004011+0000","last_became_peered":"2024-01-11T09:27:22.004011+0000","last_unstale":"2024-01-11T09:27:22.008240+0000","last_undegraded":"2024-01-11T09:27:22.008240+0000","last_fullsized":"2024-01-11T09:27:22.008240+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:20.951746+0000","last_clean_scrub_stamp":"2024-01-11T09:27:20.951746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:17:40.366748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-01-11T09:27:21.975052+0000","last_change":"2024-01-11T09:27:20.982648+0000","last_active":"2024-01-11T09:27:21.975052+0000","last_peered":"2024-01-11T09:27:21.975052+0000","last_clean":"2024-01-11T09:27:21.975052+0000","last_became_active":"2024-01-11T09:27:20.982380+0000","last_became_peered":"2024-01-11T09:27:20.982380+0000","last_unstale":"2024-01-11T09:27:21.975052+0000","last_undegraded":"2024-01-11T09:27:21.975052+0000","last_fullsized":"2024-01-11T09:27:21.975052+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:27:19.944843+0000","last_clean_scrub_stamp":"2024-01-11T09:27:19.944843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:27:04.596926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"171'311","reported_seq":976,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-01-11T09:27:20.965872+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:27:20.965872+0000","last_peered":"2024-01-11T09:27:20.965872+0000","last_clean":"2024-01-11T09:27:20.965872+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:27:20.965872+0000","last_undegraded":"2024-01-11T09:27:20.965872+0000","last_fullsized":"2024-01-11T09:27:20.965872+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1808,"num_read_kb":3778,"num_write":511,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1808,"num_read_kb":3778,"num_write":511,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":311,"ondisk_log_size":311,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575760,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312380,"kb_used_data":2804,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059460,"statfs":{"total":96636764160,"available":96316887040,"internally_reserved":0,"allocated":2871296,"data_stored":1657752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.72399999999999998,"15min":0.72599999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.91500000000000004,"5min":0.98199999999999998,"15min":0.99299999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.72199999999999998,"15min":0.72999999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.89200000000000002,"5min":0.99299999999999999,"15min":1.028},"last":0.76000000000000001}]},{"osd":1,"last update":"Thu Jan 11 09:26:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.71299999999999997,"15min":0.71799999999999997},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.42399999999999999},"max":{"1min":0.81200000000000006,"5min":1.0009999999999999,"15min":1.034},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.748,"15min":0.749},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.90900000000000003,"5min":0.96099999999999997,"15min":1.0409999999999999},"last":0.80800000000000005}]},{"osd":2,"last update":"Thu Jan 11 09:26:51 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.73399999999999999,"15min":0.748},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.46899999999999997},"max":{"1min":0.89100000000000001,"5min":1.0089999999999999,"15min":1.0880000000000001},"last":0.63},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.72699999999999998,"15min":0.73399999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50800000000000001},"max":{"1min":0.84599999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.68799999999999994}]},{"osd":3,"last update":"Thu Jan 11 09:26:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.99399999999999999,"5min":1.046,"15min":1.056},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.75,"15min":0.76100000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.92400000000000004,"5min":1.006,"15min":1.012},"last":0.752}]},{"osd":4,"last update":"Thu Jan 11 09:26:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59899999999999998,"15min":0.60199999999999998},"min":{"1min":0.47299999999999998,"5min":0.45300000000000001,"15min":0.40600000000000003},"max":{"1min":0.871,"5min":0.871,"15min":1.004},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64100000000000001,"15min":0.64300000000000002},"min":{"1min":0.433,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0009999999999999,"5min":1.016,"15min":1.0940000000000001},"last":0.78800000000000003}]},{"osd":5,"last update":"Thu Jan 11 09:26:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.63700000000000001,"15min":0.64200000000000002},"min":{"1min":0.503,"5min":0.433,"15min":0.433},"max":{"1min":0.84999999999999998,"5min":0.93400000000000005,"15min":1.0309999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.68600000000000005,"15min":0.67900000000000005},"min":{"1min":0.56000000000000005,"5min":0.41999999999999998,"15min":0.41299999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.0680000000000001},"last":0.71199999999999997}]},{"osd":6,"last update":"Thu Jan 11 09:26:51 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.76800000000000002,"15min":0.77400000000000002},"min":{"1min":0.59999999999999998,"5min":0.50900000000000001,"15min":0.47899999999999998},"max":{"1min":0.88100000000000001,"5min":0.97399999999999998,"15min":1.1200000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.749,"15min":0.754},"min":{"1min":0.46400000000000002,"5min":0.44600000000000001,"15min":0.40100000000000002},"max":{"1min":1.0389999999999999,"5min":1.073,"15min":1.073},"last":0.82699999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575759,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307684,"kb_used_data":2204,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064156,"statfs":{"total":96636764160,"available":96321695744,"internally_reserved":0,"allocated":2256896,"data_stored":1065156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.70599999999999996,"15min":0.70499999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.40500000000000003},"max":{"1min":0.91200000000000003,"5min":0.98099999999999998,"15min":1.095},"last":0.748},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.71799999999999997,"15min":0.71999999999999997},"min":{"1min":0.53500000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.92700000000000005,"5min":0.97899999999999998,"15min":1.0489999999999999},"last":0.93200000000000005}]},{"osd":1,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.72599999999999998,"15min":0.72699999999999998},"min":{"1min":0.54100000000000004,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93999999999999995,"5min":0.96499999999999997,"15min":1.077},"last":0.87},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71399999999999997,"15min":0.71899999999999997},"min":{"1min":0.49099999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.94299999999999995,"5min":0.96999999999999997,"15min":1.01},"last":0.80000000000000004}]},{"osd":2,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.74399999999999999,"15min":0.73599999999999999},"min":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.49399999999999999},"max":{"1min":1.006,"5min":1.0129999999999999,"15min":1.0309999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72999999999999998,"15min":0.75},"min":{"1min":0.54500000000000004,"5min":0.51600000000000001,"15min":0.505},"max":{"1min":0.89300000000000002,"5min":0.94599999999999995,"15min":1.161},"last":0.83499999999999996}]},{"osd":3,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76200000000000001,"15min":0.76000000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.47499999999999998},"max":{"1min":0.91300000000000003,"5min":0.999,"15min":1.0329999999999999},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76300000000000001,"15min":0.76700000000000002},"min":{"1min":0.57199999999999995,"5min":0.53700000000000003,"15min":0.503},"max":{"1min":0.91300000000000003,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.95699999999999996}]},{"osd":4,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.45900000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76100000000000001,"5min":0.82999999999999996,"15min":1.0469999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85799999999999998,"5min":0.97199999999999998,"15min":1.0569999999999999},"last":0.70399999999999996}]},{"osd":5,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.67600000000000005,"15min":0.66700000000000004},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.39200000000000002},"max":{"1min":0.89700000000000002,"5min":1.032,"15min":1.032},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.69999999999999996,"15min":0.71199999999999997},"min":{"1min":0.441,"5min":0.441,"15min":0.41699999999999998},"max":{"1min":0.93300000000000005,"5min":1.052,"15min":1.1299999999999999},"last":0.55100000000000005}]},{"osd":7,"last update":"Thu Jan 11 09:26:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.71099999999999997,"15min":0.71899999999999997},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.435},"max":{"1min":0.96799999999999997,"5min":1.034,"15min":1.1080000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.75800000000000001,"15min":0.75900000000000001},"min":{"1min":0.501,"5min":0.5,"15min":0.47299999999999998},"max":{"1min":0.95499999999999996,"5min":1.081,"15min":1.0840000000000001},"last":0.73599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575761,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307684,"kb_used_data":2204,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064156,"statfs":{"total":96636764160,"available":96321695744,"internally_reserved":0,"allocated":2256896,"data_stored":1065168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.51100000000000001,"15min":0.51800000000000002},"min":{"1min":0.33200000000000002,"5min":0.317,"15min":0.30199999999999999},"max":{"1min":0.71499999999999997,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.442},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.39900000000000002},"max":{"1min":0.74099999999999999,"5min":0.81399999999999995,"15min":0.90200000000000002},"last":0.58499999999999996}]},{"osd":2,"last update":"Thu Jan 11 09:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.495,"15min":0.505},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.83699999999999997},"last":0.622},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.36899999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.91900000000000004},"last":0.495}]},{"osd":3,"last update":"Thu Jan 11 09:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.64400000000000002,"15min":0.64600000000000002},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84699999999999998,"5min":1.002,"15min":1.002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58499999999999996,"15min":0.58399999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.40500000000000003},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.55500000000000005}]},{"osd":4,"last update":"Thu Jan 11 09:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.81799999999999995,"15min":0.81000000000000005},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.46999999999999997},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.85999999999999999,"15min":0.86299999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.085,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.81899999999999995}]},{"osd":5,"last update":"Thu Jan 11 09:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.81299999999999994,"15min":0.81100000000000005},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.55400000000000005},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.72699999999999998}]},{"osd":6,"last update":"Thu Jan 11 09:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.84999999999999998,"15min":0.84199999999999997},"min":{"1min":0.629,"5min":0.61599999999999999,"15min":0.57799999999999996},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.85299999999999998,"15min":0.84099999999999997},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.54400000000000004},"max":{"1min":1.0329999999999999,"5min":1.1499999999999999,"15min":1.2529999999999999},"last":0.88900000000000001}]},{"osd":7,"last update":"Thu Jan 11 09:26:56 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.875,"15min":0.86399999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.153,"5min":1.2130000000000001,"15min":1.218},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.84999999999999998,"15min":0.84699999999999998},"min":{"1min":0.626,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.1839999999999999,"5min":1.1950000000000001,"15min":1.244},"last":0.68799999999999994}]}]},{"osd":0,"up_from":13,"seq":55834575760,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308260,"kb_used_data":2780,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063580,"statfs":{"total":96636764160,"available":96321105920,"internally_reserved":0,"allocated":2846720,"data_stored":1655502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.52400000000000002,"15min":0.52000000000000002},"min":{"1min":0.311,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.70199999999999996,"5min":0.751,"15min":0.79600000000000004},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60299999999999998,"15min":0.59699999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.42499999999999999},"max":{"1min":0.72399999999999998,"5min":0.83699999999999997,"15min":0.94599999999999995},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55300000000000005,"15min":0.55400000000000005},"min":{"1min":0.41399999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.80600000000000005,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55200000000000005,"15min":0.57499999999999996},"min":{"1min":0.41399999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89900000000000002},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70599999999999996,"5min":0.79400000000000004,"15min":0.90600000000000003},"last":0.496},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63400000000000001,"15min":0.64300000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.79900000000000004,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.55300000000000005}]},{"osd":4,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79200000000000004,"15min":0.81699999999999995},"min":{"1min":0.59599999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.075,"5min":1.075,"15min":1.4239999999999999},"last":1.036},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.80800000000000005,"15min":0.81200000000000006},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.1659999999999999,"5min":1.2050000000000001,"15min":1.2749999999999999},"last":1.0740000000000001}]},{"osd":5,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.81599999999999995,"15min":0.83299999999999996},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.57399999999999995},"max":{"1min":1.0249999999999999,"5min":1.1140000000000001,"15min":1.391},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.84199999999999997,"15min":0.84099999999999997},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.151,"5min":1.151,"15min":1.484},"last":0.93200000000000005}]},{"osd":6,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.82599999999999996,"15min":0.82899999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.353},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.85399999999999998,"15min":0.86399999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.3069999999999999},"last":0.89400000000000002}]},{"osd":7,"last update":"Thu Jan 11 09:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.86799999999999999,"15min":0.875},"min":{"1min":0.60899999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.125,"5min":1.2010000000000001,"15min":1.246},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.872,"15min":0.86399999999999999},"min":{"1min":0.67300000000000004,"5min":0.63600000000000001,"15min":0.59999999999999998},"max":{"1min":1.048,"5min":1.157,"15min":1.272},"last":0.86899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575759,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311780,"kb_used_data":2204,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060060,"statfs":{"total":96636764160,"available":96317501440,"internally_reserved":0,"allocated":2256896,"data_stored":1065224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.497,"15min":0.5},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55100000000000005,"15min":0.55400000000000005},"min":{"1min":0.34599999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.49299999999999999}]},{"osd":1,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.45100000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79700000000000004,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60899999999999999,"15min":0.60699999999999998},"min":{"1min":0.41199999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.82699999999999996,"5min":0.93600000000000005,"15min":1},"last":0.45300000000000001}]},{"osd":3,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54900000000000004,"15min":0.55700000000000005},"min":{"1min":0.40400000000000003,"5min":0.37,"15min":0.34799999999999998},"max":{"1min":0.79500000000000004,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.439},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65600000000000003,"15min":0.66100000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.57099999999999995}]},{"osd":4,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.80700000000000005,"15min":0.81399999999999995},"min":{"1min":0.68200000000000005,"5min":0.55300000000000005,"15min":0.54000000000000004},"max":{"1min":1.1220000000000001,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81299999999999994,"15min":0.79800000000000004},"min":{"1min":0.69199999999999995,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.036,"5min":1.391,"15min":1.391},"last":0.78800000000000003}]},{"osd":5,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.878,"15min":0.86899999999999999},"min":{"1min":0.73599999999999999,"5min":0.621,"15min":0.61699999999999999},"max":{"1min":1.1240000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.88800000000000001,"15min":0.86799999999999999},"min":{"1min":0.72099999999999997,"5min":0.627,"15min":0.54700000000000004},"max":{"1min":1.238,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.83899999999999997}]},{"osd":6,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.84099999999999997,"15min":0.82999999999999996},"min":{"1min":0.71199999999999997,"5min":0.57399999999999995,"15min":0.48999999999999999},"max":{"1min":1.1180000000000001,"5min":1.534,"15min":1.534},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.72099999999999997,"5min":0.59199999999999997,"15min":0.56999999999999995},"max":{"1min":1.127,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.86699999999999999}]},{"osd":7,"last update":"Thu Jan 11 09:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.66500000000000004,"5min":0.63100000000000001,"15min":0.58799999999999997},"max":{"1min":1.2010000000000001,"5min":1.55,"15min":1.55},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.90400000000000003,"15min":0.89400000000000002},"min":{"1min":0.67900000000000005,"5min":0.54900000000000004,"15min":0.53600000000000003},"max":{"1min":1.087,"5min":1.593,"15min":1.593},"last":0.84899999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575759,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299492,"kb_used_data":2204,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072348,"statfs":{"total":96636764160,"available":96330084352,"internally_reserved":0,"allocated":2256896,"data_stored":1065168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52000000000000002,"15min":0.504},"min":{"1min":0.36499999999999999,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56799999999999995,"15min":0.57899999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.42699999999999999}]},{"osd":1,"last update":"Thu Jan 11 09:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.51300000000000001,"15min":0.51600000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78200000000000003,"5min":0.88100000000000001,"15min":0.93100000000000005},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60199999999999998,"15min":0.59099999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.36299999999999999},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 11 09:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":0.373},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.58899999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.80300000000000005,"15min":0.78800000000000003},"min":{"1min":0.56399999999999995,"5min":0.48199999999999998,"15min":0.44900000000000001},"max":{"1min":1.075,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.80700000000000005,"15min":0.80200000000000005},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.52300000000000002},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.749}]},{"osd":5,"last update":"Thu Jan 11 09:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.81999999999999995,"15min":0.81799999999999995},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.84999999999999998,"15min":0.83599999999999997},"min":{"1min":0.67300000000000004,"5min":0.63800000000000001,"15min":0.58799999999999997},"max":{"1min":2.093,"5min":2.093,"15min":2.093},"last":0.83599999999999997}]},{"osd":6,"last update":"Thu Jan 11 09:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.872,"15min":0.85199999999999998},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.58299999999999996},"max":{"1min":2.2290000000000001,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.83199999999999996,"15min":0.81599999999999995},"min":{"1min":0.626,"5min":0.60099999999999998,"15min":0.49099999999999999},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.71399999999999997}]},{"osd":7,"last update":"Thu Jan 11 09:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.871,"15min":0.86199999999999999},"min":{"1min":0.68000000000000005,"5min":0.623,"15min":0.51800000000000002},"max":{"1min":1.038,"5min":1.1659999999999999,"15min":1.196},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.89700000000000002,"15min":0.88100000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.56499999999999995},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.79000000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303600,"kb_used_data":2216,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068240,"statfs":{"total":96636764160,"available":96325877760,"internally_reserved":0,"allocated":2269184,"data_stored":1066740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.77300000000000002,"15min":0.77700000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.35099999999999998},"max":{"1min":0.86899999999999999,"5min":1.0369999999999999,"15min":1.048},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.76200000000000001,"15min":0.77200000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.48999999999999999},"max":{"1min":0.98299999999999998,"5min":1.018,"15min":1.018},"last":0.72999999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.72099999999999997,"15min":0.73099999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.72499999999999998,"15min":0.73599999999999999},"min":{"1min":0.47299999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0009999999999999,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.66200000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.753,"15min":0.75600000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.96799999999999997,"5min":1,"15min":1},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.751,"15min":0.752},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.46100000000000002},"max":{"1min":1.0349999999999999,"5min":1.054,"15min":1.0760000000000001},"last":0.80500000000000005}]},{"osd":3,"last update":"Thu Jan 11 09:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.71699999999999997,"15min":0.72899999999999998},"min":{"1min":0.48199999999999998,"5min":0.442,"15min":0.442},"max":{"1min":0.93200000000000005,"5min":0.95399999999999996,"15min":1.0109999999999999},"last":0.621},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.78400000000000003,"15min":0.78100000000000003},"min":{"1min":0.55400000000000005,"5min":0.54400000000000004,"15min":0.53700000000000003},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.63700000000000001}]},{"osd":5,"last update":"Thu Jan 11 09:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61499999999999999,"15min":0.61699999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.86599999999999999,"5min":0.89300000000000002,"15min":0.94999999999999996},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.64200000000000002,"15min":0.629},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.53600000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.65900000000000003,"15min":0.65300000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.505},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.69999999999999996,"15min":0.69799999999999995},"min":{"1min":0.45000000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.92600000000000005,"5min":0.95899999999999996,"15min":0.96399999999999997},"last":0.77700000000000002}]},{"osd":7,"last update":"Thu Jan 11 09:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.69299999999999995,"15min":0.69399999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.92300000000000004,"5min":1.139,"15min":1.139},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.746,"15min":0.74299999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.67900000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575760,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307688,"kb_used_data":2208,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064152,"statfs":{"total":96636764160,"available":96321691648,"internally_reserved":0,"allocated":2260992,"data_stored":1065710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.73499999999999999,"15min":0.73699999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.77700000000000002,"5min":1.006,"15min":1.0229999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.76100000000000001,"15min":0.75600000000000001},"min":{"1min":0.46500000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.84799999999999998,"5min":0.97599999999999998,"15min":1.1020000000000001},"last":0.73299999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.745,"15min":0.74099999999999999},"min":{"1min":0.50700000000000001,"5min":0.438,"15min":0.41399999999999998},"max":{"1min":0.997,"5min":0.997,"15min":1.046},"last":0.753},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.74399999999999999,"15min":0.75800000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82399999999999995,"5min":0.997,"15min":1.0249999999999999},"last":0.66200000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.72599999999999998,"15min":0.747},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.76600000000000001,"5min":1.014,"15min":1.0569999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.76400000000000001,"15min":0.75600000000000001},"min":{"1min":0.51700000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94599999999999995,"5min":0.97999999999999998,"15min":1.012},"last":0.64500000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.76600000000000001,"15min":0.76900000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.85799999999999998,"5min":1.016,"15min":1.016},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.76900000000000002,"15min":0.77300000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.45800000000000002},"max":{"1min":0.91400000000000003,"5min":0.99199999999999999,"15min":1.0609999999999999},"last":0.68600000000000005}]},{"osd":4,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.622,"15min":0.61199999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.64200000000000002,"15min":0.63200000000000001},"min":{"1min":0.48299999999999998,"5min":0.45800000000000002,"15min":0.38900000000000001},"max":{"1min":0.88900000000000001,"5min":0.91300000000000003,"15min":0.94899999999999995},"last":0.78400000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.64700000000000002,"15min":0.64300000000000002},"min":{"1min":0.443,"5min":0.433,"15min":0.39400000000000002},"max":{"1min":0.874,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.44500000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.80700000000000005}]},{"osd":7,"last update":"Thu Jan 11 09:26:37 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.75,"15min":0.73599999999999999},"min":{"1min":0.53700000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.89300000000000002,"5min":0.997,"15min":1.0920000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.78900000000000003,"15min":0.78100000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96899999999999997,"5min":1.01,"15min":1.137},"last":0.82799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:27:26.889 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-11T09:27:26.890 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-11T09:27:27.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.133+0000 7fa5e11ff700 1 Processor -- start 2024-01-11T09:27:27.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.133+0000 7fa5e11ff700 1 -- start start 2024-01-11T09:27:27.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.134+0000 7fa5e11ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5dc0b2b90 0x7fa5dc159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:27.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.134+0000 7fa5e11ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa5dc0b07b0 0x7fa5dc159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:27.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.135+0000 7fa5e11ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5dc0cd730 con 0x7fa5dc0b03e0 2024-01-11T09:27:27.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.135+0000 7fa5e11ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5dc0ccb20 con 0x7fa5dc0b07b0 2024-01-11T09:27:27.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.135+0000 7fa5e11ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5dc0cce00 con 0x7fa5dc0b2b90 2024-01-11T09:27:27.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.135+0000 7fa5dad9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa5dc0b03e0 0x7fa5dc0b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53572/0 (socket says 172.21.15.27:53572) 2024-01-11T09:27:27.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.135+0000 7fa5dad9d700 1 -- 172.21.15.27:0/2044665595 learned_addr learned my addr 172.21.15.27:0/2044665595 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.135+0000 7fa5db59e700 1 --2- 172.21.15.27:0/2044665595 >> v2:172.21.15.77:3300/0 conn(0x7fa5dc0b07b0 0x7fa5dc159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:27.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.135+0000 7fa5da59c700 1 --2- 172.21.15.27:0/2044665595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5dc0b2b90 0x7fa5dc159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.135+0000 7fa5d9d9b700 1 -- 172.21.15.27:0/2044665595 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418039928 0 0) 0x7fa5dc0cd730 con 0x7fa5dc0b03e0 2024-01-11T09:27:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.135+0000 7fa5d9d9b700 1 -- 172.21.15.27:0/2044665595 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5c4003570 con 0x7fa5dc0b03e0 2024-01-11T09:27:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.135+0000 7fa5d9d9b700 1 -- 172.21.15.27:0/2044665595 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 944117190 0 0) 0x7fa5c4003570 con 0x7fa5dc0b03e0 2024-01-11T09:27:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.135+0000 7fa5d9d9b700 1 -- 172.21.15.27:0/2044665595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5dc0b2b90 msgr2=0x7fa5dc159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.135+0000 7fa5d9d9b700 1 --2- 172.21.15.27:0/2044665595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5dc0b2b90 0x7fa5dc159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.136+0000 7fa5d9d9b700 1 -- 172.21.15.27:0/2044665595 >> v2:172.21.15.77:3300/0 conn(0x7fa5dc0b07b0 msgr2=0x7fa5dc159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.136+0000 7fa5d9d9b700 1 --2- 172.21.15.27:0/2044665595 >> v2:172.21.15.77:3300/0 conn(0x7fa5dc0b07b0 0x7fa5dc159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.136+0000 7fa5d9d9b700 1 -- 172.21.15.27:0/2044665595 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5dc0cd0e0 con 0x7fa5dc0b03e0 2024-01-11T09:27:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.136+0000 7fa5d9d9b700 1 -- 172.21.15.27:0/2044665595 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fa5cc001470 con 0x7fa5dc0b03e0 2024-01-11T09:27:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.136+0000 7fa5d9d9b700 1 -- 172.21.15.27:0/2044665595 >> v1:172.21.15.27:6789/0 conn(0x7fa5dc0b03e0 legacy=0x7fa5dc0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.136+0000 7fa5d9d9b700 1 --2- 172.21.15.27:0/2044665595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5c4003ae0 0x7fa5c4003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.136+0000 7fa5d9d9b700 1 --2- 172.21.15.27:0/2044665595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5c4004810 0x7fa5c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.136+0000 7fa5d9d9b700 1 --2- 172.21.15.27:0/2044665595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5dc0b2b90 0x7fa5c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.136+0000 7fa5d9d9b700 1 -- 172.21.15.27:0/2044665595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5dc0ccb20 con 0x7fa5c4003ae0 2024-01-11T09:27:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.136+0000 7fa5d9d9b700 1 -- 172.21.15.27:0/2044665595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5dc0cce00 con 0x7fa5c4004810 2024-01-11T09:27:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.136+0000 7fa5d9d9b700 1 -- 172.21.15.27:0/2044665595 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c400bb10 con 0x7fa5dc0b2b90 2024-01-11T09:27:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.136+0000 7fa5dad9d700 1 --2- 172.21.15.27:0/2044665595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5c4003ae0 0x7fa5c4003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.136+0000 7fa5da59c700 1 --2- 172.21.15.27:0/2044665595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5c4004810 0x7fa5c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.136+0000 7fa5db59e700 1 --2- 172.21.15.27:0/2044665595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5dc0b2b90 0x7fa5c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.136+0000 7fa5dad9d700 1 -- 172.21.15.27:0/2044665595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5dc0b2b90 msgr2=0x7fa5c40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.136+0000 7fa5dad9d700 1 --2- 172.21.15.27:0/2044665595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5dc0b2b90 0x7fa5c40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.136+0000 7fa5dad9d700 1 -- 172.21.15.27:0/2044665595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5c4004810 msgr2=0x7fa5c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.136+0000 7fa5dad9d700 1 --2- 172.21.15.27:0/2044665595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5c4004810 0x7fa5c4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.136+0000 7fa5dad9d700 1 -- 172.21.15.27:0/2044665595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c4003570 con 0x7fa5c4003ae0 2024-01-11T09:27:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.137+0000 7fa5dad9d700 1 --2- 172.21.15.27:0/2044665595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5c4003ae0 0x7fa5c4003ef0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc00c2d0 tx=0x7fa5cc001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc6eb3de1e0eaee3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.137+0000 7fa5d9d9b700 1 -- 172.21.15.27:0/2044665595 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc00cba0 con 0x7fa5c4003ae0 2024-01-11T09:27:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.137+0000 7fa5d9d9b700 1 -- 172.21.15.27:0/2044665595 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5cc00cd20 con 0x7fa5c4003ae0 2024-01-11T09:27:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.137+0000 7fa5d9d9b700 1 -- 172.21.15.27:0/2044665595 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc005b70 con 0x7fa5c4003ae0 2024-01-11T09:27:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.137+0000 7fa5e11ff700 1 -- 172.21.15.27:0/2044665595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5c4003ae0 msgr2=0x7fa5c4003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.137+0000 7fa5e11ff700 1 --2- 172.21.15.27:0/2044665595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5c4003ae0 0x7fa5c4003ef0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc00c2d0 tx=0x7fa5cc001dc0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.137+0000 7fa5db59e700 1 -- 172.21.15.27:0/2044665595 reap_dead start 2024-01-11T09:27:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.138+0000 7fa5e11ff700 1 -- 172.21.15.27:0/2044665595 shutdown_connections 2024-01-11T09:27:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.138+0000 7fa5e11ff700 1 -- 172.21.15.27:0/2044665595 >> 172.21.15.27:0/2044665595 conn(0x7fa5dc06ed40 msgr2=0x7fa5dc06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.138+0000 7fa5e11ff700 1 -- 172.21.15.27:0/2044665595 shutdown_connections 2024-01-11T09:27:27.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.138+0000 7fa5e11ff700 1 -- 172.21.15.27:0/2044665595 wait complete. 2024-01-11T09:27:27.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.140+0000 7fa5e11ff700 1 Processor -- start 2024-01-11T09:27:27.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.140+0000 7fa5e11ff700 1 -- start start 2024-01-11T09:27:27.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.141+0000 7fa5e11ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5c4003ae0 0x7fa5dc16b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:27.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.141+0000 7fa5e11ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5dc16ab50 0x7fa5dc167d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:27.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.141+0000 7fa5e11ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5dc167550 0x7fa5dc0c64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:27.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.141+0000 7fa5e11ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5dc16b8d0 con 0x7fa5dc167550 2024-01-11T09:27:27.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.141+0000 7fa5e11ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5dc16bbb0 con 0x7fa5c4003ae0 2024-01-11T09:27:27.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.141+0000 7fa5e11ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5dc16be90 con 0x7fa5dc16ab50 2024-01-11T09:27:27.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.142+0000 7fa5db59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5dc167550 0x7fa5dc0c64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:27.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.142+0000 7fa5db59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5dc167550 0x7fa5dc0c64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37700/0 (socket says 172.21.15.27:37700) 2024-01-11T09:27:27.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.142+0000 7fa5db59e700 1 -- 172.21.15.27:0/2914123471 learned_addr learned my addr 172.21.15.27:0/2914123471 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:27.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.142+0000 7fa5dad9d700 1 --2- 172.21.15.27:0/2914123471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5c4003ae0 0x7fa5dc16b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:27.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.142+0000 7fa5da59c700 1 --2- 172.21.15.27:0/2914123471 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5dc16ab50 0x7fa5dc167d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:27.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.142+0000 7fa5db59e700 1 -- 172.21.15.27:0/2914123471 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5dc16ab50 msgr2=0x7fa5dc167d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:27.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.142+0000 7fa5db59e700 1 --2- 172.21.15.27:0/2914123471 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5dc16ab50 0x7fa5dc167d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.142+0000 7fa5db59e700 1 -- 172.21.15.27:0/2914123471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5c4003ae0 msgr2=0x7fa5dc16b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:27.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.142+0000 7fa5db59e700 1 --2- 172.21.15.27:0/2914123471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5c4003ae0 0x7fa5dc16b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.142+0000 7fa5db59e700 1 -- 172.21.15.27:0/2914123471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5dc16c170 con 0x7fa5dc167550 2024-01-11T09:27:27.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.142+0000 7fa5db59e700 1 --2- 172.21.15.27:0/2914123471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5dc167550 0x7fa5dc0c64a0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc06fd20 tx=0x7fa5d400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:27.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.142+0000 7fa5c2ffd700 1 -- 172.21.15.27:0/2914123471 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d400e040 con 0x7fa5dc167550 2024-01-11T09:27:27.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.143+0000 7fa5c2ffd700 1 -- 172.21.15.27:0/2914123471 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d400f040 con 0x7fa5dc167550 2024-01-11T09:27:27.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.143+0000 7fa5c2ffd700 1 -- 172.21.15.27:0/2914123471 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d400fd70 con 0x7fa5dc167550 2024-01-11T09:27:27.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.143+0000 7fa5e11ff700 1 -- 172.21.15.27:0/2914123471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5dc16c450 con 0x7fa5dc167550 2024-01-11T09:27:27.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.143+0000 7fa5e11ff700 1 -- 172.21.15.27:0/2914123471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5dc1682a0 con 0x7fa5dc167550 2024-01-11T09:27:27.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.144+0000 7fa5e11ff700 1 -- 172.21.15.27:0/2914123471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5dc04c3c0 con 0x7fa5dc167550 2024-01-11T09:27:27.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.145+0000 7fa5c2ffd700 1 -- 172.21.15.27:0/2914123471 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa5d4015640 con 0x7fa5dc167550 2024-01-11T09:27:27.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.146+0000 7fa5c2ffd700 1 --2- 172.21.15.27:0/2914123471 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa5c8066f30 0x7fa5c80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:27.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.146+0000 7fa5c2ffd700 1 -- 172.21.15.27:0/2914123471 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fa5d4084e40 con 0x7fa5dc167550 2024-01-11T09:27:27.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.147+0000 7fa5dad9d700 1 --2- 172.21.15.27:0/2914123471 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa5c8066f30 0x7fa5c80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:27.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.147+0000 7fa5dad9d700 1 --2- 172.21.15.27:0/2914123471 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa5c8066f30 0x7fa5c80693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc015040 tx=0x7fa5cc012000 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:27.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.150+0000 7fa5c2ffd700 1 -- 172.21.15.27:0/2914123471 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa5d4051fd0 con 0x7fa5dc167550 2024-01-11T09:27:27.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.438+0000 7fa5e11ff700 1 -- 172.21.15.27:0/2914123471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa5dc0cd3c0 con 0x7fa5dc167550 2024-01-11T09:27:27.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.443+0000 7fa5c2ffd700 1 -- 172.21.15.27:0/2914123471 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v43) v1 ==== 190+0+0 (secure 0 0 0) 0x7fa5d40190e0 con 0x7fa5dc167550 2024-01-11T09:27:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.445+0000 7fa5e11ff700 1 -- 172.21.15.27:0/2914123471 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa5c8066f30 msgr2=0x7fa5c80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.445+0000 7fa5e11ff700 1 --2- 172.21.15.27:0/2914123471 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa5c8066f30 0x7fa5c80693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc015040 tx=0x7fa5cc012000 comp rx=0 tx=0).stop 2024-01-11T09:27:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.445+0000 7fa5e11ff700 1 -- 172.21.15.27:0/2914123471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5dc167550 msgr2=0x7fa5dc0c64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.445+0000 7fa5e11ff700 1 --2- 172.21.15.27:0/2914123471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5dc167550 0x7fa5dc0c64a0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc06fd20 tx=0x7fa5d400b040 comp rx=0 tx=0).stop 2024-01-11T09:27:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.445+0000 7fa5e11ff700 1 -- 172.21.15.27:0/2914123471 shutdown_connections 2024-01-11T09:27:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.445+0000 7fa5e11ff700 1 --2- 172.21.15.27:0/2914123471 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa5c8066f30 0x7fa5c80693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.445+0000 7fa5e11ff700 1 --2- 172.21.15.27:0/2914123471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5dc167550 0x7fa5dc0c64a0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.445+0000 7fa5e11ff700 1 --2- 172.21.15.27:0/2914123471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5c4003ae0 0x7fa5dc16b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.445+0000 7fa5e11ff700 1 --2- 172.21.15.27:0/2914123471 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5dc16ab50 0x7fa5dc167d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.445+0000 7fa5e11ff700 1 -- 172.21.15.27:0/2914123471 >> 172.21.15.27:0/2914123471 conn(0x7fa5dc06ed40 msgr2=0x7fa5dc15b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:27.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.445+0000 7fa5e11ff700 1 -- 172.21.15.27:0/2914123471 shutdown_connections 2024-01-11T09:27:27.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.445+0000 7fa5e11ff700 1 -- 172.21.15.27:0/2914123471 wait complete. 2024-01-11T09:27:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-11T09:27:27.462 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:27:27.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.663+0000 7f4bd4fff700 1 Processor -- start 2024-01-11T09:27:27.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.663+0000 7f4bd4fff700 1 -- start start 2024-01-11T09:27:27.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.665+0000 7f4bd4fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4bd00ca3e0 0x7f4bd00cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:27.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.665+0000 7f4bd4fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4bd0159e80 0x7f4bd015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:27.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.665+0000 7f4bd4fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd015d640 con 0x7f4bd00ca7b0 2024-01-11T09:27:27.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.665+0000 7f4bd4fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd015ae70 con 0x7f4bd00ca3e0 2024-01-11T09:27:27.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.665+0000 7f4bd4fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd00cc4f0 con 0x7f4bd0159e80 2024-01-11T09:27:27.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.665+0000 7f4bce59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4bd00ca3e0 0x7f4bd00cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:27.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.665+0000 7f4bce59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4bd00ca3e0 0x7f4bd00cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37716/0 (socket says 172.21.15.27:37716) 2024-01-11T09:27:27.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.665+0000 7f4bce59c700 1 -- 172.21.15.27:0/754505425 learned_addr learned my addr 172.21.15.27:0/754505425 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:27.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.665+0000 7f4bced9d700 1 --2- 172.21.15.27:0/754505425 >> v2:172.21.15.77:3300/0 conn(0x7f4bd0159e80 0x7f4bd015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:27.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.665+0000 7f4bce59c700 1 -- 172.21.15.27:0/754505425 >> v2:172.21.15.77:3300/0 conn(0x7f4bd0159e80 msgr2=0x7f4bd015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:27.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.665+0000 7f4bce59c700 1 --2- 172.21.15.27:0/754505425 >> v2:172.21.15.77:3300/0 conn(0x7f4bd0159e80 0x7f4bd015dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.665+0000 7f4bce59c700 1 -- 172.21.15.27:0/754505425 >> v1:172.21.15.77:6789/0 conn(0x7f4bd00ca7b0 legacy=0x7f4bd00c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:27.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.665+0000 7f4bce59c700 1 -- 172.21.15.27:0/754505425 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd015a5d0 con 0x7f4bd00ca3e0 2024-01-11T09:27:27.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.665+0000 7f4bced9d700 1 --2- 172.21.15.27:0/754505425 >> v2:172.21.15.77:3300/0 conn(0x7f4bd0159e80 0x7f4bd015dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:27:27.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.666+0000 7f4bce59c700 1 --2- 172.21.15.27:0/754505425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bd00ca3e0 0x7f4bd00cca20 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00099e0 tx=0x7f4bc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ea527973f05aba server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:27.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.666+0000 7f4bcd59a700 1 -- 172.21.15.27:0/754505425 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc000b7e0 con 0x7f4bd00ca3e0 2024-01-11T09:27:27.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.666+0000 7f4bcd59a700 1 -- 172.21.15.27:0/754505425 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc000b940 con 0x7f4bd00ca3e0 2024-01-11T09:27:27.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.666+0000 7f4bcd59a700 1 -- 172.21.15.27:0/754505425 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc000bce0 con 0x7f4bd00ca3e0 2024-01-11T09:27:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.666+0000 7f4bd4fff700 1 -- 172.21.15.27:0/754505425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bd00ca3e0 msgr2=0x7f4bd00cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.666+0000 7f4bd4fff700 1 --2- 172.21.15.27:0/754505425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bd00ca3e0 0x7f4bd00cca20 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00099e0 tx=0x7f4bc0009250 comp rx=0 tx=0).stop 2024-01-11T09:27:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.667+0000 7f4bd4fff700 1 -- 172.21.15.27:0/754505425 shutdown_connections 2024-01-11T09:27:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.667+0000 7f4bd4fff700 1 --2- 172.21.15.27:0/754505425 >> v2:172.21.15.77:3300/0 conn(0x7f4bd0159e80 0x7f4bd015dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.667+0000 7f4bd4fff700 1 --2- 172.21.15.27:0/754505425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bd00ca3e0 0x7f4bd00cca20 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.667+0000 7f4bd4fff700 1 -- 172.21.15.27:0/754505425 >> 172.21.15.27:0/754505425 conn(0x7f4bd00ce3e0 msgr2=0x7f4bd00ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.667+0000 7f4bd4fff700 1 -- 172.21.15.27:0/754505425 shutdown_connections 2024-01-11T09:27:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.667+0000 7f4bd4fff700 1 -- 172.21.15.27:0/754505425 wait complete. 2024-01-11T09:27:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.668+0000 7f4bd4fff700 1 Processor -- start 2024-01-11T09:27:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.668+0000 7f4bd4fff700 1 -- start start 2024-01-11T09:27:27.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.669+0000 7f4bd4fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bd00ca3e0 0x7f4bd014a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:27.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.669+0000 7f4bd4fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bd00ca7b0 0x7f4bd00b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:27.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.669+0000 7f4bd4fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bd0159e80 0x7f4bd0147e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:27.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.670+0000 7f4bd4fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd014b900 con 0x7f4bd0159e80 2024-01-11T09:27:27.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.670+0000 7f4bd4fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd014b470 con 0x7f4bd00ca7b0 2024-01-11T09:27:27.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.670+0000 7f4bd4fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd014a860 con 0x7f4bd00ca3e0 2024-01-11T09:27:27.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.670+0000 7f4bced9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bd0159e80 0x7f4bd0147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:27.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.670+0000 7f4bce59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bd00ca3e0 0x7f4bd014a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.670+0000 7f4bcdd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bd00ca7b0 0x7f4bd00b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.670+0000 7f4bced9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bd0159e80 0x7f4bd0147e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37726/0 (socket says 172.21.15.27:37726) 2024-01-11T09:27:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.670+0000 7f4bced9d700 1 -- 172.21.15.27:0/379140328 learned_addr learned my addr 172.21.15.27:0/379140328 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.670+0000 7f4bced9d700 1 -- 172.21.15.27:0/379140328 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bd00ca3e0 msgr2=0x7f4bd014a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.670+0000 7f4bced9d700 1 --2- 172.21.15.27:0/379140328 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bd00ca3e0 0x7f4bd014a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.670+0000 7f4bced9d700 1 -- 172.21.15.27:0/379140328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bd00ca7b0 msgr2=0x7f4bd00b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.670+0000 7f4bced9d700 1 --2- 172.21.15.27:0/379140328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bd00ca7b0 0x7f4bd00b1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.670+0000 7f4bced9d700 1 -- 172.21.15.27:0/379140328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc00094f0 con 0x7f4bd0159e80 2024-01-11T09:27:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.670+0000 7f4bced9d700 1 --2- 172.21.15.27:0/379140328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bd0159e80 0x7f4bd0147e30 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f4bc800cef0 tx=0x7f4bc8007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.671+0000 7f4bbe7fc700 1 -- 172.21.15.27:0/379140328 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc800ad20 con 0x7f4bd0159e80 2024-01-11T09:27:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.671+0000 7f4bbe7fc700 1 -- 172.21.15.27:0/379140328 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc8016420 con 0x7f4bd0159e80 2024-01-11T09:27:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.671+0000 7f4bbe7fc700 1 -- 172.21.15.27:0/379140328 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc800aea0 con 0x7f4bd0159e80 2024-01-11T09:27:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.671+0000 7f4bd4fff700 1 -- 172.21.15.27:0/379140328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd014ab40 con 0x7f4bd0159e80 2024-01-11T09:27:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.671+0000 7f4bd4fff700 1 -- 172.21.15.27:0/379140328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd014b100 con 0x7f4bd0159e80 2024-01-11T09:27:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.672+0000 7f4bd4fff700 1 -- 172.21.15.27:0/379140328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd014b600 con 0x7f4bd0159e80 2024-01-11T09:27:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.673+0000 7f4bbe7fc700 1 -- 172.21.15.27:0/379140328 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4bc8016a30 con 0x7f4bd0159e80 2024-01-11T09:27:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.674+0000 7f4bbe7fc700 1 --2- 172.21.15.27:0/379140328 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4bb0066f30 0x7f4bb00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.674+0000 7f4bbe7fc700 1 -- 172.21.15.27:0/379140328 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f4bc8012070 con 0x7f4bd0159e80 2024-01-11T09:27:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.675+0000 7f4bce59c700 1 --2- 172.21.15.27:0/379140328 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4bb0066f30 0x7f4bb00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.675+0000 7f4bce59c700 1 --2- 172.21.15.27:0/379140328 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4bb0066f30 0x7f4bb00693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00099e0 tx=0x7f4bc0016000 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:27.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.678+0000 7f4bbe7fc700 1 -- 172.21.15.27:0/379140328 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4bc804e5a0 con 0x7f4bd0159e80 2024-01-11T09:27:27.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.931+0000 7f4bd4fff700 1 -- 172.21.15.27:0/379140328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bd00718d0 con 0x7f4bd0159e80 2024-01-11T09:27:27.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.932+0000 7f4bbe7fc700 1 -- 172.21.15.27:0/379140328 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4601 (secure 0 0 0) 0x7f4bc8052000 con 0x7f4bd0159e80 2024-01-11T09:27:27.930 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:27:27.930 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8015,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1003297202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1003297202},{"type":"v1","addr":"172.21.15.77:6834","nonce":1003297202}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":8018,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3321059147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":3321059147},{"type":"v1","addr":"172.21.15.77:6835","nonce":3321059147}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":9202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2606639222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2606639222},{"type":"v1","addr":"172.21.15.27:6835","nonce":2606639222}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:27:21.470488+0000","modified":"2024-01-11T09:27:24.992917+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9564":{"gid":9564,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/3897592862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3897592862},{"type":"v1","addr":"172.21.15.27:6834","nonce":3897592862}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2024-01-11T09:27:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.934+0000 7f4bd4fff700 1 -- 172.21.15.27:0/379140328 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4bb0066f30 msgr2=0x7f4bb00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.934+0000 7f4bd4fff700 1 --2- 172.21.15.27:0/379140328 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4bb0066f30 0x7f4bb00693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00099e0 tx=0x7f4bc0016000 comp rx=0 tx=0).stop 2024-01-11T09:27:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.934+0000 7f4bd4fff700 1 -- 172.21.15.27:0/379140328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bd0159e80 msgr2=0x7f4bd0147e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.934+0000 7f4bd4fff700 1 --2- 172.21.15.27:0/379140328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bd0159e80 0x7f4bd0147e30 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f4bc800cef0 tx=0x7f4bc8007860 comp rx=0 tx=0).stop 2024-01-11T09:27:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.935+0000 7f4bd4fff700 1 -- 172.21.15.27:0/379140328 shutdown_connections 2024-01-11T09:27:27.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.935+0000 7f4bd4fff700 1 --2- 172.21.15.27:0/379140328 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f4bb0066f30 0x7f4bb00693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.935+0000 7f4bd4fff700 1 --2- 172.21.15.27:0/379140328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bd0159e80 0x7f4bd0147e30 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.935+0000 7f4bd4fff700 1 --2- 172.21.15.27:0/379140328 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bd00ca3e0 0x7f4bd014a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.935+0000 7f4bd4fff700 1 --2- 172.21.15.27:0/379140328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bd00ca7b0 0x7f4bd00b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:27.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.935+0000 7f4bd4fff700 1 -- 172.21.15.27:0/379140328 >> 172.21.15.27:0/379140328 conn(0x7f4bd00ce3e0 msgr2=0x7f4bd0068f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:27.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.935+0000 7f4bd4fff700 1 -- 172.21.15.27:0/379140328 shutdown_connections 2024-01-11T09:27:27.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:27.935+0000 7f4bd4fff700 1 -- 172.21.15.27:0/379140328 wait complete. 2024-01-11T09:27:27.935 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 94 2024-01-11T09:27:28.354 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 94, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-11T09:27:21.470488+0000', 'modified': '2024-01-11T09:27:24.992917+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9564}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9564': {'gid': 9564, 'name': 'c', 'rank': 0, 'incarnation': 90, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/3897592862', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 3897592862}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 3897592862}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-11T09:27:28.354 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-11T09:27:28.354 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-11T09:27:28.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.561+0000 7f536b9ff700 1 Processor -- start 2024-01-11T09:27:28.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.562+0000 7f536b9ff700 1 -- start start 2024-01-11T09:27:28.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.563+0000 7f536b9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53640686a0 0x7f53640abd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.563+0000 7f536b9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53640ac250 con 0x7f5364068ba0 2024-01-11T09:27:28.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.563+0000 7f536b9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5364067b00 con 0x7f53641596a0 2024-01-11T09:27:28.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.563+0000 7f536b9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f536406bf60 con 0x7f53640686a0 2024-01-11T09:27:28.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.563+0000 7f5368f9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5364068ba0 0x7f5364067f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53578/0 (socket says 172.21.15.27:53578) 2024-01-11T09:27:28.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.563+0000 7f5368f9a700 1 -- 172.21.15.27:0/3046831830 learned_addr learned my addr 172.21.15.27:0/3046831830 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:28.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.564+0000 7f5369f9c700 1 --2- 172.21.15.27:0/3046831830 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53640686a0 0x7f53640abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.564+0000 7f535bfff700 1 -- 172.21.15.27:0/3046831830 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397398039 0 0) 0x7f53640ac250 con 0x7f5364068ba0 2024-01-11T09:27:28.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.564+0000 7f535bfff700 1 -- 172.21.15.27:0/3046831830 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5350003570 con 0x7f5364068ba0 2024-01-11T09:27:28.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.564+0000 7f535bfff700 1 -- 172.21.15.27:0/3046831830 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2373718979 0 0) 0x7f5350003570 con 0x7f5364068ba0 2024-01-11T09:27:28.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.564+0000 7f535bfff700 1 -- 172.21.15.27:0/3046831830 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53640686a0 msgr2=0x7f53640abd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:28.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.564+0000 7f535bfff700 1 --2- 172.21.15.27:0/3046831830 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53640686a0 0x7f53640abd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.564+0000 7f535bfff700 1 -- 172.21.15.27:0/3046831830 >> v1:172.21.15.77:6789/0 conn(0x7f53641596a0 legacy=0x7f53640af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:28.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.564+0000 7f535bfff700 1 -- 172.21.15.27:0/3046831830 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536406c240 con 0x7f5364068ba0 2024-01-11T09:27:28.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.564+0000 7f535bfff700 1 -- 172.21.15.27:0/3046831830 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f5354001470 con 0x7f5364068ba0 2024-01-11T09:27:28.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.564+0000 7f535bfff700 1 -- 172.21.15.27:0/3046831830 >> v1:172.21.15.27:6789/0 conn(0x7f5364068ba0 legacy=0x7f5364067f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:28.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.564+0000 7f535bfff700 1 --2- 172.21.15.27:0/3046831830 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53640686a0 0x7f5350003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.564+0000 7f535bfff700 1 --2- 172.21.15.27:0/3046831830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5350003fd0 0x7f5350004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.564+0000 7f535bfff700 1 --2- 172.21.15.27:0/3046831830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5350004d50 0x7f53500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.564+0000 7f535bfff700 1 -- 172.21.15.27:0/3046831830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536406bf60 con 0x7f5350003fd0 2024-01-11T09:27:28.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.564+0000 7f535bfff700 1 -- 172.21.15.27:0/3046831830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535000bba0 con 0x7f5350004d50 2024-01-11T09:27:28.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.564+0000 7f535bfff700 1 -- 172.21.15.27:0/3046831830 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f535000bd60 con 0x7f53640686a0 2024-01-11T09:27:28.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.564+0000 7f5368f9a700 1 --2- 172.21.15.27:0/3046831830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5350003fd0 0x7f5350004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.564+0000 7f536979b700 1 --2- 172.21.15.27:0/3046831830 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53640686a0 0x7f5350003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.565+0000 7f5369f9c700 1 --2- 172.21.15.27:0/3046831830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5350004d50 0x7f53500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.565+0000 7f5368f9a700 1 -- 172.21.15.27:0/3046831830 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53640686a0 msgr2=0x7f5350003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:28.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.565+0000 7f5368f9a700 1 --2- 172.21.15.27:0/3046831830 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53640686a0 0x7f5350003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.565+0000 7f5368f9a700 1 -- 172.21.15.27:0/3046831830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5350004d50 msgr2=0x7f53500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:28.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.565+0000 7f5368f9a700 1 --2- 172.21.15.27:0/3046831830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5350004d50 0x7f53500051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.565+0000 7f5368f9a700 1 -- 172.21.15.27:0/3046831830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5350003570 con 0x7f5350003fd0 2024-01-11T09:27:28.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.565+0000 7f5368f9a700 1 --2- 172.21.15.27:0/3046831830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5350003fd0 0x7f5350004810 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f535400c2a0 tx=0x7f5364067a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ccf7aa9dfd1fda server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:28.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.565+0000 7f535bfff700 1 -- 172.21.15.27:0/3046831830 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535400cb70 con 0x7f5350003fd0 2024-01-11T09:27:28.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.565+0000 7f535bfff700 1 -- 172.21.15.27:0/3046831830 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535400cd10 con 0x7f5350003fd0 2024-01-11T09:27:28.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.565+0000 7f535bfff700 1 -- 172.21.15.27:0/3046831830 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5354005b70 con 0x7f5350003fd0 2024-01-11T09:27:28.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.566+0000 7f536b9ff700 1 -- 172.21.15.27:0/3046831830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5350003fd0 msgr2=0x7f5350004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:28.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.566+0000 7f536b9ff700 1 --2- 172.21.15.27:0/3046831830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5350003fd0 0x7f5350004810 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f535400c2a0 tx=0x7f5364067a70 comp rx=0 tx=0).stop 2024-01-11T09:27:28.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.566+0000 7f5369f9c700 1 -- 172.21.15.27:0/3046831830 reap_dead start 2024-01-11T09:27:28.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.566+0000 7f536b9ff700 1 -- 172.21.15.27:0/3046831830 shutdown_connections 2024-01-11T09:27:28.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.566+0000 7f536b9ff700 1 -- 172.21.15.27:0/3046831830 >> 172.21.15.27:0/3046831830 conn(0x7f536406a8b0 msgr2=0x7f53640ae520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:28.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.566+0000 7f536b9ff700 1 -- 172.21.15.27:0/3046831830 shutdown_connections 2024-01-11T09:27:28.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.566+0000 7f536b9ff700 1 -- 172.21.15.27:0/3046831830 wait complete. 2024-01-11T09:27:28.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.568+0000 7f536b9ff700 1 Processor -- start 2024-01-11T09:27:28.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.568+0000 7f536b9ff700 1 -- start start 2024-01-11T09:27:28.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.569+0000 7f536b9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5350003fd0 0x7f536415b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.569+0000 7f536b9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5364159e90 0x7f53641550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.569+0000 7f536b9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5364158250 0x7f53641586c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.569+0000 7f536b9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53640b2610 con 0x7f5364158250 2024-01-11T09:27:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.569+0000 7f536b9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536415bdf0 con 0x7f5350003fd0 2024-01-11T09:27:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.569+0000 7f536b9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5364158c00 con 0x7f5364159e90 2024-01-11T09:27:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.569+0000 7f5369f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5364158250 0x7f53641586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.569+0000 7f5369f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5364158250 0x7f53641586c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37752/0 (socket says 172.21.15.27:37752) 2024-01-11T09:27:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.569+0000 7f5369f9c700 1 -- 172.21.15.27:0/1200038391 learned_addr learned my addr 172.21.15.27:0/1200038391 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.569+0000 7f536979b700 1 --2- 172.21.15.27:0/1200038391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5350003fd0 0x7f536415b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.569+0000 7f5368f9a700 1 --2- 172.21.15.27:0/1200038391 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5364159e90 0x7f53641550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.570+0000 7f5369f9c700 1 -- 172.21.15.27:0/1200038391 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5364159e90 msgr2=0x7f53641550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.570+0000 7f5369f9c700 1 --2- 172.21.15.27:0/1200038391 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5364159e90 0x7f53641550a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.570+0000 7f5369f9c700 1 -- 172.21.15.27:0/1200038391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5350003fd0 msgr2=0x7f536415b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.570+0000 7f5369f9c700 1 --2- 172.21.15.27:0/1200038391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5350003fd0 0x7f536415b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.570+0000 7f5369f9c700 1 -- 172.21.15.27:0/1200038391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5364169c80 con 0x7f5364158250 2024-01-11T09:27:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.570+0000 7f5369f9c700 1 --2- 172.21.15.27:0/1200038391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5364158250 0x7f53641586c0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f535c009d70 tx=0x7f535c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:28.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.570+0000 7f53597fa700 1 -- 172.21.15.27:0/1200038391 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c00e040 con 0x7f5364158250 2024-01-11T09:27:28.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.570+0000 7f53597fa700 1 -- 172.21.15.27:0/1200038391 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535c00f040 con 0x7f5364158250 2024-01-11T09:27:28.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.570+0000 7f53597fa700 1 -- 172.21.15.27:0/1200038391 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c00fe70 con 0x7f5364158250 2024-01-11T09:27:28.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.570+0000 7f536b9ff700 1 -- 172.21.15.27:0/1200038391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53641555e0 con 0x7f5364158250 2024-01-11T09:27:28.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.570+0000 7f536b9ff700 1 -- 172.21.15.27:0/1200038391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536415c300 con 0x7f5364158250 2024-01-11T09:27:28.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.572+0000 7f53597fa700 1 -- 172.21.15.27:0/1200038391 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f535c015640 con 0x7f5364158250 2024-01-11T09:27:28.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.573+0000 7f53597fa700 1 --2- 172.21.15.27:0/1200038391 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5348066f30 0x7f53480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.573+0000 7f53597fa700 1 -- 172.21.15.27:0/1200038391 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f535c084ff0 con 0x7f5364158250 2024-01-11T09:27:28.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.574+0000 7f536979b700 1 --2- 172.21.15.27:0/1200038391 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5348066f30 0x7f53480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.574+0000 7f536979b700 1 --2- 172.21.15.27:0/1200038391 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5348066f30 0x7f53480693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5360003c40 tx=0x7f5360007d10 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:28.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.575+0000 7f537a0c7b80 1 Processor -- start 2024-01-11T09:27:28.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.575+0000 7f537a0c7b80 1 -- start start 2024-01-11T09:27:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.576+0000 7f537a0c7b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d08141db20 0x55d0813fedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.576+0000 7f537a0c7b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0813f93a0 0x55d081425860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.576+0000 7f537a0c7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141def0 0x55d081425da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.576+0000 7f537a0c7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d081316310 con 0x55d08141def0 2024-01-11T09:27:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.576+0000 7f537a0c7b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d081353130 con 0x55d0813f93a0 2024-01-11T09:27:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.576+0000 7f537a0c7b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d081350af0 con 0x55d08141db20 2024-01-11T09:27:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.576+0000 7f5369f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141def0 0x55d081425da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.577+0000 7f5369f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141def0 0x55d081425da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37758/0 (socket says 172.21.15.27:37758) 2024-01-11T09:27:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.577+0000 7f5369f9c700 1 -- 172.21.15.27:0/3957340545 learned_addr learned my addr 172.21.15.27:0/3957340545 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.577+0000 7f536979b700 1 --2- 172.21.15.27:0/3957340545 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d08141db20 0x55d0813fedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.577+0000 7f5368f9a700 1 --2- 172.21.15.27:0/3957340545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0813f93a0 0x55d081425860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.577+0000 7f5369f9c700 1 -- 172.21.15.27:0/3957340545 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d08141db20 msgr2=0x55d0813fedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.577+0000 7f5369f9c700 1 --2- 172.21.15.27:0/3957340545 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d08141db20 0x55d0813fedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.577+0000 7f5369f9c700 1 -- 172.21.15.27:0/3957340545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0813f93a0 msgr2=0x55d081425860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.577+0000 7f5369f9c700 1 --2- 172.21.15.27:0/3957340545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0813f93a0 0x55d081425860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.577+0000 7f5369f9c700 1 -- 172.21.15.27:0/3957340545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d08123f0a0 con 0x55d08141def0 2024-01-11T09:27:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.577+0000 7f5369f9c700 1 --2- 172.21.15.27:0/3957340545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141def0 0x55d081425da0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f535c002230 tx=0x7f535c083390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6fe388c3c6fab22 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.577+0000 7f534e7fc700 1 -- 172.21.15.27:0/3957340545 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c055c10 con 0x55d08141def0 2024-01-11T09:27:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.577+0000 7f534e7fc700 1 -- 172.21.15.27:0/3957340545 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535c055e40 con 0x55d08141def0 2024-01-11T09:27:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.577+0000 7f534e7fc700 1 -- 172.21.15.27:0/3957340545 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c059090 con 0x55d08141def0 2024-01-11T09:27:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.578+0000 7f537a0c7b80 1 -- 172.21.15.27:0/3957340545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141def0 msgr2=0x55d081425da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.578+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/3957340545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141def0 0x55d081425da0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f535c002230 tx=0x7f535c083390 comp rx=0 tx=0).stop 2024-01-11T09:27:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.578+0000 7f537a0c7b80 1 -- 172.21.15.27:0/3957340545 shutdown_connections 2024-01-11T09:27:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.578+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/3957340545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141def0 0x55d081425da0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.578+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/3957340545 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d08141db20 0x55d0813fedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.578+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/3957340545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0813f93a0 0x55d081425860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.578+0000 7f537a0c7b80 1 -- 172.21.15.27:0/3957340545 >> 172.21.15.27:0/3957340545 conn(0x55d0813fb300 msgr2=0x55d0813fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.578+0000 7f537a0c7b80 1 -- 172.21.15.27:0/3957340545 shutdown_connections 2024-01-11T09:27:28.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.578+0000 7f537a0c7b80 1 -- 172.21.15.27:0/3957340545 wait complete. 2024-01-11T09:27:28.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.579+0000 7f537a0c7b80 1 -- start start 2024-01-11T09:27:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.579+0000 7f537a0c7b80 1 Processor -- start 2024-01-11T09:27:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.580+0000 7f537a0c7b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d0813f93a0 0x55d081420ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.580+0000 7f537a0c7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141db20 0x55d08141d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.580+0000 7f537a0c7b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d08141def0 0x55d081428c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.580+0000 7f537a0c7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d08134c8e0 con 0x55d08141db20 2024-01-11T09:27:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.580+0000 7f537a0c7b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0812f8980 con 0x55d08141def0 2024-01-11T09:27:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.580+0000 7f537a0c7b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d0812f8620 con 0x55d0813f93a0 2024-01-11T09:27:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.580+0000 7f537a0c7b80 10 client.0 _ll_register_callbacks cb 0x55d08115ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T09:27:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.580+0000 7f5368f9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141db20 0x55d08141d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.580+0000 7f5368f9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141db20 0x55d08141d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37762/0 (socket says 172.21.15.27:37762) 2024-01-11T09:27:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.580+0000 7f5368f9a700 1 -- 172.21.15.27:0/587073554 learned_addr learned my addr 172.21.15.27:0/587073554 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.581+0000 7f5369f9c700 1 --2- 172.21.15.27:0/587073554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d08141def0 0x55d081428c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.581+0000 7f536979b700 1 --2- 172.21.15.27:0/587073554 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d0813f93a0 0x55d081420ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.581+0000 7f5368f9a700 1 -- 172.21.15.27:0/587073554 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d0813f93a0 msgr2=0x55d081420ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.581+0000 7f5368f9a700 1 --2- 172.21.15.27:0/587073554 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d0813f93a0 0x55d081420ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.581+0000 7f5368f9a700 1 -- 172.21.15.27:0/587073554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d08141def0 msgr2=0x55d081428c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.581+0000 7f5368f9a700 1 --2- 172.21.15.27:0/587073554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d08141def0 0x55d081428c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.581+0000 7f5368f9a700 1 -- 172.21.15.27:0/587073554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536415c300 con 0x55d08141db20 2024-01-11T09:27:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.581+0000 7f5368f9a700 1 --2- 172.21.15.27:0/587073554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141db20 0x55d08141d3a0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f535400c890 tx=0x7f5354015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cedfd11254b24b8a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.581+0000 7f53437fe700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T09:27:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.581+0000 7f53437fe700 1 -- 172.21.15.27:0/587073554 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5354024070 con 0x55d08141db20 2024-01-11T09:27:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.581+0000 7f537a0c7b80 1 -- 172.21.15.27:0/587073554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d0813f1560 con 0x55d08141db20 2024-01-11T09:27:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.581+0000 7f53437fe700 1 -- 172.21.15.27:0/587073554 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(181..181 src has 1..181) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f535400ce80 con 0x55d08141db20 2024-01-11T09:27:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.582+0000 7f53437fe700 1 -- 172.21.15.27:0/587073554 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) v2 ==== 24+0+0 (secure 0 0 0) 0x7f535400cb70 con 0x55d08141db20 2024-01-11T09:27:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.582+0000 7f537a0c7b80 10 client.9690 fetch_fsmap learned FSMap version 94 2024-01-11T09:27:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.582+0000 7f537a0c7b80 1 -- 172.21.15.27:0/587073554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) v3 -- 0x55d08131b3c0 con 0x55d08141db20 2024-01-11T09:27:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.582+0000 7f53437fe700 1 -- 172.21.15.27:0/587073554 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d0813f1560 con 0x55d08141db20 2024-01-11T09:27:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.582+0000 7f53437fe700 1 -- 172.21.15.27:0/587073554 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 94) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f5354001580 con 0x55d08141db20 2024-01-11T09:27:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.582+0000 7f537a0c7b80 10 client.9690 fetch_fsmap finished waiting for FSMap version 94 2024-01-11T09:27:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.582+0000 7f537a0c7b80 10 client.9690 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-11T09:27:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.582+0000 7f537a0c7b80 20 client.9690 populate_metadata read hostname 'smithi027' 2024-01-11T09:27:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.582+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/587073554 >> [v2:172.21.15.27:6832/3897592862,v1:172.21.15.27:6834/3897592862] conn(0x55d0814226d0 0x55d081422aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.582+0000 7f537a0c7b80 4 client.9690 mds_command: new command op to 9564 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-11T09:27:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.582+0000 7f537a0c7b80 1 -- 172.21.15.27:0/587073554 --> [v2:172.21.15.27:6832/3897592862,v1:172.21.15.27:6834/3897592862] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d0813ce0e0 con 0x55d0814226d0 2024-01-11T09:27:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.582+0000 7f536979b700 1 --2- 172.21.15.27:0/587073554 >> [v2:172.21.15.27:6832/3897592862,v1:172.21.15.27:6834/3897592862] conn(0x55d0814226d0 0x55d081422aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.583+0000 7f536979b700 1 --2- 172.21.15.27:0/587073554 >> [v2:172.21.15.27:6832/3897592862,v1:172.21.15.27:6834/3897592862] conn(0x55d0814226d0 0x55d081422aa0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=358e786562cb7813 server_cookie=afd7d5664699e4b1 in_seq=0 out_seq=0 2024-01-11T09:27:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.583+0000 7f53437fe700 10 client.9690 ms_handle_connect on v2:172.21.15.27:6832/3897592862 2024-01-11T09:27:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.584+0000 7f53437fe700 1 -- 172.21.15.27:0/587073554 <== mds.0 v2:172.21.15.27:6832/3897592862 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x55d0813ce0e0 con 0x55d0814226d0 2024-01-11T09:27:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.584+0000 7f53437fe700 10 client.9690 handle_command_reply: tid=0 2024-01-11T09:27:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.584+0000 7f537a0c7b80 1 client.9690 shutdown 2024-01-11T09:27:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.584+0000 7f537a0c7b80 1 -- 172.21.15.27:0/587073554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141db20 msgr2=0x55d08141d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.584+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/587073554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141db20 0x55d08141d3a0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f535400c890 tx=0x7f5354015000 comp rx=0 tx=0).stop 2024-01-11T09:27:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.584+0000 7f537a0c7b80 1 -- 172.21.15.27:0/587073554 shutdown_connections 2024-01-11T09:27:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.584+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/587073554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d08141def0 0x55d081428c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.584+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/587073554 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d0813f93a0 0x55d081420ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.584+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/587073554 >> [v2:172.21.15.27:6832/3897592862,v1:172.21.15.27:6834/3897592862] conn(0x55d0814226d0 0x55d081422aa0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.584+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/587073554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141db20 0x55d08141d3a0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.584+0000 7f537a0c7b80 1 -- 172.21.15.27:0/587073554 >> 172.21.15.27:0/587073554 conn(0x55d0813fb300 msgr2=0x55d0813f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.584+0000 7f53437fe700 0 client.9690 ms_handle_reset on v2:172.21.15.27:6832/3897592862 2024-01-11T09:27:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.584+0000 7f537a0c7b80 1 -- 172.21.15.27:0/587073554 shutdown_connections 2024-01-11T09:27:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.584+0000 7f537a0c7b80 1 -- 172.21.15.27:0/587073554 wait complete. 2024-01-11T09:27:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.584+0000 7f537a0c7b80 20 client.9690 trim_cache size 0 max 16384 2024-01-11T09:27:28.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.602+0000 7f537a0c7b80 1 Processor -- start 2024-01-11T09:27:28.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.602+0000 7f537a0c7b80 1 -- start start 2024-01-11T09:27:28.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.603+0000 7f537a0c7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0813f93a0 0x55d081424850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.603+0000 7f537a0c7b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d08141db20 0x55d0813f88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.603+0000 7f537a0c7b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d08141def0 0x55d08141e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.603+0000 7f537a0c7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0811e41e0 con 0x55d0813f93a0 2024-01-11T09:27:28.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.603+0000 7f537a0c7b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0813f3d80 con 0x55d08141def0 2024-01-11T09:27:28.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.603+0000 7f537a0c7b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d08142a0f0 con 0x55d08141db20 2024-01-11T09:27:28.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.603+0000 7f536979b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0813f93a0 0x55d081424850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.603+0000 7f536979b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0813f93a0 0x55d081424850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37766/0 (socket says 172.21.15.27:37766) 2024-01-11T09:27:28.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.603+0000 7f536979b700 1 -- 172.21.15.27:0/3428135231 learned_addr learned my addr 172.21.15.27:0/3428135231 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:28.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.603+0000 7f5369f9c700 1 --2- 172.21.15.27:0/3428135231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d08141def0 0x55d08141e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.603+0000 7f5368f9a700 1 --2- 172.21.15.27:0/3428135231 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d08141db20 0x55d0813f88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.603+0000 7f536979b700 1 -- 172.21.15.27:0/3428135231 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d08141db20 msgr2=0x55d0813f88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.603+0000 7f536979b700 1 --2- 172.21.15.27:0/3428135231 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d08141db20 0x55d0813f88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.604+0000 7f536979b700 1 -- 172.21.15.27:0/3428135231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d08141def0 msgr2=0x55d08141e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.604+0000 7f536979b700 1 --2- 172.21.15.27:0/3428135231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d08141def0 0x55d08141e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.604+0000 7f536979b700 1 -- 172.21.15.27:0/3428135231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0811b31d0 con 0x55d0813f93a0 2024-01-11T09:27:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.604+0000 7f5368f9a700 1 --2- 172.21.15.27:0/3428135231 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d08141db20 0x55d0813f88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:27:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.604+0000 7f536979b700 1 --2- 172.21.15.27:0/3428135231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0813f93a0 0x55d081424850 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f5360003f10 tx=0x7f5360017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf6619afd2e5e71 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.604+0000 7f534effd700 1 -- 172.21.15.27:0/3428135231 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5360027040 con 0x55d0813f93a0 2024-01-11T09:27:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.604+0000 7f534effd700 1 -- 172.21.15.27:0/3428135231 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5360015d20 con 0x55d0813f93a0 2024-01-11T09:27:28.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.604+0000 7f534effd700 1 -- 172.21.15.27:0/3428135231 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536002c630 con 0x55d0813f93a0 2024-01-11T09:27:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.605+0000 7f537a0c7b80 1 -- 172.21.15.27:0/3428135231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0813f93a0 msgr2=0x55d081424850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.605+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/3428135231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0813f93a0 0x55d081424850 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f5360003f10 tx=0x7f5360017000 comp rx=0 tx=0).stop 2024-01-11T09:27:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.605+0000 7f537a0c7b80 1 -- 172.21.15.27:0/3428135231 shutdown_connections 2024-01-11T09:27:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.605+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/3428135231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d08141def0 0x55d08141e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.605+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/3428135231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d0813f93a0 0x55d081424850 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.605+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/3428135231 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d08141db20 0x55d0813f88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.605+0000 7f537a0c7b80 1 -- 172.21.15.27:0/3428135231 >> 172.21.15.27:0/3428135231 conn(0x55d0813fb300 msgr2=0x55d081422aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.605+0000 7f537a0c7b80 1 -- 172.21.15.27:0/3428135231 shutdown_connections 2024-01-11T09:27:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.605+0000 7f537a0c7b80 1 -- 172.21.15.27:0/3428135231 wait complete. 2024-01-11T09:27:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.605+0000 7f537a0c7b80 1 -- start start 2024-01-11T09:27:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.606+0000 7f537a0c7b80 1 Processor -- start 2024-01-11T09:27:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.607+0000 7f537a0c7b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0813f93a0 0x55d0814916d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.607+0000 7f537a0c7b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141db20 0x55d081446620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.607+0000 7f537a0c7b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d08141def0 0x55d0813f99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.607+0000 7f537a0c7b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d081423fa0 con 0x55d08141db20 2024-01-11T09:27:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.607+0000 7f537a0c7b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0814a9920 con 0x55d0813f93a0 2024-01-11T09:27:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.607+0000 7f537a0c7b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d0813f2960 con 0x55d08141def0 2024-01-11T09:27:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.607+0000 7f537a0c7b80 10 client.0 _ll_register_callbacks cb 0x55d08115ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T09:27:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.607+0000 7f5368f9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141db20 0x55d081446620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.607+0000 7f5368f9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141db20 0x55d081446620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37782/0 (socket says 172.21.15.27:37782) 2024-01-11T09:27:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.607+0000 7f5368f9a700 1 -- 172.21.15.27:0/2559470737 learned_addr learned my addr 172.21.15.27:0/2559470737 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.607+0000 7f536979b700 1 --2- 172.21.15.27:0/2559470737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0813f93a0 0x55d0814916d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.607+0000 7f5369f9c700 1 --2- 172.21.15.27:0/2559470737 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d08141def0 0x55d0813f99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.607+0000 7f5368f9a700 1 -- 172.21.15.27:0/2559470737 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d08141def0 msgr2=0x55d0813f99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.607+0000 7f5368f9a700 1 --2- 172.21.15.27:0/2559470737 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d08141def0 0x55d0813f99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.607+0000 7f5368f9a700 1 -- 172.21.15.27:0/2559470737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0813f93a0 msgr2=0x55d0814916d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:28.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.607+0000 7f5368f9a700 1 --2- 172.21.15.27:0/2559470737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0813f93a0 0x55d0814916d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.607+0000 7f5368f9a700 1 -- 172.21.15.27:0/2559470737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5360002cd0 con 0x55d08141db20 2024-01-11T09:27:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.608+0000 7f5368f9a700 1 --2- 172.21.15.27:0/2559470737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141db20 0x55d081446620 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f5354009020 tx=0x7f5354024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9be155db5e148f63 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.608+0000 7f53437fe700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T09:27:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.608+0000 7f53437fe700 1 -- 172.21.15.27:0/2559470737 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535400cb70 con 0x55d08141db20 2024-01-11T09:27:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.608+0000 7f53437fe700 1 -- 172.21.15.27:0/2559470737 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(181..181 src has 1..181) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f535400ce80 con 0x55d08141db20 2024-01-11T09:27:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.608+0000 7f537a0c7b80 1 -- 172.21.15.27:0/2559470737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f535c055e40 con 0x55d08141db20 2024-01-11T09:27:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.608+0000 7f53437fe700 1 -- 172.21.15.27:0/2559470737 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) v2 ==== 24+0+0 (secure 0 0 0) 0x7f535401f410 con 0x55d08141db20 2024-01-11T09:27:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.608+0000 7f537a0c7b80 10 client.9696 fetch_fsmap learned FSMap version 94 2024-01-11T09:27:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.608+0000 7f537a0c7b80 1 -- 172.21.15.27:0/2559470737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) v3 -- 0x55d08118cd60 con 0x55d08141db20 2024-01-11T09:27:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.608+0000 7f53437fe700 1 -- 172.21.15.27:0/2559470737 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535c055e40 con 0x55d08141db20 2024-01-11T09:27:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.608+0000 7f53437fe700 1 -- 172.21.15.27:0/2559470737 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 94) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f535401f570 con 0x55d08141db20 2024-01-11T09:27:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.609+0000 7f537a0c7b80 10 client.9696 fetch_fsmap finished waiting for FSMap version 94 2024-01-11T09:27:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.609+0000 7f537a0c7b80 10 client.9696 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-11T09:27:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.609+0000 7f537a0c7b80 20 client.9696 populate_metadata read hostname 'smithi027' 2024-01-11T09:27:28.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.609+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/2559470737 >> [v2:172.21.15.27:6832/3897592862,v1:172.21.15.27:6834/3897592862] conn(0x55d08145c3a0 0x55d08145c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.609+0000 7f537a0c7b80 4 client.9696 mds_command: new command op to 9564 tid=0[{"prefix": "status"}] 2024-01-11T09:27:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.609+0000 7f537a0c7b80 1 -- 172.21.15.27:0/2559470737 --> [v2:172.21.15.27:6832/3897592862,v1:172.21.15.27:6834/3897592862] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d0812ef8d0 con 0x55d08145c3a0 2024-01-11T09:27:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.609+0000 7f536979b700 1 --2- 172.21.15.27:0/2559470737 >> [v2:172.21.15.27:6832/3897592862,v1:172.21.15.27:6834/3897592862] conn(0x55d08145c3a0 0x55d08145c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.609+0000 7f536979b700 1 --2- 172.21.15.27:0/2559470737 >> [v2:172.21.15.27:6832/3897592862,v1:172.21.15.27:6834/3897592862] conn(0x55d08145c3a0 0x55d08145c770 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6fd715a36472e9e server_cookie=5557295bc55d66bb in_seq=0 out_seq=0 2024-01-11T09:27:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.609+0000 7f53437fe700 10 client.9696 ms_handle_connect on v2:172.21.15.27:6832/3897592862 2024-01-11T09:27:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.610+0000 7f53437fe700 1 -- 172.21.15.27:0/2559470737 <== mds.0 v2:172.21.15.27:6832/3897592862 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55d0812ef8d0 con 0x55d08145c3a0 2024-01-11T09:27:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.610+0000 7f53437fe700 10 client.9696 handle_command_reply: tid=0 2024-01-11T09:27:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.610+0000 7f537a0c7b80 1 client.9696 shutdown 2024-01-11T09:27:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.610+0000 7f537a0c7b80 1 -- 172.21.15.27:0/2559470737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141db20 msgr2=0x55d081446620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.610+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/2559470737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141db20 0x55d081446620 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f5354009020 tx=0x7f5354024000 comp rx=0 tx=0).stop 2024-01-11T09:27:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.610+0000 7f537a0c7b80 1 -- 172.21.15.27:0/2559470737 shutdown_connections 2024-01-11T09:27:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.610+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/2559470737 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55d08141def0 0x55d0813f99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.610+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/2559470737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0813f93a0 0x55d0814916d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.610+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/2559470737 >> [v2:172.21.15.27:6832/3897592862,v1:172.21.15.27:6834/3897592862] conn(0x55d08145c3a0 0x55d08145c770 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.610+0000 7f537a0c7b80 1 --2- 172.21.15.27:0/2559470737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d08141db20 0x55d081446620 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.610+0000 7f537a0c7b80 1 -- 172.21.15.27:0/2559470737 >> 172.21.15.27:0/2559470737 conn(0x55d0813fb300 msgr2=0x55d0813ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:28.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.610+0000 7f53437fe700 0 client.9696 ms_handle_reset on v2:172.21.15.27:6832/3897592862 2024-01-11T09:27:28.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.610+0000 7f537a0c7b80 1 -- 172.21.15.27:0/2559470737 shutdown_connections 2024-01-11T09:27:28.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.610+0000 7f537a0c7b80 1 -- 172.21.15.27:0/2559470737 wait complete. 2024-01-11T09:27:28.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.610+0000 7f537a0c7b80 20 client.9696 trim_cache size 0 max 16384 2024-01-11T09:27:28.614 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:27:28.614 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "8fddb4a7-9d66-40f1-892d-6fe86c728ee4", 2024-01-11T09:27:28.614 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-11T09:27:28.614 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 9564, 2024-01-11T09:27:28.614 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-11T09:27:28.614 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-11T09:27:28.614 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-11T09:27:28.614 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 6.6438595669999998, 2024-01-11T09:27:28.614 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 94, 2024-01-11T09:27:28.614 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 181, 2024-01-11T09:27:28.614 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 181, 2024-01-11T09:27:28.615 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 13.311772245 2024-01-11T09:27:28.615 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:27:28.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.611+0000 7f536b9ff700 1 -- 172.21.15.27:0/1200038391 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5348066f30 msgr2=0x7f53480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:28.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.611+0000 7f536b9ff700 1 --2- 172.21.15.27:0/1200038391 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5348066f30 0x7f53480693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5360003c40 tx=0x7f5360007d10 comp rx=0 tx=0).stop 2024-01-11T09:27:28.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.611+0000 7f536b9ff700 1 -- 172.21.15.27:0/1200038391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5364158250 msgr2=0x7f53641586c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:28.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.611+0000 7f536b9ff700 1 --2- 172.21.15.27:0/1200038391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5364158250 0x7f53641586c0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f535c009d70 tx=0x7f535c00b040 comp rx=0 tx=0).stop 2024-01-11T09:27:28.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.611+0000 7f536b9ff700 1 -- 172.21.15.27:0/1200038391 shutdown_connections 2024-01-11T09:27:28.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.611+0000 7f536b9ff700 1 --2- 172.21.15.27:0/1200038391 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f5348066f30 0x7f53480693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.611+0000 7f536b9ff700 1 --2- 172.21.15.27:0/1200038391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5364158250 0x7f53641586c0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.611+0000 7f536b9ff700 1 --2- 172.21.15.27:0/1200038391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5350003fd0 0x7f536415b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.611+0000 7f536b9ff700 1 --2- 172.21.15.27:0/1200038391 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5364159e90 0x7f53641550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.611+0000 7f536b9ff700 1 -- 172.21.15.27:0/1200038391 >> 172.21.15.27:0/1200038391 conn(0x7f536406a8b0 msgr2=0x7f53640ae430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:28.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.611+0000 7f536b9ff700 1 -- 172.21.15.27:0/1200038391 shutdown_connections 2024-01-11T09:27:28.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.611+0000 7f536b9ff700 1 -- 172.21.15.27:0/1200038391 wait complete. 2024-01-11T09:27:28.626 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-11T09:27:28.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.829+0000 7fb5443ff700 1 Processor -- start 2024-01-11T09:27:28.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.829+0000 7fb5443ff700 1 -- start start 2024-01-11T09:27:28.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.831+0000 7fb5443ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb53c0c8620 0x7fb53c0af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.831+0000 7fb5443ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb53c0b09e0 0x7fb53c0accc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.831+0000 7fb5443ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb53c157390 con 0x7fb53c069d40 2024-01-11T09:27:28.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.831+0000 7fb5443ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb53c0afeb0 con 0x7fb53c0c8620 2024-01-11T09:27:28.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.831+0000 7fb5443ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c06a200 con 0x7fb53c0b09e0 2024-01-11T09:27:28.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.831+0000 7fb54299c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb53c0b09e0 0x7fb53c0accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.831+0000 7fb54299c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb53c0b09e0 0x7fb53c0accc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:37214/0 (socket says 172.21.15.27:37214) 2024-01-11T09:27:28.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.831+0000 7fb54299c700 1 -- 172.21.15.27:0/2393004336 learned_addr learned my addr 172.21.15.27:0/2393004336 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:28.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.831+0000 7fb54219b700 1 --2- 172.21.15.27:0/2393004336 >> v2:172.21.15.77:3300/0 conn(0x7fb53c0c8620 0x7fb53c0af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.832+0000 7fb54299c700 1 -- 172.21.15.27:0/2393004336 >> v2:172.21.15.77:3300/0 conn(0x7fb53c0c8620 msgr2=0x7fb53c0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:28.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.832+0000 7fb54299c700 1 --2- 172.21.15.27:0/2393004336 >> v2:172.21.15.77:3300/0 conn(0x7fb53c0c8620 0x7fb53c0af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.832+0000 7fb54299c700 1 -- 172.21.15.27:0/2393004336 >> v1:172.21.15.77:6789/0 conn(0x7fb53c069d40 legacy=0x7fb53c0b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:28.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.832+0000 7fb54299c700 1 -- 172.21.15.27:0/2393004336 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53c156d40 con 0x7fb53c0b09e0 2024-01-11T09:27:28.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.832+0000 7fb541199700 1 -- 172.21.15.27:0/2393004336 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293318264 0 0) 0x7fb53c157390 con 0x7fb53c069d40 2024-01-11T09:27:28.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.832+0000 7fb54299c700 1 --2- 172.21.15.27:0/2393004336 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb53c0b09e0 0x7fb53c0accc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb534009f50 tx=0x7fb534009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acff69c134f5136e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:28.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.832+0000 7fb541199700 1 -- 172.21.15.27:0/2393004336 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53400e070 con 0x7fb53c0b09e0 2024-01-11T09:27:28.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.832+0000 7fb541199700 1 -- 172.21.15.27:0/2393004336 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53400b9c0 con 0x7fb53c0b09e0 2024-01-11T09:27:28.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.832+0000 7fb541199700 1 -- 172.21.15.27:0/2393004336 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5340092e0 con 0x7fb53c0b09e0 2024-01-11T09:27:28.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.833+0000 7fb5443ff700 1 -- 172.21.15.27:0/2393004336 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb53c0b09e0 msgr2=0x7fb53c0accc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:28.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.833+0000 7fb5443ff700 1 --2- 172.21.15.27:0/2393004336 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb53c0b09e0 0x7fb53c0accc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb534009f50 tx=0x7fb534009840 comp rx=0 tx=0).stop 2024-01-11T09:27:28.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.833+0000 7fb5443ff700 1 -- 172.21.15.27:0/2393004336 shutdown_connections 2024-01-11T09:27:28.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.833+0000 7fb5443ff700 1 --2- 172.21.15.27:0/2393004336 >> v2:172.21.15.77:3300/0 conn(0x7fb53c0c8620 0x7fb53c0af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.833+0000 7fb5443ff700 1 --2- 172.21.15.27:0/2393004336 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb53c0b09e0 0x7fb53c0accc0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.833+0000 7fb5443ff700 1 -- 172.21.15.27:0/2393004336 >> 172.21.15.27:0/2393004336 conn(0x7fb53c0680e0 msgr2=0x7fb53c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:28.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.833+0000 7fb5443ff700 1 -- 172.21.15.27:0/2393004336 shutdown_connections 2024-01-11T09:27:28.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.833+0000 7fb5443ff700 1 -- 172.21.15.27:0/2393004336 wait complete. 2024-01-11T09:27:28.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.834+0000 7fb5443ff700 1 Processor -- start 2024-01-11T09:27:28.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.835+0000 7fb5443ff700 1 -- start start 2024-01-11T09:27:28.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.836+0000 7fb5443ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb53c069d40 0x7fb53c14c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.836+0000 7fb5443ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb53c0b09e0 0x7fb53c14e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.836+0000 7fb5443ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb53c0c8620 0x7fb53c148800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.836+0000 7fb5443ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c14cbc0 con 0x7fb53c0b09e0 2024-01-11T09:27:28.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.836+0000 7fb5443ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c14f4b0 con 0x7fb53c069d40 2024-01-11T09:27:28.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.836+0000 7fb5443ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c14eea0 con 0x7fb53c0c8620 2024-01-11T09:27:28.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.836+0000 7fb54199a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb53c0b09e0 0x7fb53c14e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.836+0000 7fb54199a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb53c0b09e0 0x7fb53c14e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37792/0 (socket says 172.21.15.27:37792) 2024-01-11T09:27:28.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.836+0000 7fb54199a700 1 -- 172.21.15.27:0/108610242 learned_addr learned my addr 172.21.15.27:0/108610242 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:28.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.836+0000 7fb54219b700 1 --2- 172.21.15.27:0/108610242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb53c069d40 0x7fb53c14c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.836+0000 7fb54299c700 1 --2- 172.21.15.27:0/108610242 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb53c0c8620 0x7fb53c148800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.836+0000 7fb54199a700 1 -- 172.21.15.27:0/108610242 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb53c0c8620 msgr2=0x7fb53c148800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.836+0000 7fb54199a700 1 --2- 172.21.15.27:0/108610242 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb53c0c8620 0x7fb53c148800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.836+0000 7fb54199a700 1 -- 172.21.15.27:0/108610242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb53c069d40 msgr2=0x7fb53c14c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.836+0000 7fb54199a700 1 --2- 172.21.15.27:0/108610242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb53c069d40 0x7fb53c14c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.836+0000 7fb54199a700 1 -- 172.21.15.27:0/108610242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb534009ae0 con 0x7fb53c0b09e0 2024-01-11T09:27:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.837+0000 7fb54219b700 1 --2- 172.21.15.27:0/108610242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb53c069d40 0x7fb53c14c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:27:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.837+0000 7fb54199a700 1 --2- 172.21.15.27:0/108610242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb53c0b09e0 0x7fb53c14e3e0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fb52c00c810 tx=0x7fb52c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.837+0000 7fb5327fc700 1 -- 172.21.15.27:0/108610242 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52c011590 con 0x7fb53c0b09e0 2024-01-11T09:27:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.837+0000 7fb5327fc700 1 -- 172.21.15.27:0/108610242 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52c011710 con 0x7fb53c0b09e0 2024-01-11T09:27:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.837+0000 7fb5327fc700 1 -- 172.21.15.27:0/108610242 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52c011ad0 con 0x7fb53c0b09e0 2024-01-11T09:27:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.837+0000 7fb5443ff700 1 -- 172.21.15.27:0/108610242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb53c14c8e0 con 0x7fb53c0b09e0 2024-01-11T09:27:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.837+0000 7fb5443ff700 1 -- 172.21.15.27:0/108610242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb53c14e920 con 0x7fb53c0b09e0 2024-01-11T09:27:28.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.838+0000 7fb5443ff700 1 -- 172.21.15.27:0/108610242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb53c0718d0 con 0x7fb53c0b09e0 2024-01-11T09:27:28.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.839+0000 7fb5327fc700 1 -- 172.21.15.27:0/108610242 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb52c01c460 con 0x7fb53c0b09e0 2024-01-11T09:27:28.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.840+0000 7fb5327fc700 1 --2- 172.21.15.27:0/108610242 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb520066f30 0x7fb5200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.840+0000 7fb5327fc700 1 -- 172.21.15.27:0/108610242 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fb52c0842a0 con 0x7fb53c0b09e0 2024-01-11T09:27:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.840+0000 7fb54219b700 1 --2- 172.21.15.27:0/108610242 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb520066f30 0x7fb5200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.841+0000 7fb54219b700 1 --2- 172.21.15.27:0/108610242 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb520066f30 0x7fb5200693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb5380113e0 tx=0x7fb5380091a0 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:28.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:28.844+0000 7fb5327fc700 1 -- 172.21.15.27:0/108610242 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb52c0514d0 con 0x7fb53c0b09e0 2024-01-11T09:27:29.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:29.097+0000 7fb5443ff700 1 -- 172.21.15.27:0/108610242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb53c14f5f0 con 0x7fb53c0b09e0 2024-01-11T09:27:29.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:29.098+0000 7fb5327fc700 1 -- 172.21.15.27:0/108610242 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v94) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb52c054af0 con 0x7fb53c0b09e0 2024-01-11T09:27:29.095 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-11T09:27:29.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:29.100+0000 7fb5443ff700 1 -- 172.21.15.27:0/108610242 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb520066f30 msgr2=0x7fb5200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:29.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:29.100+0000 7fb5443ff700 1 --2- 172.21.15.27:0/108610242 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb520066f30 0x7fb5200693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb5380113e0 tx=0x7fb5380091a0 comp rx=0 tx=0).stop 2024-01-11T09:27:29.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:29.100+0000 7fb5443ff700 1 -- 172.21.15.27:0/108610242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb53c0b09e0 msgr2=0x7fb53c14e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:29.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:29.100+0000 7fb5443ff700 1 --2- 172.21.15.27:0/108610242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb53c0b09e0 0x7fb53c14e3e0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fb52c00c810 tx=0x7fb52c007df0 comp rx=0 tx=0).stop 2024-01-11T09:27:29.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:29.100+0000 7fb5443ff700 1 -- 172.21.15.27:0/108610242 shutdown_connections 2024-01-11T09:27:29.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:29.100+0000 7fb5443ff700 1 --2- 172.21.15.27:0/108610242 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb520066f30 0x7fb5200693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:29.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:29.100+0000 7fb5443ff700 1 --2- 172.21.15.27:0/108610242 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb53c0c8620 0x7fb53c148800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:29.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:29.100+0000 7fb5443ff700 1 --2- 172.21.15.27:0/108610242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb53c069d40 0x7fb53c14c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:29.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:29.100+0000 7fb5443ff700 1 --2- 172.21.15.27:0/108610242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb53c0b09e0 0x7fb53c14e3e0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:29.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:29.100+0000 7fb5443ff700 1 -- 172.21.15.27:0/108610242 >> 172.21.15.27:0/108610242 conn(0x7fb53c0680e0 msgr2=0x7fb53c0cca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:29.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:29.100+0000 7fb5443ff700 1 -- 172.21.15.27:0/108610242 shutdown_connections 2024-01-11T09:27:29.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:29.100+0000 7fb5443ff700 1 -- 172.21.15.27:0/108610242 wait complete. 2024-01-11T09:27:29.116 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-11T09:27:29.116 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-11T09:27:29.116 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-11T09:27:29.116 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-11T09:27:29.116 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-11T09:27:29.116 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-11T09:27:29.116 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-11T09:27:29.116 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-11T09:27:29.116 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T09:27:29.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:27:29.117 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-11T09:27:29.165 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T09:27:29.166 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T09:27:29.166 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T09:27:29.166 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:27:29.166 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-11T09:27:29.166 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:27:29.166 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T09:27:29.166 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-11T09:27:29.166 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-11T09:27:29.166 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T09:27:29.166 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-11T09:27:29.166 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-11T09:27:29.166 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-11T09:27:29.167 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-11T09:27:29.167 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-11T09:27:29.167 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-11T09:27:29.167 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 37873sec preferred_lft 37873sec 2024-01-11T09:27:29.167 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-11T09:27:29.167 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:27:29.167 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T09:27:29.167 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-11T09:27:29.167 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-11T09:27:29.167 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T09:27:29.167 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff 2024-01-11T09:27:29.167 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-11T09:27:29.167 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:27:29.167 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::acd2:b3ff:fe6a:db4b/64 scope link 2024-01-11T09:27:29.168 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:27:29.168 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T09:27:29.168 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:fa:85:1b:3a:bc brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-11T09:27:29.168 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ecfa:85ff:fe1b:3abc/64 scope link 2024-01-11T09:27:29.168 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:27:29.168 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T09:27:29.168 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-11T09:27:29.168 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ec88:faff:fe17:41aa/64 scope link 2024-01-11T09:27:29.168 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:27:29.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:27:29.169 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-11T09:27:29.239 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-11T09:27:29.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:27:29.242 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-11T09:27:29.292 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-11T09:27:29.292 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-11T09:27:29.292 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-11T09:27:29.292 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-11T09:27:29.292 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-11T09:27:29.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:27:29.293 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T09:27:29.294 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T09:27:29.294 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-11T09:27:29.294 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-11T09:27:29.294 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-11T09:27:29.294 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T09:27:29.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:27:29.470 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T09:27:29.518 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-11T09:27:29.518 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-11T09:27:29.519 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T09:27:29.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:27:29.519 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-11T09:27:29.592 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T09:27:29.592 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-11T09:27:29.640 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-11T09:27:29.640 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T09:27:29.640 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-11T09:27:29.687 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-11T09:27:29.687 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-11T09:27:29.687 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:27:29.830 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:27:29.830 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T09:27:29.830 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T09:27:29.845 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:27:29.845 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T09:27:29.845 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T09:27:29.845 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T09:27:29.845 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T09:27:29.866 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:27:29.866 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T09:27:29.866 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T09:27:29.866 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T09:27:29.866 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T09:27:29.867 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790" 2024-01-11T09:27:29.867 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790:/ 2024-01-11T09:27:29.867 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=8fddb4a7-9d66-40f1-892d-6fe86c728ee4" will pass to kernel 2024-01-11T09:27:29.871 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-11T09:27:29.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:27:29.871 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-11T09:27:29.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-11T09:27:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.149+0000 7fc06cfff700 1 Processor -- start 2024-01-11T09:27:30.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.150+0000 7fc06cfff700 1 -- start start 2024-01-11T09:27:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.151+0000 7fc06cfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc0680686a0 0x7fc0680abd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.151+0000 7fc06cfff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0680ac250 con 0x7fc068068ba0 2024-01-11T09:27:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.151+0000 7fc06cfff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc068067b00 con 0x7fc0681596a0 2024-01-11T09:27:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.151+0000 7fc06cfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc06806bf60 con 0x7fc0680686a0 2024-01-11T09:27:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.151+0000 7fc065d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc068068ba0 0x7fc068067f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53584/0 (socket says 172.21.15.27:53584) 2024-01-11T09:27:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.151+0000 7fc065d9b700 1 -- 172.21.15.27:0/2663355255 learned_addr learned my addr 172.21.15.27:0/2663355255 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.151+0000 7fc066d9d700 1 --2- 172.21.15.27:0/2663355255 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc0680686a0 0x7fc0680abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:30.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.152+0000 7fc06559a700 1 -- 172.21.15.27:0/2663355255 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069462216 0 0) 0x7fc0680ac250 con 0x7fc068068ba0 2024-01-11T09:27:30.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.152+0000 7fc06559a700 1 -- 172.21.15.27:0/2663355255 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc050003570 con 0x7fc068068ba0 2024-01-11T09:27:30.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.152+0000 7fc06559a700 1 -- 172.21.15.27:0/2663355255 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632014404 0 0) 0x7fc068067b00 con 0x7fc0681596a0 2024-01-11T09:27:30.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.152+0000 7fc06559a700 1 -- 172.21.15.27:0/2663355255 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0680ac250 con 0x7fc0681596a0 2024-01-11T09:27:30.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.152+0000 7fc066d9d700 1 -- 172.21.15.27:0/2663355255 >> v1:172.21.15.77:6789/0 conn(0x7fc0681596a0 legacy=0x7fc0680af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:30.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.152+0000 7fc066d9d700 1 -- 172.21.15.27:0/2663355255 >> v1:172.21.15.27:6789/0 conn(0x7fc068068ba0 legacy=0x7fc068067f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.152+0000 7fc066d9d700 1 -- 172.21.15.27:0/2663355255 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06806c240 con 0x7fc0680686a0 2024-01-11T09:27:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.152+0000 7fc066d9d700 1 --2- 172.21.15.27:0/2663355255 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc0680686a0 0x7fc0680abd10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc060009e90 tx=0x7fc060009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d73a3fc005791ac2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.153+0000 7fc06559a700 1 -- 172.21.15.27:0/2663355255 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06000e070 con 0x7fc0680686a0 2024-01-11T09:27:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.153+0000 7fc06559a700 1 -- 172.21.15.27:0/2663355255 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06000bb80 con 0x7fc0680686a0 2024-01-11T09:27:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.153+0000 7fc06559a700 1 -- 172.21.15.27:0/2663355255 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc060012430 con 0x7fc0680686a0 2024-01-11T09:27:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.153+0000 7fc06cfff700 1 -- 172.21.15.27:0/2663355255 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc0680686a0 msgr2=0x7fc0680abd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:30.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.153+0000 7fc06cfff700 1 --2- 172.21.15.27:0/2663355255 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc0680686a0 0x7fc0680abd10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc060009e90 tx=0x7fc060009780 comp rx=0 tx=0).stop 2024-01-11T09:27:30.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.153+0000 7fc06cfff700 1 -- 172.21.15.27:0/2663355255 shutdown_connections 2024-01-11T09:27:30.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.153+0000 7fc06cfff700 1 --2- 172.21.15.27:0/2663355255 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc0680686a0 0x7fc0680abd10 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:30.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.153+0000 7fc06cfff700 1 -- 172.21.15.27:0/2663355255 >> 172.21.15.27:0/2663355255 conn(0x7fc06806a8b0 msgr2=0x7fc0680ae520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:30.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.153+0000 7fc06cfff700 1 -- 172.21.15.27:0/2663355255 shutdown_connections 2024-01-11T09:27:30.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.153+0000 7fc06cfff700 1 -- 172.21.15.27:0/2663355255 wait complete. 2024-01-11T09:27:30.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.155+0000 7fc06cfff700 1 Processor -- start 2024-01-11T09:27:30.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.155+0000 7fc06cfff700 1 -- start start 2024-01-11T09:27:30.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.156+0000 7fc06cfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0680686a0 0x7fc06815c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:30.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.156+0000 7fc06cfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc068068ba0 0x7fc0680ca710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:30.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.156+0000 7fc06cfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0681596a0 0x7fc0681699f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:30.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.156+0000 7fc06cfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0680ca280 con 0x7fc0680686a0 2024-01-11T09:27:30.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.156+0000 7fc06cfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0680c9670 con 0x7fc0681596a0 2024-01-11T09:27:30.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.156+0000 7fc06cfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0680c9950 con 0x7fc068068ba0 2024-01-11T09:27:30.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.156+0000 7fc06659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0680686a0 0x7fc06815c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:30.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.156+0000 7fc06659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0680686a0 0x7fc06815c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37794/0 (socket says 172.21.15.27:37794) 2024-01-11T09:27:30.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.156+0000 7fc06659c700 1 -- 172.21.15.27:0/4283783573 learned_addr learned my addr 172.21.15.27:0/4283783573 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:30.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.156+0000 7fc066d9d700 1 --2- 172.21.15.27:0/4283783573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0681596a0 0x7fc0681699f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:30.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.156+0000 7fc065d9b700 1 --2- 172.21.15.27:0/4283783573 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc068068ba0 0x7fc0680ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:30.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.157+0000 7fc06659c700 1 -- 172.21.15.27:0/4283783573 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc068068ba0 msgr2=0x7fc0680ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:30.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.157+0000 7fc06659c700 1 --2- 172.21.15.27:0/4283783573 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc068068ba0 0x7fc0680ca710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:30.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.157+0000 7fc06659c700 1 -- 172.21.15.27:0/4283783573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0681596a0 msgr2=0x7fc0681699f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:30.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.157+0000 7fc06659c700 1 --2- 172.21.15.27:0/4283783573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0681596a0 0x7fc0681699f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:30.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.157+0000 7fc06659c700 1 -- 172.21.15.27:0/4283783573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc060009a20 con 0x7fc0680686a0 2024-01-11T09:27:30.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.157+0000 7fc06659c700 1 --2- 172.21.15.27:0/4283783573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0680686a0 0x7fc06815c230 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fc05800c590 tx=0x7fc058007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:30.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.157+0000 7fc0567fc700 1 -- 172.21.15.27:0/4283783573 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc058010510 con 0x7fc0680686a0 2024-01-11T09:27:30.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.157+0000 7fc0567fc700 1 -- 172.21.15.27:0/4283783573 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc058010690 con 0x7fc0680686a0 2024-01-11T09:27:30.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.157+0000 7fc0567fc700 1 -- 172.21.15.27:0/4283783573 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc058010a50 con 0x7fc0680686a0 2024-01-11T09:27:30.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.157+0000 7fc06cfff700 1 -- 172.21.15.27:0/4283783573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0680c9c30 con 0x7fc0680686a0 2024-01-11T09:27:30.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.157+0000 7fc06cfff700 1 -- 172.21.15.27:0/4283783573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc06815c770 con 0x7fc0680686a0 2024-01-11T09:27:30.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.158+0000 7fc06cfff700 1 -- 172.21.15.27:0/4283783573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc06804c3c0 con 0x7fc0680686a0 2024-01-11T09:27:30.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.159+0000 7fc0567fc700 1 -- 172.21.15.27:0/4283783573 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc05801b400 con 0x7fc0680686a0 2024-01-11T09:27:30.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.161+0000 7fc0567fc700 1 --2- 172.21.15.27:0/4283783573 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc048066f30 0x7fc0480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:30.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.162+0000 7fc0567fc700 1 -- 172.21.15.27:0/4283783573 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fc0580841c0 con 0x7fc0680686a0 2024-01-11T09:27:30.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.162+0000 7fc065d9b700 1 --2- 172.21.15.27:0/4283783573 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc048066f30 0x7fc0480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:30.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.163+0000 7fc065d9b700 1 --2- 172.21.15.27:0/4283783573 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc048066f30 0x7fc0480693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc0680cb290 tx=0x7fc05c009040 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:30.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.165+0000 7fc0567fc700 1 -- 172.21.15.27:0/4283783573 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc058051350 con 0x7fc0680686a0 2024-01-11T09:27:30.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.396+0000 7fc06cfff700 1 -- 172.21.15.27:0/4283783573 --> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0680ca3c0 con 0x7fc048066f30 2024-01-11T09:27:30.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.399+0000 7fc0567fc700 1 -- 172.21.15.27:0/4283783573 <== mgr.9010 v2:172.21.15.77:6836/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fc05c00f680 con 0x7fc048066f30 2024-01-11T09:27:30.396 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T09:27:30.396 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:27:30.396 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-11T09:27:30.396 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:27:30.396 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T09:27:30.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.400+0000 7fc06cfff700 1 -- 172.21.15.27:0/4283783573 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc048066f30 msgr2=0x7fc0480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:30.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.400+0000 7fc06cfff700 1 --2- 172.21.15.27:0/4283783573 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc048066f30 0x7fc0480693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc0680cb290 tx=0x7fc05c009040 comp rx=0 tx=0).stop 2024-01-11T09:27:30.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.400+0000 7fc06cfff700 1 -- 172.21.15.27:0/4283783573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0680686a0 msgr2=0x7fc06815c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:30.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.400+0000 7fc06cfff700 1 --2- 172.21.15.27:0/4283783573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0680686a0 0x7fc06815c230 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fc05800c590 tx=0x7fc058007dd0 comp rx=0 tx=0).stop 2024-01-11T09:27:30.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.401+0000 7fc06cfff700 1 -- 172.21.15.27:0/4283783573 shutdown_connections 2024-01-11T09:27:30.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.401+0000 7fc06cfff700 1 --2- 172.21.15.27:0/4283783573 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc048066f30 0x7fc0480693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:30.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.401+0000 7fc06cfff700 1 --2- 172.21.15.27:0/4283783573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0681596a0 0x7fc0681699f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:30.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.401+0000 7fc06cfff700 1 --2- 172.21.15.27:0/4283783573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0680686a0 0x7fc06815c230 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:30.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.401+0000 7fc06cfff700 1 --2- 172.21.15.27:0/4283783573 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc068068ba0 0x7fc0680ca710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:30.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.401+0000 7fc06cfff700 1 -- 172.21.15.27:0/4283783573 >> 172.21.15.27:0/4283783573 conn(0x7fc06806a8b0 msgr2=0x7fc0680aa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:30.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.401+0000 7fc06cfff700 1 -- 172.21.15.27:0/4283783573 shutdown_connections 2024-01-11T09:27:30.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.401+0000 7fc06cfff700 1 -- 172.21.15.27:0/4283783573 wait complete. 2024-01-11T09:27:30.416 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-11T09:27:30.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.620+0000 7f83125ff700 1 Processor -- start 2024-01-11T09:27:30.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.621+0000 7f83125ff700 1 -- start start 2024-01-11T09:27:30.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.622+0000 7f83125ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f830c069540 0x7f830c162270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:30.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.622+0000 7f83125ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830c0699d0 con 0x7f830c157410 2024-01-11T09:27:30.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.622+0000 7f83125ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830c069cb0 con 0x7f830c1577e0 2024-01-11T09:27:30.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.622+0000 7f83125ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f830c069f90 con 0x7f830c069540 2024-01-11T09:27:30.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.622+0000 7f830bfff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f830c157410 0x7f830c0ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53590/0 (socket says 172.21.15.27:53590) 2024-01-11T09:27:30.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.622+0000 7f830bfff700 1 -- 172.21.15.27:0/2573040200 learned_addr learned my addr 172.21.15.27:0/2573040200 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:30.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.623+0000 7f8310b9c700 1 --2- 172.21.15.27:0/2573040200 >> v2:172.21.15.77:3300/0 conn(0x7f830c069540 0x7f830c162270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:30.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.623+0000 7f830affd700 1 -- 172.21.15.27:0/2573040200 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147284848 0 0) 0x7f830c0699d0 con 0x7f830c157410 2024-01-11T09:27:30.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.623+0000 7f830affd700 1 -- 172.21.15.27:0/2573040200 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82f4003570 con 0x7f830c157410 2024-01-11T09:27:30.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.623+0000 7f830affd700 1 -- 172.21.15.27:0/2573040200 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285469809 0 0) 0x7f830c069cb0 con 0x7f830c1577e0 2024-01-11T09:27:30.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.623+0000 7f830affd700 1 -- 172.21.15.27:0/2573040200 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f830c0699d0 con 0x7f830c1577e0 2024-01-11T09:27:30.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.623+0000 7f830affd700 1 -- 172.21.15.27:0/2573040200 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 637683457 0 0) 0x7f82f4003570 con 0x7f830c157410 2024-01-11T09:27:30.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.623+0000 7f830affd700 1 -- 172.21.15.27:0/2573040200 >> v2:172.21.15.77:3300/0 conn(0x7f830c069540 msgr2=0x7f830c162270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:30.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.623+0000 7f830affd700 1 --2- 172.21.15.27:0/2573040200 >> v2:172.21.15.77:3300/0 conn(0x7f830c069540 0x7f830c162270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:30.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.623+0000 7f830affd700 1 -- 172.21.15.27:0/2573040200 >> v1:172.21.15.77:6789/0 conn(0x7f830c1577e0 legacy=0x7f830c0ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:30.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.623+0000 7f830affd700 1 -- 172.21.15.27:0/2573040200 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830c06a270 con 0x7f830c157410 2024-01-11T09:27:30.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.623+0000 7f830affd700 1 -- 172.21.15.27:0/2573040200 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f82fc0015f0 con 0x7f830c157410 2024-01-11T09:27:30.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.623+0000 7f8310b9c700 1 --2- 172.21.15.27:0/2573040200 >> v2:172.21.15.77:3300/0 conn(0x7f830c069540 0x7f830c162270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:27:30.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.623+0000 7f830affd700 1 -- 172.21.15.27:0/2573040200 >> v1:172.21.15.27:6789/0 conn(0x7f830c157410 legacy=0x7f830c0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.623+0000 7f830affd700 1 --2- 172.21.15.27:0/2573040200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82f4003c40 0x7f82f4004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.623+0000 7f830affd700 1 --2- 172.21.15.27:0/2573040200 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f82f4004810 0x7f82f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.623+0000 7f830affd700 1 --2- 172.21.15.27:0/2573040200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f4005210 0x7f82f400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.623+0000 7f830affd700 1 -- 172.21.15.27:0/2573040200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830c069f90 con 0x7f82f4005210 2024-01-11T09:27:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.623+0000 7f830affd700 1 -- 172.21.15.27:0/2573040200 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f400c130 con 0x7f82f4003c40 2024-01-11T09:27:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.623+0000 7f830affd700 1 -- 172.21.15.27:0/2573040200 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f400c2c0 con 0x7f82f4004810 2024-01-11T09:27:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.624+0000 7f8310b9c700 1 --2- 172.21.15.27:0/2573040200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f4005210 0x7f82f400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.624+0000 7f830bfff700 1 --2- 172.21.15.27:0/2573040200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82f4003c40 0x7f82f4004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.624+0000 7f830b7fe700 1 --2- 172.21.15.27:0/2573040200 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f82f4004810 0x7f82f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.624+0000 7f8310b9c700 1 -- 172.21.15.27:0/2573040200 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f82f4004810 msgr2=0x7f82f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.624+0000 7f8310b9c700 1 --2- 172.21.15.27:0/2573040200 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f82f4004810 0x7f82f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.624+0000 7f8310b9c700 1 -- 172.21.15.27:0/2573040200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82f4003c40 msgr2=0x7f82f4004030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:30.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.624+0000 7f8310b9c700 1 --2- 172.21.15.27:0/2573040200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82f4003c40 0x7f82f4004030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:30.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.624+0000 7f8310b9c700 1 -- 172.21.15.27:0/2573040200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f4003570 con 0x7f82f4005210 2024-01-11T09:27:30.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.624+0000 7f830bfff700 1 --2- 172.21.15.27:0/2573040200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82f4003c40 0x7f82f4004030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:27:30.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.624+0000 7f8310b9c700 1 -- 172.21.15.27:0/2573040200 reap_dead start 2024-01-11T09:27:30.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.624+0000 7f8310b9c700 1 --2- 172.21.15.27:0/2573040200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f4005210 0x7f82f400ba60 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f830400e3f0 tx=0x7f830400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba19ee4a9071cd1e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:30.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.625+0000 7f830affd700 1 -- 172.21.15.27:0/2573040200 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8304009c50 con 0x7f82f4005210 2024-01-11T09:27:30.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.625+0000 7f830affd700 1 -- 172.21.15.27:0/2573040200 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f830400f040 con 0x7f82f4005210 2024-01-11T09:27:30.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.625+0000 7f830affd700 1 -- 172.21.15.27:0/2573040200 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8304014480 con 0x7f82f4005210 2024-01-11T09:27:30.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.625+0000 7f83125ff700 1 -- 172.21.15.27:0/2573040200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f4005210 msgr2=0x7f82f400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:30.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.625+0000 7f83125ff700 1 --2- 172.21.15.27:0/2573040200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f4005210 0x7f82f400ba60 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f830400e3f0 tx=0x7f830400b040 comp rx=0 tx=0).stop 2024-01-11T09:27:30.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.625+0000 7f83125ff700 1 -- 172.21.15.27:0/2573040200 shutdown_connections 2024-01-11T09:27:30.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.625+0000 7f83125ff700 1 --2- 172.21.15.27:0/2573040200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f4005210 0x7f82f400ba60 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:30.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.625+0000 7f83125ff700 1 -- 172.21.15.27:0/2573040200 >> 172.21.15.27:0/2573040200 conn(0x7f830c0680a0 msgr2=0x7f830c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:30.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.625+0000 7f83125ff700 1 -- 172.21.15.27:0/2573040200 shutdown_connections 2024-01-11T09:27:30.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.625+0000 7f83125ff700 1 -- 172.21.15.27:0/2573040200 wait complete. 2024-01-11T09:27:30.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.627+0000 7f83125ff700 1 Processor -- start 2024-01-11T09:27:30.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.627+0000 7f83125ff700 1 -- start start 2024-01-11T09:27:30.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.628+0000 7f83125ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f4005210 0x7f830c169930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:30.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.628+0000 7f83125ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f830c16cf80 0x7f830c1512d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:30.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.628+0000 7f83125ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f830c0c4fa0 0x7f830c151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:30.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.628+0000 7f83125ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830c0c3270 con 0x7f82f4005210 2024-01-11T09:27:30.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.628+0000 7f83125ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830c16d3f0 con 0x7f830c16cf80 2024-01-11T09:27:30.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.628+0000 7f83125ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f830c16d6d0 con 0x7f830c0c4fa0 2024-01-11T09:27:30.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.628+0000 7f830bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f4005210 0x7f830c169930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:30.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.628+0000 7f830bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f4005210 0x7f830c169930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37806/0 (socket says 172.21.15.27:37806) 2024-01-11T09:27:30.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.628+0000 7f830bfff700 1 -- 172.21.15.27:0/112819083 learned_addr learned my addr 172.21.15.27:0/112819083 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:30.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.628+0000 7f8310b9c700 1 --2- 172.21.15.27:0/112819083 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f830c0c4fa0 0x7f830c151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:30.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.628+0000 7f830b7fe700 1 --2- 172.21.15.27:0/112819083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f830c16cf80 0x7f830c1512d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:30.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.628+0000 7f830bfff700 1 -- 172.21.15.27:0/112819083 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f830c0c4fa0 msgr2=0x7f830c151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:30.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.628+0000 7f830bfff700 1 --2- 172.21.15.27:0/112819083 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f830c0c4fa0 0x7f830c151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:30.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.628+0000 7f830bfff700 1 -- 172.21.15.27:0/112819083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f830c16cf80 msgr2=0x7f830c1512d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:30.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.628+0000 7f830bfff700 1 --2- 172.21.15.27:0/112819083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f830c16cf80 0x7f830c1512d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:30.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.628+0000 7f830bfff700 1 -- 172.21.15.27:0/112819083 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830c16d9b0 con 0x7f82f4005210 2024-01-11T09:27:30.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.629+0000 7f830bfff700 1 --2- 172.21.15.27:0/112819083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f4005210 0x7f830c169930 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f82fc00ff30 tx=0x7f82fc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:30.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.629+0000 7f82f3fff700 1 -- 172.21.15.27:0/112819083 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc00d9d0 con 0x7f82f4005210 2024-01-11T09:27:30.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.629+0000 7f82f3fff700 1 -- 172.21.15.27:0/112819083 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82fc00db30 con 0x7f82f4005210 2024-01-11T09:27:30.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.629+0000 7f83125ff700 1 -- 172.21.15.27:0/112819083 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f830c16dc90 con 0x7f82f4005210 2024-01-11T09:27:30.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.629+0000 7f82f3fff700 1 -- 172.21.15.27:0/112819083 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc0070d0 con 0x7f82f4005210 2024-01-11T09:27:30.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.629+0000 7f83125ff700 1 -- 172.21.15.27:0/112819083 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f830c16a030 con 0x7f82f4005210 2024-01-11T09:27:30.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.630+0000 7f83125ff700 1 -- 172.21.15.27:0/112819083 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830c04c3c0 con 0x7f82f4005210 2024-01-11T09:27:30.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.631+0000 7f82f3fff700 1 -- 172.21.15.27:0/112819083 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 99) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f82fc007480 con 0x7f82f4005210 2024-01-11T09:27:30.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.632+0000 7f82f3fff700 1 --2- 172.21.15.27:0/112819083 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f82f8066f30 0x7f82f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:30.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.632+0000 7f82f3fff700 1 -- 172.21.15.27:0/112819083 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f82fc0846e0 con 0x7f82f4005210 2024-01-11T09:27:30.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.632+0000 7f830b7fe700 1 --2- 172.21.15.27:0/112819083 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f82f8066f30 0x7f82f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:30.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.633+0000 7f830b7fe700 1 --2- 172.21.15.27:0/112819083 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f82f8066f30 0x7f82f80693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8300000c00 tx=0x7f8300009040 comp rx=0 tx=0).ready entity=mgr.9010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:30.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.636+0000 7f82f3fff700 1 -- 172.21.15.27:0/112819083 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f82fc051910 con 0x7f82f4005210 2024-01-11T09:27:30.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:30.893+0000 7f83125ff700 1 -- 172.21.15.27:0/112819083 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f830c0c6ba0 con 0x7f82f4005210 2024-01-11T09:27:31.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.651+0000 7f82f3fff700 1 -- 172.21.15.27:0/112819083 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 100) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f82fc051ce0 con 0x7f82f4005210 2024-01-11T09:27:31.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.653+0000 7f82f3fff700 1 -- 172.21.15.27:0/112819083 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v100) v1 ==== 92+0+0 (secure 0 0 0) 0x7f82fc0553a0 con 0x7f82f4005210 2024-01-11T09:27:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.657+0000 7f83125ff700 1 -- 172.21.15.27:0/112819083 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f82f8066f30 msgr2=0x7f82f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.657+0000 7f83125ff700 1 --2- 172.21.15.27:0/112819083 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f82f8066f30 0x7f82f80693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8300000c00 tx=0x7f8300009040 comp rx=0 tx=0).stop 2024-01-11T09:27:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.657+0000 7f83125ff700 1 -- 172.21.15.27:0/112819083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f4005210 msgr2=0x7f830c169930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.657+0000 7f83125ff700 1 --2- 172.21.15.27:0/112819083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f4005210 0x7f830c169930 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f82fc00ff30 tx=0x7f82fc009040 comp rx=0 tx=0).stop 2024-01-11T09:27:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.657+0000 7f83125ff700 1 -- 172.21.15.27:0/112819083 shutdown_connections 2024-01-11T09:27:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.657+0000 7f83125ff700 1 --2- 172.21.15.27:0/112819083 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f82f8066f30 0x7f82f80693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.657+0000 7f83125ff700 1 --2- 172.21.15.27:0/112819083 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f830c0c4fa0 0x7f830c151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.657+0000 7f83125ff700 1 --2- 172.21.15.27:0/112819083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f4005210 0x7f830c169930 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.657+0000 7f83125ff700 1 --2- 172.21.15.27:0/112819083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f830c16cf80 0x7f830c1512d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.657+0000 7f83125ff700 1 -- 172.21.15.27:0/112819083 >> 172.21.15.27:0/112819083 conn(0x7f830c0680a0 msgr2=0x7f830c1529e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.657+0000 7f83125ff700 1 -- 172.21.15.27:0/112819083 shutdown_connections 2024-01-11T09:27:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.657+0000 7f83125ff700 1 -- 172.21.15.27:0/112819083 wait complete. 2024-01-11T09:27:31.676 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-11T09:27:31.864 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:31.867+0000 7fd631a881c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:27:31.872 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:31.875+0000 7f31879691c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:27:31.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.891+0000 7fc2855ff700 1 Processor -- start 2024-01-11T09:27:31.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.891+0000 7fc2855ff700 1 -- start start 2024-01-11T09:27:31.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.893+0000 7fc2855ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc280157410 0x7fc2800ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:31.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.893+0000 7fc2855ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2801577e0 0x7fc2800ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:31.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.893+0000 7fc2855ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2800699d0 con 0x7fc280069540 2024-01-11T09:27:31.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.893+0000 7fc2855ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc280069cb0 con 0x7fc2801577e0 2024-01-11T09:27:31.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.893+0000 7fc2855ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc280069f90 con 0x7fc280157410 2024-01-11T09:27:31.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.893+0000 7fc27e7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2801577e0 0x7fc2800ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:31.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.893+0000 7fc27effd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc280157410 0x7fc2800ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:31.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.893+0000 7fc27e7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2801577e0 0x7fc2800ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37828/0 (socket says 172.21.15.27:37828) 2024-01-11T09:27:31.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.893+0000 7fc27e7fc700 1 -- 172.21.15.27:0/1968242797 learned_addr learned my addr 172.21.15.27:0/1968242797 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:31.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.894+0000 7fc27e7fc700 1 -- 172.21.15.27:0/1968242797 >> v2:172.21.15.77:3300/0 conn(0x7fc280157410 msgr2=0x7fc2800ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:31.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.894+0000 7fc27e7fc700 1 --2- 172.21.15.27:0/1968242797 >> v2:172.21.15.77:3300/0 conn(0x7fc280157410 0x7fc2800ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:31.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.894+0000 7fc27e7fc700 1 -- 172.21.15.27:0/1968242797 >> v1:172.21.15.77:6789/0 conn(0x7fc280069540 legacy=0x7fc28015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:31.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.894+0000 7fc27dffb700 1 -- 172.21.15.27:0/1968242797 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244932411 0 0) 0x7fc2800699d0 con 0x7fc280069540 2024-01-11T09:27:31.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.894+0000 7fc27e7fc700 1 -- 172.21.15.27:0/1968242797 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28006a270 con 0x7fc2801577e0 2024-01-11T09:27:31.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.894+0000 7fc27e7fc700 1 --2- 172.21.15.27:0/1968242797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2801577e0 0x7fc2800ae000 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fc280070c90 tx=0x7fc274009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4acb920a4a94a835 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:31.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.894+0000 7fc27dffb700 1 -- 172.21.15.27:0/1968242797 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc274009e70 con 0x7fc2801577e0 2024-01-11T09:27:31.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.894+0000 7fc27dffb700 1 -- 172.21.15.27:0/1968242797 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc27400b860 con 0x7fc2801577e0 2024-01-11T09:27:31.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.894+0000 7fc27dffb700 1 -- 172.21.15.27:0/1968242797 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc27400bc00 con 0x7fc2801577e0 2024-01-11T09:27:31.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.895+0000 7fc2855ff700 1 -- 172.21.15.27:0/1968242797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2801577e0 msgr2=0x7fc2800ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:31.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.895+0000 7fc2855ff700 1 --2- 172.21.15.27:0/1968242797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2801577e0 0x7fc2800ae000 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fc280070c90 tx=0x7fc274009250 comp rx=0 tx=0).stop 2024-01-11T09:27:31.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.895+0000 7fc2855ff700 1 -- 172.21.15.27:0/1968242797 shutdown_connections 2024-01-11T09:27:31.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.895+0000 7fc2855ff700 1 --2- 172.21.15.27:0/1968242797 >> v2:172.21.15.77:3300/0 conn(0x7fc280157410 0x7fc2800ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:31.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.895+0000 7fc2855ff700 1 --2- 172.21.15.27:0/1968242797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2801577e0 0x7fc2800ae000 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:31.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.895+0000 7fc2855ff700 1 -- 172.21.15.27:0/1968242797 >> 172.21.15.27:0/1968242797 conn(0x7fc2800680a0 msgr2=0x7fc2800cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:31.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.895+0000 7fc2855ff700 1 -- 172.21.15.27:0/1968242797 shutdown_connections 2024-01-11T09:27:31.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.895+0000 7fc2855ff700 1 -- 172.21.15.27:0/1968242797 wait complete. 2024-01-11T09:27:31.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.896+0000 7fc2855ff700 1 Processor -- start 2024-01-11T09:27:31.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.897+0000 7fc2855ff700 1 -- start start 2024-01-11T09:27:31.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.897+0000 7fc2855ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc280069540 0x7fc28015a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:31.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.897+0000 7fc2855ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc280157410 0x7fc28015ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:31.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.898+0000 7fc2855ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2801577e0 0x7fc28015bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:31.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.898+0000 7fc2855ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2800c4b80 con 0x7fc280157410 2024-01-11T09:27:31.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.898+0000 7fc2855ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc280159290 con 0x7fc280069540 2024-01-11T09:27:31.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.898+0000 7fc2855ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc280159570 con 0x7fc2801577e0 2024-01-11T09:27:31.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.898+0000 7fc27e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc280157410 0x7fc28015ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:31.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.898+0000 7fc27e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc280157410 0x7fc28015ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37844/0 (socket says 172.21.15.27:37844) 2024-01-11T09:27:31.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.898+0000 7fc27e7fc700 1 -- 172.21.15.27:0/3164488608 learned_addr learned my addr 172.21.15.27:0/3164488608 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:31.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.898+0000 7fc27effd700 1 --2- 172.21.15.27:0/3164488608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc280069540 0x7fc28015a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:31.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.898+0000 7fc27f7fe700 1 --2- 172.21.15.27:0/3164488608 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2801577e0 0x7fc28015bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:31.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.898+0000 7fc27e7fc700 1 -- 172.21.15.27:0/3164488608 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2801577e0 msgr2=0x7fc28015bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:31.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.898+0000 7fc27e7fc700 1 --2- 172.21.15.27:0/3164488608 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2801577e0 0x7fc28015bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:31.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.898+0000 7fc27e7fc700 1 -- 172.21.15.27:0/3164488608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc280069540 msgr2=0x7fc28015a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:31.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.898+0000 7fc27e7fc700 1 --2- 172.21.15.27:0/3164488608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc280069540 0x7fc28015a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:31.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.898+0000 7fc27e7fc700 1 -- 172.21.15.27:0/3164488608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2740094f0 con 0x7fc280157410 2024-01-11T09:27:31.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.899+0000 7fc27e7fc700 1 --2- 172.21.15.27:0/3164488608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc280157410 0x7fc28015ba30 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fc2740094c0 tx=0x7fc274017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.899+0000 7fc266ffd700 1 -- 172.21.15.27:0/3164488608 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc274000c90 con 0x7fc280157410 2024-01-11T09:27:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.899+0000 7fc266ffd700 1 -- 172.21.15.27:0/3164488608 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc27400b9d0 con 0x7fc280157410 2024-01-11T09:27:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.899+0000 7fc266ffd700 1 -- 172.21.15.27:0/3164488608 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc274025420 con 0x7fc280157410 2024-01-11T09:27:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.899+0000 7fc2855ff700 1 -- 172.21.15.27:0/3164488608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc280159850 con 0x7fc280157410 2024-01-11T09:27:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.899+0000 7fc2855ff700 1 -- 172.21.15.27:0/3164488608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc280159e10 con 0x7fc280157410 2024-01-11T09:27:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.900+0000 7fc2855ff700 1 -- 172.21.15.27:0/3164488608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28015a050 con 0x7fc280157410 2024-01-11T09:27:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.901+0000 7fc266ffd700 1 -- 172.21.15.27:0/3164488608 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 100) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fc274023070 con 0x7fc280157410 2024-01-11T09:27:31.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.902+0000 7fc266ffd700 1 --2- 172.21.15.27:0/3164488608 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc26c066f80 0x7fc26c069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:31.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.902+0000 7fc266ffd700 1 -- 172.21.15.27:0/3164488608 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fc274085ac0 con 0x7fc280157410 2024-01-11T09:27:31.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.903+0000 7fc27effd700 1 -- 172.21.15.27:0/3164488608 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc26c066f80 msgr2=0x7fc26c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:31.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.903+0000 7fc27effd700 1 --2- 172.21.15.27:0/3164488608 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc26c066f80 0x7fc26c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:27:31.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:31.907+0000 7fc266ffd700 1 -- 172.21.15.27:0/3164488608 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc28015a050 con 0x7fc280157410 2024-01-11T09:27:32.043 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:32.046+0000 7fd631a881c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:27:32.073 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:32.076+0000 7f31879691c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:27:32.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.104+0000 7fc27effd700 1 -- 172.21.15.27:0/3164488608 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc26c066f80 msgr2=0x7fc26c069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:32.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.104+0000 7fc27effd700 1 --2- 172.21.15.27:0/3164488608 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc26c066f80 0x7fc26c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:27:32.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.154+0000 7fc2855ff700 1 -- 172.21.15.27:0/3164488608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fc28015a050 con 0x7fc280157410 2024-01-11T09:27:32.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.158+0000 7fc266ffd700 1 -- 172.21.15.27:0/3164488608 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v65)=0 v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc28015a050 con 0x7fc280157410 2024-01-11T09:27:32.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.160+0000 7fc2855ff700 1 -- 172.21.15.27:0/3164488608 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc26c066f80 msgr2=0x7fc26c069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:27:32.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.160+0000 7fc2855ff700 1 --2- 172.21.15.27:0/3164488608 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc26c066f80 0x7fc26c069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.160+0000 7fc2855ff700 1 -- 172.21.15.27:0/3164488608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc280157410 msgr2=0x7fc28015ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:32.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.160+0000 7fc2855ff700 1 --2- 172.21.15.27:0/3164488608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc280157410 0x7fc28015ba30 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fc2740094c0 tx=0x7fc274017000 comp rx=0 tx=0).stop 2024-01-11T09:27:32.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.160+0000 7fc2855ff700 1 -- 172.21.15.27:0/3164488608 shutdown_connections 2024-01-11T09:27:32.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.160+0000 7fc2855ff700 1 --2- 172.21.15.27:0/3164488608 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fc26c066f80 0x7fc26c069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.160+0000 7fc2855ff700 1 --2- 172.21.15.27:0/3164488608 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2801577e0 0x7fc28015bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.160+0000 7fc2855ff700 1 --2- 172.21.15.27:0/3164488608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc280069540 0x7fc28015a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.160+0000 7fc2855ff700 1 --2- 172.21.15.27:0/3164488608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc280157410 0x7fc28015ba30 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.160+0000 7fc2855ff700 1 -- 172.21.15.27:0/3164488608 >> 172.21.15.27:0/3164488608 conn(0x7fc2800680a0 msgr2=0x7fc2800c0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:32.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.160+0000 7fc2855ff700 1 -- 172.21.15.27:0/3164488608 shutdown_connections 2024-01-11T09:27:32.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.160+0000 7fc2855ff700 1 -- 172.21.15.27:0/3164488608 wait complete. 2024-01-11T09:27:32.176 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-11T09:27:32.358 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:32.361+0000 7fd631a881c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:27:32.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.384+0000 7fa1febff700 1 Processor -- start 2024-01-11T09:27:32.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.384+0000 7fa1febff700 1 -- start start 2024-01-11T09:27:32.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.386+0000 7fa1febff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa1f81577e0 0x7fa1f80ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:32.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.386+0000 7fa1febff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1f8069540 0x7fa1f80ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:32.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.386+0000 7fa1febff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f80699d0 con 0x7fa1f8157410 2024-01-11T09:27:32.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.386+0000 7fa1febff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1f8069cb0 con 0x7fa1f81577e0 2024-01-11T09:27:32.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.386+0000 7fa1febff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f8069f90 con 0x7fa1f8069540 2024-01-11T09:27:32.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.386+0000 7fa1f7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa1f81577e0 0x7fa1f80ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:32.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.386+0000 7fa1f7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa1f81577e0 0x7fa1f80ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37854/0 (socket says 172.21.15.27:37854) 2024-01-11T09:27:32.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.386+0000 7fa1f7fff700 1 -- 172.21.15.27:0/3034523953 learned_addr learned my addr 172.21.15.27:0/3034523953 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:32.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.386+0000 7fa1fd19c700 1 --2- 172.21.15.27:0/3034523953 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1f8069540 0x7fa1f80ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:32.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.387+0000 7fa1f7fff700 1 -- 172.21.15.27:0/3034523953 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1f8069540 msgr2=0x7fa1f80ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:32.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.387+0000 7fa1f7fff700 1 --2- 172.21.15.27:0/3034523953 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1f8069540 0x7fa1f80ae540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.387+0000 7fa1f7fff700 1 -- 172.21.15.27:0/3034523953 >> v1:172.21.15.77:6789/0 conn(0x7fa1f8157410 legacy=0x7fa1f80ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:32.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.387+0000 7fa1f7fff700 1 -- 172.21.15.27:0/3034523953 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f806a270 con 0x7fa1f81577e0 2024-01-11T09:27:32.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.387+0000 7fa1fd19c700 1 --2- 172.21.15.27:0/3034523953 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1f8069540 0x7fa1f80ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:27:32.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.387+0000 7fa1f7fff700 1 --2- 172.21.15.27:0/3034523953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1f81577e0 0x7fa1f80ae000 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40099b0 tx=0x7fa1e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d104e2b8eb719b5e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:32.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.387+0000 7fa1f77fe700 1 -- 172.21.15.27:0/3034523953 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e400b860 con 0x7fa1f81577e0 2024-01-11T09:27:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.387+0000 7fa1f77fe700 1 -- 172.21.15.27:0/3034523953 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e400b9c0 con 0x7fa1f81577e0 2024-01-11T09:27:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.387+0000 7fa1f77fe700 1 -- 172.21.15.27:0/3034523953 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e400bd60 con 0x7fa1f81577e0 2024-01-11T09:27:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.388+0000 7fa1febff700 1 -- 172.21.15.27:0/3034523953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1f81577e0 msgr2=0x7fa1f80ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.388+0000 7fa1febff700 1 --2- 172.21.15.27:0/3034523953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1f81577e0 0x7fa1f80ae000 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40099b0 tx=0x7fa1e4009250 comp rx=0 tx=0).stop 2024-01-11T09:27:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.388+0000 7fa1febff700 1 -- 172.21.15.27:0/3034523953 shutdown_connections 2024-01-11T09:27:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.388+0000 7fa1febff700 1 --2- 172.21.15.27:0/3034523953 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1f8069540 0x7fa1f80ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.388+0000 7fa1febff700 1 --2- 172.21.15.27:0/3034523953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1f81577e0 0x7fa1f80ae000 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.388+0000 7fa1febff700 1 -- 172.21.15.27:0/3034523953 >> 172.21.15.27:0/3034523953 conn(0x7fa1f80680a0 msgr2=0x7fa1f80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.388+0000 7fa1febff700 1 -- 172.21.15.27:0/3034523953 shutdown_connections 2024-01-11T09:27:32.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.388+0000 7fa1febff700 1 -- 172.21.15.27:0/3034523953 wait complete. 2024-01-11T09:27:32.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.390+0000 7fa1febff700 1 Processor -- start 2024-01-11T09:27:32.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.390+0000 7fa1febff700 1 -- start start 2024-01-11T09:27:32.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.391+0000 7fa1febff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1f8069540 0x7fa1f814bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:32.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.391+0000 7fa1febff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1f8157410 0x7fa1f814f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:32.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.391+0000 7fa1febff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1f81577e0 0x7fa1f80b0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:32.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.391+0000 7fa1febff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f814c100 con 0x7fa1f8157410 2024-01-11T09:27:32.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.391+0000 7fa1febff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f814d8b0 con 0x7fa1f8069540 2024-01-11T09:27:32.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.391+0000 7fa1febff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f814cca0 con 0x7fa1f81577e0 2024-01-11T09:27:32.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.391+0000 7fa1f7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1f8157410 0x7fa1f814f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:32.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.391+0000 7fa1f7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1f8157410 0x7fa1f814f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37868/0 (socket says 172.21.15.27:37868) 2024-01-11T09:27:32.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.391+0000 7fa1f7fff700 1 -- 172.21.15.27:0/2561281378 learned_addr learned my addr 172.21.15.27:0/2561281378 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:32.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.391+0000 7fa1fd19c700 1 --2- 172.21.15.27:0/2561281378 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1f81577e0 0x7fa1f80b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:32.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.391+0000 7fa1fc99b700 1 --2- 172.21.15.27:0/2561281378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1f8069540 0x7fa1f814bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:32.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.391+0000 7fa1f7fff700 1 -- 172.21.15.27:0/2561281378 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1f81577e0 msgr2=0x7fa1f80b0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:32.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.391+0000 7fa1f7fff700 1 --2- 172.21.15.27:0/2561281378 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1f81577e0 0x7fa1f80b0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.392+0000 7fa1f7fff700 1 -- 172.21.15.27:0/2561281378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1f8069540 msgr2=0x7fa1f814bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:32.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.392+0000 7fa1f7fff700 1 --2- 172.21.15.27:0/2561281378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1f8069540 0x7fa1f814bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.392+0000 7fa1f7fff700 1 -- 172.21.15.27:0/2561281378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e40094f0 con 0x7fa1f8157410 2024-01-11T09:27:32.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.392+0000 7fa1f7fff700 1 --2- 172.21.15.27:0/2561281378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1f8157410 0x7fa1f814f620 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40090b0 tx=0x7fa1e4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:32.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.392+0000 7fa1f4ff9700 1 -- 172.21.15.27:0/2561281378 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4011e00 con 0x7fa1f8157410 2024-01-11T09:27:32.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.392+0000 7fa1f4ff9700 1 -- 172.21.15.27:0/2561281378 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e401f070 con 0x7fa1f8157410 2024-01-11T09:27:32.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.392+0000 7fa1febff700 1 -- 172.21.15.27:0/2561281378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f814cf80 con 0x7fa1f8157410 2024-01-11T09:27:32.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.392+0000 7fa1f4ff9700 1 -- 172.21.15.27:0/2561281378 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40266d0 con 0x7fa1f8157410 2024-01-11T09:27:32.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.392+0000 7fa1febff700 1 -- 172.21.15.27:0/2561281378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f814d540 con 0x7fa1f8157410 2024-01-11T09:27:32.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.394+0000 7fa1febff700 1 -- 172.21.15.27:0/2561281378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f814da20 con 0x7fa1f8157410 2024-01-11T09:27:32.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.395+0000 7fa1f4ff9700 1 -- 172.21.15.27:0/2561281378 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 100) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fa1e4026a40 con 0x7fa1f8157410 2024-01-11T09:27:32.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.397+0000 7fa1f4ff9700 1 --2- 172.21.15.27:0/2561281378 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa1dc066f80 0x7fa1dc069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:32.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.397+0000 7fa1f4ff9700 1 -- 172.21.15.27:0/2561281378 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fa1e4021070 con 0x7fa1f8157410 2024-01-11T09:27:32.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.397+0000 7fa1fc99b700 1 -- 172.21.15.27:0/2561281378 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa1dc066f80 msgr2=0x7fa1dc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:32.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.397+0000 7fa1fc99b700 1 --2- 172.21.15.27:0/2561281378 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa1dc066f80 0x7fa1dc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:27:32.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.401+0000 7fa1f4ff9700 1 -- 172.21.15.27:0/2561281378 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa1e4053500 con 0x7fa1f8157410 2024-01-11T09:27:32.417 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:32.420+0000 7f31879691c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:27:32.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.598+0000 7fa1fc99b700 1 -- 172.21.15.27:0/2561281378 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa1dc066f80 msgr2=0x7fa1dc069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:32.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.598+0000 7fa1fc99b700 1 --2- 172.21.15.27:0/2561281378 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa1dc066f80 0x7fa1dc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:27:32.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.606+0000 7fa1febff700 1 -- 172.21.15.27:0/2561281378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fa1f804c3c0 con 0x7fa1f8157410 2024-01-11T09:27:32.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.610+0000 7fa1f4ff9700 1 -- 172.21.15.27:0/2561281378 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v66)=0 v66) v1 ==== 133+0+0 (secure 0 0 0) 0x7fa1e4056b20 con 0x7fa1f8157410 2024-01-11T09:27:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.612+0000 7fa1febff700 1 -- 172.21.15.27:0/2561281378 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa1dc066f80 msgr2=0x7fa1dc069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:27:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.612+0000 7fa1febff700 1 --2- 172.21.15.27:0/2561281378 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa1dc066f80 0x7fa1dc069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.612+0000 7fa1febff700 1 -- 172.21.15.27:0/2561281378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1f8157410 msgr2=0x7fa1f814f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.612+0000 7fa1febff700 1 --2- 172.21.15.27:0/2561281378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1f8157410 0x7fa1f814f620 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40090b0 tx=0x7fa1e4017000 comp rx=0 tx=0).stop 2024-01-11T09:27:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.613+0000 7fa1febff700 1 -- 172.21.15.27:0/2561281378 shutdown_connections 2024-01-11T09:27:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.613+0000 7fa1febff700 1 --2- 172.21.15.27:0/2561281378 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fa1dc066f80 0x7fa1dc069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.613+0000 7fa1febff700 1 --2- 172.21.15.27:0/2561281378 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1f81577e0 0x7fa1f80b0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.613+0000 7fa1febff700 1 --2- 172.21.15.27:0/2561281378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1f8069540 0x7fa1f814bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.613+0000 7fa1febff700 1 --2- 172.21.15.27:0/2561281378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1f8157410 0x7fa1f814f620 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.613+0000 7fa1febff700 1 -- 172.21.15.27:0/2561281378 >> 172.21.15.27:0/2561281378 conn(0x7fa1f80680a0 msgr2=0x7fa1f80c7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.613+0000 7fa1febff700 1 -- 172.21.15.27:0/2561281378 shutdown_connections 2024-01-11T09:27:32.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.613+0000 7fa1febff700 1 -- 172.21.15.27:0/2561281378 wait complete. 2024-01-11T09:27:32.629 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:27:32.629 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-01-11T09:27:32.685 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:27:32.685 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2024-01-11T09:27:32.736 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/test_restart 1m 2024-01-11T09:27:32.885 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:32.888+0000 7fd631a881c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:27:32.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.945+0000 7fd1913ff700 1 Processor -- start 2024-01-11T09:27:32.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.945+0000 7fd1913ff700 1 -- start start 2024-01-11T09:27:32.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.946+0000 7fd1913ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd18c069790 0x7fd18c0c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:32.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.947+0000 7fd1913ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd18c069b60 0x7fd18c1598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:32.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.947+0000 7fd1913ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd18c0cd0a0 con 0x7fd18c0c8720 2024-01-11T09:27:32.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.947+0000 7fd1913ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd18c0c7680 con 0x7fd18c069b60 2024-01-11T09:27:32.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.947+0000 7fd1913ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c0c7960 con 0x7fd18c069790 2024-01-11T09:27:32.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.947+0000 7fd18b7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd18c069b60 0x7fd18c1598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:32.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.947+0000 7fd18a7fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd18c0c8720 0x7fd18c1591a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53614/0 (socket says 172.21.15.27:53614) 2024-01-11T09:27:32.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.947+0000 7fd18a7fc700 1 -- 172.21.15.27:0/3312886235 learned_addr learned my addr 172.21.15.27:0/3312886235 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:32.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.947+0000 7fd18affd700 1 --2- 172.21.15.27:0/3312886235 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd18c069790 0x7fd18c0c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:32.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.947+0000 7fd189ffb700 1 -- 172.21.15.27:0/3312886235 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213001320 0 0) 0x7fd18c0cd0a0 con 0x7fd18c0c8720 2024-01-11T09:27:32.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.947+0000 7fd18b7fe700 1 -- 172.21.15.27:0/3312886235 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd18c069790 msgr2=0x7fd18c0c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:32.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.947+0000 7fd18b7fe700 1 --2- 172.21.15.27:0/3312886235 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd18c069790 0x7fd18c0c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.947+0000 7fd18b7fe700 1 -- 172.21.15.27:0/3312886235 >> v1:172.21.15.27:6789/0 conn(0x7fd18c0c8720 legacy=0x7fd18c1591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:32.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.947+0000 7fd18b7fe700 1 -- 172.21.15.27:0/3312886235 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18c0c7c40 con 0x7fd18c069b60 2024-01-11T09:27:32.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.948+0000 7fd18b7fe700 1 --2- 172.21.15.27:0/3312886235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd18c069b60 0x7fd18c1598b0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fd184009e60 tx=0x7fd184009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bafd3ee1a47c380 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:32.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.948+0000 7fd189ffb700 1 -- 172.21.15.27:0/3312886235 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18400e070 con 0x7fd18c069b60 2024-01-11T09:27:32.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.948+0000 7fd189ffb700 1 -- 172.21.15.27:0/3312886235 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd18400bbf0 con 0x7fd18c069b60 2024-01-11T09:27:32.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.948+0000 7fd189ffb700 1 -- 172.21.15.27:0/3312886235 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd184012430 con 0x7fd18c069b60 2024-01-11T09:27:32.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.949+0000 7fd1913ff700 1 -- 172.21.15.27:0/3312886235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd18c069b60 msgr2=0x7fd18c1598b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:32.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.949+0000 7fd1913ff700 1 --2- 172.21.15.27:0/3312886235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd18c069b60 0x7fd18c1598b0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fd184009e60 tx=0x7fd184009780 comp rx=0 tx=0).stop 2024-01-11T09:27:32.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.949+0000 7fd1913ff700 1 -- 172.21.15.27:0/3312886235 shutdown_connections 2024-01-11T09:27:32.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.949+0000 7fd1913ff700 1 --2- 172.21.15.27:0/3312886235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd18c069b60 0x7fd18c1598b0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.949+0000 7fd1913ff700 1 --2- 172.21.15.27:0/3312886235 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd18c069790 0x7fd18c0c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.949+0000 7fd1913ff700 1 -- 172.21.15.27:0/3312886235 >> 172.21.15.27:0/3312886235 conn(0x7fd18c0680a0 msgr2=0x7fd18c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:32.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.949+0000 7fd1913ff700 1 -- 172.21.15.27:0/3312886235 shutdown_connections 2024-01-11T09:27:32.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.949+0000 7fd1913ff700 1 -- 172.21.15.27:0/3312886235 wait complete. 2024-01-11T09:27:32.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.950+0000 7fd1913ff700 1 Processor -- start 2024-01-11T09:27:32.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.951+0000 7fd1913ff700 1 -- start start 2024-01-11T09:27:32.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.952+0000 7fd1913ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd18c069790 0x7fd18c15dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:32.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.952+0000 7fd1913ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd18c069b60 0x7fd18c15e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:32.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.952+0000 7fd1913ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd18c0c8720 0x7fd18c0c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:32.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.952+0000 7fd1913ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c0b1c20 con 0x7fd18c069790 2024-01-11T09:27:32.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.952+0000 7fd1913ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c15e310 con 0x7fd18c0c8720 2024-01-11T09:27:32.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.952+0000 7fd1913ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c0b0760 con 0x7fd18c069b60 2024-01-11T09:27:32.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.952+0000 7fd18affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd18c069790 0x7fd18c15dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:32.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.952+0000 7fd18affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd18c069790 0x7fd18c15dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37888/0 (socket says 172.21.15.27:37888) 2024-01-11T09:27:32.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.952+0000 7fd18affd700 1 -- 172.21.15.27:0/889468524 learned_addr learned my addr 172.21.15.27:0/889468524 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:32.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.952+0000 7fd18a7fc700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd18c069b60 0x7fd18c15e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:32.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.952+0000 7fd18b7fe700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd18c0c8720 0x7fd18c0c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:32.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.952+0000 7fd18affd700 1 -- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd18c069b60 msgr2=0x7fd18c15e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:32.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.952+0000 7fd18affd700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd18c069b60 0x7fd18c15e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.952+0000 7fd18affd700 1 -- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd18c0c8720 msgr2=0x7fd18c0c3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:32.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.952+0000 7fd18affd700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd18c0c8720 0x7fd18c0c3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:32.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.952+0000 7fd18affd700 1 -- 172.21.15.27:0/889468524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd184009a20 con 0x7fd18c069790 2024-01-11T09:27:32.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.953+0000 7fd18affd700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd18c069790 0x7fd18c15dc80 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fd17c00ea00 tx=0x7fd17c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:32.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.953+0000 7fd172ffd700 1 -- 172.21.15.27:0/889468524 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c00c6a0 con 0x7fd18c069790 2024-01-11T09:27:32.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.953+0000 7fd172ffd700 1 -- 172.21.15.27:0/889468524 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17c00c800 con 0x7fd18c069790 2024-01-11T09:27:32.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.953+0000 7fd172ffd700 1 -- 172.21.15.27:0/889468524 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c00cb80 con 0x7fd18c069790 2024-01-11T09:27:32.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.953+0000 7fd1913ff700 1 -- 172.21.15.27:0/889468524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18c0afe30 con 0x7fd18c069790 2024-01-11T09:27:32.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.953+0000 7fd1913ff700 1 -- 172.21.15.27:0/889468524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18c0b03f0 con 0x7fd18c069790 2024-01-11T09:27:32.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.954+0000 7fd1913ff700 1 -- 172.21.15.27:0/889468524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd18c04c3c0 con 0x7fd18c069790 2024-01-11T09:27:32.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.955+0000 7fd172ffd700 1 -- 172.21.15.27:0/889468524 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 100) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fd17c013470 con 0x7fd18c069790 2024-01-11T09:27:32.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.956+0000 7fd172ffd700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd178066f80 0x7fd178069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:32.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.957+0000 7fd172ffd700 1 -- 172.21.15.27:0/889468524 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fd17c084b00 con 0x7fd18c069790 2024-01-11T09:27:32.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.957+0000 7fd18a7fc700 1 -- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd178066f80 msgr2=0x7fd178069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:32.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.957+0000 7fd18a7fc700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd178066f80 0x7fd178069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:27:32.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:32.960+0000 7fd172ffd700 1 -- 172.21.15.27:0/889468524 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd17c051d30 con 0x7fd18c069790 2024-01-11T09:27:32.987 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:32.990+0000 7f31879691c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:27:33.055 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:33.058+0000 7fd631a881c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:27:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:33.158+0000 7fd18a7fc700 1 -- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd178066f80 msgr2=0x7fd178069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:33.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:33.158+0000 7fd18a7fc700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd178066f80 0x7fd178069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:27:33.168 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:33.171+0000 7f31879691c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:27:33.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:33.191+0000 7fd1913ff700 1 -- 172.21.15.27:0/889468524 --> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd18c06a110 con 0x7fd178066f80 2024-01-11T09:27:33.473 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:33.476+0000 7fd631a881c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:27:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:33.559+0000 7fd18a7fc700 1 -- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd178066f80 msgr2=0x7fd178069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:33.559+0000 7fd18a7fc700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd178066f80 0x7fd178069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-11T09:27:33.608 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:33.610+0000 7f31879691c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:27:34.249 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:34.252+0000 7fd631a881c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:27:34.338 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:34.341+0000 7fd631a881c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:27:34.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:34.360+0000 7fd18a7fc700 1 -- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd178066f80 msgr2=0x7fd178069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:34.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:34.360+0000 7fd18a7fc700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd178066f80 0x7fd178069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-11T09:27:34.436 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:34.438+0000 7f31879691c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:27:34.494 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:34.497+0000 7fd631a881c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:27:34.530 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:34.532+0000 7f31879691c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:27:34.581 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:34.584+0000 7fd631a881c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:27:34.694 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:34.697+0000 7f31879691c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:27:34.787 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:34.790+0000 7f31879691c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:27:34.799 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:34.802+0000 7fd631a881c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:27:34.896 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:34.899+0000 7fd631a881c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:27:34.992 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:34.995+0000 7fd631a881c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:27:35.015 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:35.018+0000 7f31879691c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:27:35.089 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:35.092+0000 7fd631a881c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:27:35.117 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:35.121+0000 7f31879691c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:27:35.218 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:35.221+0000 7f31879691c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:27:35.278 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:35.281+0000 7fd631a881c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:27:35.319 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:35.323+0000 7f31879691c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:27:35.378 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:35.381+0000 7fd631a881c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:27:35.520 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:35.523+0000 7f31879691c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:27:35.624 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:35.627+0000 7f31879691c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:27:35.723 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:35.726+0000 7fd631a881c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:27:35.814 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:35.816+0000 7fd631a881c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:27:35.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:35.963+0000 7fd18a7fc700 1 -- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd178066f80 msgr2=0x7fd178069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/41381 2024-01-11T09:27:35.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:35.963+0000 7fd18a7fc700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd178066f80 0x7fd178069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-11T09:27:35.988 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:35.992+0000 7f31879691c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:27:36.083 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:36.087+0000 7f31879691c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:27:36.356 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:36.359+0000 7fd631a881c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:27:36.544 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:36.546+0000 7fd631a881c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:27:36.660 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:36.664+0000 7f31879691c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:27:36.857 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:36.861+0000 7f31879691c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:27:36.922 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:36.925+0000 7fd631a881c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:27:37.259 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:37.261+0000 7f31879691c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:27:37.347 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:37.350+0000 7fd631a881c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:27:37.529 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:37.532+0000 7fd631a881c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:27:37.707 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:37.710+0000 7f31879691c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:27:37.718 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:37.721+0000 7fd631a881c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:27:37.830 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:37.833+0000 7fd631a881c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:27:37.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:37.843+0000 7fd172ffd700 1 -- 172.21.15.27:0/889468524 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 101) v1 ==== 84470+0+0 (secure 0 0 0) 0x7fd17c052100 con 0x7fd18c069790 2024-01-11T09:27:37.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:37.843+0000 7fd172ffd700 1 -- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd178066f80 msgr2=0x7fd178069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:27:37.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:37.843+0000 7fd172ffd700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd178066f80 0x7fd178069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:37.909 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:37.913+0000 7f31879691c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:27:37.965 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:37.968+0000 7fd5a7079700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:37.965 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:37.968+0000 7fd5a7079700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:37.966 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:37.968+0000 7fd5a7079700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:37.966 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:37.968+0000 7fd5a7079700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:37.966 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:37.968+0000 7fd5a7079700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:37.966 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:37.969+0000 7fd5aa07f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:37.966 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:37.969+0000 7fd5aa07f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:37.966 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:37.969+0000 7fd5aa07f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:37.966 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:37.969+0000 7fd5aa07f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:37.967 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:27:37.969+0000 7fd5aa07f700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:27:38.114 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:38.118+0000 7f31879691c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:27:38.232 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:27:38.236+0000 7f31879691c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:27:38.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:38.846+0000 7fd172ffd700 1 -- 172.21.15.27:0/889468524 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 102) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fd17c015c30 con 0x7fd18c069790 2024-01-11T09:27:38.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:38.847+0000 7fd172ffd700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd178066f80 0x7fd178069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:38.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:38.847+0000 7fd172ffd700 1 -- 172.21.15.27:0/889468524 --> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd18c06a110 con 0x7fd178066f80 2024-01-11T09:27:38.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:38.847+0000 7fd18a7fc700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd178066f80 0x7fd178069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:38.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:38.849+0000 7fd18a7fc700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd178066f80 0x7fd178069430 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd18c156e50 tx=0x7fd180007d30 comp rx=0 tx=0).ready entity=mgr.8153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:38.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:38.913+0000 7fd172ffd700 1 -- 172.21.15.27:0/889468524 <== mgr.8153 v2:172.21.15.77:6836/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fd18c06a110 con 0x7fd178066f80 2024-01-11T09:27:38.910 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path /snap_test_dir1/test_restart 2024-01-11T09:27:38.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:38.916+0000 7fd1913ff700 1 -- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd178066f80 msgr2=0x7fd178069430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:38.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:38.916+0000 7fd1913ff700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd178066f80 0x7fd178069430 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd18c156e50 tx=0x7fd180007d30 comp rx=0 tx=0).stop 2024-01-11T09:27:38.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:38.916+0000 7fd1913ff700 1 -- 172.21.15.27:0/889468524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd18c069790 msgr2=0x7fd18c15dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:38.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:38.916+0000 7fd1913ff700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd18c069790 0x7fd18c15dc80 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fd17c00ea00 tx=0x7fd17c009120 comp rx=0 tx=0).stop 2024-01-11T09:27:38.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:38.916+0000 7fd1913ff700 1 -- 172.21.15.27:0/889468524 shutdown_connections 2024-01-11T09:27:38.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:38.916+0000 7fd1913ff700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fd178066f80 0x7fd178069430 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:38.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:38.916+0000 7fd1913ff700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd18c0c8720 0x7fd18c0c3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:38.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:38.916+0000 7fd1913ff700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd18c069790 0x7fd18c15dc80 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:38.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:38.916+0000 7fd1913ff700 1 --2- 172.21.15.27:0/889468524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd18c069b60 0x7fd18c15e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:38.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:38.916+0000 7fd1913ff700 1 -- 172.21.15.27:0/889468524 >> 172.21.15.27:0/889468524 conn(0x7fd18c0680a0 msgr2=0x7fd18c0c24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:38.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:38.916+0000 7fd1913ff700 1 -- 172.21.15.27:0/889468524 shutdown_connections 2024-01-11T09:27:38.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:38.916+0000 7fd1913ff700 1 -- 172.21.15.27:0/889468524 wait complete. 2024-01-11T09:27:38.933 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2024-01-11T09:27:38.934 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/test_restart 1m 2024-01-11T09:27:39.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.142+0000 7f52ed7ff700 1 Processor -- start 2024-01-11T09:27:39.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.142+0000 7f52ed7ff700 1 -- start start 2024-01-11T09:27:39.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.143+0000 7f52ed7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52e81577e0 0x7f52e80ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:39.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.143+0000 7f52ed7ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f52e8069540 0x7f52e80ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:39.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.143+0000 7f52ed7ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e80699d0 con 0x7f52e8157410 2024-01-11T09:27:39.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.143+0000 7f52ed7ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52e8069cb0 con 0x7f52e8069540 2024-01-11T09:27:39.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.143+0000 7f52ed7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e8069f90 con 0x7f52e81577e0 2024-01-11T09:27:39.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.143+0000 7f52e77fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f52e8069540 0x7f52e80ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:39.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.144+0000 7f52e67fc700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52e81577e0 0x7f52e80ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:39.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.144+0000 7f52e6ffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f52e8157410 0x7f52e80ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:53262/0 (socket says 172.21.15.27:53262) 2024-01-11T09:27:39.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.144+0000 7f52e6ffd700 1 -- 172.21.15.27:0/2479658122 learned_addr learned my addr 172.21.15.27:0/2479658122 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:27:39.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.144+0000 7f52e5ffb700 1 -- 172.21.15.27:0/2479658122 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840497441 0 0) 0x7f52e80699d0 con 0x7f52e8157410 2024-01-11T09:27:39.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.144+0000 7f52e77fe700 1 -- 172.21.15.27:0/2479658122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52e81577e0 msgr2=0x7f52e80ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:27:39.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.144+0000 7f52e77fe700 1 --2- 172.21.15.27:0/2479658122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52e81577e0 0x7f52e80ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:39.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.144+0000 7f52e77fe700 1 -- 172.21.15.27:0/2479658122 >> v1:172.21.15.77:6789/0 conn(0x7f52e8157410 legacy=0x7f52e80ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:39.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.144+0000 7f52e77fe700 1 -- 172.21.15.27:0/2479658122 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e806a270 con 0x7f52e8069540 2024-01-11T09:27:39.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.144+0000 7f52e77fe700 1 --2- 172.21.15.27:0/2479658122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e8069540 0x7f52e80ae540 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f52e0009e60 tx=0x7f52e0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=436625cec0e0e4ec server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:39.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.145+0000 7f52e5ffb700 1 -- 172.21.15.27:0/2479658122 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e000e070 con 0x7f52e8069540 2024-01-11T09:27:39.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.145+0000 7f52e5ffb700 1 -- 172.21.15.27:0/2479658122 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e000bb60 con 0x7f52e8069540 2024-01-11T09:27:39.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.145+0000 7f52e5ffb700 1 -- 172.21.15.27:0/2479658122 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e0012430 con 0x7f52e8069540 2024-01-11T09:27:39.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.145+0000 7f52ed7ff700 1 -- 172.21.15.27:0/2479658122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e8069540 msgr2=0x7f52e80ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:39.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.145+0000 7f52ed7ff700 1 --2- 172.21.15.27:0/2479658122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e8069540 0x7f52e80ae540 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f52e0009e60 tx=0x7f52e0009780 comp rx=0 tx=0).stop 2024-01-11T09:27:39.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.145+0000 7f52ed7ff700 1 -- 172.21.15.27:0/2479658122 shutdown_connections 2024-01-11T09:27:39.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.145+0000 7f52ed7ff700 1 --2- 172.21.15.27:0/2479658122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e8069540 0x7f52e80ae540 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:39.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.145+0000 7f52ed7ff700 1 --2- 172.21.15.27:0/2479658122 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52e81577e0 0x7f52e80ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:39.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.145+0000 7f52ed7ff700 1 -- 172.21.15.27:0/2479658122 >> 172.21.15.27:0/2479658122 conn(0x7f52e80680a0 msgr2=0x7f52e80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:39.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.145+0000 7f52ed7ff700 1 -- 172.21.15.27:0/2479658122 shutdown_connections 2024-01-11T09:27:39.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.145+0000 7f52ed7ff700 1 -- 172.21.15.27:0/2479658122 wait complete. 2024-01-11T09:27:39.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.147+0000 7f52ed7ff700 1 Processor -- start 2024-01-11T09:27:39.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.147+0000 7f52ed7ff700 1 -- start start 2024-01-11T09:27:39.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.148+0000 7f52ed7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52e8069540 0x7f52e80b1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:39.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.148+0000 7f52ed7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52e8157410 0x7f52e815da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:39.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.148+0000 7f52ed7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e81577e0 0x7f52e816ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:39.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.148+0000 7f52ed7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e80b1540 con 0x7f52e81577e0 2024-01-11T09:27:39.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.148+0000 7f52ed7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e80c3ae0 con 0x7f52e8069540 2024-01-11T09:27:39.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.148+0000 7f52ed7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e816b320 con 0x7f52e8157410 2024-01-11T09:27:39.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.148+0000 7f52e77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e81577e0 0x7f52e816ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:39.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.148+0000 7f52e77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e81577e0 0x7f52e816ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43100/0 (socket says 172.21.15.27:43100) 2024-01-11T09:27:39.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.148+0000 7f52e77fe700 1 -- 172.21.15.27:0/4070947106 learned_addr learned my addr 172.21.15.27:0/4070947106 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:27:39.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.149+0000 7f52e6ffd700 1 --2- 172.21.15.27:0/4070947106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52e8069540 0x7f52e80b1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:39.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.149+0000 7f52e67fc700 1 --2- 172.21.15.27:0/4070947106 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52e8157410 0x7f52e815da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:39.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.149+0000 7f52e77fe700 1 -- 172.21.15.27:0/4070947106 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52e8157410 msgr2=0x7f52e815da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:39.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.149+0000 7f52e77fe700 1 --2- 172.21.15.27:0/4070947106 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52e8157410 0x7f52e815da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:39.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.149+0000 7f52e77fe700 1 -- 172.21.15.27:0/4070947106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52e8069540 msgr2=0x7f52e80b1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:39.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.149+0000 7f52e77fe700 1 --2- 172.21.15.27:0/4070947106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52e8069540 0x7f52e80b1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:39.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.149+0000 7f52e77fe700 1 -- 172.21.15.27:0/4070947106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e0009a20 con 0x7f52e81577e0 2024-01-11T09:27:39.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.149+0000 7f52e77fe700 1 --2- 172.21.15.27:0/4070947106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e81577e0 0x7f52e816ade0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f52e000b6c0 tx=0x7f52e0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:39.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.149+0000 7f52ceffd700 1 -- 172.21.15.27:0/4070947106 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e0012e30 con 0x7f52e81577e0 2024-01-11T09:27:39.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.149+0000 7f52ceffd700 1 -- 172.21.15.27:0/4070947106 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e001d630 con 0x7f52e81577e0 2024-01-11T09:27:39.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.149+0000 7f52ed7ff700 1 -- 172.21.15.27:0/4070947106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e80b08c0 con 0x7f52e81577e0 2024-01-11T09:27:39.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.149+0000 7f52ceffd700 1 -- 172.21.15.27:0/4070947106 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e001d9f0 con 0x7f52e81577e0 2024-01-11T09:27:39.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.150+0000 7f52ed7ff700 1 -- 172.21.15.27:0/4070947106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e80b0e80 con 0x7f52e81577e0 2024-01-11T09:27:39.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.151+0000 7f52ed7ff700 1 -- 172.21.15.27:0/4070947106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e80b1680 con 0x7f52e81577e0 2024-01-11T09:27:39.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.152+0000 7f52ceffd700 1 -- 172.21.15.27:0/4070947106 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 102) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f52e0024070 con 0x7f52e81577e0 2024-01-11T09:27:39.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.152+0000 7f52ceffd700 1 --2- 172.21.15.27:0/4070947106 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f52d4066dd0 0x7f52d4069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:27:39.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.153+0000 7f52ceffd700 1 -- 172.21.15.27:0/4070947106 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f52e0086d00 con 0x7f52e81577e0 2024-01-11T09:27:39.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.153+0000 7f52e6ffd700 1 --2- 172.21.15.27:0/4070947106 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f52d4066dd0 0x7f52d4069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:27:39.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.154+0000 7f52e6ffd700 1 --2- 172.21.15.27:0/4070947106 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f52d4066dd0 0x7f52d4069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f52e8156a00 tx=0x7f52d8009040 comp rx=0 tx=0).ready entity=mgr.8153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:27:39.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.157+0000 7f52ceffd700 1 -- 172.21.15.27:0/4070947106 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f52e80b1680 con 0x7f52e81577e0 2024-01-11T09:27:39.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.390+0000 7f52ed7ff700 1 -- 172.21.15.27:0/4070947106 --> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f52e80b1680 con 0x7f52d4066dd0 2024-01-11T09:27:39.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.437+0000 7f52ceffd700 1 -- 172.21.15.27:0/4070947106 <== mgr.8153 v2:172.21.15.77:6836/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7f52e80b1680 con 0x7f52d4066dd0 2024-01-11T09:27:39.434 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule activated for path /snap_test_dir1/test_restart 2024-01-11T09:27:39.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.439+0000 7f52ed7ff700 1 -- 172.21.15.27:0/4070947106 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f52d4066dd0 msgr2=0x7f52d4069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:39.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.439+0000 7f52ed7ff700 1 --2- 172.21.15.27:0/4070947106 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f52d4066dd0 0x7f52d4069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f52e8156a00 tx=0x7f52d8009040 comp rx=0 tx=0).stop 2024-01-11T09:27:39.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.439+0000 7f52ed7ff700 1 -- 172.21.15.27:0/4070947106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e81577e0 msgr2=0x7f52e816ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:27:39.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.439+0000 7f52ed7ff700 1 --2- 172.21.15.27:0/4070947106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e81577e0 0x7f52e816ade0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f52e000b6c0 tx=0x7f52e0019000 comp rx=0 tx=0).stop 2024-01-11T09:27:39.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.439+0000 7f52ed7ff700 1 -- 172.21.15.27:0/4070947106 shutdown_connections 2024-01-11T09:27:39.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.439+0000 7f52ed7ff700 1 --2- 172.21.15.27:0/4070947106 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f52d4066dd0 0x7f52d4069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:39.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.440+0000 7f52ed7ff700 1 --2- 172.21.15.27:0/4070947106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e81577e0 0x7f52e816ade0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:39.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.440+0000 7f52ed7ff700 1 --2- 172.21.15.27:0/4070947106 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52e8069540 0x7f52e80b1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:39.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.440+0000 7f52ed7ff700 1 --2- 172.21.15.27:0/4070947106 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f52e8157410 0x7f52e815da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:27:39.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.440+0000 7f52ed7ff700 1 -- 172.21.15.27:0/4070947106 >> 172.21.15.27:0/4070947106 conn(0x7f52e80680a0 msgr2=0x7f52e80c29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:27:39.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.440+0000 7f52ed7ff700 1 -- 172.21.15.27:0/4070947106 shutdown_connections 2024-01-11T09:27:39.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:27:39.440+0000 7f52ed7ff700 1 -- 172.21.15.27:0/4070947106 wait complete. 2024-01-11T09:27:39.457 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2024-01-11T09:27:42.025 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:27:42.030 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:28:00.445 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:28:00.449+0000 7fd59901d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:28:00.446 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:28:00.449+0000 7fd59901d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:28:00.446 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:28:00.449+0000 7fd59901d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:28:00.446 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:28:00.449+0000 7fd59901d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:28:00.446 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:28:00.449+0000 7fd59901d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:28:12.188 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:28:12.194 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:28:42.395 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:28:42.401 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:29:12.585 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:29:12.591 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:29:42.765 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:29:42.772 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:30:12.947 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:30:12.953 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:30:43.069 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:30:43.075 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:31:13.259 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:31:13.265 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:31:43.432 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:31:43.438 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:32:13.604 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:32:13.610 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:32:43.794 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:32:43.800 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:33:13.999 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:33:14.005 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:33:44.180 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:33:44.186 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:34:14.369 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:34:14.375 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:34:44.541 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:34:44.547 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:35:14.740 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:35:14.748 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:35:44.861 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:35:44.867 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:36:15.177 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:36:15.182 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:36:45.357 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:36:45.363 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:37:15.530 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:37:15.535 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:37:45.694 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:37:45.700 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:38:15.891 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:38:15.897 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:38:16.458 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:38:16.458 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-01-11T09:38:16.514 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_28_00_UTC 2024-01-11T09:38:16.514 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_29_00_UTC 2024-01-11T09:38:16.514 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_30_00_UTC 2024-01-11T09:38:16.514 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_31_00_UTC 2024-01-11T09:38:16.514 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_32_00_UTC 2024-01-11T09:38:16.514 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_33_00_UTC 2024-01-11T09:38:16.514 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_34_00_UTC 2024-01-11T09:38:16.515 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_35_00_UTC 2024-01-11T09:38:16.515 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_36_00_UTC 2024-01-11T09:38:16.515 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_37_00_UTC 2024-01-11T09:38:16.515 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_38_00_UTC 2024-01-11T09:38:16.516 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-11-09_28_00_UTC', 'scheduled-2024-01-11-09_29_00_UTC', 'scheduled-2024-01-11-09_30_00_UTC', 'scheduled-2024-01-11-09_31_00_UTC', 'scheduled-2024-01-11-09_32_00_UTC', 'scheduled-2024-01-11-09_33_00_UTC', 'scheduled-2024-01-11-09_34_00_UTC', 'scheduled-2024-01-11-09_35_00_UTC', 'scheduled-2024-01-11-09_36_00_UTC', 'scheduled-2024-01-11-09_37_00_UTC', 'scheduled-2024-01-11-09_38_00_UTC'] 2024-01-11T09:38:16.516 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2024-01-11T09:38:16.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.730+0000 7f2d7f1ff700 1 Processor -- start 2024-01-11T09:38:16.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.730+0000 7f2d7f1ff700 1 -- start start 2024-01-11T09:38:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.731+0000 7f2d7f1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2d78143190 0x7f2d780cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:38:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.731+0000 7f2d7f1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d780cb4b0 0x7f2d780cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:38:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.731+0000 7f2d7f1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d7815d320 con 0x7f2d78142dc0 2024-01-11T09:38:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.731+0000 7f2d7f1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d7815df30 con 0x7f2d78143190 2024-01-11T09:38:16.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.731+0000 7f2d7f1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7815d590 con 0x7f2d780cb4b0 2024-01-11T09:38:16.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.732+0000 7f2d7d79c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d780cb4b0 0x7f2d780cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:38:16.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.732+0000 7f2d77fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2d78143190 0x7f2d780cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:38:16.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.732+0000 7f2d77fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2d78143190 0x7f2d780cc020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:49278/0 (socket says 172.21.15.27:49278) 2024-01-11T09:38:16.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.732+0000 7f2d7d79c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d780cb4b0 0x7f2d780cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:53834/0 (socket says 172.21.15.27:53834) 2024-01-11T09:38:16.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.732+0000 7f2d77fff700 1 -- 172.21.15.27:0/4074911028 learned_addr learned my addr 172.21.15.27:0/4074911028 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:38:16.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.732+0000 7f2d7cf9b700 1 --1- 172.21.15.27:0/4074911028 >> v1:172.21.15.77:6789/0 conn(0x7f2d78142dc0 0x7f2d780cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:45814/0 (socket says 172.21.15.27:45814) 2024-01-11T09:38:16.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.732+0000 7f2d7d79c700 1 -- 172.21.15.27:0/4074911028 >> v2:172.21.15.77:3300/0 conn(0x7f2d78143190 msgr2=0x7f2d780cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:38:16.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.732+0000 7f2d7d79c700 1 --2- 172.21.15.27:0/4074911028 >> v2:172.21.15.77:3300/0 conn(0x7f2d78143190 0x7f2d780cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:38:16.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.732+0000 7f2d7d79c700 1 -- 172.21.15.27:0/4074911028 >> v1:172.21.15.77:6789/0 conn(0x7f2d78142dc0 legacy=0x7f2d780cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:38:16.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.732+0000 7f2d7d79c700 1 -- 172.21.15.27:0/4074911028 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7815d870 con 0x7f2d780cb4b0 2024-01-11T09:38:16.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.733+0000 7f2d7d79c700 1 --2- 172.21.15.27:0/4074911028 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d780cb4b0 0x7f2d780cc560 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2d70009e60 tx=0x7f2d70009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=904b3abeb4d0d61e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:38:16.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.733+0000 7f2d777fe700 1 -- 172.21.15.27:0/4074911028 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7000e070 con 0x7f2d780cb4b0 2024-01-11T09:38:16.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.733+0000 7f2d777fe700 1 -- 172.21.15.27:0/4074911028 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7000bb60 con 0x7f2d780cb4b0 2024-01-11T09:38:16.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.733+0000 7f2d777fe700 1 -- 172.21.15.27:0/4074911028 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d70012430 con 0x7f2d780cb4b0 2024-01-11T09:38:16.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.734+0000 7f2d7f1ff700 1 -- 172.21.15.27:0/4074911028 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d780cb4b0 msgr2=0x7f2d780cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:38:16.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.734+0000 7f2d7f1ff700 1 --2- 172.21.15.27:0/4074911028 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d780cb4b0 0x7f2d780cc560 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2d70009e60 tx=0x7f2d70009780 comp rx=0 tx=0).stop 2024-01-11T09:38:16.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.734+0000 7f2d7f1ff700 1 -- 172.21.15.27:0/4074911028 shutdown_connections 2024-01-11T09:38:16.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.734+0000 7f2d7f1ff700 1 --2- 172.21.15.27:0/4074911028 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d780cb4b0 0x7f2d780cc560 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:38:16.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.734+0000 7f2d7f1ff700 1 --2- 172.21.15.27:0/4074911028 >> v2:172.21.15.77:3300/0 conn(0x7f2d78143190 0x7f2d780cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:38:16.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.734+0000 7f2d7f1ff700 1 -- 172.21.15.27:0/4074911028 >> 172.21.15.27:0/4074911028 conn(0x7f2d78063ca0 msgr2=0x7f2d7806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:38:16.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.734+0000 7f2d7f1ff700 1 -- 172.21.15.27:0/4074911028 shutdown_connections 2024-01-11T09:38:16.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.734+0000 7f2d7f1ff700 1 -- 172.21.15.27:0/4074911028 wait complete. 2024-01-11T09:38:16.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.735+0000 7f2d7f1ff700 1 Processor -- start 2024-01-11T09:38:16.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.736+0000 7f2d7f1ff700 1 -- start start 2024-01-11T09:38:16.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.737+0000 7f2d7f1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d780cb4b0 0x7f2d7815b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:38:16.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.737+0000 7f2d7f1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d78142dc0 0x7f2d7813e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:38:16.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.737+0000 7f2d7f1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d78143190 0x7f2d780b3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:38:16.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.737+0000 7f2d7f1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7815bb20 con 0x7f2d78142dc0 2024-01-11T09:38:16.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.737+0000 7f2d7f1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7815be00 con 0x7f2d78143190 2024-01-11T09:38:16.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.737+0000 7f2d7f1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7816d520 con 0x7f2d780cb4b0 2024-01-11T09:38:16.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.737+0000 7f2d77fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d78142dc0 0x7f2d7813e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:38:16.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.737+0000 7f2d77fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d78142dc0 0x7f2d7813e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58328/0 (socket says 172.21.15.27:58328) 2024-01-11T09:38:16.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.737+0000 7f2d77fff700 1 -- 172.21.15.27:0/2054870395 learned_addr learned my addr 172.21.15.27:0/2054870395 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:38:16.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.737+0000 7f2d7cf9b700 1 --2- 172.21.15.27:0/2054870395 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d780cb4b0 0x7f2d7815b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:38:16.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.737+0000 7f2d7d79c700 1 --2- 172.21.15.27:0/2054870395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d78143190 0x7f2d780b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:38:16.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.737+0000 7f2d77fff700 1 -- 172.21.15.27:0/2054870395 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d780cb4b0 msgr2=0x7f2d7815b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:38:16.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.737+0000 7f2d77fff700 1 --2- 172.21.15.27:0/2054870395 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d780cb4b0 0x7f2d7815b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:38:16.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.738+0000 7f2d77fff700 1 -- 172.21.15.27:0/2054870395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d78143190 msgr2=0x7f2d780b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:38:16.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.738+0000 7f2d77fff700 1 --2- 172.21.15.27:0/2054870395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d78143190 0x7f2d780b3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:38:16.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.738+0000 7f2d77fff700 1 -- 172.21.15.27:0/2054870395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d70009a20 con 0x7f2d78142dc0 2024-01-11T09:38:16.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.738+0000 7f2d7d79c700 1 --2- 172.21.15.27:0/2054870395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d78143190 0x7f2d780b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:38:16.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.738+0000 7f2d77fff700 1 --2- 172.21.15.27:0/2054870395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d78142dc0 0x7f2d7813e760 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400e980 tx=0x7f2d64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:38:16.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.738+0000 7f2d74ff9700 1 -- 172.21.15.27:0/2054870395 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6400ee70 con 0x7f2d78142dc0 2024-01-11T09:38:16.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.738+0000 7f2d74ff9700 1 -- 172.21.15.27:0/2054870395 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6400c850 con 0x7f2d78142dc0 2024-01-11T09:38:16.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.738+0000 7f2d74ff9700 1 -- 172.21.15.27:0/2054870395 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6400cbd0 con 0x7f2d78142dc0 2024-01-11T09:38:16.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.738+0000 7f2d7f1ff700 1 -- 172.21.15.27:0/2054870395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7816d800 con 0x7f2d78142dc0 2024-01-11T09:38:16.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.738+0000 7f2d7f1ff700 1 -- 172.21.15.27:0/2054870395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7815b6d0 con 0x7f2d78142dc0 2024-01-11T09:38:16.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.739+0000 7f2d7f1ff700 1 -- 172.21.15.27:0/2054870395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7816de50 con 0x7f2d78142dc0 2024-01-11T09:38:16.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.740+0000 7f2d74ff9700 1 -- 172.21.15.27:0/2054870395 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 106) v1 ==== 84873+0+0 (secure 0 0 0) 0x7f2d64013470 con 0x7f2d78142dc0 2024-01-11T09:38:16.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.742+0000 7f2d74ff9700 1 --2- 172.21.15.27:0/2054870395 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2d5c0670a0 0x7f2d5c069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:38:16.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.742+0000 7f2d74ff9700 1 -- 172.21.15.27:0/2054870395 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6019+0+0 (secure 0 0 0) 0x7f2d64084a90 con 0x7f2d78142dc0 2024-01-11T09:38:16.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.742+0000 7f2d7cf9b700 1 --2- 172.21.15.27:0/2054870395 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2d5c0670a0 0x7f2d5c069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:38:16.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.743+0000 7f2d7cf9b700 1 --2- 172.21.15.27:0/2054870395 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2d5c0670a0 0x7f2d5c069550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c004fd0 tx=0x7f2d6c007eb0 comp rx=0 tx=0).ready entity=mgr.8153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:38:16.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.747+0000 7f2d74ff9700 1 -- 172.21.15.27:0/2054870395 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2d64051a10 con 0x7f2d78142dc0 2024-01-11T09:38:16.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.978+0000 7f2d7f1ff700 1 -- 172.21.15.27:0/2054870395 --> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2d7804c3c0 con 0x7f2d5c0670a0 2024-01-11T09:38:16.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.988+0000 7f2d74ff9700 1 -- 172.21.15.27:0/2054870395 <== mgr.8153 v2:172.21.15.77:6836/41381 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f2d7804c3c0 con 0x7f2d5c0670a0 2024-01-11T09:38:16.981 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:38:16.981 INFO:teuthology.orchestra.run.smithi027.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T09:27:38", "first": "2024-01-11T09:28:00", "last": "2024-01-11T09:38:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2024-01-11T09:38:16.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.990+0000 7f2d7f1ff700 1 -- 172.21.15.27:0/2054870395 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2d5c0670a0 msgr2=0x7f2d5c069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:38:16.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.990+0000 7f2d7f1ff700 1 --2- 172.21.15.27:0/2054870395 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2d5c0670a0 0x7f2d5c069550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c004fd0 tx=0x7f2d6c007eb0 comp rx=0 tx=0).stop 2024-01-11T09:38:16.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.990+0000 7f2d7f1ff700 1 -- 172.21.15.27:0/2054870395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d78142dc0 msgr2=0x7f2d7813e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:38:16.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.990+0000 7f2d7f1ff700 1 --2- 172.21.15.27:0/2054870395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d78142dc0 0x7f2d7813e760 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400e980 tx=0x7f2d64009250 comp rx=0 tx=0).stop 2024-01-11T09:38:16.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.990+0000 7f2d7f1ff700 1 -- 172.21.15.27:0/2054870395 shutdown_connections 2024-01-11T09:38:16.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.990+0000 7f2d7f1ff700 1 --2- 172.21.15.27:0/2054870395 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7f2d5c0670a0 0x7f2d5c069550 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:38:16.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.990+0000 7f2d7f1ff700 1 --2- 172.21.15.27:0/2054870395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d78143190 0x7f2d780b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:38:16.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.990+0000 7f2d7f1ff700 1 --2- 172.21.15.27:0/2054870395 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d780cb4b0 0x7f2d7815b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:38:16.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.990+0000 7f2d7f1ff700 1 --2- 172.21.15.27:0/2054870395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d78142dc0 0x7f2d7813e760 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:38:16.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.990+0000 7f2d7f1ff700 1 -- 172.21.15.27:0/2054870395 >> 172.21.15.27:0/2054870395 conn(0x7f2d78063ca0 msgr2=0x7f2d78141df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:38:16.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.990+0000 7f2d7f1ff700 1 -- 172.21.15.27:0/2054870395 shutdown_connections 2024-01-11T09:38:16.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:16.990+0000 7f2d7f1ff700 1 -- 172.21.15.27:0/2054870395 wait complete. 2024-01-11T09:38:17.001 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T09:27:38", "first": "2024-01-11T09:28:00", "last": "2024-01-11T09:38:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2024-01-11T09:38:17.002 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2024-01-11T00:00:00', 'created': '2024-01-11T09:27:38', 'first': '2024-01-11T09:28:00', 'last': '2024-01-11T09:38:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2024-01-11T09:38:17.002 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-01-11T09:38:17.002 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-01-11T09:38:17.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-11T09:38:17.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.209+0000 7fb3d1bff700 1 Processor -- start 2024-01-11T09:38:17.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.209+0000 7fb3d1bff700 1 -- start start 2024-01-11T09:38:17.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.210+0000 7fb3d1bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3cc0b2b90 0x7fb3cc159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:38:17.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.210+0000 7fb3d1bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb3cc0b07b0 0x7fb3cc159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:38:17.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.210+0000 7fb3d1bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3cc0cd730 con 0x7fb3cc0b03e0 2024-01-11T09:38:17.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.210+0000 7fb3d1bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3cc0ccb20 con 0x7fb3cc0b07b0 2024-01-11T09:38:17.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.210+0000 7fb3d1bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc0cce00 con 0x7fb3cc0b2b90 2024-01-11T09:38:17.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.210+0000 7fb3cbfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb3cc0b07b0 0x7fb3cc159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:38:17.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.210+0000 7fb3cbfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb3cc0b07b0 0x7fb3cc159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58344/0 (socket says 172.21.15.27:58344) 2024-01-11T09:38:17.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.210+0000 7fb3cbfff700 1 -- 172.21.15.27:0/3009434555 learned_addr learned my addr 172.21.15.27:0/3009434555 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:38:17.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.211+0000 7fb3caffd700 1 --2- 172.21.15.27:0/3009434555 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3cc0b2b90 0x7fb3cc159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:38:17.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.211+0000 7fb3cbfff700 1 -- 172.21.15.27:0/3009434555 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3cc0b2b90 msgr2=0x7fb3cc159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:38:17.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.211+0000 7fb3cbfff700 1 --2- 172.21.15.27:0/3009434555 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3cc0b2b90 0x7fb3cc159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:38:17.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.211+0000 7fb3cbfff700 1 -- 172.21.15.27:0/3009434555 >> v1:172.21.15.77:6789/0 conn(0x7fb3cc0b03e0 legacy=0x7fb3cc0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:38:17.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.211+0000 7fb3cbfff700 1 -- 172.21.15.27:0/3009434555 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3cc0cd0e0 con 0x7fb3cc0b07b0 2024-01-11T09:38:17.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.211+0000 7fb3cbfff700 1 --2- 172.21.15.27:0/3009434555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3cc0b07b0 0x7fb3cc159740 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009f20 tx=0x7fb3c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acb6f7ed0dc4afdc server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:38:17.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.211+0000 7fb3ca7fc700 1 -- 172.21.15.27:0/3009434555 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c400e070 con 0x7fb3cc0b07b0 2024-01-11T09:38:17.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.211+0000 7fb3ca7fc700 1 -- 172.21.15.27:0/3009434555 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c400ba80 con 0x7fb3cc0b07b0 2024-01-11T09:38:17.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.211+0000 7fb3ca7fc700 1 -- 172.21.15.27:0/3009434555 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c40092e0 con 0x7fb3cc0b07b0 2024-01-11T09:38:17.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.212+0000 7fb3d1bff700 1 -- 172.21.15.27:0/3009434555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3cc0b07b0 msgr2=0x7fb3cc159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:38:17.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.212+0000 7fb3d1bff700 1 --2- 172.21.15.27:0/3009434555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3cc0b07b0 0x7fb3cc159740 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009f20 tx=0x7fb3c4009840 comp rx=0 tx=0).stop 2024-01-11T09:38:17.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.212+0000 7fb3d1bff700 1 -- 172.21.15.27:0/3009434555 shutdown_connections 2024-01-11T09:38:17.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.212+0000 7fb3d1bff700 1 --2- 172.21.15.27:0/3009434555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3cc0b07b0 0x7fb3cc159740 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:38:17.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.212+0000 7fb3d1bff700 1 --2- 172.21.15.27:0/3009434555 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3cc0b2b90 0x7fb3cc159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:38:17.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.212+0000 7fb3d1bff700 1 -- 172.21.15.27:0/3009434555 >> 172.21.15.27:0/3009434555 conn(0x7fb3cc06ed40 msgr2=0x7fb3cc06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:38:17.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.212+0000 7fb3d1bff700 1 -- 172.21.15.27:0/3009434555 shutdown_connections 2024-01-11T09:38:17.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.212+0000 7fb3d1bff700 1 -- 172.21.15.27:0/3009434555 wait complete. 2024-01-11T09:38:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.214+0000 7fb3d1bff700 1 Processor -- start 2024-01-11T09:38:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.214+0000 7fb3d1bff700 1 -- start start 2024-01-11T09:38:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.215+0000 7fb3d1bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3cc0b03e0 0x7fb3cc0c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:38:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.215+0000 7fb3d1bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3cc0b07b0 0x7fb3cc0c3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:38:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.215+0000 7fb3d1bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3cc0b2b90 0x7fb3cc0af6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:38:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.215+0000 7fb3d1bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc0c43e0 con 0x7fb3cc0b07b0 2024-01-11T09:38:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.215+0000 7fb3d1bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc0c4660 con 0x7fb3cc0b2b90 2024-01-11T09:38:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.215+0000 7fb3d1bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc0c4940 con 0x7fb3cc0b03e0 2024-01-11T09:38:17.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.215+0000 7fb3caffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3cc0b07b0 0x7fb3cc0c3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:38:17.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.215+0000 7fb3caffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3cc0b07b0 0x7fb3cc0c3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58356/0 (socket says 172.21.15.27:58356) 2024-01-11T09:38:17.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.215+0000 7fb3caffd700 1 -- 172.21.15.27:0/2623283596 learned_addr learned my addr 172.21.15.27:0/2623283596 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:38:17.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.215+0000 7fb3cbfff700 1 --2- 172.21.15.27:0/2623283596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3cc0b2b90 0x7fb3cc0af6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:38:17.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.215+0000 7fb3cb7fe700 1 --2- 172.21.15.27:0/2623283596 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3cc0b03e0 0x7fb3cc0c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:38:17.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.215+0000 7fb3caffd700 1 -- 172.21.15.27:0/2623283596 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3cc0b03e0 msgr2=0x7fb3cc0c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:38:17.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.215+0000 7fb3caffd700 1 --2- 172.21.15.27:0/2623283596 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3cc0b03e0 0x7fb3cc0c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:38:17.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.215+0000 7fb3caffd700 1 -- 172.21.15.27:0/2623283596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3cc0b2b90 msgr2=0x7fb3cc0af6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:38:17.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.215+0000 7fb3caffd700 1 --2- 172.21.15.27:0/2623283596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3cc0b2b90 0x7fb3cc0af6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:38:17.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.215+0000 7fb3caffd700 1 -- 172.21.15.27:0/2623283596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c4009ae0 con 0x7fb3cc0b07b0 2024-01-11T09:38:17.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.215+0000 7fb3cb7fe700 1 --2- 172.21.15.27:0/2623283596 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3cc0b03e0 0x7fb3cc0c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:38:17.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.216+0000 7fb3caffd700 1 --2- 172.21.15.27:0/2623283596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3cc0b07b0 0x7fb3cc0c3ea0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fb3c000d860 tx=0x7fb3c0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:38:17.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.216+0000 7fb3b37fe700 1 -- 172.21.15.27:0/2623283596 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0009610 con 0x7fb3cc0b07b0 2024-01-11T09:38:17.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.216+0000 7fb3b37fe700 1 -- 172.21.15.27:0/2623283596 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c0009770 con 0x7fb3cc0b07b0 2024-01-11T09:38:17.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.216+0000 7fb3b37fe700 1 -- 172.21.15.27:0/2623283596 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0009b30 con 0x7fb3cc0b07b0 2024-01-11T09:38:17.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.216+0000 7fb3d1bff700 1 -- 172.21.15.27:0/2623283596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3cc0c4c20 con 0x7fb3cc0b07b0 2024-01-11T09:38:17.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.216+0000 7fb3d1bff700 1 -- 172.21.15.27:0/2623283596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3cc0c51e0 con 0x7fb3cc0b07b0 2024-01-11T09:38:17.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.217+0000 7fb3d1bff700 1 -- 172.21.15.27:0/2623283596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3cc0718d0 con 0x7fb3cc0b07b0 2024-01-11T09:38:17.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.218+0000 7fb3b37fe700 1 -- 172.21.15.27:0/2623283596 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 106) v1 ==== 84873+0+0 (secure 0 0 0) 0x7fb3c001e460 con 0x7fb3cc0b07b0 2024-01-11T09:38:17.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.219+0000 7fb3b37fe700 1 --2- 172.21.15.27:0/2623283596 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb3b80670a0 0x7fb3b8069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:38:17.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.219+0000 7fb3b37fe700 1 -- 172.21.15.27:0/2623283596 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6019+0+0 (secure 0 0 0) 0x7fb3c0084f60 con 0x7fb3cc0b07b0 2024-01-11T09:38:17.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.220+0000 7fb3cb7fe700 1 --2- 172.21.15.27:0/2623283596 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb3b80670a0 0x7fb3b8069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:38:17.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.220+0000 7fb3cb7fe700 1 --2- 172.21.15.27:0/2623283596 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb3b80670a0 0x7fb3b8069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc001840 tx=0x7fb3bc009040 comp rx=0 tx=0).ready entity=mgr.8153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:38:17.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.223+0000 7fb3b37fe700 1 -- 172.21.15.27:0/2623283596 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb3c0051ee0 con 0x7fb3cc0b07b0 2024-01-11T09:38:17.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.484+0000 7fb3d1bff700 1 -- 172.21.15.27:0/2623283596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fb3cc0cd3c0 con 0x7fb3cc0b07b0 2024-01-11T09:38:17.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.487+0000 7fb3b37fe700 1 -- 172.21.15.27:0/2623283596 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v106) v1 ==== 74+0+163909 (secure 0 0 0) 0x7fb3c0055500 con 0x7fb3cc0b07b0 2024-01-11T09:38:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:38:17.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.490+0000 7fb3d1bff700 1 -- 172.21.15.27:0/2623283596 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb3b80670a0 msgr2=0x7fb3b8069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:38:17.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.490+0000 7fb3d1bff700 1 --2- 172.21.15.27:0/2623283596 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb3b80670a0 0x7fb3b8069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc001840 tx=0x7fb3bc009040 comp rx=0 tx=0).stop 2024-01-11T09:38:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.490+0000 7fb3d1bff700 1 -- 172.21.15.27:0/2623283596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3cc0b07b0 msgr2=0x7fb3cc0c3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:38:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.490+0000 7fb3d1bff700 1 --2- 172.21.15.27:0/2623283596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3cc0b07b0 0x7fb3cc0c3ea0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fb3c000d860 tx=0x7fb3c0008040 comp rx=0 tx=0).stop 2024-01-11T09:38:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.490+0000 7fb3d1bff700 1 -- 172.21.15.27:0/2623283596 shutdown_connections 2024-01-11T09:38:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.490+0000 7fb3d1bff700 1 --2- 172.21.15.27:0/2623283596 >> [v2:172.21.15.77:6836/41381,v1:172.21.15.77:6837/41381] conn(0x7fb3b80670a0 0x7fb3b8069550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:38:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.490+0000 7fb3d1bff700 1 --2- 172.21.15.27:0/2623283596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3cc0b2b90 0x7fb3cc0af6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:38:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.490+0000 7fb3d1bff700 1 --2- 172.21.15.27:0/2623283596 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3cc0b03e0 0x7fb3cc0c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:38:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.490+0000 7fb3d1bff700 1 --2- 172.21.15.27:0/2623283596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3cc0b07b0 0x7fb3cc0c3ea0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:38:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.490+0000 7fb3d1bff700 1 -- 172.21.15.27:0/2623283596 >> 172.21.15.27:0/2623283596 conn(0x7fb3cc06ed40 msgr2=0x7fb3cc0cb400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:38:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.491+0000 7fb3d1bff700 1 -- 172.21.15.27:0/2623283596 shutdown_connections 2024-01-11T09:38:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:38:17.491+0000 7fb3d1bff700 1 -- 172.21.15.27:0/2623283596 wait complete. 2024-01-11T09:38:17.501 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":106,"active_gid":8153,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":41381},{"type":"v1","addr":"172.21.15.77:6837","nonce":41381}]},"active_addr":"172.21.15.77:6837/41381","active_change":"2024-01-11T09:27:37.837325+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":8150,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":182,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":2548321690}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":1160908778}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":3404938947}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":2186545586}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":2730921345}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":3966666871}]}]} 2024-01-11T09:38:17.542 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: x 2024-01-11T09:38:17.542 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-11T09:38:17.542 INFO:tasks.ceph.mgr.x:Stopping old one... 2024-01-11T09:38:17.542 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-11T09:38:17.542 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:38:17.616 INFO:tasks.ceph.mgr.x:Stopped 2024-01-11T09:38:17.616 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-11T09:38:17.620 INFO:tasks.ceph.mgr.x:Started 2024-01-11T09:38:17.882 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:17.889+0000 7fda49c811c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:38:18.061 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:18.068+0000 7fda49c811c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:38:18.373 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:18.381+0000 7fda49c811c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:38:18.893 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:18.900+0000 7fda49c811c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:38:19.064 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:19.071+0000 7fda49c811c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:38:19.474 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:19.482+0000 7fda49c811c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:38:20.247 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:20.255+0000 7fda49c811c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:38:20.334 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:20.342+0000 7fda49c811c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:38:20.489 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:20.497+0000 7fda49c811c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:38:20.577 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:20.584+0000 7fda49c811c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:38:20.795 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:20.802+0000 7fda49c811c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:38:20.891 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:20.898+0000 7fda49c811c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:38:20.987 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:20.994+0000 7fda49c811c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:38:21.082 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:21.090+0000 7fda49c811c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:38:21.271 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:21.279+0000 7fda49c811c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:38:21.368 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:21.375+0000 7fda49c811c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:38:21.710 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:21.717+0000 7fda49c811c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:38:21.799 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:21.807+0000 7fda49c811c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:38:22.336 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:22.344+0000 7fda49c811c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:38:22.523 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:22.530+0000 7fda49c811c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:38:22.898 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:22.906+0000 7fda49c811c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:38:23.315 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:23.322+0000 7fda49c811c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:38:23.499 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:23.505+0000 7fda49c811c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:38:23.686 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:23.693+0000 7fda49c811c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:38:23.795 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:23.803+0000 7fda49c811c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:38:23.957 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:23.965+0000 7fd9bea38700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:38:23.957 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:23.965+0000 7fd9bea38700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:38:23.957 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:23.965+0000 7fd9bea38700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:38:23.957 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:23.965+0000 7fd9bea38700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:38:23.958 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:23.965+0000 7fd9bea38700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:38:23.958 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:23.966+0000 7fd9c1a3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:38:23.958 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:23.966+0000 7fd9c1a3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:38:23.958 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:23.966+0000 7fd9c1a3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:38:23.958 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:23.966+0000 7fd9c1a3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:38:23.958 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:38:23.966+0000 7fd9c1a3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:38:46.013 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:38:46.019 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:39:00.952 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:39:00.960+0000 7fd9c9a8e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:39:00.953 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:39:00.960+0000 7fd9c9a8e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:39:00.953 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:39:00.960+0000 7fd9c9a8e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:39:00.953 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:39:00.960+0000 7fd9c9a8e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:39:00.953 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:39:00.960+0000 7fd9c9a8e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:39:16.137 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:39:16.142 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:39:46.317 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:39:46.323 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:40:16.433 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:40:16.440 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:40:46.622 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:40:46.627 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:41:16.811 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:41:16.816 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:41:46.951 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:41:46.956 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:42:17.066 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:42:17.071 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:42:47.271 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:42:47.277 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:43:17.469 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:43:17.475 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:43:17.621 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/test_restart 1m 2024-01-11T09:43:17.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.836+0000 7f87517ff700 1 Processor -- start 2024-01-11T09:43:17.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.836+0000 7f87517ff700 1 -- start start 2024-01-11T09:43:17.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.837+0000 7f87517ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f874c143190 0x7f874c0cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:17.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.837+0000 7f87517ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f874c0ccc70 con 0x7f874c142dc0 2024-01-11T09:43:17.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.837+0000 7f87517ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f874c15d320 con 0x7f874c0cb4b0 2024-01-11T09:43:17.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.837+0000 7f87517ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f874c15df30 con 0x7f874c143190 2024-01-11T09:43:17.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.837+0000 7f874affd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f874c142dc0 0x7f874c0cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52254/0 (socket says 172.21.15.27:52254) 2024-01-11T09:43:17.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.837+0000 7f874affd700 1 -- 172.21.15.27:0/3500379738 learned_addr learned my addr 172.21.15.27:0/3500379738 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:17.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.838+0000 7f874a7fc700 1 --2- 172.21.15.27:0/3500379738 >> v2:172.21.15.77:3300/0 conn(0x7f874c143190 0x7f874c0cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:17.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.838+0000 7f8749ffb700 1 -- 172.21.15.27:0/3500379738 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2539036978 0 0) 0x7f874c0ccc70 con 0x7f874c142dc0 2024-01-11T09:43:17.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.838+0000 7f8749ffb700 1 -- 172.21.15.27:0/3500379738 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8734003570 con 0x7f874c142dc0 2024-01-11T09:43:17.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.838+0000 7f8749ffb700 1 -- 172.21.15.27:0/3500379738 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144218227 0 0) 0x7f874c15d320 con 0x7f874c0cb4b0 2024-01-11T09:43:17.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.838+0000 7f8749ffb700 1 -- 172.21.15.27:0/3500379738 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f874c0ccc70 con 0x7f874c0cb4b0 2024-01-11T09:43:17.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.838+0000 7f874a7fc700 1 -- 172.21.15.27:0/3500379738 >> v1:172.21.15.77:6789/0 conn(0x7f874c0cb4b0 legacy=0x7f874c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:17.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.838+0000 7f8749ffb700 1 -- 172.21.15.27:0/3500379738 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1986543083 0 0) 0x7f8734003570 con 0x7f874c142dc0 2024-01-11T09:43:17.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.838+0000 7f874a7fc700 1 -- 172.21.15.27:0/3500379738 >> v1:172.21.15.27:6789/0 conn(0x7f874c142dc0 legacy=0x7f874c0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:17.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.838+0000 7f874a7fc700 1 -- 172.21.15.27:0/3500379738 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874c15d590 con 0x7f874c143190 2024-01-11T09:43:17.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.839+0000 7f874a7fc700 1 --2- 172.21.15.27:0/3500379738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f874c143190 0x7f874c0cc020 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f87400098e0 tx=0x7f87400091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=230f55077af211ee server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:17.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.839+0000 7f8749ffb700 1 -- 172.21.15.27:0/3500379738 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c001470 con 0x7f874c143190 2024-01-11T09:43:17.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.839+0000 7f8749ffb700 1 -- 172.21.15.27:0/3500379738 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8740011420 con 0x7f874c143190 2024-01-11T09:43:17.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.839+0000 7f8749ffb700 1 -- 172.21.15.27:0/3500379738 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87400117c0 con 0x7f874c143190 2024-01-11T09:43:17.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.839+0000 7f87517ff700 1 -- 172.21.15.27:0/3500379738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f874c143190 msgr2=0x7f874c0cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:17.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.839+0000 7f87517ff700 1 --2- 172.21.15.27:0/3500379738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f874c143190 0x7f874c0cc020 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f87400098e0 tx=0x7f87400091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:17.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.840+0000 7f87517ff700 1 -- 172.21.15.27:0/3500379738 shutdown_connections 2024-01-11T09:43:17.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.840+0000 7f87517ff700 1 --2- 172.21.15.27:0/3500379738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f874c143190 0x7f874c0cc020 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:17.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.840+0000 7f87517ff700 1 -- 172.21.15.27:0/3500379738 >> 172.21.15.27:0/3500379738 conn(0x7f874c063ca0 msgr2=0x7f874c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:17.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.840+0000 7f87517ff700 1 -- 172.21.15.27:0/3500379738 shutdown_connections 2024-01-11T09:43:17.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.840+0000 7f87517ff700 1 -- 172.21.15.27:0/3500379738 wait complete. 2024-01-11T09:43:17.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.841+0000 7f87517ff700 1 Processor -- start 2024-01-11T09:43:17.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.841+0000 7f87517ff700 1 -- start start 2024-01-11T09:43:17.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.842+0000 7f87517ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f874c0cb4b0 0x7f874c140150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:17.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.842+0000 7f87517ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f874c142dc0 0x7f874c0c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:17.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.842+0000 7f87517ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f874c143190 0x7f874c0b3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:17.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.842+0000 7f87517ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874c140690 con 0x7f874c0cb4b0 2024-01-11T09:43:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.842+0000 7f87517ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874c0c9420 con 0x7f874c142dc0 2024-01-11T09:43:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.842+0000 7f87517ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f874c16cde0 con 0x7f874c143190 2024-01-11T09:43:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.842+0000 7f874affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f874c0cb4b0 0x7f874c140150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.842+0000 7f874affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f874c0cb4b0 0x7f874c140150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35222/0 (socket says 172.21.15.27:35222) 2024-01-11T09:43:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.842+0000 7f874affd700 1 -- 172.21.15.27:0/1485228506 learned_addr learned my addr 172.21.15.27:0/1485228506 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.843+0000 7f874a7fc700 1 --2- 172.21.15.27:0/1485228506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f874c142dc0 0x7f874c0c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.843+0000 7f874affd700 1 -- 172.21.15.27:0/1485228506 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f874c143190 msgr2=0x7f874c0b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.843+0000 7f874affd700 1 --2- 172.21.15.27:0/1485228506 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f874c143190 0x7f874c0b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.843+0000 7f874affd700 1 -- 172.21.15.27:0/1485228506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f874c142dc0 msgr2=0x7f874c0c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.843+0000 7f874affd700 1 --2- 172.21.15.27:0/1485228506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f874c142dc0 0x7f874c0c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.843+0000 7f874affd700 1 -- 172.21.15.27:0/1485228506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8740009420 con 0x7f874c0cb4b0 2024-01-11T09:43:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.843+0000 7f874a7fc700 1 --2- 172.21.15.27:0/1485228506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f874c142dc0 0x7f874c0c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:43:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.843+0000 7f874affd700 1 --2- 172.21.15.27:0/1485228506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f874c0cb4b0 0x7f874c140150 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f873c00cb90 tx=0x7f873c001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.843+0000 7f8732ffd700 1 -- 172.21.15.27:0/1485228506 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c007360 con 0x7f874c0cb4b0 2024-01-11T09:43:17.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.843+0000 7f8732ffd700 1 -- 172.21.15.27:0/1485228506 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873c0074e0 con 0x7f874c0cb4b0 2024-01-11T09:43:17.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.843+0000 7f87517ff700 1 -- 172.21.15.27:0/1485228506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f874c16d0c0 con 0x7f874c0cb4b0 2024-01-11T09:43:17.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.843+0000 7f8732ffd700 1 -- 172.21.15.27:0/1485228506 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c004830 con 0x7f874c0cb4b0 2024-01-11T09:43:17.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.844+0000 7f87517ff700 1 -- 172.21.15.27:0/1485228506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f874c16d680 con 0x7f874c0cb4b0 2024-01-11T09:43:17.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.845+0000 7f87517ff700 1 -- 172.21.15.27:0/1485228506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f874c04c3c0 con 0x7f874c0cb4b0 2024-01-11T09:43:17.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.845+0000 7f8732ffd700 1 -- 172.21.15.27:0/1485228506 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 112) v1 ==== 84873+0+0 (secure 0 0 0) 0x7f873c004ba0 con 0x7f874c0cb4b0 2024-01-11T09:43:17.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.846+0000 7f8732ffd700 1 --2- 172.21.15.27:0/1485228506 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f87380670a0 0x7f8738069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:17.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.846+0000 7f8732ffd700 1 -- 172.21.15.27:0/1485228506 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6277+0+0 (secure 0 0 0) 0x7f873c0841e0 con 0x7f874c0cb4b0 2024-01-11T09:43:17.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.847+0000 7f874a7fc700 1 --2- 172.21.15.27:0/1485228506 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f87380670a0 0x7f8738069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:17.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.847+0000 7f874a7fc700 1 --2- 172.21.15.27:0/1485228506 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f87380670a0 0x7f8738069550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f874c0c5540 tx=0x7f8740016000 comp rx=0 tx=0).ready entity=mgr.9457 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:17.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:17.851+0000 7f8732ffd700 1 -- 172.21.15.27:0/1485228506 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f873c0510e0 con 0x7f874c0cb4b0 2024-01-11T09:43:18.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.081+0000 7f87517ff700 1 -- 172.21.15.27:0/1485228506 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f874c15d870 con 0x7f87380670a0 2024-01-11T09:43:18.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.146+0000 7f8732ffd700 1 -- 172.21.15.27:0/1485228506 <== mgr.9457 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f874c15d870 con 0x7f87380670a0 2024-01-11T09:43:18.138 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2024-01-11T09:43:18.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.149+0000 7f87517ff700 1 -- 172.21.15.27:0/1485228506 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f87380670a0 msgr2=0x7f8738069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:18.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.149+0000 7f87517ff700 1 --2- 172.21.15.27:0/1485228506 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f87380670a0 0x7f8738069550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f874c0c5540 tx=0x7f8740016000 comp rx=0 tx=0).stop 2024-01-11T09:43:18.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.149+0000 7f87517ff700 1 -- 172.21.15.27:0/1485228506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f874c0cb4b0 msgr2=0x7f874c140150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:18.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.149+0000 7f87517ff700 1 --2- 172.21.15.27:0/1485228506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f874c0cb4b0 0x7f874c140150 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f873c00cb90 tx=0x7f873c001b50 comp rx=0 tx=0).stop 2024-01-11T09:43:18.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.149+0000 7f87517ff700 1 -- 172.21.15.27:0/1485228506 shutdown_connections 2024-01-11T09:43:18.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.149+0000 7f87517ff700 1 --2- 172.21.15.27:0/1485228506 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f874c143190 0x7f874c0b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:18.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.149+0000 7f87517ff700 1 --2- 172.21.15.27:0/1485228506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f874c0cb4b0 0x7f874c140150 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:18.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.149+0000 7f87517ff700 1 --2- 172.21.15.27:0/1485228506 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f87380670a0 0x7f8738069550 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:18.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.149+0000 7f87517ff700 1 --2- 172.21.15.27:0/1485228506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f874c142dc0 0x7f874c0c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:18.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.149+0000 7f87517ff700 1 -- 172.21.15.27:0/1485228506 >> 172.21.15.27:0/1485228506 conn(0x7f874c063ca0 msgr2=0x7f874c0c4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:18.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.149+0000 7f87517ff700 1 -- 172.21.15.27:0/1485228506 shutdown_connections 2024-01-11T09:43:18.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.149+0000 7f87517ff700 1 -- 172.21.15.27:0/1485228506 wait complete. 2024-01-11T09:43:18.158 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2024-01-11T09:43:18.158 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:18.159 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-01-11T09:43:18.207 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_28_00_UTC 2024-01-11T09:43:18.207 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_29_00_UTC 2024-01-11T09:43:18.208 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_30_00_UTC 2024-01-11T09:43:18.208 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_31_00_UTC 2024-01-11T09:43:18.208 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_32_00_UTC 2024-01-11T09:43:18.208 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_33_00_UTC 2024-01-11T09:43:18.208 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_34_00_UTC 2024-01-11T09:43:18.208 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_35_00_UTC 2024-01-11T09:43:18.208 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_36_00_UTC 2024-01-11T09:43:18.208 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_37_00_UTC 2024-01-11T09:43:18.208 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_38_00_UTC 2024-01-11T09:43:18.208 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_39_00_UTC 2024-01-11T09:43:18.208 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_40_00_UTC 2024-01-11T09:43:18.208 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_41_00_UTC 2024-01-11T09:43:18.208 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_42_00_UTC 2024-01-11T09:43:18.209 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_43_00_UTC 2024-01-11T09:43:18.209 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-01-11-09_28_00_UTC', 'scheduled-2024-01-11-09_29_00_UTC', 'scheduled-2024-01-11-09_30_00_UTC', 'scheduled-2024-01-11-09_31_00_UTC', 'scheduled-2024-01-11-09_32_00_UTC', 'scheduled-2024-01-11-09_33_00_UTC', 'scheduled-2024-01-11-09_34_00_UTC', 'scheduled-2024-01-11-09_35_00_UTC', 'scheduled-2024-01-11-09_36_00_UTC', 'scheduled-2024-01-11-09_37_00_UTC', 'scheduled-2024-01-11-09_38_00_UTC', 'scheduled-2024-01-11-09_39_00_UTC', 'scheduled-2024-01-11-09_40_00_UTC', 'scheduled-2024-01-11-09_41_00_UTC', 'scheduled-2024-01-11-09_42_00_UTC', 'scheduled-2024-01-11-09_43_00_UTC'] 2024-01-11T09:43:18.210 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2024-01-11T09:43:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.420+0000 7fd462bff700 1 Processor -- start 2024-01-11T09:43:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.420+0000 7fd462bff700 1 -- start start 2024-01-11T09:43:18.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.422+0000 7fd462bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd45c0ccd70 0x7fd45c0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:18.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.422+0000 7fd462bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd45c0cf850 con 0x7fd45c15ae70 2024-01-11T09:43:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.422+0000 7fd462bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd45c0cec40 con 0x7fd45c15a1b0 2024-01-11T09:43:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.422+0000 7fd462bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45c15ac90 con 0x7fd45c0ccd70 2024-01-11T09:43:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.422+0000 7fd45bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd45c0ccd70 0x7fd45c0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.422+0000 7fd46119c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd45c15ae70 0x7fd45c15b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52264/0 (socket says 172.21.15.27:52264) 2024-01-11T09:43:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.422+0000 7fd46119c700 1 -- 172.21.15.27:0/642602262 learned_addr learned my addr 172.21.15.27:0/642602262 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.422+0000 7fd45bfff700 1 -- 172.21.15.27:0/642602262 >> v1:172.21.15.77:6789/0 conn(0x7fd45c15a1b0 legacy=0x7fd45c15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:18.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.422+0000 7fd45bfff700 1 -- 172.21.15.27:0/642602262 >> v1:172.21.15.27:6789/0 conn(0x7fd45c15ae70 legacy=0x7fd45c15b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:18.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.423+0000 7fd45bfff700 1 -- 172.21.15.27:0/642602262 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45c0cef20 con 0x7fd45c0ccd70 2024-01-11T09:43:18.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.423+0000 7fd45bfff700 1 --2- 172.21.15.27:0/642602262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd45c0ccd70 0x7fd45c0cd210 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fd4480098e0 tx=0x7fd4480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d01ed65df0c3e14 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:18.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.423+0000 7fd45b7fe700 1 -- 172.21.15.27:0/642602262 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd448009e80 con 0x7fd45c0ccd70 2024-01-11T09:43:18.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.423+0000 7fd45b7fe700 1 -- 172.21.15.27:0/642602262 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd44800b860 con 0x7fd45c0ccd70 2024-01-11T09:43:18.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.423+0000 7fd45b7fe700 1 -- 172.21.15.27:0/642602262 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44800bbe0 con 0x7fd45c0ccd70 2024-01-11T09:43:18.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.424+0000 7fd462bff700 1 -- 172.21.15.27:0/642602262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd45c0ccd70 msgr2=0x7fd45c0cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:18.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.424+0000 7fd462bff700 1 --2- 172.21.15.27:0/642602262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd45c0ccd70 0x7fd45c0cd210 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fd4480098e0 tx=0x7fd4480091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:18.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.424+0000 7fd462bff700 1 -- 172.21.15.27:0/642602262 shutdown_connections 2024-01-11T09:43:18.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.424+0000 7fd462bff700 1 --2- 172.21.15.27:0/642602262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd45c0ccd70 0x7fd45c0cd210 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:18.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.424+0000 7fd462bff700 1 -- 172.21.15.27:0/642602262 >> 172.21.15.27:0/642602262 conn(0x7fd45c06ed40 msgr2=0x7fd45c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:18.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.424+0000 7fd462bff700 1 -- 172.21.15.27:0/642602262 shutdown_connections 2024-01-11T09:43:18.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.424+0000 7fd462bff700 1 -- 172.21.15.27:0/642602262 wait complete. 2024-01-11T09:43:18.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.425+0000 7fd462bff700 1 Processor -- start 2024-01-11T09:43:18.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.425+0000 7fd462bff700 1 -- start start 2024-01-11T09:43:18.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.426+0000 7fd462bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd45c0ccd70 0x7fd45c14b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:18.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.426+0000 7fd462bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd45c15a1b0 0x7fd45c157de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:18.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.426+0000 7fd462bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd45c15ae70 0x7fd45c0b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:18.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.426+0000 7fd462bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c1586e0 con 0x7fd45c15ae70 2024-01-11T09:43:18.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.426+0000 7fd462bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c1589c0 con 0x7fd45c0ccd70 2024-01-11T09:43:18.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.426+0000 7fd462bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c158ca0 con 0x7fd45c15a1b0 2024-01-11T09:43:18.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.427+0000 7fd46119c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd45c15ae70 0x7fd45c0b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:18.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.427+0000 7fd46119c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd45c15ae70 0x7fd45c0b54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35236/0 (socket says 172.21.15.27:35236) 2024-01-11T09:43:18.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.427+0000 7fd46119c700 1 -- 172.21.15.27:0/3528944096 learned_addr learned my addr 172.21.15.27:0/3528944096 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:18.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.427+0000 7fd46099b700 1 --2- 172.21.15.27:0/3528944096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd45c0ccd70 0x7fd45c14b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:18.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.427+0000 7fd45bfff700 1 --2- 172.21.15.27:0/3528944096 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd45c15a1b0 0x7fd45c157de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:18.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.427+0000 7fd46119c700 1 -- 172.21.15.27:0/3528944096 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd45c15a1b0 msgr2=0x7fd45c157de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:18.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.427+0000 7fd46119c700 1 --2- 172.21.15.27:0/3528944096 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd45c15a1b0 0x7fd45c157de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:18.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.427+0000 7fd46119c700 1 -- 172.21.15.27:0/3528944096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd45c0ccd70 msgr2=0x7fd45c14b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:18.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.427+0000 7fd46119c700 1 --2- 172.21.15.27:0/3528944096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd45c0ccd70 0x7fd45c14b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:18.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.427+0000 7fd46119c700 1 -- 172.21.15.27:0/3528944096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd448009420 con 0x7fd45c15ae70 2024-01-11T09:43:18.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.427+0000 7fd46099b700 1 --2- 172.21.15.27:0/3528944096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd45c0ccd70 0x7fd45c14b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:18.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.427+0000 7fd46119c700 1 --2- 172.21.15.27:0/3528944096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd45c15ae70 0x7fd45c0b54d0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fd45400cc40 tx=0x7fd454001a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:18.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.428+0000 7fd458ff9700 1 -- 172.21.15.27:0/3528944096 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45400a910 con 0x7fd45c15ae70 2024-01-11T09:43:18.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.428+0000 7fd458ff9700 1 -- 172.21.15.27:0/3528944096 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45400aa70 con 0x7fd45c15ae70 2024-01-11T09:43:18.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.428+0000 7fd462bff700 1 -- 172.21.15.27:0/3528944096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd45c158f80 con 0x7fd45c15ae70 2024-01-11T09:43:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.428+0000 7fd458ff9700 1 -- 172.21.15.27:0/3528944096 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45400ae30 con 0x7fd45c15ae70 2024-01-11T09:43:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.428+0000 7fd462bff700 1 -- 172.21.15.27:0/3528944096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd45c14b570 con 0x7fd45c15ae70 2024-01-11T09:43:18.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.429+0000 7fd462bff700 1 -- 172.21.15.27:0/3528944096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd45c04c3c0 con 0x7fd45c15ae70 2024-01-11T09:43:18.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.430+0000 7fd458ff9700 1 -- 172.21.15.27:0/3528944096 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 112) v1 ==== 84873+0+0 (secure 0 0 0) 0x7fd454007300 con 0x7fd45c15ae70 2024-01-11T09:43:18.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.431+0000 7fd458ff9700 1 --2- 172.21.15.27:0/3528944096 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd4400670a0 0x7fd440069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:18.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.431+0000 7fd458ff9700 1 -- 172.21.15.27:0/3528944096 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6277+0+0 (secure 0 0 0) 0x7fd454084cd0 con 0x7fd45c15ae70 2024-01-11T09:43:18.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.431+0000 7fd46099b700 1 --2- 172.21.15.27:0/3528944096 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd4400670a0 0x7fd440069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:18.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.432+0000 7fd46099b700 1 --2- 172.21.15.27:0/3528944096 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd4400670a0 0x7fd440069550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd450007fd0 tx=0x7fd450009040 comp rx=0 tx=0).ready entity=mgr.9457 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:18.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.435+0000 7fd458ff9700 1 -- 172.21.15.27:0/3528944096 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd454051c70 con 0x7fd45c15ae70 2024-01-11T09:43:18.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.662+0000 7fd462bff700 1 -- 172.21.15.27:0/3528944096 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd45c0b2730 con 0x7fd4400670a0 2024-01-11T09:43:18.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.670+0000 7fd458ff9700 1 -- 172.21.15.27:0/3528944096 <== mgr.9457 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+361 (secure 0 0 0) 0x7fd45c0b2730 con 0x7fd4400670a0 2024-01-11T09:43:18.662 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:43:18.662 INFO:teuthology.orchestra.run.smithi027.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T09:27:38", "first": "2024-01-11T09:28:00", "last": "2024-01-11T09:43:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2024-01-11T09:43:18.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.672+0000 7fd462bff700 1 -- 172.21.15.27:0/3528944096 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd4400670a0 msgr2=0x7fd440069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:18.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.672+0000 7fd462bff700 1 --2- 172.21.15.27:0/3528944096 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd4400670a0 0x7fd440069550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd450007fd0 tx=0x7fd450009040 comp rx=0 tx=0).stop 2024-01-11T09:43:18.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.672+0000 7fd462bff700 1 -- 172.21.15.27:0/3528944096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd45c15ae70 msgr2=0x7fd45c0b54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:18.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.672+0000 7fd462bff700 1 --2- 172.21.15.27:0/3528944096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd45c15ae70 0x7fd45c0b54d0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fd45400cc40 tx=0x7fd454001a50 comp rx=0 tx=0).stop 2024-01-11T09:43:18.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.672+0000 7fd462bff700 1 -- 172.21.15.27:0/3528944096 shutdown_connections 2024-01-11T09:43:18.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.672+0000 7fd462bff700 1 --2- 172.21.15.27:0/3528944096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd45c15ae70 0x7fd45c0b54d0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:18.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.672+0000 7fd462bff700 1 --2- 172.21.15.27:0/3528944096 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd4400670a0 0x7fd440069550 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:18.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.672+0000 7fd462bff700 1 --2- 172.21.15.27:0/3528944096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd45c0ccd70 0x7fd45c14b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:18.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.672+0000 7fd462bff700 1 --2- 172.21.15.27:0/3528944096 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd45c15a1b0 0x7fd45c157de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:18.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.672+0000 7fd462bff700 1 -- 172.21.15.27:0/3528944096 >> 172.21.15.27:0/3528944096 conn(0x7fd45c06ed40 msgr2=0x7fd45c156120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:18.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.672+0000 7fd462bff700 1 -- 172.21.15.27:0/3528944096 shutdown_connections 2024-01-11T09:43:18.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.673+0000 7fd462bff700 1 -- 172.21.15.27:0/3528944096 wait complete. 2024-01-11T09:43:18.682 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T09:27:38", "first": "2024-01-11T09:28:00", "last": "2024-01-11T09:43:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2024-01-11T09:43:18.682 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2024-01-11T00:00:00', 'created': '2024-01-11T09:27:38', 'first': '2024-01-11T09:28:00', 'last': '2024-01-11T09:43:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2024-01-11T09:43:18.682 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 16 2024-01-11T09:43:18.683 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 16 2024-01-11T09:43:18.683 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/test_restart 1m 2024-01-11T09:43:18.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.893+0000 7fa0ebfff700 1 Processor -- start 2024-01-11T09:43:18.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.893+0000 7fa0ebfff700 1 -- start start 2024-01-11T09:43:18.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.895+0000 7fa0ebfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa0e4143190 0x7fa0e40cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:18.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.895+0000 7fa0ebfff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e40ccc70 con 0x7fa0e40cb4b0 2024-01-11T09:43:18.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.895+0000 7fa0ebfff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e415d320 con 0x7fa0e4142dc0 2024-01-11T09:43:18.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.895+0000 7fa0ebfff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e415df30 con 0x7fa0e4143190 2024-01-11T09:43:18.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.895+0000 7fa0ea59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa0e40cb4b0 0x7fa0e40cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52280/0 (socket says 172.21.15.27:52280) 2024-01-11T09:43:18.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.895+0000 7fa0ea59c700 1 -- 172.21.15.27:0/1878902429 learned_addr learned my addr 172.21.15.27:0/1878902429 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:18.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.895+0000 7fa0e959a700 1 --2- 172.21.15.27:0/1878902429 >> v2:172.21.15.77:3300/0 conn(0x7fa0e4143190 0x7fa0e40cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:18.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.895+0000 7fa0e8d99700 1 -- 172.21.15.27:0/1878902429 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245870431 0 0) 0x7fa0e40ccc70 con 0x7fa0e40cb4b0 2024-01-11T09:43:18.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.895+0000 7fa0e8d99700 1 -- 172.21.15.27:0/1878902429 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0d0003570 con 0x7fa0e40cb4b0 2024-01-11T09:43:18.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.896+0000 7fa0e8d99700 1 -- 172.21.15.27:0/1878902429 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2108238641 0 0) 0x7fa0d0003570 con 0x7fa0e40cb4b0 2024-01-11T09:43:18.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.896+0000 7fa0e8d99700 1 -- 172.21.15.27:0/1878902429 >> v2:172.21.15.77:3300/0 conn(0x7fa0e4143190 msgr2=0x7fa0e40cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:18.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.896+0000 7fa0e8d99700 1 --2- 172.21.15.27:0/1878902429 >> v2:172.21.15.77:3300/0 conn(0x7fa0e4143190 0x7fa0e40cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:18.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.896+0000 7fa0e8d99700 1 -- 172.21.15.27:0/1878902429 >> v1:172.21.15.77:6789/0 conn(0x7fa0e4142dc0 legacy=0x7fa0e40cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:18.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.896+0000 7fa0e8d99700 1 -- 172.21.15.27:0/1878902429 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e415d590 con 0x7fa0e40cb4b0 2024-01-11T09:43:18.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.896+0000 7fa0e8d99700 1 -- 172.21.15.27:0/1878902429 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fa0dc0017f0 con 0x7fa0e40cb4b0 2024-01-11T09:43:18.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.896+0000 7fa0e959a700 1 --2- 172.21.15.27:0/1878902429 >> v2:172.21.15.77:3300/0 conn(0x7fa0e4143190 0x7fa0e40cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:43:18.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.896+0000 7fa0e8d99700 1 -- 172.21.15.27:0/1878902429 >> v1:172.21.15.27:6789/0 conn(0x7fa0e40cb4b0 legacy=0x7fa0e40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:18.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.896+0000 7fa0e8d99700 1 --2- 172.21.15.27:0/1878902429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d0003a60 0x7fa0d0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:18.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.896+0000 7fa0e8d99700 1 --2- 172.21.15.27:0/1878902429 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0d0004810 0x7fa0d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:18.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.896+0000 7fa0e8d99700 1 --2- 172.21.15.27:0/1878902429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0d00051c0 0x7fa0d000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:18.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.896+0000 7fa0e8d99700 1 -- 172.21.15.27:0/1878902429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e415df30 con 0x7fa0d0003a60 2024-01-11T09:43:18.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.896+0000 7fa0e8d99700 1 -- 172.21.15.27:0/1878902429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d000c050 con 0x7fa0d00051c0 2024-01-11T09:43:18.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.896+0000 7fa0e8d99700 1 -- 172.21.15.27:0/1878902429 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d000c210 con 0x7fa0d0004810 2024-01-11T09:43:18.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.896+0000 7fa0e9d9b700 1 --2- 172.21.15.27:0/1878902429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d0003a60 0x7fa0d0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:18.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.896+0000 7fa0ea59c700 1 --2- 172.21.15.27:0/1878902429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0d00051c0 0x7fa0d000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:18.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.896+0000 7fa0e959a700 1 --2- 172.21.15.27:0/1878902429 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0d0004810 0x7fa0d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:18.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.897+0000 7fa0e9d9b700 1 -- 172.21.15.27:0/1878902429 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0d0004810 msgr2=0x7fa0d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:18.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.897+0000 7fa0e9d9b700 1 --2- 172.21.15.27:0/1878902429 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0d0004810 0x7fa0d0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:18.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.897+0000 7fa0e9d9b700 1 -- 172.21.15.27:0/1878902429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0d00051c0 msgr2=0x7fa0d000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:18.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.897+0000 7fa0e9d9b700 1 --2- 172.21.15.27:0/1878902429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0d00051c0 0x7fa0d000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:18.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.897+0000 7fa0e9d9b700 1 -- 172.21.15.27:0/1878902429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d0003570 con 0x7fa0d0003a60 2024-01-11T09:43:18.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.897+0000 7fa0ea59c700 1 -- 172.21.15.27:0/1878902429 reap_dead start 2024-01-11T09:43:18.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.897+0000 7fa0e9d9b700 1 --2- 172.21.15.27:0/1878902429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d0003a60 0x7fa0d0003e30 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0001960 tx=0x7fa0e0007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab18f6e13ba236c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:18.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.897+0000 7fa0e8d99700 1 -- 172.21.15.27:0/1878902429 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e000c5d0 con 0x7fa0d0003a60 2024-01-11T09:43:18.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.897+0000 7fa0e8d99700 1 -- 172.21.15.27:0/1878902429 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e000c750 con 0x7fa0d0003a60 2024-01-11T09:43:18.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.897+0000 7fa0e8d99700 1 -- 172.21.15.27:0/1878902429 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e000cb10 con 0x7fa0d0003a60 2024-01-11T09:43:18.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.898+0000 7fa0ebfff700 1 -- 172.21.15.27:0/1878902429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d0003a60 msgr2=0x7fa0d0003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:18.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.898+0000 7fa0ebfff700 1 --2- 172.21.15.27:0/1878902429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d0003a60 0x7fa0d0003e30 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0001960 tx=0x7fa0e0007d10 comp rx=0 tx=0).stop 2024-01-11T09:43:18.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.898+0000 7fa0ebfff700 1 -- 172.21.15.27:0/1878902429 shutdown_connections 2024-01-11T09:43:18.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.898+0000 7fa0ebfff700 1 --2- 172.21.15.27:0/1878902429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d0003a60 0x7fa0d0003e30 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:18.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.898+0000 7fa0ebfff700 1 -- 172.21.15.27:0/1878902429 >> 172.21.15.27:0/1878902429 conn(0x7fa0e4063ca0 msgr2=0x7fa0e406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:18.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.898+0000 7fa0ebfff700 1 -- 172.21.15.27:0/1878902429 shutdown_connections 2024-01-11T09:43:18.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.898+0000 7fa0ebfff700 1 -- 172.21.15.27:0/1878902429 wait complete. 2024-01-11T09:43:18.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.900+0000 7fa0ebfff700 1 Processor -- start 2024-01-11T09:43:18.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.900+0000 7fa0ebfff700 1 -- start start 2024-01-11T09:43:18.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.901+0000 7fa0ebfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d0003a60 0x7fa0e413ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:18.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.901+0000 7fa0ebfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0e40c5490 0x7fa0e40b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:18.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.901+0000 7fa0ebfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0e40c5900 0x7fa0e40b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:18.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.901+0000 7fa0ebfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e40b2c10 con 0x7fa0d0003a60 2024-01-11T09:43:18.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.902+0000 7fa0ebfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e413f1b0 con 0x7fa0e40c5490 2024-01-11T09:43:18.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.902+0000 7fa0ebfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e40b1180 con 0x7fa0e40c5900 2024-01-11T09:43:18.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.902+0000 7fa0e9d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d0003a60 0x7fa0e413ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:18.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.902+0000 7fa0e9d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d0003a60 0x7fa0e413ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35242/0 (socket says 172.21.15.27:35242) 2024-01-11T09:43:18.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.902+0000 7fa0e9d9b700 1 -- 172.21.15.27:0/3611041688 learned_addr learned my addr 172.21.15.27:0/3611041688 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:18.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.902+0000 7fa0e959a700 1 --2- 172.21.15.27:0/3611041688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0e40c5490 0x7fa0e40b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:18.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.902+0000 7fa0ea59c700 1 --2- 172.21.15.27:0/3611041688 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0e40c5900 0x7fa0e40b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:18.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.902+0000 7fa0e9d9b700 1 -- 172.21.15.27:0/3611041688 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0e40c5900 msgr2=0x7fa0e40b26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:18.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.902+0000 7fa0e9d9b700 1 --2- 172.21.15.27:0/3611041688 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0e40c5900 0x7fa0e40b26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:18.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.902+0000 7fa0e9d9b700 1 -- 172.21.15.27:0/3611041688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0e40c5490 msgr2=0x7fa0e40b2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:18.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.902+0000 7fa0e9d9b700 1 --2- 172.21.15.27:0/3611041688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0e40c5490 0x7fa0e40b2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:18.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.902+0000 7fa0e9d9b700 1 -- 172.21.15.27:0/3611041688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e40b13d0 con 0x7fa0d0003a60 2024-01-11T09:43:18.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.902+0000 7fa0e9d9b700 1 --2- 172.21.15.27:0/3611041688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d0003a60 0x7fa0e413ec70 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0012f40 tx=0x7fa0e0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:18.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.903+0000 7fa0d9ffb700 1 -- 172.21.15.27:0/3611041688 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e000c5d0 con 0x7fa0d0003a60 2024-01-11T09:43:18.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.903+0000 7fa0d9ffb700 1 -- 172.21.15.27:0/3611041688 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e000cad0 con 0x7fa0d0003a60 2024-01-11T09:43:18.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.903+0000 7fa0d9ffb700 1 -- 172.21.15.27:0/3611041688 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e0006a50 con 0x7fa0d0003a60 2024-01-11T09:43:18.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.903+0000 7fa0ebfff700 1 -- 172.21.15.27:0/3611041688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e40b16b0 con 0x7fa0d0003a60 2024-01-11T09:43:18.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.903+0000 7fa0ebfff700 1 -- 172.21.15.27:0/3611041688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e40b1c70 con 0x7fa0d0003a60 2024-01-11T09:43:18.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.904+0000 7fa0ebfff700 1 -- 172.21.15.27:0/3611041688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e40b1e00 con 0x7fa0d0003a60 2024-01-11T09:43:18.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.905+0000 7fa0d9ffb700 1 -- 172.21.15.27:0/3611041688 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 112) v1 ==== 84873+0+0 (secure 0 0 0) 0x7fa0e0019070 con 0x7fa0d0003a60 2024-01-11T09:43:18.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.906+0000 7fa0d9ffb700 1 --2- 172.21.15.27:0/3611041688 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa0c80670a0 0x7fa0c8069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:18.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.906+0000 7fa0d9ffb700 1 -- 172.21.15.27:0/3611041688 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6277+0+0 (secure 0 0 0) 0x7fa0e0085d90 con 0x7fa0d0003a60 2024-01-11T09:43:18.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.906+0000 7fa0e959a700 1 --2- 172.21.15.27:0/3611041688 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa0c80670a0 0x7fa0c8069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:18.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.907+0000 7fa0e959a700 1 --2- 172.21.15.27:0/3611041688 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa0c80670a0 0x7fa0c8069550 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa0e40662b0 tx=0x7fa0d40091a0 comp rx=0 tx=0).ready entity=mgr.9457 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:18.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:18.910+0000 7fa0d9ffb700 1 -- 172.21.15.27:0/3611041688 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa0e0052cd0 con 0x7fa0d0003a60 2024-01-11T09:43:19.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:19.138+0000 7fa0ebfff700 1 -- 172.21.15.27:0/3611041688 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0e404c3c0 con 0x7fa0c80670a0 2024-01-11T09:43:19.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:19.219+0000 7fa0d9ffb700 1 -- 172.21.15.27:0/3611041688 <== mgr.9457 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7fa0e404c3c0 con 0x7fa0c80670a0 2024-01-11T09:43:19.212 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule removed for path /snap_test_dir1/test_restart 2024-01-11T09:43:19.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:19.222+0000 7fa0ebfff700 1 -- 172.21.15.27:0/3611041688 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa0c80670a0 msgr2=0x7fa0c8069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:19.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:19.222+0000 7fa0ebfff700 1 --2- 172.21.15.27:0/3611041688 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa0c80670a0 0x7fa0c8069550 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa0e40662b0 tx=0x7fa0d40091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:19.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:19.222+0000 7fa0ebfff700 1 -- 172.21.15.27:0/3611041688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d0003a60 msgr2=0x7fa0e413ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:19.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:19.222+0000 7fa0ebfff700 1 --2- 172.21.15.27:0/3611041688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d0003a60 0x7fa0e413ec70 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0012f40 tx=0x7fa0e0015000 comp rx=0 tx=0).stop 2024-01-11T09:43:19.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:19.222+0000 7fa0ebfff700 1 -- 172.21.15.27:0/3611041688 shutdown_connections 2024-01-11T09:43:19.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:19.222+0000 7fa0ebfff700 1 --2- 172.21.15.27:0/3611041688 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa0e40c5900 0x7fa0e40b26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:19.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:19.222+0000 7fa0ebfff700 1 --2- 172.21.15.27:0/3611041688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0d0003a60 0x7fa0e413ec70 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:19.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:19.222+0000 7fa0ebfff700 1 --2- 172.21.15.27:0/3611041688 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa0c80670a0 0x7fa0c8069550 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:19.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:19.222+0000 7fa0ebfff700 1 --2- 172.21.15.27:0/3611041688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0e40c5490 0x7fa0e40b2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:19.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:19.222+0000 7fa0ebfff700 1 -- 172.21.15.27:0/3611041688 >> 172.21.15.27:0/3611041688 conn(0x7fa0e4063ca0 msgr2=0x7fa0e4068720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:19.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:19.222+0000 7fa0ebfff700 1 -- 172.21.15.27:0/3611041688 shutdown_connections 2024-01-11T09:43:19.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:19.222+0000 7fa0ebfff700 1 -- 172.21.15.27:0/3611041688 wait complete. 2024-01-11T09:43:19.231 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2024-01-11T09:43:19.232 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:19.232 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-01-11T09:43:19.280 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_28_00_UTC 2024-01-11T09:43:19.280 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_29_00_UTC 2024-01-11T09:43:19.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_30_00_UTC 2024-01-11T09:43:19.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_31_00_UTC 2024-01-11T09:43:19.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_32_00_UTC 2024-01-11T09:43:19.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_33_00_UTC 2024-01-11T09:43:19.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_34_00_UTC 2024-01-11T09:43:19.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_35_00_UTC 2024-01-11T09:43:19.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_36_00_UTC 2024-01-11T09:43:19.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_37_00_UTC 2024-01-11T09:43:19.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_38_00_UTC 2024-01-11T09:43:19.281 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_39_00_UTC 2024-01-11T09:43:19.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_40_00_UTC 2024-01-11T09:43:19.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_41_00_UTC 2024-01-11T09:43:19.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_42_00_UTC 2024-01-11T09:43:19.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_43_00_UTC 2024-01-11T09:43:19.283 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_28_00_UTC 2024-01-11T09:43:19.283 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:19.283 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_28_00_UTC) 2024-01-11T09:43:19.337 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_29_00_UTC 2024-01-11T09:43:19.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:19.338 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_29_00_UTC) 2024-01-11T09:43:19.392 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_30_00_UTC 2024-01-11T09:43:19.392 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:19.392 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_30_00_UTC) 2024-01-11T09:43:19.446 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_31_00_UTC 2024-01-11T09:43:19.446 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:19.446 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_31_00_UTC) 2024-01-11T09:43:19.501 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_32_00_UTC 2024-01-11T09:43:19.501 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:19.501 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_32_00_UTC) 2024-01-11T09:43:19.556 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_33_00_UTC 2024-01-11T09:43:19.556 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:19.556 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_33_00_UTC) 2024-01-11T09:43:19.610 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_34_00_UTC 2024-01-11T09:43:19.610 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:19.610 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_34_00_UTC) 2024-01-11T09:43:19.665 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_35_00_UTC 2024-01-11T09:43:19.666 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:19.666 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_35_00_UTC) 2024-01-11T09:43:19.720 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_36_00_UTC 2024-01-11T09:43:19.721 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:19.721 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_36_00_UTC) 2024-01-11T09:43:19.776 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_37_00_UTC 2024-01-11T09:43:19.776 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:19.777 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_37_00_UTC) 2024-01-11T09:43:19.831 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_38_00_UTC 2024-01-11T09:43:19.831 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:19.831 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_38_00_UTC) 2024-01-11T09:43:19.885 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_39_00_UTC 2024-01-11T09:43:19.885 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:19.886 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_39_00_UTC) 2024-01-11T09:43:19.940 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_40_00_UTC 2024-01-11T09:43:19.940 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:19.940 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_40_00_UTC) 2024-01-11T09:43:19.994 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_41_00_UTC 2024-01-11T09:43:19.995 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:19.995 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_41_00_UTC) 2024-01-11T09:43:20.050 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_42_00_UTC 2024-01-11T09:43:20.050 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:20.050 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_42_00_UTC) 2024-01-11T09:43:20.104 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_43_00_UTC 2024-01-11T09:43:20.104 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:20.104 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-01-11-09_43_00_UTC) 2024-01-11T09:43:20.158 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:20.158 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2024-01-11T09:43:20.208 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-11T09:43:20.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.418+0000 7fdd511ff700 1 Processor -- start 2024-01-11T09:43:20.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.418+0000 7fdd511ff700 1 -- start start 2024-01-11T09:43:20.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.419+0000 7fdd511ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdd4c142dc0 0x7fdd4c0cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:20.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.419+0000 7fdd511ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdd4c143190 0x7fdd4c0cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:20.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.419+0000 7fdd511ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd4c15d320 con 0x7fdd4c0cc020 2024-01-11T09:43:20.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.419+0000 7fdd511ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd4c15df30 con 0x7fdd4c142dc0 2024-01-11T09:43:20.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.419+0000 7fdd511ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c15d590 con 0x7fdd4c143190 2024-01-11T09:43:20.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.420+0000 7fdd4ad9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdd4c142dc0 0x7fdd4c0cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.420+0000 7fdd4a59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdd4c143190 0x7fdd4c0cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.420+0000 7fdd4b59e700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdd4c0cc020 0x7fdd4c0cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:60580/0 (socket says 172.21.15.27:60580) 2024-01-11T09:43:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.420+0000 7fdd4b59e700 1 -- 172.21.15.27:0/2806362595 learned_addr learned my addr 172.21.15.27:0/2806362595 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.420+0000 7fdd49d9b700 1 -- 172.21.15.27:0/2806362595 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058962366 0 0) 0x7fdd4c15d320 con 0x7fdd4c0cc020 2024-01-11T09:43:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.420+0000 7fdd49d9b700 1 -- 172.21.15.27:0/2806362595 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd34003570 con 0x7fdd4c0cc020 2024-01-11T09:43:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.420+0000 7fdd4a59c700 1 -- 172.21.15.27:0/2806362595 >> v2:172.21.15.77:3300/0 conn(0x7fdd4c142dc0 msgr2=0x7fdd4c0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.420+0000 7fdd4a59c700 1 --2- 172.21.15.27:0/2806362595 >> v2:172.21.15.77:3300/0 conn(0x7fdd4c142dc0 0x7fdd4c0cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.420+0000 7fdd4a59c700 1 -- 172.21.15.27:0/2806362595 >> v1:172.21.15.77:6789/0 conn(0x7fdd4c0cc020 legacy=0x7fdd4c0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.420+0000 7fdd4a59c700 1 -- 172.21.15.27:0/2806362595 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4c15d870 con 0x7fdd4c143190 2024-01-11T09:43:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.420+0000 7fdd4ad9d700 1 --2- 172.21.15.27:0/2806362595 >> v2:172.21.15.77:3300/0 conn(0x7fdd4c142dc0 0x7fdd4c0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:43:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.421+0000 7fdd4a59c700 1 --2- 172.21.15.27:0/2806362595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdd4c143190 0x7fdd4c0cb2e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdd400098e0 tx=0x7fdd400091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21b04e92f52c53b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.421+0000 7fdd49d9b700 1 -- 172.21.15.27:0/2806362595 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd40011630 con 0x7fdd4c143190 2024-01-11T09:43:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.421+0000 7fdd49d9b700 1 -- 172.21.15.27:0/2806362595 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd40011790 con 0x7fdd4c143190 2024-01-11T09:43:20.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.421+0000 7fdd49d9b700 1 -- 172.21.15.27:0/2806362595 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd40011b30 con 0x7fdd4c143190 2024-01-11T09:43:20.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.421+0000 7fdd511ff700 1 -- 172.21.15.27:0/2806362595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdd4c143190 msgr2=0x7fdd4c0cb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:20.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.421+0000 7fdd511ff700 1 --2- 172.21.15.27:0/2806362595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdd4c143190 0x7fdd4c0cb2e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdd400098e0 tx=0x7fdd400091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:20.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.422+0000 7fdd511ff700 1 -- 172.21.15.27:0/2806362595 shutdown_connections 2024-01-11T09:43:20.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.422+0000 7fdd511ff700 1 --2- 172.21.15.27:0/2806362595 >> v2:172.21.15.77:3300/0 conn(0x7fdd4c142dc0 0x7fdd4c0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:20.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.422+0000 7fdd511ff700 1 --2- 172.21.15.27:0/2806362595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdd4c143190 0x7fdd4c0cb2e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:20.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.422+0000 7fdd511ff700 1 -- 172.21.15.27:0/2806362595 >> 172.21.15.27:0/2806362595 conn(0x7fdd4c063ca0 msgr2=0x7fdd4c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:20.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.422+0000 7fdd511ff700 1 -- 172.21.15.27:0/2806362595 shutdown_connections 2024-01-11T09:43:20.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.422+0000 7fdd511ff700 1 -- 172.21.15.27:0/2806362595 wait complete. 2024-01-11T09:43:20.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.423+0000 7fdd511ff700 1 Processor -- start 2024-01-11T09:43:20.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.423+0000 7fdd511ff700 1 -- start start 2024-01-11T09:43:20.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.424+0000 7fdd511ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd4c0cc020 0x7fdd4c140810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:20.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.424+0000 7fdd511ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd4c142dc0 0x7fdd4c16a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:20.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.424+0000 7fdd511ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdd4c143190 0x7fdd4c0b0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:20.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.424+0000 7fdd511ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c141a90 con 0x7fdd4c142dc0 2024-01-11T09:43:20.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.424+0000 7fdd511ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c16ae10 con 0x7fdd4c0cc020 2024-01-11T09:43:20.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.424+0000 7fdd511ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c0b0560 con 0x7fdd4c143190 2024-01-11T09:43:20.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.425+0000 7fdd4a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd4c142dc0 0x7fdd4c16a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:20.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.425+0000 7fdd4a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd4c142dc0 0x7fdd4c16a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35256/0 (socket says 172.21.15.27:35256) 2024-01-11T09:43:20.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.425+0000 7fdd4a59c700 1 -- 172.21.15.27:0/2700315975 learned_addr learned my addr 172.21.15.27:0/2700315975 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:20.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.425+0000 7fdd4ad9d700 1 --2- 172.21.15.27:0/2700315975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd4c0cc020 0x7fdd4c140810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:20.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.425+0000 7fdd4b59e700 1 --2- 172.21.15.27:0/2700315975 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdd4c143190 0x7fdd4c0b0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:20.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.425+0000 7fdd4a59c700 1 -- 172.21.15.27:0/2700315975 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdd4c143190 msgr2=0x7fdd4c0b0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:20.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.425+0000 7fdd4a59c700 1 --2- 172.21.15.27:0/2700315975 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdd4c143190 0x7fdd4c0b0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:20.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.425+0000 7fdd4a59c700 1 -- 172.21.15.27:0/2700315975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd4c0cc020 msgr2=0x7fdd4c140810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:20.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.425+0000 7fdd4a59c700 1 --2- 172.21.15.27:0/2700315975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd4c0cc020 0x7fdd4c140810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:20.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.425+0000 7fdd4a59c700 1 -- 172.21.15.27:0/2700315975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c009420 con 0x7fdd4c142dc0 2024-01-11T09:43:20.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.425+0000 7fdd4a59c700 1 --2- 172.21.15.27:0/2700315975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd4c142dc0 0x7fdd4c16a8d0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fdd40009ed0 tx=0x7fdd40017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:20.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.426+0000 7fdd32ffd700 1 -- 172.21.15.27:0/2700315975 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd40011900 con 0x7fdd4c142dc0 2024-01-11T09:43:20.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.426+0000 7fdd32ffd700 1 -- 172.21.15.27:0/2700315975 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd40011a60 con 0x7fdd4c142dc0 2024-01-11T09:43:20.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.426+0000 7fdd32ffd700 1 -- 172.21.15.27:0/2700315975 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd40023620 con 0x7fdd4c142dc0 2024-01-11T09:43:20.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.426+0000 7fdd511ff700 1 -- 172.21.15.27:0/2700315975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd40009420 con 0x7fdd4c142dc0 2024-01-11T09:43:20.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.426+0000 7fdd511ff700 1 -- 172.21.15.27:0/2700315975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd4c141160 con 0x7fdd4c142dc0 2024-01-11T09:43:20.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.427+0000 7fdd511ff700 1 -- 172.21.15.27:0/2700315975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4c141c40 con 0x7fdd4c142dc0 2024-01-11T09:43:20.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.428+0000 7fdd32ffd700 1 -- 172.21.15.27:0/2700315975 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 112) v1 ==== 84873+0+0 (secure 0 0 0) 0x7fdd4002c3b0 con 0x7fdd4c142dc0 2024-01-11T09:43:20.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.429+0000 7fdd32ffd700 1 --2- 172.21.15.27:0/2700315975 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fdd380670a0 0x7fdd38069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:20.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.429+0000 7fdd32ffd700 1 -- 172.21.15.27:0/2700315975 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6277+0+0 (secure 0 0 0) 0x7fdd40085ca0 con 0x7fdd4c142dc0 2024-01-11T09:43:20.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.430+0000 7fdd4ad9d700 1 --2- 172.21.15.27:0/2700315975 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fdd380670a0 0x7fdd38069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:20.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.430+0000 7fdd4ad9d700 1 --2- 172.21.15.27:0/2700315975 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fdd380670a0 0x7fdd38069550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009f40 tx=0x7fdd3c0091a0 comp rx=0 tx=0).ready entity=mgr.9457 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:20.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.433+0000 7fdd32ffd700 1 -- 172.21.15.27:0/2700315975 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdd4c141c40 con 0x7fdd4c142dc0 2024-01-11T09:43:20.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.694+0000 7fdd511ff700 1 -- 172.21.15.27:0/2700315975 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fdd4c141c40 con 0x7fdd4c142dc0 2024-01-11T09:43:20.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.974+0000 7fdd32ffd700 1 -- 172.21.15.27:0/2700315975 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 113) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fdd40052f70 con 0x7fdd4c142dc0 2024-01-11T09:43:20.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.976+0000 7fdd32ffd700 1 -- 172.21.15.27:0/2700315975 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v113) v1 ==== 93+0+0 (secure 0 0 0) 0x7fdd400561c0 con 0x7fdd4c142dc0 2024-01-11T09:43:20.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.980+0000 7fdd511ff700 1 -- 172.21.15.27:0/2700315975 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fdd380670a0 msgr2=0x7fdd38069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:20.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.980+0000 7fdd511ff700 1 --2- 172.21.15.27:0/2700315975 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fdd380670a0 0x7fdd38069550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009f40 tx=0x7fdd3c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:20.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.980+0000 7fdd511ff700 1 -- 172.21.15.27:0/2700315975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd4c142dc0 msgr2=0x7fdd4c16a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:20.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.980+0000 7fdd511ff700 1 --2- 172.21.15.27:0/2700315975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd4c142dc0 0x7fdd4c16a8d0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fdd40009ed0 tx=0x7fdd40017000 comp rx=0 tx=0).stop 2024-01-11T09:43:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.981+0000 7fdd511ff700 1 -- 172.21.15.27:0/2700315975 shutdown_connections 2024-01-11T09:43:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.981+0000 7fdd511ff700 1 --2- 172.21.15.27:0/2700315975 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdd4c143190 0x7fdd4c0b0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.981+0000 7fdd511ff700 1 --2- 172.21.15.27:0/2700315975 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fdd380670a0 0x7fdd38069550 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.981+0000 7fdd511ff700 1 --2- 172.21.15.27:0/2700315975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd4c0cc020 0x7fdd4c140810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.981+0000 7fdd511ff700 1 --2- 172.21.15.27:0/2700315975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd4c142dc0 0x7fdd4c16a8d0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.981+0000 7fdd511ff700 1 -- 172.21.15.27:0/2700315975 >> 172.21.15.27:0/2700315975 conn(0x7fdd4c063ca0 msgr2=0x7fdd4c13ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.981+0000 7fdd511ff700 1 -- 172.21.15.27:0/2700315975 shutdown_connections 2024-01-11T09:43:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:20.982+0000 7fdd511ff700 1 -- 172.21.15.27:0/2700315975 wait complete. 2024-01-11T09:43:21.009 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T09:43:21.010 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:43:21.134 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:43:21.278 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:21.286+0000 7fcf2074e1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:43:21.284 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:21.292+0000 7f7e4dd341c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:43:21.292 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T09:43:21.292 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-11T09:43:21.292 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:21.292 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-01-11T09:43:21.373 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T09:43:21.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:43:21.374 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-11T09:43:21.447 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:43:21.458 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:21.466+0000 7fcf2074e1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:43:21.483 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:21.491+0000 7f7e4dd341c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:43:21.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.671+0000 7fb0cafff700 1 Processor -- start 2024-01-11T09:43:21.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.672+0000 7fb0cafff700 1 -- start start 2024-01-11T09:43:21.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.673+0000 7fb0cafff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c40b03e0 0x7fb0c40b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:21.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.673+0000 7fb0cafff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0c40b29c0 0x7fb0c4159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:21.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.673+0000 7fb0cafff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c40b2e00 con 0x7fb0c40b07b0 2024-01-11T09:43:21.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.673+0000 7fb0cafff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c40cd730 con 0x7fb0c40b29c0 2024-01-11T09:43:21.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.673+0000 7fb0cafff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c40ccb20 con 0x7fb0c40b03e0 2024-01-11T09:43:21.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.673+0000 7fb0c3fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb0c40b07b0 0x7fb0c4159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52300/0 (socket says 172.21.15.27:52300) 2024-01-11T09:43:21.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.673+0000 7fb0c3fff700 1 -- 172.21.15.27:0/1013324166 learned_addr learned my addr 172.21.15.27:0/1013324166 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:21.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.673+0000 7fb0c959c700 1 --2- 172.21.15.27:0/1013324166 >> v2:172.21.15.27:3300/0 conn(0x7fb0c40b29c0 0x7fb0c4159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:21.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.673+0000 7fb0c8d9b700 1 --2- 172.21.15.27:0/1013324166 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c40b03e0 0x7fb0c40b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:21.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.674+0000 7fb0c37fe700 1 -- 172.21.15.27:0/1013324166 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684861278 0 0) 0x7fb0b0001110 con 0x7fb0c40b07b0 2024-01-11T09:43:21.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.674+0000 7fb0c37fe700 1 -- 172.21.15.27:0/1013324166 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0b4003570 con 0x7fb0c40b07b0 2024-01-11T09:43:21.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.674+0000 7fb0c8d9b700 1 -- 172.21.15.27:0/1013324166 >> v2:172.21.15.27:3300/0 conn(0x7fb0c40b29c0 msgr2=0x7fb0c4159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.674+0000 7fb0c8d9b700 1 --2- 172.21.15.27:0/1013324166 >> v2:172.21.15.27:3300/0 conn(0x7fb0c40b29c0 0x7fb0c4159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.674+0000 7fb0c8d9b700 1 -- 172.21.15.27:0/1013324166 >> v1:172.21.15.27:6789/0 conn(0x7fb0c40b07b0 legacy=0x7fb0c4159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.674+0000 7fb0c8d9b700 1 -- 172.21.15.27:0/1013324166 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c40cce00 con 0x7fb0c40b03e0 2024-01-11T09:43:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.674+0000 7fb0c8d9b700 1 --2- 172.21.15.27:0/1013324166 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c40b03e0 0x7fb0c40b2480 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80098e0 tx=0x7fb0b80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65c7b6e41e533a8d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.675+0000 7fb0c37fe700 1 -- 172.21.15.27:0/1013324166 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8011630 con 0x7fb0c40b03e0 2024-01-11T09:43:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.675+0000 7fb0c37fe700 1 -- 172.21.15.27:0/1013324166 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b8011790 con 0x7fb0c40b03e0 2024-01-11T09:43:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.675+0000 7fb0c37fe700 1 -- 172.21.15.27:0/1013324166 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8011b30 con 0x7fb0c40b03e0 2024-01-11T09:43:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.675+0000 7fb0cafff700 1 -- 172.21.15.27:0/1013324166 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c40b03e0 msgr2=0x7fb0c40b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.675+0000 7fb0cafff700 1 --2- 172.21.15.27:0/1013324166 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c40b03e0 0x7fb0c40b2480 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80098e0 tx=0x7fb0b80091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.675+0000 7fb0cafff700 1 -- 172.21.15.27:0/1013324166 shutdown_connections 2024-01-11T09:43:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.675+0000 7fb0cafff700 1 --2- 172.21.15.27:0/1013324166 >> v2:172.21.15.27:3300/0 conn(0x7fb0c40b29c0 0x7fb0c4159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.675+0000 7fb0cafff700 1 --2- 172.21.15.27:0/1013324166 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c40b03e0 0x7fb0c40b2480 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.675+0000 7fb0cafff700 1 -- 172.21.15.27:0/1013324166 >> 172.21.15.27:0/1013324166 conn(0x7fb0c406ed40 msgr2=0x7fb0c406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:21.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.675+0000 7fb0cafff700 1 -- 172.21.15.27:0/1013324166 shutdown_connections 2024-01-11T09:43:21.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.676+0000 7fb0cafff700 1 -- 172.21.15.27:0/1013324166 wait complete. 2024-01-11T09:43:21.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.677+0000 7fb0cafff700 1 Processor -- start 2024-01-11T09:43:21.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.678+0000 7fb0cafff700 1 -- start start 2024-01-11T09:43:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.679+0000 7fb0cafff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c40b03e0 0x7fb0c4158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.679+0000 7fb0cafff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c40b07b0 0x7fb0c40c7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.679+0000 7fb0cafff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c40b29c0 0x7fb0c40b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.679+0000 7fb0cafff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c40a6f40 con 0x7fb0c40b07b0 2024-01-11T09:43:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.679+0000 7fb0cafff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c40aee10 con 0x7fb0c40b03e0 2024-01-11T09:43:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.679+0000 7fb0cafff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c40c92b0 con 0x7fb0c40b29c0 2024-01-11T09:43:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.679+0000 7fb0c3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c40b07b0 0x7fb0c40c7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.679+0000 7fb0c8d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c40b03e0 0x7fb0c4158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.679+0000 7fb0c959c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c40b29c0 0x7fb0c40b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.679+0000 7fb0c3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c40b07b0 0x7fb0c40c7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35290/0 (socket says 172.21.15.27:35290) 2024-01-11T09:43:21.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.679+0000 7fb0c3fff700 1 -- 172.21.15.27:0/773027165 learned_addr learned my addr 172.21.15.27:0/773027165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:21.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.679+0000 7fb0c8d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c40b03e0 0x7fb0c4158bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:43824/0 (socket says 172.21.15.27:43824) 2024-01-11T09:43:21.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.679+0000 7fb0c959c700 1 -- 172.21.15.27:0/773027165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c40b03e0 msgr2=0x7fb0c4158bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:21.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.679+0000 7fb0c959c700 1 --2- 172.21.15.27:0/773027165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c40b03e0 0x7fb0c4158bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:21.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.679+0000 7fb0c959c700 1 -- 172.21.15.27:0/773027165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c40b07b0 msgr2=0x7fb0c40c7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:21.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.679+0000 7fb0c959c700 1 --2- 172.21.15.27:0/773027165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c40b07b0 0x7fb0c40c7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:21.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.679+0000 7fb0c959c700 1 -- 172.21.15.27:0/773027165 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b8009420 con 0x7fb0c40b29c0 2024-01-11T09:43:21.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.680+0000 7fb0c959c700 1 --2- 172.21.15.27:0/773027165 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c40b29c0 0x7fb0c40b4bc0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00ecc0 tx=0x7fb0bc00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:21.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.680+0000 7fb0c0ff9700 1 -- 172.21.15.27:0/773027165 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc00c8c0 con 0x7fb0c40b29c0 2024-01-11T09:43:21.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.680+0000 7fb0c0ff9700 1 -- 172.21.15.27:0/773027165 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0bc00f040 con 0x7fb0c40b29c0 2024-01-11T09:43:21.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.680+0000 7fb0cafff700 1 -- 172.21.15.27:0/773027165 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c40af080 con 0x7fb0c40b29c0 2024-01-11T09:43:21.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.680+0000 7fb0c0ff9700 1 -- 172.21.15.27:0/773027165 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc00cc90 con 0x7fb0c40b29c0 2024-01-11T09:43:21.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.680+0000 7fb0cafff700 1 -- 172.21.15.27:0/773027165 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c40a7120 con 0x7fb0c40b29c0 2024-01-11T09:43:21.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.681+0000 7fb0cafff700 1 -- 172.21.15.27:0/773027165 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c40718d0 con 0x7fb0c40b29c0 2024-01-11T09:43:21.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.683+0000 7fb0c0ff9700 1 -- 172.21.15.27:0/773027165 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 113) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fb0bc014690 con 0x7fb0c40b29c0 2024-01-11T09:43:21.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.684+0000 7fb0c0ff9700 1 --2- 172.21.15.27:0/773027165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0a8067120 0x7fb0a80695d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:21.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.684+0000 7fb0c0ff9700 1 -- 172.21.15.27:0/773027165 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6365+0+0 (secure 0 0 0) 0x7fb0bc0853a0 con 0x7fb0c40b29c0 2024-01-11T09:43:21.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.684+0000 7fb0c8d9b700 1 -- 172.21.15.27:0/773027165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0a8067120 msgr2=0x7fb0a80695d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:21.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.684+0000 7fb0c8d9b700 1 --2- 172.21.15.27:0/773027165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0a8067120 0x7fb0a80695d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:43:21.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.689+0000 7fb0c0ff9700 1 -- 172.21.15.27:0/773027165 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb0bc0521e0 con 0x7fb0c40b29c0 2024-01-11T09:43:21.770 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:21.779+0000 7fcf2074e1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:43:21.825 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:21.834+0000 7f7e4dd341c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:43:21.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.885+0000 7fb0c8d9b700 1 -- 172.21.15.27:0/773027165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0a8067120 msgr2=0x7fb0a80695d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:21.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.885+0000 7fb0c8d9b700 1 --2- 172.21.15.27:0/773027165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0a8067120 0x7fb0a80695d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:43:21.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.939+0000 7fb0cafff700 1 -- 172.21.15.27:0/773027165 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0c40cd0e0 con 0x7fb0c40b29c0 2024-01-11T09:43:21.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.940+0000 7fb0c0ff9700 1 -- 172.21.15.27:0/773027165 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4601 (secure 0 0 0) 0x7fb0bc055800 con 0x7fb0c40b29c0 2024-01-11T09:43:21.932 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:43:21.932 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8015,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1003297202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1003297202},{"type":"v1","addr":"172.21.15.77:6834","nonce":1003297202}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":8018,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3321059147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":3321059147},{"type":"v1","addr":"172.21.15.77:6835","nonce":3321059147}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":9202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2606639222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2606639222},{"type":"v1","addr":"172.21.15.27:6835","nonce":2606639222}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:27:21.470488+0000","modified":"2024-01-11T09:27:24.992917+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9564":{"gid":9564,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/3897592862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3897592862},{"type":"v1","addr":"172.21.15.27:6834","nonce":3897592862}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2024-01-11T09:43:21.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.942+0000 7fb0cafff700 1 -- 172.21.15.27:0/773027165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0a8067120 msgr2=0x7fb0a80695d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:43:21.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.942+0000 7fb0cafff700 1 --2- 172.21.15.27:0/773027165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0a8067120 0x7fb0a80695d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:21.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.942+0000 7fb0cafff700 1 -- 172.21.15.27:0/773027165 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c40b29c0 msgr2=0x7fb0c40b4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:21.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.942+0000 7fb0cafff700 1 --2- 172.21.15.27:0/773027165 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c40b29c0 0x7fb0c40b4bc0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00ecc0 tx=0x7fb0bc00b040 comp rx=0 tx=0).stop 2024-01-11T09:43:21.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.943+0000 7fb0cafff700 1 -- 172.21.15.27:0/773027165 shutdown_connections 2024-01-11T09:43:21.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.943+0000 7fb0cafff700 1 --2- 172.21.15.27:0/773027165 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c40b29c0 0x7fb0c40b4bc0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:21.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.943+0000 7fb0cafff700 1 --2- 172.21.15.27:0/773027165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0a8067120 0x7fb0a80695d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:21.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.943+0000 7fb0cafff700 1 --2- 172.21.15.27:0/773027165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c40b03e0 0x7fb0c4158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:21.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.943+0000 7fb0cafff700 1 --2- 172.21.15.27:0/773027165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c40b07b0 0x7fb0c40c7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:21.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.943+0000 7fb0cafff700 1 -- 172.21.15.27:0/773027165 >> 172.21.15.27:0/773027165 conn(0x7fb0c406ed40 msgr2=0x7fb0c40b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:21.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.943+0000 7fb0cafff700 1 -- 172.21.15.27:0/773027165 shutdown_connections 2024-01-11T09:43:21.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:21.943+0000 7fb0cafff700 1 -- 172.21.15.27:0/773027165 wait complete. 2024-01-11T09:43:21.936 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 94 2024-01-11T09:43:21.955 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:43:22.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.219+0000 7f8aa39ff700 1 Processor -- start 2024-01-11T09:43:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.220+0000 7f8aa39ff700 1 -- start start 2024-01-11T09:43:22.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.222+0000 7f8aa39ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8a9c0ca3e0 0x7f8a9c0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:22.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.222+0000 7f8aa39ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a9c159e80 0x7f8a9c15dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.222+0000 7f8aa39ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c15d640 con 0x7f8a9c0ca7b0 2024-01-11T09:43:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.222+0000 7f8aa39ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a9c15ae70 con 0x7f8a9c0ca3e0 2024-01-11T09:43:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.222+0000 7f8aa39ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c0cc4f0 con 0x7f8a9c159e80 2024-01-11T09:43:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.222+0000 7f8aa0f9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8a9c0ca7b0 0x7f8a9c0c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52314/0 (socket says 172.21.15.27:52314) 2024-01-11T09:43:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.222+0000 7f8aa0f9a700 1 -- 172.21.15.27:0/4281552903 learned_addr learned my addr 172.21.15.27:0/4281552903 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.222+0000 7f8aa179b700 1 --2- 172.21.15.27:0/4281552903 >> v2:172.21.15.77:3300/0 conn(0x7f8a9c0ca3e0 0x7f8a9c0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.222+0000 7f8aa1f9c700 1 --2- 172.21.15.27:0/4281552903 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a9c159e80 0x7f8a9c15dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.222+0000 7f8a93fff700 1 -- 172.21.15.27:0/4281552903 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865280905 0 0) 0x7f8a9c15d640 con 0x7f8a9c0ca7b0 2024-01-11T09:43:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.222+0000 7f8a93fff700 1 -- 172.21.15.27:0/4281552903 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a88003570 con 0x7f8a9c0ca7b0 2024-01-11T09:43:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.223+0000 7f8aa179b700 1 -- 172.21.15.27:0/4281552903 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a9c159e80 msgr2=0x7f8a9c15dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.223+0000 7f8aa179b700 1 --2- 172.21.15.27:0/4281552903 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a9c159e80 0x7f8a9c15dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.223+0000 7f8aa179b700 1 -- 172.21.15.27:0/4281552903 >> v1:172.21.15.27:6789/0 conn(0x7f8a9c0ca7b0 legacy=0x7f8a9c0c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.223+0000 7f8aa179b700 1 -- 172.21.15.27:0/4281552903 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c15a5d0 con 0x7f8a9c0ca3e0 2024-01-11T09:43:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.223+0000 7f8aa1f9c700 1 --2- 172.21.15.27:0/4281552903 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a9c159e80 0x7f8a9c15dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:43:22.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.223+0000 7f8aa179b700 1 --2- 172.21.15.27:0/4281552903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0ca3e0 0x7f8a9c0cca20 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f8a98009940 tx=0x7f8a980091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7cd72f2a3391181 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:22.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.223+0000 7f8a93fff700 1 -- 172.21.15.27:0/4281552903 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a98011630 con 0x7f8a9c0ca3e0 2024-01-11T09:43:22.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.223+0000 7f8a93fff700 1 -- 172.21.15.27:0/4281552903 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a98011790 con 0x7f8a9c0ca3e0 2024-01-11T09:43:22.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.223+0000 7f8a93fff700 1 -- 172.21.15.27:0/4281552903 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a98011b50 con 0x7f8a9c0ca3e0 2024-01-11T09:43:22.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.224+0000 7f8aa39ff700 1 -- 172.21.15.27:0/4281552903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0ca3e0 msgr2=0x7f8a9c0cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:22.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.224+0000 7f8aa39ff700 1 --2- 172.21.15.27:0/4281552903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0ca3e0 0x7f8a9c0cca20 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f8a98009940 tx=0x7f8a980091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.224+0000 7f8aa39ff700 1 -- 172.21.15.27:0/4281552903 shutdown_connections 2024-01-11T09:43:22.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.224+0000 7f8aa39ff700 1 --2- 172.21.15.27:0/4281552903 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a9c159e80 0x7f8a9c15dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.224+0000 7f8aa39ff700 1 --2- 172.21.15.27:0/4281552903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c0ca3e0 0x7f8a9c0cca20 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.224+0000 7f8aa39ff700 1 -- 172.21.15.27:0/4281552903 >> 172.21.15.27:0/4281552903 conn(0x7f8a9c0ce3e0 msgr2=0x7f8a9c0ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:22.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.224+0000 7f8aa39ff700 1 -- 172.21.15.27:0/4281552903 shutdown_connections 2024-01-11T09:43:22.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.224+0000 7f8aa39ff700 1 -- 172.21.15.27:0/4281552903 wait complete. 2024-01-11T09:43:22.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.226+0000 7f8aa39ff700 1 Processor -- start 2024-01-11T09:43:22.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.227+0000 7f8aa39ff700 1 -- start start 2024-01-11T09:43:22.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.228+0000 7f8aa39ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a9c0ca3e0 0x7f8a9c147fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:22.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.228+0000 7f8aa39ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a9c0ca7b0 0x7f8a9c150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:22.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.228+0000 7f8aa39ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c159e80 0x7f8a9c1506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:22.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.228+0000 7f8aa39ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c1484e0 con 0x7f8a9c0ca7b0 2024-01-11T09:43:22.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.228+0000 7f8aa39ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c0b1f10 con 0x7f8a9c159e80 2024-01-11T09:43:22.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.228+0000 7f8aa39ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c0b1300 con 0x7f8a9c0ca3e0 2024-01-11T09:43:22.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.228+0000 7f8aa0f9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a9c0ca7b0 0x7f8a9c150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:22.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.228+0000 7f8aa179b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a9c0ca3e0 0x7f8a9c147fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:22.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.228+0000 7f8aa0f9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a9c0ca7b0 0x7f8a9c150160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35298/0 (socket says 172.21.15.27:35298) 2024-01-11T09:43:22.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.228+0000 7f8aa0f9a700 1 -- 172.21.15.27:0/1453223005 learned_addr learned my addr 172.21.15.27:0/1453223005 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:22.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.228+0000 7f8aa1f9c700 1 --2- 172.21.15.27:0/1453223005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c159e80 0x7f8a9c1506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:22.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.229+0000 7f8aa179b700 1 -- 172.21.15.27:0/1453223005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c159e80 msgr2=0x7f8a9c1506a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:22.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.229+0000 7f8aa179b700 1 --2- 172.21.15.27:0/1453223005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c159e80 0x7f8a9c1506a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.229+0000 7f8aa179b700 1 -- 172.21.15.27:0/1453223005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a9c0ca7b0 msgr2=0x7f8a9c150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:22.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.229+0000 7f8aa179b700 1 --2- 172.21.15.27:0/1453223005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a9c0ca7b0 0x7f8a9c150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.229+0000 7f8aa179b700 1 -- 172.21.15.27:0/1453223005 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a94009a20 con 0x7f8a9c0ca3e0 2024-01-11T09:43:22.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.229+0000 7f8aa1f9c700 1 --2- 172.21.15.27:0/1453223005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c159e80 0x7f8a9c1506a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:22.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.229+0000 7f8aa0f9a700 1 --2- 172.21.15.27:0/1453223005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a9c0ca7b0 0x7f8a9c150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:43:22.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.229+0000 7f8aa179b700 1 --2- 172.21.15.27:0/1453223005 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a9c0ca3e0 0x7f8a9c147fa0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8a98016000 tx=0x7f8a98017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.229+0000 7f8a917fa700 1 -- 172.21.15.27:0/1453223005 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9801f070 con 0x7f8a9c0ca3e0 2024-01-11T09:43:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.229+0000 7f8a917fa700 1 -- 172.21.15.27:0/1453223005 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a98000c90 con 0x7f8a9c0ca3e0 2024-01-11T09:43:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.229+0000 7f8aa39ff700 1 -- 172.21.15.27:0/1453223005 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a98009420 con 0x7f8a9c0ca3e0 2024-01-11T09:43:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.229+0000 7f8a917fa700 1 -- 172.21.15.27:0/1453223005 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a98011b50 con 0x7f8a9c0ca3e0 2024-01-11T09:43:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.230+0000 7f8aa39ff700 1 -- 172.21.15.27:0/1453223005 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9c0b18c0 con 0x7f8a9c0ca3e0 2024-01-11T09:43:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.231+0000 7f8aa39ff700 1 -- 172.21.15.27:0/1453223005 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a9c169cc0 con 0x7f8a9c0ca3e0 2024-01-11T09:43:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.231+0000 7f8a917fa700 1 -- 172.21.15.27:0/1453223005 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 113) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f8a98023070 con 0x7f8a9c0ca3e0 2024-01-11T09:43:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.232+0000 7f8a917fa700 1 --2- 172.21.15.27:0/1453223005 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8a80067050 0x7f8a80069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.232+0000 7f8a917fa700 1 -- 172.21.15.27:0/1453223005 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6349+0+0 (secure 0 0 0) 0x7f8a98085c30 con 0x7f8a9c0ca3e0 2024-01-11T09:43:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.232+0000 7f8aa0f9a700 1 -- 172.21.15.27:0/1453223005 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8a80067050 msgr2=0x7f8a80069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.232+0000 7f8aa0f9a700 1 --2- 172.21.15.27:0/1453223005 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8a80067050 0x7f8a80069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:43:22.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.236+0000 7f8a917fa700 1 -- 172.21.15.27:0/1453223005 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8a98052a70 con 0x7f8a9c0ca3e0 2024-01-11T09:43:22.315 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:22.324+0000 7fcf2074e1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:43:22.411 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:22.419+0000 7f7e4dd341c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:43:22.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.434+0000 7f8aa0f9a700 1 -- 172.21.15.27:0/1453223005 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8a80067050 msgr2=0x7f8a80069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:22.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.434+0000 7f8aa0f9a700 1 --2- 172.21.15.27:0/1453223005 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8a80067050 0x7f8a80069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:43:22.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.440+0000 7f8aa39ff700 1 -- 172.21.15.27:0/1453223005 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a9c0718d0 con 0x7f8a9c0ca3e0 2024-01-11T09:43:22.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.441+0000 7f8a917fa700 1 -- 172.21.15.27:0/1453223005 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) v1 ==== 74+0+16282 (secure 0 0 0) 0x7f8a9802a020 con 0x7f8a9c0ca3e0 2024-01-11T09:43:22.433 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:43:22.433 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":185,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:43:21.972966+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:27:19.009515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-01-11T09:27:20.468964+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":185,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1160908778":"2024-01-12T09:38:23.806141+0000","172.21.15.77:0/3404938947":"2024-01-12T09:38:23.806141+0000","172.21.15.77:0/2730921345":"2024-01-12T09:38:23.806141+0000","172.21.15.77:0/2548321690":"2024-01-12T09:38:23.806141+0000","172.21.15.77:0/3966666871":"2024-01-12T09:38:23.806141+0000","172.21.15.77:6837/41381":"2024-01-12T09:27:37.837169+0000","172.21.15.77:6836/41381":"2024-01-12T09:27:37.837169+0000","172.21.15.77:0/1337020578":"2024-01-12T09:27:37.837169+0000","172.21.15.77:0/2253497474":"2024-01-12T09:27:37.837169+0000","172.21.15.77:0/2186545586":"2024-01-12T09:38:23.806141+0000","172.21.15.77:0/2706846520":"2024-01-12T09:27:37.837169+0000","172.21.15.77:0/2650034185":"2024-01-12T09:27:37.837169+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":23,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:43:22.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.443+0000 7f8aa39ff700 1 -- 172.21.15.27:0/1453223005 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8a80067050 msgr2=0x7f8a80069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:43:22.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.443+0000 7f8aa39ff700 1 --2- 172.21.15.27:0/1453223005 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8a80067050 0x7f8a80069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.443+0000 7f8aa39ff700 1 -- 172.21.15.27:0/1453223005 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a9c0ca3e0 msgr2=0x7f8a9c147fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.443+0000 7f8aa39ff700 1 --2- 172.21.15.27:0/1453223005 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a9c0ca3e0 0x7f8a9c147fa0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8a98016000 tx=0x7f8a98017000 comp rx=0 tx=0).stop 2024-01-11T09:43:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.443+0000 7f8aa39ff700 1 -- 172.21.15.27:0/1453223005 shutdown_connections 2024-01-11T09:43:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.443+0000 7f8aa39ff700 1 --2- 172.21.15.27:0/1453223005 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8a80067050 0x7f8a80069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.443+0000 7f8aa39ff700 1 --2- 172.21.15.27:0/1453223005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a9c159e80 0x7f8a9c1506a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.443+0000 7f8aa39ff700 1 --2- 172.21.15.27:0/1453223005 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a9c0ca3e0 0x7f8a9c147fa0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.443+0000 7f8aa39ff700 1 --2- 172.21.15.27:0/1453223005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a9c0ca7b0 0x7f8a9c150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.443+0000 7f8aa39ff700 1 -- 172.21.15.27:0/1453223005 >> 172.21.15.27:0/1453223005 conn(0x7f8a9c0ce3e0 msgr2=0x7f8a9c0696f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.443+0000 7f8aa39ff700 1 -- 172.21.15.27:0/1453223005 shutdown_connections 2024-01-11T09:43:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.443+0000 7f8aa39ff700 1 -- 172.21.15.27:0/1453223005 wait complete. 2024-01-11T09:43:22.456 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T09:27:19.009515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '180', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T09:27:20.468964+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '185', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 185, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T09:43:22.457 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T09:43:22.485 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:22.494+0000 7fcf2074e1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:43:22.622 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:22.630+0000 7f7e4dd341c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:43:22.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.696+0000 7f331f3ff700 1 Processor -- start 2024-01-11T09:43:22.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.696+0000 7f331f3ff700 1 -- start start 2024-01-11T09:43:22.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.699+0000 7f331f3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f331813fbb0 0x7f33180ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:22.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.699+0000 7f331f3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f331815a730 0x7f33180c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:22.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.699+0000 7f331f3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33180c6b30 con 0x7f331813ff80 2024-01-11T09:43:22.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.699+0000 7f331f3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33180c6800 con 0x7f331813fbb0 2024-01-11T09:43:22.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.699+0000 7f331f3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33180cb240 con 0x7f331815a730 2024-01-11T09:43:22.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.699+0000 7f331d19b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f331813fbb0 0x7f33180ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:22.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.699+0000 7f331c99a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f331813ff80 0x7f331815a020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52320/0 (socket says 172.21.15.27:52320) 2024-01-11T09:43:22.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.699+0000 7f331c99a700 1 -- 172.21.15.27:0/3134719275 learned_addr learned my addr 172.21.15.27:0/3134719275 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:22.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.699+0000 7f331d19b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f331813fbb0 0x7f33180ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35306/0 (socket says 172.21.15.27:35306) 2024-01-11T09:43:22.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.699+0000 7f331d99c700 1 --2- 172.21.15.27:0/3134719275 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f331815a730 0x7f33180c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:22.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.699+0000 7f331d19b700 1 -- 172.21.15.27:0/3134719275 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f331815a730 msgr2=0x7f33180c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:22.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.700+0000 7f331d19b700 1 --2- 172.21.15.27:0/3134719275 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f331815a730 0x7f33180c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.700+0000 7f331d19b700 1 -- 172.21.15.27:0/3134719275 >> v1:172.21.15.27:6789/0 conn(0x7f331813ff80 legacy=0x7f331815a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:22.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.700+0000 7f331d19b700 1 -- 172.21.15.27:0/3134719275 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33180cb520 con 0x7f331813fbb0 2024-01-11T09:43:22.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.700+0000 7f330ffff700 1 -- 172.21.15.27:0/3134719275 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879803163 0 0) 0x7f33180c6b30 con 0x7f331813ff80 2024-01-11T09:43:22.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.700+0000 7f331d19b700 1 --2- 172.21.15.27:0/3134719275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f331813fbb0 0x7f33180ca7a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f331400bb70 tx=0x7f331400b3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa0bb7b34667c702 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:22.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.700+0000 7f330ffff700 1 -- 172.21.15.27:0/3134719275 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3314005c10 con 0x7f331813fbb0 2024-01-11T09:43:22.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.700+0000 7f330ffff700 1 -- 172.21.15.27:0/3134719275 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3314005d70 con 0x7f331813fbb0 2024-01-11T09:43:22.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.700+0000 7f330ffff700 1 -- 172.21.15.27:0/3134719275 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331400db60 con 0x7f331813fbb0 2024-01-11T09:43:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.701+0000 7f331f3ff700 1 -- 172.21.15.27:0/3134719275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f331813fbb0 msgr2=0x7f33180ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.701+0000 7f331f3ff700 1 --2- 172.21.15.27:0/3134719275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f331813fbb0 0x7f33180ca7a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f331400bb70 tx=0x7f331400b3e0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.701+0000 7f331f3ff700 1 -- 172.21.15.27:0/3134719275 shutdown_connections 2024-01-11T09:43:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.701+0000 7f331f3ff700 1 --2- 172.21.15.27:0/3134719275 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f331815a730 0x7f33180c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.701+0000 7f331f3ff700 1 --2- 172.21.15.27:0/3134719275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f331813fbb0 0x7f33180ca7a0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.701+0000 7f331f3ff700 1 -- 172.21.15.27:0/3134719275 >> 172.21.15.27:0/3134719275 conn(0x7f33180680a0 msgr2=0x7f33180cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.701+0000 7f331f3ff700 1 -- 172.21.15.27:0/3134719275 shutdown_connections 2024-01-11T09:43:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.701+0000 7f331f3ff700 1 -- 172.21.15.27:0/3134719275 wait complete. 2024-01-11T09:43:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.703+0000 7f331f3ff700 1 Processor -- start 2024-01-11T09:43:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.703+0000 7f331f3ff700 1 -- start start 2024-01-11T09:43:22.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.704+0000 7f331f3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f331813fbb0 0x7f3318157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:22.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.704+0000 7f331f3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f331813ff80 0x7f3318158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:22.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.704+0000 7f331f3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f331815a730 0x7f3318156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.704+0000 7f331f3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3318158e10 con 0x7f331815a730 2024-01-11T09:43:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.704+0000 7f331f3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33180c3cd0 con 0x7f331813ff80 2024-01-11T09:43:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.704+0000 7f331f3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3318158b90 con 0x7f331813fbb0 2024-01-11T09:43:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.704+0000 7f331d99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f331815a730 0x7f3318156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.704+0000 7f331d99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f331815a730 0x7f3318156020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35308/0 (socket says 172.21.15.27:35308) 2024-01-11T09:43:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.704+0000 7f331d99c700 1 -- 172.21.15.27:0/3195710604 learned_addr learned my addr 172.21.15.27:0/3195710604 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.704+0000 7f331d19b700 1 --2- 172.21.15.27:0/3195710604 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f331813fbb0 0x7f3318157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.704+0000 7f331c99a700 1 --2- 172.21.15.27:0/3195710604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f331813ff80 0x7f3318158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.704+0000 7f331d99c700 1 -- 172.21.15.27:0/3195710604 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f331813fbb0 msgr2=0x7f3318157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.704+0000 7f331d99c700 1 --2- 172.21.15.27:0/3195710604 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f331813fbb0 0x7f3318157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.704+0000 7f331d99c700 1 -- 172.21.15.27:0/3195710604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f331813ff80 msgr2=0x7f3318158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.704+0000 7f331d99c700 1 --2- 172.21.15.27:0/3195710604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f331813ff80 0x7f3318158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.705+0000 7f331d99c700 1 -- 172.21.15.27:0/3195710604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331400b660 con 0x7f331815a730 2024-01-11T09:43:22.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.705+0000 7f331d99c700 1 --2- 172.21.15.27:0/3195710604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f331815a730 0x7f3318156020 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f3310009d70 tx=0x7f331000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:22.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.705+0000 7f330d7fa700 1 -- 172.21.15.27:0/3195710604 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3310009ad0 con 0x7f331815a730 2024-01-11T09:43:22.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.705+0000 7f330d7fa700 1 -- 172.21.15.27:0/3195710604 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331000f040 con 0x7f331815a730 2024-01-11T09:43:22.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.705+0000 7f330d7fa700 1 -- 172.21.15.27:0/3195710604 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331000ccf0 con 0x7f331815a730 2024-01-11T09:43:22.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.705+0000 7f331f3ff700 1 -- 172.21.15.27:0/3195710604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33180c33a0 con 0x7f331815a730 2024-01-11T09:43:22.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.705+0000 7f331f3ff700 1 -- 172.21.15.27:0/3195710604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33180c3960 con 0x7f331815a730 2024-01-11T09:43:22.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.706+0000 7f331f3ff700 1 -- 172.21.15.27:0/3195710604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f331804c3c0 con 0x7f331815a730 2024-01-11T09:43:22.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.707+0000 7f330d7fa700 1 -- 172.21.15.27:0/3195710604 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 113) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f3310014690 con 0x7f331815a730 2024-01-11T09:43:22.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.709+0000 7f330d7fa700 1 --2- 172.21.15.27:0/3195710604 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f32fc067050 0x7f32fc069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:22.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.709+0000 7f330d7fa700 1 -- 172.21.15.27:0/3195710604 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6349+0+0 (secure 0 0 0) 0x7f33100854c0 con 0x7f331815a730 2024-01-11T09:43:22.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.709+0000 7f331d19b700 1 -- 172.21.15.27:0/3195710604 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f32fc067050 msgr2=0x7f32fc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:22.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.709+0000 7f331d19b700 1 --2- 172.21.15.27:0/3195710604 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f32fc067050 0x7f32fc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:43:22.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.713+0000 7f330d7fa700 1 -- 172.21.15.27:0/3195710604 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f33100521f0 con 0x7f331815a730 2024-01-11T09:43:22.896 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:22.904+0000 7fcf2074e1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:43:22.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.910+0000 7f331d19b700 1 -- 172.21.15.27:0/3195710604 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f32fc067050 msgr2=0x7f32fc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:22.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.910+0000 7f331d19b700 1 --2- 172.21.15.27:0/3195710604 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f32fc067050 0x7f32fc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:43:22.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.920+0000 7f331f3ff700 1 -- 172.21.15.27:0/3195710604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f33180cb800 con 0x7f331815a730 2024-01-11T09:43:22.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.921+0000 7f330d7fa700 1 -- 172.21.15.27:0/3195710604 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v185) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3310055810 con 0x7f331815a730 2024-01-11T09:43:22.913 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T09:43:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.923+0000 7f331f3ff700 1 -- 172.21.15.27:0/3195710604 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f32fc067050 msgr2=0x7f32fc069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:43:22.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.923+0000 7f331f3ff700 1 --2- 172.21.15.27:0/3195710604 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f32fc067050 0x7f32fc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.923+0000 7f331f3ff700 1 -- 172.21.15.27:0/3195710604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f331815a730 msgr2=0x7f3318156020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:22.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.923+0000 7f331f3ff700 1 --2- 172.21.15.27:0/3195710604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f331815a730 0x7f3318156020 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f3310009d70 tx=0x7f331000b040 comp rx=0 tx=0).stop 2024-01-11T09:43:22.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.923+0000 7f331f3ff700 1 -- 172.21.15.27:0/3195710604 shutdown_connections 2024-01-11T09:43:22.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.923+0000 7f331f3ff700 1 --2- 172.21.15.27:0/3195710604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f331815a730 0x7f3318156020 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.923+0000 7f331f3ff700 1 --2- 172.21.15.27:0/3195710604 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f331813fbb0 0x7f3318157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.923+0000 7f331f3ff700 1 --2- 172.21.15.27:0/3195710604 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f32fc067050 0x7f32fc069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.923+0000 7f331f3ff700 1 --2- 172.21.15.27:0/3195710604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f331813ff80 0x7f3318158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:22.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.923+0000 7f331f3ff700 1 -- 172.21.15.27:0/3195710604 >> 172.21.15.27:0/3195710604 conn(0x7f33180680a0 msgr2=0x7f33180694c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:22.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.924+0000 7f331f3ff700 1 -- 172.21.15.27:0/3195710604 shutdown_connections 2024-01-11T09:43:22.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:22.924+0000 7f331f3ff700 1 -- 172.21.15.27:0/3195710604 wait complete. 2024-01-11T09:43:22.934 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T09:43:23.075 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:23.084+0000 7f7e4dd341c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:43:23.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.148+0000 7f9f62dff700 1 Processor -- start 2024-01-11T09:43:23.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.148+0000 7f9f62dff700 1 -- start start 2024-01-11T09:43:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.149+0000 7f9f62dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f5c1598f0 0x7f9f5c1646c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.149+0000 7f9f62dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f5c15aef0 con 0x7f9f5c0cd220 2024-01-11T09:43:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.150+0000 7f9f62dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f5c0cee00 con 0x7f9f5c0cd5f0 2024-01-11T09:43:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.150+0000 7f9f62dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c15dd70 con 0x7f9f5c1598f0 2024-01-11T09:43:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.150+0000 7f9f60b9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9f5c0cd220 0x7f9f5c15d320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52332/0 (socket says 172.21.15.27:52332) 2024-01-11T09:43:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.150+0000 7f9f60b9b700 1 -- 172.21.15.27:0/3162114948 learned_addr learned my addr 172.21.15.27:0/3162114948 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:23.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.150+0000 7f9f6139c700 1 --2- 172.21.15.27:0/3162114948 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f5c1598f0 0x7f9f5c1646c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:23.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.150+0000 7f9f5b7fe700 1 -- 172.21.15.27:0/3162114948 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695088929 0 0) 0x7f9f5c15aef0 con 0x7f9f5c0cd220 2024-01-11T09:43:23.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.150+0000 7f9f5b7fe700 1 -- 172.21.15.27:0/3162114948 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f4c003570 con 0x7f9f5c0cd220 2024-01-11T09:43:23.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.150+0000 7f9f5b7fe700 1 -- 172.21.15.27:0/3162114948 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027274047 0 0) 0x7f9f5c0cee00 con 0x7f9f5c0cd5f0 2024-01-11T09:43:23.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.150+0000 7f9f5b7fe700 1 -- 172.21.15.27:0/3162114948 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f5c15aef0 con 0x7f9f5c0cd5f0 2024-01-11T09:43:23.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.151+0000 7f9f5b7fe700 1 -- 172.21.15.27:0/3162114948 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 347693346 0 0) 0x7f9f4c003570 con 0x7f9f5c0cd220 2024-01-11T09:43:23.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.151+0000 7f9f6139c700 1 -- 172.21.15.27:0/3162114948 >> v1:172.21.15.77:6789/0 conn(0x7f9f5c0cd5f0 legacy=0x7f9f5c1591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:23.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.151+0000 7f9f6139c700 1 -- 172.21.15.27:0/3162114948 >> v1:172.21.15.27:6789/0 conn(0x7f9f5c0cd220 legacy=0x7f9f5c15d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:23.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.151+0000 7f9f6139c700 1 -- 172.21.15.27:0/3162114948 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5c15e050 con 0x7f9f5c1598f0 2024-01-11T09:43:23.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.151+0000 7f9f6139c700 1 --2- 172.21.15.27:0/3162114948 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f5c1598f0 0x7f9f5c1646c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9f54009e90 tx=0x7f9f54009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2e7e3b3a0901536 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:23.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.151+0000 7f9f5b7fe700 1 -- 172.21.15.27:0/3162114948 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5400e070 con 0x7f9f5c1598f0 2024-01-11T09:43:23.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.151+0000 7f9f5b7fe700 1 -- 172.21.15.27:0/3162114948 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5400b870 con 0x7f9f5c1598f0 2024-01-11T09:43:23.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.151+0000 7f9f5b7fe700 1 -- 172.21.15.27:0/3162114948 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50001470 con 0x7f9f5c1598f0 2024-01-11T09:43:23.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.152+0000 7f9f62dff700 1 -- 172.21.15.27:0/3162114948 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f5c1598f0 msgr2=0x7f9f5c1646c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:23.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.152+0000 7f9f62dff700 1 --2- 172.21.15.27:0/3162114948 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f5c1598f0 0x7f9f5c1646c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9f54009e90 tx=0x7f9f54009780 comp rx=0 tx=0).stop 2024-01-11T09:43:23.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.152+0000 7f9f62dff700 1 -- 172.21.15.27:0/3162114948 shutdown_connections 2024-01-11T09:43:23.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.152+0000 7f9f62dff700 1 --2- 172.21.15.27:0/3162114948 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f5c1598f0 0x7f9f5c1646c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.152+0000 7f9f62dff700 1 -- 172.21.15.27:0/3162114948 >> 172.21.15.27:0/3162114948 conn(0x7f9f5c06ed40 msgr2=0x7f9f5c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:23.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.152+0000 7f9f62dff700 1 -- 172.21.15.27:0/3162114948 shutdown_connections 2024-01-11T09:43:23.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.152+0000 7f9f62dff700 1 -- 172.21.15.27:0/3162114948 wait complete. 2024-01-11T09:43:23.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.154+0000 7f9f62dff700 1 Processor -- start 2024-01-11T09:43:23.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.154+0000 7f9f62dff700 1 -- start start 2024-01-11T09:43:23.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.155+0000 7f9f62dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f5c0cd220 0x7f9f5c0b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:23.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.155+0000 7f9f62dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f5c0cd5f0 0x7f9f5c0b3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:23.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.155+0000 7f9f62dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f5c1598f0 0x7f9f5c0b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:23.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.155+0000 7f9f62dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c0b6200 con 0x7f9f5c0cd5f0 2024-01-11T09:43:23.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.155+0000 7f9f62dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c15c4f0 con 0x7f9f5c0cd220 2024-01-11T09:43:23.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.155+0000 7f9f62dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c15c980 con 0x7f9f5c1598f0 2024-01-11T09:43:23.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.155+0000 7f9f60b9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f5c0cd220 0x7f9f5c0b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:23.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.155+0000 7f9f5bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f5c0cd5f0 0x7f9f5c0b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:23.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.155+0000 7f9f60b9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f5c0cd220 0x7f9f5c0b5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:43860/0 (socket says 172.21.15.27:43860) 2024-01-11T09:43:23.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.155+0000 7f9f60b9b700 1 -- 172.21.15.27:0/2650374008 learned_addr learned my addr 172.21.15.27:0/2650374008 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:23.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.155+0000 7f9f5bfff700 1 --2- 172.21.15.27:0/2650374008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f5c0cd5f0 0x7f9f5c0b3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35320/0 (socket says 172.21.15.27:35320) 2024-01-11T09:43:23.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.156+0000 7f9f6139c700 1 --2- 172.21.15.27:0/2650374008 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f5c1598f0 0x7f9f5c0b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:23.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.156+0000 7f9f60b9b700 1 -- 172.21.15.27:0/2650374008 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f5c1598f0 msgr2=0x7f9f5c0b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:23.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.156+0000 7f9f60b9b700 1 --2- 172.21.15.27:0/2650374008 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f5c1598f0 0x7f9f5c0b38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.156+0000 7f9f60b9b700 1 -- 172.21.15.27:0/2650374008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f5c0cd5f0 msgr2=0x7f9f5c0b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:23.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.156+0000 7f9f60b9b700 1 --2- 172.21.15.27:0/2650374008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f5c0cd5f0 0x7f9f5c0b3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.156+0000 7f9f60b9b700 1 -- 172.21.15.27:0/2650374008 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f54009a20 con 0x7f9f5c0cd220 2024-01-11T09:43:23.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.156+0000 7f9f60b9b700 1 --2- 172.21.15.27:0/2650374008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f5c0cd220 0x7f9f5c0b5cc0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f9f5000cb90 tx=0x7f9f50001ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:23.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.156+0000 7f9f58ff9700 1 -- 172.21.15.27:0/2650374008 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50007150 con 0x7f9f5c0cd220 2024-01-11T09:43:23.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.157+0000 7f9f58ff9700 1 -- 172.21.15.27:0/2650374008 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f500072f0 con 0x7f9f5c0cd220 2024-01-11T09:43:23.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.157+0000 7f9f58ff9700 1 -- 172.21.15.27:0/2650374008 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f500076b0 con 0x7f9f5c0cd220 2024-01-11T09:43:23.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.157+0000 7f9f62dff700 1 -- 172.21.15.27:0/2650374008 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f5c0ca5c0 con 0x7f9f5c0cd220 2024-01-11T09:43:23.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.157+0000 7f9f62dff700 1 -- 172.21.15.27:0/2650374008 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f5c15bbc0 con 0x7f9f5c0cd220 2024-01-11T09:43:23.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.158+0000 7f9f62dff700 1 -- 172.21.15.27:0/2650374008 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5c15c7d0 con 0x7f9f5c0cd220 2024-01-11T09:43:23.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.159+0000 7f9f58ff9700 1 -- 172.21.15.27:0/2650374008 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 113) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f9f50004a40 con 0x7f9f5c0cd220 2024-01-11T09:43:23.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.160+0000 7f9f58ff9700 1 --2- 172.21.15.27:0/2650374008 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9f40067050 0x7f9f40069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:23.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.160+0000 7f9f58ff9700 1 -- 172.21.15.27:0/2650374008 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6349+0+0 (secure 0 0 0) 0x7f9f50083fc0 con 0x7f9f5c0cd220 2024-01-11T09:43:23.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.160+0000 7f9f5bfff700 1 -- 172.21.15.27:0/2650374008 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9f40067050 msgr2=0x7f9f40069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:23.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.160+0000 7f9f5bfff700 1 --2- 172.21.15.27:0/2650374008 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9f40067050 0x7f9f40069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:43:23.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.164+0000 7f9f58ff9700 1 -- 172.21.15.27:0/2650374008 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9f5004c5a0 con 0x7f9f5c0cd220 2024-01-11T09:43:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.361+0000 7f9f5bfff700 1 -- 172.21.15.27:0/2650374008 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9f40067050 msgr2=0x7f9f40069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.361+0000 7f9f5bfff700 1 --2- 172.21.15.27:0/2650374008 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9f40067050 0x7f9f40069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:43:23.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.369+0000 7f9f62dff700 1 -- 172.21.15.27:0/2650374008 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9f5c0cea90 con 0x7f9f5c0cd220 2024-01-11T09:43:23.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.369+0000 7f9f58ff9700 1 -- 172.21.15.27:0/2650374008 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v185) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9f500180a0 con 0x7f9f5c0cd220 2024-01-11T09:43:23.361 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T09:43:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.372+0000 7f9f62dff700 1 -- 172.21.15.27:0/2650374008 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9f40067050 msgr2=0x7f9f40069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:43:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.372+0000 7f9f62dff700 1 --2- 172.21.15.27:0/2650374008 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9f40067050 0x7f9f40069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.372+0000 7f9f62dff700 1 -- 172.21.15.27:0/2650374008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f5c0cd220 msgr2=0x7f9f5c0b5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.372+0000 7f9f62dff700 1 --2- 172.21.15.27:0/2650374008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f5c0cd220 0x7f9f5c0b5cc0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f9f5000cb90 tx=0x7f9f50001ab0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.372+0000 7f9f62dff700 1 -- 172.21.15.27:0/2650374008 shutdown_connections 2024-01-11T09:43:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.372+0000 7f9f62dff700 1 --2- 172.21.15.27:0/2650374008 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9f5c1598f0 0x7f9f5c0b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.372+0000 7f9f62dff700 1 --2- 172.21.15.27:0/2650374008 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9f40067050 0x7f9f40069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.372+0000 7f9f62dff700 1 --2- 172.21.15.27:0/2650374008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f5c0cd220 0x7f9f5c0b5cc0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.372+0000 7f9f62dff700 1 --2- 172.21.15.27:0/2650374008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f5c0cd5f0 0x7f9f5c0b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.372+0000 7f9f62dff700 1 -- 172.21.15.27:0/2650374008 >> 172.21.15.27:0/2650374008 conn(0x7f9f5c06ed40 msgr2=0x7f9f5c0c5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.372+0000 7f9f62dff700 1 -- 172.21.15.27:0/2650374008 shutdown_connections 2024-01-11T09:43:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.372+0000 7f9f62dff700 1 -- 172.21.15.27:0/2650374008 wait complete. 2024-01-11T09:43:23.382 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T09:43:23.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.595+0000 7f1f333ff700 1 Processor -- start 2024-01-11T09:43:23.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.595+0000 7f1f333ff700 1 -- start start 2024-01-11T09:43:23.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.597+0000 7f1f333ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1f2c0ccd70 0x7f1f2c0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:23.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.597+0000 7f1f333ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1f2c15ae70 0x7f1f2c15b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:23.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.597+0000 7f1f333ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c0cf850 con 0x7f1f2c15a1b0 2024-01-11T09:43:23.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.597+0000 7f1f333ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2c15ac90 con 0x7f1f2c15ae70 2024-01-11T09:43:23.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.597+0000 7f1f333ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c0cec40 con 0x7f1f2c0ccd70 2024-01-11T09:43:23.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.597+0000 7f1f3099a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1f2c0ccd70 0x7f1f2c0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:23.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.597+0000 7f1f3119b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1f2c15a1b0 0x7f1f2c15a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:60592/0 (socket says 172.21.15.27:60592) 2024-01-11T09:43:23.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.597+0000 7f1f3119b700 1 -- 172.21.15.27:0/3018835414 learned_addr learned my addr 172.21.15.27:0/3018835414 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:23.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.597+0000 7f1f3199c700 1 --2- 172.21.15.27:0/3018835414 >> v2:172.21.15.77:3300/0 conn(0x7f1f2c15ae70 0x7f1f2c15b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:23.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.598+0000 7f1f3099a700 1 -- 172.21.15.27:0/3018835414 >> v2:172.21.15.77:3300/0 conn(0x7f1f2c15ae70 msgr2=0x7f1f2c15b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:23.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.598+0000 7f1f3099a700 1 --2- 172.21.15.27:0/3018835414 >> v2:172.21.15.77:3300/0 conn(0x7f1f2c15ae70 0x7f1f2c15b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.598+0000 7f1f3099a700 1 -- 172.21.15.27:0/3018835414 >> v1:172.21.15.77:6789/0 conn(0x7f1f2c15a1b0 legacy=0x7f1f2c15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:23.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.598+0000 7f1f3099a700 1 -- 172.21.15.27:0/3018835414 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c0cef20 con 0x7f1f2c0ccd70 2024-01-11T09:43:23.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.598+0000 7f1f3099a700 1 --2- 172.21.15.27:0/3018835414 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1f2c0ccd70 0x7f1f2c0cd1e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0098e0 tx=0x7f1f1c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95be318c9c579d41 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.598+0000 7f1f23fff700 1 -- 172.21.15.27:0/3018835414 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c00b640 con 0x7f1f2c0ccd70 2024-01-11T09:43:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.598+0000 7f1f23fff700 1 -- 172.21.15.27:0/3018835414 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1c00b7a0 con 0x7f1f2c0ccd70 2024-01-11T09:43:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.598+0000 7f1f23fff700 1 -- 172.21.15.27:0/3018835414 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c00bb40 con 0x7f1f2c0ccd70 2024-01-11T09:43:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.599+0000 7f1f333ff700 1 -- 172.21.15.27:0/3018835414 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1f2c0ccd70 msgr2=0x7f1f2c0cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.599+0000 7f1f333ff700 1 --2- 172.21.15.27:0/3018835414 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1f2c0ccd70 0x7f1f2c0cd1e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0098e0 tx=0x7f1f1c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.599+0000 7f1f333ff700 1 -- 172.21.15.27:0/3018835414 shutdown_connections 2024-01-11T09:43:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.599+0000 7f1f333ff700 1 --2- 172.21.15.27:0/3018835414 >> v2:172.21.15.77:3300/0 conn(0x7f1f2c15ae70 0x7f1f2c15b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.599+0000 7f1f333ff700 1 --2- 172.21.15.27:0/3018835414 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1f2c0ccd70 0x7f1f2c0cd1e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.599+0000 7f1f333ff700 1 -- 172.21.15.27:0/3018835414 >> 172.21.15.27:0/3018835414 conn(0x7f1f2c06ed40 msgr2=0x7f1f2c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.599+0000 7f1f333ff700 1 -- 172.21.15.27:0/3018835414 shutdown_connections 2024-01-11T09:43:23.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.599+0000 7f1f333ff700 1 -- 172.21.15.27:0/3018835414 wait complete. 2024-01-11T09:43:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.600+0000 7f1f333ff700 1 Processor -- start 2024-01-11T09:43:23.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.601+0000 7f1f333ff700 1 -- start start 2024-01-11T09:43:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.601+0000 7f1f333ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1f2c0ccd70 0x7f1f2c158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.601+0000 7f1f333ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f2c15a1b0 0x7f1f2c14d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.602+0000 7f1f333ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f2c15ae70 0x7f1f2c0b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.602+0000 7f1f333ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c158950 con 0x7f1f2c15a1b0 2024-01-11T09:43:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.602+0000 7f1f333ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c14d580 con 0x7f1f2c15ae70 2024-01-11T09:43:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.602+0000 7f1f333ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c158020 con 0x7f1f2c0ccd70 2024-01-11T09:43:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.602+0000 7f1f3099a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f2c15a1b0 0x7f1f2c14d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.602+0000 7f1f3099a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f2c15a1b0 0x7f1f2c14d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35324/0 (socket says 172.21.15.27:35324) 2024-01-11T09:43:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.602+0000 7f1f3099a700 1 -- 172.21.15.27:0/4272271620 learned_addr learned my addr 172.21.15.27:0/4272271620 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:23.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.602+0000 7f1f3119b700 1 --2- 172.21.15.27:0/4272271620 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1f2c0ccd70 0x7f1f2c158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.602+0000 7f1f3199c700 1 --2- 172.21.15.27:0/4272271620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f2c15ae70 0x7f1f2c0b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.602+0000 7f1f3099a700 1 -- 172.21.15.27:0/4272271620 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1f2c0ccd70 msgr2=0x7f1f2c158de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.602+0000 7f1f3099a700 1 --2- 172.21.15.27:0/4272271620 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1f2c0ccd70 0x7f1f2c158de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.602+0000 7f1f3099a700 1 -- 172.21.15.27:0/4272271620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f2c15ae70 msgr2=0x7f1f2c0b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.602+0000 7f1f3099a700 1 --2- 172.21.15.27:0/4272271620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f2c15ae70 0x7f1f2c0b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.602+0000 7f1f3099a700 1 -- 172.21.15.27:0/4272271620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c009420 con 0x7f1f2c15a1b0 2024-01-11T09:43:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.603+0000 7f1f3099a700 1 --2- 172.21.15.27:0/4272271620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f2c15a1b0 0x7f1f2c14d040 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c15add0 tx=0x7f1f1c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.603+0000 7f1f217fa700 1 -- 172.21.15.27:0/4272271620 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c01bc40 con 0x7f1f2c15a1b0 2024-01-11T09:43:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.603+0000 7f1f217fa700 1 -- 172.21.15.27:0/4272271620 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1c01bdc0 con 0x7f1f2c15a1b0 2024-01-11T09:43:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.603+0000 7f1f217fa700 1 -- 172.21.15.27:0/4272271620 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c011670 con 0x7f1f2c15a1b0 2024-01-11T09:43:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.603+0000 7f1f333ff700 1 -- 172.21.15.27:0/4272271620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2c158300 con 0x7f1f2c15a1b0 2024-01-11T09:43:23.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.603+0000 7f1f333ff700 1 -- 172.21.15.27:0/4272271620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2c0ca180 con 0x7f1f2c15a1b0 2024-01-11T09:43:23.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.604+0000 7f1f333ff700 1 -- 172.21.15.27:0/4272271620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f2c158ac0 con 0x7f1f2c15a1b0 2024-01-11T09:43:23.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.605+0000 7f1f217fa700 1 -- 172.21.15.27:0/4272271620 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 113) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f1f1c011a20 con 0x7f1f2c15a1b0 2024-01-11T09:43:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.606+0000 7f1f217fa700 1 --2- 172.21.15.27:0/4272271620 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1f10067050 0x7f1f10069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.606+0000 7f1f217fa700 1 -- 172.21.15.27:0/4272271620 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6349+0+0 (secure 0 0 0) 0x7f1f1c085f30 con 0x7f1f2c15a1b0 2024-01-11T09:43:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.606+0000 7f1f3119b700 1 -- 172.21.15.27:0/4272271620 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1f10067050 msgr2=0x7f1f10069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:23.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.606+0000 7f1f3119b700 1 --2- 172.21.15.27:0/4272271620 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1f10067050 0x7f1f10069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:43:23.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.610+0000 7f1f217fa700 1 -- 172.21.15.27:0/4272271620 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1f1c052d70 con 0x7f1f2c15a1b0 2024-01-11T09:43:23.675 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:23.683+0000 7fcf2074e1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:43:23.764 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:23.772+0000 7fcf2074e1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:43:23.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.807+0000 7f1f3119b700 1 -- 172.21.15.27:0/4272271620 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1f10067050 msgr2=0x7f1f10069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:23.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.807+0000 7f1f3119b700 1 --2- 172.21.15.27:0/4272271620 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1f10067050 0x7f1f10069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:43:23.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.819+0000 7f1f333ff700 1 -- 172.21.15.27:0/4272271620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1f2c04c3c0 con 0x7f1f2c15a1b0 2024-01-11T09:43:23.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.820+0000 7f1f217fa700 1 -- 172.21.15.27:0/4272271620 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v185) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1f1c056390 con 0x7f1f2c15a1b0 2024-01-11T09:43:23.811 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T09:43:23.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.822+0000 7f1f333ff700 1 -- 172.21.15.27:0/4272271620 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1f10067050 msgr2=0x7f1f10069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:43:23.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.822+0000 7f1f333ff700 1 --2- 172.21.15.27:0/4272271620 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1f10067050 0x7f1f10069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.822+0000 7f1f333ff700 1 -- 172.21.15.27:0/4272271620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f2c15a1b0 msgr2=0x7f1f2c14d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.822+0000 7f1f333ff700 1 --2- 172.21.15.27:0/4272271620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f2c15a1b0 0x7f1f2c14d040 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c15add0 tx=0x7f1f1c017000 comp rx=0 tx=0).stop 2024-01-11T09:43:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.822+0000 7f1f333ff700 1 -- 172.21.15.27:0/4272271620 shutdown_connections 2024-01-11T09:43:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.822+0000 7f1f333ff700 1 --2- 172.21.15.27:0/4272271620 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1f10067050 0x7f1f10069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.822+0000 7f1f333ff700 1 --2- 172.21.15.27:0/4272271620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f2c15ae70 0x7f1f2c0b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.822+0000 7f1f333ff700 1 --2- 172.21.15.27:0/4272271620 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1f2c0ccd70 0x7f1f2c158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.822+0000 7f1f333ff700 1 --2- 172.21.15.27:0/4272271620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f2c15a1b0 0x7f1f2c14d040 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.822+0000 7f1f333ff700 1 -- 172.21.15.27:0/4272271620 >> 172.21.15.27:0/4272271620 conn(0x7f1f2c06ed40 msgr2=0x7f1f2c06f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.822+0000 7f1f333ff700 1 -- 172.21.15.27:0/4272271620 shutdown_connections 2024-01-11T09:43:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:23.822+0000 7f1f333ff700 1 -- 172.21.15.27:0/4272271620 wait complete. 2024-01-11T09:43:23.834 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:43:23.919 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:23.927+0000 7fcf2074e1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:43:23.927 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:23.935+0000 7f7e4dd341c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:43:24.007 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:24.016+0000 7fcf2074e1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:43:24.025 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:24.033+0000 7f7e4dd341c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:43:24.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.045+0000 7fb3ef9ff700 1 Processor -- start 2024-01-11T09:43:24.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.045+0000 7fb3ef9ff700 1 -- start start 2024-01-11T09:43:24.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.046+0000 7fb3ef9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb3e80ad020 0x7fb3e80af320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:24.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.046+0000 7fb3ef9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3e8159890 0x7fb3e80aa660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.046+0000 7fb3ef9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e80aa1d0 con 0x7fb3e80ac370 2024-01-11T09:43:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.046+0000 7fb3ef9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3e80af860 con 0x7fb3e80ad020 2024-01-11T09:43:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.046+0000 7fb3ef9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e80ac8d0 con 0x7fb3e8159890 2024-01-11T09:43:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.047+0000 7fb3ed79b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb3e80ad020 0x7fb3e80af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.047+0000 7fb3ed79b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb3e80ad020 0x7fb3e80af320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50752/0 (socket says 172.21.15.27:50752) 2024-01-11T09:43:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.047+0000 7fb3ed79b700 1 -- 172.21.15.27:0/2277373497 learned_addr learned my addr 172.21.15.27:0/2277373497 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.047+0000 7fb3edf9c700 1 --2- 172.21.15.27:0/2277373497 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3e8159890 0x7fb3e80aa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:24.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.047+0000 7fb3ed79b700 1 -- 172.21.15.27:0/2277373497 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3e8159890 msgr2=0x7fb3e80aa660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:24.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.047+0000 7fb3ed79b700 1 --2- 172.21.15.27:0/2277373497 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3e8159890 0x7fb3e80aa660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.047+0000 7fb3ed79b700 1 -- 172.21.15.27:0/2277373497 >> v1:172.21.15.27:6789/0 conn(0x7fb3e80ac370 legacy=0x7fb3e8159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:24.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.047+0000 7fb3ed79b700 1 -- 172.21.15.27:0/2277373497 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e80a98a0 con 0x7fb3e80ad020 2024-01-11T09:43:24.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.048+0000 7fb3ed79b700 1 --2- 172.21.15.27:0/2277373497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3e80ad020 0x7fb3e80af320 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40099b0 tx=0x7fb3e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c4586ae400b852 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:24.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.048+0000 7fb3dffff700 1 -- 172.21.15.27:0/2277373497 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4009ea0 con 0x7fb3e80ad020 2024-01-11T09:43:24.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.048+0000 7fb3dffff700 1 -- 172.21.15.27:0/2277373497 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e400b860 con 0x7fb3e80ad020 2024-01-11T09:43:24.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.048+0000 7fb3dffff700 1 -- 172.21.15.27:0/2277373497 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e400bc00 con 0x7fb3e80ad020 2024-01-11T09:43:24.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.048+0000 7fb3ef9ff700 1 -- 172.21.15.27:0/2277373497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3e80ad020 msgr2=0x7fb3e80af320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:24.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.048+0000 7fb3ef9ff700 1 --2- 172.21.15.27:0/2277373497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3e80ad020 0x7fb3e80af320 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40099b0 tx=0x7fb3e4009250 comp rx=0 tx=0).stop 2024-01-11T09:43:24.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.049+0000 7fb3ef9ff700 1 -- 172.21.15.27:0/2277373497 shutdown_connections 2024-01-11T09:43:24.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.049+0000 7fb3ef9ff700 1 --2- 172.21.15.27:0/2277373497 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3e8159890 0x7fb3e80aa660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.049+0000 7fb3ef9ff700 1 --2- 172.21.15.27:0/2277373497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3e80ad020 0x7fb3e80af320 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.049+0000 7fb3ef9ff700 1 -- 172.21.15.27:0/2277373497 >> 172.21.15.27:0/2277373497 conn(0x7fb3e80680e0 msgr2=0x7fb3e8069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:24.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.049+0000 7fb3ef9ff700 1 -- 172.21.15.27:0/2277373497 shutdown_connections 2024-01-11T09:43:24.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.049+0000 7fb3ef9ff700 1 -- 172.21.15.27:0/2277373497 wait complete. 2024-01-11T09:43:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.050+0000 7fb3ef9ff700 1 Processor -- start 2024-01-11T09:43:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.050+0000 7fb3ef9ff700 1 -- start start 2024-01-11T09:43:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.051+0000 7fb3ef9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3e80ac370 0x7fb3e8151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.051+0000 7fb3ef9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3e80ad020 0x7fb3e8169ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.051+0000 7fb3ef9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3e8159890 0x7fb3e815b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.051+0000 7fb3ef9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e81523c0 con 0x7fb3e80ac370 2024-01-11T09:43:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.051+0000 7fb3ef9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e8155240 con 0x7fb3e80ad020 2024-01-11T09:43:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.051+0000 7fb3ef9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e8154630 con 0x7fb3e8159890 2024-01-11T09:43:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.051+0000 7fb3ed79b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3e80ac370 0x7fb3e8151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.051+0000 7fb3ed79b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3e80ac370 0x7fb3e8151e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50760/0 (socket says 172.21.15.27:50760) 2024-01-11T09:43:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.051+0000 7fb3ed79b700 1 -- 172.21.15.27:0/3586034170 learned_addr learned my addr 172.21.15.27:0/3586034170 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.052+0000 7fb3edf9c700 1 --2- 172.21.15.27:0/3586034170 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3e8159890 0x7fb3e815b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.052+0000 7fb3ecf9a700 1 --2- 172.21.15.27:0/3586034170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3e80ad020 0x7fb3e8169ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.052+0000 7fb3ed79b700 1 -- 172.21.15.27:0/3586034170 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3e8159890 msgr2=0x7fb3e815b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.052+0000 7fb3ed79b700 1 --2- 172.21.15.27:0/3586034170 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3e8159890 0x7fb3e815b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.052+0000 7fb3ed79b700 1 -- 172.21.15.27:0/3586034170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3e80ad020 msgr2=0x7fb3e8169ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.052+0000 7fb3ed79b700 1 --2- 172.21.15.27:0/3586034170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3e80ad020 0x7fb3e8169ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.052+0000 7fb3ed79b700 1 -- 172.21.15.27:0/3586034170 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e40094f0 con 0x7fb3e80ac370 2024-01-11T09:43:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.052+0000 7fb3ed79b700 1 --2- 172.21.15.27:0/3586034170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3e80ac370 0x7fb3e8151e80 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8066180 tx=0x7fb3e4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.052+0000 7fb3dd7fa700 1 -- 172.21.15.27:0/3586034170 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4000d10 con 0x7fb3e80ac370 2024-01-11T09:43:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.052+0000 7fb3dd7fa700 1 -- 172.21.15.27:0/3586034170 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e400bbe0 con 0x7fb3e80ac370 2024-01-11T09:43:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.052+0000 7fb3dd7fa700 1 -- 172.21.15.27:0/3586034170 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4011670 con 0x7fb3e80ac370 2024-01-11T09:43:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.052+0000 7fb3ef9ff700 1 -- 172.21.15.27:0/3586034170 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e8154910 con 0x7fb3e80ac370 2024-01-11T09:43:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.053+0000 7fb3ef9ff700 1 -- 172.21.15.27:0/3586034170 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e8154ed0 con 0x7fb3e80ac370 2024-01-11T09:43:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.054+0000 7fb3ef9ff700 1 -- 172.21.15.27:0/3586034170 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e80718d0 con 0x7fb3e80ac370 2024-01-11T09:43:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.054+0000 7fb3dd7fa700 1 -- 172.21.15.27:0/3586034170 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 113) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fb3e4011aa0 con 0x7fb3e80ac370 2024-01-11T09:43:24.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.055+0000 7fb3dd7fa700 1 --2- 172.21.15.27:0/3586034170 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb3cc067050 0x7fb3cc069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.056+0000 7fb3dd7fa700 1 -- 172.21.15.27:0/3586034170 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6349+0+0 (secure 0 0 0) 0x7fb3e4086140 con 0x7fb3e80ac370 2024-01-11T09:43:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.056+0000 7fb3ecf9a700 1 -- 172.21.15.27:0/3586034170 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb3cc067050 msgr2=0x7fb3cc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.056+0000 7fb3ecf9a700 1 --2- 172.21.15.27:0/3586034170 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb3cc067050 0x7fb3cc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:43:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.059+0000 7fb3dd7fa700 1 -- 172.21.15.27:0/3586034170 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb3e4053020 con 0x7fb3e80ac370 2024-01-11T09:43:24.222 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:24.230+0000 7f7e4dd341c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:43:24.225 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:24.234+0000 7fcf2074e1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:43:24.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.257+0000 7fb3ecf9a700 1 -- 172.21.15.27:0/3586034170 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb3cc067050 msgr2=0x7fb3cc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:24.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.257+0000 7fb3ecf9a700 1 --2- 172.21.15.27:0/3586034170 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb3cc067050 0x7fb3cc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:43:24.315 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:24.323+0000 7f7e4dd341c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:43:24.320 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:24.328+0000 7fcf2074e1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:43:24.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.341+0000 7fb3ef9ff700 1 -- 172.21.15.27:0/3586034170 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3e80a9b80 con 0x7fb3e80ac370 2024-01-11T09:43:24.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.341+0000 7fb3dd7fa700 1 -- 172.21.15.27:0/3586034170 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4601 (secure 0 0 0) 0x7fb3e4056640 con 0x7fb3e80ac370 2024-01-11T09:43:24.333 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:43:24.333 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8015,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1003297202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1003297202},{"type":"v1","addr":"172.21.15.77:6834","nonce":1003297202}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":8018,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3321059147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":3321059147},{"type":"v1","addr":"172.21.15.77:6835","nonce":3321059147}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":9202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2606639222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2606639222},{"type":"v1","addr":"172.21.15.27:6835","nonce":2606639222}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:27:21.470488+0000","modified":"2024-01-11T09:27:24.992917+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9564":{"gid":9564,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/3897592862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3897592862},{"type":"v1","addr":"172.21.15.27:6834","nonce":3897592862}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2024-01-11T09:43:24.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.344+0000 7fb3ef9ff700 1 -- 172.21.15.27:0/3586034170 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb3cc067050 msgr2=0x7fb3cc069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:43:24.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.344+0000 7fb3ef9ff700 1 --2- 172.21.15.27:0/3586034170 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb3cc067050 0x7fb3cc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.344+0000 7fb3ef9ff700 1 -- 172.21.15.27:0/3586034170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3e80ac370 msgr2=0x7fb3e8151e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:24.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.344+0000 7fb3ef9ff700 1 --2- 172.21.15.27:0/3586034170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3e80ac370 0x7fb3e8151e80 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8066180 tx=0x7fb3e4017000 comp rx=0 tx=0).stop 2024-01-11T09:43:24.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.344+0000 7fb3ef9ff700 1 -- 172.21.15.27:0/3586034170 shutdown_connections 2024-01-11T09:43:24.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.344+0000 7fb3ef9ff700 1 --2- 172.21.15.27:0/3586034170 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb3e8159890 0x7fb3e815b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.344+0000 7fb3ef9ff700 1 --2- 172.21.15.27:0/3586034170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3e80ac370 0x7fb3e8151e80 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.344+0000 7fb3ef9ff700 1 --2- 172.21.15.27:0/3586034170 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb3cc067050 0x7fb3cc069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.344+0000 7fb3ef9ff700 1 --2- 172.21.15.27:0/3586034170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3e80ad020 0x7fb3e8169ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.344+0000 7fb3ef9ff700 1 -- 172.21.15.27:0/3586034170 >> 172.21.15.27:0/3586034170 conn(0x7fb3e80680e0 msgr2=0x7fb3e80abc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:24.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.344+0000 7fb3ef9ff700 1 -- 172.21.15.27:0/3586034170 shutdown_connections 2024-01-11T09:43:24.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.344+0000 7fb3ef9ff700 1 -- 172.21.15.27:0/3586034170 wait complete. 2024-01-11T09:43:24.337 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 94 2024-01-11T09:43:24.354 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:43:24.416 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:24.424+0000 7fcf2074e1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:43:24.512 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:24.520+0000 7fcf2074e1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:43:24.552 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:24.561+0000 7f7e4dd341c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:43:24.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.569+0000 7fa83e1ff700 1 Processor -- start 2024-01-11T09:43:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.569+0000 7fa83e1ff700 1 -- start start 2024-01-11T09:43:24.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.570+0000 7fa83e1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa8380c8620 0x7fa8380af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:24.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.570+0000 7fa83e1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa838069d40 0x7fa8380b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:24.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.570+0000 7fa83e1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa8380c89f0 0x7fa8380b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:24.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.570+0000 7fa83e1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa838157390 con 0x7fa8380c8620 2024-01-11T09:43:24.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.570+0000 7fa83e1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8380afeb0 con 0x7fa8380c89f0 2024-01-11T09:43:24.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.570+0000 7fa83e1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa83806a200 con 0x7fa838069d40 2024-01-11T09:43:24.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.571+0000 7fa8377fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa8380c8620 0x7fa8380af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:24.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.571+0000 7fa8377fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa8380c8620 0x7fa8380af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50764/0 (socket says 172.21.15.27:50764) 2024-01-11T09:43:24.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.571+0000 7fa837fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa8380c89f0 0x7fa8380b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:24.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.571+0000 7fa836ffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa838069d40 0x7fa8380b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:24.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.571+0000 7fa8377fe700 1 -- 172.21.15.27:0/3467590729 learned_addr learned my addr 172.21.15.27:0/3467590729 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:24.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.571+0000 7fa8377fe700 1 -- 172.21.15.27:0/3467590729 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa838069d40 msgr2=0x7fa8380b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:24.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.571+0000 7fa8377fe700 1 --2- 172.21.15.27:0/3467590729 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa838069d40 0x7fa8380b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.571+0000 7fa8377fe700 1 -- 172.21.15.27:0/3467590729 >> v2:172.21.15.77:3300/0 conn(0x7fa8380c89f0 msgr2=0x7fa8380b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:24.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.571+0000 7fa8377fe700 1 --2- 172.21.15.27:0/3467590729 >> v2:172.21.15.77:3300/0 conn(0x7fa8380c89f0 0x7fa8380b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.571+0000 7fa8377fe700 1 -- 172.21.15.27:0/3467590729 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa838156d40 con 0x7fa8380c8620 2024-01-11T09:43:24.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.571+0000 7fa8377fe700 1 --2- 172.21.15.27:0/3467590729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8380c8620 0x7fa8380af970 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fa8381545c0 tx=0x7fa82800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b308bab13823427 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:24.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.572+0000 7fa8367fc700 1 -- 172.21.15.27:0/3467590729 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82800db40 con 0x7fa8380c8620 2024-01-11T09:43:24.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.572+0000 7fa8367fc700 1 -- 172.21.15.27:0/3467590729 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82800dca0 con 0x7fa8380c8620 2024-01-11T09:43:24.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.572+0000 7fa8367fc700 1 -- 172.21.15.27:0/3467590729 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8280196b0 con 0x7fa8380c8620 2024-01-11T09:43:24.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.572+0000 7fa83e1ff700 1 -- 172.21.15.27:0/3467590729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8380c8620 msgr2=0x7fa8380af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:24.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.572+0000 7fa83e1ff700 1 --2- 172.21.15.27:0/3467590729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8380c8620 0x7fa8380af970 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fa8381545c0 tx=0x7fa82800b4f0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.572+0000 7fa83e1ff700 1 -- 172.21.15.27:0/3467590729 shutdown_connections 2024-01-11T09:43:24.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.572+0000 7fa83e1ff700 1 --2- 172.21.15.27:0/3467590729 >> v2:172.21.15.77:3300/0 conn(0x7fa8380c89f0 0x7fa8380b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.572+0000 7fa83e1ff700 1 --2- 172.21.15.27:0/3467590729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8380c8620 0x7fa8380af970 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.572+0000 7fa83e1ff700 1 --2- 172.21.15.27:0/3467590729 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa838069d40 0x7fa8380b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.572+0000 7fa83e1ff700 1 -- 172.21.15.27:0/3467590729 >> 172.21.15.27:0/3467590729 conn(0x7fa8380680e0 msgr2=0x7fa838069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:24.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.572+0000 7fa83e1ff700 1 -- 172.21.15.27:0/3467590729 shutdown_connections 2024-01-11T09:43:24.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.573+0000 7fa83e1ff700 1 -- 172.21.15.27:0/3467590729 wait complete. 2024-01-11T09:43:24.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.574+0000 7fa83e1ff700 1 Processor -- start 2024-01-11T09:43:24.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.574+0000 7fa83e1ff700 1 -- start start 2024-01-11T09:43:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.575+0000 7fa83e1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa838069d40 0x7fa838155e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.575+0000 7fa83e1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa8380c8620 0x7fa83814da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.575+0000 7fa83e1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8380c89f0 0x7fa83814df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.575+0000 7fa83e1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa83814d600 con 0x7fa838069d40 2024-01-11T09:43:24.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.575+0000 7fa83e1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa83816a270 con 0x7fa8380c89f0 2024-01-11T09:43:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.575+0000 7fa83e1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa83816e370 con 0x7fa8380c8620 2024-01-11T09:43:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.575+0000 7fa8377fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa838069d40 0x7fa838155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.575+0000 7fa8377fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa838069d40 0x7fa838155e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50770/0 (socket says 172.21.15.27:50770) 2024-01-11T09:43:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.575+0000 7fa8377fe700 1 -- 172.21.15.27:0/3730861777 learned_addr learned my addr 172.21.15.27:0/3730861777 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.575+0000 7fa836ffd700 1 --2- 172.21.15.27:0/3730861777 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa8380c8620 0x7fa83814da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.575+0000 7fa837fff700 1 --2- 172.21.15.27:0/3730861777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8380c89f0 0x7fa83814df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.576+0000 7fa8377fe700 1 -- 172.21.15.27:0/3730861777 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa8380c8620 msgr2=0x7fa83814da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.576+0000 7fa8377fe700 1 --2- 172.21.15.27:0/3730861777 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa8380c8620 0x7fa83814da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.576+0000 7fa8377fe700 1 -- 172.21.15.27:0/3730861777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8380c89f0 msgr2=0x7fa83814df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.576+0000 7fa8377fe700 1 --2- 172.21.15.27:0/3730861777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8380c89f0 0x7fa83814df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.576+0000 7fa8377fe700 1 -- 172.21.15.27:0/3730861777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82800b790 con 0x7fa838069d40 2024-01-11T09:43:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.576+0000 7fa8377fe700 1 --2- 172.21.15.27:0/3730861777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa838069d40 0x7fa838155e80 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fa82800b760 tx=0x7fa82801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.576+0000 7fa81f7fe700 1 -- 172.21.15.27:0/3730861777 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa828010470 con 0x7fa838069d40 2024-01-11T09:43:24.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.576+0000 7fa81f7fe700 1 -- 172.21.15.27:0/3730861777 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa828010610 con 0x7fa838069d40 2024-01-11T09:43:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.576+0000 7fa81f7fe700 1 -- 172.21.15.27:0/3730861777 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8280109d0 con 0x7fa838069d40 2024-01-11T09:43:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.576+0000 7fa83e1ff700 1 -- 172.21.15.27:0/3730861777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa83816e090 con 0x7fa838069d40 2024-01-11T09:43:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.577+0000 7fa83e1ff700 1 -- 172.21.15.27:0/3730861777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa83814d220 con 0x7fa838069d40 2024-01-11T09:43:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.578+0000 7fa83e1ff700 1 -- 172.21.15.27:0/3730861777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8380718d0 con 0x7fa838069d40 2024-01-11T09:43:24.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.579+0000 7fa81f7fe700 1 -- 172.21.15.27:0/3730861777 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 113) v1 ==== 84856+0+0 (secure 0 0 0) 0x7fa82802c430 con 0x7fa838069d40 2024-01-11T09:43:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.579+0000 7fa81f7fe700 1 --2- 172.21.15.27:0/3730861777 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa824067050 0x7fa824069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:24.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.580+0000 7fa81f7fe700 1 -- 172.21.15.27:0/3730861777 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6349+0+0 (secure 0 0 0) 0x7fa828014070 con 0x7fa838069d40 2024-01-11T09:43:24.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.580+0000 7fa836ffd700 1 -- 172.21.15.27:0/3730861777 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa824067050 msgr2=0x7fa824069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:24.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.580+0000 7fa836ffd700 1 --2- 172.21.15.27:0/3730861777 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa824067050 0x7fa824069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:43:24.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.588+0000 7fa81f7fe700 1 -- 172.21.15.27:0/3730861777 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa828053500 con 0x7fa838069d40 2024-01-11T09:43:24.661 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:24.669+0000 7f7e4dd341c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:43:24.702 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:24.710+0000 7fcf2074e1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:43:24.763 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:24.771+0000 7f7e4dd341c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:43:24.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.781+0000 7fa836ffd700 1 -- 172.21.15.27:0/3730861777 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa824067050 msgr2=0x7fa824069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:24.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.781+0000 7fa836ffd700 1 --2- 172.21.15.27:0/3730861777 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa824067050 0x7fa824069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:43:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.795+0000 7fa83e1ff700 1 -- 172.21.15.27:0/3730861777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa838157020 con 0x7fa838069d40 2024-01-11T09:43:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.797+0000 7fa81f7fe700 1 -- 172.21.15.27:0/3730861777 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) v1 ==== 74+0+16282 (secure 0 0 0) 0x7fa828056b20 con 0x7fa838069d40 2024-01-11T09:43:24.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:43:24.789 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":185,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:43:21.972966+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:27:19.009515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-01-11T09:27:20.468964+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":185,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1160908778":"2024-01-12T09:38:23.806141+0000","172.21.15.77:0/3404938947":"2024-01-12T09:38:23.806141+0000","172.21.15.77:0/2730921345":"2024-01-12T09:38:23.806141+0000","172.21.15.77:0/2548321690":"2024-01-12T09:38:23.806141+0000","172.21.15.77:0/3966666871":"2024-01-12T09:38:23.806141+0000","172.21.15.77:6837/41381":"2024-01-12T09:27:37.837169+0000","172.21.15.77:6836/41381":"2024-01-12T09:27:37.837169+0000","172.21.15.77:0/1337020578":"2024-01-12T09:27:37.837169+0000","172.21.15.77:0/2253497474":"2024-01-12T09:27:37.837169+0000","172.21.15.77:0/2186545586":"2024-01-12T09:38:23.806141+0000","172.21.15.77:0/2706846520":"2024-01-12T09:27:37.837169+0000","172.21.15.77:0/2650034185":"2024-01-12T09:27:37.837169+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":23,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:43:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.799+0000 7fa83e1ff700 1 -- 172.21.15.27:0/3730861777 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa824067050 msgr2=0x7fa824069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:43:24.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.799+0000 7fa83e1ff700 1 --2- 172.21.15.27:0/3730861777 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa824067050 0x7fa824069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.799+0000 7fa83e1ff700 1 -- 172.21.15.27:0/3730861777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa838069d40 msgr2=0x7fa838155e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:24.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.799+0000 7fa83e1ff700 1 --2- 172.21.15.27:0/3730861777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa838069d40 0x7fa838155e80 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fa82800b760 tx=0x7fa82801b000 comp rx=0 tx=0).stop 2024-01-11T09:43:24.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.799+0000 7fa83e1ff700 1 -- 172.21.15.27:0/3730861777 shutdown_connections 2024-01-11T09:43:24.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.799+0000 7fa83e1ff700 1 --2- 172.21.15.27:0/3730861777 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa824067050 0x7fa824069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.799+0000 7fa83e1ff700 1 --2- 172.21.15.27:0/3730861777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8380c89f0 0x7fa83814df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.799+0000 7fa83e1ff700 1 --2- 172.21.15.27:0/3730861777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa838069d40 0x7fa838155e80 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.799+0000 7fa83e1ff700 1 --2- 172.21.15.27:0/3730861777 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa8380c8620 0x7fa83814da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:24.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.799+0000 7fa83e1ff700 1 -- 172.21.15.27:0/3730861777 >> 172.21.15.27:0/3730861777 conn(0x7fa8380680e0 msgr2=0x7fa838157f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:24.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.800+0000 7fa83e1ff700 1 -- 172.21.15.27:0/3730861777 shutdown_connections 2024-01-11T09:43:24.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:24.800+0000 7fa83e1ff700 1 -- 172.21.15.27:0/3730861777 wait complete. 2024-01-11T09:43:24.800 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:24.808+0000 7fcf2074e1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:43:24.810 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-11T09:43:24.810 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:43:24.871 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:24.880+0000 7f7e4dd341c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:43:25.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.022+0000 7f99c83ff700 1 Processor -- start 2024-01-11T09:43:25.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.022+0000 7f99c83ff700 1 -- start start 2024-01-11T09:43:25.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.023+0000 7f99c83ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f99c01575e0 0x7f99c0158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:25.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.023+0000 7f99c83ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f99c006b610 0x7f99c00b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:25.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.023+0000 7f99c83ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99c0158b60 con 0x7f99c00b1800 2024-01-11T09:43:25.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.023+0000 7f99c83ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99c00cb5e0 con 0x7f99c01575e0 2024-01-11T09:43:25.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.023+0000 7f99c83ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99c01599b0 con 0x7f99c006b610 2024-01-11T09:43:25.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.023+0000 7f99c619b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f99c01575e0 0x7f99c0158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:25.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.023+0000 7f99c699c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f99c00b1800 0x7f99c00ca1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55930/0 (socket says 172.21.15.27:55930) 2024-01-11T09:43:25.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.024+0000 7f99c699c700 1 -- 172.21.15.27:0/185561380 learned_addr learned my addr 172.21.15.27:0/185561380 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:25.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.024+0000 7f99c599a700 1 --2- 172.21.15.27:0/185561380 >> v2:172.21.15.77:3300/0 conn(0x7f99c006b610 0x7f99c00b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:25.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.024+0000 7f99c619b700 1 -- 172.21.15.27:0/185561380 >> v2:172.21.15.77:3300/0 conn(0x7f99c006b610 msgr2=0x7f99c00b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:25.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.024+0000 7f99c619b700 1 --2- 172.21.15.27:0/185561380 >> v2:172.21.15.77:3300/0 conn(0x7f99c006b610 0x7f99c00b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.024+0000 7f99c619b700 1 -- 172.21.15.27:0/185561380 >> v1:172.21.15.27:6789/0 conn(0x7f99c00b1800 legacy=0x7f99c00ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:25.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.024+0000 7f99c619b700 1 -- 172.21.15.27:0/185561380 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c00b1c10 con 0x7f99c01575e0 2024-01-11T09:43:25.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.024+0000 7f99c5199700 1 -- 172.21.15.27:0/185561380 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91311539 0 0) 0x7f99b8001490 con 0x7f99c00b1800 2024-01-11T09:43:25.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.024+0000 7f99c619b700 1 --2- 172.21.15.27:0/185561380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99c01575e0 0x7f99c0158620 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f99c0064fe0 tx=0x7f99bc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c86c2043c3c73c15 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:25.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.025+0000 7f99c5199700 1 -- 172.21.15.27:0/185561380 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc011630 con 0x7f99c01575e0 2024-01-11T09:43:25.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.025+0000 7f99c5199700 1 -- 172.21.15.27:0/185561380 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99bc011790 con 0x7f99c01575e0 2024-01-11T09:43:25.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.025+0000 7f99c5199700 1 -- 172.21.15.27:0/185561380 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc011b30 con 0x7f99c01575e0 2024-01-11T09:43:25.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.025+0000 7f99c83ff700 1 -- 172.21.15.27:0/185561380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99c01575e0 msgr2=0x7f99c0158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:25.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.025+0000 7f99c83ff700 1 --2- 172.21.15.27:0/185561380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99c01575e0 0x7f99c0158620 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f99c0064fe0 tx=0x7f99bc0091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.025+0000 7f99c83ff700 1 -- 172.21.15.27:0/185561380 shutdown_connections 2024-01-11T09:43:25.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.025+0000 7f99c83ff700 1 --2- 172.21.15.27:0/185561380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99c01575e0 0x7f99c0158620 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.025+0000 7f99c83ff700 1 --2- 172.21.15.27:0/185561380 >> v2:172.21.15.77:3300/0 conn(0x7f99c006b610 0x7f99c00b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.025+0000 7f99c83ff700 1 -- 172.21.15.27:0/185561380 >> 172.21.15.27:0/185561380 conn(0x7f99c00680e0 msgr2=0x7f99c0069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:25.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.025+0000 7f99c83ff700 1 -- 172.21.15.27:0/185561380 shutdown_connections 2024-01-11T09:43:25.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.025+0000 7f99c83ff700 1 -- 172.21.15.27:0/185561380 wait complete. 2024-01-11T09:43:25.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.027+0000 7f99c83ff700 1 Processor -- start 2024-01-11T09:43:25.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.027+0000 7f99c83ff700 1 -- start start 2024-01-11T09:43:25.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.028+0000 7f99c83ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99c006b610 0x7f99c0145b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:25.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.028+0000 7f99c83ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99c00b1800 0x7f99c015c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:25.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.028+0000 7f99c83ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99c01575e0 0x7f99c015c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:25.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.028+0000 7f99c83ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c00c7230 con 0x7f99c01575e0 2024-01-11T09:43:25.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.028+0000 7f99c83ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c00c73c0 con 0x7f99c006b610 2024-01-11T09:43:25.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.028+0000 7f99c83ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c00c3d40 con 0x7f99c00b1800 2024-01-11T09:43:25.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.028+0000 7f99c699c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99c01575e0 0x7f99c015c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:25.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.028+0000 7f99c699c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99c01575e0 0x7f99c015c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50786/0 (socket says 172.21.15.27:50786) 2024-01-11T09:43:25.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.028+0000 7f99c699c700 1 -- 172.21.15.27:0/3709681957 learned_addr learned my addr 172.21.15.27:0/3709681957 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:25.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.028+0000 7f99c599a700 1 --2- 172.21.15.27:0/3709681957 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99c00b1800 0x7f99c015c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:25.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.028+0000 7f99c619b700 1 --2- 172.21.15.27:0/3709681957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99c006b610 0x7f99c0145b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:25.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.029+0000 7f99c699c700 1 -- 172.21.15.27:0/3709681957 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99c00b1800 msgr2=0x7f99c015c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:25.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.029+0000 7f99c699c700 1 --2- 172.21.15.27:0/3709681957 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99c00b1800 0x7f99c015c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.029+0000 7f99c699c700 1 -- 172.21.15.27:0/3709681957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99c006b610 msgr2=0x7f99c0145b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:25.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.029+0000 7f99c699c700 1 --2- 172.21.15.27:0/3709681957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99c006b610 0x7f99c0145b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.029+0000 7f99c699c700 1 -- 172.21.15.27:0/3709681957 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99bc009420 con 0x7f99c01575e0 2024-01-11T09:43:25.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.029+0000 7f99c699c700 1 --2- 172.21.15.27:0/3709681957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99c01575e0 0x7f99c015c610 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f99b800cc90 tx=0x7f99b8001a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:25.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.029+0000 7f99b67fc700 1 -- 172.21.15.27:0/3709681957 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b8004bb0 con 0x7f99c01575e0 2024-01-11T09:43:25.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.029+0000 7f99b67fc700 1 -- 172.21.15.27:0/3709681957 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b8004d10 con 0x7f99c01575e0 2024-01-11T09:43:25.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.029+0000 7f99b67fc700 1 -- 172.21.15.27:0/3709681957 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b800ab00 con 0x7f99c01575e0 2024-01-11T09:43:25.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.029+0000 7f99c83ff700 1 -- 172.21.15.27:0/3709681957 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c00c4020 con 0x7f99c01575e0 2024-01-11T09:43:25.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.029+0000 7f99c83ff700 1 -- 172.21.15.27:0/3709681957 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c00c45e0 con 0x7f99c01575e0 2024-01-11T09:43:25.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.031+0000 7f99c83ff700 1 -- 172.21.15.27:0/3709681957 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c00c7570 con 0x7f99c01575e0 2024-01-11T09:43:25.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.032+0000 7f99b67fc700 1 -- 172.21.15.27:0/3709681957 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 113) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f99b80070f0 con 0x7f99c01575e0 2024-01-11T09:43:25.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.033+0000 7f99b67fc700 1 --2- 172.21.15.27:0/3709681957 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f99a4067050 0x7f99a4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:25.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.033+0000 7f99b67fc700 1 -- 172.21.15.27:0/3709681957 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6349+0+0 (secure 0 0 0) 0x7f99b8084580 con 0x7f99c01575e0 2024-01-11T09:43:25.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.033+0000 7f99c619b700 1 -- 172.21.15.27:0/3709681957 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f99a4067050 msgr2=0x7f99a4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:25.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.033+0000 7f99c619b700 1 --2- 172.21.15.27:0/3709681957 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f99a4067050 0x7f99a4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:43:25.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.037+0000 7f99b67fc700 1 -- 172.21.15.27:0/3709681957 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f99c00c7570 con 0x7f99c01575e0 2024-01-11T09:43:25.082 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:25.090+0000 7f7e4dd341c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:43:25.146 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:25.155+0000 7fcf2074e1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:43:25.187 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:25.196+0000 7f7e4dd341c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:43:25.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.234+0000 7f99c619b700 1 -- 172.21.15.27:0/3709681957 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f99a4067050 msgr2=0x7f99a4069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:25.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.234+0000 7f99c619b700 1 --2- 172.21.15.27:0/3709681957 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f99a4067050 0x7f99a4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:43:25.236 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:25.245+0000 7fcf2074e1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:43:25.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.286+0000 7f99c83ff700 1 -- 172.21.15.27:0/3709681957 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99c0158b60 con 0x7f99c01575e0 2024-01-11T09:43:25.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.286+0000 7f99b67fc700 1 -- 172.21.15.27:0/3709681957 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4601 (secure 0 0 0) 0x7f99c0158b60 con 0x7f99c01575e0 2024-01-11T09:43:25.278 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:43:25.278 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8015,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1003297202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1003297202},{"type":"v1","addr":"172.21.15.77:6834","nonce":1003297202}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":8018,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3321059147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":3321059147},{"type":"v1","addr":"172.21.15.77:6835","nonce":3321059147}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":9202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2606639222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2606639222},{"type":"v1","addr":"172.21.15.27:6835","nonce":2606639222}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:27:21.470488+0000","modified":"2024-01-11T09:27:24.992917+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9564":{"gid":9564,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/3897592862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3897592862},{"type":"v1","addr":"172.21.15.27:6834","nonce":3897592862}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2024-01-11T09:43:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.289+0000 7f99c83ff700 1 -- 172.21.15.27:0/3709681957 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f99a4067050 msgr2=0x7f99a4069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:43:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.289+0000 7f99c83ff700 1 --2- 172.21.15.27:0/3709681957 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f99a4067050 0x7f99a4069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.289+0000 7f99c83ff700 1 -- 172.21.15.27:0/3709681957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99c01575e0 msgr2=0x7f99c015c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.289+0000 7f99c83ff700 1 --2- 172.21.15.27:0/3709681957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99c01575e0 0x7f99c015c610 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f99b800cc90 tx=0x7f99b8001a50 comp rx=0 tx=0).stop 2024-01-11T09:43:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.289+0000 7f99c83ff700 1 -- 172.21.15.27:0/3709681957 shutdown_connections 2024-01-11T09:43:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.289+0000 7f99c83ff700 1 --2- 172.21.15.27:0/3709681957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99c01575e0 0x7f99c015c610 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.289+0000 7f99c83ff700 1 --2- 172.21.15.27:0/3709681957 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f99a4067050 0x7f99a4069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.289+0000 7f99c83ff700 1 --2- 172.21.15.27:0/3709681957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99c006b610 0x7f99c0145b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.289+0000 7f99c83ff700 1 --2- 172.21.15.27:0/3709681957 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99c00b1800 0x7f99c015c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.289+0000 7f99c83ff700 1 -- 172.21.15.27:0/3709681957 >> 172.21.15.27:0/3709681957 conn(0x7f99c00680e0 msgr2=0x7f99c01579b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:25.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.289+0000 7f99c83ff700 1 -- 172.21.15.27:0/3709681957 shutdown_connections 2024-01-11T09:43:25.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.289+0000 7f99c83ff700 1 -- 172.21.15.27:0/3709681957 wait complete. 2024-01-11T09:43:25.282 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 94 2024-01-11T09:43:25.299 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:43:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.514+0000 7f34ef5ff700 1 Processor -- start 2024-01-11T09:43:25.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.514+0000 7f34ef5ff700 1 -- start start 2024-01-11T09:43:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.516+0000 7f34ef5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f34e80c8770 0x7f34e806d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.516+0000 7f34ef5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f34e8069af0 0x7f34e806daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.516+0000 7f34ef5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e80c82e0 con 0x7f34e8069720 2024-01-11T09:43:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.516+0000 7f34ef5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34e80c76d0 con 0x7f34e8069af0 2024-01-11T09:43:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.516+0000 7f34ef5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34e80c79b0 con 0x7f34e80c8770 2024-01-11T09:43:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.516+0000 7f34edb9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f34e8069af0 0x7f34e806daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.516+0000 7f34ed39b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f34e8069720 0x7f34e80c69f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55946/0 (socket says 172.21.15.27:55946) 2024-01-11T09:43:25.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.516+0000 7f34ed39b700 1 -- 172.21.15.27:0/2368137820 learned_addr learned my addr 172.21.15.27:0/2368137820 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:25.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.516+0000 7f34ecb9a700 1 --2- 172.21.15.27:0/2368137820 >> v2:172.21.15.77:3300/0 conn(0x7f34e80c8770 0x7f34e806d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:25.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.517+0000 7f34edb9c700 1 -- 172.21.15.27:0/2368137820 >> v2:172.21.15.77:3300/0 conn(0x7f34e80c8770 msgr2=0x7f34e806d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:25.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.517+0000 7f34edb9c700 1 --2- 172.21.15.27:0/2368137820 >> v2:172.21.15.77:3300/0 conn(0x7f34e80c8770 0x7f34e806d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.517+0000 7f34edb9c700 1 -- 172.21.15.27:0/2368137820 >> v1:172.21.15.27:6789/0 conn(0x7f34e8069720 legacy=0x7f34e80c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:25.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.517+0000 7f34edb9c700 1 -- 172.21.15.27:0/2368137820 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e80c7c90 con 0x7f34e8069af0 2024-01-11T09:43:25.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.517+0000 7f34edb9c700 1 --2- 172.21.15.27:0/2368137820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34e8069af0 0x7f34e806daa0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f34e80cd270 tx=0x7f34e0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d425ed1f3b5113b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:25.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.517+0000 7f34dffff700 1 -- 172.21.15.27:0/2368137820 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e000e070 con 0x7f34e8069af0 2024-01-11T09:43:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.517+0000 7f34dffff700 1 -- 172.21.15.27:0/2368137820 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e000bb60 con 0x7f34e8069af0 2024-01-11T09:43:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.517+0000 7f34dffff700 1 -- 172.21.15.27:0/2368137820 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0012430 con 0x7f34e8069af0 2024-01-11T09:43:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.518+0000 7f34ef5ff700 1 -- 172.21.15.27:0/2368137820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34e8069af0 msgr2=0x7f34e806daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.518+0000 7f34ef5ff700 1 --2- 172.21.15.27:0/2368137820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34e8069af0 0x7f34e806daa0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f34e80cd270 tx=0x7f34e0009780 comp rx=0 tx=0).stop 2024-01-11T09:43:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.518+0000 7f34ef5ff700 1 -- 172.21.15.27:0/2368137820 shutdown_connections 2024-01-11T09:43:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.518+0000 7f34ef5ff700 1 --2- 172.21.15.27:0/2368137820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34e8069af0 0x7f34e806daa0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.518+0000 7f34ef5ff700 1 --2- 172.21.15.27:0/2368137820 >> v2:172.21.15.77:3300/0 conn(0x7f34e80c8770 0x7f34e806d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.518+0000 7f34ef5ff700 1 -- 172.21.15.27:0/2368137820 >> 172.21.15.27:0/2368137820 conn(0x7f34e80680a0 msgr2=0x7f34e80cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.518+0000 7f34ef5ff700 1 -- 172.21.15.27:0/2368137820 shutdown_connections 2024-01-11T09:43:25.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.518+0000 7f34ef5ff700 1 -- 172.21.15.27:0/2368137820 wait complete. 2024-01-11T09:43:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.520+0000 7f34ef5ff700 1 Processor -- start 2024-01-11T09:43:25.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.520+0000 7f34ef5ff700 1 -- start start 2024-01-11T09:43:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.522+0000 7f34ef5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34e8069720 0x7f34e8151400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.522+0000 7f34ef5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e8069af0 0x7f34e80c4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.522+0000 7f34ef5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34e80c8770 0x7f34e80c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.522+0000 7f34ef5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e80c63e0 con 0x7f34e80c8770 2024-01-11T09:43:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.522+0000 7f34ef5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e80c6610 con 0x7f34e8069af0 2024-01-11T09:43:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.522+0000 7f34ef5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e8151d20 con 0x7f34e8069720 2024-01-11T09:43:25.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.522+0000 7f34edb9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34e80c8770 0x7f34e80c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.522+0000 7f34ed39b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34e8069720 0x7f34e8151400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.522+0000 7f34ecb9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e8069af0 0x7f34e80c4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.522+0000 7f34edb9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34e80c8770 0x7f34e80c5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50806/0 (socket says 172.21.15.27:50806) 2024-01-11T09:43:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.522+0000 7f34edb9c700 1 -- 172.21.15.27:0/2087639639 learned_addr learned my addr 172.21.15.27:0/2087639639 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.522+0000 7f34ecb9a700 1 -- 172.21.15.27:0/2087639639 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34e8069720 msgr2=0x7f34e8151400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.522+0000 7f34ecb9a700 1 --2- 172.21.15.27:0/2087639639 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34e8069720 0x7f34e8151400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.522+0000 7f34ecb9a700 1 -- 172.21.15.27:0/2087639639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34e80c8770 msgr2=0x7f34e80c5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.522+0000 7f34ecb9a700 1 --2- 172.21.15.27:0/2087639639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34e80c8770 0x7f34e80c5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.522+0000 7f34ecb9a700 1 -- 172.21.15.27:0/2087639639 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e0009a20 con 0x7f34e8069af0 2024-01-11T09:43:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.523+0000 7f34ecb9a700 1 --2- 172.21.15.27:0/2087639639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e8069af0 0x7f34e80c4d10 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f34d800e900 tx=0x7f34d80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.523+0000 7f34dd7fa700 1 -- 172.21.15.27:0/2087639639 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d8009450 con 0x7f34e8069af0 2024-01-11T09:43:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.523+0000 7f34dd7fa700 1 -- 172.21.15.27:0/2087639639 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d8013680 con 0x7f34e8069af0 2024-01-11T09:43:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.523+0000 7f34dd7fa700 1 -- 172.21.15.27:0/2087639639 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d8013a40 con 0x7f34e8069af0 2024-01-11T09:43:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.523+0000 7f34ef5ff700 1 -- 172.21.15.27:0/2087639639 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e8152000 con 0x7f34e8069af0 2024-01-11T09:43:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.523+0000 7f34ef5ff700 1 -- 172.21.15.27:0/2087639639 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e81525c0 con 0x7f34e8069af0 2024-01-11T09:43:25.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.524+0000 7f34ef5ff700 1 -- 172.21.15.27:0/2087639639 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e804c3c0 con 0x7f34e8069af0 2024-01-11T09:43:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.525+0000 7f34dd7fa700 1 -- 172.21.15.27:0/2087639639 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 113) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f34d80123d0 con 0x7f34e8069af0 2024-01-11T09:43:25.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.526+0000 7f34dd7fa700 1 --2- 172.21.15.27:0/2087639639 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f34cc067050 0x7f34cc069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:25.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.527+0000 7f34dd7fa700 1 -- 172.21.15.27:0/2087639639 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6349+0+0 (secure 0 0 0) 0x7f34d80848c0 con 0x7f34e8069af0 2024-01-11T09:43:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.527+0000 7f34ed39b700 1 -- 172.21.15.27:0/2087639639 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f34cc067050 msgr2=0x7f34cc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.527+0000 7f34ed39b700 1 --2- 172.21.15.27:0/2087639639 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f34cc067050 0x7f34cc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:43:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.531+0000 7f34dd7fa700 1 -- 172.21.15.27:0/2087639639 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f34d8051700 con 0x7f34e8069af0 2024-01-11T09:43:25.560 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:25.568+0000 7f7e4dd341c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:43:25.655 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:25.663+0000 7f7e4dd341c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:43:25.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.727+0000 7f34ed39b700 1 -- 172.21.15.27:0/2087639639 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f34cc067050 msgr2=0x7f34cc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:25.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.727+0000 7f34ed39b700 1 --2- 172.21.15.27:0/2087639639 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f34cc067050 0x7f34cc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:43:25.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.779+0000 7f34ef5ff700 1 -- 172.21.15.27:0/2087639639 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34e81519d0 con 0x7f34e8069af0 2024-01-11T09:43:25.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.779+0000 7f34dd7fa700 1 -- 172.21.15.27:0/2087639639 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4601 (secure 0 0 0) 0x7f34d8054d20 con 0x7f34e8069af0 2024-01-11T09:43:25.772 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:43:25.772 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8015,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1003297202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1003297202},{"type":"v1","addr":"172.21.15.77:6834","nonce":1003297202}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":8018,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3321059147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":3321059147},{"type":"v1","addr":"172.21.15.77:6835","nonce":3321059147}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":9202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2606639222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2606639222},{"type":"v1","addr":"172.21.15.27:6835","nonce":2606639222}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:27:21.470488+0000","modified":"2024-01-11T09:27:24.992917+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9564":{"gid":9564,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/3897592862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3897592862},{"type":"v1","addr":"172.21.15.27:6834","nonce":3897592862}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2024-01-11T09:43:25.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.782+0000 7f34ef5ff700 1 -- 172.21.15.27:0/2087639639 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f34cc067050 msgr2=0x7f34cc069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:43:25.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.782+0000 7f34ef5ff700 1 --2- 172.21.15.27:0/2087639639 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f34cc067050 0x7f34cc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.782+0000 7f34ef5ff700 1 -- 172.21.15.27:0/2087639639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e8069af0 msgr2=0x7f34e80c4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:25.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.782+0000 7f34ef5ff700 1 --2- 172.21.15.27:0/2087639639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e8069af0 0x7f34e80c4d10 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f34d800e900 tx=0x7f34d80091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.782+0000 7f34ef5ff700 1 -- 172.21.15.27:0/2087639639 shutdown_connections 2024-01-11T09:43:25.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.782+0000 7f34ef5ff700 1 --2- 172.21.15.27:0/2087639639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34e80c8770 0x7f34e80c5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.782+0000 7f34ef5ff700 1 --2- 172.21.15.27:0/2087639639 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34e8069720 0x7f34e8151400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.782+0000 7f34ef5ff700 1 --2- 172.21.15.27:0/2087639639 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f34cc067050 0x7f34cc069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.782+0000 7f34ef5ff700 1 --2- 172.21.15.27:0/2087639639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e8069af0 0x7f34e80c4d10 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:25.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.782+0000 7f34ef5ff700 1 -- 172.21.15.27:0/2087639639 >> 172.21.15.27:0/2087639639 conn(0x7f34e80680a0 msgr2=0x7f34e80c0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:25.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.782+0000 7f34ef5ff700 1 -- 172.21.15.27:0/2087639639 shutdown_connections 2024-01-11T09:43:25.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:25.782+0000 7f34ef5ff700 1 -- 172.21.15.27:0/2087639639 wait complete. 2024-01-11T09:43:25.776 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 94 2024-01-11T09:43:25.779 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:25.788+0000 7fcf2074e1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:43:25.793 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:43:25.968 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:25.977+0000 7fcf2074e1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:43:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.005+0000 7f73e57ff700 1 Processor -- start 2024-01-11T09:43:25.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.006+0000 7f73e57ff700 1 -- start start 2024-01-11T09:43:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.008+0000 7f73e57ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f73e01599e0 0x7f73e00b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.008+0000 7f73e57ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f73e0159db0 0x7f73e015a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.008+0000 7f73e57ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e0157e30 con 0x7f73e014fc20 2024-01-11T09:43:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.008+0000 7f73e57ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e0157500 con 0x7f73e01599e0 2024-01-11T09:43:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.008+0000 7f73e57ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e01577e0 con 0x7f73e0159db0 2024-01-11T09:43:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.008+0000 7f73deffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f73e01599e0 0x7f73e00b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.008+0000 7f73deffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f73e01599e0 0x7f73e00b0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50816/0 (socket says 172.21.15.27:50816) 2024-01-11T09:43:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.008+0000 7f73deffd700 1 -- 172.21.15.27:0/668577068 learned_addr learned my addr 172.21.15.27:0/668577068 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.009+0000 7f73de7fc700 1 --2- 172.21.15.27:0/668577068 >> v2:172.21.15.77:3300/0 conn(0x7f73e0159db0 0x7f73e015a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.009+0000 7f73deffd700 1 -- 172.21.15.27:0/668577068 >> v2:172.21.15.77:3300/0 conn(0x7f73e0159db0 msgr2=0x7f73e015a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.009+0000 7f73deffd700 1 --2- 172.21.15.27:0/668577068 >> v2:172.21.15.77:3300/0 conn(0x7f73e0159db0 0x7f73e015a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.009+0000 7f73deffd700 1 -- 172.21.15.27:0/668577068 >> v1:172.21.15.27:6789/0 conn(0x7f73e014fc20 legacy=0x7f73e0150080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:26.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.009+0000 7f73deffd700 1 -- 172.21.15.27:0/668577068 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e0157ac0 con 0x7f73e01599e0 2024-01-11T09:43:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.009+0000 7f73deffd700 1 --2- 172.21.15.27:0/668577068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e01599e0 0x7f73e00b0130 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f73e006d3f0 tx=0x7f73d000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5785e064d360a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.009+0000 7f73ddffb700 1 -- 172.21.15.27:0/668577068 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d000dbb0 con 0x7f73e01599e0 2024-01-11T09:43:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.009+0000 7f73ddffb700 1 -- 172.21.15.27:0/668577068 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d000dd10 con 0x7f73e01599e0 2024-01-11T09:43:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.009+0000 7f73ddffb700 1 -- 172.21.15.27:0/668577068 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d00196b0 con 0x7f73e01599e0 2024-01-11T09:43:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.010+0000 7f73e57ff700 1 -- 172.21.15.27:0/668577068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e01599e0 msgr2=0x7f73e00b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.010+0000 7f73e57ff700 1 --2- 172.21.15.27:0/668577068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e01599e0 0x7f73e00b0130 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f73e006d3f0 tx=0x7f73d000b4f0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.010+0000 7f73e57ff700 1 -- 172.21.15.27:0/668577068 shutdown_connections 2024-01-11T09:43:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.010+0000 7f73e57ff700 1 --2- 172.21.15.27:0/668577068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e01599e0 0x7f73e00b0130 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.010+0000 7f73e57ff700 1 --2- 172.21.15.27:0/668577068 >> v2:172.21.15.77:3300/0 conn(0x7f73e0159db0 0x7f73e015a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.010+0000 7f73e57ff700 1 -- 172.21.15.27:0/668577068 >> 172.21.15.27:0/668577068 conn(0x7f73e00680e0 msgr2=0x7f73e0069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.010+0000 7f73e57ff700 1 -- 172.21.15.27:0/668577068 shutdown_connections 2024-01-11T09:43:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.011+0000 7f73e57ff700 1 -- 172.21.15.27:0/668577068 wait complete. 2024-01-11T09:43:26.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.013+0000 7f73e57ff700 1 Processor -- start 2024-01-11T09:43:26.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.013+0000 7f73e57ff700 1 -- start start 2024-01-11T09:43:26.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.014+0000 7f73e57ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73e014fc20 0x7f73e01495e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:26.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.014+0000 7f73e57ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e01599e0 0x7f73e014c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:26.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.014+0000 7f73e57ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73e0159db0 0x7f73e014c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:26.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.014+0000 7f73e57ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e01694e0 con 0x7f73e01599e0 2024-01-11T09:43:26.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.014+0000 7f73e57ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0169780 con 0x7f73e0159db0 2024-01-11T09:43:26.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.014+0000 7f73e57ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0169a60 con 0x7f73e014fc20 2024-01-11T09:43:26.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.014+0000 7f73de7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e01599e0 0x7f73e014c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:26.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.014+0000 7f73de7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e01599e0 0x7f73e014c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50832/0 (socket says 172.21.15.27:50832) 2024-01-11T09:43:26.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.014+0000 7f73de7fc700 1 -- 172.21.15.27:0/966267855 learned_addr learned my addr 172.21.15.27:0/966267855 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:26.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.015+0000 7f73deffd700 1 --2- 172.21.15.27:0/966267855 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73e014fc20 0x7f73e01495e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:26.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.015+0000 7f73df7fe700 1 --2- 172.21.15.27:0/966267855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73e0159db0 0x7f73e014c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:26.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.015+0000 7f73de7fc700 1 -- 172.21.15.27:0/966267855 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73e014fc20 msgr2=0x7f73e01495e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:26.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.015+0000 7f73de7fc700 1 --2- 172.21.15.27:0/966267855 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73e014fc20 0x7f73e01495e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.015+0000 7f73de7fc700 1 -- 172.21.15.27:0/966267855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73e0159db0 msgr2=0x7f73e014c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:26.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.015+0000 7f73de7fc700 1 --2- 172.21.15.27:0/966267855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73e0159db0 0x7f73e014c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.015+0000 7f73de7fc700 1 -- 172.21.15.27:0/966267855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d000b790 con 0x7f73e01599e0 2024-01-11T09:43:26.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.015+0000 7f73de7fc700 1 --2- 172.21.15.27:0/966267855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e01599e0 0x7f73e014c260 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f73d400e930 tx=0x7f73d40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:26.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.016+0000 7f73c6ffd700 1 -- 172.21.15.27:0/966267855 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d4013680 con 0x7f73e01599e0 2024-01-11T09:43:26.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.016+0000 7f73c6ffd700 1 -- 172.21.15.27:0/966267855 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d40137e0 con 0x7f73e01599e0 2024-01-11T09:43:26.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.016+0000 7f73e57ff700 1 -- 172.21.15.27:0/966267855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e0169d40 con 0x7f73e01599e0 2024-01-11T09:43:26.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.016+0000 7f73c6ffd700 1 -- 172.21.15.27:0/966267855 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d4013b80 con 0x7f73e01599e0 2024-01-11T09:43:26.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.016+0000 7f73e57ff700 1 -- 172.21.15.27:0/966267855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e0149b20 con 0x7f73e01599e0 2024-01-11T09:43:26.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.017+0000 7f73e57ff700 1 -- 172.21.15.27:0/966267855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e00718d0 con 0x7f73e01599e0 2024-01-11T09:43:26.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.018+0000 7f73c6ffd700 1 -- 172.21.15.27:0/966267855 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 113) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f73d400c3e0 con 0x7f73e01599e0 2024-01-11T09:43:26.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.019+0000 7f73c6ffd700 1 --2- 172.21.15.27:0/966267855 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f73cc067050 0x7f73cc069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:26.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.019+0000 7f73c6ffd700 1 -- 172.21.15.27:0/966267855 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6349+0+0 (secure 0 0 0) 0x7f73d4084ac0 con 0x7f73e01599e0 2024-01-11T09:43:26.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.019+0000 7f73deffd700 1 -- 172.21.15.27:0/966267855 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f73cc067050 msgr2=0x7f73cc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:26.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.019+0000 7f73deffd700 1 --2- 172.21.15.27:0/966267855 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f73cc067050 0x7f73cc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:43:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.023+0000 7f73c6ffd700 1 -- 172.21.15.27:0/966267855 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f73d4051900 con 0x7f73e01599e0 2024-01-11T09:43:26.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.220+0000 7f73deffd700 1 -- 172.21.15.27:0/966267855 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f73cc067050 msgr2=0x7f73cc069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:26.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.220+0000 7f73deffd700 1 --2- 172.21.15.27:0/966267855 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f73cc067050 0x7f73cc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:43:26.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.228+0000 7f73e57ff700 1 -- 172.21.15.27:0/966267855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73e00cc980 con 0x7f73e01599e0 2024-01-11T09:43:26.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.229+0000 7f73c6ffd700 1 -- 172.21.15.27:0/966267855 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) v1 ==== 74+0+16282 (secure 0 0 0) 0x7f73d4054f20 con 0x7f73e01599e0 2024-01-11T09:43:26.222 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:43:26.222 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":185,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:43:21.972966+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:27:19.009515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-01-11T09:27:20.468964+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":185,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1160908778":"2024-01-12T09:38:23.806141+0000","172.21.15.77:0/3404938947":"2024-01-12T09:38:23.806141+0000","172.21.15.77:0/2730921345":"2024-01-12T09:38:23.806141+0000","172.21.15.77:0/2548321690":"2024-01-12T09:38:23.806141+0000","172.21.15.77:0/3966666871":"2024-01-12T09:38:23.806141+0000","172.21.15.77:6837/41381":"2024-01-12T09:27:37.837169+0000","172.21.15.77:6836/41381":"2024-01-12T09:27:37.837169+0000","172.21.15.77:0/1337020578":"2024-01-12T09:27:37.837169+0000","172.21.15.77:0/2253497474":"2024-01-12T09:27:37.837169+0000","172.21.15.77:0/2186545586":"2024-01-12T09:38:23.806141+0000","172.21.15.77:0/2706846520":"2024-01-12T09:27:37.837169+0000","172.21.15.77:0/2650034185":"2024-01-12T09:27:37.837169+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":23,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:43:26.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.231+0000 7f73e57ff700 1 -- 172.21.15.27:0/966267855 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f73cc067050 msgr2=0x7f73cc069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:43:26.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.231+0000 7f73e57ff700 1 --2- 172.21.15.27:0/966267855 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f73cc067050 0x7f73cc069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.231+0000 7f73e57ff700 1 -- 172.21.15.27:0/966267855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e01599e0 msgr2=0x7f73e014c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:26.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.231+0000 7f73e57ff700 1 --2- 172.21.15.27:0/966267855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e01599e0 0x7f73e014c260 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f73d400e930 tx=0x7f73d40091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.231+0000 7f73e57ff700 1 -- 172.21.15.27:0/966267855 shutdown_connections 2024-01-11T09:43:26.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.231+0000 7f73e57ff700 1 --2- 172.21.15.27:0/966267855 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f73cc067050 0x7f73cc069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.231+0000 7f73e57ff700 1 --2- 172.21.15.27:0/966267855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73e0159db0 0x7f73e014c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.231+0000 7f73e57ff700 1 --2- 172.21.15.27:0/966267855 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f73e014fc20 0x7f73e01495e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.231+0000 7f73e57ff700 1 --2- 172.21.15.27:0/966267855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73e01599e0 0x7f73e014c260 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.231+0000 7f73e57ff700 1 -- 172.21.15.27:0/966267855 >> 172.21.15.27:0/966267855 conn(0x7f73e00680e0 msgr2=0x7f73e01582c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:26.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.232+0000 7f73e57ff700 1 -- 172.21.15.27:0/966267855 shutdown_connections 2024-01-11T09:43:26.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.232+0000 7f73e57ff700 1 -- 172.21.15.27:0/966267855 wait complete. 2024-01-11T09:43:26.243 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-11T09:43:26.249 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:26.257+0000 7f7e4dd341c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:43:26.348 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:26.357+0000 7fcf2074e1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:43:26.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.452+0000 7f8d771ff700 1 Processor -- start 2024-01-11T09:43:26.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.452+0000 7f8d771ff700 1 -- start start 2024-01-11T09:43:26.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.454+0000 7f8d771ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d70067c30 0x7f8d701575f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:26.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.454+0000 7f8d771ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7015b100 con 0x7f8d70067860 2024-01-11T09:43:26.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.454+0000 7f8d771ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7015b350 con 0x7f8d700c8020 2024-01-11T09:43:26.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.454+0000 7f8d771ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d700afcc0 con 0x7f8d70067c30 2024-01-11T09:43:26.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.454+0000 7f8d7579c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d70067c30 0x7f8d701575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:26.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.454+0000 7f8d74f9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8d70067860 0x7f8d700c7910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55956/0 (socket says 172.21.15.27:55956) 2024-01-11T09:43:26.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.454+0000 7f8d6ffff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8d700c8020 0x7f8d70156ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:57296/0 (socket says 172.21.15.27:57296) 2024-01-11T09:43:26.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.454+0000 7f8d6ffff700 1 -- 172.21.15.27:0/2881093987 learned_addr learned my addr 172.21.15.27:0/2881093987 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:26.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.454+0000 7f8d7579c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d70067c30 0x7f8d701575f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50834/0 (socket says 172.21.15.27:50834) 2024-01-11T09:43:26.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.455+0000 7f8d6f7fe700 1 -- 172.21.15.27:0/2881093987 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610814150 0 0) 0x7f8d7015b350 con 0x7f8d700c8020 2024-01-11T09:43:26.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.455+0000 7f8d7579c700 1 -- 172.21.15.27:0/2881093987 >> v1:172.21.15.77:6789/0 conn(0x7f8d700c8020 legacy=0x7f8d70156ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:26.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.455+0000 7f8d7579c700 1 -- 172.21.15.27:0/2881093987 >> v1:172.21.15.27:6789/0 conn(0x7f8d70067860 legacy=0x7f8d700c7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:26.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.455+0000 7f8d7579c700 1 -- 172.21.15.27:0/2881093987 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7015b630 con 0x7f8d70067c30 2024-01-11T09:43:26.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.455+0000 7f8d7579c700 1 --2- 172.21.15.27:0/2881093987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d70067c30 0x7f8d701575f0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f8d68009e60 tx=0x7f8d68009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d04d32bbb75a2ac server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:26.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.455+0000 7f8d6f7fe700 1 -- 172.21.15.27:0/2881093987 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6800e070 con 0x7f8d70067c30 2024-01-11T09:43:26.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.455+0000 7f8d6f7fe700 1 -- 172.21.15.27:0/2881093987 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6800bc30 con 0x7f8d70067c30 2024-01-11T09:43:26.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.455+0000 7f8d6f7fe700 1 -- 172.21.15.27:0/2881093987 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d68012430 con 0x7f8d70067c30 2024-01-11T09:43:26.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.456+0000 7f8d771ff700 1 -- 172.21.15.27:0/2881093987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d70067c30 msgr2=0x7f8d701575f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:26.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.456+0000 7f8d771ff700 1 --2- 172.21.15.27:0/2881093987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d70067c30 0x7f8d701575f0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f8d68009e60 tx=0x7f8d68009780 comp rx=0 tx=0).stop 2024-01-11T09:43:26.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.456+0000 7f8d771ff700 1 -- 172.21.15.27:0/2881093987 shutdown_connections 2024-01-11T09:43:26.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.456+0000 7f8d771ff700 1 --2- 172.21.15.27:0/2881093987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d70067c30 0x7f8d701575f0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.456+0000 7f8d771ff700 1 -- 172.21.15.27:0/2881093987 >> 172.21.15.27:0/2881093987 conn(0x7f8d7006b170 msgr2=0x7f8d700cd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:26.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.456+0000 7f8d771ff700 1 -- 172.21.15.27:0/2881093987 shutdown_connections 2024-01-11T09:43:26.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.456+0000 7f8d771ff700 1 -- 172.21.15.27:0/2881093987 wait complete. 2024-01-11T09:43:26.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.458+0000 7f8d771ff700 1 Processor -- start 2024-01-11T09:43:26.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.458+0000 7f8d771ff700 1 -- start start 2024-01-11T09:43:26.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.459+0000 7f8d771ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d70067860 0x7f8d700c4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:26.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.459+0000 7f8d771ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d70067c30 0x7f8d700b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:26.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.459+0000 7f8d771ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d700c8020 0x7f8d70152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:26.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.459+0000 7f8d771ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d700c7730 con 0x7f8d700c8020 2024-01-11T09:43:26.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.459+0000 7f8d771ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d700c5370 con 0x7f8d70067c30 2024-01-11T09:43:26.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.459+0000 7f8d771ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7016a470 con 0x7f8d70067860 2024-01-11T09:43:26.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.459+0000 7f8d7579c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d700c8020 0x7f8d70152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:26.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.459+0000 7f8d6ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d70067c30 0x7f8d700b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:26.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.459+0000 7f8d7579c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d700c8020 0x7f8d70152b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50848/0 (socket says 172.21.15.27:50848) 2024-01-11T09:43:26.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.459+0000 7f8d7579c700 1 -- 172.21.15.27:0/3553068364 learned_addr learned my addr 172.21.15.27:0/3553068364 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:26.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.459+0000 7f8d74f9b700 1 --2- 172.21.15.27:0/3553068364 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d70067860 0x7f8d700c4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:26.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.459+0000 7f8d7579c700 1 -- 172.21.15.27:0/3553068364 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d70067860 msgr2=0x7f8d700c4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:26.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.459+0000 7f8d7579c700 1 --2- 172.21.15.27:0/3553068364 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d70067860 0x7f8d700c4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.460+0000 7f8d7579c700 1 -- 172.21.15.27:0/3553068364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d70067c30 msgr2=0x7f8d700b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:26.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.460+0000 7f8d7579c700 1 --2- 172.21.15.27:0/3553068364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d70067c30 0x7f8d700b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.460+0000 7f8d7579c700 1 -- 172.21.15.27:0/3553068364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d68009a20 con 0x7f8d700c8020 2024-01-11T09:43:26.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.460+0000 7f8d7579c700 1 --2- 172.21.15.27:0/3553068364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d700c8020 0x7f8d70152b20 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f8d680099f0 tx=0x7f8d68019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:26.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.460+0000 7f8d6cff9700 1 -- 172.21.15.27:0/3553068364 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d68012e10 con 0x7f8d700c8020 2024-01-11T09:43:26.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.460+0000 7f8d6cff9700 1 -- 172.21.15.27:0/3553068364 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6801d630 con 0x7f8d700c8020 2024-01-11T09:43:26.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.460+0000 7f8d6cff9700 1 -- 172.21.15.27:0/3553068364 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6801d9f0 con 0x7f8d700c8020 2024-01-11T09:43:26.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.460+0000 7f8d771ff700 1 -- 172.21.15.27:0/3553068364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7016a680 con 0x7f8d700c8020 2024-01-11T09:43:26.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.460+0000 7f8d771ff700 1 -- 172.21.15.27:0/3553068364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d700c4760 con 0x7f8d700c8020 2024-01-11T09:43:26.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.461+0000 7f8d771ff700 1 -- 172.21.15.27:0/3553068364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d700c8fa0 con 0x7f8d700c8020 2024-01-11T09:43:26.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.462+0000 7f8d6cff9700 1 -- 172.21.15.27:0/3553068364 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 113) v1 ==== 84856+0+0 (secure 0 0 0) 0x7f8d68024070 con 0x7f8d700c8020 2024-01-11T09:43:26.455 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:26.463+0000 7f7e4dd341c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:43:26.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.463+0000 7f8d6cff9700 1 --2- 172.21.15.27:0/3553068364 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d54067050 0x7f8d54069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:26.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.463+0000 7f8d6cff9700 1 -- 172.21.15.27:0/3553068364 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6349+0+0 (secure 0 0 0) 0x7f8d680871e0 con 0x7f8d700c8020 2024-01-11T09:43:26.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.463+0000 7f8d74f9b700 1 -- 172.21.15.27:0/3553068364 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d54067050 msgr2=0x7f8d54069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:26.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.463+0000 7f8d74f9b700 1 --2- 172.21.15.27:0/3553068364 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d54067050 0x7f8d54069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:43:26.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.467+0000 7f8d6cff9700 1 -- 172.21.15.27:0/3553068364 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8d68054020 con 0x7f8d700c8020 2024-01-11T09:43:26.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.664+0000 7f8d74f9b700 1 -- 172.21.15.27:0/3553068364 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d54067050 msgr2=0x7f8d54069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:26.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.664+0000 7f8d74f9b700 1 --2- 172.21.15.27:0/3553068364 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d54067050 0x7f8d54069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:43:26.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.716+0000 7f8d771ff700 1 -- 172.21.15.27:0/3553068364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8d7004c3c0 con 0x7f8d700c8020 2024-01-11T09:43:26.710 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-11T09:43:26.717+0000 7f27065f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-11T09:43:26.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.723+0000 7f8d6cff9700 1 -- 172.21.15.27:0/3553068364 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v95) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8d68057640 con 0x7f8d700c8020 2024-01-11T09:43:26.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.726+0000 7f8d771ff700 1 -- 172.21.15.27:0/3553068364 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d54067050 msgr2=0x7f8d54069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:43:26.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.726+0000 7f8d771ff700 1 --2- 172.21.15.27:0/3553068364 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d54067050 0x7f8d54069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.726+0000 7f8d771ff700 1 -- 172.21.15.27:0/3553068364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d700c8020 msgr2=0x7f8d70152b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:26.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.726+0000 7f8d771ff700 1 --2- 172.21.15.27:0/3553068364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d700c8020 0x7f8d70152b20 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f8d680099f0 tx=0x7f8d68019000 comp rx=0 tx=0).stop 2024-01-11T09:43:26.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.726+0000 7f8d771ff700 1 -- 172.21.15.27:0/3553068364 shutdown_connections 2024-01-11T09:43:26.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.726+0000 7f8d771ff700 1 --2- 172.21.15.27:0/3553068364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d700c8020 0x7f8d70152b20 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.726+0000 7f8d771ff700 1 --2- 172.21.15.27:0/3553068364 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d70067860 0x7f8d700c4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.726+0000 7f8d771ff700 1 --2- 172.21.15.27:0/3553068364 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d54067050 0x7f8d54069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.726+0000 7f8d771ff700 1 --2- 172.21.15.27:0/3553068364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d70067c30 0x7f8d700b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.726+0000 7f8d771ff700 1 -- 172.21.15.27:0/3553068364 >> 172.21.15.27:0/3553068364 conn(0x7f8d7006b170 msgr2=0x7f8d700c2350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:26.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.726+0000 7f8d771ff700 1 -- 172.21.15.27:0/3553068364 shutdown_connections 2024-01-11T09:43:26.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.727+0000 7f8d771ff700 1 -- 172.21.15.27:0/3553068364 wait complete. 2024-01-11T09:43:26.722 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-11T09:43:26.738 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-11T09:43:26.775 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:26.784+0000 7fcf2074e1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:43:26.785 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T09:43:26.870 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:26.877+0000 7f7e4dd341c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:43:26.959 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:26.968+0000 7fcf2074e1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:43:26.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.991+0000 7ff8b1fff700 1 Processor -- start 2024-01-11T09:43:26.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.992+0000 7ff8b1fff700 1 -- start start 2024-01-11T09:43:26.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.993+0000 7ff8b1fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8ac0ccd70 0x7ff8ac0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:26.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.993+0000 7ff8b1fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff8ac15ae70 0x7ff8ac15b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:26.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.993+0000 7ff8b1fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac0cf850 con 0x7ff8ac15a1b0 2024-01-11T09:43:26.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.993+0000 7ff8b1fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8ac15ac90 con 0x7ff8ac15ae70 2024-01-11T09:43:26.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.993+0000 7ff8b1fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac0cec40 con 0x7ff8ac0ccd70 2024-01-11T09:43:26.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.993+0000 7ff8aaffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8ac0ccd70 0x7ff8ac0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:26.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.993+0000 7ff8abfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff8ac15ae70 0x7ff8ac15b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:26.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.993+0000 7ff8ab7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff8ac15a1b0 0x7ff8ac15a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:57308/0 (socket says 172.21.15.27:57308) 2024-01-11T09:43:26.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.993+0000 7ff8ab7fe700 1 -- 172.21.15.27:0/953722755 learned_addr learned my addr 172.21.15.27:0/953722755 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:26.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.994+0000 7ff8abfff700 1 -- 172.21.15.27:0/953722755 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8ac0ccd70 msgr2=0x7ff8ac0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:26.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.994+0000 7ff8abfff700 1 --2- 172.21.15.27:0/953722755 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8ac0ccd70 0x7ff8ac0cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.994+0000 7ff8abfff700 1 -- 172.21.15.27:0/953722755 >> v1:172.21.15.77:6789/0 conn(0x7ff8ac15a1b0 legacy=0x7ff8ac15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:26.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.994+0000 7ff8abfff700 1 -- 172.21.15.27:0/953722755 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ac0cef20 con 0x7ff8ac15ae70 2024-01-11T09:43:26.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.994+0000 7ff8abfff700 1 --2- 172.21.15.27:0/953722755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8ac15ae70 0x7ff8ac15b2f0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4009e60 tx=0x7ff8a4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b2613949507a80b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:26.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.994+0000 7ff8aa7fc700 1 -- 172.21.15.27:0/953722755 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a400e070 con 0x7ff8ac15ae70 2024-01-11T09:43:26.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.994+0000 7ff8aa7fc700 1 -- 172.21.15.27:0/953722755 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a400bb60 con 0x7ff8ac15ae70 2024-01-11T09:43:26.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.994+0000 7ff8aa7fc700 1 -- 172.21.15.27:0/953722755 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4012430 con 0x7ff8ac15ae70 2024-01-11T09:43:26.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.995+0000 7ff8b1fff700 1 -- 172.21.15.27:0/953722755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8ac15ae70 msgr2=0x7ff8ac15b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:26.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.995+0000 7ff8b1fff700 1 --2- 172.21.15.27:0/953722755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8ac15ae70 0x7ff8ac15b2f0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4009e60 tx=0x7ff8a4009780 comp rx=0 tx=0).stop 2024-01-11T09:43:26.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.995+0000 7ff8b1fff700 1 -- 172.21.15.27:0/953722755 shutdown_connections 2024-01-11T09:43:26.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.995+0000 7ff8b1fff700 1 --2- 172.21.15.27:0/953722755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8ac15ae70 0x7ff8ac15b2f0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.995+0000 7ff8b1fff700 1 --2- 172.21.15.27:0/953722755 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8ac0ccd70 0x7ff8ac0cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:26.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.995+0000 7ff8b1fff700 1 -- 172.21.15.27:0/953722755 >> 172.21.15.27:0/953722755 conn(0x7ff8ac06ed40 msgr2=0x7ff8ac06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:26.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.995+0000 7ff8b1fff700 1 -- 172.21.15.27:0/953722755 shutdown_connections 2024-01-11T09:43:26.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.996+0000 7ff8b1fff700 1 -- 172.21.15.27:0/953722755 wait complete. 2024-01-11T09:43:26.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.998+0000 7ff8b1fff700 1 Processor -- start 2024-01-11T09:43:26.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:26.998+0000 7ff8b1fff700 1 -- start start 2024-01-11T09:43:26.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.000+0000 7ff8b1fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8ac0ccd70 0x7ff8ac0ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:26.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.000+0000 7ff8b1fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8ac15a1b0 0x7ff8ac169720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:26.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.000+0000 7ff8b1fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8ac15ae70 0x7ff8ac169c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:26.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.000+0000 7ff8b1fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac0b5d30 con 0x7ff8ac0ccd70 2024-01-11T09:43:26.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.000+0000 7ff8b1fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac0ca930 con 0x7ff8ac15ae70 2024-01-11T09:43:26.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.000+0000 7ff8b1fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac157cc0 con 0x7ff8ac15a1b0 2024-01-11T09:43:27.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.000+0000 7ff8ab7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8ac0ccd70 0x7ff8ac0ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:27.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.000+0000 7ff8ab7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8ac0ccd70 0x7ff8ac0ca3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50884/0 (socket says 172.21.15.27:50884) 2024-01-11T09:43:27.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.000+0000 7ff8ab7fe700 1 -- 172.21.15.27:0/2668097904 learned_addr learned my addr 172.21.15.27:0/2668097904 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:27.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.000+0000 7ff8aaffd700 1 --2- 172.21.15.27:0/2668097904 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8ac15a1b0 0x7ff8ac169720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:27.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.000+0000 7ff8abfff700 1 --2- 172.21.15.27:0/2668097904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8ac15ae70 0x7ff8ac169c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:27.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.000+0000 7ff8ab7fe700 1 -- 172.21.15.27:0/2668097904 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8ac15a1b0 msgr2=0x7ff8ac169720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:27.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.000+0000 7ff8ab7fe700 1 --2- 172.21.15.27:0/2668097904 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8ac15a1b0 0x7ff8ac169720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:27.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.000+0000 7ff8ab7fe700 1 -- 172.21.15.27:0/2668097904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8ac15ae70 msgr2=0x7ff8ac169c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:27.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.000+0000 7ff8ab7fe700 1 --2- 172.21.15.27:0/2668097904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8ac15ae70 0x7ff8ac169c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:27.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.000+0000 7ff8ab7fe700 1 -- 172.21.15.27:0/2668097904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a4009a20 con 0x7ff8ac0ccd70 2024-01-11T09:43:27.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.001+0000 7ff8ab7fe700 1 --2- 172.21.15.27:0/2668097904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8ac0ccd70 0x7ff8ac0ca3f0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7ff89c00c7c0 tx=0x7ff89c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:27.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.001+0000 7ff8937fe700 1 -- 172.21.15.27:0/2668097904 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c005b30 con 0x7ff8ac0ccd70 2024-01-11T09:43:27.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.001+0000 7ff8937fe700 1 -- 172.21.15.27:0/2668097904 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff89c005c90 con 0x7ff8ac0ccd70 2024-01-11T09:43:27.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.001+0000 7ff8b1fff700 1 -- 172.21.15.27:0/2668097904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ac16ca70 con 0x7ff8ac0ccd70 2024-01-11T09:43:27.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.001+0000 7ff8937fe700 1 -- 172.21.15.27:0/2668097904 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c004a80 con 0x7ff8ac0ccd70 2024-01-11T09:43:27.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.001+0000 7ff8b1fff700 1 -- 172.21.15.27:0/2668097904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ac0b5400 con 0x7ff8ac0ccd70 2024-01-11T09:43:27.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.002+0000 7ff8b1fff700 1 -- 172.21.15.27:0/2668097904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8ac04c3c0 con 0x7ff8ac0ccd70 2024-01-11T09:43:27.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.003+0000 7ff8937fe700 1 -- 172.21.15.27:0/2668097904 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 113) v1 ==== 84856+0+0 (secure 0 0 0) 0x7ff89c0070d0 con 0x7ff8ac0ccd70 2024-01-11T09:43:27.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.005+0000 7ff8937fe700 1 --2- 172.21.15.27:0/2668097904 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff898067050 0x7ff898069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:27.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.005+0000 7ff8937fe700 1 -- 172.21.15.27:0/2668097904 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6391+0+0 (secure 0 0 0) 0x7ff89c083d00 con 0x7ff8ac0ccd70 2024-01-11T09:43:27.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.005+0000 7ff8aaffd700 1 -- 172.21.15.27:0/2668097904 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff898067050 msgr2=0x7ff898069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:27.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.005+0000 7ff8aaffd700 1 --2- 172.21.15.27:0/2668097904 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff898067050 0x7ff898069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:43:27.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.008+0000 7ff8937fe700 1 -- 172.21.15.27:0/2668097904 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff89c04b5a0 con 0x7ff8ac0ccd70 2024-01-11T09:43:27.151 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:27.160+0000 7fcf2074e1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:43:27.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.206+0000 7ff8aaffd700 1 -- 172.21.15.27:0/2668097904 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff898067050 msgr2=0x7ff898069500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:27.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.206+0000 7ff8aaffd700 1 --2- 172.21.15.27:0/2668097904 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff898067050 0x7ff898069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:43:27.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.256+0000 7ff8b1fff700 1 -- 172.21.15.27:0/2668097904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff8ac0b2d60 con 0x7ff8ac0ccd70 2024-01-11T09:43:27.261 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:27.270+0000 7fcf2074e1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:43:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.278+0000 7ff8937fe700 1 -- 172.21.15.27:0/2668097904 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 114) v1 ==== 84453+0+0 (secure 0 0 0) 0x7ff89c050ee0 con 0x7ff8ac0ccd70 2024-01-11T09:43:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.278+0000 7ff8937fe700 1 -- 172.21.15.27:0/2668097904 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff898067050 msgr2=0x7ff898069500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:43:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.278+0000 7ff8937fe700 1 --2- 172.21.15.27:0/2668097904 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff898067050 0x7ff898069500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:27.339 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:27.346+0000 7f7e4dd341c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:43:27.340 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:27.349+0000 7fce9ad83700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:27.340 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:27.349+0000 7fce9ad83700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:27.341 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:27.349+0000 7fce9ad83700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:27.341 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:27.349+0000 7fce9ad83700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:27.341 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:27.349+0000 7fce9e58a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:27.341 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:27.349+0000 7fce9ad83700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:27.341 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:27.349+0000 7fce9e58a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:27.341 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:27.349+0000 7fce9e58a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:27.341 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:27.349+0000 7fce9e58a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:27.341 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:27.349+0000 7fce9e58a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:27.534 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:27.542+0000 7f7e4dd341c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:43:27.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.722+0000 7ff8937fe700 1 -- 172.21.15.27:0/2668097904 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v96) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff89c07f740 con 0x7ff8ac0ccd70 2024-01-11T09:43:27.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.725+0000 7ff8b1fff700 1 -- 172.21.15.27:0/2668097904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8ac0ccd70 msgr2=0x7ff8ac0ca3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:27.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.725+0000 7ff8b1fff700 1 --2- 172.21.15.27:0/2668097904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8ac0ccd70 0x7ff8ac0ca3f0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7ff89c00c7c0 tx=0x7ff89c007f70 comp rx=0 tx=0).stop 2024-01-11T09:43:27.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.725+0000 7ff8b1fff700 1 -- 172.21.15.27:0/2668097904 shutdown_connections 2024-01-11T09:43:27.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.725+0000 7ff8b1fff700 1 --2- 172.21.15.27:0/2668097904 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff898067050 0x7ff898069500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:27.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.725+0000 7ff8b1fff700 1 --2- 172.21.15.27:0/2668097904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8ac15ae70 0x7ff8ac169c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:27.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.725+0000 7ff8b1fff700 1 --2- 172.21.15.27:0/2668097904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8ac0ccd70 0x7ff8ac0ca3f0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:27.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.725+0000 7ff8b1fff700 1 --2- 172.21.15.27:0/2668097904 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8ac15a1b0 0x7ff8ac169720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:27.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.725+0000 7ff8b1fff700 1 -- 172.21.15.27:0/2668097904 >> 172.21.15.27:0/2668097904 conn(0x7ff8ac06ed40 msgr2=0x7ff8ac0c5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:27.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.725+0000 7ff8b1fff700 1 -- 172.21.15.27:0/2668097904 shutdown_connections 2024-01-11T09:43:27.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.725+0000 7ff8b1fff700 1 -- 172.21.15.27:0/2668097904 wait complete. 2024-01-11T09:43:27.735 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:27.742+0000 7f7e4dd341c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:43:27.737 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-11T09:43:27.887 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:27.894+0000 7f7e4dd341c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:43:27.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.982+0000 7fe2225ff700 1 Processor -- start 2024-01-11T09:43:27.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.983+0000 7fe2225ff700 1 -- start start 2024-01-11T09:43:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.984+0000 7fe2225ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe21c156810 0x7fe21c06e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.984+0000 7fe2225ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21c0c8590 con 0x7fe21c0ccac0 2024-01-11T09:43:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.984+0000 7fe2225ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21c0c7980 con 0x7fe21c156440 2024-01-11T09:43:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.984+0000 7fe2225ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe21c0c7c60 con 0x7fe21c156810 2024-01-11T09:43:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.984+0000 7fe21b7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe21c156810 0x7fe21c06e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.984+0000 7fe220b9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe21c0ccac0 0x7fe21c0accb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55968/0 (socket says 172.21.15.27:55968) 2024-01-11T09:43:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.984+0000 7fe220b9c700 1 -- 172.21.15.27:0/3891663162 learned_addr learned my addr 172.21.15.27:0/3891663162 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.985+0000 7fe21b7fe700 1 -- 172.21.15.27:0/3891663162 >> v1:172.21.15.77:6789/0 conn(0x7fe21c156440 legacy=0x7fe21c0c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.985+0000 7fe21b7fe700 1 -- 172.21.15.27:0/3891663162 >> v1:172.21.15.27:6789/0 conn(0x7fe21c0ccac0 legacy=0x7fe21c0accb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.985+0000 7fe21b7fe700 1 -- 172.21.15.27:0/3891663162 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c0c7f40 con 0x7fe21c156810 2024-01-11T09:43:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.985+0000 7fe21affd700 1 -- 172.21.15.27:0/3891663162 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840700097 0 0) 0x7fe21c0c7980 con 0x7fe21c156440 2024-01-11T09:43:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.985+0000 7fe21b7fe700 1 --2- 172.21.15.27:0/3891663162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c156810 0x7fe21c06e720 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fe2100098e0 tx=0x7fe2100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=572e73c3f8ac22ae server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.985+0000 7fe21affd700 1 -- 172.21.15.27:0/3891663162 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe210009e80 con 0x7fe21c156810 2024-01-11T09:43:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.985+0000 7fe21affd700 1 -- 172.21.15.27:0/3891663162 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21000b760 con 0x7fe21c156810 2024-01-11T09:43:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.985+0000 7fe21affd700 1 -- 172.21.15.27:0/3891663162 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21000bae0 con 0x7fe21c156810 2024-01-11T09:43:27.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.986+0000 7fe2225ff700 1 -- 172.21.15.27:0/3891663162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c156810 msgr2=0x7fe21c06e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:27.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.986+0000 7fe2225ff700 1 --2- 172.21.15.27:0/3891663162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c156810 0x7fe21c06e720 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fe2100098e0 tx=0x7fe2100091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:27.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.986+0000 7fe2225ff700 1 -- 172.21.15.27:0/3891663162 shutdown_connections 2024-01-11T09:43:27.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.986+0000 7fe2225ff700 1 --2- 172.21.15.27:0/3891663162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c156810 0x7fe21c06e720 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:27.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.986+0000 7fe2225ff700 1 -- 172.21.15.27:0/3891663162 >> 172.21.15.27:0/3891663162 conn(0x7fe21c0680e0 msgr2=0x7fe21c0cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:27.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.986+0000 7fe2225ff700 1 -- 172.21.15.27:0/3891663162 shutdown_connections 2024-01-11T09:43:27.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.986+0000 7fe2225ff700 1 -- 172.21.15.27:0/3891663162 wait complete. 2024-01-11T09:43:27.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.988+0000 7fe2225ff700 1 Processor -- start 2024-01-11T09:43:27.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.988+0000 7fe2225ff700 1 -- start start 2024-01-11T09:43:27.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.989+0000 7fe2225ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe21c0ccac0 0x7fe21c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:27.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.989+0000 7fe2225ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c156440 0x7fe21c0b1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:27.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.989+0000 7fe2225ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe21c156810 0x7fe21c1549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:27.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.989+0000 7fe2225ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c0b1ea0 con 0x7fe21c156440 2024-01-11T09:43:27.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.989+0000 7fe2225ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c0b08c0 con 0x7fe21c156810 2024-01-11T09:43:27.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.989+0000 7fe2225ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c0b0ba0 con 0x7fe21c0ccac0 2024-01-11T09:43:27.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.989+0000 7fe21b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c156440 0x7fe21c0b1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:27.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.989+0000 7fe21b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c156440 0x7fe21c0b1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50908/0 (socket says 172.21.15.27:50908) 2024-01-11T09:43:27.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.989+0000 7fe21b7fe700 1 -- 172.21.15.27:0/757437278 learned_addr learned my addr 172.21.15.27:0/757437278 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:27.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.989+0000 7fe21bfff700 1 --2- 172.21.15.27:0/757437278 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe21c0ccac0 0x7fe21c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:27.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.989+0000 7fe220b9c700 1 --2- 172.21.15.27:0/757437278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe21c156810 0x7fe21c1549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:27.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.989+0000 7fe21b7fe700 1 -- 172.21.15.27:0/757437278 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe21c0ccac0 msgr2=0x7fe21c15b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:27.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.989+0000 7fe21b7fe700 1 --2- 172.21.15.27:0/757437278 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe21c0ccac0 0x7fe21c15b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:27.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.989+0000 7fe21b7fe700 1 -- 172.21.15.27:0/757437278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe21c156810 msgr2=0x7fe21c1549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:27.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.989+0000 7fe21b7fe700 1 --2- 172.21.15.27:0/757437278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe21c156810 0x7fe21c1549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:27.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.989+0000 7fe21b7fe700 1 -- 172.21.15.27:0/757437278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe210009420 con 0x7fe21c156440 2024-01-11T09:43:27.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.989+0000 7fe21bfff700 1 --2- 172.21.15.27:0/757437278 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe21c0ccac0 0x7fe21c15b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:27.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.989+0000 7fe220b9c700 1 --2- 172.21.15.27:0/757437278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe21c156810 0x7fe21c1549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:27.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.990+0000 7fe21b7fe700 1 --2- 172.21.15.27:0/757437278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c156440 0x7fe21c0b1960 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fe2100030c0 tx=0x7fe210016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:27.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.991+0000 7fe203fff700 1 -- 172.21.15.27:0/757437278 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21001c480 con 0x7fe21c156440 2024-01-11T09:43:27.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.991+0000 7fe203fff700 1 -- 172.21.15.27:0/757437278 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21001c5e0 con 0x7fe21c156440 2024-01-11T09:43:27.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.991+0000 7fe2225ff700 1 -- 172.21.15.27:0/757437278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21c0b0e80 con 0x7fe21c156440 2024-01-11T09:43:27.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.991+0000 7fe203fff700 1 -- 172.21.15.27:0/757437278 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21001c980 con 0x7fe21c156440 2024-01-11T09:43:27.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.991+0000 7fe2225ff700 1 -- 172.21.15.27:0/757437278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21c0b1440 con 0x7fe21c156440 2024-01-11T09:43:27.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.992+0000 7fe2225ff700 1 -- 172.21.15.27:0/757437278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21c04c3c0 con 0x7fe21c156440 2024-01-11T09:43:27.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.993+0000 7fe203fff700 1 -- 172.21.15.27:0/757437278 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 114) v1 ==== 84453+0+0 (secure 0 0 0) 0x7fe21001a450 con 0x7fe21c156440 2024-01-11T09:43:27.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.994+0000 7fe203fff700 1 -- 172.21.15.27:0/757437278 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fe210086820 con 0x7fe21c156440 2024-01-11T09:43:27.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:27.998+0000 7fe203fff700 1 -- 172.21.15.27:0/757437278 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe2100539c0 con 0x7fe21c156440 2024-01-11T09:43:28.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:28.200+0000 7fe2225ff700 1 -- 172.21.15.27:0/757437278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe21c0c8220 con 0x7fe21c156440 2024-01-11T09:43:28.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:28.281+0000 7fe203fff700 1 -- 172.21.15.27:0/757437278 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 115) v1 ==== 84580+0+0 (secure 0 0 0) 0x7fe210053470 con 0x7fe21c156440 2024-01-11T09:43:28.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:28.281+0000 7fe203fff700 1 --2- 172.21.15.27:0/757437278 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe20806ca30 0x7fe20806ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:28.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:28.282+0000 7fe21bfff700 1 --2- 172.21.15.27:0/757437278 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe20806ca30 0x7fe20806ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:28.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:28.286+0000 7fe21bfff700 1 --2- 172.21.15.27:0/757437278 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe20806ca30 0x7fe20806ee10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe21c15a1f0 tx=0x7fe20c009040 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:28.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.001+0000 7fe203fff700 1 -- 172.21.15.27:0/757437278 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v188) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe210027690 con 0x7fe21c156440 2024-01-11T09:43:28.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.005+0000 7fe2225ff700 1 -- 172.21.15.27:0/757437278 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe20806ca30 msgr2=0x7fe20806ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:28.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.005+0000 7fe2225ff700 1 --2- 172.21.15.27:0/757437278 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe20806ca30 0x7fe20806ee10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe21c15a1f0 tx=0x7fe20c009040 comp rx=0 tx=0).stop 2024-01-11T09:43:28.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.005+0000 7fe2225ff700 1 -- 172.21.15.27:0/757437278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c156440 msgr2=0x7fe21c0b1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:28.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.005+0000 7fe2225ff700 1 --2- 172.21.15.27:0/757437278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c156440 0x7fe21c0b1960 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fe2100030c0 tx=0x7fe210016000 comp rx=0 tx=0).stop 2024-01-11T09:43:28.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.005+0000 7fe2225ff700 1 -- 172.21.15.27:0/757437278 shutdown_connections 2024-01-11T09:43:28.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.005+0000 7fe2225ff700 1 --2- 172.21.15.27:0/757437278 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe20806ca30 0x7fe20806ee10 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:28.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.005+0000 7fe2225ff700 1 --2- 172.21.15.27:0/757437278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe21c156810 0x7fe21c1549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:28.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.005+0000 7fe2225ff700 1 --2- 172.21.15.27:0/757437278 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe21c0ccac0 0x7fe21c15b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:28.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.005+0000 7fe2225ff700 1 --2- 172.21.15.27:0/757437278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe21c156440 0x7fe21c0b1960 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:28.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.006+0000 7fe2225ff700 1 -- 172.21.15.27:0/757437278 >> 172.21.15.27:0/757437278 conn(0x7fe21c0680e0 msgr2=0x7fe21c0c6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:29.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.006+0000 7fe2225ff700 1 -- 172.21.15.27:0/757437278 shutdown_connections 2024-01-11T09:43:29.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.006+0000 7fe2225ff700 1 -- 172.21.15.27:0/757437278 wait complete. 2024-01-11T09:43:29.000 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-01-11T09:43:29.020 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-11T09:43:29.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.233+0000 7f3907dff700 1 Processor -- start 2024-01-11T09:43:29.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.234+0000 7f3907dff700 1 -- start start 2024-01-11T09:43:29.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.235+0000 7f3907dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f390015a7a0 0x7f390015c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:29.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.235+0000 7f3907dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f390015c4b0 con 0x7f3900159e60 2024-01-11T09:43:29.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.235+0000 7f3907dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f390015b5d0 con 0x7f39000685e0 2024-01-11T09:43:29.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.235+0000 7f3907dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390015b8a0 con 0x7f390015a7a0 2024-01-11T09:43:29.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.235+0000 7f390539a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3900159e60 0x7f390015aec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55984/0 (socket says 172.21.15.27:55984) 2024-01-11T09:43:29.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.235+0000 7f390539a700 1 -- 172.21.15.27:0/550908862 learned_addr learned my addr 172.21.15.27:0/550908862 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:29.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.236+0000 7f3905b9b700 1 --2- 172.21.15.27:0/550908862 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f390015a7a0 0x7f390015c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:29.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.236+0000 7f3904b99700 1 -- 172.21.15.27:0/550908862 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703895216 0 0) 0x7f390015c4b0 con 0x7f3900159e60 2024-01-11T09:43:29.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.236+0000 7f3904b99700 1 -- 172.21.15.27:0/550908862 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38ec003570 con 0x7f3900159e60 2024-01-11T09:43:29.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.236+0000 7f3904b99700 1 -- 172.21.15.27:0/550908862 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4223044901 0 0) 0x7f38ec003570 con 0x7f3900159e60 2024-01-11T09:43:29.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.236+0000 7f3904b99700 1 -- 172.21.15.27:0/550908862 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f390015a7a0 msgr2=0x7f390015c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:29.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.236+0000 7f3904b99700 1 --2- 172.21.15.27:0/550908862 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f390015a7a0 0x7f390015c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:29.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.236+0000 7f3904b99700 1 -- 172.21.15.27:0/550908862 >> v1:172.21.15.77:6789/0 conn(0x7f39000685e0 legacy=0x7f3900157930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:29.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.236+0000 7f3904b99700 1 -- 172.21.15.27:0/550908862 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390015bb80 con 0x7f3900159e60 2024-01-11T09:43:29.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.236+0000 7f3904b99700 1 -- 172.21.15.27:0/550908862 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f38f0001470 con 0x7f3900159e60 2024-01-11T09:43:29.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.236+0000 7f3904b99700 1 -- 172.21.15.27:0/550908862 >> v1:172.21.15.27:6789/0 conn(0x7f3900159e60 legacy=0x7f390015aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:29.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.236+0000 7f3904b99700 1 --2- 172.21.15.27:0/550908862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec003a90 0x7f38ec003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:29.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.236+0000 7f3904b99700 1 --2- 172.21.15.27:0/550908862 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f390015a7a0 0x7f38ec004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:29.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.236+0000 7f3904b99700 1 --2- 172.21.15.27:0/550908862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38ec004d50 0x7f38ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:29.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.236+0000 7f3904b99700 1 -- 172.21.15.27:0/550908862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390015b8a0 con 0x7f38ec003a90 2024-01-11T09:43:29.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.236+0000 7f3904b99700 1 -- 172.21.15.27:0/550908862 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec00bbf0 con 0x7f38ec004d50 2024-01-11T09:43:29.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.236+0000 7f3904b99700 1 -- 172.21.15.27:0/550908862 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec00bd80 con 0x7f390015a7a0 2024-01-11T09:43:29.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.236+0000 7f3905b9b700 1 --2- 172.21.15.27:0/550908862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec003a90 0x7f38ec003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:29.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.237+0000 7f390539a700 1 --2- 172.21.15.27:0/550908862 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f390015a7a0 0x7f38ec004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:29.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.237+0000 7f390639c700 1 --2- 172.21.15.27:0/550908862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38ec004d50 0x7f38ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:29.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.237+0000 7f3905b9b700 1 -- 172.21.15.27:0/550908862 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f390015a7a0 msgr2=0x7f38ec004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:29.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.237+0000 7f3905b9b700 1 --2- 172.21.15.27:0/550908862 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f390015a7a0 0x7f38ec004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:29.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.237+0000 7f3905b9b700 1 -- 172.21.15.27:0/550908862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38ec004d50 msgr2=0x7f38ec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:29.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.237+0000 7f3905b9b700 1 --2- 172.21.15.27:0/550908862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38ec004d50 0x7f38ec0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:29.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.237+0000 7f3905b9b700 1 -- 172.21.15.27:0/550908862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ec003570 con 0x7f38ec003a90 2024-01-11T09:43:29.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.237+0000 7f3905b9b700 1 --2- 172.21.15.27:0/550908862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec003a90 0x7f38ec003ea0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009c00 tx=0x7f38fc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e19c2dc622989c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:29.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.237+0000 7f3904b99700 1 -- 172.21.15.27:0/550908862 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc00e710 con 0x7f38ec003a90 2024-01-11T09:43:29.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.237+0000 7f3904b99700 1 -- 172.21.15.27:0/550908862 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38fc00e870 con 0x7f38ec003a90 2024-01-11T09:43:29.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.237+0000 7f3904b99700 1 -- 172.21.15.27:0/550908862 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc00ec30 con 0x7f38ec003a90 2024-01-11T09:43:29.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.238+0000 7f3907dff700 1 -- 172.21.15.27:0/550908862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec003a90 msgr2=0x7f38ec003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:29.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.238+0000 7f3907dff700 1 --2- 172.21.15.27:0/550908862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec003a90 0x7f38ec003ea0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009c00 tx=0x7f38fc0091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:29.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.238+0000 7f390639c700 1 -- 172.21.15.27:0/550908862 reap_dead start 2024-01-11T09:43:29.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.238+0000 7f3907dff700 1 -- 172.21.15.27:0/550908862 shutdown_connections 2024-01-11T09:43:29.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.238+0000 7f3907dff700 1 -- 172.21.15.27:0/550908862 >> 172.21.15.27:0/550908862 conn(0x7f390006ed40 msgr2=0x7f390006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:29.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.238+0000 7f3907dff700 1 -- 172.21.15.27:0/550908862 shutdown_connections 2024-01-11T09:43:29.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.238+0000 7f3907dff700 1 -- 172.21.15.27:0/550908862 wait complete. 2024-01-11T09:43:29.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.239+0000 7f3907dff700 1 Processor -- start 2024-01-11T09:43:29.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.240+0000 7f3907dff700 1 -- start start 2024-01-11T09:43:29.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.240+0000 7f3907dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec003a90 0x7f3900154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:29.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.240+0000 7f3907dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39000b1ff0 0x7f39000b2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:29.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.240+0000 7f3907dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3900156c50 0x7f39001570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:29.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.240+0000 7f3907dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39000b29a0 con 0x7f38ec003a90 2024-01-11T09:43:29.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.240+0000 7f3907dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3900152bd0 con 0x7f3900156c50 2024-01-11T09:43:29.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.240+0000 7f3907dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3900151fc0 con 0x7f39000b1ff0 2024-01-11T09:43:29.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.241+0000 7f3905b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec003a90 0x7f3900154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.241+0000 7f3905b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec003a90 0x7f3900154e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50934/0 (socket says 172.21.15.27:50934) 2024-01-11T09:43:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.241+0000 7f3905b9b700 1 -- 172.21.15.27:0/3517529185 learned_addr learned my addr 172.21.15.27:0/3517529185 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.241+0000 7f390639c700 1 --2- 172.21.15.27:0/3517529185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3900156c50 0x7f39001570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.241+0000 7f390539a700 1 --2- 172.21.15.27:0/3517529185 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39000b1ff0 0x7f39000b2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.241+0000 7f3905b9b700 1 -- 172.21.15.27:0/3517529185 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39000b1ff0 msgr2=0x7f39000b2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.241+0000 7f3905b9b700 1 --2- 172.21.15.27:0/3517529185 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39000b1ff0 0x7f39000b2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.241+0000 7f3905b9b700 1 -- 172.21.15.27:0/3517529185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3900156c50 msgr2=0x7f39001570d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.241+0000 7f3905b9b700 1 --2- 172.21.15.27:0/3517529185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3900156c50 0x7f39001570d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.241+0000 7f3905b9b700 1 -- 172.21.15.27:0/3517529185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39001522a0 con 0x7f38ec003a90 2024-01-11T09:43:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.241+0000 7f3905b9b700 1 --2- 172.21.15.27:0/3517529185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec003a90 0x7f3900154e10 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00e4b0 tx=0x7f38fc01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.242+0000 7f38f5ffb700 1 -- 172.21.15.27:0/3517529185 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc022070 con 0x7f38ec003a90 2024-01-11T09:43:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.242+0000 7f38f5ffb700 1 -- 172.21.15.27:0/3517529185 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38fc00e9e0 con 0x7f38ec003a90 2024-01-11T09:43:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.242+0000 7f38f5ffb700 1 -- 172.21.15.27:0/3517529185 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc027470 con 0x7f38ec003a90 2024-01-11T09:43:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.242+0000 7f3907dff700 1 -- 172.21.15.27:0/3517529185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3900152580 con 0x7f38ec003a90 2024-01-11T09:43:29.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.242+0000 7f3907dff700 1 -- 172.21.15.27:0/3517529185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3900155350 con 0x7f38ec003a90 2024-01-11T09:43:29.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.243+0000 7f3907dff700 1 -- 172.21.15.27:0/3517529185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39000b29a0 con 0x7f38ec003a90 2024-01-11T09:43:29.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.244+0000 7f38f5ffb700 1 -- 172.21.15.27:0/3517529185 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 115) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f38fc01c450 con 0x7f38ec003a90 2024-01-11T09:43:29.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.245+0000 7f38f5ffb700 1 --2- 172.21.15.27:0/3517529185 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f38e4066d80 0x7f38e4069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:29.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.245+0000 7f38f5ffb700 1 -- 172.21.15.27:0/3517529185 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6297+0+0 (secure 0 0 0) 0x7f38fc085f30 con 0x7f38ec003a90 2024-01-11T09:43:29.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.245+0000 7f390539a700 1 --2- 172.21.15.27:0/3517529185 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f38e4066d80 0x7f38e4069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:29.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.246+0000 7f390539a700 1 --2- 172.21.15.27:0/3517529185 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f38e4066d80 0x7f38e4069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f38f0000e50 tx=0x7f38f0001d80 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:29.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.250+0000 7f38f5ffb700 1 -- 172.21.15.27:0/3517529185 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f38fc053200 con 0x7f38ec003a90 2024-01-11T09:43:29.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:29.456+0000 7f3907dff700 1 -- 172.21.15.27:0/3517529185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f390004c3c0 con 0x7f38ec003a90 2024-01-11T09:43:30.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.010+0000 7f38f5ffb700 1 -- 172.21.15.27:0/3517529185 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 116) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f38fc05c0c0 con 0x7f38ec003a90 2024-01-11T09:43:30.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.288+0000 7f38f5ffb700 1 -- 172.21.15.27:0/3517529185 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v189) v1 ==== 169+0+0 (secure 0 0 0) 0x7f38fc056c90 con 0x7f38ec003a90 2024-01-11T09:43:30.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.292+0000 7f3907dff700 1 -- 172.21.15.27:0/3517529185 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f38e4066d80 msgr2=0x7f38e4069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:30.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.292+0000 7f3907dff700 1 --2- 172.21.15.27:0/3517529185 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f38e4066d80 0x7f38e4069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f38f0000e50 tx=0x7f38f0001d80 comp rx=0 tx=0).stop 2024-01-11T09:43:30.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.292+0000 7f3907dff700 1 -- 172.21.15.27:0/3517529185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec003a90 msgr2=0x7f3900154e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:30.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.292+0000 7f3907dff700 1 --2- 172.21.15.27:0/3517529185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec003a90 0x7f3900154e10 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00e4b0 tx=0x7f38fc01a000 comp rx=0 tx=0).stop 2024-01-11T09:43:30.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.292+0000 7f3907dff700 1 -- 172.21.15.27:0/3517529185 shutdown_connections 2024-01-11T09:43:30.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.292+0000 7f3907dff700 1 --2- 172.21.15.27:0/3517529185 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f38e4066d80 0x7f38e4069230 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.292+0000 7f3907dff700 1 --2- 172.21.15.27:0/3517529185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3900156c50 0x7f39001570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.292+0000 7f3907dff700 1 --2- 172.21.15.27:0/3517529185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec003a90 0x7f3900154e10 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.292+0000 7f3907dff700 1 --2- 172.21.15.27:0/3517529185 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39000b1ff0 0x7f39000b2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.292+0000 7f3907dff700 1 -- 172.21.15.27:0/3517529185 >> 172.21.15.27:0/3517529185 conn(0x7f390006ed40 msgr2=0x7f39000689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:30.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.293+0000 7f3907dff700 1 -- 172.21.15.27:0/3517529185 shutdown_connections 2024-01-11T09:43:30.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.293+0000 7f3907dff700 1 -- 172.21.15.27:0/3517529185 wait complete. 2024-01-11T09:43:30.290 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-01-11T09:43:30.312 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-11T09:43:30.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.531+0000 7f9a91bff700 1 Processor -- start 2024-01-11T09:43:30.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.531+0000 7f9a91bff700 1 -- start start 2024-01-11T09:43:30.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.532+0000 7f9a91bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9a8c15a1b0 0x7f9a8c15a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:30.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.532+0000 7f9a91bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a8c0ccd70 0x7f9a8c0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:30.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.532+0000 7f9a91bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a8c0cf850 con 0x7f9a8c0685c0 2024-01-11T09:43:30.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.532+0000 7f9a91bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a8c0cec40 con 0x7f9a8c15a1b0 2024-01-11T09:43:30.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.532+0000 7f9a91bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c0cef20 con 0x7f9a8c0ccd70 2024-01-11T09:43:30.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.533+0000 7f9a8affd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a8c0ccd70 0x7f9a8c0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:30.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.533+0000 7f9a8b7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9a8c15a1b0 0x7f9a8c15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:30.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.533+0000 7f9a8bfff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9a8c0685c0 0x7f9a8c068a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:57320/0 (socket says 172.21.15.27:57320) 2024-01-11T09:43:30.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.533+0000 7f9a8bfff700 1 -- 172.21.15.27:0/3716809029 learned_addr learned my addr 172.21.15.27:0/3716809029 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.534+0000 7f9a8a7fc700 1 -- 172.21.15.27:0/3716809029 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805028117 0 0) 0x7f9a8c0cf850 con 0x7f9a8c0685c0 2024-01-11T09:43:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.534+0000 7f9a8a7fc700 1 -- 172.21.15.27:0/3716809029 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a74003570 con 0x7f9a8c0685c0 2024-01-11T09:43:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.534+0000 7f9a8b7fe700 1 -- 172.21.15.27:0/3716809029 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a8c0ccd70 msgr2=0x7f9a8c0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.534+0000 7f9a8b7fe700 1 --2- 172.21.15.27:0/3716809029 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a8c0ccd70 0x7f9a8c0cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.534+0000 7f9a8b7fe700 1 -- 172.21.15.27:0/3716809029 >> v1:172.21.15.77:6789/0 conn(0x7f9a8c0685c0 legacy=0x7f9a8c068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.534+0000 7f9a8b7fe700 1 -- 172.21.15.27:0/3716809029 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a8c0cf200 con 0x7f9a8c15a1b0 2024-01-11T09:43:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.534+0000 7f9a8affd700 1 --2- 172.21.15.27:0/3716809029 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a8c0ccd70 0x7f9a8c0cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:43:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.534+0000 7f9a8b7fe700 1 --2- 172.21.15.27:0/3716809029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a8c15a1b0 0x7f9a8c15a580 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c0098e0 tx=0x7f9a7c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64a2c67e62a189e4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.534+0000 7f9a8a7fc700 1 -- 172.21.15.27:0/3716809029 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c011630 con 0x7f9a8c15a1b0 2024-01-11T09:43:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.534+0000 7f9a8a7fc700 1 -- 172.21.15.27:0/3716809029 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7c011790 con 0x7f9a8c15a1b0 2024-01-11T09:43:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.534+0000 7f9a8a7fc700 1 -- 172.21.15.27:0/3716809029 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c011b30 con 0x7f9a8c15a1b0 2024-01-11T09:43:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.535+0000 7f9a91bff700 1 -- 172.21.15.27:0/3716809029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a8c15a1b0 msgr2=0x7f9a8c15a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.535+0000 7f9a91bff700 1 --2- 172.21.15.27:0/3716809029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a8c15a1b0 0x7f9a8c15a580 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c0098e0 tx=0x7f9a7c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.535+0000 7f9a91bff700 1 -- 172.21.15.27:0/3716809029 shutdown_connections 2024-01-11T09:43:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.535+0000 7f9a91bff700 1 --2- 172.21.15.27:0/3716809029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a8c15a1b0 0x7f9a8c15a580 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.535+0000 7f9a91bff700 1 --2- 172.21.15.27:0/3716809029 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a8c0ccd70 0x7f9a8c0cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.535+0000 7f9a91bff700 1 -- 172.21.15.27:0/3716809029 >> 172.21.15.27:0/3716809029 conn(0x7f9a8c06ed40 msgr2=0x7f9a8c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.535+0000 7f9a91bff700 1 -- 172.21.15.27:0/3716809029 shutdown_connections 2024-01-11T09:43:30.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.535+0000 7f9a91bff700 1 -- 172.21.15.27:0/3716809029 wait complete. 2024-01-11T09:43:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.537+0000 7f9a91bff700 1 Processor -- start 2024-01-11T09:43:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.537+0000 7f9a91bff700 1 -- start start 2024-01-11T09:43:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.538+0000 7f9a91bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a8c0685c0 0x7f9a8c0af6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.538+0000 7f9a91bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a8c0ccd70 0x7f9a8c169920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.538+0000 7f9a91bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a8c15a1b0 0x7f9a8c0c6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.538+0000 7f9a91bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c16af00 con 0x7f9a8c0685c0 2024-01-11T09:43:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.538+0000 7f9a91bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c16a250 con 0x7f9a8c0ccd70 2024-01-11T09:43:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.538+0000 7f9a91bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c16a530 con 0x7f9a8c15a1b0 2024-01-11T09:43:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.538+0000 7f9a8b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a8c0685c0 0x7f9a8c0af6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.538+0000 7f9a8b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a8c0685c0 0x7f9a8c0af6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50948/0 (socket says 172.21.15.27:50948) 2024-01-11T09:43:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.538+0000 7f9a8b7fe700 1 -- 172.21.15.27:0/3876784471 learned_addr learned my addr 172.21.15.27:0/3876784471 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.538+0000 7f9a8affd700 1 --2- 172.21.15.27:0/3876784471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a8c0ccd70 0x7f9a8c169920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.538+0000 7f9a8bfff700 1 --2- 172.21.15.27:0/3876784471 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a8c15a1b0 0x7f9a8c0c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.538+0000 7f9a8b7fe700 1 -- 172.21.15.27:0/3876784471 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a8c15a1b0 msgr2=0x7f9a8c0c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.538+0000 7f9a8b7fe700 1 --2- 172.21.15.27:0/3876784471 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a8c15a1b0 0x7f9a8c0c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.538+0000 7f9a8b7fe700 1 -- 172.21.15.27:0/3876784471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a8c0ccd70 msgr2=0x7f9a8c169920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:30.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.538+0000 7f9a8b7fe700 1 --2- 172.21.15.27:0/3876784471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a8c0ccd70 0x7f9a8c169920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.539+0000 7f9a8b7fe700 1 -- 172.21.15.27:0/3876784471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a80009420 con 0x7f9a8c0685c0 2024-01-11T09:43:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.539+0000 7f9a8b7fe700 1 --2- 172.21.15.27:0/3876784471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a8c0685c0 0x7f9a8c0af6d0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c069770 tx=0x7f9a7c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.539+0000 7f9a737fe700 1 -- 172.21.15.27:0/3876784471 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c000d10 con 0x7f9a8c0685c0 2024-01-11T09:43:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.539+0000 7f9a737fe700 1 -- 172.21.15.27:0/3876784471 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7c011b10 con 0x7f9a8c0685c0 2024-01-11T09:43:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.539+0000 7f9a91bff700 1 -- 172.21.15.27:0/3876784471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a7c009420 con 0x7f9a8c0685c0 2024-01-11T09:43:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.539+0000 7f9a737fe700 1 -- 172.21.15.27:0/3876784471 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c025660 con 0x7f9a8c0685c0 2024-01-11T09:43:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.539+0000 7f9a91bff700 1 -- 172.21.15.27:0/3876784471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a8c16ab90 con 0x7f9a8c0685c0 2024-01-11T09:43:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.540+0000 7f9a91bff700 1 -- 172.21.15.27:0/3876784471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a8c04c3c0 con 0x7f9a8c0685c0 2024-01-11T09:43:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.541+0000 7f9a737fe700 1 -- 172.21.15.27:0/3876784471 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 116) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f9a7c025a50 con 0x7f9a8c0685c0 2024-01-11T09:43:30.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.543+0000 7f9a737fe700 1 --2- 172.21.15.27:0/3876784471 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9a78066ea0 0x7f9a78069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:30.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.543+0000 7f9a737fe700 1 -- 172.21.15.27:0/3876784471 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f9a7c085bc0 con 0x7f9a8c0685c0 2024-01-11T09:43:30.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.543+0000 7f9a8affd700 1 --2- 172.21.15.27:0/3876784471 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9a78066ea0 0x7f9a78069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:30.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.544+0000 7f9a8affd700 1 --2- 172.21.15.27:0/3876784471 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9a78066ea0 0x7f9a78069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c0c9b00 tx=0x7f9a800091a0 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:30.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.547+0000 7f9a737fe700 1 -- 172.21.15.27:0/3876784471 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9a7c052e60 con 0x7f9a8c0685c0 2024-01-11T09:43:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.751+0000 7f9a91bff700 1 -- 172.21.15.27:0/3876784471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f9a8c0ce4a0 con 0x7f9a8c0685c0 2024-01-11T09:43:30.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.755+0000 7f9a737fe700 1 -- 172.21.15.27:0/3876784471 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v67) v1 ==== 120+0+0 (secure 0 0 0) 0x7f9a7c056480 con 0x7f9a8c0685c0 2024-01-11T09:43:30.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.757+0000 7f9a91bff700 1 -- 172.21.15.27:0/3876784471 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9a78066ea0 msgr2=0x7f9a78069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:30.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.757+0000 7f9a91bff700 1 --2- 172.21.15.27:0/3876784471 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9a78066ea0 0x7f9a78069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c0c9b00 tx=0x7f9a800091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.757+0000 7f9a91bff700 1 -- 172.21.15.27:0/3876784471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a8c0685c0 msgr2=0x7f9a8c0af6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.757+0000 7f9a91bff700 1 --2- 172.21.15.27:0/3876784471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a8c0685c0 0x7f9a8c0af6d0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c069770 tx=0x7f9a7c017000 comp rx=0 tx=0).stop 2024-01-11T09:43:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.757+0000 7f9a91bff700 1 -- 172.21.15.27:0/3876784471 shutdown_connections 2024-01-11T09:43:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.757+0000 7f9a91bff700 1 --2- 172.21.15.27:0/3876784471 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a8c15a1b0 0x7f9a8c0c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.757+0000 7f9a91bff700 1 --2- 172.21.15.27:0/3876784471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a8c0685c0 0x7f9a8c0af6d0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.757+0000 7f9a91bff700 1 --2- 172.21.15.27:0/3876784471 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9a78066ea0 0x7f9a78069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.757+0000 7f9a91bff700 1 --2- 172.21.15.27:0/3876784471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a8c0ccd70 0x7f9a8c169920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.757+0000 7f9a91bff700 1 -- 172.21.15.27:0/3876784471 >> 172.21.15.27:0/3876784471 conn(0x7f9a8c06ed40 msgr2=0x7f9a8c0c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.757+0000 7f9a91bff700 1 -- 172.21.15.27:0/3876784471 shutdown_connections 2024-01-11T09:43:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.757+0000 7f9a91bff700 1 -- 172.21.15.27:0/3876784471 wait complete. 2024-01-11T09:43:30.768 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-11T09:43:30.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.977+0000 7f7d115ff700 1 Processor -- start 2024-01-11T09:43:30.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.978+0000 7f7d115ff700 1 -- start start 2024-01-11T09:43:30.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.979+0000 7f7d115ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7d0c0ccd70 0x7f7d0c0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:30.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.979+0000 7f7d115ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7d0c15ae70 0x7f7d0c15b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:30.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.979+0000 7f7d115ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0c0cf850 con 0x7f7d0c15a1b0 2024-01-11T09:43:30.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.979+0000 7f7d115ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d0c15ac90 con 0x7f7d0c0ccd70 2024-01-11T09:43:30.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.979+0000 7f7d115ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d0c0cec40 con 0x7f7d0c15ae70 2024-01-11T09:43:30.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.979+0000 7f7d0a7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7d0c0ccd70 0x7f7d0c0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:30.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.979+0000 7f7d0a7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7d0c0ccd70 0x7f7d0c0cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50958/0 (socket says 172.21.15.27:50958) 2024-01-11T09:43:30.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.979+0000 7f7d0a7fc700 1 -- 172.21.15.27:0/649617867 learned_addr learned my addr 172.21.15.27:0/649617867 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:30.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.979+0000 7f7d0b7fe700 1 --2- 172.21.15.27:0/649617867 >> v2:172.21.15.77:3300/0 conn(0x7f7d0c15ae70 0x7f7d0c15b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:30.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.980+0000 7f7d0a7fc700 1 -- 172.21.15.27:0/649617867 >> v2:172.21.15.77:3300/0 conn(0x7f7d0c15ae70 msgr2=0x7f7d0c15b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:30.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.980+0000 7f7d0a7fc700 1 --2- 172.21.15.27:0/649617867 >> v2:172.21.15.77:3300/0 conn(0x7f7d0c15ae70 0x7f7d0c15b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.980+0000 7f7d0a7fc700 1 -- 172.21.15.27:0/649617867 >> v1:172.21.15.77:6789/0 conn(0x7f7d0c15a1b0 legacy=0x7f7d0c15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:30.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.980+0000 7f7d0a7fc700 1 -- 172.21.15.27:0/649617867 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0c0cef20 con 0x7f7d0c0ccd70 2024-01-11T09:43:30.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.980+0000 7f7d0a7fc700 1 --2- 172.21.15.27:0/649617867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d0c0ccd70 0x7f7d0c0cd210 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c089810 tx=0x7f7d00009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=791f7c511fb9688c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:30.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.980+0000 7f7d09ffb700 1 -- 172.21.15.27:0/649617867 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0000b860 con 0x7f7d0c0ccd70 2024-01-11T09:43:30.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.980+0000 7f7d09ffb700 1 -- 172.21.15.27:0/649617867 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0000b9c0 con 0x7f7d0c0ccd70 2024-01-11T09:43:30.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.980+0000 7f7d09ffb700 1 -- 172.21.15.27:0/649617867 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0000bd60 con 0x7f7d0c0ccd70 2024-01-11T09:43:30.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.981+0000 7f7d115ff700 1 -- 172.21.15.27:0/649617867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d0c0ccd70 msgr2=0x7f7d0c0cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:30.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.981+0000 7f7d115ff700 1 --2- 172.21.15.27:0/649617867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d0c0ccd70 0x7f7d0c0cd210 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c089810 tx=0x7f7d00009250 comp rx=0 tx=0).stop 2024-01-11T09:43:30.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.981+0000 7f7d115ff700 1 -- 172.21.15.27:0/649617867 shutdown_connections 2024-01-11T09:43:30.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.981+0000 7f7d115ff700 1 --2- 172.21.15.27:0/649617867 >> v2:172.21.15.77:3300/0 conn(0x7f7d0c15ae70 0x7f7d0c15b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.981+0000 7f7d115ff700 1 --2- 172.21.15.27:0/649617867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d0c0ccd70 0x7f7d0c0cd210 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.981+0000 7f7d115ff700 1 -- 172.21.15.27:0/649617867 >> 172.21.15.27:0/649617867 conn(0x7f7d0c06ed40 msgr2=0x7f7d0c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:30.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.981+0000 7f7d115ff700 1 -- 172.21.15.27:0/649617867 shutdown_connections 2024-01-11T09:43:30.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.981+0000 7f7d115ff700 1 -- 172.21.15.27:0/649617867 wait complete. 2024-01-11T09:43:30.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.982+0000 7f7d115ff700 1 Processor -- start 2024-01-11T09:43:30.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.983+0000 7f7d115ff700 1 -- start start 2024-01-11T09:43:30.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.983+0000 7f7d115ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d0c0ccd70 0x7f7d0c159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:30.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.983+0000 7f7d115ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d0c15a1b0 0x7f7d0c16a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:30.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.984+0000 7f7d115ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d0c15ae70 0x7f7d0c14b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:30.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.984+0000 7f7d115ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0c159a10 con 0x7f7d0c15a1b0 2024-01-11T09:43:30.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.984+0000 7f7d115ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0c0b4e30 con 0x7f7d0c0ccd70 2024-01-11T09:43:30.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.984+0000 7f7d115ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0c16a950 con 0x7f7d0c15ae70 2024-01-11T09:43:30.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.984+0000 7f7d0a7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d0c15a1b0 0x7f7d0c16a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:30.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.984+0000 7f7d0a7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d0c15a1b0 0x7f7d0c16a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50962/0 (socket says 172.21.15.27:50962) 2024-01-11T09:43:30.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.984+0000 7f7d0a7fc700 1 -- 172.21.15.27:0/4115512073 learned_addr learned my addr 172.21.15.27:0/4115512073 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:30.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.984+0000 7f7d0affd700 1 --2- 172.21.15.27:0/4115512073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d0c0ccd70 0x7f7d0c159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:30.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.984+0000 7f7d0b7fe700 1 --2- 172.21.15.27:0/4115512073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d0c15ae70 0x7f7d0c14b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:30.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.984+0000 7f7d0a7fc700 1 -- 172.21.15.27:0/4115512073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d0c15ae70 msgr2=0x7f7d0c14b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:30.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.984+0000 7f7d0a7fc700 1 --2- 172.21.15.27:0/4115512073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d0c15ae70 0x7f7d0c14b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.984+0000 7f7d0a7fc700 1 -- 172.21.15.27:0/4115512073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d0c0ccd70 msgr2=0x7f7d0c159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:30.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.984+0000 7f7d0a7fc700 1 --2- 172.21.15.27:0/4115512073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d0c0ccd70 0x7f7d0c159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:30.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.984+0000 7f7d0a7fc700 1 -- 172.21.15.27:0/4115512073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d000094f0 con 0x7f7d0c15a1b0 2024-01-11T09:43:30.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.984+0000 7f7d0affd700 1 --2- 172.21.15.27:0/4115512073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d0c0ccd70 0x7f7d0c159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:30.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.985+0000 7f7d0a7fc700 1 --2- 172.21.15.27:0/4115512073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d0c15a1b0 0x7f7d0c16a410 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c069770 tx=0x7f7d00017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:30.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.985+0000 7f7cf2ffd700 1 -- 172.21.15.27:0/4115512073 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d00011e60 con 0x7f7d0c15a1b0 2024-01-11T09:43:30.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.985+0000 7f7cf2ffd700 1 -- 172.21.15.27:0/4115512073 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0001a630 con 0x7f7d0c15a1b0 2024-01-11T09:43:30.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.985+0000 7f7d115ff700 1 -- 172.21.15.27:0/4115512073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d0c0b44e0 con 0x7f7d0c15a1b0 2024-01-11T09:43:30.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.985+0000 7f7cf2ffd700 1 -- 172.21.15.27:0/4115512073 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0001a9f0 con 0x7f7d0c15a1b0 2024-01-11T09:43:30.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.985+0000 7f7d115ff700 1 -- 172.21.15.27:0/4115512073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d0c14b6f0 con 0x7f7d0c15a1b0 2024-01-11T09:43:30.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.986+0000 7f7d115ff700 1 -- 172.21.15.27:0/4115512073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d0c0b49e0 con 0x7f7d0c15a1b0 2024-01-11T09:43:30.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.987+0000 7f7cf2ffd700 1 -- 172.21.15.27:0/4115512073 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 116) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f7d0002d440 con 0x7f7d0c15a1b0 2024-01-11T09:43:30.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.988+0000 7f7cf2ffd700 1 --2- 172.21.15.27:0/4115512073 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7cf8066ea0 0x7f7cf8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:30.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.988+0000 7f7cf2ffd700 1 -- 172.21.15.27:0/4115512073 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f7d000860a0 con 0x7f7d0c15a1b0 2024-01-11T09:43:30.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.989+0000 7f7d0affd700 1 --2- 172.21.15.27:0/4115512073 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7cf8066ea0 0x7f7cf8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:30.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.989+0000 7f7d0affd700 1 --2- 172.21.15.27:0/4115512073 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7cf8066ea0 0x7f7cf8069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc007fd0 tx=0x7f7cfc009040 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:30.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:30.993+0000 7f7cf2ffd700 1 -- 172.21.15.27:0/4115512073 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7d00053340 con 0x7f7d0c15a1b0 2024-01-11T09:43:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.198+0000 7f7d115ff700 1 -- 172.21.15.27:0/4115512073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f7d0c0b4fa0 con 0x7f7d0c15a1b0 2024-01-11T09:43:31.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.202+0000 7f7cf2ffd700 1 -- 172.21.15.27:0/4115512073 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v68) v1 ==== 115+0+0 (secure 0 0 0) 0x7f7d00056960 con 0x7f7d0c15a1b0 2024-01-11T09:43:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.204+0000 7f7d115ff700 1 -- 172.21.15.27:0/4115512073 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7cf8066ea0 msgr2=0x7f7cf8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.204+0000 7f7d115ff700 1 --2- 172.21.15.27:0/4115512073 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7cf8066ea0 0x7f7cf8069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc007fd0 tx=0x7f7cfc009040 comp rx=0 tx=0).stop 2024-01-11T09:43:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.204+0000 7f7d115ff700 1 -- 172.21.15.27:0/4115512073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d0c15a1b0 msgr2=0x7f7d0c16a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.204+0000 7f7d115ff700 1 --2- 172.21.15.27:0/4115512073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d0c15a1b0 0x7f7d0c16a410 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c069770 tx=0x7f7d00017000 comp rx=0 tx=0).stop 2024-01-11T09:43:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.204+0000 7f7d115ff700 1 -- 172.21.15.27:0/4115512073 shutdown_connections 2024-01-11T09:43:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.204+0000 7f7d115ff700 1 --2- 172.21.15.27:0/4115512073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7d0c15ae70 0x7f7d0c14b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.204+0000 7f7d115ff700 1 --2- 172.21.15.27:0/4115512073 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7cf8066ea0 0x7f7cf8069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.204+0000 7f7d115ff700 1 --2- 172.21.15.27:0/4115512073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d0c0ccd70 0x7f7d0c159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.204+0000 7f7d115ff700 1 --2- 172.21.15.27:0/4115512073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d0c15a1b0 0x7f7d0c16a410 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.204+0000 7f7d115ff700 1 -- 172.21.15.27:0/4115512073 >> 172.21.15.27:0/4115512073 conn(0x7f7d0c06ed40 msgr2=0x7f7d0c1561e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.205+0000 7f7d115ff700 1 -- 172.21.15.27:0/4115512073 shutdown_connections 2024-01-11T09:43:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.205+0000 7f7d115ff700 1 -- 172.21.15.27:0/4115512073 wait complete. 2024-01-11T09:43:31.216 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-11T09:43:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.425+0000 7f98e59ff700 1 Processor -- start 2024-01-11T09:43:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.425+0000 7f98e59ff700 1 -- start start 2024-01-11T09:43:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.426+0000 7f98e59ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f98e0142dc0 0x7f98e00cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.426+0000 7f98e59ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98e0143190 0x7f98e00cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.426+0000 7f98e59ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f98e00cc020 0x7f98e00cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.426+0000 7f98e59ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98e015df30 con 0x7f98e00cc020 2024-01-11T09:43:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.426+0000 7f98e59ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98e015d320 con 0x7f98e0142dc0 2024-01-11T09:43:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.426+0000 7f98e59ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e015d590 con 0x7f98e0143190 2024-01-11T09:43:31.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.427+0000 7f98df7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f98e00cc020 0x7f98e00cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:31.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.427+0000 7f98df7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f98e00cc020 0x7f98e00cc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50966/0 (socket says 172.21.15.27:50966) 2024-01-11T09:43:31.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.427+0000 7f98df7fe700 1 -- 172.21.15.27:0/2573719134 learned_addr learned my addr 172.21.15.27:0/2573719134 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:31.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.427+0000 7f98deffd700 1 --2- 172.21.15.27:0/2573719134 >> v2:172.21.15.77:3300/0 conn(0x7f98e0142dc0 0x7f98e00cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:31.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.427+0000 7f98de7fc700 1 --2- 172.21.15.27:0/2573719134 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98e0143190 0x7f98e00cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:31.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.427+0000 7f98df7fe700 1 -- 172.21.15.27:0/2573719134 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98e0143190 msgr2=0x7f98e00cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:31.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.427+0000 7f98df7fe700 1 --2- 172.21.15.27:0/2573719134 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98e0143190 0x7f98e00cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:31.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.427+0000 7f98df7fe700 1 -- 172.21.15.27:0/2573719134 >> v2:172.21.15.77:3300/0 conn(0x7f98e0142dc0 msgr2=0x7f98e00cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:31.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.427+0000 7f98df7fe700 1 --2- 172.21.15.27:0/2573719134 >> v2:172.21.15.77:3300/0 conn(0x7f98e0142dc0 0x7f98e00cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:31.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.427+0000 7f98df7fe700 1 -- 172.21.15.27:0/2573719134 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e015d870 con 0x7f98e00cc020 2024-01-11T09:43:31.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.428+0000 7f98df7fe700 1 --2- 172.21.15.27:0/2573719134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98e00cc020 0x7f98e00cc440 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f98e006f4d0 tx=0x7f98d8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43d3e172879e0ec5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:31.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.428+0000 7f98ddffb700 1 -- 172.21.15.27:0/2573719134 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d800e070 con 0x7f98e00cc020 2024-01-11T09:43:31.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.428+0000 7f98ddffb700 1 -- 172.21.15.27:0/2573719134 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d800b970 con 0x7f98e00cc020 2024-01-11T09:43:31.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.428+0000 7f98ddffb700 1 -- 172.21.15.27:0/2573719134 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d80092e0 con 0x7f98e00cc020 2024-01-11T09:43:31.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.428+0000 7f98e59ff700 1 -- 172.21.15.27:0/2573719134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98e00cc020 msgr2=0x7f98e00cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:31.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.428+0000 7f98e59ff700 1 --2- 172.21.15.27:0/2573719134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98e00cc020 0x7f98e00cc440 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f98e006f4d0 tx=0x7f98d8009840 comp rx=0 tx=0).stop 2024-01-11T09:43:31.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.428+0000 7f98e59ff700 1 -- 172.21.15.27:0/2573719134 shutdown_connections 2024-01-11T09:43:31.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.428+0000 7f98e59ff700 1 --2- 172.21.15.27:0/2573719134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98e00cc020 0x7f98e00cc440 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:31.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.428+0000 7f98e59ff700 1 --2- 172.21.15.27:0/2573719134 >> v2:172.21.15.77:3300/0 conn(0x7f98e0142dc0 0x7f98e00cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:31.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.428+0000 7f98e59ff700 1 --2- 172.21.15.27:0/2573719134 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98e0143190 0x7f98e00cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:31.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.428+0000 7f98e59ff700 1 -- 172.21.15.27:0/2573719134 >> 172.21.15.27:0/2573719134 conn(0x7f98e0063ca0 msgr2=0x7f98e006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:31.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.429+0000 7f98e59ff700 1 -- 172.21.15.27:0/2573719134 shutdown_connections 2024-01-11T09:43:31.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.429+0000 7f98e59ff700 1 -- 172.21.15.27:0/2573719134 wait complete. 2024-01-11T09:43:31.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.430+0000 7f98e59ff700 1 Processor -- start 2024-01-11T09:43:31.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.430+0000 7f98e59ff700 1 -- start start 2024-01-11T09:43:31.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.431+0000 7f98e59ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98e00cc020 0x7f98e0141dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:31.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.431+0000 7f98e59ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e0142dc0 0x7f98e00c7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:31.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.431+0000 7f98e59ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98e0143190 0x7f98e00aa500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:31.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.431+0000 7f98e59ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e0169990 con 0x7f98e0143190 2024-01-11T09:43:31.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.431+0000 7f98e59ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e00c7e10 con 0x7f98e0142dc0 2024-01-11T09:43:31.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.431+0000 7f98e59ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e00c80f0 con 0x7f98e00cc020 2024-01-11T09:43:31.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.432+0000 7f98df7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98e0143190 0x7f98e00aa500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:31.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.432+0000 7f98df7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98e0143190 0x7f98e00aa500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50982/0 (socket says 172.21.15.27:50982) 2024-01-11T09:43:31.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.432+0000 7f98df7fe700 1 -- 172.21.15.27:0/3142134955 learned_addr learned my addr 172.21.15.27:0/3142134955 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:31.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.432+0000 7f98deffd700 1 --2- 172.21.15.27:0/3142134955 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98e00cc020 0x7f98e0141dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:31.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.432+0000 7f98de7fc700 1 --2- 172.21.15.27:0/3142134955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e0142dc0 0x7f98e00c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:31.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.432+0000 7f98df7fe700 1 -- 172.21.15.27:0/3142134955 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98e00cc020 msgr2=0x7f98e0141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:31.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.432+0000 7f98df7fe700 1 --2- 172.21.15.27:0/3142134955 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98e00cc020 0x7f98e0141dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:31.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.432+0000 7f98df7fe700 1 -- 172.21.15.27:0/3142134955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e0142dc0 msgr2=0x7f98e00c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:31.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.432+0000 7f98df7fe700 1 --2- 172.21.15.27:0/3142134955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e0142dc0 0x7f98e00c7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:31.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.432+0000 7f98df7fe700 1 -- 172.21.15.27:0/3142134955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d8009ae0 con 0x7f98e0143190 2024-01-11T09:43:31.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.432+0000 7f98df7fe700 1 --2- 172.21.15.27:0/3142134955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98e0143190 0x7f98e00aa500 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f98e0064a70 tx=0x7f98d8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:31.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.432+0000 7f98c6ffd700 1 -- 172.21.15.27:0/3142134955 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d8008ad0 con 0x7f98e0143190 2024-01-11T09:43:31.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.433+0000 7f98c6ffd700 1 -- 172.21.15.27:0/3142134955 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d8008c50 con 0x7f98e0143190 2024-01-11T09:43:31.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.433+0000 7f98c6ffd700 1 -- 172.21.15.27:0/3142134955 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d8026460 con 0x7f98e0143190 2024-01-11T09:43:31.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.433+0000 7f98e59ff700 1 -- 172.21.15.27:0/3142134955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e0169c30 con 0x7f98e0143190 2024-01-11T09:43:31.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.433+0000 7f98e59ff700 1 -- 172.21.15.27:0/3142134955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e0142310 con 0x7f98e0143190 2024-01-11T09:43:31.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.434+0000 7f98e59ff700 1 -- 172.21.15.27:0/3142134955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e00c4840 con 0x7f98e0143190 2024-01-11T09:43:31.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.434+0000 7f98c6ffd700 1 -- 172.21.15.27:0/3142134955 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 116) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f98d8003040 con 0x7f98e0143190 2024-01-11T09:43:31.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.436+0000 7f98c6ffd700 1 --2- 172.21.15.27:0/3142134955 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f98cc066ea0 0x7f98cc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:31.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.436+0000 7f98c6ffd700 1 -- 172.21.15.27:0/3142134955 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f98d8086260 con 0x7f98e0143190 2024-01-11T09:43:31.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.436+0000 7f98deffd700 1 --2- 172.21.15.27:0/3142134955 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f98cc066ea0 0x7f98cc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:31.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.437+0000 7f98deffd700 1 --2- 172.21.15.27:0/3142134955 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f98cc066ea0 0x7f98cc069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009db0 tx=0x7f98d00091a0 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:31.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.439+0000 7f98c6ffd700 1 -- 172.21.15.27:0/3142134955 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f98d8053510 con 0x7f98e0143190 2024-01-11T09:43:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.645+0000 7f98e59ff700 1 -- 172.21.15.27:0/3142134955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f98e004c3c0 con 0x7f98e0143190 2024-01-11T09:43:31.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.649+0000 7f98c6ffd700 1 -- 172.21.15.27:0/3142134955 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v69) v1 ==== 104+0+0 (secure 0 0 0) 0x7f98d8056b30 con 0x7f98e0143190 2024-01-11T09:43:31.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.651+0000 7f98e59ff700 1 -- 172.21.15.27:0/3142134955 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f98cc066ea0 msgr2=0x7f98cc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:31.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.651+0000 7f98e59ff700 1 --2- 172.21.15.27:0/3142134955 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f98cc066ea0 0x7f98cc069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009db0 tx=0x7f98d00091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:31.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.651+0000 7f98e59ff700 1 -- 172.21.15.27:0/3142134955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98e0143190 msgr2=0x7f98e00aa500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:31.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.651+0000 7f98e59ff700 1 --2- 172.21.15.27:0/3142134955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98e0143190 0x7f98e00aa500 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f98e0064a70 tx=0x7f98d8018000 comp rx=0 tx=0).stop 2024-01-11T09:43:31.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.651+0000 7f98e59ff700 1 -- 172.21.15.27:0/3142134955 shutdown_connections 2024-01-11T09:43:31.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.651+0000 7f98e59ff700 1 --2- 172.21.15.27:0/3142134955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98e0143190 0x7f98e00aa500 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:31.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.651+0000 7f98e59ff700 1 --2- 172.21.15.27:0/3142134955 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f98e00cc020 0x7f98e0141dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:31.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.651+0000 7f98e59ff700 1 --2- 172.21.15.27:0/3142134955 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f98cc066ea0 0x7f98cc069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:31.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.651+0000 7f98e59ff700 1 --2- 172.21.15.27:0/3142134955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e0142dc0 0x7f98e00c7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:31.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.651+0000 7f98e59ff700 1 -- 172.21.15.27:0/3142134955 >> 172.21.15.27:0/3142134955 conn(0x7f98e0063ca0 msgr2=0x7f98e00c4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:31.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.651+0000 7f98e59ff700 1 -- 172.21.15.27:0/3142134955 shutdown_connections 2024-01-11T09:43:31.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.651+0000 7f98e59ff700 1 -- 172.21.15.27:0/3142134955 wait complete. 2024-01-11T09:43:31.662 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2024-01-11T09:43:31.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.869+0000 7fe6b15ff700 1 Processor -- start 2024-01-11T09:43:31.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.869+0000 7fe6b15ff700 1 -- start start 2024-01-11T09:43:31.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.871+0000 7fe6b15ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe6ac0b09e0 0x7fe6ac1610a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:31.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.871+0000 7fe6b15ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ac157390 con 0x7fe6ac0c8620 2024-01-11T09:43:31.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.871+0000 7fe6b15ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ac06a200 con 0x7fe6ac069d40 2024-01-11T09:43:31.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.871+0000 7fe6b15ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6ac0adcd0 con 0x7fe6ac0b09e0 2024-01-11T09:43:31.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.871+0000 7fe6aaffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe6ac0c8620 0x7fe6ac0af970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55994/0 (socket says 172.21.15.27:55994) 2024-01-11T09:43:31.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.871+0000 7fe6aaffd700 1 -- 172.21.15.27:0/180721715 learned_addr learned my addr 172.21.15.27:0/180721715 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:31.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.871+0000 7fe6ab7fe700 1 --2- 172.21.15.27:0/180721715 >> v2:172.21.15.27:3300/0 conn(0x7fe6ac0b09e0 0x7fe6ac1610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:31.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.871+0000 7fe6ab7fe700 1 -- 172.21.15.27:0/180721715 >> v1:172.21.15.77:6789/0 conn(0x7fe6ac069d40 legacy=0x7fe6ac0b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:31.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.871+0000 7fe6ab7fe700 1 -- 172.21.15.27:0/180721715 >> v1:172.21.15.27:6789/0 conn(0x7fe6ac0c8620 legacy=0x7fe6ac0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:31.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.871+0000 7fe6ab7fe700 1 -- 172.21.15.27:0/180721715 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ac156d40 con 0x7fe6ac0b09e0 2024-01-11T09:43:31.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.872+0000 7fe6ab7fe700 1 --2- 172.21.15.27:0/180721715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6ac0b09e0 0x7fe6ac1610a0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4009e60 tx=0x7fe6a4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ed079523658bb05 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:31.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.872+0000 7fe6a9ffb700 1 -- 172.21.15.27:0/180721715 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a400e070 con 0x7fe6ac0b09e0 2024-01-11T09:43:31.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.872+0000 7fe6a9ffb700 1 -- 172.21.15.27:0/180721715 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a400bc30 con 0x7fe6ac0b09e0 2024-01-11T09:43:31.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.872+0000 7fe6a9ffb700 1 -- 172.21.15.27:0/180721715 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4012430 con 0x7fe6ac0b09e0 2024-01-11T09:43:31.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.872+0000 7fe6b15ff700 1 -- 172.21.15.27:0/180721715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6ac0b09e0 msgr2=0x7fe6ac1610a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:31.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.872+0000 7fe6b15ff700 1 --2- 172.21.15.27:0/180721715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6ac0b09e0 0x7fe6ac1610a0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4009e60 tx=0x7fe6a4009780 comp rx=0 tx=0).stop 2024-01-11T09:43:31.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.873+0000 7fe6b15ff700 1 -- 172.21.15.27:0/180721715 shutdown_connections 2024-01-11T09:43:31.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.873+0000 7fe6b15ff700 1 --2- 172.21.15.27:0/180721715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6ac0b09e0 0x7fe6ac1610a0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:31.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.873+0000 7fe6b15ff700 1 -- 172.21.15.27:0/180721715 >> 172.21.15.27:0/180721715 conn(0x7fe6ac0680e0 msgr2=0x7fe6ac069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:31.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.873+0000 7fe6b15ff700 1 -- 172.21.15.27:0/180721715 shutdown_connections 2024-01-11T09:43:31.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.873+0000 7fe6b15ff700 1 -- 172.21.15.27:0/180721715 wait complete. 2024-01-11T09:43:31.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.874+0000 7fe6b15ff700 1 Processor -- start 2024-01-11T09:43:31.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.874+0000 7fe6b15ff700 1 -- start start 2024-01-11T09:43:31.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.875+0000 7fe6b15ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6ac069d40 0x7fe6ac153d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:31.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.875+0000 7fe6b15ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe6ac0b09e0 0x7fe6ac15bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.875+0000 7fe6b15ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ac0c8620 0x7fe6ac0c6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.875+0000 7fe6b15ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ac0ca210 con 0x7fe6ac069d40 2024-01-11T09:43:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.875+0000 7fe6b15ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ac0c9600 con 0x7fe6ac0c8620 2024-01-11T09:43:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.875+0000 7fe6b15ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ac0c98e0 con 0x7fe6ac0b09e0 2024-01-11T09:43:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.876+0000 7fe6aaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6ac069d40 0x7fe6ac153d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.876+0000 7fe6aaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6ac069d40 0x7fe6ac153d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51002/0 (socket says 172.21.15.27:51002) 2024-01-11T09:43:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.876+0000 7fe6aaffd700 1 -- 172.21.15.27:0/4106391084 learned_addr learned my addr 172.21.15.27:0/4106391084 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.876+0000 7fe6ab7fe700 1 --2- 172.21.15.27:0/4106391084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ac0c8620 0x7fe6ac0c6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.876+0000 7fe6aa7fc700 1 --2- 172.21.15.27:0/4106391084 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe6ac0b09e0 0x7fe6ac15bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.876+0000 7fe6aaffd700 1 -- 172.21.15.27:0/4106391084 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe6ac0b09e0 msgr2=0x7fe6ac15bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.876+0000 7fe6aaffd700 1 --2- 172.21.15.27:0/4106391084 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe6ac0b09e0 0x7fe6ac15bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.876+0000 7fe6aaffd700 1 -- 172.21.15.27:0/4106391084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ac0c8620 msgr2=0x7fe6ac0c6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.876+0000 7fe6aaffd700 1 --2- 172.21.15.27:0/4106391084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ac0c8620 0x7fe6ac0c6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:31.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.876+0000 7fe6aaffd700 1 -- 172.21.15.27:0/4106391084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a4009a20 con 0x7fe6ac069d40 2024-01-11T09:43:31.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.876+0000 7fe6aaffd700 1 --2- 172.21.15.27:0/4106391084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6ac069d40 0x7fe6ac153d20 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fe69c00c560 tx=0x7fe69c001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:31.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.877+0000 7fe692ffd700 1 -- 172.21.15.27:0/4106391084 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c005b30 con 0x7fe6ac069d40 2024-01-11T09:43:31.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.877+0000 7fe692ffd700 1 -- 172.21.15.27:0/4106391084 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69c005c90 con 0x7fe6ac069d40 2024-01-11T09:43:31.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.877+0000 7fe692ffd700 1 -- 172.21.15.27:0/4106391084 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c007320 con 0x7fe6ac069d40 2024-01-11T09:43:31.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.877+0000 7fe6b15ff700 1 -- 172.21.15.27:0/4106391084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6ac0c9bc0 con 0x7fe6ac069d40 2024-01-11T09:43:31.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.877+0000 7fe6b15ff700 1 -- 172.21.15.27:0/4106391084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6ac154260 con 0x7fe6ac069d40 2024-01-11T09:43:31.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.878+0000 7fe6b15ff700 1 -- 172.21.15.27:0/4106391084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6ac0718d0 con 0x7fe6ac069d40 2024-01-11T09:43:31.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.879+0000 7fe692ffd700 1 -- 172.21.15.27:0/4106391084 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 116) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fe69c01b3b0 con 0x7fe6ac069d40 2024-01-11T09:43:31.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.880+0000 7fe692ffd700 1 --2- 172.21.15.27:0/4106391084 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe698066ea0 0x7fe698069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:31.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.880+0000 7fe692ffd700 1 -- 172.21.15.27:0/4106391084 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5883+0+0 (secure 0 0 0) 0x7fe69c083860 con 0x7fe6ac069d40 2024-01-11T09:43:31.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.880+0000 7fe6aa7fc700 1 --2- 172.21.15.27:0/4106391084 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe698066ea0 0x7fe698069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:31.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.881+0000 7fe6aa7fc700 1 --2- 172.21.15.27:0/4106391084 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe698066ea0 0x7fe698069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac0acaa0 tx=0x7fe6a0007ec0 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:31.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:31.883+0000 7fe692ffd700 1 -- 172.21.15.27:0/4106391084 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe69c050b00 con 0x7fe6ac069d40 2024-01-11T09:43:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:32.169+0000 7fe6b15ff700 1 -- 172.21.15.27:0/4106391084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7fe6ac06bf00 con 0x7fe6ac069d40 2024-01-11T09:43:32.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:32.210+0000 7fe692ffd700 1 -- 172.21.15.27:0/4106391084 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe69c050ed0 con 0x7fe6ac069d40 2024-01-11T09:43:33.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.015+0000 7fe692ffd700 1 -- 172.21.15.27:0/4106391084 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v3009) v1 ==== 165+0+0 (secure 0 0 0) 0x7fe69c054590 con 0x7fe6ac069d40 2024-01-11T09:43:33.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.018+0000 7fe6b15ff700 1 -- 172.21.15.27:0/4106391084 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe698066ea0 msgr2=0x7fe698069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:33.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.018+0000 7fe6b15ff700 1 --2- 172.21.15.27:0/4106391084 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe698066ea0 0x7fe698069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac0acaa0 tx=0x7fe6a0007ec0 comp rx=0 tx=0).stop 2024-01-11T09:43:33.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.018+0000 7fe6b15ff700 1 -- 172.21.15.27:0/4106391084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6ac069d40 msgr2=0x7fe6ac153d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:33.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.018+0000 7fe6b15ff700 1 --2- 172.21.15.27:0/4106391084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6ac069d40 0x7fe6ac153d20 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fe69c00c560 tx=0x7fe69c001020 comp rx=0 tx=0).stop 2024-01-11T09:43:33.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.018+0000 7fe6b15ff700 1 -- 172.21.15.27:0/4106391084 shutdown_connections 2024-01-11T09:43:33.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.018+0000 7fe6b15ff700 1 --2- 172.21.15.27:0/4106391084 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe698066ea0 0x7fe698069350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:33.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.018+0000 7fe6b15ff700 1 --2- 172.21.15.27:0/4106391084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ac0c8620 0x7fe6ac0c6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:33.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.018+0000 7fe6b15ff700 1 --2- 172.21.15.27:0/4106391084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6ac069d40 0x7fe6ac153d20 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:33.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.018+0000 7fe6b15ff700 1 --2- 172.21.15.27:0/4106391084 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe6ac0b09e0 0x7fe6ac15bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:33.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.018+0000 7fe6b15ff700 1 -- 172.21.15.27:0/4106391084 >> 172.21.15.27:0/4106391084 conn(0x7fe6ac0680e0 msgr2=0x7fe6ac06dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:33.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.019+0000 7fe6b15ff700 1 -- 172.21.15.27:0/4106391084 shutdown_connections 2024-01-11T09:43:33.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.019+0000 7fe6b15ff700 1 -- 172.21.15.27:0/4106391084 wait complete. 2024-01-11T09:43:33.032 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2024-01-11T09:43:33.033 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2024-01-11T09:43:33.033 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-01-11T09:43:33.033 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2024-01-11T09:43:33.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.244+0000 7f7a955ff700 1 Processor -- start 2024-01-11T09:43:33.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.244+0000 7f7a955ff700 1 -- start start 2024-01-11T09:43:33.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.245+0000 7f7a955ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7a901599e0 0x7f7a900b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:33.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.246+0000 7f7a955ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a90157e30 con 0x7f7a90159db0 2024-01-11T09:43:33.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.246+0000 7f7a955ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a90157500 con 0x7f7a90150330 2024-01-11T09:43:33.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.246+0000 7f7a955ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a901577e0 con 0x7f7a901599e0 2024-01-11T09:43:33.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.246+0000 7f7a8effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7a901599e0 0x7f7a900b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:33.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.246+0000 7f7a8e7fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7a90159db0 0x7f7a9014fc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56010/0 (socket says 172.21.15.27:56010) 2024-01-11T09:43:33.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.246+0000 7f7a8e7fc700 1 -- 172.21.15.27:0/2426310496 learned_addr learned my addr 172.21.15.27:0/2426310496 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:33.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.246+0000 7f7a8effd700 1 -- 172.21.15.27:0/2426310496 >> v1:172.21.15.77:6789/0 conn(0x7f7a90150330 legacy=0x7f7a901582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:33.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.246+0000 7f7a8effd700 1 -- 172.21.15.27:0/2426310496 >> v1:172.21.15.27:6789/0 conn(0x7f7a90159db0 legacy=0x7f7a9014fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:33.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.246+0000 7f7a8effd700 1 -- 172.21.15.27:0/2426310496 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a90157ac0 con 0x7f7a901599e0 2024-01-11T09:43:33.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.246+0000 7f7a8dffb700 1 -- 172.21.15.27:0/2426310496 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548322868 0 0) 0x7f7a90157e30 con 0x7f7a90159db0 2024-01-11T09:43:33.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.247+0000 7f7a8effd700 1 --2- 172.21.15.27:0/2426310496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a901599e0 0x7f7a900b0130 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000bb90 tx=0x7f7a8000b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f94e80e93cc8c73 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:33.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.247+0000 7f7a8dffb700 1 -- 172.21.15.27:0/2426310496 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8000dba0 con 0x7f7a901599e0 2024-01-11T09:43:33.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.247+0000 7f7a8dffb700 1 -- 172.21.15.27:0/2426310496 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a8000dd00 con 0x7f7a901599e0 2024-01-11T09:43:33.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.247+0000 7f7a8dffb700 1 -- 172.21.15.27:0/2426310496 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a800196b0 con 0x7f7a901599e0 2024-01-11T09:43:33.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.247+0000 7f7a955ff700 1 -- 172.21.15.27:0/2426310496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a901599e0 msgr2=0x7f7a900b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:33.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.247+0000 7f7a955ff700 1 --2- 172.21.15.27:0/2426310496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a901599e0 0x7f7a900b0130 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000bb90 tx=0x7f7a8000b430 comp rx=0 tx=0).stop 2024-01-11T09:43:33.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.247+0000 7f7a955ff700 1 -- 172.21.15.27:0/2426310496 shutdown_connections 2024-01-11T09:43:33.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.247+0000 7f7a955ff700 1 --2- 172.21.15.27:0/2426310496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a901599e0 0x7f7a900b0130 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:33.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.248+0000 7f7a955ff700 1 -- 172.21.15.27:0/2426310496 >> 172.21.15.27:0/2426310496 conn(0x7f7a900680e0 msgr2=0x7f7a90069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:33.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.248+0000 7f7a955ff700 1 -- 172.21.15.27:0/2426310496 shutdown_connections 2024-01-11T09:43:33.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.248+0000 7f7a955ff700 1 -- 172.21.15.27:0/2426310496 wait complete. 2024-01-11T09:43:33.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.249+0000 7f7a955ff700 1 Processor -- start 2024-01-11T09:43:33.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.249+0000 7f7a955ff700 1 -- start start 2024-01-11T09:43:33.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.250+0000 7f7a955ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a90150330 0x7f7a90153230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:33.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.250+0000 7f7a955ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a901599e0 0x7f7a901499c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.250+0000 7f7a955ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a90159db0 0x7f7a900c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.250+0000 7f7a955ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a90165ba0 con 0x7f7a90159db0 2024-01-11T09:43:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.250+0000 7f7a955ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a90153770 con 0x7f7a90150330 2024-01-11T09:43:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.250+0000 7f7a955ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a901663e0 con 0x7f7a901599e0 2024-01-11T09:43:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.250+0000 7f7a8f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a90159db0 0x7f7a900c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.250+0000 7f7a8f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a90159db0 0x7f7a900c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51030/0 (socket says 172.21.15.27:51030) 2024-01-11T09:43:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.250+0000 7f7a8f7fe700 1 -- 172.21.15.27:0/1591695659 learned_addr learned my addr 172.21.15.27:0/1591695659 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.251+0000 7f7a8effd700 1 --2- 172.21.15.27:0/1591695659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a90150330 0x7f7a90153230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.251+0000 7f7a8e7fc700 1 --2- 172.21.15.27:0/1591695659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a901599e0 0x7f7a901499c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.251+0000 7f7a8f7fe700 1 -- 172.21.15.27:0/1591695659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a901599e0 msgr2=0x7f7a901499c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.251+0000 7f7a8f7fe700 1 --2- 172.21.15.27:0/1591695659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a901599e0 0x7f7a901499c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.251+0000 7f7a8f7fe700 1 -- 172.21.15.27:0/1591695659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a90150330 msgr2=0x7f7a90153230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.251+0000 7f7a8f7fe700 1 --2- 172.21.15.27:0/1591695659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a90150330 0x7f7a90153230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.251+0000 7f7a8f7fe700 1 -- 172.21.15.27:0/1591695659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8000b6d0 con 0x7f7a90159db0 2024-01-11T09:43:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.251+0000 7f7a8e7fc700 1 --2- 172.21.15.27:0/1591695659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a901599e0 0x7f7a901499c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:43:33.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.251+0000 7f7a8f7fe700 1 --2- 172.21.15.27:0/1591695659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a90159db0 0x7f7a900c8c10 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f7a8800c910 tx=0x7f7a88001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:33.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.251+0000 7f7a76ffd700 1 -- 172.21.15.27:0/1591695659 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8800a7a0 con 0x7f7a90159db0 2024-01-11T09:43:33.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.251+0000 7f7a76ffd700 1 -- 172.21.15.27:0/1591695659 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a8800a900 con 0x7f7a90159db0 2024-01-11T09:43:33.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.251+0000 7f7a955ff700 1 -- 172.21.15.27:0/1591695659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a90149f00 con 0x7f7a90159db0 2024-01-11T09:43:33.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.251+0000 7f7a76ffd700 1 -- 172.21.15.27:0/1591695659 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8800acc0 con 0x7f7a90159db0 2024-01-11T09:43:33.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.252+0000 7f7a955ff700 1 -- 172.21.15.27:0/1591695659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a90165d90 con 0x7f7a90159db0 2024-01-11T09:43:33.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.253+0000 7f7a955ff700 1 -- 172.21.15.27:0/1591695659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a90166550 con 0x7f7a90159db0 2024-01-11T09:43:33.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.254+0000 7f7a76ffd700 1 -- 172.21.15.27:0/1591695659 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7a88012630 con 0x7f7a90159db0 2024-01-11T09:43:33.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.255+0000 7f7a76ffd700 1 --2- 172.21.15.27:0/1591695659 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7a7c066f30 0x7f7a7c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:33.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.255+0000 7f7a76ffd700 1 -- 172.21.15.27:0/1591695659 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f7a880848d0 con 0x7f7a90159db0 2024-01-11T09:43:33.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.255+0000 7f7a8effd700 1 --2- 172.21.15.27:0/1591695659 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7a7c066f30 0x7f7a7c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:33.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.256+0000 7f7a8effd700 1 --2- 172.21.15.27:0/1591695659 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7a7c066f30 0x7f7a7c0693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000b2c0 tx=0x7f7a8001b000 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:33.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.259+0000 7f7a76ffd700 1 -- 172.21.15.27:0/1591695659 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7a88055520 con 0x7f7a90159db0 2024-01-11T09:43:33.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:33.540+0000 7f7a955ff700 1 -- 172.21.15.27:0/1591695659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f7a900718d0 con 0x7f7a90159db0 2024-01-11T09:43:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.020+0000 7f7a76ffd700 1 -- 172.21.15.27:0/1591695659 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3010) v1 ==== 166+0+0 (secure 0 0 0) 0x7f7a880550b0 con 0x7f7a90159db0 2024-01-11T09:43:34.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.022+0000 7f7a955ff700 1 -- 172.21.15.27:0/1591695659 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7a7c066f30 msgr2=0x7f7a7c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:34.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.022+0000 7f7a955ff700 1 --2- 172.21.15.27:0/1591695659 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7a7c066f30 0x7f7a7c0693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000b2c0 tx=0x7f7a8001b000 comp rx=0 tx=0).stop 2024-01-11T09:43:34.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.022+0000 7f7a955ff700 1 -- 172.21.15.27:0/1591695659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a90159db0 msgr2=0x7f7a900c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:34.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.023+0000 7f7a955ff700 1 --2- 172.21.15.27:0/1591695659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a90159db0 0x7f7a900c8c10 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f7a8800c910 tx=0x7f7a88001710 comp rx=0 tx=0).stop 2024-01-11T09:43:34.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.023+0000 7f7a955ff700 1 -- 172.21.15.27:0/1591695659 shutdown_connections 2024-01-11T09:43:34.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.023+0000 7f7a955ff700 1 --2- 172.21.15.27:0/1591695659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a90159db0 0x7f7a900c8c10 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.023+0000 7f7a955ff700 1 --2- 172.21.15.27:0/1591695659 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7a7c066f30 0x7f7a7c0693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.023+0000 7f7a955ff700 1 --2- 172.21.15.27:0/1591695659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a90150330 0x7f7a90153230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.023+0000 7f7a955ff700 1 --2- 172.21.15.27:0/1591695659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7a901599e0 0x7f7a901499c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.023+0000 7f7a955ff700 1 -- 172.21.15.27:0/1591695659 >> 172.21.15.27:0/1591695659 conn(0x7f7a900680e0 msgr2=0x7f7a9014a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:34.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.023+0000 7f7a955ff700 1 -- 172.21.15.27:0/1591695659 shutdown_connections 2024-01-11T09:43:34.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.023+0000 7f7a955ff700 1 -- 172.21.15.27:0/1591695659 wait complete. 2024-01-11T09:43:34.035 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-11T09:43:34.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.248+0000 7fe4ae9ff700 1 Processor -- start 2024-01-11T09:43:34.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.248+0000 7fe4ae9ff700 1 -- start start 2024-01-11T09:43:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.249+0000 7fe4ae9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4a81596f0 0x7fe4a80a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.249+0000 7fe4ae9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a80c80a0 con 0x7fe4a80a9de0 2024-01-11T09:43:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.249+0000 7fe4ae9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a80cbac0 con 0x7fe4a8068d30 2024-01-11T09:43:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.249+0000 7fe4ae9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80c7490 con 0x7fe4a81596f0 2024-01-11T09:43:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.249+0000 7fe4a77fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe4a80a9de0 0x7fe4a8068620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33476/0 (socket says 172.21.15.27:33476) 2024-01-11T09:43:34.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.249+0000 7fe4a77fe700 1 -- 172.21.15.27:0/2047849227 learned_addr learned my addr 172.21.15.27:0/2047849227 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.250+0000 7fe4a7fff700 1 --2- 172.21.15.27:0/2047849227 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4a81596f0 0x7fe4a80a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.250+0000 7fe4a6ffd700 1 -- 172.21.15.27:0/2047849227 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337626325 0 0) 0x7fe4a80c80a0 con 0x7fe4a80a9de0 2024-01-11T09:43:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.250+0000 7fe4a6ffd700 1 -- 172.21.15.27:0/2047849227 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe490003570 con 0x7fe4a80a9de0 2024-01-11T09:43:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.250+0000 7fe4a6ffd700 1 -- 172.21.15.27:0/2047849227 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140526120 0 0) 0x7fe4a80cbac0 con 0x7fe4a8068d30 2024-01-11T09:43:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.250+0000 7fe4a6ffd700 1 -- 172.21.15.27:0/2047849227 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a80c80a0 con 0x7fe4a8068d30 2024-01-11T09:43:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.250+0000 7fe4a6ffd700 1 -- 172.21.15.27:0/2047849227 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 412891385 0 0) 0x7fe490003570 con 0x7fe4a80a9de0 2024-01-11T09:43:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.250+0000 7fe4a6ffd700 1 -- 172.21.15.27:0/2047849227 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4a81596f0 msgr2=0x7fe4a80a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.250+0000 7fe4a6ffd700 1 --2- 172.21.15.27:0/2047849227 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4a81596f0 0x7fe4a80a98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.250+0000 7fe4a6ffd700 1 -- 172.21.15.27:0/2047849227 >> v1:172.21.15.77:6789/0 conn(0x7fe4a8068d30 legacy=0x7fe4a80c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.250+0000 7fe4a6ffd700 1 -- 172.21.15.27:0/2047849227 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a80c7770 con 0x7fe4a80a9de0 2024-01-11T09:43:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.250+0000 7fe4a6ffd700 1 -- 172.21.15.27:0/2047849227 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fe49c001470 con 0x7fe4a80a9de0 2024-01-11T09:43:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.250+0000 7fe4a6ffd700 1 -- 172.21.15.27:0/2047849227 >> v1:172.21.15.27:6789/0 conn(0x7fe4a80a9de0 legacy=0x7fe4a8068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.250+0000 7fe4a6ffd700 1 --2- 172.21.15.27:0/2047849227 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4a81596f0 0x7fe490003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.250+0000 7fe4a6ffd700 1 --2- 172.21.15.27:0/2047849227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe490004180 0x7fe490004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.250+0000 7fe4a6ffd700 1 --2- 172.21.15.27:0/2047849227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe490004d50 0x7fe4900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.250+0000 7fe4a6ffd700 1 -- 172.21.15.27:0/2047849227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80c7490 con 0x7fe490004d50 2024-01-11T09:43:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.250+0000 7fe4a6ffd700 1 -- 172.21.15.27:0/2047849227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49000bba0 con 0x7fe490004180 2024-01-11T09:43:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.250+0000 7fe4a6ffd700 1 -- 172.21.15.27:0/2047849227 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe49000bd60 con 0x7fe4a81596f0 2024-01-11T09:43:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.251+0000 7fe4acf9c700 1 --2- 172.21.15.27:0/2047849227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe490004d50 0x7fe4900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.251+0000 7fe4a77fe700 1 --2- 172.21.15.27:0/2047849227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe490004180 0x7fe490004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.251+0000 7fe4a7fff700 1 --2- 172.21.15.27:0/2047849227 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4a81596f0 0x7fe490003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.251+0000 7fe4acf9c700 1 -- 172.21.15.27:0/2047849227 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4a81596f0 msgr2=0x7fe490003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.251+0000 7fe4acf9c700 1 --2- 172.21.15.27:0/2047849227 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4a81596f0 0x7fe490003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.251+0000 7fe4acf9c700 1 -- 172.21.15.27:0/2047849227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe490004180 msgr2=0x7fe490004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.251+0000 7fe4acf9c700 1 --2- 172.21.15.27:0/2047849227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe490004180 0x7fe490004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.251+0000 7fe4acf9c700 1 -- 172.21.15.27:0/2047849227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe490003570 con 0x7fe490004d50 2024-01-11T09:43:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.251+0000 7fe4acf9c700 1 --2- 172.21.15.27:0/2047849227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe490004d50 0x7fe4900051c0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fe4a80cc980 tx=0x7fe4a00017e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e26de8bfb1f4f3b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.252+0000 7fe4a6ffd700 1 -- 172.21.15.27:0/2047849227 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0001af0 con 0x7fe490004d50 2024-01-11T09:43:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.252+0000 7fe4a6ffd700 1 -- 172.21.15.27:0/2047849227 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a000c8b0 con 0x7fe490004d50 2024-01-11T09:43:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.252+0000 7fe4a6ffd700 1 -- 172.21.15.27:0/2047849227 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a000cc50 con 0x7fe490004d50 2024-01-11T09:43:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.252+0000 7fe4ae9ff700 1 -- 172.21.15.27:0/2047849227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe490004d50 msgr2=0x7fe4900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.252+0000 7fe4ae9ff700 1 --2- 172.21.15.27:0/2047849227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe490004d50 0x7fe4900051c0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fe4a80cc980 tx=0x7fe4a00017e0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.252+0000 7fe4acf9c700 1 -- 172.21.15.27:0/2047849227 reap_dead start 2024-01-11T09:43:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.252+0000 7fe4ae9ff700 1 -- 172.21.15.27:0/2047849227 shutdown_connections 2024-01-11T09:43:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.252+0000 7fe4ae9ff700 1 -- 172.21.15.27:0/2047849227 >> 172.21.15.27:0/2047849227 conn(0x7fe4a80c9fb0 msgr2=0x7fe4a806ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.252+0000 7fe4ae9ff700 1 -- 172.21.15.27:0/2047849227 shutdown_connections 2024-01-11T09:43:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.252+0000 7fe4ae9ff700 1 -- 172.21.15.27:0/2047849227 wait complete. 2024-01-11T09:43:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.254+0000 7fe4ae9ff700 1 Processor -- start 2024-01-11T09:43:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.254+0000 7fe4ae9ff700 1 -- start start 2024-01-11T09:43:34.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.255+0000 7fe4ae9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a8153f80 0x7fe4a80b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:34.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.255+0000 7fe4ae9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4a80af8b0 0x7fe4a80afd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:34.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.255+0000 7fe4ae9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4a81583e0 0x7fe4a8158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:34.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.255+0000 7fe4ae9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a8158d90 con 0x7fe4a8153f80 2024-01-11T09:43:34.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.255+0000 7fe4ae9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80b0f10 con 0x7fe4a80af8b0 2024-01-11T09:43:34.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.255+0000 7fe4ae9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80b0300 con 0x7fe4a81583e0 2024-01-11T09:43:34.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.255+0000 7fe4a7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a8153f80 0x7fe4a80b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:34.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.255+0000 7fe4a7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a8153f80 0x7fe4a80b13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40952/0 (socket says 172.21.15.27:40952) 2024-01-11T09:43:34.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.255+0000 7fe4a7fff700 1 -- 172.21.15.27:0/3666546531 learned_addr learned my addr 172.21.15.27:0/3666546531 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:34.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.255+0000 7fe4a77fe700 1 --2- 172.21.15.27:0/3666546531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4a80af8b0 0x7fe4a80afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:34.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.255+0000 7fe4acf9c700 1 --2- 172.21.15.27:0/3666546531 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4a81583e0 0x7fe4a8158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:34.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.256+0000 7fe4a7fff700 1 -- 172.21.15.27:0/3666546531 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4a81583e0 msgr2=0x7fe4a8158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:34.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.256+0000 7fe4a7fff700 1 --2- 172.21.15.27:0/3666546531 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4a81583e0 0x7fe4a8158850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.256+0000 7fe4a7fff700 1 -- 172.21.15.27:0/3666546531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4a80af8b0 msgr2=0x7fe4a80afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:34.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.256+0000 7fe4a7fff700 1 --2- 172.21.15.27:0/3666546531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4a80af8b0 0x7fe4a80afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.256+0000 7fe4a7fff700 1 -- 172.21.15.27:0/3666546531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a80b05e0 con 0x7fe4a8153f80 2024-01-11T09:43:34.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.256+0000 7fe4a7fff700 1 --2- 172.21.15.27:0/3666546531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a8153f80 0x7fe4a80b13a0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fe498006ac0 tx=0x7fe4980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:34.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.256+0000 7fe48ffff700 1 -- 172.21.15.27:0/3666546531 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49800f940 con 0x7fe4a8153f80 2024-01-11T09:43:34.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.256+0000 7fe48ffff700 1 -- 172.21.15.27:0/3666546531 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49800faa0 con 0x7fe4a8153f80 2024-01-11T09:43:34.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.256+0000 7fe48ffff700 1 -- 172.21.15.27:0/3666546531 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49800fe20 con 0x7fe4a8153f80 2024-01-11T09:43:34.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.256+0000 7fe4ae9ff700 1 -- 172.21.15.27:0/3666546531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a80b08c0 con 0x7fe4a8153f80 2024-01-11T09:43:34.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.256+0000 7fe4ae9ff700 1 -- 172.21.15.27:0/3666546531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a8154350 con 0x7fe4a8153f80 2024-01-11T09:43:34.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.257+0000 7fe4ae9ff700 1 -- 172.21.15.27:0/3666546531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a80b11f0 con 0x7fe4a8153f80 2024-01-11T09:43:34.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.258+0000 7fe48ffff700 1 -- 172.21.15.27:0/3666546531 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe498014650 con 0x7fe4a8153f80 2024-01-11T09:43:34.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.259+0000 7fe48ffff700 1 --2- 172.21.15.27:0/3666546531 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe494066f30 0x7fe4940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:34.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.259+0000 7fe48ffff700 1 -- 172.21.15.27:0/3666546531 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5883+0+0 (secure 0 0 0) 0x7fe498018030 con 0x7fe4a8153f80 2024-01-11T09:43:34.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.259+0000 7fe4a77fe700 1 --2- 172.21.15.27:0/3666546531 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe494066f30 0x7fe4940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:34.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.260+0000 7fe4a77fe700 1 --2- 172.21.15.27:0/3666546531 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe494066f30 0x7fe4940693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe49c001200 tx=0x7fe49c001df0 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:34.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.263+0000 7fe48ffff700 1 -- 172.21.15.27:0/3666546531 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe498052120 con 0x7fe4a8153f80 2024-01-11T09:43:34.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.467+0000 7fe4ae9ff700 1 -- 172.21.15.27:0/3666546531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe4a804c3c0 con 0x7fe4a8153f80 2024-01-11T09:43:34.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.472+0000 7fe48ffff700 1 -- 172.21.15.27:0/3666546531 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v70)=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe498055740 con 0x7fe4a8153f80 2024-01-11T09:43:34.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.475+0000 7fe4ae9ff700 1 -- 172.21.15.27:0/3666546531 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe494066f30 msgr2=0x7fe4940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:34.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.475+0000 7fe4ae9ff700 1 --2- 172.21.15.27:0/3666546531 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe494066f30 0x7fe4940693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe49c001200 tx=0x7fe49c001df0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.475+0000 7fe4ae9ff700 1 -- 172.21.15.27:0/3666546531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a8153f80 msgr2=0x7fe4a80b13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:34.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.475+0000 7fe4ae9ff700 1 --2- 172.21.15.27:0/3666546531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a8153f80 0x7fe4a80b13a0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fe498006ac0 tx=0x7fe4980091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.475+0000 7fe4ae9ff700 1 -- 172.21.15.27:0/3666546531 shutdown_connections 2024-01-11T09:43:34.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.475+0000 7fe4ae9ff700 1 --2- 172.21.15.27:0/3666546531 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4a81583e0 0x7fe4a8158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.475+0000 7fe4ae9ff700 1 --2- 172.21.15.27:0/3666546531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a8153f80 0x7fe4a80b13a0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.475+0000 7fe4ae9ff700 1 --2- 172.21.15.27:0/3666546531 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe494066f30 0x7fe4940693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.475+0000 7fe4ae9ff700 1 --2- 172.21.15.27:0/3666546531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4a80af8b0 0x7fe4a80afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.475+0000 7fe4ae9ff700 1 -- 172.21.15.27:0/3666546531 >> 172.21.15.27:0/3666546531 conn(0x7fe4a80c9fb0 msgr2=0x7fe4a80c23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:34.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.475+0000 7fe4ae9ff700 1 -- 172.21.15.27:0/3666546531 shutdown_connections 2024-01-11T09:43:34.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.475+0000 7fe4ae9ff700 1 -- 172.21.15.27:0/3666546531 wait complete. 2024-01-11T09:43:34.485 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T09:43:34.485 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:43:34.577 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:43:34.685 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T09:43:34.685 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-11T09:43:34.685 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:43:34.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.893+0000 7f283d7ff700 1 Processor -- start 2024-01-11T09:43:34.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.893+0000 7f283d7ff700 1 -- start start 2024-01-11T09:43:34.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.895+0000 7f283d7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f28380ad020 0x7f28380af320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:34.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.895+0000 7f283d7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28380ac370 0x7f2838159180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:34.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.895+0000 7f283d7ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f283806b460 0x7f28381596c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:34.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.895+0000 7f283d7ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2838159c00 con 0x7f283806b460 2024-01-11T09:43:34.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.895+0000 7f283d7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28380aa1d0 con 0x7f28380ad020 2024-01-11T09:43:34.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.895+0000 7f283d7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28380af860 con 0x7f28380ac370 2024-01-11T09:43:34.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.895+0000 7f28377fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f283806b460 0x7f28381596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.895+0000 7f28377fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f283806b460 0x7f28381596c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40958/0 (socket says 172.21.15.27:40958) 2024-01-11T09:43:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.895+0000 7f28377fe700 1 -- 172.21.15.27:0/2325012889 learned_addr learned my addr 172.21.15.27:0/2325012889 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.895+0000 7f2836ffd700 1 --2- 172.21.15.27:0/2325012889 >> v2:172.21.15.77:3300/0 conn(0x7f28380ad020 0x7f28380af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.895+0000 7f28367fc700 1 --2- 172.21.15.27:0/2325012889 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28380ac370 0x7f2838159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.895+0000 7f28377fe700 1 -- 172.21.15.27:0/2325012889 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28380ac370 msgr2=0x7f2838159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.895+0000 7f28377fe700 1 --2- 172.21.15.27:0/2325012889 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28380ac370 0x7f2838159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.896+0000 7f28377fe700 1 -- 172.21.15.27:0/2325012889 >> v2:172.21.15.77:3300/0 conn(0x7f28380ad020 msgr2=0x7f28380af320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.896+0000 7f28377fe700 1 --2- 172.21.15.27:0/2325012889 >> v2:172.21.15.77:3300/0 conn(0x7f28380ad020 0x7f28380af320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.896+0000 7f28377fe700 1 -- 172.21.15.27:0/2325012889 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28380ac8d0 con 0x7f283806b460 2024-01-11T09:43:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.896+0000 7f28367fc700 1 --2- 172.21.15.27:0/2325012889 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28380ac370 0x7f2838159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.896+0000 7f2836ffd700 1 --2- 172.21.15.27:0/2325012889 >> v2:172.21.15.77:3300/0 conn(0x7f28380ad020 0x7f28380af320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.896+0000 7f28377fe700 1 --2- 172.21.15.27:0/2325012889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f283806b460 0x7f28381596c0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f2838064b00 tx=0x7f2830009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7dc6c38f11c43db server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.896+0000 7f2835ffb700 1 -- 172.21.15.27:0/2325012889 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283000e070 con 0x7f283806b460 2024-01-11T09:43:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.896+0000 7f2835ffb700 1 -- 172.21.15.27:0/2325012889 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283000ba50 con 0x7f283806b460 2024-01-11T09:43:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.896+0000 7f2835ffb700 1 -- 172.21.15.27:0/2325012889 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28300092e0 con 0x7f283806b460 2024-01-11T09:43:34.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.897+0000 7f283d7ff700 1 -- 172.21.15.27:0/2325012889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f283806b460 msgr2=0x7f28381596c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:34.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.897+0000 7f283d7ff700 1 --2- 172.21.15.27:0/2325012889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f283806b460 0x7f28381596c0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f2838064b00 tx=0x7f2830009840 comp rx=0 tx=0).stop 2024-01-11T09:43:34.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.897+0000 7f283d7ff700 1 -- 172.21.15.27:0/2325012889 shutdown_connections 2024-01-11T09:43:34.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.897+0000 7f283d7ff700 1 --2- 172.21.15.27:0/2325012889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f283806b460 0x7f28381596c0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.897+0000 7f283d7ff700 1 --2- 172.21.15.27:0/2325012889 >> v2:172.21.15.77:3300/0 conn(0x7f28380ad020 0x7f28380af320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.897+0000 7f283d7ff700 1 --2- 172.21.15.27:0/2325012889 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28380ac370 0x7f2838159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.897+0000 7f283d7ff700 1 -- 172.21.15.27:0/2325012889 >> 172.21.15.27:0/2325012889 conn(0x7f28380680e0 msgr2=0x7f2838069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:34.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.897+0000 7f283d7ff700 1 -- 172.21.15.27:0/2325012889 shutdown_connections 2024-01-11T09:43:34.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.897+0000 7f283d7ff700 1 -- 172.21.15.27:0/2325012889 wait complete. 2024-01-11T09:43:34.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.898+0000 7f283d7ff700 1 Processor -- start 2024-01-11T09:43:34.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.898+0000 7f283d7ff700 1 -- start start 2024-01-11T09:43:34.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.899+0000 7f283d7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f283806b460 0x7f2838143d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:34.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.899+0000 7f283d7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28380ac370 0x7f2838158f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:34.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.899+0000 7f283d7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28380ad020 0x7f2838159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:34.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.899+0000 7f283d7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2838144260 con 0x7f28380ad020 2024-01-11T09:43:34.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.899+0000 7f283d7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283816aff0 con 0x7f283806b460 2024-01-11T09:43:34.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.899+0000 7f283d7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f283816b290 con 0x7f28380ac370 2024-01-11T09:43:34.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.900+0000 7f28377fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28380ad020 0x7f2838159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:34.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.900+0000 7f28377fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28380ad020 0x7f2838159490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40966/0 (socket says 172.21.15.27:40966) 2024-01-11T09:43:34.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.900+0000 7f28377fe700 1 -- 172.21.15.27:0/3364716233 learned_addr learned my addr 172.21.15.27:0/3364716233 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:34.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.900+0000 7f28367fc700 1 --2- 172.21.15.27:0/3364716233 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28380ac370 0x7f2838158f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:34.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.900+0000 7f2836ffd700 1 --2- 172.21.15.27:0/3364716233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f283806b460 0x7f2838143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:34.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.900+0000 7f28377fe700 1 -- 172.21.15.27:0/3364716233 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28380ac370 msgr2=0x7f2838158f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:34.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.900+0000 7f28377fe700 1 --2- 172.21.15.27:0/3364716233 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28380ac370 0x7f2838158f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.900+0000 7f28377fe700 1 -- 172.21.15.27:0/3364716233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f283806b460 msgr2=0x7f2838143d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:34.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.900+0000 7f28377fe700 1 --2- 172.21.15.27:0/3364716233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f283806b460 0x7f2838143d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:34.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.900+0000 7f28377fe700 1 -- 172.21.15.27:0/3364716233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2830009ae0 con 0x7f28380ad020 2024-01-11T09:43:34.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.900+0000 7f28377fe700 1 --2- 172.21.15.27:0/3364716233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28380ad020 0x7f2838159490 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f2838066180 tx=0x7f2830018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:34.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.901+0000 7f281effd700 1 -- 172.21.15.27:0/3364716233 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28300032f0 con 0x7f28380ad020 2024-01-11T09:43:34.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.901+0000 7f281effd700 1 -- 172.21.15.27:0/3364716233 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2830008ce0 con 0x7f28380ad020 2024-01-11T09:43:34.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.901+0000 7f281effd700 1 -- 172.21.15.27:0/3364716233 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28300266b0 con 0x7f28380ad020 2024-01-11T09:43:34.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.901+0000 7f283d7ff700 1 -- 172.21.15.27:0/3364716233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283816b570 con 0x7f28380ad020 2024-01-11T09:43:34.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.901+0000 7f283d7ff700 1 -- 172.21.15.27:0/3364716233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f283816acb0 con 0x7f28380ad020 2024-01-11T09:43:34.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.902+0000 7f283d7ff700 1 -- 172.21.15.27:0/3364716233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28380718d0 con 0x7f28380ad020 2024-01-11T09:43:34.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.903+0000 7f281effd700 1 -- 172.21.15.27:0/3364716233 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2830003040 con 0x7f28380ad020 2024-01-11T09:43:34.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.904+0000 7f281effd700 1 --2- 172.21.15.27:0/3364716233 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2824066f30 0x7f28240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:34.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.904+0000 7f281effd700 1 -- 172.21.15.27:0/3364716233 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5883+0+0 (secure 0 0 0) 0x7f2830086460 con 0x7f28380ad020 2024-01-11T09:43:34.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.904+0000 7f2836ffd700 1 --2- 172.21.15.27:0/3364716233 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2824066f30 0x7f28240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:34.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.905+0000 7f2836ffd700 1 --2- 172.21.15.27:0/3364716233 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2824066f30 0x7f28240693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2828006fd0 tx=0x7f2828008040 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:34.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:34.908+0000 7f281effd700 1 -- 172.21.15.27:0/3364716233 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2830053580 con 0x7f28380ad020 2024-01-11T09:43:35.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.154+0000 7f283d7ff700 1 -- 172.21.15.27:0/3364716233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28380a98a0 con 0x7f28380ad020 2024-01-11T09:43:35.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.155+0000 7f281effd700 1 -- 172.21.15.27:0/3364716233 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3526 (secure 0 0 0) 0x7f2830021de0 con 0x7f28380ad020 2024-01-11T09:43:35.149 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:43:35.149 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":96,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8015,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1003297202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1003297202},{"type":"v1","addr":"172.21.15.77:6834","nonce":1003297202}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":8018,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3321059147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":3321059147},{"type":"v1","addr":"172.21.15.77:6835","nonce":3321059147}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":9202,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2606639222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":2606639222},{"type":"v1","addr":"172.21.15.27:6835","nonce":2606639222}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":9993,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/834971395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":834971395},{"type":"v1","addr":"172.21.15.27:6834","nonce":834971395}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96}],"filesystems":[]} 2024-01-11T09:43:35.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.157+0000 7f283d7ff700 1 -- 172.21.15.27:0/3364716233 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2824066f30 msgr2=0x7f28240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:35.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.157+0000 7f283d7ff700 1 --2- 172.21.15.27:0/3364716233 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2824066f30 0x7f28240693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2828006fd0 tx=0x7f2828008040 comp rx=0 tx=0).stop 2024-01-11T09:43:35.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.157+0000 7f283d7ff700 1 -- 172.21.15.27:0/3364716233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28380ad020 msgr2=0x7f2838159490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:35.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.157+0000 7f283d7ff700 1 --2- 172.21.15.27:0/3364716233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28380ad020 0x7f2838159490 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f2838066180 tx=0x7f2830018000 comp rx=0 tx=0).stop 2024-01-11T09:43:35.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.157+0000 7f283d7ff700 1 -- 172.21.15.27:0/3364716233 shutdown_connections 2024-01-11T09:43:35.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.157+0000 7f283d7ff700 1 --2- 172.21.15.27:0/3364716233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28380ad020 0x7f2838159490 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:35.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.157+0000 7f283d7ff700 1 --2- 172.21.15.27:0/3364716233 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2824066f30 0x7f28240693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:35.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.157+0000 7f283d7ff700 1 --2- 172.21.15.27:0/3364716233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f283806b460 0x7f2838143d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:35.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.157+0000 7f283d7ff700 1 --2- 172.21.15.27:0/3364716233 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f28380ac370 0x7f2838158f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:35.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.157+0000 7f283d7ff700 1 -- 172.21.15.27:0/3364716233 >> 172.21.15.27:0/3364716233 conn(0x7f28380680e0 msgr2=0x7f28380abbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:35.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.157+0000 7f283d7ff700 1 -- 172.21.15.27:0/3364716233 shutdown_connections 2024-01-11T09:43:35.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.157+0000 7f283d7ff700 1 -- 172.21.15.27:0/3364716233 wait complete. 2024-01-11T09:43:35.151 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 96 2024-01-11T09:43:35.167 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-11T09:43:35.167 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-11T09:43:35.168 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-11T09:43:35.168 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:43:35.168 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-11T09:43:35.168 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-11T09:43:35.169 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-11T09:43:35.169 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:43:35.169 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-11T09:43:35.169 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-11T09:43:35.169 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-11T09:43:35.169 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:43:35.169 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-11T09:43:35.169 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-11T09:43:35.170 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-11T09:43:35.170 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:43:35.187 INFO:tasks.ceph.mds.b:Stopped 2024-01-11T09:43:35.187 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-11T09:43:35.188 INFO:tasks.ceph.mds.d:Stopped 2024-01-11T09:43:35.188 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-11T09:43:35.189 INFO:tasks.ceph.mds.a:Stopped 2024-01-11T09:43:35.189 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-11T09:43:35.190 INFO:tasks.ceph.mds.c:Stopped 2024-01-11T09:43:35.190 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-11T09:43:35.232 INFO:tasks.ceph.mds.b:Started 2024-01-11T09:43:35.232 DEBUG:teuthology.parallel:result is None 2024-01-11T09:43:35.234 INFO:tasks.ceph.mds.a:Started 2024-01-11T09:43:35.234 DEBUG:teuthology.parallel:result is None 2024-01-11T09:43:35.235 INFO:tasks.ceph.mds.d:Started 2024-01-11T09:43:35.235 DEBUG:teuthology.parallel:result is None 2024-01-11T09:43:35.237 INFO:tasks.ceph.mds.c:Started 2024-01-11T09:43:35.237 DEBUG:teuthology.parallel:result is None 2024-01-11T09:43:35.237 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-11T09:43:35.378 INFO:tasks.ceph.mds.b.smithi077.stdout:starting mds.b at 2024-01-11T09:43:35.379 INFO:tasks.ceph.mds.d.smithi077.stdout:starting mds.d at 2024-01-11T09:43:35.387 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T09:43:35.388 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T09:43:35.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.455+0000 7fd7bddff700 1 Processor -- start 2024-01-11T09:43:35.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.455+0000 7fd7bddff700 1 -- start start 2024-01-11T09:43:35.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.456+0000 7fd7bddff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd7b8157360 0x7fd7b806f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:35.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.456+0000 7fd7bddff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7b8157730 0x7fd7b80cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:35.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.456+0000 7fd7bddff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd7b80b1290 0x7fd7b80b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:35.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.457+0000 7fd7bddff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7b80cce10 con 0x7fd7b80b1290 2024-01-11T09:43:35.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.457+0000 7fd7bddff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7b80b1c30 con 0x7fd7b8157360 2024-01-11T09:43:35.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.457+0000 7fd7bddff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b80cb540 con 0x7fd7b8157730 2024-01-11T09:43:35.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.457+0000 7fd7b7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd7b80b1290 0x7fd7b80b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:35.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.457+0000 7fd7b7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd7b80b1290 0x7fd7b80b16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41004/0 (socket says 172.21.15.27:41004) 2024-01-11T09:43:35.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.457+0000 7fd7b7fff700 1 -- 172.21.15.27:0/313673805 learned_addr learned my addr 172.21.15.27:0/313673805 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:35.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.457+0000 7fd7b77fe700 1 --2- 172.21.15.27:0/313673805 >> v2:172.21.15.77:3300/0 conn(0x7fd7b8157360 0x7fd7b806f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:35.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.457+0000 7fd7b6ffd700 1 --2- 172.21.15.27:0/313673805 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7b8157730 0x7fd7b80cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:35.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.457+0000 7fd7b7fff700 1 -- 172.21.15.27:0/313673805 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7b8157730 msgr2=0x7fd7b80cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:35.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.457+0000 7fd7b7fff700 1 --2- 172.21.15.27:0/313673805 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7b8157730 0x7fd7b80cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:35.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.457+0000 7fd7b7fff700 1 -- 172.21.15.27:0/313673805 >> v2:172.21.15.77:3300/0 conn(0x7fd7b8157360 msgr2=0x7fd7b806f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:35.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.457+0000 7fd7b7fff700 1 --2- 172.21.15.27:0/313673805 >> v2:172.21.15.77:3300/0 conn(0x7fd7b8157360 0x7fd7b806f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:35.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.457+0000 7fd7b7fff700 1 -- 172.21.15.27:0/313673805 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b80ca960 con 0x7fd7b80b1290 2024-01-11T09:43:35.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.457+0000 7fd7b7fff700 1 --2- 172.21.15.27:0/313673805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7b80b1290 0x7fd7b80b16f0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fd7b8076d60 tx=0x7fd7b0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd9ce7aaba0baee server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:35.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.458+0000 7fd7b67fc700 1 -- 172.21.15.27:0/313673805 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b000e070 con 0x7fd7b80b1290 2024-01-11T09:43:35.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.458+0000 7fd7b67fc700 1 -- 172.21.15.27:0/313673805 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b000ba50 con 0x7fd7b80b1290 2024-01-11T09:43:35.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.458+0000 7fd7b67fc700 1 -- 172.21.15.27:0/313673805 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b00092e0 con 0x7fd7b80b1290 2024-01-11T09:43:35.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.458+0000 7fd7bddff700 1 -- 172.21.15.27:0/313673805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7b80b1290 msgr2=0x7fd7b80b16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:35.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.458+0000 7fd7bddff700 1 --2- 172.21.15.27:0/313673805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7b80b1290 0x7fd7b80b16f0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fd7b8076d60 tx=0x7fd7b0009840 comp rx=0 tx=0).stop 2024-01-11T09:43:35.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.458+0000 7fd7bddff700 1 -- 172.21.15.27:0/313673805 shutdown_connections 2024-01-11T09:43:35.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.458+0000 7fd7bddff700 1 --2- 172.21.15.27:0/313673805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7b80b1290 0x7fd7b80b16f0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:35.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.458+0000 7fd7bddff700 1 --2- 172.21.15.27:0/313673805 >> v2:172.21.15.77:3300/0 conn(0x7fd7b8157360 0x7fd7b806f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:35.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.458+0000 7fd7bddff700 1 --2- 172.21.15.27:0/313673805 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7b8157730 0x7fd7b80cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:35.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.458+0000 7fd7bddff700 1 -- 172.21.15.27:0/313673805 >> 172.21.15.27:0/313673805 conn(0x7fd7b80680b0 msgr2=0x7fd7b80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:35.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.458+0000 7fd7bddff700 1 -- 172.21.15.27:0/313673805 shutdown_connections 2024-01-11T09:43:35.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.459+0000 7fd7bddff700 1 -- 172.21.15.27:0/313673805 wait complete. 2024-01-11T09:43:35.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.461+0000 7fd7bddff700 1 Processor -- start 2024-01-11T09:43:35.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.461+0000 7fd7bddff700 1 -- start start 2024-01-11T09:43:35.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.462+0000 7fd7bddff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7b80b1290 0x7fd7b80afb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:35.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.462+0000 7fd7bddff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7b8157360 0x7fd7b8149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:35.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.462+0000 7fd7bddff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7b8157730 0x7fd7b80b02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:35.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.462+0000 7fd7bddff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b815c050 con 0x7fd7b80b1290 2024-01-11T09:43:35.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.462+0000 7fd7bddff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b814a0d0 con 0x7fd7b8157360 2024-01-11T09:43:35.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.462+0000 7fd7bddff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b815b4d0 con 0x7fd7b8157730 2024-01-11T09:43:35.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.462+0000 7fd7b77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7b80b1290 0x7fd7b80afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:35.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.462+0000 7fd7b77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7b80b1290 0x7fd7b80afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41018/0 (socket says 172.21.15.27:41018) 2024-01-11T09:43:35.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.462+0000 7fd7b77fe700 1 -- 172.21.15.27:0/1878140422 learned_addr learned my addr 172.21.15.27:0/1878140422 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:35.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.463+0000 7fd7b6ffd700 1 --2- 172.21.15.27:0/1878140422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7b8157360 0x7fd7b8149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:35.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.463+0000 7fd7b7fff700 1 --2- 172.21.15.27:0/1878140422 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7b8157730 0x7fd7b80b02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:35.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.463+0000 7fd7b77fe700 1 -- 172.21.15.27:0/1878140422 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7b8157730 msgr2=0x7fd7b80b02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:35.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.463+0000 7fd7b77fe700 1 --2- 172.21.15.27:0/1878140422 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7b8157730 0x7fd7b80b02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:35.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.463+0000 7fd7b77fe700 1 -- 172.21.15.27:0/1878140422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7b8157360 msgr2=0x7fd7b8149b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:35.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.463+0000 7fd7b77fe700 1 --2- 172.21.15.27:0/1878140422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7b8157360 0x7fd7b8149b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:35.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.463+0000 7fd7b77fe700 1 -- 172.21.15.27:0/1878140422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b0009ae0 con 0x7fd7b80b1290 2024-01-11T09:43:35.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.463+0000 7fd7b77fe700 1 --2- 172.21.15.27:0/1878140422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7b80b1290 0x7fd7b80afb40 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800d890 tx=0x7fd7a8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:35.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.463+0000 7fd79f7fe700 1 -- 172.21.15.27:0/1878140422 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a80125f0 con 0x7fd7b80b1290 2024-01-11T09:43:35.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.463+0000 7fd79f7fe700 1 -- 172.21.15.27:0/1878140422 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a8012750 con 0x7fd7b80b1290 2024-01-11T09:43:35.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.463+0000 7fd79f7fe700 1 -- 172.21.15.27:0/1878140422 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8012af0 con 0x7fd7b80b1290 2024-01-11T09:43:35.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.463+0000 7fd7bddff700 1 -- 172.21.15.27:0/1878140422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7b815b720 con 0x7fd7b80b1290 2024-01-11T09:43:35.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.464+0000 7fd7bddff700 1 -- 172.21.15.27:0/1878140422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b815bce0 con 0x7fd7b80b1290 2024-01-11T09:43:35.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.465+0000 7fd7bddff700 1 -- 172.21.15.27:0/1878140422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7b815c330 con 0x7fd7b80b1290 2024-01-11T09:43:35.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.466+0000 7fd79f7fe700 1 -- 172.21.15.27:0/1878140422 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd7a801e460 con 0x7fd7b80b1290 2024-01-11T09:43:35.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.467+0000 7fd79f7fe700 1 --2- 172.21.15.27:0/1878140422 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd7a4066f30 0x7fd7a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:35.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.467+0000 7fd79f7fe700 1 -- 172.21.15.27:0/1878140422 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5883+0+0 (secure 0 0 0) 0x7fd7a8084b10 con 0x7fd7b80b1290 2024-01-11T09:43:35.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.467+0000 7fd7b6ffd700 1 --2- 172.21.15.27:0/1878140422 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd7a4066f30 0x7fd7a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:35.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.468+0000 7fd7b6ffd700 1 --2- 172.21.15.27:0/1878140422 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd7a4066f30 0x7fd7a40693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd7b80b1260 tx=0x7fd7ac008040 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:35.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.471+0000 7fd79f7fe700 1 -- 172.21.15.27:0/1878140422 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd7a8051cd0 con 0x7fd7b80b1290 2024-01-11T09:43:35.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:35.691+0000 7fd7bddff700 1 -- 172.21.15.27:0/1878140422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd7b804c3c0 con 0x7fd7b80b1290 2024-01-11T09:43:36.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:36.909+0000 7fd79f7fe700 1 -- 172.21.15.27:0/1878140422 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v194) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd7a80552f0 con 0x7fd7b80b1290 2024-01-11T09:43:36.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:36.913+0000 7fd7bddff700 1 -- 172.21.15.27:0/1878140422 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd7a4066f30 msgr2=0x7fd7a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:36.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:36.913+0000 7fd7bddff700 1 --2- 172.21.15.27:0/1878140422 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd7a4066f30 0x7fd7a40693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd7b80b1260 tx=0x7fd7ac008040 comp rx=0 tx=0).stop 2024-01-11T09:43:36.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:36.913+0000 7fd7bddff700 1 -- 172.21.15.27:0/1878140422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7b80b1290 msgr2=0x7fd7b80afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:36.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:36.913+0000 7fd7bddff700 1 --2- 172.21.15.27:0/1878140422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7b80b1290 0x7fd7b80afb40 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800d890 tx=0x7fd7a8008040 comp rx=0 tx=0).stop 2024-01-11T09:43:36.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:36.913+0000 7fd7bddff700 1 -- 172.21.15.27:0/1878140422 shutdown_connections 2024-01-11T09:43:36.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:36.913+0000 7fd7bddff700 1 --2- 172.21.15.27:0/1878140422 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd7b8157730 0x7fd7b80b02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:36.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:36.913+0000 7fd7bddff700 1 --2- 172.21.15.27:0/1878140422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7b80b1290 0x7fd7b80afb40 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:36.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:36.913+0000 7fd7bddff700 1 --2- 172.21.15.27:0/1878140422 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd7a4066f30 0x7fd7a40693e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:36.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:36.913+0000 7fd7bddff700 1 --2- 172.21.15.27:0/1878140422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7b8157360 0x7fd7b8149b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:36.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:36.913+0000 7fd7bddff700 1 -- 172.21.15.27:0/1878140422 >> 172.21.15.27:0/1878140422 conn(0x7fd7b80680b0 msgr2=0x7fd7b80ac700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:36.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:36.913+0000 7fd7bddff700 1 -- 172.21.15.27:0/1878140422 shutdown_connections 2024-01-11T09:43:36.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:36.913+0000 7fd7bddff700 1 -- 172.21.15.27:0/1878140422 wait complete. 2024-01-11T09:43:36.906 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-11T09:43:36.926 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-11T09:43:37.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.139+0000 7fee35dff700 1 Processor -- start 2024-01-11T09:43:37.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.139+0000 7fee35dff700 1 -- start start 2024-01-11T09:43:37.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.140+0000 7fee35dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee300b03e0 0x7fee300b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:37.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.140+0000 7fee35dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fee300b07b0 0x7fee300b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:37.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.140+0000 7fee35dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fee30159200 0x7fee30159640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:37.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.140+0000 7fee35dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee300cd730 con 0x7fee30159200 2024-01-11T09:43:37.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.140+0000 7fee35dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee300ccb20 con 0x7fee300b07b0 2024-01-11T09:43:37.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.140+0000 7fee35dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee300cce00 con 0x7fee300b03e0 2024-01-11T09:43:37.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.140+0000 7fee2ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fee30159200 0x7fee30159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:37.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.140+0000 7fee2ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fee30159200 0x7fee30159640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41034/0 (socket says 172.21.15.27:41034) 2024-01-11T09:43:37.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.140+0000 7fee2ffff700 1 -- 172.21.15.27:0/3405204390 learned_addr learned my addr 172.21.15.27:0/3405204390 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:37.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.141+0000 7fee2ffff700 1 -- 172.21.15.27:0/3405204390 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee300b03e0 msgr2=0x7fee300b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:37.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.141+0000 7fee2effd700 1 --2- 172.21.15.27:0/3405204390 >> v2:172.21.15.77:3300/0 conn(0x7fee300b07b0 0x7fee300b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:37.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.141+0000 7fee2f7fe700 1 --2- 172.21.15.27:0/3405204390 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee300b03e0 0x7fee300b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:37.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.141+0000 7fee2ffff700 1 --2- 172.21.15.27:0/3405204390 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee300b03e0 0x7fee300b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.141+0000 7fee2ffff700 1 -- 172.21.15.27:0/3405204390 >> v2:172.21.15.77:3300/0 conn(0x7fee300b07b0 msgr2=0x7fee300b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:37.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.141+0000 7fee2ffff700 1 --2- 172.21.15.27:0/3405204390 >> v2:172.21.15.77:3300/0 conn(0x7fee300b07b0 0x7fee300b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.141+0000 7fee2ffff700 1 -- 172.21.15.27:0/3405204390 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee300cd0e0 con 0x7fee30159200 2024-01-11T09:43:37.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.141+0000 7fee2effd700 1 --2- 172.21.15.27:0/3405204390 >> v2:172.21.15.77:3300/0 conn(0x7fee300b07b0 0x7fee300b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:43:37.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.141+0000 7fee2f7fe700 1 --2- 172.21.15.27:0/3405204390 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee300b03e0 0x7fee300b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:43:37.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.141+0000 7fee2ffff700 1 --2- 172.21.15.27:0/3405204390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee30159200 0x7fee30159640 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fee300893d0 tx=0x7fee28009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=574270b538e2d512 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:37.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.141+0000 7fee2e7fc700 1 -- 172.21.15.27:0/3405204390 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2800e070 con 0x7fee30159200 2024-01-11T09:43:37.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.141+0000 7fee2e7fc700 1 -- 172.21.15.27:0/3405204390 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2800ba50 con 0x7fee30159200 2024-01-11T09:43:37.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.141+0000 7fee2e7fc700 1 -- 172.21.15.27:0/3405204390 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee280092e0 con 0x7fee30159200 2024-01-11T09:43:37.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.142+0000 7fee35dff700 1 -- 172.21.15.27:0/3405204390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee30159200 msgr2=0x7fee30159640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:37.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.142+0000 7fee35dff700 1 --2- 172.21.15.27:0/3405204390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee30159200 0x7fee30159640 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fee300893d0 tx=0x7fee28009840 comp rx=0 tx=0).stop 2024-01-11T09:43:37.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.142+0000 7fee35dff700 1 -- 172.21.15.27:0/3405204390 shutdown_connections 2024-01-11T09:43:37.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.142+0000 7fee35dff700 1 --2- 172.21.15.27:0/3405204390 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee300b03e0 0x7fee300b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.142+0000 7fee35dff700 1 --2- 172.21.15.27:0/3405204390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee30159200 0x7fee30159640 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.142+0000 7fee35dff700 1 --2- 172.21.15.27:0/3405204390 >> v2:172.21.15.77:3300/0 conn(0x7fee300b07b0 0x7fee300b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.142+0000 7fee35dff700 1 -- 172.21.15.27:0/3405204390 >> 172.21.15.27:0/3405204390 conn(0x7fee3006ed40 msgr2=0x7fee3006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:37.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.142+0000 7fee35dff700 1 -- 172.21.15.27:0/3405204390 shutdown_connections 2024-01-11T09:43:37.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.142+0000 7fee35dff700 1 -- 172.21.15.27:0/3405204390 wait complete. 2024-01-11T09:43:37.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.144+0000 7fee35dff700 1 Processor -- start 2024-01-11T09:43:37.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.144+0000 7fee35dff700 1 -- start start 2024-01-11T09:43:37.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.146+0000 7fee35dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee300b03e0 0x7fee30158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:37.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.146+0000 7fee35dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee300b07b0 0x7fee3015bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:37.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.146+0000 7fee35dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee30159200 0x7fee3015c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:37.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.146+0000 7fee35dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3015cdd0 con 0x7fee30159200 2024-01-11T09:43:37.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.146+0000 7fee35dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee300c45f0 con 0x7fee300b03e0 2024-01-11T09:43:37.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.146+0000 7fee35dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee3015c5a0 con 0x7fee300b07b0 2024-01-11T09:43:37.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.146+0000 7fee2ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee30159200 0x7fee3015c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:37.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.146+0000 7fee2ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee30159200 0x7fee3015c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41036/0 (socket says 172.21.15.27:41036) 2024-01-11T09:43:37.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.146+0000 7fee2ffff700 1 -- 172.21.15.27:0/2006250177 learned_addr learned my addr 172.21.15.27:0/2006250177 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:37.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.146+0000 7fee2effd700 1 --2- 172.21.15.27:0/2006250177 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee300b07b0 0x7fee3015bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:37.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.146+0000 7fee2f7fe700 1 --2- 172.21.15.27:0/2006250177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee300b03e0 0x7fee30158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:37.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.146+0000 7fee2ffff700 1 -- 172.21.15.27:0/2006250177 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee300b07b0 msgr2=0x7fee3015bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:37.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.146+0000 7fee2ffff700 1 --2- 172.21.15.27:0/2006250177 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee300b07b0 0x7fee3015bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.146+0000 7fee2ffff700 1 -- 172.21.15.27:0/2006250177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee300b03e0 msgr2=0x7fee30158920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:37.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.146+0000 7fee2ffff700 1 --2- 172.21.15.27:0/2006250177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee300b03e0 0x7fee30158920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.146+0000 7fee2ffff700 1 -- 172.21.15.27:0/2006250177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee28009ae0 con 0x7fee30159200 2024-01-11T09:43:37.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.146+0000 7fee2f7fe700 1 --2- 172.21.15.27:0/2006250177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee300b03e0 0x7fee30158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:37.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.146+0000 7fee2effd700 1 --2- 172.21.15.27:0/2006250177 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee300b07b0 0x7fee3015bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:37.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.147+0000 7fee2ffff700 1 --2- 172.21.15.27:0/2006250177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee30159200 0x7fee3015c060 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fee28009fd0 tx=0x7fee28018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:37.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.147+0000 7fee177fe700 1 -- 172.21.15.27:0/2006250177 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee280032f0 con 0x7fee30159200 2024-01-11T09:43:37.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.147+0000 7fee177fe700 1 -- 172.21.15.27:0/2006250177 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee28007e60 con 0x7fee30159200 2024-01-11T09:43:37.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.147+0000 7fee35dff700 1 -- 172.21.15.27:0/2006250177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee3015c810 con 0x7fee30159200 2024-01-11T09:43:37.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.147+0000 7fee177fe700 1 -- 172.21.15.27:0/2006250177 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2801e660 con 0x7fee30159200 2024-01-11T09:43:37.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.147+0000 7fee35dff700 1 -- 172.21.15.27:0/2006250177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee300c4280 con 0x7fee30159200 2024-01-11T09:43:37.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.148+0000 7fee35dff700 1 -- 172.21.15.27:0/2006250177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee300c40e0 con 0x7fee30159200 2024-01-11T09:43:37.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.149+0000 7fee177fe700 1 -- 172.21.15.27:0/2006250177 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fee28003040 con 0x7fee30159200 2024-01-11T09:43:37.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.150+0000 7fee177fe700 1 --2- 172.21.15.27:0/2006250177 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fee1c066f30 0x7fee1c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:37.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.150+0000 7fee177fe700 1 -- 172.21.15.27:0/2006250177 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 4937+0+0 (secure 0 0 0) 0x7fee28086540 con 0x7fee30159200 2024-01-11T09:43:37.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.151+0000 7fee2f7fe700 1 --2- 172.21.15.27:0/2006250177 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fee1c066f30 0x7fee1c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:37.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.151+0000 7fee2f7fe700 1 --2- 172.21.15.27:0/2006250177 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fee1c066f30 0x7fee1c0693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fee20006fd0 tx=0x7fee20008040 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:37.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.155+0000 7fee177fe700 1 -- 172.21.15.27:0/2006250177 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fee300c40e0 con 0x7fee30159200 2024-01-11T09:43:37.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.441+0000 7fee35dff700 1 -- 172.21.15.27:0/2006250177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fee300c40e0 con 0x7fee30159200 2024-01-11T09:43:37.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.441+0000 7fee177fe700 1 -- 172.21.15.27:0/2006250177 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v48) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fee300c40e0 con 0x7fee30159200 2024-01-11T09:43:37.433 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:43:37.434 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:43:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-11T09:43:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-11T09:43:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lQm3yLhAAv6erz5hMfat8EUzRA5luMA==", 2024-01-11T09:43:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:43:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:43:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:43:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:43:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:43:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:43:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:37.434 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-11T09:43:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9l+FG8ChAACHndtVLx7qS3sidPniSycA==", 2024-01-11T09:43:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:43:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:43:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:43:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:43:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:43:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:43:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-11T09:43:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l1NnEOhAALHXsc5es7hufCthGdRzkyw==", 2024-01-11T09:43:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:43:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:43:37.435 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:43:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:43:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:43:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:43:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-11T09:43:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lruoWFhAAm8AmrZtTJhYI1+19F79RPQ==", 2024-01-11T09:43:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:43:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:43:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:43:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:43:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:43:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:43:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-11T09:43:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA2op9ltZygORAAZr1X1nCN0ce8rIVaGDpFyQ==", 2024-01-11T09:43:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:43:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:43:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:43:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:43:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:43:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-11T09:43:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA6op9lqSRPAhAAmtyCsyK5ZVGX2wfhOCMuFQ==", 2024-01-11T09:43:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:43:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:43:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:43:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:43:37.437 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:43:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-11T09:43:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA9op9lZcs8BhAA8j+U+jPOoy+fliVWZAPFYg==", 2024-01-11T09:43:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:43:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:43:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:43:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:43:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:43:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-11T09:43:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBAop9lFuCUChAAP6si/2W1mxH4ecQJvuumkw==", 2024-01-11T09:43:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:43:37.438 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:43:37.439 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:43:37.439 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:43:37.439 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.439 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:43:37.439 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:37.439 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-11T09:43:37.439 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBGop9lOpwBIRAAcpcsjwmX2IldcKHJWAgHuA==", 2024-01-11T09:43:37.439 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:43:37.439 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:43:37.439 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:43:37.439 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:43:37.439 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.439 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:43:37.439 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:37.439 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-11T09:43:37.439 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBJop9l4q8hJRAAK8mAYVRQMoWS6wBOU69HmA==", 2024-01-11T09:43:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:43:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:43:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:43:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:43:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:43:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-11T09:43:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBMop9lqMF9KBAAFgFFI2GWt9AGif4AoScThw==", 2024-01-11T09:43:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:43:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:43:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:43:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:43:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.440 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:43:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-11T09:43:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBPop9lN7eDLBAAKswPZFfs0g2AaSe6V+pbbQ==", 2024-01-11T09:43:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:43:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:43:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:43:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:43:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:43:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-11T09:43:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lT5okIBAABy8PbJJbA062VNmb1Y8qhg==", 2024-01-11T09:43:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:43:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:43:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-01-11T09:43:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-11T09:43:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:43:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-11T09:43:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lDf37LhAAqV+EabKXJEW1sS5xtu1tXw==", 2024-01-11T09:43:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:43:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:43:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T09:43:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T09:43:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T09:43:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:43:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-11T09:43:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lWayxJxAAcalM8QSOGtbCXxjYZGaNQA==", 2024-01-11T09:43:37.442 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:43:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:43:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T09:43:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T09:43:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T09:43:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:43:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-11T09:43:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lHD4tNhAAdmnFBmMrK+QhQYdpbzW0mg==", 2024-01-11T09:43:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:43:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:43:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T09:43:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T09:43:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T09:43:37.443 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:43:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-11T09:43:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAwop9lGPtDMxAAJ8FjrSEdCtQL8AhtAyqnTw==", 2024-01-11T09:43:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:43:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T09:43:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:43:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:43:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:43:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:43:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-11T09:43:37.444 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lXBZLJxAAXa1S8Vkm3MiV4igtNbg/1Q==", 2024-01-11T09:43:37.445 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:43:37.445 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T09:43:37.445 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T09:43:37.445 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:43:37.445 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.445 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:43:37.445 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:37.445 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-11T09:43:37.445 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l3ZGoHxAANqzytTTvWYSomG+RFsFDyg==", 2024-01-11T09:43:37.445 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:43:37.445 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T09:43:37.445 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T09:43:37.445 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:43:37.445 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.445 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:37.445 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-11T09:43:37.446 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:43:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.444+0000 7fee35dff700 1 -- 172.21.15.27:0/2006250177 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fee1c066f30 msgr2=0x7fee1c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.444+0000 7fee35dff700 1 --2- 172.21.15.27:0/2006250177 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fee1c066f30 0x7fee1c0693e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fee20006fd0 tx=0x7fee20008040 comp rx=0 tx=0).stop 2024-01-11T09:43:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.444+0000 7fee35dff700 1 -- 172.21.15.27:0/2006250177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee30159200 msgr2=0x7fee3015c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.444+0000 7fee35dff700 1 --2- 172.21.15.27:0/2006250177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee30159200 0x7fee3015c060 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fee28009fd0 tx=0x7fee28018000 comp rx=0 tx=0).stop 2024-01-11T09:43:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.444+0000 7fee35dff700 1 -- 172.21.15.27:0/2006250177 shutdown_connections 2024-01-11T09:43:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.444+0000 7fee35dff700 1 --2- 172.21.15.27:0/2006250177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee30159200 0x7fee3015c060 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.444+0000 7fee35dff700 1 --2- 172.21.15.27:0/2006250177 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fee1c066f30 0x7fee1c0693e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.444+0000 7fee35dff700 1 --2- 172.21.15.27:0/2006250177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee300b03e0 0x7fee30158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.444+0000 7fee35dff700 1 --2- 172.21.15.27:0/2006250177 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fee300b07b0 0x7fee3015bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.444+0000 7fee35dff700 1 -- 172.21.15.27:0/2006250177 >> 172.21.15.27:0/2006250177 conn(0x7fee3006ed40 msgr2=0x7fee300b2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.444+0000 7fee35dff700 1 -- 172.21.15.27:0/2006250177 shutdown_connections 2024-01-11T09:43:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.444+0000 7fee35dff700 1 -- 172.21.15.27:0/2006250177 wait complete. 2024-01-11T09:43:37.454 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:43:37.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.662+0000 7f5b811ff700 1 Processor -- start 2024-01-11T09:43:37.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.662+0000 7f5b811ff700 1 -- start start 2024-01-11T09:43:37.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.663+0000 7f5b811ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b7c0b09e0 0x7f5b7c1610a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:37.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.663+0000 7f5b811ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7c157390 con 0x7f5b7c069d40 2024-01-11T09:43:37.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.663+0000 7f5b811ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7c06a200 con 0x7f5b7c0c8620 2024-01-11T09:43:37.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.663+0000 7f5b811ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c0adcd0 con 0x7f5b7c0b09e0 2024-01-11T09:43:37.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.664+0000 7f5b7a59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5b7c069d40 0x7f5b7c0b02d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33492/0 (socket says 172.21.15.27:33492) 2024-01-11T09:43:37.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.664+0000 7f5b7a59c700 1 -- 172.21.15.27:0/2537866805 learned_addr learned my addr 172.21.15.27:0/2537866805 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:37.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.664+0000 7f5b7b59e700 1 --2- 172.21.15.27:0/2537866805 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b7c0b09e0 0x7f5b7c1610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:37.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.664+0000 7f5b79d9b700 1 -- 172.21.15.27:0/2537866805 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476511380 0 0) 0x7f5b7c157390 con 0x7f5b7c069d40 2024-01-11T09:43:37.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.664+0000 7f5b79d9b700 1 -- 172.21.15.27:0/2537866805 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b64003570 con 0x7f5b7c069d40 2024-01-11T09:43:37.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.664+0000 7f5b79d9b700 1 -- 172.21.15.27:0/2537866805 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834548419 0 0) 0x7f5b7c06a200 con 0x7f5b7c0c8620 2024-01-11T09:43:37.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.664+0000 7f5b79d9b700 1 -- 172.21.15.27:0/2537866805 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b7c157390 con 0x7f5b7c0c8620 2024-01-11T09:43:37.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.664+0000 7f5b79d9b700 1 -- 172.21.15.27:0/2537866805 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1146590497 0 0) 0x7f5b64003570 con 0x7f5b7c069d40 2024-01-11T09:43:37.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.664+0000 7f5b79d9b700 1 -- 172.21.15.27:0/2537866805 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b7c0b09e0 msgr2=0x7f5b7c1610a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:37.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.664+0000 7f5b79d9b700 1 --2- 172.21.15.27:0/2537866805 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b7c0b09e0 0x7f5b7c1610a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.664+0000 7f5b79d9b700 1 -- 172.21.15.27:0/2537866805 >> v1:172.21.15.77:6789/0 conn(0x7f5b7c0c8620 legacy=0x7f5b7c0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:37.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.664+0000 7f5b79d9b700 1 -- 172.21.15.27:0/2537866805 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c156d40 con 0x7f5b7c069d40 2024-01-11T09:43:37.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.664+0000 7f5b79d9b700 1 -- 172.21.15.27:0/2537866805 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f5b70001470 con 0x7f5b7c069d40 2024-01-11T09:43:37.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.664+0000 7f5b79d9b700 1 -- 172.21.15.27:0/2537866805 >> v1:172.21.15.27:6789/0 conn(0x7f5b7c069d40 legacy=0x7f5b7c0b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:37.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.665+0000 7f5b79d9b700 1 --2- 172.21.15.27:0/2537866805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b64004c80 0x7f5b64005070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:37.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.665+0000 7f5b79d9b700 1 --2- 172.21.15.27:0/2537866805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b640037b0 0x7f5b64003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:37.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.665+0000 7f5b79d9b700 1 --2- 172.21.15.27:0/2537866805 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b7c0b09e0 0x7f5b64004160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:37.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.665+0000 7f5b79d9b700 1 -- 172.21.15.27:0/2537866805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c0adcd0 con 0x7f5b640037b0 2024-01-11T09:43:37.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.665+0000 7f5b79d9b700 1 -- 172.21.15.27:0/2537866805 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6400bb40 con 0x7f5b64004c80 2024-01-11T09:43:37.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.665+0000 7f5b79d9b700 1 -- 172.21.15.27:0/2537866805 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6400df10 con 0x7f5b7c0b09e0 2024-01-11T09:43:37.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.665+0000 7f5b7ad9d700 1 --2- 172.21.15.27:0/2537866805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b64004c80 0x7f5b64005070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:37.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.665+0000 7f5b7a59c700 1 --2- 172.21.15.27:0/2537866805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b640037b0 0x7f5b64003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:37.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.665+0000 7f5b7b59e700 1 --2- 172.21.15.27:0/2537866805 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b7c0b09e0 0x7f5b64004160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:37.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.665+0000 7f5b7ad9d700 1 -- 172.21.15.27:0/2537866805 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b7c0b09e0 msgr2=0x7f5b64004160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:37.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.665+0000 7f5b7ad9d700 1 --2- 172.21.15.27:0/2537866805 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b7c0b09e0 0x7f5b64004160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.665+0000 7f5b7ad9d700 1 -- 172.21.15.27:0/2537866805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b640037b0 msgr2=0x7f5b64003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:37.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.665+0000 7f5b7ad9d700 1 --2- 172.21.15.27:0/2537866805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b640037b0 0x7f5b64003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.665+0000 7f5b7ad9d700 1 -- 172.21.15.27:0/2537866805 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b64003570 con 0x7f5b64004c80 2024-01-11T09:43:37.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.665+0000 7f5b7a59c700 1 --2- 172.21.15.27:0/2537866805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b640037b0 0x7f5b64003c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:43:37.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.666+0000 7f5b7ad9d700 1 --2- 172.21.15.27:0/2537866805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b64004c80 0x7f5b64005070 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c001a10 tx=0x7f5b6c007d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=108b2668fce127f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:37.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.666+0000 7f5b79d9b700 1 -- 172.21.15.27:0/2537866805 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c00c5d0 con 0x7f5b64004c80 2024-01-11T09:43:37.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.666+0000 7f5b79d9b700 1 -- 172.21.15.27:0/2537866805 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6c00c730 con 0x7f5b64004c80 2024-01-11T09:43:37.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.666+0000 7f5b79d9b700 1 -- 172.21.15.27:0/2537866805 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c00cad0 con 0x7f5b64004c80 2024-01-11T09:43:37.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.666+0000 7f5b811ff700 1 -- 172.21.15.27:0/2537866805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b64004c80 msgr2=0x7f5b64005070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:37.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.666+0000 7f5b811ff700 1 --2- 172.21.15.27:0/2537866805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b64004c80 0x7f5b64005070 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c001a10 tx=0x7f5b6c007d30 comp rx=0 tx=0).stop 2024-01-11T09:43:37.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.666+0000 7f5b7b59e700 1 -- 172.21.15.27:0/2537866805 reap_dead start 2024-01-11T09:43:37.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.666+0000 7f5b811ff700 1 -- 172.21.15.27:0/2537866805 shutdown_connections 2024-01-11T09:43:37.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.666+0000 7f5b811ff700 1 -- 172.21.15.27:0/2537866805 >> 172.21.15.27:0/2537866805 conn(0x7f5b7c0680e0 msgr2=0x7f5b7c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:37.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.667+0000 7f5b811ff700 1 -- 172.21.15.27:0/2537866805 shutdown_connections 2024-01-11T09:43:37.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.667+0000 7f5b811ff700 1 -- 172.21.15.27:0/2537866805 wait complete. 2024-01-11T09:43:37.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.668+0000 7f5b811ff700 1 Processor -- start 2024-01-11T09:43:37.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.668+0000 7f5b811ff700 1 -- start start 2024-01-11T09:43:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.669+0000 7f5b811ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b64004c80 0x7f5b7c0ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.669+0000 7f5b811ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b7c143600 0x7f5b7c0c9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.669+0000 7f5b811ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b7c0c74f0 0x7f5b7c0c7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.669+0000 7f5b811ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c0c7ed0 con 0x7f5b64004c80 2024-01-11T09:43:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.669+0000 7f5b811ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c0cb880 con 0x7f5b7c143600 2024-01-11T09:43:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.669+0000 7f5b811ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c0cac70 con 0x7f5b7c0c74f0 2024-01-11T09:43:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.669+0000 7f5b7ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b64004c80 0x7f5b7c0ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.669+0000 7f5b7ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b64004c80 0x7f5b7c0ca690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41056/0 (socket says 172.21.15.27:41056) 2024-01-11T09:43:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.669+0000 7f5b7ad9d700 1 -- 172.21.15.27:0/2939677319 learned_addr learned my addr 172.21.15.27:0/2939677319 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.670+0000 7f5b7b59e700 1 --2- 172.21.15.27:0/2939677319 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b7c0c74f0 0x7f5b7c0c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.670+0000 7f5b7a59c700 1 --2- 172.21.15.27:0/2939677319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b7c143600 0x7f5b7c0c9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.670+0000 7f5b7ad9d700 1 -- 172.21.15.27:0/2939677319 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b7c0c74f0 msgr2=0x7f5b7c0c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:37.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.670+0000 7f5b7ad9d700 1 --2- 172.21.15.27:0/2939677319 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b7c0c74f0 0x7f5b7c0c7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.670+0000 7f5b7ad9d700 1 -- 172.21.15.27:0/2939677319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b7c143600 msgr2=0x7f5b7c0c9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.670+0000 7f5b7ad9d700 1 --2- 172.21.15.27:0/2939677319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b7c143600 0x7f5b7c0c9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.670+0000 7f5b7ad9d700 1 -- 172.21.15.27:0/2939677319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c0caf50 con 0x7f5b64004c80 2024-01-11T09:43:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.670+0000 7f5b7b59e700 1 --2- 172.21.15.27:0/2939677319 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b7c0c74f0 0x7f5b7c0c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.670+0000 7f5b7ad9d700 1 --2- 172.21.15.27:0/2939677319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b64004c80 0x7f5b7c0ca690 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c008fd0 tx=0x7f5b6c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.670+0000 7f5b62ffd700 1 -- 172.21.15.27:0/2939677319 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c00c3f0 con 0x7f5b64004c80 2024-01-11T09:43:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.670+0000 7f5b62ffd700 1 -- 172.21.15.27:0/2939677319 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6c001360 con 0x7f5b64004c80 2024-01-11T09:43:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.670+0000 7f5b62ffd700 1 -- 172.21.15.27:0/2939677319 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c00cad0 con 0x7f5b64004c80 2024-01-11T09:43:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.670+0000 7f5b811ff700 1 -- 172.21.15.27:0/2939677319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b7c0cb230 con 0x7f5b64004c80 2024-01-11T09:43:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.670+0000 7f5b811ff700 1 -- 172.21.15.27:0/2939677319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b7c0ca1e0 con 0x7f5b64004c80 2024-01-11T09:43:37.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.671+0000 7f5b811ff700 1 -- 172.21.15.27:0/2939677319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b7c0718d0 con 0x7f5b64004c80 2024-01-11T09:43:37.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.672+0000 7f5b62ffd700 1 -- 172.21.15.27:0/2939677319 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5b6c019070 con 0x7f5b64004c80 2024-01-11T09:43:37.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.673+0000 7f5b62ffd700 1 --2- 172.21.15.27:0/2939677319 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5b68066f30 0x7f5b680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:37.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.673+0000 7f5b62ffd700 1 -- 172.21.15.27:0/2939677319 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 4937+0+0 (secure 0 0 0) 0x7f5b6c085980 con 0x7f5b64004c80 2024-01-11T09:43:37.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.674+0000 7f5b7a59c700 1 --2- 172.21.15.27:0/2939677319 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5b68066f30 0x7f5b680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:37.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.674+0000 7f5b7a59c700 1 --2- 172.21.15.27:0/2939677319 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5b68066f30 0x7f5b680693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5b70007fd0 tx=0x7f5b70001df0 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:37.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.677+0000 7f5b62ffd700 1 -- 172.21.15.27:0/2939677319 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5b6c052f10 con 0x7f5b64004c80 2024-01-11T09:43:37.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.884+0000 7f5b811ff700 1 -- 172.21.15.27:0/2939677319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b7c157020 con 0x7f5b64004c80 2024-01-11T09:43:37.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.885+0000 7f5b62ffd700 1 -- 172.21.15.27:0/2939677319 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v194) v1 ==== 74+0+12230 (secure 0 0 0) 0x7f5b6c056530 con 0x7f5b64004c80 2024-01-11T09:43:37.877 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:43:37.877 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":194,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:43:36.905204+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:43:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.888+0000 7f5b811ff700 1 -- 172.21.15.27:0/2939677319 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5b68066f30 msgr2=0x7f5b680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.888+0000 7f5b811ff700 1 --2- 172.21.15.27:0/2939677319 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5b68066f30 0x7f5b680693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5b70007fd0 tx=0x7f5b70001df0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.888+0000 7f5b811ff700 1 -- 172.21.15.27:0/2939677319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b64004c80 msgr2=0x7f5b7c0ca690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.888+0000 7f5b811ff700 1 --2- 172.21.15.27:0/2939677319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b64004c80 0x7f5b7c0ca690 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c008fd0 tx=0x7f5b6c015000 comp rx=0 tx=0).stop 2024-01-11T09:43:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.888+0000 7f5b811ff700 1 -- 172.21.15.27:0/2939677319 shutdown_connections 2024-01-11T09:43:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.888+0000 7f5b811ff700 1 --2- 172.21.15.27:0/2939677319 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b7c0c74f0 0x7f5b7c0c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.888+0000 7f5b811ff700 1 --2- 172.21.15.27:0/2939677319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b64004c80 0x7f5b7c0ca690 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.888+0000 7f5b811ff700 1 --2- 172.21.15.27:0/2939677319 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5b68066f30 0x7f5b680693e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.888+0000 7f5b811ff700 1 --2- 172.21.15.27:0/2939677319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b7c143600 0x7f5b7c0c9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:37.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.888+0000 7f5b811ff700 1 -- 172.21.15.27:0/2939677319 >> 172.21.15.27:0/2939677319 conn(0x7f5b7c0680e0 msgr2=0x7f5b7c158a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:37.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.888+0000 7f5b811ff700 1 -- 172.21.15.27:0/2939677319 shutdown_connections 2024-01-11T09:43:37.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:37.888+0000 7f5b811ff700 1 -- 172.21.15.27:0/2939677319 wait complete. 2024-01-11T09:43:37.898 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-11T09:43:37.899 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T09:43:38.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.112+0000 7f9701dff700 1 Processor -- start 2024-01-11T09:43:38.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.113+0000 7f9701dff700 1 -- start start 2024-01-11T09:43:38.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.114+0000 7f9701dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96fc15a1b0 0x7f96fc15a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:38.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.114+0000 7f9701dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f96fc15ae70 0x7f96fc15b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:38.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.114+0000 7f9701dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96fc0cf850 con 0x7f96fc0ccd70 2024-01-11T09:43:38.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.114+0000 7f9701dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96fc0cec40 con 0x7f96fc15ae70 2024-01-11T09:43:38.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.114+0000 7f9701dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc0cef20 con 0x7f96fc15a1b0 2024-01-11T09:43:38.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.114+0000 7f96fbfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f96fc15ae70 0x7f96fc15b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:38.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.114+0000 7f96fbfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f96fc15ae70 0x7f96fc15b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41058/0 (socket says 172.21.15.27:41058) 2024-01-11T09:43:38.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.114+0000 7f96fbfff700 1 -- 172.21.15.27:0/2390999399 learned_addr learned my addr 172.21.15.27:0/2390999399 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:38.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.115+0000 7f96fb7fe700 1 --2- 172.21.15.27:0/2390999399 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96fc15a1b0 0x7f96fc15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:38.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.115+0000 7f96fbfff700 1 -- 172.21.15.27:0/2390999399 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96fc15a1b0 msgr2=0x7f96fc15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:38.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.115+0000 7f96fbfff700 1 --2- 172.21.15.27:0/2390999399 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96fc15a1b0 0x7f96fc15a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.115+0000 7f96fbfff700 1 -- 172.21.15.27:0/2390999399 >> v1:172.21.15.77:6789/0 conn(0x7f96fc0ccd70 legacy=0x7f96fc0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:38.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.115+0000 7f96fbfff700 1 -- 172.21.15.27:0/2390999399 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96fc0cf200 con 0x7f96fc15ae70 2024-01-11T09:43:38.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.115+0000 7f96fb7fe700 1 --2- 172.21.15.27:0/2390999399 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96fc15a1b0 0x7f96fc15a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:43:38.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.115+0000 7f96fbfff700 1 --2- 172.21.15.27:0/2390999399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc15ae70 0x7f96fc15b2f0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f96f4009f20 tx=0x7f96f4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4044104c6804e27c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:38.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.115+0000 7f96fa7fc700 1 -- 172.21.15.27:0/2390999399 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f400e070 con 0x7f96fc15ae70 2024-01-11T09:43:38.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.116+0000 7f96fa7fc700 1 -- 172.21.15.27:0/2390999399 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f400ba80 con 0x7f96fc15ae70 2024-01-11T09:43:38.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.116+0000 7f96fa7fc700 1 -- 172.21.15.27:0/2390999399 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f40092e0 con 0x7f96fc15ae70 2024-01-11T09:43:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.116+0000 7f9701dff700 1 -- 172.21.15.27:0/2390999399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc15ae70 msgr2=0x7f96fc15b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.116+0000 7f9701dff700 1 --2- 172.21.15.27:0/2390999399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc15ae70 0x7f96fc15b2f0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f96f4009f20 tx=0x7f96f4009840 comp rx=0 tx=0).stop 2024-01-11T09:43:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.116+0000 7f9701dff700 1 -- 172.21.15.27:0/2390999399 shutdown_connections 2024-01-11T09:43:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.116+0000 7f9701dff700 1 --2- 172.21.15.27:0/2390999399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc15ae70 0x7f96fc15b2f0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.116+0000 7f9701dff700 1 --2- 172.21.15.27:0/2390999399 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96fc15a1b0 0x7f96fc15a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.116+0000 7f9701dff700 1 -- 172.21.15.27:0/2390999399 >> 172.21.15.27:0/2390999399 conn(0x7f96fc06ed40 msgr2=0x7f96fc06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.116+0000 7f9701dff700 1 -- 172.21.15.27:0/2390999399 shutdown_connections 2024-01-11T09:43:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.116+0000 7f9701dff700 1 -- 172.21.15.27:0/2390999399 wait complete. 2024-01-11T09:43:38.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.118+0000 7f9701dff700 1 Processor -- start 2024-01-11T09:43:38.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.118+0000 7f9701dff700 1 -- start start 2024-01-11T09:43:38.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.119+0000 7f9701dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96fc0ccd70 0x7f96fc0ca170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:38.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.119+0000 7f9701dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96fc15a1b0 0x7f96fc14af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:38.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.119+0000 7f9701dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc15ae70 0x7f96fc0b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:38.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.119+0000 7f9701dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc0ca6b0 con 0x7f96fc15ae70 2024-01-11T09:43:38.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.119+0000 7f9701dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc0c9ce0 con 0x7f96fc15a1b0 2024-01-11T09:43:38.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.119+0000 7f9701dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc0c9110 con 0x7f96fc0ccd70 2024-01-11T09:43:38.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.119+0000 7f96fbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc15ae70 0x7f96fc0b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:38.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.119+0000 7f96fbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc15ae70 0x7f96fc0b5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41074/0 (socket says 172.21.15.27:41074) 2024-01-11T09:43:38.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.119+0000 7f96fbfff700 1 -- 172.21.15.27:0/2014054621 learned_addr learned my addr 172.21.15.27:0/2014054621 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:38.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.119+0000 7f96fb7fe700 1 --2- 172.21.15.27:0/2014054621 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96fc0ccd70 0x7f96fc0ca170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:38.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.119+0000 7f96faffd700 1 --2- 172.21.15.27:0/2014054621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96fc15a1b0 0x7f96fc14af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:38.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.120+0000 7f96fbfff700 1 -- 172.21.15.27:0/2014054621 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96fc0ccd70 msgr2=0x7f96fc0ca170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.120+0000 7f96fbfff700 1 --2- 172.21.15.27:0/2014054621 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96fc0ccd70 0x7f96fc0ca170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.120+0000 7f96fbfff700 1 -- 172.21.15.27:0/2014054621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96fc15a1b0 msgr2=0x7f96fc14af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.120+0000 7f96fbfff700 1 --2- 172.21.15.27:0/2014054621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96fc15a1b0 0x7f96fc14af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.120+0000 7f96fbfff700 1 -- 172.21.15.27:0/2014054621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f4009ae0 con 0x7f96fc15ae70 2024-01-11T09:43:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.120+0000 7f96fbfff700 1 --2- 172.21.15.27:0/2014054621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc15ae70 0x7f96fc0b5a50 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f96f400b6c0 tx=0x7f96f4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.120+0000 7f96e37fe700 1 -- 172.21.15.27:0/2014054621 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f4008d50 con 0x7f96fc15ae70 2024-01-11T09:43:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.120+0000 7f96e37fe700 1 -- 172.21.15.27:0/2014054621 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f4026460 con 0x7f96fc15ae70 2024-01-11T09:43:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.120+0000 7f9701dff700 1 -- 172.21.15.27:0/2014054621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96fc0c93b0 con 0x7f96fc15ae70 2024-01-11T09:43:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.120+0000 7f96e37fe700 1 -- 172.21.15.27:0/2014054621 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f4026820 con 0x7f96fc15ae70 2024-01-11T09:43:38.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.120+0000 7f9701dff700 1 -- 172.21.15.27:0/2014054621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96fc14b450 con 0x7f96fc15ae70 2024-01-11T09:43:38.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.122+0000 7f9701dff700 1 -- 172.21.15.27:0/2014054621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96fc0c9e90 con 0x7f96fc15ae70 2024-01-11T09:43:38.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.122+0000 7f96e37fe700 1 -- 172.21.15.27:0/2014054621 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f96f4003040 con 0x7f96fc15ae70 2024-01-11T09:43:38.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.123+0000 7f96e37fe700 1 --2- 172.21.15.27:0/2014054621 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f96e8066f30 0x7f96e80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:38.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.123+0000 7f96e37fe700 1 -- 172.21.15.27:0/2014054621 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 4937+0+0 (secure 0 0 0) 0x7f96f40861e0 con 0x7f96fc15ae70 2024-01-11T09:43:38.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.124+0000 7f96fb7fe700 1 --2- 172.21.15.27:0/2014054621 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f96e8066f30 0x7f96e80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:38.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.124+0000 7f96fb7fe700 1 --2- 172.21.15.27:0/2014054621 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f96e8066f30 0x7f96e80693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f96ec006fd0 tx=0x7f96ec008040 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:38.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.127+0000 7f96e37fe700 1 -- 172.21.15.27:0/2014054621 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f96fc0c9e90 con 0x7f96fc15ae70 2024-01-11T09:43:38.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.329+0000 7f9701dff700 1 -- 172.21.15.27:0/2014054621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f96fc0c95a0 con 0x7f96fc15ae70 2024-01-11T09:43:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.330+0000 7f96e37fe700 1 -- 172.21.15.27:0/2014054621 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v194) v1 ==== 93+0+10 (secure 0 0 0) 0x7f96f4053750 con 0x7f96fc15ae70 2024-01-11T09:43:38.322 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T09:43:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.332+0000 7f9701dff700 1 -- 172.21.15.27:0/2014054621 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f96e8066f30 msgr2=0x7f96e80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.332+0000 7f9701dff700 1 --2- 172.21.15.27:0/2014054621 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f96e8066f30 0x7f96e80693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f96ec006fd0 tx=0x7f96ec008040 comp rx=0 tx=0).stop 2024-01-11T09:43:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.332+0000 7f9701dff700 1 -- 172.21.15.27:0/2014054621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc15ae70 msgr2=0x7f96fc0b5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.332+0000 7f9701dff700 1 --2- 172.21.15.27:0/2014054621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc15ae70 0x7f96fc0b5a50 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f96f400b6c0 tx=0x7f96f4018000 comp rx=0 tx=0).stop 2024-01-11T09:43:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.332+0000 7f9701dff700 1 -- 172.21.15.27:0/2014054621 shutdown_connections 2024-01-11T09:43:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.332+0000 7f9701dff700 1 --2- 172.21.15.27:0/2014054621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc15ae70 0x7f96fc0b5a50 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.332+0000 7f9701dff700 1 --2- 172.21.15.27:0/2014054621 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96fc0ccd70 0x7f96fc0ca170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.332+0000 7f9701dff700 1 --2- 172.21.15.27:0/2014054621 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f96e8066f30 0x7f96e80693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.332+0000 7f9701dff700 1 --2- 172.21.15.27:0/2014054621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96fc15a1b0 0x7f96fc14af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.332+0000 7f9701dff700 1 -- 172.21.15.27:0/2014054621 >> 172.21.15.27:0/2014054621 conn(0x7f96fc06ed40 msgr2=0x7f96fc156110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.332+0000 7f9701dff700 1 -- 172.21.15.27:0/2014054621 shutdown_connections 2024-01-11T09:43:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.332+0000 7f9701dff700 1 -- 172.21.15.27:0/2014054621 wait complete. 2024-01-11T09:43:38.342 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-11T09:43:38.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.549+0000 7f662c7ff700 1 Processor -- start 2024-01-11T09:43:38.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.549+0000 7f662c7ff700 1 -- start start 2024-01-11T09:43:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.551+0000 7f662c7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66240c8620 0x7f66240af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.551+0000 7f662c7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f66240b09e0 0x7f66240accc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.551+0000 7f662c7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6624157390 con 0x7f6624069d40 2024-01-11T09:43:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.551+0000 7f662c7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66240afeb0 con 0x7f66240b09e0 2024-01-11T09:43:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.551+0000 7f662c7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f662406a200 con 0x7f66240c8620 2024-01-11T09:43:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.551+0000 7f6629d9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6624069d40 0x7f66240b02d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33500/0 (socket says 172.21.15.27:33500) 2024-01-11T09:43:38.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.551+0000 7f6629d9a700 1 -- 172.21.15.27:0/2824854402 learned_addr learned my addr 172.21.15.27:0/2824854402 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:38.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.551+0000 7f662a59b700 1 --2- 172.21.15.27:0/2824854402 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66240c8620 0x7f66240af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:38.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.551+0000 7f662ad9c700 1 --2- 172.21.15.27:0/2824854402 >> v2:172.21.15.77:3300/0 conn(0x7f66240b09e0 0x7f66240accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:38.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.551+0000 7f6629599700 1 -- 172.21.15.27:0/2824854402 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835266551 0 0) 0x7f6624157390 con 0x7f6624069d40 2024-01-11T09:43:38.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.551+0000 7f6629599700 1 -- 172.21.15.27:0/2824854402 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6610003570 con 0x7f6624069d40 2024-01-11T09:43:38.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.552+0000 7f662a59b700 1 -- 172.21.15.27:0/2824854402 >> v2:172.21.15.77:3300/0 conn(0x7f66240b09e0 msgr2=0x7f66240accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:38.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.552+0000 7f662a59b700 1 --2- 172.21.15.27:0/2824854402 >> v2:172.21.15.77:3300/0 conn(0x7f66240b09e0 0x7f66240accc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.552+0000 7f662a59b700 1 -- 172.21.15.27:0/2824854402 >> v1:172.21.15.27:6789/0 conn(0x7f6624069d40 legacy=0x7f66240b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:38.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.552+0000 7f6629599700 1 -- 172.21.15.27:0/2824854402 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 481429521 0 0) 0x7f6610003570 con 0x7f6624069d40 2024-01-11T09:43:38.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.552+0000 7f662a59b700 1 -- 172.21.15.27:0/2824854402 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6624156d40 con 0x7f66240c8620 2024-01-11T09:43:38.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.552+0000 7f662a59b700 1 --2- 172.21.15.27:0/2824854402 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66240c8620 0x7f66240af970 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f66200098e0 tx=0x7f66200091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=882f3ea38ab81cff server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:38.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.552+0000 7f6629599700 1 -- 172.21.15.27:0/2824854402 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6614001470 con 0x7f66240c8620 2024-01-11T09:43:38.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.552+0000 7f6629599700 1 -- 172.21.15.27:0/2824854402 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662000f4a0 con 0x7f66240c8620 2024-01-11T09:43:38.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.552+0000 7f6629599700 1 -- 172.21.15.27:0/2824854402 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662000f840 con 0x7f66240c8620 2024-01-11T09:43:38.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.553+0000 7f662c7ff700 1 -- 172.21.15.27:0/2824854402 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66240c8620 msgr2=0x7f66240af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:38.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.553+0000 7f662c7ff700 1 --2- 172.21.15.27:0/2824854402 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66240c8620 0x7f66240af970 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f66200098e0 tx=0x7f66200091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.553+0000 7f662c7ff700 1 -- 172.21.15.27:0/2824854402 shutdown_connections 2024-01-11T09:43:38.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.553+0000 7f662c7ff700 1 --2- 172.21.15.27:0/2824854402 >> v2:172.21.15.77:3300/0 conn(0x7f66240b09e0 0x7f66240accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.553+0000 7f662c7ff700 1 --2- 172.21.15.27:0/2824854402 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66240c8620 0x7f66240af970 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.553+0000 7f662c7ff700 1 -- 172.21.15.27:0/2824854402 >> 172.21.15.27:0/2824854402 conn(0x7f66240680e0 msgr2=0x7f6624069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:38.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.553+0000 7f662c7ff700 1 -- 172.21.15.27:0/2824854402 shutdown_connections 2024-01-11T09:43:38.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.553+0000 7f662c7ff700 1 -- 172.21.15.27:0/2824854402 wait complete. 2024-01-11T09:43:38.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.554+0000 7f662c7ff700 1 Processor -- start 2024-01-11T09:43:38.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.555+0000 7f662c7ff700 1 -- start start 2024-01-11T09:43:38.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.556+0000 7f662c7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6624069d40 0x7f662413f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:38.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.556+0000 7f662c7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66240b09e0 0x7f66240aa640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:38.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.556+0000 7f662c7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66240c8620 0x7f66240cc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:38.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.556+0000 7f662c7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662415ee90 con 0x7f66240b09e0 2024-01-11T09:43:38.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.556+0000 7f662c7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66240ab5c0 con 0x7f66240c8620 2024-01-11T09:43:38.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.556+0000 7f662c7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66240caad0 con 0x7f6624069d40 2024-01-11T09:43:38.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.556+0000 7f6629d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66240b09e0 0x7f66240aa640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:38.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.556+0000 7f6629d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66240b09e0 0x7f66240aa640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41076/0 (socket says 172.21.15.27:41076) 2024-01-11T09:43:38.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.556+0000 7f6629d9a700 1 -- 172.21.15.27:0/2677576253 learned_addr learned my addr 172.21.15.27:0/2677576253 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:38.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.556+0000 7f662a59b700 1 --2- 172.21.15.27:0/2677576253 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6624069d40 0x7f662413f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:38.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.556+0000 7f662ad9c700 1 --2- 172.21.15.27:0/2677576253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66240c8620 0x7f66240cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:38.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.556+0000 7f6629d9a700 1 -- 172.21.15.27:0/2677576253 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6624069d40 msgr2=0x7f662413f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:38.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.556+0000 7f6629d9a700 1 --2- 172.21.15.27:0/2677576253 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6624069d40 0x7f662413f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.556+0000 7f6629d9a700 1 -- 172.21.15.27:0/2677576253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66240c8620 msgr2=0x7f66240cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:38.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.556+0000 7f6629d9a700 1 --2- 172.21.15.27:0/2677576253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66240c8620 0x7f66240cc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.556+0000 7f6629d9a700 1 -- 172.21.15.27:0/2677576253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6620009420 con 0x7f66240b09e0 2024-01-11T09:43:38.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.557+0000 7f6629d9a700 1 --2- 172.21.15.27:0/2677576253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66240b09e0 0x7f66240aa640 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f661400c990 tx=0x7f66140019d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:38.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.557+0000 7f661a7fc700 1 -- 172.21.15.27:0/2677576253 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6614005b30 con 0x7f66240b09e0 2024-01-11T09:43:38.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.557+0000 7f661a7fc700 1 -- 172.21.15.27:0/2677576253 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6614005c90 con 0x7f66240b09e0 2024-01-11T09:43:38.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.557+0000 7f662c7ff700 1 -- 172.21.15.27:0/2677576253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66240ab000 con 0x7f66240b09e0 2024-01-11T09:43:38.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.557+0000 7f661a7fc700 1 -- 172.21.15.27:0/2677576253 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6614007320 con 0x7f66240b09e0 2024-01-11T09:43:38.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.557+0000 7f662c7ff700 1 -- 172.21.15.27:0/2677576253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66240ca760 con 0x7f66240b09e0 2024-01-11T09:43:38.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.558+0000 7f662c7ff700 1 -- 172.21.15.27:0/2677576253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66240718d0 con 0x7f66240b09e0 2024-01-11T09:43:38.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.559+0000 7f661a7fc700 1 -- 172.21.15.27:0/2677576253 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f661401b400 con 0x7f66240b09e0 2024-01-11T09:43:38.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.560+0000 7f661a7fc700 1 --2- 172.21.15.27:0/2677576253 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6608066f30 0x7f66080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:38.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.560+0000 7f661a7fc700 1 -- 172.21.15.27:0/2677576253 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 4937+0+0 (secure 0 0 0) 0x7f6614083ff0 con 0x7f66240b09e0 2024-01-11T09:43:38.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.560+0000 7f662a59b700 1 --2- 172.21.15.27:0/2677576253 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6608066f30 0x7f66080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:38.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.561+0000 7f662a59b700 1 --2- 172.21.15.27:0/2677576253 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6608066f30 0x7f66080693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f66200091e0 tx=0x7f6620018000 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:38.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.564+0000 7f661a7fc700 1 -- 172.21.15.27:0/2677576253 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6614051520 con 0x7f66240b09e0 2024-01-11T09:43:38.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.767+0000 7f662c7ff700 1 -- 172.21.15.27:0/2677576253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f66240cac10 con 0x7f66240b09e0 2024-01-11T09:43:38.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.767+0000 7f661a7fc700 1 -- 172.21.15.27:0/2677576253 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v194) v1 ==== 84+0+65 (secure 0 0 0) 0x7f661405a030 con 0x7f66240b09e0 2024-01-11T09:43:38.759 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:43:38.759 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T09:43:38.759 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:38.760 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-11T09:43:38.760 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-11T09:43:38.760 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:38.760 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T09:43:38.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.769+0000 7f662c7ff700 1 -- 172.21.15.27:0/2677576253 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6608066f30 msgr2=0x7f66080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:38.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.769+0000 7f662c7ff700 1 --2- 172.21.15.27:0/2677576253 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6608066f30 0x7f66080693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f66200091e0 tx=0x7f6620018000 comp rx=0 tx=0).stop 2024-01-11T09:43:38.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.769+0000 7f662c7ff700 1 -- 172.21.15.27:0/2677576253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66240b09e0 msgr2=0x7f66240aa640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:38.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.769+0000 7f662c7ff700 1 --2- 172.21.15.27:0/2677576253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66240b09e0 0x7f66240aa640 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f661400c990 tx=0x7f66140019d0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.769+0000 7f662c7ff700 1 -- 172.21.15.27:0/2677576253 shutdown_connections 2024-01-11T09:43:38.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.769+0000 7f662c7ff700 1 --2- 172.21.15.27:0/2677576253 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6608066f30 0x7f66080693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.769+0000 7f662c7ff700 1 --2- 172.21.15.27:0/2677576253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66240c8620 0x7f66240cc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.769+0000 7f662c7ff700 1 --2- 172.21.15.27:0/2677576253 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6624069d40 0x7f662413f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.769+0000 7f662c7ff700 1 --2- 172.21.15.27:0/2677576253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66240b09e0 0x7f66240aa640 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.769+0000 7f662c7ff700 1 -- 172.21.15.27:0/2677576253 >> 172.21.15.27:0/2677576253 conn(0x7f66240680e0 msgr2=0x7f6624154b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:38.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.770+0000 7f662c7ff700 1 -- 172.21.15.27:0/2677576253 shutdown_connections 2024-01-11T09:43:38.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.770+0000 7f662c7ff700 1 -- 172.21.15.27:0/2677576253 wait complete. 2024-01-11T09:43:38.779 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-11T09:43:38.779 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-11T09:43:38.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.990+0000 7fd3941ff700 1 Processor -- start 2024-01-11T09:43:38.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.990+0000 7fd3941ff700 1 -- start start 2024-01-11T09:43:38.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.991+0000 7fd3941ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd38c159e60 0x7fd38c15aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:38.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.991+0000 7fd3941ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd38c15b400 0x7fd38c0685e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:38.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.991+0000 7fd3941ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c15c4b0 con 0x7fd38c15a7a0 2024-01-11T09:43:38.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.991+0000 7fd3941ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd38c15d050 con 0x7fd38c15b400 2024-01-11T09:43:38.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.991+0000 7fd3941ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c15b8a0 con 0x7fd38c159e60 2024-01-11T09:43:38.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.992+0000 7fd39179a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd38c159e60 0x7fd38c15aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:38.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.992+0000 7fd39279c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd38c15b400 0x7fd38c0685e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:38.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.992+0000 7fd391f9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd38c15a7a0 0x7fd38c15c940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:46088/0 (socket says 172.21.15.27:46088) 2024-01-11T09:43:38.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.992+0000 7fd391f9b700 1 -- 172.21.15.27:0/3644350511 learned_addr learned my addr 172.21.15.27:0/3644350511 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:38.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.992+0000 7fd39179a700 1 --2- 172.21.15.27:0/3644350511 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd38c159e60 0x7fd38c15aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:58984/0 (socket says 172.21.15.27:58984) 2024-01-11T09:43:38.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.992+0000 7fd39279c700 1 -- 172.21.15.27:0/3644350511 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd38c159e60 msgr2=0x7fd38c15aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:38.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.992+0000 7fd39279c700 1 --2- 172.21.15.27:0/3644350511 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd38c159e60 0x7fd38c15aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.992+0000 7fd39279c700 1 -- 172.21.15.27:0/3644350511 >> v1:172.21.15.77:6789/0 conn(0x7fd38c15a7a0 legacy=0x7fd38c15c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:38.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.992+0000 7fd39279c700 1 -- 172.21.15.27:0/3644350511 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd38c15bb80 con 0x7fd38c15b400 2024-01-11T09:43:38.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.992+0000 7fd39179a700 1 --2- 172.21.15.27:0/3644350511 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd38c159e60 0x7fd38c15aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:43:38.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.992+0000 7fd390f99700 1 -- 172.21.15.27:0/3644350511 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906076847 0 0) 0x7fd38c15c4b0 con 0x7fd38c15a7a0 2024-01-11T09:43:38.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.993+0000 7fd39279c700 1 --2- 172.21.15.27:0/3644350511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd38c15b400 0x7fd38c0685e0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fd384009e60 tx=0x7fd384009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a86edf976c79e241 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:38.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.993+0000 7fd390f99700 1 -- 172.21.15.27:0/3644350511 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38400e070 con 0x7fd38c15b400 2024-01-11T09:43:38.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.993+0000 7fd390f99700 1 -- 172.21.15.27:0/3644350511 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd38400bb60 con 0x7fd38c15b400 2024-01-11T09:43:38.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.993+0000 7fd390f99700 1 -- 172.21.15.27:0/3644350511 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd384012430 con 0x7fd38c15b400 2024-01-11T09:43:38.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.993+0000 7fd3941ff700 1 -- 172.21.15.27:0/3644350511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd38c15b400 msgr2=0x7fd38c0685e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:38.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.993+0000 7fd3941ff700 1 --2- 172.21.15.27:0/3644350511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd38c15b400 0x7fd38c0685e0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fd384009e60 tx=0x7fd384009780 comp rx=0 tx=0).stop 2024-01-11T09:43:38.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.994+0000 7fd3941ff700 1 -- 172.21.15.27:0/3644350511 shutdown_connections 2024-01-11T09:43:38.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.994+0000 7fd3941ff700 1 --2- 172.21.15.27:0/3644350511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd38c15b400 0x7fd38c0685e0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.994+0000 7fd3941ff700 1 --2- 172.21.15.27:0/3644350511 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd38c159e60 0x7fd38c15aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.994+0000 7fd3941ff700 1 -- 172.21.15.27:0/3644350511 >> 172.21.15.27:0/3644350511 conn(0x7fd38c06ed40 msgr2=0x7fd38c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:38.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.994+0000 7fd3941ff700 1 -- 172.21.15.27:0/3644350511 shutdown_connections 2024-01-11T09:43:38.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.994+0000 7fd3941ff700 1 -- 172.21.15.27:0/3644350511 wait complete. 2024-01-11T09:43:38.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.995+0000 7fd3941ff700 1 Processor -- start 2024-01-11T09:43:38.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.995+0000 7fd3941ff700 1 -- start start 2024-01-11T09:43:38.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.996+0000 7fd3941ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd38c15a7a0 0x7fd38c13eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:38.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.996+0000 7fd3941ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd38c15b400 0x7fd38c14d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:38.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.996+0000 7fd3941ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd38c14d850 0x7fd38c0b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:38.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.996+0000 7fd3941ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c14dcd0 con 0x7fd38c15b400 2024-01-11T09:43:38.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.996+0000 7fd3941ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c0b0e60 con 0x7fd38c14d850 2024-01-11T09:43:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.996+0000 7fd3941ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c0aff70 con 0x7fd38c15a7a0 2024-01-11T09:43:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.997+0000 7fd39179a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd38c15b400 0x7fd38c14d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.997+0000 7fd39179a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd38c15b400 0x7fd38c14d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41086/0 (socket says 172.21.15.27:41086) 2024-01-11T09:43:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.997+0000 7fd39179a700 1 -- 172.21.15.27:0/3697342261 learned_addr learned my addr 172.21.15.27:0/3697342261 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.997+0000 7fd391f9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd38c15a7a0 0x7fd38c13eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.997+0000 7fd39279c700 1 --2- 172.21.15.27:0/3697342261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd38c14d850 0x7fd38c0b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.997+0000 7fd39179a700 1 -- 172.21.15.27:0/3697342261 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd38c15a7a0 msgr2=0x7fd38c13eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.997+0000 7fd39179a700 1 --2- 172.21.15.27:0/3697342261 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd38c15a7a0 0x7fd38c13eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.997+0000 7fd39179a700 1 -- 172.21.15.27:0/3697342261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd38c14d850 msgr2=0x7fd38c0b2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.997+0000 7fd39179a700 1 --2- 172.21.15.27:0/3697342261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd38c14d850 0x7fd38c0b2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.997+0000 7fd39179a700 1 -- 172.21.15.27:0/3697342261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37c009420 con 0x7fd38c15b400 2024-01-11T09:43:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.997+0000 7fd391f9b700 1 --2- 172.21.15.27:0/3697342261 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd38c15a7a0 0x7fd38c13eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:43:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.997+0000 7fd39279c700 1 --2- 172.21.15.27:0/3697342261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd38c14d850 0x7fd38c0b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.997+0000 7fd39179a700 1 --2- 172.21.15.27:0/3697342261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd38c15b400 0x7fd38c14d310 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fd37c00ecb0 tx=0x7fd37c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.998+0000 7fd381ffb700 1 -- 172.21.15.27:0/3697342261 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd37c00c920 con 0x7fd38c15b400 2024-01-11T09:43:38.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.998+0000 7fd381ffb700 1 -- 172.21.15.27:0/3697342261 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd37c00ca80 con 0x7fd38c15b400 2024-01-11T09:43:38.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.998+0000 7fd3941ff700 1 -- 172.21.15.27:0/3697342261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd384009a20 con 0x7fd38c15b400 2024-01-11T09:43:38.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.998+0000 7fd381ffb700 1 -- 172.21.15.27:0/3697342261 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd37c00ce00 con 0x7fd38c15b400 2024-01-11T09:43:38.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.998+0000 7fd3941ff700 1 -- 172.21.15.27:0/3697342261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd38c0b05b0 con 0x7fd38c15b400 2024-01-11T09:43:38.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:38.999+0000 7fd3941ff700 1 -- 172.21.15.27:0/3697342261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd38c04c3c0 con 0x7fd38c15b400 2024-01-11T09:43:38.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:39.000+0000 7fd381ffb700 1 -- 172.21.15.27:0/3697342261 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd37c013680 con 0x7fd38c15b400 2024-01-11T09:43:38.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:39.001+0000 7fd381ffb700 1 --2- 172.21.15.27:0/3697342261 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd370066f30 0x7fd3700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:38.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:39.001+0000 7fd381ffb700 1 -- 172.21.15.27:0/3697342261 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 4937+0+0 (secure 0 0 0) 0x7fd37c084600 con 0x7fd38c15b400 2024-01-11T09:43:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:39.001+0000 7fd391f9b700 1 --2- 172.21.15.27:0/3697342261 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd370066f30 0x7fd3700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:39.002+0000 7fd391f9b700 1 --2- 172.21.15.27:0/3697342261 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd370066f30 0x7fd3700693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd388007fd0 tx=0x7fd388007d30 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:38.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:39.005+0000 7fd381ffb700 1 -- 172.21.15.27:0/3697342261 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd37c051ad0 con 0x7fd38c15b400 2024-01-11T09:43:39.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:39.211+0000 7fd3941ff700 1 -- 172.21.15.27:0/3697342261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd38c0b0fa0 con 0x7fd38c15b400 2024-01-11T09:43:40.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.045+0000 7fd381ffb700 1 -- 172.21.15.27:0/3697342261 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v195) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd37c0550f0 con 0x7fd38c15b400 2024-01-11T09:43:40.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.050+0000 7fd3941ff700 1 -- 172.21.15.27:0/3697342261 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd370066f30 msgr2=0x7fd3700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:40.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.050+0000 7fd3941ff700 1 --2- 172.21.15.27:0/3697342261 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd370066f30 0x7fd3700693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd388007fd0 tx=0x7fd388007d30 comp rx=0 tx=0).stop 2024-01-11T09:43:40.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.050+0000 7fd3941ff700 1 -- 172.21.15.27:0/3697342261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd38c15b400 msgr2=0x7fd38c14d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:40.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.050+0000 7fd3941ff700 1 --2- 172.21.15.27:0/3697342261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd38c15b400 0x7fd38c14d310 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fd37c00ecb0 tx=0x7fd37c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:40.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.051+0000 7fd3941ff700 1 -- 172.21.15.27:0/3697342261 shutdown_connections 2024-01-11T09:43:40.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.051+0000 7fd3941ff700 1 --2- 172.21.15.27:0/3697342261 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd370066f30 0x7fd3700693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:40.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.051+0000 7fd3941ff700 1 --2- 172.21.15.27:0/3697342261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd38c14d850 0x7fd38c0b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:40.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.051+0000 7fd3941ff700 1 --2- 172.21.15.27:0/3697342261 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd38c15a7a0 0x7fd38c13eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:40.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.051+0000 7fd3941ff700 1 --2- 172.21.15.27:0/3697342261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd38c15b400 0x7fd38c14d310 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:40.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.051+0000 7fd3941ff700 1 -- 172.21.15.27:0/3697342261 >> 172.21.15.27:0/3697342261 conn(0x7fd38c06ed40 msgr2=0x7fd38c15d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:40.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.051+0000 7fd3941ff700 1 -- 172.21.15.27:0/3697342261 shutdown_connections 2024-01-11T09:43:40.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.052+0000 7fd3941ff700 1 -- 172.21.15.27:0/3697342261 wait complete. 2024-01-11T09:43:40.046 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-11T09:43:40.066 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-11T09:43:40.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.285+0000 7f159cdff700 1 Processor -- start 2024-01-11T09:43:40.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.286+0000 7f159cdff700 1 -- start start 2024-01-11T09:43:40.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.287+0000 7f159cdff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1598153480 0x7f15980b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:40.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.287+0000 7f159cdff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1598153850 0x7f1598159730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:40.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.287+0000 7f159cdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1598159c70 0x7f15981561a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:40.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.287+0000 7f159cdff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1598153d70 con 0x7f1598153850 2024-01-11T09:43:40.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.287+0000 7f159cdff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f159815a0f0 con 0x7f1598153480 2024-01-11T09:43:40.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.287+0000 7f159cdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1598157ed0 con 0x7f1598159c70 2024-01-11T09:43:40.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.287+0000 7f1595d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1598153850 0x7f1598159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:40.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.287+0000 7f1595d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1598153850 0x7f1598159730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41092/0 (socket says 172.21.15.27:41092) 2024-01-11T09:43:40.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.287+0000 7f1595d9b700 1 -- 172.21.15.27:0/2996888420 learned_addr learned my addr 172.21.15.27:0/2996888420 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:40.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.288+0000 7f1596d9d700 1 --2- 172.21.15.27:0/2996888420 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1598159c70 0x7f15981561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:40.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.288+0000 7f159659c700 1 --2- 172.21.15.27:0/2996888420 >> v2:172.21.15.77:3300/0 conn(0x7f1598153480 0x7f15980b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:40.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.288+0000 7f1595d9b700 1 -- 172.21.15.27:0/2996888420 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1598159c70 msgr2=0x7f15981561a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:40.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.288+0000 7f1595d9b700 1 --2- 172.21.15.27:0/2996888420 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1598159c70 0x7f15981561a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:40.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.288+0000 7f1595d9b700 1 -- 172.21.15.27:0/2996888420 >> v2:172.21.15.77:3300/0 conn(0x7f1598153480 msgr2=0x7f15980b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:40.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.288+0000 7f1595d9b700 1 --2- 172.21.15.27:0/2996888420 >> v2:172.21.15.77:3300/0 conn(0x7f1598153480 0x7f15980b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:40.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.288+0000 7f1595d9b700 1 -- 172.21.15.27:0/2996888420 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1598157300 con 0x7f1598153850 2024-01-11T09:43:40.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.288+0000 7f1595d9b700 1 --2- 172.21.15.27:0/2996888420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1598153850 0x7f1598159730 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f159806d000 tx=0x7f158c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21bb05f3f41bf8c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:40.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.288+0000 7f159559a700 1 -- 172.21.15.27:0/2996888420 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c00b7e0 con 0x7f1598153850 2024-01-11T09:43:40.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.289+0000 7f159559a700 1 -- 172.21.15.27:0/2996888420 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158c00b940 con 0x7f1598153850 2024-01-11T09:43:40.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.289+0000 7f159559a700 1 -- 172.21.15.27:0/2996888420 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c00bce0 con 0x7f1598153850 2024-01-11T09:43:40.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.289+0000 7f159cdff700 1 -- 172.21.15.27:0/2996888420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1598153850 msgr2=0x7f1598159730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:40.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.289+0000 7f159cdff700 1 --2- 172.21.15.27:0/2996888420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1598153850 0x7f1598159730 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f159806d000 tx=0x7f158c009250 comp rx=0 tx=0).stop 2024-01-11T09:43:40.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.289+0000 7f159cdff700 1 -- 172.21.15.27:0/2996888420 shutdown_connections 2024-01-11T09:43:40.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.289+0000 7f159cdff700 1 --2- 172.21.15.27:0/2996888420 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1598159c70 0x7f15981561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:40.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.289+0000 7f159cdff700 1 --2- 172.21.15.27:0/2996888420 >> v2:172.21.15.77:3300/0 conn(0x7f1598153480 0x7f15980b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:40.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.289+0000 7f159cdff700 1 --2- 172.21.15.27:0/2996888420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1598153850 0x7f1598159730 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:40.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.289+0000 7f159cdff700 1 -- 172.21.15.27:0/2996888420 >> 172.21.15.27:0/2996888420 conn(0x7f15980680a0 msgr2=0x7f1598069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:40.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.289+0000 7f159cdff700 1 -- 172.21.15.27:0/2996888420 shutdown_connections 2024-01-11T09:43:40.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.289+0000 7f159cdff700 1 -- 172.21.15.27:0/2996888420 wait complete. 2024-01-11T09:43:40.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.291+0000 7f159cdff700 1 Processor -- start 2024-01-11T09:43:40.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.291+0000 7f159cdff700 1 -- start start 2024-01-11T09:43:40.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.292+0000 7f159cdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1598153480 0x7f15980caf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:40.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.292+0000 7f159cdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1598153850 0x7f15980cb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:40.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.292+0000 7f159cdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1598159c70 0x7f1598150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:40.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.292+0000 7f159cdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15980cba90 con 0x7f1598153850 2024-01-11T09:43:40.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.292+0000 7f159cdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15980cbcc0 con 0x7f1598153480 2024-01-11T09:43:40.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.292+0000 7f159cdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1598154470 con 0x7f1598159c70 2024-01-11T09:43:40.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.292+0000 7f1595d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1598153850 0x7f15980cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.292+0000 7f1595d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1598153850 0x7f15980cb550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41108/0 (socket says 172.21.15.27:41108) 2024-01-11T09:43:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.292+0000 7f1595d9b700 1 -- 172.21.15.27:0/3956919499 learned_addr learned my addr 172.21.15.27:0/3956919499 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.292+0000 7f1596d9d700 1 --2- 172.21.15.27:0/3956919499 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1598159c70 0x7f1598150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.292+0000 7f159659c700 1 --2- 172.21.15.27:0/3956919499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1598153480 0x7f15980caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.292+0000 7f1595d9b700 1 -- 172.21.15.27:0/3956919499 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1598159c70 msgr2=0x7f1598150bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.292+0000 7f1595d9b700 1 --2- 172.21.15.27:0/3956919499 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1598159c70 0x7f1598150bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.292+0000 7f1595d9b700 1 -- 172.21.15.27:0/3956919499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1598153480 msgr2=0x7f15980caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.293+0000 7f1595d9b700 1 --2- 172.21.15.27:0/3956919499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1598153480 0x7f15980caf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.293+0000 7f1595d9b700 1 -- 172.21.15.27:0/3956919499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158c0094f0 con 0x7f1598153850 2024-01-11T09:43:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.293+0000 7f1595d9b700 1 --2- 172.21.15.27:0/3956919499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1598153850 0x7f15980cb550 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f158c009290 tx=0x7f158c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.293+0000 7f15867fc700 1 -- 172.21.15.27:0/3956919499 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c024440 con 0x7f1598153850 2024-01-11T09:43:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.293+0000 7f159cdff700 1 -- 172.21.15.27:0/3956919499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1598154710 con 0x7f1598153850 2024-01-11T09:43:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.293+0000 7f15867fc700 1 -- 172.21.15.27:0/3956919499 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158c0245c0 con 0x7f1598153850 2024-01-11T09:43:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.293+0000 7f15867fc700 1 -- 172.21.15.27:0/3956919499 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c024980 con 0x7f1598153850 2024-01-11T09:43:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.293+0000 7f159cdff700 1 -- 172.21.15.27:0/3956919499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1598154cd0 con 0x7f1598153850 2024-01-11T09:43:40.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.294+0000 7f159cdff700 1 -- 172.21.15.27:0/3956919499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1598154ef0 con 0x7f1598153850 2024-01-11T09:43:40.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.295+0000 7f15867fc700 1 -- 172.21.15.27:0/3956919499 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f158c01a420 con 0x7f1598153850 2024-01-11T09:43:40.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.297+0000 7f15867fc700 1 --2- 172.21.15.27:0/3956919499 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1578066f30 0x7f15780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:40.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.297+0000 7f15867fc700 1 -- 172.21.15.27:0/3956919499 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5307+0+0 (secure 0 0 0) 0x7f158c086310 con 0x7f1598153850 2024-01-11T09:43:40.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.297+0000 7f159659c700 1 --2- 172.21.15.27:0/3956919499 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1578066f30 0x7f15780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:40.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.298+0000 7f159659c700 1 --2- 172.21.15.27:0/3956919499 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1578066f30 0x7f15780693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1588006fd0 tx=0x7f1588008040 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.301+0000 7f15867fc700 1 -- 172.21.15.27:0/3956919499 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f158c053670 con 0x7f1598153850 2024-01-11T09:43:40.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:40.507+0000 7f159cdff700 1 -- 172.21.15.27:0/3956919499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f159804c3c0 con 0x7f1598153850 2024-01-11T09:43:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.048+0000 7f15867fc700 1 -- 172.21.15.27:0/3956919499 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v196) v1 ==== 170+0+0 (secure 0 0 0) 0x7f158c056c90 con 0x7f1598153850 2024-01-11T09:43:41.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.055+0000 7f159cdff700 1 -- 172.21.15.27:0/3956919499 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1578066f30 msgr2=0x7f15780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:41.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.055+0000 7f159cdff700 1 --2- 172.21.15.27:0/3956919499 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1578066f30 0x7f15780693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1588006fd0 tx=0x7f1588008040 comp rx=0 tx=0).stop 2024-01-11T09:43:41.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.055+0000 7f159cdff700 1 -- 172.21.15.27:0/3956919499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1598153850 msgr2=0x7f15980cb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:41.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.055+0000 7f159cdff700 1 --2- 172.21.15.27:0/3956919499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1598153850 0x7f15980cb550 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f158c009290 tx=0x7f158c017000 comp rx=0 tx=0).stop 2024-01-11T09:43:41.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.055+0000 7f159cdff700 1 -- 172.21.15.27:0/3956919499 shutdown_connections 2024-01-11T09:43:41.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.055+0000 7f159cdff700 1 --2- 172.21.15.27:0/3956919499 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1598159c70 0x7f1598150bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:41.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.055+0000 7f159cdff700 1 --2- 172.21.15.27:0/3956919499 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1578066f30 0x7f15780693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:41.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.055+0000 7f159cdff700 1 --2- 172.21.15.27:0/3956919499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1598153480 0x7f15980caf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:41.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.055+0000 7f159cdff700 1 --2- 172.21.15.27:0/3956919499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1598153850 0x7f15980cb550 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:41.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.055+0000 7f159cdff700 1 -- 172.21.15.27:0/3956919499 >> 172.21.15.27:0/3956919499 conn(0x7f15980680a0 msgr2=0x7f159806e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:41.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.057+0000 7f159cdff700 1 -- 172.21.15.27:0/3956919499 shutdown_connections 2024-01-11T09:43:41.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.058+0000 7f159cdff700 1 -- 172.21.15.27:0/3956919499 wait complete. 2024-01-11T09:43:41.057 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-11T09:43:41.081 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-11T09:43:41.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.291+0000 7fc944dff700 1 Processor -- start 2024-01-11T09:43:41.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.292+0000 7fc944dff700 1 -- start start 2024-01-11T09:43:41.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.293+0000 7fc944dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9400ccd70 0x7fc9400cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.293+0000 7fc944dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc94015ae70 0x7fc94015b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.293+0000 7fc944dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9400cf850 con 0x7fc94015a1b0 2024-01-11T09:43:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.293+0000 7fc944dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc94015ac90 con 0x7fc9400ccd70 2024-01-11T09:43:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.293+0000 7fc944dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9400cec40 con 0x7fc94015ae70 2024-01-11T09:43:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.293+0000 7fc93dd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9400ccd70 0x7fc9400cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.293+0000 7fc93dd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9400ccd70 0x7fc9400cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41114/0 (socket says 172.21.15.27:41114) 2024-01-11T09:43:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.293+0000 7fc93dd9b700 1 -- 172.21.15.27:0/382226643 learned_addr learned my addr 172.21.15.27:0/382226643 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.293+0000 7fc93ed9d700 1 --2- 172.21.15.27:0/382226643 >> v2:172.21.15.77:3300/0 conn(0x7fc94015ae70 0x7fc94015b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.294+0000 7fc93dd9b700 1 -- 172.21.15.27:0/382226643 >> v2:172.21.15.77:3300/0 conn(0x7fc94015ae70 msgr2=0x7fc94015b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.294+0000 7fc93dd9b700 1 --2- 172.21.15.27:0/382226643 >> v2:172.21.15.77:3300/0 conn(0x7fc94015ae70 0x7fc94015b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.294+0000 7fc93dd9b700 1 -- 172.21.15.27:0/382226643 >> v1:172.21.15.27:6789/0 conn(0x7fc94015a1b0 legacy=0x7fc94015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.294+0000 7fc93dd9b700 1 -- 172.21.15.27:0/382226643 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9400cef20 con 0x7fc9400ccd70 2024-01-11T09:43:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.294+0000 7fc93dd9b700 1 --2- 172.21.15.27:0/382226643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9400ccd70 0x7fc9400cd210 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fc940089810 tx=0x7fc934009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=118c2b68ccbb9b01 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.294+0000 7fc93d59a700 1 -- 172.21.15.27:0/382226643 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc934009e70 con 0x7fc9400ccd70 2024-01-11T09:43:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.294+0000 7fc93d59a700 1 -- 172.21.15.27:0/382226643 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93400b860 con 0x7fc9400ccd70 2024-01-11T09:43:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.294+0000 7fc93d59a700 1 -- 172.21.15.27:0/382226643 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93400bc00 con 0x7fc9400ccd70 2024-01-11T09:43:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.295+0000 7fc944dff700 1 -- 172.21.15.27:0/382226643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9400ccd70 msgr2=0x7fc9400cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.295+0000 7fc944dff700 1 --2- 172.21.15.27:0/382226643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9400ccd70 0x7fc9400cd210 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fc940089810 tx=0x7fc934009250 comp rx=0 tx=0).stop 2024-01-11T09:43:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.295+0000 7fc944dff700 1 -- 172.21.15.27:0/382226643 shutdown_connections 2024-01-11T09:43:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.295+0000 7fc944dff700 1 --2- 172.21.15.27:0/382226643 >> v2:172.21.15.77:3300/0 conn(0x7fc94015ae70 0x7fc94015b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.295+0000 7fc944dff700 1 --2- 172.21.15.27:0/382226643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9400ccd70 0x7fc9400cd210 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:41.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.295+0000 7fc944dff700 1 -- 172.21.15.27:0/382226643 >> 172.21.15.27:0/382226643 conn(0x7fc94006ed40 msgr2=0x7fc94006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:41.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.295+0000 7fc944dff700 1 -- 172.21.15.27:0/382226643 shutdown_connections 2024-01-11T09:43:41.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.295+0000 7fc944dff700 1 -- 172.21.15.27:0/382226643 wait complete. 2024-01-11T09:43:41.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.296+0000 7fc944dff700 1 Processor -- start 2024-01-11T09:43:41.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.297+0000 7fc944dff700 1 -- start start 2024-01-11T09:43:41.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.298+0000 7fc944dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9400ccd70 0x7fc9400af910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:41.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.298+0000 7fc944dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc94015a1b0 0x7fc9400b2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:41.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.298+0000 7fc944dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94015ae70 0x7fc9400b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:41.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.298+0000 7fc944dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94016ab40 con 0x7fc94015ae70 2024-01-11T09:43:41.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.298+0000 7fc944dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94016b4e0 con 0x7fc94015a1b0 2024-01-11T09:43:41.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.298+0000 7fc944dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9400afe50 con 0x7fc9400ccd70 2024-01-11T09:43:41.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.298+0000 7fc93ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94015ae70 0x7fc9400b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:41.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.298+0000 7fc93ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94015ae70 0x7fc9400b27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41128/0 (socket says 172.21.15.27:41128) 2024-01-11T09:43:41.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.298+0000 7fc93ed9d700 1 -- 172.21.15.27:0/706017624 learned_addr learned my addr 172.21.15.27:0/706017624 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:41.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.298+0000 7fc93e59c700 1 --2- 172.21.15.27:0/706017624 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9400ccd70 0x7fc9400af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:41.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.298+0000 7fc93dd9b700 1 --2- 172.21.15.27:0/706017624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc94015a1b0 0x7fc9400b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:41.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.298+0000 7fc93ed9d700 1 -- 172.21.15.27:0/706017624 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9400ccd70 msgr2=0x7fc9400af910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.298+0000 7fc93ed9d700 1 --2- 172.21.15.27:0/706017624 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9400ccd70 0x7fc9400af910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.298+0000 7fc93ed9d700 1 -- 172.21.15.27:0/706017624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc94015a1b0 msgr2=0x7fc9400b2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.298+0000 7fc93ed9d700 1 --2- 172.21.15.27:0/706017624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc94015a1b0 0x7fc9400b2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.298+0000 7fc93ed9d700 1 -- 172.21.15.27:0/706017624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9340094f0 con 0x7fc94015ae70 2024-01-11T09:43:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.299+0000 7fc93ed9d700 1 --2- 172.21.15.27:0/706017624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94015ae70 0x7fc9400b27d0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fc938009e90 tx=0x7fc93800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.299+0000 7fc92e7fc700 1 -- 172.21.15.27:0/706017624 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93800cb70 con 0x7fc94015ae70 2024-01-11T09:43:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.299+0000 7fc92e7fc700 1 -- 172.21.15.27:0/706017624 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93800f040 con 0x7fc94015ae70 2024-01-11T09:43:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.299+0000 7fc92e7fc700 1 -- 172.21.15.27:0/706017624 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc938014480 con 0x7fc94015ae70 2024-01-11T09:43:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.299+0000 7fc944dff700 1 -- 172.21.15.27:0/706017624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc94016a860 con 0x7fc94015ae70 2024-01-11T09:43:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.299+0000 7fc944dff700 1 -- 172.21.15.27:0/706017624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94016b170 con 0x7fc94015ae70 2024-01-11T09:43:41.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.300+0000 7fc944dff700 1 -- 172.21.15.27:0/706017624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc94016acf0 con 0x7fc94015ae70 2024-01-11T09:43:41.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.301+0000 7fc92e7fc700 1 -- 172.21.15.27:0/706017624 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc938014830 con 0x7fc94015ae70 2024-01-11T09:43:41.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.302+0000 7fc92e7fc700 1 --2- 172.21.15.27:0/706017624 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc920066f30 0x7fc9200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:41.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.303+0000 7fc92e7fc700 1 -- 172.21.15.27:0/706017624 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5689+0+0 (secure 0 0 0) 0x7fc938084f90 con 0x7fc94015ae70 2024-01-11T09:43:41.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.303+0000 7fc93e59c700 1 --2- 172.21.15.27:0/706017624 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc920066f30 0x7fc9200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:41.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.304+0000 7fc93e59c700 1 --2- 172.21.15.27:0/706017624 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc920066f30 0x7fc9200693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc9300039f0 tx=0x7fc930007ec0 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:41.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.307+0000 7fc92e7fc700 1 -- 172.21.15.27:0/706017624 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc94016acf0 con 0x7fc94015ae70 2024-01-11T09:43:41.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:41.552+0000 7fc944dff700 1 -- 172.21.15.27:0/706017624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc94016acf0 con 0x7fc94015ae70 2024-01-11T09:43:42.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.060+0000 7fc92e7fc700 1 -- 172.21.15.27:0/706017624 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 25 v98) v1 ==== 174+0+0 (secure 0 0 0) 0x7fc94016acf0 con 0x7fc94015ae70 2024-01-11T09:43:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.071+0000 7fc944dff700 1 -- 172.21.15.27:0/706017624 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc920066f30 msgr2=0x7fc9200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.071+0000 7fc944dff700 1 --2- 172.21.15.27:0/706017624 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc920066f30 0x7fc9200693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc9300039f0 tx=0x7fc930007ec0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.071+0000 7fc944dff700 1 -- 172.21.15.27:0/706017624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94015ae70 msgr2=0x7fc9400b27d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.071+0000 7fc944dff700 1 --2- 172.21.15.27:0/706017624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94015ae70 0x7fc9400b27d0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fc938009e90 tx=0x7fc93800b040 comp rx=0 tx=0).stop 2024-01-11T09:43:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.072+0000 7fc944dff700 1 -- 172.21.15.27:0/706017624 shutdown_connections 2024-01-11T09:43:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.072+0000 7fc944dff700 1 --2- 172.21.15.27:0/706017624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94015ae70 0x7fc9400b27d0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.072+0000 7fc944dff700 1 --2- 172.21.15.27:0/706017624 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9400ccd70 0x7fc9400af910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.072+0000 7fc944dff700 1 --2- 172.21.15.27:0/706017624 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc920066f30 0x7fc9200693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.072+0000 7fc944dff700 1 --2- 172.21.15.27:0/706017624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc94015a1b0 0x7fc9400b2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.072+0000 7fc944dff700 1 -- 172.21.15.27:0/706017624 >> 172.21.15.27:0/706017624 conn(0x7fc94006ed40 msgr2=0x7fc940158a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:42.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.073+0000 7fc944dff700 1 -- 172.21.15.27:0/706017624 shutdown_connections 2024-01-11T09:43:42.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.074+0000 7fc944dff700 1 -- 172.21.15.27:0/706017624 wait complete. 2024-01-11T09:43:42.074 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 24 and data pool 25 2024-01-11T09:43:42.101 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:43:42.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.310+0000 7f5d297ff700 1 Processor -- start 2024-01-11T09:43:42.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.310+0000 7f5d297ff700 1 -- start start 2024-01-11T09:43:42.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.311+0000 7f5d297ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d240b19d0 0x7f5d24158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:42.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.311+0000 7f5d297ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5d2406b610 0x7f5d24159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:42.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.311+0000 7f5d297ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d24158b60 con 0x7f5d241575e0 2024-01-11T09:43:42.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.311+0000 7f5d297ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d240cb5e0 con 0x7f5d2406b610 2024-01-11T09:43:42.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.311+0000 7f5d297ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d241599b0 con 0x7f5d240b19d0 2024-01-11T09:43:42.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.312+0000 7f5d237fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5d2406b610 0x7f5d24159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:42.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.312+0000 7f5d22ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5d241575e0 0x7f5d240b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33512/0 (socket says 172.21.15.27:33512) 2024-01-11T09:43:42.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.312+0000 7f5d22ffd700 1 -- 172.21.15.27:0/993434933 learned_addr learned my addr 172.21.15.27:0/993434933 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:42.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.312+0000 7f5d227fc700 1 --2- 172.21.15.27:0/993434933 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d240b19d0 0x7f5d24158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:42.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.312+0000 7f5d237fe700 1 -- 172.21.15.27:0/993434933 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d240b19d0 msgr2=0x7f5d24158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:42.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.312+0000 7f5d237fe700 1 --2- 172.21.15.27:0/993434933 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d240b19d0 0x7f5d24158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.312+0000 7f5d237fe700 1 -- 172.21.15.27:0/993434933 >> v1:172.21.15.27:6789/0 conn(0x7f5d241575e0 legacy=0x7f5d240b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:42.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.312+0000 7f5d237fe700 1 -- 172.21.15.27:0/993434933 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d240c0f90 con 0x7f5d2406b610 2024-01-11T09:43:42.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.312+0000 7f5d21ffb700 1 -- 172.21.15.27:0/993434933 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711874358 0 0) 0x7f5d14001110 con 0x7f5d241575e0 2024-01-11T09:43:42.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.313+0000 7f5d237fe700 1 --2- 172.21.15.27:0/993434933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d2406b610 0x7f5d24159e40 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009e60 tx=0x7f5d1c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcbbe807b610ffb1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:42.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.313+0000 7f5d21ffb700 1 -- 172.21.15.27:0/993434933 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c00e070 con 0x7f5d2406b610 2024-01-11T09:43:42.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.313+0000 7f5d21ffb700 1 -- 172.21.15.27:0/993434933 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c00bb60 con 0x7f5d2406b610 2024-01-11T09:43:42.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.313+0000 7f5d21ffb700 1 -- 172.21.15.27:0/993434933 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c012430 con 0x7f5d2406b610 2024-01-11T09:43:42.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.313+0000 7f5d297ff700 1 -- 172.21.15.27:0/993434933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d2406b610 msgr2=0x7f5d24159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:42.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.313+0000 7f5d297ff700 1 --2- 172.21.15.27:0/993434933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d2406b610 0x7f5d24159e40 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009e60 tx=0x7f5d1c009780 comp rx=0 tx=0).stop 2024-01-11T09:43:42.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.313+0000 7f5d297ff700 1 -- 172.21.15.27:0/993434933 shutdown_connections 2024-01-11T09:43:42.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.313+0000 7f5d297ff700 1 --2- 172.21.15.27:0/993434933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d2406b610 0x7f5d24159e40 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.313+0000 7f5d297ff700 1 --2- 172.21.15.27:0/993434933 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d240b19d0 0x7f5d24158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.313+0000 7f5d297ff700 1 -- 172.21.15.27:0/993434933 >> 172.21.15.27:0/993434933 conn(0x7f5d240680e0 msgr2=0x7f5d24069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:42.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.314+0000 7f5d297ff700 1 -- 172.21.15.27:0/993434933 shutdown_connections 2024-01-11T09:43:42.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.314+0000 7f5d297ff700 1 -- 172.21.15.27:0/993434933 wait complete. 2024-01-11T09:43:42.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.315+0000 7f5d297ff700 1 Processor -- start 2024-01-11T09:43:42.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.315+0000 7f5d297ff700 1 -- start start 2024-01-11T09:43:42.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.316+0000 7f5d297ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d2406b610 0x7f5d240c1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:42.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.316+0000 7f5d297ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d240b19d0 0x7f5d240adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:42.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.316+0000 7f5d297ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d241575e0 0x7f5d2415d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:42.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.316+0000 7f5d297ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d240c85f0 con 0x7f5d240b19d0 2024-01-11T09:43:42.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.316+0000 7f5d297ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d240c7740 con 0x7f5d241575e0 2024-01-11T09:43:42.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.316+0000 7f5d297ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d240c79e0 con 0x7f5d2406b610 2024-01-11T09:43:42.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.317+0000 7f5d227fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d240b19d0 0x7f5d240adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:42.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.317+0000 7f5d227fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d240b19d0 0x7f5d240adb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41138/0 (socket says 172.21.15.27:41138) 2024-01-11T09:43:42.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.317+0000 7f5d227fc700 1 -- 172.21.15.27:0/109074437 learned_addr learned my addr 172.21.15.27:0/109074437 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:42.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.317+0000 7f5d22ffd700 1 --2- 172.21.15.27:0/109074437 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d2406b610 0x7f5d240c1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:42.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.317+0000 7f5d237fe700 1 --2- 172.21.15.27:0/109074437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d241575e0 0x7f5d2415d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:42.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.317+0000 7f5d227fc700 1 -- 172.21.15.27:0/109074437 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d2406b610 msgr2=0x7f5d240c1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:42.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.317+0000 7f5d227fc700 1 --2- 172.21.15.27:0/109074437 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d2406b610 0x7f5d240c1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.317+0000 7f5d227fc700 1 -- 172.21.15.27:0/109074437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d241575e0 msgr2=0x7f5d2415d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:42.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.317+0000 7f5d227fc700 1 --2- 172.21.15.27:0/109074437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d241575e0 0x7f5d2415d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.317+0000 7f5d227fc700 1 -- 172.21.15.27:0/109074437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1c009a20 con 0x7f5d240b19d0 2024-01-11T09:43:42.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.317+0000 7f5d227fc700 1 --2- 172.21.15.27:0/109074437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d240b19d0 0x7f5d240adb30 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f5d1800ec80 tx=0x7f5d180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:42.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.317+0000 7f5d0affd700 1 -- 172.21.15.27:0/109074437 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1800c850 con 0x7f5d240b19d0 2024-01-11T09:43:42.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.318+0000 7f5d0affd700 1 -- 172.21.15.27:0/109074437 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1800c9b0 con 0x7f5d240b19d0 2024-01-11T09:43:42.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.318+0000 7f5d0affd700 1 -- 172.21.15.27:0/109074437 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1800cd50 con 0x7f5d240b19d0 2024-01-11T09:43:42.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.318+0000 7f5d297ff700 1 -- 172.21.15.27:0/109074437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d240c7cc0 con 0x7f5d240b19d0 2024-01-11T09:43:42.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.318+0000 7f5d297ff700 1 -- 172.21.15.27:0/109074437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d240c8280 con 0x7f5d240b19d0 2024-01-11T09:43:42.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.319+0000 7f5d297ff700 1 -- 172.21.15.27:0/109074437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d240c8760 con 0x7f5d240b19d0 2024-01-11T09:43:42.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.320+0000 7f5d0affd700 1 -- 172.21.15.27:0/109074437 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5d18013680 con 0x7f5d240b19d0 2024-01-11T09:43:42.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.321+0000 7f5d0affd700 1 --2- 172.21.15.27:0/109074437 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5d10066f30 0x7f5d100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:42.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.321+0000 7f5d0affd700 1 -- 172.21.15.27:0/109074437 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f5d180847b0 con 0x7f5d240b19d0 2024-01-11T09:43:42.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.321+0000 7f5d22ffd700 1 --2- 172.21.15.27:0/109074437 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5d10066f30 0x7f5d100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:42.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.322+0000 7f5d22ffd700 1 --2- 172.21.15.27:0/109074437 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5d10066f30 0x7f5d100693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5d14007160 tx=0x7f5d14007f50 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:42.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.325+0000 7f5d0affd700 1 -- 172.21.15.27:0/109074437 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5d180519d0 con 0x7f5d240b19d0 2024-01-11T09:43:42.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.528+0000 7f5d297ff700 1 -- 172.21.15.27:0/109074437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d240718d0 con 0x7f5d240b19d0 2024-01-11T09:43:42.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.529+0000 7f5d0affd700 1 -- 172.21.15.27:0/109074437 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+15463 (secure 0 0 0) 0x7f5d18054ff0 con 0x7f5d240b19d0 2024-01-11T09:43:42.521 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:43:42.521 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":197,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:43:41.553828+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:43:39.267819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-01-11T09:43:40.563877+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:43:42.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.532+0000 7f5d297ff700 1 -- 172.21.15.27:0/109074437 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5d10066f30 msgr2=0x7f5d100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:42.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.532+0000 7f5d297ff700 1 --2- 172.21.15.27:0/109074437 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5d10066f30 0x7f5d100693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5d14007160 tx=0x7f5d14007f50 comp rx=0 tx=0).stop 2024-01-11T09:43:42.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.532+0000 7f5d297ff700 1 -- 172.21.15.27:0/109074437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d240b19d0 msgr2=0x7f5d240adb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:42.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.532+0000 7f5d297ff700 1 --2- 172.21.15.27:0/109074437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d240b19d0 0x7f5d240adb30 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f5d1800ec80 tx=0x7f5d180091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.532+0000 7f5d297ff700 1 -- 172.21.15.27:0/109074437 shutdown_connections 2024-01-11T09:43:42.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.532+0000 7f5d297ff700 1 --2- 172.21.15.27:0/109074437 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5d10066f30 0x7f5d100693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.532+0000 7f5d297ff700 1 --2- 172.21.15.27:0/109074437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d241575e0 0x7f5d2415d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.532+0000 7f5d297ff700 1 --2- 172.21.15.27:0/109074437 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d2406b610 0x7f5d240c1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.532+0000 7f5d297ff700 1 --2- 172.21.15.27:0/109074437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d240b19d0 0x7f5d240adb30 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.532+0000 7f5d297ff700 1 -- 172.21.15.27:0/109074437 >> 172.21.15.27:0/109074437 conn(0x7f5d240680e0 msgr2=0x7f5d24159e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:42.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.532+0000 7f5d297ff700 1 -- 172.21.15.27:0/109074437 shutdown_connections 2024-01-11T09:43:42.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.532+0000 7f5d297ff700 1 -- 172.21.15.27:0/109074437 wait complete. 2024-01-11T09:43:42.542 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:43:42.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.751+0000 7f4462dff700 1 Processor -- start 2024-01-11T09:43:42.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.751+0000 7f4462dff700 1 -- start start 2024-01-11T09:43:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.752+0000 7f4462dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f445c0b03e0 0x7f445c0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.752+0000 7f4462dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f445c0b07b0 0x7f445c0b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.752+0000 7f4462dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f445c0cd730 con 0x7f445c159200 2024-01-11T09:43:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.752+0000 7f4462dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f445c159d30 con 0x7f445c0b03e0 2024-01-11T09:43:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.752+0000 7f4462dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f445c0ccb20 con 0x7f445c0b07b0 2024-01-11T09:43:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.752+0000 7f4460b9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f445c0b03e0 0x7f445c0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.752+0000 7f446139c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f445c159200 0x7f445c159620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33528/0 (socket says 172.21.15.27:33528) 2024-01-11T09:43:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.752+0000 7f446139c700 1 -- 172.21.15.27:0/3766549044 learned_addr learned my addr 172.21.15.27:0/3766549044 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.752+0000 7f445bfff700 1 --2- 172.21.15.27:0/3766549044 >> v2:172.21.15.77:3300/0 conn(0x7f445c0b07b0 0x7f445c0b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.753+0000 7f4460b9b700 1 -- 172.21.15.27:0/3766549044 >> v2:172.21.15.77:3300/0 conn(0x7f445c0b07b0 msgr2=0x7f445c0b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.753+0000 7f4460b9b700 1 --2- 172.21.15.27:0/3766549044 >> v2:172.21.15.77:3300/0 conn(0x7f445c0b07b0 0x7f445c0b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.753+0000 7f4460b9b700 1 -- 172.21.15.27:0/3766549044 >> v1:172.21.15.27:6789/0 conn(0x7f445c159200 legacy=0x7f445c159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:42.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.753+0000 7f4460b9b700 1 -- 172.21.15.27:0/3766549044 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445c0cce00 con 0x7f445c0b03e0 2024-01-11T09:43:42.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.753+0000 7f445b7fe700 1 -- 172.21.15.27:0/3766549044 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555559561 0 0) 0x7f445c0cd730 con 0x7f445c159200 2024-01-11T09:43:42.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.753+0000 7f4460b9b700 1 --2- 172.21.15.27:0/3766549044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f445c0b03e0 0x7f445c0b2480 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f445c0cdfc0 tx=0x7f44500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2286077c1093448e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:42.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.753+0000 7f445b7fe700 1 -- 172.21.15.27:0/3766549044 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450011630 con 0x7f445c0b03e0 2024-01-11T09:43:42.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.753+0000 7f445b7fe700 1 -- 172.21.15.27:0/3766549044 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4450011790 con 0x7f445c0b03e0 2024-01-11T09:43:42.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.753+0000 7f445b7fe700 1 -- 172.21.15.27:0/3766549044 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450011b30 con 0x7f445c0b03e0 2024-01-11T09:43:42.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.754+0000 7f4462dff700 1 -- 172.21.15.27:0/3766549044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f445c0b03e0 msgr2=0x7f445c0b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:42.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.754+0000 7f4462dff700 1 --2- 172.21.15.27:0/3766549044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f445c0b03e0 0x7f445c0b2480 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f445c0cdfc0 tx=0x7f44500091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.754+0000 7f4462dff700 1 -- 172.21.15.27:0/3766549044 shutdown_connections 2024-01-11T09:43:42.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.754+0000 7f4462dff700 1 --2- 172.21.15.27:0/3766549044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f445c0b03e0 0x7f445c0b2480 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.754+0000 7f4462dff700 1 --2- 172.21.15.27:0/3766549044 >> v2:172.21.15.77:3300/0 conn(0x7f445c0b07b0 0x7f445c0b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.754+0000 7f4462dff700 1 -- 172.21.15.27:0/3766549044 >> 172.21.15.27:0/3766549044 conn(0x7f445c06ed40 msgr2=0x7f445c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:42.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.754+0000 7f4462dff700 1 -- 172.21.15.27:0/3766549044 shutdown_connections 2024-01-11T09:43:42.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.754+0000 7f4462dff700 1 -- 172.21.15.27:0/3766549044 wait complete. 2024-01-11T09:43:42.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.756+0000 7f4462dff700 1 Processor -- start 2024-01-11T09:43:42.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.757+0000 7f4462dff700 1 -- start start 2024-01-11T09:43:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.758+0000 7f4462dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f445c0b03e0 0x7f445c16a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.758+0000 7f4462dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f445c0b07b0 0x7f445c16ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.758+0000 7f4462dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445c159200 0x7f445c16c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.758+0000 7f4462dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445c0d0cc0 con 0x7f445c0b03e0 2024-01-11T09:43:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.758+0000 7f4462dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445c16aa40 con 0x7f445c159200 2024-01-11T09:43:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.758+0000 7f4462dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445c16b190 con 0x7f445c0b07b0 2024-01-11T09:43:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.758+0000 7f4460b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f445c0b03e0 0x7f445c16a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.758+0000 7f4460b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f445c0b03e0 0x7f445c16a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41156/0 (socket says 172.21.15.27:41156) 2024-01-11T09:43:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.758+0000 7f4460b9b700 1 -- 172.21.15.27:0/3052395571 learned_addr learned my addr 172.21.15.27:0/3052395571 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.758+0000 7f446139c700 1 --2- 172.21.15.27:0/3052395571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445c159200 0x7f445c16c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.758+0000 7f4460b9b700 1 -- 172.21.15.27:0/3052395571 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f445c0b07b0 msgr2=0x7f445c16ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.758+0000 7f445bfff700 1 --2- 172.21.15.27:0/3052395571 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f445c0b07b0 0x7f445c16ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.758+0000 7f4460b9b700 1 --2- 172.21.15.27:0/3052395571 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f445c0b07b0 0x7f445c16ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.758+0000 7f4460b9b700 1 -- 172.21.15.27:0/3052395571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445c159200 msgr2=0x7f445c16c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.758+0000 7f4460b9b700 1 --2- 172.21.15.27:0/3052395571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445c159200 0x7f445c16c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.758+0000 7f4460b9b700 1 -- 172.21.15.27:0/3052395571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4450009420 con 0x7f445c0b03e0 2024-01-11T09:43:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.758+0000 7f446139c700 1 --2- 172.21.15.27:0/3052395571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445c159200 0x7f445c16c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:43:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.758+0000 7f445bfff700 1 --2- 172.21.15.27:0/3052395571 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f445c0b07b0 0x7f445c16ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:43:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.759+0000 7f4460b9b700 1 --2- 172.21.15.27:0/3052395571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f445c0b03e0 0x7f445c16a500 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f445c069540 tx=0x7f4450017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:42.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.759+0000 7f4458ff9700 1 -- 172.21.15.27:0/3052395571 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450011b10 con 0x7f445c0b03e0 2024-01-11T09:43:42.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.759+0000 7f4458ff9700 1 -- 172.21.15.27:0/3052395571 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44500253f0 con 0x7f445c0b03e0 2024-01-11T09:43:42.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.759+0000 7f4462dff700 1 -- 172.21.15.27:0/3052395571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f445c0ce570 con 0x7f445c0b03e0 2024-01-11T09:43:42.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.759+0000 7f4458ff9700 1 -- 172.21.15.27:0/3052395571 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450025790 con 0x7f445c0b03e0 2024-01-11T09:43:42.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.759+0000 7f4462dff700 1 -- 172.21.15.27:0/3052395571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f445c0ceb30 con 0x7f445c0b03e0 2024-01-11T09:43:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.760+0000 7f4462dff700 1 -- 172.21.15.27:0/3052395571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f445c0718d0 con 0x7f445c0b03e0 2024-01-11T09:43:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.761+0000 7f4458ff9700 1 -- 172.21.15.27:0/3052395571 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4450025b80 con 0x7f445c0b03e0 2024-01-11T09:43:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.762+0000 7f4458ff9700 1 --2- 172.21.15.27:0/3052395571 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4440066f30 0x7f44400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.762+0000 7f4458ff9700 1 -- 172.21.15.27:0/3052395571 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f4450085ef0 con 0x7f445c0b03e0 2024-01-11T09:43:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.763+0000 7f445bfff700 1 --2- 172.21.15.27:0/3052395571 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4440066f30 0x7f44400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:42.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.763+0000 7f445bfff700 1 --2- 172.21.15.27:0/3052395571 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4440066f30 0x7f44400693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4448006f00 tx=0x7f4448008040 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:42.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.767+0000 7f4458ff9700 1 -- 172.21.15.27:0/3052395571 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4450053110 con 0x7f445c0b03e0 2024-01-11T09:43:42.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.970+0000 7f4462dff700 1 -- 172.21.15.27:0/3052395571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f445c0cd0e0 con 0x7f445c0b03e0 2024-01-11T09:43:42.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.971+0000 7f4458ff9700 1 -- 172.21.15.27:0/3052395571 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+15463 (secure 0 0 0) 0x7f4450056730 con 0x7f445c0b03e0 2024-01-11T09:43:42.963 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:43:42.964 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":197,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:43:41.553828+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:43:39.267819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-01-11T09:43:40.563877+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:43:42.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.974+0000 7f4462dff700 1 -- 172.21.15.27:0/3052395571 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4440066f30 msgr2=0x7f44400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:42.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.974+0000 7f4462dff700 1 --2- 172.21.15.27:0/3052395571 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4440066f30 0x7f44400693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4448006f00 tx=0x7f4448008040 comp rx=0 tx=0).stop 2024-01-11T09:43:42.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.974+0000 7f4462dff700 1 -- 172.21.15.27:0/3052395571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f445c0b03e0 msgr2=0x7f445c16a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:42.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.974+0000 7f4462dff700 1 --2- 172.21.15.27:0/3052395571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f445c0b03e0 0x7f445c16a500 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f445c069540 tx=0x7f4450017000 comp rx=0 tx=0).stop 2024-01-11T09:43:42.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.974+0000 7f4462dff700 1 -- 172.21.15.27:0/3052395571 shutdown_connections 2024-01-11T09:43:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.974+0000 7f4462dff700 1 --2- 172.21.15.27:0/3052395571 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4440066f30 0x7f44400693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.974+0000 7f4462dff700 1 --2- 172.21.15.27:0/3052395571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f445c159200 0x7f445c16c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.974+0000 7f4462dff700 1 --2- 172.21.15.27:0/3052395571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f445c0b03e0 0x7f445c16a500 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.974+0000 7f4462dff700 1 --2- 172.21.15.27:0/3052395571 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f445c0b07b0 0x7f445c16ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.974+0000 7f4462dff700 1 -- 172.21.15.27:0/3052395571 >> 172.21.15.27:0/3052395571 conn(0x7f445c06ed40 msgr2=0x7f445c158990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.974+0000 7f4462dff700 1 -- 172.21.15.27:0/3052395571 shutdown_connections 2024-01-11T09:43:42.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:42.974+0000 7f4462dff700 1 -- 172.21.15.27:0/3052395571 wait complete. 2024-01-11T09:43:42.984 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-11T09:43:43.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.197+0000 7f65c19ff700 1 Processor -- start 2024-01-11T09:43:43.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.197+0000 7f65c19ff700 1 -- start start 2024-01-11T09:43:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.198+0000 7f65c19ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f65bc1596f0 0x7f65bc0a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.199+0000 7f65c19ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f65bc068d30 0x7f65bc0c8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.199+0000 7f65c19ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65bc0c80a0 con 0x7f65bc0a9de0 2024-01-11T09:43:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.199+0000 7f65c19ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65bc0cbac0 con 0x7f65bc068d30 2024-01-11T09:43:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.199+0000 7f65c19ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65bc0c7490 con 0x7f65bc1596f0 2024-01-11T09:43:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.199+0000 7f65bb7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f65bc068d30 0x7f65bc0c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.199+0000 7f65bb7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f65bc068d30 0x7f65bc0c8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41168/0 (socket says 172.21.15.27:41168) 2024-01-11T09:43:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.199+0000 7f65bb7fe700 1 -- 172.21.15.27:0/4166762017 learned_addr learned my addr 172.21.15.27:0/4166762017 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.199+0000 7f65baffd700 1 --2- 172.21.15.27:0/4166762017 >> v2:172.21.15.77:3300/0 conn(0x7f65bc1596f0 0x7f65bc0a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.199+0000 7f65bb7fe700 1 -- 172.21.15.27:0/4166762017 >> v2:172.21.15.77:3300/0 conn(0x7f65bc1596f0 msgr2=0x7f65bc0a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.199+0000 7f65bb7fe700 1 --2- 172.21.15.27:0/4166762017 >> v2:172.21.15.77:3300/0 conn(0x7f65bc1596f0 0x7f65bc0a98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.199+0000 7f65bb7fe700 1 -- 172.21.15.27:0/4166762017 >> v1:172.21.15.77:6789/0 conn(0x7f65bc0a9de0 legacy=0x7f65bc068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.199+0000 7f65bb7fe700 1 -- 172.21.15.27:0/4166762017 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65bc0c7770 con 0x7f65bc068d30 2024-01-11T09:43:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.199+0000 7f65baffd700 1 --2- 172.21.15.27:0/4166762017 >> v2:172.21.15.77:3300/0 conn(0x7f65bc1596f0 0x7f65bc0a98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.200+0000 7f65bb7fe700 1 --2- 172.21.15.27:0/4166762017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65bc068d30 0x7f65bc0c8530 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f65bc0cc9b0 tx=0x7f65b4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a0de757a56743d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.200+0000 7f65b9ffb700 1 -- 172.21.15.27:0/4166762017 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b400e070 con 0x7f65bc068d30 2024-01-11T09:43:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.200+0000 7f65b9ffb700 1 -- 172.21.15.27:0/4166762017 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b400ba50 con 0x7f65bc068d30 2024-01-11T09:43:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.200+0000 7f65b9ffb700 1 -- 172.21.15.27:0/4166762017 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b40092e0 con 0x7f65bc068d30 2024-01-11T09:43:43.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.200+0000 7f65c19ff700 1 -- 172.21.15.27:0/4166762017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65bc068d30 msgr2=0x7f65bc0c8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:43.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.200+0000 7f65c19ff700 1 --2- 172.21.15.27:0/4166762017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65bc068d30 0x7f65bc0c8530 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f65bc0cc9b0 tx=0x7f65b4009840 comp rx=0 tx=0).stop 2024-01-11T09:43:43.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.201+0000 7f65c19ff700 1 -- 172.21.15.27:0/4166762017 shutdown_connections 2024-01-11T09:43:43.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.201+0000 7f65c19ff700 1 --2- 172.21.15.27:0/4166762017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65bc068d30 0x7f65bc0c8530 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:43.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.201+0000 7f65c19ff700 1 --2- 172.21.15.27:0/4166762017 >> v2:172.21.15.77:3300/0 conn(0x7f65bc1596f0 0x7f65bc0a98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:43.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.201+0000 7f65c19ff700 1 -- 172.21.15.27:0/4166762017 >> 172.21.15.27:0/4166762017 conn(0x7f65bc0c9fb0 msgr2=0x7f65bc06ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:43.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.201+0000 7f65c19ff700 1 -- 172.21.15.27:0/4166762017 shutdown_connections 2024-01-11T09:43:43.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.201+0000 7f65c19ff700 1 -- 172.21.15.27:0/4166762017 wait complete. 2024-01-11T09:43:43.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.203+0000 7f65c19ff700 1 Processor -- start 2024-01-11T09:43:43.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.203+0000 7f65c19ff700 1 -- start start 2024-01-11T09:43:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.204+0000 7f65c19ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65bc068d30 0x7f65bc1493c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.204+0000 7f65c19ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65bc0a9de0 0x7f65bc0b1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.204+0000 7f65c19ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65bc1596f0 0x7f65bc0b1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.204+0000 7f65c19ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc16e280 con 0x7f65bc1596f0 2024-01-11T09:43:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.204+0000 7f65c19ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc16acb0 con 0x7f65bc068d30 2024-01-11T09:43:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.204+0000 7f65c19ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc169ef0 con 0x7f65bc0a9de0 2024-01-11T09:43:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.205+0000 7f65bb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65bc1596f0 0x7f65bc0b1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.205+0000 7f65bb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65bc1596f0 0x7f65bc0b1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41178/0 (socket says 172.21.15.27:41178) 2024-01-11T09:43:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.205+0000 7f65bb7fe700 1 -- 172.21.15.27:0/2835685457 learned_addr learned my addr 172.21.15.27:0/2835685457 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.205+0000 7f65baffd700 1 --2- 172.21.15.27:0/2835685457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65bc068d30 0x7f65bc1493c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.205+0000 7f65ba7fc700 1 --2- 172.21.15.27:0/2835685457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65bc0a9de0 0x7f65bc0b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.205+0000 7f65bb7fe700 1 -- 172.21.15.27:0/2835685457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65bc0a9de0 msgr2=0x7f65bc0b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.205+0000 7f65bb7fe700 1 --2- 172.21.15.27:0/2835685457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65bc0a9de0 0x7f65bc0b1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.205+0000 7f65bb7fe700 1 -- 172.21.15.27:0/2835685457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65bc068d30 msgr2=0x7f65bc1493c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:43.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.205+0000 7f65bb7fe700 1 --2- 172.21.15.27:0/2835685457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65bc068d30 0x7f65bc1493c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:43.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.205+0000 7f65bb7fe700 1 -- 172.21.15.27:0/2835685457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b4009ae0 con 0x7f65bc1596f0 2024-01-11T09:43:43.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.205+0000 7f65bb7fe700 1 --2- 172.21.15.27:0/2835685457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65bc1596f0 0x7f65bc0b1970 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f65bc06d730 tx=0x7f65b4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:43.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.206+0000 7f65a2ffd700 1 -- 172.21.15.27:0/2835685457 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b400be60 con 0x7f65bc1596f0 2024-01-11T09:43:43.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.206+0000 7f65a2ffd700 1 -- 172.21.15.27:0/2835685457 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b4008d90 con 0x7f65bc1596f0 2024-01-11T09:43:43.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.206+0000 7f65a2ffd700 1 -- 172.21.15.27:0/2835685457 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b40266b0 con 0x7f65bc1596f0 2024-01-11T09:43:43.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.206+0000 7f65c19ff700 1 -- 172.21.15.27:0/2835685457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65bc16a1d0 con 0x7f65bc1596f0 2024-01-11T09:43:43.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.206+0000 7f65c19ff700 1 -- 172.21.15.27:0/2835685457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65bc16a790 con 0x7f65bc1596f0 2024-01-11T09:43:43.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.207+0000 7f65c19ff700 1 -- 172.21.15.27:0/2835685457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65bc04c3c0 con 0x7f65bc1596f0 2024-01-11T09:43:43.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.208+0000 7f65a2ffd700 1 -- 172.21.15.27:0/2835685457 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f65b4003040 con 0x7f65bc1596f0 2024-01-11T09:43:43.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.209+0000 7f65a2ffd700 1 --2- 172.21.15.27:0/2835685457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f65a8066f30 0x7f65a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:43.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.209+0000 7f65a2ffd700 1 -- 172.21.15.27:0/2835685457 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f65b4085fe0 con 0x7f65bc1596f0 2024-01-11T09:43:43.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.209+0000 7f65baffd700 1 --2- 172.21.15.27:0/2835685457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f65a8066f30 0x7f65a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:43.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.210+0000 7f65baffd700 1 --2- 172.21.15.27:0/2835685457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f65a8066f30 0x7f65a80693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f65ac006f00 tx=0x7f65ac008040 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:43.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.213+0000 7f65a2ffd700 1 -- 172.21.15.27:0/2835685457 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f65b4053180 con 0x7f65bc1596f0 2024-01-11T09:43:43.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:43.466+0000 7f65c19ff700 1 -- 172.21.15.27:0/2835685457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f65bc0c7a50 con 0x7f65bc1596f0 2024-01-11T09:43:44.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.074+0000 7f65a2ffd700 1 -- 172.21.15.27:0/2835685457 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v101) v1 ==== 118+0+0 (secure 0 0 0) 0x7f65b40210b0 con 0x7f65bc1596f0 2024-01-11T09:43:44.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.077+0000 7f65c19ff700 1 -- 172.21.15.27:0/2835685457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f65a8066f30 msgr2=0x7f65a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:44.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.077+0000 7f65c19ff700 1 --2- 172.21.15.27:0/2835685457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f65a8066f30 0x7f65a80693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f65ac006f00 tx=0x7f65ac008040 comp rx=0 tx=0).stop 2024-01-11T09:43:44.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.077+0000 7f65c19ff700 1 -- 172.21.15.27:0/2835685457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65bc1596f0 msgr2=0x7f65bc0b1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:44.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.077+0000 7f65c19ff700 1 --2- 172.21.15.27:0/2835685457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65bc1596f0 0x7f65bc0b1970 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f65bc06d730 tx=0x7f65b4018000 comp rx=0 tx=0).stop 2024-01-11T09:43:44.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.077+0000 7f65c19ff700 1 -- 172.21.15.27:0/2835685457 shutdown_connections 2024-01-11T09:43:44.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.077+0000 7f65c19ff700 1 --2- 172.21.15.27:0/2835685457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65bc1596f0 0x7f65bc0b1970 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:44.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.077+0000 7f65c19ff700 1 --2- 172.21.15.27:0/2835685457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f65a8066f30 0x7f65a80693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:44.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.077+0000 7f65c19ff700 1 --2- 172.21.15.27:0/2835685457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65bc068d30 0x7f65bc1493c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:44.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.077+0000 7f65c19ff700 1 --2- 172.21.15.27:0/2835685457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65bc0a9de0 0x7f65bc0b1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.077+0000 7f65c19ff700 1 -- 172.21.15.27:0/2835685457 >> 172.21.15.27:0/2835685457 conn(0x7f65bc0c9fb0 msgr2=0x7f65bc1586c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.077+0000 7f65c19ff700 1 -- 172.21.15.27:0/2835685457 shutdown_connections 2024-01-11T09:43:44.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.077+0000 7f65c19ff700 1 -- 172.21.15.27:0/2835685457 wait complete. 2024-01-11T09:43:44.089 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-11T09:43:44.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.301+0000 7f7dfa1ff700 1 Processor -- start 2024-01-11T09:43:44.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.302+0000 7f7dfa1ff700 1 -- start start 2024-01-11T09:43:44.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.303+0000 7f7dfa1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7df4069b60 0x7f7df40cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:44.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.303+0000 7f7dfa1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7df415d6d0 0x7f7df415db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:44.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.303+0000 7f7dfa1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df40cb770 con 0x7f7df4069790 2024-01-11T09:43:44.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.303+0000 7f7dfa1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7df40cab60 con 0x7f7df4069b60 2024-01-11T09:43:44.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.303+0000 7f7dfa1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df40cae40 con 0x7f7df415d6d0 2024-01-11T09:43:44.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.303+0000 7f7df2ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7df4069b60 0x7f7df40cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:44.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.303+0000 7f7df2ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7df4069b60 0x7f7df40cbc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57344/0 (socket says 172.21.15.27:57344) 2024-01-11T09:43:44.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.303+0000 7f7df2ffd700 1 -- 172.21.15.27:0/2201514694 learned_addr learned my addr 172.21.15.27:0/2201514694 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:44.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.303+0000 7f7df3fff700 1 --2- 172.21.15.27:0/2201514694 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7df415d6d0 0x7f7df415db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:44.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.304+0000 7f7df2ffd700 1 -- 172.21.15.27:0/2201514694 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7df415d6d0 msgr2=0x7f7df415db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:44.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.304+0000 7f7df2ffd700 1 --2- 172.21.15.27:0/2201514694 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7df415d6d0 0x7f7df415db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:44.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.304+0000 7f7df2ffd700 1 -- 172.21.15.27:0/2201514694 >> v1:172.21.15.77:6789/0 conn(0x7f7df4069790 legacy=0x7f7df40c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:44.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.304+0000 7f7df2ffd700 1 -- 172.21.15.27:0/2201514694 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df40cb120 con 0x7f7df4069b60 2024-01-11T09:43:44.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.304+0000 7f7df27fc700 1 -- 172.21.15.27:0/2201514694 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170708154 0 0) 0x7f7df40cb770 con 0x7f7df4069790 2024-01-11T09:43:44.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.304+0000 7f7df2ffd700 1 --2- 172.21.15.27:0/2201514694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df4069b60 0x7f7df40cbc00 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f7de80099b0 tx=0x7f7de8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=629af642ccd0e41c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:44.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.304+0000 7f7df27fc700 1 -- 172.21.15.27:0/2201514694 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de8009ea0 con 0x7f7df4069b60 2024-01-11T09:43:44.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.304+0000 7f7df27fc700 1 -- 172.21.15.27:0/2201514694 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de800b8f0 con 0x7f7df4069b60 2024-01-11T09:43:44.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.304+0000 7f7df27fc700 1 -- 172.21.15.27:0/2201514694 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de800bc90 con 0x7f7df4069b60 2024-01-11T09:43:44.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.305+0000 7f7dfa1ff700 1 -- 172.21.15.27:0/2201514694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df4069b60 msgr2=0x7f7df40cbc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:44.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.305+0000 7f7dfa1ff700 1 --2- 172.21.15.27:0/2201514694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df4069b60 0x7f7df40cbc00 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f7de80099b0 tx=0x7f7de8009250 comp rx=0 tx=0).stop 2024-01-11T09:43:44.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.305+0000 7f7dfa1ff700 1 -- 172.21.15.27:0/2201514694 shutdown_connections 2024-01-11T09:43:44.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.305+0000 7f7dfa1ff700 1 --2- 172.21.15.27:0/2201514694 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7df415d6d0 0x7f7df415db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:44.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.305+0000 7f7dfa1ff700 1 --2- 172.21.15.27:0/2201514694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df4069b60 0x7f7df40cbc00 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:44.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.305+0000 7f7dfa1ff700 1 -- 172.21.15.27:0/2201514694 >> 172.21.15.27:0/2201514694 conn(0x7f7df40680a0 msgr2=0x7f7df40cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:44.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.305+0000 7f7dfa1ff700 1 -- 172.21.15.27:0/2201514694 shutdown_connections 2024-01-11T09:43:44.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.305+0000 7f7dfa1ff700 1 -- 172.21.15.27:0/2201514694 wait complete. 2024-01-11T09:43:44.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.306+0000 7f7dfa1ff700 1 Processor -- start 2024-01-11T09:43:44.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.307+0000 7f7dfa1ff700 1 -- start start 2024-01-11T09:43:44.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.307+0000 7f7dfa1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df4069790 0x7f7df41440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:44.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.307+0000 7f7dfa1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df4069b60 0x7f7df40b11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:44.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.307+0000 7f7dfa1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7df415d6d0 0x7f7df40b1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:44.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.307+0000 7f7dfa1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df40b1c40 con 0x7f7df4069b60 2024-01-11T09:43:44.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.307+0000 7f7dfa1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df406e710 con 0x7f7df4069790 2024-01-11T09:43:44.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.307+0000 7f7dfa1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df406e9f0 con 0x7f7df415d6d0 2024-01-11T09:43:44.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.308+0000 7f7df2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df4069b60 0x7f7df40b11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:44.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.308+0000 7f7df2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df4069b60 0x7f7df40b11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57354/0 (socket says 172.21.15.27:57354) 2024-01-11T09:43:44.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.308+0000 7f7df2ffd700 1 -- 172.21.15.27:0/1402668705 learned_addr learned my addr 172.21.15.27:0/1402668705 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:44.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.308+0000 7f7df3fff700 1 --2- 172.21.15.27:0/1402668705 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7df415d6d0 0x7f7df40b1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:44.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.308+0000 7f7df37fe700 1 --2- 172.21.15.27:0/1402668705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df4069790 0x7f7df41440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:44.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.308+0000 7f7df2ffd700 1 -- 172.21.15.27:0/1402668705 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7df415d6d0 msgr2=0x7f7df40b1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:44.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.308+0000 7f7df2ffd700 1 --2- 172.21.15.27:0/1402668705 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7df415d6d0 0x7f7df40b1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:44.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.308+0000 7f7df2ffd700 1 -- 172.21.15.27:0/1402668705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df4069790 msgr2=0x7f7df41440c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:44.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.308+0000 7f7df2ffd700 1 --2- 172.21.15.27:0/1402668705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df4069790 0x7f7df41440c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:44.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.308+0000 7f7df2ffd700 1 -- 172.21.15.27:0/1402668705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de80094f0 con 0x7f7df4069b60 2024-01-11T09:43:44.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.309+0000 7f7df2ffd700 1 --2- 172.21.15.27:0/1402668705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df4069b60 0x7f7df40b11c0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f7de8006fd0 tx=0x7f7de8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:44.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.309+0000 7f7ddb7fe700 1 -- 172.21.15.27:0/1402668705 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de800bc70 con 0x7f7df4069b60 2024-01-11T09:43:44.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.309+0000 7f7ddb7fe700 1 -- 172.21.15.27:0/1402668705 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de8026450 con 0x7f7df4069b60 2024-01-11T09:43:44.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.309+0000 7f7dfa1ff700 1 -- 172.21.15.27:0/1402668705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df406ecd0 con 0x7f7df4069b60 2024-01-11T09:43:44.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.309+0000 7f7ddb7fe700 1 -- 172.21.15.27:0/1402668705 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de8026810 con 0x7f7df4069b60 2024-01-11T09:43:44.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.309+0000 7f7dfa1ff700 1 -- 172.21.15.27:0/1402668705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df40c5e40 con 0x7f7df4069b60 2024-01-11T09:43:44.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.310+0000 7f7dfa1ff700 1 -- 172.21.15.27:0/1402668705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df40c60f0 con 0x7f7df4069b60 2024-01-11T09:43:44.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.311+0000 7f7ddb7fe700 1 -- 172.21.15.27:0/1402668705 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7de8011420 con 0x7f7df4069b60 2024-01-11T09:43:44.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.312+0000 7f7ddb7fe700 1 --2- 172.21.15.27:0/1402668705 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7de0066f30 0x7f7de00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:44.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.312+0000 7f7ddb7fe700 1 -- 172.21.15.27:0/1402668705 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f7de8086420 con 0x7f7df4069b60 2024-01-11T09:43:44.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.312+0000 7f7df37fe700 1 --2- 172.21.15.27:0/1402668705 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7de0066f30 0x7f7de00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:44.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.313+0000 7f7df37fe700 1 --2- 172.21.15.27:0/1402668705 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7de0066f30 0x7f7de00693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7de4003a80 tx=0x7f7de4007f10 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:44.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.316+0000 7f7ddb7fe700 1 -- 172.21.15.27:0/1402668705 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7de8053620 con 0x7f7df4069b60 2024-01-11T09:43:44.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:44.562+0000 7f7dfa1ff700 1 -- 172.21.15.27:0/1402668705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7df404c3c0 con 0x7f7df4069b60 2024-01-11T09:43:45.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.085+0000 7f7ddb7fe700 1 -- 172.21.15.27:0/1402668705 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v102) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7de8056c40 con 0x7f7df4069b60 2024-01-11T09:43:45.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.088+0000 7f7dfa1ff700 1 -- 172.21.15.27:0/1402668705 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7de0066f30 msgr2=0x7f7de00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:45.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.088+0000 7f7dfa1ff700 1 --2- 172.21.15.27:0/1402668705 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7de0066f30 0x7f7de00693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7de4003a80 tx=0x7f7de4007f10 comp rx=0 tx=0).stop 2024-01-11T09:43:45.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.088+0000 7f7dfa1ff700 1 -- 172.21.15.27:0/1402668705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df4069b60 msgr2=0x7f7df40b11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:45.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.088+0000 7f7dfa1ff700 1 --2- 172.21.15.27:0/1402668705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df4069b60 0x7f7df40b11c0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f7de8006fd0 tx=0x7f7de8017000 comp rx=0 tx=0).stop 2024-01-11T09:43:45.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.088+0000 7f7dfa1ff700 1 -- 172.21.15.27:0/1402668705 shutdown_connections 2024-01-11T09:43:45.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.088+0000 7f7dfa1ff700 1 --2- 172.21.15.27:0/1402668705 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7df415d6d0 0x7f7df40b1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.088+0000 7f7dfa1ff700 1 --2- 172.21.15.27:0/1402668705 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7de0066f30 0x7f7de00693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.088+0000 7f7dfa1ff700 1 --2- 172.21.15.27:0/1402668705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df4069790 0x7f7df41440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.088+0000 7f7dfa1ff700 1 --2- 172.21.15.27:0/1402668705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df4069b60 0x7f7df40b11c0 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.088+0000 7f7dfa1ff700 1 -- 172.21.15.27:0/1402668705 >> 172.21.15.27:0/1402668705 conn(0x7f7df40680a0 msgr2=0x7f7df40c28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:45.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.088+0000 7f7dfa1ff700 1 -- 172.21.15.27:0/1402668705 shutdown_connections 2024-01-11T09:43:45.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.088+0000 7f7dfa1ff700 1 -- 172.21.15.27:0/1402668705 wait complete. 2024-01-11T09:43:45.100 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:43:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.309+0000 7f0ac03ff700 1 Processor -- start 2024-01-11T09:43:45.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.310+0000 7f0ac03ff700 1 -- start start 2024-01-11T09:43:45.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.311+0000 7f0ac03ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0ab80b2b90 0x7f0ab8159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:45.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.311+0000 7f0ac03ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0ab80b07b0 0x7f0ab8159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:45.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.311+0000 7f0ac03ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab80cd730 con 0x7f0ab80b03e0 2024-01-11T09:43:45.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.311+0000 7f0ac03ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab80ccb20 con 0x7f0ab80b2b90 2024-01-11T09:43:45.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.311+0000 7f0ac03ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab80cce00 con 0x7f0ab80b07b0 2024-01-11T09:43:45.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.311+0000 7f0abd99a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0ab80b2b90 0x7f0ab8159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:45.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.311+0000 7f0abd99a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0ab80b2b90 0x7f0ab8159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57370/0 (socket says 172.21.15.27:57370) 2024-01-11T09:43:45.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.311+0000 7f0abd99a700 1 -- 172.21.15.27:0/3743476626 learned_addr learned my addr 172.21.15.27:0/3743476626 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:45.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.311+0000 7f0abe99c700 1 --2- 172.21.15.27:0/3743476626 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0ab80b07b0 0x7f0ab8159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:45.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.312+0000 7f0abd99a700 1 -- 172.21.15.27:0/3743476626 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0ab80b07b0 msgr2=0x7f0ab8159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:45.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.312+0000 7f0abd99a700 1 --2- 172.21.15.27:0/3743476626 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0ab80b07b0 0x7f0ab8159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.312+0000 7f0abd99a700 1 -- 172.21.15.27:0/3743476626 >> v1:172.21.15.77:6789/0 conn(0x7f0ab80b03e0 legacy=0x7f0ab80b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:45.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.312+0000 7f0abd99a700 1 -- 172.21.15.27:0/3743476626 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab80cd0e0 con 0x7f0ab80b2b90 2024-01-11T09:43:45.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.312+0000 7f0abd99a700 1 --2- 172.21.15.27:0/3743476626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ab80b2b90 0x7f0ab8159200 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f0aa80099b0 tx=0x7f0aa8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf8c90ffccd7bbac server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:45.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.312+0000 7f0abd199700 1 -- 172.21.15.27:0/3743476626 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa800b860 con 0x7f0ab80b2b90 2024-01-11T09:43:45.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.312+0000 7f0abd199700 1 -- 172.21.15.27:0/3743476626 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa800b9c0 con 0x7f0ab80b2b90 2024-01-11T09:43:45.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.312+0000 7f0abd199700 1 -- 172.21.15.27:0/3743476626 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa800bd60 con 0x7f0ab80b2b90 2024-01-11T09:43:45.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.313+0000 7f0ac03ff700 1 -- 172.21.15.27:0/3743476626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ab80b2b90 msgr2=0x7f0ab8159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:45.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.313+0000 7f0ac03ff700 1 --2- 172.21.15.27:0/3743476626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ab80b2b90 0x7f0ab8159200 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f0aa80099b0 tx=0x7f0aa8009250 comp rx=0 tx=0).stop 2024-01-11T09:43:45.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.313+0000 7f0ac03ff700 1 -- 172.21.15.27:0/3743476626 shutdown_connections 2024-01-11T09:43:45.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.313+0000 7f0ac03ff700 1 --2- 172.21.15.27:0/3743476626 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0ab80b07b0 0x7f0ab8159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.313+0000 7f0ac03ff700 1 --2- 172.21.15.27:0/3743476626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ab80b2b90 0x7f0ab8159200 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.313+0000 7f0ac03ff700 1 -- 172.21.15.27:0/3743476626 >> 172.21.15.27:0/3743476626 conn(0x7f0ab806ed40 msgr2=0x7f0ab806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:45.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.313+0000 7f0ac03ff700 1 -- 172.21.15.27:0/3743476626 shutdown_connections 2024-01-11T09:43:45.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.313+0000 7f0ac03ff700 1 -- 172.21.15.27:0/3743476626 wait complete. 2024-01-11T09:43:45.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.314+0000 7f0ac03ff700 1 Processor -- start 2024-01-11T09:43:45.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.315+0000 7f0ac03ff700 1 -- start start 2024-01-11T09:43:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.315+0000 7f0ac03ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0ab80b03e0 0x7f0ab80c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.316+0000 7f0ac03ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ab80b07b0 0x7f0ab80c8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.316+0000 7f0ac03ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ab80b2b90 0x7f0ab8169d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.316+0000 7f0ac03ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab80c68e0 con 0x7f0ab80b07b0 2024-01-11T09:43:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.316+0000 7f0ac03ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab80c5300 con 0x7f0ab80b2b90 2024-01-11T09:43:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.316+0000 7f0ac03ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab80c55e0 con 0x7f0ab80b03e0 2024-01-11T09:43:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.316+0000 7f0abd99a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ab80b07b0 0x7f0ab80c8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.316+0000 7f0abd99a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ab80b07b0 0x7f0ab80c8470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57372/0 (socket says 172.21.15.27:57372) 2024-01-11T09:43:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.316+0000 7f0abd99a700 1 -- 172.21.15.27:0/2919589159 learned_addr learned my addr 172.21.15.27:0/2919589159 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:45.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.316+0000 7f0abe99c700 1 --2- 172.21.15.27:0/2919589159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ab80b2b90 0x7f0ab8169d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.316+0000 7f0abe19b700 1 --2- 172.21.15.27:0/2919589159 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0ab80b03e0 0x7f0ab80c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.316+0000 7f0abd99a700 1 -- 172.21.15.27:0/2919589159 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0ab80b03e0 msgr2=0x7f0ab80c63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.316+0000 7f0abd99a700 1 --2- 172.21.15.27:0/2919589159 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0ab80b03e0 0x7f0ab80c63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.316+0000 7f0abd99a700 1 -- 172.21.15.27:0/2919589159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ab80b2b90 msgr2=0x7f0ab8169d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.316+0000 7f0abd99a700 1 --2- 172.21.15.27:0/2919589159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ab80b2b90 0x7f0ab8169d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.316+0000 7f0abd99a700 1 -- 172.21.15.27:0/2919589159 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa80094f0 con 0x7f0ab80b07b0 2024-01-11T09:43:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.317+0000 7f0abd99a700 1 --2- 172.21.15.27:0/2919589159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ab80b07b0 0x7f0ab80c8470 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8069540 tx=0x7f0aa8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.317+0000 7f0aae7fc700 1 -- 172.21.15.27:0/2919589159 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8011e40 con 0x7f0ab80b07b0 2024-01-11T09:43:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.317+0000 7f0aae7fc700 1 -- 172.21.15.27:0/2919589159 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa8024440 con 0x7f0ab80b07b0 2024-01-11T09:43:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.317+0000 7f0aae7fc700 1 -- 172.21.15.27:0/2919589159 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8024800 con 0x7f0ab80b07b0 2024-01-11T09:43:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.317+0000 7f0ac03ff700 1 -- 172.21.15.27:0/2919589159 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab80c58c0 con 0x7f0ab80b07b0 2024-01-11T09:43:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.317+0000 7f0ac03ff700 1 -- 172.21.15.27:0/2919589159 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab80c5e80 con 0x7f0ab80b07b0 2024-01-11T09:43:45.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.318+0000 7f0ac03ff700 1 -- 172.21.15.27:0/2919589159 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab80d11f0 con 0x7f0ab80b07b0 2024-01-11T09:43:45.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.319+0000 7f0aae7fc700 1 -- 172.21.15.27:0/2919589159 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0aa801a420 con 0x7f0ab80b07b0 2024-01-11T09:43:45.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.320+0000 7f0aae7fc700 1 --2- 172.21.15.27:0/2919589159 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0a9c066f30 0x7f0a9c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:45.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.321+0000 7f0aae7fc700 1 -- 172.21.15.27:0/2919589159 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f0aa8086360 con 0x7f0ab80b07b0 2024-01-11T09:43:45.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.321+0000 7f0abe19b700 1 --2- 172.21.15.27:0/2919589159 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0a9c066f30 0x7f0a9c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:45.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.322+0000 7f0abe19b700 1 --2- 172.21.15.27:0/2919589159 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0a9c066f30 0x7f0a9c0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4003a80 tx=0x7f0ab4007ec0 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:45.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.325+0000 7f0aae7fc700 1 -- 172.21.15.27:0/2919589159 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0aa80534e0 con 0x7f0ab80b07b0 2024-01-11T09:43:45.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.571+0000 7f0ac03ff700 1 -- 172.21.15.27:0/2919589159 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ab80718d0 con 0x7f0ab80b07b0 2024-01-11T09:43:45.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.571+0000 7f0aae7fc700 1 -- 172.21.15.27:0/2919589159 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+4603 (secure 0 0 0) 0x7f0aa8056b00 con 0x7f0ab80b07b0 2024-01-11T09:43:45.563 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:43:45.563 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":102,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8441,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/991606883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":991606883},{"type":"v1","addr":"172.21.15.77:6834","nonce":991606883}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1996818074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1996818074},{"type":"v1","addr":"172.21.15.77:6835","nonce":1996818074}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9742,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4117588854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":4117588854},{"type":"v1","addr":"172.21.15.27:6835","nonce":4117588854}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:43:41.554157+0000","modified":"2024-01-11T09:43:45.078369+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10113":{"gid":10113,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/1557605486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1557605486},{"type":"v1","addr":"172.21.15.27:6834","nonce":1557605486}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2024-01-11T09:43:45.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.573+0000 7f0ac03ff700 1 -- 172.21.15.27:0/2919589159 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0a9c066f30 msgr2=0x7f0a9c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.573+0000 7f0ac03ff700 1 --2- 172.21.15.27:0/2919589159 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0a9c066f30 0x7f0a9c0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4003a80 tx=0x7f0ab4007ec0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.573+0000 7f0ac03ff700 1 -- 172.21.15.27:0/2919589159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ab80b07b0 msgr2=0x7f0ab80c8470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.573+0000 7f0ac03ff700 1 --2- 172.21.15.27:0/2919589159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ab80b07b0 0x7f0ab80c8470 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8069540 tx=0x7f0aa8017000 comp rx=0 tx=0).stop 2024-01-11T09:43:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.573+0000 7f0ac03ff700 1 -- 172.21.15.27:0/2919589159 shutdown_connections 2024-01-11T09:43:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.573+0000 7f0ac03ff700 1 --2- 172.21.15.27:0/2919589159 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0a9c066f30 0x7f0a9c0693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.573+0000 7f0ac03ff700 1 --2- 172.21.15.27:0/2919589159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ab80b2b90 0x7f0ab8169d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.573+0000 7f0ac03ff700 1 --2- 172.21.15.27:0/2919589159 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0ab80b03e0 0x7f0ab80c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.573+0000 7f0ac03ff700 1 --2- 172.21.15.27:0/2919589159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ab80b07b0 0x7f0ab80c8470 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.573+0000 7f0ac03ff700 1 -- 172.21.15.27:0/2919589159 >> 172.21.15.27:0/2919589159 conn(0x7f0ab806ed40 msgr2=0x7f0ab80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.574+0000 7f0ac03ff700 1 -- 172.21.15.27:0/2919589159 shutdown_connections 2024-01-11T09:43:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.574+0000 7f0ac03ff700 1 -- 172.21.15.27:0/2919589159 wait complete. 2024-01-11T09:43:45.567 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 102 2024-01-11T09:43:45.584 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:43:45.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.792+0000 7fc728fff700 1 Processor -- start 2024-01-11T09:43:45.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.792+0000 7fc728fff700 1 -- start start 2024-01-11T09:43:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.794+0000 7fc728fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7240ca3e0 0x7fc7240cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.794+0000 7fc728fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc724159e80 0x7fc72415dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.794+0000 7fc728fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72415d640 con 0x7fc7240ca7b0 2024-01-11T09:43:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.794+0000 7fc728fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72415ae70 con 0x7fc724159e80 2024-01-11T09:43:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.794+0000 7fc728fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7240cc4f0 con 0x7fc7240ca3e0 2024-01-11T09:43:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.794+0000 7fc721d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc7240ca7b0 0x7fc7240c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:32892/0 (socket says 172.21.15.27:32892) 2024-01-11T09:43:45.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.794+0000 7fc721d9b700 1 -- 172.21.15.27:0/1531544787 learned_addr learned my addr 172.21.15.27:0/1531544787 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:45.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.794+0000 7fc72259c700 1 --2- 172.21.15.27:0/1531544787 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7240ca3e0 0x7fc7240cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:45.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.794+0000 7fc722d9d700 1 --2- 172.21.15.27:0/1531544787 >> v2:172.21.15.77:3300/0 conn(0x7fc724159e80 0x7fc72415dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:45.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.794+0000 7fc72159a700 1 -- 172.21.15.27:0/1531544787 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318007268 0 0) 0x7fc72415d640 con 0x7fc7240ca7b0 2024-01-11T09:43:45.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.794+0000 7fc72159a700 1 -- 172.21.15.27:0/1531544787 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc70c003570 con 0x7fc7240ca7b0 2024-01-11T09:43:45.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.795+0000 7fc72159a700 1 -- 172.21.15.27:0/1531544787 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 493675411 0 0) 0x7fc70c003570 con 0x7fc7240ca7b0 2024-01-11T09:43:45.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.795+0000 7fc72159a700 1 -- 172.21.15.27:0/1531544787 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7240ca3e0 msgr2=0x7fc7240cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.795+0000 7fc72159a700 1 --2- 172.21.15.27:0/1531544787 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7240ca3e0 0x7fc7240cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.795+0000 7fc72159a700 1 -- 172.21.15.27:0/1531544787 >> v2:172.21.15.77:3300/0 conn(0x7fc724159e80 msgr2=0x7fc72415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.795+0000 7fc72159a700 1 --2- 172.21.15.27:0/1531544787 >> v2:172.21.15.77:3300/0 conn(0x7fc724159e80 0x7fc72415dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.795+0000 7fc72159a700 1 -- 172.21.15.27:0/1531544787 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72415a5d0 con 0x7fc7240ca7b0 2024-01-11T09:43:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.795+0000 7fc72159a700 1 -- 172.21.15.27:0/1531544787 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fc718001470 con 0x7fc7240ca7b0 2024-01-11T09:43:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.795+0000 7fc72159a700 1 -- 172.21.15.27:0/1531544787 >> v1:172.21.15.27:6789/0 conn(0x7fc7240ca7b0 legacy=0x7fc7240c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.795+0000 7fc72159a700 1 --2- 172.21.15.27:0/1531544787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc70c003ae0 0x7fc70c003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.795+0000 7fc72159a700 1 --2- 172.21.15.27:0/1531544787 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7240ca3e0 0x7fc70c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.795+0000 7fc72159a700 1 --2- 172.21.15.27:0/1531544787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc70c004d50 0x7fc70c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.795+0000 7fc72159a700 1 -- 172.21.15.27:0/1531544787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72415ae70 con 0x7fc70c003ae0 2024-01-11T09:43:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.795+0000 7fc72159a700 1 -- 172.21.15.27:0/1531544787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7240cc4f0 con 0x7fc70c004d50 2024-01-11T09:43:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.795+0000 7fc72159a700 1 -- 172.21.15.27:0/1531544787 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c00bb10 con 0x7fc7240ca3e0 2024-01-11T09:43:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.795+0000 7fc72259c700 1 --2- 172.21.15.27:0/1531544787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc70c003ae0 0x7fc70c003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:45.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.795+0000 7fc721d9b700 1 --2- 172.21.15.27:0/1531544787 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7240ca3e0 0x7fc70c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.795+0000 7fc722d9d700 1 --2- 172.21.15.27:0/1531544787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc70c004d50 0x7fc70c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.796+0000 7fc72259c700 1 -- 172.21.15.27:0/1531544787 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7240ca3e0 msgr2=0x7fc70c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.796+0000 7fc72259c700 1 --2- 172.21.15.27:0/1531544787 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7240ca3e0 0x7fc70c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.796+0000 7fc72259c700 1 -- 172.21.15.27:0/1531544787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc70c004d50 msgr2=0x7fc70c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.796+0000 7fc72259c700 1 --2- 172.21.15.27:0/1531544787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc70c004d50 0x7fc70c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.796+0000 7fc72259c700 1 -- 172.21.15.27:0/1531544787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c003570 con 0x7fc70c003ae0 2024-01-11T09:43:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.796+0000 7fc72259c700 1 --2- 172.21.15.27:0/1531544787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc70c003ae0 0x7fc70c003ef0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fc714009c30 tx=0x7fc7140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66cebe698334b8b2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.796+0000 7fc72159a700 1 -- 172.21.15.27:0/1531544787 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71400e710 con 0x7fc70c003ae0 2024-01-11T09:43:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.796+0000 7fc72159a700 1 -- 172.21.15.27:0/1531544787 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71400e870 con 0x7fc70c003ae0 2024-01-11T09:43:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.796+0000 7fc72159a700 1 -- 172.21.15.27:0/1531544787 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71400ec30 con 0x7fc70c003ae0 2024-01-11T09:43:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.797+0000 7fc728fff700 1 -- 172.21.15.27:0/1531544787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc70c003ae0 msgr2=0x7fc70c003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.797+0000 7fc728fff700 1 --2- 172.21.15.27:0/1531544787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc70c003ae0 0x7fc70c003ef0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fc714009c30 tx=0x7fc7140091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.797+0000 7fc722d9d700 1 -- 172.21.15.27:0/1531544787 reap_dead start 2024-01-11T09:43:45.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.797+0000 7fc728fff700 1 -- 172.21.15.27:0/1531544787 shutdown_connections 2024-01-11T09:43:45.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.797+0000 7fc728fff700 1 -- 172.21.15.27:0/1531544787 >> 172.21.15.27:0/1531544787 conn(0x7fc7240ce3e0 msgr2=0x7fc7240ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:45.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.797+0000 7fc728fff700 1 -- 172.21.15.27:0/1531544787 shutdown_connections 2024-01-11T09:43:45.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.797+0000 7fc728fff700 1 -- 172.21.15.27:0/1531544787 wait complete. 2024-01-11T09:43:45.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.798+0000 7fc728fff700 1 Processor -- start 2024-01-11T09:43:45.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.799+0000 7fc728fff700 1 -- start start 2024-01-11T09:43:45.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.799+0000 7fc728fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc70c003ae0 0x7fc72416dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:45.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.799+0000 7fc728fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc724169da0 0x7fc7241568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:45.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.799+0000 7fc728fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7240b1820 0x7fc7240b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.799+0000 7fc728fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72416e2a0 con 0x7fc7240b1820 2024-01-11T09:43:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.799+0000 7fc728fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72414ea90 con 0x7fc70c003ae0 2024-01-11T09:43:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.799+0000 7fc728fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72416a510 con 0x7fc724169da0 2024-01-11T09:43:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.800+0000 7fc722d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7240b1820 0x7fc7240b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.800+0000 7fc72259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc70c003ae0 0x7fc72416dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.800+0000 7fc72259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc70c003ae0 0x7fc72416dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:58564/0 (socket says 172.21.15.27:58564) 2024-01-11T09:43:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.800+0000 7fc72259c700 1 -- 172.21.15.27:0/747526185 learned_addr learned my addr 172.21.15.27:0/747526185 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.800+0000 7fc722d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7240b1820 0x7fc7240b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57384/0 (socket says 172.21.15.27:57384) 2024-01-11T09:43:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.800+0000 7fc721d9b700 1 --2- 172.21.15.27:0/747526185 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc724169da0 0x7fc7241568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.800+0000 7fc72259c700 1 -- 172.21.15.27:0/747526185 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc724169da0 msgr2=0x7fc7241568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.800+0000 7fc72259c700 1 --2- 172.21.15.27:0/747526185 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc724169da0 0x7fc7241568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.800+0000 7fc72259c700 1 -- 172.21.15.27:0/747526185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7240b1820 msgr2=0x7fc7240b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.800+0000 7fc72259c700 1 --2- 172.21.15.27:0/747526185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7240b1820 0x7fc7240b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:45.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.800+0000 7fc72259c700 1 -- 172.21.15.27:0/747526185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72416a7f0 con 0x7fc70c003ae0 2024-01-11T09:43:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.800+0000 7fc722d9d700 1 --2- 172.21.15.27:0/747526185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7240b1820 0x7fc7240b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:43:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.800+0000 7fc721d9b700 1 --2- 172.21.15.27:0/747526185 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc724169da0 0x7fc7241568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:43:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.801+0000 7fc72259c700 1 --2- 172.21.15.27:0/747526185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc70c003ae0 0x7fc72416dd60 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc71400e4b0 tx=0x7fc71401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.801+0000 7fc7127fc700 1 -- 172.21.15.27:0/747526185 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc714022070 con 0x7fc70c003ae0 2024-01-11T09:43:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.801+0000 7fc7127fc700 1 -- 172.21.15.27:0/747526185 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71400e9e0 con 0x7fc70c003ae0 2024-01-11T09:43:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.801+0000 7fc7127fc700 1 -- 172.21.15.27:0/747526185 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71401c680 con 0x7fc70c003ae0 2024-01-11T09:43:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.801+0000 7fc728fff700 1 -- 172.21.15.27:0/747526185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72416aad0 con 0x7fc70c003ae0 2024-01-11T09:43:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.801+0000 7fc728fff700 1 -- 172.21.15.27:0/747526185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72416a210 con 0x7fc70c003ae0 2024-01-11T09:43:45.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.802+0000 7fc728fff700 1 -- 172.21.15.27:0/747526185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc72416ad10 con 0x7fc70c003ae0 2024-01-11T09:43:45.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.803+0000 7fc7127fc700 1 -- 172.21.15.27:0/747526185 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc71401cab0 con 0x7fc70c003ae0 2024-01-11T09:43:45.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.804+0000 7fc7127fc700 1 --2- 172.21.15.27:0/747526185 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc704066f30 0x7fc7040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:45.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.804+0000 7fc7127fc700 1 -- 172.21.15.27:0/747526185 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5789+0+0 (secure 0 0 0) 0x7fc7140861b0 con 0x7fc70c003ae0 2024-01-11T09:43:45.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.804+0000 7fc721d9b700 1 --2- 172.21.15.27:0/747526185 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc704066f30 0x7fc7040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:45.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.805+0000 7fc721d9b700 1 --2- 172.21.15.27:0/747526185 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc704066f30 0x7fc7040693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc718001120 tx=0x7fc718001dc0 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:45.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:45.808+0000 7fc7127fc700 1 -- 172.21.15.27:0/747526185 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc7140533b0 con 0x7fc70c003ae0 2024-01-11T09:43:46.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.013+0000 7fc728fff700 1 -- 172.21.15.27:0/747526185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7240718d0 con 0x7fc70c003ae0 2024-01-11T09:43:46.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.015+0000 7fc7127fc700 1 -- 172.21.15.27:0/747526185 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v198) v1 ==== 74+0+15450 (secure 0 0 0) 0x7fc714017090 con 0x7fc70c003ae0 2024-01-11T09:43:46.007 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:43:46.007 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":198,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:43:43.064774+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:43:39.267819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-01-11T09:43:40.563877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:43:46.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.017+0000 7fc728fff700 1 -- 172.21.15.27:0/747526185 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc704066f30 msgr2=0x7fc7040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:46.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.017+0000 7fc728fff700 1 --2- 172.21.15.27:0/747526185 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc704066f30 0x7fc7040693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc718001120 tx=0x7fc718001dc0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.017+0000 7fc728fff700 1 -- 172.21.15.27:0/747526185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc70c003ae0 msgr2=0x7fc72416dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:46.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.017+0000 7fc728fff700 1 --2- 172.21.15.27:0/747526185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc70c003ae0 0x7fc72416dd60 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc71400e4b0 tx=0x7fc71401a000 comp rx=0 tx=0).stop 2024-01-11T09:43:46.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.017+0000 7fc728fff700 1 -- 172.21.15.27:0/747526185 shutdown_connections 2024-01-11T09:43:46.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.017+0000 7fc728fff700 1 --2- 172.21.15.27:0/747526185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7240b1820 0x7fc7240b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.017+0000 7fc728fff700 1 --2- 172.21.15.27:0/747526185 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc704066f30 0x7fc7040693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.017+0000 7fc728fff700 1 --2- 172.21.15.27:0/747526185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc70c003ae0 0x7fc72416dd60 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.017+0000 7fc728fff700 1 --2- 172.21.15.27:0/747526185 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc724169da0 0x7fc7241568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.017+0000 7fc728fff700 1 -- 172.21.15.27:0/747526185 >> 172.21.15.27:0/747526185 conn(0x7fc7240ce3e0 msgr2=0x7fc724069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:46.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.017+0000 7fc728fff700 1 -- 172.21.15.27:0/747526185 shutdown_connections 2024-01-11T09:43:46.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.017+0000 7fc728fff700 1 -- 172.21.15.27:0/747526185 wait complete. 2024-01-11T09:43:46.024 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-01-11T09:43:46.032+0000 7f2bbc9f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T09:43:46.027 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-11T09:43:46.027 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:43:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.237+0000 7f4e0b9ff700 1 Processor -- start 2024-01-11T09:43:46.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.237+0000 7f4e0b9ff700 1 -- start start 2024-01-11T09:43:46.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.239+0000 7f4e0b9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e040b03e0 0x7f4e040b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:46.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.239+0000 7f4e0b9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4e040b07b0 0x7f4e040b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:46.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.239+0000 7f4e0b9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4e04159200 0x7f4e04159640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:46.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.239+0000 7f4e0b9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e040cd730 con 0x7f4e04159200 2024-01-11T09:43:46.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.239+0000 7f4e0b9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e040ccb20 con 0x7f4e040b07b0 2024-01-11T09:43:46.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.239+0000 7f4e0b9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e040cce00 con 0x7f4e040b03e0 2024-01-11T09:43:46.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.239+0000 7f4e09f9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4e04159200 0x7f4e04159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:46.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.239+0000 7f4e09f9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4e04159200 0x7f4e04159640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57390/0 (socket says 172.21.15.27:57390) 2024-01-11T09:43:46.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.239+0000 7f4e09f9c700 1 -- 172.21.15.27:0/399040477 learned_addr learned my addr 172.21.15.27:0/399040477 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:46.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.239+0000 7f4e08f9a700 1 --2- 172.21.15.27:0/399040477 >> v2:172.21.15.77:3300/0 conn(0x7f4e040b07b0 0x7f4e040b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:46.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.239+0000 7f4e0979b700 1 --2- 172.21.15.27:0/399040477 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e040b03e0 0x7f4e040b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:46.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.240+0000 7f4e09f9c700 1 -- 172.21.15.27:0/399040477 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e040b03e0 msgr2=0x7f4e040b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:46.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.240+0000 7f4e09f9c700 1 --2- 172.21.15.27:0/399040477 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e040b03e0 0x7f4e040b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.240+0000 7f4e09f9c700 1 -- 172.21.15.27:0/399040477 >> v2:172.21.15.77:3300/0 conn(0x7f4e040b07b0 msgr2=0x7f4e040b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:46.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.240+0000 7f4e09f9c700 1 --2- 172.21.15.27:0/399040477 >> v2:172.21.15.77:3300/0 conn(0x7f4e040b07b0 0x7f4e040b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.240+0000 7f4e09f9c700 1 -- 172.21.15.27:0/399040477 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e040cd0e0 con 0x7f4e04159200 2024-01-11T09:43:46.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.240+0000 7f4e09f9c700 1 --2- 172.21.15.27:0/399040477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e04159200 0x7f4e04159640 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f4e040893d0 tx=0x7f4dfc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d311c02d187bfd00 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:46.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.240+0000 7f4dfbfff700 1 -- 172.21.15.27:0/399040477 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc00e070 con 0x7f4e04159200 2024-01-11T09:43:46.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.241+0000 7f4dfbfff700 1 -- 172.21.15.27:0/399040477 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dfc00b970 con 0x7f4e04159200 2024-01-11T09:43:46.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.241+0000 7f4dfbfff700 1 -- 172.21.15.27:0/399040477 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc0092e0 con 0x7f4e04159200 2024-01-11T09:43:46.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.241+0000 7f4e0b9ff700 1 -- 172.21.15.27:0/399040477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e04159200 msgr2=0x7f4e04159640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:46.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.241+0000 7f4e0b9ff700 1 --2- 172.21.15.27:0/399040477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e04159200 0x7f4e04159640 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f4e040893d0 tx=0x7f4dfc009840 comp rx=0 tx=0).stop 2024-01-11T09:43:46.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.241+0000 7f4e0b9ff700 1 -- 172.21.15.27:0/399040477 shutdown_connections 2024-01-11T09:43:46.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.241+0000 7f4e0b9ff700 1 --2- 172.21.15.27:0/399040477 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e040b03e0 0x7f4e040b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.241+0000 7f4e0b9ff700 1 --2- 172.21.15.27:0/399040477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e04159200 0x7f4e04159640 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.241+0000 7f4e0b9ff700 1 --2- 172.21.15.27:0/399040477 >> v2:172.21.15.77:3300/0 conn(0x7f4e040b07b0 0x7f4e040b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.241+0000 7f4e0b9ff700 1 -- 172.21.15.27:0/399040477 >> 172.21.15.27:0/399040477 conn(0x7f4e0406ed40 msgr2=0x7f4e0406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:46.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.241+0000 7f4e0b9ff700 1 -- 172.21.15.27:0/399040477 shutdown_connections 2024-01-11T09:43:46.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.241+0000 7f4e0b9ff700 1 -- 172.21.15.27:0/399040477 wait complete. 2024-01-11T09:43:46.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.243+0000 7f4e0b9ff700 1 Processor -- start 2024-01-11T09:43:46.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.243+0000 7f4e0b9ff700 1 -- start start 2024-01-11T09:43:46.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.244+0000 7f4e0b9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e040b03e0 0x7f4e040cefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:46.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.244+0000 7f4e0b9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e040b07b0 0x7f4e040c5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:46.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.244+0000 7f4e0b9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e04159200 0x7f4e040c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:46.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.244+0000 7f4e0b9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e040c7cc0 con 0x7f4e04159200 2024-01-11T09:43:46.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.244+0000 7f4e0b9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e040cff00 con 0x7f4e040b07b0 2024-01-11T09:43:46.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.244+0000 7f4e0b9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e040c7f80 con 0x7f4e040b03e0 2024-01-11T09:43:46.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.244+0000 7f4e09f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e04159200 0x7f4e040c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:46.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.244+0000 7f4e09f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e04159200 0x7f4e040c9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57402/0 (socket says 172.21.15.27:57402) 2024-01-11T09:43:46.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.244+0000 7f4e09f9c700 1 -- 172.21.15.27:0/101932175 learned_addr learned my addr 172.21.15.27:0/101932175 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:46.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.244+0000 7f4e0979b700 1 --2- 172.21.15.27:0/101932175 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e040b03e0 0x7f4e040cefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:46.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.244+0000 7f4e08f9a700 1 --2- 172.21.15.27:0/101932175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e040b07b0 0x7f4e040c5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:46.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.245+0000 7f4e0979b700 1 -- 172.21.15.27:0/101932175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e040b07b0 msgr2=0x7f4e040c5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:46.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.245+0000 7f4e0979b700 1 --2- 172.21.15.27:0/101932175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e040b07b0 0x7f4e040c5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.245+0000 7f4e0979b700 1 -- 172.21.15.27:0/101932175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e04159200 msgr2=0x7f4e040c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:46.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.245+0000 7f4e0979b700 1 --2- 172.21.15.27:0/101932175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e04159200 0x7f4e040c9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.245+0000 7f4e0979b700 1 -- 172.21.15.27:0/101932175 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dfc009ae0 con 0x7f4e040b03e0 2024-01-11T09:43:46.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.245+0000 7f4e09f9c700 1 --2- 172.21.15.27:0/101932175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e04159200 0x7f4e040c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:43:46.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.245+0000 7f4e0979b700 1 --2- 172.21.15.27:0/101932175 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e040b03e0 0x7f4e040cefd0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4e0000e850 tx=0x7f4e000091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:46.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.245+0000 7f4df97fa700 1 -- 172.21.15.27:0/101932175 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e00013680 con 0x7f4e040b03e0 2024-01-11T09:43:46.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.245+0000 7f4e0b9ff700 1 -- 172.21.15.27:0/101932175 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e040cf660 con 0x7f4e040b03e0 2024-01-11T09:43:46.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.245+0000 7f4df97fa700 1 -- 172.21.15.27:0/101932175 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e000137e0 con 0x7f4e040b03e0 2024-01-11T09:43:46.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.245+0000 7f4df97fa700 1 -- 172.21.15.27:0/101932175 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e00013ba0 con 0x7f4e040b03e0 2024-01-11T09:43:46.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.245+0000 7f4e0b9ff700 1 -- 172.21.15.27:0/101932175 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e040cfc20 con 0x7f4e040b03e0 2024-01-11T09:43:46.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.246+0000 7f4e0b9ff700 1 -- 172.21.15.27:0/101932175 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e040718d0 con 0x7f4e040b03e0 2024-01-11T09:43:46.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.248+0000 7f4df97fa700 1 -- 172.21.15.27:0/101932175 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4e0001b400 con 0x7f4e040b03e0 2024-01-11T09:43:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.249+0000 7f4df97fa700 1 --2- 172.21.15.27:0/101932175 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4de8066f30 0x7f4de80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.249+0000 7f4df97fa700 1 -- 172.21.15.27:0/101932175 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f4e00084410 con 0x7f4e040b03e0 2024-01-11T09:43:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.249+0000 7f4e08f9a700 1 --2- 172.21.15.27:0/101932175 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4de8066f30 0x7f4de80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.250+0000 7f4e08f9a700 1 --2- 172.21.15.27:0/101932175 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4de8066f30 0x7f4de80693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4e040b5250 tx=0x7f4df40091a0 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:46.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.254+0000 7f4df97fa700 1 -- 172.21.15.27:0/101932175 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4e000550c0 con 0x7f4e040b03e0 2024-01-11T09:43:46.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.450+0000 7f4e0b9ff700 1 -- 172.21.15.27:0/101932175 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4e040c80c0 con 0x7f4de8066f30 2024-01-11T09:43:46.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.462+0000 7f4df97fa700 1 -- 172.21.15.27:0/101932175 <== mgr.8294 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289569 (secure 0 0 0) 0x7f4df4011a50 con 0x7f4de8066f30 2024-01-11T09:43:46.458 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:43:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.466+0000 7f4e0b9ff700 1 -- 172.21.15.27:0/101932175 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4de8066f30 msgr2=0x7f4de80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.466+0000 7f4e0b9ff700 1 --2- 172.21.15.27:0/101932175 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4de8066f30 0x7f4de80693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4e040b5250 tx=0x7f4df40091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.466+0000 7f4e0b9ff700 1 -- 172.21.15.27:0/101932175 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e040b03e0 msgr2=0x7f4e040cefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.466+0000 7f4e0b9ff700 1 --2- 172.21.15.27:0/101932175 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e040b03e0 0x7f4e040cefd0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4e0000e850 tx=0x7f4e000091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.466+0000 7f4e0b9ff700 1 -- 172.21.15.27:0/101932175 shutdown_connections 2024-01-11T09:43:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.466+0000 7f4e0b9ff700 1 --2- 172.21.15.27:0/101932175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e04159200 0x7f4e040c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.466+0000 7f4e0b9ff700 1 --2- 172.21.15.27:0/101932175 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4e040b03e0 0x7f4e040cefd0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.466+0000 7f4e0b9ff700 1 --2- 172.21.15.27:0/101932175 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4de8066f30 0x7f4de80693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.466+0000 7f4e0b9ff700 1 --2- 172.21.15.27:0/101932175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e040b07b0 0x7f4e040c5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.466+0000 7f4e0b9ff700 1 -- 172.21.15.27:0/101932175 >> 172.21.15.27:0/101932175 conn(0x7f4e0406ed40 msgr2=0x7f4e040b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.466+0000 7f4e0b9ff700 1 -- 172.21.15.27:0/101932175 shutdown_connections 2024-01-11T09:43:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.466+0000 7f4e0b9ff700 1 -- 172.21.15.27:0/101932175 wait complete. 2024-01-11T09:43:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:43:46.476 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-11T09:43:45.298944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2096,"num_read_kb":4396,"num_write":615,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":382,"ondisk_log_size":382,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":78,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2460264,"kb_used_data":20520,"kb_used_omap":0,"kb_used_meta":2439680,"kb_avail":752514456,"statfs":{"total":773094113280,"available":770574802944,"internally_reserved":0,"allocated":21012480,"data_stored":10588404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2498232320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.254600"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090312+0000","last_change":"2024-01-11T09:43:42.090312+0000","last_active":"2024-01-11T09:43:42.090312+0000","last_peered":"2024-01-11T09:43:42.090312+0000","last_clean":"2024-01-11T09:43:42.090312+0000","last_became_active":"2024-01-11T09:43:42.089816+0000","last_became_peered":"2024-01-11T09:43:42.089816+0000","last_unstale":"2024-01-11T09:43:42.090312+0000","last_undegraded":"2024-01-11T09:43:42.090312+0000","last_fullsized":"2024-01-11T09:43:42.090312+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:18:25.046984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.089184+0000","last_change":"2024-01-11T09:43:42.089184+0000","last_active":"2024-01-11T09:43:42.089184+0000","last_peered":"2024-01-11T09:43:42.089184+0000","last_clean":"2024-01-11T09:43:42.089184+0000","last_became_active":"2024-01-11T09:43:42.088486+0000","last_became_peered":"2024-01-11T09:43:42.088486+0000","last_unstale":"2024-01-11T09:43:42.089184+0000","last_undegraded":"2024-01-11T09:43:42.089184+0000","last_fullsized":"2024-01-11T09:43:42.089184+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:01:29.860733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.086612+0000","last_change":"2024-01-11T09:43:42.086612+0000","last_active":"2024-01-11T09:43:42.086612+0000","last_peered":"2024-01-11T09:43:42.086612+0000","last_clean":"2024-01-11T09:43:42.086612+0000","last_became_active":"2024-01-11T09:43:42.086316+0000","last_became_peered":"2024-01-11T09:43:42.086316+0000","last_unstale":"2024-01-11T09:43:42.086612+0000","last_undegraded":"2024-01-11T09:43:42.086612+0000","last_fullsized":"2024-01-11T09:43:42.086612+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:00:48.167638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.076216+0000","last_change":"2024-01-11T09:43:42.076216+0000","last_active":"2024-01-11T09:43:42.076216+0000","last_peered":"2024-01-11T09:43:42.076216+0000","last_clean":"2024-01-11T09:43:42.076216+0000","last_became_active":"2024-01-11T09:43:42.074166+0000","last_became_peered":"2024-01-11T09:43:42.074166+0000","last_unstale":"2024-01-11T09:43:42.076216+0000","last_undegraded":"2024-01-11T09:43:42.076216+0000","last_fullsized":"2024-01-11T09:43:42.076216+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:48:40.643154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.077822+0000","last_change":"2024-01-11T09:43:42.077822+0000","last_active":"2024-01-11T09:43:42.077822+0000","last_peered":"2024-01-11T09:43:42.077822+0000","last_clean":"2024-01-11T09:43:42.077822+0000","last_became_active":"2024-01-11T09:43:42.077370+0000","last_became_peered":"2024-01-11T09:43:42.077370+0000","last_unstale":"2024-01-11T09:43:42.077822+0000","last_undegraded":"2024-01-11T09:43:42.077822+0000","last_fullsized":"2024-01-11T09:43:42.077822+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:27:02.922803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.078570+0000","last_change":"2024-01-11T09:43:42.078570+0000","last_active":"2024-01-11T09:43:42.078570+0000","last_peered":"2024-01-11T09:43:42.078570+0000","last_clean":"2024-01-11T09:43:42.078570+0000","last_became_active":"2024-01-11T09:43:42.074654+0000","last_became_peered":"2024-01-11T09:43:42.074654+0000","last_unstale":"2024-01-11T09:43:42.078570+0000","last_undegraded":"2024-01-11T09:43:42.078570+0000","last_fullsized":"2024-01-11T09:43:42.078570+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:48:31.109585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090878+0000","last_change":"2024-01-11T09:43:42.087246+0000","last_active":"2024-01-11T09:43:42.090878+0000","last_peered":"2024-01-11T09:43:42.090878+0000","last_clean":"2024-01-11T09:43:42.090878+0000","last_became_active":"2024-01-11T09:43:42.086982+0000","last_became_peered":"2024-01-11T09:43:42.086982+0000","last_unstale":"2024-01-11T09:43:42.090878+0000","last_undegraded":"2024-01-11T09:43:42.090878+0000","last_fullsized":"2024-01-11T09:43:42.090878+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:28:58.773625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090849+0000","last_change":"2024-01-11T09:43:42.081646+0000","last_active":"2024-01-11T09:43:42.090849+0000","last_peered":"2024-01-11T09:43:42.090849+0000","last_clean":"2024-01-11T09:43:42.090849+0000","last_became_active":"2024-01-11T09:43:42.081233+0000","last_became_peered":"2024-01-11T09:43:42.081233+0000","last_unstale":"2024-01-11T09:43:42.090849+0000","last_undegraded":"2024-01-11T09:43:42.090849+0000","last_fullsized":"2024-01-11T09:43:42.090849+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:47.767999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.080980+0000","last_change":"2024-01-11T09:43:42.079368+0000","last_active":"2024-01-11T09:43:42.080980+0000","last_peered":"2024-01-11T09:43:42.080980+0000","last_clean":"2024-01-11T09:43:42.080980+0000","last_became_active":"2024-01-11T09:43:42.077324+0000","last_became_peered":"2024-01-11T09:43:42.077324+0000","last_unstale":"2024-01-11T09:43:42.080980+0000","last_undegraded":"2024-01-11T09:43:42.080980+0000","last_fullsized":"2024-01-11T09:43:42.080980+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:10:05.795665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.085618+0000","last_change":"2024-01-11T09:43:42.085618+0000","last_active":"2024-01-11T09:43:42.085618+0000","last_peered":"2024-01-11T09:43:42.085618+0000","last_clean":"2024-01-11T09:43:42.085618+0000","last_became_active":"2024-01-11T09:43:42.085261+0000","last_became_peered":"2024-01-11T09:43:42.085261+0000","last_unstale":"2024-01-11T09:43:42.085618+0000","last_undegraded":"2024-01-11T09:43:42.085618+0000","last_fullsized":"2024-01-11T09:43:42.085618+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:27:15.928486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.087295+0000","last_change":"2024-01-11T09:43:42.074985+0000","last_active":"2024-01-11T09:43:42.087295+0000","last_peered":"2024-01-11T09:43:42.087295+0000","last_clean":"2024-01-11T09:43:42.087295+0000","last_became_active":"2024-01-11T09:43:42.074654+0000","last_became_peered":"2024-01-11T09:43:42.074654+0000","last_unstale":"2024-01-11T09:43:42.087295+0000","last_undegraded":"2024-01-11T09:43:42.087295+0000","last_fullsized":"2024-01-11T09:43:42.087295+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:35:36.079408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.081077+0000","last_change":"2024-01-11T09:43:42.081077+0000","last_active":"2024-01-11T09:43:42.081077+0000","last_peered":"2024-01-11T09:43:42.081077+0000","last_clean":"2024-01-11T09:43:42.081077+0000","last_became_active":"2024-01-11T09:43:42.075921+0000","last_became_peered":"2024-01-11T09:43:42.075921+0000","last_unstale":"2024-01-11T09:43:42.081077+0000","last_undegraded":"2024-01-11T09:43:42.081077+0000","last_fullsized":"2024-01-11T09:43:42.081077+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:47.767999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.079166+0000","last_change":"2024-01-11T09:43:42.079166+0000","last_active":"2024-01-11T09:43:42.079166+0000","last_peered":"2024-01-11T09:43:42.079166+0000","last_clean":"2024-01-11T09:43:42.079166+0000","last_became_active":"2024-01-11T09:43:42.078888+0000","last_became_peered":"2024-01-11T09:43:42.078888+0000","last_unstale":"2024-01-11T09:43:42.079166+0000","last_undegraded":"2024-01-11T09:43:42.079166+0000","last_fullsized":"2024-01-11T09:43:42.079166+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:21:03.439272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.088266+0000","last_change":"2024-01-11T09:43:42.081968+0000","last_active":"2024-01-11T09:43:42.088266+0000","last_peered":"2024-01-11T09:43:42.088266+0000","last_clean":"2024-01-11T09:43:42.088266+0000","last_became_active":"2024-01-11T09:43:42.081370+0000","last_became_peered":"2024-01-11T09:43:42.081370+0000","last_unstale":"2024-01-11T09:43:42.088266+0000","last_undegraded":"2024-01-11T09:43:42.088266+0000","last_fullsized":"2024-01-11T09:43:42.088266+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:11:18.246714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.077258+0000","last_change":"2024-01-11T09:43:42.077258+0000","last_active":"2024-01-11T09:43:42.077258+0000","last_peered":"2024-01-11T09:43:42.077258+0000","last_clean":"2024-01-11T09:43:42.077258+0000","last_became_active":"2024-01-11T09:43:42.076980+0000","last_became_peered":"2024-01-11T09:43:42.076980+0000","last_unstale":"2024-01-11T09:43:42.077258+0000","last_undegraded":"2024-01-11T09:43:42.077258+0000","last_fullsized":"2024-01-11T09:43:42.077258+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:44:51.740724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090831+0000","last_change":"2024-01-11T09:43:42.081561+0000","last_active":"2024-01-11T09:43:42.090831+0000","last_peered":"2024-01-11T09:43:42.090831+0000","last_clean":"2024-01-11T09:43:42.090831+0000","last_became_active":"2024-01-11T09:43:42.081012+0000","last_became_peered":"2024-01-11T09:43:42.081012+0000","last_unstale":"2024-01-11T09:43:42.090831+0000","last_undegraded":"2024-01-11T09:43:42.090831+0000","last_fullsized":"2024-01-11T09:43:42.090831+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:35:05.107900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.081243+0000","last_change":"2024-01-11T09:43:42.081243+0000","last_active":"2024-01-11T09:43:42.081243+0000","last_peered":"2024-01-11T09:43:42.081243+0000","last_clean":"2024-01-11T09:43:42.081243+0000","last_became_active":"2024-01-11T09:43:42.076361+0000","last_became_peered":"2024-01-11T09:43:42.076361+0000","last_unstale":"2024-01-11T09:43:42.081243+0000","last_undegraded":"2024-01-11T09:43:42.081243+0000","last_fullsized":"2024-01-11T09:43:42.081243+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:39:56.524103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.081190+0000","last_change":"2024-01-11T09:43:42.081190+0000","last_active":"2024-01-11T09:43:42.081190+0000","last_peered":"2024-01-11T09:43:42.081190+0000","last_clean":"2024-01-11T09:43:42.081190+0000","last_became_active":"2024-01-11T09:43:42.078543+0000","last_became_peered":"2024-01-11T09:43:42.078543+0000","last_unstale":"2024-01-11T09:43:42.081190+0000","last_undegraded":"2024-01-11T09:43:42.081190+0000","last_fullsized":"2024-01-11T09:43:42.081190+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:28:49.367007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.086232+0000","last_change":"2024-01-11T09:43:42.086232+0000","last_active":"2024-01-11T09:43:42.086232+0000","last_peered":"2024-01-11T09:43:42.086232+0000","last_clean":"2024-01-11T09:43:42.086232+0000","last_became_active":"2024-01-11T09:43:42.085935+0000","last_became_peered":"2024-01-11T09:43:42.085935+0000","last_unstale":"2024-01-11T09:43:42.086232+0000","last_undegraded":"2024-01-11T09:43:42.086232+0000","last_fullsized":"2024-01-11T09:43:42.086232+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:46:50.210897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.074134+0000","last_change":"2024-01-11T09:43:42.074134+0000","last_active":"2024-01-11T09:43:42.074134+0000","last_peered":"2024-01-11T09:43:42.074134+0000","last_clean":"2024-01-11T09:43:42.074134+0000","last_became_active":"2024-01-11T09:43:42.072966+0000","last_became_peered":"2024-01-11T09:43:42.072966+0000","last_unstale":"2024-01-11T09:43:42.074134+0000","last_undegraded":"2024-01-11T09:43:42.074134+0000","last_fullsized":"2024-01-11T09:43:42.074134+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:09:24.887933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.077625+0000","last_change":"2024-01-11T09:43:42.077625+0000","last_active":"2024-01-11T09:43:42.077625+0000","last_peered":"2024-01-11T09:43:42.077625+0000","last_clean":"2024-01-11T09:43:42.077625+0000","last_became_active":"2024-01-11T09:43:42.077041+0000","last_became_peered":"2024-01-11T09:43:42.077041+0000","last_unstale":"2024-01-11T09:43:42.077625+0000","last_undegraded":"2024-01-11T09:43:42.077625+0000","last_fullsized":"2024-01-11T09:43:42.077625+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:01:02.801676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.089655+0000","last_change":"2024-01-11T09:43:42.089655+0000","last_active":"2024-01-11T09:43:42.089655+0000","last_peered":"2024-01-11T09:43:42.089655+0000","last_clean":"2024-01-11T09:43:42.089655+0000","last_became_active":"2024-01-11T09:43:42.089316+0000","last_became_peered":"2024-01-11T09:43:42.089316+0000","last_unstale":"2024-01-11T09:43:42.089655+0000","last_undegraded":"2024-01-11T09:43:42.089655+0000","last_fullsized":"2024-01-11T09:43:42.089655+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:48.201216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.091961+0000","last_change":"2024-01-11T09:43:42.091961+0000","last_active":"2024-01-11T09:43:42.091961+0000","last_peered":"2024-01-11T09:43:42.091961+0000","last_clean":"2024-01-11T09:43:42.091961+0000","last_became_active":"2024-01-11T09:43:42.091668+0000","last_became_peered":"2024-01-11T09:43:42.091668+0000","last_unstale":"2024-01-11T09:43:42.091961+0000","last_undegraded":"2024-01-11T09:43:42.091961+0000","last_fullsized":"2024-01-11T09:43:42.091961+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:27:02.922803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090747+0000","last_change":"2024-01-11T09:43:42.081331+0000","last_active":"2024-01-11T09:43:42.090747+0000","last_peered":"2024-01-11T09:43:42.090747+0000","last_clean":"2024-01-11T09:43:42.090747+0000","last_became_active":"2024-01-11T09:43:42.080430+0000","last_became_peered":"2024-01-11T09:43:42.080430+0000","last_unstale":"2024-01-11T09:43:42.090747+0000","last_undegraded":"2024-01-11T09:43:42.090747+0000","last_fullsized":"2024-01-11T09:43:42.090747+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:21:03.439272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.096490+0000","last_change":"2024-01-11T09:43:42.096490+0000","last_active":"2024-01-11T09:43:42.096490+0000","last_peered":"2024-01-11T09:43:42.096490+0000","last_clean":"2024-01-11T09:43:42.096490+0000","last_became_active":"2024-01-11T09:43:42.096093+0000","last_became_peered":"2024-01-11T09:43:42.096093+0000","last_unstale":"2024-01-11T09:43:42.096490+0000","last_undegraded":"2024-01-11T09:43:42.096490+0000","last_fullsized":"2024-01-11T09:43:42.096490+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:56:51.522681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.087829+0000","last_change":"2024-01-11T09:43:42.081214+0000","last_active":"2024-01-11T09:43:42.087829+0000","last_peered":"2024-01-11T09:43:42.087829+0000","last_clean":"2024-01-11T09:43:42.087829+0000","last_became_active":"2024-01-11T09:43:42.080960+0000","last_became_peered":"2024-01-11T09:43:42.080960+0000","last_unstale":"2024-01-11T09:43:42.087829+0000","last_undegraded":"2024-01-11T09:43:42.087829+0000","last_fullsized":"2024-01-11T09:43:42.087829+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:39:39.259932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.080696+0000","last_change":"2024-01-11T09:43:42.080696+0000","last_active":"2024-01-11T09:43:42.080696+0000","last_peered":"2024-01-11T09:43:42.080696+0000","last_clean":"2024-01-11T09:43:42.080696+0000","last_became_active":"2024-01-11T09:43:42.080397+0000","last_became_peered":"2024-01-11T09:43:42.080397+0000","last_unstale":"2024-01-11T09:43:42.080696+0000","last_undegraded":"2024-01-11T09:43:42.080696+0000","last_fullsized":"2024-01-11T09:43:42.080696+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:22.095497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.084735+0000","last_change":"2024-01-11T09:43:42.084735+0000","last_active":"2024-01-11T09:43:42.084735+0000","last_peered":"2024-01-11T09:43:42.084735+0000","last_clean":"2024-01-11T09:43:42.084735+0000","last_became_active":"2024-01-11T09:43:42.084421+0000","last_became_peered":"2024-01-11T09:43:42.084421+0000","last_unstale":"2024-01-11T09:43:42.084735+0000","last_undegraded":"2024-01-11T09:43:42.084735+0000","last_fullsized":"2024-01-11T09:43:42.084735+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:01:09.493984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.076782+0000","last_change":"2024-01-11T09:43:42.076782+0000","last_active":"2024-01-11T09:43:42.076782+0000","last_peered":"2024-01-11T09:43:42.076782+0000","last_clean":"2024-01-11T09:43:42.076782+0000","last_became_active":"2024-01-11T09:43:42.076230+0000","last_became_peered":"2024-01-11T09:43:42.076230+0000","last_unstale":"2024-01-11T09:43:42.076782+0000","last_undegraded":"2024-01-11T09:43:42.076782+0000","last_fullsized":"2024-01-11T09:43:42.076782+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:14:13.281545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090351+0000","last_change":"2024-01-11T09:43:42.081678+0000","last_active":"2024-01-11T09:43:42.090351+0000","last_peered":"2024-01-11T09:43:42.090351+0000","last_clean":"2024-01-11T09:43:42.090351+0000","last_became_active":"2024-01-11T09:43:42.081002+0000","last_became_peered":"2024-01-11T09:43:42.081002+0000","last_unstale":"2024-01-11T09:43:42.090351+0000","last_undegraded":"2024-01-11T09:43:42.090351+0000","last_fullsized":"2024-01-11T09:43:42.090351+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:00:48.167638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.087451+0000","last_change":"2024-01-11T09:43:42.087451+0000","last_active":"2024-01-11T09:43:42.087451+0000","last_peered":"2024-01-11T09:43:42.087451+0000","last_clean":"2024-01-11T09:43:42.087451+0000","last_became_active":"2024-01-11T09:43:42.085641+0000","last_became_peered":"2024-01-11T09:43:42.085641+0000","last_unstale":"2024-01-11T09:43:42.087451+0000","last_undegraded":"2024-01-11T09:43:42.087451+0000","last_fullsized":"2024-01-11T09:43:42.087451+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:01:02.801676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.079469+0000","last_change":"2024-01-11T09:43:42.079469+0000","last_active":"2024-01-11T09:43:42.079469+0000","last_peered":"2024-01-11T09:43:42.079469+0000","last_clean":"2024-01-11T09:43:42.079469+0000","last_became_active":"2024-01-11T09:43:42.076089+0000","last_became_peered":"2024-01-11T09:43:42.076089+0000","last_unstale":"2024-01-11T09:43:42.079469+0000","last_undegraded":"2024-01-11T09:43:42.079469+0000","last_fullsized":"2024-01-11T09:43:42.079469+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:05:23.006191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090611+0000","last_change":"2024-01-11T09:43:42.078350+0000","last_active":"2024-01-11T09:43:42.090611+0000","last_peered":"2024-01-11T09:43:42.090611+0000","last_clean":"2024-01-11T09:43:42.090611+0000","last_became_active":"2024-01-11T09:43:42.078073+0000","last_became_peered":"2024-01-11T09:43:42.078073+0000","last_unstale":"2024-01-11T09:43:42.090611+0000","last_undegraded":"2024-01-11T09:43:42.090611+0000","last_fullsized":"2024-01-11T09:43:42.090611+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:56:51.522681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"197'4","reported_seq":13,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.092235+0000","last_change":"2024-01-11T09:43:41.069190+0000","last_active":"2024-01-11T09:43:42.092235+0000","last_peered":"2024-01-11T09:43:42.092235+0000","last_clean":"2024-01-11T09:43:42.092235+0000","last_became_active":"2024-01-11T09:43:41.068950+0000","last_became_peered":"2024-01-11T09:43:41.068950+0000","last_unstale":"2024-01-11T09:43:42.092235+0000","last_undegraded":"2024-01-11T09:43:42.092235+0000","last_fullsized":"2024-01-11T09:43:42.092235+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:36:40.206301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.085696+0000","last_change":"2024-01-11T09:43:42.085696+0000","last_active":"2024-01-11T09:43:42.085696+0000","last_peered":"2024-01-11T09:43:42.085696+0000","last_clean":"2024-01-11T09:43:42.085696+0000","last_became_active":"2024-01-11T09:43:42.084810+0000","last_became_peered":"2024-01-11T09:43:42.084810+0000","last_unstale":"2024-01-11T09:43:42.085696+0000","last_undegraded":"2024-01-11T09:43:42.085696+0000","last_fullsized":"2024-01-11T09:43:42.085696+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:06:31.817227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"197'1","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.085029+0000","last_change":"2024-01-11T09:43:41.066752+0000","last_active":"2024-01-11T09:43:42.085029+0000","last_peered":"2024-01-11T09:43:42.085029+0000","last_clean":"2024-01-11T09:43:42.085029+0000","last_became_active":"2024-01-11T09:43:41.066473+0000","last_became_peered":"2024-01-11T09:43:41.066473+0000","last_unstale":"2024-01-11T09:43:42.085029+0000","last_undegraded":"2024-01-11T09:43:42.085029+0000","last_fullsized":"2024-01-11T09:43:42.085029+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:36:40.206301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.086901+0000","last_change":"2024-01-11T09:43:42.076258+0000","last_active":"2024-01-11T09:43:42.086901+0000","last_peered":"2024-01-11T09:43:42.086901+0000","last_clean":"2024-01-11T09:43:42.086901+0000","last_became_active":"2024-01-11T09:43:42.076006+0000","last_became_peered":"2024-01-11T09:43:42.076006+0000","last_unstale":"2024-01-11T09:43:42.086901+0000","last_undegraded":"2024-01-11T09:43:42.086901+0000","last_fullsized":"2024-01-11T09:43:42.086901+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:35:58.603836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"197'2","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.084159+0000","last_change":"2024-01-11T09:43:41.073672+0000","last_active":"2024-01-11T09:43:42.084159+0000","last_peered":"2024-01-11T09:43:42.084159+0000","last_clean":"2024-01-11T09:43:42.084159+0000","last_became_active":"2024-01-11T09:43:41.073399+0000","last_became_peered":"2024-01-11T09:43:41.073399+0000","last_unstale":"2024-01-11T09:43:42.084159+0000","last_undegraded":"2024-01-11T09:43:42.084159+0000","last_fullsized":"2024-01-11T09:43:42.084159+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:27:59.240683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.088218+0000","last_change":"2024-01-11T09:43:42.080800+0000","last_active":"2024-01-11T09:43:42.088218+0000","last_peered":"2024-01-11T09:43:42.088218+0000","last_clean":"2024-01-11T09:43:42.088218+0000","last_became_active":"2024-01-11T09:43:42.080540+0000","last_became_peered":"2024-01-11T09:43:42.080540+0000","last_unstale":"2024-01-11T09:43:42.088218+0000","last_undegraded":"2024-01-11T09:43:42.088218+0000","last_fullsized":"2024-01-11T09:43:42.088218+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:22.095497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"197'2","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090090+0000","last_change":"2024-01-11T09:43:41.067291+0000","last_active":"2024-01-11T09:43:42.090090+0000","last_peered":"2024-01-11T09:43:42.090090+0000","last_clean":"2024-01-11T09:43:42.090090+0000","last_became_active":"2024-01-11T09:43:41.067001+0000","last_became_peered":"2024-01-11T09:43:41.067001+0000","last_unstale":"2024-01-11T09:43:42.090090+0000","last_undegraded":"2024-01-11T09:43:42.090090+0000","last_fullsized":"2024-01-11T09:43:42.090090+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:40:55.256586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.082961+0000","last_change":"2024-01-11T09:43:42.082961+0000","last_active":"2024-01-11T09:43:42.082961+0000","last_peered":"2024-01-11T09:43:42.082961+0000","last_clean":"2024-01-11T09:43:42.082961+0000","last_became_active":"2024-01-11T09:43:42.080622+0000","last_became_peered":"2024-01-11T09:43:42.080622+0000","last_unstale":"2024-01-11T09:43:42.082961+0000","last_undegraded":"2024-01-11T09:43:42.082961+0000","last_fullsized":"2024-01-11T09:43:42.082961+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:20:06.379496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"197'1","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.059906+0000","last_change":"2024-01-11T09:43:41.066081+0000","last_active":"2024-01-11T09:43:42.059906+0000","last_peered":"2024-01-11T09:43:42.059906+0000","last_clean":"2024-01-11T09:43:42.059906+0000","last_became_active":"2024-01-11T09:43:41.065820+0000","last_became_peered":"2024-01-11T09:43:41.065820+0000","last_unstale":"2024-01-11T09:43:42.059906+0000","last_undegraded":"2024-01-11T09:43:42.059906+0000","last_fullsized":"2024-01-11T09:43:42.059906+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:27:59.240683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090848+0000","last_change":"2024-01-11T09:43:42.082027+0000","last_active":"2024-01-11T09:43:42.090848+0000","last_peered":"2024-01-11T09:43:42.090848+0000","last_clean":"2024-01-11T09:43:42.090848+0000","last_became_active":"2024-01-11T09:43:42.079757+0000","last_became_peered":"2024-01-11T09:43:42.079757+0000","last_unstale":"2024-01-11T09:43:42.090848+0000","last_undegraded":"2024-01-11T09:43:42.090848+0000","last_fullsized":"2024-01-11T09:43:42.090848+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:48:40.643154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"197'1","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.094612+0000","last_change":"2024-01-11T09:43:41.075006+0000","last_active":"2024-01-11T09:43:42.094612+0000","last_peered":"2024-01-11T09:43:42.094612+0000","last_clean":"2024-01-11T09:43:42.094612+0000","last_became_active":"2024-01-11T09:43:41.074557+0000","last_became_peered":"2024-01-11T09:43:41.074557+0000","last_unstale":"2024-01-11T09:43:42.094612+0000","last_undegraded":"2024-01-11T09:43:42.094612+0000","last_fullsized":"2024-01-11T09:43:42.094612+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:35:57.597516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.086790+0000","last_change":"2024-01-11T09:43:42.077002+0000","last_active":"2024-01-11T09:43:42.086790+0000","last_peered":"2024-01-11T09:43:42.086790+0000","last_clean":"2024-01-11T09:43:42.086790+0000","last_became_active":"2024-01-11T09:43:42.076741+0000","last_became_peered":"2024-01-11T09:43:42.076741+0000","last_unstale":"2024-01-11T09:43:42.086790+0000","last_undegraded":"2024-01-11T09:43:42.086790+0000","last_fullsized":"2024-01-11T09:43:42.086790+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:20:09.297290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.068378+0000","last_change":"2024-01-11T09:43:41.066550+0000","last_active":"2024-01-11T09:43:41.068378+0000","last_peered":"2024-01-11T09:43:41.068378+0000","last_clean":"2024-01-11T09:43:41.068378+0000","last_became_active":"2024-01-11T09:43:41.066176+0000","last_became_peered":"2024-01-11T09:43:41.066176+0000","last_unstale":"2024-01-11T09:43:41.068378+0000","last_undegraded":"2024-01-11T09:43:41.068378+0000","last_fullsized":"2024-01-11T09:43:41.068378+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:35:35.073088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090380+0000","last_change":"2024-01-11T09:43:42.090380+0000","last_active":"2024-01-11T09:43:42.090380+0000","last_peered":"2024-01-11T09:43:42.090380+0000","last_clean":"2024-01-11T09:43:42.090380+0000","last_became_active":"2024-01-11T09:43:42.089950+0000","last_became_peered":"2024-01-11T09:43:42.089950+0000","last_unstale":"2024-01-11T09:43:42.090380+0000","last_undegraded":"2024-01-11T09:43:42.090380+0000","last_fullsized":"2024-01-11T09:43:42.090380+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:11:18.246714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.072412+0000","last_change":"2024-01-11T09:43:41.067450+0000","last_active":"2024-01-11T09:43:42.072412+0000","last_peered":"2024-01-11T09:43:42.072412+0000","last_clean":"2024-01-11T09:43:42.072412+0000","last_became_active":"2024-01-11T09:43:41.066475+0000","last_became_peered":"2024-01-11T09:43:41.066475+0000","last_unstale":"2024-01-11T09:43:42.072412+0000","last_undegraded":"2024-01-11T09:43:42.072412+0000","last_fullsized":"2024-01-11T09:43:42.072412+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:10:04.789345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.091223+0000","last_change":"2024-01-11T09:43:42.082956+0000","last_active":"2024-01-11T09:43:42.091223+0000","last_peered":"2024-01-11T09:43:42.091223+0000","last_clean":"2024-01-11T09:43:42.091223+0000","last_became_active":"2024-01-11T09:43:42.080949+0000","last_became_peered":"2024-01-11T09:43:42.080949+0000","last_unstale":"2024-01-11T09:43:42.091223+0000","last_undegraded":"2024-01-11T09:43:42.091223+0000","last_fullsized":"2024-01-11T09:43:42.091223+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:00:10.469996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.071270+0000","last_change":"2024-01-11T09:43:41.068988+0000","last_active":"2024-01-11T09:43:41.071270+0000","last_peered":"2024-01-11T09:43:41.071270+0000","last_clean":"2024-01-11T09:43:41.071270+0000","last_became_active":"2024-01-11T09:43:41.068718+0000","last_became_peered":"2024-01-11T09:43:41.068718+0000","last_unstale":"2024-01-11T09:43:41.071270+0000","last_undegraded":"2024-01-11T09:43:41.071270+0000","last_fullsized":"2024-01-11T09:43:41.071270+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:44:02.560921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.093087+0000","last_change":"2024-01-11T09:43:42.088859+0000","last_active":"2024-01-11T09:43:42.093087+0000","last_peered":"2024-01-11T09:43:42.093087+0000","last_clean":"2024-01-11T09:43:42.093087+0000","last_became_active":"2024-01-11T09:43:42.088201+0000","last_became_peered":"2024-01-11T09:43:42.088201+0000","last_unstale":"2024-01-11T09:43:42.093087+0000","last_undegraded":"2024-01-11T09:43:42.093087+0000","last_fullsized":"2024-01-11T09:43:42.093087+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:40:56.262906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"197'3","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.092070+0000","last_change":"2024-01-11T09:43:41.067045+0000","last_active":"2024-01-11T09:43:42.092070+0000","last_peered":"2024-01-11T09:43:42.092070+0000","last_clean":"2024-01-11T09:43:42.092070+0000","last_became_active":"2024-01-11T09:43:41.066802+0000","last_became_peered":"2024-01-11T09:43:41.066802+0000","last_unstale":"2024-01-11T09:43:42.092070+0000","last_undegraded":"2024-01-11T09:43:42.092070+0000","last_fullsized":"2024-01-11T09:43:42.092070+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:33:23.788733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090178+0000","last_change":"2024-01-11T09:43:42.090178+0000","last_active":"2024-01-11T09:43:42.090178+0000","last_peered":"2024-01-11T09:43:42.090178+0000","last_clean":"2024-01-11T09:43:42.090178+0000","last_became_active":"2024-01-11T09:43:42.089912+0000","last_became_peered":"2024-01-11T09:43:42.089912+0000","last_unstale":"2024-01-11T09:43:42.090178+0000","last_undegraded":"2024-01-11T09:43:42.090178+0000","last_fullsized":"2024-01-11T09:43:42.090178+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:36:19.587822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"197'6","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.092149+0000","last_change":"2024-01-11T09:43:41.072734+0000","last_active":"2024-01-11T09:43:42.092149+0000","last_peered":"2024-01-11T09:43:42.092149+0000","last_clean":"2024-01-11T09:43:42.092149+0000","last_became_active":"2024-01-11T09:43:41.072429+0000","last_became_peered":"2024-01-11T09:43:41.072429+0000","last_unstale":"2024-01-11T09:43:42.092149+0000","last_undegraded":"2024-01-11T09:43:42.092149+0000","last_fullsized":"2024-01-11T09:43:42.092149+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:44:02.560921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.092372+0000","last_change":"2024-01-11T09:43:42.092372+0000","last_active":"2024-01-11T09:43:42.092372+0000","last_peered":"2024-01-11T09:43:42.092372+0000","last_clean":"2024-01-11T09:43:42.092372+0000","last_became_active":"2024-01-11T09:43:42.091799+0000","last_became_peered":"2024-01-11T09:43:42.091799+0000","last_unstale":"2024-01-11T09:43:42.092372+0000","last_undegraded":"2024-01-11T09:43:42.092372+0000","last_fullsized":"2024-01-11T09:43:42.092372+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:10:05.795665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"197'2","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.092771+0000","last_change":"2024-01-11T09:43:41.072649+0000","last_active":"2024-01-11T09:43:42.092771+0000","last_peered":"2024-01-11T09:43:42.092771+0000","last_clean":"2024-01-11T09:43:42.092771+0000","last_became_active":"2024-01-11T09:43:41.072376+0000","last_became_peered":"2024-01-11T09:43:41.072376+0000","last_unstale":"2024-01-11T09:43:42.092771+0000","last_undegraded":"2024-01-11T09:43:42.092771+0000","last_fullsized":"2024-01-11T09:43:42.092771+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:55:36.074446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.089291+0000","last_change":"2024-01-11T09:43:42.083805+0000","last_active":"2024-01-11T09:43:42.089291+0000","last_peered":"2024-01-11T09:43:42.089291+0000","last_clean":"2024-01-11T09:43:42.089291+0000","last_became_active":"2024-01-11T09:43:42.083087+0000","last_became_peered":"2024-01-11T09:43:42.083087+0000","last_unstale":"2024-01-11T09:43:42.089291+0000","last_undegraded":"2024-01-11T09:43:42.089291+0000","last_fullsized":"2024-01-11T09:43:42.089291+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:25:15.467096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.068179+0000","last_change":"2024-01-11T09:43:41.064275+0000","last_active":"2024-01-11T09:43:41.068179+0000","last_peered":"2024-01-11T09:43:41.068179+0000","last_clean":"2024-01-11T09:43:41.068179+0000","last_became_active":"2024-01-11T09:43:41.063957+0000","last_became_peered":"2024-01-11T09:43:41.063957+0000","last_unstale":"2024-01-11T09:43:41.068179+0000","last_undegraded":"2024-01-11T09:43:41.068179+0000","last_fullsized":"2024-01-11T09:43:41.068179+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:35:04.101580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.091041+0000","last_change":"2024-01-11T09:43:42.086313+0000","last_active":"2024-01-11T09:43:42.091041+0000","last_peered":"2024-01-11T09:43:42.091041+0000","last_clean":"2024-01-11T09:43:42.091041+0000","last_became_active":"2024-01-11T09:43:42.085963+0000","last_became_peered":"2024-01-11T09:43:42.085963+0000","last_unstale":"2024-01-11T09:43:42.091041+0000","last_undegraded":"2024-01-11T09:43:42.091041+0000","last_fullsized":"2024-01-11T09:43:42.091041+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:15:41.955996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.060565+0000","last_change":"2024-01-11T09:43:41.062575+0000","last_active":"2024-01-11T09:43:42.060565+0000","last_peered":"2024-01-11T09:43:42.060565+0000","last_clean":"2024-01-11T09:43:42.060565+0000","last_became_active":"2024-01-11T09:43:41.062305+0000","last_became_peered":"2024-01-11T09:43:41.062305+0000","last_unstale":"2024-01-11T09:43:42.060565+0000","last_undegraded":"2024-01-11T09:43:42.060565+0000","last_fullsized":"2024-01-11T09:43:42.060565+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:40:55.256586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.084380+0000","last_change":"2024-01-11T09:43:42.084380+0000","last_active":"2024-01-11T09:43:42.084380+0000","last_peered":"2024-01-11T09:43:42.084380+0000","last_clean":"2024-01-11T09:43:42.084380+0000","last_became_active":"2024-01-11T09:43:42.084013+0000","last_became_peered":"2024-01-11T09:43:42.084013+0000","last_unstale":"2024-01-11T09:43:42.084380+0000","last_undegraded":"2024-01-11T09:43:42.084380+0000","last_fullsized":"2024-01-11T09:43:42.084380+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:42:48.428037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"197'8","reported_seq":17,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.095742+0000","last_change":"2024-01-11T09:43:41.066556+0000","last_active":"2024-01-11T09:43:42.095742+0000","last_peered":"2024-01-11T09:43:42.095742+0000","last_clean":"2024-01-11T09:43:42.095742+0000","last_became_active":"2024-01-11T09:43:41.066251+0000","last_became_peered":"2024-01-11T09:43:41.066251+0000","last_unstale":"2024-01-11T09:43:42.095742+0000","last_undegraded":"2024-01-11T09:43:42.095742+0000","last_fullsized":"2024-01-11T09:43:42.095742+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:08.927728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.089897+0000","last_change":"2024-01-11T09:43:42.089897+0000","last_active":"2024-01-11T09:43:42.089897+0000","last_peered":"2024-01-11T09:43:42.089897+0000","last_clean":"2024-01-11T09:43:42.089897+0000","last_became_active":"2024-01-11T09:43:42.089624+0000","last_became_peered":"2024-01-11T09:43:42.089624+0000","last_unstale":"2024-01-11T09:43:42.089897+0000","last_undegraded":"2024-01-11T09:43:42.089897+0000","last_fullsized":"2024-01-11T09:43:42.089897+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:21:23.493032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"197'1","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.059583+0000","last_change":"2024-01-11T09:43:41.070478+0000","last_active":"2024-01-11T09:43:42.059583+0000","last_peered":"2024-01-11T09:43:42.059583+0000","last_clean":"2024-01-11T09:43:42.059583+0000","last_became_active":"2024-01-11T09:43:41.070009+0000","last_became_peered":"2024-01-11T09:43:41.070009+0000","last_unstale":"2024-01-11T09:43:42.059583+0000","last_undegraded":"2024-01-11T09:43:42.059583+0000","last_fullsized":"2024-01-11T09:43:42.059583+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:32:09.319822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.097039+0000","last_change":"2024-01-11T09:43:42.096523+0000","last_active":"2024-01-11T09:43:42.097039+0000","last_peered":"2024-01-11T09:43:42.097039+0000","last_clean":"2024-01-11T09:43:42.097039+0000","last_became_active":"2024-01-11T09:43:42.096126+0000","last_became_peered":"2024-01-11T09:43:42.096126+0000","last_unstale":"2024-01-11T09:43:42.097039+0000","last_undegraded":"2024-01-11T09:43:42.097039+0000","last_fullsized":"2024-01-11T09:43:42.097039+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:48:31.109585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"197'2","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.093584+0000","last_change":"2024-01-11T09:43:41.069733+0000","last_active":"2024-01-11T09:43:42.093584+0000","last_peered":"2024-01-11T09:43:42.093584+0000","last_clean":"2024-01-11T09:43:42.093584+0000","last_became_active":"2024-01-11T09:43:41.069510+0000","last_became_peered":"2024-01-11T09:43:41.069510+0000","last_unstale":"2024-01-11T09:43:42.093584+0000","last_undegraded":"2024-01-11T09:43:42.093584+0000","last_fullsized":"2024-01-11T09:43:42.093584+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:20:08.290971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.079719+0000","last_change":"2024-01-11T09:43:42.070438+0000","last_active":"2024-01-11T09:43:42.079719+0000","last_peered":"2024-01-11T09:43:42.079719+0000","last_clean":"2024-01-11T09:43:42.079719+0000","last_became_active":"2024-01-11T09:43:42.070170+0000","last_became_peered":"2024-01-11T09:43:42.070170+0000","last_unstale":"2024-01-11T09:43:42.079719+0000","last_undegraded":"2024-01-11T09:43:42.079719+0000","last_fullsized":"2024-01-11T09:43:42.079719+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:01:02.801676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"197'1","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.057977+0000","last_change":"2024-01-11T09:43:41.072650+0000","last_active":"2024-01-11T09:43:42.057977+0000","last_peered":"2024-01-11T09:43:42.057977+0000","last_clean":"2024-01-11T09:43:42.057977+0000","last_became_active":"2024-01-11T09:43:41.072346+0000","last_became_peered":"2024-01-11T09:43:41.072346+0000","last_unstale":"2024-01-11T09:43:42.057977+0000","last_undegraded":"2024-01-11T09:43:42.057977+0000","last_fullsized":"2024-01-11T09:43:42.057977+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:20:40.115692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.073876+0000","last_change":"2024-01-11T09:43:42.073876+0000","last_active":"2024-01-11T09:43:42.073876+0000","last_peered":"2024-01-11T09:43:42.073876+0000","last_clean":"2024-01-11T09:43:42.073876+0000","last_became_active":"2024-01-11T09:43:42.073613+0000","last_became_peered":"2024-01-11T09:43:42.073613+0000","last_unstale":"2024-01-11T09:43:42.073876+0000","last_undegraded":"2024-01-11T09:43:42.073876+0000","last_fullsized":"2024-01-11T09:43:42.073876+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:35:05.107900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.072083+0000","last_change":"2024-01-11T09:43:41.061032+0000","last_active":"2024-01-11T09:43:42.072083+0000","last_peered":"2024-01-11T09:43:42.072083+0000","last_clean":"2024-01-11T09:43:42.072083+0000","last_became_active":"2024-01-11T09:43:41.060770+0000","last_became_peered":"2024-01-11T09:43:41.060770+0000","last_unstale":"2024-01-11T09:43:42.072083+0000","last_undegraded":"2024-01-11T09:43:42.072083+0000","last_fullsized":"2024-01-11T09:43:42.072083+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:00:47.161318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.080884+0000","last_change":"2024-01-11T09:43:42.080884+0000","last_active":"2024-01-11T09:43:42.080884+0000","last_peered":"2024-01-11T09:43:42.080884+0000","last_clean":"2024-01-11T09:43:42.080884+0000","last_became_active":"2024-01-11T09:43:42.074190+0000","last_became_peered":"2024-01-11T09:43:42.074190+0000","last_unstale":"2024-01-11T09:43:42.080884+0000","last_undegraded":"2024-01-11T09:43:42.080884+0000","last_fullsized":"2024-01-11T09:43:42.080884+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:36:05.762240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"197'2","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.092917+0000","last_change":"2024-01-11T09:43:41.070690+0000","last_active":"2024-01-11T09:43:42.092917+0000","last_peered":"2024-01-11T09:43:42.092917+0000","last_clean":"2024-01-11T09:43:42.092917+0000","last_became_active":"2024-01-11T09:43:41.069836+0000","last_became_peered":"2024-01-11T09:43:41.069836+0000","last_unstale":"2024-01-11T09:43:42.092917+0000","last_undegraded":"2024-01-11T09:43:42.092917+0000","last_fullsized":"2024-01-11T09:43:42.092917+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:36:40.206301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090815+0000","last_change":"2024-01-11T09:43:42.087040+0000","last_active":"2024-01-11T09:43:42.090815+0000","last_peered":"2024-01-11T09:43:42.090815+0000","last_clean":"2024-01-11T09:43:42.090815+0000","last_became_active":"2024-01-11T09:43:42.086230+0000","last_became_peered":"2024-01-11T09:43:42.086230+0000","last_unstale":"2024-01-11T09:43:42.090815+0000","last_undegraded":"2024-01-11T09:43:42.090815+0000","last_fullsized":"2024-01-11T09:43:42.090815+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:10:24.874233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.065354+0000","last_change":"2024-01-11T09:43:41.060470+0000","last_active":"2024-01-11T09:43:41.065354+0000","last_peered":"2024-01-11T09:43:41.065354+0000","last_clean":"2024-01-11T09:43:41.065354+0000","last_became_active":"2024-01-11T09:43:41.060208+0000","last_became_peered":"2024-01-11T09:43:41.060208+0000","last_unstale":"2024-01-11T09:43:41.065354+0000","last_undegraded":"2024-01-11T09:43:41.065354+0000","last_fullsized":"2024-01-11T09:43:41.065354+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:54.422211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.078191+0000","last_change":"2024-01-11T09:43:42.078191+0000","last_active":"2024-01-11T09:43:42.078191+0000","last_peered":"2024-01-11T09:43:42.078191+0000","last_clean":"2024-01-11T09:43:42.078191+0000","last_became_active":"2024-01-11T09:43:42.076031+0000","last_became_peered":"2024-01-11T09:43:42.076031+0000","last_unstale":"2024-01-11T09:43:42.078191+0000","last_undegraded":"2024-01-11T09:43:42.078191+0000","last_fullsized":"2024-01-11T09:43:42.078191+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:23:16.219171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.073169+0000","last_change":"2024-01-11T09:43:41.068567+0000","last_active":"2024-01-11T09:43:41.073169+0000","last_peered":"2024-01-11T09:43:41.073169+0000","last_clean":"2024-01-11T09:43:41.073169+0000","last_became_active":"2024-01-11T09:43:41.068269+0000","last_became_peered":"2024-01-11T09:43:41.068269+0000","last_unstale":"2024-01-11T09:43:41.073169+0000","last_undegraded":"2024-01-11T09:43:41.073169+0000","last_fullsized":"2024-01-11T09:43:41.073169+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:04:27.002128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.075960+0000","last_change":"2024-01-11T09:43:42.075960+0000","last_active":"2024-01-11T09:43:42.075960+0000","last_peered":"2024-01-11T09:43:42.075960+0000","last_clean":"2024-01-11T09:43:42.075960+0000","last_became_active":"2024-01-11T09:43:42.075677+0000","last_became_peered":"2024-01-11T09:43:42.075677+0000","last_unstale":"2024-01-11T09:43:42.075960+0000","last_undegraded":"2024-01-11T09:43:42.075960+0000","last_fullsized":"2024-01-11T09:43:42.075960+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:00:48.167638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.068859+0000","last_change":"2024-01-11T09:43:41.063300+0000","last_active":"2024-01-11T09:43:42.068859+0000","last_peered":"2024-01-11T09:43:42.068859+0000","last_clean":"2024-01-11T09:43:42.068859+0000","last_became_active":"2024-01-11T09:43:41.061537+0000","last_became_peered":"2024-01-11T09:43:41.061537+0000","last_unstale":"2024-01-11T09:43:42.068859+0000","last_undegraded":"2024-01-11T09:43:42.068859+0000","last_fullsized":"2024-01-11T09:43:42.068859+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:54.422211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.080672+0000","last_change":"2024-01-11T09:43:42.080672+0000","last_active":"2024-01-11T09:43:42.080672+0000","last_peered":"2024-01-11T09:43:42.080672+0000","last_clean":"2024-01-11T09:43:42.080672+0000","last_became_active":"2024-01-11T09:43:42.080429+0000","last_became_peered":"2024-01-11T09:43:42.080429+0000","last_unstale":"2024-01-11T09:43:42.080672+0000","last_undegraded":"2024-01-11T09:43:42.080672+0000","last_fullsized":"2024-01-11T09:43:42.080672+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:03:31.123957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"197'2","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.093135+0000","last_change":"2024-01-11T09:43:41.070600+0000","last_active":"2024-01-11T09:43:42.093135+0000","last_peered":"2024-01-11T09:43:42.093135+0000","last_clean":"2024-01-11T09:43:42.093135+0000","last_became_active":"2024-01-11T09:43:41.070321+0000","last_became_peered":"2024-01-11T09:43:41.070321+0000","last_unstale":"2024-01-11T09:43:42.093135+0000","last_undegraded":"2024-01-11T09:43:42.093135+0000","last_fullsized":"2024-01-11T09:43:42.093135+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:20:08.290971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.079792+0000","last_change":"2024-01-11T09:43:42.079792+0000","last_active":"2024-01-11T09:43:42.079792+0000","last_peered":"2024-01-11T09:43:42.079792+0000","last_clean":"2024-01-11T09:43:42.079792+0000","last_became_active":"2024-01-11T09:43:42.079414+0000","last_became_peered":"2024-01-11T09:43:42.079414+0000","last_unstale":"2024-01-11T09:43:42.079792+0000","last_undegraded":"2024-01-11T09:43:42.079792+0000","last_fullsized":"2024-01-11T09:43:42.079792+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:52:38.016298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.075852+0000","last_change":"2024-01-11T09:43:41.073648+0000","last_active":"2024-01-11T09:43:41.075852+0000","last_peered":"2024-01-11T09:43:41.075852+0000","last_clean":"2024-01-11T09:43:41.075852+0000","last_became_active":"2024-01-11T09:43:41.073404+0000","last_became_peered":"2024-01-11T09:43:41.073404+0000","last_unstale":"2024-01-11T09:43:41.075852+0000","last_undegraded":"2024-01-11T09:43:41.075852+0000","last_fullsized":"2024-01-11T09:43:41.075852+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:21:02.432952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.089769+0000","last_change":"2024-01-11T09:43:42.078061+0000","last_active":"2024-01-11T09:43:42.089769+0000","last_peered":"2024-01-11T09:43:42.089769+0000","last_clean":"2024-01-11T09:43:42.089769+0000","last_became_active":"2024-01-11T09:43:42.077674+0000","last_became_peered":"2024-01-11T09:43:42.077674+0000","last_unstale":"2024-01-11T09:43:42.089769+0000","last_undegraded":"2024-01-11T09:43:42.089769+0000","last_fullsized":"2024-01-11T09:43:42.089769+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:03:04.194736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.075919+0000","last_change":"2024-01-11T09:43:41.074413+0000","last_active":"2024-01-11T09:43:41.075919+0000","last_peered":"2024-01-11T09:43:41.075919+0000","last_clean":"2024-01-11T09:43:41.075919+0000","last_became_active":"2024-01-11T09:43:41.074157+0000","last_became_peered":"2024-01-11T09:43:41.074157+0000","last_unstale":"2024-01-11T09:43:41.075919+0000","last_undegraded":"2024-01-11T09:43:41.075919+0000","last_fullsized":"2024-01-11T09:43:41.075919+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:00:09.463677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.093508+0000","last_change":"2024-01-11T09:43:42.087645+0000","last_active":"2024-01-11T09:43:42.093508+0000","last_peered":"2024-01-11T09:43:42.093508+0000","last_clean":"2024-01-11T09:43:42.093508+0000","last_became_active":"2024-01-11T09:43:42.087367+0000","last_became_peered":"2024-01-11T09:43:42.087367+0000","last_unstale":"2024-01-11T09:43:42.093508+0000","last_undegraded":"2024-01-11T09:43:42.093508+0000","last_fullsized":"2024-01-11T09:43:42.093508+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:01:29.860733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.060382+0000","last_change":"2024-01-11T09:43:41.069275+0000","last_active":"2024-01-11T09:43:42.060382+0000","last_peered":"2024-01-11T09:43:42.060382+0000","last_clean":"2024-01-11T09:43:42.060382+0000","last_became_active":"2024-01-11T09:43:41.069032+0000","last_became_peered":"2024-01-11T09:43:41.069032+0000","last_unstale":"2024-01-11T09:43:42.060382+0000","last_undegraded":"2024-01-11T09:43:42.060382+0000","last_fullsized":"2024-01-11T09:43:42.060382+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:10:23.867913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.063236+0000","last_change":"2024-01-11T09:43:42.063236+0000","last_active":"2024-01-11T09:43:42.063236+0000","last_peered":"2024-01-11T09:43:42.063236+0000","last_clean":"2024-01-11T09:43:42.063236+0000","last_became_active":"2024-01-11T09:43:42.062915+0000","last_became_peered":"2024-01-11T09:43:42.062915+0000","last_unstale":"2024-01-11T09:43:42.063236+0000","last_undegraded":"2024-01-11T09:43:42.063236+0000","last_fullsized":"2024-01-11T09:43:42.063236+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:25:15.467096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.068760+0000","last_change":"2024-01-11T09:43:41.054651+0000","last_active":"2024-01-11T09:43:42.068760+0000","last_peered":"2024-01-11T09:43:42.068760+0000","last_clean":"2024-01-11T09:43:42.068760+0000","last_became_active":"2024-01-11T09:43:41.054360+0000","last_became_peered":"2024-01-11T09:43:41.054360+0000","last_unstale":"2024-01-11T09:43:42.068760+0000","last_undegraded":"2024-01-11T09:43:42.068760+0000","last_fullsized":"2024-01-11T09:43:42.068760+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:04:27.002128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.079536+0000","last_change":"2024-01-11T09:43:42.079536+0000","last_active":"2024-01-11T09:43:42.079536+0000","last_peered":"2024-01-11T09:43:42.079536+0000","last_clean":"2024-01-11T09:43:42.079536+0000","last_became_active":"2024-01-11T09:43:42.078591+0000","last_became_peered":"2024-01-11T09:43:42.078591+0000","last_unstale":"2024-01-11T09:43:42.079536+0000","last_undegraded":"2024-01-11T09:43:42.079536+0000","last_fullsized":"2024-01-11T09:43:42.079536+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:28:08.543281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.068774+0000","last_change":"2024-01-11T09:43:41.065046+0000","last_active":"2024-01-11T09:43:42.068774+0000","last_peered":"2024-01-11T09:43:42.068774+0000","last_clean":"2024-01-11T09:43:42.068774+0000","last_became_active":"2024-01-11T09:43:41.064262+0000","last_became_peered":"2024-01-11T09:43:41.064262+0000","last_unstale":"2024-01-11T09:43:42.068774+0000","last_undegraded":"2024-01-11T09:43:42.068774+0000","last_fullsized":"2024-01-11T09:43:42.068774+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:44:02.560921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.083650+0000","last_change":"2024-01-11T09:43:42.083650+0000","last_active":"2024-01-11T09:43:42.083650+0000","last_peered":"2024-01-11T09:43:42.083650+0000","last_clean":"2024-01-11T09:43:42.083650+0000","last_became_active":"2024-01-11T09:43:42.080323+0000","last_became_peered":"2024-01-11T09:43:42.080323+0000","last_unstale":"2024-01-11T09:43:42.083650+0000","last_undegraded":"2024-01-11T09:43:42.083650+0000","last_fullsized":"2024-01-11T09:43:42.083650+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:39:57.126666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.068774+0000","last_change":"2024-01-11T09:43:41.063971+0000","last_active":"2024-01-11T09:43:41.068774+0000","last_peered":"2024-01-11T09:43:41.068774+0000","last_clean":"2024-01-11T09:43:41.068774+0000","last_became_active":"2024-01-11T09:43:41.063586+0000","last_became_peered":"2024-01-11T09:43:41.063586+0000","last_unstale":"2024-01-11T09:43:41.068774+0000","last_undegraded":"2024-01-11T09:43:41.068774+0000","last_fullsized":"2024-01-11T09:43:41.068774+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:35:57.597516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.072612+0000","last_change":"2024-01-11T09:43:42.072612+0000","last_active":"2024-01-11T09:43:42.072612+0000","last_peered":"2024-01-11T09:43:42.072612+0000","last_clean":"2024-01-11T09:43:42.072612+0000","last_became_active":"2024-01-11T09:43:42.072045+0000","last_became_peered":"2024-01-11T09:43:42.072045+0000","last_unstale":"2024-01-11T09:43:42.072612+0000","last_undegraded":"2024-01-11T09:43:42.072612+0000","last_fullsized":"2024-01-11T09:43:42.072612+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:10:05.795665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.076213+0000","last_change":"2024-01-11T09:43:41.072802+0000","last_active":"2024-01-11T09:43:41.076213+0000","last_peered":"2024-01-11T09:43:41.076213+0000","last_clean":"2024-01-11T09:43:41.076213+0000","last_became_active":"2024-01-11T09:43:41.072288+0000","last_became_peered":"2024-01-11T09:43:41.072288+0000","last_unstale":"2024-01-11T09:43:41.076213+0000","last_undegraded":"2024-01-11T09:43:41.076213+0000","last_fullsized":"2024-01-11T09:43:41.076213+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:54.422211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.076624+0000","last_change":"2024-01-11T09:43:42.076624+0000","last_active":"2024-01-11T09:43:42.076624+0000","last_peered":"2024-01-11T09:43:42.076624+0000","last_clean":"2024-01-11T09:43:42.076624+0000","last_became_active":"2024-01-11T09:43:42.074176+0000","last_became_peered":"2024-01-11T09:43:42.074176+0000","last_unstale":"2024-01-11T09:43:42.076624+0000","last_undegraded":"2024-01-11T09:43:42.076624+0000","last_fullsized":"2024-01-11T09:43:42.076624+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:17:54.012106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.075970+0000","last_change":"2024-01-11T09:43:41.071431+0000","last_active":"2024-01-11T09:43:41.075970+0000","last_peered":"2024-01-11T09:43:41.075970+0000","last_clean":"2024-01-11T09:43:41.075970+0000","last_became_active":"2024-01-11T09:43:41.071073+0000","last_became_peered":"2024-01-11T09:43:41.071073+0000","last_unstale":"2024-01-11T09:43:41.075970+0000","last_undegraded":"2024-01-11T09:43:41.075970+0000","last_fullsized":"2024-01-11T09:43:41.075970+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:03:30.117638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"188'344","reported_seq":1107,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.060116+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:43:41.060116+0000","last_peered":"2024-01-11T09:43:41.060116+0000","last_clean":"2024-01-11T09:43:41.060116+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:43:41.060116+0000","last_undegraded":"2024-01-11T09:43:41.060116+0000","last_fullsized":"2024-01-11T09:43:41.060116+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2096,"num_read_kb":4396,"num_write":571,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2096,"num_read_kb":4396,"num_write":571,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":344,"ondisk_log_size":344,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300288,"kb_used_data":3000,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071552,"statfs":{"total":96636764160,"available":96329269248,"internally_reserved":0,"allocated":3072000,"data_stored":1765854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.72499999999999998,"15min":0.73999999999999999},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.36599999999999999},"max":{"1min":0.90400000000000003,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.76200000000000001,"15min":0.753},"min":{"1min":0.58799999999999997,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.89900000000000002,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.71799999999999997}]},{"osd":1,"last update":"Thu Jan 11 09:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.73199999999999998,"15min":0.73699999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41599999999999998},"max":{"1min":0.88,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.746},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.746,"15min":0.749},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.85899999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.57899999999999996}]},{"osd":2,"last update":"Thu Jan 11 09:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.747,"15min":0.746},"min":{"1min":0.60699999999999998,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.87,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.76700000000000002,"15min":0.76000000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.44500000000000001},"max":{"1min":0.91100000000000003,"5min":1.153,"15min":1.153},"last":0.65100000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.75900000000000001,"15min":0.77500000000000002},"min":{"1min":0.56499999999999995,"5min":0.53600000000000003,"15min":0.42699999999999999},"max":{"1min":0.90800000000000003,"5min":1.03,"15min":1.0660000000000001},"last":0.622},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.753,"15min":0.76200000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.92300000000000004,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.59599999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.41599999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.76200000000000001,"5min":1.107,"15min":1.107},"last":0.753},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.39700000000000002},"max":{"1min":0.73699999999999999,"5min":0.96099999999999997,"15min":1.0960000000000001},"last":0.73499999999999999}]},{"osd":5,"last update":"Thu Jan 11 09:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.63600000000000001,"15min":0.63700000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.42399999999999999},"max":{"1min":0.78600000000000003,"5min":0.90300000000000002,"15min":1.0109999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.67900000000000005,"15min":0.68100000000000005},"min":{"1min":0.50700000000000001,"5min":0.47099999999999997,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":1.133,"15min":1.133},"last":0.66000000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.77600000000000002,"15min":0.77700000000000002},"min":{"1min":0.57999999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.89500000000000002,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.76600000000000001,"15min":0.75600000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.45800000000000002},"max":{"1min":0.874,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.78800000000000003}]}]},{"osd":6,"up_from":13,"seq":55834575952,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311988,"kb_used_data":2412,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059852,"statfs":{"total":96636764160,"available":96317288448,"internally_reserved":0,"allocated":2469888,"data_stored":1175382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.71199999999999997,"15min":0.71999999999999997},"min":{"1min":0.54800000000000004,"5min":0.503,"15min":0.34599999999999997},"max":{"1min":0.94199999999999995,"5min":0.96199999999999997,"15min":1.508},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.755,"15min":0.754},"min":{"1min":0.59099999999999997,"5min":0.44400000000000001,"15min":0.39200000000000002},"max":{"1min":0.97099999999999997,"5min":1.216,"15min":1.5589999999999999},"last":0.81200000000000006}]},{"osd":1,"last update":"Thu Jan 11 09:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.748,"15min":0.753},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.92500000000000004,"5min":1.2549999999999999,"15min":1.45},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.72599999999999998,"15min":0.74099999999999999},"min":{"1min":0.56599999999999995,"5min":0.48299999999999998,"15min":0.46899999999999997},"max":{"1min":0.95299999999999996,"5min":1.0249999999999999,"15min":1.542},"last":0.878}]},{"osd":2,"last update":"Thu Jan 11 09:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.75800000000000001,"15min":0.76100000000000001},"min":{"1min":0.58599999999999997,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.98099999999999998,"5min":1.4179999999999999,"15min":1.5329999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.75700000000000001,"15min":0.76500000000000001},"min":{"1min":0.64900000000000002,"5min":0.505,"15min":0.505},"max":{"1min":0.93999999999999995,"5min":1.0449999999999999,"15min":1.647},"last":0.86099999999999999}]},{"osd":3,"last update":"Thu Jan 11 09:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.76300000000000001,"15min":0.76800000000000002},"min":{"1min":0.622,"5min":0.46899999999999997,"15min":0.45200000000000001},"max":{"1min":1.012,"5min":1.014,"15min":1.6140000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.77400000000000002,"15min":0.78600000000000003},"min":{"1min":0.60099999999999998,"5min":0.54200000000000004,"15min":0.48499999999999999},"max":{"1min":0.93200000000000005,"5min":1.165,"15min":1.4950000000000001},"last":0.80000000000000004}]},{"osd":4,"last update":"Thu Jan 11 09:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.41199999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.91700000000000004,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.61699999999999999,"15min":0.61399999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84899999999999998,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.54400000000000004}]},{"osd":5,"last update":"Thu Jan 11 09:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.67600000000000005,"15min":0.66100000000000003},"min":{"1min":0.441,"5min":0.41499999999999998,"15min":0.40000000000000002},"max":{"1min":0.97899999999999998,"5min":1.369,"15min":1.369},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71499999999999997,"15min":0.71299999999999997},"min":{"1min":0.51700000000000002,"5min":0.46700000000000003,"15min":0.45400000000000001},"max":{"1min":0.999,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.82699999999999996}]},{"osd":7,"last update":"Thu Jan 11 09:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.72299999999999998,"15min":0.71299999999999997},"min":{"1min":0.45600000000000002,"5min":0.45100000000000001,"15min":0.44400000000000001},"max":{"1min":1.089,"5min":1.484,"15min":1.484},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78200000000000003,"15min":0.77400000000000002},"min":{"1min":0.55300000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.1379999999999999,"5min":1.603,"15min":1.603},"last":0.89500000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575954,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307904,"kb_used_data":2424,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063936,"statfs":{"total":96636764160,"available":96321470464,"internally_reserved":0,"allocated":2482176,"data_stored":1176918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51800000000000002,"15min":0.51300000000000001},"min":{"1min":0.33800000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.73399999999999999,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.621,"15min":0.623},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.95099999999999996},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 11 09:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.372,"5min":0.35899999999999999,"15min":0.35099999999999998},"max":{"1min":0.82399999999999995,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.59399999999999997},"min":{"1min":0.40799999999999997,"5min":0.40600000000000003,"15min":0.39400000000000002},"max":{"1min":0.81799999999999995,"5min":0.83899999999999997,"15min":0.876},"last":0.435}]},{"osd":3,"last update":"Thu Jan 11 09:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.64900000000000002,"15min":0.65100000000000002},"min":{"1min":0.497,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.82699999999999996,"5min":0.91600000000000004,"15min":0.96699999999999997},"last":0.625},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.58599999999999997,"15min":0.58499999999999996},"min":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.53300000000000003}]},{"osd":4,"last update":"Thu Jan 11 09:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81999999999999995,"15min":0.82099999999999995},"min":{"1min":0.54000000000000004,"5min":0.53500000000000003,"15min":0.505},"max":{"1min":1.214,"5min":1.214,"15min":1.2969999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.86099999999999999,"15min":0.86199999999999999},"min":{"1min":0.68600000000000005,"5min":0.66400000000000003,"15min":0.61699999999999999},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1890000000000001},"last":0.75}]},{"osd":5,"last update":"Thu Jan 11 09:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.83699999999999997,"15min":0.82999999999999996},"min":{"1min":0.64400000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.82699999999999996,"15min":0.80800000000000005},"min":{"1min":0.63800000000000001,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.54200000000000004}]},{"osd":6,"last update":"Thu Jan 11 09:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.84899999999999998,"15min":0.84599999999999997},"min":{"1min":0.60699999999999998,"5min":0.59599999999999997,"15min":0.56799999999999995},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.85799999999999998,"15min":0.85699999999999998},"min":{"1min":0.748,"5min":0.626,"15min":0.55900000000000005},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.64300000000000002}]},{"osd":7,"last update":"Thu Jan 11 09:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.89200000000000002,"15min":0.89000000000000001},"min":{"1min":0.77400000000000002,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.86699999999999999,"15min":0.85899999999999999},"min":{"1min":0.70199999999999996,"5min":0.59299999999999997,"15min":0.54500000000000004},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.60099999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575953,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308476,"kb_used_data":2996,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063364,"statfs":{"total":96636764160,"available":96320884736,"internally_reserved":0,"allocated":3067904,"data_stored":1765772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51300000000000001,"15min":0.53000000000000003},"min":{"1min":0.35599999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.63900000000000001,"5min":0.84699999999999998,"15min":0.875},"last":0.629},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59399999999999997,"15min":0.60399999999999998},"min":{"1min":0.47399999999999998,"5min":0.41899999999999998,"15min":0.40100000000000002},"max":{"1min":0.80600000000000005,"5min":0.86399999999999999,"15min":0.874},"last":0.61099999999999999}]},{"osd":2,"last update":"Thu Jan 11 09:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56499999999999995,"15min":0.57599999999999996},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.36599999999999999},"max":{"1min":0.83099999999999996,"5min":0.84499999999999997,"15min":0.97499999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.42099999999999999,"5min":0.38500000000000001,"15min":0.38200000000000001},"max":{"1min":0.77000000000000002,"5min":0.80900000000000005,"15min":0.86399999999999999},"last":0.46400000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52900000000000003,"15min":0.55200000000000005},"min":{"1min":0.38400000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.79000000000000004,"5min":0.92600000000000005,"15min":0.95399999999999996},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64600000000000002,"15min":0.66000000000000003},"min":{"1min":0.47899999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.90300000000000002,"5min":0.92600000000000005,"15min":1.0389999999999999},"last":0.71499999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.82899999999999996,"15min":0.82299999999999995},"min":{"1min":0.65900000000000003,"5min":0.60299999999999998,"15min":0.48499999999999999},"max":{"1min":1.1339999999999999,"5min":1.1779999999999999,"15min":1.304},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.81299999999999994,"15min":0.80800000000000005},"min":{"1min":0.63300000000000001,"5min":0.58099999999999996,"15min":0.51400000000000001},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.58799999999999997}]},{"osd":5,"last update":"Thu Jan 11 09:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84599999999999997},"min":{"1min":0.65600000000000003,"5min":0.64600000000000002,"15min":0.53900000000000003},"max":{"1min":1.1140000000000001,"5min":1.194,"15min":1.194},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.84199999999999997,"15min":0.84599999999999997},"min":{"1min":0.65000000000000002,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.69499999999999995}]},{"osd":6,"last update":"Thu Jan 11 09:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.83599999999999997,"15min":0.83099999999999996},"min":{"1min":0.59599999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.86399999999999999,"15min":0.874},"min":{"1min":0.67000000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.1100000000000001,"5min":1.256,"15min":1.256},"last":0.78000000000000003}]},{"osd":7,"last update":"Thu Jan 11 09:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.86399999999999999,"15min":0.86899999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.58699999999999997},"max":{"1min":1.206,"5min":1.28,"15min":1.28},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.871,"15min":0.88100000000000001},"min":{"1min":0.63900000000000001,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.238,"5min":1.238,"15min":1.2410000000000001},"last":0.73499999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575952,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312000,"kb_used_data":2424,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059840,"statfs":{"total":96636764160,"available":96317276160,"internally_reserved":0,"allocated":2482176,"data_stored":1176048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.51300000000000001,"15min":0.50800000000000001},"min":{"1min":0.34200000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65200000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.371},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56100000000000005,"15min":0.56899999999999995},"min":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.34300000000000003},"max":{"1min":0.68200000000000005,"5min":1.131,"15min":1.6359999999999999},"last":0.57899999999999996}]},{"osd":1,"last update":"Thu Jan 11 09:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.442,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.93000000000000005},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.61299999999999999,"15min":0.60699999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.40000000000000002},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.145},"last":0.441}]},{"osd":3,"last update":"Thu Jan 11 09:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41599999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72699999999999998,"5min":0.81299999999999994,"15min":0.96399999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.66100000000000003,"15min":0.66500000000000004},"min":{"1min":0.52600000000000002,"5min":0.434,"15min":0.434},"max":{"1min":0.86699999999999999,"5min":0.97999999999999998,"15min":1.3620000000000001},"last":0.55700000000000005}]},{"osd":4,"last update":"Thu Jan 11 09:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81000000000000005,"15min":0.80700000000000005},"min":{"1min":0.65500000000000003,"5min":0.57799999999999996,"15min":0.54300000000000004},"max":{"1min":1.071,"5min":1.071,"15min":1.1519999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81399999999999995,"15min":0.81000000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.69199999999999995}]},{"osd":5,"last update":"Thu Jan 11 09:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.88600000000000001,"15min":0.88200000000000001},"min":{"1min":0.68500000000000005,"5min":0.67700000000000005,"15min":0.55800000000000005},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.429},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.87,"15min":0.87},"min":{"1min":0.73699999999999999,"5min":0.65200000000000002,"15min":0.61099999999999999},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.2170000000000001},"last":0.75600000000000001}]},{"osd":6,"last update":"Thu Jan 11 09:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.83799999999999997,"15min":0.83499999999999996},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.4139999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.876,"5min":0.876,"15min":0.877},"min":{"1min":0.66000000000000003,"5min":0.63200000000000001,"15min":0.60899999999999999},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.593},"last":0.82799999999999996}]},{"osd":7,"last update":"Thu Jan 11 09:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89900000000000002,"15min":0.90200000000000002},"min":{"1min":0.70899999999999996,"5min":0.60699999999999998,"15min":0.57299999999999995},"max":{"1min":1.1339999999999999,"5min":1.137,"15min":2.0539999999999998},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.88300000000000001,"15min":0.89200000000000002},"min":{"1min":0.63500000000000001,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2490000000000001},"last":0.86399999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575952,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299712,"kb_used_data":2424,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072128,"statfs":{"total":96636764160,"available":96329859072,"internally_reserved":0,"allocated":2482176,"data_stored":1175576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51300000000000001,"15min":0.52600000000000002},"min":{"1min":0.33900000000000002,"5min":0.32200000000000001,"15min":0.314},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.90000000000000002},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.433,"5min":0.40500000000000003,"15min":0.36099999999999999},"max":{"1min":0.90400000000000003,"5min":1.014,"15min":1.0449999999999999},"last":0.44600000000000001}]},{"osd":1,"last update":"Thu Jan 11 09:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51100000000000001,"15min":0.51500000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.67500000000000004,"5min":0.84199999999999997,"15min":1.304},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59499999999999997,"15min":0.61699999999999999},"min":{"1min":0.41599999999999998,"5min":0.38700000000000001,"15min":0.371},"max":{"1min":0.73599999999999999,"5min":0.84099999999999997,"15min":1.4590000000000001},"last":0.48699999999999999}]},{"osd":2,"last update":"Thu Jan 11 09:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.60899999999999999,"15min":0.61499999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.751,"5min":0.93700000000000006,"15min":1.04},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.66000000000000003,"15min":0.66300000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.44700000000000001},"max":{"1min":0.83799999999999997,"5min":0.89800000000000002,"15min":1.0069999999999999},"last":0.64600000000000002}]},{"osd":4,"last update":"Thu Jan 11 09:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.79300000000000004,"15min":0.79500000000000004},"min":{"1min":0.63100000000000001,"5min":0.53000000000000003,"15min":0.50600000000000001},"max":{"1min":0.90300000000000002,"5min":1.206,"15min":1.206},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80500000000000005,"15min":0.82299999999999995},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.171,"5min":1.171,"15min":1.1830000000000001},"last":1.0569999999999999}]},{"osd":5,"last update":"Thu Jan 11 09:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.83499999999999996,"15min":0.82899999999999996},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.123,"5min":1.1950000000000001,"15min":1.234},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.82899999999999996,"15min":0.85499999999999998},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.0940000000000001,"5min":1.117,"15min":1.3440000000000001},"last":1.1160000000000001}]},{"osd":6,"last update":"Thu Jan 11 09:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.84799999999999998,"15min":0.86499999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.58599999999999997},"max":{"1min":1.05,"5min":1.175,"15min":1.335},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.82299999999999995,"15min":0.82999999999999996},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.94599999999999995,"5min":1.181,"15min":1.2190000000000001},"last":0.95799999999999996}]},{"osd":7,"last update":"Thu Jan 11 09:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.87,"15min":0.875},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.052,"5min":1.264,"15min":1.264},"last":1.0429999999999999},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.89900000000000002,"15min":0.90400000000000003},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.0720000000000001,"5min":1.24,"15min":1.24},"last":0.998}]}]},{"osd":4,"up_from":13,"seq":55834575953,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307908,"kb_used_data":2428,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063932,"statfs":{"total":96636764160,"available":96321466368,"internally_reserved":0,"allocated":2486272,"data_stored":1177472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.77100000000000002,"15min":0.77900000000000003},"min":{"1min":0.44800000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.91200000000000003,"5min":1.038,"15min":1.119},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78800000000000003,"15min":0.78300000000000003},"min":{"1min":0.53300000000000003,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":0.70699999999999996}]},{"osd":1,"last update":"Thu Jan 11 09:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.75900000000000001,"15min":0.75600000000000001},"min":{"1min":0.46899999999999997,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.0190000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.74299999999999999,"15min":0.73999999999999999},"min":{"1min":0.50900000000000001,"5min":0.48799999999999999,"15min":0.45000000000000001},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":0.65700000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.77600000000000002,"15min":0.75700000000000001},"min":{"1min":0.52500000000000002,"5min":0.47099999999999997,"15min":0.44700000000000001},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.76400000000000001,"15min":0.75600000000000001},"min":{"1min":0.54700000000000004,"5min":0.53400000000000003,"15min":0.48999999999999999},"max":{"1min":1.0109999999999999,"5min":1.0249999999999999,"15min":1.0700000000000001},"last":0.72399999999999998}]},{"osd":3,"last update":"Thu Jan 11 09:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.75900000000000001,"15min":0.754},"min":{"1min":0.54000000000000004,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.80200000000000005,"15min":0.78700000000000003},"min":{"1min":0.57899999999999996,"5min":0.55600000000000005,"15min":0.53300000000000003},"max":{"1min":2.1150000000000002,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.60299999999999998}]},{"osd":5,"last update":"Thu Jan 11 09:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.63,"15min":0.62},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78300000000000003,"5min":0.97199999999999998,"15min":1.0049999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.63400000000000001}]},{"osd":6,"last update":"Thu Jan 11 09:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.66900000000000004,"15min":0.66600000000000004},"min":{"1min":0.46300000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.752,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.58599999999999997}]},{"osd":7,"last update":"Thu Jan 11 09:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.69799999999999995,"15min":0.69899999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.75,"15min":0.751},"min":{"1min":0.436,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":2.052,"5min":2.052,"15min":2.052},"last":0.74099999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575953,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311988,"kb_used_data":2412,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059852,"statfs":{"total":96636764160,"available":96317288448,"internally_reserved":0,"allocated":2469888,"data_stored":1175382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.749,"15min":0.73599999999999999},"min":{"1min":0.50900000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.88800000000000001,"5min":0.97799999999999998,"15min":1.0129999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.72599999999999998,"15min":0.73399999999999999},"min":{"1min":0.55400000000000005,"5min":0.46700000000000003,"15min":0.45600000000000002},"max":{"1min":0.94099999999999995,"5min":1.048,"15min":1.0660000000000001},"last":0.64000000000000001}]},{"osd":1,"last update":"Thu Jan 11 09:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.73199999999999998,"15min":0.74099999999999999},"min":{"1min":0.54200000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.91800000000000004,"5min":0.94699999999999995,"15min":0.97699999999999998},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.754,"15min":0.749},"min":{"1min":0.56999999999999995,"5min":0.53600000000000003,"15min":0.46700000000000003},"max":{"1min":0.96499999999999997,"5min":0.98499999999999999,"15min":1.0700000000000001},"last":0.55800000000000005}]},{"osd":2,"last update":"Thu Jan 11 09:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.505,"5min":0.48699999999999999,"15min":0.47899999999999998},"max":{"1min":0.93400000000000005,"5min":0.94299999999999995,"15min":1.028},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.748,"15min":0.749},"min":{"1min":0.59499999999999997,"5min":0.53200000000000003,"15min":0.505},"max":{"1min":0.91300000000000003,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.68100000000000005}]},{"osd":3,"last update":"Thu Jan 11 09:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.755,"15min":0.76800000000000002},"min":{"1min":0.61299999999999999,"5min":0.54200000000000004,"15min":0.49299999999999999},"max":{"1min":1.002,"5min":1.0049999999999999,"15min":1.022},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.78700000000000003,"15min":0.77500000000000002},"min":{"1min":0.64400000000000002,"5min":0.59799999999999998,"15min":0.52100000000000002},"max":{"1min":0.96599999999999997,"5min":1.004,"15min":1.1000000000000001},"last":0.80600000000000005}]},{"osd":4,"last update":"Thu Jan 11 09:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.61299999999999999,"15min":0.60999999999999999},"min":{"1min":0.41999999999999998,"5min":0.41699999999999998,"15min":0.41599999999999998},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.97199999999999998},"last":0.62},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.45100000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.84799999999999998,"5min":0.94499999999999995,"15min":1.042},"last":0.60999999999999999}]},{"osd":6,"last update":"Thu Jan 11 09:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.629,"15min":0.64500000000000002},"min":{"1min":0.5,"5min":0.44400000000000001,"15min":0.40999999999999998},"max":{"1min":0.83699999999999997,"5min":0.93100000000000005,"15min":1.0089999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.67700000000000005,"15min":0.68600000000000005},"min":{"1min":0.47799999999999998,"5min":0.46200000000000002,"15min":0.42799999999999999},"max":{"1min":0.91300000000000003,"5min":0.92100000000000004,"15min":1.0229999999999999},"last":0.79100000000000004}]},{"osd":7,"last update":"Thu Jan 11 09:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.72099999999999997,"15min":0.73699999999999999},"min":{"1min":0.51700000000000002,"5min":0.46800000000000003,"15min":0.434},"max":{"1min":0.93000000000000005,"5min":1.0309999999999999,"15min":1.0760000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.76600000000000001,"15min":0.78200000000000003},"min":{"1min":0.60999999999999999,"5min":0.5,"15min":0.47599999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0469999999999999},"last":0.59199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:43:46.482 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:43:46.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.724+0000 7f4bd05ff700 1 Processor -- start 2024-01-11T09:43:46.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.724+0000 7f4bd05ff700 1 -- start start 2024-01-11T09:43:46.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.725+0000 7f4bd05ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4bc8069d40 0x7f4bc80b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:46.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.726+0000 7f4bd05ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc8157390 con 0x7f4bc80c8620 2024-01-11T09:43:46.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.726+0000 7f4bd05ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc806a200 con 0x7f4bc80c89f0 2024-01-11T09:43:46.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.726+0000 7f4bd05ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc8156d40 con 0x7f4bc8069d40 2024-01-11T09:43:46.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.726+0000 7f4bce39b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4bc80c8620 0x7f4bc80af970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:32904/0 (socket says 172.21.15.27:32904) 2024-01-11T09:43:46.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.726+0000 7f4bce39b700 1 -- 172.21.15.27:0/1612950035 learned_addr learned my addr 172.21.15.27:0/1612950035 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:46.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.726+0000 7f4bcdb9a700 1 --2- 172.21.15.27:0/1612950035 >> v2:172.21.15.77:3300/0 conn(0x7f4bc8069d40 0x7f4bc80b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:46.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.726+0000 7f4bcd399700 1 -- 172.21.15.27:0/1612950035 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679535349 0 0) 0x7f4bc8157390 con 0x7f4bc80c8620 2024-01-11T09:43:46.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.726+0000 7f4bcd399700 1 -- 172.21.15.27:0/1612950035 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bb4003570 con 0x7f4bc80c8620 2024-01-11T09:43:46.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.727+0000 7f4bcdb9a700 1 -- 172.21.15.27:0/1612950035 >> v1:172.21.15.77:6789/0 conn(0x7f4bc80c89f0 legacy=0x7f4bc80accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:46.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.727+0000 7f4bcdb9a700 1 -- 172.21.15.27:0/1612950035 >> v1:172.21.15.27:6789/0 conn(0x7f4bc80c8620 legacy=0x7f4bc80af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:46.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.727+0000 7f4bcd399700 1 -- 172.21.15.27:0/1612950035 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 479175857 0 0) 0x7f4bb4003570 con 0x7f4bc80c8620 2024-01-11T09:43:46.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.727+0000 7f4bcdb9a700 1 -- 172.21.15.27:0/1612950035 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc8157020 con 0x7f4bc8069d40 2024-01-11T09:43:46.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.727+0000 7f4bcdb9a700 1 --2- 172.21.15.27:0/1612950035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc8069d40 0x7f4bc80b02d0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f4bb80098e0 tx=0x7f4bb80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2f3afac4d58800e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:46.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.727+0000 7f4bcd399700 1 -- 172.21.15.27:0/1612950035 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4001470 con 0x7f4bc8069d40 2024-01-11T09:43:46.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.727+0000 7f4bcd399700 1 -- 172.21.15.27:0/1612950035 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb8011420 con 0x7f4bc8069d40 2024-01-11T09:43:46.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.727+0000 7f4bcd399700 1 -- 172.21.15.27:0/1612950035 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb80117a0 con 0x7f4bc8069d40 2024-01-11T09:43:46.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.728+0000 7f4bd05ff700 1 -- 172.21.15.27:0/1612950035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc8069d40 msgr2=0x7f4bc80b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:46.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.728+0000 7f4bd05ff700 1 --2- 172.21.15.27:0/1612950035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc8069d40 0x7f4bc80b02d0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f4bb80098e0 tx=0x7f4bb80091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.728+0000 7f4bd05ff700 1 -- 172.21.15.27:0/1612950035 shutdown_connections 2024-01-11T09:43:46.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.728+0000 7f4bd05ff700 1 --2- 172.21.15.27:0/1612950035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc8069d40 0x7f4bc80b02d0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.728+0000 7f4bd05ff700 1 -- 172.21.15.27:0/1612950035 >> 172.21.15.27:0/1612950035 conn(0x7f4bc80680e0 msgr2=0x7f4bc8069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:46.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.728+0000 7f4bd05ff700 1 -- 172.21.15.27:0/1612950035 shutdown_connections 2024-01-11T09:43:46.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.728+0000 7f4bd05ff700 1 -- 172.21.15.27:0/1612950035 wait complete. 2024-01-11T09:43:46.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.729+0000 7f4bd05ff700 1 Processor -- start 2024-01-11T09:43:46.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.730+0000 7f4bd05ff700 1 -- start start 2024-01-11T09:43:46.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.731+0000 7f4bd05ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bc8069d40 0x7f4bc8151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:46.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.731+0000 7f4bd05ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc80c8620 0x7f4bc8157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:46.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.731+0000 7f4bd05ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bc80c89f0 0x7f4bc81584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:46.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.731+0000 7f4bd05ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80c7020 con 0x7f4bc8069d40 2024-01-11T09:43:46.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.731+0000 7f4bd05ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80c6410 con 0x7f4bc80c8620 2024-01-11T09:43:46.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.731+0000 7f4bd05ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80c66f0 con 0x7f4bc80c89f0 2024-01-11T09:43:46.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.731+0000 7f4bce39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bc8069d40 0x7f4bc8151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:46.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.731+0000 7f4bce39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bc8069d40 0x7f4bc8151e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57414/0 (socket says 172.21.15.27:57414) 2024-01-11T09:43:46.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.731+0000 7f4bce39b700 1 -- 172.21.15.27:0/1318231026 learned_addr learned my addr 172.21.15.27:0/1318231026 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:46.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.731+0000 7f4bcdb9a700 1 --2- 172.21.15.27:0/1318231026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc80c8620 0x7f4bc8157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:46.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.731+0000 7f4bceb9c700 1 --2- 172.21.15.27:0/1318231026 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bc80c89f0 0x7f4bc81584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:46.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.731+0000 7f4bce39b700 1 -- 172.21.15.27:0/1318231026 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bc80c89f0 msgr2=0x7f4bc81584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:46.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.731+0000 7f4bce39b700 1 --2- 172.21.15.27:0/1318231026 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bc80c89f0 0x7f4bc81584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.731+0000 7f4bce39b700 1 -- 172.21.15.27:0/1318231026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc80c8620 msgr2=0x7f4bc8157f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:46.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.731+0000 7f4bce39b700 1 --2- 172.21.15.27:0/1318231026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc80c8620 0x7f4bc8157f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.731+0000 7f4bce39b700 1 -- 172.21.15.27:0/1318231026 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb8009420 con 0x7f4bc8069d40 2024-01-11T09:43:46.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.732+0000 7f4bce39b700 1 --2- 172.21.15.27:0/1318231026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bc8069d40 0x7f4bc8151e90 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f4bc400cb90 tx=0x7f4bc4001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:46.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.732+0000 7f4bbe7fc700 1 -- 172.21.15.27:0/1318231026 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4001d10 con 0x7f4bc8069d40 2024-01-11T09:43:46.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.732+0000 7f4bbe7fc700 1 -- 172.21.15.27:0/1318231026 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc4007360 con 0x7f4bc8069d40 2024-01-11T09:43:46.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.732+0000 7f4bbe7fc700 1 -- 172.21.15.27:0/1318231026 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4007700 con 0x7f4bc8069d40 2024-01-11T09:43:46.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.732+0000 7f4bd05ff700 1 -- 172.21.15.27:0/1318231026 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc80c69d0 con 0x7f4bc8069d40 2024-01-11T09:43:46.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.732+0000 7f4bd05ff700 1 -- 172.21.15.27:0/1318231026 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc81523d0 con 0x7f4bc8069d40 2024-01-11T09:43:46.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.733+0000 7f4bd05ff700 1 -- 172.21.15.27:0/1318231026 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc80718d0 con 0x7f4bc8069d40 2024-01-11T09:43:46.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.734+0000 7f4bbe7fc700 1 -- 172.21.15.27:0/1318231026 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4bc4004a40 con 0x7f4bc8069d40 2024-01-11T09:43:46.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.736+0000 7f4bbe7fc700 1 --2- 172.21.15.27:0/1318231026 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4bac066f30 0x7f4bac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:46.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.736+0000 7f4bbe7fc700 1 -- 172.21.15.27:0/1318231026 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f4bc4083d70 con 0x7f4bc8069d40 2024-01-11T09:43:46.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.736+0000 7f4bcdb9a700 1 --2- 172.21.15.27:0/1318231026 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4bac066f30 0x7f4bac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:46.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.737+0000 7f4bcdb9a700 1 --2- 172.21.15.27:0/1318231026 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4bac066f30 0x7f4bac0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8009fd0 tx=0x7f4bb8016000 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:46.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.740+0000 7f4bbe7fc700 1 -- 172.21.15.27:0/1318231026 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4bc4051010 con 0x7f4bc8069d40 2024-01-11T09:43:46.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.942+0000 7f4bd05ff700 1 -- 172.21.15.27:0/1318231026 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4bc80c7160 con 0x7f4bac066f30 2024-01-11T09:43:46.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.951+0000 7f4bbe7fc700 1 -- 172.21.15.27:0/1318231026 <== mgr.8294 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289569 (secure 0 0 0) 0x7f4bb8011b10 con 0x7f4bac066f30 2024-01-11T09:43:46.947 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:43:46.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.955+0000 7f4bd05ff700 1 -- 172.21.15.27:0/1318231026 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4bac066f30 msgr2=0x7f4bac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.955+0000 7f4bd05ff700 1 --2- 172.21.15.27:0/1318231026 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4bac066f30 0x7f4bac0693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8009fd0 tx=0x7f4bb8016000 comp rx=0 tx=0).stop 2024-01-11T09:43:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.955+0000 7f4bd05ff700 1 -- 172.21.15.27:0/1318231026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bc8069d40 msgr2=0x7f4bc8151e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.955+0000 7f4bd05ff700 1 --2- 172.21.15.27:0/1318231026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bc8069d40 0x7f4bc8151e90 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f4bc400cb90 tx=0x7f4bc4001a80 comp rx=0 tx=0).stop 2024-01-11T09:43:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.956+0000 7f4bd05ff700 1 -- 172.21.15.27:0/1318231026 shutdown_connections 2024-01-11T09:43:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.956+0000 7f4bd05ff700 1 --2- 172.21.15.27:0/1318231026 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4bc80c89f0 0x7f4bc81584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.956+0000 7f4bd05ff700 1 --2- 172.21.15.27:0/1318231026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bc8069d40 0x7f4bc8151e90 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.956+0000 7f4bd05ff700 1 --2- 172.21.15.27:0/1318231026 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4bac066f30 0x7f4bac0693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.956+0000 7f4bd05ff700 1 --2- 172.21.15.27:0/1318231026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bc80c8620 0x7f4bc8157f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.956+0000 7f4bd05ff700 1 -- 172.21.15.27:0/1318231026 >> 172.21.15.27:0/1318231026 conn(0x7f4bc80680e0 msgr2=0x7f4bc806ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.956+0000 7f4bd05ff700 1 -- 172.21.15.27:0/1318231026 shutdown_connections 2024-01-11T09:43:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:46.956+0000 7f4bd05ff700 1 -- 172.21.15.27:0/1318231026 wait complete. 2024-01-11T09:43:46.950 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:43:46.966 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-11T09:43:45.298944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2096,"num_read_kb":4396,"num_write":615,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":382,"ondisk_log_size":382,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":78,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2460264,"kb_used_data":20520,"kb_used_omap":0,"kb_used_meta":2439680,"kb_avail":752514456,"statfs":{"total":773094113280,"available":770574802944,"internally_reserved":0,"allocated":21012480,"data_stored":10588404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2498232320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.254600"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090312+0000","last_change":"2024-01-11T09:43:42.090312+0000","last_active":"2024-01-11T09:43:42.090312+0000","last_peered":"2024-01-11T09:43:42.090312+0000","last_clean":"2024-01-11T09:43:42.090312+0000","last_became_active":"2024-01-11T09:43:42.089816+0000","last_became_peered":"2024-01-11T09:43:42.089816+0000","last_unstale":"2024-01-11T09:43:42.090312+0000","last_undegraded":"2024-01-11T09:43:42.090312+0000","last_fullsized":"2024-01-11T09:43:42.090312+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:18:25.046984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.089184+0000","last_change":"2024-01-11T09:43:42.089184+0000","last_active":"2024-01-11T09:43:42.089184+0000","last_peered":"2024-01-11T09:43:42.089184+0000","last_clean":"2024-01-11T09:43:42.089184+0000","last_became_active":"2024-01-11T09:43:42.088486+0000","last_became_peered":"2024-01-11T09:43:42.088486+0000","last_unstale":"2024-01-11T09:43:42.089184+0000","last_undegraded":"2024-01-11T09:43:42.089184+0000","last_fullsized":"2024-01-11T09:43:42.089184+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:01:29.860733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.086612+0000","last_change":"2024-01-11T09:43:42.086612+0000","last_active":"2024-01-11T09:43:42.086612+0000","last_peered":"2024-01-11T09:43:42.086612+0000","last_clean":"2024-01-11T09:43:42.086612+0000","last_became_active":"2024-01-11T09:43:42.086316+0000","last_became_peered":"2024-01-11T09:43:42.086316+0000","last_unstale":"2024-01-11T09:43:42.086612+0000","last_undegraded":"2024-01-11T09:43:42.086612+0000","last_fullsized":"2024-01-11T09:43:42.086612+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:00:48.167638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.076216+0000","last_change":"2024-01-11T09:43:42.076216+0000","last_active":"2024-01-11T09:43:42.076216+0000","last_peered":"2024-01-11T09:43:42.076216+0000","last_clean":"2024-01-11T09:43:42.076216+0000","last_became_active":"2024-01-11T09:43:42.074166+0000","last_became_peered":"2024-01-11T09:43:42.074166+0000","last_unstale":"2024-01-11T09:43:42.076216+0000","last_undegraded":"2024-01-11T09:43:42.076216+0000","last_fullsized":"2024-01-11T09:43:42.076216+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:48:40.643154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.077822+0000","last_change":"2024-01-11T09:43:42.077822+0000","last_active":"2024-01-11T09:43:42.077822+0000","last_peered":"2024-01-11T09:43:42.077822+0000","last_clean":"2024-01-11T09:43:42.077822+0000","last_became_active":"2024-01-11T09:43:42.077370+0000","last_became_peered":"2024-01-11T09:43:42.077370+0000","last_unstale":"2024-01-11T09:43:42.077822+0000","last_undegraded":"2024-01-11T09:43:42.077822+0000","last_fullsized":"2024-01-11T09:43:42.077822+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:27:02.922803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.078570+0000","last_change":"2024-01-11T09:43:42.078570+0000","last_active":"2024-01-11T09:43:42.078570+0000","last_peered":"2024-01-11T09:43:42.078570+0000","last_clean":"2024-01-11T09:43:42.078570+0000","last_became_active":"2024-01-11T09:43:42.074654+0000","last_became_peered":"2024-01-11T09:43:42.074654+0000","last_unstale":"2024-01-11T09:43:42.078570+0000","last_undegraded":"2024-01-11T09:43:42.078570+0000","last_fullsized":"2024-01-11T09:43:42.078570+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:48:31.109585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090878+0000","last_change":"2024-01-11T09:43:42.087246+0000","last_active":"2024-01-11T09:43:42.090878+0000","last_peered":"2024-01-11T09:43:42.090878+0000","last_clean":"2024-01-11T09:43:42.090878+0000","last_became_active":"2024-01-11T09:43:42.086982+0000","last_became_peered":"2024-01-11T09:43:42.086982+0000","last_unstale":"2024-01-11T09:43:42.090878+0000","last_undegraded":"2024-01-11T09:43:42.090878+0000","last_fullsized":"2024-01-11T09:43:42.090878+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:28:58.773625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090849+0000","last_change":"2024-01-11T09:43:42.081646+0000","last_active":"2024-01-11T09:43:42.090849+0000","last_peered":"2024-01-11T09:43:42.090849+0000","last_clean":"2024-01-11T09:43:42.090849+0000","last_became_active":"2024-01-11T09:43:42.081233+0000","last_became_peered":"2024-01-11T09:43:42.081233+0000","last_unstale":"2024-01-11T09:43:42.090849+0000","last_undegraded":"2024-01-11T09:43:42.090849+0000","last_fullsized":"2024-01-11T09:43:42.090849+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:47.767999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.080980+0000","last_change":"2024-01-11T09:43:42.079368+0000","last_active":"2024-01-11T09:43:42.080980+0000","last_peered":"2024-01-11T09:43:42.080980+0000","last_clean":"2024-01-11T09:43:42.080980+0000","last_became_active":"2024-01-11T09:43:42.077324+0000","last_became_peered":"2024-01-11T09:43:42.077324+0000","last_unstale":"2024-01-11T09:43:42.080980+0000","last_undegraded":"2024-01-11T09:43:42.080980+0000","last_fullsized":"2024-01-11T09:43:42.080980+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:10:05.795665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.085618+0000","last_change":"2024-01-11T09:43:42.085618+0000","last_active":"2024-01-11T09:43:42.085618+0000","last_peered":"2024-01-11T09:43:42.085618+0000","last_clean":"2024-01-11T09:43:42.085618+0000","last_became_active":"2024-01-11T09:43:42.085261+0000","last_became_peered":"2024-01-11T09:43:42.085261+0000","last_unstale":"2024-01-11T09:43:42.085618+0000","last_undegraded":"2024-01-11T09:43:42.085618+0000","last_fullsized":"2024-01-11T09:43:42.085618+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:27:15.928486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.087295+0000","last_change":"2024-01-11T09:43:42.074985+0000","last_active":"2024-01-11T09:43:42.087295+0000","last_peered":"2024-01-11T09:43:42.087295+0000","last_clean":"2024-01-11T09:43:42.087295+0000","last_became_active":"2024-01-11T09:43:42.074654+0000","last_became_peered":"2024-01-11T09:43:42.074654+0000","last_unstale":"2024-01-11T09:43:42.087295+0000","last_undegraded":"2024-01-11T09:43:42.087295+0000","last_fullsized":"2024-01-11T09:43:42.087295+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:35:36.079408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.081077+0000","last_change":"2024-01-11T09:43:42.081077+0000","last_active":"2024-01-11T09:43:42.081077+0000","last_peered":"2024-01-11T09:43:42.081077+0000","last_clean":"2024-01-11T09:43:42.081077+0000","last_became_active":"2024-01-11T09:43:42.075921+0000","last_became_peered":"2024-01-11T09:43:42.075921+0000","last_unstale":"2024-01-11T09:43:42.081077+0000","last_undegraded":"2024-01-11T09:43:42.081077+0000","last_fullsized":"2024-01-11T09:43:42.081077+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:47.767999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.079166+0000","last_change":"2024-01-11T09:43:42.079166+0000","last_active":"2024-01-11T09:43:42.079166+0000","last_peered":"2024-01-11T09:43:42.079166+0000","last_clean":"2024-01-11T09:43:42.079166+0000","last_became_active":"2024-01-11T09:43:42.078888+0000","last_became_peered":"2024-01-11T09:43:42.078888+0000","last_unstale":"2024-01-11T09:43:42.079166+0000","last_undegraded":"2024-01-11T09:43:42.079166+0000","last_fullsized":"2024-01-11T09:43:42.079166+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:21:03.439272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.088266+0000","last_change":"2024-01-11T09:43:42.081968+0000","last_active":"2024-01-11T09:43:42.088266+0000","last_peered":"2024-01-11T09:43:42.088266+0000","last_clean":"2024-01-11T09:43:42.088266+0000","last_became_active":"2024-01-11T09:43:42.081370+0000","last_became_peered":"2024-01-11T09:43:42.081370+0000","last_unstale":"2024-01-11T09:43:42.088266+0000","last_undegraded":"2024-01-11T09:43:42.088266+0000","last_fullsized":"2024-01-11T09:43:42.088266+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:11:18.246714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.077258+0000","last_change":"2024-01-11T09:43:42.077258+0000","last_active":"2024-01-11T09:43:42.077258+0000","last_peered":"2024-01-11T09:43:42.077258+0000","last_clean":"2024-01-11T09:43:42.077258+0000","last_became_active":"2024-01-11T09:43:42.076980+0000","last_became_peered":"2024-01-11T09:43:42.076980+0000","last_unstale":"2024-01-11T09:43:42.077258+0000","last_undegraded":"2024-01-11T09:43:42.077258+0000","last_fullsized":"2024-01-11T09:43:42.077258+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:44:51.740724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090831+0000","last_change":"2024-01-11T09:43:42.081561+0000","last_active":"2024-01-11T09:43:42.090831+0000","last_peered":"2024-01-11T09:43:42.090831+0000","last_clean":"2024-01-11T09:43:42.090831+0000","last_became_active":"2024-01-11T09:43:42.081012+0000","last_became_peered":"2024-01-11T09:43:42.081012+0000","last_unstale":"2024-01-11T09:43:42.090831+0000","last_undegraded":"2024-01-11T09:43:42.090831+0000","last_fullsized":"2024-01-11T09:43:42.090831+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:35:05.107900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.081243+0000","last_change":"2024-01-11T09:43:42.081243+0000","last_active":"2024-01-11T09:43:42.081243+0000","last_peered":"2024-01-11T09:43:42.081243+0000","last_clean":"2024-01-11T09:43:42.081243+0000","last_became_active":"2024-01-11T09:43:42.076361+0000","last_became_peered":"2024-01-11T09:43:42.076361+0000","last_unstale":"2024-01-11T09:43:42.081243+0000","last_undegraded":"2024-01-11T09:43:42.081243+0000","last_fullsized":"2024-01-11T09:43:42.081243+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:39:56.524103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.081190+0000","last_change":"2024-01-11T09:43:42.081190+0000","last_active":"2024-01-11T09:43:42.081190+0000","last_peered":"2024-01-11T09:43:42.081190+0000","last_clean":"2024-01-11T09:43:42.081190+0000","last_became_active":"2024-01-11T09:43:42.078543+0000","last_became_peered":"2024-01-11T09:43:42.078543+0000","last_unstale":"2024-01-11T09:43:42.081190+0000","last_undegraded":"2024-01-11T09:43:42.081190+0000","last_fullsized":"2024-01-11T09:43:42.081190+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:28:49.367007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.086232+0000","last_change":"2024-01-11T09:43:42.086232+0000","last_active":"2024-01-11T09:43:42.086232+0000","last_peered":"2024-01-11T09:43:42.086232+0000","last_clean":"2024-01-11T09:43:42.086232+0000","last_became_active":"2024-01-11T09:43:42.085935+0000","last_became_peered":"2024-01-11T09:43:42.085935+0000","last_unstale":"2024-01-11T09:43:42.086232+0000","last_undegraded":"2024-01-11T09:43:42.086232+0000","last_fullsized":"2024-01-11T09:43:42.086232+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:46:50.210897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.074134+0000","last_change":"2024-01-11T09:43:42.074134+0000","last_active":"2024-01-11T09:43:42.074134+0000","last_peered":"2024-01-11T09:43:42.074134+0000","last_clean":"2024-01-11T09:43:42.074134+0000","last_became_active":"2024-01-11T09:43:42.072966+0000","last_became_peered":"2024-01-11T09:43:42.072966+0000","last_unstale":"2024-01-11T09:43:42.074134+0000","last_undegraded":"2024-01-11T09:43:42.074134+0000","last_fullsized":"2024-01-11T09:43:42.074134+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:09:24.887933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.077625+0000","last_change":"2024-01-11T09:43:42.077625+0000","last_active":"2024-01-11T09:43:42.077625+0000","last_peered":"2024-01-11T09:43:42.077625+0000","last_clean":"2024-01-11T09:43:42.077625+0000","last_became_active":"2024-01-11T09:43:42.077041+0000","last_became_peered":"2024-01-11T09:43:42.077041+0000","last_unstale":"2024-01-11T09:43:42.077625+0000","last_undegraded":"2024-01-11T09:43:42.077625+0000","last_fullsized":"2024-01-11T09:43:42.077625+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:01:02.801676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.089655+0000","last_change":"2024-01-11T09:43:42.089655+0000","last_active":"2024-01-11T09:43:42.089655+0000","last_peered":"2024-01-11T09:43:42.089655+0000","last_clean":"2024-01-11T09:43:42.089655+0000","last_became_active":"2024-01-11T09:43:42.089316+0000","last_became_peered":"2024-01-11T09:43:42.089316+0000","last_unstale":"2024-01-11T09:43:42.089655+0000","last_undegraded":"2024-01-11T09:43:42.089655+0000","last_fullsized":"2024-01-11T09:43:42.089655+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:48.201216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.091961+0000","last_change":"2024-01-11T09:43:42.091961+0000","last_active":"2024-01-11T09:43:42.091961+0000","last_peered":"2024-01-11T09:43:42.091961+0000","last_clean":"2024-01-11T09:43:42.091961+0000","last_became_active":"2024-01-11T09:43:42.091668+0000","last_became_peered":"2024-01-11T09:43:42.091668+0000","last_unstale":"2024-01-11T09:43:42.091961+0000","last_undegraded":"2024-01-11T09:43:42.091961+0000","last_fullsized":"2024-01-11T09:43:42.091961+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:27:02.922803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090747+0000","last_change":"2024-01-11T09:43:42.081331+0000","last_active":"2024-01-11T09:43:42.090747+0000","last_peered":"2024-01-11T09:43:42.090747+0000","last_clean":"2024-01-11T09:43:42.090747+0000","last_became_active":"2024-01-11T09:43:42.080430+0000","last_became_peered":"2024-01-11T09:43:42.080430+0000","last_unstale":"2024-01-11T09:43:42.090747+0000","last_undegraded":"2024-01-11T09:43:42.090747+0000","last_fullsized":"2024-01-11T09:43:42.090747+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:21:03.439272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.096490+0000","last_change":"2024-01-11T09:43:42.096490+0000","last_active":"2024-01-11T09:43:42.096490+0000","last_peered":"2024-01-11T09:43:42.096490+0000","last_clean":"2024-01-11T09:43:42.096490+0000","last_became_active":"2024-01-11T09:43:42.096093+0000","last_became_peered":"2024-01-11T09:43:42.096093+0000","last_unstale":"2024-01-11T09:43:42.096490+0000","last_undegraded":"2024-01-11T09:43:42.096490+0000","last_fullsized":"2024-01-11T09:43:42.096490+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:56:51.522681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.087829+0000","last_change":"2024-01-11T09:43:42.081214+0000","last_active":"2024-01-11T09:43:42.087829+0000","last_peered":"2024-01-11T09:43:42.087829+0000","last_clean":"2024-01-11T09:43:42.087829+0000","last_became_active":"2024-01-11T09:43:42.080960+0000","last_became_peered":"2024-01-11T09:43:42.080960+0000","last_unstale":"2024-01-11T09:43:42.087829+0000","last_undegraded":"2024-01-11T09:43:42.087829+0000","last_fullsized":"2024-01-11T09:43:42.087829+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:39:39.259932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.080696+0000","last_change":"2024-01-11T09:43:42.080696+0000","last_active":"2024-01-11T09:43:42.080696+0000","last_peered":"2024-01-11T09:43:42.080696+0000","last_clean":"2024-01-11T09:43:42.080696+0000","last_became_active":"2024-01-11T09:43:42.080397+0000","last_became_peered":"2024-01-11T09:43:42.080397+0000","last_unstale":"2024-01-11T09:43:42.080696+0000","last_undegraded":"2024-01-11T09:43:42.080696+0000","last_fullsized":"2024-01-11T09:43:42.080696+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:22.095497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.084735+0000","last_change":"2024-01-11T09:43:42.084735+0000","last_active":"2024-01-11T09:43:42.084735+0000","last_peered":"2024-01-11T09:43:42.084735+0000","last_clean":"2024-01-11T09:43:42.084735+0000","last_became_active":"2024-01-11T09:43:42.084421+0000","last_became_peered":"2024-01-11T09:43:42.084421+0000","last_unstale":"2024-01-11T09:43:42.084735+0000","last_undegraded":"2024-01-11T09:43:42.084735+0000","last_fullsized":"2024-01-11T09:43:42.084735+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:01:09.493984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.076782+0000","last_change":"2024-01-11T09:43:42.076782+0000","last_active":"2024-01-11T09:43:42.076782+0000","last_peered":"2024-01-11T09:43:42.076782+0000","last_clean":"2024-01-11T09:43:42.076782+0000","last_became_active":"2024-01-11T09:43:42.076230+0000","last_became_peered":"2024-01-11T09:43:42.076230+0000","last_unstale":"2024-01-11T09:43:42.076782+0000","last_undegraded":"2024-01-11T09:43:42.076782+0000","last_fullsized":"2024-01-11T09:43:42.076782+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:14:13.281545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090351+0000","last_change":"2024-01-11T09:43:42.081678+0000","last_active":"2024-01-11T09:43:42.090351+0000","last_peered":"2024-01-11T09:43:42.090351+0000","last_clean":"2024-01-11T09:43:42.090351+0000","last_became_active":"2024-01-11T09:43:42.081002+0000","last_became_peered":"2024-01-11T09:43:42.081002+0000","last_unstale":"2024-01-11T09:43:42.090351+0000","last_undegraded":"2024-01-11T09:43:42.090351+0000","last_fullsized":"2024-01-11T09:43:42.090351+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:00:48.167638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.087451+0000","last_change":"2024-01-11T09:43:42.087451+0000","last_active":"2024-01-11T09:43:42.087451+0000","last_peered":"2024-01-11T09:43:42.087451+0000","last_clean":"2024-01-11T09:43:42.087451+0000","last_became_active":"2024-01-11T09:43:42.085641+0000","last_became_peered":"2024-01-11T09:43:42.085641+0000","last_unstale":"2024-01-11T09:43:42.087451+0000","last_undegraded":"2024-01-11T09:43:42.087451+0000","last_fullsized":"2024-01-11T09:43:42.087451+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:01:02.801676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.079469+0000","last_change":"2024-01-11T09:43:42.079469+0000","last_active":"2024-01-11T09:43:42.079469+0000","last_peered":"2024-01-11T09:43:42.079469+0000","last_clean":"2024-01-11T09:43:42.079469+0000","last_became_active":"2024-01-11T09:43:42.076089+0000","last_became_peered":"2024-01-11T09:43:42.076089+0000","last_unstale":"2024-01-11T09:43:42.079469+0000","last_undegraded":"2024-01-11T09:43:42.079469+0000","last_fullsized":"2024-01-11T09:43:42.079469+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:05:23.006191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090611+0000","last_change":"2024-01-11T09:43:42.078350+0000","last_active":"2024-01-11T09:43:42.090611+0000","last_peered":"2024-01-11T09:43:42.090611+0000","last_clean":"2024-01-11T09:43:42.090611+0000","last_became_active":"2024-01-11T09:43:42.078073+0000","last_became_peered":"2024-01-11T09:43:42.078073+0000","last_unstale":"2024-01-11T09:43:42.090611+0000","last_undegraded":"2024-01-11T09:43:42.090611+0000","last_fullsized":"2024-01-11T09:43:42.090611+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:56:51.522681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"197'4","reported_seq":13,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.092235+0000","last_change":"2024-01-11T09:43:41.069190+0000","last_active":"2024-01-11T09:43:42.092235+0000","last_peered":"2024-01-11T09:43:42.092235+0000","last_clean":"2024-01-11T09:43:42.092235+0000","last_became_active":"2024-01-11T09:43:41.068950+0000","last_became_peered":"2024-01-11T09:43:41.068950+0000","last_unstale":"2024-01-11T09:43:42.092235+0000","last_undegraded":"2024-01-11T09:43:42.092235+0000","last_fullsized":"2024-01-11T09:43:42.092235+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:36:40.206301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.085696+0000","last_change":"2024-01-11T09:43:42.085696+0000","last_active":"2024-01-11T09:43:42.085696+0000","last_peered":"2024-01-11T09:43:42.085696+0000","last_clean":"2024-01-11T09:43:42.085696+0000","last_became_active":"2024-01-11T09:43:42.084810+0000","last_became_peered":"2024-01-11T09:43:42.084810+0000","last_unstale":"2024-01-11T09:43:42.085696+0000","last_undegraded":"2024-01-11T09:43:42.085696+0000","last_fullsized":"2024-01-11T09:43:42.085696+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:06:31.817227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"197'1","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.085029+0000","last_change":"2024-01-11T09:43:41.066752+0000","last_active":"2024-01-11T09:43:42.085029+0000","last_peered":"2024-01-11T09:43:42.085029+0000","last_clean":"2024-01-11T09:43:42.085029+0000","last_became_active":"2024-01-11T09:43:41.066473+0000","last_became_peered":"2024-01-11T09:43:41.066473+0000","last_unstale":"2024-01-11T09:43:42.085029+0000","last_undegraded":"2024-01-11T09:43:42.085029+0000","last_fullsized":"2024-01-11T09:43:42.085029+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:36:40.206301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.086901+0000","last_change":"2024-01-11T09:43:42.076258+0000","last_active":"2024-01-11T09:43:42.086901+0000","last_peered":"2024-01-11T09:43:42.086901+0000","last_clean":"2024-01-11T09:43:42.086901+0000","last_became_active":"2024-01-11T09:43:42.076006+0000","last_became_peered":"2024-01-11T09:43:42.076006+0000","last_unstale":"2024-01-11T09:43:42.086901+0000","last_undegraded":"2024-01-11T09:43:42.086901+0000","last_fullsized":"2024-01-11T09:43:42.086901+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:35:58.603836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"197'2","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.084159+0000","last_change":"2024-01-11T09:43:41.073672+0000","last_active":"2024-01-11T09:43:42.084159+0000","last_peered":"2024-01-11T09:43:42.084159+0000","last_clean":"2024-01-11T09:43:42.084159+0000","last_became_active":"2024-01-11T09:43:41.073399+0000","last_became_peered":"2024-01-11T09:43:41.073399+0000","last_unstale":"2024-01-11T09:43:42.084159+0000","last_undegraded":"2024-01-11T09:43:42.084159+0000","last_fullsized":"2024-01-11T09:43:42.084159+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:27:59.240683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.088218+0000","last_change":"2024-01-11T09:43:42.080800+0000","last_active":"2024-01-11T09:43:42.088218+0000","last_peered":"2024-01-11T09:43:42.088218+0000","last_clean":"2024-01-11T09:43:42.088218+0000","last_became_active":"2024-01-11T09:43:42.080540+0000","last_became_peered":"2024-01-11T09:43:42.080540+0000","last_unstale":"2024-01-11T09:43:42.088218+0000","last_undegraded":"2024-01-11T09:43:42.088218+0000","last_fullsized":"2024-01-11T09:43:42.088218+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:22.095497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"197'2","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090090+0000","last_change":"2024-01-11T09:43:41.067291+0000","last_active":"2024-01-11T09:43:42.090090+0000","last_peered":"2024-01-11T09:43:42.090090+0000","last_clean":"2024-01-11T09:43:42.090090+0000","last_became_active":"2024-01-11T09:43:41.067001+0000","last_became_peered":"2024-01-11T09:43:41.067001+0000","last_unstale":"2024-01-11T09:43:42.090090+0000","last_undegraded":"2024-01-11T09:43:42.090090+0000","last_fullsized":"2024-01-11T09:43:42.090090+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:40:55.256586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.082961+0000","last_change":"2024-01-11T09:43:42.082961+0000","last_active":"2024-01-11T09:43:42.082961+0000","last_peered":"2024-01-11T09:43:42.082961+0000","last_clean":"2024-01-11T09:43:42.082961+0000","last_became_active":"2024-01-11T09:43:42.080622+0000","last_became_peered":"2024-01-11T09:43:42.080622+0000","last_unstale":"2024-01-11T09:43:42.082961+0000","last_undegraded":"2024-01-11T09:43:42.082961+0000","last_fullsized":"2024-01-11T09:43:42.082961+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:20:06.379496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"197'1","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.059906+0000","last_change":"2024-01-11T09:43:41.066081+0000","last_active":"2024-01-11T09:43:42.059906+0000","last_peered":"2024-01-11T09:43:42.059906+0000","last_clean":"2024-01-11T09:43:42.059906+0000","last_became_active":"2024-01-11T09:43:41.065820+0000","last_became_peered":"2024-01-11T09:43:41.065820+0000","last_unstale":"2024-01-11T09:43:42.059906+0000","last_undegraded":"2024-01-11T09:43:42.059906+0000","last_fullsized":"2024-01-11T09:43:42.059906+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:27:59.240683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090848+0000","last_change":"2024-01-11T09:43:42.082027+0000","last_active":"2024-01-11T09:43:42.090848+0000","last_peered":"2024-01-11T09:43:42.090848+0000","last_clean":"2024-01-11T09:43:42.090848+0000","last_became_active":"2024-01-11T09:43:42.079757+0000","last_became_peered":"2024-01-11T09:43:42.079757+0000","last_unstale":"2024-01-11T09:43:42.090848+0000","last_undegraded":"2024-01-11T09:43:42.090848+0000","last_fullsized":"2024-01-11T09:43:42.090848+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:48:40.643154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"197'1","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.094612+0000","last_change":"2024-01-11T09:43:41.075006+0000","last_active":"2024-01-11T09:43:42.094612+0000","last_peered":"2024-01-11T09:43:42.094612+0000","last_clean":"2024-01-11T09:43:42.094612+0000","last_became_active":"2024-01-11T09:43:41.074557+0000","last_became_peered":"2024-01-11T09:43:41.074557+0000","last_unstale":"2024-01-11T09:43:42.094612+0000","last_undegraded":"2024-01-11T09:43:42.094612+0000","last_fullsized":"2024-01-11T09:43:42.094612+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:35:57.597516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.086790+0000","last_change":"2024-01-11T09:43:42.077002+0000","last_active":"2024-01-11T09:43:42.086790+0000","last_peered":"2024-01-11T09:43:42.086790+0000","last_clean":"2024-01-11T09:43:42.086790+0000","last_became_active":"2024-01-11T09:43:42.076741+0000","last_became_peered":"2024-01-11T09:43:42.076741+0000","last_unstale":"2024-01-11T09:43:42.086790+0000","last_undegraded":"2024-01-11T09:43:42.086790+0000","last_fullsized":"2024-01-11T09:43:42.086790+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:20:09.297290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.068378+0000","last_change":"2024-01-11T09:43:41.066550+0000","last_active":"2024-01-11T09:43:41.068378+0000","last_peered":"2024-01-11T09:43:41.068378+0000","last_clean":"2024-01-11T09:43:41.068378+0000","last_became_active":"2024-01-11T09:43:41.066176+0000","last_became_peered":"2024-01-11T09:43:41.066176+0000","last_unstale":"2024-01-11T09:43:41.068378+0000","last_undegraded":"2024-01-11T09:43:41.068378+0000","last_fullsized":"2024-01-11T09:43:41.068378+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:35:35.073088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090380+0000","last_change":"2024-01-11T09:43:42.090380+0000","last_active":"2024-01-11T09:43:42.090380+0000","last_peered":"2024-01-11T09:43:42.090380+0000","last_clean":"2024-01-11T09:43:42.090380+0000","last_became_active":"2024-01-11T09:43:42.089950+0000","last_became_peered":"2024-01-11T09:43:42.089950+0000","last_unstale":"2024-01-11T09:43:42.090380+0000","last_undegraded":"2024-01-11T09:43:42.090380+0000","last_fullsized":"2024-01-11T09:43:42.090380+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:11:18.246714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.072412+0000","last_change":"2024-01-11T09:43:41.067450+0000","last_active":"2024-01-11T09:43:42.072412+0000","last_peered":"2024-01-11T09:43:42.072412+0000","last_clean":"2024-01-11T09:43:42.072412+0000","last_became_active":"2024-01-11T09:43:41.066475+0000","last_became_peered":"2024-01-11T09:43:41.066475+0000","last_unstale":"2024-01-11T09:43:42.072412+0000","last_undegraded":"2024-01-11T09:43:42.072412+0000","last_fullsized":"2024-01-11T09:43:42.072412+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:10:04.789345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.091223+0000","last_change":"2024-01-11T09:43:42.082956+0000","last_active":"2024-01-11T09:43:42.091223+0000","last_peered":"2024-01-11T09:43:42.091223+0000","last_clean":"2024-01-11T09:43:42.091223+0000","last_became_active":"2024-01-11T09:43:42.080949+0000","last_became_peered":"2024-01-11T09:43:42.080949+0000","last_unstale":"2024-01-11T09:43:42.091223+0000","last_undegraded":"2024-01-11T09:43:42.091223+0000","last_fullsized":"2024-01-11T09:43:42.091223+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:00:10.469996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.071270+0000","last_change":"2024-01-11T09:43:41.068988+0000","last_active":"2024-01-11T09:43:41.071270+0000","last_peered":"2024-01-11T09:43:41.071270+0000","last_clean":"2024-01-11T09:43:41.071270+0000","last_became_active":"2024-01-11T09:43:41.068718+0000","last_became_peered":"2024-01-11T09:43:41.068718+0000","last_unstale":"2024-01-11T09:43:41.071270+0000","last_undegraded":"2024-01-11T09:43:41.071270+0000","last_fullsized":"2024-01-11T09:43:41.071270+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:44:02.560921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.093087+0000","last_change":"2024-01-11T09:43:42.088859+0000","last_active":"2024-01-11T09:43:42.093087+0000","last_peered":"2024-01-11T09:43:42.093087+0000","last_clean":"2024-01-11T09:43:42.093087+0000","last_became_active":"2024-01-11T09:43:42.088201+0000","last_became_peered":"2024-01-11T09:43:42.088201+0000","last_unstale":"2024-01-11T09:43:42.093087+0000","last_undegraded":"2024-01-11T09:43:42.093087+0000","last_fullsized":"2024-01-11T09:43:42.093087+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:40:56.262906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"197'3","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.092070+0000","last_change":"2024-01-11T09:43:41.067045+0000","last_active":"2024-01-11T09:43:42.092070+0000","last_peered":"2024-01-11T09:43:42.092070+0000","last_clean":"2024-01-11T09:43:42.092070+0000","last_became_active":"2024-01-11T09:43:41.066802+0000","last_became_peered":"2024-01-11T09:43:41.066802+0000","last_unstale":"2024-01-11T09:43:42.092070+0000","last_undegraded":"2024-01-11T09:43:42.092070+0000","last_fullsized":"2024-01-11T09:43:42.092070+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:33:23.788733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090178+0000","last_change":"2024-01-11T09:43:42.090178+0000","last_active":"2024-01-11T09:43:42.090178+0000","last_peered":"2024-01-11T09:43:42.090178+0000","last_clean":"2024-01-11T09:43:42.090178+0000","last_became_active":"2024-01-11T09:43:42.089912+0000","last_became_peered":"2024-01-11T09:43:42.089912+0000","last_unstale":"2024-01-11T09:43:42.090178+0000","last_undegraded":"2024-01-11T09:43:42.090178+0000","last_fullsized":"2024-01-11T09:43:42.090178+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:36:19.587822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"197'6","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.092149+0000","last_change":"2024-01-11T09:43:41.072734+0000","last_active":"2024-01-11T09:43:42.092149+0000","last_peered":"2024-01-11T09:43:42.092149+0000","last_clean":"2024-01-11T09:43:42.092149+0000","last_became_active":"2024-01-11T09:43:41.072429+0000","last_became_peered":"2024-01-11T09:43:41.072429+0000","last_unstale":"2024-01-11T09:43:42.092149+0000","last_undegraded":"2024-01-11T09:43:42.092149+0000","last_fullsized":"2024-01-11T09:43:42.092149+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:44:02.560921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.092372+0000","last_change":"2024-01-11T09:43:42.092372+0000","last_active":"2024-01-11T09:43:42.092372+0000","last_peered":"2024-01-11T09:43:42.092372+0000","last_clean":"2024-01-11T09:43:42.092372+0000","last_became_active":"2024-01-11T09:43:42.091799+0000","last_became_peered":"2024-01-11T09:43:42.091799+0000","last_unstale":"2024-01-11T09:43:42.092372+0000","last_undegraded":"2024-01-11T09:43:42.092372+0000","last_fullsized":"2024-01-11T09:43:42.092372+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:10:05.795665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"197'2","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.092771+0000","last_change":"2024-01-11T09:43:41.072649+0000","last_active":"2024-01-11T09:43:42.092771+0000","last_peered":"2024-01-11T09:43:42.092771+0000","last_clean":"2024-01-11T09:43:42.092771+0000","last_became_active":"2024-01-11T09:43:41.072376+0000","last_became_peered":"2024-01-11T09:43:41.072376+0000","last_unstale":"2024-01-11T09:43:42.092771+0000","last_undegraded":"2024-01-11T09:43:42.092771+0000","last_fullsized":"2024-01-11T09:43:42.092771+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:55:36.074446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.089291+0000","last_change":"2024-01-11T09:43:42.083805+0000","last_active":"2024-01-11T09:43:42.089291+0000","last_peered":"2024-01-11T09:43:42.089291+0000","last_clean":"2024-01-11T09:43:42.089291+0000","last_became_active":"2024-01-11T09:43:42.083087+0000","last_became_peered":"2024-01-11T09:43:42.083087+0000","last_unstale":"2024-01-11T09:43:42.089291+0000","last_undegraded":"2024-01-11T09:43:42.089291+0000","last_fullsized":"2024-01-11T09:43:42.089291+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:25:15.467096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.068179+0000","last_change":"2024-01-11T09:43:41.064275+0000","last_active":"2024-01-11T09:43:41.068179+0000","last_peered":"2024-01-11T09:43:41.068179+0000","last_clean":"2024-01-11T09:43:41.068179+0000","last_became_active":"2024-01-11T09:43:41.063957+0000","last_became_peered":"2024-01-11T09:43:41.063957+0000","last_unstale":"2024-01-11T09:43:41.068179+0000","last_undegraded":"2024-01-11T09:43:41.068179+0000","last_fullsized":"2024-01-11T09:43:41.068179+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:35:04.101580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.091041+0000","last_change":"2024-01-11T09:43:42.086313+0000","last_active":"2024-01-11T09:43:42.091041+0000","last_peered":"2024-01-11T09:43:42.091041+0000","last_clean":"2024-01-11T09:43:42.091041+0000","last_became_active":"2024-01-11T09:43:42.085963+0000","last_became_peered":"2024-01-11T09:43:42.085963+0000","last_unstale":"2024-01-11T09:43:42.091041+0000","last_undegraded":"2024-01-11T09:43:42.091041+0000","last_fullsized":"2024-01-11T09:43:42.091041+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:15:41.955996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.060565+0000","last_change":"2024-01-11T09:43:41.062575+0000","last_active":"2024-01-11T09:43:42.060565+0000","last_peered":"2024-01-11T09:43:42.060565+0000","last_clean":"2024-01-11T09:43:42.060565+0000","last_became_active":"2024-01-11T09:43:41.062305+0000","last_became_peered":"2024-01-11T09:43:41.062305+0000","last_unstale":"2024-01-11T09:43:42.060565+0000","last_undegraded":"2024-01-11T09:43:42.060565+0000","last_fullsized":"2024-01-11T09:43:42.060565+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:40:55.256586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.084380+0000","last_change":"2024-01-11T09:43:42.084380+0000","last_active":"2024-01-11T09:43:42.084380+0000","last_peered":"2024-01-11T09:43:42.084380+0000","last_clean":"2024-01-11T09:43:42.084380+0000","last_became_active":"2024-01-11T09:43:42.084013+0000","last_became_peered":"2024-01-11T09:43:42.084013+0000","last_unstale":"2024-01-11T09:43:42.084380+0000","last_undegraded":"2024-01-11T09:43:42.084380+0000","last_fullsized":"2024-01-11T09:43:42.084380+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:42:48.428037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"197'8","reported_seq":17,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.095742+0000","last_change":"2024-01-11T09:43:41.066556+0000","last_active":"2024-01-11T09:43:42.095742+0000","last_peered":"2024-01-11T09:43:42.095742+0000","last_clean":"2024-01-11T09:43:42.095742+0000","last_became_active":"2024-01-11T09:43:41.066251+0000","last_became_peered":"2024-01-11T09:43:41.066251+0000","last_unstale":"2024-01-11T09:43:42.095742+0000","last_undegraded":"2024-01-11T09:43:42.095742+0000","last_fullsized":"2024-01-11T09:43:42.095742+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:08.927728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.089897+0000","last_change":"2024-01-11T09:43:42.089897+0000","last_active":"2024-01-11T09:43:42.089897+0000","last_peered":"2024-01-11T09:43:42.089897+0000","last_clean":"2024-01-11T09:43:42.089897+0000","last_became_active":"2024-01-11T09:43:42.089624+0000","last_became_peered":"2024-01-11T09:43:42.089624+0000","last_unstale":"2024-01-11T09:43:42.089897+0000","last_undegraded":"2024-01-11T09:43:42.089897+0000","last_fullsized":"2024-01-11T09:43:42.089897+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:21:23.493032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"197'1","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.059583+0000","last_change":"2024-01-11T09:43:41.070478+0000","last_active":"2024-01-11T09:43:42.059583+0000","last_peered":"2024-01-11T09:43:42.059583+0000","last_clean":"2024-01-11T09:43:42.059583+0000","last_became_active":"2024-01-11T09:43:41.070009+0000","last_became_peered":"2024-01-11T09:43:41.070009+0000","last_unstale":"2024-01-11T09:43:42.059583+0000","last_undegraded":"2024-01-11T09:43:42.059583+0000","last_fullsized":"2024-01-11T09:43:42.059583+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:32:09.319822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.097039+0000","last_change":"2024-01-11T09:43:42.096523+0000","last_active":"2024-01-11T09:43:42.097039+0000","last_peered":"2024-01-11T09:43:42.097039+0000","last_clean":"2024-01-11T09:43:42.097039+0000","last_became_active":"2024-01-11T09:43:42.096126+0000","last_became_peered":"2024-01-11T09:43:42.096126+0000","last_unstale":"2024-01-11T09:43:42.097039+0000","last_undegraded":"2024-01-11T09:43:42.097039+0000","last_fullsized":"2024-01-11T09:43:42.097039+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:48:31.109585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"197'2","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.093584+0000","last_change":"2024-01-11T09:43:41.069733+0000","last_active":"2024-01-11T09:43:42.093584+0000","last_peered":"2024-01-11T09:43:42.093584+0000","last_clean":"2024-01-11T09:43:42.093584+0000","last_became_active":"2024-01-11T09:43:41.069510+0000","last_became_peered":"2024-01-11T09:43:41.069510+0000","last_unstale":"2024-01-11T09:43:42.093584+0000","last_undegraded":"2024-01-11T09:43:42.093584+0000","last_fullsized":"2024-01-11T09:43:42.093584+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:20:08.290971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.079719+0000","last_change":"2024-01-11T09:43:42.070438+0000","last_active":"2024-01-11T09:43:42.079719+0000","last_peered":"2024-01-11T09:43:42.079719+0000","last_clean":"2024-01-11T09:43:42.079719+0000","last_became_active":"2024-01-11T09:43:42.070170+0000","last_became_peered":"2024-01-11T09:43:42.070170+0000","last_unstale":"2024-01-11T09:43:42.079719+0000","last_undegraded":"2024-01-11T09:43:42.079719+0000","last_fullsized":"2024-01-11T09:43:42.079719+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:01:02.801676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"197'1","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.057977+0000","last_change":"2024-01-11T09:43:41.072650+0000","last_active":"2024-01-11T09:43:42.057977+0000","last_peered":"2024-01-11T09:43:42.057977+0000","last_clean":"2024-01-11T09:43:42.057977+0000","last_became_active":"2024-01-11T09:43:41.072346+0000","last_became_peered":"2024-01-11T09:43:41.072346+0000","last_unstale":"2024-01-11T09:43:42.057977+0000","last_undegraded":"2024-01-11T09:43:42.057977+0000","last_fullsized":"2024-01-11T09:43:42.057977+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:20:40.115692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.073876+0000","last_change":"2024-01-11T09:43:42.073876+0000","last_active":"2024-01-11T09:43:42.073876+0000","last_peered":"2024-01-11T09:43:42.073876+0000","last_clean":"2024-01-11T09:43:42.073876+0000","last_became_active":"2024-01-11T09:43:42.073613+0000","last_became_peered":"2024-01-11T09:43:42.073613+0000","last_unstale":"2024-01-11T09:43:42.073876+0000","last_undegraded":"2024-01-11T09:43:42.073876+0000","last_fullsized":"2024-01-11T09:43:42.073876+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:35:05.107900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.072083+0000","last_change":"2024-01-11T09:43:41.061032+0000","last_active":"2024-01-11T09:43:42.072083+0000","last_peered":"2024-01-11T09:43:42.072083+0000","last_clean":"2024-01-11T09:43:42.072083+0000","last_became_active":"2024-01-11T09:43:41.060770+0000","last_became_peered":"2024-01-11T09:43:41.060770+0000","last_unstale":"2024-01-11T09:43:42.072083+0000","last_undegraded":"2024-01-11T09:43:42.072083+0000","last_fullsized":"2024-01-11T09:43:42.072083+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:00:47.161318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.080884+0000","last_change":"2024-01-11T09:43:42.080884+0000","last_active":"2024-01-11T09:43:42.080884+0000","last_peered":"2024-01-11T09:43:42.080884+0000","last_clean":"2024-01-11T09:43:42.080884+0000","last_became_active":"2024-01-11T09:43:42.074190+0000","last_became_peered":"2024-01-11T09:43:42.074190+0000","last_unstale":"2024-01-11T09:43:42.080884+0000","last_undegraded":"2024-01-11T09:43:42.080884+0000","last_fullsized":"2024-01-11T09:43:42.080884+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:36:05.762240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"197'2","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.092917+0000","last_change":"2024-01-11T09:43:41.070690+0000","last_active":"2024-01-11T09:43:42.092917+0000","last_peered":"2024-01-11T09:43:42.092917+0000","last_clean":"2024-01-11T09:43:42.092917+0000","last_became_active":"2024-01-11T09:43:41.069836+0000","last_became_peered":"2024-01-11T09:43:41.069836+0000","last_unstale":"2024-01-11T09:43:42.092917+0000","last_undegraded":"2024-01-11T09:43:42.092917+0000","last_fullsized":"2024-01-11T09:43:42.092917+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:36:40.206301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.090815+0000","last_change":"2024-01-11T09:43:42.087040+0000","last_active":"2024-01-11T09:43:42.090815+0000","last_peered":"2024-01-11T09:43:42.090815+0000","last_clean":"2024-01-11T09:43:42.090815+0000","last_became_active":"2024-01-11T09:43:42.086230+0000","last_became_peered":"2024-01-11T09:43:42.086230+0000","last_unstale":"2024-01-11T09:43:42.090815+0000","last_undegraded":"2024-01-11T09:43:42.090815+0000","last_fullsized":"2024-01-11T09:43:42.090815+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:10:24.874233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.065354+0000","last_change":"2024-01-11T09:43:41.060470+0000","last_active":"2024-01-11T09:43:41.065354+0000","last_peered":"2024-01-11T09:43:41.065354+0000","last_clean":"2024-01-11T09:43:41.065354+0000","last_became_active":"2024-01-11T09:43:41.060208+0000","last_became_peered":"2024-01-11T09:43:41.060208+0000","last_unstale":"2024-01-11T09:43:41.065354+0000","last_undegraded":"2024-01-11T09:43:41.065354+0000","last_fullsized":"2024-01-11T09:43:41.065354+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:54.422211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.078191+0000","last_change":"2024-01-11T09:43:42.078191+0000","last_active":"2024-01-11T09:43:42.078191+0000","last_peered":"2024-01-11T09:43:42.078191+0000","last_clean":"2024-01-11T09:43:42.078191+0000","last_became_active":"2024-01-11T09:43:42.076031+0000","last_became_peered":"2024-01-11T09:43:42.076031+0000","last_unstale":"2024-01-11T09:43:42.078191+0000","last_undegraded":"2024-01-11T09:43:42.078191+0000","last_fullsized":"2024-01-11T09:43:42.078191+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:23:16.219171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.073169+0000","last_change":"2024-01-11T09:43:41.068567+0000","last_active":"2024-01-11T09:43:41.073169+0000","last_peered":"2024-01-11T09:43:41.073169+0000","last_clean":"2024-01-11T09:43:41.073169+0000","last_became_active":"2024-01-11T09:43:41.068269+0000","last_became_peered":"2024-01-11T09:43:41.068269+0000","last_unstale":"2024-01-11T09:43:41.073169+0000","last_undegraded":"2024-01-11T09:43:41.073169+0000","last_fullsized":"2024-01-11T09:43:41.073169+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:04:27.002128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.075960+0000","last_change":"2024-01-11T09:43:42.075960+0000","last_active":"2024-01-11T09:43:42.075960+0000","last_peered":"2024-01-11T09:43:42.075960+0000","last_clean":"2024-01-11T09:43:42.075960+0000","last_became_active":"2024-01-11T09:43:42.075677+0000","last_became_peered":"2024-01-11T09:43:42.075677+0000","last_unstale":"2024-01-11T09:43:42.075960+0000","last_undegraded":"2024-01-11T09:43:42.075960+0000","last_fullsized":"2024-01-11T09:43:42.075960+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:00:48.167638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.068859+0000","last_change":"2024-01-11T09:43:41.063300+0000","last_active":"2024-01-11T09:43:42.068859+0000","last_peered":"2024-01-11T09:43:42.068859+0000","last_clean":"2024-01-11T09:43:42.068859+0000","last_became_active":"2024-01-11T09:43:41.061537+0000","last_became_peered":"2024-01-11T09:43:41.061537+0000","last_unstale":"2024-01-11T09:43:42.068859+0000","last_undegraded":"2024-01-11T09:43:42.068859+0000","last_fullsized":"2024-01-11T09:43:42.068859+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:54.422211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.080672+0000","last_change":"2024-01-11T09:43:42.080672+0000","last_active":"2024-01-11T09:43:42.080672+0000","last_peered":"2024-01-11T09:43:42.080672+0000","last_clean":"2024-01-11T09:43:42.080672+0000","last_became_active":"2024-01-11T09:43:42.080429+0000","last_became_peered":"2024-01-11T09:43:42.080429+0000","last_unstale":"2024-01-11T09:43:42.080672+0000","last_undegraded":"2024-01-11T09:43:42.080672+0000","last_fullsized":"2024-01-11T09:43:42.080672+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:03:31.123957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"197'2","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.093135+0000","last_change":"2024-01-11T09:43:41.070600+0000","last_active":"2024-01-11T09:43:42.093135+0000","last_peered":"2024-01-11T09:43:42.093135+0000","last_clean":"2024-01-11T09:43:42.093135+0000","last_became_active":"2024-01-11T09:43:41.070321+0000","last_became_peered":"2024-01-11T09:43:41.070321+0000","last_unstale":"2024-01-11T09:43:42.093135+0000","last_undegraded":"2024-01-11T09:43:42.093135+0000","last_fullsized":"2024-01-11T09:43:42.093135+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:20:08.290971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.079792+0000","last_change":"2024-01-11T09:43:42.079792+0000","last_active":"2024-01-11T09:43:42.079792+0000","last_peered":"2024-01-11T09:43:42.079792+0000","last_clean":"2024-01-11T09:43:42.079792+0000","last_became_active":"2024-01-11T09:43:42.079414+0000","last_became_peered":"2024-01-11T09:43:42.079414+0000","last_unstale":"2024-01-11T09:43:42.079792+0000","last_undegraded":"2024-01-11T09:43:42.079792+0000","last_fullsized":"2024-01-11T09:43:42.079792+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:52:38.016298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.075852+0000","last_change":"2024-01-11T09:43:41.073648+0000","last_active":"2024-01-11T09:43:41.075852+0000","last_peered":"2024-01-11T09:43:41.075852+0000","last_clean":"2024-01-11T09:43:41.075852+0000","last_became_active":"2024-01-11T09:43:41.073404+0000","last_became_peered":"2024-01-11T09:43:41.073404+0000","last_unstale":"2024-01-11T09:43:41.075852+0000","last_undegraded":"2024-01-11T09:43:41.075852+0000","last_fullsized":"2024-01-11T09:43:41.075852+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:21:02.432952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.089769+0000","last_change":"2024-01-11T09:43:42.078061+0000","last_active":"2024-01-11T09:43:42.089769+0000","last_peered":"2024-01-11T09:43:42.089769+0000","last_clean":"2024-01-11T09:43:42.089769+0000","last_became_active":"2024-01-11T09:43:42.077674+0000","last_became_peered":"2024-01-11T09:43:42.077674+0000","last_unstale":"2024-01-11T09:43:42.089769+0000","last_undegraded":"2024-01-11T09:43:42.089769+0000","last_fullsized":"2024-01-11T09:43:42.089769+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:03:04.194736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.075919+0000","last_change":"2024-01-11T09:43:41.074413+0000","last_active":"2024-01-11T09:43:41.075919+0000","last_peered":"2024-01-11T09:43:41.075919+0000","last_clean":"2024-01-11T09:43:41.075919+0000","last_became_active":"2024-01-11T09:43:41.074157+0000","last_became_peered":"2024-01-11T09:43:41.074157+0000","last_unstale":"2024-01-11T09:43:41.075919+0000","last_undegraded":"2024-01-11T09:43:41.075919+0000","last_fullsized":"2024-01-11T09:43:41.075919+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:00:09.463677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.093508+0000","last_change":"2024-01-11T09:43:42.087645+0000","last_active":"2024-01-11T09:43:42.093508+0000","last_peered":"2024-01-11T09:43:42.093508+0000","last_clean":"2024-01-11T09:43:42.093508+0000","last_became_active":"2024-01-11T09:43:42.087367+0000","last_became_peered":"2024-01-11T09:43:42.087367+0000","last_unstale":"2024-01-11T09:43:42.093508+0000","last_undegraded":"2024-01-11T09:43:42.093508+0000","last_fullsized":"2024-01-11T09:43:42.093508+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:01:29.860733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.060382+0000","last_change":"2024-01-11T09:43:41.069275+0000","last_active":"2024-01-11T09:43:42.060382+0000","last_peered":"2024-01-11T09:43:42.060382+0000","last_clean":"2024-01-11T09:43:42.060382+0000","last_became_active":"2024-01-11T09:43:41.069032+0000","last_became_peered":"2024-01-11T09:43:41.069032+0000","last_unstale":"2024-01-11T09:43:42.060382+0000","last_undegraded":"2024-01-11T09:43:42.060382+0000","last_fullsized":"2024-01-11T09:43:42.060382+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:10:23.867913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.063236+0000","last_change":"2024-01-11T09:43:42.063236+0000","last_active":"2024-01-11T09:43:42.063236+0000","last_peered":"2024-01-11T09:43:42.063236+0000","last_clean":"2024-01-11T09:43:42.063236+0000","last_became_active":"2024-01-11T09:43:42.062915+0000","last_became_peered":"2024-01-11T09:43:42.062915+0000","last_unstale":"2024-01-11T09:43:42.063236+0000","last_undegraded":"2024-01-11T09:43:42.063236+0000","last_fullsized":"2024-01-11T09:43:42.063236+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:25:15.467096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.068760+0000","last_change":"2024-01-11T09:43:41.054651+0000","last_active":"2024-01-11T09:43:42.068760+0000","last_peered":"2024-01-11T09:43:42.068760+0000","last_clean":"2024-01-11T09:43:42.068760+0000","last_became_active":"2024-01-11T09:43:41.054360+0000","last_became_peered":"2024-01-11T09:43:41.054360+0000","last_unstale":"2024-01-11T09:43:42.068760+0000","last_undegraded":"2024-01-11T09:43:42.068760+0000","last_fullsized":"2024-01-11T09:43:42.068760+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:04:27.002128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.079536+0000","last_change":"2024-01-11T09:43:42.079536+0000","last_active":"2024-01-11T09:43:42.079536+0000","last_peered":"2024-01-11T09:43:42.079536+0000","last_clean":"2024-01-11T09:43:42.079536+0000","last_became_active":"2024-01-11T09:43:42.078591+0000","last_became_peered":"2024-01-11T09:43:42.078591+0000","last_unstale":"2024-01-11T09:43:42.079536+0000","last_undegraded":"2024-01-11T09:43:42.079536+0000","last_fullsized":"2024-01-11T09:43:42.079536+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:28:08.543281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":10,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.068774+0000","last_change":"2024-01-11T09:43:41.065046+0000","last_active":"2024-01-11T09:43:42.068774+0000","last_peered":"2024-01-11T09:43:42.068774+0000","last_clean":"2024-01-11T09:43:42.068774+0000","last_became_active":"2024-01-11T09:43:41.064262+0000","last_became_peered":"2024-01-11T09:43:41.064262+0000","last_unstale":"2024-01-11T09:43:42.068774+0000","last_undegraded":"2024-01-11T09:43:42.068774+0000","last_fullsized":"2024-01-11T09:43:42.068774+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:44:02.560921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.083650+0000","last_change":"2024-01-11T09:43:42.083650+0000","last_active":"2024-01-11T09:43:42.083650+0000","last_peered":"2024-01-11T09:43:42.083650+0000","last_clean":"2024-01-11T09:43:42.083650+0000","last_became_active":"2024-01-11T09:43:42.080323+0000","last_became_peered":"2024-01-11T09:43:42.080323+0000","last_unstale":"2024-01-11T09:43:42.083650+0000","last_undegraded":"2024-01-11T09:43:42.083650+0000","last_fullsized":"2024-01-11T09:43:42.083650+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:39:57.126666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.068774+0000","last_change":"2024-01-11T09:43:41.063971+0000","last_active":"2024-01-11T09:43:41.068774+0000","last_peered":"2024-01-11T09:43:41.068774+0000","last_clean":"2024-01-11T09:43:41.068774+0000","last_became_active":"2024-01-11T09:43:41.063586+0000","last_became_peered":"2024-01-11T09:43:41.063586+0000","last_unstale":"2024-01-11T09:43:41.068774+0000","last_undegraded":"2024-01-11T09:43:41.068774+0000","last_fullsized":"2024-01-11T09:43:41.068774+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:35:57.597516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.072612+0000","last_change":"2024-01-11T09:43:42.072612+0000","last_active":"2024-01-11T09:43:42.072612+0000","last_peered":"2024-01-11T09:43:42.072612+0000","last_clean":"2024-01-11T09:43:42.072612+0000","last_became_active":"2024-01-11T09:43:42.072045+0000","last_became_peered":"2024-01-11T09:43:42.072045+0000","last_unstale":"2024-01-11T09:43:42.072612+0000","last_undegraded":"2024-01-11T09:43:42.072612+0000","last_fullsized":"2024-01-11T09:43:42.072612+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:10:05.795665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.076213+0000","last_change":"2024-01-11T09:43:41.072802+0000","last_active":"2024-01-11T09:43:41.076213+0000","last_peered":"2024-01-11T09:43:41.076213+0000","last_clean":"2024-01-11T09:43:41.076213+0000","last_became_active":"2024-01-11T09:43:41.072288+0000","last_became_peered":"2024-01-11T09:43:41.072288+0000","last_unstale":"2024-01-11T09:43:41.076213+0000","last_undegraded":"2024-01-11T09:43:41.076213+0000","last_fullsized":"2024-01-11T09:43:41.076213+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:54.422211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":8,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-01-11T09:43:42.076624+0000","last_change":"2024-01-11T09:43:42.076624+0000","last_active":"2024-01-11T09:43:42.076624+0000","last_peered":"2024-01-11T09:43:42.076624+0000","last_clean":"2024-01-11T09:43:42.076624+0000","last_became_active":"2024-01-11T09:43:42.074176+0000","last_became_peered":"2024-01-11T09:43:42.074176+0000","last_unstale":"2024-01-11T09:43:42.076624+0000","last_undegraded":"2024-01-11T09:43:42.076624+0000","last_fullsized":"2024-01-11T09:43:42.076624+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:41.042812+0000","last_clean_scrub_stamp":"2024-01-11T09:43:41.042812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:17:54.012106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":9,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.075970+0000","last_change":"2024-01-11T09:43:41.071431+0000","last_active":"2024-01-11T09:43:41.075970+0000","last_peered":"2024-01-11T09:43:41.075970+0000","last_clean":"2024-01-11T09:43:41.075970+0000","last_became_active":"2024-01-11T09:43:41.071073+0000","last_became_peered":"2024-01-11T09:43:41.071073+0000","last_unstale":"2024-01-11T09:43:41.075970+0000","last_undegraded":"2024-01-11T09:43:41.075970+0000","last_fullsized":"2024-01-11T09:43:41.075970+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:43:40.036492+0000","last_clean_scrub_stamp":"2024-01-11T09:43:40.036492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:03:30.117638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"188'344","reported_seq":1107,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-01-11T09:43:41.060116+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:43:41.060116+0000","last_peered":"2024-01-11T09:43:41.060116+0000","last_clean":"2024-01-11T09:43:41.060116+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:43:41.060116+0000","last_undegraded":"2024-01-11T09:43:41.060116+0000","last_fullsized":"2024-01-11T09:43:41.060116+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2096,"num_read_kb":4396,"num_write":571,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2096,"num_read_kb":4396,"num_write":571,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":344,"ondisk_log_size":344,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300288,"kb_used_data":3000,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071552,"statfs":{"total":96636764160,"available":96329269248,"internally_reserved":0,"allocated":3072000,"data_stored":1765854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.72499999999999998,"15min":0.73999999999999999},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.36599999999999999},"max":{"1min":0.90400000000000003,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.76200000000000001,"15min":0.753},"min":{"1min":0.58799999999999997,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.89900000000000002,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.71799999999999997}]},{"osd":1,"last update":"Thu Jan 11 09:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.73199999999999998,"15min":0.73699999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41599999999999998},"max":{"1min":0.88,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.746},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.746,"15min":0.749},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.85899999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.57899999999999996}]},{"osd":2,"last update":"Thu Jan 11 09:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.747,"15min":0.746},"min":{"1min":0.60699999999999998,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.87,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.76700000000000002,"15min":0.76000000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.44500000000000001},"max":{"1min":0.91100000000000003,"5min":1.153,"15min":1.153},"last":0.65100000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.75900000000000001,"15min":0.77500000000000002},"min":{"1min":0.56499999999999995,"5min":0.53600000000000003,"15min":0.42699999999999999},"max":{"1min":0.90800000000000003,"5min":1.03,"15min":1.0660000000000001},"last":0.622},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.753,"15min":0.76200000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.92300000000000004,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.59599999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.41599999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.76200000000000001,"5min":1.107,"15min":1.107},"last":0.753},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.39700000000000002},"max":{"1min":0.73699999999999999,"5min":0.96099999999999997,"15min":1.0960000000000001},"last":0.73499999999999999}]},{"osd":5,"last update":"Thu Jan 11 09:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.63600000000000001,"15min":0.63700000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.42399999999999999},"max":{"1min":0.78600000000000003,"5min":0.90300000000000002,"15min":1.0109999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.67900000000000005,"15min":0.68100000000000005},"min":{"1min":0.50700000000000001,"5min":0.47099999999999997,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":1.133,"15min":1.133},"last":0.66000000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.77600000000000002,"15min":0.77700000000000002},"min":{"1min":0.57999999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.89500000000000002,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.76600000000000001,"15min":0.75600000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.45800000000000002},"max":{"1min":0.874,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.78800000000000003}]}]},{"osd":6,"up_from":13,"seq":55834575952,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311988,"kb_used_data":2412,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059852,"statfs":{"total":96636764160,"available":96317288448,"internally_reserved":0,"allocated":2469888,"data_stored":1175382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.71199999999999997,"15min":0.71999999999999997},"min":{"1min":0.54800000000000004,"5min":0.503,"15min":0.34599999999999997},"max":{"1min":0.94199999999999995,"5min":0.96199999999999997,"15min":1.508},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.755,"15min":0.754},"min":{"1min":0.59099999999999997,"5min":0.44400000000000001,"15min":0.39200000000000002},"max":{"1min":0.97099999999999997,"5min":1.216,"15min":1.5589999999999999},"last":0.81200000000000006}]},{"osd":1,"last update":"Thu Jan 11 09:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.748,"15min":0.753},"min":{"1min":0.51000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.92500000000000004,"5min":1.2549999999999999,"15min":1.45},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.72599999999999998,"15min":0.74099999999999999},"min":{"1min":0.56599999999999995,"5min":0.48299999999999998,"15min":0.46899999999999997},"max":{"1min":0.95299999999999996,"5min":1.0249999999999999,"15min":1.542},"last":0.878}]},{"osd":2,"last update":"Thu Jan 11 09:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.75800000000000001,"15min":0.76100000000000001},"min":{"1min":0.58599999999999997,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.98099999999999998,"5min":1.4179999999999999,"15min":1.5329999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.75700000000000001,"15min":0.76500000000000001},"min":{"1min":0.64900000000000002,"5min":0.505,"15min":0.505},"max":{"1min":0.93999999999999995,"5min":1.0449999999999999,"15min":1.647},"last":0.86099999999999999}]},{"osd":3,"last update":"Thu Jan 11 09:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.76300000000000001,"15min":0.76800000000000002},"min":{"1min":0.622,"5min":0.46899999999999997,"15min":0.45200000000000001},"max":{"1min":1.012,"5min":1.014,"15min":1.6140000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.77400000000000002,"15min":0.78600000000000003},"min":{"1min":0.60099999999999998,"5min":0.54200000000000004,"15min":0.48499999999999999},"max":{"1min":0.93200000000000005,"5min":1.165,"15min":1.4950000000000001},"last":0.80000000000000004}]},{"osd":4,"last update":"Thu Jan 11 09:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.41199999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.91700000000000004,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.61699999999999999,"15min":0.61399999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84899999999999998,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.54400000000000004}]},{"osd":5,"last update":"Thu Jan 11 09:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.67600000000000005,"15min":0.66100000000000003},"min":{"1min":0.441,"5min":0.41499999999999998,"15min":0.40000000000000002},"max":{"1min":0.97899999999999998,"5min":1.369,"15min":1.369},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71499999999999997,"15min":0.71299999999999997},"min":{"1min":0.51700000000000002,"5min":0.46700000000000003,"15min":0.45400000000000001},"max":{"1min":0.999,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.82699999999999996}]},{"osd":7,"last update":"Thu Jan 11 09:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.72299999999999998,"15min":0.71299999999999997},"min":{"1min":0.45600000000000002,"5min":0.45100000000000001,"15min":0.44400000000000001},"max":{"1min":1.089,"5min":1.484,"15min":1.484},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78200000000000003,"15min":0.77400000000000002},"min":{"1min":0.55300000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.1379999999999999,"5min":1.603,"15min":1.603},"last":0.89500000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575954,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307904,"kb_used_data":2424,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063936,"statfs":{"total":96636764160,"available":96321470464,"internally_reserved":0,"allocated":2482176,"data_stored":1176918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51800000000000002,"15min":0.51300000000000001},"min":{"1min":0.33800000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.73399999999999999,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.621,"15min":0.623},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.95099999999999996},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 11 09:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.372,"5min":0.35899999999999999,"15min":0.35099999999999998},"max":{"1min":0.82399999999999995,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.59399999999999997},"min":{"1min":0.40799999999999997,"5min":0.40600000000000003,"15min":0.39400000000000002},"max":{"1min":0.81799999999999995,"5min":0.83899999999999997,"15min":0.876},"last":0.435}]},{"osd":3,"last update":"Thu Jan 11 09:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.64900000000000002,"15min":0.65100000000000002},"min":{"1min":0.497,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.82699999999999996,"5min":0.91600000000000004,"15min":0.96699999999999997},"last":0.625},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.58599999999999997,"15min":0.58499999999999996},"min":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.53300000000000003}]},{"osd":4,"last update":"Thu Jan 11 09:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81999999999999995,"15min":0.82099999999999995},"min":{"1min":0.54000000000000004,"5min":0.53500000000000003,"15min":0.505},"max":{"1min":1.214,"5min":1.214,"15min":1.2969999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.86099999999999999,"15min":0.86199999999999999},"min":{"1min":0.68600000000000005,"5min":0.66400000000000003,"15min":0.61699999999999999},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1890000000000001},"last":0.75}]},{"osd":5,"last update":"Thu Jan 11 09:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.83699999999999997,"15min":0.82999999999999996},"min":{"1min":0.64400000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.82699999999999996,"15min":0.80800000000000005},"min":{"1min":0.63800000000000001,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.54200000000000004}]},{"osd":6,"last update":"Thu Jan 11 09:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.84899999999999998,"15min":0.84599999999999997},"min":{"1min":0.60699999999999998,"5min":0.59599999999999997,"15min":0.56799999999999995},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.85799999999999998,"15min":0.85699999999999998},"min":{"1min":0.748,"5min":0.626,"15min":0.55900000000000005},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.64300000000000002}]},{"osd":7,"last update":"Thu Jan 11 09:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.89200000000000002,"15min":0.89000000000000001},"min":{"1min":0.77400000000000002,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.86699999999999999,"15min":0.85899999999999999},"min":{"1min":0.70199999999999996,"5min":0.59299999999999997,"15min":0.54500000000000004},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.60099999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575953,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308476,"kb_used_data":2996,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063364,"statfs":{"total":96636764160,"available":96320884736,"internally_reserved":0,"allocated":3067904,"data_stored":1765772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51300000000000001,"15min":0.53000000000000003},"min":{"1min":0.35599999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.63900000000000001,"5min":0.84699999999999998,"15min":0.875},"last":0.629},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59399999999999997,"15min":0.60399999999999998},"min":{"1min":0.47399999999999998,"5min":0.41899999999999998,"15min":0.40100000000000002},"max":{"1min":0.80600000000000005,"5min":0.86399999999999999,"15min":0.874},"last":0.61099999999999999}]},{"osd":2,"last update":"Thu Jan 11 09:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56499999999999995,"15min":0.57599999999999996},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.36599999999999999},"max":{"1min":0.83099999999999996,"5min":0.84499999999999997,"15min":0.97499999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.42099999999999999,"5min":0.38500000000000001,"15min":0.38200000000000001},"max":{"1min":0.77000000000000002,"5min":0.80900000000000005,"15min":0.86399999999999999},"last":0.46400000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52900000000000003,"15min":0.55200000000000005},"min":{"1min":0.38400000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.79000000000000004,"5min":0.92600000000000005,"15min":0.95399999999999996},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64600000000000002,"15min":0.66000000000000003},"min":{"1min":0.47899999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.90300000000000002,"5min":0.92600000000000005,"15min":1.0389999999999999},"last":0.71499999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.82899999999999996,"15min":0.82299999999999995},"min":{"1min":0.65900000000000003,"5min":0.60299999999999998,"15min":0.48499999999999999},"max":{"1min":1.1339999999999999,"5min":1.1779999999999999,"15min":1.304},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.81299999999999994,"15min":0.80800000000000005},"min":{"1min":0.63300000000000001,"5min":0.58099999999999996,"15min":0.51400000000000001},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.58799999999999997}]},{"osd":5,"last update":"Thu Jan 11 09:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84599999999999997},"min":{"1min":0.65600000000000003,"5min":0.64600000000000002,"15min":0.53900000000000003},"max":{"1min":1.1140000000000001,"5min":1.194,"15min":1.194},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.84199999999999997,"15min":0.84599999999999997},"min":{"1min":0.65000000000000002,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.69499999999999995}]},{"osd":6,"last update":"Thu Jan 11 09:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.83599999999999997,"15min":0.83099999999999996},"min":{"1min":0.59599999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.86399999999999999,"15min":0.874},"min":{"1min":0.67000000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.1100000000000001,"5min":1.256,"15min":1.256},"last":0.78000000000000003}]},{"osd":7,"last update":"Thu Jan 11 09:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.86399999999999999,"15min":0.86899999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.58699999999999997},"max":{"1min":1.206,"5min":1.28,"15min":1.28},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.871,"15min":0.88100000000000001},"min":{"1min":0.63900000000000001,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.238,"5min":1.238,"15min":1.2410000000000001},"last":0.73499999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575952,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312000,"kb_used_data":2424,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059840,"statfs":{"total":96636764160,"available":96317276160,"internally_reserved":0,"allocated":2482176,"data_stored":1176048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.51300000000000001,"15min":0.50800000000000001},"min":{"1min":0.34200000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65200000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.371},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56100000000000005,"15min":0.56899999999999995},"min":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.34300000000000003},"max":{"1min":0.68200000000000005,"5min":1.131,"15min":1.6359999999999999},"last":0.57899999999999996}]},{"osd":1,"last update":"Thu Jan 11 09:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.442,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.93000000000000005},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.61299999999999999,"15min":0.60699999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.40000000000000002},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.145},"last":0.441}]},{"osd":3,"last update":"Thu Jan 11 09:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41599999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72699999999999998,"5min":0.81299999999999994,"15min":0.96399999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.66100000000000003,"15min":0.66500000000000004},"min":{"1min":0.52600000000000002,"5min":0.434,"15min":0.434},"max":{"1min":0.86699999999999999,"5min":0.97999999999999998,"15min":1.3620000000000001},"last":0.55700000000000005}]},{"osd":4,"last update":"Thu Jan 11 09:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81000000000000005,"15min":0.80700000000000005},"min":{"1min":0.65500000000000003,"5min":0.57799999999999996,"15min":0.54300000000000004},"max":{"1min":1.071,"5min":1.071,"15min":1.1519999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81399999999999995,"15min":0.81000000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.69199999999999995}]},{"osd":5,"last update":"Thu Jan 11 09:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.88600000000000001,"15min":0.88200000000000001},"min":{"1min":0.68500000000000005,"5min":0.67700000000000005,"15min":0.55800000000000005},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.429},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.87,"15min":0.87},"min":{"1min":0.73699999999999999,"5min":0.65200000000000002,"15min":0.61099999999999999},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.2170000000000001},"last":0.75600000000000001}]},{"osd":6,"last update":"Thu Jan 11 09:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.83799999999999997,"15min":0.83499999999999996},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.4139999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.876,"5min":0.876,"15min":0.877},"min":{"1min":0.66000000000000003,"5min":0.63200000000000001,"15min":0.60899999999999999},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.593},"last":0.82799999999999996}]},{"osd":7,"last update":"Thu Jan 11 09:42:53 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89900000000000002,"15min":0.90200000000000002},"min":{"1min":0.70899999999999996,"5min":0.60699999999999998,"15min":0.57299999999999995},"max":{"1min":1.1339999999999999,"5min":1.137,"15min":2.0539999999999998},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.88300000000000001,"15min":0.89200000000000002},"min":{"1min":0.63500000000000001,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2490000000000001},"last":0.86399999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575952,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299712,"kb_used_data":2424,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072128,"statfs":{"total":96636764160,"available":96329859072,"internally_reserved":0,"allocated":2482176,"data_stored":1175576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51300000000000001,"15min":0.52600000000000002},"min":{"1min":0.33900000000000002,"5min":0.32200000000000001,"15min":0.314},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.90000000000000002},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.433,"5min":0.40500000000000003,"15min":0.36099999999999999},"max":{"1min":0.90400000000000003,"5min":1.014,"15min":1.0449999999999999},"last":0.44600000000000001}]},{"osd":1,"last update":"Thu Jan 11 09:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51100000000000001,"15min":0.51500000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.67500000000000004,"5min":0.84199999999999997,"15min":1.304},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59499999999999997,"15min":0.61699999999999999},"min":{"1min":0.41599999999999998,"5min":0.38700000000000001,"15min":0.371},"max":{"1min":0.73599999999999999,"5min":0.84099999999999997,"15min":1.4590000000000001},"last":0.48699999999999999}]},{"osd":2,"last update":"Thu Jan 11 09:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.60899999999999999,"15min":0.61499999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.751,"5min":0.93700000000000006,"15min":1.04},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.66000000000000003,"15min":0.66300000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.44700000000000001},"max":{"1min":0.83799999999999997,"5min":0.89800000000000002,"15min":1.0069999999999999},"last":0.64600000000000002}]},{"osd":4,"last update":"Thu Jan 11 09:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.79300000000000004,"15min":0.79500000000000004},"min":{"1min":0.63100000000000001,"5min":0.53000000000000003,"15min":0.50600000000000001},"max":{"1min":0.90300000000000002,"5min":1.206,"15min":1.206},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80500000000000005,"15min":0.82299999999999995},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.171,"5min":1.171,"15min":1.1830000000000001},"last":1.0569999999999999}]},{"osd":5,"last update":"Thu Jan 11 09:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.83499999999999996,"15min":0.82899999999999996},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.123,"5min":1.1950000000000001,"15min":1.234},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.82899999999999996,"15min":0.85499999999999998},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.0940000000000001,"5min":1.117,"15min":1.3440000000000001},"last":1.1160000000000001}]},{"osd":6,"last update":"Thu Jan 11 09:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.84799999999999998,"15min":0.86499999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.58599999999999997},"max":{"1min":1.05,"5min":1.175,"15min":1.335},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.82299999999999995,"15min":0.82999999999999996},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.94599999999999995,"5min":1.181,"15min":1.2190000000000001},"last":0.95799999999999996}]},{"osd":7,"last update":"Thu Jan 11 09:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.87,"15min":0.875},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.052,"5min":1.264,"15min":1.264},"last":1.0429999999999999},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.89900000000000002,"15min":0.90400000000000003},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.0720000000000001,"5min":1.24,"15min":1.24},"last":0.998}]}]},{"osd":4,"up_from":13,"seq":55834575953,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307908,"kb_used_data":2428,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063932,"statfs":{"total":96636764160,"available":96321466368,"internally_reserved":0,"allocated":2486272,"data_stored":1177472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.77100000000000002,"15min":0.77900000000000003},"min":{"1min":0.44800000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.91200000000000003,"5min":1.038,"15min":1.119},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78800000000000003,"15min":0.78300000000000003},"min":{"1min":0.53300000000000003,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":0.70699999999999996}]},{"osd":1,"last update":"Thu Jan 11 09:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.75900000000000001,"15min":0.75600000000000001},"min":{"1min":0.46899999999999997,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.0190000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.74299999999999999,"15min":0.73999999999999999},"min":{"1min":0.50900000000000001,"5min":0.48799999999999999,"15min":0.45000000000000001},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":0.65700000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.77600000000000002,"15min":0.75700000000000001},"min":{"1min":0.52500000000000002,"5min":0.47099999999999997,"15min":0.44700000000000001},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.76400000000000001,"15min":0.75600000000000001},"min":{"1min":0.54700000000000004,"5min":0.53400000000000003,"15min":0.48999999999999999},"max":{"1min":1.0109999999999999,"5min":1.0249999999999999,"15min":1.0700000000000001},"last":0.72399999999999998}]},{"osd":3,"last update":"Thu Jan 11 09:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.75900000000000001,"15min":0.754},"min":{"1min":0.54000000000000004,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.80200000000000005,"15min":0.78700000000000003},"min":{"1min":0.57899999999999996,"5min":0.55600000000000005,"15min":0.53300000000000003},"max":{"1min":2.1150000000000002,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.60299999999999998}]},{"osd":5,"last update":"Thu Jan 11 09:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.63,"15min":0.62},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78300000000000003,"5min":0.97199999999999998,"15min":1.0049999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.63400000000000001}]},{"osd":6,"last update":"Thu Jan 11 09:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.66900000000000004,"15min":0.66600000000000004},"min":{"1min":0.46300000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.752,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.58599999999999997}]},{"osd":7,"last update":"Thu Jan 11 09:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.69799999999999995,"15min":0.69899999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.75,"15min":0.751},"min":{"1min":0.436,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":2.052,"5min":2.052,"15min":2.052},"last":0.74099999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575953,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311988,"kb_used_data":2412,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059852,"statfs":{"total":96636764160,"available":96317288448,"internally_reserved":0,"allocated":2469888,"data_stored":1175382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.749,"15min":0.73599999999999999},"min":{"1min":0.50900000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.88800000000000001,"5min":0.97799999999999998,"15min":1.0129999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.72599999999999998,"15min":0.73399999999999999},"min":{"1min":0.55400000000000005,"5min":0.46700000000000003,"15min":0.45600000000000002},"max":{"1min":0.94099999999999995,"5min":1.048,"15min":1.0660000000000001},"last":0.64000000000000001}]},{"osd":1,"last update":"Thu Jan 11 09:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.73199999999999998,"15min":0.74099999999999999},"min":{"1min":0.54200000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.91800000000000004,"5min":0.94699999999999995,"15min":0.97699999999999998},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.754,"15min":0.749},"min":{"1min":0.56999999999999995,"5min":0.53600000000000003,"15min":0.46700000000000003},"max":{"1min":0.96499999999999997,"5min":0.98499999999999999,"15min":1.0700000000000001},"last":0.55800000000000005}]},{"osd":2,"last update":"Thu Jan 11 09:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.505,"5min":0.48699999999999999,"15min":0.47899999999999998},"max":{"1min":0.93400000000000005,"5min":0.94299999999999995,"15min":1.028},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.748,"15min":0.749},"min":{"1min":0.59499999999999997,"5min":0.53200000000000003,"15min":0.505},"max":{"1min":0.91300000000000003,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.68100000000000005}]},{"osd":3,"last update":"Thu Jan 11 09:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.755,"15min":0.76800000000000002},"min":{"1min":0.61299999999999999,"5min":0.54200000000000004,"15min":0.49299999999999999},"max":{"1min":1.002,"5min":1.0049999999999999,"15min":1.022},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.78700000000000003,"15min":0.77500000000000002},"min":{"1min":0.64400000000000002,"5min":0.59799999999999998,"15min":0.52100000000000002},"max":{"1min":0.96599999999999997,"5min":1.004,"15min":1.1000000000000001},"last":0.80600000000000005}]},{"osd":4,"last update":"Thu Jan 11 09:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.61299999999999999,"15min":0.60999999999999999},"min":{"1min":0.41999999999999998,"5min":0.41699999999999998,"15min":0.41599999999999998},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.97199999999999998},"last":0.62},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.45100000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.84799999999999998,"5min":0.94499999999999995,"15min":1.042},"last":0.60999999999999999}]},{"osd":6,"last update":"Thu Jan 11 09:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.629,"15min":0.64500000000000002},"min":{"1min":0.5,"5min":0.44400000000000001,"15min":0.40999999999999998},"max":{"1min":0.83699999999999997,"5min":0.93100000000000005,"15min":1.0089999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.67700000000000005,"15min":0.68600000000000005},"min":{"1min":0.47799999999999998,"5min":0.46200000000000002,"15min":0.42799999999999999},"max":{"1min":0.91300000000000003,"5min":0.92100000000000004,"15min":1.0229999999999999},"last":0.79100000000000004}]},{"osd":7,"last update":"Thu Jan 11 09:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.72099999999999997,"15min":0.73699999999999999},"min":{"1min":0.51700000000000002,"5min":0.46800000000000003,"15min":0.434},"max":{"1min":0.93000000000000005,"5min":1.0309999999999999,"15min":1.0760000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.76600000000000001,"15min":0.78200000000000003},"min":{"1min":0.60999999999999999,"5min":0.5,"15min":0.47599999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0469999999999999},"last":0.59199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:43:46.974 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-11T09:43:46.975 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-11T09:43:47.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.222+0000 7f9dcc1ff700 1 Processor -- start 2024-01-11T09:43:47.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.222+0000 7f9dcc1ff700 1 -- start start 2024-01-11T09:43:47.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.223+0000 7f9dcc1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9dc40c8860 0x7f9dc4158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:47.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.223+0000 7f9dcc1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9dc4158600 0x7f9dc40ae230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:47.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.223+0000 7f9dcc1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc40cabe0 con 0x7f9dc40b12c0 2024-01-11T09:43:47.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.223+0000 7f9dcc1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc40ae770 con 0x7f9dc40c8860 2024-01-11T09:43:47.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.223+0000 7f9dcc1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc40ae960 con 0x7f9dc4158600 2024-01-11T09:43:47.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.224+0000 7f9dc9f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9dc40c8860 0x7f9dc4158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:47.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.224+0000 7f9dc9f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9dc40c8860 0x7f9dc4158ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57428/0 (socket says 172.21.15.27:57428) 2024-01-11T09:43:47.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.224+0000 7f9dc9f9b700 1 -- 172.21.15.27:0/638688374 learned_addr learned my addr 172.21.15.27:0/638688374 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:47.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.224+0000 7f9dca79c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9dc40b12c0 0x7f9dc40cb070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:32916/0 (socket says 172.21.15.27:32916) 2024-01-11T09:43:47.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.224+0000 7f9dc979a700 1 --2- 172.21.15.27:0/638688374 >> v2:172.21.15.77:3300/0 conn(0x7f9dc4158600 0x7f9dc40ae230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:47.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.224+0000 7f9dc9f9b700 1 -- 172.21.15.27:0/638688374 >> v2:172.21.15.77:3300/0 conn(0x7f9dc4158600 msgr2=0x7f9dc40ae230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:47.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.224+0000 7f9dc9f9b700 1 --2- 172.21.15.27:0/638688374 >> v2:172.21.15.77:3300/0 conn(0x7f9dc4158600 0x7f9dc40ae230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:47.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.224+0000 7f9dc9f9b700 1 -- 172.21.15.27:0/638688374 >> v1:172.21.15.27:6789/0 conn(0x7f9dc40b12c0 legacy=0x7f9dc40cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:47.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.224+0000 7f9dc9f9b700 1 -- 172.21.15.27:0/638688374 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc40ca590 con 0x7f9dc40c8860 2024-01-11T09:43:47.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.224+0000 7f9dc979a700 1 --2- 172.21.15.27:0/638688374 >> v2:172.21.15.77:3300/0 conn(0x7f9dc4158600 0x7f9dc40ae230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:47.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.224+0000 7f9dc8f99700 1 -- 172.21.15.27:0/638688374 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427221489 0 0) 0x7f9dc40cabe0 con 0x7f9dc40b12c0 2024-01-11T09:43:47.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.224+0000 7f9dc9f9b700 1 --2- 172.21.15.27:0/638688374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dc40c8860 0x7f9dc4158ca0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f9dc406d0a0 tx=0x7f9dc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8bab73cf1bf761 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:47.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.225+0000 7f9dc8f99700 1 -- 172.21.15.27:0/638688374 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc000b860 con 0x7f9dc40c8860 2024-01-11T09:43:47.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.225+0000 7f9dc8f99700 1 -- 172.21.15.27:0/638688374 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc000b9c0 con 0x7f9dc40c8860 2024-01-11T09:43:47.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.225+0000 7f9dc8f99700 1 -- 172.21.15.27:0/638688374 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc000bd60 con 0x7f9dc40c8860 2024-01-11T09:43:47.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.225+0000 7f9dcc1ff700 1 -- 172.21.15.27:0/638688374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dc40c8860 msgr2=0x7f9dc4158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:47.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.225+0000 7f9dcc1ff700 1 --2- 172.21.15.27:0/638688374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dc40c8860 0x7f9dc4158ca0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f9dc406d0a0 tx=0x7f9dc0009250 comp rx=0 tx=0).stop 2024-01-11T09:43:47.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.225+0000 7f9dcc1ff700 1 -- 172.21.15.27:0/638688374 shutdown_connections 2024-01-11T09:43:47.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.225+0000 7f9dcc1ff700 1 --2- 172.21.15.27:0/638688374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dc40c8860 0x7f9dc4158ca0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:47.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.225+0000 7f9dcc1ff700 1 --2- 172.21.15.27:0/638688374 >> v2:172.21.15.77:3300/0 conn(0x7f9dc4158600 0x7f9dc40ae230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:47.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.225+0000 7f9dcc1ff700 1 -- 172.21.15.27:0/638688374 >> 172.21.15.27:0/638688374 conn(0x7f9dc40680a0 msgr2=0x7f9dc4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:47.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.225+0000 7f9dcc1ff700 1 -- 172.21.15.27:0/638688374 shutdown_connections 2024-01-11T09:43:47.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.225+0000 7f9dcc1ff700 1 -- 172.21.15.27:0/638688374 wait complete. 2024-01-11T09:43:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.227+0000 7f9dcc1ff700 1 Processor -- start 2024-01-11T09:43:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.227+0000 7f9dcc1ff700 1 -- start start 2024-01-11T09:43:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.228+0000 7f9dcc1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dc40b12c0 0x7f9dc416d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.228+0000 7f9dcc1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dc40c8860 0x7f9dc4154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.228+0000 7f9dcc1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9dc4158600 0x7f9dc40ac520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.228+0000 7f9dcc1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc40add80 con 0x7f9dc40c8860 2024-01-11T09:43:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.228+0000 7f9dcc1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc40ac090 con 0x7f9dc40b12c0 2024-01-11T09:43:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.228+0000 7f9dcc1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc40ab7a0 con 0x7f9dc4158600 2024-01-11T09:43:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.228+0000 7f9dc979a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dc40c8860 0x7f9dc4154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.228+0000 7f9dc979a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dc40c8860 0x7f9dc4154150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57440/0 (socket says 172.21.15.27:57440) 2024-01-11T09:43:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.228+0000 7f9dc979a700 1 -- 172.21.15.27:0/1668293187 learned_addr learned my addr 172.21.15.27:0/1668293187 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.229+0000 7f9dc9f9b700 1 --2- 172.21.15.27:0/1668293187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dc40b12c0 0x7f9dc416d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.229+0000 7f9dca79c700 1 --2- 172.21.15.27:0/1668293187 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9dc4158600 0x7f9dc40ac520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:47.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.229+0000 7f9dc979a700 1 -- 172.21.15.27:0/1668293187 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9dc4158600 msgr2=0x7f9dc40ac520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.229+0000 7f9dc979a700 1 --2- 172.21.15.27:0/1668293187 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9dc4158600 0x7f9dc40ac520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.229+0000 7f9dc979a700 1 -- 172.21.15.27:0/1668293187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dc40b12c0 msgr2=0x7f9dc416d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.229+0000 7f9dc979a700 1 --2- 172.21.15.27:0/1668293187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dc40b12c0 0x7f9dc416d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.229+0000 7f9dc979a700 1 -- 172.21.15.27:0/1668293187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc00094f0 con 0x7f9dc40c8860 2024-01-11T09:43:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.229+0000 7f9dc9f9b700 1 --2- 172.21.15.27:0/1668293187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dc40b12c0 0x7f9dc416d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:43:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.229+0000 7f9dca79c700 1 --2- 172.21.15.27:0/1668293187 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9dc4158600 0x7f9dc40ac520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:43:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.229+0000 7f9dc979a700 1 --2- 172.21.15.27:0/1668293187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dc40c8860 0x7f9dc4154150 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f9db400d860 tx=0x7f9db4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.229+0000 7f9db9ffb700 1 -- 172.21.15.27:0/1668293187 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db40125f0 con 0x7f9dc40c8860 2024-01-11T09:43:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.229+0000 7f9db9ffb700 1 -- 172.21.15.27:0/1668293187 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db4012750 con 0x7f9dc40c8860 2024-01-11T09:43:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.229+0000 7f9db9ffb700 1 -- 172.21.15.27:0/1668293187 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db4012ad0 con 0x7f9dc40c8860 2024-01-11T09:43:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.229+0000 7f9dcc1ff700 1 -- 172.21.15.27:0/1668293187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc416db90 con 0x7f9dc40c8860 2024-01-11T09:43:47.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.229+0000 7f9dcc1ff700 1 -- 172.21.15.27:0/1668293187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc40abd20 con 0x7f9dc40c8860 2024-01-11T09:43:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.230+0000 7f9dcc1ff700 1 -- 172.21.15.27:0/1668293187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc40ac200 con 0x7f9dc40c8860 2024-01-11T09:43:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.231+0000 7f9db9ffb700 1 -- 172.21.15.27:0/1668293187 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9db4014430 con 0x7f9dc40c8860 2024-01-11T09:43:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.232+0000 7f9db9ffb700 1 --2- 172.21.15.27:0/1668293187 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9da8066f30 0x7f9da80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.232+0000 7f9db9ffb700 1 -- 172.21.15.27:0/1668293187 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f9db40849d0 con 0x7f9dc40c8860 2024-01-11T09:43:47.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.233+0000 7f9dc9f9b700 1 --2- 172.21.15.27:0/1668293187 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9da8066f30 0x7f9da80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:47.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.233+0000 7f9dc9f9b700 1 --2- 172.21.15.27:0/1668293187 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9da8066f30 0x7f9da80693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9dc406d0a0 tx=0x7f9dc0017000 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:47.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.236+0000 7f9db9ffb700 1 -- 172.21.15.27:0/1668293187 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9db4055680 con 0x7f9dc40c8860 2024-01-11T09:43:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.524+0000 7f9dcc1ff700 1 -- 172.21.15.27:0/1668293187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f9dc404c3c0 con 0x7f9dc40c8860 2024-01-11T09:43:47.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.530+0000 7f9db9ffb700 1 -- 172.21.15.27:0/1668293187 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v49) v1 ==== 190+0+0 (secure 0 0 0) 0x7f9db4055210 con 0x7f9dc40c8860 2024-01-11T09:43:47.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.533+0000 7f9dcc1ff700 1 -- 172.21.15.27:0/1668293187 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9da8066f30 msgr2=0x7f9da80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:47.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.533+0000 7f9dcc1ff700 1 --2- 172.21.15.27:0/1668293187 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9da8066f30 0x7f9da80693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9dc406d0a0 tx=0x7f9dc0017000 comp rx=0 tx=0).stop 2024-01-11T09:43:47.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.533+0000 7f9dcc1ff700 1 -- 172.21.15.27:0/1668293187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dc40c8860 msgr2=0x7f9dc4154150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:47.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.533+0000 7f9dcc1ff700 1 --2- 172.21.15.27:0/1668293187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dc40c8860 0x7f9dc4154150 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f9db400d860 tx=0x7f9db4008040 comp rx=0 tx=0).stop 2024-01-11T09:43:47.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.533+0000 7f9dcc1ff700 1 -- 172.21.15.27:0/1668293187 shutdown_connections 2024-01-11T09:43:47.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.533+0000 7f9dcc1ff700 1 --2- 172.21.15.27:0/1668293187 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9dc4158600 0x7f9dc40ac520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:47.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.533+0000 7f9dcc1ff700 1 --2- 172.21.15.27:0/1668293187 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9da8066f30 0x7f9da80693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:47.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.533+0000 7f9dcc1ff700 1 --2- 172.21.15.27:0/1668293187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dc40b12c0 0x7f9dc416d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:47.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.533+0000 7f9dcc1ff700 1 --2- 172.21.15.27:0/1668293187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dc40c8860 0x7f9dc4154150 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:47.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.533+0000 7f9dcc1ff700 1 -- 172.21.15.27:0/1668293187 >> 172.21.15.27:0/1668293187 conn(0x7f9dc40680a0 msgr2=0x7f9dc4158ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:47.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.533+0000 7f9dcc1ff700 1 -- 172.21.15.27:0/1668293187 shutdown_connections 2024-01-11T09:43:47.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.533+0000 7f9dcc1ff700 1 -- 172.21.15.27:0/1668293187 wait complete. 2024-01-11T09:43:47.526 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-11T09:43:47.542 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:43:47.666 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:43:47.670 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:43:47.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.751+0000 7f0c215ff700 1 Processor -- start 2024-01-11T09:43:47.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.751+0000 7f0c215ff700 1 -- start start 2024-01-11T09:43:47.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.752+0000 7f0c215ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0c1c0b03e0 0x7f0c1c0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:47.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.752+0000 7f0c215ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0c1c0b29c0 0x7f0c1c159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:47.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.752+0000 7f0c215ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c1c0cd730 con 0x7f0c1c0b07b0 2024-01-11T09:43:47.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.752+0000 7f0c215ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c1c0b2de0 con 0x7f0c1c0b03e0 2024-01-11T09:43:47.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.752+0000 7f0c215ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c1c0ccb20 con 0x7f0c1c0b29c0 2024-01-11T09:43:47.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.753+0000 7f0c1affd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0c1c0b03e0 0x7f0c1c0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:47.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.753+0000 7f0c1affd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0c1c0b03e0 0x7f0c1c0b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57456/0 (socket says 172.21.15.27:57456) 2024-01-11T09:43:47.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.753+0000 7f0c1affd700 1 -- 172.21.15.27:0/1969493548 learned_addr learned my addr 172.21.15.27:0/1969493548 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:47.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.753+0000 7f0c1b7fe700 1 --2- 172.21.15.27:0/1969493548 >> v2:172.21.15.77:3300/0 conn(0x7f0c1c0b29c0 0x7f0c1c159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:47.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.753+0000 7f0c1affd700 1 -- 172.21.15.27:0/1969493548 >> v2:172.21.15.77:3300/0 conn(0x7f0c1c0b29c0 msgr2=0x7f0c1c159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.753+0000 7f0c1affd700 1 --2- 172.21.15.27:0/1969493548 >> v2:172.21.15.77:3300/0 conn(0x7f0c1c0b29c0 0x7f0c1c159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.753+0000 7f0c19ffb700 1 -- 172.21.15.27:0/1969493548 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302833916 0 0) 0x7f0c1c0cd730 con 0x7f0c1c0b07b0 2024-01-11T09:43:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.753+0000 7f0c1affd700 1 -- 172.21.15.27:0/1969493548 >> v1:172.21.15.77:6789/0 conn(0x7f0c1c0b07b0 legacy=0x7f0c1c159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.753+0000 7f0c1affd700 1 -- 172.21.15.27:0/1969493548 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c1c0cce00 con 0x7f0c1c0b03e0 2024-01-11T09:43:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.754+0000 7f0c1affd700 1 --2- 172.21.15.27:0/1969493548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c0b03e0 0x7f0c1c0b2480 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c0b1d10 tx=0x7f0c0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94186920ccdca9fe server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.754+0000 7f0c19ffb700 1 -- 172.21.15.27:0/1969493548 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c009e70 con 0x7f0c1c0b03e0 2024-01-11T09:43:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.754+0000 7f0c19ffb700 1 -- 172.21.15.27:0/1969493548 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0c00b860 con 0x7f0c1c0b03e0 2024-01-11T09:43:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.754+0000 7f0c19ffb700 1 -- 172.21.15.27:0/1969493548 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c00bc00 con 0x7f0c1c0b03e0 2024-01-11T09:43:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.754+0000 7f0c215ff700 1 -- 172.21.15.27:0/1969493548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c0b03e0 msgr2=0x7f0c1c0b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.754+0000 7f0c215ff700 1 --2- 172.21.15.27:0/1969493548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c0b03e0 0x7f0c1c0b2480 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c0b1d10 tx=0x7f0c0c009250 comp rx=0 tx=0).stop 2024-01-11T09:43:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.754+0000 7f0c215ff700 1 -- 172.21.15.27:0/1969493548 shutdown_connections 2024-01-11T09:43:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.754+0000 7f0c215ff700 1 --2- 172.21.15.27:0/1969493548 >> v2:172.21.15.77:3300/0 conn(0x7f0c1c0b29c0 0x7f0c1c159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.754+0000 7f0c215ff700 1 --2- 172.21.15.27:0/1969493548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c0b03e0 0x7f0c1c0b2480 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:47.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.754+0000 7f0c215ff700 1 -- 172.21.15.27:0/1969493548 >> 172.21.15.27:0/1969493548 conn(0x7f0c1c06ed40 msgr2=0x7f0c1c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:47.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.755+0000 7f0c215ff700 1 -- 172.21.15.27:0/1969493548 shutdown_connections 2024-01-11T09:43:47.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.755+0000 7f0c215ff700 1 -- 172.21.15.27:0/1969493548 wait complete. 2024-01-11T09:43:47.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.756+0000 7f0c215ff700 1 Processor -- start 2024-01-11T09:43:47.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.756+0000 7f0c215ff700 1 -- start start 2024-01-11T09:43:47.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.757+0000 7f0c215ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0c1c0b03e0 0x7f0c1c16d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.757+0000 7f0c215ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c1c0b07b0 0x7f0c1c16d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.757+0000 7f0c215ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c0b29c0 0x7f0c1c0cfa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.757+0000 7f0c215ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c0d0ad0 con 0x7f0c1c0b29c0 2024-01-11T09:43:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.757+0000 7f0c215ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c16b5a0 con 0x7f0c1c0b07b0 2024-01-11T09:43:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.757+0000 7f0c215ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c0d07f0 con 0x7f0c1c0b03e0 2024-01-11T09:43:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.757+0000 7f0c1b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c0b29c0 0x7f0c1c0cfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.757+0000 7f0c1affd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0c1c0b03e0 0x7f0c1c16d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.757+0000 7f0c1b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c0b29c0 0x7f0c1c0cfa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57462/0 (socket says 172.21.15.27:57462) 2024-01-11T09:43:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.757+0000 7f0c1b7fe700 1 -- 172.21.15.27:0/1595746597 learned_addr learned my addr 172.21.15.27:0/1595746597 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.757+0000 7f0c1a7fc700 1 --2- 172.21.15.27:0/1595746597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c1c0b07b0 0x7f0c1c16d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.758+0000 7f0c1affd700 1 -- 172.21.15.27:0/1595746597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c1c0b07b0 msgr2=0x7f0c1c16d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.758+0000 7f0c1affd700 1 --2- 172.21.15.27:0/1595746597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c1c0b07b0 0x7f0c1c16d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.758+0000 7f0c1affd700 1 -- 172.21.15.27:0/1595746597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c0b29c0 msgr2=0x7f0c1c0cfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.758+0000 7f0c1affd700 1 --2- 172.21.15.27:0/1595746597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c0b29c0 0x7f0c1c0cfa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.758+0000 7f0c1affd700 1 -- 172.21.15.27:0/1595746597 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c0094f0 con 0x7f0c1c0b03e0 2024-01-11T09:43:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.758+0000 7f0c1a7fc700 1 --2- 172.21.15.27:0/1595746597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c1c0b07b0 0x7f0c1c16d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:43:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.758+0000 7f0c1b7fe700 1 --2- 172.21.15.27:0/1595746597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c0b29c0 0x7f0c1c0cfa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:43:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.758+0000 7f0c1affd700 1 --2- 172.21.15.27:0/1595746597 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0c1c0b03e0 0x7f0c1c16d1e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c0cdfc0 tx=0x7f0c0c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.759+0000 7f0c02ffd700 1 -- 172.21.15.27:0/1595746597 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c000c90 con 0x7f0c1c0b03e0 2024-01-11T09:43:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.759+0000 7f0c02ffd700 1 -- 172.21.15.27:0/1595746597 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0c00b9d0 con 0x7f0c1c0b03e0 2024-01-11T09:43:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.759+0000 7f0c215ff700 1 -- 172.21.15.27:0/1595746597 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c1c16af50 con 0x7f0c1c0b03e0 2024-01-11T09:43:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.759+0000 7f0c02ffd700 1 -- 172.21.15.27:0/1595746597 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c025420 con 0x7f0c1c0b03e0 2024-01-11T09:43:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.759+0000 7f0c215ff700 1 -- 172.21.15.27:0/1595746597 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c1c0af090 con 0x7f0c1c0b03e0 2024-01-11T09:43:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.760+0000 7f0c215ff700 1 -- 172.21.15.27:0/1595746597 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c1c16adb0 con 0x7f0c1c0b03e0 2024-01-11T09:43:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.761+0000 7f0c02ffd700 1 -- 172.21.15.27:0/1595746597 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0c0c023070 con 0x7f0c1c0b03e0 2024-01-11T09:43:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.762+0000 7f0c02ffd700 1 --2- 172.21.15.27:0/1595746597 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c08066f30 0x7f0c080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.762+0000 7f0c02ffd700 1 -- 172.21.15.27:0/1595746597 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f0c0c0859b0 con 0x7f0c1c0b03e0 2024-01-11T09:43:47.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.762+0000 7f0c1a7fc700 1 --2- 172.21.15.27:0/1595746597 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c08066f30 0x7f0c080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:47.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.763+0000 7f0c1a7fc700 1 --2- 172.21.15.27:0/1595746597 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c08066f30 0x7f0c080693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c15cf40 tx=0x7f0c10009040 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:47.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:47.767+0000 7f0c02ffd700 1 -- 172.21.15.27:0/1595746597 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0c1c16adb0 con 0x7f0c1c0b03e0 2024-01-11T09:43:48.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.013+0000 7f0c215ff700 1 -- 172.21.15.27:0/1595746597 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c1c16adb0 con 0x7f0c1c0b03e0 2024-01-11T09:43:48.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.014+0000 7f0c02ffd700 1 -- 172.21.15.27:0/1595746597 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+4603 (secure 0 0 0) 0x7f0c1c16adb0 con 0x7f0c1c0b03e0 2024-01-11T09:43:48.006 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:43:48.006 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":102,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8441,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/991606883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":991606883},{"type":"v1","addr":"172.21.15.77:6834","nonce":991606883}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1996818074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1996818074},{"type":"v1","addr":"172.21.15.77:6835","nonce":1996818074}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9742,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4117588854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":4117588854},{"type":"v1","addr":"172.21.15.27:6835","nonce":4117588854}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:43:41.554157+0000","modified":"2024-01-11T09:43:45.078369+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10113":{"gid":10113,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/1557605486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1557605486},{"type":"v1","addr":"172.21.15.27:6834","nonce":1557605486}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2024-01-11T09:43:48.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.016+0000 7f0c215ff700 1 -- 172.21.15.27:0/1595746597 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c08066f30 msgr2=0x7f0c080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:48.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.016+0000 7f0c215ff700 1 --2- 172.21.15.27:0/1595746597 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c08066f30 0x7f0c080693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c15cf40 tx=0x7f0c10009040 comp rx=0 tx=0).stop 2024-01-11T09:43:48.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.016+0000 7f0c215ff700 1 -- 172.21.15.27:0/1595746597 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0c1c0b03e0 msgr2=0x7f0c1c16d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:48.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.016+0000 7f0c215ff700 1 --2- 172.21.15.27:0/1595746597 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0c1c0b03e0 0x7f0c1c16d1e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c0cdfc0 tx=0x7f0c0c017000 comp rx=0 tx=0).stop 2024-01-11T09:43:48.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.016+0000 7f0c215ff700 1 -- 172.21.15.27:0/1595746597 shutdown_connections 2024-01-11T09:43:48.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.016+0000 7f0c215ff700 1 --2- 172.21.15.27:0/1595746597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c1c0b29c0 0x7f0c1c0cfa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.016+0000 7f0c215ff700 1 --2- 172.21.15.27:0/1595746597 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0c1c0b03e0 0x7f0c1c16d1e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.016+0000 7f0c215ff700 1 --2- 172.21.15.27:0/1595746597 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c08066f30 0x7f0c080693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.016+0000 7f0c215ff700 1 --2- 172.21.15.27:0/1595746597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c1c0b07b0 0x7f0c1c16d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.016+0000 7f0c215ff700 1 -- 172.21.15.27:0/1595746597 >> 172.21.15.27:0/1595746597 conn(0x7f0c1c06ed40 msgr2=0x7f0c1c159530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:48.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.016+0000 7f0c215ff700 1 -- 172.21.15.27:0/1595746597 shutdown_connections 2024-01-11T09:43:48.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.016+0000 7f0c215ff700 1 -- 172.21.15.27:0/1595746597 wait complete. 2024-01-11T09:43:48.009 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 102 2024-01-11T09:43:48.429 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 102, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-11T09:43:41.554157+0000', 'modified': '2024-01-11T09:43:45.078369+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10113}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10113': {'gid': 10113, 'name': 'c', 'rank': 0, 'incarnation': 98, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/1557605486', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 1557605486}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1557605486}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-11T09:43:48.429 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-11T09:43:48.430 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-11T09:43:48.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.639+0000 7f247dbff700 1 Processor -- start 2024-01-11T09:43:48.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.639+0000 7f247dbff700 1 -- start start 2024-01-11T09:43:48.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.641+0000 7f247dbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2478156ab0 0x7f24780ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.641+0000 7f247dbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24780cc9b0 0x7f2478156e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.641+0000 7f247dbff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24780adc80 con 0x7f24781573c0 2024-01-11T09:43:48.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.641+0000 7f247dbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24780adf60 con 0x7f2478156ab0 2024-01-11T09:43:48.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.641+0000 7f247dbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24780ae240 con 0x7f24780cc9b0 2024-01-11T09:43:48.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.642+0000 7f2476ffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24780cc9b0 0x7f2478156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.642+0000 7f2476ffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24780cc9b0 0x7f2478156e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:36388/0 (socket says 172.21.15.27:36388) 2024-01-11T09:43:48.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.642+0000 7f2476ffd700 1 -- 172.21.15.27:0/1877781469 learned_addr learned my addr 172.21.15.27:0/1877781469 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:48.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.642+0000 7f24777fe700 1 --2- 172.21.15.27:0/1877781469 >> v2:172.21.15.77:3300/0 conn(0x7f2478156ab0 0x7f24780ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.642+0000 7f2476ffd700 1 -- 172.21.15.27:0/1877781469 >> v2:172.21.15.77:3300/0 conn(0x7f2478156ab0 msgr2=0x7f24780ad650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:48.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.642+0000 7f2476ffd700 1 --2- 172.21.15.27:0/1877781469 >> v2:172.21.15.77:3300/0 conn(0x7f2478156ab0 0x7f24780ad650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.642+0000 7f2476ffd700 1 -- 172.21.15.27:0/1877781469 >> v1:172.21.15.77:6789/0 conn(0x7f24781573c0 legacy=0x7f24780af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:48.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.642+0000 7f2476ffd700 1 -- 172.21.15.27:0/1877781469 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24780ae520 con 0x7f24780cc9b0 2024-01-11T09:43:48.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.642+0000 7f24777fe700 1 --2- 172.21.15.27:0/1877781469 >> v2:172.21.15.77:3300/0 conn(0x7f2478156ab0 0x7f24780ad650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:43:48.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.643+0000 7f2476ffd700 1 --2- 172.21.15.27:0/1877781469 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24780cc9b0 0x7f2478156e80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f246c0099b0 tx=0x7f246c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12636214abeb2bf0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:48.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.643+0000 7f24767fc700 1 -- 172.21.15.27:0/1877781469 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c009ea0 con 0x7f24780cc9b0 2024-01-11T09:43:48.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.643+0000 7f24767fc700 1 -- 172.21.15.27:0/1877781469 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246c00b860 con 0x7f24780cc9b0 2024-01-11T09:43:48.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.643+0000 7f24767fc700 1 -- 172.21.15.27:0/1877781469 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c00bc00 con 0x7f24780cc9b0 2024-01-11T09:43:48.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.643+0000 7f247dbff700 1 -- 172.21.15.27:0/1877781469 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24780cc9b0 msgr2=0x7f2478156e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:48.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.644+0000 7f247dbff700 1 --2- 172.21.15.27:0/1877781469 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24780cc9b0 0x7f2478156e80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f246c0099b0 tx=0x7f246c009250 comp rx=0 tx=0).stop 2024-01-11T09:43:48.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.644+0000 7f247dbff700 1 -- 172.21.15.27:0/1877781469 shutdown_connections 2024-01-11T09:43:48.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.644+0000 7f247dbff700 1 --2- 172.21.15.27:0/1877781469 >> v2:172.21.15.77:3300/0 conn(0x7f2478156ab0 0x7f24780ad650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.644+0000 7f247dbff700 1 --2- 172.21.15.27:0/1877781469 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24780cc9b0 0x7f2478156e80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.644+0000 7f247dbff700 1 -- 172.21.15.27:0/1877781469 >> 172.21.15.27:0/1877781469 conn(0x7f24780680b0 msgr2=0x7f24780cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:48.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.644+0000 7f247dbff700 1 -- 172.21.15.27:0/1877781469 shutdown_connections 2024-01-11T09:43:48.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.644+0000 7f247dbff700 1 -- 172.21.15.27:0/1877781469 wait complete. 2024-01-11T09:43:48.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.646+0000 7f247dbff700 1 Processor -- start 2024-01-11T09:43:48.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.646+0000 7f247dbff700 1 -- start start 2024-01-11T09:43:48.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.647+0000 7f247dbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24780cc9b0 0x7f247816c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.647+0000 7f247dbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2478156ab0 0x7f24780ca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.647+0000 7f247dbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24781573c0 0x7f2478155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.647+0000 7f247dbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247814b860 con 0x7f2478156ab0 2024-01-11T09:43:48.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.647+0000 7f247dbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2478169720 con 0x7f24781573c0 2024-01-11T09:43:48.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.647+0000 7f247dbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24780cb330 con 0x7f24780cc9b0 2024-01-11T09:43:48.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.648+0000 7f2476ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2478156ab0 0x7f24780ca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.648+0000 7f24777fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24780cc9b0 0x7f247816c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.648+0000 7f2476ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2478156ab0 0x7f24780ca9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57466/0 (socket says 172.21.15.27:57466) 2024-01-11T09:43:48.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.648+0000 7f2476ffd700 1 -- 172.21.15.27:0/1302170830 learned_addr learned my addr 172.21.15.27:0/1302170830 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:48.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.648+0000 7f2477fff700 1 --2- 172.21.15.27:0/1302170830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24781573c0 0x7f2478155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.648+0000 7f2476ffd700 1 -- 172.21.15.27:0/1302170830 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24780cc9b0 msgr2=0x7f247816c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:48.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.648+0000 7f2476ffd700 1 --2- 172.21.15.27:0/1302170830 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24780cc9b0 0x7f247816c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.648+0000 7f2476ffd700 1 -- 172.21.15.27:0/1302170830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24781573c0 msgr2=0x7f2478155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:48.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.648+0000 7f2476ffd700 1 --2- 172.21.15.27:0/1302170830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24781573c0 0x7f2478155500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.648+0000 7f2476ffd700 1 -- 172.21.15.27:0/1302170830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2468009420 con 0x7f2478156ab0 2024-01-11T09:43:48.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.648+0000 7f2477fff700 1 --2- 172.21.15.27:0/1302170830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24781573c0 0x7f2478155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:48.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.648+0000 7f2476ffd700 1 --2- 172.21.15.27:0/1302170830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2478156ab0 0x7f24780ca9c0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f24780ae380 tx=0x7f246c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:48.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.649+0000 7f245f7fe700 1 -- 172.21.15.27:0/1302170830 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c00bbe0 con 0x7f2478156ab0 2024-01-11T09:43:48.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.649+0000 7f245f7fe700 1 -- 172.21.15.27:0/1302170830 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246c011420 con 0x7f2478156ab0 2024-01-11T09:43:48.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.649+0000 7f245f7fe700 1 -- 172.21.15.27:0/1302170830 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c0117c0 con 0x7f2478156ab0 2024-01-11T09:43:48.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.649+0000 7f247dbff700 1 -- 172.21.15.27:0/1302170830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f246c0094f0 con 0x7f2478156ab0 2024-01-11T09:43:48.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.649+0000 7f247dbff700 1 -- 172.21.15.27:0/1302170830 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24780cb8f0 con 0x7f2478156ab0 2024-01-11T09:43:48.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.651+0000 7f245f7fe700 1 -- 172.21.15.27:0/1302170830 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f246c011bf0 con 0x7f2478156ab0 2024-01-11T09:43:48.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.652+0000 7f245f7fe700 1 --2- 172.21.15.27:0/1302170830 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2464066f30 0x7f24640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.652+0000 7f245f7fe700 1 -- 172.21.15.27:0/1302170830 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f246c085e70 con 0x7f2478156ab0 2024-01-11T09:43:48.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.652+0000 7f24777fe700 1 --2- 172.21.15.27:0/1302170830 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2464066f30 0x7f24640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.653+0000 7f24777fe700 1 --2- 172.21.15.27:0/1302170830 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2464066f30 0x7f24640693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2468009f40 tx=0x7f24680091a0 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:48.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.654+0000 7f248c24ab80 1 Processor -- start 2024-01-11T09:43:48.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.654+0000 7f248c24ab80 1 -- start start 2024-01-11T09:43:48.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.655+0000 7f248c24ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1af2b20 0x555fe1ad3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.655+0000 7f248c24ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1ace3a0 0x555fe1afa860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.655+0000 7f248c24ab80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1af2ef0 0x555fe1afada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.655+0000 7f248c24ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fe19eb310 con 0x555fe1ace3a0 2024-01-11T09:43:48.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.655+0000 7f248c24ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fe1a28130 con 0x555fe1af2b20 2024-01-11T09:43:48.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.655+0000 7f248c24ab80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x555fe1a25af0 con 0x555fe1af2ef0 2024-01-11T09:43:48.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.655+0000 7f2476ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1ace3a0 0x555fe1afa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.655+0000 7f2476ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1ace3a0 0x555fe1afa860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57468/0 (socket says 172.21.15.27:57468) 2024-01-11T09:43:48.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.655+0000 7f2476ffd700 1 -- 172.21.15.27:0/3213981014 learned_addr learned my addr 172.21.15.27:0/3213981014 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:48.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.655+0000 7f24777fe700 1 --2- 172.21.15.27:0/3213981014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1af2b20 0x555fe1ad3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.655+0000 7f2477fff700 1 --2- 172.21.15.27:0/3213981014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1af2ef0 0x555fe1afada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.655+0000 7f2476ffd700 1 -- 172.21.15.27:0/3213981014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1af2ef0 msgr2=0x555fe1afada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:48.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.655+0000 7f2476ffd700 1 --2- 172.21.15.27:0/3213981014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1af2ef0 0x555fe1afada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.655+0000 7f2476ffd700 1 -- 172.21.15.27:0/3213981014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1af2b20 msgr2=0x555fe1ad3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:48.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.655+0000 7f2476ffd700 1 --2- 172.21.15.27:0/3213981014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1af2b20 0x555fe1ad3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.655+0000 7f2476ffd700 1 -- 172.21.15.27:0/3213981014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555fe19140a0 con 0x555fe1ace3a0 2024-01-11T09:43:48.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.655+0000 7f2477fff700 1 --2- 172.21.15.27:0/3213981014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1af2ef0 0x555fe1afada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:48.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.655+0000 7f24777fe700 1 --2- 172.21.15.27:0/3213981014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1af2b20 0x555fe1ad3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:48.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.656+0000 7f2476ffd700 1 --2- 172.21.15.27:0/3213981014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1ace3a0 0x555fe1afa860 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f246c0531b0 tx=0x7f246c084200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43713bd9705ca380 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:48.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.656+0000 7f245cff9700 1 -- 172.21.15.27:0/3213981014 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c011ea0 con 0x555fe1ace3a0 2024-01-11T09:43:48.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.656+0000 7f245cff9700 1 -- 172.21.15.27:0/3213981014 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246c0573b0 con 0x555fe1ace3a0 2024-01-11T09:43:48.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.656+0000 7f245cff9700 1 -- 172.21.15.27:0/3213981014 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c056cb0 con 0x555fe1ace3a0 2024-01-11T09:43:48.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.656+0000 7f248c24ab80 1 -- 172.21.15.27:0/3213981014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1ace3a0 msgr2=0x555fe1afa860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:48.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.656+0000 7f248c24ab80 1 --2- 172.21.15.27:0/3213981014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1ace3a0 0x555fe1afa860 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f246c0531b0 tx=0x7f246c084200 comp rx=0 tx=0).stop 2024-01-11T09:43:48.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.656+0000 7f248c24ab80 1 -- 172.21.15.27:0/3213981014 shutdown_connections 2024-01-11T09:43:48.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.656+0000 7f248c24ab80 1 --2- 172.21.15.27:0/3213981014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1af2ef0 0x555fe1afada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.656+0000 7f248c24ab80 1 --2- 172.21.15.27:0/3213981014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1af2b20 0x555fe1ad3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.657+0000 7f248c24ab80 1 --2- 172.21.15.27:0/3213981014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1ace3a0 0x555fe1afa860 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.657+0000 7f248c24ab80 1 -- 172.21.15.27:0/3213981014 >> 172.21.15.27:0/3213981014 conn(0x555fe1ad0300 msgr2=0x555fe1ad2bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:48.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.657+0000 7f248c24ab80 1 -- 172.21.15.27:0/3213981014 shutdown_connections 2024-01-11T09:43:48.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.657+0000 7f248c24ab80 1 -- 172.21.15.27:0/3213981014 wait complete. 2024-01-11T09:43:48.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.657+0000 7f248c24ab80 1 -- start start 2024-01-11T09:43:48.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.658+0000 7f248c24ab80 1 Processor -- start 2024-01-11T09:43:48.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.658+0000 7f248c24ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1ace3a0 0x555fe1af0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.659+0000 7f248c24ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 0x555fe1af0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.659+0000 7f248c24ab80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1af2ef0 0x555fe1af10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.659+0000 7f248c24ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fe1a218e0 con 0x555fe1af2b20 2024-01-11T09:43:48.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.659+0000 7f248c24ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fe19cd980 con 0x555fe1ace3a0 2024-01-11T09:43:48.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.659+0000 7f248c24ab80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x555fe19cd620 con 0x555fe1af2ef0 2024-01-11T09:43:48.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.659+0000 7f248c24ab80 10 client.0 _ll_register_callbacks cb 0x555fe1830a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T09:43:48.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.659+0000 7f2476ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 0x555fe1af0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.659+0000 7f2476ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 0x555fe1af0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57480/0 (socket says 172.21.15.27:57480) 2024-01-11T09:43:48.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.659+0000 7f2476ffd700 1 -- 172.21.15.27:0/4149870014 learned_addr learned my addr 172.21.15.27:0/4149870014 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:48.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.659+0000 7f24777fe700 1 --2- 172.21.15.27:0/4149870014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1ace3a0 0x555fe1af0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.659+0000 7f2477fff700 1 --2- 172.21.15.27:0/4149870014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1af2ef0 0x555fe1af10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.659+0000 7f2476ffd700 1 -- 172.21.15.27:0/4149870014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1af2ef0 msgr2=0x555fe1af10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:48.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.659+0000 7f2476ffd700 1 --2- 172.21.15.27:0/4149870014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1af2ef0 0x555fe1af10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.659+0000 7f2476ffd700 1 -- 172.21.15.27:0/4149870014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1ace3a0 msgr2=0x555fe1af0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:48.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.659+0000 7f2476ffd700 1 --2- 172.21.15.27:0/4149870014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1ace3a0 0x555fe1af0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.659+0000 7f2476ffd700 1 -- 172.21.15.27:0/4149870014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24780cb8f0 con 0x555fe1af2b20 2024-01-11T09:43:48.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.659+0000 7f2477fff700 1 --2- 172.21.15.27:0/4149870014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1af2ef0 0x555fe1af10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:48.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.659+0000 7f2476ffd700 1 --2- 172.21.15.27:0/4149870014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 0x555fe1af0b70 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f246c057310 tx=0x7f246c027000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3023d858c212c968 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:48.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.659+0000 7f2455ffb700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T09:43:48.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.660+0000 7f2455ffb700 1 -- 172.21.15.27:0/4149870014 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c059f00 con 0x555fe1af2b20 2024-01-11T09:43:48.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.660+0000 7f2455ffb700 1 -- 172.21.15.27:0/4149870014 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(198..198 src has 1..198) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f246c05dc50 con 0x555fe1af2b20 2024-01-11T09:43:48.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.660+0000 7f248c24ab80 1 -- 172.21.15.27:0/4149870014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555fe1ac6560 con 0x555fe1af2b20 2024-01-11T09:43:48.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.660+0000 7f2455ffb700 1 -- 172.21.15.27:0/4149870014 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) v2 ==== 24+0+0 (secure 0 0 0) 0x7f246c05dea0 con 0x555fe1af2b20 2024-01-11T09:43:48.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.660+0000 7f248c24ab80 10 client.10242 fetch_fsmap learned FSMap version 102 2024-01-11T09:43:48.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.660+0000 7f248c24ab80 1 -- 172.21.15.27:0/4149870014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) v3 -- 0x555fe19f03c0 con 0x555fe1af2b20 2024-01-11T09:43:48.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.660+0000 7f2455ffb700 1 -- 172.21.15.27:0/4149870014 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246c061020 con 0x555fe1af2b20 2024-01-11T09:43:48.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.660+0000 7f2455ffb700 1 -- 172.21.15.27:0/4149870014 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 102) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f246c05d7c0 con 0x555fe1af2b20 2024-01-11T09:43:48.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.660+0000 7f248c24ab80 10 client.10242 fetch_fsmap finished waiting for FSMap version 102 2024-01-11T09:43:48.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.660+0000 7f248c24ab80 10 client.10242 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-11T09:43:48.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.660+0000 7f248c24ab80 20 client.10242 populate_metadata read hostname 'smithi027' 2024-01-11T09:43:48.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.660+0000 7f248c24ab80 1 --2- 172.21.15.27:0/4149870014 >> [v2:172.21.15.27:6833/1557605486,v1:172.21.15.27:6834/1557605486] conn(0x555fe1b00910 0x555fe1b00ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.660+0000 7f248c24ab80 4 client.10242 mds_command: new command op to 10113 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-11T09:43:48.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.660+0000 7f248c24ab80 1 -- 172.21.15.27:0/4149870014 --> [v2:172.21.15.27:6833/1557605486,v1:172.21.15.27:6834/1557605486] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555fe1aa30e0 con 0x555fe1b00910 2024-01-11T09:43:48.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.661+0000 7f24777fe700 1 --2- 172.21.15.27:0/4149870014 >> [v2:172.21.15.27:6833/1557605486,v1:172.21.15.27:6834/1557605486] conn(0x555fe1b00910 0x555fe1b00ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.661+0000 7f2455ffb700 10 client.10242 ms_handle_connect on v2:172.21.15.27:6833/1557605486 2024-01-11T09:43:48.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.661+0000 7f24777fe700 1 --2- 172.21.15.27:0/4149870014 >> [v2:172.21.15.27:6833/1557605486,v1:172.21.15.27:6834/1557605486] conn(0x555fe1b00910 0x555fe1b00ce0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2960acdc6e8fbdd server_cookie=e2518a5b25624996 in_seq=0 out_seq=0 2024-01-11T09:43:48.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.662+0000 7f2455ffb700 1 -- 172.21.15.27:0/4149870014 <== mds.0 v2:172.21.15.27:6833/1557605486 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x555fe1aa30e0 con 0x555fe1b00910 2024-01-11T09:43:48.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.662+0000 7f2455ffb700 10 client.10242 handle_command_reply: tid=0 2024-01-11T09:43:48.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.662+0000 7f248c24ab80 1 client.10242 shutdown 2024-01-11T09:43:48.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.662+0000 7f248c24ab80 1 -- 172.21.15.27:0/4149870014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 msgr2=0x555fe1af0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:48.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.662+0000 7f248c24ab80 1 --2- 172.21.15.27:0/4149870014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 0x555fe1af0b70 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f246c057310 tx=0x7f246c027000 comp rx=0 tx=0).stop 2024-01-11T09:43:48.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.662+0000 7f248c24ab80 1 -- 172.21.15.27:0/4149870014 shutdown_connections 2024-01-11T09:43:48.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.662+0000 7f248c24ab80 1 --2- 172.21.15.27:0/4149870014 >> [v2:172.21.15.27:6833/1557605486,v1:172.21.15.27:6834/1557605486] conn(0x555fe1b00910 0x555fe1b00ce0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.662+0000 7f248c24ab80 1 --2- 172.21.15.27:0/4149870014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1af2ef0 0x555fe1af10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.662+0000 7f248c24ab80 1 --2- 172.21.15.27:0/4149870014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1ace3a0 0x555fe1af0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.662+0000 7f2455ffb700 0 client.10242 ms_handle_reset on v2:172.21.15.27:6833/1557605486 2024-01-11T09:43:48.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.662+0000 7f248c24ab80 1 --2- 172.21.15.27:0/4149870014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 0x555fe1af0b70 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.662+0000 7f248c24ab80 1 -- 172.21.15.27:0/4149870014 >> 172.21.15.27:0/4149870014 conn(0x555fe1ad0300 msgr2=0x555fe1acd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:48.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.662+0000 7f248c24ab80 1 -- 172.21.15.27:0/4149870014 shutdown_connections 2024-01-11T09:43:48.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.662+0000 7f248c24ab80 1 -- 172.21.15.27:0/4149870014 wait complete. 2024-01-11T09:43:48.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.663+0000 7f248c24ab80 20 client.10242 trim_cache size 0 max 16384 2024-01-11T09:43:48.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.680+0000 7f248c24ab80 1 Processor -- start 2024-01-11T09:43:48.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.681+0000 7f248c24ab80 1 -- start start 2024-01-11T09:43:48.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.681+0000 7f248c24ab80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1ace3a0 0x555fe1af1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.681+0000 7f248c24ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 0x555fe1b00d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.681+0000 7f248c24ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1af2ef0 0x555fe1acd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.681+0000 7f248c24ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fe18b91e0 con 0x555fe1af2b20 2024-01-11T09:43:48.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.681+0000 7f248c24ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fe1af17c0 con 0x555fe1af2ef0 2024-01-11T09:43:48.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.681+0000 7f248c24ab80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x555fe1afe300 con 0x555fe1ace3a0 2024-01-11T09:43:48.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.682+0000 7f2476ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 0x555fe1b00d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.682+0000 7f2476ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 0x555fe1b00d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57484/0 (socket says 172.21.15.27:57484) 2024-01-11T09:43:48.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.682+0000 7f2476ffd700 1 -- 172.21.15.27:0/1946247036 learned_addr learned my addr 172.21.15.27:0/1946247036 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:48.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.682+0000 7f2477fff700 1 --2- 172.21.15.27:0/1946247036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1af2ef0 0x555fe1acd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.682+0000 7f24777fe700 1 --2- 172.21.15.27:0/1946247036 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1ace3a0 0x555fe1af1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.682+0000 7f2476ffd700 1 -- 172.21.15.27:0/1946247036 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1ace3a0 msgr2=0x555fe1af1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:48.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.682+0000 7f2476ffd700 1 --2- 172.21.15.27:0/1946247036 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1ace3a0 0x555fe1af1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.682+0000 7f2476ffd700 1 -- 172.21.15.27:0/1946247036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1af2ef0 msgr2=0x555fe1acd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:48.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.682+0000 7f2476ffd700 1 --2- 172.21.15.27:0/1946247036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1af2ef0 0x555fe1acd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.682+0000 7f2476ffd700 1 -- 172.21.15.27:0/1946247036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555fe18881d0 con 0x555fe1af2b20 2024-01-11T09:43:48.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.682+0000 7f24777fe700 1 --2- 172.21.15.27:0/1946247036 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1ace3a0 0x555fe1af1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:48.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.682+0000 7f2476ffd700 1 --2- 172.21.15.27:0/1946247036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 0x555fe1b00d50 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f246c056d80 tx=0x7f246c06c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b5c0c88a3072f53 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:48.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.683+0000 7f245d7fa700 1 -- 172.21.15.27:0/1946247036 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c05dea0 con 0x555fe1af2b20 2024-01-11T09:43:48.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.683+0000 7f245d7fa700 1 -- 172.21.15.27:0/1946247036 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246c061070 con 0x555fe1af2b20 2024-01-11T09:43:48.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.683+0000 7f245d7fa700 1 -- 172.21.15.27:0/1946247036 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c084120 con 0x555fe1af2b20 2024-01-11T09:43:48.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.683+0000 7f248c24ab80 1 -- 172.21.15.27:0/1946247036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 msgr2=0x555fe1b00d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:48.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.683+0000 7f248c24ab80 1 --2- 172.21.15.27:0/1946247036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 0x555fe1b00d50 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f246c056d80 tx=0x7f246c06c040 comp rx=0 tx=0).stop 2024-01-11T09:43:48.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.683+0000 7f248c24ab80 1 -- 172.21.15.27:0/1946247036 shutdown_connections 2024-01-11T09:43:48.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.683+0000 7f248c24ab80 1 --2- 172.21.15.27:0/1946247036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1af2ef0 0x555fe1acd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.683+0000 7f248c24ab80 1 --2- 172.21.15.27:0/1946247036 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1ace3a0 0x555fe1af1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.683+0000 7f248c24ab80 1 --2- 172.21.15.27:0/1946247036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 0x555fe1b00d50 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.683+0000 7f248c24ab80 1 -- 172.21.15.27:0/1946247036 >> 172.21.15.27:0/1946247036 conn(0x555fe1ad0300 msgr2=0x555fe1ace790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:48.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.683+0000 7f248c24ab80 1 -- 172.21.15.27:0/1946247036 shutdown_connections 2024-01-11T09:43:48.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.683+0000 7f248c24ab80 1 -- 172.21.15.27:0/1946247036 wait complete. 2024-01-11T09:43:48.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.684+0000 7f248c24ab80 1 -- start start 2024-01-11T09:43:48.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.684+0000 7f248c24ab80 1 Processor -- start 2024-01-11T09:43:48.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.685+0000 7f248c24ab80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1ace3a0 0x555fe1aefd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.685+0000 7f248c24ab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 0x555fe1b7ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.685+0000 7f248c24ab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1af2ef0 0x555fe1b80790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.685+0000 7f248c24ab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fe1b2a9a0 con 0x555fe1af2b20 2024-01-11T09:43:48.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.685+0000 7f248c24ab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fe1b7cfb0 con 0x555fe1af2ef0 2024-01-11T09:43:48.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.685+0000 7f248c24ab80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x555fe1afa570 con 0x555fe1ace3a0 2024-01-11T09:43:48.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.685+0000 7f248c24ab80 10 client.0 _ll_register_callbacks cb 0x555fe1830a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T09:43:48.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.685+0000 7f2476ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 0x555fe1b7ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.685+0000 7f2476ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 0x555fe1b7ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57494/0 (socket says 172.21.15.27:57494) 2024-01-11T09:43:48.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.685+0000 7f2476ffd700 1 -- 172.21.15.27:0/277957235 learned_addr learned my addr 172.21.15.27:0/277957235 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:48.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.686+0000 7f2477fff700 1 --2- 172.21.15.27:0/277957235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1af2ef0 0x555fe1b80790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.686+0000 7f2476ffd700 1 -- 172.21.15.27:0/277957235 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1ace3a0 msgr2=0x555fe1aefd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:48.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.686+0000 7f24777fe700 1 --2- 172.21.15.27:0/277957235 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1ace3a0 0x555fe1aefd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.686+0000 7f2476ffd700 1 --2- 172.21.15.27:0/277957235 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1ace3a0 0x555fe1aefd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.686+0000 7f2476ffd700 1 -- 172.21.15.27:0/277957235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1af2ef0 msgr2=0x555fe1b80790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:48.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.686+0000 7f2476ffd700 1 --2- 172.21.15.27:0/277957235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1af2ef0 0x555fe1b80790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.686+0000 7f2476ffd700 1 -- 172.21.15.27:0/277957235 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555fe19f03c0 con 0x555fe1af2b20 2024-01-11T09:43:48.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.686+0000 7f2477fff700 1 --2- 172.21.15.27:0/277957235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1af2ef0 0x555fe1b80790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:43:48.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.686+0000 7f24777fe700 1 --2- 172.21.15.27:0/277957235 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1ace3a0 0x555fe1aefd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:43:48.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.686+0000 7f2476ffd700 1 --2- 172.21.15.27:0/277957235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 0x555fe1b7ca70 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f246c0552a0 tx=0x7f246c026000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=995c0d22ef571df8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:48.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.686+0000 7f2455ffb700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T09:43:48.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.686+0000 7f2455ffb700 1 -- 172.21.15.27:0/277957235 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c05dea0 con 0x555fe1af2b20 2024-01-11T09:43:48.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.686+0000 7f2455ffb700 1 -- 172.21.15.27:0/277957235 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(198..198 src has 1..198) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f246c062080 con 0x555fe1af2b20 2024-01-11T09:43:48.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.686+0000 7f248c24ab80 1 -- 172.21.15.27:0/277957235 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f246c0573b0 con 0x555fe1af2b20 2024-01-11T09:43:48.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.686+0000 7f2455ffb700 1 -- 172.21.15.27:0/277957235 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) v2 ==== 24+0+0 (secure 0 0 0) 0x7f246c084120 con 0x555fe1af2b20 2024-01-11T09:43:48.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.686+0000 7f248c24ab80 10 client.10248 fetch_fsmap learned FSMap version 102 2024-01-11T09:43:48.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.686+0000 7f248c24ab80 1 -- 172.21.15.27:0/277957235 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) v3 -- 0x555fe1861d60 con 0x555fe1af2b20 2024-01-11T09:43:48.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.687+0000 7f2455ffb700 1 -- 172.21.15.27:0/277957235 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246c0573b0 con 0x555fe1af2b20 2024-01-11T09:43:48.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.687+0000 7f2455ffb700 1 -- 172.21.15.27:0/277957235 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 102) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f246c05d7c0 con 0x555fe1af2b20 2024-01-11T09:43:48.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.687+0000 7f248c24ab80 10 client.10248 fetch_fsmap finished waiting for FSMap version 102 2024-01-11T09:43:48.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.687+0000 7f248c24ab80 10 client.10248 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-11T09:43:48.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.687+0000 7f248c24ab80 20 client.10248 populate_metadata read hostname 'smithi027' 2024-01-11T09:43:48.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.687+0000 7f248c24ab80 1 --2- 172.21.15.27:0/277957235 >> [v2:172.21.15.27:6833/1557605486,v1:172.21.15.27:6834/1557605486] conn(0x555fe1b80cd0 0x555fe1b00ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.687+0000 7f248c24ab80 4 client.10248 mds_command: new command op to 10113 tid=0[{"prefix": "status"}] 2024-01-11T09:43:48.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.687+0000 7f248c24ab80 1 -- 172.21.15.27:0/277957235 --> [v2:172.21.15.27:6833/1557605486,v1:172.21.15.27:6834/1557605486] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555fe19c48d0 con 0x555fe1b80cd0 2024-01-11T09:43:48.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.687+0000 7f24777fe700 1 --2- 172.21.15.27:0/277957235 >> [v2:172.21.15.27:6833/1557605486,v1:172.21.15.27:6834/1557605486] conn(0x555fe1b80cd0 0x555fe1b00ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.687+0000 7f2455ffb700 10 client.10248 ms_handle_connect on v2:172.21.15.27:6833/1557605486 2024-01-11T09:43:48.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.687+0000 7f24777fe700 1 --2- 172.21.15.27:0/277957235 >> [v2:172.21.15.27:6833/1557605486,v1:172.21.15.27:6834/1557605486] conn(0x555fe1b80cd0 0x555fe1b00ce0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=271043c31d610abc server_cookie=a6148bc2057763a0 in_seq=0 out_seq=0 2024-01-11T09:43:48.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.688+0000 7f2455ffb700 1 -- 172.21.15.27:0/277957235 <== mds.0 v2:172.21.15.27:6833/1557605486 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x555fe19c48d0 con 0x555fe1b80cd0 2024-01-11T09:43:48.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.688+0000 7f2455ffb700 10 client.10248 handle_command_reply: tid=0 2024-01-11T09:43:48.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.688+0000 7f248c24ab80 1 client.10248 shutdown 2024-01-11T09:43:48.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.688+0000 7f248c24ab80 1 -- 172.21.15.27:0/277957235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 msgr2=0x555fe1b7ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:48.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.688+0000 7f248c24ab80 1 --2- 172.21.15.27:0/277957235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 0x555fe1b7ca70 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f246c0552a0 tx=0x7f246c026000 comp rx=0 tx=0).stop 2024-01-11T09:43:48.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.688+0000 7f248c24ab80 1 -- 172.21.15.27:0/277957235 shutdown_connections 2024-01-11T09:43:48.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.688+0000 7f248c24ab80 1 --2- 172.21.15.27:0/277957235 >> [v2:172.21.15.27:6833/1557605486,v1:172.21.15.27:6834/1557605486] conn(0x555fe1b80cd0 0x555fe1b00ce0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.688+0000 7f248c24ab80 1 --2- 172.21.15.27:0/277957235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555fe1af2ef0 0x555fe1b80790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.688+0000 7f248c24ab80 1 --2- 172.21.15.27:0/277957235 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x555fe1ace3a0 0x555fe1aefd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.688+0000 7f248c24ab80 1 --2- 172.21.15.27:0/277957235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555fe1af2b20 0x555fe1b7ca70 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.688+0000 7f248c24ab80 1 -- 172.21.15.27:0/277957235 >> 172.21.15.27:0/277957235 conn(0x555fe1ad0300 msgr2=0x555fe1acd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:48.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.688+0000 7f2455ffb700 0 client.10248 ms_handle_reset on v2:172.21.15.27:6833/1557605486 2024-01-11T09:43:48.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.688+0000 7f248c24ab80 1 -- 172.21.15.27:0/277957235 shutdown_connections 2024-01-11T09:43:48.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.688+0000 7f248c24ab80 1 -- 172.21.15.27:0/277957235 wait complete. 2024-01-11T09:43:48.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.688+0000 7f248c24ab80 20 client.10248 trim_cache size 0 max 16384 2024-01-11T09:43:48.685 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:43:48.685 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "8fddb4a7-9d66-40f1-892d-6fe86c728ee4", 2024-01-11T09:43:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-11T09:43:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 10113, 2024-01-11T09:43:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-11T09:43:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-11T09:43:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-11T09:43:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 6.6342556259999999, 2024-01-11T09:43:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 102, 2024-01-11T09:43:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 198, 2024-01-11T09:43:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 198, 2024-01-11T09:43:48.686 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 13.288124254 2024-01-11T09:43:48.686 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:43:48.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.689+0000 7f247dbff700 1 -- 172.21.15.27:0/1302170830 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2464066f30 msgr2=0x7f24640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:48.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.689+0000 7f247dbff700 1 --2- 172.21.15.27:0/1302170830 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2464066f30 0x7f24640693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2468009f40 tx=0x7f24680091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.689+0000 7f247dbff700 1 -- 172.21.15.27:0/1302170830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2478156ab0 msgr2=0x7f24780ca9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:48.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.689+0000 7f247dbff700 1 --2- 172.21.15.27:0/1302170830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2478156ab0 0x7f24780ca9c0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f24780ae380 tx=0x7f246c017000 comp rx=0 tx=0).stop 2024-01-11T09:43:48.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.689+0000 7f247dbff700 1 -- 172.21.15.27:0/1302170830 shutdown_connections 2024-01-11T09:43:48.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.689+0000 7f247dbff700 1 --2- 172.21.15.27:0/1302170830 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2464066f30 0x7f24640693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.689+0000 7f247dbff700 1 --2- 172.21.15.27:0/1302170830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24781573c0 0x7f2478155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.689+0000 7f247dbff700 1 --2- 172.21.15.27:0/1302170830 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24780cc9b0 0x7f247816c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.689+0000 7f247dbff700 1 --2- 172.21.15.27:0/1302170830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2478156ab0 0x7f24780ca9c0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.689+0000 7f247dbff700 1 -- 172.21.15.27:0/1302170830 >> 172.21.15.27:0/1302170830 conn(0x7f24780680b0 msgr2=0x7f24780691b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:48.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.689+0000 7f247dbff700 1 -- 172.21.15.27:0/1302170830 shutdown_connections 2024-01-11T09:43:48.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.689+0000 7f247dbff700 1 -- 172.21.15.27:0/1302170830 wait complete. 2024-01-11T09:43:48.699 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-11T09:43:48.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.906+0000 7f7072fff700 1 Processor -- start 2024-01-11T09:43:48.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.907+0000 7f7072fff700 1 -- start start 2024-01-11T09:43:48.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.908+0000 7f7072fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706c06b610 0x7f706c158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.908+0000 7f7072fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706c158b60 con 0x7f706c0b19d0 2024-01-11T09:43:48.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.908+0000 7f7072fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706c1599b0 con 0x7f706c1575e0 2024-01-11T09:43:48.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.908+0000 7f7072fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706c0cb5e0 con 0x7f706c06b610 2024-01-11T09:43:48.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.908+0000 7f706bfff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f706c0b19d0 0x7f706c159e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:32920/0 (socket says 172.21.15.27:32920) 2024-01-11T09:43:48.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.908+0000 7f706bfff700 1 -- 172.21.15.27:0/2938800919 learned_addr learned my addr 172.21.15.27:0/2938800919 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:48.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.908+0000 7f707159c700 1 --2- 172.21.15.27:0/2938800919 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706c06b610 0x7f706c158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.909+0000 7f706b7fe700 1 -- 172.21.15.27:0/2938800919 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896191491 0 0) 0x7f7058001110 con 0x7f706c0b19d0 2024-01-11T09:43:48.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.909+0000 7f706b7fe700 1 -- 172.21.15.27:0/2938800919 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f705c003570 con 0x7f706c0b19d0 2024-01-11T09:43:48.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.909+0000 7f706b7fe700 1 -- 172.21.15.27:0/2938800919 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4073617599 0 0) 0x7f705c003570 con 0x7f706c0b19d0 2024-01-11T09:43:48.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.909+0000 7f706b7fe700 1 -- 172.21.15.27:0/2938800919 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706c06b610 msgr2=0x7f706c158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:48.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.909+0000 7f706b7fe700 1 --2- 172.21.15.27:0/2938800919 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706c06b610 0x7f706c158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.909+0000 7f706b7fe700 1 -- 172.21.15.27:0/2938800919 >> v1:172.21.15.77:6789/0 conn(0x7f706c1575e0 legacy=0x7f706c0b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:48.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.909+0000 7f706b7fe700 1 -- 172.21.15.27:0/2938800919 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c0c0f90 con 0x7f706c0b19d0 2024-01-11T09:43:48.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.909+0000 7f706b7fe700 1 -- 172.21.15.27:0/2938800919 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f7058001600 con 0x7f706c0b19d0 2024-01-11T09:43:48.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.909+0000 7f706b7fe700 1 -- 172.21.15.27:0/2938800919 >> v1:172.21.15.27:6789/0 conn(0x7f706c0b19d0 legacy=0x7f706c159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:48.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.909+0000 7f706b7fe700 1 --2- 172.21.15.27:0/2938800919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f705c003a90 0x7f705c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.909+0000 7f706b7fe700 1 --2- 172.21.15.27:0/2938800919 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706c06b610 0x7f705c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.909+0000 7f706b7fe700 1 --2- 172.21.15.27:0/2938800919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705c004d50 0x7f705c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.909+0000 7f706b7fe700 1 -- 172.21.15.27:0/2938800919 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706c0cb5e0 con 0x7f705c004d50 2024-01-11T09:43:48.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.909+0000 7f706b7fe700 1 -- 172.21.15.27:0/2938800919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705c00bb40 con 0x7f705c003a90 2024-01-11T09:43:48.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.909+0000 7f706b7fe700 1 -- 172.21.15.27:0/2938800919 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f705c00bd00 con 0x7f706c06b610 2024-01-11T09:43:48.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.909+0000 7f707159c700 1 --2- 172.21.15.27:0/2938800919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705c004d50 0x7f705c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.909+0000 7f706bfff700 1 --2- 172.21.15.27:0/2938800919 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706c06b610 0x7f705c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.909+0000 7f7070d9b700 1 --2- 172.21.15.27:0/2938800919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f705c003a90 0x7f705c003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.910+0000 7f707159c700 1 -- 172.21.15.27:0/2938800919 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706c06b610 msgr2=0x7f705c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:48.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.910+0000 7f707159c700 1 --2- 172.21.15.27:0/2938800919 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706c06b610 0x7f705c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.910+0000 7f707159c700 1 -- 172.21.15.27:0/2938800919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f705c003a90 msgr2=0x7f705c003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:48.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.910+0000 7f707159c700 1 --2- 172.21.15.27:0/2938800919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f705c003a90 0x7f705c003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.910+0000 7f707159c700 1 -- 172.21.15.27:0/2938800919 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705c003570 con 0x7f705c004d50 2024-01-11T09:43:48.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.910+0000 7f7070d9b700 1 --2- 172.21.15.27:0/2938800919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f705c003a90 0x7f705c003e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:48.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.910+0000 7f707159c700 1 --2- 172.21.15.27:0/2938800919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705c004d50 0x7f705c0051c0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f706400e210 tx=0x7f706400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d3b8f7bedd2349 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:48.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.910+0000 7f706b7fe700 1 -- 172.21.15.27:0/2938800919 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706400eac0 con 0x7f705c004d50 2024-01-11T09:43:48.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.910+0000 7f706b7fe700 1 -- 172.21.15.27:0/2938800919 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706400f040 con 0x7f705c004d50 2024-01-11T09:43:48.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.910+0000 7f706b7fe700 1 -- 172.21.15.27:0/2938800919 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706400ee70 con 0x7f705c004d50 2024-01-11T09:43:48.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.911+0000 7f7072fff700 1 -- 172.21.15.27:0/2938800919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705c004d50 msgr2=0x7f705c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:48.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.911+0000 7f7072fff700 1 --2- 172.21.15.27:0/2938800919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705c004d50 0x7f705c0051c0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f706400e210 tx=0x7f706400b040 comp rx=0 tx=0).stop 2024-01-11T09:43:48.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.911+0000 7f707159c700 1 -- 172.21.15.27:0/2938800919 reap_dead start 2024-01-11T09:43:48.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.911+0000 7f7072fff700 1 -- 172.21.15.27:0/2938800919 shutdown_connections 2024-01-11T09:43:48.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.911+0000 7f7072fff700 1 -- 172.21.15.27:0/2938800919 >> 172.21.15.27:0/2938800919 conn(0x7f706c0680e0 msgr2=0x7f706c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:48.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.911+0000 7f7072fff700 1 -- 172.21.15.27:0/2938800919 shutdown_connections 2024-01-11T09:43:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.911+0000 7f7072fff700 1 -- 172.21.15.27:0/2938800919 wait complete. 2024-01-11T09:43:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.912+0000 7f7072fff700 1 Processor -- start 2024-01-11T09:43:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.913+0000 7f7072fff700 1 -- start start 2024-01-11T09:43:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.913+0000 7f7072fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f705c004d50 0x7f706c0c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.913+0000 7f7072fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f706c0c5f80 0x7f706c0c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.913+0000 7f7072fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706c0b0080 0x7f706c0b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.913+0000 7f7072fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706c0c6930 con 0x7f706c0c5f80 2024-01-11T09:43:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.913+0000 7f7072fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706c0b0a30 con 0x7f705c004d50 2024-01-11T09:43:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.913+0000 7f7072fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706c0c7730 con 0x7f706c0b0080 2024-01-11T09:43:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.914+0000 7f706bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f706c0c5f80 0x7f706c0c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.914+0000 7f706bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f706c0c5f80 0x7f706c0c63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57512/0 (socket says 172.21.15.27:57512) 2024-01-11T09:43:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.914+0000 7f706bfff700 1 -- 172.21.15.27:0/2150143906 learned_addr learned my addr 172.21.15.27:0/2150143906 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.914+0000 7f7070d9b700 1 --2- 172.21.15.27:0/2150143906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f705c004d50 0x7f706c0c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.914+0000 7f707159c700 1 --2- 172.21.15.27:0/2150143906 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706c0b0080 0x7f706c0b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.914+0000 7f706bfff700 1 -- 172.21.15.27:0/2150143906 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706c0b0080 msgr2=0x7f706c0b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.914+0000 7f706bfff700 1 --2- 172.21.15.27:0/2150143906 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706c0b0080 0x7f706c0b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.914+0000 7f706bfff700 1 -- 172.21.15.27:0/2150143906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f705c004d50 msgr2=0x7f706c0c7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.914+0000 7f706bfff700 1 --2- 172.21.15.27:0/2150143906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f705c004d50 0x7f706c0c7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.914+0000 7f706bfff700 1 -- 172.21.15.27:0/2150143906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c0c8100 con 0x7f706c0c5f80 2024-01-11T09:43:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.914+0000 7f7070d9b700 1 --2- 172.21.15.27:0/2150143906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f705c004d50 0x7f706c0c7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:43:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.914+0000 7f706bfff700 1 --2- 172.21.15.27:0/2150143906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f706c0c5f80 0x7f706c0c63f0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f70580073c0 tx=0x7f70580079d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.915+0000 7f7068ff9700 1 -- 172.21.15.27:0/2150143906 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7058007780 con 0x7f706c0c5f80 2024-01-11T09:43:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.915+0000 7f7068ff9700 1 -- 172.21.15.27:0/2150143906 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705800f8d0 con 0x7f706c0c5f80 2024-01-11T09:43:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.915+0000 7f7072fff700 1 -- 172.21.15.27:0/2150143906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f706c0b1620 con 0x7f706c0c5f80 2024-01-11T09:43:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.915+0000 7f7068ff9700 1 -- 172.21.15.27:0/2150143906 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705800fc90 con 0x7f706c0c5f80 2024-01-11T09:43:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.915+0000 7f7072fff700 1 -- 172.21.15.27:0/2150143906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f706c0c6e00 con 0x7f706c0c5f80 2024-01-11T09:43:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.916+0000 7f7072fff700 1 -- 172.21.15.27:0/2150143906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f706c0718d0 con 0x7f706c0c5f80 2024-01-11T09:43:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.917+0000 7f7068ff9700 1 -- 172.21.15.27:0/2150143906 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f70580133b0 con 0x7f706c0c5f80 2024-01-11T09:43:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.918+0000 7f7068ff9700 1 --2- 172.21.15.27:0/2150143906 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7050066f30 0x7f70500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.918+0000 7f7068ff9700 1 -- 172.21.15.27:0/2150143906 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f7058083d10 con 0x7f706c0c5f80 2024-01-11T09:43:48.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.918+0000 7f7070d9b700 1 --2- 172.21.15.27:0/2150143906 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7050066f30 0x7f70500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:48.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.919+0000 7f7070d9b700 1 --2- 172.21.15.27:0/2150143906 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7050066f30 0x7f70500693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7060000c00 tx=0x7f7060009040 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:48.922+0000 7f7068ff9700 1 -- 172.21.15.27:0/2150143906 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7058089020 con 0x7f706c0c5f80 2024-01-11T09:43:49.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:49.174+0000 7f7072fff700 1 -- 172.21.15.27:0/2150143906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f706c158b60 con 0x7f706c0c5f80 2024-01-11T09:43:49.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:49.174+0000 7f7068ff9700 1 -- 172.21.15.27:0/2150143906 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v102) v1 ==== 53+0+73 (secure 0 0 0) 0x7f705801a060 con 0x7f706c0c5f80 2024-01-11T09:43:49.166 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-11T09:43:49.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:49.177+0000 7f7072fff700 1 -- 172.21.15.27:0/2150143906 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7050066f30 msgr2=0x7f70500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:49.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:49.177+0000 7f7072fff700 1 --2- 172.21.15.27:0/2150143906 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7050066f30 0x7f70500693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7060000c00 tx=0x7f7060009040 comp rx=0 tx=0).stop 2024-01-11T09:43:49.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:49.177+0000 7f7072fff700 1 -- 172.21.15.27:0/2150143906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f706c0c5f80 msgr2=0x7f706c0c63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:49.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:49.177+0000 7f7072fff700 1 --2- 172.21.15.27:0/2150143906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f706c0c5f80 0x7f706c0c63f0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f70580073c0 tx=0x7f70580079d0 comp rx=0 tx=0).stop 2024-01-11T09:43:49.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:49.177+0000 7f7072fff700 1 -- 172.21.15.27:0/2150143906 shutdown_connections 2024-01-11T09:43:49.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:49.177+0000 7f7072fff700 1 --2- 172.21.15.27:0/2150143906 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f706c0b0080 0x7f706c0b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:49.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:49.177+0000 7f7072fff700 1 --2- 172.21.15.27:0/2150143906 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7050066f30 0x7f70500693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:49.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:49.177+0000 7f7072fff700 1 --2- 172.21.15.27:0/2150143906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f705c004d50 0x7f706c0c7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:49.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:49.177+0000 7f7072fff700 1 --2- 172.21.15.27:0/2150143906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f706c0c5f80 0x7f706c0c63f0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:49.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:49.177+0000 7f7072fff700 1 -- 172.21.15.27:0/2150143906 >> 172.21.15.27:0/2150143906 conn(0x7f706c0680e0 msgr2=0x7f706c06e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:49.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:49.177+0000 7f7072fff700 1 -- 172.21.15.27:0/2150143906 shutdown_connections 2024-01-11T09:43:49.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:49.177+0000 7f7072fff700 1 -- 172.21.15.27:0/2150143906 wait complete. 2024-01-11T09:43:49.186 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-11T09:43:49.186 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-11T09:43:49.187 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-11T09:43:49.187 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-11T09:43:49.187 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-11T09:43:49.187 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-11T09:43:49.187 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-11T09:43:49.187 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-11T09:43:49.187 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T09:43:49.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:43:49.187 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-11T09:43:49.235 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T09:43:49.235 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T09:43:49.235 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T09:43:49.235 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:43:49.236 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-11T09:43:49.236 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:43:49.236 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T09:43:49.236 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-11T09:43:49.236 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-11T09:43:49.236 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T09:43:49.236 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-11T09:43:49.236 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-11T09:43:49.236 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-11T09:43:49.236 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-11T09:43:49.236 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-11T09:43:49.236 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-11T09:43:49.236 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 36893sec preferred_lft 36893sec 2024-01-11T09:43:49.236 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-11T09:43:49.236 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:43:49.237 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T09:43:49.237 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-11T09:43:49.237 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-11T09:43:49.237 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T09:43:49.237 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff 2024-01-11T09:43:49.237 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-11T09:43:49.237 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:43:49.237 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::acd2:b3ff:fe6a:db4b/64 scope link 2024-01-11T09:43:49.237 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:43:49.237 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T09:43:49.237 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:fa:85:1b:3a:bc brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-11T09:43:49.238 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ecfa:85ff:fe1b:3abc/64 scope link 2024-01-11T09:43:49.238 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:43:49.238 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T09:43:49.238 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-11T09:43:49.238 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ec88:faff:fe17:41aa/64 scope link 2024-01-11T09:43:49.238 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:43:49.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:43:49.239 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-11T09:43:49.309 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-11T09:43:49.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:43:49.312 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-11T09:43:49.363 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-11T09:43:49.363 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-11T09:43:49.363 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-11T09:43:49.364 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-11T09:43:49.364 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-11T09:43:49.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:43:49.365 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T09:43:49.365 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T09:43:49.365 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-11T09:43:49.365 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-11T09:43:49.365 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-11T09:43:49.365 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T09:43:49.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:43:49.542 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T09:43:49.590 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-11T09:43:49.590 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-11T09:43:49.591 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T09:43:49.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:43:49.592 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-11T09:43:49.665 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T09:43:49.665 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-11T09:43:49.713 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-11T09:43:49.713 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T09:43:49.713 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-11T09:43:49.799 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-11T09:43:49.800 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-11T09:43:49.800 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:43:49.902 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:43:49.902 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T09:43:49.902 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T09:43:49.917 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:43:49.918 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T09:43:49.918 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T09:43:49.918 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T09:43:49.918 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T09:43:49.939 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:43:49.939 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T09:43:49.939 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T09:43:49.939 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T09:43:49.939 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T09:43:49.939 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790" 2024-01-11T09:43:49.939 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790:/ 2024-01-11T09:43:49.939 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=8fddb4a7-9d66-40f1-892d-6fe86c728ee4" will pass to kernel 2024-01-11T09:43:49.945 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-11T09:43:49.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:43:49.945 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-11T09:43:50.019 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-11T09:43:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.233+0000 7f01343ff700 1 Processor -- start 2024-01-11T09:43:50.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.233+0000 7f01343ff700 1 -- start start 2024-01-11T09:43:50.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.234+0000 7f01343ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f012c06bf10 0x7f012c159c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:50.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.234+0000 7f01343ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f012c0b0a40 0x7f012c15da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:50.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.235+0000 7f01343ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012c15d290 con 0x7f012c15df40 2024-01-11T09:43:50.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.235+0000 7f01343ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f012c15d570 con 0x7f012c0b0a40 2024-01-11T09:43:50.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.235+0000 7f01343ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f012c154180 con 0x7f012c06bf10 2024-01-11T09:43:50.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.235+0000 7f013199a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f012c0b0a40 0x7f012c15da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:50.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.235+0000 7f013199a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f012c0b0a40 0x7f012c15da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57522/0 (socket says 172.21.15.27:57522) 2024-01-11T09:43:50.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.235+0000 7f013199a700 1 -- 172.21.15.27:0/1873766834 learned_addr learned my addr 172.21.15.27:0/1873766834 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:50.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.235+0000 7f013219b700 1 --2- 172.21.15.27:0/1873766834 >> v2:172.21.15.77:3300/0 conn(0x7f012c06bf10 0x7f012c159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:50.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.235+0000 7f013199a700 1 -- 172.21.15.27:0/1873766834 >> v2:172.21.15.77:3300/0 conn(0x7f012c06bf10 msgr2=0x7f012c159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:50.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.235+0000 7f013199a700 1 --2- 172.21.15.27:0/1873766834 >> v2:172.21.15.77:3300/0 conn(0x7f012c06bf10 0x7f012c159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:50.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.235+0000 7f013199a700 1 -- 172.21.15.27:0/1873766834 >> v1:172.21.15.77:6789/0 conn(0x7f012c15df40 legacy=0x7f012c068720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:50.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.235+0000 7f013199a700 1 -- 172.21.15.27:0/1873766834 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012c15a510 con 0x7f012c0b0a40 2024-01-11T09:43:50.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.236+0000 7f013199a700 1 --2- 172.21.15.27:0/1873766834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f012c0b0a40 0x7f012c15da00 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f012c06f450 tx=0x7f011c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff917d29d31e846d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:50.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.236+0000 7f0131199700 1 -- 172.21.15.27:0/1873766834 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c009e70 con 0x7f012c0b0a40 2024-01-11T09:43:50.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.236+0000 7f0131199700 1 -- 172.21.15.27:0/1873766834 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011c00b860 con 0x7f012c0b0a40 2024-01-11T09:43:50.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.236+0000 7f0131199700 1 -- 172.21.15.27:0/1873766834 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c00bc00 con 0x7f012c0b0a40 2024-01-11T09:43:50.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.236+0000 7f01343ff700 1 -- 172.21.15.27:0/1873766834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f012c0b0a40 msgr2=0x7f012c15da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:50.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.236+0000 7f01343ff700 1 --2- 172.21.15.27:0/1873766834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f012c0b0a40 0x7f012c15da00 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f012c06f450 tx=0x7f011c009250 comp rx=0 tx=0).stop 2024-01-11T09:43:50.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.237+0000 7f01343ff700 1 -- 172.21.15.27:0/1873766834 shutdown_connections 2024-01-11T09:43:50.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.237+0000 7f01343ff700 1 --2- 172.21.15.27:0/1873766834 >> v2:172.21.15.77:3300/0 conn(0x7f012c06bf10 0x7f012c159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:50.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.237+0000 7f01343ff700 1 --2- 172.21.15.27:0/1873766834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f012c0b0a40 0x7f012c15da00 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:50.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.237+0000 7f01343ff700 1 -- 172.21.15.27:0/1873766834 >> 172.21.15.27:0/1873766834 conn(0x7f012c0ce3c0 msgr2=0x7f012c0ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:50.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.237+0000 7f01343ff700 1 -- 172.21.15.27:0/1873766834 shutdown_connections 2024-01-11T09:43:50.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.237+0000 7f01343ff700 1 -- 172.21.15.27:0/1873766834 wait complete. 2024-01-11T09:43:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.238+0000 7f01343ff700 1 Processor -- start 2024-01-11T09:43:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.238+0000 7f01343ff700 1 -- start start 2024-01-11T09:43:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.239+0000 7f01343ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f012c06bf10 0x7f012c152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.239+0000 7f01343ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f012c0b0a40 0x7f012c0c6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.239+0000 7f01343ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f012c15df40 0x7f012c0c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.239+0000 7f01343ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012c0c3a60 con 0x7f012c0b0a40 2024-01-11T09:43:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.239+0000 7f01343ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012c0c53d0 con 0x7f012c15df40 2024-01-11T09:43:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.239+0000 7f01343ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012c0c5f50 con 0x7f012c06bf10 2024-01-11T09:43:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.240+0000 7f013199a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f012c0b0a40 0x7f012c0c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.240+0000 7f013199a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f012c0b0a40 0x7f012c0c6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57538/0 (socket says 172.21.15.27:57538) 2024-01-11T09:43:50.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.240+0000 7f013199a700 1 -- 172.21.15.27:0/3794354757 learned_addr learned my addr 172.21.15.27:0/3794354757 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:50.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.240+0000 7f013219b700 1 --2- 172.21.15.27:0/3794354757 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f012c06bf10 0x7f012c152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:50.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.240+0000 7f013299c700 1 --2- 172.21.15.27:0/3794354757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f012c15df40 0x7f012c0c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:50.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.240+0000 7f013199a700 1 -- 172.21.15.27:0/3794354757 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f012c06bf10 msgr2=0x7f012c152cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:50.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.240+0000 7f013199a700 1 --2- 172.21.15.27:0/3794354757 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f012c06bf10 0x7f012c152cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:50.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.240+0000 7f013199a700 1 -- 172.21.15.27:0/3794354757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f012c15df40 msgr2=0x7f012c0c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:50.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.240+0000 7f013199a700 1 --2- 172.21.15.27:0/3794354757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f012c15df40 0x7f012c0c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:50.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.240+0000 7f013199a700 1 -- 172.21.15.27:0/3794354757 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011c0094f0 con 0x7f012c0b0a40 2024-01-11T09:43:50.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.241+0000 7f013199a700 1 --2- 172.21.15.27:0/3794354757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f012c0b0a40 0x7f012c0c6560 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f012c06b3c0 tx=0x7f011c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:50.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.241+0000 7f01227fc700 1 -- 172.21.15.27:0/3794354757 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c011420 con 0x7f012c0b0a40 2024-01-11T09:43:50.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.241+0000 7f01227fc700 1 -- 172.21.15.27:0/3794354757 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011c011580 con 0x7f012c0b0a40 2024-01-11T09:43:50.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.241+0000 7f01227fc700 1 -- 172.21.15.27:0/3794354757 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c011940 con 0x7f012c0b0a40 2024-01-11T09:43:50.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.241+0000 7f01343ff700 1 -- 172.21.15.27:0/3794354757 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f012c0c5620 con 0x7f012c0b0a40 2024-01-11T09:43:50.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.241+0000 7f01343ff700 1 -- 172.21.15.27:0/3794354757 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f012c0c5be0 con 0x7f012c0b0a40 2024-01-11T09:43:50.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.242+0000 7f01343ff700 1 -- 172.21.15.27:0/3794354757 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f012c0c60c0 con 0x7f012c0b0a40 2024-01-11T09:43:50.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.243+0000 7f01227fc700 1 -- 172.21.15.27:0/3794354757 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f011c011d30 con 0x7f012c0b0a40 2024-01-11T09:43:50.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.244+0000 7f01227fc700 1 --2- 172.21.15.27:0/3794354757 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0110066f30 0x7f01100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:50.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.244+0000 7f01227fc700 1 -- 172.21.15.27:0/3794354757 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f011c085b60 con 0x7f012c0b0a40 2024-01-11T09:43:50.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.245+0000 7f013219b700 1 --2- 172.21.15.27:0/3794354757 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0110066f30 0x7f01100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:50.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.245+0000 7f013219b700 1 --2- 172.21.15.27:0/3794354757 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0110066f30 0x7f01100693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0128009ca0 tx=0x7f0128008040 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:50.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.248+0000 7f01227fc700 1 -- 172.21.15.27:0/3794354757 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f011c052d80 con 0x7f012c0b0a40 2024-01-11T09:43:50.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.480+0000 7f01343ff700 1 -- 172.21.15.27:0/3794354757 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f012c04c3c0 con 0x7f0110066f30 2024-01-11T09:43:50.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.482+0000 7f01227fc700 1 -- 172.21.15.27:0/3794354757 <== mgr.8294 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f012c04c3c0 con 0x7f0110066f30 2024-01-11T09:43:50.473 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T09:43:50.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:43:50.474 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-11T09:43:50.474 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:43:50.474 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T09:43:50.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.484+0000 7f01343ff700 1 -- 172.21.15.27:0/3794354757 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0110066f30 msgr2=0x7f01100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:50.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.484+0000 7f01343ff700 1 --2- 172.21.15.27:0/3794354757 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0110066f30 0x7f01100693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0128009ca0 tx=0x7f0128008040 comp rx=0 tx=0).stop 2024-01-11T09:43:50.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.484+0000 7f01343ff700 1 -- 172.21.15.27:0/3794354757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f012c0b0a40 msgr2=0x7f012c0c6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:50.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.484+0000 7f01343ff700 1 --2- 172.21.15.27:0/3794354757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f012c0b0a40 0x7f012c0c6560 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f012c06b3c0 tx=0x7f011c017000 comp rx=0 tx=0).stop 2024-01-11T09:43:50.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.484+0000 7f01343ff700 1 -- 172.21.15.27:0/3794354757 shutdown_connections 2024-01-11T09:43:50.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.484+0000 7f01343ff700 1 --2- 172.21.15.27:0/3794354757 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0110066f30 0x7f01100693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:50.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.484+0000 7f01343ff700 1 --2- 172.21.15.27:0/3794354757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f012c15df40 0x7f012c0c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:50.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.484+0000 7f01343ff700 1 --2- 172.21.15.27:0/3794354757 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f012c06bf10 0x7f012c152cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:50.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.484+0000 7f01343ff700 1 --2- 172.21.15.27:0/3794354757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f012c0b0a40 0x7f012c0c6560 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:50.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.484+0000 7f01343ff700 1 -- 172.21.15.27:0/3794354757 >> 172.21.15.27:0/3794354757 conn(0x7f012c0ce3c0 msgr2=0x7f012c0cd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:50.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.484+0000 7f01343ff700 1 -- 172.21.15.27:0/3794354757 shutdown_connections 2024-01-11T09:43:50.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.484+0000 7f01343ff700 1 -- 172.21.15.27:0/3794354757 wait complete. 2024-01-11T09:43:50.494 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-11T09:43:50.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.703+0000 7fb5ad3ff700 1 Processor -- start 2024-01-11T09:43:50.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.703+0000 7fb5ad3ff700 1 -- start start 2024-01-11T09:43:50.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.704+0000 7fb5ad3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb5a8142dc0 0x7fb5a80cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:50.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.704+0000 7fb5ad3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5a8143190 0x7fb5a80cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:50.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.704+0000 7fb5ad3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a815d320 con 0x7fb5a80cc020 2024-01-11T09:43:50.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.704+0000 7fb5ad3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a815df30 con 0x7fb5a8143190 2024-01-11T09:43:50.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.704+0000 7fb5ad3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a815d590 con 0x7fb5a8142dc0 2024-01-11T09:43:50.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.705+0000 7fb5a67fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5a8143190 0x7fb5a80cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:50.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.705+0000 7fb5a77fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb5a80cc020 0x7fb5a80cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:32932/0 (socket says 172.21.15.27:32932) 2024-01-11T09:43:50.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.705+0000 7fb5a77fe700 1 -- 172.21.15.27:0/78068212 learned_addr learned my addr 172.21.15.27:0/78068212 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:50.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.705+0000 7fb5a6ffd700 1 --2- 172.21.15.27:0/78068212 >> v2:172.21.15.77:3300/0 conn(0x7fb5a8142dc0 0x7fb5a80cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:50.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.705+0000 7fb5a67fc700 1 -- 172.21.15.27:0/78068212 >> v2:172.21.15.77:3300/0 conn(0x7fb5a8142dc0 msgr2=0x7fb5a80cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:50.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.705+0000 7fb5a67fc700 1 --2- 172.21.15.27:0/78068212 >> v2:172.21.15.77:3300/0 conn(0x7fb5a8142dc0 0x7fb5a80cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:50.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.705+0000 7fb5a67fc700 1 -- 172.21.15.27:0/78068212 >> v1:172.21.15.27:6789/0 conn(0x7fb5a80cc020 legacy=0x7fb5a80cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:50.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.705+0000 7fb5a67fc700 1 -- 172.21.15.27:0/78068212 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a815d870 con 0x7fb5a8143190 2024-01-11T09:43:50.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.705+0000 7fb5a67fc700 1 --2- 172.21.15.27:0/78068212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5a8143190 0x7fb5a80cb2e0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fb5a806f4d0 tx=0x7fb59c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93565d5548bc1de4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.706+0000 7fb5a5ffb700 1 -- 172.21.15.27:0/78068212 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c00b6b0 con 0x7fb5a8143190 2024-01-11T09:43:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.706+0000 7fb5a5ffb700 1 -- 172.21.15.27:0/78068212 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c00b810 con 0x7fb5a8143190 2024-01-11T09:43:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.706+0000 7fb5a5ffb700 1 -- 172.21.15.27:0/78068212 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c00bbb0 con 0x7fb5a8143190 2024-01-11T09:43:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.706+0000 7fb5ad3ff700 1 -- 172.21.15.27:0/78068212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5a8143190 msgr2=0x7fb5a80cb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.706+0000 7fb5ad3ff700 1 --2- 172.21.15.27:0/78068212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5a8143190 0x7fb5a80cb2e0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fb5a806f4d0 tx=0x7fb59c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.706+0000 7fb5ad3ff700 1 -- 172.21.15.27:0/78068212 shutdown_connections 2024-01-11T09:43:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.706+0000 7fb5ad3ff700 1 --2- 172.21.15.27:0/78068212 >> v2:172.21.15.77:3300/0 conn(0x7fb5a8142dc0 0x7fb5a80cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.706+0000 7fb5ad3ff700 1 --2- 172.21.15.27:0/78068212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5a8143190 0x7fb5a80cb2e0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.706+0000 7fb5ad3ff700 1 -- 172.21.15.27:0/78068212 >> 172.21.15.27:0/78068212 conn(0x7fb5a8063ca0 msgr2=0x7fb5a806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.706+0000 7fb5ad3ff700 1 -- 172.21.15.27:0/78068212 shutdown_connections 2024-01-11T09:43:50.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.706+0000 7fb5ad3ff700 1 -- 172.21.15.27:0/78068212 wait complete. 2024-01-11T09:43:50.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.708+0000 7fb5ad3ff700 1 Processor -- start 2024-01-11T09:43:50.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.708+0000 7fb5ad3ff700 1 -- start start 2024-01-11T09:43:50.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.709+0000 7fb5ad3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5a80cc020 0x7fb5a816d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:50.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.709+0000 7fb5ad3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5a8142dc0 0x7fb5a816dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:50.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.709+0000 7fb5ad3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a8143190 0x7fb5a816e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:50.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.709+0000 7fb5ad3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a8154d60 con 0x7fb5a80cc020 2024-01-11T09:43:50.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.709+0000 7fb5ad3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a8131560 con 0x7fb5a8143190 2024-01-11T09:43:50.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.709+0000 7fb5ad3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a8154150 con 0x7fb5a8142dc0 2024-01-11T09:43:50.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.709+0000 7fb5a6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5a80cc020 0x7fb5a816d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:50.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.709+0000 7fb5a6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5a80cc020 0x7fb5a816d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57552/0 (socket says 172.21.15.27:57552) 2024-01-11T09:43:50.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.709+0000 7fb5a6ffd700 1 -- 172.21.15.27:0/3561775522 learned_addr learned my addr 172.21.15.27:0/3561775522 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:50.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.709+0000 7fb5a6ffd700 1 -- 172.21.15.27:0/3561775522 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5a8142dc0 msgr2=0x7fb5a816dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:50.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.709+0000 7fb5a6ffd700 1 --2- 172.21.15.27:0/3561775522 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5a8142dc0 0x7fb5a816dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:50.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.710+0000 7fb5a6ffd700 1 -- 172.21.15.27:0/3561775522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a8143190 msgr2=0x7fb5a816e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:50.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.710+0000 7fb5a6ffd700 1 --2- 172.21.15.27:0/3561775522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a8143190 0x7fb5a816e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:50.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.710+0000 7fb5a6ffd700 1 -- 172.21.15.27:0/3561775522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c009420 con 0x7fb5a80cc020 2024-01-11T09:43:50.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.710+0000 7fb5a6ffd700 1 --2- 172.21.15.27:0/3561775522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5a80cc020 0x7fb5a816d7f0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fb59800d890 tx=0x7fb598008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:50.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.710+0000 7fb58effd700 1 -- 172.21.15.27:0/3561775522 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59800b600 con 0x7fb5a80cc020 2024-01-11T09:43:50.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.710+0000 7fb58effd700 1 -- 172.21.15.27:0/3561775522 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59800b760 con 0x7fb5a80cc020 2024-01-11T09:43:50.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.710+0000 7fb58effd700 1 -- 172.21.15.27:0/3561775522 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59800bae0 con 0x7fb5a80cc020 2024-01-11T09:43:50.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.710+0000 7fb5ad3ff700 1 -- 172.21.15.27:0/3561775522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a8154430 con 0x7fb5a80cc020 2024-01-11T09:43:50.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.710+0000 7fb5ad3ff700 1 -- 172.21.15.27:0/3561775522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a81549f0 con 0x7fb5a80cc020 2024-01-11T09:43:50.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.711+0000 7fb5ad3ff700 1 -- 172.21.15.27:0/3561775522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a804c3c0 con 0x7fb5a80cc020 2024-01-11T09:43:50.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.712+0000 7fb58effd700 1 -- 172.21.15.27:0/3561775522 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 117) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb59801e460 con 0x7fb5a80cc020 2024-01-11T09:43:50.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.713+0000 7fb58effd700 1 --2- 172.21.15.27:0/3561775522 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb594066f00 0x7fb5940693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:50.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.713+0000 7fb58effd700 1 -- 172.21.15.27:0/3561775522 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5789+0+0 (secure 0 0 0) 0x7fb598084cd0 con 0x7fb5a80cc020 2024-01-11T09:43:50.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.714+0000 7fb5a67fc700 1 --2- 172.21.15.27:0/3561775522 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb594066f00 0x7fb5940693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:50.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.714+0000 7fb5a67fc700 1 --2- 172.21.15.27:0/3561775522 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb594066f00 0x7fb5940693b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8137b60 tx=0x7fb59c017000 comp rx=0 tx=0).ready entity=mgr.8294 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:50.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.717+0000 7fb58effd700 1 -- 172.21.15.27:0/3561775522 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb598051e50 con 0x7fb5a80cc020 2024-01-11T09:43:50.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:50.972+0000 7fb5ad3ff700 1 -- 172.21.15.27:0/3561775522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fb5a815db50 con 0x7fb5a80cc020 2024-01-11T09:43:51.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:51.896+0000 7fb58effd700 1 -- 172.21.15.27:0/3561775522 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 118) v1 ==== 84759+0+0 (secure 0 0 0) 0x7fb598052220 con 0x7fb5a80cc020 2024-01-11T09:43:51.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:51.899+0000 7fb58effd700 1 -- 172.21.15.27:0/3561775522 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v118) v1 ==== 92+0+0 (secure 0 0 0) 0x7fb598051e50 con 0x7fb5a80cc020 2024-01-11T09:43:51.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:51.902+0000 7fb5ad3ff700 1 -- 172.21.15.27:0/3561775522 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb594066f00 msgr2=0x7fb5940693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:51.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:51.902+0000 7fb5ad3ff700 1 --2- 172.21.15.27:0/3561775522 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb594066f00 0x7fb5940693b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8137b60 tx=0x7fb59c017000 comp rx=0 tx=0).stop 2024-01-11T09:43:51.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:51.902+0000 7fb5ad3ff700 1 -- 172.21.15.27:0/3561775522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5a80cc020 msgr2=0x7fb5a816d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:51.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:51.902+0000 7fb5ad3ff700 1 --2- 172.21.15.27:0/3561775522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5a80cc020 0x7fb5a816d7f0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fb59800d890 tx=0x7fb598008040 comp rx=0 tx=0).stop 2024-01-11T09:43:51.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:51.902+0000 7fb5ad3ff700 1 -- 172.21.15.27:0/3561775522 shutdown_connections 2024-01-11T09:43:51.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:51.902+0000 7fb5ad3ff700 1 --2- 172.21.15.27:0/3561775522 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb594066f00 0x7fb5940693b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:51.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:51.902+0000 7fb5ad3ff700 1 --2- 172.21.15.27:0/3561775522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5a8143190 0x7fb5a816e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:51.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:51.902+0000 7fb5ad3ff700 1 --2- 172.21.15.27:0/3561775522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5a80cc020 0x7fb5a816d7f0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:51.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:51.902+0000 7fb5ad3ff700 1 --2- 172.21.15.27:0/3561775522 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5a8142dc0 0x7fb5a816dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:51.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:51.902+0000 7fb5ad3ff700 1 -- 172.21.15.27:0/3561775522 >> 172.21.15.27:0/3561775522 conn(0x7fb5a8063ca0 msgr2=0x7fb5a806cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:51.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:51.902+0000 7fb5ad3ff700 1 -- 172.21.15.27:0/3561775522 shutdown_connections 2024-01-11T09:43:51.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:51.902+0000 7fb5ad3ff700 1 -- 172.21.15.27:0/3561775522 wait complete. 2024-01-11T09:43:51.915 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-11T09:43:52.104 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:52.111+0000 7fcca791c1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:43:52.120 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:52.128+0000 7fdbd3d1c1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:43:52.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.142+0000 7f0244dff700 1 Processor -- start 2024-01-11T09:43:52.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.142+0000 7f0244dff700 1 -- start start 2024-01-11T09:43:52.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.143+0000 7f0244dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f02400c8720 0x7f02401591a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:52.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.143+0000 7f0244dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0240069b60 0x7f02401596e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:52.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.143+0000 7f0244dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02400cd0a0 con 0x7f0240069790 2024-01-11T09:43:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.143+0000 7f0244dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02400c7680 con 0x7f02400c8720 2024-01-11T09:43:52.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.143+0000 7f0244dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02400c7960 con 0x7f0240069b60 2024-01-11T09:43:52.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.144+0000 7f023dd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f02400c8720 0x7f02401591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:52.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.144+0000 7f023e59c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0240069790 0x7f02400c69a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:53556/0 (socket says 172.21.15.27:53556) 2024-01-11T09:43:52.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.144+0000 7f023e59c700 1 -- 172.21.15.27:0/925958756 learned_addr learned my addr 172.21.15.27:0/925958756 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:52.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.144+0000 7f023ed9d700 1 --2- 172.21.15.27:0/925958756 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0240069b60 0x7f02401596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:52.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.144+0000 7f023d59a700 1 -- 172.21.15.27:0/925958756 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265703216 0 0) 0x7f02400cd0a0 con 0x7f0240069790 2024-01-11T09:43:52.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.144+0000 7f023d59a700 1 -- 172.21.15.27:0/925958756 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0228003570 con 0x7f0240069790 2024-01-11T09:43:52.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.144+0000 7f023ed9d700 1 -- 172.21.15.27:0/925958756 >> v2:172.21.15.27:3300/0 conn(0x7f02400c8720 msgr2=0x7f02401591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:52.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.144+0000 7f023ed9d700 1 --2- 172.21.15.27:0/925958756 >> v2:172.21.15.27:3300/0 conn(0x7f02400c8720 0x7f02401591a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.144+0000 7f023ed9d700 1 -- 172.21.15.27:0/925958756 >> v1:172.21.15.77:6789/0 conn(0x7f0240069790 legacy=0x7f02400c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:52.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.144+0000 7f023ed9d700 1 -- 172.21.15.27:0/925958756 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02400c7c40 con 0x7f0240069b60 2024-01-11T09:43:52.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.144+0000 7f023dd9b700 1 --2- 172.21.15.27:0/925958756 >> v2:172.21.15.27:3300/0 conn(0x7f02400c8720 0x7f02401591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:43:52.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.145+0000 7f023ed9d700 1 --2- 172.21.15.27:0/925958756 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0240069b60 0x7f02401596e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0238009e60 tx=0x7f0238009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4da28f26203f2937 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:52.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.145+0000 7f023d59a700 1 -- 172.21.15.27:0/925958756 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023800e070 con 0x7f0240069b60 2024-01-11T09:43:52.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.145+0000 7f023d59a700 1 -- 172.21.15.27:0/925958756 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023800bb40 con 0x7f0240069b60 2024-01-11T09:43:52.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.145+0000 7f023d59a700 1 -- 172.21.15.27:0/925958756 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0238012430 con 0x7f0240069b60 2024-01-11T09:43:52.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.145+0000 7f0244dff700 1 -- 172.21.15.27:0/925958756 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0240069b60 msgr2=0x7f02401596e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:52.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.145+0000 7f0244dff700 1 --2- 172.21.15.27:0/925958756 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0240069b60 0x7f02401596e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0238009e60 tx=0x7f0238009780 comp rx=0 tx=0).stop 2024-01-11T09:43:52.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.145+0000 7f0244dff700 1 -- 172.21.15.27:0/925958756 shutdown_connections 2024-01-11T09:43:52.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.145+0000 7f0244dff700 1 --2- 172.21.15.27:0/925958756 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0240069b60 0x7f02401596e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.145+0000 7f0244dff700 1 --2- 172.21.15.27:0/925958756 >> v2:172.21.15.27:3300/0 conn(0x7f02400c8720 0x7f02401591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.145+0000 7f0244dff700 1 -- 172.21.15.27:0/925958756 >> 172.21.15.27:0/925958756 conn(0x7f02400680a0 msgr2=0x7f02400cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:52.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.146+0000 7f0244dff700 1 -- 172.21.15.27:0/925958756 shutdown_connections 2024-01-11T09:43:52.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.146+0000 7f0244dff700 1 -- 172.21.15.27:0/925958756 wait complete. 2024-01-11T09:43:52.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.147+0000 7f0244dff700 1 Processor -- start 2024-01-11T09:43:52.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.147+0000 7f0244dff700 1 -- start start 2024-01-11T09:43:52.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.148+0000 7f0244dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0240069790 0x7f0240151390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:52.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.148+0000 7f0244dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0240069b60 0x7f02400aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:52.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.148+0000 7f0244dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02400c8720 0x7f02400b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:52.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.148+0000 7f0244dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02400b03c0 con 0x7f0240069b60 2024-01-11T09:43:52.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.148+0000 7f0244dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02400af7b0 con 0x7f0240069790 2024-01-11T09:43:52.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.148+0000 7f0244dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02400afa90 con 0x7f02400c8720 2024-01-11T09:43:52.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.148+0000 7f023dd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0240069b60 0x7f02400aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:52.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.148+0000 7f023e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0240069790 0x7f0240151390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:52.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.148+0000 7f023dd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0240069b60 0x7f02400aef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57588/0 (socket says 172.21.15.27:57588) 2024-01-11T09:43:52.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.148+0000 7f023dd9b700 1 -- 172.21.15.27:0/738354210 learned_addr learned my addr 172.21.15.27:0/738354210 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:52.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.149+0000 7f023ed9d700 1 --2- 172.21.15.27:0/738354210 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02400c8720 0x7f02400b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:52.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.149+0000 7f023dd9b700 1 -- 172.21.15.27:0/738354210 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02400c8720 msgr2=0x7f02400b0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:52.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.149+0000 7f023dd9b700 1 --2- 172.21.15.27:0/738354210 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02400c8720 0x7f02400b0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.149+0000 7f023dd9b700 1 -- 172.21.15.27:0/738354210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0240069790 msgr2=0x7f0240151390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:52.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.149+0000 7f023dd9b700 1 --2- 172.21.15.27:0/738354210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0240069790 0x7f0240151390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.149+0000 7f023dd9b700 1 -- 172.21.15.27:0/738354210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0234009420 con 0x7f0240069b60 2024-01-11T09:43:52.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.149+0000 7f023dd9b700 1 --2- 172.21.15.27:0/738354210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0240069b60 0x7f02400aef60 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f023400ec80 tx=0x7f02340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:52.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.149+0000 7f022e7fc700 1 -- 172.21.15.27:0/738354210 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023400c850 con 0x7f0240069b60 2024-01-11T09:43:52.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.149+0000 7f022e7fc700 1 -- 172.21.15.27:0/738354210 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023400c9b0 con 0x7f0240069b60 2024-01-11T09:43:52.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.149+0000 7f022e7fc700 1 -- 172.21.15.27:0/738354210 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023400cd50 con 0x7f0240069b60 2024-01-11T09:43:52.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.150+0000 7f0244dff700 1 -- 172.21.15.27:0/738354210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0238009a20 con 0x7f0240069b60 2024-01-11T09:43:52.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.150+0000 7f0244dff700 1 -- 172.21.15.27:0/738354210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02401518d0 con 0x7f0240069b60 2024-01-11T09:43:52.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.151+0000 7f0244dff700 1 -- 172.21.15.27:0/738354210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02400b0530 con 0x7f0240069b60 2024-01-11T09:43:52.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.152+0000 7f022e7fc700 1 -- 172.21.15.27:0/738354210 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 118) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f0234013680 con 0x7f0240069b60 2024-01-11T09:43:52.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.153+0000 7f022e7fc700 1 --2- 172.21.15.27:0/738354210 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0220066f80 0x7f0220069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:52.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.153+0000 7f022e7fc700 1 -- 172.21.15.27:0/738354210 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f0234084580 con 0x7f0240069b60 2024-01-11T09:43:52.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.153+0000 7f023e59c700 1 -- 172.21.15.27:0/738354210 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0220066f80 msgr2=0x7f0220069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:52.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.153+0000 7f023e59c700 1 --2- 172.21.15.27:0/738354210 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0220066f80 0x7f0220069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:43:52.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.157+0000 7f022e7fc700 1 -- 172.21.15.27:0/738354210 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f02340517a0 con 0x7f0240069b60 2024-01-11T09:43:52.284 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:52.291+0000 7fcca791c1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:43:52.325 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:52.334+0000 7fdbd3d1c1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:43:52.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.354+0000 7f023e59c700 1 -- 172.21.15.27:0/738354210 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0220066f80 msgr2=0x7f0220069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:52.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.354+0000 7f023e59c700 1 --2- 172.21.15.27:0/738354210 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0220066f80 0x7f0220069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:43:52.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.404+0000 7f0244dff700 1 -- 172.21.15.27:0/738354210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f02400c7f20 con 0x7f0240069b60 2024-01-11T09:43:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.408+0000 7f022e7fc700 1 -- 172.21.15.27:0/738354210 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v71)=0 v71) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0234054dc0 con 0x7f0240069b60 2024-01-11T09:43:52.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.411+0000 7f0244dff700 1 -- 172.21.15.27:0/738354210 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0220066f80 msgr2=0x7f0220069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:43:52.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.411+0000 7f0244dff700 1 --2- 172.21.15.27:0/738354210 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0220066f80 0x7f0220069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.411+0000 7f0244dff700 1 -- 172.21.15.27:0/738354210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0240069b60 msgr2=0x7f02400aef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:52.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.411+0000 7f0244dff700 1 --2- 172.21.15.27:0/738354210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0240069b60 0x7f02400aef60 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f023400ec80 tx=0x7f02340091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.411+0000 7f0244dff700 1 -- 172.21.15.27:0/738354210 shutdown_connections 2024-01-11T09:43:52.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.411+0000 7f0244dff700 1 --2- 172.21.15.27:0/738354210 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f02400c8720 0x7f02400b0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.411+0000 7f0244dff700 1 --2- 172.21.15.27:0/738354210 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0220066f80 0x7f0220069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.411+0000 7f0244dff700 1 --2- 172.21.15.27:0/738354210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0240069790 0x7f0240151390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.411+0000 7f0244dff700 1 --2- 172.21.15.27:0/738354210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0240069b60 0x7f02400aef60 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.411+0000 7f0244dff700 1 -- 172.21.15.27:0/738354210 >> 172.21.15.27:0/738354210 conn(0x7f02400680a0 msgr2=0x7f024006d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:52.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.411+0000 7f0244dff700 1 -- 172.21.15.27:0/738354210 shutdown_connections 2024-01-11T09:43:52.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.411+0000 7f0244dff700 1 -- 172.21.15.27:0/738354210 wait complete. 2024-01-11T09:43:52.421 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-11T09:43:52.598 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:52.605+0000 7fcca791c1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:43:52.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.640+0000 7f8ce85ff700 1 Processor -- start 2024-01-11T09:43:52.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.640+0000 7f8ce85ff700 1 -- start start 2024-01-11T09:43:52.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.642+0000 7f8ce85ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8ce0143190 0x7f8ce00cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:52.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.642+0000 7f8ce85ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce00ccc70 con 0x7f8ce00cb4b0 2024-01-11T09:43:52.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.642+0000 7f8ce85ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce015d320 con 0x7f8ce0142dc0 2024-01-11T09:43:52.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.642+0000 7f8ce85ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ce015df30 con 0x7f8ce0143190 2024-01-11T09:43:52.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.642+0000 7f8ce5b9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8ce0143190 0x7f8ce00cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:52.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.642+0000 7f8ce6b9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8ce00cb4b0 0x7f8ce00cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:32936/0 (socket says 172.21.15.27:32936) 2024-01-11T09:43:52.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.642+0000 7f8ce6b9c700 1 -- 172.21.15.27:0/2472112522 learned_addr learned my addr 172.21.15.27:0/2472112522 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:52.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.643+0000 7f8ce5b9a700 1 -- 172.21.15.27:0/2472112522 >> v1:172.21.15.77:6789/0 conn(0x7f8ce0142dc0 legacy=0x7f8ce00cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:52.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.643+0000 7f8ce5399700 1 -- 172.21.15.27:0/2472112522 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826006184 0 0) 0x7f8ce015d320 con 0x7f8ce0142dc0 2024-01-11T09:43:52.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.643+0000 7f8ce5b9a700 1 -- 172.21.15.27:0/2472112522 >> v1:172.21.15.27:6789/0 conn(0x7f8ce00cb4b0 legacy=0x7f8ce00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:52.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.643+0000 7f8ce5b9a700 1 -- 172.21.15.27:0/2472112522 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce015d590 con 0x7f8ce0143190 2024-01-11T09:43:52.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.643+0000 7f8ce5b9a700 1 --2- 172.21.15.27:0/2472112522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ce0143190 0x7f8ce00cc020 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f8cd00098e0 tx=0x7f8cd00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c700725d40aa027f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:52.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.643+0000 7f8ce5399700 1 -- 172.21.15.27:0/2472112522 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0009e80 con 0x7f8ce0143190 2024-01-11T09:43:52.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.643+0000 7f8ce5399700 1 -- 172.21.15.27:0/2472112522 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd000b860 con 0x7f8ce0143190 2024-01-11T09:43:52.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.643+0000 7f8ce5399700 1 -- 172.21.15.27:0/2472112522 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd000bbe0 con 0x7f8ce0143190 2024-01-11T09:43:52.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.644+0000 7f8ce85ff700 1 -- 172.21.15.27:0/2472112522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ce0143190 msgr2=0x7f8ce00cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.644+0000 7f8ce85ff700 1 --2- 172.21.15.27:0/2472112522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ce0143190 0x7f8ce00cc020 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f8cd00098e0 tx=0x7f8cd00091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.644+0000 7f8ce85ff700 1 -- 172.21.15.27:0/2472112522 shutdown_connections 2024-01-11T09:43:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.644+0000 7f8ce85ff700 1 --2- 172.21.15.27:0/2472112522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ce0143190 0x7f8ce00cc020 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.644+0000 7f8ce85ff700 1 -- 172.21.15.27:0/2472112522 >> 172.21.15.27:0/2472112522 conn(0x7f8ce0063ca0 msgr2=0x7f8ce006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.644+0000 7f8ce85ff700 1 -- 172.21.15.27:0/2472112522 shutdown_connections 2024-01-11T09:43:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.644+0000 7f8ce85ff700 1 -- 172.21.15.27:0/2472112522 wait complete. 2024-01-11T09:43:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.646+0000 7f8ce85ff700 1 Processor -- start 2024-01-11T09:43:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.646+0000 7f8ce85ff700 1 -- start start 2024-01-11T09:43:52.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.647+0000 7f8ce85ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ce00cb4b0 0x7f8ce0158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:52.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.647+0000 7f8ce85ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ce0142dc0 0x7f8ce00c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:52.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.647+0000 7f8ce85ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ce0143190 0x7f8ce01558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:52.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.647+0000 7f8ce85ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce0159220 con 0x7f8ce0142dc0 2024-01-11T09:43:52.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.647+0000 7f8ce85ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce0159ce0 con 0x7f8ce0143190 2024-01-11T09:43:52.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.647+0000 7f8ce85ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce0169be0 con 0x7f8ce00cb4b0 2024-01-11T09:43:52.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.647+0000 7f8ce5b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ce0142dc0 0x7f8ce00c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:52.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.647+0000 7f8ce5b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ce0142dc0 0x7f8ce00c9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57610/0 (socket says 172.21.15.27:57610) 2024-01-11T09:43:52.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.647+0000 7f8ce5b9a700 1 -- 172.21.15.27:0/2907921224 learned_addr learned my addr 172.21.15.27:0/2907921224 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:52.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.647+0000 7f8ce6b9c700 1 --2- 172.21.15.27:0/2907921224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ce0143190 0x7f8ce01558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:52.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.647+0000 7f8ce639b700 1 --2- 172.21.15.27:0/2907921224 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ce00cb4b0 0x7f8ce0158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:52.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.648+0000 7f8ce5b9a700 1 -- 172.21.15.27:0/2907921224 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ce00cb4b0 msgr2=0x7f8ce0158210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:52.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.648+0000 7f8ce5b9a700 1 --2- 172.21.15.27:0/2907921224 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ce00cb4b0 0x7f8ce0158210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.648+0000 7f8ce5b9a700 1 -- 172.21.15.27:0/2907921224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ce0143190 msgr2=0x7f8ce01558a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:52.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.648+0000 7f8ce5b9a700 1 --2- 172.21.15.27:0/2907921224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ce0143190 0x7f8ce01558a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.648+0000 7f8ce5b9a700 1 -- 172.21.15.27:0/2907921224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd0009420 con 0x7f8ce0142dc0 2024-01-11T09:43:52.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.648+0000 7f8ce6b9c700 1 --2- 172.21.15.27:0/2907921224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ce0143190 0x7f8ce01558a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:43:52.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.648+0000 7f8ce5b9a700 1 --2- 172.21.15.27:0/2907921224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ce0142dc0 0x7f8ce00c9d10 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0064a70 tx=0x7f8cd0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:52.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.648+0000 7f8cd67fc700 1 -- 172.21.15.27:0/2907921224 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd001f070 con 0x7f8ce0142dc0 2024-01-11T09:43:52.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.648+0000 7f8cd67fc700 1 -- 172.21.15.27:0/2907921224 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd0000c90 con 0x7f8ce0142dc0 2024-01-11T09:43:52.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.648+0000 7f8cd67fc700 1 -- 172.21.15.27:0/2907921224 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0025420 con 0x7f8ce0142dc0 2024-01-11T09:43:52.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.648+0000 7f8ce85ff700 1 -- 172.21.15.27:0/2907921224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce0158750 con 0x7f8ce0142dc0 2024-01-11T09:43:52.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.649+0000 7f8ce85ff700 1 -- 172.21.15.27:0/2907921224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce0158980 con 0x7f8ce0142dc0 2024-01-11T09:43:52.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.650+0000 7f8ce85ff700 1 -- 172.21.15.27:0/2907921224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce016a030 con 0x7f8ce0142dc0 2024-01-11T09:43:52.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.651+0000 7f8cd67fc700 1 -- 172.21.15.27:0/2907921224 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 118) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f8cd0023070 con 0x7f8ce0142dc0 2024-01-11T09:43:52.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.652+0000 7f8cd67fc700 1 --2- 172.21.15.27:0/2907921224 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8cc4066fb0 0x7f8cc4069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:52.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.652+0000 7f8cd67fc700 1 -- 172.21.15.27:0/2907921224 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f8cd0085a70 con 0x7f8ce0142dc0 2024-01-11T09:43:52.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.652+0000 7f8ce639b700 1 -- 172.21.15.27:0/2907921224 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8cc4066fb0 msgr2=0x7f8cc4069460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:52.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.652+0000 7f8ce639b700 1 --2- 172.21.15.27:0/2907921224 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8cc4066fb0 0x7f8cc4069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:43:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.656+0000 7f8cd67fc700 1 -- 172.21.15.27:0/2907921224 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8ce016a030 con 0x7f8ce0142dc0 2024-01-11T09:43:52.676 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:52.684+0000 7fdbd3d1c1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:43:52.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.853+0000 7f8ce639b700 1 -- 172.21.15.27:0/2907921224 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8cc4066fb0 msgr2=0x7f8cc4069460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:52.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.853+0000 7f8ce639b700 1 --2- 172.21.15.27:0/2907921224 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8cc4066fb0 0x7f8cc4069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:43:52.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.859+0000 7f8ce85ff700 1 -- 172.21.15.27:0/2907921224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f8ce004c3c0 con 0x7f8ce0142dc0 2024-01-11T09:43:52.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.863+0000 7f8cd67fc700 1 -- 172.21.15.27:0/2907921224 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v72)=0 v72) v1 ==== 133+0+0 (secure 0 0 0) 0x7f8cd00150c0 con 0x7f8ce0142dc0 2024-01-11T09:43:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.866+0000 7f8ce85ff700 1 -- 172.21.15.27:0/2907921224 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8cc4066fb0 msgr2=0x7f8cc4069460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:43:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.866+0000 7f8ce85ff700 1 --2- 172.21.15.27:0/2907921224 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8cc4066fb0 0x7f8cc4069460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.866+0000 7f8ce85ff700 1 -- 172.21.15.27:0/2907921224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ce0142dc0 msgr2=0x7f8ce00c9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.866+0000 7f8ce85ff700 1 --2- 172.21.15.27:0/2907921224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ce0142dc0 0x7f8ce00c9d10 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0064a70 tx=0x7f8cd0017000 comp rx=0 tx=0).stop 2024-01-11T09:43:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.866+0000 7f8ce85ff700 1 -- 172.21.15.27:0/2907921224 shutdown_connections 2024-01-11T09:43:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.866+0000 7f8ce85ff700 1 --2- 172.21.15.27:0/2907921224 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8cc4066fb0 0x7f8cc4069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.866+0000 7f8ce85ff700 1 --2- 172.21.15.27:0/2907921224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ce0143190 0x7f8ce01558a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.866+0000 7f8ce85ff700 1 --2- 172.21.15.27:0/2907921224 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ce00cb4b0 0x7f8ce0158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.866+0000 7f8ce85ff700 1 --2- 172.21.15.27:0/2907921224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ce0142dc0 0x7f8ce00c9d10 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.866+0000 7f8ce85ff700 1 -- 172.21.15.27:0/2907921224 >> 172.21.15.27:0/2907921224 conn(0x7f8ce0063ca0 msgr2=0x7f8ce006a030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:52.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.866+0000 7f8ce85ff700 1 -- 172.21.15.27:0/2907921224 shutdown_connections 2024-01-11T09:43:52.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:52.866+0000 7f8ce85ff700 1 -- 172.21.15.27:0/2907921224 wait complete. 2024-01-11T09:43:52.878 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:43:52.878 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-01-11T09:43:52.937 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-01-11T09:43:53.126 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:53.133+0000 7fcca791c1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:43:53.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.159+0000 7f7e495ff700 1 Processor -- start 2024-01-11T09:43:53.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.159+0000 7f7e495ff700 1 -- start start 2024-01-11T09:43:53.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.161+0000 7f7e495ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7e441622d0 0x7f7e441646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:53.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.161+0000 7f7e495ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e440cf850 con 0x7f7e4415a1b0 2024-01-11T09:43:53.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.161+0000 7f7e495ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e440cec40 con 0x7f7e440ccd70 2024-01-11T09:43:53.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.161+0000 7f7e495ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e4415ac90 con 0x7f7e441622d0 2024-01-11T09:43:53.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.161+0000 7f7e42ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7e4415a1b0 0x7f7e4415a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:32950/0 (socket says 172.21.15.27:32950) 2024-01-11T09:43:53.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.161+0000 7f7e42ffd700 1 -- 172.21.15.27:0/3401468931 learned_addr learned my addr 172.21.15.27:0/3401468931 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:53.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.161+0000 7f7e437fe700 1 --2- 172.21.15.27:0/3401468931 >> v2:172.21.15.77:3300/0 conn(0x7f7e441622d0 0x7f7e441646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:53.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.161+0000 7f7e41ffb700 1 -- 172.21.15.27:0/3401468931 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2390717423 0 0) 0x7f7e440cf850 con 0x7f7e4415a1b0 2024-01-11T09:43:53.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.162+0000 7f7e41ffb700 1 -- 172.21.15.27:0/3401468931 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e2c003570 con 0x7f7e4415a1b0 2024-01-11T09:43:53.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.162+0000 7f7e437fe700 1 -- 172.21.15.27:0/3401468931 >> v1:172.21.15.77:6789/0 conn(0x7f7e440ccd70 legacy=0x7f7e440cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:53.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.162+0000 7f7e437fe700 1 -- 172.21.15.27:0/3401468931 >> v1:172.21.15.27:6789/0 conn(0x7f7e4415a1b0 legacy=0x7f7e4415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:53.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.162+0000 7f7e437fe700 1 -- 172.21.15.27:0/3401468931 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e440cef20 con 0x7f7e441622d0 2024-01-11T09:43:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.162+0000 7f7e437fe700 1 --2- 172.21.15.27:0/3401468931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e441622d0 0x7f7e441646b0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c009e60 tx=0x7f7e3c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a10bfe1fb08445fd server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.162+0000 7f7e41ffb700 1 -- 172.21.15.27:0/3401468931 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c00e070 con 0x7f7e441622d0 2024-01-11T09:43:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.162+0000 7f7e41ffb700 1 -- 172.21.15.27:0/3401468931 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3c00bb60 con 0x7f7e441622d0 2024-01-11T09:43:53.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.162+0000 7f7e41ffb700 1 -- 172.21.15.27:0/3401468931 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c012430 con 0x7f7e441622d0 2024-01-11T09:43:53.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.163+0000 7f7e495ff700 1 -- 172.21.15.27:0/3401468931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e441622d0 msgr2=0x7f7e441646b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:53.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.163+0000 7f7e495ff700 1 --2- 172.21.15.27:0/3401468931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e441622d0 0x7f7e441646b0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c009e60 tx=0x7f7e3c009780 comp rx=0 tx=0).stop 2024-01-11T09:43:53.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.163+0000 7f7e495ff700 1 -- 172.21.15.27:0/3401468931 shutdown_connections 2024-01-11T09:43:53.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.163+0000 7f7e495ff700 1 --2- 172.21.15.27:0/3401468931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e441622d0 0x7f7e441646b0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:53.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.163+0000 7f7e495ff700 1 -- 172.21.15.27:0/3401468931 >> 172.21.15.27:0/3401468931 conn(0x7f7e4406ed40 msgr2=0x7f7e4406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:53.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.163+0000 7f7e495ff700 1 -- 172.21.15.27:0/3401468931 shutdown_connections 2024-01-11T09:43:53.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.163+0000 7f7e495ff700 1 -- 172.21.15.27:0/3401468931 wait complete. 2024-01-11T09:43:53.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.165+0000 7f7e495ff700 1 Processor -- start 2024-01-11T09:43:53.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.165+0000 7f7e495ff700 1 -- start start 2024-01-11T09:43:53.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.166+0000 7f7e495ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e440ccd70 0x7f7e440b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:53.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.166+0000 7f7e495ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e4415a1b0 0x7f7e4414f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.166+0000 7f7e495ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e441622d0 0x7f7e4414c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.166+0000 7f7e495ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4414e280 con 0x7f7e440ccd70 2024-01-11T09:43:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.166+0000 7f7e495ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e440b1af0 con 0x7f7e4415a1b0 2024-01-11T09:43:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.166+0000 7f7e495ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e441519f0 con 0x7f7e441622d0 2024-01-11T09:43:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.166+0000 7f7e42ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e440ccd70 0x7f7e440b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.166+0000 7f7e42ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e440ccd70 0x7f7e440b1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57612/0 (socket says 172.21.15.27:57612) 2024-01-11T09:43:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.166+0000 7f7e42ffd700 1 -- 172.21.15.27:0/2250641165 learned_addr learned my addr 172.21.15.27:0/2250641165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.166+0000 7f7e427fc700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e4415a1b0 0x7f7e4414f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.166+0000 7f7e437fe700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e441622d0 0x7f7e4414c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.166+0000 7f7e42ffd700 1 -- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e441622d0 msgr2=0x7f7e4414c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.166+0000 7f7e42ffd700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e441622d0 0x7f7e4414c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.166+0000 7f7e42ffd700 1 -- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e4415a1b0 msgr2=0x7f7e4414f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:53.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.166+0000 7f7e42ffd700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e4415a1b0 0x7f7e4414f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:53.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.166+0000 7f7e42ffd700 1 -- 172.21.15.27:0/2250641165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c009a20 con 0x7f7e440ccd70 2024-01-11T09:43:53.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.167+0000 7f7e42ffd700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e440ccd70 0x7f7e440b1f80 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400c7c0 tx=0x7f7e34007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:53.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.167+0000 7f7e2affd700 1 -- 172.21.15.27:0/2250641165 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e34005cd0 con 0x7f7e440ccd70 2024-01-11T09:43:53.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.167+0000 7f7e2affd700 1 -- 172.21.15.27:0/2250641165 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e34005e30 con 0x7f7e440ccd70 2024-01-11T09:43:53.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.167+0000 7f7e495ff700 1 -- 172.21.15.27:0/2250641165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e44151710 con 0x7f7e440ccd70 2024-01-11T09:43:53.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.167+0000 7f7e2affd700 1 -- 172.21.15.27:0/2250641165 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e34007320 con 0x7f7e440ccd70 2024-01-11T09:43:53.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.167+0000 7f7e495ff700 1 -- 172.21.15.27:0/2250641165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e440b1780 con 0x7f7e440ccd70 2024-01-11T09:43:53.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.168+0000 7f7e495ff700 1 -- 172.21.15.27:0/2250641165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e4404c3c0 con 0x7f7e440ccd70 2024-01-11T09:43:53.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.169+0000 7f7e2affd700 1 -- 172.21.15.27:0/2250641165 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 118) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f7e3401b3b0 con 0x7f7e440ccd70 2024-01-11T09:43:53.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.170+0000 7f7e2affd700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e30066f80 0x7f7e30069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:53.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.170+0000 7f7e2affd700 1 -- 172.21.15.27:0/2250641165 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f7e34083d20 con 0x7f7e440ccd70 2024-01-11T09:43:53.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.170+0000 7f7e427fc700 1 -- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e30066f80 msgr2=0x7f7e30069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:53.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.170+0000 7f7e427fc700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e30066f80 0x7f7e30069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:43:53.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.174+0000 7f7e2affd700 1 -- 172.21.15.27:0/2250641165 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7e34050ea0 con 0x7f7e440ccd70 2024-01-11T09:43:53.265 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:53.273+0000 7fdbd3d1c1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:43:53.297 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:53.304+0000 7fcca791c1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:43:53.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.371+0000 7f7e427fc700 1 -- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e30066f80 msgr2=0x7f7e30069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:53.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.371+0000 7f7e427fc700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e30066f80 0x7f7e30069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:43:53.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.406+0000 7f7e495ff700 1 -- 172.21.15.27:0/2250641165 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e44154d50 con 0x7f7e30066f80 2024-01-11T09:43:53.447 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:53.455+0000 7fdbd3d1c1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:43:53.710 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:53.718+0000 7fcca791c1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:43:53.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.772+0000 7f7e427fc700 1 -- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e30066f80 msgr2=0x7f7e30069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:53.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:53.772+0000 7f7e427fc700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e30066f80 0x7f7e30069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-11T09:43:53.886 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:53.894+0000 7fdbd3d1c1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:43:54.485 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:54.492+0000 7fcca791c1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:43:54.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:54.573+0000 7f7e427fc700 1 -- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e30066f80 msgr2=0x7f7e30069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:54.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:54.573+0000 7f7e427fc700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e30066f80 0x7f7e30069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-11T09:43:54.573 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:54.581+0000 7fcca791c1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:43:54.713 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:54.722+0000 7fdbd3d1c1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:43:54.727 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:54.735+0000 7fcca791c1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:43:54.807 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:54.815+0000 7fdbd3d1c1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:43:54.816 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:54.823+0000 7fcca791c1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:43:54.973 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:54.981+0000 7fdbd3d1c1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:43:55.032 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:55.039+0000 7fcca791c1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:43:55.067 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:55.075+0000 7fdbd3d1c1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:43:55.127 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:55.135+0000 7fcca791c1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:43:55.222 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:55.229+0000 7fcca791c1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:43:55.294 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:55.302+0000 7fdbd3d1c1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:43:55.317 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:55.324+0000 7fcca791c1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:43:55.396 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:55.405+0000 7fdbd3d1c1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:43:55.498 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:55.506+0000 7fdbd3d1c1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:43:55.507 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:55.515+0000 7fcca791c1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:43:55.599 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:55.607+0000 7fdbd3d1c1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:43:55.605 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:55.612+0000 7fcca791c1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:43:55.800 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:55.808+0000 7fdbd3d1c1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:43:55.904 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:55.913+0000 7fdbd3d1c1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:43:55.948 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:55.956+0000 7fcca791c1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:43:56.039 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:56.046+0000 7fcca791c1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:43:56.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:56.175+0000 7f7e427fc700 1 -- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e30066f80 msgr2=0x7f7e30069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:43:56.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:56.175+0000 7f7e427fc700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e30066f80 0x7f7e30069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-11T09:43:56.266 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:56.274+0000 7fdbd3d1c1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:43:56.361 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:56.369+0000 7fdbd3d1c1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:43:56.579 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:56.587+0000 7fcca791c1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:43:56.764 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:56.772+0000 7fcca791c1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:43:56.942 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:56.950+0000 7fdbd3d1c1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:43:57.142 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:57.150+0000 7fdbd3d1c1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:43:57.142 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:57.151+0000 7fcca791c1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:43:57.542 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:57.550+0000 7fdbd3d1c1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:43:57.560 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:57.569+0000 7fcca791c1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:43:57.742 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:57.751+0000 7fcca791c1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:43:57.930 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:57.938+0000 7fcca791c1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:43:57.993 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:58.002+0000 7fdbd3d1c1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:43:58.039 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:58.048+0000 7fcca791c1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:43:58.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:58.056+0000 7f7e2affd700 1 -- 172.21.15.27:0/2250641165 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 119) v1 ==== 84470+0+0 (secure 0 0 0) 0x7f7e34051270 con 0x7f7e440ccd70 2024-01-11T09:43:58.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:58.056+0000 7f7e2affd700 1 -- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e30066f80 msgr2=0x7f7e30069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:43:58.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:58.056+0000 7f7e2affd700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e30066f80 0x7f7e30069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:58.179 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:58.187+0000 7fcc1cf39700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:58.179 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:58.187+0000 7fcc1cf39700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:58.179 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:58.187+0000 7fcc1cf39700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:58.179 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:58.187+0000 7fcc1cf39700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:58.179 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:58.187+0000 7fcc1cf39700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:58.180 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:58.188+0000 7fcc1f73e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:58.180 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:58.188+0000 7fcc1f73e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:58.180 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:58.188+0000 7fcc1f73e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:58.180 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:58.188+0000 7fcc1f73e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:58.180 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:43:58.188+0000 7fcc1f73e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:43:58.206 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:58.214+0000 7fdbd3d1c1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:43:58.404 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:58.413+0000 7fdbd3d1c1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:43:58.521 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:43:58.529+0000 7fdbd3d1c1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:43:59.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.060+0000 7f7e2affd700 1 -- 172.21.15.27:0/2250641165 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 120) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f7e3405bd50 con 0x7f7e440ccd70 2024-01-11T09:43:59.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.060+0000 7f7e2affd700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e30066f80 0x7f7e300693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:59.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.060+0000 7f7e2affd700 1 -- 172.21.15.27:0/2250641165 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e44154d50 con 0x7f7e30066f80 2024-01-11T09:43:59.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.061+0000 7f7e427fc700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e30066f80 0x7f7e300693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:59.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.062+0000 7f7e427fc700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e30066f80 0x7f7e300693b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7e4414fb90 tx=0x7f7e38007ec0 comp rx=0 tx=0).ready entity=mgr.9895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:59.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.135+0000 7f7e2affd700 1 -- 172.21.15.27:0/2250641165 <== mgr.9895 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7e44154d50 con 0x7f7e30066f80 2024-01-11T09:43:59.128 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path snap_test_dir1 2024-01-11T09:43:59.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.138+0000 7f7e495ff700 1 -- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e30066f80 msgr2=0x7f7e300693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:59.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.138+0000 7f7e495ff700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e30066f80 0x7f7e300693b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7e4414fb90 tx=0x7f7e38007ec0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.138+0000 7f7e495ff700 1 -- 172.21.15.27:0/2250641165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e440ccd70 msgr2=0x7f7e440b1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:59.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.138+0000 7f7e495ff700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e440ccd70 0x7f7e440b1f80 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400c7c0 tx=0x7f7e34007e00 comp rx=0 tx=0).stop 2024-01-11T09:43:59.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.138+0000 7f7e495ff700 1 -- 172.21.15.27:0/2250641165 shutdown_connections 2024-01-11T09:43:59.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.138+0000 7f7e495ff700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e441622d0 0x7f7e4414c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.138+0000 7f7e495ff700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e440ccd70 0x7f7e440b1f80 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.138+0000 7f7e495ff700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e30066f80 0x7f7e300693b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.138+0000 7f7e495ff700 1 --2- 172.21.15.27:0/2250641165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e4415a1b0 0x7f7e4414f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.138+0000 7f7e495ff700 1 -- 172.21.15.27:0/2250641165 >> 172.21.15.27:0/2250641165 conn(0x7f7e4406ed40 msgr2=0x7f7e44154080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:59.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.138+0000 7f7e495ff700 1 -- 172.21.15.27:0/2250641165 shutdown_connections 2024-01-11T09:43:59.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.138+0000 7f7e495ff700 1 -- 172.21.15.27:0/2250641165 wait complete. 2024-01-11T09:43:59.149 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-01-11T09:43:59.149 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2024-01-11T09:43:59.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.362+0000 7f29d07ff700 1 Processor -- start 2024-01-11T09:43:59.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.362+0000 7f29d07ff700 1 -- start start 2024-01-11T09:43:59.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.363+0000 7f29d07ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f29c8156810 0x7f29c806e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:59.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.363+0000 7f29d07ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f29c80ccac0 0x7f29c806ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:59.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.363+0000 7f29d07ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c80c8590 con 0x7f29c8156440 2024-01-11T09:43:59.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.363+0000 7f29d07ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c80c7980 con 0x7f29c80ccac0 2024-01-11T09:43:59.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.363+0000 7f29d07ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c80c7c60 con 0x7f29c8156810 2024-01-11T09:43:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.364+0000 7f29cdd9a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f29c8156810 0x7f29c806e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.364+0000 7f29ced9c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f29c80ccac0 0x7f29c806ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.364+0000 7f29ce59b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f29c8156440 0x7f29c80c7120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:37404/0 (socket says 172.21.15.27:37404) 2024-01-11T09:43:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.364+0000 7f29ce59b700 1 -- 172.21.15.27:0/60791955 learned_addr learned my addr 172.21.15.27:0/60791955 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.364+0000 7f29cdd9a700 1 -- 172.21.15.27:0/60791955 >> v2:172.21.15.77:3300/0 conn(0x7f29c80ccac0 msgr2=0x7f29c806ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.364+0000 7f29cdd9a700 1 --2- 172.21.15.27:0/60791955 >> v2:172.21.15.77:3300/0 conn(0x7f29c80ccac0 0x7f29c806ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.364+0000 7f29cdd9a700 1 -- 172.21.15.27:0/60791955 >> v1:172.21.15.77:6789/0 conn(0x7f29c8156440 legacy=0x7f29c80c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.364+0000 7f29cdd9a700 1 -- 172.21.15.27:0/60791955 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c80c7f40 con 0x7f29c8156810 2024-01-11T09:43:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.364+0000 7f29cd599700 1 -- 172.21.15.27:0/60791955 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036035916 0 0) 0x7f29c80c8590 con 0x7f29c8156440 2024-01-11T09:43:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.365+0000 7f29cdd9a700 1 --2- 172.21.15.27:0/60791955 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f29c8156810 0x7f29c806e720 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f29b80098e0 tx=0x7f29b80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15794684c424c920 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.365+0000 7f29cd599700 1 -- 172.21.15.27:0/60791955 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b800b640 con 0x7f29c8156810 2024-01-11T09:43:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.365+0000 7f29cd599700 1 -- 172.21.15.27:0/60791955 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b800b7a0 con 0x7f29c8156810 2024-01-11T09:43:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.365+0000 7f29cd599700 1 -- 172.21.15.27:0/60791955 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b800bb40 con 0x7f29c8156810 2024-01-11T09:43:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.365+0000 7f29d07ff700 1 -- 172.21.15.27:0/60791955 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f29c8156810 msgr2=0x7f29c806e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:59.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.365+0000 7f29d07ff700 1 --2- 172.21.15.27:0/60791955 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f29c8156810 0x7f29c806e720 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f29b80098e0 tx=0x7f29b80091a0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.366+0000 7f29d07ff700 1 -- 172.21.15.27:0/60791955 shutdown_connections 2024-01-11T09:43:59.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.366+0000 7f29d07ff700 1 --2- 172.21.15.27:0/60791955 >> v2:172.21.15.77:3300/0 conn(0x7f29c80ccac0 0x7f29c806ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.366+0000 7f29d07ff700 1 --2- 172.21.15.27:0/60791955 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f29c8156810 0x7f29c806e720 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.366+0000 7f29d07ff700 1 -- 172.21.15.27:0/60791955 >> 172.21.15.27:0/60791955 conn(0x7f29c80680e0 msgr2=0x7f29c80cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:59.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.366+0000 7f29d07ff700 1 -- 172.21.15.27:0/60791955 shutdown_connections 2024-01-11T09:43:59.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.366+0000 7f29d07ff700 1 -- 172.21.15.27:0/60791955 wait complete. 2024-01-11T09:43:59.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.367+0000 7f29d07ff700 1 Processor -- start 2024-01-11T09:43:59.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.367+0000 7f29d07ff700 1 -- start start 2024-01-11T09:43:59.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.368+0000 7f29d07ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f29c80ccac0 0x7f29c80aa9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:59.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.368+0000 7f29d07ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29c8156440 0x7f29c80c1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:59.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.368+0000 7f29d07ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29c8156810 0x7f29c80afad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:59.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.368+0000 7f29d07ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c80aa560 con 0x7f29c8156810 2024-01-11T09:43:59.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.368+0000 7f29d07ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c80a9950 con 0x7f29c8156440 2024-01-11T09:43:59.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.368+0000 7f29d07ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c80a9c30 con 0x7f29c80ccac0 2024-01-11T09:43:59.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.368+0000 7f29ce59b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f29c80ccac0 0x7f29c80aa9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:59.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.368+0000 7f29ced9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29c8156810 0x7f29c80afad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:59.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.369+0000 7f29ce59b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f29c80ccac0 0x7f29c80aa9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:33060/0 (socket says 172.21.15.27:33060) 2024-01-11T09:43:59.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.369+0000 7f29ce59b700 1 -- 172.21.15.27:0/4223272164 learned_addr learned my addr 172.21.15.27:0/4223272164 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:59.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.369+0000 7f29cdd9a700 1 --2- 172.21.15.27:0/4223272164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29c8156440 0x7f29c80c1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:59.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.369+0000 7f29ce59b700 1 -- 172.21.15.27:0/4223272164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29c8156440 msgr2=0x7f29c80c1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:59.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.369+0000 7f29ce59b700 1 --2- 172.21.15.27:0/4223272164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29c8156440 0x7f29c80c1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.369+0000 7f29ce59b700 1 -- 172.21.15.27:0/4223272164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29c8156810 msgr2=0x7f29c80afad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:59.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.369+0000 7f29ce59b700 1 --2- 172.21.15.27:0/4223272164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29c8156810 0x7f29c80afad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.369+0000 7f29ce59b700 1 -- 172.21.15.27:0/4223272164 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b8009420 con 0x7f29c80ccac0 2024-01-11T09:43:59.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.369+0000 7f29ced9c700 1 --2- 172.21.15.27:0/4223272164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29c8156810 0x7f29c80afad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:43:59.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.369+0000 7f29ce59b700 1 --2- 172.21.15.27:0/4223272164 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f29c80ccac0 0x7f29c80aa9f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f29c400c7e0 tx=0x7f29c4007e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:59.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.369+0000 7f29be7fc700 1 -- 172.21.15.27:0/4223272164 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c4005d20 con 0x7f29c80ccac0 2024-01-11T09:43:59.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.370+0000 7f29be7fc700 1 -- 172.21.15.27:0/4223272164 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c4005e80 con 0x7f29c80ccac0 2024-01-11T09:43:59.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.370+0000 7f29be7fc700 1 -- 172.21.15.27:0/4223272164 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c4007320 con 0x7f29c80ccac0 2024-01-11T09:43:59.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.370+0000 7f29d07ff700 1 -- 172.21.15.27:0/4223272164 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c80a9f10 con 0x7f29c80ccac0 2024-01-11T09:43:59.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.370+0000 7f29d07ff700 1 -- 172.21.15.27:0/4223272164 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c8169c10 con 0x7f29c80ccac0 2024-01-11T09:43:59.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.371+0000 7f29d07ff700 1 -- 172.21.15.27:0/4223272164 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c80ade10 con 0x7f29c80ccac0 2024-01-11T09:43:59.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.372+0000 7f29be7fc700 1 -- 172.21.15.27:0/4223272164 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 120) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f29c4007690 con 0x7f29c80ccac0 2024-01-11T09:43:59.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.373+0000 7f29be7fc700 1 --2- 172.21.15.27:0/4223272164 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f29ac066dd0 0x7f29ac069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:59.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.373+0000 7f29be7fc700 1 -- 172.21.15.27:0/4223272164 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f29c4083ad0 con 0x7f29c80ccac0 2024-01-11T09:43:59.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.373+0000 7f29cdd9a700 1 --2- 172.21.15.27:0/4223272164 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f29ac066dd0 0x7f29ac069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:59.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.374+0000 7f29cdd9a700 1 --2- 172.21.15.27:0/4223272164 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f29ac066dd0 0x7f29ac069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f29c8156c00 tx=0x7f29b8017000 comp rx=0 tx=0).ready entity=mgr.9895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:59.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.378+0000 7f29be7fc700 1 -- 172.21.15.27:0/4223272164 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f29c4050dd0 con 0x7f29c80ccac0 2024-01-11T09:43:59.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.610+0000 7f29d07ff700 1 -- 172.21.15.27:0/4223272164 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f29c80a9d70 con 0x7f29ac066dd0 2024-01-11T09:43:59.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.664+0000 7f29be7fc700 1 -- 172.21.15.27:0/4223272164 <== mgr.9895 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7f29c80a9d70 con 0x7f29ac066dd0 2024-01-11T09:43:59.657 INFO:teuthology.orchestra.run.smithi027.stdout:Retention added to path snap_test_dir1 2024-01-11T09:43:59.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.667+0000 7f29d07ff700 1 -- 172.21.15.27:0/4223272164 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f29ac066dd0 msgr2=0x7f29ac069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:59.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.667+0000 7f29d07ff700 1 --2- 172.21.15.27:0/4223272164 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f29ac066dd0 0x7f29ac069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f29c8156c00 tx=0x7f29b8017000 comp rx=0 tx=0).stop 2024-01-11T09:43:59.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.667+0000 7f29d07ff700 1 -- 172.21.15.27:0/4223272164 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f29c80ccac0 msgr2=0x7f29c80aa9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:59.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.667+0000 7f29d07ff700 1 --2- 172.21.15.27:0/4223272164 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f29c80ccac0 0x7f29c80aa9f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f29c400c7e0 tx=0x7f29c4007e00 comp rx=0 tx=0).stop 2024-01-11T09:43:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.667+0000 7f29d07ff700 1 -- 172.21.15.27:0/4223272164 shutdown_connections 2024-01-11T09:43:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.667+0000 7f29d07ff700 1 --2- 172.21.15.27:0/4223272164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29c8156810 0x7f29c80afad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.667+0000 7f29d07ff700 1 --2- 172.21.15.27:0/4223272164 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f29c80ccac0 0x7f29c80aa9f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.667+0000 7f29d07ff700 1 --2- 172.21.15.27:0/4223272164 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f29ac066dd0 0x7f29ac069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.667+0000 7f29d07ff700 1 --2- 172.21.15.27:0/4223272164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29c8156440 0x7f29c80c1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.667+0000 7f29d07ff700 1 -- 172.21.15.27:0/4223272164 >> 172.21.15.27:0/4223272164 conn(0x7f29c80680e0 msgr2=0x7f29c8069660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.667+0000 7f29d07ff700 1 -- 172.21.15.27:0/4223272164 shutdown_connections 2024-01-11T09:43:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.667+0000 7f29d07ff700 1 -- 172.21.15.27:0/4223272164 wait complete. 2024-01-11T09:43:59.677 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2024-01-11T09:43:59.677 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-01-11-09_44_00 in ~1s... 2024-01-11T09:43:59.677 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2024-01-11T09:43:59.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-01-11T09:43:59.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.887+0000 7fe6275ff700 1 Processor -- start 2024-01-11T09:43:59.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.887+0000 7fe6275ff700 1 -- start start 2024-01-11T09:43:59.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.888+0000 7fe6275ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe620157410 0x7fe6200ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:59.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.888+0000 7fe6275ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe620069540 0x7fe6200c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:59.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.888+0000 7fe6275ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6200699d0 con 0x7fe6201577e0 2024-01-11T09:43:59.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.888+0000 7fe6275ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe620069cb0 con 0x7fe620157410 2024-01-11T09:43:59.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.888+0000 7fe6275ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe620069f90 con 0x7fe620069540 2024-01-11T09:43:59.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.889+0000 7fe62539b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe620157410 0x7fe6200ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:59.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.889+0000 7fe624b9a700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe6201577e0 0x7fe6200ae000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:37408/0 (socket says 172.21.15.27:37408) 2024-01-11T09:43:59.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.889+0000 7fe624b9a700 1 -- 172.21.15.27:0/588881595 learned_addr learned my addr 172.21.15.27:0/588881595 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:43:59.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.889+0000 7fe625b9c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe620069540 0x7fe6200c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:59.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.889+0000 7fe617fff700 1 -- 172.21.15.27:0/588881595 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280554972 0 0) 0x7fe6200699d0 con 0x7fe6201577e0 2024-01-11T09:43:59.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.889+0000 7fe617fff700 1 -- 172.21.15.27:0/588881595 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe60c003570 con 0x7fe6201577e0 2024-01-11T09:43:59.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.889+0000 7fe625b9c700 1 -- 172.21.15.27:0/588881595 >> v2:172.21.15.77:3300/0 conn(0x7fe620157410 msgr2=0x7fe6200ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:43:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.889+0000 7fe625b9c700 1 --2- 172.21.15.27:0/588881595 >> v2:172.21.15.77:3300/0 conn(0x7fe620157410 0x7fe6200ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.889+0000 7fe625b9c700 1 -- 172.21.15.27:0/588881595 >> v1:172.21.15.77:6789/0 conn(0x7fe6201577e0 legacy=0x7fe6200ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.889+0000 7fe625b9c700 1 -- 172.21.15.27:0/588881595 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe62006a270 con 0x7fe620069540 2024-01-11T09:43:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.889+0000 7fe625b9c700 1 --2- 172.21.15.27:0/588881595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe620069540 0x7fe6200c6ba0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe618009e60 tx=0x7fe618009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2534b320b920cbe server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.890+0000 7fe617fff700 1 -- 172.21.15.27:0/588881595 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61800e070 con 0x7fe620069540 2024-01-11T09:43:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.890+0000 7fe617fff700 1 -- 172.21.15.27:0/588881595 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61800bab0 con 0x7fe620069540 2024-01-11T09:43:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.890+0000 7fe617fff700 1 -- 172.21.15.27:0/588881595 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61800be70 con 0x7fe620069540 2024-01-11T09:43:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.890+0000 7fe6275ff700 1 -- 172.21.15.27:0/588881595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe620069540 msgr2=0x7fe6200c6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.890+0000 7fe6275ff700 1 --2- 172.21.15.27:0/588881595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe620069540 0x7fe6200c6ba0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe618009e60 tx=0x7fe618009780 comp rx=0 tx=0).stop 2024-01-11T09:43:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.890+0000 7fe6275ff700 1 -- 172.21.15.27:0/588881595 shutdown_connections 2024-01-11T09:43:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.890+0000 7fe6275ff700 1 --2- 172.21.15.27:0/588881595 >> v2:172.21.15.77:3300/0 conn(0x7fe620157410 0x7fe6200ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.890+0000 7fe6275ff700 1 --2- 172.21.15.27:0/588881595 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe620069540 0x7fe6200c6ba0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.890+0000 7fe6275ff700 1 -- 172.21.15.27:0/588881595 >> 172.21.15.27:0/588881595 conn(0x7fe6200680a0 msgr2=0x7fe6200cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:43:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.890+0000 7fe6275ff700 1 -- 172.21.15.27:0/588881595 shutdown_connections 2024-01-11T09:43:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.890+0000 7fe6275ff700 1 -- 172.21.15.27:0/588881595 wait complete. 2024-01-11T09:43:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.892+0000 7fe6275ff700 1 Processor -- start 2024-01-11T09:43:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.892+0000 7fe6275ff700 1 -- start start 2024-01-11T09:43:59.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.893+0000 7fe6275ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe620069540 0x7fe6200cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:59.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.893+0000 7fe6275ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe620157410 0x7fe6200cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:59.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.893+0000 7fe6275ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6201577e0 0x7fe6200c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:59.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.893+0000 7fe6275ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6200caa40 con 0x7fe620069540 2024-01-11T09:43:59.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.893+0000 7fe6275ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6200c9e30 con 0x7fe6201577e0 2024-01-11T09:43:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.893+0000 7fe6275ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6200ca110 con 0x7fe620157410 2024-01-11T09:43:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.893+0000 7fe62539b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe620069540 0x7fe6200cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.893+0000 7fe62539b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe620069540 0x7fe6200cb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40668/0 (socket says 172.21.15.27:40668) 2024-01-11T09:43:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.893+0000 7fe62539b700 1 -- 172.21.15.27:0/1602507187 learned_addr learned my addr 172.21.15.27:0/1602507187 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:43:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.893+0000 7fe624b9a700 1 --2- 172.21.15.27:0/1602507187 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe620157410 0x7fe6200cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.894+0000 7fe625b9c700 1 --2- 172.21.15.27:0/1602507187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6201577e0 0x7fe6200c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.894+0000 7fe62539b700 1 -- 172.21.15.27:0/1602507187 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe620157410 msgr2=0x7fe6200cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.894+0000 7fe62539b700 1 --2- 172.21.15.27:0/1602507187 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe620157410 0x7fe6200cb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.894+0000 7fe62539b700 1 -- 172.21.15.27:0/1602507187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6201577e0 msgr2=0x7fe6200c7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:43:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.894+0000 7fe62539b700 1 --2- 172.21.15.27:0/1602507187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6201577e0 0x7fe6200c7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:43:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.894+0000 7fe62539b700 1 -- 172.21.15.27:0/1602507187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe618009a20 con 0x7fe620069540 2024-01-11T09:43:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.894+0000 7fe62539b700 1 --2- 172.21.15.27:0/1602507187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe620069540 0x7fe6200cb400 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fe61c00ec80 tx=0x7fe61c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.894+0000 7fe6157fa700 1 -- 172.21.15.27:0/1602507187 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c00c850 con 0x7fe620069540 2024-01-11T09:43:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.894+0000 7fe6157fa700 1 -- 172.21.15.27:0/1602507187 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61c00c9b0 con 0x7fe620069540 2024-01-11T09:43:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.895+0000 7fe6275ff700 1 -- 172.21.15.27:0/1602507187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6200ca3f0 con 0x7fe620069540 2024-01-11T09:43:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.895+0000 7fe6157fa700 1 -- 172.21.15.27:0/1602507187 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c00cd30 con 0x7fe620069540 2024-01-11T09:43:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.895+0000 7fe6275ff700 1 -- 172.21.15.27:0/1602507187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe62016ce40 con 0x7fe620069540 2024-01-11T09:43:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.896+0000 7fe6275ff700 1 -- 172.21.15.27:0/1602507187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6200cad20 con 0x7fe620069540 2024-01-11T09:43:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.897+0000 7fe6157fa700 1 -- 172.21.15.27:0/1602507187 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 120) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fe61c013680 con 0x7fe620069540 2024-01-11T09:43:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.898+0000 7fe6157fa700 1 --2- 172.21.15.27:0/1602507187 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe604066dd0 0x7fe604069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:43:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.898+0000 7fe6157fa700 1 -- 172.21.15.27:0/1602507187 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6047+0+0 (secure 0 0 0) 0x7fe61c0848f0 con 0x7fe620069540 2024-01-11T09:43:59.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.898+0000 7fe624b9a700 1 --2- 172.21.15.27:0/1602507187 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe604066dd0 0x7fe604069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:43:59.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.899+0000 7fe624b9a700 1 --2- 172.21.15.27:0/1602507187 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe604066dd0 0x7fe604069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe620167920 tx=0x7fe610009040 comp rx=0 tx=0).ready entity=mgr.9895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:43:59.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:43:59.902+0000 7fe6157fa700 1 -- 172.21.15.27:0/1602507187 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe61c051cc0 con 0x7fe620069540 2024-01-11T09:44:00.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:44:00.110+0000 7fe6157fa700 1 -- 172.21.15.27:0/1602507187 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 121) v1 ==== 84862+0+0 (secure 0 0 0) 0x7fe61c052090 con 0x7fe620069540 2024-01-11T09:44:00.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:44:00.136+0000 7fe6275ff700 1 -- 172.21.15.27:0/1602507187 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe62004c3c0 con 0x7fe604066dd0 2024-01-11T09:44:00.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:44:00.144+0000 7fe6157fa700 1 -- 172.21.15.27:0/1602507187 <== mgr.9895 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7fe62004c3c0 con 0x7fe604066dd0 2024-01-11T09:44:00.136 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:44:00.136 INFO:teuthology.orchestra.run.smithi027.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-01-11T09:44:00.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:44:00.147+0000 7fe6275ff700 1 -- 172.21.15.27:0/1602507187 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe604066dd0 msgr2=0x7fe604069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:44:00.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:44:00.147+0000 7fe6275ff700 1 --2- 172.21.15.27:0/1602507187 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe604066dd0 0x7fe604069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe620167920 tx=0x7fe610009040 comp rx=0 tx=0).stop 2024-01-11T09:44:00.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:44:00.147+0000 7fe6275ff700 1 -- 172.21.15.27:0/1602507187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe620069540 msgr2=0x7fe6200cb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:44:00.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:44:00.147+0000 7fe6275ff700 1 --2- 172.21.15.27:0/1602507187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe620069540 0x7fe6200cb400 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fe61c00ec80 tx=0x7fe61c009120 comp rx=0 tx=0).stop 2024-01-11T09:44:00.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:44:00.147+0000 7fe6275ff700 1 -- 172.21.15.27:0/1602507187 shutdown_connections 2024-01-11T09:44:00.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:44:00.147+0000 7fe6275ff700 1 --2- 172.21.15.27:0/1602507187 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe604066dd0 0x7fe604069280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:44:00.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:44:00.147+0000 7fe6275ff700 1 --2- 172.21.15.27:0/1602507187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6201577e0 0x7fe6200c7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:44:00.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:44:00.147+0000 7fe6275ff700 1 --2- 172.21.15.27:0/1602507187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe620069540 0x7fe6200cb400 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:44:00.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:44:00.147+0000 7fe6275ff700 1 --2- 172.21.15.27:0/1602507187 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe620157410 0x7fe6200cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:44:00.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:44:00.147+0000 7fe6275ff700 1 -- 172.21.15.27:0/1602507187 >> 172.21.15.27:0/1602507187 conn(0x7fe6200680a0 msgr2=0x7fe62014b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:44:00.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:44:00.147+0000 7fe6275ff700 1 -- 172.21.15.27:0/1602507187 shutdown_connections 2024-01-11T09:44:00.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:44:00.147+0000 7fe6275ff700 1 -- 172.21.15.27:0/1602507187 wait complete. 2024-01-11T09:44:00.157 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-01-11T09:44:00.158 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2024-01-11T09:44:00.158 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:00.158 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:00.208 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-01-11T09:44:00.208 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:00.208 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:00.668 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:44:00.677+0000 7fcc0ef1d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:44:00.669 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:44:00.677+0000 7fcc0ef1d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:44:00.669 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:44:00.677+0000 7fcc0ef1d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:44:00.669 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:44:00.677+0000 7fcc0ef1d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:44:00.669 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:44:00.677+0000 7fcc0ef1d700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:44:01.209 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:01.209 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:01.260 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:01.262 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:01.262 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:01.262 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:01.262 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-01-11-09_44_00_UTC'] 2024-01-11T09:44:01.262 DEBUG:tasks.cephfs.test_snap_schedules:exec=1704966239.6776538, now = 1704966241.2624154, timo = 1 2024-01-11T09:44:02.263 DEBUG:tasks.cephfs.test_snap_schedules:expecting removal of snap snap_test_dir1/.snap/scheduled-2024-01-11-09_44_00 in ~61s... 2024-01-11T09:44:02.263 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:02.263 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:02.318 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:02.319 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:02.319 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:02.319 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:03.319 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:03.320 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:03.374 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:03.375 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:03.375 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:03.375 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:04.376 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:04.376 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:04.431 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:04.432 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:04.435 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:04.435 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:05.436 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:05.436 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:05.490 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:05.491 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:05.491 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:05.491 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:06.492 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:06.493 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:06.548 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:06.550 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:06.550 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:06.550 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:07.550 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:07.551 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:07.605 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:07.606 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:07.606 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:07.606 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:08.607 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:08.607 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:08.661 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:08.662 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:08.662 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:08.662 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:09.663 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:09.664 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:09.718 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:09.719 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:09.719 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:09.719 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:10.720 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:10.720 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:10.774 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:10.775 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:10.775 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:10.775 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:11.776 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:11.776 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:11.833 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:11.833 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:11.833 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:11.833 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:12.834 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:12.834 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:12.889 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:12.890 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:12.890 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:12.890 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:13.891 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:13.891 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:13.946 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:13.947 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:13.947 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:13.947 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:14.947 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:14.948 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:15.001 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:15.003 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:15.003 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:15.003 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:16.004 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:16.004 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:16.056 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:16.057 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:16.057 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:16.057 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:17.058 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:17.058 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:17.112 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:17.113 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:17.114 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:17.114 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:17.780 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:44:17.786 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:44:18.114 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:18.114 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:18.168 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:18.169 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:18.170 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:18.170 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:19.171 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:19.171 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:19.225 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:19.227 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:19.227 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:19.227 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:20.228 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:20.228 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:20.282 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:20.283 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:20.283 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:20.283 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:21.284 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:21.284 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:21.338 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:21.340 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:21.340 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:21.340 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:22.340 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:22.340 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:22.394 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:22.396 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:22.396 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:22.396 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:23.397 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:23.397 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:23.451 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:23.452 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:23.452 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:23.452 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:24.453 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:24.453 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:24.507 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:24.508 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:24.508 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:24.508 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:25.508 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:25.509 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:25.563 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:25.564 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:25.564 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:25.564 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:26.565 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:26.565 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:26.619 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:26.620 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:26.620 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:26.620 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:27.621 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:27.622 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:27.676 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:27.677 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:27.678 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:27.678 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:28.679 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:28.679 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:28.733 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:28.734 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:28.734 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:28.734 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:29.735 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:29.735 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:29.789 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:29.790 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:29.790 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:29.790 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:30.791 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:30.791 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:30.846 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:30.847 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:30.847 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:30.847 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:31.848 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:31.848 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:31.902 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:31.904 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:31.904 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:31.904 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:32.905 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:32.905 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:32.960 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:32.961 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:32.961 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:32.961 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:33.963 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:33.963 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:34.016 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:34.017 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:34.017 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:34.018 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:35.019 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:35.019 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:35.073 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:35.074 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:35.074 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:35.075 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:36.076 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:36.076 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:36.130 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:36.131 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:36.131 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:36.131 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:37.133 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:37.133 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:37.187 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:37.187 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:37.188 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:37.188 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:38.188 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:38.188 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:38.242 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:38.243 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:38.243 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:38.244 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:39.244 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:39.244 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:39.298 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:39.299 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:39.300 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:39.300 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:40.300 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:40.300 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:40.354 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:40.355 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:40.355 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:40.355 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:41.356 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:41.356 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:41.410 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:41.411 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:41.412 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:41.412 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:42.413 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:42.413 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:42.467 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:42.468 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:42.468 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:42.468 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:43.470 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:43.470 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:43.524 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:43.525 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:43.525 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:43.525 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:44.526 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:44.526 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:44.580 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:44.581 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:44.581 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:44.581 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:45.582 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:45.582 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:45.638 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:45.639 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:45.639 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:45.639 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:46.639 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:46.640 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:46.694 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:46.695 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:46.695 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:46.695 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:47.696 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:47.696 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:47.750 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:47.751 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:47.751 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:47.751 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:47.973 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:44:47.978 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:44:48.752 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:48.752 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:48.806 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:48.807 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:48.807 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:48.807 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:49.808 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:49.808 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:49.862 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:49.863 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:49.863 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:49.863 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:50.864 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:50.864 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:50.918 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:50.919 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:50.920 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:50.920 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:51.920 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:51.921 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:51.975 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:51.976 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:51.976 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:51.976 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:52.978 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:52.978 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:53.032 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:53.033 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:53.033 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:53.033 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:54.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:54.036 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:54.093 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:54.094 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:54.094 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:54.094 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:55.096 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:55.096 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:55.149 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:55.151 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:55.151 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:55.151 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:56.152 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:56.152 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:56.206 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:56.207 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:56.207 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:56.207 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:57.208 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:57.208 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:57.262 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:57.263 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:57.263 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:57.263 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:58.263 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:58.264 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:58.318 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:58.319 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:58.319 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:58.319 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:44:59.319 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:44:59.320 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:44:59.374 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:44:59.375 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:44:59.375 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:44:59.376 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:45:00.377 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:45:00.377 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:45:00.433 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_44_00_UTC 2024-01-11T09:45:00.434 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:45:00.434 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-01-11T09:45:00.434 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-01-11T09:45:01.434 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:45:01.435 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:45:01.489 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_45_00_UTC 2024-01-11T09:45:01.490 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-01-11-09_45_00_UTC'} 2024-01-11T09:45:01.490 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-01-11-09_45_00_UTC'} 2024-01-11T09:45:01.490 INFO:tasks.cephfs.test_snap_schedules:removed: {'scheduled-2024-01-11-09_44_00_UTC'} 2024-01-11T09:45:01.490 DEBUG:tasks.cephfs.test_snap_schedules:snapshots removed=['scheduled-2024-01-11-09_44_00_UTC'] 2024-01-11T09:45:01.490 DEBUG:tasks.cephfs.test_snap_schedules:exec=1704966239.6776538, now = 1704966301.4904194, timo = 61 2024-01-11T09:45:02.491 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-01-11T09:45:02.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.712+0000 7f6c3d1ff700 1 Processor -- start 2024-01-11T09:45:02.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.712+0000 7f6c3d1ff700 1 -- start start 2024-01-11T09:45:02.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.714+0000 7f6c3d1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6c38142dc0 0x7f6c380cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:02.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.714+0000 7f6c3d1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6c380cb2e0 0x7f6c380cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:02.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.714+0000 7f6c3d1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c380ccc70 con 0x7f6c38143190 2024-01-11T09:45:02.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.714+0000 7f6c3d1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3815df30 con 0x7f6c38142dc0 2024-01-11T09:45:02.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.714+0000 7f6c3d1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3815d320 con 0x7f6c380cb2e0 2024-01-11T09:45:02.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.714+0000 7f6c36d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6c38142dc0 0x7f6c380cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:02.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.714+0000 7f6c36d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6c38142dc0 0x7f6c380cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36350/0 (socket says 172.21.15.27:36350) 2024-01-11T09:45:02.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.714+0000 7f6c36d9d700 1 -- 172.21.15.27:0/782307501 learned_addr learned my addr 172.21.15.27:0/782307501 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:02.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.714+0000 7f6c3759e700 1 --2- 172.21.15.27:0/782307501 >> v2:172.21.15.77:3300/0 conn(0x7f6c380cb2e0 0x7f6c380cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:02.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.715+0000 7f6c36d9d700 1 -- 172.21.15.27:0/782307501 >> v2:172.21.15.77:3300/0 conn(0x7f6c380cb2e0 msgr2=0x7f6c380cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:02.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.715+0000 7f6c36d9d700 1 --2- 172.21.15.27:0/782307501 >> v2:172.21.15.77:3300/0 conn(0x7f6c380cb2e0 0x7f6c380cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:02.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.715+0000 7f6c36d9d700 1 -- 172.21.15.27:0/782307501 >> v1:172.21.15.27:6789/0 conn(0x7f6c38143190 legacy=0x7f6c380cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:02.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.715+0000 7f6c36d9d700 1 -- 172.21.15.27:0/782307501 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c380cb700 con 0x7f6c38142dc0 2024-01-11T09:45:02.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.715+0000 7f6c36d9d700 1 --2- 172.21.15.27:0/782307501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c38142dc0 0x7f6c380cada0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f6c381577b0 tx=0x7f6c28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce23c267e2f72af server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:02.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.715+0000 7f6c35d9b700 1 -- 172.21.15.27:0/782307501 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c28009e70 con 0x7f6c38142dc0 2024-01-11T09:45:02.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.715+0000 7f6c35d9b700 1 -- 172.21.15.27:0/782307501 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2800b860 con 0x7f6c38142dc0 2024-01-11T09:45:02.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.715+0000 7f6c35d9b700 1 -- 172.21.15.27:0/782307501 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2800bc00 con 0x7f6c38142dc0 2024-01-11T09:45:02.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.716+0000 7f6c3d1ff700 1 -- 172.21.15.27:0/782307501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c38142dc0 msgr2=0x7f6c380cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:02.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.716+0000 7f6c3d1ff700 1 --2- 172.21.15.27:0/782307501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c38142dc0 0x7f6c380cada0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f6c381577b0 tx=0x7f6c28009250 comp rx=0 tx=0).stop 2024-01-11T09:45:02.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.716+0000 7f6c3d1ff700 1 -- 172.21.15.27:0/782307501 shutdown_connections 2024-01-11T09:45:02.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.716+0000 7f6c3d1ff700 1 --2- 172.21.15.27:0/782307501 >> v2:172.21.15.77:3300/0 conn(0x7f6c380cb2e0 0x7f6c380cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:02.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.716+0000 7f6c3d1ff700 1 --2- 172.21.15.27:0/782307501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c38142dc0 0x7f6c380cada0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:02.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.716+0000 7f6c3d1ff700 1 -- 172.21.15.27:0/782307501 >> 172.21.15.27:0/782307501 conn(0x7f6c38063ca0 msgr2=0x7f6c3806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:02.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.716+0000 7f6c3d1ff700 1 -- 172.21.15.27:0/782307501 shutdown_connections 2024-01-11T09:45:02.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.716+0000 7f6c3d1ff700 1 -- 172.21.15.27:0/782307501 wait complete. 2024-01-11T09:45:02.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.718+0000 7f6c3d1ff700 1 Processor -- start 2024-01-11T09:45:02.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.718+0000 7f6c3d1ff700 1 -- start start 2024-01-11T09:45:02.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.719+0000 7f6c3d1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c380cb2e0 0x7f6c3813e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:02.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.719+0000 7f6c3d1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6c38142dc0 0x7f6c38140610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:02.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.719+0000 7f6c3d1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c38143190 0x7f6c38140b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:02.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.719+0000 7f6c3d1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3815a560 con 0x7f6c38143190 2024-01-11T09:45:02.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.719+0000 7f6c3d1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c380b3c10 con 0x7f6c380cb2e0 2024-01-11T09:45:02.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.719+0000 7f6c3d1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c380b3ef0 con 0x7f6c38142dc0 2024-01-11T09:45:02.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.719+0000 7f6c3759e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c38143190 0x7f6c38140b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:02.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.719+0000 7f6c3759e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c38143190 0x7f6c38140b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36364/0 (socket says 172.21.15.27:36364) 2024-01-11T09:45:02.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.719+0000 7f6c3759e700 1 -- 172.21.15.27:0/1702188004 learned_addr learned my addr 172.21.15.27:0/1702188004 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:02.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.719+0000 7f6c3759e700 1 -- 172.21.15.27:0/1702188004 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6c38142dc0 msgr2=0x7f6c38140610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:02.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.719+0000 7f6c3759e700 1 --2- 172.21.15.27:0/1702188004 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6c38142dc0 0x7f6c38140610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:02.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.719+0000 7f6c3759e700 1 -- 172.21.15.27:0/1702188004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c380cb2e0 msgr2=0x7f6c3813e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:02.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.719+0000 7f6c3759e700 1 --2- 172.21.15.27:0/1702188004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c380cb2e0 0x7f6c3813e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:02.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.719+0000 7f6c3759e700 1 -- 172.21.15.27:0/1702188004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c280094f0 con 0x7f6c38143190 2024-01-11T09:45:02.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.720+0000 7f6c3759e700 1 --2- 172.21.15.27:0/1702188004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c38143190 0x7f6c38140b50 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f6c30009e90 tx=0x7f6c3000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:02.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.720+0000 7f6c1effd700 1 -- 172.21.15.27:0/1702188004 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3000cc20 con 0x7f6c38143190 2024-01-11T09:45:02.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.720+0000 7f6c1effd700 1 -- 172.21.15.27:0/1702188004 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c3000f040 con 0x7f6c38143190 2024-01-11T09:45:02.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.720+0000 7f6c3d1ff700 1 -- 172.21.15.27:0/1702188004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c380b3920 con 0x7f6c38143190 2024-01-11T09:45:02.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.720+0000 7f6c1effd700 1 -- 172.21.15.27:0/1702188004 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c30014480 con 0x7f6c38143190 2024-01-11T09:45:02.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.720+0000 7f6c3d1ff700 1 -- 172.21.15.27:0/1702188004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c3813e7a0 con 0x7f6c38143190 2024-01-11T09:45:02.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.721+0000 7f6c3d1ff700 1 -- 172.21.15.27:0/1702188004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c380b1180 con 0x7f6c38143190 2024-01-11T09:45:02.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.722+0000 7f6c1effd700 1 -- 172.21.15.27:0/1702188004 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 122) v1 ==== 84975+0+0 (secure 0 0 0) 0x7f6c30014830 con 0x7f6c38143190 2024-01-11T09:45:02.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.723+0000 7f6c1effd700 1 --2- 172.21.15.27:0/1702188004 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6c240671c0 0x7f6c24069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:02.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.723+0000 7f6c1effd700 1 -- 172.21.15.27:0/1702188004 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6105+0+0 (secure 0 0 0) 0x7f6c30085650 con 0x7f6c38143190 2024-01-11T09:45:02.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.724+0000 7f6c36d9d700 1 --2- 172.21.15.27:0/1702188004 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6c240671c0 0x7f6c24069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:02.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.724+0000 7f6c36d9d700 1 --2- 172.21.15.27:0/1702188004 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6c240671c0 0x7f6c24069670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6c381577b0 tx=0x7f6c28017000 comp rx=0 tx=0).ready entity=mgr.9895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:02.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.728+0000 7f6c1effd700 1 -- 172.21.15.27:0/1702188004 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6c30055d80 con 0x7f6c38143190 2024-01-11T09:45:02.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:02.959+0000 7f6c3d1ff700 1 -- 172.21.15.27:0/1702188004 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c3804c3c0 con 0x7f6c240671c0 2024-01-11T09:45:03.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.033+0000 7f6c1effd700 1 -- 172.21.15.27:0/1702188004 <== mgr.9895 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6c3804c3c0 con 0x7f6c240671c0 2024-01-11T09:45:03.025 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule removed for path snap_test_dir1 2024-01-11T09:45:03.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.035+0000 7f6c3d1ff700 1 -- 172.21.15.27:0/1702188004 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6c240671c0 msgr2=0x7f6c24069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:03.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.035+0000 7f6c3d1ff700 1 --2- 172.21.15.27:0/1702188004 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6c240671c0 0x7f6c24069670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6c381577b0 tx=0x7f6c28017000 comp rx=0 tx=0).stop 2024-01-11T09:45:03.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.035+0000 7f6c3d1ff700 1 -- 172.21.15.27:0/1702188004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c38143190 msgr2=0x7f6c38140b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:03.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.035+0000 7f6c3d1ff700 1 --2- 172.21.15.27:0/1702188004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c38143190 0x7f6c38140b50 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f6c30009e90 tx=0x7f6c3000b040 comp rx=0 tx=0).stop 2024-01-11T09:45:03.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.035+0000 7f6c3d1ff700 1 -- 172.21.15.27:0/1702188004 shutdown_connections 2024-01-11T09:45:03.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.035+0000 7f6c3d1ff700 1 --2- 172.21.15.27:0/1702188004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c38143190 0x7f6c38140b50 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:03.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.035+0000 7f6c3d1ff700 1 --2- 172.21.15.27:0/1702188004 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6c240671c0 0x7f6c24069670 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:03.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.035+0000 7f6c3d1ff700 1 --2- 172.21.15.27:0/1702188004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c380cb2e0 0x7f6c3813e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:03.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.035+0000 7f6c3d1ff700 1 --2- 172.21.15.27:0/1702188004 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6c38142dc0 0x7f6c38140610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:03.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.035+0000 7f6c3d1ff700 1 -- 172.21.15.27:0/1702188004 >> 172.21.15.27:0/1702188004 conn(0x7f6c38063ca0 msgr2=0x7f6c3815f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:03.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.035+0000 7f6c3d1ff700 1 -- 172.21.15.27:0/1702188004 shutdown_connections 2024-01-11T09:45:03.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.035+0000 7f6c3d1ff700 1 -- 172.21.15.27:0/1702188004 wait complete. 2024-01-11T09:45:03.044 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-01-11T09:45:03.044 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:45:03.044 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:45:03.093 INFO:teuthology.orchestra.run.smithi027.stdout:scheduled-2024-01-11-09_45_00_UTC 2024-01-11T09:45:03.094 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-01-11-09_45_00_UTC 2024-01-11T09:45:03.094 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:45:03.094 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-01-11-09_45_00_UTC) 2024-01-11T09:45:03.148 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:45:03.148 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-01-11T09:45:03.201 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-11T09:45:03.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.410+0000 7f7ce57ff700 1 Processor -- start 2024-01-11T09:45:03.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.410+0000 7f7ce57ff700 1 -- start start 2024-01-11T09:45:03.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.412+0000 7f7ce57ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7ce015a1b0 0x7f7ce015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:03.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.412+0000 7f7ce57ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce00cf850 con 0x7f7ce015ae70 2024-01-11T09:45:03.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.412+0000 7f7ce57ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce00cec40 con 0x7f7ce00ccd70 2024-01-11T09:45:03.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.412+0000 7f7ce57ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ce00cef20 con 0x7f7ce015a1b0 2024-01-11T09:45:03.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.412+0000 7f7cdf7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7ce015ae70 0x7f7ce015b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57806/0 (socket says 172.21.15.27:57806) 2024-01-11T09:45:03.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.412+0000 7f7cdf7fe700 1 -- 172.21.15.27:0/232189142 learned_addr learned my addr 172.21.15.27:0/232189142 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.413+0000 7f7cdeffd700 1 --2- 172.21.15.27:0/232189142 >> v2:172.21.15.77:3300/0 conn(0x7f7ce015a1b0 0x7f7ce015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.413+0000 7f7cddffb700 1 -- 172.21.15.27:0/232189142 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589492668 0 0) 0x7f7ce00cf850 con 0x7f7ce015ae70 2024-01-11T09:45:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.413+0000 7f7cddffb700 1 -- 172.21.15.27:0/232189142 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cc8003570 con 0x7f7ce015ae70 2024-01-11T09:45:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.413+0000 7f7cddffb700 1 -- 172.21.15.27:0/232189142 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695002169 0 0) 0x7f7ce00cec40 con 0x7f7ce00ccd70 2024-01-11T09:45:03.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.413+0000 7f7cddffb700 1 -- 172.21.15.27:0/232189142 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ce00cf850 con 0x7f7ce00ccd70 2024-01-11T09:45:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.413+0000 7f7cddffb700 1 -- 172.21.15.27:0/232189142 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4168912332 0 0) 0x7f7cc8003570 con 0x7f7ce015ae70 2024-01-11T09:45:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.413+0000 7f7cddffb700 1 -- 172.21.15.27:0/232189142 >> v2:172.21.15.77:3300/0 conn(0x7f7ce015a1b0 msgr2=0x7f7ce015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.413+0000 7f7cddffb700 1 --2- 172.21.15.27:0/232189142 >> v2:172.21.15.77:3300/0 conn(0x7f7ce015a1b0 0x7f7ce015a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.413+0000 7f7cddffb700 1 -- 172.21.15.27:0/232189142 >> v1:172.21.15.77:6789/0 conn(0x7f7ce00ccd70 legacy=0x7f7ce00cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.413+0000 7f7cddffb700 1 -- 172.21.15.27:0/232189142 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce00cf200 con 0x7f7ce015ae70 2024-01-11T09:45:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.413+0000 7f7cddffb700 1 -- 172.21.15.27:0/232189142 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f7cd80017f0 con 0x7f7ce015ae70 2024-01-11T09:45:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.413+0000 7f7cddffb700 1 -- 172.21.15.27:0/232189142 >> v1:172.21.15.27:6789/0 conn(0x7f7ce015ae70 legacy=0x7f7ce015b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.413+0000 7f7cddffb700 1 --2- 172.21.15.27:0/232189142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cc8003c40 0x7f7cc8004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.413+0000 7f7cddffb700 1 --2- 172.21.15.27:0/232189142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cc8004810 0x7f7cc8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.413+0000 7f7cddffb700 1 --2- 172.21.15.27:0/232189142 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7cc8005210 0x7f7cc800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.413+0000 7f7cddffb700 1 -- 172.21.15.27:0/232189142 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce00cef20 con 0x7f7cc8004810 2024-01-11T09:45:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.413+0000 7f7cddffb700 1 -- 172.21.15.27:0/232189142 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc800c130 con 0x7f7cc8003c40 2024-01-11T09:45:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.413+0000 7f7cddffb700 1 -- 172.21.15.27:0/232189142 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc800c2c0 con 0x7f7cc8005210 2024-01-11T09:45:03.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.414+0000 7f7cde7fc700 1 --2- 172.21.15.27:0/232189142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cc8004810 0x7f7cc8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.414+0000 7f7cdeffd700 1 --2- 172.21.15.27:0/232189142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cc8003c40 0x7f7cc8004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.414+0000 7f7cdf7fe700 1 --2- 172.21.15.27:0/232189142 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7cc8005210 0x7f7cc800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.414+0000 7f7cde7fc700 1 -- 172.21.15.27:0/232189142 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7cc8005210 msgr2=0x7f7cc800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.414+0000 7f7cde7fc700 1 --2- 172.21.15.27:0/232189142 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7cc8005210 0x7f7cc800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.414+0000 7f7cde7fc700 1 -- 172.21.15.27:0/232189142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cc8003c40 msgr2=0x7f7cc8004030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.414+0000 7f7cde7fc700 1 --2- 172.21.15.27:0/232189142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cc8003c40 0x7f7cc8004030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.414+0000 7f7cde7fc700 1 -- 172.21.15.27:0/232189142 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc8003570 con 0x7f7cc8004810 2024-01-11T09:45:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.414+0000 7f7cdf7fe700 1 -- 172.21.15.27:0/232189142 reap_dead start 2024-01-11T09:45:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.414+0000 7f7cde7fc700 1 --2- 172.21.15.27:0/232189142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cc8004810 0x7f7cc8004c80 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4001a40 tx=0x7f7cd4007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd2e9bf189497e25 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.415+0000 7f7cddffb700 1 -- 172.21.15.27:0/232189142 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd400c590 con 0x7f7cc8004810 2024-01-11T09:45:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.415+0000 7f7cddffb700 1 -- 172.21.15.27:0/232189142 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd400c6f0 con 0x7f7cc8004810 2024-01-11T09:45:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.415+0000 7f7cddffb700 1 -- 172.21.15.27:0/232189142 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd400cab0 con 0x7f7cc8004810 2024-01-11T09:45:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.415+0000 7f7ce57ff700 1 -- 172.21.15.27:0/232189142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cc8004810 msgr2=0x7f7cc8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.415+0000 7f7ce57ff700 1 --2- 172.21.15.27:0/232189142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cc8004810 0x7f7cc8004c80 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4001a40 tx=0x7f7cd4007d30 comp rx=0 tx=0).stop 2024-01-11T09:45:03.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.415+0000 7f7ce57ff700 1 -- 172.21.15.27:0/232189142 shutdown_connections 2024-01-11T09:45:03.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.415+0000 7f7ce57ff700 1 --2- 172.21.15.27:0/232189142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cc8004810 0x7f7cc8004c80 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:03.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.415+0000 7f7ce57ff700 1 -- 172.21.15.27:0/232189142 >> 172.21.15.27:0/232189142 conn(0x7f7ce006ed40 msgr2=0x7f7ce006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:03.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.415+0000 7f7ce57ff700 1 -- 172.21.15.27:0/232189142 shutdown_connections 2024-01-11T09:45:03.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.415+0000 7f7ce57ff700 1 -- 172.21.15.27:0/232189142 wait complete. 2024-01-11T09:45:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.417+0000 7f7ce57ff700 1 Processor -- start 2024-01-11T09:45:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.417+0000 7f7ce57ff700 1 -- start start 2024-01-11T09:45:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.418+0000 7f7ce57ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7cc8004810 0x7f7ce0145f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.418+0000 7f7ce57ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce0154960 0x7f7ce0154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.418+0000 7f7ce57ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce00b4fc0 0x7f7ce00b5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.418+0000 7f7ce57ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce0155310 con 0x7f7ce0154960 2024-01-11T09:45:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.418+0000 7f7ce57ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce0157610 con 0x7f7ce00b4fc0 2024-01-11T09:45:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.418+0000 7f7ce57ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce01578f0 con 0x7f7cc8004810 2024-01-11T09:45:03.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.418+0000 7f7cde7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce0154960 0x7f7ce0154dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:03.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.418+0000 7f7cde7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce0154960 0x7f7ce0154dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36382/0 (socket says 172.21.15.27:36382) 2024-01-11T09:45:03.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.418+0000 7f7cde7fc700 1 -- 172.21.15.27:0/2018393423 learned_addr learned my addr 172.21.15.27:0/2018393423 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:03.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.418+0000 7f7cdf7fe700 1 --2- 172.21.15.27:0/2018393423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce00b4fc0 0x7f7ce00b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:03.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.418+0000 7f7cdeffd700 1 --2- 172.21.15.27:0/2018393423 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7cc8004810 0x7f7ce0145f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:03.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.418+0000 7f7cde7fc700 1 -- 172.21.15.27:0/2018393423 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7cc8004810 msgr2=0x7f7ce0145f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:03.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.418+0000 7f7cde7fc700 1 --2- 172.21.15.27:0/2018393423 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7cc8004810 0x7f7ce0145f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:03.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.419+0000 7f7cde7fc700 1 -- 172.21.15.27:0/2018393423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce00b4fc0 msgr2=0x7f7ce00b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:03.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.419+0000 7f7cde7fc700 1 --2- 172.21.15.27:0/2018393423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce00b4fc0 0x7f7ce00b5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:03.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.419+0000 7f7cde7fc700 1 -- 172.21.15.27:0/2018393423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce0157bd0 con 0x7f7ce0154960 2024-01-11T09:45:03.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.419+0000 7f7cdf7fe700 1 --2- 172.21.15.27:0/2018393423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce00b4fc0 0x7f7ce00b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:03.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.419+0000 7f7cde7fc700 1 --2- 172.21.15.27:0/2018393423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce0154960 0x7f7ce0154dd0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f7cc80057b0 tx=0x7f7cd4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:03.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.419+0000 7f7cc6ffd700 1 -- 172.21.15.27:0/2018393423 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd400c590 con 0x7f7ce0154960 2024-01-11T09:45:03.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.419+0000 7f7cc6ffd700 1 -- 172.21.15.27:0/2018393423 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd400ce60 con 0x7f7ce0154960 2024-01-11T09:45:03.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.419+0000 7f7ce57ff700 1 -- 172.21.15.27:0/2018393423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce0157eb0 con 0x7f7ce0154960 2024-01-11T09:45:03.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.419+0000 7f7cc6ffd700 1 -- 172.21.15.27:0/2018393423 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4006820 con 0x7f7ce0154960 2024-01-11T09:45:03.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.419+0000 7f7ce57ff700 1 -- 172.21.15.27:0/2018393423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce00b59a0 con 0x7f7ce0154960 2024-01-11T09:45:03.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.420+0000 7f7ce57ff700 1 -- 172.21.15.27:0/2018393423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce01583d0 con 0x7f7ce0154960 2024-01-11T09:45:03.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.421+0000 7f7cc6ffd700 1 -- 172.21.15.27:0/2018393423 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 122) v1 ==== 84975+0+0 (secure 0 0 0) 0x7f7cd4019070 con 0x7f7ce0154960 2024-01-11T09:45:03.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.422+0000 7f7cc6ffd700 1 --2- 172.21.15.27:0/2018393423 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7ccc0671c0 0x7f7ccc069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.422+0000 7f7cc6ffd700 1 -- 172.21.15.27:0/2018393423 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6105+0+0 (secure 0 0 0) 0x7f7cd4085c60 con 0x7f7ce0154960 2024-01-11T09:45:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.423+0000 7f7cdeffd700 1 --2- 172.21.15.27:0/2018393423 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7ccc0671c0 0x7f7ccc069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.423+0000 7f7cdeffd700 1 --2- 172.21.15.27:0/2018393423 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7ccc0671c0 0x7f7ccc069670 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7ce004d2d0 tx=0x7f7cd0008040 comp rx=0 tx=0).ready entity=mgr.9895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:03.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.426+0000 7f7cc6ffd700 1 -- 172.21.15.27:0/2018393423 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7ce01583d0 con 0x7f7ce0154960 2024-01-11T09:45:03.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:03.690+0000 7f7ce57ff700 1 -- 172.21.15.27:0/2018393423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f7ce01583d0 con 0x7f7ce0154960 2024-01-11T09:45:04.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.042+0000 7f7cc6ffd700 1 -- 172.21.15.27:0/2018393423 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 123) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f7cd4052de0 con 0x7f7ce0154960 2024-01-11T09:45:04.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.043+0000 7f7cc6ffd700 1 -- 172.21.15.27:0/2018393423 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v123) v1 ==== 93+0+0 (secure 0 0 0) 0x7f7cd4052a10 con 0x7f7ce0154960 2024-01-11T09:45:04.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.046+0000 7f7ce57ff700 1 -- 172.21.15.27:0/2018393423 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7ccc0671c0 msgr2=0x7f7ccc069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:04.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.046+0000 7f7ce57ff700 1 --2- 172.21.15.27:0/2018393423 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7ccc0671c0 0x7f7ccc069670 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7ce004d2d0 tx=0x7f7cd0008040 comp rx=0 tx=0).stop 2024-01-11T09:45:04.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.046+0000 7f7ce57ff700 1 -- 172.21.15.27:0/2018393423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce0154960 msgr2=0x7f7ce0154dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:04.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.046+0000 7f7ce57ff700 1 --2- 172.21.15.27:0/2018393423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce0154960 0x7f7ce0154dd0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f7cc80057b0 tx=0x7f7cd4015000 comp rx=0 tx=0).stop 2024-01-11T09:45:04.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.046+0000 7f7ce57ff700 1 -- 172.21.15.27:0/2018393423 shutdown_connections 2024-01-11T09:45:04.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.046+0000 7f7ce57ff700 1 --2- 172.21.15.27:0/2018393423 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7ccc0671c0 0x7f7ccc069670 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:04.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.046+0000 7f7ce57ff700 1 --2- 172.21.15.27:0/2018393423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ce00b4fc0 0x7f7ce00b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:04.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.046+0000 7f7ce57ff700 1 --2- 172.21.15.27:0/2018393423 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7cc8004810 0x7f7ce0145f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:04.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.046+0000 7f7ce57ff700 1 --2- 172.21.15.27:0/2018393423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce0154960 0x7f7ce0154dd0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:04.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.046+0000 7f7ce57ff700 1 -- 172.21.15.27:0/2018393423 >> 172.21.15.27:0/2018393423 conn(0x7f7ce006ed40 msgr2=0x7f7ce00d0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:04.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.046+0000 7f7ce57ff700 1 -- 172.21.15.27:0/2018393423 shutdown_connections 2024-01-11T09:45:04.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.047+0000 7f7ce57ff700 1 -- 172.21.15.27:0/2018393423 wait complete. 2024-01-11T09:45:04.065 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T09:45:04.065 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:45:04.168 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:45:04.263 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T09:45:04.263 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-11T09:45:04.263 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:45:04.263 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-01-11T09:45:04.264 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:04.270+0000 7f1b315a81c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:45:04.268 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:04.277+0000 7f89e21421c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:45:04.347 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T09:45:04.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:45:04.348 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-11T09:45:04.412 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:45:04.442 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:04.450+0000 7f1b315a81c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:45:04.473 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:04.482+0000 7f89e21421c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:45:04.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.645+0000 7f00b37ff700 1 Processor -- start 2024-01-11T09:45:04.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.645+0000 7f00b37ff700 1 -- start start 2024-01-11T09:45:04.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.646+0000 7f00b37ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f00ac0b09e0 0x7f00ac1610a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:04.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.646+0000 7f00b37ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac157390 con 0x7f00ac069d40 2024-01-11T09:45:04.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.646+0000 7f00b37ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac06a200 con 0x7f00ac0c8620 2024-01-11T09:45:04.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.646+0000 7f00b37ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00ac0adcd0 con 0x7f00ac0b09e0 2024-01-11T09:45:04.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.646+0000 7f00b0d9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f00ac069d40 0x7f00ac0b02d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46282/0 (socket says 172.21.15.27:46282) 2024-01-11T09:45:04.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.647+0000 7f00b0d9a700 1 -- 172.21.15.27:0/1088045910 learned_addr learned my addr 172.21.15.27:0/1088045910 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:04.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.647+0000 7f00b1d9c700 1 --2- 172.21.15.27:0/1088045910 >> v2:172.21.15.77:3300/0 conn(0x7f00ac0b09e0 0x7f00ac1610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:04.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.647+0000 7f00a3fff700 1 -- 172.21.15.27:0/1088045910 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956928137 0 0) 0x7f00ac06a200 con 0x7f00ac0c8620 2024-01-11T09:45:04.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.647+0000 7f00a3fff700 1 -- 172.21.15.27:0/1088045910 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0098003570 con 0x7f00ac0c8620 2024-01-11T09:45:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.647+0000 7f00a3fff700 1 -- 172.21.15.27:0/1088045910 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561505354 0 0) 0x7f00ac157390 con 0x7f00ac069d40 2024-01-11T09:45:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.647+0000 7f00a3fff700 1 -- 172.21.15.27:0/1088045910 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00ac06a200 con 0x7f00ac069d40 2024-01-11T09:45:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.647+0000 7f00b1d9c700 1 -- 172.21.15.27:0/1088045910 >> v1:172.21.15.77:6789/0 conn(0x7f00ac0c8620 legacy=0x7f00ac0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.647+0000 7f00b1d9c700 1 -- 172.21.15.27:0/1088045910 >> v1:172.21.15.27:6789/0 conn(0x7f00ac069d40 legacy=0x7f00ac0b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.647+0000 7f00b1d9c700 1 -- 172.21.15.27:0/1088045910 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac156d40 con 0x7f00ac0b09e0 2024-01-11T09:45:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.648+0000 7f00b1d9c700 1 --2- 172.21.15.27:0/1088045910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00ac0b09e0 0x7f00ac1610a0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009e60 tx=0x7f00a4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c090ff1bdc7d05d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.648+0000 7f00a3fff700 1 -- 172.21.15.27:0/1088045910 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a400e070 con 0x7f00ac0b09e0 2024-01-11T09:45:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.648+0000 7f00a3fff700 1 -- 172.21.15.27:0/1088045910 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a400bab0 con 0x7f00ac0b09e0 2024-01-11T09:45:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.648+0000 7f00a3fff700 1 -- 172.21.15.27:0/1088045910 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a400be70 con 0x7f00ac0b09e0 2024-01-11T09:45:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.648+0000 7f00b37ff700 1 -- 172.21.15.27:0/1088045910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00ac0b09e0 msgr2=0x7f00ac1610a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.648+0000 7f00b37ff700 1 --2- 172.21.15.27:0/1088045910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00ac0b09e0 0x7f00ac1610a0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009e60 tx=0x7f00a4009780 comp rx=0 tx=0).stop 2024-01-11T09:45:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.649+0000 7f00b37ff700 1 -- 172.21.15.27:0/1088045910 shutdown_connections 2024-01-11T09:45:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.649+0000 7f00b37ff700 1 --2- 172.21.15.27:0/1088045910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00ac0b09e0 0x7f00ac1610a0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.649+0000 7f00b37ff700 1 -- 172.21.15.27:0/1088045910 >> 172.21.15.27:0/1088045910 conn(0x7f00ac0680e0 msgr2=0x7f00ac069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:04.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.649+0000 7f00b37ff700 1 -- 172.21.15.27:0/1088045910 shutdown_connections 2024-01-11T09:45:04.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.649+0000 7f00b37ff700 1 -- 172.21.15.27:0/1088045910 wait complete. 2024-01-11T09:45:04.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.650+0000 7f00b37ff700 1 Processor -- start 2024-01-11T09:45:04.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.650+0000 7f00b37ff700 1 -- start start 2024-01-11T09:45:04.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.651+0000 7f00b37ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac069d40 0x7f00ac0c5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:04.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.651+0000 7f00b37ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00ac0b09e0 0x7f00ac14eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:04.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.651+0000 7f00b37ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00ac0c8620 0x7f00ac14f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:04.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.651+0000 7f00b37ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac1511a0 con 0x7f00ac069d40 2024-01-11T09:45:04.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.651+0000 7f00b37ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac16a210 con 0x7f00ac0c8620 2024-01-11T09:45:04.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.651+0000 7f00b37ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac16a4f0 con 0x7f00ac0b09e0 2024-01-11T09:45:04.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.652+0000 7f00b159b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac069d40 0x7f00ac0c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:04.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.652+0000 7f00b159b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac069d40 0x7f00ac0c5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60680/0 (socket says 172.21.15.27:60680) 2024-01-11T09:45:04.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.652+0000 7f00b159b700 1 -- 172.21.15.27:0/3887082019 learned_addr learned my addr 172.21.15.27:0/3887082019 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:04.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.652+0000 7f00b1d9c700 1 --2- 172.21.15.27:0/3887082019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00ac0c8620 0x7f00ac14f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:04.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.652+0000 7f00b0d9a700 1 --2- 172.21.15.27:0/3887082019 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00ac0b09e0 0x7f00ac14eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:04.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.652+0000 7f00b159b700 1 -- 172.21.15.27:0/3887082019 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00ac0b09e0 msgr2=0x7f00ac14eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:04.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.652+0000 7f00b159b700 1 --2- 172.21.15.27:0/3887082019 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00ac0b09e0 0x7f00ac14eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:04.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.652+0000 7f00b159b700 1 -- 172.21.15.27:0/3887082019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00ac0c8620 msgr2=0x7f00ac14f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:04.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.652+0000 7f00b159b700 1 --2- 172.21.15.27:0/3887082019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00ac0c8620 0x7f00ac14f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:04.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.652+0000 7f00b159b700 1 -- 172.21.15.27:0/3887082019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a4009a20 con 0x7f00ac069d40 2024-01-11T09:45:04.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.653+0000 7f00b159b700 1 --2- 172.21.15.27:0/3887082019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac069d40 0x7f00ac0c5660 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f00a800c590 tx=0x7f00a8007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:04.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.653+0000 7f00a17fa700 1 -- 172.21.15.27:0/3887082019 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8010510 con 0x7f00ac069d40 2024-01-11T09:45:04.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.653+0000 7f00a17fa700 1 -- 172.21.15.27:0/3887082019 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a8010670 con 0x7f00ac069d40 2024-01-11T09:45:04.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.653+0000 7f00a17fa700 1 -- 172.21.15.27:0/3887082019 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8010a10 con 0x7f00ac069d40 2024-01-11T09:45:04.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.653+0000 7f00b37ff700 1 -- 172.21.15.27:0/3887082019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ac16a7d0 con 0x7f00ac069d40 2024-01-11T09:45:04.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.653+0000 7f00b37ff700 1 -- 172.21.15.27:0/3887082019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ac0c5ba0 con 0x7f00ac069d40 2024-01-11T09:45:04.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.654+0000 7f00b37ff700 1 -- 172.21.15.27:0/3887082019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00ac16acf0 con 0x7f00ac069d40 2024-01-11T09:45:04.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.655+0000 7f00a17fa700 1 -- 172.21.15.27:0/3887082019 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 123) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f00a8025430 con 0x7f00ac069d40 2024-01-11T09:45:04.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.656+0000 7f00a17fa700 1 --2- 172.21.15.27:0/3887082019 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0090067170 0x7f0090069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:04.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.656+0000 7f00a17fa700 1 -- 172.21.15.27:0/3887082019 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6105+0+0 (secure 0 0 0) 0x7f00a8084fc0 con 0x7f00ac069d40 2024-01-11T09:45:04.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.657+0000 7f00b0d9a700 1 -- 172.21.15.27:0/3887082019 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0090067170 msgr2=0x7f0090069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:04.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.657+0000 7f00b0d9a700 1 --2- 172.21.15.27:0/3887082019 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0090067170 0x7f0090069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:45:04.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.659+0000 7f00a17fa700 1 -- 172.21.15.27:0/3887082019 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f00a8051ea0 con 0x7f00ac069d40 2024-01-11T09:45:04.758 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:04.766+0000 7f1b315a81c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:45:04.813 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:04.822+0000 7f89e21421c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:45:04.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.858+0000 7f00b0d9a700 1 -- 172.21.15.27:0/3887082019 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0090067170 msgr2=0x7f0090069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:04.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.858+0000 7f00b0d9a700 1 --2- 172.21.15.27:0/3887082019 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0090067170 0x7f0090069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:45:04.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.909+0000 7f00b37ff700 1 -- 172.21.15.27:0/3887082019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00ac16acf0 con 0x7f00ac069d40 2024-01-11T09:45:04.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.910+0000 7f00a17fa700 1 -- 172.21.15.27:0/3887082019 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+4603 (secure 0 0 0) 0x7f00ac16acf0 con 0x7f00ac069d40 2024-01-11T09:45:04.901 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:04.902 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":102,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8441,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/991606883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":991606883},{"type":"v1","addr":"172.21.15.77:6834","nonce":991606883}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1996818074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1996818074},{"type":"v1","addr":"172.21.15.77:6835","nonce":1996818074}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9742,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4117588854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":4117588854},{"type":"v1","addr":"172.21.15.27:6835","nonce":4117588854}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:43:41.554157+0000","modified":"2024-01-11T09:43:45.078369+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10113":{"gid":10113,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/1557605486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1557605486},{"type":"v1","addr":"172.21.15.27:6834","nonce":1557605486}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2024-01-11T09:45:04.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.912+0000 7f00b37ff700 1 -- 172.21.15.27:0/3887082019 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0090067170 msgr2=0x7f0090069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:45:04.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.912+0000 7f00b37ff700 1 --2- 172.21.15.27:0/3887082019 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0090067170 0x7f0090069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:04.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.912+0000 7f00b37ff700 1 -- 172.21.15.27:0/3887082019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac069d40 msgr2=0x7f00ac0c5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.912+0000 7f00b37ff700 1 --2- 172.21.15.27:0/3887082019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac069d40 0x7f00ac0c5660 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f00a800c590 tx=0x7f00a8007e00 comp rx=0 tx=0).stop 2024-01-11T09:45:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.912+0000 7f00b37ff700 1 -- 172.21.15.27:0/3887082019 shutdown_connections 2024-01-11T09:45:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.912+0000 7f00b37ff700 1 --2- 172.21.15.27:0/3887082019 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0090067170 0x7f0090069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.912+0000 7f00b37ff700 1 --2- 172.21.15.27:0/3887082019 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00ac0c8620 0x7f00ac14f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.912+0000 7f00b37ff700 1 --2- 172.21.15.27:0/3887082019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00ac069d40 0x7f00ac0c5660 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.912+0000 7f00b37ff700 1 --2- 172.21.15.27:0/3887082019 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00ac0b09e0 0x7f00ac14eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.912+0000 7f00b37ff700 1 -- 172.21.15.27:0/3887082019 >> 172.21.15.27:0/3887082019 conn(0x7f00ac0680e0 msgr2=0x7f00ac157770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.912+0000 7f00b37ff700 1 -- 172.21.15.27:0/3887082019 shutdown_connections 2024-01-11T09:45:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:04.913+0000 7f00b37ff700 1 -- 172.21.15.27:0/3887082019 wait complete. 2024-01-11T09:45:04.905 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 102 2024-01-11T09:45:04.923 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:45:05.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.132+0000 7fde4c9ff700 1 Processor -- start 2024-01-11T09:45:05.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.133+0000 7fde4c9ff700 1 -- start start 2024-01-11T09:45:05.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.134+0000 7fde4c9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fde480ca3e0 0x7fde480cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:05.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.134+0000 7fde4c9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fde480ca7b0 0x7fde48159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:05.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.134+0000 7fde4c9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4815d640 con 0x7fde480c8a20 2024-01-11T09:45:05.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.134+0000 7fde4c9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde4815ae70 con 0x7fde480ca3e0 2024-01-11T09:45:05.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.134+0000 7fde4c9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde4815a3c0 con 0x7fde480ca7b0 2024-01-11T09:45:05.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.134+0000 7fde4659c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fde480ca3e0 0x7fde480cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:05.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.134+0000 7fde46d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fde480c8a20 0x7fde4815dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46296/0 (socket says 172.21.15.27:46296) 2024-01-11T09:45:05.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.134+0000 7fde46d9d700 1 -- 172.21.15.27:0/231626632 learned_addr learned my addr 172.21.15.27:0/231626632 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:05.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.134+0000 7fde4659c700 1 --2- 172.21.15.27:0/231626632 >> v2:172.21.15.27:3300/0 conn(0x7fde480ca3e0 0x7fde480cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60690/0 (socket says 172.21.15.27:60690) 2024-01-11T09:45:05.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.135+0000 7fde45d9b700 1 --2- 172.21.15.27:0/231626632 >> v2:172.21.15.77:3300/0 conn(0x7fde480ca7b0 0x7fde48159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:05.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.135+0000 7fde4659c700 1 -- 172.21.15.27:0/231626632 >> v2:172.21.15.77:3300/0 conn(0x7fde480ca7b0 msgr2=0x7fde48159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:05.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.135+0000 7fde4659c700 1 --2- 172.21.15.27:0/231626632 >> v2:172.21.15.77:3300/0 conn(0x7fde480ca7b0 0x7fde48159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.135+0000 7fde4659c700 1 -- 172.21.15.27:0/231626632 >> v1:172.21.15.27:6789/0 conn(0x7fde480c8a20 legacy=0x7fde4815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:05.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.135+0000 7fde4659c700 1 -- 172.21.15.27:0/231626632 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4815a5d0 con 0x7fde480ca3e0 2024-01-11T09:45:05.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.135+0000 7fde4659c700 1 --2- 172.21.15.27:0/231626632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde480ca3e0 0x7fde480cca20 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fde380098e0 tx=0x7fde380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74f2ac6c7c631d11 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:05.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.135+0000 7fde4559a700 1 -- 172.21.15.27:0/231626632 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde38011630 con 0x7fde480ca3e0 2024-01-11T09:45:05.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.136+0000 7fde4559a700 1 -- 172.21.15.27:0/231626632 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde38011790 con 0x7fde480ca3e0 2024-01-11T09:45:05.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.136+0000 7fde4559a700 1 -- 172.21.15.27:0/231626632 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde38011b30 con 0x7fde480ca3e0 2024-01-11T09:45:05.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.136+0000 7fde4c9ff700 1 -- 172.21.15.27:0/231626632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde480ca3e0 msgr2=0x7fde480cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:05.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.136+0000 7fde4c9ff700 1 --2- 172.21.15.27:0/231626632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde480ca3e0 0x7fde480cca20 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fde380098e0 tx=0x7fde380091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.136+0000 7fde4c9ff700 1 -- 172.21.15.27:0/231626632 shutdown_connections 2024-01-11T09:45:05.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.136+0000 7fde4c9ff700 1 --2- 172.21.15.27:0/231626632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde480ca3e0 0x7fde480cca20 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.136+0000 7fde4c9ff700 1 --2- 172.21.15.27:0/231626632 >> v2:172.21.15.77:3300/0 conn(0x7fde480ca7b0 0x7fde48159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.136+0000 7fde4c9ff700 1 -- 172.21.15.27:0/231626632 >> 172.21.15.27:0/231626632 conn(0x7fde480ce3e0 msgr2=0x7fde480ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:05.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.136+0000 7fde4c9ff700 1 -- 172.21.15.27:0/231626632 shutdown_connections 2024-01-11T09:45:05.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.136+0000 7fde4c9ff700 1 -- 172.21.15.27:0/231626632 wait complete. 2024-01-11T09:45:05.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.138+0000 7fde4c9ff700 1 Processor -- start 2024-01-11T09:45:05.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.138+0000 7fde4c9ff700 1 -- start start 2024-01-11T09:45:05.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.139+0000 7fde4c9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde480c8a20 0x7fde48150d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:05.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.139+0000 7fde4c9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde480ca3e0 0x7fde4815f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:05.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.139+0000 7fde4c9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde480ca7b0 0x7fde4814e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:05.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.139+0000 7fde4c9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4815feb0 con 0x7fde480ca3e0 2024-01-11T09:45:05.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.139+0000 7fde4c9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde48150400 con 0x7fde480c8a20 2024-01-11T09:45:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.139+0000 7fde4c9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde48151410 con 0x7fde480ca7b0 2024-01-11T09:45:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.139+0000 7fde45d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde480ca3e0 0x7fde4815f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.139+0000 7fde45d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde480ca3e0 0x7fde4815f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60702/0 (socket says 172.21.15.27:60702) 2024-01-11T09:45:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.139+0000 7fde45d9b700 1 -- 172.21.15.27:0/2460382921 learned_addr learned my addr 172.21.15.27:0/2460382921 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.139+0000 7fde46d9d700 1 --2- 172.21.15.27:0/2460382921 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde480ca7b0 0x7fde4814e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.139+0000 7fde4659c700 1 --2- 172.21.15.27:0/2460382921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde480c8a20 0x7fde48150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.140+0000 7fde45d9b700 1 -- 172.21.15.27:0/2460382921 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde480ca7b0 msgr2=0x7fde4814e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.140+0000 7fde45d9b700 1 --2- 172.21.15.27:0/2460382921 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde480ca7b0 0x7fde4814e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.140+0000 7fde45d9b700 1 -- 172.21.15.27:0/2460382921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde480c8a20 msgr2=0x7fde48150d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.140+0000 7fde45d9b700 1 --2- 172.21.15.27:0/2460382921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde480c8a20 0x7fde48150d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.140+0000 7fde45d9b700 1 -- 172.21.15.27:0/2460382921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde38009420 con 0x7fde480ca3e0 2024-01-11T09:45:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.140+0000 7fde45d9b700 1 --2- 172.21.15.27:0/2460382921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde480ca3e0 0x7fde4815f970 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00e930 tx=0x7fde3c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.140+0000 7fde367fc700 1 -- 172.21.15.27:0/2460382921 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c013680 con 0x7fde480ca3e0 2024-01-11T09:45:05.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.140+0000 7fde367fc700 1 -- 172.21.15.27:0/2460382921 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3c0137e0 con 0x7fde480ca3e0 2024-01-11T09:45:05.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.140+0000 7fde4c9ff700 1 -- 172.21.15.27:0/2460382921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4814fad0 con 0x7fde480ca3e0 2024-01-11T09:45:05.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.140+0000 7fde367fc700 1 -- 172.21.15.27:0/2460382921 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c013b80 con 0x7fde480ca3e0 2024-01-11T09:45:05.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.141+0000 7fde4c9ff700 1 -- 172.21.15.27:0/2460382921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde48150090 con 0x7fde480ca3e0 2024-01-11T09:45:05.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.142+0000 7fde4c9ff700 1 -- 172.21.15.27:0/2460382921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde48150570 con 0x7fde480ca3e0 2024-01-11T09:45:05.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.143+0000 7fde367fc700 1 -- 172.21.15.27:0/2460382921 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 123) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fde3c00c3e0 con 0x7fde480ca3e0 2024-01-11T09:45:05.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.144+0000 7fde367fc700 1 --2- 172.21.15.27:0/2460382921 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fde28067170 0x7fde28069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:05.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.144+0000 7fde367fc700 1 -- 172.21.15.27:0/2460382921 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6105+0+0 (secure 0 0 0) 0x7fde3c084c30 con 0x7fde480ca3e0 2024-01-11T09:45:05.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.144+0000 7fde4659c700 1 -- 172.21.15.27:0/2460382921 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fde28067170 msgr2=0x7fde28069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:05.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.144+0000 7fde4659c700 1 --2- 172.21.15.27:0/2460382921 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fde28067170 0x7fde28069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:45:05.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.152+0000 7fde367fc700 1 -- 172.21.15.27:0/2460382921 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fde3c0519e0 con 0x7fde480ca3e0 2024-01-11T09:45:05.279 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:05.287+0000 7f1b315a81c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:45:05.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.345+0000 7fde4659c700 1 -- 172.21.15.27:0/2460382921 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fde28067170 msgr2=0x7fde28069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.345+0000 7fde4659c700 1 --2- 172.21.15.27:0/2460382921 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fde28067170 0x7fde28069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:45:05.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.357+0000 7fde4c9ff700 1 -- 172.21.15.27:0/2460382921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde480718d0 con 0x7fde480ca3e0 2024-01-11T09:45:05.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.359+0000 7fde367fc700 1 -- 172.21.15.27:0/2460382921 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v201) v1 ==== 74+0+15915 (secure 0 0 0) 0x7fde3c021090 con 0x7fde480ca3e0 2024-01-11T09:45:05.351 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:05.351 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":201,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:45:00.004939+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:43:39.267819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-01-11T09:43:40.563877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.26","mappings":[{"from":0,"to":1}]},{"pgid":"25.29","mappings":[{"from":0,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/3183215496":"2024-01-12T09:43:58.050773+0000","172.21.15.77:6837/59776":"2024-01-12T09:43:58.050773+0000","172.21.15.77:6836/59776":"2024-01-12T09:43:58.050773+0000","172.21.15.77:0/2374924114":"2024-01-12T09:43:58.050773+0000","172.21.15.77:0/1884196903":"2024-01-12T09:43:58.050773+0000","172.21.15.77:0/3082251031":"2024-01-12T09:43:58.050773+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:45:05.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.361+0000 7fde4c9ff700 1 -- 172.21.15.27:0/2460382921 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fde28067170 msgr2=0x7fde28069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:45:05.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.361+0000 7fde4c9ff700 1 --2- 172.21.15.27:0/2460382921 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fde28067170 0x7fde28069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.361+0000 7fde4c9ff700 1 -- 172.21.15.27:0/2460382921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde480ca3e0 msgr2=0x7fde4815f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:05.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.361+0000 7fde4c9ff700 1 --2- 172.21.15.27:0/2460382921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde480ca3e0 0x7fde4815f970 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00e930 tx=0x7fde3c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.361+0000 7fde4c9ff700 1 -- 172.21.15.27:0/2460382921 shutdown_connections 2024-01-11T09:45:05.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.361+0000 7fde4c9ff700 1 --2- 172.21.15.27:0/2460382921 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fde480ca7b0 0x7fde4814e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.361+0000 7fde4c9ff700 1 --2- 172.21.15.27:0/2460382921 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fde28067170 0x7fde28069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.361+0000 7fde4c9ff700 1 --2- 172.21.15.27:0/2460382921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde480c8a20 0x7fde48150d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.361+0000 7fde4c9ff700 1 --2- 172.21.15.27:0/2460382921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde480ca3e0 0x7fde4815f970 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.361+0000 7fde4c9ff700 1 -- 172.21.15.27:0/2460382921 >> 172.21.15.27:0/2460382921 conn(0x7fde480ce3e0 msgr2=0x7fde48069840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:05.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.361+0000 7fde4c9ff700 1 -- 172.21.15.27:0/2460382921 shutdown_connections 2024-01-11T09:45:05.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.362+0000 7fde4c9ff700 1 -- 172.21.15.27:0/2460382921 wait complete. 2024-01-11T09:45:05.372 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T09:43:39.267819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '197', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T09:43:40.563877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '198', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T09:45:05.372 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T09:45:05.388 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:05.397+0000 7f89e21421c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:45:05.449 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:05.457+0000 7f1b315a81c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:45:05.572 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:05.580+0000 7f89e21421c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:45:05.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.586+0000 7f916adff700 1 Processor -- start 2024-01-11T09:45:05.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.587+0000 7f916adff700 1 -- start start 2024-01-11T09:45:05.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.588+0000 7f916adff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f916413fbb0 0x7f91640ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:05.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.588+0000 7f916adff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f916415a730 0x7f91640c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:05.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.588+0000 7f916adff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91640c6b30 con 0x7f916413ff80 2024-01-11T09:45:05.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.588+0000 7f916adff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91640c6800 con 0x7f916415a730 2024-01-11T09:45:05.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.588+0000 7f916adff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91640cb240 con 0x7f916413fbb0 2024-01-11T09:45:05.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.588+0000 7f916939c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f916415a730 0x7f91640c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:05.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.588+0000 7f9163fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f916413ff80 0x7f916415a020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46308/0 (socket says 172.21.15.27:46308) 2024-01-11T09:45:05.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.588+0000 7f9163fff700 1 -- 172.21.15.27:0/1184166141 learned_addr learned my addr 172.21.15.27:0/1184166141 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:05.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.588+0000 7f9168b9b700 1 --2- 172.21.15.27:0/1184166141 >> v2:172.21.15.77:3300/0 conn(0x7f916413fbb0 0x7f91640ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:05.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.589+0000 7f916939c700 1 -- 172.21.15.27:0/1184166141 >> v2:172.21.15.77:3300/0 conn(0x7f916413fbb0 msgr2=0x7f91640ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:05.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.589+0000 7f916939c700 1 --2- 172.21.15.27:0/1184166141 >> v2:172.21.15.77:3300/0 conn(0x7f916413fbb0 0x7f91640ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.589+0000 7f916939c700 1 -- 172.21.15.27:0/1184166141 >> v1:172.21.15.27:6789/0 conn(0x7f916413ff80 legacy=0x7f916415a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:05.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.589+0000 7f916939c700 1 -- 172.21.15.27:0/1184166141 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91640cb520 con 0x7f916415a730 2024-01-11T09:45:05.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.589+0000 7f91637fe700 1 -- 172.21.15.27:0/1184166141 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2410746488 0 0) 0x7f91640c6b30 con 0x7f916413ff80 2024-01-11T09:45:05.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.589+0000 7f916939c700 1 --2- 172.21.15.27:0/1184166141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f916415a730 0x7f91640c6fc0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f9164064df0 tx=0x7f915c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a1dcf552db8853 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:05.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.589+0000 7f91637fe700 1 -- 172.21.15.27:0/1184166141 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c00e070 con 0x7f916415a730 2024-01-11T09:45:05.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.590+0000 7f91637fe700 1 -- 172.21.15.27:0/1184166141 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915c00bb60 con 0x7f916415a730 2024-01-11T09:45:05.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.590+0000 7f91637fe700 1 -- 172.21.15.27:0/1184166141 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c012430 con 0x7f916415a730 2024-01-11T09:45:05.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.590+0000 7f916adff700 1 -- 172.21.15.27:0/1184166141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f916415a730 msgr2=0x7f91640c6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:05.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.590+0000 7f916adff700 1 --2- 172.21.15.27:0/1184166141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f916415a730 0x7f91640c6fc0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f9164064df0 tx=0x7f915c009780 comp rx=0 tx=0).stop 2024-01-11T09:45:05.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.590+0000 7f916adff700 1 -- 172.21.15.27:0/1184166141 shutdown_connections 2024-01-11T09:45:05.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.590+0000 7f916adff700 1 --2- 172.21.15.27:0/1184166141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f916415a730 0x7f91640c6fc0 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.590+0000 7f916adff700 1 --2- 172.21.15.27:0/1184166141 >> v2:172.21.15.77:3300/0 conn(0x7f916413fbb0 0x7f91640ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.590+0000 7f916adff700 1 -- 172.21.15.27:0/1184166141 >> 172.21.15.27:0/1184166141 conn(0x7f91640680a0 msgr2=0x7f91640cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:05.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.591+0000 7f916adff700 1 -- 172.21.15.27:0/1184166141 shutdown_connections 2024-01-11T09:45:05.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.591+0000 7f916adff700 1 -- 172.21.15.27:0/1184166141 wait complete. 2024-01-11T09:45:05.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.592+0000 7f916adff700 1 Processor -- start 2024-01-11T09:45:05.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.592+0000 7f916adff700 1 -- start start 2024-01-11T09:45:05.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.593+0000 7f916adff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f916413fbb0 0x7f916413ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:05.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.593+0000 7f916adff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f916413ff80 0x7f91640c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:05.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.593+0000 7f916adff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f916415a730 0x7f916416acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:05.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.593+0000 7f916adff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91640c4910 con 0x7f916415a730 2024-01-11T09:45:05.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.593+0000 7f916adff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9164158bf0 con 0x7f916413fbb0 2024-01-11T09:45:05.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.593+0000 7f916adff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9164158ed0 con 0x7f916413ff80 2024-01-11T09:45:05.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.594+0000 7f916939c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f916415a730 0x7f916416acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:05.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.594+0000 7f9168b9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f916413fbb0 0x7f916413ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:05.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.594+0000 7f9163fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f916413ff80 0x7f91640c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:05.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.594+0000 7f9168b9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f916413fbb0 0x7f916413ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:57362/0 (socket says 172.21.15.27:57362) 2024-01-11T09:45:05.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.594+0000 7f916939c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f916415a730 0x7f916416acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60714/0 (socket says 172.21.15.27:60714) 2024-01-11T09:45:05.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.594+0000 7f9168b9b700 1 -- 172.21.15.27:0/2129323457 learned_addr learned my addr 172.21.15.27:0/2129323457 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.594+0000 7f916939c700 1 -- 172.21.15.27:0/2129323457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f916413ff80 msgr2=0x7f91640c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.594+0000 7f916939c700 1 --2- 172.21.15.27:0/2129323457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f916413ff80 0x7f91640c3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.594+0000 7f916939c700 1 -- 172.21.15.27:0/2129323457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f916413fbb0 msgr2=0x7f916413ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.594+0000 7f916939c700 1 --2- 172.21.15.27:0/2129323457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f916413fbb0 0x7f916413ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.594+0000 7f916939c700 1 -- 172.21.15.27:0/2129323457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915c009a20 con 0x7f916415a730 2024-01-11T09:45:05.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.594+0000 7f9168b9b700 1 --2- 172.21.15.27:0/2129323457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f916413fbb0 0x7f916413ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:05.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.594+0000 7f9163fff700 1 --2- 172.21.15.27:0/2129323457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f916413ff80 0x7f91640c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:05.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.594+0000 7f916939c700 1 --2- 172.21.15.27:0/2129323457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f916415a730 0x7f916416acf0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f915c016000 tx=0x7f915c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:05.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.595+0000 7f9160ff9700 1 -- 172.21.15.27:0/2129323457 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c027470 con 0x7f916415a730 2024-01-11T09:45:05.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.595+0000 7f9160ff9700 1 -- 172.21.15.27:0/2129323457 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915c027630 con 0x7f916415a730 2024-01-11T09:45:05.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.595+0000 7f916adff700 1 -- 172.21.15.27:0/2129323457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91641591b0 con 0x7f916415a730 2024-01-11T09:45:05.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.595+0000 7f9160ff9700 1 -- 172.21.15.27:0/2129323457 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c0279f0 con 0x7f916415a730 2024-01-11T09:45:05.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.595+0000 7f916adff700 1 -- 172.21.15.27:0/2129323457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916413f170 con 0x7f916415a730 2024-01-11T09:45:05.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.596+0000 7f916adff700 1 -- 172.21.15.27:0/2129323457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f916413f360 con 0x7f916415a730 2024-01-11T09:45:05.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.597+0000 7f9160ff9700 1 -- 172.21.15.27:0/2129323457 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 123) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f915c024070 con 0x7f916415a730 2024-01-11T09:45:05.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.598+0000 7f9160ff9700 1 --2- 172.21.15.27:0/2129323457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9148067170 0x7f9148069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:05.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.598+0000 7f9160ff9700 1 -- 172.21.15.27:0/2129323457 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6105+0+0 (secure 0 0 0) 0x7f915c087160 con 0x7f916415a730 2024-01-11T09:45:05.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.598+0000 7f9168b9b700 1 -- 172.21.15.27:0/2129323457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9148067170 msgr2=0x7f9148069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:05.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.598+0000 7f9168b9b700 1 --2- 172.21.15.27:0/2129323457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9148067170 0x7f9148069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:45:05.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.602+0000 7f9160ff9700 1 -- 172.21.15.27:0/2129323457 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f915c053f10 con 0x7f916415a730 2024-01-11T09:45:05.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.799+0000 7f9168b9b700 1 -- 172.21.15.27:0/2129323457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9148067170 msgr2=0x7f9148069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:05.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.799+0000 7f9168b9b700 1 --2- 172.21.15.27:0/2129323457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9148067170 0x7f9148069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:45:05.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.807+0000 7f916adff700 1 -- 172.21.15.27:0/2129323457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f91641593a0 con 0x7f916415a730 2024-01-11T09:45:05.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.807+0000 7f9160ff9700 1 -- 172.21.15.27:0/2129323457 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v201) v1 ==== 93+0+10 (secure 0 0 0) 0x7f915c057530 con 0x7f916415a730 2024-01-11T09:45:05.799 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T09:45:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.809+0000 7f916adff700 1 -- 172.21.15.27:0/2129323457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9148067170 msgr2=0x7f9148069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:45:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.809+0000 7f916adff700 1 --2- 172.21.15.27:0/2129323457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9148067170 0x7f9148069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.809+0000 7f916adff700 1 -- 172.21.15.27:0/2129323457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f916415a730 msgr2=0x7f916416acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.809+0000 7f916adff700 1 --2- 172.21.15.27:0/2129323457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f916415a730 0x7f916416acf0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f915c016000 tx=0x7f915c019000 comp rx=0 tx=0).stop 2024-01-11T09:45:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.810+0000 7f916adff700 1 -- 172.21.15.27:0/2129323457 shutdown_connections 2024-01-11T09:45:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.810+0000 7f916adff700 1 --2- 172.21.15.27:0/2129323457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f916415a730 0x7f916416acf0 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.810+0000 7f916adff700 1 --2- 172.21.15.27:0/2129323457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9148067170 0x7f9148069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.810+0000 7f916adff700 1 --2- 172.21.15.27:0/2129323457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f916413fbb0 0x7f916413ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.810+0000 7f916adff700 1 --2- 172.21.15.27:0/2129323457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f916413ff80 0x7f91640c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.810+0000 7f916adff700 1 -- 172.21.15.27:0/2129323457 >> 172.21.15.27:0/2129323457 conn(0x7f91640680a0 msgr2=0x7f91640b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.810+0000 7f916adff700 1 -- 172.21.15.27:0/2129323457 shutdown_connections 2024-01-11T09:45:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:05.810+0000 7f916adff700 1 -- 172.21.15.27:0/2129323457 wait complete. 2024-01-11T09:45:05.820 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T09:45:05.862 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:05.870+0000 7f1b315a81c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:45:06.020 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:06.028+0000 7f89e21421c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:45:06.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.083+0000 7fae0a3ff700 1 Processor -- start 2024-01-11T09:45:06.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.083+0000 7fae0a3ff700 1 -- start start 2024-01-11T09:45:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.085+0000 7fae0a3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae04155c50 0x7fae040aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.085+0000 7fae0a3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fae040694c0 0x7fae04069910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.085+0000 7fae0a3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae0406e480 con 0x7fae040c6de0 2024-01-11T09:45:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.085+0000 7fae0a3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae040cb5a0 con 0x7fae040694c0 2024-01-11T09:45:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.085+0000 7fae0a3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae0406df00 con 0x7fae04155c50 2024-01-11T09:45:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.085+0000 7fae037fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fae040694c0 0x7fae04069910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.085+0000 7fae037fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fae040694c0 0x7fae04069910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60720/0 (socket says 172.21.15.27:60720) 2024-01-11T09:45:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.085+0000 7fae037fe700 1 -- 172.21.15.27:0/2916143894 learned_addr learned my addr 172.21.15.27:0/2916143894 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.085+0000 7fae03fff700 1 --2- 172.21.15.27:0/2916143894 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae04155c50 0x7fae040aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:06.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.085+0000 7fae037fe700 1 -- 172.21.15.27:0/2916143894 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae04155c50 msgr2=0x7fae040aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:06.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.085+0000 7fae037fe700 1 --2- 172.21.15.27:0/2916143894 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae04155c50 0x7fae040aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.085+0000 7fae037fe700 1 -- 172.21.15.27:0/2916143894 >> v1:172.21.15.77:6789/0 conn(0x7fae040c6de0 legacy=0x7fae040c7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:06.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.085+0000 7fae037fe700 1 -- 172.21.15.27:0/2916143894 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae04069e50 con 0x7fae040694c0 2024-01-11T09:45:06.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.086+0000 7fae037fe700 1 --2- 172.21.15.27:0/2916143894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae040694c0 0x7fae04069910 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fadf80099b0 tx=0x7fadf8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30b77459f5926549 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:06.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.086+0000 7fae02ffd700 1 -- 172.21.15.27:0/2916143894 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf8009ea0 con 0x7fae040694c0 2024-01-11T09:45:06.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.086+0000 7fae02ffd700 1 -- 172.21.15.27:0/2916143894 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf800b8f0 con 0x7fae040694c0 2024-01-11T09:45:06.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.086+0000 7fae02ffd700 1 -- 172.21.15.27:0/2916143894 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf800bc90 con 0x7fae040694c0 2024-01-11T09:45:06.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.087+0000 7fae0a3ff700 1 -- 172.21.15.27:0/2916143894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae040694c0 msgr2=0x7fae04069910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:06.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.087+0000 7fae0a3ff700 1 --2- 172.21.15.27:0/2916143894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae040694c0 0x7fae04069910 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fadf80099b0 tx=0x7fadf8009250 comp rx=0 tx=0).stop 2024-01-11T09:45:06.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.087+0000 7fae0a3ff700 1 -- 172.21.15.27:0/2916143894 shutdown_connections 2024-01-11T09:45:06.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.087+0000 7fae0a3ff700 1 --2- 172.21.15.27:0/2916143894 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae04155c50 0x7fae040aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.087+0000 7fae0a3ff700 1 --2- 172.21.15.27:0/2916143894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae040694c0 0x7fae04069910 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.087+0000 7fae0a3ff700 1 -- 172.21.15.27:0/2916143894 >> 172.21.15.27:0/2916143894 conn(0x7fae040680a0 msgr2=0x7fae040cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:06.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.087+0000 7fae0a3ff700 1 -- 172.21.15.27:0/2916143894 shutdown_connections 2024-01-11T09:45:06.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.087+0000 7fae0a3ff700 1 -- 172.21.15.27:0/2916143894 wait complete. 2024-01-11T09:45:06.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.088+0000 7fae0a3ff700 1 Processor -- start 2024-01-11T09:45:06.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.089+0000 7fae0a3ff700 1 -- start start 2024-01-11T09:45:06.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.090+0000 7fae0a3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae040694c0 0x7fae041439b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:06.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.090+0000 7fae0a3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae040c6de0 0x7fae0416af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:06.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.090+0000 7fae0a3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae04155c50 0x7fae041569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:06.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.090+0000 7fae0a3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae04156510 con 0x7fae04155c50 2024-01-11T09:45:06.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.090+0000 7fae0a3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae04143ef0 con 0x7fae040c6de0 2024-01-11T09:45:06.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.090+0000 7fae0a3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae0416b960 con 0x7fae040694c0 2024-01-11T09:45:06.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.090+0000 7fae0899c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae04155c50 0x7fae041569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:06.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.090+0000 7fae037fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae040c6de0 0x7fae0416af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:06.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.090+0000 7fae03fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae040694c0 0x7fae041439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:06.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.090+0000 7fae0899c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae04155c50 0x7fae041569a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60726/0 (socket says 172.21.15.27:60726) 2024-01-11T09:45:06.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.090+0000 7fae0899c700 1 -- 172.21.15.27:0/2703400580 learned_addr learned my addr 172.21.15.27:0/2703400580 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:06.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.091+0000 7fae037fe700 1 -- 172.21.15.27:0/2703400580 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae040694c0 msgr2=0x7fae041439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:06.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.091+0000 7fae037fe700 1 --2- 172.21.15.27:0/2703400580 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae040694c0 0x7fae041439b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.091+0000 7fae037fe700 1 -- 172.21.15.27:0/2703400580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae04155c50 msgr2=0x7fae041569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:06.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.091+0000 7fae037fe700 1 --2- 172.21.15.27:0/2703400580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae04155c50 0x7fae041569a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.091+0000 7fae037fe700 1 -- 172.21.15.27:0/2703400580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf80094f0 con 0x7fae040c6de0 2024-01-11T09:45:06.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.091+0000 7fae03fff700 1 --2- 172.21.15.27:0/2703400580 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae040694c0 0x7fae041439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:06.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.091+0000 7fae0899c700 1 --2- 172.21.15.27:0/2703400580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae04155c50 0x7fae041569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:06.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.091+0000 7fae037fe700 1 --2- 172.21.15.27:0/2703400580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae040c6de0 0x7fae0416af80 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fae0406b730 tx=0x7fadf8017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:06.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.091+0000 7fadebfff700 1 -- 172.21.15.27:0/2703400580 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf800ba60 con 0x7fae040c6de0 2024-01-11T09:45:06.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.091+0000 7fadebfff700 1 -- 172.21.15.27:0/2703400580 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf800bbc0 con 0x7fae040c6de0 2024-01-11T09:45:06.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.091+0000 7fadebfff700 1 -- 172.21.15.27:0/2703400580 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf8025670 con 0x7fae040c6de0 2024-01-11T09:45:06.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.091+0000 7fae0a3ff700 1 -- 172.21.15.27:0/2703400580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0416bc40 con 0x7fae040c6de0 2024-01-11T09:45:06.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.091+0000 7fae0a3ff700 1 -- 172.21.15.27:0/2703400580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae041441a0 con 0x7fae040c6de0 2024-01-11T09:45:06.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.093+0000 7fae0a3ff700 1 -- 172.21.15.27:0/2703400580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae04156680 con 0x7fae040c6de0 2024-01-11T09:45:06.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.093+0000 7fadebfff700 1 -- 172.21.15.27:0/2703400580 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 123) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fadf8023070 con 0x7fae040c6de0 2024-01-11T09:45:06.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.094+0000 7fadebfff700 1 --2- 172.21.15.27:0/2703400580 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fadf0067170 0x7fadf0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:06.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.094+0000 7fadebfff700 1 -- 172.21.15.27:0/2703400580 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6161+0+0 (secure 0 0 0) 0x7fadf8085d00 con 0x7fae040c6de0 2024-01-11T09:45:06.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.094+0000 7fae03fff700 1 -- 172.21.15.27:0/2703400580 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fadf0067170 msgr2=0x7fadf0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:06.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.094+0000 7fae03fff700 1 --2- 172.21.15.27:0/2703400580 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fadf0067170 0x7fadf0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:45:06.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.099+0000 7fadebfff700 1 -- 172.21.15.27:0/2703400580 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fadf8052a70 con 0x7fae040c6de0 2024-01-11T09:45:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.295+0000 7fae03fff700 1 -- 172.21.15.27:0/2703400580 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fadf0067170 msgr2=0x7fadf0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.295+0000 7fae03fff700 1 --2- 172.21.15.27:0/2703400580 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fadf0067170 0x7fadf0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:45:06.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.305+0000 7fae0a3ff700 1 -- 172.21.15.27:0/2703400580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fae0404c3c0 con 0x7fae040c6de0 2024-01-11T09:45:06.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.306+0000 7fadebfff700 1 -- 172.21.15.27:0/2703400580 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v202) v1 ==== 104+0+11 (secure 0 0 0) 0x7fadf802a0a0 con 0x7fae040c6de0 2024-01-11T09:45:06.297 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T09:45:06.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.308+0000 7fae0a3ff700 1 -- 172.21.15.27:0/2703400580 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fadf0067170 msgr2=0x7fadf0069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:45:06.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.308+0000 7fae0a3ff700 1 --2- 172.21.15.27:0/2703400580 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fadf0067170 0x7fadf0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.308+0000 7fae0a3ff700 1 -- 172.21.15.27:0/2703400580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae040c6de0 msgr2=0x7fae0416af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:06.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.308+0000 7fae0a3ff700 1 --2- 172.21.15.27:0/2703400580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae040c6de0 0x7fae0416af80 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fae0406b730 tx=0x7fadf8017000 comp rx=0 tx=0).stop 2024-01-11T09:45:06.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.308+0000 7fae0a3ff700 1 -- 172.21.15.27:0/2703400580 shutdown_connections 2024-01-11T09:45:06.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.308+0000 7fae0a3ff700 1 --2- 172.21.15.27:0/2703400580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae04155c50 0x7fae041569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.308+0000 7fae0a3ff700 1 --2- 172.21.15.27:0/2703400580 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fae040694c0 0x7fae041439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.308+0000 7fae0a3ff700 1 --2- 172.21.15.27:0/2703400580 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fadf0067170 0x7fadf0069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.308+0000 7fae0a3ff700 1 --2- 172.21.15.27:0/2703400580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae040c6de0 0x7fae0416af80 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.308+0000 7fae0a3ff700 1 -- 172.21.15.27:0/2703400580 >> 172.21.15.27:0/2703400580 conn(0x7fae040680a0 msgr2=0x7fae040c2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:06.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.308+0000 7fae0a3ff700 1 -- 172.21.15.27:0/2703400580 shutdown_connections 2024-01-11T09:45:06.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.308+0000 7fae0a3ff700 1 -- 172.21.15.27:0/2703400580 wait complete. 2024-01-11T09:45:06.320 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T09:45:06.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.545+0000 7f55a17ff700 1 Processor -- start 2024-01-11T09:45:06.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.545+0000 7f55a17ff700 1 -- start start 2024-01-11T09:45:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.546+0000 7f55a17ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f559c143190 0x7f559c0cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.546+0000 7f55a17ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f559c0cb4b0 0x7f559c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.546+0000 7f55a17ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c15d320 con 0x7f559c142dc0 2024-01-11T09:45:06.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.546+0000 7f55a17ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f559c15df30 con 0x7f559c0cb4b0 2024-01-11T09:45:06.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.546+0000 7f55a17ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f559c15d590 con 0x7f559c143190 2024-01-11T09:45:06.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.547+0000 7f559affd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f559c142dc0 0x7f559c0cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46316/0 (socket says 172.21.15.27:46316) 2024-01-11T09:45:06.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.547+0000 7f559affd700 1 -- 172.21.15.27:0/1439288517 learned_addr learned my addr 172.21.15.27:0/1439288517 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:06.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.547+0000 7f559a7fc700 1 --2- 172.21.15.27:0/1439288517 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f559c143190 0x7f559c0cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:06.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.547+0000 7f559b7fe700 1 --2- 172.21.15.27:0/1439288517 >> v2:172.21.15.77:3300/0 conn(0x7f559c0cb4b0 0x7f559c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:06.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.547+0000 7f5599ffb700 1 -- 172.21.15.27:0/1439288517 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764483646 0 0) 0x7f559c15d320 con 0x7f559c142dc0 2024-01-11T09:45:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.547+0000 7f5599ffb700 1 -- 172.21.15.27:0/1439288517 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5584003570 con 0x7f559c142dc0 2024-01-11T09:45:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.547+0000 7f559a7fc700 1 -- 172.21.15.27:0/1439288517 >> v2:172.21.15.77:3300/0 conn(0x7f559c0cb4b0 msgr2=0x7f559c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.547+0000 7f559a7fc700 1 --2- 172.21.15.27:0/1439288517 >> v2:172.21.15.77:3300/0 conn(0x7f559c0cb4b0 0x7f559c0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.547+0000 7f559a7fc700 1 -- 172.21.15.27:0/1439288517 >> v1:172.21.15.27:6789/0 conn(0x7f559c142dc0 legacy=0x7f559c0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.547+0000 7f559a7fc700 1 -- 172.21.15.27:0/1439288517 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c15d870 con 0x7f559c143190 2024-01-11T09:45:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.547+0000 7f559b7fe700 1 --2- 172.21.15.27:0/1439288517 >> v2:172.21.15.77:3300/0 conn(0x7f559c0cb4b0 0x7f559c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.548+0000 7f559a7fc700 1 --2- 172.21.15.27:0/1439288517 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f559c143190 0x7f559c0cc020 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f55900098e0 tx=0x7f55900091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60cee94ddd33c472 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.548+0000 7f5599ffb700 1 -- 172.21.15.27:0/1439288517 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5590011630 con 0x7f559c143190 2024-01-11T09:45:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.548+0000 7f5599ffb700 1 -- 172.21.15.27:0/1439288517 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5590011790 con 0x7f559c143190 2024-01-11T09:45:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.548+0000 7f5599ffb700 1 -- 172.21.15.27:0/1439288517 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5590011b30 con 0x7f559c143190 2024-01-11T09:45:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.549+0000 7f55a17ff700 1 -- 172.21.15.27:0/1439288517 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f559c143190 msgr2=0x7f559c0cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.549+0000 7f55a17ff700 1 --2- 172.21.15.27:0/1439288517 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f559c143190 0x7f559c0cc020 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f55900098e0 tx=0x7f55900091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.549+0000 7f55a17ff700 1 -- 172.21.15.27:0/1439288517 shutdown_connections 2024-01-11T09:45:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.549+0000 7f55a17ff700 1 --2- 172.21.15.27:0/1439288517 >> v2:172.21.15.77:3300/0 conn(0x7f559c0cb4b0 0x7f559c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.549+0000 7f55a17ff700 1 --2- 172.21.15.27:0/1439288517 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f559c143190 0x7f559c0cc020 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.549+0000 7f55a17ff700 1 -- 172.21.15.27:0/1439288517 >> 172.21.15.27:0/1439288517 conn(0x7f559c063ca0 msgr2=0x7f559c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.549+0000 7f55a17ff700 1 -- 172.21.15.27:0/1439288517 shutdown_connections 2024-01-11T09:45:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.549+0000 7f55a17ff700 1 -- 172.21.15.27:0/1439288517 wait complete. 2024-01-11T09:45:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.550+0000 7f55a17ff700 1 Processor -- start 2024-01-11T09:45:06.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.551+0000 7f55a17ff700 1 -- start start 2024-01-11T09:45:06.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.551+0000 7f55a17ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f559c0cb4b0 0x7f559c16a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:06.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.551+0000 7f55a17ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f559c142dc0 0x7f559c16cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:06.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.551+0000 7f55a17ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f559c143190 0x7f559c0c4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:06.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.551+0000 7f55a17ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c0b3040 con 0x7f559c0cb4b0 2024-01-11T09:45:06.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.551+0000 7f55a17ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c16af20 con 0x7f559c142dc0 2024-01-11T09:45:06.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.551+0000 7f55a17ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f559c0b2150 con 0x7f559c143190 2024-01-11T09:45:06.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.552+0000 7f559affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f559c0cb4b0 0x7f559c16a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:06.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.552+0000 7f559affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f559c0cb4b0 0x7f559c16a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60738/0 (socket says 172.21.15.27:60738) 2024-01-11T09:45:06.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.552+0000 7f559affd700 1 -- 172.21.15.27:0/1806828210 learned_addr learned my addr 172.21.15.27:0/1806828210 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:06.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.552+0000 7f559a7fc700 1 --2- 172.21.15.27:0/1806828210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f559c142dc0 0x7f559c16cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:06.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.552+0000 7f559b7fe700 1 --2- 172.21.15.27:0/1806828210 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f559c143190 0x7f559c0c4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:06.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.552+0000 7f559affd700 1 -- 172.21.15.27:0/1806828210 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f559c143190 msgr2=0x7f559c0c4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:06.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.552+0000 7f559affd700 1 --2- 172.21.15.27:0/1806828210 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f559c143190 0x7f559c0c4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.552+0000 7f559affd700 1 -- 172.21.15.27:0/1806828210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f559c142dc0 msgr2=0x7f559c16cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:06.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.552+0000 7f559affd700 1 --2- 172.21.15.27:0/1806828210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f559c142dc0 0x7f559c16cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.552+0000 7f559affd700 1 -- 172.21.15.27:0/1806828210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5594009a20 con 0x7f559c0cb4b0 2024-01-11T09:45:06.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.552+0000 7f559b7fe700 1 --2- 172.21.15.27:0/1806828210 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f559c143190 0x7f559c0c4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:06.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.552+0000 7f559affd700 1 --2- 172.21.15.27:0/1806828210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f559c0cb4b0 0x7f559c16a9e0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f558c00c7c0 tx=0x7f558c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:06.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.553+0000 7f5582ffd700 1 -- 172.21.15.27:0/1806828210 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c005b30 con 0x7f559c0cb4b0 2024-01-11T09:45:06.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.553+0000 7f5582ffd700 1 -- 172.21.15.27:0/1806828210 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558c005cb0 con 0x7f559c0cb4b0 2024-01-11T09:45:06.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.553+0000 7f5582ffd700 1 -- 172.21.15.27:0/1806828210 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c007320 con 0x7f559c0cb4b0 2024-01-11T09:45:06.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.553+0000 7f55a17ff700 1 -- 172.21.15.27:0/1806828210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5590009420 con 0x7f559c0cb4b0 2024-01-11T09:45:06.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.553+0000 7f55a17ff700 1 -- 172.21.15.27:0/1806828210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f559c0b2710 con 0x7f559c0cb4b0 2024-01-11T09:45:06.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.554+0000 7f55a17ff700 1 -- 172.21.15.27:0/1806828210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f559c04c3c0 con 0x7f559c0cb4b0 2024-01-11T09:45:06.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.556+0000 7f5582ffd700 1 -- 172.21.15.27:0/1806828210 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 123) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f558c01b3b0 con 0x7f559c0cb4b0 2024-01-11T09:45:06.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.557+0000 7f5582ffd700 1 --2- 172.21.15.27:0/1806828210 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5588067170 0x7f5588069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:06.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.557+0000 7f5582ffd700 1 -- 172.21.15.27:0/1806828210 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f558c084530 con 0x7f559c0cb4b0 2024-01-11T09:45:06.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.557+0000 7f559a7fc700 1 -- 172.21.15.27:0/1806828210 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5588067170 msgr2=0x7f5588069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:06.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.557+0000 7f559a7fc700 1 --2- 172.21.15.27:0/1806828210 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5588067170 0x7f5588069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:45:06.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.564+0000 7f5582ffd700 1 -- 172.21.15.27:0/1806828210 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f558c051250 con 0x7f559c0cb4b0 2024-01-11T09:45:06.659 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:06.667+0000 7f1b315a81c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:45:06.748 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:06.756+0000 7f1b315a81c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:45:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.758+0000 7f559a7fc700 1 -- 172.21.15.27:0/1806828210 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5588067170 msgr2=0x7f5588069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.758+0000 7f559a7fc700 1 --2- 172.21.15.27:0/1806828210 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5588067170 0x7f5588069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:45:06.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.769+0000 7f55a17ff700 1 -- 172.21.15.27:0/1806828210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f559c06c4b0 con 0x7f559c0cb4b0 2024-01-11T09:45:06.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.769+0000 7f5582ffd700 1 -- 172.21.15.27:0/1806828210 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v202) v1 ==== 100+0+11 (secure 0 0 0) 0x7f558c017020 con 0x7f559c0cb4b0 2024-01-11T09:45:06.761 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T09:45:06.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.772+0000 7f55a17ff700 1 -- 172.21.15.27:0/1806828210 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5588067170 msgr2=0x7f5588069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:45:06.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.772+0000 7f55a17ff700 1 --2- 172.21.15.27:0/1806828210 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5588067170 0x7f5588069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.772+0000 7f55a17ff700 1 -- 172.21.15.27:0/1806828210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f559c0cb4b0 msgr2=0x7f559c16a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:06.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.772+0000 7f55a17ff700 1 --2- 172.21.15.27:0/1806828210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f559c0cb4b0 0x7f559c16a9e0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f558c00c7c0 tx=0x7f558c007e00 comp rx=0 tx=0).stop 2024-01-11T09:45:06.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.772+0000 7f55a17ff700 1 -- 172.21.15.27:0/1806828210 shutdown_connections 2024-01-11T09:45:06.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.772+0000 7f55a17ff700 1 --2- 172.21.15.27:0/1806828210 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f559c143190 0x7f559c0c4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.772+0000 7f55a17ff700 1 --2- 172.21.15.27:0/1806828210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f559c0cb4b0 0x7f559c16a9e0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.772+0000 7f55a17ff700 1 --2- 172.21.15.27:0/1806828210 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5588067170 0x7f5588069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.772+0000 7f55a17ff700 1 --2- 172.21.15.27:0/1806828210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f559c142dc0 0x7f559c16cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.772+0000 7f55a17ff700 1 -- 172.21.15.27:0/1806828210 >> 172.21.15.27:0/1806828210 conn(0x7f559c063ca0 msgr2=0x7f559c069650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:06.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.772+0000 7f55a17ff700 1 -- 172.21.15.27:0/1806828210 shutdown_connections 2024-01-11T09:45:06.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.772+0000 7f55a17ff700 1 -- 172.21.15.27:0/1806828210 wait complete. 2024-01-11T09:45:06.783 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:45:06.888 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:06.896+0000 7f89e21421c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:45:06.904 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:06.912+0000 7f1b315a81c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:45:06.985 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:06.993+0000 7f89e21421c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:45:06.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.995+0000 7fb5605ff700 1 Processor -- start 2024-01-11T09:45:06.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.995+0000 7fb5605ff700 1 -- start start 2024-01-11T09:45:06.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.996+0000 7fb5605ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5580b19d0 0x7fb558158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:06.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.996+0000 7fb5605ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb558158b60 con 0x7fb55806b610 2024-01-11T09:45:06.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.997+0000 7fb5605ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5581599b0 con 0x7fb5581575e0 2024-01-11T09:45:06.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.997+0000 7fb5605ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5580cb5e0 con 0x7fb5580b19d0 2024-01-11T09:45:06.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.997+0000 7fb55db9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5580b19d0 0x7fb558158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:06.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.997+0000 7fb55eb9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb55806b610 0x7fb558159e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46326/0 (socket says 172.21.15.27:46326) 2024-01-11T09:45:06.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.997+0000 7fb55eb9c700 1 -- 172.21.15.27:0/578277840 learned_addr learned my addr 172.21.15.27:0/578277840 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:06.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.997+0000 7fb55db9a700 1 -- 172.21.15.27:0/578277840 >> v1:172.21.15.77:6789/0 conn(0x7fb5581575e0 legacy=0x7fb5580b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:06.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.997+0000 7fb55db9a700 1 -- 172.21.15.27:0/578277840 >> v1:172.21.15.27:6789/0 conn(0x7fb55806b610 legacy=0x7fb558159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:06.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.997+0000 7fb55db9a700 1 -- 172.21.15.27:0/578277840 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5580c0f90 con 0x7fb5580b19d0 2024-01-11T09:45:06.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.998+0000 7fb55db9a700 1 --2- 172.21.15.27:0/578277840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5580b19d0 0x7fb558158620 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fb5480098e0 tx=0x7fb5480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e08e5ce3c941b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:06.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.998+0000 7fb55d399700 1 -- 172.21.15.27:0/578277840 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb548009e80 con 0x7fb5580b19d0 2024-01-11T09:45:06.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.998+0000 7fb55d399700 1 -- 172.21.15.27:0/578277840 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54800b860 con 0x7fb5580b19d0 2024-01-11T09:45:06.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.998+0000 7fb55d399700 1 -- 172.21.15.27:0/578277840 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54800bbe0 con 0x7fb5580b19d0 2024-01-11T09:45:06.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.998+0000 7fb5605ff700 1 -- 172.21.15.27:0/578277840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5580b19d0 msgr2=0x7fb558158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:06.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.998+0000 7fb5605ff700 1 --2- 172.21.15.27:0/578277840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5580b19d0 0x7fb558158620 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fb5480098e0 tx=0x7fb5480091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.998+0000 7fb5605ff700 1 -- 172.21.15.27:0/578277840 shutdown_connections 2024-01-11T09:45:06.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.999+0000 7fb5605ff700 1 --2- 172.21.15.27:0/578277840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5580b19d0 0x7fb558158620 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.999+0000 7fb5605ff700 1 -- 172.21.15.27:0/578277840 >> 172.21.15.27:0/578277840 conn(0x7fb5580680e0 msgr2=0x7fb558069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:06.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.999+0000 7fb5605ff700 1 -- 172.21.15.27:0/578277840 shutdown_connections 2024-01-11T09:45:06.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:06.999+0000 7fb5605ff700 1 -- 172.21.15.27:0/578277840 wait complete. 2024-01-11T09:45:06.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.000+0000 7fb5605ff700 1 Processor -- start 2024-01-11T09:45:06.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.000+0000 7fb5605ff700 1 -- start start 2024-01-11T09:45:06.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.001+0000 7fb5605ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb55806b610 0x7fb558152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:06.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.001+0000 7fb5605ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5580b19d0 0x7fb5580c8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:06.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.001+0000 7fb5605ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5581575e0 0x7fb558156450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:06.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.001+0000 7fb5605ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5580c8a20 con 0x7fb55806b610 2024-01-11T09:45:06.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.001+0000 7fb5605ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb558153190 con 0x7fb5580b19d0 2024-01-11T09:45:06.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.001+0000 7fb5605ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5580c7060 con 0x7fb5581575e0 2024-01-11T09:45:06.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.001+0000 7fb55e39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb55806b610 0x7fb558152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:06.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.002+0000 7fb55db9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5580b19d0 0x7fb5580c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:06.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.002+0000 7fb55e39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb55806b610 0x7fb558152c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60770/0 (socket says 172.21.15.27:60770) 2024-01-11T09:45:06.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.002+0000 7fb55e39b700 1 -- 172.21.15.27:0/1947785213 learned_addr learned my addr 172.21.15.27:0/1947785213 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:06.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.002+0000 7fb55eb9c700 1 --2- 172.21.15.27:0/1947785213 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5581575e0 0x7fb558156450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:06.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.002+0000 7fb55e39b700 1 -- 172.21.15.27:0/1947785213 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5581575e0 msgr2=0x7fb558156450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:06.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.002+0000 7fb55e39b700 1 --2- 172.21.15.27:0/1947785213 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5581575e0 0x7fb558156450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.002+0000 7fb55e39b700 1 -- 172.21.15.27:0/1947785213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5580b19d0 msgr2=0x7fb5580c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:06.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.002+0000 7fb55e39b700 1 --2- 172.21.15.27:0/1947785213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5580b19d0 0x7fb5580c8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:06.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.002+0000 7fb55e39b700 1 -- 172.21.15.27:0/1947785213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb548009420 con 0x7fb55806b610 2024-01-11T09:45:06.995 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:07.001+0000 7f1b315a81c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:45:06.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.002+0000 7fb55e39b700 1 --2- 172.21.15.27:0/1947785213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb55806b610 0x7fb558152c50 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fb55400c510 tx=0x7fb55400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:06.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.003+0000 7fb54e7fc700 1 -- 172.21.15.27:0/1947785213 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb554011590 con 0x7fb55806b610 2024-01-11T09:45:06.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.003+0000 7fb54e7fc700 1 -- 172.21.15.27:0/1947785213 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb554011710 con 0x7fb55806b610 2024-01-11T09:45:06.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.003+0000 7fb5605ff700 1 -- 172.21.15.27:0/1947785213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5580c8750 con 0x7fb55806b610 2024-01-11T09:45:06.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.003+0000 7fb54e7fc700 1 -- 172.21.15.27:0/1947785213 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb554011ad0 con 0x7fb55806b610 2024-01-11T09:45:06.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.003+0000 7fb5605ff700 1 -- 172.21.15.27:0/1947785213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5580c6a10 con 0x7fb55806b610 2024-01-11T09:45:06.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.004+0000 7fb5605ff700 1 -- 172.21.15.27:0/1947785213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5580718d0 con 0x7fb55806b610 2024-01-11T09:45:06.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.005+0000 7fb54e7fc700 1 -- 172.21.15.27:0/1947785213 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 123) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fb55401c440 con 0x7fb55806b610 2024-01-11T09:45:06.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.007+0000 7fb54e7fc700 1 --2- 172.21.15.27:0/1947785213 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb53c067170 0x7fb53c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:06.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.007+0000 7fb54e7fc700 1 -- 172.21.15.27:0/1947785213 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6161+0+0 (secure 0 0 0) 0x7fb554083e20 con 0x7fb55806b610 2024-01-11T09:45:06.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.007+0000 7fb55db9a700 1 -- 172.21.15.27:0/1947785213 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb53c067170 msgr2=0x7fb53c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:06.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.007+0000 7fb55db9a700 1 --2- 172.21.15.27:0/1947785213 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb53c067170 0x7fb53c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:45:07.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.010+0000 7fb54e7fc700 1 -- 172.21.15.27:0/1947785213 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb554050b90 con 0x7fb55806b610 2024-01-11T09:45:07.158 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:07.166+0000 7f89e21421c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:45:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.208+0000 7fb55db9a700 1 -- 172.21.15.27:0/1947785213 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb53c067170 msgr2=0x7fb53c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.208+0000 7fb55db9a700 1 --2- 172.21.15.27:0/1947785213 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb53c067170 0x7fb53c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:45:07.215 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:07.223+0000 7f1b315a81c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:45:07.250 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:07.258+0000 7f89e21421c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:45:07.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.260+0000 7fb5605ff700 1 -- 172.21.15.27:0/1947785213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb558158da0 con 0x7fb55806b610 2024-01-11T09:45:07.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.261+0000 7fb54e7fc700 1 -- 172.21.15.27:0/1947785213 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+4603 (secure 0 0 0) 0x7fb554017070 con 0x7fb55806b610 2024-01-11T09:45:07.252 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:07.252 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":102,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8441,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/991606883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":991606883},{"type":"v1","addr":"172.21.15.77:6834","nonce":991606883}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1996818074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1996818074},{"type":"v1","addr":"172.21.15.77:6835","nonce":1996818074}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9742,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4117588854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":4117588854},{"type":"v1","addr":"172.21.15.27:6835","nonce":4117588854}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:43:41.554157+0000","modified":"2024-01-11T09:43:45.078369+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10113":{"gid":10113,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/1557605486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1557605486},{"type":"v1","addr":"172.21.15.27:6834","nonce":1557605486}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2024-01-11T09:45:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.263+0000 7fb5605ff700 1 -- 172.21.15.27:0/1947785213 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb53c067170 msgr2=0x7fb53c069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:45:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.263+0000 7fb5605ff700 1 --2- 172.21.15.27:0/1947785213 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb53c067170 0x7fb53c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.263+0000 7fb5605ff700 1 -- 172.21.15.27:0/1947785213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb55806b610 msgr2=0x7fb558152c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.263+0000 7fb5605ff700 1 --2- 172.21.15.27:0/1947785213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb55806b610 0x7fb558152c50 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fb55400c510 tx=0x7fb55400c820 comp rx=0 tx=0).stop 2024-01-11T09:45:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.263+0000 7fb5605ff700 1 -- 172.21.15.27:0/1947785213 shutdown_connections 2024-01-11T09:45:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.263+0000 7fb5605ff700 1 --2- 172.21.15.27:0/1947785213 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5581575e0 0x7fb558156450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.263+0000 7fb5605ff700 1 --2- 172.21.15.27:0/1947785213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb55806b610 0x7fb558152c50 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.263+0000 7fb5605ff700 1 --2- 172.21.15.27:0/1947785213 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb53c067170 0x7fb53c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.263+0000 7fb5605ff700 1 --2- 172.21.15.27:0/1947785213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5580b19d0 0x7fb5580c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.263+0000 7fb5605ff700 1 -- 172.21.15.27:0/1947785213 >> 172.21.15.27:0/1947785213 conn(0x7fb5580680e0 msgr2=0x7fb55806f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:07.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.263+0000 7fb5605ff700 1 -- 172.21.15.27:0/1947785213 shutdown_connections 2024-01-11T09:45:07.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.263+0000 7fb5605ff700 1 -- 172.21.15.27:0/1947785213 wait complete. 2024-01-11T09:45:07.256 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 102 2024-01-11T09:45:07.273 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:45:07.313 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:07.321+0000 7f1b315a81c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:45:07.410 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:07.417+0000 7f1b315a81c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:45:07.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.488+0000 7fcc7e1ff700 1 Processor -- start 2024-01-11T09:45:07.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.488+0000 7fcc7e1ff700 1 -- start start 2024-01-11T09:45:07.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.490+0000 7fcc7e1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc780b19d0 0x7fcc78158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:07.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.490+0000 7fcc7e1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc7806b610 0x7fcc78159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:07.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.490+0000 7fcc7e1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc78158b60 con 0x7fcc781575e0 2024-01-11T09:45:07.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.490+0000 7fcc7e1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc780cb5e0 con 0x7fcc780b19d0 2024-01-11T09:45:07.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.490+0000 7fcc7e1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc781599b0 con 0x7fcc7806b610 2024-01-11T09:45:07.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.490+0000 7fcc76ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc780b19d0 0x7fcc78158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:07.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.490+0000 7fcc76ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc780b19d0 0x7fcc78158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60772/0 (socket says 172.21.15.27:60772) 2024-01-11T09:45:07.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.490+0000 7fcc76ffd700 1 -- 172.21.15.27:0/3341877967 learned_addr learned my addr 172.21.15.27:0/3341877967 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:07.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.490+0000 7fcc77fff700 1 --2- 172.21.15.27:0/3341877967 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc7806b610 0x7fcc78159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.490+0000 7fcc76ffd700 1 -- 172.21.15.27:0/3341877967 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc7806b610 msgr2=0x7fcc78159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.490+0000 7fcc76ffd700 1 --2- 172.21.15.27:0/3341877967 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc7806b610 0x7fcc78159e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.490+0000 7fcc76ffd700 1 -- 172.21.15.27:0/3341877967 >> v1:172.21.15.77:6789/0 conn(0x7fcc781575e0 legacy=0x7fcc780b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.490+0000 7fcc76ffd700 1 -- 172.21.15.27:0/3341877967 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc780c0f90 con 0x7fcc780b19d0 2024-01-11T09:45:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.490+0000 7fcc767fc700 1 -- 172.21.15.27:0/3341877967 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789720793 0 0) 0x7fcc68001110 con 0x7fcc781575e0 2024-01-11T09:45:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.491+0000 7fcc76ffd700 1 --2- 172.21.15.27:0/3341877967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc780b19d0 0x7fcc78158620 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0099b0 tx=0x7fcc6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f0c144327dcc783 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.491+0000 7fcc767fc700 1 -- 172.21.15.27:0/3341877967 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c009ea0 con 0x7fcc780b19d0 2024-01-11T09:45:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.491+0000 7fcc767fc700 1 -- 172.21.15.27:0/3341877967 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6c00b860 con 0x7fcc780b19d0 2024-01-11T09:45:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.491+0000 7fcc767fc700 1 -- 172.21.15.27:0/3341877967 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c00bc00 con 0x7fcc780b19d0 2024-01-11T09:45:07.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.492+0000 7fcc7e1ff700 1 -- 172.21.15.27:0/3341877967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc780b19d0 msgr2=0x7fcc78158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:07.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.492+0000 7fcc7e1ff700 1 --2- 172.21.15.27:0/3341877967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc780b19d0 0x7fcc78158620 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0099b0 tx=0x7fcc6c009250 comp rx=0 tx=0).stop 2024-01-11T09:45:07.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.492+0000 7fcc7e1ff700 1 -- 172.21.15.27:0/3341877967 shutdown_connections 2024-01-11T09:45:07.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.492+0000 7fcc7e1ff700 1 --2- 172.21.15.27:0/3341877967 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc7806b610 0x7fcc78159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.492+0000 7fcc7e1ff700 1 --2- 172.21.15.27:0/3341877967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc780b19d0 0x7fcc78158620 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.492+0000 7fcc7e1ff700 1 -- 172.21.15.27:0/3341877967 >> 172.21.15.27:0/3341877967 conn(0x7fcc780680e0 msgr2=0x7fcc78069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:07.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.492+0000 7fcc7e1ff700 1 -- 172.21.15.27:0/3341877967 shutdown_connections 2024-01-11T09:45:07.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.492+0000 7fcc7e1ff700 1 -- 172.21.15.27:0/3341877967 wait complete. 2024-01-11T09:45:07.485 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:07.493+0000 7f89e21421c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:45:07.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.494+0000 7fcc7e1ff700 1 Processor -- start 2024-01-11T09:45:07.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.494+0000 7fcc7e1ff700 1 -- start start 2024-01-11T09:45:07.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.495+0000 7fcc7e1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc7806b610 0x7fcc7815cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:07.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.495+0000 7fcc7e1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc780b19d0 0x7fcc7815b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:07.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.496+0000 7fcc7e1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc781575e0 0x7fcc780c6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:07.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.496+0000 7fcc7e1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc780c6dd0 con 0x7fcc781575e0 2024-01-11T09:45:07.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.496+0000 7fcc7e1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7815c890 con 0x7fcc780b19d0 2024-01-11T09:45:07.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.496+0000 7fcc7e1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7815bc80 con 0x7fcc7806b610 2024-01-11T09:45:07.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.496+0000 7fcc77fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc781575e0 0x7fcc780c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:07.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.496+0000 7fcc777fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc7806b610 0x7fcc7815cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:07.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.496+0000 7fcc77fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc781575e0 0x7fcc780c6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60774/0 (socket says 172.21.15.27:60774) 2024-01-11T09:45:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.496+0000 7fcc77fff700 1 -- 172.21.15.27:0/1936670478 learned_addr learned my addr 172.21.15.27:0/1936670478 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.496+0000 7fcc76ffd700 1 --2- 172.21.15.27:0/1936670478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc780b19d0 0x7fcc7815b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.496+0000 7fcc777fe700 1 -- 172.21.15.27:0/1936670478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc780b19d0 msgr2=0x7fcc7815b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.496+0000 7fcc777fe700 1 --2- 172.21.15.27:0/1936670478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc780b19d0 0x7fcc7815b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.496+0000 7fcc777fe700 1 -- 172.21.15.27:0/1936670478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc781575e0 msgr2=0x7fcc780c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.496+0000 7fcc777fe700 1 --2- 172.21.15.27:0/1936670478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc781575e0 0x7fcc780c6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.496+0000 7fcc777fe700 1 -- 172.21.15.27:0/1936670478 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6c0094f0 con 0x7fcc7806b610 2024-01-11T09:45:07.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.496+0000 7fcc76ffd700 1 --2- 172.21.15.27:0/1936670478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc780b19d0 0x7fcc7815b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:07.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.497+0000 7fcc777fe700 1 --2- 172.21.15.27:0/1936670478 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc7806b610 0x7fcc7815cd20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcc6800c510 tx=0x7fcc68007ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:07.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.497+0000 7fcc5f7fe700 1 -- 172.21.15.27:0/1936670478 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6800ce80 con 0x7fcc7806b610 2024-01-11T09:45:07.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.497+0000 7fcc5f7fe700 1 -- 172.21.15.27:0/1936670478 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc68011590 con 0x7fcc7806b610 2024-01-11T09:45:07.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.497+0000 7fcc7e1ff700 1 -- 172.21.15.27:0/1936670478 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7815bf60 con 0x7fcc7806b610 2024-01-11T09:45:07.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.497+0000 7fcc5f7fe700 1 -- 172.21.15.27:0/1936670478 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc68011950 con 0x7fcc7806b610 2024-01-11T09:45:07.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.497+0000 7fcc7e1ff700 1 -- 172.21.15.27:0/1936670478 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7815c520 con 0x7fcc7806b610 2024-01-11T09:45:07.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.498+0000 7fcc7e1ff700 1 -- 172.21.15.27:0/1936670478 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc780718d0 con 0x7fcc7806b610 2024-01-11T09:45:07.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.499+0000 7fcc5f7fe700 1 -- 172.21.15.27:0/1936670478 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 123) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fcc68011d40 con 0x7fcc7806b610 2024-01-11T09:45:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.500+0000 7fcc5f7fe700 1 --2- 172.21.15.27:0/1936670478 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcc64067170 0x7fcc64069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.500+0000 7fcc5f7fe700 1 -- 172.21.15.27:0/1936670478 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6161+0+0 (secure 0 0 0) 0x7fcc68083ce0 con 0x7fcc7806b610 2024-01-11T09:45:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.500+0000 7fcc76ffd700 1 -- 172.21.15.27:0/1936670478 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcc64067170 msgr2=0x7fcc64069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:07.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.500+0000 7fcc76ffd700 1 --2- 172.21.15.27:0/1936670478 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcc64067170 0x7fcc64069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:45:07.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.505+0000 7fcc5f7fe700 1 -- 172.21.15.27:0/1936670478 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcc68050a50 con 0x7fcc7806b610 2024-01-11T09:45:07.506 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:07.514+0000 7f1b315a81c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:45:07.588 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:07.596+0000 7f89e21421c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:45:07.691 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:07.699+0000 7f89e21421c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:45:07.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.701+0000 7fcc76ffd700 1 -- 172.21.15.27:0/1936670478 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcc64067170 msgr2=0x7fcc64069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:07.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.701+0000 7fcc76ffd700 1 --2- 172.21.15.27:0/1936670478 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcc64067170 0x7fcc64069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:45:07.697 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:07.705+0000 7f1b315a81c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:45:07.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.713+0000 7fcc7e1ff700 1 -- 172.21.15.27:0/1936670478 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc78158b60 con 0x7fcc7806b610 2024-01-11T09:45:07.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.714+0000 7fcc5f7fe700 1 -- 172.21.15.27:0/1936670478 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v202) v1 ==== 74+0+16027 (secure 0 0 0) 0x7fcc78158b60 con 0x7fcc7806b610 2024-01-11T09:45:07.706 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:07.706 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":202,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:45:06.040677+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:43:39.267819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-01-11T09:43:40.563877+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":202,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.26","mappings":[{"from":0,"to":1}]},{"pgid":"25.29","mappings":[{"from":0,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/3183215496":"2024-01-12T09:43:58.050773+0000","172.21.15.77:6837/59776":"2024-01-12T09:43:58.050773+0000","172.21.15.77:6836/59776":"2024-01-12T09:43:58.050773+0000","172.21.15.77:0/2374924114":"2024-01-12T09:43:58.050773+0000","172.21.15.77:0/1884196903":"2024-01-12T09:43:58.050773+0000","172.21.15.77:0/3082251031":"2024-01-12T09:43:58.050773+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:45:07.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.716+0000 7fcc7e1ff700 1 -- 172.21.15.27:0/1936670478 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcc64067170 msgr2=0x7fcc64069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:45:07.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.716+0000 7fcc7e1ff700 1 --2- 172.21.15.27:0/1936670478 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcc64067170 0x7fcc64069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.716+0000 7fcc7e1ff700 1 -- 172.21.15.27:0/1936670478 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc7806b610 msgr2=0x7fcc7815cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:07.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.716+0000 7fcc7e1ff700 1 --2- 172.21.15.27:0/1936670478 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc7806b610 0x7fcc7815cd20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcc6800c510 tx=0x7fcc68007ec0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.716+0000 7fcc7e1ff700 1 -- 172.21.15.27:0/1936670478 shutdown_connections 2024-01-11T09:45:07.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.716+0000 7fcc7e1ff700 1 --2- 172.21.15.27:0/1936670478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc781575e0 0x7fcc780c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.716+0000 7fcc7e1ff700 1 --2- 172.21.15.27:0/1936670478 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcc7806b610 0x7fcc7815cd20 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.716+0000 7fcc7e1ff700 1 --2- 172.21.15.27:0/1936670478 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcc64067170 0x7fcc64069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.716+0000 7fcc7e1ff700 1 --2- 172.21.15.27:0/1936670478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc780b19d0 0x7fcc7815b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.716+0000 7fcc7e1ff700 1 -- 172.21.15.27:0/1936670478 >> 172.21.15.27:0/1936670478 conn(0x7fcc780680e0 msgr2=0x7fcc781557d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:07.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.717+0000 7fcc7e1ff700 1 -- 172.21.15.27:0/1936670478 shutdown_connections 2024-01-11T09:45:07.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.717+0000 7fcc7e1ff700 1 -- 172.21.15.27:0/1936670478 wait complete. 2024-01-11T09:45:07.727 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-11T09:45:07.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:45:07.796 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:07.804+0000 7f1b315a81c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:45:07.797 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:07.806+0000 7f89e21421c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:45:07.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.938+0000 7fa5f3bff700 1 Processor -- start 2024-01-11T09:45:07.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.939+0000 7fa5f3bff700 1 -- start start 2024-01-11T09:45:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.941+0000 7fa5f3bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa5ec1599e0 0x7fa5ec0b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.941+0000 7fa5f3bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5ec150330 0x7fa5ec15a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.941+0000 7fa5f3bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec157e30 con 0x7fa5ec159db0 2024-01-11T09:45:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.941+0000 7fa5f3bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5ec157500 con 0x7fa5ec150330 2024-01-11T09:45:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.941+0000 7fa5f3bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5ec1577e0 con 0x7fa5ec1599e0 2024-01-11T09:45:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.941+0000 7fa5f219c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5ec150330 0x7fa5ec15a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.941+0000 7fa5f119a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa5ec159db0 0x7fa5ec14fc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46338/0 (socket says 172.21.15.27:46338) 2024-01-11T09:45:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.941+0000 7fa5f119a700 1 -- 172.21.15.27:0/3062776451 learned_addr learned my addr 172.21.15.27:0/3062776451 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:07.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.941+0000 7fa5f199b700 1 --2- 172.21.15.27:0/3062776451 >> v2:172.21.15.77:3300/0 conn(0x7fa5ec1599e0 0x7fa5ec0b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:07.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.941+0000 7fa5f0999700 1 -- 172.21.15.27:0/3062776451 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820061384 0 0) 0x7fa5ec157e30 con 0x7fa5ec159db0 2024-01-11T09:45:07.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.942+0000 7fa5f199b700 1 -- 172.21.15.27:0/3062776451 >> v2:172.21.15.27:3300/0 conn(0x7fa5ec150330 msgr2=0x7fa5ec15a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:07.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.942+0000 7fa5f199b700 1 --2- 172.21.15.27:0/3062776451 >> v2:172.21.15.27:3300/0 conn(0x7fa5ec150330 0x7fa5ec15a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.942+0000 7fa5f199b700 1 -- 172.21.15.27:0/3062776451 >> v1:172.21.15.27:6789/0 conn(0x7fa5ec159db0 legacy=0x7fa5ec14fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:07.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.942+0000 7fa5f199b700 1 -- 172.21.15.27:0/3062776451 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec157ac0 con 0x7fa5ec1599e0 2024-01-11T09:45:07.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.942+0000 7fa5f199b700 1 --2- 172.21.15.27:0/3062776451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5ec1599e0 0x7fa5ec0b0130 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fa5e80098e0 tx=0x7fa5e80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4586581ad7ec5512 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:07.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.942+0000 7fa5f0999700 1 -- 172.21.15.27:0/3062776451 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e8011630 con 0x7fa5ec1599e0 2024-01-11T09:45:07.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.942+0000 7fa5f0999700 1 -- 172.21.15.27:0/3062776451 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e8011790 con 0x7fa5ec1599e0 2024-01-11T09:45:07.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.942+0000 7fa5f0999700 1 -- 172.21.15.27:0/3062776451 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e8011b30 con 0x7fa5ec1599e0 2024-01-11T09:45:07.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.943+0000 7fa5f3bff700 1 -- 172.21.15.27:0/3062776451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5ec1599e0 msgr2=0x7fa5ec0b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:07.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.943+0000 7fa5f3bff700 1 --2- 172.21.15.27:0/3062776451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5ec1599e0 0x7fa5ec0b0130 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fa5e80098e0 tx=0x7fa5e80091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.943+0000 7fa5f3bff700 1 -- 172.21.15.27:0/3062776451 shutdown_connections 2024-01-11T09:45:07.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.943+0000 7fa5f3bff700 1 --2- 172.21.15.27:0/3062776451 >> v2:172.21.15.27:3300/0 conn(0x7fa5ec150330 0x7fa5ec15a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.943+0000 7fa5f3bff700 1 --2- 172.21.15.27:0/3062776451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5ec1599e0 0x7fa5ec0b0130 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.943+0000 7fa5f3bff700 1 -- 172.21.15.27:0/3062776451 >> 172.21.15.27:0/3062776451 conn(0x7fa5ec0680e0 msgr2=0x7fa5ec069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:07.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.943+0000 7fa5f3bff700 1 -- 172.21.15.27:0/3062776451 shutdown_connections 2024-01-11T09:45:07.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.943+0000 7fa5f3bff700 1 -- 172.21.15.27:0/3062776451 wait complete. 2024-01-11T09:45:07.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.945+0000 7fa5f3bff700 1 Processor -- start 2024-01-11T09:45:07.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.945+0000 7fa5f3bff700 1 -- start start 2024-01-11T09:45:07.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.946+0000 7fa5f3bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5ec150330 0x7fa5ec14bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:07.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.946+0000 7fa5f3bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5ec1599e0 0x7fa5ec0c9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:07.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.946+0000 7fa5f3bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ec159db0 0x7fa5ec0cb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:07.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.946+0000 7fa5f3bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec0cb0a0 con 0x7fa5ec159db0 2024-01-11T09:45:07.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.946+0000 7fa5f3bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec0c9ac0 con 0x7fa5ec150330 2024-01-11T09:45:07.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.946+0000 7fa5f3bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec0ada70 con 0x7fa5ec1599e0 2024-01-11T09:45:07.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.946+0000 7fa5f199b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5ec150330 0x7fa5ec14bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:07.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.946+0000 7fa5f219c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ec159db0 0x7fa5ec0cb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:07.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.946+0000 7fa5f199b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5ec150330 0x7fa5ec14bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:57438/0 (socket says 172.21.15.27:57438) 2024-01-11T09:45:07.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.946+0000 7fa5f119a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5ec1599e0 0x7fa5ec0c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:07.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.946+0000 7fa5f199b700 1 -- 172.21.15.27:0/1723139659 learned_addr learned my addr 172.21.15.27:0/1723139659 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:07.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.947+0000 7fa5f219c700 1 -- 172.21.15.27:0/1723139659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5ec1599e0 msgr2=0x7fa5ec0c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:07.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.947+0000 7fa5f219c700 1 --2- 172.21.15.27:0/1723139659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5ec1599e0 0x7fa5ec0c9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.947+0000 7fa5f219c700 1 -- 172.21.15.27:0/1723139659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5ec150330 msgr2=0x7fa5ec14bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:07.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.947+0000 7fa5f219c700 1 --2- 172.21.15.27:0/1723139659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5ec150330 0x7fa5ec14bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:07.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.947+0000 7fa5f219c700 1 -- 172.21.15.27:0/1723139659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e8009420 con 0x7fa5ec159db0 2024-01-11T09:45:07.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.947+0000 7fa5f199b700 1 --2- 172.21.15.27:0/1723139659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5ec150330 0x7fa5ec14bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:07.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.947+0000 7fa5f219c700 1 --2- 172.21.15.27:0/1723139659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ec159db0 0x7fa5ec0cb580 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fa5e400e170 tx=0x7fa5e400d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:07.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.947+0000 7fa5e1ffb700 1 -- 172.21.15.27:0/1723139659 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e400b2e0 con 0x7fa5ec159db0 2024-01-11T09:45:07.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.947+0000 7fa5e1ffb700 1 -- 172.21.15.27:0/1723139659 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e4011040 con 0x7fa5ec159db0 2024-01-11T09:45:07.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.947+0000 7fa5f3bff700 1 -- 172.21.15.27:0/1723139659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ec0add50 con 0x7fa5ec159db0 2024-01-11T09:45:07.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.947+0000 7fa5e1ffb700 1 -- 172.21.15.27:0/1723139659 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e400ec90 con 0x7fa5ec159db0 2024-01-11T09:45:07.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.947+0000 7fa5f3bff700 1 -- 172.21.15.27:0/1723139659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ec166470 con 0x7fa5ec159db0 2024-01-11T09:45:07.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.948+0000 7fa5f3bff700 1 -- 172.21.15.27:0/1723139659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5ec0cb2a0 con 0x7fa5ec159db0 2024-01-11T09:45:07.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.950+0000 7fa5e1ffb700 1 -- 172.21.15.27:0/1723139659 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 123) v1 ==== 84958+0+0 (secure 0 0 0) 0x7fa5e4009550 con 0x7fa5ec159db0 2024-01-11T09:45:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.950+0000 7fa5e1ffb700 1 --2- 172.21.15.27:0/1723139659 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa5d0067170 0x7fa5d0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.951+0000 7fa5e1ffb700 1 -- 172.21.15.27:0/1723139659 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6161+0+0 (secure 0 0 0) 0x7fa5e4087280 con 0x7fa5ec159db0 2024-01-11T09:45:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.951+0000 7fa5f199b700 1 -- 172.21.15.27:0/1723139659 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa5d0067170 msgr2=0x7fa5d0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.951+0000 7fa5f199b700 1 --2- 172.21.15.27:0/1723139659 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa5d0067170 0x7fa5d0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:45:07.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:07.955+0000 7fa5e1ffb700 1 -- 172.21.15.27:0/1723139659 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa5ec0cb2a0 con 0x7fa5ec159db0 2024-01-11T09:45:08.003 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:08.012+0000 7f89e21421c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:45:08.109 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:08.117+0000 7f89e21421c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:45:08.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.152+0000 7fa5f199b700 1 -- 172.21.15.27:0/1723139659 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa5d0067170 msgr2=0x7fa5d0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:08.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.152+0000 7fa5f199b700 1 --2- 172.21.15.27:0/1723139659 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa5d0067170 0x7fa5d0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:45:08.146 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:08.154+0000 7f1b315a81c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:45:08.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.203+0000 7fa5f3bff700 1 -- 172.21.15.27:0/1723139659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5ec0cb2a0 con 0x7fa5ec159db0 2024-01-11T09:45:08.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.204+0000 7fa5e1ffb700 1 -- 172.21.15.27:0/1723139659 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+4603 (secure 0 0 0) 0x7fa5ec0cb2a0 con 0x7fa5ec159db0 2024-01-11T09:45:08.195 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:08.196 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":102,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8441,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/991606883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":991606883},{"type":"v1","addr":"172.21.15.77:6834","nonce":991606883}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1996818074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1996818074},{"type":"v1","addr":"172.21.15.77:6835","nonce":1996818074}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9742,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4117588854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":4117588854},{"type":"v1","addr":"172.21.15.27:6835","nonce":4117588854}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:43:41.554157+0000","modified":"2024-01-11T09:43:45.078369+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10113":{"gid":10113,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/1557605486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1557605486},{"type":"v1","addr":"172.21.15.27:6834","nonce":1557605486}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2024-01-11T09:45:08.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.206+0000 7fa5f3bff700 1 -- 172.21.15.27:0/1723139659 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa5d0067170 msgr2=0x7fa5d0069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:45:08.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.206+0000 7fa5f3bff700 1 --2- 172.21.15.27:0/1723139659 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa5d0067170 0x7fa5d0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.206+0000 7fa5f3bff700 1 -- 172.21.15.27:0/1723139659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ec159db0 msgr2=0x7fa5ec0cb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:08.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.206+0000 7fa5f3bff700 1 --2- 172.21.15.27:0/1723139659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ec159db0 0x7fa5ec0cb580 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fa5e400e170 tx=0x7fa5e400d040 comp rx=0 tx=0).stop 2024-01-11T09:45:08.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.206+0000 7fa5f3bff700 1 -- 172.21.15.27:0/1723139659 shutdown_connections 2024-01-11T09:45:08.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.206+0000 7fa5f3bff700 1 --2- 172.21.15.27:0/1723139659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5ec159db0 0x7fa5ec0cb580 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.206+0000 7fa5f3bff700 1 --2- 172.21.15.27:0/1723139659 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa5d0067170 0x7fa5d0069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.206+0000 7fa5f3bff700 1 --2- 172.21.15.27:0/1723139659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5ec150330 0x7fa5ec14bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.206+0000 7fa5f3bff700 1 --2- 172.21.15.27:0/1723139659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5ec1599e0 0x7fa5ec0c9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.206+0000 7fa5f3bff700 1 -- 172.21.15.27:0/1723139659 >> 172.21.15.27:0/1723139659 conn(0x7fa5ec0680e0 msgr2=0x7fa5ec06e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:08.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.206+0000 7fa5f3bff700 1 -- 172.21.15.27:0/1723139659 shutdown_connections 2024-01-11T09:45:08.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.206+0000 7fa5f3bff700 1 -- 172.21.15.27:0/1723139659 wait complete. 2024-01-11T09:45:08.199 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 102 2024-01-11T09:45:08.216 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:45:08.238 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:08.246+0000 7f1b315a81c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:45:08.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.437+0000 7ff9ed5ff700 1 Processor -- start 2024-01-11T09:45:08.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.437+0000 7ff9ed5ff700 1 -- start start 2024-01-11T09:45:08.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.438+0000 7ff9ed5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9e80b2b90 0x7ff9e8159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:08.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.438+0000 7ff9ed5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9e80b07b0 0x7ff9e8159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:08.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.438+0000 7ff9ed5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e80cd730 con 0x7ff9e80b03e0 2024-01-11T09:45:08.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.438+0000 7ff9ed5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e80ccb20 con 0x7ff9e80b2b90 2024-01-11T09:45:08.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.438+0000 7ff9ed5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e80cce00 con 0x7ff9e80b07b0 2024-01-11T09:45:08.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.438+0000 7ff9e67fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9e80b2b90 0x7ff9e8159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:08.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.439+0000 7ff9e67fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9e80b2b90 0x7ff9e8159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60800/0 (socket says 172.21.15.27:60800) 2024-01-11T09:45:08.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.439+0000 7ff9e67fc700 1 -- 172.21.15.27:0/2641380169 learned_addr learned my addr 172.21.15.27:0/2641380169 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:08.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.439+0000 7ff9e77fe700 1 --2- 172.21.15.27:0/2641380169 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9e80b07b0 0x7ff9e8159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:08.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.439+0000 7ff9e77fe700 1 -- 172.21.15.27:0/2641380169 >> v2:172.21.15.27:3300/0 conn(0x7ff9e80b2b90 msgr2=0x7ff9e8159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:08.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.439+0000 7ff9e77fe700 1 --2- 172.21.15.27:0/2641380169 >> v2:172.21.15.27:3300/0 conn(0x7ff9e80b2b90 0x7ff9e8159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.439+0000 7ff9e77fe700 1 -- 172.21.15.27:0/2641380169 >> v1:172.21.15.77:6789/0 conn(0x7ff9e80b03e0 legacy=0x7ff9e80b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:08.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.439+0000 7ff9e77fe700 1 -- 172.21.15.27:0/2641380169 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e80cd0e0 con 0x7ff9e80b07b0 2024-01-11T09:45:08.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.439+0000 7ff9e5ffb700 1 -- 172.21.15.27:0/2641380169 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401635383 0 0) 0x7ff9e80cd730 con 0x7ff9e80b03e0 2024-01-11T09:45:08.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.439+0000 7ff9e77fe700 1 --2- 172.21.15.27:0/2641380169 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9e80b07b0 0x7ff9e8159740 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0009e60 tx=0x7ff9e0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1bf96e34b9b8050 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.440+0000 7ff9e5ffb700 1 -- 172.21.15.27:0/2641380169 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e000e070 con 0x7ff9e80b07b0 2024-01-11T09:45:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.440+0000 7ff9e5ffb700 1 -- 172.21.15.27:0/2641380169 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e000bb60 con 0x7ff9e80b07b0 2024-01-11T09:45:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.440+0000 7ff9e5ffb700 1 -- 172.21.15.27:0/2641380169 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0012430 con 0x7ff9e80b07b0 2024-01-11T09:45:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.440+0000 7ff9ed5ff700 1 -- 172.21.15.27:0/2641380169 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9e80b07b0 msgr2=0x7ff9e8159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.440+0000 7ff9ed5ff700 1 --2- 172.21.15.27:0/2641380169 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9e80b07b0 0x7ff9e8159740 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0009e60 tx=0x7ff9e0009780 comp rx=0 tx=0).stop 2024-01-11T09:45:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.440+0000 7ff9ed5ff700 1 -- 172.21.15.27:0/2641380169 shutdown_connections 2024-01-11T09:45:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.440+0000 7ff9ed5ff700 1 --2- 172.21.15.27:0/2641380169 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9e80b07b0 0x7ff9e8159740 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.440+0000 7ff9ed5ff700 1 --2- 172.21.15.27:0/2641380169 >> v2:172.21.15.27:3300/0 conn(0x7ff9e80b2b90 0x7ff9e8159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.440+0000 7ff9ed5ff700 1 -- 172.21.15.27:0/2641380169 >> 172.21.15.27:0/2641380169 conn(0x7ff9e806ed40 msgr2=0x7ff9e806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:08.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.440+0000 7ff9ed5ff700 1 -- 172.21.15.27:0/2641380169 shutdown_connections 2024-01-11T09:45:08.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.441+0000 7ff9ed5ff700 1 -- 172.21.15.27:0/2641380169 wait complete. 2024-01-11T09:45:08.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.442+0000 7ff9ed5ff700 1 Processor -- start 2024-01-11T09:45:08.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.442+0000 7ff9ed5ff700 1 -- start start 2024-01-11T09:45:08.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.443+0000 7ff9ed5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e80b03e0 0x7ff9e80c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:08.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.443+0000 7ff9ed5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9e80b07b0 0x7ff9e816a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:08.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.443+0000 7ff9ed5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e80b2b90 0x7ff9e8158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:08.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.443+0000 7ff9ed5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e816a760 con 0x7ff9e80b03e0 2024-01-11T09:45:08.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.443+0000 7ff9ed5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e80c88b0 con 0x7ff9e80b2b90 2024-01-11T09:45:08.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.443+0000 7ff9ed5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e8159080 con 0x7ff9e80b07b0 2024-01-11T09:45:08.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.443+0000 7ff9e6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e80b03e0 0x7ff9e80c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:08.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.443+0000 7ff9e6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e80b03e0 0x7ff9e80c75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60812/0 (socket says 172.21.15.27:60812) 2024-01-11T09:45:08.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.443+0000 7ff9e6ffd700 1 -- 172.21.15.27:0/698271002 learned_addr learned my addr 172.21.15.27:0/698271002 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:08.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.443+0000 7ff9e67fc700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9e80b07b0 0x7ff9e816a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:08.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.444+0000 7ff9e77fe700 1 --2- 172.21.15.27:0/698271002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e80b2b90 0x7ff9e8158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:08.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.444+0000 7ff9e6ffd700 1 -- 172.21.15.27:0/698271002 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9e80b07b0 msgr2=0x7ff9e816a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:08.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.444+0000 7ff9e6ffd700 1 --2- 172.21.15.27:0/698271002 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9e80b07b0 0x7ff9e816a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.444+0000 7ff9e6ffd700 1 -- 172.21.15.27:0/698271002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e80b2b90 msgr2=0x7ff9e8158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:08.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.444+0000 7ff9e6ffd700 1 --2- 172.21.15.27:0/698271002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e80b2b90 0x7ff9e8158b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.444+0000 7ff9e6ffd700 1 -- 172.21.15.27:0/698271002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e0009a20 con 0x7ff9e80b03e0 2024-01-11T09:45:08.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.444+0000 7ff9e67fc700 1 --2- 172.21.15.27:0/698271002 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9e80b07b0 0x7ff9e816a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:08.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.444+0000 7ff9e77fe700 1 --2- 172.21.15.27:0/698271002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e80b2b90 0x7ff9e8158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:08.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.444+0000 7ff9e6ffd700 1 --2- 172.21.15.27:0/698271002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e80b03e0 0x7ff9e80c75c0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800c7e0 tx=0x7ff9d8007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:08.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.445+0000 7ff9ceffd700 1 -- 172.21.15.27:0/698271002 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d80105c0 con 0x7ff9e80b03e0 2024-01-11T09:45:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.445+0000 7ff9ceffd700 1 -- 172.21.15.27:0/698271002 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d8010740 con 0x7ff9e80b03e0 2024-01-11T09:45:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.445+0000 7ff9ceffd700 1 -- 172.21.15.27:0/698271002 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8010b00 con 0x7ff9e80b03e0 2024-01-11T09:45:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.445+0000 7ff9ed5ff700 1 -- 172.21.15.27:0/698271002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e80c8d40 con 0x7ff9e80b03e0 2024-01-11T09:45:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.445+0000 7ff9ed5ff700 1 -- 172.21.15.27:0/698271002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e80c7f80 con 0x7ff9e80b03e0 2024-01-11T09:45:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.446+0000 7ff9ed5ff700 1 -- 172.21.15.27:0/698271002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e80718d0 con 0x7ff9e80b03e0 2024-01-11T09:45:08.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.447+0000 7ff9ceffd700 1 -- 172.21.15.27:0/698271002 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 123) v1 ==== 84958+0+0 (secure 0 0 0) 0x7ff9d801b3b0 con 0x7ff9e80b03e0 2024-01-11T09:45:08.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.448+0000 7ff9ceffd700 1 --2- 172.21.15.27:0/698271002 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff9d4067170 0x7ff9d4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:08.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.448+0000 7ff9ceffd700 1 -- 172.21.15.27:0/698271002 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6161+0+0 (secure 0 0 0) 0x7ff9d8084010 con 0x7ff9e80b03e0 2024-01-11T09:45:08.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.448+0000 7ff9e67fc700 1 -- 172.21.15.27:0/698271002 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff9d4067170 msgr2=0x7ff9d4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:08.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.449+0000 7ff9e67fc700 1 --2- 172.21.15.27:0/698271002 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff9d4067170 0x7ff9d4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:45:08.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.453+0000 7ff9ceffd700 1 -- 172.21.15.27:0/698271002 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff9d8050e20 con 0x7ff9e80b03e0 2024-01-11T09:45:08.487 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:08.495+0000 7f89e21421c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:45:08.581 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:08.590+0000 7f89e21421c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:45:08.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.650+0000 7ff9e67fc700 1 -- 172.21.15.27:0/698271002 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff9d4067170 msgr2=0x7ff9d4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:08.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.650+0000 7ff9e67fc700 1 --2- 172.21.15.27:0/698271002 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff9d4067170 0x7ff9d4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:45:08.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.705+0000 7ff9ed5ff700 1 -- 172.21.15.27:0/698271002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9e80cd3c0 con 0x7ff9e80b03e0 2024-01-11T09:45:08.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.706+0000 7ff9ceffd700 1 -- 172.21.15.27:0/698271002 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+4603 (secure 0 0 0) 0x7ff9d8017020 con 0x7ff9e80b03e0 2024-01-11T09:45:08.698 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:08.698 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":102,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8441,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/991606883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":991606883},{"type":"v1","addr":"172.21.15.77:6834","nonce":991606883}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1996818074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1996818074},{"type":"v1","addr":"172.21.15.77:6835","nonce":1996818074}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9742,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4117588854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":4117588854},{"type":"v1","addr":"172.21.15.27:6835","nonce":4117588854}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:43:41.554157+0000","modified":"2024-01-11T09:43:45.078369+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10113":{"gid":10113,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/1557605486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1557605486},{"type":"v1","addr":"172.21.15.27:6834","nonce":1557605486}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2024-01-11T09:45:08.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.708+0000 7ff9ed5ff700 1 -- 172.21.15.27:0/698271002 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff9d4067170 msgr2=0x7ff9d4069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:45:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.708+0000 7ff9ed5ff700 1 --2- 172.21.15.27:0/698271002 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff9d4067170 0x7ff9d4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.709+0000 7ff9ed5ff700 1 -- 172.21.15.27:0/698271002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e80b03e0 msgr2=0x7ff9e80c75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.709+0000 7ff9ed5ff700 1 --2- 172.21.15.27:0/698271002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e80b03e0 0x7ff9e80c75c0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800c7e0 tx=0x7ff9d8007df0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.709+0000 7ff9ed5ff700 1 -- 172.21.15.27:0/698271002 shutdown_connections 2024-01-11T09:45:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.709+0000 7ff9ed5ff700 1 --2- 172.21.15.27:0/698271002 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff9d4067170 0x7ff9d4069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.709+0000 7ff9ed5ff700 1 --2- 172.21.15.27:0/698271002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e80b2b90 0x7ff9e8158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.709+0000 7ff9ed5ff700 1 --2- 172.21.15.27:0/698271002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e80b03e0 0x7ff9e80c75c0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.709+0000 7ff9ed5ff700 1 --2- 172.21.15.27:0/698271002 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff9e80b07b0 0x7ff9e816a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.709+0000 7ff9ed5ff700 1 -- 172.21.15.27:0/698271002 >> 172.21.15.27:0/698271002 conn(0x7ff9e806ed40 msgr2=0x7ff9e80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.709+0000 7ff9ed5ff700 1 -- 172.21.15.27:0/698271002 shutdown_connections 2024-01-11T09:45:08.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.709+0000 7ff9ed5ff700 1 -- 172.21.15.27:0/698271002 wait complete. 2024-01-11T09:45:08.702 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 102 2024-01-11T09:45:08.718 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:45:08.781 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:08.789+0000 7f1b315a81c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:45:08.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.929+0000 7f8c813ff700 1 Processor -- start 2024-01-11T09:45:08.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.929+0000 7f8c813ff700 1 -- start start 2024-01-11T09:45:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.930+0000 7f8c813ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c7c159db0 0x7f8c7c15a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.930+0000 7f8c813ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c7c14fc20 0x7f8c7c150080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.930+0000 7f8c813ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7c157e30 con 0x7f8c7c1599e0 2024-01-11T09:45:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.930+0000 7f8c813ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c7c157500 con 0x7f8c7c14fc20 2024-01-11T09:45:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.930+0000 7f8c813ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c1577e0 con 0x7f8c7c159db0 2024-01-11T09:45:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.931+0000 7f8c7b7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c7c14fc20 0x7f8c7c150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.931+0000 7f8c7b7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c7c14fc20 0x7f8c7c150080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60816/0 (socket says 172.21.15.27:60816) 2024-01-11T09:45:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.931+0000 7f8c7b7fe700 1 -- 172.21.15.27:0/894637394 learned_addr learned my addr 172.21.15.27:0/894637394 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.931+0000 7f8c7a7fc700 1 --2- 172.21.15.27:0/894637394 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c7c159db0 0x7f8c7c15a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.931+0000 7f8c7b7fe700 1 -- 172.21.15.27:0/894637394 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c7c159db0 msgr2=0x7f8c7c15a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.931+0000 7f8c7b7fe700 1 --2- 172.21.15.27:0/894637394 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c7c159db0 0x7f8c7c15a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.931+0000 7f8c7b7fe700 1 -- 172.21.15.27:0/894637394 >> v1:172.21.15.77:6789/0 conn(0x7f8c7c1599e0 legacy=0x7f8c7c0b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:08.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.931+0000 7f8c7b7fe700 1 -- 172.21.15.27:0/894637394 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c157ac0 con 0x7f8c7c14fc20 2024-01-11T09:45:08.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.931+0000 7f8c7b7fe700 1 --2- 172.21.15.27:0/894637394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c14fc20 0x7f8c7c150080 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400d1a0 tx=0x7f8c7400ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ace32527b5aeb272 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:08.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.932+0000 7f8c79ffb700 1 -- 172.21.15.27:0/894637394 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c74010070 con 0x7f8c7c14fc20 2024-01-11T09:45:08.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.932+0000 7f8c79ffb700 1 -- 172.21.15.27:0/894637394 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7400dc00 con 0x7f8c7c14fc20 2024-01-11T09:45:08.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.932+0000 7f8c79ffb700 1 -- 172.21.15.27:0/894637394 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c74016400 con 0x7f8c7c14fc20 2024-01-11T09:45:08.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.932+0000 7f8c813ff700 1 -- 172.21.15.27:0/894637394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c14fc20 msgr2=0x7f8c7c150080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:08.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.932+0000 7f8c813ff700 1 --2- 172.21.15.27:0/894637394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c14fc20 0x7f8c7c150080 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400d1a0 tx=0x7f8c7400ba50 comp rx=0 tx=0).stop 2024-01-11T09:45:08.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.932+0000 7f8c813ff700 1 -- 172.21.15.27:0/894637394 shutdown_connections 2024-01-11T09:45:08.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.932+0000 7f8c813ff700 1 --2- 172.21.15.27:0/894637394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c14fc20 0x7f8c7c150080 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.932+0000 7f8c813ff700 1 --2- 172.21.15.27:0/894637394 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c7c159db0 0x7f8c7c15a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.932+0000 7f8c813ff700 1 -- 172.21.15.27:0/894637394 >> 172.21.15.27:0/894637394 conn(0x7f8c7c0680e0 msgr2=0x7f8c7c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:08.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.932+0000 7f8c813ff700 1 -- 172.21.15.27:0/894637394 shutdown_connections 2024-01-11T09:45:08.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.932+0000 7f8c813ff700 1 -- 172.21.15.27:0/894637394 wait complete. 2024-01-11T09:45:08.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.934+0000 7f8c813ff700 1 Processor -- start 2024-01-11T09:45:08.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.934+0000 7f8c813ff700 1 -- start start 2024-01-11T09:45:08.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.935+0000 7f8c813ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c14fc20 0x7f8c7c166e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:08.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.935+0000 7f8c813ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c7c1599e0 0x7f8c7c14b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:08.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.935+0000 7f8c813ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c7c159db0 0x7f8c7c150810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:08.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.935+0000 7f8c813ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0acac0 con 0x7f8c7c14fc20 2024-01-11T09:45:08.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.935+0000 7f8c813ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0ac190 con 0x7f8c7c1599e0 2024-01-11T09:45:08.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.935+0000 7f8c813ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0ac470 con 0x7f8c7c159db0 2024-01-11T09:45:08.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.935+0000 7f8c7affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c14fc20 0x7f8c7c166e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:08.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.935+0000 7f8c7a7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c7c1599e0 0x7f8c7c14b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:08.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.935+0000 7f8c7affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c14fc20 0x7f8c7c166e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60822/0 (socket says 172.21.15.27:60822) 2024-01-11T09:45:08.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.935+0000 7f8c7affd700 1 -- 172.21.15.27:0/3356663571 learned_addr learned my addr 172.21.15.27:0/3356663571 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:08.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.935+0000 7f8c7b7fe700 1 --2- 172.21.15.27:0/3356663571 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c7c159db0 0x7f8c7c150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:08.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.936+0000 7f8c7affd700 1 -- 172.21.15.27:0/3356663571 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c7c159db0 msgr2=0x7f8c7c150810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:08.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.936+0000 7f8c7affd700 1 --2- 172.21.15.27:0/3356663571 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c7c159db0 0x7f8c7c150810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.936+0000 7f8c7affd700 1 -- 172.21.15.27:0/3356663571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c7c1599e0 msgr2=0x7f8c7c14b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:08.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.936+0000 7f8c7affd700 1 --2- 172.21.15.27:0/3356663571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c7c1599e0 0x7f8c7c14b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:08.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.936+0000 7f8c7affd700 1 -- 172.21.15.27:0/3356663571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7400bcf0 con 0x7f8c7c14fc20 2024-01-11T09:45:08.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.936+0000 7f8c7a7fc700 1 --2- 172.21.15.27:0/3356663571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c7c1599e0 0x7f8c7c14b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:08.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.936+0000 7f8c7affd700 1 --2- 172.21.15.27:0/3356663571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c14fc20 0x7f8c7c166e00 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00c590 tx=0x7f8c6c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:08.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.936+0000 7f8c62ffd700 1 -- 172.21.15.27:0/3356663571 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c011590 con 0x7f8c7c14fc20 2024-01-11T09:45:08.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.936+0000 7f8c62ffd700 1 -- 172.21.15.27:0/3356663571 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6c011710 con 0x7f8c7c14fc20 2024-01-11T09:45:08.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.936+0000 7f8c62ffd700 1 -- 172.21.15.27:0/3356663571 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c011ad0 con 0x7f8c7c14fc20 2024-01-11T09:45:08.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.937+0000 7f8c813ff700 1 -- 172.21.15.27:0/3356663571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7c0ac750 con 0x7f8c7c14fc20 2024-01-11T09:45:08.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.937+0000 7f8c813ff700 1 -- 172.21.15.27:0/3356663571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7c14bdf0 con 0x7f8c7c14fc20 2024-01-11T09:45:08.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.938+0000 7f8c813ff700 1 -- 172.21.15.27:0/3356663571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7c0718d0 con 0x7f8c7c14fc20 2024-01-11T09:45:08.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.940+0000 7f8c62ffd700 1 -- 172.21.15.27:0/3356663571 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 123) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f8c6c01b3b0 con 0x7f8c7c14fc20 2024-01-11T09:45:08.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.941+0000 7f8c62ffd700 1 --2- 172.21.15.27:0/3356663571 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8c68067170 0x7f8c68069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.941+0000 7f8c62ffd700 1 -- 172.21.15.27:0/3356663571 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f8c6c083ed0 con 0x7f8c7c14fc20 2024-01-11T09:45:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.941+0000 7f8c7a7fc700 1 -- 172.21.15.27:0/3356663571 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8c68067170 msgr2=0x7f8c68069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.941+0000 7f8c7a7fc700 1 --2- 172.21.15.27:0/3356663571 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8c68067170 0x7f8c68069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:45:08.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:08.945+0000 7f8c62ffd700 1 -- 172.21.15.27:0/3356663571 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8c6c050ce0 con 0x7f8c7c14fc20 2024-01-11T09:45:08.970 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:08.978+0000 7f1b315a81c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:45:09.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.142+0000 7f8c7a7fc700 1 -- 172.21.15.27:0/3356663571 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8c68067170 msgr2=0x7f8c68069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:09.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.142+0000 7f8c7a7fc700 1 --2- 172.21.15.27:0/3356663571 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8c68067170 0x7f8c68069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:45:09.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.157+0000 7f8c813ff700 1 -- 172.21.15.27:0/3356663571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c7c1505c0 con 0x7f8c7c14fc20 2024-01-11T09:45:09.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.158+0000 7f8c62ffd700 1 -- 172.21.15.27:0/3356663571 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v202) v1 ==== 74+0+16027 (secure 0 0 0) 0x7f8c6c017070 con 0x7f8c7c14fc20 2024-01-11T09:45:09.150 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:09.150 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":202,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:45:06.040677+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:43:39.267819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-01-11T09:43:40.563877+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":202,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.26","mappings":[{"from":0,"to":1}]},{"pgid":"25.29","mappings":[{"from":0,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/3183215496":"2024-01-12T09:43:58.050773+0000","172.21.15.77:6837/59776":"2024-01-12T09:43:58.050773+0000","172.21.15.77:6836/59776":"2024-01-12T09:43:58.050773+0000","172.21.15.77:0/2374924114":"2024-01-12T09:43:58.050773+0000","172.21.15.77:0/1884196903":"2024-01-12T09:43:58.050773+0000","172.21.15.77:0/3082251031":"2024-01-12T09:43:58.050773+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:45:09.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.161+0000 7f8c813ff700 1 -- 172.21.15.27:0/3356663571 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8c68067170 msgr2=0x7f8c68069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:45:09.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.161+0000 7f8c813ff700 1 --2- 172.21.15.27:0/3356663571 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8c68067170 0x7f8c68069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.161+0000 7f8c813ff700 1 -- 172.21.15.27:0/3356663571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c14fc20 msgr2=0x7f8c7c166e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:09.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.161+0000 7f8c813ff700 1 --2- 172.21.15.27:0/3356663571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c14fc20 0x7f8c7c166e00 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00c590 tx=0x7f8c6c00c8a0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.161+0000 7f8c813ff700 1 -- 172.21.15.27:0/3356663571 shutdown_connections 2024-01-11T09:45:09.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.161+0000 7f8c813ff700 1 --2- 172.21.15.27:0/3356663571 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c7c159db0 0x7f8c7c150810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.161+0000 7f8c813ff700 1 --2- 172.21.15.27:0/3356663571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c14fc20 0x7f8c7c166e00 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.161+0000 7f8c813ff700 1 --2- 172.21.15.27:0/3356663571 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8c68067170 0x7f8c68069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.161+0000 7f8c813ff700 1 --2- 172.21.15.27:0/3356663571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c7c1599e0 0x7f8c7c14b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.161+0000 7f8c813ff700 1 -- 172.21.15.27:0/3356663571 >> 172.21.15.27:0/3356663571 conn(0x7f8c7c0680e0 msgr2=0x7f8c7c06e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:09.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.161+0000 7f8c813ff700 1 -- 172.21.15.27:0/3356663571 shutdown_connections 2024-01-11T09:45:09.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.161+0000 7f8c813ff700 1 -- 172.21.15.27:0/3356663571 wait complete. 2024-01-11T09:45:09.172 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-11T09:45:09.181 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:09.190+0000 7f89e21421c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:45:09.350 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:09.357+0000 7f1b315a81c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:45:09.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.388+0000 7f1db51ff700 1 Processor -- start 2024-01-11T09:45:09.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.388+0000 7f1db51ff700 1 -- start start 2024-01-11T09:45:09.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.389+0000 7f1db51ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1db0156ab0 0x7f1db00ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:09.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.389+0000 7f1db51ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1db00cc9b0 0x7f1db0156e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:09.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.389+0000 7f1db51ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db00adc80 con 0x7f1db01573c0 2024-01-11T09:45:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.389+0000 7f1db51ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1db00adf60 con 0x7f1db00cc9b0 2024-01-11T09:45:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.389+0000 7f1db51ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db00ae240 con 0x7f1db0156ab0 2024-01-11T09:45:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.390+0000 7f1dae59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1db00cc9b0 0x7f1db0156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.390+0000 7f1daf59e700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1db01573c0 0x7f1db00af540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46346/0 (socket says 172.21.15.27:46346) 2024-01-11T09:45:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.390+0000 7f1daf59e700 1 -- 172.21.15.27:0/169041573 learned_addr learned my addr 172.21.15.27:0/169041573 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.390+0000 7f1daed9d700 1 --2- 172.21.15.27:0/169041573 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1db0156ab0 0x7f1db00ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.390+0000 7f1dae59c700 1 -- 172.21.15.27:0/169041573 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1db0156ab0 msgr2=0x7f1db00ad650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.390+0000 7f1dae59c700 1 --2- 172.21.15.27:0/169041573 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1db0156ab0 0x7f1db00ad650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.390+0000 7f1dae59c700 1 -- 172.21.15.27:0/169041573 >> v1:172.21.15.27:6789/0 conn(0x7f1db01573c0 legacy=0x7f1db00af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.390+0000 7f1dae59c700 1 -- 172.21.15.27:0/169041573 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db00ae520 con 0x7f1db00cc9b0 2024-01-11T09:45:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.390+0000 7f1dadd9b700 1 -- 172.21.15.27:0/169041573 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084380935 0 0) 0x7f1db00adc80 con 0x7f1db01573c0 2024-01-11T09:45:09.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.391+0000 7f1dae59c700 1 --2- 172.21.15.27:0/169041573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db00cc9b0 0x7f1db0156e80 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f1da40098e0 tx=0x7f1da40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4395533c72c350e9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.391+0000 7f1dadd9b700 1 -- 172.21.15.27:0/169041573 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da400b640 con 0x7f1db00cc9b0 2024-01-11T09:45:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.391+0000 7f1dadd9b700 1 -- 172.21.15.27:0/169041573 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da400b7a0 con 0x7f1db00cc9b0 2024-01-11T09:45:09.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.391+0000 7f1dadd9b700 1 -- 172.21.15.27:0/169041573 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da400bb40 con 0x7f1db00cc9b0 2024-01-11T09:45:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.392+0000 7f1db51ff700 1 -- 172.21.15.27:0/169041573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db00cc9b0 msgr2=0x7f1db0156e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.392+0000 7f1db51ff700 1 --2- 172.21.15.27:0/169041573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db00cc9b0 0x7f1db0156e80 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f1da40098e0 tx=0x7f1da40091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.392+0000 7f1db51ff700 1 -- 172.21.15.27:0/169041573 shutdown_connections 2024-01-11T09:45:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.392+0000 7f1db51ff700 1 --2- 172.21.15.27:0/169041573 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1db0156ab0 0x7f1db00ad650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.392+0000 7f1db51ff700 1 --2- 172.21.15.27:0/169041573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db00cc9b0 0x7f1db0156e80 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.392+0000 7f1db51ff700 1 -- 172.21.15.27:0/169041573 >> 172.21.15.27:0/169041573 conn(0x7f1db00680b0 msgr2=0x7f1db00cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:09.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.392+0000 7f1db51ff700 1 -- 172.21.15.27:0/169041573 shutdown_connections 2024-01-11T09:45:09.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.392+0000 7f1db51ff700 1 -- 172.21.15.27:0/169041573 wait complete. 2024-01-11T09:45:09.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.393+0000 7f1db51ff700 1 Processor -- start 2024-01-11T09:45:09.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.394+0000 7f1db51ff700 1 -- start start 2024-01-11T09:45:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.394+0000 7f1db51ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1db00cc9b0 0x7f1db013fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.394+0000 7f1db51ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db0156ab0 0x7f1db006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.394+0000 7f1db51ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1db01573c0 0x7f1db0150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.394+0000 7f1db51ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db0140370 con 0x7f1db0156ab0 2024-01-11T09:45:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.394+0000 7f1db51ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db006e5c0 con 0x7f1db00cc9b0 2024-01-11T09:45:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.394+0000 7f1db51ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db006ef90 con 0x7f1db01573c0 2024-01-11T09:45:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.395+0000 7f1dae59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db0156ab0 0x7f1db006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.395+0000 7f1dae59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db0156ab0 0x7f1db006ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60838/0 (socket says 172.21.15.27:60838) 2024-01-11T09:45:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.395+0000 7f1dae59c700 1 -- 172.21.15.27:0/3548814922 learned_addr learned my addr 172.21.15.27:0/3548814922 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.395+0000 7f1daed9d700 1 --2- 172.21.15.27:0/3548814922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1db00cc9b0 0x7f1db013fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.395+0000 7f1daf59e700 1 --2- 172.21.15.27:0/3548814922 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1db01573c0 0x7f1db0150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.395+0000 7f1dae59c700 1 -- 172.21.15.27:0/3548814922 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1db01573c0 msgr2=0x7f1db0150270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.395+0000 7f1dae59c700 1 --2- 172.21.15.27:0/3548814922 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1db01573c0 0x7f1db0150270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.395+0000 7f1dae59c700 1 -- 172.21.15.27:0/3548814922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1db00cc9b0 msgr2=0x7f1db013fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.395+0000 7f1dae59c700 1 --2- 172.21.15.27:0/3548814922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1db00cc9b0 0x7f1db013fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.395+0000 7f1dae59c700 1 -- 172.21.15.27:0/3548814922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da4009420 con 0x7f1db0156ab0 2024-01-11T09:45:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.396+0000 7f1dae59c700 1 --2- 172.21.15.27:0/3548814922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db0156ab0 0x7f1db006ea50 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f1da4009f80 tx=0x7f1da4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.396+0000 7f1d96ffd700 1 -- 172.21.15.27:0/3548814922 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da400bb20 con 0x7f1db0156ab0 2024-01-11T09:45:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.396+0000 7f1d96ffd700 1 -- 172.21.15.27:0/3548814922 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da4000c90 con 0x7f1db0156ab0 2024-01-11T09:45:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.396+0000 7f1db51ff700 1 -- 172.21.15.27:0/3548814922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db006d9b0 con 0x7f1db0156ab0 2024-01-11T09:45:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.396+0000 7f1d96ffd700 1 -- 172.21.15.27:0/3548814922 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da4011630 con 0x7f1db0156ab0 2024-01-11T09:45:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.396+0000 7f1db51ff700 1 -- 172.21.15.27:0/3548814922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db006df70 con 0x7f1db0156ab0 2024-01-11T09:45:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.397+0000 7f1db51ff700 1 -- 172.21.15.27:0/3548814922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db006e750 con 0x7f1db0156ab0 2024-01-11T09:45:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.398+0000 7f1d96ffd700 1 -- 172.21.15.27:0/3548814922 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 123) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f1da40119a0 con 0x7f1db0156ab0 2024-01-11T09:45:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.399+0000 7f1d96ffd700 1 --2- 172.21.15.27:0/3548814922 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1d9c067170 0x7f1d9c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.399+0000 7f1d96ffd700 1 -- 172.21.15.27:0/3548814922 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f1da4085d70 con 0x7f1db0156ab0 2024-01-11T09:45:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.399+0000 7f1daed9d700 1 -- 172.21.15.27:0/3548814922 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1d9c067170 msgr2=0x7f1d9c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.399+0000 7f1daed9d700 1 --2- 172.21.15.27:0/3548814922 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1d9c067170 0x7f1d9c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:45:09.392 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:09.399+0000 7f89e21421c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:45:09.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.403+0000 7f1d96ffd700 1 -- 172.21.15.27:0/3548814922 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1da4052ae0 con 0x7f1db0156ab0 2024-01-11T09:45:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.600+0000 7f1daed9d700 1 -- 172.21.15.27:0/3548814922 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1d9c067170 msgr2=0x7f1d9c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.600+0000 7f1daed9d700 1 --2- 172.21.15.27:0/3548814922 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1d9c067170 0x7f1d9c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:45:09.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.653+0000 7f1db51ff700 1 -- 172.21.15.27:0/3548814922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f1db004c3c0 con 0x7f1db0156ab0 2024-01-11T09:45:09.646 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-11T09:45:09.654+0000 7f27065f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-11T09:45:09.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.660+0000 7f1d96ffd700 1 -- 172.21.15.27:0/3548814922 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v103) v1 ==== 161+0+0 (secure 0 0 0) 0x7f1da4056100 con 0x7f1db0156ab0 2024-01-11T09:45:09.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.664+0000 7f1db51ff700 1 -- 172.21.15.27:0/3548814922 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1d9c067170 msgr2=0x7f1d9c069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:45:09.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.664+0000 7f1db51ff700 1 --2- 172.21.15.27:0/3548814922 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1d9c067170 0x7f1d9c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.664+0000 7f1db51ff700 1 -- 172.21.15.27:0/3548814922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db0156ab0 msgr2=0x7f1db006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:09.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.664+0000 7f1db51ff700 1 --2- 172.21.15.27:0/3548814922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db0156ab0 0x7f1db006ea50 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f1da4009f80 tx=0x7f1da4017000 comp rx=0 tx=0).stop 2024-01-11T09:45:09.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.664+0000 7f1db51ff700 1 -- 172.21.15.27:0/3548814922 shutdown_connections 2024-01-11T09:45:09.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.664+0000 7f1db51ff700 1 --2- 172.21.15.27:0/3548814922 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1db01573c0 0x7f1db0150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.664+0000 7f1db51ff700 1 --2- 172.21.15.27:0/3548814922 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1d9c067170 0x7f1d9c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.664+0000 7f1db51ff700 1 --2- 172.21.15.27:0/3548814922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1db00cc9b0 0x7f1db013fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.664+0000 7f1db51ff700 1 --2- 172.21.15.27:0/3548814922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db0156ab0 0x7f1db006ea50 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.664+0000 7f1db51ff700 1 -- 172.21.15.27:0/3548814922 >> 172.21.15.27:0/3548814922 conn(0x7f1db00680b0 msgr2=0x7f1db00af440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:09.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.664+0000 7f1db51ff700 1 -- 172.21.15.27:0/3548814922 shutdown_connections 2024-01-11T09:45:09.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.664+0000 7f1db51ff700 1 -- 172.21.15.27:0/3548814922 wait complete. 2024-01-11T09:45:09.657 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-11T09:45:09.683 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-11T09:45:09.715 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T09:45:09.779 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:09.786+0000 7f1b315a81c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:45:09.812 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:09.821+0000 7f89e21421c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:45:09.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.937+0000 7f775dfff700 1 Processor -- start 2024-01-11T09:45:09.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.937+0000 7f775dfff700 1 -- start start 2024-01-11T09:45:09.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.938+0000 7f775dfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7758157410 0x7f77580ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:09.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.938+0000 7f775dfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7758069540 0x7f77580c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:09.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.938+0000 7f775dfff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77580699d0 con 0x7f77581577e0 2024-01-11T09:45:09.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.938+0000 7f775dfff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7758069cb0 con 0x7f7758157410 2024-01-11T09:45:09.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.939+0000 7f775dfff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7758069f90 con 0x7f7758069540 2024-01-11T09:45:09.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.939+0000 7f77577fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7758157410 0x7f77580ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:09.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.939+0000 7f7756ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f77581577e0 0x7f77580ae000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46368/0 (socket says 172.21.15.27:46368) 2024-01-11T09:45:09.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.939+0000 7f77577fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7758157410 0x7f77580ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60870/0 (socket says 172.21.15.27:60870) 2024-01-11T09:45:09.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.939+0000 7f7756ffd700 1 -- 172.21.15.27:0/1902221779 learned_addr learned my addr 172.21.15.27:0/1902221779 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:09.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.939+0000 7f7757fff700 1 --2- 172.21.15.27:0/1902221779 >> v2:172.21.15.77:3300/0 conn(0x7f7758069540 0x7f77580c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:09.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.939+0000 7f77577fe700 1 -- 172.21.15.27:0/1902221779 >> v2:172.21.15.77:3300/0 conn(0x7f7758069540 msgr2=0x7f77580c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:09.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.939+0000 7f77577fe700 1 --2- 172.21.15.27:0/1902221779 >> v2:172.21.15.77:3300/0 conn(0x7f7758069540 0x7f77580c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.939+0000 7f77577fe700 1 -- 172.21.15.27:0/1902221779 >> v1:172.21.15.27:6789/0 conn(0x7f77581577e0 legacy=0x7f77580ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:09.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.939+0000 7f77577fe700 1 -- 172.21.15.27:0/1902221779 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775806a270 con 0x7f7758157410 2024-01-11T09:45:09.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.940+0000 7f77577fe700 1 --2- 172.21.15.27:0/1902221779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7758157410 0x7f77580ccae0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f775806d030 tx=0x7f7748009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=395edb979fcb0016 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:09.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.940+0000 7f77567fc700 1 -- 172.21.15.27:0/1902221779 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7748005c10 con 0x7f7758157410 2024-01-11T09:45:09.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.940+0000 7f77567fc700 1 -- 172.21.15.27:0/1902221779 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7748005d70 con 0x7f7758157410 2024-01-11T09:45:09.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.940+0000 7f77567fc700 1 -- 172.21.15.27:0/1902221779 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7748006260 con 0x7f7758157410 2024-01-11T09:45:09.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.941+0000 7f775dfff700 1 -- 172.21.15.27:0/1902221779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7758157410 msgr2=0x7f77580ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:09.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.941+0000 7f775dfff700 1 --2- 172.21.15.27:0/1902221779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7758157410 0x7f77580ccae0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f775806d030 tx=0x7f7748009120 comp rx=0 tx=0).stop 2024-01-11T09:45:09.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.941+0000 7f775dfff700 1 -- 172.21.15.27:0/1902221779 shutdown_connections 2024-01-11T09:45:09.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.941+0000 7f775dfff700 1 --2- 172.21.15.27:0/1902221779 >> v2:172.21.15.77:3300/0 conn(0x7f7758069540 0x7f77580c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.941+0000 7f775dfff700 1 --2- 172.21.15.27:0/1902221779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7758157410 0x7f77580ccae0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.941+0000 7f775dfff700 1 -- 172.21.15.27:0/1902221779 >> 172.21.15.27:0/1902221779 conn(0x7f77580680a0 msgr2=0x7f77580cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.941+0000 7f775dfff700 1 -- 172.21.15.27:0/1902221779 shutdown_connections 2024-01-11T09:45:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.941+0000 7f775dfff700 1 -- 172.21.15.27:0/1902221779 wait complete. 2024-01-11T09:45:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.943+0000 7f775dfff700 1 Processor -- start 2024-01-11T09:45:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.943+0000 7f775dfff700 1 -- start start 2024-01-11T09:45:09.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.944+0000 7f775dfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7758069540 0x7f775814f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:09.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.944+0000 7f775dfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7758157410 0x7f775815a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:09.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.944+0000 7f775dfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77581577e0 0x7f7758167790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:09.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.944+0000 7f775dfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775814c810 con 0x7f77581577e0 2024-01-11T09:45:09.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.944+0000 7f775dfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77581489f0 con 0x7f7758157410 2024-01-11T09:45:09.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.944+0000 7f775dfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77580c8a60 con 0x7f7758069540 2024-01-11T09:45:09.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.944+0000 7f7757fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77581577e0 0x7f7758167790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:09.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.944+0000 7f7757fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77581577e0 0x7f7758167790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60886/0 (socket says 172.21.15.27:60886) 2024-01-11T09:45:09.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.944+0000 7f7757fff700 1 -- 172.21.15.27:0/3898583731 learned_addr learned my addr 172.21.15.27:0/3898583731 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.944+0000 7f77577fe700 1 --2- 172.21.15.27:0/3898583731 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7758069540 0x7f775814f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.944+0000 7f7756ffd700 1 --2- 172.21.15.27:0/3898583731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7758157410 0x7f775815a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.944+0000 7f7757fff700 1 -- 172.21.15.27:0/3898583731 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7758069540 msgr2=0x7f775814f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.944+0000 7f7757fff700 1 --2- 172.21.15.27:0/3898583731 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7758069540 0x7f775814f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.944+0000 7f7757fff700 1 -- 172.21.15.27:0/3898583731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7758157410 msgr2=0x7f775815a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.944+0000 7f7757fff700 1 --2- 172.21.15.27:0/3898583731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7758157410 0x7f775815a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.944+0000 7f7757fff700 1 -- 172.21.15.27:0/3898583731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77480093c0 con 0x7f77581577e0 2024-01-11T09:45:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.944+0000 7f77577fe700 1 --2- 172.21.15.27:0/3898583731 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7758069540 0x7f775814f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.944+0000 7f7756ffd700 1 --2- 172.21.15.27:0/3898583731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7758157410 0x7f775815a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.945+0000 7f7757fff700 1 --2- 172.21.15.27:0/3898583731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77581577e0 0x7f7758167790 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f7750009df0 tx=0x7f775000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.945+0000 7f773f7fe700 1 -- 172.21.15.27:0/3898583731 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775000cb50 con 0x7f77581577e0 2024-01-11T09:45:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.945+0000 7f773f7fe700 1 -- 172.21.15.27:0/3898583731 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f775000f040 con 0x7f77581577e0 2024-01-11T09:45:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.945+0000 7f773f7fe700 1 -- 172.21.15.27:0/3898583731 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7750014480 con 0x7f77581577e0 2024-01-11T09:45:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.945+0000 7f775dfff700 1 -- 172.21.15.27:0/3898583731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77580c8d40 con 0x7f77581577e0 2024-01-11T09:45:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.945+0000 7f775dfff700 1 -- 172.21.15.27:0/3898583731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f775814ca60 con 0x7f77581577e0 2024-01-11T09:45:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.946+0000 7f775dfff700 1 -- 172.21.15.27:0/3898583731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f775804c3c0 con 0x7f77581577e0 2024-01-11T09:45:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.947+0000 7f773f7fe700 1 -- 172.21.15.27:0/3898583731 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 123) v1 ==== 84958+0+0 (secure 0 0 0) 0x7f7750014830 con 0x7f77581577e0 2024-01-11T09:45:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.948+0000 7f773f7fe700 1 --2- 172.21.15.27:0/3898583731 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7744067170 0x7f7744069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.948+0000 7f773f7fe700 1 -- 172.21.15.27:0/3898583731 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6219+0+0 (secure 0 0 0) 0x7f77500858b0 con 0x7f77581577e0 2024-01-11T09:45:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.948+0000 7f77577fe700 1 -- 172.21.15.27:0/3898583731 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7744067170 msgr2=0x7f7744069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.948+0000 7f77577fe700 1 --2- 172.21.15.27:0/3898583731 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7744067170 0x7f7744069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:45:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:09.952+0000 7f773f7fe700 1 -- 172.21.15.27:0/3898583731 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f77500524e0 con 0x7f77581577e0 2024-01-11T09:45:09.963 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:09.970+0000 7f1b315a81c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:45:10.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.149+0000 7f77577fe700 1 -- 172.21.15.27:0/3898583731 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7744067170 msgr2=0x7f7744069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:10.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.149+0000 7f77577fe700 1 --2- 172.21.15.27:0/3898583731 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7744067170 0x7f7744069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:45:10.156 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:10.164+0000 7f1b315a81c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:45:10.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.209+0000 7f775dfff700 1 -- 172.21.15.27:0/3898583731 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f775814f600 con 0x7f77581577e0 2024-01-11T09:45:10.265 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:10.273+0000 7f1b315a81c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:45:10.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.288+0000 7f773f7fe700 1 -- 172.21.15.27:0/3898583731 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 124) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f77500528b0 con 0x7f77581577e0 2024-01-11T09:45:10.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.288+0000 7f773f7fe700 1 -- 172.21.15.27:0/3898583731 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7744067170 msgr2=0x7f7744069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:45:10.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.288+0000 7f773f7fe700 1 --2- 172.21.15.27:0/3898583731 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7744067170 0x7f7744069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:10.286 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:10.294+0000 7f89e21421c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:45:10.344 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:10.351+0000 7f1aac444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:10.344 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:10.351+0000 7f1aac444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:10.344 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:10.351+0000 7f1aac444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:10.345 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:10.351+0000 7f1aac444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:10.345 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:10.351+0000 7f1aac444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:10.345 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:10.352+0000 7f1aaf44a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:10.345 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:10.352+0000 7f1aaf44a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:10.345 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:10.352+0000 7f1aaf44a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:10.345 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:10.352+0000 7f1aaf44a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:10.345 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:10.352+0000 7f1aaf44a700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:10.482 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:10.491+0000 7f89e21421c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:45:10.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.660+0000 7f773f7fe700 1 -- 172.21.15.27:0/3898583731 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v104) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7750071460 con 0x7f77581577e0 2024-01-11T09:45:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.663+0000 7f775dfff700 1 -- 172.21.15.27:0/3898583731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77581577e0 msgr2=0x7f7758167790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.663+0000 7f775dfff700 1 --2- 172.21.15.27:0/3898583731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77581577e0 0x7f7758167790 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f7750009df0 tx=0x7f775000b040 comp rx=0 tx=0).stop 2024-01-11T09:45:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.663+0000 7f775dfff700 1 -- 172.21.15.27:0/3898583731 shutdown_connections 2024-01-11T09:45:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.663+0000 7f775dfff700 1 --2- 172.21.15.27:0/3898583731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77581577e0 0x7f7758167790 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.663+0000 7f775dfff700 1 --2- 172.21.15.27:0/3898583731 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7758069540 0x7f775814f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.663+0000 7f775dfff700 1 --2- 172.21.15.27:0/3898583731 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7744067170 0x7f7744069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.663+0000 7f775dfff700 1 --2- 172.21.15.27:0/3898583731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7758157410 0x7f775815a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.663+0000 7f775dfff700 1 -- 172.21.15.27:0/3898583731 >> 172.21.15.27:0/3898583731 conn(0x7f77580680a0 msgr2=0x7f77580af500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.663+0000 7f775dfff700 1 -- 172.21.15.27:0/3898583731 shutdown_connections 2024-01-11T09:45:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.663+0000 7f775dfff700 1 -- 172.21.15.27:0/3898583731 wait complete. 2024-01-11T09:45:10.678 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-11T09:45:10.690 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:10.698+0000 7f89e21421c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:45:10.807 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:10.816+0000 7f89e21421c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:45:10.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.892+0000 7fb438bff700 1 Processor -- start 2024-01-11T09:45:10.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.892+0000 7fb438bff700 1 -- start start 2024-01-11T09:45:10.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.893+0000 7fb438bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb434159e60 0x7fb43415aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:10.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.893+0000 7fb438bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb43415c4b0 con 0x7fb43415b400 2024-01-11T09:45:10.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.893+0000 7fb438bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb43415d050 con 0x7fb43415a7a0 2024-01-11T09:45:10.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.893+0000 7fb438bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb43415b8a0 con 0x7fb434159e60 2024-01-11T09:45:10.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.893+0000 7fb431d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb434159e60 0x7fb43415aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:10.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.893+0000 7fb432d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb43415b400 0x7fb4340685e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46378/0 (socket says 172.21.15.27:46378) 2024-01-11T09:45:10.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.893+0000 7fb432d9d700 1 -- 172.21.15.27:0/2109151882 learned_addr learned my addr 172.21.15.27:0/2109151882 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:10.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.894+0000 7fb431d9b700 1 -- 172.21.15.27:0/2109151882 >> v1:172.21.15.77:6789/0 conn(0x7fb43415a7a0 legacy=0x7fb43415c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:10.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.894+0000 7fb431d9b700 1 -- 172.21.15.27:0/2109151882 >> v1:172.21.15.27:6789/0 conn(0x7fb43415b400 legacy=0x7fb4340685e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:10.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.894+0000 7fb431d9b700 1 -- 172.21.15.27:0/2109151882 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43415bb80 con 0x7fb434159e60 2024-01-11T09:45:10.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.894+0000 7fb43159a700 1 -- 172.21.15.27:0/2109151882 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671550913 0 0) 0x7fb43415d050 con 0x7fb43415a7a0 2024-01-11T09:45:10.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.894+0000 7fb431d9b700 1 --2- 172.21.15.27:0/2109151882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb434159e60 0x7fb43415aec0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fb4280098e0 tx=0x7fb4280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f16a5e17370b600 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:10.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.894+0000 7fb43159a700 1 -- 172.21.15.27:0/2109151882 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb428009e80 con 0x7fb434159e60 2024-01-11T09:45:10.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.895+0000 7fb43159a700 1 -- 172.21.15.27:0/2109151882 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42800b760 con 0x7fb434159e60 2024-01-11T09:45:10.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.895+0000 7fb43159a700 1 -- 172.21.15.27:0/2109151882 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42800bae0 con 0x7fb434159e60 2024-01-11T09:45:10.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.895+0000 7fb438bff700 1 -- 172.21.15.27:0/2109151882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb434159e60 msgr2=0x7fb43415aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:10.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.895+0000 7fb438bff700 1 --2- 172.21.15.27:0/2109151882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb434159e60 0x7fb43415aec0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fb4280098e0 tx=0x7fb4280091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:10.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.895+0000 7fb438bff700 1 -- 172.21.15.27:0/2109151882 shutdown_connections 2024-01-11T09:45:10.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.895+0000 7fb438bff700 1 --2- 172.21.15.27:0/2109151882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb434159e60 0x7fb43415aec0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:10.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.895+0000 7fb438bff700 1 -- 172.21.15.27:0/2109151882 >> 172.21.15.27:0/2109151882 conn(0x7fb43406ed40 msgr2=0x7fb43406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:10.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.895+0000 7fb438bff700 1 -- 172.21.15.27:0/2109151882 shutdown_connections 2024-01-11T09:45:10.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.895+0000 7fb438bff700 1 -- 172.21.15.27:0/2109151882 wait complete. 2024-01-11T09:45:10.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.897+0000 7fb438bff700 1 Processor -- start 2024-01-11T09:45:10.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.898+0000 7fb438bff700 1 -- start start 2024-01-11T09:45:10.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.899+0000 7fb438bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb43415a7a0 0x7fb434155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:10.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.899+0000 7fb438bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb43415b400 0x7fb43414fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:10.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.899+0000 7fb438bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb434150170 0x7fb434169d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:10.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.899+0000 7fb438bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4341505f0 con 0x7fb43415a7a0 2024-01-11T09:45:10.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.899+0000 7fb438bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43413bad0 con 0x7fb434150170 2024-01-11T09:45:10.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.899+0000 7fb438bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4340aff70 con 0x7fb43415b400 2024-01-11T09:45:10.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.899+0000 7fb43259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb43415a7a0 0x7fb434155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:10.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.899+0000 7fb43259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb43415a7a0 0x7fb434155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60908/0 (socket says 172.21.15.27:60908) 2024-01-11T09:45:10.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.899+0000 7fb43259c700 1 -- 172.21.15.27:0/2244576027 learned_addr learned my addr 172.21.15.27:0/2244576027 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:10.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.899+0000 7fb432d9d700 1 --2- 172.21.15.27:0/2244576027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb434150170 0x7fb434169d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:10.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.899+0000 7fb431d9b700 1 --2- 172.21.15.27:0/2244576027 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb43415b400 0x7fb43414fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:10.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.899+0000 7fb43259c700 1 -- 172.21.15.27:0/2244576027 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb43415b400 msgr2=0x7fb43414fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:10.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.899+0000 7fb43259c700 1 --2- 172.21.15.27:0/2244576027 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb43415b400 0x7fb43414fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:10.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.899+0000 7fb43259c700 1 -- 172.21.15.27:0/2244576027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb434150170 msgr2=0x7fb434169d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:10.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.899+0000 7fb43259c700 1 --2- 172.21.15.27:0/2244576027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb434150170 0x7fb434169d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:10.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.900+0000 7fb43259c700 1 -- 172.21.15.27:0/2244576027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb428009420 con 0x7fb43415a7a0 2024-01-11T09:45:10.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.900+0000 7fb431d9b700 1 --2- 172.21.15.27:0/2244576027 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb43415b400 0x7fb43414fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:10.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.900+0000 7fb43259c700 1 --2- 172.21.15.27:0/2244576027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb43415a7a0 0x7fb434155f30 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fb42400c610 tx=0x7fb424007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:10.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.900+0000 7fb4227fc700 1 -- 172.21.15.27:0/2244576027 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb424011590 con 0x7fb43415a7a0 2024-01-11T09:45:10.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.900+0000 7fb4227fc700 1 -- 172.21.15.27:0/2244576027 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4240116f0 con 0x7fb43415a7a0 2024-01-11T09:45:10.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.900+0000 7fb438bff700 1 -- 172.21.15.27:0/2244576027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4340b0af0 con 0x7fb43415a7a0 2024-01-11T09:45:10.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.900+0000 7fb4227fc700 1 -- 172.21.15.27:0/2244576027 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb424011ab0 con 0x7fb43415a7a0 2024-01-11T09:45:10.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.900+0000 7fb438bff700 1 -- 172.21.15.27:0/2244576027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4340b01c0 con 0x7fb43415a7a0 2024-01-11T09:45:10.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.901+0000 7fb438bff700 1 -- 172.21.15.27:0/2244576027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb43404c3c0 con 0x7fb43415a7a0 2024-01-11T09:45:10.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.902+0000 7fb4227fc700 1 -- 172.21.15.27:0/2244576027 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 124) v1 ==== 84453+0+0 (secure 0 0 0) 0x7fb42401b3b0 con 0x7fb43415a7a0 2024-01-11T09:45:10.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.903+0000 7fb4227fc700 1 -- 172.21.15.27:0/2244576027 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 6563+0+0 (secure 0 0 0) 0x7fb424083ae0 con 0x7fb43415a7a0 2024-01-11T09:45:10.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:10.907+0000 7fb4227fc700 1 -- 172.21.15.27:0/2244576027 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb424050cc0 con 0x7fb43415a7a0 2024-01-11T09:45:11.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:11.111+0000 7fb438bff700 1 -- 172.21.15.27:0/2244576027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb43415be60 con 0x7fb43415a7a0 2024-01-11T09:45:11.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:11.287+0000 7fb4227fc700 1 -- 172.21.15.27:0/2244576027 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 125) v1 ==== 84580+0+0 (secure 0 0 0) 0x7fb424050770 con 0x7fb43415a7a0 2024-01-11T09:45:11.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:11.287+0000 7fb4227fc700 1 --2- 172.21.15.27:0/2244576027 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb41406c9b0 0x7fb41406ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:11.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:11.288+0000 7fb431d9b700 1 --2- 172.21.15.27:0/2244576027 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb41406c9b0 0x7fb41406ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:11.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:11.289+0000 7fb431d9b700 1 --2- 172.21.15.27:0/2244576027 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb41406c9b0 0x7fb41406ed90 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb4340cfdf0 tx=0x7fb428016000 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:12.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.068+0000 7fb4227fc700 1 -- 172.21.15.27:0/2244576027 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v205) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb42406faa0 con 0x7fb43415a7a0 2024-01-11T09:45:12.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.073+0000 7fb438bff700 1 -- 172.21.15.27:0/2244576027 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb41406c9b0 msgr2=0x7fb41406ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:12.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.073+0000 7fb438bff700 1 --2- 172.21.15.27:0/2244576027 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb41406c9b0 0x7fb41406ed90 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb4340cfdf0 tx=0x7fb428016000 comp rx=0 tx=0).stop 2024-01-11T09:45:12.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.073+0000 7fb438bff700 1 -- 172.21.15.27:0/2244576027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb43415a7a0 msgr2=0x7fb434155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:12.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.073+0000 7fb438bff700 1 --2- 172.21.15.27:0/2244576027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb43415a7a0 0x7fb434155f30 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fb42400c610 tx=0x7fb424007e60 comp rx=0 tx=0).stop 2024-01-11T09:45:12.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.073+0000 7fb438bff700 1 -- 172.21.15.27:0/2244576027 shutdown_connections 2024-01-11T09:45:12.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.073+0000 7fb438bff700 1 --2- 172.21.15.27:0/2244576027 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb41406c9b0 0x7fb41406ed90 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:12.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.073+0000 7fb438bff700 1 --2- 172.21.15.27:0/2244576027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb434150170 0x7fb434169d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:12.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.073+0000 7fb438bff700 1 --2- 172.21.15.27:0/2244576027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb43415a7a0 0x7fb434155f30 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:12.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.073+0000 7fb438bff700 1 --2- 172.21.15.27:0/2244576027 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb43415b400 0x7fb43414fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:12.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.073+0000 7fb438bff700 1 -- 172.21.15.27:0/2244576027 >> 172.21.15.27:0/2244576027 conn(0x7fb43406ed40 msgr2=0x7fb4340b5e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:12.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.073+0000 7fb438bff700 1 -- 172.21.15.27:0/2244576027 shutdown_connections 2024-01-11T09:45:12.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.073+0000 7fb438bff700 1 -- 172.21.15.27:0/2244576027 wait complete. 2024-01-11T09:45:12.066 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-01-11T09:45:12.087 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-11T09:45:12.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.349+0000 7f14b95ff700 1 Processor -- start 2024-01-11T09:45:12.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.349+0000 7f14b95ff700 1 -- start start 2024-01-11T09:45:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.350+0000 7f14b95ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14b4157440 0x7f14b40ae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.350+0000 7f14b95ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14b4157810 0x7f14b4069660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.350+0000 7f14b95ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b406d420 con 0x7f14b406a570 2024-01-11T09:45:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.350+0000 7f14b95ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14b406d700 con 0x7f14b4157440 2024-01-11T09:45:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.350+0000 7f14b95ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b406d9e0 con 0x7f14b4157810 2024-01-11T09:45:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.351+0000 7f14b2ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14b4157440 0x7f14b40ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.351+0000 7f14b27fc700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14b4157810 0x7f14b4069660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.351+0000 7f14b2ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14b4157440 0x7f14b40ae430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60920/0 (socket says 172.21.15.27:60920) 2024-01-11T09:45:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.351+0000 7f14b2ffd700 1 -- 172.21.15.27:0/2175074099 learned_addr learned my addr 172.21.15.27:0/2175074099 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.351+0000 7f14b2ffd700 1 -- 172.21.15.27:0/2175074099 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14b4157810 msgr2=0x7f14b4069660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.351+0000 7f14b2ffd700 1 --2- 172.21.15.27:0/2175074099 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14b4157810 0x7f14b4069660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.351+0000 7f14b2ffd700 1 -- 172.21.15.27:0/2175074099 >> v1:172.21.15.77:6789/0 conn(0x7f14b406a570 legacy=0x7f14b406e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.351+0000 7f14b2ffd700 1 -- 172.21.15.27:0/2175074099 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b406dcc0 con 0x7f14b4157440 2024-01-11T09:45:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.351+0000 7f14b2ffd700 1 --2- 172.21.15.27:0/2175074099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4157440 0x7f14b40ae430 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009990 tx=0x7f14a4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd8357de7df217fe server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.352+0000 7f14b1ffb700 1 -- 172.21.15.27:0/2175074099 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a4009e80 con 0x7f14b4157440 2024-01-11T09:45:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.352+0000 7f14b1ffb700 1 -- 172.21.15.27:0/2175074099 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a4005c10 con 0x7f14b4157440 2024-01-11T09:45:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.352+0000 7f14b1ffb700 1 -- 172.21.15.27:0/2175074099 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a400b650 con 0x7f14b4157440 2024-01-11T09:45:12.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.352+0000 7f14b95ff700 1 -- 172.21.15.27:0/2175074099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4157440 msgr2=0x7f14b40ae430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:12.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.352+0000 7f14b95ff700 1 --2- 172.21.15.27:0/2175074099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4157440 0x7f14b40ae430 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009990 tx=0x7f14a4009220 comp rx=0 tx=0).stop 2024-01-11T09:45:12.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.352+0000 7f14b95ff700 1 -- 172.21.15.27:0/2175074099 shutdown_connections 2024-01-11T09:45:12.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.352+0000 7f14b95ff700 1 --2- 172.21.15.27:0/2175074099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4157440 0x7f14b40ae430 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:12.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.352+0000 7f14b95ff700 1 --2- 172.21.15.27:0/2175074099 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14b4157810 0x7f14b4069660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:12.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.352+0000 7f14b95ff700 1 -- 172.21.15.27:0/2175074099 >> 172.21.15.27:0/2175074099 conn(0x7f14b40680e0 msgr2=0x7f14b40cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:12.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.352+0000 7f14b95ff700 1 -- 172.21.15.27:0/2175074099 shutdown_connections 2024-01-11T09:45:12.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.352+0000 7f14b95ff700 1 -- 172.21.15.27:0/2175074099 wait complete. 2024-01-11T09:45:12.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.354+0000 7f14b95ff700 1 Processor -- start 2024-01-11T09:45:12.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.354+0000 7f14b95ff700 1 -- start start 2024-01-11T09:45:12.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.355+0000 7f14b95ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14b406a570 0x7f14b414a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:12.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.355+0000 7f14b95ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4157440 0x7f14b4154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:12.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.355+0000 7f14b95ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14b4157810 0x7f14b40cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:12.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.355+0000 7f14b95ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b40c23f0 con 0x7f14b4157440 2024-01-11T09:45:12.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.355+0000 7f14b95ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b40c17e0 con 0x7f14b406a570 2024-01-11T09:45:12.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.355+0000 7f14b95ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b40c1ac0 con 0x7f14b4157810 2024-01-11T09:45:12.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.355+0000 7f14b27fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4157440 0x7f14b4154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:12.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.355+0000 7f14b27fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4157440 0x7f14b4154700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60928/0 (socket says 172.21.15.27:60928) 2024-01-11T09:45:12.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.355+0000 7f14b27fc700 1 -- 172.21.15.27:0/1706158781 learned_addr learned my addr 172.21.15.27:0/1706158781 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:12.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.355+0000 7f14b2ffd700 1 --2- 172.21.15.27:0/1706158781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14b406a570 0x7f14b414a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:12.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.356+0000 7f14b37fe700 1 --2- 172.21.15.27:0/1706158781 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14b4157810 0x7f14b40cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:12.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.356+0000 7f14b27fc700 1 -- 172.21.15.27:0/1706158781 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14b4157810 msgr2=0x7f14b40cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:12.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.356+0000 7f14b27fc700 1 --2- 172.21.15.27:0/1706158781 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14b4157810 0x7f14b40cb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:12.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.356+0000 7f14b27fc700 1 -- 172.21.15.27:0/1706158781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14b406a570 msgr2=0x7f14b414a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:12.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.356+0000 7f14b27fc700 1 --2- 172.21.15.27:0/1706158781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14b406a570 0x7f14b414a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:12.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.356+0000 7f14b27fc700 1 -- 172.21.15.27:0/1706158781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a4009480 con 0x7f14b4157440 2024-01-11T09:45:12.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.356+0000 7f14b2ffd700 1 --2- 172.21.15.27:0/1706158781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14b406a570 0x7f14b414a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:12.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.356+0000 7f14b27fc700 1 --2- 172.21.15.27:0/1706158781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4157440 0x7f14b4154700 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f14a800e930 tx=0x7f14a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:12.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.356+0000 7f149affd700 1 -- 172.21.15.27:0/1706158781 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a800eed0 con 0x7f14b4157440 2024-01-11T09:45:12.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.356+0000 7f149affd700 1 -- 172.21.15.27:0/1706158781 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a800c850 con 0x7f14b4157440 2024-01-11T09:45:12.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.356+0000 7f14b95ff700 1 -- 172.21.15.27:0/1706158781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b40c1da0 con 0x7f14b4157440 2024-01-11T09:45:12.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.356+0000 7f149affd700 1 -- 172.21.15.27:0/1706158781 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a800cbf0 con 0x7f14b4157440 2024-01-11T09:45:12.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.356+0000 7f14b95ff700 1 -- 172.21.15.27:0/1706158781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b414ac60 con 0x7f14b4157440 2024-01-11T09:45:12.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.358+0000 7f14b95ff700 1 -- 172.21.15.27:0/1706158781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b404c3c0 con 0x7f14b4157440 2024-01-11T09:45:12.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.359+0000 7f149affd700 1 -- 172.21.15.27:0/1706158781 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 125) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f14a8013470 con 0x7f14b4157440 2024-01-11T09:45:12.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.361+0000 7f149affd700 1 --2- 172.21.15.27:0/1706158781 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f14a0066e00 0x7f14a00692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:12.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.361+0000 7f149affd700 1 -- 172.21.15.27:0/1706158781 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f14a8084640 con 0x7f14b4157440 2024-01-11T09:45:12.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.361+0000 7f14b2ffd700 1 --2- 172.21.15.27:0/1706158781 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f14a0066e00 0x7f14a00692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:12.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.363+0000 7f14b2ffd700 1 --2- 172.21.15.27:0/1706158781 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f14a0066e00 0x7f14a00692b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009260 tx=0x7f14a4016000 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:12.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.366+0000 7f149affd700 1 -- 172.21.15.27:0/1706158781 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f14a80518f0 con 0x7f14b4157440 2024-01-11T09:45:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:12.570+0000 7f14b95ff700 1 -- 172.21.15.27:0/1706158781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f14b406dfa0 con 0x7f14b4157440 2024-01-11T09:45:13.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.075+0000 7f149affd700 1 -- 172.21.15.27:0/1706158781 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 126) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f14a8051cc0 con 0x7f14b4157440 2024-01-11T09:45:13.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.296+0000 7f149affd700 1 -- 172.21.15.27:0/1706158781 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v206) v1 ==== 169+0+0 (secure 0 0 0) 0x7f14a80518f0 con 0x7f14b4157440 2024-01-11T09:45:13.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.300+0000 7f14b95ff700 1 -- 172.21.15.27:0/1706158781 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f14a0066e00 msgr2=0x7f14a00692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:13.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.300+0000 7f14b95ff700 1 --2- 172.21.15.27:0/1706158781 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f14a0066e00 0x7f14a00692b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009260 tx=0x7f14a4016000 comp rx=0 tx=0).stop 2024-01-11T09:45:13.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.300+0000 7f14b95ff700 1 -- 172.21.15.27:0/1706158781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4157440 msgr2=0x7f14b4154700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:13.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.300+0000 7f14b95ff700 1 --2- 172.21.15.27:0/1706158781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4157440 0x7f14b4154700 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f14a800e930 tx=0x7f14a80091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.300+0000 7f14b95ff700 1 -- 172.21.15.27:0/1706158781 shutdown_connections 2024-01-11T09:45:13.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.300+0000 7f14b95ff700 1 --2- 172.21.15.27:0/1706158781 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f14b4157810 0x7f14b40cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.300+0000 7f14b95ff700 1 --2- 172.21.15.27:0/1706158781 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f14a0066e00 0x7f14a00692b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.300+0000 7f14b95ff700 1 --2- 172.21.15.27:0/1706158781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14b406a570 0x7f14b414a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.300+0000 7f14b95ff700 1 --2- 172.21.15.27:0/1706158781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14b4157440 0x7f14b4154700 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.300+0000 7f14b95ff700 1 -- 172.21.15.27:0/1706158781 >> 172.21.15.27:0/1706158781 conn(0x7f14b40680e0 msgr2=0x7f14b414d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:13.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.300+0000 7f14b95ff700 1 -- 172.21.15.27:0/1706158781 shutdown_connections 2024-01-11T09:45:13.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.301+0000 7f14b95ff700 1 -- 172.21.15.27:0/1706158781 wait complete. 2024-01-11T09:45:13.295 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-01-11T09:45:13.312 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-11T09:45:13.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.528+0000 7f264a5ff700 1 Processor -- start 2024-01-11T09:45:13.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.528+0000 7f264a5ff700 1 -- start start 2024-01-11T09:45:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.529+0000 7f264a5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2644155c50 0x7f26440aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.529+0000 7f264a5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26440c74f0 0x7f2644069910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.529+0000 7f264a5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264406e480 con 0x7f26440694c0 2024-01-11T09:45:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.529+0000 7f264a5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26440cb5a0 con 0x7f2644155c50 2024-01-11T09:45:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.529+0000 7f264a5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f264406df00 con 0x7f26440c74f0 2024-01-11T09:45:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.530+0000 7f2643fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2644155c50 0x7f26440aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.530+0000 7f2648b9c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26440c74f0 0x7f2644069910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.530+0000 7f26437fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f26440694c0 0x7f26440c6de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:37098/0 (socket says 172.21.15.27:37098) 2024-01-11T09:45:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.530+0000 7f2643fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2644155c50 0x7f26440aef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:57526/0 (socket says 172.21.15.27:57526) 2024-01-11T09:45:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.530+0000 7f26437fe700 1 -- 172.21.15.27:0/3097638984 learned_addr learned my addr 172.21.15.27:0/3097638984 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.530+0000 7f2642ffd700 1 -- 172.21.15.27:0/3097638984 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267405778 0 0) 0x7f264406e480 con 0x7f26440694c0 2024-01-11T09:45:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.530+0000 7f2642ffd700 1 -- 172.21.15.27:0/3097638984 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f262c003570 con 0x7f26440694c0 2024-01-11T09:45:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.530+0000 7f2643fff700 1 -- 172.21.15.27:0/3097638984 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26440c74f0 msgr2=0x7f2644069910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.530+0000 7f2643fff700 1 --2- 172.21.15.27:0/3097638984 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26440c74f0 0x7f2644069910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.531+0000 7f2643fff700 1 -- 172.21.15.27:0/3097638984 >> v1:172.21.15.77:6789/0 conn(0x7f26440694c0 legacy=0x7f26440c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.531+0000 7f2643fff700 1 -- 172.21.15.27:0/3097638984 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2644069e50 con 0x7f2644155c50 2024-01-11T09:45:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.531+0000 7f2648b9c700 1 --2- 172.21.15.27:0/3097638984 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26440c74f0 0x7f2644069910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.531+0000 7f2643fff700 1 --2- 172.21.15.27:0/3097638984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2644155c50 0x7f26440aef80 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f263400aa70 tx=0x7f263400a2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea64881832b76180 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.531+0000 7f2642ffd700 1 -- 172.21.15.27:0/3097638984 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2634005c10 con 0x7f2644155c50 2024-01-11T09:45:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.531+0000 7f2642ffd700 1 -- 172.21.15.27:0/3097638984 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2634005d70 con 0x7f2644155c50 2024-01-11T09:45:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.531+0000 7f2642ffd700 1 -- 172.21.15.27:0/3097638984 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263400ca20 con 0x7f2644155c50 2024-01-11T09:45:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.532+0000 7f264a5ff700 1 -- 172.21.15.27:0/3097638984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2644155c50 msgr2=0x7f26440aef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.532+0000 7f264a5ff700 1 --2- 172.21.15.27:0/3097638984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2644155c50 0x7f26440aef80 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f263400aa70 tx=0x7f263400a2e0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.532+0000 7f264a5ff700 1 -- 172.21.15.27:0/3097638984 shutdown_connections 2024-01-11T09:45:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.532+0000 7f264a5ff700 1 --2- 172.21.15.27:0/3097638984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2644155c50 0x7f26440aef80 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.532+0000 7f264a5ff700 1 --2- 172.21.15.27:0/3097638984 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26440c74f0 0x7f2644069910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.532+0000 7f264a5ff700 1 -- 172.21.15.27:0/3097638984 >> 172.21.15.27:0/3097638984 conn(0x7f26440680a0 msgr2=0x7f26440cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.532+0000 7f264a5ff700 1 -- 172.21.15.27:0/3097638984 shutdown_connections 2024-01-11T09:45:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.532+0000 7f264a5ff700 1 -- 172.21.15.27:0/3097638984 wait complete. 2024-01-11T09:45:13.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.533+0000 7f264a5ff700 1 Processor -- start 2024-01-11T09:45:13.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.533+0000 7f264a5ff700 1 -- start start 2024-01-11T09:45:13.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.534+0000 7f264a5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26440694c0 0x7f26440c2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:13.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.534+0000 7f264a5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26440c74f0 0x7f26440c0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:13.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.534+0000 7f264a5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2644155c50 0x7f2644144c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:13.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.534+0000 7f264a5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26440c14a0 con 0x7f2644155c50 2024-01-11T09:45:13.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.534+0000 7f264a5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26440c1f30 con 0x7f26440c74f0 2024-01-11T09:45:13.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.534+0000 7f264a5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f264415b310 con 0x7f26440694c0 2024-01-11T09:45:13.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.535+0000 7f2648b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2644155c50 0x7f2644144c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:13.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.535+0000 7f2648b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2644155c50 0x7f2644144c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60936/0 (socket says 172.21.15.27:60936) 2024-01-11T09:45:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.535+0000 7f2648b9c700 1 -- 172.21.15.27:0/2705782423 learned_addr learned my addr 172.21.15.27:0/2705782423 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.535+0000 7f2643fff700 1 --2- 172.21.15.27:0/2705782423 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26440694c0 0x7f26440c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.535+0000 7f26437fe700 1 --2- 172.21.15.27:0/2705782423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26440c74f0 0x7f26440c0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.535+0000 7f2648b9c700 1 -- 172.21.15.27:0/2705782423 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26440694c0 msgr2=0x7f26440c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.535+0000 7f2648b9c700 1 --2- 172.21.15.27:0/2705782423 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26440694c0 0x7f26440c2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.535+0000 7f2648b9c700 1 -- 172.21.15.27:0/2705782423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26440c74f0 msgr2=0x7f26440c0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.535+0000 7f2648b9c700 1 --2- 172.21.15.27:0/2705782423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26440c74f0 0x7f26440c0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.535+0000 7f2648b9c700 1 -- 172.21.15.27:0/2705782423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c009a20 con 0x7f2644155c50 2024-01-11T09:45:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.535+0000 7f2648b9c700 1 --2- 172.21.15.27:0/2705782423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2644155c50 0x7f2644144c70 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f263c009fc0 tx=0x7f263c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.536+0000 7f262bfff700 1 -- 172.21.15.27:0/2705782423 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c00cd30 con 0x7f2644155c50 2024-01-11T09:45:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.536+0000 7f262bfff700 1 -- 172.21.15.27:0/2705782423 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263c00f040 con 0x7f2644155c50 2024-01-11T09:45:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.536+0000 7f262bfff700 1 -- 172.21.15.27:0/2705782423 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c0146d0 con 0x7f2644155c50 2024-01-11T09:45:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.536+0000 7f264a5ff700 1 -- 172.21.15.27:0/2705782423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f263400a560 con 0x7f2644155c50 2024-01-11T09:45:13.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.536+0000 7f264a5ff700 1 -- 172.21.15.27:0/2705782423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26440c1690 con 0x7f2644155c50 2024-01-11T09:45:13.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.537+0000 7f264a5ff700 1 -- 172.21.15.27:0/2705782423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f264404c3c0 con 0x7f2644155c50 2024-01-11T09:45:13.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.538+0000 7f262bfff700 1 -- 172.21.15.27:0/2705782423 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 126) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f263c003020 con 0x7f2644155c50 2024-01-11T09:45:13.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.539+0000 7f262bfff700 1 --2- 172.21.15.27:0/2705782423 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2630066ea0 0x7f2630069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:13.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.540+0000 7f262bfff700 1 -- 172.21.15.27:0/2705782423 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 5653+0+0 (secure 0 0 0) 0x7f263c084ee0 con 0x7f2644155c50 2024-01-11T09:45:13.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.540+0000 7f2643fff700 1 --2- 172.21.15.27:0/2705782423 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2630066ea0 0x7f2630069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:13.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.541+0000 7f2643fff700 1 --2- 172.21.15.27:0/2705782423 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2630066ea0 0x7f2630069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f263400a320 tx=0x7f2634019000 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:13.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.544+0000 7f262bfff700 1 -- 172.21.15.27:0/2705782423 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f263c052260 con 0x7f2644155c50 2024-01-11T09:45:13.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.753+0000 7f264a5ff700 1 -- 172.21.15.27:0/2705782423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f26440aeb30 con 0x7f2644155c50 2024-01-11T09:45:13.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.757+0000 7f262bfff700 1 -- 172.21.15.27:0/2705782423 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v73) v1 ==== 120+0+0 (secure 0 0 0) 0x7f263c055880 con 0x7f2644155c50 2024-01-11T09:45:13.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.760+0000 7f264a5ff700 1 -- 172.21.15.27:0/2705782423 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2630066ea0 msgr2=0x7f2630069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:13.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.760+0000 7f264a5ff700 1 --2- 172.21.15.27:0/2705782423 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2630066ea0 0x7f2630069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f263400a320 tx=0x7f2634019000 comp rx=0 tx=0).stop 2024-01-11T09:45:13.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.760+0000 7f264a5ff700 1 -- 172.21.15.27:0/2705782423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2644155c50 msgr2=0x7f2644144c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:13.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.760+0000 7f264a5ff700 1 --2- 172.21.15.27:0/2705782423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2644155c50 0x7f2644144c70 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f263c009fc0 tx=0x7f263c00b040 comp rx=0 tx=0).stop 2024-01-11T09:45:13.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.760+0000 7f264a5ff700 1 -- 172.21.15.27:0/2705782423 shutdown_connections 2024-01-11T09:45:13.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.760+0000 7f264a5ff700 1 --2- 172.21.15.27:0/2705782423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2644155c50 0x7f2644144c70 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.760+0000 7f264a5ff700 1 --2- 172.21.15.27:0/2705782423 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26440694c0 0x7f26440c2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.760+0000 7f264a5ff700 1 --2- 172.21.15.27:0/2705782423 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2630066ea0 0x7f2630069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.760+0000 7f264a5ff700 1 --2- 172.21.15.27:0/2705782423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26440c74f0 0x7f26440c0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.760+0000 7f264a5ff700 1 -- 172.21.15.27:0/2705782423 >> 172.21.15.27:0/2705782423 conn(0x7f26440680a0 msgr2=0x7f264406e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:13.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.760+0000 7f264a5ff700 1 -- 172.21.15.27:0/2705782423 shutdown_connections 2024-01-11T09:45:13.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.760+0000 7f264a5ff700 1 -- 172.21.15.27:0/2705782423 wait complete. 2024-01-11T09:45:13.769 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-11T09:45:13.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.978+0000 7fbfb07ff700 1 Processor -- start 2024-01-11T09:45:13.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.979+0000 7fbfb07ff700 1 -- start start 2024-01-11T09:45:13.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.980+0000 7fbfb07ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbfa8157410 0x7fbfa80ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:13.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.980+0000 7fbfb07ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbfa81577e0 0x7fbfa80ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:13.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.980+0000 7fbfb07ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbfa8069540 0x7fbfa80ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:13.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.980+0000 7fbfb07ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa80699d0 con 0x7fbfa8069540 2024-01-11T09:45:13.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.980+0000 7fbfb07ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa8069cb0 con 0x7fbfa81577e0 2024-01-11T09:45:13.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.980+0000 7fbfb07ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa8069f90 con 0x7fbfa8157410 2024-01-11T09:45:13.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.980+0000 7fbfaed9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbfa8069540 0x7fbfa80ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:13.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.980+0000 7fbfaed9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbfa8069540 0x7fbfa80ae540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49650/0 (socket says 172.21.15.27:49650) 2024-01-11T09:45:13.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.980+0000 7fbfaed9c700 1 -- 172.21.15.27:0/1028355284 learned_addr learned my addr 172.21.15.27:0/1028355284 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:13.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.981+0000 7fbfadd9a700 1 --2- 172.21.15.27:0/1028355284 >> v2:172.21.15.77:3300/0 conn(0x7fbfa81577e0 0x7fbfa80ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:13.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.981+0000 7fbfae59b700 1 --2- 172.21.15.27:0/1028355284 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbfa8157410 0x7fbfa80ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:13.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.981+0000 7fbfaed9c700 1 -- 172.21.15.27:0/1028355284 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbfa8157410 msgr2=0x7fbfa80ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:13.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.981+0000 7fbfaed9c700 1 --2- 172.21.15.27:0/1028355284 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbfa8157410 0x7fbfa80ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.981+0000 7fbfaed9c700 1 -- 172.21.15.27:0/1028355284 >> v2:172.21.15.77:3300/0 conn(0x7fbfa81577e0 msgr2=0x7fbfa80ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:13.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.981+0000 7fbfaed9c700 1 --2- 172.21.15.27:0/1028355284 >> v2:172.21.15.77:3300/0 conn(0x7fbfa81577e0 0x7fbfa80ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.981+0000 7fbfaed9c700 1 -- 172.21.15.27:0/1028355284 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa806a270 con 0x7fbfa8069540 2024-01-11T09:45:13.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.981+0000 7fbfadd9a700 1 --2- 172.21.15.27:0/1028355284 >> v2:172.21.15.77:3300/0 conn(0x7fbfa81577e0 0x7fbfa80ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:13.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.981+0000 7fbfaed9c700 1 --2- 172.21.15.27:0/1028355284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfa8069540 0x7fbfa80ae540 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8070cc0 tx=0x7fbfa0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=790963efcadc77c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:13.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.981+0000 7fbfad599700 1 -- 172.21.15.27:0/1028355284 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa000e070 con 0x7fbfa8069540 2024-01-11T09:45:13.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.982+0000 7fbfad599700 1 -- 172.21.15.27:0/1028355284 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa000ba50 con 0x7fbfa8069540 2024-01-11T09:45:13.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.982+0000 7fbfad599700 1 -- 172.21.15.27:0/1028355284 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa00092e0 con 0x7fbfa8069540 2024-01-11T09:45:13.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.982+0000 7fbfb07ff700 1 -- 172.21.15.27:0/1028355284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfa8069540 msgr2=0x7fbfa80ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:13.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.982+0000 7fbfb07ff700 1 --2- 172.21.15.27:0/1028355284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfa8069540 0x7fbfa80ae540 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8070cc0 tx=0x7fbfa0009840 comp rx=0 tx=0).stop 2024-01-11T09:45:13.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.982+0000 7fbfb07ff700 1 -- 172.21.15.27:0/1028355284 shutdown_connections 2024-01-11T09:45:13.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.982+0000 7fbfb07ff700 1 --2- 172.21.15.27:0/1028355284 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbfa8157410 0x7fbfa80ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.982+0000 7fbfb07ff700 1 --2- 172.21.15.27:0/1028355284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfa8069540 0x7fbfa80ae540 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.982+0000 7fbfb07ff700 1 --2- 172.21.15.27:0/1028355284 >> v2:172.21.15.77:3300/0 conn(0x7fbfa81577e0 0x7fbfa80ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.982+0000 7fbfb07ff700 1 -- 172.21.15.27:0/1028355284 >> 172.21.15.27:0/1028355284 conn(0x7fbfa80680a0 msgr2=0x7fbfa80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:13.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.982+0000 7fbfb07ff700 1 -- 172.21.15.27:0/1028355284 shutdown_connections 2024-01-11T09:45:13.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.982+0000 7fbfb07ff700 1 -- 172.21.15.27:0/1028355284 wait complete. 2024-01-11T09:45:13.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.984+0000 7fbfb07ff700 1 Processor -- start 2024-01-11T09:45:13.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.984+0000 7fbfb07ff700 1 -- start start 2024-01-11T09:45:13.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.985+0000 7fbfb07ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbfa8069540 0x7fbfa8151ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:13.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.985+0000 7fbfb07ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfa8157410 0x7fbfa8157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:13.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.985+0000 7fbfb07ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfa81577e0 0x7fbfa8167470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:13.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.985+0000 7fbfb07ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa8152400 con 0x7fbfa81577e0 2024-01-11T09:45:13.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.985+0000 7fbfb07ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa8168000 con 0x7fbfa8157410 2024-01-11T09:45:13.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.985+0000 7fbfb07ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa81682e0 con 0x7fbfa8069540 2024-01-11T09:45:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.985+0000 7fbfaed9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfa81577e0 0x7fbfa8167470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.985+0000 7fbfaed9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfa81577e0 0x7fbfa8167470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49652/0 (socket says 172.21.15.27:49652) 2024-01-11T09:45:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.985+0000 7fbfaed9c700 1 -- 172.21.15.27:0/3743876067 learned_addr learned my addr 172.21.15.27:0/3743876067 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.985+0000 7fbfadd9a700 1 --2- 172.21.15.27:0/3743876067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfa8157410 0x7fbfa8157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.985+0000 7fbfae59b700 1 --2- 172.21.15.27:0/3743876067 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbfa8069540 0x7fbfa8151ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.986+0000 7fbfaed9c700 1 -- 172.21.15.27:0/3743876067 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbfa8069540 msgr2=0x7fbfa8151ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.986+0000 7fbfaed9c700 1 --2- 172.21.15.27:0/3743876067 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbfa8069540 0x7fbfa8151ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.986+0000 7fbfaed9c700 1 -- 172.21.15.27:0/3743876067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfa8157410 msgr2=0x7fbfa8157bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.986+0000 7fbfaed9c700 1 --2- 172.21.15.27:0/3743876067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfa8157410 0x7fbfa8157bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.986+0000 7fbfaed9c700 1 -- 172.21.15.27:0/3743876067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa0009ae0 con 0x7fbfa81577e0 2024-01-11T09:45:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.986+0000 7fbfaed9c700 1 --2- 172.21.15.27:0/3743876067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfa81577e0 0x7fbfa8167470 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0012000 tx=0x7fbfa0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.986+0000 7fbf9e7fc700 1 -- 172.21.15.27:0/3743876067 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa00032f0 con 0x7fbfa81577e0 2024-01-11T09:45:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.986+0000 7fbf9e7fc700 1 -- 172.21.15.27:0/3743876067 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa0008ce0 con 0x7fbfa81577e0 2024-01-11T09:45:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.986+0000 7fbf9e7fc700 1 -- 172.21.15.27:0/3743876067 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa00266b0 con 0x7fbfa81577e0 2024-01-11T09:45:13.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.986+0000 7fbfb07ff700 1 -- 172.21.15.27:0/3743876067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa815b160 con 0x7fbfa81577e0 2024-01-11T09:45:13.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.987+0000 7fbfb07ff700 1 -- 172.21.15.27:0/3743876067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa815ae10 con 0x7fbfa81577e0 2024-01-11T09:45:13.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.988+0000 7fbfb07ff700 1 -- 172.21.15.27:0/3743876067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa804c3c0 con 0x7fbfa81577e0 2024-01-11T09:45:13.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.988+0000 7fbf9e7fc700 1 -- 172.21.15.27:0/3743876067 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 126) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fbfa0003040 con 0x7fbfa81577e0 2024-01-11T09:45:13.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.989+0000 7fbf9e7fc700 1 --2- 172.21.15.27:0/3743876067 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbf8c066ea0 0x7fbf8c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:13.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.989+0000 7fbf9e7fc700 1 -- 172.21.15.27:0/3743876067 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 5653+0+0 (secure 0 0 0) 0x7fbfa0086230 con 0x7fbfa81577e0 2024-01-11T09:45:13.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.990+0000 7fbfae59b700 1 --2- 172.21.15.27:0/3743876067 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbf8c066ea0 0x7fbf8c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:13.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.991+0000 7fbfae59b700 1 --2- 172.21.15.27:0/3743876067 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbf8c066ea0 0x7fbf8c069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbfa80c6610 tx=0x7fbfa4008040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:13.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:13.994+0000 7fbf9e7fc700 1 -- 172.21.15.27:0/3743876067 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbfa0053510 con 0x7fbfa81577e0 2024-01-11T09:45:14.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.201+0000 7fbfb07ff700 1 -- 172.21.15.27:0/3743876067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fbfa80647f0 con 0x7fbfa81577e0 2024-01-11T09:45:14.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.207+0000 7fbf9e7fc700 1 -- 172.21.15.27:0/3743876067 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v74) v1 ==== 115+0+0 (secure 0 0 0) 0x7fbfa0056b30 con 0x7fbfa81577e0 2024-01-11T09:45:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.209+0000 7fbfb07ff700 1 -- 172.21.15.27:0/3743876067 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbf8c066ea0 msgr2=0x7fbf8c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.209+0000 7fbfb07ff700 1 --2- 172.21.15.27:0/3743876067 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbf8c066ea0 0x7fbf8c069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbfa80c6610 tx=0x7fbfa4008040 comp rx=0 tx=0).stop 2024-01-11T09:45:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.209+0000 7fbfb07ff700 1 -- 172.21.15.27:0/3743876067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfa81577e0 msgr2=0x7fbfa8167470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.209+0000 7fbfb07ff700 1 --2- 172.21.15.27:0/3743876067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfa81577e0 0x7fbfa8167470 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0012000 tx=0x7fbfa0018000 comp rx=0 tx=0).stop 2024-01-11T09:45:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.209+0000 7fbfb07ff700 1 -- 172.21.15.27:0/3743876067 shutdown_connections 2024-01-11T09:45:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.209+0000 7fbfb07ff700 1 --2- 172.21.15.27:0/3743876067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfa81577e0 0x7fbfa8167470 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.209+0000 7fbfb07ff700 1 --2- 172.21.15.27:0/3743876067 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbfa8069540 0x7fbfa8151ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.209+0000 7fbfb07ff700 1 --2- 172.21.15.27:0/3743876067 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbf8c066ea0 0x7fbf8c069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.209+0000 7fbfb07ff700 1 --2- 172.21.15.27:0/3743876067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfa8157410 0x7fbfa8157bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.209+0000 7fbfb07ff700 1 -- 172.21.15.27:0/3743876067 >> 172.21.15.27:0/3743876067 conn(0x7fbfa80680a0 msgr2=0x7fbfa80c6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:14.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.209+0000 7fbfb07ff700 1 -- 172.21.15.27:0/3743876067 shutdown_connections 2024-01-11T09:45:14.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.209+0000 7fbfb07ff700 1 -- 172.21.15.27:0/3743876067 wait complete. 2024-01-11T09:45:14.219 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-11T09:45:14.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.429+0000 7f16f63ff700 1 Processor -- start 2024-01-11T09:45:14.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.429+0000 7f16f63ff700 1 -- start start 2024-01-11T09:45:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.431+0000 7f16f63ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16f0143190 0x7f16f00cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.431+0000 7f16f63ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f16f00cb4b0 0x7f16f00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.431+0000 7f16f63ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f015d320 con 0x7f16f0142dc0 2024-01-11T09:45:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.431+0000 7f16f63ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f015df30 con 0x7f16f00cb4b0 2024-01-11T09:45:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.431+0000 7f16f63ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f015d590 con 0x7f16f0143190 2024-01-11T09:45:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.431+0000 7f16effff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f16f0142dc0 0x7f16f00cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40744/0 (socket says 172.21.15.27:40744) 2024-01-11T09:45:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.431+0000 7f16effff700 1 -- 172.21.15.27:0/1034043763 learned_addr learned my addr 172.21.15.27:0/1034043763 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.431+0000 7f16ef7fe700 1 --2- 172.21.15.27:0/1034043763 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16f0143190 0x7f16f00cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:14.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.431+0000 7f16f499c700 1 --2- 172.21.15.27:0/1034043763 >> v2:172.21.15.77:3300/0 conn(0x7f16f00cb4b0 0x7f16f00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.432+0000 7f16eeffd700 1 -- 172.21.15.27:0/1034043763 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874701934 0 0) 0x7f16f015d320 con 0x7f16f0142dc0 2024-01-11T09:45:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.432+0000 7f16eeffd700 1 -- 172.21.15.27:0/1034043763 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16d8003570 con 0x7f16f0142dc0 2024-01-11T09:45:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.432+0000 7f16eeffd700 1 -- 172.21.15.27:0/1034043763 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1093851823 0 0) 0x7f16d8003570 con 0x7f16f0142dc0 2024-01-11T09:45:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.432+0000 7f16eeffd700 1 -- 172.21.15.27:0/1034043763 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16f0143190 msgr2=0x7f16f00cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.432+0000 7f16eeffd700 1 --2- 172.21.15.27:0/1034043763 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16f0143190 0x7f16f00cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.432+0000 7f16eeffd700 1 -- 172.21.15.27:0/1034043763 >> v2:172.21.15.77:3300/0 conn(0x7f16f00cb4b0 msgr2=0x7f16f00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.432+0000 7f16eeffd700 1 --2- 172.21.15.27:0/1034043763 >> v2:172.21.15.77:3300/0 conn(0x7f16f00cb4b0 0x7f16f00cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.432+0000 7f16eeffd700 1 -- 172.21.15.27:0/1034043763 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f015d870 con 0x7f16f0142dc0 2024-01-11T09:45:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.432+0000 7f16eeffd700 1 -- 172.21.15.27:0/1034043763 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f16e0001470 con 0x7f16f0142dc0 2024-01-11T09:45:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.432+0000 7f16eeffd700 1 -- 172.21.15.27:0/1034043763 >> v1:172.21.15.27:6789/0 conn(0x7f16f0142dc0 legacy=0x7f16f00cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.432+0000 7f16eeffd700 1 --2- 172.21.15.27:0/1034043763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16d8003ae0 0x7f16d8003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.432+0000 7f16eeffd700 1 --2- 172.21.15.27:0/1034043763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16d8004810 0x7f16d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.432+0000 7f16eeffd700 1 --2- 172.21.15.27:0/1034043763 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16f0143190 0x7f16d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.432+0000 7f16eeffd700 1 -- 172.21.15.27:0/1034043763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f015df30 con 0x7f16d8003ae0 2024-01-11T09:45:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.432+0000 7f16eeffd700 1 -- 172.21.15.27:0/1034043763 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f015d590 con 0x7f16d8004810 2024-01-11T09:45:14.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.432+0000 7f16eeffd700 1 -- 172.21.15.27:0/1034043763 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d800bb10 con 0x7f16f0143190 2024-01-11T09:45:14.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.432+0000 7f16effff700 1 --2- 172.21.15.27:0/1034043763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16d8003ae0 0x7f16d8003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:14.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.432+0000 7f16ef7fe700 1 --2- 172.21.15.27:0/1034043763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16d8004810 0x7f16d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:14.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.433+0000 7f16f499c700 1 --2- 172.21.15.27:0/1034043763 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16f0143190 0x7f16d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:14.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.433+0000 7f16effff700 1 -- 172.21.15.27:0/1034043763 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16f0143190 msgr2=0x7f16d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:14.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.433+0000 7f16effff700 1 --2- 172.21.15.27:0/1034043763 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16f0143190 0x7f16d80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.433+0000 7f16effff700 1 -- 172.21.15.27:0/1034043763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16d8004810 msgr2=0x7f16d8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:14.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.433+0000 7f16effff700 1 --2- 172.21.15.27:0/1034043763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16d8004810 0x7f16d8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.433+0000 7f16effff700 1 -- 172.21.15.27:0/1034043763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d8003570 con 0x7f16d8003ae0 2024-01-11T09:45:14.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.433+0000 7f16effff700 1 --2- 172.21.15.27:0/1034043763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16d8003ae0 0x7f16d8003ef0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f16e000c2d0 tx=0x7f16e0001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8971807ecfc78f8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:14.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.433+0000 7f16eeffd700 1 -- 172.21.15.27:0/1034043763 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e000cba0 con 0x7f16d8003ae0 2024-01-11T09:45:14.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.433+0000 7f16eeffd700 1 -- 172.21.15.27:0/1034043763 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e000cd00 con 0x7f16d8003ae0 2024-01-11T09:45:14.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.433+0000 7f16eeffd700 1 -- 172.21.15.27:0/1034043763 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e0005b70 con 0x7f16d8003ae0 2024-01-11T09:45:14.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.434+0000 7f16f63ff700 1 -- 172.21.15.27:0/1034043763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16d8003ae0 msgr2=0x7f16d8003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:14.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.434+0000 7f16f63ff700 1 --2- 172.21.15.27:0/1034043763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16d8003ae0 0x7f16d8003ef0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f16e000c2d0 tx=0x7f16e0001dc0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.434+0000 7f16f499c700 1 -- 172.21.15.27:0/1034043763 reap_dead start 2024-01-11T09:45:14.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.434+0000 7f16f63ff700 1 -- 172.21.15.27:0/1034043763 shutdown_connections 2024-01-11T09:45:14.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.434+0000 7f16f63ff700 1 -- 172.21.15.27:0/1034043763 >> 172.21.15.27:0/1034043763 conn(0x7f16f0063ca0 msgr2=0x7f16f006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:14.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.434+0000 7f16f63ff700 1 -- 172.21.15.27:0/1034043763 shutdown_connections 2024-01-11T09:45:14.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.434+0000 7f16f63ff700 1 -- 172.21.15.27:0/1034043763 wait complete. 2024-01-11T09:45:14.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.436+0000 7f16f63ff700 1 Processor -- start 2024-01-11T09:45:14.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.436+0000 7f16f63ff700 1 -- start start 2024-01-11T09:45:14.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.437+0000 7f16f63ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16d8003ae0 0x7f16f00c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:14.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.437+0000 7f16f63ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f00c8960 0x7f16f0157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:14.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.437+0000 7f16f63ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16f00b1b30 0x7f16f00c45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:14.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.437+0000 7f16f63ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f00b3070 con 0x7f16f00c8960 2024-01-11T09:45:14.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.437+0000 7f16f63ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f00c9a50 con 0x7f16f00b1b30 2024-01-11T09:45:14.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.437+0000 7f16f63ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f00c8e40 con 0x7f16d8003ae0 2024-01-11T09:45:14.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.437+0000 7f16ef7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f00c8960 0x7f16f0157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:14.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.437+0000 7f16ef7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f00c8960 0x7f16f0157120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49672/0 (socket says 172.21.15.27:49672) 2024-01-11T09:45:14.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.437+0000 7f16ef7fe700 1 -- 172.21.15.27:0/272526562 learned_addr learned my addr 172.21.15.27:0/272526562 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.437+0000 7f16effff700 1 --2- 172.21.15.27:0/272526562 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16d8003ae0 0x7f16f00c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.437+0000 7f16f499c700 1 --2- 172.21.15.27:0/272526562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16f00b1b30 0x7f16f00c45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.437+0000 7f16ef7fe700 1 -- 172.21.15.27:0/272526562 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16d8003ae0 msgr2=0x7f16f00c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.437+0000 7f16ef7fe700 1 --2- 172.21.15.27:0/272526562 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16d8003ae0 0x7f16f00c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.437+0000 7f16ef7fe700 1 -- 172.21.15.27:0/272526562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16f00b1b30 msgr2=0x7f16f00c45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.437+0000 7f16ef7fe700 1 --2- 172.21.15.27:0/272526562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16f00b1b30 0x7f16f00c45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.437+0000 7f16ef7fe700 1 -- 172.21.15.27:0/272526562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f00c9120 con 0x7f16f00c8960 2024-01-11T09:45:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.438+0000 7f16ef7fe700 1 --2- 172.21.15.27:0/272526562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f00c8960 0x7f16f0157120 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f16e4011d40 tx=0x7f16e40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.438+0000 7f16d7fff700 1 -- 172.21.15.27:0/272526562 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e400f990 con 0x7f16f00c8960 2024-01-11T09:45:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.438+0000 7f16d7fff700 1 -- 172.21.15.27:0/272526562 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e400faf0 con 0x7f16f00c8960 2024-01-11T09:45:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.438+0000 7f16f63ff700 1 -- 172.21.15.27:0/272526562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f00c9400 con 0x7f16f00c8960 2024-01-11T09:45:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.438+0000 7f16d7fff700 1 -- 172.21.15.27:0/272526562 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e400fe70 con 0x7f16f00c8960 2024-01-11T09:45:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.438+0000 7f16f63ff700 1 -- 172.21.15.27:0/272526562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f00b1fd0 con 0x7f16f00c8960 2024-01-11T09:45:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.439+0000 7f16f63ff700 1 -- 172.21.15.27:0/272526562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f00c9c00 con 0x7f16f00c8960 2024-01-11T09:45:14.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.440+0000 7f16d7fff700 1 -- 172.21.15.27:0/272526562 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 126) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f16e4014650 con 0x7f16f00c8960 2024-01-11T09:45:14.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.441+0000 7f16d7fff700 1 --2- 172.21.15.27:0/272526562 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f16dc066ea0 0x7f16dc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:14.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.441+0000 7f16d7fff700 1 -- 172.21.15.27:0/272526562 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 5653+0+0 (secure 0 0 0) 0x7f16e4018030 con 0x7f16f00c8960 2024-01-11T09:45:14.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.441+0000 7f16effff700 1 --2- 172.21.15.27:0/272526562 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f16dc066ea0 0x7f16dc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:14.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.442+0000 7f16effff700 1 --2- 172.21.15.27:0/272526562 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f16dc066ea0 0x7f16dc069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f16e0015040 tx=0x7f16e0012000 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:14.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.445+0000 7f16d7fff700 1 -- 172.21.15.27:0/272526562 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f16f00c9c00 con 0x7f16f00c8960 2024-01-11T09:45:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.647+0000 7f16f63ff700 1 -- 172.21.15.27:0/272526562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f16f00c9bc0 con 0x7f16f00c8960 2024-01-11T09:45:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.653+0000 7f16d7fff700 1 -- 172.21.15.27:0/272526562 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v75) v1 ==== 104+0+0 (secure 0 0 0) 0x7f16e4052130 con 0x7f16f00c8960 2024-01-11T09:45:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.656+0000 7f16f63ff700 1 -- 172.21.15.27:0/272526562 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f16dc066ea0 msgr2=0x7f16dc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.656+0000 7f16f63ff700 1 --2- 172.21.15.27:0/272526562 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f16dc066ea0 0x7f16dc069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f16e0015040 tx=0x7f16e0012000 comp rx=0 tx=0).stop 2024-01-11T09:45:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.656+0000 7f16f63ff700 1 -- 172.21.15.27:0/272526562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f00c8960 msgr2=0x7f16f0157120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.656+0000 7f16f63ff700 1 --2- 172.21.15.27:0/272526562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f00c8960 0x7f16f0157120 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f16e4011d40 tx=0x7f16e40091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.656+0000 7f16f63ff700 1 -- 172.21.15.27:0/272526562 shutdown_connections 2024-01-11T09:45:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.656+0000 7f16f63ff700 1 --2- 172.21.15.27:0/272526562 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f16dc066ea0 0x7f16dc069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.656+0000 7f16f63ff700 1 --2- 172.21.15.27:0/272526562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16f00b1b30 0x7f16f00c45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.656+0000 7f16f63ff700 1 --2- 172.21.15.27:0/272526562 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f16d8003ae0 0x7f16f00c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.656+0000 7f16f63ff700 1 --2- 172.21.15.27:0/272526562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f00c8960 0x7f16f0157120 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.656+0000 7f16f63ff700 1 -- 172.21.15.27:0/272526562 >> 172.21.15.27:0/272526562 conn(0x7f16f0063ca0 msgr2=0x7f16f0141e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.656+0000 7f16f63ff700 1 -- 172.21.15.27:0/272526562 shutdown_connections 2024-01-11T09:45:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.656+0000 7f16f63ff700 1 -- 172.21.15.27:0/272526562 wait complete. 2024-01-11T09:45:14.665 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2024-01-11T09:45:14.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.874+0000 7f348f1ff700 1 Processor -- start 2024-01-11T09:45:14.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.874+0000 7f348f1ff700 1 -- start start 2024-01-11T09:45:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.875+0000 7f348f1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f34880b03e0 0x7f34880b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.875+0000 7f348f1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f34880b07b0 0x7f34880b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.875+0000 7f348f1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3488159200 0x7f3488159620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.875+0000 7f348f1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34880cd730 con 0x7f34880b07b0 2024-01-11T09:45:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.876+0000 7f348f1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34880ccb20 con 0x7f34880b03e0 2024-01-11T09:45:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.876+0000 7f348f1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34880cce00 con 0x7f3488159200 2024-01-11T09:45:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.876+0000 7f3487fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f34880b07b0 0x7f34880b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.876+0000 7f3487fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f34880b07b0 0x7f34880b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49680/0 (socket says 172.21.15.27:49680) 2024-01-11T09:45:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.876+0000 7f3487fff700 1 -- 172.21.15.27:0/3299140379 learned_addr learned my addr 172.21.15.27:0/3299140379 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.876+0000 7f348cf9b700 1 --2- 172.21.15.27:0/3299140379 >> v2:172.21.15.77:3300/0 conn(0x7f34880b03e0 0x7f34880b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.876+0000 7f348d79c700 1 --2- 172.21.15.27:0/3299140379 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3488159200 0x7f3488159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.876+0000 7f3487fff700 1 -- 172.21.15.27:0/3299140379 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3488159200 msgr2=0x7f3488159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.876+0000 7f3487fff700 1 --2- 172.21.15.27:0/3299140379 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3488159200 0x7f3488159620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.876+0000 7f3487fff700 1 -- 172.21.15.27:0/3299140379 >> v2:172.21.15.77:3300/0 conn(0x7f34880b03e0 msgr2=0x7f34880b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.876+0000 7f3487fff700 1 --2- 172.21.15.27:0/3299140379 >> v2:172.21.15.77:3300/0 conn(0x7f34880b03e0 0x7f34880b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.876+0000 7f3487fff700 1 -- 172.21.15.27:0/3299140379 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34880cd0e0 con 0x7f34880b07b0 2024-01-11T09:45:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.876+0000 7f348d79c700 1 --2- 172.21.15.27:0/3299140379 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3488159200 0x7f3488159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.877+0000 7f3487fff700 1 --2- 172.21.15.27:0/3299140379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34880b07b0 0x7f34880b29c0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f3488089470 tx=0x7f3474009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc83fd4388c4952a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:14.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.877+0000 7f34877fe700 1 -- 172.21.15.27:0/3299140379 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347400b860 con 0x7f34880b07b0 2024-01-11T09:45:14.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.877+0000 7f34877fe700 1 -- 172.21.15.27:0/3299140379 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347400b9c0 con 0x7f34880b07b0 2024-01-11T09:45:14.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.877+0000 7f34877fe700 1 -- 172.21.15.27:0/3299140379 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347400bd60 con 0x7f34880b07b0 2024-01-11T09:45:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.877+0000 7f348f1ff700 1 -- 172.21.15.27:0/3299140379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34880b07b0 msgr2=0x7f34880b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.877+0000 7f348f1ff700 1 --2- 172.21.15.27:0/3299140379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34880b07b0 0x7f34880b29c0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f3488089470 tx=0x7f3474009250 comp rx=0 tx=0).stop 2024-01-11T09:45:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.878+0000 7f348f1ff700 1 -- 172.21.15.27:0/3299140379 shutdown_connections 2024-01-11T09:45:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.878+0000 7f348f1ff700 1 --2- 172.21.15.27:0/3299140379 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3488159200 0x7f3488159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.878+0000 7f348f1ff700 1 --2- 172.21.15.27:0/3299140379 >> v2:172.21.15.77:3300/0 conn(0x7f34880b03e0 0x7f34880b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.878+0000 7f348f1ff700 1 --2- 172.21.15.27:0/3299140379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34880b07b0 0x7f34880b29c0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.878+0000 7f348f1ff700 1 -- 172.21.15.27:0/3299140379 >> 172.21.15.27:0/3299140379 conn(0x7f348806ed40 msgr2=0x7f348806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.878+0000 7f348f1ff700 1 -- 172.21.15.27:0/3299140379 shutdown_connections 2024-01-11T09:45:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.878+0000 7f348f1ff700 1 -- 172.21.15.27:0/3299140379 wait complete. 2024-01-11T09:45:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.879+0000 7f348f1ff700 1 Processor -- start 2024-01-11T09:45:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.879+0000 7f348f1ff700 1 -- start start 2024-01-11T09:45:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.880+0000 7f348f1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34880b03e0 0x7f348816c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.880+0000 7f348f1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34880b07b0 0x7f34880b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.880+0000 7f348f1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3488159200 0x7f34880b4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.880+0000 7f348f1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348815c6e0 con 0x7f3488159200 2024-01-11T09:45:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.880+0000 7f348f1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348815c9c0 con 0x7f34880b03e0 2024-01-11T09:45:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.880+0000 7f348f1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348815cca0 con 0x7f34880b07b0 2024-01-11T09:45:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.881+0000 7f348d79c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3488159200 0x7f34880b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.881+0000 7f348cf9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34880b03e0 0x7f348816c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.881+0000 7f348cf9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34880b03e0 0x7f348816c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:34822/0 (socket says 172.21.15.27:34822) 2024-01-11T09:45:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.881+0000 7f348cf9b700 1 -- 172.21.15.27:0/3713601578 learned_addr learned my addr 172.21.15.27:0/3713601578 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.881+0000 7f3487fff700 1 --2- 172.21.15.27:0/3713601578 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34880b07b0 0x7f34880b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:14.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.881+0000 7f348d79c700 1 -- 172.21.15.27:0/3713601578 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34880b07b0 msgr2=0x7f34880b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:14.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.881+0000 7f348d79c700 1 --2- 172.21.15.27:0/3713601578 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34880b07b0 0x7f34880b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.881+0000 7f348d79c700 1 -- 172.21.15.27:0/3713601578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34880b03e0 msgr2=0x7f348816c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:14.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.881+0000 7f348d79c700 1 --2- 172.21.15.27:0/3713601578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34880b03e0 0x7f348816c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:14.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.881+0000 7f348d79c700 1 -- 172.21.15.27:0/3713601578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34740094f0 con 0x7f3488159200 2024-01-11T09:45:14.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.882+0000 7f348d79c700 1 --2- 172.21.15.27:0/3713601578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3488159200 0x7f34880b4700 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f348000ce40 tx=0x7f34800077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:14.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.882+0000 7f3484ff9700 1 -- 172.21.15.27:0/3713601578 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348000acb0 con 0x7f3488159200 2024-01-11T09:45:14.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.882+0000 7f3484ff9700 1 -- 172.21.15.27:0/3713601578 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f348000ae30 con 0x7f3488159200 2024-01-11T09:45:14.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.882+0000 7f3484ff9700 1 -- 172.21.15.27:0/3713601578 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3480016670 con 0x7f3488159200 2024-01-11T09:45:14.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.882+0000 7f348f1ff700 1 -- 172.21.15.27:0/3713601578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34880c4c00 con 0x7f3488159200 2024-01-11T09:45:14.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.882+0000 7f348f1ff700 1 -- 172.21.15.27:0/3713601578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f348816cbe0 con 0x7f3488159200 2024-01-11T09:45:14.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.883+0000 7f348f1ff700 1 -- 172.21.15.27:0/3713601578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34880c5120 con 0x7f3488159200 2024-01-11T09:45:14.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.884+0000 7f3484ff9700 1 -- 172.21.15.27:0/3713601578 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 126) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f3480016a60 con 0x7f3488159200 2024-01-11T09:45:14.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.885+0000 7f3484ff9700 1 --2- 172.21.15.27:0/3713601578 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f346c066ea0 0x7f346c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:14.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.885+0000 7f3484ff9700 1 -- 172.21.15.27:0/3713601578 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 5653+0+0 (secure 0 0 0) 0x7f3480012070 con 0x7f3488159200 2024-01-11T09:45:14.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.885+0000 7f348cf9b700 1 --2- 172.21.15.27:0/3713601578 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f346c066ea0 0x7f346c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:14.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.886+0000 7f348cf9b700 1 --2- 172.21.15.27:0/3713601578 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f346c066ea0 0x7f346c069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f347c006fd0 tx=0x7f347c008040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:14.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:14.889+0000 7f3484ff9700 1 -- 172.21.15.27:0/3713601578 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f348004e5a0 con 0x7f3488159200 2024-01-11T09:45:15.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:15.171+0000 7f348f1ff700 1 -- 172.21.15.27:0/3713601578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f34880718d0 con 0x7f3488159200 2024-01-11T09:45:15.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:15.216+0000 7f3484ff9700 1 -- 172.21.15.27:0/3713601578 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3480052270 con 0x7f3488159200 2024-01-11T09:45:16.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.086+0000 7f3484ff9700 1 -- 172.21.15.27:0/3713601578 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3079) v1 ==== 163+0+0 (secure 0 0 0) 0x7f3480055930 con 0x7f3488159200 2024-01-11T09:45:16.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.089+0000 7f348f1ff700 1 -- 172.21.15.27:0/3713601578 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f346c066ea0 msgr2=0x7f346c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:16.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.089+0000 7f348f1ff700 1 --2- 172.21.15.27:0/3713601578 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f346c066ea0 0x7f346c069350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f347c006fd0 tx=0x7f347c008040 comp rx=0 tx=0).stop 2024-01-11T09:45:16.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.089+0000 7f348f1ff700 1 -- 172.21.15.27:0/3713601578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3488159200 msgr2=0x7f34880b4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:16.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.089+0000 7f348f1ff700 1 --2- 172.21.15.27:0/3713601578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3488159200 0x7f34880b4700 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f348000ce40 tx=0x7f34800077a0 comp rx=0 tx=0).stop 2024-01-11T09:45:16.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.089+0000 7f348f1ff700 1 -- 172.21.15.27:0/3713601578 shutdown_connections 2024-01-11T09:45:16.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.089+0000 7f348f1ff700 1 --2- 172.21.15.27:0/3713601578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3488159200 0x7f34880b4700 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:16.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.089+0000 7f348f1ff700 1 --2- 172.21.15.27:0/3713601578 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f346c066ea0 0x7f346c069350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:16.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.089+0000 7f348f1ff700 1 --2- 172.21.15.27:0/3713601578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34880b03e0 0x7f348816c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:16.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.089+0000 7f348f1ff700 1 --2- 172.21.15.27:0/3713601578 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f34880b07b0 0x7f34880b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:16.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.089+0000 7f348f1ff700 1 -- 172.21.15.27:0/3713601578 >> 172.21.15.27:0/3713601578 conn(0x7f348806ed40 msgr2=0x7f34880b2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:16.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.089+0000 7f348f1ff700 1 -- 172.21.15.27:0/3713601578 shutdown_connections 2024-01-11T09:45:16.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.089+0000 7f348f1ff700 1 -- 172.21.15.27:0/3713601578 wait complete. 2024-01-11T09:45:16.101 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... ok 2024-01-11T09:45:16.102 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command succes with --fs argument for non-default fs 2024-01-11T09:45:16.102 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2024-01-11T09:45:16.102 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2024-01-11T09:45:16.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.314+0000 7ff6003ff700 1 Processor -- start 2024-01-11T09:45:16.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.314+0000 7ff6003ff700 1 -- start start 2024-01-11T09:45:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.316+0000 7ff6003ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5f81596c0 0x7ff5f815d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.316+0000 7ff6003ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff5f8068d30 0x7ff5f80aa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.316+0000 7ff6003ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f80a9cb0 con 0x7ff5f80ca670 2024-01-11T09:45:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.316+0000 7ff6003ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f815e210 con 0x7ff5f8068d30 2024-01-11T09:45:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.316+0000 7ff6003ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f815dc50 con 0x7ff5f81596c0 2024-01-11T09:45:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.316+0000 7ff5fe99c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff5f8068d30 0x7ff5f80aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.316+0000 7ff5fe99c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff5f8068d30 0x7ff5f80aa140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49708/0 (socket says 172.21.15.27:49708) 2024-01-11T09:45:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.316+0000 7ff5fe99c700 1 -- 172.21.15.27:0/681517736 learned_addr learned my addr 172.21.15.27:0/681517736 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.316+0000 7ff5fe19b700 1 --2- 172.21.15.27:0/681517736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5f81596c0 0x7ff5f815d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.317+0000 7ff5fe99c700 1 -- 172.21.15.27:0/681517736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5f81596c0 msgr2=0x7ff5f815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.317+0000 7ff5fe99c700 1 --2- 172.21.15.27:0/681517736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5f81596c0 0x7ff5f815d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.317+0000 7ff5fe99c700 1 -- 172.21.15.27:0/681517736 >> v1:172.21.15.77:6789/0 conn(0x7ff5f80ca670 legacy=0x7ff5f8068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:16.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.317+0000 7ff5fe99c700 1 -- 172.21.15.27:0/681517736 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f815df30 con 0x7ff5f8068d30 2024-01-11T09:45:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.317+0000 7ff5fe99c700 1 --2- 172.21.15.27:0/681517736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f8068d30 0x7ff5f80aa140 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009f20 tx=0x7ff5f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b4dcbf6b3e8424 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.317+0000 7ff5fd199700 1 -- 172.21.15.27:0/681517736 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f000e070 con 0x7ff5f8068d30 2024-01-11T09:45:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.317+0000 7ff5fd199700 1 -- 172.21.15.27:0/681517736 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f000ba80 con 0x7ff5f8068d30 2024-01-11T09:45:16.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.317+0000 7ff5fd199700 1 -- 172.21.15.27:0/681517736 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f00092e0 con 0x7ff5f8068d30 2024-01-11T09:45:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.318+0000 7ff6003ff700 1 -- 172.21.15.27:0/681517736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f8068d30 msgr2=0x7ff5f80aa140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.318+0000 7ff6003ff700 1 --2- 172.21.15.27:0/681517736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f8068d30 0x7ff5f80aa140 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009f20 tx=0x7ff5f0009840 comp rx=0 tx=0).stop 2024-01-11T09:45:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.318+0000 7ff6003ff700 1 -- 172.21.15.27:0/681517736 shutdown_connections 2024-01-11T09:45:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.318+0000 7ff6003ff700 1 --2- 172.21.15.27:0/681517736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f8068d30 0x7ff5f80aa140 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.318+0000 7ff6003ff700 1 --2- 172.21.15.27:0/681517736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5f81596c0 0x7ff5f815d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.318+0000 7ff6003ff700 1 -- 172.21.15.27:0/681517736 >> 172.21.15.27:0/681517736 conn(0x7ff5f80c9f70 msgr2=0x7ff5f806deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.318+0000 7ff6003ff700 1 -- 172.21.15.27:0/681517736 shutdown_connections 2024-01-11T09:45:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.318+0000 7ff6003ff700 1 -- 172.21.15.27:0/681517736 wait complete. 2024-01-11T09:45:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.319+0000 7ff6003ff700 1 Processor -- start 2024-01-11T09:45:16.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.320+0000 7ff6003ff700 1 -- start start 2024-01-11T09:45:16.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.320+0000 7ff6003ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f8068d30 0x7ff5f80ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:16.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.320+0000 7ff6003ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5f80ca670 0x7ff5f81583f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:16.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.321+0000 7ff6003ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5f81596c0 0x7ff5f8158930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:16.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.321+0000 7ff6003ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f80ad450 con 0x7ff5f8068d30 2024-01-11T09:45:16.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.321+0000 7ff6003ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f80ac840 con 0x7ff5f81596c0 2024-01-11T09:45:16.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.321+0000 7ff6003ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f80acb20 con 0x7ff5f80ca670 2024-01-11T09:45:16.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.321+0000 7ff5fe19b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f8068d30 0x7ff5f80ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:16.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.321+0000 7ff5fe19b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f8068d30 0x7ff5f80ae280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49710/0 (socket says 172.21.15.27:49710) 2024-01-11T09:45:16.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.321+0000 7ff5fe19b700 1 -- 172.21.15.27:0/3584236984 learned_addr learned my addr 172.21.15.27:0/3584236984 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:16.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.321+0000 7ff5fe99c700 1 --2- 172.21.15.27:0/3584236984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5f81596c0 0x7ff5f8158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:16.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.321+0000 7ff5fd99a700 1 --2- 172.21.15.27:0/3584236984 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5f80ca670 0x7ff5f81583f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:16.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.321+0000 7ff5fe19b700 1 -- 172.21.15.27:0/3584236984 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5f80ca670 msgr2=0x7ff5f81583f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:16.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.321+0000 7ff5fe19b700 1 --2- 172.21.15.27:0/3584236984 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5f80ca670 0x7ff5f81583f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:16.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.321+0000 7ff5fe19b700 1 -- 172.21.15.27:0/3584236984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5f81596c0 msgr2=0x7ff5f8158930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:16.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.321+0000 7ff5fe19b700 1 --2- 172.21.15.27:0/3584236984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5f81596c0 0x7ff5f8158930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:16.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.321+0000 7ff5fe19b700 1 -- 172.21.15.27:0/3584236984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f0009ae0 con 0x7ff5f8068d30 2024-01-11T09:45:16.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.321+0000 7ff5fd99a700 1 --2- 172.21.15.27:0/3584236984 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5f80ca670 0x7ff5f81583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:16.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.322+0000 7ff5fe19b700 1 --2- 172.21.15.27:0/3584236984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f8068d30 0x7ff5f80ae280 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400d860 tx=0x7ff5f4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:16.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.322+0000 7ff5ee7fc700 1 -- 172.21.15.27:0/3584236984 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f40125f0 con 0x7ff5f8068d30 2024-01-11T09:45:16.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.322+0000 7ff5ee7fc700 1 -- 172.21.15.27:0/3584236984 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f4012770 con 0x7ff5f8068d30 2024-01-11T09:45:16.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.322+0000 7ff5ee7fc700 1 -- 172.21.15.27:0/3584236984 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4012b30 con 0x7ff5f8068d30 2024-01-11T09:45:16.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.322+0000 7ff6003ff700 1 -- 172.21.15.27:0/3584236984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f80ace00 con 0x7ff5f8068d30 2024-01-11T09:45:16.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.322+0000 7ff6003ff700 1 -- 172.21.15.27:0/3584236984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f80ae7c0 con 0x7ff5f8068d30 2024-01-11T09:45:16.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.323+0000 7ff6003ff700 1 -- 172.21.15.27:0/3584236984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f80718d0 con 0x7ff5f8068d30 2024-01-11T09:45:16.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.324+0000 7ff5ee7fc700 1 -- 172.21.15.27:0/3584236984 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff5f401e460 con 0x7ff5f8068d30 2024-01-11T09:45:16.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.325+0000 7ff5ee7fc700 1 --2- 172.21.15.27:0/3584236984 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff5dc066f30 0x7ff5dc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:16.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.325+0000 7ff5ee7fc700 1 -- 172.21.15.27:0/3584236984 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 5653+0+0 (secure 0 0 0) 0x7ff5f4084bd0 con 0x7ff5f8068d30 2024-01-11T09:45:16.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.325+0000 7ff5fd99a700 1 --2- 172.21.15.27:0/3584236984 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff5dc066f30 0x7ff5dc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:16.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.326+0000 7ff5fd99a700 1 --2- 172.21.15.27:0/3584236984 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff5dc066f30 0x7ff5dc0693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8007fd0 tx=0x7ff5e8009040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:16.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.329+0000 7ff5ee7fc700 1 -- 172.21.15.27:0/3584236984 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff5f4051dd0 con 0x7ff5f8068d30 2024-01-11T09:45:16.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:16.611+0000 7ff6003ff700 1 -- 172.21.15.27:0/3584236984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7ff5f80a9940 con 0x7ff5f8068d30 2024-01-11T09:45:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.091+0000 7ff5ee7fc700 1 -- 172.21.15.27:0/3584236984 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3080) v1 ==== 189+0+0 (secure 0 0 0) 0x7ff5f40553f0 con 0x7ff5f8068d30 2024-01-11T09:45:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.093+0000 7ff6003ff700 1 -- 172.21.15.27:0/3584236984 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff5dc066f30 msgr2=0x7ff5dc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.093+0000 7ff6003ff700 1 --2- 172.21.15.27:0/3584236984 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff5dc066f30 0x7ff5dc0693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8007fd0 tx=0x7ff5e8009040 comp rx=0 tx=0).stop 2024-01-11T09:45:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.093+0000 7ff6003ff700 1 -- 172.21.15.27:0/3584236984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f8068d30 msgr2=0x7ff5f80ae280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.093+0000 7ff6003ff700 1 --2- 172.21.15.27:0/3584236984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f8068d30 0x7ff5f80ae280 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400d860 tx=0x7ff5f4008040 comp rx=0 tx=0).stop 2024-01-11T09:45:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.093+0000 7ff6003ff700 1 -- 172.21.15.27:0/3584236984 shutdown_connections 2024-01-11T09:45:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.093+0000 7ff6003ff700 1 --2- 172.21.15.27:0/3584236984 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff5dc066f30 0x7ff5dc0693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.093+0000 7ff6003ff700 1 --2- 172.21.15.27:0/3584236984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5f81596c0 0x7ff5f8158930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.093+0000 7ff6003ff700 1 --2- 172.21.15.27:0/3584236984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f8068d30 0x7ff5f80ae280 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.093+0000 7ff6003ff700 1 --2- 172.21.15.27:0/3584236984 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5f80ca670 0x7ff5f81583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.093+0000 7ff6003ff700 1 -- 172.21.15.27:0/3584236984 >> 172.21.15.27:0/3584236984 conn(0x7ff5f80c9f70 msgr2=0x7ff5f80aa730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.094+0000 7ff6003ff700 1 -- 172.21.15.27:0/3584236984 shutdown_connections 2024-01-11T09:45:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.094+0000 7ff6003ff700 1 -- 172.21.15.27:0/3584236984 wait complete. 2024-01-11T09:45:17.105 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-11T09:45:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.316+0000 7fdc4e5ff700 1 Processor -- start 2024-01-11T09:45:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.317+0000 7fdc4e5ff700 1 -- start start 2024-01-11T09:45:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.318+0000 7fdc4e5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc48157410 0x7fdc480ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.318+0000 7fdc4e5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc481577e0 0x7fdc480ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.318+0000 7fdc4e5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc480699d0 con 0x7fdc48069540 2024-01-11T09:45:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.318+0000 7fdc4e5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc48069cb0 con 0x7fdc48157410 2024-01-11T09:45:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.318+0000 7fdc4e5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc48069f90 con 0x7fdc481577e0 2024-01-11T09:45:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.318+0000 7fdc47fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc48157410 0x7fdc480ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.318+0000 7fdc47fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc48157410 0x7fdc480ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49720/0 (socket says 172.21.15.27:49720) 2024-01-11T09:45:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.318+0000 7fdc4cb9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdc48069540 0x7fdc4815b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40746/0 (socket says 172.21.15.27:40746) 2024-01-11T09:45:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.318+0000 7fdc47fff700 1 -- 172.21.15.27:0/992312068 learned_addr learned my addr 172.21.15.27:0/992312068 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.319+0000 7fdc477fe700 1 --2- 172.21.15.27:0/992312068 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc481577e0 0x7fdc480ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.319+0000 7fdc47fff700 1 -- 172.21.15.27:0/992312068 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc481577e0 msgr2=0x7fdc480ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.319+0000 7fdc47fff700 1 --2- 172.21.15.27:0/992312068 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc481577e0 0x7fdc480ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.319+0000 7fdc47fff700 1 -- 172.21.15.27:0/992312068 >> v1:172.21.15.27:6789/0 conn(0x7fdc48069540 legacy=0x7fdc4815b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.319+0000 7fdc47fff700 1 -- 172.21.15.27:0/992312068 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4806a270 con 0x7fdc48157410 2024-01-11T09:45:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.319+0000 7fdc46ffd700 1 -- 172.21.15.27:0/992312068 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834118313 0 0) 0x7fdc480699d0 con 0x7fdc48069540 2024-01-11T09:45:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.319+0000 7fdc47fff700 1 --2- 172.21.15.27:0/992312068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc48157410 0x7fdc480ccae0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fdc380099a0 tx=0x7fdc38009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce853251db653c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.319+0000 7fdc46ffd700 1 -- 172.21.15.27:0/992312068 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc38009e90 con 0x7fdc48157410 2024-01-11T09:45:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.319+0000 7fdc46ffd700 1 -- 172.21.15.27:0/992312068 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3800b860 con 0x7fdc48157410 2024-01-11T09:45:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.319+0000 7fdc46ffd700 1 -- 172.21.15.27:0/992312068 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3800bc20 con 0x7fdc48157410 2024-01-11T09:45:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.320+0000 7fdc4e5ff700 1 -- 172.21.15.27:0/992312068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc48157410 msgr2=0x7fdc480ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.320+0000 7fdc4e5ff700 1 --2- 172.21.15.27:0/992312068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc48157410 0x7fdc480ccae0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fdc380099a0 tx=0x7fdc38009250 comp rx=0 tx=0).stop 2024-01-11T09:45:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.320+0000 7fdc4e5ff700 1 -- 172.21.15.27:0/992312068 shutdown_connections 2024-01-11T09:45:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.320+0000 7fdc4e5ff700 1 --2- 172.21.15.27:0/992312068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc48157410 0x7fdc480ccae0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.320+0000 7fdc4e5ff700 1 --2- 172.21.15.27:0/992312068 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc481577e0 0x7fdc480ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.320+0000 7fdc4e5ff700 1 -- 172.21.15.27:0/992312068 >> 172.21.15.27:0/992312068 conn(0x7fdc480680a0 msgr2=0x7fdc480cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.320+0000 7fdc4e5ff700 1 -- 172.21.15.27:0/992312068 shutdown_connections 2024-01-11T09:45:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.320+0000 7fdc4e5ff700 1 -- 172.21.15.27:0/992312068 wait complete. 2024-01-11T09:45:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.322+0000 7fdc4e5ff700 1 Processor -- start 2024-01-11T09:45:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.322+0000 7fdc4e5ff700 1 -- start start 2024-01-11T09:45:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.323+0000 7fdc4e5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc48069540 0x7fdc4816d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.323+0000 7fdc4e5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc48157410 0x7fdc4814d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.323+0000 7fdc4e5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc481577e0 0x7fdc4814be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.323+0000 7fdc4e5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4816da60 con 0x7fdc48157410 2024-01-11T09:45:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.323+0000 7fdc4e5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc480b1fa0 con 0x7fdc48069540 2024-01-11T09:45:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.323+0000 7fdc4e5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc48157e50 con 0x7fdc481577e0 2024-01-11T09:45:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.323+0000 7fdc477fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc48157410 0x7fdc4814d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.323+0000 7fdc477fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc48157410 0x7fdc4814d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49730/0 (socket says 172.21.15.27:49730) 2024-01-11T09:45:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.323+0000 7fdc477fe700 1 -- 172.21.15.27:0/2846549260 learned_addr learned my addr 172.21.15.27:0/2846549260 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.323+0000 7fdc47fff700 1 --2- 172.21.15.27:0/2846549260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc48069540 0x7fdc4816d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.323+0000 7fdc4cb9c700 1 --2- 172.21.15.27:0/2846549260 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc481577e0 0x7fdc4814be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.323+0000 7fdc477fe700 1 -- 172.21.15.27:0/2846549260 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc481577e0 msgr2=0x7fdc4814be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.323+0000 7fdc477fe700 1 --2- 172.21.15.27:0/2846549260 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc481577e0 0x7fdc4814be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.323+0000 7fdc477fe700 1 -- 172.21.15.27:0/2846549260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc48069540 msgr2=0x7fdc4816d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.323+0000 7fdc477fe700 1 --2- 172.21.15.27:0/2846549260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc48069540 0x7fdc4816d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.323+0000 7fdc477fe700 1 -- 172.21.15.27:0/2846549260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc38009490 con 0x7fdc48157410 2024-01-11T09:45:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.323+0000 7fdc4cb9c700 1 --2- 172.21.15.27:0/2846549260 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc481577e0 0x7fdc4814be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.323+0000 7fdc47fff700 1 --2- 172.21.15.27:0/2846549260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc48069540 0x7fdc4816d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:17.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.323+0000 7fdc477fe700 1 --2- 172.21.15.27:0/2846549260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc48157410 0x7fdc4814d030 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c00e900 tx=0x7fdc3c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:17.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.324+0000 7fdc2ffff700 1 -- 172.21.15.27:0/2846549260 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c00eea0 con 0x7fdc48157410 2024-01-11T09:45:17.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.324+0000 7fdc2ffff700 1 -- 172.21.15.27:0/2846549260 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3c00c850 con 0x7fdc48157410 2024-01-11T09:45:17.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.324+0000 7fdc4e5ff700 1 -- 172.21.15.27:0/2846549260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc480b1390 con 0x7fdc48157410 2024-01-11T09:45:17.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.324+0000 7fdc2ffff700 1 -- 172.21.15.27:0/2846549260 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c00cc10 con 0x7fdc48157410 2024-01-11T09:45:17.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.324+0000 7fdc4e5ff700 1 -- 172.21.15.27:0/2846549260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc480b19d0 con 0x7fdc48157410 2024-01-11T09:45:17.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.325+0000 7fdc4e5ff700 1 -- 172.21.15.27:0/2846549260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc480af720 con 0x7fdc48157410 2024-01-11T09:45:17.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.326+0000 7fdc2ffff700 1 -- 172.21.15.27:0/2846549260 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fdc3c005a00 con 0x7fdc48157410 2024-01-11T09:45:17.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.327+0000 7fdc2ffff700 1 --2- 172.21.15.27:0/2846549260 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fdc34066f30 0x7fdc340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:17.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.327+0000 7fdc2ffff700 1 -- 172.21.15.27:0/2846549260 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 5653+0+0 (secure 0 0 0) 0x7fdc3c084450 con 0x7fdc48157410 2024-01-11T09:45:17.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.327+0000 7fdc47fff700 1 --2- 172.21.15.27:0/2846549260 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fdc34066f30 0x7fdc340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:17.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.328+0000 7fdc47fff700 1 --2- 172.21.15.27:0/2846549260 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fdc34066f30 0x7fdc340693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdc380099a0 tx=0x7fdc38017000 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:17.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.331+0000 7fdc2ffff700 1 -- 172.21.15.27:0/2846549260 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdc480af720 con 0x7fdc48157410 2024-01-11T09:45:17.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.535+0000 7fdc4e5ff700 1 -- 172.21.15.27:0/2846549260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fdc480af720 con 0x7fdc48157410 2024-01-11T09:45:17.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.541+0000 7fdc2ffff700 1 -- 172.21.15.27:0/2846549260 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v76)=0 v76) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdc480af720 con 0x7fdc48157410 2024-01-11T09:45:17.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.543+0000 7fdc4e5ff700 1 -- 172.21.15.27:0/2846549260 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fdc34066f30 msgr2=0x7fdc340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:17.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.543+0000 7fdc4e5ff700 1 --2- 172.21.15.27:0/2846549260 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fdc34066f30 0x7fdc340693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdc380099a0 tx=0x7fdc38017000 comp rx=0 tx=0).stop 2024-01-11T09:45:17.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.543+0000 7fdc4e5ff700 1 -- 172.21.15.27:0/2846549260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc48157410 msgr2=0x7fdc4814d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:17.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.543+0000 7fdc4e5ff700 1 --2- 172.21.15.27:0/2846549260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc48157410 0x7fdc4814d030 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c00e900 tx=0x7fdc3c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:17.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.543+0000 7fdc4e5ff700 1 -- 172.21.15.27:0/2846549260 shutdown_connections 2024-01-11T09:45:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.543+0000 7fdc4e5ff700 1 --2- 172.21.15.27:0/2846549260 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdc481577e0 0x7fdc4814be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.543+0000 7fdc4e5ff700 1 --2- 172.21.15.27:0/2846549260 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fdc34066f30 0x7fdc340693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.543+0000 7fdc4e5ff700 1 --2- 172.21.15.27:0/2846549260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc48069540 0x7fdc4816d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.543+0000 7fdc4e5ff700 1 --2- 172.21.15.27:0/2846549260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc48157410 0x7fdc4814d030 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.543+0000 7fdc4e5ff700 1 -- 172.21.15.27:0/2846549260 >> 172.21.15.27:0/2846549260 conn(0x7fdc480680a0 msgr2=0x7fdc4814e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.543+0000 7fdc4e5ff700 1 -- 172.21.15.27:0/2846549260 shutdown_connections 2024-01-11T09:45:17.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.543+0000 7fdc4e5ff700 1 -- 172.21.15.27:0/2846549260 wait complete. 2024-01-11T09:45:17.553 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T09:45:17.553 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:45:17.649 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:45:17.749 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T09:45:17.749 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-11T09:45:17.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:45:17.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.957+0000 7f2ff0bff700 1 Processor -- start 2024-01-11T09:45:17.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.957+0000 7f2ff0bff700 1 -- start start 2024-01-11T09:45:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.958+0000 7f2ff0bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2fec0c8620 0x7f2fec0af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.958+0000 7f2ff0bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2fec0b09e0 0x7f2fec0accc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.958+0000 7f2ff0bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fec157390 con 0x7f2fec069d40 2024-01-11T09:45:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.958+0000 7f2ff0bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fec0afeb0 con 0x7f2fec0b09e0 2024-01-11T09:45:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.958+0000 7f2ff0bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec06a200 con 0x7f2fec0c8620 2024-01-11T09:45:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.958+0000 7f2fead9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2fec0b09e0 0x7f2fec0accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.959+0000 7f2fead9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2fec0b09e0 0x7f2fec0accc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49746/0 (socket says 172.21.15.27:49746) 2024-01-11T09:45:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.959+0000 7f2fead9d700 1 -- 172.21.15.27:0/2972261135 learned_addr learned my addr 172.21.15.27:0/2972261135 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.958+0000 7f2fe9d9b700 1 --1- 172.21.15.27:0/2972261135 >> v1:172.21.15.27:6789/0 conn(0x7f2fec069d40 0x7f2fec0b02d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40752/0 (socket says 172.21.15.27:40752) 2024-01-11T09:45:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.959+0000 7f2fea59c700 1 --2- 172.21.15.27:0/2972261135 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2fec0c8620 0x7f2fec0af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.959+0000 7f2fead9d700 1 -- 172.21.15.27:0/2972261135 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2fec0c8620 msgr2=0x7f2fec0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.959+0000 7f2fead9d700 1 --2- 172.21.15.27:0/2972261135 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2fec0c8620 0x7f2fec0af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.959+0000 7f2fead9d700 1 -- 172.21.15.27:0/2972261135 >> v1:172.21.15.27:6789/0 conn(0x7f2fec069d40 legacy=0x7f2fec0b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.959+0000 7f2fead9d700 1 -- 172.21.15.27:0/2972261135 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec156d40 con 0x7f2fec0b09e0 2024-01-11T09:45:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.959+0000 7f2fe959a700 1 -- 172.21.15.27:0/2972261135 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715632598 0 0) 0x7f2fec157390 con 0x7f2fec069d40 2024-01-11T09:45:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.959+0000 7f2fead9d700 1 --2- 172.21.15.27:0/2972261135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fec0b09e0 0x7f2fec0accc0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400d1a0 tx=0x7f2fe400ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de23ea0c17298177 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.960+0000 7f2fe959a700 1 -- 172.21.15.27:0/2972261135 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4010070 con 0x7f2fec0b09e0 2024-01-11T09:45:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.960+0000 7f2fe959a700 1 -- 172.21.15.27:0/2972261135 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe400de10 con 0x7f2fec0b09e0 2024-01-11T09:45:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.960+0000 7f2fe959a700 1 -- 172.21.15.27:0/2972261135 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe400b320 con 0x7f2fec0b09e0 2024-01-11T09:45:17.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.960+0000 7f2ff0bff700 1 -- 172.21.15.27:0/2972261135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fec0b09e0 msgr2=0x7f2fec0accc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:17.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.960+0000 7f2ff0bff700 1 --2- 172.21.15.27:0/2972261135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fec0b09e0 0x7f2fec0accc0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400d1a0 tx=0x7f2fe400ba50 comp rx=0 tx=0).stop 2024-01-11T09:45:17.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.960+0000 7f2ff0bff700 1 -- 172.21.15.27:0/2972261135 shutdown_connections 2024-01-11T09:45:17.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.960+0000 7f2ff0bff700 1 --2- 172.21.15.27:0/2972261135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fec0b09e0 0x7f2fec0accc0 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:17.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.960+0000 7f2ff0bff700 1 --2- 172.21.15.27:0/2972261135 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2fec0c8620 0x7f2fec0af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:17.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.960+0000 7f2ff0bff700 1 -- 172.21.15.27:0/2972261135 >> 172.21.15.27:0/2972261135 conn(0x7f2fec0680e0 msgr2=0x7f2fec069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:17.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.960+0000 7f2ff0bff700 1 -- 172.21.15.27:0/2972261135 shutdown_connections 2024-01-11T09:45:17.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.960+0000 7f2ff0bff700 1 -- 172.21.15.27:0/2972261135 wait complete. 2024-01-11T09:45:17.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.962+0000 7f2ff0bff700 1 Processor -- start 2024-01-11T09:45:17.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.962+0000 7f2ff0bff700 1 -- start start 2024-01-11T09:45:17.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.963+0000 7f2ff0bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fec069d40 0x7f2fec169940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:17.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.963+0000 7f2ff0bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fec0b09e0 0x7f2fec145d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:17.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.963+0000 7f2ff0bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2fec0c8620 0x7f2fec15c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:17.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.963+0000 7f2ff0bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec152250 con 0x7f2fec069d40 2024-01-11T09:45:17.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.963+0000 7f2ff0bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec146260 con 0x7f2fec0b09e0 2024-01-11T09:45:17.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.963+0000 7f2ff0bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec16d050 con 0x7f2fec0c8620 2024-01-11T09:45:17.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.963+0000 7f2fea59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fec069d40 0x7f2fec169940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.963+0000 7f2fea59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fec069d40 0x7f2fec169940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49762/0 (socket says 172.21.15.27:49762) 2024-01-11T09:45:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.963+0000 7f2fea59c700 1 -- 172.21.15.27:0/1008812649 learned_addr learned my addr 172.21.15.27:0/1008812649 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.963+0000 7f2fe9d9b700 1 --2- 172.21.15.27:0/1008812649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fec0b09e0 0x7f2fec145d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.964+0000 7f2fead9d700 1 --2- 172.21.15.27:0/1008812649 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2fec0c8620 0x7f2fec15c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.964+0000 7f2fea59c700 1 -- 172.21.15.27:0/1008812649 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2fec0c8620 msgr2=0x7f2fec15c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.964+0000 7f2fea59c700 1 --2- 172.21.15.27:0/1008812649 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2fec0c8620 0x7f2fec15c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.964+0000 7f2fea59c700 1 -- 172.21.15.27:0/1008812649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fec0b09e0 msgr2=0x7f2fec145d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.964+0000 7f2fea59c700 1 --2- 172.21.15.27:0/1008812649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fec0b09e0 0x7f2fec145d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.964+0000 7f2fea59c700 1 -- 172.21.15.27:0/1008812649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe400bcf0 con 0x7f2fec069d40 2024-01-11T09:45:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.964+0000 7f2fea59c700 1 --2- 172.21.15.27:0/1008812649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fec069d40 0x7f2fec169940 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00e930 tx=0x7f2fdc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.964+0000 7f2fda7fc700 1 -- 172.21.15.27:0/1008812649 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc00eed0 con 0x7f2fec069d40 2024-01-11T09:45:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.964+0000 7f2fda7fc700 1 -- 172.21.15.27:0/1008812649 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fdc00c850 con 0x7f2fec069d40 2024-01-11T09:45:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.964+0000 7f2ff0bff700 1 -- 172.21.15.27:0/1008812649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fec169f50 con 0x7f2fec069d40 2024-01-11T09:45:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.964+0000 7f2fda7fc700 1 -- 172.21.15.27:0/1008812649 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc00cbf0 con 0x7f2fec069d40 2024-01-11T09:45:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.964+0000 7f2ff0bff700 1 -- 172.21.15.27:0/1008812649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fec16a510 con 0x7f2fec069d40 2024-01-11T09:45:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.965+0000 7f2ff0bff700 1 -- 172.21.15.27:0/1008812649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fec16a850 con 0x7f2fec069d40 2024-01-11T09:45:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.966+0000 7f2fda7fc700 1 -- 172.21.15.27:0/1008812649 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2fdc013470 con 0x7f2fec069d40 2024-01-11T09:45:17.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.967+0000 7f2fda7fc700 1 --2- 172.21.15.27:0/1008812649 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2fcc066f30 0x7f2fcc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:17.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.967+0000 7f2fda7fc700 1 -- 172.21.15.27:0/1008812649 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 5653+0+0 (secure 0 0 0) 0x7f2fdc0846f0 con 0x7f2fec069d40 2024-01-11T09:45:17.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.968+0000 7f2fe9d9b700 1 --2- 172.21.15.27:0/1008812649 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2fcc066f30 0x7f2fcc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:17.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.968+0000 7f2fe9d9b700 1 --2- 172.21.15.27:0/1008812649 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2fcc066f30 0x7f2fcc0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2fec169e80 tx=0x7f2fe0007bf0 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:17.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:17.971+0000 7f2fda7fc700 1 -- 172.21.15.27:0/1008812649 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2fdc051990 con 0x7f2fec069d40 2024-01-11T09:45:18.176 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:45:18.181 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:45:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.221+0000 7f2ff0bff700 1 -- 172.21.15.27:0/1008812649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fec0718d0 con 0x7f2fec069d40 2024-01-11T09:45:18.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.222+0000 7f2fda7fc700 1 -- 172.21.15.27:0/1008812649 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3529 (secure 0 0 0) 0x7f2fdc017de0 con 0x7f2fec069d40 2024-01-11T09:45:18.214 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:18.214 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":104,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8441,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/991606883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":991606883},{"type":"v1","addr":"172.21.15.77:6834","nonce":991606883}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9733,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1996818074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1996818074},{"type":"v1","addr":"172.21.15.77:6835","nonce":1996818074}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":9742,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4117588854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":4117588854},{"type":"v1","addr":"172.21.15.27:6835","nonce":4117588854}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97},{"gid":10473,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1222757256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1222757256},{"type":"v1","addr":"172.21.15.27:6834","nonce":1222757256}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":104}],"filesystems":[]} 2024-01-11T09:45:18.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.224+0000 7f2ff0bff700 1 -- 172.21.15.27:0/1008812649 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2fcc066f30 msgr2=0x7f2fcc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:18.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.224+0000 7f2ff0bff700 1 --2- 172.21.15.27:0/1008812649 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2fcc066f30 0x7f2fcc0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2fec169e80 tx=0x7f2fe0007bf0 comp rx=0 tx=0).stop 2024-01-11T09:45:18.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.224+0000 7f2ff0bff700 1 -- 172.21.15.27:0/1008812649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fec069d40 msgr2=0x7f2fec169940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:18.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.224+0000 7f2ff0bff700 1 --2- 172.21.15.27:0/1008812649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fec069d40 0x7f2fec169940 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00e930 tx=0x7f2fdc0091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:18.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.224+0000 7f2ff0bff700 1 -- 172.21.15.27:0/1008812649 shutdown_connections 2024-01-11T09:45:18.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.224+0000 7f2ff0bff700 1 --2- 172.21.15.27:0/1008812649 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2fec0c8620 0x7f2fec15c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:18.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.224+0000 7f2ff0bff700 1 --2- 172.21.15.27:0/1008812649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fec069d40 0x7f2fec169940 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:18.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.224+0000 7f2ff0bff700 1 --2- 172.21.15.27:0/1008812649 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2fcc066f30 0x7f2fcc0693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:18.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.224+0000 7f2ff0bff700 1 --2- 172.21.15.27:0/1008812649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fec0b09e0 0x7f2fec145d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:18.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.224+0000 7f2ff0bff700 1 -- 172.21.15.27:0/1008812649 >> 172.21.15.27:0/1008812649 conn(0x7f2fec0680e0 msgr2=0x7f2fec0695b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:18.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.224+0000 7f2ff0bff700 1 -- 172.21.15.27:0/1008812649 shutdown_connections 2024-01-11T09:45:18.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.224+0000 7f2ff0bff700 1 -- 172.21.15.27:0/1008812649 wait complete. 2024-01-11T09:45:18.217 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 104 2024-01-11T09:45:18.236 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-11T09:45:18.236 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-11T09:45:18.236 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-11T09:45:18.236 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:45:18.237 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-11T09:45:18.237 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-11T09:45:18.237 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-11T09:45:18.237 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:45:18.237 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-11T09:45:18.237 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-11T09:45:18.237 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-11T09:45:18.237 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:45:18.238 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-11T09:45:18.238 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-11T09:45:18.238 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-11T09:45:18.238 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:45:18.253 INFO:tasks.ceph.mds.b:Stopped 2024-01-11T09:45:18.253 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-11T09:45:18.256 INFO:tasks.ceph.mds.d:Stopped 2024-01-11T09:45:18.256 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-11T09:45:18.287 INFO:tasks.ceph.mds.a:Stopped 2024-01-11T09:45:18.288 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-11T09:45:18.289 INFO:tasks.ceph.mds.c:Stopped 2024-01-11T09:45:18.289 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-11T09:45:18.300 INFO:tasks.ceph.mds.b:Started 2024-01-11T09:45:18.300 DEBUG:teuthology.parallel:result is None 2024-01-11T09:45:18.303 INFO:tasks.ceph.mds.d:Started 2024-01-11T09:45:18.303 DEBUG:teuthology.parallel:result is None 2024-01-11T09:45:18.333 INFO:tasks.ceph.mds.a:Started 2024-01-11T09:45:18.333 DEBUG:teuthology.parallel:result is None 2024-01-11T09:45:18.336 INFO:tasks.ceph.mds.c:Started 2024-01-11T09:45:18.336 DEBUG:teuthology.parallel:result is None 2024-01-11T09:45:18.336 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-11T09:45:18.443 INFO:tasks.ceph.mds.b.smithi077.stdout:starting mds.b at 2024-01-11T09:45:18.443 INFO:tasks.ceph.mds.d.smithi077.stdout:starting mds.d at 2024-01-11T09:45:18.488 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T09:45:18.489 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T09:45:18.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.562+0000 7f00183ff700 1 Processor -- start 2024-01-11T09:45:18.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.562+0000 7f00183ff700 1 -- start start 2024-01-11T09:45:18.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.563+0000 7f00183ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00100b0a40 0x7f001015da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:18.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.563+0000 7f00183ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f001015df40 0x7f0010068720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:18.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.563+0000 7f00183ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001015d290 con 0x7f001006bf10 2024-01-11T09:45:18.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.563+0000 7f00183ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f001015d570 con 0x7f001015df40 2024-01-11T09:45:18.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.563+0000 7f00183ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0010154180 con 0x7f00100b0a40 2024-01-11T09:45:18.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.564+0000 7f001619b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f001006bf10 0x7f0010159c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:49434/0 (socket says 172.21.15.27:49434) 2024-01-11T09:45:18.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.564+0000 7f001619b700 1 -- 172.21.15.27:0/141746919 learned_addr learned my addr 172.21.15.27:0/141746919 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:18.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.564+0000 7f001699c700 1 --2- 172.21.15.27:0/141746919 >> v2:172.21.15.77:3300/0 conn(0x7f001015df40 0x7f0010068720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:18.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.564+0000 7f001599a700 1 --2- 172.21.15.27:0/141746919 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00100b0a40 0x7f001015da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:18.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.564+0000 7f0015199700 1 -- 172.21.15.27:0/141746919 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500273161 0 0) 0x7f001015d290 con 0x7f001006bf10 2024-01-11T09:45:18.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.565+0000 7f0015199700 1 -- 172.21.15.27:0/141746919 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efffc003570 con 0x7f001006bf10 2024-01-11T09:45:18.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.565+0000 7f001699c700 1 -- 172.21.15.27:0/141746919 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00100b0a40 msgr2=0x7f001015da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:18.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.565+0000 7f001699c700 1 --2- 172.21.15.27:0/141746919 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00100b0a40 0x7f001015da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:18.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.565+0000 7f001699c700 1 -- 172.21.15.27:0/141746919 >> v1:172.21.15.77:6789/0 conn(0x7f001006bf10 legacy=0x7f0010159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:18.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.565+0000 7f001699c700 1 -- 172.21.15.27:0/141746919 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001015a510 con 0x7f001015df40 2024-01-11T09:45:18.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.565+0000 7f001599a700 1 --2- 172.21.15.27:0/141746919 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00100b0a40 0x7f001015da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:18.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.565+0000 7f001699c700 1 --2- 172.21.15.27:0/141746919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f001015df40 0x7f0010068720 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f0008009e60 tx=0x7f0008009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8972bed172d2b597 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:18.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.565+0000 7f0015199700 1 -- 172.21.15.27:0/141746919 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000800e070 con 0x7f001015df40 2024-01-11T09:45:18.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.565+0000 7f0015199700 1 -- 172.21.15.27:0/141746919 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000800bab0 con 0x7f001015df40 2024-01-11T09:45:18.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.565+0000 7f0015199700 1 -- 172.21.15.27:0/141746919 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000800be70 con 0x7f001015df40 2024-01-11T09:45:18.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.566+0000 7f00183ff700 1 -- 172.21.15.27:0/141746919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f001015df40 msgr2=0x7f0010068720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:18.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.566+0000 7f00183ff700 1 --2- 172.21.15.27:0/141746919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f001015df40 0x7f0010068720 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f0008009e60 tx=0x7f0008009780 comp rx=0 tx=0).stop 2024-01-11T09:45:18.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.566+0000 7f00183ff700 1 -- 172.21.15.27:0/141746919 shutdown_connections 2024-01-11T09:45:18.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.566+0000 7f00183ff700 1 --2- 172.21.15.27:0/141746919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f001015df40 0x7f0010068720 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:18.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.566+0000 7f00183ff700 1 --2- 172.21.15.27:0/141746919 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f00100b0a40 0x7f001015da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:18.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.566+0000 7f00183ff700 1 -- 172.21.15.27:0/141746919 >> 172.21.15.27:0/141746919 conn(0x7f00100ce3c0 msgr2=0x7f00100ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:18.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.566+0000 7f00183ff700 1 -- 172.21.15.27:0/141746919 shutdown_connections 2024-01-11T09:45:18.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.566+0000 7f00183ff700 1 -- 172.21.15.27:0/141746919 wait complete. 2024-01-11T09:45:18.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.567+0000 7f00183ff700 1 Processor -- start 2024-01-11T09:45:18.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.567+0000 7f00183ff700 1 -- start start 2024-01-11T09:45:18.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.568+0000 7f00183ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001006bf10 0x7f00100c58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:18.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.568+0000 7f00183ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00100b0a40 0x7f00100c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:18.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.568+0000 7f00183ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f001015df40 0x7f00100c6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:18.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.568+0000 7f00183ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00101531f0 con 0x7f001006bf10 2024-01-11T09:45:18.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.568+0000 7f00183ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00100b2fe0 con 0x7f00100b0a40 2024-01-11T09:45:18.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.568+0000 7f00183ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00100b2460 con 0x7f001015df40 2024-01-11T09:45:18.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.569+0000 7f001619b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001006bf10 0x7f00100c58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:18.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.569+0000 7f001619b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001006bf10 0x7f00100c58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49818/0 (socket says 172.21.15.27:49818) 2024-01-11T09:45:18.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.569+0000 7f001599a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00100b0a40 0x7f00100c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:18.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.569+0000 7f001699c700 1 --2- 172.21.15.27:0/3914162110 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f001015df40 0x7f00100c6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:18.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.569+0000 7f001619b700 1 -- 172.21.15.27:0/3914162110 learned_addr learned my addr 172.21.15.27:0/3914162110 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:18.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.569+0000 7f001599a700 1 -- 172.21.15.27:0/3914162110 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f001015df40 msgr2=0x7f00100c6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:18.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.569+0000 7f001599a700 1 --2- 172.21.15.27:0/3914162110 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f001015df40 0x7f00100c6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:18.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.569+0000 7f001599a700 1 -- 172.21.15.27:0/3914162110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001006bf10 msgr2=0x7f00100c58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:18.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.569+0000 7f001599a700 1 --2- 172.21.15.27:0/3914162110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001006bf10 0x7f00100c58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:18.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.569+0000 7f001599a700 1 -- 172.21.15.27:0/3914162110 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0000009420 con 0x7f00100b0a40 2024-01-11T09:45:18.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.569+0000 7f001619b700 1 --2- 172.21.15.27:0/3914162110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001006bf10 0x7f00100c58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:18.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.569+0000 7f001599a700 1 --2- 172.21.15.27:0/3914162110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00100b0a40 0x7f00100c5e10 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f000000ec80 tx=0x7f00000091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:18.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.570+0000 7f00067fc700 1 -- 172.21.15.27:0/3914162110 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000000c850 con 0x7f00100b0a40 2024-01-11T09:45:18.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.570+0000 7f00067fc700 1 -- 172.21.15.27:0/3914162110 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000000c9b0 con 0x7f00100b0a40 2024-01-11T09:45:18.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.570+0000 7f00183ff700 1 -- 172.21.15.27:0/3914162110 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0008009a20 con 0x7f00100b0a40 2024-01-11T09:45:18.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.570+0000 7f00067fc700 1 -- 172.21.15.27:0/3914162110 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000000cd70 con 0x7f00100b0a40 2024-01-11T09:45:18.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.570+0000 7f00183ff700 1 -- 172.21.15.27:0/3914162110 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00100b2a20 con 0x7f00100b0a40 2024-01-11T09:45:18.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.571+0000 7f00183ff700 1 -- 172.21.15.27:0/3914162110 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f001004c3c0 con 0x7f00100b0a40 2024-01-11T09:45:18.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.572+0000 7f00067fc700 1 -- 172.21.15.27:0/3914162110 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0000013680 con 0x7f00100b0a40 2024-01-11T09:45:18.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.573+0000 7f00067fc700 1 --2- 172.21.15.27:0/3914162110 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efff4066f30 0x7efff40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:18.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.573+0000 7f00067fc700 1 -- 172.21.15.27:0/3914162110 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 5653+0+0 (secure 0 0 0) 0x7f00000847c0 con 0x7f00100b0a40 2024-01-11T09:45:18.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.573+0000 7f001619b700 1 --2- 172.21.15.27:0/3914162110 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efff4066f30 0x7efff40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:18.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.574+0000 7f001619b700 1 --2- 172.21.15.27:0/3914162110 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efff4066f30 0x7efff40693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f00100c18e0 tx=0x7f000c009040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:18.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.577+0000 7f00067fc700 1 -- 172.21.15.27:0/3914162110 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f00000519c0 con 0x7f00100b0a40 2024-01-11T09:45:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:18.798+0000 7f00183ff700 1 -- 172.21.15.27:0/3914162110 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f001006c680 con 0x7f00100b0a40 2024-01-11T09:45:19.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.105+0000 7f00067fc700 1 -- 172.21.15.27:0/3914162110 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v208) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0000054fe0 con 0x7f00100b0a40 2024-01-11T09:45:19.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.108+0000 7f00183ff700 1 -- 172.21.15.27:0/3914162110 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efff4066f30 msgr2=0x7efff40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:19.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.108+0000 7f00183ff700 1 --2- 172.21.15.27:0/3914162110 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efff4066f30 0x7efff40693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f00100c18e0 tx=0x7f000c009040 comp rx=0 tx=0).stop 2024-01-11T09:45:19.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.108+0000 7f00183ff700 1 -- 172.21.15.27:0/3914162110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00100b0a40 msgr2=0x7f00100c5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:19.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.108+0000 7f00183ff700 1 --2- 172.21.15.27:0/3914162110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00100b0a40 0x7f00100c5e10 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f000000ec80 tx=0x7f00000091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:19.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.108+0000 7f00183ff700 1 -- 172.21.15.27:0/3914162110 shutdown_connections 2024-01-11T09:45:19.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.108+0000 7f00183ff700 1 --2- 172.21.15.27:0/3914162110 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f001015df40 0x7f00100c6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:19.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.108+0000 7f00183ff700 1 --2- 172.21.15.27:0/3914162110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001006bf10 0x7f00100c58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:19.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.108+0000 7f00183ff700 1 --2- 172.21.15.27:0/3914162110 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efff4066f30 0x7efff40693e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:19.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.108+0000 7f00183ff700 1 --2- 172.21.15.27:0/3914162110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00100b0a40 0x7f00100c5e10 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:19.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.108+0000 7f00183ff700 1 -- 172.21.15.27:0/3914162110 >> 172.21.15.27:0/3914162110 conn(0x7f00100ce3c0 msgr2=0x7f001006d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:19.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.108+0000 7f00183ff700 1 -- 172.21.15.27:0/3914162110 shutdown_connections 2024-01-11T09:45:19.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.108+0000 7f00183ff700 1 -- 172.21.15.27:0/3914162110 wait complete. 2024-01-11T09:45:19.102 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-11T09:45:19.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-11T09:45:19.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.327+0000 7fe0325ff700 1 Processor -- start 2024-01-11T09:45:19.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.328+0000 7fe0325ff700 1 -- start start 2024-01-11T09:45:19.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.329+0000 7fe0325ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe02c1575e0 0x7fe02c158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:19.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.329+0000 7fe0325ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c158b60 con 0x7fe02c0b19d0 2024-01-11T09:45:19.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.329+0000 7fe0325ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c1599b0 con 0x7fe02c06b610 2024-01-11T09:45:19.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.329+0000 7fe0325ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe02c0cb5e0 con 0x7fe02c1575e0 2024-01-11T09:45:19.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.329+0000 7fe02bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe02c1575e0 0x7fe02c158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:19.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.329+0000 7fe030b9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe02c0b19d0 0x7fe02c159e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40760/0 (socket says 172.21.15.27:40760) 2024-01-11T09:45:19.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.329+0000 7fe02bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe02c1575e0 0x7fe02c158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49828/0 (socket says 172.21.15.27:49828) 2024-01-11T09:45:19.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.329+0000 7fe030b9c700 1 -- 172.21.15.27:0/3200253009 learned_addr learned my addr 172.21.15.27:0/3200253009 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:19.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.330+0000 7fe02bfff700 1 -- 172.21.15.27:0/3200253009 >> v1:172.21.15.77:6789/0 conn(0x7fe02c06b610 legacy=0x7fe02c0b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:19.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.330+0000 7fe02bfff700 1 -- 172.21.15.27:0/3200253009 >> v1:172.21.15.27:6789/0 conn(0x7fe02c0b19d0 legacy=0x7fe02c159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:19.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.330+0000 7fe02bfff700 1 -- 172.21.15.27:0/3200253009 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c0c0f90 con 0x7fe02c1575e0 2024-01-11T09:45:19.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.330+0000 7fe02bfff700 1 --2- 172.21.15.27:0/3200253009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe02c1575e0 0x7fe02c158620 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fe01c0098e0 tx=0x7fe01c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=603578cdac47bed5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:19.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.330+0000 7fe02affd700 1 -- 172.21.15.27:0/3200253009 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c009e80 con 0x7fe02c1575e0 2024-01-11T09:45:19.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.330+0000 7fe02affd700 1 -- 172.21.15.27:0/3200253009 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01c00b860 con 0x7fe02c1575e0 2024-01-11T09:45:19.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.330+0000 7fe02affd700 1 -- 172.21.15.27:0/3200253009 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c00bbe0 con 0x7fe02c1575e0 2024-01-11T09:45:19.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.331+0000 7fe0325ff700 1 -- 172.21.15.27:0/3200253009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe02c1575e0 msgr2=0x7fe02c158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:19.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.331+0000 7fe0325ff700 1 --2- 172.21.15.27:0/3200253009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe02c1575e0 0x7fe02c158620 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fe01c0098e0 tx=0x7fe01c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:19.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.331+0000 7fe0325ff700 1 -- 172.21.15.27:0/3200253009 shutdown_connections 2024-01-11T09:45:19.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.331+0000 7fe0325ff700 1 --2- 172.21.15.27:0/3200253009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe02c1575e0 0x7fe02c158620 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:19.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.331+0000 7fe0325ff700 1 -- 172.21.15.27:0/3200253009 >> 172.21.15.27:0/3200253009 conn(0x7fe02c0680e0 msgr2=0x7fe02c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:19.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.331+0000 7fe0325ff700 1 -- 172.21.15.27:0/3200253009 shutdown_connections 2024-01-11T09:45:19.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.331+0000 7fe0325ff700 1 -- 172.21.15.27:0/3200253009 wait complete. 2024-01-11T09:45:19.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.332+0000 7fe0325ff700 1 Processor -- start 2024-01-11T09:45:19.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.333+0000 7fe0325ff700 1 -- start start 2024-01-11T09:45:19.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.333+0000 7fe0325ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe02c06b610 0x7fe02c0c5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:19.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.333+0000 7fe0325ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe02c0b19d0 0x7fe02c143c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:19.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.333+0000 7fe0325ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe02c1575e0 0x7fe02c06e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.334+0000 7fe0325ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c0af9e0 con 0x7fe02c0b19d0 2024-01-11T09:45:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.334+0000 7fe0325ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c0c6800 con 0x7fe02c06b610 2024-01-11T09:45:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.334+0000 7fe0325ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c0af0b0 con 0x7fe02c1575e0 2024-01-11T09:45:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.334+0000 7fe02b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe02c0b19d0 0x7fe02c143c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.334+0000 7fe02bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe02c06b610 0x7fe02c0c5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.334+0000 7fe02bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe02c06b610 0x7fe02c0c5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:34902/0 (socket says 172.21.15.27:34902) 2024-01-11T09:45:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.334+0000 7fe02bfff700 1 -- 172.21.15.27:0/1512373099 learned_addr learned my addr 172.21.15.27:0/1512373099 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.334+0000 7fe02b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe02c0b19d0 0x7fe02c143c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49838/0 (socket says 172.21.15.27:49838) 2024-01-11T09:45:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.334+0000 7fe030b9c700 1 --2- 172.21.15.27:0/1512373099 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe02c1575e0 0x7fe02c06e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.334+0000 7fe02b7fe700 1 -- 172.21.15.27:0/1512373099 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe02c1575e0 msgr2=0x7fe02c06e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.334+0000 7fe02b7fe700 1 --2- 172.21.15.27:0/1512373099 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe02c1575e0 0x7fe02c06e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.334+0000 7fe02b7fe700 1 -- 172.21.15.27:0/1512373099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe02c06b610 msgr2=0x7fe02c0c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.334+0000 7fe02b7fe700 1 --2- 172.21.15.27:0/1512373099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe02c06b610 0x7fe02c0c5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:19.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.334+0000 7fe02b7fe700 1 -- 172.21.15.27:0/1512373099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c009420 con 0x7fe02c0b19d0 2024-01-11T09:45:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.335+0000 7fe02b7fe700 1 --2- 172.21.15.27:0/1512373099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe02c0b19d0 0x7fe02c143c20 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fe02000c590 tx=0x7fe020007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.335+0000 7fe013fff700 1 -- 172.21.15.27:0/1512373099 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe020011590 con 0x7fe02c0b19d0 2024-01-11T09:45:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.335+0000 7fe013fff700 1 -- 172.21.15.27:0/1512373099 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe020011710 con 0x7fe02c0b19d0 2024-01-11T09:45:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.335+0000 7fe013fff700 1 -- 172.21.15.27:0/1512373099 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe020011ad0 con 0x7fe02c0b19d0 2024-01-11T09:45:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.335+0000 7fe0325ff700 1 -- 172.21.15.27:0/1512373099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02c0af390 con 0x7fe02c0b19d0 2024-01-11T09:45:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.335+0000 7fe0325ff700 1 -- 172.21.15.27:0/1512373099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02c0c6510 con 0x7fe02c0b19d0 2024-01-11T09:45:19.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.336+0000 7fe0325ff700 1 -- 172.21.15.27:0/1512373099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe02c0718d0 con 0x7fe02c0b19d0 2024-01-11T09:45:19.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.337+0000 7fe013fff700 1 -- 172.21.15.27:0/1512373099 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe02001b3b0 con 0x7fe02c0b19d0 2024-01-11T09:45:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.338+0000 7fe013fff700 1 --2- 172.21.15.27:0/1512373099 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe018066f60 0x7fe018069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.338+0000 7fe013fff700 1 -- 172.21.15.27:0/1512373099 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 4965+0+0 (secure 0 0 0) 0x7fe0200836d0 con 0x7fe02c0b19d0 2024-01-11T09:45:19.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.339+0000 7fe02bfff700 1 --2- 172.21.15.27:0/1512373099 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe018066f60 0x7fe018069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:19.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.340+0000 7fe02bfff700 1 --2- 172.21.15.27:0/1512373099 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe018066f60 0x7fe018069410 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe01c0091e0 tx=0x7fe01c017000 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:19.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.342+0000 7fe013fff700 1 -- 172.21.15.27:0/1512373099 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe020050b80 con 0x7fe02c0b19d0 2024-01-11T09:45:19.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.621+0000 7fe0325ff700 1 -- 172.21.15.27:0/1512373099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe02c158da0 con 0x7fe02c0b19d0 2024-01-11T09:45:19.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.621+0000 7fe013fff700 1 -- 172.21.15.27:0/1512373099 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v49) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fe020017020 con 0x7fe02c0b19d0 2024-01-11T09:45:19.613 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:19.613 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:45:19.613 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-11T09:45:19.613 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:19.613 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-11T09:45:19.613 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lQm3yLhAAv6erz5hMfat8EUzRA5luMA==", 2024-01-11T09:45:19.613 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:45:19.614 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:45:19.614 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:45:19.614 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:45:19.614 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:45:19.614 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:19.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:19.614 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-11T09:45:19.614 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9l+FG8ChAACHndtVLx7qS3sidPniSycA==", 2024-01-11T09:45:19.614 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:45:19.614 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:45:19.614 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:45:19.614 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:45:19.614 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:45:19.615 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:19.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:19.615 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-11T09:45:19.615 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l1NnEOhAALHXsc5es7hufCthGdRzkyw==", 2024-01-11T09:45:19.615 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:45:19.615 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:45:19.615 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:45:19.615 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:45:19.615 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:45:19.615 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:19.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:19.615 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-11T09:45:19.615 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lruoWFhAAm8AmrZtTJhYI1+19F79RPQ==", 2024-01-11T09:45:19.616 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:45:19.616 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:45:19.616 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:45:19.616 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:45:19.616 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:45:19.616 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:19.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:19.616 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-11T09:45:19.616 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA2op9ltZygORAAZr1X1nCN0ce8rIVaGDpFyQ==", 2024-01-11T09:45:19.616 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:45:19.616 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:45:19.616 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:45:19.616 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:45:19.617 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:19.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:19.617 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-11T09:45:19.617 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA6op9lqSRPAhAAmtyCsyK5ZVGX2wfhOCMuFQ==", 2024-01-11T09:45:19.617 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:45:19.617 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:45:19.617 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:45:19.617 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:45:19.617 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:19.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:19.617 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-11T09:45:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA9op9lZcs8BhAA8j+U+jPOoy+fliVWZAPFYg==", 2024-01-11T09:45:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:45:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:45:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:45:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:45:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-11T09:45:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBAop9lFuCUChAAP6si/2W1mxH4ecQJvuumkw==", 2024-01-11T09:45:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:45:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:45:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:45:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:45:19.619 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:19.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:19.619 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-11T09:45:19.619 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBGop9lOpwBIRAAcpcsjwmX2IldcKHJWAgHuA==", 2024-01-11T09:45:19.619 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:45:19.619 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:45:19.619 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:45:19.619 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:45:19.619 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:19.620 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:19.620 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-11T09:45:19.620 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBJop9l4q8hJRAAK8mAYVRQMoWS6wBOU69HmA==", 2024-01-11T09:45:19.620 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:45:19.620 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:45:19.620 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:45:19.620 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:45:19.620 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:19.620 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:19.620 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-11T09:45:19.620 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBMop9lqMF9KBAAFgFFI2GWt9AGif4AoScThw==", 2024-01-11T09:45:19.620 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:45:19.620 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:45:19.621 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:45:19.621 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:45:19.621 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:19.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:19.621 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-11T09:45:19.621 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBPop9lN7eDLBAAKswPZFfs0g2AaSe6V+pbbQ==", 2024-01-11T09:45:19.621 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:45:19.621 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:45:19.621 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:45:19.621 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:45:19.621 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:19.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:19.622 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-11T09:45:19.622 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lT5okIBAABy8PbJJbA062VNmb1Y8qhg==", 2024-01-11T09:45:19.622 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:45:19.622 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:45:19.622 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-01-11T09:45:19.622 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-11T09:45:19.622 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:19.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:19.622 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-11T09:45:19.622 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lDf37LhAAqV+EabKXJEW1sS5xtu1tXw==", 2024-01-11T09:45:19.622 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:45:19.622 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:45:19.623 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T09:45:19.623 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T09:45:19.623 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T09:45:19.623 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:19.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:19.623 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-11T09:45:19.623 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lWayxJxAAcalM8QSOGtbCXxjYZGaNQA==", 2024-01-11T09:45:19.623 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:45:19.623 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:45:19.623 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T09:45:19.623 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T09:45:19.623 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T09:45:19.623 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:19.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:19.624 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-11T09:45:19.624 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lHD4tNhAAdmnFBmMrK+QhQYdpbzW0mg==", 2024-01-11T09:45:19.624 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:45:19.624 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:45:19.624 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T09:45:19.624 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T09:45:19.624 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T09:45:19.624 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:19.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:19.624 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-11T09:45:19.624 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAwop9lGPtDMxAAJ8FjrSEdCtQL8AhtAyqnTw==", 2024-01-11T09:45:19.625 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:45:19.625 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T09:45:19.625 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:45:19.625 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:45:19.625 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:45:19.625 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:19.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:19.625 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-11T09:45:19.625 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lXBZLJxAAXa1S8Vkm3MiV4igtNbg/1Q==", 2024-01-11T09:45:19.625 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:45:19.625 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T09:45:19.625 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T09:45:19.626 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:45:19.626 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:19.626 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:19.626 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-11T09:45:19.626 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l3ZGoHxAANqzytTTvWYSomG+RFsFDyg==", 2024-01-11T09:45:19.626 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:45:19.626 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T09:45:19.626 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T09:45:19.626 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:45:19.626 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.626 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:19.626 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-11T09:45:19.626 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:45:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.623+0000 7fe0325ff700 1 -- 172.21.15.27:0/1512373099 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe018066f60 msgr2=0x7fe018069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.623+0000 7fe0325ff700 1 --2- 172.21.15.27:0/1512373099 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe018066f60 0x7fe018069410 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe01c0091e0 tx=0x7fe01c017000 comp rx=0 tx=0).stop 2024-01-11T09:45:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.623+0000 7fe0325ff700 1 -- 172.21.15.27:0/1512373099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe02c0b19d0 msgr2=0x7fe02c143c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.623+0000 7fe0325ff700 1 --2- 172.21.15.27:0/1512373099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe02c0b19d0 0x7fe02c143c20 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fe02000c590 tx=0x7fe020007ec0 comp rx=0 tx=0).stop 2024-01-11T09:45:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.623+0000 7fe0325ff700 1 -- 172.21.15.27:0/1512373099 shutdown_connections 2024-01-11T09:45:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.623+0000 7fe0325ff700 1 --2- 172.21.15.27:0/1512373099 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe02c1575e0 0x7fe02c06e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.623+0000 7fe0325ff700 1 --2- 172.21.15.27:0/1512373099 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe018066f60 0x7fe018069410 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.623+0000 7fe0325ff700 1 --2- 172.21.15.27:0/1512373099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe02c06b610 0x7fe02c0c5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.623+0000 7fe0325ff700 1 --2- 172.21.15.27:0/1512373099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe02c0b19d0 0x7fe02c143c20 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.623+0000 7fe0325ff700 1 -- 172.21.15.27:0/1512373099 >> 172.21.15.27:0/1512373099 conn(0x7fe02c0680e0 msgr2=0x7fe02c0695b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.624+0000 7fe0325ff700 1 -- 172.21.15.27:0/1512373099 shutdown_connections 2024-01-11T09:45:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.624+0000 7fe0325ff700 1 -- 172.21.15.27:0/1512373099 wait complete. 2024-01-11T09:45:19.637 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:45:19.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.843+0000 7f5a013ff700 1 Processor -- start 2024-01-11T09:45:19.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.844+0000 7f5a013ff700 1 -- start start 2024-01-11T09:45:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.845+0000 7f5a013ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f59fc0b19d0 0x7f59fc158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.845+0000 7f5a013ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f59fc06b610 0x7f59fc159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.845+0000 7f5a013ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59fc158b60 con 0x7f59fc1575e0 2024-01-11T09:45:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.845+0000 7f5a013ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59fc0cb5e0 con 0x7f59fc06b610 2024-01-11T09:45:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.845+0000 7f5a013ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59fc1599b0 con 0x7f59fc0b19d0 2024-01-11T09:45:19.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.845+0000 7f59fb7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f59fc06b610 0x7f59fc159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.845+0000 7f59faffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f59fc1575e0 0x7f59fc0b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40772/0 (socket says 172.21.15.27:40772) 2024-01-11T09:45:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.845+0000 7f59faffd700 1 -- 172.21.15.27:0/3250966486 learned_addr learned my addr 172.21.15.27:0/3250966486 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.846+0000 7f59fa7fc700 1 --2- 172.21.15.27:0/3250966486 >> v2:172.21.15.77:3300/0 conn(0x7f59fc0b19d0 0x7f59fc158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.846+0000 7f59fb7fe700 1 -- 172.21.15.27:0/3250966486 >> v2:172.21.15.77:3300/0 conn(0x7f59fc0b19d0 msgr2=0x7f59fc158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.846+0000 7f59fb7fe700 1 --2- 172.21.15.27:0/3250966486 >> v2:172.21.15.77:3300/0 conn(0x7f59fc0b19d0 0x7f59fc158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.846+0000 7f59fb7fe700 1 -- 172.21.15.27:0/3250966486 >> v1:172.21.15.27:6789/0 conn(0x7f59fc1575e0 legacy=0x7f59fc0b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.846+0000 7f59fb7fe700 1 -- 172.21.15.27:0/3250966486 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc0c0f90 con 0x7f59fc06b610 2024-01-11T09:45:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.846+0000 7f59f9ffb700 1 -- 172.21.15.27:0/3250966486 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177698927 0 0) 0x7f59ec001110 con 0x7f59fc1575e0 2024-01-11T09:45:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.846+0000 7f59fb7fe700 1 --2- 172.21.15.27:0/3250966486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59fc06b610 0x7f59fc159e40 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f59fc064fe0 tx=0x7f59f4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4527dd06e3a184e9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.846+0000 7f59f9ffb700 1 -- 172.21.15.27:0/3250966486 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f400e070 con 0x7f59fc06b610 2024-01-11T09:45:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.846+0000 7f59f9ffb700 1 -- 172.21.15.27:0/3250966486 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f400bc30 con 0x7f59fc06b610 2024-01-11T09:45:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.846+0000 7f59f9ffb700 1 -- 172.21.15.27:0/3250966486 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4012430 con 0x7f59fc06b610 2024-01-11T09:45:19.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.847+0000 7f5a013ff700 1 -- 172.21.15.27:0/3250966486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59fc06b610 msgr2=0x7f59fc159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.847+0000 7f5a013ff700 1 --2- 172.21.15.27:0/3250966486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59fc06b610 0x7f59fc159e40 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f59fc064fe0 tx=0x7f59f4009780 comp rx=0 tx=0).stop 2024-01-11T09:45:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.847+0000 7f5a013ff700 1 -- 172.21.15.27:0/3250966486 shutdown_connections 2024-01-11T09:45:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.847+0000 7f5a013ff700 1 --2- 172.21.15.27:0/3250966486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59fc06b610 0x7f59fc159e40 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.847+0000 7f5a013ff700 1 --2- 172.21.15.27:0/3250966486 >> v2:172.21.15.77:3300/0 conn(0x7f59fc0b19d0 0x7f59fc158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.847+0000 7f5a013ff700 1 -- 172.21.15.27:0/3250966486 >> 172.21.15.27:0/3250966486 conn(0x7f59fc0680e0 msgr2=0x7f59fc069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.847+0000 7f5a013ff700 1 -- 172.21.15.27:0/3250966486 shutdown_connections 2024-01-11T09:45:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.847+0000 7f5a013ff700 1 -- 172.21.15.27:0/3250966486 wait complete. 2024-01-11T09:45:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.848+0000 7f5a013ff700 1 Processor -- start 2024-01-11T09:45:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.849+0000 7f5a013ff700 1 -- start start 2024-01-11T09:45:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.849+0000 7f5a013ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59fc06b610 0x7f59fc15bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.850+0000 7f5a013ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59fc0b19d0 0x7f59fc15d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.850+0000 7f5a013ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59fc1575e0 0x7f59fc15dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:19.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.850+0000 7f5a013ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc15ef60 con 0x7f59fc1575e0 2024-01-11T09:45:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.850+0000 7f5a013ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc15e350 con 0x7f59fc0b19d0 2024-01-11T09:45:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.850+0000 7f5a013ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc15e630 con 0x7f59fc06b610 2024-01-11T09:45:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.850+0000 7f59fb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59fc1575e0 0x7f59fc15dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.850+0000 7f59fb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59fc1575e0 0x7f59fc15dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49862/0 (socket says 172.21.15.27:49862) 2024-01-11T09:45:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.850+0000 7f59fb7fe700 1 -- 172.21.15.27:0/2150494332 learned_addr learned my addr 172.21.15.27:0/2150494332 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.850+0000 7f59faffd700 1 --2- 172.21.15.27:0/2150494332 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59fc06b610 0x7f59fc15bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.850+0000 7f59fa7fc700 1 --2- 172.21.15.27:0/2150494332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59fc0b19d0 0x7f59fc15d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.850+0000 7f59fb7fe700 1 -- 172.21.15.27:0/2150494332 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59fc06b610 msgr2=0x7f59fc15bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:19.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.850+0000 7f59fb7fe700 1 --2- 172.21.15.27:0/2150494332 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59fc06b610 0x7f59fc15bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.850+0000 7f59fb7fe700 1 -- 172.21.15.27:0/2150494332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59fc0b19d0 msgr2=0x7f59fc15d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.850+0000 7f59fb7fe700 1 --2- 172.21.15.27:0/2150494332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59fc0b19d0 0x7f59fc15d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.850+0000 7f59fb7fe700 1 -- 172.21.15.27:0/2150494332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f4009a20 con 0x7f59fc1575e0 2024-01-11T09:45:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.851+0000 7f59fb7fe700 1 --2- 172.21.15.27:0/2150494332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59fc1575e0 0x7f59fc15dd10 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f59fc066180 tx=0x7f59f4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.851+0000 7f59e2ffd700 1 -- 172.21.15.27:0/2150494332 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4012ea0 con 0x7f59fc1575e0 2024-01-11T09:45:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.851+0000 7f59e2ffd700 1 -- 172.21.15.27:0/2150494332 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f401d630 con 0x7f59fc1575e0 2024-01-11T09:45:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.851+0000 7f59e2ffd700 1 -- 172.21.15.27:0/2150494332 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f401d9f0 con 0x7f59fc1575e0 2024-01-11T09:45:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.851+0000 7f5a013ff700 1 -- 172.21.15.27:0/2150494332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59fc15e910 con 0x7f59fc1575e0 2024-01-11T09:45:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.851+0000 7f5a013ff700 1 -- 172.21.15.27:0/2150494332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59fc152020 con 0x7f59fc1575e0 2024-01-11T09:45:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.852+0000 7f5a013ff700 1 -- 172.21.15.27:0/2150494332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59fc0b1da0 con 0x7f59fc1575e0 2024-01-11T09:45:19.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.853+0000 7f59e2ffd700 1 -- 172.21.15.27:0/2150494332 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f59f4024070 con 0x7f59fc1575e0 2024-01-11T09:45:19.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.854+0000 7f59e2ffd700 1 --2- 172.21.15.27:0/2150494332 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f59e8066f30 0x7f59e80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:19.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.854+0000 7f59e2ffd700 1 -- 172.21.15.27:0/2150494332 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f59f4086b20 con 0x7f59fc1575e0 2024-01-11T09:45:19.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.854+0000 7f59faffd700 1 --2- 172.21.15.27:0/2150494332 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f59e8066f30 0x7f59e80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:19.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.855+0000 7f59faffd700 1 --2- 172.21.15.27:0/2150494332 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f59e8066f30 0x7f59e80693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f59ec00c8f0 tx=0x7f59ec009040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:19.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:19.858+0000 7f59e2ffd700 1 -- 172.21.15.27:0/2150494332 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f59f4053fd0 con 0x7f59fc1575e0 2024-01-11T09:45:20.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.064+0000 7f5a013ff700 1 -- 172.21.15.27:0/2150494332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59fc15f0a0 con 0x7f59fc1575e0 2024-01-11T09:45:20.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.065+0000 7f59e2ffd700 1 -- 172.21.15.27:0/2150494332 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v210) v1 ==== 74+0+12275 (secure 0 0 0) 0x7f59fc15f0a0 con 0x7f59fc1575e0 2024-01-11T09:45:20.056 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:20.056 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":210,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:45:19.110282+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:45:20.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.067+0000 7f5a013ff700 1 -- 172.21.15.27:0/2150494332 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f59e8066f30 msgr2=0x7f59e80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:20.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.067+0000 7f5a013ff700 1 --2- 172.21.15.27:0/2150494332 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f59e8066f30 0x7f59e80693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f59ec00c8f0 tx=0x7f59ec009040 comp rx=0 tx=0).stop 2024-01-11T09:45:20.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.067+0000 7f5a013ff700 1 -- 172.21.15.27:0/2150494332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59fc1575e0 msgr2=0x7f59fc15dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:20.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.067+0000 7f5a013ff700 1 --2- 172.21.15.27:0/2150494332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59fc1575e0 0x7f59fc15dd10 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f59fc066180 tx=0x7f59f4019000 comp rx=0 tx=0).stop 2024-01-11T09:45:20.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.067+0000 7f5a013ff700 1 -- 172.21.15.27:0/2150494332 shutdown_connections 2024-01-11T09:45:20.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.067+0000 7f5a013ff700 1 --2- 172.21.15.27:0/2150494332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59fc1575e0 0x7f59fc15dd10 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.067+0000 7f5a013ff700 1 --2- 172.21.15.27:0/2150494332 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f59fc06b610 0x7f59fc15bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.067+0000 7f5a013ff700 1 --2- 172.21.15.27:0/2150494332 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f59e8066f30 0x7f59e80693e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.067+0000 7f5a013ff700 1 --2- 172.21.15.27:0/2150494332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59fc0b19d0 0x7f59fc15d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.067+0000 7f5a013ff700 1 -- 172.21.15.27:0/2150494332 >> 172.21.15.27:0/2150494332 conn(0x7f59fc0680e0 msgr2=0x7f59fc06e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:20.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.067+0000 7f5a013ff700 1 -- 172.21.15.27:0/2150494332 shutdown_connections 2024-01-11T09:45:20.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.067+0000 7f5a013ff700 1 -- 172.21.15.27:0/2150494332 wait complete. 2024-01-11T09:45:20.077 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-11T09:45:20.077 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T09:45:20.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.290+0000 7f9031dff700 1 Processor -- start 2024-01-11T09:45:20.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.290+0000 7f9031dff700 1 -- start start 2024-01-11T09:45:20.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.292+0000 7f9031dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f902c13ff80 0x7f902c15a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:20.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.292+0000 7f9031dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f902c15a560 0x7f902c0c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:20.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.292+0000 7f9031dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f902c0c6b30 con 0x7f902c13fbb0 2024-01-11T09:45:20.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.292+0000 7f9031dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f902c15a9a0 con 0x7f902c13ff80 2024-01-11T09:45:20.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.292+0000 7f9031dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f902c0c6800 con 0x7f902c15a560 2024-01-11T09:45:20.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.292+0000 7f902b7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f902c13fbb0 0x7f902c0ca7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40782/0 (socket says 172.21.15.27:40782) 2024-01-11T09:45:20.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.292+0000 7f902b7fe700 1 -- 172.21.15.27:0/3585053408 learned_addr learned my addr 172.21.15.27:0/3585053408 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:20.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.292+0000 7f902a7fc700 1 -- 172.21.15.27:0/3585053408 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303809932 0 0) 0x7f902c0c6b30 con 0x7f902c13fbb0 2024-01-11T09:45:20.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.292+0000 7f902bfff700 1 --2- 172.21.15.27:0/3585053408 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f902c15a560 0x7f902c0c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:20.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.292+0000 7f902a7fc700 1 -- 172.21.15.27:0/3585053408 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9014003570 con 0x7f902c13fbb0 2024-01-11T09:45:20.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.292+0000 7f902affd700 1 --2- 172.21.15.27:0/3585053408 >> v2:172.21.15.77:3300/0 conn(0x7f902c13ff80 0x7f902c15a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:20.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902a7fc700 1 -- 172.21.15.27:0/3585053408 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2333745621 0 0) 0x7f9014003570 con 0x7f902c13fbb0 2024-01-11T09:45:20.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902a7fc700 1 -- 172.21.15.27:0/3585053408 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f902c15a560 msgr2=0x7f902c0c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:20.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902a7fc700 1 --2- 172.21.15.27:0/3585053408 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f902c15a560 0x7f902c0c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902a7fc700 1 -- 172.21.15.27:0/3585053408 >> v2:172.21.15.77:3300/0 conn(0x7f902c13ff80 msgr2=0x7f902c15a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:20.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902a7fc700 1 --2- 172.21.15.27:0/3585053408 >> v2:172.21.15.77:3300/0 conn(0x7f902c13ff80 0x7f902c15a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902a7fc700 1 -- 172.21.15.27:0/3585053408 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c0cb240 con 0x7f902c13fbb0 2024-01-11T09:45:20.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902a7fc700 1 -- 172.21.15.27:0/3585053408 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f901c0015f0 con 0x7f902c13fbb0 2024-01-11T09:45:20.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902a7fc700 1 -- 172.21.15.27:0/3585053408 >> v1:172.21.15.27:6789/0 conn(0x7f902c13fbb0 legacy=0x7f902c0ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:20.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902a7fc700 1 --2- 172.21.15.27:0/3585053408 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f902c15a560 0x7f9014003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:20.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902a7fc700 1 --2- 172.21.15.27:0/3585053408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90140040b0 0x7f9014004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:20.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902a7fc700 1 --2- 172.21.15.27:0/3585053408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9014004d50 0x7f90140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:20.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902a7fc700 1 -- 172.21.15.27:0/3585053408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c15a9a0 con 0x7f90140040b0 2024-01-11T09:45:20.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902a7fc700 1 -- 172.21.15.27:0/3585053408 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c0c6800 con 0x7f9014004d50 2024-01-11T09:45:20.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902a7fc700 1 -- 172.21.15.27:0/3585053408 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901400bb60 con 0x7f902c15a560 2024-01-11T09:45:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902affd700 1 --2- 172.21.15.27:0/3585053408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90140040b0 0x7f9014004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902b7fe700 1 --2- 172.21.15.27:0/3585053408 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f902c15a560 0x7f9014003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902bfff700 1 --2- 172.21.15.27:0/3585053408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9014004d50 0x7f90140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902affd700 1 -- 172.21.15.27:0/3585053408 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f902c15a560 msgr2=0x7f9014003ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902affd700 1 --2- 172.21.15.27:0/3585053408 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f902c15a560 0x7f9014003ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902affd700 1 -- 172.21.15.27:0/3585053408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9014004d50 msgr2=0x7f90140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902affd700 1 --2- 172.21.15.27:0/3585053408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9014004d50 0x7f90140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.293+0000 7f902affd700 1 -- 172.21.15.27:0/3585053408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9014003570 con 0x7f90140040b0 2024-01-11T09:45:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.294+0000 7f902affd700 1 --2- 172.21.15.27:0/3585053408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90140040b0 0x7f9014004810 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f9020006f00 tx=0x7f90200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7510071b0a7e44e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.294+0000 7f902a7fc700 1 -- 172.21.15.27:0/3585053408 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902000e710 con 0x7f90140040b0 2024-01-11T09:45:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.294+0000 7f902a7fc700 1 -- 172.21.15.27:0/3585053408 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902000e870 con 0x7f90140040b0 2024-01-11T09:45:20.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.294+0000 7f902a7fc700 1 -- 172.21.15.27:0/3585053408 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902000ec30 con 0x7f90140040b0 2024-01-11T09:45:20.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.294+0000 7f9031dff700 1 -- 172.21.15.27:0/3585053408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90140040b0 msgr2=0x7f9014004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:20.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.294+0000 7f9031dff700 1 --2- 172.21.15.27:0/3585053408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90140040b0 0x7f9014004810 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f9020006f00 tx=0x7f90200091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.294+0000 7f902bfff700 1 -- 172.21.15.27:0/3585053408 reap_dead start 2024-01-11T09:45:20.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.295+0000 7f9031dff700 1 -- 172.21.15.27:0/3585053408 shutdown_connections 2024-01-11T09:45:20.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.295+0000 7f9031dff700 1 -- 172.21.15.27:0/3585053408 >> 172.21.15.27:0/3585053408 conn(0x7f902c0680a0 msgr2=0x7f902c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:20.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.295+0000 7f9031dff700 1 -- 172.21.15.27:0/3585053408 shutdown_connections 2024-01-11T09:45:20.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.295+0000 7f9031dff700 1 -- 172.21.15.27:0/3585053408 wait complete. 2024-01-11T09:45:20.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.297+0000 7f9031dff700 1 Processor -- start 2024-01-11T09:45:20.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.297+0000 7f9031dff700 1 -- start start 2024-01-11T09:45:20.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.298+0000 7f9031dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f90140040b0 0x7f902c155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:20.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.298+0000 7f9031dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c0b0aa0 0x7f902c0c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:20.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.298+0000 7f9031dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f902c0ca2d0 0x7f902c133ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:20.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.298+0000 7f9031dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c0ca750 con 0x7f902c0b0aa0 2024-01-11T09:45:20.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.298+0000 7f9031dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c134f20 con 0x7f902c0ca2d0 2024-01-11T09:45:20.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.298+0000 7f9031dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f902c133850 con 0x7f90140040b0 2024-01-11T09:45:20.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.298+0000 7f902affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c0b0aa0 0x7f902c0c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:20.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.299+0000 7f902b7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f90140040b0 0x7f902c155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:20.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.299+0000 7f902b7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f90140040b0 0x7f902c155d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:59866/0 (socket says 172.21.15.27:59866) 2024-01-11T09:45:20.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.299+0000 7f902b7fe700 1 -- 172.21.15.27:0/796669288 learned_addr learned my addr 172.21.15.27:0/796669288 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:20.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.299+0000 7f902bfff700 1 --2- 172.21.15.27:0/796669288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f902c0ca2d0 0x7f902c133ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:20.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.299+0000 7f902affd700 1 -- 172.21.15.27:0/796669288 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f90140040b0 msgr2=0x7f902c155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:20.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.299+0000 7f902affd700 1 --2- 172.21.15.27:0/796669288 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f90140040b0 0x7f902c155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.299+0000 7f902affd700 1 -- 172.21.15.27:0/796669288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f902c0ca2d0 msgr2=0x7f902c133ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:20.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.299+0000 7f902affd700 1 --2- 172.21.15.27:0/796669288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f902c0ca2d0 0x7f902c133ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.299+0000 7f902affd700 1 -- 172.21.15.27:0/796669288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c134960 con 0x7f902c0b0aa0 2024-01-11T09:45:20.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.299+0000 7f902affd700 1 --2- 172.21.15.27:0/796669288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c0b0aa0 0x7f902c0c9d90 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f9020019000 tx=0x7f902001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:20.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.300+0000 7f90137fe700 1 -- 172.21.15.27:0/796669288 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9020022070 con 0x7f902c0b0aa0 2024-01-11T09:45:20.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.300+0000 7f90137fe700 1 -- 172.21.15.27:0/796669288 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902000e9e0 con 0x7f902c0b0aa0 2024-01-11T09:45:20.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.300+0000 7f90137fe700 1 -- 172.21.15.27:0/796669288 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902000e710 con 0x7f902c0b0aa0 2024-01-11T09:45:20.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.300+0000 7f9031dff700 1 -- 172.21.15.27:0/796669288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f902c134c40 con 0x7f902c0b0aa0 2024-01-11T09:45:20.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.300+0000 7f9031dff700 1 -- 172.21.15.27:0/796669288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f902c156250 con 0x7f902c0b0aa0 2024-01-11T09:45:20.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.301+0000 7f9031dff700 1 -- 172.21.15.27:0/796669288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f902c0ca6d0 con 0x7f902c0b0aa0 2024-01-11T09:45:20.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.301+0000 7f90137fe700 1 -- 172.21.15.27:0/796669288 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f902001c660 con 0x7f902c0b0aa0 2024-01-11T09:45:20.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.302+0000 7f90137fe700 1 --2- 172.21.15.27:0/796669288 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9018066f30 0x7f90180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:20.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.302+0000 7f90137fe700 1 -- 172.21.15.27:0/796669288 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f90200858c0 con 0x7f902c0b0aa0 2024-01-11T09:45:20.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.303+0000 7f902b7fe700 1 --2- 172.21.15.27:0/796669288 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9018066f30 0x7f90180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:20.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.303+0000 7f902b7fe700 1 --2- 172.21.15.27:0/796669288 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9018066f30 0x7f90180693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f901c000e50 tx=0x7f901c009040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:20.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.307+0000 7f90137fe700 1 -- 172.21.15.27:0/796669288 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9020052e20 con 0x7f902c0b0aa0 2024-01-11T09:45:20.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.509+0000 7f9031dff700 1 -- 172.21.15.27:0/796669288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f902c04c3c0 con 0x7f902c0b0aa0 2024-01-11T09:45:20.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.510+0000 7f90137fe700 1 -- 172.21.15.27:0/796669288 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v210) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9020017020 con 0x7f902c0b0aa0 2024-01-11T09:45:20.502 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T09:45:20.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.512+0000 7f9031dff700 1 -- 172.21.15.27:0/796669288 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9018066f30 msgr2=0x7f90180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:20.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.512+0000 7f9031dff700 1 --2- 172.21.15.27:0/796669288 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9018066f30 0x7f90180693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f901c000e50 tx=0x7f901c009040 comp rx=0 tx=0).stop 2024-01-11T09:45:20.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.512+0000 7f9031dff700 1 -- 172.21.15.27:0/796669288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c0b0aa0 msgr2=0x7f902c0c9d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:20.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.512+0000 7f9031dff700 1 --2- 172.21.15.27:0/796669288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c0b0aa0 0x7f902c0c9d90 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f9020019000 tx=0x7f902001a000 comp rx=0 tx=0).stop 2024-01-11T09:45:20.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.512+0000 7f9031dff700 1 -- 172.21.15.27:0/796669288 shutdown_connections 2024-01-11T09:45:20.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.512+0000 7f9031dff700 1 --2- 172.21.15.27:0/796669288 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9018066f30 0x7f90180693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.512+0000 7f9031dff700 1 --2- 172.21.15.27:0/796669288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f902c0ca2d0 0x7f902c133ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.512+0000 7f9031dff700 1 --2- 172.21.15.27:0/796669288 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f90140040b0 0x7f902c155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.512+0000 7f9031dff700 1 --2- 172.21.15.27:0/796669288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c0b0aa0 0x7f902c0c9d90 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.512+0000 7f9031dff700 1 -- 172.21.15.27:0/796669288 >> 172.21.15.27:0/796669288 conn(0x7f902c0680a0 msgr2=0x7f902c0c1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:20.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.512+0000 7f9031dff700 1 -- 172.21.15.27:0/796669288 shutdown_connections 2024-01-11T09:45:20.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.513+0000 7f9031dff700 1 -- 172.21.15.27:0/796669288 wait complete. 2024-01-11T09:45:20.522 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-11T09:45:20.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.729+0000 7f130e1ff700 1 Processor -- start 2024-01-11T09:45:20.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.730+0000 7f130e1ff700 1 -- start start 2024-01-11T09:45:20.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.731+0000 7f130e1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13080b2b90 0x7f1308159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:20.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.731+0000 7f130e1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13080b07b0 0x7f1308159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:20.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.731+0000 7f130e1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13080cd730 con 0x7f13080b03e0 2024-01-11T09:45:20.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.731+0000 7f130e1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13080ccb20 con 0x7f13080b07b0 2024-01-11T09:45:20.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.731+0000 7f130e1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13080cce00 con 0x7f13080b2b90 2024-01-11T09:45:20.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.731+0000 7f13077fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f13080b03e0 0x7f13080b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40784/0 (socket says 172.21.15.27:40784) 2024-01-11T09:45:20.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.731+0000 7f1307fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13080b07b0 0x7f1308159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:20.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.731+0000 7f13077fe700 1 -- 172.21.15.27:0/3944929786 learned_addr learned my addr 172.21.15.27:0/3944929786 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.731+0000 7f1306ffd700 1 --2- 172.21.15.27:0/3944929786 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13080b2b90 0x7f1308159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.731+0000 7f1307fff700 1 -- 172.21.15.27:0/3944929786 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13080b2b90 msgr2=0x7f1308159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.731+0000 7f13067fc700 1 -- 172.21.15.27:0/3944929786 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379784767 0 0) 0x7f13080cd730 con 0x7f13080b03e0 2024-01-11T09:45:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.731+0000 7f1307fff700 1 --2- 172.21.15.27:0/3944929786 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13080b2b90 0x7f1308159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.731+0000 7f1307fff700 1 -- 172.21.15.27:0/3944929786 >> v1:172.21.15.27:6789/0 conn(0x7f13080b03e0 legacy=0x7f13080b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.732+0000 7f1307fff700 1 -- 172.21.15.27:0/3944929786 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13080cd0e0 con 0x7f13080b07b0 2024-01-11T09:45:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.732+0000 7f1306ffd700 1 --2- 172.21.15.27:0/3944929786 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13080b2b90 0x7f1308159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.732+0000 7f1307fff700 1 --2- 172.21.15.27:0/3944929786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13080b07b0 0x7f1308159740 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f1300009e60 tx=0x7f1300009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c3df859a000991c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.732+0000 7f13067fc700 1 -- 172.21.15.27:0/3944929786 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130000e070 con 0x7f13080b07b0 2024-01-11T09:45:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.732+0000 7f13067fc700 1 -- 172.21.15.27:0/3944929786 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130000bc30 con 0x7f13080b07b0 2024-01-11T09:45:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.732+0000 7f13067fc700 1 -- 172.21.15.27:0/3944929786 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1300012430 con 0x7f13080b07b0 2024-01-11T09:45:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.733+0000 7f130e1ff700 1 -- 172.21.15.27:0/3944929786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13080b07b0 msgr2=0x7f1308159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.733+0000 7f130e1ff700 1 --2- 172.21.15.27:0/3944929786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13080b07b0 0x7f1308159740 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f1300009e60 tx=0x7f1300009780 comp rx=0 tx=0).stop 2024-01-11T09:45:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.733+0000 7f130e1ff700 1 -- 172.21.15.27:0/3944929786 shutdown_connections 2024-01-11T09:45:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.733+0000 7f130e1ff700 1 --2- 172.21.15.27:0/3944929786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13080b07b0 0x7f1308159740 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.733+0000 7f130e1ff700 1 --2- 172.21.15.27:0/3944929786 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13080b2b90 0x7f1308159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.733+0000 7f130e1ff700 1 -- 172.21.15.27:0/3944929786 >> 172.21.15.27:0/3944929786 conn(0x7f130806ed40 msgr2=0x7f130806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.733+0000 7f130e1ff700 1 -- 172.21.15.27:0/3944929786 shutdown_connections 2024-01-11T09:45:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.733+0000 7f130e1ff700 1 -- 172.21.15.27:0/3944929786 wait complete. 2024-01-11T09:45:20.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.735+0000 7f130e1ff700 1 Processor -- start 2024-01-11T09:45:20.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.735+0000 7f130e1ff700 1 -- start start 2024-01-11T09:45:20.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.736+0000 7f130e1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13080b03e0 0x7f1308159470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:20.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.736+0000 7f130e1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13080b07b0 0x7f13080c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.736+0000 7f130e1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13080b2b90 0x7f13080c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.736+0000 7f130e1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13080c4210 con 0x7f13080b03e0 2024-01-11T09:45:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.736+0000 7f130e1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13080c88d0 con 0x7f13080b2b90 2024-01-11T09:45:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.736+0000 7f130e1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13080c8bb0 con 0x7f13080b07b0 2024-01-11T09:45:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.737+0000 7f13077fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13080b03e0 0x7f1308159470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.737+0000 7f13077fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13080b03e0 0x7f1308159470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49914/0 (socket says 172.21.15.27:49914) 2024-01-11T09:45:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.737+0000 7f13077fe700 1 -- 172.21.15.27:0/3550292236 learned_addr learned my addr 172.21.15.27:0/3550292236 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.737+0000 7f1307fff700 1 --2- 172.21.15.27:0/3550292236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13080b2b90 0x7f13080c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.737+0000 7f1306ffd700 1 --2- 172.21.15.27:0/3550292236 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13080b07b0 0x7f13080c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.737+0000 7f13077fe700 1 -- 172.21.15.27:0/3550292236 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13080b07b0 msgr2=0x7f13080c7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.737+0000 7f13077fe700 1 --2- 172.21.15.27:0/3550292236 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13080b07b0 0x7f13080c7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.737+0000 7f13077fe700 1 -- 172.21.15.27:0/3550292236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13080b2b90 msgr2=0x7f13080c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.737+0000 7f13077fe700 1 --2- 172.21.15.27:0/3550292236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13080b2b90 0x7f13080c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.737+0000 7f13077fe700 1 -- 172.21.15.27:0/3550292236 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1300009a20 con 0x7f13080b03e0 2024-01-11T09:45:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.737+0000 7f1307fff700 1 --2- 172.21.15.27:0/3550292236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13080b2b90 0x7f13080c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.737+0000 7f13077fe700 1 --2- 172.21.15.27:0/3550292236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13080b03e0 0x7f1308159470 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f12f800c7e0 tx=0x7f12f8007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.738+0000 7f12ef7fe700 1 -- 172.21.15.27:0/3550292236 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f8005b30 con 0x7f13080b03e0 2024-01-11T09:45:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.738+0000 7f12ef7fe700 1 -- 172.21.15.27:0/3550292236 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f8005c90 con 0x7f13080b03e0 2024-01-11T09:45:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.738+0000 7f12ef7fe700 1 -- 172.21.15.27:0/3550292236 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f8007300 con 0x7f13080b03e0 2024-01-11T09:45:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.738+0000 7f130e1ff700 1 -- 172.21.15.27:0/3550292236 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13080c8e90 con 0x7f13080b03e0 2024-01-11T09:45:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.738+0000 7f130e1ff700 1 -- 172.21.15.27:0/3550292236 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13081599b0 con 0x7f13080b03e0 2024-01-11T09:45:20.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.739+0000 7f130e1ff700 1 -- 172.21.15.27:0/3550292236 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13080718d0 con 0x7f13080b03e0 2024-01-11T09:45:20.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.740+0000 7f12ef7fe700 1 -- 172.21.15.27:0/3550292236 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f12f801b400 con 0x7f13080b03e0 2024-01-11T09:45:20.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.741+0000 7f12ef7fe700 1 --2- 172.21.15.27:0/3550292236 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f12f4066f30 0x7f12f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:20.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.742+0000 7f12ef7fe700 1 -- 172.21.15.27:0/3550292236 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f12f8083df0 con 0x7f13080b03e0 2024-01-11T09:45:20.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.742+0000 7f1306ffd700 1 --2- 172.21.15.27:0/3550292236 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f12f4066f30 0x7f12f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:20.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.742+0000 7f1306ffd700 1 --2- 172.21.15.27:0/3550292236 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f12f4066f30 0x7f12f40693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f13080ca2f0 tx=0x7f12fc008040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:20.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.746+0000 7f12ef7fe700 1 -- 172.21.15.27:0/3550292236 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f12f8051320 con 0x7f13080b03e0 2024-01-11T09:45:20.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.954+0000 7f130e1ff700 1 -- 172.21.15.27:0/3550292236 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f13080cd3c0 con 0x7f13080b03e0 2024-01-11T09:45:20.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.955+0000 7f12ef7fe700 1 -- 172.21.15.27:0/3550292236 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v210) v1 ==== 84+0+65 (secure 0 0 0) 0x7f12f8054940 con 0x7f13080b03e0 2024-01-11T09:45:20.947 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:20.948 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T09:45:20.948 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:20.948 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-11T09:45:20.948 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-11T09:45:20.948 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:20.948 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T09:45:20.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.957+0000 7f130e1ff700 1 -- 172.21.15.27:0/3550292236 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f12f4066f30 msgr2=0x7f12f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:20.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.957+0000 7f130e1ff700 1 --2- 172.21.15.27:0/3550292236 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f12f4066f30 0x7f12f40693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f13080ca2f0 tx=0x7f12fc008040 comp rx=0 tx=0).stop 2024-01-11T09:45:20.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.957+0000 7f130e1ff700 1 -- 172.21.15.27:0/3550292236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13080b03e0 msgr2=0x7f1308159470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:20.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.957+0000 7f130e1ff700 1 --2- 172.21.15.27:0/3550292236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13080b03e0 0x7f1308159470 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f12f800c7e0 tx=0x7f12f8007df0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.957+0000 7f130e1ff700 1 -- 172.21.15.27:0/3550292236 shutdown_connections 2024-01-11T09:45:20.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.957+0000 7f130e1ff700 1 --2- 172.21.15.27:0/3550292236 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f12f4066f30 0x7f12f40693e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.957+0000 7f130e1ff700 1 --2- 172.21.15.27:0/3550292236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13080b2b90 0x7f13080c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.957+0000 7f130e1ff700 1 --2- 172.21.15.27:0/3550292236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13080b03e0 0x7f1308159470 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.957+0000 7f130e1ff700 1 --2- 172.21.15.27:0/3550292236 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f13080b07b0 0x7f13080c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:20.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.957+0000 7f130e1ff700 1 -- 172.21.15.27:0/3550292236 >> 172.21.15.27:0/3550292236 conn(0x7f130806ed40 msgr2=0x7f13080b2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:20.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.957+0000 7f130e1ff700 1 -- 172.21.15.27:0/3550292236 shutdown_connections 2024-01-11T09:45:20.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:20.957+0000 7f130e1ff700 1 -- 172.21.15.27:0/3550292236 wait complete. 2024-01-11T09:45:20.967 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-11T09:45:20.968 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-11T09:45:21.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.179+0000 7f11181ff700 1 Processor -- start 2024-01-11T09:45:21.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.180+0000 7f11181ff700 1 -- start start 2024-01-11T09:45:21.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.181+0000 7f11181ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1110159250 0x7f11100b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:21.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.181+0000 7f11181ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f11100cb410 0x7f11100cb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:21.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.181+0000 7f11181ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111015a440 con 0x7f11100684d0 2024-01-11T09:45:21.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.181+0000 7f11181ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f111015ace0 con 0x7f11100cb410 2024-01-11T09:45:21.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.181+0000 7f11181ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11100b0de0 con 0x7f1110159250 2024-01-11T09:45:21.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.181+0000 7f111579a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f11100cb410 0x7f11100cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:21.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.181+0000 7f111679c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f11100684d0 0x7f11100ce2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40786/0 (socket says 172.21.15.27:40786) 2024-01-11T09:45:21.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.181+0000 7f111679c700 1 -- 172.21.15.27:0/2821898091 learned_addr learned my addr 172.21.15.27:0/2821898091 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:21.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.181+0000 7f1115f9b700 1 --2- 172.21.15.27:0/2821898091 >> v2:172.21.15.77:3300/0 conn(0x7f1110159250 0x7f11100b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:21.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.181+0000 7f111579a700 1 -- 172.21.15.27:0/2821898091 >> v2:172.21.15.77:3300/0 conn(0x7f1110159250 msgr2=0x7f11100b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:21.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.182+0000 7f111579a700 1 --2- 172.21.15.27:0/2821898091 >> v2:172.21.15.77:3300/0 conn(0x7f1110159250 0x7f11100b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:21.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.182+0000 7f111579a700 1 -- 172.21.15.27:0/2821898091 >> v1:172.21.15.27:6789/0 conn(0x7f11100684d0 legacy=0x7f11100ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:21.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.182+0000 7f111579a700 1 -- 172.21.15.27:0/2821898091 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111015a720 con 0x7f11100cb410 2024-01-11T09:45:21.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.182+0000 7f1114f99700 1 -- 172.21.15.27:0/2821898091 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125519550 0 0) 0x7f111015a440 con 0x7f11100684d0 2024-01-11T09:45:21.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.182+0000 7f1115f9b700 1 --2- 172.21.15.27:0/2821898091 >> v2:172.21.15.77:3300/0 conn(0x7f1110159250 0x7f11100b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:21.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.182+0000 7f111579a700 1 --2- 172.21.15.27:0/2821898091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11100cb410 0x7f11100cb890 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f11100ce0c0 tx=0x7f11000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=344cced0ec09745c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:21.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.182+0000 7f1114f99700 1 -- 172.21.15.27:0/2821898091 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110000b6b0 con 0x7f11100cb410 2024-01-11T09:45:21.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.182+0000 7f1114f99700 1 -- 172.21.15.27:0/2821898091 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110000b810 con 0x7f11100cb410 2024-01-11T09:45:21.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.182+0000 7f1114f99700 1 -- 172.21.15.27:0/2821898091 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110000bbb0 con 0x7f11100cb410 2024-01-11T09:45:21.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.183+0000 7f11181ff700 1 -- 172.21.15.27:0/2821898091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11100cb410 msgr2=0x7f11100cb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:21.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.183+0000 7f11181ff700 1 --2- 172.21.15.27:0/2821898091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11100cb410 0x7f11100cb890 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f11100ce0c0 tx=0x7f11000091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:21.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.183+0000 7f11181ff700 1 -- 172.21.15.27:0/2821898091 shutdown_connections 2024-01-11T09:45:21.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.183+0000 7f11181ff700 1 --2- 172.21.15.27:0/2821898091 >> v2:172.21.15.77:3300/0 conn(0x7f1110159250 0x7f11100b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:21.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.183+0000 7f11181ff700 1 --2- 172.21.15.27:0/2821898091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11100cb410 0x7f11100cb890 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:21.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.183+0000 7f11181ff700 1 -- 172.21.15.27:0/2821898091 >> 172.21.15.27:0/2821898091 conn(0x7f1110067610 msgr2=0x7f1110067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:21.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.183+0000 7f11181ff700 1 -- 172.21.15.27:0/2821898091 shutdown_connections 2024-01-11T09:45:21.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.183+0000 7f11181ff700 1 -- 172.21.15.27:0/2821898091 wait complete. 2024-01-11T09:45:21.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.185+0000 7f11181ff700 1 Processor -- start 2024-01-11T09:45:21.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.185+0000 7f11181ff700 1 -- start start 2024-01-11T09:45:21.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.187+0000 7f11181ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11100684d0 0x7f11100c6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:21.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.187+0000 7f11181ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11100cb410 0x7f11100b2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:21.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.187+0000 7f11181ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1110159250 0x7f11100b2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:21.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.187+0000 7f11181ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11100c6150 con 0x7f11100684d0 2024-01-11T09:45:21.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.187+0000 7f11181ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11100c69f0 con 0x7f1110159250 2024-01-11T09:45:21.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.187+0000 7f11181ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11100c2990 con 0x7f11100cb410 2024-01-11T09:45:21.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.187+0000 7f1115f9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11100684d0 0x7f11100c6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:21.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.187+0000 7f1115f9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11100684d0 0x7f11100c6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49938/0 (socket says 172.21.15.27:49938) 2024-01-11T09:45:21.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.187+0000 7f1115f9b700 1 -- 172.21.15.27:0/2114816624 learned_addr learned my addr 172.21.15.27:0/2114816624 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:21.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.187+0000 7f111679c700 1 --2- 172.21.15.27:0/2114816624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1110159250 0x7f11100b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:21.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.187+0000 7f111579a700 1 --2- 172.21.15.27:0/2114816624 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11100cb410 0x7f11100b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:21.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.187+0000 7f1115f9b700 1 -- 172.21.15.27:0/2114816624 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11100cb410 msgr2=0x7f11100b2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:21.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.187+0000 7f1115f9b700 1 --2- 172.21.15.27:0/2114816624 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11100cb410 0x7f11100b2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:21.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.187+0000 7f1115f9b700 1 -- 172.21.15.27:0/2114816624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1110159250 msgr2=0x7f11100b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:21.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.187+0000 7f1115f9b700 1 --2- 172.21.15.27:0/2114816624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1110159250 0x7f11100b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:21.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.188+0000 7f1115f9b700 1 -- 172.21.15.27:0/2114816624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1100009420 con 0x7f11100684d0 2024-01-11T09:45:21.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.188+0000 7f1115f9b700 1 --2- 172.21.15.27:0/2114816624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11100684d0 0x7f11100c6e80 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f110c00c830 tx=0x7f110c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:21.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.188+0000 7f1105ffb700 1 -- 172.21.15.27:0/2114816624 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c011600 con 0x7f11100684d0 2024-01-11T09:45:21.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.188+0000 7f1105ffb700 1 -- 172.21.15.27:0/2114816624 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110c011760 con 0x7f11100684d0 2024-01-11T09:45:21.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.188+0000 7f1105ffb700 1 -- 172.21.15.27:0/2114816624 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c011b20 con 0x7f11100684d0 2024-01-11T09:45:21.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.188+0000 7f11181ff700 1 -- 172.21.15.27:0/2114816624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11100c63a0 con 0x7f11100684d0 2024-01-11T09:45:21.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.188+0000 7f11181ff700 1 -- 172.21.15.27:0/2114816624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11100b39a0 con 0x7f11100684d0 2024-01-11T09:45:21.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.189+0000 7f11181ff700 1 -- 172.21.15.27:0/2114816624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11100c6cd0 con 0x7f11100684d0 2024-01-11T09:45:21.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.190+0000 7f1105ffb700 1 -- 172.21.15.27:0/2114816624 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f110c018360 con 0x7f11100684d0 2024-01-11T09:45:21.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.191+0000 7f1105ffb700 1 --2- 172.21.15.27:0/2114816624 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f10f4066f30 0x7f10f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:21.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.191+0000 7f1105ffb700 1 -- 172.21.15.27:0/2114816624 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f110c0841e0 con 0x7f11100684d0 2024-01-11T09:45:21.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.192+0000 7f111579a700 1 --2- 172.21.15.27:0/2114816624 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f10f4066f30 0x7f10f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:21.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.192+0000 7f111579a700 1 --2- 172.21.15.27:0/2114816624 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f10f4066f30 0x7f10f40693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f11000093f0 tx=0x7f1100017000 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:21.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.195+0000 7f1105ffb700 1 -- 172.21.15.27:0/2114816624 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f110c051730 con 0x7f11100684d0 2024-01-11T09:45:21.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:21.398+0000 7f11181ff700 1 -- 172.21.15.27:0/2114816624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f11100c9fe0 con 0x7f11100684d0 2024-01-11T09:45:22.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.120+0000 7f1105ffb700 1 -- 172.21.15.27:0/2114816624 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v211) v1 ==== 138+0+0 (secure 0 0 0) 0x7f110c054d50 con 0x7f11100684d0 2024-01-11T09:45:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.124+0000 7f11181ff700 1 -- 172.21.15.27:0/2114816624 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f10f4066f30 msgr2=0x7f10f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.124+0000 7f11181ff700 1 --2- 172.21.15.27:0/2114816624 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f10f4066f30 0x7f10f40693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f11000093f0 tx=0x7f1100017000 comp rx=0 tx=0).stop 2024-01-11T09:45:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.124+0000 7f11181ff700 1 -- 172.21.15.27:0/2114816624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11100684d0 msgr2=0x7f11100c6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.124+0000 7f11181ff700 1 --2- 172.21.15.27:0/2114816624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11100684d0 0x7f11100c6e80 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f110c00c830 tx=0x7f110c008040 comp rx=0 tx=0).stop 2024-01-11T09:45:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.124+0000 7f11181ff700 1 -- 172.21.15.27:0/2114816624 shutdown_connections 2024-01-11T09:45:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.124+0000 7f11181ff700 1 --2- 172.21.15.27:0/2114816624 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f10f4066f30 0x7f10f40693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.124+0000 7f11181ff700 1 --2- 172.21.15.27:0/2114816624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1110159250 0x7f11100b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.124+0000 7f11181ff700 1 --2- 172.21.15.27:0/2114816624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11100684d0 0x7f11100c6e80 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:22.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.124+0000 7f11181ff700 1 --2- 172.21.15.27:0/2114816624 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11100cb410 0x7f11100b2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:22.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.124+0000 7f11181ff700 1 -- 172.21.15.27:0/2114816624 >> 172.21.15.27:0/2114816624 conn(0x7f1110067610 msgr2=0x7f11100b0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:22.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.125+0000 7f11181ff700 1 -- 172.21.15.27:0/2114816624 shutdown_connections 2024-01-11T09:45:22.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.126+0000 7f11181ff700 1 -- 172.21.15.27:0/2114816624 wait complete. 2024-01-11T09:45:22.121 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-11T09:45:22.141 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-11T09:45:22.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.356+0000 7f36883ff700 1 Processor -- start 2024-01-11T09:45:22.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.356+0000 7f36883ff700 1 -- start start 2024-01-11T09:45:22.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.358+0000 7f36883ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f36800c8860 0x7f3680158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:22.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.358+0000 7f36883ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f36800b19d0 0x7f36800ae230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:22.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.358+0000 7f36883ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36800cabe0 con 0x7f3680158600 2024-01-11T09:45:22.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.358+0000 7f36883ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36800ae770 con 0x7f36800b19d0 2024-01-11T09:45:22.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.358+0000 7f36883ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36800ae960 con 0x7f36800c8860 2024-01-11T09:45:22.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.358+0000 7f368699c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f36800b19d0 0x7f36800ae230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:22.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.358+0000 7f368599a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3680158600 0x7f36800b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40790/0 (socket says 172.21.15.27:40790) 2024-01-11T09:45:22.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.358+0000 7f368599a700 1 -- 172.21.15.27:0/3561068621 learned_addr learned my addr 172.21.15.27:0/3561068621 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:22.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.358+0000 7f368619b700 1 --2- 172.21.15.27:0/3561068621 >> v2:172.21.15.77:3300/0 conn(0x7f36800c8860 0x7f3680158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:22.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.359+0000 7f368699c700 1 -- 172.21.15.27:0/3561068621 >> v2:172.21.15.77:3300/0 conn(0x7f36800c8860 msgr2=0x7f3680158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:22.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.359+0000 7f368699c700 1 --2- 172.21.15.27:0/3561068621 >> v2:172.21.15.77:3300/0 conn(0x7f36800c8860 0x7f3680158ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:22.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.359+0000 7f368699c700 1 -- 172.21.15.27:0/3561068621 >> v1:172.21.15.27:6789/0 conn(0x7f3680158600 legacy=0x7f36800b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:22.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.359+0000 7f368699c700 1 -- 172.21.15.27:0/3561068621 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36800ca590 con 0x7f36800b19d0 2024-01-11T09:45:22.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.359+0000 7f368619b700 1 --2- 172.21.15.27:0/3561068621 >> v2:172.21.15.77:3300/0 conn(0x7f36800c8860 0x7f3680158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:22.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.359+0000 7f3685199700 1 -- 172.21.15.27:0/3561068621 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832360939 0 0) 0x7f36800cabe0 con 0x7f3680158600 2024-01-11T09:45:22.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.359+0000 7f368699c700 1 --2- 172.21.15.27:0/3561068621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800b19d0 0x7f36800ae230 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f368006d000 tx=0x7f3678009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3612f3b8d432d93e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:22.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.359+0000 7f3685199700 1 -- 172.21.15.27:0/3561068621 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367800e070 con 0x7f36800b19d0 2024-01-11T09:45:22.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.359+0000 7f3685199700 1 -- 172.21.15.27:0/3561068621 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f367800bb60 con 0x7f36800b19d0 2024-01-11T09:45:22.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.359+0000 7f3685199700 1 -- 172.21.15.27:0/3561068621 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3678012430 con 0x7f36800b19d0 2024-01-11T09:45:22.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.360+0000 7f36883ff700 1 -- 172.21.15.27:0/3561068621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800b19d0 msgr2=0x7f36800ae230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:22.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.360+0000 7f36883ff700 1 --2- 172.21.15.27:0/3561068621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800b19d0 0x7f36800ae230 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f368006d000 tx=0x7f3678009780 comp rx=0 tx=0).stop 2024-01-11T09:45:22.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.360+0000 7f36883ff700 1 -- 172.21.15.27:0/3561068621 shutdown_connections 2024-01-11T09:45:22.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.360+0000 7f36883ff700 1 --2- 172.21.15.27:0/3561068621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800b19d0 0x7f36800ae230 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:22.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.360+0000 7f36883ff700 1 --2- 172.21.15.27:0/3561068621 >> v2:172.21.15.77:3300/0 conn(0x7f36800c8860 0x7f3680158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:22.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.360+0000 7f36883ff700 1 -- 172.21.15.27:0/3561068621 >> 172.21.15.27:0/3561068621 conn(0x7f36800680a0 msgr2=0x7f3680069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:22.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.360+0000 7f36883ff700 1 -- 172.21.15.27:0/3561068621 shutdown_connections 2024-01-11T09:45:22.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.360+0000 7f36883ff700 1 -- 172.21.15.27:0/3561068621 wait complete. 2024-01-11T09:45:22.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.361+0000 7f36883ff700 1 Processor -- start 2024-01-11T09:45:22.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.361+0000 7f36883ff700 1 -- start start 2024-01-11T09:45:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.362+0000 7f36883ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36800b19d0 0x7f3680148bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.362+0000 7f36883ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800c8860 0x7f36801568a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.362+0000 7f36883ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3680158600 0x7f36800ad760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.362+0000 7f36883ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3680156410 con 0x7f36800c8860 2024-01-11T09:45:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.362+0000 7f36883ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3680155800 con 0x7f36800b19d0 2024-01-11T09:45:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.362+0000 7f36883ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3680155ae0 con 0x7f3680158600 2024-01-11T09:45:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.363+0000 7f368599a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800c8860 0x7f36801568a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.363+0000 7f368599a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800c8860 0x7f36801568a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49960/0 (socket says 172.21.15.27:49960) 2024-01-11T09:45:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.363+0000 7f368599a700 1 -- 172.21.15.27:0/848788270 learned_addr learned my addr 172.21.15.27:0/848788270 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.363+0000 7f368619b700 1 --2- 172.21.15.27:0/848788270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36800b19d0 0x7f3680148bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.363+0000 7f368699c700 1 --2- 172.21.15.27:0/848788270 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3680158600 0x7f36800ad760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:22.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.363+0000 7f368599a700 1 -- 172.21.15.27:0/848788270 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3680158600 msgr2=0x7f36800ad760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:22.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.363+0000 7f368599a700 1 --2- 172.21.15.27:0/848788270 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3680158600 0x7f36800ad760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:22.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.363+0000 7f368599a700 1 -- 172.21.15.27:0/848788270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36800b19d0 msgr2=0x7f3680148bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:22.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.363+0000 7f368599a700 1 --2- 172.21.15.27:0/848788270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36800b19d0 0x7f3680148bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:22.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.363+0000 7f368599a700 1 -- 172.21.15.27:0/848788270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367c009420 con 0x7f36800c8860 2024-01-11T09:45:22.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.363+0000 7f368599a700 1 --2- 172.21.15.27:0/848788270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800c8860 0x7f36801568a0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f367000c5b0 tx=0x7f3670001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:22.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.364+0000 7f36767fc700 1 -- 172.21.15.27:0/848788270 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3670005b30 con 0x7f36800c8860 2024-01-11T09:45:22.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.364+0000 7f36767fc700 1 -- 172.21.15.27:0/848788270 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3670005c90 con 0x7f36800c8860 2024-01-11T09:45:22.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.364+0000 7f36767fc700 1 -- 172.21.15.27:0/848788270 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3670007300 con 0x7f36800c8860 2024-01-11T09:45:22.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.364+0000 7f36883ff700 1 -- 172.21.15.27:0/848788270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3678009a20 con 0x7f36800c8860 2024-01-11T09:45:22.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.364+0000 7f36883ff700 1 -- 172.21.15.27:0/848788270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36801560a0 con 0x7f36800c8860 2024-01-11T09:45:22.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.365+0000 7f36883ff700 1 -- 172.21.15.27:0/848788270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f368004c3c0 con 0x7f36800c8860 2024-01-11T09:45:22.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.366+0000 7f36767fc700 1 -- 172.21.15.27:0/848788270 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f367001b3b0 con 0x7f36800c8860 2024-01-11T09:45:22.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.368+0000 7f36767fc700 1 --2- 172.21.15.27:0/848788270 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3664066f30 0x7f36640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:22.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.368+0000 7f36767fc700 1 -- 172.21.15.27:0/848788270 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f3670083c80 con 0x7f36800c8860 2024-01-11T09:45:22.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.368+0000 7f368619b700 1 --2- 172.21.15.27:0/848788270 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3664066f30 0x7f36640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:22.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.369+0000 7f368619b700 1 --2- 172.21.15.27:0/848788270 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3664066f30 0x7f36640693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f36800ab4c0 tx=0x7f367c0091a0 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:22.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.372+0000 7f36767fc700 1 -- 172.21.15.27:0/848788270 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3670051040 con 0x7f36800c8860 2024-01-11T09:45:22.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:22.577+0000 7f36883ff700 1 -- 172.21.15.27:0/848788270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f36800ae550 con 0x7f36800c8860 2024-01-11T09:45:23.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.128+0000 7f36767fc700 1 -- 172.21.15.27:0/848788270 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v212) v1 ==== 170+0+0 (secure 0 0 0) 0x7f36700170d0 con 0x7f36800c8860 2024-01-11T09:45:23.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.132+0000 7f36883ff700 1 -- 172.21.15.27:0/848788270 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3664066f30 msgr2=0x7f36640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:23.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.132+0000 7f36883ff700 1 --2- 172.21.15.27:0/848788270 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3664066f30 0x7f36640693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f36800ab4c0 tx=0x7f367c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:23.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.132+0000 7f36883ff700 1 -- 172.21.15.27:0/848788270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800c8860 msgr2=0x7f36801568a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:23.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.132+0000 7f36883ff700 1 --2- 172.21.15.27:0/848788270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800c8860 0x7f36801568a0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f367000c5b0 tx=0x7f3670001070 comp rx=0 tx=0).stop 2024-01-11T09:45:23.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.134+0000 7f36883ff700 1 -- 172.21.15.27:0/848788270 shutdown_connections 2024-01-11T09:45:23.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.134+0000 7f36883ff700 1 --2- 172.21.15.27:0/848788270 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3680158600 0x7f36800ad760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:23.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.134+0000 7f36883ff700 1 --2- 172.21.15.27:0/848788270 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3664066f30 0x7f36640693e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:23.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.134+0000 7f36883ff700 1 --2- 172.21.15.27:0/848788270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36800b19d0 0x7f3680148bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:23.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.134+0000 7f36883ff700 1 --2- 172.21.15.27:0/848788270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36800c8860 0x7f36801568a0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:23.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.134+0000 7f36883ff700 1 -- 172.21.15.27:0/848788270 >> 172.21.15.27:0/848788270 conn(0x7f36800680a0 msgr2=0x7f3680158ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:23.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.135+0000 7f36883ff700 1 -- 172.21.15.27:0/848788270 shutdown_connections 2024-01-11T09:45:23.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.135+0000 7f36883ff700 1 -- 172.21.15.27:0/848788270 wait complete. 2024-01-11T09:45:23.132 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-11T09:45:23.159 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-11T09:45:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.370+0000 7f999cbff700 1 Processor -- start 2024-01-11T09:45:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.371+0000 7f999cbff700 1 -- start start 2024-01-11T09:45:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.372+0000 7f999cbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f99980ccd70 0x7f99980cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.372+0000 7f999cbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f999815ae70 0x7f999815b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.372+0000 7f999cbff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99980cf850 con 0x7f999815a1b0 2024-01-11T09:45:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.372+0000 7f999cbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f999815ac90 con 0x7f99980ccd70 2024-01-11T09:45:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.372+0000 7f999cbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99980cec40 con 0x7f999815ae70 2024-01-11T09:45:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.373+0000 7f9996d9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f999815ae70 0x7f999815b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.373+0000 7f9995d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f99980ccd70 0x7f99980cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.373+0000 7f999659c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f999815a1b0 0x7f999815a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:49446/0 (socket says 172.21.15.27:49446) 2024-01-11T09:45:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.373+0000 7f999659c700 1 -- 172.21.15.27:0/1922277509 learned_addr learned my addr 172.21.15.27:0/1922277509 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.373+0000 7f9996d9d700 1 -- 172.21.15.27:0/1922277509 >> v2:172.21.15.77:3300/0 conn(0x7f99980ccd70 msgr2=0x7f99980cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.373+0000 7f9996d9d700 1 --2- 172.21.15.27:0/1922277509 >> v2:172.21.15.77:3300/0 conn(0x7f99980ccd70 0x7f99980cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.373+0000 7f9996d9d700 1 -- 172.21.15.27:0/1922277509 >> v1:172.21.15.77:6789/0 conn(0x7f999815a1b0 legacy=0x7f999815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.373+0000 7f9996d9d700 1 -- 172.21.15.27:0/1922277509 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99980cef20 con 0x7f999815ae70 2024-01-11T09:45:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.373+0000 7f9995d9b700 1 --2- 172.21.15.27:0/1922277509 >> v2:172.21.15.77:3300/0 conn(0x7f99980ccd70 0x7f99980cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.373+0000 7f9996d9d700 1 --2- 172.21.15.27:0/1922277509 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f999815ae70 0x7f999815b2f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9990009e60 tx=0x7f9990009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae1f76576ea9bb97 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.374+0000 7f999559a700 1 -- 172.21.15.27:0/1922277509 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999000e070 con 0x7f999815ae70 2024-01-11T09:45:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.374+0000 7f999559a700 1 -- 172.21.15.27:0/1922277509 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999000bb60 con 0x7f999815ae70 2024-01-11T09:45:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.374+0000 7f999559a700 1 -- 172.21.15.27:0/1922277509 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9990012430 con 0x7f999815ae70 2024-01-11T09:45:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.374+0000 7f999cbff700 1 -- 172.21.15.27:0/1922277509 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f999815ae70 msgr2=0x7f999815b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.374+0000 7f999cbff700 1 --2- 172.21.15.27:0/1922277509 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f999815ae70 0x7f999815b2f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9990009e60 tx=0x7f9990009780 comp rx=0 tx=0).stop 2024-01-11T09:45:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.374+0000 7f999cbff700 1 -- 172.21.15.27:0/1922277509 shutdown_connections 2024-01-11T09:45:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.374+0000 7f999cbff700 1 --2- 172.21.15.27:0/1922277509 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f999815ae70 0x7f999815b2f0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.374+0000 7f999cbff700 1 --2- 172.21.15.27:0/1922277509 >> v2:172.21.15.77:3300/0 conn(0x7f99980ccd70 0x7f99980cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.374+0000 7f999cbff700 1 -- 172.21.15.27:0/1922277509 >> 172.21.15.27:0/1922277509 conn(0x7f999806ed40 msgr2=0x7f999806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.375+0000 7f999cbff700 1 -- 172.21.15.27:0/1922277509 shutdown_connections 2024-01-11T09:45:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.375+0000 7f999cbff700 1 -- 172.21.15.27:0/1922277509 wait complete. 2024-01-11T09:45:23.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.376+0000 7f999cbff700 1 Processor -- start 2024-01-11T09:45:23.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.376+0000 7f999cbff700 1 -- start start 2024-01-11T09:45:23.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.377+0000 7f999cbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99980ccd70 0x7f999814f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:23.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.377+0000 7f999cbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f999815a1b0 0x7f99980b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:23.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.377+0000 7f999cbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999815ae70 0x7f99980b4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:23.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.377+0000 7f999cbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99980b5690 con 0x7f999815ae70 2024-01-11T09:45:23.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.377+0000 7f999cbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99980b47b0 con 0x7f99980ccd70 2024-01-11T09:45:23.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.377+0000 7f999cbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99980b4a80 con 0x7f999815a1b0 2024-01-11T09:45:23.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.377+0000 7f9996d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999815ae70 0x7f99980b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:23.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.378+0000 7f9996d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999815ae70 0x7f99980b4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49972/0 (socket says 172.21.15.27:49972) 2024-01-11T09:45:23.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.378+0000 7f9996d9d700 1 -- 172.21.15.27:0/434073140 learned_addr learned my addr 172.21.15.27:0/434073140 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:23.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.378+0000 7f9995d9b700 1 --2- 172.21.15.27:0/434073140 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f999815a1b0 0x7f99980b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:23.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.378+0000 7f999659c700 1 --2- 172.21.15.27:0/434073140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99980ccd70 0x7f999814f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:23.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.378+0000 7f9996d9d700 1 -- 172.21.15.27:0/434073140 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f999815a1b0 msgr2=0x7f99980b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:23.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.378+0000 7f9996d9d700 1 --2- 172.21.15.27:0/434073140 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f999815a1b0 0x7f99980b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:23.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.378+0000 7f9996d9d700 1 -- 172.21.15.27:0/434073140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99980ccd70 msgr2=0x7f999814f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:23.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.378+0000 7f9996d9d700 1 --2- 172.21.15.27:0/434073140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99980ccd70 0x7f999814f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:23.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.378+0000 7f9996d9d700 1 -- 172.21.15.27:0/434073140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998c009420 con 0x7f999815ae70 2024-01-11T09:45:23.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.378+0000 7f9996d9d700 1 --2- 172.21.15.27:0/434073140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999815ae70 0x7f99980b4270 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f9998089840 tx=0x7f9990019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:23.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.379+0000 7f99867fc700 1 -- 172.21.15.27:0/434073140 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9990012ea0 con 0x7f999815ae70 2024-01-11T09:45:23.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.379+0000 7f99867fc700 1 -- 172.21.15.27:0/434073140 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9990027470 con 0x7f999815ae70 2024-01-11T09:45:23.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.379+0000 7f99867fc700 1 -- 172.21.15.27:0/434073140 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9990027850 con 0x7f999815ae70 2024-01-11T09:45:23.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.379+0000 7f999cbff700 1 -- 172.21.15.27:0/434073140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9990009a20 con 0x7f999815ae70 2024-01-11T09:45:23.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.379+0000 7f999cbff700 1 -- 172.21.15.27:0/434073140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99980b5040 con 0x7f999815ae70 2024-01-11T09:45:23.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.380+0000 7f999cbff700 1 -- 172.21.15.27:0/434073140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999804c3c0 con 0x7f999815ae70 2024-01-11T09:45:23.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.381+0000 7f99867fc700 1 -- 172.21.15.27:0/434073140 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9990024070 con 0x7f999815ae70 2024-01-11T09:45:23.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.383+0000 7f99867fc700 1 --2- 172.21.15.27:0/434073140 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9978066f30 0x7f99780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:23.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.383+0000 7f99867fc700 1 -- 172.21.15.27:0/434073140 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 5717+0+0 (secure 0 0 0) 0x7f9990087510 con 0x7f999815ae70 2024-01-11T09:45:23.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.383+0000 7f999659c700 1 --2- 172.21.15.27:0/434073140 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9978066f30 0x7f99780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:23.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.384+0000 7f999659c700 1 --2- 172.21.15.27:0/434073140 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9978066f30 0x7f99780693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9988007130 tx=0x7f9988007eb0 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:23.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.388+0000 7f99867fc700 1 -- 172.21.15.27:0/434073140 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9990054750 con 0x7f999815ae70 2024-01-11T09:45:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:23.635+0000 7f999cbff700 1 -- 172.21.15.27:0/434073140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f99980b2d60 con 0x7f999815ae70 2024-01-11T09:45:24.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.140+0000 7f99867fc700 1 -- 172.21.15.27:0/434073140 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 26 and data pool 27 v107) v1 ==== 174+0+0 (secure 0 0 0) 0x7f9990057d70 con 0x7f999815ae70 2024-01-11T09:45:24.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.148+0000 7f999cbff700 1 -- 172.21.15.27:0/434073140 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9978066f30 msgr2=0x7f99780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:24.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.148+0000 7f999cbff700 1 --2- 172.21.15.27:0/434073140 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9978066f30 0x7f99780693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9988007130 tx=0x7f9988007eb0 comp rx=0 tx=0).stop 2024-01-11T09:45:24.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.148+0000 7f999cbff700 1 -- 172.21.15.27:0/434073140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999815ae70 msgr2=0x7f99980b4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:24.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.148+0000 7f999cbff700 1 --2- 172.21.15.27:0/434073140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999815ae70 0x7f99980b4270 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f9998089840 tx=0x7f9990019000 comp rx=0 tx=0).stop 2024-01-11T09:45:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.149+0000 7f999cbff700 1 -- 172.21.15.27:0/434073140 shutdown_connections 2024-01-11T09:45:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.149+0000 7f999cbff700 1 --2- 172.21.15.27:0/434073140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999815ae70 0x7f99980b4270 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.149+0000 7f999cbff700 1 --2- 172.21.15.27:0/434073140 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9978066f30 0x7f99780693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.149+0000 7f999cbff700 1 --2- 172.21.15.27:0/434073140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99980ccd70 0x7f999814f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.149+0000 7f999cbff700 1 --2- 172.21.15.27:0/434073140 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f999815a1b0 0x7f99980b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:24.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.149+0000 7f999cbff700 1 -- 172.21.15.27:0/434073140 >> 172.21.15.27:0/434073140 conn(0x7f999806ed40 msgr2=0x7f99980c6b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:24.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.152+0000 7f999cbff700 1 -- 172.21.15.27:0/434073140 shutdown_connections 2024-01-11T09:45:24.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.155+0000 7f999cbff700 1 -- 172.21.15.27:0/434073140 wait complete. 2024-01-11T09:45:24.153 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 26 and data pool 27 2024-01-11T09:45:24.177 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:45:24.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.386+0000 7fc82d3ff700 1 Processor -- start 2024-01-11T09:45:24.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.386+0000 7fc82d3ff700 1 -- start start 2024-01-11T09:45:24.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.387+0000 7fc82d3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc8280c8620 0x7fc8280af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:24.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.387+0000 7fc82d3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc828069d40 0x7fc8280b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:24.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.387+0000 7fc82d3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc828157390 con 0x7fc8280c89f0 2024-01-11T09:45:24.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.387+0000 7fc82d3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8280adcd0 con 0x7fc8280c8620 2024-01-11T09:45:24.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.387+0000 7fc82d3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8280afeb0 con 0x7fc828069d40 2024-01-11T09:45:24.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.388+0000 7fc826ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc8280c8620 0x7fc8280af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:24.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.388+0000 7fc826ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc8280c8620 0x7fc8280af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51762/0 (socket says 172.21.15.27:51762) 2024-01-11T09:45:24.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.388+0000 7fc826ffd700 1 -- 172.21.15.27:0/516880781 learned_addr learned my addr 172.21.15.27:0/516880781 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:24.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.388+0000 7fc8267fc700 1 --2- 172.21.15.27:0/516880781 >> v2:172.21.15.77:3300/0 conn(0x7fc828069d40 0x7fc8280b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:24.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.388+0000 7fc826ffd700 1 -- 172.21.15.27:0/516880781 >> v2:172.21.15.77:3300/0 conn(0x7fc828069d40 msgr2=0x7fc8280b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:24.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.388+0000 7fc826ffd700 1 --2- 172.21.15.27:0/516880781 >> v2:172.21.15.77:3300/0 conn(0x7fc828069d40 0x7fc8280b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:24.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.388+0000 7fc826ffd700 1 -- 172.21.15.27:0/516880781 >> v1:172.21.15.27:6789/0 conn(0x7fc8280c89f0 legacy=0x7fc828159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:24.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.388+0000 7fc826ffd700 1 -- 172.21.15.27:0/516880781 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82806a200 con 0x7fc8280c8620 2024-01-11T09:45:24.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.388+0000 7fc825ffb700 1 -- 172.21.15.27:0/516880781 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420703810 0 0) 0x7fc828157390 con 0x7fc8280c89f0 2024-01-11T09:45:24.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.388+0000 7fc826ffd700 1 --2- 172.21.15.27:0/516880781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8280c8620 0x7fc8280af970 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fc8180099b0 tx=0x7fc818009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a862f4914fc39ed3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:24.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.389+0000 7fc825ffb700 1 -- 172.21.15.27:0/516880781 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81800b820 con 0x7fc8280c8620 2024-01-11T09:45:24.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.389+0000 7fc825ffb700 1 -- 172.21.15.27:0/516880781 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc81800b980 con 0x7fc8280c8620 2024-01-11T09:45:24.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.389+0000 7fc825ffb700 1 -- 172.21.15.27:0/516880781 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81800bd20 con 0x7fc8280c8620 2024-01-11T09:45:24.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.389+0000 7fc82d3ff700 1 -- 172.21.15.27:0/516880781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8280c8620 msgr2=0x7fc8280af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:24.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.389+0000 7fc82d3ff700 1 --2- 172.21.15.27:0/516880781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8280c8620 0x7fc8280af970 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fc8180099b0 tx=0x7fc818009250 comp rx=0 tx=0).stop 2024-01-11T09:45:24.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.389+0000 7fc82d3ff700 1 -- 172.21.15.27:0/516880781 shutdown_connections 2024-01-11T09:45:24.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.389+0000 7fc82d3ff700 1 --2- 172.21.15.27:0/516880781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8280c8620 0x7fc8280af970 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:24.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.389+0000 7fc82d3ff700 1 --2- 172.21.15.27:0/516880781 >> v2:172.21.15.77:3300/0 conn(0x7fc828069d40 0x7fc8280b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:24.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.389+0000 7fc82d3ff700 1 -- 172.21.15.27:0/516880781 >> 172.21.15.27:0/516880781 conn(0x7fc8280680e0 msgr2=0x7fc828069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:24.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.389+0000 7fc82d3ff700 1 -- 172.21.15.27:0/516880781 shutdown_connections 2024-01-11T09:45:24.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.389+0000 7fc82d3ff700 1 -- 172.21.15.27:0/516880781 wait complete. 2024-01-11T09:45:24.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.391+0000 7fc82d3ff700 1 Processor -- start 2024-01-11T09:45:24.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.391+0000 7fc82d3ff700 1 -- start start 2024-01-11T09:45:24.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.392+0000 7fc82d3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc828069d40 0x7fc8280c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:24.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.392+0000 7fc82d3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8280c8620 0x7fc828154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:24.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.392+0000 7fc82d3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8280c89f0 0x7fc8281696c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:24.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.392+0000 7fc82d3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc828155c00 con 0x7fc828069d40 2024-01-11T09:45:24.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.392+0000 7fc82d3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc828154ff0 con 0x7fc8280c8620 2024-01-11T09:45:24.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.392+0000 7fc82d3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8281552d0 con 0x7fc8280c89f0 2024-01-11T09:45:24.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.392+0000 7fc826ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc828069d40 0x7fc8280c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:24.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.392+0000 7fc826ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc828069d40 0x7fc8280c7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51768/0 (socket says 172.21.15.27:51768) 2024-01-11T09:45:24.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.392+0000 7fc826ffd700 1 -- 172.21.15.27:0/3286280224 learned_addr learned my addr 172.21.15.27:0/3286280224 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:24.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.393+0000 7fc8277fe700 1 --2- 172.21.15.27:0/3286280224 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8280c89f0 0x7fc8281696c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:24.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.393+0000 7fc8267fc700 1 --2- 172.21.15.27:0/3286280224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8280c8620 0x7fc828154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:24.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.393+0000 7fc826ffd700 1 -- 172.21.15.27:0/3286280224 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8280c89f0 msgr2=0x7fc8281696c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:24.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.393+0000 7fc826ffd700 1 --2- 172.21.15.27:0/3286280224 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8280c89f0 0x7fc8281696c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:24.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.393+0000 7fc826ffd700 1 -- 172.21.15.27:0/3286280224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8280c8620 msgr2=0x7fc828154610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:24.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.393+0000 7fc826ffd700 1 --2- 172.21.15.27:0/3286280224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8280c8620 0x7fc828154610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:24.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.393+0000 7fc826ffd700 1 -- 172.21.15.27:0/3286280224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8180094f0 con 0x7fc828069d40 2024-01-11T09:45:24.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.393+0000 7fc8267fc700 1 --2- 172.21.15.27:0/3286280224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8280c8620 0x7fc828154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:24.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.393+0000 7fc826ffd700 1 --2- 172.21.15.27:0/3286280224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc828069d40 0x7fc8280c7840 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fc828066180 tx=0x7fc818017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:24.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.393+0000 7fc80effd700 1 -- 172.21.15.27:0/3286280224 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc818024440 con 0x7fc828069d40 2024-01-11T09:45:24.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.393+0000 7fc80effd700 1 -- 172.21.15.27:0/3286280224 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8180245a0 con 0x7fc828069d40 2024-01-11T09:45:24.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.393+0000 7fc80effd700 1 -- 172.21.15.27:0/3286280224 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc818024960 con 0x7fc828069d40 2024-01-11T09:45:24.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.393+0000 7fc82d3ff700 1 -- 172.21.15.27:0/3286280224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8281555b0 con 0x7fc828069d40 2024-01-11T09:45:24.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.394+0000 7fc82d3ff700 1 -- 172.21.15.27:0/3286280224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8280c7d80 con 0x7fc828069d40 2024-01-11T09:45:24.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.395+0000 7fc82d3ff700 1 -- 172.21.15.27:0/3286280224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc828155ee0 con 0x7fc828069d40 2024-01-11T09:45:24.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.396+0000 7fc80effd700 1 -- 172.21.15.27:0/3286280224 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc81801a420 con 0x7fc828069d40 2024-01-11T09:45:24.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.398+0000 7fc80effd700 1 --2- 172.21.15.27:0/3286280224 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc814066f30 0x7fc8140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:24.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.398+0000 7fc80effd700 1 -- 172.21.15.27:0/3286280224 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5817+0+0 (secure 0 0 0) 0x7fc8180865d0 con 0x7fc828069d40 2024-01-11T09:45:24.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.398+0000 7fc8267fc700 1 --2- 172.21.15.27:0/3286280224 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc814066f30 0x7fc8140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:24.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.399+0000 7fc8267fc700 1 --2- 172.21.15.27:0/3286280224 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc814066f30 0x7fc8140693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc82806f460 tx=0x7fc81c008040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:24.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.402+0000 7fc80effd700 1 -- 172.21.15.27:0/3286280224 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc8180537e0 con 0x7fc828069d40 2024-01-11T09:45:24.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.603+0000 7fc82d3ff700 1 -- 172.21.15.27:0/3286280224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8280718d0 con 0x7fc828069d40 2024-01-11T09:45:24.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.604+0000 7fc80effd700 1 -- 172.21.15.27:0/3286280224 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+15508 (secure 0 0 0) 0x7fc818056e00 con 0x7fc828069d40 2024-01-11T09:45:24.597 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:24.597 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":213,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:45:23.637289+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:45:21.455328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-01-11T09:45:22.634298+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:45:24.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.606+0000 7fc82d3ff700 1 -- 172.21.15.27:0/3286280224 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc814066f30 msgr2=0x7fc8140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:24.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.606+0000 7fc82d3ff700 1 --2- 172.21.15.27:0/3286280224 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc814066f30 0x7fc8140693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc82806f460 tx=0x7fc81c008040 comp rx=0 tx=0).stop 2024-01-11T09:45:24.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.606+0000 7fc82d3ff700 1 -- 172.21.15.27:0/3286280224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc828069d40 msgr2=0x7fc8280c7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.606+0000 7fc82d3ff700 1 --2- 172.21.15.27:0/3286280224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc828069d40 0x7fc8280c7840 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fc828066180 tx=0x7fc818017000 comp rx=0 tx=0).stop 2024-01-11T09:45:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.606+0000 7fc82d3ff700 1 -- 172.21.15.27:0/3286280224 shutdown_connections 2024-01-11T09:45:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.606+0000 7fc82d3ff700 1 --2- 172.21.15.27:0/3286280224 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8280c89f0 0x7fc8281696c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.606+0000 7fc82d3ff700 1 --2- 172.21.15.27:0/3286280224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc828069d40 0x7fc8280c7840 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.606+0000 7fc82d3ff700 1 --2- 172.21.15.27:0/3286280224 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc814066f30 0x7fc8140693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.606+0000 7fc82d3ff700 1 --2- 172.21.15.27:0/3286280224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8280c8620 0x7fc828154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.606+0000 7fc82d3ff700 1 -- 172.21.15.27:0/3286280224 >> 172.21.15.27:0/3286280224 conn(0x7fc8280680e0 msgr2=0x7fc8280695b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.607+0000 7fc82d3ff700 1 -- 172.21.15.27:0/3286280224 shutdown_connections 2024-01-11T09:45:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.607+0000 7fc82d3ff700 1 -- 172.21.15.27:0/3286280224 wait complete. 2024-01-11T09:45:24.617 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:45:24.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.824+0000 7f9a265ff700 1 Processor -- start 2024-01-11T09:45:24.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.825+0000 7f9a265ff700 1 -- start start 2024-01-11T09:45:24.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.826+0000 7f9a265ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9a2006b610 0x7f9a20158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:24.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.826+0000 7f9a265ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a20158b60 con 0x7f9a201575e0 2024-01-11T09:45:24.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.826+0000 7f9a265ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a201599b0 con 0x7f9a200b19d0 2024-01-11T09:45:24.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.826+0000 7f9a265ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a200cb5e0 con 0x7f9a2006b610 2024-01-11T09:45:24.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.826+0000 7f9a1ffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9a201575e0 0x7f9a200b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55944/0 (socket says 172.21.15.27:55944) 2024-01-11T09:45:24.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.826+0000 7f9a1ffff700 1 -- 172.21.15.27:0/3055289942 learned_addr learned my addr 172.21.15.27:0/3055289942 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:24.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.826+0000 7f9a24b9c700 1 --2- 172.21.15.27:0/3055289942 >> v2:172.21.15.77:3300/0 conn(0x7f9a2006b610 0x7f9a20158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.827+0000 7f9a1effd700 1 -- 172.21.15.27:0/3055289942 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210431873 0 0) 0x7f9a10001110 con 0x7f9a201575e0 2024-01-11T09:45:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.827+0000 7f9a1effd700 1 -- 172.21.15.27:0/3055289942 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a08003570 con 0x7f9a201575e0 2024-01-11T09:45:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.827+0000 7f9a1effd700 1 -- 172.21.15.27:0/3055289942 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592738692 0 0) 0x7f9a201599b0 con 0x7f9a200b19d0 2024-01-11T09:45:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.827+0000 7f9a1effd700 1 -- 172.21.15.27:0/3055289942 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a10001110 con 0x7f9a200b19d0 2024-01-11T09:45:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.827+0000 7f9a1effd700 1 -- 172.21.15.27:0/3055289942 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1937916580 0 0) 0x7f9a08003570 con 0x7f9a201575e0 2024-01-11T09:45:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.827+0000 7f9a24b9c700 1 -- 172.21.15.27:0/3055289942 >> v1:172.21.15.77:6789/0 conn(0x7f9a200b19d0 legacy=0x7f9a20159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.827+0000 7f9a24b9c700 1 -- 172.21.15.27:0/3055289942 >> v1:172.21.15.27:6789/0 conn(0x7f9a201575e0 legacy=0x7f9a200b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.827+0000 7f9a24b9c700 1 -- 172.21.15.27:0/3055289942 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a200c0f90 con 0x7f9a2006b610 2024-01-11T09:45:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.827+0000 7f9a24b9c700 1 --2- 172.21.15.27:0/3055289942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a2006b610 0x7f9a20158620 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f9a18009e60 tx=0x7f9a18009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80cc78925ccfdfdc server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.827+0000 7f9a1effd700 1 -- 172.21.15.27:0/3055289942 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1800e070 con 0x7f9a2006b610 2024-01-11T09:45:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.828+0000 7f9a1effd700 1 -- 172.21.15.27:0/3055289942 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a1800bab0 con 0x7f9a2006b610 2024-01-11T09:45:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.828+0000 7f9a1effd700 1 -- 172.21.15.27:0/3055289942 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1800be70 con 0x7f9a2006b610 2024-01-11T09:45:24.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.828+0000 7f9a265ff700 1 -- 172.21.15.27:0/3055289942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a2006b610 msgr2=0x7f9a20158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:24.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.828+0000 7f9a265ff700 1 --2- 172.21.15.27:0/3055289942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a2006b610 0x7f9a20158620 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f9a18009e60 tx=0x7f9a18009780 comp rx=0 tx=0).stop 2024-01-11T09:45:24.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.828+0000 7f9a265ff700 1 -- 172.21.15.27:0/3055289942 shutdown_connections 2024-01-11T09:45:24.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.828+0000 7f9a265ff700 1 --2- 172.21.15.27:0/3055289942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a2006b610 0x7f9a20158620 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:24.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.828+0000 7f9a265ff700 1 -- 172.21.15.27:0/3055289942 >> 172.21.15.27:0/3055289942 conn(0x7f9a200680e0 msgr2=0x7f9a20069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:24.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.828+0000 7f9a265ff700 1 -- 172.21.15.27:0/3055289942 shutdown_connections 2024-01-11T09:45:24.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.828+0000 7f9a265ff700 1 -- 172.21.15.27:0/3055289942 wait complete. 2024-01-11T09:45:24.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.830+0000 7f9a265ff700 1 Processor -- start 2024-01-11T09:45:24.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.830+0000 7f9a265ff700 1 -- start start 2024-01-11T09:45:24.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.831+0000 7f9a265ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a2006b610 0x7f9a20146c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:24.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.831+0000 7f9a265ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a200b19d0 0x7f9a2015db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:24.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.831+0000 7f9a265ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a201575e0 0x7f9a2015e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:24.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.831+0000 7f9a265ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2015bb80 con 0x7f9a201575e0 2024-01-11T09:45:24.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.831+0000 7f9a265ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2015b290 con 0x7f9a2006b610 2024-01-11T09:45:24.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.831+0000 7f9a265ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a200c7160 con 0x7f9a200b19d0 2024-01-11T09:45:24.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.831+0000 7f9a24b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a201575e0 0x7f9a2015e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:24.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.831+0000 7f9a24b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a201575e0 0x7f9a2015e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51778/0 (socket says 172.21.15.27:51778) 2024-01-11T09:45:24.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.831+0000 7f9a24b9c700 1 -- 172.21.15.27:0/4250539739 learned_addr learned my addr 172.21.15.27:0/4250539739 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:24.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.831+0000 7f9a1f7fe700 1 --2- 172.21.15.27:0/4250539739 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a200b19d0 0x7f9a2015db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:24.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.831+0000 7f9a1ffff700 1 --2- 172.21.15.27:0/4250539739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a2006b610 0x7f9a20146c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:24.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.832+0000 7f9a24b9c700 1 -- 172.21.15.27:0/4250539739 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a200b19d0 msgr2=0x7f9a2015db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:24.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.832+0000 7f9a24b9c700 1 --2- 172.21.15.27:0/4250539739 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a200b19d0 0x7f9a2015db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:24.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.832+0000 7f9a24b9c700 1 -- 172.21.15.27:0/4250539739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a2006b610 msgr2=0x7f9a20146c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:24.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.832+0000 7f9a24b9c700 1 --2- 172.21.15.27:0/4250539739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a2006b610 0x7f9a20146c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:24.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.832+0000 7f9a24b9c700 1 -- 172.21.15.27:0/4250539739 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a18009a20 con 0x7f9a201575e0 2024-01-11T09:45:24.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.832+0000 7f9a1ffff700 1 --2- 172.21.15.27:0/4250539739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a2006b610 0x7f9a20146c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:24.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.832+0000 7f9a24b9c700 1 --2- 172.21.15.27:0/4250539739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a201575e0 0x7f9a2015e040 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f9a20066180 tx=0x7f9a18019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:24.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.832+0000 7f9a07fff700 1 -- 172.21.15.27:0/4250539739 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a18012e60 con 0x7f9a201575e0 2024-01-11T09:45:24.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.832+0000 7f9a07fff700 1 -- 172.21.15.27:0/4250539739 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a18026480 con 0x7f9a201575e0 2024-01-11T09:45:24.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.832+0000 7f9a265ff700 1 -- 172.21.15.27:0/4250539739 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a2015b530 con 0x7f9a201575e0 2024-01-11T09:45:24.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.832+0000 7f9a07fff700 1 -- 172.21.15.27:0/4250539739 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a18026840 con 0x7f9a201575e0 2024-01-11T09:45:24.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.833+0000 7f9a265ff700 1 -- 172.21.15.27:0/4250539739 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a20147190 con 0x7f9a201575e0 2024-01-11T09:45:24.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.834+0000 7f9a265ff700 1 -- 172.21.15.27:0/4250539739 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a2015bd30 con 0x7f9a201575e0 2024-01-11T09:45:24.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.835+0000 7f9a07fff700 1 -- 172.21.15.27:0/4250539739 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9a1801c400 con 0x7f9a201575e0 2024-01-11T09:45:24.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.835+0000 7f9a07fff700 1 --2- 172.21.15.27:0/4250539739 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9a0c066f30 0x7f9a0c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:24.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.836+0000 7f9a07fff700 1 -- 172.21.15.27:0/4250539739 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f9a18086460 con 0x7f9a201575e0 2024-01-11T09:45:24.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.836+0000 7f9a1ffff700 1 --2- 172.21.15.27:0/4250539739 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9a0c066f30 0x7f9a0c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.837+0000 7f9a1ffff700 1 --2- 172.21.15.27:0/4250539739 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9a0c066f30 0x7f9a0c0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9a2015d700 tx=0x7f9a10007c60 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:24.839+0000 7f9a07fff700 1 -- 172.21.15.27:0/4250539739 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9a2015bd30 con 0x7f9a201575e0 2024-01-11T09:45:25.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.044+0000 7f9a265ff700 1 -- 172.21.15.27:0/4250539739 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a200718d0 con 0x7f9a201575e0 2024-01-11T09:45:25.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.045+0000 7f9a07fff700 1 -- 172.21.15.27:0/4250539739 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+15508 (secure 0 0 0) 0x7f9a18053660 con 0x7f9a201575e0 2024-01-11T09:45:25.038 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:25.038 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":213,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:45:23.637289+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:45:21.455328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-01-11T09:45:22.634298+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:45:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.048+0000 7f9a265ff700 1 -- 172.21.15.27:0/4250539739 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9a0c066f30 msgr2=0x7f9a0c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.048+0000 7f9a265ff700 1 --2- 172.21.15.27:0/4250539739 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9a0c066f30 0x7f9a0c0693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9a2015d700 tx=0x7f9a10007c60 comp rx=0 tx=0).stop 2024-01-11T09:45:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.048+0000 7f9a265ff700 1 -- 172.21.15.27:0/4250539739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a201575e0 msgr2=0x7f9a2015e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.048+0000 7f9a265ff700 1 --2- 172.21.15.27:0/4250539739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a201575e0 0x7f9a2015e040 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f9a20066180 tx=0x7f9a18019000 comp rx=0 tx=0).stop 2024-01-11T09:45:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.048+0000 7f9a265ff700 1 -- 172.21.15.27:0/4250539739 shutdown_connections 2024-01-11T09:45:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.048+0000 7f9a265ff700 1 --2- 172.21.15.27:0/4250539739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a201575e0 0x7f9a2015e040 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.048+0000 7f9a265ff700 1 --2- 172.21.15.27:0/4250539739 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9a0c066f30 0x7f9a0c0693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.048+0000 7f9a265ff700 1 --2- 172.21.15.27:0/4250539739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a2006b610 0x7f9a20146c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.048+0000 7f9a265ff700 1 --2- 172.21.15.27:0/4250539739 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9a200b19d0 0x7f9a2015db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.048+0000 7f9a265ff700 1 -- 172.21.15.27:0/4250539739 >> 172.21.15.27:0/4250539739 conn(0x7f9a200680e0 msgr2=0x7f9a200b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.048+0000 7f9a265ff700 1 -- 172.21.15.27:0/4250539739 shutdown_connections 2024-01-11T09:45:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.048+0000 7f9a265ff700 1 -- 172.21.15.27:0/4250539739 wait complete. 2024-01-11T09:45:25.058 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-11T09:45:25.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.271+0000 7f799c9ff700 1 Processor -- start 2024-01-11T09:45:25.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.271+0000 7f799c9ff700 1 -- start start 2024-01-11T09:45:25.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.273+0000 7f799c9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f799813ff80 0x7f799815a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:25.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.273+0000 7f799c9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f799815a560 0x7f79980c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:25.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.273+0000 7f799c9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f799815a9c0 con 0x7f799813fbb0 2024-01-11T09:45:25.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.273+0000 7f799c9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79980c6b30 con 0x7f799815a560 2024-01-11T09:45:25.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.273+0000 7f799c9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79980c6800 con 0x7f799813ff80 2024-01-11T09:45:25.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.273+0000 7f7995d9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f799813ff80 0x7f799815a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:25.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.273+0000 7f7996d9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f799815a560 0x7f79980c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:25.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.273+0000 7f799659c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f799813fbb0 0x7f79980ca7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:53180/0 (socket says 172.21.15.27:53180) 2024-01-11T09:45:25.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.273+0000 7f799659c700 1 -- 172.21.15.27:0/2761255662 learned_addr learned my addr 172.21.15.27:0/2761255662 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:25.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.274+0000 7f7995d9b700 1 -- 172.21.15.27:0/2761255662 >> v2:172.21.15.77:3300/0 conn(0x7f799815a560 msgr2=0x7f79980c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:25.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.274+0000 7f799559a700 1 -- 172.21.15.27:0/2761255662 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895644160 0 0) 0x7f7988001160 con 0x7f799813fbb0 2024-01-11T09:45:25.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.274+0000 7f7995d9b700 1 --2- 172.21.15.27:0/2761255662 >> v2:172.21.15.77:3300/0 conn(0x7f799815a560 0x7f79980c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:25.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.274+0000 7f7995d9b700 1 -- 172.21.15.27:0/2761255662 >> v1:172.21.15.77:6789/0 conn(0x7f799813fbb0 legacy=0x7f79980ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:25.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.274+0000 7f7995d9b700 1 -- 172.21.15.27:0/2761255662 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79980cb240 con 0x7f799813ff80 2024-01-11T09:45:25.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.274+0000 7f7996d9d700 1 --2- 172.21.15.27:0/2761255662 >> v2:172.21.15.77:3300/0 conn(0x7f799815a560 0x7f79980c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:25.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.274+0000 7f7995d9b700 1 --2- 172.21.15.27:0/2761255662 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f799813ff80 0x7f799815a020 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f798c0098e0 tx=0x7f798c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=680dac7313346538 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:25.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.274+0000 7f799559a700 1 -- 172.21.15.27:0/2761255662 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c00b640 con 0x7f799813ff80 2024-01-11T09:45:25.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.275+0000 7f799559a700 1 -- 172.21.15.27:0/2761255662 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798c00b7a0 con 0x7f799813ff80 2024-01-11T09:45:25.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.275+0000 7f799559a700 1 -- 172.21.15.27:0/2761255662 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c00bb40 con 0x7f799813ff80 2024-01-11T09:45:25.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.275+0000 7f799c9ff700 1 -- 172.21.15.27:0/2761255662 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f799813ff80 msgr2=0x7f799815a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:25.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.275+0000 7f799c9ff700 1 --2- 172.21.15.27:0/2761255662 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f799813ff80 0x7f799815a020 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f798c0098e0 tx=0x7f798c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:25.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.275+0000 7f799c9ff700 1 -- 172.21.15.27:0/2761255662 shutdown_connections 2024-01-11T09:45:25.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.275+0000 7f799c9ff700 1 --2- 172.21.15.27:0/2761255662 >> v2:172.21.15.77:3300/0 conn(0x7f799815a560 0x7f79980c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:25.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.275+0000 7f799c9ff700 1 --2- 172.21.15.27:0/2761255662 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f799813ff80 0x7f799815a020 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:25.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.275+0000 7f799c9ff700 1 -- 172.21.15.27:0/2761255662 >> 172.21.15.27:0/2761255662 conn(0x7f79980680a0 msgr2=0x7f79980cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:25.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.275+0000 7f799c9ff700 1 -- 172.21.15.27:0/2761255662 shutdown_connections 2024-01-11T09:45:25.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.275+0000 7f799c9ff700 1 -- 172.21.15.27:0/2761255662 wait complete. 2024-01-11T09:45:25.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.277+0000 7f799c9ff700 1 Processor -- start 2024-01-11T09:45:25.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.277+0000 7f799c9ff700 1 -- start start 2024-01-11T09:45:25.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.278+0000 7f799c9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f799813fbb0 0x7f7998155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:25.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.278+0000 7f799c9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f799813ff80 0x7f79980c3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:25.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.278+0000 7f799c9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f799815a560 0x7f7998169ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:25.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.278+0000 7f799c9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79980c5950 con 0x7f799813fbb0 2024-01-11T09:45:25.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.278+0000 7f799c9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799815d160 con 0x7f799813ff80 2024-01-11T09:45:25.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.278+0000 7f799c9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79981563d0 con 0x7f799815a560 2024-01-11T09:45:25.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.278+0000 7f799659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f799813fbb0 0x7f7998155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:25.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.278+0000 7f799659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f799813fbb0 0x7f7998155e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51780/0 (socket says 172.21.15.27:51780) 2024-01-11T09:45:25.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.278+0000 7f799659c700 1 -- 172.21.15.27:0/3419473748 learned_addr learned my addr 172.21.15.27:0/3419473748 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:25.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.278+0000 7f7996d9d700 1 --2- 172.21.15.27:0/3419473748 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f799815a560 0x7f7998169ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:25.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.278+0000 7f7995d9b700 1 --2- 172.21.15.27:0/3419473748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f799813ff80 0x7f79980c3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:25.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.278+0000 7f799659c700 1 -- 172.21.15.27:0/3419473748 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f799815a560 msgr2=0x7f7998169ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:25.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.278+0000 7f799659c700 1 --2- 172.21.15.27:0/3419473748 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f799815a560 0x7f7998169ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:25.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.278+0000 7f799659c700 1 -- 172.21.15.27:0/3419473748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f799813ff80 msgr2=0x7f79980c3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:25.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.278+0000 7f799659c700 1 --2- 172.21.15.27:0/3419473748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f799813ff80 0x7f79980c3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:25.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.278+0000 7f799659c700 1 -- 172.21.15.27:0/3419473748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7990009a20 con 0x7f799813fbb0 2024-01-11T09:45:25.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.278+0000 7f7995d9b700 1 --2- 172.21.15.27:0/3419473748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f799813ff80 0x7f79980c3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:25.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.279+0000 7f799659c700 1 --2- 172.21.15.27:0/3419473748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f799813fbb0 0x7f7998155e90 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f798800c810 tx=0x7f7988007c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:25.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.279+0000 7f79867fc700 1 -- 172.21.15.27:0/3419473748 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7988005b80 con 0x7f799813fbb0 2024-01-11T09:45:25.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.279+0000 7f79867fc700 1 -- 172.21.15.27:0/3419473748 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7988005ce0 con 0x7f799813fbb0 2024-01-11T09:45:25.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.279+0000 7f79867fc700 1 -- 172.21.15.27:0/3419473748 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7988007340 con 0x7f799813fbb0 2024-01-11T09:45:25.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.279+0000 7f799c9ff700 1 -- 172.21.15.27:0/3419473748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f798c009420 con 0x7f799813fbb0 2024-01-11T09:45:25.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.279+0000 7f799c9ff700 1 -- 172.21.15.27:0/3419473748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79980c6100 con 0x7f799813fbb0 2024-01-11T09:45:25.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.280+0000 7f799c9ff700 1 -- 172.21.15.27:0/3419473748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f799804c3c0 con 0x7f799813fbb0 2024-01-11T09:45:25.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.281+0000 7f79867fc700 1 -- 172.21.15.27:0/3419473748 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7988007730 con 0x7f799813fbb0 2024-01-11T09:45:25.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.282+0000 7f79867fc700 1 --2- 172.21.15.27:0/3419473748 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7978066f30 0x7f79780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:25.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.282+0000 7f79867fc700 1 -- 172.21.15.27:0/3419473748 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f7988083fb0 con 0x7f799813fbb0 2024-01-11T09:45:25.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.283+0000 7f7995d9b700 1 --2- 172.21.15.27:0/3419473748 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7978066f30 0x7f79780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:25.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.283+0000 7f7995d9b700 1 --2- 172.21.15.27:0/3419473748 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7978066f30 0x7f79780693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f798c0093f0 tx=0x7f798c017000 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:25.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.286+0000 7f79867fc700 1 -- 172.21.15.27:0/3419473748 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7988054c20 con 0x7f799813fbb0 2024-01-11T09:45:25.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:25.533+0000 7f799c9ff700 1 -- 172.21.15.27:0/3419473748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f799815a9c0 con 0x7f799813fbb0 2024-01-11T09:45:26.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.145+0000 7f79867fc700 1 -- 172.21.15.27:0/3419473748 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v110) v1 ==== 118+0+0 (secure 0 0 0) 0x7f799815a9c0 con 0x7f799813fbb0 2024-01-11T09:45:26.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.148+0000 7f799c9ff700 1 -- 172.21.15.27:0/3419473748 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7978066f30 msgr2=0x7f79780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:26.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.148+0000 7f799c9ff700 1 --2- 172.21.15.27:0/3419473748 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7978066f30 0x7f79780693e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f798c0093f0 tx=0x7f798c017000 comp rx=0 tx=0).stop 2024-01-11T09:45:26.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.148+0000 7f799c9ff700 1 -- 172.21.15.27:0/3419473748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f799813fbb0 msgr2=0x7f7998155e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:26.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.148+0000 7f799c9ff700 1 --2- 172.21.15.27:0/3419473748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f799813fbb0 0x7f7998155e90 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f798800c810 tx=0x7f7988007c40 comp rx=0 tx=0).stop 2024-01-11T09:45:26.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.148+0000 7f799c9ff700 1 -- 172.21.15.27:0/3419473748 shutdown_connections 2024-01-11T09:45:26.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.148+0000 7f799c9ff700 1 --2- 172.21.15.27:0/3419473748 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f799815a560 0x7f7998169ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:26.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.148+0000 7f799c9ff700 1 --2- 172.21.15.27:0/3419473748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f799813fbb0 0x7f7998155e90 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:26.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.148+0000 7f799c9ff700 1 --2- 172.21.15.27:0/3419473748 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7978066f30 0x7f79780693e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:26.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.148+0000 7f799c9ff700 1 --2- 172.21.15.27:0/3419473748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f799813ff80 0x7f79980c3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:26.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.148+0000 7f799c9ff700 1 -- 172.21.15.27:0/3419473748 >> 172.21.15.27:0/3419473748 conn(0x7f79980680a0 msgr2=0x7f79980c0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:26.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.148+0000 7f799c9ff700 1 -- 172.21.15.27:0/3419473748 shutdown_connections 2024-01-11T09:45:26.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.148+0000 7f799c9ff700 1 -- 172.21.15.27:0/3419473748 wait complete. 2024-01-11T09:45:26.161 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-11T09:45:26.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.377+0000 7fe489bff700 1 Processor -- start 2024-01-11T09:45:26.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.378+0000 7fe489bff700 1 -- start start 2024-01-11T09:45:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.379+0000 7fe489bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe48415a1b0 0x7fe48415a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.379+0000 7fe489bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4840ccd70 0x7fe4840cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.379+0000 7fe489bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4840cf850 con 0x7fe4840685c0 2024-01-11T09:45:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.379+0000 7fe489bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4840cec40 con 0x7fe48415a1b0 2024-01-11T09:45:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.379+0000 7fe489bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4840cef20 con 0x7fe4840ccd70 2024-01-11T09:45:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.380+0000 7fe482ffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4840ccd70 0x7fe4840cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.380+0000 7fe483fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe4840685c0 0x7fe484068a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:53182/0 (socket says 172.21.15.27:53182) 2024-01-11T09:45:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.380+0000 7fe483fff700 1 -- 172.21.15.27:0/1777208492 learned_addr learned my addr 172.21.15.27:0/1777208492 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.380+0000 7fe4837fe700 1 --2- 172.21.15.27:0/1777208492 >> v2:172.21.15.77:3300/0 conn(0x7fe48415a1b0 0x7fe48415a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.380+0000 7fe4827fc700 1 -- 172.21.15.27:0/1777208492 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630574314 0 0) 0x7fe4840cf850 con 0x7fe4840685c0 2024-01-11T09:45:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.380+0000 7fe4827fc700 1 -- 172.21.15.27:0/1777208492 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe46c003570 con 0x7fe4840685c0 2024-01-11T09:45:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.380+0000 7fe482ffd700 1 -- 172.21.15.27:0/1777208492 >> v2:172.21.15.77:3300/0 conn(0x7fe48415a1b0 msgr2=0x7fe48415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.380+0000 7fe482ffd700 1 --2- 172.21.15.27:0/1777208492 >> v2:172.21.15.77:3300/0 conn(0x7fe48415a1b0 0x7fe48415a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.380+0000 7fe482ffd700 1 -- 172.21.15.27:0/1777208492 >> v1:172.21.15.77:6789/0 conn(0x7fe4840685c0 legacy=0x7fe484068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.380+0000 7fe482ffd700 1 -- 172.21.15.27:0/1777208492 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4840cf200 con 0x7fe4840ccd70 2024-01-11T09:45:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.380+0000 7fe4837fe700 1 --2- 172.21.15.27:0/1777208492 >> v2:172.21.15.77:3300/0 conn(0x7fe48415a1b0 0x7fe48415a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.381+0000 7fe482ffd700 1 --2- 172.21.15.27:0/1777208492 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4840ccd70 0x7fe4840cd1e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe4780098e0 tx=0x7fe4780091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61258d31cc1cd0b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.381+0000 7fe4827fc700 1 -- 172.21.15.27:0/1777208492 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe478011630 con 0x7fe4840ccd70 2024-01-11T09:45:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.381+0000 7fe4827fc700 1 -- 172.21.15.27:0/1777208492 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe478011790 con 0x7fe4840ccd70 2024-01-11T09:45:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.381+0000 7fe4827fc700 1 -- 172.21.15.27:0/1777208492 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe478011b30 con 0x7fe4840ccd70 2024-01-11T09:45:26.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.382+0000 7fe489bff700 1 -- 172.21.15.27:0/1777208492 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4840ccd70 msgr2=0x7fe4840cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:26.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.382+0000 7fe489bff700 1 --2- 172.21.15.27:0/1777208492 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4840ccd70 0x7fe4840cd1e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe4780098e0 tx=0x7fe4780091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:26.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.382+0000 7fe489bff700 1 -- 172.21.15.27:0/1777208492 shutdown_connections 2024-01-11T09:45:26.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.382+0000 7fe489bff700 1 --2- 172.21.15.27:0/1777208492 >> v2:172.21.15.77:3300/0 conn(0x7fe48415a1b0 0x7fe48415a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:26.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.382+0000 7fe489bff700 1 --2- 172.21.15.27:0/1777208492 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4840ccd70 0x7fe4840cd1e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:26.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.382+0000 7fe489bff700 1 -- 172.21.15.27:0/1777208492 >> 172.21.15.27:0/1777208492 conn(0x7fe48406ed40 msgr2=0x7fe48406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:26.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.382+0000 7fe489bff700 1 -- 172.21.15.27:0/1777208492 shutdown_connections 2024-01-11T09:45:26.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.382+0000 7fe489bff700 1 -- 172.21.15.27:0/1777208492 wait complete. 2024-01-11T09:45:26.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.384+0000 7fe489bff700 1 Processor -- start 2024-01-11T09:45:26.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.384+0000 7fe489bff700 1 -- start start 2024-01-11T09:45:26.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.385+0000 7fe489bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4840685c0 0x7fe4840af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:26.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.386+0000 7fe489bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4840ccd70 0x7fe4840a6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:26.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.386+0000 7fe489bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe48415a1b0 0x7fe4840a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:26.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.386+0000 7fe489bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4840c8cd0 con 0x7fe48415a1b0 2024-01-11T09:45:26.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.386+0000 7fe489bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4840b5d90 con 0x7fe4840685c0 2024-01-11T09:45:26.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.386+0000 7fe489bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4840c83a0 con 0x7fe4840ccd70 2024-01-11T09:45:26.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.386+0000 7fe483fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe48415a1b0 0x7fe4840a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:26.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.386+0000 7fe483fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe48415a1b0 0x7fe4840a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51784/0 (socket says 172.21.15.27:51784) 2024-01-11T09:45:26.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.386+0000 7fe483fff700 1 -- 172.21.15.27:0/1871165434 learned_addr learned my addr 172.21.15.27:0/1871165434 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:26.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.386+0000 7fe4837fe700 1 --2- 172.21.15.27:0/1871165434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4840685c0 0x7fe4840af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:26.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.386+0000 7fe482ffd700 1 --2- 172.21.15.27:0/1871165434 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4840ccd70 0x7fe4840a6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:26.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.386+0000 7fe483fff700 1 -- 172.21.15.27:0/1871165434 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4840ccd70 msgr2=0x7fe4840a6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:26.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.386+0000 7fe483fff700 1 --2- 172.21.15.27:0/1871165434 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4840ccd70 0x7fe4840a6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:26.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.386+0000 7fe483fff700 1 -- 172.21.15.27:0/1871165434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4840685c0 msgr2=0x7fe4840af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:26.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.386+0000 7fe483fff700 1 --2- 172.21.15.27:0/1871165434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4840685c0 0x7fe4840af6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:26.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.386+0000 7fe483fff700 1 -- 172.21.15.27:0/1871165434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe474009420 con 0x7fe48415a1b0 2024-01-11T09:45:26.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.386+0000 7fe482ffd700 1 --2- 172.21.15.27:0/1871165434 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4840ccd70 0x7fe4840a6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:26.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.387+0000 7fe483fff700 1 --2- 172.21.15.27:0/1871165434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe48415a1b0 0x7fe4840a7450 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fe47c001460 tx=0x7fe47c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:26.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.387+0000 7fe46b7fe700 1 -- 172.21.15.27:0/1871165434 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c00a8a0 con 0x7fe48415a1b0 2024-01-11T09:45:26.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.387+0000 7fe46b7fe700 1 -- 172.21.15.27:0/1871165434 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47c00aa00 con 0x7fe48415a1b0 2024-01-11T09:45:26.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.387+0000 7fe489bff700 1 -- 172.21.15.27:0/1871165434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe478009420 con 0x7fe48415a1b0 2024-01-11T09:45:26.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.387+0000 7fe46b7fe700 1 -- 172.21.15.27:0/1871165434 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c00ad80 con 0x7fe48415a1b0 2024-01-11T09:45:26.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.387+0000 7fe489bff700 1 -- 172.21.15.27:0/1871165434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4840c8960 con 0x7fe48415a1b0 2024-01-11T09:45:26.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.388+0000 7fe489bff700 1 -- 172.21.15.27:0/1871165434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe48404c3c0 con 0x7fe48415a1b0 2024-01-11T09:45:26.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.389+0000 7fe46b7fe700 1 -- 172.21.15.27:0/1871165434 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe47c007300 con 0x7fe48415a1b0 2024-01-11T09:45:26.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.390+0000 7fe46b7fe700 1 --2- 172.21.15.27:0/1871165434 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe470066f30 0x7fe4700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:26.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.390+0000 7fe46b7fe700 1 -- 172.21.15.27:0/1871165434 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5817+0+0 (secure 0 0 0) 0x7fe47c084a30 con 0x7fe48415a1b0 2024-01-11T09:45:26.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.390+0000 7fe4837fe700 1 --2- 172.21.15.27:0/1871165434 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe470066f30 0x7fe4700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:26.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.391+0000 7fe4837fe700 1 --2- 172.21.15.27:0/1871165434 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe470066f30 0x7fe4700693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe474009f40 tx=0x7fe4740091a0 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:26.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.395+0000 7fe46b7fe700 1 -- 172.21.15.27:0/1871165434 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe47c051c10 con 0x7fe48415a1b0 2024-01-11T09:45:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:26.644+0000 7fe489bff700 1 -- 172.21.15.27:0/1871165434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe484159970 con 0x7fe48415a1b0 2024-01-11T09:45:27.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.153+0000 7fe46b7fe700 1 -- 172.21.15.27:0/1871165434 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v111) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe47c055230 con 0x7fe48415a1b0 2024-01-11T09:45:27.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.155+0000 7fe489bff700 1 -- 172.21.15.27:0/1871165434 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe470066f30 msgr2=0x7fe4700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:27.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.155+0000 7fe489bff700 1 --2- 172.21.15.27:0/1871165434 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe470066f30 0x7fe4700693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe474009f40 tx=0x7fe4740091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.155+0000 7fe489bff700 1 -- 172.21.15.27:0/1871165434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe48415a1b0 msgr2=0x7fe4840a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:27.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.155+0000 7fe489bff700 1 --2- 172.21.15.27:0/1871165434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe48415a1b0 0x7fe4840a7450 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fe47c001460 tx=0x7fe47c001a40 comp rx=0 tx=0).stop 2024-01-11T09:45:27.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.155+0000 7fe489bff700 1 -- 172.21.15.27:0/1871165434 shutdown_connections 2024-01-11T09:45:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.155+0000 7fe489bff700 1 --2- 172.21.15.27:0/1871165434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe48415a1b0 0x7fe4840a7450 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.155+0000 7fe489bff700 1 --2- 172.21.15.27:0/1871165434 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe470066f30 0x7fe4700693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.155+0000 7fe489bff700 1 --2- 172.21.15.27:0/1871165434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4840685c0 0x7fe4840af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.155+0000 7fe489bff700 1 --2- 172.21.15.27:0/1871165434 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe4840ccd70 0x7fe4840a6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.155+0000 7fe489bff700 1 -- 172.21.15.27:0/1871165434 >> 172.21.15.27:0/1871165434 conn(0x7fe48406ed40 msgr2=0x7fe484158bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.155+0000 7fe489bff700 1 -- 172.21.15.27:0/1871165434 shutdown_connections 2024-01-11T09:45:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.155+0000 7fe489bff700 1 -- 172.21.15.27:0/1871165434 wait complete. 2024-01-11T09:45:27.167 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:45:27.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.375+0000 7f657c5ff700 1 Processor -- start 2024-01-11T09:45:27.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.375+0000 7f657c5ff700 1 -- start start 2024-01-11T09:45:27.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.376+0000 7f657c5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f65740ca3e0 0x7f65740cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:27.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.376+0000 7f657c5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657415d640 con 0x7f6574159e80 2024-01-11T09:45:27.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.376+0000 7f657c5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657415a5d0 con 0x7f65740ca7b0 2024-01-11T09:45:27.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.376+0000 7f657c5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657415ae70 con 0x7f65740ca3e0 2024-01-11T09:45:27.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.377+0000 7f657ab9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6574159e80 0x7f657415dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55950/0 (socket says 172.21.15.27:55950) 2024-01-11T09:45:27.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.377+0000 7f657ab9c700 1 -- 172.21.15.27:0/1230305395 learned_addr learned my addr 172.21.15.27:0/1230305395 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:27.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.377+0000 7f657a39b700 1 --2- 172.21.15.27:0/1230305395 >> v2:172.21.15.77:3300/0 conn(0x7f65740ca3e0 0x7f65740cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:27.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.377+0000 7f6579399700 1 -- 172.21.15.27:0/1230305395 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1129601507 0 0) 0x7f657415d640 con 0x7f6574159e80 2024-01-11T09:45:27.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.377+0000 7f6579399700 1 -- 172.21.15.27:0/1230305395 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6560003570 con 0x7f6574159e80 2024-01-11T09:45:27.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.377+0000 7f6579399700 1 -- 172.21.15.27:0/1230305395 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 593411265 0 0) 0x7f6560003570 con 0x7f6574159e80 2024-01-11T09:45:27.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.378+0000 7f6579399700 1 -- 172.21.15.27:0/1230305395 >> v2:172.21.15.77:3300/0 conn(0x7f65740ca3e0 msgr2=0x7f65740cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:27.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.378+0000 7f6579399700 1 --2- 172.21.15.27:0/1230305395 >> v2:172.21.15.77:3300/0 conn(0x7f65740ca3e0 0x7f65740cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.378+0000 7f6579399700 1 -- 172.21.15.27:0/1230305395 >> v1:172.21.15.77:6789/0 conn(0x7f65740ca7b0 legacy=0x7f65740c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:27.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.378+0000 7f6579399700 1 -- 172.21.15.27:0/1230305395 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657415a8b0 con 0x7f6574159e80 2024-01-11T09:45:27.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.378+0000 7f6579399700 1 -- 172.21.15.27:0/1230305395 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f656c0017f0 con 0x7f6574159e80 2024-01-11T09:45:27.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.378+0000 7f6579399700 1 -- 172.21.15.27:0/1230305395 >> v1:172.21.15.27:6789/0 conn(0x7f6574159e80 legacy=0x7f657415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:27.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.378+0000 7f6579399700 1 --2- 172.21.15.27:0/1230305395 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6560003a60 0x7f6560003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:27.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.378+0000 7f6579399700 1 --2- 172.21.15.27:0/1230305395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6560004810 0x7f6560004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.378+0000 7f6579399700 1 --2- 172.21.15.27:0/1230305395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65600051c0 0x7f656000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.378+0000 7f6579399700 1 -- 172.21.15.27:0/1230305395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657415ae70 con 0x7f65600051c0 2024-01-11T09:45:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.378+0000 7f6579399700 1 -- 172.21.15.27:0/1230305395 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656000c0e0 con 0x7f6560004810 2024-01-11T09:45:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.378+0000 7f6579399700 1 -- 172.21.15.27:0/1230305395 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656000c2a0 con 0x7f6560003a60 2024-01-11T09:45:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.378+0000 7f657ab9c700 1 --2- 172.21.15.27:0/1230305395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65600051c0 0x7f656000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.378+0000 7f657a39b700 1 --2- 172.21.15.27:0/1230305395 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6560003a60 0x7f6560003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.378+0000 7f6579b9a700 1 --2- 172.21.15.27:0/1230305395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6560004810 0x7f6560004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.379+0000 7f657ab9c700 1 -- 172.21.15.27:0/1230305395 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6560003a60 msgr2=0x7f6560003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.379+0000 7f657ab9c700 1 --2- 172.21.15.27:0/1230305395 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6560003a60 0x7f6560003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.379+0000 7f657ab9c700 1 -- 172.21.15.27:0/1230305395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6560004810 msgr2=0x7f6560004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.379+0000 7f657ab9c700 1 --2- 172.21.15.27:0/1230305395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6560004810 0x7f6560004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.379+0000 7f657ab9c700 1 -- 172.21.15.27:0/1230305395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6560003570 con 0x7f65600051c0 2024-01-11T09:45:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.379+0000 7f657ab9c700 1 -- 172.21.15.27:0/1230305395 reap_dead start 2024-01-11T09:45:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.379+0000 7f657ab9c700 1 --2- 172.21.15.27:0/1230305395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65600051c0 0x7f656000ba60 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f656c00a8d0 tx=0x7f656c002170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719ae48e19bc77d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.379+0000 7f6579399700 1 -- 172.21.15.27:0/1230305395 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c0070d0 con 0x7f65600051c0 2024-01-11T09:45:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.379+0000 7f6579399700 1 -- 172.21.15.27:0/1230305395 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656c007270 con 0x7f65600051c0 2024-01-11T09:45:27.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.379+0000 7f6579399700 1 -- 172.21.15.27:0/1230305395 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c007630 con 0x7f65600051c0 2024-01-11T09:45:27.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.380+0000 7f657c5ff700 1 -- 172.21.15.27:0/1230305395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65600051c0 msgr2=0x7f656000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:27.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.380+0000 7f657c5ff700 1 --2- 172.21.15.27:0/1230305395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65600051c0 0x7f656000ba60 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f656c00a8d0 tx=0x7f656c002170 comp rx=0 tx=0).stop 2024-01-11T09:45:27.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.380+0000 7f657c5ff700 1 -- 172.21.15.27:0/1230305395 shutdown_connections 2024-01-11T09:45:27.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.380+0000 7f657c5ff700 1 --2- 172.21.15.27:0/1230305395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65600051c0 0x7f656000ba60 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.380+0000 7f657c5ff700 1 -- 172.21.15.27:0/1230305395 >> 172.21.15.27:0/1230305395 conn(0x7f65740ce3e0 msgr2=0x7f65740ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:27.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.380+0000 7f657c5ff700 1 -- 172.21.15.27:0/1230305395 shutdown_connections 2024-01-11T09:45:27.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.380+0000 7f657c5ff700 1 -- 172.21.15.27:0/1230305395 wait complete. 2024-01-11T09:45:27.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.382+0000 7f657c5ff700 1 Processor -- start 2024-01-11T09:45:27.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.382+0000 7f657c5ff700 1 -- start start 2024-01-11T09:45:27.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.383+0000 7f657c5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65600051c0 0x7f657414ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:27.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.383+0000 7f657c5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65740b11c0 0x7f657414e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:27.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.383+0000 7f657c5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f657414eb50 0x7f657414eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:27.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.383+0000 7f657c5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65740b18c0 con 0x7f65600051c0 2024-01-11T09:45:27.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.383+0000 7f657c5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6574158f40 con 0x7f657414eb50 2024-01-11T09:45:27.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.383+0000 7f657c5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65741593d0 con 0x7f65740b11c0 2024-01-11T09:45:27.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.384+0000 7f657a39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65600051c0 0x7f657414ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:27.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.384+0000 7f657a39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65600051c0 0x7f657414ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51804/0 (socket says 172.21.15.27:51804) 2024-01-11T09:45:27.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.384+0000 7f657a39b700 1 -- 172.21.15.27:0/1736859245 learned_addr learned my addr 172.21.15.27:0/1736859245 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:27.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.384+0000 7f6579b9a700 1 --2- 172.21.15.27:0/1736859245 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65740b11c0 0x7f657414e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:27.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.384+0000 7f657ab9c700 1 --2- 172.21.15.27:0/1736859245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f657414eb50 0x7f657414eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:27.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.384+0000 7f657a39b700 1 -- 172.21.15.27:0/1736859245 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65740b11c0 msgr2=0x7f657414e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:27.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.384+0000 7f657a39b700 1 --2- 172.21.15.27:0/1736859245 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65740b11c0 0x7f657414e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.384+0000 7f657a39b700 1 -- 172.21.15.27:0/1736859245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f657414eb50 msgr2=0x7f657414eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:27.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.384+0000 7f657a39b700 1 --2- 172.21.15.27:0/1736859245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f657414eb50 0x7f657414eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.384+0000 7f657a39b700 1 -- 172.21.15.27:0/1736859245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65740b1630 con 0x7f65600051c0 2024-01-11T09:45:27.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.384+0000 7f657ab9c700 1 --2- 172.21.15.27:0/1736859245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f657414eb50 0x7f657414eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:27.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.384+0000 7f657a39b700 1 --2- 172.21.15.27:0/1736859245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65600051c0 0x7f657414ccb0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f6570011b10 tx=0x7f65700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:27.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.385+0000 7f656a7fc700 1 -- 172.21.15.27:0/1736859245 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657000f940 con 0x7f65600051c0 2024-01-11T09:45:27.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.385+0000 7f656a7fc700 1 -- 172.21.15.27:0/1736859245 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657000faa0 con 0x7f65600051c0 2024-01-11T09:45:27.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.385+0000 7f656a7fc700 1 -- 172.21.15.27:0/1736859245 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657000fe20 con 0x7f65600051c0 2024-01-11T09:45:27.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.385+0000 7f657c5ff700 1 -- 172.21.15.27:0/1736859245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f657414f530 con 0x7f65600051c0 2024-01-11T09:45:27.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.385+0000 7f657c5ff700 1 -- 172.21.15.27:0/1736859245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65741588f0 con 0x7f65600051c0 2024-01-11T09:45:27.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.386+0000 7f657c5ff700 1 -- 172.21.15.27:0/1736859245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6574159cf0 con 0x7f65600051c0 2024-01-11T09:45:27.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.387+0000 7f656a7fc700 1 -- 172.21.15.27:0/1736859245 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f6570014650 con 0x7f65600051c0 2024-01-11T09:45:27.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.388+0000 7f656a7fc700 1 --2- 172.21.15.27:0/1736859245 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6558066f30 0x7f65580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:27.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.388+0000 7f656a7fc700 1 -- 172.21.15.27:0/1736859245 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f6570018030 con 0x7f65600051c0 2024-01-11T09:45:27.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.388+0000 7f6579b9a700 1 --2- 172.21.15.27:0/1736859245 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6558066f30 0x7f65580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:27.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.389+0000 7f6579b9a700 1 --2- 172.21.15.27:0/1736859245 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6558066f30 0x7f65580693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6574069690 tx=0x7f6564007d30 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:27.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.392+0000 7f656a7fc700 1 -- 172.21.15.27:0/1736859245 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6570052170 con 0x7f65600051c0 2024-01-11T09:45:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.639+0000 7f657c5ff700 1 -- 172.21.15.27:0/1736859245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65740718d0 con 0x7f65600051c0 2024-01-11T09:45:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.640+0000 7f656a7fc700 1 -- 172.21.15.27:0/1736859245 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+4611 (secure 0 0 0) 0x7f6570022090 con 0x7f65600051c0 2024-01-11T09:45:27.633 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:27.633 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":111,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2103359888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":2103359888},{"type":"v1","addr":"172.21.15.77:6834","nonce":2103359888}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8795,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2650824649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2650824649},{"type":"v1","addr":"172.21.15.77:6835","nonce":2650824649}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106},{"gid":10596,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/2807751836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2807751836},{"type":"v1","addr":"172.21.15.27:6834","nonce":2807751836}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:45:23.637614+0000","modified":"2024-01-11T09:45:27.146191+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10599":{"gid":10599,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1143960902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1143960902},{"type":"v1","addr":"172.21.15.27:6835","nonce":1143960902}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2024-01-11T09:45:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.642+0000 7f657c5ff700 1 -- 172.21.15.27:0/1736859245 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6558066f30 msgr2=0x7f65580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.642+0000 7f657c5ff700 1 --2- 172.21.15.27:0/1736859245 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6558066f30 0x7f65580693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6574069690 tx=0x7f6564007d30 comp rx=0 tx=0).stop 2024-01-11T09:45:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.642+0000 7f657c5ff700 1 -- 172.21.15.27:0/1736859245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65600051c0 msgr2=0x7f657414ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.642+0000 7f657c5ff700 1 --2- 172.21.15.27:0/1736859245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65600051c0 0x7f657414ccb0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f6570011b10 tx=0x7f65700091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.642+0000 7f657c5ff700 1 -- 172.21.15.27:0/1736859245 shutdown_connections 2024-01-11T09:45:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.642+0000 7f657c5ff700 1 --2- 172.21.15.27:0/1736859245 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6558066f30 0x7f65580693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.642+0000 7f657c5ff700 1 --2- 172.21.15.27:0/1736859245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f657414eb50 0x7f657414eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.642+0000 7f657c5ff700 1 --2- 172.21.15.27:0/1736859245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65600051c0 0x7f657414ccb0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.642+0000 7f657c5ff700 1 --2- 172.21.15.27:0/1736859245 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f65740b11c0 0x7f657414e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.643+0000 7f657c5ff700 1 -- 172.21.15.27:0/1736859245 >> 172.21.15.27:0/1736859245 conn(0x7f65740ce3e0 msgr2=0x7f6574157360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.643+0000 7f657c5ff700 1 -- 172.21.15.27:0/1736859245 shutdown_connections 2024-01-11T09:45:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.643+0000 7f657c5ff700 1 -- 172.21.15.27:0/1736859245 wait complete. 2024-01-11T09:45:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 111 2024-01-11T09:45:27.653 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:45:27.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.860+0000 7f565ebff700 1 Processor -- start 2024-01-11T09:45:27.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.860+0000 7f565ebff700 1 -- start start 2024-01-11T09:45:27.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.861+0000 7f565ebff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56581575e0 0x7f5658158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:27.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.861+0000 7f565ebff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f56580b19d0 0x7f5658159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:27.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.861+0000 7f565ebff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5658158b60 con 0x7f565806b610 2024-01-11T09:45:27.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.861+0000 7f565ebff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56580cb5e0 con 0x7f56581575e0 2024-01-11T09:45:27.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.861+0000 7f565ebff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56581599b0 con 0x7f56580b19d0 2024-01-11T09:45:27.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.862+0000 7f565c99b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56581575e0 0x7f5658158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:27.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.862+0000 7f565c99b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56581575e0 0x7f5658158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51808/0 (socket says 172.21.15.27:51808) 2024-01-11T09:45:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.862+0000 7f565c99b700 1 -- 172.21.15.27:0/3860284443 learned_addr learned my addr 172.21.15.27:0/3860284443 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.862+0000 7f565d19c700 1 --2- 172.21.15.27:0/3860284443 >> v2:172.21.15.77:3300/0 conn(0x7f56580b19d0 0x7f5658159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.862+0000 7f565c99b700 1 -- 172.21.15.27:0/3860284443 >> v2:172.21.15.77:3300/0 conn(0x7f56580b19d0 msgr2=0x7f5658159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.862+0000 7f565c99b700 1 --2- 172.21.15.27:0/3860284443 >> v2:172.21.15.77:3300/0 conn(0x7f56580b19d0 0x7f5658159e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.862+0000 7f565c99b700 1 -- 172.21.15.27:0/3860284443 >> v1:172.21.15.77:6789/0 conn(0x7f565806b610 legacy=0x7f56580b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.862+0000 7f565c99b700 1 -- 172.21.15.27:0/3860284443 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56580c0f90 con 0x7f56581575e0 2024-01-11T09:45:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.862+0000 7f565c99b700 1 --2- 172.21.15.27:0/3860284443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56581575e0 0x7f5658158620 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f565806d070 tx=0x7f564c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70367f3653113ed server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.863+0000 7f56577fe700 1 -- 172.21.15.27:0/3860284443 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c00b860 con 0x7f56581575e0 2024-01-11T09:45:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.863+0000 7f56577fe700 1 -- 172.21.15.27:0/3860284443 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564c00b9c0 con 0x7f56581575e0 2024-01-11T09:45:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.863+0000 7f56577fe700 1 -- 172.21.15.27:0/3860284443 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c00bd60 con 0x7f56581575e0 2024-01-11T09:45:27.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.863+0000 7f565ebff700 1 -- 172.21.15.27:0/3860284443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56581575e0 msgr2=0x7f5658158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:27.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.863+0000 7f565ebff700 1 --2- 172.21.15.27:0/3860284443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56581575e0 0x7f5658158620 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f565806d070 tx=0x7f564c009250 comp rx=0 tx=0).stop 2024-01-11T09:45:27.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.863+0000 7f565ebff700 1 -- 172.21.15.27:0/3860284443 shutdown_connections 2024-01-11T09:45:27.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.863+0000 7f565ebff700 1 --2- 172.21.15.27:0/3860284443 >> v2:172.21.15.77:3300/0 conn(0x7f56580b19d0 0x7f5658159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.863+0000 7f565ebff700 1 --2- 172.21.15.27:0/3860284443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56581575e0 0x7f5658158620 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.863+0000 7f565ebff700 1 -- 172.21.15.27:0/3860284443 >> 172.21.15.27:0/3860284443 conn(0x7f56580680e0 msgr2=0x7f5658069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:27.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.863+0000 7f565ebff700 1 -- 172.21.15.27:0/3860284443 shutdown_connections 2024-01-11T09:45:27.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.864+0000 7f565ebff700 1 -- 172.21.15.27:0/3860284443 wait complete. 2024-01-11T09:45:27.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.865+0000 7f565ebff700 1 Processor -- start 2024-01-11T09:45:27.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.865+0000 7f565ebff700 1 -- start start 2024-01-11T09:45:27.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.866+0000 7f565ebff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565806b610 0x7f56580c5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:27.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.866+0000 7f565ebff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56580b19d0 0x7f56581522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:27.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.866+0000 7f565ebff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56581575e0 0x7f565816a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.866+0000 7f565ebff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5658152810 con 0x7f56581575e0 2024-01-11T09:45:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.866+0000 7f565ebff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56580c69d0 con 0x7f565806b610 2024-01-11T09:45:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.866+0000 7f565ebff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565816ab30 con 0x7f56580b19d0 2024-01-11T09:45:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.866+0000 7f565d19c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56581575e0 0x7f565816a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.866+0000 7f565d19c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56581575e0 0x7f565816a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51810/0 (socket says 172.21.15.27:51810) 2024-01-11T09:45:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.866+0000 7f565d19c700 1 -- 172.21.15.27:0/2684838854 learned_addr learned my addr 172.21.15.27:0/2684838854 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.866+0000 7f565c99b700 1 --2- 172.21.15.27:0/2684838854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565806b610 0x7f56580c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.866+0000 7f5657fff700 1 --2- 172.21.15.27:0/2684838854 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56580b19d0 0x7f56581522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.867+0000 7f565d19c700 1 -- 172.21.15.27:0/2684838854 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56580b19d0 msgr2=0x7f56581522d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.867+0000 7f565d19c700 1 --2- 172.21.15.27:0/2684838854 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56580b19d0 0x7f56581522d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.867+0000 7f565d19c700 1 -- 172.21.15.27:0/2684838854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565806b610 msgr2=0x7f56580c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.867+0000 7f565d19c700 1 --2- 172.21.15.27:0/2684838854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565806b610 0x7f56580c5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.867+0000 7f565d19c700 1 -- 172.21.15.27:0/2684838854 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564c0094f0 con 0x7f56581575e0 2024-01-11T09:45:27.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.867+0000 7f565d19c700 1 --2- 172.21.15.27:0/2684838854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56581575e0 0x7f565816a5f0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f5650009fd0 tx=0x7f5650009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:27.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.867+0000 7f5654ff9700 1 -- 172.21.15.27:0/2684838854 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565000ebf0 con 0x7f56581575e0 2024-01-11T09:45:27.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.867+0000 7f5654ff9700 1 -- 172.21.15.27:0/2684838854 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565000ed50 con 0x7f56581575e0 2024-01-11T09:45:27.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.867+0000 7f565ebff700 1 -- 172.21.15.27:0/2684838854 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56580c5dc0 con 0x7f56581575e0 2024-01-11T09:45:27.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.867+0000 7f5654ff9700 1 -- 172.21.15.27:0/2684838854 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5650014670 con 0x7f56581575e0 2024-01-11T09:45:27.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.867+0000 7f565ebff700 1 -- 172.21.15.27:0/2684838854 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56580c6380 con 0x7f56581575e0 2024-01-11T09:45:27.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.869+0000 7f565ebff700 1 -- 172.21.15.27:0/2684838854 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56580c6b80 con 0x7f56581575e0 2024-01-11T09:45:27.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.870+0000 7f5654ff9700 1 -- 172.21.15.27:0/2684838854 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f56500149e0 con 0x7f56581575e0 2024-01-11T09:45:27.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.870+0000 7f5654ff9700 1 --2- 172.21.15.27:0/2684838854 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f563c066f30 0x7f563c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:27.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.871+0000 7f5654ff9700 1 -- 172.21.15.27:0/2684838854 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f5650084b90 con 0x7f56581575e0 2024-01-11T09:45:27.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.871+0000 7f565c99b700 1 --2- 172.21.15.27:0/2684838854 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f563c066f30 0x7f563c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:27.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.871+0000 7f565c99b700 1 --2- 172.21.15.27:0/2684838854 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f563c066f30 0x7f563c0693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f565806d070 tx=0x7f564c017000 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:27.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:27.874+0000 7f5654ff9700 1 -- 172.21.15.27:0/2684838854 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f56580c6b80 con 0x7f56581575e0 2024-01-11T09:45:28.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.081+0000 7f565ebff700 1 -- 172.21.15.27:0/2684838854 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56580c6b80 con 0x7f56581575e0 2024-01-11T09:45:28.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.082+0000 7f5654ff9700 1 -- 172.21.15.27:0/2684838854 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v214) v1 ==== 74+0+15495 (secure 0 0 0) 0x7f56580c6b80 con 0x7f56581575e0 2024-01-11T09:45:28.075 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:28.075 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":214,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:45:25.139360+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:45:21.455328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-01-11T09:45:22.634298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:45:28.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.085+0000 7f565ebff700 1 -- 172.21.15.27:0/2684838854 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f563c066f30 msgr2=0x7f563c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:28.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.085+0000 7f565ebff700 1 --2- 172.21.15.27:0/2684838854 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f563c066f30 0x7f563c0693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f565806d070 tx=0x7f564c017000 comp rx=0 tx=0).stop 2024-01-11T09:45:28.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.085+0000 7f565ebff700 1 -- 172.21.15.27:0/2684838854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56581575e0 msgr2=0x7f565816a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:28.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.085+0000 7f565ebff700 1 --2- 172.21.15.27:0/2684838854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56581575e0 0x7f565816a5f0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f5650009fd0 tx=0x7f5650009900 comp rx=0 tx=0).stop 2024-01-11T09:45:28.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.085+0000 7f565ebff700 1 -- 172.21.15.27:0/2684838854 shutdown_connections 2024-01-11T09:45:28.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.085+0000 7f565ebff700 1 --2- 172.21.15.27:0/2684838854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56581575e0 0x7f565816a5f0 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:28.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.085+0000 7f565ebff700 1 --2- 172.21.15.27:0/2684838854 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f563c066f30 0x7f563c0693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:28.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.085+0000 7f565ebff700 1 --2- 172.21.15.27:0/2684838854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565806b610 0x7f56580c5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:28.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.085+0000 7f565ebff700 1 --2- 172.21.15.27:0/2684838854 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f56580b19d0 0x7f56581522d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:28.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.085+0000 7f565ebff700 1 -- 172.21.15.27:0/2684838854 >> 172.21.15.27:0/2684838854 conn(0x7f56580680e0 msgr2=0x7f5658152aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:28.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.085+0000 7f565ebff700 1 -- 172.21.15.27:0/2684838854 shutdown_connections 2024-01-11T09:45:28.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.085+0000 7f565ebff700 1 -- 172.21.15.27:0/2684838854 wait complete. 2024-01-11T09:45:28.095 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-11T09:45:28.096 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:45:28.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.303+0000 7f47fd7ff700 1 Processor -- start 2024-01-11T09:45:28.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.303+0000 7f47fd7ff700 1 -- start start 2024-01-11T09:45:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.304+0000 7f47fd7ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f47f80c8620 0x7f47f80af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.304+0000 7f47fd7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f47f8069d40 0x7f47f80b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.304+0000 7f47fd7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f8157390 con 0x7f47f80c89f0 2024-01-11T09:45:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.304+0000 7f47fd7ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f80adcd0 con 0x7f47f80c8620 2024-01-11T09:45:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.304+0000 7f47fd7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f80afeb0 con 0x7f47f8069d40 2024-01-11T09:45:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.305+0000 7f47f6ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f47f80c8620 0x7f47f80af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.305+0000 7f47f77fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f47f80c89f0 0x7f47f8159ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55966/0 (socket says 172.21.15.27:55966) 2024-01-11T09:45:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.305+0000 7f47f6ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f47f80c8620 0x7f47f80af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51822/0 (socket says 172.21.15.27:51822) 2024-01-11T09:45:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.305+0000 7f47f77fe700 1 -- 172.21.15.27:0/3895603878 learned_addr learned my addr 172.21.15.27:0/3895603878 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.305+0000 7f47f67fc700 1 --2- 172.21.15.27:0/3895603878 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f47f8069d40 0x7f47f80b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.305+0000 7f47f6ffd700 1 -- 172.21.15.27:0/3895603878 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f47f8069d40 msgr2=0x7f47f80b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:28.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.305+0000 7f47f6ffd700 1 --2- 172.21.15.27:0/3895603878 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f47f8069d40 0x7f47f80b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:28.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.305+0000 7f47f6ffd700 1 -- 172.21.15.27:0/3895603878 >> v1:172.21.15.27:6789/0 conn(0x7f47f80c89f0 legacy=0x7f47f8159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:28.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.305+0000 7f47f6ffd700 1 -- 172.21.15.27:0/3895603878 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f806a200 con 0x7f47f80c8620 2024-01-11T09:45:28.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.305+0000 7f47f6ffd700 1 --2- 172.21.15.27:0/3895603878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f80c8620 0x7f47f80af970 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f47e80098e0 tx=0x7f47e80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7653f3452b9977 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:28.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.306+0000 7f47f5ffb700 1 -- 172.21.15.27:0/3895603878 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e8009e80 con 0x7f47f80c8620 2024-01-11T09:45:28.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.306+0000 7f47f5ffb700 1 -- 172.21.15.27:0/3895603878 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e800b860 con 0x7f47f80c8620 2024-01-11T09:45:28.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.306+0000 7f47f5ffb700 1 -- 172.21.15.27:0/3895603878 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e800bbe0 con 0x7f47f80c8620 2024-01-11T09:45:28.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.306+0000 7f47fd7ff700 1 -- 172.21.15.27:0/3895603878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f80c8620 msgr2=0x7f47f80af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:28.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.306+0000 7f47fd7ff700 1 --2- 172.21.15.27:0/3895603878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f80c8620 0x7f47f80af970 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f47e80098e0 tx=0x7f47e80091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:28.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.306+0000 7f47fd7ff700 1 -- 172.21.15.27:0/3895603878 shutdown_connections 2024-01-11T09:45:28.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.306+0000 7f47fd7ff700 1 --2- 172.21.15.27:0/3895603878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f80c8620 0x7f47f80af970 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:28.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.306+0000 7f47fd7ff700 1 --2- 172.21.15.27:0/3895603878 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f47f8069d40 0x7f47f80b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:28.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.306+0000 7f47fd7ff700 1 -- 172.21.15.27:0/3895603878 >> 172.21.15.27:0/3895603878 conn(0x7f47f80680e0 msgr2=0x7f47f8069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:28.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.306+0000 7f47fd7ff700 1 -- 172.21.15.27:0/3895603878 shutdown_connections 2024-01-11T09:45:28.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.306+0000 7f47fd7ff700 1 -- 172.21.15.27:0/3895603878 wait complete. 2024-01-11T09:45:28.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.308+0000 7f47fd7ff700 1 Processor -- start 2024-01-11T09:45:28.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.309+0000 7f47fd7ff700 1 -- start start 2024-01-11T09:45:28.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.310+0000 7f47fd7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47f8069d40 0x7f47f8138f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:28.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.310+0000 7f47fd7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f47f80c8620 0x7f47f816cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:28.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.310+0000 7f47fd7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f80c89f0 0x7f47f806ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:28.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.310+0000 7f47fd7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f80a9db0 con 0x7f47f80c89f0 2024-01-11T09:45:28.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.310+0000 7f47fd7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f816d980 con 0x7f47f8069d40 2024-01-11T09:45:28.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.310+0000 7f47fd7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f816dc60 con 0x7f47f80c8620 2024-01-11T09:45:28.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.310+0000 7f47f77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f80c89f0 0x7f47f806ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:28.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.310+0000 7f47f77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f80c89f0 0x7f47f806ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51828/0 (socket says 172.21.15.27:51828) 2024-01-11T09:45:28.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.310+0000 7f47f77fe700 1 -- 172.21.15.27:0/2154192723 learned_addr learned my addr 172.21.15.27:0/2154192723 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.310+0000 7f47f6ffd700 1 --2- 172.21.15.27:0/2154192723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47f8069d40 0x7f47f8138f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.310+0000 7f47f67fc700 1 --2- 172.21.15.27:0/2154192723 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f47f80c8620 0x7f47f816cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.311+0000 7f47f77fe700 1 -- 172.21.15.27:0/2154192723 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f47f80c8620 msgr2=0x7f47f816cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.311+0000 7f47f77fe700 1 --2- 172.21.15.27:0/2154192723 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f47f80c8620 0x7f47f816cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.311+0000 7f47f77fe700 1 -- 172.21.15.27:0/2154192723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47f8069d40 msgr2=0x7f47f8138f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.311+0000 7f47f77fe700 1 --2- 172.21.15.27:0/2154192723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47f8069d40 0x7f47f8138f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.311+0000 7f47f77fe700 1 -- 172.21.15.27:0/2154192723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e8009420 con 0x7f47f80c89f0 2024-01-11T09:45:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.311+0000 7f47f77fe700 1 --2- 172.21.15.27:0/2154192723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f80c89f0 0x7f47f806ec60 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f47f000cbc0 tx=0x7f47f0001990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.311+0000 7f47deffd700 1 -- 172.21.15.27:0/2154192723 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f000a7a0 con 0x7f47f80c89f0 2024-01-11T09:45:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.311+0000 7f47deffd700 1 -- 172.21.15.27:0/2154192723 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f000a900 con 0x7f47f80c89f0 2024-01-11T09:45:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.311+0000 7f47deffd700 1 -- 172.21.15.27:0/2154192723 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f000ac80 con 0x7f47f80c89f0 2024-01-11T09:45:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.311+0000 7f47fd7ff700 1 -- 172.21.15.27:0/2154192723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f80a9a40 con 0x7f47f80c89f0 2024-01-11T09:45:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.311+0000 7f47fd7ff700 1 -- 172.21.15.27:0/2154192723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f8139450 con 0x7f47f80c89f0 2024-01-11T09:45:28.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.313+0000 7f47fd7ff700 1 -- 172.21.15.27:0/2154192723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f80a98a0 con 0x7f47f80c89f0 2024-01-11T09:45:28.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.314+0000 7f47deffd700 1 -- 172.21.15.27:0/2154192723 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f47f0007300 con 0x7f47f80c89f0 2024-01-11T09:45:28.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.315+0000 7f47deffd700 1 --2- 172.21.15.27:0/2154192723 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f47e4066f30 0x7f47e40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:28.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.315+0000 7f47deffd700 1 -- 172.21.15.27:0/2154192723 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f47f0084850 con 0x7f47f80c89f0 2024-01-11T09:45:28.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.315+0000 7f47f6ffd700 1 --2- 172.21.15.27:0/2154192723 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f47e4066f30 0x7f47e40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:28.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.316+0000 7f47f6ffd700 1 --2- 172.21.15.27:0/2154192723 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f47e4066f30 0x7f47e40693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f47e80091e0 tx=0x7f47e8017000 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:28.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.319+0000 7f47deffd700 1 -- 172.21.15.27:0/2154192723 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f47f80a98a0 con 0x7f47f80c89f0 2024-01-11T09:45:28.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.520+0000 7f47fd7ff700 1 -- 172.21.15.27:0/2154192723 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f47f80a98a0 con 0x7f47e4066f30 2024-01-11T09:45:28.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.532+0000 7f47deffd700 1 -- 172.21.15.27:0/2154192723 <== mgr.10380 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289775 (secure 0 0 0) 0x7f47f80a98a0 con 0x7f47e4066f30 2024-01-11T09:45:28.529 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:28.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.536+0000 7f47fd7ff700 1 -- 172.21.15.27:0/2154192723 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f47e4066f30 msgr2=0x7f47e40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:28.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.536+0000 7f47fd7ff700 1 --2- 172.21.15.27:0/2154192723 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f47e4066f30 0x7f47e40693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f47e80091e0 tx=0x7f47e8017000 comp rx=0 tx=0).stop 2024-01-11T09:45:28.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.536+0000 7f47fd7ff700 1 -- 172.21.15.27:0/2154192723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f80c89f0 msgr2=0x7f47f806ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:28.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.536+0000 7f47fd7ff700 1 --2- 172.21.15.27:0/2154192723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f80c89f0 0x7f47f806ec60 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f47f000cbc0 tx=0x7f47f0001990 comp rx=0 tx=0).stop 2024-01-11T09:45:28.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.536+0000 7f47fd7ff700 1 -- 172.21.15.27:0/2154192723 shutdown_connections 2024-01-11T09:45:28.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.536+0000 7f47fd7ff700 1 --2- 172.21.15.27:0/2154192723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f80c89f0 0x7f47f806ec60 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:28.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.536+0000 7f47fd7ff700 1 --2- 172.21.15.27:0/2154192723 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f47e4066f30 0x7f47e40693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:28.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.536+0000 7f47fd7ff700 1 --2- 172.21.15.27:0/2154192723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47f8069d40 0x7f47f8138f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:28.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.536+0000 7f47fd7ff700 1 --2- 172.21.15.27:0/2154192723 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f47f80c8620 0x7f47f816cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:28.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.536+0000 7f47fd7ff700 1 -- 172.21.15.27:0/2154192723 >> 172.21.15.27:0/2154192723 conn(0x7f47f80680e0 msgr2=0x7f47f8139900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:28.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.536+0000 7f47fd7ff700 1 -- 172.21.15.27:0/2154192723 shutdown_connections 2024-01-11T09:45:28.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.537+0000 7f47fd7ff700 1 -- 172.21.15.27:0/2154192723 wait complete. 2024-01-11T09:45:28.532 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:45:28.547 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-11T09:45:28.301012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":655,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":404,"ondisk_log_size":404,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":158,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2461924,"kb_used_data":22180,"kb_used_omap":0,"kb_used_meta":2439680,"kb_avail":752512796,"statfs":{"total":773094113280,"available":770573103104,"internally_reserved":0,"allocated":22712320,"data_stored":11421188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2498232320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001270"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170476+0000","last_change":"2024-01-11T09:45:24.164535+0000","last_active":"2024-01-11T09:45:24.170476+0000","last_peered":"2024-01-11T09:45:24.170476+0000","last_clean":"2024-01-11T09:45:24.170476+0000","last_became_active":"2024-01-11T09:45:24.163889+0000","last_became_peered":"2024-01-11T09:45:24.163889+0000","last_unstale":"2024-01-11T09:45:24.170476+0000","last_undegraded":"2024-01-11T09:45:24.170476+0000","last_fullsized":"2024-01-11T09:45:24.170476+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:29:27.569363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.152866+0000","last_change":"2024-01-11T09:45:24.147655+0000","last_active":"2024-01-11T09:45:24.152866+0000","last_peered":"2024-01-11T09:45:24.152866+0000","last_clean":"2024-01-11T09:45:24.152866+0000","last_became_active":"2024-01-11T09:45:24.147267+0000","last_became_peered":"2024-01-11T09:45:24.147267+0000","last_unstale":"2024-01-11T09:45:24.152866+0000","last_undegraded":"2024-01-11T09:45:24.152866+0000","last_fullsized":"2024-01-11T09:45:24.152866+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:14:09.653181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.167348+0000","last_change":"2024-01-11T09:45:24.160833+0000","last_active":"2024-01-11T09:45:24.167348+0000","last_peered":"2024-01-11T09:45:24.167348+0000","last_clean":"2024-01-11T09:45:24.167348+0000","last_became_active":"2024-01-11T09:45:24.160557+0000","last_became_peered":"2024-01-11T09:45:24.160557+0000","last_unstale":"2024-01-11T09:45:24.167348+0000","last_undegraded":"2024-01-11T09:45:24.167348+0000","last_fullsized":"2024-01-11T09:45:24.167348+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:29:13.303845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.152846+0000","last_change":"2024-01-11T09:45:24.149180+0000","last_active":"2024-01-11T09:45:24.152846+0000","last_peered":"2024-01-11T09:45:24.152846+0000","last_clean":"2024-01-11T09:45:24.152846+0000","last_became_active":"2024-01-11T09:45:24.148899+0000","last_became_peered":"2024-01-11T09:45:24.148899+0000","last_unstale":"2024-01-11T09:45:24.152846+0000","last_undegraded":"2024-01-11T09:45:24.152846+0000","last_fullsized":"2024-01-11T09:45:24.152846+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:59:56.132354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.161919+0000","last_change":"2024-01-11T09:45:24.148863+0000","last_active":"2024-01-11T09:45:24.161919+0000","last_peered":"2024-01-11T09:45:24.161919+0000","last_clean":"2024-01-11T09:45:24.161919+0000","last_became_active":"2024-01-11T09:45:24.148533+0000","last_became_peered":"2024-01-11T09:45:24.148533+0000","last_unstale":"2024-01-11T09:45:24.161919+0000","last_undegraded":"2024-01-11T09:45:24.161919+0000","last_fullsized":"2024-01-11T09:45:24.161919+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:20.218042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.162905+0000","last_change":"2024-01-11T09:45:24.153507+0000","last_active":"2024-01-11T09:45:24.162905+0000","last_peered":"2024-01-11T09:45:24.162905+0000","last_clean":"2024-01-11T09:45:24.162905+0000","last_became_active":"2024-01-11T09:45:24.153235+0000","last_became_peered":"2024-01-11T09:45:24.153235+0000","last_unstale":"2024-01-11T09:45:24.162905+0000","last_undegraded":"2024-01-11T09:45:24.162905+0000","last_fullsized":"2024-01-11T09:45:24.162905+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:01:06.710133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170737+0000","last_change":"2024-01-11T09:45:24.164133+0000","last_active":"2024-01-11T09:45:24.170737+0000","last_peered":"2024-01-11T09:45:24.170737+0000","last_clean":"2024-01-11T09:45:24.170737+0000","last_became_active":"2024-01-11T09:45:24.163856+0000","last_became_peered":"2024-01-11T09:45:24.163856+0000","last_unstale":"2024-01-11T09:45:24.170737+0000","last_undegraded":"2024-01-11T09:45:24.170737+0000","last_fullsized":"2024-01-11T09:45:24.170737+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:17:24.413093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.173279+0000","last_change":"2024-01-11T09:45:24.169914+0000","last_active":"2024-01-11T09:45:24.173279+0000","last_peered":"2024-01-11T09:45:24.173279+0000","last_clean":"2024-01-11T09:45:24.173279+0000","last_became_active":"2024-01-11T09:45:24.169438+0000","last_became_peered":"2024-01-11T09:45:24.169438+0000","last_unstale":"2024-01-11T09:45:24.173279+0000","last_undegraded":"2024-01-11T09:45:24.173279+0000","last_fullsized":"2024-01-11T09:45:24.173279+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:43:22.330951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.157010+0000","last_change":"2024-01-11T09:45:24.155318+0000","last_active":"2024-01-11T09:45:24.157010+0000","last_peered":"2024-01-11T09:45:24.157010+0000","last_clean":"2024-01-11T09:45:24.157010+0000","last_became_active":"2024-01-11T09:45:24.153405+0000","last_became_peered":"2024-01-11T09:45:24.153405+0000","last_unstale":"2024-01-11T09:45:24.157010+0000","last_undegraded":"2024-01-11T09:45:24.157010+0000","last_fullsized":"2024-01-11T09:45:24.157010+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:52:32.496845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.161534+0000","last_change":"2024-01-11T09:45:24.152214+0000","last_active":"2024-01-11T09:45:24.161534+0000","last_peered":"2024-01-11T09:45:24.161534+0000","last_clean":"2024-01-11T09:45:24.161534+0000","last_became_active":"2024-01-11T09:45:24.151938+0000","last_became_peered":"2024-01-11T09:45:24.151938+0000","last_unstale":"2024-01-11T09:45:24.161534+0000","last_undegraded":"2024-01-11T09:45:24.161534+0000","last_fullsized":"2024-01-11T09:45:24.161534+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:05.090713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.167366+0000","last_change":"2024-01-11T09:45:24.161300+0000","last_active":"2024-01-11T09:45:24.167366+0000","last_peered":"2024-01-11T09:45:24.167366+0000","last_clean":"2024-01-11T09:45:24.167366+0000","last_became_active":"2024-01-11T09:45:24.160940+0000","last_became_peered":"2024-01-11T09:45:24.160940+0000","last_unstale":"2024-01-11T09:45:24.167366+0000","last_undegraded":"2024-01-11T09:45:24.167366+0000","last_fullsized":"2024-01-11T09:45:24.167366+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:40:06.419759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.147889+0000","last_change":"2024-01-11T09:45:24.146920+0000","last_active":"2024-01-11T09:45:24.147889+0000","last_peered":"2024-01-11T09:45:24.147889+0000","last_clean":"2024-01-11T09:45:24.147889+0000","last_became_active":"2024-01-11T09:45:24.146654+0000","last_became_peered":"2024-01-11T09:45:24.146654+0000","last_unstale":"2024-01-11T09:45:24.147889+0000","last_undegraded":"2024-01-11T09:45:24.147889+0000","last_fullsized":"2024-01-11T09:45:24.147889+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:42:21.543118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.159698+0000","last_change":"2024-01-11T09:45:24.158500+0000","last_active":"2024-01-11T09:45:24.159698+0000","last_peered":"2024-01-11T09:45:24.159698+0000","last_clean":"2024-01-11T09:45:24.159698+0000","last_became_active":"2024-01-11T09:45:24.158185+0000","last_became_peered":"2024-01-11T09:45:24.158185+0000","last_unstale":"2024-01-11T09:45:24.159698+0000","last_undegraded":"2024-01-11T09:45:24.159698+0000","last_fullsized":"2024-01-11T09:45:24.159698+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:52:38.581476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.168782+0000","last_change":"2024-01-11T09:45:24.157215+0000","last_active":"2024-01-11T09:45:24.168782+0000","last_peered":"2024-01-11T09:45:24.168782+0000","last_clean":"2024-01-11T09:45:24.168782+0000","last_became_active":"2024-01-11T09:45:24.156852+0000","last_became_peered":"2024-01-11T09:45:24.156852+0000","last_unstale":"2024-01-11T09:45:24.168782+0000","last_undegraded":"2024-01-11T09:45:24.168782+0000","last_fullsized":"2024-01-11T09:45:24.168782+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:42.716516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.171326+0000","last_change":"2024-01-11T09:45:24.170581+0000","last_active":"2024-01-11T09:45:24.171326+0000","last_peered":"2024-01-11T09:45:24.171326+0000","last_clean":"2024-01-11T09:45:24.171326+0000","last_became_active":"2024-01-11T09:45:24.170233+0000","last_became_peered":"2024-01-11T09:45:24.170233+0000","last_unstale":"2024-01-11T09:45:24.171326+0000","last_undegraded":"2024-01-11T09:45:24.171326+0000","last_fullsized":"2024-01-11T09:45:24.171326+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:50:50.820643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.152444+0000","last_change":"2024-01-11T09:45:24.148308+0000","last_active":"2024-01-11T09:45:24.152444+0000","last_peered":"2024-01-11T09:45:24.152444+0000","last_clean":"2024-01-11T09:45:24.152444+0000","last_became_active":"2024-01-11T09:45:24.148023+0000","last_became_peered":"2024-01-11T09:45:24.148023+0000","last_unstale":"2024-01-11T09:45:24.152444+0000","last_undegraded":"2024-01-11T09:45:24.152444+0000","last_fullsized":"2024-01-11T09:45:24.152444+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:56:51.623335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170453+0000","last_change":"2024-01-11T09:45:24.164171+0000","last_active":"2024-01-11T09:45:24.170453+0000","last_peered":"2024-01-11T09:45:24.170453+0000","last_clean":"2024-01-11T09:45:24.170453+0000","last_became_active":"2024-01-11T09:45:24.163522+0000","last_became_peered":"2024-01-11T09:45:24.163522+0000","last_unstale":"2024-01-11T09:45:24.170453+0000","last_undegraded":"2024-01-11T09:45:24.170453+0000","last_fullsized":"2024-01-11T09:45:24.170453+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:37:20.217033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170239+0000","last_change":"2024-01-11T09:45:24.166373+0000","last_active":"2024-01-11T09:45:24.170239+0000","last_peered":"2024-01-11T09:45:24.170239+0000","last_clean":"2024-01-11T09:45:24.170239+0000","last_became_active":"2024-01-11T09:45:24.166090+0000","last_became_peered":"2024-01-11T09:45:24.166090+0000","last_unstale":"2024-01-11T09:45:24.170239+0000","last_undegraded":"2024-01-11T09:45:24.170239+0000","last_fullsized":"2024-01-11T09:45:24.170239+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:16:58.784096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.162812+0000","last_change":"2024-01-11T09:45:24.153140+0000","last_active":"2024-01-11T09:45:24.162812+0000","last_peered":"2024-01-11T09:45:24.162812+0000","last_clean":"2024-01-11T09:45:24.162812+0000","last_became_active":"2024-01-11T09:45:24.152890+0000","last_became_peered":"2024-01-11T09:45:24.152890+0000","last_unstale":"2024-01-11T09:45:24.162812+0000","last_undegraded":"2024-01-11T09:45:24.162812+0000","last_fullsized":"2024-01-11T09:45:24.162812+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:39.406018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.162866+0000","last_change":"2024-01-11T09:45:24.156832+0000","last_active":"2024-01-11T09:45:24.162866+0000","last_peered":"2024-01-11T09:45:24.162866+0000","last_clean":"2024-01-11T09:45:24.162866+0000","last_became_active":"2024-01-11T09:45:24.156541+0000","last_became_peered":"2024-01-11T09:45:24.156541+0000","last_unstale":"2024-01-11T09:45:24.162866+0000","last_undegraded":"2024-01-11T09:45:24.162866+0000","last_fullsized":"2024-01-11T09:45:24.162866+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:49.368305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.169864+0000","last_change":"2024-01-11T09:45:24.167154+0000","last_active":"2024-01-11T09:45:24.169864+0000","last_peered":"2024-01-11T09:45:24.169864+0000","last_clean":"2024-01-11T09:45:24.169864+0000","last_became_active":"2024-01-11T09:45:24.166862+0000","last_became_peered":"2024-01-11T09:45:24.166862+0000","last_unstale":"2024-01-11T09:45:24.169864+0000","last_undegraded":"2024-01-11T09:45:24.169864+0000","last_fullsized":"2024-01-11T09:45:24.169864+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:15:00.803121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.147395+0000","last_change":"2024-01-11T09:45:24.145558+0000","last_active":"2024-01-11T09:45:24.147395+0000","last_peered":"2024-01-11T09:45:24.147395+0000","last_clean":"2024-01-11T09:45:24.147395+0000","last_became_active":"2024-01-11T09:45:24.143216+0000","last_became_peered":"2024-01-11T09:45:24.143216+0000","last_unstale":"2024-01-11T09:45:24.147395+0000","last_undegraded":"2024-01-11T09:45:24.147395+0000","last_fullsized":"2024-01-11T09:45:24.147395+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:32:19.368822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.161415+0000","last_change":"2024-01-11T09:45:24.158003+0000","last_active":"2024-01-11T09:45:24.161415+0000","last_peered":"2024-01-11T09:45:24.161415+0000","last_clean":"2024-01-11T09:45:24.161415+0000","last_became_active":"2024-01-11T09:45:24.157710+0000","last_became_peered":"2024-01-11T09:45:24.157710+0000","last_unstale":"2024-01-11T09:45:24.161415+0000","last_undegraded":"2024-01-11T09:45:24.161415+0000","last_fullsized":"2024-01-11T09:45:24.161415+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:52:18.543824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.169953+0000","last_change":"2024-01-11T09:45:24.168887+0000","last_active":"2024-01-11T09:45:24.169953+0000","last_peered":"2024-01-11T09:45:24.169953+0000","last_clean":"2024-01-11T09:45:24.169953+0000","last_became_active":"2024-01-11T09:45:24.167138+0000","last_became_peered":"2024-01-11T09:45:24.167138+0000","last_unstale":"2024-01-11T09:45:24.169953+0000","last_undegraded":"2024-01-11T09:45:24.169953+0000","last_fullsized":"2024-01-11T09:45:24.169953+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:43:22.330951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.151363+0000","last_change":"2024-01-11T09:45:24.145841+0000","last_active":"2024-01-11T09:45:24.151363+0000","last_peered":"2024-01-11T09:45:24.151363+0000","last_clean":"2024-01-11T09:45:24.151363+0000","last_became_active":"2024-01-11T09:45:24.145584+0000","last_became_peered":"2024-01-11T09:45:24.145584+0000","last_unstale":"2024-01-11T09:45:24.151363+0000","last_undegraded":"2024-01-11T09:45:24.151363+0000","last_fullsized":"2024-01-11T09:45:24.151363+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:47.970580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.169381+0000","last_change":"2024-01-11T09:45:24.158803+0000","last_active":"2024-01-11T09:45:24.169381+0000","last_peered":"2024-01-11T09:45:24.169381+0000","last_clean":"2024-01-11T09:45:24.169381+0000","last_became_active":"2024-01-11T09:45:24.158533+0000","last_became_peered":"2024-01-11T09:45:24.158533+0000","last_unstale":"2024-01-11T09:45:24.169381+0000","last_undegraded":"2024-01-11T09:45:24.169381+0000","last_fullsized":"2024-01-11T09:45:24.169381+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:57:46.243868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.155273+0000","last_change":"2024-01-11T09:45:24.150865+0000","last_active":"2024-01-11T09:45:24.155273+0000","last_peered":"2024-01-11T09:45:24.155273+0000","last_clean":"2024-01-11T09:45:24.155273+0000","last_became_active":"2024-01-11T09:45:24.148370+0000","last_became_peered":"2024-01-11T09:45:24.148370+0000","last_unstale":"2024-01-11T09:45:24.155273+0000","last_undegraded":"2024-01-11T09:45:24.155273+0000","last_fullsized":"2024-01-11T09:45:24.155273+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:27.774010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.171257+0000","last_change":"2024-01-11T09:45:24.166152+0000","last_active":"2024-01-11T09:45:24.171257+0000","last_peered":"2024-01-11T09:45:24.171257+0000","last_clean":"2024-01-11T09:45:24.171257+0000","last_became_active":"2024-01-11T09:45:24.165889+0000","last_became_peered":"2024-01-11T09:45:24.165889+0000","last_unstale":"2024-01-11T09:45:24.171257+0000","last_undegraded":"2024-01-11T09:45:24.171257+0000","last_fullsized":"2024-01-11T09:45:24.171257+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:01:06.710133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.150341+0000","last_change":"2024-01-11T09:45:24.148965+0000","last_active":"2024-01-11T09:45:24.150341+0000","last_peered":"2024-01-11T09:45:24.150341+0000","last_clean":"2024-01-11T09:45:24.150341+0000","last_became_active":"2024-01-11T09:45:24.148481+0000","last_became_peered":"2024-01-11T09:45:24.148481+0000","last_unstale":"2024-01-11T09:45:24.150341+0000","last_undegraded":"2024-01-11T09:45:24.150341+0000","last_fullsized":"2024-01-11T09:45:24.150341+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:39.406018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.149821+0000","last_change":"2024-01-11T09:45:24.146908+0000","last_active":"2024-01-11T09:45:24.149821+0000","last_peered":"2024-01-11T09:45:24.149821+0000","last_clean":"2024-01-11T09:45:24.149821+0000","last_became_active":"2024-01-11T09:45:24.146543+0000","last_became_peered":"2024-01-11T09:45:24.146543+0000","last_unstale":"2024-01-11T09:45:24.149821+0000","last_undegraded":"2024-01-11T09:45:24.149821+0000","last_fullsized":"2024-01-11T09:45:24.149821+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:40.983480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.160800+0000","last_change":"2024-01-11T09:45:24.155055+0000","last_active":"2024-01-11T09:45:24.160800+0000","last_peered":"2024-01-11T09:45:24.160800+0000","last_clean":"2024-01-11T09:45:24.160800+0000","last_became_active":"2024-01-11T09:45:24.154781+0000","last_became_peered":"2024-01-11T09:45:24.154781+0000","last_unstale":"2024-01-11T09:45:24.160800+0000","last_undegraded":"2024-01-11T09:45:24.160800+0000","last_fullsized":"2024-01-11T09:45:24.160800+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:38:28.859392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170578+0000","last_change":"2024-01-11T09:45:24.167922+0000","last_active":"2024-01-11T09:45:24.170578+0000","last_peered":"2024-01-11T09:45:24.170578+0000","last_clean":"2024-01-11T09:45:24.170578+0000","last_became_active":"2024-01-11T09:45:24.167641+0000","last_became_peered":"2024-01-11T09:45:24.167641+0000","last_unstale":"2024-01-11T09:45:24.170578+0000","last_undegraded":"2024-01-11T09:45:24.170578+0000","last_fullsized":"2024-01-11T09:45:24.170578+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:17.184569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.168556+0000","last_change":"2024-01-11T09:45:24.167467+0000","last_active":"2024-01-11T09:45:24.168556+0000","last_peered":"2024-01-11T09:45:24.168556+0000","last_clean":"2024-01-11T09:45:24.168556+0000","last_became_active":"2024-01-11T09:45:24.167175+0000","last_became_peered":"2024-01-11T09:45:24.167175+0000","last_unstale":"2024-01-11T09:45:24.168556+0000","last_undegraded":"2024-01-11T09:45:24.168556+0000","last_fullsized":"2024-01-11T09:45:24.168556+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:52:36.044653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"213'4","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.168564+0000","last_change":"2024-01-11T09:45:23.150803+0000","last_active":"2024-01-11T09:45:24.168564+0000","last_peered":"2024-01-11T09:45:24.168564+0000","last_clean":"2024-01-11T09:45:24.168564+0000","last_became_active":"2024-01-11T09:45:23.149761+0000","last_became_peered":"2024-01-11T09:45:23.149761+0000","last_unstale":"2024-01-11T09:45:24.168564+0000","last_undegraded":"2024-01-11T09:45:24.168564+0000","last_fullsized":"2024-01-11T09:45:24.168564+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:33:40.495800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.155574+0000","last_change":"2024-01-11T09:45:24.150915+0000","last_active":"2024-01-11T09:45:24.155574+0000","last_peered":"2024-01-11T09:45:24.155574+0000","last_clean":"2024-01-11T09:45:24.155574+0000","last_became_active":"2024-01-11T09:45:24.149015+0000","last_became_peered":"2024-01-11T09:45:24.149015+0000","last_unstale":"2024-01-11T09:45:24.155574+0000","last_undegraded":"2024-01-11T09:45:24.155574+0000","last_fullsized":"2024-01-11T09:45:24.155574+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:14:14.599129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"213'1","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.168906+0000","last_change":"2024-01-11T09:45:23.148530+0000","last_active":"2024-01-11T09:45:24.168906+0000","last_peered":"2024-01-11T09:45:24.168906+0000","last_clean":"2024-01-11T09:45:24.168906+0000","last_became_active":"2024-01-11T09:45:23.148208+0000","last_became_peered":"2024-01-11T09:45:23.148208+0000","last_unstale":"2024-01-11T09:45:24.168906+0000","last_undegraded":"2024-01-11T09:45:24.168906+0000","last_fullsized":"2024-01-11T09:45:24.168906+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:44:55.075364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170535+0000","last_change":"2024-01-11T09:45:24.165254+0000","last_active":"2024-01-11T09:45:24.170535+0000","last_peered":"2024-01-11T09:45:24.170535+0000","last_clean":"2024-01-11T09:45:24.170535+0000","last_became_active":"2024-01-11T09:45:24.163213+0000","last_became_peered":"2024-01-11T09:45:24.163213+0000","last_unstale":"2024-01-11T09:45:24.170535+0000","last_undegraded":"2024-01-11T09:45:24.170535+0000","last_fullsized":"2024-01-11T09:45:24.170535+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:29:21.907384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"213'2","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.173466+0000","last_change":"2024-01-11T09:45:23.144976+0000","last_active":"2024-01-11T09:45:24.173466+0000","last_peered":"2024-01-11T09:45:24.173466+0000","last_clean":"2024-01-11T09:45:24.173466+0000","last_became_active":"2024-01-11T09:45:23.144660+0000","last_became_peered":"2024-01-11T09:45:23.144660+0000","last_unstale":"2024-01-11T09:45:24.173466+0000","last_undegraded":"2024-01-11T09:45:24.173466+0000","last_fullsized":"2024-01-11T09:45:24.173466+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:59:40.579473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.168826+0000","last_change":"2024-01-11T09:45:24.164335+0000","last_active":"2024-01-11T09:45:24.168826+0000","last_peered":"2024-01-11T09:45:24.168826+0000","last_clean":"2024-01-11T09:45:24.168826+0000","last_became_active":"2024-01-11T09:45:24.164063+0000","last_became_peered":"2024-01-11T09:45:24.164063+0000","last_unstale":"2024-01-11T09:45:24.168826+0000","last_undegraded":"2024-01-11T09:45:24.168826+0000","last_fullsized":"2024-01-11T09:45:24.168826+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:42:21.543118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"213'2","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.172142+0000","last_change":"2024-01-11T09:45:23.150649+0000","last_active":"2024-01-11T09:45:24.172142+0000","last_peered":"2024-01-11T09:45:24.172142+0000","last_clean":"2024-01-11T09:45:24.172142+0000","last_became_active":"2024-01-11T09:45:23.150425+0000","last_became_peered":"2024-01-11T09:45:23.150425+0000","last_unstale":"2024-01-11T09:45:24.172142+0000","last_undegraded":"2024-01-11T09:45:24.172142+0000","last_fullsized":"2024-01-11T09:45:24.172142+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:14:08.645320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.150811+0000","last_change":"2024-01-11T09:45:24.148848+0000","last_active":"2024-01-11T09:45:24.150811+0000","last_peered":"2024-01-11T09:45:24.150811+0000","last_clean":"2024-01-11T09:45:24.150811+0000","last_became_active":"2024-01-11T09:45:24.148573+0000","last_became_peered":"2024-01-11T09:45:24.148573+0000","last_unstale":"2024-01-11T09:45:24.150811+0000","last_undegraded":"2024-01-11T09:45:24.150811+0000","last_fullsized":"2024-01-11T09:45:24.150811+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:29:30.529875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"213'1","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.164581+0000","last_change":"2024-01-11T09:45:23.154575+0000","last_active":"2024-01-11T09:45:24.164581+0000","last_peered":"2024-01-11T09:45:24.164581+0000","last_clean":"2024-01-11T09:45:24.164581+0000","last_became_active":"2024-01-11T09:45:23.154313+0000","last_became_peered":"2024-01-11T09:45:23.154313+0000","last_unstale":"2024-01-11T09:45:24.164581+0000","last_undegraded":"2024-01-11T09:45:24.164581+0000","last_fullsized":"2024-01-11T09:45:24.164581+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:14:08.645320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.162735+0000","last_change":"2024-01-11T09:45:24.150698+0000","last_active":"2024-01-11T09:45:24.162735+0000","last_peered":"2024-01-11T09:45:24.162735+0000","last_clean":"2024-01-11T09:45:24.162735+0000","last_became_active":"2024-01-11T09:45:24.149431+0000","last_became_peered":"2024-01-11T09:45:24.149431+0000","last_unstale":"2024-01-11T09:45:24.162735+0000","last_undegraded":"2024-01-11T09:45:24.162735+0000","last_fullsized":"2024-01-11T09:45:24.162735+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:48:06.986946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.143494+0000","last_change":"2024-01-11T09:45:23.145331+0000","last_active":"2024-01-11T09:45:24.143494+0000","last_peered":"2024-01-11T09:45:24.143494+0000","last_clean":"2024-01-11T09:45:24.143494+0000","last_became_active":"2024-01-11T09:45:23.145064+0000","last_became_peered":"2024-01-11T09:45:23.145064+0000","last_unstale":"2024-01-11T09:45:24.143494+0000","last_undegraded":"2024-01-11T09:45:24.143494+0000","last_fullsized":"2024-01-11T09:45:24.143494+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:56:41.726387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.154619+0000","last_change":"2024-01-11T09:45:24.150754+0000","last_active":"2024-01-11T09:45:24.154619+0000","last_peered":"2024-01-11T09:45:24.154619+0000","last_clean":"2024-01-11T09:45:24.154619+0000","last_became_active":"2024-01-11T09:45:24.150485+0000","last_became_peered":"2024-01-11T09:45:24.150485+0000","last_unstale":"2024-01-11T09:45:24.154619+0000","last_undegraded":"2024-01-11T09:45:24.154619+0000","last_fullsized":"2024-01-11T09:45:24.154619+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:32:19.368822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.139464+0000","last_change":"2024-01-11T09:45:23.152843+0000","last_active":"2024-01-11T09:45:24.139464+0000","last_peered":"2024-01-11T09:45:24.139464+0000","last_clean":"2024-01-11T09:45:24.139464+0000","last_became_active":"2024-01-11T09:45:23.152579+0000","last_became_peered":"2024-01-11T09:45:23.152579+0000","last_unstale":"2024-01-11T09:45:24.139464+0000","last_undegraded":"2024-01-11T09:45:24.139464+0000","last_fullsized":"2024-01-11T09:45:24.139464+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:48:05.979085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.159625+0000","last_change":"2024-01-11T09:45:24.155035+0000","last_active":"2024-01-11T09:45:24.159625+0000","last_peered":"2024-01-11T09:45:24.159625+0000","last_clean":"2024-01-11T09:45:24.159625+0000","last_became_active":"2024-01-11T09:45:24.153980+0000","last_became_peered":"2024-01-11T09:45:24.153980+0000","last_unstale":"2024-01-11T09:45:24.159625+0000","last_undegraded":"2024-01-11T09:45:24.159625+0000","last_fullsized":"2024-01-11T09:45:24.159625+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:52:20.741771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.143930+0000","last_change":"2024-01-11T09:45:23.152897+0000","last_active":"2024-01-11T09:45:24.143930+0000","last_peered":"2024-01-11T09:45:24.143930+0000","last_clean":"2024-01-11T09:45:24.143930+0000","last_became_active":"2024-01-11T09:45:23.152620+0000","last_became_peered":"2024-01-11T09:45:23.152620+0000","last_unstale":"2024-01-11T09:45:24.143930+0000","last_undegraded":"2024-01-11T09:45:24.143930+0000","last_fullsized":"2024-01-11T09:45:24.143930+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:02:41.619972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.171300+0000","last_change":"2024-01-11T09:45:24.166531+0000","last_active":"2024-01-11T09:45:24.171300+0000","last_peered":"2024-01-11T09:45:24.171300+0000","last_clean":"2024-01-11T09:45:24.171300+0000","last_became_active":"2024-01-11T09:45:24.164071+0000","last_became_peered":"2024-01-11T09:45:24.164071+0000","last_unstale":"2024-01-11T09:45:24.171300+0000","last_undegraded":"2024-01-11T09:45:24.171300+0000","last_fullsized":"2024-01-11T09:45:24.171300+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:41:10.078741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.139847+0000","last_change":"2024-01-11T09:45:23.150869+0000","last_active":"2024-01-11T09:45:24.139847+0000","last_peered":"2024-01-11T09:45:24.139847+0000","last_clean":"2024-01-11T09:45:24.139847+0000","last_became_active":"2024-01-11T09:45:23.150604+0000","last_became_peered":"2024-01-11T09:45:23.150604+0000","last_unstale":"2024-01-11T09:45:24.139847+0000","last_undegraded":"2024-01-11T09:45:24.139847+0000","last_fullsized":"2024-01-11T09:45:24.139847+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:19.210181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.152867+0000","last_change":"2024-01-11T09:45:24.150019+0000","last_active":"2024-01-11T09:45:24.152867+0000","last_peered":"2024-01-11T09:45:24.152867+0000","last_clean":"2024-01-11T09:45:24.152867+0000","last_became_active":"2024-01-11T09:45:24.148994+0000","last_became_peered":"2024-01-11T09:45:24.148994+0000","last_unstale":"2024-01-11T09:45:24.152867+0000","last_undegraded":"2024-01-11T09:45:24.152867+0000","last_fullsized":"2024-01-11T09:45:24.152867+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:29:01.556722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"213'3","reported_seq":13,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.171173+0000","last_change":"2024-01-11T09:45:23.154355+0000","last_active":"2024-01-11T09:45:24.171173+0000","last_peered":"2024-01-11T09:45:24.171173+0000","last_clean":"2024-01-11T09:45:24.171173+0000","last_became_active":"2024-01-11T09:45:23.153262+0000","last_became_peered":"2024-01-11T09:45:23.153262+0000","last_unstale":"2024-01-11T09:45:24.171173+0000","last_undegraded":"2024-01-11T09:45:24.171173+0000","last_fullsized":"2024-01-11T09:45:24.171173+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:32:15.216292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.171076+0000","last_change":"2024-01-11T09:45:24.169731+0000","last_active":"2024-01-11T09:45:24.171076+0000","last_peered":"2024-01-11T09:45:24.171076+0000","last_clean":"2024-01-11T09:45:24.171076+0000","last_became_active":"2024-01-11T09:45:24.169381+0000","last_became_peered":"2024-01-11T09:45:24.169381+0000","last_unstale":"2024-01-11T09:45:24.171076+0000","last_undegraded":"2024-01-11T09:45:24.171076+0000","last_fullsized":"2024-01-11T09:45:24.171076+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:04:41.533247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"213'6","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.171979+0000","last_change":"2024-01-11T09:45:23.152136+0000","last_active":"2024-01-11T09:45:24.171979+0000","last_peered":"2024-01-11T09:45:24.171979+0000","last_clean":"2024-01-11T09:45:24.171979+0000","last_became_active":"2024-01-11T09:45:23.151819+0000","last_became_peered":"2024-01-11T09:45:23.151819+0000","last_unstale":"2024-01-11T09:45:24.171979+0000","last_undegraded":"2024-01-11T09:45:24.171979+0000","last_fullsized":"2024-01-11T09:45:24.171979+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:41:09.070880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.163042+0000","last_change":"2024-01-11T09:45:24.157881+0000","last_active":"2024-01-11T09:45:24.163042+0000","last_peered":"2024-01-11T09:45:24.163042+0000","last_clean":"2024-01-11T09:45:24.163042+0000","last_became_active":"2024-01-11T09:45:24.157224+0000","last_became_peered":"2024-01-11T09:45:24.157224+0000","last_unstale":"2024-01-11T09:45:24.163042+0000","last_undegraded":"2024-01-11T09:45:24.163042+0000","last_fullsized":"2024-01-11T09:45:24.163042+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:32:16.224153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"213'2","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.165110+0000","last_change":"2024-01-11T09:45:23.155269+0000","last_active":"2024-01-11T09:45:24.165110+0000","last_peered":"2024-01-11T09:45:24.165110+0000","last_clean":"2024-01-11T09:45:24.165110+0000","last_became_active":"2024-01-11T09:45:23.154990+0000","last_became_peered":"2024-01-11T09:45:23.154990+0000","last_unstale":"2024-01-11T09:45:24.165110+0000","last_undegraded":"2024-01-11T09:45:24.165110+0000","last_fullsized":"2024-01-11T09:45:24.165110+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:32:15.216292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.152306+0000","last_change":"2024-01-11T09:45:24.147696+0000","last_active":"2024-01-11T09:45:24.152306+0000","last_peered":"2024-01-11T09:45:24.152306+0000","last_clean":"2024-01-11T09:45:24.152306+0000","last_became_active":"2024-01-11T09:45:24.147422+0000","last_became_peered":"2024-01-11T09:45:24.147422+0000","last_unstale":"2024-01-11T09:45:24.152306+0000","last_undegraded":"2024-01-11T09:45:24.152306+0000","last_fullsized":"2024-01-11T09:45:24.152306+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:32:40.240733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.138676+0000","last_change":"2024-01-11T09:45:23.152794+0000","last_active":"2024-01-11T09:45:24.138676+0000","last_peered":"2024-01-11T09:45:24.138676+0000","last_clean":"2024-01-11T09:45:24.138676+0000","last_became_active":"2024-01-11T09:45:23.152495+0000","last_became_peered":"2024-01-11T09:45:23.152495+0000","last_unstale":"2024-01-11T09:45:24.138676+0000","last_undegraded":"2024-01-11T09:45:24.138676+0000","last_fullsized":"2024-01-11T09:45:24.138676+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:01:05.702272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.171333+0000","last_change":"2024-01-11T09:45:24.166383+0000","last_active":"2024-01-11T09:45:24.171333+0000","last_peered":"2024-01-11T09:45:24.171333+0000","last_clean":"2024-01-11T09:45:24.171333+0000","last_became_active":"2024-01-11T09:45:24.159441+0000","last_became_peered":"2024-01-11T09:45:24.159441+0000","last_unstale":"2024-01-11T09:45:24.171333+0000","last_undegraded":"2024-01-11T09:45:24.171333+0000","last_fullsized":"2024-01-11T09:45:24.171333+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:25:32.122001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.130735+0000","last_change":"2024-01-11T09:45:23.151388+0000","last_active":"2024-01-11T09:45:24.130735+0000","last_peered":"2024-01-11T09:45:24.130735+0000","last_clean":"2024-01-11T09:45:24.130735+0000","last_became_active":"2024-01-11T09:45:23.151127+0000","last_became_peered":"2024-01-11T09:45:23.151127+0000","last_unstale":"2024-01-11T09:45:24.130735+0000","last_undegraded":"2024-01-11T09:45:24.130735+0000","last_fullsized":"2024-01-11T09:45:24.130735+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:43:21.323090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170718+0000","last_change":"2024-01-11T09:45:24.154643+0000","last_active":"2024-01-11T09:45:24.170718+0000","last_peered":"2024-01-11T09:45:24.170718+0000","last_clean":"2024-01-11T09:45:24.170718+0000","last_became_active":"2024-01-11T09:45:24.154293+0000","last_became_peered":"2024-01-11T09:45:24.154293+0000","last_unstale":"2024-01-11T09:45:24.170718+0000","last_undegraded":"2024-01-11T09:45:24.170718+0000","last_fullsized":"2024-01-11T09:45:24.170718+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:00:36.090834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"213'8","reported_seq":18,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.174799+0000","last_change":"2024-01-11T09:45:23.146296+0000","last_active":"2024-01-11T09:45:24.174799+0000","last_peered":"2024-01-11T09:45:24.174799+0000","last_clean":"2024-01-11T09:45:24.174799+0000","last_became_active":"2024-01-11T09:45:23.146026+0000","last_became_peered":"2024-01-11T09:45:23.146026+0000","last_unstale":"2024-01-11T09:45:24.174799+0000","last_undegraded":"2024-01-11T09:45:24.174799+0000","last_fullsized":"2024-01-11T09:45:24.174799+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:38:30.060556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.157400+0000","last_change":"2024-01-11T09:45:24.152987+0000","last_active":"2024-01-11T09:45:24.157400+0000","last_peered":"2024-01-11T09:45:24.157400+0000","last_clean":"2024-01-11T09:45:24.157400+0000","last_became_active":"2024-01-11T09:45:24.152677+0000","last_became_peered":"2024-01-11T09:45:24.152677+0000","last_unstale":"2024-01-11T09:45:24.157400+0000","last_undegraded":"2024-01-11T09:45:24.157400+0000","last_fullsized":"2024-01-11T09:45:24.157400+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:03.360851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"213'1","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.167919+0000","last_change":"2024-01-11T09:45:23.152322+0000","last_active":"2024-01-11T09:45:24.167919+0000","last_peered":"2024-01-11T09:45:24.167919+0000","last_clean":"2024-01-11T09:45:24.167919+0000","last_became_active":"2024-01-11T09:45:23.152006+0000","last_became_peered":"2024-01-11T09:45:23.152006+0000","last_unstale":"2024-01-11T09:45:24.167919+0000","last_undegraded":"2024-01-11T09:45:24.167919+0000","last_fullsized":"2024-01-11T09:45:24.167919+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:46.962719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.143849+0000","last_change":"2024-01-11T09:45:24.140693+0000","last_active":"2024-01-11T09:45:24.143849+0000","last_peered":"2024-01-11T09:45:24.143849+0000","last_clean":"2024-01-11T09:45:24.143849+0000","last_became_active":"2024-01-11T09:45:24.140376+0000","last_became_peered":"2024-01-11T09:45:24.140376+0000","last_unstale":"2024-01-11T09:45:24.143849+0000","last_undegraded":"2024-01-11T09:45:24.143849+0000","last_fullsized":"2024-01-11T09:45:24.143849+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:52:17.873609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"213'2","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170711+0000","last_change":"2024-01-11T09:45:23.147655+0000","last_active":"2024-01-11T09:45:24.170711+0000","last_peered":"2024-01-11T09:45:24.170711+0000","last_clean":"2024-01-11T09:45:24.170711+0000","last_became_active":"2024-01-11T09:45:23.147203+0000","last_became_peered":"2024-01-11T09:45:23.147203+0000","last_unstale":"2024-01-11T09:45:24.170711+0000","last_undegraded":"2024-01-11T09:45:24.170711+0000","last_fullsized":"2024-01-11T09:45:24.170711+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:16:31.626739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.166957+0000","last_change":"2024-01-11T09:45:24.159728+0000","last_active":"2024-01-11T09:45:24.166957+0000","last_peered":"2024-01-11T09:45:24.166957+0000","last_clean":"2024-01-11T09:45:24.166957+0000","last_became_active":"2024-01-11T09:45:24.159489+0000","last_became_peered":"2024-01-11T09:45:24.159489+0000","last_unstale":"2024-01-11T09:45:24.166957+0000","last_undegraded":"2024-01-11T09:45:24.166957+0000","last_fullsized":"2024-01-11T09:45:24.166957+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:21:50.096457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"213'1","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.167618+0000","last_change":"2024-01-11T09:45:23.153536+0000","last_active":"2024-01-11T09:45:24.167618+0000","last_peered":"2024-01-11T09:45:24.167618+0000","last_clean":"2024-01-11T09:45:24.167618+0000","last_became_active":"2024-01-11T09:45:23.152838+0000","last_became_peered":"2024-01-11T09:45:23.152838+0000","last_unstale":"2024-01-11T09:45:24.167618+0000","last_undegraded":"2024-01-11T09:45:24.167618+0000","last_fullsized":"2024-01-11T09:45:24.167618+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:14:59.795260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.153884+0000","last_change":"2024-01-11T09:45:24.149619+0000","last_active":"2024-01-11T09:45:24.153884+0000","last_peered":"2024-01-11T09:45:24.153884+0000","last_clean":"2024-01-11T09:45:24.153884+0000","last_became_active":"2024-01-11T09:45:24.148547+0000","last_became_peered":"2024-01-11T09:45:24.148547+0000","last_unstale":"2024-01-11T09:45:24.153884+0000","last_undegraded":"2024-01-11T09:45:24.153884+0000","last_fullsized":"2024-01-11T09:45:24.153884+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:28:18.397764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.152091+0000","last_change":"2024-01-11T09:45:23.155273+0000","last_active":"2024-01-11T09:45:24.152091+0000","last_peered":"2024-01-11T09:45:24.152091+0000","last_clean":"2024-01-11T09:45:24.152091+0000","last_became_active":"2024-01-11T09:45:23.155023+0000","last_became_peered":"2024-01-11T09:45:23.155023+0000","last_unstale":"2024-01-11T09:45:24.152091+0000","last_undegraded":"2024-01-11T09:45:24.152091+0000","last_fullsized":"2024-01-11T09:45:24.152091+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:03:59.152882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.151452+0000","last_change":"2024-01-11T09:45:24.145805+0000","last_active":"2024-01-11T09:45:24.151452+0000","last_peered":"2024-01-11T09:45:24.151452+0000","last_clean":"2024-01-11T09:45:24.151452+0000","last_became_active":"2024-01-11T09:45:24.145497+0000","last_became_peered":"2024-01-11T09:45:24.145497+0000","last_unstale":"2024-01-11T09:45:24.151452+0000","last_undegraded":"2024-01-11T09:45:24.151452+0000","last_fullsized":"2024-01-11T09:45:24.151452+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:27.774010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"213'2","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.172984+0000","last_change":"2024-01-11T09:45:23.143823+0000","last_active":"2024-01-11T09:45:24.172984+0000","last_peered":"2024-01-11T09:45:24.172984+0000","last_clean":"2024-01-11T09:45:24.172984+0000","last_became_active":"2024-01-11T09:45:23.141580+0000","last_became_peered":"2024-01-11T09:45:23.141580+0000","last_unstale":"2024-01-11T09:45:24.172984+0000","last_undegraded":"2024-01-11T09:45:24.172984+0000","last_fullsized":"2024-01-11T09:45:24.172984+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:19.210181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.172210+0000","last_change":"2024-01-11T09:45:24.168028+0000","last_active":"2024-01-11T09:45:24.172210+0000","last_peered":"2024-01-11T09:45:24.172210+0000","last_clean":"2024-01-11T09:45:24.172210+0000","last_became_active":"2024-01-11T09:45:24.167726+0000","last_became_peered":"2024-01-11T09:45:24.167726+0000","last_unstale":"2024-01-11T09:45:24.172210+0000","last_undegraded":"2024-01-11T09:45:24.172210+0000","last_fullsized":"2024-01-11T09:45:24.172210+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:48:06.986946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.152607+0000","last_change":"2024-01-11T09:45:23.154111+0000","last_active":"2024-01-11T09:45:24.152607+0000","last_peered":"2024-01-11T09:45:24.152607+0000","last_clean":"2024-01-11T09:45:24.152607+0000","last_became_active":"2024-01-11T09:45:23.151742+0000","last_became_peered":"2024-01-11T09:45:23.151742+0000","last_unstale":"2024-01-11T09:45:24.152607+0000","last_undegraded":"2024-01-11T09:45:24.152607+0000","last_fullsized":"2024-01-11T09:45:24.152607+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:38.398157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.150062+0000","last_change":"2024-01-11T09:45:24.144965+0000","last_active":"2024-01-11T09:45:24.150062+0000","last_peered":"2024-01-11T09:45:24.150062+0000","last_clean":"2024-01-11T09:45:24.150062+0000","last_became_active":"2024-01-11T09:45:24.143991+0000","last_became_peered":"2024-01-11T09:45:24.143991+0000","last_unstale":"2024-01-11T09:45:24.150062+0000","last_undegraded":"2024-01-11T09:45:24.150062+0000","last_fullsized":"2024-01-11T09:45:24.150062+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:56:42.734248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.152877+0000","last_change":"2024-01-11T09:45:23.155951+0000","last_active":"2024-01-11T09:45:24.152877+0000","last_peered":"2024-01-11T09:45:24.152877+0000","last_clean":"2024-01-11T09:45:24.152877+0000","last_became_active":"2024-01-11T09:45:23.155673+0000","last_became_peered":"2024-01-11T09:45:23.155673+0000","last_unstale":"2024-01-11T09:45:24.152877+0000","last_undegraded":"2024-01-11T09:45:24.152877+0000","last_fullsized":"2024-01-11T09:45:24.152877+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:52:37.573615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.156483+0000","last_change":"2024-01-11T09:45:24.150105+0000","last_active":"2024-01-11T09:45:24.156483+0000","last_peered":"2024-01-11T09:45:24.156483+0000","last_clean":"2024-01-11T09:45:24.156483+0000","last_became_active":"2024-01-11T09:45:24.149005+0000","last_became_peered":"2024-01-11T09:45:24.149005+0000","last_unstale":"2024-01-11T09:45:24.156483+0000","last_undegraded":"2024-01-11T09:45:24.156483+0000","last_fullsized":"2024-01-11T09:45:24.156483+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:25:27.316660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.143649+0000","last_change":"2024-01-11T09:45:23.154230+0000","last_active":"2024-01-11T09:45:24.143649+0000","last_peered":"2024-01-11T09:45:24.143649+0000","last_clean":"2024-01-11T09:45:24.143649+0000","last_became_active":"2024-01-11T09:45:23.153324+0000","last_became_peered":"2024-01-11T09:45:23.153324+0000","last_unstale":"2024-01-11T09:45:24.143649+0000","last_undegraded":"2024-01-11T09:45:24.143649+0000","last_fullsized":"2024-01-11T09:45:24.143649+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:52:16.865748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.152685+0000","last_change":"2024-01-11T09:45:24.149162+0000","last_active":"2024-01-11T09:45:24.152685+0000","last_peered":"2024-01-11T09:45:24.152685+0000","last_clean":"2024-01-11T09:45:24.152685+0000","last_became_active":"2024-01-11T09:45:24.148510+0000","last_became_peered":"2024-01-11T09:45:24.148510+0000","last_unstale":"2024-01-11T09:45:24.152685+0000","last_undegraded":"2024-01-11T09:45:24.152685+0000","last_fullsized":"2024-01-11T09:45:24.152685+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:56:51.623335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"213'2","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.173126+0000","last_change":"2024-01-11T09:45:23.148348+0000","last_active":"2024-01-11T09:45:24.173126+0000","last_peered":"2024-01-11T09:45:24.173126+0000","last_clean":"2024-01-11T09:45:24.173126+0000","last_became_active":"2024-01-11T09:45:23.147723+0000","last_became_peered":"2024-01-11T09:45:23.147723+0000","last_unstale":"2024-01-11T09:45:24.173126+0000","last_undegraded":"2024-01-11T09:45:24.173126+0000","last_fullsized":"2024-01-11T09:45:24.173126+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:48:39.924839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170058+0000","last_change":"2024-01-11T09:45:24.162504+0000","last_active":"2024-01-11T09:45:24.170058+0000","last_peered":"2024-01-11T09:45:24.170058+0000","last_clean":"2024-01-11T09:45:24.170058+0000","last_became_active":"2024-01-11T09:45:24.162179+0000","last_became_peered":"2024-01-11T09:45:24.162179+0000","last_unstale":"2024-01-11T09:45:24.170058+0000","last_undegraded":"2024-01-11T09:45:24.170058+0000","last_fullsized":"2024-01-11T09:45:24.170058+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:47:23.503381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.144038+0000","last_change":"2024-01-11T09:45:23.149174+0000","last_active":"2024-01-11T09:45:24.144038+0000","last_peered":"2024-01-11T09:45:24.144038+0000","last_clean":"2024-01-11T09:45:24.144038+0000","last_became_active":"2024-01-11T09:45:23.148798+0000","last_became_peered":"2024-01-11T09:45:23.148798+0000","last_unstale":"2024-01-11T09:45:24.144038+0000","last_undegraded":"2024-01-11T09:45:24.144038+0000","last_fullsized":"2024-01-11T09:45:24.144038+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:52:31.488984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.159270+0000","last_change":"2024-01-11T09:45:24.157902+0000","last_active":"2024-01-11T09:45:24.159270+0000","last_peered":"2024-01-11T09:45:24.159270+0000","last_clean":"2024-01-11T09:45:24.159270+0000","last_became_active":"2024-01-11T09:45:24.157443+0000","last_became_peered":"2024-01-11T09:45:24.157443+0000","last_unstale":"2024-01-11T09:45:24.159270+0000","last_undegraded":"2024-01-11T09:45:24.159270+0000","last_fullsized":"2024-01-11T09:45:24.159270+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:41:10.078741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.143814+0000","last_change":"2024-01-11T09:45:23.148631+0000","last_active":"2024-01-11T09:45:24.143814+0000","last_peered":"2024-01-11T09:45:24.143814+0000","last_clean":"2024-01-11T09:45:24.143814+0000","last_became_active":"2024-01-11T09:45:23.147091+0000","last_became_peered":"2024-01-11T09:45:23.147091+0000","last_unstale":"2024-01-11T09:45:24.143814+0000","last_undegraded":"2024-01-11T09:45:24.143814+0000","last_fullsized":"2024-01-11T09:45:24.143814+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:49:14.796444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.156960+0000","last_change":"2024-01-11T09:45:24.154386+0000","last_active":"2024-01-11T09:45:24.156960+0000","last_peered":"2024-01-11T09:45:24.156960+0000","last_clean":"2024-01-11T09:45:24.156960+0000","last_became_active":"2024-01-11T09:45:24.154017+0000","last_became_peered":"2024-01-11T09:45:24.154017+0000","last_unstale":"2024-01-11T09:45:24.156960+0000","last_undegraded":"2024-01-11T09:45:24.156960+0000","last_fullsized":"2024-01-11T09:45:24.156960+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:33:41.503661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.151333+0000","last_change":"2024-01-11T09:45:23.147642+0000","last_active":"2024-01-11T09:45:24.151333+0000","last_peered":"2024-01-11T09:45:24.151333+0000","last_clean":"2024-01-11T09:45:24.151333+0000","last_became_active":"2024-01-11T09:45:23.147387+0000","last_became_peered":"2024-01-11T09:45:23.147387+0000","last_unstale":"2024-01-11T09:45:24.151333+0000","last_undegraded":"2024-01-11T09:45:24.151333+0000","last_fullsized":"2024-01-11T09:45:24.151333+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:49:33.086505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.168803+0000","last_change":"2024-01-11T09:45:24.158240+0000","last_active":"2024-01-11T09:45:24.168803+0000","last_peered":"2024-01-11T09:45:24.168803+0000","last_clean":"2024-01-11T09:45:24.168803+0000","last_became_active":"2024-01-11T09:45:24.157938+0000","last_became_peered":"2024-01-11T09:45:24.157938+0000","last_unstale":"2024-01-11T09:45:24.168803+0000","last_undegraded":"2024-01-11T09:45:24.168803+0000","last_fullsized":"2024-01-11T09:45:24.168803+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:08:44.293978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.142853+0000","last_change":"2024-01-11T09:45:23.151324+0000","last_active":"2024-01-11T09:45:24.142853+0000","last_peered":"2024-01-11T09:45:24.142853+0000","last_clean":"2024-01-11T09:45:24.142853+0000","last_became_active":"2024-01-11T09:45:23.150822+0000","last_became_peered":"2024-01-11T09:45:23.150822+0000","last_unstale":"2024-01-11T09:45:24.142853+0000","last_undegraded":"2024-01-11T09:45:24.142853+0000","last_fullsized":"2024-01-11T09:45:24.142853+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:39.975619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.158767+0000","last_change":"2024-01-11T09:45:24.157004+0000","last_active":"2024-01-11T09:45:24.158767+0000","last_peered":"2024-01-11T09:45:24.158767+0000","last_clean":"2024-01-11T09:45:24.158767+0000","last_became_active":"2024-01-11T09:45:24.156446+0000","last_became_peered":"2024-01-11T09:45:24.156446+0000","last_unstale":"2024-01-11T09:45:24.158767+0000","last_undegraded":"2024-01-11T09:45:24.158767+0000","last_fullsized":"2024-01-11T09:45:24.158767+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:28:18.397764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.144031+0000","last_change":"2024-01-11T09:45:23.151649+0000","last_active":"2024-01-11T09:45:24.144031+0000","last_peered":"2024-01-11T09:45:24.144031+0000","last_clean":"2024-01-11T09:45:24.144031+0000","last_became_active":"2024-01-11T09:45:23.151399+0000","last_became_peered":"2024-01-11T09:45:23.151399+0000","last_unstale":"2024-01-11T09:45:24.144031+0000","last_undegraded":"2024-01-11T09:45:24.144031+0000","last_fullsized":"2024-01-11T09:45:24.144031+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:02.352990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170415+0000","last_change":"2024-01-11T09:45:24.163398+0000","last_active":"2024-01-11T09:45:24.170415+0000","last_peered":"2024-01-11T09:45:24.170415+0000","last_clean":"2024-01-11T09:45:24.170415+0000","last_became_active":"2024-01-11T09:45:24.159101+0000","last_became_peered":"2024-01-11T09:45:24.159101+0000","last_unstale":"2024-01-11T09:45:24.170415+0000","last_undegraded":"2024-01-11T09:45:24.170415+0000","last_fullsized":"2024-01-11T09:45:24.170415+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:40.983480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.138863+0000","last_change":"2024-01-11T09:45:23.154043+0000","last_active":"2024-01-11T09:45:24.138863+0000","last_peered":"2024-01-11T09:45:24.138863+0000","last_clean":"2024-01-11T09:45:24.138863+0000","last_became_active":"2024-01-11T09:45:23.153529+0000","last_became_peered":"2024-01-11T09:45:23.153529+0000","last_unstale":"2024-01-11T09:45:24.138863+0000","last_undegraded":"2024-01-11T09:45:24.138863+0000","last_fullsized":"2024-01-11T09:45:24.138863+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:48.360444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.156341+0000","last_change":"2024-01-11T09:45:24.153383+0000","last_active":"2024-01-11T09:45:24.156341+0000","last_peered":"2024-01-11T09:45:24.156341+0000","last_clean":"2024-01-11T09:45:24.156341+0000","last_became_active":"2024-01-11T09:45:24.153089+0000","last_became_peered":"2024-01-11T09:45:24.153089+0000","last_unstale":"2024-01-11T09:45:24.156341+0000","last_undegraded":"2024-01-11T09:45:24.156341+0000","last_fullsized":"2024-01-11T09:45:24.156341+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:49:15.804305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.143264+0000","last_change":"2024-01-11T09:45:23.158577+0000","last_active":"2024-01-11T09:45:24.143264+0000","last_peered":"2024-01-11T09:45:24.143264+0000","last_clean":"2024-01-11T09:45:24.143264+0000","last_became_active":"2024-01-11T09:45:23.158304+0000","last_became_peered":"2024-01-11T09:45:23.158304+0000","last_unstale":"2024-01-11T09:45:24.143264+0000","last_undegraded":"2024-01-11T09:45:24.143264+0000","last_fullsized":"2024-01-11T09:45:24.143264+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:59:55.124493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.171012+0000","last_change":"2024-01-11T09:45:24.169860+0000","last_active":"2024-01-11T09:45:24.171012+0000","last_peered":"2024-01-11T09:45:24.171012+0000","last_clean":"2024-01-11T09:45:24.171012+0000","last_became_active":"2024-01-11T09:45:24.169370+0000","last_became_peered":"2024-01-11T09:45:24.169370+0000","last_unstale":"2024-01-11T09:45:24.171012+0000","last_undegraded":"2024-01-11T09:45:24.171012+0000","last_fullsized":"2024-01-11T09:45:24.171012+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:15:38.555186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"213'1","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.165883+0000","last_change":"2024-01-11T09:45:23.147910+0000","last_active":"2024-01-11T09:45:24.165883+0000","last_peered":"2024-01-11T09:45:24.165883+0000","last_clean":"2024-01-11T09:45:24.165883+0000","last_became_active":"2024-01-11T09:45:23.147626+0000","last_became_peered":"2024-01-11T09:45:23.147626+0000","last_unstale":"2024-01-11T09:45:24.165883+0000","last_undegraded":"2024-01-11T09:45:24.165883+0000","last_fullsized":"2024-01-11T09:45:24.165883+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:32:18.360961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"205'366","reported_seq":1199,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.140481+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:45:24.140481+0000","last_peered":"2024-01-11T09:45:24.140481+0000","last_clean":"2024-01-11T09:45:24.140481+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:45:24.140481+0000","last_undegraded":"2024-01-11T09:45:24.140481+0000","last_fullsized":"2024-01-11T09:45:24.140481+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":611,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":611,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":366,"ondisk_log_size":366,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575972,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300492,"kb_used_data":3204,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071348,"statfs":{"total":96636764160,"available":96329060352,"internally_reserved":0,"allocated":3280896,"data_stored":1867384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:44:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70499999999999996,"15min":0.73699999999999999},"min":{"1min":0.51800000000000002,"5min":0.39400000000000002,"15min":0.36599999999999999},"max":{"1min":0.95699999999999996,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.73399999999999999,"15min":0.746},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.91800000000000004,"5min":1.0609999999999999,"15min":1.0669999999999999},"last":0.71399999999999997}]},{"osd":1,"last update":"Thu Jan 11 09:44:27 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.71699999999999997,"15min":0.73199999999999998},"min":{"1min":0.49299999999999999,"5min":0.45600000000000002,"15min":0.41599999999999998},"max":{"1min":0.88200000000000001,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.72999999999999998,"15min":0.74299999999999999},"min":{"1min":0.54100000000000004,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.90400000000000003,"5min":1.0169999999999999,"15min":1.0820000000000001},"last":0.65300000000000002}]},{"osd":2,"last update":"Thu Jan 11 09:44:27 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.746},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.92600000000000005,"5min":0.94399999999999995,"15min":1.0600000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.73699999999999999,"15min":0.753},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44500000000000001},"max":{"1min":0.81299999999999994,"5min":0.96999999999999997,"15min":1.153},"last":0.58499999999999996}]},{"osd":3,"last update":"Thu Jan 11 09:44:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73699999999999999,"15min":0.76900000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94299999999999995,"5min":0.96699999999999997,"15min":1.0660000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.73599999999999999,"15min":0.75700000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93100000000000005,"5min":1,"15min":1.0880000000000001},"last":0.79700000000000004}]},{"osd":4,"last update":"Thu Jan 11 09:44:27 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.60599999999999998,"15min":0.60399999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.107},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.63,"15min":0.63600000000000001},"min":{"1min":0.45700000000000002,"5min":0.44400000000000001,"15min":0.39700000000000002},"max":{"1min":0.995,"5min":0.995,"15min":1.0960000000000001},"last":0.89000000000000001}]},{"osd":5,"last update":"Thu Jan 11 09:44:27 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.61799999999999999,"15min":0.63600000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86699999999999999,"5min":0.90300000000000002,"15min":1.0109999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.67500000000000004,"15min":0.68300000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":1.133},"last":1.1899999999999999}]},{"osd":6,"last update":"Thu Jan 11 09:44:27 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.76100000000000001,"15min":0.77500000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.2110000000000001},"last":1.232},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.75700000000000001,"15min":0.76000000000000001},"min":{"1min":0.55800000000000005,"5min":0.48799999999999999,"15min":0.45800000000000002},"max":{"1min":1.0269999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.95899999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575971,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312204,"kb_used_data":2628,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059636,"statfs":{"total":96636764160,"available":96317067264,"internally_reserved":0,"allocated":2691072,"data_stored":1282434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67800000000000005,"15min":0.71399999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83899999999999997,"5min":0.94199999999999995,"15min":1.508},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.71899999999999997,"15min":0.748},"min":{"1min":0.501,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.92600000000000005,"5min":0.97099999999999997,"15min":1.5589999999999999},"last":0.97099999999999997}]},{"osd":1,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.69199999999999995,"15min":0.74099999999999999},"min":{"1min":0.42799999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84799999999999998,"5min":0.94699999999999995,"15min":1.45},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.70599999999999996,"15min":0.73699999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.79400000000000004,"5min":0.98299999999999998,"15min":1.542},"last":0.95499999999999996}]},{"osd":2,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.72599999999999998,"15min":0.752},"min":{"1min":0.52500000000000002,"5min":0.501,"15min":0.44500000000000001},"max":{"1min":0.85899999999999999,"5min":1.034,"15min":1.5329999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.72399999999999998,"15min":0.75700000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89500000000000002,"5min":0.93999999999999995,"15min":1.647},"last":1}]},{"osd":3,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.747,"15min":0.76500000000000001},"min":{"1min":0.54900000000000004,"5min":0.502,"15min":0.45200000000000001},"max":{"1min":0.88200000000000001,"5min":1.012,"15min":1.6140000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.72699999999999998,"15min":0.77400000000000002},"min":{"1min":0.53600000000000003,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.83499999999999996,"5min":0.96499999999999997,"15min":1.4950000000000001},"last":0.91300000000000003}]},{"osd":4,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86899999999999999,"5min":0.91700000000000004,"15min":1.0309999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61499999999999999,"15min":0.61199999999999999},"min":{"1min":0.42499999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.86199999999999999,"5min":0.94799999999999995,"15min":1.1259999999999999},"last":0.69499999999999995}]},{"osd":5,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64700000000000002,"15min":0.65900000000000003},"min":{"1min":0.45800000000000002,"5min":0.41499999999999998,"15min":0.40000000000000002},"max":{"1min":0.92700000000000005,"5min":0.97899999999999998,"15min":1.369},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.49099999999999999,"5min":0.46700000000000003,"15min":0.45400000000000001},"max":{"1min":0.89400000000000002,"5min":0.999,"15min":1.0960000000000001},"last":0.71999999999999997}]},{"osd":7,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70099999999999996,"15min":0.70799999999999996},"min":{"1min":0.53000000000000003,"5min":0.45100000000000001,"15min":0.44400000000000001},"max":{"1min":0.91900000000000004,"5min":1.089,"15min":1.484},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.76100000000000001,"15min":0.76900000000000002},"min":{"1min":0.48699999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0609999999999999,"5min":1.145,"15min":1.603},"last":0.86099999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308100,"kb_used_data":2620,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063740,"statfs":{"total":96636764160,"available":96321269760,"internally_reserved":0,"allocated":2682880,"data_stored":1275536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52500000000000002,"15min":0.51500000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.315},"max":{"1min":0.66000000000000003,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61399999999999999,"15min":0.622},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.748,"5min":0.85199999999999998,"15min":0.95099999999999996},"last":0.41899999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.51300000000000001,"15min":0.51400000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.64000000000000001,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.37},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60099999999999998,"15min":0.59399999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39400000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.39300000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64400000000000002,"15min":0.65000000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.42699999999999999},"max":{"1min":0.81200000000000006,"5min":0.91600000000000004,"15min":0.96699999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.40400000000000003},"max":{"1min":0.70499999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.45400000000000001}]},{"osd":4,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.81899999999999995,"15min":0.82399999999999995},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.505},"max":{"1min":1.1080000000000001,"5min":1.214,"15min":1.2969999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.879,"5min":0.85999999999999999,"15min":0.86399999999999999},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.61699999999999999},"max":{"1min":1.0129999999999999,"5min":1.1879999999999999,"15min":1.1890000000000001},"last":0.71899999999999997}]},{"osd":5,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83899999999999997,"15min":0.83299999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.093,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.81799999999999995,"15min":0.80700000000000005},"min":{"1min":0.54200000000000004,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":1.0569999999999999,"5min":1.341,"15min":1.341},"last":0.66100000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.84599999999999997,"15min":0.84899999999999998},"min":{"1min":0.67000000000000004,"5min":0.59599999999999997,"15min":0.56799999999999995},"max":{"1min":1.073,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.85099999999999998,"15min":0.85499999999999998},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.55900000000000005},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.82399999999999995}]},{"osd":7,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89600000000000002,"15min":0.89100000000000001},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.143,"5min":1.421,"15min":1.421},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.86499999999999999,"15min":0.85899999999999999},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.54500000000000004},"max":{"1min":1.083,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.74399999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575972,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308688,"kb_used_data":3208,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063152,"statfs":{"total":96636764160,"available":96320667648,"internally_reserved":0,"allocated":3284992,"data_stored":1872338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34499999999999997,"5min":0.34300000000000003,"15min":0.32100000000000001},"max":{"1min":0.754,"5min":0.84699999999999998,"15min":0.875},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.60099999999999998,"15min":0.60199999999999998},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.81599999999999995,"5min":0.86399999999999999,"15min":0.874},"last":0.52000000000000002}]},{"osd":2,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.56799999999999995,"15min":0.57299999999999995},"min":{"1min":0.42499999999999999,"5min":0.40400000000000003,"15min":0.36599999999999999},"max":{"1min":0.79900000000000004,"5min":0.84499999999999997,"15min":0.97499999999999998},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.438,"5min":0.39600000000000002,"15min":0.38500000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.86399999999999999},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54000000000000004,"15min":0.54600000000000004},"min":{"1min":0.42499999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69199999999999995,"5min":0.92600000000000005,"15min":0.95399999999999996},"last":0.432},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64700000000000002,"15min":0.65600000000000003},"min":{"1min":0.46800000000000003,"5min":0.45900000000000002,"15min":0.41999999999999998},"max":{"1min":0.77800000000000002,"5min":0.92600000000000005,"15min":1.0389999999999999},"last":0.61099999999999999}]},{"osd":4,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.83099999999999996,"15min":0.82299999999999995},"min":{"1min":0.60499999999999998,"5min":0.60299999999999998,"15min":0.57399999999999995},"max":{"1min":1.0569999999999999,"5min":1.1779999999999999,"15min":1.304},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.81699999999999995,"15min":0.81000000000000005},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.272,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.77100000000000002}]},{"osd":5,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.84299999999999997,"15min":0.84399999999999997},"min":{"1min":0.67200000000000004,"5min":0.65000000000000002,"15min":0.53900000000000003},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.84599999999999997,"15min":0.84799999999999998},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.153,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.81899999999999995}]},{"osd":6,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83499999999999996,"15min":0.82799999999999996},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.55700000000000005},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.871,"5min":0.872,"15min":0.875},"min":{"1min":0.63800000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.1699999999999999,"5min":1.256,"15min":1.256},"last":0.79800000000000004}]},{"osd":7,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.877,"15min":0.871},"min":{"1min":0.66100000000000003,"5min":0.59099999999999997,"15min":0.58699999999999997},"max":{"1min":1.226,"5min":1.28,"15min":1.28},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.88100000000000001,"15min":0.879},"min":{"1min":0.65300000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.1259999999999999,"5min":1.238,"15min":1.2410000000000001},"last":0.95099999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575971,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312188,"kb_used_data":2612,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059652,"statfs":{"total":96636764160,"available":96317083648,"internally_reserved":0,"allocated":2674688,"data_stored":1275480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.49299999999999999,"15min":0.505},"min":{"1min":0.379,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.627,"5min":0.78800000000000003,"15min":0.85399999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55200000000000005,"15min":0.56499999999999995},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34300000000000003},"max":{"1min":0.71099999999999997,"5min":1.131,"15min":1.6359999999999999},"last":0.61399999999999999}]},{"osd":1,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.38200000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.751,"5min":0.88800000000000001,"15min":0.93000000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60199999999999998,"15min":0.60399999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.40000000000000002},"max":{"1min":0.77400000000000002,"5min":0.97299999999999998,"15min":1.145},"last":0.56699999999999995}]},{"osd":3,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.42999999999999999,"5min":0.377,"15min":0.36199999999999999},"max":{"1min":0.76800000000000002,"5min":0.83099999999999996,"15min":0.96399999999999997},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.66200000000000003},"min":{"1min":0.50600000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.876,"5min":0.92300000000000004,"15min":1.3620000000000001},"last":0.59399999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79000000000000004,"15min":0.80300000000000005},"min":{"1min":0.63400000000000001,"5min":0.57799999999999996,"15min":0.54300000000000004},"max":{"1min":0.97199999999999998,"5min":1.071,"15min":1.1519999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.80100000000000005,"15min":0.80400000000000005},"min":{"1min":0.56200000000000006,"5min":0.505,"15min":0.505},"max":{"1min":1.1439999999999999,"5min":1.161,"15min":1.161},"last":0.92400000000000004}]},{"osd":5,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.85899999999999999,"15min":0.877},"min":{"1min":0.59699999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.228,"5min":1.228,"15min":1.429},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.85799999999999998,"15min":0.86899999999999999},"min":{"1min":0.65900000000000003,"5min":0.64700000000000002,"15min":0.61099999999999999},"max":{"1min":1.0629999999999999,"5min":1.1850000000000001,"15min":1.2170000000000001},"last":0.875}]},{"osd":6,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.81899999999999995,"15min":0.83099999999999996},"min":{"1min":0.59199999999999997,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.091,"5min":1.1659999999999999,"15min":1.4139999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.87,"5min":0.84799999999999998,"15min":0.873},"min":{"1min":0.68300000000000005,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.173,"5min":1.2010000000000001,"15min":1.593},"last":0.67400000000000004}]},{"osd":7,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.88,"15min":0.89800000000000002},"min":{"1min":0.72299999999999998,"5min":0.623,"15min":0.57299999999999995},"max":{"1min":1.111,"5min":1.137,"15min":2.0539999999999998},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.88500000000000001,"15min":0.88600000000000001},"min":{"1min":0.67600000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1919999999999999,"5min":1.2270000000000001,"15min":1.2490000000000001},"last":0.71899999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575971,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299924,"kb_used_data":2636,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071916,"statfs":{"total":96636764160,"available":96329641984,"internally_reserved":0,"allocated":2699264,"data_stored":1282062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:44:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51500000000000001,"15min":0.52300000000000002},"min":{"1min":0.39100000000000001,"5min":0.33900000000000002,"15min":0.314},"max":{"1min":0.64400000000000002,"5min":0.89700000000000002,"15min":0.90000000000000002},"last":0.374},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.58899999999999997,"15min":0.59399999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36099999999999999},"max":{"1min":0.72299999999999998,"5min":1.014,"15min":1.0449999999999999},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:44:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.505,"15min":0.51300000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.70899999999999996,"5min":0.84199999999999997,"15min":1.304},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60299999999999998,"15min":0.61199999999999999},"min":{"1min":0.433,"5min":0.41599999999999998,"15min":0.371},"max":{"1min":0.78600000000000003,"5min":0.84099999999999997,"15min":1.4590000000000001},"last":0.42499999999999999}]},{"osd":2,"last update":"Thu Jan 11 09:44:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59999999999999998,"15min":0.61299999999999999},"min":{"1min":0.44400000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76400000000000001,"5min":0.93700000000000006,"15min":1.04},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65600000000000003,"15min":0.66100000000000003},"min":{"1min":0.51100000000000001,"5min":0.47299999999999998,"15min":0.44700000000000001},"max":{"1min":0.85699999999999998,"5min":0.89800000000000002,"15min":0.92000000000000004},"last":0.48299999999999998}]},{"osd":4,"last update":"Thu Jan 11 09:44:38 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.78700000000000003,"15min":0.79400000000000004},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.50600000000000001},"max":{"1min":1.048,"5min":1.206,"15min":1.206},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.80700000000000005,"15min":0.82399999999999995},"min":{"1min":0.56299999999999994,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.089,"5min":1.171,"15min":1.1830000000000001},"last":0.61699999999999999}]},{"osd":5,"last update":"Thu Jan 11 09:44:38 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.82299999999999995,"15min":0.82799999999999996},"min":{"1min":0.60599999999999998,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.1619999999999999,"5min":1.1950000000000001,"15min":1.234},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.83099999999999996,"15min":0.85299999999999998},"min":{"1min":0.57999999999999996,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.1160000000000001,"5min":1.117,"15min":1.3440000000000001},"last":0.55300000000000005}]},{"osd":6,"last update":"Thu Jan 11 09:44:38 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.84699999999999998,"15min":0.85999999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0249999999999999,"5min":1.175,"15min":1.2170000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.82499999999999996,"15min":0.83199999999999996},"min":{"1min":0.57499999999999996,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.145,"5min":1.181,"15min":1.2190000000000001},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 11 09:44:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.86599999999999999,"15min":0.874},"min":{"1min":0.66300000000000003,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.2030000000000001,"5min":1.264,"15min":1.264},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.88500000000000001,"15min":0.90200000000000002},"min":{"1min":0.57899999999999996,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.071,"5min":1.24,"15min":1.24},"last":0.66600000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575972,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308124,"kb_used_data":2644,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063716,"statfs":{"total":96636764160,"available":96321245184,"internally_reserved":0,"allocated":2707456,"data_stored":1283520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:44:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77600000000000002},"min":{"1min":0.45900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.90900000000000003,"5min":0.98499999999999999,"15min":1.119},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.77700000000000002,"15min":0.77900000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.92300000000000004,"5min":1.853,"15min":1.853},"last":0.78900000000000003}]},{"osd":1,"last update":"Thu Jan 11 09:44:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.748,"15min":0.752},"min":{"1min":0.437,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84699999999999998,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.73399999999999999,"15min":0.73899999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.0369999999999999,"5min":1.639,"15min":1.639},"last":0.67000000000000004}]},{"osd":2,"last update":"Thu Jan 11 09:44:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.75700000000000001,"15min":0.754},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.44700000000000001},"max":{"1min":1.0109999999999999,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.75900000000000001,"15min":0.75600000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.92200000000000004,"5min":1.0249999999999999,"15min":1.0700000000000001},"last":0.73499999999999999}]},{"osd":3,"last update":"Thu Jan 11 09:44:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.74099999999999999,"15min":0.751},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.85799999999999998,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.78800000000000003,"15min":0.78300000000000003},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.93200000000000005,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.69699999999999995}]},{"osd":5,"last update":"Thu Jan 11 09:44:29 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.63300000000000001,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":1.0049999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.65500000000000003,"15min":0.64400000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95799999999999996,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.64000000000000001}]},{"osd":6,"last update":"Thu Jan 11 09:44:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.66900000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0760000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.41099999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.98899999999999999,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.80200000000000005}]},{"osd":7,"last update":"Thu Jan 11 09:44:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.70099999999999996,"15min":0.69899999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83399999999999996,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.76000000000000001,"15min":0.751},"min":{"1min":0.46999999999999997,"5min":0.436,"15min":0.41699999999999998},"max":{"1min":1.1299999999999999,"5min":2.052,"15min":2.052},"last":0.77900000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575972,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312204,"kb_used_data":2628,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059636,"statfs":{"total":96636764160,"available":96317067264,"internally_reserved":0,"allocated":2691072,"data_stored":1282434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.74099999999999999,"15min":0.73299999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":0.95099999999999996,"5min":0.97799999999999998,"15min":1.0029999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71199999999999997,"15min":0.72399999999999998},"min":{"1min":0.52000000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.91100000000000003,"5min":1.048,"15min":1.0549999999999999},"last":0.64700000000000002}]},{"osd":1,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71899999999999997,"15min":0.73499999999999999},"min":{"1min":0.48499999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.88300000000000001,"5min":0.95699999999999996,"15min":0.97699999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.74299999999999999,"15min":0.74199999999999999},"min":{"1min":0.5,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.92600000000000005,"5min":1.004,"15min":1.0269999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.72599999999999998,"15min":0.72999999999999998},"min":{"1min":0.504,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92600000000000005,"5min":1.0269999999999999,"15min":1.028},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.748,"5min":0.74299999999999999,"15min":0.746},"min":{"1min":0.53500000000000003,"5min":0.439,"15min":0.439},"max":{"1min":0.89900000000000002,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.61199999999999999}]},{"osd":3,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.74299999999999999,"15min":0.75700000000000001},"min":{"1min":0.58499999999999996,"5min":0.52800000000000002,"15min":0.49299999999999999},"max":{"1min":0.90200000000000002,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.625},{"interface":"front","average":{"1min":0.751,"5min":0.76400000000000001,"15min":0.76900000000000002},"min":{"1min":0.56999999999999995,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.94499999999999995,"5min":1.004,"15min":1.1000000000000001},"last":0.59999999999999998}]},{"osd":4,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61799999999999999,"15min":0.61099999999999999},"min":{"1min":0.443,"5min":0.41999999999999998,"15min":0.41699999999999998},"max":{"1min":0.91000000000000003,"5min":0.94699999999999995,"15min":0.97199999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.63500000000000001,"15min":0.63300000000000001},"min":{"1min":0.46999999999999997,"5min":0.40200000000000002,"15min":0.375},"max":{"1min":0.85799999999999998,"5min":0.93100000000000005,"15min":0.94499999999999995},"last":0.52500000000000002}]},{"osd":6,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63300000000000001,"15min":0.64100000000000001},"min":{"1min":0.48099999999999998,"5min":0.42099999999999999,"15min":0.40999999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":1.0089999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68300000000000005,"15min":0.68700000000000006},"min":{"1min":0.47299999999999998,"5min":0.46200000000000002,"15min":0.42799999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":1.0229999999999999},"last":0.56699999999999995}]},{"osd":7,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.434},"max":{"1min":0.88600000000000001,"5min":1.0309999999999999,"15min":1.0760000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77700000000000002},"min":{"1min":0.56799999999999995,"5min":0.52700000000000002,"15min":0.47599999999999998},"max":{"1min":0.97099999999999997,"5min":1.0429999999999999,"15min":1.0469999999999999},"last":0.748}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:45:28.552 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:45:28.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.794+0000 7f11cc1ff700 1 Processor -- start 2024-01-11T09:45:28.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.794+0000 7f11cc1ff700 1 -- start start 2024-01-11T09:45:28.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.795+0000 7f11cc1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f11c40c8620 0x7f11c40af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:28.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.795+0000 7f11cc1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11c4069d40 0x7f11c40b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:28.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.795+0000 7f11cc1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c4157390 con 0x7f11c40c89f0 2024-01-11T09:45:28.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.795+0000 7f11cc1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c40adcd0 con 0x7f11c40c8620 2024-01-11T09:45:28.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.795+0000 7f11cc1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c40afeb0 con 0x7f11c4069d40 2024-01-11T09:45:28.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.796+0000 7f11c9f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f11c40c8620 0x7f11c40af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:28.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.796+0000 7f11c9f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f11c40c8620 0x7f11c40af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51836/0 (socket says 172.21.15.27:51836) 2024-01-11T09:45:28.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.796+0000 7f11c9f9b700 1 -- 172.21.15.27:0/3759871635 learned_addr learned my addr 172.21.15.27:0/3759871635 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:28.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.796+0000 7f11c979a700 1 --2- 172.21.15.27:0/3759871635 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11c4069d40 0x7f11c40b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:28.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.796+0000 7f11c9f9b700 1 -- 172.21.15.27:0/3759871635 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11c4069d40 msgr2=0x7f11c40b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:28.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.796+0000 7f11c9f9b700 1 --2- 172.21.15.27:0/3759871635 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11c4069d40 0x7f11c40b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:28.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.796+0000 7f11c9f9b700 1 -- 172.21.15.27:0/3759871635 >> v1:172.21.15.77:6789/0 conn(0x7f11c40c89f0 legacy=0x7f11c4159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:28.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.796+0000 7f11c9f9b700 1 -- 172.21.15.27:0/3759871635 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c406a200 con 0x7f11c40c8620 2024-01-11T09:45:28.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.796+0000 7f11c9f9b700 1 --2- 172.21.15.27:0/3759871635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40c8620 0x7f11c40af970 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f11c00099b0 tx=0x7f11c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c15f34739d5f9aba server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:28.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.797+0000 7f11c8f99700 1 -- 172.21.15.27:0/3759871635 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c0009ea0 con 0x7f11c40c8620 2024-01-11T09:45:28.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.797+0000 7f11c8f99700 1 -- 172.21.15.27:0/3759871635 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c000b860 con 0x7f11c40c8620 2024-01-11T09:45:28.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.797+0000 7f11c8f99700 1 -- 172.21.15.27:0/3759871635 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c000bc00 con 0x7f11c40c8620 2024-01-11T09:45:28.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.797+0000 7f11cc1ff700 1 -- 172.21.15.27:0/3759871635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40c8620 msgr2=0x7f11c40af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:28.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.797+0000 7f11cc1ff700 1 --2- 172.21.15.27:0/3759871635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40c8620 0x7f11c40af970 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f11c00099b0 tx=0x7f11c0009250 comp rx=0 tx=0).stop 2024-01-11T09:45:28.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.797+0000 7f11cc1ff700 1 -- 172.21.15.27:0/3759871635 shutdown_connections 2024-01-11T09:45:28.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.797+0000 7f11cc1ff700 1 --2- 172.21.15.27:0/3759871635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40c8620 0x7f11c40af970 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:28.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.797+0000 7f11cc1ff700 1 --2- 172.21.15.27:0/3759871635 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11c4069d40 0x7f11c40b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:28.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.797+0000 7f11cc1ff700 1 -- 172.21.15.27:0/3759871635 >> 172.21.15.27:0/3759871635 conn(0x7f11c40680e0 msgr2=0x7f11c4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:28.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.797+0000 7f11cc1ff700 1 -- 172.21.15.27:0/3759871635 shutdown_connections 2024-01-11T09:45:28.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.797+0000 7f11cc1ff700 1 -- 172.21.15.27:0/3759871635 wait complete. 2024-01-11T09:45:28.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.799+0000 7f11cc1ff700 1 Processor -- start 2024-01-11T09:45:28.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.799+0000 7f11cc1ff700 1 -- start start 2024-01-11T09:45:28.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.800+0000 7f11cc1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11c4069d40 0x7f11c414d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:28.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.800+0000 7f11cc1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40c8620 0x7f11c4143580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:28.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.800+0000 7f11cc1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c40c89f0 0x7f11c4144e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:28.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.800+0000 7f11cc1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c414c6b0 con 0x7f11c40c8620 2024-01-11T09:45:28.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.800+0000 7f11cc1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c414baa0 con 0x7f11c40c89f0 2024-01-11T09:45:28.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.800+0000 7f11cc1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c414bd80 con 0x7f11c4069d40 2024-01-11T09:45:28.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.800+0000 7f11c979a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40c8620 0x7f11c4143580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:28.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.800+0000 7f11c979a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40c8620 0x7f11c4143580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51850/0 (socket says 172.21.15.27:51850) 2024-01-11T09:45:28.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.800+0000 7f11c979a700 1 -- 172.21.15.27:0/233812662 learned_addr learned my addr 172.21.15.27:0/233812662 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:28.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.800+0000 7f11c9f9b700 1 --2- 172.21.15.27:0/233812662 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11c4069d40 0x7f11c414d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:28.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.800+0000 7f11ca79c700 1 --2- 172.21.15.27:0/233812662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c40c89f0 0x7f11c4144e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:28.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.800+0000 7f11c979a700 1 -- 172.21.15.27:0/233812662 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11c4069d40 msgr2=0x7f11c414d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:28.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.800+0000 7f11c979a700 1 --2- 172.21.15.27:0/233812662 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11c4069d40 0x7f11c414d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:28.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.800+0000 7f11c979a700 1 -- 172.21.15.27:0/233812662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c40c89f0 msgr2=0x7f11c4144e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:28.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.800+0000 7f11c979a700 1 --2- 172.21.15.27:0/233812662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c40c89f0 0x7f11c4144e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:28.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.800+0000 7f11c979a700 1 -- 172.21.15.27:0/233812662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c00094f0 con 0x7f11c40c8620 2024-01-11T09:45:28.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.800+0000 7f11ca79c700 1 --2- 172.21.15.27:0/233812662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c40c89f0 0x7f11c4144e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:28.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.801+0000 7f11c979a700 1 --2- 172.21.15.27:0/233812662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40c8620 0x7f11c4143580 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f11b400ec50 tx=0x7f11b40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:28.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.801+0000 7f11b9ffb700 1 -- 172.21.15.27:0/233812662 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b400c9d0 con 0x7f11c40c8620 2024-01-11T09:45:28.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.801+0000 7f11b9ffb700 1 -- 172.21.15.27:0/233812662 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b400cb30 con 0x7f11c40c8620 2024-01-11T09:45:28.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.801+0000 7f11cc1ff700 1 -- 172.21.15.27:0/233812662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c414c060 con 0x7f11c40c8620 2024-01-11T09:45:28.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.801+0000 7f11b9ffb700 1 -- 172.21.15.27:0/233812662 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b4013470 con 0x7f11c40c8620 2024-01-11T09:45:28.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.801+0000 7f11cc1ff700 1 -- 172.21.15.27:0/233812662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c414ddd0 con 0x7f11c40c8620 2024-01-11T09:45:28.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.802+0000 7f11cc1ff700 1 -- 172.21.15.27:0/233812662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c414c860 con 0x7f11c40c8620 2024-01-11T09:45:28.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.803+0000 7f11b9ffb700 1 -- 172.21.15.27:0/233812662 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f11b4013860 con 0x7f11c40c8620 2024-01-11T09:45:28.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.804+0000 7f11b9ffb700 1 --2- 172.21.15.27:0/233812662 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f11a8066f30 0x7f11a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:28.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.804+0000 7f11b9ffb700 1 -- 172.21.15.27:0/233812662 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f11b4084960 con 0x7f11c40c8620 2024-01-11T09:45:28.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.805+0000 7f11c9f9b700 1 --2- 172.21.15.27:0/233812662 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f11a8066f30 0x7f11a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:28.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.805+0000 7f11c9f9b700 1 --2- 172.21.15.27:0/233812662 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f11a8066f30 0x7f11a80693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f11c00099b0 tx=0x7f11c0017000 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:28.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:28.808+0000 7f11b9ffb700 1 -- 172.21.15.27:0/233812662 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f11c414c860 con 0x7f11c40c8620 2024-01-11T09:45:28.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.007+0000 7f11cc1ff700 1 -- 172.21.15.27:0/233812662 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f11c414c860 con 0x7f11a8066f30 2024-01-11T09:45:29.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.016+0000 7f11b9ffb700 1 -- 172.21.15.27:0/233812662 <== mgr.10380 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289775 (secure 0 0 0) 0x7f11c414c860 con 0x7f11a8066f30 2024-01-11T09:45:29.012 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.020+0000 7f11cc1ff700 1 -- 172.21.15.27:0/233812662 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f11a8066f30 msgr2=0x7f11a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.020+0000 7f11cc1ff700 1 --2- 172.21.15.27:0/233812662 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f11a8066f30 0x7f11a80693e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f11c00099b0 tx=0x7f11c0017000 comp rx=0 tx=0).stop 2024-01-11T09:45:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.020+0000 7f11cc1ff700 1 -- 172.21.15.27:0/233812662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40c8620 msgr2=0x7f11c4143580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.020+0000 7f11cc1ff700 1 --2- 172.21.15.27:0/233812662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40c8620 0x7f11c4143580 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f11b400ec50 tx=0x7f11b40091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.021+0000 7f11cc1ff700 1 -- 172.21.15.27:0/233812662 shutdown_connections 2024-01-11T09:45:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.021+0000 7f11cc1ff700 1 --2- 172.21.15.27:0/233812662 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f11a8066f30 0x7f11a80693e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.021+0000 7f11cc1ff700 1 --2- 172.21.15.27:0/233812662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11c40c89f0 0x7f11c4144e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.021+0000 7f11cc1ff700 1 --2- 172.21.15.27:0/233812662 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11c4069d40 0x7f11c414d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.021+0000 7f11cc1ff700 1 --2- 172.21.15.27:0/233812662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40c8620 0x7f11c4143580 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.021+0000 7f11cc1ff700 1 -- 172.21.15.27:0/233812662 >> 172.21.15.27:0/233812662 conn(0x7f11c40680e0 msgr2=0x7f11c4156600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.021+0000 7f11cc1ff700 1 -- 172.21.15.27:0/233812662 shutdown_connections 2024-01-11T09:45:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.021+0000 7f11cc1ff700 1 -- 172.21.15.27:0/233812662 wait complete. 2024-01-11T09:45:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:45:29.031 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-11T09:45:28.301012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":655,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":404,"ondisk_log_size":404,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":158,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2461924,"kb_used_data":22180,"kb_used_omap":0,"kb_used_meta":2439680,"kb_avail":752512796,"statfs":{"total":773094113280,"available":770573103104,"internally_reserved":0,"allocated":22712320,"data_stored":11421188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2498232320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001270"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170476+0000","last_change":"2024-01-11T09:45:24.164535+0000","last_active":"2024-01-11T09:45:24.170476+0000","last_peered":"2024-01-11T09:45:24.170476+0000","last_clean":"2024-01-11T09:45:24.170476+0000","last_became_active":"2024-01-11T09:45:24.163889+0000","last_became_peered":"2024-01-11T09:45:24.163889+0000","last_unstale":"2024-01-11T09:45:24.170476+0000","last_undegraded":"2024-01-11T09:45:24.170476+0000","last_fullsized":"2024-01-11T09:45:24.170476+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:29:27.569363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.152866+0000","last_change":"2024-01-11T09:45:24.147655+0000","last_active":"2024-01-11T09:45:24.152866+0000","last_peered":"2024-01-11T09:45:24.152866+0000","last_clean":"2024-01-11T09:45:24.152866+0000","last_became_active":"2024-01-11T09:45:24.147267+0000","last_became_peered":"2024-01-11T09:45:24.147267+0000","last_unstale":"2024-01-11T09:45:24.152866+0000","last_undegraded":"2024-01-11T09:45:24.152866+0000","last_fullsized":"2024-01-11T09:45:24.152866+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:14:09.653181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.167348+0000","last_change":"2024-01-11T09:45:24.160833+0000","last_active":"2024-01-11T09:45:24.167348+0000","last_peered":"2024-01-11T09:45:24.167348+0000","last_clean":"2024-01-11T09:45:24.167348+0000","last_became_active":"2024-01-11T09:45:24.160557+0000","last_became_peered":"2024-01-11T09:45:24.160557+0000","last_unstale":"2024-01-11T09:45:24.167348+0000","last_undegraded":"2024-01-11T09:45:24.167348+0000","last_fullsized":"2024-01-11T09:45:24.167348+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:29:13.303845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.152846+0000","last_change":"2024-01-11T09:45:24.149180+0000","last_active":"2024-01-11T09:45:24.152846+0000","last_peered":"2024-01-11T09:45:24.152846+0000","last_clean":"2024-01-11T09:45:24.152846+0000","last_became_active":"2024-01-11T09:45:24.148899+0000","last_became_peered":"2024-01-11T09:45:24.148899+0000","last_unstale":"2024-01-11T09:45:24.152846+0000","last_undegraded":"2024-01-11T09:45:24.152846+0000","last_fullsized":"2024-01-11T09:45:24.152846+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:59:56.132354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.161919+0000","last_change":"2024-01-11T09:45:24.148863+0000","last_active":"2024-01-11T09:45:24.161919+0000","last_peered":"2024-01-11T09:45:24.161919+0000","last_clean":"2024-01-11T09:45:24.161919+0000","last_became_active":"2024-01-11T09:45:24.148533+0000","last_became_peered":"2024-01-11T09:45:24.148533+0000","last_unstale":"2024-01-11T09:45:24.161919+0000","last_undegraded":"2024-01-11T09:45:24.161919+0000","last_fullsized":"2024-01-11T09:45:24.161919+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:20.218042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.162905+0000","last_change":"2024-01-11T09:45:24.153507+0000","last_active":"2024-01-11T09:45:24.162905+0000","last_peered":"2024-01-11T09:45:24.162905+0000","last_clean":"2024-01-11T09:45:24.162905+0000","last_became_active":"2024-01-11T09:45:24.153235+0000","last_became_peered":"2024-01-11T09:45:24.153235+0000","last_unstale":"2024-01-11T09:45:24.162905+0000","last_undegraded":"2024-01-11T09:45:24.162905+0000","last_fullsized":"2024-01-11T09:45:24.162905+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:01:06.710133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170737+0000","last_change":"2024-01-11T09:45:24.164133+0000","last_active":"2024-01-11T09:45:24.170737+0000","last_peered":"2024-01-11T09:45:24.170737+0000","last_clean":"2024-01-11T09:45:24.170737+0000","last_became_active":"2024-01-11T09:45:24.163856+0000","last_became_peered":"2024-01-11T09:45:24.163856+0000","last_unstale":"2024-01-11T09:45:24.170737+0000","last_undegraded":"2024-01-11T09:45:24.170737+0000","last_fullsized":"2024-01-11T09:45:24.170737+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:17:24.413093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.173279+0000","last_change":"2024-01-11T09:45:24.169914+0000","last_active":"2024-01-11T09:45:24.173279+0000","last_peered":"2024-01-11T09:45:24.173279+0000","last_clean":"2024-01-11T09:45:24.173279+0000","last_became_active":"2024-01-11T09:45:24.169438+0000","last_became_peered":"2024-01-11T09:45:24.169438+0000","last_unstale":"2024-01-11T09:45:24.173279+0000","last_undegraded":"2024-01-11T09:45:24.173279+0000","last_fullsized":"2024-01-11T09:45:24.173279+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:43:22.330951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.157010+0000","last_change":"2024-01-11T09:45:24.155318+0000","last_active":"2024-01-11T09:45:24.157010+0000","last_peered":"2024-01-11T09:45:24.157010+0000","last_clean":"2024-01-11T09:45:24.157010+0000","last_became_active":"2024-01-11T09:45:24.153405+0000","last_became_peered":"2024-01-11T09:45:24.153405+0000","last_unstale":"2024-01-11T09:45:24.157010+0000","last_undegraded":"2024-01-11T09:45:24.157010+0000","last_fullsized":"2024-01-11T09:45:24.157010+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:52:32.496845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.161534+0000","last_change":"2024-01-11T09:45:24.152214+0000","last_active":"2024-01-11T09:45:24.161534+0000","last_peered":"2024-01-11T09:45:24.161534+0000","last_clean":"2024-01-11T09:45:24.161534+0000","last_became_active":"2024-01-11T09:45:24.151938+0000","last_became_peered":"2024-01-11T09:45:24.151938+0000","last_unstale":"2024-01-11T09:45:24.161534+0000","last_undegraded":"2024-01-11T09:45:24.161534+0000","last_fullsized":"2024-01-11T09:45:24.161534+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:05.090713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.167366+0000","last_change":"2024-01-11T09:45:24.161300+0000","last_active":"2024-01-11T09:45:24.167366+0000","last_peered":"2024-01-11T09:45:24.167366+0000","last_clean":"2024-01-11T09:45:24.167366+0000","last_became_active":"2024-01-11T09:45:24.160940+0000","last_became_peered":"2024-01-11T09:45:24.160940+0000","last_unstale":"2024-01-11T09:45:24.167366+0000","last_undegraded":"2024-01-11T09:45:24.167366+0000","last_fullsized":"2024-01-11T09:45:24.167366+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:40:06.419759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.147889+0000","last_change":"2024-01-11T09:45:24.146920+0000","last_active":"2024-01-11T09:45:24.147889+0000","last_peered":"2024-01-11T09:45:24.147889+0000","last_clean":"2024-01-11T09:45:24.147889+0000","last_became_active":"2024-01-11T09:45:24.146654+0000","last_became_peered":"2024-01-11T09:45:24.146654+0000","last_unstale":"2024-01-11T09:45:24.147889+0000","last_undegraded":"2024-01-11T09:45:24.147889+0000","last_fullsized":"2024-01-11T09:45:24.147889+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:42:21.543118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.159698+0000","last_change":"2024-01-11T09:45:24.158500+0000","last_active":"2024-01-11T09:45:24.159698+0000","last_peered":"2024-01-11T09:45:24.159698+0000","last_clean":"2024-01-11T09:45:24.159698+0000","last_became_active":"2024-01-11T09:45:24.158185+0000","last_became_peered":"2024-01-11T09:45:24.158185+0000","last_unstale":"2024-01-11T09:45:24.159698+0000","last_undegraded":"2024-01-11T09:45:24.159698+0000","last_fullsized":"2024-01-11T09:45:24.159698+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:52:38.581476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.168782+0000","last_change":"2024-01-11T09:45:24.157215+0000","last_active":"2024-01-11T09:45:24.168782+0000","last_peered":"2024-01-11T09:45:24.168782+0000","last_clean":"2024-01-11T09:45:24.168782+0000","last_became_active":"2024-01-11T09:45:24.156852+0000","last_became_peered":"2024-01-11T09:45:24.156852+0000","last_unstale":"2024-01-11T09:45:24.168782+0000","last_undegraded":"2024-01-11T09:45:24.168782+0000","last_fullsized":"2024-01-11T09:45:24.168782+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:42.716516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.171326+0000","last_change":"2024-01-11T09:45:24.170581+0000","last_active":"2024-01-11T09:45:24.171326+0000","last_peered":"2024-01-11T09:45:24.171326+0000","last_clean":"2024-01-11T09:45:24.171326+0000","last_became_active":"2024-01-11T09:45:24.170233+0000","last_became_peered":"2024-01-11T09:45:24.170233+0000","last_unstale":"2024-01-11T09:45:24.171326+0000","last_undegraded":"2024-01-11T09:45:24.171326+0000","last_fullsized":"2024-01-11T09:45:24.171326+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:50:50.820643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.152444+0000","last_change":"2024-01-11T09:45:24.148308+0000","last_active":"2024-01-11T09:45:24.152444+0000","last_peered":"2024-01-11T09:45:24.152444+0000","last_clean":"2024-01-11T09:45:24.152444+0000","last_became_active":"2024-01-11T09:45:24.148023+0000","last_became_peered":"2024-01-11T09:45:24.148023+0000","last_unstale":"2024-01-11T09:45:24.152444+0000","last_undegraded":"2024-01-11T09:45:24.152444+0000","last_fullsized":"2024-01-11T09:45:24.152444+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:56:51.623335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170453+0000","last_change":"2024-01-11T09:45:24.164171+0000","last_active":"2024-01-11T09:45:24.170453+0000","last_peered":"2024-01-11T09:45:24.170453+0000","last_clean":"2024-01-11T09:45:24.170453+0000","last_became_active":"2024-01-11T09:45:24.163522+0000","last_became_peered":"2024-01-11T09:45:24.163522+0000","last_unstale":"2024-01-11T09:45:24.170453+0000","last_undegraded":"2024-01-11T09:45:24.170453+0000","last_fullsized":"2024-01-11T09:45:24.170453+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:37:20.217033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170239+0000","last_change":"2024-01-11T09:45:24.166373+0000","last_active":"2024-01-11T09:45:24.170239+0000","last_peered":"2024-01-11T09:45:24.170239+0000","last_clean":"2024-01-11T09:45:24.170239+0000","last_became_active":"2024-01-11T09:45:24.166090+0000","last_became_peered":"2024-01-11T09:45:24.166090+0000","last_unstale":"2024-01-11T09:45:24.170239+0000","last_undegraded":"2024-01-11T09:45:24.170239+0000","last_fullsized":"2024-01-11T09:45:24.170239+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:16:58.784096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.162812+0000","last_change":"2024-01-11T09:45:24.153140+0000","last_active":"2024-01-11T09:45:24.162812+0000","last_peered":"2024-01-11T09:45:24.162812+0000","last_clean":"2024-01-11T09:45:24.162812+0000","last_became_active":"2024-01-11T09:45:24.152890+0000","last_became_peered":"2024-01-11T09:45:24.152890+0000","last_unstale":"2024-01-11T09:45:24.162812+0000","last_undegraded":"2024-01-11T09:45:24.162812+0000","last_fullsized":"2024-01-11T09:45:24.162812+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:39.406018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.162866+0000","last_change":"2024-01-11T09:45:24.156832+0000","last_active":"2024-01-11T09:45:24.162866+0000","last_peered":"2024-01-11T09:45:24.162866+0000","last_clean":"2024-01-11T09:45:24.162866+0000","last_became_active":"2024-01-11T09:45:24.156541+0000","last_became_peered":"2024-01-11T09:45:24.156541+0000","last_unstale":"2024-01-11T09:45:24.162866+0000","last_undegraded":"2024-01-11T09:45:24.162866+0000","last_fullsized":"2024-01-11T09:45:24.162866+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:49.368305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.169864+0000","last_change":"2024-01-11T09:45:24.167154+0000","last_active":"2024-01-11T09:45:24.169864+0000","last_peered":"2024-01-11T09:45:24.169864+0000","last_clean":"2024-01-11T09:45:24.169864+0000","last_became_active":"2024-01-11T09:45:24.166862+0000","last_became_peered":"2024-01-11T09:45:24.166862+0000","last_unstale":"2024-01-11T09:45:24.169864+0000","last_undegraded":"2024-01-11T09:45:24.169864+0000","last_fullsized":"2024-01-11T09:45:24.169864+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:15:00.803121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.147395+0000","last_change":"2024-01-11T09:45:24.145558+0000","last_active":"2024-01-11T09:45:24.147395+0000","last_peered":"2024-01-11T09:45:24.147395+0000","last_clean":"2024-01-11T09:45:24.147395+0000","last_became_active":"2024-01-11T09:45:24.143216+0000","last_became_peered":"2024-01-11T09:45:24.143216+0000","last_unstale":"2024-01-11T09:45:24.147395+0000","last_undegraded":"2024-01-11T09:45:24.147395+0000","last_fullsized":"2024-01-11T09:45:24.147395+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:32:19.368822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.161415+0000","last_change":"2024-01-11T09:45:24.158003+0000","last_active":"2024-01-11T09:45:24.161415+0000","last_peered":"2024-01-11T09:45:24.161415+0000","last_clean":"2024-01-11T09:45:24.161415+0000","last_became_active":"2024-01-11T09:45:24.157710+0000","last_became_peered":"2024-01-11T09:45:24.157710+0000","last_unstale":"2024-01-11T09:45:24.161415+0000","last_undegraded":"2024-01-11T09:45:24.161415+0000","last_fullsized":"2024-01-11T09:45:24.161415+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:52:18.543824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.169953+0000","last_change":"2024-01-11T09:45:24.168887+0000","last_active":"2024-01-11T09:45:24.169953+0000","last_peered":"2024-01-11T09:45:24.169953+0000","last_clean":"2024-01-11T09:45:24.169953+0000","last_became_active":"2024-01-11T09:45:24.167138+0000","last_became_peered":"2024-01-11T09:45:24.167138+0000","last_unstale":"2024-01-11T09:45:24.169953+0000","last_undegraded":"2024-01-11T09:45:24.169953+0000","last_fullsized":"2024-01-11T09:45:24.169953+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:43:22.330951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.151363+0000","last_change":"2024-01-11T09:45:24.145841+0000","last_active":"2024-01-11T09:45:24.151363+0000","last_peered":"2024-01-11T09:45:24.151363+0000","last_clean":"2024-01-11T09:45:24.151363+0000","last_became_active":"2024-01-11T09:45:24.145584+0000","last_became_peered":"2024-01-11T09:45:24.145584+0000","last_unstale":"2024-01-11T09:45:24.151363+0000","last_undegraded":"2024-01-11T09:45:24.151363+0000","last_fullsized":"2024-01-11T09:45:24.151363+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:47.970580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.169381+0000","last_change":"2024-01-11T09:45:24.158803+0000","last_active":"2024-01-11T09:45:24.169381+0000","last_peered":"2024-01-11T09:45:24.169381+0000","last_clean":"2024-01-11T09:45:24.169381+0000","last_became_active":"2024-01-11T09:45:24.158533+0000","last_became_peered":"2024-01-11T09:45:24.158533+0000","last_unstale":"2024-01-11T09:45:24.169381+0000","last_undegraded":"2024-01-11T09:45:24.169381+0000","last_fullsized":"2024-01-11T09:45:24.169381+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:57:46.243868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.155273+0000","last_change":"2024-01-11T09:45:24.150865+0000","last_active":"2024-01-11T09:45:24.155273+0000","last_peered":"2024-01-11T09:45:24.155273+0000","last_clean":"2024-01-11T09:45:24.155273+0000","last_became_active":"2024-01-11T09:45:24.148370+0000","last_became_peered":"2024-01-11T09:45:24.148370+0000","last_unstale":"2024-01-11T09:45:24.155273+0000","last_undegraded":"2024-01-11T09:45:24.155273+0000","last_fullsized":"2024-01-11T09:45:24.155273+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:27.774010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.171257+0000","last_change":"2024-01-11T09:45:24.166152+0000","last_active":"2024-01-11T09:45:24.171257+0000","last_peered":"2024-01-11T09:45:24.171257+0000","last_clean":"2024-01-11T09:45:24.171257+0000","last_became_active":"2024-01-11T09:45:24.165889+0000","last_became_peered":"2024-01-11T09:45:24.165889+0000","last_unstale":"2024-01-11T09:45:24.171257+0000","last_undegraded":"2024-01-11T09:45:24.171257+0000","last_fullsized":"2024-01-11T09:45:24.171257+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:01:06.710133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.150341+0000","last_change":"2024-01-11T09:45:24.148965+0000","last_active":"2024-01-11T09:45:24.150341+0000","last_peered":"2024-01-11T09:45:24.150341+0000","last_clean":"2024-01-11T09:45:24.150341+0000","last_became_active":"2024-01-11T09:45:24.148481+0000","last_became_peered":"2024-01-11T09:45:24.148481+0000","last_unstale":"2024-01-11T09:45:24.150341+0000","last_undegraded":"2024-01-11T09:45:24.150341+0000","last_fullsized":"2024-01-11T09:45:24.150341+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:39.406018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.149821+0000","last_change":"2024-01-11T09:45:24.146908+0000","last_active":"2024-01-11T09:45:24.149821+0000","last_peered":"2024-01-11T09:45:24.149821+0000","last_clean":"2024-01-11T09:45:24.149821+0000","last_became_active":"2024-01-11T09:45:24.146543+0000","last_became_peered":"2024-01-11T09:45:24.146543+0000","last_unstale":"2024-01-11T09:45:24.149821+0000","last_undegraded":"2024-01-11T09:45:24.149821+0000","last_fullsized":"2024-01-11T09:45:24.149821+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:40.983480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.160800+0000","last_change":"2024-01-11T09:45:24.155055+0000","last_active":"2024-01-11T09:45:24.160800+0000","last_peered":"2024-01-11T09:45:24.160800+0000","last_clean":"2024-01-11T09:45:24.160800+0000","last_became_active":"2024-01-11T09:45:24.154781+0000","last_became_peered":"2024-01-11T09:45:24.154781+0000","last_unstale":"2024-01-11T09:45:24.160800+0000","last_undegraded":"2024-01-11T09:45:24.160800+0000","last_fullsized":"2024-01-11T09:45:24.160800+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:38:28.859392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170578+0000","last_change":"2024-01-11T09:45:24.167922+0000","last_active":"2024-01-11T09:45:24.170578+0000","last_peered":"2024-01-11T09:45:24.170578+0000","last_clean":"2024-01-11T09:45:24.170578+0000","last_became_active":"2024-01-11T09:45:24.167641+0000","last_became_peered":"2024-01-11T09:45:24.167641+0000","last_unstale":"2024-01-11T09:45:24.170578+0000","last_undegraded":"2024-01-11T09:45:24.170578+0000","last_fullsized":"2024-01-11T09:45:24.170578+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:17.184569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.168556+0000","last_change":"2024-01-11T09:45:24.167467+0000","last_active":"2024-01-11T09:45:24.168556+0000","last_peered":"2024-01-11T09:45:24.168556+0000","last_clean":"2024-01-11T09:45:24.168556+0000","last_became_active":"2024-01-11T09:45:24.167175+0000","last_became_peered":"2024-01-11T09:45:24.167175+0000","last_unstale":"2024-01-11T09:45:24.168556+0000","last_undegraded":"2024-01-11T09:45:24.168556+0000","last_fullsized":"2024-01-11T09:45:24.168556+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:52:36.044653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"213'4","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.168564+0000","last_change":"2024-01-11T09:45:23.150803+0000","last_active":"2024-01-11T09:45:24.168564+0000","last_peered":"2024-01-11T09:45:24.168564+0000","last_clean":"2024-01-11T09:45:24.168564+0000","last_became_active":"2024-01-11T09:45:23.149761+0000","last_became_peered":"2024-01-11T09:45:23.149761+0000","last_unstale":"2024-01-11T09:45:24.168564+0000","last_undegraded":"2024-01-11T09:45:24.168564+0000","last_fullsized":"2024-01-11T09:45:24.168564+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:33:40.495800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.155574+0000","last_change":"2024-01-11T09:45:24.150915+0000","last_active":"2024-01-11T09:45:24.155574+0000","last_peered":"2024-01-11T09:45:24.155574+0000","last_clean":"2024-01-11T09:45:24.155574+0000","last_became_active":"2024-01-11T09:45:24.149015+0000","last_became_peered":"2024-01-11T09:45:24.149015+0000","last_unstale":"2024-01-11T09:45:24.155574+0000","last_undegraded":"2024-01-11T09:45:24.155574+0000","last_fullsized":"2024-01-11T09:45:24.155574+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:14:14.599129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"213'1","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.168906+0000","last_change":"2024-01-11T09:45:23.148530+0000","last_active":"2024-01-11T09:45:24.168906+0000","last_peered":"2024-01-11T09:45:24.168906+0000","last_clean":"2024-01-11T09:45:24.168906+0000","last_became_active":"2024-01-11T09:45:23.148208+0000","last_became_peered":"2024-01-11T09:45:23.148208+0000","last_unstale":"2024-01-11T09:45:24.168906+0000","last_undegraded":"2024-01-11T09:45:24.168906+0000","last_fullsized":"2024-01-11T09:45:24.168906+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:44:55.075364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170535+0000","last_change":"2024-01-11T09:45:24.165254+0000","last_active":"2024-01-11T09:45:24.170535+0000","last_peered":"2024-01-11T09:45:24.170535+0000","last_clean":"2024-01-11T09:45:24.170535+0000","last_became_active":"2024-01-11T09:45:24.163213+0000","last_became_peered":"2024-01-11T09:45:24.163213+0000","last_unstale":"2024-01-11T09:45:24.170535+0000","last_undegraded":"2024-01-11T09:45:24.170535+0000","last_fullsized":"2024-01-11T09:45:24.170535+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:29:21.907384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"213'2","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.173466+0000","last_change":"2024-01-11T09:45:23.144976+0000","last_active":"2024-01-11T09:45:24.173466+0000","last_peered":"2024-01-11T09:45:24.173466+0000","last_clean":"2024-01-11T09:45:24.173466+0000","last_became_active":"2024-01-11T09:45:23.144660+0000","last_became_peered":"2024-01-11T09:45:23.144660+0000","last_unstale":"2024-01-11T09:45:24.173466+0000","last_undegraded":"2024-01-11T09:45:24.173466+0000","last_fullsized":"2024-01-11T09:45:24.173466+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:59:40.579473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.168826+0000","last_change":"2024-01-11T09:45:24.164335+0000","last_active":"2024-01-11T09:45:24.168826+0000","last_peered":"2024-01-11T09:45:24.168826+0000","last_clean":"2024-01-11T09:45:24.168826+0000","last_became_active":"2024-01-11T09:45:24.164063+0000","last_became_peered":"2024-01-11T09:45:24.164063+0000","last_unstale":"2024-01-11T09:45:24.168826+0000","last_undegraded":"2024-01-11T09:45:24.168826+0000","last_fullsized":"2024-01-11T09:45:24.168826+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:42:21.543118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"213'2","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.172142+0000","last_change":"2024-01-11T09:45:23.150649+0000","last_active":"2024-01-11T09:45:24.172142+0000","last_peered":"2024-01-11T09:45:24.172142+0000","last_clean":"2024-01-11T09:45:24.172142+0000","last_became_active":"2024-01-11T09:45:23.150425+0000","last_became_peered":"2024-01-11T09:45:23.150425+0000","last_unstale":"2024-01-11T09:45:24.172142+0000","last_undegraded":"2024-01-11T09:45:24.172142+0000","last_fullsized":"2024-01-11T09:45:24.172142+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:14:08.645320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.150811+0000","last_change":"2024-01-11T09:45:24.148848+0000","last_active":"2024-01-11T09:45:24.150811+0000","last_peered":"2024-01-11T09:45:24.150811+0000","last_clean":"2024-01-11T09:45:24.150811+0000","last_became_active":"2024-01-11T09:45:24.148573+0000","last_became_peered":"2024-01-11T09:45:24.148573+0000","last_unstale":"2024-01-11T09:45:24.150811+0000","last_undegraded":"2024-01-11T09:45:24.150811+0000","last_fullsized":"2024-01-11T09:45:24.150811+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:29:30.529875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"213'1","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.164581+0000","last_change":"2024-01-11T09:45:23.154575+0000","last_active":"2024-01-11T09:45:24.164581+0000","last_peered":"2024-01-11T09:45:24.164581+0000","last_clean":"2024-01-11T09:45:24.164581+0000","last_became_active":"2024-01-11T09:45:23.154313+0000","last_became_peered":"2024-01-11T09:45:23.154313+0000","last_unstale":"2024-01-11T09:45:24.164581+0000","last_undegraded":"2024-01-11T09:45:24.164581+0000","last_fullsized":"2024-01-11T09:45:24.164581+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:14:08.645320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.162735+0000","last_change":"2024-01-11T09:45:24.150698+0000","last_active":"2024-01-11T09:45:24.162735+0000","last_peered":"2024-01-11T09:45:24.162735+0000","last_clean":"2024-01-11T09:45:24.162735+0000","last_became_active":"2024-01-11T09:45:24.149431+0000","last_became_peered":"2024-01-11T09:45:24.149431+0000","last_unstale":"2024-01-11T09:45:24.162735+0000","last_undegraded":"2024-01-11T09:45:24.162735+0000","last_fullsized":"2024-01-11T09:45:24.162735+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:48:06.986946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.143494+0000","last_change":"2024-01-11T09:45:23.145331+0000","last_active":"2024-01-11T09:45:24.143494+0000","last_peered":"2024-01-11T09:45:24.143494+0000","last_clean":"2024-01-11T09:45:24.143494+0000","last_became_active":"2024-01-11T09:45:23.145064+0000","last_became_peered":"2024-01-11T09:45:23.145064+0000","last_unstale":"2024-01-11T09:45:24.143494+0000","last_undegraded":"2024-01-11T09:45:24.143494+0000","last_fullsized":"2024-01-11T09:45:24.143494+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:56:41.726387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.154619+0000","last_change":"2024-01-11T09:45:24.150754+0000","last_active":"2024-01-11T09:45:24.154619+0000","last_peered":"2024-01-11T09:45:24.154619+0000","last_clean":"2024-01-11T09:45:24.154619+0000","last_became_active":"2024-01-11T09:45:24.150485+0000","last_became_peered":"2024-01-11T09:45:24.150485+0000","last_unstale":"2024-01-11T09:45:24.154619+0000","last_undegraded":"2024-01-11T09:45:24.154619+0000","last_fullsized":"2024-01-11T09:45:24.154619+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:32:19.368822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.139464+0000","last_change":"2024-01-11T09:45:23.152843+0000","last_active":"2024-01-11T09:45:24.139464+0000","last_peered":"2024-01-11T09:45:24.139464+0000","last_clean":"2024-01-11T09:45:24.139464+0000","last_became_active":"2024-01-11T09:45:23.152579+0000","last_became_peered":"2024-01-11T09:45:23.152579+0000","last_unstale":"2024-01-11T09:45:24.139464+0000","last_undegraded":"2024-01-11T09:45:24.139464+0000","last_fullsized":"2024-01-11T09:45:24.139464+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:48:05.979085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.159625+0000","last_change":"2024-01-11T09:45:24.155035+0000","last_active":"2024-01-11T09:45:24.159625+0000","last_peered":"2024-01-11T09:45:24.159625+0000","last_clean":"2024-01-11T09:45:24.159625+0000","last_became_active":"2024-01-11T09:45:24.153980+0000","last_became_peered":"2024-01-11T09:45:24.153980+0000","last_unstale":"2024-01-11T09:45:24.159625+0000","last_undegraded":"2024-01-11T09:45:24.159625+0000","last_fullsized":"2024-01-11T09:45:24.159625+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:52:20.741771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.143930+0000","last_change":"2024-01-11T09:45:23.152897+0000","last_active":"2024-01-11T09:45:24.143930+0000","last_peered":"2024-01-11T09:45:24.143930+0000","last_clean":"2024-01-11T09:45:24.143930+0000","last_became_active":"2024-01-11T09:45:23.152620+0000","last_became_peered":"2024-01-11T09:45:23.152620+0000","last_unstale":"2024-01-11T09:45:24.143930+0000","last_undegraded":"2024-01-11T09:45:24.143930+0000","last_fullsized":"2024-01-11T09:45:24.143930+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:02:41.619972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.171300+0000","last_change":"2024-01-11T09:45:24.166531+0000","last_active":"2024-01-11T09:45:24.171300+0000","last_peered":"2024-01-11T09:45:24.171300+0000","last_clean":"2024-01-11T09:45:24.171300+0000","last_became_active":"2024-01-11T09:45:24.164071+0000","last_became_peered":"2024-01-11T09:45:24.164071+0000","last_unstale":"2024-01-11T09:45:24.171300+0000","last_undegraded":"2024-01-11T09:45:24.171300+0000","last_fullsized":"2024-01-11T09:45:24.171300+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:41:10.078741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.139847+0000","last_change":"2024-01-11T09:45:23.150869+0000","last_active":"2024-01-11T09:45:24.139847+0000","last_peered":"2024-01-11T09:45:24.139847+0000","last_clean":"2024-01-11T09:45:24.139847+0000","last_became_active":"2024-01-11T09:45:23.150604+0000","last_became_peered":"2024-01-11T09:45:23.150604+0000","last_unstale":"2024-01-11T09:45:24.139847+0000","last_undegraded":"2024-01-11T09:45:24.139847+0000","last_fullsized":"2024-01-11T09:45:24.139847+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:19.210181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.152867+0000","last_change":"2024-01-11T09:45:24.150019+0000","last_active":"2024-01-11T09:45:24.152867+0000","last_peered":"2024-01-11T09:45:24.152867+0000","last_clean":"2024-01-11T09:45:24.152867+0000","last_became_active":"2024-01-11T09:45:24.148994+0000","last_became_peered":"2024-01-11T09:45:24.148994+0000","last_unstale":"2024-01-11T09:45:24.152867+0000","last_undegraded":"2024-01-11T09:45:24.152867+0000","last_fullsized":"2024-01-11T09:45:24.152867+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:29:01.556722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"213'3","reported_seq":13,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.171173+0000","last_change":"2024-01-11T09:45:23.154355+0000","last_active":"2024-01-11T09:45:24.171173+0000","last_peered":"2024-01-11T09:45:24.171173+0000","last_clean":"2024-01-11T09:45:24.171173+0000","last_became_active":"2024-01-11T09:45:23.153262+0000","last_became_peered":"2024-01-11T09:45:23.153262+0000","last_unstale":"2024-01-11T09:45:24.171173+0000","last_undegraded":"2024-01-11T09:45:24.171173+0000","last_fullsized":"2024-01-11T09:45:24.171173+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:32:15.216292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.171076+0000","last_change":"2024-01-11T09:45:24.169731+0000","last_active":"2024-01-11T09:45:24.171076+0000","last_peered":"2024-01-11T09:45:24.171076+0000","last_clean":"2024-01-11T09:45:24.171076+0000","last_became_active":"2024-01-11T09:45:24.169381+0000","last_became_peered":"2024-01-11T09:45:24.169381+0000","last_unstale":"2024-01-11T09:45:24.171076+0000","last_undegraded":"2024-01-11T09:45:24.171076+0000","last_fullsized":"2024-01-11T09:45:24.171076+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:04:41.533247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"213'6","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.171979+0000","last_change":"2024-01-11T09:45:23.152136+0000","last_active":"2024-01-11T09:45:24.171979+0000","last_peered":"2024-01-11T09:45:24.171979+0000","last_clean":"2024-01-11T09:45:24.171979+0000","last_became_active":"2024-01-11T09:45:23.151819+0000","last_became_peered":"2024-01-11T09:45:23.151819+0000","last_unstale":"2024-01-11T09:45:24.171979+0000","last_undegraded":"2024-01-11T09:45:24.171979+0000","last_fullsized":"2024-01-11T09:45:24.171979+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:41:09.070880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.163042+0000","last_change":"2024-01-11T09:45:24.157881+0000","last_active":"2024-01-11T09:45:24.163042+0000","last_peered":"2024-01-11T09:45:24.163042+0000","last_clean":"2024-01-11T09:45:24.163042+0000","last_became_active":"2024-01-11T09:45:24.157224+0000","last_became_peered":"2024-01-11T09:45:24.157224+0000","last_unstale":"2024-01-11T09:45:24.163042+0000","last_undegraded":"2024-01-11T09:45:24.163042+0000","last_fullsized":"2024-01-11T09:45:24.163042+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:32:16.224153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"213'2","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.165110+0000","last_change":"2024-01-11T09:45:23.155269+0000","last_active":"2024-01-11T09:45:24.165110+0000","last_peered":"2024-01-11T09:45:24.165110+0000","last_clean":"2024-01-11T09:45:24.165110+0000","last_became_active":"2024-01-11T09:45:23.154990+0000","last_became_peered":"2024-01-11T09:45:23.154990+0000","last_unstale":"2024-01-11T09:45:24.165110+0000","last_undegraded":"2024-01-11T09:45:24.165110+0000","last_fullsized":"2024-01-11T09:45:24.165110+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:32:15.216292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.152306+0000","last_change":"2024-01-11T09:45:24.147696+0000","last_active":"2024-01-11T09:45:24.152306+0000","last_peered":"2024-01-11T09:45:24.152306+0000","last_clean":"2024-01-11T09:45:24.152306+0000","last_became_active":"2024-01-11T09:45:24.147422+0000","last_became_peered":"2024-01-11T09:45:24.147422+0000","last_unstale":"2024-01-11T09:45:24.152306+0000","last_undegraded":"2024-01-11T09:45:24.152306+0000","last_fullsized":"2024-01-11T09:45:24.152306+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:32:40.240733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.138676+0000","last_change":"2024-01-11T09:45:23.152794+0000","last_active":"2024-01-11T09:45:24.138676+0000","last_peered":"2024-01-11T09:45:24.138676+0000","last_clean":"2024-01-11T09:45:24.138676+0000","last_became_active":"2024-01-11T09:45:23.152495+0000","last_became_peered":"2024-01-11T09:45:23.152495+0000","last_unstale":"2024-01-11T09:45:24.138676+0000","last_undegraded":"2024-01-11T09:45:24.138676+0000","last_fullsized":"2024-01-11T09:45:24.138676+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:01:05.702272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.171333+0000","last_change":"2024-01-11T09:45:24.166383+0000","last_active":"2024-01-11T09:45:24.171333+0000","last_peered":"2024-01-11T09:45:24.171333+0000","last_clean":"2024-01-11T09:45:24.171333+0000","last_became_active":"2024-01-11T09:45:24.159441+0000","last_became_peered":"2024-01-11T09:45:24.159441+0000","last_unstale":"2024-01-11T09:45:24.171333+0000","last_undegraded":"2024-01-11T09:45:24.171333+0000","last_fullsized":"2024-01-11T09:45:24.171333+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:25:32.122001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.130735+0000","last_change":"2024-01-11T09:45:23.151388+0000","last_active":"2024-01-11T09:45:24.130735+0000","last_peered":"2024-01-11T09:45:24.130735+0000","last_clean":"2024-01-11T09:45:24.130735+0000","last_became_active":"2024-01-11T09:45:23.151127+0000","last_became_peered":"2024-01-11T09:45:23.151127+0000","last_unstale":"2024-01-11T09:45:24.130735+0000","last_undegraded":"2024-01-11T09:45:24.130735+0000","last_fullsized":"2024-01-11T09:45:24.130735+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:43:21.323090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170718+0000","last_change":"2024-01-11T09:45:24.154643+0000","last_active":"2024-01-11T09:45:24.170718+0000","last_peered":"2024-01-11T09:45:24.170718+0000","last_clean":"2024-01-11T09:45:24.170718+0000","last_became_active":"2024-01-11T09:45:24.154293+0000","last_became_peered":"2024-01-11T09:45:24.154293+0000","last_unstale":"2024-01-11T09:45:24.170718+0000","last_undegraded":"2024-01-11T09:45:24.170718+0000","last_fullsized":"2024-01-11T09:45:24.170718+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:00:36.090834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"213'8","reported_seq":18,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.174799+0000","last_change":"2024-01-11T09:45:23.146296+0000","last_active":"2024-01-11T09:45:24.174799+0000","last_peered":"2024-01-11T09:45:24.174799+0000","last_clean":"2024-01-11T09:45:24.174799+0000","last_became_active":"2024-01-11T09:45:23.146026+0000","last_became_peered":"2024-01-11T09:45:23.146026+0000","last_unstale":"2024-01-11T09:45:24.174799+0000","last_undegraded":"2024-01-11T09:45:24.174799+0000","last_fullsized":"2024-01-11T09:45:24.174799+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:38:30.060556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.157400+0000","last_change":"2024-01-11T09:45:24.152987+0000","last_active":"2024-01-11T09:45:24.157400+0000","last_peered":"2024-01-11T09:45:24.157400+0000","last_clean":"2024-01-11T09:45:24.157400+0000","last_became_active":"2024-01-11T09:45:24.152677+0000","last_became_peered":"2024-01-11T09:45:24.152677+0000","last_unstale":"2024-01-11T09:45:24.157400+0000","last_undegraded":"2024-01-11T09:45:24.157400+0000","last_fullsized":"2024-01-11T09:45:24.157400+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:03.360851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"213'1","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.167919+0000","last_change":"2024-01-11T09:45:23.152322+0000","last_active":"2024-01-11T09:45:24.167919+0000","last_peered":"2024-01-11T09:45:24.167919+0000","last_clean":"2024-01-11T09:45:24.167919+0000","last_became_active":"2024-01-11T09:45:23.152006+0000","last_became_peered":"2024-01-11T09:45:23.152006+0000","last_unstale":"2024-01-11T09:45:24.167919+0000","last_undegraded":"2024-01-11T09:45:24.167919+0000","last_fullsized":"2024-01-11T09:45:24.167919+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:46.962719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.143849+0000","last_change":"2024-01-11T09:45:24.140693+0000","last_active":"2024-01-11T09:45:24.143849+0000","last_peered":"2024-01-11T09:45:24.143849+0000","last_clean":"2024-01-11T09:45:24.143849+0000","last_became_active":"2024-01-11T09:45:24.140376+0000","last_became_peered":"2024-01-11T09:45:24.140376+0000","last_unstale":"2024-01-11T09:45:24.143849+0000","last_undegraded":"2024-01-11T09:45:24.143849+0000","last_fullsized":"2024-01-11T09:45:24.143849+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:52:17.873609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"213'2","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170711+0000","last_change":"2024-01-11T09:45:23.147655+0000","last_active":"2024-01-11T09:45:24.170711+0000","last_peered":"2024-01-11T09:45:24.170711+0000","last_clean":"2024-01-11T09:45:24.170711+0000","last_became_active":"2024-01-11T09:45:23.147203+0000","last_became_peered":"2024-01-11T09:45:23.147203+0000","last_unstale":"2024-01-11T09:45:24.170711+0000","last_undegraded":"2024-01-11T09:45:24.170711+0000","last_fullsized":"2024-01-11T09:45:24.170711+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:16:31.626739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.166957+0000","last_change":"2024-01-11T09:45:24.159728+0000","last_active":"2024-01-11T09:45:24.166957+0000","last_peered":"2024-01-11T09:45:24.166957+0000","last_clean":"2024-01-11T09:45:24.166957+0000","last_became_active":"2024-01-11T09:45:24.159489+0000","last_became_peered":"2024-01-11T09:45:24.159489+0000","last_unstale":"2024-01-11T09:45:24.166957+0000","last_undegraded":"2024-01-11T09:45:24.166957+0000","last_fullsized":"2024-01-11T09:45:24.166957+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:21:50.096457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"213'1","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.167618+0000","last_change":"2024-01-11T09:45:23.153536+0000","last_active":"2024-01-11T09:45:24.167618+0000","last_peered":"2024-01-11T09:45:24.167618+0000","last_clean":"2024-01-11T09:45:24.167618+0000","last_became_active":"2024-01-11T09:45:23.152838+0000","last_became_peered":"2024-01-11T09:45:23.152838+0000","last_unstale":"2024-01-11T09:45:24.167618+0000","last_undegraded":"2024-01-11T09:45:24.167618+0000","last_fullsized":"2024-01-11T09:45:24.167618+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:14:59.795260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.153884+0000","last_change":"2024-01-11T09:45:24.149619+0000","last_active":"2024-01-11T09:45:24.153884+0000","last_peered":"2024-01-11T09:45:24.153884+0000","last_clean":"2024-01-11T09:45:24.153884+0000","last_became_active":"2024-01-11T09:45:24.148547+0000","last_became_peered":"2024-01-11T09:45:24.148547+0000","last_unstale":"2024-01-11T09:45:24.153884+0000","last_undegraded":"2024-01-11T09:45:24.153884+0000","last_fullsized":"2024-01-11T09:45:24.153884+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:28:18.397764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.152091+0000","last_change":"2024-01-11T09:45:23.155273+0000","last_active":"2024-01-11T09:45:24.152091+0000","last_peered":"2024-01-11T09:45:24.152091+0000","last_clean":"2024-01-11T09:45:24.152091+0000","last_became_active":"2024-01-11T09:45:23.155023+0000","last_became_peered":"2024-01-11T09:45:23.155023+0000","last_unstale":"2024-01-11T09:45:24.152091+0000","last_undegraded":"2024-01-11T09:45:24.152091+0000","last_fullsized":"2024-01-11T09:45:24.152091+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:03:59.152882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.151452+0000","last_change":"2024-01-11T09:45:24.145805+0000","last_active":"2024-01-11T09:45:24.151452+0000","last_peered":"2024-01-11T09:45:24.151452+0000","last_clean":"2024-01-11T09:45:24.151452+0000","last_became_active":"2024-01-11T09:45:24.145497+0000","last_became_peered":"2024-01-11T09:45:24.145497+0000","last_unstale":"2024-01-11T09:45:24.151452+0000","last_undegraded":"2024-01-11T09:45:24.151452+0000","last_fullsized":"2024-01-11T09:45:24.151452+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:27.774010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"213'2","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.172984+0000","last_change":"2024-01-11T09:45:23.143823+0000","last_active":"2024-01-11T09:45:24.172984+0000","last_peered":"2024-01-11T09:45:24.172984+0000","last_clean":"2024-01-11T09:45:24.172984+0000","last_became_active":"2024-01-11T09:45:23.141580+0000","last_became_peered":"2024-01-11T09:45:23.141580+0000","last_unstale":"2024-01-11T09:45:24.172984+0000","last_undegraded":"2024-01-11T09:45:24.172984+0000","last_fullsized":"2024-01-11T09:45:24.172984+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:19.210181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.172210+0000","last_change":"2024-01-11T09:45:24.168028+0000","last_active":"2024-01-11T09:45:24.172210+0000","last_peered":"2024-01-11T09:45:24.172210+0000","last_clean":"2024-01-11T09:45:24.172210+0000","last_became_active":"2024-01-11T09:45:24.167726+0000","last_became_peered":"2024-01-11T09:45:24.167726+0000","last_unstale":"2024-01-11T09:45:24.172210+0000","last_undegraded":"2024-01-11T09:45:24.172210+0000","last_fullsized":"2024-01-11T09:45:24.172210+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:48:06.986946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.152607+0000","last_change":"2024-01-11T09:45:23.154111+0000","last_active":"2024-01-11T09:45:24.152607+0000","last_peered":"2024-01-11T09:45:24.152607+0000","last_clean":"2024-01-11T09:45:24.152607+0000","last_became_active":"2024-01-11T09:45:23.151742+0000","last_became_peered":"2024-01-11T09:45:23.151742+0000","last_unstale":"2024-01-11T09:45:24.152607+0000","last_undegraded":"2024-01-11T09:45:24.152607+0000","last_fullsized":"2024-01-11T09:45:24.152607+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:38.398157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.150062+0000","last_change":"2024-01-11T09:45:24.144965+0000","last_active":"2024-01-11T09:45:24.150062+0000","last_peered":"2024-01-11T09:45:24.150062+0000","last_clean":"2024-01-11T09:45:24.150062+0000","last_became_active":"2024-01-11T09:45:24.143991+0000","last_became_peered":"2024-01-11T09:45:24.143991+0000","last_unstale":"2024-01-11T09:45:24.150062+0000","last_undegraded":"2024-01-11T09:45:24.150062+0000","last_fullsized":"2024-01-11T09:45:24.150062+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:56:42.734248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.152877+0000","last_change":"2024-01-11T09:45:23.155951+0000","last_active":"2024-01-11T09:45:24.152877+0000","last_peered":"2024-01-11T09:45:24.152877+0000","last_clean":"2024-01-11T09:45:24.152877+0000","last_became_active":"2024-01-11T09:45:23.155673+0000","last_became_peered":"2024-01-11T09:45:23.155673+0000","last_unstale":"2024-01-11T09:45:24.152877+0000","last_undegraded":"2024-01-11T09:45:24.152877+0000","last_fullsized":"2024-01-11T09:45:24.152877+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:52:37.573615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.156483+0000","last_change":"2024-01-11T09:45:24.150105+0000","last_active":"2024-01-11T09:45:24.156483+0000","last_peered":"2024-01-11T09:45:24.156483+0000","last_clean":"2024-01-11T09:45:24.156483+0000","last_became_active":"2024-01-11T09:45:24.149005+0000","last_became_peered":"2024-01-11T09:45:24.149005+0000","last_unstale":"2024-01-11T09:45:24.156483+0000","last_undegraded":"2024-01-11T09:45:24.156483+0000","last_fullsized":"2024-01-11T09:45:24.156483+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:25:27.316660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.143649+0000","last_change":"2024-01-11T09:45:23.154230+0000","last_active":"2024-01-11T09:45:24.143649+0000","last_peered":"2024-01-11T09:45:24.143649+0000","last_clean":"2024-01-11T09:45:24.143649+0000","last_became_active":"2024-01-11T09:45:23.153324+0000","last_became_peered":"2024-01-11T09:45:23.153324+0000","last_unstale":"2024-01-11T09:45:24.143649+0000","last_undegraded":"2024-01-11T09:45:24.143649+0000","last_fullsized":"2024-01-11T09:45:24.143649+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:52:16.865748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.152685+0000","last_change":"2024-01-11T09:45:24.149162+0000","last_active":"2024-01-11T09:45:24.152685+0000","last_peered":"2024-01-11T09:45:24.152685+0000","last_clean":"2024-01-11T09:45:24.152685+0000","last_became_active":"2024-01-11T09:45:24.148510+0000","last_became_peered":"2024-01-11T09:45:24.148510+0000","last_unstale":"2024-01-11T09:45:24.152685+0000","last_undegraded":"2024-01-11T09:45:24.152685+0000","last_fullsized":"2024-01-11T09:45:24.152685+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:56:51.623335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"213'2","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.173126+0000","last_change":"2024-01-11T09:45:23.148348+0000","last_active":"2024-01-11T09:45:24.173126+0000","last_peered":"2024-01-11T09:45:24.173126+0000","last_clean":"2024-01-11T09:45:24.173126+0000","last_became_active":"2024-01-11T09:45:23.147723+0000","last_became_peered":"2024-01-11T09:45:23.147723+0000","last_unstale":"2024-01-11T09:45:24.173126+0000","last_undegraded":"2024-01-11T09:45:24.173126+0000","last_fullsized":"2024-01-11T09:45:24.173126+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:48:39.924839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170058+0000","last_change":"2024-01-11T09:45:24.162504+0000","last_active":"2024-01-11T09:45:24.170058+0000","last_peered":"2024-01-11T09:45:24.170058+0000","last_clean":"2024-01-11T09:45:24.170058+0000","last_became_active":"2024-01-11T09:45:24.162179+0000","last_became_peered":"2024-01-11T09:45:24.162179+0000","last_unstale":"2024-01-11T09:45:24.170058+0000","last_undegraded":"2024-01-11T09:45:24.170058+0000","last_fullsized":"2024-01-11T09:45:24.170058+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:47:23.503381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.144038+0000","last_change":"2024-01-11T09:45:23.149174+0000","last_active":"2024-01-11T09:45:24.144038+0000","last_peered":"2024-01-11T09:45:24.144038+0000","last_clean":"2024-01-11T09:45:24.144038+0000","last_became_active":"2024-01-11T09:45:23.148798+0000","last_became_peered":"2024-01-11T09:45:23.148798+0000","last_unstale":"2024-01-11T09:45:24.144038+0000","last_undegraded":"2024-01-11T09:45:24.144038+0000","last_fullsized":"2024-01-11T09:45:24.144038+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:52:31.488984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.159270+0000","last_change":"2024-01-11T09:45:24.157902+0000","last_active":"2024-01-11T09:45:24.159270+0000","last_peered":"2024-01-11T09:45:24.159270+0000","last_clean":"2024-01-11T09:45:24.159270+0000","last_became_active":"2024-01-11T09:45:24.157443+0000","last_became_peered":"2024-01-11T09:45:24.157443+0000","last_unstale":"2024-01-11T09:45:24.159270+0000","last_undegraded":"2024-01-11T09:45:24.159270+0000","last_fullsized":"2024-01-11T09:45:24.159270+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:41:10.078741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.143814+0000","last_change":"2024-01-11T09:45:23.148631+0000","last_active":"2024-01-11T09:45:24.143814+0000","last_peered":"2024-01-11T09:45:24.143814+0000","last_clean":"2024-01-11T09:45:24.143814+0000","last_became_active":"2024-01-11T09:45:23.147091+0000","last_became_peered":"2024-01-11T09:45:23.147091+0000","last_unstale":"2024-01-11T09:45:24.143814+0000","last_undegraded":"2024-01-11T09:45:24.143814+0000","last_fullsized":"2024-01-11T09:45:24.143814+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:49:14.796444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.156960+0000","last_change":"2024-01-11T09:45:24.154386+0000","last_active":"2024-01-11T09:45:24.156960+0000","last_peered":"2024-01-11T09:45:24.156960+0000","last_clean":"2024-01-11T09:45:24.156960+0000","last_became_active":"2024-01-11T09:45:24.154017+0000","last_became_peered":"2024-01-11T09:45:24.154017+0000","last_unstale":"2024-01-11T09:45:24.156960+0000","last_undegraded":"2024-01-11T09:45:24.156960+0000","last_fullsized":"2024-01-11T09:45:24.156960+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:33:41.503661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.151333+0000","last_change":"2024-01-11T09:45:23.147642+0000","last_active":"2024-01-11T09:45:24.151333+0000","last_peered":"2024-01-11T09:45:24.151333+0000","last_clean":"2024-01-11T09:45:24.151333+0000","last_became_active":"2024-01-11T09:45:23.147387+0000","last_became_peered":"2024-01-11T09:45:23.147387+0000","last_unstale":"2024-01-11T09:45:24.151333+0000","last_undegraded":"2024-01-11T09:45:24.151333+0000","last_fullsized":"2024-01-11T09:45:24.151333+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:49:33.086505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.168803+0000","last_change":"2024-01-11T09:45:24.158240+0000","last_active":"2024-01-11T09:45:24.168803+0000","last_peered":"2024-01-11T09:45:24.168803+0000","last_clean":"2024-01-11T09:45:24.168803+0000","last_became_active":"2024-01-11T09:45:24.157938+0000","last_became_peered":"2024-01-11T09:45:24.157938+0000","last_unstale":"2024-01-11T09:45:24.168803+0000","last_undegraded":"2024-01-11T09:45:24.168803+0000","last_fullsized":"2024-01-11T09:45:24.168803+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:08:44.293978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.142853+0000","last_change":"2024-01-11T09:45:23.151324+0000","last_active":"2024-01-11T09:45:24.142853+0000","last_peered":"2024-01-11T09:45:24.142853+0000","last_clean":"2024-01-11T09:45:24.142853+0000","last_became_active":"2024-01-11T09:45:23.150822+0000","last_became_peered":"2024-01-11T09:45:23.150822+0000","last_unstale":"2024-01-11T09:45:24.142853+0000","last_undegraded":"2024-01-11T09:45:24.142853+0000","last_fullsized":"2024-01-11T09:45:24.142853+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:39.975619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.158767+0000","last_change":"2024-01-11T09:45:24.157004+0000","last_active":"2024-01-11T09:45:24.158767+0000","last_peered":"2024-01-11T09:45:24.158767+0000","last_clean":"2024-01-11T09:45:24.158767+0000","last_became_active":"2024-01-11T09:45:24.156446+0000","last_became_peered":"2024-01-11T09:45:24.156446+0000","last_unstale":"2024-01-11T09:45:24.158767+0000","last_undegraded":"2024-01-11T09:45:24.158767+0000","last_fullsized":"2024-01-11T09:45:24.158767+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:28:18.397764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.144031+0000","last_change":"2024-01-11T09:45:23.151649+0000","last_active":"2024-01-11T09:45:24.144031+0000","last_peered":"2024-01-11T09:45:24.144031+0000","last_clean":"2024-01-11T09:45:24.144031+0000","last_became_active":"2024-01-11T09:45:23.151399+0000","last_became_peered":"2024-01-11T09:45:23.151399+0000","last_unstale":"2024-01-11T09:45:24.144031+0000","last_undegraded":"2024-01-11T09:45:24.144031+0000","last_fullsized":"2024-01-11T09:45:24.144031+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:02.352990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.170415+0000","last_change":"2024-01-11T09:45:24.163398+0000","last_active":"2024-01-11T09:45:24.170415+0000","last_peered":"2024-01-11T09:45:24.170415+0000","last_clean":"2024-01-11T09:45:24.170415+0000","last_became_active":"2024-01-11T09:45:24.159101+0000","last_became_peered":"2024-01-11T09:45:24.159101+0000","last_unstale":"2024-01-11T09:45:24.170415+0000","last_undegraded":"2024-01-11T09:45:24.170415+0000","last_fullsized":"2024-01-11T09:45:24.170415+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:40.983480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.138863+0000","last_change":"2024-01-11T09:45:23.154043+0000","last_active":"2024-01-11T09:45:24.138863+0000","last_peered":"2024-01-11T09:45:24.138863+0000","last_clean":"2024-01-11T09:45:24.138863+0000","last_became_active":"2024-01-11T09:45:23.153529+0000","last_became_peered":"2024-01-11T09:45:23.153529+0000","last_unstale":"2024-01-11T09:45:24.138863+0000","last_undegraded":"2024-01-11T09:45:24.138863+0000","last_fullsized":"2024-01-11T09:45:24.138863+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:48.360444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.156341+0000","last_change":"2024-01-11T09:45:24.153383+0000","last_active":"2024-01-11T09:45:24.156341+0000","last_peered":"2024-01-11T09:45:24.156341+0000","last_clean":"2024-01-11T09:45:24.156341+0000","last_became_active":"2024-01-11T09:45:24.153089+0000","last_became_peered":"2024-01-11T09:45:24.153089+0000","last_unstale":"2024-01-11T09:45:24.156341+0000","last_undegraded":"2024-01-11T09:45:24.156341+0000","last_fullsized":"2024-01-11T09:45:24.156341+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:49:15.804305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.143264+0000","last_change":"2024-01-11T09:45:23.158577+0000","last_active":"2024-01-11T09:45:24.143264+0000","last_peered":"2024-01-11T09:45:24.143264+0000","last_clean":"2024-01-11T09:45:24.143264+0000","last_became_active":"2024-01-11T09:45:23.158304+0000","last_became_peered":"2024-01-11T09:45:23.158304+0000","last_unstale":"2024-01-11T09:45:24.143264+0000","last_undegraded":"2024-01-11T09:45:24.143264+0000","last_fullsized":"2024-01-11T09:45:24.143264+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:59:55.124493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.171012+0000","last_change":"2024-01-11T09:45:24.169860+0000","last_active":"2024-01-11T09:45:24.171012+0000","last_peered":"2024-01-11T09:45:24.171012+0000","last_clean":"2024-01-11T09:45:24.171012+0000","last_became_active":"2024-01-11T09:45:24.169370+0000","last_became_peered":"2024-01-11T09:45:24.169370+0000","last_unstale":"2024-01-11T09:45:24.171012+0000","last_undegraded":"2024-01-11T09:45:24.171012+0000","last_fullsized":"2024-01-11T09:45:24.171012+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:15:38.555186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"213'1","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.165883+0000","last_change":"2024-01-11T09:45:23.147910+0000","last_active":"2024-01-11T09:45:24.165883+0000","last_peered":"2024-01-11T09:45:24.165883+0000","last_clean":"2024-01-11T09:45:24.165883+0000","last_became_active":"2024-01-11T09:45:23.147626+0000","last_became_peered":"2024-01-11T09:45:23.147626+0000","last_unstale":"2024-01-11T09:45:24.165883+0000","last_undegraded":"2024-01-11T09:45:24.165883+0000","last_fullsized":"2024-01-11T09:45:24.165883+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:32:18.360961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"205'366","reported_seq":1199,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-01-11T09:45:24.140481+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:45:24.140481+0000","last_peered":"2024-01-11T09:45:24.140481+0000","last_clean":"2024-01-11T09:45:24.140481+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:45:24.140481+0000","last_undegraded":"2024-01-11T09:45:24.140481+0000","last_fullsized":"2024-01-11T09:45:24.140481+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":611,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":611,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":366,"ondisk_log_size":366,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575972,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300492,"kb_used_data":3204,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071348,"statfs":{"total":96636764160,"available":96329060352,"internally_reserved":0,"allocated":3280896,"data_stored":1867384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:44:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70499999999999996,"15min":0.73699999999999999},"min":{"1min":0.51800000000000002,"5min":0.39400000000000002,"15min":0.36599999999999999},"max":{"1min":0.95699999999999996,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.73399999999999999,"15min":0.746},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.91800000000000004,"5min":1.0609999999999999,"15min":1.0669999999999999},"last":0.71399999999999997}]},{"osd":1,"last update":"Thu Jan 11 09:44:27 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.71699999999999997,"15min":0.73199999999999998},"min":{"1min":0.49299999999999999,"5min":0.45600000000000002,"15min":0.41599999999999998},"max":{"1min":0.88200000000000001,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.72999999999999998,"15min":0.74299999999999999},"min":{"1min":0.54100000000000004,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.90400000000000003,"5min":1.0169999999999999,"15min":1.0820000000000001},"last":0.65300000000000002}]},{"osd":2,"last update":"Thu Jan 11 09:44:27 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.746},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.92600000000000005,"5min":0.94399999999999995,"15min":1.0600000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.73699999999999999,"15min":0.753},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44500000000000001},"max":{"1min":0.81299999999999994,"5min":0.96999999999999997,"15min":1.153},"last":0.58499999999999996}]},{"osd":3,"last update":"Thu Jan 11 09:44:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73699999999999999,"15min":0.76900000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94299999999999995,"5min":0.96699999999999997,"15min":1.0660000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.73599999999999999,"15min":0.75700000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.93100000000000005,"5min":1,"15min":1.0880000000000001},"last":0.79700000000000004}]},{"osd":4,"last update":"Thu Jan 11 09:44:27 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.60599999999999998,"15min":0.60399999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.107},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.63,"15min":0.63600000000000001},"min":{"1min":0.45700000000000002,"5min":0.44400000000000001,"15min":0.39700000000000002},"max":{"1min":0.995,"5min":0.995,"15min":1.0960000000000001},"last":0.89000000000000001}]},{"osd":5,"last update":"Thu Jan 11 09:44:27 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.61799999999999999,"15min":0.63600000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86699999999999999,"5min":0.90300000000000002,"15min":1.0109999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.67500000000000004,"15min":0.68300000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":1.133},"last":1.1899999999999999}]},{"osd":6,"last update":"Thu Jan 11 09:44:27 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.76100000000000001,"15min":0.77500000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.2110000000000001},"last":1.232},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.75700000000000001,"15min":0.76000000000000001},"min":{"1min":0.55800000000000005,"5min":0.48799999999999999,"15min":0.45800000000000002},"max":{"1min":1.0269999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.95899999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575971,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312204,"kb_used_data":2628,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059636,"statfs":{"total":96636764160,"available":96317067264,"internally_reserved":0,"allocated":2691072,"data_stored":1282434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67800000000000005,"15min":0.71399999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83899999999999997,"5min":0.94199999999999995,"15min":1.508},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.71899999999999997,"15min":0.748},"min":{"1min":0.501,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.92600000000000005,"5min":0.97099999999999997,"15min":1.5589999999999999},"last":0.97099999999999997}]},{"osd":1,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.69199999999999995,"15min":0.74099999999999999},"min":{"1min":0.42799999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84799999999999998,"5min":0.94699999999999995,"15min":1.45},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.70599999999999996,"15min":0.73699999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.79400000000000004,"5min":0.98299999999999998,"15min":1.542},"last":0.95499999999999996}]},{"osd":2,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.72599999999999998,"15min":0.752},"min":{"1min":0.52500000000000002,"5min":0.501,"15min":0.44500000000000001},"max":{"1min":0.85899999999999999,"5min":1.034,"15min":1.5329999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.72399999999999998,"15min":0.75700000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89500000000000002,"5min":0.93999999999999995,"15min":1.647},"last":1}]},{"osd":3,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.747,"15min":0.76500000000000001},"min":{"1min":0.54900000000000004,"5min":0.502,"15min":0.45200000000000001},"max":{"1min":0.88200000000000001,"5min":1.012,"15min":1.6140000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.72699999999999998,"15min":0.77400000000000002},"min":{"1min":0.53600000000000003,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.83499999999999996,"5min":0.96499999999999997,"15min":1.4950000000000001},"last":0.91300000000000003}]},{"osd":4,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86899999999999999,"5min":0.91700000000000004,"15min":1.0309999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61499999999999999,"15min":0.61199999999999999},"min":{"1min":0.42499999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.86199999999999999,"5min":0.94799999999999995,"15min":1.1259999999999999},"last":0.69499999999999995}]},{"osd":5,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64700000000000002,"15min":0.65900000000000003},"min":{"1min":0.45800000000000002,"5min":0.41499999999999998,"15min":0.40000000000000002},"max":{"1min":0.92700000000000005,"5min":0.97899999999999998,"15min":1.369},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.49099999999999999,"5min":0.46700000000000003,"15min":0.45400000000000001},"max":{"1min":0.89400000000000002,"5min":0.999,"15min":1.0960000000000001},"last":0.71999999999999997}]},{"osd":7,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70099999999999996,"15min":0.70799999999999996},"min":{"1min":0.53000000000000003,"5min":0.45100000000000001,"15min":0.44400000000000001},"max":{"1min":0.91900000000000004,"5min":1.089,"15min":1.484},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.76100000000000001,"15min":0.76900000000000002},"min":{"1min":0.48699999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0609999999999999,"5min":1.145,"15min":1.603},"last":0.86099999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308100,"kb_used_data":2620,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063740,"statfs":{"total":96636764160,"available":96321269760,"internally_reserved":0,"allocated":2682880,"data_stored":1275536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52500000000000002,"15min":0.51500000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.315},"max":{"1min":0.66000000000000003,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61399999999999999,"15min":0.622},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.748,"5min":0.85199999999999998,"15min":0.95099999999999996},"last":0.41899999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.51300000000000001,"15min":0.51400000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.64000000000000001,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.37},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60099999999999998,"15min":0.59399999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39400000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.39300000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64400000000000002,"15min":0.65000000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.42699999999999999},"max":{"1min":0.81200000000000006,"5min":0.91600000000000004,"15min":0.96699999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.40400000000000003},"max":{"1min":0.70499999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.45400000000000001}]},{"osd":4,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.81899999999999995,"15min":0.82399999999999995},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.505},"max":{"1min":1.1080000000000001,"5min":1.214,"15min":1.2969999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.879,"5min":0.85999999999999999,"15min":0.86399999999999999},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.61699999999999999},"max":{"1min":1.0129999999999999,"5min":1.1879999999999999,"15min":1.1890000000000001},"last":0.71899999999999997}]},{"osd":5,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83899999999999997,"15min":0.83299999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.093,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.81799999999999995,"15min":0.80700000000000005},"min":{"1min":0.54200000000000004,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":1.0569999999999999,"5min":1.341,"15min":1.341},"last":0.66100000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.84599999999999997,"15min":0.84899999999999998},"min":{"1min":0.67000000000000004,"5min":0.59599999999999997,"15min":0.56799999999999995},"max":{"1min":1.073,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.85099999999999998,"15min":0.85499999999999998},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.55900000000000005},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.82399999999999995}]},{"osd":7,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89600000000000002,"15min":0.89100000000000001},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.143,"5min":1.421,"15min":1.421},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.86499999999999999,"15min":0.85899999999999999},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.54500000000000004},"max":{"1min":1.083,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.74399999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575972,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308688,"kb_used_data":3208,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063152,"statfs":{"total":96636764160,"available":96320667648,"internally_reserved":0,"allocated":3284992,"data_stored":1872338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34499999999999997,"5min":0.34300000000000003,"15min":0.32100000000000001},"max":{"1min":0.754,"5min":0.84699999999999998,"15min":0.875},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.60099999999999998,"15min":0.60199999999999998},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.81599999999999995,"5min":0.86399999999999999,"15min":0.874},"last":0.52000000000000002}]},{"osd":2,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.56799999999999995,"15min":0.57299999999999995},"min":{"1min":0.42499999999999999,"5min":0.40400000000000003,"15min":0.36599999999999999},"max":{"1min":0.79900000000000004,"5min":0.84499999999999997,"15min":0.97499999999999998},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.438,"5min":0.39600000000000002,"15min":0.38500000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.86399999999999999},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54000000000000004,"15min":0.54600000000000004},"min":{"1min":0.42499999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69199999999999995,"5min":0.92600000000000005,"15min":0.95399999999999996},"last":0.432},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64700000000000002,"15min":0.65600000000000003},"min":{"1min":0.46800000000000003,"5min":0.45900000000000002,"15min":0.41999999999999998},"max":{"1min":0.77800000000000002,"5min":0.92600000000000005,"15min":1.0389999999999999},"last":0.61099999999999999}]},{"osd":4,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.83099999999999996,"15min":0.82299999999999995},"min":{"1min":0.60499999999999998,"5min":0.60299999999999998,"15min":0.57399999999999995},"max":{"1min":1.0569999999999999,"5min":1.1779999999999999,"15min":1.304},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.81699999999999995,"15min":0.81000000000000005},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.272,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.77100000000000002}]},{"osd":5,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.84299999999999997,"15min":0.84399999999999997},"min":{"1min":0.67200000000000004,"5min":0.65000000000000002,"15min":0.53900000000000003},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.84599999999999997,"15min":0.84799999999999998},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.153,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.81899999999999995}]},{"osd":6,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83499999999999996,"15min":0.82799999999999996},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.55700000000000005},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.871,"5min":0.872,"15min":0.875},"min":{"1min":0.63800000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.1699999999999999,"5min":1.256,"15min":1.256},"last":0.79800000000000004}]},{"osd":7,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.877,"15min":0.871},"min":{"1min":0.66100000000000003,"5min":0.59099999999999997,"15min":0.58699999999999997},"max":{"1min":1.226,"5min":1.28,"15min":1.28},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.88100000000000001,"15min":0.879},"min":{"1min":0.65300000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.1259999999999999,"5min":1.238,"15min":1.2410000000000001},"last":0.95099999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575971,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312188,"kb_used_data":2612,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059652,"statfs":{"total":96636764160,"available":96317083648,"internally_reserved":0,"allocated":2674688,"data_stored":1275480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.49299999999999999,"15min":0.505},"min":{"1min":0.379,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.627,"5min":0.78800000000000003,"15min":0.85399999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55200000000000005,"15min":0.56499999999999995},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34300000000000003},"max":{"1min":0.71099999999999997,"5min":1.131,"15min":1.6359999999999999},"last":0.61399999999999999}]},{"osd":1,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.38200000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.751,"5min":0.88800000000000001,"15min":0.93000000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60199999999999998,"15min":0.60399999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.40000000000000002},"max":{"1min":0.77400000000000002,"5min":0.97299999999999998,"15min":1.145},"last":0.56699999999999995}]},{"osd":3,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.42999999999999999,"5min":0.377,"15min":0.36199999999999999},"max":{"1min":0.76800000000000002,"5min":0.83099999999999996,"15min":0.96399999999999997},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.66200000000000003},"min":{"1min":0.50600000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.876,"5min":0.92300000000000004,"15min":1.3620000000000001},"last":0.59399999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79000000000000004,"15min":0.80300000000000005},"min":{"1min":0.63400000000000001,"5min":0.57799999999999996,"15min":0.54300000000000004},"max":{"1min":0.97199999999999998,"5min":1.071,"15min":1.1519999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.80100000000000005,"15min":0.80400000000000005},"min":{"1min":0.56200000000000006,"5min":0.505,"15min":0.505},"max":{"1min":1.1439999999999999,"5min":1.161,"15min":1.161},"last":0.92400000000000004}]},{"osd":5,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.85899999999999999,"15min":0.877},"min":{"1min":0.59699999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.228,"5min":1.228,"15min":1.429},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.85799999999999998,"15min":0.86899999999999999},"min":{"1min":0.65900000000000003,"5min":0.64700000000000002,"15min":0.61099999999999999},"max":{"1min":1.0629999999999999,"5min":1.1850000000000001,"15min":1.2170000000000001},"last":0.875}]},{"osd":6,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.81899999999999995,"15min":0.83099999999999996},"min":{"1min":0.59199999999999997,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.091,"5min":1.1659999999999999,"15min":1.4139999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.87,"5min":0.84799999999999998,"15min":0.873},"min":{"1min":0.68300000000000005,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.173,"5min":1.2010000000000001,"15min":1.593},"last":0.67400000000000004}]},{"osd":7,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.88,"15min":0.89800000000000002},"min":{"1min":0.72299999999999998,"5min":0.623,"15min":0.57299999999999995},"max":{"1min":1.111,"5min":1.137,"15min":2.0539999999999998},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.88500000000000001,"15min":0.88600000000000001},"min":{"1min":0.67600000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1919999999999999,"5min":1.2270000000000001,"15min":1.2490000000000001},"last":0.71899999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575971,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299924,"kb_used_data":2636,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071916,"statfs":{"total":96636764160,"available":96329641984,"internally_reserved":0,"allocated":2699264,"data_stored":1282062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:44:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51500000000000001,"15min":0.52300000000000002},"min":{"1min":0.39100000000000001,"5min":0.33900000000000002,"15min":0.314},"max":{"1min":0.64400000000000002,"5min":0.89700000000000002,"15min":0.90000000000000002},"last":0.374},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.58899999999999997,"15min":0.59399999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36099999999999999},"max":{"1min":0.72299999999999998,"5min":1.014,"15min":1.0449999999999999},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:44:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.505,"15min":0.51300000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.70899999999999996,"5min":0.84199999999999997,"15min":1.304},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60299999999999998,"15min":0.61199999999999999},"min":{"1min":0.433,"5min":0.41599999999999998,"15min":0.371},"max":{"1min":0.78600000000000003,"5min":0.84099999999999997,"15min":1.4590000000000001},"last":0.42499999999999999}]},{"osd":2,"last update":"Thu Jan 11 09:44:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59999999999999998,"15min":0.61299999999999999},"min":{"1min":0.44400000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76400000000000001,"5min":0.93700000000000006,"15min":1.04},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65600000000000003,"15min":0.66100000000000003},"min":{"1min":0.51100000000000001,"5min":0.47299999999999998,"15min":0.44700000000000001},"max":{"1min":0.85699999999999998,"5min":0.89800000000000002,"15min":0.92000000000000004},"last":0.48299999999999998}]},{"osd":4,"last update":"Thu Jan 11 09:44:38 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.78700000000000003,"15min":0.79400000000000004},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.50600000000000001},"max":{"1min":1.048,"5min":1.206,"15min":1.206},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.80700000000000005,"15min":0.82399999999999995},"min":{"1min":0.56299999999999994,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.089,"5min":1.171,"15min":1.1830000000000001},"last":0.61699999999999999}]},{"osd":5,"last update":"Thu Jan 11 09:44:38 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.82299999999999995,"15min":0.82799999999999996},"min":{"1min":0.60599999999999998,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.1619999999999999,"5min":1.1950000000000001,"15min":1.234},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.83099999999999996,"15min":0.85299999999999998},"min":{"1min":0.57999999999999996,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.1160000000000001,"5min":1.117,"15min":1.3440000000000001},"last":0.55300000000000005}]},{"osd":6,"last update":"Thu Jan 11 09:44:38 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.84699999999999998,"15min":0.85999999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0249999999999999,"5min":1.175,"15min":1.2170000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.82499999999999996,"15min":0.83199999999999996},"min":{"1min":0.57499999999999996,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.145,"5min":1.181,"15min":1.2190000000000001},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 11 09:44:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.86599999999999999,"15min":0.874},"min":{"1min":0.66300000000000003,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.2030000000000001,"5min":1.264,"15min":1.264},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.88500000000000001,"15min":0.90200000000000002},"min":{"1min":0.57899999999999996,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.071,"5min":1.24,"15min":1.24},"last":0.66600000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575972,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308124,"kb_used_data":2644,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063716,"statfs":{"total":96636764160,"available":96321245184,"internally_reserved":0,"allocated":2707456,"data_stored":1283520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:44:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77600000000000002},"min":{"1min":0.45900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.90900000000000003,"5min":0.98499999999999999,"15min":1.119},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.77700000000000002,"15min":0.77900000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.92300000000000004,"5min":1.853,"15min":1.853},"last":0.78900000000000003}]},{"osd":1,"last update":"Thu Jan 11 09:44:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.748,"15min":0.752},"min":{"1min":0.437,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84699999999999998,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.73399999999999999,"15min":0.73899999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.0369999999999999,"5min":1.639,"15min":1.639},"last":0.67000000000000004}]},{"osd":2,"last update":"Thu Jan 11 09:44:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.75700000000000001,"15min":0.754},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.44700000000000001},"max":{"1min":1.0109999999999999,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.75900000000000001,"15min":0.75600000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.92200000000000004,"5min":1.0249999999999999,"15min":1.0700000000000001},"last":0.73499999999999999}]},{"osd":3,"last update":"Thu Jan 11 09:44:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.74099999999999999,"15min":0.751},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.85799999999999998,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.78800000000000003,"15min":0.78300000000000003},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.93200000000000005,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.69699999999999995}]},{"osd":5,"last update":"Thu Jan 11 09:44:29 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.63300000000000001,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":1.0049999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.65500000000000003,"15min":0.64400000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95799999999999996,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.64000000000000001}]},{"osd":6,"last update":"Thu Jan 11 09:44:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.66900000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0760000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.41099999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.98899999999999999,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.80200000000000005}]},{"osd":7,"last update":"Thu Jan 11 09:44:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.70099999999999996,"15min":0.69899999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83399999999999996,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.76000000000000001,"15min":0.751},"min":{"1min":0.46999999999999997,"5min":0.436,"15min":0.41699999999999998},"max":{"1min":1.1299999999999999,"5min":2.052,"15min":2.052},"last":0.77900000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575972,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312204,"kb_used_data":2628,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059636,"statfs":{"total":96636764160,"available":96317067264,"internally_reserved":0,"allocated":2691072,"data_stored":1282434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.74099999999999999,"15min":0.73299999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":0.95099999999999996,"5min":0.97799999999999998,"15min":1.0029999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71199999999999997,"15min":0.72399999999999998},"min":{"1min":0.52000000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.91100000000000003,"5min":1.048,"15min":1.0549999999999999},"last":0.64700000000000002}]},{"osd":1,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71899999999999997,"15min":0.73499999999999999},"min":{"1min":0.48499999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.88300000000000001,"5min":0.95699999999999996,"15min":0.97699999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.74299999999999999,"15min":0.74199999999999999},"min":{"1min":0.5,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.92600000000000005,"5min":1.004,"15min":1.0269999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.72599999999999998,"15min":0.72999999999999998},"min":{"1min":0.504,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92600000000000005,"5min":1.0269999999999999,"15min":1.028},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.748,"5min":0.74299999999999999,"15min":0.746},"min":{"1min":0.53500000000000003,"5min":0.439,"15min":0.439},"max":{"1min":0.89900000000000002,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.61199999999999999}]},{"osd":3,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.74299999999999999,"15min":0.75700000000000001},"min":{"1min":0.58499999999999996,"5min":0.52800000000000002,"15min":0.49299999999999999},"max":{"1min":0.90200000000000002,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.625},{"interface":"front","average":{"1min":0.751,"5min":0.76400000000000001,"15min":0.76900000000000002},"min":{"1min":0.56999999999999995,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.94499999999999995,"5min":1.004,"15min":1.1000000000000001},"last":0.59999999999999998}]},{"osd":4,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61799999999999999,"15min":0.61099999999999999},"min":{"1min":0.443,"5min":0.41999999999999998,"15min":0.41699999999999998},"max":{"1min":0.91000000000000003,"5min":0.94699999999999995,"15min":0.97199999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.63500000000000001,"15min":0.63300000000000001},"min":{"1min":0.46999999999999997,"5min":0.40200000000000002,"15min":0.375},"max":{"1min":0.85799999999999998,"5min":0.93100000000000005,"15min":0.94499999999999995},"last":0.52500000000000002}]},{"osd":6,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63300000000000001,"15min":0.64100000000000001},"min":{"1min":0.48099999999999998,"5min":0.42099999999999999,"15min":0.40999999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":1.0089999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68300000000000005,"15min":0.68700000000000006},"min":{"1min":0.47299999999999998,"5min":0.46200000000000002,"15min":0.42799999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":1.0229999999999999},"last":0.56699999999999995}]},{"osd":7,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.434},"max":{"1min":0.88600000000000001,"5min":1.0309999999999999,"15min":1.0760000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77700000000000002},"min":{"1min":0.56799999999999995,"5min":0.52700000000000002,"15min":0.47599999999999998},"max":{"1min":0.97099999999999997,"5min":1.0429999999999999,"15min":1.0469999999999999},"last":0.748}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:45:29.036 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-11T09:45:29.036 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-11T09:45:29.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.287+0000 7f42157ff700 1 Processor -- start 2024-01-11T09:45:29.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.287+0000 7f42157ff700 1 -- start start 2024-01-11T09:45:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.288+0000 7f42157ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f42100b03e0 0x7f42100b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.288+0000 7f42157ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f42100b29c0 0x7f4210159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.288+0000 7f42157ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42100cd730 con 0x7f42100b07b0 2024-01-11T09:45:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.288+0000 7f42157ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42100b2de0 con 0x7f42100b03e0 2024-01-11T09:45:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.288+0000 7f42157ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42100ccb20 con 0x7f42100b29c0 2024-01-11T09:45:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.289+0000 7f420e7fc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f42100b07b0 0x7f4210159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:53214/0 (socket says 172.21.15.27:53214) 2024-01-11T09:45:29.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.289+0000 7f420e7fc700 1 -- 172.21.15.27:0/3462266213 learned_addr learned my addr 172.21.15.27:0/3462266213 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:29.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.289+0000 7f420effd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f42100b03e0 0x7f42100b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:29.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.289+0000 7f420f7fe700 1 --2- 172.21.15.27:0/3462266213 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f42100b29c0 0x7f4210159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:29.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.289+0000 7f420dffb700 1 -- 172.21.15.27:0/3462266213 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507583621 0 0) 0x7f42100cd730 con 0x7f42100b07b0 2024-01-11T09:45:29.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.290+0000 7f420f7fe700 1 -- 172.21.15.27:0/3462266213 >> v2:172.21.15.77:3300/0 conn(0x7f42100b03e0 msgr2=0x7f42100b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:29.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.290+0000 7f420f7fe700 1 --2- 172.21.15.27:0/3462266213 >> v2:172.21.15.77:3300/0 conn(0x7f42100b03e0 0x7f42100b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:29.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.290+0000 7f420f7fe700 1 -- 172.21.15.27:0/3462266213 >> v1:172.21.15.77:6789/0 conn(0x7f42100b07b0 legacy=0x7f4210159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:29.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.290+0000 7f420f7fe700 1 -- 172.21.15.27:0/3462266213 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42100cce00 con 0x7f42100b29c0 2024-01-11T09:45:29.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.290+0000 7f420effd700 1 --2- 172.21.15.27:0/3462266213 >> v2:172.21.15.77:3300/0 conn(0x7f42100b03e0 0x7f42100b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:29.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.290+0000 7f420f7fe700 1 --2- 172.21.15.27:0/3462266213 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f42100b29c0 0x7f4210159910 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4208009e60 tx=0x7f4208009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c208a84484c437e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:29.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.290+0000 7f420dffb700 1 -- 172.21.15.27:0/3462266213 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420800e070 con 0x7f42100b29c0 2024-01-11T09:45:29.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.290+0000 7f420dffb700 1 -- 172.21.15.27:0/3462266213 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f420800bb60 con 0x7f42100b29c0 2024-01-11T09:45:29.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.290+0000 7f420dffb700 1 -- 172.21.15.27:0/3462266213 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4208012430 con 0x7f42100b29c0 2024-01-11T09:45:29.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.291+0000 7f42157ff700 1 -- 172.21.15.27:0/3462266213 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f42100b29c0 msgr2=0x7f4210159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:29.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.291+0000 7f42157ff700 1 --2- 172.21.15.27:0/3462266213 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f42100b29c0 0x7f4210159910 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4208009e60 tx=0x7f4208009780 comp rx=0 tx=0).stop 2024-01-11T09:45:29.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.291+0000 7f42157ff700 1 -- 172.21.15.27:0/3462266213 shutdown_connections 2024-01-11T09:45:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.291+0000 7f42157ff700 1 --2- 172.21.15.27:0/3462266213 >> v2:172.21.15.77:3300/0 conn(0x7f42100b03e0 0x7f42100b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.291+0000 7f42157ff700 1 --2- 172.21.15.27:0/3462266213 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f42100b29c0 0x7f4210159910 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.291+0000 7f42157ff700 1 -- 172.21.15.27:0/3462266213 >> 172.21.15.27:0/3462266213 conn(0x7f421006ed40 msgr2=0x7f421006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.291+0000 7f42157ff700 1 -- 172.21.15.27:0/3462266213 shutdown_connections 2024-01-11T09:45:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.291+0000 7f42157ff700 1 -- 172.21.15.27:0/3462266213 wait complete. 2024-01-11T09:45:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.292+0000 7f42157ff700 1 Processor -- start 2024-01-11T09:45:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.293+0000 7f42157ff700 1 -- start start 2024-01-11T09:45:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.293+0000 7f42157ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f42100b03e0 0x7f42100c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.294+0000 7f42157ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42100b07b0 0x7f42100ca480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.294+0000 7f42157ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42100b29c0 0x7f42100a84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.294+0000 7f42157ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42100a7440 con 0x7f42100b07b0 2024-01-11T09:45:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.294+0000 7f42157ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42100a7720 con 0x7f42100b29c0 2024-01-11T09:45:29.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.294+0000 7f42157ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42100a7a00 con 0x7f42100b03e0 2024-01-11T09:45:29.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.294+0000 7f420e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42100b07b0 0x7f42100ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:29.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.294+0000 7f420effd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f42100b03e0 0x7f42100c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:29.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.294+0000 7f420effd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f42100b03e0 0x7f42100c69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:34880/0 (socket says 172.21.15.27:34880) 2024-01-11T09:45:29.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.294+0000 7f420effd700 1 -- 172.21.15.27:0/2486422441 learned_addr learned my addr 172.21.15.27:0/2486422441 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:29.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.294+0000 7f420f7fe700 1 --2- 172.21.15.27:0/2486422441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42100b29c0 0x7f42100a84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:29.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.295+0000 7f420e7fc700 1 -- 172.21.15.27:0/2486422441 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f42100b03e0 msgr2=0x7f42100c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:29.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.295+0000 7f420e7fc700 1 --2- 172.21.15.27:0/2486422441 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f42100b03e0 0x7f42100c69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:29.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.295+0000 7f420e7fc700 1 -- 172.21.15.27:0/2486422441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42100b29c0 msgr2=0x7f42100a84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:29.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.295+0000 7f420e7fc700 1 --2- 172.21.15.27:0/2486422441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42100b29c0 0x7f42100a84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:29.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.295+0000 7f420e7fc700 1 -- 172.21.15.27:0/2486422441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4200009420 con 0x7f42100b07b0 2024-01-11T09:45:29.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.295+0000 7f420e7fc700 1 --2- 172.21.15.27:0/2486422441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42100b07b0 0x7f42100ca480 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f42040011e0 tx=0x7f4204007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:29.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.295+0000 7f41f6ffd700 1 -- 172.21.15.27:0/2486422441 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4204005b30 con 0x7f42100b07b0 2024-01-11T09:45:29.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.295+0000 7f41f6ffd700 1 -- 172.21.15.27:0/2486422441 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4204005c90 con 0x7f42100b07b0 2024-01-11T09:45:29.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.295+0000 7f42157ff700 1 -- 172.21.15.27:0/2486422441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4208009a20 con 0x7f42100b07b0 2024-01-11T09:45:29.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.295+0000 7f41f6ffd700 1 -- 172.21.15.27:0/2486422441 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4204007320 con 0x7f42100b07b0 2024-01-11T09:45:29.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.295+0000 7f42157ff700 1 -- 172.21.15.27:0/2486422441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42100a7fc0 con 0x7f42100b07b0 2024-01-11T09:45:29.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.296+0000 7f42157ff700 1 -- 172.21.15.27:0/2486422441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42100a81e0 con 0x7f42100b07b0 2024-01-11T09:45:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.297+0000 7f41f6ffd700 1 -- 172.21.15.27:0/2486422441 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f420401b3b0 con 0x7f42100b07b0 2024-01-11T09:45:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.298+0000 7f41f6ffd700 1 --2- 172.21.15.27:0/2486422441 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f41fc066f30 0x7f41fc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.298+0000 7f41f6ffd700 1 -- 172.21.15.27:0/2486422441 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f42040839d0 con 0x7f42100b07b0 2024-01-11T09:45:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.299+0000 7f420effd700 1 --2- 172.21.15.27:0/2486422441 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f41fc066f30 0x7f41fc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:29.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.299+0000 7f420effd700 1 --2- 172.21.15.27:0/2486422441 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f41fc066f30 0x7f41fc0693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4200011310 tx=0x7f42000091a0 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:29.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.302+0000 7f41f6ffd700 1 -- 172.21.15.27:0/2486422441 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4204050b30 con 0x7f42100b07b0 2024-01-11T09:45:29.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.585+0000 7f42157ff700 1 -- 172.21.15.27:0/2486422441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f421004c3c0 con 0x7f42100b07b0 2024-01-11T09:45:29.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.591+0000 7f41f6ffd700 1 -- 172.21.15.27:0/2486422441 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v50) v1 ==== 190+0+0 (secure 0 0 0) 0x7f42040170e0 con 0x7f42100b07b0 2024-01-11T09:45:29.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.594+0000 7f42157ff700 1 -- 172.21.15.27:0/2486422441 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f41fc066f30 msgr2=0x7f41fc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:29.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.594+0000 7f42157ff700 1 --2- 172.21.15.27:0/2486422441 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f41fc066f30 0x7f41fc0693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4200011310 tx=0x7f42000091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:29.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.594+0000 7f42157ff700 1 -- 172.21.15.27:0/2486422441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42100b07b0 msgr2=0x7f42100ca480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:29.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.594+0000 7f42157ff700 1 --2- 172.21.15.27:0/2486422441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42100b07b0 0x7f42100ca480 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f42040011e0 tx=0x7f4204007d30 comp rx=0 tx=0).stop 2024-01-11T09:45:29.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.594+0000 7f42157ff700 1 -- 172.21.15.27:0/2486422441 shutdown_connections 2024-01-11T09:45:29.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.594+0000 7f42157ff700 1 --2- 172.21.15.27:0/2486422441 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f41fc066f30 0x7f41fc0693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:29.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.594+0000 7f42157ff700 1 --2- 172.21.15.27:0/2486422441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42100b29c0 0x7f42100a84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:29.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.594+0000 7f42157ff700 1 --2- 172.21.15.27:0/2486422441 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f42100b03e0 0x7f42100c69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:29.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.594+0000 7f42157ff700 1 --2- 172.21.15.27:0/2486422441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42100b07b0 0x7f42100ca480 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:29.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.594+0000 7f42157ff700 1 -- 172.21.15.27:0/2486422441 >> 172.21.15.27:0/2486422441 conn(0x7f421006ed40 msgr2=0x7f42101595f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:29.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.594+0000 7f42157ff700 1 -- 172.21.15.27:0/2486422441 shutdown_connections 2024-01-11T09:45:29.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.594+0000 7f42157ff700 1 -- 172.21.15.27:0/2486422441 wait complete. 2024-01-11T09:45:29.588 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-11T09:45:29.604 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:45:29.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.811+0000 7f7b075ff700 1 Processor -- start 2024-01-11T09:45:29.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.811+0000 7f7b075ff700 1 -- start start 2024-01-11T09:45:29.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.813+0000 7f7b075ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7b001592d0 0x7f7b0015a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:29.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.813+0000 7f7b075ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b00157e30 con 0x7f7b00159db0 2024-01-11T09:45:29.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.813+0000 7f7b075ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b00157500 con 0x7f7b001599e0 2024-01-11T09:45:29.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.813+0000 7f7b075ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b001577e0 con 0x7f7b001592d0 2024-01-11T09:45:29.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.813+0000 7f7b04b9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7b00159db0 0x7f7b0014fc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55970/0 (socket says 172.21.15.27:55970) 2024-01-11T09:45:29.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.813+0000 7f7b04b9a700 1 -- 172.21.15.27:0/770284376 learned_addr learned my addr 172.21.15.27:0/770284376 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:29.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.813+0000 7f7b05b9c700 1 --2- 172.21.15.27:0/770284376 >> v2:172.21.15.27:3300/0 conn(0x7f7b001592d0 0x7f7b0015a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:29.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.814+0000 7f7af7fff700 1 -- 172.21.15.27:0/770284376 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10567638 0 0) 0x7f7b00157e30 con 0x7f7b00159db0 2024-01-11T09:45:29.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.814+0000 7f7af7fff700 1 -- 172.21.15.27:0/770284376 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7aec004680 con 0x7f7b00159db0 2024-01-11T09:45:29.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.814+0000 7f7b05b9c700 1 -- 172.21.15.27:0/770284376 >> v1:172.21.15.77:6789/0 conn(0x7f7b001599e0 legacy=0x7f7b000b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:29.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.814+0000 7f7b05b9c700 1 -- 172.21.15.27:0/770284376 >> v1:172.21.15.27:6789/0 conn(0x7f7b00159db0 legacy=0x7f7b0014fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:29.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.814+0000 7f7b05b9c700 1 -- 172.21.15.27:0/770284376 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b00157ac0 con 0x7f7b001592d0 2024-01-11T09:45:29.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.814+0000 7f7b05b9c700 1 --2- 172.21.15.27:0/770284376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b001592d0 0x7f7b0015a230 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f7af8009e60 tx=0x7f7af8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecf798d8b2ccbd49 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:29.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.814+0000 7f7af7fff700 1 -- 172.21.15.27:0/770284376 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af800e070 con 0x7f7b001592d0 2024-01-11T09:45:29.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.814+0000 7f7af7fff700 1 -- 172.21.15.27:0/770284376 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af800bb60 con 0x7f7b001592d0 2024-01-11T09:45:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.814+0000 7f7af7fff700 1 -- 172.21.15.27:0/770284376 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8012430 con 0x7f7b001592d0 2024-01-11T09:45:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.815+0000 7f7b075ff700 1 -- 172.21.15.27:0/770284376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b001592d0 msgr2=0x7f7b0015a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.815+0000 7f7b075ff700 1 --2- 172.21.15.27:0/770284376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b001592d0 0x7f7b0015a230 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f7af8009e60 tx=0x7f7af8009780 comp rx=0 tx=0).stop 2024-01-11T09:45:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.815+0000 7f7b075ff700 1 -- 172.21.15.27:0/770284376 shutdown_connections 2024-01-11T09:45:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.815+0000 7f7b075ff700 1 --2- 172.21.15.27:0/770284376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b001592d0 0x7f7b0015a230 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.815+0000 7f7b075ff700 1 -- 172.21.15.27:0/770284376 >> 172.21.15.27:0/770284376 conn(0x7f7b000680e0 msgr2=0x7f7b00069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.815+0000 7f7b075ff700 1 -- 172.21.15.27:0/770284376 shutdown_connections 2024-01-11T09:45:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.815+0000 7f7b075ff700 1 -- 172.21.15.27:0/770284376 wait complete. 2024-01-11T09:45:29.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.817+0000 7f7b075ff700 1 Processor -- start 2024-01-11T09:45:29.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.817+0000 7f7b075ff700 1 -- start start 2024-01-11T09:45:29.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.818+0000 7f7b075ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b001592d0 0x7f7b00149800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:29.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.818+0000 7f7b075ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b001599e0 0x7f7b0006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:29.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.818+0000 7f7b075ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b00159db0 0x7f7b0006f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.818+0000 7f7b075ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b00168530 con 0x7f7b00159db0 2024-01-11T09:45:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.818+0000 7f7b075ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b00167920 con 0x7f7b001592d0 2024-01-11T09:45:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.818+0000 7f7b075ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b00167c00 con 0x7f7b001599e0 2024-01-11T09:45:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.818+0000 7f7b05b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b00159db0 0x7f7b0006f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.818+0000 7f7b05b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b00159db0 0x7f7b0006f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51868/0 (socket says 172.21.15.27:51868) 2024-01-11T09:45:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.818+0000 7f7b05b9c700 1 -- 172.21.15.27:0/2906994493 learned_addr learned my addr 172.21.15.27:0/2906994493 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.818+0000 7f7b04b9a700 1 --2- 172.21.15.27:0/2906994493 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b001599e0 0x7f7b0006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.818+0000 7f7b0539b700 1 --2- 172.21.15.27:0/2906994493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b001592d0 0x7f7b00149800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.818+0000 7f7b05b9c700 1 -- 172.21.15.27:0/2906994493 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b001599e0 msgr2=0x7f7b0006ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.818+0000 7f7b05b9c700 1 --2- 172.21.15.27:0/2906994493 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b001599e0 0x7f7b0006ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.818+0000 7f7b05b9c700 1 -- 172.21.15.27:0/2906994493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b001592d0 msgr2=0x7f7b00149800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.818+0000 7f7b05b9c700 1 --2- 172.21.15.27:0/2906994493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b001592d0 0x7f7b00149800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:29.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.818+0000 7f7b05b9c700 1 -- 172.21.15.27:0/2906994493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af8009a20 con 0x7f7b00159db0 2024-01-11T09:45:29.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.819+0000 7f7b05b9c700 1 --2- 172.21.15.27:0/2906994493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b00159db0 0x7f7b0006f1c0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f7af8013000 tx=0x7f7af8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:29.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.819+0000 7f7af57fa700 1 -- 172.21.15.27:0/2906994493 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8012e00 con 0x7f7b00159db0 2024-01-11T09:45:29.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.819+0000 7f7af57fa700 1 -- 172.21.15.27:0/2906994493 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af801d630 con 0x7f7b00159db0 2024-01-11T09:45:29.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.819+0000 7f7b075ff700 1 -- 172.21.15.27:0/2906994493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b00167ee0 con 0x7f7b00159db0 2024-01-11T09:45:29.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.819+0000 7f7af57fa700 1 -- 172.21.15.27:0/2906994493 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af801d9f0 con 0x7f7b00159db0 2024-01-11T09:45:29.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.819+0000 7f7b075ff700 1 -- 172.21.15.27:0/2906994493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b00149d40 con 0x7f7b00159db0 2024-01-11T09:45:29.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.820+0000 7f7b075ff700 1 -- 172.21.15.27:0/2906994493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b001686e0 con 0x7f7b00159db0 2024-01-11T09:45:29.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.821+0000 7f7af57fa700 1 -- 172.21.15.27:0/2906994493 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7af8024070 con 0x7f7b00159db0 2024-01-11T09:45:29.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.822+0000 7f7af57fa700 1 --2- 172.21.15.27:0/2906994493 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7ae4066f30 0x7f7ae40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:29.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.822+0000 7f7af57fa700 1 -- 172.21.15.27:0/2906994493 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f7af8086d80 con 0x7f7b00159db0 2024-01-11T09:45:29.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.823+0000 7f7b0539b700 1 --2- 172.21.15.27:0/2906994493 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7ae4066f30 0x7f7ae40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:29.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.824+0000 7f7b0539b700 1 --2- 172.21.15.27:0/2906994493 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7ae4066f30 0x7f7ae40693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7afc003860 tx=0x7f7afc009040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:29.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:29.827+0000 7f7af57fa700 1 -- 172.21.15.27:0/2906994493 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7b001686e0 con 0x7f7b00159db0 2024-01-11T09:45:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.070+0000 7f7b075ff700 1 -- 172.21.15.27:0/2906994493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b001686e0 con 0x7f7b00159db0 2024-01-11T09:45:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.070+0000 7f7af57fa700 1 -- 172.21.15.27:0/2906994493 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+4611 (secure 0 0 0) 0x7f7b001686e0 con 0x7f7b00159db0 2024-01-11T09:45:30.062 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:30.062 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":111,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2103359888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":2103359888},{"type":"v1","addr":"172.21.15.77:6834","nonce":2103359888}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8795,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2650824649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2650824649},{"type":"v1","addr":"172.21.15.77:6835","nonce":2650824649}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106},{"gid":10596,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/2807751836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2807751836},{"type":"v1","addr":"172.21.15.27:6834","nonce":2807751836}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:45:23.637614+0000","modified":"2024-01-11T09:45:27.146191+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10599":{"gid":10599,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1143960902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1143960902},{"type":"v1","addr":"172.21.15.27:6835","nonce":1143960902}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2024-01-11T09:45:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.073+0000 7f7b075ff700 1 -- 172.21.15.27:0/2906994493 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7ae4066f30 msgr2=0x7f7ae40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.073+0000 7f7b075ff700 1 --2- 172.21.15.27:0/2906994493 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7ae4066f30 0x7f7ae40693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7afc003860 tx=0x7f7afc009040 comp rx=0 tx=0).stop 2024-01-11T09:45:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.073+0000 7f7b075ff700 1 -- 172.21.15.27:0/2906994493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b00159db0 msgr2=0x7f7b0006f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.073+0000 7f7b075ff700 1 --2- 172.21.15.27:0/2906994493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b00159db0 0x7f7b0006f1c0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f7af8013000 tx=0x7f7af8019000 comp rx=0 tx=0).stop 2024-01-11T09:45:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.073+0000 7f7b075ff700 1 -- 172.21.15.27:0/2906994493 shutdown_connections 2024-01-11T09:45:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.073+0000 7f7b075ff700 1 --2- 172.21.15.27:0/2906994493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b00159db0 0x7f7b0006f1c0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.073+0000 7f7b075ff700 1 --2- 172.21.15.27:0/2906994493 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7ae4066f30 0x7f7ae40693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.073+0000 7f7b075ff700 1 --2- 172.21.15.27:0/2906994493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b001592d0 0x7f7b00149800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.073+0000 7f7b075ff700 1 --2- 172.21.15.27:0/2906994493 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b001599e0 0x7f7b0006ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.073+0000 7f7b075ff700 1 -- 172.21.15.27:0/2906994493 >> 172.21.15.27:0/2906994493 conn(0x7f7b000680e0 msgr2=0x7f7b000cbe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.073+0000 7f7b075ff700 1 -- 172.21.15.27:0/2906994493 shutdown_connections 2024-01-11T09:45:30.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.073+0000 7f7b075ff700 1 -- 172.21.15.27:0/2906994493 wait complete. 2024-01-11T09:45:30.067 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 111 2024-01-11T09:45:30.099 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-01-11T09:45:30.108+0000 7f566a1f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T09:45:30.485 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 111, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-11T09:45:23.637614+0000', 'modified': '2024-01-11T09:45:27.146191+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10599}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10599': {'gid': 10599, 'name': 'a', 'rank': 0, 'incarnation': 107, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6835/1143960902', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 1143960902}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 1143960902}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-11T09:45:30.485 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-11T09:45:30.486 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-11T09:45:30.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.697+0000 7f24e759e700 1 Processor -- start 2024-01-11T09:45:30.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.697+0000 7f24e759e700 1 -- start start 2024-01-11T09:45:30.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.698+0000 7f24e759e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f24e8159520 0x7f24e80b0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.698+0000 7f24e759e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f24e80b2a30 0x7f24e80b0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.698+0000 7f24e759e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e80cb340 con 0x7f24e81598f0 2024-01-11T09:45:30.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.698+0000 7f24e759e700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24e80cb620 con 0x7f24e80b2a30 2024-01-11T09:45:30.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.698+0000 7f24e759e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24e80cb900 con 0x7f24e8159520 2024-01-11T09:45:30.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.699+0000 7f24e6d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f24e80b2a30 0x7f24e80b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.699+0000 7f24e6d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f24e80b2a30 0x7f24e80b0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51878/0 (socket says 172.21.15.27:51878) 2024-01-11T09:45:30.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.699+0000 7f24e6d9d700 1 -- 172.21.15.27:0/1960065908 learned_addr learned my addr 172.21.15.27:0/1960065908 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:30.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.699+0000 7f24e659c700 1 --2- 172.21.15.27:0/1960065908 >> v2:172.21.15.77:3300/0 conn(0x7f24e8159520 0x7f24e80b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.699+0000 7f24e6d9d700 1 -- 172.21.15.27:0/1960065908 >> v2:172.21.15.77:3300/0 conn(0x7f24e8159520 msgr2=0x7f24e80b0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:30.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.699+0000 7f24e6d9d700 1 --2- 172.21.15.27:0/1960065908 >> v2:172.21.15.77:3300/0 conn(0x7f24e8159520 0x7f24e80b0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.699+0000 7f24e6d9d700 1 -- 172.21.15.27:0/1960065908 >> v1:172.21.15.77:6789/0 conn(0x7f24e81598f0 legacy=0x7f24e80b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:30.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.699+0000 7f24e6d9d700 1 -- 172.21.15.27:0/1960065908 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e80cbbe0 con 0x7f24e80b2a30 2024-01-11T09:45:30.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.699+0000 7f24e659c700 1 --2- 172.21.15.27:0/1960065908 >> v2:172.21.15.77:3300/0 conn(0x7f24e8159520 0x7f24e80b0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:30.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.699+0000 7f24e6d9d700 1 --2- 172.21.15.27:0/1960065908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e80b2a30 0x7f24e80b0570 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f24e8089310 tx=0x7f24e0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb9db7d950aef38b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:30.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.699+0000 7f24e559a700 1 -- 172.21.15.27:0/1960065908 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e000e070 con 0x7f24e80b2a30 2024-01-11T09:45:30.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.700+0000 7f24e559a700 1 -- 172.21.15.27:0/1960065908 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e000ba50 con 0x7f24e80b2a30 2024-01-11T09:45:30.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.700+0000 7f24e559a700 1 -- 172.21.15.27:0/1960065908 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e00092e0 con 0x7f24e80b2a30 2024-01-11T09:45:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.700+0000 7f24e759e700 1 -- 172.21.15.27:0/1960065908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e80b2a30 msgr2=0x7f24e80b0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.700+0000 7f24e759e700 1 --2- 172.21.15.27:0/1960065908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e80b2a30 0x7f24e80b0570 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f24e8089310 tx=0x7f24e0009840 comp rx=0 tx=0).stop 2024-01-11T09:45:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.700+0000 7f24e759e700 1 -- 172.21.15.27:0/1960065908 shutdown_connections 2024-01-11T09:45:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.700+0000 7f24e759e700 1 --2- 172.21.15.27:0/1960065908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e80b2a30 0x7f24e80b0570 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.700+0000 7f24e759e700 1 --2- 172.21.15.27:0/1960065908 >> v2:172.21.15.77:3300/0 conn(0x7f24e8159520 0x7f24e80b0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.700+0000 7f24e759e700 1 -- 172.21.15.27:0/1960065908 >> 172.21.15.27:0/1960065908 conn(0x7f24e806ed40 msgr2=0x7f24e806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.700+0000 7f24e759e700 1 -- 172.21.15.27:0/1960065908 shutdown_connections 2024-01-11T09:45:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.700+0000 7f24e759e700 1 -- 172.21.15.27:0/1960065908 wait complete. 2024-01-11T09:45:30.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.702+0000 7f24e759e700 1 Processor -- start 2024-01-11T09:45:30.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.702+0000 7f24e759e700 1 -- start start 2024-01-11T09:45:30.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.703+0000 7f24e759e700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24e80b2a30 0x7f24e816a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.703+0000 7f24e759e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e8159520 0x7f24e8157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.703+0000 7f24e759e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24e81598f0 0x7f24e80c5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.703+0000 7f24e759e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e80c4df0 con 0x7f24e8159520 2024-01-11T09:45:30.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.703+0000 7f24e759e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e80aff30 con 0x7f24e81598f0 2024-01-11T09:45:30.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.703+0000 7f24e759e700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e816a6e0 con 0x7f24e80b2a30 2024-01-11T09:45:30.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.703+0000 7f24e5d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e8159520 0x7f24e8157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.703+0000 7f24e5d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e8159520 0x7f24e8157a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51884/0 (socket says 172.21.15.27:51884) 2024-01-11T09:45:30.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.703+0000 7f24e5d9b700 1 -- 172.21.15.27:0/3588497553 learned_addr learned my addr 172.21.15.27:0/3588497553 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:30.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.703+0000 7f24e6d9d700 1 --2- 172.21.15.27:0/3588497553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24e81598f0 0x7f24e80c5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.703+0000 7f24e5d9b700 1 -- 172.21.15.27:0/3588497553 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24e80b2a30 msgr2=0x7f24e816a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:30.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.703+0000 7f24e659c700 1 --2- 172.21.15.27:0/3588497553 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24e80b2a30 0x7f24e816a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.703+0000 7f24e5d9b700 1 --2- 172.21.15.27:0/3588497553 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24e80b2a30 0x7f24e816a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.703+0000 7f24e5d9b700 1 -- 172.21.15.27:0/3588497553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24e81598f0 msgr2=0x7f24e80c5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.703+0000 7f24e5d9b700 1 --2- 172.21.15.27:0/3588497553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24e81598f0 0x7f24e80c5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.703+0000 7f24e5d9b700 1 -- 172.21.15.27:0/3588497553 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e0009ae0 con 0x7f24e8159520 2024-01-11T09:45:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.703+0000 7f24e6d9d700 1 --2- 172.21.15.27:0/3588497553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24e81598f0 0x7f24e80c5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.703+0000 7f24e659c700 1 --2- 172.21.15.27:0/3588497553 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24e80b2a30 0x7f24e816a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.704+0000 7f24e5d9b700 1 --2- 172.21.15.27:0/3588497553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e8159520 0x7f24e8157a60 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00c4e0 tx=0x7f24dc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.704+0000 7f24d67fc700 1 -- 172.21.15.27:0/3588497553 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc011590 con 0x7f24e8159520 2024-01-11T09:45:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.704+0000 7f24d67fc700 1 -- 172.21.15.27:0/3588497553 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24dc011710 con 0x7f24e8159520 2024-01-11T09:45:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.704+0000 7f24d67fc700 1 -- 172.21.15.27:0/3588497553 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc011ad0 con 0x7f24e8159520 2024-01-11T09:45:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.704+0000 7f24e759e700 1 -- 172.21.15.27:0/3588497553 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e8157fa0 con 0x7f24e8159520 2024-01-11T09:45:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.704+0000 7f24e759e700 1 -- 172.21.15.27:0/3588497553 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e80c44c0 con 0x7f24e8159520 2024-01-11T09:45:30.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.706+0000 7f24d67fc700 1 -- 172.21.15.27:0/3588497553 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f24dc01b3b0 con 0x7f24e8159520 2024-01-11T09:45:30.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.707+0000 7f24d67fc700 1 --2- 172.21.15.27:0/3588497553 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f24c8066f30 0x7f24c80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.708+0000 7f24d67fc700 1 -- 172.21.15.27:0/3588497553 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f24dc083b70 con 0x7f24e8159520 2024-01-11T09:45:30.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.708+0000 7f24e659c700 1 --2- 172.21.15.27:0/3588497553 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f24c8066f30 0x7f24c80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.709+0000 7f24e659c700 1 --2- 172.21.15.27:0/3588497553 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f24c8066f30 0x7f24c80693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f24d8006fd0 tx=0x7f24d8008040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:30.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.709+0000 7f24fb5f0b80 1 Processor -- start 2024-01-11T09:45:30.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.709+0000 7f24fb5f0b80 1 -- start start 2024-01-11T09:45:30.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.710+0000 7f24fb5f0b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 0x5571713689d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.710+0000 7f24fb5f0b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55717136e880 0x55717138ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.710+0000 7f24fb5f0b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 0x557171394160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.710+0000 7f24fb5f0b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557171285310 con 0x5571713904c0 2024-01-11T09:45:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.710+0000 7f24fb5f0b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571712c2130 con 0x557171368600 2024-01-11T09:45:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.710+0000 7f24fb5f0b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x5571712bfaf0 con 0x55717136e880 2024-01-11T09:45:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.710+0000 7f24e6d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 0x557171394160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.710+0000 7f24e6d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 0x557171394160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51886/0 (socket says 172.21.15.27:51886) 2024-01-11T09:45:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.710+0000 7f24e6d9d700 1 -- 172.21.15.27:0/4284246819 learned_addr learned my addr 172.21.15.27:0/4284246819 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.710+0000 7f24e5d9b700 1 --2- 172.21.15.27:0/4284246819 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55717136e880 0x55717138ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.710+0000 7f24e659c700 1 --2- 172.21.15.27:0/4284246819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 0x5571713689d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.710+0000 7f24e6d9d700 1 -- 172.21.15.27:0/4284246819 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55717136e880 msgr2=0x55717138ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.711+0000 7f24e6d9d700 1 --2- 172.21.15.27:0/4284246819 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55717136e880 0x55717138ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.711+0000 7f24e6d9d700 1 -- 172.21.15.27:0/4284246819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 msgr2=0x5571713689d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.711+0000 7f24e6d9d700 1 --2- 172.21.15.27:0/4284246819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 0x5571713689d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.711+0000 7f24e6d9d700 1 -- 172.21.15.27:0/4284246819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571711ae0a0 con 0x5571713904c0 2024-01-11T09:45:30.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.711+0000 7f24e659c700 1 --2- 172.21.15.27:0/4284246819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 0x5571713689d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:30.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.711+0000 7f24e6d9d700 1 --2- 172.21.15.27:0/4284246819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 0x557171394160 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009fd0 tx=0x7f24e0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa993d22da98684e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:30.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.711+0000 7f24c77fe700 1 -- 172.21.15.27:0/4284246819 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e000be60 con 0x5571713904c0 2024-01-11T09:45:30.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.711+0000 7f24c77fe700 1 -- 172.21.15.27:0/4284246819 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e0007200 con 0x5571713904c0 2024-01-11T09:45:30.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.711+0000 7f24c77fe700 1 -- 172.21.15.27:0/4284246819 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e0007610 con 0x5571713904c0 2024-01-11T09:45:30.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.712+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/4284246819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 msgr2=0x557171394160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:30.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.712+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/4284246819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 0x557171394160 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009fd0 tx=0x7f24e0018000 comp rx=0 tx=0).stop 2024-01-11T09:45:30.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.712+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/4284246819 shutdown_connections 2024-01-11T09:45:30.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.712+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/4284246819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 0x557171394160 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.712+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/4284246819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 0x5571713689d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.712+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/4284246819 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55717136e880 0x55717138ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.712+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/4284246819 >> 172.21.15.27:0/4284246819 conn(0x55717136db20 msgr2=0x55717136def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:30.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.712+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/4284246819 shutdown_connections 2024-01-11T09:45:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.712+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/4284246819 wait complete. 2024-01-11T09:45:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.713+0000 7f24fb5f0b80 1 -- start start 2024-01-11T09:45:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.713+0000 7f24fb5f0b80 1 Processor -- start 2024-01-11T09:45:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.714+0000 7f24fb5f0b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 0x5571713b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.714+0000 7f24fb5f0b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55717136e880 0x55717138d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.714+0000 7f24fb5f0b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 0x55717138dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.714+0000 7f24fb5f0b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571712bb8e0 con 0x5571713904c0 2024-01-11T09:45:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.714+0000 7f24fb5f0b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x557171267980 con 0x557171368600 2024-01-11T09:45:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.714+0000 7f24fb5f0b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x557171267620 con 0x55717136e880 2024-01-11T09:45:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.714+0000 7f24fb5f0b80 10 client.0 _ll_register_callbacks cb 0x5571710caa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T09:45:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.714+0000 7f24e6d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 0x55717138dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.714+0000 7f24e6d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 0x55717138dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51888/0 (socket says 172.21.15.27:51888) 2024-01-11T09:45:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.714+0000 7f24e6d9d700 1 -- 172.21.15.27:0/1502081855 learned_addr learned my addr 172.21.15.27:0/1502081855 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.714+0000 7f24e659c700 1 --2- 172.21.15.27:0/1502081855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 0x5571713b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.714+0000 7f24e5d9b700 1 --2- 172.21.15.27:0/1502081855 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55717136e880 0x55717138d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.715+0000 7f24e6d9d700 1 -- 172.21.15.27:0/1502081855 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55717136e880 msgr2=0x55717138d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:30.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.715+0000 7f24e6d9d700 1 --2- 172.21.15.27:0/1502081855 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55717136e880 0x55717138d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.715+0000 7f24e6d9d700 1 -- 172.21.15.27:0/1502081855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 msgr2=0x5571713b4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:30.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.715+0000 7f24e6d9d700 1 --2- 172.21.15.27:0/1502081855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 0x5571713b4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.715+0000 7f24e6d9d700 1 -- 172.21.15.27:0/1502081855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e80cbbe0 con 0x5571713904c0 2024-01-11T09:45:30.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.715+0000 7f24e6d9d700 1 --2- 172.21.15.27:0/1502081855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 0x55717138dca0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009ab0 tx=0x7f24e001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=199e3587dd206d3e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:30.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.715+0000 7f24c4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T09:45:30.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.715+0000 7f24c4ff9700 1 -- 172.21.15.27:0/1502081855 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e002d070 con 0x5571713904c0 2024-01-11T09:45:30.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.715+0000 7f24c4ff9700 1 -- 172.21.15.27:0/1502081855 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(214..214 src has 1..214) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f24e0007c10 con 0x5571713904c0 2024-01-11T09:45:30.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.715+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/1502081855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557171360560 con 0x5571713904c0 2024-01-11T09:45:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.715+0000 7f24c4ff9700 1 -- 172.21.15.27:0/1502081855 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=111) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24e001e430 con 0x5571713904c0 2024-01-11T09:45:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.715+0000 7f24fb5f0b80 10 client.10728 fetch_fsmap learned FSMap version 111 2024-01-11T09:45:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.715+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/1502081855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=111,monmap=2+}) v3 -- 0x55717128a3c0 con 0x5571713904c0 2024-01-11T09:45:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.716+0000 7f24c4ff9700 1 -- 172.21.15.27:0/1502081855 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557171360560 con 0x5571713904c0 2024-01-11T09:45:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.716+0000 7f24c4ff9700 1 -- 172.21.15.27:0/1502081855 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 111) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f24e0024030 con 0x5571713904c0 2024-01-11T09:45:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.716+0000 7f24fb5f0b80 10 client.10728 fetch_fsmap finished waiting for FSMap version 111 2024-01-11T09:45:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.716+0000 7f24fb5f0b80 10 client.10728 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-11T09:45:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.716+0000 7f24fb5f0b80 20 client.10728 populate_metadata read hostname 'smithi027' 2024-01-11T09:45:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.716+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/1502081855 >> [v2:172.21.15.27:6833/1143960902,v1:172.21.15.27:6835/1143960902] conn(0x557171393a50 0x5571713915f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.716+0000 7f24fb5f0b80 4 client.10728 mds_command: new command op to 10599 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-11T09:45:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.716+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/1502081855 --> [v2:172.21.15.27:6833/1143960902,v1:172.21.15.27:6835/1143960902] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55717133d0e0 con 0x557171393a50 2024-01-11T09:45:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.716+0000 7f24e659c700 1 --2- 172.21.15.27:0/1502081855 >> [v2:172.21.15.27:6833/1143960902,v1:172.21.15.27:6835/1143960902] conn(0x557171393a50 0x5571713915f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.717+0000 7f24c4ff9700 10 client.10728 ms_handle_connect on v2:172.21.15.27:6833/1143960902 2024-01-11T09:45:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.717+0000 7f24e659c700 1 --2- 172.21.15.27:0/1502081855 >> [v2:172.21.15.27:6833/1143960902,v1:172.21.15.27:6835/1143960902] conn(0x557171393a50 0x5571713915f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9a35ed6c2c5415a server_cookie=ec5e01fc6e348d14 in_seq=0 out_seq=0 2024-01-11T09:45:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.717+0000 7f24c4ff9700 1 -- 172.21.15.27:0/1502081855 <== mds.0 v2:172.21.15.27:6833/1143960902 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x55717133d0e0 con 0x557171393a50 2024-01-11T09:45:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.717+0000 7f24c4ff9700 10 client.10728 handle_command_reply: tid=0 2024-01-11T09:45:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.717+0000 7f24fb5f0b80 1 client.10728 shutdown 2024-01-11T09:45:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.718+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/1502081855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 msgr2=0x55717138dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.718+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/1502081855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 0x55717138dca0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009ab0 tx=0x7f24e001f000 comp rx=0 tx=0).stop 2024-01-11T09:45:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.718+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/1502081855 shutdown_connections 2024-01-11T09:45:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.718+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/1502081855 >> [v2:172.21.15.27:6833/1143960902,v1:172.21.15.27:6835/1143960902] conn(0x557171393a50 0x5571713915f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.718+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/1502081855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 0x55717138dca0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.718+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/1502081855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 0x5571713b4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.718+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/1502081855 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55717136e880 0x55717138d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.718+0000 7f24c4ff9700 0 client.10728 ms_handle_reset on v2:172.21.15.27:6833/1143960902 2024-01-11T09:45:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.718+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/1502081855 >> 172.21.15.27:0/1502081855 conn(0x55717136db20 msgr2=0x55717136bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.718+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/1502081855 shutdown_connections 2024-01-11T09:45:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.718+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/1502081855 wait complete. 2024-01-11T09:45:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.718+0000 7f24fb5f0b80 20 client.10728 trim_cache size 0 max 16384 2024-01-11T09:45:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.736+0000 7f24fb5f0b80 1 Processor -- start 2024-01-11T09:45:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.736+0000 7f24fb5f0b80 1 -- start start 2024-01-11T09:45:30.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.737+0000 7f24fb5f0b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 0x55717138f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.737+0000 7f24fb5f0b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55717136e880 0x5571713617d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.737+0000 7f24fb5f0b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 0x557171361d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.737+0000 7f24fb5f0b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571711531e0 con 0x5571713904c0 2024-01-11T09:45:30.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.737+0000 7f24fb5f0b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571713b1110 con 0x557171368600 2024-01-11T09:45:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.737+0000 7f24fb5f0b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x557171368d50 con 0x55717136e880 2024-01-11T09:45:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.737+0000 7f24e6d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 0x557171361d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.737+0000 7f24e6d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 0x557171361d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51904/0 (socket says 172.21.15.27:51904) 2024-01-11T09:45:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.737+0000 7f24e6d9d700 1 -- 172.21.15.27:0/2646905832 learned_addr learned my addr 172.21.15.27:0/2646905832 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.737+0000 7f24e659c700 1 --2- 172.21.15.27:0/2646905832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 0x55717138f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.737+0000 7f24e5d9b700 1 --2- 172.21.15.27:0/2646905832 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55717136e880 0x5571713617d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.737+0000 7f24e6d9d700 1 -- 172.21.15.27:0/2646905832 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55717136e880 msgr2=0x5571713617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.737+0000 7f24e6d9d700 1 --2- 172.21.15.27:0/2646905832 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55717136e880 0x5571713617d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.737+0000 7f24e6d9d700 1 -- 172.21.15.27:0/2646905832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 msgr2=0x55717138f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.737+0000 7f24e6d9d700 1 --2- 172.21.15.27:0/2646905832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 0x55717138f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.737+0000 7f24e6d9d700 1 -- 172.21.15.27:0/2646905832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571711221d0 con 0x5571713904c0 2024-01-11T09:45:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.738+0000 7f24e6d9d700 1 --2- 172.21.15.27:0/2646905832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 0x557171361d10 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f24e000b020 tx=0x7f24e002a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ff0a86761fa8be server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.738+0000 7f24c7fff700 1 -- 172.21.15.27:0/2646905832 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e002d070 con 0x5571713904c0 2024-01-11T09:45:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.738+0000 7f24c7fff700 1 -- 172.21.15.27:0/2646905832 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e0007620 con 0x5571713904c0 2024-01-11T09:45:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.738+0000 7f24c7fff700 1 -- 172.21.15.27:0/2646905832 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e00094e0 con 0x5571713904c0 2024-01-11T09:45:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.738+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/2646905832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 msgr2=0x557171361d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.738+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/2646905832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 0x557171361d10 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f24e000b020 tx=0x7f24e002a050 comp rx=0 tx=0).stop 2024-01-11T09:45:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.738+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/2646905832 shutdown_connections 2024-01-11T09:45:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.738+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/2646905832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5571713904c0 0x557171361d10 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.738+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/2646905832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 0x55717138f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.738+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/2646905832 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55717136e880 0x5571713617d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.738+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/2646905832 >> 172.21.15.27:0/2646905832 conn(0x55717136db20 msgr2=0x55717136def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.738+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/2646905832 shutdown_connections 2024-01-11T09:45:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.738+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/2646905832 wait complete. 2024-01-11T09:45:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.739+0000 7f24fb5f0b80 1 -- start start 2024-01-11T09:45:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.739+0000 7f24fb5f0b80 1 Processor -- start 2024-01-11T09:45:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.740+0000 7f24fb5f0b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 0x557171416da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.740+0000 7f24fb5f0b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55717136e880 0x557171394720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.740+0000 7f24fb5f0b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5571713904c0 0x55717136cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.740+0000 7f24fb5f0b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571713c18e0 con 0x55717136e880 2024-01-11T09:45:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.740+0000 7f24fb5f0b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x557171417860 con 0x557171368600 2024-01-11T09:45:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.740+0000 7f24fb5f0b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x5571713b4a00 con 0x5571713904c0 2024-01-11T09:45:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.740+0000 7f24fb5f0b80 10 client.0 _ll_register_callbacks cb 0x7f24dc050d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T09:45:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.741+0000 7f24e5d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55717136e880 0x557171394720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.741+0000 7f24e659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 0x557171416da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.741+0000 7f24e659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 0x557171416da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:45506/0 (socket says 172.21.15.27:45506) 2024-01-11T09:45:30.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.741+0000 7f24e659c700 1 -- 172.21.15.27:0/1447817717 learned_addr learned my addr 172.21.15.27:0/1447817717 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:30.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.741+0000 7f24e6d9d700 1 --2- 172.21.15.27:0/1447817717 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5571713904c0 0x55717136cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.741+0000 7f24e659c700 1 -- 172.21.15.27:0/1447817717 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5571713904c0 msgr2=0x55717136cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:30.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.741+0000 7f24e659c700 1 --2- 172.21.15.27:0/1447817717 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5571713904c0 0x55717136cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.741+0000 7f24e659c700 1 -- 172.21.15.27:0/1447817717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55717136e880 msgr2=0x557171394720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:30.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.741+0000 7f24e659c700 1 --2- 172.21.15.27:0/1447817717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55717136e880 0x557171394720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.741+0000 7f24e659c700 1 -- 172.21.15.27:0/1447817717 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55717128a3c0 con 0x557171368600 2024-01-11T09:45:30.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.741+0000 7f24e5d9b700 1 --2- 172.21.15.27:0/1447817717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55717136e880 0x557171394720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:30.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.741+0000 7f24e659c700 1 --2- 172.21.15.27:0/1447817717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 0x557171416da0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f24d8008890 tx=0x7f24d8027250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4624d4c69f88636d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:30.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.741+0000 7f24c4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-01-11T09:45:30.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.742+0000 7f24c4ff9700 1 -- 172.21.15.27:0/1447817717 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d800b050 con 0x557171368600 2024-01-11T09:45:30.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.742+0000 7f24c4ff9700 1 -- 172.21.15.27:0/1447817717 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(214..214 src has 1..214) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f24d802a470 con 0x557171368600 2024-01-11T09:45:30.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.742+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/1447817717 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f24e0007200 con 0x557171368600 2024-01-11T09:45:30.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.742+0000 7f24c4ff9700 1 -- 172.21.15.27:0/1447817717 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=111) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24d802a840 con 0x557171368600 2024-01-11T09:45:30.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.742+0000 7f24fb5f0b80 10 client.10345 fetch_fsmap learned FSMap version 111 2024-01-11T09:45:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.742+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/1447817717 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=111,monmap=2+}) v3 -- 0x5571710fbd60 con 0x557171368600 2024-01-11T09:45:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.742+0000 7f24c4ff9700 1 -- 172.21.15.27:0/1447817717 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e0007200 con 0x557171368600 2024-01-11T09:45:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.742+0000 7f24c4ff9700 1 -- 172.21.15.27:0/1447817717 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 111) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f24d802a9a0 con 0x557171368600 2024-01-11T09:45:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.742+0000 7f24fb5f0b80 10 client.10345 fetch_fsmap finished waiting for FSMap version 111 2024-01-11T09:45:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.742+0000 7f24fb5f0b80 10 client.10345 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-11T09:45:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.742+0000 7f24fb5f0b80 20 client.10345 populate_metadata read hostname 'smithi027' 2024-01-11T09:45:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.742+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/1447817717 >> [v2:172.21.15.27:6833/1143960902,v1:172.21.15.27:6835/1143960902] conn(0x55717136d290 0x5571713c0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.742+0000 7f24fb5f0b80 4 client.10345 mds_command: new command op to 10599 tid=0[{"prefix": "status"}] 2024-01-11T09:45:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.742+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/1447817717 --> [v2:172.21.15.27:6833/1143960902,v1:172.21.15.27:6835/1143960902] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55717125e8d0 con 0x55717136d290 2024-01-11T09:45:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.743+0000 7f24e5d9b700 1 --2- 172.21.15.27:0/1447817717 >> [v2:172.21.15.27:6833/1143960902,v1:172.21.15.27:6835/1143960902] conn(0x55717136d290 0x5571713c0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.743+0000 7f24c4ff9700 10 client.10345 ms_handle_connect on v2:172.21.15.27:6833/1143960902 2024-01-11T09:45:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.743+0000 7f24e5d9b700 1 --2- 172.21.15.27:0/1447817717 >> [v2:172.21.15.27:6833/1143960902,v1:172.21.15.27:6835/1143960902] conn(0x55717136d290 0x5571713c0500 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7223a750bcee8dbd server_cookie=f597793400d27315 in_seq=0 out_seq=0 2024-01-11T09:45:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.743+0000 7f24c4ff9700 1 -- 172.21.15.27:0/1447817717 <== mds.0 v2:172.21.15.27:6833/1143960902 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55717125e8d0 con 0x55717136d290 2024-01-11T09:45:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.743+0000 7f24c4ff9700 10 client.10345 handle_command_reply: tid=0 2024-01-11T09:45:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.743+0000 7f24fb5f0b80 1 client.10345 shutdown 2024-01-11T09:45:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.743+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/1447817717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 msgr2=0x557171416da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.743+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/1447817717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 0x557171416da0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f24d8008890 tx=0x7f24d8027250 comp rx=0 tx=0).stop 2024-01-11T09:45:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.744+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/1447817717 shutdown_connections 2024-01-11T09:45:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.744+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/1447817717 >> [v2:172.21.15.27:6833/1143960902,v1:172.21.15.27:6835/1143960902] conn(0x55717136d290 0x5571713c0500 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.744+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/1447817717 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5571713904c0 0x55717136cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.744+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/1447817717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557171368600 0x557171416da0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.744+0000 7f24fb5f0b80 1 --2- 172.21.15.27:0/1447817717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55717136e880 0x557171394720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.744+0000 7f24c4ff9700 0 client.10345 ms_handle_reset on v2:172.21.15.27:6833/1143960902 2024-01-11T09:45:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.744+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/1447817717 >> 172.21.15.27:0/1447817717 conn(0x55717136db20 msgr2=0x5571713617d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.744+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/1447817717 shutdown_connections 2024-01-11T09:45:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.744+0000 7f24fb5f0b80 1 -- 172.21.15.27:0/1447817717 wait complete. 2024-01-11T09:45:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.744+0000 7f24fb5f0b80 20 client.10345 trim_cache size 0 max 16384 2024-01-11T09:45:30.740 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:45:30.740 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "8fddb4a7-9d66-40f1-892d-6fe86c728ee4", 2024-01-11T09:45:30.740 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-11T09:45:30.740 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 10599, 2024-01-11T09:45:30.740 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-11T09:45:30.740 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-11T09:45:30.740 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-11T09:45:30.740 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 6.6030844240000004, 2024-01-11T09:45:30.740 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 111, 2024-01-11T09:45:30.740 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 214, 2024-01-11T09:45:30.740 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 214, 2024-01-11T09:45:30.740 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 12.240163900000001 2024-01-11T09:45:30.741 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:45:30.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.744+0000 7f24e759e700 1 -- 172.21.15.27:0/3588497553 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f24c8066f30 msgr2=0x7f24c80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:30.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.744+0000 7f24e759e700 1 --2- 172.21.15.27:0/3588497553 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f24c8066f30 0x7f24c80693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f24d8006fd0 tx=0x7f24d8008040 comp rx=0 tx=0).stop 2024-01-11T09:45:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.744+0000 7f24e759e700 1 -- 172.21.15.27:0/3588497553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e8159520 msgr2=0x7f24e8157a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.744+0000 7f24e759e700 1 --2- 172.21.15.27:0/3588497553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e8159520 0x7f24e8157a60 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00c4e0 tx=0x7f24dc00c8a0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.745+0000 7f24e759e700 1 -- 172.21.15.27:0/3588497553 shutdown_connections 2024-01-11T09:45:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.745+0000 7f24e759e700 1 --2- 172.21.15.27:0/3588497553 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f24c8066f30 0x7f24c80693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.745+0000 7f24e759e700 1 --2- 172.21.15.27:0/3588497553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24e81598f0 0x7f24e80c5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.745+0000 7f24e759e700 1 --2- 172.21.15.27:0/3588497553 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f24e80b2a30 0x7f24e816a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.745+0000 7f24e759e700 1 --2- 172.21.15.27:0/3588497553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e8159520 0x7f24e8157a60 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.745+0000 7f24e759e700 1 -- 172.21.15.27:0/3588497553 >> 172.21.15.27:0/3588497553 conn(0x7f24e806ed40 msgr2=0x7f24e8158b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.745+0000 7f24e759e700 1 -- 172.21.15.27:0/3588497553 shutdown_connections 2024-01-11T09:45:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.745+0000 7f24e759e700 1 -- 172.21.15.27:0/3588497553 wait complete. 2024-01-11T09:45:30.753 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-11T09:45:30.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.962+0000 7fc8dc1ff700 1 Processor -- start 2024-01-11T09:45:30.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.962+0000 7fc8dc1ff700 1 -- start start 2024-01-11T09:45:30.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.963+0000 7fc8dc1ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc8d40b2b90 0x7fc8d4159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.963+0000 7fc8dc1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d40cd730 con 0x7fc8d40b07b0 2024-01-11T09:45:30.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.963+0000 7fc8dc1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d40ccb20 con 0x7fc8d40b03e0 2024-01-11T09:45:30.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.963+0000 7fc8dc1ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8d40cce00 con 0x7fc8d40b2b90 2024-01-11T09:45:30.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.963+0000 7fc8d979a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc8d40b2b90 0x7fc8d4159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.963+0000 7fc8d979a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc8d40b2b90 0x7fc8d4159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51908/0 (socket says 172.21.15.27:51908) 2024-01-11T09:45:30.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.963+0000 7fc8d979a700 1 -- 172.21.15.27:0/3150621398 learned_addr learned my addr 172.21.15.27:0/3150621398 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:30.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.963+0000 7fc8da79c700 1 --1- 172.21.15.27:0/3150621398 >> v1:172.21.15.27:6789/0 conn(0x7fc8d40b07b0 0x7fc8d4159740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55982/0 (socket says 172.21.15.27:55982) 2024-01-11T09:45:30.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.964+0000 7fc8d979a700 1 -- 172.21.15.27:0/3150621398 >> v1:172.21.15.77:6789/0 conn(0x7fc8d40b03e0 legacy=0x7fc8d40b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:30.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.964+0000 7fc8d979a700 1 -- 172.21.15.27:0/3150621398 >> v1:172.21.15.27:6789/0 conn(0x7fc8d40b07b0 legacy=0x7fc8d4159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:30.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.964+0000 7fc8d979a700 1 -- 172.21.15.27:0/3150621398 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d40cd0e0 con 0x7fc8d40b2b90 2024-01-11T09:45:30.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.964+0000 7fc8d979a700 1 --2- 172.21.15.27:0/3150621398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8d40b2b90 0x7fc8d4159200 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fc8c40099b0 tx=0x7fc8c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75606ad644b6f7d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:30.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.964+0000 7fc8d8f99700 1 -- 172.21.15.27:0/3150621398 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c400b860 con 0x7fc8d40b2b90 2024-01-11T09:45:30.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.964+0000 7fc8d8f99700 1 -- 172.21.15.27:0/3150621398 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c400b9c0 con 0x7fc8d40b2b90 2024-01-11T09:45:30.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.964+0000 7fc8d8f99700 1 -- 172.21.15.27:0/3150621398 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c400bd60 con 0x7fc8d40b2b90 2024-01-11T09:45:30.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.965+0000 7fc8dc1ff700 1 -- 172.21.15.27:0/3150621398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8d40b2b90 msgr2=0x7fc8d4159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:30.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.965+0000 7fc8dc1ff700 1 --2- 172.21.15.27:0/3150621398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8d40b2b90 0x7fc8d4159200 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fc8c40099b0 tx=0x7fc8c4009250 comp rx=0 tx=0).stop 2024-01-11T09:45:30.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.965+0000 7fc8dc1ff700 1 -- 172.21.15.27:0/3150621398 shutdown_connections 2024-01-11T09:45:30.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.965+0000 7fc8dc1ff700 1 --2- 172.21.15.27:0/3150621398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8d40b2b90 0x7fc8d4159200 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.965+0000 7fc8dc1ff700 1 -- 172.21.15.27:0/3150621398 >> 172.21.15.27:0/3150621398 conn(0x7fc8d406ed40 msgr2=0x7fc8d406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:30.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.965+0000 7fc8dc1ff700 1 -- 172.21.15.27:0/3150621398 shutdown_connections 2024-01-11T09:45:30.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.965+0000 7fc8dc1ff700 1 -- 172.21.15.27:0/3150621398 wait complete. 2024-01-11T09:45:30.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.966+0000 7fc8dc1ff700 1 Processor -- start 2024-01-11T09:45:30.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.967+0000 7fc8dc1ff700 1 -- start start 2024-01-11T09:45:30.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.967+0000 7fc8dc1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8d40b03e0 0x7fc8d40ca610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.967+0000 7fc8dc1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8d40b07b0 0x7fc8d40c8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.967+0000 7fc8dc1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8d40b2b90 0x7fc8d4169b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.967+0000 7fc8dc1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d40cab50 con 0x7fc8d40b07b0 2024-01-11T09:45:30.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.967+0000 7fc8dc1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d40c85c0 con 0x7fc8d40b03e0 2024-01-11T09:45:30.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.967+0000 7fc8dc1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d40c90a0 con 0x7fc8d40b2b90 2024-01-11T09:45:30.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.968+0000 7fc8d979a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8d40b07b0 0x7fc8d40c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.968+0000 7fc8d979a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8d40b07b0 0x7fc8d40c8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51924/0 (socket says 172.21.15.27:51924) 2024-01-11T09:45:30.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.968+0000 7fc8d979a700 1 -- 172.21.15.27:0/2904724897 learned_addr learned my addr 172.21.15.27:0/2904724897 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:30.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.968+0000 7fc8d9f9b700 1 --2- 172.21.15.27:0/2904724897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8d40b03e0 0x7fc8d40ca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.968+0000 7fc8da79c700 1 --2- 172.21.15.27:0/2904724897 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8d40b2b90 0x7fc8d4169b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.968+0000 7fc8d979a700 1 -- 172.21.15.27:0/2904724897 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8d40b2b90 msgr2=0x7fc8d4169b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:30.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.968+0000 7fc8d979a700 1 --2- 172.21.15.27:0/2904724897 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8d40b2b90 0x7fc8d4169b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.968+0000 7fc8d979a700 1 -- 172.21.15.27:0/2904724897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8d40b03e0 msgr2=0x7fc8d40ca610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:30.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.968+0000 7fc8d979a700 1 --2- 172.21.15.27:0/2904724897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8d40b03e0 0x7fc8d40ca610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:30.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.968+0000 7fc8d979a700 1 -- 172.21.15.27:0/2904724897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c40094f0 con 0x7fc8d40b07b0 2024-01-11T09:45:30.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.968+0000 7fc8da79c700 1 --2- 172.21.15.27:0/2904724897 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8d40b2b90 0x7fc8d4169b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:30.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.969+0000 7fc8d979a700 1 --2- 172.21.15.27:0/2904724897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8d40b07b0 0x7fc8d40c8080 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4069540 tx=0x7fc8c4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:30.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.969+0000 7fc8c9ffb700 1 -- 172.21.15.27:0/2904724897 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4024440 con 0x7fc8d40b07b0 2024-01-11T09:45:30.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.969+0000 7fc8c9ffb700 1 -- 172.21.15.27:0/2904724897 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c40245a0 con 0x7fc8d40b07b0 2024-01-11T09:45:30.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.969+0000 7fc8c9ffb700 1 -- 172.21.15.27:0/2904724897 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4024940 con 0x7fc8d40b07b0 2024-01-11T09:45:30.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.969+0000 7fc8dc1ff700 1 -- 172.21.15.27:0/2904724897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d40c5030 con 0x7fc8d40b07b0 2024-01-11T09:45:30.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.969+0000 7fc8dc1ff700 1 -- 172.21.15.27:0/2904724897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d40c8ae0 con 0x7fc8d40b07b0 2024-01-11T09:45:30.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.970+0000 7fc8dc1ff700 1 -- 172.21.15.27:0/2904724897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d40c51c0 con 0x7fc8d40b07b0 2024-01-11T09:45:30.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.971+0000 7fc8c9ffb700 1 -- 172.21.15.27:0/2904724897 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fc8c401a420 con 0x7fc8d40b07b0 2024-01-11T09:45:30.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.972+0000 7fc8c9ffb700 1 --2- 172.21.15.27:0/2904724897 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc8b8066f30 0x7fc8b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:30.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.972+0000 7fc8c9ffb700 1 -- 172.21.15.27:0/2904724897 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5817+0+0 (secure 0 0 0) 0x7fc8c4086560 con 0x7fc8d40b07b0 2024-01-11T09:45:30.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.972+0000 7fc8d9f9b700 1 --2- 172.21.15.27:0/2904724897 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc8b8066f30 0x7fc8b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:30.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.973+0000 7fc8d9f9b700 1 --2- 172.21.15.27:0/2904724897 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc8b8066f30 0x7fc8b80693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0003a30 tx=0x7fc8d0007ec0 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:30.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:30.977+0000 7fc8c9ffb700 1 -- 172.21.15.27:0/2904724897 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc8c40536c0 con 0x7fc8d40b07b0 2024-01-11T09:45:31.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:31.228+0000 7fc8dc1ff700 1 -- 172.21.15.27:0/2904724897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc8d40cd3c0 con 0x7fc8d40b07b0 2024-01-11T09:45:31.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:31.228+0000 7fc8c9ffb700 1 -- 172.21.15.27:0/2904724897 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v111) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc8c4056ce0 con 0x7fc8d40b07b0 2024-01-11T09:45:31.220 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-11T09:45:31.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:31.230+0000 7fc8dc1ff700 1 -- 172.21.15.27:0/2904724897 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc8b8066f30 msgr2=0x7fc8b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:31.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:31.230+0000 7fc8dc1ff700 1 --2- 172.21.15.27:0/2904724897 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc8b8066f30 0x7fc8b80693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0003a30 tx=0x7fc8d0007ec0 comp rx=0 tx=0).stop 2024-01-11T09:45:31.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:31.230+0000 7fc8dc1ff700 1 -- 172.21.15.27:0/2904724897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8d40b07b0 msgr2=0x7fc8d40c8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:31.230+0000 7fc8dc1ff700 1 --2- 172.21.15.27:0/2904724897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8d40b07b0 0x7fc8d40c8080 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4069540 tx=0x7fc8c4017000 comp rx=0 tx=0).stop 2024-01-11T09:45:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:31.230+0000 7fc8dc1ff700 1 -- 172.21.15.27:0/2904724897 shutdown_connections 2024-01-11T09:45:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:31.230+0000 7fc8dc1ff700 1 --2- 172.21.15.27:0/2904724897 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc8d40b2b90 0x7fc8d4169b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:31.230+0000 7fc8dc1ff700 1 --2- 172.21.15.27:0/2904724897 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc8b8066f30 0x7fc8b80693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:31.230+0000 7fc8dc1ff700 1 --2- 172.21.15.27:0/2904724897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8d40b03e0 0x7fc8d40ca610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:31.230+0000 7fc8dc1ff700 1 --2- 172.21.15.27:0/2904724897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8d40b07b0 0x7fc8d40c8080 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:31.230+0000 7fc8dc1ff700 1 -- 172.21.15.27:0/2904724897 >> 172.21.15.27:0/2904724897 conn(0x7fc8d406ed40 msgr2=0x7fc8d40703a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:31.231+0000 7fc8dc1ff700 1 -- 172.21.15.27:0/2904724897 shutdown_connections 2024-01-11T09:45:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:31.231+0000 7fc8dc1ff700 1 -- 172.21.15.27:0/2904724897 wait complete. 2024-01-11T09:45:31.240 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-11T09:45:31.240 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-11T09:45:31.240 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-11T09:45:31.240 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-11T09:45:31.240 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-11T09:45:31.240 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-11T09:45:31.240 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-11T09:45:31.240 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-11T09:45:31.240 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T09:45:31.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:45:31.241 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-11T09:45:31.290 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T09:45:31.290 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T09:45:31.291 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T09:45:31.291 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:45:31.291 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-11T09:45:31.291 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:45:31.291 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T09:45:31.291 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-11T09:45:31.291 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-11T09:45:31.291 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T09:45:31.292 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-11T09:45:31.292 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-11T09:45:31.292 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-11T09:45:31.292 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-11T09:45:31.292 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-11T09:45:31.292 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-11T09:45:31.292 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 36791sec preferred_lft 36791sec 2024-01-11T09:45:31.293 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-11T09:45:31.293 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:45:31.293 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T09:45:31.293 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-11T09:45:31.293 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-11T09:45:31.293 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T09:45:31.293 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff 2024-01-11T09:45:31.294 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-11T09:45:31.294 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:45:31.294 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::acd2:b3ff:fe6a:db4b/64 scope link 2024-01-11T09:45:31.294 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:45:31.294 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T09:45:31.294 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:fa:85:1b:3a:bc brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-11T09:45:31.294 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ecfa:85ff:fe1b:3abc/64 scope link 2024-01-11T09:45:31.295 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:45:31.295 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T09:45:31.295 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-11T09:45:31.295 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ec88:faff:fe17:41aa/64 scope link 2024-01-11T09:45:31.295 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:45:31.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:45:31.296 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-11T09:45:31.366 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-11T09:45:31.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:45:31.370 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-11T09:45:31.420 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-11T09:45:31.420 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-11T09:45:31.421 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-11T09:45:31.421 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-11T09:45:31.421 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-11T09:45:31.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:45:31.422 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T09:45:31.422 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T09:45:31.422 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-11T09:45:31.422 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-11T09:45:31.422 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-11T09:45:31.422 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T09:45:31.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:45:31.599 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T09:45:31.646 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-11T09:45:31.647 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-11T09:45:31.647 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T09:45:31.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:45:31.648 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-11T09:45:31.722 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T09:45:31.722 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-11T09:45:31.770 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-11T09:45:31.770 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T09:45:31.770 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-11T09:45:31.817 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-11T09:45:31.817 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-11T09:45:31.817 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:45:31.920 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:45:31.921 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T09:45:31.921 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T09:45:31.935 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:45:31.936 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T09:45:31.936 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T09:45:31.936 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T09:45:31.936 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T09:45:31.958 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:45:31.958 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T09:45:31.958 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T09:45:31.958 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T09:45:31.958 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T09:45:31.958 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790" 2024-01-11T09:45:31.958 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790:/ 2024-01-11T09:45:31.958 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=8fddb4a7-9d66-40f1-892d-6fe86c728ee4" will pass to kernel 2024-01-11T09:45:31.962 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-11T09:45:31.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:45:31.963 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-11T09:45:32.036 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-11T09:45:32.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.253+0000 7f9ae3fff700 1 Processor -- start 2024-01-11T09:45:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.253+0000 7f9ae3fff700 1 -- start start 2024-01-11T09:45:32.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.254+0000 7f9ae3fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9adc0b08a0 0x7f9adc0684d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:32.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.254+0000 7f9ae3fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9adc068a10 0x7f9adc0ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:32.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.254+0000 7f9ae3fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9adc0b0d20 con 0x7f9adc159250 2024-01-11T09:45:32.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.254+0000 7f9ae3fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9adc15ace0 con 0x7f9adc0b08a0 2024-01-11T09:45:32.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.254+0000 7f9ae3fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc0cacf0 con 0x7f9adc068a10 2024-01-11T09:45:32.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.254+0000 7f9ae1d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9adc159250 0x7f9adc0cb410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55996/0 (socket says 172.21.15.27:55996) 2024-01-11T09:45:32.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.254+0000 7f9ae1d9b700 1 -- 172.21.15.27:0/2890019307 learned_addr learned my addr 172.21.15.27:0/2890019307 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:32.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.254+0000 7f9ae159a700 1 --2- 172.21.15.27:0/2890019307 >> v2:172.21.15.27:3300/0 conn(0x7f9adc0b08a0 0x7f9adc0684d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:32.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.255+0000 7f9ae259c700 1 --2- 172.21.15.27:0/2890019307 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9adc068a10 0x7f9adc0ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:32.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.255+0000 7f9ae0d99700 1 -- 172.21.15.27:0/2890019307 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798389727 0 0) 0x7f9adc0b0d20 con 0x7f9adc159250 2024-01-11T09:45:32.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.255+0000 7f9ae0d99700 1 -- 172.21.15.27:0/2890019307 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac8003570 con 0x7f9adc159250 2024-01-11T09:45:32.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.255+0000 7f9ae159a700 1 -- 172.21.15.27:0/2890019307 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9adc068a10 msgr2=0x7f9adc0ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:32.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.255+0000 7f9ae159a700 1 --2- 172.21.15.27:0/2890019307 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9adc068a10 0x7f9adc0ce2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:32.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.255+0000 7f9ae159a700 1 -- 172.21.15.27:0/2890019307 >> v1:172.21.15.27:6789/0 conn(0x7f9adc159250 legacy=0x7f9adc0cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:32.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.255+0000 7f9ae159a700 1 -- 172.21.15.27:0/2890019307 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9adc15a440 con 0x7f9adc0b08a0 2024-01-11T09:45:32.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.255+0000 7f9ae159a700 1 --2- 172.21.15.27:0/2890019307 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9adc0b08a0 0x7f9adc0684d0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f9acc0098e0 tx=0x7f9acc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d40fe6fdefd0296f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:32.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.255+0000 7f9ae0d99700 1 -- 172.21.15.27:0/2890019307 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc011630 con 0x7f9adc0b08a0 2024-01-11T09:45:32.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.256+0000 7f9ae0d99700 1 -- 172.21.15.27:0/2890019307 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9acc011790 con 0x7f9adc0b08a0 2024-01-11T09:45:32.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.256+0000 7f9ae0d99700 1 -- 172.21.15.27:0/2890019307 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc011b30 con 0x7f9adc0b08a0 2024-01-11T09:45:32.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.256+0000 7f9ae3fff700 1 -- 172.21.15.27:0/2890019307 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9adc0b08a0 msgr2=0x7f9adc0684d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:32.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.256+0000 7f9ae3fff700 1 --2- 172.21.15.27:0/2890019307 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9adc0b08a0 0x7f9adc0684d0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f9acc0098e0 tx=0x7f9acc0091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:32.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.256+0000 7f9ae3fff700 1 -- 172.21.15.27:0/2890019307 shutdown_connections 2024-01-11T09:45:32.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.256+0000 7f9ae3fff700 1 --2- 172.21.15.27:0/2890019307 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9adc068a10 0x7f9adc0ce2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:32.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.256+0000 7f9ae3fff700 1 --2- 172.21.15.27:0/2890019307 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9adc0b08a0 0x7f9adc0684d0 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:32.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.256+0000 7f9ae3fff700 1 -- 172.21.15.27:0/2890019307 >> 172.21.15.27:0/2890019307 conn(0x7f9adc067610 msgr2=0x7f9adc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:32.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.256+0000 7f9ae3fff700 1 -- 172.21.15.27:0/2890019307 shutdown_connections 2024-01-11T09:45:32.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.256+0000 7f9ae3fff700 1 -- 172.21.15.27:0/2890019307 wait complete. 2024-01-11T09:45:32.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.258+0000 7f9ae3fff700 1 Processor -- start 2024-01-11T09:45:32.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.258+0000 7f9ae3fff700 1 -- start start 2024-01-11T09:45:32.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.259+0000 7f9ae3fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9adc068a10 0x7f9adc14a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:32.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.259+0000 7f9ae3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9adc0b08a0 0x7f9adc0c1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:32.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.259+0000 7f9ae3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9adc159250 0x7f9adc0c1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:32.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.259+0000 7f9ae3fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc0c50c0 con 0x7f9adc159250 2024-01-11T09:45:32.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.259+0000 7f9ae3fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc0c6480 con 0x7f9adc0b08a0 2024-01-11T09:45:32.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.259+0000 7f9ae3fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc0c5e70 con 0x7f9adc068a10 2024-01-11T09:45:32.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.259+0000 7f9ae259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9adc159250 0x7f9adc0c1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:32.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.259+0000 7f9ae259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9adc159250 0x7f9adc0c1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51950/0 (socket says 172.21.15.27:51950) 2024-01-11T09:45:32.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.259+0000 7f9ae259c700 1 -- 172.21.15.27:0/2896495204 learned_addr learned my addr 172.21.15.27:0/2896495204 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:32.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.259+0000 7f9ae1d9b700 1 --2- 172.21.15.27:0/2896495204 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9adc068a10 0x7f9adc14a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:32.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.259+0000 7f9ae159a700 1 --2- 172.21.15.27:0/2896495204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9adc0b08a0 0x7f9adc0c1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.260+0000 7f9ae259c700 1 -- 172.21.15.27:0/2896495204 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9adc068a10 msgr2=0x7f9adc14a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.260+0000 7f9ae259c700 1 --2- 172.21.15.27:0/2896495204 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9adc068a10 0x7f9adc14a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.260+0000 7f9ae259c700 1 -- 172.21.15.27:0/2896495204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9adc0b08a0 msgr2=0x7f9adc0c1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.260+0000 7f9ae259c700 1 --2- 172.21.15.27:0/2896495204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9adc0b08a0 0x7f9adc0c1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.260+0000 7f9ae259c700 1 -- 172.21.15.27:0/2896495204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9acc009420 con 0x7f9adc159250 2024-01-11T09:45:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.260+0000 7f9ae159a700 1 --2- 172.21.15.27:0/2896495204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9adc0b08a0 0x7f9adc0c1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.260+0000 7f9ae259c700 1 --2- 172.21.15.27:0/2896495204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9adc159250 0x7f9adc0c1fa0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4009df0 tx=0x7f9ad400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.260+0000 7f9ad1ffb700 1 -- 172.21.15.27:0/2896495204 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad400cb50 con 0x7f9adc159250 2024-01-11T09:45:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.260+0000 7f9ad1ffb700 1 -- 172.21.15.27:0/2896495204 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad400f040 con 0x7f9adc159250 2024-01-11T09:45:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.260+0000 7f9ae3fff700 1 -- 172.21.15.27:0/2896495204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9adc0b3720 con 0x7f9adc159250 2024-01-11T09:45:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.260+0000 7f9ad1ffb700 1 -- 172.21.15.27:0/2896495204 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4014480 con 0x7f9adc159250 2024-01-11T09:45:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.260+0000 7f9ae3fff700 1 -- 172.21.15.27:0/2896495204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9adc0c6110 con 0x7f9adc159250 2024-01-11T09:45:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.261+0000 7f9ae3fff700 1 -- 172.21.15.27:0/2896495204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9adc0c6610 con 0x7f9adc159250 2024-01-11T09:45:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.262+0000 7f9ad1ffb700 1 -- 172.21.15.27:0/2896495204 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9ad4014830 con 0x7f9adc159250 2024-01-11T09:45:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.263+0000 7f9ad1ffb700 1 --2- 172.21.15.27:0/2896495204 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9ac0066f30 0x7f9ac00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:32.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.263+0000 7f9ad1ffb700 1 -- 172.21.15.27:0/2896495204 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f9ad4084f90 con 0x7f9adc159250 2024-01-11T09:45:32.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.264+0000 7f9ae1d9b700 1 --2- 172.21.15.27:0/2896495204 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9ac0066f30 0x7f9ac00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:32.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.264+0000 7f9ae1d9b700 1 --2- 172.21.15.27:0/2896495204 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9ac0066f30 0x7f9ac00693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8007fd0 tx=0x7f9ad8009040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:32.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.267+0000 7f9ad1ffb700 1 -- 172.21.15.27:0/2896495204 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9ad40520f0 con 0x7f9adc159250 2024-01-11T09:45:32.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.514+0000 7f9ae3fff700 1 -- 172.21.15.27:0/2896495204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9adc04c3c0 con 0x7f9adc159250 2024-01-11T09:45:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.923+0000 7f9ad1ffb700 1 -- 172.21.15.27:0/2896495204 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v112) v1 ==== 136+0+0 (secure 0 0 0) 0x7f9ad4055710 con 0x7f9adc159250 2024-01-11T09:45:32.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.926+0000 7f9ae3fff700 1 -- 172.21.15.27:0/2896495204 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9ac0066f30 msgr2=0x7f9ac00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:32.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.926+0000 7f9ae3fff700 1 --2- 172.21.15.27:0/2896495204 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9ac0066f30 0x7f9ac00693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8007fd0 tx=0x7f9ad8009040 comp rx=0 tx=0).stop 2024-01-11T09:45:32.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.926+0000 7f9ae3fff700 1 -- 172.21.15.27:0/2896495204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9adc159250 msgr2=0x7f9adc0c1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:32.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.926+0000 7f9ae3fff700 1 --2- 172.21.15.27:0/2896495204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9adc159250 0x7f9adc0c1fa0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4009df0 tx=0x7f9ad400b040 comp rx=0 tx=0).stop 2024-01-11T09:45:32.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.926+0000 7f9ae3fff700 1 -- 172.21.15.27:0/2896495204 shutdown_connections 2024-01-11T09:45:32.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.926+0000 7f9ae3fff700 1 --2- 172.21.15.27:0/2896495204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9adc159250 0x7f9adc0c1fa0 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:32.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.926+0000 7f9ae3fff700 1 --2- 172.21.15.27:0/2896495204 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9adc068a10 0x7f9adc14a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:32.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.926+0000 7f9ae3fff700 1 --2- 172.21.15.27:0/2896495204 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9ac0066f30 0x7f9ac00693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:32.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.926+0000 7f9ae3fff700 1 --2- 172.21.15.27:0/2896495204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9adc0b08a0 0x7f9adc0c1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:32.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.926+0000 7f9ae3fff700 1 -- 172.21.15.27:0/2896495204 >> 172.21.15.27:0/2896495204 conn(0x7f9adc067610 msgr2=0x7f9adc158810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:32.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.926+0000 7f9ae3fff700 1 -- 172.21.15.27:0/2896495204 shutdown_connections 2024-01-11T09:45:32.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:32.926+0000 7f9ae3fff700 1 -- 172.21.15.27:0/2896495204 wait complete. 2024-01-11T09:45:32.936 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:45:33.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.144+0000 7fb4143ff700 1 Processor -- start 2024-01-11T09:45:33.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.144+0000 7fb4143ff700 1 -- start start 2024-01-11T09:45:33.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.146+0000 7fb4143ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb40c0ca3e0 0x7fb40c0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:33.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.146+0000 7fb4143ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb40c0ca7b0 0x7fb40c159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:33.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.146+0000 7fb4143ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb40c15d640 con 0x7fb40c0c8a20 2024-01-11T09:45:33.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.146+0000 7fb4143ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb40c15ae70 con 0x7fb40c0ca3e0 2024-01-11T09:45:33.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.146+0000 7fb4143ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c15a3c0 con 0x7fb40c0ca7b0 2024-01-11T09:45:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.147+0000 7fb41219b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb40c0ca3e0 0x7fb40c0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.147+0000 7fb41199a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb40c0ca7b0 0x7fb40c159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.147+0000 7fb41219b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb40c0ca3e0 0x7fb40c0cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:45548/0 (socket says 172.21.15.27:45548) 2024-01-11T09:45:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.147+0000 7fb41199a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb40c0ca7b0 0x7fb40c159e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:34966/0 (socket says 172.21.15.27:34966) 2024-01-11T09:45:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.147+0000 7fb41219b700 1 -- 172.21.15.27:0/220900615 learned_addr learned my addr 172.21.15.27:0/220900615 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.147+0000 7fb41219b700 1 -- 172.21.15.27:0/220900615 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb40c0ca7b0 msgr2=0x7fb40c159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.147+0000 7fb41219b700 1 --2- 172.21.15.27:0/220900615 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb40c0ca7b0 0x7fb40c159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.147+0000 7fb41219b700 1 -- 172.21.15.27:0/220900615 >> v1:172.21.15.77:6789/0 conn(0x7fb40c0c8a20 legacy=0x7fb40c15dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.147+0000 7fb41219b700 1 -- 172.21.15.27:0/220900615 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb40c15a5d0 con 0x7fb40c0ca3e0 2024-01-11T09:45:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.148+0000 7fb41219b700 1 --2- 172.21.15.27:0/220900615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb40c0ca3e0 0x7fb40c0cca20 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fb4080099b0 tx=0x7fb408009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cb948499d8f2fb2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.148+0000 7fb411199700 1 -- 172.21.15.27:0/220900615 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb408009ea0 con 0x7fb40c0ca3e0 2024-01-11T09:45:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.148+0000 7fb411199700 1 -- 172.21.15.27:0/220900615 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb40800b860 con 0x7fb40c0ca3e0 2024-01-11T09:45:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.148+0000 7fb411199700 1 -- 172.21.15.27:0/220900615 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40800bc00 con 0x7fb40c0ca3e0 2024-01-11T09:45:33.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.148+0000 7fb4143ff700 1 -- 172.21.15.27:0/220900615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb40c0ca3e0 msgr2=0x7fb40c0cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:33.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.148+0000 7fb4143ff700 1 --2- 172.21.15.27:0/220900615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb40c0ca3e0 0x7fb40c0cca20 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fb4080099b0 tx=0x7fb408009250 comp rx=0 tx=0).stop 2024-01-11T09:45:33.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.149+0000 7fb4143ff700 1 -- 172.21.15.27:0/220900615 shutdown_connections 2024-01-11T09:45:33.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.149+0000 7fb4143ff700 1 --2- 172.21.15.27:0/220900615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb40c0ca3e0 0x7fb40c0cca20 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:33.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.149+0000 7fb4143ff700 1 --2- 172.21.15.27:0/220900615 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb40c0ca7b0 0x7fb40c159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:33.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.149+0000 7fb4143ff700 1 -- 172.21.15.27:0/220900615 >> 172.21.15.27:0/220900615 conn(0x7fb40c0ce3e0 msgr2=0x7fb40c0ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:33.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.149+0000 7fb4143ff700 1 -- 172.21.15.27:0/220900615 shutdown_connections 2024-01-11T09:45:33.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.149+0000 7fb4143ff700 1 -- 172.21.15.27:0/220900615 wait complete. 2024-01-11T09:45:33.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.151+0000 7fb4143ff700 1 Processor -- start 2024-01-11T09:45:33.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.151+0000 7fb4143ff700 1 -- start start 2024-01-11T09:45:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.152+0000 7fb4143ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c0c8a20 0x7fb40c14f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.152+0000 7fb4143ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb40c0ca3e0 0x7fb40c14eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.152+0000 7fb4143ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb40c0ca7b0 0x7fb40c0c6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.152+0000 7fb4143ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c0b1450 con 0x7fb40c0c8a20 2024-01-11T09:45:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.152+0000 7fb4143ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c14feb0 con 0x7fb40c0ca3e0 2024-01-11T09:45:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.152+0000 7fb4143ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c0b1880 con 0x7fb40c0ca7b0 2024-01-11T09:45:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.152+0000 7fb41219b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c0c8a20 0x7fb40c14f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.152+0000 7fb41219b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c0c8a20 0x7fb40c14f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51958/0 (socket says 172.21.15.27:51958) 2024-01-11T09:45:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.152+0000 7fb41219b700 1 -- 172.21.15.27:0/3370922457 learned_addr learned my addr 172.21.15.27:0/3370922457 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.153+0000 7fb41199a700 1 --2- 172.21.15.27:0/3370922457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb40c0ca3e0 0x7fb40c14eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.153+0000 7fb41299c700 1 --2- 172.21.15.27:0/3370922457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb40c0ca7b0 0x7fb40c0c6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.153+0000 7fb41219b700 1 -- 172.21.15.27:0/3370922457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb40c0ca7b0 msgr2=0x7fb40c0c6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.153+0000 7fb41219b700 1 --2- 172.21.15.27:0/3370922457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb40c0ca7b0 0x7fb40c0c6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.153+0000 7fb41219b700 1 -- 172.21.15.27:0/3370922457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb40c0ca3e0 msgr2=0x7fb40c14eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.153+0000 7fb41219b700 1 --2- 172.21.15.27:0/3370922457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb40c0ca3e0 0x7fb40c14eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.153+0000 7fb41219b700 1 -- 172.21.15.27:0/3370922457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4080094f0 con 0x7fb40c0c8a20 2024-01-11T09:45:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.153+0000 7fb41299c700 1 --2- 172.21.15.27:0/3370922457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb40c0ca7b0 0x7fb40c0c6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.153+0000 7fb41219b700 1 --2- 172.21.15.27:0/3370922457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c0c8a20 0x7fb40c14f970 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fb40c064980 tx=0x7fb408017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.153+0000 7fb4027fc700 1 -- 172.21.15.27:0/3370922457 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40800bbe0 con 0x7fb40c0c8a20 2024-01-11T09:45:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.154+0000 7fb4027fc700 1 -- 172.21.15.27:0/3370922457 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb408011420 con 0x7fb40c0c8a20 2024-01-11T09:45:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.153+0000 7fb4143ff700 1 -- 172.21.15.27:0/3370922457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb40c1500e0 con 0x7fb40c0c8a20 2024-01-11T09:45:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.154+0000 7fb4027fc700 1 -- 172.21.15.27:0/3370922457 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4080117c0 con 0x7fb40c0c8a20 2024-01-11T09:45:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.154+0000 7fb4143ff700 1 -- 172.21.15.27:0/3370922457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb40c1506a0 con 0x7fb40c0c8a20 2024-01-11T09:45:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.155+0000 7fb4143ff700 1 -- 172.21.15.27:0/3370922457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb40c0c8e60 con 0x7fb40c0c8a20 2024-01-11T09:45:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.156+0000 7fb4027fc700 1 -- 172.21.15.27:0/3370922457 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb408011bf0 con 0x7fb40c0c8a20 2024-01-11T09:45:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.156+0000 7fb4027fc700 1 --2- 172.21.15.27:0/3370922457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb3f0066f30 0x7fb3f00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.157+0000 7fb4027fc700 1 -- 172.21.15.27:0/3370922457 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5817+0+0 (secure 0 0 0) 0x7fb408085ea0 con 0x7fb40c0c8a20 2024-01-11T09:45:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.157+0000 7fb41199a700 1 --2- 172.21.15.27:0/3370922457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb3f0066f30 0x7fb3f00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:33.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.158+0000 7fb41199a700 1 --2- 172.21.15.27:0/3370922457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb3f0066f30 0x7fb3f00693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb40c0c6220 tx=0x7fb3fc0091a0 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.161+0000 7fb4027fc700 1 -- 172.21.15.27:0/3370922457 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb408053000 con 0x7fb40c0c8a20 2024-01-11T09:45:33.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.364+0000 7fb4143ff700 1 -- 172.21.15.27:0/3370922457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb40c0b1650 con 0x7fb40c0c8a20 2024-01-11T09:45:33.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.365+0000 7fb4027fc700 1 -- 172.21.15.27:0/3370922457 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v214) v1 ==== 74+0+15495 (secure 0 0 0) 0x7fb408056620 con 0x7fb40c0c8a20 2024-01-11T09:45:33.357 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:33.357 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":214,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:45:25.139360+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:45:21.455328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-01-11T09:45:22.634298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:45:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.368+0000 7fb4143ff700 1 -- 172.21.15.27:0/3370922457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb3f0066f30 msgr2=0x7fb3f00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.368+0000 7fb4143ff700 1 --2- 172.21.15.27:0/3370922457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb3f0066f30 0x7fb3f00693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb40c0c6220 tx=0x7fb3fc0091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.368+0000 7fb4143ff700 1 -- 172.21.15.27:0/3370922457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c0c8a20 msgr2=0x7fb40c14f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.368+0000 7fb4143ff700 1 --2- 172.21.15.27:0/3370922457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c0c8a20 0x7fb40c14f970 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fb40c064980 tx=0x7fb408017000 comp rx=0 tx=0).stop 2024-01-11T09:45:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.368+0000 7fb4143ff700 1 -- 172.21.15.27:0/3370922457 shutdown_connections 2024-01-11T09:45:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.368+0000 7fb4143ff700 1 --2- 172.21.15.27:0/3370922457 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb40c0ca7b0 0x7fb40c0c6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.368+0000 7fb4143ff700 1 --2- 172.21.15.27:0/3370922457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c0c8a20 0x7fb40c14f970 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.368+0000 7fb4143ff700 1 --2- 172.21.15.27:0/3370922457 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb3f0066f30 0x7fb3f00693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.368+0000 7fb4143ff700 1 --2- 172.21.15.27:0/3370922457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb40c0ca3e0 0x7fb40c14eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.368+0000 7fb4143ff700 1 -- 172.21.15.27:0/3370922457 >> 172.21.15.27:0/3370922457 conn(0x7fb40c0ce3e0 msgr2=0x7fb40c068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:33.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.368+0000 7fb4143ff700 1 -- 172.21.15.27:0/3370922457 shutdown_connections 2024-01-11T09:45:33.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.368+0000 7fb4143ff700 1 -- 172.21.15.27:0/3370922457 wait complete. 2024-01-11T09:45:33.377 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T09:45:21.455328+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '213', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T09:45:22.634298+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '214', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T09:45:33.378 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T09:45:33.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.589+0000 7faa1fdff700 1 Processor -- start 2024-01-11T09:45:33.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.589+0000 7faa1fdff700 1 -- start start 2024-01-11T09:45:33.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.590+0000 7faa1fdff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faa18069bd0 0x7faa180aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:33.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.590+0000 7faa1fdff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faa180c6de0 0x7faa180c7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:33.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.590+0000 7faa1fdff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1806e480 con 0x7faa18155c50 2024-01-11T09:45:33.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.590+0000 7faa1fdff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa180cb5a0 con 0x7faa180c6de0 2024-01-11T09:45:33.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.590+0000 7faa1fdff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa1806df00 con 0x7faa18069bd0 2024-01-11T09:45:33.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.591+0000 7faa1db9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7faa18155c50 0x7faa180694c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56000/0 (socket says 172.21.15.27:56000) 2024-01-11T09:45:33.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.591+0000 7faa1db9b700 1 -- 172.21.15.27:0/656719366 learned_addr learned my addr 172.21.15.27:0/656719366 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:33.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.591+0000 7faa1e39c700 1 --2- 172.21.15.27:0/656719366 >> v2:172.21.15.27:3300/0 conn(0x7faa180c6de0 0x7faa180c7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:33.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.591+0000 7faa1d39a700 1 --2- 172.21.15.27:0/656719366 >> v2:172.21.15.77:3300/0 conn(0x7faa18069bd0 0x7faa180aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:33.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.591+0000 7faa1cb99700 1 -- 172.21.15.27:0/656719366 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116572214 0 0) 0x7faa1806e480 con 0x7faa18155c50 2024-01-11T09:45:33.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.591+0000 7faa1cb99700 1 -- 172.21.15.27:0/656719366 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa04003570 con 0x7faa18155c50 2024-01-11T09:45:33.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.591+0000 7faa1e39c700 1 -- 172.21.15.27:0/656719366 >> v2:172.21.15.77:3300/0 conn(0x7faa18069bd0 msgr2=0x7faa180aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:33.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.591+0000 7faa1e39c700 1 --2- 172.21.15.27:0/656719366 >> v2:172.21.15.77:3300/0 conn(0x7faa18069bd0 0x7faa180aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:33.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.591+0000 7faa1e39c700 1 -- 172.21.15.27:0/656719366 >> v1:172.21.15.27:6789/0 conn(0x7faa18155c50 legacy=0x7faa180694c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:33.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.591+0000 7faa1e39c700 1 -- 172.21.15.27:0/656719366 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa180c7a60 con 0x7faa180c6de0 2024-01-11T09:45:33.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.591+0000 7faa1d39a700 1 --2- 172.21.15.27:0/656719366 >> v2:172.21.15.77:3300/0 conn(0x7faa18069bd0 0x7faa180aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:33.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.592+0000 7faa1e39c700 1 --2- 172.21.15.27:0/656719366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa180c6de0 0x7faa180c7220 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7faa1806d2a0 tx=0x7faa10009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41ce442a12827c3d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:33.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.592+0000 7faa1cb99700 1 -- 172.21.15.27:0/656719366 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1000e070 con 0x7faa180c6de0 2024-01-11T09:45:33.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.592+0000 7faa1cb99700 1 -- 172.21.15.27:0/656719366 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1000bb40 con 0x7faa180c6de0 2024-01-11T09:45:33.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.592+0000 7faa1cb99700 1 -- 172.21.15.27:0/656719366 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa10012430 con 0x7faa180c6de0 2024-01-11T09:45:33.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.592+0000 7faa1fdff700 1 -- 172.21.15.27:0/656719366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa180c6de0 msgr2=0x7faa180c7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:33.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.592+0000 7faa1fdff700 1 --2- 172.21.15.27:0/656719366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa180c6de0 0x7faa180c7220 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7faa1806d2a0 tx=0x7faa10009780 comp rx=0 tx=0).stop 2024-01-11T09:45:33.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.592+0000 7faa1fdff700 1 -- 172.21.15.27:0/656719366 shutdown_connections 2024-01-11T09:45:33.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.592+0000 7faa1fdff700 1 --2- 172.21.15.27:0/656719366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa180c6de0 0x7faa180c7220 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:33.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.592+0000 7faa1fdff700 1 --2- 172.21.15.27:0/656719366 >> v2:172.21.15.77:3300/0 conn(0x7faa18069bd0 0x7faa180aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:33.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.592+0000 7faa1fdff700 1 -- 172.21.15.27:0/656719366 >> 172.21.15.27:0/656719366 conn(0x7faa180680a0 msgr2=0x7faa180cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:33.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.592+0000 7faa1fdff700 1 -- 172.21.15.27:0/656719366 shutdown_connections 2024-01-11T09:45:33.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.593+0000 7faa1fdff700 1 -- 172.21.15.27:0/656719366 wait complete. 2024-01-11T09:45:33.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.594+0000 7faa1fdff700 1 Processor -- start 2024-01-11T09:45:33.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.594+0000 7faa1fdff700 1 -- start start 2024-01-11T09:45:33.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.595+0000 7faa1fdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faa18069bd0 0x7faa1813f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:33.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.595+0000 7faa1fdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa180c6de0 0x7faa18152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:33.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.595+0000 7faa1fdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa18155c50 0x7faa180c9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:33.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.595+0000 7faa1fdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa180ae730 con 0x7faa18155c50 2024-01-11T09:45:33.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.595+0000 7faa1fdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa180aac50 con 0x7faa180c6de0 2024-01-11T09:45:33.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.595+0000 7faa1fdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa180ae170 con 0x7faa18069bd0 2024-01-11T09:45:33.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.596+0000 7faa1e39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa18155c50 0x7faa180c9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:33.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.596+0000 7faa1e39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa18155c50 0x7faa180c9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51988/0 (socket says 172.21.15.27:51988) 2024-01-11T09:45:33.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.596+0000 7faa1e39c700 1 -- 172.21.15.27:0/1871171187 learned_addr learned my addr 172.21.15.27:0/1871171187 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:33.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.596+0000 7faa1db9b700 1 --2- 172.21.15.27:0/1871171187 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faa18069bd0 0x7faa1813f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:33.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.596+0000 7faa1d39a700 1 --2- 172.21.15.27:0/1871171187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa180c6de0 0x7faa18152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:33.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.596+0000 7faa1e39c700 1 -- 172.21.15.27:0/1871171187 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faa18069bd0 msgr2=0x7faa1813f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:33.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.596+0000 7faa1e39c700 1 --2- 172.21.15.27:0/1871171187 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faa18069bd0 0x7faa1813f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:33.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.596+0000 7faa1e39c700 1 -- 172.21.15.27:0/1871171187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa180c6de0 msgr2=0x7faa18152b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:33.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.596+0000 7faa1e39c700 1 --2- 172.21.15.27:0/1871171187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa180c6de0 0x7faa18152b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:33.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.596+0000 7faa1e39c700 1 -- 172.21.15.27:0/1871171187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa08009420 con 0x7faa18155c50 2024-01-11T09:45:33.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.596+0000 7faa1e39c700 1 --2- 172.21.15.27:0/1871171187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa18155c50 0x7faa180c9900 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7faa10009fd0 tx=0x7faa10019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:33.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.597+0000 7faa0dffb700 1 -- 172.21.15.27:0/1871171187 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa10027470 con 0x7faa18155c50 2024-01-11T09:45:33.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.597+0000 7faa0dffb700 1 -- 172.21.15.27:0/1871171187 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa100275f0 con 0x7faa18155c50 2024-01-11T09:45:33.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.597+0000 7faa0dffb700 1 -- 172.21.15.27:0/1871171187 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa100279b0 con 0x7faa18155c50 2024-01-11T09:45:33.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.597+0000 7faa1fdff700 1 -- 172.21.15.27:0/1871171187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa10009a20 con 0x7faa18155c50 2024-01-11T09:45:33.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.597+0000 7faa1fdff700 1 -- 172.21.15.27:0/1871171187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa180aa8e0 con 0x7faa18155c50 2024-01-11T09:45:33.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.598+0000 7faa1fdff700 1 -- 172.21.15.27:0/1871171187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa180aa740 con 0x7faa18155c50 2024-01-11T09:45:33.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.599+0000 7faa0dffb700 1 -- 172.21.15.27:0/1871171187 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7faa10024070 con 0x7faa18155c50 2024-01-11T09:45:33.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.600+0000 7faa0dffb700 1 --2- 172.21.15.27:0/1871171187 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa9fc066f30 0x7fa9fc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:33.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.600+0000 7faa0dffb700 1 -- 172.21.15.27:0/1871171187 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5817+0+0 (secure 0 0 0) 0x7faa10086f10 con 0x7faa18155c50 2024-01-11T09:45:33.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.600+0000 7faa1db9b700 1 --2- 172.21.15.27:0/1871171187 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa9fc066f30 0x7fa9fc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:33.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.601+0000 7faa1db9b700 1 --2- 172.21.15.27:0/1871171187 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa9fc066f30 0x7fa9fc0693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faa14001360 tx=0x7faa14009040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.604+0000 7faa0dffb700 1 -- 172.21.15.27:0/1871171187 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faa180aa740 con 0x7faa18155c50 2024-01-11T09:45:33.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.806+0000 7faa1fdff700 1 -- 172.21.15.27:0/1871171187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faa180aa740 con 0x7faa18155c50 2024-01-11T09:45:33.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.807+0000 7faa0dffb700 1 -- 172.21.15.27:0/1871171187 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v214) v1 ==== 93+0+10 (secure 0 0 0) 0x7faa180aa740 con 0x7faa18155c50 2024-01-11T09:45:33.799 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T09:45:33.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.809+0000 7faa1fdff700 1 -- 172.21.15.27:0/1871171187 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa9fc066f30 msgr2=0x7fa9fc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:33.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.809+0000 7faa1fdff700 1 --2- 172.21.15.27:0/1871171187 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa9fc066f30 0x7fa9fc0693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faa14001360 tx=0x7faa14009040 comp rx=0 tx=0).stop 2024-01-11T09:45:33.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.809+0000 7faa1fdff700 1 -- 172.21.15.27:0/1871171187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa18155c50 msgr2=0x7faa180c9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:33.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.809+0000 7faa1fdff700 1 --2- 172.21.15.27:0/1871171187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa18155c50 0x7faa180c9900 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7faa10009fd0 tx=0x7faa10019000 comp rx=0 tx=0).stop 2024-01-11T09:45:33.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.809+0000 7faa1fdff700 1 -- 172.21.15.27:0/1871171187 shutdown_connections 2024-01-11T09:45:33.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.809+0000 7faa1fdff700 1 --2- 172.21.15.27:0/1871171187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa18155c50 0x7faa180c9900 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:33.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.809+0000 7faa1fdff700 1 --2- 172.21.15.27:0/1871171187 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faa18069bd0 0x7faa1813f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:33.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.809+0000 7faa1fdff700 1 --2- 172.21.15.27:0/1871171187 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa9fc066f30 0x7fa9fc0693e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:33.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.809+0000 7faa1fdff700 1 --2- 172.21.15.27:0/1871171187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa180c6de0 0x7faa18152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:33.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.809+0000 7faa1fdff700 1 -- 172.21.15.27:0/1871171187 >> 172.21.15.27:0/1871171187 conn(0x7faa180680a0 msgr2=0x7faa180b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:33.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.809+0000 7faa1fdff700 1 -- 172.21.15.27:0/1871171187 shutdown_connections 2024-01-11T09:45:33.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:33.809+0000 7faa1fdff700 1 -- 172.21.15.27:0/1871171187 wait complete. 2024-01-11T09:45:33.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T09:45:34.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.030+0000 7fcfb85ff700 1 Processor -- start 2024-01-11T09:45:34.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.030+0000 7fcfb85ff700 1 -- start start 2024-01-11T09:45:34.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.031+0000 7fcfb85ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcfb0157410 0x7fcfb00ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:34.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.031+0000 7fcfb85ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcfb01577e0 0x7fcfb00ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:34.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.031+0000 7fcfb85ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfb00699d0 con 0x7fcfb0069540 2024-01-11T09:45:34.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.031+0000 7fcfb85ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfb0069cb0 con 0x7fcfb01577e0 2024-01-11T09:45:34.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.031+0000 7fcfb85ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfb0069f90 con 0x7fcfb0157410 2024-01-11T09:45:34.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.031+0000 7fcfb5b9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcfb01577e0 0x7fcfb00ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:34.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.032+0000 7fcfb6b9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcfb0069540 0x7fcfb015b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41624/0 (socket says 172.21.15.27:41624) 2024-01-11T09:45:34.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.032+0000 7fcfb6b9c700 1 -- 172.21.15.27:0/4142736465 learned_addr learned my addr 172.21.15.27:0/4142736465 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:34.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.032+0000 7fcfb639b700 1 --2- 172.21.15.27:0/4142736465 >> v2:172.21.15.77:3300/0 conn(0x7fcfb0157410 0x7fcfb00ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:34.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.032+0000 7fcfb5b9a700 1 -- 172.21.15.27:0/4142736465 >> v2:172.21.15.77:3300/0 conn(0x7fcfb0157410 msgr2=0x7fcfb00ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:34.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.032+0000 7fcfb5b9a700 1 --2- 172.21.15.27:0/4142736465 >> v2:172.21.15.77:3300/0 conn(0x7fcfb0157410 0x7fcfb00ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.032+0000 7fcfb5b9a700 1 -- 172.21.15.27:0/4142736465 >> v1:172.21.15.27:6789/0 conn(0x7fcfb0069540 legacy=0x7fcfb015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:34.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.032+0000 7fcfb5b9a700 1 -- 172.21.15.27:0/4142736465 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb006a270 con 0x7fcfb01577e0 2024-01-11T09:45:34.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.032+0000 7fcfb5399700 1 -- 172.21.15.27:0/4142736465 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966872550 0 0) 0x7fcfb00699d0 con 0x7fcfb0069540 2024-01-11T09:45:34.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.032+0000 7fcfb5b9a700 1 --2- 172.21.15.27:0/4142736465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfb01577e0 0x7fcfb00ae000 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0070c90 tx=0x7fcfa00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ece03a75398c50 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:34.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.033+0000 7fcfb5399700 1 -- 172.21.15.27:0/4142736465 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa000b6b0 con 0x7fcfb01577e0 2024-01-11T09:45:34.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.033+0000 7fcfb5399700 1 -- 172.21.15.27:0/4142736465 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa000b810 con 0x7fcfb01577e0 2024-01-11T09:45:34.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.033+0000 7fcfb5399700 1 -- 172.21.15.27:0/4142736465 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa000bbb0 con 0x7fcfb01577e0 2024-01-11T09:45:34.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.033+0000 7fcfb85ff700 1 -- 172.21.15.27:0/4142736465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfb01577e0 msgr2=0x7fcfb00ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:34.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.033+0000 7fcfb85ff700 1 --2- 172.21.15.27:0/4142736465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfb01577e0 0x7fcfb00ae000 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0070c90 tx=0x7fcfa00091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.033+0000 7fcfb85ff700 1 -- 172.21.15.27:0/4142736465 shutdown_connections 2024-01-11T09:45:34.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.033+0000 7fcfb85ff700 1 --2- 172.21.15.27:0/4142736465 >> v2:172.21.15.77:3300/0 conn(0x7fcfb0157410 0x7fcfb00ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.033+0000 7fcfb85ff700 1 --2- 172.21.15.27:0/4142736465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfb01577e0 0x7fcfb00ae000 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.033+0000 7fcfb85ff700 1 -- 172.21.15.27:0/4142736465 >> 172.21.15.27:0/4142736465 conn(0x7fcfb00680a0 msgr2=0x7fcfb00cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:34.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.033+0000 7fcfb85ff700 1 -- 172.21.15.27:0/4142736465 shutdown_connections 2024-01-11T09:45:34.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.033+0000 7fcfb85ff700 1 -- 172.21.15.27:0/4142736465 wait complete. 2024-01-11T09:45:34.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.035+0000 7fcfb85ff700 1 Processor -- start 2024-01-11T09:45:34.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.035+0000 7fcfb85ff700 1 -- start start 2024-01-11T09:45:34.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.036+0000 7fcfb85ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfb0069540 0x7fcfb0156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:34.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.036+0000 7fcfb85ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfb0157410 0x7fcfb00c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:34.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.036+0000 7fcfb85ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcfb01577e0 0x7fcfb00b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:34.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.036+0000 7fcfb85ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb01567d0 con 0x7fcfb0157410 2024-01-11T09:45:34.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.036+0000 7fcfb85ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb016e080 con 0x7fcfb0069540 2024-01-11T09:45:34.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.036+0000 7fcfb85ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb00c54e0 con 0x7fcfb01577e0 2024-01-11T09:45:34.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.036+0000 7fcfb5b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfb0157410 0x7fcfb00c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:34.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.036+0000 7fcfb5b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfb0157410 0x7fcfb00c5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47356/0 (socket says 172.21.15.27:47356) 2024-01-11T09:45:34.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.036+0000 7fcfb5b9a700 1 -- 172.21.15.27:0/726223410 learned_addr learned my addr 172.21.15.27:0/726223410 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:34.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.036+0000 7fcfb5b9a700 1 -- 172.21.15.27:0/726223410 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcfb01577e0 msgr2=0x7fcfb00b0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:34.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.036+0000 7fcfb5b9a700 1 --2- 172.21.15.27:0/726223410 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcfb01577e0 0x7fcfb00b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.036+0000 7fcfb5b9a700 1 -- 172.21.15.27:0/726223410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfb0069540 msgr2=0x7fcfb0156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:34.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.036+0000 7fcfb5b9a700 1 --2- 172.21.15.27:0/726223410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfb0069540 0x7fcfb0156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.036+0000 7fcfb5b9a700 1 -- 172.21.15.27:0/726223410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa0009420 con 0x7fcfb0157410 2024-01-11T09:45:34.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.037+0000 7fcfb5b9a700 1 --2- 172.21.15.27:0/726223410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfb0157410 0x7fcfb00c5cb0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0013000 tx=0x7fcfa0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:34.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.037+0000 7fcfa67fc700 1 -- 172.21.15.27:0/726223410 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0011420 con 0x7fcfb0157410 2024-01-11T09:45:34.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.037+0000 7fcfa67fc700 1 -- 172.21.15.27:0/726223410 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa0011580 con 0x7fcfb0157410 2024-01-11T09:45:34.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.037+0000 7fcfa67fc700 1 -- 172.21.15.27:0/726223410 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0011940 con 0x7fcfb0157410 2024-01-11T09:45:34.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.037+0000 7fcfb85ff700 1 -- 172.21.15.27:0/726223410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb016d470 con 0x7fcfb0157410 2024-01-11T09:45:34.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.037+0000 7fcfb85ff700 1 -- 172.21.15.27:0/726223410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb016da30 con 0x7fcfb0157410 2024-01-11T09:45:34.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.038+0000 7fcfb85ff700 1 -- 172.21.15.27:0/726223410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb004c3c0 con 0x7fcfb0157410 2024-01-11T09:45:34.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.039+0000 7fcfa67fc700 1 -- 172.21.15.27:0/726223410 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fcfa0011d30 con 0x7fcfb0157410 2024-01-11T09:45:34.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.040+0000 7fcfa67fc700 1 --2- 172.21.15.27:0/726223410 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcf94067000 0x7fcf940694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:34.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.040+0000 7fcfa67fc700 1 -- 172.21.15.27:0/726223410 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5817+0+0 (secure 0 0 0) 0x7fcfa0085b00 con 0x7fcfb0157410 2024-01-11T09:45:34.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.040+0000 7fcfb639b700 1 --2- 172.21.15.27:0/726223410 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcf94067000 0x7fcf940694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:34.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.041+0000 7fcfb639b700 1 --2- 172.21.15.27:0/726223410 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcf94067000 0x7fcf940694b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcfac009a40 tx=0x7fcfac008040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:34.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.045+0000 7fcfa67fc700 1 -- 172.21.15.27:0/726223410 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcfa0087e60 con 0x7fcfb0157410 2024-01-11T09:45:34.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.248+0000 7fcfb85ff700 1 -- 172.21.15.27:0/726223410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcfb016e1c0 con 0x7fcfb0157410 2024-01-11T09:45:34.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.249+0000 7fcfa67fc700 1 -- 172.21.15.27:0/726223410 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v214) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcfa00150a0 con 0x7fcfb0157410 2024-01-11T09:45:34.240 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T09:45:34.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.251+0000 7fcfb85ff700 1 -- 172.21.15.27:0/726223410 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcf94067000 msgr2=0x7fcf940694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:34.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.251+0000 7fcfb85ff700 1 --2- 172.21.15.27:0/726223410 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcf94067000 0x7fcf940694b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcfac009a40 tx=0x7fcfac008040 comp rx=0 tx=0).stop 2024-01-11T09:45:34.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.251+0000 7fcfb85ff700 1 -- 172.21.15.27:0/726223410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfb0157410 msgr2=0x7fcfb00c5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:34.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.251+0000 7fcfb85ff700 1 --2- 172.21.15.27:0/726223410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfb0157410 0x7fcfb00c5cb0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0013000 tx=0x7fcfa0017000 comp rx=0 tx=0).stop 2024-01-11T09:45:34.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.251+0000 7fcfb85ff700 1 -- 172.21.15.27:0/726223410 shutdown_connections 2024-01-11T09:45:34.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.251+0000 7fcfb85ff700 1 --2- 172.21.15.27:0/726223410 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcfb01577e0 0x7fcfb00b0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.251+0000 7fcfb85ff700 1 --2- 172.21.15.27:0/726223410 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcf94067000 0x7fcf940694b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.251+0000 7fcfb85ff700 1 --2- 172.21.15.27:0/726223410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfb0069540 0x7fcfb0156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.251+0000 7fcfb85ff700 1 --2- 172.21.15.27:0/726223410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfb0157410 0x7fcfb00c5cb0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.251+0000 7fcfb85ff700 1 -- 172.21.15.27:0/726223410 >> 172.21.15.27:0/726223410 conn(0x7fcfb00680a0 msgr2=0x7fcfb00caa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:34.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.251+0000 7fcfb85ff700 1 -- 172.21.15.27:0/726223410 shutdown_connections 2024-01-11T09:45:34.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.251+0000 7fcfb85ff700 1 -- 172.21.15.27:0/726223410 wait complete. 2024-01-11T09:45:34.261 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T09:45:34.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.472+0000 7f912e9ff700 1 Processor -- start 2024-01-11T09:45:34.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.472+0000 7f912e9ff700 1 -- start start 2024-01-11T09:45:34.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.473+0000 7f912e9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f91280ccd70 0x7f91280cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:34.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.473+0000 7f912e9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f912815ae70 0x7f912815b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:34.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.473+0000 7f912e9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91280cf850 con 0x7f912815a1b0 2024-01-11T09:45:34.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.473+0000 7f912e9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f912815ac90 con 0x7f912815ae70 2024-01-11T09:45:34.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.473+0000 7f912e9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91280cec40 con 0x7f91280ccd70 2024-01-11T09:45:34.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.474+0000 7f912cf9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f912815ae70 0x7f912815b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:34.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.474+0000 7f912cf9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f912815ae70 0x7f912815b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47366/0 (socket says 172.21.15.27:47366) 2024-01-11T09:45:34.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.474+0000 7f912cf9c700 1 -- 172.21.15.27:0/1100565008 learned_addr learned my addr 172.21.15.27:0/1100565008 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:34.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.474+0000 7f91277fe700 1 --2- 172.21.15.27:0/1100565008 >> v2:172.21.15.77:3300/0 conn(0x7f91280ccd70 0x7f91280cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:34.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.474+0000 7f912cf9c700 1 -- 172.21.15.27:0/1100565008 >> v2:172.21.15.77:3300/0 conn(0x7f91280ccd70 msgr2=0x7f91280cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:34.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.474+0000 7f912cf9c700 1 --2- 172.21.15.27:0/1100565008 >> v2:172.21.15.77:3300/0 conn(0x7f91280ccd70 0x7f91280cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.474+0000 7f912cf9c700 1 -- 172.21.15.27:0/1100565008 >> v1:172.21.15.77:6789/0 conn(0x7f912815a1b0 legacy=0x7f912815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:34.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.474+0000 7f912cf9c700 1 -- 172.21.15.27:0/1100565008 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91280cef20 con 0x7f912815ae70 2024-01-11T09:45:34.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.474+0000 7f912cf9c700 1 --2- 172.21.15.27:0/1100565008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912815ae70 0x7f912815b2a0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f91280696f0 tx=0x7f9120009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e267974279b1d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:34.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.475+0000 7f9126ffd700 1 -- 172.21.15.27:0/1100565008 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912000e070 con 0x7f912815ae70 2024-01-11T09:45:34.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.475+0000 7f9126ffd700 1 -- 172.21.15.27:0/1100565008 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912000ba20 con 0x7f912815ae70 2024-01-11T09:45:34.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.475+0000 7f9126ffd700 1 -- 172.21.15.27:0/1100565008 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91200092e0 con 0x7f912815ae70 2024-01-11T09:45:34.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.475+0000 7f912e9ff700 1 -- 172.21.15.27:0/1100565008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912815ae70 msgr2=0x7f912815b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:34.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.475+0000 7f912e9ff700 1 --2- 172.21.15.27:0/1100565008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912815ae70 0x7f912815b2a0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f91280696f0 tx=0x7f9120009840 comp rx=0 tx=0).stop 2024-01-11T09:45:34.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.475+0000 7f912e9ff700 1 -- 172.21.15.27:0/1100565008 shutdown_connections 2024-01-11T09:45:34.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.475+0000 7f912e9ff700 1 --2- 172.21.15.27:0/1100565008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912815ae70 0x7f912815b2a0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.475+0000 7f912e9ff700 1 --2- 172.21.15.27:0/1100565008 >> v2:172.21.15.77:3300/0 conn(0x7f91280ccd70 0x7f91280cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.475+0000 7f912e9ff700 1 -- 172.21.15.27:0/1100565008 >> 172.21.15.27:0/1100565008 conn(0x7f912806ed40 msgr2=0x7f912806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:34.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.475+0000 7f912e9ff700 1 -- 172.21.15.27:0/1100565008 shutdown_connections 2024-01-11T09:45:34.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.476+0000 7f912e9ff700 1 -- 172.21.15.27:0/1100565008 wait complete. 2024-01-11T09:45:34.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.477+0000 7f912e9ff700 1 Processor -- start 2024-01-11T09:45:34.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.477+0000 7f912e9ff700 1 -- start start 2024-01-11T09:45:34.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.478+0000 7f912e9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f91280ccd70 0x7f912815de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:34.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.478+0000 7f912e9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912815a1b0 0x7f912815edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:34.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.478+0000 7f912e9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f912815ae70 0x7f91280c5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:34.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.478+0000 7f912e9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912816a4f0 con 0x7f912815a1b0 2024-01-11T09:45:34.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.478+0000 7f912e9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91280c78a0 con 0x7f912815ae70 2024-01-11T09:45:34.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.478+0000 7f912e9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f912816a210 con 0x7f91280ccd70 2024-01-11T09:45:34.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.478+0000 7f91277fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912815a1b0 0x7f912815edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:34.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.479+0000 7f91277fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912815a1b0 0x7f912815edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47380/0 (socket says 172.21.15.27:47380) 2024-01-11T09:45:34.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.479+0000 7f91277fe700 1 -- 172.21.15.27:0/4198778667 learned_addr learned my addr 172.21.15.27:0/4198778667 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:34.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.479+0000 7f9127fff700 1 --2- 172.21.15.27:0/4198778667 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f91280ccd70 0x7f912815de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:34.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.479+0000 7f912cf9c700 1 --2- 172.21.15.27:0/4198778667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f912815ae70 0x7f91280c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:34.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.479+0000 7f91277fe700 1 -- 172.21.15.27:0/4198778667 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f91280ccd70 msgr2=0x7f912815de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:34.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.479+0000 7f91277fe700 1 --2- 172.21.15.27:0/4198778667 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f91280ccd70 0x7f912815de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.479+0000 7f91277fe700 1 -- 172.21.15.27:0/4198778667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f912815ae70 msgr2=0x7f91280c5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:34.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.479+0000 7f91277fe700 1 --2- 172.21.15.27:0/4198778667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f912815ae70 0x7f91280c5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.479+0000 7f91277fe700 1 -- 172.21.15.27:0/4198778667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9120009ae0 con 0x7f912815a1b0 2024-01-11T09:45:34.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.479+0000 7f91277fe700 1 --2- 172.21.15.27:0/4198778667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912815a1b0 0x7f912815edc0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f911c00e950 tx=0x7f911c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:34.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.479+0000 7f910ffff700 1 -- 172.21.15.27:0/4198778667 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c013680 con 0x7f912815a1b0 2024-01-11T09:45:34.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.479+0000 7f910ffff700 1 -- 172.21.15.27:0/4198778667 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f911c0137e0 con 0x7f912815a1b0 2024-01-11T09:45:34.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.479+0000 7f910ffff700 1 -- 172.21.15.27:0/4198778667 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c013ba0 con 0x7f912815a1b0 2024-01-11T09:45:34.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.480+0000 7f912e9ff700 1 -- 172.21.15.27:0/4198778667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f912815d3b0 con 0x7f912815a1b0 2024-01-11T09:45:34.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.480+0000 7f912e9ff700 1 -- 172.21.15.27:0/4198778667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f912815d970 con 0x7f912815a1b0 2024-01-11T09:45:34.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.481+0000 7f912e9ff700 1 -- 172.21.15.27:0/4198778667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f912815db90 con 0x7f912815a1b0 2024-01-11T09:45:34.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.482+0000 7f910ffff700 1 -- 172.21.15.27:0/4198778667 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f911c0123d0 con 0x7f912815a1b0 2024-01-11T09:45:34.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.483+0000 7f910ffff700 1 --2- 172.21.15.27:0/4198778667 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9114066f30 0x7f91140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:34.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.483+0000 7f910ffff700 1 -- 172.21.15.27:0/4198778667 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f911c084740 con 0x7f912815a1b0 2024-01-11T09:45:34.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.483+0000 7f9127fff700 1 --2- 172.21.15.27:0/4198778667 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9114066f30 0x7f91140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:34.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.484+0000 7f9127fff700 1 --2- 172.21.15.27:0/4198778667 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9114066f30 0x7f91140693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9118003a80 tx=0x7f9118007ec0 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:34.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.487+0000 7f910ffff700 1 -- 172.21.15.27:0/4198778667 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f911c051920 con 0x7f912815a1b0 2024-01-11T09:45:34.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.691+0000 7f912e9ff700 1 -- 172.21.15.27:0/4198778667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f912804c3c0 con 0x7f912815a1b0 2024-01-11T09:45:34.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.692+0000 7f910ffff700 1 -- 172.21.15.27:0/4198778667 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v214) v1 ==== 100+0+11 (secure 0 0 0) 0x7f911c054f40 con 0x7f912815a1b0 2024-01-11T09:45:34.683 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T09:45:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.694+0000 7f912e9ff700 1 -- 172.21.15.27:0/4198778667 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9114066f30 msgr2=0x7f91140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.694+0000 7f912e9ff700 1 --2- 172.21.15.27:0/4198778667 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9114066f30 0x7f91140693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9118003a80 tx=0x7f9118007ec0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.694+0000 7f912e9ff700 1 -- 172.21.15.27:0/4198778667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912815a1b0 msgr2=0x7f912815edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.694+0000 7f912e9ff700 1 --2- 172.21.15.27:0/4198778667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912815a1b0 0x7f912815edc0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f911c00e950 tx=0x7f911c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.694+0000 7f912e9ff700 1 -- 172.21.15.27:0/4198778667 shutdown_connections 2024-01-11T09:45:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.694+0000 7f912e9ff700 1 --2- 172.21.15.27:0/4198778667 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9114066f30 0x7f91140693e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.694+0000 7f912e9ff700 1 --2- 172.21.15.27:0/4198778667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f912815ae70 0x7f91280c5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.694+0000 7f912e9ff700 1 --2- 172.21.15.27:0/4198778667 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f91280ccd70 0x7f912815de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.694+0000 7f912e9ff700 1 --2- 172.21.15.27:0/4198778667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f912815a1b0 0x7f912815edc0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.694+0000 7f912e9ff700 1 -- 172.21.15.27:0/4198778667 >> 172.21.15.27:0/4198778667 conn(0x7f912806ed40 msgr2=0x7f912806f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.694+0000 7f912e9ff700 1 -- 172.21.15.27:0/4198778667 shutdown_connections 2024-01-11T09:45:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.694+0000 7f912e9ff700 1 -- 172.21.15.27:0/4198778667 wait complete. 2024-01-11T09:45:34.704 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-11T09:45:34.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.912+0000 7f1360dff700 1 Processor -- start 2024-01-11T09:45:34.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.912+0000 7f1360dff700 1 -- start start 2024-01-11T09:45:34.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.914+0000 7f1360dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f135c1596c0 0x7f135c15d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:34.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.914+0000 7f1360dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f135c0ca670 0x7f135c068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:34.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.914+0000 7f1360dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f135c068fc0 con 0x7f135c068b60 2024-01-11T09:45:34.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.914+0000 7f1360dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f135c0a9cb0 con 0x7f135c0ca670 2024-01-11T09:45:34.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.914+0000 7f1360dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f135c15e210 con 0x7f135c1596c0 2024-01-11T09:45:34.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.914+0000 7f1359d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f135c0ca670 0x7f135c068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:34.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.914+0000 7f1359d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f135c0ca670 0x7f135c068620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47384/0 (socket says 172.21.15.27:47384) 2024-01-11T09:45:34.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.914+0000 7f1359d9b700 1 -- 172.21.15.27:0/7057555 learned_addr learned my addr 172.21.15.27:0/7057555 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:34.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.914+0000 7f135a59c700 1 --2- 172.21.15.27:0/7057555 >> v2:172.21.15.77:3300/0 conn(0x7f135c1596c0 0x7f135c15d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:34.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.914+0000 7f1359d9b700 1 -- 172.21.15.27:0/7057555 >> v2:172.21.15.77:3300/0 conn(0x7f135c1596c0 msgr2=0x7f135c15d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:34.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.914+0000 7f1359d9b700 1 --2- 172.21.15.27:0/7057555 >> v2:172.21.15.77:3300/0 conn(0x7f135c1596c0 0x7f135c15d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.914+0000 7f1359d9b700 1 -- 172.21.15.27:0/7057555 >> v1:172.21.15.77:6789/0 conn(0x7f135c068b60 legacy=0x7f135c159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:34.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.914+0000 7f1359d9b700 1 -- 172.21.15.27:0/7057555 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135c15dc50 con 0x7f135c0ca670 2024-01-11T09:45:34.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.915+0000 7f1359d9b700 1 --2- 172.21.15.27:0/7057555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f135c0ca670 0x7f135c068620 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f135c0c3a20 tx=0x7f1350009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e34380113c0f732a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:34.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.915+0000 7f135959a700 1 -- 172.21.15.27:0/7057555 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1350009e70 con 0x7f135c0ca670 2024-01-11T09:45:34.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.915+0000 7f135959a700 1 -- 172.21.15.27:0/7057555 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135000b860 con 0x7f135c0ca670 2024-01-11T09:45:34.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.915+0000 7f135959a700 1 -- 172.21.15.27:0/7057555 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135000bc00 con 0x7f135c0ca670 2024-01-11T09:45:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.915+0000 7f1360dff700 1 -- 172.21.15.27:0/7057555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f135c0ca670 msgr2=0x7f135c068620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.915+0000 7f1360dff700 1 --2- 172.21.15.27:0/7057555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f135c0ca670 0x7f135c068620 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f135c0c3a20 tx=0x7f1350009250 comp rx=0 tx=0).stop 2024-01-11T09:45:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.916+0000 7f1360dff700 1 -- 172.21.15.27:0/7057555 shutdown_connections 2024-01-11T09:45:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.916+0000 7f1360dff700 1 --2- 172.21.15.27:0/7057555 >> v2:172.21.15.77:3300/0 conn(0x7f135c1596c0 0x7f135c15d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.916+0000 7f1360dff700 1 --2- 172.21.15.27:0/7057555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f135c0ca670 0x7f135c068620 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.916+0000 7f1360dff700 1 -- 172.21.15.27:0/7057555 >> 172.21.15.27:0/7057555 conn(0x7f135c0c9f70 msgr2=0x7f135c06deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.916+0000 7f1360dff700 1 -- 172.21.15.27:0/7057555 shutdown_connections 2024-01-11T09:45:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.916+0000 7f1360dff700 1 -- 172.21.15.27:0/7057555 wait complete. 2024-01-11T09:45:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.917+0000 7f1360dff700 1 Processor -- start 2024-01-11T09:45:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.917+0000 7f1360dff700 1 -- start start 2024-01-11T09:45:34.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.918+0000 7f1360dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f135c068b60 0x7f135c0c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:34.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.918+0000 7f1360dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f135c0ca670 0x7f135c16ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:34.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.918+0000 7f1360dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f135c1596c0 0x7f135c169cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:34.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.918+0000 7f1360dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c0afb60 con 0x7f135c0ca670 2024-01-11T09:45:34.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.918+0000 7f1360dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c16a1f0 con 0x7f135c068b60 2024-01-11T09:45:34.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.918+0000 7f1360dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f135c16a450 con 0x7f135c1596c0 2024-01-11T09:45:34.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.918+0000 7f1359d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f135c0ca670 0x7f135c16ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:34.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.919+0000 7f135a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f135c068b60 0x7f135c0c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:34.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.919+0000 7f135a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f135c068b60 0x7f135c0c8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:34168/0 (socket says 172.21.15.27:34168) 2024-01-11T09:45:34.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.919+0000 7f1359d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f135c0ca670 0x7f135c16ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47398/0 (socket says 172.21.15.27:47398) 2024-01-11T09:45:34.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.919+0000 7f135a59c700 1 -- 172.21.15.27:0/2833038314 learned_addr learned my addr 172.21.15.27:0/2833038314 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:34.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.919+0000 7f135ad9d700 1 --2- 172.21.15.27:0/2833038314 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f135c1596c0 0x7f135c169cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:34.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.919+0000 7f1359d9b700 1 -- 172.21.15.27:0/2833038314 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f135c1596c0 msgr2=0x7f135c169cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:34.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.919+0000 7f1359d9b700 1 --2- 172.21.15.27:0/2833038314 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f135c1596c0 0x7f135c169cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.919+0000 7f1359d9b700 1 -- 172.21.15.27:0/2833038314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f135c068b60 msgr2=0x7f135c0c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:34.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.919+0000 7f1359d9b700 1 --2- 172.21.15.27:0/2833038314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f135c068b60 0x7f135c0c8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:34.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.919+0000 7f1359d9b700 1 -- 172.21.15.27:0/2833038314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13500094f0 con 0x7f135c0ca670 2024-01-11T09:45:34.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.919+0000 7f1359d9b700 1 --2- 172.21.15.27:0/2833038314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f135c0ca670 0x7f135c16ae40 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f135c069a50 tx=0x7f1350017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:34.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.919+0000 7f134a7fc700 1 -- 172.21.15.27:0/2833038314 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135000b9d0 con 0x7f135c0ca670 2024-01-11T09:45:34.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.920+0000 7f134a7fc700 1 -- 172.21.15.27:0/2833038314 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135000bb50 con 0x7f135c0ca670 2024-01-11T09:45:34.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.920+0000 7f1360dff700 1 -- 172.21.15.27:0/2833038314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f135c16a730 con 0x7f135c0ca670 2024-01-11T09:45:34.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.920+0000 7f134a7fc700 1 -- 172.21.15.27:0/2833038314 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1350025670 con 0x7f135c0ca670 2024-01-11T09:45:34.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.920+0000 7f1360dff700 1 -- 172.21.15.27:0/2833038314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f135c0c29d0 con 0x7f135c0ca670 2024-01-11T09:45:34.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.921+0000 7f1360dff700 1 -- 172.21.15.27:0/2833038314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f135c0af930 con 0x7f135c0ca670 2024-01-11T09:45:34.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.922+0000 7f134a7fc700 1 -- 172.21.15.27:0/2833038314 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f1350023070 con 0x7f135c0ca670 2024-01-11T09:45:34.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.923+0000 7f134a7fc700 1 --2- 172.21.15.27:0/2833038314 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f133c066f30 0x7f133c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:34.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.923+0000 7f134a7fc700 1 -- 172.21.15.27:0/2833038314 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f1350085750 con 0x7f135c0ca670 2024-01-11T09:45:34.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.923+0000 7f135a59c700 1 --2- 172.21.15.27:0/2833038314 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f133c066f30 0x7f133c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:34.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.924+0000 7f135a59c700 1 --2- 172.21.15.27:0/2833038314 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f133c066f30 0x7f133c0693e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f135c06bfc0 tx=0x7f134c0091a0 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:34.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:34.927+0000 7f134a7fc700 1 -- 172.21.15.27:0/2833038314 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f135c0af930 con 0x7f135c0ca670 2024-01-11T09:45:35.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.132+0000 7f1360dff700 1 -- 172.21.15.27:0/2833038314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f135c0afca0 con 0x7f135c0ca670 2024-01-11T09:45:35.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.132+0000 7f134a7fc700 1 -- 172.21.15.27:0/2833038314 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v214) v1 ==== 84+0+209 (secure 0 0 0) 0x7f1350052950 con 0x7f135c0ca670 2024-01-11T09:45:35.124 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:35.124 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T09:45:35.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:35.125 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-11T09:45:35.125 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-11T09:45:35.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:35.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:35.125 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 26, 2024-01-11T09:45:35.125 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": "cephfs_metadata" 2024-01-11T09:45:35.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:35.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:35.125 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 27, 2024-01-11T09:45:35.125 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": "cephfs_data" 2024-01-11T09:45:35.125 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:35.125 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T09:45:35.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.135+0000 7f1360dff700 1 -- 172.21.15.27:0/2833038314 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f133c066f30 msgr2=0x7f133c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:35.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.135+0000 7f1360dff700 1 --2- 172.21.15.27:0/2833038314 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f133c066f30 0x7f133c0693e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f135c06bfc0 tx=0x7f134c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:35.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.135+0000 7f1360dff700 1 -- 172.21.15.27:0/2833038314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f135c0ca670 msgr2=0x7f135c16ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:35.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.135+0000 7f1360dff700 1 --2- 172.21.15.27:0/2833038314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f135c0ca670 0x7f135c16ae40 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f135c069a50 tx=0x7f1350017000 comp rx=0 tx=0).stop 2024-01-11T09:45:35.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.135+0000 7f1360dff700 1 -- 172.21.15.27:0/2833038314 shutdown_connections 2024-01-11T09:45:35.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.135+0000 7f1360dff700 1 --2- 172.21.15.27:0/2833038314 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f135c1596c0 0x7f135c169cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:35.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.135+0000 7f1360dff700 1 --2- 172.21.15.27:0/2833038314 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f133c066f30 0x7f133c0693e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:35.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.135+0000 7f1360dff700 1 --2- 172.21.15.27:0/2833038314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f135c068b60 0x7f135c0c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:35.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.135+0000 7f1360dff700 1 --2- 172.21.15.27:0/2833038314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f135c0ca670 0x7f135c16ae40 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:35.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.135+0000 7f1360dff700 1 -- 172.21.15.27:0/2833038314 >> 172.21.15.27:0/2833038314 conn(0x7f135c0c9f70 msgr2=0x7f135c068120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:35.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.135+0000 7f1360dff700 1 -- 172.21.15.27:0/2833038314 shutdown_connections 2024-01-11T09:45:35.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.135+0000 7f1360dff700 1 -- 172.21.15.27:0/2833038314 wait complete. 2024-01-11T09:45:35.144 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2024-01-11T09:45:35.145 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2024-01-11T09:45:35.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.356+0000 7f3e83dff700 1 Processor -- start 2024-01-11T09:45:35.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.357+0000 7f3e83dff700 1 -- start start 2024-01-11T09:45:35.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.358+0000 7f3e83dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3e7c159e60 0x7f3e7c15aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:35.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.358+0000 7f3e83dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3e7c15b400 0x7f3e7c0685e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:35.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.358+0000 7f3e83dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7c15c4b0 con 0x7f3e7c15a7a0 2024-01-11T09:45:35.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.358+0000 7f3e83dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e7c15d050 con 0x7f3e7c15b400 2024-01-11T09:45:35.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.358+0000 7f3e83dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7c15b8a0 con 0x7f3e7c159e60 2024-01-11T09:45:35.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.358+0000 7f3e81b9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3e7c15a7a0 0x7f3e7c15c940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41628/0 (socket says 172.21.15.27:41628) 2024-01-11T09:45:35.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.358+0000 7f3e81b9b700 1 -- 172.21.15.27:0/3683975318 learned_addr learned my addr 172.21.15.27:0/3683975318 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:35.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.358+0000 7f3e8239c700 1 --2- 172.21.15.27:0/3683975318 >> v2:172.21.15.27:3300/0 conn(0x7f3e7c15b400 0x7f3e7c0685e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:35.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.358+0000 7f3e8139a700 1 --2- 172.21.15.27:0/3683975318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3e7c159e60 0x7f3e7c15aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:35.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.359+0000 7f3e80b99700 1 -- 172.21.15.27:0/3683975318 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501023084 0 0) 0x7f3e7c15c4b0 con 0x7f3e7c15a7a0 2024-01-11T09:45:35.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.359+0000 7f3e80b99700 1 -- 172.21.15.27:0/3683975318 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e68003570 con 0x7f3e7c15a7a0 2024-01-11T09:45:35.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.359+0000 7f3e8239c700 1 -- 172.21.15.27:0/3683975318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3e7c159e60 msgr2=0x7f3e7c15aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:35.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.359+0000 7f3e8239c700 1 --2- 172.21.15.27:0/3683975318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3e7c159e60 0x7f3e7c15aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:35.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.359+0000 7f3e8239c700 1 -- 172.21.15.27:0/3683975318 >> v1:172.21.15.27:6789/0 conn(0x7f3e7c15a7a0 legacy=0x7f3e7c15c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:35.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.359+0000 7f3e8239c700 1 -- 172.21.15.27:0/3683975318 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e7c15bb80 con 0x7f3e7c15b400 2024-01-11T09:45:35.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.359+0000 7f3e8239c700 1 --2- 172.21.15.27:0/3683975318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e7c15b400 0x7f3e7c0685e0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f3e74009e60 tx=0x7f3e74009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9527fa5b92ad957d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:35.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.359+0000 7f3e80b99700 1 -- 172.21.15.27:0/3683975318 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7400e070 con 0x7f3e7c15b400 2024-01-11T09:45:35.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.359+0000 7f3e80b99700 1 -- 172.21.15.27:0/3683975318 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e7400bab0 con 0x7f3e7c15b400 2024-01-11T09:45:35.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.359+0000 7f3e80b99700 1 -- 172.21.15.27:0/3683975318 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7400be70 con 0x7f3e7c15b400 2024-01-11T09:45:35.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.360+0000 7f3e83dff700 1 -- 172.21.15.27:0/3683975318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e7c15b400 msgr2=0x7f3e7c0685e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:35.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.360+0000 7f3e83dff700 1 --2- 172.21.15.27:0/3683975318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e7c15b400 0x7f3e7c0685e0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f3e74009e60 tx=0x7f3e74009780 comp rx=0 tx=0).stop 2024-01-11T09:45:35.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.360+0000 7f3e83dff700 1 -- 172.21.15.27:0/3683975318 shutdown_connections 2024-01-11T09:45:35.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.360+0000 7f3e83dff700 1 --2- 172.21.15.27:0/3683975318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e7c15b400 0x7f3e7c0685e0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:35.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.360+0000 7f3e83dff700 1 --2- 172.21.15.27:0/3683975318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3e7c159e60 0x7f3e7c15aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:35.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.360+0000 7f3e83dff700 1 -- 172.21.15.27:0/3683975318 >> 172.21.15.27:0/3683975318 conn(0x7f3e7c06ed40 msgr2=0x7f3e7c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:35.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.360+0000 7f3e83dff700 1 -- 172.21.15.27:0/3683975318 shutdown_connections 2024-01-11T09:45:35.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.360+0000 7f3e83dff700 1 -- 172.21.15.27:0/3683975318 wait complete. 2024-01-11T09:45:35.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.362+0000 7f3e83dff700 1 Processor -- start 2024-01-11T09:45:35.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.362+0000 7f3e83dff700 1 -- start start 2024-01-11T09:45:35.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.363+0000 7f3e83dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e7c15a7a0 0x7f3e7c15fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:35.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.363+0000 7f3e83dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e7c15b400 0x7f3e7c14f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:35.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.363+0000 7f3e83dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3e7c0aff70 0x7f3e7c14fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:35.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.363+0000 7f3e83dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7c1694a0 con 0x7f3e7c15a7a0 2024-01-11T09:45:35.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.363+0000 7f3e83dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7c0b0450 con 0x7f3e7c15b400 2024-01-11T09:45:35.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.363+0000 7f3e83dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7c0b0730 con 0x7f3e7c0aff70 2024-01-11T09:45:35.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.363+0000 7f3e81b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e7c15a7a0 0x7f3e7c15fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:35.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.363+0000 7f3e81b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e7c15a7a0 0x7f3e7c15fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47406/0 (socket says 172.21.15.27:47406) 2024-01-11T09:45:35.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.363+0000 7f3e81b9b700 1 -- 172.21.15.27:0/1154198344 learned_addr learned my addr 172.21.15.27:0/1154198344 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:35.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.363+0000 7f3e8239c700 1 --2- 172.21.15.27:0/1154198344 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3e7c0aff70 0x7f3e7c14fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:35.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.363+0000 7f3e8139a700 1 --2- 172.21.15.27:0/1154198344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e7c15b400 0x7f3e7c14f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.363+0000 7f3e81b9b700 1 -- 172.21.15.27:0/1154198344 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3e7c0aff70 msgr2=0x7f3e7c14fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.363+0000 7f3e81b9b700 1 --2- 172.21.15.27:0/1154198344 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3e7c0aff70 0x7f3e7c14fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.364+0000 7f3e81b9b700 1 -- 172.21.15.27:0/1154198344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e7c15b400 msgr2=0x7f3e7c14f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.364+0000 7f3e81b9b700 1 --2- 172.21.15.27:0/1154198344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e7c15b400 0x7f3e7c14f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.364+0000 7f3e81b9b700 1 -- 172.21.15.27:0/1154198344 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e74009a20 con 0x7f3e7c15a7a0 2024-01-11T09:45:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.364+0000 7f3e81b9b700 1 --2- 172.21.15.27:0/1154198344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e7c15a7a0 0x7f3e7c15fc20 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f3e7800c7c0 tx=0x7f3e78007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.364+0000 7f3e71ffb700 1 -- 172.21.15.27:0/1154198344 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e78005b30 con 0x7f3e7c15a7a0 2024-01-11T09:45:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.364+0000 7f3e71ffb700 1 -- 172.21.15.27:0/1154198344 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e78005c90 con 0x7f3e7c15a7a0 2024-01-11T09:45:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.364+0000 7f3e83dff700 1 -- 172.21.15.27:0/1154198344 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e7c0b0a10 con 0x7f3e7c15a7a0 2024-01-11T09:45:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.364+0000 7f3e71ffb700 1 -- 172.21.15.27:0/1154198344 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e78007340 con 0x7f3e7c15a7a0 2024-01-11T09:45:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.364+0000 7f3e83dff700 1 -- 172.21.15.27:0/1154198344 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e7c0b2630 con 0x7f3e7c15a7a0 2024-01-11T09:45:35.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.365+0000 7f3e83dff700 1 -- 172.21.15.27:0/1154198344 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e7c04c3c0 con 0x7f3e7c15a7a0 2024-01-11T09:45:35.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.366+0000 7f3e71ffb700 1 -- 172.21.15.27:0/1154198344 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3e7801b400 con 0x7f3e7c15a7a0 2024-01-11T09:45:35.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.367+0000 7f3e71ffb700 1 --2- 172.21.15.27:0/1154198344 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3e60066f30 0x7f3e600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:35.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.367+0000 7f3e71ffb700 1 -- 172.21.15.27:0/1154198344 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f3e78084130 con 0x7f3e7c15a7a0 2024-01-11T09:45:35.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.367+0000 7f3e8139a700 1 --2- 172.21.15.27:0/1154198344 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3e60066f30 0x7f3e600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:35.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.368+0000 7f3e8139a700 1 --2- 172.21.15.27:0/1154198344 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3e60066f30 0x7f3e600693e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0ceb70 tx=0x7f3e6c008040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:35.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.371+0000 7f3e71ffb700 1 -- 172.21.15.27:0/1154198344 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3e78051310 con 0x7f3e7c15a7a0 2024-01-11T09:45:35.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.575+0000 7f3e83dff700 1 -- 172.21.15.27:0/1154198344 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3e7c15be60 con 0x7f3e7c15a7a0 2024-01-11T09:45:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.940+0000 7f3e71ffb700 1 -- 172.21.15.27:0/1154198344 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v215) v1 ==== 144+0+0 (secure 0 0 0) 0x7f3e78054930 con 0x7f3e7c15a7a0 2024-01-11T09:45:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.944+0000 7f3e83dff700 1 -- 172.21.15.27:0/1154198344 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3e60066f30 msgr2=0x7f3e600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.944+0000 7f3e83dff700 1 --2- 172.21.15.27:0/1154198344 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3e60066f30 0x7f3e600693e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0ceb70 tx=0x7f3e6c008040 comp rx=0 tx=0).stop 2024-01-11T09:45:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.944+0000 7f3e83dff700 1 -- 172.21.15.27:0/1154198344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e7c15a7a0 msgr2=0x7f3e7c15fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.944+0000 7f3e83dff700 1 --2- 172.21.15.27:0/1154198344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e7c15a7a0 0x7f3e7c15fc20 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f3e7800c7c0 tx=0x7f3e78007e00 comp rx=0 tx=0).stop 2024-01-11T09:45:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.944+0000 7f3e83dff700 1 -- 172.21.15.27:0/1154198344 shutdown_connections 2024-01-11T09:45:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.944+0000 7f3e83dff700 1 --2- 172.21.15.27:0/1154198344 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3e7c0aff70 0x7f3e7c14fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:35.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.944+0000 7f3e83dff700 1 --2- 172.21.15.27:0/1154198344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e7c15a7a0 0x7f3e7c15fc20 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:35.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.944+0000 7f3e83dff700 1 --2- 172.21.15.27:0/1154198344 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3e60066f30 0x7f3e600693e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:35.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.944+0000 7f3e83dff700 1 --2- 172.21.15.27:0/1154198344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e7c15b400 0x7f3e7c14f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:35.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.944+0000 7f3e83dff700 1 -- 172.21.15.27:0/1154198344 >> 172.21.15.27:0/1154198344 conn(0x7f3e7c06ed40 msgr2=0x7f3e7c0d0950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:35.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.945+0000 7f3e83dff700 1 -- 172.21.15.27:0/1154198344 shutdown_connections 2024-01-11T09:45:35.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:35.945+0000 7f3e83dff700 1 -- 172.21.15.27:0/1154198344 wait complete. 2024-01-11T09:45:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'backup_fs_metadata' created 2024-01-11T09:45:35.958 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-11T09:45:36.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.173+0000 7f37a59ff700 1 Processor -- start 2024-01-11T09:45:36.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.173+0000 7f37a59ff700 1 -- start start 2024-01-11T09:45:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.175+0000 7f37a59ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f37a00c8860 0x7f37a0158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.175+0000 7f37a59ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f37a00b19d0 0x7f37a00ae230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.175+0000 7f37a59ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a00cabe0 con 0x7f37a0158600 2024-01-11T09:45:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.175+0000 7f37a59ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a00ae770 con 0x7f37a00c8860 2024-01-11T09:45:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.175+0000 7f37a59ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a00ae960 con 0x7f37a00b19d0 2024-01-11T09:45:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.175+0000 7f379effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f37a00c8860 0x7f37a0158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.175+0000 7f379effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f37a00c8860 0x7f37a0158ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47422/0 (socket says 172.21.15.27:47422) 2024-01-11T09:45:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.175+0000 7f379e7fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f37a0158600 0x7f37a00b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41642/0 (socket says 172.21.15.27:41642) 2024-01-11T09:45:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.175+0000 7f379effd700 1 -- 172.21.15.27:0/595941434 learned_addr learned my addr 172.21.15.27:0/595941434 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.175+0000 7f379f7fe700 1 --2- 172.21.15.27:0/595941434 >> v2:172.21.15.77:3300/0 conn(0x7f37a00b19d0 0x7f37a00ae230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.176+0000 7f379effd700 1 -- 172.21.15.27:0/595941434 >> v2:172.21.15.77:3300/0 conn(0x7f37a00b19d0 msgr2=0x7f37a00ae230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.176+0000 7f379effd700 1 --2- 172.21.15.27:0/595941434 >> v2:172.21.15.77:3300/0 conn(0x7f37a00b19d0 0x7f37a00ae230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.176+0000 7f379effd700 1 -- 172.21.15.27:0/595941434 >> v1:172.21.15.27:6789/0 conn(0x7f37a0158600 legacy=0x7f37a00b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.176+0000 7f379effd700 1 -- 172.21.15.27:0/595941434 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a00ca590 con 0x7f37a00c8860 2024-01-11T09:45:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.176+0000 7f379dffb700 1 -- 172.21.15.27:0/595941434 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269648241 0 0) 0x7f37a00cabe0 con 0x7f37a0158600 2024-01-11T09:45:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.176+0000 7f379effd700 1 --2- 172.21.15.27:0/595941434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a00c8860 0x7f37a0158ca0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f37a0155670 tx=0x7f3790009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a27d7bd32ec1ad3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.176+0000 7f379dffb700 1 -- 172.21.15.27:0/595941434 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3790009e70 con 0x7f37a00c8860 2024-01-11T09:45:36.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.176+0000 7f379dffb700 1 -- 172.21.15.27:0/595941434 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379000b860 con 0x7f37a00c8860 2024-01-11T09:45:36.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.176+0000 7f379dffb700 1 -- 172.21.15.27:0/595941434 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379000bc00 con 0x7f37a00c8860 2024-01-11T09:45:36.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.177+0000 7f37a59ff700 1 -- 172.21.15.27:0/595941434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a00c8860 msgr2=0x7f37a0158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:36.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.177+0000 7f37a59ff700 1 --2- 172.21.15.27:0/595941434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a00c8860 0x7f37a0158ca0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f37a0155670 tx=0x7f3790009250 comp rx=0 tx=0).stop 2024-01-11T09:45:36.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.177+0000 7f37a59ff700 1 -- 172.21.15.27:0/595941434 shutdown_connections 2024-01-11T09:45:36.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.177+0000 7f37a59ff700 1 --2- 172.21.15.27:0/595941434 >> v2:172.21.15.77:3300/0 conn(0x7f37a00b19d0 0x7f37a00ae230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:36.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.177+0000 7f37a59ff700 1 --2- 172.21.15.27:0/595941434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a00c8860 0x7f37a0158ca0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:36.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.177+0000 7f37a59ff700 1 -- 172.21.15.27:0/595941434 >> 172.21.15.27:0/595941434 conn(0x7f37a00680a0 msgr2=0x7f37a0069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:36.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.177+0000 7f37a59ff700 1 -- 172.21.15.27:0/595941434 shutdown_connections 2024-01-11T09:45:36.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.177+0000 7f37a59ff700 1 -- 172.21.15.27:0/595941434 wait complete. 2024-01-11T09:45:36.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.178+0000 7f37a59ff700 1 Processor -- start 2024-01-11T09:45:36.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.179+0000 7f37a59ff700 1 -- start start 2024-01-11T09:45:36.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.179+0000 7f37a59ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f37a00b19d0 0x7f37a00b1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:36.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.180+0000 7f37a59ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a00c8860 0x7f37a0154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:36.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.180+0000 7f37a59ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37a0158600 0x7f37a015c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:36.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.180+0000 7f37a59ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a015e170 con 0x7f37a00c8860 2024-01-11T09:45:36.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.180+0000 7f37a59ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a015e300 con 0x7f37a0158600 2024-01-11T09:45:36.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.180+0000 7f37a59ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a00b1860 con 0x7f37a00b19d0 2024-01-11T09:45:36.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.180+0000 7f379e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a00c8860 0x7f37a0154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:36.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.180+0000 7f379e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a00c8860 0x7f37a0154d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47432/0 (socket says 172.21.15.27:47432) 2024-01-11T09:45:36.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.180+0000 7f379e7fc700 1 -- 172.21.15.27:0/1579967050 learned_addr learned my addr 172.21.15.27:0/1579967050 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:36.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.180+0000 7f379effd700 1 --2- 172.21.15.27:0/1579967050 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f37a00b19d0 0x7f37a00b1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:36.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.180+0000 7f379f7fe700 1 --2- 172.21.15.27:0/1579967050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37a0158600 0x7f37a015c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:36.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.180+0000 7f379e7fc700 1 -- 172.21.15.27:0/1579967050 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f37a00b19d0 msgr2=0x7f37a00b1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:36.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.180+0000 7f379e7fc700 1 --2- 172.21.15.27:0/1579967050 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f37a00b19d0 0x7f37a00b1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:36.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.180+0000 7f379e7fc700 1 -- 172.21.15.27:0/1579967050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37a0158600 msgr2=0x7f37a015c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:36.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.180+0000 7f379e7fc700 1 --2- 172.21.15.27:0/1579967050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37a0158600 0x7f37a015c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:36.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.180+0000 7f379e7fc700 1 -- 172.21.15.27:0/1579967050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37900094f0 con 0x7f37a00c8860 2024-01-11T09:45:36.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.181+0000 7f379e7fc700 1 --2- 172.21.15.27:0/1579967050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a00c8860 0x7f37a0154d20 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f379400c5b0 tx=0x7f3794001040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:36.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.181+0000 7f3786ffd700 1 -- 172.21.15.27:0/1579967050 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3794005a70 con 0x7f37a00c8860 2024-01-11T09:45:36.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.181+0000 7f3786ffd700 1 -- 172.21.15.27:0/1579967050 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3794005bd0 con 0x7f37a00c8860 2024-01-11T09:45:36.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.181+0000 7f3786ffd700 1 -- 172.21.15.27:0/1579967050 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37940070d0 con 0x7f37a00c8860 2024-01-11T09:45:36.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.181+0000 7f37a59ff700 1 -- 172.21.15.27:0/1579967050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a00b04b0 con 0x7f37a00c8860 2024-01-11T09:45:36.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.181+0000 7f37a59ff700 1 -- 172.21.15.27:0/1579967050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a00afba0 con 0x7f37a00c8860 2024-01-11T09:45:36.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.182+0000 7f37a59ff700 1 -- 172.21.15.27:0/1579967050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a004c3c0 con 0x7f37a00c8860 2024-01-11T09:45:36.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.184+0000 7f3786ffd700 1 -- 172.21.15.27:0/1579967050 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f3794007480 con 0x7f37a00c8860 2024-01-11T09:45:36.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.186+0000 7f3786ffd700 1 --2- 172.21.15.27:0/1579967050 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f378c066f30 0x7f378c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:36.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.186+0000 7f3786ffd700 1 -- 172.21.15.27:0/1579967050 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6190+0+0 (secure 0 0 0) 0x7f3794084100 con 0x7f37a00c8860 2024-01-11T09:45:36.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.186+0000 7f379effd700 1 --2- 172.21.15.27:0/1579967050 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f378c066f30 0x7f378c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:36.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.187+0000 7f379effd700 1 --2- 172.21.15.27:0/1579967050 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f378c066f30 0x7f378c0693e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f37a0155670 tx=0x7f3790017000 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:36.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.190+0000 7f3786ffd700 1 -- 172.21.15.27:0/1579967050 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3794051170 con 0x7f37a00c8860 2024-01-11T09:45:36.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.395+0000 7f37a59ff700 1 -- 172.21.15.27:0/1579967050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f37a00ae440 con 0x7f37a00c8860 2024-01-11T09:45:36.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.948+0000 7f3786ffd700 1 -- 172.21.15.27:0/1579967050 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v216) v1 ==== 176+0+0 (secure 0 0 0) 0x7f37940170d0 con 0x7f37a00c8860 2024-01-11T09:45:36.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.954+0000 7f37a59ff700 1 -- 172.21.15.27:0/1579967050 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f378c066f30 msgr2=0x7f378c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:36.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.954+0000 7f37a59ff700 1 --2- 172.21.15.27:0/1579967050 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f378c066f30 0x7f378c0693e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f37a0155670 tx=0x7f3790017000 comp rx=0 tx=0).stop 2024-01-11T09:45:36.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.954+0000 7f37a59ff700 1 -- 172.21.15.27:0/1579967050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a00c8860 msgr2=0x7f37a0154d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:36.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.954+0000 7f37a59ff700 1 --2- 172.21.15.27:0/1579967050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a00c8860 0x7f37a0154d20 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f379400c5b0 tx=0x7f3794001040 comp rx=0 tx=0).stop 2024-01-11T09:45:36.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.957+0000 7f37a59ff700 1 -- 172.21.15.27:0/1579967050 shutdown_connections 2024-01-11T09:45:36.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.957+0000 7f37a59ff700 1 --2- 172.21.15.27:0/1579967050 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f378c066f30 0x7f378c0693e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:36.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.957+0000 7f37a59ff700 1 --2- 172.21.15.27:0/1579967050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37a0158600 0x7f37a015c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:36.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.958+0000 7f37a59ff700 1 --2- 172.21.15.27:0/1579967050 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f37a00b19d0 0x7f37a00b1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:36.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.958+0000 7f37a59ff700 1 --2- 172.21.15.27:0/1579967050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a00c8860 0x7f37a0154d20 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:36.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.958+0000 7f37a59ff700 1 -- 172.21.15.27:0/1579967050 >> 172.21.15.27:0/1579967050 conn(0x7f37a00680a0 msgr2=0x7f37a006dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:36.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.961+0000 7f37a59ff700 1 -- 172.21.15.27:0/1579967050 shutdown_connections 2024-01-11T09:45:36.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:36.962+0000 7f37a59ff700 1 -- 172.21.15.27:0/1579967050 wait complete. 2024-01-11T09:45:36.960 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'backup_fs_data' created 2024-01-11T09:45:36.983 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2024-01-11T09:45:37.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.195+0000 7f8ffc3ff700 1 Processor -- start 2024-01-11T09:45:37.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.195+0000 7f8ffc3ff700 1 -- start start 2024-01-11T09:45:37.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.196+0000 7f8ffc3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8ff4067860 0x7f8ff406c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:37.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.196+0000 7f8ffc3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8ff406cd90 0x7f8ff40cae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:37.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.196+0000 7f8ffc3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff406c3c0 con 0x7f8ff4067c30 2024-01-11T09:45:37.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.196+0000 7f8ffc3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff40cb9c0 con 0x7f8ff4067860 2024-01-11T09:45:37.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.196+0000 7f8ffc3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff406f650 con 0x7f8ff406cd90 2024-01-11T09:45:37.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.196+0000 7f8ffa19b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8ff4067860 0x7f8ff406c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:37.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.196+0000 7f8ffa99c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8ff4067c30 0x7f8ff40c7900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41650/0 (socket says 172.21.15.27:41650) 2024-01-11T09:45:37.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.196+0000 7f8ffa99c700 1 -- 172.21.15.27:0/2796849086 learned_addr learned my addr 172.21.15.27:0/2796849086 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:37.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.197+0000 7f8ff999a700 1 --2- 172.21.15.27:0/2796849086 >> v2:172.21.15.77:3300/0 conn(0x7f8ff406cd90 0x7f8ff40cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:37.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.197+0000 7f8ffa19b700 1 -- 172.21.15.27:0/2796849086 >> v2:172.21.15.77:3300/0 conn(0x7f8ff406cd90 msgr2=0x7f8ff40cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:37.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.197+0000 7f8ffa19b700 1 --2- 172.21.15.27:0/2796849086 >> v2:172.21.15.77:3300/0 conn(0x7f8ff406cd90 0x7f8ff40cae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:37.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.197+0000 7f8ffa19b700 1 -- 172.21.15.27:0/2796849086 >> v1:172.21.15.27:6789/0 conn(0x7f8ff4067c30 legacy=0x7f8ff40c7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:37.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.197+0000 7f8ff9199700 1 -- 172.21.15.27:0/2796849086 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172608722 0 0) 0x7f8ff406c3c0 con 0x7f8ff4067c30 2024-01-11T09:45:37.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.197+0000 7f8ffa19b700 1 -- 172.21.15.27:0/2796849086 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff406ba90 con 0x7f8ff4067860 2024-01-11T09:45:37.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.197+0000 7f8ffa19b700 1 --2- 172.21.15.27:0/2796849086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ff4067860 0x7f8ff406c850 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f8ff40ce3d0 tx=0x7f8ff00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa15ffd38c6e710 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:37.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.198+0000 7f8ff9199700 1 -- 172.21.15.27:0/2796849086 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff000b6b0 con 0x7f8ff4067860 2024-01-11T09:45:37.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.198+0000 7f8ff9199700 1 -- 172.21.15.27:0/2796849086 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff000b810 con 0x7f8ff4067860 2024-01-11T09:45:37.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.198+0000 7f8ff9199700 1 -- 172.21.15.27:0/2796849086 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff000bbb0 con 0x7f8ff4067860 2024-01-11T09:45:37.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.198+0000 7f8ffc3ff700 1 -- 172.21.15.27:0/2796849086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ff4067860 msgr2=0x7f8ff406c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:37.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.198+0000 7f8ffc3ff700 1 --2- 172.21.15.27:0/2796849086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ff4067860 0x7f8ff406c850 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f8ff40ce3d0 tx=0x7f8ff00091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:37.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.198+0000 7f8ffc3ff700 1 -- 172.21.15.27:0/2796849086 shutdown_connections 2024-01-11T09:45:37.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.198+0000 7f8ffc3ff700 1 --2- 172.21.15.27:0/2796849086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ff4067860 0x7f8ff406c850 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:37.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.198+0000 7f8ffc3ff700 1 --2- 172.21.15.27:0/2796849086 >> v2:172.21.15.77:3300/0 conn(0x7f8ff406cd90 0x7f8ff40cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:37.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.198+0000 7f8ffc3ff700 1 -- 172.21.15.27:0/2796849086 >> 172.21.15.27:0/2796849086 conn(0x7f8ff406b1f0 msgr2=0x7f8ff40cd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:37.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.198+0000 7f8ffc3ff700 1 -- 172.21.15.27:0/2796849086 shutdown_connections 2024-01-11T09:45:37.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.198+0000 7f8ffc3ff700 1 -- 172.21.15.27:0/2796849086 wait complete. 2024-01-11T09:45:37.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.200+0000 7f8ffc3ff700 1 Processor -- start 2024-01-11T09:45:37.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.200+0000 7f8ffc3ff700 1 -- start start 2024-01-11T09:45:37.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.201+0000 7f8ffc3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ff4067860 0x7f8ff415d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:37.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.201+0000 7f8ffc3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ff4067c30 0x7f8ff4141c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:37.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.201+0000 7f8ffc3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ff406cd90 0x7f8ff40c3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:37.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.201+0000 7f8ffc3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40ad050 con 0x7f8ff4067c30 2024-01-11T09:45:37.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.201+0000 7f8ffc3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40ad330 con 0x7f8ff406cd90 2024-01-11T09:45:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.201+0000 7f8ffc3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40ad610 con 0x7f8ff4067860 2024-01-11T09:45:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.201+0000 7f8ff999a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ff4067c30 0x7f8ff4141c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.201+0000 7f8ff999a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ff4067c30 0x7f8ff4141c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47456/0 (socket says 172.21.15.27:47456) 2024-01-11T09:45:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.201+0000 7f8ff999a700 1 -- 172.21.15.27:0/3153959722 learned_addr learned my addr 172.21.15.27:0/3153959722 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.201+0000 7f8ff999a700 1 -- 172.21.15.27:0/3153959722 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ff4067860 msgr2=0x7f8ff415d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.202+0000 7f8ffa19b700 1 --2- 172.21.15.27:0/3153959722 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ff4067860 0x7f8ff415d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.201+0000 7f8ffa99c700 1 --2- 172.21.15.27:0/3153959722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ff406cd90 0x7f8ff40c3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.202+0000 7f8ff999a700 1 --2- 172.21.15.27:0/3153959722 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ff4067860 0x7f8ff415d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.202+0000 7f8ff999a700 1 -- 172.21.15.27:0/3153959722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ff406cd90 msgr2=0x7f8ff40c3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.202+0000 7f8ff999a700 1 --2- 172.21.15.27:0/3153959722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ff406cd90 0x7f8ff40c3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.202+0000 7f8ff999a700 1 -- 172.21.15.27:0/3153959722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff0009420 con 0x7f8ff4067c30 2024-01-11T09:45:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.202+0000 7f8ffa19b700 1 --2- 172.21.15.27:0/3153959722 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ff4067860 0x7f8ff415d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.202+0000 7f8ffa99c700 1 --2- 172.21.15.27:0/3153959722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ff406cd90 0x7f8ff40c3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.202+0000 7f8ff999a700 1 --2- 172.21.15.27:0/3153959722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ff4067c30 0x7f8ff4141c30 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400d860 tx=0x7f8fe4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:37.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.202+0000 7f8fea7fc700 1 -- 172.21.15.27:0/3153959722 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4009610 con 0x7f8ff4067c30 2024-01-11T09:45:37.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.202+0000 7f8fea7fc700 1 -- 172.21.15.27:0/3153959722 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe4009770 con 0x7f8ff4067c30 2024-01-11T09:45:37.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.202+0000 7f8ffc3ff700 1 -- 172.21.15.27:0/3153959722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff40ad8f0 con 0x7f8ff4067c30 2024-01-11T09:45:37.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.202+0000 7f8fea7fc700 1 -- 172.21.15.27:0/3153959722 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4009b30 con 0x7f8ff4067c30 2024-01-11T09:45:37.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.202+0000 7f8ffc3ff700 1 -- 172.21.15.27:0/3153959722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff4142170 con 0x7f8ff4067c30 2024-01-11T09:45:37.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.203+0000 7f8ffc3ff700 1 -- 172.21.15.27:0/3153959722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff4159d90 con 0x7f8ff4067c30 2024-01-11T09:45:37.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.205+0000 7f8fea7fc700 1 -- 172.21.15.27:0/3153959722 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8fe401e460 con 0x7f8ff4067c30 2024-01-11T09:45:37.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.206+0000 7f8fea7fc700 1 --2- 172.21.15.27:0/3153959722 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8fd8066f30 0x7f8fd80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:37.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.206+0000 7f8fea7fc700 1 -- 172.21.15.27:0/3153959722 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6575+0+0 (secure 0 0 0) 0x7f8fe4085880 con 0x7f8ff4067c30 2024-01-11T09:45:37.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.207+0000 7f8ffa19b700 1 --2- 172.21.15.27:0/3153959722 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8fd8066f30 0x7f8fd80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:37.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.207+0000 7f8ffa19b700 1 --2- 172.21.15.27:0/3153959722 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8fd8066f30 0x7f8fd80693e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8ff00097b0 tx=0x7f8ff0017000 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:37.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.210+0000 7f8fea7fc700 1 -- 172.21.15.27:0/3153959722 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8fe4052770 con 0x7f8ff4067c30 2024-01-11T09:45:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.458+0000 7f8ffc3ff700 1 -- 172.21.15.27:0/3153959722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7f8ff40ada80 con 0x7f8ff4067c30 2024-01-11T09:45:37.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.956+0000 7f8fea7fc700 1 -- 172.21.15.27:0/3153959722 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 28 and data pool 29 v113) v1 ==== 183+0+0 (secure 0 0 0) 0x7f8fe4055d90 con 0x7f8ff4067c30 2024-01-11T09:45:37.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.962+0000 7f8ffc3ff700 1 -- 172.21.15.27:0/3153959722 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8fd8066f30 msgr2=0x7f8fd80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.962+0000 7f8ffc3ff700 1 --2- 172.21.15.27:0/3153959722 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8fd8066f30 0x7f8fd80693e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8ff00097b0 tx=0x7f8ff0017000 comp rx=0 tx=0).stop 2024-01-11T09:45:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.962+0000 7f8ffc3ff700 1 -- 172.21.15.27:0/3153959722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ff4067c30 msgr2=0x7f8ff4141c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.962+0000 7f8ffc3ff700 1 --2- 172.21.15.27:0/3153959722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ff4067c30 0x7f8ff4141c30 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400d860 tx=0x7f8fe4008040 comp rx=0 tx=0).stop 2024-01-11T09:45:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.963+0000 7f8ffc3ff700 1 -- 172.21.15.27:0/3153959722 shutdown_connections 2024-01-11T09:45:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.963+0000 7f8ffc3ff700 1 --2- 172.21.15.27:0/3153959722 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8fd8066f30 0x7f8fd80693e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.963+0000 7f8ffc3ff700 1 --2- 172.21.15.27:0/3153959722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ff406cd90 0x7f8ff40c3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.963+0000 7f8ffc3ff700 1 --2- 172.21.15.27:0/3153959722 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ff4067860 0x7f8ff415d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.963+0000 7f8ffc3ff700 1 --2- 172.21.15.27:0/3153959722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ff4067c30 0x7f8ff4141c30 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.963+0000 7f8ffc3ff700 1 -- 172.21.15.27:0/3153959722 >> 172.21.15.27:0/3153959722 conn(0x7f8ff406b1f0 msgr2=0x7f8ff40cd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:37.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.964+0000 7f8ffc3ff700 1 -- 172.21.15.27:0/3153959722 shutdown_connections 2024-01-11T09:45:37.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:37.964+0000 7f8ffc3ff700 1 -- 172.21.15.27:0/3153959722 wait complete. 2024-01-11T09:45:37.958 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 28 and data pool 29 2024-01-11T09:45:37.987 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:45:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.195+0000 7f791d3ff700 1 Processor -- start 2024-01-11T09:45:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.196+0000 7f791d3ff700 1 -- start start 2024-01-11T09:45:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.197+0000 7f791d3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79181575e0 0x7f7918158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.197+0000 7f791d3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f791806b610 0x7f79180b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.197+0000 7f791d3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7918158b60 con 0x7f79180b1800 2024-01-11T09:45:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.197+0000 7f791d3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79180cb5e0 con 0x7f791806b610 2024-01-11T09:45:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.197+0000 7f791d3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79180b1c80 con 0x7f79181575e0 2024-01-11T09:45:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.197+0000 7f7916ffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79181575e0 0x7f7918158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.197+0000 7f79167fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f791806b610 0x7f79180b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.197+0000 7f79177fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f79180b1800 0x7f79180ca1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:34158/0 (socket says 172.21.15.27:34158) 2024-01-11T09:45:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.197+0000 7f7916ffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79181575e0 0x7f7918158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:40264/0 (socket says 172.21.15.27:40264) 2024-01-11T09:45:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.197+0000 7f79167fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f791806b610 0x7f79180b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:34234/0 (socket says 172.21.15.27:34234) 2024-01-11T09:45:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.197+0000 7f79177fe700 1 -- 172.21.15.27:0/2729231692 learned_addr learned my addr 172.21.15.27:0/2729231692 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.198+0000 7f79167fc700 1 -- 172.21.15.27:0/2729231692 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79181575e0 msgr2=0x7f7918158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.198+0000 7f79167fc700 1 --2- 172.21.15.27:0/2729231692 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79181575e0 0x7f7918158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.198+0000 7f79167fc700 1 -- 172.21.15.27:0/2729231692 >> v1:172.21.15.77:6789/0 conn(0x7f79180b1800 legacy=0x7f79180ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.198+0000 7f79167fc700 1 -- 172.21.15.27:0/2729231692 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79181599b0 con 0x7f791806b610 2024-01-11T09:45:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.198+0000 7f7915ffb700 1 -- 172.21.15.27:0/2729231692 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49415227 0 0) 0x7f7910001490 con 0x7f79180b1800 2024-01-11T09:45:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.198+0000 7f79167fc700 1 --2- 172.21.15.27:0/2729231692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f791806b610 0x7f79180b12c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f790c0098e0 tx=0x7f790c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5bd39825d232ed5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.199+0000 7f7915ffb700 1 -- 172.21.15.27:0/2729231692 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790c00b640 con 0x7f791806b610 2024-01-11T09:45:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.199+0000 7f7915ffb700 1 -- 172.21.15.27:0/2729231692 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790c00b7a0 con 0x7f791806b610 2024-01-11T09:45:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.199+0000 7f7915ffb700 1 -- 172.21.15.27:0/2729231692 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790c00bb40 con 0x7f791806b610 2024-01-11T09:45:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.199+0000 7f791d3ff700 1 -- 172.21.15.27:0/2729231692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f791806b610 msgr2=0x7f79180b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.199+0000 7f791d3ff700 1 --2- 172.21.15.27:0/2729231692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f791806b610 0x7f79180b12c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f790c0098e0 tx=0x7f790c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.199+0000 7f791d3ff700 1 -- 172.21.15.27:0/2729231692 shutdown_connections 2024-01-11T09:45:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.199+0000 7f791d3ff700 1 --2- 172.21.15.27:0/2729231692 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79181575e0 0x7f7918158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.199+0000 7f791d3ff700 1 --2- 172.21.15.27:0/2729231692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f791806b610 0x7f79180b12c0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.199+0000 7f791d3ff700 1 -- 172.21.15.27:0/2729231692 >> 172.21.15.27:0/2729231692 conn(0x7f79180680e0 msgr2=0x7f7918069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.199+0000 7f791d3ff700 1 -- 172.21.15.27:0/2729231692 shutdown_connections 2024-01-11T09:45:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.199+0000 7f791d3ff700 1 -- 172.21.15.27:0/2729231692 wait complete. 2024-01-11T09:45:38.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.201+0000 7f791d3ff700 1 Processor -- start 2024-01-11T09:45:38.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.201+0000 7f791d3ff700 1 -- start start 2024-01-11T09:45:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.202+0000 7f791d3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f791806b610 0x7f791815d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.202+0000 7f791d3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79180b1800 0x7f791816dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.202+0000 7f791d3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79181575e0 0x7f7918153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.202+0000 7f791d3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791816e160 con 0x7f791806b610 2024-01-11T09:45:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.202+0000 7f791d3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791815c0b0 con 0x7f79180b1800 2024-01-11T09:45:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.202+0000 7f791d3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791815c390 con 0x7f79181575e0 2024-01-11T09:45:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.202+0000 7f7916ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f791806b610 0x7f791815d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.202+0000 7f7916ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f791806b610 0x7f791815d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47458/0 (socket says 172.21.15.27:47458) 2024-01-11T09:45:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.202+0000 7f7916ffd700 1 -- 172.21.15.27:0/1432150718 learned_addr learned my addr 172.21.15.27:0/1432150718 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.202+0000 7f79177fe700 1 --2- 172.21.15.27:0/1432150718 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79181575e0 0x7f7918153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.202+0000 7f79167fc700 1 --2- 172.21.15.27:0/1432150718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79180b1800 0x7f791816dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.203+0000 7f7916ffd700 1 -- 172.21.15.27:0/1432150718 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79181575e0 msgr2=0x7f7918153d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.203+0000 7f7916ffd700 1 --2- 172.21.15.27:0/1432150718 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79181575e0 0x7f7918153d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.203+0000 7f7916ffd700 1 -- 172.21.15.27:0/1432150718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79180b1800 msgr2=0x7f791816dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.203+0000 7f7916ffd700 1 --2- 172.21.15.27:0/1432150718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79180b1800 0x7f791816dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.203+0000 7f7916ffd700 1 -- 172.21.15.27:0/1432150718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790c009420 con 0x7f791806b610 2024-01-11T09:45:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.203+0000 7f7916ffd700 1 --2- 172.21.15.27:0/1432150718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f791806b610 0x7f791815d150 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f790800e930 tx=0x7f79080091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.203+0000 7f78feffd700 1 -- 172.21.15.27:0/1432150718 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7908013680 con 0x7f791806b610 2024-01-11T09:45:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.203+0000 7f78feffd700 1 -- 172.21.15.27:0/1432150718 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79080137e0 con 0x7f791806b610 2024-01-11T09:45:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.203+0000 7f78feffd700 1 -- 172.21.15.27:0/1432150718 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7908013b80 con 0x7f791806b610 2024-01-11T09:45:38.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.203+0000 7f791d3ff700 1 -- 172.21.15.27:0/1432150718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f791815c670 con 0x7f791806b610 2024-01-11T09:45:38.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.204+0000 7f791d3ff700 1 -- 172.21.15.27:0/1432150718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f791815cc30 con 0x7f791806b610 2024-01-11T09:45:38.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.205+0000 7f791d3ff700 1 -- 172.21.15.27:0/1432150718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f791815cfa0 con 0x7f791806b610 2024-01-11T09:45:38.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.206+0000 7f78feffd700 1 -- 172.21.15.27:0/1432150718 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f790800c3e0 con 0x7f791806b610 2024-01-11T09:45:38.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.207+0000 7f78feffd700 1 --2- 172.21.15.27:0/1432150718 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7904066f30 0x7f79040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.207+0000 7f78feffd700 1 -- 172.21.15.27:0/1432150718 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f7908084a50 con 0x7f791806b610 2024-01-11T09:45:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.208+0000 7f79167fc700 1 --2- 172.21.15.27:0/1432150718 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7904066f30 0x7f79040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.208+0000 7f79167fc700 1 --2- 172.21.15.27:0/1432150718 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7904066f30 0x7f79040693e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f790c0093f0 tx=0x7f790c017000 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:38.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.211+0000 7f78feffd700 1 -- 172.21.15.27:0/1432150718 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7908051850 con 0x7f791806b610 2024-01-11T09:45:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.417+0000 7f791d3ff700 1 -- 172.21.15.27:0/1432150718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79180718d0 con 0x7f791806b610 2024-01-11T09:45:38.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.418+0000 7f78feffd700 1 -- 172.21.15.27:0/1432150718 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v217) v1 ==== 74+0+18740 (secure 0 0 0) 0x7f7908054e70 con 0x7f791806b610 2024-01-11T09:45:38.410 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:38.410 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":217,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:45:37.459599+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:45:21.455328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-01-11T09:45:22.634298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-01-11T09:45:35.631727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-01-11T09:45:36.452195+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:45:38.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.420+0000 7f791d3ff700 1 -- 172.21.15.27:0/1432150718 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7904066f30 msgr2=0x7f79040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:38.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.420+0000 7f791d3ff700 1 --2- 172.21.15.27:0/1432150718 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7904066f30 0x7f79040693e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f790c0093f0 tx=0x7f790c017000 comp rx=0 tx=0).stop 2024-01-11T09:45:38.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.420+0000 7f791d3ff700 1 -- 172.21.15.27:0/1432150718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f791806b610 msgr2=0x7f791815d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:38.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.420+0000 7f791d3ff700 1 --2- 172.21.15.27:0/1432150718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f791806b610 0x7f791815d150 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f790800e930 tx=0x7f79080091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:38.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.420+0000 7f791d3ff700 1 -- 172.21.15.27:0/1432150718 shutdown_connections 2024-01-11T09:45:38.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.420+0000 7f791d3ff700 1 --2- 172.21.15.27:0/1432150718 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79181575e0 0x7f7918153d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:38.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.420+0000 7f791d3ff700 1 --2- 172.21.15.27:0/1432150718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f791806b610 0x7f791815d150 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:38.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.420+0000 7f791d3ff700 1 --2- 172.21.15.27:0/1432150718 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7904066f30 0x7f79040693e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:38.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.420+0000 7f791d3ff700 1 --2- 172.21.15.27:0/1432150718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79180b1800 0x7f791816dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:38.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.420+0000 7f791d3ff700 1 -- 172.21.15.27:0/1432150718 >> 172.21.15.27:0/1432150718 conn(0x7f79180680e0 msgr2=0x7f791806f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:38.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.420+0000 7f791d3ff700 1 -- 172.21.15.27:0/1432150718 shutdown_connections 2024-01-11T09:45:38.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.420+0000 7f791d3ff700 1 -- 172.21.15.27:0/1432150718 wait complete. 2024-01-11T09:45:38.430 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:45:38.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.638+0000 7f19f3dff700 1 Processor -- start 2024-01-11T09:45:38.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.639+0000 7f19f3dff700 1 -- start start 2024-01-11T09:45:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.640+0000 7f19f3dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f19ec0b03e0 0x7f19ec0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.640+0000 7f19f3dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ec0cd730 con 0x7f19ec0b07b0 2024-01-11T09:45:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.640+0000 7f19f3dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ec0b2de0 con 0x7f19ec0b29c0 2024-01-11T09:45:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.640+0000 7f19f3dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19ec0ccb20 con 0x7f19ec0b03e0 2024-01-11T09:45:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.640+0000 7f19f139a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f19ec0b07b0 0x7f19ec159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41652/0 (socket says 172.21.15.27:41652) 2024-01-11T09:45:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.640+0000 7f19f139a700 1 -- 172.21.15.27:0/2598445977 learned_addr learned my addr 172.21.15.27:0/2598445977 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.640+0000 7f19f1b9b700 1 --2- 172.21.15.27:0/2598445977 >> v2:172.21.15.77:3300/0 conn(0x7f19ec0b03e0 0x7f19ec0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:38.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.641+0000 7f19f1b9b700 1 -- 172.21.15.27:0/2598445977 >> v1:172.21.15.77:6789/0 conn(0x7f19ec0b29c0 legacy=0x7f19ec0aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:38.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.641+0000 7f19f0b99700 1 -- 172.21.15.27:0/2598445977 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870100644 0 0) 0x7f19ec0cd730 con 0x7f19ec0b07b0 2024-01-11T09:45:38.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.641+0000 7f19f1b9b700 1 -- 172.21.15.27:0/2598445977 >> v1:172.21.15.27:6789/0 conn(0x7f19ec0b07b0 legacy=0x7f19ec159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:38.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.641+0000 7f19f1b9b700 1 -- 172.21.15.27:0/2598445977 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec0cce00 con 0x7f19ec0b03e0 2024-01-11T09:45:38.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.641+0000 7f19f1b9b700 1 --2- 172.21.15.27:0/2598445977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19ec0b03e0 0x7f19ec0b2480 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f19e8009860 tx=0x7f19e80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75fab968e75c149d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:38.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.641+0000 7f19f0b99700 1 -- 172.21.15.27:0/2598445977 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e8011630 con 0x7f19ec0b03e0 2024-01-11T09:45:38.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.641+0000 7f19f0b99700 1 -- 172.21.15.27:0/2598445977 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e8011790 con 0x7f19ec0b03e0 2024-01-11T09:45:38.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.641+0000 7f19f0b99700 1 -- 172.21.15.27:0/2598445977 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e8011b50 con 0x7f19ec0b03e0 2024-01-11T09:45:38.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.642+0000 7f19f3dff700 1 -- 172.21.15.27:0/2598445977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19ec0b03e0 msgr2=0x7f19ec0b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:38.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.642+0000 7f19f3dff700 1 --2- 172.21.15.27:0/2598445977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19ec0b03e0 0x7f19ec0b2480 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f19e8009860 tx=0x7f19e80091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:38.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.642+0000 7f19f3dff700 1 -- 172.21.15.27:0/2598445977 shutdown_connections 2024-01-11T09:45:38.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.642+0000 7f19f3dff700 1 --2- 172.21.15.27:0/2598445977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19ec0b03e0 0x7f19ec0b2480 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:38.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.642+0000 7f19f3dff700 1 -- 172.21.15.27:0/2598445977 >> 172.21.15.27:0/2598445977 conn(0x7f19ec06ed40 msgr2=0x7f19ec06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:38.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.642+0000 7f19f3dff700 1 -- 172.21.15.27:0/2598445977 shutdown_connections 2024-01-11T09:45:38.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.642+0000 7f19f3dff700 1 -- 172.21.15.27:0/2598445977 wait complete. 2024-01-11T09:45:38.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.644+0000 7f19f3dff700 1 Processor -- start 2024-01-11T09:45:38.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.644+0000 7f19f3dff700 1 -- start start 2024-01-11T09:45:38.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.646+0000 7f19f3dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19ec0b03e0 0x7f19ec0c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:38.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.646+0000 7f19f3dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19ec0b07b0 0x7f19ec16aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:38.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.646+0000 7f19f3dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19ec0b29c0 0x7f19ec0ac2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:38.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.646+0000 7f19f3dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec0c7310 con 0x7f19ec0b03e0 2024-01-11T09:45:38.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.646+0000 7f19f3dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec16b0e0 con 0x7f19ec0b29c0 2024-01-11T09:45:38.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.646+0000 7f19f3dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec0a6f40 con 0x7f19ec0b07b0 2024-01-11T09:45:38.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.646+0000 7f19f1b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19ec0b03e0 0x7f19ec0c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:38.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.646+0000 7f19f1b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19ec0b03e0 0x7f19ec0c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47470/0 (socket says 172.21.15.27:47470) 2024-01-11T09:45:38.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.646+0000 7f19f1b9b700 1 -- 172.21.15.27:0/4017115964 learned_addr learned my addr 172.21.15.27:0/4017115964 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:38.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.646+0000 7f19f1b9b700 1 -- 172.21.15.27:0/4017115964 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19ec0b07b0 msgr2=0x7f19ec16aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:38.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.646+0000 7f19f239c700 1 --2- 172.21.15.27:0/4017115964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19ec0b29c0 0x7f19ec0ac2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:38.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.646+0000 7f19f139a700 1 --2- 172.21.15.27:0/4017115964 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19ec0b07b0 0x7f19ec16aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:38.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.646+0000 7f19f1b9b700 1 --2- 172.21.15.27:0/4017115964 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19ec0b07b0 0x7f19ec16aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:38.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.646+0000 7f19f1b9b700 1 -- 172.21.15.27:0/4017115964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19ec0b29c0 msgr2=0x7f19ec0ac2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:38.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.646+0000 7f19f1b9b700 1 --2- 172.21.15.27:0/4017115964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19ec0b29c0 0x7f19ec0ac2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:38.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.646+0000 7f19f1b9b700 1 -- 172.21.15.27:0/4017115964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e8009420 con 0x7f19ec0b03e0 2024-01-11T09:45:38.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.646+0000 7f19f139a700 1 --2- 172.21.15.27:0/4017115964 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19ec0b07b0 0x7f19ec16aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:38.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.646+0000 7f19f239c700 1 --2- 172.21.15.27:0/4017115964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19ec0b29c0 0x7f19ec0ac2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:38.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.647+0000 7f19f1b9b700 1 --2- 172.21.15.27:0/4017115964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19ec0b03e0 0x7f19ec0c6dd0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f19ec069540 tx=0x7f19e8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:38.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.647+0000 7f19e1ffb700 1 -- 172.21.15.27:0/4017115964 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e801a450 con 0x7f19ec0b03e0 2024-01-11T09:45:38.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.647+0000 7f19e1ffb700 1 -- 172.21.15.27:0/4017115964 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e801a5f0 con 0x7f19ec0b03e0 2024-01-11T09:45:38.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.647+0000 7f19f3dff700 1 -- 172.21.15.27:0/4017115964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ec0ae900 con 0x7f19ec0b03e0 2024-01-11T09:45:38.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.647+0000 7f19e1ffb700 1 -- 172.21.15.27:0/4017115964 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e801a9b0 con 0x7f19ec0b03e0 2024-01-11T09:45:38.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.647+0000 7f19f3dff700 1 -- 172.21.15.27:0/4017115964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ec0aeec0 con 0x7f19ec0b03e0 2024-01-11T09:45:38.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.648+0000 7f19f3dff700 1 -- 172.21.15.27:0/4017115964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19ec0af4f0 con 0x7f19ec0b03e0 2024-01-11T09:45:38.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.649+0000 7f19e1ffb700 1 -- 172.21.15.27:0/4017115964 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f19e8024410 con 0x7f19ec0b03e0 2024-01-11T09:45:38.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.650+0000 7f19e1ffb700 1 --2- 172.21.15.27:0/4017115964 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f19d0066f30 0x7f19d00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:38.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.650+0000 7f19e1ffb700 1 -- 172.21.15.27:0/4017115964 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f19e8086680 con 0x7f19ec0b03e0 2024-01-11T09:45:38.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.650+0000 7f19f139a700 1 --2- 172.21.15.27:0/4017115964 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f19d0066f30 0x7f19d00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:38.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.651+0000 7f19f139a700 1 --2- 172.21.15.27:0/4017115964 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f19d0066f30 0x7f19d00693e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f19ec0cf4d0 tx=0x7f19dc009040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:38.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.654+0000 7f19e1ffb700 1 -- 172.21.15.27:0/4017115964 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f19e80535d0 con 0x7f19ec0b03e0 2024-01-11T09:45:38.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.858+0000 7f19f3dff700 1 -- 172.21.15.27:0/4017115964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19ec0718d0 con 0x7f19ec0b03e0 2024-01-11T09:45:38.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.859+0000 7f19e1ffb700 1 -- 172.21.15.27:0/4017115964 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v217) v1 ==== 74+0+18740 (secure 0 0 0) 0x7f19e805c090 con 0x7f19ec0b03e0 2024-01-11T09:45:38.850 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:38.851 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":217,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:45:37.459599+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:45:21.455328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-01-11T09:45:22.634298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-01-11T09:45:35.631727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-01-11T09:45:36.452195+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:45:38.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.861+0000 7f19f3dff700 1 -- 172.21.15.27:0/4017115964 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f19d0066f30 msgr2=0x7f19d00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:38.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.861+0000 7f19f3dff700 1 --2- 172.21.15.27:0/4017115964 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f19d0066f30 0x7f19d00693e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f19ec0cf4d0 tx=0x7f19dc009040 comp rx=0 tx=0).stop 2024-01-11T09:45:38.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.861+0000 7f19f3dff700 1 -- 172.21.15.27:0/4017115964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19ec0b03e0 msgr2=0x7f19ec0c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:38.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.861+0000 7f19f3dff700 1 --2- 172.21.15.27:0/4017115964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19ec0b03e0 0x7f19ec0c6dd0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f19ec069540 tx=0x7f19e8016000 comp rx=0 tx=0).stop 2024-01-11T09:45:38.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.861+0000 7f19f3dff700 1 -- 172.21.15.27:0/4017115964 shutdown_connections 2024-01-11T09:45:38.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.861+0000 7f19f3dff700 1 --2- 172.21.15.27:0/4017115964 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f19d0066f30 0x7f19d00693e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:38.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.861+0000 7f19f3dff700 1 --2- 172.21.15.27:0/4017115964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19ec0b29c0 0x7f19ec0ac2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:38.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.861+0000 7f19f3dff700 1 --2- 172.21.15.27:0/4017115964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19ec0b03e0 0x7f19ec0c6dd0 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:38.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.861+0000 7f19f3dff700 1 --2- 172.21.15.27:0/4017115964 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19ec0b07b0 0x7f19ec16aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:38.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.861+0000 7f19f3dff700 1 -- 172.21.15.27:0/4017115964 >> 172.21.15.27:0/4017115964 conn(0x7f19ec06ed40 msgr2=0x7f19ec15c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:38.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.861+0000 7f19f3dff700 1 -- 172.21.15.27:0/4017115964 shutdown_connections 2024-01-11T09:45:38.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:38.861+0000 7f19f3dff700 1 -- 172.21.15.27:0/4017115964 wait complete. 2024-01-11T09:45:38.871 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2024-01-11T09:45:39.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.089+0000 7f2d359ff700 1 Processor -- start 2024-01-11T09:45:39.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.089+0000 7f2d359ff700 1 -- start start 2024-01-11T09:45:39.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.090+0000 7f2d359ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2d30157410 0x7f2d300ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.090+0000 7f2d359ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2d30069540 0x7f2d300c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.090+0000 7f2d359ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d300699d0 con 0x7f2d301577e0 2024-01-11T09:45:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.090+0000 7f2d359ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d30069cb0 con 0x7f2d30157410 2024-01-11T09:45:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.090+0000 7f2d359ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d30069f90 con 0x7f2d30069540 2024-01-11T09:45:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.090+0000 7f2d2effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2d30157410 0x7f2d300ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.090+0000 7f2d2e7fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2d301577e0 0x7f2d300ae000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41664/0 (socket says 172.21.15.27:41664) 2024-01-11T09:45:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.090+0000 7f2d2effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2d30157410 0x7f2d300ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47480/0 (socket says 172.21.15.27:47480) 2024-01-11T09:45:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.090+0000 7f2d2e7fc700 1 -- 172.21.15.27:0/2285395146 learned_addr learned my addr 172.21.15.27:0/2285395146 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.090+0000 7f2d2f7fe700 1 --2- 172.21.15.27:0/2285395146 >> v2:172.21.15.77:3300/0 conn(0x7f2d30069540 0x7f2d300c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.091+0000 7f2d2effd700 1 -- 172.21.15.27:0/2285395146 >> v2:172.21.15.77:3300/0 conn(0x7f2d30069540 msgr2=0x7f2d300c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.091+0000 7f2d2effd700 1 --2- 172.21.15.27:0/2285395146 >> v2:172.21.15.77:3300/0 conn(0x7f2d30069540 0x7f2d300c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.091+0000 7f2d2effd700 1 -- 172.21.15.27:0/2285395146 >> v1:172.21.15.27:6789/0 conn(0x7f2d301577e0 legacy=0x7f2d300ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.091+0000 7f2d2effd700 1 -- 172.21.15.27:0/2285395146 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3006a270 con 0x7f2d30157410 2024-01-11T09:45:39.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.091+0000 7f2d2dffb700 1 -- 172.21.15.27:0/2285395146 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052400424 0 0) 0x7f2d300699d0 con 0x7f2d301577e0 2024-01-11T09:45:39.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.091+0000 7f2d2effd700 1 --2- 172.21.15.27:0/2285395146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30157410 0x7f2d300ccae0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f2d3006d030 tx=0x7f2d20009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e26f2de65c6d4a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:39.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.091+0000 7f2d2dffb700 1 -- 172.21.15.27:0/2285395146 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d20005c10 con 0x7f2d30157410 2024-01-11T09:45:39.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.092+0000 7f2d2dffb700 1 -- 172.21.15.27:0/2285395146 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d20005d70 con 0x7f2d30157410 2024-01-11T09:45:39.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.092+0000 7f2d2dffb700 1 -- 172.21.15.27:0/2285395146 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d20006260 con 0x7f2d30157410 2024-01-11T09:45:39.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.092+0000 7f2d359ff700 1 -- 172.21.15.27:0/2285395146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30157410 msgr2=0x7f2d300ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:39.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.092+0000 7f2d359ff700 1 --2- 172.21.15.27:0/2285395146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30157410 0x7f2d300ccae0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f2d3006d030 tx=0x7f2d20009120 comp rx=0 tx=0).stop 2024-01-11T09:45:39.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.092+0000 7f2d359ff700 1 -- 172.21.15.27:0/2285395146 shutdown_connections 2024-01-11T09:45:39.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.092+0000 7f2d359ff700 1 --2- 172.21.15.27:0/2285395146 >> v2:172.21.15.77:3300/0 conn(0x7f2d30069540 0x7f2d300c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:39.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.092+0000 7f2d359ff700 1 --2- 172.21.15.27:0/2285395146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30157410 0x7f2d300ccae0 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:39.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.092+0000 7f2d359ff700 1 -- 172.21.15.27:0/2285395146 >> 172.21.15.27:0/2285395146 conn(0x7f2d300680a0 msgr2=0x7f2d300cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:39.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.092+0000 7f2d359ff700 1 -- 172.21.15.27:0/2285395146 shutdown_connections 2024-01-11T09:45:39.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.092+0000 7f2d359ff700 1 -- 172.21.15.27:0/2285395146 wait complete. 2024-01-11T09:45:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.094+0000 7f2d359ff700 1 Processor -- start 2024-01-11T09:45:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.094+0000 7f2d359ff700 1 -- start start 2024-01-11T09:45:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.095+0000 7f2d359ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30069540 0x7f2d3014e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.095+0000 7f2d359ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d30157410 0x7f2d300c3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.095+0000 7f2d359ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d301577e0 0x7f2d3015dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.095+0000 7f2d359ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d300c4cf0 con 0x7f2d30069540 2024-01-11T09:45:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.095+0000 7f2d359ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d300c3c50 con 0x7f2d30157410 2024-01-11T09:45:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.095+0000 7f2d359ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d300c3f30 con 0x7f2d301577e0 2024-01-11T09:45:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.095+0000 7f2d2effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30069540 0x7f2d3014e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.095+0000 7f2d2effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30069540 0x7f2d3014e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47496/0 (socket says 172.21.15.27:47496) 2024-01-11T09:45:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.095+0000 7f2d2effd700 1 -- 172.21.15.27:0/4017640861 learned_addr learned my addr 172.21.15.27:0/4017640861 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.095+0000 7f2d2e7fc700 1 --2- 172.21.15.27:0/4017640861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d30157410 0x7f2d300c3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.095+0000 7f2d2f7fe700 1 --2- 172.21.15.27:0/4017640861 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d301577e0 0x7f2d3015dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.096+0000 7f2d2effd700 1 -- 172.21.15.27:0/4017640861 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d301577e0 msgr2=0x7f2d3015dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.096+0000 7f2d2effd700 1 --2- 172.21.15.27:0/4017640861 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d301577e0 0x7f2d3015dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.096+0000 7f2d2effd700 1 -- 172.21.15.27:0/4017640861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d30157410 msgr2=0x7f2d300c3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.096+0000 7f2d2effd700 1 --2- 172.21.15.27:0/4017640861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d30157410 0x7f2d300c3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.096+0000 7f2d2effd700 1 -- 172.21.15.27:0/4017640861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d200093c0 con 0x7f2d30069540 2024-01-11T09:45:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.096+0000 7f2d2e7fc700 1 --2- 172.21.15.27:0/4017640861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d30157410 0x7f2d300c3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.096+0000 7f2d2effd700 1 --2- 172.21.15.27:0/4017640861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30069540 0x7f2d3014e1d0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f2d20009f40 tx=0x7f2d20016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.096+0000 7f2d16ffd700 1 -- 172.21.15.27:0/4017640861 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2001d880 con 0x7f2d30069540 2024-01-11T09:45:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.096+0000 7f2d16ffd700 1 -- 172.21.15.27:0/4017640861 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2001da00 con 0x7f2d30069540 2024-01-11T09:45:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.096+0000 7f2d359ff700 1 -- 172.21.15.27:0/4017640861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d300c4210 con 0x7f2d30069540 2024-01-11T09:45:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.096+0000 7f2d16ffd700 1 -- 172.21.15.27:0/4017640861 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2001ddc0 con 0x7f2d30069540 2024-01-11T09:45:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.096+0000 7f2d359ff700 1 -- 172.21.15.27:0/4017640861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d300c47d0 con 0x7f2d30069540 2024-01-11T09:45:39.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.097+0000 7f2d359ff700 1 -- 172.21.15.27:0/4017640861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d300c49f0 con 0x7f2d30069540 2024-01-11T09:45:39.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.098+0000 7f2d16ffd700 1 -- 172.21.15.27:0/4017640861 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2d2001b690 con 0x7f2d30069540 2024-01-11T09:45:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.099+0000 7f2d16ffd700 1 --2- 172.21.15.27:0/4017640861 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2d1c066f30 0x7f2d1c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.099+0000 7f2d16ffd700 1 -- 172.21.15.27:0/4017640861 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f2d200867d0 con 0x7f2d30069540 2024-01-11T09:45:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.100+0000 7f2d2e7fc700 1 --2- 172.21.15.27:0/4017640861 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2d1c066f30 0x7f2d1c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:39.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.100+0000 7f2d2e7fc700 1 --2- 172.21.15.27:0/4017640861 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2d1c066f30 0x7f2d1c0693e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2d30156890 tx=0x7f2d24009040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.103+0000 7f2d16ffd700 1 -- 172.21.15.27:0/4017640861 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2d200535d0 con 0x7f2d30069540 2024-01-11T09:45:39.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.354+0000 7f2d359ff700 1 -- 172.21.15.27:0/4017640861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2d3004c3c0 con 0x7f2d30069540 2024-01-11T09:45:39.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.974+0000 7f2d16ffd700 1 -- 172.21.15.27:0/4017640861 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v116) v1 ==== 121+0+0 (secure 0 0 0) 0x7f2d20056bf0 con 0x7f2d30069540 2024-01-11T09:45:39.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.977+0000 7f2d359ff700 1 -- 172.21.15.27:0/4017640861 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2d1c066f30 msgr2=0x7f2d1c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:39.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.977+0000 7f2d359ff700 1 --2- 172.21.15.27:0/4017640861 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2d1c066f30 0x7f2d1c0693e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2d30156890 tx=0x7f2d24009040 comp rx=0 tx=0).stop 2024-01-11T09:45:39.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.977+0000 7f2d359ff700 1 -- 172.21.15.27:0/4017640861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30069540 msgr2=0x7f2d3014e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:39.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.977+0000 7f2d359ff700 1 --2- 172.21.15.27:0/4017640861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30069540 0x7f2d3014e1d0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f2d20009f40 tx=0x7f2d20016000 comp rx=0 tx=0).stop 2024-01-11T09:45:39.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.977+0000 7f2d359ff700 1 -- 172.21.15.27:0/4017640861 shutdown_connections 2024-01-11T09:45:39.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.977+0000 7f2d359ff700 1 --2- 172.21.15.27:0/4017640861 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2d301577e0 0x7f2d3015dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:39.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.977+0000 7f2d359ff700 1 --2- 172.21.15.27:0/4017640861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30069540 0x7f2d3014e1d0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:39.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.977+0000 7f2d359ff700 1 --2- 172.21.15.27:0/4017640861 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2d1c066f30 0x7f2d1c0693e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:39.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.977+0000 7f2d359ff700 1 --2- 172.21.15.27:0/4017640861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d30157410 0x7f2d300c3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:39.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.977+0000 7f2d359ff700 1 -- 172.21.15.27:0/4017640861 >> 172.21.15.27:0/4017640861 conn(0x7f2d300680a0 msgr2=0x7f2d300c7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:39.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.977+0000 7f2d359ff700 1 -- 172.21.15.27:0/4017640861 shutdown_connections 2024-01-11T09:45:39.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:39.977+0000 7f2d359ff700 1 -- 172.21.15.27:0/4017640861 wait complete. 2024-01-11T09:45:39.991 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2024-01-11T09:45:40.100 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-01-11T09:45:40.108+0000 7f2bc3ff1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T09:45:40.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.207+0000 7f0e105ff700 1 Processor -- start 2024-01-11T09:45:40.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.208+0000 7f0e105ff700 1 -- start start 2024-01-11T09:45:40.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.209+0000 7f0e105ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0e08155c80 0x7f0e08068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:40.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.209+0000 7f0e105ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0e08159f50 0x7f0e080c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:40.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.209+0000 7f0e105ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e08068fd0 con 0x7f0e08068b60 2024-01-11T09:45:40.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.209+0000 7f0e105ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e0806ad00 con 0x7f0e08155c80 2024-01-11T09:45:40.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.209+0000 7f0e105ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e08156b90 con 0x7f0e08159f50 2024-01-11T09:45:40.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.209+0000 7f0e0e39b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0e08155c80 0x7f0e08068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:40.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.209+0000 7f0e0e39b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0e08155c80 0x7f0e08068620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47502/0 (socket says 172.21.15.27:47502) 2024-01-11T09:45:40.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.209+0000 7f0e0e39b700 1 -- 172.21.15.27:0/3793549864 learned_addr learned my addr 172.21.15.27:0/3793549864 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:40.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.209+0000 7f0e0eb9c700 1 --2- 172.21.15.27:0/3793549864 >> v2:172.21.15.77:3300/0 conn(0x7f0e08159f50 0x7f0e080c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:40.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.210+0000 7f0e0e39b700 1 -- 172.21.15.27:0/3793549864 >> v2:172.21.15.77:3300/0 conn(0x7f0e08159f50 msgr2=0x7f0e080c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:40.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.210+0000 7f0e0e39b700 1 --2- 172.21.15.27:0/3793549864 >> v2:172.21.15.77:3300/0 conn(0x7f0e08159f50 0x7f0e080c70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:40.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.210+0000 7f0e0e39b700 1 -- 172.21.15.27:0/3793549864 >> v1:172.21.15.77:6789/0 conn(0x7f0e08068b60 legacy=0x7f0e08159840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:40.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.210+0000 7f0e0e39b700 1 -- 172.21.15.27:0/3793549864 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e081562d0 con 0x7f0e08155c80 2024-01-11T09:45:40.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.210+0000 7f0e0eb9c700 1 --2- 172.21.15.27:0/3793549864 >> v2:172.21.15.77:3300/0 conn(0x7f0e08159f50 0x7f0e080c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:40.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.210+0000 7f0e0e39b700 1 --2- 172.21.15.27:0/3793549864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e08155c80 0x7f0e08068620 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f0e080cc640 tx=0x7f0e0400b4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6902f250dd05826 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:40.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.210+0000 7f0e0d399700 1 -- 172.21.15.27:0/3793549864 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0400db10 con 0x7f0e08155c80 2024-01-11T09:45:40.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.210+0000 7f0e0d399700 1 -- 172.21.15.27:0/3793549864 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e0400dc70 con 0x7f0e08155c80 2024-01-11T09:45:40.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.210+0000 7f0e0d399700 1 -- 172.21.15.27:0/3793549864 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e04017410 con 0x7f0e08155c80 2024-01-11T09:45:40.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.211+0000 7f0e105ff700 1 -- 172.21.15.27:0/3793549864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e08155c80 msgr2=0x7f0e08068620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:40.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.211+0000 7f0e105ff700 1 --2- 172.21.15.27:0/3793549864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e08155c80 0x7f0e08068620 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f0e080cc640 tx=0x7f0e0400b4e0 comp rx=0 tx=0).stop 2024-01-11T09:45:40.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.211+0000 7f0e105ff700 1 -- 172.21.15.27:0/3793549864 shutdown_connections 2024-01-11T09:45:40.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.211+0000 7f0e105ff700 1 --2- 172.21.15.27:0/3793549864 >> v2:172.21.15.77:3300/0 conn(0x7f0e08159f50 0x7f0e080c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:40.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.211+0000 7f0e105ff700 1 --2- 172.21.15.27:0/3793549864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e08155c80 0x7f0e08068620 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:40.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.211+0000 7f0e105ff700 1 -- 172.21.15.27:0/3793549864 >> 172.21.15.27:0/3793549864 conn(0x7f0e080c9fb0 msgr2=0x7f0e0806ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:40.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.211+0000 7f0e105ff700 1 -- 172.21.15.27:0/3793549864 shutdown_connections 2024-01-11T09:45:40.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.211+0000 7f0e105ff700 1 -- 172.21.15.27:0/3793549864 wait complete. 2024-01-11T09:45:40.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.213+0000 7f0e105ff700 1 Processor -- start 2024-01-11T09:45:40.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.213+0000 7f0e105ff700 1 -- start start 2024-01-11T09:45:40.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.214+0000 7f0e105ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e08068b60 0x7f0e0813ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:40.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.214+0000 7f0e105ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e08155c80 0x7f0e08152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:40.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.214+0000 7f0e105ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e08159f50 0x7f0e081532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:40.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.215+0000 7f0e105ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e08152920 con 0x7f0e08068b60 2024-01-11T09:45:40.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.215+0000 7f0e105ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e080b11c0 con 0x7f0e08155c80 2024-01-11T09:45:40.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.215+0000 7f0e105ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e08151ff0 con 0x7f0e08159f50 2024-01-11T09:45:40.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.215+0000 7f0e0e39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e08068b60 0x7f0e0813ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:40.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.215+0000 7f0e0e39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e08068b60 0x7f0e0813ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47510/0 (socket says 172.21.15.27:47510) 2024-01-11T09:45:40.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.215+0000 7f0e0e39b700 1 -- 172.21.15.27:0/1645943914 learned_addr learned my addr 172.21.15.27:0/1645943914 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:40.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.215+0000 7f0e0db9a700 1 --2- 172.21.15.27:0/1645943914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e08155c80 0x7f0e08152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:40.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.215+0000 7f0e0eb9c700 1 --2- 172.21.15.27:0/1645943914 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e08159f50 0x7f0e081532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:40.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.215+0000 7f0e0e39b700 1 -- 172.21.15.27:0/1645943914 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e08159f50 msgr2=0x7f0e081532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:40.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.215+0000 7f0e0e39b700 1 --2- 172.21.15.27:0/1645943914 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e08159f50 0x7f0e081532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:40.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.215+0000 7f0e0e39b700 1 -- 172.21.15.27:0/1645943914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e08155c80 msgr2=0x7f0e08152db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:40.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.215+0000 7f0e0e39b700 1 --2- 172.21.15.27:0/1645943914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e08155c80 0x7f0e08152db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:40.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.215+0000 7f0e0e39b700 1 -- 172.21.15.27:0/1645943914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0400b780 con 0x7f0e08068b60 2024-01-11T09:45:40.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.216+0000 7f0e0e39b700 1 --2- 172.21.15.27:0/1645943914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e08068b60 0x7f0e0813ff20 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f0e0400bfd0 tx=0x7f0e04019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:40.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.216+0000 7f0dfe7fc700 1 -- 172.21.15.27:0/1645943914 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e04002f90 con 0x7f0e08068b60 2024-01-11T09:45:40.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.216+0000 7f0dfe7fc700 1 -- 172.21.15.27:0/1645943914 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e04022610 con 0x7f0e08068b60 2024-01-11T09:45:40.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.216+0000 7f0e105ff700 1 -- 172.21.15.27:0/1645943914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e081522d0 con 0x7f0e08068b60 2024-01-11T09:45:40.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.216+0000 7f0dfe7fc700 1 -- 172.21.15.27:0/1645943914 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e04022990 con 0x7f0e08068b60 2024-01-11T09:45:40.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.216+0000 7f0e105ff700 1 -- 172.21.15.27:0/1645943914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0814dc80 con 0x7f0e08068b60 2024-01-11T09:45:40.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.217+0000 7f0e105ff700 1 -- 172.21.15.27:0/1645943914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e08152c00 con 0x7f0e08068b60 2024-01-11T09:45:40.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.218+0000 7f0dfe7fc700 1 -- 172.21.15.27:0/1645943914 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0e04015070 con 0x7f0e08068b60 2024-01-11T09:45:40.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.219+0000 7f0dfe7fc700 1 --2- 172.21.15.27:0/1645943914 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0dec066f30 0x7f0dec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:40.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.219+0000 7f0dfe7fc700 1 -- 172.21.15.27:0/1645943914 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f0e04088740 con 0x7f0e08068b60 2024-01-11T09:45:40.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.219+0000 7f0e0db9a700 1 --2- 172.21.15.27:0/1645943914 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0dec066f30 0x7f0dec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:40.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.220+0000 7f0e0db9a700 1 --2- 172.21.15.27:0/1645943914 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0dec066f30 0x7f0dec0693e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0e080b18c0 tx=0x7f0df8007ec0 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:40.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.223+0000 7f0dfe7fc700 1 -- 172.21.15.27:0/1645943914 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0e04055540 con 0x7f0e08068b60 2024-01-11T09:45:40.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.469+0000 7f0e105ff700 1 -- 172.21.15.27:0/1645943914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f0e0804c3c0 con 0x7f0e08068b60 2024-01-11T09:45:40.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.985+0000 7f0dfe7fc700 1 -- 172.21.15.27:0/1645943914 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v117) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0e04058b60 con 0x7f0e08068b60 2024-01-11T09:45:40.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.987+0000 7f0e105ff700 1 -- 172.21.15.27:0/1645943914 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0dec066f30 msgr2=0x7f0dec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:40.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.987+0000 7f0e105ff700 1 --2- 172.21.15.27:0/1645943914 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0dec066f30 0x7f0dec0693e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0e080b18c0 tx=0x7f0df8007ec0 comp rx=0 tx=0).stop 2024-01-11T09:45:40.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.987+0000 7f0e105ff700 1 -- 172.21.15.27:0/1645943914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e08068b60 msgr2=0x7f0e0813ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:40.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.987+0000 7f0e105ff700 1 --2- 172.21.15.27:0/1645943914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e08068b60 0x7f0e0813ff20 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f0e0400bfd0 tx=0x7f0e04019000 comp rx=0 tx=0).stop 2024-01-11T09:45:40.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.987+0000 7f0e105ff700 1 -- 172.21.15.27:0/1645943914 shutdown_connections 2024-01-11T09:45:40.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.987+0000 7f0e105ff700 1 --2- 172.21.15.27:0/1645943914 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e08159f50 0x7f0e081532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:40.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.987+0000 7f0e105ff700 1 --2- 172.21.15.27:0/1645943914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e08068b60 0x7f0e0813ff20 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:40.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.987+0000 7f0e105ff700 1 --2- 172.21.15.27:0/1645943914 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0dec066f30 0x7f0dec0693e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:40.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.987+0000 7f0e105ff700 1 --2- 172.21.15.27:0/1645943914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e08155c80 0x7f0e08152db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:40.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.987+0000 7f0e105ff700 1 -- 172.21.15.27:0/1645943914 >> 172.21.15.27:0/1645943914 conn(0x7f0e080c9fb0 msgr2=0x7f0e080ca620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:40.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.987+0000 7f0e105ff700 1 -- 172.21.15.27:0/1645943914 shutdown_connections 2024-01-11T09:45:40.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:40.988+0000 7f0e105ff700 1 -- 172.21.15.27:0/1645943914 wait complete. 2024-01-11T09:45:40.999 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:45:41.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.207+0000 7f8f2f9ff700 1 Processor -- start 2024-01-11T09:45:41.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.207+0000 7f8f2f9ff700 1 -- start start 2024-01-11T09:45:41.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.208+0000 7f8f2f9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f280b03e0 0x7f8f280b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:41.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.208+0000 7f8f2f9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8f280b07b0 0x7f8f280b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:41.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.208+0000 7f8f2f9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8f28159200 0x7f8f28159640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:41.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.208+0000 7f8f2f9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f280cd730 con 0x7f8f28159200 2024-01-11T09:45:41.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.208+0000 7f8f2f9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f280ccb20 con 0x7f8f280b07b0 2024-01-11T09:45:41.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.208+0000 7f8f2f9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f280cce00 con 0x7f8f280b03e0 2024-01-11T09:45:41.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.209+0000 7f8f2df9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8f28159200 0x7f8f28159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:41.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.209+0000 7f8f2cf9a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8f280b07b0 0x7f8f280b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:41.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.209+0000 7f8f2d79b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f280b03e0 0x7f8f280b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.209+0000 7f8f2cf9a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8f280b07b0 0x7f8f280b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:34302/0 (socket says 172.21.15.27:34302) 2024-01-11T09:45:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.209+0000 7f8f2df9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8f28159200 0x7f8f28159640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47514/0 (socket says 172.21.15.27:47514) 2024-01-11T09:45:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.209+0000 7f8f2cf9a700 1 -- 172.21.15.27:0/1404196023 learned_addr learned my addr 172.21.15.27:0/1404196023 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.209+0000 7f8f2df9c700 1 -- 172.21.15.27:0/1404196023 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f280b03e0 msgr2=0x7f8f280b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.209+0000 7f8f2df9c700 1 --2- 172.21.15.27:0/1404196023 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f280b03e0 0x7f8f280b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.209+0000 7f8f2df9c700 1 -- 172.21.15.27:0/1404196023 >> v2:172.21.15.77:3300/0 conn(0x7f8f280b07b0 msgr2=0x7f8f280b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.209+0000 7f8f2df9c700 1 --2- 172.21.15.27:0/1404196023 >> v2:172.21.15.77:3300/0 conn(0x7f8f280b07b0 0x7f8f280b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.209+0000 7f8f2df9c700 1 -- 172.21.15.27:0/1404196023 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f280cd0e0 con 0x7f8f28159200 2024-01-11T09:45:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.209+0000 7f8f2cf9a700 1 --2- 172.21.15.27:0/1404196023 >> v2:172.21.15.77:3300/0 conn(0x7f8f280b07b0 0x7f8f280b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.210+0000 7f8f2df9c700 1 --2- 172.21.15.27:0/1404196023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f28159200 0x7f8f28159640 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f8f280893d0 tx=0x7f8f20009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f4545c01cdd8e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:41.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.210+0000 7f8f1ffff700 1 -- 172.21.15.27:0/1404196023 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2000e070 con 0x7f8f28159200 2024-01-11T09:45:41.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.210+0000 7f8f1ffff700 1 -- 172.21.15.27:0/1404196023 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2000bab0 con 0x7f8f28159200 2024-01-11T09:45:41.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.210+0000 7f8f1ffff700 1 -- 172.21.15.27:0/1404196023 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2000be70 con 0x7f8f28159200 2024-01-11T09:45:41.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.210+0000 7f8f2f9ff700 1 -- 172.21.15.27:0/1404196023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f28159200 msgr2=0x7f8f28159640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:41.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.210+0000 7f8f2f9ff700 1 --2- 172.21.15.27:0/1404196023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f28159200 0x7f8f28159640 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f8f280893d0 tx=0x7f8f20009780 comp rx=0 tx=0).stop 2024-01-11T09:45:41.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.211+0000 7f8f2f9ff700 1 -- 172.21.15.27:0/1404196023 shutdown_connections 2024-01-11T09:45:41.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.211+0000 7f8f2f9ff700 1 --2- 172.21.15.27:0/1404196023 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f280b03e0 0x7f8f280b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.211+0000 7f8f2f9ff700 1 --2- 172.21.15.27:0/1404196023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f28159200 0x7f8f28159640 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.211+0000 7f8f2f9ff700 1 --2- 172.21.15.27:0/1404196023 >> v2:172.21.15.77:3300/0 conn(0x7f8f280b07b0 0x7f8f280b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.211+0000 7f8f2f9ff700 1 -- 172.21.15.27:0/1404196023 >> 172.21.15.27:0/1404196023 conn(0x7f8f2806ed40 msgr2=0x7f8f2806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:41.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.211+0000 7f8f2f9ff700 1 -- 172.21.15.27:0/1404196023 shutdown_connections 2024-01-11T09:45:41.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.211+0000 7f8f2f9ff700 1 -- 172.21.15.27:0/1404196023 wait complete. 2024-01-11T09:45:41.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.212+0000 7f8f2f9ff700 1 Processor -- start 2024-01-11T09:45:41.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.212+0000 7f8f2f9ff700 1 -- start start 2024-01-11T09:45:41.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.213+0000 7f8f2f9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f280b03e0 0x7f8f28157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:41.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.213+0000 7f8f2f9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f280b07b0 0x7f8f280c6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:41.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.213+0000 7f8f2f9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f28159200 0x7f8f280c5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:41.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.213+0000 7f8f2f9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f280c42f0 con 0x7f8f28159200 2024-01-11T09:45:41.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.213+0000 7f8f2f9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f280c45d0 con 0x7f8f280b07b0 2024-01-11T09:45:41.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.213+0000 7f8f2f9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f280c48b0 con 0x7f8f280b03e0 2024-01-11T09:45:41.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.214+0000 7f8f2d79b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f280b03e0 0x7f8f28157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:41.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.214+0000 7f8f2df9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f28159200 0x7f8f280c5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:41.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.214+0000 7f8f2d79b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f280b03e0 0x7f8f28157b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:40320/0 (socket says 172.21.15.27:40320) 2024-01-11T09:45:41.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.214+0000 7f8f2d79b700 1 -- 172.21.15.27:0/3400435974 learned_addr learned my addr 172.21.15.27:0/3400435974 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:41.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.214+0000 7f8f2cf9a700 1 --2- 172.21.15.27:0/3400435974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f280b07b0 0x7f8f280c6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:41.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.214+0000 7f8f2d79b700 1 -- 172.21.15.27:0/3400435974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f280b07b0 msgr2=0x7f8f280c6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:41.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.214+0000 7f8f2d79b700 1 --2- 172.21.15.27:0/3400435974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f280b07b0 0x7f8f280c6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.214+0000 7f8f2d79b700 1 -- 172.21.15.27:0/3400435974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f28159200 msgr2=0x7f8f280c5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:41.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.214+0000 7f8f2d79b700 1 --2- 172.21.15.27:0/3400435974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f28159200 0x7f8f280c5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.214+0000 7f8f2d79b700 1 -- 172.21.15.27:0/3400435974 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f180094f0 con 0x7f8f280b03e0 2024-01-11T09:45:41.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.214+0000 7f8f2df9c700 1 --2- 172.21.15.27:0/3400435974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f28159200 0x7f8f280c5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:41.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.214+0000 7f8f2d79b700 1 --2- 172.21.15.27:0/3400435974 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f280b03e0 0x7f8f28157b40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8f2400e930 tx=0x7f8f240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:41.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.215+0000 7f8f1d7fa700 1 -- 172.21.15.27:0/3400435974 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f24013680 con 0x7f8f280b03e0 2024-01-11T09:45:41.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.215+0000 7f8f1d7fa700 1 -- 172.21.15.27:0/3400435974 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f240137e0 con 0x7f8f280b03e0 2024-01-11T09:45:41.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.215+0000 7f8f1d7fa700 1 -- 172.21.15.27:0/3400435974 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f24013b80 con 0x7f8f280b03e0 2024-01-11T09:45:41.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.215+0000 7f8f2f9ff700 1 -- 172.21.15.27:0/3400435974 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f20009a20 con 0x7f8f280b03e0 2024-01-11T09:45:41.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.215+0000 7f8f2f9ff700 1 -- 172.21.15.27:0/3400435974 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f280c4e70 con 0x7f8f280b03e0 2024-01-11T09:45:41.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.216+0000 7f8f2f9ff700 1 -- 172.21.15.27:0/3400435974 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f280c51e0 con 0x7f8f280b03e0 2024-01-11T09:45:41.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.217+0000 7f8f1d7fa700 1 -- 172.21.15.27:0/3400435974 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8f24005a00 con 0x7f8f280b03e0 2024-01-11T09:45:41.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.218+0000 7f8f1d7fa700 1 --2- 172.21.15.27:0/3400435974 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8f0c066f30 0x7f8f0c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:41.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.218+0000 7f8f1d7fa700 1 -- 172.21.15.27:0/3400435974 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f8f24084bd0 con 0x7f8f280b03e0 2024-01-11T09:45:41.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.218+0000 7f8f2cf9a700 1 --2- 172.21.15.27:0/3400435974 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8f0c066f30 0x7f8f0c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:41.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.219+0000 7f8f2cf9a700 1 --2- 172.21.15.27:0/3400435974 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8f0c066f30 0x7f8f0c0693e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8f280c9650 tx=0x7f8f18009250 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:41.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.222+0000 7f8f1d7fa700 1 -- 172.21.15.27:0/3400435974 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8f24051a50 con 0x7f8f280b03e0 2024-01-11T09:45:41.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.475+0000 7f8f2f9ff700 1 -- 172.21.15.27:0/3400435974 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f280718d0 con 0x7f8f280b03e0 2024-01-11T09:45:41.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.477+0000 7f8f1d7fa700 1 -- 172.21.15.27:0/3400435974 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+5690 (secure 0 0 0) 0x7f8f24017090 con 0x7f8f280b03e0 2024-01-11T09:45:41.468 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:41.468 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":117,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2103359888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":2103359888},{"type":"v1","addr":"172.21.15.77:6834","nonce":2103359888}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8795,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2650824649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2650824649},{"type":"v1","addr":"172.21.15.77:6835","nonce":2650824649}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:45:23.637614+0000","modified":"2024-01-11T09:45:27.146191+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10599":{"gid":10599,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1143960902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1143960902},{"type":"v1","addr":"172.21.15.27:6835","nonce":1143960902}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:45:37.460120+0000","modified":"2024-01-11T09:45:40.977702+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10596},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10596":{"gid":10596,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6834/2807751836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2807751836},{"type":"v1","addr":"172.21.15.27:6834","nonce":2807751836}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":14}]} 2024-01-11T09:45:41.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.479+0000 7f8f2f9ff700 1 -- 172.21.15.27:0/3400435974 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8f0c066f30 msgr2=0x7f8f0c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:41.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.479+0000 7f8f2f9ff700 1 --2- 172.21.15.27:0/3400435974 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8f0c066f30 0x7f8f0c0693e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8f280c9650 tx=0x7f8f18009250 comp rx=0 tx=0).stop 2024-01-11T09:45:41.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.479+0000 7f8f2f9ff700 1 -- 172.21.15.27:0/3400435974 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f280b03e0 msgr2=0x7f8f28157b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:41.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.479+0000 7f8f2f9ff700 1 --2- 172.21.15.27:0/3400435974 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f280b03e0 0x7f8f28157b40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8f2400e930 tx=0x7f8f240091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.479+0000 7f8f2f9ff700 1 -- 172.21.15.27:0/3400435974 shutdown_connections 2024-01-11T09:45:41.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.479+0000 7f8f2f9ff700 1 --2- 172.21.15.27:0/3400435974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f28159200 0x7f8f280c5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.479+0000 7f8f2f9ff700 1 --2- 172.21.15.27:0/3400435974 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f280b03e0 0x7f8f28157b40 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.479+0000 7f8f2f9ff700 1 --2- 172.21.15.27:0/3400435974 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8f0c066f30 0x7f8f0c0693e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.479+0000 7f8f2f9ff700 1 --2- 172.21.15.27:0/3400435974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f280b07b0 0x7f8f280c6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.479+0000 7f8f2f9ff700 1 -- 172.21.15.27:0/3400435974 >> 172.21.15.27:0/3400435974 conn(0x7f8f2806ed40 msgr2=0x7f8f28158980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:41.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.479+0000 7f8f2f9ff700 1 -- 172.21.15.27:0/3400435974 shutdown_connections 2024-01-11T09:45:41.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.479+0000 7f8f2f9ff700 1 -- 172.21.15.27:0/3400435974 wait complete. 2024-01-11T09:45:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 117 2024-01-11T09:45:41.488 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:45:41.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.695+0000 7fbee67ff700 1 Processor -- start 2024-01-11T09:45:41.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.695+0000 7fbee67ff700 1 -- start start 2024-01-11T09:45:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.697+0000 7fbee67ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee0159db0 0x7fbee015a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.697+0000 7fbee67ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbee014fc20 0x7fbee0150080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.697+0000 7fbee67ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee0157e30 con 0x7fbee01599e0 2024-01-11T09:45:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.697+0000 7fbee67ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbee0157500 con 0x7fbee014fc20 2024-01-11T09:45:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.697+0000 7fbee67ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbee01577e0 con 0x7fbee0159db0 2024-01-11T09:45:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.697+0000 7fbee4d9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbee014fc20 0x7fbee0150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.697+0000 7fbee4d9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbee014fc20 0x7fbee0150080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47536/0 (socket says 172.21.15.27:47536) 2024-01-11T09:45:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.697+0000 7fbee4d9c700 1 -- 172.21.15.27:0/2625778458 learned_addr learned my addr 172.21.15.27:0/2625778458 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:41.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.697+0000 7fbedf7fe700 1 --2- 172.21.15.27:0/2625778458 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee0159db0 0x7fbee015a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:41.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.697+0000 7fbee4d9c700 1 -- 172.21.15.27:0/2625778458 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee0159db0 msgr2=0x7fbee015a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:41.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.698+0000 7fbee4d9c700 1 --2- 172.21.15.27:0/2625778458 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee0159db0 0x7fbee015a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.698+0000 7fbee4d9c700 1 -- 172.21.15.27:0/2625778458 >> v1:172.21.15.77:6789/0 conn(0x7fbee01599e0 legacy=0x7fbee00b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:41.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.698+0000 7fbee4d9c700 1 -- 172.21.15.27:0/2625778458 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee0157ac0 con 0x7fbee014fc20 2024-01-11T09:45:41.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.698+0000 7fbedf7fe700 1 --2- 172.21.15.27:0/2625778458 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee0159db0 0x7fbee015a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:41.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.698+0000 7fbee4d9c700 1 --2- 172.21.15.27:0/2625778458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee014fc20 0x7fbee0150080 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fbed800d1a0 tx=0x7fbed800ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc719d629c64fcb3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:41.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.698+0000 7fbedeffd700 1 -- 172.21.15.27:0/2625778458 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed8010070 con 0x7fbee014fc20 2024-01-11T09:45:41.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.698+0000 7fbedeffd700 1 -- 172.21.15.27:0/2625778458 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed800dcb0 con 0x7fbee014fc20 2024-01-11T09:45:41.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.698+0000 7fbedeffd700 1 -- 172.21.15.27:0/2625778458 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed8016630 con 0x7fbee014fc20 2024-01-11T09:45:41.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.699+0000 7fbee67ff700 1 -- 172.21.15.27:0/2625778458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee014fc20 msgr2=0x7fbee0150080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:41.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.699+0000 7fbee67ff700 1 --2- 172.21.15.27:0/2625778458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee014fc20 0x7fbee0150080 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fbed800d1a0 tx=0x7fbed800ba50 comp rx=0 tx=0).stop 2024-01-11T09:45:41.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.699+0000 7fbee67ff700 1 -- 172.21.15.27:0/2625778458 shutdown_connections 2024-01-11T09:45:41.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.699+0000 7fbee67ff700 1 --2- 172.21.15.27:0/2625778458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee014fc20 0x7fbee0150080 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.699+0000 7fbee67ff700 1 --2- 172.21.15.27:0/2625778458 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee0159db0 0x7fbee015a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.699+0000 7fbee67ff700 1 -- 172.21.15.27:0/2625778458 >> 172.21.15.27:0/2625778458 conn(0x7fbee00680e0 msgr2=0x7fbee0069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:41.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.699+0000 7fbee67ff700 1 -- 172.21.15.27:0/2625778458 shutdown_connections 2024-01-11T09:45:41.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.699+0000 7fbee67ff700 1 -- 172.21.15.27:0/2625778458 wait complete. 2024-01-11T09:45:41.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.701+0000 7fbee67ff700 1 Processor -- start 2024-01-11T09:45:41.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.701+0000 7fbee67ff700 1 -- start start 2024-01-11T09:45:41.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.702+0000 7fbee67ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee014fc20 0x7fbee0149b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:41.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.702+0000 7fbee67ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee01599e0 0x7fbee014c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:41.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.702+0000 7fbee67ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee0159db0 0x7fbee014c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:41.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.702+0000 7fbee67ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee016de70 con 0x7fbee01599e0 2024-01-11T09:45:41.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.702+0000 7fbee67ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee0156be0 con 0x7fbee014fc20 2024-01-11T09:45:41.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.702+0000 7fbee67ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbee016d540 con 0x7fbee0159db0 2024-01-11T09:45:41.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.702+0000 7fbedf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee01599e0 0x7fbee014c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:41.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.702+0000 7fbedf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee01599e0 0x7fbee014c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47550/0 (socket says 172.21.15.27:47550) 2024-01-11T09:45:41.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.702+0000 7fbedf7fe700 1 -- 172.21.15.27:0/983996294 learned_addr learned my addr 172.21.15.27:0/983996294 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:41.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.702+0000 7fbedffff700 1 --2- 172.21.15.27:0/983996294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee014fc20 0x7fbee0149b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:41.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.702+0000 7fbee4d9c700 1 --2- 172.21.15.27:0/983996294 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee0159db0 0x7fbee014c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:41.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.702+0000 7fbedf7fe700 1 -- 172.21.15.27:0/983996294 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee0159db0 msgr2=0x7fbee014c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:41.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.702+0000 7fbedf7fe700 1 --2- 172.21.15.27:0/983996294 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee0159db0 0x7fbee014c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.702+0000 7fbedf7fe700 1 -- 172.21.15.27:0/983996294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee014fc20 msgr2=0x7fbee0149b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:41.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.702+0000 7fbedf7fe700 1 --2- 172.21.15.27:0/983996294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee014fc20 0x7fbee0149b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.702+0000 7fbedf7fe700 1 -- 172.21.15.27:0/983996294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed800bcf0 con 0x7fbee01599e0 2024-01-11T09:45:41.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.703+0000 7fbedf7fe700 1 --2- 172.21.15.27:0/983996294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee01599e0 0x7fbee014c2c0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fbed400d860 tx=0x7fbed4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:41.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.703+0000 7fbec7fff700 1 -- 172.21.15.27:0/983996294 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed40125f0 con 0x7fbee01599e0 2024-01-11T09:45:41.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.703+0000 7fbec7fff700 1 -- 172.21.15.27:0/983996294 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed4012750 con 0x7fbee01599e0 2024-01-11T09:45:41.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.703+0000 7fbec7fff700 1 -- 172.21.15.27:0/983996294 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed4012af0 con 0x7fbee01599e0 2024-01-11T09:45:41.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.703+0000 7fbee67ff700 1 -- 172.21.15.27:0/983996294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee016d820 con 0x7fbee01599e0 2024-01-11T09:45:41.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.703+0000 7fbee67ff700 1 -- 172.21.15.27:0/983996294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee016e300 con 0x7fbee01599e0 2024-01-11T09:45:41.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.704+0000 7fbee67ff700 1 -- 172.21.15.27:0/983996294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee00718d0 con 0x7fbee01599e0 2024-01-11T09:45:41.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.706+0000 7fbec7fff700 1 -- 172.21.15.27:0/983996294 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fbed4013480 con 0x7fbee01599e0 2024-01-11T09:45:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.706+0000 7fbec7fff700 1 --2- 172.21.15.27:0/983996294 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbecc066f30 0x7fbecc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.707+0000 7fbec7fff700 1 -- 172.21.15.27:0/983996294 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fbed4084ae0 con 0x7fbee01599e0 2024-01-11T09:45:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.707+0000 7fbedffff700 1 --2- 172.21.15.27:0/983996294 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbecc066f30 0x7fbecc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:41.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.708+0000 7fbedffff700 1 --2- 172.21.15.27:0/983996294 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbecc066f30 0x7fbecc0693e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbed0003a80 tx=0x7fbed0007ec0 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:41.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.711+0000 7fbec7fff700 1 -- 172.21.15.27:0/983996294 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbed40518e0 con 0x7fbee01599e0 2024-01-11T09:45:41.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.912+0000 7fbee67ff700 1 -- 172.21.15.27:0/983996294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbee01505c0 con 0x7fbee01599e0 2024-01-11T09:45:41.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.913+0000 7fbec7fff700 1 -- 172.21.15.27:0/983996294 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v218) v1 ==== 74+0+18727 (secure 0 0 0) 0x7fbed4054f00 con 0x7fbee01599e0 2024-01-11T09:45:41.905 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:41.905 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":218,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:45:38.954297+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:45:21.455328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-01-11T09:45:22.634298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-01-11T09:45:35.631727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-01-11T09:45:36.452195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:45:41.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.915+0000 7fbee67ff700 1 -- 172.21.15.27:0/983996294 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbecc066f30 msgr2=0x7fbecc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:41.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.915+0000 7fbee67ff700 1 --2- 172.21.15.27:0/983996294 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbecc066f30 0x7fbecc0693e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbed0003a80 tx=0x7fbed0007ec0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.915+0000 7fbee67ff700 1 -- 172.21.15.27:0/983996294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee01599e0 msgr2=0x7fbee014c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:41.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.915+0000 7fbee67ff700 1 --2- 172.21.15.27:0/983996294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee01599e0 0x7fbee014c2c0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fbed400d860 tx=0x7fbed4008040 comp rx=0 tx=0).stop 2024-01-11T09:45:41.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.915+0000 7fbee67ff700 1 -- 172.21.15.27:0/983996294 shutdown_connections 2024-01-11T09:45:41.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.915+0000 7fbee67ff700 1 --2- 172.21.15.27:0/983996294 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee0159db0 0x7fbee014c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.915+0000 7fbee67ff700 1 --2- 172.21.15.27:0/983996294 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbecc066f30 0x7fbecc0693e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.915+0000 7fbee67ff700 1 --2- 172.21.15.27:0/983996294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee014fc20 0x7fbee0149b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.915+0000 7fbee67ff700 1 --2- 172.21.15.27:0/983996294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee01599e0 0x7fbee014c2c0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:41.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.915+0000 7fbee67ff700 1 -- 172.21.15.27:0/983996294 >> 172.21.15.27:0/983996294 conn(0x7fbee00680e0 msgr2=0x7fbee006d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:41.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.915+0000 7fbee67ff700 1 -- 172.21.15.27:0/983996294 shutdown_connections 2024-01-11T09:45:41.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:41.915+0000 7fbee67ff700 1 -- 172.21.15.27:0/983996294 wait complete. 2024-01-11T09:45:41.925 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-11T09:45:41.925 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:45:42.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.134+0000 7fb6561ff700 1 Processor -- start 2024-01-11T09:45:42.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.135+0000 7fb6561ff700 1 -- start start 2024-01-11T09:45:42.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.136+0000 7fb6561ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb6501575e0 0x7fb650158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:42.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.136+0000 7fb6561ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6500b19d0 0x7fb650159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:42.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.136+0000 7fb6561ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb650158b60 con 0x7fb65006b610 2024-01-11T09:45:42.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.136+0000 7fb6561ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6500cb5e0 con 0x7fb6500b19d0 2024-01-11T09:45:42.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.136+0000 7fb6561ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6501599b0 con 0x7fb6501575e0 2024-01-11T09:45:42.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.136+0000 7fb64ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6500b19d0 0x7fb650159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:42.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.136+0000 7fb64effd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb65006b610 0x7fb6500b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41678/0 (socket says 172.21.15.27:41678) 2024-01-11T09:45:42.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.136+0000 7fb64effd700 1 -- 172.21.15.27:0/2021907888 learned_addr learned my addr 172.21.15.27:0/2021907888 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.136+0000 7fb64f7fe700 1 --2- 172.21.15.27:0/2021907888 >> v2:172.21.15.77:3300/0 conn(0x7fb6501575e0 0x7fb650158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.137+0000 7fb64e7fc700 1 -- 172.21.15.27:0/2021907888 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426810040 0 0) 0x7fb644001110 con 0x7fb65006b610 2024-01-11T09:45:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.137+0000 7fb64e7fc700 1 -- 172.21.15.27:0/2021907888 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb638003570 con 0x7fb65006b610 2024-01-11T09:45:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.137+0000 7fb64ffff700 1 -- 172.21.15.27:0/2021907888 >> v2:172.21.15.77:3300/0 conn(0x7fb6501575e0 msgr2=0x7fb650158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.137+0000 7fb64ffff700 1 --2- 172.21.15.27:0/2021907888 >> v2:172.21.15.77:3300/0 conn(0x7fb6501575e0 0x7fb650158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.137+0000 7fb64ffff700 1 -- 172.21.15.27:0/2021907888 >> v1:172.21.15.27:6789/0 conn(0x7fb65006b610 legacy=0x7fb6500b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.137+0000 7fb64ffff700 1 -- 172.21.15.27:0/2021907888 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6500c0f90 con 0x7fb6500b19d0 2024-01-11T09:45:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.137+0000 7fb64ffff700 1 --2- 172.21.15.27:0/2021907888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6500b19d0 0x7fb650159e40 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fb650064b00 tx=0x7fb648009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7decb4de86fbed3a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.137+0000 7fb64e7fc700 1 -- 172.21.15.27:0/2021907888 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64800e070 con 0x7fb6500b19d0 2024-01-11T09:45:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.137+0000 7fb64e7fc700 1 -- 172.21.15.27:0/2021907888 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb64800bb60 con 0x7fb6500b19d0 2024-01-11T09:45:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.137+0000 7fb64e7fc700 1 -- 172.21.15.27:0/2021907888 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb648012430 con 0x7fb6500b19d0 2024-01-11T09:45:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.138+0000 7fb6561ff700 1 -- 172.21.15.27:0/2021907888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6500b19d0 msgr2=0x7fb650159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.138+0000 7fb6561ff700 1 --2- 172.21.15.27:0/2021907888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6500b19d0 0x7fb650159e40 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fb650064b00 tx=0x7fb648009780 comp rx=0 tx=0).stop 2024-01-11T09:45:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.138+0000 7fb6561ff700 1 -- 172.21.15.27:0/2021907888 shutdown_connections 2024-01-11T09:45:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.138+0000 7fb6561ff700 1 --2- 172.21.15.27:0/2021907888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6500b19d0 0x7fb650159e40 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.138+0000 7fb6561ff700 1 --2- 172.21.15.27:0/2021907888 >> v2:172.21.15.77:3300/0 conn(0x7fb6501575e0 0x7fb650158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.138+0000 7fb6561ff700 1 -- 172.21.15.27:0/2021907888 >> 172.21.15.27:0/2021907888 conn(0x7fb6500680e0 msgr2=0x7fb650069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.138+0000 7fb6561ff700 1 -- 172.21.15.27:0/2021907888 shutdown_connections 2024-01-11T09:45:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.138+0000 7fb6561ff700 1 -- 172.21.15.27:0/2021907888 wait complete. 2024-01-11T09:45:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.140+0000 7fb6561ff700 1 Processor -- start 2024-01-11T09:45:42.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.140+0000 7fb6561ff700 1 -- start start 2024-01-11T09:45:42.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.142+0000 7fb6561ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb65006b610 0x7fb650153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.142+0000 7fb6561ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb6500b19d0 0x7fb65006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.142+0000 7fb6561ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6501575e0 0x7fb65006f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.142+0000 7fb6561ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6500a5490 con 0x7fb65006b610 2024-01-11T09:45:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.142+0000 7fb6561ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6500c2c80 con 0x7fb6501575e0 2024-01-11T09:45:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.142+0000 7fb6561ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6500c82e0 con 0x7fb6500b19d0 2024-01-11T09:45:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.142+0000 7fb64f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb65006b610 0x7fb650153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.142+0000 7fb64f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb65006b610 0x7fb650153e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47564/0 (socket says 172.21.15.27:47564) 2024-01-11T09:45:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.142+0000 7fb64f7fe700 1 -- 172.21.15.27:0/714880850 learned_addr learned my addr 172.21.15.27:0/714880850 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.142+0000 7fb64effd700 1 --2- 172.21.15.27:0/714880850 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb6500b19d0 0x7fb65006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.142+0000 7fb64f7fe700 1 -- 172.21.15.27:0/714880850 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb6500b19d0 msgr2=0x7fb65006ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.142+0000 7fb64ffff700 1 --2- 172.21.15.27:0/714880850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6501575e0 0x7fb65006f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.142+0000 7fb64f7fe700 1 --2- 172.21.15.27:0/714880850 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb6500b19d0 0x7fb65006ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.142+0000 7fb64f7fe700 1 -- 172.21.15.27:0/714880850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6501575e0 msgr2=0x7fb65006f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.142+0000 7fb64f7fe700 1 --2- 172.21.15.27:0/714880850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6501575e0 0x7fb65006f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.142+0000 7fb64f7fe700 1 -- 172.21.15.27:0/714880850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb648009a20 con 0x7fb65006b610 2024-01-11T09:45:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.142+0000 7fb64effd700 1 --2- 172.21.15.27:0/714880850 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb6500b19d0 0x7fb65006ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.142+0000 7fb64ffff700 1 --2- 172.21.15.27:0/714880850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6501575e0 0x7fb65006f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.143+0000 7fb64f7fe700 1 --2- 172.21.15.27:0/714880850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb65006b610 0x7fb650153e30 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fb64000d860 tx=0x7fb640008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.143+0000 7fb6377fe700 1 -- 172.21.15.27:0/714880850 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb640009610 con 0x7fb65006b610 2024-01-11T09:45:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.143+0000 7fb6377fe700 1 -- 172.21.15.27:0/714880850 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb640009770 con 0x7fb65006b610 2024-01-11T09:45:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.143+0000 7fb6377fe700 1 -- 172.21.15.27:0/714880850 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb640009b30 con 0x7fb65006b610 2024-01-11T09:45:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.143+0000 7fb6561ff700 1 -- 172.21.15.27:0/714880850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6500c85c0 con 0x7fb65006b610 2024-01-11T09:45:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.143+0000 7fb6561ff700 1 -- 172.21.15.27:0/714880850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6500c2910 con 0x7fb65006b610 2024-01-11T09:45:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.144+0000 7fb6561ff700 1 -- 172.21.15.27:0/714880850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6500718d0 con 0x7fb65006b610 2024-01-11T09:45:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.145+0000 7fb6377fe700 1 -- 172.21.15.27:0/714880850 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fb64001e460 con 0x7fb65006b610 2024-01-11T09:45:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.146+0000 7fb6377fe700 1 --2- 172.21.15.27:0/714880850 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb63c066f30 0x7fb63c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.146+0000 7fb6377fe700 1 -- 172.21.15.27:0/714880850 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fb640084f90 con 0x7fb65006b610 2024-01-11T09:45:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.147+0000 7fb64effd700 1 --2- 172.21.15.27:0/714880850 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb63c066f30 0x7fb63c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.147+0000 7fb64effd700 1 --2- 172.21.15.27:0/714880850 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb63c066f30 0x7fb63c0693e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb6440058d0 tx=0x7fb644009040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:42.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.150+0000 7fb6377fe700 1 -- 172.21.15.27:0/714880850 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb640051d90 con 0x7fb65006b610 2024-01-11T09:45:42.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.347+0000 7fb6561ff700 1 -- 172.21.15.27:0/714880850 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb650159330 con 0x7fb63c066f30 2024-01-11T09:45:42.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.364+0000 7fb6377fe700 1 -- 172.21.15.27:0/714880850 <== mgr.10380 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528648 (secure 0 0 0) 0x7fb650159330 con 0x7fb63c066f30 2024-01-11T09:45:42.363 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:42.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.370+0000 7fb6561ff700 1 -- 172.21.15.27:0/714880850 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb63c066f30 msgr2=0x7fb63c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:42.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.370+0000 7fb6561ff700 1 --2- 172.21.15.27:0/714880850 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb63c066f30 0x7fb63c0693e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb6440058d0 tx=0x7fb644009040 comp rx=0 tx=0).stop 2024-01-11T09:45:42.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.370+0000 7fb6561ff700 1 -- 172.21.15.27:0/714880850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb65006b610 msgr2=0x7fb650153e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:42.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.370+0000 7fb6561ff700 1 --2- 172.21.15.27:0/714880850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb65006b610 0x7fb650153e30 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fb64000d860 tx=0x7fb640008040 comp rx=0 tx=0).stop 2024-01-11T09:45:42.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.370+0000 7fb6561ff700 1 -- 172.21.15.27:0/714880850 shutdown_connections 2024-01-11T09:45:42.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.370+0000 7fb6561ff700 1 --2- 172.21.15.27:0/714880850 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb63c066f30 0x7fb63c0693e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:42.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.370+0000 7fb6561ff700 1 --2- 172.21.15.27:0/714880850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6501575e0 0x7fb65006f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:42.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.370+0000 7fb6561ff700 1 --2- 172.21.15.27:0/714880850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb65006b610 0x7fb650153e30 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:42.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.370+0000 7fb6561ff700 1 --2- 172.21.15.27:0/714880850 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb6500b19d0 0x7fb65006ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:42.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.370+0000 7fb6561ff700 1 -- 172.21.15.27:0/714880850 >> 172.21.15.27:0/714880850 conn(0x7fb6500680e0 msgr2=0x7fb6501556e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:42.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.370+0000 7fb6561ff700 1 -- 172.21.15.27:0/714880850 shutdown_connections 2024-01-11T09:45:42.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.370+0000 7fb6561ff700 1 -- 172.21.15.27:0/714880850 wait complete. 2024-01-11T09:45:42.369 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:45:42.379 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-01-11T09:45:42.304619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596193,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":702,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2462408,"kb_used_data":22664,"kb_used_omap":0,"kb_used_meta":2439680,"kb_avail":752512312,"statfs":{"total":773094113280,"available":770572607488,"internally_reserved":0,"allocated":23207936,"data_stored":11679610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2498232320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001482"},"pg_stats":[{"pgid":"1.0","version":"205'366","reported_seq":1203,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954188+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:45:37.954188+0000","last_peered":"2024-01-11T09:45:37.954188+0000","last_clean":"2024-01-11T09:45:37.954188+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:45:37.954188+0000","last_undegraded":"2024-01-11T09:45:37.954188+0000","last_fullsized":"2024-01-11T09:45:37.954188+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":611,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"213'1","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.973258+0000","last_change":"2024-01-11T09:45:23.147910+0000","last_active":"2024-01-11T09:45:37.973258+0000","last_peered":"2024-01-11T09:45:37.973258+0000","last_clean":"2024-01-11T09:45:37.973258+0000","last_became_active":"2024-01-11T09:45:23.147626+0000","last_became_peered":"2024-01-11T09:45:23.147626+0000","last_unstale":"2024-01-11T09:45:37.973258+0000","last_undegraded":"2024-01-11T09:45:37.973258+0000","last_fullsized":"2024-01-11T09:45:37.973258+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:32:18.360961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.956527+0000","last_change":"2024-01-11T09:45:24.169860+0000","last_active":"2024-01-11T09:45:37.956527+0000","last_peered":"2024-01-11T09:45:37.956527+0000","last_clean":"2024-01-11T09:45:37.956527+0000","last_became_active":"2024-01-11T09:45:24.169370+0000","last_became_peered":"2024-01-11T09:45:24.169370+0000","last_unstale":"2024-01-11T09:45:37.956527+0000","last_undegraded":"2024-01-11T09:45:37.956527+0000","last_fullsized":"2024-01-11T09:45:37.956527+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:15:38.555186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"217'2","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.983247+0000","last_change":"2024-01-11T09:45:36.976365+0000","last_active":"2024-01-11T09:45:37.983247+0000","last_peered":"2024-01-11T09:45:37.983247+0000","last_clean":"2024-01-11T09:45:37.983247+0000","last_became_active":"2024-01-11T09:45:36.976041+0000","last_became_peered":"2024-01-11T09:45:36.976041+0000","last_unstale":"2024-01-11T09:45:37.983247+0000","last_undegraded":"2024-01-11T09:45:37.983247+0000","last_fullsized":"2024-01-11T09:45:37.983247+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:40.586757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.972306+0000","last_change":"2024-01-11T09:45:37.968694+0000","last_active":"2024-01-11T09:45:37.972306+0000","last_peered":"2024-01-11T09:45:37.972306+0000","last_clean":"2024-01-11T09:45:37.972306+0000","last_became_active":"2024-01-11T09:45:37.966845+0000","last_became_peered":"2024-01-11T09:45:37.966845+0000","last_unstale":"2024-01-11T09:45:37.972306+0000","last_undegraded":"2024-01-11T09:45:37.972306+0000","last_fullsized":"2024-01-11T09:45:37.972306+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:47:37.322693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.969312+0000","last_change":"2024-01-11T09:45:23.158577+0000","last_active":"2024-01-11T09:45:37.969312+0000","last_peered":"2024-01-11T09:45:37.969312+0000","last_clean":"2024-01-11T09:45:37.969312+0000","last_became_active":"2024-01-11T09:45:23.158304+0000","last_became_peered":"2024-01-11T09:45:23.158304+0000","last_unstale":"2024-01-11T09:45:37.969312+0000","last_undegraded":"2024-01-11T09:45:37.969312+0000","last_fullsized":"2024-01-11T09:45:37.969312+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:59:55.124493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.948096+0000","last_change":"2024-01-11T09:45:24.153383+0000","last_active":"2024-01-11T09:45:37.948096+0000","last_peered":"2024-01-11T09:45:37.948096+0000","last_clean":"2024-01-11T09:45:37.948096+0000","last_became_active":"2024-01-11T09:45:24.153089+0000","last_became_peered":"2024-01-11T09:45:24.153089+0000","last_unstale":"2024-01-11T09:45:37.948096+0000","last_undegraded":"2024-01-11T09:45:37.948096+0000","last_fullsized":"2024-01-11T09:45:37.948096+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:49:15.804305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954712+0000","last_change":"2024-01-11T09:45:36.973450+0000","last_active":"2024-01-11T09:45:37.954712+0000","last_peered":"2024-01-11T09:45:37.954712+0000","last_clean":"2024-01-11T09:45:37.954712+0000","last_became_active":"2024-01-11T09:45:36.973089+0000","last_became_peered":"2024-01-11T09:45:36.973089+0000","last_unstale":"2024-01-11T09:45:37.954712+0000","last_undegraded":"2024-01-11T09:45:37.954712+0000","last_fullsized":"2024-01-11T09:45:37.954712+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:39:18.270597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.986808+0000","last_change":"2024-01-11T09:45:37.983052+0000","last_active":"2024-01-11T09:45:37.986808+0000","last_peered":"2024-01-11T09:45:37.986808+0000","last_clean":"2024-01-11T09:45:37.986808+0000","last_became_active":"2024-01-11T09:45:37.982783+0000","last_became_peered":"2024-01-11T09:45:37.982783+0000","last_unstale":"2024-01-11T09:45:37.986808+0000","last_undegraded":"2024-01-11T09:45:37.986808+0000","last_fullsized":"2024-01-11T09:45:37.986808+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:13:51.430167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949982+0000","last_change":"2024-01-11T09:45:23.154043+0000","last_active":"2024-01-11T09:45:37.949982+0000","last_peered":"2024-01-11T09:45:37.949982+0000","last_clean":"2024-01-11T09:45:37.949982+0000","last_became_active":"2024-01-11T09:45:23.153529+0000","last_became_peered":"2024-01-11T09:45:23.153529+0000","last_unstale":"2024-01-11T09:45:37.949982+0000","last_undegraded":"2024-01-11T09:45:37.949982+0000","last_fullsized":"2024-01-11T09:45:37.949982+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:48.360444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.955840+0000","last_change":"2024-01-11T09:45:24.163398+0000","last_active":"2024-01-11T09:45:37.955840+0000","last_peered":"2024-01-11T09:45:37.955840+0000","last_clean":"2024-01-11T09:45:37.955840+0000","last_became_active":"2024-01-11T09:45:24.159101+0000","last_became_peered":"2024-01-11T09:45:24.159101+0000","last_unstale":"2024-01-11T09:45:37.955840+0000","last_undegraded":"2024-01-11T09:45:37.955840+0000","last_fullsized":"2024-01-11T09:45:37.955840+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:40.983480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"217'2","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.988007+0000","last_change":"2024-01-11T09:45:36.974931+0000","last_active":"2024-01-11T09:45:37.988007+0000","last_peered":"2024-01-11T09:45:37.988007+0000","last_clean":"2024-01-11T09:45:37.988007+0000","last_became_active":"2024-01-11T09:45:36.974149+0000","last_became_peered":"2024-01-11T09:45:36.974149+0000","last_unstale":"2024-01-11T09:45:37.988007+0000","last_undegraded":"2024-01-11T09:45:37.988007+0000","last_fullsized":"2024-01-11T09:45:37.988007+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:15:49.588159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.978949+0000","last_change":"2024-01-11T09:45:37.968718+0000","last_active":"2024-01-11T09:45:37.978949+0000","last_peered":"2024-01-11T09:45:37.978949+0000","last_clean":"2024-01-11T09:45:37.978949+0000","last_became_active":"2024-01-11T09:45:37.968431+0000","last_became_peered":"2024-01-11T09:45:37.968431+0000","last_unstale":"2024-01-11T09:45:37.978949+0000","last_undegraded":"2024-01-11T09:45:37.978949+0000","last_fullsized":"2024-01-11T09:45:37.978949+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:17:38.232406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.958199+0000","last_change":"2024-01-11T09:45:23.151649+0000","last_active":"2024-01-11T09:45:37.958199+0000","last_peered":"2024-01-11T09:45:37.958199+0000","last_clean":"2024-01-11T09:45:37.958199+0000","last_became_active":"2024-01-11T09:45:23.151399+0000","last_became_peered":"2024-01-11T09:45:23.151399+0000","last_unstale":"2024-01-11T09:45:37.958199+0000","last_undegraded":"2024-01-11T09:45:37.958199+0000","last_fullsized":"2024-01-11T09:45:37.958199+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:02.352990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.967480+0000","last_change":"2024-01-11T09:45:24.157004+0000","last_active":"2024-01-11T09:45:37.967480+0000","last_peered":"2024-01-11T09:45:37.967480+0000","last_clean":"2024-01-11T09:45:37.967480+0000","last_became_active":"2024-01-11T09:45:24.156446+0000","last_became_peered":"2024-01-11T09:45:24.156446+0000","last_unstale":"2024-01-11T09:45:37.967480+0000","last_undegraded":"2024-01-11T09:45:37.967480+0000","last_fullsized":"2024-01-11T09:45:37.967480+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:28:18.397764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.972338+0000","last_change":"2024-01-11T09:45:36.971222+0000","last_active":"2024-01-11T09:45:37.972338+0000","last_peered":"2024-01-11T09:45:37.972338+0000","last_clean":"2024-01-11T09:45:37.972338+0000","last_became_active":"2024-01-11T09:45:36.970967+0000","last_became_peered":"2024-01-11T09:45:36.970967+0000","last_unstale":"2024-01-11T09:45:37.972338+0000","last_undegraded":"2024-01-11T09:45:37.972338+0000","last_fullsized":"2024-01-11T09:45:37.972338+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:55.529264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.983024+0000","last_change":"2024-01-11T09:45:37.974201+0000","last_active":"2024-01-11T09:45:37.983024+0000","last_peered":"2024-01-11T09:45:37.983024+0000","last_clean":"2024-01-11T09:45:37.983024+0000","last_became_active":"2024-01-11T09:45:37.973939+0000","last_became_peered":"2024-01-11T09:45:37.973939+0000","last_unstale":"2024-01-11T09:45:37.983024+0000","last_undegraded":"2024-01-11T09:45:37.983024+0000","last_fullsized":"2024-01-11T09:45:37.983024+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:37:34.036345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954500+0000","last_change":"2024-01-11T09:45:23.151324+0000","last_active":"2024-01-11T09:45:37.954500+0000","last_peered":"2024-01-11T09:45:37.954500+0000","last_clean":"2024-01-11T09:45:37.954500+0000","last_became_active":"2024-01-11T09:45:23.150822+0000","last_became_peered":"2024-01-11T09:45:23.150822+0000","last_unstale":"2024-01-11T09:45:37.954500+0000","last_undegraded":"2024-01-11T09:45:37.954500+0000","last_fullsized":"2024-01-11T09:45:37.954500+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:39.975619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.956687+0000","last_change":"2024-01-11T09:45:24.158240+0000","last_active":"2024-01-11T09:45:37.956687+0000","last_peered":"2024-01-11T09:45:37.956687+0000","last_clean":"2024-01-11T09:45:37.956687+0000","last_became_active":"2024-01-11T09:45:24.157938+0000","last_became_peered":"2024-01-11T09:45:24.157938+0000","last_unstale":"2024-01-11T09:45:37.956687+0000","last_undegraded":"2024-01-11T09:45:37.956687+0000","last_fullsized":"2024-01-11T09:45:37.956687+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:08:44.293978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.955218+0000","last_change":"2024-01-11T09:45:36.974347+0000","last_active":"2024-01-11T09:45:37.955218+0000","last_peered":"2024-01-11T09:45:37.955218+0000","last_clean":"2024-01-11T09:45:37.955218+0000","last_became_active":"2024-01-11T09:45:36.973974+0000","last_became_peered":"2024-01-11T09:45:36.973974+0000","last_unstale":"2024-01-11T09:45:37.955218+0000","last_undegraded":"2024-01-11T09:45:37.955218+0000","last_fullsized":"2024-01-11T09:45:37.955218+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:06:58.345396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984617+0000","last_change":"2024-01-11T09:45:37.976200+0000","last_active":"2024-01-11T09:45:37.984617+0000","last_peered":"2024-01-11T09:45:37.984617+0000","last_clean":"2024-01-11T09:45:37.984617+0000","last_became_active":"2024-01-11T09:45:37.975890+0000","last_became_peered":"2024-01-11T09:45:37.975890+0000","last_unstale":"2024-01-11T09:45:37.984617+0000","last_undegraded":"2024-01-11T09:45:37.984617+0000","last_fullsized":"2024-01-11T09:45:37.984617+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:58:38.459913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960857+0000","last_change":"2024-01-11T09:45:23.147642+0000","last_active":"2024-01-11T09:45:37.960857+0000","last_peered":"2024-01-11T09:45:37.960857+0000","last_clean":"2024-01-11T09:45:37.960857+0000","last_became_active":"2024-01-11T09:45:23.147387+0000","last_became_peered":"2024-01-11T09:45:23.147387+0000","last_unstale":"2024-01-11T09:45:37.960857+0000","last_undegraded":"2024-01-11T09:45:37.960857+0000","last_fullsized":"2024-01-11T09:45:37.960857+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:49:33.086505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950588+0000","last_change":"2024-01-11T09:45:24.154386+0000","last_active":"2024-01-11T09:45:37.950588+0000","last_peered":"2024-01-11T09:45:37.950588+0000","last_clean":"2024-01-11T09:45:37.950588+0000","last_became_active":"2024-01-11T09:45:24.154017+0000","last_became_peered":"2024-01-11T09:45:24.154017+0000","last_unstale":"2024-01-11T09:45:37.950588+0000","last_undegraded":"2024-01-11T09:45:37.950588+0000","last_fullsized":"2024-01-11T09:45:37.950588+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:33:41.503661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.956518+0000","last_change":"2024-01-11T09:45:36.973464+0000","last_active":"2024-01-11T09:45:37.956518+0000","last_peered":"2024-01-11T09:45:37.956518+0000","last_clean":"2024-01-11T09:45:37.956518+0000","last_became_active":"2024-01-11T09:45:36.973216+0000","last_became_peered":"2024-01-11T09:45:36.973216+0000","last_unstale":"2024-01-11T09:45:37.956518+0000","last_undegraded":"2024-01-11T09:45:37.956518+0000","last_fullsized":"2024-01-11T09:45:37.956518+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:57:54.872386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.985401+0000","last_change":"2024-01-11T09:45:37.978430+0000","last_active":"2024-01-11T09:45:37.985401+0000","last_peered":"2024-01-11T09:45:37.985401+0000","last_clean":"2024-01-11T09:45:37.985401+0000","last_became_active":"2024-01-11T09:45:37.977974+0000","last_became_peered":"2024-01-11T09:45:37.977974+0000","last_unstale":"2024-01-11T09:45:37.985401+0000","last_undegraded":"2024-01-11T09:45:37.985401+0000","last_fullsized":"2024-01-11T09:45:37.985401+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:57:55.878951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.967642+0000","last_change":"2024-01-11T09:45:23.148631+0000","last_active":"2024-01-11T09:45:37.967642+0000","last_peered":"2024-01-11T09:45:37.967642+0000","last_clean":"2024-01-11T09:45:37.967642+0000","last_became_active":"2024-01-11T09:45:23.147091+0000","last_became_peered":"2024-01-11T09:45:23.147091+0000","last_unstale":"2024-01-11T09:45:37.967642+0000","last_undegraded":"2024-01-11T09:45:37.967642+0000","last_fullsized":"2024-01-11T09:45:37.967642+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:49:14.796444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949340+0000","last_change":"2024-01-11T09:45:24.157902+0000","last_active":"2024-01-11T09:45:37.949340+0000","last_peered":"2024-01-11T09:45:37.949340+0000","last_clean":"2024-01-11T09:45:37.949340+0000","last_became_active":"2024-01-11T09:45:24.157443+0000","last_became_peered":"2024-01-11T09:45:24.157443+0000","last_unstale":"2024-01-11T09:45:37.949340+0000","last_undegraded":"2024-01-11T09:45:37.949340+0000","last_fullsized":"2024-01-11T09:45:37.949340+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:41:10.078741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949378+0000","last_change":"2024-01-11T09:45:36.967374+0000","last_active":"2024-01-11T09:45:37.949378+0000","last_peered":"2024-01-11T09:45:37.949378+0000","last_clean":"2024-01-11T09:45:37.949378+0000","last_became_active":"2024-01-11T09:45:36.967001+0000","last_became_peered":"2024-01-11T09:45:36.967001+0000","last_unstale":"2024-01-11T09:45:37.949378+0000","last_undegraded":"2024-01-11T09:45:37.949378+0000","last_fullsized":"2024-01-11T09:45:37.949378+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:42:34.355866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.972812+0000","last_change":"2024-01-11T09:45:37.964730+0000","last_active":"2024-01-11T09:45:37.972812+0000","last_peered":"2024-01-11T09:45:37.972812+0000","last_clean":"2024-01-11T09:45:37.972812+0000","last_became_active":"2024-01-11T09:45:37.964406+0000","last_became_peered":"2024-01-11T09:45:37.964406+0000","last_unstale":"2024-01-11T09:45:37.972812+0000","last_undegraded":"2024-01-11T09:45:37.972812+0000","last_fullsized":"2024-01-11T09:45:37.972812+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:29:35.726697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.6","version":"214'1","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954993+0000","last_change":"2024-01-11T09:45:23.149174+0000","last_active":"2024-01-11T09:45:37.954993+0000","last_peered":"2024-01-11T09:45:37.954993+0000","last_clean":"2024-01-11T09:45:37.954993+0000","last_became_active":"2024-01-11T09:45:23.148798+0000","last_became_peered":"2024-01-11T09:45:23.148798+0000","last_unstale":"2024-01-11T09:45:37.954993+0000","last_undegraded":"2024-01-11T09:45:37.954993+0000","last_fullsized":"2024-01-11T09:45:37.954993+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:52:31.488984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.963297+0000","last_change":"2024-01-11T09:45:24.162504+0000","last_active":"2024-01-11T09:45:37.963297+0000","last_peered":"2024-01-11T09:45:37.963297+0000","last_clean":"2024-01-11T09:45:37.963297+0000","last_became_active":"2024-01-11T09:45:24.162179+0000","last_became_peered":"2024-01-11T09:45:24.162179+0000","last_unstale":"2024-01-11T09:45:37.963297+0000","last_undegraded":"2024-01-11T09:45:37.963297+0000","last_fullsized":"2024-01-11T09:45:37.963297+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:47:23.503381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.966665+0000","last_change":"2024-01-11T09:45:36.973086+0000","last_active":"2024-01-11T09:45:37.966665+0000","last_peered":"2024-01-11T09:45:37.966665+0000","last_clean":"2024-01-11T09:45:37.966665+0000","last_became_active":"2024-01-11T09:45:36.972786+0000","last_became_peered":"2024-01-11T09:45:36.972786+0000","last_unstale":"2024-01-11T09:45:37.966665+0000","last_undegraded":"2024-01-11T09:45:37.966665+0000","last_fullsized":"2024-01-11T09:45:37.966665+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:47:36.316129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.972164+0000","last_change":"2024-01-11T09:45:37.965771+0000","last_active":"2024-01-11T09:45:37.972164+0000","last_peered":"2024-01-11T09:45:37.972164+0000","last_clean":"2024-01-11T09:45:37.972164+0000","last_became_active":"2024-01-11T09:45:37.965461+0000","last_became_peered":"2024-01-11T09:45:37.965461+0000","last_unstale":"2024-01-11T09:45:37.972164+0000","last_undegraded":"2024-01-11T09:45:37.972164+0000","last_fullsized":"2024-01-11T09:45:37.972164+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:29:35.726697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.7","version":"213'2","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960712+0000","last_change":"2024-01-11T09:45:23.148348+0000","last_active":"2024-01-11T09:45:37.960712+0000","last_peered":"2024-01-11T09:45:37.960712+0000","last_clean":"2024-01-11T09:45:37.960712+0000","last_became_active":"2024-01-11T09:45:23.147723+0000","last_became_peered":"2024-01-11T09:45:23.147723+0000","last_unstale":"2024-01-11T09:45:37.960712+0000","last_undegraded":"2024-01-11T09:45:37.960712+0000","last_fullsized":"2024-01-11T09:45:37.960712+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:48:39.924839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.971682+0000","last_change":"2024-01-11T09:45:24.149162+0000","last_active":"2024-01-11T09:45:37.971682+0000","last_peered":"2024-01-11T09:45:37.971682+0000","last_clean":"2024-01-11T09:45:37.971682+0000","last_became_active":"2024-01-11T09:45:24.148510+0000","last_became_peered":"2024-01-11T09:45:24.148510+0000","last_unstale":"2024-01-11T09:45:37.971682+0000","last_undegraded":"2024-01-11T09:45:37.971682+0000","last_fullsized":"2024-01-11T09:45:37.971682+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:56:51.623335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.955067+0000","last_change":"2024-01-11T09:45:36.972976+0000","last_active":"2024-01-11T09:45:37.955067+0000","last_peered":"2024-01-11T09:45:37.955067+0000","last_clean":"2024-01-11T09:45:37.955067+0000","last_became_active":"2024-01-11T09:45:36.972185+0000","last_became_peered":"2024-01-11T09:45:36.972185+0000","last_unstale":"2024-01-11T09:45:37.955067+0000","last_undegraded":"2024-01-11T09:45:37.955067+0000","last_fullsized":"2024-01-11T09:45:37.955067+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:52:51.394224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.989010+0000","last_change":"2024-01-11T09:45:37.986910+0000","last_active":"2024-01-11T09:45:37.989010+0000","last_peered":"2024-01-11T09:45:37.989010+0000","last_clean":"2024-01-11T09:45:37.989010+0000","last_became_active":"2024-01-11T09:45:37.986238+0000","last_became_peered":"2024-01-11T09:45:37.986238+0000","last_unstale":"2024-01-11T09:45:37.989010+0000","last_undegraded":"2024-01-11T09:45:37.989010+0000","last_fullsized":"2024-01-11T09:45:37.989010+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:39:29.824585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.961220+0000","last_change":"2024-01-11T09:45:23.154230+0000","last_active":"2024-01-11T09:45:37.961220+0000","last_peered":"2024-01-11T09:45:37.961220+0000","last_clean":"2024-01-11T09:45:37.961220+0000","last_became_active":"2024-01-11T09:45:23.153324+0000","last_became_peered":"2024-01-11T09:45:23.153324+0000","last_unstale":"2024-01-11T09:45:37.961220+0000","last_undegraded":"2024-01-11T09:45:37.961220+0000","last_fullsized":"2024-01-11T09:45:37.961220+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:52:16.865748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950176+0000","last_change":"2024-01-11T09:45:24.150105+0000","last_active":"2024-01-11T09:45:37.950176+0000","last_peered":"2024-01-11T09:45:37.950176+0000","last_clean":"2024-01-11T09:45:37.950176+0000","last_became_active":"2024-01-11T09:45:24.149005+0000","last_became_peered":"2024-01-11T09:45:24.149005+0000","last_unstale":"2024-01-11T09:45:37.950176+0000","last_undegraded":"2024-01-11T09:45:37.950176+0000","last_fullsized":"2024-01-11T09:45:37.950176+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:25:27.316660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"217'2","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.985232+0000","last_change":"2024-01-11T09:45:36.972899+0000","last_active":"2024-01-11T09:45:37.985232+0000","last_peered":"2024-01-11T09:45:37.985232+0000","last_clean":"2024-01-11T09:45:37.985232+0000","last_became_active":"2024-01-11T09:45:36.972632+0000","last_became_peered":"2024-01-11T09:45:36.972632+0000","last_unstale":"2024-01-11T09:45:37.985232+0000","last_undegraded":"2024-01-11T09:45:37.985232+0000","last_fullsized":"2024-01-11T09:45:37.985232+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:30:48.523200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.978604+0000","last_change":"2024-01-11T09:45:37.973953+0000","last_active":"2024-01-11T09:45:37.978604+0000","last_peered":"2024-01-11T09:45:37.978604+0000","last_clean":"2024-01-11T09:45:37.978604+0000","last_became_active":"2024-01-11T09:45:37.973677+0000","last_became_peered":"2024-01-11T09:45:37.973677+0000","last_unstale":"2024-01-11T09:45:37.978604+0000","last_undegraded":"2024-01-11T09:45:37.978604+0000","last_fullsized":"2024-01-11T09:45:37.978604+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:04:16.164163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.958878+0000","last_change":"2024-01-11T09:45:23.155951+0000","last_active":"2024-01-11T09:45:37.958878+0000","last_peered":"2024-01-11T09:45:37.958878+0000","last_clean":"2024-01-11T09:45:37.958878+0000","last_became_active":"2024-01-11T09:45:23.155673+0000","last_became_peered":"2024-01-11T09:45:23.155673+0000","last_unstale":"2024-01-11T09:45:37.958878+0000","last_undegraded":"2024-01-11T09:45:37.958878+0000","last_fullsized":"2024-01-11T09:45:37.958878+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:52:37.573615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.947596+0000","last_change":"2024-01-11T09:45:24.144965+0000","last_active":"2024-01-11T09:45:37.947596+0000","last_peered":"2024-01-11T09:45:37.947596+0000","last_clean":"2024-01-11T09:45:37.947596+0000","last_became_active":"2024-01-11T09:45:24.143991+0000","last_became_peered":"2024-01-11T09:45:24.143991+0000","last_unstale":"2024-01-11T09:45:37.947596+0000","last_undegraded":"2024-01-11T09:45:37.947596+0000","last_fullsized":"2024-01-11T09:45:37.947596+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:56:42.734248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"217'1","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.986143+0000","last_change":"2024-01-11T09:45:36.974662+0000","last_active":"2024-01-11T09:45:37.986143+0000","last_peered":"2024-01-11T09:45:37.986143+0000","last_clean":"2024-01-11T09:45:37.986143+0000","last_became_active":"2024-01-11T09:45:36.974322+0000","last_became_peered":"2024-01-11T09:45:36.974322+0000","last_unstale":"2024-01-11T09:45:37.986143+0000","last_undegraded":"2024-01-11T09:45:37.986143+0000","last_fullsized":"2024-01-11T09:45:37.986143+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:04:54.345995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984029+0000","last_change":"2024-01-11T09:45:37.977839+0000","last_active":"2024-01-11T09:45:37.984029+0000","last_peered":"2024-01-11T09:45:37.984029+0000","last_clean":"2024-01-11T09:45:37.984029+0000","last_became_active":"2024-01-11T09:45:37.977406+0000","last_became_peered":"2024-01-11T09:45:37.977406+0000","last_unstale":"2024-01-11T09:45:37.984029+0000","last_undegraded":"2024-01-11T09:45:37.984029+0000","last_fullsized":"2024-01-11T09:45:37.984029+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:10:33.893904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.956444+0000","last_change":"2024-01-11T09:45:23.154111+0000","last_active":"2024-01-11T09:45:37.956444+0000","last_peered":"2024-01-11T09:45:37.956444+0000","last_clean":"2024-01-11T09:45:37.956444+0000","last_became_active":"2024-01-11T09:45:23.151742+0000","last_became_peered":"2024-01-11T09:45:23.151742+0000","last_unstale":"2024-01-11T09:45:37.956444+0000","last_undegraded":"2024-01-11T09:45:37.956444+0000","last_fullsized":"2024-01-11T09:45:37.956444+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:38.398157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960261+0000","last_change":"2024-01-11T09:45:24.168028+0000","last_active":"2024-01-11T09:45:37.960261+0000","last_peered":"2024-01-11T09:45:37.960261+0000","last_clean":"2024-01-11T09:45:37.960261+0000","last_became_active":"2024-01-11T09:45:24.167726+0000","last_became_peered":"2024-01-11T09:45:24.167726+0000","last_unstale":"2024-01-11T09:45:37.960261+0000","last_undegraded":"2024-01-11T09:45:37.960261+0000","last_fullsized":"2024-01-11T09:45:37.960261+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:48:06.986946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949293+0000","last_change":"2024-01-11T09:45:36.973677+0000","last_active":"2024-01-11T09:45:37.949293+0000","last_peered":"2024-01-11T09:45:37.949293+0000","last_clean":"2024-01-11T09:45:37.949293+0000","last_became_active":"2024-01-11T09:45:36.971802+0000","last_became_peered":"2024-01-11T09:45:36.971802+0000","last_unstale":"2024-01-11T09:45:37.949293+0000","last_undegraded":"2024-01-11T09:45:37.949293+0000","last_fullsized":"2024-01-11T09:45:37.949293+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:04:54.345995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.983824+0000","last_change":"2024-01-11T09:45:37.975972+0000","last_active":"2024-01-11T09:45:37.983824+0000","last_peered":"2024-01-11T09:45:37.983824+0000","last_clean":"2024-01-11T09:45:37.983824+0000","last_became_active":"2024-01-11T09:45:37.975672+0000","last_became_peered":"2024-01-11T09:45:37.975672+0000","last_unstale":"2024-01-11T09:45:37.983824+0000","last_undegraded":"2024-01-11T09:45:37.983824+0000","last_fullsized":"2024-01-11T09:45:37.983824+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:23:48.270235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"213'2","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949253+0000","last_change":"2024-01-11T09:45:23.143823+0000","last_active":"2024-01-11T09:45:37.949253+0000","last_peered":"2024-01-11T09:45:37.949253+0000","last_clean":"2024-01-11T09:45:37.949253+0000","last_became_active":"2024-01-11T09:45:23.141580+0000","last_became_peered":"2024-01-11T09:45:23.141580+0000","last_unstale":"2024-01-11T09:45:37.949253+0000","last_undegraded":"2024-01-11T09:45:37.949253+0000","last_fullsized":"2024-01-11T09:45:37.949253+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:19.210181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949363+0000","last_change":"2024-01-11T09:45:24.145805+0000","last_active":"2024-01-11T09:45:37.949363+0000","last_peered":"2024-01-11T09:45:37.949363+0000","last_clean":"2024-01-11T09:45:37.949363+0000","last_became_active":"2024-01-11T09:45:24.145497+0000","last_became_peered":"2024-01-11T09:45:24.145497+0000","last_unstale":"2024-01-11T09:45:37.949363+0000","last_undegraded":"2024-01-11T09:45:37.949363+0000","last_fullsized":"2024-01-11T09:45:37.949363+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:27.774010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"217'1","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.982488+0000","last_change":"2024-01-11T09:45:36.977294+0000","last_active":"2024-01-11T09:45:37.982488+0000","last_peered":"2024-01-11T09:45:37.982488+0000","last_clean":"2024-01-11T09:45:37.982488+0000","last_became_active":"2024-01-11T09:45:36.976307+0000","last_became_peered":"2024-01-11T09:45:36.976307+0000","last_unstale":"2024-01-11T09:45:37.982488+0000","last_undegraded":"2024-01-11T09:45:37.982488+0000","last_fullsized":"2024-01-11T09:45:37.982488+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:08:59.135044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.983796+0000","last_change":"2024-01-11T09:45:37.975856+0000","last_active":"2024-01-11T09:45:37.983796+0000","last_peered":"2024-01-11T09:45:37.983796+0000","last_clean":"2024-01-11T09:45:37.983796+0000","last_became_active":"2024-01-11T09:45:37.975580+0000","last_became_peered":"2024-01-11T09:45:37.975580+0000","last_unstale":"2024-01-11T09:45:37.983796+0000","last_undegraded":"2024-01-11T09:45:37.983796+0000","last_fullsized":"2024-01-11T09:45:37.983796+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:47:32.728151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960674+0000","last_change":"2024-01-11T09:45:23.155273+0000","last_active":"2024-01-11T09:45:37.960674+0000","last_peered":"2024-01-11T09:45:37.960674+0000","last_clean":"2024-01-11T09:45:37.960674+0000","last_became_active":"2024-01-11T09:45:23.155023+0000","last_became_peered":"2024-01-11T09:45:23.155023+0000","last_unstale":"2024-01-11T09:45:37.960674+0000","last_undegraded":"2024-01-11T09:45:37.960674+0000","last_fullsized":"2024-01-11T09:45:37.960674+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:03:59.152882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950637+0000","last_change":"2024-01-11T09:45:24.149619+0000","last_active":"2024-01-11T09:45:37.950637+0000","last_peered":"2024-01-11T09:45:37.950637+0000","last_clean":"2024-01-11T09:45:37.950637+0000","last_became_active":"2024-01-11T09:45:24.148547+0000","last_became_peered":"2024-01-11T09:45:24.148547+0000","last_unstale":"2024-01-11T09:45:37.950637+0000","last_undegraded":"2024-01-11T09:45:37.950637+0000","last_fullsized":"2024-01-11T09:45:37.950637+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:28:18.397764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949238+0000","last_change":"2024-01-11T09:45:36.966307+0000","last_active":"2024-01-11T09:45:37.949238+0000","last_peered":"2024-01-11T09:45:37.949238+0000","last_clean":"2024-01-11T09:45:37.949238+0000","last_became_active":"2024-01-11T09:45:36.966015+0000","last_became_peered":"2024-01-11T09:45:36.966015+0000","last_unstale":"2024-01-11T09:45:37.949238+0000","last_undegraded":"2024-01-11T09:45:37.949238+0000","last_fullsized":"2024-01-11T09:45:37.949238+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:17.903461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984037+0000","last_change":"2024-01-11T09:45:37.974824+0000","last_active":"2024-01-11T09:45:37.984037+0000","last_peered":"2024-01-11T09:45:37.984037+0000","last_clean":"2024-01-11T09:45:37.984037+0000","last_became_active":"2024-01-11T09:45:37.974505+0000","last_became_peered":"2024-01-11T09:45:37.974505+0000","last_unstale":"2024-01-11T09:45:37.984037+0000","last_undegraded":"2024-01-11T09:45:37.984037+0000","last_fullsized":"2024-01-11T09:45:37.984037+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:21:58.660078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.d","version":"213'1","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950970+0000","last_change":"2024-01-11T09:45:23.153536+0000","last_active":"2024-01-11T09:45:37.950970+0000","last_peered":"2024-01-11T09:45:37.950970+0000","last_clean":"2024-01-11T09:45:37.950970+0000","last_became_active":"2024-01-11T09:45:23.152838+0000","last_became_peered":"2024-01-11T09:45:23.152838+0000","last_unstale":"2024-01-11T09:45:37.950970+0000","last_undegraded":"2024-01-11T09:45:37.950970+0000","last_fullsized":"2024-01-11T09:45:37.950970+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:14:59.795260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960771+0000","last_change":"2024-01-11T09:45:24.159728+0000","last_active":"2024-01-11T09:45:37.960771+0000","last_peered":"2024-01-11T09:45:37.960771+0000","last_clean":"2024-01-11T09:45:37.960771+0000","last_became_active":"2024-01-11T09:45:24.159489+0000","last_became_peered":"2024-01-11T09:45:24.159489+0000","last_unstale":"2024-01-11T09:45:37.960771+0000","last_undegraded":"2024-01-11T09:45:37.960771+0000","last_fullsized":"2024-01-11T09:45:37.960771+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:21:50.096457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"217'2","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984991+0000","last_change":"2024-01-11T09:45:36.978438+0000","last_active":"2024-01-11T09:45:37.984991+0000","last_peered":"2024-01-11T09:45:37.984991+0000","last_clean":"2024-01-11T09:45:37.984991+0000","last_became_active":"2024-01-11T09:45:36.977894+0000","last_became_peered":"2024-01-11T09:45:36.977894+0000","last_unstale":"2024-01-11T09:45:37.984991+0000","last_undegraded":"2024-01-11T09:45:37.984991+0000","last_fullsized":"2024-01-11T09:45:37.984991+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:15:51.367934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.977525+0000","last_change":"2024-01-11T09:45:37.966951+0000","last_active":"2024-01-11T09:45:37.977525+0000","last_peered":"2024-01-11T09:45:37.977525+0000","last_clean":"2024-01-11T09:45:37.977525+0000","last_became_active":"2024-01-11T09:45:37.966637+0000","last_became_peered":"2024-01-11T09:45:37.966637+0000","last_unstale":"2024-01-11T09:45:37.977525+0000","last_undegraded":"2024-01-11T09:45:37.977525+0000","last_fullsized":"2024-01-11T09:45:37.977525+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:51:04.639956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"213'2","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950295+0000","last_change":"2024-01-11T09:45:23.147655+0000","last_active":"2024-01-11T09:45:37.950295+0000","last_peered":"2024-01-11T09:45:37.950295+0000","last_clean":"2024-01-11T09:45:37.950295+0000","last_became_active":"2024-01-11T09:45:23.147203+0000","last_became_peered":"2024-01-11T09:45:23.147203+0000","last_unstale":"2024-01-11T09:45:37.950295+0000","last_undegraded":"2024-01-11T09:45:37.950295+0000","last_fullsized":"2024-01-11T09:45:37.950295+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:16:31.626739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950216+0000","last_change":"2024-01-11T09:45:24.140693+0000","last_active":"2024-01-11T09:45:37.950216+0000","last_peered":"2024-01-11T09:45:37.950216+0000","last_clean":"2024-01-11T09:45:37.950216+0000","last_became_active":"2024-01-11T09:45:24.140376+0000","last_became_peered":"2024-01-11T09:45:24.140376+0000","last_unstale":"2024-01-11T09:45:37.950216+0000","last_undegraded":"2024-01-11T09:45:37.950216+0000","last_fullsized":"2024-01-11T09:45:37.950216+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:52:17.873609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.966554+0000","last_change":"2024-01-11T09:45:36.972519+0000","last_active":"2024-01-11T09:45:37.966554+0000","last_peered":"2024-01-11T09:45:37.966554+0000","last_clean":"2024-01-11T09:45:37.966554+0000","last_became_active":"2024-01-11T09:45:36.971278+0000","last_became_peered":"2024-01-11T09:45:36.971278+0000","last_unstale":"2024-01-11T09:45:37.966554+0000","last_undegraded":"2024-01-11T09:45:37.966554+0000","last_fullsized":"2024-01-11T09:45:37.966554+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:29.997317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.978814+0000","last_change":"2024-01-11T09:45:37.962984+0000","last_active":"2024-01-11T09:45:37.978814+0000","last_peered":"2024-01-11T09:45:37.978814+0000","last_clean":"2024-01-11T09:45:37.978814+0000","last_became_active":"2024-01-11T09:45:37.962727+0000","last_became_peered":"2024-01-11T09:45:37.962727+0000","last_unstale":"2024-01-11T09:45:37.978814+0000","last_undegraded":"2024-01-11T09:45:37.978814+0000","last_fullsized":"2024-01-11T09:45:37.978814+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:29:35.726697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.f","version":"213'1","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.966530+0000","last_change":"2024-01-11T09:45:23.152322+0000","last_active":"2024-01-11T09:45:37.966530+0000","last_peered":"2024-01-11T09:45:37.966530+0000","last_clean":"2024-01-11T09:45:37.966530+0000","last_became_active":"2024-01-11T09:45:23.152006+0000","last_became_peered":"2024-01-11T09:45:23.152006+0000","last_unstale":"2024-01-11T09:45:37.966530+0000","last_undegraded":"2024-01-11T09:45:37.966530+0000","last_fullsized":"2024-01-11T09:45:37.966530+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:46.962719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950186+0000","last_change":"2024-01-11T09:45:24.152987+0000","last_active":"2024-01-11T09:45:37.950186+0000","last_peered":"2024-01-11T09:45:37.950186+0000","last_clean":"2024-01-11T09:45:37.950186+0000","last_became_active":"2024-01-11T09:45:24.152677+0000","last_became_peered":"2024-01-11T09:45:24.152677+0000","last_unstale":"2024-01-11T09:45:37.950186+0000","last_undegraded":"2024-01-11T09:45:37.950186+0000","last_fullsized":"2024-01-11T09:45:37.950186+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:03.360851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.947837+0000","last_change":"2024-01-11T09:45:36.972308+0000","last_active":"2024-01-11T09:45:37.947837+0000","last_peered":"2024-01-11T09:45:37.947837+0000","last_clean":"2024-01-11T09:45:37.947837+0000","last_became_active":"2024-01-11T09:45:36.972050+0000","last_became_peered":"2024-01-11T09:45:36.972050+0000","last_unstale":"2024-01-11T09:45:37.947837+0000","last_undegraded":"2024-01-11T09:45:37.947837+0000","last_fullsized":"2024-01-11T09:45:37.947837+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:57:04.436083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.982544+0000","last_change":"2024-01-11T09:45:37.975993+0000","last_active":"2024-01-11T09:45:37.982544+0000","last_peered":"2024-01-11T09:45:37.982544+0000","last_clean":"2024-01-11T09:45:37.982544+0000","last_became_active":"2024-01-11T09:45:37.975734+0000","last_became_peered":"2024-01-11T09:45:37.975734+0000","last_unstale":"2024-01-11T09:45:37.982544+0000","last_undegraded":"2024-01-11T09:45:37.982544+0000","last_fullsized":"2024-01-11T09:45:37.982544+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:52:52.400789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.10","version":"213'8","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.955930+0000","last_change":"2024-01-11T09:45:23.146296+0000","last_active":"2024-01-11T09:45:37.955930+0000","last_peered":"2024-01-11T09:45:37.955930+0000","last_clean":"2024-01-11T09:45:37.955930+0000","last_became_active":"2024-01-11T09:45:23.146026+0000","last_became_peered":"2024-01-11T09:45:23.146026+0000","last_unstale":"2024-01-11T09:45:37.955930+0000","last_undegraded":"2024-01-11T09:45:37.955930+0000","last_fullsized":"2024-01-11T09:45:37.955930+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:38:30.060556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.958796+0000","last_change":"2024-01-11T09:45:24.154643+0000","last_active":"2024-01-11T09:45:37.958796+0000","last_peered":"2024-01-11T09:45:37.958796+0000","last_clean":"2024-01-11T09:45:37.958796+0000","last_became_active":"2024-01-11T09:45:24.154293+0000","last_became_peered":"2024-01-11T09:45:24.154293+0000","last_unstale":"2024-01-11T09:45:37.958796+0000","last_undegraded":"2024-01-11T09:45:37.958796+0000","last_fullsized":"2024-01-11T09:45:37.958796+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:00:36.090834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.961988+0000","last_change":"2024-01-11T09:45:36.971692+0000","last_active":"2024-01-11T09:45:37.961988+0000","last_peered":"2024-01-11T09:45:37.961988+0000","last_clean":"2024-01-11T09:45:37.961988+0000","last_became_active":"2024-01-11T09:45:36.971379+0000","last_became_peered":"2024-01-11T09:45:36.971379+0000","last_unstale":"2024-01-11T09:45:37.961988+0000","last_undegraded":"2024-01-11T09:45:37.961988+0000","last_fullsized":"2024-01-11T09:45:37.961988+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:11:30.110533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.972230+0000","last_change":"2024-01-11T09:45:37.966212+0000","last_active":"2024-01-11T09:45:37.972230+0000","last_peered":"2024-01-11T09:45:37.972230+0000","last_clean":"2024-01-11T09:45:37.972230+0000","last_became_active":"2024-01-11T09:45:37.965957+0000","last_became_peered":"2024-01-11T09:45:37.965957+0000","last_unstale":"2024-01-11T09:45:37.972230+0000","last_undegraded":"2024-01-11T09:45:37.972230+0000","last_fullsized":"2024-01-11T09:45:37.972230+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:59:05.825428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949035+0000","last_change":"2024-01-11T09:45:23.151388+0000","last_active":"2024-01-11T09:45:37.949035+0000","last_peered":"2024-01-11T09:45:37.949035+0000","last_clean":"2024-01-11T09:45:37.949035+0000","last_became_active":"2024-01-11T09:45:23.151127+0000","last_became_peered":"2024-01-11T09:45:23.151127+0000","last_unstale":"2024-01-11T09:45:37.949035+0000","last_undegraded":"2024-01-11T09:45:37.949035+0000","last_fullsized":"2024-01-11T09:45:37.949035+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:43:21.323090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.955745+0000","last_change":"2024-01-11T09:45:24.166383+0000","last_active":"2024-01-11T09:45:37.955745+0000","last_peered":"2024-01-11T09:45:37.955745+0000","last_clean":"2024-01-11T09:45:37.955745+0000","last_became_active":"2024-01-11T09:45:24.159441+0000","last_became_peered":"2024-01-11T09:45:24.159441+0000","last_unstale":"2024-01-11T09:45:37.955745+0000","last_undegraded":"2024-01-11T09:45:37.955745+0000","last_fullsized":"2024-01-11T09:45:37.955745+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:25:32.122001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.955791+0000","last_change":"2024-01-11T09:45:36.976096+0000","last_active":"2024-01-11T09:45:37.955791+0000","last_peered":"2024-01-11T09:45:37.955791+0000","last_clean":"2024-01-11T09:45:37.955791+0000","last_became_active":"2024-01-11T09:45:36.975634+0000","last_became_peered":"2024-01-11T09:45:36.975634+0000","last_unstale":"2024-01-11T09:45:37.955791+0000","last_undegraded":"2024-01-11T09:45:37.955791+0000","last_fullsized":"2024-01-11T09:45:37.955791+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:49:00.158472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.986212+0000","last_change":"2024-01-11T09:45:37.980563+0000","last_active":"2024-01-11T09:45:37.986212+0000","last_peered":"2024-01-11T09:45:37.986212+0000","last_clean":"2024-01-11T09:45:37.986212+0000","last_became_active":"2024-01-11T09:45:37.980290+0000","last_became_peered":"2024-01-11T09:45:37.980290+0000","last_unstale":"2024-01-11T09:45:37.986212+0000","last_undegraded":"2024-01-11T09:45:37.986212+0000","last_fullsized":"2024-01-11T09:45:37.986212+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:59:05.825428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950826+0000","last_change":"2024-01-11T09:45:23.152794+0000","last_active":"2024-01-11T09:45:37.950826+0000","last_peered":"2024-01-11T09:45:37.950826+0000","last_clean":"2024-01-11T09:45:37.950826+0000","last_became_active":"2024-01-11T09:45:23.152495+0000","last_became_peered":"2024-01-11T09:45:23.152495+0000","last_unstale":"2024-01-11T09:45:37.950826+0000","last_undegraded":"2024-01-11T09:45:37.950826+0000","last_fullsized":"2024-01-11T09:45:37.950826+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:01:05.702272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949452+0000","last_change":"2024-01-11T09:45:24.147696+0000","last_active":"2024-01-11T09:45:37.949452+0000","last_peered":"2024-01-11T09:45:37.949452+0000","last_clean":"2024-01-11T09:45:37.949452+0000","last_became_active":"2024-01-11T09:45:24.147422+0000","last_became_peered":"2024-01-11T09:45:24.147422+0000","last_unstale":"2024-01-11T09:45:37.949452+0000","last_undegraded":"2024-01-11T09:45:37.949452+0000","last_fullsized":"2024-01-11T09:45:37.949452+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:32:40.240733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"217'6","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.993710+0000","last_change":"2024-01-11T09:45:36.963386+0000","last_active":"2024-01-11T09:45:37.993710+0000","last_peered":"2024-01-11T09:45:37.993710+0000","last_clean":"2024-01-11T09:45:37.993710+0000","last_became_active":"2024-01-11T09:45:36.962553+0000","last_became_peered":"2024-01-11T09:45:36.962553+0000","last_unstale":"2024-01-11T09:45:37.993710+0000","last_undegraded":"2024-01-11T09:45:37.993710+0000","last_fullsized":"2024-01-11T09:45:37.993710+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:15:51.367934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.972328+0000","last_change":"2024-01-11T09:45:37.965023+0000","last_active":"2024-01-11T09:45:37.972328+0000","last_peered":"2024-01-11T09:45:37.972328+0000","last_clean":"2024-01-11T09:45:37.972328+0000","last_became_active":"2024-01-11T09:45:37.963795+0000","last_became_peered":"2024-01-11T09:45:37.963795+0000","last_unstale":"2024-01-11T09:45:37.972328+0000","last_undegraded":"2024-01-11T09:45:37.972328+0000","last_fullsized":"2024-01-11T09:45:37.972328+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:28:12.821267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.13","version":"213'2","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.951250+0000","last_change":"2024-01-11T09:45:23.155269+0000","last_active":"2024-01-11T09:45:37.951250+0000","last_peered":"2024-01-11T09:45:37.951250+0000","last_clean":"2024-01-11T09:45:37.951250+0000","last_became_active":"2024-01-11T09:45:23.154990+0000","last_became_peered":"2024-01-11T09:45:23.154990+0000","last_unstale":"2024-01-11T09:45:37.951250+0000","last_undegraded":"2024-01-11T09:45:37.951250+0000","last_fullsized":"2024-01-11T09:45:37.951250+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:32:15.216292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.961636+0000","last_change":"2024-01-11T09:45:24.157881+0000","last_active":"2024-01-11T09:45:37.961636+0000","last_peered":"2024-01-11T09:45:37.961636+0000","last_clean":"2024-01-11T09:45:37.961636+0000","last_became_active":"2024-01-11T09:45:24.157224+0000","last_became_peered":"2024-01-11T09:45:24.157224+0000","last_unstale":"2024-01-11T09:45:37.961636+0000","last_undegraded":"2024-01-11T09:45:37.961636+0000","last_fullsized":"2024-01-11T09:45:37.961636+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:32:16.224153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"217'3","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.991193+0000","last_change":"2024-01-11T09:45:36.971737+0000","last_active":"2024-01-11T09:45:37.991193+0000","last_peered":"2024-01-11T09:45:37.991193+0000","last_clean":"2024-01-11T09:45:37.991193+0000","last_became_active":"2024-01-11T09:45:36.969536+0000","last_became_peered":"2024-01-11T09:45:36.969536+0000","last_unstale":"2024-01-11T09:45:37.991193+0000","last_undegraded":"2024-01-11T09:45:37.991193+0000","last_fullsized":"2024-01-11T09:45:37.991193+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:36:41.577470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.969343+0000","last_change":"2024-01-11T09:45:37.966563+0000","last_active":"2024-01-11T09:45:37.969343+0000","last_peered":"2024-01-11T09:45:37.969343+0000","last_clean":"2024-01-11T09:45:37.969343+0000","last_became_active":"2024-01-11T09:45:37.966273+0000","last_became_peered":"2024-01-11T09:45:37.966273+0000","last_unstale":"2024-01-11T09:45:37.969343+0000","last_undegraded":"2024-01-11T09:45:37.969343+0000","last_fullsized":"2024-01-11T09:45:37.969343+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:39:19.277161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.14","version":"214'8","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.953956+0000","last_change":"2024-01-11T09:45:23.152136+0000","last_active":"2024-01-11T09:45:37.953956+0000","last_peered":"2024-01-11T09:45:37.953956+0000","last_clean":"2024-01-11T09:45:37.953956+0000","last_became_active":"2024-01-11T09:45:23.151819+0000","last_became_peered":"2024-01-11T09:45:23.151819+0000","last_unstale":"2024-01-11T09:45:37.953956+0000","last_undegraded":"2024-01-11T09:45:37.953956+0000","last_fullsized":"2024-01-11T09:45:37.953956+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:41:09.070880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2749,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960221+0000","last_change":"2024-01-11T09:45:24.169731+0000","last_active":"2024-01-11T09:45:37.960221+0000","last_peered":"2024-01-11T09:45:37.960221+0000","last_clean":"2024-01-11T09:45:37.960221+0000","last_became_active":"2024-01-11T09:45:24.169381+0000","last_became_peered":"2024-01-11T09:45:24.169381+0000","last_unstale":"2024-01-11T09:45:37.960221+0000","last_undegraded":"2024-01-11T09:45:37.960221+0000","last_fullsized":"2024-01-11T09:45:37.960221+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:04:41.533247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.956354+0000","last_change":"2024-01-11T09:45:36.964088+0000","last_active":"2024-01-11T09:45:37.956354+0000","last_peered":"2024-01-11T09:45:37.956354+0000","last_clean":"2024-01-11T09:45:37.956354+0000","last_became_active":"2024-01-11T09:45:36.963596+0000","last_became_peered":"2024-01-11T09:45:36.963596+0000","last_unstale":"2024-01-11T09:45:37.956354+0000","last_undegraded":"2024-01-11T09:45:37.956354+0000","last_fullsized":"2024-01-11T09:45:37.956354+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:28:10.306993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984996+0000","last_change":"2024-01-11T09:45:37.977762+0000","last_active":"2024-01-11T09:45:37.984996+0000","last_peered":"2024-01-11T09:45:37.984996+0000","last_clean":"2024-01-11T09:45:37.984996+0000","last_became_active":"2024-01-11T09:45:37.977432+0000","last_became_peered":"2024-01-11T09:45:37.977432+0000","last_unstale":"2024-01-11T09:45:37.984996+0000","last_undegraded":"2024-01-11T09:45:37.984996+0000","last_fullsized":"2024-01-11T09:45:37.984996+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:28:11.313558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.15","version":"213'3","reported_seq":17,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950781+0000","last_change":"2024-01-11T09:45:23.154355+0000","last_active":"2024-01-11T09:45:37.950781+0000","last_peered":"2024-01-11T09:45:37.950781+0000","last_clean":"2024-01-11T09:45:37.950781+0000","last_became_active":"2024-01-11T09:45:23.153262+0000","last_became_peered":"2024-01-11T09:45:23.153262+0000","last_unstale":"2024-01-11T09:45:37.950781+0000","last_undegraded":"2024-01-11T09:45:37.950781+0000","last_fullsized":"2024-01-11T09:45:37.950781+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:32:15.216292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954159+0000","last_change":"2024-01-11T09:45:24.150019+0000","last_active":"2024-01-11T09:45:37.954159+0000","last_peered":"2024-01-11T09:45:37.954159+0000","last_clean":"2024-01-11T09:45:37.954159+0000","last_became_active":"2024-01-11T09:45:24.148994+0000","last_became_peered":"2024-01-11T09:45:24.148994+0000","last_unstale":"2024-01-11T09:45:37.954159+0000","last_undegraded":"2024-01-11T09:45:37.954159+0000","last_fullsized":"2024-01-11T09:45:37.954159+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:29:01.556722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"217'2","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.986712+0000","last_change":"2024-01-11T09:45:36.972442+0000","last_active":"2024-01-11T09:45:37.986712+0000","last_peered":"2024-01-11T09:45:37.986712+0000","last_clean":"2024-01-11T09:45:37.986712+0000","last_became_active":"2024-01-11T09:45:36.972165+0000","last_became_peered":"2024-01-11T09:45:36.972165+0000","last_unstale":"2024-01-11T09:45:37.986712+0000","last_undegraded":"2024-01-11T09:45:37.986712+0000","last_fullsized":"2024-01-11T09:45:37.986712+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:28:31.210512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.983856+0000","last_change":"2024-01-11T09:45:37.979778+0000","last_active":"2024-01-11T09:45:37.983856+0000","last_peered":"2024-01-11T09:45:37.983856+0000","last_clean":"2024-01-11T09:45:37.983856+0000","last_became_active":"2024-01-11T09:45:37.977950+0000","last_became_peered":"2024-01-11T09:45:37.977950+0000","last_unstale":"2024-01-11T09:45:37.983856+0000","last_undegraded":"2024-01-11T09:45:37.983856+0000","last_fullsized":"2024-01-11T09:45:37.983856+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:07:41.051644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950589+0000","last_change":"2024-01-11T09:45:23.150869+0000","last_active":"2024-01-11T09:45:37.950589+0000","last_peered":"2024-01-11T09:45:37.950589+0000","last_clean":"2024-01-11T09:45:37.950589+0000","last_became_active":"2024-01-11T09:45:23.150604+0000","last_became_peered":"2024-01-11T09:45:23.150604+0000","last_unstale":"2024-01-11T09:45:37.950589+0000","last_undegraded":"2024-01-11T09:45:37.950589+0000","last_fullsized":"2024-01-11T09:45:37.950589+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:19.210181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960616+0000","last_change":"2024-01-11T09:45:24.166531+0000","last_active":"2024-01-11T09:45:37.960616+0000","last_peered":"2024-01-11T09:45:37.960616+0000","last_clean":"2024-01-11T09:45:37.960616+0000","last_became_active":"2024-01-11T09:45:24.164071+0000","last_became_peered":"2024-01-11T09:45:24.164071+0000","last_unstale":"2024-01-11T09:45:37.960616+0000","last_undegraded":"2024-01-11T09:45:37.960616+0000","last_fullsized":"2024-01-11T09:45:37.960616+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:41:10.078741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"217'8","reported_seq":18,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.991634+0000","last_change":"2024-01-11T09:45:36.966889+0000","last_active":"2024-01-11T09:45:37.991634+0000","last_peered":"2024-01-11T09:45:37.991634+0000","last_clean":"2024-01-11T09:45:37.991634+0000","last_became_active":"2024-01-11T09:45:36.965424+0000","last_became_peered":"2024-01-11T09:45:36.965424+0000","last_unstale":"2024-01-11T09:45:37.991634+0000","last_undegraded":"2024-01-11T09:45:37.991634+0000","last_fullsized":"2024-01-11T09:45:37.991634+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:50:36.781518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.957865+0000","last_change":"2024-01-11T09:45:37.952454+0000","last_active":"2024-01-11T09:45:37.957865+0000","last_peered":"2024-01-11T09:45:37.957865+0000","last_clean":"2024-01-11T09:45:37.957865+0000","last_became_active":"2024-01-11T09:45:37.952102+0000","last_became_peered":"2024-01-11T09:45:37.952102+0000","last_unstale":"2024-01-11T09:45:37.957865+0000","last_undegraded":"2024-01-11T09:45:37.957865+0000","last_fullsized":"2024-01-11T09:45:37.957865+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:52:26.173608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.968052+0000","last_change":"2024-01-11T09:45:23.152897+0000","last_active":"2024-01-11T09:45:37.968052+0000","last_peered":"2024-01-11T09:45:37.968052+0000","last_clean":"2024-01-11T09:45:37.968052+0000","last_became_active":"2024-01-11T09:45:23.152620+0000","last_became_peered":"2024-01-11T09:45:23.152620+0000","last_unstale":"2024-01-11T09:45:37.968052+0000","last_undegraded":"2024-01-11T09:45:37.968052+0000","last_fullsized":"2024-01-11T09:45:37.968052+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:02:41.619972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954942+0000","last_change":"2024-01-11T09:45:24.155035+0000","last_active":"2024-01-11T09:45:37.954942+0000","last_peered":"2024-01-11T09:45:37.954942+0000","last_clean":"2024-01-11T09:45:37.954942+0000","last_became_active":"2024-01-11T09:45:24.153980+0000","last_became_peered":"2024-01-11T09:45:24.153980+0000","last_unstale":"2024-01-11T09:45:37.954942+0000","last_undegraded":"2024-01-11T09:45:37.954942+0000","last_fullsized":"2024-01-11T09:45:37.954942+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:52:20.741771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954953+0000","last_change":"2024-01-11T09:45:36.967350+0000","last_active":"2024-01-11T09:45:37.954953+0000","last_peered":"2024-01-11T09:45:37.954953+0000","last_clean":"2024-01-11T09:45:37.954953+0000","last_became_active":"2024-01-11T09:45:36.964560+0000","last_became_peered":"2024-01-11T09:45:36.964560+0000","last_unstale":"2024-01-11T09:45:37.954953+0000","last_undegraded":"2024-01-11T09:45:37.954953+0000","last_fullsized":"2024-01-11T09:45:37.954953+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:44:19.194643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.988960+0000","last_change":"2024-01-11T09:45:37.986871+0000","last_active":"2024-01-11T09:45:37.988960+0000","last_peered":"2024-01-11T09:45:37.988960+0000","last_clean":"2024-01-11T09:45:37.988960+0000","last_became_active":"2024-01-11T09:45:37.986318+0000","last_became_peered":"2024-01-11T09:45:37.986318+0000","last_unstale":"2024-01-11T09:45:37.988960+0000","last_undegraded":"2024-01-11T09:45:37.988960+0000","last_fullsized":"2024-01-11T09:45:37.988960+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:27:26.360142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949746+0000","last_change":"2024-01-11T09:45:23.152843+0000","last_active":"2024-01-11T09:45:37.949746+0000","last_peered":"2024-01-11T09:45:37.949746+0000","last_clean":"2024-01-11T09:45:37.949746+0000","last_became_active":"2024-01-11T09:45:23.152579+0000","last_became_peered":"2024-01-11T09:45:23.152579+0000","last_unstale":"2024-01-11T09:45:37.949746+0000","last_undegraded":"2024-01-11T09:45:37.949746+0000","last_fullsized":"2024-01-11T09:45:37.949746+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:48:05.979085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954041+0000","last_change":"2024-01-11T09:45:24.150754+0000","last_active":"2024-01-11T09:45:37.954041+0000","last_peered":"2024-01-11T09:45:37.954041+0000","last_clean":"2024-01-11T09:45:37.954041+0000","last_became_active":"2024-01-11T09:45:24.150485+0000","last_became_peered":"2024-01-11T09:45:24.150485+0000","last_unstale":"2024-01-11T09:45:37.954041+0000","last_undegraded":"2024-01-11T09:45:37.954041+0000","last_fullsized":"2024-01-11T09:45:37.954041+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:32:19.368822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"217'1","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.987045+0000","last_change":"2024-01-11T09:45:36.974169+0000","last_active":"2024-01-11T09:45:37.987045+0000","last_peered":"2024-01-11T09:45:37.987045+0000","last_clean":"2024-01-11T09:45:37.987045+0000","last_became_active":"2024-01-11T09:45:36.973911+0000","last_became_peered":"2024-01-11T09:45:36.973911+0000","last_unstale":"2024-01-11T09:45:37.987045+0000","last_undegraded":"2024-01-11T09:45:37.987045+0000","last_fullsized":"2024-01-11T09:45:37.987045+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:46:15.891808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.981819+0000","last_change":"2024-01-11T09:45:37.975235+0000","last_active":"2024-01-11T09:45:37.981819+0000","last_peered":"2024-01-11T09:45:37.981819+0000","last_clean":"2024-01-11T09:45:37.981819+0000","last_became_active":"2024-01-11T09:45:37.972196+0000","last_became_peered":"2024-01-11T09:45:37.972196+0000","last_unstale":"2024-01-11T09:45:37.981819+0000","last_undegraded":"2024-01-11T09:45:37.981819+0000","last_fullsized":"2024-01-11T09:45:37.981819+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:36:42.584034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.955181+0000","last_change":"2024-01-11T09:45:23.145331+0000","last_active":"2024-01-11T09:45:37.955181+0000","last_peered":"2024-01-11T09:45:37.955181+0000","last_clean":"2024-01-11T09:45:37.955181+0000","last_became_active":"2024-01-11T09:45:23.145064+0000","last_became_peered":"2024-01-11T09:45:23.145064+0000","last_unstale":"2024-01-11T09:45:37.955181+0000","last_undegraded":"2024-01-11T09:45:37.955181+0000","last_fullsized":"2024-01-11T09:45:37.955181+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:56:41.726387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.961053+0000","last_change":"2024-01-11T09:45:24.150698+0000","last_active":"2024-01-11T09:45:37.961053+0000","last_peered":"2024-01-11T09:45:37.961053+0000","last_clean":"2024-01-11T09:45:37.961053+0000","last_became_active":"2024-01-11T09:45:24.149431+0000","last_became_peered":"2024-01-11T09:45:24.149431+0000","last_unstale":"2024-01-11T09:45:37.961053+0000","last_undegraded":"2024-01-11T09:45:37.961053+0000","last_fullsized":"2024-01-11T09:45:37.961053+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:48:06.986946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"217'4","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.990945+0000","last_change":"2024-01-11T09:45:36.972087+0000","last_active":"2024-01-11T09:45:37.990945+0000","last_peered":"2024-01-11T09:45:37.990945+0000","last_clean":"2024-01-11T09:45:37.990945+0000","last_became_active":"2024-01-11T09:45:36.970149+0000","last_became_peered":"2024-01-11T09:45:36.970149+0000","last_unstale":"2024-01-11T09:45:37.990945+0000","last_undegraded":"2024-01-11T09:45:37.990945+0000","last_fullsized":"2024-01-11T09:45:37.990945+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:39:48.476885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984778+0000","last_change":"2024-01-11T09:45:37.976386+0000","last_active":"2024-01-11T09:45:37.984778+0000","last_peered":"2024-01-11T09:45:37.984778+0000","last_clean":"2024-01-11T09:45:37.984778+0000","last_became_active":"2024-01-11T09:45:37.975081+0000","last_became_peered":"2024-01-11T09:45:37.975081+0000","last_unstale":"2024-01-11T09:45:37.984778+0000","last_undegraded":"2024-01-11T09:45:37.984778+0000","last_fullsized":"2024-01-11T09:45:37.984778+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:11:31.117098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1a","version":"213'1","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.961545+0000","last_change":"2024-01-11T09:45:23.154575+0000","last_active":"2024-01-11T09:45:37.961545+0000","last_peered":"2024-01-11T09:45:37.961545+0000","last_clean":"2024-01-11T09:45:37.961545+0000","last_became_active":"2024-01-11T09:45:23.154313+0000","last_became_peered":"2024-01-11T09:45:23.154313+0000","last_unstale":"2024-01-11T09:45:37.961545+0000","last_undegraded":"2024-01-11T09:45:37.961545+0000","last_fullsized":"2024-01-11T09:45:37.961545+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:14:08.645320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954142+0000","last_change":"2024-01-11T09:45:24.148848+0000","last_active":"2024-01-11T09:45:37.954142+0000","last_peered":"2024-01-11T09:45:37.954142+0000","last_clean":"2024-01-11T09:45:37.954142+0000","last_became_active":"2024-01-11T09:45:24.148573+0000","last_became_peered":"2024-01-11T09:45:24.148573+0000","last_unstale":"2024-01-11T09:45:37.954142+0000","last_undegraded":"2024-01-11T09:45:37.954142+0000","last_fullsized":"2024-01-11T09:45:37.954142+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:29:30.529875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"217'2","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984837+0000","last_change":"2024-01-11T09:45:36.965686+0000","last_active":"2024-01-11T09:45:37.984837+0000","last_peered":"2024-01-11T09:45:37.984837+0000","last_clean":"2024-01-11T09:45:37.984837+0000","last_became_active":"2024-01-11T09:45:36.964782+0000","last_became_peered":"2024-01-11T09:45:36.964782+0000","last_unstale":"2024-01-11T09:45:37.984837+0000","last_undegraded":"2024-01-11T09:45:37.984837+0000","last_fullsized":"2024-01-11T09:45:37.984837+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:18:37.841014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.983269+0000","last_change":"2024-01-11T09:45:37.978823+0000","last_active":"2024-01-11T09:45:37.983269+0000","last_peered":"2024-01-11T09:45:37.983269+0000","last_clean":"2024-01-11T09:45:37.983269+0000","last_became_active":"2024-01-11T09:45:37.978561+0000","last_became_peered":"2024-01-11T09:45:37.978561+0000","last_unstale":"2024-01-11T09:45:37.983269+0000","last_undegraded":"2024-01-11T09:45:37.983269+0000","last_fullsized":"2024-01-11T09:45:37.983269+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:06:59.351961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1c","version":"213'2","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954064+0000","last_change":"2024-01-11T09:45:23.150649+0000","last_active":"2024-01-11T09:45:37.954064+0000","last_peered":"2024-01-11T09:45:37.954064+0000","last_clean":"2024-01-11T09:45:37.954064+0000","last_became_active":"2024-01-11T09:45:23.150425+0000","last_became_peered":"2024-01-11T09:45:23.150425+0000","last_unstale":"2024-01-11T09:45:37.954064+0000","last_undegraded":"2024-01-11T09:45:37.954064+0000","last_fullsized":"2024-01-11T09:45:37.954064+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:14:08.645320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950548+0000","last_change":"2024-01-11T09:45:24.164335+0000","last_active":"2024-01-11T09:45:37.950548+0000","last_peered":"2024-01-11T09:45:37.950548+0000","last_clean":"2024-01-11T09:45:37.950548+0000","last_became_active":"2024-01-11T09:45:24.164063+0000","last_became_peered":"2024-01-11T09:45:24.164063+0000","last_unstale":"2024-01-11T09:45:37.950548+0000","last_undegraded":"2024-01-11T09:45:37.950548+0000","last_fullsized":"2024-01-11T09:45:37.950548+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:42:21.543118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"217'1","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.986047+0000","last_change":"2024-01-11T09:45:36.973376+0000","last_active":"2024-01-11T09:45:37.986047+0000","last_peered":"2024-01-11T09:45:37.986047+0000","last_clean":"2024-01-11T09:45:37.986047+0000","last_became_active":"2024-01-11T09:45:36.973100+0000","last_became_peered":"2024-01-11T09:45:36.973100+0000","last_unstale":"2024-01-11T09:45:37.986047+0000","last_undegraded":"2024-01-11T09:45:37.986047+0000","last_fullsized":"2024-01-11T09:45:37.986047+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:08:57.106726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.983126+0000","last_change":"2024-01-11T09:45:37.976372+0000","last_active":"2024-01-11T09:45:37.983126+0000","last_peered":"2024-01-11T09:45:37.983126+0000","last_clean":"2024-01-11T09:45:37.983126+0000","last_became_active":"2024-01-11T09:45:37.976103+0000","last_became_peered":"2024-01-11T09:45:37.976103+0000","last_unstale":"2024-01-11T09:45:37.983126+0000","last_undegraded":"2024-01-11T09:45:37.983126+0000","last_fullsized":"2024-01-11T09:45:37.983126+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:31.003882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1d","version":"213'2","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960565+0000","last_change":"2024-01-11T09:45:23.144976+0000","last_active":"2024-01-11T09:45:37.960565+0000","last_peered":"2024-01-11T09:45:37.960565+0000","last_clean":"2024-01-11T09:45:37.960565+0000","last_became_active":"2024-01-11T09:45:23.144660+0000","last_became_peered":"2024-01-11T09:45:23.144660+0000","last_unstale":"2024-01-11T09:45:37.960565+0000","last_undegraded":"2024-01-11T09:45:37.960565+0000","last_fullsized":"2024-01-11T09:45:37.960565+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:59:40.579473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.957059+0000","last_change":"2024-01-11T09:45:24.165254+0000","last_active":"2024-01-11T09:45:37.957059+0000","last_peered":"2024-01-11T09:45:37.957059+0000","last_clean":"2024-01-11T09:45:37.957059+0000","last_became_active":"2024-01-11T09:45:24.163213+0000","last_became_peered":"2024-01-11T09:45:24.163213+0000","last_unstale":"2024-01-11T09:45:37.957059+0000","last_undegraded":"2024-01-11T09:45:37.957059+0000","last_fullsized":"2024-01-11T09:45:37.957059+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:29:21.907384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"217'1","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.986835+0000","last_change":"2024-01-11T09:45:36.967799+0000","last_active":"2024-01-11T09:45:37.986835+0000","last_peered":"2024-01-11T09:45:37.986835+0000","last_clean":"2024-01-11T09:45:37.986835+0000","last_became_active":"2024-01-11T09:45:36.967519+0000","last_became_peered":"2024-01-11T09:45:36.967519+0000","last_unstale":"2024-01-11T09:45:37.986835+0000","last_undegraded":"2024-01-11T09:45:37.986835+0000","last_fullsized":"2024-01-11T09:45:37.986835+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:17.903461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.972388+0000","last_change":"2024-01-11T09:45:37.967983+0000","last_active":"2024-01-11T09:45:37.972388+0000","last_peered":"2024-01-11T09:45:37.972388+0000","last_clean":"2024-01-11T09:45:37.972388+0000","last_became_active":"2024-01-11T09:45:37.967172+0000","last_became_peered":"2024-01-11T09:45:37.967172+0000","last_unstale":"2024-01-11T09:45:37.972388+0000","last_undegraded":"2024-01-11T09:45:37.972388+0000","last_fullsized":"2024-01-11T09:45:37.972388+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:31.003882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1e","version":"213'1","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949928+0000","last_change":"2024-01-11T09:45:23.148530+0000","last_active":"2024-01-11T09:45:37.949928+0000","last_peered":"2024-01-11T09:45:37.949928+0000","last_clean":"2024-01-11T09:45:37.949928+0000","last_became_active":"2024-01-11T09:45:23.148208+0000","last_became_peered":"2024-01-11T09:45:23.148208+0000","last_unstale":"2024-01-11T09:45:37.949928+0000","last_undegraded":"2024-01-11T09:45:37.949928+0000","last_fullsized":"2024-01-11T09:45:37.949928+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:44:55.075364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.951770+0000","last_change":"2024-01-11T09:45:24.150915+0000","last_active":"2024-01-11T09:45:37.951770+0000","last_peered":"2024-01-11T09:45:37.951770+0000","last_clean":"2024-01-11T09:45:37.951770+0000","last_became_active":"2024-01-11T09:45:24.149015+0000","last_became_peered":"2024-01-11T09:45:24.149015+0000","last_unstale":"2024-01-11T09:45:37.951770+0000","last_undegraded":"2024-01-11T09:45:37.951770+0000","last_fullsized":"2024-01-11T09:45:37.951770+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:14:14.599129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949362+0000","last_change":"2024-01-11T09:45:36.959784+0000","last_active":"2024-01-11T09:45:37.949362+0000","last_peered":"2024-01-11T09:45:37.949362+0000","last_clean":"2024-01-11T09:45:37.949362+0000","last_became_active":"2024-01-11T09:45:36.958855+0000","last_became_peered":"2024-01-11T09:45:36.958855+0000","last_unstale":"2024-01-11T09:45:37.949362+0000","last_undegraded":"2024-01-11T09:45:37.949362+0000","last_fullsized":"2024-01-11T09:45:37.949362+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:57:59.056615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.986150+0000","last_change":"2024-01-11T09:45:37.982374+0000","last_active":"2024-01-11T09:45:37.986150+0000","last_peered":"2024-01-11T09:45:37.986150+0000","last_clean":"2024-01-11T09:45:37.986150+0000","last_became_active":"2024-01-11T09:45:37.982081+0000","last_became_peered":"2024-01-11T09:45:37.982081+0000","last_unstale":"2024-01-11T09:45:37.986150+0000","last_undegraded":"2024-01-11T09:45:37.986150+0000","last_fullsized":"2024-01-11T09:45:37.986150+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:56.535829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1f","version":"213'4","reported_seq":18,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.967298+0000","last_change":"2024-01-11T09:45:23.150803+0000","last_active":"2024-01-11T09:45:37.967298+0000","last_peered":"2024-01-11T09:45:37.967298+0000","last_clean":"2024-01-11T09:45:37.967298+0000","last_became_active":"2024-01-11T09:45:23.149761+0000","last_became_peered":"2024-01-11T09:45:23.149761+0000","last_unstale":"2024-01-11T09:45:37.967298+0000","last_undegraded":"2024-01-11T09:45:37.967298+0000","last_fullsized":"2024-01-11T09:45:37.967298+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:33:40.495800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949294+0000","last_change":"2024-01-11T09:45:24.167467+0000","last_active":"2024-01-11T09:45:37.949294+0000","last_peered":"2024-01-11T09:45:37.949294+0000","last_clean":"2024-01-11T09:45:37.949294+0000","last_became_active":"2024-01-11T09:45:24.167175+0000","last_became_peered":"2024-01-11T09:45:24.167175+0000","last_unstale":"2024-01-11T09:45:37.949294+0000","last_undegraded":"2024-01-11T09:45:37.949294+0000","last_fullsized":"2024-01-11T09:45:37.949294+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:52:36.044653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.948006+0000","last_change":"2024-01-11T09:45:36.964388+0000","last_active":"2024-01-11T09:45:37.948006+0000","last_peered":"2024-01-11T09:45:37.948006+0000","last_clean":"2024-01-11T09:45:37.948006+0000","last_became_active":"2024-01-11T09:45:36.964114+0000","last_became_peered":"2024-01-11T09:45:36.964114+0000","last_unstale":"2024-01-11T09:45:37.948006+0000","last_undegraded":"2024-01-11T09:45:37.948006+0000","last_fullsized":"2024-01-11T09:45:37.948006+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:28:31.210512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.973257+0000","last_change":"2024-01-11T09:45:37.969941+0000","last_active":"2024-01-11T09:45:37.973257+0000","last_peered":"2024-01-11T09:45:37.973257+0000","last_clean":"2024-01-11T09:45:37.973257+0000","last_became_active":"2024-01-11T09:45:37.969684+0000","last_became_peered":"2024-01-11T09:45:37.969684+0000","last_unstale":"2024-01-11T09:45:37.973257+0000","last_undegraded":"2024-01-11T09:45:37.973257+0000","last_fullsized":"2024-01-11T09:45:37.973257+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:57:46.266155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984003+0000","last_change":"2024-01-11T09:45:37.977272+0000","last_active":"2024-01-11T09:45:37.984003+0000","last_peered":"2024-01-11T09:45:37.984003+0000","last_clean":"2024-01-11T09:45:37.984003+0000","last_became_active":"2024-01-11T09:45:37.975292+0000","last_became_peered":"2024-01-11T09:45:37.975292+0000","last_unstale":"2024-01-11T09:45:37.984003+0000","last_undegraded":"2024-01-11T09:45:37.984003+0000","last_fullsized":"2024-01-11T09:45:37.984003+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:47:04.067575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.957660+0000","last_change":"2024-01-11T09:45:24.167922+0000","last_active":"2024-01-11T09:45:37.957660+0000","last_peered":"2024-01-11T09:45:37.957660+0000","last_clean":"2024-01-11T09:45:37.957660+0000","last_became_active":"2024-01-11T09:45:24.167641+0000","last_became_peered":"2024-01-11T09:45:24.167641+0000","last_unstale":"2024-01-11T09:45:37.957660+0000","last_undegraded":"2024-01-11T09:45:37.957660+0000","last_fullsized":"2024-01-11T09:45:37.957660+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:17.184569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.973931+0000","last_change":"2024-01-11T09:45:37.970672+0000","last_active":"2024-01-11T09:45:37.973931+0000","last_peered":"2024-01-11T09:45:37.973931+0000","last_clean":"2024-01-11T09:45:37.973931+0000","last_became_active":"2024-01-11T09:45:37.970368+0000","last_became_peered":"2024-01-11T09:45:37.970368+0000","last_unstale":"2024-01-11T09:45:37.973931+0000","last_undegraded":"2024-01-11T09:45:37.973931+0000","last_fullsized":"2024-01-11T09:45:37.973931+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:02:35.657312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.948942+0000","last_change":"2024-01-11T09:45:24.155055+0000","last_active":"2024-01-11T09:45:37.948942+0000","last_peered":"2024-01-11T09:45:37.948942+0000","last_clean":"2024-01-11T09:45:37.948942+0000","last_became_active":"2024-01-11T09:45:24.154781+0000","last_became_peered":"2024-01-11T09:45:24.154781+0000","last_unstale":"2024-01-11T09:45:37.948942+0000","last_undegraded":"2024-01-11T09:45:37.948942+0000","last_fullsized":"2024-01-11T09:45:37.948942+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:38:28.859392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.947904+0000","last_change":"2024-01-11T09:45:24.146908+0000","last_active":"2024-01-11T09:45:37.947904+0000","last_peered":"2024-01-11T09:45:37.947904+0000","last_clean":"2024-01-11T09:45:37.947904+0000","last_became_active":"2024-01-11T09:45:24.146543+0000","last_became_peered":"2024-01-11T09:45:24.146543+0000","last_unstale":"2024-01-11T09:45:37.947904+0000","last_undegraded":"2024-01-11T09:45:37.947904+0000","last_fullsized":"2024-01-11T09:45:37.947904+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:40.983480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.975579+0000","last_change":"2024-01-11T09:45:37.971778+0000","last_active":"2024-01-11T09:45:37.975579+0000","last_peered":"2024-01-11T09:45:37.975579+0000","last_clean":"2024-01-11T09:45:37.975579+0000","last_became_active":"2024-01-11T09:45:37.971369+0000","last_became_peered":"2024-01-11T09:45:37.971369+0000","last_unstale":"2024-01-11T09:45:37.975579+0000","last_undegraded":"2024-01-11T09:45:37.975579+0000","last_fullsized":"2024-01-11T09:45:37.975579+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:57:26.930467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.988545+0000","last_change":"2024-01-11T09:45:37.984875+0000","last_active":"2024-01-11T09:45:37.988545+0000","last_peered":"2024-01-11T09:45:37.988545+0000","last_clean":"2024-01-11T09:45:37.988545+0000","last_became_active":"2024-01-11T09:45:37.984603+0000","last_became_peered":"2024-01-11T09:45:37.984603+0000","last_unstale":"2024-01-11T09:45:37.988545+0000","last_undegraded":"2024-01-11T09:45:37.988545+0000","last_fullsized":"2024-01-11T09:45:37.988545+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:32:49.312144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.967593+0000","last_change":"2024-01-11T09:45:24.148965+0000","last_active":"2024-01-11T09:45:37.967593+0000","last_peered":"2024-01-11T09:45:37.967593+0000","last_clean":"2024-01-11T09:45:37.967593+0000","last_became_active":"2024-01-11T09:45:24.148481+0000","last_became_peered":"2024-01-11T09:45:24.148481+0000","last_unstale":"2024-01-11T09:45:37.967593+0000","last_undegraded":"2024-01-11T09:45:37.967593+0000","last_fullsized":"2024-01-11T09:45:37.967593+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:39.406018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960501+0000","last_change":"2024-01-11T09:45:24.166152+0000","last_active":"2024-01-11T09:45:37.960501+0000","last_peered":"2024-01-11T09:45:37.960501+0000","last_clean":"2024-01-11T09:45:37.960501+0000","last_became_active":"2024-01-11T09:45:24.165889+0000","last_became_peered":"2024-01-11T09:45:24.165889+0000","last_unstale":"2024-01-11T09:45:37.960501+0000","last_undegraded":"2024-01-11T09:45:37.960501+0000","last_fullsized":"2024-01-11T09:45:37.960501+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:01:06.710133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.972409+0000","last_change":"2024-01-11T09:45:37.968430+0000","last_active":"2024-01-11T09:45:37.972409+0000","last_peered":"2024-01-11T09:45:37.972409+0000","last_clean":"2024-01-11T09:45:37.972409+0000","last_became_active":"2024-01-11T09:45:37.967571+0000","last_became_peered":"2024-01-11T09:45:37.967571+0000","last_unstale":"2024-01-11T09:45:37.972409+0000","last_undegraded":"2024-01-11T09:45:37.972409+0000","last_fullsized":"2024-01-11T09:45:37.972409+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:57:26.930467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.967232+0000","last_change":"2024-01-11T09:45:24.150865+0000","last_active":"2024-01-11T09:45:37.967232+0000","last_peered":"2024-01-11T09:45:37.967232+0000","last_clean":"2024-01-11T09:45:37.967232+0000","last_became_active":"2024-01-11T09:45:24.148370+0000","last_became_peered":"2024-01-11T09:45:24.148370+0000","last_unstale":"2024-01-11T09:45:37.967232+0000","last_undegraded":"2024-01-11T09:45:37.967232+0000","last_fullsized":"2024-01-11T09:45:37.967232+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:27.774010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.980650+0000","last_change":"2024-01-11T09:45:37.976027+0000","last_active":"2024-01-11T09:45:37.980650+0000","last_peered":"2024-01-11T09:45:37.980650+0000","last_clean":"2024-01-11T09:45:37.980650+0000","last_became_active":"2024-01-11T09:45:37.975753+0000","last_became_peered":"2024-01-11T09:45:37.975753+0000","last_unstale":"2024-01-11T09:45:37.980650+0000","last_undegraded":"2024-01-11T09:45:37.980650+0000","last_fullsized":"2024-01-11T09:45:37.980650+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:18:38.847578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.962139+0000","last_change":"2024-01-11T09:45:24.158803+0000","last_active":"2024-01-11T09:45:37.962139+0000","last_peered":"2024-01-11T09:45:37.962139+0000","last_clean":"2024-01-11T09:45:37.962139+0000","last_became_active":"2024-01-11T09:45:24.158533+0000","last_became_peered":"2024-01-11T09:45:24.158533+0000","last_unstale":"2024-01-11T09:45:37.962139+0000","last_undegraded":"2024-01-11T09:45:37.962139+0000","last_fullsized":"2024-01-11T09:45:37.962139+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:57:46.243868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.970344+0000","last_change":"2024-01-11T09:45:37.967437+0000","last_active":"2024-01-11T09:45:37.970344+0000","last_peered":"2024-01-11T09:45:37.970344+0000","last_clean":"2024-01-11T09:45:37.970344+0000","last_became_active":"2024-01-11T09:45:37.967161+0000","last_became_peered":"2024-01-11T09:45:37.967161+0000","last_unstale":"2024-01-11T09:45:37.970344+0000","last_undegraded":"2024-01-11T09:45:37.970344+0000","last_fullsized":"2024-01-11T09:45:37.970344+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:54:27.964289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950338+0000","last_change":"2024-01-11T09:45:24.145841+0000","last_active":"2024-01-11T09:45:37.950338+0000","last_peered":"2024-01-11T09:45:37.950338+0000","last_clean":"2024-01-11T09:45:37.950338+0000","last_became_active":"2024-01-11T09:45:24.145584+0000","last_became_peered":"2024-01-11T09:45:24.145584+0000","last_unstale":"2024-01-11T09:45:37.950338+0000","last_undegraded":"2024-01-11T09:45:37.950338+0000","last_fullsized":"2024-01-11T09:45:37.950338+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:47.970580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.979066+0000","last_change":"2024-01-11T09:45:37.967560+0000","last_active":"2024-01-11T09:45:37.979066+0000","last_peered":"2024-01-11T09:45:37.979066+0000","last_clean":"2024-01-11T09:45:37.979066+0000","last_became_active":"2024-01-11T09:45:37.967314+0000","last_became_peered":"2024-01-11T09:45:37.967314+0000","last_unstale":"2024-01-11T09:45:37.979066+0000","last_undegraded":"2024-01-11T09:45:37.979066+0000","last_fullsized":"2024-01-11T09:45:37.979066+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:58:00.063180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.980237+0000","last_change":"2024-01-11T09:45:37.974580+0000","last_active":"2024-01-11T09:45:37.980237+0000","last_peered":"2024-01-11T09:45:37.980237+0000","last_clean":"2024-01-11T09:45:37.980237+0000","last_became_active":"2024-01-11T09:45:37.974273+0000","last_became_peered":"2024-01-11T09:45:37.974273+0000","last_unstale":"2024-01-11T09:45:37.980237+0000","last_undegraded":"2024-01-11T09:45:37.980237+0000","last_fullsized":"2024-01-11T09:45:37.980237+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:28:23.210266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954058+0000","last_change":"2024-01-11T09:45:24.168887+0000","last_active":"2024-01-11T09:45:37.954058+0000","last_peered":"2024-01-11T09:45:37.954058+0000","last_clean":"2024-01-11T09:45:37.954058+0000","last_became_active":"2024-01-11T09:45:24.167138+0000","last_became_peered":"2024-01-11T09:45:24.167138+0000","last_unstale":"2024-01-11T09:45:37.954058+0000","last_undegraded":"2024-01-11T09:45:37.954058+0000","last_fullsized":"2024-01-11T09:45:37.954058+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:43:22.330951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.982381+0000","last_change":"2024-01-11T09:45:37.980159+0000","last_active":"2024-01-11T09:45:37.982381+0000","last_peered":"2024-01-11T09:45:37.982381+0000","last_clean":"2024-01-11T09:45:37.982381+0000","last_became_active":"2024-01-11T09:45:37.976703+0000","last_became_peered":"2024-01-11T09:45:37.976703+0000","last_unstale":"2024-01-11T09:45:37.982381+0000","last_undegraded":"2024-01-11T09:45:37.982381+0000","last_fullsized":"2024-01-11T09:45:37.982381+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:49:01.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950736+0000","last_change":"2024-01-11T09:45:24.158003+0000","last_active":"2024-01-11T09:45:37.950736+0000","last_peered":"2024-01-11T09:45:37.950736+0000","last_clean":"2024-01-11T09:45:37.950736+0000","last_became_active":"2024-01-11T09:45:24.157710+0000","last_became_peered":"2024-01-11T09:45:24.157710+0000","last_unstale":"2024-01-11T09:45:37.950736+0000","last_undegraded":"2024-01-11T09:45:37.950736+0000","last_fullsized":"2024-01-11T09:45:37.950736+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:52:18.543824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.971932+0000","last_change":"2024-01-11T09:45:37.965176+0000","last_active":"2024-01-11T09:45:37.971932+0000","last_peered":"2024-01-11T09:45:37.971932+0000","last_clean":"2024-01-11T09:45:37.971932+0000","last_became_active":"2024-01-11T09:45:37.964838+0000","last_became_peered":"2024-01-11T09:45:37.964838+0000","last_unstale":"2024-01-11T09:45:37.971932+0000","last_undegraded":"2024-01-11T09:45:37.971932+0000","last_fullsized":"2024-01-11T09:45:37.971932+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:32:49.312144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950589+0000","last_change":"2024-01-11T09:45:24.145558+0000","last_active":"2024-01-11T09:45:37.950589+0000","last_peered":"2024-01-11T09:45:37.950589+0000","last_clean":"2024-01-11T09:45:37.950589+0000","last_became_active":"2024-01-11T09:45:24.143216+0000","last_became_peered":"2024-01-11T09:45:24.143216+0000","last_unstale":"2024-01-11T09:45:37.950589+0000","last_undegraded":"2024-01-11T09:45:37.950589+0000","last_fullsized":"2024-01-11T09:45:37.950589+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:32:19.368822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.988413+0000","last_change":"2024-01-11T09:45:37.983715+0000","last_active":"2024-01-11T09:45:37.988413+0000","last_peered":"2024-01-11T09:45:37.988413+0000","last_clean":"2024-01-11T09:45:37.988413+0000","last_became_active":"2024-01-11T09:45:37.983461+0000","last_became_peered":"2024-01-11T09:45:37.983461+0000","last_unstale":"2024-01-11T09:45:37.988413+0000","last_undegraded":"2024-01-11T09:45:37.988413+0000","last_fullsized":"2024-01-11T09:45:37.988413+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:47:37.322693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954769+0000","last_change":"2024-01-11T09:45:24.167154+0000","last_active":"2024-01-11T09:45:37.954769+0000","last_peered":"2024-01-11T09:45:37.954769+0000","last_clean":"2024-01-11T09:45:37.954769+0000","last_became_active":"2024-01-11T09:45:24.166862+0000","last_became_peered":"2024-01-11T09:45:24.166862+0000","last_unstale":"2024-01-11T09:45:37.954769+0000","last_undegraded":"2024-01-11T09:45:37.954769+0000","last_fullsized":"2024-01-11T09:45:37.954769+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:15:00.803121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.979055+0000","last_change":"2024-01-11T09:45:37.972382+0000","last_active":"2024-01-11T09:45:37.979055+0000","last_peered":"2024-01-11T09:45:37.979055+0000","last_clean":"2024-01-11T09:45:37.979055+0000","last_became_active":"2024-01-11T09:45:37.970224+0000","last_became_peered":"2024-01-11T09:45:37.970224+0000","last_unstale":"2024-01-11T09:45:37.979055+0000","last_undegraded":"2024-01-11T09:45:37.979055+0000","last_fullsized":"2024-01-11T09:45:37.979055+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:46:18.798102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.966130+0000","last_change":"2024-01-11T09:45:24.156832+0000","last_active":"2024-01-11T09:45:37.966130+0000","last_peered":"2024-01-11T09:45:37.966130+0000","last_clean":"2024-01-11T09:45:37.966130+0000","last_became_active":"2024-01-11T09:45:24.156541+0000","last_became_peered":"2024-01-11T09:45:24.156541+0000","last_unstale":"2024-01-11T09:45:37.966130+0000","last_undegraded":"2024-01-11T09:45:37.966130+0000","last_fullsized":"2024-01-11T09:45:37.966130+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:49.368305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.979956+0000","last_change":"2024-01-11T09:45:37.967199+0000","last_active":"2024-01-11T09:45:37.979956+0000","last_peered":"2024-01-11T09:45:37.979956+0000","last_clean":"2024-01-11T09:45:37.979956+0000","last_became_active":"2024-01-11T09:45:37.966928+0000","last_became_peered":"2024-01-11T09:45:37.966928+0000","last_unstale":"2024-01-11T09:45:37.979956+0000","last_undegraded":"2024-01-11T09:45:37.979956+0000","last_fullsized":"2024-01-11T09:45:37.979956+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:29:41.388676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950156+0000","last_change":"2024-01-11T09:45:24.153140+0000","last_active":"2024-01-11T09:45:37.950156+0000","last_peered":"2024-01-11T09:45:37.950156+0000","last_clean":"2024-01-11T09:45:37.950156+0000","last_became_active":"2024-01-11T09:45:24.152890+0000","last_became_peered":"2024-01-11T09:45:24.152890+0000","last_unstale":"2024-01-11T09:45:37.950156+0000","last_undegraded":"2024-01-11T09:45:37.950156+0000","last_fullsized":"2024-01-11T09:45:37.950156+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:39.406018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.986364+0000","last_change":"2024-01-11T09:45:37.979637+0000","last_active":"2024-01-11T09:45:37.986364+0000","last_peered":"2024-01-11T09:45:37.986364+0000","last_clean":"2024-01-11T09:45:37.986364+0000","last_became_active":"2024-01-11T09:45:37.979380+0000","last_became_peered":"2024-01-11T09:45:37.979380+0000","last_unstale":"2024-01-11T09:45:37.986364+0000","last_undegraded":"2024-01-11T09:45:37.986364+0000","last_fullsized":"2024-01-11T09:45:37.986364+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:15:52.374499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.951564+0000","last_change":"2024-01-11T09:45:24.166373+0000","last_active":"2024-01-11T09:45:37.951564+0000","last_peered":"2024-01-11T09:45:37.951564+0000","last_clean":"2024-01-11T09:45:37.951564+0000","last_became_active":"2024-01-11T09:45:24.166090+0000","last_became_peered":"2024-01-11T09:45:24.166090+0000","last_unstale":"2024-01-11T09:45:37.951564+0000","last_undegraded":"2024-01-11T09:45:37.951564+0000","last_fullsized":"2024-01-11T09:45:37.951564+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:16:58.784096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.981477+0000","last_change":"2024-01-11T09:45:37.979710+0000","last_active":"2024-01-11T09:45:37.981477+0000","last_peered":"2024-01-11T09:45:37.981477+0000","last_clean":"2024-01-11T09:45:37.981477+0000","last_became_active":"2024-01-11T09:45:37.979392+0000","last_became_peered":"2024-01-11T09:45:37.979392+0000","last_unstale":"2024-01-11T09:45:37.981477+0000","last_undegraded":"2024-01-11T09:45:37.981477+0000","last_fullsized":"2024-01-11T09:45:37.981477+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:08:58.113291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.963215+0000","last_change":"2024-01-11T09:45:24.164171+0000","last_active":"2024-01-11T09:45:37.963215+0000","last_peered":"2024-01-11T09:45:37.963215+0000","last_clean":"2024-01-11T09:45:37.963215+0000","last_became_active":"2024-01-11T09:45:24.163522+0000","last_became_peered":"2024-01-11T09:45:24.163522+0000","last_unstale":"2024-01-11T09:45:37.963215+0000","last_undegraded":"2024-01-11T09:45:37.963215+0000","last_fullsized":"2024-01-11T09:45:37.963215+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:37:20.217033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.982287+0000","last_change":"2024-01-11T09:45:37.978335+0000","last_active":"2024-01-11T09:45:37.982287+0000","last_peered":"2024-01-11T09:45:37.982287+0000","last_clean":"2024-01-11T09:45:37.982287+0000","last_became_active":"2024-01-11T09:45:37.978064+0000","last_became_peered":"2024-01-11T09:45:37.978064+0000","last_unstale":"2024-01-11T09:45:37.982287+0000","last_undegraded":"2024-01-11T09:45:37.982287+0000","last_fullsized":"2024-01-11T09:45:37.982287+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:26:56.082604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.948834+0000","last_change":"2024-01-11T09:45:24.148308+0000","last_active":"2024-01-11T09:45:37.948834+0000","last_peered":"2024-01-11T09:45:37.948834+0000","last_clean":"2024-01-11T09:45:37.948834+0000","last_became_active":"2024-01-11T09:45:24.148023+0000","last_became_peered":"2024-01-11T09:45:24.148023+0000","last_unstale":"2024-01-11T09:45:37.948834+0000","last_undegraded":"2024-01-11T09:45:37.948834+0000","last_fullsized":"2024-01-11T09:45:37.948834+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:56:51.623335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984817+0000","last_change":"2024-01-11T09:45:37.977229+0000","last_active":"2024-01-11T09:45:37.984817+0000","last_peered":"2024-01-11T09:45:37.984817+0000","last_clean":"2024-01-11T09:45:37.984817+0000","last_became_active":"2024-01-11T09:45:37.976956+0000","last_became_peered":"2024-01-11T09:45:37.976956+0000","last_unstale":"2024-01-11T09:45:37.984817+0000","last_undegraded":"2024-01-11T09:45:37.984817+0000","last_fullsized":"2024-01-11T09:45:37.984817+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:30:49.529765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.958690+0000","last_change":"2024-01-11T09:45:24.170581+0000","last_active":"2024-01-11T09:45:37.958690+0000","last_peered":"2024-01-11T09:45:37.958690+0000","last_clean":"2024-01-11T09:45:37.958690+0000","last_became_active":"2024-01-11T09:45:24.170233+0000","last_became_peered":"2024-01-11T09:45:24.170233+0000","last_unstale":"2024-01-11T09:45:37.958690+0000","last_undegraded":"2024-01-11T09:45:37.958690+0000","last_fullsized":"2024-01-11T09:45:37.958690+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:50:50.820643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.982420+0000","last_change":"2024-01-11T09:45:37.971726+0000","last_active":"2024-01-11T09:45:37.982420+0000","last_peered":"2024-01-11T09:45:37.982420+0000","last_clean":"2024-01-11T09:45:37.982420+0000","last_became_active":"2024-01-11T09:45:37.971126+0000","last_became_peered":"2024-01-11T09:45:37.971126+0000","last_unstale":"2024-01-11T09:45:37.982420+0000","last_undegraded":"2024-01-11T09:45:37.982420+0000","last_fullsized":"2024-01-11T09:45:37.982420+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:08:58.113291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.956787+0000","last_change":"2024-01-11T09:45:24.157215+0000","last_active":"2024-01-11T09:45:37.956787+0000","last_peered":"2024-01-11T09:45:37.956787+0000","last_clean":"2024-01-11T09:45:37.956787+0000","last_became_active":"2024-01-11T09:45:24.156852+0000","last_became_peered":"2024-01-11T09:45:24.156852+0000","last_unstale":"2024-01-11T09:45:37.956787+0000","last_undegraded":"2024-01-11T09:45:37.956787+0000","last_fullsized":"2024-01-11T09:45:37.956787+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:42.716516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.974315+0000","last_change":"2024-01-11T09:45:37.971871+0000","last_active":"2024-01-11T09:45:37.974315+0000","last_peered":"2024-01-11T09:45:37.974315+0000","last_clean":"2024-01-11T09:45:37.974315+0000","last_became_active":"2024-01-11T09:45:37.971223+0000","last_became_peered":"2024-01-11T09:45:37.971223+0000","last_unstale":"2024-01-11T09:45:37.974315+0000","last_undegraded":"2024-01-11T09:45:37.974315+0000","last_fullsized":"2024-01-11T09:45:37.974315+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:26:56.082604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.967678+0000","last_change":"2024-01-11T09:45:24.158500+0000","last_active":"2024-01-11T09:45:37.967678+0000","last_peered":"2024-01-11T09:45:37.967678+0000","last_clean":"2024-01-11T09:45:37.967678+0000","last_became_active":"2024-01-11T09:45:24.158185+0000","last_became_peered":"2024-01-11T09:45:24.158185+0000","last_unstale":"2024-01-11T09:45:37.967678+0000","last_undegraded":"2024-01-11T09:45:37.967678+0000","last_fullsized":"2024-01-11T09:45:37.967678+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:52:38.581476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.970263+0000","last_change":"2024-01-11T09:45:37.968152+0000","last_active":"2024-01-11T09:45:37.970263+0000","last_peered":"2024-01-11T09:45:37.970263+0000","last_clean":"2024-01-11T09:45:37.970263+0000","last_became_active":"2024-01-11T09:45:37.967824+0000","last_became_peered":"2024-01-11T09:45:37.967824+0000","last_unstale":"2024-01-11T09:45:37.970263+0000","last_undegraded":"2024-01-11T09:45:37.970263+0000","last_fullsized":"2024-01-11T09:45:37.970263+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:09:00.141609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.972493+0000","last_change":"2024-01-11T09:45:24.146920+0000","last_active":"2024-01-11T09:45:37.972493+0000","last_peered":"2024-01-11T09:45:37.972493+0000","last_clean":"2024-01-11T09:45:37.972493+0000","last_became_active":"2024-01-11T09:45:24.146654+0000","last_became_peered":"2024-01-11T09:45:24.146654+0000","last_unstale":"2024-01-11T09:45:37.972493+0000","last_undegraded":"2024-01-11T09:45:37.972493+0000","last_fullsized":"2024-01-11T09:45:37.972493+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:42:21.543118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.963957+0000","last_change":"2024-01-11T09:45:37.960537+0000","last_active":"2024-01-11T09:45:37.963957+0000","last_peered":"2024-01-11T09:45:37.963957+0000","last_clean":"2024-01-11T09:45:37.963957+0000","last_became_active":"2024-01-11T09:45:37.959337+0000","last_became_peered":"2024-01-11T09:45:37.959337+0000","last_unstale":"2024-01-11T09:45:37.963957+0000","last_undegraded":"2024-01-11T09:45:37.963957+0000","last_fullsized":"2024-01-11T09:45:37.963957+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:04:55.352559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960969+0000","last_change":"2024-01-11T09:45:24.161300+0000","last_active":"2024-01-11T09:45:37.960969+0000","last_peered":"2024-01-11T09:45:37.960969+0000","last_clean":"2024-01-11T09:45:37.960969+0000","last_became_active":"2024-01-11T09:45:24.160940+0000","last_became_peered":"2024-01-11T09:45:24.160940+0000","last_unstale":"2024-01-11T09:45:37.960969+0000","last_undegraded":"2024-01-11T09:45:37.960969+0000","last_fullsized":"2024-01-11T09:45:37.960969+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:40:06.419759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984070+0000","last_change":"2024-01-11T09:45:37.973489+0000","last_active":"2024-01-11T09:45:37.984070+0000","last_peered":"2024-01-11T09:45:37.984070+0000","last_clean":"2024-01-11T09:45:37.984070+0000","last_became_active":"2024-01-11T09:45:37.971292+0000","last_became_peered":"2024-01-11T09:45:37.971292+0000","last_unstale":"2024-01-11T09:45:37.984070+0000","last_undegraded":"2024-01-11T09:45:37.984070+0000","last_fullsized":"2024-01-11T09:45:37.984070+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:47:37.322693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.951480+0000","last_change":"2024-01-11T09:45:24.152214+0000","last_active":"2024-01-11T09:45:37.951480+0000","last_peered":"2024-01-11T09:45:37.951480+0000","last_clean":"2024-01-11T09:45:37.951480+0000","last_became_active":"2024-01-11T09:45:24.151938+0000","last_became_peered":"2024-01-11T09:45:24.151938+0000","last_unstale":"2024-01-11T09:45:37.951480+0000","last_undegraded":"2024-01-11T09:45:37.951480+0000","last_fullsized":"2024-01-11T09:45:37.951480+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:05.090713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.983891+0000","last_change":"2024-01-11T09:45:37.979205+0000","last_active":"2024-01-11T09:45:37.983891+0000","last_peered":"2024-01-11T09:45:37.983891+0000","last_clean":"2024-01-11T09:45:37.983891+0000","last_became_active":"2024-01-11T09:45:37.977222+0000","last_became_peered":"2024-01-11T09:45:37.977222+0000","last_unstale":"2024-01-11T09:45:37.983891+0000","last_undegraded":"2024-01-11T09:45:37.983891+0000","last_fullsized":"2024-01-11T09:45:37.983891+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:07:03.144535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950339+0000","last_change":"2024-01-11T09:45:24.155318+0000","last_active":"2024-01-11T09:45:37.950339+0000","last_peered":"2024-01-11T09:45:37.950339+0000","last_clean":"2024-01-11T09:45:37.950339+0000","last_became_active":"2024-01-11T09:45:24.153405+0000","last_became_peered":"2024-01-11T09:45:24.153405+0000","last_unstale":"2024-01-11T09:45:37.950339+0000","last_undegraded":"2024-01-11T09:45:37.950339+0000","last_fullsized":"2024-01-11T09:45:37.950339+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:52:32.496845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.983145+0000","last_change":"2024-01-11T09:45:37.977665+0000","last_active":"2024-01-11T09:45:37.983145+0000","last_peered":"2024-01-11T09:45:37.983145+0000","last_clean":"2024-01-11T09:45:37.983145+0000","last_became_active":"2024-01-11T09:45:37.974163+0000","last_became_peered":"2024-01-11T09:45:37.974163+0000","last_unstale":"2024-01-11T09:45:37.983145+0000","last_undegraded":"2024-01-11T09:45:37.983145+0000","last_fullsized":"2024-01-11T09:45:37.983145+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:31.003882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.961527+0000","last_change":"2024-01-11T09:45:24.169914+0000","last_active":"2024-01-11T09:45:37.961527+0000","last_peered":"2024-01-11T09:45:37.961527+0000","last_clean":"2024-01-11T09:45:37.961527+0000","last_became_active":"2024-01-11T09:45:24.169438+0000","last_became_peered":"2024-01-11T09:45:24.169438+0000","last_unstale":"2024-01-11T09:45:37.961527+0000","last_undegraded":"2024-01-11T09:45:37.961527+0000","last_fullsized":"2024-01-11T09:45:37.961527+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:43:22.330951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.974323+0000","last_change":"2024-01-11T09:45:37.970756+0000","last_active":"2024-01-11T09:45:37.974323+0000","last_peered":"2024-01-11T09:45:37.974323+0000","last_clean":"2024-01-11T09:45:37.974323+0000","last_became_active":"2024-01-11T09:45:37.970446+0000","last_became_peered":"2024-01-11T09:45:37.970446+0000","last_unstale":"2024-01-11T09:45:37.974323+0000","last_undegraded":"2024-01-11T09:45:37.974323+0000","last_fullsized":"2024-01-11T09:45:37.974323+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:06:21.286305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.958611+0000","last_change":"2024-01-11T09:45:24.164133+0000","last_active":"2024-01-11T09:45:37.958611+0000","last_peered":"2024-01-11T09:45:37.958611+0000","last_clean":"2024-01-11T09:45:37.958611+0000","last_became_active":"2024-01-11T09:45:24.163856+0000","last_became_peered":"2024-01-11T09:45:24.163856+0000","last_unstale":"2024-01-11T09:45:37.958611+0000","last_undegraded":"2024-01-11T09:45:37.958611+0000","last_fullsized":"2024-01-11T09:45:37.958611+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:17:24.413093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.978680+0000","last_change":"2024-01-11T09:45:37.968969+0000","last_active":"2024-01-11T09:45:37.978680+0000","last_peered":"2024-01-11T09:45:37.978680+0000","last_clean":"2024-01-11T09:45:37.978680+0000","last_became_active":"2024-01-11T09:45:37.968664+0000","last_became_peered":"2024-01-11T09:45:37.968664+0000","last_unstale":"2024-01-11T09:45:37.978680+0000","last_undegraded":"2024-01-11T09:45:37.978680+0000","last_fullsized":"2024-01-11T09:45:37.978680+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:08:58.113291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.961712+0000","last_change":"2024-01-11T09:45:24.153507+0000","last_active":"2024-01-11T09:45:37.961712+0000","last_peered":"2024-01-11T09:45:37.961712+0000","last_clean":"2024-01-11T09:45:37.961712+0000","last_became_active":"2024-01-11T09:45:24.153235+0000","last_became_peered":"2024-01-11T09:45:24.153235+0000","last_unstale":"2024-01-11T09:45:37.961712+0000","last_undegraded":"2024-01-11T09:45:37.961712+0000","last_fullsized":"2024-01-11T09:45:37.961712+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:01:06.710133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.988445+0000","last_change":"2024-01-11T09:45:37.985398+0000","last_active":"2024-01-11T09:45:37.988445+0000","last_peered":"2024-01-11T09:45:37.988445+0000","last_clean":"2024-01-11T09:45:37.988445+0000","last_became_active":"2024-01-11T09:45:37.984360+0000","last_became_peered":"2024-01-11T09:45:37.984360+0000","last_unstale":"2024-01-11T09:45:37.988445+0000","last_undegraded":"2024-01-11T09:45:37.988445+0000","last_fullsized":"2024-01-11T09:45:37.988445+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:42:31.879941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.963305+0000","last_change":"2024-01-11T09:45:24.148863+0000","last_active":"2024-01-11T09:45:37.963305+0000","last_peered":"2024-01-11T09:45:37.963305+0000","last_clean":"2024-01-11T09:45:37.963305+0000","last_became_active":"2024-01-11T09:45:24.148533+0000","last_became_peered":"2024-01-11T09:45:24.148533+0000","last_unstale":"2024-01-11T09:45:37.963305+0000","last_undegraded":"2024-01-11T09:45:37.963305+0000","last_fullsized":"2024-01-11T09:45:37.963305+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:20.218042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.957184+0000","last_change":"2024-01-11T09:45:37.954469+0000","last_active":"2024-01-11T09:45:37.957184+0000","last_peered":"2024-01-11T09:45:37.957184+0000","last_clean":"2024-01-11T09:45:37.957184+0000","last_became_active":"2024-01-11T09:45:37.954186+0000","last_became_peered":"2024-01-11T09:45:37.954186+0000","last_unstale":"2024-01-11T09:45:37.957184+0000","last_undegraded":"2024-01-11T09:45:37.957184+0000","last_fullsized":"2024-01-11T09:45:37.957184+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:06:21.286305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954687+0000","last_change":"2024-01-11T09:45:24.149180+0000","last_active":"2024-01-11T09:45:37.954687+0000","last_peered":"2024-01-11T09:45:37.954687+0000","last_clean":"2024-01-11T09:45:37.954687+0000","last_became_active":"2024-01-11T09:45:24.148899+0000","last_became_peered":"2024-01-11T09:45:24.148899+0000","last_unstale":"2024-01-11T09:45:37.954687+0000","last_undegraded":"2024-01-11T09:45:37.954687+0000","last_fullsized":"2024-01-11T09:45:37.954687+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:59:56.132354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.979980+0000","last_change":"2024-01-11T09:45:37.971700+0000","last_active":"2024-01-11T09:45:37.979980+0000","last_peered":"2024-01-11T09:45:37.979980+0000","last_clean":"2024-01-11T09:45:37.979980+0000","last_became_active":"2024-01-11T09:45:37.971458+0000","last_became_peered":"2024-01-11T09:45:37.971458+0000","last_unstale":"2024-01-11T09:45:37.979980+0000","last_undegraded":"2024-01-11T09:45:37.979980+0000","last_fullsized":"2024-01-11T09:45:37.979980+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:04:55.352559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960669+0000","last_change":"2024-01-11T09:45:24.160833+0000","last_active":"2024-01-11T09:45:37.960669+0000","last_peered":"2024-01-11T09:45:37.960669+0000","last_clean":"2024-01-11T09:45:37.960669+0000","last_became_active":"2024-01-11T09:45:24.160557+0000","last_became_peered":"2024-01-11T09:45:24.160557+0000","last_unstale":"2024-01-11T09:45:37.960669+0000","last_undegraded":"2024-01-11T09:45:37.960669+0000","last_fullsized":"2024-01-11T09:45:37.960669+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:29:13.303845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.971889+0000","last_change":"2024-01-11T09:45:37.963861+0000","last_active":"2024-01-11T09:45:37.971889+0000","last_peered":"2024-01-11T09:45:37.971889+0000","last_clean":"2024-01-11T09:45:37.971889+0000","last_became_active":"2024-01-11T09:45:37.963598+0000","last_became_peered":"2024-01-11T09:45:37.963598+0000","last_unstale":"2024-01-11T09:45:37.971889+0000","last_undegraded":"2024-01-11T09:45:37.971889+0000","last_fullsized":"2024-01-11T09:45:37.971889+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:52:26.173608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950401+0000","last_change":"2024-01-11T09:45:24.147655+0000","last_active":"2024-01-11T09:45:37.950401+0000","last_peered":"2024-01-11T09:45:37.950401+0000","last_clean":"2024-01-11T09:45:37.950401+0000","last_became_active":"2024-01-11T09:45:24.147267+0000","last_became_peered":"2024-01-11T09:45:24.147267+0000","last_unstale":"2024-01-11T09:45:37.950401+0000","last_undegraded":"2024-01-11T09:45:37.950401+0000","last_fullsized":"2024-01-11T09:45:37.950401+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:14:09.653181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.986187+0000","last_change":"2024-01-11T09:45:37.980685+0000","last_active":"2024-01-11T09:45:37.986187+0000","last_peered":"2024-01-11T09:45:37.986187+0000","last_clean":"2024-01-11T09:45:37.986187+0000","last_became_active":"2024-01-11T09:45:37.980384+0000","last_became_peered":"2024-01-11T09:45:37.980384+0000","last_unstale":"2024-01-11T09:45:37.986187+0000","last_undegraded":"2024-01-11T09:45:37.986187+0000","last_fullsized":"2024-01-11T09:45:37.986187+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:39:49.483450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.963124+0000","last_change":"2024-01-11T09:45:24.164535+0000","last_active":"2024-01-11T09:45:37.963124+0000","last_peered":"2024-01-11T09:45:37.963124+0000","last_clean":"2024-01-11T09:45:37.963124+0000","last_became_active":"2024-01-11T09:45:24.163889+0000","last_became_peered":"2024-01-11T09:45:24.163889+0000","last_unstale":"2024-01-11T09:45:37.963124+0000","last_undegraded":"2024-01-11T09:45:37.963124+0000","last_fullsized":"2024-01-11T09:45:37.963124+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:29:27.569363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3519,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":611,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":366,"ondisk_log_size":366,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575975,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300556,"kb_used_data":3268,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071284,"statfs":{"total":96636764160,"available":96328994816,"internally_reserved":0,"allocated":3346432,"data_stored":1904090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.69599999999999995,"15min":0.73299999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.36599999999999999},"max":{"1min":0.93000000000000005,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.878},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.72599999999999998,"15min":0.747},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.026,"5min":1.0609999999999999,"15min":1.0669999999999999},"last":0.73099999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.70699999999999996,"15min":0.73299999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41599999999999998},"max":{"1min":0.91600000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.71799999999999997,"15min":0.74199999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.95899999999999996,"5min":0.999,"15min":1.0820000000000001},"last":0.89400000000000002}]},{"osd":2,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.73599999999999999,"15min":0.74399999999999999},"min":{"1min":0.44500000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.93799999999999994,"5min":0.94399999999999995,"15min":1.0600000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73199999999999998,"15min":0.754},"min":{"1min":0.503,"5min":0.44800000000000001,"15min":0.44500000000000001},"max":{"1min":0.92800000000000005,"5min":0.96999999999999997,"15min":1.153},"last":0.86099999999999999}]},{"osd":3,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73799999999999999,"15min":0.76700000000000002},"min":{"1min":0.51600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.93000000000000005,"5min":0.96699999999999997,"15min":1.0660000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73699999999999999,"15min":0.75600000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.94099999999999995,"5min":1,"15min":1.0880000000000001},"last":0.71199999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.432,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0720000000000001,"5min":1.0860000000000001,"15min":1.107},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.629,"15min":0.63800000000000001},"min":{"1min":0.47699999999999998,"5min":0.44400000000000001,"15min":0.39700000000000002},"max":{"1min":0.89000000000000001,"5min":0.995,"15min":1.0960000000000001},"last":0.58899999999999997}]},{"osd":5,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.61199999999999999,"15min":0.63800000000000001},"min":{"1min":0.48999999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86599999999999999,"5min":0.86699999999999999,"15min":1.0109999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.67900000000000005,"15min":0.68400000000000005},"min":{"1min":0.46899999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.65700000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.76300000000000001,"15min":0.77900000000000003},"min":{"1min":0.55900000000000005,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.747,"15min":0.76000000000000001},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.45800000000000002},"max":{"1min":0.95899999999999996,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.83299999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575974,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312260,"kb_used_data":2684,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059580,"statfs":{"total":96636764160,"available":96317009920,"internally_reserved":0,"allocated":2748416,"data_stored":1311513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67800000000000005,"15min":0.71399999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83899999999999997,"5min":0.94199999999999995,"15min":1.508},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.71899999999999997,"15min":0.748},"min":{"1min":0.501,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.92600000000000005,"5min":0.97099999999999997,"15min":1.5589999999999999},"last":0.65500000000000003}]},{"osd":1,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.69199999999999995,"15min":0.74099999999999999},"min":{"1min":0.42799999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84799999999999998,"5min":0.94699999999999995,"15min":1.45},"last":0.747},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.70599999999999996,"15min":0.73699999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.79400000000000004,"5min":0.98299999999999998,"15min":1.542},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.72599999999999998,"15min":0.752},"min":{"1min":0.52500000000000002,"5min":0.501,"15min":0.44500000000000001},"max":{"1min":0.85899999999999999,"5min":1.034,"15min":1.5329999999999999},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.72399999999999998,"15min":0.75700000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89500000000000002,"5min":0.93999999999999995,"15min":1.647},"last":0.51500000000000001}]},{"osd":3,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.747,"15min":0.76500000000000001},"min":{"1min":0.54900000000000004,"5min":0.502,"15min":0.45200000000000001},"max":{"1min":0.88200000000000001,"5min":1.012,"15min":1.6140000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.72699999999999998,"15min":0.77400000000000002},"min":{"1min":0.53600000000000003,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.83499999999999996,"5min":0.96499999999999997,"15min":1.4950000000000001},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86899999999999999,"5min":0.91700000000000004,"15min":1.0309999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61499999999999999,"15min":0.61199999999999999},"min":{"1min":0.42499999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.86199999999999999,"5min":0.94799999999999995,"15min":1.1259999999999999},"last":0.88500000000000001}]},{"osd":5,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64700000000000002,"15min":0.65900000000000003},"min":{"1min":0.45800000000000002,"5min":0.41499999999999998,"15min":0.40000000000000002},"max":{"1min":0.92700000000000005,"5min":0.97899999999999998,"15min":1.369},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.49099999999999999,"5min":0.46700000000000003,"15min":0.45400000000000001},"max":{"1min":0.89400000000000002,"5min":0.999,"15min":1.0960000000000001},"last":0.78500000000000003}]},{"osd":7,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70099999999999996,"15min":0.70799999999999996},"min":{"1min":0.53000000000000003,"5min":0.45100000000000001,"15min":0.44400000000000001},"max":{"1min":0.91900000000000004,"5min":1.089,"15min":1.484},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.755,"5min":0.76100000000000001,"15min":0.76900000000000002},"min":{"1min":0.48699999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0609999999999999,"5min":1.145,"15min":1.603},"last":0.98299999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575976,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308164,"kb_used_data":2684,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063676,"statfs":{"total":96636764160,"available":96321204224,"internally_reserved":0,"allocated":2748416,"data_stored":1311049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52500000000000002,"15min":0.51500000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.315},"max":{"1min":0.66000000000000003,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61399999999999999,"15min":0.622},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.748,"5min":0.85199999999999998,"15min":0.95099999999999996},"last":0.47799999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.51300000000000001,"15min":0.51400000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.64000000000000001,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60099999999999998,"15min":0.59399999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39400000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.63800000000000001}]},{"osd":3,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64400000000000002,"15min":0.65000000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.42699999999999999},"max":{"1min":0.81200000000000006,"5min":0.91600000000000004,"15min":0.96699999999999997},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.40400000000000003},"max":{"1min":0.70499999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.436}]},{"osd":4,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.81899999999999995,"15min":0.82399999999999995},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.505},"max":{"1min":1.1080000000000001,"5min":1.214,"15min":1.2969999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.879,"5min":0.85999999999999999,"15min":0.86399999999999999},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.61699999999999999},"max":{"1min":1.0129999999999999,"5min":1.1879999999999999,"15min":1.1890000000000001},"last":0.72299999999999998}]},{"osd":5,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83899999999999997,"15min":0.83299999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.093,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.81799999999999995,"15min":0.80700000000000005},"min":{"1min":0.54200000000000004,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":1.0569999999999999,"5min":1.341,"15min":1.341},"last":0.753}]},{"osd":6,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.84599999999999997,"15min":0.84899999999999998},"min":{"1min":0.67000000000000004,"5min":0.59599999999999997,"15min":0.56799999999999995},"max":{"1min":1.073,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.85099999999999998,"15min":0.85499999999999998},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.55900000000000005},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.624}]},{"osd":7,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89600000000000002,"15min":0.89100000000000001},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.143,"5min":1.421,"15min":1.421},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.86499999999999999,"15min":0.85899999999999999},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.54500000000000004},"max":{"1min":1.083,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.81200000000000006}]}]},{"osd":0,"up_from":13,"seq":55834575975,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308744,"kb_used_data":3264,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063096,"statfs":{"total":96636764160,"available":96320610304,"internally_reserved":0,"allocated":3342336,"data_stored":1902005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34499999999999997,"5min":0.34300000000000003,"15min":0.32100000000000001},"max":{"1min":0.754,"5min":0.84699999999999998,"15min":0.875},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.60099999999999998,"15min":0.60199999999999998},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.81599999999999995,"5min":0.86399999999999999,"15min":0.874},"last":0.51300000000000001}]},{"osd":2,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.56799999999999995,"15min":0.57299999999999995},"min":{"1min":0.42499999999999999,"5min":0.40400000000000003,"15min":0.36599999999999999},"max":{"1min":0.79900000000000004,"5min":0.84499999999999997,"15min":0.97499999999999998},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.438,"5min":0.39600000000000002,"15min":0.38500000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.86399999999999999},"last":0.55400000000000005}]},{"osd":3,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54000000000000004,"15min":0.54600000000000004},"min":{"1min":0.42499999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69199999999999995,"5min":0.92600000000000005,"15min":0.95399999999999996},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64700000000000002,"15min":0.65600000000000003},"min":{"1min":0.46800000000000003,"5min":0.45900000000000002,"15min":0.41999999999999998},"max":{"1min":0.77800000000000002,"5min":0.92600000000000005,"15min":1.0389999999999999},"last":0.52800000000000002}]},{"osd":4,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.83099999999999996,"15min":0.82299999999999995},"min":{"1min":0.60499999999999998,"5min":0.60299999999999998,"15min":0.57399999999999995},"max":{"1min":1.0569999999999999,"5min":1.1779999999999999,"15min":1.304},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.81699999999999995,"15min":0.81000000000000005},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.272,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.78000000000000003}]},{"osd":5,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.84299999999999997,"15min":0.84399999999999997},"min":{"1min":0.67200000000000004,"5min":0.65000000000000002,"15min":0.53900000000000003},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.84599999999999997,"15min":0.84799999999999998},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.153,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.80500000000000005}]},{"osd":6,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83499999999999996,"15min":0.82799999999999996},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.55700000000000005},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.871,"5min":0.872,"15min":0.875},"min":{"1min":0.63800000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.1699999999999999,"5min":1.256,"15min":1.256},"last":0.73699999999999999}]},{"osd":7,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.877,"15min":0.871},"min":{"1min":0.66100000000000003,"5min":0.59099999999999997,"15min":0.58699999999999997},"max":{"1min":1.226,"5min":1.28,"15min":1.28},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.88100000000000001,"15min":0.879},"min":{"1min":0.65300000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.1259999999999999,"5min":1.238,"15min":1.2410000000000001},"last":0.65900000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575974,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312268,"kb_used_data":2692,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059572,"statfs":{"total":96636764160,"available":96317001728,"internally_reserved":0,"allocated":2756608,"data_stored":1312495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.49299999999999999,"15min":0.505},"min":{"1min":0.379,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.627,"5min":0.78800000000000003,"15min":0.85399999999999998},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55200000000000005,"15min":0.56499999999999995},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34300000000000003},"max":{"1min":0.71099999999999997,"5min":1.131,"15min":1.6359999999999999},"last":0.63}]},{"osd":1,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.38200000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.751,"5min":0.88800000000000001,"15min":0.93000000000000005},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60199999999999998,"15min":0.60399999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.40000000000000002},"max":{"1min":0.77400000000000002,"5min":0.97299999999999998,"15min":1.145},"last":0.73099999999999998}]},{"osd":3,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.42999999999999999,"5min":0.377,"15min":0.36199999999999999},"max":{"1min":0.76800000000000002,"5min":0.83099999999999996,"15min":0.96399999999999997},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.66200000000000003},"min":{"1min":0.50600000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.876,"5min":0.92300000000000004,"15min":1.3620000000000001},"last":0.754}]},{"osd":4,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79000000000000004,"15min":0.80300000000000005},"min":{"1min":0.63400000000000001,"5min":0.57799999999999996,"15min":0.54300000000000004},"max":{"1min":0.97199999999999998,"5min":1.071,"15min":1.1519999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.80100000000000005,"15min":0.80400000000000005},"min":{"1min":0.56200000000000006,"5min":0.505,"15min":0.505},"max":{"1min":1.1439999999999999,"5min":1.161,"15min":1.161},"last":0.80600000000000005}]},{"osd":5,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.85899999999999999,"15min":0.877},"min":{"1min":0.59699999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.228,"5min":1.228,"15min":1.429},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.85799999999999998,"15min":0.86899999999999999},"min":{"1min":0.65900000000000003,"5min":0.64700000000000002,"15min":0.61099999999999999},"max":{"1min":1.0629999999999999,"5min":1.1850000000000001,"15min":1.2170000000000001},"last":0.84399999999999997}]},{"osd":6,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.81899999999999995,"15min":0.83099999999999996},"min":{"1min":0.59199999999999997,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.091,"5min":1.1659999999999999,"15min":1.4139999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.87,"5min":0.84799999999999998,"15min":0.873},"min":{"1min":0.68300000000000005,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.173,"5min":1.2010000000000001,"15min":1.593},"last":0.96599999999999997}]},{"osd":7,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.88,"15min":0.89800000000000002},"min":{"1min":0.72299999999999998,"5min":0.623,"15min":0.57299999999999995},"max":{"1min":1.111,"5min":1.137,"15min":2.0539999999999998},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.88500000000000001,"15min":0.88600000000000001},"min":{"1min":0.67600000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1919999999999999,"5min":1.2270000000000001,"15min":1.2490000000000001},"last":1.069}]}]},{"osd":3,"up_from":13,"seq":55834575974,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299988,"kb_used_data":2700,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071852,"statfs":{"total":96636764160,"available":96329576448,"internally_reserved":0,"allocated":2764800,"data_stored":1312691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.502,"15min":0.52000000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.314},"max":{"1min":0.76100000000000001,"5min":0.89700000000000002,"15min":0.90000000000000002},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.56200000000000006,"15min":0.58699999999999997},"min":{"1min":0.372,"5min":0.36899999999999999,"15min":0.36099999999999999},"max":{"1min":0.67900000000000005,"5min":1.014,"15min":1.0449999999999999},"last":0.44600000000000001}]},{"osd":1,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.495,"15min":0.51100000000000001},"min":{"1min":0.29399999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63500000000000001,"5min":0.84199999999999997,"15min":1.304},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.38700000000000001},"max":{"1min":0.75,"5min":0.84099999999999997,"15min":1.4590000000000001},"last":0.47499999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58299999999999996,"15min":0.60599999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.72999999999999998,"5min":0.93700000000000006,"15min":1.04},"last":0.502},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.64200000000000002,"15min":0.65800000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44700000000000001},"max":{"1min":0.83499999999999996,"5min":0.89800000000000002,"15min":0.92000000000000004},"last":0.52700000000000002}]},{"osd":4,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76100000000000001,"15min":0.79000000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.92900000000000005,"5min":1.206,"15min":1.206},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.77800000000000002,"15min":0.81299999999999994},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.871,"5min":1.171,"15min":1.1830000000000001},"last":0.74099999999999999}]},{"osd":5,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.79600000000000004,"15min":0.82099999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.95399999999999996,"5min":1.1619999999999999,"15min":1.234},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81000000000000005,"15min":0.84799999999999998},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.3440000000000001},"last":0.83399999999999996}]},{"osd":6,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82999999999999996,"15min":0.85899999999999999},"min":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.077,"5min":1.1739999999999999,"15min":1.2170000000000001},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.80500000000000005,"15min":0.82899999999999996},"min":{"1min":0.55300000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.97099999999999997,"5min":1.181,"15min":1.2190000000000001},"last":0.96699999999999997}]},{"osd":7,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.83699999999999997,"15min":0.86799999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0269999999999999,"5min":1.224,"15min":1.264},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.86099999999999999,"15min":0.89500000000000002},"min":{"1min":0.59299999999999997,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.0209999999999999,"5min":1.1539999999999999,"15min":1.24},"last":0.93200000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575975,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308176,"kb_used_data":2696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063664,"statfs":{"total":96636764160,"available":96321191936,"internally_reserved":0,"allocated":2760704,"data_stored":1314310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.75800000000000001,"15min":0.77200000000000002},"min":{"1min":0.54000000000000004,"5min":0.41199999999999998,"15min":0.39900000000000002},"max":{"1min":0.91200000000000003,"5min":0.98499999999999999,"15min":1.119},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.98499999999999999,"5min":1.853,"15min":1.853},"last":0.72499999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.73099999999999998,"15min":0.74399999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.41399999999999998},"max":{"1min":0.90500000000000003,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.72999999999999998,"15min":0.73599999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.95899999999999996,"5min":1.639,"15min":1.639},"last":0.621}]},{"osd":2,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.747,"15min":0.751},"min":{"1min":0.54800000000000004,"5min":0.46500000000000002,"15min":0.44700000000000001},"max":{"1min":0.94499999999999995,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.75800000000000001,"15min":0.752},"min":{"1min":0.503,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.002,"5min":1.0249999999999999,"15min":1.0700000000000001},"last":0.55500000000000005}]},{"osd":3,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.73899999999999999,"15min":0.745},"min":{"1min":0.52100000000000002,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.94799999999999995,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.78300000000000003,"15min":0.78000000000000003},"min":{"1min":0.56499999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.02,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.69399999999999995}]},{"osd":5,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.64100000000000001,"15min":0.624},"min":{"1min":0.42999999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.90300000000000002,"5min":0.91500000000000004,"15min":1.0049999999999999},"last":0.503},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.66600000000000004,"15min":0.64600000000000002},"min":{"1min":0.46700000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89500000000000002,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.78600000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.69099999999999995,"15min":0.67100000000000004},"min":{"1min":0.51000000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83899999999999997,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.72699999999999998,"15min":0.70699999999999996},"min":{"1min":0.51800000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95199999999999996,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.80600000000000005}]},{"osd":7,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.72499999999999998,"15min":0.70099999999999996},"min":{"1min":0.56399999999999995,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0489999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.78000000000000003,"15min":0.754},"min":{"1min":0.57499999999999996,"5min":0.436,"15min":0.41699999999999998},"max":{"1min":1.02,"5min":2.052,"15min":2.052},"last":0.89100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575975,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312252,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059588,"statfs":{"total":96636764160,"available":96317018112,"internally_reserved":0,"allocated":2740224,"data_stored":1311457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.74099999999999999,"15min":0.73299999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":0.95099999999999996,"5min":0.97799999999999998,"15min":1.0029999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71199999999999997,"15min":0.72399999999999998},"min":{"1min":0.52000000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.91100000000000003,"5min":1.048,"15min":1.0549999999999999},"last":0.82599999999999996}]},{"osd":1,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71899999999999997,"15min":0.73499999999999999},"min":{"1min":0.48499999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.88300000000000001,"5min":0.95699999999999996,"15min":0.97699999999999998},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.74299999999999999,"15min":0.74199999999999999},"min":{"1min":0.5,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.92600000000000005,"5min":1.004,"15min":1.0269999999999999},"last":0.84999999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.72599999999999998,"15min":0.72999999999999998},"min":{"1min":0.504,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92600000000000005,"5min":1.0269999999999999,"15min":1.028},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.74299999999999999,"15min":0.746},"min":{"1min":0.53500000000000003,"5min":0.439,"15min":0.439},"max":{"1min":0.89900000000000002,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.73399999999999999}]},{"osd":3,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.74299999999999999,"15min":0.75700000000000001},"min":{"1min":0.58499999999999996,"5min":0.52800000000000002,"15min":0.49299999999999999},"max":{"1min":0.90200000000000002,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.76400000000000001,"15min":0.76900000000000002},"min":{"1min":0.56999999999999995,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.94499999999999995,"5min":1.004,"15min":1.1000000000000001},"last":0.753}]},{"osd":4,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61799999999999999,"15min":0.61099999999999999},"min":{"1min":0.443,"5min":0.41999999999999998,"15min":0.41699999999999998},"max":{"1min":0.91000000000000003,"5min":0.94699999999999995,"15min":0.97199999999999998},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.63500000000000001,"15min":0.63300000000000001},"min":{"1min":0.46999999999999997,"5min":0.40200000000000002,"15min":0.375},"max":{"1min":0.85799999999999998,"5min":0.93100000000000005,"15min":0.94499999999999995},"last":0.55600000000000005}]},{"osd":6,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63300000000000001,"15min":0.64100000000000001},"min":{"1min":0.48099999999999998,"5min":0.42099999999999999,"15min":0.40999999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":1.0089999999999999},"last":0.628},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68300000000000005,"15min":0.68700000000000006},"min":{"1min":0.47299999999999998,"5min":0.46200000000000002,"15min":0.42799999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":1.0229999999999999},"last":0.60699999999999998}]},{"osd":7,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.434},"max":{"1min":0.88600000000000001,"5min":1.0309999999999999,"15min":1.0760000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77700000000000002},"min":{"1min":0.56799999999999995,"5min":0.52700000000000002,"15min":0.47599999999999998},"max":{"1min":0.97099999999999997,"5min":1.0429999999999999,"15min":1.0469999999999999},"last":0.68000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:45:42.389 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:45:42.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.628+0000 7ff891dff700 1 Processor -- start 2024-01-11T09:45:42.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.628+0000 7ff891dff700 1 -- start start 2024-01-11T09:45:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.629+0000 7ff891dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff88c0b03e0 0x7ff88c0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.629+0000 7ff891dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff88c0b07b0 0x7ff88c0b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.629+0000 7ff891dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88c0cd730 con 0x7ff88c159200 2024-01-11T09:45:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.629+0000 7ff891dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff88c159d50 con 0x7ff88c0b07b0 2024-01-11T09:45:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.629+0000 7ff891dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0ccb20 con 0x7ff88c0b03e0 2024-01-11T09:45:42.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.630+0000 7ff88b7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff88c0b03e0 0x7ff88c0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:42.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.630+0000 7ff88affd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff88c0b07b0 0x7ff88c0b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:42.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.630+0000 7ff88b7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff88c0b03e0 0x7ff88c0b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:40344/0 (socket says 172.21.15.27:40344) 2024-01-11T09:45:42.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.630+0000 7ff88b7fe700 1 -- 172.21.15.27:0/3575678888 learned_addr learned my addr 172.21.15.27:0/3575678888 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:42.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.630+0000 7ff88affd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff88c0b07b0 0x7ff88c0b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:34358/0 (socket says 172.21.15.27:34358) 2024-01-11T09:45:42.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.630+0000 7ff88bfff700 1 --1- 172.21.15.27:0/3575678888 >> v1:172.21.15.77:6789/0 conn(0x7ff88c159200 0x7ff88c159640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:34196/0 (socket says 172.21.15.27:34196) 2024-01-11T09:45:42.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.630+0000 7ff88b7fe700 1 -- 172.21.15.27:0/3575678888 >> v2:172.21.15.77:3300/0 conn(0x7ff88c0b07b0 msgr2=0x7ff88c0b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:42.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.630+0000 7ff88b7fe700 1 --2- 172.21.15.27:0/3575678888 >> v2:172.21.15.77:3300/0 conn(0x7ff88c0b07b0 0x7ff88c0b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:42.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.630+0000 7ff88b7fe700 1 -- 172.21.15.27:0/3575678888 >> v1:172.21.15.77:6789/0 conn(0x7ff88c159200 legacy=0x7ff88c159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:42.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.630+0000 7ff88b7fe700 1 -- 172.21.15.27:0/3575678888 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88c0cce00 con 0x7ff88c0b03e0 2024-01-11T09:45:42.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.631+0000 7ff88b7fe700 1 --2- 172.21.15.27:0/3575678888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff88c0b03e0 0x7ff88c0b2480 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff87c0099b0 tx=0x7ff87c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29366e58c433dca7 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.631+0000 7ff88a7fc700 1 -- 172.21.15.27:0/3575678888 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c009ea0 con 0x7ff88c0b03e0 2024-01-11T09:45:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.631+0000 7ff88a7fc700 1 -- 172.21.15.27:0/3575678888 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87c00b860 con 0x7ff88c0b03e0 2024-01-11T09:45:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.631+0000 7ff88a7fc700 1 -- 172.21.15.27:0/3575678888 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c00bc00 con 0x7ff88c0b03e0 2024-01-11T09:45:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.631+0000 7ff891dff700 1 -- 172.21.15.27:0/3575678888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff88c0b03e0 msgr2=0x7ff88c0b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.631+0000 7ff891dff700 1 --2- 172.21.15.27:0/3575678888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff88c0b03e0 0x7ff88c0b2480 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff87c0099b0 tx=0x7ff87c009250 comp rx=0 tx=0).stop 2024-01-11T09:45:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.632+0000 7ff891dff700 1 -- 172.21.15.27:0/3575678888 shutdown_connections 2024-01-11T09:45:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.632+0000 7ff891dff700 1 --2- 172.21.15.27:0/3575678888 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff88c0b03e0 0x7ff88c0b2480 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.632+0000 7ff891dff700 1 --2- 172.21.15.27:0/3575678888 >> v2:172.21.15.77:3300/0 conn(0x7ff88c0b07b0 0x7ff88c0b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.632+0000 7ff891dff700 1 -- 172.21.15.27:0/3575678888 >> 172.21.15.27:0/3575678888 conn(0x7ff88c06ed40 msgr2=0x7ff88c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.632+0000 7ff891dff700 1 -- 172.21.15.27:0/3575678888 shutdown_connections 2024-01-11T09:45:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.632+0000 7ff891dff700 1 -- 172.21.15.27:0/3575678888 wait complete. 2024-01-11T09:45:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.634+0000 7ff891dff700 1 Processor -- start 2024-01-11T09:45:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.634+0000 7ff891dff700 1 -- start start 2024-01-11T09:45:42.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.635+0000 7ff891dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff88c0b03e0 0x7ff88c0ae760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:42.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.635+0000 7ff891dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff88c0b07b0 0x7ff88c0c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.635+0000 7ff891dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff88c159200 0x7ff88c0cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.635+0000 7ff891dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c169e90 con 0x7ff88c0b03e0 2024-01-11T09:45:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.635+0000 7ff891dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0c9910 con 0x7ff88c159200 2024-01-11T09:45:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.635+0000 7ff891dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0c9bf0 con 0x7ff88c0b07b0 2024-01-11T09:45:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.636+0000 7ff88b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff88c0b03e0 0x7ff88c0ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.636+0000 7ff88b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff88c0b03e0 0x7ff88c0ae760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47578/0 (socket says 172.21.15.27:47578) 2024-01-11T09:45:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.636+0000 7ff88b7fe700 1 -- 172.21.15.27:0/1979034141 learned_addr learned my addr 172.21.15.27:0/1979034141 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.636+0000 7ff88affd700 1 --2- 172.21.15.27:0/1979034141 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff88c0b07b0 0x7ff88c0c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.636+0000 7ff88bfff700 1 --2- 172.21.15.27:0/1979034141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff88c159200 0x7ff88c0cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.636+0000 7ff88b7fe700 1 -- 172.21.15.27:0/1979034141 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff88c0b07b0 msgr2=0x7ff88c0c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.636+0000 7ff88b7fe700 1 --2- 172.21.15.27:0/1979034141 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff88c0b07b0 0x7ff88c0c8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.636+0000 7ff88b7fe700 1 -- 172.21.15.27:0/1979034141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff88c159200 msgr2=0x7ff88c0cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.636+0000 7ff88b7fe700 1 --2- 172.21.15.27:0/1979034141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff88c159200 0x7ff88c0cefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.636+0000 7ff88b7fe700 1 -- 172.21.15.27:0/1979034141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c0094f0 con 0x7ff88c0b03e0 2024-01-11T09:45:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.636+0000 7ff88bfff700 1 --2- 172.21.15.27:0/1979034141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff88c159200 0x7ff88c0cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.636+0000 7ff88b7fe700 1 --2- 172.21.15.27:0/1979034141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff88c0b03e0 0x7ff88c0ae760 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7ff88c06fd20 tx=0x7ff87c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.637+0000 7ff8737fe700 1 -- 172.21.15.27:0/1979034141 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c00bbe0 con 0x7ff88c0b03e0 2024-01-11T09:45:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.637+0000 7ff8737fe700 1 -- 172.21.15.27:0/1979034141 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87c011420 con 0x7ff88c0b03e0 2024-01-11T09:45:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.637+0000 7ff8737fe700 1 -- 172.21.15.27:0/1979034141 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c0117c0 con 0x7ff88c0b03e0 2024-01-11T09:45:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.637+0000 7ff891dff700 1 -- 172.21.15.27:0/1979034141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff88c0c9ed0 con 0x7ff88c0b03e0 2024-01-11T09:45:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.637+0000 7ff891dff700 1 -- 172.21.15.27:0/1979034141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff88c0aeca0 con 0x7ff88c0b03e0 2024-01-11T09:45:42.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.638+0000 7ff891dff700 1 -- 172.21.15.27:0/1979034141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88c0718d0 con 0x7ff88c0b03e0 2024-01-11T09:45:42.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.639+0000 7ff8737fe700 1 -- 172.21.15.27:0/1979034141 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff87c011bf0 con 0x7ff88c0b03e0 2024-01-11T09:45:42.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.639+0000 7ff8737fe700 1 --2- 172.21.15.27:0/1979034141 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff878066f30 0x7ff8780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:42.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.639+0000 7ff8737fe700 1 -- 172.21.15.27:0/1979034141 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6681+0+0 (secure 0 0 0) 0x7ff87c086330 con 0x7ff88c0b03e0 2024-01-11T09:45:42.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.640+0000 7ff88affd700 1 --2- 172.21.15.27:0/1979034141 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff878066f30 0x7ff8780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:42.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.641+0000 7ff88affd700 1 --2- 172.21.15.27:0/1979034141 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff878066f30 0x7ff8780693e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff88c0b5860 tx=0x7ff880008040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:42.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.644+0000 7ff8737fe700 1 -- 172.21.15.27:0/1979034141 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff87c053130 con 0x7ff88c0b03e0 2024-01-11T09:45:42.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.846+0000 7ff891dff700 1 -- 172.21.15.27:0/1979034141 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff88c0cd0e0 con 0x7ff878066f30 2024-01-11T09:45:42.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.861+0000 7ff8737fe700 1 -- 172.21.15.27:0/1979034141 <== mgr.10380 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528648 (secure 0 0 0) 0x7ff88c0cd0e0 con 0x7ff878066f30 2024-01-11T09:45:42.860 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:42.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.867+0000 7ff891dff700 1 -- 172.21.15.27:0/1979034141 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff878066f30 msgr2=0x7ff8780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:42.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.867+0000 7ff891dff700 1 --2- 172.21.15.27:0/1979034141 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff878066f30 0x7ff8780693e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff88c0b5860 tx=0x7ff880008040 comp rx=0 tx=0).stop 2024-01-11T09:45:42.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.867+0000 7ff891dff700 1 -- 172.21.15.27:0/1979034141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff88c0b03e0 msgr2=0x7ff88c0ae760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:42.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.867+0000 7ff891dff700 1 --2- 172.21.15.27:0/1979034141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff88c0b03e0 0x7ff88c0ae760 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7ff88c06fd20 tx=0x7ff87c017000 comp rx=0 tx=0).stop 2024-01-11T09:45:42.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.867+0000 7ff891dff700 1 -- 172.21.15.27:0/1979034141 shutdown_connections 2024-01-11T09:45:42.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.867+0000 7ff891dff700 1 --2- 172.21.15.27:0/1979034141 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff878066f30 0x7ff8780693e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:42.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.867+0000 7ff891dff700 1 --2- 172.21.15.27:0/1979034141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff88c159200 0x7ff88c0cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:42.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.867+0000 7ff891dff700 1 --2- 172.21.15.27:0/1979034141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff88c0b03e0 0x7ff88c0ae760 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:42.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.867+0000 7ff891dff700 1 --2- 172.21.15.27:0/1979034141 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff88c0b07b0 0x7ff88c0c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:42.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.867+0000 7ff891dff700 1 -- 172.21.15.27:0/1979034141 >> 172.21.15.27:0/1979034141 conn(0x7ff88c06ed40 msgr2=0x7ff88c0b2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:42.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.867+0000 7ff891dff700 1 -- 172.21.15.27:0/1979034141 shutdown_connections 2024-01-11T09:45:42.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:42.868+0000 7ff891dff700 1 -- 172.21.15.27:0/1979034141 wait complete. 2024-01-11T09:45:42.865 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:45:42.877 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-01-11T09:45:42.304619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596193,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":702,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2462408,"kb_used_data":22664,"kb_used_omap":0,"kb_used_meta":2439680,"kb_avail":752512312,"statfs":{"total":773094113280,"available":770572607488,"internally_reserved":0,"allocated":23207936,"data_stored":11679610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2498232320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001482"},"pg_stats":[{"pgid":"1.0","version":"205'366","reported_seq":1203,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954188+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:45:37.954188+0000","last_peered":"2024-01-11T09:45:37.954188+0000","last_clean":"2024-01-11T09:45:37.954188+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:45:37.954188+0000","last_undegraded":"2024-01-11T09:45:37.954188+0000","last_fullsized":"2024-01-11T09:45:37.954188+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":611,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"213'1","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.973258+0000","last_change":"2024-01-11T09:45:23.147910+0000","last_active":"2024-01-11T09:45:37.973258+0000","last_peered":"2024-01-11T09:45:37.973258+0000","last_clean":"2024-01-11T09:45:37.973258+0000","last_became_active":"2024-01-11T09:45:23.147626+0000","last_became_peered":"2024-01-11T09:45:23.147626+0000","last_unstale":"2024-01-11T09:45:37.973258+0000","last_undegraded":"2024-01-11T09:45:37.973258+0000","last_fullsized":"2024-01-11T09:45:37.973258+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:32:18.360961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.956527+0000","last_change":"2024-01-11T09:45:24.169860+0000","last_active":"2024-01-11T09:45:37.956527+0000","last_peered":"2024-01-11T09:45:37.956527+0000","last_clean":"2024-01-11T09:45:37.956527+0000","last_became_active":"2024-01-11T09:45:24.169370+0000","last_became_peered":"2024-01-11T09:45:24.169370+0000","last_unstale":"2024-01-11T09:45:37.956527+0000","last_undegraded":"2024-01-11T09:45:37.956527+0000","last_fullsized":"2024-01-11T09:45:37.956527+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:15:38.555186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"217'2","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.983247+0000","last_change":"2024-01-11T09:45:36.976365+0000","last_active":"2024-01-11T09:45:37.983247+0000","last_peered":"2024-01-11T09:45:37.983247+0000","last_clean":"2024-01-11T09:45:37.983247+0000","last_became_active":"2024-01-11T09:45:36.976041+0000","last_became_peered":"2024-01-11T09:45:36.976041+0000","last_unstale":"2024-01-11T09:45:37.983247+0000","last_undegraded":"2024-01-11T09:45:37.983247+0000","last_fullsized":"2024-01-11T09:45:37.983247+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:40.586757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.972306+0000","last_change":"2024-01-11T09:45:37.968694+0000","last_active":"2024-01-11T09:45:37.972306+0000","last_peered":"2024-01-11T09:45:37.972306+0000","last_clean":"2024-01-11T09:45:37.972306+0000","last_became_active":"2024-01-11T09:45:37.966845+0000","last_became_peered":"2024-01-11T09:45:37.966845+0000","last_unstale":"2024-01-11T09:45:37.972306+0000","last_undegraded":"2024-01-11T09:45:37.972306+0000","last_fullsized":"2024-01-11T09:45:37.972306+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:47:37.322693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.969312+0000","last_change":"2024-01-11T09:45:23.158577+0000","last_active":"2024-01-11T09:45:37.969312+0000","last_peered":"2024-01-11T09:45:37.969312+0000","last_clean":"2024-01-11T09:45:37.969312+0000","last_became_active":"2024-01-11T09:45:23.158304+0000","last_became_peered":"2024-01-11T09:45:23.158304+0000","last_unstale":"2024-01-11T09:45:37.969312+0000","last_undegraded":"2024-01-11T09:45:37.969312+0000","last_fullsized":"2024-01-11T09:45:37.969312+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:59:55.124493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.948096+0000","last_change":"2024-01-11T09:45:24.153383+0000","last_active":"2024-01-11T09:45:37.948096+0000","last_peered":"2024-01-11T09:45:37.948096+0000","last_clean":"2024-01-11T09:45:37.948096+0000","last_became_active":"2024-01-11T09:45:24.153089+0000","last_became_peered":"2024-01-11T09:45:24.153089+0000","last_unstale":"2024-01-11T09:45:37.948096+0000","last_undegraded":"2024-01-11T09:45:37.948096+0000","last_fullsized":"2024-01-11T09:45:37.948096+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:49:15.804305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954712+0000","last_change":"2024-01-11T09:45:36.973450+0000","last_active":"2024-01-11T09:45:37.954712+0000","last_peered":"2024-01-11T09:45:37.954712+0000","last_clean":"2024-01-11T09:45:37.954712+0000","last_became_active":"2024-01-11T09:45:36.973089+0000","last_became_peered":"2024-01-11T09:45:36.973089+0000","last_unstale":"2024-01-11T09:45:37.954712+0000","last_undegraded":"2024-01-11T09:45:37.954712+0000","last_fullsized":"2024-01-11T09:45:37.954712+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:39:18.270597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.986808+0000","last_change":"2024-01-11T09:45:37.983052+0000","last_active":"2024-01-11T09:45:37.986808+0000","last_peered":"2024-01-11T09:45:37.986808+0000","last_clean":"2024-01-11T09:45:37.986808+0000","last_became_active":"2024-01-11T09:45:37.982783+0000","last_became_peered":"2024-01-11T09:45:37.982783+0000","last_unstale":"2024-01-11T09:45:37.986808+0000","last_undegraded":"2024-01-11T09:45:37.986808+0000","last_fullsized":"2024-01-11T09:45:37.986808+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:13:51.430167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949982+0000","last_change":"2024-01-11T09:45:23.154043+0000","last_active":"2024-01-11T09:45:37.949982+0000","last_peered":"2024-01-11T09:45:37.949982+0000","last_clean":"2024-01-11T09:45:37.949982+0000","last_became_active":"2024-01-11T09:45:23.153529+0000","last_became_peered":"2024-01-11T09:45:23.153529+0000","last_unstale":"2024-01-11T09:45:37.949982+0000","last_undegraded":"2024-01-11T09:45:37.949982+0000","last_fullsized":"2024-01-11T09:45:37.949982+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:48.360444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.955840+0000","last_change":"2024-01-11T09:45:24.163398+0000","last_active":"2024-01-11T09:45:37.955840+0000","last_peered":"2024-01-11T09:45:37.955840+0000","last_clean":"2024-01-11T09:45:37.955840+0000","last_became_active":"2024-01-11T09:45:24.159101+0000","last_became_peered":"2024-01-11T09:45:24.159101+0000","last_unstale":"2024-01-11T09:45:37.955840+0000","last_undegraded":"2024-01-11T09:45:37.955840+0000","last_fullsized":"2024-01-11T09:45:37.955840+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:40.983480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"217'2","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.988007+0000","last_change":"2024-01-11T09:45:36.974931+0000","last_active":"2024-01-11T09:45:37.988007+0000","last_peered":"2024-01-11T09:45:37.988007+0000","last_clean":"2024-01-11T09:45:37.988007+0000","last_became_active":"2024-01-11T09:45:36.974149+0000","last_became_peered":"2024-01-11T09:45:36.974149+0000","last_unstale":"2024-01-11T09:45:37.988007+0000","last_undegraded":"2024-01-11T09:45:37.988007+0000","last_fullsized":"2024-01-11T09:45:37.988007+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:15:49.588159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.978949+0000","last_change":"2024-01-11T09:45:37.968718+0000","last_active":"2024-01-11T09:45:37.978949+0000","last_peered":"2024-01-11T09:45:37.978949+0000","last_clean":"2024-01-11T09:45:37.978949+0000","last_became_active":"2024-01-11T09:45:37.968431+0000","last_became_peered":"2024-01-11T09:45:37.968431+0000","last_unstale":"2024-01-11T09:45:37.978949+0000","last_undegraded":"2024-01-11T09:45:37.978949+0000","last_fullsized":"2024-01-11T09:45:37.978949+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:17:38.232406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.958199+0000","last_change":"2024-01-11T09:45:23.151649+0000","last_active":"2024-01-11T09:45:37.958199+0000","last_peered":"2024-01-11T09:45:37.958199+0000","last_clean":"2024-01-11T09:45:37.958199+0000","last_became_active":"2024-01-11T09:45:23.151399+0000","last_became_peered":"2024-01-11T09:45:23.151399+0000","last_unstale":"2024-01-11T09:45:37.958199+0000","last_undegraded":"2024-01-11T09:45:37.958199+0000","last_fullsized":"2024-01-11T09:45:37.958199+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:02.352990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.967480+0000","last_change":"2024-01-11T09:45:24.157004+0000","last_active":"2024-01-11T09:45:37.967480+0000","last_peered":"2024-01-11T09:45:37.967480+0000","last_clean":"2024-01-11T09:45:37.967480+0000","last_became_active":"2024-01-11T09:45:24.156446+0000","last_became_peered":"2024-01-11T09:45:24.156446+0000","last_unstale":"2024-01-11T09:45:37.967480+0000","last_undegraded":"2024-01-11T09:45:37.967480+0000","last_fullsized":"2024-01-11T09:45:37.967480+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:28:18.397764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.972338+0000","last_change":"2024-01-11T09:45:36.971222+0000","last_active":"2024-01-11T09:45:37.972338+0000","last_peered":"2024-01-11T09:45:37.972338+0000","last_clean":"2024-01-11T09:45:37.972338+0000","last_became_active":"2024-01-11T09:45:36.970967+0000","last_became_peered":"2024-01-11T09:45:36.970967+0000","last_unstale":"2024-01-11T09:45:37.972338+0000","last_undegraded":"2024-01-11T09:45:37.972338+0000","last_fullsized":"2024-01-11T09:45:37.972338+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:55.529264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.983024+0000","last_change":"2024-01-11T09:45:37.974201+0000","last_active":"2024-01-11T09:45:37.983024+0000","last_peered":"2024-01-11T09:45:37.983024+0000","last_clean":"2024-01-11T09:45:37.983024+0000","last_became_active":"2024-01-11T09:45:37.973939+0000","last_became_peered":"2024-01-11T09:45:37.973939+0000","last_unstale":"2024-01-11T09:45:37.983024+0000","last_undegraded":"2024-01-11T09:45:37.983024+0000","last_fullsized":"2024-01-11T09:45:37.983024+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:37:34.036345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954500+0000","last_change":"2024-01-11T09:45:23.151324+0000","last_active":"2024-01-11T09:45:37.954500+0000","last_peered":"2024-01-11T09:45:37.954500+0000","last_clean":"2024-01-11T09:45:37.954500+0000","last_became_active":"2024-01-11T09:45:23.150822+0000","last_became_peered":"2024-01-11T09:45:23.150822+0000","last_unstale":"2024-01-11T09:45:37.954500+0000","last_undegraded":"2024-01-11T09:45:37.954500+0000","last_fullsized":"2024-01-11T09:45:37.954500+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:39.975619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.956687+0000","last_change":"2024-01-11T09:45:24.158240+0000","last_active":"2024-01-11T09:45:37.956687+0000","last_peered":"2024-01-11T09:45:37.956687+0000","last_clean":"2024-01-11T09:45:37.956687+0000","last_became_active":"2024-01-11T09:45:24.157938+0000","last_became_peered":"2024-01-11T09:45:24.157938+0000","last_unstale":"2024-01-11T09:45:37.956687+0000","last_undegraded":"2024-01-11T09:45:37.956687+0000","last_fullsized":"2024-01-11T09:45:37.956687+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:08:44.293978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.955218+0000","last_change":"2024-01-11T09:45:36.974347+0000","last_active":"2024-01-11T09:45:37.955218+0000","last_peered":"2024-01-11T09:45:37.955218+0000","last_clean":"2024-01-11T09:45:37.955218+0000","last_became_active":"2024-01-11T09:45:36.973974+0000","last_became_peered":"2024-01-11T09:45:36.973974+0000","last_unstale":"2024-01-11T09:45:37.955218+0000","last_undegraded":"2024-01-11T09:45:37.955218+0000","last_fullsized":"2024-01-11T09:45:37.955218+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:06:58.345396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984617+0000","last_change":"2024-01-11T09:45:37.976200+0000","last_active":"2024-01-11T09:45:37.984617+0000","last_peered":"2024-01-11T09:45:37.984617+0000","last_clean":"2024-01-11T09:45:37.984617+0000","last_became_active":"2024-01-11T09:45:37.975890+0000","last_became_peered":"2024-01-11T09:45:37.975890+0000","last_unstale":"2024-01-11T09:45:37.984617+0000","last_undegraded":"2024-01-11T09:45:37.984617+0000","last_fullsized":"2024-01-11T09:45:37.984617+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:58:38.459913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960857+0000","last_change":"2024-01-11T09:45:23.147642+0000","last_active":"2024-01-11T09:45:37.960857+0000","last_peered":"2024-01-11T09:45:37.960857+0000","last_clean":"2024-01-11T09:45:37.960857+0000","last_became_active":"2024-01-11T09:45:23.147387+0000","last_became_peered":"2024-01-11T09:45:23.147387+0000","last_unstale":"2024-01-11T09:45:37.960857+0000","last_undegraded":"2024-01-11T09:45:37.960857+0000","last_fullsized":"2024-01-11T09:45:37.960857+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:49:33.086505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950588+0000","last_change":"2024-01-11T09:45:24.154386+0000","last_active":"2024-01-11T09:45:37.950588+0000","last_peered":"2024-01-11T09:45:37.950588+0000","last_clean":"2024-01-11T09:45:37.950588+0000","last_became_active":"2024-01-11T09:45:24.154017+0000","last_became_peered":"2024-01-11T09:45:24.154017+0000","last_unstale":"2024-01-11T09:45:37.950588+0000","last_undegraded":"2024-01-11T09:45:37.950588+0000","last_fullsized":"2024-01-11T09:45:37.950588+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:33:41.503661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.956518+0000","last_change":"2024-01-11T09:45:36.973464+0000","last_active":"2024-01-11T09:45:37.956518+0000","last_peered":"2024-01-11T09:45:37.956518+0000","last_clean":"2024-01-11T09:45:37.956518+0000","last_became_active":"2024-01-11T09:45:36.973216+0000","last_became_peered":"2024-01-11T09:45:36.973216+0000","last_unstale":"2024-01-11T09:45:37.956518+0000","last_undegraded":"2024-01-11T09:45:37.956518+0000","last_fullsized":"2024-01-11T09:45:37.956518+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:57:54.872386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.985401+0000","last_change":"2024-01-11T09:45:37.978430+0000","last_active":"2024-01-11T09:45:37.985401+0000","last_peered":"2024-01-11T09:45:37.985401+0000","last_clean":"2024-01-11T09:45:37.985401+0000","last_became_active":"2024-01-11T09:45:37.977974+0000","last_became_peered":"2024-01-11T09:45:37.977974+0000","last_unstale":"2024-01-11T09:45:37.985401+0000","last_undegraded":"2024-01-11T09:45:37.985401+0000","last_fullsized":"2024-01-11T09:45:37.985401+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:57:55.878951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.967642+0000","last_change":"2024-01-11T09:45:23.148631+0000","last_active":"2024-01-11T09:45:37.967642+0000","last_peered":"2024-01-11T09:45:37.967642+0000","last_clean":"2024-01-11T09:45:37.967642+0000","last_became_active":"2024-01-11T09:45:23.147091+0000","last_became_peered":"2024-01-11T09:45:23.147091+0000","last_unstale":"2024-01-11T09:45:37.967642+0000","last_undegraded":"2024-01-11T09:45:37.967642+0000","last_fullsized":"2024-01-11T09:45:37.967642+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:49:14.796444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949340+0000","last_change":"2024-01-11T09:45:24.157902+0000","last_active":"2024-01-11T09:45:37.949340+0000","last_peered":"2024-01-11T09:45:37.949340+0000","last_clean":"2024-01-11T09:45:37.949340+0000","last_became_active":"2024-01-11T09:45:24.157443+0000","last_became_peered":"2024-01-11T09:45:24.157443+0000","last_unstale":"2024-01-11T09:45:37.949340+0000","last_undegraded":"2024-01-11T09:45:37.949340+0000","last_fullsized":"2024-01-11T09:45:37.949340+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:41:10.078741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949378+0000","last_change":"2024-01-11T09:45:36.967374+0000","last_active":"2024-01-11T09:45:37.949378+0000","last_peered":"2024-01-11T09:45:37.949378+0000","last_clean":"2024-01-11T09:45:37.949378+0000","last_became_active":"2024-01-11T09:45:36.967001+0000","last_became_peered":"2024-01-11T09:45:36.967001+0000","last_unstale":"2024-01-11T09:45:37.949378+0000","last_undegraded":"2024-01-11T09:45:37.949378+0000","last_fullsized":"2024-01-11T09:45:37.949378+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:42:34.355866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.972812+0000","last_change":"2024-01-11T09:45:37.964730+0000","last_active":"2024-01-11T09:45:37.972812+0000","last_peered":"2024-01-11T09:45:37.972812+0000","last_clean":"2024-01-11T09:45:37.972812+0000","last_became_active":"2024-01-11T09:45:37.964406+0000","last_became_peered":"2024-01-11T09:45:37.964406+0000","last_unstale":"2024-01-11T09:45:37.972812+0000","last_undegraded":"2024-01-11T09:45:37.972812+0000","last_fullsized":"2024-01-11T09:45:37.972812+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:29:35.726697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.6","version":"214'1","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954993+0000","last_change":"2024-01-11T09:45:23.149174+0000","last_active":"2024-01-11T09:45:37.954993+0000","last_peered":"2024-01-11T09:45:37.954993+0000","last_clean":"2024-01-11T09:45:37.954993+0000","last_became_active":"2024-01-11T09:45:23.148798+0000","last_became_peered":"2024-01-11T09:45:23.148798+0000","last_unstale":"2024-01-11T09:45:37.954993+0000","last_undegraded":"2024-01-11T09:45:37.954993+0000","last_fullsized":"2024-01-11T09:45:37.954993+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:52:31.488984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.963297+0000","last_change":"2024-01-11T09:45:24.162504+0000","last_active":"2024-01-11T09:45:37.963297+0000","last_peered":"2024-01-11T09:45:37.963297+0000","last_clean":"2024-01-11T09:45:37.963297+0000","last_became_active":"2024-01-11T09:45:24.162179+0000","last_became_peered":"2024-01-11T09:45:24.162179+0000","last_unstale":"2024-01-11T09:45:37.963297+0000","last_undegraded":"2024-01-11T09:45:37.963297+0000","last_fullsized":"2024-01-11T09:45:37.963297+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:47:23.503381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.966665+0000","last_change":"2024-01-11T09:45:36.973086+0000","last_active":"2024-01-11T09:45:37.966665+0000","last_peered":"2024-01-11T09:45:37.966665+0000","last_clean":"2024-01-11T09:45:37.966665+0000","last_became_active":"2024-01-11T09:45:36.972786+0000","last_became_peered":"2024-01-11T09:45:36.972786+0000","last_unstale":"2024-01-11T09:45:37.966665+0000","last_undegraded":"2024-01-11T09:45:37.966665+0000","last_fullsized":"2024-01-11T09:45:37.966665+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:47:36.316129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.972164+0000","last_change":"2024-01-11T09:45:37.965771+0000","last_active":"2024-01-11T09:45:37.972164+0000","last_peered":"2024-01-11T09:45:37.972164+0000","last_clean":"2024-01-11T09:45:37.972164+0000","last_became_active":"2024-01-11T09:45:37.965461+0000","last_became_peered":"2024-01-11T09:45:37.965461+0000","last_unstale":"2024-01-11T09:45:37.972164+0000","last_undegraded":"2024-01-11T09:45:37.972164+0000","last_fullsized":"2024-01-11T09:45:37.972164+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:29:35.726697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.7","version":"213'2","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960712+0000","last_change":"2024-01-11T09:45:23.148348+0000","last_active":"2024-01-11T09:45:37.960712+0000","last_peered":"2024-01-11T09:45:37.960712+0000","last_clean":"2024-01-11T09:45:37.960712+0000","last_became_active":"2024-01-11T09:45:23.147723+0000","last_became_peered":"2024-01-11T09:45:23.147723+0000","last_unstale":"2024-01-11T09:45:37.960712+0000","last_undegraded":"2024-01-11T09:45:37.960712+0000","last_fullsized":"2024-01-11T09:45:37.960712+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:48:39.924839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.971682+0000","last_change":"2024-01-11T09:45:24.149162+0000","last_active":"2024-01-11T09:45:37.971682+0000","last_peered":"2024-01-11T09:45:37.971682+0000","last_clean":"2024-01-11T09:45:37.971682+0000","last_became_active":"2024-01-11T09:45:24.148510+0000","last_became_peered":"2024-01-11T09:45:24.148510+0000","last_unstale":"2024-01-11T09:45:37.971682+0000","last_undegraded":"2024-01-11T09:45:37.971682+0000","last_fullsized":"2024-01-11T09:45:37.971682+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:56:51.623335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.955067+0000","last_change":"2024-01-11T09:45:36.972976+0000","last_active":"2024-01-11T09:45:37.955067+0000","last_peered":"2024-01-11T09:45:37.955067+0000","last_clean":"2024-01-11T09:45:37.955067+0000","last_became_active":"2024-01-11T09:45:36.972185+0000","last_became_peered":"2024-01-11T09:45:36.972185+0000","last_unstale":"2024-01-11T09:45:37.955067+0000","last_undegraded":"2024-01-11T09:45:37.955067+0000","last_fullsized":"2024-01-11T09:45:37.955067+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:52:51.394224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.989010+0000","last_change":"2024-01-11T09:45:37.986910+0000","last_active":"2024-01-11T09:45:37.989010+0000","last_peered":"2024-01-11T09:45:37.989010+0000","last_clean":"2024-01-11T09:45:37.989010+0000","last_became_active":"2024-01-11T09:45:37.986238+0000","last_became_peered":"2024-01-11T09:45:37.986238+0000","last_unstale":"2024-01-11T09:45:37.989010+0000","last_undegraded":"2024-01-11T09:45:37.989010+0000","last_fullsized":"2024-01-11T09:45:37.989010+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:39:29.824585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.961220+0000","last_change":"2024-01-11T09:45:23.154230+0000","last_active":"2024-01-11T09:45:37.961220+0000","last_peered":"2024-01-11T09:45:37.961220+0000","last_clean":"2024-01-11T09:45:37.961220+0000","last_became_active":"2024-01-11T09:45:23.153324+0000","last_became_peered":"2024-01-11T09:45:23.153324+0000","last_unstale":"2024-01-11T09:45:37.961220+0000","last_undegraded":"2024-01-11T09:45:37.961220+0000","last_fullsized":"2024-01-11T09:45:37.961220+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:52:16.865748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950176+0000","last_change":"2024-01-11T09:45:24.150105+0000","last_active":"2024-01-11T09:45:37.950176+0000","last_peered":"2024-01-11T09:45:37.950176+0000","last_clean":"2024-01-11T09:45:37.950176+0000","last_became_active":"2024-01-11T09:45:24.149005+0000","last_became_peered":"2024-01-11T09:45:24.149005+0000","last_unstale":"2024-01-11T09:45:37.950176+0000","last_undegraded":"2024-01-11T09:45:37.950176+0000","last_fullsized":"2024-01-11T09:45:37.950176+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:25:27.316660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"217'2","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.985232+0000","last_change":"2024-01-11T09:45:36.972899+0000","last_active":"2024-01-11T09:45:37.985232+0000","last_peered":"2024-01-11T09:45:37.985232+0000","last_clean":"2024-01-11T09:45:37.985232+0000","last_became_active":"2024-01-11T09:45:36.972632+0000","last_became_peered":"2024-01-11T09:45:36.972632+0000","last_unstale":"2024-01-11T09:45:37.985232+0000","last_undegraded":"2024-01-11T09:45:37.985232+0000","last_fullsized":"2024-01-11T09:45:37.985232+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:30:48.523200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.978604+0000","last_change":"2024-01-11T09:45:37.973953+0000","last_active":"2024-01-11T09:45:37.978604+0000","last_peered":"2024-01-11T09:45:37.978604+0000","last_clean":"2024-01-11T09:45:37.978604+0000","last_became_active":"2024-01-11T09:45:37.973677+0000","last_became_peered":"2024-01-11T09:45:37.973677+0000","last_unstale":"2024-01-11T09:45:37.978604+0000","last_undegraded":"2024-01-11T09:45:37.978604+0000","last_fullsized":"2024-01-11T09:45:37.978604+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:04:16.164163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.958878+0000","last_change":"2024-01-11T09:45:23.155951+0000","last_active":"2024-01-11T09:45:37.958878+0000","last_peered":"2024-01-11T09:45:37.958878+0000","last_clean":"2024-01-11T09:45:37.958878+0000","last_became_active":"2024-01-11T09:45:23.155673+0000","last_became_peered":"2024-01-11T09:45:23.155673+0000","last_unstale":"2024-01-11T09:45:37.958878+0000","last_undegraded":"2024-01-11T09:45:37.958878+0000","last_fullsized":"2024-01-11T09:45:37.958878+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:52:37.573615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.947596+0000","last_change":"2024-01-11T09:45:24.144965+0000","last_active":"2024-01-11T09:45:37.947596+0000","last_peered":"2024-01-11T09:45:37.947596+0000","last_clean":"2024-01-11T09:45:37.947596+0000","last_became_active":"2024-01-11T09:45:24.143991+0000","last_became_peered":"2024-01-11T09:45:24.143991+0000","last_unstale":"2024-01-11T09:45:37.947596+0000","last_undegraded":"2024-01-11T09:45:37.947596+0000","last_fullsized":"2024-01-11T09:45:37.947596+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:56:42.734248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"217'1","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.986143+0000","last_change":"2024-01-11T09:45:36.974662+0000","last_active":"2024-01-11T09:45:37.986143+0000","last_peered":"2024-01-11T09:45:37.986143+0000","last_clean":"2024-01-11T09:45:37.986143+0000","last_became_active":"2024-01-11T09:45:36.974322+0000","last_became_peered":"2024-01-11T09:45:36.974322+0000","last_unstale":"2024-01-11T09:45:37.986143+0000","last_undegraded":"2024-01-11T09:45:37.986143+0000","last_fullsized":"2024-01-11T09:45:37.986143+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:04:54.345995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984029+0000","last_change":"2024-01-11T09:45:37.977839+0000","last_active":"2024-01-11T09:45:37.984029+0000","last_peered":"2024-01-11T09:45:37.984029+0000","last_clean":"2024-01-11T09:45:37.984029+0000","last_became_active":"2024-01-11T09:45:37.977406+0000","last_became_peered":"2024-01-11T09:45:37.977406+0000","last_unstale":"2024-01-11T09:45:37.984029+0000","last_undegraded":"2024-01-11T09:45:37.984029+0000","last_fullsized":"2024-01-11T09:45:37.984029+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:10:33.893904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.956444+0000","last_change":"2024-01-11T09:45:23.154111+0000","last_active":"2024-01-11T09:45:37.956444+0000","last_peered":"2024-01-11T09:45:37.956444+0000","last_clean":"2024-01-11T09:45:37.956444+0000","last_became_active":"2024-01-11T09:45:23.151742+0000","last_became_peered":"2024-01-11T09:45:23.151742+0000","last_unstale":"2024-01-11T09:45:37.956444+0000","last_undegraded":"2024-01-11T09:45:37.956444+0000","last_fullsized":"2024-01-11T09:45:37.956444+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:38.398157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960261+0000","last_change":"2024-01-11T09:45:24.168028+0000","last_active":"2024-01-11T09:45:37.960261+0000","last_peered":"2024-01-11T09:45:37.960261+0000","last_clean":"2024-01-11T09:45:37.960261+0000","last_became_active":"2024-01-11T09:45:24.167726+0000","last_became_peered":"2024-01-11T09:45:24.167726+0000","last_unstale":"2024-01-11T09:45:37.960261+0000","last_undegraded":"2024-01-11T09:45:37.960261+0000","last_fullsized":"2024-01-11T09:45:37.960261+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:48:06.986946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949293+0000","last_change":"2024-01-11T09:45:36.973677+0000","last_active":"2024-01-11T09:45:37.949293+0000","last_peered":"2024-01-11T09:45:37.949293+0000","last_clean":"2024-01-11T09:45:37.949293+0000","last_became_active":"2024-01-11T09:45:36.971802+0000","last_became_peered":"2024-01-11T09:45:36.971802+0000","last_unstale":"2024-01-11T09:45:37.949293+0000","last_undegraded":"2024-01-11T09:45:37.949293+0000","last_fullsized":"2024-01-11T09:45:37.949293+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:04:54.345995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.983824+0000","last_change":"2024-01-11T09:45:37.975972+0000","last_active":"2024-01-11T09:45:37.983824+0000","last_peered":"2024-01-11T09:45:37.983824+0000","last_clean":"2024-01-11T09:45:37.983824+0000","last_became_active":"2024-01-11T09:45:37.975672+0000","last_became_peered":"2024-01-11T09:45:37.975672+0000","last_unstale":"2024-01-11T09:45:37.983824+0000","last_undegraded":"2024-01-11T09:45:37.983824+0000","last_fullsized":"2024-01-11T09:45:37.983824+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:23:48.270235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"213'2","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949253+0000","last_change":"2024-01-11T09:45:23.143823+0000","last_active":"2024-01-11T09:45:37.949253+0000","last_peered":"2024-01-11T09:45:37.949253+0000","last_clean":"2024-01-11T09:45:37.949253+0000","last_became_active":"2024-01-11T09:45:23.141580+0000","last_became_peered":"2024-01-11T09:45:23.141580+0000","last_unstale":"2024-01-11T09:45:37.949253+0000","last_undegraded":"2024-01-11T09:45:37.949253+0000","last_fullsized":"2024-01-11T09:45:37.949253+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:19.210181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949363+0000","last_change":"2024-01-11T09:45:24.145805+0000","last_active":"2024-01-11T09:45:37.949363+0000","last_peered":"2024-01-11T09:45:37.949363+0000","last_clean":"2024-01-11T09:45:37.949363+0000","last_became_active":"2024-01-11T09:45:24.145497+0000","last_became_peered":"2024-01-11T09:45:24.145497+0000","last_unstale":"2024-01-11T09:45:37.949363+0000","last_undegraded":"2024-01-11T09:45:37.949363+0000","last_fullsized":"2024-01-11T09:45:37.949363+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:27.774010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"217'1","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.982488+0000","last_change":"2024-01-11T09:45:36.977294+0000","last_active":"2024-01-11T09:45:37.982488+0000","last_peered":"2024-01-11T09:45:37.982488+0000","last_clean":"2024-01-11T09:45:37.982488+0000","last_became_active":"2024-01-11T09:45:36.976307+0000","last_became_peered":"2024-01-11T09:45:36.976307+0000","last_unstale":"2024-01-11T09:45:37.982488+0000","last_undegraded":"2024-01-11T09:45:37.982488+0000","last_fullsized":"2024-01-11T09:45:37.982488+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:08:59.135044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.983796+0000","last_change":"2024-01-11T09:45:37.975856+0000","last_active":"2024-01-11T09:45:37.983796+0000","last_peered":"2024-01-11T09:45:37.983796+0000","last_clean":"2024-01-11T09:45:37.983796+0000","last_became_active":"2024-01-11T09:45:37.975580+0000","last_became_peered":"2024-01-11T09:45:37.975580+0000","last_unstale":"2024-01-11T09:45:37.983796+0000","last_undegraded":"2024-01-11T09:45:37.983796+0000","last_fullsized":"2024-01-11T09:45:37.983796+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:47:32.728151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960674+0000","last_change":"2024-01-11T09:45:23.155273+0000","last_active":"2024-01-11T09:45:37.960674+0000","last_peered":"2024-01-11T09:45:37.960674+0000","last_clean":"2024-01-11T09:45:37.960674+0000","last_became_active":"2024-01-11T09:45:23.155023+0000","last_became_peered":"2024-01-11T09:45:23.155023+0000","last_unstale":"2024-01-11T09:45:37.960674+0000","last_undegraded":"2024-01-11T09:45:37.960674+0000","last_fullsized":"2024-01-11T09:45:37.960674+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:03:59.152882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950637+0000","last_change":"2024-01-11T09:45:24.149619+0000","last_active":"2024-01-11T09:45:37.950637+0000","last_peered":"2024-01-11T09:45:37.950637+0000","last_clean":"2024-01-11T09:45:37.950637+0000","last_became_active":"2024-01-11T09:45:24.148547+0000","last_became_peered":"2024-01-11T09:45:24.148547+0000","last_unstale":"2024-01-11T09:45:37.950637+0000","last_undegraded":"2024-01-11T09:45:37.950637+0000","last_fullsized":"2024-01-11T09:45:37.950637+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:28:18.397764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949238+0000","last_change":"2024-01-11T09:45:36.966307+0000","last_active":"2024-01-11T09:45:37.949238+0000","last_peered":"2024-01-11T09:45:37.949238+0000","last_clean":"2024-01-11T09:45:37.949238+0000","last_became_active":"2024-01-11T09:45:36.966015+0000","last_became_peered":"2024-01-11T09:45:36.966015+0000","last_unstale":"2024-01-11T09:45:37.949238+0000","last_undegraded":"2024-01-11T09:45:37.949238+0000","last_fullsized":"2024-01-11T09:45:37.949238+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:17.903461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984037+0000","last_change":"2024-01-11T09:45:37.974824+0000","last_active":"2024-01-11T09:45:37.984037+0000","last_peered":"2024-01-11T09:45:37.984037+0000","last_clean":"2024-01-11T09:45:37.984037+0000","last_became_active":"2024-01-11T09:45:37.974505+0000","last_became_peered":"2024-01-11T09:45:37.974505+0000","last_unstale":"2024-01-11T09:45:37.984037+0000","last_undegraded":"2024-01-11T09:45:37.984037+0000","last_fullsized":"2024-01-11T09:45:37.984037+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:21:58.660078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.d","version":"213'1","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950970+0000","last_change":"2024-01-11T09:45:23.153536+0000","last_active":"2024-01-11T09:45:37.950970+0000","last_peered":"2024-01-11T09:45:37.950970+0000","last_clean":"2024-01-11T09:45:37.950970+0000","last_became_active":"2024-01-11T09:45:23.152838+0000","last_became_peered":"2024-01-11T09:45:23.152838+0000","last_unstale":"2024-01-11T09:45:37.950970+0000","last_undegraded":"2024-01-11T09:45:37.950970+0000","last_fullsized":"2024-01-11T09:45:37.950970+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:14:59.795260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960771+0000","last_change":"2024-01-11T09:45:24.159728+0000","last_active":"2024-01-11T09:45:37.960771+0000","last_peered":"2024-01-11T09:45:37.960771+0000","last_clean":"2024-01-11T09:45:37.960771+0000","last_became_active":"2024-01-11T09:45:24.159489+0000","last_became_peered":"2024-01-11T09:45:24.159489+0000","last_unstale":"2024-01-11T09:45:37.960771+0000","last_undegraded":"2024-01-11T09:45:37.960771+0000","last_fullsized":"2024-01-11T09:45:37.960771+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:21:50.096457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"217'2","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984991+0000","last_change":"2024-01-11T09:45:36.978438+0000","last_active":"2024-01-11T09:45:37.984991+0000","last_peered":"2024-01-11T09:45:37.984991+0000","last_clean":"2024-01-11T09:45:37.984991+0000","last_became_active":"2024-01-11T09:45:36.977894+0000","last_became_peered":"2024-01-11T09:45:36.977894+0000","last_unstale":"2024-01-11T09:45:37.984991+0000","last_undegraded":"2024-01-11T09:45:37.984991+0000","last_fullsized":"2024-01-11T09:45:37.984991+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:15:51.367934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.977525+0000","last_change":"2024-01-11T09:45:37.966951+0000","last_active":"2024-01-11T09:45:37.977525+0000","last_peered":"2024-01-11T09:45:37.977525+0000","last_clean":"2024-01-11T09:45:37.977525+0000","last_became_active":"2024-01-11T09:45:37.966637+0000","last_became_peered":"2024-01-11T09:45:37.966637+0000","last_unstale":"2024-01-11T09:45:37.977525+0000","last_undegraded":"2024-01-11T09:45:37.977525+0000","last_fullsized":"2024-01-11T09:45:37.977525+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:51:04.639956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"213'2","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950295+0000","last_change":"2024-01-11T09:45:23.147655+0000","last_active":"2024-01-11T09:45:37.950295+0000","last_peered":"2024-01-11T09:45:37.950295+0000","last_clean":"2024-01-11T09:45:37.950295+0000","last_became_active":"2024-01-11T09:45:23.147203+0000","last_became_peered":"2024-01-11T09:45:23.147203+0000","last_unstale":"2024-01-11T09:45:37.950295+0000","last_undegraded":"2024-01-11T09:45:37.950295+0000","last_fullsized":"2024-01-11T09:45:37.950295+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:16:31.626739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950216+0000","last_change":"2024-01-11T09:45:24.140693+0000","last_active":"2024-01-11T09:45:37.950216+0000","last_peered":"2024-01-11T09:45:37.950216+0000","last_clean":"2024-01-11T09:45:37.950216+0000","last_became_active":"2024-01-11T09:45:24.140376+0000","last_became_peered":"2024-01-11T09:45:24.140376+0000","last_unstale":"2024-01-11T09:45:37.950216+0000","last_undegraded":"2024-01-11T09:45:37.950216+0000","last_fullsized":"2024-01-11T09:45:37.950216+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:52:17.873609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.966554+0000","last_change":"2024-01-11T09:45:36.972519+0000","last_active":"2024-01-11T09:45:37.966554+0000","last_peered":"2024-01-11T09:45:37.966554+0000","last_clean":"2024-01-11T09:45:37.966554+0000","last_became_active":"2024-01-11T09:45:36.971278+0000","last_became_peered":"2024-01-11T09:45:36.971278+0000","last_unstale":"2024-01-11T09:45:37.966554+0000","last_undegraded":"2024-01-11T09:45:37.966554+0000","last_fullsized":"2024-01-11T09:45:37.966554+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:29.997317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.978814+0000","last_change":"2024-01-11T09:45:37.962984+0000","last_active":"2024-01-11T09:45:37.978814+0000","last_peered":"2024-01-11T09:45:37.978814+0000","last_clean":"2024-01-11T09:45:37.978814+0000","last_became_active":"2024-01-11T09:45:37.962727+0000","last_became_peered":"2024-01-11T09:45:37.962727+0000","last_unstale":"2024-01-11T09:45:37.978814+0000","last_undegraded":"2024-01-11T09:45:37.978814+0000","last_fullsized":"2024-01-11T09:45:37.978814+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:29:35.726697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.f","version":"213'1","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.966530+0000","last_change":"2024-01-11T09:45:23.152322+0000","last_active":"2024-01-11T09:45:37.966530+0000","last_peered":"2024-01-11T09:45:37.966530+0000","last_clean":"2024-01-11T09:45:37.966530+0000","last_became_active":"2024-01-11T09:45:23.152006+0000","last_became_peered":"2024-01-11T09:45:23.152006+0000","last_unstale":"2024-01-11T09:45:37.966530+0000","last_undegraded":"2024-01-11T09:45:37.966530+0000","last_fullsized":"2024-01-11T09:45:37.966530+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:46.962719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950186+0000","last_change":"2024-01-11T09:45:24.152987+0000","last_active":"2024-01-11T09:45:37.950186+0000","last_peered":"2024-01-11T09:45:37.950186+0000","last_clean":"2024-01-11T09:45:37.950186+0000","last_became_active":"2024-01-11T09:45:24.152677+0000","last_became_peered":"2024-01-11T09:45:24.152677+0000","last_unstale":"2024-01-11T09:45:37.950186+0000","last_undegraded":"2024-01-11T09:45:37.950186+0000","last_fullsized":"2024-01-11T09:45:37.950186+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:03.360851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.947837+0000","last_change":"2024-01-11T09:45:36.972308+0000","last_active":"2024-01-11T09:45:37.947837+0000","last_peered":"2024-01-11T09:45:37.947837+0000","last_clean":"2024-01-11T09:45:37.947837+0000","last_became_active":"2024-01-11T09:45:36.972050+0000","last_became_peered":"2024-01-11T09:45:36.972050+0000","last_unstale":"2024-01-11T09:45:37.947837+0000","last_undegraded":"2024-01-11T09:45:37.947837+0000","last_fullsized":"2024-01-11T09:45:37.947837+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:57:04.436083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.982544+0000","last_change":"2024-01-11T09:45:37.975993+0000","last_active":"2024-01-11T09:45:37.982544+0000","last_peered":"2024-01-11T09:45:37.982544+0000","last_clean":"2024-01-11T09:45:37.982544+0000","last_became_active":"2024-01-11T09:45:37.975734+0000","last_became_peered":"2024-01-11T09:45:37.975734+0000","last_unstale":"2024-01-11T09:45:37.982544+0000","last_undegraded":"2024-01-11T09:45:37.982544+0000","last_fullsized":"2024-01-11T09:45:37.982544+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:52:52.400789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.10","version":"213'8","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.955930+0000","last_change":"2024-01-11T09:45:23.146296+0000","last_active":"2024-01-11T09:45:37.955930+0000","last_peered":"2024-01-11T09:45:37.955930+0000","last_clean":"2024-01-11T09:45:37.955930+0000","last_became_active":"2024-01-11T09:45:23.146026+0000","last_became_peered":"2024-01-11T09:45:23.146026+0000","last_unstale":"2024-01-11T09:45:37.955930+0000","last_undegraded":"2024-01-11T09:45:37.955930+0000","last_fullsized":"2024-01-11T09:45:37.955930+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:38:30.060556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.958796+0000","last_change":"2024-01-11T09:45:24.154643+0000","last_active":"2024-01-11T09:45:37.958796+0000","last_peered":"2024-01-11T09:45:37.958796+0000","last_clean":"2024-01-11T09:45:37.958796+0000","last_became_active":"2024-01-11T09:45:24.154293+0000","last_became_peered":"2024-01-11T09:45:24.154293+0000","last_unstale":"2024-01-11T09:45:37.958796+0000","last_undegraded":"2024-01-11T09:45:37.958796+0000","last_fullsized":"2024-01-11T09:45:37.958796+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:00:36.090834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.961988+0000","last_change":"2024-01-11T09:45:36.971692+0000","last_active":"2024-01-11T09:45:37.961988+0000","last_peered":"2024-01-11T09:45:37.961988+0000","last_clean":"2024-01-11T09:45:37.961988+0000","last_became_active":"2024-01-11T09:45:36.971379+0000","last_became_peered":"2024-01-11T09:45:36.971379+0000","last_unstale":"2024-01-11T09:45:37.961988+0000","last_undegraded":"2024-01-11T09:45:37.961988+0000","last_fullsized":"2024-01-11T09:45:37.961988+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:11:30.110533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.972230+0000","last_change":"2024-01-11T09:45:37.966212+0000","last_active":"2024-01-11T09:45:37.972230+0000","last_peered":"2024-01-11T09:45:37.972230+0000","last_clean":"2024-01-11T09:45:37.972230+0000","last_became_active":"2024-01-11T09:45:37.965957+0000","last_became_peered":"2024-01-11T09:45:37.965957+0000","last_unstale":"2024-01-11T09:45:37.972230+0000","last_undegraded":"2024-01-11T09:45:37.972230+0000","last_fullsized":"2024-01-11T09:45:37.972230+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:59:05.825428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949035+0000","last_change":"2024-01-11T09:45:23.151388+0000","last_active":"2024-01-11T09:45:37.949035+0000","last_peered":"2024-01-11T09:45:37.949035+0000","last_clean":"2024-01-11T09:45:37.949035+0000","last_became_active":"2024-01-11T09:45:23.151127+0000","last_became_peered":"2024-01-11T09:45:23.151127+0000","last_unstale":"2024-01-11T09:45:37.949035+0000","last_undegraded":"2024-01-11T09:45:37.949035+0000","last_fullsized":"2024-01-11T09:45:37.949035+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:43:21.323090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.955745+0000","last_change":"2024-01-11T09:45:24.166383+0000","last_active":"2024-01-11T09:45:37.955745+0000","last_peered":"2024-01-11T09:45:37.955745+0000","last_clean":"2024-01-11T09:45:37.955745+0000","last_became_active":"2024-01-11T09:45:24.159441+0000","last_became_peered":"2024-01-11T09:45:24.159441+0000","last_unstale":"2024-01-11T09:45:37.955745+0000","last_undegraded":"2024-01-11T09:45:37.955745+0000","last_fullsized":"2024-01-11T09:45:37.955745+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:25:32.122001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.955791+0000","last_change":"2024-01-11T09:45:36.976096+0000","last_active":"2024-01-11T09:45:37.955791+0000","last_peered":"2024-01-11T09:45:37.955791+0000","last_clean":"2024-01-11T09:45:37.955791+0000","last_became_active":"2024-01-11T09:45:36.975634+0000","last_became_peered":"2024-01-11T09:45:36.975634+0000","last_unstale":"2024-01-11T09:45:37.955791+0000","last_undegraded":"2024-01-11T09:45:37.955791+0000","last_fullsized":"2024-01-11T09:45:37.955791+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:49:00.158472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.986212+0000","last_change":"2024-01-11T09:45:37.980563+0000","last_active":"2024-01-11T09:45:37.986212+0000","last_peered":"2024-01-11T09:45:37.986212+0000","last_clean":"2024-01-11T09:45:37.986212+0000","last_became_active":"2024-01-11T09:45:37.980290+0000","last_became_peered":"2024-01-11T09:45:37.980290+0000","last_unstale":"2024-01-11T09:45:37.986212+0000","last_undegraded":"2024-01-11T09:45:37.986212+0000","last_fullsized":"2024-01-11T09:45:37.986212+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:59:05.825428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950826+0000","last_change":"2024-01-11T09:45:23.152794+0000","last_active":"2024-01-11T09:45:37.950826+0000","last_peered":"2024-01-11T09:45:37.950826+0000","last_clean":"2024-01-11T09:45:37.950826+0000","last_became_active":"2024-01-11T09:45:23.152495+0000","last_became_peered":"2024-01-11T09:45:23.152495+0000","last_unstale":"2024-01-11T09:45:37.950826+0000","last_undegraded":"2024-01-11T09:45:37.950826+0000","last_fullsized":"2024-01-11T09:45:37.950826+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:01:05.702272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949452+0000","last_change":"2024-01-11T09:45:24.147696+0000","last_active":"2024-01-11T09:45:37.949452+0000","last_peered":"2024-01-11T09:45:37.949452+0000","last_clean":"2024-01-11T09:45:37.949452+0000","last_became_active":"2024-01-11T09:45:24.147422+0000","last_became_peered":"2024-01-11T09:45:24.147422+0000","last_unstale":"2024-01-11T09:45:37.949452+0000","last_undegraded":"2024-01-11T09:45:37.949452+0000","last_fullsized":"2024-01-11T09:45:37.949452+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:32:40.240733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"217'6","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.993710+0000","last_change":"2024-01-11T09:45:36.963386+0000","last_active":"2024-01-11T09:45:37.993710+0000","last_peered":"2024-01-11T09:45:37.993710+0000","last_clean":"2024-01-11T09:45:37.993710+0000","last_became_active":"2024-01-11T09:45:36.962553+0000","last_became_peered":"2024-01-11T09:45:36.962553+0000","last_unstale":"2024-01-11T09:45:37.993710+0000","last_undegraded":"2024-01-11T09:45:37.993710+0000","last_fullsized":"2024-01-11T09:45:37.993710+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:15:51.367934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.972328+0000","last_change":"2024-01-11T09:45:37.965023+0000","last_active":"2024-01-11T09:45:37.972328+0000","last_peered":"2024-01-11T09:45:37.972328+0000","last_clean":"2024-01-11T09:45:37.972328+0000","last_became_active":"2024-01-11T09:45:37.963795+0000","last_became_peered":"2024-01-11T09:45:37.963795+0000","last_unstale":"2024-01-11T09:45:37.972328+0000","last_undegraded":"2024-01-11T09:45:37.972328+0000","last_fullsized":"2024-01-11T09:45:37.972328+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:28:12.821267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.13","version":"213'2","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.951250+0000","last_change":"2024-01-11T09:45:23.155269+0000","last_active":"2024-01-11T09:45:37.951250+0000","last_peered":"2024-01-11T09:45:37.951250+0000","last_clean":"2024-01-11T09:45:37.951250+0000","last_became_active":"2024-01-11T09:45:23.154990+0000","last_became_peered":"2024-01-11T09:45:23.154990+0000","last_unstale":"2024-01-11T09:45:37.951250+0000","last_undegraded":"2024-01-11T09:45:37.951250+0000","last_fullsized":"2024-01-11T09:45:37.951250+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:32:15.216292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.961636+0000","last_change":"2024-01-11T09:45:24.157881+0000","last_active":"2024-01-11T09:45:37.961636+0000","last_peered":"2024-01-11T09:45:37.961636+0000","last_clean":"2024-01-11T09:45:37.961636+0000","last_became_active":"2024-01-11T09:45:24.157224+0000","last_became_peered":"2024-01-11T09:45:24.157224+0000","last_unstale":"2024-01-11T09:45:37.961636+0000","last_undegraded":"2024-01-11T09:45:37.961636+0000","last_fullsized":"2024-01-11T09:45:37.961636+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:32:16.224153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"217'3","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.991193+0000","last_change":"2024-01-11T09:45:36.971737+0000","last_active":"2024-01-11T09:45:37.991193+0000","last_peered":"2024-01-11T09:45:37.991193+0000","last_clean":"2024-01-11T09:45:37.991193+0000","last_became_active":"2024-01-11T09:45:36.969536+0000","last_became_peered":"2024-01-11T09:45:36.969536+0000","last_unstale":"2024-01-11T09:45:37.991193+0000","last_undegraded":"2024-01-11T09:45:37.991193+0000","last_fullsized":"2024-01-11T09:45:37.991193+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:36:41.577470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.969343+0000","last_change":"2024-01-11T09:45:37.966563+0000","last_active":"2024-01-11T09:45:37.969343+0000","last_peered":"2024-01-11T09:45:37.969343+0000","last_clean":"2024-01-11T09:45:37.969343+0000","last_became_active":"2024-01-11T09:45:37.966273+0000","last_became_peered":"2024-01-11T09:45:37.966273+0000","last_unstale":"2024-01-11T09:45:37.969343+0000","last_undegraded":"2024-01-11T09:45:37.969343+0000","last_fullsized":"2024-01-11T09:45:37.969343+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:39:19.277161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.14","version":"214'8","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.953956+0000","last_change":"2024-01-11T09:45:23.152136+0000","last_active":"2024-01-11T09:45:37.953956+0000","last_peered":"2024-01-11T09:45:37.953956+0000","last_clean":"2024-01-11T09:45:37.953956+0000","last_became_active":"2024-01-11T09:45:23.151819+0000","last_became_peered":"2024-01-11T09:45:23.151819+0000","last_unstale":"2024-01-11T09:45:37.953956+0000","last_undegraded":"2024-01-11T09:45:37.953956+0000","last_fullsized":"2024-01-11T09:45:37.953956+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:41:09.070880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2749,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960221+0000","last_change":"2024-01-11T09:45:24.169731+0000","last_active":"2024-01-11T09:45:37.960221+0000","last_peered":"2024-01-11T09:45:37.960221+0000","last_clean":"2024-01-11T09:45:37.960221+0000","last_became_active":"2024-01-11T09:45:24.169381+0000","last_became_peered":"2024-01-11T09:45:24.169381+0000","last_unstale":"2024-01-11T09:45:37.960221+0000","last_undegraded":"2024-01-11T09:45:37.960221+0000","last_fullsized":"2024-01-11T09:45:37.960221+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:04:41.533247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.956354+0000","last_change":"2024-01-11T09:45:36.964088+0000","last_active":"2024-01-11T09:45:37.956354+0000","last_peered":"2024-01-11T09:45:37.956354+0000","last_clean":"2024-01-11T09:45:37.956354+0000","last_became_active":"2024-01-11T09:45:36.963596+0000","last_became_peered":"2024-01-11T09:45:36.963596+0000","last_unstale":"2024-01-11T09:45:37.956354+0000","last_undegraded":"2024-01-11T09:45:37.956354+0000","last_fullsized":"2024-01-11T09:45:37.956354+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:28:10.306993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984996+0000","last_change":"2024-01-11T09:45:37.977762+0000","last_active":"2024-01-11T09:45:37.984996+0000","last_peered":"2024-01-11T09:45:37.984996+0000","last_clean":"2024-01-11T09:45:37.984996+0000","last_became_active":"2024-01-11T09:45:37.977432+0000","last_became_peered":"2024-01-11T09:45:37.977432+0000","last_unstale":"2024-01-11T09:45:37.984996+0000","last_undegraded":"2024-01-11T09:45:37.984996+0000","last_fullsized":"2024-01-11T09:45:37.984996+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:28:11.313558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.15","version":"213'3","reported_seq":17,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950781+0000","last_change":"2024-01-11T09:45:23.154355+0000","last_active":"2024-01-11T09:45:37.950781+0000","last_peered":"2024-01-11T09:45:37.950781+0000","last_clean":"2024-01-11T09:45:37.950781+0000","last_became_active":"2024-01-11T09:45:23.153262+0000","last_became_peered":"2024-01-11T09:45:23.153262+0000","last_unstale":"2024-01-11T09:45:37.950781+0000","last_undegraded":"2024-01-11T09:45:37.950781+0000","last_fullsized":"2024-01-11T09:45:37.950781+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:32:15.216292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954159+0000","last_change":"2024-01-11T09:45:24.150019+0000","last_active":"2024-01-11T09:45:37.954159+0000","last_peered":"2024-01-11T09:45:37.954159+0000","last_clean":"2024-01-11T09:45:37.954159+0000","last_became_active":"2024-01-11T09:45:24.148994+0000","last_became_peered":"2024-01-11T09:45:24.148994+0000","last_unstale":"2024-01-11T09:45:37.954159+0000","last_undegraded":"2024-01-11T09:45:37.954159+0000","last_fullsized":"2024-01-11T09:45:37.954159+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:29:01.556722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"217'2","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.986712+0000","last_change":"2024-01-11T09:45:36.972442+0000","last_active":"2024-01-11T09:45:37.986712+0000","last_peered":"2024-01-11T09:45:37.986712+0000","last_clean":"2024-01-11T09:45:37.986712+0000","last_became_active":"2024-01-11T09:45:36.972165+0000","last_became_peered":"2024-01-11T09:45:36.972165+0000","last_unstale":"2024-01-11T09:45:37.986712+0000","last_undegraded":"2024-01-11T09:45:37.986712+0000","last_fullsized":"2024-01-11T09:45:37.986712+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:28:31.210512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.983856+0000","last_change":"2024-01-11T09:45:37.979778+0000","last_active":"2024-01-11T09:45:37.983856+0000","last_peered":"2024-01-11T09:45:37.983856+0000","last_clean":"2024-01-11T09:45:37.983856+0000","last_became_active":"2024-01-11T09:45:37.977950+0000","last_became_peered":"2024-01-11T09:45:37.977950+0000","last_unstale":"2024-01-11T09:45:37.983856+0000","last_undegraded":"2024-01-11T09:45:37.983856+0000","last_fullsized":"2024-01-11T09:45:37.983856+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:07:41.051644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950589+0000","last_change":"2024-01-11T09:45:23.150869+0000","last_active":"2024-01-11T09:45:37.950589+0000","last_peered":"2024-01-11T09:45:37.950589+0000","last_clean":"2024-01-11T09:45:37.950589+0000","last_became_active":"2024-01-11T09:45:23.150604+0000","last_became_peered":"2024-01-11T09:45:23.150604+0000","last_unstale":"2024-01-11T09:45:37.950589+0000","last_undegraded":"2024-01-11T09:45:37.950589+0000","last_fullsized":"2024-01-11T09:45:37.950589+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:19.210181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960616+0000","last_change":"2024-01-11T09:45:24.166531+0000","last_active":"2024-01-11T09:45:37.960616+0000","last_peered":"2024-01-11T09:45:37.960616+0000","last_clean":"2024-01-11T09:45:37.960616+0000","last_became_active":"2024-01-11T09:45:24.164071+0000","last_became_peered":"2024-01-11T09:45:24.164071+0000","last_unstale":"2024-01-11T09:45:37.960616+0000","last_undegraded":"2024-01-11T09:45:37.960616+0000","last_fullsized":"2024-01-11T09:45:37.960616+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:41:10.078741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"217'8","reported_seq":18,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.991634+0000","last_change":"2024-01-11T09:45:36.966889+0000","last_active":"2024-01-11T09:45:37.991634+0000","last_peered":"2024-01-11T09:45:37.991634+0000","last_clean":"2024-01-11T09:45:37.991634+0000","last_became_active":"2024-01-11T09:45:36.965424+0000","last_became_peered":"2024-01-11T09:45:36.965424+0000","last_unstale":"2024-01-11T09:45:37.991634+0000","last_undegraded":"2024-01-11T09:45:37.991634+0000","last_fullsized":"2024-01-11T09:45:37.991634+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:50:36.781518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.957865+0000","last_change":"2024-01-11T09:45:37.952454+0000","last_active":"2024-01-11T09:45:37.957865+0000","last_peered":"2024-01-11T09:45:37.957865+0000","last_clean":"2024-01-11T09:45:37.957865+0000","last_became_active":"2024-01-11T09:45:37.952102+0000","last_became_peered":"2024-01-11T09:45:37.952102+0000","last_unstale":"2024-01-11T09:45:37.957865+0000","last_undegraded":"2024-01-11T09:45:37.957865+0000","last_fullsized":"2024-01-11T09:45:37.957865+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:52:26.173608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.968052+0000","last_change":"2024-01-11T09:45:23.152897+0000","last_active":"2024-01-11T09:45:37.968052+0000","last_peered":"2024-01-11T09:45:37.968052+0000","last_clean":"2024-01-11T09:45:37.968052+0000","last_became_active":"2024-01-11T09:45:23.152620+0000","last_became_peered":"2024-01-11T09:45:23.152620+0000","last_unstale":"2024-01-11T09:45:37.968052+0000","last_undegraded":"2024-01-11T09:45:37.968052+0000","last_fullsized":"2024-01-11T09:45:37.968052+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:02:41.619972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954942+0000","last_change":"2024-01-11T09:45:24.155035+0000","last_active":"2024-01-11T09:45:37.954942+0000","last_peered":"2024-01-11T09:45:37.954942+0000","last_clean":"2024-01-11T09:45:37.954942+0000","last_became_active":"2024-01-11T09:45:24.153980+0000","last_became_peered":"2024-01-11T09:45:24.153980+0000","last_unstale":"2024-01-11T09:45:37.954942+0000","last_undegraded":"2024-01-11T09:45:37.954942+0000","last_fullsized":"2024-01-11T09:45:37.954942+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:52:20.741771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954953+0000","last_change":"2024-01-11T09:45:36.967350+0000","last_active":"2024-01-11T09:45:37.954953+0000","last_peered":"2024-01-11T09:45:37.954953+0000","last_clean":"2024-01-11T09:45:37.954953+0000","last_became_active":"2024-01-11T09:45:36.964560+0000","last_became_peered":"2024-01-11T09:45:36.964560+0000","last_unstale":"2024-01-11T09:45:37.954953+0000","last_undegraded":"2024-01-11T09:45:37.954953+0000","last_fullsized":"2024-01-11T09:45:37.954953+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:44:19.194643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.988960+0000","last_change":"2024-01-11T09:45:37.986871+0000","last_active":"2024-01-11T09:45:37.988960+0000","last_peered":"2024-01-11T09:45:37.988960+0000","last_clean":"2024-01-11T09:45:37.988960+0000","last_became_active":"2024-01-11T09:45:37.986318+0000","last_became_peered":"2024-01-11T09:45:37.986318+0000","last_unstale":"2024-01-11T09:45:37.988960+0000","last_undegraded":"2024-01-11T09:45:37.988960+0000","last_fullsized":"2024-01-11T09:45:37.988960+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:27:26.360142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949746+0000","last_change":"2024-01-11T09:45:23.152843+0000","last_active":"2024-01-11T09:45:37.949746+0000","last_peered":"2024-01-11T09:45:37.949746+0000","last_clean":"2024-01-11T09:45:37.949746+0000","last_became_active":"2024-01-11T09:45:23.152579+0000","last_became_peered":"2024-01-11T09:45:23.152579+0000","last_unstale":"2024-01-11T09:45:37.949746+0000","last_undegraded":"2024-01-11T09:45:37.949746+0000","last_fullsized":"2024-01-11T09:45:37.949746+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:48:05.979085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954041+0000","last_change":"2024-01-11T09:45:24.150754+0000","last_active":"2024-01-11T09:45:37.954041+0000","last_peered":"2024-01-11T09:45:37.954041+0000","last_clean":"2024-01-11T09:45:37.954041+0000","last_became_active":"2024-01-11T09:45:24.150485+0000","last_became_peered":"2024-01-11T09:45:24.150485+0000","last_unstale":"2024-01-11T09:45:37.954041+0000","last_undegraded":"2024-01-11T09:45:37.954041+0000","last_fullsized":"2024-01-11T09:45:37.954041+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:32:19.368822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"217'1","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.987045+0000","last_change":"2024-01-11T09:45:36.974169+0000","last_active":"2024-01-11T09:45:37.987045+0000","last_peered":"2024-01-11T09:45:37.987045+0000","last_clean":"2024-01-11T09:45:37.987045+0000","last_became_active":"2024-01-11T09:45:36.973911+0000","last_became_peered":"2024-01-11T09:45:36.973911+0000","last_unstale":"2024-01-11T09:45:37.987045+0000","last_undegraded":"2024-01-11T09:45:37.987045+0000","last_fullsized":"2024-01-11T09:45:37.987045+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:46:15.891808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.981819+0000","last_change":"2024-01-11T09:45:37.975235+0000","last_active":"2024-01-11T09:45:37.981819+0000","last_peered":"2024-01-11T09:45:37.981819+0000","last_clean":"2024-01-11T09:45:37.981819+0000","last_became_active":"2024-01-11T09:45:37.972196+0000","last_became_peered":"2024-01-11T09:45:37.972196+0000","last_unstale":"2024-01-11T09:45:37.981819+0000","last_undegraded":"2024-01-11T09:45:37.981819+0000","last_fullsized":"2024-01-11T09:45:37.981819+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:36:42.584034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.955181+0000","last_change":"2024-01-11T09:45:23.145331+0000","last_active":"2024-01-11T09:45:37.955181+0000","last_peered":"2024-01-11T09:45:37.955181+0000","last_clean":"2024-01-11T09:45:37.955181+0000","last_became_active":"2024-01-11T09:45:23.145064+0000","last_became_peered":"2024-01-11T09:45:23.145064+0000","last_unstale":"2024-01-11T09:45:37.955181+0000","last_undegraded":"2024-01-11T09:45:37.955181+0000","last_fullsized":"2024-01-11T09:45:37.955181+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:56:41.726387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.961053+0000","last_change":"2024-01-11T09:45:24.150698+0000","last_active":"2024-01-11T09:45:37.961053+0000","last_peered":"2024-01-11T09:45:37.961053+0000","last_clean":"2024-01-11T09:45:37.961053+0000","last_became_active":"2024-01-11T09:45:24.149431+0000","last_became_peered":"2024-01-11T09:45:24.149431+0000","last_unstale":"2024-01-11T09:45:37.961053+0000","last_undegraded":"2024-01-11T09:45:37.961053+0000","last_fullsized":"2024-01-11T09:45:37.961053+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:48:06.986946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"217'4","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.990945+0000","last_change":"2024-01-11T09:45:36.972087+0000","last_active":"2024-01-11T09:45:37.990945+0000","last_peered":"2024-01-11T09:45:37.990945+0000","last_clean":"2024-01-11T09:45:37.990945+0000","last_became_active":"2024-01-11T09:45:36.970149+0000","last_became_peered":"2024-01-11T09:45:36.970149+0000","last_unstale":"2024-01-11T09:45:37.990945+0000","last_undegraded":"2024-01-11T09:45:37.990945+0000","last_fullsized":"2024-01-11T09:45:37.990945+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:39:48.476885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984778+0000","last_change":"2024-01-11T09:45:37.976386+0000","last_active":"2024-01-11T09:45:37.984778+0000","last_peered":"2024-01-11T09:45:37.984778+0000","last_clean":"2024-01-11T09:45:37.984778+0000","last_became_active":"2024-01-11T09:45:37.975081+0000","last_became_peered":"2024-01-11T09:45:37.975081+0000","last_unstale":"2024-01-11T09:45:37.984778+0000","last_undegraded":"2024-01-11T09:45:37.984778+0000","last_fullsized":"2024-01-11T09:45:37.984778+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:11:31.117098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1a","version":"213'1","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.961545+0000","last_change":"2024-01-11T09:45:23.154575+0000","last_active":"2024-01-11T09:45:37.961545+0000","last_peered":"2024-01-11T09:45:37.961545+0000","last_clean":"2024-01-11T09:45:37.961545+0000","last_became_active":"2024-01-11T09:45:23.154313+0000","last_became_peered":"2024-01-11T09:45:23.154313+0000","last_unstale":"2024-01-11T09:45:37.961545+0000","last_undegraded":"2024-01-11T09:45:37.961545+0000","last_fullsized":"2024-01-11T09:45:37.961545+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:14:08.645320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954142+0000","last_change":"2024-01-11T09:45:24.148848+0000","last_active":"2024-01-11T09:45:37.954142+0000","last_peered":"2024-01-11T09:45:37.954142+0000","last_clean":"2024-01-11T09:45:37.954142+0000","last_became_active":"2024-01-11T09:45:24.148573+0000","last_became_peered":"2024-01-11T09:45:24.148573+0000","last_unstale":"2024-01-11T09:45:37.954142+0000","last_undegraded":"2024-01-11T09:45:37.954142+0000","last_fullsized":"2024-01-11T09:45:37.954142+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:29:30.529875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"217'2","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984837+0000","last_change":"2024-01-11T09:45:36.965686+0000","last_active":"2024-01-11T09:45:37.984837+0000","last_peered":"2024-01-11T09:45:37.984837+0000","last_clean":"2024-01-11T09:45:37.984837+0000","last_became_active":"2024-01-11T09:45:36.964782+0000","last_became_peered":"2024-01-11T09:45:36.964782+0000","last_unstale":"2024-01-11T09:45:37.984837+0000","last_undegraded":"2024-01-11T09:45:37.984837+0000","last_fullsized":"2024-01-11T09:45:37.984837+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:18:37.841014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.983269+0000","last_change":"2024-01-11T09:45:37.978823+0000","last_active":"2024-01-11T09:45:37.983269+0000","last_peered":"2024-01-11T09:45:37.983269+0000","last_clean":"2024-01-11T09:45:37.983269+0000","last_became_active":"2024-01-11T09:45:37.978561+0000","last_became_peered":"2024-01-11T09:45:37.978561+0000","last_unstale":"2024-01-11T09:45:37.983269+0000","last_undegraded":"2024-01-11T09:45:37.983269+0000","last_fullsized":"2024-01-11T09:45:37.983269+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:06:59.351961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1c","version":"213'2","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954064+0000","last_change":"2024-01-11T09:45:23.150649+0000","last_active":"2024-01-11T09:45:37.954064+0000","last_peered":"2024-01-11T09:45:37.954064+0000","last_clean":"2024-01-11T09:45:37.954064+0000","last_became_active":"2024-01-11T09:45:23.150425+0000","last_became_peered":"2024-01-11T09:45:23.150425+0000","last_unstale":"2024-01-11T09:45:37.954064+0000","last_undegraded":"2024-01-11T09:45:37.954064+0000","last_fullsized":"2024-01-11T09:45:37.954064+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:14:08.645320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950548+0000","last_change":"2024-01-11T09:45:24.164335+0000","last_active":"2024-01-11T09:45:37.950548+0000","last_peered":"2024-01-11T09:45:37.950548+0000","last_clean":"2024-01-11T09:45:37.950548+0000","last_became_active":"2024-01-11T09:45:24.164063+0000","last_became_peered":"2024-01-11T09:45:24.164063+0000","last_unstale":"2024-01-11T09:45:37.950548+0000","last_undegraded":"2024-01-11T09:45:37.950548+0000","last_fullsized":"2024-01-11T09:45:37.950548+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:42:21.543118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"217'1","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.986047+0000","last_change":"2024-01-11T09:45:36.973376+0000","last_active":"2024-01-11T09:45:37.986047+0000","last_peered":"2024-01-11T09:45:37.986047+0000","last_clean":"2024-01-11T09:45:37.986047+0000","last_became_active":"2024-01-11T09:45:36.973100+0000","last_became_peered":"2024-01-11T09:45:36.973100+0000","last_unstale":"2024-01-11T09:45:37.986047+0000","last_undegraded":"2024-01-11T09:45:37.986047+0000","last_fullsized":"2024-01-11T09:45:37.986047+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:08:57.106726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.983126+0000","last_change":"2024-01-11T09:45:37.976372+0000","last_active":"2024-01-11T09:45:37.983126+0000","last_peered":"2024-01-11T09:45:37.983126+0000","last_clean":"2024-01-11T09:45:37.983126+0000","last_became_active":"2024-01-11T09:45:37.976103+0000","last_became_peered":"2024-01-11T09:45:37.976103+0000","last_unstale":"2024-01-11T09:45:37.983126+0000","last_undegraded":"2024-01-11T09:45:37.983126+0000","last_fullsized":"2024-01-11T09:45:37.983126+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:31.003882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1d","version":"213'2","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960565+0000","last_change":"2024-01-11T09:45:23.144976+0000","last_active":"2024-01-11T09:45:37.960565+0000","last_peered":"2024-01-11T09:45:37.960565+0000","last_clean":"2024-01-11T09:45:37.960565+0000","last_became_active":"2024-01-11T09:45:23.144660+0000","last_became_peered":"2024-01-11T09:45:23.144660+0000","last_unstale":"2024-01-11T09:45:37.960565+0000","last_undegraded":"2024-01-11T09:45:37.960565+0000","last_fullsized":"2024-01-11T09:45:37.960565+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:59:40.579473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.957059+0000","last_change":"2024-01-11T09:45:24.165254+0000","last_active":"2024-01-11T09:45:37.957059+0000","last_peered":"2024-01-11T09:45:37.957059+0000","last_clean":"2024-01-11T09:45:37.957059+0000","last_became_active":"2024-01-11T09:45:24.163213+0000","last_became_peered":"2024-01-11T09:45:24.163213+0000","last_unstale":"2024-01-11T09:45:37.957059+0000","last_undegraded":"2024-01-11T09:45:37.957059+0000","last_fullsized":"2024-01-11T09:45:37.957059+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:29:21.907384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"217'1","reported_seq":11,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.986835+0000","last_change":"2024-01-11T09:45:36.967799+0000","last_active":"2024-01-11T09:45:37.986835+0000","last_peered":"2024-01-11T09:45:37.986835+0000","last_clean":"2024-01-11T09:45:37.986835+0000","last_became_active":"2024-01-11T09:45:36.967519+0000","last_became_peered":"2024-01-11T09:45:36.967519+0000","last_unstale":"2024-01-11T09:45:37.986835+0000","last_undegraded":"2024-01-11T09:45:37.986835+0000","last_fullsized":"2024-01-11T09:45:37.986835+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:17.903461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.972388+0000","last_change":"2024-01-11T09:45:37.967983+0000","last_active":"2024-01-11T09:45:37.972388+0000","last_peered":"2024-01-11T09:45:37.972388+0000","last_clean":"2024-01-11T09:45:37.972388+0000","last_became_active":"2024-01-11T09:45:37.967172+0000","last_became_peered":"2024-01-11T09:45:37.967172+0000","last_unstale":"2024-01-11T09:45:37.972388+0000","last_undegraded":"2024-01-11T09:45:37.972388+0000","last_fullsized":"2024-01-11T09:45:37.972388+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:31.003882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1e","version":"213'1","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949928+0000","last_change":"2024-01-11T09:45:23.148530+0000","last_active":"2024-01-11T09:45:37.949928+0000","last_peered":"2024-01-11T09:45:37.949928+0000","last_clean":"2024-01-11T09:45:37.949928+0000","last_became_active":"2024-01-11T09:45:23.148208+0000","last_became_peered":"2024-01-11T09:45:23.148208+0000","last_unstale":"2024-01-11T09:45:37.949928+0000","last_undegraded":"2024-01-11T09:45:37.949928+0000","last_fullsized":"2024-01-11T09:45:37.949928+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:44:55.075364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.951770+0000","last_change":"2024-01-11T09:45:24.150915+0000","last_active":"2024-01-11T09:45:37.951770+0000","last_peered":"2024-01-11T09:45:37.951770+0000","last_clean":"2024-01-11T09:45:37.951770+0000","last_became_active":"2024-01-11T09:45:24.149015+0000","last_became_peered":"2024-01-11T09:45:24.149015+0000","last_unstale":"2024-01-11T09:45:37.951770+0000","last_undegraded":"2024-01-11T09:45:37.951770+0000","last_fullsized":"2024-01-11T09:45:37.951770+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:14:14.599129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949362+0000","last_change":"2024-01-11T09:45:36.959784+0000","last_active":"2024-01-11T09:45:37.949362+0000","last_peered":"2024-01-11T09:45:37.949362+0000","last_clean":"2024-01-11T09:45:37.949362+0000","last_became_active":"2024-01-11T09:45:36.958855+0000","last_became_peered":"2024-01-11T09:45:36.958855+0000","last_unstale":"2024-01-11T09:45:37.949362+0000","last_undegraded":"2024-01-11T09:45:37.949362+0000","last_fullsized":"2024-01-11T09:45:37.949362+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:57:59.056615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.986150+0000","last_change":"2024-01-11T09:45:37.982374+0000","last_active":"2024-01-11T09:45:37.986150+0000","last_peered":"2024-01-11T09:45:37.986150+0000","last_clean":"2024-01-11T09:45:37.986150+0000","last_became_active":"2024-01-11T09:45:37.982081+0000","last_became_peered":"2024-01-11T09:45:37.982081+0000","last_unstale":"2024-01-11T09:45:37.986150+0000","last_undegraded":"2024-01-11T09:45:37.986150+0000","last_fullsized":"2024-01-11T09:45:37.986150+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:56.535829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1f","version":"213'4","reported_seq":18,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.967298+0000","last_change":"2024-01-11T09:45:23.150803+0000","last_active":"2024-01-11T09:45:37.967298+0000","last_peered":"2024-01-11T09:45:37.967298+0000","last_clean":"2024-01-11T09:45:37.967298+0000","last_became_active":"2024-01-11T09:45:23.149761+0000","last_became_peered":"2024-01-11T09:45:23.149761+0000","last_unstale":"2024-01-11T09:45:37.967298+0000","last_undegraded":"2024-01-11T09:45:37.967298+0000","last_fullsized":"2024-01-11T09:45:37.967298+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:22.112719+0000","last_clean_scrub_stamp":"2024-01-11T09:45:22.112719+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:33:40.495800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.949294+0000","last_change":"2024-01-11T09:45:24.167467+0000","last_active":"2024-01-11T09:45:37.949294+0000","last_peered":"2024-01-11T09:45:37.949294+0000","last_clean":"2024-01-11T09:45:37.949294+0000","last_became_active":"2024-01-11T09:45:24.167175+0000","last_became_peered":"2024-01-11T09:45:24.167175+0000","last_unstale":"2024-01-11T09:45:37.949294+0000","last_undegraded":"2024-01-11T09:45:37.949294+0000","last_fullsized":"2024-01-11T09:45:37.949294+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:52:36.044653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":10,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.948006+0000","last_change":"2024-01-11T09:45:36.964388+0000","last_active":"2024-01-11T09:45:37.948006+0000","last_peered":"2024-01-11T09:45:37.948006+0000","last_clean":"2024-01-11T09:45:37.948006+0000","last_became_active":"2024-01-11T09:45:36.964114+0000","last_became_peered":"2024-01-11T09:45:36.964114+0000","last_unstale":"2024-01-11T09:45:37.948006+0000","last_undegraded":"2024-01-11T09:45:37.948006+0000","last_fullsized":"2024-01-11T09:45:37.948006+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:35.933327+0000","last_clean_scrub_stamp":"2024-01-11T09:45:35.933327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:28:31.210512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.973257+0000","last_change":"2024-01-11T09:45:37.969941+0000","last_active":"2024-01-11T09:45:37.973257+0000","last_peered":"2024-01-11T09:45:37.973257+0000","last_clean":"2024-01-11T09:45:37.973257+0000","last_became_active":"2024-01-11T09:45:37.969684+0000","last_became_peered":"2024-01-11T09:45:37.969684+0000","last_unstale":"2024-01-11T09:45:37.973257+0000","last_undegraded":"2024-01-11T09:45:37.973257+0000","last_fullsized":"2024-01-11T09:45:37.973257+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:57:46.266155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984003+0000","last_change":"2024-01-11T09:45:37.977272+0000","last_active":"2024-01-11T09:45:37.984003+0000","last_peered":"2024-01-11T09:45:37.984003+0000","last_clean":"2024-01-11T09:45:37.984003+0000","last_became_active":"2024-01-11T09:45:37.975292+0000","last_became_peered":"2024-01-11T09:45:37.975292+0000","last_unstale":"2024-01-11T09:45:37.984003+0000","last_undegraded":"2024-01-11T09:45:37.984003+0000","last_fullsized":"2024-01-11T09:45:37.984003+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:47:04.067575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.957660+0000","last_change":"2024-01-11T09:45:24.167922+0000","last_active":"2024-01-11T09:45:37.957660+0000","last_peered":"2024-01-11T09:45:37.957660+0000","last_clean":"2024-01-11T09:45:37.957660+0000","last_became_active":"2024-01-11T09:45:24.167641+0000","last_became_peered":"2024-01-11T09:45:24.167641+0000","last_unstale":"2024-01-11T09:45:37.957660+0000","last_undegraded":"2024-01-11T09:45:37.957660+0000","last_fullsized":"2024-01-11T09:45:37.957660+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:17.184569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.973931+0000","last_change":"2024-01-11T09:45:37.970672+0000","last_active":"2024-01-11T09:45:37.973931+0000","last_peered":"2024-01-11T09:45:37.973931+0000","last_clean":"2024-01-11T09:45:37.973931+0000","last_became_active":"2024-01-11T09:45:37.970368+0000","last_became_peered":"2024-01-11T09:45:37.970368+0000","last_unstale":"2024-01-11T09:45:37.973931+0000","last_undegraded":"2024-01-11T09:45:37.973931+0000","last_fullsized":"2024-01-11T09:45:37.973931+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:02:35.657312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.948942+0000","last_change":"2024-01-11T09:45:24.155055+0000","last_active":"2024-01-11T09:45:37.948942+0000","last_peered":"2024-01-11T09:45:37.948942+0000","last_clean":"2024-01-11T09:45:37.948942+0000","last_became_active":"2024-01-11T09:45:24.154781+0000","last_became_peered":"2024-01-11T09:45:24.154781+0000","last_unstale":"2024-01-11T09:45:37.948942+0000","last_undegraded":"2024-01-11T09:45:37.948942+0000","last_fullsized":"2024-01-11T09:45:37.948942+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:38:28.859392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.947904+0000","last_change":"2024-01-11T09:45:24.146908+0000","last_active":"2024-01-11T09:45:37.947904+0000","last_peered":"2024-01-11T09:45:37.947904+0000","last_clean":"2024-01-11T09:45:37.947904+0000","last_became_active":"2024-01-11T09:45:24.146543+0000","last_became_peered":"2024-01-11T09:45:24.146543+0000","last_unstale":"2024-01-11T09:45:37.947904+0000","last_undegraded":"2024-01-11T09:45:37.947904+0000","last_fullsized":"2024-01-11T09:45:37.947904+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:10:40.983480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.975579+0000","last_change":"2024-01-11T09:45:37.971778+0000","last_active":"2024-01-11T09:45:37.975579+0000","last_peered":"2024-01-11T09:45:37.975579+0000","last_clean":"2024-01-11T09:45:37.975579+0000","last_became_active":"2024-01-11T09:45:37.971369+0000","last_became_peered":"2024-01-11T09:45:37.971369+0000","last_unstale":"2024-01-11T09:45:37.975579+0000","last_undegraded":"2024-01-11T09:45:37.975579+0000","last_fullsized":"2024-01-11T09:45:37.975579+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:57:26.930467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.988545+0000","last_change":"2024-01-11T09:45:37.984875+0000","last_active":"2024-01-11T09:45:37.988545+0000","last_peered":"2024-01-11T09:45:37.988545+0000","last_clean":"2024-01-11T09:45:37.988545+0000","last_became_active":"2024-01-11T09:45:37.984603+0000","last_became_peered":"2024-01-11T09:45:37.984603+0000","last_unstale":"2024-01-11T09:45:37.988545+0000","last_undegraded":"2024-01-11T09:45:37.988545+0000","last_fullsized":"2024-01-11T09:45:37.988545+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:32:49.312144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.967593+0000","last_change":"2024-01-11T09:45:24.148965+0000","last_active":"2024-01-11T09:45:37.967593+0000","last_peered":"2024-01-11T09:45:37.967593+0000","last_clean":"2024-01-11T09:45:37.967593+0000","last_became_active":"2024-01-11T09:45:24.148481+0000","last_became_peered":"2024-01-11T09:45:24.148481+0000","last_unstale":"2024-01-11T09:45:37.967593+0000","last_undegraded":"2024-01-11T09:45:37.967593+0000","last_fullsized":"2024-01-11T09:45:37.967593+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:39.406018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960501+0000","last_change":"2024-01-11T09:45:24.166152+0000","last_active":"2024-01-11T09:45:37.960501+0000","last_peered":"2024-01-11T09:45:37.960501+0000","last_clean":"2024-01-11T09:45:37.960501+0000","last_became_active":"2024-01-11T09:45:24.165889+0000","last_became_peered":"2024-01-11T09:45:24.165889+0000","last_unstale":"2024-01-11T09:45:37.960501+0000","last_undegraded":"2024-01-11T09:45:37.960501+0000","last_fullsized":"2024-01-11T09:45:37.960501+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:01:06.710133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.972409+0000","last_change":"2024-01-11T09:45:37.968430+0000","last_active":"2024-01-11T09:45:37.972409+0000","last_peered":"2024-01-11T09:45:37.972409+0000","last_clean":"2024-01-11T09:45:37.972409+0000","last_became_active":"2024-01-11T09:45:37.967571+0000","last_became_peered":"2024-01-11T09:45:37.967571+0000","last_unstale":"2024-01-11T09:45:37.972409+0000","last_undegraded":"2024-01-11T09:45:37.972409+0000","last_fullsized":"2024-01-11T09:45:37.972409+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:57:26.930467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.967232+0000","last_change":"2024-01-11T09:45:24.150865+0000","last_active":"2024-01-11T09:45:37.967232+0000","last_peered":"2024-01-11T09:45:37.967232+0000","last_clean":"2024-01-11T09:45:37.967232+0000","last_became_active":"2024-01-11T09:45:24.148370+0000","last_became_peered":"2024-01-11T09:45:24.148370+0000","last_unstale":"2024-01-11T09:45:37.967232+0000","last_undegraded":"2024-01-11T09:45:37.967232+0000","last_fullsized":"2024-01-11T09:45:37.967232+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:27.774010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.980650+0000","last_change":"2024-01-11T09:45:37.976027+0000","last_active":"2024-01-11T09:45:37.980650+0000","last_peered":"2024-01-11T09:45:37.980650+0000","last_clean":"2024-01-11T09:45:37.980650+0000","last_became_active":"2024-01-11T09:45:37.975753+0000","last_became_peered":"2024-01-11T09:45:37.975753+0000","last_unstale":"2024-01-11T09:45:37.980650+0000","last_undegraded":"2024-01-11T09:45:37.980650+0000","last_fullsized":"2024-01-11T09:45:37.980650+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:18:38.847578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.962139+0000","last_change":"2024-01-11T09:45:24.158803+0000","last_active":"2024-01-11T09:45:37.962139+0000","last_peered":"2024-01-11T09:45:37.962139+0000","last_clean":"2024-01-11T09:45:37.962139+0000","last_became_active":"2024-01-11T09:45:24.158533+0000","last_became_peered":"2024-01-11T09:45:24.158533+0000","last_unstale":"2024-01-11T09:45:37.962139+0000","last_undegraded":"2024-01-11T09:45:37.962139+0000","last_fullsized":"2024-01-11T09:45:37.962139+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:57:46.243868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.970344+0000","last_change":"2024-01-11T09:45:37.967437+0000","last_active":"2024-01-11T09:45:37.970344+0000","last_peered":"2024-01-11T09:45:37.970344+0000","last_clean":"2024-01-11T09:45:37.970344+0000","last_became_active":"2024-01-11T09:45:37.967161+0000","last_became_peered":"2024-01-11T09:45:37.967161+0000","last_unstale":"2024-01-11T09:45:37.970344+0000","last_undegraded":"2024-01-11T09:45:37.970344+0000","last_fullsized":"2024-01-11T09:45:37.970344+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:54:27.964289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950338+0000","last_change":"2024-01-11T09:45:24.145841+0000","last_active":"2024-01-11T09:45:37.950338+0000","last_peered":"2024-01-11T09:45:37.950338+0000","last_clean":"2024-01-11T09:45:37.950338+0000","last_became_active":"2024-01-11T09:45:24.145584+0000","last_became_peered":"2024-01-11T09:45:24.145584+0000","last_unstale":"2024-01-11T09:45:37.950338+0000","last_undegraded":"2024-01-11T09:45:37.950338+0000","last_fullsized":"2024-01-11T09:45:37.950338+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:40:47.970580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.979066+0000","last_change":"2024-01-11T09:45:37.967560+0000","last_active":"2024-01-11T09:45:37.979066+0000","last_peered":"2024-01-11T09:45:37.979066+0000","last_clean":"2024-01-11T09:45:37.979066+0000","last_became_active":"2024-01-11T09:45:37.967314+0000","last_became_peered":"2024-01-11T09:45:37.967314+0000","last_unstale":"2024-01-11T09:45:37.979066+0000","last_undegraded":"2024-01-11T09:45:37.979066+0000","last_fullsized":"2024-01-11T09:45:37.979066+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:58:00.063180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.980237+0000","last_change":"2024-01-11T09:45:37.974580+0000","last_active":"2024-01-11T09:45:37.980237+0000","last_peered":"2024-01-11T09:45:37.980237+0000","last_clean":"2024-01-11T09:45:37.980237+0000","last_became_active":"2024-01-11T09:45:37.974273+0000","last_became_peered":"2024-01-11T09:45:37.974273+0000","last_unstale":"2024-01-11T09:45:37.980237+0000","last_undegraded":"2024-01-11T09:45:37.980237+0000","last_fullsized":"2024-01-11T09:45:37.980237+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:28:23.210266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954058+0000","last_change":"2024-01-11T09:45:24.168887+0000","last_active":"2024-01-11T09:45:37.954058+0000","last_peered":"2024-01-11T09:45:37.954058+0000","last_clean":"2024-01-11T09:45:37.954058+0000","last_became_active":"2024-01-11T09:45:24.167138+0000","last_became_peered":"2024-01-11T09:45:24.167138+0000","last_unstale":"2024-01-11T09:45:37.954058+0000","last_undegraded":"2024-01-11T09:45:37.954058+0000","last_fullsized":"2024-01-11T09:45:37.954058+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:43:22.330951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.982381+0000","last_change":"2024-01-11T09:45:37.980159+0000","last_active":"2024-01-11T09:45:37.982381+0000","last_peered":"2024-01-11T09:45:37.982381+0000","last_clean":"2024-01-11T09:45:37.982381+0000","last_became_active":"2024-01-11T09:45:37.976703+0000","last_became_peered":"2024-01-11T09:45:37.976703+0000","last_unstale":"2024-01-11T09:45:37.982381+0000","last_undegraded":"2024-01-11T09:45:37.982381+0000","last_fullsized":"2024-01-11T09:45:37.982381+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:49:01.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950736+0000","last_change":"2024-01-11T09:45:24.158003+0000","last_active":"2024-01-11T09:45:37.950736+0000","last_peered":"2024-01-11T09:45:37.950736+0000","last_clean":"2024-01-11T09:45:37.950736+0000","last_became_active":"2024-01-11T09:45:24.157710+0000","last_became_peered":"2024-01-11T09:45:24.157710+0000","last_unstale":"2024-01-11T09:45:37.950736+0000","last_undegraded":"2024-01-11T09:45:37.950736+0000","last_fullsized":"2024-01-11T09:45:37.950736+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:52:18.543824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.971932+0000","last_change":"2024-01-11T09:45:37.965176+0000","last_active":"2024-01-11T09:45:37.971932+0000","last_peered":"2024-01-11T09:45:37.971932+0000","last_clean":"2024-01-11T09:45:37.971932+0000","last_became_active":"2024-01-11T09:45:37.964838+0000","last_became_peered":"2024-01-11T09:45:37.964838+0000","last_unstale":"2024-01-11T09:45:37.971932+0000","last_undegraded":"2024-01-11T09:45:37.971932+0000","last_fullsized":"2024-01-11T09:45:37.971932+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:32:49.312144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950589+0000","last_change":"2024-01-11T09:45:24.145558+0000","last_active":"2024-01-11T09:45:37.950589+0000","last_peered":"2024-01-11T09:45:37.950589+0000","last_clean":"2024-01-11T09:45:37.950589+0000","last_became_active":"2024-01-11T09:45:24.143216+0000","last_became_peered":"2024-01-11T09:45:24.143216+0000","last_unstale":"2024-01-11T09:45:37.950589+0000","last_undegraded":"2024-01-11T09:45:37.950589+0000","last_fullsized":"2024-01-11T09:45:37.950589+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:32:19.368822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.988413+0000","last_change":"2024-01-11T09:45:37.983715+0000","last_active":"2024-01-11T09:45:37.988413+0000","last_peered":"2024-01-11T09:45:37.988413+0000","last_clean":"2024-01-11T09:45:37.988413+0000","last_became_active":"2024-01-11T09:45:37.983461+0000","last_became_peered":"2024-01-11T09:45:37.983461+0000","last_unstale":"2024-01-11T09:45:37.988413+0000","last_undegraded":"2024-01-11T09:45:37.988413+0000","last_fullsized":"2024-01-11T09:45:37.988413+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:47:37.322693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954769+0000","last_change":"2024-01-11T09:45:24.167154+0000","last_active":"2024-01-11T09:45:37.954769+0000","last_peered":"2024-01-11T09:45:37.954769+0000","last_clean":"2024-01-11T09:45:37.954769+0000","last_became_active":"2024-01-11T09:45:24.166862+0000","last_became_peered":"2024-01-11T09:45:24.166862+0000","last_unstale":"2024-01-11T09:45:37.954769+0000","last_undegraded":"2024-01-11T09:45:37.954769+0000","last_fullsized":"2024-01-11T09:45:37.954769+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:15:00.803121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.979055+0000","last_change":"2024-01-11T09:45:37.972382+0000","last_active":"2024-01-11T09:45:37.979055+0000","last_peered":"2024-01-11T09:45:37.979055+0000","last_clean":"2024-01-11T09:45:37.979055+0000","last_became_active":"2024-01-11T09:45:37.970224+0000","last_became_peered":"2024-01-11T09:45:37.970224+0000","last_unstale":"2024-01-11T09:45:37.979055+0000","last_undegraded":"2024-01-11T09:45:37.979055+0000","last_fullsized":"2024-01-11T09:45:37.979055+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:46:18.798102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.966130+0000","last_change":"2024-01-11T09:45:24.156832+0000","last_active":"2024-01-11T09:45:37.966130+0000","last_peered":"2024-01-11T09:45:37.966130+0000","last_clean":"2024-01-11T09:45:37.966130+0000","last_became_active":"2024-01-11T09:45:24.156541+0000","last_became_peered":"2024-01-11T09:45:24.156541+0000","last_unstale":"2024-01-11T09:45:37.966130+0000","last_undegraded":"2024-01-11T09:45:37.966130+0000","last_fullsized":"2024-01-11T09:45:37.966130+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:49.368305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.979956+0000","last_change":"2024-01-11T09:45:37.967199+0000","last_active":"2024-01-11T09:45:37.979956+0000","last_peered":"2024-01-11T09:45:37.979956+0000","last_clean":"2024-01-11T09:45:37.979956+0000","last_became_active":"2024-01-11T09:45:37.966928+0000","last_became_peered":"2024-01-11T09:45:37.966928+0000","last_unstale":"2024-01-11T09:45:37.979956+0000","last_undegraded":"2024-01-11T09:45:37.979956+0000","last_fullsized":"2024-01-11T09:45:37.979956+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:29:41.388676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950156+0000","last_change":"2024-01-11T09:45:24.153140+0000","last_active":"2024-01-11T09:45:37.950156+0000","last_peered":"2024-01-11T09:45:37.950156+0000","last_clean":"2024-01-11T09:45:37.950156+0000","last_became_active":"2024-01-11T09:45:24.152890+0000","last_became_peered":"2024-01-11T09:45:24.152890+0000","last_unstale":"2024-01-11T09:45:37.950156+0000","last_undegraded":"2024-01-11T09:45:37.950156+0000","last_fullsized":"2024-01-11T09:45:37.950156+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:39.406018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.986364+0000","last_change":"2024-01-11T09:45:37.979637+0000","last_active":"2024-01-11T09:45:37.986364+0000","last_peered":"2024-01-11T09:45:37.986364+0000","last_clean":"2024-01-11T09:45:37.986364+0000","last_became_active":"2024-01-11T09:45:37.979380+0000","last_became_peered":"2024-01-11T09:45:37.979380+0000","last_unstale":"2024-01-11T09:45:37.986364+0000","last_undegraded":"2024-01-11T09:45:37.986364+0000","last_fullsized":"2024-01-11T09:45:37.986364+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:15:52.374499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.951564+0000","last_change":"2024-01-11T09:45:24.166373+0000","last_active":"2024-01-11T09:45:37.951564+0000","last_peered":"2024-01-11T09:45:37.951564+0000","last_clean":"2024-01-11T09:45:37.951564+0000","last_became_active":"2024-01-11T09:45:24.166090+0000","last_became_peered":"2024-01-11T09:45:24.166090+0000","last_unstale":"2024-01-11T09:45:37.951564+0000","last_undegraded":"2024-01-11T09:45:37.951564+0000","last_fullsized":"2024-01-11T09:45:37.951564+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:16:58.784096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.981477+0000","last_change":"2024-01-11T09:45:37.979710+0000","last_active":"2024-01-11T09:45:37.981477+0000","last_peered":"2024-01-11T09:45:37.981477+0000","last_clean":"2024-01-11T09:45:37.981477+0000","last_became_active":"2024-01-11T09:45:37.979392+0000","last_became_peered":"2024-01-11T09:45:37.979392+0000","last_unstale":"2024-01-11T09:45:37.981477+0000","last_undegraded":"2024-01-11T09:45:37.981477+0000","last_fullsized":"2024-01-11T09:45:37.981477+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:08:58.113291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.963215+0000","last_change":"2024-01-11T09:45:24.164171+0000","last_active":"2024-01-11T09:45:37.963215+0000","last_peered":"2024-01-11T09:45:37.963215+0000","last_clean":"2024-01-11T09:45:37.963215+0000","last_became_active":"2024-01-11T09:45:24.163522+0000","last_became_peered":"2024-01-11T09:45:24.163522+0000","last_unstale":"2024-01-11T09:45:37.963215+0000","last_undegraded":"2024-01-11T09:45:37.963215+0000","last_fullsized":"2024-01-11T09:45:37.963215+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:37:20.217033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.982287+0000","last_change":"2024-01-11T09:45:37.978335+0000","last_active":"2024-01-11T09:45:37.982287+0000","last_peered":"2024-01-11T09:45:37.982287+0000","last_clean":"2024-01-11T09:45:37.982287+0000","last_became_active":"2024-01-11T09:45:37.978064+0000","last_became_peered":"2024-01-11T09:45:37.978064+0000","last_unstale":"2024-01-11T09:45:37.982287+0000","last_undegraded":"2024-01-11T09:45:37.982287+0000","last_fullsized":"2024-01-11T09:45:37.982287+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:26:56.082604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.948834+0000","last_change":"2024-01-11T09:45:24.148308+0000","last_active":"2024-01-11T09:45:37.948834+0000","last_peered":"2024-01-11T09:45:37.948834+0000","last_clean":"2024-01-11T09:45:37.948834+0000","last_became_active":"2024-01-11T09:45:24.148023+0000","last_became_peered":"2024-01-11T09:45:24.148023+0000","last_unstale":"2024-01-11T09:45:37.948834+0000","last_undegraded":"2024-01-11T09:45:37.948834+0000","last_fullsized":"2024-01-11T09:45:37.948834+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:56:51.623335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984817+0000","last_change":"2024-01-11T09:45:37.977229+0000","last_active":"2024-01-11T09:45:37.984817+0000","last_peered":"2024-01-11T09:45:37.984817+0000","last_clean":"2024-01-11T09:45:37.984817+0000","last_became_active":"2024-01-11T09:45:37.976956+0000","last_became_peered":"2024-01-11T09:45:37.976956+0000","last_unstale":"2024-01-11T09:45:37.984817+0000","last_undegraded":"2024-01-11T09:45:37.984817+0000","last_fullsized":"2024-01-11T09:45:37.984817+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:30:49.529765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.958690+0000","last_change":"2024-01-11T09:45:24.170581+0000","last_active":"2024-01-11T09:45:37.958690+0000","last_peered":"2024-01-11T09:45:37.958690+0000","last_clean":"2024-01-11T09:45:37.958690+0000","last_became_active":"2024-01-11T09:45:24.170233+0000","last_became_peered":"2024-01-11T09:45:24.170233+0000","last_unstale":"2024-01-11T09:45:37.958690+0000","last_undegraded":"2024-01-11T09:45:37.958690+0000","last_fullsized":"2024-01-11T09:45:37.958690+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:50:50.820643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.982420+0000","last_change":"2024-01-11T09:45:37.971726+0000","last_active":"2024-01-11T09:45:37.982420+0000","last_peered":"2024-01-11T09:45:37.982420+0000","last_clean":"2024-01-11T09:45:37.982420+0000","last_became_active":"2024-01-11T09:45:37.971126+0000","last_became_peered":"2024-01-11T09:45:37.971126+0000","last_unstale":"2024-01-11T09:45:37.982420+0000","last_undegraded":"2024-01-11T09:45:37.982420+0000","last_fullsized":"2024-01-11T09:45:37.982420+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:08:58.113291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.956787+0000","last_change":"2024-01-11T09:45:24.157215+0000","last_active":"2024-01-11T09:45:37.956787+0000","last_peered":"2024-01-11T09:45:37.956787+0000","last_clean":"2024-01-11T09:45:37.956787+0000","last_became_active":"2024-01-11T09:45:24.156852+0000","last_became_peered":"2024-01-11T09:45:24.156852+0000","last_unstale":"2024-01-11T09:45:37.956787+0000","last_undegraded":"2024-01-11T09:45:37.956787+0000","last_fullsized":"2024-01-11T09:45:37.956787+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:42.716516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.974315+0000","last_change":"2024-01-11T09:45:37.971871+0000","last_active":"2024-01-11T09:45:37.974315+0000","last_peered":"2024-01-11T09:45:37.974315+0000","last_clean":"2024-01-11T09:45:37.974315+0000","last_became_active":"2024-01-11T09:45:37.971223+0000","last_became_peered":"2024-01-11T09:45:37.971223+0000","last_unstale":"2024-01-11T09:45:37.974315+0000","last_undegraded":"2024-01-11T09:45:37.974315+0000","last_fullsized":"2024-01-11T09:45:37.974315+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:26:56.082604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.967678+0000","last_change":"2024-01-11T09:45:24.158500+0000","last_active":"2024-01-11T09:45:37.967678+0000","last_peered":"2024-01-11T09:45:37.967678+0000","last_clean":"2024-01-11T09:45:37.967678+0000","last_became_active":"2024-01-11T09:45:24.158185+0000","last_became_peered":"2024-01-11T09:45:24.158185+0000","last_unstale":"2024-01-11T09:45:37.967678+0000","last_undegraded":"2024-01-11T09:45:37.967678+0000","last_fullsized":"2024-01-11T09:45:37.967678+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:52:38.581476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.970263+0000","last_change":"2024-01-11T09:45:37.968152+0000","last_active":"2024-01-11T09:45:37.970263+0000","last_peered":"2024-01-11T09:45:37.970263+0000","last_clean":"2024-01-11T09:45:37.970263+0000","last_became_active":"2024-01-11T09:45:37.967824+0000","last_became_peered":"2024-01-11T09:45:37.967824+0000","last_unstale":"2024-01-11T09:45:37.970263+0000","last_undegraded":"2024-01-11T09:45:37.970263+0000","last_fullsized":"2024-01-11T09:45:37.970263+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:09:00.141609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.972493+0000","last_change":"2024-01-11T09:45:24.146920+0000","last_active":"2024-01-11T09:45:37.972493+0000","last_peered":"2024-01-11T09:45:37.972493+0000","last_clean":"2024-01-11T09:45:37.972493+0000","last_became_active":"2024-01-11T09:45:24.146654+0000","last_became_peered":"2024-01-11T09:45:24.146654+0000","last_unstale":"2024-01-11T09:45:37.972493+0000","last_undegraded":"2024-01-11T09:45:37.972493+0000","last_fullsized":"2024-01-11T09:45:37.972493+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:42:21.543118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.963957+0000","last_change":"2024-01-11T09:45:37.960537+0000","last_active":"2024-01-11T09:45:37.963957+0000","last_peered":"2024-01-11T09:45:37.963957+0000","last_clean":"2024-01-11T09:45:37.963957+0000","last_became_active":"2024-01-11T09:45:37.959337+0000","last_became_peered":"2024-01-11T09:45:37.959337+0000","last_unstale":"2024-01-11T09:45:37.963957+0000","last_undegraded":"2024-01-11T09:45:37.963957+0000","last_fullsized":"2024-01-11T09:45:37.963957+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:04:55.352559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960969+0000","last_change":"2024-01-11T09:45:24.161300+0000","last_active":"2024-01-11T09:45:37.960969+0000","last_peered":"2024-01-11T09:45:37.960969+0000","last_clean":"2024-01-11T09:45:37.960969+0000","last_became_active":"2024-01-11T09:45:24.160940+0000","last_became_peered":"2024-01-11T09:45:24.160940+0000","last_unstale":"2024-01-11T09:45:37.960969+0000","last_undegraded":"2024-01-11T09:45:37.960969+0000","last_fullsized":"2024-01-11T09:45:37.960969+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:40:06.419759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.984070+0000","last_change":"2024-01-11T09:45:37.973489+0000","last_active":"2024-01-11T09:45:37.984070+0000","last_peered":"2024-01-11T09:45:37.984070+0000","last_clean":"2024-01-11T09:45:37.984070+0000","last_became_active":"2024-01-11T09:45:37.971292+0000","last_became_peered":"2024-01-11T09:45:37.971292+0000","last_unstale":"2024-01-11T09:45:37.984070+0000","last_undegraded":"2024-01-11T09:45:37.984070+0000","last_fullsized":"2024-01-11T09:45:37.984070+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:47:37.322693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.951480+0000","last_change":"2024-01-11T09:45:24.152214+0000","last_active":"2024-01-11T09:45:37.951480+0000","last_peered":"2024-01-11T09:45:37.951480+0000","last_clean":"2024-01-11T09:45:37.951480+0000","last_became_active":"2024-01-11T09:45:24.151938+0000","last_became_peered":"2024-01-11T09:45:24.151938+0000","last_unstale":"2024-01-11T09:45:37.951480+0000","last_undegraded":"2024-01-11T09:45:37.951480+0000","last_fullsized":"2024-01-11T09:45:37.951480+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:28:05.090713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.983891+0000","last_change":"2024-01-11T09:45:37.979205+0000","last_active":"2024-01-11T09:45:37.983891+0000","last_peered":"2024-01-11T09:45:37.983891+0000","last_clean":"2024-01-11T09:45:37.983891+0000","last_became_active":"2024-01-11T09:45:37.977222+0000","last_became_peered":"2024-01-11T09:45:37.977222+0000","last_unstale":"2024-01-11T09:45:37.983891+0000","last_undegraded":"2024-01-11T09:45:37.983891+0000","last_fullsized":"2024-01-11T09:45:37.983891+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:07:03.144535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950339+0000","last_change":"2024-01-11T09:45:24.155318+0000","last_active":"2024-01-11T09:45:37.950339+0000","last_peered":"2024-01-11T09:45:37.950339+0000","last_clean":"2024-01-11T09:45:37.950339+0000","last_became_active":"2024-01-11T09:45:24.153405+0000","last_became_peered":"2024-01-11T09:45:24.153405+0000","last_unstale":"2024-01-11T09:45:37.950339+0000","last_undegraded":"2024-01-11T09:45:37.950339+0000","last_fullsized":"2024-01-11T09:45:37.950339+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:52:32.496845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.983145+0000","last_change":"2024-01-11T09:45:37.977665+0000","last_active":"2024-01-11T09:45:37.983145+0000","last_peered":"2024-01-11T09:45:37.983145+0000","last_clean":"2024-01-11T09:45:37.983145+0000","last_became_active":"2024-01-11T09:45:37.974163+0000","last_became_peered":"2024-01-11T09:45:37.974163+0000","last_unstale":"2024-01-11T09:45:37.983145+0000","last_undegraded":"2024-01-11T09:45:37.983145+0000","last_fullsized":"2024-01-11T09:45:37.983145+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:12:31.003882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.961527+0000","last_change":"2024-01-11T09:45:24.169914+0000","last_active":"2024-01-11T09:45:37.961527+0000","last_peered":"2024-01-11T09:45:37.961527+0000","last_clean":"2024-01-11T09:45:37.961527+0000","last_became_active":"2024-01-11T09:45:24.169438+0000","last_became_peered":"2024-01-11T09:45:24.169438+0000","last_unstale":"2024-01-11T09:45:37.961527+0000","last_undegraded":"2024-01-11T09:45:37.961527+0000","last_fullsized":"2024-01-11T09:45:37.961527+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:43:22.330951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.974323+0000","last_change":"2024-01-11T09:45:37.970756+0000","last_active":"2024-01-11T09:45:37.974323+0000","last_peered":"2024-01-11T09:45:37.974323+0000","last_clean":"2024-01-11T09:45:37.974323+0000","last_became_active":"2024-01-11T09:45:37.970446+0000","last_became_peered":"2024-01-11T09:45:37.970446+0000","last_unstale":"2024-01-11T09:45:37.974323+0000","last_undegraded":"2024-01-11T09:45:37.974323+0000","last_fullsized":"2024-01-11T09:45:37.974323+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:06:21.286305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.958611+0000","last_change":"2024-01-11T09:45:24.164133+0000","last_active":"2024-01-11T09:45:37.958611+0000","last_peered":"2024-01-11T09:45:37.958611+0000","last_clean":"2024-01-11T09:45:37.958611+0000","last_became_active":"2024-01-11T09:45:24.163856+0000","last_became_peered":"2024-01-11T09:45:24.163856+0000","last_unstale":"2024-01-11T09:45:37.958611+0000","last_undegraded":"2024-01-11T09:45:37.958611+0000","last_fullsized":"2024-01-11T09:45:37.958611+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:17:24.413093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.978680+0000","last_change":"2024-01-11T09:45:37.968969+0000","last_active":"2024-01-11T09:45:37.978680+0000","last_peered":"2024-01-11T09:45:37.978680+0000","last_clean":"2024-01-11T09:45:37.978680+0000","last_became_active":"2024-01-11T09:45:37.968664+0000","last_became_peered":"2024-01-11T09:45:37.968664+0000","last_unstale":"2024-01-11T09:45:37.978680+0000","last_undegraded":"2024-01-11T09:45:37.978680+0000","last_fullsized":"2024-01-11T09:45:37.978680+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:08:58.113291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.961712+0000","last_change":"2024-01-11T09:45:24.153507+0000","last_active":"2024-01-11T09:45:37.961712+0000","last_peered":"2024-01-11T09:45:37.961712+0000","last_clean":"2024-01-11T09:45:37.961712+0000","last_became_active":"2024-01-11T09:45:24.153235+0000","last_became_peered":"2024-01-11T09:45:24.153235+0000","last_unstale":"2024-01-11T09:45:37.961712+0000","last_undegraded":"2024-01-11T09:45:37.961712+0000","last_fullsized":"2024-01-11T09:45:37.961712+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:01:06.710133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.988445+0000","last_change":"2024-01-11T09:45:37.985398+0000","last_active":"2024-01-11T09:45:37.988445+0000","last_peered":"2024-01-11T09:45:37.988445+0000","last_clean":"2024-01-11T09:45:37.988445+0000","last_became_active":"2024-01-11T09:45:37.984360+0000","last_became_peered":"2024-01-11T09:45:37.984360+0000","last_unstale":"2024-01-11T09:45:37.988445+0000","last_undegraded":"2024-01-11T09:45:37.988445+0000","last_fullsized":"2024-01-11T09:45:37.988445+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:42:31.879941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.963305+0000","last_change":"2024-01-11T09:45:24.148863+0000","last_active":"2024-01-11T09:45:37.963305+0000","last_peered":"2024-01-11T09:45:37.963305+0000","last_clean":"2024-01-11T09:45:37.963305+0000","last_became_active":"2024-01-11T09:45:24.148533+0000","last_became_peered":"2024-01-11T09:45:24.148533+0000","last_unstale":"2024-01-11T09:45:37.963305+0000","last_undegraded":"2024-01-11T09:45:37.963305+0000","last_fullsized":"2024-01-11T09:45:37.963305+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:47:20.218042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.957184+0000","last_change":"2024-01-11T09:45:37.954469+0000","last_active":"2024-01-11T09:45:37.957184+0000","last_peered":"2024-01-11T09:45:37.957184+0000","last_clean":"2024-01-11T09:45:37.957184+0000","last_became_active":"2024-01-11T09:45:37.954186+0000","last_became_peered":"2024-01-11T09:45:37.954186+0000","last_unstale":"2024-01-11T09:45:37.957184+0000","last_undegraded":"2024-01-11T09:45:37.957184+0000","last_fullsized":"2024-01-11T09:45:37.957184+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:06:21.286305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.954687+0000","last_change":"2024-01-11T09:45:24.149180+0000","last_active":"2024-01-11T09:45:37.954687+0000","last_peered":"2024-01-11T09:45:37.954687+0000","last_clean":"2024-01-11T09:45:37.954687+0000","last_became_active":"2024-01-11T09:45:24.148899+0000","last_became_peered":"2024-01-11T09:45:24.148899+0000","last_unstale":"2024-01-11T09:45:37.954687+0000","last_undegraded":"2024-01-11T09:45:37.954687+0000","last_fullsized":"2024-01-11T09:45:37.954687+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:59:56.132354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.979980+0000","last_change":"2024-01-11T09:45:37.971700+0000","last_active":"2024-01-11T09:45:37.979980+0000","last_peered":"2024-01-11T09:45:37.979980+0000","last_clean":"2024-01-11T09:45:37.979980+0000","last_became_active":"2024-01-11T09:45:37.971458+0000","last_became_peered":"2024-01-11T09:45:37.971458+0000","last_unstale":"2024-01-11T09:45:37.979980+0000","last_undegraded":"2024-01-11T09:45:37.979980+0000","last_fullsized":"2024-01-11T09:45:37.979980+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:04:55.352559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.960669+0000","last_change":"2024-01-11T09:45:24.160833+0000","last_active":"2024-01-11T09:45:37.960669+0000","last_peered":"2024-01-11T09:45:37.960669+0000","last_clean":"2024-01-11T09:45:37.960669+0000","last_became_active":"2024-01-11T09:45:24.160557+0000","last_became_peered":"2024-01-11T09:45:24.160557+0000","last_unstale":"2024-01-11T09:45:37.960669+0000","last_undegraded":"2024-01-11T09:45:37.960669+0000","last_fullsized":"2024-01-11T09:45:37.960669+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:29:13.303845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.971889+0000","last_change":"2024-01-11T09:45:37.963861+0000","last_active":"2024-01-11T09:45:37.971889+0000","last_peered":"2024-01-11T09:45:37.971889+0000","last_clean":"2024-01-11T09:45:37.971889+0000","last_became_active":"2024-01-11T09:45:37.963598+0000","last_became_peered":"2024-01-11T09:45:37.963598+0000","last_unstale":"2024-01-11T09:45:37.971889+0000","last_undegraded":"2024-01-11T09:45:37.971889+0000","last_fullsized":"2024-01-11T09:45:37.971889+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:52:26.173608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.950401+0000","last_change":"2024-01-11T09:45:24.147655+0000","last_active":"2024-01-11T09:45:37.950401+0000","last_peered":"2024-01-11T09:45:37.950401+0000","last_clean":"2024-01-11T09:45:37.950401+0000","last_became_active":"2024-01-11T09:45:24.147267+0000","last_became_peered":"2024-01-11T09:45:24.147267+0000","last_unstale":"2024-01-11T09:45:37.950401+0000","last_undegraded":"2024-01-11T09:45:37.950401+0000","last_fullsized":"2024-01-11T09:45:37.950401+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:14:09.653181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":9,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.986187+0000","last_change":"2024-01-11T09:45:37.980685+0000","last_active":"2024-01-11T09:45:37.986187+0000","last_peered":"2024-01-11T09:45:37.986187+0000","last_clean":"2024-01-11T09:45:37.986187+0000","last_became_active":"2024-01-11T09:45:37.980384+0000","last_became_peered":"2024-01-11T09:45:37.980384+0000","last_unstale":"2024-01-11T09:45:37.986187+0000","last_undegraded":"2024-01-11T09:45:37.986187+0000","last_fullsized":"2024-01-11T09:45:37.986187+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:36.939892+0000","last_clean_scrub_stamp":"2024-01-11T09:45:36.939892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:39:49.483450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":13,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-01-11T09:45:37.963124+0000","last_change":"2024-01-11T09:45:24.164535+0000","last_active":"2024-01-11T09:45:37.963124+0000","last_peered":"2024-01-11T09:45:37.963124+0000","last_clean":"2024-01-11T09:45:37.963124+0000","last_became_active":"2024-01-11T09:45:24.163889+0000","last_became_peered":"2024-01-11T09:45:24.163889+0000","last_unstale":"2024-01-11T09:45:37.963124+0000","last_undegraded":"2024-01-11T09:45:37.963124+0000","last_fullsized":"2024-01-11T09:45:37.963124+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:45:23.120580+0000","last_clean_scrub_stamp":"2024-01-11T09:45:23.120580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:29:27.569363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3519,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2288,"num_read_kb":4808,"num_write":611,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":366,"ondisk_log_size":366,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575975,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300556,"kb_used_data":3268,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071284,"statfs":{"total":96636764160,"available":96328994816,"internally_reserved":0,"allocated":3346432,"data_stored":1904090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.69599999999999995,"15min":0.73299999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.36599999999999999},"max":{"1min":0.93000000000000005,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.878},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.72599999999999998,"15min":0.747},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.026,"5min":1.0609999999999999,"15min":1.0669999999999999},"last":0.73099999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.70699999999999996,"15min":0.73299999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41599999999999998},"max":{"1min":0.91600000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.71799999999999997,"15min":0.74199999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.95899999999999996,"5min":0.999,"15min":1.0820000000000001},"last":0.89400000000000002}]},{"osd":2,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.73599999999999999,"15min":0.74399999999999999},"min":{"1min":0.44500000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.93799999999999994,"5min":0.94399999999999995,"15min":1.0600000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73199999999999998,"15min":0.754},"min":{"1min":0.503,"5min":0.44800000000000001,"15min":0.44500000000000001},"max":{"1min":0.92800000000000005,"5min":0.96999999999999997,"15min":1.153},"last":0.86099999999999999}]},{"osd":3,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73799999999999999,"15min":0.76700000000000002},"min":{"1min":0.51600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.93000000000000005,"5min":0.96699999999999997,"15min":1.0660000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73699999999999999,"15min":0.75600000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.94099999999999995,"5min":1,"15min":1.0880000000000001},"last":0.71199999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.432,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0720000000000001,"5min":1.0860000000000001,"15min":1.107},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.629,"15min":0.63800000000000001},"min":{"1min":0.47699999999999998,"5min":0.44400000000000001,"15min":0.39700000000000002},"max":{"1min":0.89000000000000001,"5min":0.995,"15min":1.0960000000000001},"last":0.58899999999999997}]},{"osd":5,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.61199999999999999,"15min":0.63800000000000001},"min":{"1min":0.48999999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86599999999999999,"5min":0.86699999999999999,"15min":1.0109999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.67900000000000005,"15min":0.68400000000000005},"min":{"1min":0.46899999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.65700000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.76300000000000001,"15min":0.77900000000000003},"min":{"1min":0.55900000000000005,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.747,"15min":0.76000000000000001},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.45800000000000002},"max":{"1min":0.95899999999999996,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.83299999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575974,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312260,"kb_used_data":2684,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059580,"statfs":{"total":96636764160,"available":96317009920,"internally_reserved":0,"allocated":2748416,"data_stored":1311513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67800000000000005,"15min":0.71399999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83899999999999997,"5min":0.94199999999999995,"15min":1.508},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.71899999999999997,"15min":0.748},"min":{"1min":0.501,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.92600000000000005,"5min":0.97099999999999997,"15min":1.5589999999999999},"last":0.65500000000000003}]},{"osd":1,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.69199999999999995,"15min":0.74099999999999999},"min":{"1min":0.42799999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84799999999999998,"5min":0.94699999999999995,"15min":1.45},"last":0.747},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.70599999999999996,"15min":0.73699999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.79400000000000004,"5min":0.98299999999999998,"15min":1.542},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.72599999999999998,"15min":0.752},"min":{"1min":0.52500000000000002,"5min":0.501,"15min":0.44500000000000001},"max":{"1min":0.85899999999999999,"5min":1.034,"15min":1.5329999999999999},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.72399999999999998,"15min":0.75700000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.89500000000000002,"5min":0.93999999999999995,"15min":1.647},"last":0.51500000000000001}]},{"osd":3,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.747,"15min":0.76500000000000001},"min":{"1min":0.54900000000000004,"5min":0.502,"15min":0.45200000000000001},"max":{"1min":0.88200000000000001,"5min":1.012,"15min":1.6140000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.72699999999999998,"15min":0.77400000000000002},"min":{"1min":0.53600000000000003,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.83499999999999996,"5min":0.96499999999999997,"15min":1.4950000000000001},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86899999999999999,"5min":0.91700000000000004,"15min":1.0309999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61499999999999999,"15min":0.61199999999999999},"min":{"1min":0.42499999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.86199999999999999,"5min":0.94799999999999995,"15min":1.1259999999999999},"last":0.88500000000000001}]},{"osd":5,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64700000000000002,"15min":0.65900000000000003},"min":{"1min":0.45800000000000002,"5min":0.41499999999999998,"15min":0.40000000000000002},"max":{"1min":0.92700000000000005,"5min":0.97899999999999998,"15min":1.369},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.49099999999999999,"5min":0.46700000000000003,"15min":0.45400000000000001},"max":{"1min":0.89400000000000002,"5min":0.999,"15min":1.0960000000000001},"last":0.78500000000000003}]},{"osd":7,"last update":"Thu Jan 11 09:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70099999999999996,"15min":0.70799999999999996},"min":{"1min":0.53000000000000003,"5min":0.45100000000000001,"15min":0.44400000000000001},"max":{"1min":0.91900000000000004,"5min":1.089,"15min":1.484},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.755,"5min":0.76100000000000001,"15min":0.76900000000000002},"min":{"1min":0.48699999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0609999999999999,"5min":1.145,"15min":1.603},"last":0.98299999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575976,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308164,"kb_used_data":2684,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063676,"statfs":{"total":96636764160,"available":96321204224,"internally_reserved":0,"allocated":2748416,"data_stored":1311049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52500000000000002,"15min":0.51500000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.315},"max":{"1min":0.66000000000000003,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61399999999999999,"15min":0.622},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.748,"5min":0.85199999999999998,"15min":0.95099999999999996},"last":0.47799999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.51300000000000001,"15min":0.51400000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.64000000000000001,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60099999999999998,"15min":0.59399999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39400000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.63800000000000001}]},{"osd":3,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64400000000000002,"15min":0.65000000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.42699999999999999},"max":{"1min":0.81200000000000006,"5min":0.91600000000000004,"15min":0.96699999999999997},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.40400000000000003},"max":{"1min":0.70499999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.436}]},{"osd":4,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.81899999999999995,"15min":0.82399999999999995},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.505},"max":{"1min":1.1080000000000001,"5min":1.214,"15min":1.2969999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.879,"5min":0.85999999999999999,"15min":0.86399999999999999},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.61699999999999999},"max":{"1min":1.0129999999999999,"5min":1.1879999999999999,"15min":1.1890000000000001},"last":0.72299999999999998}]},{"osd":5,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83899999999999997,"15min":0.83299999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.093,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.81799999999999995,"15min":0.80700000000000005},"min":{"1min":0.54200000000000004,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":1.0569999999999999,"5min":1.341,"15min":1.341},"last":0.753}]},{"osd":6,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.84599999999999997,"15min":0.84899999999999998},"min":{"1min":0.67000000000000004,"5min":0.59599999999999997,"15min":0.56799999999999995},"max":{"1min":1.073,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.85099999999999998,"15min":0.85499999999999998},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.55900000000000005},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.624}]},{"osd":7,"last update":"Thu Jan 11 09:44:41 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89600000000000002,"15min":0.89100000000000001},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.143,"5min":1.421,"15min":1.421},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.86499999999999999,"15min":0.85899999999999999},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.54500000000000004},"max":{"1min":1.083,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.81200000000000006}]}]},{"osd":0,"up_from":13,"seq":55834575975,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308744,"kb_used_data":3264,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063096,"statfs":{"total":96636764160,"available":96320610304,"internally_reserved":0,"allocated":3342336,"data_stored":1902005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34499999999999997,"5min":0.34300000000000003,"15min":0.32100000000000001},"max":{"1min":0.754,"5min":0.84699999999999998,"15min":0.875},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.60099999999999998,"15min":0.60199999999999998},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.81599999999999995,"5min":0.86399999999999999,"15min":0.874},"last":0.51300000000000001}]},{"osd":2,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.56799999999999995,"15min":0.57299999999999995},"min":{"1min":0.42499999999999999,"5min":0.40400000000000003,"15min":0.36599999999999999},"max":{"1min":0.79900000000000004,"5min":0.84499999999999997,"15min":0.97499999999999998},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.438,"5min":0.39600000000000002,"15min":0.38500000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.86399999999999999},"last":0.55400000000000005}]},{"osd":3,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54000000000000004,"15min":0.54600000000000004},"min":{"1min":0.42499999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69199999999999995,"5min":0.92600000000000005,"15min":0.95399999999999996},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64700000000000002,"15min":0.65600000000000003},"min":{"1min":0.46800000000000003,"5min":0.45900000000000002,"15min":0.41999999999999998},"max":{"1min":0.77800000000000002,"5min":0.92600000000000005,"15min":1.0389999999999999},"last":0.52800000000000002}]},{"osd":4,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.83099999999999996,"15min":0.82299999999999995},"min":{"1min":0.60499999999999998,"5min":0.60299999999999998,"15min":0.57399999999999995},"max":{"1min":1.0569999999999999,"5min":1.1779999999999999,"15min":1.304},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.81699999999999995,"15min":0.81000000000000005},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.272,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.78000000000000003}]},{"osd":5,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.84299999999999997,"15min":0.84399999999999997},"min":{"1min":0.67200000000000004,"5min":0.65000000000000002,"15min":0.53900000000000003},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.84599999999999997,"15min":0.84799999999999998},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.153,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.80500000000000005}]},{"osd":6,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83499999999999996,"15min":0.82799999999999996},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.55700000000000005},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.871,"5min":0.872,"15min":0.875},"min":{"1min":0.63800000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.1699999999999999,"5min":1.256,"15min":1.256},"last":0.73699999999999999}]},{"osd":7,"last update":"Thu Jan 11 09:45:09 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.877,"15min":0.871},"min":{"1min":0.66100000000000003,"5min":0.59099999999999997,"15min":0.58699999999999997},"max":{"1min":1.226,"5min":1.28,"15min":1.28},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.88100000000000001,"15min":0.879},"min":{"1min":0.65300000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.1259999999999999,"5min":1.238,"15min":1.2410000000000001},"last":0.65900000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575974,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312268,"kb_used_data":2692,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059572,"statfs":{"total":96636764160,"available":96317001728,"internally_reserved":0,"allocated":2756608,"data_stored":1312495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.49299999999999999,"15min":0.505},"min":{"1min":0.379,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.627,"5min":0.78800000000000003,"15min":0.85399999999999998},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55200000000000005,"15min":0.56499999999999995},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34300000000000003},"max":{"1min":0.71099999999999997,"5min":1.131,"15min":1.6359999999999999},"last":0.63}]},{"osd":1,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.38200000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.751,"5min":0.88800000000000001,"15min":0.93000000000000005},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60199999999999998,"15min":0.60399999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.40000000000000002},"max":{"1min":0.77400000000000002,"5min":0.97299999999999998,"15min":1.145},"last":0.73099999999999998}]},{"osd":3,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.42999999999999999,"5min":0.377,"15min":0.36199999999999999},"max":{"1min":0.76800000000000002,"5min":0.83099999999999996,"15min":0.96399999999999997},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.66200000000000003},"min":{"1min":0.50600000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.876,"5min":0.92300000000000004,"15min":1.3620000000000001},"last":0.754}]},{"osd":4,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79000000000000004,"15min":0.80300000000000005},"min":{"1min":0.63400000000000001,"5min":0.57799999999999996,"15min":0.54300000000000004},"max":{"1min":0.97199999999999998,"5min":1.071,"15min":1.1519999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.80100000000000005,"15min":0.80400000000000005},"min":{"1min":0.56200000000000006,"5min":0.505,"15min":0.505},"max":{"1min":1.1439999999999999,"5min":1.161,"15min":1.161},"last":0.80600000000000005}]},{"osd":5,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.85899999999999999,"15min":0.877},"min":{"1min":0.59699999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.228,"5min":1.228,"15min":1.429},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.85799999999999998,"15min":0.86899999999999999},"min":{"1min":0.65900000000000003,"5min":0.64700000000000002,"15min":0.61099999999999999},"max":{"1min":1.0629999999999999,"5min":1.1850000000000001,"15min":1.2170000000000001},"last":0.84399999999999997}]},{"osd":6,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.81899999999999995,"15min":0.83099999999999996},"min":{"1min":0.59199999999999997,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.091,"5min":1.1659999999999999,"15min":1.4139999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.87,"5min":0.84799999999999998,"15min":0.873},"min":{"1min":0.68300000000000005,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.173,"5min":1.2010000000000001,"15min":1.593},"last":0.96599999999999997}]},{"osd":7,"last update":"Thu Jan 11 09:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.88,"15min":0.89800000000000002},"min":{"1min":0.72299999999999998,"5min":0.623,"15min":0.57299999999999995},"max":{"1min":1.111,"5min":1.137,"15min":2.0539999999999998},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.88500000000000001,"15min":0.88600000000000001},"min":{"1min":0.67600000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1919999999999999,"5min":1.2270000000000001,"15min":1.2490000000000001},"last":1.069}]}]},{"osd":3,"up_from":13,"seq":55834575974,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299988,"kb_used_data":2700,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071852,"statfs":{"total":96636764160,"available":96329576448,"internally_reserved":0,"allocated":2764800,"data_stored":1312691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.502,"15min":0.52000000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.314},"max":{"1min":0.76100000000000001,"5min":0.89700000000000002,"15min":0.90000000000000002},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.56200000000000006,"15min":0.58699999999999997},"min":{"1min":0.372,"5min":0.36899999999999999,"15min":0.36099999999999999},"max":{"1min":0.67900000000000005,"5min":1.014,"15min":1.0449999999999999},"last":0.44600000000000001}]},{"osd":1,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.495,"15min":0.51100000000000001},"min":{"1min":0.29399999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63500000000000001,"5min":0.84199999999999997,"15min":1.304},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.38700000000000001},"max":{"1min":0.75,"5min":0.84099999999999997,"15min":1.4590000000000001},"last":0.47499999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58299999999999996,"15min":0.60599999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.72999999999999998,"5min":0.93700000000000006,"15min":1.04},"last":0.502},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.64200000000000002,"15min":0.65800000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44700000000000001},"max":{"1min":0.83499999999999996,"5min":0.89800000000000002,"15min":0.92000000000000004},"last":0.52700000000000002}]},{"osd":4,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76100000000000001,"15min":0.79000000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.92900000000000005,"5min":1.206,"15min":1.206},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.77800000000000002,"15min":0.81299999999999994},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.871,"5min":1.171,"15min":1.1830000000000001},"last":0.74099999999999999}]},{"osd":5,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.79600000000000004,"15min":0.82099999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.95399999999999996,"5min":1.1619999999999999,"15min":1.234},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81000000000000005,"15min":0.84799999999999998},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.3440000000000001},"last":0.83399999999999996}]},{"osd":6,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82999999999999996,"15min":0.85899999999999999},"min":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.077,"5min":1.1739999999999999,"15min":1.2170000000000001},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.80500000000000005,"15min":0.82899999999999996},"min":{"1min":0.55300000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.97099999999999997,"5min":1.181,"15min":1.2190000000000001},"last":0.96699999999999997}]},{"osd":7,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.83699999999999997,"15min":0.86799999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0269999999999999,"5min":1.224,"15min":1.264},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.86099999999999999,"15min":0.89500000000000002},"min":{"1min":0.59299999999999997,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.0209999999999999,"5min":1.1539999999999999,"15min":1.24},"last":0.93200000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575975,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308176,"kb_used_data":2696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063664,"statfs":{"total":96636764160,"available":96321191936,"internally_reserved":0,"allocated":2760704,"data_stored":1314310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.75800000000000001,"15min":0.77200000000000002},"min":{"1min":0.54000000000000004,"5min":0.41199999999999998,"15min":0.39900000000000002},"max":{"1min":0.91200000000000003,"5min":0.98499999999999999,"15min":1.119},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.98499999999999999,"5min":1.853,"15min":1.853},"last":0.72499999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.73099999999999998,"15min":0.74399999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.41399999999999998},"max":{"1min":0.90500000000000003,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.72999999999999998,"15min":0.73599999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.95899999999999996,"5min":1.639,"15min":1.639},"last":0.621}]},{"osd":2,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.747,"15min":0.751},"min":{"1min":0.54800000000000004,"5min":0.46500000000000002,"15min":0.44700000000000001},"max":{"1min":0.94499999999999995,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.75800000000000001,"15min":0.752},"min":{"1min":0.503,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.002,"5min":1.0249999999999999,"15min":1.0700000000000001},"last":0.55500000000000005}]},{"osd":3,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.73899999999999999,"15min":0.745},"min":{"1min":0.52100000000000002,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.94799999999999995,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.78300000000000003,"15min":0.78000000000000003},"min":{"1min":0.56499999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.02,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.69399999999999995}]},{"osd":5,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.64100000000000001,"15min":0.624},"min":{"1min":0.42999999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.90300000000000002,"5min":0.91500000000000004,"15min":1.0049999999999999},"last":0.503},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.66600000000000004,"15min":0.64600000000000002},"min":{"1min":0.46700000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89500000000000002,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.78600000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.69099999999999995,"15min":0.67100000000000004},"min":{"1min":0.51000000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83899999999999997,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.72699999999999998,"15min":0.70699999999999996},"min":{"1min":0.51800000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95199999999999996,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.80600000000000005}]},{"osd":7,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.72499999999999998,"15min":0.70099999999999996},"min":{"1min":0.56399999999999995,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0489999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.78000000000000003,"15min":0.754},"min":{"1min":0.57499999999999996,"5min":0.436,"15min":0.41699999999999998},"max":{"1min":1.02,"5min":2.052,"15min":2.052},"last":0.89100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575975,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312252,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059588,"statfs":{"total":96636764160,"available":96317018112,"internally_reserved":0,"allocated":2740224,"data_stored":1311457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.74099999999999999,"15min":0.73299999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":0.95099999999999996,"5min":0.97799999999999998,"15min":1.0029999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71199999999999997,"15min":0.72399999999999998},"min":{"1min":0.52000000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.91100000000000003,"5min":1.048,"15min":1.0549999999999999},"last":0.82599999999999996}]},{"osd":1,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71899999999999997,"15min":0.73499999999999999},"min":{"1min":0.48499999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.88300000000000001,"5min":0.95699999999999996,"15min":0.97699999999999998},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.74299999999999999,"15min":0.74199999999999999},"min":{"1min":0.5,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.92600000000000005,"5min":1.004,"15min":1.0269999999999999},"last":0.84999999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.72599999999999998,"15min":0.72999999999999998},"min":{"1min":0.504,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92600000000000005,"5min":1.0269999999999999,"15min":1.028},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.74299999999999999,"15min":0.746},"min":{"1min":0.53500000000000003,"5min":0.439,"15min":0.439},"max":{"1min":0.89900000000000002,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.73399999999999999}]},{"osd":3,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.74299999999999999,"15min":0.75700000000000001},"min":{"1min":0.58499999999999996,"5min":0.52800000000000002,"15min":0.49299999999999999},"max":{"1min":0.90200000000000002,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.76400000000000001,"15min":0.76900000000000002},"min":{"1min":0.56999999999999995,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.94499999999999995,"5min":1.004,"15min":1.1000000000000001},"last":0.753}]},{"osd":4,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61799999999999999,"15min":0.61099999999999999},"min":{"1min":0.443,"5min":0.41999999999999998,"15min":0.41699999999999998},"max":{"1min":0.91000000000000003,"5min":0.94699999999999995,"15min":0.97199999999999998},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.63500000000000001,"15min":0.63300000000000001},"min":{"1min":0.46999999999999997,"5min":0.40200000000000002,"15min":0.375},"max":{"1min":0.85799999999999998,"5min":0.93100000000000005,"15min":0.94499999999999995},"last":0.55600000000000005}]},{"osd":6,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63300000000000001,"15min":0.64100000000000001},"min":{"1min":0.48099999999999998,"5min":0.42099999999999999,"15min":0.40999999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":1.0089999999999999},"last":0.628},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68300000000000005,"15min":0.68700000000000006},"min":{"1min":0.47299999999999998,"5min":0.46200000000000002,"15min":0.42799999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":1.0229999999999999},"last":0.60699999999999998}]},{"osd":7,"last update":"Thu Jan 11 09:45:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.434},"max":{"1min":0.88600000000000001,"5min":1.0309999999999999,"15min":1.0760000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77700000000000002},"min":{"1min":0.56799999999999995,"5min":0.52700000000000002,"15min":0.47599999999999998},"max":{"1min":0.97099999999999997,"5min":1.0429999999999999,"15min":1.0469999999999999},"last":0.68000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:45:42.885 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-11T09:45:42.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:45:43.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.125+0000 7f186a5ff700 1 Processor -- start 2024-01-11T09:45:43.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.125+0000 7f186a5ff700 1 -- start start 2024-01-11T09:45:43.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.126+0000 7f186a5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18640b19d0 0x7f1864158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:43.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.126+0000 7f186a5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f186406b610 0x7f1864159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:43.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.126+0000 7f186a5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1864158b60 con 0x7f18641575e0 2024-01-11T09:45:43.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.126+0000 7f186a5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18640cb5e0 con 0x7f186406b610 2024-01-11T09:45:43.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.126+0000 7f186a5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18641599b0 con 0x7f18640b19d0 2024-01-11T09:45:43.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.127+0000 7f1868b9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f186406b610 0x7f1864159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:43.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.127+0000 7f1868b9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f186406b610 0x7f1864159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47582/0 (socket says 172.21.15.27:47582) 2024-01-11T09:45:43.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.127+0000 7f1868b9c700 1 -- 172.21.15.27:0/3173877260 learned_addr learned my addr 172.21.15.27:0/3173877260 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:43.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.127+0000 7f18637fe700 1 --2- 172.21.15.27:0/3173877260 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18640b19d0 0x7f1864158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:43.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.127+0000 7f1868b9c700 1 -- 172.21.15.27:0/3173877260 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18640b19d0 msgr2=0x7f1864158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:43.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.127+0000 7f1868b9c700 1 --2- 172.21.15.27:0/3173877260 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18640b19d0 0x7f1864158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:43.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.127+0000 7f1868b9c700 1 -- 172.21.15.27:0/3173877260 >> v1:172.21.15.77:6789/0 conn(0x7f18641575e0 legacy=0x7f18640b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:43.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.127+0000 7f1868b9c700 1 -- 172.21.15.27:0/3173877260 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18640c0f90 con 0x7f186406b610 2024-01-11T09:45:43.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.127+0000 7f1868b9c700 1 --2- 172.21.15.27:0/3173877260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f186406b610 0x7f1864159e40 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f185c009f20 tx=0x7f185c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d1c440640fd1805 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:43.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.128+0000 7f1862ffd700 1 -- 172.21.15.27:0/3173877260 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c00e070 con 0x7f186406b610 2024-01-11T09:45:43.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.128+0000 7f1862ffd700 1 -- 172.21.15.27:0/3173877260 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185c00b9d0 con 0x7f186406b610 2024-01-11T09:45:43.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.128+0000 7f1862ffd700 1 -- 172.21.15.27:0/3173877260 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c0092e0 con 0x7f186406b610 2024-01-11T09:45:43.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.128+0000 7f186a5ff700 1 -- 172.21.15.27:0/3173877260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f186406b610 msgr2=0x7f1864159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:43.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.128+0000 7f186a5ff700 1 --2- 172.21.15.27:0/3173877260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f186406b610 0x7f1864159e40 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f185c009f20 tx=0x7f185c009840 comp rx=0 tx=0).stop 2024-01-11T09:45:43.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.128+0000 7f186a5ff700 1 -- 172.21.15.27:0/3173877260 shutdown_connections 2024-01-11T09:45:43.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.128+0000 7f186a5ff700 1 --2- 172.21.15.27:0/3173877260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f186406b610 0x7f1864159e40 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:43.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.128+0000 7f186a5ff700 1 --2- 172.21.15.27:0/3173877260 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18640b19d0 0x7f1864158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:43.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.128+0000 7f186a5ff700 1 -- 172.21.15.27:0/3173877260 >> 172.21.15.27:0/3173877260 conn(0x7f18640680e0 msgr2=0x7f1864069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:43.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.128+0000 7f186a5ff700 1 -- 172.21.15.27:0/3173877260 shutdown_connections 2024-01-11T09:45:43.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.129+0000 7f186a5ff700 1 -- 172.21.15.27:0/3173877260 wait complete. 2024-01-11T09:45:43.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.130+0000 7f186a5ff700 1 Processor -- start 2024-01-11T09:45:43.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.130+0000 7f186a5ff700 1 -- start start 2024-01-11T09:45:43.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.131+0000 7f186a5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f186406b610 0x7f1864140680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:43.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.131+0000 7f186a5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18640b19d0 0x7f186415e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:43.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.131+0000 7f186a5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18641575e0 0x7f186415e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:43.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.131+0000 7f186a5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186415c710 con 0x7f186406b610 2024-01-11T09:45:43.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.131+0000 7f186a5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186415bb00 con 0x7f18641575e0 2024-01-11T09:45:43.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.131+0000 7f186a5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f186415bde0 con 0x7f18640b19d0 2024-01-11T09:45:43.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.131+0000 7f1863fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f186406b610 0x7f1864140680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:43.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.131+0000 7f1863fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f186406b610 0x7f1864140680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47588/0 (socket says 172.21.15.27:47588) 2024-01-11T09:45:43.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.131+0000 7f1863fff700 1 -- 172.21.15.27:0/2796664287 learned_addr learned my addr 172.21.15.27:0/2796664287 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:43.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.132+0000 7f18637fe700 1 --2- 172.21.15.27:0/2796664287 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18640b19d0 0x7f186415e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:43.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.132+0000 7f1868b9c700 1 --2- 172.21.15.27:0/2796664287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18641575e0 0x7f186415e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:43.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.132+0000 7f1863fff700 1 -- 172.21.15.27:0/2796664287 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18640b19d0 msgr2=0x7f186415e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:43.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.132+0000 7f1863fff700 1 --2- 172.21.15.27:0/2796664287 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18640b19d0 0x7f186415e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:43.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.132+0000 7f1863fff700 1 -- 172.21.15.27:0/2796664287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18641575e0 msgr2=0x7f186415e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:43.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.132+0000 7f1863fff700 1 --2- 172.21.15.27:0/2796664287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18641575e0 0x7f186415e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:43.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.132+0000 7f1863fff700 1 -- 172.21.15.27:0/2796664287 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c009ae0 con 0x7f186406b610 2024-01-11T09:45:43.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.132+0000 7f18637fe700 1 --2- 172.21.15.27:0/2796664287 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18640b19d0 0x7f186415e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:43.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.132+0000 7f1868b9c700 1 --2- 172.21.15.27:0/2796664287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18641575e0 0x7f186415e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:43.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.132+0000 7f1863fff700 1 --2- 172.21.15.27:0/2796664287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f186406b610 0x7f1864140680 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f185400c4e0 tx=0x7f185400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:43.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.132+0000 7f184bfff700 1 -- 172.21.15.27:0/2796664287 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854011590 con 0x7f186406b610 2024-01-11T09:45:43.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.132+0000 7f184bfff700 1 -- 172.21.15.27:0/2796664287 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1854011710 con 0x7f186406b610 2024-01-11T09:45:43.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.132+0000 7f186a5ff700 1 -- 172.21.15.27:0/2796664287 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f186415c0c0 con 0x7f186406b610 2024-01-11T09:45:43.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.132+0000 7f184bfff700 1 -- 172.21.15.27:0/2796664287 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854011ad0 con 0x7f186406b610 2024-01-11T09:45:43.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.133+0000 7f186a5ff700 1 -- 172.21.15.27:0/2796664287 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1864140bc0 con 0x7f186406b610 2024-01-11T09:45:43.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.134+0000 7f186a5ff700 1 -- 172.21.15.27:0/2796664287 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18640718d0 con 0x7f186406b610 2024-01-11T09:45:43.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.135+0000 7f184bfff700 1 -- 172.21.15.27:0/2796664287 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f185401b3b0 con 0x7f186406b610 2024-01-11T09:45:43.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.136+0000 7f184bfff700 1 --2- 172.21.15.27:0/2796664287 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1850066f30 0x7f18500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:43.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.136+0000 7f184bfff700 1 -- 172.21.15.27:0/2796664287 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f1854083c30 con 0x7f186406b610 2024-01-11T09:45:43.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.136+0000 7f18637fe700 1 --2- 172.21.15.27:0/2796664287 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1850066f30 0x7f18500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:43.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.137+0000 7f18637fe700 1 --2- 172.21.15.27:0/2796664287 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1850066f30 0x7f18500693e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f186415f0e0 tx=0x7f1858008040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:43.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.140+0000 7f184bfff700 1 -- 172.21.15.27:0/2796664287 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1854050ad0 con 0x7f186406b610 2024-01-11T09:45:43.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.385+0000 7f186a5ff700 1 -- 172.21.15.27:0/2796664287 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1864158b60 con 0x7f186406b610 2024-01-11T09:45:43.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.386+0000 7f184bfff700 1 -- 172.21.15.27:0/2796664287 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+5690 (secure 0 0 0) 0x7f1864158b60 con 0x7f186406b610 2024-01-11T09:45:43.378 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:43.378 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":117,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2103359888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":2103359888},{"type":"v1","addr":"172.21.15.77:6834","nonce":2103359888}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8795,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2650824649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2650824649},{"type":"v1","addr":"172.21.15.77:6835","nonce":2650824649}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:45:23.637614+0000","modified":"2024-01-11T09:45:27.146191+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10599":{"gid":10599,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1143960902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1143960902},{"type":"v1","addr":"172.21.15.27:6835","nonce":1143960902}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:45:37.460120+0000","modified":"2024-01-11T09:45:40.977702+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10596},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10596":{"gid":10596,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6834/2807751836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2807751836},{"type":"v1","addr":"172.21.15.27:6834","nonce":2807751836}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":14}]} 2024-01-11T09:45:43.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.388+0000 7f186a5ff700 1 -- 172.21.15.27:0/2796664287 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1850066f30 msgr2=0x7f18500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:43.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.388+0000 7f186a5ff700 1 --2- 172.21.15.27:0/2796664287 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1850066f30 0x7f18500693e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f186415f0e0 tx=0x7f1858008040 comp rx=0 tx=0).stop 2024-01-11T09:45:43.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.388+0000 7f186a5ff700 1 -- 172.21.15.27:0/2796664287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f186406b610 msgr2=0x7f1864140680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:43.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.388+0000 7f186a5ff700 1 --2- 172.21.15.27:0/2796664287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f186406b610 0x7f1864140680 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f185400c4e0 tx=0x7f185400c8a0 comp rx=0 tx=0).stop 2024-01-11T09:45:43.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.388+0000 7f186a5ff700 1 -- 172.21.15.27:0/2796664287 shutdown_connections 2024-01-11T09:45:43.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.388+0000 7f186a5ff700 1 --2- 172.21.15.27:0/2796664287 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1850066f30 0x7f18500693e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:43.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.388+0000 7f186a5ff700 1 --2- 172.21.15.27:0/2796664287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18641575e0 0x7f186415e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:43.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.388+0000 7f186a5ff700 1 --2- 172.21.15.27:0/2796664287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f186406b610 0x7f1864140680 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:43.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.388+0000 7f186a5ff700 1 --2- 172.21.15.27:0/2796664287 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18640b19d0 0x7f186415e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:43.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.388+0000 7f186a5ff700 1 -- 172.21.15.27:0/2796664287 >> 172.21.15.27:0/2796664287 conn(0x7f18640680e0 msgr2=0x7f1864154990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:43.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.388+0000 7f186a5ff700 1 -- 172.21.15.27:0/2796664287 shutdown_connections 2024-01-11T09:45:43.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:43.389+0000 7f186a5ff700 1 -- 172.21.15.27:0/2796664287 wait complete. 2024-01-11T09:45:43.381 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 117 2024-01-11T09:45:43.801 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-11T09:45:37.460120+0000', 'modified': '2024-01-11T09:45:40.977702+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10596}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10596': {'gid': 10596, 'name': 'c', 'rank': 0, 'incarnation': 113, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.27:6834/2807751836', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2807751836}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2807751836}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-11T09:45:43.802 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-11T09:45:43.802 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-11T09:45:44.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.012+0000 7ff0905ff700 1 Processor -- start 2024-01-11T09:45:44.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.013+0000 7ff0905ff700 1 -- start start 2024-01-11T09:45:44.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.014+0000 7ff0905ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff0880686a0 0x7ff0880abd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.014+0000 7ff0905ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0880ac250 con 0x7ff088068ba0 2024-01-11T09:45:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.014+0000 7ff0905ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff088067b00 con 0x7ff0881596a0 2024-01-11T09:45:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.014+0000 7ff0905ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08806bf60 con 0x7ff0880686a0 2024-01-11T09:45:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.014+0000 7ff08db9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff088068ba0 0x7ff088067f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58814/0 (socket says 172.21.15.27:58814) 2024-01-11T09:45:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.014+0000 7ff08db9a700 1 -- 172.21.15.27:0/1110331671 learned_addr learned my addr 172.21.15.27:0/1110331671 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:44.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.014+0000 7ff08eb9c700 1 --2- 172.21.15.27:0/1110331671 >> v2:172.21.15.77:3300/0 conn(0x7ff0880686a0 0x7ff0880abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.015+0000 7ff08d399700 1 -- 172.21.15.27:0/1110331671 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267369830 0 0) 0x7ff0880ac250 con 0x7ff088068ba0 2024-01-11T09:45:44.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.015+0000 7ff08d399700 1 -- 172.21.15.27:0/1110331671 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff074003570 con 0x7ff088068ba0 2024-01-11T09:45:44.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.015+0000 7ff08d399700 1 -- 172.21.15.27:0/1110331671 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 386270225 0 0) 0x7ff088067b00 con 0x7ff0881596a0 2024-01-11T09:45:44.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.015+0000 7ff08d399700 1 -- 172.21.15.27:0/1110331671 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0880ac250 con 0x7ff0881596a0 2024-01-11T09:45:44.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.015+0000 7ff08eb9c700 1 -- 172.21.15.27:0/1110331671 >> v1:172.21.15.77:6789/0 conn(0x7ff0881596a0 legacy=0x7ff0880af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.015+0000 7ff08eb9c700 1 -- 172.21.15.27:0/1110331671 >> v1:172.21.15.27:6789/0 conn(0x7ff088068ba0 legacy=0x7ff088067f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.015+0000 7ff08d399700 1 -- 172.21.15.27:0/1110331671 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4275425511 0 0) 0x7ff074003570 con 0x7ff088068ba0 2024-01-11T09:45:44.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.015+0000 7ff08eb9c700 1 -- 172.21.15.27:0/1110331671 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08806c240 con 0x7ff0880686a0 2024-01-11T09:45:44.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.015+0000 7ff08eb9c700 1 --2- 172.21.15.27:0/1110331671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0880686a0 0x7ff0880abd10 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7ff080009e60 tx=0x7ff080009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ece5bbec0995db5a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:44.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.016+0000 7ff08d399700 1 -- 172.21.15.27:0/1110331671 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08000e070 con 0x7ff0880686a0 2024-01-11T09:45:44.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.016+0000 7ff08d399700 1 -- 172.21.15.27:0/1110331671 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08000b7a0 con 0x7ff0880686a0 2024-01-11T09:45:44.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.016+0000 7ff08d399700 1 -- 172.21.15.27:0/1110331671 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078001470 con 0x7ff0880686a0 2024-01-11T09:45:44.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.016+0000 7ff0905ff700 1 -- 172.21.15.27:0/1110331671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0880686a0 msgr2=0x7ff0880abd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.016+0000 7ff0905ff700 1 --2- 172.21.15.27:0/1110331671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0880686a0 0x7ff0880abd10 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7ff080009e60 tx=0x7ff080009780 comp rx=0 tx=0).stop 2024-01-11T09:45:44.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.016+0000 7ff0905ff700 1 -- 172.21.15.27:0/1110331671 shutdown_connections 2024-01-11T09:45:44.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.016+0000 7ff0905ff700 1 --2- 172.21.15.27:0/1110331671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0880686a0 0x7ff0880abd10 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.016+0000 7ff0905ff700 1 -- 172.21.15.27:0/1110331671 >> 172.21.15.27:0/1110331671 conn(0x7ff08806a8b0 msgr2=0x7ff0880ae520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:44.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.016+0000 7ff0905ff700 1 -- 172.21.15.27:0/1110331671 shutdown_connections 2024-01-11T09:45:44.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.016+0000 7ff0905ff700 1 -- 172.21.15.27:0/1110331671 wait complete. 2024-01-11T09:45:44.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.018+0000 7ff0905ff700 1 Processor -- start 2024-01-11T09:45:44.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.018+0000 7ff0905ff700 1 -- start start 2024-01-11T09:45:44.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.019+0000 7ff0905ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0880686a0 0x7ff0880c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.019+0000 7ff0905ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff088068ba0 0x7ff0880b1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.019+0000 7ff0905ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0881596a0 0x7ff0881566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.019+0000 7ff0905ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff088159da0 con 0x7ff0880686a0 2024-01-11T09:45:44.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.019+0000 7ff0905ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08815a080 con 0x7ff088068ba0 2024-01-11T09:45:44.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.019+0000 7ff0905ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08815a360 con 0x7ff0881596a0 2024-01-11T09:45:44.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.019+0000 7ff08e39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0880686a0 0x7ff0880c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.019+0000 7ff08e39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0880686a0 0x7ff0880c8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44532/0 (socket says 172.21.15.27:44532) 2024-01-11T09:45:44.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.019+0000 7ff08e39b700 1 -- 172.21.15.27:0/2596000069 learned_addr learned my addr 172.21.15.27:0/2596000069 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:44.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.019+0000 7ff08db9a700 1 --2- 172.21.15.27:0/2596000069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff088068ba0 0x7ff0880b1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.019+0000 7ff08eb9c700 1 --2- 172.21.15.27:0/2596000069 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0881596a0 0x7ff0881566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.020+0000 7ff08e39b700 1 -- 172.21.15.27:0/2596000069 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0881596a0 msgr2=0x7ff0881566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.020+0000 7ff08e39b700 1 --2- 172.21.15.27:0/2596000069 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0881596a0 0x7ff0881566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.020+0000 7ff08e39b700 1 -- 172.21.15.27:0/2596000069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff088068ba0 msgr2=0x7ff0880b1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.020+0000 7ff08e39b700 1 --2- 172.21.15.27:0/2596000069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff088068ba0 0x7ff0880b1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.020+0000 7ff08e39b700 1 -- 172.21.15.27:0/2596000069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff080009a20 con 0x7ff0880686a0 2024-01-11T09:45:44.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.020+0000 7ff08e39b700 1 --2- 172.21.15.27:0/2596000069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0880686a0 0x7ff0880c8b40 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7ff08400c590 tx=0x7ff084007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:44.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.020+0000 7ff07e7fc700 1 -- 172.21.15.27:0/2596000069 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084001550 con 0x7ff0880686a0 2024-01-11T09:45:44.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.020+0000 7ff07e7fc700 1 -- 172.21.15.27:0/2596000069 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0840016d0 con 0x7ff0880686a0 2024-01-11T09:45:44.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.020+0000 7ff0905ff700 1 -- 172.21.15.27:0/2596000069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08815a640 con 0x7ff0880686a0 2024-01-11T09:45:44.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.020+0000 7ff07e7fc700 1 -- 172.21.15.27:0/2596000069 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084001a90 con 0x7ff0880686a0 2024-01-11T09:45:44.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.020+0000 7ff0905ff700 1 -- 172.21.15.27:0/2596000069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff088159a70 con 0x7ff0880686a0 2024-01-11T09:45:44.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.022+0000 7ff07e7fc700 1 -- 172.21.15.27:0/2596000069 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff08401b400 con 0x7ff0880686a0 2024-01-11T09:45:44.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.023+0000 7ff07e7fc700 1 --2- 172.21.15.27:0/2596000069 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff06c066f30 0x7ff06c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.023+0000 7ff07e7fc700 1 -- 172.21.15.27:0/2596000069 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6681+0+0 (secure 0 0 0) 0x7ff084084750 con 0x7ff0880686a0 2024-01-11T09:45:44.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.024+0000 7ff08db9a700 1 --2- 172.21.15.27:0/2596000069 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff06c066f30 0x7ff06c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.024+0000 7ff08db9a700 1 --2- 172.21.15.27:0/2596000069 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff06c066f30 0x7ff06c0693e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff0880cc400 tx=0x7ff078000bc0 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:44.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.025+0000 7ff09ed6bb80 1 Processor -- start 2024-01-11T09:45:44.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.025+0000 7ff09ed6bb80 1 -- start start 2024-01-11T09:45:44.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.026+0000 7ff09ed6bb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 0x5626c52c2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.026+0000 7ff09ed6bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52bd3a0 0x5626c52e9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.026+0000 7ff09ed6bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 0x5626c52e9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.026+0000 7ff09ed6bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c51da310 con 0x5626c52e1ef0 2024-01-11T09:45:44.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.026+0000 7ff09ed6bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c5217130 con 0x5626c52bd3a0 2024-01-11T09:45:44.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.026+0000 7ff09ed6bb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626c5214af0 con 0x5626c52e1b20 2024-01-11T09:45:44.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.027+0000 7ff08eb9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 0x5626c52e9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.027+0000 7ff08e39b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 0x5626c52c2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.027+0000 7ff08db9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52bd3a0 0x5626c52e9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.027+0000 7ff08eb9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 0x5626c52e9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44538/0 (socket says 172.21.15.27:44538) 2024-01-11T09:45:44.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.027+0000 7ff08eb9c700 1 -- 172.21.15.27:0/1844787844 learned_addr learned my addr 172.21.15.27:0/1844787844 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:44.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.027+0000 7ff08eb9c700 1 -- 172.21.15.27:0/1844787844 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 msgr2=0x5626c52c2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:44.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.027+0000 7ff08eb9c700 1 --2- 172.21.15.27:0/1844787844 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 0x5626c52c2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.027+0000 7ff08eb9c700 1 -- 172.21.15.27:0/1844787844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52bd3a0 msgr2=0x5626c52e9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:44.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.027+0000 7ff08eb9c700 1 --2- 172.21.15.27:0/1844787844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52bd3a0 0x5626c52e9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.027+0000 7ff08eb9c700 1 -- 172.21.15.27:0/1844787844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626c51030a0 con 0x5626c52e1ef0 2024-01-11T09:45:44.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.027+0000 7ff08eb9c700 1 --2- 172.21.15.27:0/1844787844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 0x5626c52e9da0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7ff0800099f0 tx=0x7ff080017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb0e6c71f3ac7c04 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:44.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.027+0000 7ff06b7fe700 1 -- 172.21.15.27:0/1844787844 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08000bda0 con 0x5626c52e1ef0 2024-01-11T09:45:44.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.028+0000 7ff06b7fe700 1 -- 172.21.15.27:0/1844787844 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff080027460 con 0x5626c52e1ef0 2024-01-11T09:45:44.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.028+0000 7ff06b7fe700 1 -- 172.21.15.27:0/1844787844 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0800277e0 con 0x5626c52e1ef0 2024-01-11T09:45:44.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.028+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/1844787844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 msgr2=0x5626c52e9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.028+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/1844787844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 0x5626c52e9da0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7ff0800099f0 tx=0x7ff080017000 comp rx=0 tx=0).stop 2024-01-11T09:45:44.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.028+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/1844787844 shutdown_connections 2024-01-11T09:45:44.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.028+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/1844787844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 0x5626c52e9da0 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.028+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/1844787844 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 0x5626c52c2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.028+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/1844787844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52bd3a0 0x5626c52e9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.028+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/1844787844 >> 172.21.15.27:0/1844787844 conn(0x5626c52bf300 msgr2=0x5626c52c1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:44.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.028+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/1844787844 shutdown_connections 2024-01-11T09:45:44.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.028+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/1844787844 wait complete. 2024-01-11T09:45:44.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.029+0000 7ff09ed6bb80 1 -- start start 2024-01-11T09:45:44.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.029+0000 7ff09ed6bb80 1 Processor -- start 2024-01-11T09:45:44.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.030+0000 7ff09ed6bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52bd3a0 0x5626c52e08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.030+0000 7ff09ed6bb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 0x5626c52e0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.030+0000 7ff09ed6bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52e1ef0 0x5626c52e3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.030+0000 7ff09ed6bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c52108e0 con 0x5626c52bd3a0 2024-01-11T09:45:44.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.030+0000 7ff09ed6bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c51bc980 con 0x5626c52e1ef0 2024-01-11T09:45:44.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.030+0000 7ff09ed6bb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626c51bc620 con 0x5626c52e1b20 2024-01-11T09:45:44.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.031+0000 7ff09ed6bb80 10 client.0 _ll_register_callbacks cb 0x5626c501fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T09:45:44.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.031+0000 7ff08e39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52bd3a0 0x5626c52e08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.031+0000 7ff08e39b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52bd3a0 0x5626c52e08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44540/0 (socket says 172.21.15.27:44540) 2024-01-11T09:45:44.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.031+0000 7ff08e39b700 1 -- 172.21.15.27:0/2008846736 learned_addr learned my addr 172.21.15.27:0/2008846736 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:44.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.031+0000 7ff08db9a700 1 --2- 172.21.15.27:0/2008846736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 0x5626c52e0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.031+0000 7ff08eb9c700 1 --2- 172.21.15.27:0/2008846736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52e1ef0 0x5626c52e3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.031+0000 7ff08e39b700 1 -- 172.21.15.27:0/2008846736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 msgr2=0x5626c52e0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:44.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.031+0000 7ff08e39b700 1 --2- 172.21.15.27:0/2008846736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 0x5626c52e0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.031+0000 7ff08e39b700 1 -- 172.21.15.27:0/2008846736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52e1ef0 msgr2=0x5626c52e3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:44.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.031+0000 7ff08e39b700 1 --2- 172.21.15.27:0/2008846736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52e1ef0 0x5626c52e3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.031+0000 7ff08e39b700 1 -- 172.21.15.27:0/2008846736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08806c240 con 0x5626c52bd3a0 2024-01-11T09:45:44.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.031+0000 7ff08e39b700 1 --2- 172.21.15.27:0/2008846736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52bd3a0 0x5626c52e08c0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7ff0840551a0 tx=0x7ff084082760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b18b7e3b33ed715c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:44.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.031+0000 7ff068ff9700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T09:45:44.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.032+0000 7ff068ff9700 1 -- 172.21.15.27:0/2008846736 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084055210 con 0x5626c52bd3a0 2024-01-11T09:45:44.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.032+0000 7ff068ff9700 1 -- 172.21.15.27:0/2008846736 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(218..218 src has 1..218) v4 ==== 6681+0+0 (secure 0 0 0) 0x7ff084058460 con 0x5626c52bd3a0 2024-01-11T09:45:44.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.032+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/2008846736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5626c52b5560 con 0x5626c52bd3a0 2024-01-11T09:45:44.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.032+0000 7ff068ff9700 1 -- 172.21.15.27:0/2008846736 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff08405c1b0 con 0x5626c52bd3a0 2024-01-11T09:45:44.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.032+0000 7ff09ed6bb80 10 client.10878 fetch_fsmap learned FSMap version 117 2024-01-11T09:45:44.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.032+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/2008846736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=2+}) v3 -- 0x5626c51df3c0 con 0x5626c52bd3a0 2024-01-11T09:45:44.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.032+0000 7ff068ff9700 1 -- 172.21.15.27:0/2008846736 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5626c52b5560 con 0x5626c52bd3a0 2024-01-11T09:45:44.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.032+0000 7ff068ff9700 1 -- 172.21.15.27:0/2008846736 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 117) v1 ==== 3653+0+0 (secure 0 0 0) 0x7ff08405bd20 con 0x5626c52bd3a0 2024-01-11T09:45:44.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.032+0000 7ff09ed6bb80 10 client.10878 fetch_fsmap finished waiting for FSMap version 117 2024-01-11T09:45:44.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.032+0000 7ff09ed6bb80 10 client.10878 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-11T09:45:44.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.032+0000 7ff09ed6bb80 20 client.10878 populate_metadata read hostname 'smithi027' 2024-01-11T09:45:44.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.032+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/2008846736 >> [v2:172.21.15.27:6832/2807751836,v1:172.21.15.27:6834/2807751836] conn(0x5626c531a650 0x5626c531aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.032+0000 7ff09ed6bb80 4 client.10878 mds_command: new command op to 10596 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-11T09:45:44.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.032+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/2008846736 --> [v2:172.21.15.27:6832/2807751836,v1:172.21.15.27:6834/2807751836] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5626c52920e0 con 0x5626c531a650 2024-01-11T09:45:44.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.033+0000 7ff08db9a700 1 --2- 172.21.15.27:0/2008846736 >> [v2:172.21.15.27:6832/2807751836,v1:172.21.15.27:6834/2807751836] conn(0x5626c531a650 0x5626c531aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.033+0000 7ff08db9a700 1 --2- 172.21.15.27:0/2008846736 >> [v2:172.21.15.27:6832/2807751836,v1:172.21.15.27:6834/2807751836] conn(0x5626c531a650 0x5626c531aa20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e97689ca24c9f400 server_cookie=975d5891055033b8 in_seq=0 out_seq=0 2024-01-11T09:45:44.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.033+0000 7ff068ff9700 10 client.10878 ms_handle_connect on v2:172.21.15.27:6832/2807751836 2024-01-11T09:45:44.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.034+0000 7ff068ff9700 1 -- 172.21.15.27:0/2008846736 <== mds.0 v2:172.21.15.27:6832/2807751836 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x5626c52920e0 con 0x5626c531a650 2024-01-11T09:45:44.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.034+0000 7ff068ff9700 10 client.10878 handle_command_reply: tid=0 2024-01-11T09:45:44.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.034+0000 7ff09ed6bb80 1 client.10878 shutdown 2024-01-11T09:45:44.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.034+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/2008846736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52bd3a0 msgr2=0x5626c52e08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.034+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/2008846736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52bd3a0 0x5626c52e08c0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7ff0840551a0 tx=0x7ff084082760 comp rx=0 tx=0).stop 2024-01-11T09:45:44.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.034+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/2008846736 shutdown_connections 2024-01-11T09:45:44.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.034+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/2008846736 >> [v2:172.21.15.27:6832/2807751836,v1:172.21.15.27:6834/2807751836] conn(0x5626c531a650 0x5626c531aa20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.034+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/2008846736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52e1ef0 0x5626c52e3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.034+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/2008846736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52bd3a0 0x5626c52e08c0 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.034+0000 7ff068ff9700 0 client.10878 ms_handle_reset on v2:172.21.15.27:6832/2807751836 2024-01-11T09:45:44.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.034+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/2008846736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 0x5626c52e0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.034+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/2008846736 >> 172.21.15.27:0/2008846736 conn(0x5626c52bf300 msgr2=0x5626c52bda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:44.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.034+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/2008846736 shutdown_connections 2024-01-11T09:45:44.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.035+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/2008846736 wait complete. 2024-01-11T09:45:44.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.035+0000 7ff09ed6bb80 20 client.10878 trim_cache size 0 max 16384 2024-01-11T09:45:44.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.052+0000 7ff09ed6bb80 1 Processor -- start 2024-01-11T09:45:44.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.052+0000 7ff09ed6bb80 1 -- start start 2024-01-11T09:45:44.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.053+0000 7ff09ed6bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52bd3a0 0x5626c53154c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.053+0000 7ff09ed6bb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 0x5626c5315a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.054+0000 7ff09ed6bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 0x5626c52b67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.054+0000 7ff09ed6bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c50a81e0 con 0x5626c52e1ef0 2024-01-11T09:45:44.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.054+0000 7ff09ed6bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c52bff70 con 0x5626c52bd3a0 2024-01-11T09:45:44.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.054+0000 7ff09ed6bb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626c52e17d0 con 0x5626c52e1b20 2024-01-11T09:45:44.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.054+0000 7ff08eb9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 0x5626c52b67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.054+0000 7ff08eb9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 0x5626c52b67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44552/0 (socket says 172.21.15.27:44552) 2024-01-11T09:45:44.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.054+0000 7ff08eb9c700 1 -- 172.21.15.27:0/1850360009 learned_addr learned my addr 172.21.15.27:0/1850360009 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:44.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.054+0000 7ff08e39b700 1 --2- 172.21.15.27:0/1850360009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52bd3a0 0x5626c53154c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.054+0000 7ff08db9a700 1 --2- 172.21.15.27:0/1850360009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 0x5626c5315a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.054+0000 7ff08eb9c700 1 -- 172.21.15.27:0/1850360009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 msgr2=0x5626c5315a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:44.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.054+0000 7ff08eb9c700 1 --2- 172.21.15.27:0/1850360009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 0x5626c5315a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.054+0000 7ff08eb9c700 1 -- 172.21.15.27:0/1850360009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52bd3a0 msgr2=0x5626c53154c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:44.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.054+0000 7ff08eb9c700 1 --2- 172.21.15.27:0/1850360009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52bd3a0 0x5626c53154c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.054+0000 7ff08eb9c700 1 -- 172.21.15.27:0/1850360009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626c50771d0 con 0x5626c52e1ef0 2024-01-11T09:45:44.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.054+0000 7ff08e39b700 1 --2- 172.21.15.27:0/1850360009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52bd3a0 0x5626c53154c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:44.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.054+0000 7ff08db9a700 1 --2- 172.21.15.27:0/1850360009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 0x5626c5315a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:44.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.055+0000 7ff08eb9c700 1 --2- 172.21.15.27:0/1850360009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 0x5626c52b67d0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7ff080003000 tx=0x7ff08001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c32033bbe2a160 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:44.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.055+0000 7ff06bfff700 1 -- 172.21.15.27:0/1850360009 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08002d050 con 0x5626c52e1ef0 2024-01-11T09:45:44.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.055+0000 7ff06bfff700 1 -- 172.21.15.27:0/1850360009 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08002e040 con 0x5626c52e1ef0 2024-01-11T09:45:44.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.055+0000 7ff06bfff700 1 -- 172.21.15.27:0/1850360009 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff080027b90 con 0x5626c52e1ef0 2024-01-11T09:45:44.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.055+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/1850360009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 msgr2=0x5626c52b67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.055+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/1850360009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 0x5626c52b67d0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7ff080003000 tx=0x7ff08001f000 comp rx=0 tx=0).stop 2024-01-11T09:45:44.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.055+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/1850360009 shutdown_connections 2024-01-11T09:45:44.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.055+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/1850360009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 0x5626c52b67d0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.055+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/1850360009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52bd3a0 0x5626c53154c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.055+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/1850360009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 0x5626c5315a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.055+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/1850360009 >> 172.21.15.27:0/1850360009 conn(0x5626c52bf300 msgr2=0x5626c52e38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:44.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.055+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/1850360009 shutdown_connections 2024-01-11T09:45:44.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.056+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/1850360009 wait complete. 2024-01-11T09:45:44.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.056+0000 7ff09ed6bb80 1 -- start start 2024-01-11T09:45:44.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.057+0000 7ff09ed6bb80 1 Processor -- start 2024-01-11T09:45:44.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.058+0000 7ff09ed6bb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52bd3a0 0x5626c52e3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.058+0000 7ff09ed6bb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 0x5626c5315920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.058+0000 7ff09ed6bb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 0x5626c5355080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.058+0000 7ff09ed6bb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c531a170 con 0x5626c52e1ef0 2024-01-11T09:45:44.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.058+0000 7ff09ed6bb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c536d730 con 0x5626c52bd3a0 2024-01-11T09:45:44.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.058+0000 7ff09ed6bb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626c536d020 con 0x5626c52e1b20 2024-01-11T09:45:44.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.058+0000 7ff09ed6bb80 10 client.0 _ll_register_callbacks cb 0x5626c501fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T09:45:44.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.058+0000 7ff08eb9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 0x5626c5355080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.059+0000 7ff08e39b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52bd3a0 0x5626c52e3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.059+0000 7ff08db9a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 0x5626c5315920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.059+0000 7ff08eb9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 0x5626c5355080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44566/0 (socket says 172.21.15.27:44566) 2024-01-11T09:45:44.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.059+0000 7ff08eb9c700 1 -- 172.21.15.27:0/1047998493 learned_addr learned my addr 172.21.15.27:0/1047998493 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:44.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.059+0000 7ff08e39b700 1 -- 172.21.15.27:0/1047998493 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 msgr2=0x5626c5315920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:44.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.059+0000 7ff08e39b700 1 --2- 172.21.15.27:0/1047998493 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 0x5626c5315920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.059+0000 7ff08e39b700 1 -- 172.21.15.27:0/1047998493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 msgr2=0x5626c5355080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:44.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.059+0000 7ff08e39b700 1 --2- 172.21.15.27:0/1047998493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 0x5626c5355080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.059+0000 7ff08e39b700 1 -- 172.21.15.27:0/1047998493 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5626c51030a0 con 0x5626c52bd3a0 2024-01-11T09:45:44.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.059+0000 7ff08eb9c700 1 --2- 172.21.15.27:0/1047998493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 0x5626c5355080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:44.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.059+0000 7ff08db9a700 1 --2- 172.21.15.27:0/1047998493 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 0x5626c5315920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:44.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.059+0000 7ff08e39b700 1 --2- 172.21.15.27:0/1047998493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52bd3a0 0x5626c52e3a10 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7ff084011020 tx=0x7ff084069000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bed64cd153443a00 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:44.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.059+0000 7ff068ff9700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-01-11T09:45:44.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.059+0000 7ff068ff9700 1 -- 172.21.15.27:0/1047998493 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08405c1b0 con 0x5626c52bd3a0 2024-01-11T09:45:44.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.060+0000 7ff068ff9700 1 -- 172.21.15.27:0/1047998493 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(218..218 src has 1..218) v4 ==== 6681+0+0 (secure 0 0 0) 0x7ff084074020 con 0x5626c52bd3a0 2024-01-11T09:45:44.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.060+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/1047998493 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff080027460 con 0x5626c52bd3a0 2024-01-11T09:45:44.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.060+0000 7ff068ff9700 1 -- 172.21.15.27:0/1047998493 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0840582a0 con 0x5626c52bd3a0 2024-01-11T09:45:44.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.060+0000 7ff09ed6bb80 10 client.10480 fetch_fsmap learned FSMap version 117 2024-01-11T09:45:44.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.060+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/1047998493 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=2+}) v3 -- 0x5626c5050d60 con 0x5626c52bd3a0 2024-01-11T09:45:44.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.060+0000 7ff068ff9700 1 -- 172.21.15.27:0/1047998493 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff080027460 con 0x5626c52bd3a0 2024-01-11T09:45:44.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.060+0000 7ff068ff9700 1 -- 172.21.15.27:0/1047998493 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 117) v1 ==== 3653+0+0 (secure 0 0 0) 0x7ff08405bd20 con 0x5626c52bd3a0 2024-01-11T09:45:44.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.060+0000 7ff09ed6bb80 10 client.10480 fetch_fsmap finished waiting for FSMap version 117 2024-01-11T09:45:44.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.060+0000 7ff09ed6bb80 10 client.10480 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-11T09:45:44.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.060+0000 7ff09ed6bb80 20 client.10480 populate_metadata read hostname 'smithi027' 2024-01-11T09:45:44.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.060+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/1047998493 >> [v2:172.21.15.27:6832/2807751836,v1:172.21.15.27:6834/2807751836] conn(0x5626c52c1d50 0x5626c52c2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.060+0000 7ff09ed6bb80 4 client.10480 mds_command: new command op to 10596 tid=0[{"prefix": "status"}] 2024-01-11T09:45:44.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.060+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/1047998493 --> [v2:172.21.15.27:6832/2807751836,v1:172.21.15.27:6834/2807751836] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5626c51b38d0 con 0x5626c52c1d50 2024-01-11T09:45:44.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.060+0000 7ff08db9a700 1 --2- 172.21.15.27:0/1047998493 >> [v2:172.21.15.27:6832/2807751836,v1:172.21.15.27:6834/2807751836] conn(0x5626c52c1d50 0x5626c52c2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.061+0000 7ff08db9a700 1 --2- 172.21.15.27:0/1047998493 >> [v2:172.21.15.27:6832/2807751836,v1:172.21.15.27:6834/2807751836] conn(0x5626c52c1d50 0x5626c52c2120 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35f1cbd95335f401 server_cookie=7a2a9227991a7d1f in_seq=0 out_seq=0 2024-01-11T09:45:44.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.061+0000 7ff068ff9700 10 client.10480 ms_handle_connect on v2:172.21.15.27:6832/2807751836 2024-01-11T09:45:44.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.061+0000 7ff068ff9700 1 -- 172.21.15.27:0/1047998493 <== mds.0 v2:172.21.15.27:6832/2807751836 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+338 (crc 0 0 0) 0x5626c51b38d0 con 0x5626c52c1d50 2024-01-11T09:45:44.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.061+0000 7ff068ff9700 10 client.10480 handle_command_reply: tid=0 2024-01-11T09:45:44.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.061+0000 7ff09ed6bb80 1 client.10480 shutdown 2024-01-11T09:45:44.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.061+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/1047998493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52bd3a0 msgr2=0x5626c52e3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.061+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/1047998493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52bd3a0 0x5626c52e3a10 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7ff084011020 tx=0x7ff084069000 comp rx=0 tx=0).stop 2024-01-11T09:45:44.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.061+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/1047998493 shutdown_connections 2024-01-11T09:45:44.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.061+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/1047998493 >> [v2:172.21.15.27:6832/2807751836,v1:172.21.15.27:6834/2807751836] conn(0x5626c52c1d50 0x5626c52c2120 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.061+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/1047998493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5626c52e1ef0 0x5626c5355080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.061+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/1047998493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626c52bd3a0 0x5626c52e3a10 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.061+0000 7ff09ed6bb80 1 --2- 172.21.15.27:0/1047998493 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5626c52e1b20 0x5626c5315920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.061+0000 7ff068ff9700 0 client.10480 ms_handle_reset on v2:172.21.15.27:6832/2807751836 2024-01-11T09:45:44.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.061+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/1047998493 >> 172.21.15.27:0/1047998493 conn(0x5626c52bf300 msgr2=0x5626c52be5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:44.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.062+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/1047998493 shutdown_connections 2024-01-11T09:45:44.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.062+0000 7ff09ed6bb80 1 -- 172.21.15.27:0/1047998493 wait complete. 2024-01-11T09:45:44.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.062+0000 7ff09ed6bb80 20 client.10480 trim_cache size 0 max 16384 2024-01-11T09:45:44.057 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:45:44.057 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "8fddb4a7-9d66-40f1-892d-6fe86c728ee4", 2024-01-11T09:45:44.057 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-11T09:45:44.057 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 10596, 2024-01-11T09:45:44.058 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-11T09:45:44.058 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-11T09:45:44.058 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "backup_fs", 2024-01-11T09:45:44.058 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 6.1068825259999997, 2024-01-11T09:45:44.058 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 117, 2024-01-11T09:45:44.058 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 218, 2024-01-11T09:45:44.058 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 218, 2024-01-11T09:45:44.058 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 25.560873733000001 2024-01-11T09:45:44.058 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:45:44.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.062+0000 7ff0905ff700 1 -- 172.21.15.27:0/2596000069 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff06c066f30 msgr2=0x7ff06c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.062+0000 7ff0905ff700 1 --2- 172.21.15.27:0/2596000069 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff06c066f30 0x7ff06c0693e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff0880cc400 tx=0x7ff078000bc0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.062+0000 7ff0905ff700 1 -- 172.21.15.27:0/2596000069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0880686a0 msgr2=0x7ff0880c8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.062+0000 7ff0905ff700 1 --2- 172.21.15.27:0/2596000069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0880686a0 0x7ff0880c8b40 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7ff08400c590 tx=0x7ff084007dd0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.062+0000 7ff0905ff700 1 -- 172.21.15.27:0/2596000069 shutdown_connections 2024-01-11T09:45:44.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.062+0000 7ff0905ff700 1 --2- 172.21.15.27:0/2596000069 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff0881596a0 0x7ff0881566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.062+0000 7ff0905ff700 1 --2- 172.21.15.27:0/2596000069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0880686a0 0x7ff0880c8b40 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.062+0000 7ff0905ff700 1 --2- 172.21.15.27:0/2596000069 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff06c066f30 0x7ff06c0693e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.062+0000 7ff0905ff700 1 --2- 172.21.15.27:0/2596000069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff088068ba0 0x7ff0880b1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.062+0000 7ff0905ff700 1 -- 172.21.15.27:0/2596000069 >> 172.21.15.27:0/2596000069 conn(0x7ff08806a8b0 msgr2=0x7ff0880ac4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:44.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.063+0000 7ff0905ff700 1 -- 172.21.15.27:0/2596000069 shutdown_connections 2024-01-11T09:45:44.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.063+0000 7ff0905ff700 1 -- 172.21.15.27:0/2596000069 wait complete. 2024-01-11T09:45:44.071 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-11T09:45:44.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.280+0000 7f8d8d5ff700 1 Processor -- start 2024-01-11T09:45:44.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.280+0000 7f8d8d5ff700 1 -- start start 2024-01-11T09:45:44.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.282+0000 7f8d8d5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d880cb5c0 0x7f8d880cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.282+0000 7f8d8d5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d880af390 con 0x7f8d88159550 2024-01-11T09:45:44.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.282+0000 7f8d8d5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d8815a220 con 0x7f8d88159920 2024-01-11T09:45:44.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.282+0000 7f8d8d5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8815ada0 con 0x7f8d880cb5c0 2024-01-11T09:45:44.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.283+0000 7f8d86ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8d88159550 0x7f8d880c58e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58826/0 (socket says 172.21.15.27:58826) 2024-01-11T09:45:44.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.283+0000 7f8d86ffd700 1 -- 172.21.15.27:0/3631791433 learned_addr learned my addr 172.21.15.27:0/3631791433 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:44.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.283+0000 7f8d877fe700 1 --2- 172.21.15.27:0/3631791433 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d880cb5c0 0x7f8d880cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.283+0000 7f8d85ffb700 1 -- 172.21.15.27:0/3631791433 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2546794393 0 0) 0x7f8d880af390 con 0x7f8d88159550 2024-01-11T09:45:44.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.283+0000 7f8d85ffb700 1 -- 172.21.15.27:0/3631791433 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d70003570 con 0x7f8d88159550 2024-01-11T09:45:44.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.283+0000 7f8d85ffb700 1 -- 172.21.15.27:0/3631791433 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933650506 0 0) 0x7f8d8815a220 con 0x7f8d88159920 2024-01-11T09:45:44.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.283+0000 7f8d85ffb700 1 -- 172.21.15.27:0/3631791433 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d880af390 con 0x7f8d88159920 2024-01-11T09:45:44.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.283+0000 7f8d877fe700 1 -- 172.21.15.27:0/3631791433 >> v1:172.21.15.77:6789/0 conn(0x7f8d88159920 legacy=0x7f8d880af770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.283+0000 7f8d877fe700 1 -- 172.21.15.27:0/3631791433 >> v1:172.21.15.27:6789/0 conn(0x7f8d88159550 legacy=0x7f8d880c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.283+0000 7f8d877fe700 1 -- 172.21.15.27:0/3631791433 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8815a500 con 0x7f8d880cb5c0 2024-01-11T09:45:44.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.284+0000 7f8d877fe700 1 --2- 172.21.15.27:0/3631791433 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d880cb5c0 0x7f8d880cb9d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8d80009e90 tx=0x7f8d80009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6a5fa139af602dc server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:44.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.284+0000 7f8d85ffb700 1 -- 172.21.15.27:0/3631791433 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8000e070 con 0x7f8d880cb5c0 2024-01-11T09:45:44.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.284+0000 7f8d85ffb700 1 -- 172.21.15.27:0/3631791433 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8000bb80 con 0x7f8d880cb5c0 2024-01-11T09:45:44.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.284+0000 7f8d85ffb700 1 -- 172.21.15.27:0/3631791433 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d80012430 con 0x7f8d880cb5c0 2024-01-11T09:45:44.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.284+0000 7f8d8d5ff700 1 -- 172.21.15.27:0/3631791433 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d880cb5c0 msgr2=0x7f8d880cb9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.284+0000 7f8d8d5ff700 1 --2- 172.21.15.27:0/3631791433 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d880cb5c0 0x7f8d880cb9d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8d80009e90 tx=0x7f8d80009780 comp rx=0 tx=0).stop 2024-01-11T09:45:44.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.285+0000 7f8d8d5ff700 1 -- 172.21.15.27:0/3631791433 shutdown_connections 2024-01-11T09:45:44.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.285+0000 7f8d8d5ff700 1 --2- 172.21.15.27:0/3631791433 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d880cb5c0 0x7f8d880cb9d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.285+0000 7f8d8d5ff700 1 -- 172.21.15.27:0/3631791433 >> 172.21.15.27:0/3631791433 conn(0x7f8d8806ed40 msgr2=0x7f8d8806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:44.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.285+0000 7f8d8d5ff700 1 -- 172.21.15.27:0/3631791433 shutdown_connections 2024-01-11T09:45:44.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.285+0000 7f8d8d5ff700 1 -- 172.21.15.27:0/3631791433 wait complete. 2024-01-11T09:45:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.287+0000 7f8d8d5ff700 1 Processor -- start 2024-01-11T09:45:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.287+0000 7f8d8d5ff700 1 -- start start 2024-01-11T09:45:44.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.288+0000 7f8d8d5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d880cb5c0 0x7f8d88154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.288+0000 7f8d8d5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d88159550 0x7f8d880b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.288+0000 7f8d8d5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d88159920 0x7f8d880b2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.288+0000 7f8d8d5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d88169430 con 0x7f8d88159920 2024-01-11T09:45:44.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.288+0000 7f8d8d5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d881551c0 con 0x7f8d88159550 2024-01-11T09:45:44.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.288+0000 7f8d8d5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d88169f70 con 0x7f8d880cb5c0 2024-01-11T09:45:44.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.289+0000 7f8d877fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d88159920 0x7f8d880b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.289+0000 7f8d877fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d88159920 0x7f8d880b2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44574/0 (socket says 172.21.15.27:44574) 2024-01-11T09:45:44.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.289+0000 7f8d877fe700 1 -- 172.21.15.27:0/1905296033 learned_addr learned my addr 172.21.15.27:0/1905296033 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:44.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.289+0000 7f8d86ffd700 1 --2- 172.21.15.27:0/1905296033 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d880cb5c0 0x7f8d88154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.289+0000 7f8d867fc700 1 --2- 172.21.15.27:0/1905296033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d88159550 0x7f8d880b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.289+0000 7f8d877fe700 1 -- 172.21.15.27:0/1905296033 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d880cb5c0 msgr2=0x7f8d88154c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.289+0000 7f8d877fe700 1 --2- 172.21.15.27:0/1905296033 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d880cb5c0 0x7f8d88154c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.289+0000 7f8d877fe700 1 -- 172.21.15.27:0/1905296033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d88159550 msgr2=0x7f8d880b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.289+0000 7f8d877fe700 1 --2- 172.21.15.27:0/1905296033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d88159550 0x7f8d880b1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.289+0000 7f8d877fe700 1 -- 172.21.15.27:0/1905296033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d80009a20 con 0x7f8d88159920 2024-01-11T09:45:44.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.289+0000 7f8d867fc700 1 --2- 172.21.15.27:0/1905296033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d88159550 0x7f8d880b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:44.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.289+0000 7f8d877fe700 1 --2- 172.21.15.27:0/1905296033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d88159920 0x7f8d880b2060 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f8d8000b610 tx=0x7f8d80019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:44.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.290+0000 7f8d6effd700 1 -- 172.21.15.27:0/1905296033 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d80027470 con 0x7f8d88159920 2024-01-11T09:45:44.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.290+0000 7f8d6effd700 1 -- 172.21.15.27:0/1905296033 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d800275f0 con 0x7f8d88159920 2024-01-11T09:45:44.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.290+0000 7f8d8d5ff700 1 -- 172.21.15.27:0/1905296033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d881582e0 con 0x7f8d88159920 2024-01-11T09:45:44.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.290+0000 7f8d6effd700 1 -- 172.21.15.27:0/1905296033 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d800279b0 con 0x7f8d88159920 2024-01-11T09:45:44.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.290+0000 7f8d8d5ff700 1 -- 172.21.15.27:0/1905296033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d880b2690 con 0x7f8d88159920 2024-01-11T09:45:44.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.291+0000 7f8d8d5ff700 1 -- 172.21.15.27:0/1905296033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d8804c3c0 con 0x7f8d88159920 2024-01-11T09:45:44.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.292+0000 7f8d6effd700 1 -- 172.21.15.27:0/1905296033 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8d80024070 con 0x7f8d88159920 2024-01-11T09:45:44.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.293+0000 7f8d6effd700 1 --2- 172.21.15.27:0/1905296033 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d74066f30 0x7f8d740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.293+0000 7f8d6effd700 1 -- 172.21.15.27:0/1905296033 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f8d80087130 con 0x7f8d88159920 2024-01-11T09:45:44.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.293+0000 7f8d86ffd700 1 --2- 172.21.15.27:0/1905296033 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d74066f30 0x7f8d740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.294+0000 7f8d86ffd700 1 --2- 172.21.15.27:0/1905296033 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d74066f30 0x7f8d740693e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8d78007fd0 tx=0x7f8d78009040 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:44.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.297+0000 7f8d6effd700 1 -- 172.21.15.27:0/1905296033 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8d80053f30 con 0x7f8d88159920 2024-01-11T09:45:44.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.530+0000 7f8d8d5ff700 1 -- 172.21.15.27:0/1905296033 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d8816a0b0 con 0x7f8d74066f30 2024-01-11T09:45:44.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.533+0000 7f8d6effd700 1 -- 172.21.15.27:0/1905296033 <== mgr.10380 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7f8d8816a0b0 con 0x7f8d74066f30 2024-01-11T09:45:44.524 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T09:45:44.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:44.524 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-11T09:45:44.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:45:44.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:45:44.525 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "backup_fs" 2024-01-11T09:45:44.525 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:45:44.525 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T09:45:44.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.535+0000 7f8d8d5ff700 1 -- 172.21.15.27:0/1905296033 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d74066f30 msgr2=0x7f8d740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.535+0000 7f8d8d5ff700 1 --2- 172.21.15.27:0/1905296033 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d74066f30 0x7f8d740693e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8d78007fd0 tx=0x7f8d78009040 comp rx=0 tx=0).stop 2024-01-11T09:45:44.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.535+0000 7f8d8d5ff700 1 -- 172.21.15.27:0/1905296033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d88159920 msgr2=0x7f8d880b2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.535+0000 7f8d8d5ff700 1 --2- 172.21.15.27:0/1905296033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d88159920 0x7f8d880b2060 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f8d8000b610 tx=0x7f8d80019000 comp rx=0 tx=0).stop 2024-01-11T09:45:44.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.535+0000 7f8d8d5ff700 1 -- 172.21.15.27:0/1905296033 shutdown_connections 2024-01-11T09:45:44.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.535+0000 7f8d8d5ff700 1 --2- 172.21.15.27:0/1905296033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d88159920 0x7f8d880b2060 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.535+0000 7f8d8d5ff700 1 --2- 172.21.15.27:0/1905296033 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d880cb5c0 0x7f8d88154c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.535+0000 7f8d8d5ff700 1 --2- 172.21.15.27:0/1905296033 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d74066f30 0x7f8d740693e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.535+0000 7f8d8d5ff700 1 --2- 172.21.15.27:0/1905296033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d88159550 0x7f8d880b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.535+0000 7f8d8d5ff700 1 -- 172.21.15.27:0/1905296033 >> 172.21.15.27:0/1905296033 conn(0x7f8d8806ed40 msgr2=0x7f8d880cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:44.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.535+0000 7f8d8d5ff700 1 -- 172.21.15.27:0/1905296033 shutdown_connections 2024-01-11T09:45:44.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.535+0000 7f8d8d5ff700 1 -- 172.21.15.27:0/1905296033 wait complete. 2024-01-11T09:45:44.544 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-11T09:45:44.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.753+0000 7f66885ff700 1 Processor -- start 2024-01-11T09:45:44.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.753+0000 7f66885ff700 1 -- start start 2024-01-11T09:45:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.755+0000 7f66885ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f66801577e0 0x7f66800ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.755+0000 7f66885ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66800699d0 con 0x7f6680157410 2024-01-11T09:45:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.755+0000 7f66885ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6680069cb0 con 0x7f6680069540 2024-01-11T09:45:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.755+0000 7f66885ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6680069f90 con 0x7f66801577e0 2024-01-11T09:45:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.755+0000 7f6685b9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f66801577e0 0x7f66800ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.755+0000 7f668639b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6680157410 0x7f66800ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58838/0 (socket says 172.21.15.27:58838) 2024-01-11T09:45:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.755+0000 7f668639b700 1 -- 172.21.15.27:0/4191945427 learned_addr learned my addr 172.21.15.27:0/4191945427 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:44.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.755+0000 7f6685b9a700 1 -- 172.21.15.27:0/4191945427 >> v1:172.21.15.77:6789/0 conn(0x7f6680069540 legacy=0x7f66800c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.755+0000 7f6685b9a700 1 -- 172.21.15.27:0/4191945427 >> v1:172.21.15.27:6789/0 conn(0x7f6680157410 legacy=0x7f66800ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.755+0000 7f6685b9a700 1 -- 172.21.15.27:0/4191945427 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668006a270 con 0x7f66801577e0 2024-01-11T09:45:44.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.755+0000 7f6685399700 1 -- 172.21.15.27:0/4191945427 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932410433 0 0) 0x7f66800699d0 con 0x7f6680157410 2024-01-11T09:45:44.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.756+0000 7f6685b9a700 1 --2- 172.21.15.27:0/4191945427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66801577e0 0x7f66800ae000 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f66700098e0 tx=0x7f66700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5acce57698a79d23 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:44.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.756+0000 7f6685399700 1 -- 172.21.15.27:0/4191945427 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670009e80 con 0x7f66801577e0 2024-01-11T09:45:44.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.756+0000 7f6685399700 1 -- 172.21.15.27:0/4191945427 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667000b860 con 0x7f66801577e0 2024-01-11T09:45:44.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.756+0000 7f6685399700 1 -- 172.21.15.27:0/4191945427 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667000bbe0 con 0x7f66801577e0 2024-01-11T09:45:44.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.757+0000 7f66885ff700 1 -- 172.21.15.27:0/4191945427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66801577e0 msgr2=0x7f66800ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.757+0000 7f66885ff700 1 --2- 172.21.15.27:0/4191945427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66801577e0 0x7f66800ae000 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f66700098e0 tx=0x7f66700091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.757+0000 7f66885ff700 1 -- 172.21.15.27:0/4191945427 shutdown_connections 2024-01-11T09:45:44.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.757+0000 7f66885ff700 1 --2- 172.21.15.27:0/4191945427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66801577e0 0x7f66800ae000 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.757+0000 7f66885ff700 1 -- 172.21.15.27:0/4191945427 >> 172.21.15.27:0/4191945427 conn(0x7f66800680a0 msgr2=0x7f66800cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:44.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.757+0000 7f66885ff700 1 -- 172.21.15.27:0/4191945427 shutdown_connections 2024-01-11T09:45:44.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.757+0000 7f66885ff700 1 -- 172.21.15.27:0/4191945427 wait complete. 2024-01-11T09:45:44.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.758+0000 7f66885ff700 1 Processor -- start 2024-01-11T09:45:44.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.758+0000 7f66885ff700 1 -- start start 2024-01-11T09:45:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.759+0000 7f66885ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6680069540 0x7f668016a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.759+0000 7f66885ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6680157410 0x7f6680158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.759+0000 7f66885ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66801577e0 0x7f6680154000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.759+0000 7f66885ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668016a8c0 con 0x7f6680069540 2024-01-11T09:45:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.759+0000 7f66885ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668015b580 con 0x7f6680157410 2024-01-11T09:45:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.759+0000 7f66885ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f668015ba10 con 0x7f66801577e0 2024-01-11T09:45:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.759+0000 7f668639b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6680069540 0x7f668016a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.760+0000 7f668639b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6680069540 0x7f668016a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44584/0 (socket says 172.21.15.27:44584) 2024-01-11T09:45:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.760+0000 7f668639b700 1 -- 172.21.15.27:0/3561067565 learned_addr learned my addr 172.21.15.27:0/3561067565 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.760+0000 7f6686b9c700 1 --2- 172.21.15.27:0/3561067565 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66801577e0 0x7f6680154000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.760+0000 7f6685b9a700 1 --2- 172.21.15.27:0/3561067565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6680157410 0x7f6680158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.760+0000 7f668639b700 1 -- 172.21.15.27:0/3561067565 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66801577e0 msgr2=0x7f6680154000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.760+0000 7f668639b700 1 --2- 172.21.15.27:0/3561067565 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66801577e0 0x7f6680154000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.760+0000 7f668639b700 1 -- 172.21.15.27:0/3561067565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6680157410 msgr2=0x7f6680158810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.760+0000 7f668639b700 1 --2- 172.21.15.27:0/3561067565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6680157410 0x7f6680158810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.760+0000 7f668639b700 1 -- 172.21.15.27:0/3561067565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6670009420 con 0x7f6680069540 2024-01-11T09:45:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.760+0000 7f6686b9c700 1 --2- 172.21.15.27:0/3561067565 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66801577e0 0x7f6680154000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.760+0000 7f668639b700 1 --2- 172.21.15.27:0/3561067565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6680069540 0x7f668016a380 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f667c00c810 tx=0x7f667c007c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.760+0000 7f66767fc700 1 -- 172.21.15.27:0/3561067565 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c0072e0 con 0x7f6680069540 2024-01-11T09:45:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.761+0000 7f66767fc700 1 -- 172.21.15.27:0/3561067565 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667c007440 con 0x7f6680069540 2024-01-11T09:45:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.761+0000 7f66767fc700 1 -- 172.21.15.27:0/3561067565 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c0057f0 con 0x7f6680069540 2024-01-11T09:45:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.761+0000 7f66885ff700 1 -- 172.21.15.27:0/3561067565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f668015a970 con 0x7f6680069540 2024-01-11T09:45:44.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.761+0000 7f66885ff700 1 -- 172.21.15.27:0/3561067565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f668015af30 con 0x7f6680069540 2024-01-11T09:45:44.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.762+0000 7f66885ff700 1 -- 172.21.15.27:0/3561067565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f668015b860 con 0x7f6680069540 2024-01-11T09:45:44.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.763+0000 7f66767fc700 1 -- 172.21.15.27:0/3561067565 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 127) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f667c005c40 con 0x7f6680069540 2024-01-11T09:45:44.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.764+0000 7f66767fc700 1 --2- 172.21.15.27:0/3561067565 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6664066f30 0x7f66640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:44.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.764+0000 7f66767fc700 1 -- 172.21.15.27:0/3561067565 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f667c083e40 con 0x7f6680069540 2024-01-11T09:45:44.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.764+0000 7f6685b9a700 1 --2- 172.21.15.27:0/3561067565 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6664066f30 0x7f66640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:44.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.765+0000 7f6685b9a700 1 --2- 172.21.15.27:0/3561067565 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6664066f30 0x7f66640693e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f66700098e0 tx=0x7f6670017000 comp rx=0 tx=0).ready entity=mgr.10380 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:44.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:44.768+0000 7f66767fc700 1 -- 172.21.15.27:0/3561067565 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f667c050ce0 con 0x7f6680069540 2024-01-11T09:45:45.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:45.025+0000 7f66885ff700 1 -- 172.21.15.27:0/3561067565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f668004c3c0 con 0x7f6680069540 2024-01-11T09:45:45.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.005+0000 7f66767fc700 1 -- 172.21.15.27:0/3561067565 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 128) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f667c0510b0 con 0x7f6680069540 2024-01-11T09:45:45.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.008+0000 7f66767fc700 1 -- 172.21.15.27:0/3561067565 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v128) v1 ==== 92+0+0 (secure 0 0 0) 0x7f667c054770 con 0x7f6680069540 2024-01-11T09:45:46.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.011+0000 7f66885ff700 1 -- 172.21.15.27:0/3561067565 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6664066f30 msgr2=0x7f66640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:46.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.011+0000 7f66885ff700 1 --2- 172.21.15.27:0/3561067565 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6664066f30 0x7f66640693e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f66700098e0 tx=0x7f6670017000 comp rx=0 tx=0).stop 2024-01-11T09:45:46.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.011+0000 7f66885ff700 1 -- 172.21.15.27:0/3561067565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6680069540 msgr2=0x7f668016a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:46.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.011+0000 7f66885ff700 1 --2- 172.21.15.27:0/3561067565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6680069540 0x7f668016a380 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f667c00c810 tx=0x7f667c007c00 comp rx=0 tx=0).stop 2024-01-11T09:45:46.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.011+0000 7f66885ff700 1 -- 172.21.15.27:0/3561067565 shutdown_connections 2024-01-11T09:45:46.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.011+0000 7f66885ff700 1 --2- 172.21.15.27:0/3561067565 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66801577e0 0x7f6680154000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.011+0000 7f66885ff700 1 --2- 172.21.15.27:0/3561067565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6680069540 0x7f668016a380 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.011+0000 7f66885ff700 1 --2- 172.21.15.27:0/3561067565 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6664066f30 0x7f66640693e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.011+0000 7f66885ff700 1 --2- 172.21.15.27:0/3561067565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6680157410 0x7f6680158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.011+0000 7f66885ff700 1 -- 172.21.15.27:0/3561067565 >> 172.21.15.27:0/3561067565 conn(0x7f66800680a0 msgr2=0x7f66800af6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:46.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.011+0000 7f66885ff700 1 -- 172.21.15.27:0/3561067565 shutdown_connections 2024-01-11T09:45:46.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.011+0000 7f66885ff700 1 -- 172.21.15.27:0/3561067565 wait complete. 2024-01-11T09:45:46.025 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-11T09:45:46.216 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:46.225+0000 7f7d771361c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:45:46.224 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:46.233+0000 7fed702671c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:45:46.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.247+0000 7f53b15ff700 1 Processor -- start 2024-01-11T09:45:46.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.247+0000 7f53b15ff700 1 -- start start 2024-01-11T09:45:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.248+0000 7f53b15ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53ac15a1b0 0x7f53ac15a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.248+0000 7f53b15ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f53ac15ae70 0x7f53ac15b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.248+0000 7f53b15ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53ac0cf850 con 0x7f53ac0ccd70 2024-01-11T09:45:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.248+0000 7f53b15ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53ac0cec40 con 0x7f53ac15ae70 2024-01-11T09:45:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.248+0000 7f53b15ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac0cef20 con 0x7f53ac15a1b0 2024-01-11T09:45:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.249+0000 7f53ab7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f53ac15ae70 0x7f53ac15b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.249+0000 7f53aaffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53ac15a1b0 0x7f53ac15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.249+0000 7f53aaffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53ac15a1b0 0x7f53ac15a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:60036/0 (socket says 172.21.15.27:60036) 2024-01-11T09:45:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.249+0000 7f53ab7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f53ac15ae70 0x7f53ac15b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44620/0 (socket says 172.21.15.27:44620) 2024-01-11T09:45:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.249+0000 7f53aa7fc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f53ac0ccd70 0x7f53ac0cd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:40440/0 (socket says 172.21.15.27:40440) 2024-01-11T09:45:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.249+0000 7f53aaffd700 1 -- 172.21.15.27:0/1421048255 learned_addr learned my addr 172.21.15.27:0/1421048255 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.249+0000 7f53aaffd700 1 -- 172.21.15.27:0/1421048255 >> v2:172.21.15.27:3300/0 conn(0x7f53ac15ae70 msgr2=0x7f53ac15b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:46.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.249+0000 7f53aaffd700 1 --2- 172.21.15.27:0/1421048255 >> v2:172.21.15.27:3300/0 conn(0x7f53ac15ae70 0x7f53ac15b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.249+0000 7f53aaffd700 1 -- 172.21.15.27:0/1421048255 >> v1:172.21.15.77:6789/0 conn(0x7f53ac0ccd70 legacy=0x7f53ac0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:46.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.249+0000 7f53aaffd700 1 -- 172.21.15.27:0/1421048255 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ac0cf200 con 0x7f53ac15a1b0 2024-01-11T09:45:46.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.249+0000 7f53a9ffb700 1 -- 172.21.15.27:0/1421048255 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030612103 0 0) 0x7f53ac0cf850 con 0x7f53ac0ccd70 2024-01-11T09:45:46.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.249+0000 7f53ab7fe700 1 --2- 172.21.15.27:0/1421048255 >> v2:172.21.15.27:3300/0 conn(0x7f53ac15ae70 0x7f53ac15b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:46.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.250+0000 7f53aaffd700 1 --2- 172.21.15.27:0/1421048255 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53ac15a1b0 0x7f53ac15a580 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f539c0099b0 tx=0x7f539c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63259d0ae83d727a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:46.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.250+0000 7f53a9ffb700 1 -- 172.21.15.27:0/1421048255 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c009ea0 con 0x7f53ac15a1b0 2024-01-11T09:45:46.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.250+0000 7f53a9ffb700 1 -- 172.21.15.27:0/1421048255 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f539c00b860 con 0x7f53ac15a1b0 2024-01-11T09:45:46.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.250+0000 7f53a9ffb700 1 -- 172.21.15.27:0/1421048255 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c00bc00 con 0x7f53ac15a1b0 2024-01-11T09:45:46.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.250+0000 7f53b15ff700 1 -- 172.21.15.27:0/1421048255 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53ac15a1b0 msgr2=0x7f53ac15a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:46.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.250+0000 7f53b15ff700 1 --2- 172.21.15.27:0/1421048255 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53ac15a1b0 0x7f53ac15a580 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f539c0099b0 tx=0x7f539c009250 comp rx=0 tx=0).stop 2024-01-11T09:45:46.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.251+0000 7f53b15ff700 1 -- 172.21.15.27:0/1421048255 shutdown_connections 2024-01-11T09:45:46.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.251+0000 7f53b15ff700 1 --2- 172.21.15.27:0/1421048255 >> v2:172.21.15.27:3300/0 conn(0x7f53ac15ae70 0x7f53ac15b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.251+0000 7f53b15ff700 1 --2- 172.21.15.27:0/1421048255 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53ac15a1b0 0x7f53ac15a580 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.251+0000 7f53b15ff700 1 -- 172.21.15.27:0/1421048255 >> 172.21.15.27:0/1421048255 conn(0x7f53ac06ed40 msgr2=0x7f53ac06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:46.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.251+0000 7f53b15ff700 1 -- 172.21.15.27:0/1421048255 shutdown_connections 2024-01-11T09:45:46.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.251+0000 7f53b15ff700 1 -- 172.21.15.27:0/1421048255 wait complete. 2024-01-11T09:45:46.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.253+0000 7f53b15ff700 1 Processor -- start 2024-01-11T09:45:46.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.253+0000 7f53b15ff700 1 -- start start 2024-01-11T09:45:46.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.254+0000 7f53b15ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53ac0ccd70 0x7f53ac168c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:46.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.254+0000 7f53b15ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53ac15a1b0 0x7f53ac0b5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:46.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.254+0000 7f53b15ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53ac15ae70 0x7f53ac15dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:46.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.254+0000 7f53b15ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac14fc80 con 0x7f53ac0ccd70 2024-01-11T09:45:46.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.254+0000 7f53b15ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac15e130 con 0x7f53ac15ae70 2024-01-11T09:45:46.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.254+0000 7f53b15ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac153390 con 0x7f53ac15a1b0 2024-01-11T09:45:46.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.255+0000 7f53aaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53ac0ccd70 0x7f53ac168c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.255+0000 7f53aa7fc700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53ac15a1b0 0x7f53ac0b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.255+0000 7f53aaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53ac0ccd70 0x7f53ac168c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44632/0 (socket says 172.21.15.27:44632) 2024-01-11T09:45:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.255+0000 7f53aaffd700 1 -- 172.21.15.27:0/2758931274 learned_addr learned my addr 172.21.15.27:0/2758931274 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.255+0000 7f53ab7fe700 1 --2- 172.21.15.27:0/2758931274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53ac15ae70 0x7f53ac15dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.255+0000 7f53aaffd700 1 -- 172.21.15.27:0/2758931274 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53ac15a1b0 msgr2=0x7f53ac0b5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.255+0000 7f53aaffd700 1 --2- 172.21.15.27:0/2758931274 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53ac15a1b0 0x7f53ac0b5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.255+0000 7f53aaffd700 1 -- 172.21.15.27:0/2758931274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53ac15ae70 msgr2=0x7f53ac15dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.255+0000 7f53aaffd700 1 --2- 172.21.15.27:0/2758931274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53ac15ae70 0x7f53ac15dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.255+0000 7f53aaffd700 1 -- 172.21.15.27:0/2758931274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a4009a20 con 0x7f53ac0ccd70 2024-01-11T09:45:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.255+0000 7f53aa7fc700 1 --2- 172.21.15.27:0/2758931274 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53ac15a1b0 0x7f53ac0b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.255+0000 7f53aaffd700 1 --2- 172.21.15.27:0/2758931274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53ac0ccd70 0x7f53ac168c70 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f53ac0cf060 tx=0x7f539c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.256+0000 7f5392ffd700 1 -- 172.21.15.27:0/2758931274 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c00bbe0 con 0x7f53ac0ccd70 2024-01-11T09:45:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.256+0000 7f5392ffd700 1 -- 172.21.15.27:0/2758931274 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f539c011420 con 0x7f53ac0ccd70 2024-01-11T09:45:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.256+0000 7f53b15ff700 1 -- 172.21.15.27:0/2758931274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f539c0094f0 con 0x7f53ac0ccd70 2024-01-11T09:45:46.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.256+0000 7f5392ffd700 1 -- 172.21.15.27:0/2758931274 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c0117e0 con 0x7f53ac0ccd70 2024-01-11T09:45:46.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.256+0000 7f53b15ff700 1 -- 172.21.15.27:0/2758931274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53ac15e620 con 0x7f53ac0ccd70 2024-01-11T09:45:46.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.257+0000 7f53b15ff700 1 -- 172.21.15.27:0/2758931274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53ac04c3c0 con 0x7f53ac0ccd70 2024-01-11T09:45:46.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.258+0000 7f5392ffd700 1 -- 172.21.15.27:0/2758931274 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 128) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f539c011bd0 con 0x7f53ac0ccd70 2024-01-11T09:45:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.259+0000 7f5392ffd700 1 --2- 172.21.15.27:0/2758931274 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5398066f80 0x7f5398069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.259+0000 7f5392ffd700 1 -- 172.21.15.27:0/2758931274 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f539c0861e0 con 0x7f53ac0ccd70 2024-01-11T09:45:46.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.259+0000 7f53aa7fc700 1 -- 172.21.15.27:0/2758931274 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5398066f80 msgr2=0x7f5398069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:46.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.259+0000 7f53aa7fc700 1 --2- 172.21.15.27:0/2758931274 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5398066f80 0x7f5398069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:45:46.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.263+0000 7f5392ffd700 1 -- 172.21.15.27:0/2758931274 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f539c0530a0 con 0x7f53ac0ccd70 2024-01-11T09:45:46.397 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:46.403+0000 7f7d771361c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:45:46.420 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:46.429+0000 7fed702671c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:45:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.460+0000 7f53aa7fc700 1 -- 172.21.15.27:0/2758931274 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5398066f80 msgr2=0x7f5398069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.460+0000 7f53aa7fc700 1 --2- 172.21.15.27:0/2758931274 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5398066f80 0x7f5398069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:45:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.510+0000 7f53b15ff700 1 -- 172.21.15.27:0/2758931274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f53ac067610 con 0x7f53ac0ccd70 2024-01-11T09:45:46.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.515+0000 7f5392ffd700 1 -- 172.21.15.27:0/2758931274 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v77)=0 v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7f539c0566c0 con 0x7f53ac0ccd70 2024-01-11T09:45:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.518+0000 7f53b15ff700 1 -- 172.21.15.27:0/2758931274 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5398066f80 msgr2=0x7f5398069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:45:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.518+0000 7f53b15ff700 1 --2- 172.21.15.27:0/2758931274 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5398066f80 0x7f5398069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.518+0000 7f53b15ff700 1 -- 172.21.15.27:0/2758931274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53ac0ccd70 msgr2=0x7f53ac168c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.518+0000 7f53b15ff700 1 --2- 172.21.15.27:0/2758931274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53ac0ccd70 0x7f53ac168c70 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f53ac0cf060 tx=0x7f539c017000 comp rx=0 tx=0).stop 2024-01-11T09:45:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.518+0000 7f53b15ff700 1 -- 172.21.15.27:0/2758931274 shutdown_connections 2024-01-11T09:45:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.518+0000 7f53b15ff700 1 --2- 172.21.15.27:0/2758931274 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5398066f80 0x7f5398069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.518+0000 7f53b15ff700 1 --2- 172.21.15.27:0/2758931274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53ac15ae70 0x7f53ac15dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.518+0000 7f53b15ff700 1 --2- 172.21.15.27:0/2758931274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53ac0ccd70 0x7f53ac168c70 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.518+0000 7f53b15ff700 1 --2- 172.21.15.27:0/2758931274 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f53ac15a1b0 0x7f53ac0b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.518+0000 7f53b15ff700 1 -- 172.21.15.27:0/2758931274 >> 172.21.15.27:0/2758931274 conn(0x7f53ac06ed40 msgr2=0x7f53ac0c57f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.518+0000 7f53b15ff700 1 -- 172.21.15.27:0/2758931274 shutdown_connections 2024-01-11T09:45:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.518+0000 7f53b15ff700 1 -- 172.21.15.27:0/2758931274 wait complete. 2024-01-11T09:45:46.528 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-11T09:45:46.709 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:46.717+0000 7f7d771361c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:45:46.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.742+0000 7f83cf9ff700 1 Processor -- start 2024-01-11T09:45:46.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.743+0000 7f83cf9ff700 1 -- start start 2024-01-11T09:45:46.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.744+0000 7f83cf9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f83c815a1b0 0x7f83c815a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:46.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.744+0000 7f83cf9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f83c80ccd70 0x7f83c80cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:46.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.744+0000 7f83cf9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c80cf850 con 0x7f83c80685c0 2024-01-11T09:45:46.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.744+0000 7f83cf9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83c80cec40 con 0x7f83c80ccd70 2024-01-11T09:45:46.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.744+0000 7f83cf9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c80cef20 con 0x7f83c815a1b0 2024-01-11T09:45:46.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.744+0000 7f83ccf9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f83c80ccd70 0x7f83c80cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:46.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.744+0000 7f83cd79b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f83c815a1b0 0x7f83c815a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:46.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.744+0000 7f83ccf9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f83c80ccd70 0x7f83c80cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44640/0 (socket says 172.21.15.27:44640) 2024-01-11T09:45:46.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.745+0000 7f83ccf9a700 1 -- 172.21.15.27:0/2139331594 learned_addr learned my addr 172.21.15.27:0/2139331594 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:46.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.744+0000 7f83cdf9c700 1 --1- 172.21.15.27:0/2139331594 >> v1:172.21.15.77:6789/0 conn(0x7f83c80685c0 0x7f83c8068a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:40448/0 (socket says 172.21.15.27:40448) 2024-01-11T09:45:46.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.745+0000 7f83bffff700 1 -- 172.21.15.27:0/2139331594 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103096698 0 0) 0x7f83c80cf850 con 0x7f83c80685c0 2024-01-11T09:45:46.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.745+0000 7f83bffff700 1 -- 172.21.15.27:0/2139331594 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b4003570 con 0x7f83c80685c0 2024-01-11T09:45:46.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.745+0000 7f83ccf9a700 1 -- 172.21.15.27:0/2139331594 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f83c815a1b0 msgr2=0x7f83c815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:46.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.745+0000 7f83ccf9a700 1 --2- 172.21.15.27:0/2139331594 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f83c815a1b0 0x7f83c815a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.745+0000 7f83ccf9a700 1 -- 172.21.15.27:0/2139331594 >> v1:172.21.15.77:6789/0 conn(0x7f83c80685c0 legacy=0x7f83c8068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:46.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.745+0000 7f83ccf9a700 1 -- 172.21.15.27:0/2139331594 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c80cf200 con 0x7f83c80ccd70 2024-01-11T09:45:46.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.745+0000 7f83cd79b700 1 --2- 172.21.15.27:0/2139331594 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f83c815a1b0 0x7f83c815a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:46.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.745+0000 7f83ccf9a700 1 --2- 172.21.15.27:0/2139331594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c80ccd70 0x7f83c80cd1e0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f83b80099b0 tx=0x7f83b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a02262f7e297f37d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:46.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.746+0000 7f83bffff700 1 -- 172.21.15.27:0/2139331594 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b800b680 con 0x7f83c80ccd70 2024-01-11T09:45:46.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.746+0000 7f83bffff700 1 -- 172.21.15.27:0/2139331594 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b800b7e0 con 0x7f83c80ccd70 2024-01-11T09:45:46.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.746+0000 7f83bffff700 1 -- 172.21.15.27:0/2139331594 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b800bb80 con 0x7f83c80ccd70 2024-01-11T09:45:46.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.746+0000 7f83cf9ff700 1 -- 172.21.15.27:0/2139331594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c80ccd70 msgr2=0x7f83c80cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:46.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.747+0000 7f83cf9ff700 1 --2- 172.21.15.27:0/2139331594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c80ccd70 0x7f83c80cd1e0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f83b80099b0 tx=0x7f83b8009250 comp rx=0 tx=0).stop 2024-01-11T09:45:46.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.747+0000 7f83cf9ff700 1 -- 172.21.15.27:0/2139331594 shutdown_connections 2024-01-11T09:45:46.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.747+0000 7f83cf9ff700 1 --2- 172.21.15.27:0/2139331594 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f83c815a1b0 0x7f83c815a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.747+0000 7f83cf9ff700 1 --2- 172.21.15.27:0/2139331594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c80ccd70 0x7f83c80cd1e0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.747+0000 7f83cf9ff700 1 -- 172.21.15.27:0/2139331594 >> 172.21.15.27:0/2139331594 conn(0x7f83c806ed40 msgr2=0x7f83c806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:46.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.747+0000 7f83cf9ff700 1 -- 172.21.15.27:0/2139331594 shutdown_connections 2024-01-11T09:45:46.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.747+0000 7f83cf9ff700 1 -- 172.21.15.27:0/2139331594 wait complete. 2024-01-11T09:45:46.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.748+0000 7f83cf9ff700 1 Processor -- start 2024-01-11T09:45:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.748+0000 7f83cf9ff700 1 -- start start 2024-01-11T09:45:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.749+0000 7f83cf9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f83c80685c0 0x7f83c8156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.749+0000 7f83cf9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83c80ccd70 0x7f83c815ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.749+0000 7f83cf9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c815a1b0 0x7f83c815f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.749+0000 7f83cf9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c816a690 con 0x7f83c815a1b0 2024-01-11T09:45:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.749+0000 7f83cf9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c816a390 con 0x7f83c80ccd70 2024-01-11T09:45:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.749+0000 7f83cf9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c8156ad0 con 0x7f83c80685c0 2024-01-11T09:45:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.750+0000 7f83cdf9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c815a1b0 0x7f83c815f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.750+0000 7f83cdf9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c815a1b0 0x7f83c815f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44654/0 (socket says 172.21.15.27:44654) 2024-01-11T09:45:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.750+0000 7f83cdf9c700 1 -- 172.21.15.27:0/165671409 learned_addr learned my addr 172.21.15.27:0/165671409 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.750+0000 7f83ccf9a700 1 --2- 172.21.15.27:0/165671409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83c80ccd70 0x7f83c815ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.750+0000 7f83cd79b700 1 --2- 172.21.15.27:0/165671409 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f83c80685c0 0x7f83c8156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.750+0000 7f83cdf9c700 1 -- 172.21.15.27:0/165671409 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f83c80685c0 msgr2=0x7f83c8156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.750+0000 7f83cdf9c700 1 --2- 172.21.15.27:0/165671409 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f83c80685c0 0x7f83c8156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.750+0000 7f83cdf9c700 1 -- 172.21.15.27:0/165671409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83c80ccd70 msgr2=0x7f83c815ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.750+0000 7f83cdf9c700 1 --2- 172.21.15.27:0/165671409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83c80ccd70 0x7f83c815ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.750+0000 7f83cdf9c700 1 -- 172.21.15.27:0/165671409 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c4009420 con 0x7f83c815a1b0 2024-01-11T09:45:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.750+0000 7f83cdf9c700 1 --2- 172.21.15.27:0/165671409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c815a1b0 0x7f83c815f1c0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f83c000cd30 tx=0x7f83c0001b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:46.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.751+0000 7f83bd7fa700 1 -- 172.21.15.27:0/165671409 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c000ce90 con 0x7f83c815a1b0 2024-01-11T09:45:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.751+0000 7f83bd7fa700 1 -- 172.21.15.27:0/165671409 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83c0005df0 con 0x7f83c815a1b0 2024-01-11T09:45:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.751+0000 7f83bd7fa700 1 -- 172.21.15.27:0/165671409 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c000aac0 con 0x7f83c815a1b0 2024-01-11T09:45:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.751+0000 7f83cf9ff700 1 -- 172.21.15.27:0/165671409 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b80094f0 con 0x7f83c815a1b0 2024-01-11T09:45:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.751+0000 7f83cf9ff700 1 -- 172.21.15.27:0/165671409 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c8169780 con 0x7f83c815a1b0 2024-01-11T09:45:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.752+0000 7f83cf9ff700 1 -- 172.21.15.27:0/165671409 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c804c3c0 con 0x7f83c815a1b0 2024-01-11T09:45:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.753+0000 7f83bd7fa700 1 -- 172.21.15.27:0/165671409 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 128) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f83c00070f0 con 0x7f83c815a1b0 2024-01-11T09:45:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.754+0000 7f83bd7fa700 1 --2- 172.21.15.27:0/165671409 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f83ac066f80 0x7f83ac069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.754+0000 7f83bd7fa700 1 -- 172.21.15.27:0/165671409 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f83c0084b50 con 0x7f83c815a1b0 2024-01-11T09:45:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.754+0000 7f83cd79b700 1 -- 172.21.15.27:0/165671409 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f83ac066f80 msgr2=0x7f83ac069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.754+0000 7f83cd79b700 1 --2- 172.21.15.27:0/165671409 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f83ac066f80 0x7f83ac069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:45:46.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.762+0000 7f83bd7fa700 1 -- 172.21.15.27:0/165671409 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f83c0051950 con 0x7f83c815a1b0 2024-01-11T09:45:46.770 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:46.779+0000 7fed702671c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:45:46.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.955+0000 7f83cd79b700 1 -- 172.21.15.27:0/165671409 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f83ac066f80 msgr2=0x7f83ac069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:46.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.955+0000 7f83cd79b700 1 --2- 172.21.15.27:0/165671409 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f83ac066f80 0x7f83ac069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:45:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.968+0000 7f83cf9ff700 1 -- 172.21.15.27:0/165671409 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f83c80ce0e0 con 0x7f83c815a1b0 2024-01-11T09:45:46.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.972+0000 7f83bd7fa700 1 -- 172.21.15.27:0/165671409 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v78)=0 v78) v1 ==== 133+0+0 (secure 0 0 0) 0x7f83c0054f70 con 0x7f83c815a1b0 2024-01-11T09:45:46.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.974+0000 7f83cf9ff700 1 -- 172.21.15.27:0/165671409 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f83ac066f80 msgr2=0x7f83ac069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:45:46.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.974+0000 7f83cf9ff700 1 --2- 172.21.15.27:0/165671409 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f83ac066f80 0x7f83ac069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.974+0000 7f83cf9ff700 1 -- 172.21.15.27:0/165671409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c815a1b0 msgr2=0x7f83c815f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.974+0000 7f83cf9ff700 1 --2- 172.21.15.27:0/165671409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c815a1b0 0x7f83c815f1c0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f83c000cd30 tx=0x7f83c0001b60 comp rx=0 tx=0).stop 2024-01-11T09:45:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.975+0000 7f83cf9ff700 1 -- 172.21.15.27:0/165671409 shutdown_connections 2024-01-11T09:45:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.975+0000 7f83cf9ff700 1 --2- 172.21.15.27:0/165671409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83c815a1b0 0x7f83c815f1c0 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.975+0000 7f83cf9ff700 1 --2- 172.21.15.27:0/165671409 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f83c80685c0 0x7f83c8156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.975+0000 7f83cf9ff700 1 --2- 172.21.15.27:0/165671409 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f83ac066f80 0x7f83ac069430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.975+0000 7f83cf9ff700 1 --2- 172.21.15.27:0/165671409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83c80ccd70 0x7f83c815ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.975+0000 7f83cf9ff700 1 -- 172.21.15.27:0/165671409 >> 172.21.15.27:0/165671409 conn(0x7f83c806ed40 msgr2=0x7f83c815b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.975+0000 7f83cf9ff700 1 -- 172.21.15.27:0/165671409 shutdown_connections 2024-01-11T09:45:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:46.975+0000 7f83cf9ff700 1 -- 172.21.15.27:0/165671409 wait complete. 2024-01-11T09:45:46.985 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:45:46.985 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2024-01-11T09:45:47.040 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-01-11T09:45:47.237 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:47.246+0000 7f7d771361c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:45:47.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.259+0000 7f0c635ff700 1 Processor -- start 2024-01-11T09:45:47.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.259+0000 7f0c635ff700 1 -- start start 2024-01-11T09:45:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.261+0000 7f0c635ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0c5c157440 0x7f0c5c0ae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.261+0000 7f0c635ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0c5c069d70 0x7f0c5c161180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.261+0000 7f0c635ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c5c06d420 con 0x7f0c5c157810 2024-01-11T09:45:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.261+0000 7f0c635ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c5c06d700 con 0x7f0c5c069d70 2024-01-11T09:45:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.261+0000 7f0c635ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c5c06d9e0 con 0x7f0c5c157440 2024-01-11T09:45:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.261+0000 7f0c61b9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0c5c069d70 0x7f0c5c161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:47.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.261+0000 7f0c6139b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0c5c157440 0x7f0c5c0ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:47.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.261+0000 7f0c61b9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0c5c069d70 0x7f0c5c161180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44656/0 (socket says 172.21.15.27:44656) 2024-01-11T09:45:47.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.261+0000 7f0c61b9c700 1 -- 172.21.15.27:0/2543048860 learned_addr learned my addr 172.21.15.27:0/2543048860 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.262+0000 7f0c6139b700 1 -- 172.21.15.27:0/2543048860 >> v2:172.21.15.27:3300/0 conn(0x7f0c5c069d70 msgr2=0x7f0c5c161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.262+0000 7f0c6139b700 1 --2- 172.21.15.27:0/2543048860 >> v2:172.21.15.27:3300/0 conn(0x7f0c5c069d70 0x7f0c5c161180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.262+0000 7f0c6139b700 1 -- 172.21.15.27:0/2543048860 >> v1:172.21.15.77:6789/0 conn(0x7f0c5c157810 legacy=0x7f0c5c069660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.262+0000 7f0c6139b700 1 -- 172.21.15.27:0/2543048860 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5c06dcc0 con 0x7f0c5c157440 2024-01-11T09:45:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.262+0000 7f0c53fff700 1 -- 172.21.15.27:0/2543048860 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751673849 0 0) 0x7f0c5c06d420 con 0x7f0c5c157810 2024-01-11T09:45:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.262+0000 7f0c6139b700 1 --2- 172.21.15.27:0/2543048860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c5c157440 0x7f0c5c0ae430 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c070ce0 tx=0x7f0c58009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58fbc99790f22e2a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:47.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.262+0000 7f0c53fff700 1 -- 172.21.15.27:0/2543048860 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c58005c10 con 0x7f0c5c157440 2024-01-11T09:45:47.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.262+0000 7f0c53fff700 1 -- 172.21.15.27:0/2543048860 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c58005d70 con 0x7f0c5c157440 2024-01-11T09:45:47.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.262+0000 7f0c53fff700 1 -- 172.21.15.27:0/2543048860 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c58006260 con 0x7f0c5c157440 2024-01-11T09:45:47.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.263+0000 7f0c635ff700 1 -- 172.21.15.27:0/2543048860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c5c157440 msgr2=0x7f0c5c0ae430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:47.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.263+0000 7f0c635ff700 1 --2- 172.21.15.27:0/2543048860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c5c157440 0x7f0c5c0ae430 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c070ce0 tx=0x7f0c58009120 comp rx=0 tx=0).stop 2024-01-11T09:45:47.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.263+0000 7f0c635ff700 1 -- 172.21.15.27:0/2543048860 shutdown_connections 2024-01-11T09:45:47.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.263+0000 7f0c635ff700 1 --2- 172.21.15.27:0/2543048860 >> v2:172.21.15.27:3300/0 conn(0x7f0c5c069d70 0x7f0c5c161180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:47.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.263+0000 7f0c635ff700 1 --2- 172.21.15.27:0/2543048860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c5c157440 0x7f0c5c0ae430 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:47.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.263+0000 7f0c635ff700 1 -- 172.21.15.27:0/2543048860 >> 172.21.15.27:0/2543048860 conn(0x7f0c5c0680e0 msgr2=0x7f0c5c0cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:47.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.263+0000 7f0c635ff700 1 -- 172.21.15.27:0/2543048860 shutdown_connections 2024-01-11T09:45:47.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.263+0000 7f0c635ff700 1 -- 172.21.15.27:0/2543048860 wait complete. 2024-01-11T09:45:47.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.265+0000 7f0c635ff700 1 Processor -- start 2024-01-11T09:45:47.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.265+0000 7f0c635ff700 1 -- start start 2024-01-11T09:45:47.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.266+0000 7f0c635ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c5c069d70 0x7f0c5c0aa480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:47.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.266+0000 7f0c635ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0c5c157440 0x7f0c5c0aaf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:47.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.266+0000 7f0c635ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c5c157810 0x7f0c5c14b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:47.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.266+0000 7f0c635ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5c0ab460 con 0x7f0c5c157810 2024-01-11T09:45:47.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.266+0000 7f0c635ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5c0ab650 con 0x7f0c5c069d70 2024-01-11T09:45:47.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.266+0000 7f0c635ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5c1586a0 con 0x7f0c5c157440 2024-01-11T09:45:47.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.266+0000 7f0c6139b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c5c069d70 0x7f0c5c0aa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:47.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.266+0000 7f0c61b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c5c157810 0x7f0c5c14b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:47.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.266+0000 7f0c6139b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c5c069d70 0x7f0c5c0aa480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:44602/0 (socket says 172.21.15.27:44602) 2024-01-11T09:45:47.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.266+0000 7f0c6139b700 1 -- 172.21.15.27:0/1469694896 learned_addr learned my addr 172.21.15.27:0/1469694896 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:47.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.266+0000 7f0c60b9a700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0c5c157440 0x7f0c5c0aaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:47.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.267+0000 7f0c6139b700 1 -- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0c5c157440 msgr2=0x7f0c5c0aaf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:47.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.267+0000 7f0c6139b700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0c5c157440 0x7f0c5c0aaf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:47.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.267+0000 7f0c6139b700 1 -- 172.21.15.27:0/1469694896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c5c157810 msgr2=0x7f0c5c14b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:47.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.267+0000 7f0c6139b700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c5c157810 0x7f0c5c14b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:47.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.267+0000 7f0c6139b700 1 -- 172.21.15.27:0/1469694896 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c580093c0 con 0x7f0c5c069d70 2024-01-11T09:45:47.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.267+0000 7f0c6139b700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c5c069d70 0x7f0c5c0aa480 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f0c58006210 tx=0x7f0c58016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:47.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.267+0000 7f0c517fa700 1 -- 172.21.15.27:0/1469694896 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5801a700 con 0x7f0c5c069d70 2024-01-11T09:45:47.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.268+0000 7f0c517fa700 1 -- 172.21.15.27:0/1469694896 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5801a860 con 0x7f0c5c069d70 2024-01-11T09:45:47.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.268+0000 7f0c635ff700 1 -- 172.21.15.27:0/1469694896 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c5c157d70 con 0x7f0c5c069d70 2024-01-11T09:45:47.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.268+0000 7f0c517fa700 1 -- 172.21.15.27:0/1469694896 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5801ac20 con 0x7f0c5c069d70 2024-01-11T09:45:47.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.268+0000 7f0c635ff700 1 -- 172.21.15.27:0/1469694896 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c5c158330 con 0x7f0c5c069d70 2024-01-11T09:45:47.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.269+0000 7f0c635ff700 1 -- 172.21.15.27:0/1469694896 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c5c04c3c0 con 0x7f0c5c069d70 2024-01-11T09:45:47.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.270+0000 7f0c517fa700 1 -- 172.21.15.27:0/1469694896 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 128) v1 ==== 84759+0+0 (secure 0 0 0) 0x7f0c58011630 con 0x7f0c5c069d70 2024-01-11T09:45:47.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.271+0000 7f0c517fa700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c40066f80 0x7f0c40069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:47.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.271+0000 7f0c517fa700 1 -- 172.21.15.27:0/1469694896 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f0c58085e20 con 0x7f0c5c069d70 2024-01-11T09:45:47.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.271+0000 7f0c60b9a700 1 -- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c40066f80 msgr2=0x7f0c40069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:47.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.271+0000 7f0c60b9a700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c40066f80 0x7f0c40069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:45:47.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.276+0000 7f0c517fa700 1 -- 172.21.15.27:0/1469694896 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0c58052c20 con 0x7f0c5c069d70 2024-01-11T09:45:47.342 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:47.351+0000 7fed702671c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:45:47.409 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:47.418+0000 7f7d771361c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:45:47.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.472+0000 7f0c60b9a700 1 -- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c40066f80 msgr2=0x7f0c40069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:47.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.472+0000 7f0c60b9a700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c40066f80 0x7f0c40069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:45:47.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.506+0000 7f0c635ff700 1 -- 172.21.15.27:0/1469694896 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c5c1587e0 con 0x7f0c40066f80 2024-01-11T09:45:47.523 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:47.531+0000 7fed702671c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:45:47.825 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:47.834+0000 7f7d771361c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:45:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.873+0000 7f0c60b9a700 1 -- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c40066f80 msgr2=0x7f0c40069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:47.873+0000 7f0c60b9a700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c40066f80 0x7f0c40069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-11T09:45:47.960 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:47.968+0000 7fed702671c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:45:48.391 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:45:48.396 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:45:48.614 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:48.621+0000 7f7d771361c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:45:48.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:48.674+0000 7f0c60b9a700 1 -- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c40066f80 msgr2=0x7f0c40069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:48.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:48.674+0000 7f0c60b9a700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c40066f80 0x7f0c40069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-11T09:45:48.702 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:48.710+0000 7f7d771361c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:45:48.788 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:48.796+0000 7fed702671c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:45:48.857 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:48.865+0000 7f7d771361c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:45:48.880 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:48.889+0000 7fed702671c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:45:48.944 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:48.952+0000 7f7d771361c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:45:49.046 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:49.054+0000 7fed702671c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:45:49.139 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:49.148+0000 7fed702671c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:45:49.162 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:49.170+0000 7f7d771361c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:45:49.257 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:49.265+0000 7f7d771361c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:45:49.353 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:49.361+0000 7f7d771361c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:45:49.367 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:49.375+0000 7fed702671c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:45:49.448 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:49.456+0000 7f7d771361c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:45:49.468 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:49.477+0000 7fed702671c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:45:49.569 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:49.577+0000 7fed702671c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:45:49.640 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:49.648+0000 7f7d771361c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:45:49.669 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:49.677+0000 7fed702671c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:45:49.738 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:49.746+0000 7f7d771361c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:45:49.869 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:49.877+0000 7fed702671c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:45:49.973 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:49.981+0000 7fed702671c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:45:50.080 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:50.088+0000 7f7d771361c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:45:50.171 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:50.178+0000 7f7d771361c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:45:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:50.276+0000 7f0c60b9a700 1 -- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c40066f80 msgr2=0x7f0c40069430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:50.276+0000 7f0c60b9a700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c40066f80 0x7f0c40069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-11T09:45:50.338 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:50.347+0000 7fed702671c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:45:50.433 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:50.441+0000 7fed702671c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:45:50.710 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:50.718+0000 7f7d771361c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:45:50.898 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:50.906+0000 7f7d771361c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:45:51.010 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:51.019+0000 7fed702671c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:45:51.208 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:51.216+0000 7fed702671c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:45:51.279 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:51.287+0000 7f7d771361c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:45:51.607 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:51.615+0000 7fed702671c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:45:51.702 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:51.710+0000 7f7d771361c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:45:51.883 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:51.891+0000 7f7d771361c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:45:52.059 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:52.067+0000 7fed702671c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:45:52.071 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.079+0000 7f7d771361c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:45:52.180 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.188+0000 7f7d771361c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:45:52.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:52.201+0000 7f0c517fa700 1 -- 172.21.15.27:0/1469694896 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 129) v1 ==== 84470+0+0 (secure 0 0 0) 0x7f0c5808b020 con 0x7f0c5c069d70 2024-01-11T09:45:52.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:52.201+0000 7f0c517fa700 1 -- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c40066f80 msgr2=0x7f0c40069430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:45:52.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:52.201+0000 7f0c517fa700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c40066f80 0x7f0c40069430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:52.261 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:52.269+0000 7fed702671c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:45:52.363 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.369+0000 7f7cebf78700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.363 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.369+0000 7f7cebf78700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.364 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.369+0000 7f7cebf78700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.364 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.369+0000 7f7cebf78700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.364 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.369+0000 7f7cebf78700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.364 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.369+0000 7f7ceef7e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.364 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.369+0000 7f7ceef7e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.364 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.369+0000 7f7ceef7e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.364 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.369+0000 7f7ceef7e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.364 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.369+0000 7f7ceef7e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.374 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.382+0000 7f7ceef7e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.374 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.382+0000 7f7ceef7e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.374 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.382+0000 7f7ceef7e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.374 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.382+0000 7f7ceef7e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.374 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.382+0000 7f7ceef7e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.376 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.384+0000 7f7cebf78700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.376 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.384+0000 7f7cebf78700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.376 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.384+0000 7f7cebf78700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.377 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.384+0000 7f7cebf78700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.377 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:52.384+0000 7f7cebf78700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:45:52.466 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:52.475+0000 7fed702671c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:45:52.582 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:52.591+0000 7fed702671c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:45:53.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.205+0000 7f0c517fa700 1 -- 172.21.15.27:0/1469694896 <== mon.1 v2:172.21.15.77:3300/0 8 ==== mgrmap(e 130) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f0c58052ff0 con 0x7f0c5c069d70 2024-01-11T09:45:53.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.205+0000 7f0c517fa700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c40066f80 0x7f0c400693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:53.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.205+0000 7f0c517fa700 1 -- 172.21.15.27:0/1469694896 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c40071670 con 0x7f0c40066f80 2024-01-11T09:45:53.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.206+0000 7f0c60b9a700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c40066f80 0x7f0c400693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:53.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.208+0000 7f0c60b9a700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c40066f80 0x7f0c400693b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c0aaeb0 tx=0x7f0c4c007ec0 comp rx=0 tx=0).ready entity=mgr.10501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:53.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.271+0000 7f0c517fa700 1 -- 172.21.15.27:0/1469694896 <== mgr.10501 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+53 (secure 0 0 0) 0x7f0c40071670 con 0x7f0c40066f80 2024-01-11T09:45:53.263 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path mandatory_fs_argument_test_dir 2024-01-11T09:45:53.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.274+0000 7f0c635ff700 1 -- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c40066f80 msgr2=0x7f0c400693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:53.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.274+0000 7f0c635ff700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c40066f80 0x7f0c400693b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c0aaeb0 tx=0x7f0c4c007ec0 comp rx=0 tx=0).stop 2024-01-11T09:45:53.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.274+0000 7f0c635ff700 1 -- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c5c069d70 msgr2=0x7f0c5c0aa480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:53.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.274+0000 7f0c635ff700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c5c069d70 0x7f0c5c0aa480 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f0c58006210 tx=0x7f0c58016000 comp rx=0 tx=0).stop 2024-01-11T09:45:53.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.274+0000 7f0c635ff700 1 -- 172.21.15.27:0/1469694896 shutdown_connections 2024-01-11T09:45:53.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.274+0000 7f0c635ff700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c5c157810 0x7f0c5c14b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:53.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.274+0000 7f0c635ff700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0c40066f80 0x7f0c400693b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:53.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.274+0000 7f0c635ff700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c5c069d70 0x7f0c5c0aa480 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:53.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.274+0000 7f0c635ff700 1 --2- 172.21.15.27:0/1469694896 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0c5c157440 0x7f0c5c0aaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:53.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.274+0000 7f0c635ff700 1 -- 172.21.15.27:0/1469694896 >> 172.21.15.27:0/1469694896 conn(0x7f0c5c0680e0 msgr2=0x7f0c5c0c7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:53.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.274+0000 7f0c635ff700 1 -- 172.21.15.27:0/1469694896 shutdown_connections 2024-01-11T09:45:53.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.274+0000 7f0c635ff700 1 -- 172.21.15.27:0/1469694896 wait complete. 2024-01-11T09:45:53.285 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path mandatory_fs_argument_test_dir 2024-01-11T09:45:53.285 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-01-11T09:45:53.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.498+0000 7fa600dff700 1 Processor -- start 2024-01-11T09:45:53.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.498+0000 7fa600dff700 1 -- start start 2024-01-11T09:45:53.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.499+0000 7fa600dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5fc156440 0x7fa5fc0c7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:53.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.499+0000 7fa600dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5fc156810 0x7fa5fc06e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:53.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.499+0000 7fa600dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc0c8590 con 0x7fa5fc0ccac0 2024-01-11T09:45:53.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.499+0000 7fa600dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5fc0c7980 con 0x7fa5fc156440 2024-01-11T09:45:53.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.499+0000 7fa600dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc0c7c60 con 0x7fa5fc156810 2024-01-11T09:45:53.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.500+0000 7fa5fa59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5fc156440 0x7fa5fc0c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:53.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.500+0000 7fa5fa59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5fc156440 0x7fa5fc0c7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44668/0 (socket says 172.21.15.27:44668) 2024-01-11T09:45:53.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.500+0000 7fa5fa59c700 1 -- 172.21.15.27:0/1092185298 learned_addr learned my addr 172.21.15.27:0/1092185298 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:53.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.500+0000 7fa5f9d9b700 1 --2- 172.21.15.27:0/1092185298 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5fc156810 0x7fa5fc06e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:53.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.500+0000 7fa5fa59c700 1 -- 172.21.15.27:0/1092185298 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5fc156810 msgr2=0x7fa5fc06e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:53.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.500+0000 7fa5fa59c700 1 --2- 172.21.15.27:0/1092185298 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5fc156810 0x7fa5fc06e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:53.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.500+0000 7fa5fa59c700 1 -- 172.21.15.27:0/1092185298 >> v1:172.21.15.77:6789/0 conn(0x7fa5fc0ccac0 legacy=0x7fa5fc06d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:53.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.500+0000 7fa5fa59c700 1 -- 172.21.15.27:0/1092185298 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc0c7f40 con 0x7fa5fc156440 2024-01-11T09:45:53.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.500+0000 7fa5f959a700 1 -- 172.21.15.27:0/1092185298 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654664157 0 0) 0x7fa5fc0c8590 con 0x7fa5fc0ccac0 2024-01-11T09:45:53.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.500+0000 7fa5fa59c700 1 --2- 172.21.15.27:0/1092185298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc156440 0x7fa5fc0c7120 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec009990 tx=0x7fa5ec009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4834af59d52b5f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:53.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.501+0000 7fa5f959a700 1 -- 172.21.15.27:0/1092185298 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec009e80 con 0x7fa5fc156440 2024-01-11T09:45:53.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.501+0000 7fa5f959a700 1 -- 172.21.15.27:0/1092185298 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ec005c10 con 0x7fa5fc156440 2024-01-11T09:45:53.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.501+0000 7fa5f959a700 1 -- 172.21.15.27:0/1092185298 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec00b650 con 0x7fa5fc156440 2024-01-11T09:45:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.501+0000 7fa600dff700 1 -- 172.21.15.27:0/1092185298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc156440 msgr2=0x7fa5fc0c7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.501+0000 7fa600dff700 1 --2- 172.21.15.27:0/1092185298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc156440 0x7fa5fc0c7120 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec009990 tx=0x7fa5ec009220 comp rx=0 tx=0).stop 2024-01-11T09:45:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.501+0000 7fa600dff700 1 -- 172.21.15.27:0/1092185298 shutdown_connections 2024-01-11T09:45:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.501+0000 7fa600dff700 1 --2- 172.21.15.27:0/1092185298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc156440 0x7fa5fc0c7120 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.501+0000 7fa600dff700 1 --2- 172.21.15.27:0/1092185298 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5fc156810 0x7fa5fc06e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.501+0000 7fa600dff700 1 -- 172.21.15.27:0/1092185298 >> 172.21.15.27:0/1092185298 conn(0x7fa5fc0680e0 msgr2=0x7fa5fc0cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.502+0000 7fa600dff700 1 -- 172.21.15.27:0/1092185298 shutdown_connections 2024-01-11T09:45:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.502+0000 7fa600dff700 1 -- 172.21.15.27:0/1092185298 wait complete. 2024-01-11T09:45:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.503+0000 7fa600dff700 1 Processor -- start 2024-01-11T09:45:53.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.503+0000 7fa600dff700 1 -- start start 2024-01-11T09:45:53.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.504+0000 7fa600dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc0ccac0 0x7fa5fc0c5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:53.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.504+0000 7fa600dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5fc156440 0x7fa5fc0c0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:53.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.504+0000 7fa600dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5fc156810 0x7fa5fc0caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:53.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.504+0000 7fa600dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc0c5490 con 0x7fa5fc0ccac0 2024-01-11T09:45:53.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.504+0000 7fa600dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc0c31b0 con 0x7fa5fc156440 2024-01-11T09:45:53.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.504+0000 7fa600dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc0c3490 con 0x7fa5fc156810 2024-01-11T09:45:53.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.504+0000 7fa5fa59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc0ccac0 0x7fa5fc0c5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:53.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.504+0000 7fa5fa59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc0ccac0 0x7fa5fc0c5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44680/0 (socket says 172.21.15.27:44680) 2024-01-11T09:45:53.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.504+0000 7fa5fa59c700 1 -- 172.21.15.27:0/887971326 learned_addr learned my addr 172.21.15.27:0/887971326 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:53.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.505+0000 7fa5f9d9b700 1 --2- 172.21.15.27:0/887971326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5fc156440 0x7fa5fc0c0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:53.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.505+0000 7fa5fad9d700 1 --2- 172.21.15.27:0/887971326 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5fc156810 0x7fa5fc0caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:53.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.505+0000 7fa5fa59c700 1 -- 172.21.15.27:0/887971326 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5fc156810 msgr2=0x7fa5fc0caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:53.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.505+0000 7fa5fa59c700 1 --2- 172.21.15.27:0/887971326 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5fc156810 0x7fa5fc0caa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:53.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.505+0000 7fa5fa59c700 1 -- 172.21.15.27:0/887971326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5fc156440 msgr2=0x7fa5fc0c0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:53.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.505+0000 7fa5fa59c700 1 --2- 172.21.15.27:0/887971326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5fc156440 0x7fa5fc0c0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:53.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.505+0000 7fa5fa59c700 1 -- 172.21.15.27:0/887971326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec009480 con 0x7fa5fc0ccac0 2024-01-11T09:45:53.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.505+0000 7fa5f9d9b700 1 --2- 172.21.15.27:0/887971326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5fc156440 0x7fa5fc0c0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:45:53.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.505+0000 7fa5fa59c700 1 --2- 172.21.15.27:0/887971326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc0ccac0 0x7fa5fc0c5930 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00b3c0 tx=0x7fa5ec016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:53.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.505+0000 7fa5ea7fc700 1 -- 172.21.15.27:0/887971326 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec009e80 con 0x7fa5fc0ccac0 2024-01-11T09:45:53.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.506+0000 7fa5ea7fc700 1 -- 172.21.15.27:0/887971326 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ec020070 con 0x7fa5fc0ccac0 2024-01-11T09:45:53.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.506+0000 7fa600dff700 1 -- 172.21.15.27:0/887971326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5fc0c3770 con 0x7fa5fc0ccac0 2024-01-11T09:45:53.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.506+0000 7fa5ea7fc700 1 -- 172.21.15.27:0/887971326 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec011d10 con 0x7fa5fc0ccac0 2024-01-11T09:45:53.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.506+0000 7fa600dff700 1 -- 172.21.15.27:0/887971326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5fc0c3d30 con 0x7fa5fc0ccac0 2024-01-11T09:45:53.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.507+0000 7fa600dff700 1 -- 172.21.15.27:0/887971326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5fc04c3c0 con 0x7fa5fc0ccac0 2024-01-11T09:45:53.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.508+0000 7fa5ea7fc700 1 -- 172.21.15.27:0/887971326 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 130) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fa5ec022030 con 0x7fa5fc0ccac0 2024-01-11T09:45:53.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.509+0000 7fa5ea7fc700 1 --2- 172.21.15.27:0/887971326 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa5dc066dd0 0x7fa5dc069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:53.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.510+0000 7fa5ea7fc700 1 -- 172.21.15.27:0/887971326 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7fa5ec086400 con 0x7fa5fc0ccac0 2024-01-11T09:45:53.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.510+0000 7fa5f9d9b700 1 --2- 172.21.15.27:0/887971326 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa5dc066dd0 0x7fa5dc069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:53.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.511+0000 7fa5f9d9b700 1 --2- 172.21.15.27:0/887971326 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa5dc066dd0 0x7fa5dc069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0c7560 tx=0x7fa5f0008040 comp rx=0 tx=0).ready entity=mgr.10501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:53.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.514+0000 7fa5ea7fc700 1 -- 172.21.15.27:0/887971326 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa5ec053450 con 0x7fa5fc0ccac0 2024-01-11T09:45:53.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.744+0000 7fa600dff700 1 -- 172.21.15.27:0/887971326 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5fc0c8220 con 0x7fa5dc066dd0 2024-01-11T09:45:53.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.790+0000 7fa5ea7fc700 1 -- 172.21.15.27:0/887971326 <== mgr.10501 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fa5fc0c8220 con 0x7fa5dc066dd0 2024-01-11T09:45:53.782 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule activated for path mandatory_fs_argument_test_dir 2024-01-11T09:45:53.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.793+0000 7fa600dff700 1 -- 172.21.15.27:0/887971326 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa5dc066dd0 msgr2=0x7fa5dc069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:53.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.793+0000 7fa600dff700 1 --2- 172.21.15.27:0/887971326 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa5dc066dd0 0x7fa5dc069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0c7560 tx=0x7fa5f0008040 comp rx=0 tx=0).stop 2024-01-11T09:45:53.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.793+0000 7fa600dff700 1 -- 172.21.15.27:0/887971326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc0ccac0 msgr2=0x7fa5fc0c5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:53.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.793+0000 7fa600dff700 1 --2- 172.21.15.27:0/887971326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc0ccac0 0x7fa5fc0c5930 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00b3c0 tx=0x7fa5ec016000 comp rx=0 tx=0).stop 2024-01-11T09:45:53.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.793+0000 7fa600dff700 1 -- 172.21.15.27:0/887971326 shutdown_connections 2024-01-11T09:45:53.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.793+0000 7fa600dff700 1 --2- 172.21.15.27:0/887971326 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa5fc156810 0x7fa5fc0caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:53.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.793+0000 7fa600dff700 1 --2- 172.21.15.27:0/887971326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5fc0ccac0 0x7fa5fc0c5930 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:53.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.793+0000 7fa600dff700 1 --2- 172.21.15.27:0/887971326 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa5dc066dd0 0x7fa5dc069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:53.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.793+0000 7fa600dff700 1 --2- 172.21.15.27:0/887971326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5fc156440 0x7fa5fc0c0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:53.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.793+0000 7fa600dff700 1 -- 172.21.15.27:0/887971326 >> 172.21.15.27:0/887971326 conn(0x7fa5fc0680e0 msgr2=0x7fa5fc0c6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:53.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.793+0000 7fa600dff700 1 -- 172.21.15.27:0/887971326 shutdown_connections 2024-01-11T09:45:53.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:53.793+0000 7fa600dff700 1 -- 172.21.15.27:0/887971326 wait complete. 2024-01-11T09:45:53.803 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path mandatory_fs_argument_test_dir 2024-01-11T09:45:53.804 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-01-11T09:45:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.021+0000 7faca6dff700 1 Processor -- start 2024-01-11T09:45:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.022+0000 7faca6dff700 1 -- start start 2024-01-11T09:45:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.023+0000 7faca6dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faca013fbb0 0x7faca00ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.023+0000 7faca6dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faca013ff80 0x7faca015a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.023+0000 7faca6dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faca00c6b30 con 0x7faca00cace0 2024-01-11T09:45:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.023+0000 7faca6dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faca00c6800 con 0x7faca013fbb0 2024-01-11T09:45:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.023+0000 7faca6dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faca00cb240 con 0x7faca013ff80 2024-01-11T09:45:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.023+0000 7faca539c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7faca00cace0 0x7faca006e6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42116/0 (socket says 172.21.15.27:42116) 2024-01-11T09:45:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.023+0000 7faca539c700 1 -- 172.21.15.27:0/3001436454 learned_addr learned my addr 172.21.15.27:0/3001436454 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.023+0000 7fac9ffff700 1 --2- 172.21.15.27:0/3001436454 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faca013ff80 0x7faca015a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.023+0000 7faca4b9b700 1 --2- 172.21.15.27:0/3001436454 >> v2:172.21.15.77:3300/0 conn(0x7faca013fbb0 0x7faca00ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:54.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.024+0000 7fac9f7fe700 1 -- 172.21.15.27:0/3001436454 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583669989 0 0) 0x7faca00c6b30 con 0x7faca00cace0 2024-01-11T09:45:54.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.024+0000 7fac9f7fe700 1 -- 172.21.15.27:0/3001436454 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac90003570 con 0x7faca00cace0 2024-01-11T09:45:54.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.024+0000 7fac9ffff700 1 -- 172.21.15.27:0/3001436454 >> v2:172.21.15.77:3300/0 conn(0x7faca013fbb0 msgr2=0x7faca00ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:54.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.024+0000 7fac9ffff700 1 --2- 172.21.15.27:0/3001436454 >> v2:172.21.15.77:3300/0 conn(0x7faca013fbb0 0x7faca00ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:54.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.024+0000 7fac9ffff700 1 -- 172.21.15.27:0/3001436454 >> v1:172.21.15.27:6789/0 conn(0x7faca00cace0 legacy=0x7faca006e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:54.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.024+0000 7fac9ffff700 1 -- 172.21.15.27:0/3001436454 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca00cb520 con 0x7faca013ff80 2024-01-11T09:45:54.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.024+0000 7faca4b9b700 1 --2- 172.21.15.27:0/3001436454 >> v2:172.21.15.77:3300/0 conn(0x7faca013fbb0 0x7faca00ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:54.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.024+0000 7fac9ffff700 1 --2- 172.21.15.27:0/3001436454 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faca013ff80 0x7faca015a020 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fac8c0098e0 tx=0x7fac8c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c7547679d71ef71 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:54.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.024+0000 7fac9f7fe700 1 -- 172.21.15.27:0/3001436454 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c011630 con 0x7faca013ff80 2024-01-11T09:45:54.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.025+0000 7fac9f7fe700 1 -- 172.21.15.27:0/3001436454 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8c011790 con 0x7faca013ff80 2024-01-11T09:45:54.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.025+0000 7fac9f7fe700 1 -- 172.21.15.27:0/3001436454 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c011b30 con 0x7faca013ff80 2024-01-11T09:45:54.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.025+0000 7faca6dff700 1 -- 172.21.15.27:0/3001436454 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faca013ff80 msgr2=0x7faca015a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:54.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.025+0000 7faca6dff700 1 --2- 172.21.15.27:0/3001436454 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faca013ff80 0x7faca015a020 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fac8c0098e0 tx=0x7fac8c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:54.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.025+0000 7faca6dff700 1 -- 172.21.15.27:0/3001436454 shutdown_connections 2024-01-11T09:45:54.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.025+0000 7faca6dff700 1 --2- 172.21.15.27:0/3001436454 >> v2:172.21.15.77:3300/0 conn(0x7faca013fbb0 0x7faca00ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:54.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.025+0000 7faca6dff700 1 --2- 172.21.15.27:0/3001436454 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faca013ff80 0x7faca015a020 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:54.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.025+0000 7faca6dff700 1 -- 172.21.15.27:0/3001436454 >> 172.21.15.27:0/3001436454 conn(0x7faca00680a0 msgr2=0x7faca00cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:54.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.025+0000 7faca6dff700 1 -- 172.21.15.27:0/3001436454 shutdown_connections 2024-01-11T09:45:54.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.025+0000 7faca6dff700 1 -- 172.21.15.27:0/3001436454 wait complete. 2024-01-11T09:45:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.027+0000 7faca6dff700 1 Processor -- start 2024-01-11T09:45:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.027+0000 7faca6dff700 1 -- start start 2024-01-11T09:45:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.028+0000 7faca6dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faca00cace0 0x7faca013acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.028+0000 7faca6dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faca013fbb0 0x7faca013b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.028+0000 7faca6dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faca013ff80 0x7faca0169390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.028+0000 7faca6dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca013b730 con 0x7faca013fbb0 2024-01-11T09:45:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.028+0000 7faca6dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca00b13c0 con 0x7faca013ff80 2024-01-11T09:45:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.028+0000 7faca6dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faca00b16a0 con 0x7faca00cace0 2024-01-11T09:45:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.028+0000 7fac9ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faca013fbb0 0x7faca013b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.028+0000 7fac9ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faca013fbb0 0x7faca013b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47626/0 (socket says 172.21.15.27:47626) 2024-01-11T09:45:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.028+0000 7fac9ffff700 1 -- 172.21.15.27:0/1252736242 learned_addr learned my addr 172.21.15.27:0/1252736242 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.028+0000 7faca4b9b700 1 --2- 172.21.15.27:0/1252736242 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faca00cace0 0x7faca013acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:54.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.028+0000 7faca539c700 1 --2- 172.21.15.27:0/1252736242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faca013ff80 0x7faca0169390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:54.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.029+0000 7fac9ffff700 1 -- 172.21.15.27:0/1252736242 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faca00cace0 msgr2=0x7faca013acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:54.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.029+0000 7fac9ffff700 1 --2- 172.21.15.27:0/1252736242 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faca00cace0 0x7faca013acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:54.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.029+0000 7fac9ffff700 1 -- 172.21.15.27:0/1252736242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faca013ff80 msgr2=0x7faca0169390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:54.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.029+0000 7fac9ffff700 1 --2- 172.21.15.27:0/1252736242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faca013ff80 0x7faca0169390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:54.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.029+0000 7fac9ffff700 1 -- 172.21.15.27:0/1252736242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac940093c0 con 0x7faca013fbb0 2024-01-11T09:45:54.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.029+0000 7fac9ffff700 1 --2- 172.21.15.27:0/1252736242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faca013fbb0 0x7faca013b1f0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7faca00cb380 tx=0x7fac8c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:54.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.029+0000 7fac9cff9700 1 -- 172.21.15.27:0/1252736242 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c000d10 con 0x7faca013fbb0 2024-01-11T09:45:54.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.029+0000 7fac9cff9700 1 -- 172.21.15.27:0/1252736242 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8c011900 con 0x7faca013fbb0 2024-01-11T09:45:54.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.029+0000 7faca6dff700 1 -- 172.21.15.27:0/1252736242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac8c009420 con 0x7faca013fbb0 2024-01-11T09:45:54.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.029+0000 7fac9cff9700 1 -- 172.21.15.27:0/1252736242 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c026430 con 0x7faca013fbb0 2024-01-11T09:45:54.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.029+0000 7faca6dff700 1 -- 172.21.15.27:0/1252736242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faca00b1c60 con 0x7faca013fbb0 2024-01-11T09:45:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.031+0000 7faca6dff700 1 -- 172.21.15.27:0/1252736242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faca00c36f0 con 0x7faca013fbb0 2024-01-11T09:45:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.031+0000 7fac9cff9700 1 -- 172.21.15.27:0/1252736242 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 130) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fac8c023030 con 0x7faca013fbb0 2024-01-11T09:45:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.032+0000 7fac9cff9700 1 --2- 172.21.15.27:0/1252736242 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fac84066dd0 0x7fac84069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.032+0000 7fac9cff9700 1 -- 172.21.15.27:0/1252736242 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7fac8c085d40 con 0x7faca013fbb0 2024-01-11T09:45:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.033+0000 7faca4b9b700 1 --2- 172.21.15.27:0/1252736242 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fac84066dd0 0x7fac84069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.033+0000 7faca4b9b700 1 --2- 172.21.15.27:0/1252736242 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fac84066dd0 0x7fac84069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7faca013b9d0 tx=0x7fac94009120 comp rx=0 tx=0).ready entity=mgr.10501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.036+0000 7fac9cff9700 1 -- 172.21.15.27:0/1252736242 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fac8c089030 con 0x7faca013fbb0 2024-01-11T09:45:54.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.253+0000 7fac9cff9700 1 -- 172.21.15.27:0/1252736242 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 131) v1 ==== 84964+0+0 (secure 0 0 0) 0x7fac8c0531e0 con 0x7faca013fbb0 2024-01-11T09:45:54.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.272+0000 7faca6dff700 1 -- 172.21.15.27:0/1252736242 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7faca00cb800 con 0x7fac84066dd0 2024-01-11T09:45:54.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.329+0000 7fac9cff9700 1 -- 172.21.15.27:0/1252736242 <== mgr.10501 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7faca00cb800 con 0x7fac84066dd0 2024-01-11T09:45:54.321 INFO:teuthology.orchestra.run.smithi027.stdout:Retention added to path mandatory_fs_argument_test_dir 2024-01-11T09:45:54.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.331+0000 7faca6dff700 1 -- 172.21.15.27:0/1252736242 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fac84066dd0 msgr2=0x7fac84069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:54.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.331+0000 7faca6dff700 1 --2- 172.21.15.27:0/1252736242 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fac84066dd0 0x7fac84069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7faca013b9d0 tx=0x7fac94009120 comp rx=0 tx=0).stop 2024-01-11T09:45:54.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.332+0000 7faca6dff700 1 -- 172.21.15.27:0/1252736242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faca013fbb0 msgr2=0x7faca013b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:54.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.332+0000 7faca6dff700 1 --2- 172.21.15.27:0/1252736242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faca013fbb0 0x7faca013b1f0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7faca00cb380 tx=0x7fac8c017000 comp rx=0 tx=0).stop 2024-01-11T09:45:54.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.332+0000 7faca6dff700 1 -- 172.21.15.27:0/1252736242 shutdown_connections 2024-01-11T09:45:54.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.332+0000 7faca6dff700 1 --2- 172.21.15.27:0/1252736242 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fac84066dd0 0x7fac84069280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:54.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.332+0000 7faca6dff700 1 --2- 172.21.15.27:0/1252736242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faca013ff80 0x7faca0169390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:54.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.332+0000 7faca6dff700 1 --2- 172.21.15.27:0/1252736242 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faca00cace0 0x7faca013acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:54.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.332+0000 7faca6dff700 1 --2- 172.21.15.27:0/1252736242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faca013fbb0 0x7faca013b1f0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:54.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.332+0000 7faca6dff700 1 -- 172.21.15.27:0/1252736242 >> 172.21.15.27:0/1252736242 conn(0x7faca00680a0 msgr2=0x7faca00c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:54.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.332+0000 7faca6dff700 1 -- 172.21.15.27:0/1252736242 shutdown_connections 2024-01-11T09:45:54.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.332+0000 7faca6dff700 1 -- 172.21.15.27:0/1252736242 wait complete. 2024-01-11T09:45:54.341 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path mandatory_fs_argument_test_dir 2024-01-11T09:45:54.341 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list mandatory_fs_argument_test_dir --fs backup_fs --format json 2024-01-11T09:45:54.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.553+0000 7f01d41ff700 1 Processor -- start 2024-01-11T09:45:54.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.553+0000 7f01d41ff700 1 -- start start 2024-01-11T09:45:54.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.554+0000 7f01d41ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f01cc068be0 0x7f01cc0ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:54.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.554+0000 7f01d41ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01cc0b0d20 con 0x7f01cc0b08a0 2024-01-11T09:45:54.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.554+0000 7f01d41ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01cc15a440 con 0x7f01cc159250 2024-01-11T09:45:54.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.554+0000 7f01d41ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01cc15ace0 con 0x7f01cc068be0 2024-01-11T09:45:54.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.555+0000 7f01d179a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f01cc0b08a0 0x7f01cc0684d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42126/0 (socket says 172.21.15.27:42126) 2024-01-11T09:45:54.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.555+0000 7f01d179a700 1 -- 172.21.15.27:0/2065264727 learned_addr learned my addr 172.21.15.27:0/2065264727 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:54.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.555+0000 7f01d279c700 1 --2- 172.21.15.27:0/2065264727 >> v2:172.21.15.27:3300/0 conn(0x7f01cc068be0 0x7f01cc0ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:54.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.555+0000 7f01d0f99700 1 -- 172.21.15.27:0/2065264727 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2613362725 0 0) 0x7f01cc0b0d20 con 0x7f01cc0b08a0 2024-01-11T09:45:54.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.555+0000 7f01d279c700 1 -- 172.21.15.27:0/2065264727 >> v1:172.21.15.77:6789/0 conn(0x7f01cc159250 legacy=0x7f01cc0cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:54.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.555+0000 7f01d279c700 1 -- 172.21.15.27:0/2065264727 >> v1:172.21.15.27:6789/0 conn(0x7f01cc0b08a0 legacy=0x7f01cc0684d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:54.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.555+0000 7f01d279c700 1 -- 172.21.15.27:0/2065264727 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01cc15a720 con 0x7f01cc068be0 2024-01-11T09:45:54.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.555+0000 7f01d279c700 1 --2- 172.21.15.27:0/2065264727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01cc068be0 0x7f01cc0ce2d0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f01c4009de0 tx=0x7f01c4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81d75f2a3df650de server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.556+0000 7f01d0f99700 1 -- 172.21.15.27:0/2065264727 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c400e070 con 0x7f01cc068be0 2024-01-11T09:45:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.556+0000 7f01d0f99700 1 -- 172.21.15.27:0/2065264727 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c400bbb0 con 0x7f01cc068be0 2024-01-11T09:45:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.556+0000 7f01d0f99700 1 -- 172.21.15.27:0/2065264727 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c4012430 con 0x7f01cc068be0 2024-01-11T09:45:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.556+0000 7f01d41ff700 1 -- 172.21.15.27:0/2065264727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01cc068be0 msgr2=0x7f01cc0ce2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.556+0000 7f01d41ff700 1 --2- 172.21.15.27:0/2065264727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01cc068be0 0x7f01cc0ce2d0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f01c4009de0 tx=0x7f01c4009780 comp rx=0 tx=0).stop 2024-01-11T09:45:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.556+0000 7f01d41ff700 1 -- 172.21.15.27:0/2065264727 shutdown_connections 2024-01-11T09:45:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.556+0000 7f01d41ff700 1 --2- 172.21.15.27:0/2065264727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01cc068be0 0x7f01cc0ce2d0 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.556+0000 7f01d41ff700 1 -- 172.21.15.27:0/2065264727 >> 172.21.15.27:0/2065264727 conn(0x7f01cc067610 msgr2=0x7f01cc067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.556+0000 7f01d41ff700 1 -- 172.21.15.27:0/2065264727 shutdown_connections 2024-01-11T09:45:54.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.556+0000 7f01d41ff700 1 -- 172.21.15.27:0/2065264727 wait complete. 2024-01-11T09:45:54.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.558+0000 7f01d41ff700 1 Processor -- start 2024-01-11T09:45:54.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.558+0000 7f01d41ff700 1 -- start start 2024-01-11T09:45:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.559+0000 7f01d41ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01cc068be0 0x7f01cc0c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.559+0000 7f01d41ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01cc0b08a0 0x7f01cc143ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.559+0000 7f01d41ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01cc159250 0x7f01cc0c2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.559+0000 7f01d41ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc0c7770 con 0x7f01cc068be0 2024-01-11T09:45:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.559+0000 7f01d41ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc0c7a50 con 0x7f01cc0b08a0 2024-01-11T09:45:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.559+0000 7f01d41ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc0c7d30 con 0x7f01cc159250 2024-01-11T09:45:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.559+0000 7f01d1f9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01cc068be0 0x7f01cc0c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.559+0000 7f01d1f9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01cc068be0 0x7f01cc0c8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47638/0 (socket says 172.21.15.27:47638) 2024-01-11T09:45:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.559+0000 7f01d1f9b700 1 -- 172.21.15.27:0/4094862883 learned_addr learned my addr 172.21.15.27:0/4094862883 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.559+0000 7f01d179a700 1 --2- 172.21.15.27:0/4094862883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01cc0b08a0 0x7f01cc143ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.559+0000 7f01d279c700 1 --2- 172.21.15.27:0/4094862883 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01cc159250 0x7f01cc0c2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.560+0000 7f01d1f9b700 1 -- 172.21.15.27:0/4094862883 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01cc159250 msgr2=0x7f01cc0c2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.560+0000 7f01d1f9b700 1 --2- 172.21.15.27:0/4094862883 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01cc159250 0x7f01cc0c2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.560+0000 7f01d1f9b700 1 -- 172.21.15.27:0/4094862883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01cc0b08a0 msgr2=0x7f01cc143ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:54.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.560+0000 7f01d1f9b700 1 --2- 172.21.15.27:0/4094862883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01cc0b08a0 0x7f01cc143ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:54.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.560+0000 7f01d1f9b700 1 -- 172.21.15.27:0/4094862883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c4009a20 con 0x7f01cc068be0 2024-01-11T09:45:54.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.560+0000 7f01d1f9b700 1 --2- 172.21.15.27:0/4094862883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01cc068be0 0x7f01cc0c8810 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f01c800c810 tx=0x7f01c8007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:54.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.560+0000 7f01c1ffb700 1 -- 172.21.15.27:0/4094862883 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c8011590 con 0x7f01cc068be0 2024-01-11T09:45:54.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.560+0000 7f01c1ffb700 1 -- 172.21.15.27:0/4094862883 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c80116f0 con 0x7f01cc068be0 2024-01-11T09:45:54.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.560+0000 7f01d41ff700 1 -- 172.21.15.27:0/4094862883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01cc0c8010 con 0x7f01cc068be0 2024-01-11T09:45:54.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.560+0000 7f01c1ffb700 1 -- 172.21.15.27:0/4094862883 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c8011ab0 con 0x7f01cc068be0 2024-01-11T09:45:54.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.560+0000 7f01d41ff700 1 -- 172.21.15.27:0/4094862883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01cc0c7460 con 0x7f01cc068be0 2024-01-11T09:45:54.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.561+0000 7f01d41ff700 1 -- 172.21.15.27:0/4094862883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01cc159090 con 0x7f01cc068be0 2024-01-11T09:45:54.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.562+0000 7f01c1ffb700 1 -- 172.21.15.27:0/4094862883 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 131) v1 ==== 84964+0+0 (secure 0 0 0) 0x7f01c801b3b0 con 0x7f01cc068be0 2024-01-11T09:45:54.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.563+0000 7f01c1ffb700 1 --2- 172.21.15.27:0/4094862883 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f01b00671c0 0x7f01b0069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:54.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.563+0000 7f01c1ffb700 1 -- 172.21.15.27:0/4094862883 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f01c8084360 con 0x7f01cc068be0 2024-01-11T09:45:54.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.563+0000 7f01d179a700 1 --2- 172.21.15.27:0/4094862883 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f01b00671c0 0x7f01b0069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:54.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.564+0000 7f01d179a700 1 --2- 172.21.15.27:0/4094862883 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f01b00671c0 0x7f01b0069670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f01cc0c7060 tx=0x7f01bc007f30 comp rx=0 tx=0).ready entity=mgr.10501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:54.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.567+0000 7f01c1ffb700 1 -- 172.21.15.27:0/4094862883 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f01c8050e70 con 0x7f01cc068be0 2024-01-11T09:45:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.796+0000 7f01d41ff700 1 -- 172.21.15.27:0/4094862883 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f01cc04c3c0 con 0x7f01b00671c0 2024-01-11T09:45:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.803+0000 7f01c1ffb700 1 -- 172.21.15.27:0/4094862883 <== mgr.10501 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+88 (secure 0 0 0) 0x7f01cc04c3c0 con 0x7f01b00671c0 2024-01-11T09:45:54.795 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:54.795 INFO:teuthology.orchestra.run.smithi027.stdout:{"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2024-01-11T09:45:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.805+0000 7f01d41ff700 1 -- 172.21.15.27:0/4094862883 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f01b00671c0 msgr2=0x7f01b0069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.805+0000 7f01d41ff700 1 --2- 172.21.15.27:0/4094862883 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f01b00671c0 0x7f01b0069670 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f01cc0c7060 tx=0x7f01bc007f30 comp rx=0 tx=0).stop 2024-01-11T09:45:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.805+0000 7f01d41ff700 1 -- 172.21.15.27:0/4094862883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01cc068be0 msgr2=0x7f01cc0c8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.805+0000 7f01d41ff700 1 --2- 172.21.15.27:0/4094862883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01cc068be0 0x7f01cc0c8810 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f01c800c810 tx=0x7f01c8007ee0 comp rx=0 tx=0).stop 2024-01-11T09:45:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.805+0000 7f01d41ff700 1 -- 172.21.15.27:0/4094862883 shutdown_connections 2024-01-11T09:45:54.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.805+0000 7f01d41ff700 1 --2- 172.21.15.27:0/4094862883 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01cc159250 0x7f01cc0c2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:54.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.805+0000 7f01d41ff700 1 --2- 172.21.15.27:0/4094862883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01cc068be0 0x7f01cc0c8810 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:54.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.805+0000 7f01d41ff700 1 --2- 172.21.15.27:0/4094862883 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f01b00671c0 0x7f01b0069670 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:54.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.805+0000 7f01d41ff700 1 --2- 172.21.15.27:0/4094862883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01cc0b08a0 0x7f01cc143ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:54.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.806+0000 7f01d41ff700 1 -- 172.21.15.27:0/4094862883 >> 172.21.15.27:0/4094862883 conn(0x7f01cc067610 msgr2=0x7f01cc1588a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:54.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.806+0000 7f01d41ff700 1 -- 172.21.15.27:0/4094862883 shutdown_connections 2024-01-11T09:45:54.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:54.806+0000 7f01d41ff700 1 -- 172.21.15.27:0/4094862883 wait complete. 2024-01-11T09:45:54.815 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2024-01-11T09:45:54.815 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status mandatory_fs_argument_test_dir --fs backup_fs --format json 2024-01-11T09:45:55.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.028+0000 7f21a3dff700 1 Processor -- start 2024-01-11T09:45:55.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.028+0000 7f21a3dff700 1 -- start start 2024-01-11T09:45:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.029+0000 7f21a3dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f219c159250 0x7f219c0b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.029+0000 7f21a3dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f219c0cb410 0x7f219c0cb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.029+0000 7f21a3dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c15a440 con 0x7f219c0684d0 2024-01-11T09:45:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.030+0000 7f21a3dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f219c15ace0 con 0x7f219c159250 2024-01-11T09:45:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.030+0000 7f21a3dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f219c0b0de0 con 0x7f219c0cb410 2024-01-11T09:45:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.030+0000 7f21a139a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f219c0cb410 0x7f219c0cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.030+0000 7f21a1b9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f219c159250 0x7f219c0b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.030+0000 7f21a239c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f219c0684d0 0x7f219c0ce2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:59976/0 (socket says 172.21.15.27:59976) 2024-01-11T09:45:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.030+0000 7f21a239c700 1 -- 172.21.15.27:0/580076185 learned_addr learned my addr 172.21.15.27:0/580076185 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.031+0000 7f21a0b99700 1 -- 172.21.15.27:0/580076185 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072420285 0 0) 0x7f219c15a440 con 0x7f219c0684d0 2024-01-11T09:45:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.031+0000 7f21a0b99700 1 -- 172.21.15.27:0/580076185 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2188003570 con 0x7f219c0684d0 2024-01-11T09:45:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.031+0000 7f21a139a700 1 -- 172.21.15.27:0/580076185 >> v2:172.21.15.77:3300/0 conn(0x7f219c159250 msgr2=0x7f219c0b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.031+0000 7f21a139a700 1 --2- 172.21.15.27:0/580076185 >> v2:172.21.15.77:3300/0 conn(0x7f219c159250 0x7f219c0b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.031+0000 7f21a139a700 1 -- 172.21.15.27:0/580076185 >> v1:172.21.15.77:6789/0 conn(0x7f219c0684d0 legacy=0x7f219c0ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.031+0000 7f21a139a700 1 -- 172.21.15.27:0/580076185 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219c15a720 con 0x7f219c0cb410 2024-01-11T09:45:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.031+0000 7f21a1b9b700 1 --2- 172.21.15.27:0/580076185 >> v2:172.21.15.77:3300/0 conn(0x7f219c159250 0x7f219c0b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.031+0000 7f21a139a700 1 --2- 172.21.15.27:0/580076185 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f219c0cb410 0x7f219c0cb860 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f218c0098e0 tx=0x7f218c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae2af17dd2000844 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.031+0000 7f21a0b99700 1 -- 172.21.15.27:0/580076185 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218c011630 con 0x7f219c0cb410 2024-01-11T09:45:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.031+0000 7f21a0b99700 1 -- 172.21.15.27:0/580076185 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f218c011790 con 0x7f219c0cb410 2024-01-11T09:45:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.031+0000 7f21a0b99700 1 -- 172.21.15.27:0/580076185 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218c011b10 con 0x7f219c0cb410 2024-01-11T09:45:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.032+0000 7f21a3dff700 1 -- 172.21.15.27:0/580076185 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f219c0cb410 msgr2=0x7f219c0cb860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.032+0000 7f21a3dff700 1 --2- 172.21.15.27:0/580076185 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f219c0cb410 0x7f219c0cb860 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f218c0098e0 tx=0x7f218c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.032+0000 7f21a3dff700 1 -- 172.21.15.27:0/580076185 shutdown_connections 2024-01-11T09:45:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.032+0000 7f21a3dff700 1 --2- 172.21.15.27:0/580076185 >> v2:172.21.15.77:3300/0 conn(0x7f219c159250 0x7f219c0b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.032+0000 7f21a3dff700 1 --2- 172.21.15.27:0/580076185 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f219c0cb410 0x7f219c0cb860 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.032+0000 7f21a3dff700 1 -- 172.21.15.27:0/580076185 >> 172.21.15.27:0/580076185 conn(0x7f219c067610 msgr2=0x7f219c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.032+0000 7f21a3dff700 1 -- 172.21.15.27:0/580076185 shutdown_connections 2024-01-11T09:45:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.032+0000 7f21a3dff700 1 -- 172.21.15.27:0/580076185 wait complete. 2024-01-11T09:45:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.033+0000 7f21a3dff700 1 Processor -- start 2024-01-11T09:45:55.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.034+0000 7f21a3dff700 1 -- start start 2024-01-11T09:45:55.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.034+0000 7f21a3dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f219c0684d0 0x7f219c156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:55.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.034+0000 7f21a3dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f219c0cb410 0x7f219c0c3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:55.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.034+0000 7f21a3dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f219c159250 0x7f219c0c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:55.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.034+0000 7f21a3dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f219c0c2260 con 0x7f219c0684d0 2024-01-11T09:45:55.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.034+0000 7f21a3dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f219c0c46e0 con 0x7f219c0cb410 2024-01-11T09:45:55.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.034+0000 7f21a3dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f219c0c1c10 con 0x7f219c159250 2024-01-11T09:45:55.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.035+0000 7f21a1b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f219c0684d0 0x7f219c156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:55.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.035+0000 7f21a1b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f219c0684d0 0x7f219c156850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47652/0 (socket says 172.21.15.27:47652) 2024-01-11T09:45:55.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.035+0000 7f21a1b9b700 1 -- 172.21.15.27:0/1807583197 learned_addr learned my addr 172.21.15.27:0/1807583197 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:55.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.035+0000 7f21a239c700 1 --2- 172.21.15.27:0/1807583197 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f219c159250 0x7f219c0c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:55.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.035+0000 7f21a139a700 1 --2- 172.21.15.27:0/1807583197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f219c0cb410 0x7f219c0c3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:55.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.035+0000 7f21a1b9b700 1 -- 172.21.15.27:0/1807583197 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f219c159250 msgr2=0x7f219c0c67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:55.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.035+0000 7f21a1b9b700 1 --2- 172.21.15.27:0/1807583197 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f219c159250 0x7f219c0c67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.035+0000 7f21a1b9b700 1 -- 172.21.15.27:0/1807583197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f219c0cb410 msgr2=0x7f219c0c3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:55.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.035+0000 7f21a1b9b700 1 --2- 172.21.15.27:0/1807583197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f219c0cb410 0x7f219c0c3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.035+0000 7f21a1b9b700 1 -- 172.21.15.27:0/1807583197 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2198009420 con 0x7f219c0684d0 2024-01-11T09:45:55.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.035+0000 7f21a1b9b700 1 --2- 172.21.15.27:0/1807583197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f219c0684d0 0x7f219c156850 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f219800ec80 tx=0x7f21980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:55.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.036+0000 7f2191ffb700 1 -- 172.21.15.27:0/1807583197 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219800c9d0 con 0x7f219c0684d0 2024-01-11T09:45:55.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.036+0000 7f21a3dff700 1 -- 172.21.15.27:0/1807583197 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f218c009420 con 0x7f219c0684d0 2024-01-11T09:45:55.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.036+0000 7f2191ffb700 1 -- 172.21.15.27:0/1807583197 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219800cb30 con 0x7f219c0684d0 2024-01-11T09:45:55.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.036+0000 7f2191ffb700 1 -- 172.21.15.27:0/1807583197 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219800ceb0 con 0x7f219c0684d0 2024-01-11T09:45:55.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.036+0000 7f21a3dff700 1 -- 172.21.15.27:0/1807583197 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f219c156d90 con 0x7f219c0684d0 2024-01-11T09:45:55.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.037+0000 7f21a3dff700 1 -- 172.21.15.27:0/1807583197 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f219c0c2540 con 0x7f219c0684d0 2024-01-11T09:45:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.038+0000 7f2191ffb700 1 -- 172.21.15.27:0/1807583197 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 131) v1 ==== 84964+0+0 (secure 0 0 0) 0x7f2198013680 con 0x7f219c0684d0 2024-01-11T09:45:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.039+0000 7f2191ffb700 1 --2- 172.21.15.27:0/1807583197 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f21800671c0 0x7f2180069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.039+0000 7f2191ffb700 1 -- 172.21.15.27:0/1807583197 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f2198085240 con 0x7f219c0684d0 2024-01-11T09:45:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.039+0000 7f21a139a700 1 --2- 172.21.15.27:0/1807583197 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f21800671c0 0x7f2180069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.040+0000 7f21a139a700 1 --2- 172.21.15.27:0/1807583197 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f21800671c0 0x7f2180069670 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f218c0091e0 tx=0x7f218c017000 comp rx=0 tx=0).ready entity=mgr.10501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:55.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.043+0000 7f2191ffb700 1 -- 172.21.15.27:0/1807583197 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2198051d50 con 0x7f219c0684d0 2024-01-11T09:45:55.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.273+0000 7f21a3dff700 1 -- 172.21.15.27:0/1807583197 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f219c0c23a0 con 0x7f21800671c0 2024-01-11T09:45:55.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.280+0000 7f2191ffb700 1 -- 172.21.15.27:0/1807583197 <== mgr.10501 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+338 (secure 0 0 0) 0x7f218c000cb0 con 0x7f21800671c0 2024-01-11T09:45:55.272 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:55.272 INFO:teuthology.orchestra.run.smithi027.stdout:[{"fs": "backup_fs", "subvol": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T09:45:53", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-01-11T09:45:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.283+0000 7f21a3dff700 1 -- 172.21.15.27:0/1807583197 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f21800671c0 msgr2=0x7f2180069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.283+0000 7f21a3dff700 1 --2- 172.21.15.27:0/1807583197 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f21800671c0 0x7f2180069670 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f218c0091e0 tx=0x7f218c017000 comp rx=0 tx=0).stop 2024-01-11T09:45:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.283+0000 7f21a3dff700 1 -- 172.21.15.27:0/1807583197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f219c0684d0 msgr2=0x7f219c156850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.283+0000 7f21a3dff700 1 --2- 172.21.15.27:0/1807583197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f219c0684d0 0x7f219c156850 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f219800ec80 tx=0x7f21980091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.283+0000 7f21a3dff700 1 -- 172.21.15.27:0/1807583197 shutdown_connections 2024-01-11T09:45:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.283+0000 7f21a3dff700 1 --2- 172.21.15.27:0/1807583197 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f219c159250 0x7f219c0c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.283+0000 7f21a3dff700 1 --2- 172.21.15.27:0/1807583197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f219c0684d0 0x7f219c156850 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.283+0000 7f21a3dff700 1 --2- 172.21.15.27:0/1807583197 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f21800671c0 0x7f2180069670 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.283+0000 7f21a3dff700 1 --2- 172.21.15.27:0/1807583197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f219c0cb410 0x7f219c0c3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.283+0000 7f21a3dff700 1 -- 172.21.15.27:0/1807583197 >> 172.21.15.27:0/1807583197 conn(0x7f219c067610 msgr2=0x7f219c0afb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:55.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.283+0000 7f21a3dff700 1 -- 172.21.15.27:0/1807583197 shutdown_connections 2024-01-11T09:45:55.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.283+0000 7f21a3dff700 1 -- 172.21.15.27:0/1807583197 wait complete. 2024-01-11T09:45:55.292 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "backup_fs", "subvol": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2024-01-11T00:00:00", "created": "2024-01-11T09:45:53", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-01-11T09:45:55.293 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-01-11T09:45:55.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.507+0000 7f7ee87ff700 1 Processor -- start 2024-01-11T09:45:55.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.507+0000 7f7ee87ff700 1 -- start start 2024-01-11T09:45:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.509+0000 7f7ee87ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7ee00ccd70 0x7f7ee00cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.509+0000 7f7ee87ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ee015ae70 0x7f7ee015b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.509+0000 7f7ee87ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee00cf850 con 0x7f7ee015a1b0 2024-01-11T09:45:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.509+0000 7f7ee87ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ee015ac90 con 0x7f7ee00ccd70 2024-01-11T09:45:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.509+0000 7f7ee87ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee00cec40 con 0x7f7ee015ae70 2024-01-11T09:45:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.509+0000 7f7ee5d9a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7ee00ccd70 0x7f7ee00cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.509+0000 7f7ee6d9c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ee015ae70 0x7f7ee015b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.509+0000 7f7ee659b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7ee015a1b0 0x7f7ee015a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:59992/0 (socket says 172.21.15.27:59992) 2024-01-11T09:45:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.509+0000 7f7ee659b700 1 -- 172.21.15.27:0/840053335 learned_addr learned my addr 172.21.15.27:0/840053335 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:55.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.510+0000 7f7ee6d9c700 1 -- 172.21.15.27:0/840053335 >> v2:172.21.15.77:3300/0 conn(0x7f7ee00ccd70 msgr2=0x7f7ee00cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:55.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.510+0000 7f7ee6d9c700 1 --2- 172.21.15.27:0/840053335 >> v2:172.21.15.77:3300/0 conn(0x7f7ee00ccd70 0x7f7ee00cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.510+0000 7f7ee6d9c700 1 -- 172.21.15.27:0/840053335 >> v1:172.21.15.77:6789/0 conn(0x7f7ee015a1b0 legacy=0x7f7ee015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:55.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.510+0000 7f7ee6d9c700 1 -- 172.21.15.27:0/840053335 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee00cef20 con 0x7f7ee015ae70 2024-01-11T09:45:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.510+0000 7f7ee5d9a700 1 --2- 172.21.15.27:0/840053335 >> v2:172.21.15.77:3300/0 conn(0x7f7ee00ccd70 0x7f7ee00cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.510+0000 7f7ee6d9c700 1 --2- 172.21.15.27:0/840053335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ee015ae70 0x7f7ee015b2f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009e60 tx=0x7f7ed8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fed97a62f492091 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.510+0000 7f7ee5599700 1 -- 172.21.15.27:0/840053335 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed800e070 con 0x7f7ee015ae70 2024-01-11T09:45:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.511+0000 7f7ee5599700 1 -- 172.21.15.27:0/840053335 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed800bb60 con 0x7f7ee015ae70 2024-01-11T09:45:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.511+0000 7f7ee5599700 1 -- 172.21.15.27:0/840053335 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8012430 con 0x7f7ee015ae70 2024-01-11T09:45:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.511+0000 7f7ee87ff700 1 -- 172.21.15.27:0/840053335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ee015ae70 msgr2=0x7f7ee015b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.511+0000 7f7ee87ff700 1 --2- 172.21.15.27:0/840053335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ee015ae70 0x7f7ee015b2f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009e60 tx=0x7f7ed8009780 comp rx=0 tx=0).stop 2024-01-11T09:45:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.511+0000 7f7ee87ff700 1 -- 172.21.15.27:0/840053335 shutdown_connections 2024-01-11T09:45:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.511+0000 7f7ee87ff700 1 --2- 172.21.15.27:0/840053335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ee015ae70 0x7f7ee015b2f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.511+0000 7f7ee87ff700 1 --2- 172.21.15.27:0/840053335 >> v2:172.21.15.77:3300/0 conn(0x7f7ee00ccd70 0x7f7ee00cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.511+0000 7f7ee87ff700 1 -- 172.21.15.27:0/840053335 >> 172.21.15.27:0/840053335 conn(0x7f7ee006ed40 msgr2=0x7f7ee006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:55.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.511+0000 7f7ee87ff700 1 -- 172.21.15.27:0/840053335 shutdown_connections 2024-01-11T09:45:55.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.511+0000 7f7ee87ff700 1 -- 172.21.15.27:0/840053335 wait complete. 2024-01-11T09:45:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.513+0000 7f7ee87ff700 1 Processor -- start 2024-01-11T09:45:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.513+0000 7f7ee87ff700 1 -- start start 2024-01-11T09:45:55.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.514+0000 7f7ee87ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ee00ccd70 0x7f7ee014b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:55.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.514+0000 7f7ee87ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee015a1b0 0x7f7ee00ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:55.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.514+0000 7f7ee87ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ee015ae70 0x7f7ee00b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:55.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.514+0000 7f7ee87ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee00c9a00 con 0x7f7ee015a1b0 2024-01-11T09:45:55.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.514+0000 7f7ee87ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee00a7b20 con 0x7f7ee00ccd70 2024-01-11T09:45:55.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.514+0000 7f7ee87ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee00a6f10 con 0x7f7ee015ae70 2024-01-11T09:45:55.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.514+0000 7f7ee5d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee015a1b0 0x7f7ee00ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:55.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.514+0000 7f7ee5d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee015a1b0 0x7f7ee00ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47668/0 (socket says 172.21.15.27:47668) 2024-01-11T09:45:55.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.514+0000 7f7ee5d9a700 1 -- 172.21.15.27:0/2661069729 learned_addr learned my addr 172.21.15.27:0/2661069729 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:55.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.514+0000 7f7ee6d9c700 1 --2- 172.21.15.27:0/2661069729 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ee015ae70 0x7f7ee00b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.514+0000 7f7ee659b700 1 --2- 172.21.15.27:0/2661069729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ee00ccd70 0x7f7ee014b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.515+0000 7f7ee5d9a700 1 -- 172.21.15.27:0/2661069729 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ee015ae70 msgr2=0x7f7ee00b54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.515+0000 7f7ee5d9a700 1 --2- 172.21.15.27:0/2661069729 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ee015ae70 0x7f7ee00b54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.515+0000 7f7ee5d9a700 1 -- 172.21.15.27:0/2661069729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ee00ccd70 msgr2=0x7f7ee014b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.515+0000 7f7ee5d9a700 1 --2- 172.21.15.27:0/2661069729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ee00ccd70 0x7f7ee014b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.515+0000 7f7ee5d9a700 1 -- 172.21.15.27:0/2661069729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed0009420 con 0x7f7ee015a1b0 2024-01-11T09:45:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.515+0000 7f7ee5d9a700 1 --2- 172.21.15.27:0/2661069729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee015a1b0 0x7f7ee00ca1e0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f7ed000ec80 tx=0x7f7ed00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.515+0000 7f7ed67fc700 1 -- 172.21.15.27:0/2661069729 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed000c850 con 0x7f7ee015a1b0 2024-01-11T09:45:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.515+0000 7f7ed67fc700 1 -- 172.21.15.27:0/2661069729 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed000c9b0 con 0x7f7ee015a1b0 2024-01-11T09:45:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.515+0000 7f7ee87ff700 1 -- 172.21.15.27:0/2661069729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed8009a20 con 0x7f7ee015a1b0 2024-01-11T09:45:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.515+0000 7f7ed67fc700 1 -- 172.21.15.27:0/2661069729 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed000cd30 con 0x7f7ee015a1b0 2024-01-11T09:45:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.516+0000 7f7ee87ff700 1 -- 172.21.15.27:0/2661069729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee00a7550 con 0x7f7ee015a1b0 2024-01-11T09:45:55.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.517+0000 7f7ee87ff700 1 -- 172.21.15.27:0/2661069729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee00a78d0 con 0x7f7ee015a1b0 2024-01-11T09:45:55.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.518+0000 7f7ed67fc700 1 -- 172.21.15.27:0/2661069729 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 131) v1 ==== 84964+0+0 (secure 0 0 0) 0x7f7ed0013680 con 0x7f7ee015a1b0 2024-01-11T09:45:55.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.518+0000 7f7ed67fc700 1 --2- 172.21.15.27:0/2661069729 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7ec40671c0 0x7f7ec4069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:55.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.519+0000 7f7ed67fc700 1 -- 172.21.15.27:0/2661069729 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f7ed00850a0 con 0x7f7ee015a1b0 2024-01-11T09:45:55.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.519+0000 7f7ee659b700 1 --2- 172.21.15.27:0/2661069729 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7ec40671c0 0x7f7ec4069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:55.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.520+0000 7f7ee659b700 1 --2- 172.21.15.27:0/2661069729 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7ec40671c0 0x7f7ec4069670 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7edc007130 tx=0x7f7edc007eb0 comp rx=0 tx=0).ready entity=mgr.10501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:55.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.523+0000 7f7ed67fc700 1 -- 172.21.15.27:0/2661069729 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7ed0055640 con 0x7f7ee015a1b0 2024-01-11T09:45:55.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.755+0000 7f7ee87ff700 1 -- 172.21.15.27:0/2661069729 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ee004c3c0 con 0x7f7ec40671c0 2024-01-11T09:45:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.811+0000 7f7ed67fc700 1 -- 172.21.15.27:0/2661069729 <== mgr.10501 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f7ee004c3c0 con 0x7f7ec40671c0 2024-01-11T09:45:55.803 INFO:teuthology.orchestra.run.smithi027.stdout:Retention removed from path mandatory_fs_argument_test_dir 2024-01-11T09:45:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.813+0000 7f7ee87ff700 1 -- 172.21.15.27:0/2661069729 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7ec40671c0 msgr2=0x7f7ec4069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.813+0000 7f7ee87ff700 1 --2- 172.21.15.27:0/2661069729 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7ec40671c0 0x7f7ec4069670 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7edc007130 tx=0x7f7edc007eb0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.813+0000 7f7ee87ff700 1 -- 172.21.15.27:0/2661069729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee015a1b0 msgr2=0x7f7ee00ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.813+0000 7f7ee87ff700 1 --2- 172.21.15.27:0/2661069729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee015a1b0 0x7f7ee00ca1e0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f7ed000ec80 tx=0x7f7ed00091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.814+0000 7f7ee87ff700 1 -- 172.21.15.27:0/2661069729 shutdown_connections 2024-01-11T09:45:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.814+0000 7f7ee87ff700 1 --2- 172.21.15.27:0/2661069729 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7ee015ae70 0x7f7ee00b54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.814+0000 7f7ee87ff700 1 --2- 172.21.15.27:0/2661069729 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7ec40671c0 0x7f7ec4069670 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.814+0000 7f7ee87ff700 1 --2- 172.21.15.27:0/2661069729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ee00ccd70 0x7f7ee014b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.814+0000 7f7ee87ff700 1 --2- 172.21.15.27:0/2661069729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ee015a1b0 0x7f7ee00ca1e0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.814+0000 7f7ee87ff700 1 -- 172.21.15.27:0/2661069729 >> 172.21.15.27:0/2661069729 conn(0x7f7ee006ed40 msgr2=0x7f7ee006f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.814+0000 7f7ee87ff700 1 -- 172.21.15.27:0/2661069729 shutdown_connections 2024-01-11T09:45:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:55.814+0000 7f7ee87ff700 1 -- 172.21.15.27:0/2661069729 wait complete. 2024-01-11T09:45:55.823 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention removed from path mandatory_fs_argument_test_dir 2024-01-11T09:45:55.823 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-01-11T09:45:56.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.037+0000 7f01789ff700 1 Processor -- start 2024-01-11T09:45:56.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.037+0000 7f01789ff700 1 -- start start 2024-01-11T09:45:56.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.039+0000 7f01789ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f017415a7a0 0x7f017415c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:56.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.039+0000 7f01789ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017415c4b0 con 0x7f0174159e60 2024-01-11T09:45:56.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.039+0000 7f01789ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017415b5d0 con 0x7f01740685e0 2024-01-11T09:45:56.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.039+0000 7f01789ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017415b8a0 con 0x7f017415a7a0 2024-01-11T09:45:56.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.039+0000 7f0171d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0174159e60 0x7f017415aec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42142/0 (socket says 172.21.15.27:42142) 2024-01-11T09:45:56.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.039+0000 7f0171d9b700 1 -- 172.21.15.27:0/38370733 learned_addr learned my addr 172.21.15.27:0/38370733 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:56.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.039+0000 7f017259c700 1 --2- 172.21.15.27:0/38370733 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f017415a7a0 0x7f017415c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:56.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.039+0000 7f017159a700 1 -- 172.21.15.27:0/38370733 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023926177 0 0) 0x7f017415c4b0 con 0x7f0174159e60 2024-01-11T09:45:56.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.040+0000 7f017159a700 1 -- 172.21.15.27:0/38370733 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f015c003570 con 0x7f0174159e60 2024-01-11T09:45:56.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.040+0000 7f017159a700 1 -- 172.21.15.27:0/38370733 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1427167928 0 0) 0x7f015c003570 con 0x7f0174159e60 2024-01-11T09:45:56.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.040+0000 7f017159a700 1 -- 172.21.15.27:0/38370733 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f017415a7a0 msgr2=0x7f017415c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:56.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.040+0000 7f017159a700 1 --2- 172.21.15.27:0/38370733 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f017415a7a0 0x7f017415c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:56.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.040+0000 7f017159a700 1 -- 172.21.15.27:0/38370733 >> v1:172.21.15.77:6789/0 conn(0x7f01740685e0 legacy=0x7f0174157930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:56.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.040+0000 7f017159a700 1 -- 172.21.15.27:0/38370733 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017415bb80 con 0x7f0174159e60 2024-01-11T09:45:56.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.040+0000 7f017159a700 1 -- 172.21.15.27:0/38370733 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f0168001470 con 0x7f0174159e60 2024-01-11T09:45:56.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.040+0000 7f017159a700 1 -- 172.21.15.27:0/38370733 >> v1:172.21.15.27:6789/0 conn(0x7f0174159e60 legacy=0x7f017415aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:56.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.040+0000 7f017159a700 1 --2- 172.21.15.27:0/38370733 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f017415a7a0 0x7f015c003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:56.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.040+0000 7f017159a700 1 --2- 172.21.15.27:0/38370733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f015c003fd0 0x7f015c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:56.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.040+0000 7f017159a700 1 --2- 172.21.15.27:0/38370733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f015c004d50 0x7f015c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:56.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.040+0000 7f017159a700 1 -- 172.21.15.27:0/38370733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017415b8a0 con 0x7f015c004d50 2024-01-11T09:45:56.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.040+0000 7f017159a700 1 -- 172.21.15.27:0/38370733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c00bb40 con 0x7f015c003fd0 2024-01-11T09:45:56.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.040+0000 7f017159a700 1 -- 172.21.15.27:0/38370733 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c00bd00 con 0x7f017415a7a0 2024-01-11T09:45:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.040+0000 7f017259c700 1 --2- 172.21.15.27:0/38370733 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f017415a7a0 0x7f015c003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.040+0000 7f0172d9d700 1 --2- 172.21.15.27:0/38370733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f015c004d50 0x7f015c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.040+0000 7f0171d9b700 1 --2- 172.21.15.27:0/38370733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f015c003fd0 0x7f015c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.041+0000 7f017259c700 1 -- 172.21.15.27:0/38370733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f015c003fd0 msgr2=0x7f015c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.041+0000 7f017259c700 1 --2- 172.21.15.27:0/38370733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f015c003fd0 0x7f015c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.041+0000 7f017259c700 1 -- 172.21.15.27:0/38370733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f015c004d50 msgr2=0x7f015c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.041+0000 7f017259c700 1 --2- 172.21.15.27:0/38370733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f015c004d50 0x7f015c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.041+0000 7f017259c700 1 -- 172.21.15.27:0/38370733 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c003570 con 0x7f017415a7a0 2024-01-11T09:45:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.041+0000 7f0172d9d700 1 --2- 172.21.15.27:0/38370733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f015c004d50 0x7f015c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:45:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.041+0000 7f017259c700 1 --2- 172.21.15.27:0/38370733 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f017415a7a0 0x7f015c003a90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f0164009c00 tx=0x7f01640091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c9f19c4b02da0f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.041+0000 7f017159a700 1 -- 172.21.15.27:0/38370733 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016400e710 con 0x7f017415a7a0 2024-01-11T09:45:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.041+0000 7f017159a700 1 -- 172.21.15.27:0/38370733 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016400e870 con 0x7f017415a7a0 2024-01-11T09:45:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.041+0000 7f017159a700 1 -- 172.21.15.27:0/38370733 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016400ec30 con 0x7f017415a7a0 2024-01-11T09:45:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.042+0000 7f01789ff700 1 -- 172.21.15.27:0/38370733 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f017415a7a0 msgr2=0x7f015c003a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.042+0000 7f01789ff700 1 --2- 172.21.15.27:0/38370733 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f017415a7a0 0x7f015c003a90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f0164009c00 tx=0x7f01640091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:56.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.042+0000 7f0172d9d700 1 -- 172.21.15.27:0/38370733 reap_dead start 2024-01-11T09:45:56.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.042+0000 7f01789ff700 1 -- 172.21.15.27:0/38370733 shutdown_connections 2024-01-11T09:45:56.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.042+0000 7f01789ff700 1 -- 172.21.15.27:0/38370733 >> 172.21.15.27:0/38370733 conn(0x7f017406ed40 msgr2=0x7f017406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:56.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.042+0000 7f01789ff700 1 -- 172.21.15.27:0/38370733 shutdown_connections 2024-01-11T09:45:56.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.042+0000 7f01789ff700 1 -- 172.21.15.27:0/38370733 wait complete. 2024-01-11T09:45:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.044+0000 7f01789ff700 1 Processor -- start 2024-01-11T09:45:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.044+0000 7f01789ff700 1 -- start start 2024-01-11T09:45:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.045+0000 7f01789ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f017415a7a0 0x7f01740d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.045+0000 7f01789ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0174147980 0x7f017416a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.045+0000 7f01789ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f017416a8a0 0x7f017416d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.045+0000 7f01789ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0174169c30 con 0x7f017415a7a0 2024-01-11T09:45:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.045+0000 7f01789ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0174169ed0 con 0x7f017416a8a0 2024-01-11T09:45:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.045+0000 7f01789ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01740b2170 con 0x7f0174147980 2024-01-11T09:45:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.045+0000 7f017259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f017415a7a0 0x7f01740d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.045+0000 7f017259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f017415a7a0 0x7f01740d08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47680/0 (socket says 172.21.15.27:47680) 2024-01-11T09:45:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.045+0000 7f017259c700 1 -- 172.21.15.27:0/2777336280 learned_addr learned my addr 172.21.15.27:0/2777336280 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.045+0000 7f0171d9b700 1 --2- 172.21.15.27:0/2777336280 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0174147980 0x7f017416a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.045+0000 7f0172d9d700 1 --2- 172.21.15.27:0/2777336280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f017416a8a0 0x7f017416d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:56.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.045+0000 7f017259c700 1 -- 172.21.15.27:0/2777336280 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0174147980 msgr2=0x7f017416a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:56.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.045+0000 7f017259c700 1 --2- 172.21.15.27:0/2777336280 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0174147980 0x7f017416a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:56.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.045+0000 7f017259c700 1 -- 172.21.15.27:0/2777336280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f017416a8a0 msgr2=0x7f017416d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:56.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.045+0000 7f017259c700 1 --2- 172.21.15.27:0/2777336280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f017416a8a0 0x7f017416d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:56.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.045+0000 7f017259c700 1 -- 172.21.15.27:0/2777336280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01740b2450 con 0x7f017415a7a0 2024-01-11T09:45:56.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.046+0000 7f017259c700 1 --2- 172.21.15.27:0/2777336280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f017415a7a0 0x7f01740d08f0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f015c00be90 tx=0x7f016401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:56.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.046+0000 7f01627fc700 1 -- 172.21.15.27:0/2777336280 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0164022070 con 0x7f017415a7a0 2024-01-11T09:45:56.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.046+0000 7f01627fc700 1 -- 172.21.15.27:0/2777336280 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016400ebf0 con 0x7f017415a7a0 2024-01-11T09:45:56.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.046+0000 7f01627fc700 1 -- 172.21.15.27:0/2777336280 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01640276a0 con 0x7f017415a7a0 2024-01-11T09:45:56.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.046+0000 7f01789ff700 1 -- 172.21.15.27:0/2777336280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01740b2730 con 0x7f017415a7a0 2024-01-11T09:45:56.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.046+0000 7f01789ff700 1 -- 172.21.15.27:0/2777336280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01740d0e30 con 0x7f017415a7a0 2024-01-11T09:45:56.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.047+0000 7f01789ff700 1 -- 172.21.15.27:0/2777336280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01740b2c50 con 0x7f017415a7a0 2024-01-11T09:45:56.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.048+0000 7f01627fc700 1 -- 172.21.15.27:0/2777336280 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 131) v1 ==== 84964+0+0 (secure 0 0 0) 0x7f0164027ad0 con 0x7f017415a7a0 2024-01-11T09:45:56.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.049+0000 7f01627fc700 1 --2- 172.21.15.27:0/2777336280 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f01540671c0 0x7f0154069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:56.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.049+0000 7f01627fc700 1 -- 172.21.15.27:0/2777336280 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f0164086980 con 0x7f017415a7a0 2024-01-11T09:45:56.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.049+0000 7f0171d9b700 1 --2- 172.21.15.27:0/2777336280 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f01540671c0 0x7f0154069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:56.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.050+0000 7f0171d9b700 1 --2- 172.21.15.27:0/2777336280 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f01540671c0 0x7f0154069670 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0168000f60 tx=0x7f0168009040 comp rx=0 tx=0).ready entity=mgr.10501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:56.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.054+0000 7f01627fc700 1 -- 172.21.15.27:0/2777336280 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f01740b2c50 con 0x7f017415a7a0 2024-01-11T09:45:56.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.292+0000 7f01789ff700 1 -- 172.21.15.27:0/2777336280 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f017416a010 con 0x7f01540671c0 2024-01-11T09:45:56.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.348+0000 7f01627fc700 1 -- 172.21.15.27:0/2777336280 <== mgr.10501 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+61 (secure 0 0 0) 0x7f01680070b0 con 0x7f01540671c0 2024-01-11T09:45:56.339 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule deactivated for path mandatory_fs_argument_test_dir 2024-01-11T09:45:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.350+0000 7f01789ff700 1 -- 172.21.15.27:0/2777336280 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f01540671c0 msgr2=0x7f0154069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.350+0000 7f01789ff700 1 --2- 172.21.15.27:0/2777336280 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f01540671c0 0x7f0154069670 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0168000f60 tx=0x7f0168009040 comp rx=0 tx=0).stop 2024-01-11T09:45:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.350+0000 7f01789ff700 1 -- 172.21.15.27:0/2777336280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f017415a7a0 msgr2=0x7f01740d08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.350+0000 7f01789ff700 1 --2- 172.21.15.27:0/2777336280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f017415a7a0 0x7f01740d08f0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f015c00be90 tx=0x7f016401a000 comp rx=0 tx=0).stop 2024-01-11T09:45:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.350+0000 7f01789ff700 1 -- 172.21.15.27:0/2777336280 shutdown_connections 2024-01-11T09:45:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.350+0000 7f01789ff700 1 --2- 172.21.15.27:0/2777336280 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f01540671c0 0x7f0154069670 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.350+0000 7f01789ff700 1 --2- 172.21.15.27:0/2777336280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f017416a8a0 0x7f017416d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.350+0000 7f01789ff700 1 --2- 172.21.15.27:0/2777336280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f017415a7a0 0x7f01740d08f0 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.350+0000 7f01789ff700 1 --2- 172.21.15.27:0/2777336280 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0174147980 0x7f017416a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.350+0000 7f01789ff700 1 -- 172.21.15.27:0/2777336280 >> 172.21.15.27:0/2777336280 conn(0x7f017406ed40 msgr2=0x7f0174156f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.350+0000 7f01789ff700 1 -- 172.21.15.27:0/2777336280 shutdown_connections 2024-01-11T09:45:56.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.350+0000 7f01789ff700 1 -- 172.21.15.27:0/2777336280 wait complete. 2024-01-11T09:45:56.360 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path mandatory_fs_argument_test_dir 2024-01-11T09:45:56.360 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-01-11T09:45:56.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.572+0000 7f8ad6fff700 1 Processor -- start 2024-01-11T09:45:56.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.572+0000 7f8ad6fff700 1 -- start start 2024-01-11T09:45:56.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.573+0000 7f8ad6fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8ad0069d70 0x7f8ad0162190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:56.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.573+0000 7f8ad6fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad006d420 con 0x7f8ad0157440 2024-01-11T09:45:56.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.573+0000 7f8ad6fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad006d700 con 0x7f8ad0157810 2024-01-11T09:45:56.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.573+0000 7f8ad6fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ad006d9e0 con 0x7f8ad0069d70 2024-01-11T09:45:56.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.574+0000 7f8ad4d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8ad0157440 0x7f8ad00ae430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42158/0 (socket says 172.21.15.27:42158) 2024-01-11T09:45:56.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.574+0000 7f8ad4d9b700 1 -- 172.21.15.27:0/1250423470 learned_addr learned my addr 172.21.15.27:0/1250423470 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:56.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.574+0000 7f8ad559c700 1 --2- 172.21.15.27:0/1250423470 >> v2:172.21.15.77:3300/0 conn(0x7f8ad0069d70 0x7f8ad0162190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:56.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.574+0000 7f8acf7fe700 1 -- 172.21.15.27:0/1250423470 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097004508 0 0) 0x7f8ad006d420 con 0x7f8ad0157440 2024-01-11T09:45:56.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.574+0000 7f8acf7fe700 1 -- 172.21.15.27:0/1250423470 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac0003570 con 0x7f8ad0157440 2024-01-11T09:45:56.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.574+0000 7f8acf7fe700 1 -- 172.21.15.27:0/1250423470 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036773049 0 0) 0x7f8ad006d700 con 0x7f8ad0157810 2024-01-11T09:45:56.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.574+0000 7f8acf7fe700 1 -- 172.21.15.27:0/1250423470 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ad006d420 con 0x7f8ad0157810 2024-01-11T09:45:56.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.574+0000 7f8ad559c700 1 -- 172.21.15.27:0/1250423470 >> v1:172.21.15.77:6789/0 conn(0x7f8ad0157810 legacy=0x7f8ad0069660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:56.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.574+0000 7f8ad559c700 1 -- 172.21.15.27:0/1250423470 >> v1:172.21.15.27:6789/0 conn(0x7f8ad0157440 legacy=0x7f8ad00ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:56.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.575+0000 7f8ad559c700 1 -- 172.21.15.27:0/1250423470 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad006dcc0 con 0x7f8ad0069d70 2024-01-11T09:45:56.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.575+0000 7f8ad559c700 1 --2- 172.21.15.27:0/1250423470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ad0069d70 0x7f8ad0162190 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009e60 tx=0x7f8ac8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c18536f51bbf444 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:56.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.575+0000 7f8acf7fe700 1 -- 172.21.15.27:0/1250423470 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac800e070 con 0x7f8ad0069d70 2024-01-11T09:45:56.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.575+0000 7f8acf7fe700 1 -- 172.21.15.27:0/1250423470 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac800bab0 con 0x7f8ad0069d70 2024-01-11T09:45:56.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.575+0000 7f8acf7fe700 1 -- 172.21.15.27:0/1250423470 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac800be70 con 0x7f8ad0069d70 2024-01-11T09:45:56.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.576+0000 7f8ad6fff700 1 -- 172.21.15.27:0/1250423470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ad0069d70 msgr2=0x7f8ad0162190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:56.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.576+0000 7f8ad6fff700 1 --2- 172.21.15.27:0/1250423470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ad0069d70 0x7f8ad0162190 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009e60 tx=0x7f8ac8009780 comp rx=0 tx=0).stop 2024-01-11T09:45:56.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.576+0000 7f8ad6fff700 1 -- 172.21.15.27:0/1250423470 shutdown_connections 2024-01-11T09:45:56.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.576+0000 7f8ad6fff700 1 --2- 172.21.15.27:0/1250423470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ad0069d70 0x7f8ad0162190 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:56.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.576+0000 7f8ad6fff700 1 -- 172.21.15.27:0/1250423470 >> 172.21.15.27:0/1250423470 conn(0x7f8ad00680e0 msgr2=0x7f8ad00cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:56.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.576+0000 7f8ad6fff700 1 -- 172.21.15.27:0/1250423470 shutdown_connections 2024-01-11T09:45:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.576+0000 7f8ad6fff700 1 -- 172.21.15.27:0/1250423470 wait complete. 2024-01-11T09:45:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.577+0000 7f8ad6fff700 1 Processor -- start 2024-01-11T09:45:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.577+0000 7f8ad6fff700 1 -- start start 2024-01-11T09:45:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.578+0000 7f8ad6fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad0069d70 0x7f8ad00b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.578+0000 7f8ad6fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ad0157440 0x7f8ad00c2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.578+0000 7f8ad6fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ad0157810 0x7f8ad00c0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.578+0000 7f8ad6fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad00af7f0 con 0x7f8ad0069d70 2024-01-11T09:45:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.578+0000 7f8ad6fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad00aef50 con 0x7f8ad0157440 2024-01-11T09:45:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.578+0000 7f8ad6fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad0150520 con 0x7f8ad0157810 2024-01-11T09:45:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.579+0000 7f8ad4d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad0069d70 0x7f8ad00b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.579+0000 7f8ad4d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad0069d70 0x7f8ad00b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47682/0 (socket says 172.21.15.27:47682) 2024-01-11T09:45:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.579+0000 7f8ad4d9b700 1 -- 172.21.15.27:0/3474463024 learned_addr learned my addr 172.21.15.27:0/3474463024 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:56.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.579+0000 7f8acffff700 1 --2- 172.21.15.27:0/3474463024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ad0157440 0x7f8ad00c2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.579+0000 7f8ad559c700 1 --2- 172.21.15.27:0/3474463024 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ad0157810 0x7f8ad00c0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.579+0000 7f8ad4d9b700 1 -- 172.21.15.27:0/3474463024 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ad0157810 msgr2=0x7f8ad00c0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.579+0000 7f8ad4d9b700 1 --2- 172.21.15.27:0/3474463024 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ad0157810 0x7f8ad00c0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.579+0000 7f8ad4d9b700 1 -- 172.21.15.27:0/3474463024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ad0157440 msgr2=0x7f8ad00c2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.579+0000 7f8ad4d9b700 1 --2- 172.21.15.27:0/3474463024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ad0157440 0x7f8ad00c2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.579+0000 7f8ad4d9b700 1 -- 172.21.15.27:0/3474463024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac8009a20 con 0x7f8ad0069d70 2024-01-11T09:45:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.579+0000 7f8ad4d9b700 1 --2- 172.21.15.27:0/3474463024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad0069d70 0x7f8ad00b12c0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400c790 tx=0x7f8ac4007ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.580+0000 7f8accff9700 1 -- 172.21.15.27:0/3474463024 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac4007300 con 0x7f8ad0069d70 2024-01-11T09:45:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.580+0000 7f8accff9700 1 -- 172.21.15.27:0/3474463024 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac4007460 con 0x7f8ad0069d70 2024-01-11T09:45:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.580+0000 7f8accff9700 1 -- 172.21.15.27:0/3474463024 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac40057f0 con 0x7f8ad0069d70 2024-01-11T09:45:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.580+0000 7f8ad6fff700 1 -- 172.21.15.27:0/3474463024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad00af1a0 con 0x7f8ad0069d70 2024-01-11T09:45:56.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.580+0000 7f8ad6fff700 1 -- 172.21.15.27:0/3474463024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad00c26a0 con 0x7f8ad0069d70 2024-01-11T09:45:56.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.581+0000 7f8ad6fff700 1 -- 172.21.15.27:0/3474463024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad00afad0 con 0x7f8ad0069d70 2024-01-11T09:45:56.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.582+0000 7f8accff9700 1 -- 172.21.15.27:0/3474463024 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 131) v1 ==== 84964+0+0 (secure 0 0 0) 0x7f8ac4005ba0 con 0x7f8ad0069d70 2024-01-11T09:45:56.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.583+0000 7f8accff9700 1 --2- 172.21.15.27:0/3474463024 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8ab40671c0 0x7f8ab4069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:56.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.583+0000 7f8accff9700 1 -- 172.21.15.27:0/3474463024 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f8ac4084930 con 0x7f8ad0069d70 2024-01-11T09:45:56.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.583+0000 7f8acffff700 1 --2- 172.21.15.27:0/3474463024 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8ab40671c0 0x7f8ab4069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:56.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.584+0000 7f8acffff700 1 --2- 172.21.15.27:0/3474463024 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8ab40671c0 0x7f8ab4069670 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0154d90 tx=0x7f8abc009040 comp rx=0 tx=0).ready entity=mgr.10501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:56.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.587+0000 7f8accff9700 1 -- 172.21.15.27:0/3474463024 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8ac40514c0 con 0x7f8ad0069d70 2024-01-11T09:45:56.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.815+0000 7f8ad6fff700 1 -- 172.21.15.27:0/3474463024 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ad004c3c0 con 0x7f8ab40671c0 2024-01-11T09:45:56.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.881+0000 7f8accff9700 1 -- 172.21.15.27:0/3474463024 <== mgr.10501 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7f8ad004c3c0 con 0x7f8ab40671c0 2024-01-11T09:45:56.873 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule removed for path mandatory_fs_argument_test_dir 2024-01-11T09:45:56.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.884+0000 7f8ad6fff700 1 -- 172.21.15.27:0/3474463024 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8ab40671c0 msgr2=0x7f8ab4069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:56.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.884+0000 7f8ad6fff700 1 --2- 172.21.15.27:0/3474463024 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8ab40671c0 0x7f8ab4069670 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0154d90 tx=0x7f8abc009040 comp rx=0 tx=0).stop 2024-01-11T09:45:56.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.884+0000 7f8ad6fff700 1 -- 172.21.15.27:0/3474463024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad0069d70 msgr2=0x7f8ad00b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:56.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.884+0000 7f8ad6fff700 1 --2- 172.21.15.27:0/3474463024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad0069d70 0x7f8ad00b12c0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400c790 tx=0x7f8ac4007ce0 comp rx=0 tx=0).stop 2024-01-11T09:45:56.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.884+0000 7f8ad6fff700 1 -- 172.21.15.27:0/3474463024 shutdown_connections 2024-01-11T09:45:56.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.884+0000 7f8ad6fff700 1 --2- 172.21.15.27:0/3474463024 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8ad0157810 0x7f8ad00c0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:56.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.884+0000 7f8ad6fff700 1 --2- 172.21.15.27:0/3474463024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad0069d70 0x7f8ad00b12c0 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:56.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.884+0000 7f8ad6fff700 1 --2- 172.21.15.27:0/3474463024 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8ab40671c0 0x7f8ab4069670 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:56.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.884+0000 7f8ad6fff700 1 --2- 172.21.15.27:0/3474463024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ad0157440 0x7f8ad00c2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.884+0000 7f8ad6fff700 1 -- 172.21.15.27:0/3474463024 >> 172.21.15.27:0/3474463024 conn(0x7f8ad00680e0 msgr2=0x7f8ad00cb770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.884+0000 7f8ad6fff700 1 -- 172.21.15.27:0/3474463024 shutdown_connections 2024-01-11T09:45:56.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:56.884+0000 7f8ad6fff700 1 -- 172.21.15.27:0/3474463024 wait complete. 2024-01-11T09:45:56.894 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path mandatory_fs_argument_test_dir 2024-01-11T09:45:56.894 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:45:56.894 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2024-01-11T09:45:56.946 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-11T09:45:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.159+0000 7fa1f43ff700 1 Processor -- start 2024-01-11T09:45:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.159+0000 7fa1f43ff700 1 -- start start 2024-01-11T09:45:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.161+0000 7fa1f43ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1ec13fbb0 0x7fa1ec0ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.161+0000 7fa1f43ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa1ec13ff80 0x7fa1ec15a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.161+0000 7fa1f43ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ec0c6b30 con 0x7fa1ec0cace0 2024-01-11T09:45:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.161+0000 7fa1f43ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ec0c6800 con 0x7fa1ec13ff80 2024-01-11T09:45:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.161+0000 7fa1f43ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0cb240 con 0x7fa1ec13fbb0 2024-01-11T09:45:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.161+0000 7fa1f219b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1ec13fbb0 0x7fa1ec0ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.161+0000 7fa1f199a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa1ec13ff80 0x7fa1ec15a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.161+0000 7fa1f219b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1ec13fbb0 0x7fa1ec0ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:48268/0 (socket says 172.21.15.27:48268) 2024-01-11T09:45:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.161+0000 7fa1f219b700 1 -- 172.21.15.27:0/3788770257 learned_addr learned my addr 172.21.15.27:0/3788770257 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.162+0000 7fa1f199a700 1 -- 172.21.15.27:0/3788770257 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1ec13fbb0 msgr2=0x7fa1ec0ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.162+0000 7fa1f199a700 1 --2- 172.21.15.27:0/3788770257 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1ec13fbb0 0x7fa1ec0ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.162+0000 7fa1f199a700 1 -- 172.21.15.27:0/3788770257 >> v1:172.21.15.77:6789/0 conn(0x7fa1ec0cace0 legacy=0x7fa1ec06e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.162+0000 7fa1f199a700 1 -- 172.21.15.27:0/3788770257 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec0cb520 con 0x7fa1ec13ff80 2024-01-11T09:45:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.162+0000 7fa1f1199700 1 -- 172.21.15.27:0/3788770257 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866038905 0 0) 0x7fa1ec0c6b30 con 0x7fa1ec0cace0 2024-01-11T09:45:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.162+0000 7fa1f199a700 1 --2- 172.21.15.27:0/3788770257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1ec13ff80 0x7fa1ec15a020 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc0098e0 tx=0x7fa1dc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39ac5c16def3221b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.162+0000 7fa1f1199700 1 -- 172.21.15.27:0/3788770257 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc00b640 con 0x7fa1ec13ff80 2024-01-11T09:45:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.162+0000 7fa1f1199700 1 -- 172.21.15.27:0/3788770257 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1dc00b7a0 con 0x7fa1ec13ff80 2024-01-11T09:45:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.163+0000 7fa1f1199700 1 -- 172.21.15.27:0/3788770257 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc00bb40 con 0x7fa1ec13ff80 2024-01-11T09:45:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.163+0000 7fa1f43ff700 1 -- 172.21.15.27:0/3788770257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1ec13ff80 msgr2=0x7fa1ec15a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.163+0000 7fa1f43ff700 1 --2- 172.21.15.27:0/3788770257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1ec13ff80 0x7fa1ec15a020 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc0098e0 tx=0x7fa1dc0091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.163+0000 7fa1f43ff700 1 -- 172.21.15.27:0/3788770257 shutdown_connections 2024-01-11T09:45:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.163+0000 7fa1f43ff700 1 --2- 172.21.15.27:0/3788770257 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1ec13fbb0 0x7fa1ec0ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.163+0000 7fa1f43ff700 1 --2- 172.21.15.27:0/3788770257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1ec13ff80 0x7fa1ec15a020 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.163+0000 7fa1f43ff700 1 -- 172.21.15.27:0/3788770257 >> 172.21.15.27:0/3788770257 conn(0x7fa1ec0680a0 msgr2=0x7fa1ec0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.163+0000 7fa1f43ff700 1 -- 172.21.15.27:0/3788770257 shutdown_connections 2024-01-11T09:45:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.163+0000 7fa1f43ff700 1 -- 172.21.15.27:0/3788770257 wait complete. 2024-01-11T09:45:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.165+0000 7fa1f43ff700 1 Processor -- start 2024-01-11T09:45:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.165+0000 7fa1f43ff700 1 -- start start 2024-01-11T09:45:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.166+0000 7fa1f43ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1ec0cace0 0x7fa1ec0c5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.166+0000 7fa1f43ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1ec13fbb0 0x7fa1ec1574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.166+0000 7fa1f43ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1ec13ff80 0x7fa1ec13cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.166+0000 7fa1f43ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0c3680 con 0x7fa1ec13fbb0 2024-01-11T09:45:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.166+0000 7fa1f43ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0c5eb0 con 0x7fa1ec13ff80 2024-01-11T09:45:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.166+0000 7fa1f43ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec169960 con 0x7fa1ec0cace0 2024-01-11T09:45:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.166+0000 7fa1f199a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1ec13fbb0 0x7fa1ec1574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.166+0000 7fa1f199a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1ec13fbb0 0x7fa1ec1574a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47684/0 (socket says 172.21.15.27:47684) 2024-01-11T09:45:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.166+0000 7fa1f199a700 1 -- 172.21.15.27:0/2544702270 learned_addr learned my addr 172.21.15.27:0/2544702270 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.166+0000 7fa1f299c700 1 --2- 172.21.15.27:0/2544702270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1ec13ff80 0x7fa1ec13cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.166+0000 7fa1f219b700 1 --2- 172.21.15.27:0/2544702270 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1ec0cace0 0x7fa1ec0c5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.166+0000 7fa1f199a700 1 -- 172.21.15.27:0/2544702270 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1ec0cace0 msgr2=0x7fa1ec0c5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.166+0000 7fa1f199a700 1 --2- 172.21.15.27:0/2544702270 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1ec0cace0 0x7fa1ec0c5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.166+0000 7fa1f199a700 1 -- 172.21.15.27:0/2544702270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1ec13ff80 msgr2=0x7fa1ec13cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.166+0000 7fa1f199a700 1 --2- 172.21.15.27:0/2544702270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1ec13ff80 0x7fa1ec13cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.166+0000 7fa1f199a700 1 -- 172.21.15.27:0/2544702270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc009420 con 0x7fa1ec13fbb0 2024-01-11T09:45:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.166+0000 7fa1f219b700 1 --2- 172.21.15.27:0/2544702270 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1ec0cace0 0x7fa1ec0c5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:45:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.167+0000 7fa1f199a700 1 --2- 172.21.15.27:0/2544702270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1ec13fbb0 0x7fa1ec1574a0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc009f80 tx=0x7fa1dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.167+0000 7fa1e27fc700 1 -- 172.21.15.27:0/2544702270 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc00bb20 con 0x7fa1ec13fbb0 2024-01-11T09:45:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.167+0000 7fa1e27fc700 1 -- 172.21.15.27:0/2544702270 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1dc011420 con 0x7fa1ec13fbb0 2024-01-11T09:45:57.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.167+0000 7fa1e27fc700 1 -- 172.21.15.27:0/2544702270 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc0117a0 con 0x7fa1ec13fbb0 2024-01-11T09:45:57.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.167+0000 7fa1f43ff700 1 -- 172.21.15.27:0/2544702270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ec0c6120 con 0x7fa1ec13fbb0 2024-01-11T09:45:57.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.167+0000 7fa1f43ff700 1 -- 172.21.15.27:0/2544702270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ec0c3310 con 0x7fa1ec13fbb0 2024-01-11T09:45:57.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.168+0000 7fa1f43ff700 1 -- 172.21.15.27:0/2544702270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ec04c3c0 con 0x7fa1ec13fbb0 2024-01-11T09:45:57.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.169+0000 7fa1e27fc700 1 -- 172.21.15.27:0/2544702270 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 132) v1 ==== 85020+0+0 (secure 0 0 0) 0x7fa1dc011b50 con 0x7fa1ec13fbb0 2024-01-11T09:45:57.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.170+0000 7fa1e27fc700 1 --2- 172.21.15.27:0/2544702270 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa1d0067250 0x7fa1d0069700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:57.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.170+0000 7fa1e27fc700 1 -- 172.21.15.27:0/2544702270 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7fa1dc086570 con 0x7fa1ec13fbb0 2024-01-11T09:45:57.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.171+0000 7fa1f219b700 1 --2- 172.21.15.27:0/2544702270 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa1d0067250 0x7fa1d0069700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:57.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.171+0000 7fa1f219b700 1 --2- 172.21.15.27:0/2544702270 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa1d0067250 0x7fa1d0069700 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8006f60 tx=0x7fa1e8008040 comp rx=0 tx=0).ready entity=mgr.10501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:57.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.174+0000 7fa1e27fc700 1 -- 172.21.15.27:0/2544702270 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa1dc052f80 con 0x7fa1ec13fbb0 2024-01-11T09:45:57.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.431+0000 7fa1f43ff700 1 -- 172.21.15.27:0/2544702270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fa1ec0cb800 con 0x7fa1ec13fbb0 2024-01-11T09:45:57.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.935+0000 7fa1e27fc700 1 -- 172.21.15.27:0/2544702270 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 133) v1 ==== 85003+0+0 (secure 0 0 0) 0x7fa1dc00fda0 con 0x7fa1ec13fbb0 2024-01-11T09:45:57.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.936+0000 7fa1e27fc700 1 -- 172.21.15.27:0/2544702270 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v133) v1 ==== 93+0+0 (secure 0 0 0) 0x7fa1dc056a10 con 0x7fa1ec13fbb0 2024-01-11T09:45:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.940+0000 7fa1f43ff700 1 -- 172.21.15.27:0/2544702270 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa1d0067250 msgr2=0x7fa1d0069700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.940+0000 7fa1f43ff700 1 --2- 172.21.15.27:0/2544702270 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa1d0067250 0x7fa1d0069700 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8006f60 tx=0x7fa1e8008040 comp rx=0 tx=0).stop 2024-01-11T09:45:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.940+0000 7fa1f43ff700 1 -- 172.21.15.27:0/2544702270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1ec13fbb0 msgr2=0x7fa1ec1574a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.940+0000 7fa1f43ff700 1 --2- 172.21.15.27:0/2544702270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1ec13fbb0 0x7fa1ec1574a0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc009f80 tx=0x7fa1dc017000 comp rx=0 tx=0).stop 2024-01-11T09:45:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.940+0000 7fa1f43ff700 1 -- 172.21.15.27:0/2544702270 shutdown_connections 2024-01-11T09:45:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.940+0000 7fa1f43ff700 1 --2- 172.21.15.27:0/2544702270 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa1d0067250 0x7fa1d0069700 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.940+0000 7fa1f43ff700 1 --2- 172.21.15.27:0/2544702270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1ec13ff80 0x7fa1ec13cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.940+0000 7fa1f43ff700 1 --2- 172.21.15.27:0/2544702270 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa1ec0cace0 0x7fa1ec0c5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.940+0000 7fa1f43ff700 1 --2- 172.21.15.27:0/2544702270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1ec13fbb0 0x7fa1ec1574a0 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.940+0000 7fa1f43ff700 1 -- 172.21.15.27:0/2544702270 >> 172.21.15.27:0/2544702270 conn(0x7fa1ec0680a0 msgr2=0x7fa1ec13eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.940+0000 7fa1f43ff700 1 -- 172.21.15.27:0/2544702270 shutdown_connections 2024-01-11T09:45:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:57.940+0000 7fa1f43ff700 1 -- 172.21.15.27:0/2544702270 wait complete. 2024-01-11T09:45:57.952 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T09:45:57.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:45:58.046 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:45:58.133 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T09:45:58.134 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-11T09:45:58.134 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:45:58.134 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-01-11T09:45:58.149 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:58.157+0000 7feb4febe1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:45:58.155 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:58.163+0000 7f89e6eae1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:45:58.217 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T09:45:58.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:45:58.217 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-11T09:45:58.334 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:58.342+0000 7f89e6eae1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:45:58.345 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:45:58.357 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:58.366+0000 7feb4febe1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:45:58.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.560+0000 7f86f87ff700 1 Processor -- start 2024-01-11T09:45:58.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.560+0000 7f86f87ff700 1 -- start start 2024-01-11T09:45:58.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.563+0000 7f86f87ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f86f00c8620 0x7f86f00af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:58.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.563+0000 7f86f87ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f0157390 con 0x7f86f00b09e0 2024-01-11T09:45:58.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.563+0000 7f86f87ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f00afeb0 con 0x7f86f0069d40 2024-01-11T09:45:58.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.563+0000 7f86f87ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f006a200 con 0x7f86f00c8620 2024-01-11T09:45:58.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.563+0000 7f86f6d9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f86f00b09e0 0x7f86f00accc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42176/0 (socket says 172.21.15.27:42176) 2024-01-11T09:45:58.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.563+0000 7f86f6d9c700 1 -- 172.21.15.27:0/3419123957 learned_addr learned my addr 172.21.15.27:0/3419123957 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:45:58.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.563+0000 7f86f659b700 1 --2- 172.21.15.27:0/3419123957 >> v2:172.21.15.77:3300/0 conn(0x7f86f00c8620 0x7f86f00af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:58.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.563+0000 7f86f5599700 1 -- 172.21.15.27:0/3419123957 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069655657 0 0) 0x7f86f0157390 con 0x7f86f00b09e0 2024-01-11T09:45:58.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.564+0000 7f86f5599700 1 -- 172.21.15.27:0/3419123957 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86dc003570 con 0x7f86f00b09e0 2024-01-11T09:45:58.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.564+0000 7f86f5599700 1 -- 172.21.15.27:0/3419123957 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591138389 0 0) 0x7f86f00afeb0 con 0x7f86f0069d40 2024-01-11T09:45:58.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.564+0000 7f86f659b700 1 -- 172.21.15.27:0/3419123957 >> v1:172.21.15.77:6789/0 conn(0x7f86f0069d40 legacy=0x7f86f00b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:58.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.564+0000 7f86f659b700 1 -- 172.21.15.27:0/3419123957 >> v1:172.21.15.27:6789/0 conn(0x7f86f00b09e0 legacy=0x7f86f00accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:58.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.564+0000 7f86f659b700 1 -- 172.21.15.27:0/3419123957 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f0156d40 con 0x7f86f00c8620 2024-01-11T09:45:58.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.564+0000 7f86f659b700 1 --2- 172.21.15.27:0/3419123957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86f00c8620 0x7f86f00af970 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f86ec009910 tx=0x7f86ec0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed7bfb7597879633 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:58.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.564+0000 7f86f5599700 1 -- 172.21.15.27:0/3419123957 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ec00b640 con 0x7f86f00c8620 2024-01-11T09:45:58.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.564+0000 7f86f5599700 1 -- 172.21.15.27:0/3419123957 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86ec00b7a0 con 0x7f86f00c8620 2024-01-11T09:45:58.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.564+0000 7f86f5599700 1 -- 172.21.15.27:0/3419123957 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ec00bb60 con 0x7f86f00c8620 2024-01-11T09:45:58.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.565+0000 7f86f87ff700 1 -- 172.21.15.27:0/3419123957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86f00c8620 msgr2=0x7f86f00af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:58.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.565+0000 7f86f87ff700 1 --2- 172.21.15.27:0/3419123957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86f00c8620 0x7f86f00af970 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f86ec009910 tx=0x7f86ec0091a0 comp rx=0 tx=0).stop 2024-01-11T09:45:58.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.565+0000 7f86f87ff700 1 -- 172.21.15.27:0/3419123957 shutdown_connections 2024-01-11T09:45:58.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.565+0000 7f86f87ff700 1 --2- 172.21.15.27:0/3419123957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86f00c8620 0x7f86f00af970 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:58.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.565+0000 7f86f87ff700 1 -- 172.21.15.27:0/3419123957 >> 172.21.15.27:0/3419123957 conn(0x7f86f00680e0 msgr2=0x7f86f0069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:58.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.565+0000 7f86f87ff700 1 -- 172.21.15.27:0/3419123957 shutdown_connections 2024-01-11T09:45:58.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.565+0000 7f86f87ff700 1 -- 172.21.15.27:0/3419123957 wait complete. 2024-01-11T09:45:58.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.567+0000 7f86f87ff700 1 Processor -- start 2024-01-11T09:45:58.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.567+0000 7f86f87ff700 1 -- start start 2024-01-11T09:45:58.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.569+0000 7f86f87ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86f0069d40 0x7f86f0149f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:58.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.569+0000 7f86f87ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f86f00b09e0 0x7f86f015b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:58.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.569+0000 7f86f87ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86f00c8620 0x7f86f0167510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:58.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.569+0000 7f86f87ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f0150960 con 0x7f86f00c8620 2024-01-11T09:45:58.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.569+0000 7f86f87ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f014fd50 con 0x7f86f0069d40 2024-01-11T09:45:58.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.569+0000 7f86f87ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f0150030 con 0x7f86f00b09e0 2024-01-11T09:45:58.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.569+0000 7f86f659b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86f0069d40 0x7f86f0149f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:58.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.569+0000 7f86f6d9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86f00c8620 0x7f86f0167510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:58.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.569+0000 7f86f5d9a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f86f00b09e0 0x7f86f015b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:58.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.569+0000 7f86f6d9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86f00c8620 0x7f86f0167510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47708/0 (socket says 172.21.15.27:47708) 2024-01-11T09:45:58.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.569+0000 7f86f659b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86f0069d40 0x7f86f0149f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:46584/0 (socket says 172.21.15.27:46584) 2024-01-11T09:45:58.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.569+0000 7f86f6d9c700 1 -- 172.21.15.27:0/1457103232 learned_addr learned my addr 172.21.15.27:0/1457103232 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:58.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.570+0000 7f86f6d9c700 1 -- 172.21.15.27:0/1457103232 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f86f00b09e0 msgr2=0x7f86f015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:58.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.570+0000 7f86f6d9c700 1 --2- 172.21.15.27:0/1457103232 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f86f00b09e0 0x7f86f015b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:58.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.570+0000 7f86f6d9c700 1 -- 172.21.15.27:0/1457103232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86f0069d40 msgr2=0x7f86f0149f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:58.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.570+0000 7f86f6d9c700 1 --2- 172.21.15.27:0/1457103232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86f0069d40 0x7f86f0149f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:58.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.570+0000 7f86f6d9c700 1 -- 172.21.15.27:0/1457103232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ec009420 con 0x7f86f00c8620 2024-01-11T09:45:58.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.570+0000 7f86f6d9c700 1 --2- 172.21.15.27:0/1457103232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86f00c8620 0x7f86f0167510 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f86e800cc40 tx=0x7f86e8001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:58.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.570+0000 7f86e67fc700 1 -- 172.21.15.27:0/1457103232 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e800a8b0 con 0x7f86f00c8620 2024-01-11T09:45:58.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.570+0000 7f86e67fc700 1 -- 172.21.15.27:0/1457103232 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86e800aa10 con 0x7f86f00c8620 2024-01-11T09:45:58.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.570+0000 7f86e67fc700 1 -- 172.21.15.27:0/1457103232 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e800ad90 con 0x7f86f00c8620 2024-01-11T09:45:58.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.570+0000 7f86f87ff700 1 -- 172.21.15.27:0/1457103232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f0150310 con 0x7f86f00c8620 2024-01-11T09:45:58.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.570+0000 7f86f87ff700 1 -- 172.21.15.27:0/1457103232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f015b620 con 0x7f86f00c8620 2024-01-11T09:45:58.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.572+0000 7f86f87ff700 1 -- 172.21.15.27:0/1457103232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f00718d0 con 0x7f86f00c8620 2024-01-11T09:45:58.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.572+0000 7f86e67fc700 1 -- 172.21.15.27:0/1457103232 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 133) v1 ==== 85003+0+0 (secure 0 0 0) 0x7f86e8007300 con 0x7f86f00c8620 2024-01-11T09:45:58.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.573+0000 7f86e67fc700 1 --2- 172.21.15.27:0/1457103232 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f86d4067200 0x7f86d40696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:58.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.573+0000 7f86e67fc700 1 -- 172.21.15.27:0/1457103232 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f86e8085210 con 0x7f86f00c8620 2024-01-11T09:45:58.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.573+0000 7f86f659b700 1 -- 172.21.15.27:0/1457103232 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f86d4067200 msgr2=0x7f86d40696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:58.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.574+0000 7f86f659b700 1 --2- 172.21.15.27:0/1457103232 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f86d4067200 0x7f86d40696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:45:58.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.578+0000 7f86e67fc700 1 -- 172.21.15.27:0/1457103232 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f86e8051c50 con 0x7f86f00c8620 2024-01-11T09:45:58.648 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:58.656+0000 7f89e6eae1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:45:58.703 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:58.712+0000 7feb4febe1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:45:58.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.775+0000 7f86f659b700 1 -- 172.21.15.27:0/1457103232 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f86d4067200 msgr2=0x7f86d40696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:58.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.775+0000 7f86f659b700 1 --2- 172.21.15.27:0/1457103232 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f86d4067200 0x7f86d40696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:45:58.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.830+0000 7f86f87ff700 1 -- 172.21.15.27:0/1457103232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86f0150aa0 con 0x7f86f00c8620 2024-01-11T09:45:58.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.831+0000 7f86e67fc700 1 -- 172.21.15.27:0/1457103232 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+5690 (secure 0 0 0) 0x7f86e8055270 con 0x7f86f00c8620 2024-01-11T09:45:58.822 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:58.823 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":117,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2103359888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":2103359888},{"type":"v1","addr":"172.21.15.77:6834","nonce":2103359888}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8795,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2650824649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2650824649},{"type":"v1","addr":"172.21.15.77:6835","nonce":2650824649}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:45:23.637614+0000","modified":"2024-01-11T09:45:27.146191+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10599":{"gid":10599,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1143960902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1143960902},{"type":"v1","addr":"172.21.15.27:6835","nonce":1143960902}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:45:37.460120+0000","modified":"2024-01-11T09:45:40.977702+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10596},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10596":{"gid":10596,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6834/2807751836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2807751836},{"type":"v1","addr":"172.21.15.27:6834","nonce":2807751836}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":14}]} 2024-01-11T09:45:58.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.833+0000 7f86f87ff700 1 -- 172.21.15.27:0/1457103232 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f86d4067200 msgr2=0x7f86d40696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:45:58.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.833+0000 7f86f87ff700 1 --2- 172.21.15.27:0/1457103232 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f86d4067200 0x7f86d40696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:58.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.833+0000 7f86f87ff700 1 -- 172.21.15.27:0/1457103232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86f00c8620 msgr2=0x7f86f0167510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:58.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.833+0000 7f86f87ff700 1 --2- 172.21.15.27:0/1457103232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86f00c8620 0x7f86f0167510 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f86e800cc40 tx=0x7f86e8001aa0 comp rx=0 tx=0).stop 2024-01-11T09:45:58.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.834+0000 7f86f87ff700 1 -- 172.21.15.27:0/1457103232 shutdown_connections 2024-01-11T09:45:58.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.834+0000 7f86f87ff700 1 --2- 172.21.15.27:0/1457103232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86f00c8620 0x7f86f0167510 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:58.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.834+0000 7f86f87ff700 1 --2- 172.21.15.27:0/1457103232 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f86d4067200 0x7f86d40696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:58.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.834+0000 7f86f87ff700 1 --2- 172.21.15.27:0/1457103232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86f0069d40 0x7f86f0149f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:58.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.834+0000 7f86f87ff700 1 --2- 172.21.15.27:0/1457103232 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f86f00b09e0 0x7f86f015b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:58.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.834+0000 7f86f87ff700 1 -- 172.21.15.27:0/1457103232 >> 172.21.15.27:0/1457103232 conn(0x7f86f00680e0 msgr2=0x7f86f014a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:58.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.834+0000 7f86f87ff700 1 -- 172.21.15.27:0/1457103232 shutdown_connections 2024-01-11T09:45:58.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:58.834+0000 7f86f87ff700 1 -- 172.21.15.27:0/1457103232 wait complete. 2024-01-11T09:45:58.826 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 117 2024-01-11T09:45:58.845 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:45:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.096+0000 7f7b531ff700 1 Processor -- start 2024-01-11T09:45:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.096+0000 7f7b531ff700 1 -- start start 2024-01-11T09:45:59.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.098+0000 7f7b531ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7b4c0cbfa0 0x7f7b4c0ca230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:59.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.098+0000 7f7b531ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b4c06e770 0x7f7b4c06bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:59.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.098+0000 7f7b531ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b4c0b00c0 con 0x7f7b4c0cc370 2024-01-11T09:45:59.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.098+0000 7f7b531ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b4c0ca770 con 0x7f7b4c0cbfa0 2024-01-11T09:45:59.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.098+0000 7f7b531ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c06c660 con 0x7f7b4c06e770 2024-01-11T09:45:59.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.098+0000 7f7b50f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7b4c0cbfa0 0x7f7b4c0ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:59.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.098+0000 7f7b50f9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7b4c0cbfa0 0x7f7b4c0ca230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47716/0 (socket says 172.21.15.27:47716) 2024-01-11T09:45:59.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.098+0000 7f7b50f9b700 1 -- 172.21.15.27:0/3599596883 learned_addr learned my addr 172.21.15.27:0/3599596883 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:59.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.098+0000 7f7b4bfff700 1 --2- 172.21.15.27:0/3599596883 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b4c06e770 0x7f7b4c06bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:59.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.098+0000 7f7b50f9b700 1 -- 172.21.15.27:0/3599596883 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b4c06e770 msgr2=0x7f7b4c06bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:59.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.098+0000 7f7b50f9b700 1 --2- 172.21.15.27:0/3599596883 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b4c06e770 0x7f7b4c06bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.098+0000 7f7b50f9b700 1 -- 172.21.15.27:0/3599596883 >> v1:172.21.15.77:6789/0 conn(0x7f7b4c0cc370 legacy=0x7f7b4c0b16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:59.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.098+0000 7f7b50f9b700 1 -- 172.21.15.27:0/3599596883 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4c06c940 con 0x7f7b4c0cbfa0 2024-01-11T09:45:59.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.099+0000 7f7b50f9b700 1 --2- 172.21.15.27:0/3599596883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b4c0cbfa0 0x7f7b4c0ca230 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f7b400099b0 tx=0x7f7b40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f020d22c3749af server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.099+0000 7f7b4b7fe700 1 -- 172.21.15.27:0/3599596883 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b40009ea0 con 0x7f7b4c0cbfa0 2024-01-11T09:45:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.099+0000 7f7b4b7fe700 1 -- 172.21.15.27:0/3599596883 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4000b860 con 0x7f7b4c0cbfa0 2024-01-11T09:45:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.099+0000 7f7b4b7fe700 1 -- 172.21.15.27:0/3599596883 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4000bc00 con 0x7f7b4c0cbfa0 2024-01-11T09:45:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.100+0000 7f7b531ff700 1 -- 172.21.15.27:0/3599596883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b4c0cbfa0 msgr2=0x7f7b4c0ca230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.100+0000 7f7b531ff700 1 --2- 172.21.15.27:0/3599596883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b4c0cbfa0 0x7f7b4c0ca230 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f7b400099b0 tx=0x7f7b40009250 comp rx=0 tx=0).stop 2024-01-11T09:45:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.100+0000 7f7b531ff700 1 -- 172.21.15.27:0/3599596883 shutdown_connections 2024-01-11T09:45:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.100+0000 7f7b531ff700 1 --2- 172.21.15.27:0/3599596883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b4c0cbfa0 0x7f7b4c0ca230 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.100+0000 7f7b531ff700 1 --2- 172.21.15.27:0/3599596883 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b4c06e770 0x7f7b4c06bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.100+0000 7f7b531ff700 1 -- 172.21.15.27:0/3599596883 >> 172.21.15.27:0/3599596883 conn(0x7f7b4c06b200 msgr2=0x7f7b4c0aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.100+0000 7f7b531ff700 1 -- 172.21.15.27:0/3599596883 shutdown_connections 2024-01-11T09:45:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.100+0000 7f7b531ff700 1 -- 172.21.15.27:0/3599596883 wait complete. 2024-01-11T09:45:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.101+0000 7f7b531ff700 1 Processor -- start 2024-01-11T09:45:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.101+0000 7f7b531ff700 1 -- start start 2024-01-11T09:45:59.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.102+0000 7f7b531ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b4c06e770 0x7f7b4c0c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:59.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.102+0000 7f7b531ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b4c0cbfa0 0x7f7b4c145db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:59.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.102+0000 7f7b531ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b4c0cc370 0x7f7b4c0c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:59.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.102+0000 7f7b531ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c0c7fc0 con 0x7f7b4c0cc370 2024-01-11T09:45:59.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.102+0000 7f7b531ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c152fc0 con 0x7f7b4c0cbfa0 2024-01-11T09:45:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.102+0000 7f7b531ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c15f1f0 con 0x7f7b4c06e770 2024-01-11T09:45:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.103+0000 7f7b5179c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b4c0cc370 0x7f7b4c0c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.103+0000 7f7b5179c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b4c0cc370 0x7f7b4c0c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47724/0 (socket says 172.21.15.27:47724) 2024-01-11T09:45:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.103+0000 7f7b5179c700 1 -- 172.21.15.27:0/2942277539 learned_addr learned my addr 172.21.15.27:0/2942277539 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.103+0000 7f7b4bfff700 1 --2- 172.21.15.27:0/2942277539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b4c0cbfa0 0x7f7b4c145db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.103+0000 7f7b50f9b700 1 --2- 172.21.15.27:0/2942277539 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b4c06e770 0x7f7b4c0c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.103+0000 7f7b5179c700 1 -- 172.21.15.27:0/2942277539 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b4c06e770 msgr2=0x7f7b4c0c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.103+0000 7f7b5179c700 1 --2- 172.21.15.27:0/2942277539 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b4c06e770 0x7f7b4c0c64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.103+0000 7f7b5179c700 1 -- 172.21.15.27:0/2942277539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b4c0cbfa0 msgr2=0x7f7b4c145db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.103+0000 7f7b5179c700 1 --2- 172.21.15.27:0/2942277539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b4c0cbfa0 0x7f7b4c145db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.103+0000 7f7b5179c700 1 -- 172.21.15.27:0/2942277539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b400094f0 con 0x7f7b4c0cc370 2024-01-11T09:45:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.103+0000 7f7b5179c700 1 --2- 172.21.15.27:0/2942277539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b4c0cc370 0x7f7b4c0c6c00 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f7b4400c840 tx=0x7f7b44001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.104+0000 7f7b48ff9700 1 -- 172.21.15.27:0/2942277539 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b44012630 con 0x7f7b4c0cc370 2024-01-11T09:45:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.104+0000 7f7b48ff9700 1 -- 172.21.15.27:0/2942277539 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b440127b0 con 0x7f7b4c0cc370 2024-01-11T09:45:59.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.104+0000 7f7b531ff700 1 -- 172.21.15.27:0/2942277539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4c1462f0 con 0x7f7b4c0cc370 2024-01-11T09:45:59.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.104+0000 7f7b48ff9700 1 -- 172.21.15.27:0/2942277539 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b44012b70 con 0x7f7b4c0cc370 2024-01-11T09:45:59.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.104+0000 7f7b531ff700 1 -- 172.21.15.27:0/2942277539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4c0c7ce0 con 0x7f7b4c0cc370 2024-01-11T09:45:59.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.105+0000 7f7b531ff700 1 -- 172.21.15.27:0/2942277539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b4c15f580 con 0x7f7b4c0cc370 2024-01-11T09:45:59.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.106+0000 7f7b48ff9700 1 -- 172.21.15.27:0/2942277539 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 133) v1 ==== 85003+0+0 (secure 0 0 0) 0x7f7b4401a3a0 con 0x7f7b4c0cc370 2024-01-11T09:45:59.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.107+0000 7f7b48ff9700 1 --2- 172.21.15.27:0/2942277539 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7b30067200 0x7f7b300696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:59.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.107+0000 7f7b48ff9700 1 -- 172.21.15.27:0/2942277539 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f7b44084fa0 con 0x7f7b4c0cc370 2024-01-11T09:45:59.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.107+0000 7f7b50f9b700 1 -- 172.21.15.27:0/2942277539 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7b30067200 msgr2=0x7f7b300696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:59.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.107+0000 7f7b50f9b700 1 --2- 172.21.15.27:0/2942277539 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7b30067200 0x7f7b300696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:45:59.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.112+0000 7f7b48ff9700 1 -- 172.21.15.27:0/2942277539 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7b4c15f580 con 0x7f7b4c0cc370 2024-01-11T09:45:59.171 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:59.178+0000 7f89e6eae1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:45:59.286 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:59.294+0000 7feb4febe1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:45:59.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.308+0000 7f7b50f9b700 1 -- 172.21.15.27:0/2942277539 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7b30067200 msgr2=0x7f7b300696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:59.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.308+0000 7f7b50f9b700 1 --2- 172.21.15.27:0/2942277539 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7b30067200 0x7f7b300696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:45:59.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.336+0000 7f7b531ff700 1 -- 172.21.15.27:0/2942277539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b4c15f580 con 0x7f7b4c0cc370 2024-01-11T09:45:59.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.338+0000 7f7b48ff9700 1 -- 172.21.15.27:0/2942277539 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v219) v1 ==== 74+0+19093 (secure 0 0 0) 0x7f7b4c15f580 con 0x7f7b4c0cc370 2024-01-11T09:45:59.330 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:45:59.331 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":219,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:45:52.192269+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:45:21.455328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-01-11T09:45:22.634298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-01-11T09:45:35.631727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-01-11T09:45:36.452195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/2192171652":"2024-01-12T09:45:52.192235+0000","172.21.15.77:6837/59776":"2024-01-12T09:45:52.192235+0000","172.21.15.77:6836/59776":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/2654175097":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/652362578":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/2439942664":"2024-01-12T09:45:52.192235+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:45:59.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.340+0000 7f7b531ff700 1 -- 172.21.15.27:0/2942277539 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7b30067200 msgr2=0x7f7b300696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:45:59.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.340+0000 7f7b531ff700 1 --2- 172.21.15.27:0/2942277539 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7b30067200 0x7f7b300696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.340+0000 7f7b531ff700 1 -- 172.21.15.27:0/2942277539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b4c0cc370 msgr2=0x7f7b4c0c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:59.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.340+0000 7f7b531ff700 1 --2- 172.21.15.27:0/2942277539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b4c0cc370 0x7f7b4c0c6c00 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f7b4400c840 tx=0x7f7b44001670 comp rx=0 tx=0).stop 2024-01-11T09:45:59.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.340+0000 7f7b531ff700 1 -- 172.21.15.27:0/2942277539 shutdown_connections 2024-01-11T09:45:59.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.340+0000 7f7b531ff700 1 --2- 172.21.15.27:0/2942277539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b4c0cc370 0x7f7b4c0c6c00 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.340+0000 7f7b531ff700 1 --2- 172.21.15.27:0/2942277539 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b4c06e770 0x7f7b4c0c64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.340+0000 7f7b531ff700 1 --2- 172.21.15.27:0/2942277539 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7b30067200 0x7f7b300696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.340+0000 7f7b531ff700 1 --2- 172.21.15.27:0/2942277539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b4c0cbfa0 0x7f7b4c145db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.340+0000 7f7b531ff700 1 -- 172.21.15.27:0/2942277539 >> 172.21.15.27:0/2942277539 conn(0x7f7b4c06b200 msgr2=0x7f7b4c156a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:59.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.340+0000 7f7b531ff700 1 -- 172.21.15.27:0/2942277539 shutdown_connections 2024-01-11T09:45:59.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.340+0000 7f7b531ff700 1 -- 172.21.15.27:0/2942277539 wait complete. 2024-01-11T09:45:59.341 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:59.349+0000 7f89e6eae1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:45:59.351 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T09:45:21.455328+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '213', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T09:45:22.634298+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '214', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-01-11T09:45:35.631727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '217', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2024-01-11T09:45:36.452195+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '218', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}}] 2024-01-11T09:45:59.352 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T09:45:59.476 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:59.483+0000 7feb4febe1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:45:59.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.566+0000 7f89b49ff700 1 Processor -- start 2024-01-11T09:45:59.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.567+0000 7f89b49ff700 1 -- start start 2024-01-11T09:45:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.568+0000 7f89b49ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89b013ff80 0x7f89b015a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.568+0000 7f89b49ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f89b015a560 0x7f89b00c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.568+0000 7f89b49ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b00c6b30 con 0x7f89b013fbb0 2024-01-11T09:45:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.568+0000 7f89b49ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89b00c6800 con 0x7f89b013ff80 2024-01-11T09:45:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.568+0000 7f89b49ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89b015a960 con 0x7f89b015a560 2024-01-11T09:45:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.568+0000 7f89add9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89b013ff80 0x7f89b015a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.568+0000 7f89add9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89b013ff80 0x7f89b015a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47740/0 (socket says 172.21.15.27:47740) 2024-01-11T09:45:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.568+0000 7f89add9b700 1 -- 172.21.15.27:0/2453860075 learned_addr learned my addr 172.21.15.27:0/2453860075 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.568+0000 7f89aed9d700 1 --2- 172.21.15.27:0/2453860075 >> v2:172.21.15.77:3300/0 conn(0x7f89b015a560 0x7f89b00c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.569+0000 7f89add9b700 1 -- 172.21.15.27:0/2453860075 >> v2:172.21.15.77:3300/0 conn(0x7f89b015a560 msgr2=0x7f89b00c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:45:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.569+0000 7f89add9b700 1 --2- 172.21.15.27:0/2453860075 >> v2:172.21.15.77:3300/0 conn(0x7f89b015a560 0x7f89b00c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.569+0000 7f89add9b700 1 -- 172.21.15.27:0/2453860075 >> v1:172.21.15.77:6789/0 conn(0x7f89b013fbb0 legacy=0x7f89b00ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.569+0000 7f89add9b700 1 -- 172.21.15.27:0/2453860075 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b00cb240 con 0x7f89b013ff80 2024-01-11T09:45:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.569+0000 7f89add9b700 1 --2- 172.21.15.27:0/2453860075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b013ff80 0x7f89b015a020 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f89b006fe10 tx=0x7f89a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=831b9964857b1694 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.569+0000 7f89ad59a700 1 -- 172.21.15.27:0/2453860075 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a4009e70 con 0x7f89b013ff80 2024-01-11T09:45:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.569+0000 7f89ad59a700 1 -- 172.21.15.27:0/2453860075 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a400b860 con 0x7f89b013ff80 2024-01-11T09:45:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.569+0000 7f89ad59a700 1 -- 172.21.15.27:0/2453860075 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a400bc00 con 0x7f89b013ff80 2024-01-11T09:45:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.570+0000 7f89b49ff700 1 -- 172.21.15.27:0/2453860075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b013ff80 msgr2=0x7f89b015a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.570+0000 7f89b49ff700 1 --2- 172.21.15.27:0/2453860075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b013ff80 0x7f89b015a020 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f89b006fe10 tx=0x7f89a4009250 comp rx=0 tx=0).stop 2024-01-11T09:45:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.570+0000 7f89b49ff700 1 -- 172.21.15.27:0/2453860075 shutdown_connections 2024-01-11T09:45:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.570+0000 7f89b49ff700 1 --2- 172.21.15.27:0/2453860075 >> v2:172.21.15.77:3300/0 conn(0x7f89b015a560 0x7f89b00c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.570+0000 7f89b49ff700 1 --2- 172.21.15.27:0/2453860075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b013ff80 0x7f89b015a020 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.570+0000 7f89b49ff700 1 -- 172.21.15.27:0/2453860075 >> 172.21.15.27:0/2453860075 conn(0x7f89b00680a0 msgr2=0x7f89b00cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.570+0000 7f89b49ff700 1 -- 172.21.15.27:0/2453860075 shutdown_connections 2024-01-11T09:45:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.570+0000 7f89b49ff700 1 -- 172.21.15.27:0/2453860075 wait complete. 2024-01-11T09:45:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.572+0000 7f89b49ff700 1 Processor -- start 2024-01-11T09:45:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.572+0000 7f89b49ff700 1 -- start start 2024-01-11T09:45:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.573+0000 7f89b49ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f89b013fbb0 0x7f89b0155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.573+0000 7f89b49ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b013ff80 0x7f89b00af210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.573+0000 7f89b49ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89b015a560 0x7f89b013c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.573+0000 7f89b49ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b0166d30 con 0x7f89b013ff80 2024-01-11T09:45:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.573+0000 7f89b49ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b0157b60 con 0x7f89b015a560 2024-01-11T09:45:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.573+0000 7f89b49ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b00b0300 con 0x7f89b013fbb0 2024-01-11T09:45:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.573+0000 7f89add9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b013ff80 0x7f89b00af210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.573+0000 7f89add9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b013ff80 0x7f89b00af210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47750/0 (socket says 172.21.15.27:47750) 2024-01-11T09:45:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.573+0000 7f89add9b700 1 -- 172.21.15.27:0/553671898 learned_addr learned my addr 172.21.15.27:0/553671898 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:45:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.573+0000 7f89aed9d700 1 --2- 172.21.15.27:0/553671898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89b015a560 0x7f89b013c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.573+0000 7f89ae59c700 1 --2- 172.21.15.27:0/553671898 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f89b013fbb0 0x7f89b0155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:45:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.574+0000 7f89add9b700 1 -- 172.21.15.27:0/553671898 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f89b013fbb0 msgr2=0x7f89b0155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.574+0000 7f89add9b700 1 --2- 172.21.15.27:0/553671898 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f89b013fbb0 0x7f89b0155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.574+0000 7f89add9b700 1 -- 172.21.15.27:0/553671898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89b015a560 msgr2=0x7f89b013c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.574+0000 7f89add9b700 1 --2- 172.21.15.27:0/553671898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89b015a560 0x7f89b013c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.574+0000 7f89add9b700 1 -- 172.21.15.27:0/553671898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a40094f0 con 0x7f89b013ff80 2024-01-11T09:45:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.574+0000 7f89add9b700 1 --2- 172.21.15.27:0/553671898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b013ff80 0x7f89b00af210 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f89a4009290 tx=0x7f89a4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:45:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.574+0000 7f899e7fc700 1 -- 172.21.15.27:0/553671898 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a401e070 con 0x7f89b013ff80 2024-01-11T09:45:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.574+0000 7f899e7fc700 1 -- 172.21.15.27:0/553671898 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a400b9d0 con 0x7f89b013ff80 2024-01-11T09:45:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.574+0000 7f89b49ff700 1 -- 172.21.15.27:0/553671898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b0156250 con 0x7f89b013ff80 2024-01-11T09:45:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.574+0000 7f899e7fc700 1 -- 172.21.15.27:0/553671898 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a4010470 con 0x7f89b013ff80 2024-01-11T09:45:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.574+0000 7f89b49ff700 1 -- 172.21.15.27:0/553671898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b00afd40 con 0x7f89b013ff80 2024-01-11T09:45:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.576+0000 7f89b49ff700 1 -- 172.21.15.27:0/553671898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b0157d10 con 0x7f89b013ff80 2024-01-11T09:45:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.577+0000 7f899e7fc700 1 -- 172.21.15.27:0/553671898 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 133) v1 ==== 85003+0+0 (secure 0 0 0) 0x7f89a4010820 con 0x7f89b013ff80 2024-01-11T09:45:59.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.578+0000 7f899e7fc700 1 --2- 172.21.15.27:0/553671898 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8990067200 0x7f89900696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:45:59.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.578+0000 7f899e7fc700 1 -- 172.21.15.27:0/553671898 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f89a4014070 con 0x7f89b013ff80 2024-01-11T09:45:59.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.578+0000 7f89ae59c700 1 -- 172.21.15.27:0/553671898 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8990067200 msgr2=0x7f89900696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.578+0000 7f89ae59c700 1 --2- 172.21.15.27:0/553671898 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8990067200 0x7f89900696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:45:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.582+0000 7f899e7fc700 1 -- 172.21.15.27:0/553671898 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f89b0157d10 con 0x7f89b013ff80 2024-01-11T09:45:59.756 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:45:59.764+0000 7f89e6eae1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:45:59.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.779+0000 7f89ae59c700 1 -- 172.21.15.27:0/553671898 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8990067200 msgr2=0x7f89900696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:45:59.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.779+0000 7f89ae59c700 1 --2- 172.21.15.27:0/553671898 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8990067200 0x7f89900696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:45:59.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.786+0000 7f89b49ff700 1 -- 172.21.15.27:0/553671898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f89b0157cd0 con 0x7f89b013ff80 2024-01-11T09:45:59.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.787+0000 7f899e7fc700 1 -- 172.21.15.27:0/553671898 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v219) v1 ==== 93+0+10 (secure 0 0 0) 0x7f89a4027030 con 0x7f89b013ff80 2024-01-11T09:45:59.779 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T09:45:59.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.789+0000 7f89b49ff700 1 -- 172.21.15.27:0/553671898 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8990067200 msgr2=0x7f89900696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:45:59.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.789+0000 7f89b49ff700 1 --2- 172.21.15.27:0/553671898 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8990067200 0x7f89900696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.789+0000 7f89b49ff700 1 -- 172.21.15.27:0/553671898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b013ff80 msgr2=0x7f89b00af210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:45:59.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.789+0000 7f89b49ff700 1 --2- 172.21.15.27:0/553671898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b013ff80 0x7f89b00af210 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f89a4009290 tx=0x7f89a4019000 comp rx=0 tx=0).stop 2024-01-11T09:45:59.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.789+0000 7f89b49ff700 1 -- 172.21.15.27:0/553671898 shutdown_connections 2024-01-11T09:45:59.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.789+0000 7f89b49ff700 1 --2- 172.21.15.27:0/553671898 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8990067200 0x7f89900696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.789+0000 7f89b49ff700 1 --2- 172.21.15.27:0/553671898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89b015a560 0x7f89b013c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.789+0000 7f89b49ff700 1 --2- 172.21.15.27:0/553671898 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f89b013fbb0 0x7f89b0155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.789+0000 7f89b49ff700 1 --2- 172.21.15.27:0/553671898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89b013ff80 0x7f89b00af210 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:45:59.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.789+0000 7f89b49ff700 1 -- 172.21.15.27:0/553671898 >> 172.21.15.27:0/553671898 conn(0x7f89b00680a0 msgr2=0x7f89b00c8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:45:59.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.790+0000 7f89b49ff700 1 -- 172.21.15.27:0/553671898 shutdown_connections 2024-01-11T09:45:59.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:45:59.790+0000 7f89b49ff700 1 -- 172.21.15.27:0/553671898 wait complete. 2024-01-11T09:45:59.801 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T09:45:59.934 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:45:59.942+0000 7feb4febe1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:46:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.022+0000 7f2babfff700 1 Processor -- start 2024-01-11T09:46:00.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.022+0000 7f2babfff700 1 -- start start 2024-01-11T09:46:00.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.023+0000 7f2babfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2ba40cd220 0x7f2ba415d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:00.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.023+0000 7f2babfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba415d860 0x7f2ba41591e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:00.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.023+0000 7f2babfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2ba40cd5f0 0x7f2ba4159720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:00.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.023+0000 7f2babfff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ba415aef0 con 0x7f2ba40cd5f0 2024-01-11T09:46:00.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.023+0000 7f2babfff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ba40cee00 con 0x7f2ba40cd220 2024-01-11T09:46:00.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.023+0000 7f2babfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba415dd70 con 0x7f2ba415d860 2024-01-11T09:46:00.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.024+0000 7f2baa59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2ba40cd5f0 0x7f2ba4159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:00.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.024+0000 7f2ba959a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba415d860 0x7f2ba41591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:00.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.024+0000 7f2ba9d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2ba40cd220 0x7f2ba415d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:00.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.024+0000 7f2ba9d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2ba40cd220 0x7f2ba415d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:46622/0 (socket says 172.21.15.27:46622) 2024-01-11T09:46:00.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.024+0000 7f2baa59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2ba40cd5f0 0x7f2ba4159720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47764/0 (socket says 172.21.15.27:47764) 2024-01-11T09:46:00.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.024+0000 7f2ba959a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba415d860 0x7f2ba41591e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:48308/0 (socket says 172.21.15.27:48308) 2024-01-11T09:46:00.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.024+0000 7f2ba9d9b700 1 -- 172.21.15.27:0/3819947152 learned_addr learned my addr 172.21.15.27:0/3819947152 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:00.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.024+0000 7f2ba9d9b700 1 -- 172.21.15.27:0/3819947152 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba415d860 msgr2=0x7f2ba41591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:00.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.024+0000 7f2ba9d9b700 1 --2- 172.21.15.27:0/3819947152 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba415d860 0x7f2ba41591e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.024+0000 7f2ba9d9b700 1 -- 172.21.15.27:0/3819947152 >> v2:172.21.15.27:3300/0 conn(0x7f2ba40cd5f0 msgr2=0x7f2ba4159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.024+0000 7f2ba9d9b700 1 --2- 172.21.15.27:0/3819947152 >> v2:172.21.15.27:3300/0 conn(0x7f2ba40cd5f0 0x7f2ba4159720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.024+0000 7f2ba9d9b700 1 -- 172.21.15.27:0/3819947152 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba415e050 con 0x7f2ba40cd220 2024-01-11T09:46:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.024+0000 7f2ba959a700 1 --2- 172.21.15.27:0/3819947152 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba415d860 0x7f2ba41591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.024+0000 7f2ba9d9b700 1 --2- 172.21.15.27:0/3819947152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba40cd220 0x7f2ba415d320 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40ce150 tx=0x7f2ba0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f64762e0b83f86cc server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.025+0000 7f2ba8d99700 1 -- 172.21.15.27:0/3819947152 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba000b640 con 0x7f2ba40cd220 2024-01-11T09:46:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.025+0000 7f2ba8d99700 1 -- 172.21.15.27:0/3819947152 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba000b7a0 con 0x7f2ba40cd220 2024-01-11T09:46:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.025+0000 7f2ba8d99700 1 -- 172.21.15.27:0/3819947152 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba000bb60 con 0x7f2ba40cd220 2024-01-11T09:46:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.025+0000 7f2babfff700 1 -- 172.21.15.27:0/3819947152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba40cd220 msgr2=0x7f2ba415d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.025+0000 7f2babfff700 1 --2- 172.21.15.27:0/3819947152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba40cd220 0x7f2ba415d320 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40ce150 tx=0x7f2ba0009250 comp rx=0 tx=0).stop 2024-01-11T09:46:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.026+0000 7f2babfff700 1 -- 172.21.15.27:0/3819947152 shutdown_connections 2024-01-11T09:46:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.026+0000 7f2babfff700 1 --2- 172.21.15.27:0/3819947152 >> v2:172.21.15.27:3300/0 conn(0x7f2ba40cd5f0 0x7f2ba4159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.026+0000 7f2babfff700 1 --2- 172.21.15.27:0/3819947152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba40cd220 0x7f2ba415d320 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.026+0000 7f2babfff700 1 --2- 172.21.15.27:0/3819947152 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba415d860 0x7f2ba41591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.026+0000 7f2babfff700 1 -- 172.21.15.27:0/3819947152 >> 172.21.15.27:0/3819947152 conn(0x7f2ba406ed40 msgr2=0x7f2ba406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:00.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.026+0000 7f2babfff700 1 -- 172.21.15.27:0/3819947152 shutdown_connections 2024-01-11T09:46:00.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.026+0000 7f2babfff700 1 -- 172.21.15.27:0/3819947152 wait complete. 2024-01-11T09:46:00.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.027+0000 7f2babfff700 1 Processor -- start 2024-01-11T09:46:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.027+0000 7f2babfff700 1 -- start start 2024-01-11T09:46:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.028+0000 7f2babfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba40cd220 0x7f2ba416df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.028+0000 7f2babfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40cd5f0 0x7f2ba40c82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.028+0000 7f2babfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba415d860 0x7f2ba4169f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.028+0000 7f2babfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba416e440 con 0x7f2ba40cd5f0 2024-01-11T09:46:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.028+0000 7f2babfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba416a960 con 0x7f2ba415d860 2024-01-11T09:46:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.028+0000 7f2babfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba416ac40 con 0x7f2ba40cd220 2024-01-11T09:46:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.028+0000 7f2ba959a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40cd5f0 0x7f2ba40c82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.028+0000 7f2ba9d9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba40cd220 0x7f2ba416df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.029+0000 7f2baa59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba415d860 0x7f2ba4169f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.029+0000 7f2ba9d9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba40cd220 0x7f2ba416df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:48324/0 (socket says 172.21.15.27:48324) 2024-01-11T09:46:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.029+0000 7f2ba9d9b700 1 -- 172.21.15.27:0/1047318547 learned_addr learned my addr 172.21.15.27:0/1047318547 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.029+0000 7f2ba959a700 1 -- 172.21.15.27:0/1047318547 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba40cd220 msgr2=0x7f2ba416df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.029+0000 7f2ba959a700 1 --2- 172.21.15.27:0/1047318547 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba40cd220 0x7f2ba416df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.029+0000 7f2ba959a700 1 -- 172.21.15.27:0/1047318547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba415d860 msgr2=0x7f2ba4169f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.029+0000 7f2ba959a700 1 --2- 172.21.15.27:0/1047318547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba415d860 0x7f2ba4169f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.029+0000 7f2ba959a700 1 -- 172.21.15.27:0/1047318547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b94009420 con 0x7f2ba40cd5f0 2024-01-11T09:46:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.029+0000 7f2ba9d9b700 1 --2- 172.21.15.27:0/1047318547 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba40cd220 0x7f2ba416df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.029+0000 7f2baa59c700 1 --2- 172.21.15.27:0/1047318547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba415d860 0x7f2ba4169f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.029+0000 7f2ba959a700 1 --2- 172.21.15.27:0/1047318547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40cd5f0 0x7f2ba40c82c0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f2b9400ea00 tx=0x7f2b940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.030+0000 7f2b99ffb700 1 -- 172.21.15.27:0/1047318547 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9400c6a0 con 0x7f2ba40cd5f0 2024-01-11T09:46:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.030+0000 7f2b99ffb700 1 -- 172.21.15.27:0/1047318547 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9400c800 con 0x7f2ba40cd5f0 2024-01-11T09:46:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.030+0000 7f2babfff700 1 -- 172.21.15.27:0/1047318547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba00094f0 con 0x7f2ba40cd5f0 2024-01-11T09:46:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.030+0000 7f2b99ffb700 1 -- 172.21.15.27:0/1047318547 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9400cba0 con 0x7f2ba40cd5f0 2024-01-11T09:46:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.030+0000 7f2babfff700 1 -- 172.21.15.27:0/1047318547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba416d9e0 con 0x7f2ba40cd5f0 2024-01-11T09:46:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.031+0000 7f2babfff700 1 -- 172.21.15.27:0/1047318547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba416ae60 con 0x7f2ba40cd5f0 2024-01-11T09:46:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.032+0000 7f2b99ffb700 1 -- 172.21.15.27:0/1047318547 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 133) v1 ==== 85003+0+0 (secure 0 0 0) 0x7f2b94013470 con 0x7f2ba40cd5f0 2024-01-11T09:46:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.033+0000 7f2b99ffb700 1 --2- 172.21.15.27:0/1047318547 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2b88067200 0x7f2b880696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.033+0000 7f2b99ffb700 1 -- 172.21.15.27:0/1047318547 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f2b94085000 con 0x7f2ba40cd5f0 2024-01-11T09:46:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.033+0000 7f2ba9d9b700 1 -- 172.21.15.27:0/1047318547 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2b88067200 msgr2=0x7f2b880696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.033+0000 7f2ba9d9b700 1 --2- 172.21.15.27:0/1047318547 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2b88067200 0x7f2b880696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:46:00.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.037+0000 7f2b99ffb700 1 -- 172.21.15.27:0/1047318547 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2b94051a10 con 0x7f2ba40cd5f0 2024-01-11T09:46:00.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.234+0000 7f2ba9d9b700 1 -- 172.21.15.27:0/1047318547 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2b88067200 msgr2=0x7f2b880696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:00.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.234+0000 7f2ba9d9b700 1 --2- 172.21.15.27:0/1047318547 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2b88067200 0x7f2b880696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:46:00.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.240+0000 7f2babfff700 1 -- 172.21.15.27:0/1047318547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2ba416dc00 con 0x7f2ba40cd5f0 2024-01-11T09:46:00.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.240+0000 7f2b99ffb700 1 -- 172.21.15.27:0/1047318547 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v219) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2b94055030 con 0x7f2ba40cd5f0 2024-01-11T09:46:00.233 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T09:46:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.243+0000 7f2babfff700 1 -- 172.21.15.27:0/1047318547 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2b88067200 msgr2=0x7f2b880696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:46:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.243+0000 7f2babfff700 1 --2- 172.21.15.27:0/1047318547 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2b88067200 0x7f2b880696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.243+0000 7f2babfff700 1 -- 172.21.15.27:0/1047318547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40cd5f0 msgr2=0x7f2ba40c82c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.243+0000 7f2babfff700 1 --2- 172.21.15.27:0/1047318547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40cd5f0 0x7f2ba40c82c0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f2b9400ea00 tx=0x7f2b940091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.243+0000 7f2babfff700 1 -- 172.21.15.27:0/1047318547 shutdown_connections 2024-01-11T09:46:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.243+0000 7f2babfff700 1 --2- 172.21.15.27:0/1047318547 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2b88067200 0x7f2b880696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.243+0000 7f2babfff700 1 --2- 172.21.15.27:0/1047318547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ba415d860 0x7f2ba4169f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.243+0000 7f2babfff700 1 --2- 172.21.15.27:0/1047318547 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2ba40cd220 0x7f2ba416df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.243+0000 7f2babfff700 1 --2- 172.21.15.27:0/1047318547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba40cd5f0 0x7f2ba40c82c0 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.243+0000 7f2babfff700 1 -- 172.21.15.27:0/1047318547 >> 172.21.15.27:0/1047318547 conn(0x7f2ba406ed40 msgr2=0x7f2ba4152b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.243+0000 7f2babfff700 1 -- 172.21.15.27:0/1047318547 shutdown_connections 2024-01-11T09:46:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.243+0000 7f2babfff700 1 -- 172.21.15.27:0/1047318547 wait complete. 2024-01-11T09:46:00.256 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T09:46:00.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.467+0000 7f3b659ff700 1 Processor -- start 2024-01-11T09:46:00.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.468+0000 7f3b659ff700 1 -- start start 2024-01-11T09:46:00.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.469+0000 7f3b659ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3b60142dc0 0x7f3b600cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:00.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.469+0000 7f3b659ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3b600cb2e0 0x7f3b600cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:00.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.469+0000 7f3b659ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b600ccc70 con 0x7f3b60143190 2024-01-11T09:46:00.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.469+0000 7f3b659ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6015df30 con 0x7f3b600cb2e0 2024-01-11T09:46:00.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.469+0000 7f3b659ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6015d320 con 0x7f3b60142dc0 2024-01-11T09:46:00.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.469+0000 7f3b5f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3b600cb2e0 0x7f3b600cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:00.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.469+0000 7f3b5f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3b600cb2e0 0x7f3b600cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47780/0 (socket says 172.21.15.27:47780) 2024-01-11T09:46:00.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.469+0000 7f3b5f7fe700 1 -- 172.21.15.27:0/709759494 learned_addr learned my addr 172.21.15.27:0/709759494 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:00.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.469+0000 7f3b5effd700 1 --2- 172.21.15.27:0/709759494 >> v2:172.21.15.77:3300/0 conn(0x7f3b60142dc0 0x7f3b600cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:00.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.470+0000 7f3b5f7fe700 1 -- 172.21.15.27:0/709759494 >> v2:172.21.15.77:3300/0 conn(0x7f3b60142dc0 msgr2=0x7f3b600cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:00.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.470+0000 7f3b5f7fe700 1 --2- 172.21.15.27:0/709759494 >> v2:172.21.15.77:3300/0 conn(0x7f3b60142dc0 0x7f3b600cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.470+0000 7f3b5f7fe700 1 -- 172.21.15.27:0/709759494 >> v1:172.21.15.77:6789/0 conn(0x7f3b60143190 legacy=0x7f3b600cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:00.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.470+0000 7f3b5f7fe700 1 -- 172.21.15.27:0/709759494 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b600cb700 con 0x7f3b600cb2e0 2024-01-11T09:46:00.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.470+0000 7f3b5effd700 1 --2- 172.21.15.27:0/709759494 >> v2:172.21.15.77:3300/0 conn(0x7f3b60142dc0 0x7f3b600cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:00.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.470+0000 7f3b5f7fe700 1 --2- 172.21.15.27:0/709759494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b600cb2e0 0x7f3b600cc730 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f3b6006f4d0 tx=0x7f3b58009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee2b0fb3617f2ad3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:00.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.470+0000 7f3b5dffb700 1 -- 172.21.15.27:0/709759494 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5800e070 con 0x7f3b600cb2e0 2024-01-11T09:46:00.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.470+0000 7f3b5dffb700 1 -- 172.21.15.27:0/709759494 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5800ba50 con 0x7f3b600cb2e0 2024-01-11T09:46:00.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.470+0000 7f3b5dffb700 1 -- 172.21.15.27:0/709759494 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b580092e0 con 0x7f3b600cb2e0 2024-01-11T09:46:00.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.471+0000 7f3b659ff700 1 -- 172.21.15.27:0/709759494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b600cb2e0 msgr2=0x7f3b600cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:00.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.471+0000 7f3b659ff700 1 --2- 172.21.15.27:0/709759494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b600cb2e0 0x7f3b600cc730 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f3b6006f4d0 tx=0x7f3b58009840 comp rx=0 tx=0).stop 2024-01-11T09:46:00.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.471+0000 7f3b659ff700 1 -- 172.21.15.27:0/709759494 shutdown_connections 2024-01-11T09:46:00.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.471+0000 7f3b659ff700 1 --2- 172.21.15.27:0/709759494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b600cb2e0 0x7f3b600cc730 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.471+0000 7f3b659ff700 1 --2- 172.21.15.27:0/709759494 >> v2:172.21.15.77:3300/0 conn(0x7f3b60142dc0 0x7f3b600cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.471+0000 7f3b659ff700 1 -- 172.21.15.27:0/709759494 >> 172.21.15.27:0/709759494 conn(0x7f3b60063ca0 msgr2=0x7f3b6006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:00.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.471+0000 7f3b659ff700 1 -- 172.21.15.27:0/709759494 shutdown_connections 2024-01-11T09:46:00.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.471+0000 7f3b659ff700 1 -- 172.21.15.27:0/709759494 wait complete. 2024-01-11T09:46:00.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.473+0000 7f3b659ff700 1 Processor -- start 2024-01-11T09:46:00.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.473+0000 7f3b659ff700 1 -- start start 2024-01-11T09:46:00.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.475+0000 7f3b659ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b600cb2e0 0x7f3b600c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:00.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.475+0000 7f3b659ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b60142dc0 0x7f3b6016afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:00.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.475+0000 7f3b659ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b60143190 0x7f3b6016b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:00.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.475+0000 7f3b659ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b600c5530 con 0x7f3b60143190 2024-01-11T09:46:00.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.475+0000 7f3b659ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b600c5810 con 0x7f3b60142dc0 2024-01-11T09:46:00.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.475+0000 7f3b659ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b600c5af0 con 0x7f3b600cb2e0 2024-01-11T09:46:00.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.475+0000 7f3b5f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b60143190 0x7f3b6016b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:00.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.475+0000 7f3b5f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b60143190 0x7f3b6016b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47794/0 (socket says 172.21.15.27:47794) 2024-01-11T09:46:00.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.475+0000 7f3b5f7fe700 1 -- 172.21.15.27:0/3259379221 learned_addr learned my addr 172.21.15.27:0/3259379221 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:00.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.475+0000 7f3b5e7fc700 1 --2- 172.21.15.27:0/3259379221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b60142dc0 0x7f3b6016afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:00.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.475+0000 7f3b5effd700 1 --2- 172.21.15.27:0/3259379221 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b600cb2e0 0x7f3b600c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:00.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.475+0000 7f3b5f7fe700 1 -- 172.21.15.27:0/3259379221 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b600cb2e0 msgr2=0x7f3b600c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:00.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.475+0000 7f3b5f7fe700 1 --2- 172.21.15.27:0/3259379221 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b600cb2e0 0x7f3b600c65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.475+0000 7f3b5f7fe700 1 -- 172.21.15.27:0/3259379221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b60142dc0 msgr2=0x7f3b6016afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:00.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.476+0000 7f3b5f7fe700 1 --2- 172.21.15.27:0/3259379221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b60142dc0 0x7f3b6016afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.476+0000 7f3b5f7fe700 1 -- 172.21.15.27:0/3259379221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b58009ae0 con 0x7f3b60143190 2024-01-11T09:46:00.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.476+0000 7f3b5f7fe700 1 --2- 172.21.15.27:0/3259379221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b60143190 0x7f3b6016b520 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f3b58009880 tx=0x7f3b58018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:00.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.476+0000 7f3b46ffd700 1 -- 172.21.15.27:0/3259379221 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5800be60 con 0x7f3b60143190 2024-01-11T09:46:00.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.476+0000 7f3b46ffd700 1 -- 172.21.15.27:0/3259379221 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b58008d20 con 0x7f3b60143190 2024-01-11T09:46:00.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.476+0000 7f3b659ff700 1 -- 172.21.15.27:0/3259379221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b600c5dd0 con 0x7f3b60143190 2024-01-11T09:46:00.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.476+0000 7f3b46ffd700 1 -- 172.21.15.27:0/3259379221 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b580266b0 con 0x7f3b60143190 2024-01-11T09:46:00.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.476+0000 7f3b659ff700 1 -- 172.21.15.27:0/3259379221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b600c6b10 con 0x7f3b60143190 2024-01-11T09:46:00.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.478+0000 7f3b659ff700 1 -- 172.21.15.27:0/3259379221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b600c6df0 con 0x7f3b60143190 2024-01-11T09:46:00.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.478+0000 7f3b46ffd700 1 -- 172.21.15.27:0/3259379221 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 133) v1 ==== 85003+0+0 (secure 0 0 0) 0x7f3b58003040 con 0x7f3b60143190 2024-01-11T09:46:00.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.479+0000 7f3b46ffd700 1 --2- 172.21.15.27:0/3259379221 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3b4c067200 0x7f3b4c0696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:00.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.479+0000 7f3b46ffd700 1 -- 172.21.15.27:0/3259379221 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f3b580868a0 con 0x7f3b60143190 2024-01-11T09:46:00.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.479+0000 7f3b5effd700 1 -- 172.21.15.27:0/3259379221 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3b4c067200 msgr2=0x7f3b4c0696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:00.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.479+0000 7f3b5effd700 1 --2- 172.21.15.27:0/3259379221 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3b4c067200 0x7f3b4c0696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:46:00.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.483+0000 7f3b46ffd700 1 -- 172.21.15.27:0/3259379221 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3b58053290 con 0x7f3b60143190 2024-01-11T09:46:00.537 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:00.545+0000 7f89e6eae1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:46:00.625 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:00.633+0000 7f89e6eae1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:46:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.680+0000 7f3b5effd700 1 -- 172.21.15.27:0/3259379221 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3b4c067200 msgr2=0x7f3b4c0696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.680+0000 7f3b5effd700 1 --2- 172.21.15.27:0/3259379221 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3b4c067200 0x7f3b4c0696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:46:00.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.687+0000 7f3b659ff700 1 -- 172.21.15.27:0/3259379221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3b6004c3c0 con 0x7f3b60143190 2024-01-11T09:46:00.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.687+0000 7f3b46ffd700 1 -- 172.21.15.27:0/3259379221 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v219) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3b580568b0 con 0x7f3b60143190 2024-01-11T09:46:00.680 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T09:46:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.690+0000 7f3b659ff700 1 -- 172.21.15.27:0/3259379221 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3b4c067200 msgr2=0x7f3b4c0696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:46:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.690+0000 7f3b659ff700 1 --2- 172.21.15.27:0/3259379221 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3b4c067200 0x7f3b4c0696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.690+0000 7f3b659ff700 1 -- 172.21.15.27:0/3259379221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b60143190 msgr2=0x7f3b6016b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.690+0000 7f3b659ff700 1 --2- 172.21.15.27:0/3259379221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b60143190 0x7f3b6016b520 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f3b58009880 tx=0x7f3b58018000 comp rx=0 tx=0).stop 2024-01-11T09:46:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.690+0000 7f3b659ff700 1 -- 172.21.15.27:0/3259379221 shutdown_connections 2024-01-11T09:46:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.690+0000 7f3b659ff700 1 --2- 172.21.15.27:0/3259379221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b60143190 0x7f3b6016b520 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.690+0000 7f3b659ff700 1 --2- 172.21.15.27:0/3259379221 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3b600cb2e0 0x7f3b600c65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.690+0000 7f3b659ff700 1 --2- 172.21.15.27:0/3259379221 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3b4c067200 0x7f3b4c0696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.690+0000 7f3b659ff700 1 --2- 172.21.15.27:0/3259379221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b60142dc0 0x7f3b6016afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.690+0000 7f3b659ff700 1 -- 172.21.15.27:0/3259379221 >> 172.21.15.27:0/3259379221 conn(0x7f3b60063ca0 msgr2=0x7f3b60141bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.690+0000 7f3b659ff700 1 -- 172.21.15.27:0/3259379221 shutdown_connections 2024-01-11T09:46:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.690+0000 7f3b659ff700 1 -- 172.21.15.27:0/3259379221 wait complete. 2024-01-11T09:46:00.701 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-01-11T09:46:00.780 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:00.788+0000 7f89e6eae1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:46:00.786 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:00.793+0000 7feb4febe1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:46:00.867 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:00.875+0000 7f89e6eae1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:46:00.882 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:00.890+0000 7feb4febe1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:46:00.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.912+0000 7fbf165ff700 1 Processor -- start 2024-01-11T09:46:00.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.912+0000 7fbf165ff700 1 -- start start 2024-01-11T09:46:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.913+0000 7fbf165ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbf1015a1b0 0x7fbf1015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.914+0000 7fbf165ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf1015ae70 0x7fbf1015b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.914+0000 7fbf165ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf100cf850 con 0x7fbf100ccd70 2024-01-11T09:46:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.914+0000 7fbf165ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf100cec40 con 0x7fbf1015a1b0 2024-01-11T09:46:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.914+0000 7fbf165ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100cef20 con 0x7fbf1015ae70 2024-01-11T09:46:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.914+0000 7fbf0f7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbf100ccd70 0x7fbf100cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42188/0 (socket says 172.21.15.27:42188) 2024-01-11T09:46:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.914+0000 7fbf0f7fe700 1 -- 172.21.15.27:0/3097142350 learned_addr learned my addr 172.21.15.27:0/3097142350 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.914+0000 7fbf14b9c700 1 --2- 172.21.15.27:0/3097142350 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf1015ae70 0x7fbf1015b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:00.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.914+0000 7fbf0ffff700 1 --2- 172.21.15.27:0/3097142350 >> v2:172.21.15.77:3300/0 conn(0x7fbf1015a1b0 0x7fbf1015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:00.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.914+0000 7fbf0effd700 1 -- 172.21.15.27:0/3097142350 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140635159 0 0) 0x7fbf100cf850 con 0x7fbf100ccd70 2024-01-11T09:46:00.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.914+0000 7fbf0effd700 1 -- 172.21.15.27:0/3097142350 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbef8003570 con 0x7fbf100ccd70 2024-01-11T09:46:00.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.915+0000 7fbf14b9c700 1 -- 172.21.15.27:0/3097142350 >> v2:172.21.15.77:3300/0 conn(0x7fbf1015a1b0 msgr2=0x7fbf1015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:00.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.915+0000 7fbf14b9c700 1 --2- 172.21.15.27:0/3097142350 >> v2:172.21.15.77:3300/0 conn(0x7fbf1015a1b0 0x7fbf1015a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.915+0000 7fbf14b9c700 1 -- 172.21.15.27:0/3097142350 >> v1:172.21.15.27:6789/0 conn(0x7fbf100ccd70 legacy=0x7fbf100cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:00.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.915+0000 7fbf14b9c700 1 -- 172.21.15.27:0/3097142350 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf100cf200 con 0x7fbf1015ae70 2024-01-11T09:46:00.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.915+0000 7fbf0ffff700 1 --2- 172.21.15.27:0/3097142350 >> v2:172.21.15.77:3300/0 conn(0x7fbf1015a1b0 0x7fbf1015a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:00.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.915+0000 7fbf14b9c700 1 --2- 172.21.15.27:0/3097142350 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf1015ae70 0x7fbf1015b2f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fbf08009e60 tx=0x7fbf08009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=154dc227f2d4ef11 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:00.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.915+0000 7fbf0effd700 1 -- 172.21.15.27:0/3097142350 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0800e070 con 0x7fbf1015ae70 2024-01-11T09:46:00.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.915+0000 7fbf0effd700 1 -- 172.21.15.27:0/3097142350 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0800bab0 con 0x7fbf1015ae70 2024-01-11T09:46:00.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.915+0000 7fbf0effd700 1 -- 172.21.15.27:0/3097142350 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0800be70 con 0x7fbf1015ae70 2024-01-11T09:46:00.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.916+0000 7fbf165ff700 1 -- 172.21.15.27:0/3097142350 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf1015ae70 msgr2=0x7fbf1015b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:00.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.916+0000 7fbf165ff700 1 --2- 172.21.15.27:0/3097142350 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf1015ae70 0x7fbf1015b2f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fbf08009e60 tx=0x7fbf08009780 comp rx=0 tx=0).stop 2024-01-11T09:46:00.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.916+0000 7fbf165ff700 1 -- 172.21.15.27:0/3097142350 shutdown_connections 2024-01-11T09:46:00.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.916+0000 7fbf165ff700 1 --2- 172.21.15.27:0/3097142350 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf1015ae70 0x7fbf1015b2f0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.916+0000 7fbf165ff700 1 --2- 172.21.15.27:0/3097142350 >> v2:172.21.15.77:3300/0 conn(0x7fbf1015a1b0 0x7fbf1015a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.916+0000 7fbf165ff700 1 -- 172.21.15.27:0/3097142350 >> 172.21.15.27:0/3097142350 conn(0x7fbf1006ed40 msgr2=0x7fbf1006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:00.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.916+0000 7fbf165ff700 1 -- 172.21.15.27:0/3097142350 shutdown_connections 2024-01-11T09:46:00.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.916+0000 7fbf165ff700 1 -- 172.21.15.27:0/3097142350 wait complete. 2024-01-11T09:46:00.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.917+0000 7fbf165ff700 1 Processor -- start 2024-01-11T09:46:00.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.918+0000 7fbf165ff700 1 -- start start 2024-01-11T09:46:00.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.918+0000 7fbf165ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf100ccd70 0x7fbf10158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:00.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.919+0000 7fbf165ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf1015a1b0 0x7fbf100b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:00.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.919+0000 7fbf165ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf1015ae70 0x7fbf10152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:00.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.919+0000 7fbf165ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100b5180 con 0x7fbf100ccd70 2024-01-11T09:46:00.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.919+0000 7fbf165ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100b4570 con 0x7fbf1015ae70 2024-01-11T09:46:00.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.919+0000 7fbf165ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100b4850 con 0x7fbf1015a1b0 2024-01-11T09:46:00.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.919+0000 7fbf0ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf100ccd70 0x7fbf10158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:00.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.919+0000 7fbf0ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf100ccd70 0x7fbf10158aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47802/0 (socket says 172.21.15.27:47802) 2024-01-11T09:46:00.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.919+0000 7fbf0ffff700 1 -- 172.21.15.27:0/3189292988 learned_addr learned my addr 172.21.15.27:0/3189292988 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:00.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.919+0000 7fbf0f7fe700 1 --2- 172.21.15.27:0/3189292988 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf1015a1b0 0x7fbf100b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:00.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.919+0000 7fbf14b9c700 1 --2- 172.21.15.27:0/3189292988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf1015ae70 0x7fbf10152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:00.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.919+0000 7fbf0ffff700 1 -- 172.21.15.27:0/3189292988 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf1015a1b0 msgr2=0x7fbf100b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:00.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.919+0000 7fbf0ffff700 1 --2- 172.21.15.27:0/3189292988 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf1015a1b0 0x7fbf100b3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.919+0000 7fbf0ffff700 1 -- 172.21.15.27:0/3189292988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf1015ae70 msgr2=0x7fbf10152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:00.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.919+0000 7fbf0ffff700 1 --2- 172.21.15.27:0/3189292988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf1015ae70 0x7fbf10152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:00.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.919+0000 7fbf0ffff700 1 -- 172.21.15.27:0/3189292988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf00009420 con 0x7fbf100ccd70 2024-01-11T09:46:00.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.919+0000 7fbf0f7fe700 1 --2- 172.21.15.27:0/3189292988 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf1015a1b0 0x7fbf100b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:00.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.920+0000 7fbf0ffff700 1 --2- 172.21.15.27:0/3189292988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf100ccd70 0x7fbf10158aa0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fbf0000ec80 tx=0x7fbf000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:00.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.921+0000 7fbef7fff700 1 -- 172.21.15.27:0/3189292988 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0000c920 con 0x7fbf100ccd70 2024-01-11T09:46:00.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.921+0000 7fbef7fff700 1 -- 172.21.15.27:0/3189292988 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0000ca80 con 0x7fbf100ccd70 2024-01-11T09:46:00.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.921+0000 7fbef7fff700 1 -- 172.21.15.27:0/3189292988 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0000ce00 con 0x7fbf100ccd70 2024-01-11T09:46:00.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.921+0000 7fbf165ff700 1 -- 172.21.15.27:0/3189292988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf08009a20 con 0x7fbf100ccd70 2024-01-11T09:46:00.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.921+0000 7fbf165ff700 1 -- 172.21.15.27:0/3189292988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf100b4e10 con 0x7fbf100ccd70 2024-01-11T09:46:00.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.923+0000 7fbf165ff700 1 -- 172.21.15.27:0/3189292988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf100883f0 con 0x7fbf100ccd70 2024-01-11T09:46:00.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.924+0000 7fbef7fff700 1 -- 172.21.15.27:0/3189292988 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 133) v1 ==== 85003+0+0 (secure 0 0 0) 0x7fbf00013680 con 0x7fbf100ccd70 2024-01-11T09:46:00.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.924+0000 7fbef7fff700 1 --2- 172.21.15.27:0/3189292988 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbefc067200 0x7fbefc0696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:00.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.925+0000 7fbef7fff700 1 -- 172.21.15.27:0/3189292988 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7fbf00085550 con 0x7fbf100ccd70 2024-01-11T09:46:00.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.925+0000 7fbf0f7fe700 1 -- 172.21.15.27:0/3189292988 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbefc067200 msgr2=0x7fbefc0696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:00.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.925+0000 7fbf0f7fe700 1 --2- 172.21.15.27:0/3189292988 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbefc067200 0x7fbefc0696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:46:00.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:00.933+0000 7fbef7fff700 1 -- 172.21.15.27:0/3189292988 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbf00051f60 con 0x7fbf100ccd70 2024-01-11T09:46:01.060 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:01.067+0000 7feb4febe1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:46:01.087 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:01.095+0000 7f89e6eae1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:46:01.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.126+0000 7fbf0f7fe700 1 -- 172.21.15.27:0/3189292988 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbefc067200 msgr2=0x7fbefc0696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:01.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.126+0000 7fbf0f7fe700 1 --2- 172.21.15.27:0/3189292988 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbefc067200 0x7fbefc0696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:46:01.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.142+0000 7fbf165ff700 1 -- 172.21.15.27:0/3189292988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbf1015d5a0 con 0x7fbf100ccd70 2024-01-11T09:46:01.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.142+0000 7fbef7fff700 1 -- 172.21.15.27:0/3189292988 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v219) v1 ==== 107+0+11 (secure 0 0 0) 0x7fbf00055580 con 0x7fbf100ccd70 2024-01-11T09:46:01.135 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T09:46:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.145+0000 7fbf165ff700 1 -- 172.21.15.27:0/3189292988 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbefc067200 msgr2=0x7fbefc0696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:46:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.145+0000 7fbf165ff700 1 --2- 172.21.15.27:0/3189292988 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbefc067200 0x7fbefc0696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.145+0000 7fbf165ff700 1 -- 172.21.15.27:0/3189292988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf100ccd70 msgr2=0x7fbf10158aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.145+0000 7fbf165ff700 1 --2- 172.21.15.27:0/3189292988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf100ccd70 0x7fbf10158aa0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fbf0000ec80 tx=0x7fbf000091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.145+0000 7fbf165ff700 1 -- 172.21.15.27:0/3189292988 shutdown_connections 2024-01-11T09:46:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.145+0000 7fbf165ff700 1 --2- 172.21.15.27:0/3189292988 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbefc067200 0x7fbefc0696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.145+0000 7fbf165ff700 1 --2- 172.21.15.27:0/3189292988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf1015ae70 0x7fbf10152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.145+0000 7fbf165ff700 1 --2- 172.21.15.27:0/3189292988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf100ccd70 0x7fbf10158aa0 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.145+0000 7fbf165ff700 1 --2- 172.21.15.27:0/3189292988 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbf1015a1b0 0x7fbf100b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.145+0000 7fbf165ff700 1 -- 172.21.15.27:0/3189292988 >> 172.21.15.27:0/3189292988 conn(0x7fbf1006ed40 msgr2=0x7fbf100c5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.145+0000 7fbf165ff700 1 -- 172.21.15.27:0/3189292988 shutdown_connections 2024-01-11T09:46:01.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.145+0000 7fbf165ff700 1 -- 172.21.15.27:0/3189292988 wait complete. 2024-01-11T09:46:01.154 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:01.162+0000 7feb4febe1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:46:01.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-01-11T09:46:01.181 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:01.189+0000 7f89e6eae1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:46:01.277 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:01.284+0000 7f89e6eae1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:46:01.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.371+0000 7f1ccf9ff700 1 Processor -- start 2024-01-11T09:46:01.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.372+0000 7f1ccf9ff700 1 -- start start 2024-01-11T09:46:01.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.373+0000 7f1ccf9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1cc8143190 0x7f1cc80cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:01.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.373+0000 7f1ccf9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1cc80cb4b0 0x7f1cc80cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:01.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.373+0000 7f1ccf9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc815d320 con 0x7f1cc8142dc0 2024-01-11T09:46:01.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.373+0000 7f1ccf9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc815df30 con 0x7f1cc8143190 2024-01-11T09:46:01.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.373+0000 7f1ccf9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc815d590 con 0x7f1cc80cb4b0 2024-01-11T09:46:01.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.373+0000 7f1ccd79b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1cc8142dc0 0x7f1cc80cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42200/0 (socket says 172.21.15.27:42200) 2024-01-11T09:46:01.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.373+0000 7f1ccd79b700 1 -- 172.21.15.27:0/2057747201 learned_addr learned my addr 172.21.15.27:0/2057747201 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:01.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.373+0000 7f1cccf9a700 1 --2- 172.21.15.27:0/2057747201 >> v2:172.21.15.27:3300/0 conn(0x7f1cc8143190 0x7f1cc80cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:01.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.373+0000 7f1ccdf9c700 1 --2- 172.21.15.27:0/2057747201 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1cc80cb4b0 0x7f1cc80cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:01.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.374+0000 7f1cbffff700 1 -- 172.21.15.27:0/2057747201 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735484320 0 0) 0x7f1cc815d320 con 0x7f1cc8142dc0 2024-01-11T09:46:01.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.374+0000 7f1cbffff700 1 -- 172.21.15.27:0/2057747201 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cb4003570 con 0x7f1cc8142dc0 2024-01-11T09:46:01.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.374+0000 7f1cccf9a700 1 -- 172.21.15.27:0/2057747201 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1cc80cb4b0 msgr2=0x7f1cc80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:01.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.374+0000 7f1cccf9a700 1 --2- 172.21.15.27:0/2057747201 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1cc80cb4b0 0x7f1cc80cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.374+0000 7f1cccf9a700 1 -- 172.21.15.27:0/2057747201 >> v1:172.21.15.27:6789/0 conn(0x7f1cc8142dc0 legacy=0x7f1cc80cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:01.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.374+0000 7f1cccf9a700 1 -- 172.21.15.27:0/2057747201 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc815d870 con 0x7f1cc8143190 2024-01-11T09:46:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.374+0000 7f1cccf9a700 1 --2- 172.21.15.27:0/2057747201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cc8143190 0x7f1cc80cc020 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f1cb80098e0 tx=0x7f1cb80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70bd27d0620eefa8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.374+0000 7f1cbffff700 1 -- 172.21.15.27:0/2057747201 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8011630 con 0x7f1cc8143190 2024-01-11T09:46:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.374+0000 7f1cbffff700 1 -- 172.21.15.27:0/2057747201 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb8011790 con 0x7f1cc8143190 2024-01-11T09:46:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.374+0000 7f1cbffff700 1 -- 172.21.15.27:0/2057747201 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8011b30 con 0x7f1cc8143190 2024-01-11T09:46:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.375+0000 7f1ccf9ff700 1 -- 172.21.15.27:0/2057747201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cc8143190 msgr2=0x7f1cc80cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.375+0000 7f1ccf9ff700 1 --2- 172.21.15.27:0/2057747201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cc8143190 0x7f1cc80cc020 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f1cb80098e0 tx=0x7f1cb80091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.375+0000 7f1ccf9ff700 1 -- 172.21.15.27:0/2057747201 shutdown_connections 2024-01-11T09:46:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.375+0000 7f1ccf9ff700 1 --2- 172.21.15.27:0/2057747201 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1cc80cb4b0 0x7f1cc80cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.375+0000 7f1ccf9ff700 1 --2- 172.21.15.27:0/2057747201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cc8143190 0x7f1cc80cc020 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.375+0000 7f1ccf9ff700 1 -- 172.21.15.27:0/2057747201 >> 172.21.15.27:0/2057747201 conn(0x7f1cc8063ca0 msgr2=0x7f1cc806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.375+0000 7f1ccf9ff700 1 -- 172.21.15.27:0/2057747201 shutdown_connections 2024-01-11T09:46:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.375+0000 7f1ccf9ff700 1 -- 172.21.15.27:0/2057747201 wait complete. 2024-01-11T09:46:01.371 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:01.378+0000 7f89e6eae1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:46:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.377+0000 7f1ccf9ff700 1 Processor -- start 2024-01-11T09:46:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.377+0000 7f1ccf9ff700 1 -- start start 2024-01-11T09:46:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.378+0000 7f1ccf9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cc80cb4b0 0x7f1cc813e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.378+0000 7f1ccf9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1cc8142dc0 0x7f1cc813e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.378+0000 7f1ccf9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc8143190 0x7f1cc8169a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.378+0000 7f1ccf9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc80c5e50 con 0x7f1cc80cb4b0 2024-01-11T09:46:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.378+0000 7f1ccf9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc8169350 con 0x7f1cc8143190 2024-01-11T09:46:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.378+0000 7f1ccf9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc813e6e0 con 0x7f1cc8142dc0 2024-01-11T09:46:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.378+0000 7f1ccd79b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cc80cb4b0 0x7f1cc813e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.378+0000 7f1ccd79b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cc80cb4b0 0x7f1cc813e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47820/0 (socket says 172.21.15.27:47820) 2024-01-11T09:46:01.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.378+0000 7f1ccd79b700 1 -- 172.21.15.27:0/1016732513 learned_addr learned my addr 172.21.15.27:0/1016732513 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:01.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.378+0000 7f1cccf9a700 1 --2- 172.21.15.27:0/1016732513 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1cc8142dc0 0x7f1cc813e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:01.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.378+0000 7f1ccdf9c700 1 --2- 172.21.15.27:0/1016732513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc8143190 0x7f1cc8169a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:01.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.379+0000 7f1cccf9a700 1 -- 172.21.15.27:0/1016732513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc8143190 msgr2=0x7f1cc8169a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:01.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.379+0000 7f1cccf9a700 1 --2- 172.21.15.27:0/1016732513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc8143190 0x7f1cc8169a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.379+0000 7f1cccf9a700 1 -- 172.21.15.27:0/1016732513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cc80cb4b0 msgr2=0x7f1cc813e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:01.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.379+0000 7f1cccf9a700 1 --2- 172.21.15.27:0/1016732513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cc80cb4b0 0x7f1cc813e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.379+0000 7f1cccf9a700 1 -- 172.21.15.27:0/1016732513 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb8009420 con 0x7f1cc8142dc0 2024-01-11T09:46:01.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.379+0000 7f1ccd79b700 1 --2- 172.21.15.27:0/1016732513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cc80cb4b0 0x7f1cc813e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:01.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.379+0000 7f1cccf9a700 1 --2- 172.21.15.27:0/1016732513 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1cc8142dc0 0x7f1cc813e1a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8016000 tx=0x7f1cb8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:01.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.379+0000 7f1cbd7fa700 1 -- 172.21.15.27:0/1016732513 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8011900 con 0x7f1cc8142dc0 2024-01-11T09:46:01.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.379+0000 7f1cbd7fa700 1 -- 172.21.15.27:0/1016732513 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb8011a60 con 0x7f1cc8142dc0 2024-01-11T09:46:01.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.379+0000 7f1cbd7fa700 1 -- 172.21.15.27:0/1016732513 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8025640 con 0x7f1cc8142dc0 2024-01-11T09:46:01.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.379+0000 7f1ccf9ff700 1 -- 172.21.15.27:0/1016732513 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc813f0d0 con 0x7f1cc8142dc0 2024-01-11T09:46:01.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.380+0000 7f1ccf9ff700 1 -- 172.21.15.27:0/1016732513 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc80696d0 con 0x7f1cc8142dc0 2024-01-11T09:46:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.381+0000 7f1ccf9ff700 1 -- 172.21.15.27:0/1016732513 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc804c3c0 con 0x7f1cc8142dc0 2024-01-11T09:46:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.382+0000 7f1cbd7fa700 1 -- 172.21.15.27:0/1016732513 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 133) v1 ==== 85003+0+0 (secure 0 0 0) 0x7f1cb80193b0 con 0x7f1cc8142dc0 2024-01-11T09:46:01.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.383+0000 7f1cbd7fa700 1 --2- 172.21.15.27:0/1016732513 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1cac067200 0x7f1cac0696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:01.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.383+0000 7f1cbd7fa700 1 -- 172.21.15.27:0/1016732513 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f1cb8086560 con 0x7f1cc8142dc0 2024-01-11T09:46:01.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.383+0000 7f1ccd79b700 1 -- 172.21.15.27:0/1016732513 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1cac067200 msgr2=0x7f1cac0696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:01.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.383+0000 7f1ccd79b700 1 --2- 172.21.15.27:0/1016732513 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1cac067200 0x7f1cac0696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:46:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.388+0000 7f1cbd7fa700 1 -- 172.21.15.27:0/1016732513 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1cb8052fb0 con 0x7f1cc8142dc0 2024-01-11T09:46:01.393 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:01.401+0000 7feb4febe1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:46:01.496 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:01.504+0000 7feb4febe1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:46:01.559 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:01.567+0000 7f89e6eae1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:46:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.584+0000 7f1ccd79b700 1 -- 172.21.15.27:0/1016732513 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1cac067200 msgr2=0x7f1cac0696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.584+0000 7f1ccd79b700 1 --2- 172.21.15.27:0/1016732513 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1cac067200 0x7f1cac0696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:46:01.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.592+0000 7f1ccf9ff700 1 -- 172.21.15.27:0/1016732513 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f1cc815db50 con 0x7f1cc8142dc0 2024-01-11T09:46:01.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.593+0000 7f1cbd7fa700 1 -- 172.21.15.27:0/1016732513 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v219) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1cb80565d0 con 0x7f1cc8142dc0 2024-01-11T09:46:01.585 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T09:46:01.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.595+0000 7f1ccf9ff700 1 -- 172.21.15.27:0/1016732513 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1cac067200 msgr2=0x7f1cac0696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:46:01.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.595+0000 7f1ccf9ff700 1 --2- 172.21.15.27:0/1016732513 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1cac067200 0x7f1cac0696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.595+0000 7f1ccf9ff700 1 -- 172.21.15.27:0/1016732513 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1cc8142dc0 msgr2=0x7f1cc813e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:01.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.595+0000 7f1ccf9ff700 1 --2- 172.21.15.27:0/1016732513 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1cc8142dc0 0x7f1cc813e1a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8016000 tx=0x7f1cb8017000 comp rx=0 tx=0).stop 2024-01-11T09:46:01.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.595+0000 7f1ccf9ff700 1 -- 172.21.15.27:0/1016732513 shutdown_connections 2024-01-11T09:46:01.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.595+0000 7f1ccf9ff700 1 --2- 172.21.15.27:0/1016732513 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1cac067200 0x7f1cac0696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.595+0000 7f1ccf9ff700 1 --2- 172.21.15.27:0/1016732513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cc8143190 0x7f1cc8169a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.595+0000 7f1ccf9ff700 1 --2- 172.21.15.27:0/1016732513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cc80cb4b0 0x7f1cc813e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.595+0000 7f1ccf9ff700 1 --2- 172.21.15.27:0/1016732513 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1cc8142dc0 0x7f1cc813e1a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.595+0000 7f1ccf9ff700 1 -- 172.21.15.27:0/1016732513 >> 172.21.15.27:0/1016732513 conn(0x7f1cc8063ca0 msgr2=0x7f1cc8141df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.595+0000 7f1ccf9ff700 1 -- 172.21.15.27:0/1016732513 shutdown_connections 2024-01-11T09:46:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.595+0000 7f1ccf9ff700 1 -- 172.21.15.27:0/1016732513 wait complete. 2024-01-11T09:46:01.599 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:01.607+0000 7feb4febe1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:46:01.606 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:46:01.656 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:01.664+0000 7f89e6eae1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:46:01.706 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:01.714+0000 7feb4febe1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:46:01.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.816+0000 7f71b27ff700 1 Processor -- start 2024-01-11T09:46:01.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.816+0000 7f71b27ff700 1 -- start start 2024-01-11T09:46:01.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.817+0000 7f71b27ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f71ac0b03e0 0x7f71ac0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:01.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.817+0000 7f71b27ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71ac0b2e00 con 0x7f71ac0b07b0 2024-01-11T09:46:01.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.817+0000 7f71b27ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71ac0cd730 con 0x7f71ac0b29c0 2024-01-11T09:46:01.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.817+0000 7f71b27ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac0ccb20 con 0x7f71ac0b03e0 2024-01-11T09:46:01.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.817+0000 7f71ab7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f71ac0b07b0 0x7f71ac159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42202/0 (socket says 172.21.15.27:42202) 2024-01-11T09:46:01.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.818+0000 7f71ab7fe700 1 -- 172.21.15.27:0/378495669 learned_addr learned my addr 172.21.15.27:0/378495669 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:01.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.818+0000 7f71abfff700 1 --2- 172.21.15.27:0/378495669 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f71ac0b03e0 0x7f71ac0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:01.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.818+0000 7f71aaffd700 1 -- 172.21.15.27:0/378495669 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326022515 0 0) 0x7f71a0001110 con 0x7f71ac0b07b0 2024-01-11T09:46:01.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.818+0000 7f71aaffd700 1 -- 172.21.15.27:0/378495669 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7194003570 con 0x7f71ac0b07b0 2024-01-11T09:46:01.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.818+0000 7f71aaffd700 1 -- 172.21.15.27:0/378495669 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586041842 0 0) 0x7f71ac0cd730 con 0x7f71ac0b29c0 2024-01-11T09:46:01.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.818+0000 7f71aaffd700 1 -- 172.21.15.27:0/378495669 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71a0001110 con 0x7f71ac0b29c0 2024-01-11T09:46:01.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.818+0000 7f71abfff700 1 -- 172.21.15.27:0/378495669 >> v1:172.21.15.77:6789/0 conn(0x7f71ac0b29c0 legacy=0x7f71ac0aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:01.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.818+0000 7f71abfff700 1 -- 172.21.15.27:0/378495669 >> v1:172.21.15.27:6789/0 conn(0x7f71ac0b07b0 legacy=0x7f71ac159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:01.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.818+0000 7f71abfff700 1 -- 172.21.15.27:0/378495669 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ac0cce00 con 0x7f71ac0b03e0 2024-01-11T09:46:01.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.818+0000 7f71abfff700 1 --2- 172.21.15.27:0/378495669 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f71ac0b03e0 0x7f71ac0b2480 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f719c009910 tx=0x7f719c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6c9cf35db61f46b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:01.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.819+0000 7f71aaffd700 1 -- 172.21.15.27:0/378495669 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c011630 con 0x7f71ac0b03e0 2024-01-11T09:46:01.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.819+0000 7f71aaffd700 1 -- 172.21.15.27:0/378495669 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719c011790 con 0x7f71ac0b03e0 2024-01-11T09:46:01.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.819+0000 7f71aaffd700 1 -- 172.21.15.27:0/378495669 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c011b30 con 0x7f71ac0b03e0 2024-01-11T09:46:01.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.819+0000 7f71b27ff700 1 -- 172.21.15.27:0/378495669 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f71ac0b03e0 msgr2=0x7f71ac0b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:01.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.819+0000 7f71b27ff700 1 --2- 172.21.15.27:0/378495669 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f71ac0b03e0 0x7f71ac0b2480 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f719c009910 tx=0x7f719c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.819+0000 7f71b27ff700 1 -- 172.21.15.27:0/378495669 shutdown_connections 2024-01-11T09:46:01.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.819+0000 7f71b27ff700 1 --2- 172.21.15.27:0/378495669 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f71ac0b03e0 0x7f71ac0b2480 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.819+0000 7f71b27ff700 1 -- 172.21.15.27:0/378495669 >> 172.21.15.27:0/378495669 conn(0x7f71ac06ed40 msgr2=0x7f71ac06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:01.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.820+0000 7f71b27ff700 1 -- 172.21.15.27:0/378495669 shutdown_connections 2024-01-11T09:46:01.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.820+0000 7f71b27ff700 1 -- 172.21.15.27:0/378495669 wait complete. 2024-01-11T09:46:01.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.821+0000 7f71b27ff700 1 Processor -- start 2024-01-11T09:46:01.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.821+0000 7f71b27ff700 1 -- start start 2024-01-11T09:46:01.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.822+0000 7f71b27ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f71ac0b03e0 0x7f71ac0a8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:01.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.822+0000 7f71b27ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71ac0b07b0 0x7f71ac0a6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:01.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.822+0000 7f71b27ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac0b29c0 0x7f71ac169b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:01.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.822+0000 7f71b27ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac0a8c90 con 0x7f71ac0b29c0 2024-01-11T09:46:01.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.822+0000 7f71b27ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac0a7480 con 0x7f71ac0b07b0 2024-01-11T09:46:01.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.822+0000 7f71b27ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac0aec80 con 0x7f71ac0b03e0 2024-01-11T09:46:01.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.823+0000 7f71b0d9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac0b29c0 0x7f71ac169b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:01.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.823+0000 7f71abfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f71ac0b03e0 0x7f71ac0a8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:01.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.823+0000 7f71ab7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71ac0b07b0 0x7f71ac0a6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:01.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.823+0000 7f71b0d9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac0b29c0 0x7f71ac169b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47836/0 (socket says 172.21.15.27:47836) 2024-01-11T09:46:01.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.823+0000 7f71b0d9c700 1 -- 172.21.15.27:0/2986157010 learned_addr learned my addr 172.21.15.27:0/2986157010 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:01.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.823+0000 7f71b0d9c700 1 -- 172.21.15.27:0/2986157010 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f71ac0b03e0 msgr2=0x7f71ac0a8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:01.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.823+0000 7f71b0d9c700 1 --2- 172.21.15.27:0/2986157010 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f71ac0b03e0 0x7f71ac0a8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.823+0000 7f71b0d9c700 1 -- 172.21.15.27:0/2986157010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71ac0b07b0 msgr2=0x7f71ac0a6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:01.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.823+0000 7f71b0d9c700 1 --2- 172.21.15.27:0/2986157010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71ac0b07b0 0x7f71ac0a6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:01.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.823+0000 7f71b0d9c700 1 -- 172.21.15.27:0/2986157010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719c009420 con 0x7f71ac0b29c0 2024-01-11T09:46:01.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.823+0000 7f71b0d9c700 1 --2- 172.21.15.27:0/2986157010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac0b29c0 0x7f71ac169b80 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f71a400c860 tx=0x7f71a4002700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:01.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.824+0000 7f7193fff700 1 -- 172.21.15.27:0/2986157010 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a4002030 con 0x7f71ac0b29c0 2024-01-11T09:46:01.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.824+0000 7f71b27ff700 1 -- 172.21.15.27:0/2986157010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71ac0a82c0 con 0x7f71ac0b29c0 2024-01-11T09:46:01.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.824+0000 7f7193fff700 1 -- 172.21.15.27:0/2986157010 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a40021b0 con 0x7f71ac0b29c0 2024-01-11T09:46:01.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.824+0000 7f7193fff700 1 -- 172.21.15.27:0/2986157010 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a4012630 con 0x7f71ac0b29c0 2024-01-11T09:46:01.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.824+0000 7f71b27ff700 1 -- 172.21.15.27:0/2986157010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71ac0a7990 con 0x7f71ac0b29c0 2024-01-11T09:46:01.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.825+0000 7f71b27ff700 1 -- 172.21.15.27:0/2986157010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71ac0a8450 con 0x7f71ac0b29c0 2024-01-11T09:46:01.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.826+0000 7f7193fff700 1 -- 172.21.15.27:0/2986157010 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 133) v1 ==== 85003+0+0 (secure 0 0 0) 0x7f71a4012a20 con 0x7f71ac0b29c0 2024-01-11T09:46:01.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.827+0000 7f7193fff700 1 --2- 172.21.15.27:0/2986157010 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7198067200 0x7f71980696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:01.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.827+0000 7f7193fff700 1 -- 172.21.15.27:0/2986157010 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f71a4084e30 con 0x7f71ac0b29c0 2024-01-11T09:46:01.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.827+0000 7f71abfff700 1 -- 172.21.15.27:0/2986157010 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7198067200 msgr2=0x7f71980696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:01.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.827+0000 7f71abfff700 1 --2- 172.21.15.27:0/2986157010 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7198067200 0x7f71980696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:46:01.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:01.831+0000 7f7193fff700 1 -- 172.21.15.27:0/2986157010 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f71a4051840 con 0x7f71ac0b29c0 2024-01-11T09:46:01.914 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:01.922+0000 7feb4febe1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:46:02.000 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:02.008+0000 7f89e6eae1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:46:02.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.028+0000 7f71abfff700 1 -- 172.21.15.27:0/2986157010 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7198067200 msgr2=0x7f71980696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:02.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.028+0000 7f71abfff700 1 --2- 172.21.15.27:0/2986157010 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7198067200 0x7f71980696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:46:02.026 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:02.034+0000 7feb4febe1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:46:02.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.082+0000 7f71b27ff700 1 -- 172.21.15.27:0/2986157010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71ac0cd0e0 con 0x7f71ac0b29c0 2024-01-11T09:46:02.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.083+0000 7f7193fff700 1 -- 172.21.15.27:0/2986157010 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+5690 (secure 0 0 0) 0x7f71a4054e60 con 0x7f71ac0b29c0 2024-01-11T09:46:02.075 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:02.076 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":117,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2103359888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":2103359888},{"type":"v1","addr":"172.21.15.77:6834","nonce":2103359888}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8795,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2650824649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2650824649},{"type":"v1","addr":"172.21.15.77:6835","nonce":2650824649}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:45:23.637614+0000","modified":"2024-01-11T09:45:27.146191+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10599":{"gid":10599,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1143960902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1143960902},{"type":"v1","addr":"172.21.15.27:6835","nonce":1143960902}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:45:37.460120+0000","modified":"2024-01-11T09:45:40.977702+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10596},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10596":{"gid":10596,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6834/2807751836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2807751836},{"type":"v1","addr":"172.21.15.27:6834","nonce":2807751836}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":14}]} 2024-01-11T09:46:02.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.085+0000 7f71b27ff700 1 -- 172.21.15.27:0/2986157010 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7198067200 msgr2=0x7f71980696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:46:02.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.085+0000 7f71b27ff700 1 --2- 172.21.15.27:0/2986157010 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7198067200 0x7f71980696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.085+0000 7f71b27ff700 1 -- 172.21.15.27:0/2986157010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac0b29c0 msgr2=0x7f71ac169b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:02.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.085+0000 7f71b27ff700 1 --2- 172.21.15.27:0/2986157010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac0b29c0 0x7f71ac169b80 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f71a400c860 tx=0x7f71a4002700 comp rx=0 tx=0).stop 2024-01-11T09:46:02.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.085+0000 7f71b27ff700 1 -- 172.21.15.27:0/2986157010 shutdown_connections 2024-01-11T09:46:02.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.085+0000 7f71b27ff700 1 --2- 172.21.15.27:0/2986157010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac0b29c0 0x7f71ac169b80 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.085+0000 7f71b27ff700 1 --2- 172.21.15.27:0/2986157010 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f71ac0b03e0 0x7f71ac0a8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.085+0000 7f71b27ff700 1 --2- 172.21.15.27:0/2986157010 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7198067200 0x7f71980696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.085+0000 7f71b27ff700 1 --2- 172.21.15.27:0/2986157010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71ac0b07b0 0x7f71ac0a6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.085+0000 7f71b27ff700 1 -- 172.21.15.27:0/2986157010 >> 172.21.15.27:0/2986157010 conn(0x7f71ac06ed40 msgr2=0x7f71ac0ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:02.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.085+0000 7f71b27ff700 1 -- 172.21.15.27:0/2986157010 shutdown_connections 2024-01-11T09:46:02.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.086+0000 7f71b27ff700 1 -- 172.21.15.27:0/2986157010 wait complete. 2024-01-11T09:46:02.079 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 117 2024-01-11T09:46:02.091 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:02.099+0000 7f89e6eae1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:46:02.096 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:46:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.309+0000 7f8904bff700 1 Processor -- start 2024-01-11T09:46:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.310+0000 7f8904bff700 1 -- start start 2024-01-11T09:46:02.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.311+0000 7f8904bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f89001575e0 0x7f8900158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:02.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.311+0000 7f8904bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89000b19d0 0x7f8900159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:02.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.311+0000 7f8904bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8900158b60 con 0x7f890006b610 2024-01-11T09:46:02.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.311+0000 7f8904bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89000cb5e0 con 0x7f89000b19d0 2024-01-11T09:46:02.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.311+0000 7f8904bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89001599b0 con 0x7f89001575e0 2024-01-11T09:46:02.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.311+0000 7f88fed9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89000b19d0 0x7f8900159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.311+0000 7f88fe59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f89001575e0 0x7f8900158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.311+0000 7f88fed9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89000b19d0 0x7f8900159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47838/0 (socket says 172.21.15.27:47838) 2024-01-11T09:46:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.311+0000 7f88fed9d700 1 -- 172.21.15.27:0/2342168682 learned_addr learned my addr 172.21.15.27:0/2342168682 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.311+0000 7f88fdd9b700 1 --1- 172.21.15.27:0/2342168682 >> v1:172.21.15.77:6789/0 conn(0x7f890006b610 0x7f89000b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:60082/0 (socket says 172.21.15.27:60082) 2024-01-11T09:46:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.312+0000 7f88fe59c700 1 -- 172.21.15.27:0/2342168682 >> v2:172.21.15.27:3300/0 conn(0x7f89000b19d0 msgr2=0x7f8900159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.312+0000 7f88fe59c700 1 --2- 172.21.15.27:0/2342168682 >> v2:172.21.15.27:3300/0 conn(0x7f89000b19d0 0x7f8900159e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.312+0000 7f88fe59c700 1 -- 172.21.15.27:0/2342168682 >> v1:172.21.15.77:6789/0 conn(0x7f890006b610 legacy=0x7f89000b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.312+0000 7f88fe59c700 1 -- 172.21.15.27:0/2342168682 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89000c0f90 con 0x7f89001575e0 2024-01-11T09:46:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.312+0000 7f88fd59a700 1 -- 172.21.15.27:0/2342168682 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985820141 0 0) 0x7f88f4001110 con 0x7f890006b610 2024-01-11T09:46:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.312+0000 7f88fed9d700 1 --2- 172.21.15.27:0/2342168682 >> v2:172.21.15.27:3300/0 conn(0x7f89000b19d0 0x7f8900159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:02.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.312+0000 7f88fe59c700 1 --2- 172.21.15.27:0/2342168682 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f89001575e0 0x7f8900158620 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f88f00098e0 tx=0x7f88f00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da9f2b23a5195526 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:02.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.312+0000 7f88fd59a700 1 -- 172.21.15.27:0/2342168682 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f000b640 con 0x7f89001575e0 2024-01-11T09:46:02.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.312+0000 7f88fd59a700 1 -- 172.21.15.27:0/2342168682 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f000b7a0 con 0x7f89001575e0 2024-01-11T09:46:02.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.312+0000 7f88fd59a700 1 -- 172.21.15.27:0/2342168682 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f000bb40 con 0x7f89001575e0 2024-01-11T09:46:02.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.313+0000 7f8904bff700 1 -- 172.21.15.27:0/2342168682 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f89001575e0 msgr2=0x7f8900158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:02.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.313+0000 7f8904bff700 1 --2- 172.21.15.27:0/2342168682 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f89001575e0 0x7f8900158620 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f88f00098e0 tx=0x7f88f00091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.313+0000 7f8904bff700 1 -- 172.21.15.27:0/2342168682 shutdown_connections 2024-01-11T09:46:02.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.313+0000 7f8904bff700 1 --2- 172.21.15.27:0/2342168682 >> v2:172.21.15.27:3300/0 conn(0x7f89000b19d0 0x7f8900159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.313+0000 7f8904bff700 1 --2- 172.21.15.27:0/2342168682 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f89001575e0 0x7f8900158620 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.313+0000 7f8904bff700 1 -- 172.21.15.27:0/2342168682 >> 172.21.15.27:0/2342168682 conn(0x7f89000680e0 msgr2=0x7f8900069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:02.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.313+0000 7f8904bff700 1 -- 172.21.15.27:0/2342168682 shutdown_connections 2024-01-11T09:46:02.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.313+0000 7f8904bff700 1 -- 172.21.15.27:0/2342168682 wait complete. 2024-01-11T09:46:02.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.315+0000 7f8904bff700 1 Processor -- start 2024-01-11T09:46:02.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.315+0000 7f8904bff700 1 -- start start 2024-01-11T09:46:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.316+0000 7f8904bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f890006b610 0x7f89001454d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.316+0000 7f8904bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f89000b19d0 0x7f890006e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.316+0000 7f8904bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89001575e0 0x7f8900151e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.316+0000 7f8904bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89000c6790 con 0x7f890006b610 2024-01-11T09:46:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.316+0000 7f8904bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89000c5b80 con 0x7f89001575e0 2024-01-11T09:46:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.316+0000 7f8904bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89000c5e60 con 0x7f89000b19d0 2024-01-11T09:46:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.316+0000 7f88fe59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f890006b610 0x7f89001454d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.316+0000 7f88fe59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f890006b610 0x7f89001454d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47852/0 (socket says 172.21.15.27:47852) 2024-01-11T09:46:02.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.316+0000 7f88fe59c700 1 -- 172.21.15.27:0/2761809522 learned_addr learned my addr 172.21.15.27:0/2761809522 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:02.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.316+0000 7f88fdd9b700 1 --2- 172.21.15.27:0/2761809522 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f89000b19d0 0x7f890006e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:02.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.316+0000 7f88fed9d700 1 --2- 172.21.15.27:0/2761809522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89001575e0 0x7f8900151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:02.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.317+0000 7f88fe59c700 1 -- 172.21.15.27:0/2761809522 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f89000b19d0 msgr2=0x7f890006e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:02.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.317+0000 7f88fe59c700 1 --2- 172.21.15.27:0/2761809522 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f89000b19d0 0x7f890006e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.317+0000 7f88fe59c700 1 -- 172.21.15.27:0/2761809522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89001575e0 msgr2=0x7f8900151e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:02.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.317+0000 7f88fe59c700 1 --2- 172.21.15.27:0/2761809522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89001575e0 0x7f8900151e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.317+0000 7f88fe59c700 1 -- 172.21.15.27:0/2761809522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f8009ae0 con 0x7f890006b610 2024-01-11T09:46:02.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.317+0000 7f88fe59c700 1 --2- 172.21.15.27:0/2761809522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f890006b610 0x7f89001454d0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f88f00093f0 tx=0x7f88f0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:02.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.317+0000 7f88ee7fc700 1 -- 172.21.15.27:0/2761809522 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f000bb20 con 0x7f890006b610 2024-01-11T09:46:02.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.317+0000 7f88ee7fc700 1 -- 172.21.15.27:0/2761809522 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f0000c90 con 0x7f890006b610 2024-01-11T09:46:02.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.317+0000 7f8904bff700 1 -- 172.21.15.27:0/2761809522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f0009420 con 0x7f890006b610 2024-01-11T09:46:02.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.318+0000 7f88ee7fc700 1 -- 172.21.15.27:0/2761809522 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0011650 con 0x7f890006b610 2024-01-11T09:46:02.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.318+0000 7f8904bff700 1 -- 172.21.15.27:0/2761809522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89000c6420 con 0x7f890006b610 2024-01-11T09:46:02.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.319+0000 7f8904bff700 1 -- 172.21.15.27:0/2761809522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89000718d0 con 0x7f890006b610 2024-01-11T09:46:02.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.320+0000 7f88ee7fc700 1 -- 172.21.15.27:0/2761809522 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 133) v1 ==== 85003+0+0 (secure 0 0 0) 0x7f88f0011a40 con 0x7f890006b610 2024-01-11T09:46:02.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.322+0000 7f88ee7fc700 1 --2- 172.21.15.27:0/2761809522 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f88e0067200 0x7f88e00696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:02.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.322+0000 7f88ee7fc700 1 -- 172.21.15.27:0/2761809522 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f88f0086270 con 0x7f890006b610 2024-01-11T09:46:02.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.322+0000 7f88fdd9b700 1 -- 172.21.15.27:0/2761809522 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f88e0067200 msgr2=0x7f88e00696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:02.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.322+0000 7f88fdd9b700 1 --2- 172.21.15.27:0/2761809522 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f88e0067200 0x7f88e00696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:46:02.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.326+0000 7f88ee7fc700 1 -- 172.21.15.27:0/2761809522 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f88f0052c80 con 0x7f890006b610 2024-01-11T09:46:02.404 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:02.411+0000 7feb4febe1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:46:02.499 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:02.507+0000 7feb4febe1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:46:02.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.523+0000 7f88fdd9b700 1 -- 172.21.15.27:0/2761809522 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f88e0067200 msgr2=0x7f88e00696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:02.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.523+0000 7f88fdd9b700 1 --2- 172.21.15.27:0/2761809522 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f88e0067200 0x7f88e00696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:46:02.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.533+0000 7f8904bff700 1 -- 172.21.15.27:0/2761809522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8900158da0 con 0x7f890006b610 2024-01-11T09:46:02.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.534+0000 7f88ee7fc700 1 -- 172.21.15.27:0/2761809522 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v219) v1 ==== 74+0+19093 (secure 0 0 0) 0x7f88f00562a0 con 0x7f890006b610 2024-01-11T09:46:02.527 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:02.527 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":219,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:45:52.192269+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:45:21.455328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-01-11T09:45:22.634298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-01-11T09:45:35.631727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-01-11T09:45:36.452195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/2192171652":"2024-01-12T09:45:52.192235+0000","172.21.15.77:6837/59776":"2024-01-12T09:45:52.192235+0000","172.21.15.77:6836/59776":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/2654175097":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/652362578":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/2439942664":"2024-01-12T09:45:52.192235+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:46:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.537+0000 7f8904bff700 1 -- 172.21.15.27:0/2761809522 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f88e0067200 msgr2=0x7f88e00696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:46:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.537+0000 7f8904bff700 1 --2- 172.21.15.27:0/2761809522 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f88e0067200 0x7f88e00696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.537+0000 7f8904bff700 1 -- 172.21.15.27:0/2761809522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f890006b610 msgr2=0x7f89001454d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.537+0000 7f8904bff700 1 --2- 172.21.15.27:0/2761809522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f890006b610 0x7f89001454d0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f88f00093f0 tx=0x7f88f0017000 comp rx=0 tx=0).stop 2024-01-11T09:46:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.537+0000 7f8904bff700 1 -- 172.21.15.27:0/2761809522 shutdown_connections 2024-01-11T09:46:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.537+0000 7f8904bff700 1 --2- 172.21.15.27:0/2761809522 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f88e0067200 0x7f88e00696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.537+0000 7f8904bff700 1 --2- 172.21.15.27:0/2761809522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89001575e0 0x7f8900151e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.537+0000 7f8904bff700 1 --2- 172.21.15.27:0/2761809522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f890006b610 0x7f89001454d0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.537+0000 7f8904bff700 1 --2- 172.21.15.27:0/2761809522 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f89000b19d0 0x7f890006e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.537+0000 7f8904bff700 1 -- 172.21.15.27:0/2761809522 >> 172.21.15.27:0/2761809522 conn(0x7f89000680e0 msgr2=0x7f89001558f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.537+0000 7f8904bff700 1 -- 172.21.15.27:0/2761809522 shutdown_connections 2024-01-11T09:46:02.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.537+0000 7f8904bff700 1 -- 172.21.15.27:0/2761809522 wait complete. 2024-01-11T09:46:02.548 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-11T09:46:02.549 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:46:02.634 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:02.642+0000 7f89e6eae1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:46:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.759+0000 7f28745ff700 1 Processor -- start 2024-01-11T09:46:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.759+0000 7f28745ff700 1 -- start start 2024-01-11T09:46:02.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.760+0000 7f28745ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f286c0c8a30 0x7f286c0c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:02.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.760+0000 7f28745ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f286c0cc280 0x7f286c0cc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:02.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.760+0000 7f28745ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286c0c93c0 con 0x7f286c157500 2024-01-11T09:46:02.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.760+0000 7f28745ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f286c06c430 con 0x7f286c0cc280 2024-01-11T09:46:02.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.760+0000 7f28745ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286c157c00 con 0x7f286c0c8a30 2024-01-11T09:46:02.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.761+0000 7f2872b9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f286c0cc280 0x7f286c0cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:02.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.761+0000 7f287239b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f286c157500 0x7f286c06b990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42210/0 (socket says 172.21.15.27:42210) 2024-01-11T09:46:02.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.761+0000 7f287239b700 1 -- 172.21.15.27:0/825791318 learned_addr learned my addr 172.21.15.27:0/825791318 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:02.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.761+0000 7f2871b9a700 1 --2- 172.21.15.27:0/825791318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f286c0c8a30 0x7f286c0c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:02.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.761+0000 7f2871399700 1 -- 172.21.15.27:0/825791318 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495911594 0 0) 0x7f286c0c93c0 con 0x7f286c157500 2024-01-11T09:46:02.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.761+0000 7f2871399700 1 -- 172.21.15.27:0/825791318 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2858003570 con 0x7f286c157500 2024-01-11T09:46:02.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.761+0000 7f2872b9c700 1 -- 172.21.15.27:0/825791318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f286c0c8a30 msgr2=0x7f286c0c8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:02.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.761+0000 7f2872b9c700 1 --2- 172.21.15.27:0/825791318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f286c0c8a30 0x7f286c0c8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.761+0000 7f2872b9c700 1 -- 172.21.15.27:0/825791318 >> v1:172.21.15.27:6789/0 conn(0x7f286c157500 legacy=0x7f286c06b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:02.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.761+0000 7f2872b9c700 1 -- 172.21.15.27:0/825791318 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286c157ee0 con 0x7f286c0cc280 2024-01-11T09:46:02.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.762+0000 7f2872b9c700 1 --2- 172.21.15.27:0/825791318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f286c0cc280 0x7f286c0cc700 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f2864009e60 tx=0x7f2864009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=875f640c71c7a784 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:02.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.762+0000 7f2871399700 1 -- 172.21.15.27:0/825791318 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286400e070 con 0x7f286c0cc280 2024-01-11T09:46:02.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.762+0000 7f2871399700 1 -- 172.21.15.27:0/825791318 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286400bab0 con 0x7f286c0cc280 2024-01-11T09:46:02.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.762+0000 7f2871399700 1 -- 172.21.15.27:0/825791318 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286400be70 con 0x7f286c0cc280 2024-01-11T09:46:02.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.763+0000 7f28745ff700 1 -- 172.21.15.27:0/825791318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f286c0cc280 msgr2=0x7f286c0cc700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:02.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.763+0000 7f28745ff700 1 --2- 172.21.15.27:0/825791318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f286c0cc280 0x7f286c0cc700 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f2864009e60 tx=0x7f2864009780 comp rx=0 tx=0).stop 2024-01-11T09:46:02.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.763+0000 7f28745ff700 1 -- 172.21.15.27:0/825791318 shutdown_connections 2024-01-11T09:46:02.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.763+0000 7f28745ff700 1 --2- 172.21.15.27:0/825791318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f286c0cc280 0x7f286c0cc700 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.763+0000 7f28745ff700 1 --2- 172.21.15.27:0/825791318 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f286c0c8a30 0x7f286c0c8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.763+0000 7f28745ff700 1 -- 172.21.15.27:0/825791318 >> 172.21.15.27:0/825791318 conn(0x7f286c0ce3f0 msgr2=0x7f286c0ce7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:02.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.763+0000 7f28745ff700 1 -- 172.21.15.27:0/825791318 shutdown_connections 2024-01-11T09:46:02.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.763+0000 7f28745ff700 1 -- 172.21.15.27:0/825791318 wait complete. 2024-01-11T09:46:02.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.765+0000 7f28745ff700 1 Processor -- start 2024-01-11T09:46:02.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.765+0000 7f28745ff700 1 -- start start 2024-01-11T09:46:02.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.767+0000 7f28745ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f286c0c8a30 0x7f286c0c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:02.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.767+0000 7f28745ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f286c0cc280 0x7f286c0b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:02.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.767+0000 7f28745ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f286c157500 0x7f286c159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:02.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.767+0000 7f28745ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c0688a0 con 0x7f286c0c8a30 2024-01-11T09:46:02.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.767+0000 7f28745ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c0b3810 con 0x7f286c0cc280 2024-01-11T09:46:02.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.767+0000 7f28745ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286c0b3ab0 con 0x7f286c157500 2024-01-11T09:46:02.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.767+0000 7f287239b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f286c0c8a30 0x7f286c0c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:02.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.767+0000 7f287239b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f286c0c8a30 0x7f286c0c7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47872/0 (socket says 172.21.15.27:47872) 2024-01-11T09:46:02.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.767+0000 7f287239b700 1 -- 172.21.15.27:0/2416401285 learned_addr learned my addr 172.21.15.27:0/2416401285 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:02.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.767+0000 7f2872b9c700 1 --2- 172.21.15.27:0/2416401285 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f286c157500 0x7f286c159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:02.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.767+0000 7f2871b9a700 1 --2- 172.21.15.27:0/2416401285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f286c0cc280 0x7f286c0b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:02.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.768+0000 7f287239b700 1 -- 172.21.15.27:0/2416401285 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f286c157500 msgr2=0x7f286c159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:02.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.768+0000 7f287239b700 1 --2- 172.21.15.27:0/2416401285 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f286c157500 0x7f286c159d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.768+0000 7f287239b700 1 -- 172.21.15.27:0/2416401285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f286c0cc280 msgr2=0x7f286c0b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:02.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.768+0000 7f287239b700 1 --2- 172.21.15.27:0/2416401285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f286c0cc280 0x7f286c0b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:02.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.768+0000 7f287239b700 1 -- 172.21.15.27:0/2416401285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2864009a20 con 0x7f286c0c8a30 2024-01-11T09:46:02.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.768+0000 7f287239b700 1 --2- 172.21.15.27:0/2416401285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f286c0c8a30 0x7f286c0c7e50 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f286800c7c0 tx=0x7f2868007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:02.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.768+0000 7f28627fc700 1 -- 172.21.15.27:0/2416401285 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2868005cd0 con 0x7f286c0c8a30 2024-01-11T09:46:02.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.768+0000 7f28627fc700 1 -- 172.21.15.27:0/2416401285 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2868005e30 con 0x7f286c0c8a30 2024-01-11T09:46:02.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.768+0000 7f28745ff700 1 -- 172.21.15.27:0/2416401285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f286c0b3d90 con 0x7f286c0c8a30 2024-01-11T09:46:02.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.768+0000 7f28627fc700 1 -- 172.21.15.27:0/2416401285 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2868007320 con 0x7f286c0c8a30 2024-01-11T09:46:02.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.768+0000 7f28745ff700 1 -- 172.21.15.27:0/2416401285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286c0b4070 con 0x7f286c0c8a30 2024-01-11T09:46:02.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.769+0000 7f28745ff700 1 -- 172.21.15.27:0/2416401285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286c0718d0 con 0x7f286c0c8a30 2024-01-11T09:46:02.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.770+0000 7f28627fc700 1 -- 172.21.15.27:0/2416401285 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 133) v1 ==== 85003+0+0 (secure 0 0 0) 0x7f286801b3b0 con 0x7f286c0c8a30 2024-01-11T09:46:02.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.771+0000 7f28627fc700 1 --2- 172.21.15.27:0/2416401285 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2850067200 0x7f28500696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:02.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.771+0000 7f28627fc700 1 -- 172.21.15.27:0/2416401285 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f28680844f0 con 0x7f286c0c8a30 2024-01-11T09:46:02.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.772+0000 7f2871b9a700 1 -- 172.21.15.27:0/2416401285 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2850067200 msgr2=0x7f28500696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:02.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.772+0000 7f2871b9a700 1 --2- 172.21.15.27:0/2416401285 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2850067200 0x7f28500696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:46:02.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.776+0000 7f28627fc700 1 -- 172.21.15.27:0/2416401285 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2868050f40 con 0x7f286c0c8a30 2024-01-11T09:46:02.822 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:02.830+0000 7f89e6eae1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:46:02.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.973+0000 7f2871b9a700 1 -- 172.21.15.27:0/2416401285 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2850067200 msgr2=0x7f28500696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:02.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:02.973+0000 7f2871b9a700 1 --2- 172.21.15.27:0/2416401285 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2850067200 0x7f28500696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:46:03.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.028+0000 7f28745ff700 1 -- 172.21.15.27:0/2416401285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f286c06c6f0 con 0x7f286c0c8a30 2024-01-11T09:46:03.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.029+0000 7f28627fc700 1 -- 172.21.15.27:0/2416401285 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+5690 (secure 0 0 0) 0x7f2868054560 con 0x7f286c0c8a30 2024-01-11T09:46:03.021 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:03.022 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":117,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2103359888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":2103359888},{"type":"v1","addr":"172.21.15.77:6834","nonce":2103359888}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8795,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2650824649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2650824649},{"type":"v1","addr":"172.21.15.77:6835","nonce":2650824649}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:45:23.637614+0000","modified":"2024-01-11T09:45:27.146191+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10599":{"gid":10599,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1143960902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1143960902},{"type":"v1","addr":"172.21.15.27:6835","nonce":1143960902}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:45:37.460120+0000","modified":"2024-01-11T09:45:40.977702+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10596},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10596":{"gid":10596,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6834/2807751836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2807751836},{"type":"v1","addr":"172.21.15.27:6834","nonce":2807751836}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":14}]} 2024-01-11T09:46:03.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.031+0000 7f28745ff700 1 -- 172.21.15.27:0/2416401285 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2850067200 msgr2=0x7f28500696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:46:03.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.031+0000 7f28745ff700 1 --2- 172.21.15.27:0/2416401285 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2850067200 0x7f28500696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.031+0000 7f28745ff700 1 -- 172.21.15.27:0/2416401285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f286c0c8a30 msgr2=0x7f286c0c7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:03.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.031+0000 7f28745ff700 1 --2- 172.21.15.27:0/2416401285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f286c0c8a30 0x7f286c0c7e50 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f286800c7c0 tx=0x7f2868007e00 comp rx=0 tx=0).stop 2024-01-11T09:46:03.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.031+0000 7f28745ff700 1 -- 172.21.15.27:0/2416401285 shutdown_connections 2024-01-11T09:46:03.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.031+0000 7f28745ff700 1 --2- 172.21.15.27:0/2416401285 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f286c157500 0x7f286c159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.031+0000 7f28745ff700 1 --2- 172.21.15.27:0/2416401285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f286c0c8a30 0x7f286c0c7e50 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.031+0000 7f28745ff700 1 --2- 172.21.15.27:0/2416401285 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2850067200 0x7f28500696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.031+0000 7f28745ff700 1 --2- 172.21.15.27:0/2416401285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f286c0cc280 0x7f286c0b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.031+0000 7f28745ff700 1 -- 172.21.15.27:0/2416401285 >> 172.21.15.27:0/2416401285 conn(0x7f286c0ce3f0 msgr2=0x7f286c154380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:03.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.031+0000 7f28745ff700 1 -- 172.21.15.27:0/2416401285 shutdown_connections 2024-01-11T09:46:03.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.031+0000 7f28745ff700 1 -- 172.21.15.27:0/2416401285 wait complete. 2024-01-11T09:46:03.025 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 117 2024-01-11T09:46:03.042 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:46:03.099 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:03.107+0000 7feb4febe1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:46:03.205 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:03.212+0000 7f89e6eae1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:46:03.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.260+0000 7efdf89ff700 1 Processor -- start 2024-01-11T09:46:03.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.261+0000 7efdf89ff700 1 -- start start 2024-01-11T09:46:03.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.262+0000 7efdf89ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdf4157500 0x7efdf406b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:03.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.262+0000 7efdf89ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efdf406bed0 0x7efdf40c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:03.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.262+0000 7efdf89ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf40cd290 con 0x7efdf40c8f70 2024-01-11T09:46:03.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.262+0000 7efdf89ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf40c93f0 con 0x7efdf406bed0 2024-01-11T09:46:03.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.262+0000 7efdf89ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf406c430 con 0x7efdf4157500 2024-01-11T09:46:03.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.262+0000 7efdf2d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efdf40c8f70 0x7efdf406ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42216/0 (socket says 172.21.15.27:42216) 2024-01-11T09:46:03.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.262+0000 7efdf2d9d700 1 -- 172.21.15.27:0/648731502 learned_addr learned my addr 172.21.15.27:0/648731502 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:03.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.262+0000 7efdf259c700 1 --2- 172.21.15.27:0/648731502 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdf4157500 0x7efdf406b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:03.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.262+0000 7efdf1d9b700 1 --2- 172.21.15.27:0/648731502 >> v2:172.21.15.77:3300/0 conn(0x7efdf406bed0 0x7efdf40c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:03.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.263+0000 7efdf159a700 1 -- 172.21.15.27:0/648731502 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314429447 0 0) 0x7efdf40cd290 con 0x7efdf40c8f70 2024-01-11T09:46:03.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.263+0000 7efdf159a700 1 -- 172.21.15.27:0/648731502 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efddc003570 con 0x7efdf40c8f70 2024-01-11T09:46:03.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.263+0000 7efdf259c700 1 -- 172.21.15.27:0/648731502 >> v2:172.21.15.77:3300/0 conn(0x7efdf406bed0 msgr2=0x7efdf40c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:03.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.263+0000 7efdf259c700 1 --2- 172.21.15.27:0/648731502 >> v2:172.21.15.77:3300/0 conn(0x7efdf406bed0 0x7efdf40c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.263+0000 7efdf259c700 1 -- 172.21.15.27:0/648731502 >> v1:172.21.15.27:6789/0 conn(0x7efdf40c8f70 legacy=0x7efdf406ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:03.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.263+0000 7efdf259c700 1 -- 172.21.15.27:0/648731502 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf4157c00 con 0x7efdf4157500 2024-01-11T09:46:03.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.263+0000 7efdf259c700 1 --2- 172.21.15.27:0/648731502 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdf4157500 0x7efdf406b990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7efde40098e0 tx=0x7efde40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5325669aac52140b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:03.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.264+0000 7efdf159a700 1 -- 172.21.15.27:0/648731502 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde4011630 con 0x7efdf4157500 2024-01-11T09:46:03.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.264+0000 7efdf159a700 1 -- 172.21.15.27:0/648731502 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde4011790 con 0x7efdf4157500 2024-01-11T09:46:03.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.264+0000 7efdf159a700 1 -- 172.21.15.27:0/648731502 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde4011b30 con 0x7efdf4157500 2024-01-11T09:46:03.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.264+0000 7efdf89ff700 1 -- 172.21.15.27:0/648731502 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdf4157500 msgr2=0x7efdf406b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:03.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.264+0000 7efdf89ff700 1 --2- 172.21.15.27:0/648731502 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdf4157500 0x7efdf406b990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7efde40098e0 tx=0x7efde40091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.264+0000 7efdf89ff700 1 -- 172.21.15.27:0/648731502 shutdown_connections 2024-01-11T09:46:03.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.264+0000 7efdf89ff700 1 --2- 172.21.15.27:0/648731502 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdf4157500 0x7efdf406b990 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.264+0000 7efdf89ff700 1 --2- 172.21.15.27:0/648731502 >> v2:172.21.15.77:3300/0 conn(0x7efdf406bed0 0x7efdf40c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.264+0000 7efdf89ff700 1 -- 172.21.15.27:0/648731502 >> 172.21.15.27:0/648731502 conn(0x7efdf40ce3f0 msgr2=0x7efdf40ce7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:03.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.265+0000 7efdf89ff700 1 -- 172.21.15.27:0/648731502 shutdown_connections 2024-01-11T09:46:03.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.265+0000 7efdf89ff700 1 -- 172.21.15.27:0/648731502 wait complete. 2024-01-11T09:46:03.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.266+0000 7efdf89ff700 1 Processor -- start 2024-01-11T09:46:03.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.266+0000 7efdf89ff700 1 -- start start 2024-01-11T09:46:03.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.267+0000 7efdf89ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdf406bed0 0x7efdf413f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:03.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.267+0000 7efdf89ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdf40c8f70 0x7efdf40c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:03.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.267+0000 7efdf89ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdf4157500 0x7efdf4154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:03.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.267+0000 7efdf89ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf4152300 con 0x7efdf406bed0 2024-01-11T09:46:03.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.267+0000 7efdf89ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf40c8980 con 0x7efdf40c8f70 2024-01-11T09:46:03.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.267+0000 7efdf89ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf40c8c60 con 0x7efdf4157500 2024-01-11T09:46:03.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.267+0000 7efdf259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdf406bed0 0x7efdf413f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:03.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.267+0000 7efdf1d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdf40c8f70 0x7efdf40c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:03.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.268+0000 7efdf1d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdf40c8f70 0x7efdf40c7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:46734/0 (socket says 172.21.15.27:46734) 2024-01-11T09:46:03.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.268+0000 7efdf2d9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdf4157500 0x7efdf4154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:03.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.268+0000 7efdf1d9b700 1 -- 172.21.15.27:0/162279327 learned_addr learned my addr 172.21.15.27:0/162279327 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:03.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.268+0000 7efdf259c700 1 -- 172.21.15.27:0/162279327 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdf4157500 msgr2=0x7efdf4154e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:03.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.268+0000 7efdf259c700 1 --2- 172.21.15.27:0/162279327 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdf4157500 0x7efdf4154e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.268+0000 7efdf259c700 1 -- 172.21.15.27:0/162279327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdf40c8f70 msgr2=0x7efdf40c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:03.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.268+0000 7efdf259c700 1 --2- 172.21.15.27:0/162279327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdf40c8f70 0x7efdf40c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.268+0000 7efdf259c700 1 -- 172.21.15.27:0/162279327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde4009420 con 0x7efdf406bed0 2024-01-11T09:46:03.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.268+0000 7efdf1d9b700 1 --2- 172.21.15.27:0/162279327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdf40c8f70 0x7efdf40c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:03.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.268+0000 7efdf259c700 1 --2- 172.21.15.27:0/162279327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdf406bed0 0x7efdf413f7a0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7efde4016000 tx=0x7efde4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:03.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.268+0000 7efde27fc700 1 -- 172.21.15.27:0/162279327 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde4000d10 con 0x7efdf406bed0 2024-01-11T09:46:03.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.269+0000 7efde27fc700 1 -- 172.21.15.27:0/162279327 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde4011b10 con 0x7efdf406bed0 2024-01-11T09:46:03.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.269+0000 7efde27fc700 1 -- 172.21.15.27:0/162279327 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde4025640 con 0x7efdf406bed0 2024-01-11T09:46:03.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.269+0000 7efdf89ff700 1 -- 172.21.15.27:0/162279327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf40b1860 con 0x7efdf406bed0 2024-01-11T09:46:03.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.269+0000 7efdf89ff700 1 -- 172.21.15.27:0/162279327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf40b1e20 con 0x7efdf406bed0 2024-01-11T09:46:03.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.270+0000 7efdf89ff700 1 -- 172.21.15.27:0/162279327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf40b2010 con 0x7efdf406bed0 2024-01-11T09:46:03.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.271+0000 7efde27fc700 1 -- 172.21.15.27:0/162279327 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 133) v1 ==== 85003+0+0 (secure 0 0 0) 0x7efde40193b0 con 0x7efdf406bed0 2024-01-11T09:46:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.272+0000 7efde27fc700 1 --2- 172.21.15.27:0/162279327 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efdd4067200 0x7efdd40696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.272+0000 7efde27fc700 1 -- 172.21.15.27:0/162279327 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7efde40866b0 con 0x7efdf406bed0 2024-01-11T09:46:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.272+0000 7efdf1d9b700 1 -- 172.21.15.27:0/162279327 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efdd4067200 msgr2=0x7efdd40696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.272+0000 7efdf1d9b700 1 --2- 172.21.15.27:0/162279327 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efdd4067200 0x7efdd40696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:46:03.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.276+0000 7efde27fc700 1 -- 172.21.15.27:0/162279327 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efde4053100 con 0x7efdf406bed0 2024-01-11T09:46:03.306 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:03.314+0000 7feb4febe1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:46:03.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.473+0000 7efdf1d9b700 1 -- 172.21.15.27:0/162279327 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efdd4067200 msgr2=0x7efdd40696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:03.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.473+0000 7efdf1d9b700 1 --2- 172.21.15.27:0/162279327 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efdd4067200 0x7efdd40696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:46:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.525+0000 7efdf89ff700 1 -- 172.21.15.27:0/162279327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdf40718d0 con 0x7efdf406bed0 2024-01-11T09:46:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.526+0000 7efde27fc700 1 -- 172.21.15.27:0/162279327 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+5690 (secure 0 0 0) 0x7efde4056720 con 0x7efdf406bed0 2024-01-11T09:46:03.520 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:03.520 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":117,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2103359888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":2103359888},{"type":"v1","addr":"172.21.15.77:6834","nonce":2103359888}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8795,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2650824649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2650824649},{"type":"v1","addr":"172.21.15.77:6835","nonce":2650824649}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:45:23.637614+0000","modified":"2024-01-11T09:45:27.146191+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10599":{"gid":10599,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1143960902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1143960902},{"type":"v1","addr":"172.21.15.27:6835","nonce":1143960902}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:45:37.460120+0000","modified":"2024-01-11T09:45:40.977702+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10596},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10596":{"gid":10596,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6834/2807751836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2807751836},{"type":"v1","addr":"172.21.15.27:6834","nonce":2807751836}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":14}]} 2024-01-11T09:46:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.528+0000 7efdf89ff700 1 -- 172.21.15.27:0/162279327 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efdd4067200 msgr2=0x7efdd40696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:46:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.528+0000 7efdf89ff700 1 --2- 172.21.15.27:0/162279327 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efdd4067200 0x7efdd40696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.528+0000 7efdf89ff700 1 -- 172.21.15.27:0/162279327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdf406bed0 msgr2=0x7efdf413f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.528+0000 7efdf89ff700 1 --2- 172.21.15.27:0/162279327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdf406bed0 0x7efdf413f7a0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7efde4016000 tx=0x7efde4017000 comp rx=0 tx=0).stop 2024-01-11T09:46:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.528+0000 7efdf89ff700 1 -- 172.21.15.27:0/162279327 shutdown_connections 2024-01-11T09:46:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.529+0000 7efdf89ff700 1 --2- 172.21.15.27:0/162279327 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efdf4157500 0x7efdf4154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.529+0000 7efdf89ff700 1 --2- 172.21.15.27:0/162279327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdf406bed0 0x7efdf413f7a0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.529+0000 7efdf89ff700 1 --2- 172.21.15.27:0/162279327 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efdd4067200 0x7efdd40696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.529+0000 7efdf89ff700 1 --2- 172.21.15.27:0/162279327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdf40c8f70 0x7efdf40c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.529+0000 7efdf89ff700 1 -- 172.21.15.27:0/162279327 >> 172.21.15.27:0/162279327 conn(0x7efdf40ce3f0 msgr2=0x7efdf4155dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.529+0000 7efdf89ff700 1 -- 172.21.15.27:0/162279327 shutdown_connections 2024-01-11T09:46:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.529+0000 7efdf89ff700 1 -- 172.21.15.27:0/162279327 wait complete. 2024-01-11T09:46:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 117 2024-01-11T09:46:03.539 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:46:03.630 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:03.638+0000 7f89e6eae1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:46:03.713 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:03.721+0000 7feb4febe1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:46:03.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.749+0000 7f2a04bff700 1 Processor -- start 2024-01-11T09:46:03.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.749+0000 7f2a04bff700 1 -- start start 2024-01-11T09:46:03.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.750+0000 7f2a04bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2a000ca7b0 0x7f2a000cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:03.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.750+0000 7f2a04bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a00159e80 0x7f2a0015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:03.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.750+0000 7f2a04bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0015d640 con 0x7f2a000ca3e0 2024-01-11T09:46:03.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.750+0000 7f2a04bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a0015ae70 con 0x7f2a000ca7b0 2024-01-11T09:46:03.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.750+0000 7f2a04bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a000cc4f0 con 0x7f2a00159e80 2024-01-11T09:46:03.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.750+0000 7f29fdd9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2a000ca7b0 0x7f2a000cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:03.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.751+0000 7f29fed9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a00159e80 0x7f2a0015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:03.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.751+0000 7f29fe59c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2a000ca3e0 0x7f2a000c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:60092/0 (socket says 172.21.15.27:60092) 2024-01-11T09:46:03.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.751+0000 7f29fe59c700 1 -- 172.21.15.27:0/1646953491 learned_addr learned my addr 172.21.15.27:0/1646953491 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:03.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.751+0000 7f29fd59a700 1 -- 172.21.15.27:0/1646953491 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784218242 0 0) 0x7f2a0015d640 con 0x7f2a000ca3e0 2024-01-11T09:46:03.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.751+0000 7f29fd59a700 1 -- 172.21.15.27:0/1646953491 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e8003570 con 0x7f2a000ca3e0 2024-01-11T09:46:03.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.751+0000 7f29fdd9b700 1 -- 172.21.15.27:0/1646953491 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a00159e80 msgr2=0x7f2a0015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:03.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.751+0000 7f29fdd9b700 1 --2- 172.21.15.27:0/1646953491 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a00159e80 0x7f2a0015dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.751+0000 7f29fdd9b700 1 -- 172.21.15.27:0/1646953491 >> v1:172.21.15.77:6789/0 conn(0x7f2a000ca3e0 legacy=0x7f2a000c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:03.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.751+0000 7f29fdd9b700 1 -- 172.21.15.27:0/1646953491 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0015a5d0 con 0x7f2a000ca7b0 2024-01-11T09:46:03.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.751+0000 7f29fed9d700 1 --2- 172.21.15.27:0/1646953491 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a00159e80 0x7f2a0015dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:03.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.751+0000 7f29fdd9b700 1 --2- 172.21.15.27:0/1646953491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a000ca7b0 0x7f2a000cca20 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009940 tx=0x7f29f40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de2f9978154e7abb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:03.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.752+0000 7f29fd59a700 1 -- 172.21.15.27:0/1646953491 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f4011630 con 0x7f2a000ca7b0 2024-01-11T09:46:03.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.752+0000 7f29fd59a700 1 -- 172.21.15.27:0/1646953491 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f4011790 con 0x7f2a000ca7b0 2024-01-11T09:46:03.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.752+0000 7f29fd59a700 1 -- 172.21.15.27:0/1646953491 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f4011b50 con 0x7f2a000ca7b0 2024-01-11T09:46:03.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.752+0000 7f2a04bff700 1 -- 172.21.15.27:0/1646953491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a000ca7b0 msgr2=0x7f2a000cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:03.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.752+0000 7f2a04bff700 1 --2- 172.21.15.27:0/1646953491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a000ca7b0 0x7f2a000cca20 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009940 tx=0x7f29f40091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.752+0000 7f2a04bff700 1 -- 172.21.15.27:0/1646953491 shutdown_connections 2024-01-11T09:46:03.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.752+0000 7f2a04bff700 1 --2- 172.21.15.27:0/1646953491 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a00159e80 0x7f2a0015dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.752+0000 7f2a04bff700 1 --2- 172.21.15.27:0/1646953491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a000ca7b0 0x7f2a000cca20 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.752+0000 7f2a04bff700 1 -- 172.21.15.27:0/1646953491 >> 172.21.15.27:0/1646953491 conn(0x7f2a000ce3e0 msgr2=0x7f2a000ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:03.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.753+0000 7f2a04bff700 1 -- 172.21.15.27:0/1646953491 shutdown_connections 2024-01-11T09:46:03.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.753+0000 7f2a04bff700 1 -- 172.21.15.27:0/1646953491 wait complete. 2024-01-11T09:46:03.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.754+0000 7f2a04bff700 1 Processor -- start 2024-01-11T09:46:03.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.754+0000 7f2a04bff700 1 -- start start 2024-01-11T09:46:03.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.755+0000 7f2a04bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a000ca3e0 0x7f2a0016aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:03.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.755+0000 7f2a04bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a000ca7b0 0x7f2a0014a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:03.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.755+0000 7f2a04bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a00159e80 0x7f2a000c4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:03.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.755+0000 7f2a04bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0014b5c0 con 0x7f2a000ca3e0 2024-01-11T09:46:03.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.755+0000 7f2a04bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0014a9b0 con 0x7f2a000ca7b0 2024-01-11T09:46:03.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.755+0000 7f2a04bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0014ac90 con 0x7f2a00159e80 2024-01-11T09:46:03.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.755+0000 7f29fe59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a000ca3e0 0x7f2a0016aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:03.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.755+0000 7f29fdd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a000ca7b0 0x7f2a0014a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:03.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.755+0000 7f29fe59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a000ca3e0 0x7f2a0016aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47884/0 (socket says 172.21.15.27:47884) 2024-01-11T09:46:03.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.755+0000 7f29fe59c700 1 -- 172.21.15.27:0/1923353384 learned_addr learned my addr 172.21.15.27:0/1923353384 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:03.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.755+0000 7f29fed9d700 1 --2- 172.21.15.27:0/1923353384 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a00159e80 0x7f2a000c4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:03.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.756+0000 7f29fe59c700 1 -- 172.21.15.27:0/1923353384 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a00159e80 msgr2=0x7f2a000c4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:03.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.756+0000 7f29fe59c700 1 --2- 172.21.15.27:0/1923353384 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a00159e80 0x7f2a000c4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.756+0000 7f29fe59c700 1 -- 172.21.15.27:0/1923353384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a000ca7b0 msgr2=0x7f2a0014a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:03.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.756+0000 7f29fe59c700 1 --2- 172.21.15.27:0/1923353384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a000ca7b0 0x7f2a0014a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.756+0000 7f29fe59c700 1 -- 172.21.15.27:0/1923353384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f8009a20 con 0x7f2a000ca3e0 2024-01-11T09:46:03.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.756+0000 7f29fe59c700 1 --2- 172.21.15.27:0/1923353384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a000ca3e0 0x7f2a0016aae0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f29f000c7c0 tx=0x7f29f0007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:03.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.756+0000 7f29ee7fc700 1 -- 172.21.15.27:0/1923353384 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f0005b30 con 0x7f2a000ca3e0 2024-01-11T09:46:03.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.756+0000 7f29ee7fc700 1 -- 172.21.15.27:0/1923353384 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f0005cb0 con 0x7f2a000ca3e0 2024-01-11T09:46:03.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.756+0000 7f29ee7fc700 1 -- 172.21.15.27:0/1923353384 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f0007320 con 0x7f2a000ca3e0 2024-01-11T09:46:03.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.756+0000 7f2a04bff700 1 -- 172.21.15.27:0/1923353384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f4009420 con 0x7f2a000ca3e0 2024-01-11T09:46:03.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.757+0000 7f2a04bff700 1 -- 172.21.15.27:0/1923353384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0014b250 con 0x7f2a000ca3e0 2024-01-11T09:46:03.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.758+0000 7f2a04bff700 1 -- 172.21.15.27:0/1923353384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a000718d0 con 0x7f2a000ca3e0 2024-01-11T09:46:03.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.759+0000 7f29ee7fc700 1 -- 172.21.15.27:0/1923353384 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 133) v1 ==== 85003+0+0 (secure 0 0 0) 0x7f29f001b3b0 con 0x7f2a000ca3e0 2024-01-11T09:46:03.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.760+0000 7f29ee7fc700 1 --2- 172.21.15.27:0/1923353384 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f29e0067200 0x7f29e00696b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:03.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.760+0000 7f29ee7fc700 1 -- 172.21.15.27:0/1923353384 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6939+0+0 (secure 0 0 0) 0x7f29f00842c0 con 0x7f2a000ca3e0 2024-01-11T09:46:03.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.760+0000 7f29fdd9b700 1 -- 172.21.15.27:0/1923353384 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f29e0067200 msgr2=0x7f29e00696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:03.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.760+0000 7f29fdd9b700 1 --2- 172.21.15.27:0/1923353384 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f29e0067200 0x7f29e00696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:46:03.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.764+0000 7f29ee7fc700 1 -- 172.21.15.27:0/1923353384 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f29f0050dd0 con 0x7f2a000ca3e0 2024-01-11T09:46:03.815 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:03.823+0000 7f89e6eae1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:46:03.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.961+0000 7f29fdd9b700 1 -- 172.21.15.27:0/1923353384 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f29e0067200 msgr2=0x7f29e00696b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:03.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.961+0000 7f29fdd9b700 1 --2- 172.21.15.27:0/1923353384 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f29e0067200 0x7f29e00696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:46:03.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.971+0000 7f2a04bff700 1 -- 172.21.15.27:0/1923353384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a0015a8b0 con 0x7f2a000ca3e0 2024-01-11T09:46:03.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.972+0000 7f29ee7fc700 1 -- 172.21.15.27:0/1923353384 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v219) v1 ==== 74+0+19093 (secure 0 0 0) 0x7f29f00543f0 con 0x7f2a000ca3e0 2024-01-11T09:46:03.965 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:03.965 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":219,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:45:52.192269+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:45:21.455328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-01-11T09:45:22.634298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-01-11T09:45:35.631727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-01-11T09:45:36.452195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/2192171652":"2024-01-12T09:45:52.192235+0000","172.21.15.77:6837/59776":"2024-01-12T09:45:52.192235+0000","172.21.15.77:6836/59776":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/2654175097":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/652362578":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/2439942664":"2024-01-12T09:45:52.192235+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:46:03.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.974+0000 7f2a04bff700 1 -- 172.21.15.27:0/1923353384 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f29e0067200 msgr2=0x7f29e00696b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:46:03.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.974+0000 7f2a04bff700 1 --2- 172.21.15.27:0/1923353384 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f29e0067200 0x7f29e00696b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.974+0000 7f2a04bff700 1 -- 172.21.15.27:0/1923353384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a000ca3e0 msgr2=0x7f2a0016aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:03.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.974+0000 7f2a04bff700 1 --2- 172.21.15.27:0/1923353384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a000ca3e0 0x7f2a0016aae0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f29f000c7c0 tx=0x7f29f0007e00 comp rx=0 tx=0).stop 2024-01-11T09:46:03.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.974+0000 7f2a04bff700 1 -- 172.21.15.27:0/1923353384 shutdown_connections 2024-01-11T09:46:03.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.974+0000 7f2a04bff700 1 --2- 172.21.15.27:0/1923353384 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a00159e80 0x7f2a000c4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.974+0000 7f2a04bff700 1 --2- 172.21.15.27:0/1923353384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a000ca3e0 0x7f2a0016aae0 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.974+0000 7f2a04bff700 1 --2- 172.21.15.27:0/1923353384 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f29e0067200 0x7f29e00696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.974+0000 7f2a04bff700 1 --2- 172.21.15.27:0/1923353384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a000ca7b0 0x7f2a0014a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:03.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.974+0000 7f2a04bff700 1 -- 172.21.15.27:0/1923353384 >> 172.21.15.27:0/1923353384 conn(0x7f2a000ce3e0 msgr2=0x7f2a00069790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:03.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.974+0000 7f2a04bff700 1 -- 172.21.15.27:0/1923353384 shutdown_connections 2024-01-11T09:46:03.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:03.975+0000 7f2a04bff700 1 -- 172.21.15.27:0/1923353384 wait complete. 2024-01-11T09:46:03.985 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-11T09:46:04.005 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.013+0000 7f89e6eae1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:46:04.116 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.123+0000 7f89e6eae1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:46:04.185 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:04.192+0000 7feb4febe1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:46:04.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.207+0000 7f9ed21ff700 1 Processor -- start 2024-01-11T09:46:04.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.207+0000 7f9ed21ff700 1 -- start start 2024-01-11T09:46:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.208+0000 7f9ed21ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ecc0b0a40 0x7f9ecc15da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.208+0000 7f9ed21ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9ecc15df40 0x7f9ecc068720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.208+0000 7f9ed21ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ecc15d290 con 0x7f9ecc06bf10 2024-01-11T09:46:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.208+0000 7f9ed21ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ecc15d570 con 0x7f9ecc15df40 2024-01-11T09:46:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.208+0000 7f9ed21ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc154180 con 0x7f9ecc0b0a40 2024-01-11T09:46:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.209+0000 7f9ecaffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ecc0b0a40 0x7f9ecc15da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.209+0000 7f9ecb7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9ecc06bf10 0x7f9ecc159c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:58980/0 (socket says 172.21.15.27:58980) 2024-01-11T09:46:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.209+0000 7f9ecb7fe700 1 -- 172.21.15.27:0/2739712194 learned_addr learned my addr 172.21.15.27:0/2739712194 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.209+0000 7f9ecbfff700 1 --2- 172.21.15.27:0/2739712194 >> v2:172.21.15.77:3300/0 conn(0x7f9ecc15df40 0x7f9ecc068720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.209+0000 7f9ecaffd700 1 -- 172.21.15.27:0/2739712194 >> v2:172.21.15.77:3300/0 conn(0x7f9ecc15df40 msgr2=0x7f9ecc068720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.209+0000 7f9ecaffd700 1 --2- 172.21.15.27:0/2739712194 >> v2:172.21.15.77:3300/0 conn(0x7f9ecc15df40 0x7f9ecc068720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.209+0000 7f9ecaffd700 1 -- 172.21.15.27:0/2739712194 >> v1:172.21.15.77:6789/0 conn(0x7f9ecc06bf10 legacy=0x7f9ecc159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:04.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.209+0000 7f9ecaffd700 1 -- 172.21.15.27:0/2739712194 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ecc15a510 con 0x7f9ecc0b0a40 2024-01-11T09:46:04.204 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.209+0000 7f8961cc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.204 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.209+0000 7f8961cc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.204 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.209+0000 7f8961cc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.204 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.209+0000 7f8961cc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.205 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.209+0000 7f8961cc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.205 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.210+0000 7f8964cca700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.205 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.210+0000 7f8964cca700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.205 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.210+0000 7f8964cca700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.205 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.210+0000 7f8964cca700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.205 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.210+0000 7f8964cca700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.210+0000 7f9ecaffd700 1 --2- 172.21.15.27:0/2739712194 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ecc0b0a40 0x7f9ecc15da00 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9ec00098e0 tx=0x7f9ec00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad2651cca8f20969 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:04.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.210+0000 7f9eca7fc700 1 -- 172.21.15.27:0/2739712194 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec000b640 con 0x7f9ecc0b0a40 2024-01-11T09:46:04.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.211+0000 7f9eca7fc700 1 -- 172.21.15.27:0/2739712194 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec000b7a0 con 0x7f9ecc0b0a40 2024-01-11T09:46:04.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.211+0000 7f9eca7fc700 1 -- 172.21.15.27:0/2739712194 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec000bb40 con 0x7f9ecc0b0a40 2024-01-11T09:46:04.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.211+0000 7f9ed21ff700 1 -- 172.21.15.27:0/2739712194 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ecc0b0a40 msgr2=0x7f9ecc15da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:04.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.211+0000 7f9ed21ff700 1 --2- 172.21.15.27:0/2739712194 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ecc0b0a40 0x7f9ecc15da00 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9ec00098e0 tx=0x7f9ec00091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:04.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.211+0000 7f9ed21ff700 1 -- 172.21.15.27:0/2739712194 shutdown_connections 2024-01-11T09:46:04.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.211+0000 7f9ed21ff700 1 --2- 172.21.15.27:0/2739712194 >> v2:172.21.15.77:3300/0 conn(0x7f9ecc15df40 0x7f9ecc068720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:04.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.211+0000 7f9ed21ff700 1 --2- 172.21.15.27:0/2739712194 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ecc0b0a40 0x7f9ecc15da00 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:04.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.211+0000 7f9ed21ff700 1 -- 172.21.15.27:0/2739712194 >> 172.21.15.27:0/2739712194 conn(0x7f9ecc0ce3c0 msgr2=0x7f9ecc0ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:04.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.211+0000 7f9ed21ff700 1 -- 172.21.15.27:0/2739712194 shutdown_connections 2024-01-11T09:46:04.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.212+0000 7f9ed21ff700 1 -- 172.21.15.27:0/2739712194 wait complete. 2024-01-11T09:46:04.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.213+0000 7f9ed21ff700 1 Processor -- start 2024-01-11T09:46:04.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.214+0000 7f9ed21ff700 1 -- start start 2024-01-11T09:46:04.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.215+0000 7f9ed21ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ecc06bf10 0x7f9ecc0c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:04.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.215+0000 7f9ed21ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ecc0b0a40 0x7f9ecc0c8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:04.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.215+0000 7f9ed21ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ecc15df40 0x7f9ecc0c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:04.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.215+0000 7f9ed21ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc0c9280 con 0x7f9ecc0b0a40 2024-01-11T09:46:04.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.215+0000 7f9ed21ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc0c1aa0 con 0x7f9ecc15df40 2024-01-11T09:46:04.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.215+0000 7f9ed21ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc0c2060 con 0x7f9ecc06bf10 2024-01-11T09:46:04.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.215+0000 7f9ecaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ecc0b0a40 0x7f9ecc0c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:04.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.215+0000 7f9ecb7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ecc06bf10 0x7f9ecc0c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.215+0000 7f9ecaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ecc0b0a40 0x7f9ecc0c8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42672/0 (socket says 172.21.15.27:42672) 2024-01-11T09:46:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.215+0000 7f9ecaffd700 1 -- 172.21.15.27:0/1578296440 learned_addr learned my addr 172.21.15.27:0/1578296440 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.215+0000 7f9ecbfff700 1 --2- 172.21.15.27:0/1578296440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ecc15df40 0x7f9ecc0c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.215+0000 7f9ecaffd700 1 -- 172.21.15.27:0/1578296440 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ecc06bf10 msgr2=0x7f9ecc0c6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.215+0000 7f9ecaffd700 1 --2- 172.21.15.27:0/1578296440 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ecc06bf10 0x7f9ecc0c6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.215+0000 7f9ecaffd700 1 -- 172.21.15.27:0/1578296440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ecc15df40 msgr2=0x7f9ecc0c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.215+0000 7f9ecaffd700 1 --2- 172.21.15.27:0/1578296440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ecc15df40 0x7f9ecc0c8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.215+0000 7f9ecaffd700 1 -- 172.21.15.27:0/1578296440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec0009420 con 0x7f9ecc0b0a40 2024-01-11T09:46:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.215+0000 7f9ecbfff700 1 --2- 172.21.15.27:0/1578296440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ecc15df40 0x7f9ecc0c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.216+0000 7f9ecaffd700 1 --2- 172.21.15.27:0/1578296440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ecc0b0a40 0x7f9ecc0c8800 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc1542c0 tx=0x7f9ec0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:04.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.216+0000 7f9eb37fe700 1 -- 172.21.15.27:0/1578296440 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec001bc20 con 0x7f9ecc0b0a40 2024-01-11T09:46:04.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.216+0000 7f9eb37fe700 1 -- 172.21.15.27:0/1578296440 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec001bd80 con 0x7f9ecc0b0a40 2024-01-11T09:46:04.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.216+0000 7f9ed21ff700 1 -- 172.21.15.27:0/1578296440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ecc0c4d60 con 0x7f9ecc0b0a40 2024-01-11T09:46:04.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.216+0000 7f9eb37fe700 1 -- 172.21.15.27:0/1578296440 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec0011670 con 0x7f9ecc0b0a40 2024-01-11T09:46:04.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.216+0000 7f9ed21ff700 1 -- 172.21.15.27:0/1578296440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ecc0c1cf0 con 0x7f9ecc0b0a40 2024-01-11T09:46:04.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.217+0000 7f9ed21ff700 1 -- 172.21.15.27:0/1578296440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ecc0cef20 con 0x7f9ecc0b0a40 2024-01-11T09:46:04.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.219+0000 7f9eb37fe700 1 -- 172.21.15.27:0/1578296440 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 134) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f9ec0011a20 con 0x7f9ecc0b0a40 2024-01-11T09:46:04.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.220+0000 7f9eb37fe700 1 -- 172.21.15.27:0/1578296440 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 7326+0+0 (secure 0 0 0) 0x7f9ec0085ad0 con 0x7f9ecc0b0a40 2024-01-11T09:46:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.224+0000 7f9eb37fe700 1 -- 172.21.15.27:0/1578296440 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9ec0052ab0 con 0x7f9ecc0b0a40 2024-01-11T09:46:04.217 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.224+0000 7f8961cc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.217 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.224+0000 7f8961cc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.217 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.224+0000 7f8961cc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.218 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.224+0000 7f8961cc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.218 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.224+0000 7f8961cc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.218 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.226+0000 7f8964cca700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.218 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.226+0000 7f8964cca700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.219 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.226+0000 7f8964cca700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.219 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.226+0000 7f8964cca700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.219 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:04.226+0000 7f8964cca700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:04.390 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:04.397+0000 7feb4febe1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:46:04.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.480+0000 7f9ed21ff700 1 -- 172.21.15.27:0/1578296440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9ecc0c21d0 con 0x7f9ecc0b0a40 2024-01-11T09:46:04.473 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-11T09:46:04.480+0000 7f27065f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-11T09:46:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.488+0000 7f9eb37fe700 1 -- 172.21.15.27:0/1578296440 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v118) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9ec00560d0 con 0x7f9ecc0b0a40 2024-01-11T09:46:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.491+0000 7f9ed21ff700 1 -- 172.21.15.27:0/1578296440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ecc0b0a40 msgr2=0x7f9ecc0c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.491+0000 7f9ed21ff700 1 --2- 172.21.15.27:0/1578296440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ecc0b0a40 0x7f9ecc0c8800 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc1542c0 tx=0x7f9ec0017000 comp rx=0 tx=0).stop 2024-01-11T09:46:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.491+0000 7f9ed21ff700 1 -- 172.21.15.27:0/1578296440 shutdown_connections 2024-01-11T09:46:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.491+0000 7f9ed21ff700 1 --2- 172.21.15.27:0/1578296440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ecc15df40 0x7f9ecc0c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.491+0000 7f9ed21ff700 1 --2- 172.21.15.27:0/1578296440 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9ecc06bf10 0x7f9ecc0c6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.491+0000 7f9ed21ff700 1 --2- 172.21.15.27:0/1578296440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ecc0b0a40 0x7f9ecc0c8800 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.491+0000 7f9ed21ff700 1 -- 172.21.15.27:0/1578296440 >> 172.21.15.27:0/1578296440 conn(0x7f9ecc0ce3c0 msgr2=0x7f9ecc0b3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.491+0000 7f9ed21ff700 1 -- 172.21.15.27:0/1578296440 shutdown_connections 2024-01-11T09:46:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.492+0000 7f9ed21ff700 1 -- 172.21.15.27:0/1578296440 wait complete. 2024-01-11T09:46:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-11T09:46:04.488 INFO:tasks.ceph.mds.a.smithi027.stderr: -3290> 2024-01-11T09:45:30.108+0000 7f566a1f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T09:46:04.511 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-11T09:46:04.543 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T09:46:04.604 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:04.612+0000 7feb4febe1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:46:04.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.728+0000 7fa35b3ff700 1 Processor -- start 2024-01-11T09:46:04.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.728+0000 7fa35b3ff700 1 -- start start 2024-01-11T09:46:04.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.729+0000 7fa35b3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa3541622d0 0x7fa3541646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:04.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.729+0000 7fa35b3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3540cf850 con 0x7fa3540ccd70 2024-01-11T09:46:04.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.729+0000 7fa35b3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3540cec40 con 0x7fa35415a1b0 2024-01-11T09:46:04.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.729+0000 7fa35b3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa35415ac90 con 0x7fa3541622d0 2024-01-11T09:46:04.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.730+0000 7fa35899a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa3540ccd70 0x7fa3540cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37178/0 (socket says 172.21.15.27:37178) 2024-01-11T09:46:04.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.730+0000 7fa35899a700 1 -- 172.21.15.27:0/1534126431 learned_addr learned my addr 172.21.15.27:0/1534126431 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:04.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.730+0000 7fa35999c700 1 --2- 172.21.15.27:0/1534126431 >> v2:172.21.15.77:3300/0 conn(0x7fa3541622d0 0x7fa3541646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.730+0000 7fa34bfff700 1 -- 172.21.15.27:0/1534126431 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125034881 0 0) 0x7fa3540cf850 con 0x7fa3540ccd70 2024-01-11T09:46:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.730+0000 7fa34bfff700 1 -- 172.21.15.27:0/1534126431 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa340003570 con 0x7fa3540ccd70 2024-01-11T09:46:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.730+0000 7fa35999c700 1 -- 172.21.15.27:0/1534126431 >> v1:172.21.15.77:6789/0 conn(0x7fa35415a1b0 legacy=0x7fa35415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.730+0000 7fa34bfff700 1 -- 172.21.15.27:0/1534126431 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893566489 0 0) 0x7fa3540cec40 con 0x7fa35415a1b0 2024-01-11T09:46:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.730+0000 7fa35999c700 1 -- 172.21.15.27:0/1534126431 >> v1:172.21.15.27:6789/0 conn(0x7fa3540ccd70 legacy=0x7fa3540cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.730+0000 7fa35999c700 1 -- 172.21.15.27:0/1534126431 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3540cef20 con 0x7fa3541622d0 2024-01-11T09:46:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.731+0000 7fa35999c700 1 --2- 172.21.15.27:0/1534126431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3541622d0 0x7fa3541646b0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fa34c009e60 tx=0x7fa34c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78cf8c45a0c82267 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.731+0000 7fa34bfff700 1 -- 172.21.15.27:0/1534126431 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34c00e070 con 0x7fa3541622d0 2024-01-11T09:46:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.731+0000 7fa34bfff700 1 -- 172.21.15.27:0/1534126431 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34c00b850 con 0x7fa3541622d0 2024-01-11T09:46:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.731+0000 7fa34bfff700 1 -- 172.21.15.27:0/1534126431 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa344001470 con 0x7fa3541622d0 2024-01-11T09:46:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.731+0000 7fa35b3ff700 1 -- 172.21.15.27:0/1534126431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3541622d0 msgr2=0x7fa3541646b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.731+0000 7fa35b3ff700 1 --2- 172.21.15.27:0/1534126431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3541622d0 0x7fa3541646b0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fa34c009e60 tx=0x7fa34c009780 comp rx=0 tx=0).stop 2024-01-11T09:46:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.732+0000 7fa35b3ff700 1 -- 172.21.15.27:0/1534126431 shutdown_connections 2024-01-11T09:46:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.732+0000 7fa35b3ff700 1 --2- 172.21.15.27:0/1534126431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3541622d0 0x7fa3541646b0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.732+0000 7fa35b3ff700 1 -- 172.21.15.27:0/1534126431 >> 172.21.15.27:0/1534126431 conn(0x7fa35406ed40 msgr2=0x7fa35406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.732+0000 7fa35b3ff700 1 -- 172.21.15.27:0/1534126431 shutdown_connections 2024-01-11T09:46:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.732+0000 7fa35b3ff700 1 -- 172.21.15.27:0/1534126431 wait complete. 2024-01-11T09:46:04.726 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:04.733+0000 7feb4febe1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:46:04.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.733+0000 7fa35b3ff700 1 Processor -- start 2024-01-11T09:46:04.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.733+0000 7fa35b3ff700 1 -- start start 2024-01-11T09:46:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.734+0000 7fa35b3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3540ccd70 0x7fa35414fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.734+0000 7fa35b3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa35415a1b0 0x7fa3540b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.734+0000 7fa35b3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3541622d0 0x7fa3540b38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.734+0000 7fa35b3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3540b4760 con 0x7fa3541622d0 2024-01-11T09:46:04.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.734+0000 7fa35b3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3540c9700 con 0x7fa3540ccd70 2024-01-11T09:46:04.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.734+0000 7fa35b3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3540c99e0 con 0x7fa35415a1b0 2024-01-11T09:46:04.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.735+0000 7fa35999c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3541622d0 0x7fa3540b38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:04.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.735+0000 7fa35919b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3540ccd70 0x7fa35414fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:04.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.735+0000 7fa35899a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa35415a1b0 0x7fa3540b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:04.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.735+0000 7fa35919b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3540ccd70 0x7fa35414fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:48406/0 (socket says 172.21.15.27:48406) 2024-01-11T09:46:04.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.735+0000 7fa35919b700 1 -- 172.21.15.27:0/3591582154 learned_addr learned my addr 172.21.15.27:0/3591582154 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:04.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.735+0000 7fa35999c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3541622d0 0x7fa3540b38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42696/0 (socket says 172.21.15.27:42696) 2024-01-11T09:46:04.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.735+0000 7fa35899a700 1 -- 172.21.15.27:0/3591582154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3540ccd70 msgr2=0x7fa35414fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:04.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.735+0000 7fa35899a700 1 --2- 172.21.15.27:0/3591582154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3540ccd70 0x7fa35414fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:04.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.735+0000 7fa35899a700 1 -- 172.21.15.27:0/3591582154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3541622d0 msgr2=0x7fa3540b38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:04.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.735+0000 7fa35899a700 1 --2- 172.21.15.27:0/3591582154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3541622d0 0x7fa3540b38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:04.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.735+0000 7fa35899a700 1 -- 172.21.15.27:0/3591582154 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34c009a20 con 0x7fa35415a1b0 2024-01-11T09:46:04.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.736+0000 7fa35899a700 1 --2- 172.21.15.27:0/3591582154 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa35415a1b0 0x7fa3540b3380 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa34400cb60 tx=0x7fa344001a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:04.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.736+0000 7fa3497fa700 1 -- 172.21.15.27:0/3591582154 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa344007150 con 0x7fa35415a1b0 2024-01-11T09:46:04.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.736+0000 7fa3497fa700 1 -- 172.21.15.27:0/3591582154 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3440072d0 con 0x7fa35415a1b0 2024-01-11T09:46:04.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.736+0000 7fa3497fa700 1 -- 172.21.15.27:0/3591582154 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa344007690 con 0x7fa35415a1b0 2024-01-11T09:46:04.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.736+0000 7fa35b3ff700 1 -- 172.21.15.27:0/3591582154 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3540c9cc0 con 0x7fa35415a1b0 2024-01-11T09:46:04.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.736+0000 7fa35b3ff700 1 -- 172.21.15.27:0/3591582154 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3540ca280 con 0x7fa35415a1b0 2024-01-11T09:46:04.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.738+0000 7fa35b3ff700 1 -- 172.21.15.27:0/3591582154 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa35404c3c0 con 0x7fa35415a1b0 2024-01-11T09:46:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.738+0000 7fa3497fa700 1 -- 172.21.15.27:0/3591582154 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 134) v1 ==== 84453+0+0 (secure 0 0 0) 0x7fa344004a40 con 0x7fa35415a1b0 2024-01-11T09:46:04.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.740+0000 7fa3497fa700 1 -- 172.21.15.27:0/3591582154 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7412+0+0 (secure 0 0 0) 0x7fa344083a50 con 0x7fa35415a1b0 2024-01-11T09:46:04.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.744+0000 7fa3497fa700 1 -- 172.21.15.27:0/3591582154 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa344050960 con 0x7fa35415a1b0 2024-01-11T09:46:04.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:04.994+0000 7fa35b3ff700 1 -- 172.21.15.27:0/3591582154 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa3540cae50 con 0x7fa35415a1b0 2024-01-11T09:46:05.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.143+0000 7fa3497fa700 1 -- 172.21.15.27:0/3591582154 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mgrmap(e 135) v1 ==== 84580+0+0 (secure 0 0 0) 0x7fa344004cf0 con 0x7fa35415a1b0 2024-01-11T09:46:05.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.143+0000 7fa3497fa700 1 --2- 172.21.15.27:0/3591582154 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa33806f070 0x7fa338071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:05.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.144+0000 7fa35919b700 1 --2- 172.21.15.27:0/3591582154 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa33806f070 0x7fa338071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.147+0000 7fa35919b700 1 --2- 172.21.15.27:0/3591582154 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa33806f070 0x7fa338071450 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa350007fd0 tx=0x7fa350007df0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:05.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.488+0000 7fa3497fa700 1 -- 172.21.15.27:0/3591582154 <== mon.2 v2:172.21.15.77:3301/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v119) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa34404f080 con 0x7fa35415a1b0 2024-01-11T09:46:05.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.491+0000 7fa35b3ff700 1 -- 172.21.15.27:0/3591582154 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa33806f070 msgr2=0x7fa338071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:05.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.491+0000 7fa35b3ff700 1 --2- 172.21.15.27:0/3591582154 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa33806f070 0x7fa338071450 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa350007fd0 tx=0x7fa350007df0 comp rx=0 tx=0).stop 2024-01-11T09:46:05.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.491+0000 7fa35b3ff700 1 -- 172.21.15.27:0/3591582154 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa35415a1b0 msgr2=0x7fa3540b3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:05.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.491+0000 7fa35b3ff700 1 --2- 172.21.15.27:0/3591582154 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa35415a1b0 0x7fa3540b3380 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa34400cb60 tx=0x7fa344001a60 comp rx=0 tx=0).stop 2024-01-11T09:46:05.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.491+0000 7fa35b3ff700 1 -- 172.21.15.27:0/3591582154 shutdown_connections 2024-01-11T09:46:05.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.491+0000 7fa35b3ff700 1 --2- 172.21.15.27:0/3591582154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3541622d0 0x7fa3540b38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:05.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.491+0000 7fa35b3ff700 1 --2- 172.21.15.27:0/3591582154 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa33806f070 0x7fa338071450 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:05.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.491+0000 7fa35b3ff700 1 --2- 172.21.15.27:0/3591582154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3540ccd70 0x7fa35414fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:05.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.491+0000 7fa35b3ff700 1 --2- 172.21.15.27:0/3591582154 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa35415a1b0 0x7fa3540b3380 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:05.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.491+0000 7fa35b3ff700 1 -- 172.21.15.27:0/3591582154 >> 172.21.15.27:0/3591582154 conn(0x7fa35406ed40 msgr2=0x7fa35415a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:05.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.492+0000 7fa35b3ff700 1 -- 172.21.15.27:0/3591582154 shutdown_connections 2024-01-11T09:46:05.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.492+0000 7fa35b3ff700 1 -- 172.21.15.27:0/3591582154 wait complete. 2024-01-11T09:46:05.504 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-11T09:46:05.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.759+0000 7f8b4bdff700 1 Processor -- start 2024-01-11T09:46:05.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.759+0000 7f8b4bdff700 1 -- start start 2024-01-11T09:46:05.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.761+0000 7f8b4bdff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b4415a7a0 0x7f8b4415c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:05.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.761+0000 7f8b4bdff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8b44159e60 0x7f8b4415aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:05.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.761+0000 7f8b4bdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b4415b400 0x7f8b440685e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.761+0000 7f8b4bdff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b4415c4b0 con 0x7f8b4415a7a0 2024-01-11T09:46:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.761+0000 7f8b4bdff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b4415b8a0 con 0x7f8b44159e60 2024-01-11T09:46:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.761+0000 7f8b4bdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4415bb80 con 0x7f8b4415b400 2024-01-11T09:46:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.761+0000 7f8b49b9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b4415a7a0 0x7f8b4415c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.761+0000 7f8b49b9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b4415a7a0 0x7f8b4415c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42706/0 (socket says 172.21.15.27:42706) 2024-01-11T09:46:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.761+0000 7f8b49b9b700 1 -- 172.21.15.27:0/3919294571 learned_addr learned my addr 172.21.15.27:0/3919294571 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.761+0000 7f8b4939a700 1 --2- 172.21.15.27:0/3919294571 >> v2:172.21.15.77:3300/0 conn(0x7f8b44159e60 0x7f8b4415aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.761+0000 7f8b4a39c700 1 --2- 172.21.15.27:0/3919294571 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b4415b400 0x7f8b440685e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.761+0000 7f8b49b9b700 1 -- 172.21.15.27:0/3919294571 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b4415b400 msgr2=0x7f8b440685e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.761+0000 7f8b49b9b700 1 --2- 172.21.15.27:0/3919294571 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b4415b400 0x7f8b440685e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.761+0000 7f8b49b9b700 1 -- 172.21.15.27:0/3919294571 >> v2:172.21.15.77:3300/0 conn(0x7f8b44159e60 msgr2=0x7f8b4415aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.761+0000 7f8b49b9b700 1 --2- 172.21.15.27:0/3919294571 >> v2:172.21.15.77:3300/0 conn(0x7f8b44159e60 0x7f8b4415aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.761+0000 7f8b49b9b700 1 -- 172.21.15.27:0/3919294571 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4415be60 con 0x7f8b4415a7a0 2024-01-11T09:46:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.761+0000 7f8b4939a700 1 --2- 172.21.15.27:0/3919294571 >> v2:172.21.15.77:3300/0 conn(0x7f8b44159e60 0x7f8b4415aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:05.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.762+0000 7f8b49b9b700 1 --2- 172.21.15.27:0/3919294571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4415a7a0 0x7f8b4415c940 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f8b440ce530 tx=0x7f8b40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af74322f2d474fbf server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:05.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.762+0000 7f8b48b99700 1 -- 172.21.15.27:0/3919294571 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4000b860 con 0x7f8b4415a7a0 2024-01-11T09:46:05.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.762+0000 7f8b48b99700 1 -- 172.21.15.27:0/3919294571 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b4000b9c0 con 0x7f8b4415a7a0 2024-01-11T09:46:05.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.762+0000 7f8b48b99700 1 -- 172.21.15.27:0/3919294571 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4000bd60 con 0x7f8b4415a7a0 2024-01-11T09:46:05.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.762+0000 7f8b4bdff700 1 -- 172.21.15.27:0/3919294571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4415a7a0 msgr2=0x7f8b4415c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:05.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.762+0000 7f8b4bdff700 1 --2- 172.21.15.27:0/3919294571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4415a7a0 0x7f8b4415c940 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f8b440ce530 tx=0x7f8b40009250 comp rx=0 tx=0).stop 2024-01-11T09:46:05.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.763+0000 7f8b4bdff700 1 -- 172.21.15.27:0/3919294571 shutdown_connections 2024-01-11T09:46:05.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.763+0000 7f8b4bdff700 1 --2- 172.21.15.27:0/3919294571 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b4415b400 0x7f8b440685e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:05.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.763+0000 7f8b4bdff700 1 --2- 172.21.15.27:0/3919294571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4415a7a0 0x7f8b4415c940 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:05.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.763+0000 7f8b4bdff700 1 --2- 172.21.15.27:0/3919294571 >> v2:172.21.15.77:3300/0 conn(0x7f8b44159e60 0x7f8b4415aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:05.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.763+0000 7f8b4bdff700 1 -- 172.21.15.27:0/3919294571 >> 172.21.15.27:0/3919294571 conn(0x7f8b4406ed40 msgr2=0x7f8b4406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:05.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.763+0000 7f8b4bdff700 1 -- 172.21.15.27:0/3919294571 shutdown_connections 2024-01-11T09:46:05.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.763+0000 7f8b4bdff700 1 -- 172.21.15.27:0/3919294571 wait complete. 2024-01-11T09:46:05.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.764+0000 7f8b4bdff700 1 Processor -- start 2024-01-11T09:46:05.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.764+0000 7f8b4bdff700 1 -- start start 2024-01-11T09:46:05.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.765+0000 7f8b4bdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b4415a7a0 0x7f8b441510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:05.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.765+0000 7f8b4bdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b4415b400 0x7f8b440b1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:05.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.765+0000 7f8b4bdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4414d330 0x7f8b441538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:05.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.765+0000 7f8b4bdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b44151ad0 con 0x7f8b4414d330 2024-01-11T09:46:05.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.765+0000 7f8b4bdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b440b2960 con 0x7f8b4415b400 2024-01-11T09:46:05.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.765+0000 7f8b4bdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b440b1e90 con 0x7f8b4415a7a0 2024-01-11T09:46:05.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.765+0000 7f8b4a39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4414d330 0x7f8b441538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:05.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.765+0000 7f8b4a39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4414d330 0x7f8b441538c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42722/0 (socket says 172.21.15.27:42722) 2024-01-11T09:46:05.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.765+0000 7f8b4a39c700 1 -- 172.21.15.27:0/1736159517 learned_addr learned my addr 172.21.15.27:0/1736159517 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:05.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.766+0000 7f8b4939a700 1 --2- 172.21.15.27:0/1736159517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b4415b400 0x7f8b440b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:05.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.766+0000 7f8b49b9b700 1 --2- 172.21.15.27:0/1736159517 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b4415a7a0 0x7f8b441510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:05.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.766+0000 7f8b4a39c700 1 -- 172.21.15.27:0/1736159517 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b4415a7a0 msgr2=0x7f8b441510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:05.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.766+0000 7f8b4a39c700 1 --2- 172.21.15.27:0/1736159517 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b4415a7a0 0x7f8b441510d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:05.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.766+0000 7f8b4a39c700 1 -- 172.21.15.27:0/1736159517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b4415b400 msgr2=0x7f8b440b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:05.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.766+0000 7f8b4a39c700 1 --2- 172.21.15.27:0/1736159517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b4415b400 0x7f8b440b1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:05.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.766+0000 7f8b4a39c700 1 -- 172.21.15.27:0/1736159517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b400094f0 con 0x7f8b4414d330 2024-01-11T09:46:05.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.766+0000 7f8b4a39c700 1 --2- 172.21.15.27:0/1736159517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4414d330 0x7f8b441538c0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c007f50 tx=0x7f8b3c0098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:05.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.766+0000 7f8b39ffb700 1 -- 172.21.15.27:0/1736159517 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c00eb70 con 0x7f8b4414d330 2024-01-11T09:46:05.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.766+0000 7f8b39ffb700 1 -- 172.21.15.27:0/1736159517 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3c00ecf0 con 0x7f8b4414d330 2024-01-11T09:46:05.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.766+0000 7f8b39ffb700 1 -- 172.21.15.27:0/1736159517 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c014670 con 0x7f8b4414d330 2024-01-11T09:46:05.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.766+0000 7f8b4bdff700 1 -- 172.21.15.27:0/1736159517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b440b20c0 con 0x7f8b4414d330 2024-01-11T09:46:05.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.767+0000 7f8b4bdff700 1 -- 172.21.15.27:0/1736159517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b440b2680 con 0x7f8b4414d330 2024-01-11T09:46:05.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.768+0000 7f8b4bdff700 1 -- 172.21.15.27:0/1736159517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b44151c40 con 0x7f8b4414d330 2024-01-11T09:46:05.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.769+0000 7f8b39ffb700 1 -- 172.21.15.27:0/1736159517 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 135) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f8b3c014a60 con 0x7f8b4414d330 2024-01-11T09:46:05.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.769+0000 7f8b39ffb700 1 --2- 172.21.15.27:0/1736159517 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8b28066d80 0x7f8b28069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:05.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.769+0000 7f8b39ffb700 1 -- 172.21.15.27:0/1736159517 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7412+0+0 (secure 0 0 0) 0x7f8b3c084f40 con 0x7f8b4414d330 2024-01-11T09:46:05.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.770+0000 7f8b49b9b700 1 --2- 172.21.15.27:0/1736159517 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8b28066d80 0x7f8b28069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:05.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.771+0000 7f8b49b9b700 1 --2- 172.21.15.27:0/1736159517 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8b28066d80 0x7f8b28069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8b440ce530 tx=0x7f8b40017000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:05.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.774+0000 7f8b39ffb700 1 -- 172.21.15.27:0/1736159517 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8b44151c40 con 0x7f8b4414d330 2024-01-11T09:46:05.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:05.981+0000 7f8b4bdff700 1 -- 172.21.15.27:0/1736159517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8b44151c40 con 0x7f8b4414d330 2024-01-11T09:46:06.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:06.971+0000 7f8b39ffb700 1 -- 172.21.15.27:0/1736159517 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 136) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f8b3c012dc0 con 0x7f8b4414d330 2024-01-11T09:46:06.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:06.971+0000 7f8b39ffb700 1 -- 172.21.15.27:0/1736159517 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v222) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8b3c04e410 con 0x7f8b4414d330 2024-01-11T09:46:06.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:06.975+0000 7f8b4bdff700 1 -- 172.21.15.27:0/1736159517 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8b28066d80 msgr2=0x7f8b28069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:06.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:06.975+0000 7f8b4bdff700 1 --2- 172.21.15.27:0/1736159517 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8b28066d80 0x7f8b28069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8b440ce530 tx=0x7f8b40017000 comp rx=0 tx=0).stop 2024-01-11T09:46:06.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:06.975+0000 7f8b4bdff700 1 -- 172.21.15.27:0/1736159517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4414d330 msgr2=0x7f8b441538c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:06.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:06.975+0000 7f8b4bdff700 1 --2- 172.21.15.27:0/1736159517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4414d330 0x7f8b441538c0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c007f50 tx=0x7f8b3c0098a0 comp rx=0 tx=0).stop 2024-01-11T09:46:06.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:06.975+0000 7f8b4bdff700 1 -- 172.21.15.27:0/1736159517 shutdown_connections 2024-01-11T09:46:06.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:06.975+0000 7f8b4bdff700 1 --2- 172.21.15.27:0/1736159517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4414d330 0x7f8b441538c0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:06.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:06.975+0000 7f8b4bdff700 1 --2- 172.21.15.27:0/1736159517 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b4415a7a0 0x7f8b441510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:06.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:06.975+0000 7f8b4bdff700 1 --2- 172.21.15.27:0/1736159517 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8b28066d80 0x7f8b28069230 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:06.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:06.975+0000 7f8b4bdff700 1 --2- 172.21.15.27:0/1736159517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b4415b400 0x7f8b440b1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:06.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:06.975+0000 7f8b4bdff700 1 -- 172.21.15.27:0/1736159517 >> 172.21.15.27:0/1736159517 conn(0x7f8b4406ed40 msgr2=0x7f8b4414dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:06.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:06.976+0000 7f8b4bdff700 1 -- 172.21.15.27:0/1736159517 shutdown_connections 2024-01-11T09:46:06.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:06.976+0000 7f8b4bdff700 1 -- 172.21.15.27:0/1736159517 wait complete. 2024-01-11T09:46:06.969 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-01-11T09:46:06.989 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-11T09:46:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.206+0000 7feadadff700 1 Processor -- start 2024-01-11T09:46:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.206+0000 7feadadff700 1 -- start start 2024-01-11T09:46:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.207+0000 7feadadff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fead4156440 0x7fead40c7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.207+0000 7feadadff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fead406ee30 0x7fead40accb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.207+0000 7feadadff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead40c8590 con 0x7fead4156810 2024-01-11T09:46:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.207+0000 7feadadff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fead40c7980 con 0x7fead406ee30 2024-01-11T09:46:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.208+0000 7feadadff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead40c7c60 con 0x7fead4156440 2024-01-11T09:46:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.208+0000 7fead939c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fead406ee30 0x7fead40accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.208+0000 7fead3fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fead4156810 0x7fead406e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37194/0 (socket says 172.21.15.27:37194) 2024-01-11T09:46:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.208+0000 7fead3fff700 1 -- 172.21.15.27:0/1421683630 learned_addr learned my addr 172.21.15.27:0/1421683630 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.208+0000 7fead8b9b700 1 --2- 172.21.15.27:0/1421683630 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fead4156440 0x7fead40c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.208+0000 7fead37fe700 1 -- 172.21.15.27:0/1421683630 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24362696 0 0) 0x7fead40c8590 con 0x7fead4156810 2024-01-11T09:46:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.208+0000 7fead37fe700 1 -- 172.21.15.27:0/1421683630 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feac4003570 con 0x7fead4156810 2024-01-11T09:46:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.208+0000 7fead939c700 1 -- 172.21.15.27:0/1421683630 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fead4156440 msgr2=0x7fead40c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.208+0000 7fead939c700 1 --2- 172.21.15.27:0/1421683630 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fead4156440 0x7fead40c7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.208+0000 7fead939c700 1 -- 172.21.15.27:0/1421683630 >> v1:172.21.15.27:6789/0 conn(0x7fead4156810 legacy=0x7fead406e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.208+0000 7fead939c700 1 -- 172.21.15.27:0/1421683630 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead40c7f40 con 0x7fead406ee30 2024-01-11T09:46:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.209+0000 7fead939c700 1 --2- 172.21.15.27:0/1421683630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fead406ee30 0x7fead40accb0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7feacc009e60 tx=0x7feacc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aa7c0a9ac162324 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.209+0000 7fead37fe700 1 -- 172.21.15.27:0/1421683630 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc00e070 con 0x7fead406ee30 2024-01-11T09:46:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.209+0000 7fead37fe700 1 -- 172.21.15.27:0/1421683630 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feacc00bab0 con 0x7fead406ee30 2024-01-11T09:46:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.209+0000 7fead37fe700 1 -- 172.21.15.27:0/1421683630 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc00be70 con 0x7fead406ee30 2024-01-11T09:46:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.209+0000 7feadadff700 1 -- 172.21.15.27:0/1421683630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fead406ee30 msgr2=0x7fead40accb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.209+0000 7feadadff700 1 --2- 172.21.15.27:0/1421683630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fead406ee30 0x7fead40accb0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7feacc009e60 tx=0x7feacc009780 comp rx=0 tx=0).stop 2024-01-11T09:46:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.210+0000 7feadadff700 1 -- 172.21.15.27:0/1421683630 shutdown_connections 2024-01-11T09:46:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.210+0000 7feadadff700 1 --2- 172.21.15.27:0/1421683630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fead406ee30 0x7fead40accb0 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.210+0000 7feadadff700 1 --2- 172.21.15.27:0/1421683630 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fead4156440 0x7fead40c7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.210+0000 7feadadff700 1 -- 172.21.15.27:0/1421683630 >> 172.21.15.27:0/1421683630 conn(0x7fead40680e0 msgr2=0x7fead40cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.210+0000 7feadadff700 1 -- 172.21.15.27:0/1421683630 shutdown_connections 2024-01-11T09:46:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.210+0000 7feadadff700 1 -- 172.21.15.27:0/1421683630 wait complete. 2024-01-11T09:46:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.211+0000 7feadadff700 1 Processor -- start 2024-01-11T09:46:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.211+0000 7feadadff700 1 -- start start 2024-01-11T09:46:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.212+0000 7feadadff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fead406ee30 0x7fead40bf990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.212+0000 7feadadff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fead4156440 0x7fead4151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.212+0000 7feadadff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fead4156810 0x7fead415c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.212+0000 7feadadff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead40bfed0 con 0x7fead406ee30 2024-01-11T09:46:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.212+0000 7feadadff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead40b06a0 con 0x7fead4156440 2024-01-11T09:46:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.212+0000 7feadadff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead40afa90 con 0x7fead4156810 2024-01-11T09:46:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.212+0000 7fead8b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fead406ee30 0x7fead40bf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.213+0000 7fead8b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fead406ee30 0x7fead40bf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42742/0 (socket says 172.21.15.27:42742) 2024-01-11T09:46:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.213+0000 7fead8b9b700 1 -- 172.21.15.27:0/4281858773 learned_addr learned my addr 172.21.15.27:0/4281858773 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.213+0000 7fead3fff700 1 --2- 172.21.15.27:0/4281858773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fead4156440 0x7fead4151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.213+0000 7fead939c700 1 --2- 172.21.15.27:0/4281858773 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fead4156810 0x7fead415c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.213+0000 7fead8b9b700 1 -- 172.21.15.27:0/4281858773 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fead4156810 msgr2=0x7fead415c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.213+0000 7fead8b9b700 1 --2- 172.21.15.27:0/4281858773 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fead4156810 0x7fead415c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.213+0000 7fead8b9b700 1 -- 172.21.15.27:0/4281858773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fead4156440 msgr2=0x7fead4151690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.213+0000 7fead8b9b700 1 --2- 172.21.15.27:0/4281858773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fead4156440 0x7fead4151690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.213+0000 7fead8b9b700 1 -- 172.21.15.27:0/4281858773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc009a20 con 0x7fead406ee30 2024-01-11T09:46:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.213+0000 7fead939c700 1 --2- 172.21.15.27:0/4281858773 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fead4156810 0x7fead415c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.213+0000 7fead3fff700 1 --2- 172.21.15.27:0/4281858773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fead4156440 0x7fead4151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.213+0000 7fead8b9b700 1 --2- 172.21.15.27:0/4281858773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fead406ee30 0x7fead40bf990 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7feac800ecb0 tx=0x7feac8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.213+0000 7fead0ff9700 1 -- 172.21.15.27:0/4281858773 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac800c9d0 con 0x7fead406ee30 2024-01-11T09:46:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.213+0000 7fead0ff9700 1 -- 172.21.15.27:0/4281858773 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac800cb30 con 0x7fead406ee30 2024-01-11T09:46:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.214+0000 7feadadff700 1 -- 172.21.15.27:0/4281858773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead40afd70 con 0x7fead406ee30 2024-01-11T09:46:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.214+0000 7fead0ff9700 1 -- 172.21.15.27:0/4281858773 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac800ceb0 con 0x7fead406ee30 2024-01-11T09:46:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.214+0000 7feadadff700 1 -- 172.21.15.27:0/4281858773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead40b0330 con 0x7fead406ee30 2024-01-11T09:46:07.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.215+0000 7feadadff700 1 -- 172.21.15.27:0/4281858773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fead40b0980 con 0x7fead406ee30 2024-01-11T09:46:07.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.216+0000 7fead0ff9700 1 -- 172.21.15.27:0/4281858773 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 136) v1 ==== 84890+0+0 (secure 0 0 0) 0x7feac8013680 con 0x7fead406ee30 2024-01-11T09:46:07.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.217+0000 7fead0ff9700 1 --2- 172.21.15.27:0/4281858773 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7feab80670e0 0x7feab8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:07.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.217+0000 7fead0ff9700 1 -- 172.21.15.27:0/4281858773 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6974+0+0 (secure 0 0 0) 0x7feac8085360 con 0x7fead406ee30 2024-01-11T09:46:07.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.217+0000 7fead3fff700 1 --2- 172.21.15.27:0/4281858773 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7feab80670e0 0x7feab8069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:07.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.218+0000 7fead3fff700 1 --2- 172.21.15.27:0/4281858773 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7feab80670e0 0x7feab8069590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fead40c6200 tx=0x7feac0009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:07.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.221+0000 7fead0ff9700 1 -- 172.21.15.27:0/4281858773 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feac8051e60 con 0x7fead406ee30 2024-01-11T09:46:07.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:07.429+0000 7feadadff700 1 -- 172.21.15.27:0/4281858773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fead404c3c0 con 0x7fead406ee30 2024-01-11T09:46:08.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.145+0000 7fead0ff9700 1 -- 172.21.15.27:0/4281858773 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v223) v1 ==== 169+0+0 (secure 0 0 0) 0x7feac8055480 con 0x7fead406ee30 2024-01-11T09:46:08.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.149+0000 7feadadff700 1 -- 172.21.15.27:0/4281858773 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7feab80670e0 msgr2=0x7feab8069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:08.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.149+0000 7feadadff700 1 --2- 172.21.15.27:0/4281858773 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7feab80670e0 0x7feab8069590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fead40c6200 tx=0x7feac0009040 comp rx=0 tx=0).stop 2024-01-11T09:46:08.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.149+0000 7feadadff700 1 -- 172.21.15.27:0/4281858773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fead406ee30 msgr2=0x7fead40bf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:08.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.149+0000 7feadadff700 1 --2- 172.21.15.27:0/4281858773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fead406ee30 0x7fead40bf990 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7feac800ecb0 tx=0x7feac8009120 comp rx=0 tx=0).stop 2024-01-11T09:46:08.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.150+0000 7feadadff700 1 -- 172.21.15.27:0/4281858773 shutdown_connections 2024-01-11T09:46:08.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.150+0000 7feadadff700 1 --2- 172.21.15.27:0/4281858773 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fead4156810 0x7fead415c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.150+0000 7feadadff700 1 --2- 172.21.15.27:0/4281858773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fead406ee30 0x7fead40bf990 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.150+0000 7feadadff700 1 --2- 172.21.15.27:0/4281858773 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7feab80670e0 0x7feab8069590 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.150+0000 7feadadff700 1 --2- 172.21.15.27:0/4281858773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fead4156440 0x7fead4151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.150+0000 7feadadff700 1 -- 172.21.15.27:0/4281858773 >> 172.21.15.27:0/4281858773 conn(0x7fead40680e0 msgr2=0x7fead40ad440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:08.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.150+0000 7feadadff700 1 -- 172.21.15.27:0/4281858773 shutdown_connections 2024-01-11T09:46:08.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.150+0000 7feadadff700 1 -- 172.21.15.27:0/4281858773 wait complete. 2024-01-11T09:46:08.144 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-01-11T09:46:08.164 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:46:08.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.376+0000 7f8694bff700 1 Processor -- start 2024-01-11T09:46:08.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.376+0000 7f8694bff700 1 -- start start 2024-01-11T09:46:08.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.377+0000 7f8694bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f86900b19d0 0x7f8690158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:08.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.377+0000 7f8694bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f869006b610 0x7f8690159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:08.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.377+0000 7f8694bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8690158b60 con 0x7f86901575e0 2024-01-11T09:46:08.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.377+0000 7f8694bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86900cb5e0 con 0x7f86900b19d0 2024-01-11T09:46:08.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.377+0000 7f8694bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86901599b0 con 0x7f869006b610 2024-01-11T09:46:08.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.378+0000 7f868e59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f86901575e0 0x7f86900b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37200/0 (socket says 172.21.15.27:37200) 2024-01-11T09:46:08.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.378+0000 7f868e59c700 1 -- 172.21.15.27:0/2601790330 learned_addr learned my addr 172.21.15.27:0/2601790330 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:08.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.378+0000 7f868dd9b700 1 --2- 172.21.15.27:0/2601790330 >> v2:172.21.15.77:3300/0 conn(0x7f86900b19d0 0x7f8690158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:08.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.378+0000 7f868ed9d700 1 --2- 172.21.15.27:0/2601790330 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f869006b610 0x7f8690159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:08.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.378+0000 7f868d59a700 1 -- 172.21.15.27:0/2601790330 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193757573 0 0) 0x7f8680001110 con 0x7f86901575e0 2024-01-11T09:46:08.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.378+0000 7f868d59a700 1 -- 172.21.15.27:0/2601790330 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8678003570 con 0x7f86901575e0 2024-01-11T09:46:08.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.378+0000 7f868d59a700 1 -- 172.21.15.27:0/2601790330 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 416706832 0 0) 0x7f8678003570 con 0x7f86901575e0 2024-01-11T09:46:08.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.378+0000 7f868d59a700 1 -- 172.21.15.27:0/2601790330 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f869006b610 msgr2=0x7f8690159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:08.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.378+0000 7f868d59a700 1 --2- 172.21.15.27:0/2601790330 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f869006b610 0x7f8690159e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.378+0000 7f868d59a700 1 -- 172.21.15.27:0/2601790330 >> v2:172.21.15.77:3300/0 conn(0x7f86900b19d0 msgr2=0x7f8690158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:08.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.378+0000 7f868d59a700 1 --2- 172.21.15.27:0/2601790330 >> v2:172.21.15.77:3300/0 conn(0x7f86900b19d0 0x7f8690158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.378+0000 7f868d59a700 1 -- 172.21.15.27:0/2601790330 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86900c0f90 con 0x7f86901575e0 2024-01-11T09:46:08.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.378+0000 7f868d59a700 1 -- 172.21.15.27:0/2601790330 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f8680001600 con 0x7f86901575e0 2024-01-11T09:46:08.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.379+0000 7f868d59a700 1 -- 172.21.15.27:0/2601790330 >> v1:172.21.15.27:6789/0 conn(0x7f86901575e0 legacy=0x7f86900b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:08.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.379+0000 7f868d59a700 1 --2- 172.21.15.27:0/2601790330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8678003a60 0x7f8678003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:08.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.379+0000 7f868d59a700 1 --2- 172.21.15.27:0/2601790330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8678004810 0x7f8678004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:08.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.379+0000 7f868d59a700 1 --2- 172.21.15.27:0/2601790330 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f869006b610 0x7f86780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:08.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.379+0000 7f868d59a700 1 -- 172.21.15.27:0/2601790330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86900cb5e0 con 0x7f8678004810 2024-01-11T09:46:08.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.379+0000 7f868d59a700 1 -- 172.21.15.27:0/2601790330 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86901599b0 con 0x7f8678003a60 2024-01-11T09:46:08.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.379+0000 7f868d59a700 1 -- 172.21.15.27:0/2601790330 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f867800ba60 con 0x7f869006b610 2024-01-11T09:46:08.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.379+0000 7f868dd9b700 1 --2- 172.21.15.27:0/2601790330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8678004810 0x7f8678004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:08.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.379+0000 7f868e59c700 1 --2- 172.21.15.27:0/2601790330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8678003a60 0x7f8678003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:08.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.379+0000 7f868ed9d700 1 --2- 172.21.15.27:0/2601790330 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f869006b610 0x7f86780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:08.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.379+0000 7f868dd9b700 1 -- 172.21.15.27:0/2601790330 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f869006b610 msgr2=0x7f86780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:08.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.379+0000 7f868dd9b700 1 --2- 172.21.15.27:0/2601790330 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f869006b610 0x7f86780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.379+0000 7f868dd9b700 1 -- 172.21.15.27:0/2601790330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8678003a60 msgr2=0x7f8678003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:08.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.379+0000 7f868dd9b700 1 --2- 172.21.15.27:0/2601790330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8678003a60 0x7f8678003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.379+0000 7f868dd9b700 1 -- 172.21.15.27:0/2601790330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8678003570 con 0x7f8678004810 2024-01-11T09:46:08.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.380+0000 7f868dd9b700 1 --2- 172.21.15.27:0/2601790330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8678004810 0x7f8678004c80 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f8684009c30 tx=0x7f86840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62c3408276751ac6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:08.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.380+0000 7f868d59a700 1 -- 172.21.15.27:0/2601790330 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868400e710 con 0x7f8678004810 2024-01-11T09:46:08.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.380+0000 7f868d59a700 1 -- 172.21.15.27:0/2601790330 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868400e870 con 0x7f8678004810 2024-01-11T09:46:08.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.380+0000 7f868d59a700 1 -- 172.21.15.27:0/2601790330 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868400ec30 con 0x7f8678004810 2024-01-11T09:46:08.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.380+0000 7f8694bff700 1 -- 172.21.15.27:0/2601790330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8678004810 msgr2=0x7f8678004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:08.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.380+0000 7f8694bff700 1 --2- 172.21.15.27:0/2601790330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8678004810 0x7f8678004c80 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f8684009c30 tx=0x7f86840091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.380+0000 7f868ed9d700 1 -- 172.21.15.27:0/2601790330 reap_dead start 2024-01-11T09:46:08.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.381+0000 7f8694bff700 1 -- 172.21.15.27:0/2601790330 shutdown_connections 2024-01-11T09:46:08.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.381+0000 7f8694bff700 1 -- 172.21.15.27:0/2601790330 >> 172.21.15.27:0/2601790330 conn(0x7f86900680e0 msgr2=0x7f8690069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:08.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.381+0000 7f8694bff700 1 -- 172.21.15.27:0/2601790330 shutdown_connections 2024-01-11T09:46:08.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.381+0000 7f8694bff700 1 -- 172.21.15.27:0/2601790330 wait complete. 2024-01-11T09:46:08.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.383+0000 7f8694bff700 1 Processor -- start 2024-01-11T09:46:08.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.383+0000 7f8694bff700 1 -- start start 2024-01-11T09:46:08.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.384+0000 7f8694bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8678004810 0x7f86901533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:08.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.384+0000 7f8694bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86900a7180 0x7f86900a75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:08.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.384+0000 7f8694bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86900a5490 0x7f86900a5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:08.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.384+0000 7f8694bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86900a7b30 con 0x7f86900a5490 2024-01-11T09:46:08.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.384+0000 7f8694bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86900a6180 con 0x7f86900a7180 2024-01-11T09:46:08.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.384+0000 7f8694bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86900a6460 con 0x7f8678004810 2024-01-11T09:46:08.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.385+0000 7f868ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86900a5490 0x7f86900a5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:08.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.385+0000 7f868ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86900a5490 0x7f86900a5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42768/0 (socket says 172.21.15.27:42768) 2024-01-11T09:46:08.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.385+0000 7f868ed9d700 1 -- 172.21.15.27:0/2731412984 learned_addr learned my addr 172.21.15.27:0/2731412984 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:08.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.385+0000 7f868e59c700 1 --2- 172.21.15.27:0/2731412984 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8678004810 0x7f86901533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:08.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.385+0000 7f868dd9b700 1 --2- 172.21.15.27:0/2731412984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86900a7180 0x7f86900a75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:08.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.385+0000 7f868ed9d700 1 -- 172.21.15.27:0/2731412984 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8678004810 msgr2=0x7f86901533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:08.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.385+0000 7f868ed9d700 1 --2- 172.21.15.27:0/2731412984 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8678004810 0x7f86901533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.385+0000 7f868ed9d700 1 -- 172.21.15.27:0/2731412984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86900a7180 msgr2=0x7f86900a75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:08.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.385+0000 7f868ed9d700 1 --2- 172.21.15.27:0/2731412984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86900a7180 0x7f86900a75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.385+0000 7f868ed9d700 1 -- 172.21.15.27:0/2731412984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86900a6740 con 0x7f86900a5490 2024-01-11T09:46:08.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.385+0000 7f868ed9d700 1 --2- 172.21.15.27:0/2731412984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86900a5490 0x7f86900a5900 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f8690070d30 tx=0x7f868800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:08.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.385+0000 7f867e7fc700 1 -- 172.21.15.27:0/2731412984 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868800e040 con 0x7f86900a5490 2024-01-11T09:46:08.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.386+0000 7f8694bff700 1 -- 172.21.15.27:0/2731412984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86900a6a20 con 0x7f86900a5490 2024-01-11T09:46:08.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.386+0000 7f867e7fc700 1 -- 172.21.15.27:0/2731412984 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868800f040 con 0x7f86900a5490 2024-01-11T09:46:08.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.386+0000 7f867e7fc700 1 -- 172.21.15.27:0/2731412984 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868800fd70 con 0x7f86900a5490 2024-01-11T09:46:08.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.386+0000 7f8694bff700 1 -- 172.21.15.27:0/2731412984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8690153920 con 0x7f86900a5490 2024-01-11T09:46:08.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.387+0000 7f8694bff700 1 -- 172.21.15.27:0/2731412984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86900c5650 con 0x7f86900a5490 2024-01-11T09:46:08.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.388+0000 7f867e7fc700 1 -- 172.21.15.27:0/2731412984 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 136) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f8688015640 con 0x7f86900a5490 2024-01-11T09:46:08.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.390+0000 7f867e7fc700 1 --2- 172.21.15.27:0/2731412984 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f86700670e0 0x7f8670069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:08.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.390+0000 7f867e7fc700 1 -- 172.21.15.27:0/2731412984 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f8688085530 con 0x7f86900a5490 2024-01-11T09:46:08.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.390+0000 7f868e59c700 1 --2- 172.21.15.27:0/2731412984 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f86700670e0 0x7f8670069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:08.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.391+0000 7f868e59c700 1 --2- 172.21.15.27:0/2731412984 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f86700670e0 0x7f8670069590 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8680000e10 tx=0x7f8680007a80 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:08.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.394+0000 7f867e7fc700 1 -- 172.21.15.27:0/2731412984 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f86900c5650 con 0x7f86900a5490 2024-01-11T09:46:08.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.598+0000 7f8694bff700 1 -- 172.21.15.27:0/2731412984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86900c5650 con 0x7f86900a5490 2024-01-11T09:46:08.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.599+0000 7f867e7fc700 1 -- 172.21.15.27:0/2731412984 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v223) v1 ==== 74+0+16559 (secure 0 0 0) 0x7f86900c5650 con 0x7f86900a5490 2024-01-11T09:46:08.590 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:08.591 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":223,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:46:08.139060+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-01-11T09:45:35.631727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-01-11T09:45:36.452195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/1143960902":"2024-01-12T09:46:04.482215+0000","172.21.15.27:6833/1143960902":"2024-01-12T09:46:04.482215+0000","172.21.15.77:0/3729201132":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/3605354921":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/4084841078":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/3487780416":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/534676732":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/3735282980":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/1375879589":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/2192171652":"2024-01-12T09:45:52.192235+0000","172.21.15.77:6837/59776":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/2635904665":"2024-01-12T09:46:04.127727+0000","172.21.15.77:6836/59776":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/2654175097":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/652362578":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/749658672":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/2439942664":"2024-01-12T09:45:52.192235+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:46:08.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.601+0000 7f8694bff700 1 -- 172.21.15.27:0/2731412984 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f86700670e0 msgr2=0x7f8670069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:08.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.601+0000 7f8694bff700 1 --2- 172.21.15.27:0/2731412984 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f86700670e0 0x7f8670069590 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8680000e10 tx=0x7f8680007a80 comp rx=0 tx=0).stop 2024-01-11T09:46:08.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.601+0000 7f8694bff700 1 -- 172.21.15.27:0/2731412984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86900a5490 msgr2=0x7f86900a5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:08.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.601+0000 7f8694bff700 1 --2- 172.21.15.27:0/2731412984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86900a5490 0x7f86900a5900 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f8690070d30 tx=0x7f868800b040 comp rx=0 tx=0).stop 2024-01-11T09:46:08.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.601+0000 7f8694bff700 1 -- 172.21.15.27:0/2731412984 shutdown_connections 2024-01-11T09:46:08.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.601+0000 7f8694bff700 1 --2- 172.21.15.27:0/2731412984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86900a5490 0x7f86900a5900 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.601+0000 7f8694bff700 1 --2- 172.21.15.27:0/2731412984 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8678004810 0x7f86901533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.601+0000 7f8694bff700 1 --2- 172.21.15.27:0/2731412984 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f86700670e0 0x7f8670069590 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.601+0000 7f8694bff700 1 --2- 172.21.15.27:0/2731412984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86900a7180 0x7f86900a75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.601+0000 7f8694bff700 1 -- 172.21.15.27:0/2731412984 >> 172.21.15.27:0/2731412984 conn(0x7f86900680e0 msgr2=0x7f869015a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.601+0000 7f8694bff700 1 -- 172.21.15.27:0/2731412984 shutdown_connections 2024-01-11T09:46:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.601+0000 7f8694bff700 1 -- 172.21.15.27:0/2731412984 wait complete. 2024-01-11T09:46:08.611 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-01-11T09:45:35.631727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '217', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2024-01-11T09:45:36.452195+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '218', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}}] 2024-01-11T09:46:08.612 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T09:46:08.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.823+0000 7f5b5a3ff700 1 Processor -- start 2024-01-11T09:46:08.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.823+0000 7f5b5a3ff700 1 -- start start 2024-01-11T09:46:08.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.824+0000 7f5b5a3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b540cd5f0 0x7f5b541591e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:08.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.824+0000 7f5b5a3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5b54159720 0x7f5b540cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:08.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.824+0000 7f5b5a3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5415aef0 con 0x7f5b540cd220 2024-01-11T09:46:08.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.824+0000 7f5b5a3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b540cee00 con 0x7f5b54159720 2024-01-11T09:46:08.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.824+0000 7f5b5a3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b54159b60 con 0x7f5b540cd5f0 2024-01-11T09:46:08.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.825+0000 7f5b53fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5b540cd220 0x7f5b5415d320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37214/0 (socket says 172.21.15.27:37214) 2024-01-11T09:46:08.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.825+0000 7f5b53fff700 1 -- 172.21.15.27:0/1853609901 learned_addr learned my addr 172.21.15.27:0/1853609901 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:08.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.825+0000 7f5b5899c700 1 --2- 172.21.15.27:0/1853609901 >> v2:172.21.15.27:3300/0 conn(0x7f5b54159720 0x7f5b540cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:08.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.825+0000 7f5b537fe700 1 --2- 172.21.15.27:0/1853609901 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b540cd5f0 0x7f5b541591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:08.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.825+0000 7f5b52ffd700 1 -- 172.21.15.27:0/1853609901 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245226088 0 0) 0x7f5b5415aef0 con 0x7f5b540cd220 2024-01-11T09:46:08.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.825+0000 7f5b52ffd700 1 -- 172.21.15.27:0/1853609901 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b3c003570 con 0x7f5b540cd220 2024-01-11T09:46:08.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.825+0000 7f5b5899c700 1 -- 172.21.15.27:0/1853609901 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b540cd5f0 msgr2=0x7f5b541591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:08.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.825+0000 7f5b5899c700 1 --2- 172.21.15.27:0/1853609901 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b540cd5f0 0x7f5b541591e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.825+0000 7f5b5899c700 1 -- 172.21.15.27:0/1853609901 >> v1:172.21.15.27:6789/0 conn(0x7f5b540cd220 legacy=0x7f5b5415d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:08.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.825+0000 7f5b5899c700 1 -- 172.21.15.27:0/1853609901 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5415dd70 con 0x7f5b54159720 2024-01-11T09:46:08.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.826+0000 7f5b5899c700 1 --2- 172.21.15.27:0/1853609901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b54159720 0x7f5b540cf290 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c009e60 tx=0x7f5b4c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d23f4e642b6d3d0e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:08.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.826+0000 7f5b52ffd700 1 -- 172.21.15.27:0/1853609901 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c00e070 con 0x7f5b54159720 2024-01-11T09:46:08.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.826+0000 7f5b52ffd700 1 -- 172.21.15.27:0/1853609901 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b4c00bab0 con 0x7f5b54159720 2024-01-11T09:46:08.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.826+0000 7f5b52ffd700 1 -- 172.21.15.27:0/1853609901 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c00be70 con 0x7f5b54159720 2024-01-11T09:46:08.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.826+0000 7f5b5a3ff700 1 -- 172.21.15.27:0/1853609901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b54159720 msgr2=0x7f5b540cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:08.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.826+0000 7f5b5a3ff700 1 --2- 172.21.15.27:0/1853609901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b54159720 0x7f5b540cf290 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c009e60 tx=0x7f5b4c009780 comp rx=0 tx=0).stop 2024-01-11T09:46:08.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.826+0000 7f5b5a3ff700 1 -- 172.21.15.27:0/1853609901 shutdown_connections 2024-01-11T09:46:08.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.826+0000 7f5b5a3ff700 1 --2- 172.21.15.27:0/1853609901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b54159720 0x7f5b540cf290 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.826+0000 7f5b5a3ff700 1 --2- 172.21.15.27:0/1853609901 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b540cd5f0 0x7f5b541591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.826+0000 7f5b5a3ff700 1 -- 172.21.15.27:0/1853609901 >> 172.21.15.27:0/1853609901 conn(0x7f5b5406ed40 msgr2=0x7f5b5406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:08.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.826+0000 7f5b5a3ff700 1 -- 172.21.15.27:0/1853609901 shutdown_connections 2024-01-11T09:46:08.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.827+0000 7f5b5a3ff700 1 -- 172.21.15.27:0/1853609901 wait complete. 2024-01-11T09:46:08.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.828+0000 7f5b5a3ff700 1 Processor -- start 2024-01-11T09:46:08.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.828+0000 7f5b5a3ff700 1 -- start start 2024-01-11T09:46:08.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.829+0000 7f5b5a3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b540cd220 0x7f5b5414cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:08.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.829+0000 7f5b5a3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b540cd5f0 0x7f5b540b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:08.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.829+0000 7f5b5a3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b54159720 0x7f5b5414c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:08.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.829+0000 7f5b5a3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5416cbf0 con 0x7f5b54159720 2024-01-11T09:46:08.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.829+0000 7f5b5a3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5414ca20 con 0x7f5b540cd5f0 2024-01-11T09:46:08.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.829+0000 7f5b5a3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5416ce40 con 0x7f5b540cd220 2024-01-11T09:46:08.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.829+0000 7f5b5899c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b54159720 0x7f5b5414c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:08.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.829+0000 7f5b5899c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b54159720 0x7f5b5414c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42776/0 (socket says 172.21.15.27:42776) 2024-01-11T09:46:08.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.829+0000 7f5b5899c700 1 -- 172.21.15.27:0/4059891145 learned_addr learned my addr 172.21.15.27:0/4059891145 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:08.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.829+0000 7f5b537fe700 1 --2- 172.21.15.27:0/4059891145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b540cd5f0 0x7f5b540b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:08.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.830+0000 7f5b53fff700 1 --2- 172.21.15.27:0/4059891145 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b540cd220 0x7f5b5414cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:08.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.830+0000 7f5b5899c700 1 -- 172.21.15.27:0/4059891145 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b540cd220 msgr2=0x7f5b5414cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:08.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.830+0000 7f5b5899c700 1 --2- 172.21.15.27:0/4059891145 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b540cd220 0x7f5b5414cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.830+0000 7f5b5899c700 1 -- 172.21.15.27:0/4059891145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b540cd5f0 msgr2=0x7f5b540b0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:08.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.830+0000 7f5b5899c700 1 --2- 172.21.15.27:0/4059891145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b540cd5f0 0x7f5b540b0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:08.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.830+0000 7f5b5899c700 1 -- 172.21.15.27:0/4059891145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b4c009a20 con 0x7f5b54159720 2024-01-11T09:46:08.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.830+0000 7f5b5899c700 1 --2- 172.21.15.27:0/4059891145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b54159720 0x7f5b5414c4e0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c00b610 tx=0x7f5b4c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:08.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.830+0000 7f5b3bfff700 1 -- 172.21.15.27:0/4059891145 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c012e80 con 0x7f5b54159720 2024-01-11T09:46:08.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.830+0000 7f5b3bfff700 1 -- 172.21.15.27:0/4059891145 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b4c027470 con 0x7f5b54159720 2024-01-11T09:46:08.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.830+0000 7f5b3bfff700 1 -- 172.21.15.27:0/4059891145 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c027830 con 0x7f5b54159720 2024-01-11T09:46:08.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.830+0000 7f5b5a3ff700 1 -- 172.21.15.27:0/4059891145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b5416d120 con 0x7f5b54159720 2024-01-11T09:46:08.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.831+0000 7f5b5a3ff700 1 -- 172.21.15.27:0/4059891145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b5416d6e0 con 0x7f5b54159720 2024-01-11T09:46:08.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.832+0000 7f5b5a3ff700 1 -- 172.21.15.27:0/4059891145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b5416d900 con 0x7f5b54159720 2024-01-11T09:46:08.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.832+0000 7f5b3bfff700 1 -- 172.21.15.27:0/4059891145 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 136) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f5b4c024070 con 0x7f5b54159720 2024-01-11T09:46:08.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.833+0000 7f5b3bfff700 1 --2- 172.21.15.27:0/4059891145 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5b400670e0 0x7f5b40069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:08.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.833+0000 7f5b3bfff700 1 -- 172.21.15.27:0/4059891145 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f5b4c087650 con 0x7f5b54159720 2024-01-11T09:46:08.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.834+0000 7f5b53fff700 1 --2- 172.21.15.27:0/4059891145 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5b400670e0 0x7f5b40069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:08.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.834+0000 7f5b53fff700 1 --2- 172.21.15.27:0/4059891145 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5b400670e0 0x7f5b40069590 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5b440058d0 tx=0x7f5b44009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:08.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.838+0000 7f5b3bfff700 1 -- 172.21.15.27:0/4059891145 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5b4c0542f0 con 0x7f5b54159720 2024-01-11T09:46:08.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:08.971+0000 7f5b3bfff700 1 -- 172.21.15.27:0/4059891145 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f5b4c003040 con 0x7f5b54159720 2024-01-11T09:46:09.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.047+0000 7f5b5a3ff700 1 -- 172.21.15.27:0/4059891145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5b5404c3c0 con 0x7f5b54159720 2024-01-11T09:46:09.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.047+0000 7f5b3bfff700 1 -- 172.21.15.27:0/4059891145 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v223) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5b4c057d80 con 0x7f5b54159720 2024-01-11T09:46:09.039 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T09:46:09.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.049+0000 7f5b5a3ff700 1 -- 172.21.15.27:0/4059891145 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5b400670e0 msgr2=0x7f5b40069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:09.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.049+0000 7f5b5a3ff700 1 --2- 172.21.15.27:0/4059891145 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5b400670e0 0x7f5b40069590 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5b440058d0 tx=0x7f5b44009040 comp rx=0 tx=0).stop 2024-01-11T09:46:09.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.049+0000 7f5b5a3ff700 1 -- 172.21.15.27:0/4059891145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b54159720 msgr2=0x7f5b5414c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:09.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.049+0000 7f5b5a3ff700 1 --2- 172.21.15.27:0/4059891145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b54159720 0x7f5b5414c4e0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c00b610 tx=0x7f5b4c019000 comp rx=0 tx=0).stop 2024-01-11T09:46:09.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.049+0000 7f5b5a3ff700 1 -- 172.21.15.27:0/4059891145 shutdown_connections 2024-01-11T09:46:09.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.049+0000 7f5b5a3ff700 1 --2- 172.21.15.27:0/4059891145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b54159720 0x7f5b5414c4e0 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.049+0000 7f5b5a3ff700 1 --2- 172.21.15.27:0/4059891145 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5b540cd220 0x7f5b5414cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.049+0000 7f5b5a3ff700 1 --2- 172.21.15.27:0/4059891145 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5b400670e0 0x7f5b40069590 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.049+0000 7f5b5a3ff700 1 --2- 172.21.15.27:0/4059891145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b540cd5f0 0x7f5b540b0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.049+0000 7f5b5a3ff700 1 -- 172.21.15.27:0/4059891145 >> 172.21.15.27:0/4059891145 conn(0x7f5b5406ed40 msgr2=0x7f5b5415bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:09.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.050+0000 7f5b5a3ff700 1 -- 172.21.15.27:0/4059891145 shutdown_connections 2024-01-11T09:46:09.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.050+0000 7f5b5a3ff700 1 -- 172.21.15.27:0/4059891145 wait complete. 2024-01-11T09:46:09.059 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-01-11T09:46:09.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.276+0000 7fe3a39ff700 1 Processor -- start 2024-01-11T09:46:09.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.277+0000 7fe3a39ff700 1 -- start start 2024-01-11T09:46:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.278+0000 7fe3a39ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe39c157410 0x7fe39c0ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.278+0000 7fe3a39ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe39c1577e0 0x7fe39c0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.278+0000 7fe3a39ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39c0699d0 con 0x7fe39c069540 2024-01-11T09:46:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.278+0000 7fe3a39ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe39c069cb0 con 0x7fe39c1577e0 2024-01-11T09:46:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.278+0000 7fe3a39ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe39c069f90 con 0x7fe39c157410 2024-01-11T09:46:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.278+0000 7fe3a0f9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe39c1577e0 0x7fe39c0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:09.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.278+0000 7fe3a0f9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe39c1577e0 0x7fe39c0ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42788/0 (socket says 172.21.15.27:42788) 2024-01-11T09:46:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.278+0000 7fe3a0f9a700 1 -- 172.21.15.27:0/2111796465 learned_addr learned my addr 172.21.15.27:0/2111796465 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.278+0000 7fe3a179b700 1 --2- 172.21.15.27:0/2111796465 >> v2:172.21.15.77:3300/0 conn(0x7fe39c157410 0x7fe39c0ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.279+0000 7fe3a0f9a700 1 -- 172.21.15.27:0/2111796465 >> v2:172.21.15.77:3300/0 conn(0x7fe39c157410 msgr2=0x7fe39c0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.279+0000 7fe3a0f9a700 1 --2- 172.21.15.27:0/2111796465 >> v2:172.21.15.77:3300/0 conn(0x7fe39c157410 0x7fe39c0ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.279+0000 7fe3a0f9a700 1 -- 172.21.15.27:0/2111796465 >> v1:172.21.15.77:6789/0 conn(0x7fe39c069540 legacy=0x7fe39c15b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:09.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.279+0000 7fe3a0f9a700 1 -- 172.21.15.27:0/2111796465 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39c06a270 con 0x7fe39c1577e0 2024-01-11T09:46:09.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.279+0000 7fe3a0f9a700 1 --2- 172.21.15.27:0/2111796465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c1577e0 0x7fe39c0ae000 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fe39c070c90 tx=0x7fe38c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655f5c5719d2e660 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:09.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.279+0000 7fe393fff700 1 -- 172.21.15.27:0/2111796465 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38c009e70 con 0x7fe39c1577e0 2024-01-11T09:46:09.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.279+0000 7fe393fff700 1 -- 172.21.15.27:0/2111796465 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38c00b860 con 0x7fe39c1577e0 2024-01-11T09:46:09.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.279+0000 7fe393fff700 1 -- 172.21.15.27:0/2111796465 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38c00bc00 con 0x7fe39c1577e0 2024-01-11T09:46:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.280+0000 7fe3a39ff700 1 -- 172.21.15.27:0/2111796465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c1577e0 msgr2=0x7fe39c0ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.280+0000 7fe3a39ff700 1 --2- 172.21.15.27:0/2111796465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c1577e0 0x7fe39c0ae000 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fe39c070c90 tx=0x7fe38c009250 comp rx=0 tx=0).stop 2024-01-11T09:46:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.280+0000 7fe3a39ff700 1 -- 172.21.15.27:0/2111796465 shutdown_connections 2024-01-11T09:46:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.280+0000 7fe3a39ff700 1 --2- 172.21.15.27:0/2111796465 >> v2:172.21.15.77:3300/0 conn(0x7fe39c157410 0x7fe39c0ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.280+0000 7fe3a39ff700 1 --2- 172.21.15.27:0/2111796465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c1577e0 0x7fe39c0ae000 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.280+0000 7fe3a39ff700 1 -- 172.21.15.27:0/2111796465 >> 172.21.15.27:0/2111796465 conn(0x7fe39c0680a0 msgr2=0x7fe39c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.280+0000 7fe3a39ff700 1 -- 172.21.15.27:0/2111796465 shutdown_connections 2024-01-11T09:46:09.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.280+0000 7fe3a39ff700 1 -- 172.21.15.27:0/2111796465 wait complete. 2024-01-11T09:46:09.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.282+0000 7fe3a39ff700 1 Processor -- start 2024-01-11T09:46:09.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.282+0000 7fe3a39ff700 1 -- start start 2024-01-11T09:46:09.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.283+0000 7fe3a39ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe39c069540 0x7fe39c16dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:09.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.283+0000 7fe3a39ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe39c157410 0x7fe39c14ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:09.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.283+0000 7fe3a39ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c1577e0 0x7fe39c151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:09.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.283+0000 7fe3a39ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c1509f0 con 0x7fe39c1577e0 2024-01-11T09:46:09.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.283+0000 7fe3a39ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c16e270 con 0x7fe39c069540 2024-01-11T09:46:09.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.283+0000 7fe3a39ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c14efa0 con 0x7fe39c157410 2024-01-11T09:46:09.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.283+0000 7fe3a179b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe39c069540 0x7fe39c16dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:09.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.283+0000 7fe3a1f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c1577e0 0x7fe39c151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:09.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.283+0000 7fe3a179b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe39c069540 0x7fe39c16dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:48468/0 (socket says 172.21.15.27:48468) 2024-01-11T09:46:09.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.283+0000 7fe3a179b700 1 -- 172.21.15.27:0/2270792074 learned_addr learned my addr 172.21.15.27:0/2270792074 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:09.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.283+0000 7fe3a0f9a700 1 --2- 172.21.15.27:0/2270792074 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe39c157410 0x7fe39c14ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.284+0000 7fe3a1f9c700 1 -- 172.21.15.27:0/2270792074 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe39c157410 msgr2=0x7fe39c14ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.284+0000 7fe3a1f9c700 1 --2- 172.21.15.27:0/2270792074 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe39c157410 0x7fe39c14ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.284+0000 7fe3a1f9c700 1 -- 172.21.15.27:0/2270792074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe39c069540 msgr2=0x7fe39c16dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.284+0000 7fe3a1f9c700 1 --2- 172.21.15.27:0/2270792074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe39c069540 0x7fe39c16dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.284+0000 7fe3a1f9c700 1 -- 172.21.15.27:0/2270792074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38c0094f0 con 0x7fe39c1577e0 2024-01-11T09:46:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.284+0000 7fe3a179b700 1 --2- 172.21.15.27:0/2270792074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe39c069540 0x7fe39c16dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.284+0000 7fe3a1f9c700 1 --2- 172.21.15.27:0/2270792074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c1577e0 0x7fe39c151460 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7fe39400c790 tx=0x7fe3940015f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.284+0000 7fe3917fa700 1 -- 172.21.15.27:0/2270792074 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe394012630 con 0x7fe39c1577e0 2024-01-11T09:46:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.284+0000 7fe3917fa700 1 -- 172.21.15.27:0/2270792074 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3940127b0 con 0x7fe39c1577e0 2024-01-11T09:46:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.284+0000 7fe3917fa700 1 -- 172.21.15.27:0/2270792074 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe394012b70 con 0x7fe39c1577e0 2024-01-11T09:46:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.284+0000 7fe3a39ff700 1 -- 172.21.15.27:0/2270792074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe39c150c60 con 0x7fe39c1577e0 2024-01-11T09:46:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.284+0000 7fe3a39ff700 1 -- 172.21.15.27:0/2270792074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39c14ec50 con 0x7fe39c1577e0 2024-01-11T09:46:09.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.285+0000 7fe3a39ff700 1 -- 172.21.15.27:0/2270792074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe39c150850 con 0x7fe39c1577e0 2024-01-11T09:46:09.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.286+0000 7fe3917fa700 1 -- 172.21.15.27:0/2270792074 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fe394003030 con 0x7fe39c1577e0 2024-01-11T09:46:09.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.287+0000 7fe3917fa700 1 --2- 172.21.15.27:0/2270792074 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe380067170 0x7fe380069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:09.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.287+0000 7fe3917fa700 1 -- 172.21.15.27:0/2270792074 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6560+0+0 (secure 0 0 0) 0x7fe394084a00 con 0x7fe39c1577e0 2024-01-11T09:46:09.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.288+0000 7fe3a179b700 1 --2- 172.21.15.27:0/2270792074 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe380067170 0x7fe380069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:09.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.289+0000 7fe3a179b700 1 --2- 172.21.15.27:0/2270792074 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe380067170 0x7fe380069620 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe398011520 tx=0x7fe398009120 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:09.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.292+0000 7fe3917fa700 1 -- 172.21.15.27:0/2270792074 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe39c150850 con 0x7fe39c1577e0 2024-01-11T09:46:09.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.503+0000 7fe3a39ff700 1 -- 172.21.15.27:0/2270792074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe39c151180 con 0x7fe39c1577e0 2024-01-11T09:46:09.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.504+0000 7fe3917fa700 1 -- 172.21.15.27:0/2270792074 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v223) v1 ==== 107+0+11 (secure 0 0 0) 0x7fe39c151180 con 0x7fe39c1577e0 2024-01-11T09:46:09.495 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T09:46:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.506+0000 7fe3a39ff700 1 -- 172.21.15.27:0/2270792074 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe380067170 msgr2=0x7fe380069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.506+0000 7fe3a39ff700 1 --2- 172.21.15.27:0/2270792074 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe380067170 0x7fe380069620 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe398011520 tx=0x7fe398009120 comp rx=0 tx=0).stop 2024-01-11T09:46:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.506+0000 7fe3a39ff700 1 -- 172.21.15.27:0/2270792074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c1577e0 msgr2=0x7fe39c151460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.506+0000 7fe3a39ff700 1 --2- 172.21.15.27:0/2270792074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c1577e0 0x7fe39c151460 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7fe39400c790 tx=0x7fe3940015f0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.506+0000 7fe3a39ff700 1 -- 172.21.15.27:0/2270792074 shutdown_connections 2024-01-11T09:46:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.506+0000 7fe3a39ff700 1 --2- 172.21.15.27:0/2270792074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe39c1577e0 0x7fe39c151460 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.506+0000 7fe3a39ff700 1 --2- 172.21.15.27:0/2270792074 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe380067170 0x7fe380069620 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.506+0000 7fe3a39ff700 1 --2- 172.21.15.27:0/2270792074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe39c069540 0x7fe39c16dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.506+0000 7fe3a39ff700 1 --2- 172.21.15.27:0/2270792074 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe39c157410 0x7fe39c14ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.506+0000 7fe3a39ff700 1 -- 172.21.15.27:0/2270792074 >> 172.21.15.27:0/2270792074 conn(0x7fe39c0680a0 msgr2=0x7fe39c0c1e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.506+0000 7fe3a39ff700 1 -- 172.21.15.27:0/2270792074 shutdown_connections 2024-01-11T09:46:09.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.506+0000 7fe3a39ff700 1 -- 172.21.15.27:0/2270792074 wait complete. 2024-01-11T09:46:09.515 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-01-11T09:46:09.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.727+0000 7f3d36bff700 1 Processor -- start 2024-01-11T09:46:09.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.727+0000 7f3d36bff700 1 -- start start 2024-01-11T09:46:09.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.728+0000 7f3d36bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3d300ccd70 0x7f3d300cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:09.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.728+0000 7f3d36bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d300cf850 con 0x7f3d3015ae70 2024-01-11T09:46:09.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.728+0000 7f3d36bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d300cec40 con 0x7f3d3015a1b0 2024-01-11T09:46:09.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.728+0000 7f3d36bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d3015ac90 con 0x7f3d300ccd70 2024-01-11T09:46:09.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.729+0000 7f3d3519c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3d3015ae70 0x7f3d3015b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37216/0 (socket says 172.21.15.27:37216) 2024-01-11T09:46:09.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.729+0000 7f3d3519c700 1 -- 172.21.15.27:0/2096498318 learned_addr learned my addr 172.21.15.27:0/2096498318 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:09.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.729+0000 7f3d2ffff700 1 --2- 172.21.15.27:0/2096498318 >> v2:172.21.15.77:3300/0 conn(0x7f3d300ccd70 0x7f3d300cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:09.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.729+0000 7f3d2f7fe700 1 -- 172.21.15.27:0/2096498318 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525043126 0 0) 0x7f3d300cf850 con 0x7f3d3015ae70 2024-01-11T09:46:09.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.729+0000 7f3d2f7fe700 1 -- 172.21.15.27:0/2096498318 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d20003570 con 0x7f3d3015ae70 2024-01-11T09:46:09.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.729+0000 7f3d2ffff700 1 -- 172.21.15.27:0/2096498318 >> v1:172.21.15.77:6789/0 conn(0x7f3d3015a1b0 legacy=0x7f3d3015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:09.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.729+0000 7f3d2f7fe700 1 -- 172.21.15.27:0/2096498318 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877348321 0 0) 0x7f3d300cec40 con 0x7f3d3015a1b0 2024-01-11T09:46:09.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.729+0000 7f3d2ffff700 1 -- 172.21.15.27:0/2096498318 >> v1:172.21.15.27:6789/0 conn(0x7f3d3015ae70 legacy=0x7f3d3015b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:09.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.729+0000 7f3d2ffff700 1 -- 172.21.15.27:0/2096498318 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d300cef20 con 0x7f3d300ccd70 2024-01-11T09:46:09.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.730+0000 7f3d2ffff700 1 --2- 172.21.15.27:0/2096498318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d300ccd70 0x7f3d300cd210 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0098e0 tx=0x7f3d1c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=597d1f4c65309401 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:09.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.730+0000 7f3d2f7fe700 1 -- 172.21.15.27:0/2096498318 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c00b640 con 0x7f3d300ccd70 2024-01-11T09:46:09.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.730+0000 7f3d2f7fe700 1 -- 172.21.15.27:0/2096498318 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c00b7a0 con 0x7f3d300ccd70 2024-01-11T09:46:09.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.730+0000 7f3d2f7fe700 1 -- 172.21.15.27:0/2096498318 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c00bb40 con 0x7f3d300ccd70 2024-01-11T09:46:09.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.730+0000 7f3d36bff700 1 -- 172.21.15.27:0/2096498318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d300ccd70 msgr2=0x7f3d300cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:09.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.730+0000 7f3d36bff700 1 --2- 172.21.15.27:0/2096498318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d300ccd70 0x7f3d300cd210 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0098e0 tx=0x7f3d1c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.731+0000 7f3d36bff700 1 -- 172.21.15.27:0/2096498318 shutdown_connections 2024-01-11T09:46:09.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.731+0000 7f3d36bff700 1 --2- 172.21.15.27:0/2096498318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d300ccd70 0x7f3d300cd210 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.731+0000 7f3d36bff700 1 -- 172.21.15.27:0/2096498318 >> 172.21.15.27:0/2096498318 conn(0x7f3d3006ed40 msgr2=0x7f3d3006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:09.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.731+0000 7f3d36bff700 1 -- 172.21.15.27:0/2096498318 shutdown_connections 2024-01-11T09:46:09.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.731+0000 7f3d36bff700 1 -- 172.21.15.27:0/2096498318 wait complete. 2024-01-11T09:46:09.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.732+0000 7f3d36bff700 1 Processor -- start 2024-01-11T09:46:09.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.732+0000 7f3d36bff700 1 -- start start 2024-01-11T09:46:09.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.733+0000 7f3d36bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d300ccd70 0x7f3d30158930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:09.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.733+0000 7f3d36bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d3015a1b0 0x7f3d30169740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:09.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.733+0000 7f3d36bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3d3015ae70 0x7f3d300c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:09.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.733+0000 7f3d36bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d30169c80 con 0x7f3d3015a1b0 2024-01-11T09:46:09.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.733+0000 7f3d36bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d300b5040 con 0x7f3d300ccd70 2024-01-11T09:46:09.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.733+0000 7f3d36bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d300b4430 con 0x7f3d3015ae70 2024-01-11T09:46:09.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.733+0000 7f3d2ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d3015a1b0 0x7f3d30169740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:09.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.733+0000 7f3d2ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d3015a1b0 0x7f3d30169740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42804/0 (socket says 172.21.15.27:42804) 2024-01-11T09:46:09.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.733+0000 7f3d2ffff700 1 -- 172.21.15.27:0/3216247626 learned_addr learned my addr 172.21.15.27:0/3216247626 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:09.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.733+0000 7f3d3499b700 1 --2- 172.21.15.27:0/3216247626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d300ccd70 0x7f3d30158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:09.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.734+0000 7f3d3519c700 1 --2- 172.21.15.27:0/3216247626 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3d3015ae70 0x7f3d300c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:09.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.734+0000 7f3d2ffff700 1 -- 172.21.15.27:0/3216247626 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3d3015ae70 msgr2=0x7f3d300c9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:09.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.734+0000 7f3d2ffff700 1 --2- 172.21.15.27:0/3216247626 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3d3015ae70 0x7f3d300c9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.734+0000 7f3d2ffff700 1 -- 172.21.15.27:0/3216247626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d300ccd70 msgr2=0x7f3d30158930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:09.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.734+0000 7f3d2ffff700 1 --2- 172.21.15.27:0/3216247626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d300ccd70 0x7f3d30158930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.734+0000 7f3d2ffff700 1 -- 172.21.15.27:0/3216247626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c009420 con 0x7f3d3015a1b0 2024-01-11T09:46:09.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.734+0000 7f3d2ffff700 1 --2- 172.21.15.27:0/3216247626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d3015a1b0 0x7f3d30169740 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f3d30069770 tx=0x7f3d1c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:09.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.734+0000 7f3d2cff9700 1 -- 172.21.15.27:0/3216247626 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c01f070 con 0x7f3d3015a1b0 2024-01-11T09:46:09.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.735+0000 7f3d2cff9700 1 -- 172.21.15.27:0/3216247626 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c002f70 con 0x7f3d3015a1b0 2024-01-11T09:46:09.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.735+0000 7f3d36bff700 1 -- 172.21.15.27:0/3216247626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d300b4710 con 0x7f3d3015a1b0 2024-01-11T09:46:09.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.735+0000 7f3d2cff9700 1 -- 172.21.15.27:0/3216247626 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c00b910 con 0x7f3d3015a1b0 2024-01-11T09:46:09.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.735+0000 7f3d36bff700 1 -- 172.21.15.27:0/3216247626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d300b4cd0 con 0x7f3d3015a1b0 2024-01-11T09:46:09.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.736+0000 7f3d36bff700 1 -- 172.21.15.27:0/3216247626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d300b51f0 con 0x7f3d3015a1b0 2024-01-11T09:46:09.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.737+0000 7f3d2cff9700 1 -- 172.21.15.27:0/3216247626 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f3d1c023070 con 0x7f3d3015a1b0 2024-01-11T09:46:09.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.737+0000 7f3d2cff9700 1 --2- 172.21.15.27:0/3216247626 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3d14067170 0x7f3d14069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:09.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.738+0000 7f3d2cff9700 1 -- 172.21.15.27:0/3216247626 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f3d1c085ea0 con 0x7f3d3015a1b0 2024-01-11T09:46:09.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.738+0000 7f3d3499b700 1 --2- 172.21.15.27:0/3216247626 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3d14067170 0x7f3d14069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:09.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.739+0000 7f3d3499b700 1 --2- 172.21.15.27:0/3216247626 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3d14067170 0x7f3d14069620 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3d24007fd0 tx=0x7f3d24007d30 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:09.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.742+0000 7f3d2cff9700 1 -- 172.21.15.27:0/3216247626 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3d300b51f0 con 0x7f3d3015a1b0 2024-01-11T09:46:09.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.945+0000 7f3d36bff700 1 -- 172.21.15.27:0/3216247626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f3d300b51b0 con 0x7f3d3015a1b0 2024-01-11T09:46:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.945+0000 7f3d2cff9700 1 -- 172.21.15.27:0/3216247626 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v223) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3d1c052a60 con 0x7f3d3015a1b0 2024-01-11T09:46:09.937 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T09:46:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.948+0000 7f3d36bff700 1 -- 172.21.15.27:0/3216247626 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3d14067170 msgr2=0x7f3d14069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.948+0000 7f3d36bff700 1 --2- 172.21.15.27:0/3216247626 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3d14067170 0x7f3d14069620 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3d24007fd0 tx=0x7f3d24007d30 comp rx=0 tx=0).stop 2024-01-11T09:46:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.948+0000 7f3d36bff700 1 -- 172.21.15.27:0/3216247626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d3015a1b0 msgr2=0x7f3d30169740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.948+0000 7f3d36bff700 1 --2- 172.21.15.27:0/3216247626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d3015a1b0 0x7f3d30169740 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f3d30069770 tx=0x7f3d1c017000 comp rx=0 tx=0).stop 2024-01-11T09:46:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.948+0000 7f3d36bff700 1 -- 172.21.15.27:0/3216247626 shutdown_connections 2024-01-11T09:46:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.948+0000 7f3d36bff700 1 --2- 172.21.15.27:0/3216247626 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3d3015ae70 0x7f3d300c9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.948+0000 7f3d36bff700 1 --2- 172.21.15.27:0/3216247626 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3d14067170 0x7f3d14069620 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.948+0000 7f3d36bff700 1 --2- 172.21.15.27:0/3216247626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d300ccd70 0x7f3d30158930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.948+0000 7f3d36bff700 1 --2- 172.21.15.27:0/3216247626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d3015a1b0 0x7f3d30169740 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.948+0000 7f3d36bff700 1 -- 172.21.15.27:0/3216247626 >> 172.21.15.27:0/3216247626 conn(0x7f3d3006ed40 msgr2=0x7f3d300d0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.948+0000 7f3d36bff700 1 -- 172.21.15.27:0/3216247626 shutdown_connections 2024-01-11T09:46:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:09.948+0000 7f3d36bff700 1 -- 172.21.15.27:0/3216247626 wait complete. 2024-01-11T09:46:09.958 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:46:10.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.169+0000 7fb85dfff700 1 Processor -- start 2024-01-11T09:46:10.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.170+0000 7fb85dfff700 1 -- start start 2024-01-11T09:46:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.171+0000 7fb85dfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb8580b03e0 0x7fb8580b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.171+0000 7fb85dfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8580b29c0 0x7fb858159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.171+0000 7fb85dfff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8580cd730 con 0x7fb8580b07b0 2024-01-11T09:46:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.171+0000 7fb85dfff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8580b2de0 con 0x7fb8580b03e0 2024-01-11T09:46:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.171+0000 7fb85dfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8580ccb20 con 0x7fb8580b29c0 2024-01-11T09:46:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.171+0000 7fb856ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb8580b07b0 0x7fb858159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37230/0 (socket says 172.21.15.27:37230) 2024-01-11T09:46:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.171+0000 7fb856ffd700 1 -- 172.21.15.27:0/1404986304 learned_addr learned my addr 172.21.15.27:0/1404986304 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.171+0000 7fb857fff700 1 --2- 172.21.15.27:0/1404986304 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8580b29c0 0x7fb858159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.171+0000 7fb8577fe700 1 --2- 172.21.15.27:0/1404986304 >> v2:172.21.15.77:3300/0 conn(0x7fb8580b03e0 0x7fb8580b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.171+0000 7fb8567fc700 1 -- 172.21.15.27:0/1404986304 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541920214 0 0) 0x7fb8580cd730 con 0x7fb8580b07b0 2024-01-11T09:46:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.172+0000 7fb8567fc700 1 -- 172.21.15.27:0/1404986304 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb840003570 con 0x7fb8580b07b0 2024-01-11T09:46:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.172+0000 7fb8567fc700 1 -- 172.21.15.27:0/1404986304 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1763687239 0 0) 0x7fb840003570 con 0x7fb8580b07b0 2024-01-11T09:46:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.172+0000 7fb857fff700 1 -- 172.21.15.27:0/1404986304 >> v2:172.21.15.77:3300/0 conn(0x7fb8580b03e0 msgr2=0x7fb8580b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.172+0000 7fb857fff700 1 --2- 172.21.15.27:0/1404986304 >> v2:172.21.15.77:3300/0 conn(0x7fb8580b03e0 0x7fb8580b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.172+0000 7fb857fff700 1 -- 172.21.15.27:0/1404986304 >> v1:172.21.15.27:6789/0 conn(0x7fb8580b07b0 legacy=0x7fb858159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.172+0000 7fb857fff700 1 -- 172.21.15.27:0/1404986304 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8580cce00 con 0x7fb8580b29c0 2024-01-11T09:46:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.172+0000 7fb857fff700 1 --2- 172.21.15.27:0/1404986304 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8580b29c0 0x7fb858159910 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb850009e60 tx=0x7fb850009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbdd83ab286c235 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.172+0000 7fb8567fc700 1 -- 172.21.15.27:0/1404986304 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85000e070 con 0x7fb8580b29c0 2024-01-11T09:46:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.173+0000 7fb8567fc700 1 -- 172.21.15.27:0/1404986304 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85000b7a0 con 0x7fb8580b29c0 2024-01-11T09:46:10.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.173+0000 7fb8567fc700 1 -- 172.21.15.27:0/1404986304 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c001470 con 0x7fb8580b29c0 2024-01-11T09:46:10.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.173+0000 7fb85dfff700 1 -- 172.21.15.27:0/1404986304 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8580b29c0 msgr2=0x7fb858159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:10.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.173+0000 7fb85dfff700 1 --2- 172.21.15.27:0/1404986304 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8580b29c0 0x7fb858159910 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb850009e60 tx=0x7fb850009780 comp rx=0 tx=0).stop 2024-01-11T09:46:10.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.173+0000 7fb85dfff700 1 -- 172.21.15.27:0/1404986304 shutdown_connections 2024-01-11T09:46:10.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.173+0000 7fb85dfff700 1 --2- 172.21.15.27:0/1404986304 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8580b29c0 0x7fb858159910 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:10.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.173+0000 7fb85dfff700 1 --2- 172.21.15.27:0/1404986304 >> v2:172.21.15.77:3300/0 conn(0x7fb8580b03e0 0x7fb8580b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:10.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.173+0000 7fb85dfff700 1 -- 172.21.15.27:0/1404986304 >> 172.21.15.27:0/1404986304 conn(0x7fb85806ed40 msgr2=0x7fb85806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:10.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.173+0000 7fb85dfff700 1 -- 172.21.15.27:0/1404986304 shutdown_connections 2024-01-11T09:46:10.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.173+0000 7fb85dfff700 1 -- 172.21.15.27:0/1404986304 wait complete. 2024-01-11T09:46:10.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.175+0000 7fb85dfff700 1 Processor -- start 2024-01-11T09:46:10.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.175+0000 7fb85dfff700 1 -- start start 2024-01-11T09:46:10.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.176+0000 7fb85dfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8580b03e0 0x7fb8580c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:10.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.176+0000 7fb85dfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8580b07b0 0x7fb85816a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:10.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.176+0000 7fb85dfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8580b29c0 0x7fb858158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:10.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.176+0000 7fb85dfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8580c9030 con 0x7fb8580b29c0 2024-01-11T09:46:10.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.176+0000 7fb85dfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8580c8660 con 0x7fb8580b03e0 2024-01-11T09:46:10.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.176+0000 7fb85dfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8580af190 con 0x7fb8580b07b0 2024-01-11T09:46:10.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.176+0000 7fb857fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8580b29c0 0x7fb858158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:10.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.176+0000 7fb857fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8580b29c0 0x7fb858158ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42814/0 (socket says 172.21.15.27:42814) 2024-01-11T09:46:10.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.176+0000 7fb857fff700 1 -- 172.21.15.27:0/398814445 learned_addr learned my addr 172.21.15.27:0/398814445 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:10.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.176+0000 7fb8577fe700 1 --2- 172.21.15.27:0/398814445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8580b03e0 0x7fb8580c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:10.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.176+0000 7fb856ffd700 1 --2- 172.21.15.27:0/398814445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8580b07b0 0x7fb85816a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:10.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.176+0000 7fb857fff700 1 -- 172.21.15.27:0/398814445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8580b07b0 msgr2=0x7fb85816a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:10.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.176+0000 7fb857fff700 1 --2- 172.21.15.27:0/398814445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8580b07b0 0x7fb85816a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:10.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.176+0000 7fb857fff700 1 -- 172.21.15.27:0/398814445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8580b03e0 msgr2=0x7fb8580c8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:10.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.176+0000 7fb857fff700 1 --2- 172.21.15.27:0/398814445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8580b03e0 0x7fb8580c8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:10.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.176+0000 7fb857fff700 1 -- 172.21.15.27:0/398814445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb850009a20 con 0x7fb8580b29c0 2024-01-11T09:46:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.176+0000 7fb856ffd700 1 --2- 172.21.15.27:0/398814445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8580b07b0 0x7fb85816a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.177+0000 7fb857fff700 1 --2- 172.21.15.27:0/398814445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8580b29c0 0x7fb858158ab0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fb85000b610 tx=0x7fb850018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.177+0000 7fb83f7fe700 1 -- 172.21.15.27:0/398814445 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb850012ca0 con 0x7fb8580b29c0 2024-01-11T09:46:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.177+0000 7fb83f7fe700 1 -- 172.21.15.27:0/398814445 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb850012e00 con 0x7fb8580b29c0 2024-01-11T09:46:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.177+0000 7fb83f7fe700 1 -- 172.21.15.27:0/398814445 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85001e680 con 0x7fb8580b29c0 2024-01-11T09:46:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.177+0000 7fb85dfff700 1 -- 172.21.15.27:0/398814445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85816a970 con 0x7fb8580b29c0 2024-01-11T09:46:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.177+0000 7fb85dfff700 1 -- 172.21.15.27:0/398814445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8580c7d30 con 0x7fb8580b29c0 2024-01-11T09:46:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.178+0000 7fb85dfff700 1 -- 172.21.15.27:0/398814445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8580c87d0 con 0x7fb8580b29c0 2024-01-11T09:46:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.179+0000 7fb83f7fe700 1 -- 172.21.15.27:0/398814445 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fb85001ea70 con 0x7fb8580b29c0 2024-01-11T09:46:10.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.180+0000 7fb83f7fe700 1 --2- 172.21.15.27:0/398814445 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb844067170 0x7fb844069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:10.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.180+0000 7fb83f7fe700 1 -- 172.21.15.27:0/398814445 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6560+0+0 (secure 0 0 0) 0x7fb850086b00 con 0x7fb8580b29c0 2024-01-11T09:46:10.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.180+0000 7fb8577fe700 1 --2- 172.21.15.27:0/398814445 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb844067170 0x7fb844069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:10.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.181+0000 7fb8577fe700 1 --2- 172.21.15.27:0/398814445 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb844067170 0x7fb844069620 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb848009db0 tx=0x7fb8480091a0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:10.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.184+0000 7fb83f7fe700 1 -- 172.21.15.27:0/398814445 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb850088e60 con 0x7fb8580b29c0 2024-01-11T09:46:10.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.429+0000 7fb85dfff700 1 -- 172.21.15.27:0/398814445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8580718d0 con 0x7fb8580b29c0 2024-01-11T09:46:10.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.430+0000 7fb83f7fe700 1 -- 172.21.15.27:0/398814445 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+4614 (secure 0 0 0) 0x7fb850053600 con 0x7fb8580b29c0 2024-01-11T09:46:10.422 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:10.422 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":119,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2103359888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":2103359888},{"type":"v1","addr":"172.21.15.77:6834","nonce":2103359888}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8795,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2650824649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2650824649},{"type":"v1","addr":"172.21.15.77:6835","nonce":2650824649}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106},{"gid":11157,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1211538286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1211538286},{"type":"v1","addr":"172.21.15.27:6835","nonce":1211538286}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:45:37.460120+0000","modified":"2024-01-11T09:45:40.977702+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10596},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10596":{"gid":10596,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6834/2807751836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2807751836},{"type":"v1","addr":"172.21.15.27:6834","nonce":2807751836}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":14}]} 2024-01-11T09:46:10.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.432+0000 7fb85dfff700 1 -- 172.21.15.27:0/398814445 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb844067170 msgr2=0x7fb844069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:10.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.432+0000 7fb85dfff700 1 --2- 172.21.15.27:0/398814445 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb844067170 0x7fb844069620 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb848009db0 tx=0x7fb8480091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:10.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.432+0000 7fb85dfff700 1 -- 172.21.15.27:0/398814445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8580b29c0 msgr2=0x7fb858158ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:10.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.432+0000 7fb85dfff700 1 --2- 172.21.15.27:0/398814445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8580b29c0 0x7fb858158ab0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fb85000b610 tx=0x7fb850018000 comp rx=0 tx=0).stop 2024-01-11T09:46:10.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.433+0000 7fb85dfff700 1 -- 172.21.15.27:0/398814445 shutdown_connections 2024-01-11T09:46:10.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.433+0000 7fb85dfff700 1 --2- 172.21.15.27:0/398814445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8580b29c0 0x7fb858158ab0 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:10.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.433+0000 7fb85dfff700 1 --2- 172.21.15.27:0/398814445 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb844067170 0x7fb844069620 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:10.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.433+0000 7fb85dfff700 1 --2- 172.21.15.27:0/398814445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8580b03e0 0x7fb8580c8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:10.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.433+0000 7fb85dfff700 1 --2- 172.21.15.27:0/398814445 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb8580b07b0 0x7fb85816a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:10.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.433+0000 7fb85dfff700 1 -- 172.21.15.27:0/398814445 >> 172.21.15.27:0/398814445 conn(0x7fb85806ed40 msgr2=0x7fb8580cbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:10.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.433+0000 7fb85dfff700 1 -- 172.21.15.27:0/398814445 shutdown_connections 2024-01-11T09:46:10.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.433+0000 7fb85dfff700 1 -- 172.21.15.27:0/398814445 wait complete. 2024-01-11T09:46:10.426 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 119 2024-01-11T09:46:10.442 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:46:10.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.650+0000 7f94db7ff700 1 Processor -- start 2024-01-11T09:46:10.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.650+0000 7f94db7ff700 1 -- start start 2024-01-11T09:46:10.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.652+0000 7f94db7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94d40ca3e0 0x7f94d40cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:10.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.652+0000 7f94db7ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94d40ca7b0 0x7f94d4159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:10.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.652+0000 7f94db7ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d415d640 con 0x7f94d40c8a20 2024-01-11T09:46:10.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.652+0000 7f94db7ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94d415ae70 con 0x7f94d40ca7b0 2024-01-11T09:46:10.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.652+0000 7f94db7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d415a3c0 con 0x7f94d40ca3e0 2024-01-11T09:46:10.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.652+0000 7f94d8d9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94d40ca7b0 0x7f94d4159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:10.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.652+0000 7f94d8d9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94d40ca7b0 0x7f94d4159e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42824/0 (socket says 172.21.15.27:42824) 2024-01-11T09:46:10.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.652+0000 7f94d8d9a700 1 -- 172.21.15.27:0/4005057427 learned_addr learned my addr 172.21.15.27:0/4005057427 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:10.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.652+0000 7f94d959b700 1 --2- 172.21.15.27:0/4005057427 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94d40ca3e0 0x7f94d40cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:10.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.653+0000 7f94d8d9a700 1 -- 172.21.15.27:0/4005057427 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94d40ca3e0 msgr2=0x7f94d40cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:10.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.653+0000 7f94d8d9a700 1 --2- 172.21.15.27:0/4005057427 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94d40ca3e0 0x7f94d40cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:10.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.653+0000 7f94d8d9a700 1 -- 172.21.15.27:0/4005057427 >> v1:172.21.15.77:6789/0 conn(0x7f94d40c8a20 legacy=0x7f94d415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:10.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.653+0000 7f94d8d9a700 1 -- 172.21.15.27:0/4005057427 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d415a5d0 con 0x7f94d40ca7b0 2024-01-11T09:46:10.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.653+0000 7f94d8d9a700 1 --2- 172.21.15.27:0/4005057427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d40ca7b0 0x7f94d4159e80 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f94c40099b0 tx=0x7f94c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef508dc165fb31 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:10.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.653+0000 7f94cbfff700 1 -- 172.21.15.27:0/4005057427 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c400b860 con 0x7f94d40ca7b0 2024-01-11T09:46:10.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.653+0000 7f94cbfff700 1 -- 172.21.15.27:0/4005057427 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c400b9c0 con 0x7f94d40ca7b0 2024-01-11T09:46:10.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.653+0000 7f94cbfff700 1 -- 172.21.15.27:0/4005057427 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c400bd60 con 0x7f94d40ca7b0 2024-01-11T09:46:10.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.654+0000 7f94db7ff700 1 -- 172.21.15.27:0/4005057427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d40ca7b0 msgr2=0x7f94d4159e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:10.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.654+0000 7f94db7ff700 1 --2- 172.21.15.27:0/4005057427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d40ca7b0 0x7f94d4159e80 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f94c40099b0 tx=0x7f94c4009250 comp rx=0 tx=0).stop 2024-01-11T09:46:10.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.654+0000 7f94db7ff700 1 -- 172.21.15.27:0/4005057427 shutdown_connections 2024-01-11T09:46:10.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.654+0000 7f94db7ff700 1 --2- 172.21.15.27:0/4005057427 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94d40ca3e0 0x7f94d40cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:10.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.654+0000 7f94db7ff700 1 --2- 172.21.15.27:0/4005057427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d40ca7b0 0x7f94d4159e80 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:10.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.654+0000 7f94db7ff700 1 -- 172.21.15.27:0/4005057427 >> 172.21.15.27:0/4005057427 conn(0x7f94d40ce3e0 msgr2=0x7f94d40ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:10.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.654+0000 7f94db7ff700 1 -- 172.21.15.27:0/4005057427 shutdown_connections 2024-01-11T09:46:10.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.654+0000 7f94db7ff700 1 -- 172.21.15.27:0/4005057427 wait complete. 2024-01-11T09:46:10.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.656+0000 7f94db7ff700 1 Processor -- start 2024-01-11T09:46:10.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.656+0000 7f94db7ff700 1 -- start start 2024-01-11T09:46:10.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.657+0000 7f94db7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94d40c8a20 0x7f94d40b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:10.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.657+0000 7f94db7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d40ca3e0 0x7f94d406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:10.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.657+0000 7f94db7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d40ca7b0 0x7f94d41439e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:10.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.657+0000 7f94db7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d40c6530 con 0x7f94d40ca7b0 2024-01-11T09:46:10.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.657+0000 7f94db7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d40b4080 con 0x7f94d40ca3e0 2024-01-11T09:46:10.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.657+0000 7f94db7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d40c5920 con 0x7f94d40c8a20 2024-01-11T09:46:10.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.657+0000 7f94d9d9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d40ca7b0 0x7f94d41439e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:10.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.657+0000 7f94d9d9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d40ca7b0 0x7f94d41439e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42832/0 (socket says 172.21.15.27:42832) 2024-01-11T09:46:10.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.657+0000 7f94d9d9c700 1 -- 172.21.15.27:0/3158265350 learned_addr learned my addr 172.21.15.27:0/3158265350 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:10.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.657+0000 7f94d8d9a700 1 --2- 172.21.15.27:0/3158265350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d40ca3e0 0x7f94d406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.657+0000 7f94d959b700 1 --2- 172.21.15.27:0/3158265350 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94d40c8a20 0x7f94d40b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.657+0000 7f94d9d9c700 1 -- 172.21.15.27:0/3158265350 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94d40c8a20 msgr2=0x7f94d40b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.657+0000 7f94d9d9c700 1 --2- 172.21.15.27:0/3158265350 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94d40c8a20 0x7f94d40b3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.657+0000 7f94d9d9c700 1 -- 172.21.15.27:0/3158265350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d40ca3e0 msgr2=0x7f94d406d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.657+0000 7f94d9d9c700 1 --2- 172.21.15.27:0/3158265350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d40ca3e0 0x7f94d406d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.657+0000 7f94d9d9c700 1 -- 172.21.15.27:0/3158265350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c40094f0 con 0x7f94d40ca7b0 2024-01-11T09:46:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.658+0000 7f94d9d9c700 1 --2- 172.21.15.27:0/3158265350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d40ca7b0 0x7f94d41439e0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f94cc00c840 tx=0x7f94cc001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:10.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.658+0000 7f94c97fa700 1 -- 172.21.15.27:0/3158265350 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94cc012630 con 0x7f94d40ca7b0 2024-01-11T09:46:10.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.658+0000 7f94c97fa700 1 -- 172.21.15.27:0/3158265350 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94cc0127b0 con 0x7f94d40ca7b0 2024-01-11T09:46:10.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.658+0000 7f94c97fa700 1 -- 172.21.15.27:0/3158265350 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94cc012b70 con 0x7f94d40ca7b0 2024-01-11T09:46:10.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.658+0000 7f94db7ff700 1 -- 172.21.15.27:0/3158265350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d40c5c00 con 0x7f94d40ca7b0 2024-01-11T09:46:10.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.658+0000 7f94db7ff700 1 -- 172.21.15.27:0/3158265350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d40c61c0 con 0x7f94d40ca7b0 2024-01-11T09:46:10.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.659+0000 7f94db7ff700 1 -- 172.21.15.27:0/3158265350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d40718d0 con 0x7f94d40ca7b0 2024-01-11T09:46:10.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.660+0000 7f94c97fa700 1 -- 172.21.15.27:0/3158265350 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f94cc003030 con 0x7f94d40ca7b0 2024-01-11T09:46:10.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.661+0000 7f94c97fa700 1 --2- 172.21.15.27:0/3158265350 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f94b8067170 0x7f94b8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:10.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.661+0000 7f94c97fa700 1 -- 172.21.15.27:0/3158265350 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f94cc084ba0 con 0x7f94d40ca7b0 2024-01-11T09:46:10.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.661+0000 7f94d959b700 1 --2- 172.21.15.27:0/3158265350 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f94b8067170 0x7f94b8069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:10.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.662+0000 7f94d959b700 1 --2- 172.21.15.27:0/3158265350 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f94b8067170 0x7f94b8069620 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f94d0006fd0 tx=0x7f94d0008040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.665+0000 7f94c97fa700 1 -- 172.21.15.27:0/3158265350 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f94cc051830 con 0x7f94d40ca7b0 2024-01-11T09:46:10.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.867+0000 7f94db7ff700 1 -- 172.21.15.27:0/3158265350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94d415ceb0 con 0x7f94d40ca7b0 2024-01-11T09:46:10.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.867+0000 7f94c97fa700 1 -- 172.21.15.27:0/3158265350 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v223) v1 ==== 74+0+16559 (secure 0 0 0) 0x7f94cc054e50 con 0x7f94d40ca7b0 2024-01-11T09:46:10.859 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:10.859 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":223,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:46:08.139060+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-01-11T09:45:35.631727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-01-11T09:45:36.452195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/1143960902":"2024-01-12T09:46:04.482215+0000","172.21.15.27:6833/1143960902":"2024-01-12T09:46:04.482215+0000","172.21.15.77:0/3729201132":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/3605354921":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/4084841078":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/3487780416":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/534676732":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/3735282980":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/1375879589":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/2192171652":"2024-01-12T09:45:52.192235+0000","172.21.15.77:6837/59776":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/2635904665":"2024-01-12T09:46:04.127727+0000","172.21.15.77:6836/59776":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/2654175097":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/652362578":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/749658672":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/2439942664":"2024-01-12T09:45:52.192235+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:46:10.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.870+0000 7f94db7ff700 1 -- 172.21.15.27:0/3158265350 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f94b8067170 msgr2=0x7f94b8069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:10.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.870+0000 7f94db7ff700 1 --2- 172.21.15.27:0/3158265350 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f94b8067170 0x7f94b8069620 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f94d0006fd0 tx=0x7f94d0008040 comp rx=0 tx=0).stop 2024-01-11T09:46:10.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.870+0000 7f94db7ff700 1 -- 172.21.15.27:0/3158265350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d40ca7b0 msgr2=0x7f94d41439e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:10.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.870+0000 7f94db7ff700 1 --2- 172.21.15.27:0/3158265350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d40ca7b0 0x7f94d41439e0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f94cc00c840 tx=0x7f94cc001670 comp rx=0 tx=0).stop 2024-01-11T09:46:10.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.870+0000 7f94db7ff700 1 -- 172.21.15.27:0/3158265350 shutdown_connections 2024-01-11T09:46:10.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.870+0000 7f94db7ff700 1 --2- 172.21.15.27:0/3158265350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94d40ca7b0 0x7f94d41439e0 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:10.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.870+0000 7f94db7ff700 1 --2- 172.21.15.27:0/3158265350 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f94d40c8a20 0x7f94d40b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:10.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.870+0000 7f94db7ff700 1 --2- 172.21.15.27:0/3158265350 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f94b8067170 0x7f94b8069620 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:10.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.870+0000 7f94db7ff700 1 --2- 172.21.15.27:0/3158265350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d40ca3e0 0x7f94d406d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:10.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.870+0000 7f94db7ff700 1 -- 172.21.15.27:0/3158265350 >> 172.21.15.27:0/3158265350 conn(0x7f94d40ce3e0 msgr2=0x7f94d415b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:10.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.870+0000 7f94db7ff700 1 -- 172.21.15.27:0/3158265350 shutdown_connections 2024-01-11T09:46:10.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:10.870+0000 7f94db7ff700 1 -- 172.21.15.27:0/3158265350 wait complete. 2024-01-11T09:46:10.879 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2024-01-11T09:46:10.880 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:46:11.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.090+0000 7f8fe23ff700 1 Processor -- start 2024-01-11T09:46:11.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.090+0000 7f8fe23ff700 1 -- start start 2024-01-11T09:46:11.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.091+0000 7f8fe23ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fdc0b09e0 0x7f8fdc1610a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:11.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.091+0000 7f8fe23ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fdc157390 con 0x7f8fdc069d40 2024-01-11T09:46:11.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.091+0000 7f8fe23ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fdc06a200 con 0x7f8fdc0c8620 2024-01-11T09:46:11.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.091+0000 7f8fe23ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc0adcd0 con 0x7f8fdc0b09e0 2024-01-11T09:46:11.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.092+0000 7f8fdb7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8fdc069d40 0x7f8fdc0b02d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37244/0 (socket says 172.21.15.27:37244) 2024-01-11T09:46:11.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.092+0000 7f8fdb7fe700 1 -- 172.21.15.27:0/611698175 learned_addr learned my addr 172.21.15.27:0/611698175 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:11.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.092+0000 7f8fe099c700 1 --2- 172.21.15.27:0/611698175 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fdc0b09e0 0x7f8fdc1610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:11.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.092+0000 7f8fdaffd700 1 -- 172.21.15.27:0/611698175 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101958485 0 0) 0x7f8fdc157390 con 0x7f8fdc069d40 2024-01-11T09:46:11.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.092+0000 7f8fdaffd700 1 -- 172.21.15.27:0/611698175 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fc4003570 con 0x7f8fdc069d40 2024-01-11T09:46:11.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.092+0000 7f8fdaffd700 1 -- 172.21.15.27:0/611698175 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562785001 0 0) 0x7f8fdc06a200 con 0x7f8fdc0c8620 2024-01-11T09:46:11.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.092+0000 7f8fdaffd700 1 -- 172.21.15.27:0/611698175 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fdc157390 con 0x7f8fdc0c8620 2024-01-11T09:46:11.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.092+0000 7f8fe099c700 1 -- 172.21.15.27:0/611698175 >> v1:172.21.15.77:6789/0 conn(0x7f8fdc0c8620 legacy=0x7f8fdc0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:11.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.092+0000 7f8fe099c700 1 -- 172.21.15.27:0/611698175 >> v1:172.21.15.27:6789/0 conn(0x7f8fdc069d40 legacy=0x7f8fdc0b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:11.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.092+0000 7f8fe099c700 1 -- 172.21.15.27:0/611698175 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fdc156d40 con 0x7f8fdc0b09e0 2024-01-11T09:46:11.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.093+0000 7f8fe099c700 1 --2- 172.21.15.27:0/611698175 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fdc0b09e0 0x7f8fdc1610a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009e90 tx=0x7f8fd4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35e0c65d987bea17 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:11.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.093+0000 7f8fdaffd700 1 -- 172.21.15.27:0/611698175 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd400e070 con 0x7f8fdc0b09e0 2024-01-11T09:46:11.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.093+0000 7f8fdaffd700 1 -- 172.21.15.27:0/611698175 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd400bb80 con 0x7f8fdc0b09e0 2024-01-11T09:46:11.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.093+0000 7f8fdaffd700 1 -- 172.21.15.27:0/611698175 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4012430 con 0x7f8fdc0b09e0 2024-01-11T09:46:11.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.093+0000 7f8fe23ff700 1 -- 172.21.15.27:0/611698175 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fdc0b09e0 msgr2=0x7f8fdc1610a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:11.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.093+0000 7f8fe23ff700 1 --2- 172.21.15.27:0/611698175 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fdc0b09e0 0x7f8fdc1610a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009e90 tx=0x7f8fd4009780 comp rx=0 tx=0).stop 2024-01-11T09:46:11.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.094+0000 7f8fe23ff700 1 -- 172.21.15.27:0/611698175 shutdown_connections 2024-01-11T09:46:11.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.094+0000 7f8fe23ff700 1 --2- 172.21.15.27:0/611698175 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fdc0b09e0 0x7f8fdc1610a0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.094+0000 7f8fe23ff700 1 -- 172.21.15.27:0/611698175 >> 172.21.15.27:0/611698175 conn(0x7f8fdc0680e0 msgr2=0x7f8fdc069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:11.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.094+0000 7f8fe23ff700 1 -- 172.21.15.27:0/611698175 shutdown_connections 2024-01-11T09:46:11.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.094+0000 7f8fe23ff700 1 -- 172.21.15.27:0/611698175 wait complete. 2024-01-11T09:46:11.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.095+0000 7f8fe23ff700 1 Processor -- start 2024-01-11T09:46:11.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.095+0000 7f8fe23ff700 1 -- start start 2024-01-11T09:46:11.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.096+0000 7f8fe23ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fdc069d40 0x7f8fdc158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:11.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.096+0000 7f8fe23ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fdc0b09e0 0x7f8fdc0c9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:11.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.096+0000 7f8fe23ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fdc0c8620 0x7f8fdc0ca850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:11.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.096+0000 7f8fe23ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc0c9580 con 0x7f8fdc0c8620 2024-01-11T09:46:11.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.096+0000 7f8fe23ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc0ca3c0 con 0x7f8fdc0b09e0 2024-01-11T09:46:11.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.096+0000 7f8fe23ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc0c97b0 con 0x7f8fdc069d40 2024-01-11T09:46:11.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.097+0000 7f8fe099c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fdc0c8620 0x7f8fdc0ca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:11.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.097+0000 7f8fdb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fdc0b09e0 0x7f8fdc0c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:11.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.097+0000 7f8fe099c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fdc0c8620 0x7f8fdc0ca850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42846/0 (socket says 172.21.15.27:42846) 2024-01-11T09:46:11.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.097+0000 7f8fe099c700 1 -- 172.21.15.27:0/2733551242 learned_addr learned my addr 172.21.15.27:0/2733551242 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:11.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.097+0000 7f8fdbfff700 1 --2- 172.21.15.27:0/2733551242 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fdc069d40 0x7f8fdc158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:11.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.097+0000 7f8fe099c700 1 -- 172.21.15.27:0/2733551242 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fdc069d40 msgr2=0x7f8fdc158a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:11.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.097+0000 7f8fe099c700 1 --2- 172.21.15.27:0/2733551242 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fdc069d40 0x7f8fdc158a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.097+0000 7f8fe099c700 1 -- 172.21.15.27:0/2733551242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fdc0b09e0 msgr2=0x7f8fdc0c9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:11.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.097+0000 7f8fe099c700 1 --2- 172.21.15.27:0/2733551242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fdc0b09e0 0x7f8fdc0c9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.097+0000 7f8fe099c700 1 -- 172.21.15.27:0/2733551242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd4009a20 con 0x7f8fdc0c8620 2024-01-11T09:46:11.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.097+0000 7f8fe099c700 1 --2- 172.21.15.27:0/2733551242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fdc0c8620 0x7f8fdc0ca850 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4013000 tx=0x7f8fd4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:11.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.098+0000 7f8fc3fff700 1 -- 172.21.15.27:0/2733551242 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd401b420 con 0x7f8fdc0c8620 2024-01-11T09:46:11.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.098+0000 7f8fc3fff700 1 -- 172.21.15.27:0/2733551242 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd401b5a0 con 0x7f8fdc0c8620 2024-01-11T09:46:11.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.098+0000 7f8fe23ff700 1 -- 172.21.15.27:0/2733551242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fdc0c9a90 con 0x7f8fdc0c8620 2024-01-11T09:46:11.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.098+0000 7f8fc3fff700 1 -- 172.21.15.27:0/2733551242 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd401b960 con 0x7f8fdc0c8620 2024-01-11T09:46:11.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.098+0000 7f8fe23ff700 1 -- 172.21.15.27:0/2733551242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fdc0ca0d0 con 0x7f8fdc0c8620 2024-01-11T09:46:11.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.099+0000 7f8fe23ff700 1 -- 172.21.15.27:0/2733551242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fdc0caeb0 con 0x7f8fdc0c8620 2024-01-11T09:46:11.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.100+0000 7f8fc3fff700 1 -- 172.21.15.27:0/2733551242 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f8fd401bd50 con 0x7f8fdc0c8620 2024-01-11T09:46:11.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.101+0000 7f8fc3fff700 1 --2- 172.21.15.27:0/2733551242 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8fc8067170 0x7f8fc8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:11.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.101+0000 7f8fc3fff700 1 -- 172.21.15.27:0/2733551242 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f8fd4086ae0 con 0x7f8fdc0c8620 2024-01-11T09:46:11.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.102+0000 7f8fdbfff700 1 --2- 172.21.15.27:0/2733551242 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8fc8067170 0x7f8fc8069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:11.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.102+0000 7f8fdbfff700 1 --2- 172.21.15.27:0/2733551242 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8fc8067170 0x7f8fc8069620 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc007fd0 tx=0x7f8fcc007d50 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:11.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.106+0000 7f8fc3fff700 1 -- 172.21.15.27:0/2733551242 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8fd40536a0 con 0x7f8fdc0c8620 2024-01-11T09:46:11.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.355+0000 7f8fe23ff700 1 -- 172.21.15.27:0/2733551242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fdc0718d0 con 0x7f8fdc0c8620 2024-01-11T09:46:11.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.355+0000 7f8fc3fff700 1 -- 172.21.15.27:0/2733551242 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+4614 (secure 0 0 0) 0x7f8fd4056c00 con 0x7f8fdc0c8620 2024-01-11T09:46:11.347 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:11.347 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":119,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2103359888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":2103359888},{"type":"v1","addr":"172.21.15.77:6834","nonce":2103359888}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8795,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2650824649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2650824649},{"type":"v1","addr":"172.21.15.77:6835","nonce":2650824649}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106},{"gid":11157,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1211538286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1211538286},{"type":"v1","addr":"172.21.15.27:6835","nonce":1211538286}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:45:37.460120+0000","modified":"2024-01-11T09:45:40.977702+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10596},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10596":{"gid":10596,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6834/2807751836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2807751836},{"type":"v1","addr":"172.21.15.27:6834","nonce":2807751836}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":14}]} 2024-01-11T09:46:11.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.358+0000 7f8fe23ff700 1 -- 172.21.15.27:0/2733551242 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8fc8067170 msgr2=0x7f8fc8069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:11.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.358+0000 7f8fe23ff700 1 --2- 172.21.15.27:0/2733551242 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8fc8067170 0x7f8fc8069620 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc007fd0 tx=0x7f8fcc007d50 comp rx=0 tx=0).stop 2024-01-11T09:46:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.358+0000 7f8fe23ff700 1 -- 172.21.15.27:0/2733551242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fdc0c8620 msgr2=0x7f8fdc0ca850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.358+0000 7f8fe23ff700 1 --2- 172.21.15.27:0/2733551242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fdc0c8620 0x7f8fdc0ca850 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4013000 tx=0x7f8fd4019000 comp rx=0 tx=0).stop 2024-01-11T09:46:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.358+0000 7f8fe23ff700 1 -- 172.21.15.27:0/2733551242 shutdown_connections 2024-01-11T09:46:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.358+0000 7f8fe23ff700 1 --2- 172.21.15.27:0/2733551242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fdc0c8620 0x7f8fdc0ca850 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.358+0000 7f8fe23ff700 1 --2- 172.21.15.27:0/2733551242 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fdc069d40 0x7f8fdc158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.358+0000 7f8fe23ff700 1 --2- 172.21.15.27:0/2733551242 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8fc8067170 0x7f8fc8069620 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.358+0000 7f8fe23ff700 1 --2- 172.21.15.27:0/2733551242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fdc0b09e0 0x7f8fdc0c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.358+0000 7f8fe23ff700 1 -- 172.21.15.27:0/2733551242 >> 172.21.15.27:0/2733551242 conn(0x7f8fdc0680e0 msgr2=0x7f8fdc155150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.358+0000 7f8fe23ff700 1 -- 172.21.15.27:0/2733551242 shutdown_connections 2024-01-11T09:46:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.358+0000 7f8fe23ff700 1 -- 172.21.15.27:0/2733551242 wait complete. 2024-01-11T09:46:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 119 2024-01-11T09:46:11.367 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:46:11.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.575+0000 7f1882bff700 1 Processor -- start 2024-01-11T09:46:11.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.575+0000 7f1882bff700 1 -- start start 2024-01-11T09:46:11.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.577+0000 7f1882bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f187c1575e0 0x7f187c158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:11.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.577+0000 7f1882bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f187c06b610 0x7f187c0b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:11.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.577+0000 7f1882bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f187c158b60 con 0x7f187c0b1800 2024-01-11T09:46:11.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.577+0000 7f1882bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f187c0cb5e0 con 0x7f187c1575e0 2024-01-11T09:46:11.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.577+0000 7f1882bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f187c0b1c80 con 0x7f187c06b610 2024-01-11T09:46:11.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.577+0000 7f188119c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f187c0b1800 0x7f187c0ca1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37246/0 (socket says 172.21.15.27:37246) 2024-01-11T09:46:11.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.577+0000 7f188119c700 1 -- 172.21.15.27:0/1937152581 learned_addr learned my addr 172.21.15.27:0/1937152581 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:11.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.577+0000 7f188099b700 1 --2- 172.21.15.27:0/1937152581 >> v2:172.21.15.77:3300/0 conn(0x7f187c1575e0 0x7f187c158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:11.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.577+0000 7f187bfff700 1 --2- 172.21.15.27:0/1937152581 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f187c06b610 0x7f187c0b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:11.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f187b7fe700 1 -- 172.21.15.27:0/1937152581 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063214400 0 0) 0x7f1874001490 con 0x7f187c0b1800 2024-01-11T09:46:11.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f187b7fe700 1 -- 172.21.15.27:0/1937152581 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f186c003570 con 0x7f187c0b1800 2024-01-11T09:46:11.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f187b7fe700 1 -- 172.21.15.27:0/1937152581 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2997673641 0 0) 0x7f186c003570 con 0x7f187c0b1800 2024-01-11T09:46:11.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f187b7fe700 1 -- 172.21.15.27:0/1937152581 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f187c06b610 msgr2=0x7f187c0b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:11.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f187b7fe700 1 --2- 172.21.15.27:0/1937152581 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f187c06b610 0x7f187c0b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f187b7fe700 1 -- 172.21.15.27:0/1937152581 >> v2:172.21.15.77:3300/0 conn(0x7f187c1575e0 msgr2=0x7f187c158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:11.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f187b7fe700 1 --2- 172.21.15.27:0/1937152581 >> v2:172.21.15.77:3300/0 conn(0x7f187c1575e0 0x7f187c158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f187b7fe700 1 -- 172.21.15.27:0/1937152581 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187c1599b0 con 0x7f187c0b1800 2024-01-11T09:46:11.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f187b7fe700 1 -- 172.21.15.27:0/1937152581 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f1874001980 con 0x7f187c0b1800 2024-01-11T09:46:11.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f188099b700 1 --2- 172.21.15.27:0/1937152581 >> v2:172.21.15.77:3300/0 conn(0x7f187c1575e0 0x7f187c158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:11.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f187b7fe700 1 -- 172.21.15.27:0/1937152581 >> v1:172.21.15.27:6789/0 conn(0x7f187c0b1800 legacy=0x7f187c0ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:11.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f187b7fe700 1 --2- 172.21.15.27:0/1937152581 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f187c06b610 0x7f186c003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f187b7fe700 1 --2- 172.21.15.27:0/1937152581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f186c003fe0 0x7f186c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f187b7fe700 1 --2- 172.21.15.27:0/1937152581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f186c004d50 0x7f186c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f187b7fe700 1 -- 172.21.15.27:0/1937152581 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187c0cb5e0 con 0x7f186c003fe0 2024-01-11T09:46:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f187b7fe700 1 -- 172.21.15.27:0/1937152581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187c0b1c80 con 0x7f186c004d50 2024-01-11T09:46:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f187b7fe700 1 -- 172.21.15.27:0/1937152581 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f186c00ba90 con 0x7f187c06b610 2024-01-11T09:46:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f187bfff700 1 --2- 172.21.15.27:0/1937152581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f186c003fe0 0x7f186c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f188099b700 1 --2- 172.21.15.27:0/1937152581 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f187c06b610 0x7f186c003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.578+0000 7f188119c700 1 --2- 172.21.15.27:0/1937152581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f186c004d50 0x7f186c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.579+0000 7f187bfff700 1 -- 172.21.15.27:0/1937152581 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f187c06b610 msgr2=0x7f186c003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.579+0000 7f187bfff700 1 --2- 172.21.15.27:0/1937152581 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f187c06b610 0x7f186c003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.579+0000 7f187bfff700 1 -- 172.21.15.27:0/1937152581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f186c004d50 msgr2=0x7f186c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.579+0000 7f187bfff700 1 --2- 172.21.15.27:0/1937152581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f186c004d50 0x7f186c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.579+0000 7f187bfff700 1 -- 172.21.15.27:0/1937152581 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186c003570 con 0x7f186c003fe0 2024-01-11T09:46:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.579+0000 7f187bfff700 1 --2- 172.21.15.27:0/1937152581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f186c003fe0 0x7f186c004810 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f1868009c00 tx=0x7f18680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bcee7f6aa878f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.579+0000 7f187b7fe700 1 -- 172.21.15.27:0/1937152581 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186800e6b0 con 0x7f186c003fe0 2024-01-11T09:46:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.579+0000 7f187b7fe700 1 -- 172.21.15.27:0/1937152581 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186800e810 con 0x7f186c003fe0 2024-01-11T09:46:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.579+0000 7f187b7fe700 1 -- 172.21.15.27:0/1937152581 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186800ebd0 con 0x7f186c003fe0 2024-01-11T09:46:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.580+0000 7f1882bff700 1 -- 172.21.15.27:0/1937152581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f186c003fe0 msgr2=0x7f186c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.580+0000 7f1882bff700 1 --2- 172.21.15.27:0/1937152581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f186c003fe0 0x7f186c004810 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f1868009c00 tx=0x7f18680091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.580+0000 7f188119c700 1 -- 172.21.15.27:0/1937152581 reap_dead start 2024-01-11T09:46:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.580+0000 7f1882bff700 1 -- 172.21.15.27:0/1937152581 shutdown_connections 2024-01-11T09:46:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.580+0000 7f1882bff700 1 -- 172.21.15.27:0/1937152581 >> 172.21.15.27:0/1937152581 conn(0x7f187c0680e0 msgr2=0x7f187c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.580+0000 7f1882bff700 1 -- 172.21.15.27:0/1937152581 shutdown_connections 2024-01-11T09:46:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.580+0000 7f1882bff700 1 -- 172.21.15.27:0/1937152581 wait complete. 2024-01-11T09:46:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.581+0000 7f1882bff700 1 Processor -- start 2024-01-11T09:46:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.582+0000 7f1882bff700 1 -- start start 2024-01-11T09:46:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.582+0000 7f1882bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f186c003fe0 0x7f187c150c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.582+0000 7f1882bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f187c169810 0x7f187c0c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.582+0000 7f1882bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f187c0c5fb0 0x7f187c0afbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.583+0000 7f1882bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187c0af150 con 0x7f187c0c5fb0 2024-01-11T09:46:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.583+0000 7f1882bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187c0c68e0 con 0x7f187c169810 2024-01-11T09:46:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.583+0000 7f1882bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f187c0af3f0 con 0x7f186c003fe0 2024-01-11T09:46:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.583+0000 7f188119c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f187c0c5fb0 0x7f187c0afbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.583+0000 7f188119c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f187c0c5fb0 0x7f187c0afbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42866/0 (socket says 172.21.15.27:42866) 2024-01-11T09:46:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.583+0000 7f188119c700 1 -- 172.21.15.27:0/3856422491 learned_addr learned my addr 172.21.15.27:0/3856422491 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.583+0000 7f187bfff700 1 --2- 172.21.15.27:0/3856422491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f187c169810 0x7f187c0c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.583+0000 7f188099b700 1 --2- 172.21.15.27:0/3856422491 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f186c003fe0 0x7f187c150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.583+0000 7f188119c700 1 -- 172.21.15.27:0/3856422491 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f186c003fe0 msgr2=0x7f187c150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:11.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.583+0000 7f188119c700 1 --2- 172.21.15.27:0/3856422491 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f186c003fe0 0x7f187c150c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.583+0000 7f188119c700 1 -- 172.21.15.27:0/3856422491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f187c169810 msgr2=0x7f187c0c5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:11.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.583+0000 7f188119c700 1 --2- 172.21.15.27:0/3856422491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f187c169810 0x7f187c0c5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.583+0000 7f188119c700 1 -- 172.21.15.27:0/3856422491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187c0af6d0 con 0x7f187c0c5fb0 2024-01-11T09:46:11.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.583+0000 7f187bfff700 1 --2- 172.21.15.27:0/3856422491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f187c169810 0x7f187c0c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:11.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.584+0000 7f188119c700 1 --2- 172.21.15.27:0/3856422491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f187c0c5fb0 0x7f187c0afbf0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f1874000f80 tx=0x7f187400a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:11.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.584+0000 7f1878ff9700 1 -- 172.21.15.27:0/3856422491 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1874004e20 con 0x7f187c0c5fb0 2024-01-11T09:46:11.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.584+0000 7f1878ff9700 1 -- 172.21.15.27:0/3856422491 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1874007ac0 con 0x7f187c0c5fb0 2024-01-11T09:46:11.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.584+0000 7f1882bff700 1 -- 172.21.15.27:0/3856422491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f187c0c6bc0 con 0x7f187c0c5fb0 2024-01-11T09:46:11.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.584+0000 7f1878ff9700 1 -- 172.21.15.27:0/3856422491 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1874013640 con 0x7f187c0c5fb0 2024-01-11T09:46:11.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.584+0000 7f1882bff700 1 -- 172.21.15.27:0/3856422491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f187c151160 con 0x7f187c0c5fb0 2024-01-11T09:46:11.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.585+0000 7f1882bff700 1 -- 172.21.15.27:0/3856422491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f187c0718d0 con 0x7f187c0c5fb0 2024-01-11T09:46:11.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.586+0000 7f1878ff9700 1 -- 172.21.15.27:0/3856422491 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f1874013a30 con 0x7f187c0c5fb0 2024-01-11T09:46:11.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.587+0000 7f1878ff9700 1 --2- 172.21.15.27:0/3856422491 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1860067170 0x7f1860069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:11.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.587+0000 7f1878ff9700 1 -- 172.21.15.27:0/3856422491 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f18740853b0 con 0x7f187c0c5fb0 2024-01-11T09:46:11.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.588+0000 7f188099b700 1 --2- 172.21.15.27:0/3856422491 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1860067170 0x7f1860069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:11.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.588+0000 7f188099b700 1 --2- 172.21.15.27:0/3856422491 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1860067170 0x7f1860069620 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1870006f00 tx=0x7f18700091a0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:11.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.591+0000 7f1878ff9700 1 -- 172.21.15.27:0/3856422491 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1874051f70 con 0x7f187c0c5fb0 2024-01-11T09:46:11.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.838+0000 7f1882bff700 1 -- 172.21.15.27:0/3856422491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f187c158b60 con 0x7f187c0c5fb0 2024-01-11T09:46:11.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.839+0000 7f1878ff9700 1 -- 172.21.15.27:0/3856422491 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+4614 (secure 0 0 0) 0x7f187c158b60 con 0x7f187c0c5fb0 2024-01-11T09:46:11.830 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:11.830 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":119,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2103359888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":2103359888},{"type":"v1","addr":"172.21.15.77:6834","nonce":2103359888}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8795,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2650824649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2650824649},{"type":"v1","addr":"172.21.15.77:6835","nonce":2650824649}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106},{"gid":11157,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1211538286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1211538286},{"type":"v1","addr":"172.21.15.27:6835","nonce":1211538286}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:45:37.460120+0000","modified":"2024-01-11T09:45:40.977702+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10596},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10596":{"gid":10596,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6834/2807751836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2807751836},{"type":"v1","addr":"172.21.15.27:6834","nonce":2807751836}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":14}]} 2024-01-11T09:46:11.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.841+0000 7f1882bff700 1 -- 172.21.15.27:0/3856422491 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1860067170 msgr2=0x7f1860069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:11.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.841+0000 7f1882bff700 1 --2- 172.21.15.27:0/3856422491 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1860067170 0x7f1860069620 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1870006f00 tx=0x7f18700091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.841+0000 7f1882bff700 1 -- 172.21.15.27:0/3856422491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f187c0c5fb0 msgr2=0x7f187c0afbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:11.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.841+0000 7f1882bff700 1 --2- 172.21.15.27:0/3856422491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f187c0c5fb0 0x7f187c0afbf0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f1874000f80 tx=0x7f187400a1a0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.841+0000 7f1882bff700 1 -- 172.21.15.27:0/3856422491 shutdown_connections 2024-01-11T09:46:11.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.841+0000 7f1882bff700 1 --2- 172.21.15.27:0/3856422491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f187c0c5fb0 0x7f187c0afbf0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.841+0000 7f1882bff700 1 --2- 172.21.15.27:0/3856422491 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f186c003fe0 0x7f187c150c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.841+0000 7f1882bff700 1 --2- 172.21.15.27:0/3856422491 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1860067170 0x7f1860069620 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.841+0000 7f1882bff700 1 --2- 172.21.15.27:0/3856422491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f187c169810 0x7f187c0c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:11.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.841+0000 7f1882bff700 1 -- 172.21.15.27:0/3856422491 >> 172.21.15.27:0/3856422491 conn(0x7f187c0680e0 msgr2=0x7f187c15a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:11.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.841+0000 7f1882bff700 1 -- 172.21.15.27:0/3856422491 shutdown_connections 2024-01-11T09:46:11.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:11.841+0000 7f1882bff700 1 -- 172.21.15.27:0/3856422491 wait complete. 2024-01-11T09:46:11.835 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 119 2024-01-11T09:46:11.851 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:46:12.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.058+0000 7f1d17bff700 1 Processor -- start 2024-01-11T09:46:12.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.058+0000 7f1d17bff700 1 -- start start 2024-01-11T09:46:12.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.059+0000 7f1d17bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1d100b09e0 0x7f1d101610a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:12.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.059+0000 7f1d17bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d10157390 con 0x7f1d10069d40 2024-01-11T09:46:12.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.059+0000 7f1d17bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d1006a200 con 0x7f1d100c8620 2024-01-11T09:46:12.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.059+0000 7f1d17bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d100adcd0 con 0x7f1d100b09e0 2024-01-11T09:46:12.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.060+0000 7f1d1619c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1d100b09e0 0x7f1d101610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:12.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.060+0000 7f1d1519a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1d10069d40 0x7f1d100b02d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37248/0 (socket says 172.21.15.27:37248) 2024-01-11T09:46:12.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.060+0000 7f1d1519a700 1 -- 172.21.15.27:0/789848277 learned_addr learned my addr 172.21.15.27:0/789848277 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:12.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.060+0000 7f1d1619c700 1 -- 172.21.15.27:0/789848277 >> v1:172.21.15.77:6789/0 conn(0x7f1d100c8620 legacy=0x7f1d100af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:12.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.060+0000 7f1d1619c700 1 -- 172.21.15.27:0/789848277 >> v1:172.21.15.27:6789/0 conn(0x7f1d10069d40 legacy=0x7f1d100b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:12.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.060+0000 7f1d1619c700 1 -- 172.21.15.27:0/789848277 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d10156d40 con 0x7f1d100b09e0 2024-01-11T09:46:12.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.060+0000 7f1d14999700 1 -- 172.21.15.27:0/789848277 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353333828 0 0) 0x7f1d10157390 con 0x7f1d10069d40 2024-01-11T09:46:12.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.061+0000 7f1d1619c700 1 --2- 172.21.15.27:0/789848277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d100b09e0 0x7f1d101610a0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f1d08009e60 tx=0x7f1d08009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef94811fbf04397d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:12.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.061+0000 7f1d14999700 1 -- 172.21.15.27:0/789848277 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0800e070 con 0x7f1d100b09e0 2024-01-11T09:46:12.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.061+0000 7f1d14999700 1 -- 172.21.15.27:0/789848277 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0800bc30 con 0x7f1d100b09e0 2024-01-11T09:46:12.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.061+0000 7f1d14999700 1 -- 172.21.15.27:0/789848277 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d08012430 con 0x7f1d100b09e0 2024-01-11T09:46:12.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.061+0000 7f1d17bff700 1 -- 172.21.15.27:0/789848277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d100b09e0 msgr2=0x7f1d101610a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:12.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.061+0000 7f1d17bff700 1 --2- 172.21.15.27:0/789848277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d100b09e0 0x7f1d101610a0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f1d08009e60 tx=0x7f1d08009780 comp rx=0 tx=0).stop 2024-01-11T09:46:12.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.061+0000 7f1d17bff700 1 -- 172.21.15.27:0/789848277 shutdown_connections 2024-01-11T09:46:12.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.061+0000 7f1d17bff700 1 --2- 172.21.15.27:0/789848277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d100b09e0 0x7f1d101610a0 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:12.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.061+0000 7f1d17bff700 1 -- 172.21.15.27:0/789848277 >> 172.21.15.27:0/789848277 conn(0x7f1d100680e0 msgr2=0x7f1d10069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:12.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.061+0000 7f1d17bff700 1 -- 172.21.15.27:0/789848277 shutdown_connections 2024-01-11T09:46:12.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.062+0000 7f1d17bff700 1 -- 172.21.15.27:0/789848277 wait complete. 2024-01-11T09:46:12.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.063+0000 7f1d17bff700 1 Processor -- start 2024-01-11T09:46:12.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.063+0000 7f1d17bff700 1 -- start start 2024-01-11T09:46:12.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.064+0000 7f1d17bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1d10069d40 0x7f1d100c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:12.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.064+0000 7f1d17bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d100b09e0 0x7f1d1014f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:12.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.064+0000 7f1d17bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d100c8620 0x7f1d100c26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:12.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.064+0000 7f1d17bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d100c2be0 con 0x7f1d100b09e0 2024-01-11T09:46:12.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.064+0000 7f1d17bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d100c4e10 con 0x7f1d100c8620 2024-01-11T09:46:12.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.064+0000 7f1d17bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d100c3970 con 0x7f1d10069d40 2024-01-11T09:46:12.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.064+0000 7f1d1519a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d100b09e0 0x7f1d1014f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:12.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.064+0000 7f1d1519a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d100b09e0 0x7f1d1014f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42878/0 (socket says 172.21.15.27:42878) 2024-01-11T09:46:12.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.064+0000 7f1d1519a700 1 -- 172.21.15.27:0/679820764 learned_addr learned my addr 172.21.15.27:0/679820764 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:12.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.064+0000 7f1d1619c700 1 --2- 172.21.15.27:0/679820764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d100c8620 0x7f1d100c26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:12.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.064+0000 7f1d1599b700 1 --2- 172.21.15.27:0/679820764 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1d10069d40 0x7f1d100c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:12.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.065+0000 7f1d1519a700 1 -- 172.21.15.27:0/679820764 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1d10069d40 msgr2=0x7f1d100c9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:12.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.065+0000 7f1d1519a700 1 --2- 172.21.15.27:0/679820764 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1d10069d40 0x7f1d100c9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:12.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.065+0000 7f1d1519a700 1 -- 172.21.15.27:0/679820764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d100c8620 msgr2=0x7f1d100c26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:12.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.065+0000 7f1d1519a700 1 --2- 172.21.15.27:0/679820764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d100c8620 0x7f1d100c26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:12.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.065+0000 7f1d1519a700 1 -- 172.21.15.27:0/679820764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d08009a20 con 0x7f1d100b09e0 2024-01-11T09:46:12.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.065+0000 7f1d1519a700 1 --2- 172.21.15.27:0/679820764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d100b09e0 0x7f1d1014f870 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000c7e0 tx=0x7f1d00007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:12.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.065+0000 7f1d05ffb700 1 -- 172.21.15.27:0/679820764 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d00005b30 con 0x7f1d100b09e0 2024-01-11T09:46:12.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.065+0000 7f1d05ffb700 1 -- 172.21.15.27:0/679820764 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d00005c90 con 0x7f1d100b09e0 2024-01-11T09:46:12.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.065+0000 7f1d17bff700 1 -- 172.21.15.27:0/679820764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d100c2df0 con 0x7f1d100b09e0 2024-01-11T09:46:12.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.065+0000 7f1d05ffb700 1 -- 172.21.15.27:0/679820764 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d00007300 con 0x7f1d100b09e0 2024-01-11T09:46:12.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.065+0000 7f1d17bff700 1 -- 172.21.15.27:0/679820764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d100c33b0 con 0x7f1d100b09e0 2024-01-11T09:46:12.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.066+0000 7f1d17bff700 1 -- 172.21.15.27:0/679820764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d100c81a0 con 0x7f1d100b09e0 2024-01-11T09:46:12.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.067+0000 7f1d05ffb700 1 -- 172.21.15.27:0/679820764 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f1d0001b400 con 0x7f1d100b09e0 2024-01-11T09:46:12.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.068+0000 7f1d05ffb700 1 --2- 172.21.15.27:0/679820764 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1cf4067170 0x7f1cf4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:12.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.068+0000 7f1d05ffb700 1 -- 172.21.15.27:0/679820764 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f1d00084840 con 0x7f1d100b09e0 2024-01-11T09:46:12.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.069+0000 7f1d1599b700 1 --2- 172.21.15.27:0/679820764 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1cf4067170 0x7f1cf4069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:12.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.069+0000 7f1d1599b700 1 --2- 172.21.15.27:0/679820764 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1cf4067170 0x7f1cf4069620 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c0037c0 tx=0x7f1d0c009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:12.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.072+0000 7f1d05ffb700 1 -- 172.21.15.27:0/679820764 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1d00051400 con 0x7f1d100b09e0 2024-01-11T09:46:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.282+0000 7f1d17bff700 1 -- 172.21.15.27:0/679820764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d100718d0 con 0x7f1d100b09e0 2024-01-11T09:46:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.283+0000 7f1d05ffb700 1 -- 172.21.15.27:0/679820764 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v223) v1 ==== 74+0+16559 (secure 0 0 0) 0x7f1d00054a20 con 0x7f1d100b09e0 2024-01-11T09:46:12.275 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:12.275 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":223,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:46:08.139060+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-01-11T09:45:35.631727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-01-11T09:45:36.452195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/1143960902":"2024-01-12T09:46:04.482215+0000","172.21.15.27:6833/1143960902":"2024-01-12T09:46:04.482215+0000","172.21.15.77:0/3729201132":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/3605354921":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/4084841078":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/3487780416":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/534676732":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/3735282980":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/1375879589":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/2192171652":"2024-01-12T09:45:52.192235+0000","172.21.15.77:6837/59776":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/2635904665":"2024-01-12T09:46:04.127727+0000","172.21.15.77:6836/59776":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/2654175097":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/652362578":"2024-01-12T09:45:52.192235+0000","172.21.15.77:0/749658672":"2024-01-12T09:46:04.127727+0000","172.21.15.77:0/2439942664":"2024-01-12T09:45:52.192235+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:46:12.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.285+0000 7f1d17bff700 1 -- 172.21.15.27:0/679820764 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1cf4067170 msgr2=0x7f1cf4069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.285+0000 7f1d17bff700 1 --2- 172.21.15.27:0/679820764 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1cf4067170 0x7f1cf4069620 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c0037c0 tx=0x7f1d0c009040 comp rx=0 tx=0).stop 2024-01-11T09:46:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.285+0000 7f1d17bff700 1 -- 172.21.15.27:0/679820764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d100b09e0 msgr2=0x7f1d1014f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.285+0000 7f1d17bff700 1 --2- 172.21.15.27:0/679820764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d100b09e0 0x7f1d1014f870 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000c7e0 tx=0x7f1d00007df0 comp rx=0 tx=0).stop 2024-01-11T09:46:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.286+0000 7f1d17bff700 1 -- 172.21.15.27:0/679820764 shutdown_connections 2024-01-11T09:46:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.286+0000 7f1d17bff700 1 --2- 172.21.15.27:0/679820764 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1cf4067170 0x7f1cf4069620 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.286+0000 7f1d17bff700 1 --2- 172.21.15.27:0/679820764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d100c8620 0x7f1d100c26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.286+0000 7f1d17bff700 1 --2- 172.21.15.27:0/679820764 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1d10069d40 0x7f1d100c9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.286+0000 7f1d17bff700 1 --2- 172.21.15.27:0/679820764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d100b09e0 0x7f1d1014f870 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.286+0000 7f1d17bff700 1 -- 172.21.15.27:0/679820764 >> 172.21.15.27:0/679820764 conn(0x7f1d100680e0 msgr2=0x7f1d100cca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.286+0000 7f1d17bff700 1 -- 172.21.15.27:0/679820764 shutdown_connections 2024-01-11T09:46:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.286+0000 7f1d17bff700 1 -- 172.21.15.27:0/679820764 wait complete. 2024-01-11T09:46:12.296 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2024-01-11T09:46:12.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.507+0000 7f0e387ff700 1 Processor -- start 2024-01-11T09:46:12.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.507+0000 7f0e387ff700 1 -- start start 2024-01-11T09:46:12.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.508+0000 7f0e387ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0e3006bf10 0x7f0e30159c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:12.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.508+0000 7f0e387ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e30068720 0x7f0e30068b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:12.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.508+0000 7f0e387ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e3015d290 con 0x7f0e300b0a40 2024-01-11T09:46:12.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.508+0000 7f0e387ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e30069080 con 0x7f0e3006bf10 2024-01-11T09:46:12.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.509+0000 7f0e387ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3015d570 con 0x7f0e30068720 2024-01-11T09:46:12.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.509+0000 7f0e36d9c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e30068720 0x7f0e30068b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:12.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.509+0000 7f0e3659b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0e3006bf10 0x7f0e30159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:12.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.509+0000 7f0e36d9c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e30068720 0x7f0e30068b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:54886/0 (socket says 172.21.15.27:54886) 2024-01-11T09:46:12.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.509+0000 7f0e36d9c700 1 -- 172.21.15.27:0/985465562 learned_addr learned my addr 172.21.15.27:0/985465562 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:12.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.509+0000 7f0e35d9a700 1 --1- 172.21.15.27:0/985465562 >> v1:172.21.15.77:6789/0 conn(0x7f0e300b0a40 0x7f0e3015da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:59060/0 (socket says 172.21.15.27:59060) 2024-01-11T09:46:12.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.510+0000 7f0e36d9c700 1 -- 172.21.15.27:0/985465562 >> v2:172.21.15.77:3300/0 conn(0x7f0e3006bf10 msgr2=0x7f0e30159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:12.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.510+0000 7f0e36d9c700 1 --2- 172.21.15.27:0/985465562 >> v2:172.21.15.77:3300/0 conn(0x7f0e3006bf10 0x7f0e30159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:12.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.510+0000 7f0e36d9c700 1 -- 172.21.15.27:0/985465562 >> v1:172.21.15.77:6789/0 conn(0x7f0e300b0a40 legacy=0x7f0e3015da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:12.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.509+0000 7f0e35599700 1 -- 172.21.15.27:0/985465562 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220767305 0 0) 0x7f0e3015d290 con 0x7f0e300b0a40 2024-01-11T09:46:12.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.510+0000 7f0e36d9c700 1 -- 172.21.15.27:0/985465562 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e30154180 con 0x7f0e30068720 2024-01-11T09:46:12.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.510+0000 7f0e36d9c700 1 --2- 172.21.15.27:0/985465562 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e30068720 0x7f0e30068b40 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009f20 tx=0x7f0e28009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=696b6d5a7204a00a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:12.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.510+0000 7f0e35599700 1 -- 172.21.15.27:0/985465562 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2800e070 con 0x7f0e30068720 2024-01-11T09:46:12.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.510+0000 7f0e35599700 1 -- 172.21.15.27:0/985465562 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2800b9d0 con 0x7f0e30068720 2024-01-11T09:46:12.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.510+0000 7f0e35599700 1 -- 172.21.15.27:0/985465562 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e280092e0 con 0x7f0e30068720 2024-01-11T09:46:12.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.511+0000 7f0e387ff700 1 -- 172.21.15.27:0/985465562 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e30068720 msgr2=0x7f0e30068b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:12.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.511+0000 7f0e387ff700 1 --2- 172.21.15.27:0/985465562 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e30068720 0x7f0e30068b40 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009f20 tx=0x7f0e28009840 comp rx=0 tx=0).stop 2024-01-11T09:46:12.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.511+0000 7f0e387ff700 1 -- 172.21.15.27:0/985465562 shutdown_connections 2024-01-11T09:46:12.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.511+0000 7f0e387ff700 1 --2- 172.21.15.27:0/985465562 >> v2:172.21.15.77:3300/0 conn(0x7f0e3006bf10 0x7f0e30159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:12.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.511+0000 7f0e387ff700 1 --2- 172.21.15.27:0/985465562 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e30068720 0x7f0e30068b40 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:12.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.511+0000 7f0e387ff700 1 -- 172.21.15.27:0/985465562 >> 172.21.15.27:0/985465562 conn(0x7f0e300ce3c0 msgr2=0x7f0e300ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:12.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.511+0000 7f0e387ff700 1 -- 172.21.15.27:0/985465562 shutdown_connections 2024-01-11T09:46:12.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.511+0000 7f0e387ff700 1 -- 172.21.15.27:0/985465562 wait complete. 2024-01-11T09:46:12.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.512+0000 7f0e387ff700 1 Processor -- start 2024-01-11T09:46:12.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.513+0000 7f0e387ff700 1 -- start start 2024-01-11T09:46:12.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.513+0000 7f0e387ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e30068720 0x7f0e30154340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:12.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.513+0000 7f0e387ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e3006bf10 0x7f0e3016cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:12.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.513+0000 7f0e387ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e300b0a40 0x7f0e3016d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:12.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.513+0000 7f0e387ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e30157300 con 0x7f0e3006bf10 2024-01-11T09:46:12.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.513+0000 7f0e387ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3016a580 con 0x7f0e300b0a40 2024-01-11T09:46:12.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.513+0000 7f0e387ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e300c5d20 con 0x7f0e30068720 2024-01-11T09:46:12.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.514+0000 7f0e35d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e3006bf10 0x7f0e3016cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:12.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.514+0000 7f0e3659b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e30068720 0x7f0e30154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:12.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.514+0000 7f0e35d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e3006bf10 0x7f0e3016cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42884/0 (socket says 172.21.15.27:42884) 2024-01-11T09:46:12.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.514+0000 7f0e35d9a700 1 -- 172.21.15.27:0/1999449602 learned_addr learned my addr 172.21.15.27:0/1999449602 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:12.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.514+0000 7f0e36d9c700 1 --2- 172.21.15.27:0/1999449602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e300b0a40 0x7f0e3016d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:12.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.514+0000 7f0e3659b700 1 -- 172.21.15.27:0/1999449602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e300b0a40 msgr2=0x7f0e3016d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:12.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.514+0000 7f0e3659b700 1 --2- 172.21.15.27:0/1999449602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e300b0a40 0x7f0e3016d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:12.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.514+0000 7f0e3659b700 1 -- 172.21.15.27:0/1999449602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e3006bf10 msgr2=0x7f0e3016cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:12.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.514+0000 7f0e3659b700 1 --2- 172.21.15.27:0/1999449602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e3006bf10 0x7f0e3016cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:12.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.514+0000 7f0e3659b700 1 -- 172.21.15.27:0/1999449602 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e28009ae0 con 0x7f0e30068720 2024-01-11T09:46:12.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.514+0000 7f0e35d9a700 1 --2- 172.21.15.27:0/1999449602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e3006bf10 0x7f0e3016cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:12.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.515+0000 7f0e3659b700 1 --2- 172.21.15.27:0/1999449602 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e30068720 0x7f0e30154340 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00e900 tx=0x7f0e2c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:12.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.515+0000 7f0e267fc700 1 -- 172.21.15.27:0/1999449602 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c013680 con 0x7f0e30068720 2024-01-11T09:46:12.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.515+0000 7f0e267fc700 1 -- 172.21.15.27:0/1999449602 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2c0137e0 con 0x7f0e30068720 2024-01-11T09:46:12.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.515+0000 7f0e387ff700 1 -- 172.21.15.27:0/1999449602 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e30154880 con 0x7f0e30068720 2024-01-11T09:46:12.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.515+0000 7f0e267fc700 1 -- 172.21.15.27:0/1999449602 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c013b80 con 0x7f0e30068720 2024-01-11T09:46:12.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.515+0000 7f0e387ff700 1 -- 172.21.15.27:0/1999449602 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e30157020 con 0x7f0e30068720 2024-01-11T09:46:12.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.516+0000 7f0e387ff700 1 -- 172.21.15.27:0/1999449602 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e3004c3c0 con 0x7f0e30068720 2024-01-11T09:46:12.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.517+0000 7f0e267fc700 1 -- 172.21.15.27:0/1999449602 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f0e2c0123d0 con 0x7f0e30068720 2024-01-11T09:46:12.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.518+0000 7f0e267fc700 1 --2- 172.21.15.27:0/1999449602 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0e14067170 0x7f0e14069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:12.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.518+0000 7f0e267fc700 1 -- 172.21.15.27:0/1999449602 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f0e2c084c50 con 0x7f0e30068720 2024-01-11T09:46:12.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.518+0000 7f0e35d9a700 1 --2- 172.21.15.27:0/1999449602 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0e14067170 0x7f0e14069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:12.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.519+0000 7f0e35d9a700 1 --2- 172.21.15.27:0/1999449602 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0e14067170 0x7f0e14069620 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0e300b1220 tx=0x7f0e20007e10 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:12.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.523+0000 7f0e267fc700 1 -- 172.21.15.27:0/1999449602 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0e2c051860 con 0x7f0e30068720 2024-01-11T09:46:12.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.776+0000 7f0e387ff700 1 -- 172.21.15.27:0/1999449602 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7f0e3015a510 con 0x7f0e30068720 2024-01-11T09:46:12.769 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-11T09:46:12.778+0000 7f27065f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-11T09:46:12.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.786+0000 7f0e267fc700 1 -- 172.21.15.27:0/1999449602 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v120) v1 ==== 167+0+0 (secure 0 0 0) 0x7f0e2c017090 con 0x7f0e30068720 2024-01-11T09:46:12.779 INFO:tasks.ceph.mds.c.smithi027.stderr: -1830> 2024-01-11T09:45:40.108+0000 7f2bc3ff1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T09:46:12.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.788+0000 7f0e387ff700 1 -- 172.21.15.27:0/1999449602 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0e14067170 msgr2=0x7f0e14069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:12.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.788+0000 7f0e387ff700 1 --2- 172.21.15.27:0/1999449602 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0e14067170 0x7f0e14069620 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0e300b1220 tx=0x7f0e20007e10 comp rx=0 tx=0).stop 2024-01-11T09:46:12.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.788+0000 7f0e387ff700 1 -- 172.21.15.27:0/1999449602 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e30068720 msgr2=0x7f0e30154340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:12.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.788+0000 7f0e387ff700 1 --2- 172.21.15.27:0/1999449602 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e30068720 0x7f0e30154340 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00e900 tx=0x7f0e2c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.788+0000 7f0e387ff700 1 -- 172.21.15.27:0/1999449602 shutdown_connections 2024-01-11T09:46:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.789+0000 7f0e387ff700 1 --2- 172.21.15.27:0/1999449602 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0e14067170 0x7f0e14069620 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.789+0000 7f0e387ff700 1 --2- 172.21.15.27:0/1999449602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e300b0a40 0x7f0e3016d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.789+0000 7f0e387ff700 1 --2- 172.21.15.27:0/1999449602 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0e30068720 0x7f0e30154340 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.789+0000 7f0e387ff700 1 --2- 172.21.15.27:0/1999449602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e3006bf10 0x7f0e3016cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.789+0000 7f0e387ff700 1 -- 172.21.15.27:0/1999449602 >> 172.21.15.27:0/1999449602 conn(0x7f0e300ce3c0 msgr2=0x7f0e300ce790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.789+0000 7f0e387ff700 1 -- 172.21.15.27:0/1999449602 shutdown_connections 2024-01-11T09:46:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:12.789+0000 7f0e387ff700 1 -- 172.21.15.27:0/1999449602 wait complete. 2024-01-11T09:46:12.781 INFO:teuthology.orchestra.run.smithi027.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-11T09:46:12.804 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2024-01-11T09:46:12.826 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T09:46:13.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.017+0000 7fe973dff700 1 Processor -- start 2024-01-11T09:46:13.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.017+0000 7fe973dff700 1 -- start start 2024-01-11T09:46:13.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.018+0000 7fe973dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe96c067860 0x7fe96c06c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:13.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.019+0000 7fe973dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe96c06c3c0 con 0x7fe96c06cd90 2024-01-11T09:46:13.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.019+0000 7fe973dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe96c0cb9c0 con 0x7fe96c067c30 2024-01-11T09:46:13.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.019+0000 7fe973dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe96c06f650 con 0x7fe96c067860 2024-01-11T09:46:13.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.019+0000 7fe971b9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe96c067860 0x7fe96c06c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:13.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.019+0000 7fe97139a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe96c06cd90 0x7fe96c0cae90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37266/0 (socket says 172.21.15.27:37266) 2024-01-11T09:46:13.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.019+0000 7fe97139a700 1 -- 172.21.15.27:0/2036551134 learned_addr learned my addr 172.21.15.27:0/2036551134 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:13.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.019+0000 7fe970b99700 1 -- 172.21.15.27:0/2036551134 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023232156 0 0) 0x7fe96c06c3c0 con 0x7fe96c06cd90 2024-01-11T09:46:13.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.019+0000 7fe970b99700 1 -- 172.21.15.27:0/2036551134 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe958003570 con 0x7fe96c06cd90 2024-01-11T09:46:13.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.019+0000 7fe971b9b700 1 -- 172.21.15.27:0/2036551134 >> v1:172.21.15.77:6789/0 conn(0x7fe96c067c30 legacy=0x7fe96c0c7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:13.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.019+0000 7fe971b9b700 1 -- 172.21.15.27:0/2036551134 >> v1:172.21.15.27:6789/0 conn(0x7fe96c06cd90 legacy=0x7fe96c0cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:13.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.019+0000 7fe971b9b700 1 -- 172.21.15.27:0/2036551134 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe96c06ba90 con 0x7fe96c067860 2024-01-11T09:46:13.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.020+0000 7fe971b9b700 1 --2- 172.21.15.27:0/2036551134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe96c067860 0x7fe96c06c850 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7fe9680098e0 tx=0x7fe9680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b72bd48e607d261 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:13.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.020+0000 7fe970b99700 1 -- 172.21.15.27:0/2036551134 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe96800b640 con 0x7fe96c067860 2024-01-11T09:46:13.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.020+0000 7fe970b99700 1 -- 172.21.15.27:0/2036551134 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe96800b7a0 con 0x7fe96c067860 2024-01-11T09:46:13.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.020+0000 7fe970b99700 1 -- 172.21.15.27:0/2036551134 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe96800bb40 con 0x7fe96c067860 2024-01-11T09:46:13.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.020+0000 7fe973dff700 1 -- 172.21.15.27:0/2036551134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe96c067860 msgr2=0x7fe96c06c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:13.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.020+0000 7fe973dff700 1 --2- 172.21.15.27:0/2036551134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe96c067860 0x7fe96c06c850 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7fe9680098e0 tx=0x7fe9680091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:13.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.021+0000 7fe973dff700 1 -- 172.21.15.27:0/2036551134 shutdown_connections 2024-01-11T09:46:13.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.021+0000 7fe973dff700 1 --2- 172.21.15.27:0/2036551134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe96c067860 0x7fe96c06c850 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:13.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.021+0000 7fe973dff700 1 -- 172.21.15.27:0/2036551134 >> 172.21.15.27:0/2036551134 conn(0x7fe96c06b1f0 msgr2=0x7fe96c0cd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:13.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.021+0000 7fe973dff700 1 -- 172.21.15.27:0/2036551134 shutdown_connections 2024-01-11T09:46:13.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.021+0000 7fe973dff700 1 -- 172.21.15.27:0/2036551134 wait complete. 2024-01-11T09:46:13.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.022+0000 7fe973dff700 1 Processor -- start 2024-01-11T09:46:13.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.022+0000 7fe973dff700 1 -- start start 2024-01-11T09:46:13.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.023+0000 7fe973dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe96c067860 0x7fe96c0b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:13.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.023+0000 7fe973dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe96c067c30 0x7fe96c0ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:13.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.023+0000 7fe973dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe96c06cd90 0x7fe96c0c30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:13.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.023+0000 7fe973dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96c159580 con 0x7fe96c067860 2024-01-11T09:46:13.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.023+0000 7fe973dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96c0b08d0 con 0x7fe96c067c30 2024-01-11T09:46:13.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.023+0000 7fe973dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe96c0c2c10 con 0x7fe96c06cd90 2024-01-11T09:46:13.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.024+0000 7fe971b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe96c067860 0x7fe96c0b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:13.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.024+0000 7fe971b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe96c067860 0x7fe96c0b25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42914/0 (socket says 172.21.15.27:42914) 2024-01-11T09:46:13.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.024+0000 7fe971b9b700 1 -- 172.21.15.27:0/3185777044 learned_addr learned my addr 172.21.15.27:0/3185777044 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:13.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.024+0000 7fe97139a700 1 --2- 172.21.15.27:0/3185777044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe96c067c30 0x7fe96c0ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:13.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.024+0000 7fe97239c700 1 --2- 172.21.15.27:0/3185777044 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe96c06cd90 0x7fe96c0c30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:13.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.024+0000 7fe971b9b700 1 -- 172.21.15.27:0/3185777044 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe96c06cd90 msgr2=0x7fe96c0c30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:13.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.024+0000 7fe971b9b700 1 --2- 172.21.15.27:0/3185777044 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe96c06cd90 0x7fe96c0c30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:13.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.024+0000 7fe971b9b700 1 -- 172.21.15.27:0/3185777044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe96c067c30 msgr2=0x7fe96c0ad6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:13.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.024+0000 7fe971b9b700 1 --2- 172.21.15.27:0/3185777044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe96c067c30 0x7fe96c0ad6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:13.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.024+0000 7fe971b9b700 1 -- 172.21.15.27:0/3185777044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe968009420 con 0x7fe96c067860 2024-01-11T09:46:13.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.024+0000 7fe971b9b700 1 --2- 172.21.15.27:0/3185777044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe96c067860 0x7fe96c0b25d0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7fe968009f80 tx=0x7fe968017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:13.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.025+0000 7fe961ffb700 1 -- 172.21.15.27:0/3185777044 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe96800bb20 con 0x7fe96c067860 2024-01-11T09:46:13.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.025+0000 7fe961ffb700 1 -- 172.21.15.27:0/3185777044 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe968011420 con 0x7fe96c067860 2024-01-11T09:46:13.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.025+0000 7fe961ffb700 1 -- 172.21.15.27:0/3185777044 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9680117c0 con 0x7fe96c067860 2024-01-11T09:46:13.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.025+0000 7fe973dff700 1 -- 172.21.15.27:0/3185777044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96c0b0b20 con 0x7fe96c067860 2024-01-11T09:46:13.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.025+0000 7fe973dff700 1 -- 172.21.15.27:0/3185777044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe96c0b10e0 con 0x7fe96c067860 2024-01-11T09:46:13.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.026+0000 7fe973dff700 1 -- 172.21.15.27:0/3185777044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe96c0c2ef0 con 0x7fe96c067860 2024-01-11T09:46:13.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.027+0000 7fe961ffb700 1 -- 172.21.15.27:0/3185777044 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fe968011b30 con 0x7fe96c067860 2024-01-11T09:46:13.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.027+0000 7fe961ffb700 1 --2- 172.21.15.27:0/3185777044 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe950067170 0x7fe950069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:13.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.028+0000 7fe961ffb700 1 -- 172.21.15.27:0/3185777044 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 6646+0+0 (secure 0 0 0) 0x7fe9680863b0 con 0x7fe96c067860 2024-01-11T09:46:13.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.028+0000 7fe97139a700 1 --2- 172.21.15.27:0/3185777044 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe950067170 0x7fe950069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:13.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.029+0000 7fe97139a700 1 --2- 172.21.15.27:0/3185777044 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe950067170 0x7fe950069620 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe96c0acd90 tx=0x7fe95c009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:13.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.032+0000 7fe961ffb700 1 -- 172.21.15.27:0/3185777044 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe968052f40 con 0x7fe96c067860 2024-01-11T09:46:13.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.283+0000 7fe973dff700 1 -- 172.21.15.27:0/3185777044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe96c04c3c0 con 0x7fe96c067860 2024-01-11T09:46:13.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.783+0000 7fe961ffb700 1 -- 172.21.15.27:0/3185777044 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v121) v1 ==== 107+0+0 (secure 0 0 0) 0x7fe968056560 con 0x7fe96c067860 2024-01-11T09:46:13.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.786+0000 7fe973dff700 1 -- 172.21.15.27:0/3185777044 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe950067170 msgr2=0x7fe950069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.786+0000 7fe973dff700 1 --2- 172.21.15.27:0/3185777044 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe950067170 0x7fe950069620 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe96c0acd90 tx=0x7fe95c009040 comp rx=0 tx=0).stop 2024-01-11T09:46:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.786+0000 7fe973dff700 1 -- 172.21.15.27:0/3185777044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe96c067860 msgr2=0x7fe96c0b25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.786+0000 7fe973dff700 1 --2- 172.21.15.27:0/3185777044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe96c067860 0x7fe96c0b25d0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7fe968009f80 tx=0x7fe968017000 comp rx=0 tx=0).stop 2024-01-11T09:46:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.786+0000 7fe973dff700 1 -- 172.21.15.27:0/3185777044 shutdown_connections 2024-01-11T09:46:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.786+0000 7fe973dff700 1 --2- 172.21.15.27:0/3185777044 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe96c06cd90 0x7fe96c0c30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.786+0000 7fe973dff700 1 --2- 172.21.15.27:0/3185777044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe96c067860 0x7fe96c0b25d0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.786+0000 7fe973dff700 1 --2- 172.21.15.27:0/3185777044 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe950067170 0x7fe950069620 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.786+0000 7fe973dff700 1 --2- 172.21.15.27:0/3185777044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe96c067c30 0x7fe96c0ad6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.786+0000 7fe973dff700 1 -- 172.21.15.27:0/3185777044 >> 172.21.15.27:0/3185777044 conn(0x7fe96c06b1f0 msgr2=0x7fe96c141a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.786+0000 7fe973dff700 1 -- 172.21.15.27:0/3185777044 shutdown_connections 2024-01-11T09:46:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:13.786+0000 7fe973dff700 1 -- 172.21.15.27:0/3185777044 wait complete. 2024-01-11T09:46:13.796 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2024-01-11T09:46:14.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.050+0000 7fdcbafff700 1 Processor -- start 2024-01-11T09:46:14.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.051+0000 7fdcbafff700 1 -- start start 2024-01-11T09:46:14.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.052+0000 7fdcbafff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdcb4156440 0x7fdcb40c7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:14.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.052+0000 7fdcbafff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdcb406ee30 0x7fdcb40accb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:14.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.052+0000 7fdcbafff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb40c8590 con 0x7fdcb4156810 2024-01-11T09:46:14.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.052+0000 7fdcbafff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb40c7980 con 0x7fdcb406ee30 2024-01-11T09:46:14.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.052+0000 7fdcbafff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb40c7c60 con 0x7fdcb4156440 2024-01-11T09:46:14.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.052+0000 7fdcb959c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdcb406ee30 0x7fdcb40accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:14.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.052+0000 7fdcb959c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdcb406ee30 0x7fdcb40accb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60312/0 (socket says 172.21.15.27:60312) 2024-01-11T09:46:14.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.052+0000 7fdcb959c700 1 -- 172.21.15.27:0/3725949672 learned_addr learned my addr 172.21.15.27:0/3725949672 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:14.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.052+0000 7fdcb8d9b700 1 --2- 172.21.15.27:0/3725949672 >> v2:172.21.15.77:3300/0 conn(0x7fdcb4156440 0x7fdcb40c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:14.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.053+0000 7fdcb959c700 1 -- 172.21.15.27:0/3725949672 >> v2:172.21.15.77:3300/0 conn(0x7fdcb4156440 msgr2=0x7fdcb40c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:14.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.053+0000 7fdcb959c700 1 --2- 172.21.15.27:0/3725949672 >> v2:172.21.15.77:3300/0 conn(0x7fdcb4156440 0x7fdcb40c7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:14.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.053+0000 7fdcb959c700 1 -- 172.21.15.27:0/3725949672 >> v1:172.21.15.77:6789/0 conn(0x7fdcb4156810 legacy=0x7fdcb406e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:14.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.053+0000 7fdcb959c700 1 -- 172.21.15.27:0/3725949672 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb40c7f40 con 0x7fdcb406ee30 2024-01-11T09:46:14.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.053+0000 7fdcb959c700 1 --2- 172.21.15.27:0/3725949672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcb406ee30 0x7fdcb40accb0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7fdcb40bfbd0 tx=0x7fdcac009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f58162d22d220a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:14.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.053+0000 7fdcb37fe700 1 -- 172.21.15.27:0/3725949672 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac00e070 con 0x7fdcb406ee30 2024-01-11T09:46:14.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.053+0000 7fdcb37fe700 1 -- 172.21.15.27:0/3725949672 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcac00ba20 con 0x7fdcb406ee30 2024-01-11T09:46:14.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.053+0000 7fdcb37fe700 1 -- 172.21.15.27:0/3725949672 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac0092e0 con 0x7fdcb406ee30 2024-01-11T09:46:14.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.054+0000 7fdcbafff700 1 -- 172.21.15.27:0/3725949672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcb406ee30 msgr2=0x7fdcb40accb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:14.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.054+0000 7fdcbafff700 1 --2- 172.21.15.27:0/3725949672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcb406ee30 0x7fdcb40accb0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7fdcb40bfbd0 tx=0x7fdcac009840 comp rx=0 tx=0).stop 2024-01-11T09:46:14.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.054+0000 7fdcbafff700 1 -- 172.21.15.27:0/3725949672 shutdown_connections 2024-01-11T09:46:14.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.054+0000 7fdcbafff700 1 --2- 172.21.15.27:0/3725949672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcb406ee30 0x7fdcb40accb0 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:14.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.054+0000 7fdcbafff700 1 --2- 172.21.15.27:0/3725949672 >> v2:172.21.15.77:3300/0 conn(0x7fdcb4156440 0x7fdcb40c7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:14.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.054+0000 7fdcbafff700 1 -- 172.21.15.27:0/3725949672 >> 172.21.15.27:0/3725949672 conn(0x7fdcb40680e0 msgr2=0x7fdcb40cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:14.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.054+0000 7fdcbafff700 1 -- 172.21.15.27:0/3725949672 shutdown_connections 2024-01-11T09:46:14.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.054+0000 7fdcbafff700 1 -- 172.21.15.27:0/3725949672 wait complete. 2024-01-11T09:46:14.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.055+0000 7fdcbafff700 1 Processor -- start 2024-01-11T09:46:14.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.056+0000 7fdcbafff700 1 -- start start 2024-01-11T09:46:14.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.056+0000 7fdcbafff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdcb406ee30 0x7fdcb4153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:14.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.056+0000 7fdcbafff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcb4156440 0x7fdcb41575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:14.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.056+0000 7fdcbafff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcb4156810 0x7fdcb4157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:14.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.057+0000 7fdcbafff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb40c29c0 con 0x7fdcb4156810 2024-01-11T09:46:14.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.057+0000 7fdcbafff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb4153570 con 0x7fdcb4156440 2024-01-11T09:46:14.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.057+0000 7fdcbafff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb40bfc00 con 0x7fdcb406ee30 2024-01-11T09:46:14.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.057+0000 7fdcb959c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcb4156810 0x7fdcb4157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:14.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.057+0000 7fdcb8d9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdcb406ee30 0x7fdcb4153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:14.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.057+0000 7fdcb8d9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdcb406ee30 0x7fdcb4153030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:40090/0 (socket says 172.21.15.27:40090) 2024-01-11T09:46:14.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.057+0000 7fdcb8d9b700 1 -- 172.21.15.27:0/1847708761 learned_addr learned my addr 172.21.15.27:0/1847708761 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:14.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.057+0000 7fdcb3fff700 1 --2- 172.21.15.27:0/1847708761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcb4156440 0x7fdcb41575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:14.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.057+0000 7fdcb959c700 1 -- 172.21.15.27:0/1847708761 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdcb406ee30 msgr2=0x7fdcb4153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:14.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.057+0000 7fdcb959c700 1 --2- 172.21.15.27:0/1847708761 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdcb406ee30 0x7fdcb4153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:14.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.057+0000 7fdcb959c700 1 -- 172.21.15.27:0/1847708761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcb4156440 msgr2=0x7fdcb41575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:14.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.057+0000 7fdcb959c700 1 --2- 172.21.15.27:0/1847708761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcb4156440 0x7fdcb41575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:14.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.057+0000 7fdcb959c700 1 -- 172.21.15.27:0/1847708761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcac009ae0 con 0x7fdcb4156810 2024-01-11T09:46:14.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.057+0000 7fdcb8d9b700 1 --2- 172.21.15.27:0/1847708761 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdcb406ee30 0x7fdcb4153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:14.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.058+0000 7fdcb959c700 1 --2- 172.21.15.27:0/1847708761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcb4156810 0x7fdcb4157e90 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7fdcac015000 tx=0x7fdcac018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:14.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.058+0000 7fdcb0ff9700 1 -- 172.21.15.27:0/1847708761 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac008a80 con 0x7fdcb4156810 2024-01-11T09:46:14.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.058+0000 7fdcb0ff9700 1 -- 172.21.15.27:0/1847708761 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcac008c20 con 0x7fdcb4156810 2024-01-11T09:46:14.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.058+0000 7fdcbafff700 1 -- 172.21.15.27:0/1847708761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb40bfee0 con 0x7fdcb4156810 2024-01-11T09:46:14.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.058+0000 7fdcb0ff9700 1 -- 172.21.15.27:0/1847708761 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac01c410 con 0x7fdcb4156810 2024-01-11T09:46:14.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.058+0000 7fdcbafff700 1 -- 172.21.15.27:0/1847708761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb40c04a0 con 0x7fdcb4156810 2024-01-11T09:46:14.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.059+0000 7fdcbafff700 1 -- 172.21.15.27:0/1847708761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb404c3c0 con 0x7fdcb4156810 2024-01-11T09:46:14.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.060+0000 7fdcb0ff9700 1 -- 172.21.15.27:0/1847708761 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fdcac003040 con 0x7fdcb4156810 2024-01-11T09:46:14.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.061+0000 7fdcb0ff9700 1 --2- 172.21.15.27:0/1847708761 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fdc98067170 0x7fdc98069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:14.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.061+0000 7fdcb0ff9700 1 -- 172.21.15.27:0/1847708761 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 6646+0+0 (secure 0 0 0) 0x7fdcac086350 con 0x7fdcb4156810 2024-01-11T09:46:14.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.061+0000 7fdcb8d9b700 1 --2- 172.21.15.27:0/1847708761 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fdc98067170 0x7fdc98069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:14.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.062+0000 7fdcb8d9b700 1 --2- 172.21.15.27:0/1847708761 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fdc98067170 0x7fdc98069620 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdca8011520 tx=0x7fdca8009120 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:14.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.065+0000 7fdcb0ff9700 1 -- 172.21.15.27:0/1847708761 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdcac052dd0 con 0x7fdcb4156810 2024-01-11T09:46:14.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:14.271+0000 7fdcbafff700 1 -- 172.21.15.27:0/1847708761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdcb40c8220 con 0x7fdcb4156810 2024-01-11T09:46:15.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.176+0000 7fdcb0ff9700 1 -- 172.21.15.27:0/1847708761 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v225) v1 ==== 190+0+0 (secure 0 0 0) 0x7fdcac0563f0 con 0x7fdcb4156810 2024-01-11T09:46:15.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.181+0000 7fdcbafff700 1 -- 172.21.15.27:0/1847708761 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fdc98067170 msgr2=0x7fdc98069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:15.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.181+0000 7fdcbafff700 1 --2- 172.21.15.27:0/1847708761 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fdc98067170 0x7fdc98069620 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdca8011520 tx=0x7fdca8009120 comp rx=0 tx=0).stop 2024-01-11T09:46:15.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.181+0000 7fdcbafff700 1 -- 172.21.15.27:0/1847708761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcb4156810 msgr2=0x7fdcb4157e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:15.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.181+0000 7fdcbafff700 1 --2- 172.21.15.27:0/1847708761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcb4156810 0x7fdcb4157e90 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7fdcac015000 tx=0x7fdcac018000 comp rx=0 tx=0).stop 2024-01-11T09:46:15.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.182+0000 7fdcbafff700 1 -- 172.21.15.27:0/1847708761 shutdown_connections 2024-01-11T09:46:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.182+0000 7fdcbafff700 1 --2- 172.21.15.27:0/1847708761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcb4156810 0x7fdcb4157e90 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.182+0000 7fdcbafff700 1 --2- 172.21.15.27:0/1847708761 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdcb406ee30 0x7fdcb4153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.182+0000 7fdcbafff700 1 --2- 172.21.15.27:0/1847708761 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fdc98067170 0x7fdc98069620 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.182+0000 7fdcbafff700 1 --2- 172.21.15.27:0/1847708761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcb4156440 0x7fdcb41575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.182+0000 7fdcbafff700 1 -- 172.21.15.27:0/1847708761 >> 172.21.15.27:0/1847708761 conn(0x7fdcb40680e0 msgr2=0x7fdcb4069b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.182+0000 7fdcbafff700 1 -- 172.21.15.27:0/1847708761 shutdown_connections 2024-01-11T09:46:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.182+0000 7fdcbafff700 1 -- 172.21.15.27:0/1847708761 wait complete. 2024-01-11T09:46:15.176 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'backup_fs_metadata' removed 2024-01-11T09:46:15.198 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2024-01-11T09:46:15.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.415+0000 7f9eb37ff700 1 Processor -- start 2024-01-11T09:46:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.415+0000 7f9eb37ff700 1 -- start start 2024-01-11T09:46:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.416+0000 7f9eb37ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9eac15a7a0 0x7f9eac15c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.416+0000 7f9eb37ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9eac159e60 0x7f9eac15aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.416+0000 7f9eb37ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eac15c4b0 con 0x7f9eac15b400 2024-01-11T09:46:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.417+0000 7f9eb37ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eac15b8a0 con 0x7f9eac159e60 2024-01-11T09:46:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.417+0000 7f9eb37ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac15bb80 con 0x7f9eac15a7a0 2024-01-11T09:46:15.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.417+0000 7f9eb0d9a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9eac159e60 0x7f9eac15aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:15.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.417+0000 7f9eb159b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9eac15a7a0 0x7f9eac15c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:15.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.417+0000 7f9eb0d9a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9eac159e60 0x7f9eac15aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:41032/0 (socket says 172.21.15.27:41032) 2024-01-11T09:46:15.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.417+0000 7f9eb0d9a700 1 -- 172.21.15.27:0/3144010820 learned_addr learned my addr 172.21.15.27:0/3144010820 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:15.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.418+0000 7f9eb0d9a700 1 -- 172.21.15.27:0/3144010820 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9eac15a7a0 msgr2=0x7f9eac15c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:15.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.418+0000 7f9eb0d9a700 1 --2- 172.21.15.27:0/3144010820 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9eac15a7a0 0x7f9eac15c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:15.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.418+0000 7f9eb0d9a700 1 -- 172.21.15.27:0/3144010820 >> v1:172.21.15.77:6789/0 conn(0x7f9eac15b400 legacy=0x7f9eac0685e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:15.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.418+0000 7f9eb0d9a700 1 -- 172.21.15.27:0/3144010820 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eac15be60 con 0x7f9eac159e60 2024-01-11T09:46:15.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.418+0000 7f9eb0d9a700 1 --2- 172.21.15.27:0/3144010820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9eac159e60 0x7f9eac15aec0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0099b0 tx=0x7f9e9c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48d919a2a6e0e1cb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:15.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.418+0000 7f9ea3fff700 1 -- 172.21.15.27:0/3144010820 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c009ea0 con 0x7f9eac159e60 2024-01-11T09:46:15.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.418+0000 7f9ea3fff700 1 -- 172.21.15.27:0/3144010820 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9c00b860 con 0x7f9eac159e60 2024-01-11T09:46:15.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.418+0000 7f9ea3fff700 1 -- 172.21.15.27:0/3144010820 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c00bc00 con 0x7f9eac159e60 2024-01-11T09:46:15.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.419+0000 7f9eb37ff700 1 -- 172.21.15.27:0/3144010820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9eac159e60 msgr2=0x7f9eac15aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:15.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.419+0000 7f9eb37ff700 1 --2- 172.21.15.27:0/3144010820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9eac159e60 0x7f9eac15aec0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0099b0 tx=0x7f9e9c009250 comp rx=0 tx=0).stop 2024-01-11T09:46:15.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.419+0000 7f9eb37ff700 1 -- 172.21.15.27:0/3144010820 shutdown_connections 2024-01-11T09:46:15.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.419+0000 7f9eb37ff700 1 --2- 172.21.15.27:0/3144010820 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9eac15a7a0 0x7f9eac15c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:15.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.419+0000 7f9eb37ff700 1 --2- 172.21.15.27:0/3144010820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9eac159e60 0x7f9eac15aec0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:15.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.419+0000 7f9eb37ff700 1 -- 172.21.15.27:0/3144010820 >> 172.21.15.27:0/3144010820 conn(0x7f9eac06ed40 msgr2=0x7f9eac06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:15.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.419+0000 7f9eb37ff700 1 -- 172.21.15.27:0/3144010820 shutdown_connections 2024-01-11T09:46:15.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.419+0000 7f9eb37ff700 1 -- 172.21.15.27:0/3144010820 wait complete. 2024-01-11T09:46:15.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.420+0000 7f9eb37ff700 1 Processor -- start 2024-01-11T09:46:15.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.420+0000 7f9eb37ff700 1 -- start start 2024-01-11T09:46:15.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.421+0000 7f9eb37ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9eac15a7a0 0x7f9eac169ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:15.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.421+0000 7f9eb37ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eac15b400 0x7f9eac1466e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:15.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.421+0000 7f9eb37ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9eac0b19f0 0x7f9eac144290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:15.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.421+0000 7f9eb37ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac16a0c0 con 0x7f9eac15b400 2024-01-11T09:46:15.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.421+0000 7f9eb37ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac16a3a0 con 0x7f9eac15a7a0 2024-01-11T09:46:15.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.421+0000 7f9eb37ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac16a680 con 0x7f9eac0b19f0 2024-01-11T09:46:15.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.422+0000 7f9eb0d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eac15b400 0x7f9eac1466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:15.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.422+0000 7f9eb0d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eac15b400 0x7f9eac1466e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60338/0 (socket says 172.21.15.27:60338) 2024-01-11T09:46:15.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.422+0000 7f9eb0d9a700 1 -- 172.21.15.27:0/2372817441 learned_addr learned my addr 172.21.15.27:0/2372817441 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:15.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.422+0000 7f9eb159b700 1 --2- 172.21.15.27:0/2372817441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9eac15a7a0 0x7f9eac169ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:15.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.422+0000 7f9eb1d9c700 1 --2- 172.21.15.27:0/2372817441 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9eac0b19f0 0x7f9eac144290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:15.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.422+0000 7f9eb0d9a700 1 -- 172.21.15.27:0/2372817441 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9eac0b19f0 msgr2=0x7f9eac144290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:15.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.422+0000 7f9eb0d9a700 1 --2- 172.21.15.27:0/2372817441 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9eac0b19f0 0x7f9eac144290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:15.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.422+0000 7f9eb0d9a700 1 -- 172.21.15.27:0/2372817441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9eac15a7a0 msgr2=0x7f9eac169ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:15.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.422+0000 7f9eb0d9a700 1 --2- 172.21.15.27:0/2372817441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9eac15a7a0 0x7f9eac169ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:15.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.422+0000 7f9eb0d9a700 1 -- 172.21.15.27:0/2372817441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9c0094f0 con 0x7f9eac15b400 2024-01-11T09:46:15.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.422+0000 7f9eb0d9a700 1 --2- 172.21.15.27:0/2372817441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eac15b400 0x7f9eac1466e0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f9eac088e90 tx=0x7f9e9c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:15.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.423+0000 7f9ea17fa700 1 -- 172.21.15.27:0/2372817441 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c000d10 con 0x7f9eac15b400 2024-01-11T09:46:15.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.423+0000 7f9ea17fa700 1 -- 172.21.15.27:0/2372817441 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9c00bbe0 con 0x7f9eac15b400 2024-01-11T09:46:15.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.423+0000 7f9ea17fa700 1 -- 172.21.15.27:0/2372817441 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c011670 con 0x7f9eac15b400 2024-01-11T09:46:15.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.423+0000 7f9eb37ff700 1 -- 172.21.15.27:0/2372817441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eac16a960 con 0x7f9eac15b400 2024-01-11T09:46:15.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.423+0000 7f9eb37ff700 1 -- 172.21.15.27:0/2372817441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eac1447d0 con 0x7f9eac15b400 2024-01-11T09:46:15.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.424+0000 7f9eb37ff700 1 -- 172.21.15.27:0/2372817441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eac04c3c0 con 0x7f9eac15b400 2024-01-11T09:46:15.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.425+0000 7f9ea17fa700 1 -- 172.21.15.27:0/2372817441 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f9e9c011aa0 con 0x7f9eac15b400 2024-01-11T09:46:15.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.426+0000 7f9ea17fa700 1 --2- 172.21.15.27:0/2372817441 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9e90067170 0x7f9e90069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:15.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.426+0000 7f9ea17fa700 1 -- 172.21.15.27:0/2372817441 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f9e9c086270 con 0x7f9eac15b400 2024-01-11T09:46:15.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.427+0000 7f9eb159b700 1 --2- 172.21.15.27:0/2372817441 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9e90067170 0x7f9e90069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:15.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.427+0000 7f9eb159b700 1 --2- 172.21.15.27:0/2372817441 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9e90067170 0x7f9e90069620 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8009db0 tx=0x7f9ea80091a0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:15.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.431+0000 7f9ea17fa700 1 -- 172.21.15.27:0/2372817441 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9e9c052fc0 con 0x7f9eac15b400 2024-01-11T09:46:15.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:15.637+0000 7f9eb37ff700 1 -- 172.21.15.27:0/2372817441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9eac16a200 con 0x7f9eac15b400 2024-01-11T09:46:16.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.182+0000 7f9ea17fa700 1 -- 172.21.15.27:0/2372817441 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v226) v1 ==== 178+0+0 (secure 0 0 0) 0x7f9eac16a200 con 0x7f9eac15b400 2024-01-11T09:46:16.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.186+0000 7f9eb37ff700 1 -- 172.21.15.27:0/2372817441 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9e90067170 msgr2=0x7f9e90069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:16.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.186+0000 7f9eb37ff700 1 --2- 172.21.15.27:0/2372817441 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9e90067170 0x7f9e90069620 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8009db0 tx=0x7f9ea80091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:16.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.186+0000 7f9eb37ff700 1 -- 172.21.15.27:0/2372817441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eac15b400 msgr2=0x7f9eac1466e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:16.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.186+0000 7f9eb37ff700 1 --2- 172.21.15.27:0/2372817441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eac15b400 0x7f9eac1466e0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f9eac088e90 tx=0x7f9e9c017000 comp rx=0 tx=0).stop 2024-01-11T09:46:16.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.186+0000 7f9eb37ff700 1 -- 172.21.15.27:0/2372817441 shutdown_connections 2024-01-11T09:46:16.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.186+0000 7f9eb37ff700 1 --2- 172.21.15.27:0/2372817441 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9eac0b19f0 0x7f9eac144290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:16.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.186+0000 7f9eb37ff700 1 --2- 172.21.15.27:0/2372817441 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9e90067170 0x7f9e90069620 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:16.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.186+0000 7f9eb37ff700 1 --2- 172.21.15.27:0/2372817441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9eac15a7a0 0x7f9eac169ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:16.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.186+0000 7f9eb37ff700 1 --2- 172.21.15.27:0/2372817441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9eac15b400 0x7f9eac1466e0 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:16.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.186+0000 7f9eb37ff700 1 -- 172.21.15.27:0/2372817441 >> 172.21.15.27:0/2372817441 conn(0x7f9eac06ed40 msgr2=0x7f9eac1580f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:16.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.186+0000 7f9eb37ff700 1 -- 172.21.15.27:0/2372817441 shutdown_connections 2024-01-11T09:46:16.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.187+0000 7f9eb37ff700 1 -- 172.21.15.27:0/2372817441 wait complete. 2024-01-11T09:46:16.181 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'backup_fs_data' removed 2024-01-11T09:46:16.200 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-11T09:46:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.411+0000 7f8fb09ff700 1 Processor -- start 2024-01-11T09:46:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.412+0000 7f8fb09ff700 1 -- start start 2024-01-11T09:46:16.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.413+0000 7f8fb09ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fac15a1b0 0x7f8fac15a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:16.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.413+0000 7f8fb09ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fac0cf850 con 0x7f8fac0ccd70 2024-01-11T09:46:16.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.413+0000 7f8fb09ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fac0cec40 con 0x7f8fac15ae70 2024-01-11T09:46:16.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.413+0000 7f8fb09ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac0cef20 con 0x7f8fac15a1b0 2024-01-11T09:46:16.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.413+0000 7f8fa9d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8fac0ccd70 0x7f8fac0cd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39798/0 (socket says 172.21.15.27:39798) 2024-01-11T09:46:16.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.413+0000 7f8fa9d9b700 1 -- 172.21.15.27:0/1374234748 learned_addr learned my addr 172.21.15.27:0/1374234748 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.414+0000 7f8faa59c700 1 --2- 172.21.15.27:0/1374234748 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fac15a1b0 0x7f8fac15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.414+0000 7f8fa959a700 1 -- 172.21.15.27:0/1374234748 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949945938 0 0) 0x7f8fac0cf850 con 0x7f8fac0ccd70 2024-01-11T09:46:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.414+0000 7f8fa959a700 1 -- 172.21.15.27:0/1374234748 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f94003570 con 0x7f8fac0ccd70 2024-01-11T09:46:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.414+0000 7f8fa959a700 1 -- 172.21.15.27:0/1374234748 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017108142 0 0) 0x7f8fac0cec40 con 0x7f8fac15ae70 2024-01-11T09:46:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.414+0000 7f8fa959a700 1 -- 172.21.15.27:0/1374234748 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fac0cf850 con 0x7f8fac15ae70 2024-01-11T09:46:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.414+0000 7f8fa959a700 1 -- 172.21.15.27:0/1374234748 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 588753408 0 0) 0x7f8f94003570 con 0x7f8fac0ccd70 2024-01-11T09:46:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.414+0000 7f8fa959a700 1 -- 172.21.15.27:0/1374234748 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fac15a1b0 msgr2=0x7f8fac15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.414+0000 7f8fa959a700 1 --2- 172.21.15.27:0/1374234748 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fac15a1b0 0x7f8fac15a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.414+0000 7f8fa959a700 1 -- 172.21.15.27:0/1374234748 >> v1:172.21.15.77:6789/0 conn(0x7f8fac15ae70 legacy=0x7f8fac15b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.414+0000 7f8fa959a700 1 -- 172.21.15.27:0/1374234748 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fac0cf200 con 0x7f8fac0ccd70 2024-01-11T09:46:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.414+0000 7f8fa959a700 1 -- 172.21.15.27:0/1374234748 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f8fa0001470 con 0x7f8fac0ccd70 2024-01-11T09:46:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.414+0000 7f8fa959a700 1 -- 172.21.15.27:0/1374234748 >> v1:172.21.15.27:6789/0 conn(0x7f8fac0ccd70 legacy=0x7f8fac0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.414+0000 7f8fa959a700 1 --2- 172.21.15.27:0/1374234748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f94003c20 0x7f8f94004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.414+0000 7f8fa959a700 1 --2- 172.21.15.27:0/1374234748 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fac15a1b0 0x7f8f94004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:16.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.414+0000 7f8fa959a700 1 --2- 172.21.15.27:0/1374234748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f94004d50 0x7f8f940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:16.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.414+0000 7f8fa959a700 1 -- 172.21.15.27:0/1374234748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac0cef20 con 0x7f8f94003c20 2024-01-11T09:46:16.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.414+0000 7f8fa959a700 1 -- 172.21.15.27:0/1374234748 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9400bb40 con 0x7f8f94004d50 2024-01-11T09:46:16.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.414+0000 7f8fa959a700 1 -- 172.21.15.27:0/1374234748 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9400bd00 con 0x7f8fac15a1b0 2024-01-11T09:46:16.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.414+0000 7f8faa59c700 1 --2- 172.21.15.27:0/1374234748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f94003c20 0x7f8f94004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:16.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.415+0000 7f8faad9d700 1 --2- 172.21.15.27:0/1374234748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f94004d50 0x7f8f940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:16.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.415+0000 7f8fa9d9b700 1 --2- 172.21.15.27:0/1374234748 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fac15a1b0 0x7f8f94004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:16.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.415+0000 7f8faa59c700 1 -- 172.21.15.27:0/1374234748 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fac15a1b0 msgr2=0x7f8f94004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:16.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.415+0000 7f8faa59c700 1 --2- 172.21.15.27:0/1374234748 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fac15a1b0 0x7f8f94004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:16.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.415+0000 7f8faa59c700 1 -- 172.21.15.27:0/1374234748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f94004d50 msgr2=0x7f8f940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:16.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.415+0000 7f8faa59c700 1 --2- 172.21.15.27:0/1374234748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f94004d50 0x7f8f940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:16.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.415+0000 7f8faa59c700 1 -- 172.21.15.27:0/1374234748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f94003570 con 0x7f8f94003c20 2024-01-11T09:46:16.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.415+0000 7f8faad9d700 1 --2- 172.21.15.27:0/1374234748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f94004d50 0x7f8f940051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:16.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.415+0000 7f8faa59c700 1 --2- 172.21.15.27:0/1374234748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f94003c20 0x7f8f94004010 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c009c00 tx=0x7f8f9c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=452af0ca77a28795 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:16.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.415+0000 7f8fa959a700 1 -- 172.21.15.27:0/1374234748 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c00e710 con 0x7f8f94003c20 2024-01-11T09:46:16.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.415+0000 7f8fa959a700 1 -- 172.21.15.27:0/1374234748 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9c00e870 con 0x7f8f94003c20 2024-01-11T09:46:16.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.415+0000 7f8fa959a700 1 -- 172.21.15.27:0/1374234748 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c00ec30 con 0x7f8f94003c20 2024-01-11T09:46:16.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.416+0000 7f8fb09ff700 1 -- 172.21.15.27:0/1374234748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f94003c20 msgr2=0x7f8f94004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:16.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.416+0000 7f8fb09ff700 1 --2- 172.21.15.27:0/1374234748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f94003c20 0x7f8f94004010 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c009c00 tx=0x7f8f9c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:16.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.416+0000 7f8faad9d700 1 -- 172.21.15.27:0/1374234748 reap_dead start 2024-01-11T09:46:16.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.416+0000 7f8fb09ff700 1 -- 172.21.15.27:0/1374234748 shutdown_connections 2024-01-11T09:46:16.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.416+0000 7f8fb09ff700 1 -- 172.21.15.27:0/1374234748 >> 172.21.15.27:0/1374234748 conn(0x7f8fac06ed40 msgr2=0x7f8fac06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:16.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.416+0000 7f8fb09ff700 1 -- 172.21.15.27:0/1374234748 shutdown_connections 2024-01-11T09:46:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.416+0000 7f8fb09ff700 1 -- 172.21.15.27:0/1374234748 wait complete. 2024-01-11T09:46:16.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.417+0000 7f8fb09ff700 1 Processor -- start 2024-01-11T09:46:16.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.418+0000 7f8fb09ff700 1 -- start start 2024-01-11T09:46:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.418+0000 7f8fb09ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f94003c20 0x7f8fac0b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.419+0000 7f8fb09ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fac0b5d60 0x7f8fac156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.419+0000 7f8fb09ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fac0b61d0 0x7f8fac1571f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.419+0000 7f8fb09ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac0b36f0 con 0x7f8fac0b61d0 2024-01-11T09:46:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.419+0000 7f8fb09ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac0c9e90 con 0x7f8fac0b5d60 2024-01-11T09:46:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.419+0000 7f8fb09ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac0ca170 con 0x7f8f94003c20 2024-01-11T09:46:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.419+0000 7f8faad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fac0b61d0 0x7f8fac1571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.419+0000 7f8faad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fac0b61d0 0x7f8fac1571f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60352/0 (socket says 172.21.15.27:60352) 2024-01-11T09:46:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.419+0000 7f8faad9d700 1 -- 172.21.15.27:0/49262527 learned_addr learned my addr 172.21.15.27:0/49262527 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.419+0000 7f8faa59c700 1 --2- 172.21.15.27:0/49262527 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f94003c20 0x7f8fac0b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.419+0000 7f8fa9d9b700 1 --2- 172.21.15.27:0/49262527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fac0b5d60 0x7f8fac156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.419+0000 7f8faad9d700 1 -- 172.21.15.27:0/49262527 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f94003c20 msgr2=0x7f8fac0b54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.419+0000 7f8faad9d700 1 --2- 172.21.15.27:0/49262527 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f94003c20 0x7f8fac0b54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:16.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.419+0000 7f8faad9d700 1 -- 172.21.15.27:0/49262527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fac0b5d60 msgr2=0x7f8fac156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:16.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.419+0000 7f8faad9d700 1 --2- 172.21.15.27:0/49262527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fac0b5d60 0x7f8fac156cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:16.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.419+0000 7f8faad9d700 1 -- 172.21.15.27:0/49262527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fac0ca450 con 0x7f8fac0b61d0 2024-01-11T09:46:16.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.419+0000 7f8faa59c700 1 --2- 172.21.15.27:0/49262527 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f94003c20 0x7f8fac0b54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:16.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.419+0000 7f8faad9d700 1 --2- 172.21.15.27:0/49262527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fac0b61d0 0x7f8fac1571f0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4001350 tx=0x7f8fa4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:16.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.420+0000 7f8f9a7fc700 1 -- 172.21.15.27:0/49262527 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4017070 con 0x7f8fac0b61d0 2024-01-11T09:46:16.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.420+0000 7f8f9a7fc700 1 -- 172.21.15.27:0/49262527 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa40032e0 con 0x7f8fac0b61d0 2024-01-11T09:46:16.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.420+0000 7f8fb09ff700 1 -- 172.21.15.27:0/49262527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fac0b3380 con 0x7f8fac0b61d0 2024-01-11T09:46:16.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.420+0000 7f8f9a7fc700 1 -- 172.21.15.27:0/49262527 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4006ba0 con 0x7f8fac0b61d0 2024-01-11T09:46:16.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.420+0000 7f8fb09ff700 1 -- 172.21.15.27:0/49262527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fac157730 con 0x7f8fac0b61d0 2024-01-11T09:46:16.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.421+0000 7f8fb09ff700 1 -- 172.21.15.27:0/49262527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fac04c3c0 con 0x7f8fac0b61d0 2024-01-11T09:46:16.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.422+0000 7f8f9a7fc700 1 -- 172.21.15.27:0/49262527 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f8fa4001680 con 0x7f8fac0b61d0 2024-01-11T09:46:16.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.423+0000 7f8f9a7fc700 1 --2- 172.21.15.27:0/49262527 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8f8c067170 0x7f8f8c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:16.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.423+0000 7f8f9a7fc700 1 -- 172.21.15.27:0/49262527 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f8fa40844a0 con 0x7f8fac0b61d0 2024-01-11T09:46:16.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.423+0000 7f8faa59c700 1 --2- 172.21.15.27:0/49262527 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8f8c067170 0x7f8f8c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:16.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.424+0000 7f8faa59c700 1 --2- 172.21.15.27:0/49262527 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8f8c067170 0x7f8f8c069620 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c0091e0 tx=0x7f8f9c01a000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:16.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.427+0000 7f8f9a7fc700 1 -- 172.21.15.27:0/49262527 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8fa4051390 con 0x7f8fac0b61d0 2024-01-11T09:46:16.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.628+0000 7f8fb09ff700 1 -- 172.21.15.27:0/49262527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f8fac0b3830 con 0x7f8fac0b61d0 2024-01-11T09:46:16.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.632+0000 7f8f9a7fc700 1 -- 172.21.15.27:0/49262527 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v79) v1 ==== 120+0+0 (secure 0 0 0) 0x7f8fa40549b0 con 0x7f8fac0b61d0 2024-01-11T09:46:16.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.635+0000 7f8fb09ff700 1 -- 172.21.15.27:0/49262527 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8f8c067170 msgr2=0x7f8f8c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:16.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.635+0000 7f8fb09ff700 1 --2- 172.21.15.27:0/49262527 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8f8c067170 0x7f8f8c069620 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c0091e0 tx=0x7f8f9c01a000 comp rx=0 tx=0).stop 2024-01-11T09:46:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.635+0000 7f8fb09ff700 1 -- 172.21.15.27:0/49262527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fac0b61d0 msgr2=0x7f8fac1571f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.635+0000 7f8fb09ff700 1 --2- 172.21.15.27:0/49262527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fac0b61d0 0x7f8fac1571f0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4001350 tx=0x7f8fa4009040 comp rx=0 tx=0).stop 2024-01-11T09:46:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.635+0000 7f8fb09ff700 1 -- 172.21.15.27:0/49262527 shutdown_connections 2024-01-11T09:46:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.635+0000 7f8fb09ff700 1 --2- 172.21.15.27:0/49262527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fac0b61d0 0x7f8fac1571f0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.635+0000 7f8fb09ff700 1 --2- 172.21.15.27:0/49262527 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8f94003c20 0x7f8fac0b54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.635+0000 7f8fb09ff700 1 --2- 172.21.15.27:0/49262527 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8f8c067170 0x7f8f8c069620 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.635+0000 7f8fb09ff700 1 --2- 172.21.15.27:0/49262527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fac0b5d60 0x7f8fac156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:16.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.635+0000 7f8fb09ff700 1 -- 172.21.15.27:0/49262527 >> 172.21.15.27:0/49262527 conn(0x7f8fac06ed40 msgr2=0x7f8fac156120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:16.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.635+0000 7f8fb09ff700 1 -- 172.21.15.27:0/49262527 shutdown_connections 2024-01-11T09:46:16.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.635+0000 7f8fb09ff700 1 -- 172.21.15.27:0/49262527 wait complete. 2024-01-11T09:46:16.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-11T09:46:16.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.855+0000 7f2e519ff700 1 Processor -- start 2024-01-11T09:46:16.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.855+0000 7f2e519ff700 1 -- start start 2024-01-11T09:46:16.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.857+0000 7f2e519ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2e4c1577e0 0x7f2e4c0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:16.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.857+0000 7f2e519ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e4c0699d0 con 0x7f2e4c069540 2024-01-11T09:46:16.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.857+0000 7f2e519ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e4c069cb0 con 0x7f2e4c157410 2024-01-11T09:46:16.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.857+0000 7f2e519ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e4c069f90 con 0x7f2e4c1577e0 2024-01-11T09:46:16.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.857+0000 7f2e4b7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2e4c069540 0x7f2e4c0c6ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39804/0 (socket says 172.21.15.27:39804) 2024-01-11T09:46:16.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.857+0000 7f2e4b7fe700 1 -- 172.21.15.27:0/3861723123 learned_addr learned my addr 172.21.15.27:0/3861723123 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:16.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.857+0000 7f2e4a7fc700 1 --2- 172.21.15.27:0/3861723123 >> v2:172.21.15.77:3300/0 conn(0x7f2e4c1577e0 0x7f2e4c0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.858+0000 7f2e49ffb700 1 -- 172.21.15.27:0/3861723123 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259325619 0 0) 0x7f2e4c0699d0 con 0x7f2e4c069540 2024-01-11T09:46:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.858+0000 7f2e49ffb700 1 -- 172.21.15.27:0/3861723123 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e34003570 con 0x7f2e4c069540 2024-01-11T09:46:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.858+0000 7f2e4a7fc700 1 -- 172.21.15.27:0/3861723123 >> v1:172.21.15.77:6789/0 conn(0x7f2e4c157410 legacy=0x7f2e4c0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.858+0000 7f2e49ffb700 1 -- 172.21.15.27:0/3861723123 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219240233 0 0) 0x7f2e4c069cb0 con 0x7f2e4c157410 2024-01-11T09:46:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.858+0000 7f2e4a7fc700 1 -- 172.21.15.27:0/3861723123 >> v1:172.21.15.27:6789/0 conn(0x7f2e4c069540 legacy=0x7f2e4c0c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.858+0000 7f2e4a7fc700 1 -- 172.21.15.27:0/3861723123 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4c06a270 con 0x7f2e4c1577e0 2024-01-11T09:46:16.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.858+0000 7f2e4a7fc700 1 --2- 172.21.15.27:0/3861723123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e4c1577e0 0x7f2e4c0ae000 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f2e400098e0 tx=0x7f2e400091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d062220b73cd2010 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.858+0000 7f2e49ffb700 1 -- 172.21.15.27:0/3861723123 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4000b640 con 0x7f2e4c1577e0 2024-01-11T09:46:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.858+0000 7f2e49ffb700 1 -- 172.21.15.27:0/3861723123 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4000b7a0 con 0x7f2e4c1577e0 2024-01-11T09:46:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.858+0000 7f2e49ffb700 1 -- 172.21.15.27:0/3861723123 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4000bb40 con 0x7f2e4c1577e0 2024-01-11T09:46:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.859+0000 7f2e519ff700 1 -- 172.21.15.27:0/3861723123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e4c1577e0 msgr2=0x7f2e4c0ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.859+0000 7f2e519ff700 1 --2- 172.21.15.27:0/3861723123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e4c1577e0 0x7f2e4c0ae000 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f2e400098e0 tx=0x7f2e400091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.859+0000 7f2e519ff700 1 -- 172.21.15.27:0/3861723123 shutdown_connections 2024-01-11T09:46:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.859+0000 7f2e519ff700 1 --2- 172.21.15.27:0/3861723123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e4c1577e0 0x7f2e4c0ae000 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.859+0000 7f2e519ff700 1 -- 172.21.15.27:0/3861723123 >> 172.21.15.27:0/3861723123 conn(0x7f2e4c0680a0 msgr2=0x7f2e4c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.859+0000 7f2e519ff700 1 -- 172.21.15.27:0/3861723123 shutdown_connections 2024-01-11T09:46:16.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.859+0000 7f2e519ff700 1 -- 172.21.15.27:0/3861723123 wait complete. 2024-01-11T09:46:16.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.861+0000 7f2e519ff700 1 Processor -- start 2024-01-11T09:46:16.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.861+0000 7f2e519ff700 1 -- start start 2024-01-11T09:46:16.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.861+0000 7f2e519ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e4c069540 0x7f2e4c153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:16.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.862+0000 7f2e519ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e4c157410 0x7f2e4c150b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:16.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.862+0000 7f2e519ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e4c1577e0 0x7f2e4c15bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:16.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.862+0000 7f2e519ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c1546e0 con 0x7f2e4c1577e0 2024-01-11T09:46:16.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.862+0000 7f2e519ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c0b1ec0 con 0x7f2e4c157410 2024-01-11T09:46:16.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.862+0000 7f2e519ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c1544b0 con 0x7f2e4c069540 2024-01-11T09:46:16.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.862+0000 7f2e4b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e4c1577e0 0x7f2e4c15bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.862+0000 7f2e4b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e4c1577e0 0x7f2e4c15bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60362/0 (socket says 172.21.15.27:60362) 2024-01-11T09:46:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.862+0000 7f2e4b7fe700 1 -- 172.21.15.27:0/1106299968 learned_addr learned my addr 172.21.15.27:0/1106299968 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.862+0000 7f2e4a7fc700 1 --2- 172.21.15.27:0/1106299968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e4c157410 0x7f2e4c150b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.862+0000 7f2e4affd700 1 --2- 172.21.15.27:0/1106299968 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e4c069540 0x7f2e4c153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.862+0000 7f2e4b7fe700 1 -- 172.21.15.27:0/1106299968 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e4c069540 msgr2=0x7f2e4c153f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.862+0000 7f2e4b7fe700 1 --2- 172.21.15.27:0/1106299968 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e4c069540 0x7f2e4c153f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.862+0000 7f2e4b7fe700 1 -- 172.21.15.27:0/1106299968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e4c157410 msgr2=0x7f2e4c150b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.862+0000 7f2e4b7fe700 1 --2- 172.21.15.27:0/1106299968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e4c157410 0x7f2e4c150b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.862+0000 7f2e4b7fe700 1 -- 172.21.15.27:0/1106299968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e40009420 con 0x7f2e4c1577e0 2024-01-11T09:46:16.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.863+0000 7f2e4b7fe700 1 --2- 172.21.15.27:0/1106299968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e4c1577e0 0x7f2e4c15bfc0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f2e44001460 tx=0x7f2e44001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:16.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.863+0000 7f2e32ffd700 1 -- 172.21.15.27:0/1106299968 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4400a940 con 0x7f2e4c1577e0 2024-01-11T09:46:16.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.863+0000 7f2e32ffd700 1 -- 172.21.15.27:0/1106299968 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4400aaa0 con 0x7f2e4c1577e0 2024-01-11T09:46:16.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.863+0000 7f2e32ffd700 1 -- 172.21.15.27:0/1106299968 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4400ae20 con 0x7f2e4c1577e0 2024-01-11T09:46:16.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.863+0000 7f2e519ff700 1 -- 172.21.15.27:0/1106299968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e4c0ca9a0 con 0x7f2e4c1577e0 2024-01-11T09:46:16.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.863+0000 7f2e519ff700 1 -- 172.21.15.27:0/1106299968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e4c0ca350 con 0x7f2e4c1577e0 2024-01-11T09:46:16.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.864+0000 7f2e519ff700 1 -- 172.21.15.27:0/1106299968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e4c0cab30 con 0x7f2e4c1577e0 2024-01-11T09:46:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.865+0000 7f2e32ffd700 1 -- 172.21.15.27:0/1106299968 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f2e44007300 con 0x7f2e4c1577e0 2024-01-11T09:46:16.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.866+0000 7f2e32ffd700 1 --2- 172.21.15.27:0/1106299968 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2e38067170 0x7f2e38069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:16.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.867+0000 7f2e32ffd700 1 -- 172.21.15.27:0/1106299968 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f2e44084d60 con 0x7f2e4c1577e0 2024-01-11T09:46:16.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.867+0000 7f2e4affd700 1 --2- 172.21.15.27:0/1106299968 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2e38067170 0x7f2e38069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:16.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.868+0000 7f2e4affd700 1 --2- 172.21.15.27:0/1106299968 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2e38067170 0x7f2e38069620 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c0038f0 tx=0x7f2e3c009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:16.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:16.870+0000 7f2e32ffd700 1 -- 172.21.15.27:0/1106299968 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2e44051ce0 con 0x7f2e4c1577e0 2024-01-11T09:46:17.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.082+0000 7f2e519ff700 1 -- 172.21.15.27:0/1106299968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f2e4c04c3c0 con 0x7f2e4c1577e0 2024-01-11T09:46:17.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.088+0000 7f2e32ffd700 1 -- 172.21.15.27:0/1106299968 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v80) v1 ==== 115+0+0 (secure 0 0 0) 0x7f2e44055300 con 0x7f2e4c1577e0 2024-01-11T09:46:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.091+0000 7f2e519ff700 1 -- 172.21.15.27:0/1106299968 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2e38067170 msgr2=0x7f2e38069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.091+0000 7f2e519ff700 1 --2- 172.21.15.27:0/1106299968 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2e38067170 0x7f2e38069620 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c0038f0 tx=0x7f2e3c009040 comp rx=0 tx=0).stop 2024-01-11T09:46:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.091+0000 7f2e519ff700 1 -- 172.21.15.27:0/1106299968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e4c1577e0 msgr2=0x7f2e4c15bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.091+0000 7f2e519ff700 1 --2- 172.21.15.27:0/1106299968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e4c1577e0 0x7f2e4c15bfc0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f2e44001460 tx=0x7f2e44001ae0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.091+0000 7f2e519ff700 1 -- 172.21.15.27:0/1106299968 shutdown_connections 2024-01-11T09:46:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.091+0000 7f2e519ff700 1 --2- 172.21.15.27:0/1106299968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e4c1577e0 0x7f2e4c15bfc0 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.091+0000 7f2e519ff700 1 --2- 172.21.15.27:0/1106299968 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e4c069540 0x7f2e4c153f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.091+0000 7f2e519ff700 1 --2- 172.21.15.27:0/1106299968 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2e38067170 0x7f2e38069620 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.091+0000 7f2e519ff700 1 --2- 172.21.15.27:0/1106299968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e4c157410 0x7f2e4c150b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.091+0000 7f2e519ff700 1 -- 172.21.15.27:0/1106299968 >> 172.21.15.27:0/1106299968 conn(0x7f2e4c0680a0 msgr2=0x7f2e4c0c7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.091+0000 7f2e519ff700 1 -- 172.21.15.27:0/1106299968 shutdown_connections 2024-01-11T09:46:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.091+0000 7f2e519ff700 1 -- 172.21.15.27:0/1106299968 wait complete. 2024-01-11T09:46:17.100 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-11T09:46:17.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.311+0000 7f1984fff700 1 Processor -- start 2024-01-11T09:46:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.311+0000 7f1984fff700 1 -- start start 2024-01-11T09:46:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.312+0000 7f1984fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1980143190 0x7f19800cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.312+0000 7f1984fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19800ccc70 con 0x7f1980142dc0 2024-01-11T09:46:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.312+0000 7f1984fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f198015d320 con 0x7f19800cb4b0 2024-01-11T09:46:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.312+0000 7f1984fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f198015df30 con 0x7f1980143190 2024-01-11T09:46:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.313+0000 7f197e59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1980142dc0 0x7f19800cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39816/0 (socket says 172.21.15.27:39816) 2024-01-11T09:46:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.313+0000 7f197e59c700 1 -- 172.21.15.27:0/43040837 learned_addr learned my addr 172.21.15.27:0/43040837 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.313+0000 7f197dd9b700 1 --2- 172.21.15.27:0/43040837 >> v2:172.21.15.77:3300/0 conn(0x7f1980143190 0x7f19800cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.313+0000 7f197d59a700 1 -- 172.21.15.27:0/43040837 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455831699 0 0) 0x7f19800ccc70 con 0x7f1980142dc0 2024-01-11T09:46:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.313+0000 7f197d59a700 1 -- 172.21.15.27:0/43040837 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1968003570 con 0x7f1980142dc0 2024-01-11T09:46:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.313+0000 7f197d59a700 1 -- 172.21.15.27:0/43040837 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779816581 0 0) 0x7f198015d320 con 0x7f19800cb4b0 2024-01-11T09:46:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.313+0000 7f197d59a700 1 -- 172.21.15.27:0/43040837 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19800ccc70 con 0x7f19800cb4b0 2024-01-11T09:46:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.313+0000 7f197dd9b700 1 -- 172.21.15.27:0/43040837 >> v1:172.21.15.77:6789/0 conn(0x7f19800cb4b0 legacy=0x7f19800cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.313+0000 7f197dd9b700 1 -- 172.21.15.27:0/43040837 >> v1:172.21.15.27:6789/0 conn(0x7f1980142dc0 legacy=0x7f19800cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.313+0000 7f197dd9b700 1 -- 172.21.15.27:0/43040837 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198015d590 con 0x7f1980143190 2024-01-11T09:46:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.314+0000 7f197dd9b700 1 --2- 172.21.15.27:0/43040837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1980143190 0x7f19800cc020 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f19740098e0 tx=0x7f19740091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b74c321ff9afa675 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.314+0000 7f197d59a700 1 -- 172.21.15.27:0/43040837 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1974011630 con 0x7f1980143190 2024-01-11T09:46:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.314+0000 7f197d59a700 1 -- 172.21.15.27:0/43040837 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1974011790 con 0x7f1980143190 2024-01-11T09:46:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.314+0000 7f197d59a700 1 -- 172.21.15.27:0/43040837 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1974011b30 con 0x7f1980143190 2024-01-11T09:46:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.314+0000 7f1984fff700 1 -- 172.21.15.27:0/43040837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1980143190 msgr2=0x7f19800cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.314+0000 7f1984fff700 1 --2- 172.21.15.27:0/43040837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1980143190 0x7f19800cc020 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f19740098e0 tx=0x7f19740091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.315+0000 7f1984fff700 1 -- 172.21.15.27:0/43040837 shutdown_connections 2024-01-11T09:46:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.315+0000 7f1984fff700 1 --2- 172.21.15.27:0/43040837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1980143190 0x7f19800cc020 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.315+0000 7f1984fff700 1 -- 172.21.15.27:0/43040837 >> 172.21.15.27:0/43040837 conn(0x7f1980063ca0 msgr2=0x7f198006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.315+0000 7f1984fff700 1 -- 172.21.15.27:0/43040837 shutdown_connections 2024-01-11T09:46:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.315+0000 7f1984fff700 1 -- 172.21.15.27:0/43040837 wait complete. 2024-01-11T09:46:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.316+0000 7f1984fff700 1 Processor -- start 2024-01-11T09:46:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.316+0000 7f1984fff700 1 -- start start 2024-01-11T09:46:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.317+0000 7f1984fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19800cb4b0 0x7f19800ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.317+0000 7f1984fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1980142dc0 0x7f19800c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.317+0000 7f1984fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1980143190 0x7f198016d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.317+0000 7f1984fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19800ca8d0 con 0x7f1980142dc0 2024-01-11T09:46:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.317+0000 7f1984fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19800c8ac0 con 0x7f19800cb4b0 2024-01-11T09:46:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.317+0000 7f1984fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19800c5800 con 0x7f1980143190 2024-01-11T09:46:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.317+0000 7f197dd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1980142dc0 0x7f19800c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.317+0000 7f197dd9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1980142dc0 0x7f19800c8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60376/0 (socket says 172.21.15.27:60376) 2024-01-11T09:46:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.318+0000 7f197dd9b700 1 -- 172.21.15.27:0/2142583664 learned_addr learned my addr 172.21.15.27:0/2142583664 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.318+0000 7f197e59c700 1 --2- 172.21.15.27:0/2142583664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19800cb4b0 0x7f19800ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.318+0000 7f197ed9d700 1 --2- 172.21.15.27:0/2142583664 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1980143190 0x7f198016d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.318+0000 7f197dd9b700 1 -- 172.21.15.27:0/2142583664 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1980143190 msgr2=0x7f198016d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.318+0000 7f197dd9b700 1 --2- 172.21.15.27:0/2142583664 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1980143190 0x7f198016d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.318+0000 7f197dd9b700 1 -- 172.21.15.27:0/2142583664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19800cb4b0 msgr2=0x7f19800ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.318+0000 7f197dd9b700 1 --2- 172.21.15.27:0/2142583664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19800cb4b0 0x7f19800ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.318+0000 7f197dd9b700 1 -- 172.21.15.27:0/2142583664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1974009420 con 0x7f1980142dc0 2024-01-11T09:46:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.318+0000 7f197ed9d700 1 --2- 172.21.15.27:0/2142583664 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1980143190 0x7f198016d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.318+0000 7f197dd9b700 1 --2- 172.21.15.27:0/2142583664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1980142dc0 0x7f19800c8580 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f1980064a70 tx=0x7f1974017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:17.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.318+0000 7f196e7fc700 1 -- 172.21.15.27:0/2142583664 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197401f070 con 0x7f1980142dc0 2024-01-11T09:46:17.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.319+0000 7f196e7fc700 1 -- 172.21.15.27:0/2142583664 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1974000bc0 con 0x7f1980142dc0 2024-01-11T09:46:17.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.319+0000 7f196e7fc700 1 -- 172.21.15.27:0/2142583664 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1974025420 con 0x7f1980142dc0 2024-01-11T09:46:17.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.319+0000 7f1984fff700 1 -- 172.21.15.27:0/2142583664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19800c8cd0 con 0x7f1980142dc0 2024-01-11T09:46:17.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.319+0000 7f1984fff700 1 -- 172.21.15.27:0/2142583664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19800c9290 con 0x7f1980142dc0 2024-01-11T09:46:17.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.320+0000 7f1984fff700 1 -- 172.21.15.27:0/2142583664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19800c5990 con 0x7f1980142dc0 2024-01-11T09:46:17.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.321+0000 7f196e7fc700 1 -- 172.21.15.27:0/2142583664 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f1974023070 con 0x7f1980142dc0 2024-01-11T09:46:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.322+0000 7f196e7fc700 1 --2- 172.21.15.27:0/2142583664 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1960067170 0x7f1960069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.322+0000 7f196e7fc700 1 -- 172.21.15.27:0/2142583664 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f1974085bc0 con 0x7f1980142dc0 2024-01-11T09:46:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.322+0000 7f197e59c700 1 --2- 172.21.15.27:0/2142583664 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1960067170 0x7f1960069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.323+0000 7f197e59c700 1 --2- 172.21.15.27:0/2142583664 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1960067170 0x7f1960069620 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1970007fd0 tx=0x7f1970009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.326+0000 7f196e7fc700 1 -- 172.21.15.27:0/2142583664 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1974052ab0 con 0x7f1980142dc0 2024-01-11T09:46:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.529+0000 7f1984fff700 1 -- 172.21.15.27:0/2142583664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f198004c3c0 con 0x7f1980142dc0 2024-01-11T09:46:17.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.535+0000 7f196e7fc700 1 -- 172.21.15.27:0/2142583664 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v81) v1 ==== 104+0+0 (secure 0 0 0) 0x7f197402a0a0 con 0x7f1980142dc0 2024-01-11T09:46:17.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.537+0000 7f1984fff700 1 -- 172.21.15.27:0/2142583664 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1960067170 msgr2=0x7f1960069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:17.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.537+0000 7f1984fff700 1 --2- 172.21.15.27:0/2142583664 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1960067170 0x7f1960069620 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1970007fd0 tx=0x7f1970009040 comp rx=0 tx=0).stop 2024-01-11T09:46:17.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.537+0000 7f1984fff700 1 -- 172.21.15.27:0/2142583664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1980142dc0 msgr2=0x7f19800c8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:17.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.537+0000 7f1984fff700 1 --2- 172.21.15.27:0/2142583664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1980142dc0 0x7f19800c8580 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f1980064a70 tx=0x7f1974017000 comp rx=0 tx=0).stop 2024-01-11T09:46:17.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.537+0000 7f1984fff700 1 -- 172.21.15.27:0/2142583664 shutdown_connections 2024-01-11T09:46:17.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.537+0000 7f1984fff700 1 --2- 172.21.15.27:0/2142583664 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1980143190 0x7f198016d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.537+0000 7f1984fff700 1 --2- 172.21.15.27:0/2142583664 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1960067170 0x7f1960069620 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.537+0000 7f1984fff700 1 --2- 172.21.15.27:0/2142583664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19800cb4b0 0x7f19800ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.537+0000 7f1984fff700 1 --2- 172.21.15.27:0/2142583664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1980142dc0 0x7f19800c8580 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.537+0000 7f1984fff700 1 -- 172.21.15.27:0/2142583664 >> 172.21.15.27:0/2142583664 conn(0x7f1980063ca0 msgr2=0x7f19800c4bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:17.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.537+0000 7f1984fff700 1 -- 172.21.15.27:0/2142583664 shutdown_connections 2024-01-11T09:46:17.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.537+0000 7f1984fff700 1 -- 172.21.15.27:0/2142583664 wait complete. 2024-01-11T09:46:17.547 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2024-01-11T09:46:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.754+0000 7fe7e89ff700 1 Processor -- start 2024-01-11T09:46:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.755+0000 7fe7e89ff700 1 -- start start 2024-01-11T09:46:17.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.756+0000 7fe7e89ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7e41575e0 0x7fe7e4158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.756+0000 7fe7e89ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe7e406b610 0x7fe7e40b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.756+0000 7fe7e89ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe7e40b1800 0x7fe7e40ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.756+0000 7fe7e89ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e40cb5e0 con 0x7fe7e41575e0 2024-01-11T09:46:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.756+0000 7fe7e89ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e4158b60 con 0x7fe7e406b610 2024-01-11T09:46:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.756+0000 7fe7e89ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e40b1c60 con 0x7fe7e40b1800 2024-01-11T09:46:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.756+0000 7fe7e259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7e41575e0 0x7fe7e4158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.756+0000 7fe7e259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7e41575e0 0x7fe7e4158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60380/0 (socket says 172.21.15.27:60380) 2024-01-11T09:46:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.756+0000 7fe7e259c700 1 -- 172.21.15.27:0/3144715097 learned_addr learned my addr 172.21.15.27:0/3144715097 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.756+0000 7fe7e1d9b700 1 --2- 172.21.15.27:0/3144715097 >> v2:172.21.15.77:3300/0 conn(0x7fe7e406b610 0x7fe7e40b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.756+0000 7fe7e2d9d700 1 --2- 172.21.15.27:0/3144715097 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe7e40b1800 0x7fe7e40ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.757+0000 7fe7e259c700 1 -- 172.21.15.27:0/3144715097 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe7e40b1800 msgr2=0x7fe7e40ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.757+0000 7fe7e259c700 1 --2- 172.21.15.27:0/3144715097 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe7e40b1800 0x7fe7e40ca1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.757+0000 7fe7e259c700 1 -- 172.21.15.27:0/3144715097 >> v2:172.21.15.77:3300/0 conn(0x7fe7e406b610 msgr2=0x7fe7e40b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:17.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.757+0000 7fe7e259c700 1 --2- 172.21.15.27:0/3144715097 >> v2:172.21.15.77:3300/0 conn(0x7fe7e406b610 0x7fe7e40b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.757+0000 7fe7e259c700 1 -- 172.21.15.27:0/3144715097 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e41599b0 con 0x7fe7e41575e0 2024-01-11T09:46:17.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.757+0000 7fe7e1d9b700 1 --2- 172.21.15.27:0/3144715097 >> v2:172.21.15.77:3300/0 conn(0x7fe7e406b610 0x7fe7e40b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:17.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.757+0000 7fe7e2d9d700 1 --2- 172.21.15.27:0/3144715097 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe7e40b1800 0x7fe7e40ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:17.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.757+0000 7fe7e259c700 1 --2- 172.21.15.27:0/3144715097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e41575e0 0x7fe7e4158620 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4064fe0 tx=0x7fe7d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609f9c771b35cac1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:17.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.757+0000 7fe7e159a700 1 -- 172.21.15.27:0/3144715097 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d400b860 con 0x7fe7e41575e0 2024-01-11T09:46:17.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.757+0000 7fe7e159a700 1 -- 172.21.15.27:0/3144715097 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d400b9c0 con 0x7fe7e41575e0 2024-01-11T09:46:17.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.757+0000 7fe7e159a700 1 -- 172.21.15.27:0/3144715097 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d400bd60 con 0x7fe7e41575e0 2024-01-11T09:46:17.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.758+0000 7fe7e89ff700 1 -- 172.21.15.27:0/3144715097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e41575e0 msgr2=0x7fe7e4158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:17.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.758+0000 7fe7e89ff700 1 --2- 172.21.15.27:0/3144715097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e41575e0 0x7fe7e4158620 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4064fe0 tx=0x7fe7d4009250 comp rx=0 tx=0).stop 2024-01-11T09:46:17.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.758+0000 7fe7e89ff700 1 -- 172.21.15.27:0/3144715097 shutdown_connections 2024-01-11T09:46:17.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.758+0000 7fe7e89ff700 1 --2- 172.21.15.27:0/3144715097 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe7e40b1800 0x7fe7e40ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.758+0000 7fe7e89ff700 1 --2- 172.21.15.27:0/3144715097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e41575e0 0x7fe7e4158620 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.758+0000 7fe7e89ff700 1 --2- 172.21.15.27:0/3144715097 >> v2:172.21.15.77:3300/0 conn(0x7fe7e406b610 0x7fe7e40b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.758+0000 7fe7e89ff700 1 -- 172.21.15.27:0/3144715097 >> 172.21.15.27:0/3144715097 conn(0x7fe7e40680e0 msgr2=0x7fe7e4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:17.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.758+0000 7fe7e89ff700 1 -- 172.21.15.27:0/3144715097 shutdown_connections 2024-01-11T09:46:17.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.758+0000 7fe7e89ff700 1 -- 172.21.15.27:0/3144715097 wait complete. 2024-01-11T09:46:17.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.759+0000 7fe7e89ff700 1 Processor -- start 2024-01-11T09:46:17.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.760+0000 7fe7e89ff700 1 -- start start 2024-01-11T09:46:17.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.761+0000 7fe7e89ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe7e406b610 0x7fe7e40c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:17.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.761+0000 7fe7e89ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e40b1800 0x7fe7e416c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:17.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.761+0000 7fe7e89ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7e41575e0 0x7fe7e416d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:17.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.761+0000 7fe7e89ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e4140770 con 0x7fe7e40b1800 2024-01-11T09:46:17.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.761+0000 7fe7e89ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e416ce50 con 0x7fe7e41575e0 2024-01-11T09:46:17.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.761+0000 7fe7e89ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e4153de0 con 0x7fe7e406b610 2024-01-11T09:46:17.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.761+0000 7fe7e1d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e40b1800 0x7fe7e416c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:17.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.761+0000 7fe7e1d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e40b1800 0x7fe7e416c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60382/0 (socket says 172.21.15.27:60382) 2024-01-11T09:46:17.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.761+0000 7fe7e1d9b700 1 -- 172.21.15.27:0/3798064554 learned_addr learned my addr 172.21.15.27:0/3798064554 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:17.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.761+0000 7fe7e259c700 1 --2- 172.21.15.27:0/3798064554 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe7e406b610 0x7fe7e40c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:17.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.761+0000 7fe7e2d9d700 1 --2- 172.21.15.27:0/3798064554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7e41575e0 0x7fe7e416d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:17.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.761+0000 7fe7e1d9b700 1 -- 172.21.15.27:0/3798064554 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe7e406b610 msgr2=0x7fe7e40c74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:17.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.761+0000 7fe7e1d9b700 1 --2- 172.21.15.27:0/3798064554 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe7e406b610 0x7fe7e40c74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.761+0000 7fe7e1d9b700 1 -- 172.21.15.27:0/3798064554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7e41575e0 msgr2=0x7fe7e416d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:17.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.761+0000 7fe7e1d9b700 1 --2- 172.21.15.27:0/3798064554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7e41575e0 0x7fe7e416d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:17.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.761+0000 7fe7e1d9b700 1 -- 172.21.15.27:0/3798064554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d40094f0 con 0x7fe7e40b1800 2024-01-11T09:46:17.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.762+0000 7fe7e1d9b700 1 --2- 172.21.15.27:0/3798064554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e40b1800 0x7fe7e416c910 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7fe7d800e8d0 tx=0x7fe7d8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:17.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.762+0000 7fe7d27fc700 1 -- 172.21.15.27:0/3798064554 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d8013410 con 0x7fe7e40b1800 2024-01-11T09:46:17.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.762+0000 7fe7d27fc700 1 -- 172.21.15.27:0/3798064554 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d8013570 con 0x7fe7e40b1800 2024-01-11T09:46:17.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.762+0000 7fe7d27fc700 1 -- 172.21.15.27:0/3798064554 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d8013910 con 0x7fe7e40b1800 2024-01-11T09:46:17.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.762+0000 7fe7e89ff700 1 -- 172.21.15.27:0/3798064554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e40c7a90 con 0x7fe7e40b1800 2024-01-11T09:46:17.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.762+0000 7fe7e89ff700 1 -- 172.21.15.27:0/3798064554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e40c8050 con 0x7fe7e40b1800 2024-01-11T09:46:17.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.763+0000 7fe7e89ff700 1 -- 172.21.15.27:0/3798064554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e40718d0 con 0x7fe7e40b1800 2024-01-11T09:46:17.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.764+0000 7fe7d27fc700 1 -- 172.21.15.27:0/3798064554 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fe7d8013d00 con 0x7fe7e40b1800 2024-01-11T09:46:17.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.765+0000 7fe7d27fc700 1 --2- 172.21.15.27:0/3798064554 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe7c4067170 0x7fe7c4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:17.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.766+0000 7fe7d27fc700 1 -- 172.21.15.27:0/3798064554 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 5782+0+0 (secure 0 0 0) 0x7fe7d80847b0 con 0x7fe7e40b1800 2024-01-11T09:46:17.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.766+0000 7fe7e259c700 1 --2- 172.21.15.27:0/3798064554 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe7c4067170 0x7fe7c4069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:17.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.767+0000 7fe7e259c700 1 --2- 172.21.15.27:0/3798064554 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe7c4067170 0x7fe7c4069620 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4064fe0 tx=0x7fe7d4017000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:17.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:17.770+0000 7fe7d27fc700 1 -- 172.21.15.27:0/3798064554 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe7d8051590 con 0x7fe7e40b1800 2024-01-11T09:46:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.055+0000 7fe7e89ff700 1 -- 172.21.15.27:0/3798064554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7fe7e40c0f90 con 0x7fe7e40b1800 2024-01-11T09:46:18.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.193+0000 7fe7d27fc700 1 -- 172.21.15.27:0/3798064554 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3133) v1 ==== 186+0+0 (secure 0 0 0) 0x7fe7d8054bb0 con 0x7fe7e40b1800 2024-01-11T09:46:18.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.196+0000 7fe7e89ff700 1 -- 172.21.15.27:0/3798064554 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe7c4067170 msgr2=0x7fe7c4069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:18.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.196+0000 7fe7e89ff700 1 --2- 172.21.15.27:0/3798064554 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe7c4067170 0x7fe7c4069620 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4064fe0 tx=0x7fe7d4017000 comp rx=0 tx=0).stop 2024-01-11T09:46:18.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.196+0000 7fe7e89ff700 1 -- 172.21.15.27:0/3798064554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e40b1800 msgr2=0x7fe7e416c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:18.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.196+0000 7fe7e89ff700 1 --2- 172.21.15.27:0/3798064554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e40b1800 0x7fe7e416c910 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7fe7d800e8d0 tx=0x7fe7d8008040 comp rx=0 tx=0).stop 2024-01-11T09:46:18.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.196+0000 7fe7e89ff700 1 -- 172.21.15.27:0/3798064554 shutdown_connections 2024-01-11T09:46:18.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.196+0000 7fe7e89ff700 1 --2- 172.21.15.27:0/3798064554 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe7c4067170 0x7fe7c4069620 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:18.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.196+0000 7fe7e89ff700 1 --2- 172.21.15.27:0/3798064554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7e41575e0 0x7fe7e416d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:18.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.196+0000 7fe7e89ff700 1 --2- 172.21.15.27:0/3798064554 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe7e406b610 0x7fe7e40c74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:18.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.196+0000 7fe7e89ff700 1 --2- 172.21.15.27:0/3798064554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7e40b1800 0x7fe7e416c910 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:18.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.196+0000 7fe7e89ff700 1 -- 172.21.15.27:0/3798064554 >> 172.21.15.27:0/3798064554 conn(0x7fe7e40680e0 msgr2=0x7fe7e415a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:18.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.196+0000 7fe7e89ff700 1 -- 172.21.15.27:0/3798064554 shutdown_connections 2024-01-11T09:46:18.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.196+0000 7fe7e89ff700 1 -- 172.21.15.27:0/3798064554 wait complete. 2024-01-11T09:46:18.205 INFO:tasks.cephfs_test_runner:Test command succes with --fs argument for non-default fs ... ok 2024-01-11T09:46:18.205 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command fails without --fs argument in presence of multiple fs 2024-01-11T09:46:18.205 INFO:tasks.cephfs_test_runner:test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2024-01-11T09:46:18.206 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2024-01-11T09:46:18.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.414+0000 7f77b9dff700 1 Processor -- start 2024-01-11T09:46:18.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.414+0000 7f77b9dff700 1 -- start start 2024-01-11T09:46:18.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.415+0000 7f77b9dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77b40b03e0 0x7f77b40b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:18.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.415+0000 7f77b9dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77b40b07b0 0x7f77b40b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.415+0000 7f77b9dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f77b4159200 0x7f77b4159640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.415+0000 7f77b9dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77b40cd730 con 0x7f77b40b03e0 2024-01-11T09:46:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.415+0000 7f77b9dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77b40ccb20 con 0x7f77b4159200 2024-01-11T09:46:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.415+0000 7f77b9dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b40cce00 con 0x7f77b40b07b0 2024-01-11T09:46:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.416+0000 7f77b37fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77b40b03e0 0x7f77b40b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.416+0000 7f77b37fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77b40b03e0 0x7f77b40b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60384/0 (socket says 172.21.15.27:60384) 2024-01-11T09:46:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.416+0000 7f77b37fe700 1 -- 172.21.15.27:0/2017958607 learned_addr learned my addr 172.21.15.27:0/2017958607 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.416+0000 7f77b2ffd700 1 --2- 172.21.15.27:0/2017958607 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77b40b07b0 0x7f77b40b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.416+0000 7f77b3fff700 1 --2- 172.21.15.27:0/2017958607 >> v2:172.21.15.77:3300/0 conn(0x7f77b4159200 0x7f77b4159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.416+0000 7f77b37fe700 1 -- 172.21.15.27:0/2017958607 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77b40b07b0 msgr2=0x7f77b40b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.416+0000 7f77b37fe700 1 --2- 172.21.15.27:0/2017958607 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77b40b07b0 0x7f77b40b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.416+0000 7f77b37fe700 1 -- 172.21.15.27:0/2017958607 >> v2:172.21.15.77:3300/0 conn(0x7f77b4159200 msgr2=0x7f77b4159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:18.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.416+0000 7f77b37fe700 1 --2- 172.21.15.27:0/2017958607 >> v2:172.21.15.77:3300/0 conn(0x7f77b4159200 0x7f77b4159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:18.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.416+0000 7f77b37fe700 1 -- 172.21.15.27:0/2017958607 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b40cd0e0 con 0x7f77b40b03e0 2024-01-11T09:46:18.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.416+0000 7f77b3fff700 1 --2- 172.21.15.27:0/2017958607 >> v2:172.21.15.77:3300/0 conn(0x7f77b4159200 0x7f77b4159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.416+0000 7f77b37fe700 1 --2- 172.21.15.27:0/2017958607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b40b03e0 0x7f77b40b2480 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f77b40b5f00 tx=0x7f77a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a4ba296ee11f9ed server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.417+0000 7f77b27fc700 1 -- 172.21.15.27:0/2017958607 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a400b640 con 0x7f77b40b03e0 2024-01-11T09:46:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.417+0000 7f77b27fc700 1 -- 172.21.15.27:0/2017958607 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a400b7a0 con 0x7f77b40b03e0 2024-01-11T09:46:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.417+0000 7f77b27fc700 1 -- 172.21.15.27:0/2017958607 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a400bb60 con 0x7f77b40b03e0 2024-01-11T09:46:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.417+0000 7f77b9dff700 1 -- 172.21.15.27:0/2017958607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b40b03e0 msgr2=0x7f77b40b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.417+0000 7f77b9dff700 1 --2- 172.21.15.27:0/2017958607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b40b03e0 0x7f77b40b2480 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f77b40b5f00 tx=0x7f77a4009250 comp rx=0 tx=0).stop 2024-01-11T09:46:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.417+0000 7f77b9dff700 1 -- 172.21.15.27:0/2017958607 shutdown_connections 2024-01-11T09:46:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.417+0000 7f77b9dff700 1 --2- 172.21.15.27:0/2017958607 >> v2:172.21.15.77:3300/0 conn(0x7f77b4159200 0x7f77b4159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.417+0000 7f77b9dff700 1 --2- 172.21.15.27:0/2017958607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b40b03e0 0x7f77b40b2480 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:18.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.417+0000 7f77b9dff700 1 --2- 172.21.15.27:0/2017958607 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77b40b07b0 0x7f77b40b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:18.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.417+0000 7f77b9dff700 1 -- 172.21.15.27:0/2017958607 >> 172.21.15.27:0/2017958607 conn(0x7f77b406ed40 msgr2=0x7f77b406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:18.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.417+0000 7f77b9dff700 1 -- 172.21.15.27:0/2017958607 shutdown_connections 2024-01-11T09:46:18.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.418+0000 7f77b9dff700 1 -- 172.21.15.27:0/2017958607 wait complete. 2024-01-11T09:46:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.419+0000 7f77b9dff700 1 Processor -- start 2024-01-11T09:46:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.419+0000 7f77b9dff700 1 -- start start 2024-01-11T09:46:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.420+0000 7f77b9dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b40b03e0 0x7f77b416aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.420+0000 7f77b9dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77b40b07b0 0x7f77b4159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.420+0000 7f77b9dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77b4159200 0x7f77b40af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.420+0000 7f77b9dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b4159b80 con 0x7f77b40b03e0 2024-01-11T09:46:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.420+0000 7f77b9dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b415bdd0 con 0x7f77b4159200 2024-01-11T09:46:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.420+0000 7f77b9dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b415a0c0 con 0x7f77b40b07b0 2024-01-11T09:46:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.420+0000 7f77b37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b40b03e0 0x7f77b416aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.420+0000 7f77b37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b40b03e0 0x7f77b416aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60392/0 (socket says 172.21.15.27:60392) 2024-01-11T09:46:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.420+0000 7f77b37fe700 1 -- 172.21.15.27:0/4095819949 learned_addr learned my addr 172.21.15.27:0/4095819949 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.421+0000 7f77b3fff700 1 --2- 172.21.15.27:0/4095819949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77b4159200 0x7f77b40af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.421+0000 7f77b2ffd700 1 --2- 172.21.15.27:0/4095819949 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77b40b07b0 0x7f77b4159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.421+0000 7f77b37fe700 1 -- 172.21.15.27:0/4095819949 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77b40b07b0 msgr2=0x7f77b4159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.421+0000 7f77b37fe700 1 --2- 172.21.15.27:0/4095819949 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77b40b07b0 0x7f77b4159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.421+0000 7f77b37fe700 1 -- 172.21.15.27:0/4095819949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77b4159200 msgr2=0x7f77b40af3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.421+0000 7f77b37fe700 1 --2- 172.21.15.27:0/4095819949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77b4159200 0x7f77b40af3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:18.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.421+0000 7f77b37fe700 1 -- 172.21.15.27:0/4095819949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ac009a20 con 0x7f77b40b03e0 2024-01-11T09:46:18.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.421+0000 7f77b37fe700 1 --2- 172.21.15.27:0/4095819949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b40b03e0 0x7f77b416aa10 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f77a4006fd0 tx=0x7f77a4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:18.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.421+0000 7f779b7fe700 1 -- 172.21.15.27:0/4095819949 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a4000d10 con 0x7f77b40b03e0 2024-01-11T09:46:18.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.421+0000 7f779b7fe700 1 -- 172.21.15.27:0/4095819949 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a400b910 con 0x7f77b40b03e0 2024-01-11T09:46:18.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.421+0000 7f779b7fe700 1 -- 172.21.15.27:0/4095819949 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a4025420 con 0x7f77b40b03e0 2024-01-11T09:46:18.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.422+0000 7f77b9dff700 1 -- 172.21.15.27:0/4095819949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a40094f0 con 0x7f77b40b03e0 2024-01-11T09:46:18.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.422+0000 7f77b9dff700 1 -- 172.21.15.27:0/4095819949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b415c040 con 0x7f77b40b03e0 2024-01-11T09:46:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.423+0000 7f77b9dff700 1 -- 172.21.15.27:0/4095819949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b40718d0 con 0x7f77b40b03e0 2024-01-11T09:46:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.423+0000 7f779b7fe700 1 -- 172.21.15.27:0/4095819949 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f77a4023070 con 0x7f77b40b03e0 2024-01-11T09:46:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.424+0000 7f779b7fe700 1 --2- 172.21.15.27:0/4095819949 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f77a0067170 0x7f77a0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.424+0000 7f779b7fe700 1 -- 172.21.15.27:0/4095819949 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f77a4085bf0 con 0x7f77b40b03e0 2024-01-11T09:46:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.425+0000 7f77b2ffd700 1 --2- 172.21.15.27:0/4095819949 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f77a0067170 0x7f77a0069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.425+0000 7f77b2ffd700 1 --2- 172.21.15.27:0/4095819949 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f77a0067170 0x7f77a0069620 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f77b40b4de0 tx=0x7f77a80091a0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:18.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.428+0000 7f779b7fe700 1 -- 172.21.15.27:0/4095819949 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f77a4052c40 con 0x7f77b40b03e0 2024-01-11T09:46:18.580 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:46:18.587 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:46:18.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:18.730+0000 7f77b9dff700 1 -- 172.21.15.27:0/4095819949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7f77b40cd3f0 con 0x7f77b40b03e0 2024-01-11T09:46:19.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.193+0000 7f779b7fe700 1 -- 172.21.15.27:0/4095819949 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3134) v1 ==== 190+0+0 (secure 0 0 0) 0x7f77a4056260 con 0x7f77b40b03e0 2024-01-11T09:46:19.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.196+0000 7f77b9dff700 1 -- 172.21.15.27:0/4095819949 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f77a0067170 msgr2=0x7f77a0069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:19.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.196+0000 7f77b9dff700 1 --2- 172.21.15.27:0/4095819949 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f77a0067170 0x7f77a0069620 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f77b40b4de0 tx=0x7f77a80091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:19.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.196+0000 7f77b9dff700 1 -- 172.21.15.27:0/4095819949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b40b03e0 msgr2=0x7f77b416aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:19.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.196+0000 7f77b9dff700 1 --2- 172.21.15.27:0/4095819949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b40b03e0 0x7f77b416aa10 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f77a4006fd0 tx=0x7f77a4017000 comp rx=0 tx=0).stop 2024-01-11T09:46:19.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.196+0000 7f77b9dff700 1 -- 172.21.15.27:0/4095819949 shutdown_connections 2024-01-11T09:46:19.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.196+0000 7f77b9dff700 1 --2- 172.21.15.27:0/4095819949 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f77a0067170 0x7f77a0069620 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:19.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.196+0000 7f77b9dff700 1 --2- 172.21.15.27:0/4095819949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77b4159200 0x7f77b40af3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:19.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.196+0000 7f77b9dff700 1 --2- 172.21.15.27:0/4095819949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b40b03e0 0x7f77b416aa10 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:19.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.196+0000 7f77b9dff700 1 --2- 172.21.15.27:0/4095819949 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77b40b07b0 0x7f77b4159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:19.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.196+0000 7f77b9dff700 1 -- 172.21.15.27:0/4095819949 >> 172.21.15.27:0/4095819949 conn(0x7f77b406ed40 msgr2=0x7f77b40ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:19.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.196+0000 7f77b9dff700 1 -- 172.21.15.27:0/4095819949 shutdown_connections 2024-01-11T09:46:19.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.196+0000 7f77b9dff700 1 -- 172.21.15.27:0/4095819949 wait complete. 2024-01-11T09:46:19.207 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-11T09:46:19.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.461+0000 7f39699ff700 1 Processor -- start 2024-01-11T09:46:19.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.461+0000 7f39699ff700 1 -- start start 2024-01-11T09:46:19.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.462+0000 7f39699ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3964157410 0x7f39640ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:19.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.462+0000 7f39699ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39641577e0 0x7f39640ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:19.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.462+0000 7f39699ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39640699d0 con 0x7f3964069540 2024-01-11T09:46:19.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.462+0000 7f39699ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3964069cb0 con 0x7f3964157410 2024-01-11T09:46:19.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.462+0000 7f39699ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3964069f90 con 0x7f39641577e0 2024-01-11T09:46:19.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.463+0000 7f39627fc700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39641577e0 0x7f39640ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:19.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.463+0000 7f3962ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3964157410 0x7f39640ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:19.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.463+0000 7f39637fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3964069540 0x7f396415b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:53056/0 (socket says 172.21.15.27:53056) 2024-01-11T09:46:19.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.463+0000 7f39627fc700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39641577e0 0x7f39640ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:40204/0 (socket says 172.21.15.27:40204) 2024-01-11T09:46:19.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.463+0000 7f39637fe700 1 -- 172.21.15.27:0/3861637802 learned_addr learned my addr 172.21.15.27:0/3861637802 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:19.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.463+0000 7f3961ffb700 1 -- 172.21.15.27:0/3861637802 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660919392 0 0) 0x7f39640699d0 con 0x7f3964069540 2024-01-11T09:46:19.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.463+0000 7f39627fc700 1 -- 172.21.15.27:0/3861637802 >> v2:172.21.15.77:3300/0 conn(0x7f3964157410 msgr2=0x7f39640ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:19.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.463+0000 7f39627fc700 1 --2- 172.21.15.27:0/3861637802 >> v2:172.21.15.77:3300/0 conn(0x7f3964157410 0x7f39640ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:19.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.463+0000 7f39627fc700 1 -- 172.21.15.27:0/3861637802 >> v1:172.21.15.77:6789/0 conn(0x7f3964069540 legacy=0x7f396415b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:19.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.463+0000 7f39627fc700 1 -- 172.21.15.27:0/3861637802 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396406a270 con 0x7f39641577e0 2024-01-11T09:46:19.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.464+0000 7f39627fc700 1 --2- 172.21.15.27:0/3861637802 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39641577e0 0x7f39640ae000 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f39580098e0 tx=0x7f39580091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99245d21e02dba75 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:19.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.464+0000 7f3961ffb700 1 -- 172.21.15.27:0/3861637802 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395800b640 con 0x7f39641577e0 2024-01-11T09:46:19.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.464+0000 7f3961ffb700 1 -- 172.21.15.27:0/3861637802 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395800b7a0 con 0x7f39641577e0 2024-01-11T09:46:19.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.464+0000 7f3961ffb700 1 -- 172.21.15.27:0/3861637802 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395800bb40 con 0x7f39641577e0 2024-01-11T09:46:19.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.464+0000 7f39699ff700 1 -- 172.21.15.27:0/3861637802 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39641577e0 msgr2=0x7f39640ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:19.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.464+0000 7f39699ff700 1 --2- 172.21.15.27:0/3861637802 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39641577e0 0x7f39640ae000 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f39580098e0 tx=0x7f39580091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:19.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.464+0000 7f39699ff700 1 -- 172.21.15.27:0/3861637802 shutdown_connections 2024-01-11T09:46:19.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.464+0000 7f39699ff700 1 --2- 172.21.15.27:0/3861637802 >> v2:172.21.15.77:3300/0 conn(0x7f3964157410 0x7f39640ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:19.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.464+0000 7f39699ff700 1 --2- 172.21.15.27:0/3861637802 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f39641577e0 0x7f39640ae000 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:19.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.464+0000 7f39699ff700 1 -- 172.21.15.27:0/3861637802 >> 172.21.15.27:0/3861637802 conn(0x7f39640680a0 msgr2=0x7f39640cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:19.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.465+0000 7f39699ff700 1 -- 172.21.15.27:0/3861637802 shutdown_connections 2024-01-11T09:46:19.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.465+0000 7f39699ff700 1 -- 172.21.15.27:0/3861637802 wait complete. 2024-01-11T09:46:19.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.466+0000 7f39699ff700 1 Processor -- start 2024-01-11T09:46:19.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.466+0000 7f39699ff700 1 -- start start 2024-01-11T09:46:19.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.467+0000 7f39699ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3964069540 0x7f396416a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:19.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.467+0000 7f39699ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3964157410 0x7f3964150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:19.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.467+0000 7f39699ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39641577e0 0x7f39640c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:19.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.467+0000 7f39699ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3964169c80 con 0x7f3964157410 2024-01-11T09:46:19.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.467+0000 7f39699ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39640ca700 con 0x7f39641577e0 2024-01-11T09:46:19.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.467+0000 7f39699ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39640ca9e0 con 0x7f3964069540 2024-01-11T09:46:19.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.467+0000 7f39627fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3964157410 0x7f3964150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.468+0000 7f39627fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3964157410 0x7f3964150a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60404/0 (socket says 172.21.15.27:60404) 2024-01-11T09:46:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.468+0000 7f39627fc700 1 -- 172.21.15.27:0/3860105529 learned_addr learned my addr 172.21.15.27:0/3860105529 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.468+0000 7f3962ffd700 1 --2- 172.21.15.27:0/3860105529 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3964069540 0x7f396416a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.468+0000 7f39637fe700 1 --2- 172.21.15.27:0/3860105529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39641577e0 0x7f39640c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.468+0000 7f39627fc700 1 -- 172.21.15.27:0/3860105529 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3964069540 msgr2=0x7f396416a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.468+0000 7f39627fc700 1 --2- 172.21.15.27:0/3860105529 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3964069540 0x7f396416a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.468+0000 7f39627fc700 1 -- 172.21.15.27:0/3860105529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39641577e0 msgr2=0x7f39640c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.468+0000 7f39627fc700 1 --2- 172.21.15.27:0/3860105529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39641577e0 0x7f39640c9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.468+0000 7f39627fc700 1 -- 172.21.15.27:0/3860105529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3958009420 con 0x7f3964157410 2024-01-11T09:46:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.468+0000 7f39627fc700 1 --2- 172.21.15.27:0/3860105529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3964157410 0x7f3964150a00 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f3958013000 tx=0x7f3958017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.468+0000 7f394affd700 1 -- 172.21.15.27:0/3860105529 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395800bb20 con 0x7f3964157410 2024-01-11T09:46:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.469+0000 7f394affd700 1 -- 172.21.15.27:0/3860105529 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3958000c90 con 0x7f3964157410 2024-01-11T09:46:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.469+0000 7f394affd700 1 -- 172.21.15.27:0/3860105529 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958011630 con 0x7f3964157410 2024-01-11T09:46:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.469+0000 7f39699ff700 1 -- 172.21.15.27:0/3860105529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3964169910 con 0x7f3964157410 2024-01-11T09:46:19.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.469+0000 7f39699ff700 1 -- 172.21.15.27:0/3860105529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3964150f40 con 0x7f3964157410 2024-01-11T09:46:19.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.470+0000 7f39699ff700 1 -- 172.21.15.27:0/3860105529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3964169e30 con 0x7f3964157410 2024-01-11T09:46:19.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.471+0000 7f394affd700 1 -- 172.21.15.27:0/3860105529 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f39580119a0 con 0x7f3964157410 2024-01-11T09:46:19.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.472+0000 7f394affd700 1 --2- 172.21.15.27:0/3860105529 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3950067170 0x7f3950069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:19.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.472+0000 7f394affd700 1 -- 172.21.15.27:0/3860105529 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f3958085b40 con 0x7f3964157410 2024-01-11T09:46:19.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.472+0000 7f3962ffd700 1 --2- 172.21.15.27:0/3860105529 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3950067170 0x7f3950069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:19.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.473+0000 7f3962ffd700 1 --2- 172.21.15.27:0/3860105529 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3950067170 0x7f3950069620 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3954009da0 tx=0x7f3954009120 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:19.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.476+0000 7f394affd700 1 -- 172.21.15.27:0/3860105529 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3964169e30 con 0x7f3964157410 2024-01-11T09:46:19.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.679+0000 7f39699ff700 1 -- 172.21.15.27:0/3860105529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f396404c3c0 con 0x7f3964157410 2024-01-11T09:46:19.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.683+0000 7f394affd700 1 -- 172.21.15.27:0/3860105529 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v82)=0 v82) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3958052a30 con 0x7f3964157410 2024-01-11T09:46:19.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.686+0000 7f39699ff700 1 -- 172.21.15.27:0/3860105529 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3950067170 msgr2=0x7f3950069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:19.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.686+0000 7f39699ff700 1 --2- 172.21.15.27:0/3860105529 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3950067170 0x7f3950069620 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3954009da0 tx=0x7f3954009120 comp rx=0 tx=0).stop 2024-01-11T09:46:19.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.686+0000 7f39699ff700 1 -- 172.21.15.27:0/3860105529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3964157410 msgr2=0x7f3964150a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:19.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.686+0000 7f39699ff700 1 --2- 172.21.15.27:0/3860105529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3964157410 0x7f3964150a00 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f3958013000 tx=0x7f3958017000 comp rx=0 tx=0).stop 2024-01-11T09:46:19.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.686+0000 7f39699ff700 1 -- 172.21.15.27:0/3860105529 shutdown_connections 2024-01-11T09:46:19.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.686+0000 7f39699ff700 1 --2- 172.21.15.27:0/3860105529 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3950067170 0x7f3950069620 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:19.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.686+0000 7f39699ff700 1 --2- 172.21.15.27:0/3860105529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39641577e0 0x7f39640c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:19.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.686+0000 7f39699ff700 1 --2- 172.21.15.27:0/3860105529 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3964069540 0x7f396416a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:19.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.686+0000 7f39699ff700 1 --2- 172.21.15.27:0/3860105529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3964157410 0x7f3964150a00 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:19.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.686+0000 7f39699ff700 1 -- 172.21.15.27:0/3860105529 >> 172.21.15.27:0/3860105529 conn(0x7f39640680a0 msgr2=0x7f39640c0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:19.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.686+0000 7f39699ff700 1 -- 172.21.15.27:0/3860105529 shutdown_connections 2024-01-11T09:46:19.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:19.686+0000 7f39699ff700 1 -- 172.21.15.27:0/3860105529 wait complete. 2024-01-11T09:46:19.696 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T09:46:19.696 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:46:19.791 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:46:19.884 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T09:46:19.884 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-11T09:46:19.884 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:46:20.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.093+0000 7f68441ff700 1 Processor -- start 2024-01-11T09:46:20.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.093+0000 7f68441ff700 1 -- start start 2024-01-11T09:46:20.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.094+0000 7f68441ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f683c0b03e0 0x7f683c0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:20.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.094+0000 7f68441ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f683c0b07b0 0x7f683c0b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:20.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.094+0000 7f68441ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f683c159200 0x7f683c159620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:20.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.094+0000 7f68441ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f683c0cd730 con 0x7f683c0b07b0 2024-01-11T09:46:20.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.094+0000 7f68441ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f683c0ccb20 con 0x7f683c0b03e0 2024-01-11T09:46:20.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.094+0000 7f68441ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f683c0cce00 con 0x7f683c159200 2024-01-11T09:46:20.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.094+0000 7f684179a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f683c0b07b0 0x7f683c0b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:20.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.094+0000 7f684179a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f683c0b07b0 0x7f683c0b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60410/0 (socket says 172.21.15.27:60410) 2024-01-11T09:46:20.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.094+0000 7f684179a700 1 -- 172.21.15.27:0/493922264 learned_addr learned my addr 172.21.15.27:0/493922264 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:20.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.095+0000 7f684279c700 1 --2- 172.21.15.27:0/493922264 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f683c159200 0x7f683c159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:20.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.095+0000 7f6841f9b700 1 --2- 172.21.15.27:0/493922264 >> v2:172.21.15.77:3300/0 conn(0x7f683c0b03e0 0x7f683c0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:20.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.095+0000 7f684179a700 1 -- 172.21.15.27:0/493922264 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f683c159200 msgr2=0x7f683c159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:20.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.095+0000 7f684179a700 1 --2- 172.21.15.27:0/493922264 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f683c159200 0x7f683c159620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:20.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.095+0000 7f684179a700 1 -- 172.21.15.27:0/493922264 >> v2:172.21.15.77:3300/0 conn(0x7f683c0b03e0 msgr2=0x7f683c0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:20.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.095+0000 7f684179a700 1 --2- 172.21.15.27:0/493922264 >> v2:172.21.15.77:3300/0 conn(0x7f683c0b03e0 0x7f683c0b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:20.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.095+0000 7f684179a700 1 -- 172.21.15.27:0/493922264 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f683c0cd0e0 con 0x7f683c0b07b0 2024-01-11T09:46:20.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.095+0000 7f684179a700 1 --2- 172.21.15.27:0/493922264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0b07b0 0x7f683c0b29c0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f683c089470 tx=0x7f682c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2124b05d5b08595 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:20.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.095+0000 7f6840f99700 1 -- 172.21.15.27:0/493922264 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682c00b640 con 0x7f683c0b07b0 2024-01-11T09:46:20.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.096+0000 7f6840f99700 1 -- 172.21.15.27:0/493922264 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682c00b7a0 con 0x7f683c0b07b0 2024-01-11T09:46:20.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.096+0000 7f6840f99700 1 -- 172.21.15.27:0/493922264 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682c00bb60 con 0x7f683c0b07b0 2024-01-11T09:46:20.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.096+0000 7f68441ff700 1 -- 172.21.15.27:0/493922264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0b07b0 msgr2=0x7f683c0b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:20.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.096+0000 7f68441ff700 1 --2- 172.21.15.27:0/493922264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0b07b0 0x7f683c0b29c0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f683c089470 tx=0x7f682c009250 comp rx=0 tx=0).stop 2024-01-11T09:46:20.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.096+0000 7f68441ff700 1 -- 172.21.15.27:0/493922264 shutdown_connections 2024-01-11T09:46:20.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.096+0000 7f68441ff700 1 --2- 172.21.15.27:0/493922264 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f683c159200 0x7f683c159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:20.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.096+0000 7f68441ff700 1 --2- 172.21.15.27:0/493922264 >> v2:172.21.15.77:3300/0 conn(0x7f683c0b03e0 0x7f683c0b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:20.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.096+0000 7f68441ff700 1 --2- 172.21.15.27:0/493922264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0b07b0 0x7f683c0b29c0 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:20.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.096+0000 7f68441ff700 1 -- 172.21.15.27:0/493922264 >> 172.21.15.27:0/493922264 conn(0x7f683c06ed40 msgr2=0x7f683c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:20.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.096+0000 7f68441ff700 1 -- 172.21.15.27:0/493922264 shutdown_connections 2024-01-11T09:46:20.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.096+0000 7f68441ff700 1 -- 172.21.15.27:0/493922264 wait complete. 2024-01-11T09:46:20.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.098+0000 7f68441ff700 1 Processor -- start 2024-01-11T09:46:20.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.098+0000 7f68441ff700 1 -- start start 2024-01-11T09:46:20.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.099+0000 7f68441ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f683c0b03e0 0x7f683c1578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:20.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.099+0000 7f68441ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0b07b0 0x7f683c159f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:20.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.099+0000 7f68441ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f683c159200 0x7f683c0b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:20.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.099+0000 7f68441ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c15c120 con 0x7f683c0b07b0 2024-01-11T09:46:20.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.099+0000 7f68441ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c15b270 con 0x7f683c0b03e0 2024-01-11T09:46:20.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.099+0000 7f68441ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f683c15b510 con 0x7f683c159200 2024-01-11T09:46:20.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.099+0000 7f684179a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0b07b0 0x7f683c159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:20.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.099+0000 7f684179a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0b07b0 0x7f683c159f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60418/0 (socket says 172.21.15.27:60418) 2024-01-11T09:46:20.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.099+0000 7f684179a700 1 -- 172.21.15.27:0/1301699519 learned_addr learned my addr 172.21.15.27:0/1301699519 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:20.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.099+0000 7f6841f9b700 1 --2- 172.21.15.27:0/1301699519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f683c0b03e0 0x7f683c1578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:20.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.099+0000 7f684279c700 1 --2- 172.21.15.27:0/1301699519 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f683c159200 0x7f683c0b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:20.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.100+0000 7f684179a700 1 -- 172.21.15.27:0/1301699519 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f683c159200 msgr2=0x7f683c0b3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:20.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.100+0000 7f684179a700 1 --2- 172.21.15.27:0/1301699519 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f683c159200 0x7f683c0b3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:20.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.100+0000 7f684179a700 1 -- 172.21.15.27:0/1301699519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f683c0b03e0 msgr2=0x7f683c1578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:20.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.100+0000 7f684179a700 1 --2- 172.21.15.27:0/1301699519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f683c0b03e0 0x7f683c1578c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:20.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.100+0000 7f684179a700 1 -- 172.21.15.27:0/1301699519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682c0094f0 con 0x7f683c0b07b0 2024-01-11T09:46:20.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.100+0000 7f6841f9b700 1 --2- 172.21.15.27:0/1301699519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f683c0b03e0 0x7f683c1578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:20.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.100+0000 7f684179a700 1 --2- 172.21.15.27:0/1301699519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0b07b0 0x7f683c159f20 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f683c069540 tx=0x7f682c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:20.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.100+0000 7f6831ffb700 1 -- 172.21.15.27:0/1301699519 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682c00b910 con 0x7f683c0b07b0 2024-01-11T09:46:20.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.100+0000 7f6831ffb700 1 -- 172.21.15.27:0/1301699519 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682c00ba70 con 0x7f683c0b07b0 2024-01-11T09:46:20.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.100+0000 7f6831ffb700 1 -- 172.21.15.27:0/1301699519 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682c011630 con 0x7f683c0b07b0 2024-01-11T09:46:20.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.100+0000 7f68441ff700 1 -- 172.21.15.27:0/1301699519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f683c15b7f0 con 0x7f683c0b07b0 2024-01-11T09:46:20.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.100+0000 7f68441ff700 1 -- 172.21.15.27:0/1301699519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f683c15bdb0 con 0x7f683c0b07b0 2024-01-11T09:46:20.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.101+0000 7f68441ff700 1 -- 172.21.15.27:0/1301699519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f683c0718d0 con 0x7f683c0b07b0 2024-01-11T09:46:20.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.102+0000 7f6831ffb700 1 -- 172.21.15.27:0/1301699519 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f682c0119a0 con 0x7f683c0b07b0 2024-01-11T09:46:20.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.103+0000 7f6831ffb700 1 --2- 172.21.15.27:0/1301699519 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6820067170 0x7f6820069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:20.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.103+0000 7f6831ffb700 1 -- 172.21.15.27:0/1301699519 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f682c086020 con 0x7f683c0b07b0 2024-01-11T09:46:20.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.103+0000 7f6841f9b700 1 --2- 172.21.15.27:0/1301699519 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6820067170 0x7f6820069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:20.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.104+0000 7f6841f9b700 1 --2- 172.21.15.27:0/1301699519 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6820067170 0x7f6820069620 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f68380113e0 tx=0x7f68380091a0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:20.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.107+0000 7f6831ffb700 1 -- 172.21.15.27:0/1301699519 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f682c052fb0 con 0x7f683c0b07b0 2024-01-11T09:46:20.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.357+0000 7f68441ff700 1 -- 172.21.15.27:0/1301699519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f683c0cd3c0 con 0x7f683c0b07b0 2024-01-11T09:46:20.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.358+0000 7f6831ffb700 1 -- 172.21.15.27:0/1301699519 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3533 (secure 0 0 0) 0x7f682c015de0 con 0x7f683c0b07b0 2024-01-11T09:46:20.350 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:20.350 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2103359888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":2103359888},{"type":"v1","addr":"172.21.15.77:6834","nonce":2103359888}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105},{"gid":8795,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2650824649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2650824649},{"type":"v1","addr":"172.21.15.77:6835","nonce":2650824649}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106},{"gid":11157,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1211538286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1211538286},{"type":"v1","addr":"172.21.15.27:6835","nonce":1211538286}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":119},{"gid":11256,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/438917589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":438917589},{"type":"v1","addr":"172.21.15.27:6834","nonce":438917589}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":121}],"filesystems":[]} 2024-01-11T09:46:20.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.360+0000 7f68441ff700 1 -- 172.21.15.27:0/1301699519 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6820067170 msgr2=0x7f6820069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:20.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.360+0000 7f68441ff700 1 --2- 172.21.15.27:0/1301699519 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6820067170 0x7f6820069620 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f68380113e0 tx=0x7f68380091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:20.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.360+0000 7f68441ff700 1 -- 172.21.15.27:0/1301699519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0b07b0 msgr2=0x7f683c159f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:20.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.360+0000 7f68441ff700 1 --2- 172.21.15.27:0/1301699519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0b07b0 0x7f683c159f20 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f683c069540 tx=0x7f682c017000 comp rx=0 tx=0).stop 2024-01-11T09:46:20.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.360+0000 7f68441ff700 1 -- 172.21.15.27:0/1301699519 shutdown_connections 2024-01-11T09:46:20.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.360+0000 7f68441ff700 1 --2- 172.21.15.27:0/1301699519 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f683c159200 0x7f683c0b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:20.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.360+0000 7f68441ff700 1 --2- 172.21.15.27:0/1301699519 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6820067170 0x7f6820069620 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:20.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.360+0000 7f68441ff700 1 --2- 172.21.15.27:0/1301699519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f683c0b03e0 0x7f683c1578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:20.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.360+0000 7f68441ff700 1 --2- 172.21.15.27:0/1301699519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0b07b0 0x7f683c159f20 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:20.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.360+0000 7f68441ff700 1 -- 172.21.15.27:0/1301699519 >> 172.21.15.27:0/1301699519 conn(0x7f683c06ed40 msgr2=0x7f683c0d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:20.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.361+0000 7f68441ff700 1 -- 172.21.15.27:0/1301699519 shutdown_connections 2024-01-11T09:46:20.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.361+0000 7f68441ff700 1 -- 172.21.15.27:0/1301699519 wait complete. 2024-01-11T09:46:20.354 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-01-11T09:46:20.370 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-11T09:46:20.370 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-11T09:46:20.371 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-11T09:46:20.371 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:46:20.371 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-11T09:46:20.371 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-11T09:46:20.371 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-11T09:46:20.371 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:46:20.372 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-11T09:46:20.372 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-11T09:46:20.372 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-11T09:46:20.372 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:46:20.372 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-11T09:46:20.372 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-11T09:46:20.372 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-11T09:46:20.373 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:46:20.384 INFO:tasks.ceph.mds.a:Stopped 2024-01-11T09:46:20.385 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-11T09:46:20.386 INFO:tasks.ceph.mds.c:Stopped 2024-01-11T09:46:20.386 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-11T09:46:20.391 INFO:tasks.ceph.mds.d:Stopped 2024-01-11T09:46:20.391 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-11T09:46:20.391 INFO:tasks.ceph.mds.b:Stopped 2024-01-11T09:46:20.391 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-11T09:46:20.429 INFO:tasks.ceph.mds.a:Started 2024-01-11T09:46:20.429 DEBUG:teuthology.parallel:result is None 2024-01-11T09:46:20.432 INFO:tasks.ceph.mds.c:Started 2024-01-11T09:46:20.432 DEBUG:teuthology.parallel:result is None 2024-01-11T09:46:20.435 INFO:tasks.ceph.mds.d:Started 2024-01-11T09:46:20.435 DEBUG:teuthology.parallel:result is None 2024-01-11T09:46:20.438 INFO:tasks.ceph.mds.b:Started 2024-01-11T09:46:20.438 DEBUG:teuthology.parallel:result is None 2024-01-11T09:46:20.438 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-11T09:46:20.581 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T09:46:20.581 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T09:46:20.582 INFO:tasks.ceph.mds.d.smithi077.stdout:starting mds.d at 2024-01-11T09:46:20.582 INFO:tasks.ceph.mds.b.smithi077.stdout:starting mds.b at 2024-01-11T09:46:20.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.658+0000 7f2236bff700 1 Processor -- start 2024-01-11T09:46:20.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.658+0000 7f2236bff700 1 -- start start 2024-01-11T09:46:20.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.659+0000 7f2236bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2230155a60 0x7f2230155e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:20.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.659+0000 7f2236bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22300aeb50 con 0x7f22300aef80 2024-01-11T09:46:20.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.659+0000 7f2236bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22300abf20 con 0x7f22300b0360 2024-01-11T09:46:20.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.659+0000 7f2236bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22300ae590 con 0x7f2230155a60 2024-01-11T09:46:20.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.659+0000 7f222ffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f22300aef80 0x7f22300afc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39834/0 (socket says 172.21.15.27:39834) 2024-01-11T09:46:20.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.660+0000 7f222ffff700 1 -- 172.21.15.27:0/1012906467 learned_addr learned my addr 172.21.15.27:0/1012906467 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:20.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.660+0000 7f223499b700 1 --2- 172.21.15.27:0/1012906467 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2230155a60 0x7f2230155e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:20.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.660+0000 7f222f7fe700 1 -- 172.21.15.27:0/1012906467 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084683622 0 0) 0x7f22300aeb50 con 0x7f22300aef80 2024-01-11T09:46:20.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.660+0000 7f222f7fe700 1 -- 172.21.15.27:0/1012906467 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2220003570 con 0x7f22300aef80 2024-01-11T09:46:20.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.660+0000 7f222f7fe700 1 -- 172.21.15.27:0/1012906467 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246233676 0 0) 0x7f22300abf20 con 0x7f22300b0360 2024-01-11T09:46:20.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.660+0000 7f222f7fe700 1 -- 172.21.15.27:0/1012906467 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22300aeb50 con 0x7f22300b0360 2024-01-11T09:46:20.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.660+0000 7f222f7fe700 1 -- 172.21.15.27:0/1012906467 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1087224569 0 0) 0x7f2220003570 con 0x7f22300aef80 2024-01-11T09:46:20.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.660+0000 7f222f7fe700 1 -- 172.21.15.27:0/1012906467 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2230155a60 msgr2=0x7f2230155e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:20.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.660+0000 7f222f7fe700 1 --2- 172.21.15.27:0/1012906467 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2230155a60 0x7f2230155e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:20.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.660+0000 7f222f7fe700 1 -- 172.21.15.27:0/1012906467 >> v1:172.21.15.77:6789/0 conn(0x7f22300b0360 legacy=0x7f22300ac3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:20.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.660+0000 7f222f7fe700 1 -- 172.21.15.27:0/1012906467 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22300ae870 con 0x7f22300aef80 2024-01-11T09:46:20.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.660+0000 7f222f7fe700 1 -- 172.21.15.27:0/1012906467 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f221c001470 con 0x7f22300aef80 2024-01-11T09:46:20.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.660+0000 7f223499b700 1 --2- 172.21.15.27:0/1012906467 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2230155a60 0x7f2230155e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:20.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.660+0000 7f222f7fe700 1 -- 172.21.15.27:0/1012906467 >> v1:172.21.15.27:6789/0 conn(0x7f22300aef80 legacy=0x7f22300afc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:20.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.661+0000 7f222f7fe700 1 --2- 172.21.15.27:0/1012906467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2220003c20 0x7f2220004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:20.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.661+0000 7f222f7fe700 1 --2- 172.21.15.27:0/1012906467 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2230155a60 0x7f2220004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:20.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.661+0000 7f222f7fe700 1 --2- 172.21.15.27:0/1012906467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2220004d50 0x7f22200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.661+0000 7f222f7fe700 1 -- 172.21.15.27:0/1012906467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22300ae590 con 0x7f2220003c20 2024-01-11T09:46:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.661+0000 7f222f7fe700 1 -- 172.21.15.27:0/1012906467 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222000bb70 con 0x7f2220004d50 2024-01-11T09:46:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.661+0000 7f222f7fe700 1 -- 172.21.15.27:0/1012906467 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f222000df40 con 0x7f2230155a60 2024-01-11T09:46:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.661+0000 7f223499b700 1 --2- 172.21.15.27:0/1012906467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2220003c20 0x7f2220004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.661+0000 7f222ffff700 1 --2- 172.21.15.27:0/1012906467 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2230155a60 0x7f2220004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.661+0000 7f223519c700 1 --2- 172.21.15.27:0/1012906467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2220004d50 0x7f22200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.661+0000 7f223499b700 1 -- 172.21.15.27:0/1012906467 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2230155a60 msgr2=0x7f2220004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.661+0000 7f223499b700 1 --2- 172.21.15.27:0/1012906467 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2230155a60 0x7f2220004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.661+0000 7f223499b700 1 -- 172.21.15.27:0/1012906467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2220004d50 msgr2=0x7f22200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.661+0000 7f223499b700 1 --2- 172.21.15.27:0/1012906467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2220004d50 0x7f22200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.661+0000 7f223499b700 1 -- 172.21.15.27:0/1012906467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2224009420 con 0x7f2220003c20 2024-01-11T09:46:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.662+0000 7f223499b700 1 --2- 172.21.15.27:0/1012906467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2220003c20 0x7f2220004010 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f2224009df0 tx=0x7f22240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ea1f2f59a3afee server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:20.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.662+0000 7f222f7fe700 1 -- 172.21.15.27:0/1012906467 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2224009740 con 0x7f2220003c20 2024-01-11T09:46:20.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.662+0000 7f222f7fe700 1 -- 172.21.15.27:0/1012906467 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f222400e920 con 0x7f2220003c20 2024-01-11T09:46:20.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.662+0000 7f222f7fe700 1 -- 172.21.15.27:0/1012906467 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222400ece0 con 0x7f2220003c20 2024-01-11T09:46:20.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.662+0000 7f2236bff700 1 -- 172.21.15.27:0/1012906467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2220003c20 msgr2=0x7f2220004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:20.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.662+0000 7f2236bff700 1 --2- 172.21.15.27:0/1012906467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2220003c20 0x7f2220004010 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f2224009df0 tx=0x7f22240091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:20.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.662+0000 7f223519c700 1 -- 172.21.15.27:0/1012906467 reap_dead start 2024-01-11T09:46:20.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.662+0000 7f2236bff700 1 -- 172.21.15.27:0/1012906467 shutdown_connections 2024-01-11T09:46:20.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.662+0000 7f2236bff700 1 -- 172.21.15.27:0/1012906467 >> 172.21.15.27:0/1012906467 conn(0x7f22300680b0 msgr2=0x7f22300cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:20.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.663+0000 7f2236bff700 1 -- 172.21.15.27:0/1012906467 shutdown_connections 2024-01-11T09:46:20.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.663+0000 7f2236bff700 1 -- 172.21.15.27:0/1012906467 wait complete. 2024-01-11T09:46:20.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.664+0000 7f2236bff700 1 Processor -- start 2024-01-11T09:46:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.664+0000 7f2236bff700 1 -- start start 2024-01-11T09:46:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.665+0000 7f2236bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2220003c20 0x7f22300ca290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.665+0000 7f2236bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2230169770 0x7f223013e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.665+0000 7f2236bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f223006ea60 0x7f223006eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.665+0000 7f2236bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2230141e90 con 0x7f223006ea60 2024-01-11T09:46:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.665+0000 7f2236bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22300cb510 con 0x7f2230169770 2024-01-11T09:46:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.665+0000 7f2236bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f223006f3f0 con 0x7f2220003c20 2024-01-11T09:46:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.666+0000 7f223499b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2220003c20 0x7f22300ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.666+0000 7f223519c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f223006ea60 0x7f223006eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.666+0000 7f222ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2230169770 0x7f223013e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.666+0000 7f222ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2230169770 0x7f223013e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:41242/0 (socket says 172.21.15.27:41242) 2024-01-11T09:46:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.666+0000 7f222ffff700 1 -- 172.21.15.27:0/1658916697 learned_addr learned my addr 172.21.15.27:0/1658916697 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:20.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.666+0000 7f223519c700 1 -- 172.21.15.27:0/1658916697 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2220003c20 msgr2=0x7f22300ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:20.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.666+0000 7f223519c700 1 --2- 172.21.15.27:0/1658916697 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2220003c20 0x7f22300ca290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:20.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.666+0000 7f223519c700 1 -- 172.21.15.27:0/1658916697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2230169770 msgr2=0x7f223013e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:20.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.666+0000 7f223519c700 1 --2- 172.21.15.27:0/1658916697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2230169770 0x7f223013e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:20.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.666+0000 7f223519c700 1 -- 172.21.15.27:0/1658916697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223013eec0 con 0x7f223006ea60 2024-01-11T09:46:20.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.666+0000 7f222ffff700 1 --2- 172.21.15.27:0/1658916697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2230169770 0x7f223013e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:20.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.666+0000 7f223519c700 1 --2- 172.21.15.27:0/1658916697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f223006ea60 0x7f223006eeb0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f2230076d60 tx=0x7f22280017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:20.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.666+0000 7f222cff9700 1 -- 172.21.15.27:0/1658916697 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2228006db0 con 0x7f223006ea60 2024-01-11T09:46:20.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.667+0000 7f222cff9700 1 -- 172.21.15.27:0/1658916697 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f222800c5a0 con 0x7f223006ea60 2024-01-11T09:46:20.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.667+0000 7f2236bff700 1 -- 172.21.15.27:0/1658916697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22300ca900 con 0x7f223006ea60 2024-01-11T09:46:20.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.667+0000 7f222cff9700 1 -- 172.21.15.27:0/1658916697 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222800c960 con 0x7f223006ea60 2024-01-11T09:46:20.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.667+0000 7f2236bff700 1 -- 172.21.15.27:0/1658916697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22300caec0 con 0x7f223006ea60 2024-01-11T09:46:20.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.668+0000 7f2236bff700 1 -- 172.21.15.27:0/1658916697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f223004c3c0 con 0x7f223006ea60 2024-01-11T09:46:20.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.669+0000 7f222cff9700 1 -- 172.21.15.27:0/1658916697 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f222800cd50 con 0x7f223006ea60 2024-01-11T09:46:20.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.669+0000 7f222cff9700 1 --2- 172.21.15.27:0/1658916697 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2214067170 0x7f2214069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:20.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.670+0000 7f222cff9700 1 -- 172.21.15.27:0/1658916697 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f2228083f10 con 0x7f223006ea60 2024-01-11T09:46:20.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.670+0000 7f223499b700 1 --2- 172.21.15.27:0/1658916697 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2214067170 0x7f2214069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:20.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.671+0000 7f223499b700 1 --2- 172.21.15.27:0/1658916697 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2214067170 0x7f2214069620 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f22240091e0 tx=0x7f222401a000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.674+0000 7f222cff9700 1 -- 172.21.15.27:0/1658916697 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2228050f50 con 0x7f223006ea60 2024-01-11T09:46:20.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:20.895+0000 7f2236bff700 1 -- 172.21.15.27:0/1658916697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f22300abbb0 con 0x7f223006ea60 2024-01-11T09:46:21.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.700+0000 7f222cff9700 1 -- 172.21.15.27:0/1658916697 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v231) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2228054570 con 0x7f223006ea60 2024-01-11T09:46:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.703+0000 7f2236bff700 1 -- 172.21.15.27:0/1658916697 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2214067170 msgr2=0x7f2214069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.703+0000 7f2236bff700 1 --2- 172.21.15.27:0/1658916697 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2214067170 0x7f2214069620 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f22240091e0 tx=0x7f222401a000 comp rx=0 tx=0).stop 2024-01-11T09:46:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.703+0000 7f2236bff700 1 -- 172.21.15.27:0/1658916697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f223006ea60 msgr2=0x7f223006eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.703+0000 7f2236bff700 1 --2- 172.21.15.27:0/1658916697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f223006ea60 0x7f223006eeb0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f2230076d60 tx=0x7f22280017a0 comp rx=0 tx=0).stop 2024-01-11T09:46:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.703+0000 7f2236bff700 1 -- 172.21.15.27:0/1658916697 shutdown_connections 2024-01-11T09:46:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.703+0000 7f2236bff700 1 --2- 172.21.15.27:0/1658916697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f223006ea60 0x7f223006eeb0 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.703+0000 7f2236bff700 1 --2- 172.21.15.27:0/1658916697 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2220003c20 0x7f22300ca290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.703+0000 7f2236bff700 1 --2- 172.21.15.27:0/1658916697 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2214067170 0x7f2214069620 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.703+0000 7f2236bff700 1 --2- 172.21.15.27:0/1658916697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2230169770 0x7f223013e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.703+0000 7f2236bff700 1 -- 172.21.15.27:0/1658916697 >> 172.21.15.27:0/1658916697 conn(0x7f22300680b0 msgr2=0x7f2230150570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.703+0000 7f2236bff700 1 -- 172.21.15.27:0/1658916697 shutdown_connections 2024-01-11T09:46:21.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.703+0000 7f2236bff700 1 -- 172.21.15.27:0/1658916697 wait complete. 2024-01-11T09:46:21.696 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-11T09:46:21.715 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-11T09:46:21.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.927+0000 7fc201bff700 1 Processor -- start 2024-01-11T09:46:21.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.927+0000 7fc201bff700 1 -- start start 2024-01-11T09:46:21.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.928+0000 7fc201bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc1fc1596c0 0x7fc1fc15d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:21.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.928+0000 7fc201bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc1fc0ca670 0x7fc1fc068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:21.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.928+0000 7fc201bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc1fc068b60 0x7fc1fc159a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:21.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.928+0000 7fc201bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1fc068fe0 con 0x7fc1fc068b60 2024-01-11T09:46:21.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.928+0000 7fc201bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1fc0a9cb0 con 0x7fc1fc0ca670 2024-01-11T09:46:21.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.928+0000 7fc201bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1fc15e210 con 0x7fc1fc1596c0 2024-01-11T09:46:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.928+0000 7fc1fbfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc1fc068b60 0x7fc1fc159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.929+0000 7fc1fbfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc1fc068b60 0x7fc1fc159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60474/0 (socket says 172.21.15.27:60474) 2024-01-11T09:46:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.929+0000 7fc1fbfff700 1 -- 172.21.15.27:0/3413683512 learned_addr learned my addr 172.21.15.27:0/3413683512 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.929+0000 7fc1faffd700 1 --2- 172.21.15.27:0/3413683512 >> v2:172.21.15.77:3300/0 conn(0x7fc1fc0ca670 0x7fc1fc068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.929+0000 7fc1fb7fe700 1 --2- 172.21.15.27:0/3413683512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc1fc1596c0 0x7fc1fc15d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.929+0000 7fc1fbfff700 1 -- 172.21.15.27:0/3413683512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc1fc1596c0 msgr2=0x7fc1fc15d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.929+0000 7fc1fbfff700 1 --2- 172.21.15.27:0/3413683512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc1fc1596c0 0x7fc1fc15d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.929+0000 7fc1fbfff700 1 -- 172.21.15.27:0/3413683512 >> v2:172.21.15.77:3300/0 conn(0x7fc1fc0ca670 msgr2=0x7fc1fc068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.929+0000 7fc1fbfff700 1 --2- 172.21.15.27:0/3413683512 >> v2:172.21.15.77:3300/0 conn(0x7fc1fc0ca670 0x7fc1fc068620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.929+0000 7fc1fbfff700 1 -- 172.21.15.27:0/3413683512 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1fc15dc50 con 0x7fc1fc068b60 2024-01-11T09:46:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.929+0000 7fc1faffd700 1 --2- 172.21.15.27:0/3413683512 >> v2:172.21.15.77:3300/0 conn(0x7fc1fc0ca670 0x7fc1fc068620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.929+0000 7fc1fbfff700 1 --2- 172.21.15.27:0/3413683512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc068b60 0x7fc1fc159a90 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc0c3980 tx=0x7fc1f4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=145f72143af7cdfa server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.930+0000 7fc1fa7fc700 1 -- 172.21.15.27:0/3413683512 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f400e070 con 0x7fc1fc068b60 2024-01-11T09:46:21.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.930+0000 7fc1fa7fc700 1 -- 172.21.15.27:0/3413683512 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f400ba50 con 0x7fc1fc068b60 2024-01-11T09:46:21.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.930+0000 7fc1fa7fc700 1 -- 172.21.15.27:0/3413683512 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f40092e0 con 0x7fc1fc068b60 2024-01-11T09:46:21.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.930+0000 7fc201bff700 1 -- 172.21.15.27:0/3413683512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc068b60 msgr2=0x7fc1fc159a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:21.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.930+0000 7fc201bff700 1 --2- 172.21.15.27:0/3413683512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc068b60 0x7fc1fc159a90 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc0c3980 tx=0x7fc1f4009840 comp rx=0 tx=0).stop 2024-01-11T09:46:21.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.930+0000 7fc201bff700 1 -- 172.21.15.27:0/3413683512 shutdown_connections 2024-01-11T09:46:21.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.930+0000 7fc201bff700 1 --2- 172.21.15.27:0/3413683512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc1fc1596c0 0x7fc1fc15d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:21.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.930+0000 7fc201bff700 1 --2- 172.21.15.27:0/3413683512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc068b60 0x7fc1fc159a90 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:21.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.930+0000 7fc201bff700 1 --2- 172.21.15.27:0/3413683512 >> v2:172.21.15.77:3300/0 conn(0x7fc1fc0ca670 0x7fc1fc068620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:21.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.930+0000 7fc201bff700 1 -- 172.21.15.27:0/3413683512 >> 172.21.15.27:0/3413683512 conn(0x7fc1fc0c9f70 msgr2=0x7fc1fc06deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:21.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.930+0000 7fc201bff700 1 -- 172.21.15.27:0/3413683512 shutdown_connections 2024-01-11T09:46:21.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.930+0000 7fc201bff700 1 -- 172.21.15.27:0/3413683512 wait complete. 2024-01-11T09:46:21.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.932+0000 7fc201bff700 1 Processor -- start 2024-01-11T09:46:21.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.932+0000 7fc201bff700 1 -- start start 2024-01-11T09:46:21.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.933+0000 7fc201bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc1fc068b60 0x7fc1fc145da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:21.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.933+0000 7fc201bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1fc0ca670 0x7fc1fc0aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:21.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.933+0000 7fc201bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1596c0 0x7fc1fc0c3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:21.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.933+0000 7fc201bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1fc1462e0 con 0x7fc1fc1596c0 2024-01-11T09:46:21.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.933+0000 7fc201bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1fc0c6840 con 0x7fc1fc0ca670 2024-01-11T09:46:21.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.933+0000 7fc201bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1fc0c59b0 con 0x7fc1fc068b60 2024-01-11T09:46:21.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.933+0000 7fc1fbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1596c0 0x7fc1fc0c3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:21.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.933+0000 7fc1fb7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc1fc068b60 0x7fc1fc145da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:21.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.933+0000 7fc1fbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1596c0 0x7fc1fc0c3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60478/0 (socket says 172.21.15.27:60478) 2024-01-11T09:46:21.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.933+0000 7fc1fbfff700 1 -- 172.21.15.27:0/4057388414 learned_addr learned my addr 172.21.15.27:0/4057388414 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:21.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.934+0000 7fc1faffd700 1 --2- 172.21.15.27:0/4057388414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1fc0ca670 0x7fc1fc0aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:21.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.934+0000 7fc1fbfff700 1 -- 172.21.15.27:0/4057388414 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc1fc068b60 msgr2=0x7fc1fc145da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:21.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.934+0000 7fc1fbfff700 1 --2- 172.21.15.27:0/4057388414 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc1fc068b60 0x7fc1fc145da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:21.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.934+0000 7fc1fbfff700 1 -- 172.21.15.27:0/4057388414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1fc0ca670 msgr2=0x7fc1fc0aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:21.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.934+0000 7fc1fbfff700 1 --2- 172.21.15.27:0/4057388414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1fc0ca670 0x7fc1fc0aff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:21.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.934+0000 7fc1fbfff700 1 -- 172.21.15.27:0/4057388414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f4009ae0 con 0x7fc1fc1596c0 2024-01-11T09:46:21.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.934+0000 7fc1fbfff700 1 --2- 172.21.15.27:0/4057388414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1596c0 0x7fc1fc0c3a70 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4009880 tx=0x7fc1f4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:21.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.934+0000 7fc1e37fe700 1 -- 172.21.15.27:0/4057388414 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f40032f0 con 0x7fc1fc1596c0 2024-01-11T09:46:21.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.934+0000 7fc1e37fe700 1 -- 172.21.15.27:0/4057388414 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f4008d20 con 0x7fc1fc1596c0 2024-01-11T09:46:21.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.934+0000 7fc1e37fe700 1 -- 172.21.15.27:0/4057388414 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4026690 con 0x7fc1fc1596c0 2024-01-11T09:46:21.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.935+0000 7fc201bff700 1 -- 172.21.15.27:0/4057388414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1fc0c6cd0 con 0x7fc1fc1596c0 2024-01-11T09:46:21.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.935+0000 7fc201bff700 1 -- 172.21.15.27:0/4057388414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1fc0c5f10 con 0x7fc1fc1596c0 2024-01-11T09:46:21.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.936+0000 7fc201bff700 1 -- 172.21.15.27:0/4057388414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1fc0718d0 con 0x7fc1fc1596c0 2024-01-11T09:46:21.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.937+0000 7fc1e37fe700 1 -- 172.21.15.27:0/4057388414 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fc1f4003040 con 0x7fc1fc1596c0 2024-01-11T09:46:21.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.937+0000 7fc1e37fe700 1 --2- 172.21.15.27:0/4057388414 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc1e8067170 0x7fc1e8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:21.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.938+0000 7fc1e37fe700 1 -- 172.21.15.27:0/4057388414 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 4965+0+0 (secure 0 0 0) 0x7fc1f4086060 con 0x7fc1fc1596c0 2024-01-11T09:46:21.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.938+0000 7fc1fb7fe700 1 --2- 172.21.15.27:0/4057388414 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc1e8067170 0x7fc1e8069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:21.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.939+0000 7fc1fb7fe700 1 --2- 172.21.15.27:0/4057388414 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc1e8067170 0x7fc1e8069620 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc0af9e0 tx=0x7fc1ec008040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:21.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:21.942+0000 7fc1e37fe700 1 -- 172.21.15.27:0/4057388414 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc1f40531f0 con 0x7fc1fc1596c0 2024-01-11T09:46:22.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.225+0000 7fc201bff700 1 -- 172.21.15.27:0/4057388414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc1fc0cc800 con 0x7fc1fc1596c0 2024-01-11T09:46:22.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.226+0000 7fc1e37fe700 1 -- 172.21.15.27:0/4057388414 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v51) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fc1f4056810 con 0x7fc1fc1596c0 2024-01-11T09:46:22.217 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:22.217 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:46:22.217 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-11T09:46:22.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:22.218 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-11T09:46:22.218 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lQm3yLhAAv6erz5hMfat8EUzRA5luMA==", 2024-01-11T09:46:22.218 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:46:22.218 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:46:22.218 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:46:22.218 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:46:22.218 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:46:22.218 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.218 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:22.218 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:22.218 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-11T09:46:22.218 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9l+FG8ChAACHndtVLx7qS3sidPniSycA==", 2024-01-11T09:46:22.218 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:46:22.218 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:46:22.218 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:46:22.219 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:46:22.219 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:46:22.219 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:22.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:22.219 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-11T09:46:22.219 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l1NnEOhAALHXsc5es7hufCthGdRzkyw==", 2024-01-11T09:46:22.219 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:46:22.219 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:46:22.219 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:46:22.219 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:46:22.219 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:46:22.219 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:22.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:22.219 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-11T09:46:22.220 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lruoWFhAAm8AmrZtTJhYI1+19F79RPQ==", 2024-01-11T09:46:22.220 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:46:22.220 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:46:22.220 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:46:22.220 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:46:22.220 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:46:22.220 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.220 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:22.220 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:22.220 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-11T09:46:22.220 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA2op9ltZygORAAZr1X1nCN0ce8rIVaGDpFyQ==", 2024-01-11T09:46:22.220 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:46:22.220 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:46:22.221 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:46:22.221 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:46:22.221 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.221 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:22.221 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:22.221 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-11T09:46:22.221 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA6op9lqSRPAhAAmtyCsyK5ZVGX2wfhOCMuFQ==", 2024-01-11T09:46:22.221 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:46:22.221 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:46:22.221 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:46:22.221 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:46:22.221 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.221 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:22.222 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:22.222 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-11T09:46:22.222 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA9op9lZcs8BhAA8j+U+jPOoy+fliVWZAPFYg==", 2024-01-11T09:46:22.222 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:46:22.222 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:46:22.222 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:46:22.223 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:46:22.223 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:22.223 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:22.223 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-11T09:46:22.223 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBAop9lFuCUChAAP6si/2W1mxH4ecQJvuumkw==", 2024-01-11T09:46:22.223 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:46:22.223 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:46:22.223 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:46:22.223 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:46:22.223 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:22.223 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:22.223 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-11T09:46:22.223 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBGop9lOpwBIRAAcpcsjwmX2IldcKHJWAgHuA==", 2024-01-11T09:46:22.224 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:46:22.224 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:46:22.224 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:46:22.224 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:46:22.224 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:22.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:22.224 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-11T09:46:22.224 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBJop9l4q8hJRAAK8mAYVRQMoWS6wBOU69HmA==", 2024-01-11T09:46:22.224 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:46:22.224 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:46:22.224 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:46:22.224 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:46:22.225 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:22.225 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:22.225 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-11T09:46:22.225 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBMop9lqMF9KBAAFgFFI2GWt9AGif4AoScThw==", 2024-01-11T09:46:22.225 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:46:22.225 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:46:22.225 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:46:22.225 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:46:22.225 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:22.225 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:22.225 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-11T09:46:22.225 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBPop9lN7eDLBAAKswPZFfs0g2AaSe6V+pbbQ==", 2024-01-11T09:46:22.225 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:46:22.226 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:46:22.226 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:46:22.226 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:46:22.226 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.226 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:22.226 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:22.226 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-11T09:46:22.226 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lT5okIBAABy8PbJJbA062VNmb1Y8qhg==", 2024-01-11T09:46:22.226 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:46:22.226 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:46:22.226 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-01-11T09:46:22.226 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-11T09:46:22.226 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.226 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:22.227 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:22.227 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-11T09:46:22.227 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lDf37LhAAqV+EabKXJEW1sS5xtu1tXw==", 2024-01-11T09:46:22.227 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:46:22.227 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:46:22.227 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T09:46:22.227 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T09:46:22.227 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T09:46:22.227 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.227 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:22.227 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:22.227 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-11T09:46:22.227 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lWayxJxAAcalM8QSOGtbCXxjYZGaNQA==", 2024-01-11T09:46:22.227 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:46:22.227 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:46:22.228 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T09:46:22.228 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T09:46:22.228 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T09:46:22.228 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.228 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:22.228 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:22.228 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-11T09:46:22.228 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lHD4tNhAAdmnFBmMrK+QhQYdpbzW0mg==", 2024-01-11T09:46:22.228 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:46:22.228 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:46:22.228 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T09:46:22.229 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T09:46:22.229 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T09:46:22.229 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:22.229 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:22.229 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-11T09:46:22.229 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAwop9lGPtDMxAAJ8FjrSEdCtQL8AhtAyqnTw==", 2024-01-11T09:46:22.229 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:46:22.229 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T09:46:22.229 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:46:22.229 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:46:22.229 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:46:22.229 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:22.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:22.230 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-11T09:46:22.230 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lXBZLJxAAXa1S8Vkm3MiV4igtNbg/1Q==", 2024-01-11T09:46:22.230 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:46:22.230 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T09:46:22.230 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T09:46:22.230 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:46:22.230 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:22.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:22.230 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-11T09:46:22.230 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l3ZGoHxAANqzytTTvWYSomG+RFsFDyg==", 2024-01-11T09:46:22.230 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:46:22.230 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T09:46:22.230 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T09:46:22.231 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:46:22.231 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.231 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:22.231 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-11T09:46:22.231 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:46:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.228+0000 7fc201bff700 1 -- 172.21.15.27:0/4057388414 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc1e8067170 msgr2=0x7fc1e8069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.228+0000 7fc201bff700 1 --2- 172.21.15.27:0/4057388414 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc1e8067170 0x7fc1e8069620 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc0af9e0 tx=0x7fc1ec008040 comp rx=0 tx=0).stop 2024-01-11T09:46:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.228+0000 7fc201bff700 1 -- 172.21.15.27:0/4057388414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1596c0 msgr2=0x7fc1fc0c3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.228+0000 7fc201bff700 1 --2- 172.21.15.27:0/4057388414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1596c0 0x7fc1fc0c3a70 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4009880 tx=0x7fc1f4018000 comp rx=0 tx=0).stop 2024-01-11T09:46:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.228+0000 7fc201bff700 1 -- 172.21.15.27:0/4057388414 shutdown_connections 2024-01-11T09:46:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.228+0000 7fc201bff700 1 --2- 172.21.15.27:0/4057388414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1596c0 0x7fc1fc0c3a70 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.228+0000 7fc201bff700 1 --2- 172.21.15.27:0/4057388414 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc1fc068b60 0x7fc1fc145da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.228+0000 7fc201bff700 1 --2- 172.21.15.27:0/4057388414 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc1e8067170 0x7fc1e8069620 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.228+0000 7fc201bff700 1 --2- 172.21.15.27:0/4057388414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1fc0ca670 0x7fc1fc0aff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.228+0000 7fc201bff700 1 -- 172.21.15.27:0/4057388414 >> 172.21.15.27:0/4057388414 conn(0x7fc1fc0c9f70 msgr2=0x7fc1fc158690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.228+0000 7fc201bff700 1 -- 172.21.15.27:0/4057388414 shutdown_connections 2024-01-11T09:46:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.228+0000 7fc201bff700 1 -- 172.21.15.27:0/4057388414 wait complete. 2024-01-11T09:46:22.238 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:46:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.446+0000 7f7e083ff700 1 Processor -- start 2024-01-11T09:46:22.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.446+0000 7f7e083ff700 1 -- start start 2024-01-11T09:46:22.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.447+0000 7f7e083ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e0015cef0 0x7f7e000ce430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:22.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.447+0000 7f7e083ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7e000cd650 0x7f7e00163530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:22.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.447+0000 7f7e083ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e000cc510 con 0x7f7e000ce970 2024-01-11T09:46:22.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.447+0000 7f7e083ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e000c6350 con 0x7f7e000cd650 2024-01-11T09:46:22.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.447+0000 7f7e083ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0015d2c0 con 0x7f7e0015cef0 2024-01-11T09:46:22.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.447+0000 7f7e0599a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7e000ce970 0x7f7e000ccf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39836/0 (socket says 172.21.15.27:39836) 2024-01-11T09:46:22.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.447+0000 7f7e0599a700 1 -- 172.21.15.27:0/2965001798 learned_addr learned my addr 172.21.15.27:0/2965001798 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:22.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.448+0000 7f7e0619b700 1 --2- 172.21.15.27:0/2965001798 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e0015cef0 0x7f7e000ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:22.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.448+0000 7f7e0699c700 1 --2- 172.21.15.27:0/2965001798 >> v2:172.21.15.77:3300/0 conn(0x7f7e000cd650 0x7f7e00163530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:22.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.448+0000 7f7e05199700 1 -- 172.21.15.27:0/2965001798 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693531483 0 0) 0x7f7e000cc510 con 0x7f7e000ce970 2024-01-11T09:46:22.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.448+0000 7f7e05199700 1 -- 172.21.15.27:0/2965001798 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dec003570 con 0x7f7e000ce970 2024-01-11T09:46:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.448+0000 7f7e0619b700 1 -- 172.21.15.27:0/2965001798 >> v2:172.21.15.77:3300/0 conn(0x7f7e000cd650 msgr2=0x7f7e00163530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.448+0000 7f7e0619b700 1 --2- 172.21.15.27:0/2965001798 >> v2:172.21.15.77:3300/0 conn(0x7f7e000cd650 0x7f7e00163530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.448+0000 7f7e0619b700 1 -- 172.21.15.27:0/2965001798 >> v1:172.21.15.27:6789/0 conn(0x7f7e000ce970 legacy=0x7f7e000ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.448+0000 7f7e0619b700 1 -- 172.21.15.27:0/2965001798 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e000cb900 con 0x7f7e0015cef0 2024-01-11T09:46:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.448+0000 7f7e0619b700 1 --2- 172.21.15.27:0/2965001798 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e0015cef0 0x7f7e000ce430 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc0098d0 tx=0x7f7dfc009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=195ec0b9d2033f71 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.449+0000 7f7e05199700 1 -- 172.21.15.27:0/2965001798 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc005c10 con 0x7f7e0015cef0 2024-01-11T09:46:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.449+0000 7f7e05199700 1 -- 172.21.15.27:0/2965001798 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dfc005d70 con 0x7f7e0015cef0 2024-01-11T09:46:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.449+0000 7f7e05199700 1 -- 172.21.15.27:0/2965001798 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc006280 con 0x7f7e0015cef0 2024-01-11T09:46:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.449+0000 7f7e083ff700 1 -- 172.21.15.27:0/2965001798 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e0015cef0 msgr2=0x7f7e000ce430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.449+0000 7f7e083ff700 1 --2- 172.21.15.27:0/2965001798 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e0015cef0 0x7f7e000ce430 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc0098d0 tx=0x7f7dfc009120 comp rx=0 tx=0).stop 2024-01-11T09:46:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.449+0000 7f7e083ff700 1 -- 172.21.15.27:0/2965001798 shutdown_connections 2024-01-11T09:46:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.449+0000 7f7e083ff700 1 --2- 172.21.15.27:0/2965001798 >> v2:172.21.15.77:3300/0 conn(0x7f7e000cd650 0x7f7e00163530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.449+0000 7f7e083ff700 1 --2- 172.21.15.27:0/2965001798 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e0015cef0 0x7f7e000ce430 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.449+0000 7f7e083ff700 1 -- 172.21.15.27:0/2965001798 >> 172.21.15.27:0/2965001798 conn(0x7f7e00099880 msgr2=0x7f7e0006f520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:22.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.450+0000 7f7e083ff700 1 -- 172.21.15.27:0/2965001798 shutdown_connections 2024-01-11T09:46:22.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.450+0000 7f7e083ff700 1 -- 172.21.15.27:0/2965001798 wait complete. 2024-01-11T09:46:22.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.451+0000 7f7e083ff700 1 Processor -- start 2024-01-11T09:46:22.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.451+0000 7f7e083ff700 1 -- start start 2024-01-11T09:46:22.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.452+0000 7f7e083ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e000cd650 0x7f7e001691f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:22.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.452+0000 7f7e083ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e000ce970 0x7f7e00159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:22.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.452+0000 7f7e083ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e0015cef0 0x7f7e00159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:22.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.452+0000 7f7e083ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e00169c60 con 0x7f7e000ce970 2024-01-11T09:46:22.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.452+0000 7f7e083ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e00169f40 con 0x7f7e000cd650 2024-01-11T09:46:22.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.452+0000 7f7e083ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0016a220 con 0x7f7e0015cef0 2024-01-11T09:46:22.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.452+0000 7f7e0599a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e000ce970 0x7f7e00159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:22.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.452+0000 7f7e0599a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e000ce970 0x7f7e00159ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60490/0 (socket says 172.21.15.27:60490) 2024-01-11T09:46:22.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.452+0000 7f7e0599a700 1 -- 172.21.15.27:0/3478551700 learned_addr learned my addr 172.21.15.27:0/3478551700 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:22.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.452+0000 7f7e0699c700 1 --2- 172.21.15.27:0/3478551700 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e0015cef0 0x7f7e00159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:22.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.453+0000 7f7e0619b700 1 --2- 172.21.15.27:0/3478551700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e000cd650 0x7f7e001691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:22.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.453+0000 7f7e0599a700 1 -- 172.21.15.27:0/3478551700 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e0015cef0 msgr2=0x7f7e00159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:22.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.453+0000 7f7e0599a700 1 --2- 172.21.15.27:0/3478551700 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e0015cef0 0x7f7e00159ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:22.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.453+0000 7f7e0599a700 1 -- 172.21.15.27:0/3478551700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e000cd650 msgr2=0x7f7e001691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:22.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.453+0000 7f7e0599a700 1 --2- 172.21.15.27:0/3478551700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e000cd650 0x7f7e001691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:22.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.453+0000 7f7e0599a700 1 -- 172.21.15.27:0/3478551700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dfc0093c0 con 0x7f7e000ce970 2024-01-11T09:46:22.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.453+0000 7f7e0699c700 1 --2- 172.21.15.27:0/3478551700 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e0015cef0 0x7f7e00159ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:22.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.453+0000 7f7e0599a700 1 --2- 172.21.15.27:0/3478551700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e000ce970 0x7f7e00159ab0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f7df000c590 tx=0x7f7df0007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:22.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.453+0000 7f7df67fc700 1 -- 172.21.15.27:0/3478551700 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df0005b30 con 0x7f7e000ce970 2024-01-11T09:46:22.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.453+0000 7f7df67fc700 1 -- 172.21.15.27:0/3478551700 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df0005cb0 con 0x7f7e000ce970 2024-01-11T09:46:22.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.453+0000 7f7df67fc700 1 -- 172.21.15.27:0/3478551700 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df0007320 con 0x7f7e000ce970 2024-01-11T09:46:22.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.453+0000 7f7e083ff700 1 -- 172.21.15.27:0/3478551700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0016a500 con 0x7f7e000ce970 2024-01-11T09:46:22.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.453+0000 7f7e083ff700 1 -- 172.21.15.27:0/3478551700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e00169730 con 0x7f7e000ce970 2024-01-11T09:46:22.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.454+0000 7f7e083ff700 1 -- 172.21.15.27:0/3478551700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e0016aa20 con 0x7f7e000ce970 2024-01-11T09:46:22.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.455+0000 7f7df67fc700 1 -- 172.21.15.27:0/3478551700 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f7df001b3b0 con 0x7f7e000ce970 2024-01-11T09:46:22.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.457+0000 7f7df67fc700 1 --2- 172.21.15.27:0/3478551700 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7de40671a0 0x7f7de4069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:22.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.457+0000 7f7df67fc700 1 -- 172.21.15.27:0/3478551700 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f7df0083ca0 con 0x7f7e000ce970 2024-01-11T09:46:22.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.457+0000 7f7e0619b700 1 --2- 172.21.15.27:0/3478551700 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7de40671a0 0x7f7de4069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:22.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.458+0000 7f7e0619b700 1 --2- 172.21.15.27:0/3478551700 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7de40671a0 0x7f7de4069650 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7e000b3100 tx=0x7f7dfc016000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:22.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.461+0000 7f7df67fc700 1 -- 172.21.15.27:0/3478551700 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7e0016aa20 con 0x7f7e000ce970 2024-01-11T09:46:22.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.666+0000 7f7e083ff700 1 -- 172.21.15.27:0/3478551700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e0016aa20 con 0x7f7e000ce970 2024-01-11T09:46:22.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.666+0000 7f7df67fc700 1 -- 172.21.15.27:0/3478551700 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v231) v1 ==== 74+0+12275 (secure 0 0 0) 0x7f7e0016aa20 con 0x7f7e000ce970 2024-01-11T09:46:22.658 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:22.658 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":231,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:46:21.696036+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:46:22.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.669+0000 7f7e083ff700 1 -- 172.21.15.27:0/3478551700 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7de40671a0 msgr2=0x7f7de4069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:22.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.669+0000 7f7e083ff700 1 --2- 172.21.15.27:0/3478551700 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7de40671a0 0x7f7de4069650 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7e000b3100 tx=0x7f7dfc016000 comp rx=0 tx=0).stop 2024-01-11T09:46:22.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.669+0000 7f7e083ff700 1 -- 172.21.15.27:0/3478551700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e000ce970 msgr2=0x7f7e00159ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:22.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.669+0000 7f7e083ff700 1 --2- 172.21.15.27:0/3478551700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e000ce970 0x7f7e00159ab0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f7df000c590 tx=0x7f7df0007e00 comp rx=0 tx=0).stop 2024-01-11T09:46:22.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.669+0000 7f7e083ff700 1 -- 172.21.15.27:0/3478551700 shutdown_connections 2024-01-11T09:46:22.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.669+0000 7f7e083ff700 1 --2- 172.21.15.27:0/3478551700 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7e0015cef0 0x7f7e00159ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:22.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.669+0000 7f7e083ff700 1 --2- 172.21.15.27:0/3478551700 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7de40671a0 0x7f7de4069650 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:22.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.669+0000 7f7e083ff700 1 --2- 172.21.15.27:0/3478551700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e000cd650 0x7f7e001691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:22.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.669+0000 7f7e083ff700 1 --2- 172.21.15.27:0/3478551700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e000ce970 0x7f7e00159ab0 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:22.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.669+0000 7f7e083ff700 1 -- 172.21.15.27:0/3478551700 >> 172.21.15.27:0/3478551700 conn(0x7f7e00099880 msgr2=0x7f7e000686b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:22.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.669+0000 7f7e083ff700 1 -- 172.21.15.27:0/3478551700 shutdown_connections 2024-01-11T09:46:22.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.669+0000 7f7e083ff700 1 -- 172.21.15.27:0/3478551700 wait complete. 2024-01-11T09:46:22.679 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-11T09:46:22.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T09:46:22.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.890+0000 7fcf2d1ff700 1 Processor -- start 2024-01-11T09:46:22.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.891+0000 7fcf2d1ff700 1 -- start start 2024-01-11T09:46:22.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.892+0000 7fcf2d1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcf28159ac0 0x7fcf28068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.892+0000 7fcf2d1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf280c80a0 con 0x7fcf281596f0 2024-01-11T09:46:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.892+0000 7fcf2d1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf280cbac0 con 0x7fcf280ca640 2024-01-11T09:46:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.892+0000 7fcf2d1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf280a9fb0 con 0x7fcf28159ac0 2024-01-11T09:46:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.892+0000 7fcf26d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcf281596f0 0x7fcf280a98a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39838/0 (socket says 172.21.15.27:39838) 2024-01-11T09:46:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.892+0000 7fcf26d9d700 1 -- 172.21.15.27:0/2932084902 learned_addr learned my addr 172.21.15.27:0/2932084902 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.892+0000 7fcf2659c700 1 --2- 172.21.15.27:0/2932084902 >> v2:172.21.15.77:3300/0 conn(0x7fcf28159ac0 0x7fcf28068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.893+0000 7fcf25d9b700 1 -- 172.21.15.27:0/2932084902 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516885484 0 0) 0x7fcf280c80a0 con 0x7fcf281596f0 2024-01-11T09:46:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.893+0000 7fcf25d9b700 1 -- 172.21.15.27:0/2932084902 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf10003570 con 0x7fcf281596f0 2024-01-11T09:46:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.893+0000 7fcf25d9b700 1 -- 172.21.15.27:0/2932084902 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687030984 0 0) 0x7fcf280cbac0 con 0x7fcf280ca640 2024-01-11T09:46:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.893+0000 7fcf25d9b700 1 -- 172.21.15.27:0/2932084902 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf280c80a0 con 0x7fcf280ca640 2024-01-11T09:46:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.893+0000 7fcf2659c700 1 -- 172.21.15.27:0/2932084902 >> v1:172.21.15.77:6789/0 conn(0x7fcf280ca640 legacy=0x7fcf280c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.893+0000 7fcf2659c700 1 -- 172.21.15.27:0/2932084902 >> v1:172.21.15.27:6789/0 conn(0x7fcf281596f0 legacy=0x7fcf280a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:22.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.893+0000 7fcf2659c700 1 -- 172.21.15.27:0/2932084902 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf280c7490 con 0x7fcf28159ac0 2024-01-11T09:46:22.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.893+0000 7fcf2659c700 1 --2- 172.21.15.27:0/2932084902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf28159ac0 0x7fcf28068620 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0098e0 tx=0x7fcf1c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bceca1fdb363205 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:22.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.894+0000 7fcf25d9b700 1 -- 172.21.15.27:0/2932084902 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c011630 con 0x7fcf28159ac0 2024-01-11T09:46:22.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.894+0000 7fcf25d9b700 1 -- 172.21.15.27:0/2932084902 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1c011790 con 0x7fcf28159ac0 2024-01-11T09:46:22.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.894+0000 7fcf25d9b700 1 -- 172.21.15.27:0/2932084902 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c011b30 con 0x7fcf28159ac0 2024-01-11T09:46:22.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.894+0000 7fcf2d1ff700 1 -- 172.21.15.27:0/2932084902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf28159ac0 msgr2=0x7fcf28068620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:22.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.894+0000 7fcf2d1ff700 1 --2- 172.21.15.27:0/2932084902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf28159ac0 0x7fcf28068620 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0098e0 tx=0x7fcf1c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:22.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.894+0000 7fcf2d1ff700 1 -- 172.21.15.27:0/2932084902 shutdown_connections 2024-01-11T09:46:22.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.894+0000 7fcf2d1ff700 1 --2- 172.21.15.27:0/2932084902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf28159ac0 0x7fcf28068620 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:22.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.894+0000 7fcf2d1ff700 1 -- 172.21.15.27:0/2932084902 >> 172.21.15.27:0/2932084902 conn(0x7fcf280c9fb0 msgr2=0x7fcf2806ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:22.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.894+0000 7fcf2d1ff700 1 -- 172.21.15.27:0/2932084902 shutdown_connections 2024-01-11T09:46:22.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.894+0000 7fcf2d1ff700 1 -- 172.21.15.27:0/2932084902 wait complete. 2024-01-11T09:46:22.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.896+0000 7fcf2d1ff700 1 Processor -- start 2024-01-11T09:46:22.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.896+0000 7fcf2d1ff700 1 -- start start 2024-01-11T09:46:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.897+0000 7fcf2d1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf280ca640 0x7fcf281561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.897+0000 7fcf2d1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf281596f0 0x7fcf280c5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.897+0000 7fcf2d1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf28159ac0 0x7fcf2815e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.897+0000 7fcf2d1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf280b1870 con 0x7fcf281596f0 2024-01-11T09:46:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.897+0000 7fcf2d1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf280b0c60 con 0x7fcf280ca640 2024-01-11T09:46:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.897+0000 7fcf2d1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf280b0f40 con 0x7fcf28159ac0 2024-01-11T09:46:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.897+0000 7fcf2659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf281596f0 0x7fcf280c5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.897+0000 7fcf2659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf281596f0 0x7fcf280c5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60498/0 (socket says 172.21.15.27:60498) 2024-01-11T09:46:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.897+0000 7fcf2659c700 1 -- 172.21.15.27:0/1695793049 learned_addr learned my addr 172.21.15.27:0/1695793049 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.898+0000 7fcf2759e700 1 --2- 172.21.15.27:0/1695793049 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf28159ac0 0x7fcf2815e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.898+0000 7fcf26d9d700 1 --2- 172.21.15.27:0/1695793049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf280ca640 0x7fcf281561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.898+0000 7fcf2659c700 1 -- 172.21.15.27:0/1695793049 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf28159ac0 msgr2=0x7fcf2815e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.898+0000 7fcf2659c700 1 --2- 172.21.15.27:0/1695793049 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf28159ac0 0x7fcf2815e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.898+0000 7fcf2659c700 1 -- 172.21.15.27:0/1695793049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf280ca640 msgr2=0x7fcf281561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.898+0000 7fcf2659c700 1 --2- 172.21.15.27:0/1695793049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf280ca640 0x7fcf281561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.898+0000 7fcf2659c700 1 -- 172.21.15.27:0/1695793049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c009420 con 0x7fcf281596f0 2024-01-11T09:46:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.898+0000 7fcf2759e700 1 --2- 172.21.15.27:0/1695793049 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf28159ac0 0x7fcf2815e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:46:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.898+0000 7fcf26d9d700 1 --2- 172.21.15.27:0/1695793049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf280ca640 0x7fcf281561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:46:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.898+0000 7fcf2659c700 1 --2- 172.21.15.27:0/1695793049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf281596f0 0x7fcf280c5cf0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fcf2806d730 tx=0x7fcf1c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.898+0000 7fcf0effd700 1 -- 172.21.15.27:0/1695793049 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c01f070 con 0x7fcf281596f0 2024-01-11T09:46:22.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.899+0000 7fcf0effd700 1 -- 172.21.15.27:0/1695793049 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1c000c90 con 0x7fcf281596f0 2024-01-11T09:46:22.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.899+0000 7fcf0effd700 1 -- 172.21.15.27:0/1695793049 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c025420 con 0x7fcf281596f0 2024-01-11T09:46:22.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.899+0000 7fcf2d1ff700 1 -- 172.21.15.27:0/1695793049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf280b1220 con 0x7fcf281596f0 2024-01-11T09:46:22.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.899+0000 7fcf2d1ff700 1 -- 172.21.15.27:0/1695793049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf28156730 con 0x7fcf281596f0 2024-01-11T09:46:22.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.900+0000 7fcf2d1ff700 1 -- 172.21.15.27:0/1695793049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf280c3b00 con 0x7fcf281596f0 2024-01-11T09:46:22.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.901+0000 7fcf0effd700 1 -- 172.21.15.27:0/1695793049 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fcf1c023070 con 0x7fcf281596f0 2024-01-11T09:46:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.902+0000 7fcf0effd700 1 --2- 172.21.15.27:0/1695793049 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcf14067170 0x7fcf14069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.902+0000 7fcf0effd700 1 -- 172.21.15.27:0/1695793049 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 4965+0+0 (secure 0 0 0) 0x7fcf1c085920 con 0x7fcf281596f0 2024-01-11T09:46:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.902+0000 7fcf26d9d700 1 --2- 172.21.15.27:0/1695793049 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcf14067170 0x7fcf14069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.903+0000 7fcf26d9d700 1 --2- 172.21.15.27:0/1695793049 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcf14067170 0x7fcf14069620 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcf280c1ad0 tx=0x7fcf18009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:22.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:22.906+0000 7fcf0effd700 1 -- 172.21.15.27:0/1695793049 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcf1c052be0 con 0x7fcf281596f0 2024-01-11T09:46:23.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.108+0000 7fcf2d1ff700 1 -- 172.21.15.27:0/1695793049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcf2804c3c0 con 0x7fcf281596f0 2024-01-11T09:46:23.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.108+0000 7fcf0effd700 1 -- 172.21.15.27:0/1695793049 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v231) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcf1c02a020 con 0x7fcf281596f0 2024-01-11T09:46:23.100 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T09:46:23.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.110+0000 7fcf2d1ff700 1 -- 172.21.15.27:0/1695793049 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcf14067170 msgr2=0x7fcf14069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:23.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.110+0000 7fcf2d1ff700 1 --2- 172.21.15.27:0/1695793049 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcf14067170 0x7fcf14069620 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcf280c1ad0 tx=0x7fcf18009040 comp rx=0 tx=0).stop 2024-01-11T09:46:23.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.110+0000 7fcf2d1ff700 1 -- 172.21.15.27:0/1695793049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf281596f0 msgr2=0x7fcf280c5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:23.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.110+0000 7fcf2d1ff700 1 --2- 172.21.15.27:0/1695793049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf281596f0 0x7fcf280c5cf0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fcf2806d730 tx=0x7fcf1c017000 comp rx=0 tx=0).stop 2024-01-11T09:46:23.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.110+0000 7fcf2d1ff700 1 -- 172.21.15.27:0/1695793049 shutdown_connections 2024-01-11T09:46:23.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.110+0000 7fcf2d1ff700 1 --2- 172.21.15.27:0/1695793049 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf28159ac0 0x7fcf2815e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:23.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.110+0000 7fcf2d1ff700 1 --2- 172.21.15.27:0/1695793049 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcf14067170 0x7fcf14069620 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:23.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.110+0000 7fcf2d1ff700 1 --2- 172.21.15.27:0/1695793049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf280ca640 0x7fcf281561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:23.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.110+0000 7fcf2d1ff700 1 --2- 172.21.15.27:0/1695793049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf281596f0 0x7fcf280c5cf0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:23.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.110+0000 7fcf2d1ff700 1 -- 172.21.15.27:0/1695793049 >> 172.21.15.27:0/1695793049 conn(0x7fcf280c9fb0 msgr2=0x7fcf280c27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:23.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.111+0000 7fcf2d1ff700 1 -- 172.21.15.27:0/1695793049 shutdown_connections 2024-01-11T09:46:23.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.111+0000 7fcf2d1ff700 1 -- 172.21.15.27:0/1695793049 wait complete. 2024-01-11T09:46:23.121 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-11T09:46:23.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.329+0000 7f5a489ff700 1 Processor -- start 2024-01-11T09:46:23.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.329+0000 7f5a489ff700 1 -- start start 2024-01-11T09:46:23.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.330+0000 7f5a489ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a44159a20 0x7f5a4406bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:23.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.330+0000 7f5a489ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5a440b2730 0x7f5a440b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:23.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.331+0000 7f5a489ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4406fba0 con 0x7f5a44159650 2024-01-11T09:46:23.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.331+0000 7f5a489ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a4406e700 con 0x7f5a440b2730 2024-01-11T09:46:23.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.331+0000 7f5a489ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4406c360 con 0x7f5a44159a20 2024-01-11T09:46:23.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.331+0000 7f5a42d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5a440b2730 0x7f5a440b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:23.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.331+0000 7f5a42d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5a440b2730 0x7f5a440b2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60512/0 (socket says 172.21.15.27:60512) 2024-01-11T09:46:23.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.331+0000 7f5a42d9d700 1 -- 172.21.15.27:0/1370923902 learned_addr learned my addr 172.21.15.27:0/1370923902 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:23.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.331+0000 7f5a41d9b700 1 --2- 172.21.15.27:0/1370923902 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a44159a20 0x7f5a4406bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:23.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.331+0000 7f5a42d9d700 1 -- 172.21.15.27:0/1370923902 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a44159a20 msgr2=0x7f5a4406bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:23.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.331+0000 7f5a42d9d700 1 --2- 172.21.15.27:0/1370923902 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a44159a20 0x7f5a4406bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:23.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.331+0000 7f5a42d9d700 1 -- 172.21.15.27:0/1370923902 >> v1:172.21.15.27:6789/0 conn(0x7f5a44159650 legacy=0x7f5a440b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:23.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.331+0000 7f5a42d9d700 1 -- 172.21.15.27:0/1370923902 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4406c640 con 0x7f5a440b2730 2024-01-11T09:46:23.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.332+0000 7f5a42d9d700 1 --2- 172.21.15.27:0/1370923902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a440b2730 0x7f5a440b2bd0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c009f20 tx=0x7f5a3c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bee4a1902cd9b1f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:23.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.332+0000 7f5a4159a700 1 -- 172.21.15.27:0/1370923902 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c00e070 con 0x7f5a440b2730 2024-01-11T09:46:23.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.332+0000 7f5a4159a700 1 -- 172.21.15.27:0/1370923902 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3c00ba80 con 0x7f5a440b2730 2024-01-11T09:46:23.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.332+0000 7f5a4159a700 1 -- 172.21.15.27:0/1370923902 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c0092e0 con 0x7f5a440b2730 2024-01-11T09:46:23.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.332+0000 7f5a489ff700 1 -- 172.21.15.27:0/1370923902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a440b2730 msgr2=0x7f5a440b2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:23.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.332+0000 7f5a489ff700 1 --2- 172.21.15.27:0/1370923902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a440b2730 0x7f5a440b2bd0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c009f20 tx=0x7f5a3c009840 comp rx=0 tx=0).stop 2024-01-11T09:46:23.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.333+0000 7f5a489ff700 1 -- 172.21.15.27:0/1370923902 shutdown_connections 2024-01-11T09:46:23.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.333+0000 7f5a489ff700 1 --2- 172.21.15.27:0/1370923902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a440b2730 0x7f5a440b2bd0 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:23.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.333+0000 7f5a489ff700 1 --2- 172.21.15.27:0/1370923902 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a44159a20 0x7f5a4406bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:23.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.333+0000 7f5a489ff700 1 -- 172.21.15.27:0/1370923902 >> 172.21.15.27:0/1370923902 conn(0x7f5a4406b200 msgr2=0x7f5a440aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:23.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.333+0000 7f5a489ff700 1 -- 172.21.15.27:0/1370923902 shutdown_connections 2024-01-11T09:46:23.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.333+0000 7f5a489ff700 1 -- 172.21.15.27:0/1370923902 wait complete. 2024-01-11T09:46:23.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.334+0000 7f5a489ff700 1 Processor -- start 2024-01-11T09:46:23.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.334+0000 7f5a489ff700 1 -- start start 2024-01-11T09:46:23.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.335+0000 7f5a489ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a440b2730 0x7f5a4414ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:23.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.335+0000 7f5a489ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a44159650 0x7f5a44150a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:23.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.335+0000 7f5a489ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a44159a20 0x7f5a4415be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:23.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.335+0000 7f5a489ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4414fae0 con 0x7f5a440b2730 2024-01-11T09:46:23.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.335+0000 7f5a489ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a44169bb0 con 0x7f5a44159650 2024-01-11T09:46:23.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.335+0000 7f5a489ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4414ef70 con 0x7f5a44159a20 2024-01-11T09:46:23.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.335+0000 7f5a4259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a440b2730 0x7f5a4414ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:23.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.335+0000 7f5a4259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a440b2730 0x7f5a4414ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60516/0 (socket says 172.21.15.27:60516) 2024-01-11T09:46:23.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.336+0000 7f5a4259c700 1 -- 172.21.15.27:0/870706365 learned_addr learned my addr 172.21.15.27:0/870706365 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:23.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.336+0000 7f5a42d9d700 1 --2- 172.21.15.27:0/870706365 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a44159a20 0x7f5a4415be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:23.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.336+0000 7f5a41d9b700 1 --2- 172.21.15.27:0/870706365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a44159650 0x7f5a44150a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:23.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.336+0000 7f5a4259c700 1 -- 172.21.15.27:0/870706365 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a44159a20 msgr2=0x7f5a4415be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:23.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.336+0000 7f5a4259c700 1 --2- 172.21.15.27:0/870706365 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a44159a20 0x7f5a4415be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:23.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.336+0000 7f5a4259c700 1 -- 172.21.15.27:0/870706365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a44159650 msgr2=0x7f5a44150a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:23.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.336+0000 7f5a4259c700 1 --2- 172.21.15.27:0/870706365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a44159650 0x7f5a44150a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:23.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.336+0000 7f5a4259c700 1 -- 172.21.15.27:0/870706365 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3c009ae0 con 0x7f5a440b2730 2024-01-11T09:46:23.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.336+0000 7f5a4259c700 1 --2- 172.21.15.27:0/870706365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a440b2730 0x7f5a4414ea30 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400c590 tx=0x7f5a3400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:23.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.336+0000 7f5a327fc700 1 -- 172.21.15.27:0/870706365 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a34011590 con 0x7f5a440b2730 2024-01-11T09:46:23.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.336+0000 7f5a327fc700 1 -- 172.21.15.27:0/870706365 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a340116f0 con 0x7f5a440b2730 2024-01-11T09:46:23.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.336+0000 7f5a489ff700 1 -- 172.21.15.27:0/870706365 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a44150fd0 con 0x7f5a440b2730 2024-01-11T09:46:23.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.336+0000 7f5a327fc700 1 -- 172.21.15.27:0/870706365 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a34011ab0 con 0x7f5a440b2730 2024-01-11T09:46:23.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.337+0000 7f5a489ff700 1 -- 172.21.15.27:0/870706365 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4414f2f0 con 0x7f5a440b2730 2024-01-11T09:46:23.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.338+0000 7f5a489ff700 1 -- 172.21.15.27:0/870706365 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a440718d0 con 0x7f5a440b2730 2024-01-11T09:46:23.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.339+0000 7f5a327fc700 1 -- 172.21.15.27:0/870706365 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f5a3401b3b0 con 0x7f5a440b2730 2024-01-11T09:46:23.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.340+0000 7f5a327fc700 1 --2- 172.21.15.27:0/870706365 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5a24067170 0x7f5a24069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:23.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.340+0000 7f5a327fc700 1 -- 172.21.15.27:0/870706365 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f5a34083bb0 con 0x7f5a440b2730 2024-01-11T09:46:23.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.340+0000 7f5a41d9b700 1 --2- 172.21.15.27:0/870706365 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5a24067170 0x7f5a24069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:23.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.341+0000 7f5a41d9b700 1 --2- 172.21.15.27:0/870706365 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5a24067170 0x7f5a24069620 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5a4414fc40 tx=0x7f5a38008040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:23.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.343+0000 7f5a327fc700 1 -- 172.21.15.27:0/870706365 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a34050dd0 con 0x7f5a440b2730 2024-01-11T09:46:23.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.548+0000 7f5a489ff700 1 -- 172.21.15.27:0/870706365 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5a4406c920 con 0x7f5a440b2730 2024-01-11T09:46:23.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.549+0000 7f5a327fc700 1 -- 172.21.15.27:0/870706365 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v231) v1 ==== 84+0+65 (secure 0 0 0) 0x7f5a34017020 con 0x7f5a440b2730 2024-01-11T09:46:23.541 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:23.541 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T09:46:23.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:23.541 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-11T09:46:23.541 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-11T09:46:23.541 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:23.541 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T09:46:23.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.551+0000 7f5a489ff700 1 -- 172.21.15.27:0/870706365 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5a24067170 msgr2=0x7f5a24069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:23.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.551+0000 7f5a489ff700 1 --2- 172.21.15.27:0/870706365 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5a24067170 0x7f5a24069620 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5a4414fc40 tx=0x7f5a38008040 comp rx=0 tx=0).stop 2024-01-11T09:46:23.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.551+0000 7f5a489ff700 1 -- 172.21.15.27:0/870706365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a440b2730 msgr2=0x7f5a4414ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:23.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.551+0000 7f5a489ff700 1 --2- 172.21.15.27:0/870706365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a440b2730 0x7f5a4414ea30 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400c590 tx=0x7f5a3400c950 comp rx=0 tx=0).stop 2024-01-11T09:46:23.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.551+0000 7f5a489ff700 1 -- 172.21.15.27:0/870706365 shutdown_connections 2024-01-11T09:46:23.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.551+0000 7f5a489ff700 1 --2- 172.21.15.27:0/870706365 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5a44159a20 0x7f5a4415be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:23.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.551+0000 7f5a489ff700 1 --2- 172.21.15.27:0/870706365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a440b2730 0x7f5a4414ea30 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:23.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.551+0000 7f5a489ff700 1 --2- 172.21.15.27:0/870706365 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5a24067170 0x7f5a24069620 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:23.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.551+0000 7f5a489ff700 1 --2- 172.21.15.27:0/870706365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a44159650 0x7f5a44150a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:23.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.551+0000 7f5a489ff700 1 -- 172.21.15.27:0/870706365 >> 172.21.15.27:0/870706365 conn(0x7f5a4406b200 msgr2=0x7f5a4406bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:23.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.551+0000 7f5a489ff700 1 -- 172.21.15.27:0/870706365 shutdown_connections 2024-01-11T09:46:23.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.551+0000 7f5a489ff700 1 -- 172.21.15.27:0/870706365 wait complete. 2024-01-11T09:46:23.561 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-11T09:46:23.561 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-11T09:46:23.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.774+0000 7f01a6fff700 1 Processor -- start 2024-01-11T09:46:23.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.774+0000 7f01a6fff700 1 -- start start 2024-01-11T09:46:23.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.775+0000 7f01a6fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f01a0156440 0x7f01a00c7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:23.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.775+0000 7f01a6fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a00c8590 con 0x7f01a006ee30 2024-01-11T09:46:23.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.775+0000 7f01a6fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a00c7980 con 0x7f01a0156810 2024-01-11T09:46:23.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.775+0000 7f01a6fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a00c7c60 con 0x7f01a0156440 2024-01-11T09:46:23.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.775+0000 7f01a4d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f01a0156440 0x7f01a00c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:23.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.775+0000 7f01a4d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f01a0156440 0x7f01a00c7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60520/0 (socket says 172.21.15.27:60520) 2024-01-11T09:46:23.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.775+0000 7f01a4d9b700 1 -- 172.21.15.27:0/3437463209 learned_addr learned my addr 172.21.15.27:0/3437463209 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:23.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.776+0000 7f01a4d9b700 1 -- 172.21.15.27:0/3437463209 >> v1:172.21.15.77:6789/0 conn(0x7f01a0156810 legacy=0x7f01a006e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.776+0000 7f01a4d9b700 1 -- 172.21.15.27:0/3437463209 >> v1:172.21.15.27:6789/0 conn(0x7f01a006ee30 legacy=0x7f01a00accb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.776+0000 7f01a4d9b700 1 -- 172.21.15.27:0/3437463209 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a00c7f40 con 0x7f01a0156440 2024-01-11T09:46:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.776+0000 7f01a4d9b700 1 --2- 172.21.15.27:0/3437463209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a0156440 0x7f01a00c7120 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f0194009990 tx=0x7f0194009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d6278b5423cd94b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.776+0000 7f019f7fe700 1 -- 172.21.15.27:0/3437463209 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0194005c10 con 0x7f01a0156440 2024-01-11T09:46:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.776+0000 7f019f7fe700 1 -- 172.21.15.27:0/3437463209 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0194005d70 con 0x7f01a0156440 2024-01-11T09:46:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.776+0000 7f019f7fe700 1 -- 172.21.15.27:0/3437463209 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019400b860 con 0x7f01a0156440 2024-01-11T09:46:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.777+0000 7f01a6fff700 1 -- 172.21.15.27:0/3437463209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a0156440 msgr2=0x7f01a00c7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.777+0000 7f01a6fff700 1 --2- 172.21.15.27:0/3437463209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a0156440 0x7f01a00c7120 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f0194009990 tx=0x7f0194009220 comp rx=0 tx=0).stop 2024-01-11T09:46:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.777+0000 7f01a6fff700 1 -- 172.21.15.27:0/3437463209 shutdown_connections 2024-01-11T09:46:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.777+0000 7f01a6fff700 1 --2- 172.21.15.27:0/3437463209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a0156440 0x7f01a00c7120 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.777+0000 7f01a6fff700 1 -- 172.21.15.27:0/3437463209 >> 172.21.15.27:0/3437463209 conn(0x7f01a00680e0 msgr2=0x7f01a00cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.777+0000 7f01a6fff700 1 -- 172.21.15.27:0/3437463209 shutdown_connections 2024-01-11T09:46:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.777+0000 7f01a6fff700 1 -- 172.21.15.27:0/3437463209 wait complete. 2024-01-11T09:46:23.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.778+0000 7f01a6fff700 1 Processor -- start 2024-01-11T09:46:23.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.779+0000 7f01a6fff700 1 -- start start 2024-01-11T09:46:23.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.779+0000 7f01a6fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a006ee30 0x7f01a01558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:23.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.779+0000 7f01a6fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01a0156440 0x7f01a00aef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:23.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.779+0000 7f01a6fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01a0156810 0x7f01a00af490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:23.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.780+0000 7f01a6fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a00b1990 con 0x7f01a006ee30 2024-01-11T09:46:23.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.780+0000 7f01a6fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a00c4200 con 0x7f01a0156440 2024-01-11T09:46:23.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.780+0000 7f01a6fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a00c3650 con 0x7f01a0156810 2024-01-11T09:46:23.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.780+0000 7f01a4d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a006ee30 0x7f01a01558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:23.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.780+0000 7f01a4d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a006ee30 0x7f01a01558a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60536/0 (socket says 172.21.15.27:60536) 2024-01-11T09:46:23.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.780+0000 7f01a4d9b700 1 -- 172.21.15.27:0/3227139590 learned_addr learned my addr 172.21.15.27:0/3227139590 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:23.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.780+0000 7f01a4d9b700 1 -- 172.21.15.27:0/3227139590 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01a0156810 msgr2=0x7f01a00af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:23.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.780+0000 7f01a4d9b700 1 --2- 172.21.15.27:0/3227139590 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01a0156810 0x7f01a00af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:23.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.780+0000 7f01a4d9b700 1 -- 172.21.15.27:0/3227139590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01a0156440 msgr2=0x7f01a00aef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:23.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.780+0000 7f01a4d9b700 1 --2- 172.21.15.27:0/3227139590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01a0156440 0x7f01a00aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:23.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.780+0000 7f01a4d9b700 1 -- 172.21.15.27:0/3227139590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0194009480 con 0x7f01a006ee30 2024-01-11T09:46:23.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.780+0000 7f01a4d9b700 1 --2- 172.21.15.27:0/3227139590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a006ee30 0x7f01a01558a0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f01a0069380 tx=0x7f0194016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:23.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.781+0000 7f019cff9700 1 -- 172.21.15.27:0/3227139590 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0194011b60 con 0x7f01a006ee30 2024-01-11T09:46:23.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.781+0000 7f019cff9700 1 -- 172.21.15.27:0/3227139590 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0194011ce0 con 0x7f01a006ee30 2024-01-11T09:46:23.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.781+0000 7f01a6fff700 1 -- 172.21.15.27:0/3227139590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a00c38d0 con 0x7f01a006ee30 2024-01-11T09:46:23.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.781+0000 7f019cff9700 1 -- 172.21.15.27:0/3227139590 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019401b680 con 0x7f01a006ee30 2024-01-11T09:46:23.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.781+0000 7f01a6fff700 1 -- 172.21.15.27:0/3227139590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a00c3e90 con 0x7f01a006ee30 2024-01-11T09:46:23.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.782+0000 7f01a6fff700 1 -- 172.21.15.27:0/3227139590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a004c3c0 con 0x7f01a006ee30 2024-01-11T09:46:23.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.783+0000 7f019cff9700 1 -- 172.21.15.27:0/3227139590 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f019401ba70 con 0x7f01a006ee30 2024-01-11T09:46:23.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.784+0000 7f019cff9700 1 --2- 172.21.15.27:0/3227139590 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0184067170 0x7f0184069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:23.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.784+0000 7f019cff9700 1 -- 172.21.15.27:0/3227139590 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f0194086220 con 0x7f01a006ee30 2024-01-11T09:46:23.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.784+0000 7f019ffff700 1 --2- 172.21.15.27:0/3227139590 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0184067170 0x7f0184069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:23.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.785+0000 7f019ffff700 1 --2- 172.21.15.27:0/3227139590 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0184067170 0x7f0184069620 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f01a00b2200 tx=0x7f018c007b40 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:23.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.788+0000 7f019cff9700 1 -- 172.21.15.27:0/3227139590 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0194053440 con 0x7f01a006ee30 2024-01-11T09:46:23.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:23.992+0000 7f01a6fff700 1 -- 172.21.15.27:0/3227139590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f01a00c8220 con 0x7f01a006ee30 2024-01-11T09:46:24.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.213+0000 7f019cff9700 1 -- 172.21.15.27:0/3227139590 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v232) v1 ==== 138+0+0 (secure 0 0 0) 0x7f019402a0b0 con 0x7f01a006ee30 2024-01-11T09:46:24.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.219+0000 7f01a6fff700 1 -- 172.21.15.27:0/3227139590 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0184067170 msgr2=0x7f0184069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:24.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.219+0000 7f01a6fff700 1 --2- 172.21.15.27:0/3227139590 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0184067170 0x7f0184069620 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f01a00b2200 tx=0x7f018c007b40 comp rx=0 tx=0).stop 2024-01-11T09:46:24.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.219+0000 7f01a6fff700 1 -- 172.21.15.27:0/3227139590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a006ee30 msgr2=0x7f01a01558a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:24.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.219+0000 7f01a6fff700 1 --2- 172.21.15.27:0/3227139590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a006ee30 0x7f01a01558a0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f01a0069380 tx=0x7f0194016000 comp rx=0 tx=0).stop 2024-01-11T09:46:24.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.219+0000 7f01a6fff700 1 -- 172.21.15.27:0/3227139590 shutdown_connections 2024-01-11T09:46:24.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.219+0000 7f01a6fff700 1 --2- 172.21.15.27:0/3227139590 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01a0156810 0x7f01a00af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:24.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.219+0000 7f01a6fff700 1 --2- 172.21.15.27:0/3227139590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01a006ee30 0x7f01a01558a0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:24.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.219+0000 7f01a6fff700 1 --2- 172.21.15.27:0/3227139590 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0184067170 0x7f0184069620 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:24.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.219+0000 7f01a6fff700 1 --2- 172.21.15.27:0/3227139590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01a0156440 0x7f01a00aef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:24.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.219+0000 7f01a6fff700 1 -- 172.21.15.27:0/3227139590 >> 172.21.15.27:0/3227139590 conn(0x7f01a00680e0 msgr2=0x7f01a006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:24.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.220+0000 7f01a6fff700 1 -- 172.21.15.27:0/3227139590 shutdown_connections 2024-01-11T09:46:24.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.221+0000 7f01a6fff700 1 -- 172.21.15.27:0/3227139590 wait complete. 2024-01-11T09:46:24.215 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-11T09:46:24.233 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-11T09:46:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.448+0000 7fa764dff700 1 Processor -- start 2024-01-11T09:46:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.448+0000 7fa764dff700 1 -- start start 2024-01-11T09:46:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.449+0000 7fa764dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa7600b03e0 0x7fa7600b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.449+0000 7fa764dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7600b29c0 0x7fa760159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.449+0000 7fa764dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7600cd730 con 0x7fa7600b07b0 2024-01-11T09:46:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.449+0000 7fa764dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7600b2de0 con 0x7fa7600b03e0 2024-01-11T09:46:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.449+0000 7fa764dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7600ccb20 con 0x7fa7600b29c0 2024-01-11T09:46:24.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.450+0000 7fa75ed9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7600b29c0 0x7fa760159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:24.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.450+0000 7fa75e59c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa7600b03e0 0x7fa7600b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:24.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.450+0000 7fa75ed9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7600b29c0 0x7fa760159910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:47720/0 (socket says 172.21.15.27:47720) 2024-01-11T09:46:24.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.450+0000 7fa75ed9d700 1 -- 172.21.15.27:0/3238731425 learned_addr learned my addr 172.21.15.27:0/3238731425 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:24.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.450+0000 7fa75dd9b700 1 --1- 172.21.15.27:0/3238731425 >> v1:172.21.15.77:6789/0 conn(0x7fa7600b07b0 0x7fa760159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:39708/0 (socket says 172.21.15.27:39708) 2024-01-11T09:46:24.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.450+0000 7fa75ed9d700 1 -- 172.21.15.27:0/3238731425 >> v2:172.21.15.77:3300/0 conn(0x7fa7600b03e0 msgr2=0x7fa7600b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:24.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.450+0000 7fa75ed9d700 1 --2- 172.21.15.27:0/3238731425 >> v2:172.21.15.77:3300/0 conn(0x7fa7600b03e0 0x7fa7600b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:24.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.450+0000 7fa75d59a700 1 -- 172.21.15.27:0/3238731425 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578761331 0 0) 0x7fa7600cd730 con 0x7fa7600b07b0 2024-01-11T09:46:24.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.450+0000 7fa75ed9d700 1 -- 172.21.15.27:0/3238731425 >> v1:172.21.15.77:6789/0 conn(0x7fa7600b07b0 legacy=0x7fa760159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:24.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.450+0000 7fa75ed9d700 1 -- 172.21.15.27:0/3238731425 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7600cce00 con 0x7fa7600b29c0 2024-01-11T09:46:24.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.450+0000 7fa75e59c700 1 --2- 172.21.15.27:0/3238731425 >> v2:172.21.15.77:3300/0 conn(0x7fa7600b03e0 0x7fa7600b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:24.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.450+0000 7fa75ed9d700 1 --2- 172.21.15.27:0/3238731425 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7600b29c0 0x7fa760159910 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa758009f20 tx=0x7fa758009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86e9ca2e220c513a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:24.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.451+0000 7fa75d59a700 1 -- 172.21.15.27:0/3238731425 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75800e070 con 0x7fa7600b29c0 2024-01-11T09:46:24.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.451+0000 7fa75d59a700 1 -- 172.21.15.27:0/3238731425 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75800b9d0 con 0x7fa7600b29c0 2024-01-11T09:46:24.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.451+0000 7fa75d59a700 1 -- 172.21.15.27:0/3238731425 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7580092e0 con 0x7fa7600b29c0 2024-01-11T09:46:24.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.451+0000 7fa764dff700 1 -- 172.21.15.27:0/3238731425 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7600b29c0 msgr2=0x7fa760159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:24.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.451+0000 7fa764dff700 1 --2- 172.21.15.27:0/3238731425 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7600b29c0 0x7fa760159910 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa758009f20 tx=0x7fa758009840 comp rx=0 tx=0).stop 2024-01-11T09:46:24.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.451+0000 7fa764dff700 1 -- 172.21.15.27:0/3238731425 shutdown_connections 2024-01-11T09:46:24.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.451+0000 7fa764dff700 1 --2- 172.21.15.27:0/3238731425 >> v2:172.21.15.77:3300/0 conn(0x7fa7600b03e0 0x7fa7600b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:24.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.451+0000 7fa764dff700 1 --2- 172.21.15.27:0/3238731425 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7600b29c0 0x7fa760159910 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:24.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.451+0000 7fa764dff700 1 -- 172.21.15.27:0/3238731425 >> 172.21.15.27:0/3238731425 conn(0x7fa76006ed40 msgr2=0x7fa76006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:24.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.451+0000 7fa764dff700 1 -- 172.21.15.27:0/3238731425 shutdown_connections 2024-01-11T09:46:24.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.452+0000 7fa764dff700 1 -- 172.21.15.27:0/3238731425 wait complete. 2024-01-11T09:46:24.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.453+0000 7fa764dff700 1 Processor -- start 2024-01-11T09:46:24.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.453+0000 7fa764dff700 1 -- start start 2024-01-11T09:46:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.454+0000 7fa764dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7600b03e0 0x7fa7600c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.454+0000 7fa764dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7600b07b0 0x7fa7600b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.454+0000 7fa764dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7600b29c0 0x7fa7600b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.454+0000 7fa764dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7600a73a0 con 0x7fa7600b29c0 2024-01-11T09:46:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.454+0000 7fa764dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7600c6f00 con 0x7fa7600b03e0 2024-01-11T09:46:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.454+0000 7fa764dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7600d1450 con 0x7fa7600b07b0 2024-01-11T09:46:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.455+0000 7fa75ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7600b29c0 0x7fa7600b49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.455+0000 7fa75ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7600b29c0 0x7fa7600b49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48388/0 (socket says 172.21.15.27:48388) 2024-01-11T09:46:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.455+0000 7fa75ed9d700 1 -- 172.21.15.27:0/1338930566 learned_addr learned my addr 172.21.15.27:0/1338930566 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.455+0000 7fa75e59c700 1 --2- 172.21.15.27:0/1338930566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7600b03e0 0x7fa7600c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.455+0000 7fa75dd9b700 1 --2- 172.21.15.27:0/1338930566 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7600b07b0 0x7fa7600b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.455+0000 7fa75ed9d700 1 -- 172.21.15.27:0/1338930566 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7600b07b0 msgr2=0x7fa7600b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.455+0000 7fa75ed9d700 1 --2- 172.21.15.27:0/1338930566 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7600b07b0 0x7fa7600b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:24.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.455+0000 7fa75ed9d700 1 -- 172.21.15.27:0/1338930566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7600b03e0 msgr2=0x7fa7600c69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:24.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.455+0000 7fa75ed9d700 1 --2- 172.21.15.27:0/1338930566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7600b03e0 0x7fa7600c69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:24.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.455+0000 7fa75ed9d700 1 -- 172.21.15.27:0/1338930566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa750009420 con 0x7fa7600b29c0 2024-01-11T09:46:24.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.455+0000 7fa75ed9d700 1 --2- 172.21.15.27:0/1338930566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7600b29c0 0x7fa7600b49e0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fa75800b610 tx=0x7fa758018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:24.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.456+0000 7fa74e7fc700 1 -- 172.21.15.27:0/1338930566 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa758008ad0 con 0x7fa7600b29c0 2024-01-11T09:46:24.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.456+0000 7fa764dff700 1 -- 172.21.15.27:0/1338930566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa758009ae0 con 0x7fa7600b29c0 2024-01-11T09:46:24.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.456+0000 7fa74e7fc700 1 -- 172.21.15.27:0/1338930566 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa758008c30 con 0x7fa7600b29c0 2024-01-11T09:46:24.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.456+0000 7fa74e7fc700 1 -- 172.21.15.27:0/1338930566 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa758025460 con 0x7fa7600b29c0 2024-01-11T09:46:24.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.456+0000 7fa764dff700 1 -- 172.21.15.27:0/1338930566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7600d0e90 con 0x7fa7600b29c0 2024-01-11T09:46:24.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.457+0000 7fa764dff700 1 -- 172.21.15.27:0/1338930566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7600a7510 con 0x7fa7600b29c0 2024-01-11T09:46:24.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.458+0000 7fa74e7fc700 1 -- 172.21.15.27:0/1338930566 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fa758003040 con 0x7fa7600b29c0 2024-01-11T09:46:24.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.459+0000 7fa74e7fc700 1 --2- 172.21.15.27:0/1338930566 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa740067170 0x7fa740069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:24.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.459+0000 7fa74e7fc700 1 -- 172.21.15.27:0/1338930566 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fa7580868a0 con 0x7fa7600b29c0 2024-01-11T09:46:24.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.460+0000 7fa75e59c700 1 --2- 172.21.15.27:0/1338930566 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa740067170 0x7fa740069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:24.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.460+0000 7fa75e59c700 1 --2- 172.21.15.27:0/1338930566 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa740067170 0x7fa740069620 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa750009f40 tx=0x7fa7500091a0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:24.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.463+0000 7fa74e7fc700 1 -- 172.21.15.27:0/1338930566 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa758053840 con 0x7fa7600b29c0 2024-01-11T09:46:24.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:24.681+0000 7fa764dff700 1 -- 172.21.15.27:0/1338930566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa76004c3c0 con 0x7fa7600b29c0 2024-01-11T09:46:25.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.219+0000 7fa74e7fc700 1 -- 172.21.15.27:0/1338930566 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v233) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa75801a3a0 con 0x7fa7600b29c0 2024-01-11T09:46:25.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.225+0000 7fa764dff700 1 -- 172.21.15.27:0/1338930566 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa740067170 msgr2=0x7fa740069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:25.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.225+0000 7fa764dff700 1 --2- 172.21.15.27:0/1338930566 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa740067170 0x7fa740069620 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa750009f40 tx=0x7fa7500091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.225+0000 7fa764dff700 1 -- 172.21.15.27:0/1338930566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7600b29c0 msgr2=0x7fa7600b49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.225+0000 7fa764dff700 1 --2- 172.21.15.27:0/1338930566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7600b29c0 0x7fa7600b49e0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fa75800b610 tx=0x7fa758018000 comp rx=0 tx=0).stop 2024-01-11T09:46:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.225+0000 7fa764dff700 1 -- 172.21.15.27:0/1338930566 shutdown_connections 2024-01-11T09:46:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.225+0000 7fa764dff700 1 --2- 172.21.15.27:0/1338930566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7600b29c0 0x7fa7600b49e0 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.225+0000 7fa764dff700 1 --2- 172.21.15.27:0/1338930566 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa740067170 0x7fa740069620 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.225+0000 7fa764dff700 1 --2- 172.21.15.27:0/1338930566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7600b03e0 0x7fa7600c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.225+0000 7fa764dff700 1 --2- 172.21.15.27:0/1338930566 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7600b07b0 0x7fa7600b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.225+0000 7fa764dff700 1 -- 172.21.15.27:0/1338930566 >> 172.21.15.27:0/1338930566 conn(0x7fa76006ed40 msgr2=0x7fa760159ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.226+0000 7fa764dff700 1 -- 172.21.15.27:0/1338930566 shutdown_connections 2024-01-11T09:46:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.226+0000 7fa764dff700 1 -- 172.21.15.27:0/1338930566 wait complete. 2024-01-11T09:46:25.227 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-11T09:46:25.249 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-11T09:46:25.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.460+0000 7fab7f5ff700 1 Processor -- start 2024-01-11T09:46:25.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.460+0000 7fab7f5ff700 1 -- start start 2024-01-11T09:46:25.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.462+0000 7fab7f5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fab78068d30 0x7fab780c8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:25.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.462+0000 7fab7f5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab780c80a0 con 0x7fab781596f0 2024-01-11T09:46:25.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.462+0000 7fab7f5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab780cbac0 con 0x7fab78159ac0 2024-01-11T09:46:25.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.462+0000 7fab7f5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab780a9fb0 con 0x7fab78068d30 2024-01-11T09:46:25.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.462+0000 7fab7d39b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fab781596f0 0x7fab780a98a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52360/0 (socket says 172.21.15.27:52360) 2024-01-11T09:46:25.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.462+0000 7fab7d39b700 1 -- 172.21.15.27:0/3519566468 learned_addr learned my addr 172.21.15.27:0/3519566468 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:25.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.462+0000 7fab7db9c700 1 --2- 172.21.15.27:0/3519566468 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fab78068d30 0x7fab780c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:25.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.462+0000 7fab6ffff700 1 -- 172.21.15.27:0/3519566468 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703868133 0 0) 0x7fab780c80a0 con 0x7fab781596f0 2024-01-11T09:46:25.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.463+0000 7fab6ffff700 1 -- 172.21.15.27:0/3519566468 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab64003570 con 0x7fab781596f0 2024-01-11T09:46:25.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.463+0000 7fab6ffff700 1 -- 172.21.15.27:0/3519566468 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782199144 0 0) 0x7fab780cbac0 con 0x7fab78159ac0 2024-01-11T09:46:25.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.463+0000 7fab6ffff700 1 -- 172.21.15.27:0/3519566468 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab780c80a0 con 0x7fab78159ac0 2024-01-11T09:46:25.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.463+0000 7fab7db9c700 1 -- 172.21.15.27:0/3519566468 >> v1:172.21.15.77:6789/0 conn(0x7fab78159ac0 legacy=0x7fab78068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:25.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.463+0000 7fab7db9c700 1 -- 172.21.15.27:0/3519566468 >> v1:172.21.15.27:6789/0 conn(0x7fab781596f0 legacy=0x7fab780a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:25.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.463+0000 7fab7db9c700 1 -- 172.21.15.27:0/3519566468 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab780c7490 con 0x7fab78068d30 2024-01-11T09:46:25.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.463+0000 7fab7db9c700 1 --2- 172.21.15.27:0/3519566468 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fab78068d30 0x7fab780c8530 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fab70009e90 tx=0x7fab70009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95de4a679256ddb0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:25.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.463+0000 7fab6ffff700 1 -- 172.21.15.27:0/3519566468 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7000e070 con 0x7fab78068d30 2024-01-11T09:46:25.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.463+0000 7fab6ffff700 1 -- 172.21.15.27:0/3519566468 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7000bb80 con 0x7fab78068d30 2024-01-11T09:46:25.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.463+0000 7fab6ffff700 1 -- 172.21.15.27:0/3519566468 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab70012430 con 0x7fab78068d30 2024-01-11T09:46:25.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.464+0000 7fab7f5ff700 1 -- 172.21.15.27:0/3519566468 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fab78068d30 msgr2=0x7fab780c8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:25.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.464+0000 7fab7f5ff700 1 --2- 172.21.15.27:0/3519566468 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fab78068d30 0x7fab780c8530 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fab70009e90 tx=0x7fab70009780 comp rx=0 tx=0).stop 2024-01-11T09:46:25.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.464+0000 7fab7f5ff700 1 -- 172.21.15.27:0/3519566468 shutdown_connections 2024-01-11T09:46:25.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.464+0000 7fab7f5ff700 1 --2- 172.21.15.27:0/3519566468 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fab78068d30 0x7fab780c8530 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:25.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.464+0000 7fab7f5ff700 1 -- 172.21.15.27:0/3519566468 >> 172.21.15.27:0/3519566468 conn(0x7fab780c9fb0 msgr2=0x7fab7806ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:25.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.464+0000 7fab7f5ff700 1 -- 172.21.15.27:0/3519566468 shutdown_connections 2024-01-11T09:46:25.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.464+0000 7fab7f5ff700 1 -- 172.21.15.27:0/3519566468 wait complete. 2024-01-11T09:46:25.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.466+0000 7fab7f5ff700 1 Processor -- start 2024-01-11T09:46:25.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.466+0000 7fab7f5ff700 1 -- start start 2024-01-11T09:46:25.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.467+0000 7fab7f5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab78068d30 0x7fab78146400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:25.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.467+0000 7fab7f5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fab781596f0 0x7fab780af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:25.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.467+0000 7fab7f5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab78159ac0 0x7fab780b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:25.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.467+0000 7fab7f5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab780ad5a0 con 0x7fab78159ac0 2024-01-11T09:46:25.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.467+0000 7fab7f5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab780ac040 con 0x7fab78068d30 2024-01-11T09:46:25.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.467+0000 7fab7f5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab780ac320 con 0x7fab781596f0 2024-01-11T09:46:25.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.467+0000 7fab7db9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab78159ac0 0x7fab780b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:25.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.467+0000 7fab7db9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab78159ac0 0x7fab780b0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48390/0 (socket says 172.21.15.27:48390) 2024-01-11T09:46:25.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.467+0000 7fab7db9c700 1 -- 172.21.15.27:0/114583984 learned_addr learned my addr 172.21.15.27:0/114583984 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:25.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.467+0000 7fab7cb9a700 1 --2- 172.21.15.27:0/114583984 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fab781596f0 0x7fab780af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:25.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.467+0000 7fab7d39b700 1 --2- 172.21.15.27:0/114583984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab78068d30 0x7fab78146400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:25.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.467+0000 7fab7db9c700 1 -- 172.21.15.27:0/114583984 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fab781596f0 msgr2=0x7fab780af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:25.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.467+0000 7fab7db9c700 1 --2- 172.21.15.27:0/114583984 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fab781596f0 0x7fab780af8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:25.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.467+0000 7fab7db9c700 1 -- 172.21.15.27:0/114583984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab78068d30 msgr2=0x7fab78146400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:25.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.467+0000 7fab7db9c700 1 --2- 172.21.15.27:0/114583984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab78068d30 0x7fab78146400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:25.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.467+0000 7fab7db9c700 1 -- 172.21.15.27:0/114583984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab70009a20 con 0x7fab78159ac0 2024-01-11T09:46:25.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.468+0000 7fab7db9c700 1 --2- 172.21.15.27:0/114583984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab78159ac0 0x7fab780b0640 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fab70013000 tx=0x7fab70019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:25.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.468+0000 7fab6d7fa700 1 -- 172.21.15.27:0/114583984 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab700263f0 con 0x7fab78159ac0 2024-01-11T09:46:25.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.468+0000 7fab7f5ff700 1 -- 172.21.15.27:0/114583984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab780ac600 con 0x7fab78159ac0 2024-01-11T09:46:25.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.468+0000 7fab6d7fa700 1 -- 172.21.15.27:0/114583984 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab70026570 con 0x7fab78159ac0 2024-01-11T09:46:25.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.468+0000 7fab6d7fa700 1 -- 172.21.15.27:0/114583984 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab70026930 con 0x7fab78159ac0 2024-01-11T09:46:25.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.468+0000 7fab7f5ff700 1 -- 172.21.15.27:0/114583984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab780acbc0 con 0x7fab78159ac0 2024-01-11T09:46:25.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.469+0000 7fab7f5ff700 1 -- 172.21.15.27:0/114583984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab7804c3c0 con 0x7fab78159ac0 2024-01-11T09:46:25.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.470+0000 7fab6d7fa700 1 -- 172.21.15.27:0/114583984 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fab70026ce0 con 0x7fab78159ac0 2024-01-11T09:46:25.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.471+0000 7fab6d7fa700 1 --2- 172.21.15.27:0/114583984 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fab5c067170 0x7fab5c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:25.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.471+0000 7fab6d7fa700 1 -- 172.21.15.27:0/114583984 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5717+0+0 (secure 0 0 0) 0x7fab70086b50 con 0x7fab78159ac0 2024-01-11T09:46:25.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.472+0000 7fab7d39b700 1 --2- 172.21.15.27:0/114583984 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fab5c067170 0x7fab5c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:25.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.473+0000 7fab7d39b700 1 --2- 172.21.15.27:0/114583984 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fab5c067170 0x7fab5c069620 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fab74007fd0 tx=0x7fab74009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.476+0000 7fab6d7fa700 1 -- 172.21.15.27:0/114583984 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fab70053970 con 0x7fab78159ac0 2024-01-11T09:46:25.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:25.724+0000 7fab7f5ff700 1 -- 172.21.15.27:0/114583984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fab780c7770 con 0x7fab78159ac0 2024-01-11T09:46:26.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.235+0000 7fab6d7fa700 1 -- 172.21.15.27:0/114583984 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 30 and data pool 31 v123) v1 ==== 174+0+0 (secure 0 0 0) 0x7fab70056f90 con 0x7fab78159ac0 2024-01-11T09:46:26.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.239+0000 7fab7f5ff700 1 -- 172.21.15.27:0/114583984 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fab5c067170 msgr2=0x7fab5c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:26.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.239+0000 7fab7f5ff700 1 --2- 172.21.15.27:0/114583984 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fab5c067170 0x7fab5c069620 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fab74007fd0 tx=0x7fab74009040 comp rx=0 tx=0).stop 2024-01-11T09:46:26.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.239+0000 7fab7f5ff700 1 -- 172.21.15.27:0/114583984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab78159ac0 msgr2=0x7fab780b0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:26.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.239+0000 7fab7f5ff700 1 --2- 172.21.15.27:0/114583984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab78159ac0 0x7fab780b0640 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fab70013000 tx=0x7fab70019000 comp rx=0 tx=0).stop 2024-01-11T09:46:26.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.239+0000 7fab7f5ff700 1 -- 172.21.15.27:0/114583984 shutdown_connections 2024-01-11T09:46:26.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.239+0000 7fab7f5ff700 1 --2- 172.21.15.27:0/114583984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab78159ac0 0x7fab780b0640 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:26.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.239+0000 7fab7f5ff700 1 --2- 172.21.15.27:0/114583984 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fab5c067170 0x7fab5c069620 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:26.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.239+0000 7fab7f5ff700 1 --2- 172.21.15.27:0/114583984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab78068d30 0x7fab78146400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:26.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.239+0000 7fab7f5ff700 1 --2- 172.21.15.27:0/114583984 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fab781596f0 0x7fab780af8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:26.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.239+0000 7fab7f5ff700 1 -- 172.21.15.27:0/114583984 >> 172.21.15.27:0/114583984 conn(0x7fab780c9fb0 msgr2=0x7fab780684a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:26.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.239+0000 7fab7f5ff700 1 -- 172.21.15.27:0/114583984 shutdown_connections 2024-01-11T09:46:26.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.239+0000 7fab7f5ff700 1 -- 172.21.15.27:0/114583984 wait complete. 2024-01-11T09:46:26.235 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 30 and data pool 31 2024-01-11T09:46:26.255 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:46:26.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.463+0000 7f9e92bff700 1 Processor -- start 2024-01-11T09:46:26.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.463+0000 7f9e92bff700 1 -- start start 2024-01-11T09:46:26.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.464+0000 7f9e92bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e8c0b2b90 0x7f9e8c159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:26.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.464+0000 7f9e92bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9e8c0b07b0 0x7f9e8c159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:26.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.464+0000 7f9e92bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e8c0cd730 con 0x7f9e8c0b03e0 2024-01-11T09:46:26.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.464+0000 7f9e92bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e8c0ccb20 con 0x7f9e8c0b07b0 2024-01-11T09:46:26.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.464+0000 7f9e92bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c0cce00 con 0x7f9e8c0b2b90 2024-01-11T09:46:26.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.464+0000 7f9e9119c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9e8c0b07b0 0x7f9e8c159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:26.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.464+0000 7f9e9119c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9e8c0b07b0 0x7f9e8c159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48404/0 (socket says 172.21.15.27:48404) 2024-01-11T09:46:26.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.464+0000 7f9e9119c700 1 -- 172.21.15.27:0/3958479784 learned_addr learned my addr 172.21.15.27:0/3958479784 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:26.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.464+0000 7f9e8bfff700 1 --2- 172.21.15.27:0/3958479784 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e8c0b2b90 0x7f9e8c159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:26.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.465+0000 7f9e9119c700 1 -- 172.21.15.27:0/3958479784 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e8c0b2b90 msgr2=0x7f9e8c159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:26.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.465+0000 7f9e9119c700 1 --2- 172.21.15.27:0/3958479784 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e8c0b2b90 0x7f9e8c159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:26.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.465+0000 7f9e9119c700 1 -- 172.21.15.27:0/3958479784 >> v1:172.21.15.77:6789/0 conn(0x7f9e8c0b03e0 legacy=0x7f9e8c0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:26.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.465+0000 7f9e9119c700 1 -- 172.21.15.27:0/3958479784 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8c0cd0e0 con 0x7f9e8c0b07b0 2024-01-11T09:46:26.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.465+0000 7f9e8bfff700 1 --2- 172.21.15.27:0/3958479784 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e8c0b2b90 0x7f9e8c159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:26.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.465+0000 7f9e9119c700 1 --2- 172.21.15.27:0/3958479784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e8c0b07b0 0x7f9e8c159740 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f9e84009f20 tx=0x7f9e84009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea2405539d2706f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:26.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.465+0000 7f9e8b7fe700 1 -- 172.21.15.27:0/3958479784 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8400e070 con 0x7f9e8c0b07b0 2024-01-11T09:46:26.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.465+0000 7f9e8b7fe700 1 -- 172.21.15.27:0/3958479784 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8400ba80 con 0x7f9e8c0b07b0 2024-01-11T09:46:26.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.465+0000 7f9e8b7fe700 1 -- 172.21.15.27:0/3958479784 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840092e0 con 0x7f9e8c0b07b0 2024-01-11T09:46:26.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.466+0000 7f9e92bff700 1 -- 172.21.15.27:0/3958479784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e8c0b07b0 msgr2=0x7f9e8c159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:26.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.466+0000 7f9e92bff700 1 --2- 172.21.15.27:0/3958479784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e8c0b07b0 0x7f9e8c159740 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f9e84009f20 tx=0x7f9e84009840 comp rx=0 tx=0).stop 2024-01-11T09:46:26.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.466+0000 7f9e92bff700 1 -- 172.21.15.27:0/3958479784 shutdown_connections 2024-01-11T09:46:26.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.466+0000 7f9e92bff700 1 --2- 172.21.15.27:0/3958479784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e8c0b07b0 0x7f9e8c159740 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:26.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.466+0000 7f9e92bff700 1 --2- 172.21.15.27:0/3958479784 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e8c0b2b90 0x7f9e8c159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:26.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.466+0000 7f9e92bff700 1 -- 172.21.15.27:0/3958479784 >> 172.21.15.27:0/3958479784 conn(0x7f9e8c06ed40 msgr2=0x7f9e8c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:26.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.466+0000 7f9e92bff700 1 -- 172.21.15.27:0/3958479784 shutdown_connections 2024-01-11T09:46:26.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.466+0000 7f9e92bff700 1 -- 172.21.15.27:0/3958479784 wait complete. 2024-01-11T09:46:26.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.467+0000 7f9e92bff700 1 Processor -- start 2024-01-11T09:46:26.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.468+0000 7f9e92bff700 1 -- start start 2024-01-11T09:46:26.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.468+0000 7f9e92bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e8c0b03e0 0x7f9e8c1589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:26.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.469+0000 7f9e92bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e8c0b07b0 0x7f9e8c15c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:26.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.469+0000 7f9e92bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e8c0b2b90 0x7f9e8c15c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:26.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.469+0000 7f9e92bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c16a300 con 0x7f9e8c0b2b90 2024-01-11T09:46:26.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.469+0000 7f9e92bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c16a5e0 con 0x7f9e8c0b03e0 2024-01-11T09:46:26.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.469+0000 7f9e92bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c16a8c0 con 0x7f9e8c0b07b0 2024-01-11T09:46:26.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.469+0000 7f9e9119c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e8c0b2b90 0x7f9e8c15c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:26.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.469+0000 7f9e9119c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e8c0b2b90 0x7f9e8c15c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48414/0 (socket says 172.21.15.27:48414) 2024-01-11T09:46:26.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.469+0000 7f9e9119c700 1 -- 172.21.15.27:0/591199297 learned_addr learned my addr 172.21.15.27:0/591199297 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:26.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.469+0000 7f9e8bfff700 1 --2- 172.21.15.27:0/591199297 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e8c0b07b0 0x7f9e8c15c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:26.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.469+0000 7f9e9099b700 1 --2- 172.21.15.27:0/591199297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e8c0b03e0 0x7f9e8c1589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:26.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.469+0000 7f9e9119c700 1 -- 172.21.15.27:0/591199297 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e8c0b07b0 msgr2=0x7f9e8c15c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:26.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.469+0000 7f9e9119c700 1 --2- 172.21.15.27:0/591199297 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e8c0b07b0 0x7f9e8c15c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:26.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.469+0000 7f9e9119c700 1 -- 172.21.15.27:0/591199297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e8c0b03e0 msgr2=0x7f9e8c1589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.469+0000 7f9e9119c700 1 --2- 172.21.15.27:0/591199297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e8c0b03e0 0x7f9e8c1589b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.469+0000 7f9e9119c700 1 -- 172.21.15.27:0/591199297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e84009ae0 con 0x7f9e8c0b2b90 2024-01-11T09:46:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.469+0000 7f9e8bfff700 1 --2- 172.21.15.27:0/591199297 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e8c0b07b0 0x7f9e8c15c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.469+0000 7f9e9099b700 1 --2- 172.21.15.27:0/591199297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e8c0b03e0 0x7f9e8c1589b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.470+0000 7f9e9119c700 1 --2- 172.21.15.27:0/591199297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e8c0b2b90 0x7f9e8c15c770 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f9e8400b6c0 tx=0x7f9e84018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.470+0000 7f9e88ff9700 1 -- 172.21.15.27:0/591199297 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840032f0 con 0x7f9e8c0b2b90 2024-01-11T09:46:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.470+0000 7f9e88ff9700 1 -- 172.21.15.27:0/591199297 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e84026460 con 0x7f9e8c0b2b90 2024-01-11T09:46:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.470+0000 7f9e88ff9700 1 -- 172.21.15.27:0/591199297 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e84026800 con 0x7f9e8c0b2b90 2024-01-11T09:46:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.470+0000 7f9e92bff700 1 -- 172.21.15.27:0/591199297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e8c16aba0 con 0x7f9e8c0b2b90 2024-01-11T09:46:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.470+0000 7f9e92bff700 1 -- 172.21.15.27:0/591199297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e8c169fb0 con 0x7f9e8c0b2b90 2024-01-11T09:46:26.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.471+0000 7f9e92bff700 1 -- 172.21.15.27:0/591199297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e8c16ade0 con 0x7f9e8c0b2b90 2024-01-11T09:46:26.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.472+0000 7f9e88ff9700 1 -- 172.21.15.27:0/591199297 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f9e84003040 con 0x7f9e8c0b2b90 2024-01-11T09:46:26.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.473+0000 7f9e88ff9700 1 --2- 172.21.15.27:0/591199297 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9e70067170 0x7f9e70069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:26.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.473+0000 7f9e88ff9700 1 -- 172.21.15.27:0/591199297 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f9e84086a40 con 0x7f9e8c0b2b90 2024-01-11T09:46:26.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.473+0000 7f9e9099b700 1 --2- 172.21.15.27:0/591199297 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9e70067170 0x7f9e70069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:26.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.474+0000 7f9e9099b700 1 --2- 172.21.15.27:0/591199297 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9e70067170 0x7f9e70069620 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9e80007fd0 tx=0x7f9e80009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:26.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.477+0000 7f9e88ff9700 1 -- 172.21.15.27:0/591199297 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9e8c16ade0 con 0x7f9e8c0b2b90 2024-01-11T09:46:26.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.683+0000 7f9e92bff700 1 -- 172.21.15.27:0/591199297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e8c16ade0 con 0x7f9e8c0b2b90 2024-01-11T09:46:26.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.684+0000 7f9e88ff9700 1 -- 172.21.15.27:0/591199297 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v234) v1 ==== 74+0+15508 (secure 0 0 0) 0x7f9e8c16ade0 con 0x7f9e8c0b2b90 2024-01-11T09:46:26.675 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:26.676 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":234,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:46:25.725536+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:46:24.049904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-01-11T09:46:24.737495+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:46:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.686+0000 7f9e92bff700 1 -- 172.21.15.27:0/591199297 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9e70067170 msgr2=0x7f9e70069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.686+0000 7f9e92bff700 1 --2- 172.21.15.27:0/591199297 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9e70067170 0x7f9e70069620 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9e80007fd0 tx=0x7f9e80009040 comp rx=0 tx=0).stop 2024-01-11T09:46:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.686+0000 7f9e92bff700 1 -- 172.21.15.27:0/591199297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e8c0b2b90 msgr2=0x7f9e8c15c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.686+0000 7f9e92bff700 1 --2- 172.21.15.27:0/591199297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e8c0b2b90 0x7f9e8c15c770 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f9e8400b6c0 tx=0x7f9e84018000 comp rx=0 tx=0).stop 2024-01-11T09:46:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.686+0000 7f9e92bff700 1 -- 172.21.15.27:0/591199297 shutdown_connections 2024-01-11T09:46:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.686+0000 7f9e92bff700 1 --2- 172.21.15.27:0/591199297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e8c0b2b90 0x7f9e8c15c770 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.686+0000 7f9e92bff700 1 --2- 172.21.15.27:0/591199297 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9e70067170 0x7f9e70069620 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.686+0000 7f9e92bff700 1 --2- 172.21.15.27:0/591199297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e8c0b03e0 0x7f9e8c1589b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.686+0000 7f9e92bff700 1 --2- 172.21.15.27:0/591199297 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9e8c0b07b0 0x7f9e8c15c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.686+0000 7f9e92bff700 1 -- 172.21.15.27:0/591199297 >> 172.21.15.27:0/591199297 conn(0x7f9e8c06ed40 msgr2=0x7f9e8c0cb400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.686+0000 7f9e92bff700 1 -- 172.21.15.27:0/591199297 shutdown_connections 2024-01-11T09:46:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.686+0000 7f9e92bff700 1 -- 172.21.15.27:0/591199297 wait complete. 2024-01-11T09:46:26.696 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:46:26.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.904+0000 7fcb38fff700 1 Processor -- start 2024-01-11T09:46:26.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.905+0000 7fcb38fff700 1 -- start start 2024-01-11T09:46:26.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.906+0000 7fcb38fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcb340ca3e0 0x7fcb340cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:26.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.906+0000 7fcb38fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb340ca7b0 0x7fcb34159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.906+0000 7fcb38fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb3415d640 con 0x7fcb340c8a20 2024-01-11T09:46:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.906+0000 7fcb38fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb3415ae70 con 0x7fcb340ca3e0 2024-01-11T09:46:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.906+0000 7fcb38fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3415a3c0 con 0x7fcb340ca7b0 2024-01-11T09:46:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.906+0000 7fcb32d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcb340c8a20 0x7fcb3415dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52372/0 (socket says 172.21.15.27:52372) 2024-01-11T09:46:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.906+0000 7fcb32d9d700 1 -- 172.21.15.27:0/4291631171 learned_addr learned my addr 172.21.15.27:0/4291631171 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.906+0000 7fcb31d9b700 1 --2- 172.21.15.27:0/4291631171 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb340ca7b0 0x7fcb34159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.906+0000 7fcb3259c700 1 --2- 172.21.15.27:0/4291631171 >> v2:172.21.15.77:3300/0 conn(0x7fcb340ca3e0 0x7fcb340cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.907+0000 7fcb3159a700 1 -- 172.21.15.27:0/4291631171 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717393458 0 0) 0x7fcb3415d640 con 0x7fcb340c8a20 2024-01-11T09:46:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.907+0000 7fcb3159a700 1 -- 172.21.15.27:0/4291631171 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb1c003570 con 0x7fcb340c8a20 2024-01-11T09:46:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.907+0000 7fcb3259c700 1 -- 172.21.15.27:0/4291631171 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb340ca7b0 msgr2=0x7fcb34159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.907+0000 7fcb3259c700 1 --2- 172.21.15.27:0/4291631171 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb340ca7b0 0x7fcb34159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.907+0000 7fcb3259c700 1 -- 172.21.15.27:0/4291631171 >> v1:172.21.15.27:6789/0 conn(0x7fcb340c8a20 legacy=0x7fcb3415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.907+0000 7fcb3259c700 1 -- 172.21.15.27:0/4291631171 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3415a5d0 con 0x7fcb340ca3e0 2024-01-11T09:46:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.907+0000 7fcb3259c700 1 --2- 172.21.15.27:0/4291631171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb340ca3e0 0x7fcb340cca20 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fcb240098e0 tx=0x7fcb240091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63a81deed5eb60e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.908+0000 7fcb3159a700 1 -- 172.21.15.27:0/4291631171 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb24011630 con 0x7fcb340ca3e0 2024-01-11T09:46:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.908+0000 7fcb3159a700 1 -- 172.21.15.27:0/4291631171 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb24011790 con 0x7fcb340ca3e0 2024-01-11T09:46:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.908+0000 7fcb3159a700 1 -- 172.21.15.27:0/4291631171 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb24011b30 con 0x7fcb340ca3e0 2024-01-11T09:46:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.908+0000 7fcb38fff700 1 -- 172.21.15.27:0/4291631171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb340ca3e0 msgr2=0x7fcb340cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.908+0000 7fcb38fff700 1 --2- 172.21.15.27:0/4291631171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb340ca3e0 0x7fcb340cca20 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fcb240098e0 tx=0x7fcb240091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.908+0000 7fcb38fff700 1 -- 172.21.15.27:0/4291631171 shutdown_connections 2024-01-11T09:46:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.908+0000 7fcb38fff700 1 --2- 172.21.15.27:0/4291631171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb340ca3e0 0x7fcb340cca20 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.908+0000 7fcb38fff700 1 --2- 172.21.15.27:0/4291631171 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb340ca7b0 0x7fcb34159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.908+0000 7fcb38fff700 1 -- 172.21.15.27:0/4291631171 >> 172.21.15.27:0/4291631171 conn(0x7fcb340ce3e0 msgr2=0x7fcb340ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.908+0000 7fcb38fff700 1 -- 172.21.15.27:0/4291631171 shutdown_connections 2024-01-11T09:46:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.908+0000 7fcb38fff700 1 -- 172.21.15.27:0/4291631171 wait complete. 2024-01-11T09:46:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.910+0000 7fcb38fff700 1 Processor -- start 2024-01-11T09:46:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.910+0000 7fcb38fff700 1 -- start start 2024-01-11T09:46:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.911+0000 7fcb38fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb340c8a20 0x7fcb34150ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.911+0000 7fcb38fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb340ca3e0 0x7fcb3415e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.911+0000 7fcb38fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb340ca7b0 0x7fcb3415ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.911+0000 7fcb38fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb34160800 con 0x7fcb340c8a20 2024-01-11T09:46:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.911+0000 7fcb38fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb34160370 con 0x7fcb340ca3e0 2024-01-11T09:46:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.911+0000 7fcb38fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3415f760 con 0x7fcb340ca7b0 2024-01-11T09:46:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.911+0000 7fcb3259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb340c8a20 0x7fcb34150ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.911+0000 7fcb3259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb340c8a20 0x7fcb34150ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48420/0 (socket says 172.21.15.27:48420) 2024-01-11T09:46:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.911+0000 7fcb3259c700 1 -- 172.21.15.27:0/942502177 learned_addr learned my addr 172.21.15.27:0/942502177 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.911+0000 7fcb32d9d700 1 --2- 172.21.15.27:0/942502177 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb340ca7b0 0x7fcb3415ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.911+0000 7fcb31d9b700 1 --2- 172.21.15.27:0/942502177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb340ca3e0 0x7fcb3415e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.912+0000 7fcb3259c700 1 -- 172.21.15.27:0/942502177 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb340ca7b0 msgr2=0x7fcb3415ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.912+0000 7fcb3259c700 1 --2- 172.21.15.27:0/942502177 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb340ca7b0 0x7fcb3415ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.912+0000 7fcb3259c700 1 -- 172.21.15.27:0/942502177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb340ca3e0 msgr2=0x7fcb3415e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.912+0000 7fcb3259c700 1 --2- 172.21.15.27:0/942502177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb340ca3e0 0x7fcb3415e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.912+0000 7fcb3259c700 1 -- 172.21.15.27:0/942502177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb24009420 con 0x7fcb340c8a20 2024-01-11T09:46:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.912+0000 7fcb3259c700 1 --2- 172.21.15.27:0/942502177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb340c8a20 0x7fcb34150ce0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7fcb34064980 tx=0x7fcb24017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.912+0000 7fcb227fc700 1 -- 172.21.15.27:0/942502177 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb24000d10 con 0x7fcb340c8a20 2024-01-11T09:46:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.912+0000 7fcb227fc700 1 -- 172.21.15.27:0/942502177 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb24011b10 con 0x7fcb340c8a20 2024-01-11T09:46:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.912+0000 7fcb227fc700 1 -- 172.21.15.27:0/942502177 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb24025660 con 0x7fcb340c8a20 2024-01-11T09:46:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.912+0000 7fcb38fff700 1 -- 172.21.15.27:0/942502177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3415fa40 con 0x7fcb340c8a20 2024-01-11T09:46:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.912+0000 7fcb38fff700 1 -- 172.21.15.27:0/942502177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb34160000 con 0x7fcb340c8a20 2024-01-11T09:46:26.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.914+0000 7fcb38fff700 1 -- 172.21.15.27:0/942502177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb340718d0 con 0x7fcb340c8a20 2024-01-11T09:46:26.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.914+0000 7fcb227fc700 1 -- 172.21.15.27:0/942502177 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fcb24025a50 con 0x7fcb340c8a20 2024-01-11T09:46:26.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.916+0000 7fcb227fc700 1 --2- 172.21.15.27:0/942502177 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcb14067170 0x7fcb14069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:26.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.916+0000 7fcb227fc700 1 -- 172.21.15.27:0/942502177 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5817+0+0 (secure 0 0 0) 0x7fcb240860c0 con 0x7fcb340c8a20 2024-01-11T09:46:26.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.916+0000 7fcb31d9b700 1 --2- 172.21.15.27:0/942502177 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcb14067170 0x7fcb14069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:26.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.917+0000 7fcb31d9b700 1 --2- 172.21.15.27:0/942502177 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcb14067170 0x7fcb14069620 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcb3416d4d0 tx=0x7fcb280091a0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:26.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:26.919+0000 7fcb227fc700 1 -- 172.21.15.27:0/942502177 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcb24053030 con 0x7fcb340c8a20 2024-01-11T09:46:27.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.127+0000 7fcb38fff700 1 -- 172.21.15.27:0/942502177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb3406c840 con 0x7fcb340c8a20 2024-01-11T09:46:27.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.128+0000 7fcb227fc700 1 -- 172.21.15.27:0/942502177 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v235) v1 ==== 74+0+15495 (secure 0 0 0) 0x7fcb24056650 con 0x7fcb340c8a20 2024-01-11T09:46:27.119 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:27.120 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":235,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:46:26.932357+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:46:24.049904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-01-11T09:46:24.737495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:46:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.130+0000 7fcb38fff700 1 -- 172.21.15.27:0/942502177 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcb14067170 msgr2=0x7fcb14069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.130+0000 7fcb38fff700 1 --2- 172.21.15.27:0/942502177 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcb14067170 0x7fcb14069620 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcb3416d4d0 tx=0x7fcb280091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.130+0000 7fcb38fff700 1 -- 172.21.15.27:0/942502177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb340c8a20 msgr2=0x7fcb34150ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.130+0000 7fcb38fff700 1 --2- 172.21.15.27:0/942502177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb340c8a20 0x7fcb34150ce0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7fcb34064980 tx=0x7fcb24017000 comp rx=0 tx=0).stop 2024-01-11T09:46:27.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.130+0000 7fcb38fff700 1 -- 172.21.15.27:0/942502177 shutdown_connections 2024-01-11T09:46:27.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.130+0000 7fcb38fff700 1 --2- 172.21.15.27:0/942502177 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcb340ca7b0 0x7fcb3415ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:27.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.130+0000 7fcb38fff700 1 --2- 172.21.15.27:0/942502177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb340c8a20 0x7fcb34150ce0 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:27.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.130+0000 7fcb38fff700 1 --2- 172.21.15.27:0/942502177 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcb14067170 0x7fcb14069620 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:27.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.130+0000 7fcb38fff700 1 --2- 172.21.15.27:0/942502177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb340ca3e0 0x7fcb3415e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:27.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.131+0000 7fcb38fff700 1 -- 172.21.15.27:0/942502177 >> 172.21.15.27:0/942502177 conn(0x7fcb340ce3e0 msgr2=0x7fcb340688c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:27.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.131+0000 7fcb38fff700 1 -- 172.21.15.27:0/942502177 shutdown_connections 2024-01-11T09:46:27.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.131+0000 7fcb38fff700 1 -- 172.21.15.27:0/942502177 wait complete. 2024-01-11T09:46:27.140 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-11T09:46:27.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.353+0000 7f5ea09ff700 1 Processor -- start 2024-01-11T09:46:27.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.354+0000 7f5ea09ff700 1 -- start start 2024-01-11T09:46:27.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.355+0000 7f5ea09ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e9c0cb4b0 0x7f5e9c0cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:27.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.355+0000 7f5ea09ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9c0ccc70 con 0x7f5e9c142dc0 2024-01-11T09:46:27.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.355+0000 7f5ea09ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9c15d320 con 0x7f5e9c143190 2024-01-11T09:46:27.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.355+0000 7f5ea09ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c15df30 con 0x7f5e9c0cb4b0 2024-01-11T09:46:27.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.355+0000 7f5e9a59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5e9c142dc0 0x7f5e9c0cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52378/0 (socket says 172.21.15.27:52378) 2024-01-11T09:46:27.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.355+0000 7f5e9a59c700 1 -- 172.21.15.27:0/232664621 learned_addr learned my addr 172.21.15.27:0/232664621 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:27.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.355+0000 7f5e9ad9d700 1 --2- 172.21.15.27:0/232664621 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e9c0cb4b0 0x7f5e9c0cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:27.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.356+0000 7f5e9959a700 1 -- 172.21.15.27:0/232664621 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831773053 0 0) 0x7f5e9c0ccc70 con 0x7f5e9c142dc0 2024-01-11T09:46:27.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.356+0000 7f5e9959a700 1 -- 172.21.15.27:0/232664621 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e84003570 con 0x7f5e9c142dc0 2024-01-11T09:46:27.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.356+0000 7f5e9959a700 1 -- 172.21.15.27:0/232664621 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929936225 0 0) 0x7f5e9c15d320 con 0x7f5e9c143190 2024-01-11T09:46:27.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.356+0000 7f5e9959a700 1 -- 172.21.15.27:0/232664621 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e9c0ccc70 con 0x7f5e9c143190 2024-01-11T09:46:27.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.356+0000 7f5e9ad9d700 1 -- 172.21.15.27:0/232664621 >> v1:172.21.15.77:6789/0 conn(0x7f5e9c143190 legacy=0x7f5e9c0cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:27.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.356+0000 7f5e9ad9d700 1 -- 172.21.15.27:0/232664621 >> v1:172.21.15.27:6789/0 conn(0x7f5e9c142dc0 legacy=0x7f5e9c0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:27.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.356+0000 7f5e9ad9d700 1 -- 172.21.15.27:0/232664621 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9c15d590 con 0x7f5e9c0cb4b0 2024-01-11T09:46:27.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.356+0000 7f5e9ad9d700 1 --2- 172.21.15.27:0/232664621 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e9c0cb4b0 0x7f5e9c0cc730 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009e90 tx=0x7f5e94009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2350c40e09ac7d92 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.357+0000 7f5e9959a700 1 -- 172.21.15.27:0/232664621 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9400e070 con 0x7f5e9c0cb4b0 2024-01-11T09:46:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.357+0000 7f5e9959a700 1 -- 172.21.15.27:0/232664621 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9400bb80 con 0x7f5e9c0cb4b0 2024-01-11T09:46:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.357+0000 7f5e9959a700 1 -- 172.21.15.27:0/232664621 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e94012430 con 0x7f5e9c0cb4b0 2024-01-11T09:46:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.357+0000 7f5ea09ff700 1 -- 172.21.15.27:0/232664621 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e9c0cb4b0 msgr2=0x7f5e9c0cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.357+0000 7f5ea09ff700 1 --2- 172.21.15.27:0/232664621 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e9c0cb4b0 0x7f5e9c0cc730 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009e90 tx=0x7f5e94009780 comp rx=0 tx=0).stop 2024-01-11T09:46:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.357+0000 7f5ea09ff700 1 -- 172.21.15.27:0/232664621 shutdown_connections 2024-01-11T09:46:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.357+0000 7f5ea09ff700 1 --2- 172.21.15.27:0/232664621 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e9c0cb4b0 0x7f5e9c0cc730 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.357+0000 7f5ea09ff700 1 -- 172.21.15.27:0/232664621 >> 172.21.15.27:0/232664621 conn(0x7f5e9c063ca0 msgr2=0x7f5e9c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.357+0000 7f5ea09ff700 1 -- 172.21.15.27:0/232664621 shutdown_connections 2024-01-11T09:46:27.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.357+0000 7f5ea09ff700 1 -- 172.21.15.27:0/232664621 wait complete. 2024-01-11T09:46:27.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.359+0000 7f5ea09ff700 1 Processor -- start 2024-01-11T09:46:27.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.359+0000 7f5ea09ff700 1 -- start start 2024-01-11T09:46:27.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.360+0000 7f5ea09ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e9c0cb4b0 0x7f5e9c158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:27.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.360+0000 7f5ea09ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e9c142dc0 0x7f5e9c141e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:27.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.360+0000 7f5ea09ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e9c143190 0x7f5e9c0c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:27.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.360+0000 7f5ea09ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c0b2170 con 0x7f5e9c143190 2024-01-11T09:46:27.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.360+0000 7f5ea09ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c0b2450 con 0x7f5e9c142dc0 2024-01-11T09:46:27.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.360+0000 7f5ea09ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c1423b0 con 0x7f5e9c0cb4b0 2024-01-11T09:46:27.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.360+0000 7f5e9ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e9c143190 0x7f5e9c0c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:27.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.360+0000 7f5e9ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e9c143190 0x7f5e9c0c5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48430/0 (socket says 172.21.15.27:48430) 2024-01-11T09:46:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.360+0000 7f5e9ad9d700 1 -- 172.21.15.27:0/817028114 learned_addr learned my addr 172.21.15.27:0/817028114 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.360+0000 7f5e9a59c700 1 --2- 172.21.15.27:0/817028114 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e9c0cb4b0 0x7f5e9c158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.360+0000 7f5e99d9b700 1 --2- 172.21.15.27:0/817028114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e9c142dc0 0x7f5e9c141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.361+0000 7f5e9ad9d700 1 -- 172.21.15.27:0/817028114 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e9c0cb4b0 msgr2=0x7f5e9c158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.361+0000 7f5e9ad9d700 1 --2- 172.21.15.27:0/817028114 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e9c0cb4b0 0x7f5e9c158f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.361+0000 7f5e9ad9d700 1 -- 172.21.15.27:0/817028114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e9c142dc0 msgr2=0x7f5e9c141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.361+0000 7f5e9ad9d700 1 --2- 172.21.15.27:0/817028114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e9c142dc0 0x7f5e9c141e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.361+0000 7f5e9ad9d700 1 -- 172.21.15.27:0/817028114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e94009a20 con 0x7f5e9c143190 2024-01-11T09:46:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.361+0000 7f5e99d9b700 1 --2- 172.21.15.27:0/817028114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e9c142dc0 0x7f5e9c141e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.361+0000 7f5e9ad9d700 1 --2- 172.21.15.27:0/817028114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e9c143190 0x7f5e9c0c5490 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c064a70 tx=0x7f5e94019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.361+0000 7f5e8a7fc700 1 -- 172.21.15.27:0/817028114 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e94026480 con 0x7f5e9c143190 2024-01-11T09:46:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.361+0000 7f5e8a7fc700 1 -- 172.21.15.27:0/817028114 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e94026600 con 0x7f5e9c143190 2024-01-11T09:46:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.361+0000 7f5e8a7fc700 1 -- 172.21.15.27:0/817028114 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e940269c0 con 0x7f5e9c143190 2024-01-11T09:46:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.361+0000 7f5ea09ff700 1 -- 172.21.15.27:0/817028114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e9c0ca870 con 0x7f5e9c143190 2024-01-11T09:46:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.361+0000 7f5ea09ff700 1 -- 172.21.15.27:0/817028114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e9c1598f0 con 0x7f5e9c143190 2024-01-11T09:46:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.363+0000 7f5ea09ff700 1 -- 172.21.15.27:0/817028114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e9c0b25e0 con 0x7f5e9c143190 2024-01-11T09:46:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.363+0000 7f5e8a7fc700 1 -- 172.21.15.27:0/817028114 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f5e9401c400 con 0x7f5e9c143190 2024-01-11T09:46:27.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.365+0000 7f5e8a7fc700 1 --2- 172.21.15.27:0/817028114 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5e7c067170 0x7f5e7c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:27.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.365+0000 7f5e8a7fc700 1 -- 172.21.15.27:0/817028114 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f5e94086e50 con 0x7f5e9c143190 2024-01-11T09:46:27.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.365+0000 7f5e9a59c700 1 --2- 172.21.15.27:0/817028114 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5e7c067170 0x7f5e7c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:27.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.366+0000 7f5e9a59c700 1 --2- 172.21.15.27:0/817028114 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5e7c067170 0x7f5e7c069620 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c007fd0 tx=0x7f5e8c009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:27.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.369+0000 7f5e8a7fc700 1 -- 172.21.15.27:0/817028114 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5e940576a0 con 0x7f5e9c143190 2024-01-11T09:46:27.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:27.619+0000 7f5ea09ff700 1 -- 172.21.15.27:0/817028114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5e9c04c3c0 con 0x7f5e9c143190 2024-01-11T09:46:28.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.258+0000 7f5e8a7fc700 1 -- 172.21.15.27:0/817028114 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v126) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5e9402b0b0 con 0x7f5e9c143190 2024-01-11T09:46:28.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.261+0000 7f5ea09ff700 1 -- 172.21.15.27:0/817028114 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5e7c067170 msgr2=0x7f5e7c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:28.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.261+0000 7f5ea09ff700 1 --2- 172.21.15.27:0/817028114 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5e7c067170 0x7f5e7c069620 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c007fd0 tx=0x7f5e8c009040 comp rx=0 tx=0).stop 2024-01-11T09:46:28.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.261+0000 7f5ea09ff700 1 -- 172.21.15.27:0/817028114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e9c143190 msgr2=0x7f5e9c0c5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:28.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.261+0000 7f5ea09ff700 1 --2- 172.21.15.27:0/817028114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e9c143190 0x7f5e9c0c5490 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c064a70 tx=0x7f5e94019000 comp rx=0 tx=0).stop 2024-01-11T09:46:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.261+0000 7f5ea09ff700 1 -- 172.21.15.27:0/817028114 shutdown_connections 2024-01-11T09:46:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.261+0000 7f5ea09ff700 1 --2- 172.21.15.27:0/817028114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e9c143190 0x7f5e9c0c5490 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.261+0000 7f5ea09ff700 1 --2- 172.21.15.27:0/817028114 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5e9c0cb4b0 0x7f5e9c158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.261+0000 7f5ea09ff700 1 --2- 172.21.15.27:0/817028114 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5e7c067170 0x7f5e7c069620 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.261+0000 7f5ea09ff700 1 --2- 172.21.15.27:0/817028114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e9c142dc0 0x7f5e9c141e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.261+0000 7f5ea09ff700 1 -- 172.21.15.27:0/817028114 >> 172.21.15.27:0/817028114 conn(0x7f5e9c063ca0 msgr2=0x7f5e9c068720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.262+0000 7f5ea09ff700 1 -- 172.21.15.27:0/817028114 shutdown_connections 2024-01-11T09:46:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.262+0000 7f5ea09ff700 1 -- 172.21.15.27:0/817028114 wait complete. 2024-01-11T09:46:28.274 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-11T09:46:28.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.489+0000 7fc2c29ff700 1 Processor -- start 2024-01-11T09:46:28.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.489+0000 7fc2c29ff700 1 -- start start 2024-01-11T09:46:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.490+0000 7fc2c29ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc2bc157410 0x7fc2bc0ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.490+0000 7fc2c29ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2bc1577e0 0x7fc2bc0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.490+0000 7fc2c29ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2bc069540 0x7fc2bc0ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.490+0000 7fc2c29ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2bc0699d0 con 0x7fc2bc069540 2024-01-11T09:46:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.490+0000 7fc2c29ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2bc069cb0 con 0x7fc2bc157410 2024-01-11T09:46:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.490+0000 7fc2c29ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc069f90 con 0x7fc2bc1577e0 2024-01-11T09:46:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.491+0000 7fc2c0f9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2bc069540 0x7fc2bc0ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.491+0000 7fc2c0f9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2bc069540 0x7fc2bc0ae540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48432/0 (socket says 172.21.15.27:48432) 2024-01-11T09:46:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.491+0000 7fc2c0f9c700 1 -- 172.21.15.27:0/4157817849 learned_addr learned my addr 172.21.15.27:0/4157817849 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.491+0000 7fc2bb7fe700 1 --2- 172.21.15.27:0/4157817849 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2bc1577e0 0x7fc2bc0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.491+0000 7fc2bbfff700 1 --2- 172.21.15.27:0/4157817849 >> v2:172.21.15.77:3300/0 conn(0x7fc2bc157410 0x7fc2bc0ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.491+0000 7fc2c0f9c700 1 -- 172.21.15.27:0/4157817849 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2bc1577e0 msgr2=0x7fc2bc0ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.491+0000 7fc2c0f9c700 1 --2- 172.21.15.27:0/4157817849 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2bc1577e0 0x7fc2bc0ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.491+0000 7fc2c0f9c700 1 -- 172.21.15.27:0/4157817849 >> v2:172.21.15.77:3300/0 conn(0x7fc2bc157410 msgr2=0x7fc2bc0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.491+0000 7fc2c0f9c700 1 --2- 172.21.15.27:0/4157817849 >> v2:172.21.15.77:3300/0 conn(0x7fc2bc157410 0x7fc2bc0ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.491+0000 7fc2c0f9c700 1 -- 172.21.15.27:0/4157817849 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc06a270 con 0x7fc2bc069540 2024-01-11T09:46:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.491+0000 7fc2bbfff700 1 --2- 172.21.15.27:0/4157817849 >> v2:172.21.15.77:3300/0 conn(0x7fc2bc157410 0x7fc2bc0ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.492+0000 7fc2c0f9c700 1 --2- 172.21.15.27:0/4157817849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc069540 0x7fc2bc0ae540 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc070c90 tx=0x7fc2b4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2809938a7af759 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.492+0000 7fc2baffd700 1 -- 172.21.15.27:0/4157817849 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b400e070 con 0x7fc2bc069540 2024-01-11T09:46:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.492+0000 7fc2baffd700 1 -- 172.21.15.27:0/4157817849 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b400ba20 con 0x7fc2bc069540 2024-01-11T09:46:28.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.492+0000 7fc2baffd700 1 -- 172.21.15.27:0/4157817849 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b40092e0 con 0x7fc2bc069540 2024-01-11T09:46:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.492+0000 7fc2c29ff700 1 -- 172.21.15.27:0/4157817849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc069540 msgr2=0x7fc2bc0ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.492+0000 7fc2c29ff700 1 --2- 172.21.15.27:0/4157817849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc069540 0x7fc2bc0ae540 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc070c90 tx=0x7fc2b4009840 comp rx=0 tx=0).stop 2024-01-11T09:46:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.492+0000 7fc2c29ff700 1 -- 172.21.15.27:0/4157817849 shutdown_connections 2024-01-11T09:46:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.492+0000 7fc2c29ff700 1 --2- 172.21.15.27:0/4157817849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc069540 0x7fc2bc0ae540 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.492+0000 7fc2c29ff700 1 --2- 172.21.15.27:0/4157817849 >> v2:172.21.15.77:3300/0 conn(0x7fc2bc157410 0x7fc2bc0ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.492+0000 7fc2c29ff700 1 --2- 172.21.15.27:0/4157817849 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2bc1577e0 0x7fc2bc0ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.492+0000 7fc2c29ff700 1 -- 172.21.15.27:0/4157817849 >> 172.21.15.27:0/4157817849 conn(0x7fc2bc0680a0 msgr2=0x7fc2bc0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.493+0000 7fc2c29ff700 1 -- 172.21.15.27:0/4157817849 shutdown_connections 2024-01-11T09:46:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.493+0000 7fc2c29ff700 1 -- 172.21.15.27:0/4157817849 wait complete. 2024-01-11T09:46:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.494+0000 7fc2c29ff700 1 Processor -- start 2024-01-11T09:46:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.494+0000 7fc2c29ff700 1 -- start start 2024-01-11T09:46:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.495+0000 7fc2c29ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc069540 0x7fc2bc0c4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.495+0000 7fc2c29ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2bc157410 0x7fc2bc15c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.495+0000 7fc2c29ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2bc1577e0 0x7fc2bc15cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.495+0000 7fc2c29ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc0c54d0 con 0x7fc2bc069540 2024-01-11T09:46:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.495+0000 7fc2c29ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc0aef60 con 0x7fc2bc1577e0 2024-01-11T09:46:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.495+0000 7fc2c29ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc0c5700 con 0x7fc2bc157410 2024-01-11T09:46:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.495+0000 7fc2bbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc069540 0x7fc2bc0c4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.495+0000 7fc2bbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc069540 0x7fc2bc0c4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48436/0 (socket says 172.21.15.27:48436) 2024-01-11T09:46:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.495+0000 7fc2bbfff700 1 -- 172.21.15.27:0/2480534841 learned_addr learned my addr 172.21.15.27:0/2480534841 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.496+0000 7fc2c0f9c700 1 --2- 172.21.15.27:0/2480534841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2bc1577e0 0x7fc2bc15cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.496+0000 7fc2bbfff700 1 -- 172.21.15.27:0/2480534841 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2bc157410 msgr2=0x7fc2bc15c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.496+0000 7fc2bb7fe700 1 --2- 172.21.15.27:0/2480534841 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2bc157410 0x7fc2bc15c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.496+0000 7fc2bbfff700 1 --2- 172.21.15.27:0/2480534841 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2bc157410 0x7fc2bc15c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.496+0000 7fc2bbfff700 1 -- 172.21.15.27:0/2480534841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2bc1577e0 msgr2=0x7fc2bc15cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:28.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.496+0000 7fc2bbfff700 1 --2- 172.21.15.27:0/2480534841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2bc1577e0 0x7fc2bc15cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:28.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.496+0000 7fc2bbfff700 1 -- 172.21.15.27:0/2480534841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b4009ae0 con 0x7fc2bc069540 2024-01-11T09:46:28.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.496+0000 7fc2c0f9c700 1 --2- 172.21.15.27:0/2480534841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2bc1577e0 0x7fc2bc15cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:46:28.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.496+0000 7fc2bb7fe700 1 --2- 172.21.15.27:0/2480534841 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2bc157410 0x7fc2bc15c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:46:28.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.496+0000 7fc2bbfff700 1 --2- 172.21.15.27:0/2480534841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc069540 0x7fc2bc0c4f90 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac00e920 tx=0x7fc2ac008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:28.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.496+0000 7fc2a3fff700 1 -- 172.21.15.27:0/2480534841 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac005a00 con 0x7fc2bc069540 2024-01-11T09:46:28.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.496+0000 7fc2a3fff700 1 -- 172.21.15.27:0/2480534841 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2ac005b60 con 0x7fc2bc069540 2024-01-11T09:46:28.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.496+0000 7fc2a3fff700 1 -- 172.21.15.27:0/2480534841 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac006020 con 0x7fc2bc069540 2024-01-11T09:46:28.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.496+0000 7fc2c29ff700 1 -- 172.21.15.27:0/2480534841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2bc0c59e0 con 0x7fc2bc069540 2024-01-11T09:46:28.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.496+0000 7fc2c29ff700 1 -- 172.21.15.27:0/2480534841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2bc0c5fa0 con 0x7fc2bc069540 2024-01-11T09:46:28.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.497+0000 7fc2c29ff700 1 -- 172.21.15.27:0/2480534841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2bc04c3c0 con 0x7fc2bc069540 2024-01-11T09:46:28.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.499+0000 7fc2a3fff700 1 -- 172.21.15.27:0/2480534841 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fc2ac006410 con 0x7fc2bc069540 2024-01-11T09:46:28.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.499+0000 7fc2a3fff700 1 --2- 172.21.15.27:0/2480534841 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc2a8067170 0x7fc2a8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:28.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.500+0000 7fc2a3fff700 1 -- 172.21.15.27:0/2480534841 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5817+0+0 (secure 0 0 0) 0x7fc2ac084b40 con 0x7fc2bc069540 2024-01-11T09:46:28.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.500+0000 7fc2bb7fe700 1 --2- 172.21.15.27:0/2480534841 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc2a8067170 0x7fc2a8069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:28.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.501+0000 7fc2bb7fe700 1 --2- 172.21.15.27:0/2480534841 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc2a8067170 0x7fc2a8069620 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0006f00 tx=0x7fc2b0008040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:28.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.504+0000 7fc2a3fff700 1 -- 172.21.15.27:0/2480534841 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc2ac0519a0 con 0x7fc2bc069540 2024-01-11T09:46:28.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:28.756+0000 7fc2c29ff700 1 -- 172.21.15.27:0/2480534841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc2bc0cb2a0 con 0x7fc2bc069540 2024-01-11T09:46:29.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.260+0000 7fc2a3fff700 1 -- 172.21.15.27:0/2480534841 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v127) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc2ac054fc0 con 0x7fc2bc069540 2024-01-11T09:46:29.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.263+0000 7fc2c29ff700 1 -- 172.21.15.27:0/2480534841 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc2a8067170 msgr2=0x7fc2a8069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:29.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.263+0000 7fc2c29ff700 1 --2- 172.21.15.27:0/2480534841 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc2a8067170 0x7fc2a8069620 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0006f00 tx=0x7fc2b0008040 comp rx=0 tx=0).stop 2024-01-11T09:46:29.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.263+0000 7fc2c29ff700 1 -- 172.21.15.27:0/2480534841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc069540 msgr2=0x7fc2bc0c4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:29.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.263+0000 7fc2c29ff700 1 --2- 172.21.15.27:0/2480534841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc069540 0x7fc2bc0c4f90 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac00e920 tx=0x7fc2ac008040 comp rx=0 tx=0).stop 2024-01-11T09:46:29.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.263+0000 7fc2c29ff700 1 -- 172.21.15.27:0/2480534841 shutdown_connections 2024-01-11T09:46:29.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.263+0000 7fc2c29ff700 1 --2- 172.21.15.27:0/2480534841 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc2a8067170 0x7fc2a8069620 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:29.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.263+0000 7fc2c29ff700 1 --2- 172.21.15.27:0/2480534841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc2bc1577e0 0x7fc2bc15cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:29.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.263+0000 7fc2c29ff700 1 --2- 172.21.15.27:0/2480534841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2bc069540 0x7fc2bc0c4f90 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:29.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.263+0000 7fc2c29ff700 1 --2- 172.21.15.27:0/2480534841 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc2bc157410 0x7fc2bc15c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:29.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.263+0000 7fc2c29ff700 1 -- 172.21.15.27:0/2480534841 >> 172.21.15.27:0/2480534841 conn(0x7fc2bc0680a0 msgr2=0x7fc2bc0c6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:29.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.263+0000 7fc2c29ff700 1 -- 172.21.15.27:0/2480534841 shutdown_connections 2024-01-11T09:46:29.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.264+0000 7fc2c29ff700 1 -- 172.21.15.27:0/2480534841 wait complete. 2024-01-11T09:46:29.274 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:46:29.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.483+0000 7f8fd1fff700 1 Processor -- start 2024-01-11T09:46:29.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.483+0000 7f8fd1fff700 1 -- start start 2024-01-11T09:46:29.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.484+0000 7f8fd1fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8fcc0b03e0 0x7f8fcc0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:29.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.484+0000 7f8fd1fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fcc0b07b0 0x7f8fcc0b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:29.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.484+0000 7f8fd1fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8fcc159200 0x7f8fcc159640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:29.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.484+0000 7f8fd1fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fcc0cd730 con 0x7f8fcc159200 2024-01-11T09:46:29.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.484+0000 7f8fd1fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fcc0ccb20 con 0x7f8fcc0b03e0 2024-01-11T09:46:29.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.484+0000 7f8fd1fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc0cce00 con 0x7f8fcc0b07b0 2024-01-11T09:46:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.485+0000 7f8fcbfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8fcc159200 0x7f8fcc159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.485+0000 7f8fcbfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8fcc159200 0x7f8fcc159640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48438/0 (socket says 172.21.15.27:48438) 2024-01-11T09:46:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.485+0000 7f8fcbfff700 1 -- 172.21.15.27:0/3365275443 learned_addr learned my addr 172.21.15.27:0/3365275443 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.485+0000 7f8fcaffd700 1 --2- 172.21.15.27:0/3365275443 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fcc0b07b0 0x7f8fcc0b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.485+0000 7f8fcb7fe700 1 --2- 172.21.15.27:0/3365275443 >> v2:172.21.15.77:3300/0 conn(0x7f8fcc0b03e0 0x7f8fcc0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.485+0000 7f8fcbfff700 1 -- 172.21.15.27:0/3365275443 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fcc0b07b0 msgr2=0x7f8fcc0b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.485+0000 7f8fcbfff700 1 --2- 172.21.15.27:0/3365275443 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fcc0b07b0 0x7f8fcc0b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.485+0000 7f8fcbfff700 1 -- 172.21.15.27:0/3365275443 >> v2:172.21.15.77:3300/0 conn(0x7f8fcc0b03e0 msgr2=0x7f8fcc0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.485+0000 7f8fcbfff700 1 --2- 172.21.15.27:0/3365275443 >> v2:172.21.15.77:3300/0 conn(0x7f8fcc0b03e0 0x7f8fcc0b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.485+0000 7f8fcbfff700 1 -- 172.21.15.27:0/3365275443 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc0cd0e0 con 0x7f8fcc159200 2024-01-11T09:46:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.485+0000 7f8fcbfff700 1 --2- 172.21.15.27:0/3365275443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc159200 0x7f8fcc159640 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc089470 tx=0x7f8fc4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a58ebee7e70914d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.486+0000 7f8fca7fc700 1 -- 172.21.15.27:0/3365275443 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc400e070 con 0x7f8fcc159200 2024-01-11T09:46:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.486+0000 7f8fca7fc700 1 -- 172.21.15.27:0/3365275443 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc400ba50 con 0x7f8fcc159200 2024-01-11T09:46:29.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.486+0000 7f8fca7fc700 1 -- 172.21.15.27:0/3365275443 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40092e0 con 0x7f8fcc159200 2024-01-11T09:46:29.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.486+0000 7f8fd1fff700 1 -- 172.21.15.27:0/3365275443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc159200 msgr2=0x7f8fcc159640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:29.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.486+0000 7f8fd1fff700 1 --2- 172.21.15.27:0/3365275443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc159200 0x7f8fcc159640 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc089470 tx=0x7f8fc4009840 comp rx=0 tx=0).stop 2024-01-11T09:46:29.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.486+0000 7f8fd1fff700 1 -- 172.21.15.27:0/3365275443 shutdown_connections 2024-01-11T09:46:29.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.486+0000 7f8fd1fff700 1 --2- 172.21.15.27:0/3365275443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc159200 0x7f8fcc159640 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:29.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.486+0000 7f8fd1fff700 1 --2- 172.21.15.27:0/3365275443 >> v2:172.21.15.77:3300/0 conn(0x7f8fcc0b03e0 0x7f8fcc0b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:29.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.486+0000 7f8fd1fff700 1 --2- 172.21.15.27:0/3365275443 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fcc0b07b0 0x7f8fcc0b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:29.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.486+0000 7f8fd1fff700 1 -- 172.21.15.27:0/3365275443 >> 172.21.15.27:0/3365275443 conn(0x7f8fcc06ed40 msgr2=0x7f8fcc06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:29.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.486+0000 7f8fd1fff700 1 -- 172.21.15.27:0/3365275443 shutdown_connections 2024-01-11T09:46:29.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.486+0000 7f8fd1fff700 1 -- 172.21.15.27:0/3365275443 wait complete. 2024-01-11T09:46:29.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.488+0000 7f8fd1fff700 1 Processor -- start 2024-01-11T09:46:29.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.489+0000 7f8fd1fff700 1 -- start start 2024-01-11T09:46:29.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.490+0000 7f8fd1fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc0b03e0 0x7f8fcc15c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:29.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.490+0000 7f8fd1fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fcc0b07b0 0x7f8fcc0cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:29.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.490+0000 7f8fd1fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fcc159200 0x7f8fcc0d0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:29.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.490+0000 7f8fd1fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc15b830 con 0x7f8fcc0b03e0 2024-01-11T09:46:29.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.490+0000 7f8fd1fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc15bb10 con 0x7f8fcc0b07b0 2024-01-11T09:46:29.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.490+0000 7f8fd1fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc15bdf0 con 0x7f8fcc159200 2024-01-11T09:46:29.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.490+0000 7f8fcb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc0b03e0 0x7f8fcc15c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:29.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.490+0000 7f8fcb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc0b03e0 0x7f8fcc15c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48454/0 (socket says 172.21.15.27:48454) 2024-01-11T09:46:29.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.490+0000 7f8fcb7fe700 1 -- 172.21.15.27:0/678141651 learned_addr learned my addr 172.21.15.27:0/678141651 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:29.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.490+0000 7f8fcaffd700 1 --2- 172.21.15.27:0/678141651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fcc0b07b0 0x7f8fcc0cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:29.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.490+0000 7f8fcbfff700 1 --2- 172.21.15.27:0/678141651 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fcc159200 0x7f8fcc0d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:29.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.491+0000 7f8fcb7fe700 1 -- 172.21.15.27:0/678141651 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fcc159200 msgr2=0x7f8fcc0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:29.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.491+0000 7f8fcb7fe700 1 --2- 172.21.15.27:0/678141651 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fcc159200 0x7f8fcc0d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:29.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.491+0000 7f8fcb7fe700 1 -- 172.21.15.27:0/678141651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fcc0b07b0 msgr2=0x7f8fcc0cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:29.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.491+0000 7f8fcb7fe700 1 --2- 172.21.15.27:0/678141651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fcc0b07b0 0x7f8fcc0cb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:29.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.491+0000 7f8fcb7fe700 1 -- 172.21.15.27:0/678141651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc4009ae0 con 0x7f8fcc0b03e0 2024-01-11T09:46:29.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.491+0000 7f8fcb7fe700 1 --2- 172.21.15.27:0/678141651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc0b03e0 0x7f8fcc15c8d0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc00d860 tx=0x7f8fbc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:29.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.491+0000 7f8fb37fe700 1 -- 172.21.15.27:0/678141651 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc0125f0 con 0x7f8fcc0b03e0 2024-01-11T09:46:29.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.491+0000 7f8fb37fe700 1 -- 172.21.15.27:0/678141651 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fbc012750 con 0x7f8fcc0b03e0 2024-01-11T09:46:29.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.491+0000 7f8fd1fff700 1 -- 172.21.15.27:0/678141651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fcc15c0d0 con 0x7f8fcc0b03e0 2024-01-11T09:46:29.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.491+0000 7f8fb37fe700 1 -- 172.21.15.27:0/678141651 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc012b10 con 0x7f8fcc0b03e0 2024-01-11T09:46:29.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.491+0000 7f8fd1fff700 1 -- 172.21.15.27:0/678141651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fcc15b400 con 0x7f8fcc0b03e0 2024-01-11T09:46:29.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.493+0000 7f8fd1fff700 1 -- 172.21.15.27:0/678141651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fcc0718d0 con 0x7f8fcc0b03e0 2024-01-11T09:46:29.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.493+0000 7f8fb37fe700 1 -- 172.21.15.27:0/678141651 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f8fbc01e460 con 0x7f8fcc0b03e0 2024-01-11T09:46:29.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.494+0000 7f8fb37fe700 1 --2- 172.21.15.27:0/678141651 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8fb8067170 0x7f8fb8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:29.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.494+0000 7f8fb37fe700 1 -- 172.21.15.27:0/678141651 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f8fbc084df0 con 0x7f8fcc0b03e0 2024-01-11T09:46:29.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.495+0000 7f8fcaffd700 1 --2- 172.21.15.27:0/678141651 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8fb8067170 0x7f8fb8069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:29.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.495+0000 7f8fcaffd700 1 --2- 172.21.15.27:0/678141651 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8fb8067170 0x7f8fb8069620 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0006fd0 tx=0x7f8fc0008040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:29.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.499+0000 7f8fb37fe700 1 -- 172.21.15.27:0/678141651 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8fbc0557f0 con 0x7f8fcc0b03e0 2024-01-11T09:46:29.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.747+0000 7f8fd1fff700 1 -- 172.21.15.27:0/678141651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fcc15c720 con 0x7f8fcc0b03e0 2024-01-11T09:46:29.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.748+0000 7f8fb37fe700 1 -- 172.21.15.27:0/678141651 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+4609 (secure 0 0 0) 0x7f8fbc055380 con 0x7f8fcc0b03e0 2024-01-11T09:46:29.739 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:29.739 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":127,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1568813392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1568813392},{"type":"v1","addr":"172.21.15.77:6834","nonce":1568813392}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122},{"gid":10969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1352491019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1352491019},{"type":"v1","addr":"172.21.15.77:6835","nonce":1352491019}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122},{"gid":11331,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3477661846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3477661846},{"type":"v1","addr":"172.21.15.27:6834","nonce":3477661846}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:46:25.726042+0000","modified":"2024-01-11T09:46:29.257798+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11340":{"gid":11340,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/321681407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":321681407},{"type":"v1","addr":"172.21.15.27:6835","nonce":321681407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2024-01-11T09:46:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.750+0000 7f8fd1fff700 1 -- 172.21.15.27:0/678141651 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8fb8067170 msgr2=0x7f8fb8069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.750+0000 7f8fd1fff700 1 --2- 172.21.15.27:0/678141651 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8fb8067170 0x7f8fb8069620 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0006fd0 tx=0x7f8fc0008040 comp rx=0 tx=0).stop 2024-01-11T09:46:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.750+0000 7f8fd1fff700 1 -- 172.21.15.27:0/678141651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc0b03e0 msgr2=0x7f8fcc15c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.750+0000 7f8fd1fff700 1 --2- 172.21.15.27:0/678141651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc0b03e0 0x7f8fcc15c8d0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc00d860 tx=0x7f8fbc008040 comp rx=0 tx=0).stop 2024-01-11T09:46:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.750+0000 7f8fd1fff700 1 -- 172.21.15.27:0/678141651 shutdown_connections 2024-01-11T09:46:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.750+0000 7f8fd1fff700 1 --2- 172.21.15.27:0/678141651 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8fcc159200 0x7f8fcc0d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.750+0000 7f8fd1fff700 1 --2- 172.21.15.27:0/678141651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc0b03e0 0x7f8fcc15c8d0 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.750+0000 7f8fd1fff700 1 --2- 172.21.15.27:0/678141651 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8fb8067170 0x7f8fb8069620 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.750+0000 7f8fd1fff700 1 --2- 172.21.15.27:0/678141651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fcc0b07b0 0x7f8fcc0cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:29.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.750+0000 7f8fd1fff700 1 -- 172.21.15.27:0/678141651 >> 172.21.15.27:0/678141651 conn(0x7f8fcc06ed40 msgr2=0x7f8fcc1588f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:29.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.750+0000 7f8fd1fff700 1 -- 172.21.15.27:0/678141651 shutdown_connections 2024-01-11T09:46:29.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.750+0000 7f8fd1fff700 1 -- 172.21.15.27:0/678141651 wait complete. 2024-01-11T09:46:29.743 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 127 2024-01-11T09:46:29.760 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:46:29.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.968+0000 7ff865fff700 1 Processor -- start 2024-01-11T09:46:29.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.969+0000 7ff865fff700 1 -- start start 2024-01-11T09:46:29.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.970+0000 7ff865fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff8601596c0 0x7ff86015d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:29.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.970+0000 7ff865fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8600a9cb0 con 0x7ff860068d30 2024-01-11T09:46:29.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.970+0000 7ff865fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86015e210 con 0x7ff8600ca670 2024-01-11T09:46:29.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.970+0000 7ff865fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff86015dc50 con 0x7ff8601596c0 2024-01-11T09:46:29.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.970+0000 7ff85f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff8601596c0 0x7ff86015d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:29.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.970+0000 7ff85f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff8601596c0 0x7ff86015d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48458/0 (socket says 172.21.15.27:48458) 2024-01-11T09:46:29.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.970+0000 7ff85f7fe700 1 -- 172.21.15.27:0/3802763621 learned_addr learned my addr 172.21.15.27:0/3802763621 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:29.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.971+0000 7ff85f7fe700 1 -- 172.21.15.27:0/3802763621 >> v1:172.21.15.77:6789/0 conn(0x7ff8600ca670 legacy=0x7ff860068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:29.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.971+0000 7ff85f7fe700 1 -- 172.21.15.27:0/3802763621 >> v1:172.21.15.27:6789/0 conn(0x7ff860068d30 legacy=0x7ff8600aa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:29.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.971+0000 7ff85f7fe700 1 -- 172.21.15.27:0/3802763621 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86015df30 con 0x7ff8601596c0 2024-01-11T09:46:29.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.971+0000 7ff85f7fe700 1 --2- 172.21.15.27:0/3802763621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8601596c0 0x7ff86015d300 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7ff8500099b0 tx=0x7ff850009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc6d7cc9fe843edc server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:29.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.971+0000 7ff85e7fc700 1 -- 172.21.15.27:0/3802763621 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85000b860 con 0x7ff8601596c0 2024-01-11T09:46:29.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.971+0000 7ff85e7fc700 1 -- 172.21.15.27:0/3802763621 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff85000b9c0 con 0x7ff8601596c0 2024-01-11T09:46:29.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.971+0000 7ff85e7fc700 1 -- 172.21.15.27:0/3802763621 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85000bd60 con 0x7ff8601596c0 2024-01-11T09:46:29.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.972+0000 7ff865fff700 1 -- 172.21.15.27:0/3802763621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8601596c0 msgr2=0x7ff86015d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:29.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.972+0000 7ff865fff700 1 --2- 172.21.15.27:0/3802763621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8601596c0 0x7ff86015d300 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7ff8500099b0 tx=0x7ff850009250 comp rx=0 tx=0).stop 2024-01-11T09:46:29.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.972+0000 7ff865fff700 1 -- 172.21.15.27:0/3802763621 shutdown_connections 2024-01-11T09:46:29.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.972+0000 7ff865fff700 1 --2- 172.21.15.27:0/3802763621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8601596c0 0x7ff86015d300 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:29.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.972+0000 7ff865fff700 1 -- 172.21.15.27:0/3802763621 >> 172.21.15.27:0/3802763621 conn(0x7ff8600c9f70 msgr2=0x7ff86006deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:29.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.972+0000 7ff865fff700 1 -- 172.21.15.27:0/3802763621 shutdown_connections 2024-01-11T09:46:29.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.972+0000 7ff865fff700 1 -- 172.21.15.27:0/3802763621 wait complete. 2024-01-11T09:46:29.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.973+0000 7ff865fff700 1 Processor -- start 2024-01-11T09:46:29.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.974+0000 7ff865fff700 1 -- start start 2024-01-11T09:46:29.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.974+0000 7ff865fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff860068d30 0x7ff8600b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:29.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.974+0000 7ff865fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8600ca670 0x7ff86006b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:29.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.974+0000 7ff865fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8601596c0 0x7ff86006bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:29.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.974+0000 7ff865fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86016ab50 con 0x7ff8601596c0 2024-01-11T09:46:29.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.975+0000 7ff865fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86016e560 con 0x7ff860068d30 2024-01-11T09:46:29.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.975+0000 7ff865fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff86016a220 con 0x7ff8600ca670 2024-01-11T09:46:29.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.975+0000 7ff85ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8601596c0 0x7ff86006bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:29.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.975+0000 7ff85ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8601596c0 0x7ff86006bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48472/0 (socket says 172.21.15.27:48472) 2024-01-11T09:46:29.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.975+0000 7ff85ffff700 1 -- 172.21.15.27:0/3839741498 learned_addr learned my addr 172.21.15.27:0/3839741498 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:29.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.975+0000 7ff85f7fe700 1 --2- 172.21.15.27:0/3839741498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff860068d30 0x7ff8600b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:29.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.975+0000 7ff85effd700 1 --2- 172.21.15.27:0/3839741498 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8600ca670 0x7ff86006b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:29.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.975+0000 7ff85ffff700 1 -- 172.21.15.27:0/3839741498 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8600ca670 msgr2=0x7ff86006b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:29.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.975+0000 7ff85ffff700 1 --2- 172.21.15.27:0/3839741498 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8600ca670 0x7ff86006b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:29.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.975+0000 7ff85ffff700 1 -- 172.21.15.27:0/3839741498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff860068d30 msgr2=0x7ff8600b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:29.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.975+0000 7ff85ffff700 1 --2- 172.21.15.27:0/3839741498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff860068d30 0x7ff8600b2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:29.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.975+0000 7ff85ffff700 1 -- 172.21.15.27:0/3839741498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8500094f0 con 0x7ff8601596c0 2024-01-11T09:46:29.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.976+0000 7ff85ffff700 1 --2- 172.21.15.27:0/3839741498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8601596c0 0x7ff86006bea0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7ff85800c810 tx=0x7ff858001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:29.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.976+0000 7ff8477fe700 1 -- 172.21.15.27:0/3839741498 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff858012630 con 0x7ff8601596c0 2024-01-11T09:46:29.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.976+0000 7ff8477fe700 1 -- 172.21.15.27:0/3839741498 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8580127b0 con 0x7ff8601596c0 2024-01-11T09:46:29.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.976+0000 7ff8477fe700 1 -- 172.21.15.27:0/3839741498 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff858012b70 con 0x7ff8601596c0 2024-01-11T09:46:29.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.976+0000 7ff865fff700 1 -- 172.21.15.27:0/3839741498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff86016a500 con 0x7ff8601596c0 2024-01-11T09:46:29.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.976+0000 7ff865fff700 1 -- 172.21.15.27:0/3839741498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86016afe0 con 0x7ff8601596c0 2024-01-11T09:46:29.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.977+0000 7ff865fff700 1 -- 172.21.15.27:0/3839741498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8600718d0 con 0x7ff8601596c0 2024-01-11T09:46:29.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.978+0000 7ff8477fe700 1 -- 172.21.15.27:0/3839741498 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7ff85801b400 con 0x7ff8601596c0 2024-01-11T09:46:29.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.979+0000 7ff8477fe700 1 --2- 172.21.15.27:0/3839741498 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff84c067170 0x7ff84c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:29.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.979+0000 7ff8477fe700 1 -- 172.21.15.27:0/3839741498 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5817+0+0 (secure 0 0 0) 0x7ff858084af0 con 0x7ff8601596c0 2024-01-11T09:46:29.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.980+0000 7ff85f7fe700 1 --2- 172.21.15.27:0/3839741498 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff84c067170 0x7ff84c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:29.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.980+0000 7ff85f7fe700 1 --2- 172.21.15.27:0/3839741498 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff84c067170 0x7ff84c069620 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff8500099b0 tx=0x7ff850017000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:29.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:29.983+0000 7ff8477fe700 1 -- 172.21.15.27:0/3839741498 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff8580519c0 con 0x7ff8601596c0 2024-01-11T09:46:30.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.186+0000 7ff865fff700 1 -- 172.21.15.27:0/3839741498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8600a9940 con 0x7ff8601596c0 2024-01-11T09:46:30.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.187+0000 7ff8477fe700 1 -- 172.21.15.27:0/3839741498 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v235) v1 ==== 74+0+15495 (secure 0 0 0) 0x7ff858054fe0 con 0x7ff8601596c0 2024-01-11T09:46:30.179 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:30.179 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":235,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:46:26.932357+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:46:24.049904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-01-11T09:46:24.737495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:46:30.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.189+0000 7ff865fff700 1 -- 172.21.15.27:0/3839741498 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff84c067170 msgr2=0x7ff84c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:30.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.189+0000 7ff865fff700 1 --2- 172.21.15.27:0/3839741498 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff84c067170 0x7ff84c069620 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff8500099b0 tx=0x7ff850017000 comp rx=0 tx=0).stop 2024-01-11T09:46:30.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.189+0000 7ff865fff700 1 -- 172.21.15.27:0/3839741498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8601596c0 msgr2=0x7ff86006bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:30.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.189+0000 7ff865fff700 1 --2- 172.21.15.27:0/3839741498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8601596c0 0x7ff86006bea0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7ff85800c810 tx=0x7ff858001650 comp rx=0 tx=0).stop 2024-01-11T09:46:30.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.189+0000 7ff865fff700 1 -- 172.21.15.27:0/3839741498 shutdown_connections 2024-01-11T09:46:30.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.189+0000 7ff865fff700 1 --2- 172.21.15.27:0/3839741498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8601596c0 0x7ff86006bea0 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:30.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.189+0000 7ff865fff700 1 --2- 172.21.15.27:0/3839741498 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff84c067170 0x7ff84c069620 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:30.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.189+0000 7ff865fff700 1 --2- 172.21.15.27:0/3839741498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff860068d30 0x7ff8600b2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:30.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.189+0000 7ff865fff700 1 --2- 172.21.15.27:0/3839741498 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff8600ca670 0x7ff86006b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:30.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.189+0000 7ff865fff700 1 -- 172.21.15.27:0/3839741498 >> 172.21.15.27:0/3839741498 conn(0x7ff8600c9f70 msgr2=0x7ff8600c2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:30.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.190+0000 7ff865fff700 1 -- 172.21.15.27:0/3839741498 shutdown_connections 2024-01-11T09:46:30.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.190+0000 7ff865fff700 1 -- 172.21.15.27:0/3839741498 wait complete. 2024-01-11T09:46:30.199 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-11T09:46:30.199 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:46:30.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.410+0000 7f84509ff700 1 Processor -- start 2024-01-11T09:46:30.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.410+0000 7f84509ff700 1 -- start start 2024-01-11T09:46:30.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.411+0000 7f84509ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f844c0b03e0 0x7f844c0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:30.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.411+0000 7f84509ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c0cd730 con 0x7f844c0b29c0 2024-01-11T09:46:30.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.411+0000 7f84509ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c0b2de0 con 0x7f844c0b07b0 2024-01-11T09:46:30.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.411+0000 7f84509ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f844c0ccb20 con 0x7f844c0b03e0 2024-01-11T09:46:30.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.412+0000 7f844ad9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f844c0b29c0 0x7f844c0aefc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52400/0 (socket says 172.21.15.27:52400) 2024-01-11T09:46:30.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.412+0000 7f844ad9d700 1 -- 172.21.15.27:0/2985273484 learned_addr learned my addr 172.21.15.27:0/2985273484 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:30.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.412+0000 7f844a59c700 1 --2- 172.21.15.27:0/2985273484 >> v2:172.21.15.77:3300/0 conn(0x7f844c0b03e0 0x7f844c0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:30.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.412+0000 7f844959a700 1 -- 172.21.15.27:0/2985273484 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550006026 0 0) 0x7f844c0cd730 con 0x7f844c0b29c0 2024-01-11T09:46:30.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.412+0000 7f844959a700 1 -- 172.21.15.27:0/2985273484 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8434003570 con 0x7f844c0b29c0 2024-01-11T09:46:30.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.412+0000 7f844959a700 1 -- 172.21.15.27:0/2985273484 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979045436 0 0) 0x7f844c0b2de0 con 0x7f844c0b07b0 2024-01-11T09:46:30.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.412+0000 7f844959a700 1 -- 172.21.15.27:0/2985273484 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f844c0cd730 con 0x7f844c0b07b0 2024-01-11T09:46:30.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.412+0000 7f844959a700 1 -- 172.21.15.27:0/2985273484 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3872974872 0 0) 0x7f8434003570 con 0x7f844c0b29c0 2024-01-11T09:46:30.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.412+0000 7f844a59c700 1 -- 172.21.15.27:0/2985273484 >> v1:172.21.15.77:6789/0 conn(0x7f844c0b07b0 legacy=0x7f844c159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:30.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.412+0000 7f844a59c700 1 -- 172.21.15.27:0/2985273484 >> v1:172.21.15.27:6789/0 conn(0x7f844c0b29c0 legacy=0x7f844c0aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:30.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.412+0000 7f844a59c700 1 -- 172.21.15.27:0/2985273484 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c0cce00 con 0x7f844c0b03e0 2024-01-11T09:46:30.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.413+0000 7f844a59c700 1 --2- 172.21.15.27:0/2985273484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f844c0b03e0 0x7f844c0b2480 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f843c0098e0 tx=0x7f843c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db75aa52a62f2698 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:30.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.413+0000 7f844959a700 1 -- 172.21.15.27:0/2985273484 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84440017f0 con 0x7f844c0b03e0 2024-01-11T09:46:30.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.413+0000 7f844959a700 1 -- 172.21.15.27:0/2985273484 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843c011420 con 0x7f844c0b03e0 2024-01-11T09:46:30.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.413+0000 7f844959a700 1 -- 172.21.15.27:0/2985273484 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c0117c0 con 0x7f844c0b03e0 2024-01-11T09:46:30.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.414+0000 7f84509ff700 1 -- 172.21.15.27:0/2985273484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f844c0b03e0 msgr2=0x7f844c0b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:30.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.414+0000 7f84509ff700 1 --2- 172.21.15.27:0/2985273484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f844c0b03e0 0x7f844c0b2480 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f843c0098e0 tx=0x7f843c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:30.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.414+0000 7f84509ff700 1 -- 172.21.15.27:0/2985273484 shutdown_connections 2024-01-11T09:46:30.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.414+0000 7f84509ff700 1 --2- 172.21.15.27:0/2985273484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f844c0b03e0 0x7f844c0b2480 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:30.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.414+0000 7f84509ff700 1 -- 172.21.15.27:0/2985273484 >> 172.21.15.27:0/2985273484 conn(0x7f844c06ed40 msgr2=0x7f844c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:30.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.414+0000 7f84509ff700 1 -- 172.21.15.27:0/2985273484 shutdown_connections 2024-01-11T09:46:30.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.414+0000 7f84509ff700 1 -- 172.21.15.27:0/2985273484 wait complete. 2024-01-11T09:46:30.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.415+0000 7f84509ff700 1 Processor -- start 2024-01-11T09:46:30.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.415+0000 7f84509ff700 1 -- start start 2024-01-11T09:46:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.416+0000 7f84509ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f844c0b03e0 0x7f844c16a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.416+0000 7f84509ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f844c0b07b0 0x7f844c16afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.416+0000 7f84509ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f844c0b29c0 0x7f844c16cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.416+0000 7f84509ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c0aeb30 con 0x7f844c0b03e0 2024-01-11T09:46:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.416+0000 7f84509ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c16ad60 con 0x7f844c0b29c0 2024-01-11T09:46:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.416+0000 7f84509ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844c0abc80 con 0x7f844c0b07b0 2024-01-11T09:46:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.417+0000 7f844a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f844c0b03e0 0x7f844c16a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.417+0000 7f844a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f844c0b03e0 0x7f844c16a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48476/0 (socket says 172.21.15.27:48476) 2024-01-11T09:46:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.417+0000 7f844a59c700 1 -- 172.21.15.27:0/3641462428 learned_addr learned my addr 172.21.15.27:0/3641462428 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.417+0000 7f844a59c700 1 -- 172.21.15.27:0/3641462428 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f844c0b07b0 msgr2=0x7f844c16afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:30.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.417+0000 7f844a59c700 1 --2- 172.21.15.27:0/3641462428 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f844c0b07b0 0x7f844c16afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.417+0000 7f844a59c700 1 -- 172.21.15.27:0/3641462428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f844c0b29c0 msgr2=0x7f844c16cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.417+0000 7f844ad9d700 1 --2- 172.21.15.27:0/3641462428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f844c0b29c0 0x7f844c16cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.417+0000 7f844a59c700 1 --2- 172.21.15.27:0/3641462428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f844c0b29c0 0x7f844c16cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.417+0000 7f844a59c700 1 -- 172.21.15.27:0/3641462428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c009420 con 0x7f844c0b03e0 2024-01-11T09:46:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.417+0000 7f844ad9d700 1 --2- 172.21.15.27:0/3641462428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f844c0b29c0 0x7f844c16cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:46:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.417+0000 7f844a59c700 1 --2- 172.21.15.27:0/3641462428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f844c0b03e0 0x7f844c16a820 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f844c069540 tx=0x7f843c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.418+0000 7f843a7fc700 1 -- 172.21.15.27:0/3641462428 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c01cc90 con 0x7f844c0b03e0 2024-01-11T09:46:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.418+0000 7f843a7fc700 1 -- 172.21.15.27:0/3641462428 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843c01cdf0 con 0x7f844c0b03e0 2024-01-11T09:46:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.418+0000 7f84509ff700 1 -- 172.21.15.27:0/3641462428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844c0adf20 con 0x7f844c0b03e0 2024-01-11T09:46:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.418+0000 7f843a7fc700 1 -- 172.21.15.27:0/3641462428 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c01a6a0 con 0x7f844c0b03e0 2024-01-11T09:46:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.418+0000 7f84509ff700 1 -- 172.21.15.27:0/3641462428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c0ae4e0 con 0x7f844c0b03e0 2024-01-11T09:46:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.419+0000 7f84509ff700 1 -- 172.21.15.27:0/3641462428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844c0718d0 con 0x7f844c0b03e0 2024-01-11T09:46:30.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.420+0000 7f843a7fc700 1 -- 172.21.15.27:0/3641462428 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f843c01aa50 con 0x7f844c0b03e0 2024-01-11T09:46:30.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.421+0000 7f843a7fc700 1 --2- 172.21.15.27:0/3641462428 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f842c067170 0x7f842c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:30.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.421+0000 7f843a7fc700 1 -- 172.21.15.27:0/3641462428 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f843c086380 con 0x7f844c0b03e0 2024-01-11T09:46:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.421+0000 7f8449d9b700 1 --2- 172.21.15.27:0/3641462428 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f842c067170 0x7f842c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.422+0000 7f8449d9b700 1 --2- 172.21.15.27:0/3641462428 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f842c067170 0x7f842c069620 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f84400026f0 tx=0x7f84400079d0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.425+0000 7f843a7fc700 1 -- 172.21.15.27:0/3641462428 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f843c0532f0 con 0x7f844c0b03e0 2024-01-11T09:46:30.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.624+0000 7f84509ff700 1 -- 172.21.15.27:0/3641462428 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f844c0aec70 con 0x7f842c067170 2024-01-11T09:46:30.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.634+0000 7f843a7fc700 1 -- 172.21.15.27:0/3641462428 <== mgr.9107 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289646 (secure 0 0 0) 0x7f8440014400 con 0x7f842c067170 2024-01-11T09:46:30.630 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:30.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.638+0000 7f84509ff700 1 -- 172.21.15.27:0/3641462428 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f842c067170 msgr2=0x7f842c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:30.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.638+0000 7f84509ff700 1 --2- 172.21.15.27:0/3641462428 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f842c067170 0x7f842c069620 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f84400026f0 tx=0x7f84400079d0 comp rx=0 tx=0).stop 2024-01-11T09:46:30.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.638+0000 7f84509ff700 1 -- 172.21.15.27:0/3641462428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f844c0b03e0 msgr2=0x7f844c16a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:30.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.638+0000 7f84509ff700 1 --2- 172.21.15.27:0/3641462428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f844c0b03e0 0x7f844c16a820 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f844c069540 tx=0x7f843c016000 comp rx=0 tx=0).stop 2024-01-11T09:46:30.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.639+0000 7f84509ff700 1 -- 172.21.15.27:0/3641462428 shutdown_connections 2024-01-11T09:46:30.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.639+0000 7f84509ff700 1 --2- 172.21.15.27:0/3641462428 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f842c067170 0x7f842c069620 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:30.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.639+0000 7f84509ff700 1 --2- 172.21.15.27:0/3641462428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f844c0b29c0 0x7f844c16cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:30.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.639+0000 7f84509ff700 1 --2- 172.21.15.27:0/3641462428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f844c0b03e0 0x7f844c16a820 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:30.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.639+0000 7f84509ff700 1 --2- 172.21.15.27:0/3641462428 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f844c0b07b0 0x7f844c16afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:30.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.639+0000 7f84509ff700 1 -- 172.21.15.27:0/3641462428 >> 172.21.15.27:0/3641462428 conn(0x7f844c06ed40 msgr2=0x7f844c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:30.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.639+0000 7f84509ff700 1 -- 172.21.15.27:0/3641462428 shutdown_connections 2024-01-11T09:46:30.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.639+0000 7f84509ff700 1 -- 172.21.15.27:0/3641462428 wait complete. 2024-01-11T09:46:30.634 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:46:30.648 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-01-11T09:46:30.158435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":695,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":426,"ondisk_log_size":426,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2451816,"kb_used_data":24360,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752522904,"statfs":{"total":773094113280,"available":770583453696,"internally_reserved":0,"allocated":24944640,"data_stored":12579076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.944946"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.956201+0000","last_change":"2024-01-11T09:46:25.952190+0000","last_active":"2024-01-11T09:46:25.956201+0000","last_peered":"2024-01-11T09:46:25.956201+0000","last_clean":"2024-01-11T09:46:25.956201+0000","last_became_active":"2024-01-11T09:46:25.951733+0000","last_became_peered":"2024-01-11T09:46:25.951733+0000","last_unstale":"2024-01-11T09:46:25.956201+0000","last_undegraded":"2024-01-11T09:46:25.956201+0000","last_fullsized":"2024-01-11T09:46:25.956201+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:59:15.007682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960088+0000","last_change":"2024-01-11T09:46:25.957907+0000","last_active":"2024-01-11T09:46:25.960088+0000","last_peered":"2024-01-11T09:46:25.960088+0000","last_clean":"2024-01-11T09:46:25.960088+0000","last_became_active":"2024-01-11T09:46:25.956251+0000","last_became_peered":"2024-01-11T09:46:25.956251+0000","last_unstale":"2024-01-11T09:46:25.960088+0000","last_undegraded":"2024-01-11T09:46:25.960088+0000","last_fullsized":"2024-01-11T09:46:25.960088+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:53:35.139255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960472+0000","last_change":"2024-01-11T09:46:25.956076+0000","last_active":"2024-01-11T09:46:25.960472+0000","last_peered":"2024-01-11T09:46:25.960472+0000","last_clean":"2024-01-11T09:46:25.960472+0000","last_became_active":"2024-01-11T09:46:25.955819+0000","last_became_peered":"2024-01-11T09:46:25.955819+0000","last_unstale":"2024-01-11T09:46:25.960472+0000","last_undegraded":"2024-01-11T09:46:25.960472+0000","last_fullsized":"2024-01-11T09:46:25.960472+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:27:34.516236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.952790+0000","last_change":"2024-01-11T09:46:25.947296+0000","last_active":"2024-01-11T09:46:25.952790+0000","last_peered":"2024-01-11T09:46:25.952790+0000","last_clean":"2024-01-11T09:46:25.952790+0000","last_became_active":"2024-01-11T09:46:25.946057+0000","last_became_peered":"2024-01-11T09:46:25.946057+0000","last_unstale":"2024-01-11T09:46:25.952790+0000","last_undegraded":"2024-01-11T09:46:25.952790+0000","last_fullsized":"2024-01-11T09:46:25.952790+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:15:19.562588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.955304+0000","last_change":"2024-01-11T09:46:25.953859+0000","last_active":"2024-01-11T09:46:25.955304+0000","last_peered":"2024-01-11T09:46:25.955304+0000","last_clean":"2024-01-11T09:46:25.955304+0000","last_became_active":"2024-01-11T09:46:25.953566+0000","last_became_peered":"2024-01-11T09:46:25.953566+0000","last_unstale":"2024-01-11T09:46:25.955304+0000","last_undegraded":"2024-01-11T09:46:25.955304+0000","last_fullsized":"2024-01-11T09:46:25.955304+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:42:55.688728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.953888+0000","last_change":"2024-01-11T09:46:25.945778+0000","last_active":"2024-01-11T09:46:25.953888+0000","last_peered":"2024-01-11T09:46:25.953888+0000","last_clean":"2024-01-11T09:46:25.953888+0000","last_became_active":"2024-01-11T09:46:25.944907+0000","last_became_peered":"2024-01-11T09:46:25.944907+0000","last_unstale":"2024-01-11T09:46:25.953888+0000","last_undegraded":"2024-01-11T09:46:25.953888+0000","last_fullsized":"2024-01-11T09:46:25.953888+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:59:02.206221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.963832+0000","last_change":"2024-01-11T09:46:25.960002+0000","last_active":"2024-01-11T09:46:25.963832+0000","last_peered":"2024-01-11T09:46:25.963832+0000","last_clean":"2024-01-11T09:46:25.963832+0000","last_became_active":"2024-01-11T09:46:25.957845+0000","last_became_peered":"2024-01-11T09:46:25.957845+0000","last_unstale":"2024-01-11T09:46:25.963832+0000","last_undegraded":"2024-01-11T09:46:25.963832+0000","last_fullsized":"2024-01-11T09:46:25.963832+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:58:52.892728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.963536+0000","last_change":"2024-01-11T09:46:25.958630+0000","last_active":"2024-01-11T09:46:25.963536+0000","last_peered":"2024-01-11T09:46:25.963536+0000","last_clean":"2024-01-11T09:46:25.963536+0000","last_became_active":"2024-01-11T09:46:25.957768+0000","last_became_peered":"2024-01-11T09:46:25.957768+0000","last_unstale":"2024-01-11T09:46:25.963536+0000","last_undegraded":"2024-01-11T09:46:25.963536+0000","last_fullsized":"2024-01-11T09:46:25.963536+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:59:15.007682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.956638+0000","last_change":"2024-01-11T09:46:25.954318+0000","last_active":"2024-01-11T09:46:25.956638+0000","last_peered":"2024-01-11T09:46:25.956638+0000","last_clean":"2024-01-11T09:46:25.956638+0000","last_became_active":"2024-01-11T09:46:25.953690+0000","last_became_peered":"2024-01-11T09:46:25.953690+0000","last_unstale":"2024-01-11T09:46:25.956638+0000","last_undegraded":"2024-01-11T09:46:25.956638+0000","last_fullsized":"2024-01-11T09:46:25.956638+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:28:55.053850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.961820+0000","last_change":"2024-01-11T09:46:25.957195+0000","last_active":"2024-01-11T09:46:25.961820+0000","last_peered":"2024-01-11T09:46:25.961820+0000","last_clean":"2024-01-11T09:46:25.961820+0000","last_became_active":"2024-01-11T09:46:25.956882+0000","last_became_peered":"2024-01-11T09:46:25.956882+0000","last_unstale":"2024-01-11T09:46:25.961820+0000","last_undegraded":"2024-01-11T09:46:25.961820+0000","last_fullsized":"2024-01-11T09:46:25.961820+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:37:41.519655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.963173+0000","last_change":"2024-01-11T09:46:25.960943+0000","last_active":"2024-01-11T09:46:25.963173+0000","last_peered":"2024-01-11T09:46:25.963173+0000","last_clean":"2024-01-11T09:46:25.963173+0000","last_became_active":"2024-01-11T09:46:25.960111+0000","last_became_peered":"2024-01-11T09:46:25.960111+0000","last_unstale":"2024-01-11T09:46:25.963173+0000","last_undegraded":"2024-01-11T09:46:25.963173+0000","last_fullsized":"2024-01-11T09:46:25.963173+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:04.562589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.961397+0000","last_change":"2024-01-11T09:46:25.955367+0000","last_active":"2024-01-11T09:46:25.961397+0000","last_peered":"2024-01-11T09:46:25.961397+0000","last_clean":"2024-01-11T09:46:25.961397+0000","last_became_active":"2024-01-11T09:46:25.955123+0000","last_became_peered":"2024-01-11T09:46:25.955123+0000","last_unstale":"2024-01-11T09:46:25.961397+0000","last_undegraded":"2024-01-11T09:46:25.961397+0000","last_fullsized":"2024-01-11T09:46:25.961397+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:07:13.288202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.963023+0000","last_change":"2024-01-11T09:46:25.958036+0000","last_active":"2024-01-11T09:46:25.963023+0000","last_peered":"2024-01-11T09:46:25.963023+0000","last_clean":"2024-01-11T09:46:25.963023+0000","last_became_active":"2024-01-11T09:46:25.957739+0000","last_became_peered":"2024-01-11T09:46:25.957739+0000","last_unstale":"2024-01-11T09:46:25.963023+0000","last_undegraded":"2024-01-11T09:46:25.963023+0000","last_fullsized":"2024-01-11T09:46:25.963023+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:04.562589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.955491+0000","last_change":"2024-01-11T09:46:25.951789+0000","last_active":"2024-01-11T09:46:25.955491+0000","last_peered":"2024-01-11T09:46:25.955491+0000","last_clean":"2024-01-11T09:46:25.955491+0000","last_became_active":"2024-01-11T09:46:25.950427+0000","last_became_peered":"2024-01-11T09:46:25.950427+0000","last_unstale":"2024-01-11T09:46:25.955491+0000","last_undegraded":"2024-01-11T09:46:25.955491+0000","last_fullsized":"2024-01-11T09:46:25.955491+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:10.125414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.963896+0000","last_change":"2024-01-11T09:46:25.962921+0000","last_active":"2024-01-11T09:46:25.963896+0000","last_peered":"2024-01-11T09:46:25.963896+0000","last_clean":"2024-01-11T09:46:25.963896+0000","last_became_active":"2024-01-11T09:46:25.962306+0000","last_became_peered":"2024-01-11T09:46:25.962306+0000","last_unstale":"2024-01-11T09:46:25.963896+0000","last_undegraded":"2024-01-11T09:46:25.963896+0000","last_fullsized":"2024-01-11T09:46:25.963896+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:49:33.997022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958634+0000","last_change":"2024-01-11T09:46:25.951750+0000","last_active":"2024-01-11T09:46:25.958634+0000","last_peered":"2024-01-11T09:46:25.958634+0000","last_clean":"2024-01-11T09:46:25.958634+0000","last_became_active":"2024-01-11T09:46:25.951499+0000","last_became_peered":"2024-01-11T09:46:25.951499+0000","last_unstale":"2024-01-11T09:46:25.958634+0000","last_undegraded":"2024-01-11T09:46:25.958634+0000","last_fullsized":"2024-01-11T09:46:25.958634+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:48:16.354636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.952702+0000","last_change":"2024-01-11T09:46:25.950974+0000","last_active":"2024-01-11T09:46:25.952702+0000","last_peered":"2024-01-11T09:46:25.952702+0000","last_clean":"2024-01-11T09:46:25.952702+0000","last_became_active":"2024-01-11T09:46:25.950595+0000","last_became_peered":"2024-01-11T09:46:25.950595+0000","last_unstale":"2024-01-11T09:46:25.952702+0000","last_undegraded":"2024-01-11T09:46:25.952702+0000","last_fullsized":"2024-01-11T09:46:25.952702+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:58.843659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.955429+0000","last_change":"2024-01-11T09:46:25.952186+0000","last_active":"2024-01-11T09:46:25.955429+0000","last_peered":"2024-01-11T09:46:25.955429+0000","last_clean":"2024-01-11T09:46:25.955429+0000","last_became_active":"2024-01-11T09:46:25.951898+0000","last_became_peered":"2024-01-11T09:46:25.951898+0000","last_unstale":"2024-01-11T09:46:25.955429+0000","last_undegraded":"2024-01-11T09:46:25.955429+0000","last_fullsized":"2024-01-11T09:46:25.955429+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:54:00.534598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.963483+0000","last_change":"2024-01-11T09:46:25.957991+0000","last_active":"2024-01-11T09:46:25.963483+0000","last_peered":"2024-01-11T09:46:25.963483+0000","last_clean":"2024-01-11T09:46:25.963483+0000","last_became_active":"2024-01-11T09:46:25.957482+0000","last_became_peered":"2024-01-11T09:46:25.957482+0000","last_unstale":"2024-01-11T09:46:25.963483+0000","last_undegraded":"2024-01-11T09:46:25.963483+0000","last_fullsized":"2024-01-11T09:46:25.963483+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:54:00.534598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.949569+0000","last_change":"2024-01-11T09:46:25.942788+0000","last_active":"2024-01-11T09:46:25.949569+0000","last_peered":"2024-01-11T09:46:25.949569+0000","last_clean":"2024-01-11T09:46:25.949569+0000","last_became_active":"2024-01-11T09:46:25.941771+0000","last_became_peered":"2024-01-11T09:46:25.941771+0000","last_unstale":"2024-01-11T09:46:25.949569+0000","last_undegraded":"2024-01-11T09:46:25.949569+0000","last_fullsized":"2024-01-11T09:46:25.949569+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:42:47.356295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960319+0000","last_change":"2024-01-11T09:46:25.954694+0000","last_active":"2024-01-11T09:46:25.960319+0000","last_peered":"2024-01-11T09:46:25.960319+0000","last_clean":"2024-01-11T09:46:25.960319+0000","last_became_active":"2024-01-11T09:46:25.952808+0000","last_became_peered":"2024-01-11T09:46:25.952808+0000","last_unstale":"2024-01-11T09:46:25.960319+0000","last_undegraded":"2024-01-11T09:46:25.960319+0000","last_fullsized":"2024-01-11T09:46:25.960319+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:10.125414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.964163+0000","last_change":"2024-01-11T09:46:25.962654+0000","last_active":"2024-01-11T09:46:25.964163+0000","last_peered":"2024-01-11T09:46:25.964163+0000","last_clean":"2024-01-11T09:46:25.964163+0000","last_became_active":"2024-01-11T09:46:25.962073+0000","last_became_peered":"2024-01-11T09:46:25.962073+0000","last_unstale":"2024-01-11T09:46:25.964163+0000","last_undegraded":"2024-01-11T09:46:25.964163+0000","last_fullsized":"2024-01-11T09:46:25.964163+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:56:15.022301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.963312+0000","last_change":"2024-01-11T09:46:25.956810+0000","last_active":"2024-01-11T09:46:25.963312+0000","last_peered":"2024-01-11T09:46:25.963312+0000","last_clean":"2024-01-11T09:46:25.963312+0000","last_became_active":"2024-01-11T09:46:25.955894+0000","last_became_peered":"2024-01-11T09:46:25.955894+0000","last_unstale":"2024-01-11T09:46:25.963312+0000","last_undegraded":"2024-01-11T09:46:25.963312+0000","last_fullsized":"2024-01-11T09:46:25.963312+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:53:35.139255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.951416+0000","last_change":"2024-01-11T09:46:25.947721+0000","last_active":"2024-01-11T09:46:25.951416+0000","last_peered":"2024-01-11T09:46:25.951416+0000","last_clean":"2024-01-11T09:46:25.951416+0000","last_became_active":"2024-01-11T09:46:25.947454+0000","last_became_peered":"2024-01-11T09:46:25.947454+0000","last_unstale":"2024-01-11T09:46:25.951416+0000","last_undegraded":"2024-01-11T09:46:25.951416+0000","last_fullsized":"2024-01-11T09:46:25.951416+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:42:44.083747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.946476+0000","last_change":"2024-01-11T09:46:25.939265+0000","last_active":"2024-01-11T09:46:25.946476+0000","last_peered":"2024-01-11T09:46:25.946476+0000","last_clean":"2024-01-11T09:46:25.946476+0000","last_became_active":"2024-01-11T09:46:25.938964+0000","last_became_peered":"2024-01-11T09:46:25.938964+0000","last_unstale":"2024-01-11T09:46:25.946476+0000","last_undegraded":"2024-01-11T09:46:25.946476+0000","last_fullsized":"2024-01-11T09:46:25.946476+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:20:24.956940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.962886+0000","last_change":"2024-01-11T09:46:25.953966+0000","last_active":"2024-01-11T09:46:25.962886+0000","last_peered":"2024-01-11T09:46:25.962886+0000","last_clean":"2024-01-11T09:46:25.962886+0000","last_became_active":"2024-01-11T09:46:25.953566+0000","last_became_peered":"2024-01-11T09:46:25.953566+0000","last_unstale":"2024-01-11T09:46:25.962886+0000","last_undegraded":"2024-01-11T09:46:25.962886+0000","last_fullsized":"2024-01-11T09:46:25.962886+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:45:16.842085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.962801+0000","last_change":"2024-01-11T09:46:25.959751+0000","last_active":"2024-01-11T09:46:25.962801+0000","last_peered":"2024-01-11T09:46:25.962801+0000","last_clean":"2024-01-11T09:46:25.962801+0000","last_became_active":"2024-01-11T09:46:25.959431+0000","last_became_peered":"2024-01-11T09:46:25.959431+0000","last_unstale":"2024-01-11T09:46:25.962801+0000","last_undegraded":"2024-01-11T09:46:25.962801+0000","last_fullsized":"2024-01-11T09:46:25.962801+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:58:52.892728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958104+0000","last_change":"2024-01-11T09:46:25.956345+0000","last_active":"2024-01-11T09:46:25.958104+0000","last_peered":"2024-01-11T09:46:25.958104+0000","last_clean":"2024-01-11T09:46:25.958104+0000","last_became_active":"2024-01-11T09:46:25.956080+0000","last_became_peered":"2024-01-11T09:46:25.956080+0000","last_unstale":"2024-01-11T09:46:25.958104+0000","last_undegraded":"2024-01-11T09:46:25.958104+0000","last_fullsized":"2024-01-11T09:46:25.958104+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:30:20.657242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960243+0000","last_change":"2024-01-11T09:46:25.953176+0000","last_active":"2024-01-11T09:46:25.960243+0000","last_peered":"2024-01-11T09:46:25.960243+0000","last_clean":"2024-01-11T09:46:25.960243+0000","last_became_active":"2024-01-11T09:46:25.952276+0000","last_became_peered":"2024-01-11T09:46:25.952276+0000","last_unstale":"2024-01-11T09:46:25.960243+0000","last_undegraded":"2024-01-11T09:46:25.960243+0000","last_fullsized":"2024-01-11T09:46:25.960243+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:37:41.519655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.951599+0000","last_change":"2024-01-11T09:46:25.949114+0000","last_active":"2024-01-11T09:46:25.951599+0000","last_peered":"2024-01-11T09:46:25.951599+0000","last_clean":"2024-01-11T09:46:25.951599+0000","last_became_active":"2024-01-11T09:46:25.948844+0000","last_became_peered":"2024-01-11T09:46:25.948844+0000","last_unstale":"2024-01-11T09:46:25.951599+0000","last_undegraded":"2024-01-11T09:46:25.951599+0000","last_fullsized":"2024-01-11T09:46:25.951599+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:31:58.203452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.962503+0000","last_change":"2024-01-11T09:46:25.954895+0000","last_active":"2024-01-11T09:46:25.962503+0000","last_peered":"2024-01-11T09:46:25.962503+0000","last_clean":"2024-01-11T09:46:25.962503+0000","last_became_active":"2024-01-11T09:46:25.954472+0000","last_became_peered":"2024-01-11T09:46:25.954472+0000","last_unstale":"2024-01-11T09:46:25.962503+0000","last_undegraded":"2024-01-11T09:46:25.962503+0000","last_fullsized":"2024-01-11T09:46:25.962503+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:34:26.903086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.957489+0000","last_change":"2024-01-11T09:46:25.955503+0000","last_active":"2024-01-11T09:46:25.957489+0000","last_peered":"2024-01-11T09:46:25.957489+0000","last_clean":"2024-01-11T09:46:25.957489+0000","last_became_active":"2024-01-11T09:46:25.954870+0000","last_became_peered":"2024-01-11T09:46:25.954870+0000","last_unstale":"2024-01-11T09:46:25.957489+0000","last_undegraded":"2024-01-11T09:46:25.957489+0000","last_fullsized":"2024-01-11T09:46:25.957489+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:06:12.133349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.956147+0000","last_change":"2024-01-11T09:46:25.947226+0000","last_active":"2024-01-11T09:46:25.956147+0000","last_peered":"2024-01-11T09:46:25.956147+0000","last_clean":"2024-01-11T09:46:25.956147+0000","last_became_active":"2024-01-11T09:46:25.946806+0000","last_became_peered":"2024-01-11T09:46:25.946806+0000","last_unstale":"2024-01-11T09:46:25.956147+0000","last_undegraded":"2024-01-11T09:46:25.956147+0000","last_fullsized":"2024-01-11T09:46:25.956147+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:25:09.589100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"234'1","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.242548+0000","last_change":"2024-01-11T09:46:25.236672+0000","last_active":"2024-01-11T09:46:26.242548+0000","last_peered":"2024-01-11T09:46:26.242548+0000","last_clean":"2024-01-11T09:46:26.242548+0000","last_became_active":"2024-01-11T09:46:25.236363+0000","last_became_peered":"2024-01-11T09:46:25.236363+0000","last_unstale":"2024-01-11T09:46:26.242548+0000","last_undegraded":"2024-01-11T09:46:26.242548+0000","last_fullsized":"2024-01-11T09:46:26.242548+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:01:24.091035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960428+0000","last_change":"2024-01-11T09:46:25.951411+0000","last_active":"2024-01-11T09:46:25.960428+0000","last_peered":"2024-01-11T09:46:25.960428+0000","last_clean":"2024-01-11T09:46:25.960428+0000","last_became_active":"2024-01-11T09:46:25.951161+0000","last_became_peered":"2024-01-11T09:46:25.951161+0000","last_unstale":"2024-01-11T09:46:25.960428+0000","last_undegraded":"2024-01-11T09:46:25.960428+0000","last_fullsized":"2024-01-11T09:46:25.960428+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:45:16.842085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"234'2","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.247880+0000","last_change":"2024-01-11T09:46:25.238481+0000","last_active":"2024-01-11T09:46:26.247880+0000","last_peered":"2024-01-11T09:46:26.247880+0000","last_clean":"2024-01-11T09:46:26.247880+0000","last_became_active":"2024-01-11T09:46:25.238203+0000","last_became_peered":"2024-01-11T09:46:25.238203+0000","last_unstale":"2024-01-11T09:46:26.247880+0000","last_undegraded":"2024-01-11T09:46:26.247880+0000","last_fullsized":"2024-01-11T09:46:26.247880+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:42:43.074601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958017+0000","last_change":"2024-01-11T09:46:25.954029+0000","last_active":"2024-01-11T09:46:25.958017+0000","last_peered":"2024-01-11T09:46:25.958017+0000","last_clean":"2024-01-11T09:46:25.958017+0000","last_became_active":"2024-01-11T09:46:25.953690+0000","last_became_peered":"2024-01-11T09:46:25.953690+0000","last_unstale":"2024-01-11T09:46:25.958017+0000","last_undegraded":"2024-01-11T09:46:25.958017+0000","last_fullsized":"2024-01-11T09:46:25.958017+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:26:46.984194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"234'2","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.247094+0000","last_change":"2024-01-11T09:46:25.227884+0000","last_active":"2024-01-11T09:46:26.247094+0000","last_peered":"2024-01-11T09:46:26.247094+0000","last_clean":"2024-01-11T09:46:26.247094+0000","last_became_active":"2024-01-11T09:46:25.227605+0000","last_became_peered":"2024-01-11T09:46:25.227605+0000","last_unstale":"2024-01-11T09:46:26.247094+0000","last_undegraded":"2024-01-11T09:46:26.247094+0000","last_fullsized":"2024-01-11T09:46:26.247094+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:09.116268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.957373+0000","last_change":"2024-01-11T09:46:25.950375+0000","last_active":"2024-01-11T09:46:25.957373+0000","last_peered":"2024-01-11T09:46:25.957373+0000","last_clean":"2024-01-11T09:46:25.957373+0000","last_became_active":"2024-01-11T09:46:25.950114+0000","last_became_peered":"2024-01-11T09:46:25.950114+0000","last_unstale":"2024-01-11T09:46:25.957373+0000","last_undegraded":"2024-01-11T09:46:25.957373+0000","last_fullsized":"2024-01-11T09:46:25.957373+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:25.226766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"234'1","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.242911+0000","last_change":"2024-01-11T09:46:25.233693+0000","last_active":"2024-01-11T09:46:26.242911+0000","last_peered":"2024-01-11T09:46:26.242911+0000","last_clean":"2024-01-11T09:46:26.242911+0000","last_became_active":"2024-01-11T09:46:25.233410+0000","last_became_peered":"2024-01-11T09:46:25.233410+0000","last_unstale":"2024-01-11T09:46:26.242911+0000","last_undegraded":"2024-01-11T09:46:26.242911+0000","last_fullsized":"2024-01-11T09:46:26.242911+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:03:42.704031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958402+0000","last_change":"2024-01-11T09:46:25.955874+0000","last_active":"2024-01-11T09:46:25.958402+0000","last_peered":"2024-01-11T09:46:25.958402+0000","last_clean":"2024-01-11T09:46:25.958402+0000","last_became_active":"2024-01-11T09:46:25.954789+0000","last_became_peered":"2024-01-11T09:46:25.954789+0000","last_unstale":"2024-01-11T09:46:25.958402+0000","last_undegraded":"2024-01-11T09:46:25.958402+0000","last_fullsized":"2024-01-11T09:46:25.958402+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:07.160563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"234'1","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.244983+0000","last_change":"2024-01-11T09:46:25.235297+0000","last_active":"2024-01-11T09:46:26.244983+0000","last_peered":"2024-01-11T09:46:26.244983+0000","last_clean":"2024-01-11T09:46:26.244983+0000","last_became_active":"2024-01-11T09:46:25.234744+0000","last_became_peered":"2024-01-11T09:46:25.234744+0000","last_unstale":"2024-01-11T09:46:26.244983+0000","last_undegraded":"2024-01-11T09:46:26.244983+0000","last_fullsized":"2024-01-11T09:46:26.244983+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:51:44.543489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958203+0000","last_change":"2024-01-11T09:46:25.948668+0000","last_active":"2024-01-11T09:46:25.958203+0000","last_peered":"2024-01-11T09:46:25.958203+0000","last_clean":"2024-01-11T09:46:25.958203+0000","last_became_active":"2024-01-11T09:46:25.948372+0000","last_became_peered":"2024-01-11T09:46:25.948372+0000","last_unstale":"2024-01-11T09:46:25.958203+0000","last_undegraded":"2024-01-11T09:46:25.958203+0000","last_fullsized":"2024-01-11T09:46:25.958203+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:41.154829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.948817+0000","last_change":"2024-01-11T09:46:25.238227+0000","last_active":"2024-01-11T09:46:25.948817+0000","last_peered":"2024-01-11T09:46:25.948817+0000","last_clean":"2024-01-11T09:46:25.948817+0000","last_became_active":"2024-01-11T09:46:25.238013+0000","last_became_peered":"2024-01-11T09:46:25.238013+0000","last_unstale":"2024-01-11T09:46:25.948817+0000","last_undegraded":"2024-01-11T09:46:25.948817+0000","last_fullsized":"2024-01-11T09:46:25.948817+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:36:18.777304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.954020+0000","last_change":"2024-01-11T09:46:25.949014+0000","last_active":"2024-01-11T09:46:25.954020+0000","last_peered":"2024-01-11T09:46:25.954020+0000","last_clean":"2024-01-11T09:46:25.954020+0000","last_became_active":"2024-01-11T09:46:25.948717+0000","last_became_peered":"2024-01-11T09:46:25.948717+0000","last_unstale":"2024-01-11T09:46:25.954020+0000","last_undegraded":"2024-01-11T09:46:25.954020+0000","last_fullsized":"2024-01-11T09:46:25.954020+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:07.160563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.934325+0000","last_change":"2024-01-11T09:46:25.225980+0000","last_active":"2024-01-11T09:46:25.934325+0000","last_peered":"2024-01-11T09:46:25.934325+0000","last_clean":"2024-01-11T09:46:25.934325+0000","last_became_active":"2024-01-11T09:46:25.225444+0000","last_became_peered":"2024-01-11T09:46:25.225444+0000","last_unstale":"2024-01-11T09:46:25.934325+0000","last_undegraded":"2024-01-11T09:46:25.934325+0000","last_fullsized":"2024-01-11T09:46:25.934325+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:30:19.648096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.955447+0000","last_change":"2024-01-11T09:46:25.953219+0000","last_active":"2024-01-11T09:46:25.955447+0000","last_peered":"2024-01-11T09:46:25.955447+0000","last_clean":"2024-01-11T09:46:25.955447+0000","last_became_active":"2024-01-11T09:46:25.952967+0000","last_became_peered":"2024-01-11T09:46:25.952967+0000","last_unstale":"2024-01-11T09:46:25.955447+0000","last_undegraded":"2024-01-11T09:46:25.955447+0000","last_fullsized":"2024-01-11T09:46:25.955447+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:07:13.288202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.942322+0000","last_change":"2024-01-11T09:46:25.237685+0000","last_active":"2024-01-11T09:46:25.942322+0000","last_peered":"2024-01-11T09:46:25.942322+0000","last_clean":"2024-01-11T09:46:25.942322+0000","last_became_active":"2024-01-11T09:46:25.237419+0000","last_became_peered":"2024-01-11T09:46:25.237419+0000","last_unstale":"2024-01-11T09:46:25.942322+0000","last_undegraded":"2024-01-11T09:46:25.942322+0000","last_fullsized":"2024-01-11T09:46:25.942322+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:52.221452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958490+0000","last_change":"2024-01-11T09:46:25.950838+0000","last_active":"2024-01-11T09:46:25.958490+0000","last_peered":"2024-01-11T09:46:25.958490+0000","last_clean":"2024-01-11T09:46:25.958490+0000","last_became_active":"2024-01-11T09:46:25.950551+0000","last_became_peered":"2024-01-11T09:46:25.950551+0000","last_unstale":"2024-01-11T09:46:25.958490+0000","last_undegraded":"2024-01-11T09:46:25.958490+0000","last_fullsized":"2024-01-11T09:46:25.958490+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:39:05.971547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.935051+0000","last_change":"2024-01-11T09:46:25.232121+0000","last_active":"2024-01-11T09:46:25.935051+0000","last_peered":"2024-01-11T09:46:25.935051+0000","last_clean":"2024-01-11T09:46:25.935051+0000","last_became_active":"2024-01-11T09:46:25.231859+0000","last_became_peered":"2024-01-11T09:46:25.231859+0000","last_unstale":"2024-01-11T09:46:25.935051+0000","last_undegraded":"2024-01-11T09:46:25.935051+0000","last_fullsized":"2024-01-11T09:46:25.935051+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:07:12.279056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958248+0000","last_change":"2024-01-11T09:46:25.948806+0000","last_active":"2024-01-11T09:46:25.958248+0000","last_peered":"2024-01-11T09:46:25.958248+0000","last_clean":"2024-01-11T09:46:25.958248+0000","last_became_active":"2024-01-11T09:46:25.948526+0000","last_became_peered":"2024-01-11T09:46:25.948526+0000","last_unstale":"2024-01-11T09:46:25.958248+0000","last_undegraded":"2024-01-11T09:46:25.958248+0000","last_fullsized":"2024-01-11T09:46:25.958248+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:51:19.504238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"234'3","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.248539+0000","last_change":"2024-01-11T09:46:25.238242+0000","last_active":"2024-01-11T09:46:26.248539+0000","last_peered":"2024-01-11T09:46:26.248539+0000","last_clean":"2024-01-11T09:46:26.248539+0000","last_became_active":"2024-01-11T09:46:25.237965+0000","last_became_peered":"2024-01-11T09:46:25.237965+0000","last_unstale":"2024-01-11T09:46:26.248539+0000","last_undegraded":"2024-01-11T09:46:26.248539+0000","last_fullsized":"2024-01-11T09:46:26.248539+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:58:02.388540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.959891+0000","last_change":"2024-01-11T09:46:25.950108+0000","last_active":"2024-01-11T09:46:25.959891+0000","last_peered":"2024-01-11T09:46:25.959891+0000","last_clean":"2024-01-11T09:46:25.959891+0000","last_became_active":"2024-01-11T09:46:25.949796+0000","last_became_peered":"2024-01-11T09:46:25.949796+0000","last_unstale":"2024-01-11T09:46:25.959891+0000","last_undegraded":"2024-01-11T09:46:25.959891+0000","last_fullsized":"2024-01-11T09:46:25.959891+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:11:07.968385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"234'6","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.250731+0000","last_change":"2024-01-11T09:46:25.239177+0000","last_active":"2024-01-11T09:46:26.250731+0000","last_peered":"2024-01-11T09:46:26.250731+0000","last_clean":"2024-01-11T09:46:26.250731+0000","last_became_active":"2024-01-11T09:46:25.238865+0000","last_became_peered":"2024-01-11T09:46:25.238865+0000","last_unstale":"2024-01-11T09:46:26.250731+0000","last_undegraded":"2024-01-11T09:46:26.250731+0000","last_fullsized":"2024-01-11T09:46:26.250731+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:29:50.151952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.962578+0000","last_change":"2024-01-11T09:46:25.951823+0000","last_active":"2024-01-11T09:46:25.962578+0000","last_peered":"2024-01-11T09:46:25.962578+0000","last_clean":"2024-01-11T09:46:25.962578+0000","last_became_active":"2024-01-11T09:46:25.951493+0000","last_became_peered":"2024-01-11T09:46:25.951493+0000","last_unstale":"2024-01-11T09:46:25.962578+0000","last_undegraded":"2024-01-11T09:46:25.962578+0000","last_fullsized":"2024-01-11T09:46:25.962578+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:59:22.974857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"234'2","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.248169+0000","last_change":"2024-01-11T09:46:25.244293+0000","last_active":"2024-01-11T09:46:26.248169+0000","last_peered":"2024-01-11T09:46:26.248169+0000","last_clean":"2024-01-11T09:46:26.248169+0000","last_became_active":"2024-01-11T09:46:25.243959+0000","last_became_peered":"2024-01-11T09:46:25.243959+0000","last_unstale":"2024-01-11T09:46:26.248169+0000","last_undegraded":"2024-01-11T09:46:26.248169+0000","last_fullsized":"2024-01-11T09:46:26.248169+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:13:10.353715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.959593+0000","last_change":"2024-01-11T09:46:25.958245+0000","last_active":"2024-01-11T09:46:25.959593+0000","last_peered":"2024-01-11T09:46:25.959593+0000","last_clean":"2024-01-11T09:46:25.959593+0000","last_became_active":"2024-01-11T09:46:25.957893+0000","last_became_peered":"2024-01-11T09:46:25.957893+0000","last_unstale":"2024-01-11T09:46:25.959593+0000","last_undegraded":"2024-01-11T09:46:25.959593+0000","last_fullsized":"2024-01-11T09:46:25.959593+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:43:22.062566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.935100+0000","last_change":"2024-01-11T09:46:25.231846+0000","last_active":"2024-01-11T09:46:25.935100+0000","last_peered":"2024-01-11T09:46:25.935100+0000","last_clean":"2024-01-11T09:46:25.935100+0000","last_became_active":"2024-01-11T09:46:25.229498+0000","last_became_peered":"2024-01-11T09:46:25.229498+0000","last_unstale":"2024-01-11T09:46:25.935100+0000","last_undegraded":"2024-01-11T09:46:25.935100+0000","last_fullsized":"2024-01-11T09:46:25.935100+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:53:34.130109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.953370+0000","last_change":"2024-01-11T09:46:25.947333+0000","last_active":"2024-01-11T09:46:25.953370+0000","last_peered":"2024-01-11T09:46:25.953370+0000","last_clean":"2024-01-11T09:46:25.953370+0000","last_became_active":"2024-01-11T09:46:25.946307+0000","last_became_peered":"2024-01-11T09:46:25.946307+0000","last_unstale":"2024-01-11T09:46:25.953370+0000","last_undegraded":"2024-01-11T09:46:25.953370+0000","last_fullsized":"2024-01-11T09:46:25.953370+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:32:47.063220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.948916+0000","last_change":"2024-01-11T09:46:25.239044+0000","last_active":"2024-01-11T09:46:25.948916+0000","last_peered":"2024-01-11T09:46:25.948916+0000","last_clean":"2024-01-11T09:46:25.948916+0000","last_became_active":"2024-01-11T09:46:25.238780+0000","last_became_peered":"2024-01-11T09:46:25.238780+0000","last_unstale":"2024-01-11T09:46:25.948916+0000","last_undegraded":"2024-01-11T09:46:25.948916+0000","last_fullsized":"2024-01-11T09:46:25.948916+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:34:37.940301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.964090+0000","last_change":"2024-01-11T09:46:25.962027+0000","last_active":"2024-01-11T09:46:25.964090+0000","last_peered":"2024-01-11T09:46:25.964090+0000","last_clean":"2024-01-11T09:46:25.964090+0000","last_became_active":"2024-01-11T09:46:25.961646+0000","last_became_peered":"2024-01-11T09:46:25.961646+0000","last_unstale":"2024-01-11T09:46:25.964090+0000","last_undegraded":"2024-01-11T09:46:25.964090+0000","last_fullsized":"2024-01-11T09:46:25.964090+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:39:05.971547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"234'8","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.249576+0000","last_change":"2024-01-11T09:46:25.238694+0000","last_active":"2024-01-11T09:46:26.249576+0000","last_peered":"2024-01-11T09:46:26.249576+0000","last_clean":"2024-01-11T09:46:26.249576+0000","last_became_active":"2024-01-11T09:46:25.237565+0000","last_became_peered":"2024-01-11T09:46:25.237565+0000","last_unstale":"2024-01-11T09:46:26.249576+0000","last_undegraded":"2024-01-11T09:46:26.249576+0000","last_fullsized":"2024-01-11T09:46:26.249576+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:25:08.579955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.954111+0000","last_change":"2024-01-11T09:46:25.946186+0000","last_active":"2024-01-11T09:46:25.954111+0000","last_peered":"2024-01-11T09:46:25.954111+0000","last_clean":"2024-01-11T09:46:25.954111+0000","last_became_active":"2024-01-11T09:46:25.945907+0000","last_became_peered":"2024-01-11T09:46:25.945907+0000","last_unstale":"2024-01-11T09:46:25.954111+0000","last_undegraded":"2024-01-11T09:46:25.954111+0000","last_fullsized":"2024-01-11T09:46:25.954111+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:26:51.516156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"234'1","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.242887+0000","last_change":"2024-01-11T09:46:25.241687+0000","last_active":"2024-01-11T09:46:26.242887+0000","last_peered":"2024-01-11T09:46:26.242887+0000","last_clean":"2024-01-11T09:46:26.242887+0000","last_became_active":"2024-01-11T09:46:25.241369+0000","last_became_peered":"2024-01-11T09:46:25.241369+0000","last_unstale":"2024-01-11T09:46:26.242887+0000","last_undegraded":"2024-01-11T09:46:26.242887+0000","last_fullsized":"2024-01-11T09:46:26.242887+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:40.145683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958691+0000","last_change":"2024-01-11T09:46:25.951531+0000","last_active":"2024-01-11T09:46:25.958691+0000","last_peered":"2024-01-11T09:46:25.958691+0000","last_clean":"2024-01-11T09:46:25.958691+0000","last_became_active":"2024-01-11T09:46:25.950791+0000","last_became_peered":"2024-01-11T09:46:25.950791+0000","last_unstale":"2024-01-11T09:46:25.958691+0000","last_undegraded":"2024-01-11T09:46:25.958691+0000","last_fullsized":"2024-01-11T09:46:25.958691+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:09:35.408321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"234'2","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.249024+0000","last_change":"2024-01-11T09:46:25.234950+0000","last_active":"2024-01-11T09:46:26.249024+0000","last_peered":"2024-01-11T09:46:26.249024+0000","last_clean":"2024-01-11T09:46:26.249024+0000","last_became_active":"2024-01-11T09:46:25.234510+0000","last_became_peered":"2024-01-11T09:46:25.234510+0000","last_unstale":"2024-01-11T09:46:26.249024+0000","last_undegraded":"2024-01-11T09:46:26.249024+0000","last_fullsized":"2024-01-11T09:46:26.249024+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:33:13.743309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.963426+0000","last_change":"2024-01-11T09:46:25.961077+0000","last_active":"2024-01-11T09:46:25.963426+0000","last_peered":"2024-01-11T09:46:25.963426+0000","last_clean":"2024-01-11T09:46:25.963426+0000","last_became_active":"2024-01-11T09:46:25.960690+0000","last_became_peered":"2024-01-11T09:46:25.960690+0000","last_unstale":"2024-01-11T09:46:25.963426+0000","last_undegraded":"2024-01-11T09:46:25.963426+0000","last_fullsized":"2024-01-11T09:46:25.963426+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:59:52.045856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"234'1","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.246122+0000","last_change":"2024-01-11T09:46:25.235005+0000","last_active":"2024-01-11T09:46:26.246122+0000","last_peered":"2024-01-11T09:46:26.246122+0000","last_clean":"2024-01-11T09:46:26.246122+0000","last_became_active":"2024-01-11T09:46:25.234748+0000","last_became_peered":"2024-01-11T09:46:25.234748+0000","last_unstale":"2024-01-11T09:46:26.246122+0000","last_undegraded":"2024-01-11T09:46:26.246122+0000","last_fullsized":"2024-01-11T09:46:26.246122+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:29:50.151952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.957997+0000","last_change":"2024-01-11T09:46:25.952457+0000","last_active":"2024-01-11T09:46:25.957997+0000","last_peered":"2024-01-11T09:46:25.957997+0000","last_clean":"2024-01-11T09:46:25.957997+0000","last_became_active":"2024-01-11T09:46:25.948394+0000","last_became_peered":"2024-01-11T09:46:25.948394+0000","last_unstale":"2024-01-11T09:46:25.957997+0000","last_undegraded":"2024-01-11T09:46:25.957997+0000","last_fullsized":"2024-01-11T09:46:25.957997+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:49:33.997022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.944053+0000","last_change":"2024-01-11T09:46:25.242589+0000","last_active":"2024-01-11T09:46:25.944053+0000","last_peered":"2024-01-11T09:46:25.944053+0000","last_clean":"2024-01-11T09:46:25.944053+0000","last_became_active":"2024-01-11T09:46:25.241879+0000","last_became_peered":"2024-01-11T09:46:25.241879+0000","last_unstale":"2024-01-11T09:46:25.944053+0000","last_undegraded":"2024-01-11T09:46:25.944053+0000","last_fullsized":"2024-01-11T09:46:25.944053+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:40.145683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958806+0000","last_change":"2024-01-11T09:46:25.953778+0000","last_active":"2024-01-11T09:46:25.958806+0000","last_peered":"2024-01-11T09:46:25.958806+0000","last_clean":"2024-01-11T09:46:25.958806+0000","last_became_active":"2024-01-11T09:46:25.953471+0000","last_became_peered":"2024-01-11T09:46:25.953471+0000","last_unstale":"2024-01-11T09:46:25.958806+0000","last_undegraded":"2024-01-11T09:46:25.958806+0000","last_fullsized":"2024-01-11T09:46:25.958806+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:30:20.657242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"234'2","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.246689+0000","last_change":"2024-01-11T09:46:25.236418+0000","last_active":"2024-01-11T09:46:26.246689+0000","last_peered":"2024-01-11T09:46:26.246689+0000","last_clean":"2024-01-11T09:46:26.246689+0000","last_became_active":"2024-01-11T09:46:25.236155+0000","last_became_peered":"2024-01-11T09:46:25.236155+0000","last_unstale":"2024-01-11T09:46:26.246689+0000","last_undegraded":"2024-01-11T09:46:26.246689+0000","last_fullsized":"2024-01-11T09:46:26.246689+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:59:13.998537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.957442+0000","last_change":"2024-01-11T09:46:25.954393+0000","last_active":"2024-01-11T09:46:25.957442+0000","last_peered":"2024-01-11T09:46:25.957442+0000","last_clean":"2024-01-11T09:46:25.957442+0000","last_became_active":"2024-01-11T09:46:25.954139+0000","last_became_peered":"2024-01-11T09:46:25.954139+0000","last_unstale":"2024-01-11T09:46:25.957442+0000","last_undegraded":"2024-01-11T09:46:25.957442+0000","last_fullsized":"2024-01-11T09:46:25.957442+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:33:14.752455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.943596+0000","last_change":"2024-01-11T09:46:25.243586+0000","last_active":"2024-01-11T09:46:25.943596+0000","last_peered":"2024-01-11T09:46:25.943596+0000","last_clean":"2024-01-11T09:46:25.943596+0000","last_became_active":"2024-01-11T09:46:25.243184+0000","last_became_peered":"2024-01-11T09:46:25.243184+0000","last_unstale":"2024-01-11T09:46:25.943596+0000","last_undegraded":"2024-01-11T09:46:25.943596+0000","last_fullsized":"2024-01-11T09:46:25.943596+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:01:24.091035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960346+0000","last_change":"2024-01-11T09:46:25.953319+0000","last_active":"2024-01-11T09:46:25.960346+0000","last_peered":"2024-01-11T09:46:25.960346+0000","last_clean":"2024-01-11T09:46:25.960346+0000","last_became_active":"2024-01-11T09:46:25.953063+0000","last_became_peered":"2024-01-11T09:46:25.953063+0000","last_unstale":"2024-01-11T09:46:25.960346+0000","last_undegraded":"2024-01-11T09:46:25.960346+0000","last_fullsized":"2024-01-11T09:46:25.960346+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:31:56.044420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.945981+0000","last_change":"2024-01-11T09:46:25.243348+0000","last_active":"2024-01-11T09:46:25.945981+0000","last_peered":"2024-01-11T09:46:25.945981+0000","last_clean":"2024-01-11T09:46:25.945981+0000","last_became_active":"2024-01-11T09:46:25.243043+0000","last_became_peered":"2024-01-11T09:46:25.243043+0000","last_unstale":"2024-01-11T09:46:25.945981+0000","last_undegraded":"2024-01-11T09:46:25.945981+0000","last_fullsized":"2024-01-11T09:46:25.945981+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:13:10.353715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.962789+0000","last_change":"2024-01-11T09:46:25.956539+0000","last_active":"2024-01-11T09:46:25.962789+0000","last_peered":"2024-01-11T09:46:25.962789+0000","last_clean":"2024-01-11T09:46:25.962789+0000","last_became_active":"2024-01-11T09:46:25.953109+0000","last_became_peered":"2024-01-11T09:46:25.953109+0000","last_unstale":"2024-01-11T09:46:25.962789+0000","last_undegraded":"2024-01-11T09:46:25.962789+0000","last_fullsized":"2024-01-11T09:46:25.962789+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:39:37.637123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.948443+0000","last_change":"2024-01-11T09:46:25.234212+0000","last_active":"2024-01-11T09:46:25.948443+0000","last_peered":"2024-01-11T09:46:25.948443+0000","last_clean":"2024-01-11T09:46:25.948443+0000","last_became_active":"2024-01-11T09:46:25.233918+0000","last_became_peered":"2024-01-11T09:46:25.233918+0000","last_unstale":"2024-01-11T09:46:25.948443+0000","last_undegraded":"2024-01-11T09:46:25.948443+0000","last_fullsized":"2024-01-11T09:46:25.948443+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:39:17.328191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.961103+0000","last_change":"2024-01-11T09:46:25.953810+0000","last_active":"2024-01-11T09:46:25.961103+0000","last_peered":"2024-01-11T09:46:25.961103+0000","last_clean":"2024-01-11T09:46:25.961103+0000","last_became_active":"2024-01-11T09:46:25.953201+0000","last_became_peered":"2024-01-11T09:46:25.953201+0000","last_unstale":"2024-01-11T09:46:25.961103+0000","last_undegraded":"2024-01-11T09:46:25.961103+0000","last_fullsized":"2024-01-11T09:46:25.961103+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:25.226766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"234'2","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.248570+0000","last_change":"2024-01-11T09:46:25.245359+0000","last_active":"2024-01-11T09:46:26.248570+0000","last_peered":"2024-01-11T09:46:26.248570+0000","last_clean":"2024-01-11T09:46:26.248570+0000","last_became_active":"2024-01-11T09:46:25.244994+0000","last_became_peered":"2024-01-11T09:46:25.244994+0000","last_unstale":"2024-01-11T09:46:26.248570+0000","last_undegraded":"2024-01-11T09:46:26.248570+0000","last_fullsized":"2024-01-11T09:46:26.248570+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:42:13.269515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.956181+0000","last_change":"2024-01-11T09:46:25.952007+0000","last_active":"2024-01-11T09:46:25.956181+0000","last_peered":"2024-01-11T09:46:25.956181+0000","last_clean":"2024-01-11T09:46:25.956181+0000","last_became_active":"2024-01-11T09:46:25.951401+0000","last_became_peered":"2024-01-11T09:46:25.951401+0000","last_unstale":"2024-01-11T09:46:25.956181+0000","last_undegraded":"2024-01-11T09:46:25.956181+0000","last_fullsized":"2024-01-11T09:46:25.956181+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:29:18.608960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.939276+0000","last_change":"2024-01-11T09:46:25.236787+0000","last_active":"2024-01-11T09:46:25.939276+0000","last_peered":"2024-01-11T09:46:25.939276+0000","last_clean":"2024-01-11T09:46:25.939276+0000","last_became_active":"2024-01-11T09:46:25.236503+0000","last_became_peered":"2024-01-11T09:46:25.236503+0000","last_unstale":"2024-01-11T09:46:25.939276+0000","last_undegraded":"2024-01-11T09:46:25.939276+0000","last_fullsized":"2024-01-11T09:46:25.939276+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:40:17.481276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960608+0000","last_change":"2024-01-11T09:46:25.953624+0000","last_active":"2024-01-11T09:46:25.960608+0000","last_peered":"2024-01-11T09:46:25.960608+0000","last_clean":"2024-01-11T09:46:25.960608+0000","last_became_active":"2024-01-11T09:46:25.953347+0000","last_became_peered":"2024-01-11T09:46:25.953347+0000","last_unstale":"2024-01-11T09:46:25.960608+0000","last_undegraded":"2024-01-11T09:46:25.960608+0000","last_fullsized":"2024-01-11T09:46:25.960608+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:56:15.022301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.946699+0000","last_change":"2024-01-11T09:46:25.238938+0000","last_active":"2024-01-11T09:46:25.946699+0000","last_peered":"2024-01-11T09:46:25.946699+0000","last_clean":"2024-01-11T09:46:25.946699+0000","last_became_active":"2024-01-11T09:46:25.238214+0000","last_became_peered":"2024-01-11T09:46:25.238214+0000","last_unstale":"2024-01-11T09:46:25.946699+0000","last_undegraded":"2024-01-11T09:46:25.946699+0000","last_fullsized":"2024-01-11T09:46:25.946699+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:30:58.825981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960351+0000","last_change":"2024-01-11T09:46:25.954867+0000","last_active":"2024-01-11T09:46:25.960351+0000","last_peered":"2024-01-11T09:46:25.960351+0000","last_clean":"2024-01-11T09:46:25.960351+0000","last_became_active":"2024-01-11T09:46:25.954613+0000","last_became_peered":"2024-01-11T09:46:25.954613+0000","last_unstale":"2024-01-11T09:46:25.960351+0000","last_undegraded":"2024-01-11T09:46:25.960351+0000","last_fullsized":"2024-01-11T09:46:25.960351+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:11:07.968385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.944325+0000","last_change":"2024-01-11T09:46:25.244239+0000","last_active":"2024-01-11T09:46:25.944325+0000","last_peered":"2024-01-11T09:46:25.944325+0000","last_clean":"2024-01-11T09:46:25.944325+0000","last_became_active":"2024-01-11T09:46:25.243663+0000","last_became_peered":"2024-01-11T09:46:25.243663+0000","last_unstale":"2024-01-11T09:46:25.944325+0000","last_undegraded":"2024-01-11T09:46:25.944325+0000","last_fullsized":"2024-01-11T09:46:25.944325+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:23:51.951171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.964142+0000","last_change":"2024-01-11T09:46:25.962471+0000","last_active":"2024-01-11T09:46:25.964142+0000","last_peered":"2024-01-11T09:46:25.964142+0000","last_clean":"2024-01-11T09:46:25.964142+0000","last_became_active":"2024-01-11T09:46:25.959247+0000","last_became_peered":"2024-01-11T09:46:25.959247+0000","last_unstale":"2024-01-11T09:46:25.964142+0000","last_undegraded":"2024-01-11T09:46:25.964142+0000","last_fullsized":"2024-01-11T09:46:25.964142+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:31:56.044420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.950613+0000","last_change":"2024-01-11T09:46:25.242486+0000","last_active":"2024-01-11T09:46:25.950613+0000","last_peered":"2024-01-11T09:46:25.950613+0000","last_clean":"2024-01-11T09:46:25.950613+0000","last_became_active":"2024-01-11T09:46:25.242231+0000","last_became_peered":"2024-01-11T09:46:25.242231+0000","last_unstale":"2024-01-11T09:46:25.950613+0000","last_undegraded":"2024-01-11T09:46:25.950613+0000","last_fullsized":"2024-01-11T09:46:25.950613+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:23:51.951171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958879+0000","last_change":"2024-01-11T09:46:25.954929+0000","last_active":"2024-01-11T09:46:25.958879+0000","last_peered":"2024-01-11T09:46:25.958879+0000","last_clean":"2024-01-11T09:46:25.958879+0000","last_became_active":"2024-01-11T09:46:25.951640+0000","last_became_peered":"2024-01-11T09:46:25.951640+0000","last_unstale":"2024-01-11T09:46:25.958879+0000","last_undegraded":"2024-01-11T09:46:25.958879+0000","last_fullsized":"2024-01-11T09:46:25.958879+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:28:55.053850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.935714+0000","last_change":"2024-01-11T09:46:25.234604+0000","last_active":"2024-01-11T09:46:25.935714+0000","last_peered":"2024-01-11T09:46:25.935714+0000","last_clean":"2024-01-11T09:46:25.935714+0000","last_became_active":"2024-01-11T09:46:25.234348+0000","last_became_peered":"2024-01-11T09:46:25.234348+0000","last_unstale":"2024-01-11T09:46:25.935714+0000","last_undegraded":"2024-01-11T09:46:25.935714+0000","last_fullsized":"2024-01-11T09:46:25.935714+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:01:05.259333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960490+0000","last_change":"2024-01-11T09:46:25.956295+0000","last_active":"2024-01-11T09:46:25.960490+0000","last_peered":"2024-01-11T09:46:25.960490+0000","last_clean":"2024-01-11T09:46:25.960490+0000","last_became_active":"2024-01-11T09:46:25.949169+0000","last_became_peered":"2024-01-11T09:46:25.949169+0000","last_unstale":"2024-01-11T09:46:25.960490+0000","last_undegraded":"2024-01-11T09:46:25.960490+0000","last_fullsized":"2024-01-11T09:46:25.960490+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:07.160563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.934226+0000","last_change":"2024-01-11T09:46:25.231375+0000","last_active":"2024-01-11T09:46:25.934226+0000","last_peered":"2024-01-11T09:46:25.934226+0000","last_clean":"2024-01-11T09:46:25.934226+0000","last_became_active":"2024-01-11T09:46:25.231115+0000","last_became_peered":"2024-01-11T09:46:25.231115+0000","last_unstale":"2024-01-11T09:46:25.934226+0000","last_undegraded":"2024-01-11T09:46:25.934226+0000","last_fullsized":"2024-01-11T09:46:25.934226+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:06:11.124203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960030+0000","last_change":"2024-01-11T09:46:25.957641+0000","last_active":"2024-01-11T09:46:25.960030+0000","last_peered":"2024-01-11T09:46:25.960030+0000","last_clean":"2024-01-11T09:46:25.960030+0000","last_became_active":"2024-01-11T09:46:25.955768+0000","last_became_peered":"2024-01-11T09:46:25.955768+0000","last_unstale":"2024-01-11T09:46:25.960030+0000","last_undegraded":"2024-01-11T09:46:25.960030+0000","last_fullsized":"2024-01-11T09:46:25.960030+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:58.843659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.934820+0000","last_change":"2024-01-11T09:46:25.239061+0000","last_active":"2024-01-11T09:46:25.934820+0000","last_peered":"2024-01-11T09:46:25.934820+0000","last_clean":"2024-01-11T09:46:25.934820+0000","last_became_active":"2024-01-11T09:46:25.238802+0000","last_became_peered":"2024-01-11T09:46:25.238802+0000","last_unstale":"2024-01-11T09:46:25.934820+0000","last_undegraded":"2024-01-11T09:46:25.934820+0000","last_fullsized":"2024-01-11T09:46:25.934820+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:57.834513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.951329+0000","last_change":"2024-01-11T09:46:25.948656+0000","last_active":"2024-01-11T09:46:25.951329+0000","last_peered":"2024-01-11T09:46:25.951329+0000","last_clean":"2024-01-11T09:46:25.951329+0000","last_became_active":"2024-01-11T09:46:25.948273+0000","last_became_peered":"2024-01-11T09:46:25.948273+0000","last_unstale":"2024-01-11T09:46:25.951329+0000","last_undegraded":"2024-01-11T09:46:25.951329+0000","last_fullsized":"2024-01-11T09:46:25.951329+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:36:07.163334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"234'4","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.247953+0000","last_change":"2024-01-11T09:46:25.235720+0000","last_active":"2024-01-11T09:46:26.247953+0000","last_peered":"2024-01-11T09:46:26.247953+0000","last_clean":"2024-01-11T09:46:26.247953+0000","last_became_active":"2024-01-11T09:46:25.235445+0000","last_became_peered":"2024-01-11T09:46:25.235445+0000","last_unstale":"2024-01-11T09:46:26.247953+0000","last_undegraded":"2024-01-11T09:46:26.247953+0000","last_fullsized":"2024-01-11T09:46:26.247953+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:53:59.525452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"221'388","reported_seq":1295,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.945190+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:46:25.945190+0000","last_peered":"2024-01-11T09:46:25.945190+0000","last_clean":"2024-01-11T09:46:25.945190+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:46:25.945190+0000","last_undegraded":"2024-01-11T09:46:25.945190+0000","last_fullsized":"2024-01-11T09:46:25.945190+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":651,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":651,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":388,"ondisk_log_size":388,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575982,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300764,"kb_used_data":3476,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071076,"statfs":{"total":96636764160,"available":96328781824,"internally_reserved":0,"allocated":3559424,"data_stored":2014654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.69599999999999995,"15min":0.73299999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.36599999999999999},"max":{"1min":0.93000000000000005,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.72599999999999998,"15min":0.747},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.026,"5min":1.0609999999999999,"15min":1.0669999999999999},"last":0.80100000000000005}]},{"osd":1,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.70699999999999996,"15min":0.73299999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41599999999999998},"max":{"1min":0.91600000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.71799999999999997,"15min":0.74199999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.95899999999999996,"5min":0.999,"15min":1.0820000000000001},"last":0.61399999999999999}]},{"osd":2,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.73599999999999999,"15min":0.74399999999999999},"min":{"1min":0.44500000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.93799999999999994,"5min":0.94399999999999995,"15min":1.0600000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73199999999999998,"15min":0.754},"min":{"1min":0.503,"5min":0.44800000000000001,"15min":0.44500000000000001},"max":{"1min":0.92800000000000005,"5min":0.96999999999999997,"15min":1.153},"last":0.66000000000000003}]},{"osd":3,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73799999999999999,"15min":0.76700000000000002},"min":{"1min":0.51600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.93000000000000005,"5min":0.96699999999999997,"15min":1.0660000000000001},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73699999999999999,"15min":0.75600000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.94099999999999995,"5min":1,"15min":1.0880000000000001},"last":0.74199999999999999}]},{"osd":4,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.432,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0720000000000001,"5min":1.0860000000000001,"15min":1.107},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.629,"15min":0.63800000000000001},"min":{"1min":0.47699999999999998,"5min":0.44400000000000001,"15min":0.39700000000000002},"max":{"1min":0.89000000000000001,"5min":0.995,"15min":1.0960000000000001},"last":0.76300000000000001}]},{"osd":5,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.61199999999999999,"15min":0.63800000000000001},"min":{"1min":0.48999999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86599999999999999,"5min":0.86699999999999999,"15min":1.0109999999999999},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.67900000000000005,"15min":0.68400000000000005},"min":{"1min":0.46899999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.92200000000000004}]},{"osd":6,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.76300000000000001,"15min":0.77900000000000003},"min":{"1min":0.55900000000000005,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.747,"15min":0.76000000000000001},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.45800000000000002},"max":{"1min":0.95899999999999996,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":1.002}]}]},{"osd":6,"up_from":13,"seq":55834575981,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312476,"kb_used_data":2900,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059364,"statfs":{"total":96636764160,"available":96316788736,"internally_reserved":0,"allocated":2969600,"data_stored":1424792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.65200000000000002,"15min":0.70399999999999996},"min":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89100000000000001,"5min":0.94199999999999995,"15min":0.999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.71299999999999997,"15min":0.73899999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.216},"last":0.60899999999999999}]},{"osd":1,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.67500000000000004,"15min":0.73199999999999998},"min":{"1min":0.44900000000000001,"5min":0.42799999999999999,"15min":0.41699999999999998},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":1.3129999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.68899999999999995,"15min":0.72499999999999998},"min":{"1min":0.44400000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.95499999999999996,"5min":0.98299999999999998,"15min":1.0840000000000001},"last":0.503}]},{"osd":2,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.70699999999999996,"15min":0.74299999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.0069999999999999,"5min":1.034,"15min":1.4179999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.72099999999999997,"15min":0.75},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":0.439}]},{"osd":3,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.72999999999999998,"15min":0.754},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.92600000000000005,"5min":1.012,"15min":1.014},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.70599999999999996,"15min":0.76200000000000001},"min":{"1min":0.498,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.91300000000000003,"5min":0.95999999999999996,"15min":1.165},"last":0.64300000000000002}]},{"osd":4,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.38400000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78000000000000003,"5min":0.91700000000000004,"15min":1.0309999999999999},"last":0.627},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.626,"15min":0.61599999999999999},"min":{"1min":0.40799999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92100000000000004,"5min":0.94799999999999995,"15min":1.1259999999999999},"last":0.66300000000000003}]},{"osd":5,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.65000000000000002,"15min":0.65900000000000003},"min":{"1min":0.44600000000000001,"5min":0.41499999999999998,"15min":0.40000000000000002},"max":{"1min":0.93100000000000005,"5min":0.97899999999999998,"15min":1.369},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70299999999999996,"15min":0.70499999999999996},"min":{"1min":0.56799999999999995,"5min":0.46700000000000003,"15min":0.45400000000000001},"max":{"1min":0.85999999999999999,"5min":0.999,"15min":1.0960000000000001},"last":0.67700000000000005}]},{"osd":7,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.70699999999999996,"15min":0.70899999999999996},"min":{"1min":0.47199999999999998,"5min":0.45600000000000002,"15min":0.45100000000000001},"max":{"1min":0.91300000000000003,"5min":1.089,"15min":1.484},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.754,"15min":0.76700000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.98299999999999998,"5min":1.1379999999999999,"15min":1.603},"last":0.74099999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575983,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308392,"kb_used_data":2912,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063448,"statfs":{"total":96636764160,"available":96320970752,"internally_reserved":0,"allocated":2981888,"data_stored":1425876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.51400000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.315},"max":{"1min":0.69799999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.439},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.59899999999999998,"15min":0.61899999999999999},"min":{"1min":0.41899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.75600000000000001,"5min":0.85199999999999998,"15min":0.95099999999999996},"last":0.48399999999999999}]},{"osd":2,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.50700000000000001,"15min":0.51200000000000001},"min":{"1min":0.34999999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66600000000000004,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.84099999999999997,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.505}]},{"osd":3,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.46400000000000002,"5min":0.436,"15min":0.42699999999999999},"max":{"1min":0.85199999999999998,"5min":0.91600000000000004,"15min":0.96699999999999997},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.58099999999999996,"15min":0.58199999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.71299999999999997,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.42099999999999999}]},{"osd":4,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.79500000000000004,"15min":0.81599999999999995},"min":{"1min":0.54600000000000004,"5min":0.51400000000000001,"15min":0.505},"max":{"1min":0.875,"5min":1.214,"15min":1.2969999999999999},"last":0.751},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.84399999999999997,"15min":0.85899999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1000000000000001,"5min":1.1879999999999999,"15min":1.1890000000000001},"last":0.73199999999999998}]},{"osd":5,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.82599999999999996,"15min":0.82799999999999996},"min":{"1min":0.51800000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.996,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.80800000000000005,"15min":0.80700000000000005},"min":{"1min":0.61899999999999999,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":1.0609999999999999,"5min":1.341,"15min":1.341},"last":0.67000000000000004}]},{"osd":6,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.83499999999999996,"15min":0.84699999999999998},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.56799999999999995},"max":{"1min":1.0620000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.84499999999999997,"15min":0.85399999999999998},"min":{"1min":0.60099999999999998,"5min":0.56899999999999995,"15min":0.55900000000000005},"max":{"1min":1.077,"5min":1.177,"15min":1.177},"last":0.83799999999999997}]},{"osd":7,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.88800000000000001,"15min":0.88900000000000001},"min":{"1min":0.64400000000000002,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.1359999999999999,"5min":1.421,"15min":1.421},"last":0.622},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.86299999999999999,"15min":0.85699999999999998},"min":{"1min":0.64500000000000002,"5min":0.60099999999999998,"15min":0.54500000000000004},"max":{"1min":0.96099999999999997,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.71299999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575982,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308956,"kb_used_data":3476,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062884,"statfs":{"total":96636764160,"available":96320393216,"internally_reserved":0,"allocated":3559424,"data_stored":2014618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.50700000000000001,"15min":0.51800000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.70999999999999996,"5min":0.84699999999999998,"15min":0.875},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.58399999999999996,"15min":0.59599999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77200000000000002,"5min":0.81599999999999995,"15min":0.874},"last":0.502}]},{"osd":2,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.53500000000000003,"15min":0.56499999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.61499999999999999,"5min":0.83099999999999996,"15min":0.97499999999999998},"last":0.308},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.56299999999999994,"15min":0.57999999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71899999999999997,"5min":0.84699999999999998,"15min":0.86399999999999999},"last":0.33700000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.51400000000000001,"15min":0.54000000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.73699999999999999,"5min":0.79000000000000004,"15min":0.95399999999999996},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.621,"15min":0.64800000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80200000000000005,"5min":0.90300000000000002,"15min":1.0389999999999999},"last":0.47299999999999998}]},{"osd":4,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.81399999999999995,"15min":0.81599999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0629999999999999,"5min":1.1779999999999999,"15min":1.304},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.80000000000000004,"15min":0.80600000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.93000000000000005,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.48799999999999999}]},{"osd":5,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.82299999999999995,"15min":0.83899999999999997},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0069999999999999,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.82899999999999996,"15min":0.84199999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.1100000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.623}]},{"osd":6,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.81000000000000005,"15min":0.82399999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.1000000000000001,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.84199999999999997,"15min":0.86599999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":0.95899999999999996,"5min":1.256,"15min":1.256},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.85599999999999998,"15min":0.86599999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.151,"5min":1.28,"15min":1.28},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.85899999999999999,"15min":0.872},"min":{"1min":0.57499999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.054,"5min":1.238,"15min":1.2410000000000001},"last":0.64100000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575981,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300184,"kb_used_data":2896,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071656,"statfs":{"total":96636764160,"available":96329375744,"internally_reserved":0,"allocated":2965504,"data_stored":1424306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48699999999999999,"15min":0.502},"min":{"1min":0.32800000000000001,"5min":0.29899999999999999,"15min":0.27900000000000003},"max":{"1min":0.625,"5min":0.72599999999999998,"15min":0.85399999999999998},"last":0.436},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.56499999999999995},"min":{"1min":0.38800000000000001,"5min":0.34399999999999997,"15min":0.34300000000000003},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":1.6359999999999999},"last":0.51300000000000001}]},{"osd":1,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.39500000000000002,"5min":0.38200000000000001,"15min":0.33500000000000002},"max":{"1min":0.80900000000000005,"5min":0.88800000000000001,"15min":0.93000000000000005},"last":0.5},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58899999999999997,"15min":0.59899999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.73099999999999998,"5min":0.97299999999999998,"15min":1.145},"last":0.46100000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.75,"5min":0.83099999999999996,"15min":0.96399999999999997},"last":0.376},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.63,"15min":0.65500000000000003},"min":{"1min":0.442,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.83599999999999997,"5min":0.90700000000000003,"15min":1.3620000000000001},"last":0.59999999999999998}]},{"osd":4,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.78200000000000003,"15min":0.79800000000000004},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.54300000000000004},"max":{"1min":0.95299999999999996,"5min":1.071,"15min":1.1519999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.78700000000000003,"15min":0.80000000000000004},"min":{"1min":0.54500000000000004,"5min":0.505,"15min":0.505},"max":{"1min":0.96299999999999997,"5min":1.161,"15min":1.161},"last":0.61099999999999999}]},{"osd":5,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.83099999999999996,"15min":0.86799999999999999},"min":{"1min":0.625,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.996,"5min":1.228,"15min":1.429},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.85099999999999998,"15min":0.86299999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.0389999999999999,"5min":1.1850000000000001,"15min":1.2170000000000001},"last":0.73899999999999999}]},{"osd":6,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.80000000000000004,"15min":0.82599999999999996},"min":{"1min":0.55500000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0049999999999999,"5min":1.1659999999999999,"15min":1.4139999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.83499999999999996,"15min":0.86399999999999999},"min":{"1min":0.63400000000000001,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.96599999999999997,"5min":1.2010000000000001,"15min":1.593},"last":0.77300000000000002}]},{"osd":7,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.86799999999999999,"15min":0.89200000000000002},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.57299999999999995},"max":{"1min":1.0129999999999999,"5min":1.137,"15min":2.0539999999999998},"last":0.627},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.86499999999999999,"15min":0.88100000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.55500000000000005},"max":{"1min":1.069,"5min":1.2270000000000001,"15min":1.2490000000000001},"last":0.75600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575981,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300184,"kb_used_data":2896,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071656,"statfs":{"total":96636764160,"available":96329375744,"internally_reserved":0,"allocated":2965504,"data_stored":1424770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.502,"15min":0.52000000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.314},"max":{"1min":0.76100000000000001,"5min":0.89700000000000002,"15min":0.90000000000000002},"last":0.318},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.56200000000000006,"15min":0.58699999999999997},"min":{"1min":0.372,"5min":0.36899999999999999,"15min":0.36099999999999999},"max":{"1min":0.67900000000000005,"5min":1.014,"15min":1.0449999999999999},"last":0.46999999999999997}]},{"osd":1,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.495,"15min":0.51100000000000001},"min":{"1min":0.29399999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63500000000000001,"5min":0.84199999999999997,"15min":1.304},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.38700000000000001},"max":{"1min":0.75,"5min":0.84099999999999997,"15min":1.4590000000000001},"last":0.372}]},{"osd":2,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58299999999999996,"15min":0.60599999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.72999999999999998,"5min":0.93700000000000006,"15min":1.04},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.64200000000000002,"15min":0.65800000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44700000000000001},"max":{"1min":0.83499999999999996,"5min":0.89800000000000002,"15min":0.92000000000000004},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76100000000000001,"15min":0.79000000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.92900000000000005,"5min":1.206,"15min":1.206},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.77800000000000002,"15min":0.81299999999999994},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.871,"5min":1.171,"15min":1.1830000000000001},"last":0.82699999999999996}]},{"osd":5,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.79600000000000004,"15min":0.82099999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.95399999999999996,"5min":1.1619999999999999,"15min":1.234},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81000000000000005,"15min":0.84799999999999998},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.3440000000000001},"last":0.93700000000000006}]},{"osd":6,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82999999999999996,"15min":0.85899999999999999},"min":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.077,"5min":1.1739999999999999,"15min":1.2170000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.80500000000000005,"15min":0.82899999999999996},"min":{"1min":0.55300000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.97099999999999997,"5min":1.181,"15min":1.2190000000000001},"last":0.81399999999999995}]},{"osd":7,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.83699999999999997,"15min":0.86799999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0269999999999999,"5min":1.224,"15min":1.264},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.86099999999999999,"15min":0.89500000000000002},"min":{"1min":0.59299999999999997,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.0209999999999999,"5min":1.1539999999999999,"15min":1.24},"last":0.89500000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575982,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308392,"kb_used_data":2912,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063448,"statfs":{"total":96636764160,"available":96320970752,"internally_reserved":0,"allocated":2981888,"data_stored":1425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.75800000000000001,"15min":0.77200000000000002},"min":{"1min":0.54000000000000004,"5min":0.41199999999999998,"15min":0.39900000000000002},"max":{"1min":0.91200000000000003,"5min":0.98499999999999999,"15min":1.119},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.98499999999999999,"5min":1.853,"15min":1.853},"last":0.64000000000000001}]},{"osd":1,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.73099999999999998,"15min":0.74399999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.41399999999999998},"max":{"1min":0.90500000000000003,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.72999999999999998,"15min":0.73599999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.95899999999999996,"5min":1.639,"15min":1.639},"last":0.72699999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.747,"15min":0.751},"min":{"1min":0.54800000000000004,"5min":0.46500000000000002,"15min":0.44700000000000001},"max":{"1min":0.94499999999999995,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.75800000000000001,"15min":0.752},"min":{"1min":0.503,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.002,"5min":1.0249999999999999,"15min":1.0700000000000001},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.73899999999999999,"15min":0.745},"min":{"1min":0.52100000000000002,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.94799999999999995,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.441},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.78300000000000003,"15min":0.78000000000000003},"min":{"1min":0.56499999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.02,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.61399999999999999}]},{"osd":5,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.64100000000000001,"15min":0.624},"min":{"1min":0.42999999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.90300000000000002,"5min":0.91500000000000004,"15min":1.0049999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.66600000000000004,"15min":0.64600000000000002},"min":{"1min":0.46700000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89500000000000002,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.56899999999999995}]},{"osd":6,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.69099999999999995,"15min":0.67100000000000004},"min":{"1min":0.51000000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83899999999999997,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.72699999999999998,"15min":0.70699999999999996},"min":{"1min":0.51800000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95199999999999996,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.58799999999999997}]},{"osd":7,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.72499999999999998,"15min":0.70099999999999996},"min":{"1min":0.56399999999999995,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0489999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.78000000000000003,"15min":0.754},"min":{"1min":0.57499999999999996,"5min":0.436,"15min":0.41699999999999998},"max":{"1min":1.02,"5min":2.052,"15min":2.052},"last":0.74299999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575982,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312468,"kb_used_data":2892,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059372,"statfs":{"total":96636764160,"available":96316796928,"internally_reserved":0,"allocated":2961408,"data_stored":1424216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.71699999999999997,"15min":0.72899999999999998},"min":{"1min":0.46800000000000003,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71099999999999997,"15min":0.72399999999999998},"min":{"1min":0.51000000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.56699999999999995}]},{"osd":1,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71499999999999997,"15min":0.73399999999999999},"min":{"1min":0.46800000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91400000000000003,"5min":0.95699999999999996,"15min":0.97699999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.73299999999999998,"15min":0.73799999999999999},"min":{"1min":0.51900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.874,"5min":1.004,"15min":1.0269999999999999},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.72299999999999998,"15min":0.72999999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1,"5min":1.0269999999999999,"15min":1.028},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.73199999999999998,"15min":0.74199999999999999},"min":{"1min":0.50900000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.98799999999999999},"last":0.71799999999999997}]},{"osd":3,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.74399999999999999,"15min":0.75600000000000001},"min":{"1min":0.56200000000000006,"5min":0.52800000000000002,"15min":0.49299999999999999},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76100000000000001,"15min":0.77000000000000002},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.874}]},{"osd":4,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.63300000000000001,"15min":0.61399999999999999},"min":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41699999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.64300000000000002,"15min":0.63400000000000001},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.375},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":1.0189999999999999}]},{"osd":6,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63700000000000001,"15min":0.64300000000000002},"min":{"1min":0.433,"5min":0.42099999999999999,"15min":0.40999999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":1.0089999999999999},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69199999999999995,"15min":0.68600000000000005},"min":{"1min":0.48699999999999999,"5min":0.46200000000000002,"15min":0.42799999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.79500000000000004}]},{"osd":7,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.73199999999999998,"15min":0.73299999999999998},"min":{"1min":0.58299999999999996,"5min":0.47299999999999998,"15min":0.434},"max":{"1min":0.94799999999999995,"5min":1.0149999999999999,"15min":1.0760000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.77800000000000002,"15min":0.77600000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.47599999999999998},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":1.056}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:46:30.654 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:46:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.898+0000 7f74bffff700 1 Processor -- start 2024-01-11T09:46:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.898+0000 7f74bffff700 1 -- start start 2024-01-11T09:46:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.899+0000 7f74bffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74b80c8620 0x7f74b80af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:30.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.899+0000 7f74bffff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74b8069d40 0x7f74b80b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.899+0000 7f74bffff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b8157390 con 0x7f74b80c89f0 2024-01-11T09:46:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.900+0000 7f74bffff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b80adcd0 con 0x7f74b80c8620 2024-01-11T09:46:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.900+0000 7f74bffff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b80afeb0 con 0x7f74b8069d40 2024-01-11T09:46:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.900+0000 7f74bdd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74b80c8620 0x7f74b80af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.900+0000 7f74bdd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74b80c8620 0x7f74b80af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48488/0 (socket says 172.21.15.27:48488) 2024-01-11T09:46:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.900+0000 7f74bdd9b700 1 -- 172.21.15.27:0/1825907195 learned_addr learned my addr 172.21.15.27:0/1825907195 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.900+0000 7f74be59c700 1 --1- 172.21.15.27:0/1825907195 >> v1:172.21.15.27:6789/0 conn(0x7f74b80c89f0 0x7f74b8159ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52416/0 (socket says 172.21.15.27:52416) 2024-01-11T09:46:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.900+0000 7f74bd59a700 1 --2- 172.21.15.27:0/1825907195 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74b8069d40 0x7f74b80b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.900+0000 7f74bdd9b700 1 -- 172.21.15.27:0/1825907195 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74b8069d40 msgr2=0x7f74b80b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.900+0000 7f74bdd9b700 1 --2- 172.21.15.27:0/1825907195 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74b8069d40 0x7f74b80b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.900+0000 7f74bdd9b700 1 -- 172.21.15.27:0/1825907195 >> v1:172.21.15.27:6789/0 conn(0x7f74b80c89f0 legacy=0x7f74b8159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:30.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.900+0000 7f74bdd9b700 1 -- 172.21.15.27:0/1825907195 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b806a200 con 0x7f74b80c8620 2024-01-11T09:46:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.900+0000 7f74bd59a700 1 --2- 172.21.15.27:0/1825907195 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74b8069d40 0x7f74b80b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.901+0000 7f74bdd9b700 1 --2- 172.21.15.27:0/1825907195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b80c8620 0x7f74b80af970 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f74b40099b0 tx=0x7f74b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d48c7fc15786ec server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.901+0000 7f74bcd99700 1 -- 172.21.15.27:0/1825907195 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b400b860 con 0x7f74b80c8620 2024-01-11T09:46:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.901+0000 7f74bcd99700 1 -- 172.21.15.27:0/1825907195 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b400b9c0 con 0x7f74b80c8620 2024-01-11T09:46:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.901+0000 7f74bcd99700 1 -- 172.21.15.27:0/1825907195 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b400bd60 con 0x7f74b80c8620 2024-01-11T09:46:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.901+0000 7f74bffff700 1 -- 172.21.15.27:0/1825907195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b80c8620 msgr2=0x7f74b80af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.901+0000 7f74bffff700 1 --2- 172.21.15.27:0/1825907195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b80c8620 0x7f74b80af970 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f74b40099b0 tx=0x7f74b4009250 comp rx=0 tx=0).stop 2024-01-11T09:46:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.901+0000 7f74bffff700 1 -- 172.21.15.27:0/1825907195 shutdown_connections 2024-01-11T09:46:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.901+0000 7f74bffff700 1 --2- 172.21.15.27:0/1825907195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b80c8620 0x7f74b80af970 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.901+0000 7f74bffff700 1 --2- 172.21.15.27:0/1825907195 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74b8069d40 0x7f74b80b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.902+0000 7f74bffff700 1 -- 172.21.15.27:0/1825907195 >> 172.21.15.27:0/1825907195 conn(0x7f74b80680e0 msgr2=0x7f74b8069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.902+0000 7f74bffff700 1 -- 172.21.15.27:0/1825907195 shutdown_connections 2024-01-11T09:46:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.902+0000 7f74bffff700 1 -- 172.21.15.27:0/1825907195 wait complete. 2024-01-11T09:46:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.903+0000 7f74bffff700 1 Processor -- start 2024-01-11T09:46:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.903+0000 7f74bffff700 1 -- start start 2024-01-11T09:46:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.904+0000 7f74bffff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74b8069d40 0x7f74b8167c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.904+0000 7f74bffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74b80c8620 0x7f74b81587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.904+0000 7f74bffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b80c89f0 0x7f74b8158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.904+0000 7f74bffff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b8168bc0 con 0x7f74b80c89f0 2024-01-11T09:46:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.904+0000 7f74bffff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b8169280 con 0x7f74b80c8620 2024-01-11T09:46:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.904+0000 7f74bffff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b8168600 con 0x7f74b8069d40 2024-01-11T09:46:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.904+0000 7f74be59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b80c89f0 0x7f74b8158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.904+0000 7f74be59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b80c89f0 0x7f74b8158d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48498/0 (socket says 172.21.15.27:48498) 2024-01-11T09:46:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.904+0000 7f74be59c700 1 -- 172.21.15.27:0/3956894176 learned_addr learned my addr 172.21.15.27:0/3956894176 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.905+0000 7f74bdd9b700 1 --2- 172.21.15.27:0/3956894176 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74b8069d40 0x7f74b8167c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.905+0000 7f74bd59a700 1 --2- 172.21.15.27:0/3956894176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74b80c8620 0x7f74b81587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.905+0000 7f74be59c700 1 -- 172.21.15.27:0/3956894176 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74b8069d40 msgr2=0x7f74b8167c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.905+0000 7f74be59c700 1 --2- 172.21.15.27:0/3956894176 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74b8069d40 0x7f74b8167c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.905+0000 7f74be59c700 1 -- 172.21.15.27:0/3956894176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74b80c8620 msgr2=0x7f74b81587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.905+0000 7f74be59c700 1 --2- 172.21.15.27:0/3956894176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74b80c8620 0x7f74b81587f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.905+0000 7f74be59c700 1 -- 172.21.15.27:0/3956894176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b40094f0 con 0x7f74b80c89f0 2024-01-11T09:46:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.905+0000 7f74be59c700 1 --2- 172.21.15.27:0/3956894176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b80c89f0 0x7f74b8158d30 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f74b000cc70 tx=0x7f74b0001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.905+0000 7f74adffb700 1 -- 172.21.15.27:0/3956894176 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b0005df0 con 0x7f74b80c89f0 2024-01-11T09:46:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.905+0000 7f74adffb700 1 -- 172.21.15.27:0/3956894176 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b000a640 con 0x7f74b80c89f0 2024-01-11T09:46:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.905+0000 7f74bffff700 1 -- 172.21.15.27:0/3956894176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b81688e0 con 0x7f74b80c89f0 2024-01-11T09:46:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.905+0000 7f74adffb700 1 -- 172.21.15.27:0/3956894176 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b000a9c0 con 0x7f74b80c89f0 2024-01-11T09:46:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.906+0000 7f74bffff700 1 -- 172.21.15.27:0/3956894176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b8159970 con 0x7f74b80c89f0 2024-01-11T09:46:30.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.907+0000 7f74bffff700 1 -- 172.21.15.27:0/3956894176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b80718d0 con 0x7f74b80c89f0 2024-01-11T09:46:30.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.908+0000 7f74adffb700 1 -- 172.21.15.27:0/3956894176 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f74b00070f0 con 0x7f74b80c89f0 2024-01-11T09:46:30.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.909+0000 7f74adffb700 1 --2- 172.21.15.27:0/3956894176 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f749c067170 0x7f749c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:30.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.909+0000 7f74adffb700 1 -- 172.21.15.27:0/3956894176 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f74b0084890 con 0x7f74b80c89f0 2024-01-11T09:46:30.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.909+0000 7f74bdd9b700 1 --2- 172.21.15.27:0/3956894176 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f749c067170 0x7f749c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:30.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.910+0000 7f74bdd9b700 1 --2- 172.21.15.27:0/3956894176 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f749c067170 0x7f749c069620 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f74b40099b0 tx=0x7f74b4017000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:30.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:30.913+0000 7f74adffb700 1 -- 172.21.15.27:0/3956894176 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f74b0051930 con 0x7f74b80c89f0 2024-01-11T09:46:31.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.112+0000 7f74bffff700 1 -- 172.21.15.27:0/3956894176 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f74b8156d40 con 0x7f749c067170 2024-01-11T09:46:31.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.121+0000 7f74adffb700 1 -- 172.21.15.27:0/3956894176 <== mgr.9107 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289646 (secure 0 0 0) 0x7f74b8156d40 con 0x7f749c067170 2024-01-11T09:46:31.117 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:31.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.126+0000 7f74bffff700 1 -- 172.21.15.27:0/3956894176 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f749c067170 msgr2=0x7f749c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:31.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.126+0000 7f74bffff700 1 --2- 172.21.15.27:0/3956894176 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f749c067170 0x7f749c069620 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f74b40099b0 tx=0x7f74b4017000 comp rx=0 tx=0).stop 2024-01-11T09:46:31.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.126+0000 7f74bffff700 1 -- 172.21.15.27:0/3956894176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b80c89f0 msgr2=0x7f74b8158d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:31.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.126+0000 7f74bffff700 1 --2- 172.21.15.27:0/3956894176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b80c89f0 0x7f74b8158d30 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f74b000cc70 tx=0x7f74b0001a40 comp rx=0 tx=0).stop 2024-01-11T09:46:31.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.126+0000 7f74bffff700 1 -- 172.21.15.27:0/3956894176 shutdown_connections 2024-01-11T09:46:31.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.126+0000 7f74bffff700 1 --2- 172.21.15.27:0/3956894176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b80c89f0 0x7f74b8158d30 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.126+0000 7f74bffff700 1 --2- 172.21.15.27:0/3956894176 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74b8069d40 0x7f74b8167c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.126+0000 7f74bffff700 1 --2- 172.21.15.27:0/3956894176 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f749c067170 0x7f749c069620 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.126+0000 7f74bffff700 1 --2- 172.21.15.27:0/3956894176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74b80c8620 0x7f74b81587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.126+0000 7f74bffff700 1 -- 172.21.15.27:0/3956894176 >> 172.21.15.27:0/3956894176 conn(0x7f74b80680e0 msgr2=0x7f74b81547b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:31.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.126+0000 7f74bffff700 1 -- 172.21.15.27:0/3956894176 shutdown_connections 2024-01-11T09:46:31.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.126+0000 7f74bffff700 1 -- 172.21.15.27:0/3956894176 wait complete. 2024-01-11T09:46:31.120 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:46:31.136 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-01-11T09:46:30.158435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":695,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":426,"ondisk_log_size":426,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2451816,"kb_used_data":24360,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752522904,"statfs":{"total":773094113280,"available":770583453696,"internally_reserved":0,"allocated":24944640,"data_stored":12579076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.944946"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.956201+0000","last_change":"2024-01-11T09:46:25.952190+0000","last_active":"2024-01-11T09:46:25.956201+0000","last_peered":"2024-01-11T09:46:25.956201+0000","last_clean":"2024-01-11T09:46:25.956201+0000","last_became_active":"2024-01-11T09:46:25.951733+0000","last_became_peered":"2024-01-11T09:46:25.951733+0000","last_unstale":"2024-01-11T09:46:25.956201+0000","last_undegraded":"2024-01-11T09:46:25.956201+0000","last_fullsized":"2024-01-11T09:46:25.956201+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:59:15.007682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960088+0000","last_change":"2024-01-11T09:46:25.957907+0000","last_active":"2024-01-11T09:46:25.960088+0000","last_peered":"2024-01-11T09:46:25.960088+0000","last_clean":"2024-01-11T09:46:25.960088+0000","last_became_active":"2024-01-11T09:46:25.956251+0000","last_became_peered":"2024-01-11T09:46:25.956251+0000","last_unstale":"2024-01-11T09:46:25.960088+0000","last_undegraded":"2024-01-11T09:46:25.960088+0000","last_fullsized":"2024-01-11T09:46:25.960088+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:53:35.139255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960472+0000","last_change":"2024-01-11T09:46:25.956076+0000","last_active":"2024-01-11T09:46:25.960472+0000","last_peered":"2024-01-11T09:46:25.960472+0000","last_clean":"2024-01-11T09:46:25.960472+0000","last_became_active":"2024-01-11T09:46:25.955819+0000","last_became_peered":"2024-01-11T09:46:25.955819+0000","last_unstale":"2024-01-11T09:46:25.960472+0000","last_undegraded":"2024-01-11T09:46:25.960472+0000","last_fullsized":"2024-01-11T09:46:25.960472+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:27:34.516236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.952790+0000","last_change":"2024-01-11T09:46:25.947296+0000","last_active":"2024-01-11T09:46:25.952790+0000","last_peered":"2024-01-11T09:46:25.952790+0000","last_clean":"2024-01-11T09:46:25.952790+0000","last_became_active":"2024-01-11T09:46:25.946057+0000","last_became_peered":"2024-01-11T09:46:25.946057+0000","last_unstale":"2024-01-11T09:46:25.952790+0000","last_undegraded":"2024-01-11T09:46:25.952790+0000","last_fullsized":"2024-01-11T09:46:25.952790+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:15:19.562588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.955304+0000","last_change":"2024-01-11T09:46:25.953859+0000","last_active":"2024-01-11T09:46:25.955304+0000","last_peered":"2024-01-11T09:46:25.955304+0000","last_clean":"2024-01-11T09:46:25.955304+0000","last_became_active":"2024-01-11T09:46:25.953566+0000","last_became_peered":"2024-01-11T09:46:25.953566+0000","last_unstale":"2024-01-11T09:46:25.955304+0000","last_undegraded":"2024-01-11T09:46:25.955304+0000","last_fullsized":"2024-01-11T09:46:25.955304+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:42:55.688728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.953888+0000","last_change":"2024-01-11T09:46:25.945778+0000","last_active":"2024-01-11T09:46:25.953888+0000","last_peered":"2024-01-11T09:46:25.953888+0000","last_clean":"2024-01-11T09:46:25.953888+0000","last_became_active":"2024-01-11T09:46:25.944907+0000","last_became_peered":"2024-01-11T09:46:25.944907+0000","last_unstale":"2024-01-11T09:46:25.953888+0000","last_undegraded":"2024-01-11T09:46:25.953888+0000","last_fullsized":"2024-01-11T09:46:25.953888+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:59:02.206221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.963832+0000","last_change":"2024-01-11T09:46:25.960002+0000","last_active":"2024-01-11T09:46:25.963832+0000","last_peered":"2024-01-11T09:46:25.963832+0000","last_clean":"2024-01-11T09:46:25.963832+0000","last_became_active":"2024-01-11T09:46:25.957845+0000","last_became_peered":"2024-01-11T09:46:25.957845+0000","last_unstale":"2024-01-11T09:46:25.963832+0000","last_undegraded":"2024-01-11T09:46:25.963832+0000","last_fullsized":"2024-01-11T09:46:25.963832+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:58:52.892728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.963536+0000","last_change":"2024-01-11T09:46:25.958630+0000","last_active":"2024-01-11T09:46:25.963536+0000","last_peered":"2024-01-11T09:46:25.963536+0000","last_clean":"2024-01-11T09:46:25.963536+0000","last_became_active":"2024-01-11T09:46:25.957768+0000","last_became_peered":"2024-01-11T09:46:25.957768+0000","last_unstale":"2024-01-11T09:46:25.963536+0000","last_undegraded":"2024-01-11T09:46:25.963536+0000","last_fullsized":"2024-01-11T09:46:25.963536+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:59:15.007682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.956638+0000","last_change":"2024-01-11T09:46:25.954318+0000","last_active":"2024-01-11T09:46:25.956638+0000","last_peered":"2024-01-11T09:46:25.956638+0000","last_clean":"2024-01-11T09:46:25.956638+0000","last_became_active":"2024-01-11T09:46:25.953690+0000","last_became_peered":"2024-01-11T09:46:25.953690+0000","last_unstale":"2024-01-11T09:46:25.956638+0000","last_undegraded":"2024-01-11T09:46:25.956638+0000","last_fullsized":"2024-01-11T09:46:25.956638+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:28:55.053850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.961820+0000","last_change":"2024-01-11T09:46:25.957195+0000","last_active":"2024-01-11T09:46:25.961820+0000","last_peered":"2024-01-11T09:46:25.961820+0000","last_clean":"2024-01-11T09:46:25.961820+0000","last_became_active":"2024-01-11T09:46:25.956882+0000","last_became_peered":"2024-01-11T09:46:25.956882+0000","last_unstale":"2024-01-11T09:46:25.961820+0000","last_undegraded":"2024-01-11T09:46:25.961820+0000","last_fullsized":"2024-01-11T09:46:25.961820+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:37:41.519655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.963173+0000","last_change":"2024-01-11T09:46:25.960943+0000","last_active":"2024-01-11T09:46:25.963173+0000","last_peered":"2024-01-11T09:46:25.963173+0000","last_clean":"2024-01-11T09:46:25.963173+0000","last_became_active":"2024-01-11T09:46:25.960111+0000","last_became_peered":"2024-01-11T09:46:25.960111+0000","last_unstale":"2024-01-11T09:46:25.963173+0000","last_undegraded":"2024-01-11T09:46:25.963173+0000","last_fullsized":"2024-01-11T09:46:25.963173+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:04.562589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.961397+0000","last_change":"2024-01-11T09:46:25.955367+0000","last_active":"2024-01-11T09:46:25.961397+0000","last_peered":"2024-01-11T09:46:25.961397+0000","last_clean":"2024-01-11T09:46:25.961397+0000","last_became_active":"2024-01-11T09:46:25.955123+0000","last_became_peered":"2024-01-11T09:46:25.955123+0000","last_unstale":"2024-01-11T09:46:25.961397+0000","last_undegraded":"2024-01-11T09:46:25.961397+0000","last_fullsized":"2024-01-11T09:46:25.961397+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:07:13.288202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.963023+0000","last_change":"2024-01-11T09:46:25.958036+0000","last_active":"2024-01-11T09:46:25.963023+0000","last_peered":"2024-01-11T09:46:25.963023+0000","last_clean":"2024-01-11T09:46:25.963023+0000","last_became_active":"2024-01-11T09:46:25.957739+0000","last_became_peered":"2024-01-11T09:46:25.957739+0000","last_unstale":"2024-01-11T09:46:25.963023+0000","last_undegraded":"2024-01-11T09:46:25.963023+0000","last_fullsized":"2024-01-11T09:46:25.963023+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:04.562589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.955491+0000","last_change":"2024-01-11T09:46:25.951789+0000","last_active":"2024-01-11T09:46:25.955491+0000","last_peered":"2024-01-11T09:46:25.955491+0000","last_clean":"2024-01-11T09:46:25.955491+0000","last_became_active":"2024-01-11T09:46:25.950427+0000","last_became_peered":"2024-01-11T09:46:25.950427+0000","last_unstale":"2024-01-11T09:46:25.955491+0000","last_undegraded":"2024-01-11T09:46:25.955491+0000","last_fullsized":"2024-01-11T09:46:25.955491+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:10.125414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.963896+0000","last_change":"2024-01-11T09:46:25.962921+0000","last_active":"2024-01-11T09:46:25.963896+0000","last_peered":"2024-01-11T09:46:25.963896+0000","last_clean":"2024-01-11T09:46:25.963896+0000","last_became_active":"2024-01-11T09:46:25.962306+0000","last_became_peered":"2024-01-11T09:46:25.962306+0000","last_unstale":"2024-01-11T09:46:25.963896+0000","last_undegraded":"2024-01-11T09:46:25.963896+0000","last_fullsized":"2024-01-11T09:46:25.963896+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:49:33.997022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958634+0000","last_change":"2024-01-11T09:46:25.951750+0000","last_active":"2024-01-11T09:46:25.958634+0000","last_peered":"2024-01-11T09:46:25.958634+0000","last_clean":"2024-01-11T09:46:25.958634+0000","last_became_active":"2024-01-11T09:46:25.951499+0000","last_became_peered":"2024-01-11T09:46:25.951499+0000","last_unstale":"2024-01-11T09:46:25.958634+0000","last_undegraded":"2024-01-11T09:46:25.958634+0000","last_fullsized":"2024-01-11T09:46:25.958634+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:48:16.354636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.952702+0000","last_change":"2024-01-11T09:46:25.950974+0000","last_active":"2024-01-11T09:46:25.952702+0000","last_peered":"2024-01-11T09:46:25.952702+0000","last_clean":"2024-01-11T09:46:25.952702+0000","last_became_active":"2024-01-11T09:46:25.950595+0000","last_became_peered":"2024-01-11T09:46:25.950595+0000","last_unstale":"2024-01-11T09:46:25.952702+0000","last_undegraded":"2024-01-11T09:46:25.952702+0000","last_fullsized":"2024-01-11T09:46:25.952702+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:58.843659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.955429+0000","last_change":"2024-01-11T09:46:25.952186+0000","last_active":"2024-01-11T09:46:25.955429+0000","last_peered":"2024-01-11T09:46:25.955429+0000","last_clean":"2024-01-11T09:46:25.955429+0000","last_became_active":"2024-01-11T09:46:25.951898+0000","last_became_peered":"2024-01-11T09:46:25.951898+0000","last_unstale":"2024-01-11T09:46:25.955429+0000","last_undegraded":"2024-01-11T09:46:25.955429+0000","last_fullsized":"2024-01-11T09:46:25.955429+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:54:00.534598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.963483+0000","last_change":"2024-01-11T09:46:25.957991+0000","last_active":"2024-01-11T09:46:25.963483+0000","last_peered":"2024-01-11T09:46:25.963483+0000","last_clean":"2024-01-11T09:46:25.963483+0000","last_became_active":"2024-01-11T09:46:25.957482+0000","last_became_peered":"2024-01-11T09:46:25.957482+0000","last_unstale":"2024-01-11T09:46:25.963483+0000","last_undegraded":"2024-01-11T09:46:25.963483+0000","last_fullsized":"2024-01-11T09:46:25.963483+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:54:00.534598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.949569+0000","last_change":"2024-01-11T09:46:25.942788+0000","last_active":"2024-01-11T09:46:25.949569+0000","last_peered":"2024-01-11T09:46:25.949569+0000","last_clean":"2024-01-11T09:46:25.949569+0000","last_became_active":"2024-01-11T09:46:25.941771+0000","last_became_peered":"2024-01-11T09:46:25.941771+0000","last_unstale":"2024-01-11T09:46:25.949569+0000","last_undegraded":"2024-01-11T09:46:25.949569+0000","last_fullsized":"2024-01-11T09:46:25.949569+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:42:47.356295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960319+0000","last_change":"2024-01-11T09:46:25.954694+0000","last_active":"2024-01-11T09:46:25.960319+0000","last_peered":"2024-01-11T09:46:25.960319+0000","last_clean":"2024-01-11T09:46:25.960319+0000","last_became_active":"2024-01-11T09:46:25.952808+0000","last_became_peered":"2024-01-11T09:46:25.952808+0000","last_unstale":"2024-01-11T09:46:25.960319+0000","last_undegraded":"2024-01-11T09:46:25.960319+0000","last_fullsized":"2024-01-11T09:46:25.960319+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:10.125414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.964163+0000","last_change":"2024-01-11T09:46:25.962654+0000","last_active":"2024-01-11T09:46:25.964163+0000","last_peered":"2024-01-11T09:46:25.964163+0000","last_clean":"2024-01-11T09:46:25.964163+0000","last_became_active":"2024-01-11T09:46:25.962073+0000","last_became_peered":"2024-01-11T09:46:25.962073+0000","last_unstale":"2024-01-11T09:46:25.964163+0000","last_undegraded":"2024-01-11T09:46:25.964163+0000","last_fullsized":"2024-01-11T09:46:25.964163+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:56:15.022301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.963312+0000","last_change":"2024-01-11T09:46:25.956810+0000","last_active":"2024-01-11T09:46:25.963312+0000","last_peered":"2024-01-11T09:46:25.963312+0000","last_clean":"2024-01-11T09:46:25.963312+0000","last_became_active":"2024-01-11T09:46:25.955894+0000","last_became_peered":"2024-01-11T09:46:25.955894+0000","last_unstale":"2024-01-11T09:46:25.963312+0000","last_undegraded":"2024-01-11T09:46:25.963312+0000","last_fullsized":"2024-01-11T09:46:25.963312+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:53:35.139255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.951416+0000","last_change":"2024-01-11T09:46:25.947721+0000","last_active":"2024-01-11T09:46:25.951416+0000","last_peered":"2024-01-11T09:46:25.951416+0000","last_clean":"2024-01-11T09:46:25.951416+0000","last_became_active":"2024-01-11T09:46:25.947454+0000","last_became_peered":"2024-01-11T09:46:25.947454+0000","last_unstale":"2024-01-11T09:46:25.951416+0000","last_undegraded":"2024-01-11T09:46:25.951416+0000","last_fullsized":"2024-01-11T09:46:25.951416+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:42:44.083747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.946476+0000","last_change":"2024-01-11T09:46:25.939265+0000","last_active":"2024-01-11T09:46:25.946476+0000","last_peered":"2024-01-11T09:46:25.946476+0000","last_clean":"2024-01-11T09:46:25.946476+0000","last_became_active":"2024-01-11T09:46:25.938964+0000","last_became_peered":"2024-01-11T09:46:25.938964+0000","last_unstale":"2024-01-11T09:46:25.946476+0000","last_undegraded":"2024-01-11T09:46:25.946476+0000","last_fullsized":"2024-01-11T09:46:25.946476+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:20:24.956940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.962886+0000","last_change":"2024-01-11T09:46:25.953966+0000","last_active":"2024-01-11T09:46:25.962886+0000","last_peered":"2024-01-11T09:46:25.962886+0000","last_clean":"2024-01-11T09:46:25.962886+0000","last_became_active":"2024-01-11T09:46:25.953566+0000","last_became_peered":"2024-01-11T09:46:25.953566+0000","last_unstale":"2024-01-11T09:46:25.962886+0000","last_undegraded":"2024-01-11T09:46:25.962886+0000","last_fullsized":"2024-01-11T09:46:25.962886+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:45:16.842085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.962801+0000","last_change":"2024-01-11T09:46:25.959751+0000","last_active":"2024-01-11T09:46:25.962801+0000","last_peered":"2024-01-11T09:46:25.962801+0000","last_clean":"2024-01-11T09:46:25.962801+0000","last_became_active":"2024-01-11T09:46:25.959431+0000","last_became_peered":"2024-01-11T09:46:25.959431+0000","last_unstale":"2024-01-11T09:46:25.962801+0000","last_undegraded":"2024-01-11T09:46:25.962801+0000","last_fullsized":"2024-01-11T09:46:25.962801+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:58:52.892728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958104+0000","last_change":"2024-01-11T09:46:25.956345+0000","last_active":"2024-01-11T09:46:25.958104+0000","last_peered":"2024-01-11T09:46:25.958104+0000","last_clean":"2024-01-11T09:46:25.958104+0000","last_became_active":"2024-01-11T09:46:25.956080+0000","last_became_peered":"2024-01-11T09:46:25.956080+0000","last_unstale":"2024-01-11T09:46:25.958104+0000","last_undegraded":"2024-01-11T09:46:25.958104+0000","last_fullsized":"2024-01-11T09:46:25.958104+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:30:20.657242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960243+0000","last_change":"2024-01-11T09:46:25.953176+0000","last_active":"2024-01-11T09:46:25.960243+0000","last_peered":"2024-01-11T09:46:25.960243+0000","last_clean":"2024-01-11T09:46:25.960243+0000","last_became_active":"2024-01-11T09:46:25.952276+0000","last_became_peered":"2024-01-11T09:46:25.952276+0000","last_unstale":"2024-01-11T09:46:25.960243+0000","last_undegraded":"2024-01-11T09:46:25.960243+0000","last_fullsized":"2024-01-11T09:46:25.960243+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:37:41.519655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.951599+0000","last_change":"2024-01-11T09:46:25.949114+0000","last_active":"2024-01-11T09:46:25.951599+0000","last_peered":"2024-01-11T09:46:25.951599+0000","last_clean":"2024-01-11T09:46:25.951599+0000","last_became_active":"2024-01-11T09:46:25.948844+0000","last_became_peered":"2024-01-11T09:46:25.948844+0000","last_unstale":"2024-01-11T09:46:25.951599+0000","last_undegraded":"2024-01-11T09:46:25.951599+0000","last_fullsized":"2024-01-11T09:46:25.951599+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:31:58.203452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.962503+0000","last_change":"2024-01-11T09:46:25.954895+0000","last_active":"2024-01-11T09:46:25.962503+0000","last_peered":"2024-01-11T09:46:25.962503+0000","last_clean":"2024-01-11T09:46:25.962503+0000","last_became_active":"2024-01-11T09:46:25.954472+0000","last_became_peered":"2024-01-11T09:46:25.954472+0000","last_unstale":"2024-01-11T09:46:25.962503+0000","last_undegraded":"2024-01-11T09:46:25.962503+0000","last_fullsized":"2024-01-11T09:46:25.962503+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:34:26.903086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.957489+0000","last_change":"2024-01-11T09:46:25.955503+0000","last_active":"2024-01-11T09:46:25.957489+0000","last_peered":"2024-01-11T09:46:25.957489+0000","last_clean":"2024-01-11T09:46:25.957489+0000","last_became_active":"2024-01-11T09:46:25.954870+0000","last_became_peered":"2024-01-11T09:46:25.954870+0000","last_unstale":"2024-01-11T09:46:25.957489+0000","last_undegraded":"2024-01-11T09:46:25.957489+0000","last_fullsized":"2024-01-11T09:46:25.957489+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:06:12.133349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.956147+0000","last_change":"2024-01-11T09:46:25.947226+0000","last_active":"2024-01-11T09:46:25.956147+0000","last_peered":"2024-01-11T09:46:25.956147+0000","last_clean":"2024-01-11T09:46:25.956147+0000","last_became_active":"2024-01-11T09:46:25.946806+0000","last_became_peered":"2024-01-11T09:46:25.946806+0000","last_unstale":"2024-01-11T09:46:25.956147+0000","last_undegraded":"2024-01-11T09:46:25.956147+0000","last_fullsized":"2024-01-11T09:46:25.956147+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:25:09.589100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"234'1","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.242548+0000","last_change":"2024-01-11T09:46:25.236672+0000","last_active":"2024-01-11T09:46:26.242548+0000","last_peered":"2024-01-11T09:46:26.242548+0000","last_clean":"2024-01-11T09:46:26.242548+0000","last_became_active":"2024-01-11T09:46:25.236363+0000","last_became_peered":"2024-01-11T09:46:25.236363+0000","last_unstale":"2024-01-11T09:46:26.242548+0000","last_undegraded":"2024-01-11T09:46:26.242548+0000","last_fullsized":"2024-01-11T09:46:26.242548+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:01:24.091035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960428+0000","last_change":"2024-01-11T09:46:25.951411+0000","last_active":"2024-01-11T09:46:25.960428+0000","last_peered":"2024-01-11T09:46:25.960428+0000","last_clean":"2024-01-11T09:46:25.960428+0000","last_became_active":"2024-01-11T09:46:25.951161+0000","last_became_peered":"2024-01-11T09:46:25.951161+0000","last_unstale":"2024-01-11T09:46:25.960428+0000","last_undegraded":"2024-01-11T09:46:25.960428+0000","last_fullsized":"2024-01-11T09:46:25.960428+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:45:16.842085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"234'2","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.247880+0000","last_change":"2024-01-11T09:46:25.238481+0000","last_active":"2024-01-11T09:46:26.247880+0000","last_peered":"2024-01-11T09:46:26.247880+0000","last_clean":"2024-01-11T09:46:26.247880+0000","last_became_active":"2024-01-11T09:46:25.238203+0000","last_became_peered":"2024-01-11T09:46:25.238203+0000","last_unstale":"2024-01-11T09:46:26.247880+0000","last_undegraded":"2024-01-11T09:46:26.247880+0000","last_fullsized":"2024-01-11T09:46:26.247880+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:42:43.074601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958017+0000","last_change":"2024-01-11T09:46:25.954029+0000","last_active":"2024-01-11T09:46:25.958017+0000","last_peered":"2024-01-11T09:46:25.958017+0000","last_clean":"2024-01-11T09:46:25.958017+0000","last_became_active":"2024-01-11T09:46:25.953690+0000","last_became_peered":"2024-01-11T09:46:25.953690+0000","last_unstale":"2024-01-11T09:46:25.958017+0000","last_undegraded":"2024-01-11T09:46:25.958017+0000","last_fullsized":"2024-01-11T09:46:25.958017+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:26:46.984194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"234'2","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.247094+0000","last_change":"2024-01-11T09:46:25.227884+0000","last_active":"2024-01-11T09:46:26.247094+0000","last_peered":"2024-01-11T09:46:26.247094+0000","last_clean":"2024-01-11T09:46:26.247094+0000","last_became_active":"2024-01-11T09:46:25.227605+0000","last_became_peered":"2024-01-11T09:46:25.227605+0000","last_unstale":"2024-01-11T09:46:26.247094+0000","last_undegraded":"2024-01-11T09:46:26.247094+0000","last_fullsized":"2024-01-11T09:46:26.247094+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:09.116268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.957373+0000","last_change":"2024-01-11T09:46:25.950375+0000","last_active":"2024-01-11T09:46:25.957373+0000","last_peered":"2024-01-11T09:46:25.957373+0000","last_clean":"2024-01-11T09:46:25.957373+0000","last_became_active":"2024-01-11T09:46:25.950114+0000","last_became_peered":"2024-01-11T09:46:25.950114+0000","last_unstale":"2024-01-11T09:46:25.957373+0000","last_undegraded":"2024-01-11T09:46:25.957373+0000","last_fullsized":"2024-01-11T09:46:25.957373+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:25.226766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"234'1","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.242911+0000","last_change":"2024-01-11T09:46:25.233693+0000","last_active":"2024-01-11T09:46:26.242911+0000","last_peered":"2024-01-11T09:46:26.242911+0000","last_clean":"2024-01-11T09:46:26.242911+0000","last_became_active":"2024-01-11T09:46:25.233410+0000","last_became_peered":"2024-01-11T09:46:25.233410+0000","last_unstale":"2024-01-11T09:46:26.242911+0000","last_undegraded":"2024-01-11T09:46:26.242911+0000","last_fullsized":"2024-01-11T09:46:26.242911+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:03:42.704031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958402+0000","last_change":"2024-01-11T09:46:25.955874+0000","last_active":"2024-01-11T09:46:25.958402+0000","last_peered":"2024-01-11T09:46:25.958402+0000","last_clean":"2024-01-11T09:46:25.958402+0000","last_became_active":"2024-01-11T09:46:25.954789+0000","last_became_peered":"2024-01-11T09:46:25.954789+0000","last_unstale":"2024-01-11T09:46:25.958402+0000","last_undegraded":"2024-01-11T09:46:25.958402+0000","last_fullsized":"2024-01-11T09:46:25.958402+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:07.160563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"234'1","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.244983+0000","last_change":"2024-01-11T09:46:25.235297+0000","last_active":"2024-01-11T09:46:26.244983+0000","last_peered":"2024-01-11T09:46:26.244983+0000","last_clean":"2024-01-11T09:46:26.244983+0000","last_became_active":"2024-01-11T09:46:25.234744+0000","last_became_peered":"2024-01-11T09:46:25.234744+0000","last_unstale":"2024-01-11T09:46:26.244983+0000","last_undegraded":"2024-01-11T09:46:26.244983+0000","last_fullsized":"2024-01-11T09:46:26.244983+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:51:44.543489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958203+0000","last_change":"2024-01-11T09:46:25.948668+0000","last_active":"2024-01-11T09:46:25.958203+0000","last_peered":"2024-01-11T09:46:25.958203+0000","last_clean":"2024-01-11T09:46:25.958203+0000","last_became_active":"2024-01-11T09:46:25.948372+0000","last_became_peered":"2024-01-11T09:46:25.948372+0000","last_unstale":"2024-01-11T09:46:25.958203+0000","last_undegraded":"2024-01-11T09:46:25.958203+0000","last_fullsized":"2024-01-11T09:46:25.958203+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:41.154829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.948817+0000","last_change":"2024-01-11T09:46:25.238227+0000","last_active":"2024-01-11T09:46:25.948817+0000","last_peered":"2024-01-11T09:46:25.948817+0000","last_clean":"2024-01-11T09:46:25.948817+0000","last_became_active":"2024-01-11T09:46:25.238013+0000","last_became_peered":"2024-01-11T09:46:25.238013+0000","last_unstale":"2024-01-11T09:46:25.948817+0000","last_undegraded":"2024-01-11T09:46:25.948817+0000","last_fullsized":"2024-01-11T09:46:25.948817+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:36:18.777304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.954020+0000","last_change":"2024-01-11T09:46:25.949014+0000","last_active":"2024-01-11T09:46:25.954020+0000","last_peered":"2024-01-11T09:46:25.954020+0000","last_clean":"2024-01-11T09:46:25.954020+0000","last_became_active":"2024-01-11T09:46:25.948717+0000","last_became_peered":"2024-01-11T09:46:25.948717+0000","last_unstale":"2024-01-11T09:46:25.954020+0000","last_undegraded":"2024-01-11T09:46:25.954020+0000","last_fullsized":"2024-01-11T09:46:25.954020+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:07.160563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.934325+0000","last_change":"2024-01-11T09:46:25.225980+0000","last_active":"2024-01-11T09:46:25.934325+0000","last_peered":"2024-01-11T09:46:25.934325+0000","last_clean":"2024-01-11T09:46:25.934325+0000","last_became_active":"2024-01-11T09:46:25.225444+0000","last_became_peered":"2024-01-11T09:46:25.225444+0000","last_unstale":"2024-01-11T09:46:25.934325+0000","last_undegraded":"2024-01-11T09:46:25.934325+0000","last_fullsized":"2024-01-11T09:46:25.934325+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:30:19.648096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.955447+0000","last_change":"2024-01-11T09:46:25.953219+0000","last_active":"2024-01-11T09:46:25.955447+0000","last_peered":"2024-01-11T09:46:25.955447+0000","last_clean":"2024-01-11T09:46:25.955447+0000","last_became_active":"2024-01-11T09:46:25.952967+0000","last_became_peered":"2024-01-11T09:46:25.952967+0000","last_unstale":"2024-01-11T09:46:25.955447+0000","last_undegraded":"2024-01-11T09:46:25.955447+0000","last_fullsized":"2024-01-11T09:46:25.955447+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:07:13.288202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.942322+0000","last_change":"2024-01-11T09:46:25.237685+0000","last_active":"2024-01-11T09:46:25.942322+0000","last_peered":"2024-01-11T09:46:25.942322+0000","last_clean":"2024-01-11T09:46:25.942322+0000","last_became_active":"2024-01-11T09:46:25.237419+0000","last_became_peered":"2024-01-11T09:46:25.237419+0000","last_unstale":"2024-01-11T09:46:25.942322+0000","last_undegraded":"2024-01-11T09:46:25.942322+0000","last_fullsized":"2024-01-11T09:46:25.942322+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:52.221452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958490+0000","last_change":"2024-01-11T09:46:25.950838+0000","last_active":"2024-01-11T09:46:25.958490+0000","last_peered":"2024-01-11T09:46:25.958490+0000","last_clean":"2024-01-11T09:46:25.958490+0000","last_became_active":"2024-01-11T09:46:25.950551+0000","last_became_peered":"2024-01-11T09:46:25.950551+0000","last_unstale":"2024-01-11T09:46:25.958490+0000","last_undegraded":"2024-01-11T09:46:25.958490+0000","last_fullsized":"2024-01-11T09:46:25.958490+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:39:05.971547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.935051+0000","last_change":"2024-01-11T09:46:25.232121+0000","last_active":"2024-01-11T09:46:25.935051+0000","last_peered":"2024-01-11T09:46:25.935051+0000","last_clean":"2024-01-11T09:46:25.935051+0000","last_became_active":"2024-01-11T09:46:25.231859+0000","last_became_peered":"2024-01-11T09:46:25.231859+0000","last_unstale":"2024-01-11T09:46:25.935051+0000","last_undegraded":"2024-01-11T09:46:25.935051+0000","last_fullsized":"2024-01-11T09:46:25.935051+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:07:12.279056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958248+0000","last_change":"2024-01-11T09:46:25.948806+0000","last_active":"2024-01-11T09:46:25.958248+0000","last_peered":"2024-01-11T09:46:25.958248+0000","last_clean":"2024-01-11T09:46:25.958248+0000","last_became_active":"2024-01-11T09:46:25.948526+0000","last_became_peered":"2024-01-11T09:46:25.948526+0000","last_unstale":"2024-01-11T09:46:25.958248+0000","last_undegraded":"2024-01-11T09:46:25.958248+0000","last_fullsized":"2024-01-11T09:46:25.958248+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:51:19.504238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"234'3","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.248539+0000","last_change":"2024-01-11T09:46:25.238242+0000","last_active":"2024-01-11T09:46:26.248539+0000","last_peered":"2024-01-11T09:46:26.248539+0000","last_clean":"2024-01-11T09:46:26.248539+0000","last_became_active":"2024-01-11T09:46:25.237965+0000","last_became_peered":"2024-01-11T09:46:25.237965+0000","last_unstale":"2024-01-11T09:46:26.248539+0000","last_undegraded":"2024-01-11T09:46:26.248539+0000","last_fullsized":"2024-01-11T09:46:26.248539+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:58:02.388540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.959891+0000","last_change":"2024-01-11T09:46:25.950108+0000","last_active":"2024-01-11T09:46:25.959891+0000","last_peered":"2024-01-11T09:46:25.959891+0000","last_clean":"2024-01-11T09:46:25.959891+0000","last_became_active":"2024-01-11T09:46:25.949796+0000","last_became_peered":"2024-01-11T09:46:25.949796+0000","last_unstale":"2024-01-11T09:46:25.959891+0000","last_undegraded":"2024-01-11T09:46:25.959891+0000","last_fullsized":"2024-01-11T09:46:25.959891+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:11:07.968385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"234'6","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.250731+0000","last_change":"2024-01-11T09:46:25.239177+0000","last_active":"2024-01-11T09:46:26.250731+0000","last_peered":"2024-01-11T09:46:26.250731+0000","last_clean":"2024-01-11T09:46:26.250731+0000","last_became_active":"2024-01-11T09:46:25.238865+0000","last_became_peered":"2024-01-11T09:46:25.238865+0000","last_unstale":"2024-01-11T09:46:26.250731+0000","last_undegraded":"2024-01-11T09:46:26.250731+0000","last_fullsized":"2024-01-11T09:46:26.250731+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:29:50.151952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.962578+0000","last_change":"2024-01-11T09:46:25.951823+0000","last_active":"2024-01-11T09:46:25.962578+0000","last_peered":"2024-01-11T09:46:25.962578+0000","last_clean":"2024-01-11T09:46:25.962578+0000","last_became_active":"2024-01-11T09:46:25.951493+0000","last_became_peered":"2024-01-11T09:46:25.951493+0000","last_unstale":"2024-01-11T09:46:25.962578+0000","last_undegraded":"2024-01-11T09:46:25.962578+0000","last_fullsized":"2024-01-11T09:46:25.962578+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:59:22.974857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"234'2","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.248169+0000","last_change":"2024-01-11T09:46:25.244293+0000","last_active":"2024-01-11T09:46:26.248169+0000","last_peered":"2024-01-11T09:46:26.248169+0000","last_clean":"2024-01-11T09:46:26.248169+0000","last_became_active":"2024-01-11T09:46:25.243959+0000","last_became_peered":"2024-01-11T09:46:25.243959+0000","last_unstale":"2024-01-11T09:46:26.248169+0000","last_undegraded":"2024-01-11T09:46:26.248169+0000","last_fullsized":"2024-01-11T09:46:26.248169+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:13:10.353715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.959593+0000","last_change":"2024-01-11T09:46:25.958245+0000","last_active":"2024-01-11T09:46:25.959593+0000","last_peered":"2024-01-11T09:46:25.959593+0000","last_clean":"2024-01-11T09:46:25.959593+0000","last_became_active":"2024-01-11T09:46:25.957893+0000","last_became_peered":"2024-01-11T09:46:25.957893+0000","last_unstale":"2024-01-11T09:46:25.959593+0000","last_undegraded":"2024-01-11T09:46:25.959593+0000","last_fullsized":"2024-01-11T09:46:25.959593+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:43:22.062566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.935100+0000","last_change":"2024-01-11T09:46:25.231846+0000","last_active":"2024-01-11T09:46:25.935100+0000","last_peered":"2024-01-11T09:46:25.935100+0000","last_clean":"2024-01-11T09:46:25.935100+0000","last_became_active":"2024-01-11T09:46:25.229498+0000","last_became_peered":"2024-01-11T09:46:25.229498+0000","last_unstale":"2024-01-11T09:46:25.935100+0000","last_undegraded":"2024-01-11T09:46:25.935100+0000","last_fullsized":"2024-01-11T09:46:25.935100+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:53:34.130109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.953370+0000","last_change":"2024-01-11T09:46:25.947333+0000","last_active":"2024-01-11T09:46:25.953370+0000","last_peered":"2024-01-11T09:46:25.953370+0000","last_clean":"2024-01-11T09:46:25.953370+0000","last_became_active":"2024-01-11T09:46:25.946307+0000","last_became_peered":"2024-01-11T09:46:25.946307+0000","last_unstale":"2024-01-11T09:46:25.953370+0000","last_undegraded":"2024-01-11T09:46:25.953370+0000","last_fullsized":"2024-01-11T09:46:25.953370+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:32:47.063220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.948916+0000","last_change":"2024-01-11T09:46:25.239044+0000","last_active":"2024-01-11T09:46:25.948916+0000","last_peered":"2024-01-11T09:46:25.948916+0000","last_clean":"2024-01-11T09:46:25.948916+0000","last_became_active":"2024-01-11T09:46:25.238780+0000","last_became_peered":"2024-01-11T09:46:25.238780+0000","last_unstale":"2024-01-11T09:46:25.948916+0000","last_undegraded":"2024-01-11T09:46:25.948916+0000","last_fullsized":"2024-01-11T09:46:25.948916+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:34:37.940301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.964090+0000","last_change":"2024-01-11T09:46:25.962027+0000","last_active":"2024-01-11T09:46:25.964090+0000","last_peered":"2024-01-11T09:46:25.964090+0000","last_clean":"2024-01-11T09:46:25.964090+0000","last_became_active":"2024-01-11T09:46:25.961646+0000","last_became_peered":"2024-01-11T09:46:25.961646+0000","last_unstale":"2024-01-11T09:46:25.964090+0000","last_undegraded":"2024-01-11T09:46:25.964090+0000","last_fullsized":"2024-01-11T09:46:25.964090+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:39:05.971547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"234'8","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.249576+0000","last_change":"2024-01-11T09:46:25.238694+0000","last_active":"2024-01-11T09:46:26.249576+0000","last_peered":"2024-01-11T09:46:26.249576+0000","last_clean":"2024-01-11T09:46:26.249576+0000","last_became_active":"2024-01-11T09:46:25.237565+0000","last_became_peered":"2024-01-11T09:46:25.237565+0000","last_unstale":"2024-01-11T09:46:26.249576+0000","last_undegraded":"2024-01-11T09:46:26.249576+0000","last_fullsized":"2024-01-11T09:46:26.249576+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:25:08.579955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.954111+0000","last_change":"2024-01-11T09:46:25.946186+0000","last_active":"2024-01-11T09:46:25.954111+0000","last_peered":"2024-01-11T09:46:25.954111+0000","last_clean":"2024-01-11T09:46:25.954111+0000","last_became_active":"2024-01-11T09:46:25.945907+0000","last_became_peered":"2024-01-11T09:46:25.945907+0000","last_unstale":"2024-01-11T09:46:25.954111+0000","last_undegraded":"2024-01-11T09:46:25.954111+0000","last_fullsized":"2024-01-11T09:46:25.954111+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:26:51.516156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"234'1","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.242887+0000","last_change":"2024-01-11T09:46:25.241687+0000","last_active":"2024-01-11T09:46:26.242887+0000","last_peered":"2024-01-11T09:46:26.242887+0000","last_clean":"2024-01-11T09:46:26.242887+0000","last_became_active":"2024-01-11T09:46:25.241369+0000","last_became_peered":"2024-01-11T09:46:25.241369+0000","last_unstale":"2024-01-11T09:46:26.242887+0000","last_undegraded":"2024-01-11T09:46:26.242887+0000","last_fullsized":"2024-01-11T09:46:26.242887+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:40.145683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958691+0000","last_change":"2024-01-11T09:46:25.951531+0000","last_active":"2024-01-11T09:46:25.958691+0000","last_peered":"2024-01-11T09:46:25.958691+0000","last_clean":"2024-01-11T09:46:25.958691+0000","last_became_active":"2024-01-11T09:46:25.950791+0000","last_became_peered":"2024-01-11T09:46:25.950791+0000","last_unstale":"2024-01-11T09:46:25.958691+0000","last_undegraded":"2024-01-11T09:46:25.958691+0000","last_fullsized":"2024-01-11T09:46:25.958691+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:09:35.408321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"234'2","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.249024+0000","last_change":"2024-01-11T09:46:25.234950+0000","last_active":"2024-01-11T09:46:26.249024+0000","last_peered":"2024-01-11T09:46:26.249024+0000","last_clean":"2024-01-11T09:46:26.249024+0000","last_became_active":"2024-01-11T09:46:25.234510+0000","last_became_peered":"2024-01-11T09:46:25.234510+0000","last_unstale":"2024-01-11T09:46:26.249024+0000","last_undegraded":"2024-01-11T09:46:26.249024+0000","last_fullsized":"2024-01-11T09:46:26.249024+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:33:13.743309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.963426+0000","last_change":"2024-01-11T09:46:25.961077+0000","last_active":"2024-01-11T09:46:25.963426+0000","last_peered":"2024-01-11T09:46:25.963426+0000","last_clean":"2024-01-11T09:46:25.963426+0000","last_became_active":"2024-01-11T09:46:25.960690+0000","last_became_peered":"2024-01-11T09:46:25.960690+0000","last_unstale":"2024-01-11T09:46:25.963426+0000","last_undegraded":"2024-01-11T09:46:25.963426+0000","last_fullsized":"2024-01-11T09:46:25.963426+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:59:52.045856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"234'1","reported_seq":11,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.246122+0000","last_change":"2024-01-11T09:46:25.235005+0000","last_active":"2024-01-11T09:46:26.246122+0000","last_peered":"2024-01-11T09:46:26.246122+0000","last_clean":"2024-01-11T09:46:26.246122+0000","last_became_active":"2024-01-11T09:46:25.234748+0000","last_became_peered":"2024-01-11T09:46:25.234748+0000","last_unstale":"2024-01-11T09:46:26.246122+0000","last_undegraded":"2024-01-11T09:46:26.246122+0000","last_fullsized":"2024-01-11T09:46:26.246122+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:29:50.151952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.957997+0000","last_change":"2024-01-11T09:46:25.952457+0000","last_active":"2024-01-11T09:46:25.957997+0000","last_peered":"2024-01-11T09:46:25.957997+0000","last_clean":"2024-01-11T09:46:25.957997+0000","last_became_active":"2024-01-11T09:46:25.948394+0000","last_became_peered":"2024-01-11T09:46:25.948394+0000","last_unstale":"2024-01-11T09:46:25.957997+0000","last_undegraded":"2024-01-11T09:46:25.957997+0000","last_fullsized":"2024-01-11T09:46:25.957997+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:49:33.997022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.944053+0000","last_change":"2024-01-11T09:46:25.242589+0000","last_active":"2024-01-11T09:46:25.944053+0000","last_peered":"2024-01-11T09:46:25.944053+0000","last_clean":"2024-01-11T09:46:25.944053+0000","last_became_active":"2024-01-11T09:46:25.241879+0000","last_became_peered":"2024-01-11T09:46:25.241879+0000","last_unstale":"2024-01-11T09:46:25.944053+0000","last_undegraded":"2024-01-11T09:46:25.944053+0000","last_fullsized":"2024-01-11T09:46:25.944053+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:40.145683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958806+0000","last_change":"2024-01-11T09:46:25.953778+0000","last_active":"2024-01-11T09:46:25.958806+0000","last_peered":"2024-01-11T09:46:25.958806+0000","last_clean":"2024-01-11T09:46:25.958806+0000","last_became_active":"2024-01-11T09:46:25.953471+0000","last_became_peered":"2024-01-11T09:46:25.953471+0000","last_unstale":"2024-01-11T09:46:25.958806+0000","last_undegraded":"2024-01-11T09:46:25.958806+0000","last_fullsized":"2024-01-11T09:46:25.958806+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:30:20.657242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"234'2","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.246689+0000","last_change":"2024-01-11T09:46:25.236418+0000","last_active":"2024-01-11T09:46:26.246689+0000","last_peered":"2024-01-11T09:46:26.246689+0000","last_clean":"2024-01-11T09:46:26.246689+0000","last_became_active":"2024-01-11T09:46:25.236155+0000","last_became_peered":"2024-01-11T09:46:25.236155+0000","last_unstale":"2024-01-11T09:46:26.246689+0000","last_undegraded":"2024-01-11T09:46:26.246689+0000","last_fullsized":"2024-01-11T09:46:26.246689+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:59:13.998537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.957442+0000","last_change":"2024-01-11T09:46:25.954393+0000","last_active":"2024-01-11T09:46:25.957442+0000","last_peered":"2024-01-11T09:46:25.957442+0000","last_clean":"2024-01-11T09:46:25.957442+0000","last_became_active":"2024-01-11T09:46:25.954139+0000","last_became_peered":"2024-01-11T09:46:25.954139+0000","last_unstale":"2024-01-11T09:46:25.957442+0000","last_undegraded":"2024-01-11T09:46:25.957442+0000","last_fullsized":"2024-01-11T09:46:25.957442+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:33:14.752455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.943596+0000","last_change":"2024-01-11T09:46:25.243586+0000","last_active":"2024-01-11T09:46:25.943596+0000","last_peered":"2024-01-11T09:46:25.943596+0000","last_clean":"2024-01-11T09:46:25.943596+0000","last_became_active":"2024-01-11T09:46:25.243184+0000","last_became_peered":"2024-01-11T09:46:25.243184+0000","last_unstale":"2024-01-11T09:46:25.943596+0000","last_undegraded":"2024-01-11T09:46:25.943596+0000","last_fullsized":"2024-01-11T09:46:25.943596+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:01:24.091035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960346+0000","last_change":"2024-01-11T09:46:25.953319+0000","last_active":"2024-01-11T09:46:25.960346+0000","last_peered":"2024-01-11T09:46:25.960346+0000","last_clean":"2024-01-11T09:46:25.960346+0000","last_became_active":"2024-01-11T09:46:25.953063+0000","last_became_peered":"2024-01-11T09:46:25.953063+0000","last_unstale":"2024-01-11T09:46:25.960346+0000","last_undegraded":"2024-01-11T09:46:25.960346+0000","last_fullsized":"2024-01-11T09:46:25.960346+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:31:56.044420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.945981+0000","last_change":"2024-01-11T09:46:25.243348+0000","last_active":"2024-01-11T09:46:25.945981+0000","last_peered":"2024-01-11T09:46:25.945981+0000","last_clean":"2024-01-11T09:46:25.945981+0000","last_became_active":"2024-01-11T09:46:25.243043+0000","last_became_peered":"2024-01-11T09:46:25.243043+0000","last_unstale":"2024-01-11T09:46:25.945981+0000","last_undegraded":"2024-01-11T09:46:25.945981+0000","last_fullsized":"2024-01-11T09:46:25.945981+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:13:10.353715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.962789+0000","last_change":"2024-01-11T09:46:25.956539+0000","last_active":"2024-01-11T09:46:25.962789+0000","last_peered":"2024-01-11T09:46:25.962789+0000","last_clean":"2024-01-11T09:46:25.962789+0000","last_became_active":"2024-01-11T09:46:25.953109+0000","last_became_peered":"2024-01-11T09:46:25.953109+0000","last_unstale":"2024-01-11T09:46:25.962789+0000","last_undegraded":"2024-01-11T09:46:25.962789+0000","last_fullsized":"2024-01-11T09:46:25.962789+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:39:37.637123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.948443+0000","last_change":"2024-01-11T09:46:25.234212+0000","last_active":"2024-01-11T09:46:25.948443+0000","last_peered":"2024-01-11T09:46:25.948443+0000","last_clean":"2024-01-11T09:46:25.948443+0000","last_became_active":"2024-01-11T09:46:25.233918+0000","last_became_peered":"2024-01-11T09:46:25.233918+0000","last_unstale":"2024-01-11T09:46:25.948443+0000","last_undegraded":"2024-01-11T09:46:25.948443+0000","last_fullsized":"2024-01-11T09:46:25.948443+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:39:17.328191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.961103+0000","last_change":"2024-01-11T09:46:25.953810+0000","last_active":"2024-01-11T09:46:25.961103+0000","last_peered":"2024-01-11T09:46:25.961103+0000","last_clean":"2024-01-11T09:46:25.961103+0000","last_became_active":"2024-01-11T09:46:25.953201+0000","last_became_peered":"2024-01-11T09:46:25.953201+0000","last_unstale":"2024-01-11T09:46:25.961103+0000","last_undegraded":"2024-01-11T09:46:25.961103+0000","last_fullsized":"2024-01-11T09:46:25.961103+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:25.226766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"234'2","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.248570+0000","last_change":"2024-01-11T09:46:25.245359+0000","last_active":"2024-01-11T09:46:26.248570+0000","last_peered":"2024-01-11T09:46:26.248570+0000","last_clean":"2024-01-11T09:46:26.248570+0000","last_became_active":"2024-01-11T09:46:25.244994+0000","last_became_peered":"2024-01-11T09:46:25.244994+0000","last_unstale":"2024-01-11T09:46:26.248570+0000","last_undegraded":"2024-01-11T09:46:26.248570+0000","last_fullsized":"2024-01-11T09:46:26.248570+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:42:13.269515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.956181+0000","last_change":"2024-01-11T09:46:25.952007+0000","last_active":"2024-01-11T09:46:25.956181+0000","last_peered":"2024-01-11T09:46:25.956181+0000","last_clean":"2024-01-11T09:46:25.956181+0000","last_became_active":"2024-01-11T09:46:25.951401+0000","last_became_peered":"2024-01-11T09:46:25.951401+0000","last_unstale":"2024-01-11T09:46:25.956181+0000","last_undegraded":"2024-01-11T09:46:25.956181+0000","last_fullsized":"2024-01-11T09:46:25.956181+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:29:18.608960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.939276+0000","last_change":"2024-01-11T09:46:25.236787+0000","last_active":"2024-01-11T09:46:25.939276+0000","last_peered":"2024-01-11T09:46:25.939276+0000","last_clean":"2024-01-11T09:46:25.939276+0000","last_became_active":"2024-01-11T09:46:25.236503+0000","last_became_peered":"2024-01-11T09:46:25.236503+0000","last_unstale":"2024-01-11T09:46:25.939276+0000","last_undegraded":"2024-01-11T09:46:25.939276+0000","last_fullsized":"2024-01-11T09:46:25.939276+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:40:17.481276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960608+0000","last_change":"2024-01-11T09:46:25.953624+0000","last_active":"2024-01-11T09:46:25.960608+0000","last_peered":"2024-01-11T09:46:25.960608+0000","last_clean":"2024-01-11T09:46:25.960608+0000","last_became_active":"2024-01-11T09:46:25.953347+0000","last_became_peered":"2024-01-11T09:46:25.953347+0000","last_unstale":"2024-01-11T09:46:25.960608+0000","last_undegraded":"2024-01-11T09:46:25.960608+0000","last_fullsized":"2024-01-11T09:46:25.960608+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:56:15.022301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.946699+0000","last_change":"2024-01-11T09:46:25.238938+0000","last_active":"2024-01-11T09:46:25.946699+0000","last_peered":"2024-01-11T09:46:25.946699+0000","last_clean":"2024-01-11T09:46:25.946699+0000","last_became_active":"2024-01-11T09:46:25.238214+0000","last_became_peered":"2024-01-11T09:46:25.238214+0000","last_unstale":"2024-01-11T09:46:25.946699+0000","last_undegraded":"2024-01-11T09:46:25.946699+0000","last_fullsized":"2024-01-11T09:46:25.946699+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:30:58.825981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960351+0000","last_change":"2024-01-11T09:46:25.954867+0000","last_active":"2024-01-11T09:46:25.960351+0000","last_peered":"2024-01-11T09:46:25.960351+0000","last_clean":"2024-01-11T09:46:25.960351+0000","last_became_active":"2024-01-11T09:46:25.954613+0000","last_became_peered":"2024-01-11T09:46:25.954613+0000","last_unstale":"2024-01-11T09:46:25.960351+0000","last_undegraded":"2024-01-11T09:46:25.960351+0000","last_fullsized":"2024-01-11T09:46:25.960351+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:11:07.968385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.944325+0000","last_change":"2024-01-11T09:46:25.244239+0000","last_active":"2024-01-11T09:46:25.944325+0000","last_peered":"2024-01-11T09:46:25.944325+0000","last_clean":"2024-01-11T09:46:25.944325+0000","last_became_active":"2024-01-11T09:46:25.243663+0000","last_became_peered":"2024-01-11T09:46:25.243663+0000","last_unstale":"2024-01-11T09:46:25.944325+0000","last_undegraded":"2024-01-11T09:46:25.944325+0000","last_fullsized":"2024-01-11T09:46:25.944325+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:23:51.951171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.964142+0000","last_change":"2024-01-11T09:46:25.962471+0000","last_active":"2024-01-11T09:46:25.964142+0000","last_peered":"2024-01-11T09:46:25.964142+0000","last_clean":"2024-01-11T09:46:25.964142+0000","last_became_active":"2024-01-11T09:46:25.959247+0000","last_became_peered":"2024-01-11T09:46:25.959247+0000","last_unstale":"2024-01-11T09:46:25.964142+0000","last_undegraded":"2024-01-11T09:46:25.964142+0000","last_fullsized":"2024-01-11T09:46:25.964142+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:31:56.044420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.950613+0000","last_change":"2024-01-11T09:46:25.242486+0000","last_active":"2024-01-11T09:46:25.950613+0000","last_peered":"2024-01-11T09:46:25.950613+0000","last_clean":"2024-01-11T09:46:25.950613+0000","last_became_active":"2024-01-11T09:46:25.242231+0000","last_became_peered":"2024-01-11T09:46:25.242231+0000","last_unstale":"2024-01-11T09:46:25.950613+0000","last_undegraded":"2024-01-11T09:46:25.950613+0000","last_fullsized":"2024-01-11T09:46:25.950613+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:23:51.951171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.958879+0000","last_change":"2024-01-11T09:46:25.954929+0000","last_active":"2024-01-11T09:46:25.958879+0000","last_peered":"2024-01-11T09:46:25.958879+0000","last_clean":"2024-01-11T09:46:25.958879+0000","last_became_active":"2024-01-11T09:46:25.951640+0000","last_became_peered":"2024-01-11T09:46:25.951640+0000","last_unstale":"2024-01-11T09:46:25.958879+0000","last_undegraded":"2024-01-11T09:46:25.958879+0000","last_fullsized":"2024-01-11T09:46:25.958879+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:28:55.053850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.935714+0000","last_change":"2024-01-11T09:46:25.234604+0000","last_active":"2024-01-11T09:46:25.935714+0000","last_peered":"2024-01-11T09:46:25.935714+0000","last_clean":"2024-01-11T09:46:25.935714+0000","last_became_active":"2024-01-11T09:46:25.234348+0000","last_became_peered":"2024-01-11T09:46:25.234348+0000","last_unstale":"2024-01-11T09:46:25.935714+0000","last_undegraded":"2024-01-11T09:46:25.935714+0000","last_fullsized":"2024-01-11T09:46:25.935714+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:01:05.259333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960490+0000","last_change":"2024-01-11T09:46:25.956295+0000","last_active":"2024-01-11T09:46:25.960490+0000","last_peered":"2024-01-11T09:46:25.960490+0000","last_clean":"2024-01-11T09:46:25.960490+0000","last_became_active":"2024-01-11T09:46:25.949169+0000","last_became_peered":"2024-01-11T09:46:25.949169+0000","last_unstale":"2024-01-11T09:46:25.960490+0000","last_undegraded":"2024-01-11T09:46:25.960490+0000","last_fullsized":"2024-01-11T09:46:25.960490+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:07.160563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.934226+0000","last_change":"2024-01-11T09:46:25.231375+0000","last_active":"2024-01-11T09:46:25.934226+0000","last_peered":"2024-01-11T09:46:25.934226+0000","last_clean":"2024-01-11T09:46:25.934226+0000","last_became_active":"2024-01-11T09:46:25.231115+0000","last_became_peered":"2024-01-11T09:46:25.231115+0000","last_unstale":"2024-01-11T09:46:25.934226+0000","last_undegraded":"2024-01-11T09:46:25.934226+0000","last_fullsized":"2024-01-11T09:46:25.934226+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:06:11.124203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.960030+0000","last_change":"2024-01-11T09:46:25.957641+0000","last_active":"2024-01-11T09:46:25.960030+0000","last_peered":"2024-01-11T09:46:25.960030+0000","last_clean":"2024-01-11T09:46:25.960030+0000","last_became_active":"2024-01-11T09:46:25.955768+0000","last_became_peered":"2024-01-11T09:46:25.955768+0000","last_unstale":"2024-01-11T09:46:25.960030+0000","last_undegraded":"2024-01-11T09:46:25.960030+0000","last_fullsized":"2024-01-11T09:46:25.960030+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:58.843659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":10,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.934820+0000","last_change":"2024-01-11T09:46:25.239061+0000","last_active":"2024-01-11T09:46:25.934820+0000","last_peered":"2024-01-11T09:46:25.934820+0000","last_clean":"2024-01-11T09:46:25.934820+0000","last_became_active":"2024-01-11T09:46:25.238802+0000","last_became_peered":"2024-01-11T09:46:25.238802+0000","last_unstale":"2024-01-11T09:46:25.934820+0000","last_undegraded":"2024-01-11T09:46:25.934820+0000","last_fullsized":"2024-01-11T09:46:25.934820+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:57.834513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.951329+0000","last_change":"2024-01-11T09:46:25.948656+0000","last_active":"2024-01-11T09:46:25.951329+0000","last_peered":"2024-01-11T09:46:25.951329+0000","last_clean":"2024-01-11T09:46:25.951329+0000","last_became_active":"2024-01-11T09:46:25.948273+0000","last_became_peered":"2024-01-11T09:46:25.948273+0000","last_unstale":"2024-01-11T09:46:25.951329+0000","last_undegraded":"2024-01-11T09:46:25.951329+0000","last_fullsized":"2024-01-11T09:46:25.951329+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:36:07.163334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"234'4","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:26.247953+0000","last_change":"2024-01-11T09:46:25.235720+0000","last_active":"2024-01-11T09:46:26.247953+0000","last_peered":"2024-01-11T09:46:26.247953+0000","last_clean":"2024-01-11T09:46:26.247953+0000","last_became_active":"2024-01-11T09:46:25.235445+0000","last_became_peered":"2024-01-11T09:46:25.235445+0000","last_unstale":"2024-01-11T09:46:26.247953+0000","last_undegraded":"2024-01-11T09:46:26.247953+0000","last_fullsized":"2024-01-11T09:46:26.247953+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:53:59.525452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"221'388","reported_seq":1295,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-01-11T09:46:25.945190+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:46:25.945190+0000","last_peered":"2024-01-11T09:46:25.945190+0000","last_clean":"2024-01-11T09:46:25.945190+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:46:25.945190+0000","last_undegraded":"2024-01-11T09:46:25.945190+0000","last_fullsized":"2024-01-11T09:46:25.945190+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":651,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":651,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":388,"ondisk_log_size":388,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575982,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300764,"kb_used_data":3476,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071076,"statfs":{"total":96636764160,"available":96328781824,"internally_reserved":0,"allocated":3559424,"data_stored":2014654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.69599999999999995,"15min":0.73299999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.36599999999999999},"max":{"1min":0.93000000000000005,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.72599999999999998,"15min":0.747},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.026,"5min":1.0609999999999999,"15min":1.0669999999999999},"last":0.80100000000000005}]},{"osd":1,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.70699999999999996,"15min":0.73299999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41599999999999998},"max":{"1min":0.91600000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.71799999999999997,"15min":0.74199999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.95899999999999996,"5min":0.999,"15min":1.0820000000000001},"last":0.61399999999999999}]},{"osd":2,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.73599999999999999,"15min":0.74399999999999999},"min":{"1min":0.44500000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.93799999999999994,"5min":0.94399999999999995,"15min":1.0600000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73199999999999998,"15min":0.754},"min":{"1min":0.503,"5min":0.44800000000000001,"15min":0.44500000000000001},"max":{"1min":0.92800000000000005,"5min":0.96999999999999997,"15min":1.153},"last":0.66000000000000003}]},{"osd":3,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73799999999999999,"15min":0.76700000000000002},"min":{"1min":0.51600000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.93000000000000005,"5min":0.96699999999999997,"15min":1.0660000000000001},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73699999999999999,"15min":0.75600000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.94099999999999995,"5min":1,"15min":1.0880000000000001},"last":0.74199999999999999}]},{"osd":4,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.432,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0720000000000001,"5min":1.0860000000000001,"15min":1.107},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.629,"15min":0.63800000000000001},"min":{"1min":0.47699999999999998,"5min":0.44400000000000001,"15min":0.39700000000000002},"max":{"1min":0.89000000000000001,"5min":0.995,"15min":1.0960000000000001},"last":0.76300000000000001}]},{"osd":5,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.61199999999999999,"15min":0.63800000000000001},"min":{"1min":0.48999999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86599999999999999,"5min":0.86699999999999999,"15min":1.0109999999999999},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.67900000000000005,"15min":0.68400000000000005},"min":{"1min":0.46899999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.92200000000000004}]},{"osd":6,"last update":"Thu Jan 11 09:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.76300000000000001,"15min":0.77900000000000003},"min":{"1min":0.55900000000000005,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.747,"15min":0.76000000000000001},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.45800000000000002},"max":{"1min":0.95899999999999996,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":1.002}]}]},{"osd":6,"up_from":13,"seq":55834575981,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312476,"kb_used_data":2900,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059364,"statfs":{"total":96636764160,"available":96316788736,"internally_reserved":0,"allocated":2969600,"data_stored":1424792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.65200000000000002,"15min":0.70399999999999996},"min":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89100000000000001,"5min":0.94199999999999995,"15min":0.999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.71299999999999997,"15min":0.73899999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.216},"last":0.60899999999999999}]},{"osd":1,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.67500000000000004,"15min":0.73199999999999998},"min":{"1min":0.44900000000000001,"5min":0.42799999999999999,"15min":0.41699999999999998},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":1.3129999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.68899999999999995,"15min":0.72499999999999998},"min":{"1min":0.44400000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.95499999999999996,"5min":0.98299999999999998,"15min":1.0840000000000001},"last":0.503}]},{"osd":2,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.70699999999999996,"15min":0.74299999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.0069999999999999,"5min":1.034,"15min":1.4179999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.72099999999999997,"15min":0.75},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":0.439}]},{"osd":3,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.72999999999999998,"15min":0.754},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.92600000000000005,"5min":1.012,"15min":1.014},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.70599999999999996,"15min":0.76200000000000001},"min":{"1min":0.498,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.91300000000000003,"5min":0.95999999999999996,"15min":1.165},"last":0.64300000000000002}]},{"osd":4,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.38400000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78000000000000003,"5min":0.91700000000000004,"15min":1.0309999999999999},"last":0.627},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.626,"15min":0.61599999999999999},"min":{"1min":0.40799999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92100000000000004,"5min":0.94799999999999995,"15min":1.1259999999999999},"last":0.66300000000000003}]},{"osd":5,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.65000000000000002,"15min":0.65900000000000003},"min":{"1min":0.44600000000000001,"5min":0.41499999999999998,"15min":0.40000000000000002},"max":{"1min":0.93100000000000005,"5min":0.97899999999999998,"15min":1.369},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70299999999999996,"15min":0.70499999999999996},"min":{"1min":0.56799999999999995,"5min":0.46700000000000003,"15min":0.45400000000000001},"max":{"1min":0.85999999999999999,"5min":0.999,"15min":1.0960000000000001},"last":0.67700000000000005}]},{"osd":7,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.70699999999999996,"15min":0.70899999999999996},"min":{"1min":0.47199999999999998,"5min":0.45600000000000002,"15min":0.45100000000000001},"max":{"1min":0.91300000000000003,"5min":1.089,"15min":1.484},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.754,"15min":0.76700000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.98299999999999998,"5min":1.1379999999999999,"15min":1.603},"last":0.74099999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575983,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308392,"kb_used_data":2912,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063448,"statfs":{"total":96636764160,"available":96320970752,"internally_reserved":0,"allocated":2981888,"data_stored":1425876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.51400000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.315},"max":{"1min":0.69799999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.439},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.59899999999999998,"15min":0.61899999999999999},"min":{"1min":0.41899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.75600000000000001,"5min":0.85199999999999998,"15min":0.95099999999999996},"last":0.48399999999999999}]},{"osd":2,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.50700000000000001,"15min":0.51200000000000001},"min":{"1min":0.34999999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66600000000000004,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.84099999999999997,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.505}]},{"osd":3,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.46400000000000002,"5min":0.436,"15min":0.42699999999999999},"max":{"1min":0.85199999999999998,"5min":0.91600000000000004,"15min":0.96699999999999997},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.58099999999999996,"15min":0.58199999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.71299999999999997,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.42099999999999999}]},{"osd":4,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.79500000000000004,"15min":0.81599999999999995},"min":{"1min":0.54600000000000004,"5min":0.51400000000000001,"15min":0.505},"max":{"1min":0.875,"5min":1.214,"15min":1.2969999999999999},"last":0.751},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.84399999999999997,"15min":0.85899999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1000000000000001,"5min":1.1879999999999999,"15min":1.1890000000000001},"last":0.73199999999999998}]},{"osd":5,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.82599999999999996,"15min":0.82799999999999996},"min":{"1min":0.51800000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.996,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.80800000000000005,"15min":0.80700000000000005},"min":{"1min":0.61899999999999999,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":1.0609999999999999,"5min":1.341,"15min":1.341},"last":0.67000000000000004}]},{"osd":6,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.83499999999999996,"15min":0.84699999999999998},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.56799999999999995},"max":{"1min":1.0620000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.84499999999999997,"15min":0.85399999999999998},"min":{"1min":0.60099999999999998,"5min":0.56899999999999995,"15min":0.55900000000000005},"max":{"1min":1.077,"5min":1.177,"15min":1.177},"last":0.83799999999999997}]},{"osd":7,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.88800000000000001,"15min":0.88900000000000001},"min":{"1min":0.64400000000000002,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.1359999999999999,"5min":1.421,"15min":1.421},"last":0.622},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.86299999999999999,"15min":0.85699999999999998},"min":{"1min":0.64500000000000002,"5min":0.60099999999999998,"15min":0.54500000000000004},"max":{"1min":0.96099999999999997,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.71299999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575982,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308956,"kb_used_data":3476,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062884,"statfs":{"total":96636764160,"available":96320393216,"internally_reserved":0,"allocated":3559424,"data_stored":2014618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.50700000000000001,"15min":0.51800000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.70999999999999996,"5min":0.84699999999999998,"15min":0.875},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.58399999999999996,"15min":0.59599999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77200000000000002,"5min":0.81599999999999995,"15min":0.874},"last":0.502}]},{"osd":2,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.53500000000000003,"15min":0.56499999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.61499999999999999,"5min":0.83099999999999996,"15min":0.97499999999999998},"last":0.308},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.56299999999999994,"15min":0.57999999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71899999999999997,"5min":0.84699999999999998,"15min":0.86399999999999999},"last":0.33700000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.51400000000000001,"15min":0.54000000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.73699999999999999,"5min":0.79000000000000004,"15min":0.95399999999999996},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.621,"15min":0.64800000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80200000000000005,"5min":0.90300000000000002,"15min":1.0389999999999999},"last":0.47299999999999998}]},{"osd":4,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.81399999999999995,"15min":0.81599999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0629999999999999,"5min":1.1779999999999999,"15min":1.304},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.80000000000000004,"15min":0.80600000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.93000000000000005,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.48799999999999999}]},{"osd":5,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.82299999999999995,"15min":0.83899999999999997},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0069999999999999,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.82899999999999996,"15min":0.84199999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.1100000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.623}]},{"osd":6,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.81000000000000005,"15min":0.82399999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.1000000000000001,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.84199999999999997,"15min":0.86599999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":0.95899999999999996,"5min":1.256,"15min":1.256},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.85599999999999998,"15min":0.86599999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.151,"5min":1.28,"15min":1.28},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.85899999999999999,"15min":0.872},"min":{"1min":0.57499999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.054,"5min":1.238,"15min":1.2410000000000001},"last":0.64100000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575981,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300184,"kb_used_data":2896,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071656,"statfs":{"total":96636764160,"available":96329375744,"internally_reserved":0,"allocated":2965504,"data_stored":1424306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48699999999999999,"15min":0.502},"min":{"1min":0.32800000000000001,"5min":0.29899999999999999,"15min":0.27900000000000003},"max":{"1min":0.625,"5min":0.72599999999999998,"15min":0.85399999999999998},"last":0.436},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.56499999999999995},"min":{"1min":0.38800000000000001,"5min":0.34399999999999997,"15min":0.34300000000000003},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":1.6359999999999999},"last":0.51300000000000001}]},{"osd":1,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.39500000000000002,"5min":0.38200000000000001,"15min":0.33500000000000002},"max":{"1min":0.80900000000000005,"5min":0.88800000000000001,"15min":0.93000000000000005},"last":0.5},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58899999999999997,"15min":0.59899999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.73099999999999998,"5min":0.97299999999999998,"15min":1.145},"last":0.46100000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.75,"5min":0.83099999999999996,"15min":0.96399999999999997},"last":0.376},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.63,"15min":0.65500000000000003},"min":{"1min":0.442,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.83599999999999997,"5min":0.90700000000000003,"15min":1.3620000000000001},"last":0.59999999999999998}]},{"osd":4,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.78200000000000003,"15min":0.79800000000000004},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.54300000000000004},"max":{"1min":0.95299999999999996,"5min":1.071,"15min":1.1519999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.78700000000000003,"15min":0.80000000000000004},"min":{"1min":0.54500000000000004,"5min":0.505,"15min":0.505},"max":{"1min":0.96299999999999997,"5min":1.161,"15min":1.161},"last":0.61099999999999999}]},{"osd":5,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.83099999999999996,"15min":0.86799999999999999},"min":{"1min":0.625,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.996,"5min":1.228,"15min":1.429},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.85099999999999998,"15min":0.86299999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.0389999999999999,"5min":1.1850000000000001,"15min":1.2170000000000001},"last":0.73899999999999999}]},{"osd":6,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.80000000000000004,"15min":0.82599999999999996},"min":{"1min":0.55500000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0049999999999999,"5min":1.1659999999999999,"15min":1.4139999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.83499999999999996,"15min":0.86399999999999999},"min":{"1min":0.63400000000000001,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.96599999999999997,"5min":1.2010000000000001,"15min":1.593},"last":0.77300000000000002}]},{"osd":7,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.86799999999999999,"15min":0.89200000000000002},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.57299999999999995},"max":{"1min":1.0129999999999999,"5min":1.137,"15min":2.0539999999999998},"last":0.627},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.86499999999999999,"15min":0.88100000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.55500000000000005},"max":{"1min":1.069,"5min":1.2270000000000001,"15min":1.2490000000000001},"last":0.75600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575981,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300184,"kb_used_data":2896,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071656,"statfs":{"total":96636764160,"available":96329375744,"internally_reserved":0,"allocated":2965504,"data_stored":1424770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.502,"15min":0.52000000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.314},"max":{"1min":0.76100000000000001,"5min":0.89700000000000002,"15min":0.90000000000000002},"last":0.318},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.56200000000000006,"15min":0.58699999999999997},"min":{"1min":0.372,"5min":0.36899999999999999,"15min":0.36099999999999999},"max":{"1min":0.67900000000000005,"5min":1.014,"15min":1.0449999999999999},"last":0.46999999999999997}]},{"osd":1,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.495,"15min":0.51100000000000001},"min":{"1min":0.29399999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63500000000000001,"5min":0.84199999999999997,"15min":1.304},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.38700000000000001},"max":{"1min":0.75,"5min":0.84099999999999997,"15min":1.4590000000000001},"last":0.372}]},{"osd":2,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58299999999999996,"15min":0.60599999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.72999999999999998,"5min":0.93700000000000006,"15min":1.04},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.64200000000000002,"15min":0.65800000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44700000000000001},"max":{"1min":0.83499999999999996,"5min":0.89800000000000002,"15min":0.92000000000000004},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76100000000000001,"15min":0.79000000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.92900000000000005,"5min":1.206,"15min":1.206},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.77800000000000002,"15min":0.81299999999999994},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.871,"5min":1.171,"15min":1.1830000000000001},"last":0.82699999999999996}]},{"osd":5,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.79600000000000004,"15min":0.82099999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.95399999999999996,"5min":1.1619999999999999,"15min":1.234},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81000000000000005,"15min":0.84799999999999998},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.3440000000000001},"last":0.93700000000000006}]},{"osd":6,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82999999999999996,"15min":0.85899999999999999},"min":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.077,"5min":1.1739999999999999,"15min":1.2170000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.80500000000000005,"15min":0.82899999999999996},"min":{"1min":0.55300000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.97099999999999997,"5min":1.181,"15min":1.2190000000000001},"last":0.81399999999999995}]},{"osd":7,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.83699999999999997,"15min":0.86799999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0269999999999999,"5min":1.224,"15min":1.264},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.86099999999999999,"15min":0.89500000000000002},"min":{"1min":0.59299999999999997,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.0209999999999999,"5min":1.1539999999999999,"15min":1.24},"last":0.89500000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575982,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308392,"kb_used_data":2912,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063448,"statfs":{"total":96636764160,"available":96320970752,"internally_reserved":0,"allocated":2981888,"data_stored":1425844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.75800000000000001,"15min":0.77200000000000002},"min":{"1min":0.54000000000000004,"5min":0.41199999999999998,"15min":0.39900000000000002},"max":{"1min":0.91200000000000003,"5min":0.98499999999999999,"15min":1.119},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.98499999999999999,"5min":1.853,"15min":1.853},"last":0.64000000000000001}]},{"osd":1,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.73099999999999998,"15min":0.74399999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.41399999999999998},"max":{"1min":0.90500000000000003,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.72999999999999998,"15min":0.73599999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.95899999999999996,"5min":1.639,"15min":1.639},"last":0.72699999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.747,"15min":0.751},"min":{"1min":0.54800000000000004,"5min":0.46500000000000002,"15min":0.44700000000000001},"max":{"1min":0.94499999999999995,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.75800000000000001,"15min":0.752},"min":{"1min":0.503,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.002,"5min":1.0249999999999999,"15min":1.0700000000000001},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.73899999999999999,"15min":0.745},"min":{"1min":0.52100000000000002,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.94799999999999995,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.441},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.78300000000000003,"15min":0.78000000000000003},"min":{"1min":0.56499999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.02,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.61399999999999999}]},{"osd":5,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.64100000000000001,"15min":0.624},"min":{"1min":0.42999999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.90300000000000002,"5min":0.91500000000000004,"15min":1.0049999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.66600000000000004,"15min":0.64600000000000002},"min":{"1min":0.46700000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89500000000000002,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.56899999999999995}]},{"osd":6,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.69099999999999995,"15min":0.67100000000000004},"min":{"1min":0.51000000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83899999999999997,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.72699999999999998,"15min":0.70699999999999996},"min":{"1min":0.51800000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95199999999999996,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.58799999999999997}]},{"osd":7,"last update":"Thu Jan 11 09:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.72499999999999998,"15min":0.70099999999999996},"min":{"1min":0.56399999999999995,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0489999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.78000000000000003,"15min":0.754},"min":{"1min":0.57499999999999996,"5min":0.436,"15min":0.41699999999999998},"max":{"1min":1.02,"5min":2.052,"15min":2.052},"last":0.74299999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575982,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312468,"kb_used_data":2892,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059372,"statfs":{"total":96636764160,"available":96316796928,"internally_reserved":0,"allocated":2961408,"data_stored":1424216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.71699999999999997,"15min":0.72899999999999998},"min":{"1min":0.46800000000000003,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71099999999999997,"15min":0.72399999999999998},"min":{"1min":0.51000000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.56699999999999995}]},{"osd":1,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71499999999999997,"15min":0.73399999999999999},"min":{"1min":0.46800000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91400000000000003,"5min":0.95699999999999996,"15min":0.97699999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.73299999999999998,"15min":0.73799999999999999},"min":{"1min":0.51900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.874,"5min":1.004,"15min":1.0269999999999999},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.72299999999999998,"15min":0.72999999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1,"5min":1.0269999999999999,"15min":1.028},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.73199999999999998,"15min":0.74199999999999999},"min":{"1min":0.50900000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.98799999999999999},"last":0.71799999999999997}]},{"osd":3,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.74399999999999999,"15min":0.75600000000000001},"min":{"1min":0.56200000000000006,"5min":0.52800000000000002,"15min":0.49299999999999999},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76100000000000001,"15min":0.77000000000000002},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.874}]},{"osd":4,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.63300000000000001,"15min":0.61399999999999999},"min":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41699999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.64300000000000002,"15min":0.63400000000000001},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.375},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":1.0189999999999999}]},{"osd":6,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63700000000000001,"15min":0.64300000000000002},"min":{"1min":0.433,"5min":0.42099999999999999,"15min":0.40999999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":1.0089999999999999},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69199999999999995,"15min":0.68600000000000005},"min":{"1min":0.48699999999999999,"5min":0.46200000000000002,"15min":0.42799999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.79500000000000004}]},{"osd":7,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.73199999999999998,"15min":0.73299999999999998},"min":{"1min":0.58299999999999996,"5min":0.47299999999999998,"15min":0.434},"max":{"1min":0.94799999999999995,"5min":1.0149999999999999,"15min":1.0760000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.77800000000000002,"15min":0.77600000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.47599999999999998},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":1.056}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:46:31.140 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-11T09:46:31.140 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-11T09:46:31.196 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-01-11T09:46:31.204+0000 7f17f05f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T09:46:31.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.391+0000 7fb5f97ff700 1 Processor -- start 2024-01-11T09:46:31.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.391+0000 7fb5f97ff700 1 -- start start 2024-01-11T09:46:31.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.392+0000 7fb5f97ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5f41596c0 0x7fb5f415d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:31.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.392+0000 7fb5f97ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb5f40ca670 0x7fb5f4068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:31.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.392+0000 7fb5f97ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5f4068b60 0x7fb5f4159a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:31.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.392+0000 7fb5f97ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5f4068fc0 con 0x7fb5f41596c0 2024-01-11T09:46:31.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.392+0000 7fb5f97ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5f40a9cb0 con 0x7fb5f40ca670 2024-01-11T09:46:31.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.392+0000 7fb5f97ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f415e210 con 0x7fb5f4068b60 2024-01-11T09:46:31.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.393+0000 7fb5f2ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5f41596c0 0x7fb5f415d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:31.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.393+0000 7fb5f2ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5f41596c0 0x7fb5f415d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48510/0 (socket says 172.21.15.27:48510) 2024-01-11T09:46:31.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.393+0000 7fb5f2ffd700 1 -- 172.21.15.27:0/3459706298 learned_addr learned my addr 172.21.15.27:0/3459706298 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:31.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.393+0000 7fb5f27fc700 1 --2- 172.21.15.27:0/3459706298 >> v2:172.21.15.77:3300/0 conn(0x7fb5f40ca670 0x7fb5f4068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:31.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.393+0000 7fb5f37fe700 1 --2- 172.21.15.27:0/3459706298 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5f4068b60 0x7fb5f4159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:31.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.393+0000 7fb5f2ffd700 1 -- 172.21.15.27:0/3459706298 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5f4068b60 msgr2=0x7fb5f4159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:31.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.393+0000 7fb5f2ffd700 1 --2- 172.21.15.27:0/3459706298 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5f4068b60 0x7fb5f4159a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.393+0000 7fb5f2ffd700 1 -- 172.21.15.27:0/3459706298 >> v2:172.21.15.77:3300/0 conn(0x7fb5f40ca670 msgr2=0x7fb5f4068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:31.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.393+0000 7fb5f2ffd700 1 --2- 172.21.15.27:0/3459706298 >> v2:172.21.15.77:3300/0 conn(0x7fb5f40ca670 0x7fb5f4068620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.393+0000 7fb5f2ffd700 1 -- 172.21.15.27:0/3459706298 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f415dc50 con 0x7fb5f41596c0 2024-01-11T09:46:31.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.394+0000 7fb5f2ffd700 1 --2- 172.21.15.27:0/3459706298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f41596c0 0x7fb5f415d300 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fb5f40b2c00 tx=0x7fb5e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7d261110a7f2fc8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:31.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.394+0000 7fb5f1ffb700 1 -- 172.21.15.27:0/3459706298 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e400b640 con 0x7fb5f41596c0 2024-01-11T09:46:31.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.394+0000 7fb5f1ffb700 1 -- 172.21.15.27:0/3459706298 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e400b7a0 con 0x7fb5f41596c0 2024-01-11T09:46:31.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.394+0000 7fb5f1ffb700 1 -- 172.21.15.27:0/3459706298 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e400bb60 con 0x7fb5f41596c0 2024-01-11T09:46:31.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.394+0000 7fb5f97ff700 1 -- 172.21.15.27:0/3459706298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f41596c0 msgr2=0x7fb5f415d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:31.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.394+0000 7fb5f97ff700 1 --2- 172.21.15.27:0/3459706298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f41596c0 0x7fb5f415d300 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fb5f40b2c00 tx=0x7fb5e4009250 comp rx=0 tx=0).stop 2024-01-11T09:46:31.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.394+0000 7fb5f97ff700 1 -- 172.21.15.27:0/3459706298 shutdown_connections 2024-01-11T09:46:31.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.394+0000 7fb5f97ff700 1 --2- 172.21.15.27:0/3459706298 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5f4068b60 0x7fb5f4159a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.394+0000 7fb5f97ff700 1 --2- 172.21.15.27:0/3459706298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f41596c0 0x7fb5f415d300 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.394+0000 7fb5f97ff700 1 --2- 172.21.15.27:0/3459706298 >> v2:172.21.15.77:3300/0 conn(0x7fb5f40ca670 0x7fb5f4068620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.394+0000 7fb5f97ff700 1 -- 172.21.15.27:0/3459706298 >> 172.21.15.27:0/3459706298 conn(0x7fb5f40c9f70 msgr2=0x7fb5f406deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:31.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.395+0000 7fb5f97ff700 1 -- 172.21.15.27:0/3459706298 shutdown_connections 2024-01-11T09:46:31.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.395+0000 7fb5f97ff700 1 -- 172.21.15.27:0/3459706298 wait complete. 2024-01-11T09:46:31.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.396+0000 7fb5f97ff700 1 Processor -- start 2024-01-11T09:46:31.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.396+0000 7fb5f97ff700 1 -- start start 2024-01-11T09:46:31.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.397+0000 7fb5f97ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f4068b60 0x7fb5f40aa2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:31.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.397+0000 7fb5f97ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5f40ca670 0x7fb5f40c2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:31.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.397+0000 7fb5f97ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5f41596c0 0x7fb5f40c2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:31.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.397+0000 7fb5f97ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f40ac2a0 con 0x7fb5f4068b60 2024-01-11T09:46:31.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.397+0000 7fb5f97ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f415cc50 con 0x7fb5f41596c0 2024-01-11T09:46:31.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.397+0000 7fb5f97ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f40ac560 con 0x7fb5f40ca670 2024-01-11T09:46:31.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.397+0000 7fb5f2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f4068b60 0x7fb5f40aa2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:31.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.397+0000 7fb5f2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f4068b60 0x7fb5f40aa2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48526/0 (socket says 172.21.15.27:48526) 2024-01-11T09:46:31.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.397+0000 7fb5f2ffd700 1 -- 172.21.15.27:0/3578557749 learned_addr learned my addr 172.21.15.27:0/3578557749 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:31.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.397+0000 7fb5f27fc700 1 --2- 172.21.15.27:0/3578557749 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5f40ca670 0x7fb5f40c2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:31.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.398+0000 7fb5f37fe700 1 --2- 172.21.15.27:0/3578557749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5f41596c0 0x7fb5f40c2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:31.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.398+0000 7fb5f2ffd700 1 -- 172.21.15.27:0/3578557749 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5f40ca670 msgr2=0x7fb5f40c2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:31.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.398+0000 7fb5f2ffd700 1 --2- 172.21.15.27:0/3578557749 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5f40ca670 0x7fb5f40c2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.398+0000 7fb5f2ffd700 1 -- 172.21.15.27:0/3578557749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5f41596c0 msgr2=0x7fb5f40c2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:31.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.398+0000 7fb5f2ffd700 1 --2- 172.21.15.27:0/3578557749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5f41596c0 0x7fb5f40c2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.398+0000 7fb5f2ffd700 1 -- 172.21.15.27:0/3578557749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e40094f0 con 0x7fb5f4068b60 2024-01-11T09:46:31.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.398+0000 7fb5f2ffd700 1 --2- 172.21.15.27:0/3578557749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f4068b60 0x7fb5f40aa2d0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4006fd0 tx=0x7fb5e4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:31.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.398+0000 7fb5daffd700 1 -- 172.21.15.27:0/3578557749 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e400b910 con 0x7fb5f4068b60 2024-01-11T09:46:31.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.398+0000 7fb5daffd700 1 -- 172.21.15.27:0/3578557749 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e400ba70 con 0x7fb5f4068b60 2024-01-11T09:46:31.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.398+0000 7fb5f97ff700 1 -- 172.21.15.27:0/3578557749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f415c3b0 con 0x7fb5f4068b60 2024-01-11T09:46:31.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.398+0000 7fb5daffd700 1 -- 172.21.15.27:0/3578557749 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4011650 con 0x7fb5f4068b60 2024-01-11T09:46:31.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.399+0000 7fb5f97ff700 1 -- 172.21.15.27:0/3578557749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f415c970 con 0x7fb5f4068b60 2024-01-11T09:46:31.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.400+0000 7fb5f97ff700 1 -- 172.21.15.27:0/3578557749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f404c3c0 con 0x7fb5f4068b60 2024-01-11T09:46:31.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.401+0000 7fb5daffd700 1 -- 172.21.15.27:0/3578557749 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fb5e40119c0 con 0x7fb5f4068b60 2024-01-11T09:46:31.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.401+0000 7fb5daffd700 1 --2- 172.21.15.27:0/3578557749 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb5e0067170 0x7fb5e0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:31.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.402+0000 7fb5daffd700 1 -- 172.21.15.27:0/3578557749 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5817+0+0 (secure 0 0 0) 0x7fb5e4085f70 con 0x7fb5f4068b60 2024-01-11T09:46:31.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.402+0000 7fb5f27fc700 1 --2- 172.21.15.27:0/3578557749 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb5e0067170 0x7fb5e0069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:31.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.403+0000 7fb5f27fc700 1 --2- 172.21.15.27:0/3578557749 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb5e0067170 0x7fb5e0069620 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb5f416cc40 tx=0x7fb5e80091a0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:31.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.406+0000 7fb5daffd700 1 -- 172.21.15.27:0/3578557749 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb5e40b75a0 con 0x7fb5f4068b60 2024-01-11T09:46:31.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.691+0000 7fb5f97ff700 1 -- 172.21.15.27:0/3578557749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fb5f40b10e0 con 0x7fb5f4068b60 2024-01-11T09:46:31.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.695+0000 7fb5daffd700 1 -- 172.21.15.27:0/3578557749 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v52) v1 ==== 190+0+0 (secure 0 0 0) 0x7fb5e4052ee0 con 0x7fb5f4068b60 2024-01-11T09:46:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.697+0000 7fb5f97ff700 1 -- 172.21.15.27:0/3578557749 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb5e0067170 msgr2=0x7fb5e0069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.697+0000 7fb5f97ff700 1 --2- 172.21.15.27:0/3578557749 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb5e0067170 0x7fb5e0069620 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb5f416cc40 tx=0x7fb5e80091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.697+0000 7fb5f97ff700 1 -- 172.21.15.27:0/3578557749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f4068b60 msgr2=0x7fb5f40aa2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.697+0000 7fb5f97ff700 1 --2- 172.21.15.27:0/3578557749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f4068b60 0x7fb5f40aa2d0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4006fd0 tx=0x7fb5e4017000 comp rx=0 tx=0).stop 2024-01-11T09:46:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.698+0000 7fb5f97ff700 1 -- 172.21.15.27:0/3578557749 shutdown_connections 2024-01-11T09:46:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.698+0000 7fb5f97ff700 1 --2- 172.21.15.27:0/3578557749 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb5e0067170 0x7fb5e0069620 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.698+0000 7fb5f97ff700 1 --2- 172.21.15.27:0/3578557749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5f41596c0 0x7fb5f40c2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.698+0000 7fb5f97ff700 1 --2- 172.21.15.27:0/3578557749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f4068b60 0x7fb5f40aa2d0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.698+0000 7fb5f97ff700 1 --2- 172.21.15.27:0/3578557749 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb5f40ca670 0x7fb5f40c2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.698+0000 7fb5f97ff700 1 -- 172.21.15.27:0/3578557749 >> 172.21.15.27:0/3578557749 conn(0x7fb5f40c9f70 msgr2=0x7fb5f40c3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.698+0000 7fb5f97ff700 1 -- 172.21.15.27:0/3578557749 shutdown_connections 2024-01-11T09:46:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.698+0000 7fb5f97ff700 1 -- 172.21.15.27:0/3578557749 wait complete. 2024-01-11T09:46:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-11T09:46:31.707 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:46:31.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.957+0000 7fa3025ff700 1 Processor -- start 2024-01-11T09:46:31.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.957+0000 7fa3025ff700 1 -- start start 2024-01-11T09:46:31.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.959+0000 7fa3025ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa2fc0ca670 0x7fa2fc068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:31.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.959+0000 7fa3025ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa2fc068b60 0x7fa2fc0aa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:31.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.959+0000 7fa3025ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2fc15da10 con 0x7fa2fc1596c0 2024-01-11T09:46:31.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.959+0000 7fa3025ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2fc068fc0 con 0x7fa2fc0ca670 2024-01-11T09:46:31.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.959+0000 7fa3025ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2fc0a9cb0 con 0x7fa2fc068b60 2024-01-11T09:46:31.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.959+0000 7fa2fb7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa2fc0ca670 0x7fa2fc068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:31.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.959+0000 7fa2fb7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa2fc0ca670 0x7fa2fc068620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48532/0 (socket says 172.21.15.27:48532) 2024-01-11T09:46:31.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.959+0000 7fa2fb7fe700 1 -- 172.21.15.27:0/3875034783 learned_addr learned my addr 172.21.15.27:0/3875034783 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:31.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.959+0000 7fa300b9c700 1 --2- 172.21.15.27:0/3875034783 >> v2:172.21.15.77:3300/0 conn(0x7fa2fc068b60 0x7fa2fc0aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:31.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.959+0000 7fa2fb7fe700 1 -- 172.21.15.27:0/3875034783 >> v2:172.21.15.77:3300/0 conn(0x7fa2fc068b60 msgr2=0x7fa2fc0aa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:31.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.959+0000 7fa2fb7fe700 1 --2- 172.21.15.27:0/3875034783 >> v2:172.21.15.77:3300/0 conn(0x7fa2fc068b60 0x7fa2fc0aa140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.959+0000 7fa2fb7fe700 1 -- 172.21.15.27:0/3875034783 >> v1:172.21.15.77:6789/0 conn(0x7fa2fc1596c0 legacy=0x7fa2fc15d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:31.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.959+0000 7fa2fb7fe700 1 -- 172.21.15.27:0/3875034783 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2fc15e210 con 0x7fa2fc0ca670 2024-01-11T09:46:31.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.959+0000 7fa300b9c700 1 --2- 172.21.15.27:0/3875034783 >> v2:172.21.15.77:3300/0 conn(0x7fa2fc068b60 0x7fa2fc0aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:31.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.960+0000 7fa2fb7fe700 1 --2- 172.21.15.27:0/3875034783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2fc0ca670 0x7fa2fc068620 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc069a50 tx=0x7fa2f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4929246933b11e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.960+0000 7fa2faffd700 1 -- 172.21.15.27:0/3875034783 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f000b860 con 0x7fa2fc0ca670 2024-01-11T09:46:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.960+0000 7fa2faffd700 1 -- 172.21.15.27:0/3875034783 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f000b9c0 con 0x7fa2fc0ca670 2024-01-11T09:46:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.960+0000 7fa2faffd700 1 -- 172.21.15.27:0/3875034783 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f000bd60 con 0x7fa2fc0ca670 2024-01-11T09:46:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.960+0000 7fa3025ff700 1 -- 172.21.15.27:0/3875034783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2fc0ca670 msgr2=0x7fa2fc068620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.960+0000 7fa3025ff700 1 --2- 172.21.15.27:0/3875034783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2fc0ca670 0x7fa2fc068620 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc069a50 tx=0x7fa2f0009250 comp rx=0 tx=0).stop 2024-01-11T09:46:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.961+0000 7fa3025ff700 1 -- 172.21.15.27:0/3875034783 shutdown_connections 2024-01-11T09:46:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.961+0000 7fa3025ff700 1 --2- 172.21.15.27:0/3875034783 >> v2:172.21.15.77:3300/0 conn(0x7fa2fc068b60 0x7fa2fc0aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.961+0000 7fa3025ff700 1 --2- 172.21.15.27:0/3875034783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2fc0ca670 0x7fa2fc068620 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.961+0000 7fa3025ff700 1 -- 172.21.15.27:0/3875034783 >> 172.21.15.27:0/3875034783 conn(0x7fa2fc0c9f70 msgr2=0x7fa2fc06deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:31.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.961+0000 7fa3025ff700 1 -- 172.21.15.27:0/3875034783 shutdown_connections 2024-01-11T09:46:31.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.961+0000 7fa3025ff700 1 -- 172.21.15.27:0/3875034783 wait complete. 2024-01-11T09:46:31.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.962+0000 7fa3025ff700 1 Processor -- start 2024-01-11T09:46:31.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.962+0000 7fa3025ff700 1 -- start start 2024-01-11T09:46:31.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.963+0000 7fa3025ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2fc068b60 0x7fa2fc16add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.963+0000 7fa3025ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2fc0ca670 0x7fa2fc06a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.963+0000 7fa3025ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa2fc1596c0 0x7fa2fc0c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.963+0000 7fa3025ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc0c3b80 con 0x7fa2fc068b60 2024-01-11T09:46:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.963+0000 7fa3025ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc16b310 con 0x7fa2fc0ca670 2024-01-11T09:46:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.963+0000 7fa3025ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc06b0a0 con 0x7fa2fc1596c0 2024-01-11T09:46:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.963+0000 7fa2fbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2fc068b60 0x7fa2fc16add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.963+0000 7fa2fbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2fc068b60 0x7fa2fc16add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48536/0 (socket says 172.21.15.27:48536) 2024-01-11T09:46:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.963+0000 7fa2fbfff700 1 -- 172.21.15.27:0/3561412233 learned_addr learned my addr 172.21.15.27:0/3561412233 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.964+0000 7fa2fb7fe700 1 --2- 172.21.15.27:0/3561412233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2fc0ca670 0x7fa2fc06a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.964+0000 7fa300b9c700 1 --2- 172.21.15.27:0/3561412233 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa2fc1596c0 0x7fa2fc0c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.964+0000 7fa2fbfff700 1 -- 172.21.15.27:0/3561412233 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa2fc1596c0 msgr2=0x7fa2fc0c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.964+0000 7fa2fbfff700 1 --2- 172.21.15.27:0/3561412233 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa2fc1596c0 0x7fa2fc0c7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.964+0000 7fa2fbfff700 1 -- 172.21.15.27:0/3561412233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2fc0ca670 msgr2=0x7fa2fc06a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.964+0000 7fa2fbfff700 1 --2- 172.21.15.27:0/3561412233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2fc0ca670 0x7fa2fc06a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:31.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.964+0000 7fa2fbfff700 1 -- 172.21.15.27:0/3561412233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f00094f0 con 0x7fa2fc068b60 2024-01-11T09:46:31.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.964+0000 7fa2fbfff700 1 --2- 172.21.15.27:0/3561412233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2fc068b60 0x7fa2fc16add0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00c590 tx=0x7fa2ec00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:31.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.964+0000 7fa2e3fff700 1 -- 172.21.15.27:0/3561412233 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec011590 con 0x7fa2fc068b60 2024-01-11T09:46:31.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.964+0000 7fa2e3fff700 1 -- 172.21.15.27:0/3561412233 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ec011710 con 0x7fa2fc068b60 2024-01-11T09:46:31.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.964+0000 7fa2e3fff700 1 -- 172.21.15.27:0/3561412233 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec011ad0 con 0x7fa2fc068b60 2024-01-11T09:46:31.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.965+0000 7fa3025ff700 1 -- 172.21.15.27:0/3561412233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2fc06b380 con 0x7fa2fc068b60 2024-01-11T09:46:31.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.965+0000 7fa3025ff700 1 -- 172.21.15.27:0/3561412233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2fc06b940 con 0x7fa2fc068b60 2024-01-11T09:46:31.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.966+0000 7fa3025ff700 1 -- 172.21.15.27:0/3561412233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2fc0718d0 con 0x7fa2fc068b60 2024-01-11T09:46:31.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.967+0000 7fa2e3fff700 1 -- 172.21.15.27:0/3561412233 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fa2ec01b3b0 con 0x7fa2fc068b60 2024-01-11T09:46:31.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.967+0000 7fa2e3fff700 1 --2- 172.21.15.27:0/3561412233 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa2e8067170 0x7fa2e8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:31.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.968+0000 7fa2e3fff700 1 -- 172.21.15.27:0/3561412233 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5817+0+0 (secure 0 0 0) 0x7fa2ec083c10 con 0x7fa2fc068b60 2024-01-11T09:46:31.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.968+0000 7fa2fb7fe700 1 --2- 172.21.15.27:0/3561412233 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa2e8067170 0x7fa2e8069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:31.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.969+0000 7fa2fb7fe700 1 --2- 172.21.15.27:0/3561412233 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa2e8067170 0x7fa2e8069620 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc069a50 tx=0x7fa2f0017000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:31.972+0000 7fa2e3fff700 1 -- 172.21.15.27:0/3561412233 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa2ec050b80 con 0x7fa2fc068b60 2024-01-11T09:46:32.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.219+0000 7fa3025ff700 1 -- 172.21.15.27:0/3561412233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2fc15da10 con 0x7fa2fc068b60 2024-01-11T09:46:32.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.220+0000 7fa2e3fff700 1 -- 172.21.15.27:0/3561412233 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+4609 (secure 0 0 0) 0x7fa2fc15da10 con 0x7fa2fc068b60 2024-01-11T09:46:32.211 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:32.211 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":127,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1568813392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1568813392},{"type":"v1","addr":"172.21.15.77:6834","nonce":1568813392}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122},{"gid":10969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1352491019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1352491019},{"type":"v1","addr":"172.21.15.77:6835","nonce":1352491019}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122},{"gid":11331,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3477661846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3477661846},{"type":"v1","addr":"172.21.15.27:6834","nonce":3477661846}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:46:25.726042+0000","modified":"2024-01-11T09:46:29.257798+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11340":{"gid":11340,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/321681407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":321681407},{"type":"v1","addr":"172.21.15.27:6835","nonce":321681407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2024-01-11T09:46:32.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.222+0000 7fa3025ff700 1 -- 172.21.15.27:0/3561412233 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa2e8067170 msgr2=0x7fa2e8069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:32.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.222+0000 7fa3025ff700 1 --2- 172.21.15.27:0/3561412233 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa2e8067170 0x7fa2e8069620 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc069a50 tx=0x7fa2f0017000 comp rx=0 tx=0).stop 2024-01-11T09:46:32.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.222+0000 7fa3025ff700 1 -- 172.21.15.27:0/3561412233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2fc068b60 msgr2=0x7fa2fc16add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:32.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.222+0000 7fa3025ff700 1 --2- 172.21.15.27:0/3561412233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2fc068b60 0x7fa2fc16add0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00c590 tx=0x7fa2ec00c8a0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.222+0000 7fa3025ff700 1 -- 172.21.15.27:0/3561412233 shutdown_connections 2024-01-11T09:46:32.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.222+0000 7fa3025ff700 1 --2- 172.21.15.27:0/3561412233 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa2fc1596c0 0x7fa2fc0c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.222+0000 7fa3025ff700 1 --2- 172.21.15.27:0/3561412233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2fc068b60 0x7fa2fc16add0 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.222+0000 7fa3025ff700 1 --2- 172.21.15.27:0/3561412233 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa2e8067170 0x7fa2e8069620 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.222+0000 7fa3025ff700 1 --2- 172.21.15.27:0/3561412233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2fc0ca670 0x7fa2fc06a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.222+0000 7fa3025ff700 1 -- 172.21.15.27:0/3561412233 >> 172.21.15.27:0/3561412233 conn(0x7fa2fc0c9f70 msgr2=0x7fa2fc0c2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:32.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.222+0000 7fa3025ff700 1 -- 172.21.15.27:0/3561412233 shutdown_connections 2024-01-11T09:46:32.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.222+0000 7fa3025ff700 1 -- 172.21.15.27:0/3561412233 wait complete. 2024-01-11T09:46:32.215 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 127 2024-01-11T09:46:32.634 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 127, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-11T09:46:25.726042+0000', 'modified': '2024-01-11T09:46:29.257798+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11340}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11340': {'gid': 11340, 'name': 'c', 'rank': 0, 'incarnation': 123, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6835/321681407', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 321681407}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 321681407}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-11T09:46:32.634 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-11T09:46:32.634 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-11T09:46:32.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.844+0000 7f66eedff700 1 Processor -- start 2024-01-11T09:46:32.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.844+0000 7f66eedff700 1 -- start start 2024-01-11T09:46:32.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.845+0000 7f66eedff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f66e80b0a40 0x7f66e815da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.846+0000 7f66eedff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66e815df40 0x7f66e8068720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.846+0000 7f66eedff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66e815d290 con 0x7f66e806bf10 2024-01-11T09:46:32.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.846+0000 7f66eedff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e815d570 con 0x7f66e80b0a40 2024-01-11T09:46:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.846+0000 7f66eedff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e8154180 con 0x7f66e815df40 2024-01-11T09:46:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.846+0000 7f66ecb9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f66e806bf10 0x7f66e8159c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52432/0 (socket says 172.21.15.27:52432) 2024-01-11T09:46:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.846+0000 7f66ecb9b700 1 -- 172.21.15.27:0/3897168011 learned_addr learned my addr 172.21.15.27:0/3897168011 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.846+0000 7f66e7fff700 1 --2- 172.21.15.27:0/3897168011 >> v2:172.21.15.77:3300/0 conn(0x7f66e80b0a40 0x7f66e815da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.846+0000 7f66ed39c700 1 --2- 172.21.15.27:0/3897168011 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66e815df40 0x7f66e8068720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.846+0000 7f66e77fe700 1 -- 172.21.15.27:0/3897168011 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320532904 0 0) 0x7f66e815d290 con 0x7f66e806bf10 2024-01-11T09:46:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.846+0000 7f66e77fe700 1 -- 172.21.15.27:0/3897168011 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66d8003570 con 0x7f66e806bf10 2024-01-11T09:46:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.846+0000 7f66e7fff700 1 -- 172.21.15.27:0/3897168011 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66e815df40 msgr2=0x7f66e8068720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.846+0000 7f66e7fff700 1 --2- 172.21.15.27:0/3897168011 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66e815df40 0x7f66e8068720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.846+0000 7f66e7fff700 1 -- 172.21.15.27:0/3897168011 >> v1:172.21.15.27:6789/0 conn(0x7f66e806bf10 legacy=0x7f66e8159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.846+0000 7f66e7fff700 1 -- 172.21.15.27:0/3897168011 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e815a510 con 0x7f66e80b0a40 2024-01-11T09:46:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.846+0000 7f66ed39c700 1 --2- 172.21.15.27:0/3897168011 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66e815df40 0x7f66e8068720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.847+0000 7f66e7fff700 1 --2- 172.21.15.27:0/3897168011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66e80b0a40 0x7f66e815da00 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f66d40098e0 tx=0x7f66d40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=945a1276f97e9f1d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:32.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.847+0000 7f66e77fe700 1 -- 172.21.15.27:0/3897168011 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d4011630 con 0x7f66e80b0a40 2024-01-11T09:46:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.847+0000 7f66e77fe700 1 -- 172.21.15.27:0/3897168011 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d4011790 con 0x7f66e80b0a40 2024-01-11T09:46:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.847+0000 7f66e77fe700 1 -- 172.21.15.27:0/3897168011 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d4011b30 con 0x7f66e80b0a40 2024-01-11T09:46:32.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.848+0000 7f66eedff700 1 -- 172.21.15.27:0/3897168011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66e80b0a40 msgr2=0x7f66e815da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:32.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.848+0000 7f66eedff700 1 --2- 172.21.15.27:0/3897168011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66e80b0a40 0x7f66e815da00 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f66d40098e0 tx=0x7f66d40091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.848+0000 7f66eedff700 1 -- 172.21.15.27:0/3897168011 shutdown_connections 2024-01-11T09:46:32.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.848+0000 7f66eedff700 1 --2- 172.21.15.27:0/3897168011 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66e815df40 0x7f66e8068720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.848+0000 7f66eedff700 1 --2- 172.21.15.27:0/3897168011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66e80b0a40 0x7f66e815da00 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.848+0000 7f66eedff700 1 -- 172.21.15.27:0/3897168011 >> 172.21.15.27:0/3897168011 conn(0x7f66e80ce3c0 msgr2=0x7f66e80ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:32.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.848+0000 7f66eedff700 1 -- 172.21.15.27:0/3897168011 shutdown_connections 2024-01-11T09:46:32.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.848+0000 7f66eedff700 1 -- 172.21.15.27:0/3897168011 wait complete. 2024-01-11T09:46:32.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.849+0000 7f66eedff700 1 Processor -- start 2024-01-11T09:46:32.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.850+0000 7f66eedff700 1 -- start start 2024-01-11T09:46:32.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.851+0000 7f66eedff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66e806bf10 0x7f66e8155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.851+0000 7f66eedff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66e80b0a40 0x7f66e815c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.851+0000 7f66eedff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66e815df40 0x7f66e8151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.851+0000 7f66eedff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e8152d60 con 0x7f66e806bf10 2024-01-11T09:46:32.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.851+0000 7f66eedff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e8155660 con 0x7f66e815df40 2024-01-11T09:46:32.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.851+0000 7f66eedff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e815c960 con 0x7f66e80b0a40 2024-01-11T09:46:32.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.851+0000 7f66ecb9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66e806bf10 0x7f66e8155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.851+0000 7f66ecb9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66e806bf10 0x7f66e8155af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48550/0 (socket says 172.21.15.27:48550) 2024-01-11T09:46:32.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.851+0000 7f66ecb9b700 1 -- 172.21.15.27:0/3339313926 learned_addr learned my addr 172.21.15.27:0/3339313926 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:32.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.851+0000 7f66e7fff700 1 --2- 172.21.15.27:0/3339313926 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66e80b0a40 0x7f66e815c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.851+0000 7f66ed39c700 1 --2- 172.21.15.27:0/3339313926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66e815df40 0x7f66e8151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.851+0000 7f66ecb9b700 1 -- 172.21.15.27:0/3339313926 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66e80b0a40 msgr2=0x7f66e815c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:32.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.851+0000 7f66ecb9b700 1 --2- 172.21.15.27:0/3339313926 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66e80b0a40 0x7f66e815c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.851+0000 7f66ecb9b700 1 -- 172.21.15.27:0/3339313926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66e815df40 msgr2=0x7f66e8151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:32.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.851+0000 7f66ecb9b700 1 --2- 172.21.15.27:0/3339313926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66e815df40 0x7f66e8151c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.851+0000 7f66ecb9b700 1 -- 172.21.15.27:0/3339313926 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e0009a20 con 0x7f66e806bf10 2024-01-11T09:46:32.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.851+0000 7f66ed39c700 1 --2- 172.21.15.27:0/3339313926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66e815df40 0x7f66e8151c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:32.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.852+0000 7f66ecb9b700 1 --2- 172.21.15.27:0/3339313926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66e806bf10 0x7f66e8155af0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00c590 tx=0x7f66dc007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:32.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.852+0000 7f66e4ff9700 1 -- 172.21.15.27:0/3339313926 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66dc005b30 con 0x7f66e806bf10 2024-01-11T09:46:32.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.852+0000 7f66e4ff9700 1 -- 172.21.15.27:0/3339313926 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66dc005cb0 con 0x7f66e806bf10 2024-01-11T09:46:32.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.852+0000 7f66e4ff9700 1 -- 172.21.15.27:0/3339313926 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66dc007320 con 0x7f66e806bf10 2024-01-11T09:46:32.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.852+0000 7f66eedff700 1 -- 172.21.15.27:0/3339313926 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d4009420 con 0x7f66e806bf10 2024-01-11T09:46:32.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.852+0000 7f66eedff700 1 -- 172.21.15.27:0/3339313926 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e81552f0 con 0x7f66e806bf10 2024-01-11T09:46:32.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.854+0000 7f66e4ff9700 1 -- 172.21.15.27:0/3339313926 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f66dc01b3b0 con 0x7f66e806bf10 2024-01-11T09:46:32.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.855+0000 7f66e4ff9700 1 --2- 172.21.15.27:0/3339313926 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f66cc067170 0x7f66cc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.855+0000 7f66e4ff9700 1 -- 172.21.15.27:0/3339313926 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f66dc083f90 con 0x7f66e806bf10 2024-01-11T09:46:32.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.855+0000 7f66e7fff700 1 --2- 172.21.15.27:0/3339313926 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f66cc067170 0x7f66cc069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.856+0000 7f66e7fff700 1 --2- 172.21.15.27:0/3339313926 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f66cc067170 0x7f66cc069620 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f66e81574a0 tx=0x7f66d4017000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:32.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.857+0000 7f66fd55cb80 1 Processor -- start 2024-01-11T09:46:32.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.857+0000 7f66fd55cb80 1 -- start start 2024-01-11T09:46:32.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.858+0000 7f66fd55cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3039fb20 0x563b30380df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.858+0000 7f66fd55cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3037b3a0 0x563b303a7860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.858+0000 7f66fd55cb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fef0 0x563b303a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.858+0000 7f66fd55cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b30298310 con 0x563b3039fb20 2024-01-11T09:46:32.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.858+0000 7f66fd55cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b302d5130 con 0x563b3037b3a0 2024-01-11T09:46:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.858+0000 7f66fd55cb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x563b302d2af0 con 0x563b3039fef0 2024-01-11T09:46:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.858+0000 7f66ecb9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3039fb20 0x563b30380df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.858+0000 7f66ecb9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3039fb20 0x563b30380df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48552/0 (socket says 172.21.15.27:48552) 2024-01-11T09:46:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.858+0000 7f66ecb9b700 1 -- 172.21.15.27:0/1659933297 learned_addr learned my addr 172.21.15.27:0/1659933297 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.858+0000 7f66e7fff700 1 --2- 172.21.15.27:0/1659933297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3037b3a0 0x563b303a7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.858+0000 7f66ed39c700 1 --2- 172.21.15.27:0/1659933297 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fef0 0x563b303a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.858+0000 7f66ecb9b700 1 -- 172.21.15.27:0/1659933297 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fef0 msgr2=0x563b303a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.858+0000 7f66ecb9b700 1 --2- 172.21.15.27:0/1659933297 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fef0 0x563b303a7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.858+0000 7f66ecb9b700 1 -- 172.21.15.27:0/1659933297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3037b3a0 msgr2=0x563b303a7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.858+0000 7f66ecb9b700 1 --2- 172.21.15.27:0/1659933297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3037b3a0 0x563b303a7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.858+0000 7f66ecb9b700 1 -- 172.21.15.27:0/1659933297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b301c10a0 con 0x563b3039fb20 2024-01-11T09:46:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.858+0000 7f66ed39c700 1 --2- 172.21.15.27:0/1659933297 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fef0 0x563b303a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.859+0000 7f66ecb9b700 1 --2- 172.21.15.27:0/1659933297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3039fb20 0x563b30380df0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00cce0 tx=0x7f66dc0548c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddbc53dd775e0a76 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.859+0000 7f66d1ffb700 1 -- 172.21.15.27:0/1659933297 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66dc0551f0 con 0x563b3039fb20 2024-01-11T09:46:32.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.859+0000 7f66d1ffb700 1 -- 172.21.15.27:0/1659933297 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66dc054af0 con 0x563b3039fb20 2024-01-11T09:46:32.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.859+0000 7f66d1ffb700 1 -- 172.21.15.27:0/1659933297 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66dc057d40 con 0x563b3039fb20 2024-01-11T09:46:32.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.860+0000 7f66fd55cb80 1 -- 172.21.15.27:0/1659933297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3039fb20 msgr2=0x563b30380df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:32.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.860+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/1659933297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3039fb20 0x563b30380df0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00cce0 tx=0x7f66dc0548c0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.860+0000 7f66fd55cb80 1 -- 172.21.15.27:0/1659933297 shutdown_connections 2024-01-11T09:46:32.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.860+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/1659933297 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fef0 0x563b303a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.860+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/1659933297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3039fb20 0x563b30380df0 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.860+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/1659933297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3037b3a0 0x563b303a7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.860+0000 7f66fd55cb80 1 -- 172.21.15.27:0/1659933297 >> 172.21.15.27:0/1659933297 conn(0x563b3037d300 msgr2=0x563b3037fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:32.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.860+0000 7f66fd55cb80 1 -- 172.21.15.27:0/1659933297 shutdown_connections 2024-01-11T09:46:32.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.860+0000 7f66fd55cb80 1 -- 172.21.15.27:0/1659933297 wait complete. 2024-01-11T09:46:32.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.860+0000 7f66fd55cb80 1 -- start start 2024-01-11T09:46:32.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.861+0000 7f66fd55cb80 1 Processor -- start 2024-01-11T09:46:32.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.862+0000 7f66fd55cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3037b3a0 0x563b303aa8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.862+0000 7f66fd55cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3039fb20 0x563b303a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.862+0000 7f66fd55cb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fef0 0x563b303ad280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.862+0000 7f66fd55cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b302ce8e0 con 0x563b3037b3a0 2024-01-11T09:46:32.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.862+0000 7f66fd55cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b3027a980 con 0x563b3039fb20 2024-01-11T09:46:32.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.862+0000 7f66fd55cb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x563b3027a620 con 0x563b3039fef0 2024-01-11T09:46:32.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.862+0000 7f66fd55cb80 10 client.0 _ll_register_callbacks cb 0x563b300dda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T09:46:32.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.862+0000 7f66ecb9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3037b3a0 0x563b303aa8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.862+0000 7f66ecb9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3037b3a0 0x563b303aa8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48562/0 (socket says 172.21.15.27:48562) 2024-01-11T09:46:32.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.862+0000 7f66ecb9b700 1 -- 172.21.15.27:0/265158988 learned_addr learned my addr 172.21.15.27:0/265158988 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.862+0000 7f66e7fff700 1 --2- 172.21.15.27:0/265158988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3039fb20 0x563b303a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.862+0000 7f66ed39c700 1 --2- 172.21.15.27:0/265158988 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fef0 0x563b303ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.862+0000 7f66ecb9b700 1 -- 172.21.15.27:0/265158988 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fef0 msgr2=0x563b303ad280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.862+0000 7f66ecb9b700 1 --2- 172.21.15.27:0/265158988 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fef0 0x563b303ad280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.862+0000 7f66ecb9b700 1 -- 172.21.15.27:0/265158988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3039fb20 msgr2=0x563b303a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.862+0000 7f66ecb9b700 1 --2- 172.21.15.27:0/265158988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3039fb20 0x563b303a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.862+0000 7f66ecb9b700 1 -- 172.21.15.27:0/265158988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e81552f0 con 0x563b3037b3a0 2024-01-11T09:46:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.863+0000 7f66ecb9b700 1 --2- 172.21.15.27:0/265158988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3037b3a0 0x563b303aa8e0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00cce0 tx=0x7f66dc06c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c86c7c1a72e0cd3c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.863+0000 7f66c6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T09:46:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.863+0000 7f66c6ffd700 1 -- 172.21.15.27:0/265158988 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66dc0551f0 con 0x563b3037b3a0 2024-01-11T09:46:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.863+0000 7f66c6ffd700 1 -- 172.21.15.27:0/265158988 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(235..235 src has 1..235) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f66dc05ba90 con 0x563b3037b3a0 2024-01-11T09:46:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.863+0000 7f66fd55cb80 1 -- 172.21.15.27:0/265158988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563b30373560 con 0x563b3037b3a0 2024-01-11T09:46:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.863+0000 7f66c6ffd700 1 -- 172.21.15.27:0/265158988 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=127) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66dc057d40 con 0x563b3037b3a0 2024-01-11T09:46:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.863+0000 7f66fd55cb80 10 client.11466 fetch_fsmap learned FSMap version 127 2024-01-11T09:46:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.863+0000 7f66fd55cb80 1 -- 172.21.15.27:0/265158988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=127,monmap=2+}) v3 -- 0x563b3029d3c0 con 0x563b3037b3a0 2024-01-11T09:46:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.863+0000 7f66c6ffd700 1 -- 172.21.15.27:0/265158988 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563b30373560 con 0x563b3037b3a0 2024-01-11T09:46:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.863+0000 7f66c6ffd700 1 -- 172.21.15.27:0/265158988 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 127) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f66dc0822e0 con 0x563b3037b3a0 2024-01-11T09:46:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.864+0000 7f66fd55cb80 10 client.11466 fetch_fsmap finished waiting for FSMap version 127 2024-01-11T09:46:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.864+0000 7f66fd55cb80 10 client.11466 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-11T09:46:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.864+0000 7f66fd55cb80 20 client.11466 populate_metadata read hostname 'smithi027' 2024-01-11T09:46:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.864+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/265158988 >> [v2:172.21.15.27:6833/321681407,v1:172.21.15.27:6835/321681407] conn(0x563b303d85b0 0x563b303d8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.864+0000 7f66fd55cb80 4 client.11466 mds_command: new command op to 11340 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-11T09:46:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.864+0000 7f66fd55cb80 1 -- 172.21.15.27:0/265158988 --> [v2:172.21.15.27:6833/321681407,v1:172.21.15.27:6835/321681407] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563b303500e0 con 0x563b303d85b0 2024-01-11T09:46:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.864+0000 7f66e7fff700 1 --2- 172.21.15.27:0/265158988 >> [v2:172.21.15.27:6833/321681407,v1:172.21.15.27:6835/321681407] conn(0x563b303d85b0 0x563b303d8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.864+0000 7f66e7fff700 1 --2- 172.21.15.27:0/265158988 >> [v2:172.21.15.27:6833/321681407,v1:172.21.15.27:6835/321681407] conn(0x563b303d85b0 0x563b303d8980 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af4f1d2aa8cdeecc server_cookie=d15e0c8aa59de6c8 in_seq=0 out_seq=0 2024-01-11T09:46:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.864+0000 7f66c6ffd700 10 client.11466 ms_handle_connect on v2:172.21.15.27:6833/321681407 2024-01-11T09:46:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.865+0000 7f66c6ffd700 1 -- 172.21.15.27:0/265158988 <== mds.0 v2:172.21.15.27:6833/321681407 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x563b303500e0 con 0x563b303d85b0 2024-01-11T09:46:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.865+0000 7f66c6ffd700 10 client.11466 handle_command_reply: tid=0 2024-01-11T09:46:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.865+0000 7f66fd55cb80 1 client.11466 shutdown 2024-01-11T09:46:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.865+0000 7f66fd55cb80 1 -- 172.21.15.27:0/265158988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3037b3a0 msgr2=0x563b303aa8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.865+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/265158988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3037b3a0 0x563b303aa8e0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00cce0 tx=0x7f66dc06c000 comp rx=0 tx=0).stop 2024-01-11T09:46:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.866+0000 7f66fd55cb80 1 -- 172.21.15.27:0/265158988 shutdown_connections 2024-01-11T09:46:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.866+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/265158988 >> [v2:172.21.15.27:6833/321681407,v1:172.21.15.27:6835/321681407] conn(0x563b303d85b0 0x563b303d8980 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.866+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/265158988 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fef0 0x563b303ad280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.866+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/265158988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3037b3a0 0x563b303aa8e0 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.866+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/265158988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3039fb20 0x563b303a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.866+0000 7f66c6ffd700 0 client.11466 ms_handle_reset on v2:172.21.15.27:6833/321681407 2024-01-11T09:46:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.866+0000 7f66fd55cb80 1 -- 172.21.15.27:0/265158988 >> 172.21.15.27:0/265158988 conn(0x563b3037d300 msgr2=0x563b303a77d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.866+0000 7f66fd55cb80 1 -- 172.21.15.27:0/265158988 shutdown_connections 2024-01-11T09:46:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.866+0000 7f66fd55cb80 1 -- 172.21.15.27:0/265158988 wait complete. 2024-01-11T09:46:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.866+0000 7f66fd55cb80 20 client.11466 trim_cache size 0 max 16384 2024-01-11T09:46:32.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.883+0000 7f66fd55cb80 1 Processor -- start 2024-01-11T09:46:32.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.883+0000 7f66fd55cb80 1 -- start start 2024-01-11T09:46:32.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.884+0000 7f66fd55cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3037b3a0 0x563b303d4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.884+0000 7f66fd55cb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fb20 0x563b303d4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.884+0000 7f66fd55cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3039fef0 0x563b303a7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.884+0000 7f66fd55cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b301661e0 con 0x563b3039fef0 2024-01-11T09:46:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.884+0000 7f66fd55cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b303d8140 con 0x563b3037b3a0 2024-01-11T09:46:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.884+0000 7f66fd55cb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x563b303a29e0 con 0x563b3039fb20 2024-01-11T09:46:32.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.885+0000 7f66ed39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3039fef0 0x563b303a7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.885+0000 7f66ed39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3039fef0 0x563b303a7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48564/0 (socket says 172.21.15.27:48564) 2024-01-11T09:46:32.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.885+0000 7f66ed39c700 1 -- 172.21.15.27:0/2542822850 learned_addr learned my addr 172.21.15.27:0/2542822850 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:32.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.885+0000 7f66ecb9b700 1 --2- 172.21.15.27:0/2542822850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3037b3a0 0x563b303d4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.885+0000 7f66e7fff700 1 --2- 172.21.15.27:0/2542822850 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fb20 0x563b303d4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.885+0000 7f66ed39c700 1 -- 172.21.15.27:0/2542822850 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fb20 msgr2=0x563b303d4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:32.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.885+0000 7f66ed39c700 1 --2- 172.21.15.27:0/2542822850 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fb20 0x563b303d4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.885+0000 7f66ed39c700 1 -- 172.21.15.27:0/2542822850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3037b3a0 msgr2=0x563b303d4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:32.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.885+0000 7f66ed39c700 1 --2- 172.21.15.27:0/2542822850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3037b3a0 0x563b303d4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.885+0000 7f66ed39c700 1 -- 172.21.15.27:0/2542822850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b301351d0 con 0x563b3039fef0 2024-01-11T09:46:32.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.885+0000 7f66ed39c700 1 --2- 172.21.15.27:0/2542822850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3039fef0 0x563b303a7340 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f66e0007970 tx=0x7f66e000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4efd1985593a08b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:32.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.885+0000 7f66d27fc700 1 -- 172.21.15.27:0/2542822850 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e000e040 con 0x563b3039fef0 2024-01-11T09:46:32.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.886+0000 7f66d27fc700 1 -- 172.21.15.27:0/2542822850 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e000f040 con 0x563b3039fef0 2024-01-11T09:46:32.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.886+0000 7f66d27fc700 1 -- 172.21.15.27:0/2542822850 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e00168f0 con 0x563b3039fef0 2024-01-11T09:46:32.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.886+0000 7f66fd55cb80 1 -- 172.21.15.27:0/2542822850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3039fef0 msgr2=0x563b303a7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:32.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.886+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/2542822850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3039fef0 0x563b303a7340 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f66e0007970 tx=0x7f66e000b040 comp rx=0 tx=0).stop 2024-01-11T09:46:32.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.886+0000 7f66fd55cb80 1 -- 172.21.15.27:0/2542822850 shutdown_connections 2024-01-11T09:46:32.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.886+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/2542822850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3039fef0 0x563b303a7340 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.886+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/2542822850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3037b3a0 0x563b303d4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.886+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/2542822850 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fb20 0x563b303d4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.886+0000 7f66fd55cb80 1 -- 172.21.15.27:0/2542822850 >> 172.21.15.27:0/2542822850 conn(0x563b3037d300 msgr2=0x563b303adf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:32.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.886+0000 7f66fd55cb80 1 -- 172.21.15.27:0/2542822850 shutdown_connections 2024-01-11T09:46:32.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.886+0000 7f66fd55cb80 1 -- 172.21.15.27:0/2542822850 wait complete. 2024-01-11T09:46:32.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.887+0000 7f66fd55cb80 1 -- start start 2024-01-11T09:46:32.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.887+0000 7f66fd55cb80 1 Processor -- start 2024-01-11T09:46:32.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.888+0000 7f66fd55cb80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3037b3a0 0x563b30429950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.888+0000 7f66fd55cb80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fb20 0x563b303adf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.888+0000 7f66fd55cb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3039fef0 0x563b303ae450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.888+0000 7f66fd55cb80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b303ab020 con 0x563b3037b3a0 2024-01-11T09:46:32.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.888+0000 7f66fd55cb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b30380c70 con 0x563b3039fef0 2024-01-11T09:46:32.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.888+0000 7f66fd55cb80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x563b303a7760 con 0x563b3039fb20 2024-01-11T09:46:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.888+0000 7f66fd55cb80 10 client.0 _ll_register_callbacks cb 0x563b300dda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T09:46:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.888+0000 7f66ecb9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3037b3a0 0x563b30429950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.888+0000 7f66ecb9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3037b3a0 0x563b30429950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48566/0 (socket says 172.21.15.27:48566) 2024-01-11T09:46:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.888+0000 7f66ecb9b700 1 -- 172.21.15.27:0/3275047464 learned_addr learned my addr 172.21.15.27:0/3275047464 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.888+0000 7f66ed39c700 1 --2- 172.21.15.27:0/3275047464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3039fef0 0x563b303ae450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.889+0000 7f66e7fff700 1 --2- 172.21.15.27:0/3275047464 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fb20 0x563b303adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.889+0000 7f66ecb9b700 1 -- 172.21.15.27:0/3275047464 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fb20 msgr2=0x563b303adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.889+0000 7f66ecb9b700 1 --2- 172.21.15.27:0/3275047464 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fb20 0x563b303adf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.889+0000 7f66ecb9b700 1 -- 172.21.15.27:0/3275047464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3039fef0 msgr2=0x563b303ae450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.889+0000 7f66ecb9b700 1 --2- 172.21.15.27:0/3275047464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3039fef0 0x563b303ae450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.889+0000 7f66ecb9b700 1 -- 172.21.15.27:0/3275047464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e0007670 con 0x563b3037b3a0 2024-01-11T09:46:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.889+0000 7f66ecb9b700 1 --2- 172.21.15.27:0/3275047464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3037b3a0 0x563b30429950 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f66dc0070a0 tx=0x7f66dc07d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3a7016844140d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.889+0000 7f66c6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T09:46:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.889+0000 7f66c6ffd700 1 -- 172.21.15.27:0/3275047464 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66dc00e040 con 0x563b3037b3a0 2024-01-11T09:46:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.889+0000 7f66c6ffd700 1 -- 172.21.15.27:0/3275047464 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(235..235 src has 1..235) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f66dc0551f0 con 0x563b3037b3a0 2024-01-11T09:46:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.889+0000 7f66fd55cb80 1 -- 172.21.15.27:0/3275047464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f66dc054af0 con 0x563b3037b3a0 2024-01-11T09:46:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.889+0000 7f66c6ffd700 1 -- 172.21.15.27:0/3275047464 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=127) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66dc067120 con 0x563b3037b3a0 2024-01-11T09:46:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.889+0000 7f66fd55cb80 10 client.11472 fetch_fsmap learned FSMap version 127 2024-01-11T09:46:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.889+0000 7f66fd55cb80 1 -- 172.21.15.27:0/3275047464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=127,monmap=2+}) v3 -- 0x563b3010ed60 con 0x563b3037b3a0 2024-01-11T09:46:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.890+0000 7f66c6ffd700 1 -- 172.21.15.27:0/3275047464 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66dc074080 con 0x563b3037b3a0 2024-01-11T09:46:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.890+0000 7f66c6ffd700 1 -- 172.21.15.27:0/3275047464 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 127) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f66dc0508e0 con 0x563b3037b3a0 2024-01-11T09:46:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.890+0000 7f66fd55cb80 10 client.11472 fetch_fsmap finished waiting for FSMap version 127 2024-01-11T09:46:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.890+0000 7f66fd55cb80 10 client.11472 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-11T09:46:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.890+0000 7f66fd55cb80 20 client.11472 populate_metadata read hostname 'smithi027' 2024-01-11T09:46:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.890+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/3275047464 >> [v2:172.21.15.27:6833/321681407,v1:172.21.15.27:6835/321681407] conn(0x563b303d48a0 0x563b3037fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.890+0000 7f66fd55cb80 4 client.11472 mds_command: new command op to 11340 tid=0[{"prefix": "status"}] 2024-01-11T09:46:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.890+0000 7f66fd55cb80 1 -- 172.21.15.27:0/3275047464 --> [v2:172.21.15.27:6833/321681407,v1:172.21.15.27:6835/321681407] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563b302718d0 con 0x563b303d48a0 2024-01-11T09:46:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.890+0000 7f66e7fff700 1 --2- 172.21.15.27:0/3275047464 >> [v2:172.21.15.27:6833/321681407,v1:172.21.15.27:6835/321681407] conn(0x563b303d48a0 0x563b3037fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.890+0000 7f66e7fff700 1 --2- 172.21.15.27:0/3275047464 >> [v2:172.21.15.27:6833/321681407,v1:172.21.15.27:6835/321681407] conn(0x563b303d48a0 0x563b3037fdc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f149e52c8fa5ce00 server_cookie=1bfbe359fc29f69 in_seq=0 out_seq=0 2024-01-11T09:46:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.890+0000 7f66c6ffd700 10 client.11472 ms_handle_connect on v2:172.21.15.27:6833/321681407 2024-01-11T09:46:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.891+0000 7f66c6ffd700 1 -- 172.21.15.27:0/3275047464 <== mds.0 v2:172.21.15.27:6833/321681407 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x563b302718d0 con 0x563b303d48a0 2024-01-11T09:46:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.891+0000 7f66c6ffd700 10 client.11472 handle_command_reply: tid=0 2024-01-11T09:46:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.891+0000 7f66fd55cb80 1 client.11472 shutdown 2024-01-11T09:46:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.891+0000 7f66fd55cb80 1 -- 172.21.15.27:0/3275047464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3037b3a0 msgr2=0x563b30429950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.891+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/3275047464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3037b3a0 0x563b30429950 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f66dc0070a0 tx=0x7f66dc07d000 comp rx=0 tx=0).stop 2024-01-11T09:46:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.891+0000 7f66fd55cb80 1 -- 172.21.15.27:0/3275047464 shutdown_connections 2024-01-11T09:46:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.891+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/3275047464 >> [v2:172.21.15.27:6833/321681407,v1:172.21.15.27:6835/321681407] conn(0x563b303d48a0 0x563b3037fdc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.891+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/3275047464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b3039fef0 0x563b303ae450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.891+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/3275047464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b3037b3a0 0x563b30429950 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.891+0000 7f66c6ffd700 0 client.11472 ms_handle_reset on v2:172.21.15.27:6833/321681407 2024-01-11T09:46:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.891+0000 7f66fd55cb80 1 --2- 172.21.15.27:0/3275047464 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x563b3039fb20 0x563b303adf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.891+0000 7f66fd55cb80 1 -- 172.21.15.27:0/3275047464 >> 172.21.15.27:0/3275047464 conn(0x563b3037d300 msgr2=0x563b303de9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.891+0000 7f66fd55cb80 1 -- 172.21.15.27:0/3275047464 shutdown_connections 2024-01-11T09:46:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.891+0000 7f66fd55cb80 1 -- 172.21.15.27:0/3275047464 wait complete. 2024-01-11T09:46:32.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.891+0000 7f66fd55cb80 20 client.11472 trim_cache size 0 max 16384 2024-01-11T09:46:32.889 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:46:32.889 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "8fddb4a7-9d66-40f1-892d-6fe86c728ee4", 2024-01-11T09:46:32.889 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-11T09:46:32.889 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 11340, 2024-01-11T09:46:32.889 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-11T09:46:32.890 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-11T09:46:32.890 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-11T09:46:32.890 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 6.6574665719999997, 2024-01-11T09:46:32.890 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 127, 2024-01-11T09:46:32.890 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 235, 2024-01-11T09:46:32.890 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 235, 2024-01-11T09:46:32.890 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 12.294934831000001 2024-01-11T09:46:32.890 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:46:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.892+0000 7f66eedff700 1 -- 172.21.15.27:0/3339313926 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f66cc067170 msgr2=0x7f66cc069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.892+0000 7f66eedff700 1 --2- 172.21.15.27:0/3339313926 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f66cc067170 0x7f66cc069620 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f66e81574a0 tx=0x7f66d4017000 comp rx=0 tx=0).stop 2024-01-11T09:46:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.892+0000 7f66eedff700 1 -- 172.21.15.27:0/3339313926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66e806bf10 msgr2=0x7f66e8155af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.892+0000 7f66eedff700 1 --2- 172.21.15.27:0/3339313926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66e806bf10 0x7f66e8155af0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00c590 tx=0x7f66dc007e00 comp rx=0 tx=0).stop 2024-01-11T09:46:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.892+0000 7f66eedff700 1 -- 172.21.15.27:0/3339313926 shutdown_connections 2024-01-11T09:46:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.892+0000 7f66eedff700 1 --2- 172.21.15.27:0/3339313926 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f66cc067170 0x7f66cc069620 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.892+0000 7f66eedff700 1 --2- 172.21.15.27:0/3339313926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66e815df40 0x7f66e8151c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.892+0000 7f66eedff700 1 --2- 172.21.15.27:0/3339313926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66e806bf10 0x7f66e8155af0 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.892+0000 7f66eedff700 1 --2- 172.21.15.27:0/3339313926 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f66e80b0a40 0x7f66e815c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:32.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.892+0000 7f66eedff700 1 -- 172.21.15.27:0/3339313926 >> 172.21.15.27:0/3339313926 conn(0x7f66e80ce3c0 msgr2=0x7f66e815cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:32.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.892+0000 7f66eedff700 1 -- 172.21.15.27:0/3339313926 shutdown_connections 2024-01-11T09:46:32.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:32.892+0000 7f66eedff700 1 -- 172.21.15.27:0/3339313926 wait complete. 2024-01-11T09:46:32.902 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-11T09:46:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.111+0000 7fafb97ff700 1 Processor -- start 2024-01-11T09:46:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.111+0000 7fafb97ff700 1 -- start start 2024-01-11T09:46:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.112+0000 7fafb97ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fafb4069d40 0x7fafb40b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.112+0000 7fafb97ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fafb40c89f0 0x7fafb40b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.112+0000 7fafb97ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb4157390 con 0x7fafb40c8620 2024-01-11T09:46:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.113+0000 7fafb97ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb40adcd0 con 0x7fafb40c89f0 2024-01-11T09:46:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.113+0000 7fafb97ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb406a200 con 0x7fafb4069d40 2024-01-11T09:46:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.113+0000 7fafb2ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fafb40c8620 0x7fafb40af970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52434/0 (socket says 172.21.15.27:52434) 2024-01-11T09:46:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.113+0000 7fafb2ffd700 1 -- 172.21.15.27:0/1860375546 learned_addr learned my addr 172.21.15.27:0/1860375546 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.113+0000 7fafb37fe700 1 --2- 172.21.15.27:0/1860375546 >> v2:172.21.15.27:3300/0 conn(0x7fafb40c89f0 0x7fafb40b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.113+0000 7fafb27fc700 1 --2- 172.21.15.27:0/1860375546 >> v2:172.21.15.77:3300/0 conn(0x7fafb4069d40 0x7fafb40b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.113+0000 7fafb37fe700 1 -- 172.21.15.27:0/1860375546 >> v2:172.21.15.77:3300/0 conn(0x7fafb4069d40 msgr2=0x7fafb40b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.113+0000 7fafb37fe700 1 --2- 172.21.15.27:0/1860375546 >> v2:172.21.15.77:3300/0 conn(0x7fafb4069d40 0x7fafb40b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.113+0000 7fafb37fe700 1 -- 172.21.15.27:0/1860375546 >> v1:172.21.15.27:6789/0 conn(0x7fafb40c8620 legacy=0x7fafb40af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.113+0000 7fafb37fe700 1 -- 172.21.15.27:0/1860375546 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb4156d40 con 0x7fafb40c89f0 2024-01-11T09:46:33.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.114+0000 7fafb37fe700 1 --2- 172.21.15.27:0/1860375546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb40c89f0 0x7fafb40b0810 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7fafb4157300 tx=0x7fafac009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=718074818ac1ae9a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.114+0000 7fafb1ffb700 1 -- 172.21.15.27:0/1860375546 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac00e070 con 0x7fafb40c89f0 2024-01-11T09:46:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.114+0000 7fafb1ffb700 1 -- 172.21.15.27:0/1860375546 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafac00bb60 con 0x7fafb40c89f0 2024-01-11T09:46:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.114+0000 7fafb1ffb700 1 -- 172.21.15.27:0/1860375546 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac012430 con 0x7fafb40c89f0 2024-01-11T09:46:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.114+0000 7fafb97ff700 1 -- 172.21.15.27:0/1860375546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb40c89f0 msgr2=0x7fafb40b0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.114+0000 7fafb97ff700 1 --2- 172.21.15.27:0/1860375546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb40c89f0 0x7fafb40b0810 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7fafb4157300 tx=0x7fafac009780 comp rx=0 tx=0).stop 2024-01-11T09:46:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.114+0000 7fafb97ff700 1 -- 172.21.15.27:0/1860375546 shutdown_connections 2024-01-11T09:46:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.114+0000 7fafb97ff700 1 --2- 172.21.15.27:0/1860375546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb40c89f0 0x7fafb40b0810 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.114+0000 7fafb97ff700 1 --2- 172.21.15.27:0/1860375546 >> v2:172.21.15.77:3300/0 conn(0x7fafb4069d40 0x7fafb40b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.114+0000 7fafb97ff700 1 -- 172.21.15.27:0/1860375546 >> 172.21.15.27:0/1860375546 conn(0x7fafb40680e0 msgr2=0x7fafb4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.115+0000 7fafb97ff700 1 -- 172.21.15.27:0/1860375546 shutdown_connections 2024-01-11T09:46:33.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.115+0000 7fafb97ff700 1 -- 172.21.15.27:0/1860375546 wait complete. 2024-01-11T09:46:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.116+0000 7fafb97ff700 1 Processor -- start 2024-01-11T09:46:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.116+0000 7fafb97ff700 1 -- start start 2024-01-11T09:46:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.117+0000 7fafb97ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb4069d40 0x7fafb4147aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.117+0000 7fafb97ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb40c8620 0x7fafb40a74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.117+0000 7fafb97ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafb40c89f0 0x7fafb416ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.117+0000 7fafb97ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb4155490 con 0x7fafb4069d40 2024-01-11T09:46:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.117+0000 7fafb97ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb40caa90 con 0x7fafb40c8620 2024-01-11T09:46:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.117+0000 7fafb97ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb40c9e80 con 0x7fafb40c89f0 2024-01-11T09:46:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.117+0000 7fafb2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb4069d40 0x7fafb4147aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.117+0000 7fafb2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb4069d40 0x7fafb4147aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48576/0 (socket says 172.21.15.27:48576) 2024-01-11T09:46:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.117+0000 7fafb2ffd700 1 -- 172.21.15.27:0/179283838 learned_addr learned my addr 172.21.15.27:0/179283838 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.118+0000 7fafb27fc700 1 --2- 172.21.15.27:0/179283838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb40c8620 0x7fafb40a74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:33.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.118+0000 7fafb37fe700 1 --2- 172.21.15.27:0/179283838 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafb40c89f0 0x7fafb416ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:33.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.118+0000 7fafb2ffd700 1 -- 172.21.15.27:0/179283838 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafb40c89f0 msgr2=0x7fafb416ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:33.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.118+0000 7fafb2ffd700 1 --2- 172.21.15.27:0/179283838 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafb40c89f0 0x7fafb416ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:33.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.118+0000 7fafb2ffd700 1 -- 172.21.15.27:0/179283838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb40c8620 msgr2=0x7fafb40a74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:33.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.118+0000 7fafb2ffd700 1 --2- 172.21.15.27:0/179283838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb40c8620 0x7fafb40a74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:33.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.118+0000 7fafb2ffd700 1 -- 172.21.15.27:0/179283838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac009a20 con 0x7fafb4069d40 2024-01-11T09:46:33.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.118+0000 7fafb2ffd700 1 --2- 172.21.15.27:0/179283838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb4069d40 0x7fafb4147aa0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fafa400c7c0 tx=0x7fafa4007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.118+0000 7faf9affd700 1 -- 172.21.15.27:0/179283838 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4005b30 con 0x7fafb4069d40 2024-01-11T09:46:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.118+0000 7faf9affd700 1 -- 172.21.15.27:0/179283838 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa4005cb0 con 0x7fafb4069d40 2024-01-11T09:46:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.118+0000 7faf9affd700 1 -- 172.21.15.27:0/179283838 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4004a80 con 0x7fafb4069d40 2024-01-11T09:46:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.118+0000 7fafb97ff700 1 -- 172.21.15.27:0/179283838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb40ca160 con 0x7fafb4069d40 2024-01-11T09:46:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.118+0000 7fafb97ff700 1 -- 172.21.15.27:0/179283838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb40ca720 con 0x7fafb4069d40 2024-01-11T09:46:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.120+0000 7fafb97ff700 1 -- 172.21.15.27:0/179283838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb40718d0 con 0x7fafb4069d40 2024-01-11T09:46:33.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.121+0000 7faf9affd700 1 -- 172.21.15.27:0/179283838 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fafa40070f0 con 0x7fafb4069d40 2024-01-11T09:46:33.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.122+0000 7faf9affd700 1 --2- 172.21.15.27:0/179283838 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fafa00671a0 0x7fafa0069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:33.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.122+0000 7faf9affd700 1 -- 172.21.15.27:0/179283838 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5817+0+0 (secure 0 0 0) 0x7fafa4083d80 con 0x7fafb4069d40 2024-01-11T09:46:33.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.122+0000 7fafb27fc700 1 --2- 172.21.15.27:0/179283838 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fafa00671a0 0x7fafa0069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:33.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.123+0000 7fafb27fc700 1 --2- 172.21.15.27:0/179283838 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fafa00671a0 0x7fafa0069650 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fafb40cb230 tx=0x7fafa8008040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:33.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.125+0000 7faf9affd700 1 -- 172.21.15.27:0/179283838 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fafa4050d70 con 0x7fafb4069d40 2024-01-11T09:46:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.379+0000 7fafb97ff700 1 -- 172.21.15.27:0/179283838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fafb4157020 con 0x7fafb4069d40 2024-01-11T09:46:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.380+0000 7faf9affd700 1 -- 172.21.15.27:0/179283838 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v127) v1 ==== 53+0+73 (secure 0 0 0) 0x7fafa4018030 con 0x7fafb4069d40 2024-01-11T09:46:33.371 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-11T09:46:33.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.382+0000 7fafb97ff700 1 -- 172.21.15.27:0/179283838 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fafa00671a0 msgr2=0x7fafa0069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:33.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.382+0000 7fafb97ff700 1 --2- 172.21.15.27:0/179283838 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fafa00671a0 0x7fafa0069650 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fafb40cb230 tx=0x7fafa8008040 comp rx=0 tx=0).stop 2024-01-11T09:46:33.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.382+0000 7fafb97ff700 1 -- 172.21.15.27:0/179283838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb4069d40 msgr2=0x7fafb4147aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:33.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.382+0000 7fafb97ff700 1 --2- 172.21.15.27:0/179283838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb4069d40 0x7fafb4147aa0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fafa400c7c0 tx=0x7fafa4007f70 comp rx=0 tx=0).stop 2024-01-11T09:46:33.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.382+0000 7fafb97ff700 1 -- 172.21.15.27:0/179283838 shutdown_connections 2024-01-11T09:46:33.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.382+0000 7fafb97ff700 1 --2- 172.21.15.27:0/179283838 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafb40c89f0 0x7fafb416ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:33.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.382+0000 7fafb97ff700 1 --2- 172.21.15.27:0/179283838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb4069d40 0x7fafb4147aa0 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:33.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.382+0000 7fafb97ff700 1 --2- 172.21.15.27:0/179283838 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fafa00671a0 0x7fafa0069650 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:33.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.382+0000 7fafb97ff700 1 --2- 172.21.15.27:0/179283838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb40c8620 0x7fafb40a74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.382+0000 7fafb97ff700 1 -- 172.21.15.27:0/179283838 >> 172.21.15.27:0/179283838 conn(0x7fafb40680e0 msgr2=0x7fafb40ade10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.382+0000 7fafb97ff700 1 -- 172.21.15.27:0/179283838 shutdown_connections 2024-01-11T09:46:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:33.382+0000 7fafb97ff700 1 -- 172.21.15.27:0/179283838 wait complete. 2024-01-11T09:46:33.392 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-11T09:46:33.392 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-11T09:46:33.392 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-11T09:46:33.392 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-11T09:46:33.392 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-11T09:46:33.392 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-11T09:46:33.392 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-11T09:46:33.392 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-11T09:46:33.393 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T09:46:33.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:46:33.393 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-11T09:46:33.441 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T09:46:33.441 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T09:46:33.441 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T09:46:33.441 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:46:33.441 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-11T09:46:33.441 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:46:33.441 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T09:46:33.441 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-11T09:46:33.442 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-11T09:46:33.442 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T09:46:33.442 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-11T09:46:33.442 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-11T09:46:33.442 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-11T09:46:33.442 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-11T09:46:33.442 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-11T09:46:33.442 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-11T09:46:33.442 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 36728sec preferred_lft 36728sec 2024-01-11T09:46:33.442 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-11T09:46:33.442 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:46:33.442 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T09:46:33.442 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-11T09:46:33.442 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-11T09:46:33.442 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T09:46:33.443 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff 2024-01-11T09:46:33.443 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-11T09:46:33.443 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:46:33.443 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::acd2:b3ff:fe6a:db4b/64 scope link 2024-01-11T09:46:33.443 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:46:33.443 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T09:46:33.443 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:fa:85:1b:3a:bc brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-11T09:46:33.443 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ecfa:85ff:fe1b:3abc/64 scope link 2024-01-11T09:46:33.443 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:46:33.443 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T09:46:33.443 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-11T09:46:33.443 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ec88:faff:fe17:41aa/64 scope link 2024-01-11T09:46:33.443 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:46:33.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:46:33.444 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-11T09:46:33.514 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-11T09:46:33.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:46:33.517 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-11T09:46:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-11T09:46:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-11T09:46:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-11T09:46:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-11T09:46:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-11T09:46:33.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:46:33.569 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T09:46:33.569 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T09:46:33.569 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-11T09:46:33.569 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-11T09:46:33.569 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-11T09:46:33.569 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T09:46:33.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:46:33.745 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T09:46:33.793 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-11T09:46:33.793 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-11T09:46:33.794 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T09:46:33.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:46:33.794 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-11T09:46:33.867 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T09:46:33.867 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-11T09:46:33.914 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-11T09:46:33.915 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T09:46:33.915 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-11T09:46:33.964 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-11T09:46:33.965 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-11T09:46:33.965 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:46:34.068 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:46:34.068 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T09:46:34.068 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T09:46:34.083 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:46:34.083 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T09:46:34.083 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T09:46:34.083 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T09:46:34.083 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T09:46:34.103 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:46:34.103 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T09:46:34.103 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T09:46:34.103 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T09:46:34.104 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T09:46:34.104 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790" 2024-01-11T09:46:34.104 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790:/ 2024-01-11T09:46:34.104 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=8fddb4a7-9d66-40f1-892d-6fe86c728ee4" will pass to kernel 2024-01-11T09:46:34.107 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-11T09:46:34.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:46:34.108 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-11T09:46:34.180 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-11T09:46:34.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.393+0000 7fc05a5ff700 1 Processor -- start 2024-01-11T09:46:34.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.393+0000 7fc05a5ff700 1 -- start start 2024-01-11T09:46:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.394+0000 7fc05a5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc054155ba0 0x7fc05406ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.394+0000 7fc05a5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc054069d70 0x7fc0540c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.394+0000 7fc05a5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0540c6f30 con 0x7fc05406e310 2024-01-11T09:46:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.394+0000 7fc05a5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc05406e860 con 0x7fc054155ba0 2024-01-11T09:46:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.394+0000 7fc05a5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0540c68e0 con 0x7fc054069d70 2024-01-11T09:46:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.394+0000 7fc053fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc054155ba0 0x7fc05406ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.394+0000 7fc053fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc054155ba0 0x7fc05406ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40232/0 (socket says 172.21.15.27:40232) 2024-01-11T09:46:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.394+0000 7fc053fff700 1 -- 172.21.15.27:0/735083629 learned_addr learned my addr 172.21.15.27:0/735083629 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.395+0000 7fc058b9c700 1 --2- 172.21.15.27:0/735083629 >> v2:172.21.15.77:3300/0 conn(0x7fc054069d70 0x7fc0540c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.395+0000 7fc053fff700 1 -- 172.21.15.27:0/735083629 >> v2:172.21.15.77:3300/0 conn(0x7fc054069d70 msgr2=0x7fc0540c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.395+0000 7fc053fff700 1 --2- 172.21.15.27:0/735083629 >> v2:172.21.15.77:3300/0 conn(0x7fc054069d70 0x7fc0540c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.395+0000 7fc053fff700 1 -- 172.21.15.27:0/735083629 >> v1:172.21.15.27:6789/0 conn(0x7fc05406e310 legacy=0x7fc054069660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.395+0000 7fc053fff700 1 -- 172.21.15.27:0/735083629 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0540c6bc0 con 0x7fc054155ba0 2024-01-11T09:46:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.395+0000 7fc053fff700 1 --2- 172.21.15.27:0/735083629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc054155ba0 0x7fc05406ddd0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fc05406ac70 tx=0x7fc04400b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a419c4a49f15ba91 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.395+0000 7fc052ffd700 1 -- 172.21.15.27:0/735083629 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04400db10 con 0x7fc054155ba0 2024-01-11T09:46:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.395+0000 7fc052ffd700 1 -- 172.21.15.27:0/735083629 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc04400dc90 con 0x7fc054155ba0 2024-01-11T09:46:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.395+0000 7fc052ffd700 1 -- 172.21.15.27:0/735083629 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc044005c50 con 0x7fc054155ba0 2024-01-11T09:46:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.396+0000 7fc05a5ff700 1 -- 172.21.15.27:0/735083629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc054155ba0 msgr2=0x7fc05406ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.396+0000 7fc05a5ff700 1 --2- 172.21.15.27:0/735083629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc054155ba0 0x7fc05406ddd0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fc05406ac70 tx=0x7fc04400b4b0 comp rx=0 tx=0).stop 2024-01-11T09:46:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.396+0000 7fc05a5ff700 1 -- 172.21.15.27:0/735083629 shutdown_connections 2024-01-11T09:46:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.396+0000 7fc05a5ff700 1 --2- 172.21.15.27:0/735083629 >> v2:172.21.15.77:3300/0 conn(0x7fc054069d70 0x7fc0540c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.396+0000 7fc05a5ff700 1 --2- 172.21.15.27:0/735083629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc054155ba0 0x7fc05406ddd0 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.396+0000 7fc05a5ff700 1 -- 172.21.15.27:0/735083629 >> 172.21.15.27:0/735083629 conn(0x7fc0540680e0 msgr2=0x7fc0540cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.396+0000 7fc05a5ff700 1 -- 172.21.15.27:0/735083629 shutdown_connections 2024-01-11T09:46:34.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.396+0000 7fc05a5ff700 1 -- 172.21.15.27:0/735083629 wait complete. 2024-01-11T09:46:34.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.397+0000 7fc05a5ff700 1 Processor -- start 2024-01-11T09:46:34.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.398+0000 7fc05a5ff700 1 -- start start 2024-01-11T09:46:34.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.399+0000 7fc05a5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc054069d70 0x7fc054151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:34.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.399+0000 7fc05a5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc05406e310 0x7fc054169e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:34.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.399+0000 7fc05a5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc054155ba0 0x7fc0540c0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:34.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.399+0000 7fc05a5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05416b050 con 0x7fc054155ba0 2024-01-11T09:46:34.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.399+0000 7fc05a5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0540b0860 con 0x7fc05406e310 2024-01-11T09:46:34.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.399+0000 7fc05a5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc054151bb0 con 0x7fc054069d70 2024-01-11T09:46:34.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.399+0000 7fc058b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc054155ba0 0x7fc0540c0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.399+0000 7fc058b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc054155ba0 0x7fc0540c0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40246/0 (socket says 172.21.15.27:40246) 2024-01-11T09:46:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.399+0000 7fc058b9c700 1 -- 172.21.15.27:0/3859473370 learned_addr learned my addr 172.21.15.27:0/3859473370 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.399+0000 7fc053fff700 1 --2- 172.21.15.27:0/3859473370 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc054069d70 0x7fc054151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.399+0000 7fc0537fe700 1 --2- 172.21.15.27:0/3859473370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc05406e310 0x7fc054169e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.399+0000 7fc058b9c700 1 -- 172.21.15.27:0/3859473370 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc054069d70 msgr2=0x7fc054151670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.399+0000 7fc058b9c700 1 --2- 172.21.15.27:0/3859473370 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc054069d70 0x7fc054151670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.399+0000 7fc058b9c700 1 -- 172.21.15.27:0/3859473370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc05406e310 msgr2=0x7fc054169e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.399+0000 7fc058b9c700 1 --2- 172.21.15.27:0/3859473370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc05406e310 0x7fc054169e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.399+0000 7fc058b9c700 1 -- 172.21.15.27:0/3859473370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04400b730 con 0x7fc054155ba0 2024-01-11T09:46:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.400+0000 7fc058b9c700 1 --2- 172.21.15.27:0/3859473370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc054155ba0 0x7fc0540c0880 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7fc04c007f20 tx=0x7fc04c0098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.400+0000 7fc03bfff700 1 -- 172.21.15.27:0/3859473370 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04c00eca0 con 0x7fc054155ba0 2024-01-11T09:46:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.400+0000 7fc03bfff700 1 -- 172.21.15.27:0/3859473370 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc04c00ee00 con 0x7fc054155ba0 2024-01-11T09:46:34.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.400+0000 7fc03bfff700 1 -- 172.21.15.27:0/3859473370 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04c014670 con 0x7fc054155ba0 2024-01-11T09:46:34.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.400+0000 7fc05a5ff700 1 -- 172.21.15.27:0/3859473370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc05416a7d0 con 0x7fc054155ba0 2024-01-11T09:46:34.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.400+0000 7fc05a5ff700 1 -- 172.21.15.27:0/3859473370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc05416ad90 con 0x7fc054155ba0 2024-01-11T09:46:34.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.401+0000 7fc05a5ff700 1 -- 172.21.15.27:0/3859473370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc05404c3c0 con 0x7fc054155ba0 2024-01-11T09:46:34.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.402+0000 7fc03bfff700 1 -- 172.21.15.27:0/3859473370 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fc04c014a20 con 0x7fc054155ba0 2024-01-11T09:46:34.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.403+0000 7fc03bfff700 1 --2- 172.21.15.27:0/3859473370 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc040067170 0x7fc040069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:34.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.404+0000 7fc03bfff700 1 -- 172.21.15.27:0/3859473370 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5817+0+0 (secure 0 0 0) 0x7fc04c084f70 con 0x7fc054155ba0 2024-01-11T09:46:34.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.404+0000 7fc053fff700 1 --2- 172.21.15.27:0/3859473370 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc040067170 0x7fc040069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:34.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.405+0000 7fc053fff700 1 --2- 172.21.15.27:0/3859473370 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc040067170 0x7fc040069620 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc05406ac70 tx=0x7fc044018000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:34.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.408+0000 7fc03bfff700 1 -- 172.21.15.27:0/3859473370 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc04c051ee0 con 0x7fc054155ba0 2024-01-11T09:46:34.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.662+0000 7fc05a5ff700 1 -- 172.21.15.27:0/3859473370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc0540ac030 con 0x7fc054155ba0 2024-01-11T09:46:34.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.948+0000 7fc03bfff700 1 -- 172.21.15.27:0/3859473370 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v128) v1 ==== 136+0+0 (secure 0 0 0) 0x7fc04c009930 con 0x7fc054155ba0 2024-01-11T09:46:34.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.950+0000 7fc05a5ff700 1 -- 172.21.15.27:0/3859473370 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc040067170 msgr2=0x7fc040069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:34.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.950+0000 7fc05a5ff700 1 --2- 172.21.15.27:0/3859473370 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc040067170 0x7fc040069620 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc05406ac70 tx=0x7fc044018000 comp rx=0 tx=0).stop 2024-01-11T09:46:34.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.950+0000 7fc05a5ff700 1 -- 172.21.15.27:0/3859473370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc054155ba0 msgr2=0x7fc0540c0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:34.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.950+0000 7fc05a5ff700 1 --2- 172.21.15.27:0/3859473370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc054155ba0 0x7fc0540c0880 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7fc04c007f20 tx=0x7fc04c0098a0 comp rx=0 tx=0).stop 2024-01-11T09:46:34.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.950+0000 7fc05a5ff700 1 -- 172.21.15.27:0/3859473370 shutdown_connections 2024-01-11T09:46:34.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.950+0000 7fc05a5ff700 1 --2- 172.21.15.27:0/3859473370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc054155ba0 0x7fc0540c0880 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:34.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.950+0000 7fc05a5ff700 1 --2- 172.21.15.27:0/3859473370 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc054069d70 0x7fc054151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:34.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.950+0000 7fc05a5ff700 1 --2- 172.21.15.27:0/3859473370 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc040067170 0x7fc040069620 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:34.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.950+0000 7fc05a5ff700 1 --2- 172.21.15.27:0/3859473370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc05406e310 0x7fc054169e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:34.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.950+0000 7fc05a5ff700 1 -- 172.21.15.27:0/3859473370 >> 172.21.15.27:0/3859473370 conn(0x7fc0540680e0 msgr2=0x7fc05406db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:34.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.950+0000 7fc05a5ff700 1 -- 172.21.15.27:0/3859473370 shutdown_connections 2024-01-11T09:46:34.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:34.950+0000 7fc05a5ff700 1 -- 172.21.15.27:0/3859473370 wait complete. 2024-01-11T09:46:34.960 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:46:35.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.170+0000 7fe16bfff700 1 Processor -- start 2024-01-11T09:46:35.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.170+0000 7fe16bfff700 1 -- start start 2024-01-11T09:46:35.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.171+0000 7fe16bfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe164069d40 0x7fe1640b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:35.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.171+0000 7fe16bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe1640c89f0 0x7fe1640b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:35.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.171+0000 7fe16bfff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe164157390 con 0x7fe1640c8620 2024-01-11T09:46:35.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.171+0000 7fe16bfff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1640adcd0 con 0x7fe1640c89f0 2024-01-11T09:46:35.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.171+0000 7fe16bfff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe16406a200 con 0x7fe164069d40 2024-01-11T09:46:35.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.172+0000 7fe16a59c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe1640c89f0 0x7fe1640b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:35.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.172+0000 7fe169d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe1640c8620 0x7fe1640af970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48132/0 (socket says 172.21.15.27:48132) 2024-01-11T09:46:35.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.172+0000 7fe169d9b700 1 -- 172.21.15.27:0/1291979569 learned_addr learned my addr 172.21.15.27:0/1291979569 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:35.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.172+0000 7fe16959a700 1 --2- 172.21.15.27:0/1291979569 >> v2:172.21.15.77:3300/0 conn(0x7fe164069d40 0x7fe1640b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:35.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.172+0000 7fe16a59c700 1 -- 172.21.15.27:0/1291979569 >> v2:172.21.15.77:3300/0 conn(0x7fe164069d40 msgr2=0x7fe1640b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:35.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.172+0000 7fe16a59c700 1 --2- 172.21.15.27:0/1291979569 >> v2:172.21.15.77:3300/0 conn(0x7fe164069d40 0x7fe1640b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.172+0000 7fe16a59c700 1 -- 172.21.15.27:0/1291979569 >> v1:172.21.15.27:6789/0 conn(0x7fe1640c8620 legacy=0x7fe1640af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:35.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.172+0000 7fe16a59c700 1 -- 172.21.15.27:0/1291979569 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe164156d40 con 0x7fe1640c89f0 2024-01-11T09:46:35.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.173+0000 7fe16a59c700 1 --2- 172.21.15.27:0/1291979569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1640c89f0 0x7fe1640b0810 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fe164157300 tx=0x7fe15c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a704bffbb5a652f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:35.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.173+0000 7fe168d99700 1 -- 172.21.15.27:0/1291979569 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15c00e070 con 0x7fe1640c89f0 2024-01-11T09:46:35.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.173+0000 7fe168d99700 1 -- 172.21.15.27:0/1291979569 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15c00bb60 con 0x7fe1640c89f0 2024-01-11T09:46:35.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.173+0000 7fe168d99700 1 -- 172.21.15.27:0/1291979569 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15c012430 con 0x7fe1640c89f0 2024-01-11T09:46:35.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.173+0000 7fe16bfff700 1 -- 172.21.15.27:0/1291979569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1640c89f0 msgr2=0x7fe1640b0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:35.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.173+0000 7fe16bfff700 1 --2- 172.21.15.27:0/1291979569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1640c89f0 0x7fe1640b0810 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fe164157300 tx=0x7fe15c009780 comp rx=0 tx=0).stop 2024-01-11T09:46:35.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.173+0000 7fe16bfff700 1 -- 172.21.15.27:0/1291979569 shutdown_connections 2024-01-11T09:46:35.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.173+0000 7fe16bfff700 1 --2- 172.21.15.27:0/1291979569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1640c89f0 0x7fe1640b0810 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.173+0000 7fe16bfff700 1 --2- 172.21.15.27:0/1291979569 >> v2:172.21.15.77:3300/0 conn(0x7fe164069d40 0x7fe1640b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.173+0000 7fe16bfff700 1 -- 172.21.15.27:0/1291979569 >> 172.21.15.27:0/1291979569 conn(0x7fe1640680e0 msgr2=0x7fe164069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:35.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.173+0000 7fe16bfff700 1 -- 172.21.15.27:0/1291979569 shutdown_connections 2024-01-11T09:46:35.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.174+0000 7fe16bfff700 1 -- 172.21.15.27:0/1291979569 wait complete. 2024-01-11T09:46:35.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.175+0000 7fe16bfff700 1 Processor -- start 2024-01-11T09:46:35.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.175+0000 7fe16bfff700 1 -- start start 2024-01-11T09:46:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.176+0000 7fe16bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe164069d40 0x7fe164167b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.176+0000 7fe16bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1640c8620 0x7fe16416a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.176+0000 7fe16bfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1640c89f0 0x7fe16414a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.176+0000 7fe16bfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1640c7680 con 0x7fe1640c8620 2024-01-11T09:46:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.176+0000 7fe16bfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe164168040 con 0x7fe164069d40 2024-01-11T09:46:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.176+0000 7fe16bfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16416b590 con 0x7fe1640c89f0 2024-01-11T09:46:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.176+0000 7fe16959a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1640c8620 0x7fe16416a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.176+0000 7fe16959a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1640c8620 0x7fe16416a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40258/0 (socket says 172.21.15.27:40258) 2024-01-11T09:46:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.176+0000 7fe16959a700 1 -- 172.21.15.27:0/1510884873 learned_addr learned my addr 172.21.15.27:0/1510884873 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.176+0000 7fe169d9b700 1 --2- 172.21.15.27:0/1510884873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe164069d40 0x7fe164167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.177+0000 7fe16a59c700 1 --2- 172.21.15.27:0/1510884873 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1640c89f0 0x7fe16414a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:35.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.177+0000 7fe16959a700 1 -- 172.21.15.27:0/1510884873 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1640c89f0 msgr2=0x7fe16414a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:35.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.177+0000 7fe16959a700 1 --2- 172.21.15.27:0/1510884873 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1640c89f0 0x7fe16414a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.177+0000 7fe16959a700 1 -- 172.21.15.27:0/1510884873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe164069d40 msgr2=0x7fe164167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:35.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.177+0000 7fe16959a700 1 --2- 172.21.15.27:0/1510884873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe164069d40 0x7fe164167b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.177+0000 7fe16959a700 1 -- 172.21.15.27:0/1510884873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15c009a20 con 0x7fe1640c8620 2024-01-11T09:46:35.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.177+0000 7fe16a59c700 1 --2- 172.21.15.27:0/1510884873 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1640c89f0 0x7fe16414a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:35.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.177+0000 7fe16959a700 1 --2- 172.21.15.27:0/1510884873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1640c8620 0x7fe16416a870 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fe15400ec80 tx=0x7fe1540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:35.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.177+0000 7fe159ffb700 1 -- 172.21.15.27:0/1510884873 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15400c850 con 0x7fe1640c8620 2024-01-11T09:46:35.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.177+0000 7fe159ffb700 1 -- 172.21.15.27:0/1510884873 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15400c9b0 con 0x7fe1640c8620 2024-01-11T09:46:35.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.177+0000 7fe159ffb700 1 -- 172.21.15.27:0/1510884873 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15400cd50 con 0x7fe1640c8620 2024-01-11T09:46:35.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.177+0000 7fe16bfff700 1 -- 172.21.15.27:0/1510884873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16416b870 con 0x7fe1640c8620 2024-01-11T09:46:35.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.178+0000 7fe16bfff700 1 -- 172.21.15.27:0/1510884873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16416be30 con 0x7fe1640c8620 2024-01-11T09:46:35.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.179+0000 7fe16bfff700 1 -- 172.21.15.27:0/1510884873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1640718d0 con 0x7fe1640c8620 2024-01-11T09:46:35.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.180+0000 7fe159ffb700 1 -- 172.21.15.27:0/1510884873 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fe154013680 con 0x7fe1640c8620 2024-01-11T09:46:35.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.181+0000 7fe159ffb700 1 --2- 172.21.15.27:0/1510884873 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe148067170 0x7fe148069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:35.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.181+0000 7fe159ffb700 1 -- 172.21.15.27:0/1510884873 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5817+0+0 (secure 0 0 0) 0x7fe154084bc0 con 0x7fe1640c8620 2024-01-11T09:46:35.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.181+0000 7fe169d9b700 1 --2- 172.21.15.27:0/1510884873 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe148067170 0x7fe148069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:35.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.182+0000 7fe169d9b700 1 --2- 172.21.15.27:0/1510884873 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe148067170 0x7fe148069620 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe160007130 tx=0x7fe160007eb0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:35.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.185+0000 7fe159ffb700 1 -- 172.21.15.27:0/1510884873 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe154051b30 con 0x7fe1640c8620 2024-01-11T09:46:35.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.389+0000 7fe16bfff700 1 -- 172.21.15.27:0/1510884873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe164157020 con 0x7fe1640c8620 2024-01-11T09:46:35.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.390+0000 7fe159ffb700 1 -- 172.21.15.27:0/1510884873 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v235) v1 ==== 74+0+15495 (secure 0 0 0) 0x7fe154055150 con 0x7fe1640c8620 2024-01-11T09:46:35.382 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:35.382 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":235,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:46:26.932357+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:46:24.049904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-01-11T09:46:24.737495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:46:35.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.393+0000 7fe16bfff700 1 -- 172.21.15.27:0/1510884873 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe148067170 msgr2=0x7fe148069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:35.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.393+0000 7fe16bfff700 1 --2- 172.21.15.27:0/1510884873 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe148067170 0x7fe148069620 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe160007130 tx=0x7fe160007eb0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.393+0000 7fe16bfff700 1 -- 172.21.15.27:0/1510884873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1640c8620 msgr2=0x7fe16416a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:35.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.393+0000 7fe16bfff700 1 --2- 172.21.15.27:0/1510884873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1640c8620 0x7fe16416a870 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fe15400ec80 tx=0x7fe1540091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.393+0000 7fe16bfff700 1 -- 172.21.15.27:0/1510884873 shutdown_connections 2024-01-11T09:46:35.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.393+0000 7fe16bfff700 1 --2- 172.21.15.27:0/1510884873 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe1640c89f0 0x7fe16414a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.393+0000 7fe16bfff700 1 --2- 172.21.15.27:0/1510884873 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe148067170 0x7fe148069620 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.393+0000 7fe16bfff700 1 --2- 172.21.15.27:0/1510884873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe164069d40 0x7fe164167b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.393+0000 7fe16bfff700 1 --2- 172.21.15.27:0/1510884873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1640c8620 0x7fe16416a870 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.393+0000 7fe16bfff700 1 -- 172.21.15.27:0/1510884873 >> 172.21.15.27:0/1510884873 conn(0x7fe1640680e0 msgr2=0x7fe164158590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:35.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.393+0000 7fe16bfff700 1 -- 172.21.15.27:0/1510884873 shutdown_connections 2024-01-11T09:46:35.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.393+0000 7fe16bfff700 1 -- 172.21.15.27:0/1510884873 wait complete. 2024-01-11T09:46:35.403 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T09:46:24.049904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '234', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T09:46:24.737495+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '235', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T09:46:35.403 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T09:46:35.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.614+0000 7f0a975ff700 1 Processor -- start 2024-01-11T09:46:35.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.614+0000 7f0a975ff700 1 -- start start 2024-01-11T09:46:35.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.615+0000 7f0a975ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a90157410 0x7f0a900ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:35.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.615+0000 7f0a975ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0a901577e0 0x7f0a900ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:35.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.615+0000 7f0a975ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a900699d0 con 0x7f0a90069540 2024-01-11T09:46:35.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.615+0000 7f0a975ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a90069cb0 con 0x7f0a901577e0 2024-01-11T09:46:35.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.615+0000 7f0a975ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a90069f90 con 0x7f0a90157410 2024-01-11T09:46:35.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.615+0000 7f0a94b9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0a901577e0 0x7f0a900ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:35.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.615+0000 7f0a95b9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0a90069540 0x7f0a9015b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48140/0 (socket says 172.21.15.27:48140) 2024-01-11T09:46:35.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.616+0000 7f0a95b9c700 1 -- 172.21.15.27:0/3915489489 learned_addr learned my addr 172.21.15.27:0/3915489489 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:35.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.616+0000 7f0a9539b700 1 --2- 172.21.15.27:0/3915489489 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a90157410 0x7f0a900ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:35.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.616+0000 7f0a94b9a700 1 -- 172.21.15.27:0/3915489489 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a90157410 msgr2=0x7f0a900ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:35.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.616+0000 7f0a94b9a700 1 --2- 172.21.15.27:0/3915489489 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a90157410 0x7f0a900ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.616+0000 7f0a94b9a700 1 -- 172.21.15.27:0/3915489489 >> v1:172.21.15.27:6789/0 conn(0x7f0a90069540 legacy=0x7f0a9015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:35.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.616+0000 7f0a94b9a700 1 -- 172.21.15.27:0/3915489489 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9006a270 con 0x7f0a901577e0 2024-01-11T09:46:35.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.616+0000 7f0a87fff700 1 -- 172.21.15.27:0/3915489489 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 196815667 0 0) 0x7f0a900699d0 con 0x7f0a90069540 2024-01-11T09:46:35.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.616+0000 7f0a94b9a700 1 --2- 172.21.15.27:0/3915489489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a901577e0 0x7f0a900ae000 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f0a800098e0 tx=0x7f0a800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2fc3aa10eecbe72 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:35.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.616+0000 7f0a87fff700 1 -- 172.21.15.27:0/3915489489 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8000b640 con 0x7f0a901577e0 2024-01-11T09:46:35.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.617+0000 7f0a87fff700 1 -- 172.21.15.27:0/3915489489 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8000b7a0 con 0x7f0a901577e0 2024-01-11T09:46:35.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.617+0000 7f0a87fff700 1 -- 172.21.15.27:0/3915489489 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8000bb40 con 0x7f0a901577e0 2024-01-11T09:46:35.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.617+0000 7f0a975ff700 1 -- 172.21.15.27:0/3915489489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a901577e0 msgr2=0x7f0a900ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:35.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.617+0000 7f0a975ff700 1 --2- 172.21.15.27:0/3915489489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a901577e0 0x7f0a900ae000 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f0a800098e0 tx=0x7f0a800091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.617+0000 7f0a975ff700 1 -- 172.21.15.27:0/3915489489 shutdown_connections 2024-01-11T09:46:35.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.617+0000 7f0a975ff700 1 --2- 172.21.15.27:0/3915489489 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a90157410 0x7f0a900ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.617+0000 7f0a975ff700 1 --2- 172.21.15.27:0/3915489489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a901577e0 0x7f0a900ae000 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.617+0000 7f0a975ff700 1 -- 172.21.15.27:0/3915489489 >> 172.21.15.27:0/3915489489 conn(0x7f0a900680a0 msgr2=0x7f0a900cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:35.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.617+0000 7f0a975ff700 1 -- 172.21.15.27:0/3915489489 shutdown_connections 2024-01-11T09:46:35.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.617+0000 7f0a975ff700 1 -- 172.21.15.27:0/3915489489 wait complete. 2024-01-11T09:46:35.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.619+0000 7f0a975ff700 1 Processor -- start 2024-01-11T09:46:35.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.619+0000 7f0a975ff700 1 -- start start 2024-01-11T09:46:35.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.620+0000 7f0a975ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a90069540 0x7f0a9016a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:35.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.620+0000 7f0a975ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a90157410 0x7f0a9014d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:35.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.620+0000 7f0a975ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a901577e0 0x7f0a9015f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:35.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.620+0000 7f0a975ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9016cd60 con 0x7f0a901577e0 2024-01-11T09:46:35.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.620+0000 7f0a975ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9016d040 con 0x7f0a90157410 2024-01-11T09:46:35.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.620+0000 7f0a975ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9016d320 con 0x7f0a90069540 2024-01-11T09:46:35.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.620+0000 7f0a95b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a901577e0 0x7f0a9015f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:35.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.620+0000 7f0a95b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a901577e0 0x7f0a9015f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40276/0 (socket says 172.21.15.27:40276) 2024-01-11T09:46:35.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.620+0000 7f0a95b9c700 1 -- 172.21.15.27:0/3479557111 learned_addr learned my addr 172.21.15.27:0/3479557111 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:35.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.620+0000 7f0a9539b700 1 --2- 172.21.15.27:0/3479557111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a90069540 0x7f0a9016a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:35.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.620+0000 7f0a94b9a700 1 --2- 172.21.15.27:0/3479557111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a90157410 0x7f0a9014d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:35.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.621+0000 7f0a95b9c700 1 -- 172.21.15.27:0/3479557111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a90069540 msgr2=0x7f0a9016a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.621+0000 7f0a95b9c700 1 --2- 172.21.15.27:0/3479557111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a90069540 0x7f0a9016a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.621+0000 7f0a95b9c700 1 -- 172.21.15.27:0/3479557111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a90157410 msgr2=0x7f0a9014d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.621+0000 7f0a95b9c700 1 --2- 172.21.15.27:0/3479557111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a90157410 0x7f0a9014d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.621+0000 7f0a95b9c700 1 -- 172.21.15.27:0/3479557111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a80009420 con 0x7f0a901577e0 2024-01-11T09:46:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.621+0000 7f0a95b9c700 1 --2- 172.21.15.27:0/3479557111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a901577e0 0x7f0a9015f5b0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f0a8800ccc0 tx=0x7f0a880019e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.621+0000 7f0a857fa700 1 -- 172.21.15.27:0/3479557111 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8800a880 con 0x7f0a901577e0 2024-01-11T09:46:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.621+0000 7f0a857fa700 1 -- 172.21.15.27:0/3479557111 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8800aa00 con 0x7f0a901577e0 2024-01-11T09:46:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.621+0000 7f0a857fa700 1 -- 172.21.15.27:0/3479557111 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8800adc0 con 0x7f0a901577e0 2024-01-11T09:46:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.621+0000 7f0a975ff700 1 -- 172.21.15.27:0/3479557111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a9016d600 con 0x7f0a901577e0 2024-01-11T09:46:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.621+0000 7f0a975ff700 1 -- 172.21.15.27:0/3479557111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a9014d890 con 0x7f0a901577e0 2024-01-11T09:46:35.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.622+0000 7f0a975ff700 1 -- 172.21.15.27:0/3479557111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a9004c3c0 con 0x7f0a901577e0 2024-01-11T09:46:35.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.623+0000 7f0a857fa700 1 -- 172.21.15.27:0/3479557111 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f0a88007300 con 0x7f0a901577e0 2024-01-11T09:46:35.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.625+0000 7f0a857fa700 1 --2- 172.21.15.27:0/3479557111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0a74067170 0x7f0a74069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:35.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.625+0000 7f0a857fa700 1 -- 172.21.15.27:0/3479557111 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f0a88084bd0 con 0x7f0a901577e0 2024-01-11T09:46:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.625+0000 7f0a9539b700 1 --2- 172.21.15.27:0/3479557111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0a74067170 0x7f0a74069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.626+0000 7f0a9539b700 1 --2- 172.21.15.27:0/3479557111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0a74067170 0x7f0a74069620 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0a900adc50 tx=0x7f0a8c009120 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:35.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.629+0000 7f0a857fa700 1 -- 172.21.15.27:0/3479557111 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0a88051bc0 con 0x7f0a901577e0 2024-01-11T09:46:35.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.832+0000 7f0a975ff700 1 -- 172.21.15.27:0/3479557111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0a900c1980 con 0x7f0a901577e0 2024-01-11T09:46:35.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.833+0000 7f0a857fa700 1 -- 172.21.15.27:0/3479557111 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v235) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0a880551e0 con 0x7f0a901577e0 2024-01-11T09:46:35.824 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T09:46:35.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.835+0000 7f0a975ff700 1 -- 172.21.15.27:0/3479557111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0a74067170 msgr2=0x7f0a74069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:35.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.835+0000 7f0a975ff700 1 --2- 172.21.15.27:0/3479557111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0a74067170 0x7f0a74069620 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0a900adc50 tx=0x7f0a8c009120 comp rx=0 tx=0).stop 2024-01-11T09:46:35.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.835+0000 7f0a975ff700 1 -- 172.21.15.27:0/3479557111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a901577e0 msgr2=0x7f0a9015f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:35.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.835+0000 7f0a975ff700 1 --2- 172.21.15.27:0/3479557111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a901577e0 0x7f0a9015f5b0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f0a8800ccc0 tx=0x7f0a880019e0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.835+0000 7f0a975ff700 1 -- 172.21.15.27:0/3479557111 shutdown_connections 2024-01-11T09:46:35.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.835+0000 7f0a975ff700 1 --2- 172.21.15.27:0/3479557111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a901577e0 0x7f0a9015f5b0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.835+0000 7f0a975ff700 1 --2- 172.21.15.27:0/3479557111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0a90069540 0x7f0a9016a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.835+0000 7f0a975ff700 1 --2- 172.21.15.27:0/3479557111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0a74067170 0x7f0a74069620 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.835+0000 7f0a975ff700 1 --2- 172.21.15.27:0/3479557111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a90157410 0x7f0a9014d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:35.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.835+0000 7f0a975ff700 1 -- 172.21.15.27:0/3479557111 >> 172.21.15.27:0/3479557111 conn(0x7f0a900680a0 msgr2=0x7f0a900c24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:35.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.835+0000 7f0a975ff700 1 -- 172.21.15.27:0/3479557111 shutdown_connections 2024-01-11T09:46:35.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:35.835+0000 7f0a975ff700 1 -- 172.21.15.27:0/3479557111 wait complete. 2024-01-11T09:46:35.844 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T09:46:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.056+0000 7f3bb85ff700 1 Processor -- start 2024-01-11T09:46:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.057+0000 7f3bb85ff700 1 -- start start 2024-01-11T09:46:36.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.058+0000 7f3bb85ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3bb00ccd70 0x7f3bb00cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:36.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.058+0000 7f3bb85ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3bb015ae70 0x7f3bb015b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:36.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.058+0000 7f3bb85ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb00cf850 con 0x7f3bb015a1b0 2024-01-11T09:46:36.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.058+0000 7f3bb85ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb015ac90 con 0x7f3bb00ccd70 2024-01-11T09:46:36.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.058+0000 7f3bb85ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb00cec40 con 0x7f3bb015ae70 2024-01-11T09:46:36.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.058+0000 7f3bb6b9c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3bb015ae70 0x7f3bb015b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:36.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.058+0000 7f3bb639b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3bb015a1b0 0x7f3bb015a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:55440/0 (socket says 172.21.15.27:55440) 2024-01-11T09:46:36.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.059+0000 7f3bb639b700 1 -- 172.21.15.27:0/1951387266 learned_addr learned my addr 172.21.15.27:0/1951387266 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:36.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.058+0000 7f3bb5b9a700 1 --2- 172.21.15.27:0/1951387266 >> v2:172.21.15.77:3300/0 conn(0x7f3bb00ccd70 0x7f3bb00cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:36.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.059+0000 7f3bb5399700 1 -- 172.21.15.27:0/1951387266 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719320972 0 0) 0x7f3bb00cf850 con 0x7f3bb015a1b0 2024-01-11T09:46:36.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.059+0000 7f3bb5399700 1 -- 172.21.15.27:0/1951387266 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b9c003570 con 0x7f3bb015a1b0 2024-01-11T09:46:36.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.059+0000 7f3bb6b9c700 1 -- 172.21.15.27:0/1951387266 >> v2:172.21.15.77:3300/0 conn(0x7f3bb00ccd70 msgr2=0x7f3bb00cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:36.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.059+0000 7f3bb6b9c700 1 --2- 172.21.15.27:0/1951387266 >> v2:172.21.15.77:3300/0 conn(0x7f3bb00ccd70 0x7f3bb00cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.059+0000 7f3bb6b9c700 1 -- 172.21.15.27:0/1951387266 >> v1:172.21.15.77:6789/0 conn(0x7f3bb015a1b0 legacy=0x7f3bb015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:36.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.059+0000 7f3bb6b9c700 1 -- 172.21.15.27:0/1951387266 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb00cef20 con 0x7f3bb015ae70 2024-01-11T09:46:36.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.059+0000 7f3bb5b9a700 1 --2- 172.21.15.27:0/1951387266 >> v2:172.21.15.77:3300/0 conn(0x7f3bb00ccd70 0x7f3bb00cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:36.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.059+0000 7f3bb6b9c700 1 --2- 172.21.15.27:0/1951387266 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3bb015ae70 0x7f3bb015b2f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8009e60 tx=0x7f3ba8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4f64a6037445cbe server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.060+0000 7f3bb5399700 1 -- 172.21.15.27:0/1951387266 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba800e070 con 0x7f3bb015ae70 2024-01-11T09:46:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.060+0000 7f3bb5399700 1 -- 172.21.15.27:0/1951387266 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba800bab0 con 0x7f3bb015ae70 2024-01-11T09:46:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.060+0000 7f3bb5399700 1 -- 172.21.15.27:0/1951387266 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba800be70 con 0x7f3bb015ae70 2024-01-11T09:46:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.060+0000 7f3bb85ff700 1 -- 172.21.15.27:0/1951387266 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3bb015ae70 msgr2=0x7f3bb015b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.060+0000 7f3bb85ff700 1 --2- 172.21.15.27:0/1951387266 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3bb015ae70 0x7f3bb015b2f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8009e60 tx=0x7f3ba8009780 comp rx=0 tx=0).stop 2024-01-11T09:46:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.060+0000 7f3bb85ff700 1 -- 172.21.15.27:0/1951387266 shutdown_connections 2024-01-11T09:46:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.060+0000 7f3bb85ff700 1 --2- 172.21.15.27:0/1951387266 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3bb015ae70 0x7f3bb015b2f0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.060+0000 7f3bb85ff700 1 --2- 172.21.15.27:0/1951387266 >> v2:172.21.15.77:3300/0 conn(0x7f3bb00ccd70 0x7f3bb00cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.060+0000 7f3bb85ff700 1 -- 172.21.15.27:0/1951387266 >> 172.21.15.27:0/1951387266 conn(0x7f3bb006ed40 msgr2=0x7f3bb006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.060+0000 7f3bb85ff700 1 -- 172.21.15.27:0/1951387266 shutdown_connections 2024-01-11T09:46:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.060+0000 7f3bb85ff700 1 -- 172.21.15.27:0/1951387266 wait complete. 2024-01-11T09:46:36.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.062+0000 7f3bb85ff700 1 Processor -- start 2024-01-11T09:46:36.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.062+0000 7f3bb85ff700 1 -- start start 2024-01-11T09:46:36.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.063+0000 7f3bb85ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3bb00ccd70 0x7f3bb0147f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:36.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.063+0000 7f3bb85ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bb015a1b0 0x7f3bb016cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:36.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.063+0000 7f3bb85ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bb015ae70 0x7f3bb00685c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:36.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.063+0000 7f3bb85ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb0157e60 con 0x7f3bb015a1b0 2024-01-11T09:46:36.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.063+0000 7f3bb85ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb016d5c0 con 0x7f3bb015ae70 2024-01-11T09:46:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.063+0000 7f3bb85ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb016d8a0 con 0x7f3bb00ccd70 2024-01-11T09:46:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.063+0000 7f3bb5b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bb015a1b0 0x7f3bb016cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.063+0000 7f3bb5b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bb015a1b0 0x7f3bb016cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40288/0 (socket says 172.21.15.27:40288) 2024-01-11T09:46:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.063+0000 7f3bb5b9a700 1 -- 172.21.15.27:0/3310467305 learned_addr learned my addr 172.21.15.27:0/3310467305 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.064+0000 7f3bb639b700 1 --2- 172.21.15.27:0/3310467305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3bb00ccd70 0x7f3bb0147f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.064+0000 7f3bb6b9c700 1 --2- 172.21.15.27:0/3310467305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bb015ae70 0x7f3bb00685c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.064+0000 7f3bb5b9a700 1 -- 172.21.15.27:0/3310467305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3bb00ccd70 msgr2=0x7f3bb0147f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.064+0000 7f3bb5b9a700 1 --2- 172.21.15.27:0/3310467305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3bb00ccd70 0x7f3bb0147f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.064+0000 7f3bb5b9a700 1 -- 172.21.15.27:0/3310467305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bb015ae70 msgr2=0x7f3bb00685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.064+0000 7f3bb5b9a700 1 --2- 172.21.15.27:0/3310467305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bb015ae70 0x7f3bb00685c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.064+0000 7f3bb5b9a700 1 -- 172.21.15.27:0/3310467305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba0009420 con 0x7f3bb015a1b0 2024-01-11T09:46:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.064+0000 7f3bb6b9c700 1 --2- 172.21.15.27:0/3310467305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bb015ae70 0x7f3bb00685c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:46:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.064+0000 7f3bb5b9a700 1 --2- 172.21.15.27:0/3310467305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bb015a1b0 0x7f3bb016cc80 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f3ba000ec80 tx=0x7f3ba00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.064+0000 7f3ba67fc700 1 -- 172.21.15.27:0/3310467305 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba000c9d0 con 0x7f3bb015a1b0 2024-01-11T09:46:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.064+0000 7f3ba67fc700 1 -- 172.21.15.27:0/3310467305 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba000cb30 con 0x7f3bb015a1b0 2024-01-11T09:46:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.064+0000 7f3ba67fc700 1 -- 172.21.15.27:0/3310467305 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba000ceb0 con 0x7f3bb015a1b0 2024-01-11T09:46:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.064+0000 7f3bb85ff700 1 -- 172.21.15.27:0/3310467305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ba8009a20 con 0x7f3bb015a1b0 2024-01-11T09:46:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.064+0000 7f3bb85ff700 1 -- 172.21.15.27:0/3310467305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb016db80 con 0x7f3bb015a1b0 2024-01-11T09:46:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.065+0000 7f3bb85ff700 1 -- 172.21.15.27:0/3310467305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb01582c0 con 0x7f3bb015a1b0 2024-01-11T09:46:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.067+0000 7f3ba67fc700 1 -- 172.21.15.27:0/3310467305 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f3ba0013680 con 0x7f3bb015a1b0 2024-01-11T09:46:36.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.067+0000 7f3ba67fc700 1 --2- 172.21.15.27:0/3310467305 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3b94067170 0x7f3b94069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:36.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.067+0000 7f3ba67fc700 1 -- 172.21.15.27:0/3310467305 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f3ba0084ca0 con 0x7f3bb015a1b0 2024-01-11T09:46:36.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.068+0000 7f3bb639b700 1 --2- 172.21.15.27:0/3310467305 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3b94067170 0x7f3b94069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:36.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.068+0000 7f3bb639b700 1 --2- 172.21.15.27:0/3310467305 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3b94067170 0x7f3b94069620 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3bac0058d0 tx=0x7f3bac009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:36.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.072+0000 7f3ba67fc700 1 -- 172.21.15.27:0/3310467305 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3ba0051c10 con 0x7f3bb015a1b0 2024-01-11T09:46:36.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.276+0000 7f3bb85ff700 1 -- 172.21.15.27:0/3310467305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3bb00c75c0 con 0x7f3bb015a1b0 2024-01-11T09:46:36.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.276+0000 7f3ba67fc700 1 -- 172.21.15.27:0/3310467305 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v235) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3ba0055230 con 0x7f3bb015a1b0 2024-01-11T09:46:36.268 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T09:46:36.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.278+0000 7f3bb85ff700 1 -- 172.21.15.27:0/3310467305 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3b94067170 msgr2=0x7f3b94069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:36.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.278+0000 7f3bb85ff700 1 --2- 172.21.15.27:0/3310467305 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3b94067170 0x7f3b94069620 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3bac0058d0 tx=0x7f3bac009040 comp rx=0 tx=0).stop 2024-01-11T09:46:36.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.278+0000 7f3bb85ff700 1 -- 172.21.15.27:0/3310467305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bb015a1b0 msgr2=0x7f3bb016cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:36.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.278+0000 7f3bb85ff700 1 --2- 172.21.15.27:0/3310467305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bb015a1b0 0x7f3bb016cc80 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f3ba000ec80 tx=0x7f3ba00091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.278+0000 7f3bb85ff700 1 -- 172.21.15.27:0/3310467305 shutdown_connections 2024-01-11T09:46:36.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.279+0000 7f3bb85ff700 1 --2- 172.21.15.27:0/3310467305 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3b94067170 0x7f3b94069620 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.279+0000 7f3bb85ff700 1 --2- 172.21.15.27:0/3310467305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bb015ae70 0x7f3bb00685c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.279+0000 7f3bb85ff700 1 --2- 172.21.15.27:0/3310467305 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3bb00ccd70 0x7f3bb0147f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.279+0000 7f3bb85ff700 1 --2- 172.21.15.27:0/3310467305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bb015a1b0 0x7f3bb016cc80 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.279+0000 7f3bb85ff700 1 -- 172.21.15.27:0/3310467305 >> 172.21.15.27:0/3310467305 conn(0x7f3bb006ed40 msgr2=0x7f3bb01531a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:36.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.279+0000 7f3bb85ff700 1 -- 172.21.15.27:0/3310467305 shutdown_connections 2024-01-11T09:46:36.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.279+0000 7f3bb85ff700 1 -- 172.21.15.27:0/3310467305 wait complete. 2024-01-11T09:46:36.288 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T09:46:36.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.500+0000 7f8a843ff700 1 Processor -- start 2024-01-11T09:46:36.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.500+0000 7f8a843ff700 1 -- start start 2024-01-11T09:46:36.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.501+0000 7f8a843ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8a7c1596f0 0x7f8a7c0a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:36.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.501+0000 7f8a843ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a7c068d30 0x7f8a7c0c8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:36.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.501+0000 7f8a843ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a7c0c80a0 con 0x7f8a7c0a9de0 2024-01-11T09:46:36.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.501+0000 7f8a843ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a7c0cbac0 con 0x7f8a7c1596f0 2024-01-11T09:46:36.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.501+0000 7f8a843ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7c0c7490 con 0x7f8a7c068d30 2024-01-11T09:46:36.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.502+0000 7f8a8219b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8a7c1596f0 0x7f8a7c0a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:36.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.502+0000 7f8a8219b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8a7c1596f0 0x7f8a7c0a98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40292/0 (socket says 172.21.15.27:40292) 2024-01-11T09:46:36.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.502+0000 7f8a8219b700 1 -- 172.21.15.27:0/3354013997 learned_addr learned my addr 172.21.15.27:0/3354013997 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.502+0000 7f8a8219b700 1 -- 172.21.15.27:0/3354013997 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a7c068d30 msgr2=0x7f8a7c0c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.502+0000 7f8a8219b700 1 --2- 172.21.15.27:0/3354013997 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a7c068d30 0x7f8a7c0c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.502+0000 7f8a8219b700 1 -- 172.21.15.27:0/3354013997 >> v1:172.21.15.77:6789/0 conn(0x7f8a7c0a9de0 legacy=0x7f8a7c068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.502+0000 7f8a8219b700 1 -- 172.21.15.27:0/3354013997 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a7c0c7770 con 0x7f8a7c1596f0 2024-01-11T09:46:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.502+0000 7f8a8219b700 1 --2- 172.21.15.27:0/3354013997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a7c1596f0 0x7f8a7c0a98a0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f8a780099b0 tx=0x7f8a78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=475f7246f912ad55 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.502+0000 7f8a81199700 1 -- 172.21.15.27:0/3354013997 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7800b820 con 0x7f8a7c1596f0 2024-01-11T09:46:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.503+0000 7f8a81199700 1 -- 172.21.15.27:0/3354013997 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7800b980 con 0x7f8a7c1596f0 2024-01-11T09:46:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.503+0000 7f8a81199700 1 -- 172.21.15.27:0/3354013997 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7800bd20 con 0x7f8a7c1596f0 2024-01-11T09:46:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.503+0000 7f8a843ff700 1 -- 172.21.15.27:0/3354013997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a7c1596f0 msgr2=0x7f8a7c0a98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.503+0000 7f8a843ff700 1 --2- 172.21.15.27:0/3354013997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a7c1596f0 0x7f8a7c0a98a0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f8a780099b0 tx=0x7f8a78009250 comp rx=0 tx=0).stop 2024-01-11T09:46:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.503+0000 7f8a843ff700 1 -- 172.21.15.27:0/3354013997 shutdown_connections 2024-01-11T09:46:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.503+0000 7f8a843ff700 1 --2- 172.21.15.27:0/3354013997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a7c1596f0 0x7f8a7c0a98a0 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.503+0000 7f8a843ff700 1 --2- 172.21.15.27:0/3354013997 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a7c068d30 0x7f8a7c0c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.503+0000 7f8a843ff700 1 -- 172.21.15.27:0/3354013997 >> 172.21.15.27:0/3354013997 conn(0x7f8a7c0c9fb0 msgr2=0x7f8a7c06ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.503+0000 7f8a843ff700 1 -- 172.21.15.27:0/3354013997 shutdown_connections 2024-01-11T09:46:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.503+0000 7f8a843ff700 1 -- 172.21.15.27:0/3354013997 wait complete. 2024-01-11T09:46:36.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.505+0000 7f8a843ff700 1 Processor -- start 2024-01-11T09:46:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.506+0000 7f8a843ff700 1 -- start start 2024-01-11T09:46:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.507+0000 7f8a843ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a7c068d30 0x7f8a7c16d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.507+0000 7f8a843ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a7c0a9de0 0x7f8a7c1489d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.507+0000 7f8a843ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a7c1596f0 0x7f8a7c1569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.507+0000 7f8a843ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7c16e4b0 con 0x7f8a7c1596f0 2024-01-11T09:46:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.507+0000 7f8a843ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7c169f80 con 0x7f8a7c0a9de0 2024-01-11T09:46:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.507+0000 7f8a843ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7c16a220 con 0x7f8a7c068d30 2024-01-11T09:46:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.507+0000 7f8a8299c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a7c1596f0 0x7f8a7c1569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.507+0000 7f8a8299c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a7c1596f0 0x7f8a7c1569c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40296/0 (socket says 172.21.15.27:40296) 2024-01-11T09:46:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.507+0000 7f8a8299c700 1 -- 172.21.15.27:0/3255062039 learned_addr learned my addr 172.21.15.27:0/3255062039 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.508+0000 7f8a8299c700 1 -- 172.21.15.27:0/3255062039 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a7c068d30 msgr2=0x7f8a7c16d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.508+0000 7f8a8219b700 1 --2- 172.21.15.27:0/3255062039 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a7c068d30 0x7f8a7c16d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.508+0000 7f8a8199a700 1 --2- 172.21.15.27:0/3255062039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a7c0a9de0 0x7f8a7c1489d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.508+0000 7f8a8299c700 1 --2- 172.21.15.27:0/3255062039 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a7c068d30 0x7f8a7c16d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.508+0000 7f8a8299c700 1 -- 172.21.15.27:0/3255062039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a7c0a9de0 msgr2=0x7f8a7c1489d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.508+0000 7f8a8299c700 1 --2- 172.21.15.27:0/3255062039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a7c0a9de0 0x7f8a7c1489d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.508+0000 7f8a8299c700 1 -- 172.21.15.27:0/3255062039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a780094f0 con 0x7f8a7c1596f0 2024-01-11T09:46:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.508+0000 7f8a8219b700 1 --2- 172.21.15.27:0/3255062039 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a7c068d30 0x7f8a7c16d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:46:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.508+0000 7f8a8199a700 1 --2- 172.21.15.27:0/3255062039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a7c0a9de0 0x7f8a7c1489d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:46:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.508+0000 7f8a8299c700 1 --2- 172.21.15.27:0/3255062039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a7c1596f0 0x7f8a7c1569c0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f8a7400bb60 tx=0x7f8a74009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.508+0000 7f8a727fc700 1 -- 172.21.15.27:0/3255062039 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7400c940 con 0x7f8a7c1596f0 2024-01-11T09:46:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.508+0000 7f8a727fc700 1 -- 172.21.15.27:0/3255062039 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7400cac0 con 0x7f8a7c1596f0 2024-01-11T09:46:36.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.508+0000 7f8a843ff700 1 -- 172.21.15.27:0/3255062039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a7c16dc10 con 0x7f8a7c1596f0 2024-01-11T09:46:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.508+0000 7f8a727fc700 1 -- 172.21.15.27:0/3255062039 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7400ce80 con 0x7f8a7c1596f0 2024-01-11T09:46:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.508+0000 7f8a843ff700 1 -- 172.21.15.27:0/3255062039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a7c16e1d0 con 0x7f8a7c1596f0 2024-01-11T09:46:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.509+0000 7f8a843ff700 1 -- 172.21.15.27:0/3255062039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a7c0ab960 con 0x7f8a7c1596f0 2024-01-11T09:46:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.510+0000 7f8a727fc700 1 -- 172.21.15.27:0/3255062039 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f8a74014620 con 0x7f8a7c1596f0 2024-01-11T09:46:36.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.511+0000 7f8a727fc700 1 --2- 172.21.15.27:0/3255062039 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8a60067170 0x7f8a60069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:36.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.511+0000 7f8a727fc700 1 -- 172.21.15.27:0/3255062039 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f8a740849a0 con 0x7f8a7c1596f0 2024-01-11T09:46:36.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.512+0000 7f8a8219b700 1 --2- 172.21.15.27:0/3255062039 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8a60067170 0x7f8a60069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:36.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.512+0000 7f8a8219b700 1 --2- 172.21.15.27:0/3255062039 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8a60067170 0x7f8a60069620 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8a780099b0 tx=0x7f8a78016000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:36.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.515+0000 7f8a727fc700 1 -- 172.21.15.27:0/3255062039 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8a7404c5a0 con 0x7f8a7c1596f0 2024-01-11T09:46:36.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.723+0000 7f8a843ff700 1 -- 172.21.15.27:0/3255062039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8a7c04c3c0 con 0x7f8a7c1596f0 2024-01-11T09:46:36.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.723+0000 7f8a727fc700 1 -- 172.21.15.27:0/3255062039 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v235) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8a74055300 con 0x7f8a7c1596f0 2024-01-11T09:46:36.715 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T09:46:36.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.726+0000 7f8a843ff700 1 -- 172.21.15.27:0/3255062039 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8a60067170 msgr2=0x7f8a60069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:36.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.726+0000 7f8a843ff700 1 --2- 172.21.15.27:0/3255062039 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8a60067170 0x7f8a60069620 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8a780099b0 tx=0x7f8a78016000 comp rx=0 tx=0).stop 2024-01-11T09:46:36.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.726+0000 7f8a843ff700 1 -- 172.21.15.27:0/3255062039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a7c1596f0 msgr2=0x7f8a7c1569c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:36.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.726+0000 7f8a843ff700 1 --2- 172.21.15.27:0/3255062039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a7c1596f0 0x7f8a7c1569c0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f8a7400bb60 tx=0x7f8a74009810 comp rx=0 tx=0).stop 2024-01-11T09:46:36.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.726+0000 7f8a843ff700 1 -- 172.21.15.27:0/3255062039 shutdown_connections 2024-01-11T09:46:36.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.726+0000 7f8a843ff700 1 --2- 172.21.15.27:0/3255062039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a7c1596f0 0x7f8a7c1569c0 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.726+0000 7f8a843ff700 1 --2- 172.21.15.27:0/3255062039 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8a7c068d30 0x7f8a7c16d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.726+0000 7f8a843ff700 1 --2- 172.21.15.27:0/3255062039 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8a60067170 0x7f8a60069620 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.726+0000 7f8a843ff700 1 --2- 172.21.15.27:0/3255062039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a7c0a9de0 0x7f8a7c1489d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.726+0000 7f8a843ff700 1 -- 172.21.15.27:0/3255062039 >> 172.21.15.27:0/3255062039 conn(0x7f8a7c0c9fb0 msgr2=0x7f8a7c0c2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:36.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.726+0000 7f8a843ff700 1 -- 172.21.15.27:0/3255062039 shutdown_connections 2024-01-11T09:46:36.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.726+0000 7f8a843ff700 1 -- 172.21.15.27:0/3255062039 wait complete. 2024-01-11T09:46:36.735 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-11T09:46:36.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.943+0000 7f083e1ff700 1 Processor -- start 2024-01-11T09:46:36.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.944+0000 7f083e1ff700 1 -- start start 2024-01-11T09:46:36.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.945+0000 7f083e1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08381596c0 0x7f083815d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:36.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.945+0000 7f083e1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08380a9cb0 con 0x7f0838068d30 2024-01-11T09:46:36.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.945+0000 7f083e1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083815e210 con 0x7f08380ca670 2024-01-11T09:46:36.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.945+0000 7f083e1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083815dc50 con 0x7f08381596c0 2024-01-11T09:46:36.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.945+0000 7f0837fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0838068d30 0x7f08380aa140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48142/0 (socket says 172.21.15.27:48142) 2024-01-11T09:46:36.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.945+0000 7f0837fff700 1 -- 172.21.15.27:0/1143337799 learned_addr learned my addr 172.21.15.27:0/1143337799 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:36.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.946+0000 7f08377fe700 1 --2- 172.21.15.27:0/1143337799 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08381596c0 0x7f083815d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.946+0000 7f08367fc700 1 -- 172.21.15.27:0/1143337799 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068897554 0 0) 0x7f08380a9cb0 con 0x7f0838068d30 2024-01-11T09:46:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.946+0000 7f08367fc700 1 -- 172.21.15.27:0/1143337799 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0820003570 con 0x7f0838068d30 2024-01-11T09:46:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.946+0000 7f08367fc700 1 -- 172.21.15.27:0/1143337799 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299191853 0 0) 0x7f083815e210 con 0x7f08380ca670 2024-01-11T09:46:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.946+0000 7f08367fc700 1 -- 172.21.15.27:0/1143337799 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08380a9cb0 con 0x7f08380ca670 2024-01-11T09:46:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.946+0000 7f08367fc700 1 -- 172.21.15.27:0/1143337799 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2744366116 0 0) 0x7f0820003570 con 0x7f0838068d30 2024-01-11T09:46:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.946+0000 7f08377fe700 1 -- 172.21.15.27:0/1143337799 >> v1:172.21.15.77:6789/0 conn(0x7f08380ca670 legacy=0x7f0838068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.946+0000 7f08377fe700 1 -- 172.21.15.27:0/1143337799 >> v1:172.21.15.27:6789/0 conn(0x7f0838068d30 legacy=0x7f08380aa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.946+0000 7f08377fe700 1 -- 172.21.15.27:0/1143337799 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083815df30 con 0x7f08381596c0 2024-01-11T09:46:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.946+0000 7f08377fe700 1 --2- 172.21.15.27:0/1143337799 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08381596c0 0x7f083815d300 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0828009910 tx=0x7f08280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a4d2208f2b8cb02 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.947+0000 7f08367fc700 1 -- 172.21.15.27:0/1143337799 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08300017f0 con 0x7f08381596c0 2024-01-11T09:46:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.947+0000 7f08367fc700 1 -- 172.21.15.27:0/1143337799 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0828011420 con 0x7f08381596c0 2024-01-11T09:46:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.947+0000 7f08367fc700 1 -- 172.21.15.27:0/1143337799 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08280117c0 con 0x7f08381596c0 2024-01-11T09:46:36.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.947+0000 7f083e1ff700 1 -- 172.21.15.27:0/1143337799 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08381596c0 msgr2=0x7f083815d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:36.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.947+0000 7f083e1ff700 1 --2- 172.21.15.27:0/1143337799 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08381596c0 0x7f083815d300 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0828009910 tx=0x7f08280091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.947+0000 7f083e1ff700 1 -- 172.21.15.27:0/1143337799 shutdown_connections 2024-01-11T09:46:36.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.947+0000 7f083e1ff700 1 --2- 172.21.15.27:0/1143337799 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08381596c0 0x7f083815d300 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.947+0000 7f083e1ff700 1 -- 172.21.15.27:0/1143337799 >> 172.21.15.27:0/1143337799 conn(0x7f08380c9f70 msgr2=0x7f083806deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:36.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.947+0000 7f083e1ff700 1 -- 172.21.15.27:0/1143337799 shutdown_connections 2024-01-11T09:46:36.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.947+0000 7f083e1ff700 1 -- 172.21.15.27:0/1143337799 wait complete. 2024-01-11T09:46:36.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.949+0000 7f083e1ff700 1 Processor -- start 2024-01-11T09:46:36.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.949+0000 7f083e1ff700 1 -- start start 2024-01-11T09:46:36.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.950+0000 7f083e1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0838068d30 0x7f0838156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:36.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.950+0000 7f083e1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08380ca670 0x7f083816a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:36.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.950+0000 7f083e1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08381596c0 0x7f083816a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:36.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.950+0000 7f083e1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083815b4c0 con 0x7f08380ca670 2024-01-11T09:46:36.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.950+0000 7f083e1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083816ac70 con 0x7f0838068d30 2024-01-11T09:46:36.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.950+0000 7f083e1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083816af50 con 0x7f08381596c0 2024-01-11T09:46:36.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.951+0000 7f0836ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08380ca670 0x7f083816a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:36.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.951+0000 7f08377fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0838068d30 0x7f0838156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:36.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.951+0000 7f08377fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0838068d30 0x7f0838156590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:44514/0 (socket says 172.21.15.27:44514) 2024-01-11T09:46:36.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.951+0000 7f08377fe700 1 -- 172.21.15.27:0/1030672966 learned_addr learned my addr 172.21.15.27:0/1030672966 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:36.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.951+0000 7f0837fff700 1 --2- 172.21.15.27:0/1030672966 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08381596c0 0x7f083816a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:36.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.951+0000 7f0836ffd700 1 -- 172.21.15.27:0/1030672966 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08381596c0 msgr2=0x7f083816a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:36.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.951+0000 7f0836ffd700 1 --2- 172.21.15.27:0/1030672966 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08381596c0 0x7f083816a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.951+0000 7f0836ffd700 1 -- 172.21.15.27:0/1030672966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0838068d30 msgr2=0x7f0838156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:36.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.951+0000 7f0836ffd700 1 --2- 172.21.15.27:0/1030672966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0838068d30 0x7f0838156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:36.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.951+0000 7f0836ffd700 1 -- 172.21.15.27:0/1030672966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0828009420 con 0x7f08380ca670 2024-01-11T09:46:36.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.951+0000 7f0836ffd700 1 --2- 172.21.15.27:0/1030672966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08380ca670 0x7f083816a090 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f08380caa40 tx=0x7f082c007d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:36.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.952+0000 7f081f7fe700 1 -- 172.21.15.27:0/1030672966 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c011590 con 0x7f08380ca670 2024-01-11T09:46:36.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.952+0000 7f081f7fe700 1 -- 172.21.15.27:0/1030672966 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082c0116f0 con 0x7f08380ca670 2024-01-11T09:46:36.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.952+0000 7f083e1ff700 1 -- 172.21.15.27:0/1030672966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083815ae70 con 0x7f08380ca670 2024-01-11T09:46:36.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.952+0000 7f081f7fe700 1 -- 172.21.15.27:0/1030672966 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c011ab0 con 0x7f08380ca670 2024-01-11T09:46:36.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.952+0000 7f083e1ff700 1 -- 172.21.15.27:0/1030672966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08380b1090 con 0x7f08380ca670 2024-01-11T09:46:36.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.953+0000 7f083e1ff700 1 -- 172.21.15.27:0/1030672966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f083815b630 con 0x7f08380ca670 2024-01-11T09:46:36.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.954+0000 7f081f7fe700 1 -- 172.21.15.27:0/1030672966 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f082c01c460 con 0x7f08380ca670 2024-01-11T09:46:36.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.955+0000 7f081f7fe700 1 --2- 172.21.15.27:0/1030672966 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0824067170 0x7f0824069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:36.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.955+0000 7f081f7fe700 1 -- 172.21.15.27:0/1030672966 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f082c084490 con 0x7f08380ca670 2024-01-11T09:46:36.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.956+0000 7f08377fe700 1 --2- 172.21.15.27:0/1030672966 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0824067170 0x7f0824069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:36.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.956+0000 7f08377fe700 1 --2- 172.21.15.27:0/1030672966 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0824067170 0x7f0824069620 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0828009860 tx=0x7f0828016000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:36.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:36.960+0000 7f081f7fe700 1 -- 172.21.15.27:0/1030672966 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f082c0513e0 con 0x7f08380ca670 2024-01-11T09:46:37.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.168+0000 7f083e1ff700 1 -- 172.21.15.27:0/1030672966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f08380718d0 con 0x7f08380ca670 2024-01-11T09:46:37.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.169+0000 7f081f7fe700 1 -- 172.21.15.27:0/1030672966 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v235) v1 ==== 84+0+209 (secure 0 0 0) 0x7f082c054a00 con 0x7f08380ca670 2024-01-11T09:46:37.160 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:37.160 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T09:46:37.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:37.161 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-11T09:46:37.161 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-11T09:46:37.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:37.161 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:37.161 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 30, 2024-01-11T09:46:37.161 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": "cephfs_metadata" 2024-01-11T09:46:37.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:37.161 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:37.161 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 31, 2024-01-11T09:46:37.161 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": "cephfs_data" 2024-01-11T09:46:37.161 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:37.161 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T09:46:37.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.171+0000 7f083e1ff700 1 -- 172.21.15.27:0/1030672966 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0824067170 msgr2=0x7f0824069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:37.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.171+0000 7f083e1ff700 1 --2- 172.21.15.27:0/1030672966 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0824067170 0x7f0824069620 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0828009860 tx=0x7f0828016000 comp rx=0 tx=0).stop 2024-01-11T09:46:37.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.171+0000 7f083e1ff700 1 -- 172.21.15.27:0/1030672966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08380ca670 msgr2=0x7f083816a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:37.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.171+0000 7f083e1ff700 1 --2- 172.21.15.27:0/1030672966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08380ca670 0x7f083816a090 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f08380caa40 tx=0x7f082c007d50 comp rx=0 tx=0).stop 2024-01-11T09:46:37.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.171+0000 7f083e1ff700 1 -- 172.21.15.27:0/1030672966 shutdown_connections 2024-01-11T09:46:37.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.171+0000 7f083e1ff700 1 --2- 172.21.15.27:0/1030672966 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f08381596c0 0x7f083816a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:37.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.171+0000 7f083e1ff700 1 --2- 172.21.15.27:0/1030672966 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0824067170 0x7f0824069620 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:37.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.171+0000 7f083e1ff700 1 --2- 172.21.15.27:0/1030672966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0838068d30 0x7f0838156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:37.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.171+0000 7f083e1ff700 1 --2- 172.21.15.27:0/1030672966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08380ca670 0x7f083816a090 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:37.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.171+0000 7f083e1ff700 1 -- 172.21.15.27:0/1030672966 >> 172.21.15.27:0/1030672966 conn(0x7f08380c9f70 msgr2=0x7f0838158460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:37.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.171+0000 7f083e1ff700 1 -- 172.21.15.27:0/1030672966 shutdown_connections 2024-01-11T09:46:37.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.171+0000 7f083e1ff700 1 -- 172.21.15.27:0/1030672966 wait complete. 2024-01-11T09:46:37.180 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2024-01-11T09:46:37.181 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2024-01-11T09:46:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.392+0000 7f7ebebff700 1 Processor -- start 2024-01-11T09:46:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.392+0000 7f7ebebff700 1 -- start start 2024-01-11T09:46:37.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.393+0000 7f7ebebff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7eb8155ba0 0x7f7eb806ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:37.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.393+0000 7f7ebebff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eb80c6f30 con 0x7f7eb806e310 2024-01-11T09:46:37.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.393+0000 7f7ebebff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eb806e860 con 0x7f7eb8069d70 2024-01-11T09:46:37.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.393+0000 7f7ebebff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eb80c68e0 con 0x7f7eb8155ba0 2024-01-11T09:46:37.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.394+0000 7f7ebc99b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7eb8155ba0 0x7f7eb806ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:37.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.394+0000 7f7eb7fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7eb806e310 0x7f7eb8069660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48148/0 (socket says 172.21.15.27:48148) 2024-01-11T09:46:37.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.394+0000 7f7eb7fff700 1 -- 172.21.15.27:0/2937072323 learned_addr learned my addr 172.21.15.27:0/2937072323 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:37.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.394+0000 7f7ebc99b700 1 -- 172.21.15.27:0/2937072323 >> v1:172.21.15.77:6789/0 conn(0x7f7eb8069d70 legacy=0x7f7eb80c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:37.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.394+0000 7f7ebc99b700 1 -- 172.21.15.27:0/2937072323 >> v1:172.21.15.27:6789/0 conn(0x7f7eb806e310 legacy=0x7f7eb8069660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:37.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.394+0000 7f7eb77fe700 1 -- 172.21.15.27:0/2937072323 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071204082 0 0) 0x7f7eb80c6f30 con 0x7f7eb806e310 2024-01-11T09:46:37.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.394+0000 7f7ebc99b700 1 -- 172.21.15.27:0/2937072323 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb80c6bc0 con 0x7f7eb8155ba0 2024-01-11T09:46:37.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.395+0000 7f7ebc99b700 1 --2- 172.21.15.27:0/2937072323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb8155ba0 0x7f7eb806ddd0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f7eac0098d0 tx=0x7f7eac009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b981f705212132a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:37.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.395+0000 7f7eb77fe700 1 -- 172.21.15.27:0/2937072323 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac009e70 con 0x7f7eb8155ba0 2024-01-11T09:46:37.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.395+0000 7f7eb77fe700 1 -- 172.21.15.27:0/2937072323 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eac005c10 con 0x7f7eb8155ba0 2024-01-11T09:46:37.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.395+0000 7f7eb77fe700 1 -- 172.21.15.27:0/2937072323 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac00b650 con 0x7f7eb8155ba0 2024-01-11T09:46:37.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.395+0000 7f7ebebff700 1 -- 172.21.15.27:0/2937072323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb8155ba0 msgr2=0x7f7eb806ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:37.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.395+0000 7f7ebebff700 1 --2- 172.21.15.27:0/2937072323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb8155ba0 0x7f7eb806ddd0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f7eac0098d0 tx=0x7f7eac009120 comp rx=0 tx=0).stop 2024-01-11T09:46:37.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.395+0000 7f7ebebff700 1 -- 172.21.15.27:0/2937072323 shutdown_connections 2024-01-11T09:46:37.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.395+0000 7f7ebebff700 1 --2- 172.21.15.27:0/2937072323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb8155ba0 0x7f7eb806ddd0 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:37.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.395+0000 7f7ebebff700 1 -- 172.21.15.27:0/2937072323 >> 172.21.15.27:0/2937072323 conn(0x7f7eb80680e0 msgr2=0x7f7eb80cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:37.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.396+0000 7f7ebebff700 1 -- 172.21.15.27:0/2937072323 shutdown_connections 2024-01-11T09:46:37.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.396+0000 7f7ebebff700 1 -- 172.21.15.27:0/2937072323 wait complete. 2024-01-11T09:46:37.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.397+0000 7f7ebebff700 1 Processor -- start 2024-01-11T09:46:37.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.397+0000 7f7ebebff700 1 -- start start 2024-01-11T09:46:37.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.398+0000 7f7ebebff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7eb8069d70 0x7f7eb80c3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:37.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.398+0000 7f7ebebff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb806e310 0x7f7eb80ad250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:37.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.398+0000 7f7ebebff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7eb8155ba0 0x7f7eb80ad790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:37.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.398+0000 7f7ebebff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb8169fb0 con 0x7f7eb806e310 2024-01-11T09:46:37.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.398+0000 7f7ebebff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb80c4680 con 0x7f7eb8155ba0 2024-01-11T09:46:37.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.398+0000 7f7ebebff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb80c4960 con 0x7f7eb8069d70 2024-01-11T09:46:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.398+0000 7f7eb7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb806e310 0x7f7eb80ad250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.399+0000 7f7eb7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb806e310 0x7f7eb80ad250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40330/0 (socket says 172.21.15.27:40330) 2024-01-11T09:46:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.399+0000 7f7eb7fff700 1 -- 172.21.15.27:0/1425314390 learned_addr learned my addr 172.21.15.27:0/1425314390 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.399+0000 7f7ebc99b700 1 --2- 172.21.15.27:0/1425314390 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7eb8069d70 0x7f7eb80c3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.399+0000 7f7ebd19c700 1 --2- 172.21.15.27:0/1425314390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7eb8155ba0 0x7f7eb80ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.399+0000 7f7eb7fff700 1 -- 172.21.15.27:0/1425314390 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7eb8069d70 msgr2=0x7f7eb80c3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.399+0000 7f7eb7fff700 1 --2- 172.21.15.27:0/1425314390 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7eb8069d70 0x7f7eb80c3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.399+0000 7f7eb7fff700 1 -- 172.21.15.27:0/1425314390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7eb8155ba0 msgr2=0x7f7eb80ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.399+0000 7f7eb7fff700 1 --2- 172.21.15.27:0/1425314390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7eb8155ba0 0x7f7eb80ad790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.399+0000 7f7eb7fff700 1 -- 172.21.15.27:0/1425314390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eac0093c0 con 0x7f7eb806e310 2024-01-11T09:46:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.399+0000 7f7ebd19c700 1 --2- 172.21.15.27:0/1425314390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7eb8155ba0 0x7f7eb80ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.399+0000 7f7eb7fff700 1 --2- 172.21.15.27:0/1425314390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb806e310 0x7f7eb80ad250 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f7ea400c810 tx=0x7f7ea4007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.399+0000 7f7eb4ff9700 1 -- 172.21.15.27:0/1425314390 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea4005c00 con 0x7f7eb806e310 2024-01-11T09:46:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.399+0000 7f7eb4ff9700 1 -- 172.21.15.27:0/1425314390 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea4005d60 con 0x7f7eb806e310 2024-01-11T09:46:37.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.399+0000 7f7eb4ff9700 1 -- 172.21.15.27:0/1425314390 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea4007320 con 0x7f7eb806e310 2024-01-11T09:46:37.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.399+0000 7f7ebebff700 1 -- 172.21.15.27:0/1425314390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb80c4c40 con 0x7f7eb806e310 2024-01-11T09:46:37.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.400+0000 7f7ebebff700 1 -- 172.21.15.27:0/1425314390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb80c4f20 con 0x7f7eb806e310 2024-01-11T09:46:37.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.400+0000 7f7ebebff700 1 -- 172.21.15.27:0/1425314390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb816a160 con 0x7f7eb806e310 2024-01-11T09:46:37.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.402+0000 7f7eb4ff9700 1 -- 172.21.15.27:0/1425314390 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f7ea401b3b0 con 0x7f7eb806e310 2024-01-11T09:46:37.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.403+0000 7f7eb4ff9700 1 --2- 172.21.15.27:0/1425314390 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e9c067170 0x7f7e9c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:37.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.403+0000 7f7eb4ff9700 1 -- 172.21.15.27:0/1425314390 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f7ea4083e30 con 0x7f7eb806e310 2024-01-11T09:46:37.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.403+0000 7f7ebc99b700 1 --2- 172.21.15.27:0/1425314390 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e9c067170 0x7f7e9c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:37.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.404+0000 7f7ebc99b700 1 --2- 172.21.15.27:0/1425314390 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e9c067170 0x7f7e9c069620 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7eac009390 tx=0x7f7eac017000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:37.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.407+0000 7f7eb4ff9700 1 -- 172.21.15.27:0/1425314390 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7eb816a160 con 0x7f7eb806e310 2024-01-11T09:46:37.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.608+0000 7f7ebebff700 1 -- 172.21.15.27:0/1425314390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7eb804c3c0 con 0x7f7eb806e310 2024-01-11T09:46:37.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.962+0000 7f7eb4ff9700 1 -- 172.21.15.27:0/1425314390 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v236) v1 ==== 144+0+0 (secure 0 0 0) 0x7f7ea4017070 con 0x7f7eb806e310 2024-01-11T09:46:37.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.966+0000 7f7ebebff700 1 -- 172.21.15.27:0/1425314390 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e9c067170 msgr2=0x7f7e9c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:37.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.966+0000 7f7ebebff700 1 --2- 172.21.15.27:0/1425314390 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e9c067170 0x7f7e9c069620 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7eac009390 tx=0x7f7eac017000 comp rx=0 tx=0).stop 2024-01-11T09:46:37.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.966+0000 7f7ebebff700 1 -- 172.21.15.27:0/1425314390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb806e310 msgr2=0x7f7eb80ad250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:37.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.966+0000 7f7ebebff700 1 --2- 172.21.15.27:0/1425314390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb806e310 0x7f7eb80ad250 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f7ea400c810 tx=0x7f7ea4007d90 comp rx=0 tx=0).stop 2024-01-11T09:46:37.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.967+0000 7f7ebebff700 1 -- 172.21.15.27:0/1425314390 shutdown_connections 2024-01-11T09:46:37.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.967+0000 7f7ebebff700 1 --2- 172.21.15.27:0/1425314390 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7e9c067170 0x7f7e9c069620 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:37.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.967+0000 7f7ebebff700 1 --2- 172.21.15.27:0/1425314390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7eb8155ba0 0x7f7eb80ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:37.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.967+0000 7f7ebebff700 1 --2- 172.21.15.27:0/1425314390 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7eb8069d70 0x7f7eb80c3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:37.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.967+0000 7f7ebebff700 1 --2- 172.21.15.27:0/1425314390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb806e310 0x7f7eb80ad250 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:37.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.967+0000 7f7ebebff700 1 -- 172.21.15.27:0/1425314390 >> 172.21.15.27:0/1425314390 conn(0x7f7eb80680e0 msgr2=0x7f7eb80c73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:37.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.967+0000 7f7ebebff700 1 -- 172.21.15.27:0/1425314390 shutdown_connections 2024-01-11T09:46:37.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:37.968+0000 7f7ebebff700 1 -- 172.21.15.27:0/1425314390 wait complete. 2024-01-11T09:46:37.964 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'backup_fs_metadata' created 2024-01-11T09:46:37.982 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-11T09:46:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.198+0000 7f18e6dff700 1 Processor -- start 2024-01-11T09:46:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.198+0000 7f18e6dff700 1 -- start start 2024-01-11T09:46:38.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.199+0000 7f18e6dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18e00b03e0 0x7f18e00b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:38.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.199+0000 7f18e6dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18e00b07b0 0x7f18e00b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:38.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.199+0000 7f18e6dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f18e0159200 0x7f18e0159640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:38.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.199+0000 7f18e6dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e00cd730 con 0x7f18e00b03e0 2024-01-11T09:46:38.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.199+0000 7f18e6dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e00ccb20 con 0x7f18e0159200 2024-01-11T09:46:38.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.199+0000 7f18e6dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e00cce00 con 0x7f18e00b07b0 2024-01-11T09:46:38.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.199+0000 7f18e4b9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18e00b03e0 0x7f18e00b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:38.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.199+0000 7f18e4b9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18e00b03e0 0x7f18e00b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40344/0 (socket says 172.21.15.27:40344) 2024-01-11T09:46:38.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.199+0000 7f18e4b9b700 1 -- 172.21.15.27:0/425448511 learned_addr learned my addr 172.21.15.27:0/425448511 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:38.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.200+0000 7f18e539c700 1 --2- 172.21.15.27:0/425448511 >> v2:172.21.15.77:3300/0 conn(0x7f18e0159200 0x7f18e0159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.200+0000 7f18dffff700 1 --2- 172.21.15.27:0/425448511 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18e00b07b0 0x7f18e00b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.200+0000 7f18e4b9b700 1 -- 172.21.15.27:0/425448511 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18e00b07b0 msgr2=0x7f18e00b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.200+0000 7f18e4b9b700 1 --2- 172.21.15.27:0/425448511 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18e00b07b0 0x7f18e00b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.200+0000 7f18e4b9b700 1 -- 172.21.15.27:0/425448511 >> v2:172.21.15.77:3300/0 conn(0x7f18e0159200 msgr2=0x7f18e0159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.200+0000 7f18e4b9b700 1 --2- 172.21.15.27:0/425448511 >> v2:172.21.15.77:3300/0 conn(0x7f18e0159200 0x7f18e0159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.200+0000 7f18e4b9b700 1 -- 172.21.15.27:0/425448511 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e00cd0e0 con 0x7f18e00b03e0 2024-01-11T09:46:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.200+0000 7f18e539c700 1 --2- 172.21.15.27:0/425448511 >> v2:172.21.15.77:3300/0 conn(0x7f18e0159200 0x7f18e0159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.200+0000 7f18e4b9b700 1 --2- 172.21.15.27:0/425448511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18e00b03e0 0x7f18e00b2480 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f18e00b5f00 tx=0x7f18d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb21d620485e0892 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.200+0000 7f18df7fe700 1 -- 172.21.15.27:0/425448511 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d400b860 con 0x7f18e00b03e0 2024-01-11T09:46:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.201+0000 7f18df7fe700 1 -- 172.21.15.27:0/425448511 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d400b9c0 con 0x7f18e00b03e0 2024-01-11T09:46:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.201+0000 7f18df7fe700 1 -- 172.21.15.27:0/425448511 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d400bd60 con 0x7f18e00b03e0 2024-01-11T09:46:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.201+0000 7f18e6dff700 1 -- 172.21.15.27:0/425448511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18e00b03e0 msgr2=0x7f18e00b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.201+0000 7f18e6dff700 1 --2- 172.21.15.27:0/425448511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18e00b03e0 0x7f18e00b2480 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f18e00b5f00 tx=0x7f18d4009250 comp rx=0 tx=0).stop 2024-01-11T09:46:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.201+0000 7f18e6dff700 1 -- 172.21.15.27:0/425448511 shutdown_connections 2024-01-11T09:46:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.201+0000 7f18e6dff700 1 --2- 172.21.15.27:0/425448511 >> v2:172.21.15.77:3300/0 conn(0x7f18e0159200 0x7f18e0159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.201+0000 7f18e6dff700 1 --2- 172.21.15.27:0/425448511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18e00b03e0 0x7f18e00b2480 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.201+0000 7f18e6dff700 1 --2- 172.21.15.27:0/425448511 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18e00b07b0 0x7f18e00b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.201+0000 7f18e6dff700 1 -- 172.21.15.27:0/425448511 >> 172.21.15.27:0/425448511 conn(0x7f18e006ed40 msgr2=0x7f18e006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.201+0000 7f18e6dff700 1 -- 172.21.15.27:0/425448511 shutdown_connections 2024-01-11T09:46:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.201+0000 7f18e6dff700 1 -- 172.21.15.27:0/425448511 wait complete. 2024-01-11T09:46:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.203+0000 7f18e6dff700 1 Processor -- start 2024-01-11T09:46:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.203+0000 7f18e6dff700 1 -- start start 2024-01-11T09:46:38.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.204+0000 7f18e6dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18e00b03e0 0x7f18e00c8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:38.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.204+0000 7f18e6dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18e00b07b0 0x7f18e015b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:38.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.204+0000 7f18e6dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18e0159200 0x7f18e015b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:38.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.204+0000 7f18e6dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e00c9da0 con 0x7f18e0159200 2024-01-11T09:46:38.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.204+0000 7f18e6dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e0146660 con 0x7f18e00b07b0 2024-01-11T09:46:38.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.204+0000 7f18e6dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e00c9190 con 0x7f18e00b03e0 2024-01-11T09:46:38.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.204+0000 7f18e539c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18e0159200 0x7f18e015b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:38.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.204+0000 7f18e539c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18e0159200 0x7f18e015b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40356/0 (socket says 172.21.15.27:40356) 2024-01-11T09:46:38.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.204+0000 7f18e539c700 1 -- 172.21.15.27:0/1546921592 learned_addr learned my addr 172.21.15.27:0/1546921592 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:38.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.204+0000 7f18e4b9b700 1 --2- 172.21.15.27:0/1546921592 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18e00b03e0 0x7f18e00c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:38.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.204+0000 7f18dffff700 1 --2- 172.21.15.27:0/1546921592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18e00b07b0 0x7f18e015b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:38.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.205+0000 7f18e539c700 1 -- 172.21.15.27:0/1546921592 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18e00b03e0 msgr2=0x7f18e00c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.205+0000 7f18e539c700 1 --2- 172.21.15.27:0/1546921592 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18e00b03e0 0x7f18e00c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.205+0000 7f18e539c700 1 -- 172.21.15.27:0/1546921592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18e00b07b0 msgr2=0x7f18e015b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.205+0000 7f18e539c700 1 --2- 172.21.15.27:0/1546921592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18e00b07b0 0x7f18e015b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.205+0000 7f18e539c700 1 -- 172.21.15.27:0/1546921592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d40094f0 con 0x7f18e0159200 2024-01-11T09:46:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.205+0000 7f18e539c700 1 --2- 172.21.15.27:0/1546921592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18e0159200 0x7f18e015b9d0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f18d800d180 tx=0x7f18d8009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.205+0000 7f18dcff9700 1 -- 172.21.15.27:0/1546921592 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d800dcc0 con 0x7f18e0159200 2024-01-11T09:46:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.205+0000 7f18dcff9700 1 -- 172.21.15.27:0/1546921592 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d800f040 con 0x7f18e0159200 2024-01-11T09:46:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.205+0000 7f18e6dff700 1 -- 172.21.15.27:0/1546921592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e00c9470 con 0x7f18e0159200 2024-01-11T09:46:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.205+0000 7f18dcff9700 1 -- 172.21.15.27:0/1546921592 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d8015690 con 0x7f18e0159200 2024-01-11T09:46:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.205+0000 7f18e6dff700 1 -- 172.21.15.27:0/1546921592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e00c9a30 con 0x7f18e0159200 2024-01-11T09:46:38.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.206+0000 7f18e6dff700 1 -- 172.21.15.27:0/1546921592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e00c9f50 con 0x7f18e0159200 2024-01-11T09:46:38.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.208+0000 7f18dcff9700 1 -- 172.21.15.27:0/1546921592 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f18d801f440 con 0x7f18e0159200 2024-01-11T09:46:38.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.209+0000 7f18dcff9700 1 --2- 172.21.15.27:0/1546921592 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f18c4067170 0x7f18c4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:38.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.209+0000 7f18dcff9700 1 -- 172.21.15.27:0/1546921592 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6190+0+0 (secure 0 0 0) 0x7f18d80852d0 con 0x7f18e0159200 2024-01-11T09:46:38.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.209+0000 7f18e4b9b700 1 --2- 172.21.15.27:0/1546921592 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f18c4067170 0x7f18c4069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:38.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.210+0000 7f18e4b9b700 1 --2- 172.21.15.27:0/1546921592 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f18c4067170 0x7f18c4069620 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f18e00b5f00 tx=0x7f18d4017000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:38.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.213+0000 7f18dcff9700 1 -- 172.21.15.27:0/1546921592 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f18d8052040 con 0x7f18e0159200 2024-01-11T09:46:38.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.415+0000 7f18e6dff700 1 -- 172.21.15.27:0/1546921592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f18e004c3c0 con 0x7f18e0159200 2024-01-11T09:46:38.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.984+0000 7f18dcff9700 1 -- 172.21.15.27:0/1546921592 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v237) v1 ==== 176+0+0 (secure 0 0 0) 0x7f18d801a0d0 con 0x7f18e0159200 2024-01-11T09:46:38.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.990+0000 7f18e6dff700 1 -- 172.21.15.27:0/1546921592 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f18c4067170 msgr2=0x7f18c4069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:38.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.990+0000 7f18e6dff700 1 --2- 172.21.15.27:0/1546921592 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f18c4067170 0x7f18c4069620 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f18e00b5f00 tx=0x7f18d4017000 comp rx=0 tx=0).stop 2024-01-11T09:46:38.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.990+0000 7f18e6dff700 1 -- 172.21.15.27:0/1546921592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18e0159200 msgr2=0x7f18e015b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:38.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.990+0000 7f18e6dff700 1 --2- 172.21.15.27:0/1546921592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18e0159200 0x7f18e015b9d0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f18d800d180 tx=0x7f18d8009a50 comp rx=0 tx=0).stop 2024-01-11T09:46:38.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.991+0000 7f18e6dff700 1 -- 172.21.15.27:0/1546921592 shutdown_connections 2024-01-11T09:46:38.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.991+0000 7f18e6dff700 1 --2- 172.21.15.27:0/1546921592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18e0159200 0x7f18e015b9d0 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:38.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.991+0000 7f18e6dff700 1 --2- 172.21.15.27:0/1546921592 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f18e00b03e0 0x7f18e00c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:38.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.991+0000 7f18e6dff700 1 --2- 172.21.15.27:0/1546921592 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f18c4067170 0x7f18c4069620 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:38.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.991+0000 7f18e6dff700 1 --2- 172.21.15.27:0/1546921592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18e00b07b0 0x7f18e015b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:38.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.991+0000 7f18e6dff700 1 -- 172.21.15.27:0/1546921592 >> 172.21.15.27:0/1546921592 conn(0x7f18e006ed40 msgr2=0x7f18e00ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:38.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.992+0000 7f18e6dff700 1 -- 172.21.15.27:0/1546921592 shutdown_connections 2024-01-11T09:46:38.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:38.993+0000 7f18e6dff700 1 -- 172.21.15.27:0/1546921592 wait complete. 2024-01-11T09:46:38.988 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'backup_fs_data' created 2024-01-11T09:46:39.008 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2024-01-11T09:46:39.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.219+0000 7fc4575ff700 1 Processor -- start 2024-01-11T09:46:39.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.219+0000 7fc4575ff700 1 -- start start 2024-01-11T09:46:39.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.220+0000 7fc4575ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc450142dc0 0x7fc4500cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:39.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.220+0000 7fc4575ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc450143190 0x7fc4500cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:39.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.221+0000 7fc4575ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45015d320 con 0x7fc4500cc020 2024-01-11T09:46:39.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.221+0000 7fc4575ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc45015df30 con 0x7fc450142dc0 2024-01-11T09:46:39.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.221+0000 7fc4575ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45015d590 con 0x7fc450143190 2024-01-11T09:46:39.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.221+0000 7fc45539b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc450142dc0 0x7fc4500cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:39.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.221+0000 7fc45539b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc450142dc0 0x7fc4500cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40370/0 (socket says 172.21.15.27:40370) 2024-01-11T09:46:39.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.221+0000 7fc45539b700 1 -- 172.21.15.27:0/2403078583 learned_addr learned my addr 172.21.15.27:0/2403078583 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:39.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.221+0000 7fc454b9a700 1 --2- 172.21.15.27:0/2403078583 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc450143190 0x7fc4500cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:39.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.221+0000 7fc45539b700 1 -- 172.21.15.27:0/2403078583 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc450143190 msgr2=0x7fc4500cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:39.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.221+0000 7fc45539b700 1 --2- 172.21.15.27:0/2403078583 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc450143190 0x7fc4500cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:39.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.221+0000 7fc45539b700 1 -- 172.21.15.27:0/2403078583 >> v1:172.21.15.77:6789/0 conn(0x7fc4500cc020 legacy=0x7fc4500cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:39.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.221+0000 7fc45539b700 1 -- 172.21.15.27:0/2403078583 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45015d870 con 0x7fc450142dc0 2024-01-11T09:46:39.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.221+0000 7fc447fff700 1 -- 172.21.15.27:0/2403078583 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367941973 0 0) 0x7fc45015d320 con 0x7fc4500cc020 2024-01-11T09:46:39.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.222+0000 7fc45539b700 1 --2- 172.21.15.27:0/2403078583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc450142dc0 0x7fc4500cada0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fc44c0099b0 tx=0x7fc44c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fefa4b7ec8ffa764 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:39.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.222+0000 7fc447fff700 1 -- 172.21.15.27:0/2403078583 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c009ea0 con 0x7fc450142dc0 2024-01-11T09:46:39.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.222+0000 7fc447fff700 1 -- 172.21.15.27:0/2403078583 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44c00b860 con 0x7fc450142dc0 2024-01-11T09:46:39.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.222+0000 7fc447fff700 1 -- 172.21.15.27:0/2403078583 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c00bc00 con 0x7fc450142dc0 2024-01-11T09:46:39.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.223+0000 7fc4575ff700 1 -- 172.21.15.27:0/2403078583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc450142dc0 msgr2=0x7fc4500cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:39.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.223+0000 7fc4575ff700 1 --2- 172.21.15.27:0/2403078583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc450142dc0 0x7fc4500cada0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fc44c0099b0 tx=0x7fc44c009250 comp rx=0 tx=0).stop 2024-01-11T09:46:39.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.223+0000 7fc4575ff700 1 -- 172.21.15.27:0/2403078583 shutdown_connections 2024-01-11T09:46:39.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.223+0000 7fc4575ff700 1 --2- 172.21.15.27:0/2403078583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc450142dc0 0x7fc4500cada0 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:39.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.223+0000 7fc4575ff700 1 --2- 172.21.15.27:0/2403078583 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc450143190 0x7fc4500cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:39.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.223+0000 7fc4575ff700 1 -- 172.21.15.27:0/2403078583 >> 172.21.15.27:0/2403078583 conn(0x7fc450063ca0 msgr2=0x7fc45006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:39.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.223+0000 7fc4575ff700 1 -- 172.21.15.27:0/2403078583 shutdown_connections 2024-01-11T09:46:39.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.223+0000 7fc4575ff700 1 -- 172.21.15.27:0/2403078583 wait complete. 2024-01-11T09:46:39.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.224+0000 7fc4575ff700 1 Processor -- start 2024-01-11T09:46:39.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.224+0000 7fc4575ff700 1 -- start start 2024-01-11T09:46:39.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.225+0000 7fc4575ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4500cc020 0x7fc4500c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:39.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.225+0000 7fc4575ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc450142dc0 0x7fc4501596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:39.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.225+0000 7fc4575ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc450143190 0x7fc4500c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:39.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.225+0000 7fc4575ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4500c96d0 con 0x7fc450143190 2024-01-11T09:46:39.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.225+0000 7fc4575ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45016c150 con 0x7fc450142dc0 2024-01-11T09:46:39.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.225+0000 7fc4575ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45016b540 con 0x7fc4500cc020 2024-01-11T09:46:39.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.226+0000 7fc455b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc450143190 0x7fc4500c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:39.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.226+0000 7fc455b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc450143190 0x7fc4500c5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40380/0 (socket says 172.21.15.27:40380) 2024-01-11T09:46:39.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.226+0000 7fc455b9c700 1 -- 172.21.15.27:0/3604468939 learned_addr learned my addr 172.21.15.27:0/3604468939 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:39.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.226+0000 7fc45539b700 1 --2- 172.21.15.27:0/3604468939 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4500cc020 0x7fc4500c68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:39.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.226+0000 7fc454b9a700 1 --2- 172.21.15.27:0/3604468939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc450142dc0 0x7fc4501596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:39.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.226+0000 7fc455b9c700 1 -- 172.21.15.27:0/3604468939 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4500cc020 msgr2=0x7fc4500c68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:39.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.226+0000 7fc455b9c700 1 --2- 172.21.15.27:0/3604468939 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4500cc020 0x7fc4500c68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:39.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.226+0000 7fc455b9c700 1 -- 172.21.15.27:0/3604468939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc450142dc0 msgr2=0x7fc4501596d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:39.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.226+0000 7fc455b9c700 1 --2- 172.21.15.27:0/3604468939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc450142dc0 0x7fc4501596d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:39.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.226+0000 7fc455b9c700 1 -- 172.21.15.27:0/3604468939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44c0094f0 con 0x7fc450143190 2024-01-11T09:46:39.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.226+0000 7fc45539b700 1 --2- 172.21.15.27:0/3604468939 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4500cc020 0x7fc4500c68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:39.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.226+0000 7fc455b9c700 1 --2- 172.21.15.27:0/3604468939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc450143190 0x7fc4500c5ea0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fc448001560 tx=0x7fc448001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:39.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.227+0000 7fc4457fa700 1 -- 172.21.15.27:0/3604468939 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc448003030 con 0x7fc450143190 2024-01-11T09:46:39.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.227+0000 7fc4457fa700 1 -- 172.21.15.27:0/3604468939 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44800a7e0 con 0x7fc450143190 2024-01-11T09:46:39.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.227+0000 7fc4457fa700 1 -- 172.21.15.27:0/3604468939 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44800aba0 con 0x7fc450143190 2024-01-11T09:46:39.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.227+0000 7fc4575ff700 1 -- 172.21.15.27:0/3604468939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45016b820 con 0x7fc450143190 2024-01-11T09:46:39.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.227+0000 7fc4575ff700 1 -- 172.21.15.27:0/3604468939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45016bde0 con 0x7fc450143190 2024-01-11T09:46:39.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.228+0000 7fc4575ff700 1 -- 172.21.15.27:0/3604468939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45004c3c0 con 0x7fc450143190 2024-01-11T09:46:39.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.229+0000 7fc4457fa700 1 -- 172.21.15.27:0/3604468939 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fc448012420 con 0x7fc450143190 2024-01-11T09:46:39.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.230+0000 7fc4457fa700 1 --2- 172.21.15.27:0/3604468939 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc434067170 0x7fc434069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:39.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.230+0000 7fc4457fa700 1 -- 172.21.15.27:0/3604468939 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6575+0+0 (secure 0 0 0) 0x7fc448004070 con 0x7fc450143190 2024-01-11T09:46:39.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.230+0000 7fc45539b700 1 --2- 172.21.15.27:0/3604468939 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc434067170 0x7fc434069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:39.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.231+0000 7fc45539b700 1 --2- 172.21.15.27:0/3604468939 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc434067170 0x7fc434069620 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc44c0099b0 tx=0x7fc44c017000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:39.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.235+0000 7fc4457fa700 1 -- 172.21.15.27:0/3604468939 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc448051780 con 0x7fc450143190 2024-01-11T09:46:39.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.481+0000 7fc4575ff700 1 -- 172.21.15.27:0/3604468939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7fc4500cbdb0 con 0x7fc450143190 2024-01-11T09:46:39.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.978+0000 7fc4457fa700 1 -- 172.21.15.27:0/3604468939 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 32 and data pool 33 v129) v1 ==== 183+0+0 (secure 0 0 0) 0x7fc448054da0 con 0x7fc450143190 2024-01-11T09:46:39.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.988+0000 7fc4575ff700 1 -- 172.21.15.27:0/3604468939 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc434067170 msgr2=0x7fc434069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:39.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.988+0000 7fc4575ff700 1 --2- 172.21.15.27:0/3604468939 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc434067170 0x7fc434069620 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc44c0099b0 tx=0x7fc44c017000 comp rx=0 tx=0).stop 2024-01-11T09:46:39.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.988+0000 7fc4575ff700 1 -- 172.21.15.27:0/3604468939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc450143190 msgr2=0x7fc4500c5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:39.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.988+0000 7fc4575ff700 1 --2- 172.21.15.27:0/3604468939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc450143190 0x7fc4500c5ea0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fc448001560 tx=0x7fc448001770 comp rx=0 tx=0).stop 2024-01-11T09:46:39.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.989+0000 7fc4575ff700 1 -- 172.21.15.27:0/3604468939 shutdown_connections 2024-01-11T09:46:39.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.989+0000 7fc4575ff700 1 --2- 172.21.15.27:0/3604468939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc450143190 0x7fc4500c5ea0 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:39.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.989+0000 7fc4575ff700 1 --2- 172.21.15.27:0/3604468939 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4500cc020 0x7fc4500c68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:39.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.989+0000 7fc4575ff700 1 --2- 172.21.15.27:0/3604468939 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc434067170 0x7fc434069620 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:39.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.989+0000 7fc4575ff700 1 --2- 172.21.15.27:0/3604468939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc450142dc0 0x7fc4501596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:39.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.989+0000 7fc4575ff700 1 -- 172.21.15.27:0/3604468939 >> 172.21.15.27:0/3604468939 conn(0x7fc450063ca0 msgr2=0x7fc450141fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:39.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.990+0000 7fc4575ff700 1 -- 172.21.15.27:0/3604468939 shutdown_connections 2024-01-11T09:46:39.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:39.990+0000 7fc4575ff700 1 -- 172.21.15.27:0/3604468939 wait complete. 2024-01-11T09:46:39.991 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 32 and data pool 33 2024-01-11T09:46:40.021 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:46:40.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.232+0000 7fcebb9ff700 1 Processor -- start 2024-01-11T09:46:40.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.232+0000 7fcebb9ff700 1 -- start start 2024-01-11T09:46:40.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.233+0000 7fcebb9ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fceb41575e0 0x7fceb4158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:40.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.233+0000 7fcebb9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fceb406b610 0x7fceb40b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:40.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.233+0000 7fcebb9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb4158b60 con 0x7fceb40b1800 2024-01-11T09:46:40.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.233+0000 7fcebb9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb40cb5e0 con 0x7fceb406b610 2024-01-11T09:46:40.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.233+0000 7fcebb9ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb41599b0 con 0x7fceb41575e0 2024-01-11T09:46:40.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.233+0000 7fceb8f9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fceb406b610 0x7fceb40b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:40.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.233+0000 7fceb9f9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fceb40b1800 0x7fceb40ca1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48162/0 (socket says 172.21.15.27:48162) 2024-01-11T09:46:40.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.233+0000 7fceb9f9c700 1 -- 172.21.15.27:0/180776849 learned_addr learned my addr 172.21.15.27:0/180776849 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:40.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.234+0000 7fceb979b700 1 --2- 172.21.15.27:0/180776849 >> v2:172.21.15.77:3300/0 conn(0x7fceb41575e0 0x7fceb4158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:40.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.234+0000 7fceb8f9a700 1 -- 172.21.15.27:0/180776849 >> v2:172.21.15.77:3300/0 conn(0x7fceb41575e0 msgr2=0x7fceb4158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:40.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.234+0000 7fceb8f9a700 1 --2- 172.21.15.27:0/180776849 >> v2:172.21.15.77:3300/0 conn(0x7fceb41575e0 0x7fceb4158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:40.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.234+0000 7fceb8f9a700 1 -- 172.21.15.27:0/180776849 >> v1:172.21.15.27:6789/0 conn(0x7fceb40b1800 legacy=0x7fceb40ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:40.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.234+0000 7fceb8f9a700 1 -- 172.21.15.27:0/180776849 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb40b1c10 con 0x7fceb406b610 2024-01-11T09:46:40.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.234+0000 7fceabfff700 1 -- 172.21.15.27:0/180776849 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273428423 0 0) 0x7fceac001490 con 0x7fceb40b1800 2024-01-11T09:46:40.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.234+0000 7fceb979b700 1 --2- 172.21.15.27:0/180776849 >> v2:172.21.15.77:3300/0 conn(0x7fceb41575e0 0x7fceb4158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:40.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.234+0000 7fceb8f9a700 1 --2- 172.21.15.27:0/180776849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb406b610 0x7fceb40b12c0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fceb4064b00 tx=0x7fcea40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=488af5e61401177e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:40.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.234+0000 7fceabfff700 1 -- 172.21.15.27:0/180776849 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea400b6b0 con 0x7fceb406b610 2024-01-11T09:46:40.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.234+0000 7fceabfff700 1 -- 172.21.15.27:0/180776849 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea400b810 con 0x7fceb406b610 2024-01-11T09:46:40.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.234+0000 7fceabfff700 1 -- 172.21.15.27:0/180776849 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea400bbb0 con 0x7fceb406b610 2024-01-11T09:46:40.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.235+0000 7fcebb9ff700 1 -- 172.21.15.27:0/180776849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb406b610 msgr2=0x7fceb40b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:40.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.235+0000 7fcebb9ff700 1 --2- 172.21.15.27:0/180776849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb406b610 0x7fceb40b12c0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fceb4064b00 tx=0x7fcea40091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:40.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.235+0000 7fcebb9ff700 1 -- 172.21.15.27:0/180776849 shutdown_connections 2024-01-11T09:46:40.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.235+0000 7fcebb9ff700 1 --2- 172.21.15.27:0/180776849 >> v2:172.21.15.77:3300/0 conn(0x7fceb41575e0 0x7fceb4158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:40.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.235+0000 7fcebb9ff700 1 --2- 172.21.15.27:0/180776849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb406b610 0x7fceb40b12c0 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:40.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.235+0000 7fcebb9ff700 1 -- 172.21.15.27:0/180776849 >> 172.21.15.27:0/180776849 conn(0x7fceb40680e0 msgr2=0x7fceb4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:40.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.235+0000 7fcebb9ff700 1 -- 172.21.15.27:0/180776849 shutdown_connections 2024-01-11T09:46:40.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.235+0000 7fcebb9ff700 1 -- 172.21.15.27:0/180776849 wait complete. 2024-01-11T09:46:40.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.237+0000 7fcebb9ff700 1 Processor -- start 2024-01-11T09:46:40.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.237+0000 7fcebb9ff700 1 -- start start 2024-01-11T09:46:40.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.238+0000 7fcebb9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fceb406b610 0x7fceb4140080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:40.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.238+0000 7fcebb9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fceb40b1800 0x7fceb4140990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:40.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.238+0000 7fcebb9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb41575e0 0x7fceb415b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:40.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.238+0000 7fcebb9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb415c760 con 0x7fceb41575e0 2024-01-11T09:46:40.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.238+0000 7fcebb9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb415bb50 con 0x7fceb406b610 2024-01-11T09:46:40.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.238+0000 7fcebb9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb415be30 con 0x7fceb40b1800 2024-01-11T09:46:40.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.238+0000 7fceb9f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb41575e0 0x7fceb415b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:40.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.238+0000 7fceb9f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb41575e0 0x7fceb415b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40398/0 (socket says 172.21.15.27:40398) 2024-01-11T09:46:40.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.238+0000 7fceb9f9c700 1 -- 172.21.15.27:0/2648284238 learned_addr learned my addr 172.21.15.27:0/2648284238 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:40.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.238+0000 7fceb979b700 1 --2- 172.21.15.27:0/2648284238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fceb406b610 0x7fceb4140080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:40.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.238+0000 7fceb8f9a700 1 --2- 172.21.15.27:0/2648284238 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fceb40b1800 0x7fceb4140990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:40.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.238+0000 7fceb9f9c700 1 -- 172.21.15.27:0/2648284238 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fceb40b1800 msgr2=0x7fceb4140990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:40.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.238+0000 7fceb9f9c700 1 --2- 172.21.15.27:0/2648284238 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fceb40b1800 0x7fceb4140990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:40.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.238+0000 7fceb9f9c700 1 -- 172.21.15.27:0/2648284238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fceb406b610 msgr2=0x7fceb4140080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:40.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.238+0000 7fceb9f9c700 1 --2- 172.21.15.27:0/2648284238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fceb406b610 0x7fceb4140080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:40.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.238+0000 7fceb9f9c700 1 -- 172.21.15.27:0/2648284238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea4009420 con 0x7fceb41575e0 2024-01-11T09:46:40.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.238+0000 7fceb8f9a700 1 --2- 172.21.15.27:0/2648284238 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fceb40b1800 0x7fceb4140990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:40.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.239+0000 7fceb9f9c700 1 --2- 172.21.15.27:0/2648284238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb41575e0 0x7fceb415b100 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fceac00cc90 tx=0x7fceac001a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:40.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.239+0000 7fcea97fa700 1 -- 172.21.15.27:0/2648284238 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceac004bb0 con 0x7fceb41575e0 2024-01-11T09:46:40.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.239+0000 7fcea97fa700 1 -- 172.21.15.27:0/2648284238 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceac004d10 con 0x7fceb41575e0 2024-01-11T09:46:40.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.239+0000 7fcebb9ff700 1 -- 172.21.15.27:0/2648284238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb415c110 con 0x7fceb41575e0 2024-01-11T09:46:40.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.239+0000 7fcea97fa700 1 -- 172.21.15.27:0/2648284238 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceac00abb0 con 0x7fceb41575e0 2024-01-11T09:46:40.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.239+0000 7fcebb9ff700 1 -- 172.21.15.27:0/2648284238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb41405c0 con 0x7fceb41575e0 2024-01-11T09:46:40.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.240+0000 7fcebb9ff700 1 -- 172.21.15.27:0/2648284238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb4070d70 con 0x7fceb41575e0 2024-01-11T09:46:40.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.241+0000 7fcea97fa700 1 -- 172.21.15.27:0/2648284238 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fceac0070f0 con 0x7fceb41575e0 2024-01-11T09:46:40.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.242+0000 7fcea97fa700 1 --2- 172.21.15.27:0/2648284238 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fce98067170 0x7fce98069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:40.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.242+0000 7fcea97fa700 1 -- 172.21.15.27:0/2648284238 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fceac0848e0 con 0x7fceb41575e0 2024-01-11T09:46:40.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.243+0000 7fceb979b700 1 --2- 172.21.15.27:0/2648284238 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fce98067170 0x7fce98069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:40.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.243+0000 7fceb979b700 1 --2- 172.21.15.27:0/2648284238 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fce98067170 0x7fce98069620 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fceb0006f00 tx=0x7fceb0008040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:40.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.246+0000 7fcea97fa700 1 -- 172.21.15.27:0/2648284238 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fceb4070d70 con 0x7fceb41575e0 2024-01-11T09:46:40.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.451+0000 7fcebb9ff700 1 -- 172.21.15.27:0/2648284238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fceb4158b60 con 0x7fceb41575e0 2024-01-11T09:46:40.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.452+0000 7fcea97fa700 1 -- 172.21.15.27:0/2648284238 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v238) v1 ==== 74+0+18740 (secure 0 0 0) 0x7fceb4158b60 con 0x7fceb41575e0 2024-01-11T09:46:40.444 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:40.445 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":238,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:46:39.483585+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:46:24.049904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-01-11T09:46:24.737495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-01-11T09:46:37.665603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"238","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-01-11T09:46:38.472134+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"238","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:46:40.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.454+0000 7fcebb9ff700 1 -- 172.21.15.27:0/2648284238 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fce98067170 msgr2=0x7fce98069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:40.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.454+0000 7fcebb9ff700 1 --2- 172.21.15.27:0/2648284238 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fce98067170 0x7fce98069620 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fceb0006f00 tx=0x7fceb0008040 comp rx=0 tx=0).stop 2024-01-11T09:46:40.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.454+0000 7fcebb9ff700 1 -- 172.21.15.27:0/2648284238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb41575e0 msgr2=0x7fceb415b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:40.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.454+0000 7fcebb9ff700 1 --2- 172.21.15.27:0/2648284238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb41575e0 0x7fceb415b100 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fceac00cc90 tx=0x7fceac001a50 comp rx=0 tx=0).stop 2024-01-11T09:46:40.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.454+0000 7fcebb9ff700 1 -- 172.21.15.27:0/2648284238 shutdown_connections 2024-01-11T09:46:40.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.454+0000 7fcebb9ff700 1 --2- 172.21.15.27:0/2648284238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceb41575e0 0x7fceb415b100 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:40.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.454+0000 7fcebb9ff700 1 --2- 172.21.15.27:0/2648284238 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fce98067170 0x7fce98069620 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:40.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.454+0000 7fcebb9ff700 1 --2- 172.21.15.27:0/2648284238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fceb406b610 0x7fceb4140080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:40.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.454+0000 7fcebb9ff700 1 --2- 172.21.15.27:0/2648284238 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fceb40b1800 0x7fceb4140990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:40.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.454+0000 7fcebb9ff700 1 -- 172.21.15.27:0/2648284238 >> 172.21.15.27:0/2648284238 conn(0x7fceb40680e0 msgr2=0x7fceb41579b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:40.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.454+0000 7fcebb9ff700 1 -- 172.21.15.27:0/2648284238 shutdown_connections 2024-01-11T09:46:40.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.454+0000 7fcebb9ff700 1 -- 172.21.15.27:0/2648284238 wait complete. 2024-01-11T09:46:40.465 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:46:40.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.672+0000 7fa99efff700 1 Processor -- start 2024-01-11T09:46:40.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.672+0000 7fa99efff700 1 -- start start 2024-01-11T09:46:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.673+0000 7fa99efff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa9980ca670 0x7fa998068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.673+0000 7fa99efff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa998068b60 0x7fa9980aa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.673+0000 7fa99efff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99815da10 con 0x7fa9981596c0 2024-01-11T09:46:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.673+0000 7fa99efff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa998068fc0 con 0x7fa998068b60 2024-01-11T09:46:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.673+0000 7fa99efff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9980a9cb0 con 0x7fa9980ca670 2024-01-11T09:46:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.673+0000 7fa99cd9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa9981596c0 0x7fa99815d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48176/0 (socket says 172.21.15.27:48176) 2024-01-11T09:46:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.673+0000 7fa99cd9b700 1 -- 172.21.15.27:0/1318120000 learned_addr learned my addr 172.21.15.27:0/1318120000 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.673+0000 7fa99d59c700 1 --2- 172.21.15.27:0/1318120000 >> v2:172.21.15.27:3300/0 conn(0x7fa998068b60 0x7fa9980aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.674+0000 7fa997fff700 1 --2- 172.21.15.27:0/1318120000 >> v2:172.21.15.77:3300/0 conn(0x7fa9980ca670 0x7fa998068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.674+0000 7fa9977fe700 1 -- 172.21.15.27:0/1318120000 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298733002 0 0) 0x7fa98c001110 con 0x7fa9981596c0 2024-01-11T09:46:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.674+0000 7fa9977fe700 1 -- 172.21.15.27:0/1318120000 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa988003570 con 0x7fa9981596c0 2024-01-11T09:46:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.674+0000 7fa99d59c700 1 -- 172.21.15.27:0/1318120000 >> v2:172.21.15.77:3300/0 conn(0x7fa9980ca670 msgr2=0x7fa998068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.674+0000 7fa99d59c700 1 --2- 172.21.15.27:0/1318120000 >> v2:172.21.15.77:3300/0 conn(0x7fa9980ca670 0x7fa998068620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.674+0000 7fa99d59c700 1 -- 172.21.15.27:0/1318120000 >> v1:172.21.15.27:6789/0 conn(0x7fa9981596c0 legacy=0x7fa99815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.674+0000 7fa99d59c700 1 -- 172.21.15.27:0/1318120000 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99815e210 con 0x7fa998068b60 2024-01-11T09:46:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.674+0000 7fa99d59c700 1 --2- 172.21.15.27:0/1318120000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa998068b60 0x7fa9980aa140 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7fa9980b2c00 tx=0x7fa990009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=173d545919e44637 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.675+0000 7fa9977fe700 1 -- 172.21.15.27:0/1318120000 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99000e070 con 0x7fa998068b60 2024-01-11T09:46:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.675+0000 7fa9977fe700 1 -- 172.21.15.27:0/1318120000 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99000bab0 con 0x7fa998068b60 2024-01-11T09:46:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.675+0000 7fa9977fe700 1 -- 172.21.15.27:0/1318120000 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99000be70 con 0x7fa998068b60 2024-01-11T09:46:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.675+0000 7fa99efff700 1 -- 172.21.15.27:0/1318120000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa998068b60 msgr2=0x7fa9980aa140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.675+0000 7fa99efff700 1 --2- 172.21.15.27:0/1318120000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa998068b60 0x7fa9980aa140 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7fa9980b2c00 tx=0x7fa990009780 comp rx=0 tx=0).stop 2024-01-11T09:46:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.675+0000 7fa99efff700 1 -- 172.21.15.27:0/1318120000 shutdown_connections 2024-01-11T09:46:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.675+0000 7fa99efff700 1 --2- 172.21.15.27:0/1318120000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa998068b60 0x7fa9980aa140 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.675+0000 7fa99efff700 1 --2- 172.21.15.27:0/1318120000 >> v2:172.21.15.77:3300/0 conn(0x7fa9980ca670 0x7fa998068620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.675+0000 7fa99efff700 1 -- 172.21.15.27:0/1318120000 >> 172.21.15.27:0/1318120000 conn(0x7fa9980c9f70 msgr2=0x7fa99806deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.675+0000 7fa99efff700 1 -- 172.21.15.27:0/1318120000 shutdown_connections 2024-01-11T09:46:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.675+0000 7fa99efff700 1 -- 172.21.15.27:0/1318120000 wait complete. 2024-01-11T09:46:40.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.677+0000 7fa99efff700 1 Processor -- start 2024-01-11T09:46:40.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.677+0000 7fa99efff700 1 -- start start 2024-01-11T09:46:40.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.678+0000 7fa99efff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa998068b60 0x7fa998145e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:40.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.678+0000 7fa99efff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa9980ca670 0x7fa9980c4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:40.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.678+0000 7fa99efff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9981596c0 0x7fa99816ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:40.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.678+0000 7fa99efff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99815bec0 con 0x7fa9981596c0 2024-01-11T09:46:40.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.678+0000 7fa99efff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99815c1a0 con 0x7fa998068b60 2024-01-11T09:46:40.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.678+0000 7fa99efff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99815c480 con 0x7fa9980ca670 2024-01-11T09:46:40.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.678+0000 7fa99d59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9981596c0 0x7fa99816ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:40.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.678+0000 7fa99d59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9981596c0 0x7fa99816ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40420/0 (socket says 172.21.15.27:40420) 2024-01-11T09:46:40.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.678+0000 7fa99d59c700 1 -- 172.21.15.27:0/1054992793 learned_addr learned my addr 172.21.15.27:0/1054992793 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:40.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.678+0000 7fa99cd9b700 1 --2- 172.21.15.27:0/1054992793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa998068b60 0x7fa998145e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:40.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.678+0000 7fa997fff700 1 --2- 172.21.15.27:0/1054992793 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa9980ca670 0x7fa9980c4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:40.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.678+0000 7fa99d59c700 1 -- 172.21.15.27:0/1054992793 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa9980ca670 msgr2=0x7fa9980c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.679+0000 7fa99d59c700 1 --2- 172.21.15.27:0/1054992793 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa9980ca670 0x7fa9980c4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.679+0000 7fa99d59c700 1 -- 172.21.15.27:0/1054992793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa998068b60 msgr2=0x7fa998145e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.679+0000 7fa99d59c700 1 --2- 172.21.15.27:0/1054992793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa998068b60 0x7fa998145e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.679+0000 7fa99d59c700 1 -- 172.21.15.27:0/1054992793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa990009a20 con 0x7fa9981596c0 2024-01-11T09:46:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.679+0000 7fa997fff700 1 --2- 172.21.15.27:0/1054992793 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa9980ca670 0x7fa9980c4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.679+0000 7fa99cd9b700 1 --2- 172.21.15.27:0/1054992793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa998068b60 0x7fa998145e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.679+0000 7fa99d59c700 1 --2- 172.21.15.27:0/1054992793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9981596c0 0x7fa99816ad60 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fa99806d490 tx=0x7fa990019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.679+0000 7fa994ff9700 1 -- 172.21.15.27:0/1054992793 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa990027470 con 0x7fa9981596c0 2024-01-11T09:46:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.679+0000 7fa994ff9700 1 -- 172.21.15.27:0/1054992793 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9900275d0 con 0x7fa9981596c0 2024-01-11T09:46:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.679+0000 7fa99efff700 1 -- 172.21.15.27:0/1054992793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99815c760 con 0x7fa9981596c0 2024-01-11T09:46:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.679+0000 7fa994ff9700 1 -- 172.21.15.27:0/1054992793 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa990027990 con 0x7fa9981596c0 2024-01-11T09:46:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.679+0000 7fa99efff700 1 -- 172.21.15.27:0/1054992793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9980c3bc0 con 0x7fa9981596c0 2024-01-11T09:46:40.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.680+0000 7fa99efff700 1 -- 172.21.15.27:0/1054992793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9980c3db0 con 0x7fa9981596c0 2024-01-11T09:46:40.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.681+0000 7fa994ff9700 1 -- 172.21.15.27:0/1054992793 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fa990024070 con 0x7fa9981596c0 2024-01-11T09:46:40.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.683+0000 7fa994ff9700 1 --2- 172.21.15.27:0/1054992793 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa97c067170 0x7fa97c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:40.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.683+0000 7fa994ff9700 1 -- 172.21.15.27:0/1054992793 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fa990087670 con 0x7fa9981596c0 2024-01-11T09:46:40.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.683+0000 7fa99cd9b700 1 --2- 172.21.15.27:0/1054992793 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa97c067170 0x7fa97c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:40.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.684+0000 7fa99cd9b700 1 --2- 172.21.15.27:0/1054992793 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa97c067170 0x7fa97c069620 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa98c001270 tx=0x7fa98c009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:40.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.687+0000 7fa994ff9700 1 -- 172.21.15.27:0/1054992793 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa9980c3db0 con 0x7fa9981596c0 2024-01-11T09:46:40.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.891+0000 7fa99efff700 1 -- 172.21.15.27:0/1054992793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa99815cc80 con 0x7fa9981596c0 2024-01-11T09:46:40.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.892+0000 7fa994ff9700 1 -- 172.21.15.27:0/1054992793 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v238) v1 ==== 74+0+18740 (secure 0 0 0) 0x7fa990054280 con 0x7fa9981596c0 2024-01-11T09:46:40.885 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:40.885 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":238,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:46:39.483585+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:46:24.049904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-01-11T09:46:24.737495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-01-11T09:46:37.665603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"238","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-01-11T09:46:38.472134+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"238","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:46:40.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.895+0000 7fa99efff700 1 -- 172.21.15.27:0/1054992793 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa97c067170 msgr2=0x7fa97c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:40.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.895+0000 7fa99efff700 1 --2- 172.21.15.27:0/1054992793 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa97c067170 0x7fa97c069620 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa98c001270 tx=0x7fa98c009040 comp rx=0 tx=0).stop 2024-01-11T09:46:40.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.895+0000 7fa99efff700 1 -- 172.21.15.27:0/1054992793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9981596c0 msgr2=0x7fa99816ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:40.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.895+0000 7fa99efff700 1 --2- 172.21.15.27:0/1054992793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9981596c0 0x7fa99816ad60 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fa99806d490 tx=0x7fa990019000 comp rx=0 tx=0).stop 2024-01-11T09:46:40.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.895+0000 7fa99efff700 1 -- 172.21.15.27:0/1054992793 shutdown_connections 2024-01-11T09:46:40.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.895+0000 7fa99efff700 1 --2- 172.21.15.27:0/1054992793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9981596c0 0x7fa99816ad60 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:40.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.895+0000 7fa99efff700 1 --2- 172.21.15.27:0/1054992793 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa97c067170 0x7fa97c069620 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:40.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.895+0000 7fa99efff700 1 --2- 172.21.15.27:0/1054992793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa998068b60 0x7fa998145e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:40.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.895+0000 7fa99efff700 1 --2- 172.21.15.27:0/1054992793 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa9980ca670 0x7fa9980c4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:40.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.895+0000 7fa99efff700 1 -- 172.21.15.27:0/1054992793 >> 172.21.15.27:0/1054992793 conn(0x7fa9980c9f70 msgr2=0x7fa9980aaf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:40.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.895+0000 7fa99efff700 1 -- 172.21.15.27:0/1054992793 shutdown_connections 2024-01-11T09:46:40.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:40.895+0000 7fa99efff700 1 -- 172.21.15.27:0/1054992793 wait complete. 2024-01-11T09:46:40.906 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2024-01-11T09:46:41.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.119+0000 7fe7037ff700 1 Processor -- start 2024-01-11T09:46:41.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.119+0000 7fe7037ff700 1 -- start start 2024-01-11T09:46:41.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.120+0000 7fe7037ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe6fc13ff80 0x7fe6fc15a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:41.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.120+0000 7fe7037ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe6fc15a560 0x7fe6fc0c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:41.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.120+0000 7fe7037ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6fc15a9c0 con 0x7fe6fc13fbb0 2024-01-11T09:46:41.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.120+0000 7fe7037ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6fc0c6b30 con 0x7fe6fc15a560 2024-01-11T09:46:41.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.120+0000 7fe7037ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc0c6800 con 0x7fe6fc13ff80 2024-01-11T09:46:41.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.121+0000 7fe700d9a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe6fc13ff80 0x7fe6fc15a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:41.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.121+0000 7fe701d9c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe6fc15a560 0x7fe6fc0c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:41.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.121+0000 7fe70159b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe6fc13fbb0 0x7fe6fc0ca7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:55486/0 (socket says 172.21.15.27:55486) 2024-01-11T09:46:41.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.121+0000 7fe70159b700 1 -- 172.21.15.27:0/4221868224 learned_addr learned my addr 172.21.15.27:0/4221868224 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:41.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.121+0000 7fe6f3fff700 1 -- 172.21.15.27:0/4221868224 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368059955 0 0) 0x7fe6f8001160 con 0x7fe6fc13fbb0 2024-01-11T09:46:41.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.121+0000 7fe700d9a700 1 -- 172.21.15.27:0/4221868224 >> v2:172.21.15.77:3300/0 conn(0x7fe6fc15a560 msgr2=0x7fe6fc0c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:41.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.121+0000 7fe700d9a700 1 --2- 172.21.15.27:0/4221868224 >> v2:172.21.15.77:3300/0 conn(0x7fe6fc15a560 0x7fe6fc0c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:41.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.121+0000 7fe700d9a700 1 -- 172.21.15.27:0/4221868224 >> v1:172.21.15.77:6789/0 conn(0x7fe6fc13fbb0 legacy=0x7fe6fc0ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:41.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.121+0000 7fe700d9a700 1 -- 172.21.15.27:0/4221868224 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6fc0cb240 con 0x7fe6fc13ff80 2024-01-11T09:46:41.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.122+0000 7fe700d9a700 1 --2- 172.21.15.27:0/4221868224 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe6fc13ff80 0x7fe6fc15a020 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec0098e0 tx=0x7fe6ec0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c91963f39f7fdda7 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:41.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.122+0000 7fe6f3fff700 1 -- 172.21.15.27:0/4221868224 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec00b640 con 0x7fe6fc13ff80 2024-01-11T09:46:41.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.122+0000 7fe6f3fff700 1 -- 172.21.15.27:0/4221868224 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ec00b7a0 con 0x7fe6fc13ff80 2024-01-11T09:46:41.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.122+0000 7fe6f3fff700 1 -- 172.21.15.27:0/4221868224 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec00bb40 con 0x7fe6fc13ff80 2024-01-11T09:46:41.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.122+0000 7fe7037ff700 1 -- 172.21.15.27:0/4221868224 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe6fc13ff80 msgr2=0x7fe6fc15a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:41.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.122+0000 7fe7037ff700 1 --2- 172.21.15.27:0/4221868224 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe6fc13ff80 0x7fe6fc15a020 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec0098e0 tx=0x7fe6ec0091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:41.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.123+0000 7fe7037ff700 1 -- 172.21.15.27:0/4221868224 shutdown_connections 2024-01-11T09:46:41.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.123+0000 7fe7037ff700 1 --2- 172.21.15.27:0/4221868224 >> v2:172.21.15.77:3300/0 conn(0x7fe6fc15a560 0x7fe6fc0c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:41.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.123+0000 7fe7037ff700 1 --2- 172.21.15.27:0/4221868224 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe6fc13ff80 0x7fe6fc15a020 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:41.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.123+0000 7fe7037ff700 1 -- 172.21.15.27:0/4221868224 >> 172.21.15.27:0/4221868224 conn(0x7fe6fc0680a0 msgr2=0x7fe6fc0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:41.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.123+0000 7fe7037ff700 1 -- 172.21.15.27:0/4221868224 shutdown_connections 2024-01-11T09:46:41.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.123+0000 7fe7037ff700 1 -- 172.21.15.27:0/4221868224 wait complete. 2024-01-11T09:46:41.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.125+0000 7fe7037ff700 1 Processor -- start 2024-01-11T09:46:41.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.125+0000 7fe7037ff700 1 -- start start 2024-01-11T09:46:41.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.126+0000 7fe7037ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6fc13fbb0 0x7fe6fc16d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:41.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.126+0000 7fe7037ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe6fc13ff80 0x7fe6fc1577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:41.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.126+0000 7fe7037ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6fc15a560 0x7fe6fc134ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:41.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.126+0000 7fe7037ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc16e430 con 0x7fe6fc13fbb0 2024-01-11T09:46:41.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.126+0000 7fe7037ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc0c42d0 con 0x7fe6fc15a560 2024-01-11T09:46:41.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.126+0000 7fe7037ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc157ce0 con 0x7fe6fc13ff80 2024-01-11T09:46:41.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.126+0000 7fe70159b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6fc13fbb0 0x7fe6fc16d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:41.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.127+0000 7fe70159b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6fc13fbb0 0x7fe6fc16d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40424/0 (socket says 172.21.15.27:40424) 2024-01-11T09:46:41.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.127+0000 7fe70159b700 1 -- 172.21.15.27:0/2896599335 learned_addr learned my addr 172.21.15.27:0/2896599335 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:41.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.127+0000 7fe700d9a700 1 --2- 172.21.15.27:0/2896599335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe6fc13ff80 0x7fe6fc1577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:41.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.127+0000 7fe701d9c700 1 --2- 172.21.15.27:0/2896599335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6fc15a560 0x7fe6fc134ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:41.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.127+0000 7fe70159b700 1 -- 172.21.15.27:0/2896599335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe6fc13ff80 msgr2=0x7fe6fc1577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:41.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.127+0000 7fe70159b700 1 --2- 172.21.15.27:0/2896599335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe6fc13ff80 0x7fe6fc1577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:41.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.127+0000 7fe70159b700 1 -- 172.21.15.27:0/2896599335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6fc15a560 msgr2=0x7fe6fc134ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:41.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.127+0000 7fe70159b700 1 --2- 172.21.15.27:0/2896599335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6fc15a560 0x7fe6fc134ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:41.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.127+0000 7fe70159b700 1 -- 172.21.15.27:0/2896599335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ec009420 con 0x7fe6fc13fbb0 2024-01-11T09:46:41.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.127+0000 7fe70159b700 1 --2- 172.21.15.27:0/2896599335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6fc13fbb0 0x7fe6fc16d4b0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fe6f800c810 tx=0x7fe6f8007c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:41.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.127+0000 7fe6f17fa700 1 -- 172.21.15.27:0/2896599335 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f8005b80 con 0x7fe6fc13fbb0 2024-01-11T09:46:41.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.128+0000 7fe6f17fa700 1 -- 172.21.15.27:0/2896599335 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f8005d00 con 0x7fe6fc13fbb0 2024-01-11T09:46:41.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.128+0000 7fe6f17fa700 1 -- 172.21.15.27:0/2896599335 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f8007320 con 0x7fe6fc13fbb0 2024-01-11T09:46:41.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.128+0000 7fe7037ff700 1 -- 172.21.15.27:0/2896599335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6fc135010 con 0x7fe6fc13fbb0 2024-01-11T09:46:41.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.128+0000 7fe7037ff700 1 -- 172.21.15.27:0/2896599335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6fc16de70 con 0x7fe6fc13fbb0 2024-01-11T09:46:41.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.129+0000 7fe7037ff700 1 -- 172.21.15.27:0/2896599335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6fc04c3c0 con 0x7fe6fc13fbb0 2024-01-11T09:46:41.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.130+0000 7fe6f17fa700 1 -- 172.21.15.27:0/2896599335 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fe6f8007750 con 0x7fe6fc13fbb0 2024-01-11T09:46:41.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.131+0000 7fe6f17fa700 1 --2- 172.21.15.27:0/2896599335 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe6e0067170 0x7fe6e0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:41.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.131+0000 7fe6f17fa700 1 -- 172.21.15.27:0/2896599335 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fe6f8083fa0 con 0x7fe6fc13fbb0 2024-01-11T09:46:41.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.131+0000 7fe700d9a700 1 --2- 172.21.15.27:0/2896599335 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe6e0067170 0x7fe6e0069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:41.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.132+0000 7fe700d9a700 1 --2- 172.21.15.27:0/2896599335 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe6e0067170 0x7fe6e0069620 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec0093f0 tx=0x7fe6ec017000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:41.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.135+0000 7fe6f17fa700 1 -- 172.21.15.27:0/2896599335 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe6f8050bb0 con 0x7fe6fc13fbb0 2024-01-11T09:46:41.197 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-01-11T09:46:41.205+0000 7fc9719f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T09:46:41.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.391+0000 7fe7037ff700 1 -- 172.21.15.27:0/2896599335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe6fc15a9c0 con 0x7fe6fc13fbb0 2024-01-11T09:46:41.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.996+0000 7fe6f17fa700 1 -- 172.21.15.27:0/2896599335 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v131) v1 ==== 121+0+0 (secure 0 0 0) 0x7fe6fc15a9c0 con 0x7fe6fc13fbb0 2024-01-11T09:46:41.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.998+0000 7fe7037ff700 1 -- 172.21.15.27:0/2896599335 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe6e0067170 msgr2=0x7fe6e0069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:41.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.998+0000 7fe7037ff700 1 --2- 172.21.15.27:0/2896599335 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe6e0067170 0x7fe6e0069620 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec0093f0 tx=0x7fe6ec017000 comp rx=0 tx=0).stop 2024-01-11T09:46:41.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.998+0000 7fe7037ff700 1 -- 172.21.15.27:0/2896599335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6fc13fbb0 msgr2=0x7fe6fc16d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:41.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.998+0000 7fe7037ff700 1 --2- 172.21.15.27:0/2896599335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6fc13fbb0 0x7fe6fc16d4b0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fe6f800c810 tx=0x7fe6f8007c40 comp rx=0 tx=0).stop 2024-01-11T09:46:41.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.999+0000 7fe7037ff700 1 -- 172.21.15.27:0/2896599335 shutdown_connections 2024-01-11T09:46:41.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.999+0000 7fe7037ff700 1 --2- 172.21.15.27:0/2896599335 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe6e0067170 0x7fe6e0069620 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:41.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.999+0000 7fe7037ff700 1 --2- 172.21.15.27:0/2896599335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6fc15a560 0x7fe6fc134ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:41.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.999+0000 7fe7037ff700 1 --2- 172.21.15.27:0/2896599335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6fc13fbb0 0x7fe6fc16d4b0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:41.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.999+0000 7fe7037ff700 1 --2- 172.21.15.27:0/2896599335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe6fc13ff80 0x7fe6fc1577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:41.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.999+0000 7fe7037ff700 1 -- 172.21.15.27:0/2896599335 >> 172.21.15.27:0/2896599335 conn(0x7fe6fc0680a0 msgr2=0x7fe6fc13ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:41.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.999+0000 7fe7037ff700 1 -- 172.21.15.27:0/2896599335 shutdown_connections 2024-01-11T09:46:41.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:41.999+0000 7fe7037ff700 1 -- 172.21.15.27:0/2896599335 wait complete. 2024-01-11T09:46:42.011 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2024-01-11T09:46:42.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.225+0000 7f2bffdff700 1 Processor -- start 2024-01-11T09:46:42.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.225+0000 7f2bffdff700 1 -- start start 2024-01-11T09:46:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.226+0000 7f2bffdff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2bf81577e0 0x7f2bf80ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.226+0000 7f2bffdff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2bf8069540 0x7f2bf80ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.226+0000 7f2bffdff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf80699d0 con 0x7f2bf8157410 2024-01-11T09:46:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.226+0000 7f2bffdff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bf8069cb0 con 0x7f2bf81577e0 2024-01-11T09:46:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.226+0000 7f2bffdff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bf8069f90 con 0x7f2bf8069540 2024-01-11T09:46:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.226+0000 7f2bfd39a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2bf81577e0 0x7f2bf80ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.227+0000 7f2bfd39a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2bf81577e0 0x7f2bf80ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40432/0 (socket says 172.21.15.27:40432) 2024-01-11T09:46:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.226+0000 7f2bfdb9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2bf8157410 0x7f2bf80ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48184/0 (socket says 172.21.15.27:48184) 2024-01-11T09:46:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.227+0000 7f2bfd39a700 1 -- 172.21.15.27:0/1244736809 learned_addr learned my addr 172.21.15.27:0/1244736809 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.227+0000 7f2bfe39c700 1 --2- 172.21.15.27:0/1244736809 >> v2:172.21.15.77:3300/0 conn(0x7f2bf8069540 0x7f2bf80ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.227+0000 7f2bfcb99700 1 -- 172.21.15.27:0/1244736809 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139175710 0 0) 0x7f2bf80699d0 con 0x7f2bf8157410 2024-01-11T09:46:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.227+0000 7f2bfd39a700 1 -- 172.21.15.27:0/1244736809 >> v2:172.21.15.77:3300/0 conn(0x7f2bf8069540 msgr2=0x7f2bf80ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.227+0000 7f2bfd39a700 1 --2- 172.21.15.27:0/1244736809 >> v2:172.21.15.77:3300/0 conn(0x7f2bf8069540 0x7f2bf80ae540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.227+0000 7f2bfd39a700 1 -- 172.21.15.27:0/1244736809 >> v1:172.21.15.27:6789/0 conn(0x7f2bf8157410 legacy=0x7f2bf80ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.227+0000 7f2bfd39a700 1 -- 172.21.15.27:0/1244736809 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf806a270 con 0x7f2bf81577e0 2024-01-11T09:46:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.227+0000 7f2bfd39a700 1 --2- 172.21.15.27:0/1244736809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bf81577e0 0x7f2bf80ae000 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f2bf806d030 tx=0x7f2be8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b08e998901cc3ee5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.227+0000 7f2bfcb99700 1 -- 172.21.15.27:0/1244736809 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be8009e70 con 0x7f2bf81577e0 2024-01-11T09:46:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.228+0000 7f2bfcb99700 1 -- 172.21.15.27:0/1244736809 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be800b860 con 0x7f2bf81577e0 2024-01-11T09:46:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.228+0000 7f2bfcb99700 1 -- 172.21.15.27:0/1244736809 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be800bc00 con 0x7f2bf81577e0 2024-01-11T09:46:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.228+0000 7f2bffdff700 1 -- 172.21.15.27:0/1244736809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bf81577e0 msgr2=0x7f2bf80ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.228+0000 7f2bffdff700 1 --2- 172.21.15.27:0/1244736809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bf81577e0 0x7f2bf80ae000 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f2bf806d030 tx=0x7f2be8009250 comp rx=0 tx=0).stop 2024-01-11T09:46:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.228+0000 7f2bffdff700 1 -- 172.21.15.27:0/1244736809 shutdown_connections 2024-01-11T09:46:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.228+0000 7f2bffdff700 1 --2- 172.21.15.27:0/1244736809 >> v2:172.21.15.77:3300/0 conn(0x7f2bf8069540 0x7f2bf80ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.228+0000 7f2bffdff700 1 --2- 172.21.15.27:0/1244736809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bf81577e0 0x7f2bf80ae000 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.228+0000 7f2bffdff700 1 -- 172.21.15.27:0/1244736809 >> 172.21.15.27:0/1244736809 conn(0x7f2bf80680a0 msgr2=0x7f2bf80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.228+0000 7f2bffdff700 1 -- 172.21.15.27:0/1244736809 shutdown_connections 2024-01-11T09:46:42.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.228+0000 7f2bffdff700 1 -- 172.21.15.27:0/1244736809 wait complete. 2024-01-11T09:46:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.230+0000 7f2bffdff700 1 Processor -- start 2024-01-11T09:46:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.230+0000 7f2bffdff700 1 -- start start 2024-01-11T09:46:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.231+0000 7f2bffdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bf8069540 0x7f2bf80c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.231+0000 7f2bffdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2bf8157410 0x7f2bf80cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.231+0000 7f2bffdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf81577e0 0x7f2bf814e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.231+0000 7f2bffdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf80cbe20 con 0x7f2bf8069540 2024-01-11T09:46:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.231+0000 7f2bffdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf80c4f50 con 0x7f2bf81577e0 2024-01-11T09:46:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.231+0000 7f2bffdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf80c4340 con 0x7f2bf8157410 2024-01-11T09:46:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.231+0000 7f2bfdb9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bf8069540 0x7f2bf80c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.231+0000 7f2bfdb9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bf8069540 0x7f2bf80c3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40448/0 (socket says 172.21.15.27:40448) 2024-01-11T09:46:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.231+0000 7f2bfdb9b700 1 -- 172.21.15.27:0/3950436851 learned_addr learned my addr 172.21.15.27:0/3950436851 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.231+0000 7f2bfd39a700 1 --2- 172.21.15.27:0/3950436851 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2bf8157410 0x7f2bf80cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.231+0000 7f2bfe39c700 1 --2- 172.21.15.27:0/3950436851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf81577e0 0x7f2bf814e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.232+0000 7f2bfdb9b700 1 -- 172.21.15.27:0/3950436851 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2bf8157410 msgr2=0x7f2bf80cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.232+0000 7f2bfdb9b700 1 --2- 172.21.15.27:0/3950436851 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2bf8157410 0x7f2bf80cb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.232+0000 7f2bfdb9b700 1 -- 172.21.15.27:0/3950436851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf81577e0 msgr2=0x7f2bf814e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.232+0000 7f2bfdb9b700 1 --2- 172.21.15.27:0/3950436851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf81577e0 0x7f2bf814e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.232+0000 7f2bfdb9b700 1 -- 172.21.15.27:0/3950436851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be80094f0 con 0x7f2bf8069540 2024-01-11T09:46:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.232+0000 7f2bfe39c700 1 --2- 172.21.15.27:0/3950436851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf81577e0 0x7f2bf814e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:46:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.232+0000 7f2bfdb9b700 1 --2- 172.21.15.27:0/3950436851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bf8069540 0x7f2bf80c3c80 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400c810 tx=0x7f2bf4007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.232+0000 7f2bedffb700 1 -- 172.21.15.27:0/3950436851 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf4007300 con 0x7f2bf8069540 2024-01-11T09:46:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.232+0000 7f2bedffb700 1 -- 172.21.15.27:0/3950436851 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf4007460 con 0x7f2bf8069540 2024-01-11T09:46:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.232+0000 7f2bffdff700 1 -- 172.21.15.27:0/3950436851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf80c4620 con 0x7f2bf8069540 2024-01-11T09:46:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.232+0000 7f2bedffb700 1 -- 172.21.15.27:0/3950436851 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf40057f0 con 0x7f2bf8069540 2024-01-11T09:46:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.232+0000 7f2bffdff700 1 -- 172.21.15.27:0/3950436851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf80c4be0 con 0x7f2bf8069540 2024-01-11T09:46:42.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.233+0000 7f2bffdff700 1 -- 172.21.15.27:0/3950436851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf804c3c0 con 0x7f2bf8069540 2024-01-11T09:46:42.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.234+0000 7f2bedffb700 1 -- 172.21.15.27:0/3950436851 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f2bf4005b80 con 0x7f2bf8069540 2024-01-11T09:46:42.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.235+0000 7f2bedffb700 1 --2- 172.21.15.27:0/3950436851 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2bdc067170 0x7f2bdc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:42.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.235+0000 7f2bedffb700 1 -- 172.21.15.27:0/3950436851 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f2bf4084740 con 0x7f2bf8069540 2024-01-11T09:46:42.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.236+0000 7f2bfd39a700 1 --2- 172.21.15.27:0/3950436851 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2bdc067170 0x7f2bdc069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:42.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.236+0000 7f2bfd39a700 1 --2- 172.21.15.27:0/3950436851 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2bdc067170 0x7f2bdc069620 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2bf806d560 tx=0x7f2be8017000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:42.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.239+0000 7f2bedffb700 1 -- 172.21.15.27:0/3950436851 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2bf40513d0 con 0x7f2bf8069540 2024-01-11T09:46:42.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:42.491+0000 7f2bffdff700 1 -- 172.21.15.27:0/3950436851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2bf815f4a0 con 0x7f2bf8069540 2024-01-11T09:46:42.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.003+0000 7f2bedffb700 1 -- 172.21.15.27:0/3950436851 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v132) v1 ==== 125+0+0 (secure 0 0 0) 0x7f2bf40549f0 con 0x7f2bf8069540 2024-01-11T09:46:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.006+0000 7f2bffdff700 1 -- 172.21.15.27:0/3950436851 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2bdc067170 msgr2=0x7f2bdc069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.006+0000 7f2bffdff700 1 --2- 172.21.15.27:0/3950436851 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2bdc067170 0x7f2bdc069620 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2bf806d560 tx=0x7f2be8017000 comp rx=0 tx=0).stop 2024-01-11T09:46:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.006+0000 7f2bffdff700 1 -- 172.21.15.27:0/3950436851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bf8069540 msgr2=0x7f2bf80c3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.006+0000 7f2bffdff700 1 --2- 172.21.15.27:0/3950436851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bf8069540 0x7f2bf80c3c80 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400c810 tx=0x7f2bf4007d20 comp rx=0 tx=0).stop 2024-01-11T09:46:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.006+0000 7f2bffdff700 1 -- 172.21.15.27:0/3950436851 shutdown_connections 2024-01-11T09:46:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.006+0000 7f2bffdff700 1 --2- 172.21.15.27:0/3950436851 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2bdc067170 0x7f2bdc069620 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2bf806d560 tx=0x7f2be8017000 comp rx=0 tx=0).stop 2024-01-11T09:46:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.006+0000 7f2bffdff700 1 --2- 172.21.15.27:0/3950436851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf81577e0 0x7f2bf814e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.006+0000 7f2bffdff700 1 --2- 172.21.15.27:0/3950436851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bf8069540 0x7f2bf80c3c80 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.006+0000 7f2bffdff700 1 --2- 172.21.15.27:0/3950436851 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2bf8157410 0x7f2bf80cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.006+0000 7f2bffdff700 1 -- 172.21.15.27:0/3950436851 >> 172.21.15.27:0/3950436851 conn(0x7f2bf80680a0 msgr2=0x7f2bf80c94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.006+0000 7f2bffdff700 1 -- 172.21.15.27:0/3950436851 shutdown_connections 2024-01-11T09:46:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.006+0000 7f2bffdff700 1 -- 172.21.15.27:0/3950436851 wait complete. 2024-01-11T09:46:43.018 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:46:43.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.227+0000 7f389d1ff700 1 Processor -- start 2024-01-11T09:46:43.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.227+0000 7f389d1ff700 1 -- start start 2024-01-11T09:46:43.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.228+0000 7f389d1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f38980b0500 0x7f389814fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:43.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.228+0000 7f389d1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3898150160 0x7f38981599e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:43.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.228+0000 7f389d1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3898157e30 con 0x7f38980b0130 2024-01-11T09:46:43.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.228+0000 7f389d1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f389806a450 con 0x7f38980b0500 2024-01-11T09:46:43.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.228+0000 7f389d1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3898157500 con 0x7f3898150160 2024-01-11T09:46:43.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.229+0000 7f3896d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f38980b0130 0x7f3898069d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48190/0 (socket says 172.21.15.27:48190) 2024-01-11T09:46:43.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.229+0000 7f3896d9d700 1 -- 172.21.15.27:0/3638500192 learned_addr learned my addr 172.21.15.27:0/3638500192 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:43.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.229+0000 7f389759e700 1 --2- 172.21.15.27:0/3638500192 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3898150160 0x7f38981599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:43.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.229+0000 7f389659c700 1 --2- 172.21.15.27:0/3638500192 >> v2:172.21.15.77:3300/0 conn(0x7f38980b0500 0x7f389814fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:43.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.229+0000 7f3895d9b700 1 -- 172.21.15.27:0/3638500192 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693610419 0 0) 0x7f3898157e30 con 0x7f38980b0130 2024-01-11T09:46:43.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.229+0000 7f3895d9b700 1 -- 172.21.15.27:0/3638500192 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3880003570 con 0x7f38980b0130 2024-01-11T09:46:43.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.229+0000 7f3895d9b700 1 -- 172.21.15.27:0/3638500192 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4024345514 0 0) 0x7f3880003570 con 0x7f38980b0130 2024-01-11T09:46:43.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.229+0000 7f3895d9b700 1 -- 172.21.15.27:0/3638500192 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3898150160 msgr2=0x7f38981599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:43.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.229+0000 7f3895d9b700 1 --2- 172.21.15.27:0/3638500192 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3898150160 0x7f38981599e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.229+0000 7f3895d9b700 1 -- 172.21.15.27:0/3638500192 >> v2:172.21.15.77:3300/0 conn(0x7f38980b0500 msgr2=0x7f389814fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:43.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.229+0000 7f3895d9b700 1 --2- 172.21.15.27:0/3638500192 >> v2:172.21.15.77:3300/0 conn(0x7f38980b0500 0x7f389814fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.229+0000 7f3895d9b700 1 -- 172.21.15.27:0/3638500192 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38981577e0 con 0x7f38980b0130 2024-01-11T09:46:43.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.230+0000 7f3895d9b700 1 -- 172.21.15.27:0/3638500192 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f3888001470 con 0x7f38980b0130 2024-01-11T09:46:43.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.230+0000 7f3895d9b700 1 -- 172.21.15.27:0/3638500192 >> v1:172.21.15.27:6789/0 conn(0x7f38980b0130 legacy=0x7f3898069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:43.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.230+0000 7f3895d9b700 1 --2- 172.21.15.27:0/3638500192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3880003a60 0x7f3880003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:43.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.230+0000 7f3895d9b700 1 --2- 172.21.15.27:0/3638500192 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3898150160 0x7f3880004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:43.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.230+0000 7f3895d9b700 1 --2- 172.21.15.27:0/3638500192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3880004d50 0x7f38800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:43.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.230+0000 7f3895d9b700 1 -- 172.21.15.27:0/3638500192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389806a450 con 0x7f3880003a60 2024-01-11T09:46:43.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.230+0000 7f3895d9b700 1 -- 172.21.15.27:0/3638500192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3898157500 con 0x7f3880004d50 2024-01-11T09:46:43.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.230+0000 7f3895d9b700 1 -- 172.21.15.27:0/3638500192 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f388000ba60 con 0x7f3898150160 2024-01-11T09:46:43.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.230+0000 7f3896d9d700 1 --2- 172.21.15.27:0/3638500192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3880003a60 0x7f3880003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:43.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.230+0000 7f389659c700 1 --2- 172.21.15.27:0/3638500192 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3898150160 0x7f3880004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:43.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.230+0000 7f389759e700 1 --2- 172.21.15.27:0/3638500192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3880004d50 0x7f38800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:43.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.230+0000 7f3896d9d700 1 -- 172.21.15.27:0/3638500192 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3898150160 msgr2=0x7f3880004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:43.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.230+0000 7f3896d9d700 1 --2- 172.21.15.27:0/3638500192 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3898150160 0x7f3880004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.230+0000 7f3896d9d700 1 -- 172.21.15.27:0/3638500192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3880004d50 msgr2=0x7f38800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:43.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.230+0000 7f3896d9d700 1 --2- 172.21.15.27:0/3638500192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3880004d50 0x7f38800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.230+0000 7f3896d9d700 1 -- 172.21.15.27:0/3638500192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3880003570 con 0x7f3880003a60 2024-01-11T09:46:43.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.230+0000 7f389759e700 1 --2- 172.21.15.27:0/3638500192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3880004d50 0x7f38800051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:43.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.231+0000 7f3896d9d700 1 --2- 172.21.15.27:0/3638500192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3880003a60 0x7f3880003e70 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f388800c3c0 tx=0x7f3888001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40f74ca153b3d2c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:43.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.231+0000 7f3895d9b700 1 -- 172.21.15.27:0/3638500192 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388800cc30 con 0x7f3880003a60 2024-01-11T09:46:43.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.231+0000 7f3895d9b700 1 -- 172.21.15.27:0/3638500192 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388800cdd0 con 0x7f3880003a60 2024-01-11T09:46:43.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.231+0000 7f3895d9b700 1 -- 172.21.15.27:0/3638500192 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3888005b70 con 0x7f3880003a60 2024-01-11T09:46:43.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.231+0000 7f389d1ff700 1 -- 172.21.15.27:0/3638500192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3880003a60 msgr2=0x7f3880003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:43.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.231+0000 7f389d1ff700 1 --2- 172.21.15.27:0/3638500192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3880003a60 0x7f3880003e70 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f388800c3c0 tx=0x7f3888001e90 comp rx=0 tx=0).stop 2024-01-11T09:46:43.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.231+0000 7f389759e700 1 -- 172.21.15.27:0/3638500192 reap_dead start 2024-01-11T09:46:43.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.231+0000 7f389d1ff700 1 -- 172.21.15.27:0/3638500192 shutdown_connections 2024-01-11T09:46:43.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.232+0000 7f389d1ff700 1 -- 172.21.15.27:0/3638500192 >> 172.21.15.27:0/3638500192 conn(0x7f38980680e0 msgr2=0x7f3898069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:43.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.232+0000 7f389d1ff700 1 -- 172.21.15.27:0/3638500192 shutdown_connections 2024-01-11T09:46:43.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.232+0000 7f389d1ff700 1 -- 172.21.15.27:0/3638500192 wait complete. 2024-01-11T09:46:43.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.233+0000 7f389d1ff700 1 Processor -- start 2024-01-11T09:46:43.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.233+0000 7f389d1ff700 1 -- start start 2024-01-11T09:46:43.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.234+0000 7f389d1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3880003a60 0x7f389816a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:43.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.234+0000 7f389d1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f389816a210 0x7f3898155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:43.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.234+0000 7f389d1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3898154050 0x7f389816c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:43.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.234+0000 7f389d1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389816a680 con 0x7f389816a210 2024-01-11T09:46:43.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.234+0000 7f389d1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3898156590 con 0x7f3898154050 2024-01-11T09:46:43.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.234+0000 7f389d1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3898156870 con 0x7f3880003a60 2024-01-11T09:46:43.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.235+0000 7f389659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f389816a210 0x7f3898155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:43.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.235+0000 7f389659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f389816a210 0x7f3898155d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40456/0 (socket says 172.21.15.27:40456) 2024-01-11T09:46:43.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.235+0000 7f389659c700 1 -- 172.21.15.27:0/1274774227 learned_addr learned my addr 172.21.15.27:0/1274774227 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:43.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.235+0000 7f3896d9d700 1 --2- 172.21.15.27:0/1274774227 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3880003a60 0x7f389816a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:43.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.235+0000 7f389759e700 1 --2- 172.21.15.27:0/1274774227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3898154050 0x7f389816c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:43.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.235+0000 7f389659c700 1 -- 172.21.15.27:0/1274774227 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3880003a60 msgr2=0x7f389816a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:43.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.235+0000 7f389659c700 1 --2- 172.21.15.27:0/1274774227 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3880003a60 0x7f389816a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.235+0000 7f389659c700 1 -- 172.21.15.27:0/1274774227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3898154050 msgr2=0x7f389816c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:43.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.235+0000 7f389659c700 1 --2- 172.21.15.27:0/1274774227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3898154050 0x7f389816c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.235+0000 7f389659c700 1 -- 172.21.15.27:0/1274774227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3898156b50 con 0x7f389816a210 2024-01-11T09:46:43.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.235+0000 7f389659c700 1 --2- 172.21.15.27:0/1274774227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f389816a210 0x7f3898155d40 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f388c011d40 tx=0x7f388c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:43.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.236+0000 7f387effd700 1 -- 172.21.15.27:0/1274774227 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c00f990 con 0x7f389816a210 2024-01-11T09:46:43.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.236+0000 7f387effd700 1 -- 172.21.15.27:0/1274774227 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388c00faf0 con 0x7f389816a210 2024-01-11T09:46:43.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.236+0000 7f387effd700 1 -- 172.21.15.27:0/1274774227 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c00feb0 con 0x7f389816a210 2024-01-11T09:46:43.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.236+0000 7f389d1ff700 1 -- 172.21.15.27:0/1274774227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3898156e30 con 0x7f389816a210 2024-01-11T09:46:43.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.236+0000 7f389d1ff700 1 -- 172.21.15.27:0/1274774227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3898156280 con 0x7f389816a210 2024-01-11T09:46:43.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.237+0000 7f389d1ff700 1 -- 172.21.15.27:0/1274774227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38980c9cb0 con 0x7f389816a210 2024-01-11T09:46:43.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.238+0000 7f387effd700 1 -- 172.21.15.27:0/1274774227 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f388c014650 con 0x7f389816a210 2024-01-11T09:46:43.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.239+0000 7f387effd700 1 --2- 172.21.15.27:0/1274774227 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f38840671a0 0x7f3884069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:43.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.239+0000 7f387effd700 1 -- 172.21.15.27:0/1274774227 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f388c018030 con 0x7f389816a210 2024-01-11T09:46:43.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.239+0000 7f3896d9d700 1 --2- 172.21.15.27:0/1274774227 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f38840671a0 0x7f3884069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:43.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.240+0000 7f3896d9d700 1 --2- 172.21.15.27:0/1274774227 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f38840671a0 0x7f3884069650 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f388800c3c0 tx=0x7f3888015000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:43.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.243+0000 7f387effd700 1 -- 172.21.15.27:0/1274774227 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f388c0524f0 con 0x7f389816a210 2024-01-11T09:46:43.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.488+0000 7f389d1ff700 1 -- 172.21.15.27:0/1274774227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38980718d0 con 0x7f389816a210 2024-01-11T09:46:43.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.489+0000 7f387effd700 1 -- 172.21.15.27:0/1274774227 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5688 (secure 0 0 0) 0x7f388c055b10 con 0x7f389816a210 2024-01-11T09:46:43.510 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:43.511 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":132,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1568813392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1568813392},{"type":"v1","addr":"172.21.15.77:6834","nonce":1568813392}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122},{"gid":10969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1352491019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1352491019},{"type":"v1","addr":"172.21.15.77:6835","nonce":1352491019}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:46:25.726042+0000","modified":"2024-01-11T09:46:29.257798+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11340":{"gid":11340,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/321681407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":321681407},{"type":"v1","addr":"172.21.15.27:6835","nonce":321681407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:46:39.483941+0000","modified":"2024-01-11T09:46:42.996786+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11331},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11331":{"gid":11331,"name":"a","rank":0,"incarnation":129,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6834/3477661846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3477661846},{"type":"v1","addr":"172.21.15.27:6834","nonce":3477661846}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":16}]} 2024-01-11T09:46:43.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.491+0000 7f389d1ff700 1 -- 172.21.15.27:0/1274774227 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f38840671a0 msgr2=0x7f3884069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:43.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.491+0000 7f389d1ff700 1 --2- 172.21.15.27:0/1274774227 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f38840671a0 0x7f3884069650 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f388800c3c0 tx=0x7f3888015000 comp rx=0 tx=0).stop 2024-01-11T09:46:43.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.491+0000 7f389d1ff700 1 -- 172.21.15.27:0/1274774227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f389816a210 msgr2=0x7f3898155d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:43.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.491+0000 7f389d1ff700 1 --2- 172.21.15.27:0/1274774227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f389816a210 0x7f3898155d40 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f388c011d40 tx=0x7f388c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.491+0000 7f389d1ff700 1 -- 172.21.15.27:0/1274774227 shutdown_connections 2024-01-11T09:46:43.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.491+0000 7f389d1ff700 1 --2- 172.21.15.27:0/1274774227 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f38840671a0 0x7f3884069650 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.491+0000 7f389d1ff700 1 --2- 172.21.15.27:0/1274774227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3898154050 0x7f389816c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.491+0000 7f389d1ff700 1 --2- 172.21.15.27:0/1274774227 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3880003a60 0x7f389816a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.491+0000 7f389d1ff700 1 --2- 172.21.15.27:0/1274774227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f389816a210 0x7f3898155d40 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.491+0000 7f389d1ff700 1 -- 172.21.15.27:0/1274774227 >> 172.21.15.27:0/1274774227 conn(0x7f38980680e0 msgr2=0x7f389814be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:43.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.491+0000 7f389d1ff700 1 -- 172.21.15.27:0/1274774227 shutdown_connections 2024-01-11T09:46:43.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.491+0000 7f389d1ff700 1 -- 172.21.15.27:0/1274774227 wait complete. 2024-01-11T09:46:43.513 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 132 2024-01-11T09:46:43.513 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:46:43.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.720+0000 7fd414dff700 1 Processor -- start 2024-01-11T09:46:43.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.720+0000 7fd414dff700 1 -- start start 2024-01-11T09:46:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.721+0000 7fd414dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4100b09e0 0x7fd4101610a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.721+0000 7fd414dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd410157390 con 0x7fd410069d40 2024-01-11T09:46:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.721+0000 7fd414dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd41006a200 con 0x7fd4100c8620 2024-01-11T09:46:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.721+0000 7fd414dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4100adcd0 con 0x7fd4100b09e0 2024-01-11T09:46:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.722+0000 7fd40dd9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd410069d40 0x7fd4100b02d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48198/0 (socket says 172.21.15.27:48198) 2024-01-11T09:46:43.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.722+0000 7fd40dd9b700 1 -- 172.21.15.27:0/1263919391 learned_addr learned my addr 172.21.15.27:0/1263919391 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:43.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.722+0000 7fd40ed9d700 1 --2- 172.21.15.27:0/1263919391 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4100b09e0 0x7fd4101610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:43.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.722+0000 7fd40d59a700 1 -- 172.21.15.27:0/1263919391 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364796320 0 0) 0x7fd410157390 con 0x7fd410069d40 2024-01-11T09:46:43.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.722+0000 7fd40d59a700 1 -- 172.21.15.27:0/1263919391 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f8003570 con 0x7fd410069d40 2024-01-11T09:46:43.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.722+0000 7fd40d59a700 1 -- 172.21.15.27:0/1263919391 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8540855 0 0) 0x7fd41006a200 con 0x7fd4100c8620 2024-01-11T09:46:43.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.722+0000 7fd40d59a700 1 -- 172.21.15.27:0/1263919391 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd410157390 con 0x7fd4100c8620 2024-01-11T09:46:43.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.722+0000 7fd40d59a700 1 -- 172.21.15.27:0/1263919391 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 411340878 0 0) 0x7fd3f8003570 con 0x7fd410069d40 2024-01-11T09:46:43.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.722+0000 7fd40d59a700 1 -- 172.21.15.27:0/1263919391 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4100b09e0 msgr2=0x7fd4101610a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:43.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.722+0000 7fd40d59a700 1 --2- 172.21.15.27:0/1263919391 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4100b09e0 0x7fd4101610a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.722+0000 7fd40d59a700 1 -- 172.21.15.27:0/1263919391 >> v1:172.21.15.77:6789/0 conn(0x7fd4100c8620 legacy=0x7fd4100af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:43.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.722+0000 7fd40d59a700 1 -- 172.21.15.27:0/1263919391 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd410156d40 con 0x7fd410069d40 2024-01-11T09:46:43.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.722+0000 7fd40d59a700 1 -- 172.21.15.27:0/1263919391 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fd404001470 con 0x7fd410069d40 2024-01-11T09:46:43.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.722+0000 7fd40ed9d700 1 --2- 172.21.15.27:0/1263919391 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4100b09e0 0x7fd4101610a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:43.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.723+0000 7fd40d59a700 1 -- 172.21.15.27:0/1263919391 >> v1:172.21.15.27:6789/0 conn(0x7fd410069d40 legacy=0x7fd4100b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:43.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.723+0000 7fd40d59a700 1 --2- 172.21.15.27:0/1263919391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3f8003c20 0x7fd3f8004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:43.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.723+0000 7fd40d59a700 1 --2- 172.21.15.27:0/1263919391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3f8004810 0x7fd3f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:43.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.723+0000 7fd40d59a700 1 --2- 172.21.15.27:0/1263919391 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4100b09e0 0x7fd3f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:43.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.723+0000 7fd40d59a700 1 -- 172.21.15.27:0/1263919391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4100adcd0 con 0x7fd3f8003c20 2024-01-11T09:46:43.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.723+0000 7fd40d59a700 1 -- 172.21.15.27:0/1263919391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f800bb10 con 0x7fd3f8004810 2024-01-11T09:46:43.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.723+0000 7fd40d59a700 1 -- 172.21.15.27:0/1263919391 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f800dee0 con 0x7fd4100b09e0 2024-01-11T09:46:43.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.723+0000 7fd40e59c700 1 --2- 172.21.15.27:0/1263919391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3f8003c20 0x7fd3f8004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:43.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.723+0000 7fd40dd9b700 1 --2- 172.21.15.27:0/1263919391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3f8004810 0x7fd3f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:43.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.723+0000 7fd40ed9d700 1 --2- 172.21.15.27:0/1263919391 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4100b09e0 0x7fd3f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:43.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.723+0000 7fd40e59c700 1 -- 172.21.15.27:0/1263919391 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4100b09e0 msgr2=0x7fd3f80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:43.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.723+0000 7fd40e59c700 1 --2- 172.21.15.27:0/1263919391 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4100b09e0 0x7fd3f80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.723+0000 7fd40e59c700 1 -- 172.21.15.27:0/1263919391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3f8004810 msgr2=0x7fd3f8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:43.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.723+0000 7fd40e59c700 1 --2- 172.21.15.27:0/1263919391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3f8004810 0x7fd3f8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.723+0000 7fd40e59c700 1 -- 172.21.15.27:0/1263919391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd408009a20 con 0x7fd3f8003c20 2024-01-11T09:46:43.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.723+0000 7fd40ed9d700 1 --2- 172.21.15.27:0/1263919391 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd4100b09e0 0x7fd3f80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:43.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.724+0000 7fd40e59c700 1 --2- 172.21.15.27:0/1263919391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3f8003c20 0x7fd3f8004010 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fd400001a10 tx=0x7fd400007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7986680f681bc7e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:43.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.724+0000 7fd40d59a700 1 -- 172.21.15.27:0/1263919391 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40000c5d0 con 0x7fd3f8003c20 2024-01-11T09:46:43.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.724+0000 7fd40d59a700 1 -- 172.21.15.27:0/1263919391 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40000c730 con 0x7fd3f8003c20 2024-01-11T09:46:43.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.724+0000 7fd40d59a700 1 -- 172.21.15.27:0/1263919391 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40000cad0 con 0x7fd3f8003c20 2024-01-11T09:46:43.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.724+0000 7fd414dff700 1 -- 172.21.15.27:0/1263919391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3f8003c20 msgr2=0x7fd3f8004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:43.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.724+0000 7fd414dff700 1 --2- 172.21.15.27:0/1263919391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3f8003c20 0x7fd3f8004010 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fd400001a10 tx=0x7fd400007d30 comp rx=0 tx=0).stop 2024-01-11T09:46:43.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.724+0000 7fd40ed9d700 1 -- 172.21.15.27:0/1263919391 reap_dead start 2024-01-11T09:46:43.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.724+0000 7fd414dff700 1 -- 172.21.15.27:0/1263919391 shutdown_connections 2024-01-11T09:46:43.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.724+0000 7fd414dff700 1 -- 172.21.15.27:0/1263919391 >> 172.21.15.27:0/1263919391 conn(0x7fd4100680e0 msgr2=0x7fd410069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:43.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.725+0000 7fd414dff700 1 -- 172.21.15.27:0/1263919391 shutdown_connections 2024-01-11T09:46:43.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.725+0000 7fd414dff700 1 -- 172.21.15.27:0/1263919391 wait complete. 2024-01-11T09:46:43.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.726+0000 7fd414dff700 1 Processor -- start 2024-01-11T09:46:43.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.726+0000 7fd414dff700 1 -- start start 2024-01-11T09:46:43.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.727+0000 7fd414dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd3f8003c20 0x7fd410153a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:43.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.727+0000 7fd414dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4101662b0 0x7fd410168e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:43.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.727+0000 7fd414dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd410149660 0x7fd4100af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:43.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.727+0000 7fd414dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd410166720 con 0x7fd410149660 2024-01-11T09:46:43.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.727+0000 7fd414dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4101696b0 con 0x7fd4101662b0 2024-01-11T09:46:43.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.727+0000 7fd414dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd410169990 con 0x7fd3f8003c20 2024-01-11T09:46:43.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.727+0000 7fd40ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd410149660 0x7fd4100af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.727+0000 7fd40ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd410149660 0x7fd4100af440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40468/0 (socket says 172.21.15.27:40468) 2024-01-11T09:46:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.727+0000 7fd40ed9d700 1 -- 172.21.15.27:0/2477969657 learned_addr learned my addr 172.21.15.27:0/2477969657 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.727+0000 7fd40dd9b700 1 --2- 172.21.15.27:0/2477969657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4101662b0 0x7fd410168e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.728+0000 7fd40e59c700 1 --2- 172.21.15.27:0/2477969657 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd3f8003c20 0x7fd410153a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.728+0000 7fd40ed9d700 1 -- 172.21.15.27:0/2477969657 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd3f8003c20 msgr2=0x7fd410153a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.728+0000 7fd40ed9d700 1 --2- 172.21.15.27:0/2477969657 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd3f8003c20 0x7fd410153a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.728+0000 7fd40ed9d700 1 -- 172.21.15.27:0/2477969657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4101662b0 msgr2=0x7fd410168e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.728+0000 7fd40ed9d700 1 --2- 172.21.15.27:0/2477969657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4101662b0 0x7fd410168e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.728+0000 7fd40ed9d700 1 -- 172.21.15.27:0/2477969657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd410152070 con 0x7fd410149660 2024-01-11T09:46:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.728+0000 7fd40e59c700 1 --2- 172.21.15.27:0/2477969657 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd3f8003c20 0x7fd410153a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:43.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.728+0000 7fd40ed9d700 1 --2- 172.21.15.27:0/2477969657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd410149660 0x7fd4100af440 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fd410070d50 tx=0x7fd40800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.728+0000 7fd3fe7fc700 1 -- 172.21.15.27:0/2477969657 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40800e070 con 0x7fd410149660 2024-01-11T09:46:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.728+0000 7fd3fe7fc700 1 -- 172.21.15.27:0/2477969657 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd408011d00 con 0x7fd410149660 2024-01-11T09:46:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.728+0000 7fd3fe7fc700 1 -- 172.21.15.27:0/2477969657 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4080166c0 con 0x7fd410149660 2024-01-11T09:46:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.728+0000 7fd414dff700 1 -- 172.21.15.27:0/2477969657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd410152350 con 0x7fd410149660 2024-01-11T09:46:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.728+0000 7fd414dff700 1 -- 172.21.15.27:0/2477969657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd410149b00 con 0x7fd410149660 2024-01-11T09:46:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.730+0000 7fd414dff700 1 -- 172.21.15.27:0/2477969657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd410152540 con 0x7fd410149660 2024-01-11T09:46:43.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.731+0000 7fd3fe7fc700 1 -- 172.21.15.27:0/2477969657 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fd408016ab0 con 0x7fd410149660 2024-01-11T09:46:43.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.731+0000 7fd3fe7fc700 1 --2- 172.21.15.27:0/2477969657 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd3f00671a0 0x7fd3f0069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:43.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.732+0000 7fd3fe7fc700 1 -- 172.21.15.27:0/2477969657 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fd4080859f0 con 0x7fd410149660 2024-01-11T09:46:43.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.732+0000 7fd40e59c700 1 --2- 172.21.15.27:0/2477969657 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd3f00671a0 0x7fd3f0069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:43.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.733+0000 7fd40e59c700 1 --2- 172.21.15.27:0/2477969657 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd3f00671a0 0x7fd3f0069650 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd400001ed0 tx=0x7fd400015000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:43.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.735+0000 7fd3fe7fc700 1 -- 172.21.15.27:0/2477969657 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd4080524f0 con 0x7fd410149660 2024-01-11T09:46:43.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.940+0000 7fd414dff700 1 -- 172.21.15.27:0/2477969657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4100718d0 con 0x7fd410149660 2024-01-11T09:46:43.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.941+0000 7fd3fe7fc700 1 -- 172.21.15.27:0/2477969657 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v239) v1 ==== 74+0+18727 (secure 0 0 0) 0x7fd408055b10 con 0x7fd410149660 2024-01-11T09:46:43.934 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:43.934 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":239,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:46:40.984944+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:46:24.049904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-01-11T09:46:24.737495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-01-11T09:46:37.665603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"238","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-01-11T09:46:38.472134+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"239","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:46:43.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.944+0000 7fd414dff700 1 -- 172.21.15.27:0/2477969657 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd3f00671a0 msgr2=0x7fd3f0069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:43.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.944+0000 7fd414dff700 1 --2- 172.21.15.27:0/2477969657 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd3f00671a0 0x7fd3f0069650 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd400001ed0 tx=0x7fd400015000 comp rx=0 tx=0).stop 2024-01-11T09:46:43.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.944+0000 7fd414dff700 1 -- 172.21.15.27:0/2477969657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd410149660 msgr2=0x7fd4100af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:43.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.944+0000 7fd414dff700 1 --2- 172.21.15.27:0/2477969657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd410149660 0x7fd4100af440 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fd410070d50 tx=0x7fd40800b040 comp rx=0 tx=0).stop 2024-01-11T09:46:43.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.944+0000 7fd414dff700 1 -- 172.21.15.27:0/2477969657 shutdown_connections 2024-01-11T09:46:43.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.944+0000 7fd414dff700 1 --2- 172.21.15.27:0/2477969657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd410149660 0x7fd4100af440 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.944+0000 7fd414dff700 1 --2- 172.21.15.27:0/2477969657 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd3f8003c20 0x7fd410153a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.944+0000 7fd414dff700 1 --2- 172.21.15.27:0/2477969657 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd3f00671a0 0x7fd3f0069650 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.944+0000 7fd414dff700 1 --2- 172.21.15.27:0/2477969657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4101662b0 0x7fd410168e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:43.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.944+0000 7fd414dff700 1 -- 172.21.15.27:0/2477969657 >> 172.21.15.27:0/2477969657 conn(0x7fd4100680e0 msgr2=0x7fd4100695b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:43.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.944+0000 7fd414dff700 1 -- 172.21.15.27:0/2477969657 shutdown_connections 2024-01-11T09:46:43.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:43.944+0000 7fd414dff700 1 -- 172.21.15.27:0/2477969657 wait complete. 2024-01-11T09:46:43.954 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-11T09:46:43.955 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:46:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.163+0000 7f079f5ff700 1 Processor -- start 2024-01-11T09:46:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.163+0000 7f079f5ff700 1 -- start start 2024-01-11T09:46:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.164+0000 7f079f5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f07980b2b90 0x7f0798159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.164+0000 7f079f5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07980b07b0 0x7f0798159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.164+0000 7f079f5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07980cd730 con 0x7f07980b03e0 2024-01-11T09:46:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.164+0000 7f079f5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07980ccb20 con 0x7f07980b2b90 2024-01-11T09:46:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.164+0000 7f079f5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07980cce00 con 0x7f07980b07b0 2024-01-11T09:46:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.164+0000 7f079d39b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f07980b03e0 0x7f07980b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35098/0 (socket says 172.21.15.27:35098) 2024-01-11T09:46:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.164+0000 7f079d39b700 1 -- 172.21.15.27:0/3091857398 learned_addr learned my addr 172.21.15.27:0/3091857398 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:44.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.165+0000 7f079cb9a700 1 --2- 172.21.15.27:0/3091857398 >> v2:172.21.15.77:3300/0 conn(0x7f07980b2b90 0x7f0798159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:44.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.165+0000 7f079db9c700 1 --2- 172.21.15.27:0/3091857398 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07980b07b0 0x7f0798159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:44.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.165+0000 7f078ffff700 1 -- 172.21.15.27:0/3091857398 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566295600 0 0) 0x7f07980cd730 con 0x7f07980b03e0 2024-01-11T09:46:44.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.165+0000 7f078ffff700 1 -- 172.21.15.27:0/3091857398 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0784003570 con 0x7f07980b03e0 2024-01-11T09:46:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.165+0000 7f078ffff700 1 -- 172.21.15.27:0/3091857398 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3390233551 0 0) 0x7f0784003570 con 0x7f07980b03e0 2024-01-11T09:46:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.165+0000 7f079cb9a700 1 -- 172.21.15.27:0/3091857398 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07980b07b0 msgr2=0x7f0798159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.165+0000 7f079cb9a700 1 --2- 172.21.15.27:0/3091857398 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07980b07b0 0x7f0798159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.165+0000 7f079cb9a700 1 -- 172.21.15.27:0/3091857398 >> v1:172.21.15.27:6789/0 conn(0x7f07980b03e0 legacy=0x7f07980b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.165+0000 7f079cb9a700 1 -- 172.21.15.27:0/3091857398 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07980cd0e0 con 0x7f07980b2b90 2024-01-11T09:46:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.165+0000 7f079db9c700 1 --2- 172.21.15.27:0/3091857398 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07980b07b0 0x7f0798159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.166+0000 7f079cb9a700 1 --2- 172.21.15.27:0/3091857398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07980b2b90 0x7f0798159200 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f07880098e0 tx=0x7f07880091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8657186a66ade3a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.166+0000 7f078ffff700 1 -- 172.21.15.27:0/3091857398 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794001470 con 0x7f07980b2b90 2024-01-11T09:46:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.166+0000 7f078ffff700 1 -- 172.21.15.27:0/3091857398 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0788011420 con 0x7f07980b2b90 2024-01-11T09:46:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.166+0000 7f078ffff700 1 -- 172.21.15.27:0/3091857398 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07880117c0 con 0x7f07980b2b90 2024-01-11T09:46:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.166+0000 7f079f5ff700 1 -- 172.21.15.27:0/3091857398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07980b2b90 msgr2=0x7f0798159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.166+0000 7f079f5ff700 1 --2- 172.21.15.27:0/3091857398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07980b2b90 0x7f0798159200 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f07880098e0 tx=0x7f07880091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.166+0000 7f079f5ff700 1 -- 172.21.15.27:0/3091857398 shutdown_connections 2024-01-11T09:46:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.166+0000 7f079f5ff700 1 --2- 172.21.15.27:0/3091857398 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07980b07b0 0x7f0798159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.166+0000 7f079f5ff700 1 --2- 172.21.15.27:0/3091857398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07980b2b90 0x7f0798159200 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.166+0000 7f079f5ff700 1 -- 172.21.15.27:0/3091857398 >> 172.21.15.27:0/3091857398 conn(0x7f079806ed40 msgr2=0x7f079806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.167+0000 7f079f5ff700 1 -- 172.21.15.27:0/3091857398 shutdown_connections 2024-01-11T09:46:44.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.167+0000 7f079f5ff700 1 -- 172.21.15.27:0/3091857398 wait complete. 2024-01-11T09:46:44.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.168+0000 7f079f5ff700 1 Processor -- start 2024-01-11T09:46:44.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.168+0000 7f079f5ff700 1 -- start start 2024-01-11T09:46:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.169+0000 7f079f5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07980b03e0 0x7f07981476a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.169+0000 7f079f5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07980b07b0 0x7f079815c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.169+0000 7f079f5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07980b2b90 0x7f079815c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.169+0000 7f079f5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07980ca6b0 con 0x7f07980b2b90 2024-01-11T09:46:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.169+0000 7f079f5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07980ca990 con 0x7f07980b03e0 2024-01-11T09:46:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.169+0000 7f079f5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07980c3f40 con 0x7f07980b07b0 2024-01-11T09:46:44.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.169+0000 7f079db9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07980b2b90 0x7f079815c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.169+0000 7f079db9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07980b2b90 0x7f079815c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52554/0 (socket says 172.21.15.27:52554) 2024-01-11T09:46:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.169+0000 7f079db9c700 1 -- 172.21.15.27:0/183026893 learned_addr learned my addr 172.21.15.27:0/183026893 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.169+0000 7f079d39b700 1 --2- 172.21.15.27:0/183026893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07980b03e0 0x7f07981476a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.169+0000 7f079cb9a700 1 --2- 172.21.15.27:0/183026893 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07980b07b0 0x7f079815c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.170+0000 7f079db9c700 1 -- 172.21.15.27:0/183026893 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07980b07b0 msgr2=0x7f079815c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.170+0000 7f079db9c700 1 --2- 172.21.15.27:0/183026893 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07980b07b0 0x7f079815c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.170+0000 7f079db9c700 1 -- 172.21.15.27:0/183026893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07980b03e0 msgr2=0x7f07981476a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.170+0000 7f079db9c700 1 --2- 172.21.15.27:0/183026893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07980b03e0 0x7f07981476a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.170+0000 7f079db9c700 1 -- 172.21.15.27:0/183026893 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0790009a20 con 0x7f07980b2b90 2024-01-11T09:46:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.170+0000 7f079db9c700 1 --2- 172.21.15.27:0/183026893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07980b2b90 0x7f079815c770 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f0790009fc0 tx=0x7f079000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.170+0000 7f078d7fa700 1 -- 172.21.15.27:0/183026893 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079000cbd0 con 0x7f07980b2b90 2024-01-11T09:46:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.170+0000 7f078d7fa700 1 -- 172.21.15.27:0/183026893 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079000f040 con 0x7f07980b2b90 2024-01-11T09:46:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.170+0000 7f078d7fa700 1 -- 172.21.15.27:0/183026893 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0790014480 con 0x7f07980b2b90 2024-01-11T09:46:44.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.170+0000 7f079f5ff700 1 -- 172.21.15.27:0/183026893 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0788009420 con 0x7f07980b2b90 2024-01-11T09:46:44.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.171+0000 7f079f5ff700 1 -- 172.21.15.27:0/183026893 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07980c4500 con 0x7f07980b2b90 2024-01-11T09:46:44.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.172+0000 7f079f5ff700 1 -- 172.21.15.27:0/183026893 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07980c46f0 con 0x7f07980b2b90 2024-01-11T09:46:44.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.173+0000 7f078d7fa700 1 -- 172.21.15.27:0/183026893 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f0790003020 con 0x7f07980b2b90 2024-01-11T09:46:44.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.173+0000 7f078d7fa700 1 --2- 172.21.15.27:0/183026893 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f077c067170 0x7f077c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:44.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.174+0000 7f078d7fa700 1 -- 172.21.15.27:0/183026893 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f0790085760 con 0x7f07980b2b90 2024-01-11T09:46:44.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.174+0000 7f079d39b700 1 --2- 172.21.15.27:0/183026893 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f077c067170 0x7f077c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:44.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.175+0000 7f079d39b700 1 --2- 172.21.15.27:0/183026893 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f077c067170 0x7f077c069620 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0794001ed0 tx=0x7f07940019d0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:44.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.178+0000 7f078d7fa700 1 -- 172.21.15.27:0/183026893 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0790052190 con 0x7f07980b2b90 2024-01-11T09:46:44.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.376+0000 7f079f5ff700 1 -- 172.21.15.27:0/183026893 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f07980718d0 con 0x7f077c067170 2024-01-11T09:46:44.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.392+0000 7f078d7fa700 1 -- 172.21.15.27:0/183026893 <== mgr.9107 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528629 (secure 0 0 0) 0x7f07980718d0 con 0x7f077c067170 2024-01-11T09:46:44.392 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:44.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.398+0000 7f079f5ff700 1 -- 172.21.15.27:0/183026893 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f077c067170 msgr2=0x7f077c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.398+0000 7f079f5ff700 1 --2- 172.21.15.27:0/183026893 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f077c067170 0x7f077c069620 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0794001ed0 tx=0x7f07940019d0 comp rx=0 tx=0).stop 2024-01-11T09:46:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.398+0000 7f079f5ff700 1 -- 172.21.15.27:0/183026893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07980b2b90 msgr2=0x7f079815c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.398+0000 7f079f5ff700 1 --2- 172.21.15.27:0/183026893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07980b2b90 0x7f079815c770 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f0790009fc0 tx=0x7f079000b040 comp rx=0 tx=0).stop 2024-01-11T09:46:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.398+0000 7f079f5ff700 1 -- 172.21.15.27:0/183026893 shutdown_connections 2024-01-11T09:46:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.398+0000 7f079f5ff700 1 --2- 172.21.15.27:0/183026893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07980b2b90 0x7f079815c770 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.398+0000 7f079f5ff700 1 --2- 172.21.15.27:0/183026893 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f077c067170 0x7f077c069620 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.398+0000 7f079f5ff700 1 --2- 172.21.15.27:0/183026893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07980b03e0 0x7f07981476a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.398+0000 7f079f5ff700 1 --2- 172.21.15.27:0/183026893 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f07980b07b0 0x7f079815c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.398+0000 7f079f5ff700 1 -- 172.21.15.27:0/183026893 >> 172.21.15.27:0/183026893 conn(0x7f079806ed40 msgr2=0x7f07980d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.398+0000 7f079f5ff700 1 -- 172.21.15.27:0/183026893 shutdown_connections 2024-01-11T09:46:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.399+0000 7f079f5ff700 1 -- 172.21.15.27:0/183026893 wait complete. 2024-01-11T09:46:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:46:44.409 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-01-11T09:46:44.161892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596193,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":742,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":467,"ondisk_log_size":467,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":247,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2452008,"kb_used_data":24552,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752522712,"statfs":{"total":773094113280,"available":770583257088,"internally_reserved":0,"allocated":25141248,"data_stored":12693590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3265,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001522"},"pg_stats":[{"pgid":"33.1","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.995461+0000","last_change":"2024-01-11T09:46:39.995461+0000","last_active":"2024-01-11T09:46:39.995461+0000","last_peered":"2024-01-11T09:46:39.995461+0000","last_clean":"2024-01-11T09:46:39.995461+0000","last_became_active":"2024-01-11T09:46:39.995177+0000","last_became_peered":"2024-01-11T09:46:39.995177+0000","last_unstale":"2024-01-11T09:46:39.995461+0000","last_undegraded":"2024-01-11T09:46:39.995461+0000","last_fullsized":"2024-01-11T09:46:39.995461+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:53:48.890166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.002322+0000","last_change":"2024-01-11T09:46:38.998858+0000","last_active":"2024-01-11T09:46:39.002322+0000","last_peered":"2024-01-11T09:46:39.002322+0000","last_clean":"2024-01-11T09:46:39.002322+0000","last_became_active":"2024-01-11T09:46:38.998596+0000","last_became_peered":"2024-01-11T09:46:38.998596+0000","last_unstale":"2024-01-11T09:46:39.002322+0000","last_undegraded":"2024-01-11T09:46:39.002322+0000","last_fullsized":"2024-01-11T09:46:39.002322+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:29:24.327601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3f","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.992628+0000","last_change":"2024-01-11T09:46:25.952190+0000","last_active":"2024-01-11T09:46:38.992628+0000","last_peered":"2024-01-11T09:46:38.992628+0000","last_clean":"2024-01-11T09:46:38.992628+0000","last_became_active":"2024-01-11T09:46:25.951733+0000","last_became_peered":"2024-01-11T09:46:25.951733+0000","last_unstale":"2024-01-11T09:46:38.992628+0000","last_undegraded":"2024-01-11T09:46:38.992628+0000","last_fullsized":"2024-01-11T09:46:38.992628+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:59:15.007682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.999859+0000","last_change":"2024-01-11T09:46:39.999859+0000","last_active":"2024-01-11T09:46:39.999859+0000","last_peered":"2024-01-11T09:46:39.999859+0000","last_clean":"2024-01-11T09:46:39.999859+0000","last_became_active":"2024-01-11T09:46:39.999334+0000","last_became_peered":"2024-01-11T09:46:39.999334+0000","last_unstale":"2024-01-11T09:46:39.999859+0000","last_undegraded":"2024-01-11T09:46:39.999859+0000","last_fullsized":"2024-01-11T09:46:39.999859+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:56:27.940894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.001251+0000","last_change":"2024-01-11T09:46:38.997477+0000","last_active":"2024-01-11T09:46:39.001251+0000","last_peered":"2024-01-11T09:46:39.001251+0000","last_clean":"2024-01-11T09:46:39.001251+0000","last_became_active":"2024-01-11T09:46:38.997202+0000","last_became_peered":"2024-01-11T09:46:38.997202+0000","last_unstale":"2024-01-11T09:46:39.001251+0000","last_undegraded":"2024-01-11T09:46:39.001251+0000","last_fullsized":"2024-01-11T09:46:39.001251+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:39:50.380011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990492+0000","last_change":"2024-01-11T09:46:25.957907+0000","last_active":"2024-01-11T09:46:38.990492+0000","last_peered":"2024-01-11T09:46:38.990492+0000","last_clean":"2024-01-11T09:46:38.990492+0000","last_became_active":"2024-01-11T09:46:25.956251+0000","last_became_peered":"2024-01-11T09:46:25.956251+0000","last_unstale":"2024-01-11T09:46:38.990492+0000","last_undegraded":"2024-01-11T09:46:38.990492+0000","last_fullsized":"2024-01-11T09:46:38.990492+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:53:35.139255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.995469+0000","last_change":"2024-01-11T09:46:39.995469+0000","last_active":"2024-01-11T09:46:39.995469+0000","last_peered":"2024-01-11T09:46:39.995469+0000","last_clean":"2024-01-11T09:46:39.995469+0000","last_became_active":"2024-01-11T09:46:39.995211+0000","last_became_peered":"2024-01-11T09:46:39.995211+0000","last_unstale":"2024-01-11T09:46:39.995469+0000","last_undegraded":"2024-01-11T09:46:39.995469+0000","last_fullsized":"2024-01-11T09:46:39.995469+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:11:21.719296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986698+0000","last_change":"2024-01-11T09:46:38.979782+0000","last_active":"2024-01-11T09:46:38.986698+0000","last_peered":"2024-01-11T09:46:38.986698+0000","last_clean":"2024-01-11T09:46:38.986698+0000","last_became_active":"2024-01-11T09:46:38.979495+0000","last_became_peered":"2024-01-11T09:46:38.979495+0000","last_unstale":"2024-01-11T09:46:38.986698+0000","last_undegraded":"2024-01-11T09:46:38.986698+0000","last_fullsized":"2024-01-11T09:46:38.986698+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:27:17.589227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.983955+0000","last_change":"2024-01-11T09:46:25.956076+0000","last_active":"2024-01-11T09:46:38.983955+0000","last_peered":"2024-01-11T09:46:38.983955+0000","last_clean":"2024-01-11T09:46:38.983955+0000","last_became_active":"2024-01-11T09:46:25.955819+0000","last_became_peered":"2024-01-11T09:46:25.955819+0000","last_unstale":"2024-01-11T09:46:38.983955+0000","last_undegraded":"2024-01-11T09:46:38.983955+0000","last_fullsized":"2024-01-11T09:46:38.983955+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:27:34.516236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.997183+0000","last_change":"2024-01-11T09:46:39.997183+0000","last_active":"2024-01-11T09:46:39.997183+0000","last_peered":"2024-01-11T09:46:39.997183+0000","last_clean":"2024-01-11T09:46:39.997183+0000","last_became_active":"2024-01-11T09:46:39.996739+0000","last_became_peered":"2024-01-11T09:46:39.996739+0000","last_unstale":"2024-01-11T09:46:39.997183+0000","last_undegraded":"2024-01-11T09:46:39.997183+0000","last_fullsized":"2024-01-11T09:46:39.997183+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:43:01.107206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.999835+0000","last_change":"2024-01-11T09:46:38.995869+0000","last_active":"2024-01-11T09:46:38.999835+0000","last_peered":"2024-01-11T09:46:38.999835+0000","last_clean":"2024-01-11T09:46:38.999835+0000","last_became_active":"2024-01-11T09:46:38.995586+0000","last_became_peered":"2024-01-11T09:46:38.995586+0000","last_unstale":"2024-01-11T09:46:38.999835+0000","last_undegraded":"2024-01-11T09:46:38.999835+0000","last_fullsized":"2024-01-11T09:46:38.999835+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:47:57.662961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984870+0000","last_change":"2024-01-11T09:46:25.947296+0000","last_active":"2024-01-11T09:46:38.984870+0000","last_peered":"2024-01-11T09:46:38.984870+0000","last_clean":"2024-01-11T09:46:38.984870+0000","last_became_active":"2024-01-11T09:46:25.946057+0000","last_became_peered":"2024-01-11T09:46:25.946057+0000","last_unstale":"2024-01-11T09:46:38.984870+0000","last_undegraded":"2024-01-11T09:46:38.984870+0000","last_fullsized":"2024-01-11T09:46:38.984870+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:15:19.562588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.015760+0000","last_change":"2024-01-11T09:46:40.015760+0000","last_active":"2024-01-11T09:46:40.015760+0000","last_peered":"2024-01-11T09:46:40.015760+0000","last_clean":"2024-01-11T09:46:40.015760+0000","last_became_active":"2024-01-11T09:46:40.015414+0000","last_became_peered":"2024-01-11T09:46:40.015414+0000","last_unstale":"2024-01-11T09:46:40.015760+0000","last_undegraded":"2024-01-11T09:46:40.015760+0000","last_fullsized":"2024-01-11T09:46:40.015760+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:27:00.735104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.002209+0000","last_change":"2024-01-11T09:46:38.998098+0000","last_active":"2024-01-11T09:46:39.002209+0000","last_peered":"2024-01-11T09:46:39.002209+0000","last_clean":"2024-01-11T09:46:39.002209+0000","last_became_active":"2024-01-11T09:46:38.997822+0000","last_became_peered":"2024-01-11T09:46:38.997822+0000","last_unstale":"2024-01-11T09:46:39.002209+0000","last_undegraded":"2024-01-11T09:46:39.002209+0000","last_fullsized":"2024-01-11T09:46:39.002209+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:51:50.752675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.989215+0000","last_change":"2024-01-11T09:46:25.953859+0000","last_active":"2024-01-11T09:46:38.989215+0000","last_peered":"2024-01-11T09:46:38.989215+0000","last_clean":"2024-01-11T09:46:38.989215+0000","last_became_active":"2024-01-11T09:46:25.953566+0000","last_became_peered":"2024-01-11T09:46:25.953566+0000","last_unstale":"2024-01-11T09:46:38.989215+0000","last_undegraded":"2024-01-11T09:46:38.989215+0000","last_fullsized":"2024-01-11T09:46:38.989215+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:42:55.688728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.001149+0000","last_change":"2024-01-11T09:46:40.001149+0000","last_active":"2024-01-11T09:46:40.001149+0000","last_peered":"2024-01-11T09:46:40.001149+0000","last_clean":"2024-01-11T09:46:40.001149+0000","last_became_active":"2024-01-11T09:46:40.000842+0000","last_became_peered":"2024-01-11T09:46:40.000842+0000","last_unstale":"2024-01-11T09:46:40.001149+0000","last_undegraded":"2024-01-11T09:46:40.001149+0000","last_fullsized":"2024-01-11T09:46:40.001149+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:30:34.408153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.997032+0000","last_change":"2024-01-11T09:46:38.992594+0000","last_active":"2024-01-11T09:46:38.997032+0000","last_peered":"2024-01-11T09:46:38.997032+0000","last_clean":"2024-01-11T09:46:38.997032+0000","last_became_active":"2024-01-11T09:46:38.992036+0000","last_became_peered":"2024-01-11T09:46:38.992036+0000","last_unstale":"2024-01-11T09:46:38.997032+0000","last_undegraded":"2024-01-11T09:46:38.997032+0000","last_fullsized":"2024-01-11T09:46:38.997032+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:27:17.589227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990993+0000","last_change":"2024-01-11T09:46:25.945778+0000","last_active":"2024-01-11T09:46:38.990993+0000","last_peered":"2024-01-11T09:46:38.990993+0000","last_clean":"2024-01-11T09:46:38.990993+0000","last_became_active":"2024-01-11T09:46:25.944907+0000","last_became_peered":"2024-01-11T09:46:25.944907+0000","last_unstale":"2024-01-11T09:46:38.990993+0000","last_undegraded":"2024-01-11T09:46:38.990993+0000","last_fullsized":"2024-01-11T09:46:38.990993+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:59:02.206221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.002961+0000","last_change":"2024-01-11T09:46:40.002961+0000","last_active":"2024-01-11T09:46:40.002961+0000","last_peered":"2024-01-11T09:46:40.002961+0000","last_clean":"2024-01-11T09:46:40.002961+0000","last_became_active":"2024-01-11T09:46:40.002649+0000","last_became_peered":"2024-01-11T09:46:40.002649+0000","last_unstale":"2024-01-11T09:46:40.002961+0000","last_undegraded":"2024-01-11T09:46:40.002961+0000","last_fullsized":"2024-01-11T09:46:40.002961+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:57:29.373530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.007915+0000","last_change":"2024-01-11T09:46:39.003378+0000","last_active":"2024-01-11T09:46:39.007915+0000","last_peered":"2024-01-11T09:46:39.007915+0000","last_clean":"2024-01-11T09:46:39.007915+0000","last_became_active":"2024-01-11T09:46:39.003118+0000","last_became_peered":"2024-01-11T09:46:39.003118+0000","last_unstale":"2024-01-11T09:46:39.007915+0000","last_undegraded":"2024-01-11T09:46:39.007915+0000","last_fullsized":"2024-01-11T09:46:39.007915+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:21:59.323435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.979967+0000","last_change":"2024-01-11T09:46:25.960002+0000","last_active":"2024-01-11T09:46:38.979967+0000","last_peered":"2024-01-11T09:46:38.979967+0000","last_clean":"2024-01-11T09:46:38.979967+0000","last_became_active":"2024-01-11T09:46:25.957845+0000","last_became_peered":"2024-01-11T09:46:25.957845+0000","last_unstale":"2024-01-11T09:46:38.979967+0000","last_undegraded":"2024-01-11T09:46:38.979967+0000","last_fullsized":"2024-01-11T09:46:38.979967+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:58:52.892728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.008739+0000","last_change":"2024-01-11T09:46:40.008739+0000","last_active":"2024-01-11T09:46:40.008739+0000","last_peered":"2024-01-11T09:46:40.008739+0000","last_clean":"2024-01-11T09:46:40.008739+0000","last_became_active":"2024-01-11T09:46:40.008481+0000","last_became_peered":"2024-01-11T09:46:40.008481+0000","last_unstale":"2024-01-11T09:46:40.008739+0000","last_undegraded":"2024-01-11T09:46:40.008739+0000","last_fullsized":"2024-01-11T09:46:40.008739+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:59:43.532603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"238'2","reported_seq":11,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.016124+0000","last_change":"2024-01-11T09:46:38.993477+0000","last_active":"2024-01-11T09:46:40.016124+0000","last_peered":"2024-01-11T09:46:40.016124+0000","last_clean":"2024-01-11T09:46:40.016124+0000","last_became_active":"2024-01-11T09:46:38.993223+0000","last_became_peered":"2024-01-11T09:46:38.993223+0000","last_unstale":"2024-01-11T09:46:40.016124+0000","last_undegraded":"2024-01-11T09:46:40.016124+0000","last_fullsized":"2024-01-11T09:46:40.016124+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:30:33.400129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986067+0000","last_change":"2024-01-11T09:46:25.958630+0000","last_active":"2024-01-11T09:46:38.986067+0000","last_peered":"2024-01-11T09:46:38.986067+0000","last_clean":"2024-01-11T09:46:38.986067+0000","last_became_active":"2024-01-11T09:46:25.957768+0000","last_became_peered":"2024-01-11T09:46:25.957768+0000","last_unstale":"2024-01-11T09:46:38.986067+0000","last_undegraded":"2024-01-11T09:46:38.986067+0000","last_fullsized":"2024-01-11T09:46:38.986067+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:59:15.007682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.997099+0000","last_change":"2024-01-11T09:46:39.997099+0000","last_active":"2024-01-11T09:46:39.997099+0000","last_peered":"2024-01-11T09:46:39.997099+0000","last_clean":"2024-01-11T09:46:39.997099+0000","last_became_active":"2024-01-11T09:46:39.996495+0000","last_became_peered":"2024-01-11T09:46:39.996495+0000","last_unstale":"2024-01-11T09:46:39.997099+0000","last_undegraded":"2024-01-11T09:46:39.997099+0000","last_fullsized":"2024-01-11T09:46:39.997099+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:09:57.054111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.996762+0000","last_change":"2024-01-11T09:46:38.994074+0000","last_active":"2024-01-11T09:46:38.996762+0000","last_peered":"2024-01-11T09:46:38.996762+0000","last_clean":"2024-01-11T09:46:38.996762+0000","last_became_active":"2024-01-11T09:46:38.993430+0000","last_became_peered":"2024-01-11T09:46:38.993430+0000","last_unstale":"2024-01-11T09:46:38.996762+0000","last_undegraded":"2024-01-11T09:46:38.996762+0000","last_fullsized":"2024-01-11T09:46:38.996762+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:37.969654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.983431+0000","last_change":"2024-01-11T09:46:25.954318+0000","last_active":"2024-01-11T09:46:38.983431+0000","last_peered":"2024-01-11T09:46:38.983431+0000","last_clean":"2024-01-11T09:46:38.983431+0000","last_became_active":"2024-01-11T09:46:25.953690+0000","last_became_peered":"2024-01-11T09:46:25.953690+0000","last_unstale":"2024-01-11T09:46:38.983431+0000","last_undegraded":"2024-01-11T09:46:38.983431+0000","last_fullsized":"2024-01-11T09:46:38.983431+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:28:55.053850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.991235+0000","last_change":"2024-01-11T09:46:39.991235+0000","last_active":"2024-01-11T09:46:39.991235+0000","last_peered":"2024-01-11T09:46:39.991235+0000","last_clean":"2024-01-11T09:46:39.991235+0000","last_became_active":"2024-01-11T09:46:39.990973+0000","last_became_peered":"2024-01-11T09:46:39.990973+0000","last_unstale":"2024-01-11T09:46:39.991235+0000","last_undegraded":"2024-01-11T09:46:39.991235+0000","last_fullsized":"2024-01-11T09:46:39.991235+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:54:48.495716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.993092+0000","last_change":"2024-01-11T09:46:38.990904+0000","last_active":"2024-01-11T09:46:38.993092+0000","last_peered":"2024-01-11T09:46:38.993092+0000","last_clean":"2024-01-11T09:46:38.993092+0000","last_became_active":"2024-01-11T09:46:38.990203+0000","last_became_peered":"2024-01-11T09:46:38.990203+0000","last_unstale":"2024-01-11T09:46:38.993092+0000","last_undegraded":"2024-01-11T09:46:38.993092+0000","last_fullsized":"2024-01-11T09:46:38.993092+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:25:22.331988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.981288+0000","last_change":"2024-01-11T09:46:25.957195+0000","last_active":"2024-01-11T09:46:38.981288+0000","last_peered":"2024-01-11T09:46:38.981288+0000","last_clean":"2024-01-11T09:46:38.981288+0000","last_became_active":"2024-01-11T09:46:25.956882+0000","last_became_peered":"2024-01-11T09:46:25.956882+0000","last_unstale":"2024-01-11T09:46:38.981288+0000","last_undegraded":"2024-01-11T09:46:38.981288+0000","last_fullsized":"2024-01-11T09:46:38.981288+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:37:41.519655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.010637+0000","last_change":"2024-01-11T09:46:40.010637+0000","last_active":"2024-01-11T09:46:40.010637+0000","last_peered":"2024-01-11T09:46:40.010637+0000","last_clean":"2024-01-11T09:46:40.010637+0000","last_became_active":"2024-01-11T09:46:40.009994+0000","last_became_peered":"2024-01-11T09:46:40.009994+0000","last_unstale":"2024-01-11T09:46:40.010637+0000","last_undegraded":"2024-01-11T09:46:40.010637+0000","last_fullsized":"2024-01-11T09:46:40.010637+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:54:35.025661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.994806+0000","last_change":"2024-01-11T09:46:38.993541+0000","last_active":"2024-01-11T09:46:38.994806+0000","last_peered":"2024-01-11T09:46:38.994806+0000","last_clean":"2024-01-11T09:46:38.994806+0000","last_became_active":"2024-01-11T09:46:38.992246+0000","last_became_peered":"2024-01-11T09:46:38.992246+0000","last_unstale":"2024-01-11T09:46:38.994806+0000","last_undegraded":"2024-01-11T09:46:38.994806+0000","last_fullsized":"2024-01-11T09:46:38.994806+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:18:20.631011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.978013+0000","last_change":"2024-01-11T09:46:25.960943+0000","last_active":"2024-01-11T09:46:38.978013+0000","last_peered":"2024-01-11T09:46:38.978013+0000","last_clean":"2024-01-11T09:46:38.978013+0000","last_became_active":"2024-01-11T09:46:25.960111+0000","last_became_peered":"2024-01-11T09:46:25.960111+0000","last_unstale":"2024-01-11T09:46:38.978013+0000","last_undegraded":"2024-01-11T09:46:38.978013+0000","last_fullsized":"2024-01-11T09:46:38.978013+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:04.562589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.017170+0000","last_change":"2024-01-11T09:46:40.017170+0000","last_active":"2024-01-11T09:46:40.017170+0000","last_peered":"2024-01-11T09:46:40.017170+0000","last_clean":"2024-01-11T09:46:40.017170+0000","last_became_active":"2024-01-11T09:46:40.013179+0000","last_became_peered":"2024-01-11T09:46:40.013179+0000","last_unstale":"2024-01-11T09:46:40.017170+0000","last_undegraded":"2024-01-11T09:46:40.017170+0000","last_fullsized":"2024-01-11T09:46:40.017170+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:10:42.456629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"238'2","reported_seq":11,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.013664+0000","last_change":"2024-01-11T09:46:38.991352+0000","last_active":"2024-01-11T09:46:40.013664+0000","last_peered":"2024-01-11T09:46:40.013664+0000","last_clean":"2024-01-11T09:46:40.013664+0000","last_became_active":"2024-01-11T09:46:38.991069+0000","last_became_peered":"2024-01-11T09:46:38.991069+0000","last_unstale":"2024-01-11T09:46:40.013664+0000","last_undegraded":"2024-01-11T09:46:40.013664+0000","last_fullsized":"2024-01-11T09:46:40.013664+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:10:22.443975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986956+0000","last_change":"2024-01-11T09:46:25.955367+0000","last_active":"2024-01-11T09:46:38.986956+0000","last_peered":"2024-01-11T09:46:38.986956+0000","last_clean":"2024-01-11T09:46:38.986956+0000","last_became_active":"2024-01-11T09:46:25.955123+0000","last_became_peered":"2024-01-11T09:46:25.955123+0000","last_unstale":"2024-01-11T09:46:38.986956+0000","last_undegraded":"2024-01-11T09:46:38.986956+0000","last_fullsized":"2024-01-11T09:46:38.986956+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:07:13.288202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.986527+0000","last_change":"2024-01-11T09:46:39.986527+0000","last_active":"2024-01-11T09:46:39.986527+0000","last_peered":"2024-01-11T09:46:39.986527+0000","last_clean":"2024-01-11T09:46:39.986527+0000","last_became_active":"2024-01-11T09:46:39.986243+0000","last_became_peered":"2024-01-11T09:46:39.986243+0000","last_unstale":"2024-01-11T09:46:39.986527+0000","last_undegraded":"2024-01-11T09:46:39.986527+0000","last_fullsized":"2024-01-11T09:46:39.986527+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:20.911474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.002782+0000","last_change":"2024-01-11T09:46:38.999657+0000","last_active":"2024-01-11T09:46:39.002782+0000","last_peered":"2024-01-11T09:46:39.002782+0000","last_clean":"2024-01-11T09:46:39.002782+0000","last_became_active":"2024-01-11T09:46:38.997446+0000","last_became_peered":"2024-01-11T09:46:38.997446+0000","last_unstale":"2024-01-11T09:46:39.002782+0000","last_undegraded":"2024-01-11T09:46:39.002782+0000","last_fullsized":"2024-01-11T09:46:39.002782+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:26:59.727081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980464+0000","last_change":"2024-01-11T09:46:25.958036+0000","last_active":"2024-01-11T09:46:38.980464+0000","last_peered":"2024-01-11T09:46:38.980464+0000","last_clean":"2024-01-11T09:46:38.980464+0000","last_became_active":"2024-01-11T09:46:25.957739+0000","last_became_peered":"2024-01-11T09:46:25.957739+0000","last_unstale":"2024-01-11T09:46:38.980464+0000","last_undegraded":"2024-01-11T09:46:38.980464+0000","last_fullsized":"2024-01-11T09:46:38.980464+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:04.562589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.999065+0000","last_change":"2024-01-11T09:46:39.999065+0000","last_active":"2024-01-11T09:46:39.999065+0000","last_peered":"2024-01-11T09:46:39.999065+0000","last_clean":"2024-01-11T09:46:39.999065+0000","last_became_active":"2024-01-11T09:46:39.998807+0000","last_became_peered":"2024-01-11T09:46:39.998807+0000","last_unstale":"2024-01-11T09:46:39.999065+0000","last_undegraded":"2024-01-11T09:46:39.999065+0000","last_fullsized":"2024-01-11T09:46:39.999065+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:30:59.947250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"238'1","reported_seq":10,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.983321+0000","last_change":"2024-01-11T09:46:39.000116+0000","last_active":"2024-01-11T09:46:39.983321+0000","last_peered":"2024-01-11T09:46:39.983321+0000","last_clean":"2024-01-11T09:46:39.983321+0000","last_became_active":"2024-01-11T09:46:38.999835+0000","last_became_peered":"2024-01-11T09:46:38.999835+0000","last_unstale":"2024-01-11T09:46:39.983321+0000","last_undegraded":"2024-01-11T09:46:39.983321+0000","last_fullsized":"2024-01-11T09:46:39.983321+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:53.897717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.988805+0000","last_change":"2024-01-11T09:46:25.951789+0000","last_active":"2024-01-11T09:46:38.988805+0000","last_peered":"2024-01-11T09:46:38.988805+0000","last_clean":"2024-01-11T09:46:38.988805+0000","last_became_active":"2024-01-11T09:46:25.950427+0000","last_became_peered":"2024-01-11T09:46:25.950427+0000","last_unstale":"2024-01-11T09:46:38.988805+0000","last_undegraded":"2024-01-11T09:46:38.988805+0000","last_fullsized":"2024-01-11T09:46:38.988805+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:10.125414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.003667+0000","last_change":"2024-01-11T09:46:40.003667+0000","last_active":"2024-01-11T09:46:40.003667+0000","last_peered":"2024-01-11T09:46:40.003667+0000","last_clean":"2024-01-11T09:46:40.003667+0000","last_became_active":"2024-01-11T09:46:40.003357+0000","last_became_peered":"2024-01-11T09:46:40.003357+0000","last_unstale":"2024-01-11T09:46:40.003667+0000","last_undegraded":"2024-01-11T09:46:40.003667+0000","last_fullsized":"2024-01-11T09:46:40.003667+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:56:27.940894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"238'2","reported_seq":11,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.023050+0000","last_change":"2024-01-11T09:46:38.989486+0000","last_active":"2024-01-11T09:46:40.023050+0000","last_peered":"2024-01-11T09:46:40.023050+0000","last_clean":"2024-01-11T09:46:40.023050+0000","last_became_active":"2024-01-11T09:46:38.989219+0000","last_became_peered":"2024-01-11T09:46:38.989219+0000","last_unstale":"2024-01-11T09:46:40.023050+0000","last_undegraded":"2024-01-11T09:46:40.023050+0000","last_fullsized":"2024-01-11T09:46:40.023050+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:41:48.456551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980046+0000","last_change":"2024-01-11T09:46:25.962921+0000","last_active":"2024-01-11T09:46:38.980046+0000","last_peered":"2024-01-11T09:46:38.980046+0000","last_clean":"2024-01-11T09:46:38.980046+0000","last_became_active":"2024-01-11T09:46:25.962306+0000","last_became_peered":"2024-01-11T09:46:25.962306+0000","last_unstale":"2024-01-11T09:46:38.980046+0000","last_undegraded":"2024-01-11T09:46:38.980046+0000","last_fullsized":"2024-01-11T09:46:38.980046+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:49:33.997022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.010679+0000","last_change":"2024-01-11T09:46:40.010679+0000","last_active":"2024-01-11T09:46:40.010679+0000","last_peered":"2024-01-11T09:46:40.010679+0000","last_clean":"2024-01-11T09:46:40.010679+0000","last_became_active":"2024-01-11T09:46:40.007871+0000","last_became_peered":"2024-01-11T09:46:40.007871+0000","last_unstale":"2024-01-11T09:46:40.010679+0000","last_undegraded":"2024-01-11T09:46:40.010679+0000","last_fullsized":"2024-01-11T09:46:40.010679+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:18:21.639035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"238'1","reported_seq":10,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.987717+0000","last_change":"2024-01-11T09:46:38.997759+0000","last_active":"2024-01-11T09:46:39.987717+0000","last_peered":"2024-01-11T09:46:39.987717+0000","last_clean":"2024-01-11T09:46:39.987717+0000","last_became_active":"2024-01-11T09:46:38.997461+0000","last_became_peered":"2024-01-11T09:46:38.997461+0000","last_unstale":"2024-01-11T09:46:39.987717+0000","last_undegraded":"2024-01-11T09:46:39.987717+0000","last_fullsized":"2024-01-11T09:46:39.987717+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:11:20.711273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.981007+0000","last_change":"2024-01-11T09:46:25.951750+0000","last_active":"2024-01-11T09:46:38.981007+0000","last_peered":"2024-01-11T09:46:38.981007+0000","last_clean":"2024-01-11T09:46:38.981007+0000","last_became_active":"2024-01-11T09:46:25.951499+0000","last_became_peered":"2024-01-11T09:46:25.951499+0000","last_unstale":"2024-01-11T09:46:38.981007+0000","last_undegraded":"2024-01-11T09:46:38.981007+0000","last_fullsized":"2024-01-11T09:46:38.981007+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:48:16.354636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.988056+0000","last_change":"2024-01-11T09:46:39.988056+0000","last_active":"2024-01-11T09:46:39.988056+0000","last_peered":"2024-01-11T09:46:39.988056+0000","last_clean":"2024-01-11T09:46:39.988056+0000","last_became_active":"2024-01-11T09:46:39.987747+0000","last_became_peered":"2024-01-11T09:46:39.987747+0000","last_unstale":"2024-01-11T09:46:39.988056+0000","last_undegraded":"2024-01-11T09:46:39.988056+0000","last_fullsized":"2024-01-11T09:46:39.988056+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:04:48.614151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"238'8","reported_seq":17,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.027417+0000","last_change":"2024-01-11T09:46:39.001969+0000","last_active":"2024-01-11T09:46:40.027417+0000","last_peered":"2024-01-11T09:46:40.027417+0000","last_clean":"2024-01-11T09:46:40.027417+0000","last_became_active":"2024-01-11T09:46:39.001149+0000","last_became_peered":"2024-01-11T09:46:39.001149+0000","last_unstale":"2024-01-11T09:46:40.027417+0000","last_undegraded":"2024-01-11T09:46:40.027417+0000","last_fullsized":"2024-01-11T09:46:40.027417+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:39:18.714435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986090+0000","last_change":"2024-01-11T09:46:25.950974+0000","last_active":"2024-01-11T09:46:38.986090+0000","last_peered":"2024-01-11T09:46:38.986090+0000","last_clean":"2024-01-11T09:46:38.986090+0000","last_became_active":"2024-01-11T09:46:25.950595+0000","last_became_peered":"2024-01-11T09:46:25.950595+0000","last_unstale":"2024-01-11T09:46:38.986090+0000","last_undegraded":"2024-01-11T09:46:38.986090+0000","last_fullsized":"2024-01-11T09:46:38.986090+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:58.843659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.999320+0000","last_change":"2024-01-11T09:46:39.999320+0000","last_active":"2024-01-11T09:46:39.999320+0000","last_peered":"2024-01-11T09:46:39.999320+0000","last_clean":"2024-01-11T09:46:39.999320+0000","last_became_active":"2024-01-11T09:46:39.999031+0000","last_became_peered":"2024-01-11T09:46:39.999031+0000","last_unstale":"2024-01-11T09:46:39.999320+0000","last_undegraded":"2024-01-11T09:46:39.999320+0000","last_fullsized":"2024-01-11T09:46:39.999320+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:03:39.674343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.993071+0000","last_change":"2024-01-11T09:46:38.990498+0000","last_active":"2024-01-11T09:46:38.993071+0000","last_peered":"2024-01-11T09:46:38.993071+0000","last_clean":"2024-01-11T09:46:38.993071+0000","last_became_active":"2024-01-11T09:46:38.990250+0000","last_became_peered":"2024-01-11T09:46:38.990250+0000","last_unstale":"2024-01-11T09:46:38.993071+0000","last_undegraded":"2024-01-11T09:46:38.993071+0000","last_fullsized":"2024-01-11T09:46:38.993071+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:27:47.259124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990576+0000","last_change":"2024-01-11T09:46:25.952186+0000","last_active":"2024-01-11T09:46:38.990576+0000","last_peered":"2024-01-11T09:46:38.990576+0000","last_clean":"2024-01-11T09:46:38.990576+0000","last_became_active":"2024-01-11T09:46:25.951898+0000","last_became_peered":"2024-01-11T09:46:25.951898+0000","last_unstale":"2024-01-11T09:46:38.990576+0000","last_undegraded":"2024-01-11T09:46:38.990576+0000","last_fullsized":"2024-01-11T09:46:38.990576+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:54:00.534598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.017717+0000","last_change":"2024-01-11T09:46:40.017717+0000","last_active":"2024-01-11T09:46:40.017717+0000","last_peered":"2024-01-11T09:46:40.017717+0000","last_clean":"2024-01-11T09:46:40.017717+0000","last_became_active":"2024-01-11T09:46:40.017424+0000","last_became_peered":"2024-01-11T09:46:40.017424+0000","last_unstale":"2024-01-11T09:46:40.017717+0000","last_undegraded":"2024-01-11T09:46:40.017717+0000","last_fullsized":"2024-01-11T09:46:40.017717+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:14:20.834540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.992273+0000","last_change":"2024-01-11T09:46:38.990154+0000","last_active":"2024-01-11T09:46:38.992273+0000","last_peered":"2024-01-11T09:46:38.992273+0000","last_clean":"2024-01-11T09:46:38.992273+0000","last_became_active":"2024-01-11T09:46:38.989880+0000","last_became_peered":"2024-01-11T09:46:38.989880+0000","last_unstale":"2024-01-11T09:46:38.992273+0000","last_undegraded":"2024-01-11T09:46:38.992273+0000","last_fullsized":"2024-01-11T09:46:38.992273+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:19:16.773855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986052+0000","last_change":"2024-01-11T09:46:25.957991+0000","last_active":"2024-01-11T09:46:38.986052+0000","last_peered":"2024-01-11T09:46:38.986052+0000","last_clean":"2024-01-11T09:46:38.986052+0000","last_became_active":"2024-01-11T09:46:25.957482+0000","last_became_peered":"2024-01-11T09:46:25.957482+0000","last_unstale":"2024-01-11T09:46:38.986052+0000","last_undegraded":"2024-01-11T09:46:38.986052+0000","last_fullsized":"2024-01-11T09:46:38.986052+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:54:00.534598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.995141+0000","last_change":"2024-01-11T09:46:39.995141+0000","last_active":"2024-01-11T09:46:39.995141+0000","last_peered":"2024-01-11T09:46:39.995141+0000","last_clean":"2024-01-11T09:46:39.995141+0000","last_became_active":"2024-01-11T09:46:39.994673+0000","last_became_peered":"2024-01-11T09:46:39.994673+0000","last_unstale":"2024-01-11T09:46:39.995141+0000","last_undegraded":"2024-01-11T09:46:39.995141+0000","last_fullsized":"2024-01-11T09:46:39.995141+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:44:58.685583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"238'2","reported_seq":11,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.020072+0000","last_change":"2024-01-11T09:46:39.001870+0000","last_active":"2024-01-11T09:46:40.020072+0000","last_peered":"2024-01-11T09:46:40.020072+0000","last_clean":"2024-01-11T09:46:40.020072+0000","last_became_active":"2024-01-11T09:46:39.001632+0000","last_became_peered":"2024-01-11T09:46:39.001632+0000","last_unstale":"2024-01-11T09:46:40.020072+0000","last_undegraded":"2024-01-11T09:46:40.020072+0000","last_fullsized":"2024-01-11T09:46:40.020072+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:59:42.524580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984733+0000","last_change":"2024-01-11T09:46:25.942788+0000","last_active":"2024-01-11T09:46:38.984733+0000","last_peered":"2024-01-11T09:46:38.984733+0000","last_clean":"2024-01-11T09:46:38.984733+0000","last_became_active":"2024-01-11T09:46:25.941771+0000","last_became_peered":"2024-01-11T09:46:25.941771+0000","last_unstale":"2024-01-11T09:46:38.984733+0000","last_undegraded":"2024-01-11T09:46:38.984733+0000","last_fullsized":"2024-01-11T09:46:38.984733+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:42:47.356295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.020514+0000","last_change":"2024-01-11T09:46:40.020514+0000","last_active":"2024-01-11T09:46:40.020514+0000","last_peered":"2024-01-11T09:46:40.020514+0000","last_clean":"2024-01-11T09:46:40.020514+0000","last_became_active":"2024-01-11T09:46:40.020274+0000","last_became_peered":"2024-01-11T09:46:40.020274+0000","last_unstale":"2024-01-11T09:46:40.020514+0000","last_undegraded":"2024-01-11T09:46:40.020514+0000","last_fullsized":"2024-01-11T09:46:40.020514+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:51:27.846178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"238'6","reported_seq":15,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.014817+0000","last_change":"2024-01-11T09:46:38.996034+0000","last_active":"2024-01-11T09:46:40.014817+0000","last_peered":"2024-01-11T09:46:40.014817+0000","last_clean":"2024-01-11T09:46:40.014817+0000","last_became_active":"2024-01-11T09:46:38.995771+0000","last_became_peered":"2024-01-11T09:46:38.995771+0000","last_unstale":"2024-01-11T09:46:40.014817+0000","last_undegraded":"2024-01-11T09:46:40.014817+0000","last_fullsized":"2024-01-11T09:46:40.014817+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:08:08.704554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986656+0000","last_change":"2024-01-11T09:46:25.954694+0000","last_active":"2024-01-11T09:46:38.986656+0000","last_peered":"2024-01-11T09:46:38.986656+0000","last_clean":"2024-01-11T09:46:38.986656+0000","last_became_active":"2024-01-11T09:46:25.952808+0000","last_became_peered":"2024-01-11T09:46:25.952808+0000","last_unstale":"2024-01-11T09:46:38.986656+0000","last_undegraded":"2024-01-11T09:46:38.986656+0000","last_fullsized":"2024-01-11T09:46:38.986656+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:10.125414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.004319+0000","last_change":"2024-01-11T09:46:40.004319+0000","last_active":"2024-01-11T09:46:40.004319+0000","last_peered":"2024-01-11T09:46:40.004319+0000","last_clean":"2024-01-11T09:46:40.004319+0000","last_became_active":"2024-01-11T09:46:40.004054+0000","last_became_peered":"2024-01-11T09:46:40.004054+0000","last_unstale":"2024-01-11T09:46:40.004319+0000","last_undegraded":"2024-01-11T09:46:40.004319+0000","last_fullsized":"2024-01-11T09:46:40.004319+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:24:06.711228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"238'3","reported_seq":12,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.023788+0000","last_change":"2024-01-11T09:46:38.994771+0000","last_active":"2024-01-11T09:46:40.023788+0000","last_peered":"2024-01-11T09:46:40.023788+0000","last_clean":"2024-01-11T09:46:40.023788+0000","last_became_active":"2024-01-11T09:46:38.994461+0000","last_became_peered":"2024-01-11T09:46:38.994461+0000","last_unstale":"2024-01-11T09:46:40.023788+0000","last_undegraded":"2024-01-11T09:46:40.023788+0000","last_fullsized":"2024-01-11T09:46:40.023788+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:45:29.584972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984134+0000","last_change":"2024-01-11T09:46:25.962654+0000","last_active":"2024-01-11T09:46:38.984134+0000","last_peered":"2024-01-11T09:46:38.984134+0000","last_clean":"2024-01-11T09:46:38.984134+0000","last_became_active":"2024-01-11T09:46:25.962073+0000","last_became_peered":"2024-01-11T09:46:25.962073+0000","last_unstale":"2024-01-11T09:46:38.984134+0000","last_undegraded":"2024-01-11T09:46:38.984134+0000","last_fullsized":"2024-01-11T09:46:38.984134+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:56:15.022301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.019981+0000","last_change":"2024-01-11T09:46:40.019981+0000","last_active":"2024-01-11T09:46:40.019981+0000","last_peered":"2024-01-11T09:46:40.019981+0000","last_clean":"2024-01-11T09:46:40.019981+0000","last_became_active":"2024-01-11T09:46:40.019280+0000","last_became_peered":"2024-01-11T09:46:40.019280+0000","last_unstale":"2024-01-11T09:46:40.019981+0000","last_undegraded":"2024-01-11T09:46:40.019981+0000","last_fullsized":"2024-01-11T09:46:40.019981+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:09:57.054111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.999886+0000","last_change":"2024-01-11T09:46:38.993119+0000","last_active":"2024-01-11T09:46:38.999886+0000","last_peered":"2024-01-11T09:46:38.999886+0000","last_clean":"2024-01-11T09:46:38.999886+0000","last_became_active":"2024-01-11T09:46:38.992846+0000","last_became_peered":"2024-01-11T09:46:38.992846+0000","last_unstale":"2024-01-11T09:46:38.999886+0000","last_undegraded":"2024-01-11T09:46:38.999886+0000","last_fullsized":"2024-01-11T09:46:38.999886+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:17.305476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986439+0000","last_change":"2024-01-11T09:46:25.956810+0000","last_active":"2024-01-11T09:46:38.986439+0000","last_peered":"2024-01-11T09:46:38.986439+0000","last_clean":"2024-01-11T09:46:38.986439+0000","last_became_active":"2024-01-11T09:46:25.955894+0000","last_became_peered":"2024-01-11T09:46:25.955894+0000","last_unstale":"2024-01-11T09:46:38.986439+0000","last_undegraded":"2024-01-11T09:46:38.986439+0000","last_fullsized":"2024-01-11T09:46:38.986439+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:53:35.139255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.994807+0000","last_change":"2024-01-11T09:46:39.994807+0000","last_active":"2024-01-11T09:46:39.994807+0000","last_peered":"2024-01-11T09:46:39.994807+0000","last_clean":"2024-01-11T09:46:39.994807+0000","last_became_active":"2024-01-11T09:46:39.994284+0000","last_became_peered":"2024-01-11T09:46:39.994284+0000","last_unstale":"2024-01-11T09:46:39.994807+0000","last_undegraded":"2024-01-11T09:46:39.994807+0000","last_fullsized":"2024-01-11T09:46:39.994807+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:41:33.450510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.001864+0000","last_change":"2024-01-11T09:46:38.998113+0000","last_active":"2024-01-11T09:46:39.001864+0000","last_peered":"2024-01-11T09:46:39.001864+0000","last_clean":"2024-01-11T09:46:39.001864+0000","last_became_active":"2024-01-11T09:46:38.997748+0000","last_became_peered":"2024-01-11T09:46:38.997748+0000","last_unstale":"2024-01-11T09:46:39.001864+0000","last_undegraded":"2024-01-11T09:46:39.001864+0000","last_fullsized":"2024-01-11T09:46:39.001864+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:38:11.656185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.989104+0000","last_change":"2024-01-11T09:46:25.947721+0000","last_active":"2024-01-11T09:46:38.989104+0000","last_peered":"2024-01-11T09:46:38.989104+0000","last_clean":"2024-01-11T09:46:38.989104+0000","last_became_active":"2024-01-11T09:46:25.947454+0000","last_became_peered":"2024-01-11T09:46:25.947454+0000","last_unstale":"2024-01-11T09:46:38.989104+0000","last_undegraded":"2024-01-11T09:46:38.989104+0000","last_fullsized":"2024-01-11T09:46:38.989104+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:42:44.083747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.994606+0000","last_change":"2024-01-11T09:46:39.994606+0000","last_active":"2024-01-11T09:46:39.994606+0000","last_peered":"2024-01-11T09:46:39.994606+0000","last_clean":"2024-01-11T09:46:39.994606+0000","last_became_active":"2024-01-11T09:46:39.994309+0000","last_became_peered":"2024-01-11T09:46:39.994309+0000","last_unstale":"2024-01-11T09:46:39.994606+0000","last_undegraded":"2024-01-11T09:46:39.994606+0000","last_fullsized":"2024-01-11T09:46:39.994606+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:37:32.540165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.996928+0000","last_change":"2024-01-11T09:46:38.991811+0000","last_active":"2024-01-11T09:46:38.996928+0000","last_peered":"2024-01-11T09:46:38.996928+0000","last_clean":"2024-01-11T09:46:38.996928+0000","last_became_active":"2024-01-11T09:46:38.990698+0000","last_became_peered":"2024-01-11T09:46:38.990698+0000","last_unstale":"2024-01-11T09:46:38.996928+0000","last_undegraded":"2024-01-11T09:46:38.996928+0000","last_fullsized":"2024-01-11T09:46:38.996928+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:54:34.017638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.995074+0000","last_change":"2024-01-11T09:46:25.939265+0000","last_active":"2024-01-11T09:46:38.995074+0000","last_peered":"2024-01-11T09:46:38.995074+0000","last_clean":"2024-01-11T09:46:38.995074+0000","last_became_active":"2024-01-11T09:46:25.938964+0000","last_became_peered":"2024-01-11T09:46:25.938964+0000","last_unstale":"2024-01-11T09:46:38.995074+0000","last_undegraded":"2024-01-11T09:46:38.995074+0000","last_fullsized":"2024-01-11T09:46:38.995074+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:20:24.956940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.013869+0000","last_change":"2024-01-11T09:46:40.013869+0000","last_active":"2024-01-11T09:46:40.013869+0000","last_peered":"2024-01-11T09:46:40.013869+0000","last_clean":"2024-01-11T09:46:40.013869+0000","last_became_active":"2024-01-11T09:46:40.012391+0000","last_became_peered":"2024-01-11T09:46:40.012391+0000","last_unstale":"2024-01-11T09:46:40.013869+0000","last_undegraded":"2024-01-11T09:46:40.013869+0000","last_fullsized":"2024-01-11T09:46:40.013869+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:15:01.956891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.007912+0000","last_change":"2024-01-11T09:46:39.003374+0000","last_active":"2024-01-11T09:46:39.007912+0000","last_peered":"2024-01-11T09:46:39.007912+0000","last_clean":"2024-01-11T09:46:39.007912+0000","last_became_active":"2024-01-11T09:46:39.003062+0000","last_became_peered":"2024-01-11T09:46:39.003062+0000","last_unstale":"2024-01-11T09:46:39.007912+0000","last_undegraded":"2024-01-11T09:46:39.007912+0000","last_fullsized":"2024-01-11T09:46:39.007912+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:31:12.578015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.981127+0000","last_change":"2024-01-11T09:46:25.953966+0000","last_active":"2024-01-11T09:46:38.981127+0000","last_peered":"2024-01-11T09:46:38.981127+0000","last_clean":"2024-01-11T09:46:38.981127+0000","last_became_active":"2024-01-11T09:46:25.953566+0000","last_became_peered":"2024-01-11T09:46:25.953566+0000","last_unstale":"2024-01-11T09:46:38.981127+0000","last_undegraded":"2024-01-11T09:46:38.981127+0000","last_fullsized":"2024-01-11T09:46:38.981127+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:45:16.842085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.010817+0000","last_change":"2024-01-11T09:46:40.010817+0000","last_active":"2024-01-11T09:46:40.010817+0000","last_peered":"2024-01-11T09:46:40.010817+0000","last_clean":"2024-01-11T09:46:40.010817+0000","last_became_active":"2024-01-11T09:46:40.010334+0000","last_became_peered":"2024-01-11T09:46:40.010334+0000","last_unstale":"2024-01-11T09:46:40.010817+0000","last_undegraded":"2024-01-11T09:46:40.010817+0000","last_fullsized":"2024-01-11T09:46:40.010817+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:10:14.041470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"238'1","reported_seq":10,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.980203+0000","last_change":"2024-01-11T09:46:38.997134+0000","last_active":"2024-01-11T09:46:39.980203+0000","last_peered":"2024-01-11T09:46:39.980203+0000","last_clean":"2024-01-11T09:46:39.980203+0000","last_became_active":"2024-01-11T09:46:38.996819+0000","last_became_peered":"2024-01-11T09:46:38.996819+0000","last_unstale":"2024-01-11T09:46:39.980203+0000","last_undegraded":"2024-01-11T09:46:39.980203+0000","last_fullsized":"2024-01-11T09:46:39.980203+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:03:27.598867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.978052+0000","last_change":"2024-01-11T09:46:25.959751+0000","last_active":"2024-01-11T09:46:38.978052+0000","last_peered":"2024-01-11T09:46:38.978052+0000","last_clean":"2024-01-11T09:46:38.978052+0000","last_became_active":"2024-01-11T09:46:25.959431+0000","last_became_peered":"2024-01-11T09:46:25.959431+0000","last_unstale":"2024-01-11T09:46:38.978052+0000","last_undegraded":"2024-01-11T09:46:38.978052+0000","last_fullsized":"2024-01-11T09:46:38.978052+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:58:52.892728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.993126+0000","last_change":"2024-01-11T09:46:39.993126+0000","last_active":"2024-01-11T09:46:39.993126+0000","last_peered":"2024-01-11T09:46:39.993126+0000","last_clean":"2024-01-11T09:46:39.993126+0000","last_became_active":"2024-01-11T09:46:39.992838+0000","last_became_peered":"2024-01-11T09:46:39.992838+0000","last_unstale":"2024-01-11T09:46:39.993126+0000","last_undegraded":"2024-01-11T09:46:39.993126+0000","last_fullsized":"2024-01-11T09:46:39.993126+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:49:09.210651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"238'1","reported_seq":10,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.985657+0000","last_change":"2024-01-11T09:46:38.995368+0000","last_active":"2024-01-11T09:46:39.985657+0000","last_peered":"2024-01-11T09:46:39.985657+0000","last_clean":"2024-01-11T09:46:39.985657+0000","last_became_active":"2024-01-11T09:46:38.995087+0000","last_became_peered":"2024-01-11T09:46:38.995087+0000","last_unstale":"2024-01-11T09:46:39.985657+0000","last_undegraded":"2024-01-11T09:46:39.985657+0000","last_fullsized":"2024-01-11T09:46:39.985657+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:47:57.662961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990411+0000","last_change":"2024-01-11T09:46:25.956345+0000","last_active":"2024-01-11T09:46:38.990411+0000","last_peered":"2024-01-11T09:46:38.990411+0000","last_clean":"2024-01-11T09:46:38.990411+0000","last_became_active":"2024-01-11T09:46:25.956080+0000","last_became_peered":"2024-01-11T09:46:25.956080+0000","last_unstale":"2024-01-11T09:46:38.990411+0000","last_undegraded":"2024-01-11T09:46:38.990411+0000","last_fullsized":"2024-01-11T09:46:38.990411+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:30:20.657242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.996253+0000","last_change":"2024-01-11T09:46:39.996253+0000","last_active":"2024-01-11T09:46:39.996253+0000","last_peered":"2024-01-11T09:46:39.996253+0000","last_clean":"2024-01-11T09:46:39.996253+0000","last_became_active":"2024-01-11T09:46:39.995997+0000","last_became_peered":"2024-01-11T09:46:39.995997+0000","last_unstale":"2024-01-11T09:46:39.996253+0000","last_undegraded":"2024-01-11T09:46:39.996253+0000","last_fullsized":"2024-01-11T09:46:39.996253+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:41:49.464575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"238'2","reported_seq":11,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.022163+0000","last_change":"2024-01-11T09:46:38.996864+0000","last_active":"2024-01-11T09:46:40.022163+0000","last_peered":"2024-01-11T09:46:40.022163+0000","last_clean":"2024-01-11T09:46:40.022163+0000","last_became_active":"2024-01-11T09:46:38.996594+0000","last_became_peered":"2024-01-11T09:46:38.996594+0000","last_unstale":"2024-01-11T09:46:40.022163+0000","last_undegraded":"2024-01-11T09:46:40.022163+0000","last_fullsized":"2024-01-11T09:46:40.022163+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:37:54.262542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.977660+0000","last_change":"2024-01-11T09:46:25.953176+0000","last_active":"2024-01-11T09:46:38.977660+0000","last_peered":"2024-01-11T09:46:38.977660+0000","last_clean":"2024-01-11T09:46:38.977660+0000","last_became_active":"2024-01-11T09:46:25.952276+0000","last_became_peered":"2024-01-11T09:46:25.952276+0000","last_unstale":"2024-01-11T09:46:38.977660+0000","last_undegraded":"2024-01-11T09:46:38.977660+0000","last_fullsized":"2024-01-11T09:46:38.977660+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:37:41.519655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.017999+0000","last_change":"2024-01-11T09:46:40.017999+0000","last_active":"2024-01-11T09:46:40.017999+0000","last_peered":"2024-01-11T09:46:40.017999+0000","last_clean":"2024-01-11T09:46:40.017999+0000","last_became_active":"2024-01-11T09:46:40.017756+0000","last_became_peered":"2024-01-11T09:46:40.017756+0000","last_unstale":"2024-01-11T09:46:40.017999+0000","last_undegraded":"2024-01-11T09:46:40.017999+0000","last_fullsized":"2024-01-11T09:46:40.017999+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:03:04.200008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"238'2","reported_seq":11,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.023981+0000","last_change":"2024-01-11T09:46:38.994164+0000","last_active":"2024-01-11T09:46:40.023981+0000","last_peered":"2024-01-11T09:46:40.023981+0000","last_clean":"2024-01-11T09:46:40.023981+0000","last_became_active":"2024-01-11T09:46:38.992666+0000","last_became_peered":"2024-01-11T09:46:38.992666+0000","last_unstale":"2024-01-11T09:46:40.023981+0000","last_undegraded":"2024-01-11T09:46:40.023981+0000","last_fullsized":"2024-01-11T09:46:40.023981+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:59:05.635616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.989167+0000","last_change":"2024-01-11T09:46:25.949114+0000","last_active":"2024-01-11T09:46:38.989167+0000","last_peered":"2024-01-11T09:46:38.989167+0000","last_clean":"2024-01-11T09:46:38.989167+0000","last_became_active":"2024-01-11T09:46:25.948844+0000","last_became_peered":"2024-01-11T09:46:25.948844+0000","last_unstale":"2024-01-11T09:46:38.989167+0000","last_undegraded":"2024-01-11T09:46:38.989167+0000","last_fullsized":"2024-01-11T09:46:38.989167+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:31:58.203452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.996228+0000","last_change":"2024-01-11T09:46:39.996228+0000","last_active":"2024-01-11T09:46:39.996228+0000","last_peered":"2024-01-11T09:46:39.996228+0000","last_clean":"2024-01-11T09:46:39.996228+0000","last_became_active":"2024-01-11T09:46:39.995964+0000","last_became_peered":"2024-01-11T09:46:39.995964+0000","last_unstale":"2024-01-11T09:46:39.996228+0000","last_undegraded":"2024-01-11T09:46:39.996228+0000","last_fullsized":"2024-01-11T09:46:39.996228+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:06:31.107627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"238'1","reported_seq":10,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.994751+0000","last_change":"2024-01-11T09:46:38.998074+0000","last_active":"2024-01-11T09:46:39.994751+0000","last_peered":"2024-01-11T09:46:39.994751+0000","last_clean":"2024-01-11T09:46:39.994751+0000","last_became_active":"2024-01-11T09:46:38.997784+0000","last_became_peered":"2024-01-11T09:46:38.997784+0000","last_unstale":"2024-01-11T09:46:39.994751+0000","last_undegraded":"2024-01-11T09:46:39.994751+0000","last_fullsized":"2024-01-11T09:46:39.994751+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:22.868301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984877+0000","last_change":"2024-01-11T09:46:25.954895+0000","last_active":"2024-01-11T09:46:38.984877+0000","last_peered":"2024-01-11T09:46:38.984877+0000","last_clean":"2024-01-11T09:46:38.984877+0000","last_became_active":"2024-01-11T09:46:25.954472+0000","last_became_peered":"2024-01-11T09:46:25.954472+0000","last_unstale":"2024-01-11T09:46:38.984877+0000","last_undegraded":"2024-01-11T09:46:38.984877+0000","last_fullsized":"2024-01-11T09:46:38.984877+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:34:26.903086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.004791+0000","last_change":"2024-01-11T09:46:40.004791+0000","last_active":"2024-01-11T09:46:40.004791+0000","last_peered":"2024-01-11T09:46:40.004791+0000","last_clean":"2024-01-11T09:46:40.004791+0000","last_became_active":"2024-01-11T09:46:40.004293+0000","last_became_peered":"2024-01-11T09:46:40.004293+0000","last_unstale":"2024-01-11T09:46:40.004791+0000","last_undegraded":"2024-01-11T09:46:40.004791+0000","last_fullsized":"2024-01-11T09:46:40.004791+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:20:57.093675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"238'4","reported_seq":13,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.021842+0000","last_change":"2024-01-11T09:46:39.000920+0000","last_active":"2024-01-11T09:46:40.021842+0000","last_peered":"2024-01-11T09:46:40.021842+0000","last_clean":"2024-01-11T09:46:40.021842+0000","last_became_active":"2024-01-11T09:46:39.000661+0000","last_became_peered":"2024-01-11T09:46:39.000661+0000","last_unstale":"2024-01-11T09:46:40.021842+0000","last_undegraded":"2024-01-11T09:46:40.021842+0000","last_fullsized":"2024-01-11T09:46:40.021842+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:10:41.448606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.987884+0000","last_change":"2024-01-11T09:46:25.955503+0000","last_active":"2024-01-11T09:46:38.987884+0000","last_peered":"2024-01-11T09:46:38.987884+0000","last_clean":"2024-01-11T09:46:38.987884+0000","last_became_active":"2024-01-11T09:46:25.954870+0000","last_became_peered":"2024-01-11T09:46:25.954870+0000","last_unstale":"2024-01-11T09:46:38.987884+0000","last_undegraded":"2024-01-11T09:46:38.987884+0000","last_fullsized":"2024-01-11T09:46:38.987884+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:06:12.133349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.999548+0000","last_change":"2024-01-11T09:46:39.999548+0000","last_active":"2024-01-11T09:46:39.999548+0000","last_peered":"2024-01-11T09:46:39.999548+0000","last_clean":"2024-01-11T09:46:39.999548+0000","last_became_active":"2024-01-11T09:46:39.999236+0000","last_became_peered":"2024-01-11T09:46:39.999236+0000","last_unstale":"2024-01-11T09:46:39.999548+0000","last_undegraded":"2024-01-11T09:46:39.999548+0000","last_fullsized":"2024-01-11T09:46:39.999548+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:59:06.643639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984281+0000","last_change":"2024-01-11T09:46:25.947226+0000","last_active":"2024-01-11T09:46:38.984281+0000","last_peered":"2024-01-11T09:46:38.984281+0000","last_clean":"2024-01-11T09:46:38.984281+0000","last_became_active":"2024-01-11T09:46:25.946806+0000","last_became_peered":"2024-01-11T09:46:25.946806+0000","last_unstale":"2024-01-11T09:46:38.984281+0000","last_undegraded":"2024-01-11T09:46:38.984281+0000","last_fullsized":"2024-01-11T09:46:38.984281+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:25:09.589100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"234'1","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980946+0000","last_change":"2024-01-11T09:46:25.236672+0000","last_active":"2024-01-11T09:46:38.980946+0000","last_peered":"2024-01-11T09:46:38.980946+0000","last_clean":"2024-01-11T09:46:38.980946+0000","last_became_active":"2024-01-11T09:46:25.236363+0000","last_became_peered":"2024-01-11T09:46:25.236363+0000","last_unstale":"2024-01-11T09:46:38.980946+0000","last_undegraded":"2024-01-11T09:46:38.980946+0000","last_fullsized":"2024-01-11T09:46:38.980946+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:01:24.091035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.005404+0000","last_change":"2024-01-11T09:46:40.005404+0000","last_active":"2024-01-11T09:46:40.005404+0000","last_peered":"2024-01-11T09:46:40.005404+0000","last_clean":"2024-01-11T09:46:40.005404+0000","last_became_active":"2024-01-11T09:46:40.005086+0000","last_became_peered":"2024-01-11T09:46:40.005086+0000","last_unstale":"2024-01-11T09:46:40.005404+0000","last_undegraded":"2024-01-11T09:46:40.005404+0000","last_fullsized":"2024-01-11T09:46:40.005404+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:27:48.267147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.977442+0000","last_change":"2024-01-11T09:46:25.951411+0000","last_active":"2024-01-11T09:46:38.977442+0000","last_peered":"2024-01-11T09:46:38.977442+0000","last_clean":"2024-01-11T09:46:38.977442+0000","last_became_active":"2024-01-11T09:46:25.951161+0000","last_became_peered":"2024-01-11T09:46:25.951161+0000","last_unstale":"2024-01-11T09:46:38.977442+0000","last_undegraded":"2024-01-11T09:46:38.977442+0000","last_fullsized":"2024-01-11T09:46:38.977442+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:45:16.842085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"234'2","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984056+0000","last_change":"2024-01-11T09:46:25.238481+0000","last_active":"2024-01-11T09:46:38.984056+0000","last_peered":"2024-01-11T09:46:38.984056+0000","last_clean":"2024-01-11T09:46:38.984056+0000","last_became_active":"2024-01-11T09:46:25.238203+0000","last_became_peered":"2024-01-11T09:46:25.238203+0000","last_unstale":"2024-01-11T09:46:38.984056+0000","last_undegraded":"2024-01-11T09:46:38.984056+0000","last_fullsized":"2024-01-11T09:46:38.984056+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:42:43.074601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.016036+0000","last_change":"2024-01-11T09:46:40.016036+0000","last_active":"2024-01-11T09:46:40.016036+0000","last_peered":"2024-01-11T09:46:40.016036+0000","last_clean":"2024-01-11T09:46:40.016036+0000","last_became_active":"2024-01-11T09:46:40.015608+0000","last_became_peered":"2024-01-11T09:46:40.015608+0000","last_unstale":"2024-01-11T09:46:40.016036+0000","last_undegraded":"2024-01-11T09:46:40.016036+0000","last_fullsized":"2024-01-11T09:46:40.016036+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:44:19.712167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986139+0000","last_change":"2024-01-11T09:46:25.954029+0000","last_active":"2024-01-11T09:46:38.986139+0000","last_peered":"2024-01-11T09:46:38.986139+0000","last_clean":"2024-01-11T09:46:38.986139+0000","last_became_active":"2024-01-11T09:46:25.953690+0000","last_became_peered":"2024-01-11T09:46:25.953690+0000","last_unstale":"2024-01-11T09:46:38.986139+0000","last_undegraded":"2024-01-11T09:46:38.986139+0000","last_fullsized":"2024-01-11T09:46:38.986139+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:26:46.984194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"234'2","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.985187+0000","last_change":"2024-01-11T09:46:25.227884+0000","last_active":"2024-01-11T09:46:38.985187+0000","last_peered":"2024-01-11T09:46:38.985187+0000","last_clean":"2024-01-11T09:46:38.985187+0000","last_became_active":"2024-01-11T09:46:25.227605+0000","last_became_peered":"2024-01-11T09:46:25.227605+0000","last_unstale":"2024-01-11T09:46:38.985187+0000","last_undegraded":"2024-01-11T09:46:38.985187+0000","last_fullsized":"2024-01-11T09:46:38.985187+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:09.116268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.010140+0000","last_change":"2024-01-11T09:46:40.010140+0000","last_active":"2024-01-11T09:46:40.010140+0000","last_peered":"2024-01-11T09:46:40.010140+0000","last_clean":"2024-01-11T09:46:40.010140+0000","last_became_active":"2024-01-11T09:46:40.009493+0000","last_became_peered":"2024-01-11T09:46:40.009493+0000","last_unstale":"2024-01-11T09:46:40.010140+0000","last_undegraded":"2024-01-11T09:46:40.010140+0000","last_fullsized":"2024-01-11T09:46:40.010140+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:25:23.340011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.977338+0000","last_change":"2024-01-11T09:46:25.950375+0000","last_active":"2024-01-11T09:46:38.977338+0000","last_peered":"2024-01-11T09:46:38.977338+0000","last_clean":"2024-01-11T09:46:38.977338+0000","last_became_active":"2024-01-11T09:46:25.950114+0000","last_became_peered":"2024-01-11T09:46:25.950114+0000","last_unstale":"2024-01-11T09:46:38.977338+0000","last_undegraded":"2024-01-11T09:46:38.977338+0000","last_fullsized":"2024-01-11T09:46:38.977338+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:25.226766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"234'1","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.993589+0000","last_change":"2024-01-11T09:46:25.233693+0000","last_active":"2024-01-11T09:46:38.993589+0000","last_peered":"2024-01-11T09:46:38.993589+0000","last_clean":"2024-01-11T09:46:38.993589+0000","last_became_active":"2024-01-11T09:46:25.233410+0000","last_became_peered":"2024-01-11T09:46:25.233410+0000","last_unstale":"2024-01-11T09:46:38.993589+0000","last_undegraded":"2024-01-11T09:46:38.993589+0000","last_fullsized":"2024-01-11T09:46:38.993589+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:03:42.704031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.003083+0000","last_change":"2024-01-11T09:46:40.003083+0000","last_active":"2024-01-11T09:46:40.003083+0000","last_peered":"2024-01-11T09:46:40.003083+0000","last_clean":"2024-01-11T09:46:40.003083+0000","last_became_active":"2024-01-11T09:46:40.002819+0000","last_became_peered":"2024-01-11T09:46:40.002819+0000","last_unstale":"2024-01-11T09:46:40.003083+0000","last_undegraded":"2024-01-11T09:46:40.003083+0000","last_fullsized":"2024-01-11T09:46:40.003083+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:47:54.704748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.981687+0000","last_change":"2024-01-11T09:46:25.955874+0000","last_active":"2024-01-11T09:46:38.981687+0000","last_peered":"2024-01-11T09:46:38.981687+0000","last_clean":"2024-01-11T09:46:38.981687+0000","last_became_active":"2024-01-11T09:46:25.954789+0000","last_became_peered":"2024-01-11T09:46:25.954789+0000","last_unstale":"2024-01-11T09:46:38.981687+0000","last_undegraded":"2024-01-11T09:46:38.981687+0000","last_fullsized":"2024-01-11T09:46:38.981687+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:07.160563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"234'1","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.988905+0000","last_change":"2024-01-11T09:46:25.235297+0000","last_active":"2024-01-11T09:46:38.988905+0000","last_peered":"2024-01-11T09:46:38.988905+0000","last_clean":"2024-01-11T09:46:38.988905+0000","last_became_active":"2024-01-11T09:46:25.234744+0000","last_became_peered":"2024-01-11T09:46:25.234744+0000","last_unstale":"2024-01-11T09:46:38.988905+0000","last_undegraded":"2024-01-11T09:46:38.988905+0000","last_fullsized":"2024-01-11T09:46:38.988905+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:51:44.543489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.995389+0000","last_change":"2024-01-11T09:46:39.995389+0000","last_active":"2024-01-11T09:46:39.995389+0000","last_peered":"2024-01-11T09:46:39.995389+0000","last_clean":"2024-01-11T09:46:39.995389+0000","last_became_active":"2024-01-11T09:46:39.995111+0000","last_became_peered":"2024-01-11T09:46:39.995111+0000","last_unstale":"2024-01-11T09:46:39.995389+0000","last_undegraded":"2024-01-11T09:46:39.995389+0000","last_fullsized":"2024-01-11T09:46:39.995389+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:07:27.039113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980841+0000","last_change":"2024-01-11T09:46:25.948668+0000","last_active":"2024-01-11T09:46:38.980841+0000","last_peered":"2024-01-11T09:46:38.980841+0000","last_clean":"2024-01-11T09:46:38.980841+0000","last_became_active":"2024-01-11T09:46:25.948372+0000","last_became_peered":"2024-01-11T09:46:25.948372+0000","last_unstale":"2024-01-11T09:46:38.980841+0000","last_undegraded":"2024-01-11T09:46:38.980841+0000","last_fullsized":"2024-01-11T09:46:38.980841+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:41.154829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980164+0000","last_change":"2024-01-11T09:46:25.238227+0000","last_active":"2024-01-11T09:46:38.980164+0000","last_peered":"2024-01-11T09:46:38.980164+0000","last_clean":"2024-01-11T09:46:38.980164+0000","last_became_active":"2024-01-11T09:46:25.238013+0000","last_became_peered":"2024-01-11T09:46:25.238013+0000","last_unstale":"2024-01-11T09:46:38.980164+0000","last_undegraded":"2024-01-11T09:46:38.980164+0000","last_fullsized":"2024-01-11T09:46:38.980164+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:36:18.777304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.999708+0000","last_change":"2024-01-11T09:46:39.999708+0000","last_active":"2024-01-11T09:46:39.999708+0000","last_peered":"2024-01-11T09:46:39.999708+0000","last_clean":"2024-01-11T09:46:39.999708+0000","last_became_active":"2024-01-11T09:46:39.999341+0000","last_became_peered":"2024-01-11T09:46:39.999341+0000","last_unstale":"2024-01-11T09:46:39.999708+0000","last_undegraded":"2024-01-11T09:46:39.999708+0000","last_fullsized":"2024-01-11T09:46:39.999708+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:09:49.159232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.988745+0000","last_change":"2024-01-11T09:46:25.949014+0000","last_active":"2024-01-11T09:46:38.988745+0000","last_peered":"2024-01-11T09:46:38.988745+0000","last_clean":"2024-01-11T09:46:38.988745+0000","last_became_active":"2024-01-11T09:46:25.948717+0000","last_became_peered":"2024-01-11T09:46:25.948717+0000","last_unstale":"2024-01-11T09:46:38.988745+0000","last_undegraded":"2024-01-11T09:46:38.988745+0000","last_fullsized":"2024-01-11T09:46:38.988745+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:07.160563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980927+0000","last_change":"2024-01-11T09:46:25.225980+0000","last_active":"2024-01-11T09:46:38.980927+0000","last_peered":"2024-01-11T09:46:38.980927+0000","last_clean":"2024-01-11T09:46:38.980927+0000","last_became_active":"2024-01-11T09:46:25.225444+0000","last_became_peered":"2024-01-11T09:46:25.225444+0000","last_unstale":"2024-01-11T09:46:38.980927+0000","last_undegraded":"2024-01-11T09:46:38.980927+0000","last_fullsized":"2024-01-11T09:46:38.980927+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:30:19.648096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.989639+0000","last_change":"2024-01-11T09:46:39.989639+0000","last_active":"2024-01-11T09:46:39.989639+0000","last_peered":"2024-01-11T09:46:39.989639+0000","last_clean":"2024-01-11T09:46:39.989639+0000","last_became_active":"2024-01-11T09:46:39.989397+0000","last_became_peered":"2024-01-11T09:46:39.989397+0000","last_unstale":"2024-01-11T09:46:39.989639+0000","last_undegraded":"2024-01-11T09:46:39.989639+0000","last_fullsized":"2024-01-11T09:46:39.989639+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:31:00.593411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990657+0000","last_change":"2024-01-11T09:46:25.953219+0000","last_active":"2024-01-11T09:46:38.990657+0000","last_peered":"2024-01-11T09:46:38.990657+0000","last_clean":"2024-01-11T09:46:38.990657+0000","last_became_active":"2024-01-11T09:46:25.952967+0000","last_became_peered":"2024-01-11T09:46:25.952967+0000","last_unstale":"2024-01-11T09:46:38.990657+0000","last_undegraded":"2024-01-11T09:46:38.990657+0000","last_fullsized":"2024-01-11T09:46:38.990657+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:07:13.288202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986207+0000","last_change":"2024-01-11T09:46:25.237685+0000","last_active":"2024-01-11T09:46:38.986207+0000","last_peered":"2024-01-11T09:46:38.986207+0000","last_clean":"2024-01-11T09:46:38.986207+0000","last_became_active":"2024-01-11T09:46:25.237419+0000","last_became_peered":"2024-01-11T09:46:25.237419+0000","last_unstale":"2024-01-11T09:46:38.986207+0000","last_undegraded":"2024-01-11T09:46:38.986207+0000","last_fullsized":"2024-01-11T09:46:38.986207+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:52.221452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.995097+0000","last_change":"2024-01-11T09:46:39.995097+0000","last_active":"2024-01-11T09:46:39.995097+0000","last_peered":"2024-01-11T09:46:39.995097+0000","last_clean":"2024-01-11T09:46:39.995097+0000","last_became_active":"2024-01-11T09:46:39.994814+0000","last_became_peered":"2024-01-11T09:46:39.994814+0000","last_unstale":"2024-01-11T09:46:39.995097+0000","last_undegraded":"2024-01-11T09:46:39.995097+0000","last_fullsized":"2024-01-11T09:46:39.995097+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:54:22.183997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.979295+0000","last_change":"2024-01-11T09:46:25.950838+0000","last_active":"2024-01-11T09:46:38.979295+0000","last_peered":"2024-01-11T09:46:38.979295+0000","last_clean":"2024-01-11T09:46:38.979295+0000","last_became_active":"2024-01-11T09:46:25.950551+0000","last_became_peered":"2024-01-11T09:46:25.950551+0000","last_unstale":"2024-01-11T09:46:38.979295+0000","last_undegraded":"2024-01-11T09:46:38.979295+0000","last_fullsized":"2024-01-11T09:46:38.979295+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:39:05.971547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.982892+0000","last_change":"2024-01-11T09:46:25.232121+0000","last_active":"2024-01-11T09:46:38.982892+0000","last_peered":"2024-01-11T09:46:38.982892+0000","last_clean":"2024-01-11T09:46:38.982892+0000","last_became_active":"2024-01-11T09:46:25.231859+0000","last_became_peered":"2024-01-11T09:46:25.231859+0000","last_unstale":"2024-01-11T09:46:38.982892+0000","last_undegraded":"2024-01-11T09:46:38.982892+0000","last_fullsized":"2024-01-11T09:46:38.982892+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:07:12.279056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.013866+0000","last_change":"2024-01-11T09:46:40.013866+0000","last_active":"2024-01-11T09:46:40.013866+0000","last_peered":"2024-01-11T09:46:40.013866+0000","last_clean":"2024-01-11T09:46:40.013866+0000","last_became_active":"2024-01-11T09:46:40.013285+0000","last_became_peered":"2024-01-11T09:46:40.013285+0000","last_unstale":"2024-01-11T09:46:40.013866+0000","last_undegraded":"2024-01-11T09:46:40.013866+0000","last_fullsized":"2024-01-11T09:46:40.013866+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:44:19.712167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984855+0000","last_change":"2024-01-11T09:46:25.948806+0000","last_active":"2024-01-11T09:46:38.984855+0000","last_peered":"2024-01-11T09:46:38.984855+0000","last_clean":"2024-01-11T09:46:38.984855+0000","last_became_active":"2024-01-11T09:46:25.948526+0000","last_became_peered":"2024-01-11T09:46:25.948526+0000","last_unstale":"2024-01-11T09:46:38.984855+0000","last_undegraded":"2024-01-11T09:46:38.984855+0000","last_fullsized":"2024-01-11T09:46:38.984855+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:51:19.504238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"234'3","reported_seq":16,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.991439+0000","last_change":"2024-01-11T09:46:25.238242+0000","last_active":"2024-01-11T09:46:38.991439+0000","last_peered":"2024-01-11T09:46:38.991439+0000","last_clean":"2024-01-11T09:46:38.991439+0000","last_became_active":"2024-01-11T09:46:25.237965+0000","last_became_peered":"2024-01-11T09:46:25.237965+0000","last_unstale":"2024-01-11T09:46:38.991439+0000","last_undegraded":"2024-01-11T09:46:38.991439+0000","last_fullsized":"2024-01-11T09:46:38.991439+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:58:02.388540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.003570+0000","last_change":"2024-01-11T09:46:40.003570+0000","last_active":"2024-01-11T09:46:40.003570+0000","last_peered":"2024-01-11T09:46:40.003570+0000","last_clean":"2024-01-11T09:46:40.003570+0000","last_became_active":"2024-01-11T09:46:40.002034+0000","last_became_peered":"2024-01-11T09:46:40.002034+0000","last_unstale":"2024-01-11T09:46:40.003570+0000","last_undegraded":"2024-01-11T09:46:40.003570+0000","last_fullsized":"2024-01-11T09:46:40.003570+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:51:27.846178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.981368+0000","last_change":"2024-01-11T09:46:25.950108+0000","last_active":"2024-01-11T09:46:38.981368+0000","last_peered":"2024-01-11T09:46:38.981368+0000","last_clean":"2024-01-11T09:46:38.981368+0000","last_became_active":"2024-01-11T09:46:25.949796+0000","last_became_peered":"2024-01-11T09:46:25.949796+0000","last_unstale":"2024-01-11T09:46:38.981368+0000","last_undegraded":"2024-01-11T09:46:38.981368+0000","last_fullsized":"2024-01-11T09:46:38.981368+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:11:07.968385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"235'8","reported_seq":21,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990505+0000","last_change":"2024-01-11T09:46:25.239177+0000","last_active":"2024-01-11T09:46:38.990505+0000","last_peered":"2024-01-11T09:46:38.990505+0000","last_clean":"2024-01-11T09:46:38.990505+0000","last_became_active":"2024-01-11T09:46:25.238865+0000","last_became_peered":"2024-01-11T09:46:25.238865+0000","last_unstale":"2024-01-11T09:46:38.990505+0000","last_undegraded":"2024-01-11T09:46:38.990505+0000","last_fullsized":"2024-01-11T09:46:38.990505+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:29:50.151952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2749,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.017512+0000","last_change":"2024-01-11T09:46:40.017512+0000","last_active":"2024-01-11T09:46:40.017512+0000","last_peered":"2024-01-11T09:46:40.017512+0000","last_clean":"2024-01-11T09:46:40.017512+0000","last_became_active":"2024-01-11T09:46:40.017221+0000","last_became_peered":"2024-01-11T09:46:40.017221+0000","last_unstale":"2024-01-11T09:46:40.017512+0000","last_undegraded":"2024-01-11T09:46:40.017512+0000","last_fullsized":"2024-01-11T09:46:40.017512+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:03:28.606890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984235+0000","last_change":"2024-01-11T09:46:25.951823+0000","last_active":"2024-01-11T09:46:38.984235+0000","last_peered":"2024-01-11T09:46:38.984235+0000","last_clean":"2024-01-11T09:46:38.984235+0000","last_became_active":"2024-01-11T09:46:25.951493+0000","last_became_peered":"2024-01-11T09:46:25.951493+0000","last_unstale":"2024-01-11T09:46:38.984235+0000","last_undegraded":"2024-01-11T09:46:38.984235+0000","last_fullsized":"2024-01-11T09:46:38.984235+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:59:22.974857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"234'2","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990118+0000","last_change":"2024-01-11T09:46:25.244293+0000","last_active":"2024-01-11T09:46:38.990118+0000","last_peered":"2024-01-11T09:46:38.990118+0000","last_clean":"2024-01-11T09:46:38.990118+0000","last_became_active":"2024-01-11T09:46:25.243959+0000","last_became_peered":"2024-01-11T09:46:25.243959+0000","last_unstale":"2024-01-11T09:46:38.990118+0000","last_undegraded":"2024-01-11T09:46:38.990118+0000","last_fullsized":"2024-01-11T09:46:38.990118+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:13:10.353715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.007024+0000","last_change":"2024-01-11T09:46:40.007024+0000","last_active":"2024-01-11T09:46:40.007024+0000","last_peered":"2024-01-11T09:46:40.007024+0000","last_clean":"2024-01-11T09:46:40.007024+0000","last_became_active":"2024-01-11T09:46:40.006049+0000","last_became_peered":"2024-01-11T09:46:40.006049+0000","last_unstale":"2024-01-11T09:46:40.007024+0000","last_undegraded":"2024-01-11T09:46:40.007024+0000","last_fullsized":"2024-01-11T09:46:40.007024+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:45:58.839035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.982248+0000","last_change":"2024-01-11T09:46:25.958245+0000","last_active":"2024-01-11T09:46:38.982248+0000","last_peered":"2024-01-11T09:46:38.982248+0000","last_clean":"2024-01-11T09:46:38.982248+0000","last_became_active":"2024-01-11T09:46:25.957893+0000","last_became_peered":"2024-01-11T09:46:25.957893+0000","last_unstale":"2024-01-11T09:46:38.982248+0000","last_undegraded":"2024-01-11T09:46:38.982248+0000","last_fullsized":"2024-01-11T09:46:38.982248+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:43:22.062566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.982221+0000","last_change":"2024-01-11T09:46:25.231846+0000","last_active":"2024-01-11T09:46:38.982221+0000","last_peered":"2024-01-11T09:46:38.982221+0000","last_clean":"2024-01-11T09:46:38.982221+0000","last_became_active":"2024-01-11T09:46:25.229498+0000","last_became_peered":"2024-01-11T09:46:25.229498+0000","last_unstale":"2024-01-11T09:46:38.982221+0000","last_undegraded":"2024-01-11T09:46:38.982221+0000","last_fullsized":"2024-01-11T09:46:38.982221+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:53:34.130109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.001720+0000","last_change":"2024-01-11T09:46:40.001720+0000","last_active":"2024-01-11T09:46:40.001720+0000","last_peered":"2024-01-11T09:46:40.001720+0000","last_clean":"2024-01-11T09:46:40.001720+0000","last_became_active":"2024-01-11T09:46:40.001315+0000","last_became_peered":"2024-01-11T09:46:40.001315+0000","last_unstale":"2024-01-11T09:46:40.001720+0000","last_undegraded":"2024-01-11T09:46:40.001720+0000","last_fullsized":"2024-01-11T09:46:40.001720+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:14:18.913296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984870+0000","last_change":"2024-01-11T09:46:25.947333+0000","last_active":"2024-01-11T09:46:38.984870+0000","last_peered":"2024-01-11T09:46:38.984870+0000","last_clean":"2024-01-11T09:46:38.984870+0000","last_became_active":"2024-01-11T09:46:25.946307+0000","last_became_peered":"2024-01-11T09:46:25.946307+0000","last_unstale":"2024-01-11T09:46:38.984870+0000","last_undegraded":"2024-01-11T09:46:38.984870+0000","last_fullsized":"2024-01-11T09:46:38.984870+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:32:47.063220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980245+0000","last_change":"2024-01-11T09:46:25.239044+0000","last_active":"2024-01-11T09:46:38.980245+0000","last_peered":"2024-01-11T09:46:38.980245+0000","last_clean":"2024-01-11T09:46:38.980245+0000","last_became_active":"2024-01-11T09:46:25.238780+0000","last_became_peered":"2024-01-11T09:46:25.238780+0000","last_unstale":"2024-01-11T09:46:38.980245+0000","last_undegraded":"2024-01-11T09:46:38.980245+0000","last_fullsized":"2024-01-11T09:46:38.980245+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:34:37.940301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.002357+0000","last_change":"2024-01-11T09:46:40.002357+0000","last_active":"2024-01-11T09:46:40.002357+0000","last_peered":"2024-01-11T09:46:40.002357+0000","last_clean":"2024-01-11T09:46:40.002357+0000","last_became_active":"2024-01-11T09:46:39.994174+0000","last_became_peered":"2024-01-11T09:46:39.994174+0000","last_unstale":"2024-01-11T09:46:40.002357+0000","last_undegraded":"2024-01-11T09:46:40.002357+0000","last_fullsized":"2024-01-11T09:46:40.002357+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:49:09.210651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980062+0000","last_change":"2024-01-11T09:46:25.962027+0000","last_active":"2024-01-11T09:46:38.980062+0000","last_peered":"2024-01-11T09:46:38.980062+0000","last_clean":"2024-01-11T09:46:38.980062+0000","last_became_active":"2024-01-11T09:46:25.961646+0000","last_became_peered":"2024-01-11T09:46:25.961646+0000","last_unstale":"2024-01-11T09:46:38.980062+0000","last_undegraded":"2024-01-11T09:46:38.980062+0000","last_fullsized":"2024-01-11T09:46:38.980062+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:39:05.971547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"234'8","reported_seq":21,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.977073+0000","last_change":"2024-01-11T09:46:25.238694+0000","last_active":"2024-01-11T09:46:38.977073+0000","last_peered":"2024-01-11T09:46:38.977073+0000","last_clean":"2024-01-11T09:46:38.977073+0000","last_became_active":"2024-01-11T09:46:25.237565+0000","last_became_peered":"2024-01-11T09:46:25.237565+0000","last_unstale":"2024-01-11T09:46:38.977073+0000","last_undegraded":"2024-01-11T09:46:38.977073+0000","last_fullsized":"2024-01-11T09:46:38.977073+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:25:08.579955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.014333+0000","last_change":"2024-01-11T09:46:40.014333+0000","last_active":"2024-01-11T09:46:40.014333+0000","last_peered":"2024-01-11T09:46:40.014333+0000","last_clean":"2024-01-11T09:46:40.014333+0000","last_became_active":"2024-01-11T09:46:40.014085+0000","last_became_peered":"2024-01-11T09:46:40.014085+0000","last_unstale":"2024-01-11T09:46:40.014333+0000","last_undegraded":"2024-01-11T09:46:40.014333+0000","last_fullsized":"2024-01-11T09:46:40.014333+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:34:54.101440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.994750+0000","last_change":"2024-01-11T09:46:25.946186+0000","last_active":"2024-01-11T09:46:38.994750+0000","last_peered":"2024-01-11T09:46:38.994750+0000","last_clean":"2024-01-11T09:46:38.994750+0000","last_became_active":"2024-01-11T09:46:25.945907+0000","last_became_peered":"2024-01-11T09:46:25.945907+0000","last_unstale":"2024-01-11T09:46:38.994750+0000","last_undegraded":"2024-01-11T09:46:38.994750+0000","last_fullsized":"2024-01-11T09:46:38.994750+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:26:51.516156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"234'1","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.981243+0000","last_change":"2024-01-11T09:46:25.241687+0000","last_active":"2024-01-11T09:46:38.981243+0000","last_peered":"2024-01-11T09:46:38.981243+0000","last_clean":"2024-01-11T09:46:38.981243+0000","last_became_active":"2024-01-11T09:46:25.241369+0000","last_became_peered":"2024-01-11T09:46:25.241369+0000","last_unstale":"2024-01-11T09:46:38.981243+0000","last_undegraded":"2024-01-11T09:46:38.981243+0000","last_fullsized":"2024-01-11T09:46:38.981243+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:40.145683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.999085+0000","last_change":"2024-01-11T09:46:39.999085+0000","last_active":"2024-01-11T09:46:39.999085+0000","last_peered":"2024-01-11T09:46:39.999085+0000","last_clean":"2024-01-11T09:46:39.999085+0000","last_became_active":"2024-01-11T09:46:39.997666+0000","last_became_peered":"2024-01-11T09:46:39.997666+0000","last_unstale":"2024-01-11T09:46:39.999085+0000","last_undegraded":"2024-01-11T09:46:39.999085+0000","last_fullsized":"2024-01-11T09:46:39.999085+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:48:30.105547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984370+0000","last_change":"2024-01-11T09:46:25.951531+0000","last_active":"2024-01-11T09:46:38.984370+0000","last_peered":"2024-01-11T09:46:38.984370+0000","last_clean":"2024-01-11T09:46:38.984370+0000","last_became_active":"2024-01-11T09:46:25.950791+0000","last_became_peered":"2024-01-11T09:46:25.950791+0000","last_unstale":"2024-01-11T09:46:38.984370+0000","last_undegraded":"2024-01-11T09:46:38.984370+0000","last_fullsized":"2024-01-11T09:46:38.984370+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:09:35.408321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"234'2","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.982795+0000","last_change":"2024-01-11T09:46:25.234950+0000","last_active":"2024-01-11T09:46:38.982795+0000","last_peered":"2024-01-11T09:46:38.982795+0000","last_clean":"2024-01-11T09:46:38.982795+0000","last_became_active":"2024-01-11T09:46:25.234510+0000","last_became_peered":"2024-01-11T09:46:25.234510+0000","last_unstale":"2024-01-11T09:46:38.982795+0000","last_undegraded":"2024-01-11T09:46:38.982795+0000","last_fullsized":"2024-01-11T09:46:38.982795+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:33:13.743309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.002994+0000","last_change":"2024-01-11T09:46:40.002994+0000","last_active":"2024-01-11T09:46:40.002994+0000","last_peered":"2024-01-11T09:46:40.002994+0000","last_clean":"2024-01-11T09:46:40.002994+0000","last_became_active":"2024-01-11T09:46:40.002700+0000","last_became_peered":"2024-01-11T09:46:40.002700+0000","last_unstale":"2024-01-11T09:46:40.002994+0000","last_undegraded":"2024-01-11T09:46:40.002994+0000","last_fullsized":"2024-01-11T09:46:40.002994+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:54.905740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.982536+0000","last_change":"2024-01-11T09:46:25.961077+0000","last_active":"2024-01-11T09:46:38.982536+0000","last_peered":"2024-01-11T09:46:38.982536+0000","last_clean":"2024-01-11T09:46:38.982536+0000","last_became_active":"2024-01-11T09:46:25.960690+0000","last_became_peered":"2024-01-11T09:46:25.960690+0000","last_unstale":"2024-01-11T09:46:38.982536+0000","last_undegraded":"2024-01-11T09:46:38.982536+0000","last_fullsized":"2024-01-11T09:46:38.982536+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:59:52.045856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"234'1","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.982750+0000","last_change":"2024-01-11T09:46:25.235005+0000","last_active":"2024-01-11T09:46:38.982750+0000","last_peered":"2024-01-11T09:46:38.982750+0000","last_clean":"2024-01-11T09:46:38.982750+0000","last_became_active":"2024-01-11T09:46:25.234748+0000","last_became_peered":"2024-01-11T09:46:25.234748+0000","last_unstale":"2024-01-11T09:46:38.982750+0000","last_undegraded":"2024-01-11T09:46:38.982750+0000","last_fullsized":"2024-01-11T09:46:38.982750+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:29:50.151952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.011411+0000","last_change":"2024-01-11T09:46:40.011411+0000","last_active":"2024-01-11T09:46:40.011411+0000","last_peered":"2024-01-11T09:46:40.011411+0000","last_clean":"2024-01-11T09:46:40.011411+0000","last_became_active":"2024-01-11T09:46:40.011120+0000","last_became_peered":"2024-01-11T09:46:40.011120+0000","last_unstale":"2024-01-11T09:46:40.011411+0000","last_undegraded":"2024-01-11T09:46:40.011411+0000","last_fullsized":"2024-01-11T09:46:40.011411+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:59:36.725768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986243+0000","last_change":"2024-01-11T09:46:25.952457+0000","last_active":"2024-01-11T09:46:38.986243+0000","last_peered":"2024-01-11T09:46:38.986243+0000","last_clean":"2024-01-11T09:46:38.986243+0000","last_became_active":"2024-01-11T09:46:25.948394+0000","last_became_peered":"2024-01-11T09:46:25.948394+0000","last_unstale":"2024-01-11T09:46:38.986243+0000","last_undegraded":"2024-01-11T09:46:38.986243+0000","last_fullsized":"2024-01-11T09:46:38.986243+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:49:33.997022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.977796+0000","last_change":"2024-01-11T09:46:25.242589+0000","last_active":"2024-01-11T09:46:38.977796+0000","last_peered":"2024-01-11T09:46:38.977796+0000","last_clean":"2024-01-11T09:46:38.977796+0000","last_became_active":"2024-01-11T09:46:25.241879+0000","last_became_peered":"2024-01-11T09:46:25.241879+0000","last_unstale":"2024-01-11T09:46:38.977796+0000","last_undegraded":"2024-01-11T09:46:38.977796+0000","last_fullsized":"2024-01-11T09:46:38.977796+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:40.145683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.998699+0000","last_change":"2024-01-11T09:46:39.998699+0000","last_active":"2024-01-11T09:46:39.998699+0000","last_peered":"2024-01-11T09:46:39.998699+0000","last_clean":"2024-01-11T09:46:39.998699+0000","last_became_active":"2024-01-11T09:46:39.998426+0000","last_became_peered":"2024-01-11T09:46:39.998426+0000","last_unstale":"2024-01-11T09:46:39.998699+0000","last_undegraded":"2024-01-11T09:46:39.998699+0000","last_fullsized":"2024-01-11T09:46:39.998699+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:26:11.710312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986453+0000","last_change":"2024-01-11T09:46:25.953778+0000","last_active":"2024-01-11T09:46:38.986453+0000","last_peered":"2024-01-11T09:46:38.986453+0000","last_clean":"2024-01-11T09:46:38.986453+0000","last_became_active":"2024-01-11T09:46:25.953471+0000","last_became_peered":"2024-01-11T09:46:25.953471+0000","last_unstale":"2024-01-11T09:46:38.986453+0000","last_undegraded":"2024-01-11T09:46:38.986453+0000","last_fullsized":"2024-01-11T09:46:38.986453+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:30:20.657242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"234'2","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.982618+0000","last_change":"2024-01-11T09:46:25.236418+0000","last_active":"2024-01-11T09:46:38.982618+0000","last_peered":"2024-01-11T09:46:38.982618+0000","last_clean":"2024-01-11T09:46:38.982618+0000","last_became_active":"2024-01-11T09:46:25.236155+0000","last_became_peered":"2024-01-11T09:46:25.236155+0000","last_unstale":"2024-01-11T09:46:38.982618+0000","last_undegraded":"2024-01-11T09:46:38.982618+0000","last_fullsized":"2024-01-11T09:46:38.982618+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:59:13.998537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.018015+0000","last_change":"2024-01-11T09:46:40.018015+0000","last_active":"2024-01-11T09:46:40.018015+0000","last_peered":"2024-01-11T09:46:40.018015+0000","last_clean":"2024-01-11T09:46:40.018015+0000","last_became_active":"2024-01-11T09:46:40.017803+0000","last_became_peered":"2024-01-11T09:46:40.017803+0000","last_unstale":"2024-01-11T09:46:40.018015+0000","last_undegraded":"2024-01-11T09:46:40.018015+0000","last_fullsized":"2024-01-11T09:46:40.018015+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:16:06.119899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.989798+0000","last_change":"2024-01-11T09:46:25.954393+0000","last_active":"2024-01-11T09:46:38.989798+0000","last_peered":"2024-01-11T09:46:38.989798+0000","last_clean":"2024-01-11T09:46:38.989798+0000","last_became_active":"2024-01-11T09:46:25.954139+0000","last_became_peered":"2024-01-11T09:46:25.954139+0000","last_unstale":"2024-01-11T09:46:38.989798+0000","last_undegraded":"2024-01-11T09:46:38.989798+0000","last_fullsized":"2024-01-11T09:46:38.989798+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:33:14.752455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.977474+0000","last_change":"2024-01-11T09:46:25.243586+0000","last_active":"2024-01-11T09:46:38.977474+0000","last_peered":"2024-01-11T09:46:38.977474+0000","last_clean":"2024-01-11T09:46:38.977474+0000","last_became_active":"2024-01-11T09:46:25.243184+0000","last_became_peered":"2024-01-11T09:46:25.243184+0000","last_unstale":"2024-01-11T09:46:38.977474+0000","last_undegraded":"2024-01-11T09:46:38.977474+0000","last_fullsized":"2024-01-11T09:46:38.977474+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:01:24.091035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.002684+0000","last_change":"2024-01-11T09:46:40.002684+0000","last_active":"2024-01-11T09:46:40.002684+0000","last_peered":"2024-01-11T09:46:40.002684+0000","last_clean":"2024-01-11T09:46:40.002684+0000","last_became_active":"2024-01-11T09:46:40.002048+0000","last_became_peered":"2024-01-11T09:46:40.002048+0000","last_unstale":"2024-01-11T09:46:40.002684+0000","last_undegraded":"2024-01-11T09:46:40.002684+0000","last_fullsized":"2024-01-11T09:46:40.002684+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:58:47.077944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.977074+0000","last_change":"2024-01-11T09:46:25.953319+0000","last_active":"2024-01-11T09:46:38.977074+0000","last_peered":"2024-01-11T09:46:38.977074+0000","last_clean":"2024-01-11T09:46:38.977074+0000","last_became_active":"2024-01-11T09:46:25.953063+0000","last_became_peered":"2024-01-11T09:46:25.953063+0000","last_unstale":"2024-01-11T09:46:38.977074+0000","last_undegraded":"2024-01-11T09:46:38.977074+0000","last_fullsized":"2024-01-11T09:46:38.977074+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:31:56.044420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980122+0000","last_change":"2024-01-11T09:46:25.243348+0000","last_active":"2024-01-11T09:46:38.980122+0000","last_peered":"2024-01-11T09:46:38.980122+0000","last_clean":"2024-01-11T09:46:38.980122+0000","last_became_active":"2024-01-11T09:46:25.243043+0000","last_became_peered":"2024-01-11T09:46:25.243043+0000","last_unstale":"2024-01-11T09:46:38.980122+0000","last_undegraded":"2024-01-11T09:46:38.980122+0000","last_fullsized":"2024-01-11T09:46:38.980122+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:13:10.353715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.998085+0000","last_change":"2024-01-11T09:46:39.998085+0000","last_active":"2024-01-11T09:46:39.998085+0000","last_peered":"2024-01-11T09:46:39.998085+0000","last_clean":"2024-01-11T09:46:39.998085+0000","last_became_active":"2024-01-11T09:46:39.994400+0000","last_became_peered":"2024-01-11T09:46:39.994400+0000","last_unstale":"2024-01-11T09:46:39.998085+0000","last_undegraded":"2024-01-11T09:46:39.998085+0000","last_fullsized":"2024-01-11T09:46:39.998085+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:31:00.593411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986541+0000","last_change":"2024-01-11T09:46:25.956539+0000","last_active":"2024-01-11T09:46:38.986541+0000","last_peered":"2024-01-11T09:46:38.986541+0000","last_clean":"2024-01-11T09:46:38.986541+0000","last_became_active":"2024-01-11T09:46:25.953109+0000","last_became_peered":"2024-01-11T09:46:25.953109+0000","last_unstale":"2024-01-11T09:46:38.986541+0000","last_undegraded":"2024-01-11T09:46:38.986541+0000","last_fullsized":"2024-01-11T09:46:38.986541+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:39:37.637123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980073+0000","last_change":"2024-01-11T09:46:25.234212+0000","last_active":"2024-01-11T09:46:38.980073+0000","last_peered":"2024-01-11T09:46:38.980073+0000","last_clean":"2024-01-11T09:46:38.980073+0000","last_became_active":"2024-01-11T09:46:25.233918+0000","last_became_peered":"2024-01-11T09:46:25.233918+0000","last_unstale":"2024-01-11T09:46:38.980073+0000","last_undegraded":"2024-01-11T09:46:38.980073+0000","last_fullsized":"2024-01-11T09:46:38.980073+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:39:17.328191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.001121+0000","last_change":"2024-01-11T09:46:40.001121+0000","last_active":"2024-01-11T09:46:40.001121+0000","last_peered":"2024-01-11T09:46:40.001121+0000","last_clean":"2024-01-11T09:46:40.001121+0000","last_became_active":"2024-01-11T09:46:40.000620+0000","last_became_peered":"2024-01-11T09:46:40.000620+0000","last_unstale":"2024-01-11T09:46:40.001121+0000","last_undegraded":"2024-01-11T09:46:40.001121+0000","last_fullsized":"2024-01-11T09:46:40.001121+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:40:21.583291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.981029+0000","last_change":"2024-01-11T09:46:25.953810+0000","last_active":"2024-01-11T09:46:38.981029+0000","last_peered":"2024-01-11T09:46:38.981029+0000","last_clean":"2024-01-11T09:46:38.981029+0000","last_became_active":"2024-01-11T09:46:25.953201+0000","last_became_peered":"2024-01-11T09:46:25.953201+0000","last_unstale":"2024-01-11T09:46:38.981029+0000","last_undegraded":"2024-01-11T09:46:38.981029+0000","last_fullsized":"2024-01-11T09:46:38.981029+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:25.226766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"234'2","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.995159+0000","last_change":"2024-01-11T09:46:25.245359+0000","last_active":"2024-01-11T09:46:38.995159+0000","last_peered":"2024-01-11T09:46:38.995159+0000","last_clean":"2024-01-11T09:46:38.995159+0000","last_became_active":"2024-01-11T09:46:25.244994+0000","last_became_peered":"2024-01-11T09:46:25.244994+0000","last_unstale":"2024-01-11T09:46:38.995159+0000","last_undegraded":"2024-01-11T09:46:38.995159+0000","last_fullsized":"2024-01-11T09:46:38.995159+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:42:13.269515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.994498+0000","last_change":"2024-01-11T09:46:39.994498+0000","last_active":"2024-01-11T09:46:39.994498+0000","last_peered":"2024-01-11T09:46:39.994498+0000","last_clean":"2024-01-11T09:46:39.994498+0000","last_became_active":"2024-01-11T09:46:39.994203+0000","last_became_peered":"2024-01-11T09:46:39.994203+0000","last_unstale":"2024-01-11T09:46:39.994498+0000","last_undegraded":"2024-01-11T09:46:39.994498+0000","last_fullsized":"2024-01-11T09:46:39.994498+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:50:15.419725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.994342+0000","last_change":"2024-01-11T09:46:25.952007+0000","last_active":"2024-01-11T09:46:38.994342+0000","last_peered":"2024-01-11T09:46:38.994342+0000","last_clean":"2024-01-11T09:46:38.994342+0000","last_became_active":"2024-01-11T09:46:25.951401+0000","last_became_peered":"2024-01-11T09:46:25.951401+0000","last_unstale":"2024-01-11T09:46:38.994342+0000","last_undegraded":"2024-01-11T09:46:38.994342+0000","last_fullsized":"2024-01-11T09:46:38.994342+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:29:18.608960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"235'1","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990258+0000","last_change":"2024-01-11T09:46:25.236787+0000","last_active":"2024-01-11T09:46:38.990258+0000","last_peered":"2024-01-11T09:46:38.990258+0000","last_clean":"2024-01-11T09:46:38.990258+0000","last_became_active":"2024-01-11T09:46:25.236503+0000","last_became_peered":"2024-01-11T09:46:25.236503+0000","last_unstale":"2024-01-11T09:46:38.990258+0000","last_undegraded":"2024-01-11T09:46:38.990258+0000","last_fullsized":"2024-01-11T09:46:38.990258+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:40:17.481276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.017366+0000","last_change":"2024-01-11T09:46:40.017366+0000","last_active":"2024-01-11T09:46:40.017366+0000","last_peered":"2024-01-11T09:46:40.017366+0000","last_clean":"2024-01-11T09:46:40.017366+0000","last_became_active":"2024-01-11T09:46:40.016774+0000","last_became_peered":"2024-01-11T09:46:40.016774+0000","last_unstale":"2024-01-11T09:46:40.017366+0000","last_undegraded":"2024-01-11T09:46:40.017366+0000","last_fullsized":"2024-01-11T09:46:40.017366+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:08:22.898981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.977786+0000","last_change":"2024-01-11T09:46:25.953624+0000","last_active":"2024-01-11T09:46:38.977786+0000","last_peered":"2024-01-11T09:46:38.977786+0000","last_clean":"2024-01-11T09:46:38.977786+0000","last_became_active":"2024-01-11T09:46:25.953347+0000","last_became_peered":"2024-01-11T09:46:25.953347+0000","last_unstale":"2024-01-11T09:46:38.977786+0000","last_undegraded":"2024-01-11T09:46:38.977786+0000","last_fullsized":"2024-01-11T09:46:38.977786+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:56:15.022301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.982895+0000","last_change":"2024-01-11T09:46:25.238938+0000","last_active":"2024-01-11T09:46:38.982895+0000","last_peered":"2024-01-11T09:46:38.982895+0000","last_clean":"2024-01-11T09:46:38.982895+0000","last_became_active":"2024-01-11T09:46:25.238214+0000","last_became_peered":"2024-01-11T09:46:25.238214+0000","last_unstale":"2024-01-11T09:46:38.982895+0000","last_undegraded":"2024-01-11T09:46:38.982895+0000","last_fullsized":"2024-01-11T09:46:38.982895+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:30:58.825981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.993325+0000","last_change":"2024-01-11T09:46:39.993325+0000","last_active":"2024-01-11T09:46:39.993325+0000","last_peered":"2024-01-11T09:46:39.993325+0000","last_clean":"2024-01-11T09:46:39.993325+0000","last_became_active":"2024-01-11T09:46:39.993033+0000","last_became_peered":"2024-01-11T09:46:39.993033+0000","last_unstale":"2024-01-11T09:46:39.993325+0000","last_undegraded":"2024-01-11T09:46:39.993325+0000","last_fullsized":"2024-01-11T09:46:39.993325+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:06:12.594569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.978097+0000","last_change":"2024-01-11T09:46:25.954867+0000","last_active":"2024-01-11T09:46:38.978097+0000","last_peered":"2024-01-11T09:46:38.978097+0000","last_clean":"2024-01-11T09:46:38.978097+0000","last_became_active":"2024-01-11T09:46:25.954613+0000","last_became_peered":"2024-01-11T09:46:25.954613+0000","last_unstale":"2024-01-11T09:46:38.978097+0000","last_undegraded":"2024-01-11T09:46:38.978097+0000","last_fullsized":"2024-01-11T09:46:38.978097+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:11:07.968385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.978067+0000","last_change":"2024-01-11T09:46:25.244239+0000","last_active":"2024-01-11T09:46:38.978067+0000","last_peered":"2024-01-11T09:46:38.978067+0000","last_clean":"2024-01-11T09:46:38.978067+0000","last_became_active":"2024-01-11T09:46:25.243663+0000","last_became_peered":"2024-01-11T09:46:25.243663+0000","last_unstale":"2024-01-11T09:46:38.978067+0000","last_undegraded":"2024-01-11T09:46:38.978067+0000","last_fullsized":"2024-01-11T09:46:38.978067+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:23:51.951171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.997383+0000","last_change":"2024-01-11T09:46:39.997383+0000","last_active":"2024-01-11T09:46:39.997383+0000","last_peered":"2024-01-11T09:46:39.997383+0000","last_clean":"2024-01-11T09:46:39.997383+0000","last_became_active":"2024-01-11T09:46:39.995199+0000","last_became_peered":"2024-01-11T09:46:39.995199+0000","last_unstale":"2024-01-11T09:46:39.997383+0000","last_undegraded":"2024-01-11T09:46:39.997383+0000","last_fullsized":"2024-01-11T09:46:39.997383+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:59:28.758593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984067+0000","last_change":"2024-01-11T09:46:25.962471+0000","last_active":"2024-01-11T09:46:38.984067+0000","last_peered":"2024-01-11T09:46:38.984067+0000","last_clean":"2024-01-11T09:46:38.984067+0000","last_became_active":"2024-01-11T09:46:25.959247+0000","last_became_peered":"2024-01-11T09:46:25.959247+0000","last_unstale":"2024-01-11T09:46:38.984067+0000","last_undegraded":"2024-01-11T09:46:38.984067+0000","last_fullsized":"2024-01-11T09:46:38.984067+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:31:56.044420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980624+0000","last_change":"2024-01-11T09:46:25.242486+0000","last_active":"2024-01-11T09:46:38.980624+0000","last_peered":"2024-01-11T09:46:38.980624+0000","last_clean":"2024-01-11T09:46:38.980624+0000","last_became_active":"2024-01-11T09:46:25.242231+0000","last_became_peered":"2024-01-11T09:46:25.242231+0000","last_unstale":"2024-01-11T09:46:38.980624+0000","last_undegraded":"2024-01-11T09:46:38.980624+0000","last_fullsized":"2024-01-11T09:46:38.980624+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:23:51.951171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.020863+0000","last_change":"2024-01-11T09:46:40.020863+0000","last_active":"2024-01-11T09:46:40.020863+0000","last_peered":"2024-01-11T09:46:40.020863+0000","last_clean":"2024-01-11T09:46:40.020863+0000","last_became_active":"2024-01-11T09:46:40.020559+0000","last_became_peered":"2024-01-11T09:46:40.020559+0000","last_unstale":"2024-01-11T09:46:40.020863+0000","last_undegraded":"2024-01-11T09:46:40.020863+0000","last_fullsized":"2024-01-11T09:46:40.020863+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:23.876325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986674+0000","last_change":"2024-01-11T09:46:25.954929+0000","last_active":"2024-01-11T09:46:38.986674+0000","last_peered":"2024-01-11T09:46:38.986674+0000","last_clean":"2024-01-11T09:46:38.986674+0000","last_became_active":"2024-01-11T09:46:25.951640+0000","last_became_peered":"2024-01-11T09:46:25.951640+0000","last_unstale":"2024-01-11T09:46:38.986674+0000","last_undegraded":"2024-01-11T09:46:38.986674+0000","last_fullsized":"2024-01-11T09:46:38.986674+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:28:55.053850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.991096+0000","last_change":"2024-01-11T09:46:25.234604+0000","last_active":"2024-01-11T09:46:38.991096+0000","last_peered":"2024-01-11T09:46:38.991096+0000","last_clean":"2024-01-11T09:46:38.991096+0000","last_became_active":"2024-01-11T09:46:25.234348+0000","last_became_peered":"2024-01-11T09:46:25.234348+0000","last_unstale":"2024-01-11T09:46:38.991096+0000","last_undegraded":"2024-01-11T09:46:38.991096+0000","last_fullsized":"2024-01-11T09:46:38.991096+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:01:05.259333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.999682+0000","last_change":"2024-01-11T09:46:39.999682+0000","last_active":"2024-01-11T09:46:39.999682+0000","last_peered":"2024-01-11T09:46:39.999682+0000","last_clean":"2024-01-11T09:46:39.999682+0000","last_became_active":"2024-01-11T09:46:39.999423+0000","last_became_peered":"2024-01-11T09:46:39.999423+0000","last_unstale":"2024-01-11T09:46:39.999682+0000","last_undegraded":"2024-01-11T09:46:39.999682+0000","last_fullsized":"2024-01-11T09:46:39.999682+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:48:30.105547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986233+0000","last_change":"2024-01-11T09:46:25.956295+0000","last_active":"2024-01-11T09:46:38.986233+0000","last_peered":"2024-01-11T09:46:38.986233+0000","last_clean":"2024-01-11T09:46:38.986233+0000","last_became_active":"2024-01-11T09:46:25.949169+0000","last_became_peered":"2024-01-11T09:46:25.949169+0000","last_unstale":"2024-01-11T09:46:38.986233+0000","last_undegraded":"2024-01-11T09:46:38.986233+0000","last_fullsized":"2024-01-11T09:46:38.986233+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:07.160563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980864+0000","last_change":"2024-01-11T09:46:25.231375+0000","last_active":"2024-01-11T09:46:38.980864+0000","last_peered":"2024-01-11T09:46:38.980864+0000","last_clean":"2024-01-11T09:46:38.980864+0000","last_became_active":"2024-01-11T09:46:25.231115+0000","last_became_peered":"2024-01-11T09:46:25.231115+0000","last_unstale":"2024-01-11T09:46:38.980864+0000","last_undegraded":"2024-01-11T09:46:38.980864+0000","last_fullsized":"2024-01-11T09:46:38.980864+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:06:11.124203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3f","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.994881+0000","last_change":"2024-01-11T09:46:39.994881+0000","last_active":"2024-01-11T09:46:39.994881+0000","last_peered":"2024-01-11T09:46:39.994881+0000","last_clean":"2024-01-11T09:46:39.994881+0000","last_became_active":"2024-01-11T09:46:39.993737+0000","last_became_peered":"2024-01-11T09:46:39.993737+0000","last_unstale":"2024-01-11T09:46:39.994881+0000","last_undegraded":"2024-01-11T09:46:39.994881+0000","last_fullsized":"2024-01-11T09:46:39.994881+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:48:18.089926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990906+0000","last_change":"2024-01-11T09:46:25.957641+0000","last_active":"2024-01-11T09:46:38.990906+0000","last_peered":"2024-01-11T09:46:38.990906+0000","last_clean":"2024-01-11T09:46:38.990906+0000","last_became_active":"2024-01-11T09:46:25.955768+0000","last_became_peered":"2024-01-11T09:46:25.955768+0000","last_unstale":"2024-01-11T09:46:38.990906+0000","last_undegraded":"2024-01-11T09:46:38.990906+0000","last_fullsized":"2024-01-11T09:46:38.990906+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:58.843659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.981087+0000","last_change":"2024-01-11T09:46:25.239061+0000","last_active":"2024-01-11T09:46:38.981087+0000","last_peered":"2024-01-11T09:46:38.981087+0000","last_clean":"2024-01-11T09:46:38.981087+0000","last_became_active":"2024-01-11T09:46:25.238802+0000","last_became_peered":"2024-01-11T09:46:25.238802+0000","last_unstale":"2024-01-11T09:46:38.981087+0000","last_undegraded":"2024-01-11T09:46:38.981087+0000","last_fullsized":"2024-01-11T09:46:38.981087+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:57.834513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.996552+0000","last_change":"2024-01-11T09:46:39.996552+0000","last_active":"2024-01-11T09:46:39.996552+0000","last_peered":"2024-01-11T09:46:39.996552+0000","last_clean":"2024-01-11T09:46:39.996552+0000","last_became_active":"2024-01-11T09:46:39.996063+0000","last_became_peered":"2024-01-11T09:46:39.996063+0000","last_unstale":"2024-01-11T09:46:39.996552+0000","last_undegraded":"2024-01-11T09:46:39.996552+0000","last_fullsized":"2024-01-11T09:46:39.996552+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:19:31.710935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990166+0000","last_change":"2024-01-11T09:46:25.948656+0000","last_active":"2024-01-11T09:46:38.990166+0000","last_peered":"2024-01-11T09:46:38.990166+0000","last_clean":"2024-01-11T09:46:38.990166+0000","last_became_active":"2024-01-11T09:46:25.948273+0000","last_became_peered":"2024-01-11T09:46:25.948273+0000","last_unstale":"2024-01-11T09:46:38.990166+0000","last_undegraded":"2024-01-11T09:46:38.990166+0000","last_fullsized":"2024-01-11T09:46:38.990166+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:36:07.163334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"234'4","reported_seq":17,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984460+0000","last_change":"2024-01-11T09:46:25.235720+0000","last_active":"2024-01-11T09:46:38.984460+0000","last_peered":"2024-01-11T09:46:38.984460+0000","last_clean":"2024-01-11T09:46:38.984460+0000","last_became_active":"2024-01-11T09:46:25.235445+0000","last_became_peered":"2024-01-11T09:46:25.235445+0000","last_unstale":"2024-01-11T09:46:38.984460+0000","last_undegraded":"2024-01-11T09:46:38.984460+0000","last_fullsized":"2024-01-11T09:46:38.984460+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:53:59.525452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"221'388","reported_seq":1298,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.979950+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:46:38.979950+0000","last_peered":"2024-01-11T09:46:38.979950+0000","last_clean":"2024-01-11T09:46:38.979950+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:46:38.979950+0000","last_undegraded":"2024-01-11T09:46:38.979950+0000","last_fullsized":"2024-01-11T09:46:38.979950+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":651,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":3519,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":651,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":388,"ondisk_log_size":388,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575984,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300788,"kb_used_data":3500,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071052,"statfs":{"total":96636764160,"available":96328757248,"internally_reserved":0,"allocated":3584000,"data_stored":2028665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.68899999999999995,"15min":0.72799999999999998},"min":{"1min":0.39100000000000001,"5min":0.376,"15min":0.36599999999999999},"max":{"1min":0.89200000000000002,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.71499999999999997,"15min":0.74299999999999999},"min":{"1min":0.40400000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83799999999999997,"5min":1.0609999999999999,"15min":1.0669999999999999},"last":0.66700000000000004}]},{"osd":1,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.69799999999999995,"15min":0.72499999999999998},"min":{"1min":0.48099999999999998,"5min":0.41799999999999998,"15min":0.41599999999999998},"max":{"1min":0.82099999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.70499999999999996,"15min":0.73599999999999999},"min":{"1min":0.504,"5min":0.434,"15min":0.434},"max":{"1min":0.89400000000000002,"5min":0.999,"15min":1.0820000000000001},"last":0.53100000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.71799999999999997,"15min":0.73499999999999999},"min":{"1min":0.46899999999999997,"5min":0.443,"15min":0.443},"max":{"1min":0.875,"5min":0.94399999999999995,"15min":1.0600000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.71099999999999997,"15min":0.747},"min":{"1min":0.48499999999999999,"5min":0.44800000000000001,"15min":0.44500000000000001},"max":{"1min":0.86099999999999999,"5min":0.96999999999999997,"15min":1.153},"last":0.623}]},{"osd":3,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73599999999999999,"15min":0.76200000000000001},"min":{"1min":0.51500000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":1.0660000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.72299999999999998,"15min":0.747},"min":{"1min":0.51000000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.77100000000000002,"5min":1,"15min":1.0880000000000001},"last":0.59299999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61099999999999999,"15min":0.60599999999999998},"min":{"1min":0.45900000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79600000000000004,"5min":1.0860000000000001,"15min":1.107},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64000000000000001,"15min":0.63900000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39700000000000002},"max":{"1min":0.871,"5min":0.995,"15min":1.0960000000000001},"last":0.75}]},{"osd":5,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.622,"15min":0.64100000000000001},"min":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86099999999999999,"5min":0.86699999999999999,"15min":1.0109999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.69199999999999995,"15min":0.68799999999999994},"min":{"1min":0.41499999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.61099999999999999}]},{"osd":6,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.78000000000000003},"min":{"1min":0.56000000000000005,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.96899999999999997,"5min":1.232,"15min":1.232},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.002,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.78400000000000003}]}]},{"osd":6,"up_from":13,"seq":55834575983,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312500,"kb_used_data":2924,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059340,"statfs":{"total":96636764160,"available":96316764160,"internally_reserved":0,"allocated":2994176,"data_stored":1438803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.65200000000000002,"15min":0.70399999999999996},"min":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89100000000000001,"5min":0.94199999999999995,"15min":0.999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.71299999999999997,"15min":0.73899999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.216},"last":0.65600000000000003}]},{"osd":1,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.67500000000000004,"15min":0.73199999999999998},"min":{"1min":0.44900000000000001,"5min":0.42799999999999999,"15min":0.41699999999999998},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":1.3129999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.68899999999999995,"15min":0.72499999999999998},"min":{"1min":0.44400000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.95499999999999996,"5min":0.98299999999999998,"15min":1.0840000000000001},"last":0.51900000000000002}]},{"osd":2,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.70699999999999996,"15min":0.74299999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.0069999999999999,"5min":1.034,"15min":1.4179999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.72099999999999997,"15min":0.75},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":0.46500000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.72999999999999998,"15min":0.754},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.92600000000000005,"5min":1.012,"15min":1.014},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.70599999999999996,"15min":0.76200000000000001},"min":{"1min":0.498,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.91300000000000003,"5min":0.95999999999999996,"15min":1.165},"last":0.64300000000000002}]},{"osd":4,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.38400000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78000000000000003,"5min":0.91700000000000004,"15min":1.0309999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.626,"15min":0.61599999999999999},"min":{"1min":0.40799999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92100000000000004,"5min":0.94799999999999995,"15min":1.1259999999999999},"last":0.56299999999999994}]},{"osd":5,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.65000000000000002,"15min":0.65900000000000003},"min":{"1min":0.44600000000000001,"5min":0.41499999999999998,"15min":0.40000000000000002},"max":{"1min":0.93100000000000005,"5min":0.97899999999999998,"15min":1.369},"last":0.441},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70299999999999996,"15min":0.70499999999999996},"min":{"1min":0.56799999999999995,"5min":0.46700000000000003,"15min":0.45400000000000001},"max":{"1min":0.85999999999999999,"5min":0.999,"15min":1.0960000000000001},"last":0.58499999999999996}]},{"osd":7,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.70699999999999996,"15min":0.70899999999999996},"min":{"1min":0.47199999999999998,"5min":0.45600000000000002,"15min":0.45100000000000001},"max":{"1min":0.91300000000000003,"5min":1.089,"15min":1.484},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.754,"15min":0.76700000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.98299999999999998,"5min":1.1379999999999999,"15min":1.603},"last":0.68600000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575985,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308416,"kb_used_data":2936,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063424,"statfs":{"total":96636764160,"available":96320946176,"internally_reserved":0,"allocated":3006464,"data_stored":1441100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.51400000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.315},"max":{"1min":0.69799999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.59899999999999998,"15min":0.61899999999999999},"min":{"1min":0.41899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.75600000000000001,"5min":0.85199999999999998,"15min":0.95099999999999996},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.50700000000000001,"15min":0.51200000000000001},"min":{"1min":0.34999999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66600000000000004,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.84099999999999997,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.38400000000000001}]},{"osd":3,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.46400000000000002,"5min":0.436,"15min":0.42699999999999999},"max":{"1min":0.85199999999999998,"5min":0.91600000000000004,"15min":0.96699999999999997},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.58099999999999996,"15min":0.58199999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.71299999999999997,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.48399999999999999}]},{"osd":4,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.79500000000000004,"15min":0.81599999999999995},"min":{"1min":0.54600000000000004,"5min":0.51400000000000001,"15min":0.505},"max":{"1min":0.875,"5min":1.214,"15min":1.2969999999999999},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.84399999999999997,"15min":0.85899999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1000000000000001,"5min":1.1879999999999999,"15min":1.1890000000000001},"last":0.81599999999999995}]},{"osd":5,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.82599999999999996,"15min":0.82799999999999996},"min":{"1min":0.51800000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.996,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.80800000000000005,"15min":0.80700000000000005},"min":{"1min":0.61899999999999999,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":1.0609999999999999,"5min":1.341,"15min":1.341},"last":0.746}]},{"osd":6,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.83499999999999996,"15min":0.84699999999999998},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.56799999999999995},"max":{"1min":1.0620000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.84499999999999997,"15min":0.85399999999999998},"min":{"1min":0.60099999999999998,"5min":0.56899999999999995,"15min":0.55900000000000005},"max":{"1min":1.077,"5min":1.177,"15min":1.177},"last":0.94199999999999995}]},{"osd":7,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.88800000000000001,"15min":0.88900000000000001},"min":{"1min":0.64400000000000002,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.1359999999999999,"5min":1.421,"15min":1.421},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.86299999999999999,"15min":0.85699999999999998},"min":{"1min":0.64500000000000002,"5min":0.60099999999999998,"15min":0.54500000000000004},"max":{"1min":0.96099999999999997,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.84699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575984,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308980,"kb_used_data":3500,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062860,"statfs":{"total":96636764160,"available":96320368640,"internally_reserved":0,"allocated":3584000,"data_stored":2028629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.50700000000000001,"15min":0.51800000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.70999999999999996,"5min":0.84699999999999998,"15min":0.875},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.58399999999999996,"15min":0.59599999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77200000000000002,"5min":0.81599999999999995,"15min":0.874},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.53500000000000003,"15min":0.56499999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.61499999999999999,"5min":0.83099999999999996,"15min":0.97499999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.56299999999999994,"15min":0.57999999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71899999999999997,"5min":0.84699999999999998,"15min":0.86399999999999999},"last":0.42999999999999999}]},{"osd":3,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.51400000000000001,"15min":0.54000000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.73699999999999999,"5min":0.79000000000000004,"15min":0.95399999999999996},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.621,"15min":0.64800000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80200000000000005,"5min":0.90300000000000002,"15min":1.0389999999999999},"last":0.55600000000000005}]},{"osd":4,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.81399999999999995,"15min":0.81599999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0629999999999999,"5min":1.1779999999999999,"15min":1.304},"last":0.746},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.80000000000000004,"15min":0.80600000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.93000000000000005,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.82999999999999996}]},{"osd":5,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.82299999999999995,"15min":0.83899999999999997},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0069999999999999,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.82899999999999996,"15min":0.84199999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.1100000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.89000000000000001}]},{"osd":6,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.81000000000000005,"15min":0.82399999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.1000000000000001,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.872},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.84199999999999997,"15min":0.86599999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":0.95899999999999996,"5min":1.256,"15min":1.256},"last":0.79000000000000004}]},{"osd":7,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.85599999999999998,"15min":0.86599999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.151,"5min":1.28,"15min":1.28},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.85899999999999999,"15min":0.872},"min":{"1min":0.57499999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.054,"5min":1.238,"15min":1.2410000000000001},"last":0.91500000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575983,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300208,"kb_used_data":2920,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071632,"statfs":{"total":96636764160,"available":96329351168,"internally_reserved":0,"allocated":2990080,"data_stored":1438317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48699999999999999,"15min":0.502},"min":{"1min":0.32800000000000001,"5min":0.29899999999999999,"15min":0.27900000000000003},"max":{"1min":0.625,"5min":0.72599999999999998,"15min":0.85399999999999998},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.56499999999999995},"min":{"1min":0.38800000000000001,"5min":0.34399999999999997,"15min":0.34300000000000003},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":1.6359999999999999},"last":0.48999999999999999}]},{"osd":1,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.39500000000000002,"5min":0.38200000000000001,"15min":0.33500000000000002},"max":{"1min":0.80900000000000005,"5min":0.88800000000000001,"15min":0.93000000000000005},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58899999999999997,"15min":0.59899999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.73099999999999998,"5min":0.97299999999999998,"15min":1.145},"last":0.41099999999999998}]},{"osd":3,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.75,"5min":0.83099999999999996,"15min":0.96399999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.63,"15min":0.65500000000000003},"min":{"1min":0.442,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.83599999999999997,"5min":0.90700000000000003,"15min":1.3620000000000001},"last":0.46800000000000003}]},{"osd":4,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.78200000000000003,"15min":0.79800000000000004},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.54300000000000004},"max":{"1min":0.95299999999999996,"5min":1.071,"15min":1.1519999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.78700000000000003,"15min":0.80000000000000004},"min":{"1min":0.54500000000000004,"5min":0.505,"15min":0.505},"max":{"1min":0.96299999999999997,"5min":1.161,"15min":1.161},"last":0.79700000000000004}]},{"osd":5,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.83099999999999996,"15min":0.86799999999999999},"min":{"1min":0.625,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.996,"5min":1.228,"15min":1.429},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.85099999999999998,"15min":0.86299999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.0389999999999999,"5min":1.1850000000000001,"15min":1.2170000000000001},"last":0.64900000000000002}]},{"osd":6,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.80000000000000004,"15min":0.82599999999999996},"min":{"1min":0.55500000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0049999999999999,"5min":1.1659999999999999,"15min":1.4139999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.83499999999999996,"15min":0.86399999999999999},"min":{"1min":0.63400000000000001,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.96599999999999997,"5min":1.2010000000000001,"15min":1.593},"last":0.76400000000000001}]},{"osd":7,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.86799999999999999,"15min":0.89200000000000002},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.57299999999999995},"max":{"1min":1.0129999999999999,"5min":1.137,"15min":2.0539999999999998},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.86499999999999999,"15min":0.88100000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.55500000000000005},"max":{"1min":1.069,"5min":1.2270000000000001,"15min":1.2490000000000001},"last":0.82699999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575983,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300208,"kb_used_data":2920,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071632,"statfs":{"total":96636764160,"available":96329351168,"internally_reserved":0,"allocated":2990080,"data_stored":1438781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.502,"15min":0.52000000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.314},"max":{"1min":0.76100000000000001,"5min":0.89700000000000002,"15min":0.90000000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.56200000000000006,"15min":0.58699999999999997},"min":{"1min":0.372,"5min":0.36899999999999999,"15min":0.36099999999999999},"max":{"1min":0.67900000000000005,"5min":1.014,"15min":1.0449999999999999},"last":0.47799999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.495,"15min":0.51100000000000001},"min":{"1min":0.29399999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63500000000000001,"5min":0.84199999999999997,"15min":1.304},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.38700000000000001},"max":{"1min":0.75,"5min":0.84099999999999997,"15min":1.4590000000000001},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58299999999999996,"15min":0.60599999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.72999999999999998,"5min":0.93700000000000006,"15min":1.04},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.64200000000000002,"15min":0.65800000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44700000000000001},"max":{"1min":0.83499999999999996,"5min":0.89800000000000002,"15min":0.92000000000000004},"last":0.45200000000000001}]},{"osd":4,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76100000000000001,"15min":0.79000000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.92900000000000005,"5min":1.206,"15min":1.206},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.77800000000000002,"15min":0.81299999999999994},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.871,"5min":1.171,"15min":1.1830000000000001},"last":0.76300000000000001}]},{"osd":5,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.79600000000000004,"15min":0.82099999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.95399999999999996,"5min":1.1619999999999999,"15min":1.234},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81000000000000005,"15min":0.84799999999999998},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.3440000000000001},"last":0.77500000000000002}]},{"osd":6,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82999999999999996,"15min":0.85899999999999999},"min":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.077,"5min":1.1739999999999999,"15min":1.2170000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.80500000000000005,"15min":0.82899999999999996},"min":{"1min":0.55300000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.97099999999999997,"5min":1.181,"15min":1.2190000000000001},"last":0.78800000000000003}]},{"osd":7,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.83699999999999997,"15min":0.86799999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0269999999999999,"5min":1.224,"15min":1.264},"last":0.749},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.86099999999999999,"15min":0.89500000000000002},"min":{"1min":0.59299999999999997,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.0209999999999999,"5min":1.1539999999999999,"15min":1.24},"last":0.66700000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575984,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308416,"kb_used_data":2936,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063424,"statfs":{"total":96636764160,"available":96320946176,"internally_reserved":0,"allocated":3006464,"data_stored":1441068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.72299999999999998,"15min":0.76000000000000001},"min":{"1min":0.42099999999999999,"5min":0.41199999999999998,"15min":0.39900000000000002},"max":{"1min":0.85799999999999998,"5min":0.97799999999999998,"15min":1.119},"last":0.747},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.73799999999999999,"15min":0.76700000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86399999999999999,"5min":1.853,"15min":1.853},"last":0.72799999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.69899999999999995,"15min":0.73499999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.83599999999999997,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.69799999999999995,"15min":0.72699999999999998},"min":{"1min":0.438,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81399999999999995,"5min":1.639,"15min":1.639},"last":0.84299999999999997}]},{"osd":2,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.71999999999999997,"15min":0.746},"min":{"1min":0.50600000000000001,"5min":0.46500000000000002,"15min":0.44700000000000001},"max":{"1min":0.76400000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.71999999999999997,"15min":0.74199999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.93300000000000005,"5min":1.0249999999999999,"15min":1.0700000000000001},"last":0.78200000000000003}]},{"osd":3,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.70199999999999996,"15min":0.73299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.75900000000000001,"15min":0.77300000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.97499999999999998,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.82699999999999996}]},{"osd":5,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63500000000000001,"15min":0.623},"min":{"1min":0.41999999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.89900000000000002,"5min":0.91500000000000004,"15min":0.97199999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.66700000000000004,"15min":0.64700000000000002},"min":{"1min":0.45900000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95099999999999996,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.53300000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.68999999999999995,"15min":0.67300000000000004},"min":{"1min":0.441,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.012,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.71899999999999997,"15min":0.70499999999999996},"min":{"1min":0.46000000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0609999999999999,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.81100000000000005}]},{"osd":7,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.72899999999999998,"15min":0.70199999999999996},"min":{"1min":0.495,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.075,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.77900000000000003,"15min":0.754},"min":{"1min":0.56699999999999995,"5min":0.436,"15min":0.41699999999999998},"max":{"1min":1.0409999999999999,"5min":2.052,"15min":2.052},"last":0.86899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575984,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312492,"kb_used_data":2916,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059348,"statfs":{"total":96636764160,"available":96316772352,"internally_reserved":0,"allocated":2985984,"data_stored":1438227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.71699999999999997,"15min":0.72899999999999998},"min":{"1min":0.46800000000000003,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71099999999999997,"15min":0.72399999999999998},"min":{"1min":0.51000000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.66000000000000003}]},{"osd":1,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71499999999999997,"15min":0.73399999999999999},"min":{"1min":0.46800000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91400000000000003,"5min":0.95699999999999996,"15min":0.97699999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.73299999999999998,"15min":0.73799999999999999},"min":{"1min":0.51900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.874,"5min":1.004,"15min":1.0269999999999999},"last":0.63700000000000001}]},{"osd":2,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.72299999999999998,"15min":0.72999999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1,"5min":1.0269999999999999,"15min":1.028},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.73199999999999998,"15min":0.74199999999999999},"min":{"1min":0.50900000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.98799999999999999},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.74399999999999999,"15min":0.75600000000000001},"min":{"1min":0.56200000000000006,"5min":0.52800000000000002,"15min":0.49299999999999999},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76100000000000001,"15min":0.77000000000000002},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.63300000000000001,"15min":0.61399999999999999},"min":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41699999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.64300000000000002,"15min":0.63400000000000001},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.375},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.44}]},{"osd":6,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63700000000000001,"15min":0.64300000000000002},"min":{"1min":0.433,"5min":0.42099999999999999,"15min":0.40999999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":1.0089999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69199999999999995,"15min":0.68600000000000005},"min":{"1min":0.48699999999999999,"5min":0.46200000000000002,"15min":0.42799999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.38500000000000001}]},{"osd":7,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.73199999999999998,"15min":0.73299999999999998},"min":{"1min":0.58299999999999996,"5min":0.47299999999999998,"15min":0.434},"max":{"1min":0.94799999999999995,"5min":1.0149999999999999,"15min":1.0760000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.77800000000000002,"15min":0.77600000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.47599999999999998},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.623}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:46:44.418 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:46:44.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.657+0000 7effad5ff700 1 Processor -- start 2024-01-11T09:46:44.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.657+0000 7effad5ff700 1 -- start start 2024-01-11T09:46:44.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.658+0000 7effad5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7effa80b03e0 0x7effa80b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:44.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.659+0000 7effad5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa80cd730 con 0x7effa80b07b0 2024-01-11T09:46:44.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.659+0000 7effad5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa80b2de0 con 0x7effa80b29c0 2024-01-11T09:46:44.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.659+0000 7effad5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa80ccb20 con 0x7effa80b03e0 2024-01-11T09:46:44.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.659+0000 7effa6ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7effa80b03e0 0x7effa80b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:44.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.659+0000 7effa67fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7effa80b07b0 0x7effa8159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35100/0 (socket says 172.21.15.27:35100) 2024-01-11T09:46:44.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.659+0000 7effa67fc700 1 -- 172.21.15.27:0/3148058755 learned_addr learned my addr 172.21.15.27:0/3148058755 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:44.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.659+0000 7effa6ffd700 1 -- 172.21.15.27:0/3148058755 >> v1:172.21.15.77:6789/0 conn(0x7effa80b29c0 legacy=0x7effa80aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:44.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.659+0000 7effa6ffd700 1 -- 172.21.15.27:0/3148058755 >> v1:172.21.15.27:6789/0 conn(0x7effa80b07b0 legacy=0x7effa8159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:44.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.659+0000 7effa6ffd700 1 -- 172.21.15.27:0/3148058755 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa80cce00 con 0x7effa80b03e0 2024-01-11T09:46:44.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.659+0000 7effa5ffb700 1 -- 172.21.15.27:0/3148058755 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455597197 0 0) 0x7effa80cd730 con 0x7effa80b07b0 2024-01-11T09:46:44.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.660+0000 7effa6ffd700 1 --2- 172.21.15.27:0/3148058755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effa80b03e0 0x7effa80b2480 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7eff980098e0 tx=0x7eff980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2c716900e34e0b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:44.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.660+0000 7effa5ffb700 1 -- 172.21.15.27:0/3148058755 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff98009e80 con 0x7effa80b03e0 2024-01-11T09:46:44.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.660+0000 7effa5ffb700 1 -- 172.21.15.27:0/3148058755 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9800b860 con 0x7effa80b03e0 2024-01-11T09:46:44.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.660+0000 7effa5ffb700 1 -- 172.21.15.27:0/3148058755 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9800bbe0 con 0x7effa80b03e0 2024-01-11T09:46:44.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.660+0000 7effad5ff700 1 -- 172.21.15.27:0/3148058755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effa80b03e0 msgr2=0x7effa80b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:44.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.660+0000 7effad5ff700 1 --2- 172.21.15.27:0/3148058755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effa80b03e0 0x7effa80b2480 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7eff980098e0 tx=0x7eff980091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:44.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.661+0000 7effad5ff700 1 -- 172.21.15.27:0/3148058755 shutdown_connections 2024-01-11T09:46:44.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.661+0000 7effad5ff700 1 --2- 172.21.15.27:0/3148058755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effa80b03e0 0x7effa80b2480 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:44.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.661+0000 7effad5ff700 1 -- 172.21.15.27:0/3148058755 >> 172.21.15.27:0/3148058755 conn(0x7effa806ed40 msgr2=0x7effa806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:44.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.661+0000 7effad5ff700 1 -- 172.21.15.27:0/3148058755 shutdown_connections 2024-01-11T09:46:44.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.661+0000 7effad5ff700 1 -- 172.21.15.27:0/3148058755 wait complete. 2024-01-11T09:46:44.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.662+0000 7effad5ff700 1 Processor -- start 2024-01-11T09:46:44.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.662+0000 7effad5ff700 1 -- start start 2024-01-11T09:46:44.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.663+0000 7effad5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effa80b03e0 0x7effa815bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:44.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.663+0000 7effad5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effa80b07b0 0x7effa80a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:44.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.663+0000 7effad5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7effa80b29c0 0x7effa816a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:44.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.663+0000 7effad5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa80c4ae0 con 0x7effa80b03e0 2024-01-11T09:46:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.663+0000 7effad5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa815c140 con 0x7effa80b07b0 2024-01-11T09:46:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.663+0000 7effad5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa80c3ed0 con 0x7effa80b29c0 2024-01-11T09:46:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.663+0000 7effa6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effa80b03e0 0x7effa815bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.663+0000 7effa6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effa80b03e0 0x7effa815bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52558/0 (socket says 172.21.15.27:52558) 2024-01-11T09:46:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.663+0000 7effa6ffd700 1 -- 172.21.15.27:0/1703827073 learned_addr learned my addr 172.21.15.27:0/1703827073 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.663+0000 7effa67fc700 1 --2- 172.21.15.27:0/1703827073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effa80b07b0 0x7effa80a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.664+0000 7effa77fe700 1 --2- 172.21.15.27:0/1703827073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7effa80b29c0 0x7effa816a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.664+0000 7effa6ffd700 1 -- 172.21.15.27:0/1703827073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7effa80b29c0 msgr2=0x7effa816a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:44.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.664+0000 7effa6ffd700 1 --2- 172.21.15.27:0/1703827073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7effa80b29c0 0x7effa816a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:44.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.664+0000 7effa6ffd700 1 -- 172.21.15.27:0/1703827073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effa80b07b0 msgr2=0x7effa80a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:44.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.664+0000 7effa6ffd700 1 --2- 172.21.15.27:0/1703827073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effa80b07b0 0x7effa80a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:44.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.664+0000 7effa6ffd700 1 -- 172.21.15.27:0/1703827073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff98009420 con 0x7effa80b03e0 2024-01-11T09:46:44.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.664+0000 7effa6ffd700 1 --2- 172.21.15.27:0/1703827073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effa80b03e0 0x7effa815bc00 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7effa8069540 tx=0x7eff98017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:44.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.664+0000 7eff8effd700 1 -- 172.21.15.27:0/1703827073 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9801f070 con 0x7effa80b03e0 2024-01-11T09:46:44.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.665+0000 7eff8effd700 1 -- 172.21.15.27:0/1703827073 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff98000c90 con 0x7effa80b03e0 2024-01-11T09:46:44.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.665+0000 7effad5ff700 1 -- 172.21.15.27:0/1703827073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa80c41b0 con 0x7effa80b03e0 2024-01-11T09:46:44.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.665+0000 7eff8effd700 1 -- 172.21.15.27:0/1703827073 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff98025420 con 0x7effa80b03e0 2024-01-11T09:46:44.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.665+0000 7effad5ff700 1 -- 172.21.15.27:0/1703827073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa80c4770 con 0x7effa80b03e0 2024-01-11T09:46:44.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.666+0000 7effad5ff700 1 -- 172.21.15.27:0/1703827073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa80718d0 con 0x7effa80b03e0 2024-01-11T09:46:44.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.667+0000 7eff8effd700 1 -- 172.21.15.27:0/1703827073 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7eff98023070 con 0x7effa80b03e0 2024-01-11T09:46:44.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.667+0000 7eff8effd700 1 --2- 172.21.15.27:0/1703827073 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7eff94067170 0x7eff94069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:44.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.668+0000 7eff8effd700 1 -- 172.21.15.27:0/1703827073 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6681+0+0 (secure 0 0 0) 0x7eff98086130 con 0x7effa80b03e0 2024-01-11T09:46:44.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.668+0000 7effa67fc700 1 --2- 172.21.15.27:0/1703827073 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7eff94067170 0x7eff94069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:44.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.669+0000 7effa67fc700 1 --2- 172.21.15.27:0/1703827073 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7eff94067170 0x7eff94069620 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7effa80c3e80 tx=0x7eff9c007d10 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:44.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.672+0000 7eff8effd700 1 -- 172.21.15.27:0/1703827073 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7eff98052d40 con 0x7effa80b03e0 2024-01-11T09:46:44.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.870+0000 7effad5ff700 1 -- 172.21.15.27:0/1703827073 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7effa80c4c20 con 0x7eff94067170 2024-01-11T09:46:44.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.885+0000 7eff8effd700 1 -- 172.21.15.27:0/1703827073 <== mgr.9107 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528629 (secure 0 0 0) 0x7eff9c001400 con 0x7eff94067170 2024-01-11T09:46:44.885 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:44.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.891+0000 7effad5ff700 1 -- 172.21.15.27:0/1703827073 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7eff94067170 msgr2=0x7eff94069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:44.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.891+0000 7effad5ff700 1 --2- 172.21.15.27:0/1703827073 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7eff94067170 0x7eff94069620 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7effa80c3e80 tx=0x7eff9c007d10 comp rx=0 tx=0).stop 2024-01-11T09:46:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.891+0000 7effad5ff700 1 -- 172.21.15.27:0/1703827073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effa80b03e0 msgr2=0x7effa815bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.891+0000 7effad5ff700 1 --2- 172.21.15.27:0/1703827073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effa80b03e0 0x7effa815bc00 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7effa8069540 tx=0x7eff98017000 comp rx=0 tx=0).stop 2024-01-11T09:46:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.891+0000 7effad5ff700 1 -- 172.21.15.27:0/1703827073 shutdown_connections 2024-01-11T09:46:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.891+0000 7effad5ff700 1 --2- 172.21.15.27:0/1703827073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7effa80b29c0 0x7effa816a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.891+0000 7effad5ff700 1 --2- 172.21.15.27:0/1703827073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effa80b03e0 0x7effa815bc00 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.891+0000 7effad5ff700 1 --2- 172.21.15.27:0/1703827073 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7eff94067170 0x7eff94069620 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.891+0000 7effad5ff700 1 --2- 172.21.15.27:0/1703827073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effa80b07b0 0x7effa80a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.891+0000 7effad5ff700 1 -- 172.21.15.27:0/1703827073 >> 172.21.15.27:0/1703827073 conn(0x7effa806ed40 msgr2=0x7effa80c95d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.891+0000 7effad5ff700 1 -- 172.21.15.27:0/1703827073 shutdown_connections 2024-01-11T09:46:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:44.891+0000 7effad5ff700 1 -- 172.21.15.27:0/1703827073 wait complete. 2024-01-11T09:46:44.891 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:46:44.902 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-01-11T09:46:44.161892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596193,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":742,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":467,"ondisk_log_size":467,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":247,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2452008,"kb_used_data":24552,"kb_used_omap":0,"kb_used_meta":2427392,"kb_avail":752522712,"statfs":{"total":773094113280,"available":770583257088,"internally_reserved":0,"allocated":25141248,"data_stored":12693590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485649408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3265,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001522"},"pg_stats":[{"pgid":"33.1","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.995461+0000","last_change":"2024-01-11T09:46:39.995461+0000","last_active":"2024-01-11T09:46:39.995461+0000","last_peered":"2024-01-11T09:46:39.995461+0000","last_clean":"2024-01-11T09:46:39.995461+0000","last_became_active":"2024-01-11T09:46:39.995177+0000","last_became_peered":"2024-01-11T09:46:39.995177+0000","last_unstale":"2024-01-11T09:46:39.995461+0000","last_undegraded":"2024-01-11T09:46:39.995461+0000","last_fullsized":"2024-01-11T09:46:39.995461+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:53:48.890166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.002322+0000","last_change":"2024-01-11T09:46:38.998858+0000","last_active":"2024-01-11T09:46:39.002322+0000","last_peered":"2024-01-11T09:46:39.002322+0000","last_clean":"2024-01-11T09:46:39.002322+0000","last_became_active":"2024-01-11T09:46:38.998596+0000","last_became_peered":"2024-01-11T09:46:38.998596+0000","last_unstale":"2024-01-11T09:46:39.002322+0000","last_undegraded":"2024-01-11T09:46:39.002322+0000","last_fullsized":"2024-01-11T09:46:39.002322+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:29:24.327601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3f","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.992628+0000","last_change":"2024-01-11T09:46:25.952190+0000","last_active":"2024-01-11T09:46:38.992628+0000","last_peered":"2024-01-11T09:46:38.992628+0000","last_clean":"2024-01-11T09:46:38.992628+0000","last_became_active":"2024-01-11T09:46:25.951733+0000","last_became_peered":"2024-01-11T09:46:25.951733+0000","last_unstale":"2024-01-11T09:46:38.992628+0000","last_undegraded":"2024-01-11T09:46:38.992628+0000","last_fullsized":"2024-01-11T09:46:38.992628+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:59:15.007682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.999859+0000","last_change":"2024-01-11T09:46:39.999859+0000","last_active":"2024-01-11T09:46:39.999859+0000","last_peered":"2024-01-11T09:46:39.999859+0000","last_clean":"2024-01-11T09:46:39.999859+0000","last_became_active":"2024-01-11T09:46:39.999334+0000","last_became_peered":"2024-01-11T09:46:39.999334+0000","last_unstale":"2024-01-11T09:46:39.999859+0000","last_undegraded":"2024-01-11T09:46:39.999859+0000","last_fullsized":"2024-01-11T09:46:39.999859+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:56:27.940894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.001251+0000","last_change":"2024-01-11T09:46:38.997477+0000","last_active":"2024-01-11T09:46:39.001251+0000","last_peered":"2024-01-11T09:46:39.001251+0000","last_clean":"2024-01-11T09:46:39.001251+0000","last_became_active":"2024-01-11T09:46:38.997202+0000","last_became_peered":"2024-01-11T09:46:38.997202+0000","last_unstale":"2024-01-11T09:46:39.001251+0000","last_undegraded":"2024-01-11T09:46:39.001251+0000","last_fullsized":"2024-01-11T09:46:39.001251+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:39:50.380011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990492+0000","last_change":"2024-01-11T09:46:25.957907+0000","last_active":"2024-01-11T09:46:38.990492+0000","last_peered":"2024-01-11T09:46:38.990492+0000","last_clean":"2024-01-11T09:46:38.990492+0000","last_became_active":"2024-01-11T09:46:25.956251+0000","last_became_peered":"2024-01-11T09:46:25.956251+0000","last_unstale":"2024-01-11T09:46:38.990492+0000","last_undegraded":"2024-01-11T09:46:38.990492+0000","last_fullsized":"2024-01-11T09:46:38.990492+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:53:35.139255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.995469+0000","last_change":"2024-01-11T09:46:39.995469+0000","last_active":"2024-01-11T09:46:39.995469+0000","last_peered":"2024-01-11T09:46:39.995469+0000","last_clean":"2024-01-11T09:46:39.995469+0000","last_became_active":"2024-01-11T09:46:39.995211+0000","last_became_peered":"2024-01-11T09:46:39.995211+0000","last_unstale":"2024-01-11T09:46:39.995469+0000","last_undegraded":"2024-01-11T09:46:39.995469+0000","last_fullsized":"2024-01-11T09:46:39.995469+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:11:21.719296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986698+0000","last_change":"2024-01-11T09:46:38.979782+0000","last_active":"2024-01-11T09:46:38.986698+0000","last_peered":"2024-01-11T09:46:38.986698+0000","last_clean":"2024-01-11T09:46:38.986698+0000","last_became_active":"2024-01-11T09:46:38.979495+0000","last_became_peered":"2024-01-11T09:46:38.979495+0000","last_unstale":"2024-01-11T09:46:38.986698+0000","last_undegraded":"2024-01-11T09:46:38.986698+0000","last_fullsized":"2024-01-11T09:46:38.986698+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:27:17.589227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.983955+0000","last_change":"2024-01-11T09:46:25.956076+0000","last_active":"2024-01-11T09:46:38.983955+0000","last_peered":"2024-01-11T09:46:38.983955+0000","last_clean":"2024-01-11T09:46:38.983955+0000","last_became_active":"2024-01-11T09:46:25.955819+0000","last_became_peered":"2024-01-11T09:46:25.955819+0000","last_unstale":"2024-01-11T09:46:38.983955+0000","last_undegraded":"2024-01-11T09:46:38.983955+0000","last_fullsized":"2024-01-11T09:46:38.983955+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:27:34.516236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.997183+0000","last_change":"2024-01-11T09:46:39.997183+0000","last_active":"2024-01-11T09:46:39.997183+0000","last_peered":"2024-01-11T09:46:39.997183+0000","last_clean":"2024-01-11T09:46:39.997183+0000","last_became_active":"2024-01-11T09:46:39.996739+0000","last_became_peered":"2024-01-11T09:46:39.996739+0000","last_unstale":"2024-01-11T09:46:39.997183+0000","last_undegraded":"2024-01-11T09:46:39.997183+0000","last_fullsized":"2024-01-11T09:46:39.997183+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:43:01.107206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.999835+0000","last_change":"2024-01-11T09:46:38.995869+0000","last_active":"2024-01-11T09:46:38.999835+0000","last_peered":"2024-01-11T09:46:38.999835+0000","last_clean":"2024-01-11T09:46:38.999835+0000","last_became_active":"2024-01-11T09:46:38.995586+0000","last_became_peered":"2024-01-11T09:46:38.995586+0000","last_unstale":"2024-01-11T09:46:38.999835+0000","last_undegraded":"2024-01-11T09:46:38.999835+0000","last_fullsized":"2024-01-11T09:46:38.999835+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:47:57.662961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984870+0000","last_change":"2024-01-11T09:46:25.947296+0000","last_active":"2024-01-11T09:46:38.984870+0000","last_peered":"2024-01-11T09:46:38.984870+0000","last_clean":"2024-01-11T09:46:38.984870+0000","last_became_active":"2024-01-11T09:46:25.946057+0000","last_became_peered":"2024-01-11T09:46:25.946057+0000","last_unstale":"2024-01-11T09:46:38.984870+0000","last_undegraded":"2024-01-11T09:46:38.984870+0000","last_fullsized":"2024-01-11T09:46:38.984870+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:15:19.562588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.015760+0000","last_change":"2024-01-11T09:46:40.015760+0000","last_active":"2024-01-11T09:46:40.015760+0000","last_peered":"2024-01-11T09:46:40.015760+0000","last_clean":"2024-01-11T09:46:40.015760+0000","last_became_active":"2024-01-11T09:46:40.015414+0000","last_became_peered":"2024-01-11T09:46:40.015414+0000","last_unstale":"2024-01-11T09:46:40.015760+0000","last_undegraded":"2024-01-11T09:46:40.015760+0000","last_fullsized":"2024-01-11T09:46:40.015760+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:27:00.735104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.002209+0000","last_change":"2024-01-11T09:46:38.998098+0000","last_active":"2024-01-11T09:46:39.002209+0000","last_peered":"2024-01-11T09:46:39.002209+0000","last_clean":"2024-01-11T09:46:39.002209+0000","last_became_active":"2024-01-11T09:46:38.997822+0000","last_became_peered":"2024-01-11T09:46:38.997822+0000","last_unstale":"2024-01-11T09:46:39.002209+0000","last_undegraded":"2024-01-11T09:46:39.002209+0000","last_fullsized":"2024-01-11T09:46:39.002209+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:51:50.752675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.989215+0000","last_change":"2024-01-11T09:46:25.953859+0000","last_active":"2024-01-11T09:46:38.989215+0000","last_peered":"2024-01-11T09:46:38.989215+0000","last_clean":"2024-01-11T09:46:38.989215+0000","last_became_active":"2024-01-11T09:46:25.953566+0000","last_became_peered":"2024-01-11T09:46:25.953566+0000","last_unstale":"2024-01-11T09:46:38.989215+0000","last_undegraded":"2024-01-11T09:46:38.989215+0000","last_fullsized":"2024-01-11T09:46:38.989215+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:42:55.688728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.001149+0000","last_change":"2024-01-11T09:46:40.001149+0000","last_active":"2024-01-11T09:46:40.001149+0000","last_peered":"2024-01-11T09:46:40.001149+0000","last_clean":"2024-01-11T09:46:40.001149+0000","last_became_active":"2024-01-11T09:46:40.000842+0000","last_became_peered":"2024-01-11T09:46:40.000842+0000","last_unstale":"2024-01-11T09:46:40.001149+0000","last_undegraded":"2024-01-11T09:46:40.001149+0000","last_fullsized":"2024-01-11T09:46:40.001149+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:30:34.408153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.997032+0000","last_change":"2024-01-11T09:46:38.992594+0000","last_active":"2024-01-11T09:46:38.997032+0000","last_peered":"2024-01-11T09:46:38.997032+0000","last_clean":"2024-01-11T09:46:38.997032+0000","last_became_active":"2024-01-11T09:46:38.992036+0000","last_became_peered":"2024-01-11T09:46:38.992036+0000","last_unstale":"2024-01-11T09:46:38.997032+0000","last_undegraded":"2024-01-11T09:46:38.997032+0000","last_fullsized":"2024-01-11T09:46:38.997032+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:27:17.589227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990993+0000","last_change":"2024-01-11T09:46:25.945778+0000","last_active":"2024-01-11T09:46:38.990993+0000","last_peered":"2024-01-11T09:46:38.990993+0000","last_clean":"2024-01-11T09:46:38.990993+0000","last_became_active":"2024-01-11T09:46:25.944907+0000","last_became_peered":"2024-01-11T09:46:25.944907+0000","last_unstale":"2024-01-11T09:46:38.990993+0000","last_undegraded":"2024-01-11T09:46:38.990993+0000","last_fullsized":"2024-01-11T09:46:38.990993+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:59:02.206221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.002961+0000","last_change":"2024-01-11T09:46:40.002961+0000","last_active":"2024-01-11T09:46:40.002961+0000","last_peered":"2024-01-11T09:46:40.002961+0000","last_clean":"2024-01-11T09:46:40.002961+0000","last_became_active":"2024-01-11T09:46:40.002649+0000","last_became_peered":"2024-01-11T09:46:40.002649+0000","last_unstale":"2024-01-11T09:46:40.002961+0000","last_undegraded":"2024-01-11T09:46:40.002961+0000","last_fullsized":"2024-01-11T09:46:40.002961+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:57:29.373530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.007915+0000","last_change":"2024-01-11T09:46:39.003378+0000","last_active":"2024-01-11T09:46:39.007915+0000","last_peered":"2024-01-11T09:46:39.007915+0000","last_clean":"2024-01-11T09:46:39.007915+0000","last_became_active":"2024-01-11T09:46:39.003118+0000","last_became_peered":"2024-01-11T09:46:39.003118+0000","last_unstale":"2024-01-11T09:46:39.007915+0000","last_undegraded":"2024-01-11T09:46:39.007915+0000","last_fullsized":"2024-01-11T09:46:39.007915+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:21:59.323435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.979967+0000","last_change":"2024-01-11T09:46:25.960002+0000","last_active":"2024-01-11T09:46:38.979967+0000","last_peered":"2024-01-11T09:46:38.979967+0000","last_clean":"2024-01-11T09:46:38.979967+0000","last_became_active":"2024-01-11T09:46:25.957845+0000","last_became_peered":"2024-01-11T09:46:25.957845+0000","last_unstale":"2024-01-11T09:46:38.979967+0000","last_undegraded":"2024-01-11T09:46:38.979967+0000","last_fullsized":"2024-01-11T09:46:38.979967+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:58:52.892728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.008739+0000","last_change":"2024-01-11T09:46:40.008739+0000","last_active":"2024-01-11T09:46:40.008739+0000","last_peered":"2024-01-11T09:46:40.008739+0000","last_clean":"2024-01-11T09:46:40.008739+0000","last_became_active":"2024-01-11T09:46:40.008481+0000","last_became_peered":"2024-01-11T09:46:40.008481+0000","last_unstale":"2024-01-11T09:46:40.008739+0000","last_undegraded":"2024-01-11T09:46:40.008739+0000","last_fullsized":"2024-01-11T09:46:40.008739+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:59:43.532603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"238'2","reported_seq":11,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.016124+0000","last_change":"2024-01-11T09:46:38.993477+0000","last_active":"2024-01-11T09:46:40.016124+0000","last_peered":"2024-01-11T09:46:40.016124+0000","last_clean":"2024-01-11T09:46:40.016124+0000","last_became_active":"2024-01-11T09:46:38.993223+0000","last_became_peered":"2024-01-11T09:46:38.993223+0000","last_unstale":"2024-01-11T09:46:40.016124+0000","last_undegraded":"2024-01-11T09:46:40.016124+0000","last_fullsized":"2024-01-11T09:46:40.016124+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:30:33.400129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986067+0000","last_change":"2024-01-11T09:46:25.958630+0000","last_active":"2024-01-11T09:46:38.986067+0000","last_peered":"2024-01-11T09:46:38.986067+0000","last_clean":"2024-01-11T09:46:38.986067+0000","last_became_active":"2024-01-11T09:46:25.957768+0000","last_became_peered":"2024-01-11T09:46:25.957768+0000","last_unstale":"2024-01-11T09:46:38.986067+0000","last_undegraded":"2024-01-11T09:46:38.986067+0000","last_fullsized":"2024-01-11T09:46:38.986067+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:59:15.007682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.997099+0000","last_change":"2024-01-11T09:46:39.997099+0000","last_active":"2024-01-11T09:46:39.997099+0000","last_peered":"2024-01-11T09:46:39.997099+0000","last_clean":"2024-01-11T09:46:39.997099+0000","last_became_active":"2024-01-11T09:46:39.996495+0000","last_became_peered":"2024-01-11T09:46:39.996495+0000","last_unstale":"2024-01-11T09:46:39.997099+0000","last_undegraded":"2024-01-11T09:46:39.997099+0000","last_fullsized":"2024-01-11T09:46:39.997099+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:09:57.054111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.996762+0000","last_change":"2024-01-11T09:46:38.994074+0000","last_active":"2024-01-11T09:46:38.996762+0000","last_peered":"2024-01-11T09:46:38.996762+0000","last_clean":"2024-01-11T09:46:38.996762+0000","last_became_active":"2024-01-11T09:46:38.993430+0000","last_became_peered":"2024-01-11T09:46:38.993430+0000","last_unstale":"2024-01-11T09:46:38.996762+0000","last_undegraded":"2024-01-11T09:46:38.996762+0000","last_fullsized":"2024-01-11T09:46:38.996762+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:37.969654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.983431+0000","last_change":"2024-01-11T09:46:25.954318+0000","last_active":"2024-01-11T09:46:38.983431+0000","last_peered":"2024-01-11T09:46:38.983431+0000","last_clean":"2024-01-11T09:46:38.983431+0000","last_became_active":"2024-01-11T09:46:25.953690+0000","last_became_peered":"2024-01-11T09:46:25.953690+0000","last_unstale":"2024-01-11T09:46:38.983431+0000","last_undegraded":"2024-01-11T09:46:38.983431+0000","last_fullsized":"2024-01-11T09:46:38.983431+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:28:55.053850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.991235+0000","last_change":"2024-01-11T09:46:39.991235+0000","last_active":"2024-01-11T09:46:39.991235+0000","last_peered":"2024-01-11T09:46:39.991235+0000","last_clean":"2024-01-11T09:46:39.991235+0000","last_became_active":"2024-01-11T09:46:39.990973+0000","last_became_peered":"2024-01-11T09:46:39.990973+0000","last_unstale":"2024-01-11T09:46:39.991235+0000","last_undegraded":"2024-01-11T09:46:39.991235+0000","last_fullsized":"2024-01-11T09:46:39.991235+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:54:48.495716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.993092+0000","last_change":"2024-01-11T09:46:38.990904+0000","last_active":"2024-01-11T09:46:38.993092+0000","last_peered":"2024-01-11T09:46:38.993092+0000","last_clean":"2024-01-11T09:46:38.993092+0000","last_became_active":"2024-01-11T09:46:38.990203+0000","last_became_peered":"2024-01-11T09:46:38.990203+0000","last_unstale":"2024-01-11T09:46:38.993092+0000","last_undegraded":"2024-01-11T09:46:38.993092+0000","last_fullsized":"2024-01-11T09:46:38.993092+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:25:22.331988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.981288+0000","last_change":"2024-01-11T09:46:25.957195+0000","last_active":"2024-01-11T09:46:38.981288+0000","last_peered":"2024-01-11T09:46:38.981288+0000","last_clean":"2024-01-11T09:46:38.981288+0000","last_became_active":"2024-01-11T09:46:25.956882+0000","last_became_peered":"2024-01-11T09:46:25.956882+0000","last_unstale":"2024-01-11T09:46:38.981288+0000","last_undegraded":"2024-01-11T09:46:38.981288+0000","last_fullsized":"2024-01-11T09:46:38.981288+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:37:41.519655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.010637+0000","last_change":"2024-01-11T09:46:40.010637+0000","last_active":"2024-01-11T09:46:40.010637+0000","last_peered":"2024-01-11T09:46:40.010637+0000","last_clean":"2024-01-11T09:46:40.010637+0000","last_became_active":"2024-01-11T09:46:40.009994+0000","last_became_peered":"2024-01-11T09:46:40.009994+0000","last_unstale":"2024-01-11T09:46:40.010637+0000","last_undegraded":"2024-01-11T09:46:40.010637+0000","last_fullsized":"2024-01-11T09:46:40.010637+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:54:35.025661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.994806+0000","last_change":"2024-01-11T09:46:38.993541+0000","last_active":"2024-01-11T09:46:38.994806+0000","last_peered":"2024-01-11T09:46:38.994806+0000","last_clean":"2024-01-11T09:46:38.994806+0000","last_became_active":"2024-01-11T09:46:38.992246+0000","last_became_peered":"2024-01-11T09:46:38.992246+0000","last_unstale":"2024-01-11T09:46:38.994806+0000","last_undegraded":"2024-01-11T09:46:38.994806+0000","last_fullsized":"2024-01-11T09:46:38.994806+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:18:20.631011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.978013+0000","last_change":"2024-01-11T09:46:25.960943+0000","last_active":"2024-01-11T09:46:38.978013+0000","last_peered":"2024-01-11T09:46:38.978013+0000","last_clean":"2024-01-11T09:46:38.978013+0000","last_became_active":"2024-01-11T09:46:25.960111+0000","last_became_peered":"2024-01-11T09:46:25.960111+0000","last_unstale":"2024-01-11T09:46:38.978013+0000","last_undegraded":"2024-01-11T09:46:38.978013+0000","last_fullsized":"2024-01-11T09:46:38.978013+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:04.562589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.017170+0000","last_change":"2024-01-11T09:46:40.017170+0000","last_active":"2024-01-11T09:46:40.017170+0000","last_peered":"2024-01-11T09:46:40.017170+0000","last_clean":"2024-01-11T09:46:40.017170+0000","last_became_active":"2024-01-11T09:46:40.013179+0000","last_became_peered":"2024-01-11T09:46:40.013179+0000","last_unstale":"2024-01-11T09:46:40.017170+0000","last_undegraded":"2024-01-11T09:46:40.017170+0000","last_fullsized":"2024-01-11T09:46:40.017170+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:10:42.456629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"238'2","reported_seq":11,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.013664+0000","last_change":"2024-01-11T09:46:38.991352+0000","last_active":"2024-01-11T09:46:40.013664+0000","last_peered":"2024-01-11T09:46:40.013664+0000","last_clean":"2024-01-11T09:46:40.013664+0000","last_became_active":"2024-01-11T09:46:38.991069+0000","last_became_peered":"2024-01-11T09:46:38.991069+0000","last_unstale":"2024-01-11T09:46:40.013664+0000","last_undegraded":"2024-01-11T09:46:40.013664+0000","last_fullsized":"2024-01-11T09:46:40.013664+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:10:22.443975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986956+0000","last_change":"2024-01-11T09:46:25.955367+0000","last_active":"2024-01-11T09:46:38.986956+0000","last_peered":"2024-01-11T09:46:38.986956+0000","last_clean":"2024-01-11T09:46:38.986956+0000","last_became_active":"2024-01-11T09:46:25.955123+0000","last_became_peered":"2024-01-11T09:46:25.955123+0000","last_unstale":"2024-01-11T09:46:38.986956+0000","last_undegraded":"2024-01-11T09:46:38.986956+0000","last_fullsized":"2024-01-11T09:46:38.986956+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:07:13.288202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.986527+0000","last_change":"2024-01-11T09:46:39.986527+0000","last_active":"2024-01-11T09:46:39.986527+0000","last_peered":"2024-01-11T09:46:39.986527+0000","last_clean":"2024-01-11T09:46:39.986527+0000","last_became_active":"2024-01-11T09:46:39.986243+0000","last_became_peered":"2024-01-11T09:46:39.986243+0000","last_unstale":"2024-01-11T09:46:39.986527+0000","last_undegraded":"2024-01-11T09:46:39.986527+0000","last_fullsized":"2024-01-11T09:46:39.986527+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:20.911474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.002782+0000","last_change":"2024-01-11T09:46:38.999657+0000","last_active":"2024-01-11T09:46:39.002782+0000","last_peered":"2024-01-11T09:46:39.002782+0000","last_clean":"2024-01-11T09:46:39.002782+0000","last_became_active":"2024-01-11T09:46:38.997446+0000","last_became_peered":"2024-01-11T09:46:38.997446+0000","last_unstale":"2024-01-11T09:46:39.002782+0000","last_undegraded":"2024-01-11T09:46:39.002782+0000","last_fullsized":"2024-01-11T09:46:39.002782+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:26:59.727081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980464+0000","last_change":"2024-01-11T09:46:25.958036+0000","last_active":"2024-01-11T09:46:38.980464+0000","last_peered":"2024-01-11T09:46:38.980464+0000","last_clean":"2024-01-11T09:46:38.980464+0000","last_became_active":"2024-01-11T09:46:25.957739+0000","last_became_peered":"2024-01-11T09:46:25.957739+0000","last_unstale":"2024-01-11T09:46:38.980464+0000","last_undegraded":"2024-01-11T09:46:38.980464+0000","last_fullsized":"2024-01-11T09:46:38.980464+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:04.562589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.999065+0000","last_change":"2024-01-11T09:46:39.999065+0000","last_active":"2024-01-11T09:46:39.999065+0000","last_peered":"2024-01-11T09:46:39.999065+0000","last_clean":"2024-01-11T09:46:39.999065+0000","last_became_active":"2024-01-11T09:46:39.998807+0000","last_became_peered":"2024-01-11T09:46:39.998807+0000","last_unstale":"2024-01-11T09:46:39.999065+0000","last_undegraded":"2024-01-11T09:46:39.999065+0000","last_fullsized":"2024-01-11T09:46:39.999065+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:30:59.947250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"238'1","reported_seq":10,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.983321+0000","last_change":"2024-01-11T09:46:39.000116+0000","last_active":"2024-01-11T09:46:39.983321+0000","last_peered":"2024-01-11T09:46:39.983321+0000","last_clean":"2024-01-11T09:46:39.983321+0000","last_became_active":"2024-01-11T09:46:38.999835+0000","last_became_peered":"2024-01-11T09:46:38.999835+0000","last_unstale":"2024-01-11T09:46:39.983321+0000","last_undegraded":"2024-01-11T09:46:39.983321+0000","last_fullsized":"2024-01-11T09:46:39.983321+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:53.897717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.988805+0000","last_change":"2024-01-11T09:46:25.951789+0000","last_active":"2024-01-11T09:46:38.988805+0000","last_peered":"2024-01-11T09:46:38.988805+0000","last_clean":"2024-01-11T09:46:38.988805+0000","last_became_active":"2024-01-11T09:46:25.950427+0000","last_became_peered":"2024-01-11T09:46:25.950427+0000","last_unstale":"2024-01-11T09:46:38.988805+0000","last_undegraded":"2024-01-11T09:46:38.988805+0000","last_fullsized":"2024-01-11T09:46:38.988805+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:10.125414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.003667+0000","last_change":"2024-01-11T09:46:40.003667+0000","last_active":"2024-01-11T09:46:40.003667+0000","last_peered":"2024-01-11T09:46:40.003667+0000","last_clean":"2024-01-11T09:46:40.003667+0000","last_became_active":"2024-01-11T09:46:40.003357+0000","last_became_peered":"2024-01-11T09:46:40.003357+0000","last_unstale":"2024-01-11T09:46:40.003667+0000","last_undegraded":"2024-01-11T09:46:40.003667+0000","last_fullsized":"2024-01-11T09:46:40.003667+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:56:27.940894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"238'2","reported_seq":11,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.023050+0000","last_change":"2024-01-11T09:46:38.989486+0000","last_active":"2024-01-11T09:46:40.023050+0000","last_peered":"2024-01-11T09:46:40.023050+0000","last_clean":"2024-01-11T09:46:40.023050+0000","last_became_active":"2024-01-11T09:46:38.989219+0000","last_became_peered":"2024-01-11T09:46:38.989219+0000","last_unstale":"2024-01-11T09:46:40.023050+0000","last_undegraded":"2024-01-11T09:46:40.023050+0000","last_fullsized":"2024-01-11T09:46:40.023050+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:41:48.456551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980046+0000","last_change":"2024-01-11T09:46:25.962921+0000","last_active":"2024-01-11T09:46:38.980046+0000","last_peered":"2024-01-11T09:46:38.980046+0000","last_clean":"2024-01-11T09:46:38.980046+0000","last_became_active":"2024-01-11T09:46:25.962306+0000","last_became_peered":"2024-01-11T09:46:25.962306+0000","last_unstale":"2024-01-11T09:46:38.980046+0000","last_undegraded":"2024-01-11T09:46:38.980046+0000","last_fullsized":"2024-01-11T09:46:38.980046+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:49:33.997022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.010679+0000","last_change":"2024-01-11T09:46:40.010679+0000","last_active":"2024-01-11T09:46:40.010679+0000","last_peered":"2024-01-11T09:46:40.010679+0000","last_clean":"2024-01-11T09:46:40.010679+0000","last_became_active":"2024-01-11T09:46:40.007871+0000","last_became_peered":"2024-01-11T09:46:40.007871+0000","last_unstale":"2024-01-11T09:46:40.010679+0000","last_undegraded":"2024-01-11T09:46:40.010679+0000","last_fullsized":"2024-01-11T09:46:40.010679+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:18:21.639035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"238'1","reported_seq":10,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.987717+0000","last_change":"2024-01-11T09:46:38.997759+0000","last_active":"2024-01-11T09:46:39.987717+0000","last_peered":"2024-01-11T09:46:39.987717+0000","last_clean":"2024-01-11T09:46:39.987717+0000","last_became_active":"2024-01-11T09:46:38.997461+0000","last_became_peered":"2024-01-11T09:46:38.997461+0000","last_unstale":"2024-01-11T09:46:39.987717+0000","last_undegraded":"2024-01-11T09:46:39.987717+0000","last_fullsized":"2024-01-11T09:46:39.987717+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:11:20.711273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.981007+0000","last_change":"2024-01-11T09:46:25.951750+0000","last_active":"2024-01-11T09:46:38.981007+0000","last_peered":"2024-01-11T09:46:38.981007+0000","last_clean":"2024-01-11T09:46:38.981007+0000","last_became_active":"2024-01-11T09:46:25.951499+0000","last_became_peered":"2024-01-11T09:46:25.951499+0000","last_unstale":"2024-01-11T09:46:38.981007+0000","last_undegraded":"2024-01-11T09:46:38.981007+0000","last_fullsized":"2024-01-11T09:46:38.981007+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:48:16.354636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.988056+0000","last_change":"2024-01-11T09:46:39.988056+0000","last_active":"2024-01-11T09:46:39.988056+0000","last_peered":"2024-01-11T09:46:39.988056+0000","last_clean":"2024-01-11T09:46:39.988056+0000","last_became_active":"2024-01-11T09:46:39.987747+0000","last_became_peered":"2024-01-11T09:46:39.987747+0000","last_unstale":"2024-01-11T09:46:39.988056+0000","last_undegraded":"2024-01-11T09:46:39.988056+0000","last_fullsized":"2024-01-11T09:46:39.988056+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:04:48.614151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"238'8","reported_seq":17,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.027417+0000","last_change":"2024-01-11T09:46:39.001969+0000","last_active":"2024-01-11T09:46:40.027417+0000","last_peered":"2024-01-11T09:46:40.027417+0000","last_clean":"2024-01-11T09:46:40.027417+0000","last_became_active":"2024-01-11T09:46:39.001149+0000","last_became_peered":"2024-01-11T09:46:39.001149+0000","last_unstale":"2024-01-11T09:46:40.027417+0000","last_undegraded":"2024-01-11T09:46:40.027417+0000","last_fullsized":"2024-01-11T09:46:40.027417+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:39:18.714435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986090+0000","last_change":"2024-01-11T09:46:25.950974+0000","last_active":"2024-01-11T09:46:38.986090+0000","last_peered":"2024-01-11T09:46:38.986090+0000","last_clean":"2024-01-11T09:46:38.986090+0000","last_became_active":"2024-01-11T09:46:25.950595+0000","last_became_peered":"2024-01-11T09:46:25.950595+0000","last_unstale":"2024-01-11T09:46:38.986090+0000","last_undegraded":"2024-01-11T09:46:38.986090+0000","last_fullsized":"2024-01-11T09:46:38.986090+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:58.843659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.999320+0000","last_change":"2024-01-11T09:46:39.999320+0000","last_active":"2024-01-11T09:46:39.999320+0000","last_peered":"2024-01-11T09:46:39.999320+0000","last_clean":"2024-01-11T09:46:39.999320+0000","last_became_active":"2024-01-11T09:46:39.999031+0000","last_became_peered":"2024-01-11T09:46:39.999031+0000","last_unstale":"2024-01-11T09:46:39.999320+0000","last_undegraded":"2024-01-11T09:46:39.999320+0000","last_fullsized":"2024-01-11T09:46:39.999320+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:03:39.674343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.993071+0000","last_change":"2024-01-11T09:46:38.990498+0000","last_active":"2024-01-11T09:46:38.993071+0000","last_peered":"2024-01-11T09:46:38.993071+0000","last_clean":"2024-01-11T09:46:38.993071+0000","last_became_active":"2024-01-11T09:46:38.990250+0000","last_became_peered":"2024-01-11T09:46:38.990250+0000","last_unstale":"2024-01-11T09:46:38.993071+0000","last_undegraded":"2024-01-11T09:46:38.993071+0000","last_fullsized":"2024-01-11T09:46:38.993071+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:27:47.259124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990576+0000","last_change":"2024-01-11T09:46:25.952186+0000","last_active":"2024-01-11T09:46:38.990576+0000","last_peered":"2024-01-11T09:46:38.990576+0000","last_clean":"2024-01-11T09:46:38.990576+0000","last_became_active":"2024-01-11T09:46:25.951898+0000","last_became_peered":"2024-01-11T09:46:25.951898+0000","last_unstale":"2024-01-11T09:46:38.990576+0000","last_undegraded":"2024-01-11T09:46:38.990576+0000","last_fullsized":"2024-01-11T09:46:38.990576+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:54:00.534598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.017717+0000","last_change":"2024-01-11T09:46:40.017717+0000","last_active":"2024-01-11T09:46:40.017717+0000","last_peered":"2024-01-11T09:46:40.017717+0000","last_clean":"2024-01-11T09:46:40.017717+0000","last_became_active":"2024-01-11T09:46:40.017424+0000","last_became_peered":"2024-01-11T09:46:40.017424+0000","last_unstale":"2024-01-11T09:46:40.017717+0000","last_undegraded":"2024-01-11T09:46:40.017717+0000","last_fullsized":"2024-01-11T09:46:40.017717+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:14:20.834540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.992273+0000","last_change":"2024-01-11T09:46:38.990154+0000","last_active":"2024-01-11T09:46:38.992273+0000","last_peered":"2024-01-11T09:46:38.992273+0000","last_clean":"2024-01-11T09:46:38.992273+0000","last_became_active":"2024-01-11T09:46:38.989880+0000","last_became_peered":"2024-01-11T09:46:38.989880+0000","last_unstale":"2024-01-11T09:46:38.992273+0000","last_undegraded":"2024-01-11T09:46:38.992273+0000","last_fullsized":"2024-01-11T09:46:38.992273+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:19:16.773855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986052+0000","last_change":"2024-01-11T09:46:25.957991+0000","last_active":"2024-01-11T09:46:38.986052+0000","last_peered":"2024-01-11T09:46:38.986052+0000","last_clean":"2024-01-11T09:46:38.986052+0000","last_became_active":"2024-01-11T09:46:25.957482+0000","last_became_peered":"2024-01-11T09:46:25.957482+0000","last_unstale":"2024-01-11T09:46:38.986052+0000","last_undegraded":"2024-01-11T09:46:38.986052+0000","last_fullsized":"2024-01-11T09:46:38.986052+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:54:00.534598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.995141+0000","last_change":"2024-01-11T09:46:39.995141+0000","last_active":"2024-01-11T09:46:39.995141+0000","last_peered":"2024-01-11T09:46:39.995141+0000","last_clean":"2024-01-11T09:46:39.995141+0000","last_became_active":"2024-01-11T09:46:39.994673+0000","last_became_peered":"2024-01-11T09:46:39.994673+0000","last_unstale":"2024-01-11T09:46:39.995141+0000","last_undegraded":"2024-01-11T09:46:39.995141+0000","last_fullsized":"2024-01-11T09:46:39.995141+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:44:58.685583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"238'2","reported_seq":11,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.020072+0000","last_change":"2024-01-11T09:46:39.001870+0000","last_active":"2024-01-11T09:46:40.020072+0000","last_peered":"2024-01-11T09:46:40.020072+0000","last_clean":"2024-01-11T09:46:40.020072+0000","last_became_active":"2024-01-11T09:46:39.001632+0000","last_became_peered":"2024-01-11T09:46:39.001632+0000","last_unstale":"2024-01-11T09:46:40.020072+0000","last_undegraded":"2024-01-11T09:46:40.020072+0000","last_fullsized":"2024-01-11T09:46:40.020072+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:59:42.524580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984733+0000","last_change":"2024-01-11T09:46:25.942788+0000","last_active":"2024-01-11T09:46:38.984733+0000","last_peered":"2024-01-11T09:46:38.984733+0000","last_clean":"2024-01-11T09:46:38.984733+0000","last_became_active":"2024-01-11T09:46:25.941771+0000","last_became_peered":"2024-01-11T09:46:25.941771+0000","last_unstale":"2024-01-11T09:46:38.984733+0000","last_undegraded":"2024-01-11T09:46:38.984733+0000","last_fullsized":"2024-01-11T09:46:38.984733+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:42:47.356295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.020514+0000","last_change":"2024-01-11T09:46:40.020514+0000","last_active":"2024-01-11T09:46:40.020514+0000","last_peered":"2024-01-11T09:46:40.020514+0000","last_clean":"2024-01-11T09:46:40.020514+0000","last_became_active":"2024-01-11T09:46:40.020274+0000","last_became_peered":"2024-01-11T09:46:40.020274+0000","last_unstale":"2024-01-11T09:46:40.020514+0000","last_undegraded":"2024-01-11T09:46:40.020514+0000","last_fullsized":"2024-01-11T09:46:40.020514+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:51:27.846178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"238'6","reported_seq":15,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.014817+0000","last_change":"2024-01-11T09:46:38.996034+0000","last_active":"2024-01-11T09:46:40.014817+0000","last_peered":"2024-01-11T09:46:40.014817+0000","last_clean":"2024-01-11T09:46:40.014817+0000","last_became_active":"2024-01-11T09:46:38.995771+0000","last_became_peered":"2024-01-11T09:46:38.995771+0000","last_unstale":"2024-01-11T09:46:40.014817+0000","last_undegraded":"2024-01-11T09:46:40.014817+0000","last_fullsized":"2024-01-11T09:46:40.014817+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:08:08.704554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986656+0000","last_change":"2024-01-11T09:46:25.954694+0000","last_active":"2024-01-11T09:46:38.986656+0000","last_peered":"2024-01-11T09:46:38.986656+0000","last_clean":"2024-01-11T09:46:38.986656+0000","last_became_active":"2024-01-11T09:46:25.952808+0000","last_became_peered":"2024-01-11T09:46:25.952808+0000","last_unstale":"2024-01-11T09:46:38.986656+0000","last_undegraded":"2024-01-11T09:46:38.986656+0000","last_fullsized":"2024-01-11T09:46:38.986656+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:10.125414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.004319+0000","last_change":"2024-01-11T09:46:40.004319+0000","last_active":"2024-01-11T09:46:40.004319+0000","last_peered":"2024-01-11T09:46:40.004319+0000","last_clean":"2024-01-11T09:46:40.004319+0000","last_became_active":"2024-01-11T09:46:40.004054+0000","last_became_peered":"2024-01-11T09:46:40.004054+0000","last_unstale":"2024-01-11T09:46:40.004319+0000","last_undegraded":"2024-01-11T09:46:40.004319+0000","last_fullsized":"2024-01-11T09:46:40.004319+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:24:06.711228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"238'3","reported_seq":12,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.023788+0000","last_change":"2024-01-11T09:46:38.994771+0000","last_active":"2024-01-11T09:46:40.023788+0000","last_peered":"2024-01-11T09:46:40.023788+0000","last_clean":"2024-01-11T09:46:40.023788+0000","last_became_active":"2024-01-11T09:46:38.994461+0000","last_became_peered":"2024-01-11T09:46:38.994461+0000","last_unstale":"2024-01-11T09:46:40.023788+0000","last_undegraded":"2024-01-11T09:46:40.023788+0000","last_fullsized":"2024-01-11T09:46:40.023788+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:45:29.584972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984134+0000","last_change":"2024-01-11T09:46:25.962654+0000","last_active":"2024-01-11T09:46:38.984134+0000","last_peered":"2024-01-11T09:46:38.984134+0000","last_clean":"2024-01-11T09:46:38.984134+0000","last_became_active":"2024-01-11T09:46:25.962073+0000","last_became_peered":"2024-01-11T09:46:25.962073+0000","last_unstale":"2024-01-11T09:46:38.984134+0000","last_undegraded":"2024-01-11T09:46:38.984134+0000","last_fullsized":"2024-01-11T09:46:38.984134+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:56:15.022301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.019981+0000","last_change":"2024-01-11T09:46:40.019981+0000","last_active":"2024-01-11T09:46:40.019981+0000","last_peered":"2024-01-11T09:46:40.019981+0000","last_clean":"2024-01-11T09:46:40.019981+0000","last_became_active":"2024-01-11T09:46:40.019280+0000","last_became_peered":"2024-01-11T09:46:40.019280+0000","last_unstale":"2024-01-11T09:46:40.019981+0000","last_undegraded":"2024-01-11T09:46:40.019981+0000","last_fullsized":"2024-01-11T09:46:40.019981+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:09:57.054111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.999886+0000","last_change":"2024-01-11T09:46:38.993119+0000","last_active":"2024-01-11T09:46:38.999886+0000","last_peered":"2024-01-11T09:46:38.999886+0000","last_clean":"2024-01-11T09:46:38.999886+0000","last_became_active":"2024-01-11T09:46:38.992846+0000","last_became_peered":"2024-01-11T09:46:38.992846+0000","last_unstale":"2024-01-11T09:46:38.999886+0000","last_undegraded":"2024-01-11T09:46:38.999886+0000","last_fullsized":"2024-01-11T09:46:38.999886+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:17.305476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986439+0000","last_change":"2024-01-11T09:46:25.956810+0000","last_active":"2024-01-11T09:46:38.986439+0000","last_peered":"2024-01-11T09:46:38.986439+0000","last_clean":"2024-01-11T09:46:38.986439+0000","last_became_active":"2024-01-11T09:46:25.955894+0000","last_became_peered":"2024-01-11T09:46:25.955894+0000","last_unstale":"2024-01-11T09:46:38.986439+0000","last_undegraded":"2024-01-11T09:46:38.986439+0000","last_fullsized":"2024-01-11T09:46:38.986439+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:53:35.139255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.994807+0000","last_change":"2024-01-11T09:46:39.994807+0000","last_active":"2024-01-11T09:46:39.994807+0000","last_peered":"2024-01-11T09:46:39.994807+0000","last_clean":"2024-01-11T09:46:39.994807+0000","last_became_active":"2024-01-11T09:46:39.994284+0000","last_became_peered":"2024-01-11T09:46:39.994284+0000","last_unstale":"2024-01-11T09:46:39.994807+0000","last_undegraded":"2024-01-11T09:46:39.994807+0000","last_fullsized":"2024-01-11T09:46:39.994807+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:41:33.450510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.001864+0000","last_change":"2024-01-11T09:46:38.998113+0000","last_active":"2024-01-11T09:46:39.001864+0000","last_peered":"2024-01-11T09:46:39.001864+0000","last_clean":"2024-01-11T09:46:39.001864+0000","last_became_active":"2024-01-11T09:46:38.997748+0000","last_became_peered":"2024-01-11T09:46:38.997748+0000","last_unstale":"2024-01-11T09:46:39.001864+0000","last_undegraded":"2024-01-11T09:46:39.001864+0000","last_fullsized":"2024-01-11T09:46:39.001864+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:38:11.656185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.989104+0000","last_change":"2024-01-11T09:46:25.947721+0000","last_active":"2024-01-11T09:46:38.989104+0000","last_peered":"2024-01-11T09:46:38.989104+0000","last_clean":"2024-01-11T09:46:38.989104+0000","last_became_active":"2024-01-11T09:46:25.947454+0000","last_became_peered":"2024-01-11T09:46:25.947454+0000","last_unstale":"2024-01-11T09:46:38.989104+0000","last_undegraded":"2024-01-11T09:46:38.989104+0000","last_fullsized":"2024-01-11T09:46:38.989104+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:42:44.083747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.994606+0000","last_change":"2024-01-11T09:46:39.994606+0000","last_active":"2024-01-11T09:46:39.994606+0000","last_peered":"2024-01-11T09:46:39.994606+0000","last_clean":"2024-01-11T09:46:39.994606+0000","last_became_active":"2024-01-11T09:46:39.994309+0000","last_became_peered":"2024-01-11T09:46:39.994309+0000","last_unstale":"2024-01-11T09:46:39.994606+0000","last_undegraded":"2024-01-11T09:46:39.994606+0000","last_fullsized":"2024-01-11T09:46:39.994606+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:37:32.540165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.996928+0000","last_change":"2024-01-11T09:46:38.991811+0000","last_active":"2024-01-11T09:46:38.996928+0000","last_peered":"2024-01-11T09:46:38.996928+0000","last_clean":"2024-01-11T09:46:38.996928+0000","last_became_active":"2024-01-11T09:46:38.990698+0000","last_became_peered":"2024-01-11T09:46:38.990698+0000","last_unstale":"2024-01-11T09:46:38.996928+0000","last_undegraded":"2024-01-11T09:46:38.996928+0000","last_fullsized":"2024-01-11T09:46:38.996928+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:54:34.017638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.995074+0000","last_change":"2024-01-11T09:46:25.939265+0000","last_active":"2024-01-11T09:46:38.995074+0000","last_peered":"2024-01-11T09:46:38.995074+0000","last_clean":"2024-01-11T09:46:38.995074+0000","last_became_active":"2024-01-11T09:46:25.938964+0000","last_became_peered":"2024-01-11T09:46:25.938964+0000","last_unstale":"2024-01-11T09:46:38.995074+0000","last_undegraded":"2024-01-11T09:46:38.995074+0000","last_fullsized":"2024-01-11T09:46:38.995074+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:20:24.956940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.013869+0000","last_change":"2024-01-11T09:46:40.013869+0000","last_active":"2024-01-11T09:46:40.013869+0000","last_peered":"2024-01-11T09:46:40.013869+0000","last_clean":"2024-01-11T09:46:40.013869+0000","last_became_active":"2024-01-11T09:46:40.012391+0000","last_became_peered":"2024-01-11T09:46:40.012391+0000","last_unstale":"2024-01-11T09:46:40.013869+0000","last_undegraded":"2024-01-11T09:46:40.013869+0000","last_fullsized":"2024-01-11T09:46:40.013869+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:15:01.956891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":9,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.007912+0000","last_change":"2024-01-11T09:46:39.003374+0000","last_active":"2024-01-11T09:46:39.007912+0000","last_peered":"2024-01-11T09:46:39.007912+0000","last_clean":"2024-01-11T09:46:39.007912+0000","last_became_active":"2024-01-11T09:46:39.003062+0000","last_became_peered":"2024-01-11T09:46:39.003062+0000","last_unstale":"2024-01-11T09:46:39.007912+0000","last_undegraded":"2024-01-11T09:46:39.007912+0000","last_fullsized":"2024-01-11T09:46:39.007912+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:31:12.578015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.981127+0000","last_change":"2024-01-11T09:46:25.953966+0000","last_active":"2024-01-11T09:46:38.981127+0000","last_peered":"2024-01-11T09:46:38.981127+0000","last_clean":"2024-01-11T09:46:38.981127+0000","last_became_active":"2024-01-11T09:46:25.953566+0000","last_became_peered":"2024-01-11T09:46:25.953566+0000","last_unstale":"2024-01-11T09:46:38.981127+0000","last_undegraded":"2024-01-11T09:46:38.981127+0000","last_fullsized":"2024-01-11T09:46:38.981127+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:45:16.842085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.010817+0000","last_change":"2024-01-11T09:46:40.010817+0000","last_active":"2024-01-11T09:46:40.010817+0000","last_peered":"2024-01-11T09:46:40.010817+0000","last_clean":"2024-01-11T09:46:40.010817+0000","last_became_active":"2024-01-11T09:46:40.010334+0000","last_became_peered":"2024-01-11T09:46:40.010334+0000","last_unstale":"2024-01-11T09:46:40.010817+0000","last_undegraded":"2024-01-11T09:46:40.010817+0000","last_fullsized":"2024-01-11T09:46:40.010817+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:10:14.041470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"238'1","reported_seq":10,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.980203+0000","last_change":"2024-01-11T09:46:38.997134+0000","last_active":"2024-01-11T09:46:39.980203+0000","last_peered":"2024-01-11T09:46:39.980203+0000","last_clean":"2024-01-11T09:46:39.980203+0000","last_became_active":"2024-01-11T09:46:38.996819+0000","last_became_peered":"2024-01-11T09:46:38.996819+0000","last_unstale":"2024-01-11T09:46:39.980203+0000","last_undegraded":"2024-01-11T09:46:39.980203+0000","last_fullsized":"2024-01-11T09:46:39.980203+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:03:27.598867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.978052+0000","last_change":"2024-01-11T09:46:25.959751+0000","last_active":"2024-01-11T09:46:38.978052+0000","last_peered":"2024-01-11T09:46:38.978052+0000","last_clean":"2024-01-11T09:46:38.978052+0000","last_became_active":"2024-01-11T09:46:25.959431+0000","last_became_peered":"2024-01-11T09:46:25.959431+0000","last_unstale":"2024-01-11T09:46:38.978052+0000","last_undegraded":"2024-01-11T09:46:38.978052+0000","last_fullsized":"2024-01-11T09:46:38.978052+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:58:52.892728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.993126+0000","last_change":"2024-01-11T09:46:39.993126+0000","last_active":"2024-01-11T09:46:39.993126+0000","last_peered":"2024-01-11T09:46:39.993126+0000","last_clean":"2024-01-11T09:46:39.993126+0000","last_became_active":"2024-01-11T09:46:39.992838+0000","last_became_peered":"2024-01-11T09:46:39.992838+0000","last_unstale":"2024-01-11T09:46:39.993126+0000","last_undegraded":"2024-01-11T09:46:39.993126+0000","last_fullsized":"2024-01-11T09:46:39.993126+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:49:09.210651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"238'1","reported_seq":10,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.985657+0000","last_change":"2024-01-11T09:46:38.995368+0000","last_active":"2024-01-11T09:46:39.985657+0000","last_peered":"2024-01-11T09:46:39.985657+0000","last_clean":"2024-01-11T09:46:39.985657+0000","last_became_active":"2024-01-11T09:46:38.995087+0000","last_became_peered":"2024-01-11T09:46:38.995087+0000","last_unstale":"2024-01-11T09:46:39.985657+0000","last_undegraded":"2024-01-11T09:46:39.985657+0000","last_fullsized":"2024-01-11T09:46:39.985657+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:47:57.662961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990411+0000","last_change":"2024-01-11T09:46:25.956345+0000","last_active":"2024-01-11T09:46:38.990411+0000","last_peered":"2024-01-11T09:46:38.990411+0000","last_clean":"2024-01-11T09:46:38.990411+0000","last_became_active":"2024-01-11T09:46:25.956080+0000","last_became_peered":"2024-01-11T09:46:25.956080+0000","last_unstale":"2024-01-11T09:46:38.990411+0000","last_undegraded":"2024-01-11T09:46:38.990411+0000","last_fullsized":"2024-01-11T09:46:38.990411+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:30:20.657242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.996253+0000","last_change":"2024-01-11T09:46:39.996253+0000","last_active":"2024-01-11T09:46:39.996253+0000","last_peered":"2024-01-11T09:46:39.996253+0000","last_clean":"2024-01-11T09:46:39.996253+0000","last_became_active":"2024-01-11T09:46:39.995997+0000","last_became_peered":"2024-01-11T09:46:39.995997+0000","last_unstale":"2024-01-11T09:46:39.996253+0000","last_undegraded":"2024-01-11T09:46:39.996253+0000","last_fullsized":"2024-01-11T09:46:39.996253+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:41:49.464575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"238'2","reported_seq":11,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.022163+0000","last_change":"2024-01-11T09:46:38.996864+0000","last_active":"2024-01-11T09:46:40.022163+0000","last_peered":"2024-01-11T09:46:40.022163+0000","last_clean":"2024-01-11T09:46:40.022163+0000","last_became_active":"2024-01-11T09:46:38.996594+0000","last_became_peered":"2024-01-11T09:46:38.996594+0000","last_unstale":"2024-01-11T09:46:40.022163+0000","last_undegraded":"2024-01-11T09:46:40.022163+0000","last_fullsized":"2024-01-11T09:46:40.022163+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:37:54.262542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.977660+0000","last_change":"2024-01-11T09:46:25.953176+0000","last_active":"2024-01-11T09:46:38.977660+0000","last_peered":"2024-01-11T09:46:38.977660+0000","last_clean":"2024-01-11T09:46:38.977660+0000","last_became_active":"2024-01-11T09:46:25.952276+0000","last_became_peered":"2024-01-11T09:46:25.952276+0000","last_unstale":"2024-01-11T09:46:38.977660+0000","last_undegraded":"2024-01-11T09:46:38.977660+0000","last_fullsized":"2024-01-11T09:46:38.977660+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:37:41.519655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.017999+0000","last_change":"2024-01-11T09:46:40.017999+0000","last_active":"2024-01-11T09:46:40.017999+0000","last_peered":"2024-01-11T09:46:40.017999+0000","last_clean":"2024-01-11T09:46:40.017999+0000","last_became_active":"2024-01-11T09:46:40.017756+0000","last_became_peered":"2024-01-11T09:46:40.017756+0000","last_unstale":"2024-01-11T09:46:40.017999+0000","last_undegraded":"2024-01-11T09:46:40.017999+0000","last_fullsized":"2024-01-11T09:46:40.017999+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:03:04.200008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"238'2","reported_seq":11,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.023981+0000","last_change":"2024-01-11T09:46:38.994164+0000","last_active":"2024-01-11T09:46:40.023981+0000","last_peered":"2024-01-11T09:46:40.023981+0000","last_clean":"2024-01-11T09:46:40.023981+0000","last_became_active":"2024-01-11T09:46:38.992666+0000","last_became_peered":"2024-01-11T09:46:38.992666+0000","last_unstale":"2024-01-11T09:46:40.023981+0000","last_undegraded":"2024-01-11T09:46:40.023981+0000","last_fullsized":"2024-01-11T09:46:40.023981+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:59:05.635616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.989167+0000","last_change":"2024-01-11T09:46:25.949114+0000","last_active":"2024-01-11T09:46:38.989167+0000","last_peered":"2024-01-11T09:46:38.989167+0000","last_clean":"2024-01-11T09:46:38.989167+0000","last_became_active":"2024-01-11T09:46:25.948844+0000","last_became_peered":"2024-01-11T09:46:25.948844+0000","last_unstale":"2024-01-11T09:46:38.989167+0000","last_undegraded":"2024-01-11T09:46:38.989167+0000","last_fullsized":"2024-01-11T09:46:38.989167+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:31:58.203452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.996228+0000","last_change":"2024-01-11T09:46:39.996228+0000","last_active":"2024-01-11T09:46:39.996228+0000","last_peered":"2024-01-11T09:46:39.996228+0000","last_clean":"2024-01-11T09:46:39.996228+0000","last_became_active":"2024-01-11T09:46:39.995964+0000","last_became_peered":"2024-01-11T09:46:39.995964+0000","last_unstale":"2024-01-11T09:46:39.996228+0000","last_undegraded":"2024-01-11T09:46:39.996228+0000","last_fullsized":"2024-01-11T09:46:39.996228+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:06:31.107627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"238'1","reported_seq":10,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.994751+0000","last_change":"2024-01-11T09:46:38.998074+0000","last_active":"2024-01-11T09:46:39.994751+0000","last_peered":"2024-01-11T09:46:39.994751+0000","last_clean":"2024-01-11T09:46:39.994751+0000","last_became_active":"2024-01-11T09:46:38.997784+0000","last_became_peered":"2024-01-11T09:46:38.997784+0000","last_unstale":"2024-01-11T09:46:39.994751+0000","last_undegraded":"2024-01-11T09:46:39.994751+0000","last_fullsized":"2024-01-11T09:46:39.994751+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:22.868301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984877+0000","last_change":"2024-01-11T09:46:25.954895+0000","last_active":"2024-01-11T09:46:38.984877+0000","last_peered":"2024-01-11T09:46:38.984877+0000","last_clean":"2024-01-11T09:46:38.984877+0000","last_became_active":"2024-01-11T09:46:25.954472+0000","last_became_peered":"2024-01-11T09:46:25.954472+0000","last_unstale":"2024-01-11T09:46:38.984877+0000","last_undegraded":"2024-01-11T09:46:38.984877+0000","last_fullsized":"2024-01-11T09:46:38.984877+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:34:26.903086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.004791+0000","last_change":"2024-01-11T09:46:40.004791+0000","last_active":"2024-01-11T09:46:40.004791+0000","last_peered":"2024-01-11T09:46:40.004791+0000","last_clean":"2024-01-11T09:46:40.004791+0000","last_became_active":"2024-01-11T09:46:40.004293+0000","last_became_peered":"2024-01-11T09:46:40.004293+0000","last_unstale":"2024-01-11T09:46:40.004791+0000","last_undegraded":"2024-01-11T09:46:40.004791+0000","last_fullsized":"2024-01-11T09:46:40.004791+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:20:57.093675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"238'4","reported_seq":13,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.021842+0000","last_change":"2024-01-11T09:46:39.000920+0000","last_active":"2024-01-11T09:46:40.021842+0000","last_peered":"2024-01-11T09:46:40.021842+0000","last_clean":"2024-01-11T09:46:40.021842+0000","last_became_active":"2024-01-11T09:46:39.000661+0000","last_became_peered":"2024-01-11T09:46:39.000661+0000","last_unstale":"2024-01-11T09:46:40.021842+0000","last_undegraded":"2024-01-11T09:46:40.021842+0000","last_fullsized":"2024-01-11T09:46:40.021842+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:37.955035+0000","last_clean_scrub_stamp":"2024-01-11T09:46:37.955035+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:10:41.448606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.987884+0000","last_change":"2024-01-11T09:46:25.955503+0000","last_active":"2024-01-11T09:46:38.987884+0000","last_peered":"2024-01-11T09:46:38.987884+0000","last_clean":"2024-01-11T09:46:38.987884+0000","last_became_active":"2024-01-11T09:46:25.954870+0000","last_became_peered":"2024-01-11T09:46:25.954870+0000","last_unstale":"2024-01-11T09:46:38.987884+0000","last_undegraded":"2024-01-11T09:46:38.987884+0000","last_fullsized":"2024-01-11T09:46:38.987884+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:06:12.133349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.999548+0000","last_change":"2024-01-11T09:46:39.999548+0000","last_active":"2024-01-11T09:46:39.999548+0000","last_peered":"2024-01-11T09:46:39.999548+0000","last_clean":"2024-01-11T09:46:39.999548+0000","last_became_active":"2024-01-11T09:46:39.999236+0000","last_became_peered":"2024-01-11T09:46:39.999236+0000","last_unstale":"2024-01-11T09:46:39.999548+0000","last_undegraded":"2024-01-11T09:46:39.999548+0000","last_fullsized":"2024-01-11T09:46:39.999548+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:59:06.643639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984281+0000","last_change":"2024-01-11T09:46:25.947226+0000","last_active":"2024-01-11T09:46:38.984281+0000","last_peered":"2024-01-11T09:46:38.984281+0000","last_clean":"2024-01-11T09:46:38.984281+0000","last_became_active":"2024-01-11T09:46:25.946806+0000","last_became_peered":"2024-01-11T09:46:25.946806+0000","last_unstale":"2024-01-11T09:46:38.984281+0000","last_undegraded":"2024-01-11T09:46:38.984281+0000","last_fullsized":"2024-01-11T09:46:38.984281+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:25:09.589100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"234'1","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980946+0000","last_change":"2024-01-11T09:46:25.236672+0000","last_active":"2024-01-11T09:46:38.980946+0000","last_peered":"2024-01-11T09:46:38.980946+0000","last_clean":"2024-01-11T09:46:38.980946+0000","last_became_active":"2024-01-11T09:46:25.236363+0000","last_became_peered":"2024-01-11T09:46:25.236363+0000","last_unstale":"2024-01-11T09:46:38.980946+0000","last_undegraded":"2024-01-11T09:46:38.980946+0000","last_fullsized":"2024-01-11T09:46:38.980946+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:01:24.091035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.005404+0000","last_change":"2024-01-11T09:46:40.005404+0000","last_active":"2024-01-11T09:46:40.005404+0000","last_peered":"2024-01-11T09:46:40.005404+0000","last_clean":"2024-01-11T09:46:40.005404+0000","last_became_active":"2024-01-11T09:46:40.005086+0000","last_became_peered":"2024-01-11T09:46:40.005086+0000","last_unstale":"2024-01-11T09:46:40.005404+0000","last_undegraded":"2024-01-11T09:46:40.005404+0000","last_fullsized":"2024-01-11T09:46:40.005404+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:27:48.267147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.977442+0000","last_change":"2024-01-11T09:46:25.951411+0000","last_active":"2024-01-11T09:46:38.977442+0000","last_peered":"2024-01-11T09:46:38.977442+0000","last_clean":"2024-01-11T09:46:38.977442+0000","last_became_active":"2024-01-11T09:46:25.951161+0000","last_became_peered":"2024-01-11T09:46:25.951161+0000","last_unstale":"2024-01-11T09:46:38.977442+0000","last_undegraded":"2024-01-11T09:46:38.977442+0000","last_fullsized":"2024-01-11T09:46:38.977442+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:45:16.842085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"234'2","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984056+0000","last_change":"2024-01-11T09:46:25.238481+0000","last_active":"2024-01-11T09:46:38.984056+0000","last_peered":"2024-01-11T09:46:38.984056+0000","last_clean":"2024-01-11T09:46:38.984056+0000","last_became_active":"2024-01-11T09:46:25.238203+0000","last_became_peered":"2024-01-11T09:46:25.238203+0000","last_unstale":"2024-01-11T09:46:38.984056+0000","last_undegraded":"2024-01-11T09:46:38.984056+0000","last_fullsized":"2024-01-11T09:46:38.984056+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:42:43.074601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.016036+0000","last_change":"2024-01-11T09:46:40.016036+0000","last_active":"2024-01-11T09:46:40.016036+0000","last_peered":"2024-01-11T09:46:40.016036+0000","last_clean":"2024-01-11T09:46:40.016036+0000","last_became_active":"2024-01-11T09:46:40.015608+0000","last_became_peered":"2024-01-11T09:46:40.015608+0000","last_unstale":"2024-01-11T09:46:40.016036+0000","last_undegraded":"2024-01-11T09:46:40.016036+0000","last_fullsized":"2024-01-11T09:46:40.016036+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:44:19.712167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986139+0000","last_change":"2024-01-11T09:46:25.954029+0000","last_active":"2024-01-11T09:46:38.986139+0000","last_peered":"2024-01-11T09:46:38.986139+0000","last_clean":"2024-01-11T09:46:38.986139+0000","last_became_active":"2024-01-11T09:46:25.953690+0000","last_became_peered":"2024-01-11T09:46:25.953690+0000","last_unstale":"2024-01-11T09:46:38.986139+0000","last_undegraded":"2024-01-11T09:46:38.986139+0000","last_fullsized":"2024-01-11T09:46:38.986139+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:26:46.984194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"234'2","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.985187+0000","last_change":"2024-01-11T09:46:25.227884+0000","last_active":"2024-01-11T09:46:38.985187+0000","last_peered":"2024-01-11T09:46:38.985187+0000","last_clean":"2024-01-11T09:46:38.985187+0000","last_became_active":"2024-01-11T09:46:25.227605+0000","last_became_peered":"2024-01-11T09:46:25.227605+0000","last_unstale":"2024-01-11T09:46:38.985187+0000","last_undegraded":"2024-01-11T09:46:38.985187+0000","last_fullsized":"2024-01-11T09:46:38.985187+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:09.116268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.010140+0000","last_change":"2024-01-11T09:46:40.010140+0000","last_active":"2024-01-11T09:46:40.010140+0000","last_peered":"2024-01-11T09:46:40.010140+0000","last_clean":"2024-01-11T09:46:40.010140+0000","last_became_active":"2024-01-11T09:46:40.009493+0000","last_became_peered":"2024-01-11T09:46:40.009493+0000","last_unstale":"2024-01-11T09:46:40.010140+0000","last_undegraded":"2024-01-11T09:46:40.010140+0000","last_fullsized":"2024-01-11T09:46:40.010140+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:25:23.340011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.977338+0000","last_change":"2024-01-11T09:46:25.950375+0000","last_active":"2024-01-11T09:46:38.977338+0000","last_peered":"2024-01-11T09:46:38.977338+0000","last_clean":"2024-01-11T09:46:38.977338+0000","last_became_active":"2024-01-11T09:46:25.950114+0000","last_became_peered":"2024-01-11T09:46:25.950114+0000","last_unstale":"2024-01-11T09:46:38.977338+0000","last_undegraded":"2024-01-11T09:46:38.977338+0000","last_fullsized":"2024-01-11T09:46:38.977338+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:25.226766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"234'1","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.993589+0000","last_change":"2024-01-11T09:46:25.233693+0000","last_active":"2024-01-11T09:46:38.993589+0000","last_peered":"2024-01-11T09:46:38.993589+0000","last_clean":"2024-01-11T09:46:38.993589+0000","last_became_active":"2024-01-11T09:46:25.233410+0000","last_became_peered":"2024-01-11T09:46:25.233410+0000","last_unstale":"2024-01-11T09:46:38.993589+0000","last_undegraded":"2024-01-11T09:46:38.993589+0000","last_fullsized":"2024-01-11T09:46:38.993589+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:03:42.704031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.003083+0000","last_change":"2024-01-11T09:46:40.003083+0000","last_active":"2024-01-11T09:46:40.003083+0000","last_peered":"2024-01-11T09:46:40.003083+0000","last_clean":"2024-01-11T09:46:40.003083+0000","last_became_active":"2024-01-11T09:46:40.002819+0000","last_became_peered":"2024-01-11T09:46:40.002819+0000","last_unstale":"2024-01-11T09:46:40.003083+0000","last_undegraded":"2024-01-11T09:46:40.003083+0000","last_fullsized":"2024-01-11T09:46:40.003083+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:47:54.704748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.981687+0000","last_change":"2024-01-11T09:46:25.955874+0000","last_active":"2024-01-11T09:46:38.981687+0000","last_peered":"2024-01-11T09:46:38.981687+0000","last_clean":"2024-01-11T09:46:38.981687+0000","last_became_active":"2024-01-11T09:46:25.954789+0000","last_became_peered":"2024-01-11T09:46:25.954789+0000","last_unstale":"2024-01-11T09:46:38.981687+0000","last_undegraded":"2024-01-11T09:46:38.981687+0000","last_fullsized":"2024-01-11T09:46:38.981687+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:07.160563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"234'1","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.988905+0000","last_change":"2024-01-11T09:46:25.235297+0000","last_active":"2024-01-11T09:46:38.988905+0000","last_peered":"2024-01-11T09:46:38.988905+0000","last_clean":"2024-01-11T09:46:38.988905+0000","last_became_active":"2024-01-11T09:46:25.234744+0000","last_became_peered":"2024-01-11T09:46:25.234744+0000","last_unstale":"2024-01-11T09:46:38.988905+0000","last_undegraded":"2024-01-11T09:46:38.988905+0000","last_fullsized":"2024-01-11T09:46:38.988905+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:51:44.543489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.995389+0000","last_change":"2024-01-11T09:46:39.995389+0000","last_active":"2024-01-11T09:46:39.995389+0000","last_peered":"2024-01-11T09:46:39.995389+0000","last_clean":"2024-01-11T09:46:39.995389+0000","last_became_active":"2024-01-11T09:46:39.995111+0000","last_became_peered":"2024-01-11T09:46:39.995111+0000","last_unstale":"2024-01-11T09:46:39.995389+0000","last_undegraded":"2024-01-11T09:46:39.995389+0000","last_fullsized":"2024-01-11T09:46:39.995389+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:07:27.039113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980841+0000","last_change":"2024-01-11T09:46:25.948668+0000","last_active":"2024-01-11T09:46:38.980841+0000","last_peered":"2024-01-11T09:46:38.980841+0000","last_clean":"2024-01-11T09:46:38.980841+0000","last_became_active":"2024-01-11T09:46:25.948372+0000","last_became_peered":"2024-01-11T09:46:25.948372+0000","last_unstale":"2024-01-11T09:46:38.980841+0000","last_undegraded":"2024-01-11T09:46:38.980841+0000","last_fullsized":"2024-01-11T09:46:38.980841+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:41.154829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980164+0000","last_change":"2024-01-11T09:46:25.238227+0000","last_active":"2024-01-11T09:46:38.980164+0000","last_peered":"2024-01-11T09:46:38.980164+0000","last_clean":"2024-01-11T09:46:38.980164+0000","last_became_active":"2024-01-11T09:46:25.238013+0000","last_became_peered":"2024-01-11T09:46:25.238013+0000","last_unstale":"2024-01-11T09:46:38.980164+0000","last_undegraded":"2024-01-11T09:46:38.980164+0000","last_fullsized":"2024-01-11T09:46:38.980164+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:36:18.777304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.999708+0000","last_change":"2024-01-11T09:46:39.999708+0000","last_active":"2024-01-11T09:46:39.999708+0000","last_peered":"2024-01-11T09:46:39.999708+0000","last_clean":"2024-01-11T09:46:39.999708+0000","last_became_active":"2024-01-11T09:46:39.999341+0000","last_became_peered":"2024-01-11T09:46:39.999341+0000","last_unstale":"2024-01-11T09:46:39.999708+0000","last_undegraded":"2024-01-11T09:46:39.999708+0000","last_fullsized":"2024-01-11T09:46:39.999708+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:09:49.159232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.988745+0000","last_change":"2024-01-11T09:46:25.949014+0000","last_active":"2024-01-11T09:46:38.988745+0000","last_peered":"2024-01-11T09:46:38.988745+0000","last_clean":"2024-01-11T09:46:38.988745+0000","last_became_active":"2024-01-11T09:46:25.948717+0000","last_became_peered":"2024-01-11T09:46:25.948717+0000","last_unstale":"2024-01-11T09:46:38.988745+0000","last_undegraded":"2024-01-11T09:46:38.988745+0000","last_fullsized":"2024-01-11T09:46:38.988745+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:07.160563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980927+0000","last_change":"2024-01-11T09:46:25.225980+0000","last_active":"2024-01-11T09:46:38.980927+0000","last_peered":"2024-01-11T09:46:38.980927+0000","last_clean":"2024-01-11T09:46:38.980927+0000","last_became_active":"2024-01-11T09:46:25.225444+0000","last_became_peered":"2024-01-11T09:46:25.225444+0000","last_unstale":"2024-01-11T09:46:38.980927+0000","last_undegraded":"2024-01-11T09:46:38.980927+0000","last_fullsized":"2024-01-11T09:46:38.980927+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:30:19.648096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.989639+0000","last_change":"2024-01-11T09:46:39.989639+0000","last_active":"2024-01-11T09:46:39.989639+0000","last_peered":"2024-01-11T09:46:39.989639+0000","last_clean":"2024-01-11T09:46:39.989639+0000","last_became_active":"2024-01-11T09:46:39.989397+0000","last_became_peered":"2024-01-11T09:46:39.989397+0000","last_unstale":"2024-01-11T09:46:39.989639+0000","last_undegraded":"2024-01-11T09:46:39.989639+0000","last_fullsized":"2024-01-11T09:46:39.989639+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:31:00.593411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990657+0000","last_change":"2024-01-11T09:46:25.953219+0000","last_active":"2024-01-11T09:46:38.990657+0000","last_peered":"2024-01-11T09:46:38.990657+0000","last_clean":"2024-01-11T09:46:38.990657+0000","last_became_active":"2024-01-11T09:46:25.952967+0000","last_became_peered":"2024-01-11T09:46:25.952967+0000","last_unstale":"2024-01-11T09:46:38.990657+0000","last_undegraded":"2024-01-11T09:46:38.990657+0000","last_fullsized":"2024-01-11T09:46:38.990657+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:07:13.288202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986207+0000","last_change":"2024-01-11T09:46:25.237685+0000","last_active":"2024-01-11T09:46:38.986207+0000","last_peered":"2024-01-11T09:46:38.986207+0000","last_clean":"2024-01-11T09:46:38.986207+0000","last_became_active":"2024-01-11T09:46:25.237419+0000","last_became_peered":"2024-01-11T09:46:25.237419+0000","last_unstale":"2024-01-11T09:46:38.986207+0000","last_undegraded":"2024-01-11T09:46:38.986207+0000","last_fullsized":"2024-01-11T09:46:38.986207+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:07:52.221452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.995097+0000","last_change":"2024-01-11T09:46:39.995097+0000","last_active":"2024-01-11T09:46:39.995097+0000","last_peered":"2024-01-11T09:46:39.995097+0000","last_clean":"2024-01-11T09:46:39.995097+0000","last_became_active":"2024-01-11T09:46:39.994814+0000","last_became_peered":"2024-01-11T09:46:39.994814+0000","last_unstale":"2024-01-11T09:46:39.995097+0000","last_undegraded":"2024-01-11T09:46:39.995097+0000","last_fullsized":"2024-01-11T09:46:39.995097+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:54:22.183997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.979295+0000","last_change":"2024-01-11T09:46:25.950838+0000","last_active":"2024-01-11T09:46:38.979295+0000","last_peered":"2024-01-11T09:46:38.979295+0000","last_clean":"2024-01-11T09:46:38.979295+0000","last_became_active":"2024-01-11T09:46:25.950551+0000","last_became_peered":"2024-01-11T09:46:25.950551+0000","last_unstale":"2024-01-11T09:46:38.979295+0000","last_undegraded":"2024-01-11T09:46:38.979295+0000","last_fullsized":"2024-01-11T09:46:38.979295+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:39:05.971547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.982892+0000","last_change":"2024-01-11T09:46:25.232121+0000","last_active":"2024-01-11T09:46:38.982892+0000","last_peered":"2024-01-11T09:46:38.982892+0000","last_clean":"2024-01-11T09:46:38.982892+0000","last_became_active":"2024-01-11T09:46:25.231859+0000","last_became_peered":"2024-01-11T09:46:25.231859+0000","last_unstale":"2024-01-11T09:46:38.982892+0000","last_undegraded":"2024-01-11T09:46:38.982892+0000","last_fullsized":"2024-01-11T09:46:38.982892+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:07:12.279056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.013866+0000","last_change":"2024-01-11T09:46:40.013866+0000","last_active":"2024-01-11T09:46:40.013866+0000","last_peered":"2024-01-11T09:46:40.013866+0000","last_clean":"2024-01-11T09:46:40.013866+0000","last_became_active":"2024-01-11T09:46:40.013285+0000","last_became_peered":"2024-01-11T09:46:40.013285+0000","last_unstale":"2024-01-11T09:46:40.013866+0000","last_undegraded":"2024-01-11T09:46:40.013866+0000","last_fullsized":"2024-01-11T09:46:40.013866+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:44:19.712167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984855+0000","last_change":"2024-01-11T09:46:25.948806+0000","last_active":"2024-01-11T09:46:38.984855+0000","last_peered":"2024-01-11T09:46:38.984855+0000","last_clean":"2024-01-11T09:46:38.984855+0000","last_became_active":"2024-01-11T09:46:25.948526+0000","last_became_peered":"2024-01-11T09:46:25.948526+0000","last_unstale":"2024-01-11T09:46:38.984855+0000","last_undegraded":"2024-01-11T09:46:38.984855+0000","last_fullsized":"2024-01-11T09:46:38.984855+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:51:19.504238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"234'3","reported_seq":16,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.991439+0000","last_change":"2024-01-11T09:46:25.238242+0000","last_active":"2024-01-11T09:46:38.991439+0000","last_peered":"2024-01-11T09:46:38.991439+0000","last_clean":"2024-01-11T09:46:38.991439+0000","last_became_active":"2024-01-11T09:46:25.237965+0000","last_became_peered":"2024-01-11T09:46:25.237965+0000","last_unstale":"2024-01-11T09:46:38.991439+0000","last_undegraded":"2024-01-11T09:46:38.991439+0000","last_fullsized":"2024-01-11T09:46:38.991439+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:58:02.388540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.003570+0000","last_change":"2024-01-11T09:46:40.003570+0000","last_active":"2024-01-11T09:46:40.003570+0000","last_peered":"2024-01-11T09:46:40.003570+0000","last_clean":"2024-01-11T09:46:40.003570+0000","last_became_active":"2024-01-11T09:46:40.002034+0000","last_became_peered":"2024-01-11T09:46:40.002034+0000","last_unstale":"2024-01-11T09:46:40.003570+0000","last_undegraded":"2024-01-11T09:46:40.003570+0000","last_fullsized":"2024-01-11T09:46:40.003570+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:51:27.846178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.981368+0000","last_change":"2024-01-11T09:46:25.950108+0000","last_active":"2024-01-11T09:46:38.981368+0000","last_peered":"2024-01-11T09:46:38.981368+0000","last_clean":"2024-01-11T09:46:38.981368+0000","last_became_active":"2024-01-11T09:46:25.949796+0000","last_became_peered":"2024-01-11T09:46:25.949796+0000","last_unstale":"2024-01-11T09:46:38.981368+0000","last_undegraded":"2024-01-11T09:46:38.981368+0000","last_fullsized":"2024-01-11T09:46:38.981368+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:11:07.968385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"235'8","reported_seq":21,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990505+0000","last_change":"2024-01-11T09:46:25.239177+0000","last_active":"2024-01-11T09:46:38.990505+0000","last_peered":"2024-01-11T09:46:38.990505+0000","last_clean":"2024-01-11T09:46:38.990505+0000","last_became_active":"2024-01-11T09:46:25.238865+0000","last_became_peered":"2024-01-11T09:46:25.238865+0000","last_unstale":"2024-01-11T09:46:38.990505+0000","last_undegraded":"2024-01-11T09:46:38.990505+0000","last_fullsized":"2024-01-11T09:46:38.990505+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:29:50.151952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2749,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.017512+0000","last_change":"2024-01-11T09:46:40.017512+0000","last_active":"2024-01-11T09:46:40.017512+0000","last_peered":"2024-01-11T09:46:40.017512+0000","last_clean":"2024-01-11T09:46:40.017512+0000","last_became_active":"2024-01-11T09:46:40.017221+0000","last_became_peered":"2024-01-11T09:46:40.017221+0000","last_unstale":"2024-01-11T09:46:40.017512+0000","last_undegraded":"2024-01-11T09:46:40.017512+0000","last_fullsized":"2024-01-11T09:46:40.017512+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:03:28.606890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984235+0000","last_change":"2024-01-11T09:46:25.951823+0000","last_active":"2024-01-11T09:46:38.984235+0000","last_peered":"2024-01-11T09:46:38.984235+0000","last_clean":"2024-01-11T09:46:38.984235+0000","last_became_active":"2024-01-11T09:46:25.951493+0000","last_became_peered":"2024-01-11T09:46:25.951493+0000","last_unstale":"2024-01-11T09:46:38.984235+0000","last_undegraded":"2024-01-11T09:46:38.984235+0000","last_fullsized":"2024-01-11T09:46:38.984235+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:59:22.974857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"234'2","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990118+0000","last_change":"2024-01-11T09:46:25.244293+0000","last_active":"2024-01-11T09:46:38.990118+0000","last_peered":"2024-01-11T09:46:38.990118+0000","last_clean":"2024-01-11T09:46:38.990118+0000","last_became_active":"2024-01-11T09:46:25.243959+0000","last_became_peered":"2024-01-11T09:46:25.243959+0000","last_unstale":"2024-01-11T09:46:38.990118+0000","last_undegraded":"2024-01-11T09:46:38.990118+0000","last_fullsized":"2024-01-11T09:46:38.990118+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:13:10.353715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.007024+0000","last_change":"2024-01-11T09:46:40.007024+0000","last_active":"2024-01-11T09:46:40.007024+0000","last_peered":"2024-01-11T09:46:40.007024+0000","last_clean":"2024-01-11T09:46:40.007024+0000","last_became_active":"2024-01-11T09:46:40.006049+0000","last_became_peered":"2024-01-11T09:46:40.006049+0000","last_unstale":"2024-01-11T09:46:40.007024+0000","last_undegraded":"2024-01-11T09:46:40.007024+0000","last_fullsized":"2024-01-11T09:46:40.007024+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:45:58.839035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.982248+0000","last_change":"2024-01-11T09:46:25.958245+0000","last_active":"2024-01-11T09:46:38.982248+0000","last_peered":"2024-01-11T09:46:38.982248+0000","last_clean":"2024-01-11T09:46:38.982248+0000","last_became_active":"2024-01-11T09:46:25.957893+0000","last_became_peered":"2024-01-11T09:46:25.957893+0000","last_unstale":"2024-01-11T09:46:38.982248+0000","last_undegraded":"2024-01-11T09:46:38.982248+0000","last_fullsized":"2024-01-11T09:46:38.982248+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:43:22.062566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.982221+0000","last_change":"2024-01-11T09:46:25.231846+0000","last_active":"2024-01-11T09:46:38.982221+0000","last_peered":"2024-01-11T09:46:38.982221+0000","last_clean":"2024-01-11T09:46:38.982221+0000","last_became_active":"2024-01-11T09:46:25.229498+0000","last_became_peered":"2024-01-11T09:46:25.229498+0000","last_unstale":"2024-01-11T09:46:38.982221+0000","last_undegraded":"2024-01-11T09:46:38.982221+0000","last_fullsized":"2024-01-11T09:46:38.982221+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:53:34.130109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.001720+0000","last_change":"2024-01-11T09:46:40.001720+0000","last_active":"2024-01-11T09:46:40.001720+0000","last_peered":"2024-01-11T09:46:40.001720+0000","last_clean":"2024-01-11T09:46:40.001720+0000","last_became_active":"2024-01-11T09:46:40.001315+0000","last_became_peered":"2024-01-11T09:46:40.001315+0000","last_unstale":"2024-01-11T09:46:40.001720+0000","last_undegraded":"2024-01-11T09:46:40.001720+0000","last_fullsized":"2024-01-11T09:46:40.001720+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:14:18.913296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984870+0000","last_change":"2024-01-11T09:46:25.947333+0000","last_active":"2024-01-11T09:46:38.984870+0000","last_peered":"2024-01-11T09:46:38.984870+0000","last_clean":"2024-01-11T09:46:38.984870+0000","last_became_active":"2024-01-11T09:46:25.946307+0000","last_became_peered":"2024-01-11T09:46:25.946307+0000","last_unstale":"2024-01-11T09:46:38.984870+0000","last_undegraded":"2024-01-11T09:46:38.984870+0000","last_fullsized":"2024-01-11T09:46:38.984870+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:32:47.063220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980245+0000","last_change":"2024-01-11T09:46:25.239044+0000","last_active":"2024-01-11T09:46:38.980245+0000","last_peered":"2024-01-11T09:46:38.980245+0000","last_clean":"2024-01-11T09:46:38.980245+0000","last_became_active":"2024-01-11T09:46:25.238780+0000","last_became_peered":"2024-01-11T09:46:25.238780+0000","last_unstale":"2024-01-11T09:46:38.980245+0000","last_undegraded":"2024-01-11T09:46:38.980245+0000","last_fullsized":"2024-01-11T09:46:38.980245+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:34:37.940301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.002357+0000","last_change":"2024-01-11T09:46:40.002357+0000","last_active":"2024-01-11T09:46:40.002357+0000","last_peered":"2024-01-11T09:46:40.002357+0000","last_clean":"2024-01-11T09:46:40.002357+0000","last_became_active":"2024-01-11T09:46:39.994174+0000","last_became_peered":"2024-01-11T09:46:39.994174+0000","last_unstale":"2024-01-11T09:46:40.002357+0000","last_undegraded":"2024-01-11T09:46:40.002357+0000","last_fullsized":"2024-01-11T09:46:40.002357+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:49:09.210651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980062+0000","last_change":"2024-01-11T09:46:25.962027+0000","last_active":"2024-01-11T09:46:38.980062+0000","last_peered":"2024-01-11T09:46:38.980062+0000","last_clean":"2024-01-11T09:46:38.980062+0000","last_became_active":"2024-01-11T09:46:25.961646+0000","last_became_peered":"2024-01-11T09:46:25.961646+0000","last_unstale":"2024-01-11T09:46:38.980062+0000","last_undegraded":"2024-01-11T09:46:38.980062+0000","last_fullsized":"2024-01-11T09:46:38.980062+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:39:05.971547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"234'8","reported_seq":21,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.977073+0000","last_change":"2024-01-11T09:46:25.238694+0000","last_active":"2024-01-11T09:46:38.977073+0000","last_peered":"2024-01-11T09:46:38.977073+0000","last_clean":"2024-01-11T09:46:38.977073+0000","last_became_active":"2024-01-11T09:46:25.237565+0000","last_became_peered":"2024-01-11T09:46:25.237565+0000","last_unstale":"2024-01-11T09:46:38.977073+0000","last_undegraded":"2024-01-11T09:46:38.977073+0000","last_fullsized":"2024-01-11T09:46:38.977073+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:25:08.579955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.014333+0000","last_change":"2024-01-11T09:46:40.014333+0000","last_active":"2024-01-11T09:46:40.014333+0000","last_peered":"2024-01-11T09:46:40.014333+0000","last_clean":"2024-01-11T09:46:40.014333+0000","last_became_active":"2024-01-11T09:46:40.014085+0000","last_became_peered":"2024-01-11T09:46:40.014085+0000","last_unstale":"2024-01-11T09:46:40.014333+0000","last_undegraded":"2024-01-11T09:46:40.014333+0000","last_fullsized":"2024-01-11T09:46:40.014333+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:34:54.101440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.994750+0000","last_change":"2024-01-11T09:46:25.946186+0000","last_active":"2024-01-11T09:46:38.994750+0000","last_peered":"2024-01-11T09:46:38.994750+0000","last_clean":"2024-01-11T09:46:38.994750+0000","last_became_active":"2024-01-11T09:46:25.945907+0000","last_became_peered":"2024-01-11T09:46:25.945907+0000","last_unstale":"2024-01-11T09:46:38.994750+0000","last_undegraded":"2024-01-11T09:46:38.994750+0000","last_fullsized":"2024-01-11T09:46:38.994750+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:26:51.516156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"234'1","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.981243+0000","last_change":"2024-01-11T09:46:25.241687+0000","last_active":"2024-01-11T09:46:38.981243+0000","last_peered":"2024-01-11T09:46:38.981243+0000","last_clean":"2024-01-11T09:46:38.981243+0000","last_became_active":"2024-01-11T09:46:25.241369+0000","last_became_peered":"2024-01-11T09:46:25.241369+0000","last_unstale":"2024-01-11T09:46:38.981243+0000","last_undegraded":"2024-01-11T09:46:38.981243+0000","last_fullsized":"2024-01-11T09:46:38.981243+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:40.145683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.999085+0000","last_change":"2024-01-11T09:46:39.999085+0000","last_active":"2024-01-11T09:46:39.999085+0000","last_peered":"2024-01-11T09:46:39.999085+0000","last_clean":"2024-01-11T09:46:39.999085+0000","last_became_active":"2024-01-11T09:46:39.997666+0000","last_became_peered":"2024-01-11T09:46:39.997666+0000","last_unstale":"2024-01-11T09:46:39.999085+0000","last_undegraded":"2024-01-11T09:46:39.999085+0000","last_fullsized":"2024-01-11T09:46:39.999085+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:48:30.105547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984370+0000","last_change":"2024-01-11T09:46:25.951531+0000","last_active":"2024-01-11T09:46:38.984370+0000","last_peered":"2024-01-11T09:46:38.984370+0000","last_clean":"2024-01-11T09:46:38.984370+0000","last_became_active":"2024-01-11T09:46:25.950791+0000","last_became_peered":"2024-01-11T09:46:25.950791+0000","last_unstale":"2024-01-11T09:46:38.984370+0000","last_undegraded":"2024-01-11T09:46:38.984370+0000","last_fullsized":"2024-01-11T09:46:38.984370+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:09:35.408321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"234'2","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.982795+0000","last_change":"2024-01-11T09:46:25.234950+0000","last_active":"2024-01-11T09:46:38.982795+0000","last_peered":"2024-01-11T09:46:38.982795+0000","last_clean":"2024-01-11T09:46:38.982795+0000","last_became_active":"2024-01-11T09:46:25.234510+0000","last_became_peered":"2024-01-11T09:46:25.234510+0000","last_unstale":"2024-01-11T09:46:38.982795+0000","last_undegraded":"2024-01-11T09:46:38.982795+0000","last_fullsized":"2024-01-11T09:46:38.982795+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:33:13.743309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.002994+0000","last_change":"2024-01-11T09:46:40.002994+0000","last_active":"2024-01-11T09:46:40.002994+0000","last_peered":"2024-01-11T09:46:40.002994+0000","last_clean":"2024-01-11T09:46:40.002994+0000","last_became_active":"2024-01-11T09:46:40.002700+0000","last_became_peered":"2024-01-11T09:46:40.002700+0000","last_unstale":"2024-01-11T09:46:40.002994+0000","last_undegraded":"2024-01-11T09:46:40.002994+0000","last_fullsized":"2024-01-11T09:46:40.002994+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:54.905740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.982536+0000","last_change":"2024-01-11T09:46:25.961077+0000","last_active":"2024-01-11T09:46:38.982536+0000","last_peered":"2024-01-11T09:46:38.982536+0000","last_clean":"2024-01-11T09:46:38.982536+0000","last_became_active":"2024-01-11T09:46:25.960690+0000","last_became_peered":"2024-01-11T09:46:25.960690+0000","last_unstale":"2024-01-11T09:46:38.982536+0000","last_undegraded":"2024-01-11T09:46:38.982536+0000","last_fullsized":"2024-01-11T09:46:38.982536+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:59:52.045856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"234'1","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.982750+0000","last_change":"2024-01-11T09:46:25.235005+0000","last_active":"2024-01-11T09:46:38.982750+0000","last_peered":"2024-01-11T09:46:38.982750+0000","last_clean":"2024-01-11T09:46:38.982750+0000","last_became_active":"2024-01-11T09:46:25.234748+0000","last_became_peered":"2024-01-11T09:46:25.234748+0000","last_unstale":"2024-01-11T09:46:38.982750+0000","last_undegraded":"2024-01-11T09:46:38.982750+0000","last_fullsized":"2024-01-11T09:46:38.982750+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:29:50.151952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.011411+0000","last_change":"2024-01-11T09:46:40.011411+0000","last_active":"2024-01-11T09:46:40.011411+0000","last_peered":"2024-01-11T09:46:40.011411+0000","last_clean":"2024-01-11T09:46:40.011411+0000","last_became_active":"2024-01-11T09:46:40.011120+0000","last_became_peered":"2024-01-11T09:46:40.011120+0000","last_unstale":"2024-01-11T09:46:40.011411+0000","last_undegraded":"2024-01-11T09:46:40.011411+0000","last_fullsized":"2024-01-11T09:46:40.011411+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:59:36.725768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986243+0000","last_change":"2024-01-11T09:46:25.952457+0000","last_active":"2024-01-11T09:46:38.986243+0000","last_peered":"2024-01-11T09:46:38.986243+0000","last_clean":"2024-01-11T09:46:38.986243+0000","last_became_active":"2024-01-11T09:46:25.948394+0000","last_became_peered":"2024-01-11T09:46:25.948394+0000","last_unstale":"2024-01-11T09:46:38.986243+0000","last_undegraded":"2024-01-11T09:46:38.986243+0000","last_fullsized":"2024-01-11T09:46:38.986243+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:49:33.997022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.977796+0000","last_change":"2024-01-11T09:46:25.242589+0000","last_active":"2024-01-11T09:46:38.977796+0000","last_peered":"2024-01-11T09:46:38.977796+0000","last_clean":"2024-01-11T09:46:38.977796+0000","last_became_active":"2024-01-11T09:46:25.241879+0000","last_became_peered":"2024-01-11T09:46:25.241879+0000","last_unstale":"2024-01-11T09:46:38.977796+0000","last_undegraded":"2024-01-11T09:46:38.977796+0000","last_fullsized":"2024-01-11T09:46:38.977796+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:11:40.145683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.998699+0000","last_change":"2024-01-11T09:46:39.998699+0000","last_active":"2024-01-11T09:46:39.998699+0000","last_peered":"2024-01-11T09:46:39.998699+0000","last_clean":"2024-01-11T09:46:39.998699+0000","last_became_active":"2024-01-11T09:46:39.998426+0000","last_became_peered":"2024-01-11T09:46:39.998426+0000","last_unstale":"2024-01-11T09:46:39.998699+0000","last_undegraded":"2024-01-11T09:46:39.998699+0000","last_fullsized":"2024-01-11T09:46:39.998699+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:26:11.710312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986453+0000","last_change":"2024-01-11T09:46:25.953778+0000","last_active":"2024-01-11T09:46:38.986453+0000","last_peered":"2024-01-11T09:46:38.986453+0000","last_clean":"2024-01-11T09:46:38.986453+0000","last_became_active":"2024-01-11T09:46:25.953471+0000","last_became_peered":"2024-01-11T09:46:25.953471+0000","last_unstale":"2024-01-11T09:46:38.986453+0000","last_undegraded":"2024-01-11T09:46:38.986453+0000","last_fullsized":"2024-01-11T09:46:38.986453+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:30:20.657242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"234'2","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.982618+0000","last_change":"2024-01-11T09:46:25.236418+0000","last_active":"2024-01-11T09:46:38.982618+0000","last_peered":"2024-01-11T09:46:38.982618+0000","last_clean":"2024-01-11T09:46:38.982618+0000","last_became_active":"2024-01-11T09:46:25.236155+0000","last_became_peered":"2024-01-11T09:46:25.236155+0000","last_unstale":"2024-01-11T09:46:38.982618+0000","last_undegraded":"2024-01-11T09:46:38.982618+0000","last_fullsized":"2024-01-11T09:46:38.982618+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:59:13.998537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.018015+0000","last_change":"2024-01-11T09:46:40.018015+0000","last_active":"2024-01-11T09:46:40.018015+0000","last_peered":"2024-01-11T09:46:40.018015+0000","last_clean":"2024-01-11T09:46:40.018015+0000","last_became_active":"2024-01-11T09:46:40.017803+0000","last_became_peered":"2024-01-11T09:46:40.017803+0000","last_unstale":"2024-01-11T09:46:40.018015+0000","last_undegraded":"2024-01-11T09:46:40.018015+0000","last_fullsized":"2024-01-11T09:46:40.018015+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:16:06.119899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.989798+0000","last_change":"2024-01-11T09:46:25.954393+0000","last_active":"2024-01-11T09:46:38.989798+0000","last_peered":"2024-01-11T09:46:38.989798+0000","last_clean":"2024-01-11T09:46:38.989798+0000","last_became_active":"2024-01-11T09:46:25.954139+0000","last_became_peered":"2024-01-11T09:46:25.954139+0000","last_unstale":"2024-01-11T09:46:38.989798+0000","last_undegraded":"2024-01-11T09:46:38.989798+0000","last_fullsized":"2024-01-11T09:46:38.989798+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:33:14.752455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.977474+0000","last_change":"2024-01-11T09:46:25.243586+0000","last_active":"2024-01-11T09:46:38.977474+0000","last_peered":"2024-01-11T09:46:38.977474+0000","last_clean":"2024-01-11T09:46:38.977474+0000","last_became_active":"2024-01-11T09:46:25.243184+0000","last_became_peered":"2024-01-11T09:46:25.243184+0000","last_unstale":"2024-01-11T09:46:38.977474+0000","last_undegraded":"2024-01-11T09:46:38.977474+0000","last_fullsized":"2024-01-11T09:46:38.977474+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:01:24.091035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.002684+0000","last_change":"2024-01-11T09:46:40.002684+0000","last_active":"2024-01-11T09:46:40.002684+0000","last_peered":"2024-01-11T09:46:40.002684+0000","last_clean":"2024-01-11T09:46:40.002684+0000","last_became_active":"2024-01-11T09:46:40.002048+0000","last_became_peered":"2024-01-11T09:46:40.002048+0000","last_unstale":"2024-01-11T09:46:40.002684+0000","last_undegraded":"2024-01-11T09:46:40.002684+0000","last_fullsized":"2024-01-11T09:46:40.002684+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:58:47.077944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.977074+0000","last_change":"2024-01-11T09:46:25.953319+0000","last_active":"2024-01-11T09:46:38.977074+0000","last_peered":"2024-01-11T09:46:38.977074+0000","last_clean":"2024-01-11T09:46:38.977074+0000","last_became_active":"2024-01-11T09:46:25.953063+0000","last_became_peered":"2024-01-11T09:46:25.953063+0000","last_unstale":"2024-01-11T09:46:38.977074+0000","last_undegraded":"2024-01-11T09:46:38.977074+0000","last_fullsized":"2024-01-11T09:46:38.977074+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:31:56.044420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980122+0000","last_change":"2024-01-11T09:46:25.243348+0000","last_active":"2024-01-11T09:46:38.980122+0000","last_peered":"2024-01-11T09:46:38.980122+0000","last_clean":"2024-01-11T09:46:38.980122+0000","last_became_active":"2024-01-11T09:46:25.243043+0000","last_became_peered":"2024-01-11T09:46:25.243043+0000","last_unstale":"2024-01-11T09:46:38.980122+0000","last_undegraded":"2024-01-11T09:46:38.980122+0000","last_fullsized":"2024-01-11T09:46:38.980122+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:13:10.353715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.998085+0000","last_change":"2024-01-11T09:46:39.998085+0000","last_active":"2024-01-11T09:46:39.998085+0000","last_peered":"2024-01-11T09:46:39.998085+0000","last_clean":"2024-01-11T09:46:39.998085+0000","last_became_active":"2024-01-11T09:46:39.994400+0000","last_became_peered":"2024-01-11T09:46:39.994400+0000","last_unstale":"2024-01-11T09:46:39.998085+0000","last_undegraded":"2024-01-11T09:46:39.998085+0000","last_fullsized":"2024-01-11T09:46:39.998085+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:31:00.593411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986541+0000","last_change":"2024-01-11T09:46:25.956539+0000","last_active":"2024-01-11T09:46:38.986541+0000","last_peered":"2024-01-11T09:46:38.986541+0000","last_clean":"2024-01-11T09:46:38.986541+0000","last_became_active":"2024-01-11T09:46:25.953109+0000","last_became_peered":"2024-01-11T09:46:25.953109+0000","last_unstale":"2024-01-11T09:46:38.986541+0000","last_undegraded":"2024-01-11T09:46:38.986541+0000","last_fullsized":"2024-01-11T09:46:38.986541+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:39:37.637123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980073+0000","last_change":"2024-01-11T09:46:25.234212+0000","last_active":"2024-01-11T09:46:38.980073+0000","last_peered":"2024-01-11T09:46:38.980073+0000","last_clean":"2024-01-11T09:46:38.980073+0000","last_became_active":"2024-01-11T09:46:25.233918+0000","last_became_peered":"2024-01-11T09:46:25.233918+0000","last_unstale":"2024-01-11T09:46:38.980073+0000","last_undegraded":"2024-01-11T09:46:38.980073+0000","last_fullsized":"2024-01-11T09:46:38.980073+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:39:17.328191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.001121+0000","last_change":"2024-01-11T09:46:40.001121+0000","last_active":"2024-01-11T09:46:40.001121+0000","last_peered":"2024-01-11T09:46:40.001121+0000","last_clean":"2024-01-11T09:46:40.001121+0000","last_became_active":"2024-01-11T09:46:40.000620+0000","last_became_peered":"2024-01-11T09:46:40.000620+0000","last_unstale":"2024-01-11T09:46:40.001121+0000","last_undegraded":"2024-01-11T09:46:40.001121+0000","last_fullsized":"2024-01-11T09:46:40.001121+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:40:21.583291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.981029+0000","last_change":"2024-01-11T09:46:25.953810+0000","last_active":"2024-01-11T09:46:38.981029+0000","last_peered":"2024-01-11T09:46:38.981029+0000","last_clean":"2024-01-11T09:46:38.981029+0000","last_became_active":"2024-01-11T09:46:25.953201+0000","last_became_peered":"2024-01-11T09:46:25.953201+0000","last_unstale":"2024-01-11T09:46:38.981029+0000","last_undegraded":"2024-01-11T09:46:38.981029+0000","last_fullsized":"2024-01-11T09:46:38.981029+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:43:25.226766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"234'2","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.995159+0000","last_change":"2024-01-11T09:46:25.245359+0000","last_active":"2024-01-11T09:46:38.995159+0000","last_peered":"2024-01-11T09:46:38.995159+0000","last_clean":"2024-01-11T09:46:38.995159+0000","last_became_active":"2024-01-11T09:46:25.244994+0000","last_became_peered":"2024-01-11T09:46:25.244994+0000","last_unstale":"2024-01-11T09:46:38.995159+0000","last_undegraded":"2024-01-11T09:46:38.995159+0000","last_fullsized":"2024-01-11T09:46:38.995159+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:42:13.269515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.994498+0000","last_change":"2024-01-11T09:46:39.994498+0000","last_active":"2024-01-11T09:46:39.994498+0000","last_peered":"2024-01-11T09:46:39.994498+0000","last_clean":"2024-01-11T09:46:39.994498+0000","last_became_active":"2024-01-11T09:46:39.994203+0000","last_became_peered":"2024-01-11T09:46:39.994203+0000","last_unstale":"2024-01-11T09:46:39.994498+0000","last_undegraded":"2024-01-11T09:46:39.994498+0000","last_fullsized":"2024-01-11T09:46:39.994498+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:50:15.419725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.994342+0000","last_change":"2024-01-11T09:46:25.952007+0000","last_active":"2024-01-11T09:46:38.994342+0000","last_peered":"2024-01-11T09:46:38.994342+0000","last_clean":"2024-01-11T09:46:38.994342+0000","last_became_active":"2024-01-11T09:46:25.951401+0000","last_became_peered":"2024-01-11T09:46:25.951401+0000","last_unstale":"2024-01-11T09:46:38.994342+0000","last_undegraded":"2024-01-11T09:46:38.994342+0000","last_fullsized":"2024-01-11T09:46:38.994342+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:29:18.608960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"235'1","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990258+0000","last_change":"2024-01-11T09:46:25.236787+0000","last_active":"2024-01-11T09:46:38.990258+0000","last_peered":"2024-01-11T09:46:38.990258+0000","last_clean":"2024-01-11T09:46:38.990258+0000","last_became_active":"2024-01-11T09:46:25.236503+0000","last_became_peered":"2024-01-11T09:46:25.236503+0000","last_unstale":"2024-01-11T09:46:38.990258+0000","last_undegraded":"2024-01-11T09:46:38.990258+0000","last_fullsized":"2024-01-11T09:46:38.990258+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:40:17.481276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.017366+0000","last_change":"2024-01-11T09:46:40.017366+0000","last_active":"2024-01-11T09:46:40.017366+0000","last_peered":"2024-01-11T09:46:40.017366+0000","last_clean":"2024-01-11T09:46:40.017366+0000","last_became_active":"2024-01-11T09:46:40.016774+0000","last_became_peered":"2024-01-11T09:46:40.016774+0000","last_unstale":"2024-01-11T09:46:40.017366+0000","last_undegraded":"2024-01-11T09:46:40.017366+0000","last_fullsized":"2024-01-11T09:46:40.017366+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:08:22.898981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.977786+0000","last_change":"2024-01-11T09:46:25.953624+0000","last_active":"2024-01-11T09:46:38.977786+0000","last_peered":"2024-01-11T09:46:38.977786+0000","last_clean":"2024-01-11T09:46:38.977786+0000","last_became_active":"2024-01-11T09:46:25.953347+0000","last_became_peered":"2024-01-11T09:46:25.953347+0000","last_unstale":"2024-01-11T09:46:38.977786+0000","last_undegraded":"2024-01-11T09:46:38.977786+0000","last_fullsized":"2024-01-11T09:46:38.977786+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:56:15.022301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.982895+0000","last_change":"2024-01-11T09:46:25.238938+0000","last_active":"2024-01-11T09:46:38.982895+0000","last_peered":"2024-01-11T09:46:38.982895+0000","last_clean":"2024-01-11T09:46:38.982895+0000","last_became_active":"2024-01-11T09:46:25.238214+0000","last_became_peered":"2024-01-11T09:46:25.238214+0000","last_unstale":"2024-01-11T09:46:38.982895+0000","last_undegraded":"2024-01-11T09:46:38.982895+0000","last_fullsized":"2024-01-11T09:46:38.982895+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:30:58.825981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.993325+0000","last_change":"2024-01-11T09:46:39.993325+0000","last_active":"2024-01-11T09:46:39.993325+0000","last_peered":"2024-01-11T09:46:39.993325+0000","last_clean":"2024-01-11T09:46:39.993325+0000","last_became_active":"2024-01-11T09:46:39.993033+0000","last_became_peered":"2024-01-11T09:46:39.993033+0000","last_unstale":"2024-01-11T09:46:39.993325+0000","last_undegraded":"2024-01-11T09:46:39.993325+0000","last_fullsized":"2024-01-11T09:46:39.993325+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:06:12.594569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.978097+0000","last_change":"2024-01-11T09:46:25.954867+0000","last_active":"2024-01-11T09:46:38.978097+0000","last_peered":"2024-01-11T09:46:38.978097+0000","last_clean":"2024-01-11T09:46:38.978097+0000","last_became_active":"2024-01-11T09:46:25.954613+0000","last_became_peered":"2024-01-11T09:46:25.954613+0000","last_unstale":"2024-01-11T09:46:38.978097+0000","last_undegraded":"2024-01-11T09:46:38.978097+0000","last_fullsized":"2024-01-11T09:46:38.978097+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:11:07.968385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.978067+0000","last_change":"2024-01-11T09:46:25.244239+0000","last_active":"2024-01-11T09:46:38.978067+0000","last_peered":"2024-01-11T09:46:38.978067+0000","last_clean":"2024-01-11T09:46:38.978067+0000","last_became_active":"2024-01-11T09:46:25.243663+0000","last_became_peered":"2024-01-11T09:46:25.243663+0000","last_unstale":"2024-01-11T09:46:38.978067+0000","last_undegraded":"2024-01-11T09:46:38.978067+0000","last_fullsized":"2024-01-11T09:46:38.978067+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:23:51.951171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.997383+0000","last_change":"2024-01-11T09:46:39.997383+0000","last_active":"2024-01-11T09:46:39.997383+0000","last_peered":"2024-01-11T09:46:39.997383+0000","last_clean":"2024-01-11T09:46:39.997383+0000","last_became_active":"2024-01-11T09:46:39.995199+0000","last_became_peered":"2024-01-11T09:46:39.995199+0000","last_unstale":"2024-01-11T09:46:39.997383+0000","last_undegraded":"2024-01-11T09:46:39.997383+0000","last_fullsized":"2024-01-11T09:46:39.997383+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:59:28.758593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984067+0000","last_change":"2024-01-11T09:46:25.962471+0000","last_active":"2024-01-11T09:46:38.984067+0000","last_peered":"2024-01-11T09:46:38.984067+0000","last_clean":"2024-01-11T09:46:38.984067+0000","last_became_active":"2024-01-11T09:46:25.959247+0000","last_became_peered":"2024-01-11T09:46:25.959247+0000","last_unstale":"2024-01-11T09:46:38.984067+0000","last_undegraded":"2024-01-11T09:46:38.984067+0000","last_fullsized":"2024-01-11T09:46:38.984067+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:31:56.044420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980624+0000","last_change":"2024-01-11T09:46:25.242486+0000","last_active":"2024-01-11T09:46:38.980624+0000","last_peered":"2024-01-11T09:46:38.980624+0000","last_clean":"2024-01-11T09:46:38.980624+0000","last_became_active":"2024-01-11T09:46:25.242231+0000","last_became_peered":"2024-01-11T09:46:25.242231+0000","last_unstale":"2024-01-11T09:46:38.980624+0000","last_undegraded":"2024-01-11T09:46:38.980624+0000","last_fullsized":"2024-01-11T09:46:38.980624+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:23:51.951171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:40.020863+0000","last_change":"2024-01-11T09:46:40.020863+0000","last_active":"2024-01-11T09:46:40.020863+0000","last_peered":"2024-01-11T09:46:40.020863+0000","last_clean":"2024-01-11T09:46:40.020863+0000","last_became_active":"2024-01-11T09:46:40.020559+0000","last_became_peered":"2024-01-11T09:46:40.020559+0000","last_unstale":"2024-01-11T09:46:40.020863+0000","last_undegraded":"2024-01-11T09:46:40.020863+0000","last_fullsized":"2024-01-11T09:46:40.020863+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:05:23.876325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986674+0000","last_change":"2024-01-11T09:46:25.954929+0000","last_active":"2024-01-11T09:46:38.986674+0000","last_peered":"2024-01-11T09:46:38.986674+0000","last_clean":"2024-01-11T09:46:38.986674+0000","last_became_active":"2024-01-11T09:46:25.951640+0000","last_became_peered":"2024-01-11T09:46:25.951640+0000","last_unstale":"2024-01-11T09:46:38.986674+0000","last_undegraded":"2024-01-11T09:46:38.986674+0000","last_fullsized":"2024-01-11T09:46:38.986674+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:28:55.053850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.991096+0000","last_change":"2024-01-11T09:46:25.234604+0000","last_active":"2024-01-11T09:46:38.991096+0000","last_peered":"2024-01-11T09:46:38.991096+0000","last_clean":"2024-01-11T09:46:38.991096+0000","last_became_active":"2024-01-11T09:46:25.234348+0000","last_became_peered":"2024-01-11T09:46:25.234348+0000","last_unstale":"2024-01-11T09:46:38.991096+0000","last_undegraded":"2024-01-11T09:46:38.991096+0000","last_fullsized":"2024-01-11T09:46:38.991096+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:01:05.259333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.999682+0000","last_change":"2024-01-11T09:46:39.999682+0000","last_active":"2024-01-11T09:46:39.999682+0000","last_peered":"2024-01-11T09:46:39.999682+0000","last_clean":"2024-01-11T09:46:39.999682+0000","last_became_active":"2024-01-11T09:46:39.999423+0000","last_became_peered":"2024-01-11T09:46:39.999423+0000","last_unstale":"2024-01-11T09:46:39.999682+0000","last_undegraded":"2024-01-11T09:46:39.999682+0000","last_fullsized":"2024-01-11T09:46:39.999682+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:48:30.105547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.986233+0000","last_change":"2024-01-11T09:46:25.956295+0000","last_active":"2024-01-11T09:46:38.986233+0000","last_peered":"2024-01-11T09:46:38.986233+0000","last_clean":"2024-01-11T09:46:38.986233+0000","last_became_active":"2024-01-11T09:46:25.949169+0000","last_became_peered":"2024-01-11T09:46:25.949169+0000","last_unstale":"2024-01-11T09:46:38.986233+0000","last_undegraded":"2024-01-11T09:46:38.986233+0000","last_fullsized":"2024-01-11T09:46:38.986233+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:43:07.160563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.980864+0000","last_change":"2024-01-11T09:46:25.231375+0000","last_active":"2024-01-11T09:46:38.980864+0000","last_peered":"2024-01-11T09:46:38.980864+0000","last_clean":"2024-01-11T09:46:38.980864+0000","last_became_active":"2024-01-11T09:46:25.231115+0000","last_became_peered":"2024-01-11T09:46:25.231115+0000","last_unstale":"2024-01-11T09:46:38.980864+0000","last_undegraded":"2024-01-11T09:46:38.980864+0000","last_fullsized":"2024-01-11T09:46:38.980864+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:06:11.124203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3f","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.994881+0000","last_change":"2024-01-11T09:46:39.994881+0000","last_active":"2024-01-11T09:46:39.994881+0000","last_peered":"2024-01-11T09:46:39.994881+0000","last_clean":"2024-01-11T09:46:39.994881+0000","last_became_active":"2024-01-11T09:46:39.993737+0000","last_became_peered":"2024-01-11T09:46:39.993737+0000","last_unstale":"2024-01-11T09:46:39.994881+0000","last_undegraded":"2024-01-11T09:46:39.994881+0000","last_fullsized":"2024-01-11T09:46:39.994881+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:48:18.089926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990906+0000","last_change":"2024-01-11T09:46:25.957641+0000","last_active":"2024-01-11T09:46:38.990906+0000","last_peered":"2024-01-11T09:46:38.990906+0000","last_clean":"2024-01-11T09:46:38.990906+0000","last_became_active":"2024-01-11T09:46:25.955768+0000","last_became_peered":"2024-01-11T09:46:25.955768+0000","last_unstale":"2024-01-11T09:46:38.990906+0000","last_undegraded":"2024-01-11T09:46:38.990906+0000","last_fullsized":"2024-01-11T09:46:38.990906+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:58.843659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":13,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.981087+0000","last_change":"2024-01-11T09:46:25.239061+0000","last_active":"2024-01-11T09:46:38.981087+0000","last_peered":"2024-01-11T09:46:38.981087+0000","last_clean":"2024-01-11T09:46:38.981087+0000","last_became_active":"2024-01-11T09:46:25.238802+0000","last_became_peered":"2024-01-11T09:46:25.238802+0000","last_unstale":"2024-01-11T09:46:38.981087+0000","last_undegraded":"2024-01-11T09:46:38.981087+0000","last_fullsized":"2024-01-11T09:46:38.981087+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:05:57.834513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":8,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-01-11T09:46:39.996552+0000","last_change":"2024-01-11T09:46:39.996552+0000","last_active":"2024-01-11T09:46:39.996552+0000","last_peered":"2024-01-11T09:46:39.996552+0000","last_clean":"2024-01-11T09:46:39.996552+0000","last_became_active":"2024-01-11T09:46:39.996063+0000","last_became_peered":"2024-01-11T09:46:39.996063+0000","last_unstale":"2024-01-11T09:46:39.996552+0000","last_undegraded":"2024-01-11T09:46:39.996552+0000","last_fullsized":"2024-01-11T09:46:39.996552+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:38.963058+0000","last_clean_scrub_stamp":"2024-01-11T09:46:38.963058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:19:31.710935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.990166+0000","last_change":"2024-01-11T09:46:25.948656+0000","last_active":"2024-01-11T09:46:38.990166+0000","last_peered":"2024-01-11T09:46:38.990166+0000","last_clean":"2024-01-11T09:46:38.990166+0000","last_became_active":"2024-01-11T09:46:25.948273+0000","last_became_peered":"2024-01-11T09:46:25.948273+0000","last_unstale":"2024-01-11T09:46:38.990166+0000","last_undegraded":"2024-01-11T09:46:38.990166+0000","last_fullsized":"2024-01-11T09:46:38.990166+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:25.212148+0000","last_clean_scrub_stamp":"2024-01-11T09:46:25.212148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:36:07.163334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"234'4","reported_seq":17,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.984460+0000","last_change":"2024-01-11T09:46:25.235720+0000","last_active":"2024-01-11T09:46:38.984460+0000","last_peered":"2024-01-11T09:46:38.984460+0000","last_clean":"2024-01-11T09:46:38.984460+0000","last_became_active":"2024-01-11T09:46:25.235445+0000","last_became_peered":"2024-01-11T09:46:25.235445+0000","last_unstale":"2024-01-11T09:46:38.984460+0000","last_undegraded":"2024-01-11T09:46:38.984460+0000","last_fullsized":"2024-01-11T09:46:38.984460+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:46:24.203002+0000","last_clean_scrub_stamp":"2024-01-11T09:46:24.203002+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:53:59.525452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"221'388","reported_seq":1298,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-01-11T09:46:38.979950+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:46:38.979950+0000","last_peered":"2024-01-11T09:46:38.979950+0000","last_clean":"2024-01-11T09:46:38.979950+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:46:38.979950+0000","last_undegraded":"2024-01-11T09:46:38.979950+0000","last_fullsized":"2024-01-11T09:46:38.979950+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":651,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":3519,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2480,"num_read_kb":5220,"num_write":651,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":388,"ondisk_log_size":388,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575984,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300788,"kb_used_data":3500,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071052,"statfs":{"total":96636764160,"available":96328757248,"internally_reserved":0,"allocated":3584000,"data_stored":2028665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.68899999999999995,"15min":0.72799999999999998},"min":{"1min":0.39100000000000001,"5min":0.376,"15min":0.36599999999999999},"max":{"1min":0.89200000000000002,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.71499999999999997,"15min":0.74299999999999999},"min":{"1min":0.40400000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83799999999999997,"5min":1.0609999999999999,"15min":1.0669999999999999},"last":0.66700000000000004}]},{"osd":1,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.69799999999999995,"15min":0.72499999999999998},"min":{"1min":0.48099999999999998,"5min":0.41799999999999998,"15min":0.41599999999999998},"max":{"1min":0.82099999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.70499999999999996,"15min":0.73599999999999999},"min":{"1min":0.504,"5min":0.434,"15min":0.434},"max":{"1min":0.89400000000000002,"5min":0.999,"15min":1.0820000000000001},"last":0.53100000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.71799999999999997,"15min":0.73499999999999999},"min":{"1min":0.46899999999999997,"5min":0.443,"15min":0.443},"max":{"1min":0.875,"5min":0.94399999999999995,"15min":1.0600000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.71099999999999997,"15min":0.747},"min":{"1min":0.48499999999999999,"5min":0.44800000000000001,"15min":0.44500000000000001},"max":{"1min":0.86099999999999999,"5min":0.96999999999999997,"15min":1.153},"last":0.623}]},{"osd":3,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73599999999999999,"15min":0.76200000000000001},"min":{"1min":0.51500000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":1.0660000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.72299999999999998,"15min":0.747},"min":{"1min":0.51000000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.77100000000000002,"5min":1,"15min":1.0880000000000001},"last":0.59299999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61099999999999999,"15min":0.60599999999999998},"min":{"1min":0.45900000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79600000000000004,"5min":1.0860000000000001,"15min":1.107},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64000000000000001,"15min":0.63900000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39700000000000002},"max":{"1min":0.871,"5min":0.995,"15min":1.0960000000000001},"last":0.75}]},{"osd":5,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.622,"15min":0.64100000000000001},"min":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86099999999999999,"5min":0.86699999999999999,"15min":1.0109999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.69199999999999995,"15min":0.68799999999999994},"min":{"1min":0.41499999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.61099999999999999}]},{"osd":6,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.78000000000000003},"min":{"1min":0.56000000000000005,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.96899999999999997,"5min":1.232,"15min":1.232},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.002,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.78400000000000003}]}]},{"osd":6,"up_from":13,"seq":55834575983,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312500,"kb_used_data":2924,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059340,"statfs":{"total":96636764160,"available":96316764160,"internally_reserved":0,"allocated":2994176,"data_stored":1438803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.65200000000000002,"15min":0.70399999999999996},"min":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89100000000000001,"5min":0.94199999999999995,"15min":0.999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.71299999999999997,"15min":0.73899999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.216},"last":0.65600000000000003}]},{"osd":1,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.67500000000000004,"15min":0.73199999999999998},"min":{"1min":0.44900000000000001,"5min":0.42799999999999999,"15min":0.41699999999999998},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":1.3129999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.68899999999999995,"15min":0.72499999999999998},"min":{"1min":0.44400000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.95499999999999996,"5min":0.98299999999999998,"15min":1.0840000000000001},"last":0.51900000000000002}]},{"osd":2,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.70699999999999996,"15min":0.74299999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.0069999999999999,"5min":1.034,"15min":1.4179999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.72099999999999997,"15min":0.75},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":0.46500000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.72999999999999998,"15min":0.754},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.92600000000000005,"5min":1.012,"15min":1.014},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.70599999999999996,"15min":0.76200000000000001},"min":{"1min":0.498,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.91300000000000003,"5min":0.95999999999999996,"15min":1.165},"last":0.64300000000000002}]},{"osd":4,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.38400000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78000000000000003,"5min":0.91700000000000004,"15min":1.0309999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.626,"15min":0.61599999999999999},"min":{"1min":0.40799999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92100000000000004,"5min":0.94799999999999995,"15min":1.1259999999999999},"last":0.56299999999999994}]},{"osd":5,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.65000000000000002,"15min":0.65900000000000003},"min":{"1min":0.44600000000000001,"5min":0.41499999999999998,"15min":0.40000000000000002},"max":{"1min":0.93100000000000005,"5min":0.97899999999999998,"15min":1.369},"last":0.441},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70299999999999996,"15min":0.70499999999999996},"min":{"1min":0.56799999999999995,"5min":0.46700000000000003,"15min":0.45400000000000001},"max":{"1min":0.85999999999999999,"5min":0.999,"15min":1.0960000000000001},"last":0.58499999999999996}]},{"osd":7,"last update":"Thu Jan 11 09:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.70699999999999996,"15min":0.70899999999999996},"min":{"1min":0.47199999999999998,"5min":0.45600000000000002,"15min":0.45100000000000001},"max":{"1min":0.91300000000000003,"5min":1.089,"15min":1.484},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.754,"15min":0.76700000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.98299999999999998,"5min":1.1379999999999999,"15min":1.603},"last":0.68600000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575985,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308416,"kb_used_data":2936,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063424,"statfs":{"total":96636764160,"available":96320946176,"internally_reserved":0,"allocated":3006464,"data_stored":1441100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.51400000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.315},"max":{"1min":0.69799999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.59899999999999998,"15min":0.61899999999999999},"min":{"1min":0.41899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.75600000000000001,"5min":0.85199999999999998,"15min":0.95099999999999996},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.50700000000000001,"15min":0.51200000000000001},"min":{"1min":0.34999999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66600000000000004,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.84099999999999997,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.38400000000000001}]},{"osd":3,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.46400000000000002,"5min":0.436,"15min":0.42699999999999999},"max":{"1min":0.85199999999999998,"5min":0.91600000000000004,"15min":0.96699999999999997},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.58099999999999996,"15min":0.58199999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.71299999999999997,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.48399999999999999}]},{"osd":4,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.79500000000000004,"15min":0.81599999999999995},"min":{"1min":0.54600000000000004,"5min":0.51400000000000001,"15min":0.505},"max":{"1min":0.875,"5min":1.214,"15min":1.2969999999999999},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.84399999999999997,"15min":0.85899999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1000000000000001,"5min":1.1879999999999999,"15min":1.1890000000000001},"last":0.81599999999999995}]},{"osd":5,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.82599999999999996,"15min":0.82799999999999996},"min":{"1min":0.51800000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.996,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.80800000000000005,"15min":0.80700000000000005},"min":{"1min":0.61899999999999999,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":1.0609999999999999,"5min":1.341,"15min":1.341},"last":0.746}]},{"osd":6,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.83499999999999996,"15min":0.84699999999999998},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.56799999999999995},"max":{"1min":1.0620000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.84499999999999997,"15min":0.85399999999999998},"min":{"1min":0.60099999999999998,"5min":0.56899999999999995,"15min":0.55900000000000005},"max":{"1min":1.077,"5min":1.177,"15min":1.177},"last":0.94199999999999995}]},{"osd":7,"last update":"Thu Jan 11 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.88800000000000001,"15min":0.88900000000000001},"min":{"1min":0.64400000000000002,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.1359999999999999,"5min":1.421,"15min":1.421},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.86299999999999999,"15min":0.85699999999999998},"min":{"1min":0.64500000000000002,"5min":0.60099999999999998,"15min":0.54500000000000004},"max":{"1min":0.96099999999999997,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.84699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575984,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308980,"kb_used_data":3500,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062860,"statfs":{"total":96636764160,"available":96320368640,"internally_reserved":0,"allocated":3584000,"data_stored":2028629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.50700000000000001,"15min":0.51800000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.70999999999999996,"5min":0.84699999999999998,"15min":0.875},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.58399999999999996,"15min":0.59599999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77200000000000002,"5min":0.81599999999999995,"15min":0.874},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.53500000000000003,"15min":0.56499999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.61499999999999999,"5min":0.83099999999999996,"15min":0.97499999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.56299999999999994,"15min":0.57999999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71899999999999997,"5min":0.84699999999999998,"15min":0.86399999999999999},"last":0.42999999999999999}]},{"osd":3,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.51400000000000001,"15min":0.54000000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.73699999999999999,"5min":0.79000000000000004,"15min":0.95399999999999996},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.621,"15min":0.64800000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80200000000000005,"5min":0.90300000000000002,"15min":1.0389999999999999},"last":0.55600000000000005}]},{"osd":4,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.81399999999999995,"15min":0.81599999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0629999999999999,"5min":1.1779999999999999,"15min":1.304},"last":0.746},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.80000000000000004,"15min":0.80600000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.93000000000000005,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.82999999999999996}]},{"osd":5,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.82299999999999995,"15min":0.83899999999999997},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0069999999999999,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.82899999999999996,"15min":0.84199999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.1100000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.89000000000000001}]},{"osd":6,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.81000000000000005,"15min":0.82399999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.1000000000000001,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.872},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.84199999999999997,"15min":0.86599999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":0.95899999999999996,"5min":1.256,"15min":1.256},"last":0.79000000000000004}]},{"osd":7,"last update":"Thu Jan 11 09:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.85599999999999998,"15min":0.86599999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.151,"5min":1.28,"15min":1.28},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.85899999999999999,"15min":0.872},"min":{"1min":0.57499999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.054,"5min":1.238,"15min":1.2410000000000001},"last":0.91500000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575983,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300208,"kb_used_data":2920,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071632,"statfs":{"total":96636764160,"available":96329351168,"internally_reserved":0,"allocated":2990080,"data_stored":1438317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48699999999999999,"15min":0.502},"min":{"1min":0.32800000000000001,"5min":0.29899999999999999,"15min":0.27900000000000003},"max":{"1min":0.625,"5min":0.72599999999999998,"15min":0.85399999999999998},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.56499999999999995},"min":{"1min":0.38800000000000001,"5min":0.34399999999999997,"15min":0.34300000000000003},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":1.6359999999999999},"last":0.48999999999999999}]},{"osd":1,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.39500000000000002,"5min":0.38200000000000001,"15min":0.33500000000000002},"max":{"1min":0.80900000000000005,"5min":0.88800000000000001,"15min":0.93000000000000005},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58899999999999997,"15min":0.59899999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.73099999999999998,"5min":0.97299999999999998,"15min":1.145},"last":0.41099999999999998}]},{"osd":3,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.75,"5min":0.83099999999999996,"15min":0.96399999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.63,"15min":0.65500000000000003},"min":{"1min":0.442,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.83599999999999997,"5min":0.90700000000000003,"15min":1.3620000000000001},"last":0.46800000000000003}]},{"osd":4,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.78200000000000003,"15min":0.79800000000000004},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.54300000000000004},"max":{"1min":0.95299999999999996,"5min":1.071,"15min":1.1519999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.78700000000000003,"15min":0.80000000000000004},"min":{"1min":0.54500000000000004,"5min":0.505,"15min":0.505},"max":{"1min":0.96299999999999997,"5min":1.161,"15min":1.161},"last":0.79700000000000004}]},{"osd":5,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.83099999999999996,"15min":0.86799999999999999},"min":{"1min":0.625,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.996,"5min":1.228,"15min":1.429},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.85099999999999998,"15min":0.86299999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.0389999999999999,"5min":1.1850000000000001,"15min":1.2170000000000001},"last":0.64900000000000002}]},{"osd":6,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.80000000000000004,"15min":0.82599999999999996},"min":{"1min":0.55500000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0049999999999999,"5min":1.1659999999999999,"15min":1.4139999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.83499999999999996,"15min":0.86399999999999999},"min":{"1min":0.63400000000000001,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.96599999999999997,"5min":1.2010000000000001,"15min":1.593},"last":0.76400000000000001}]},{"osd":7,"last update":"Thu Jan 11 09:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.86799999999999999,"15min":0.89200000000000002},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.57299999999999995},"max":{"1min":1.0129999999999999,"5min":1.137,"15min":2.0539999999999998},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.86499999999999999,"15min":0.88100000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.55500000000000005},"max":{"1min":1.069,"5min":1.2270000000000001,"15min":1.2490000000000001},"last":0.82699999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575983,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300208,"kb_used_data":2920,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071632,"statfs":{"total":96636764160,"available":96329351168,"internally_reserved":0,"allocated":2990080,"data_stored":1438781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.502,"15min":0.52000000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.314},"max":{"1min":0.76100000000000001,"5min":0.89700000000000002,"15min":0.90000000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.56200000000000006,"15min":0.58699999999999997},"min":{"1min":0.372,"5min":0.36899999999999999,"15min":0.36099999999999999},"max":{"1min":0.67900000000000005,"5min":1.014,"15min":1.0449999999999999},"last":0.47799999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.495,"15min":0.51100000000000001},"min":{"1min":0.29399999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63500000000000001,"5min":0.84199999999999997,"15min":1.304},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.38700000000000001},"max":{"1min":0.75,"5min":0.84099999999999997,"15min":1.4590000000000001},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58299999999999996,"15min":0.60599999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.72999999999999998,"5min":0.93700000000000006,"15min":1.04},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.64200000000000002,"15min":0.65800000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44700000000000001},"max":{"1min":0.83499999999999996,"5min":0.89800000000000002,"15min":0.92000000000000004},"last":0.45200000000000001}]},{"osd":4,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76100000000000001,"15min":0.79000000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.92900000000000005,"5min":1.206,"15min":1.206},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.77800000000000002,"15min":0.81299999999999994},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.871,"5min":1.171,"15min":1.1830000000000001},"last":0.76300000000000001}]},{"osd":5,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.79600000000000004,"15min":0.82099999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.95399999999999996,"5min":1.1619999999999999,"15min":1.234},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81000000000000005,"15min":0.84799999999999998},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.3440000000000001},"last":0.77500000000000002}]},{"osd":6,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82999999999999996,"15min":0.85899999999999999},"min":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.077,"5min":1.1739999999999999,"15min":1.2170000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.80500000000000005,"15min":0.82899999999999996},"min":{"1min":0.55300000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.97099999999999997,"5min":1.181,"15min":1.2190000000000001},"last":0.78800000000000003}]},{"osd":7,"last update":"Thu Jan 11 09:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.83699999999999997,"15min":0.86799999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0269999999999999,"5min":1.224,"15min":1.264},"last":0.749},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.86099999999999999,"15min":0.89500000000000002},"min":{"1min":0.59299999999999997,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.0209999999999999,"5min":1.1539999999999999,"15min":1.24},"last":0.66700000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575984,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308416,"kb_used_data":2936,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063424,"statfs":{"total":96636764160,"available":96320946176,"internally_reserved":0,"allocated":3006464,"data_stored":1441068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.72299999999999998,"15min":0.76000000000000001},"min":{"1min":0.42099999999999999,"5min":0.41199999999999998,"15min":0.39900000000000002},"max":{"1min":0.85799999999999998,"5min":0.97799999999999998,"15min":1.119},"last":0.747},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.73799999999999999,"15min":0.76700000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86399999999999999,"5min":1.853,"15min":1.853},"last":0.72799999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.69899999999999995,"15min":0.73499999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.83599999999999997,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.69799999999999995,"15min":0.72699999999999998},"min":{"1min":0.438,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81399999999999995,"5min":1.639,"15min":1.639},"last":0.84299999999999997}]},{"osd":2,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.71999999999999997,"15min":0.746},"min":{"1min":0.50600000000000001,"5min":0.46500000000000002,"15min":0.44700000000000001},"max":{"1min":0.76400000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.71999999999999997,"15min":0.74199999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.93300000000000005,"5min":1.0249999999999999,"15min":1.0700000000000001},"last":0.78200000000000003}]},{"osd":3,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.70199999999999996,"15min":0.73299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.75900000000000001,"15min":0.77300000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.97499999999999998,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.82699999999999996}]},{"osd":5,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63500000000000001,"15min":0.623},"min":{"1min":0.41999999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.89900000000000002,"5min":0.91500000000000004,"15min":0.97199999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.66700000000000004,"15min":0.64700000000000002},"min":{"1min":0.45900000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95099999999999996,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.53300000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.68999999999999995,"15min":0.67300000000000004},"min":{"1min":0.441,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.012,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.71899999999999997,"15min":0.70499999999999996},"min":{"1min":0.46000000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0609999999999999,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.81100000000000005}]},{"osd":7,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.72899999999999998,"15min":0.70199999999999996},"min":{"1min":0.495,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.075,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.77900000000000003,"15min":0.754},"min":{"1min":0.56699999999999995,"5min":0.436,"15min":0.41699999999999998},"max":{"1min":1.0409999999999999,"5min":2.052,"15min":2.052},"last":0.86899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575984,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312492,"kb_used_data":2916,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059348,"statfs":{"total":96636764160,"available":96316772352,"internally_reserved":0,"allocated":2985984,"data_stored":1438227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.71699999999999997,"15min":0.72899999999999998},"min":{"1min":0.46800000000000003,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71099999999999997,"15min":0.72399999999999998},"min":{"1min":0.51000000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.66000000000000003}]},{"osd":1,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71499999999999997,"15min":0.73399999999999999},"min":{"1min":0.46800000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91400000000000003,"5min":0.95699999999999996,"15min":0.97699999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.73299999999999998,"15min":0.73799999999999999},"min":{"1min":0.51900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.874,"5min":1.004,"15min":1.0269999999999999},"last":0.63700000000000001}]},{"osd":2,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.72299999999999998,"15min":0.72999999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1,"5min":1.0269999999999999,"15min":1.028},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.73199999999999998,"15min":0.74199999999999999},"min":{"1min":0.50900000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.98799999999999999},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.74399999999999999,"15min":0.75600000000000001},"min":{"1min":0.56200000000000006,"5min":0.52800000000000002,"15min":0.49299999999999999},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76100000000000001,"15min":0.77000000000000002},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.63300000000000001,"15min":0.61399999999999999},"min":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41699999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.64300000000000002,"15min":0.63400000000000001},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.375},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.44}]},{"osd":6,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63700000000000001,"15min":0.64300000000000002},"min":{"1min":0.433,"5min":0.42099999999999999,"15min":0.40999999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":1.0089999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69199999999999995,"15min":0.68600000000000005},"min":{"1min":0.48699999999999999,"5min":0.46200000000000002,"15min":0.42799999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.38500000000000001}]},{"osd":7,"last update":"Thu Jan 11 09:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.73199999999999998,"15min":0.73299999999999998},"min":{"1min":0.58299999999999996,"5min":0.47299999999999998,"15min":0.434},"max":{"1min":0.94799999999999995,"5min":1.0149999999999999,"15min":1.0760000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.77800000000000002,"15min":0.77600000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.47599999999999998},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.623}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:46:44.910 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-11T09:46:44.910 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:46:45.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.152+0000 7f1b739ff700 1 Processor -- start 2024-01-11T09:46:45.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.152+0000 7f1b739ff700 1 -- start start 2024-01-11T09:46:45.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.153+0000 7f1b739ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1b6c0b2b90 0x7f1b6c159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:45.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.153+0000 7f1b739ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1b6c0b07b0 0x7f1b6c159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:45.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.153+0000 7f1b739ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6c0cd730 con 0x7f1b6c0b03e0 2024-01-11T09:46:45.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.153+0000 7f1b739ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b6c0ccb20 con 0x7f1b6c0b07b0 2024-01-11T09:46:45.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.153+0000 7f1b739ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b6c0cce00 con 0x7f1b6c0b2b90 2024-01-11T09:46:45.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.154+0000 7f1b71f9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1b6c0b07b0 0x7f1b6c159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:45.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.154+0000 7f1b71f9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1b6c0b07b0 0x7f1b6c159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52560/0 (socket says 172.21.15.27:52560) 2024-01-11T09:46:45.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.154+0000 7f1b71f9c700 1 -- 172.21.15.27:0/3788181619 learned_addr learned my addr 172.21.15.27:0/3788181619 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:45.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.154+0000 7f1b70f9a700 1 --2- 172.21.15.27:0/3788181619 >> v2:172.21.15.77:3300/0 conn(0x7f1b6c0b2b90 0x7f1b6c159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:45.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.154+0000 7f1b71f9c700 1 -- 172.21.15.27:0/3788181619 >> v2:172.21.15.77:3300/0 conn(0x7f1b6c0b2b90 msgr2=0x7f1b6c159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:45.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.154+0000 7f1b71f9c700 1 --2- 172.21.15.27:0/3788181619 >> v2:172.21.15.77:3300/0 conn(0x7f1b6c0b2b90 0x7f1b6c159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:45.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.154+0000 7f1b71f9c700 1 -- 172.21.15.27:0/3788181619 >> v1:172.21.15.77:6789/0 conn(0x7f1b6c0b03e0 legacy=0x7f1b6c0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:45.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.154+0000 7f1b71f9c700 1 -- 172.21.15.27:0/3788181619 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6c0cd0e0 con 0x7f1b6c0b07b0 2024-01-11T09:46:45.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.154+0000 7f1b70f9a700 1 --2- 172.21.15.27:0/3788181619 >> v2:172.21.15.77:3300/0 conn(0x7f1b6c0b2b90 0x7f1b6c159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:45.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.154+0000 7f1b71f9c700 1 --2- 172.21.15.27:0/3788181619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b6c0b07b0 0x7f1b6c159740 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0cdfc0 tx=0x7f1b64009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fc92074bbbfb37d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.155+0000 7f1b63fff700 1 -- 172.21.15.27:0/3788181619 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6400e070 con 0x7f1b6c0b07b0 2024-01-11T09:46:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.155+0000 7f1b63fff700 1 -- 172.21.15.27:0/3788181619 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6400ba50 con 0x7f1b6c0b07b0 2024-01-11T09:46:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.155+0000 7f1b63fff700 1 -- 172.21.15.27:0/3788181619 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b640092e0 con 0x7f1b6c0b07b0 2024-01-11T09:46:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.155+0000 7f1b739ff700 1 -- 172.21.15.27:0/3788181619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b6c0b07b0 msgr2=0x7f1b6c159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.155+0000 7f1b739ff700 1 --2- 172.21.15.27:0/3788181619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b6c0b07b0 0x7f1b6c159740 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0cdfc0 tx=0x7f1b64009840 comp rx=0 tx=0).stop 2024-01-11T09:46:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.155+0000 7f1b739ff700 1 -- 172.21.15.27:0/3788181619 shutdown_connections 2024-01-11T09:46:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.155+0000 7f1b739ff700 1 --2- 172.21.15.27:0/3788181619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b6c0b07b0 0x7f1b6c159740 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.155+0000 7f1b739ff700 1 --2- 172.21.15.27:0/3788181619 >> v2:172.21.15.77:3300/0 conn(0x7f1b6c0b2b90 0x7f1b6c159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.155+0000 7f1b739ff700 1 -- 172.21.15.27:0/3788181619 >> 172.21.15.27:0/3788181619 conn(0x7f1b6c06ed40 msgr2=0x7f1b6c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.155+0000 7f1b739ff700 1 -- 172.21.15.27:0/3788181619 shutdown_connections 2024-01-11T09:46:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.155+0000 7f1b739ff700 1 -- 172.21.15.27:0/3788181619 wait complete. 2024-01-11T09:46:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.157+0000 7f1b739ff700 1 Processor -- start 2024-01-11T09:46:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.157+0000 7f1b739ff700 1 -- start start 2024-01-11T09:46:45.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.158+0000 7f1b739ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1b6c0b03e0 0x7f1b6c158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:45.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.158+0000 7f1b739ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b6c0b07b0 0x7f1b6c0cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:45.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.158+0000 7f1b739ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b6c0b2b90 0x7f1b6c0cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:45.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.158+0000 7f1b739ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c0c4870 con 0x7f1b6c0b07b0 2024-01-11T09:46:45.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.158+0000 7f1b739ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c0c93c0 con 0x7f1b6c0b2b90 2024-01-11T09:46:45.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.158+0000 7f1b739ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c0c3f40 con 0x7f1b6c0b03e0 2024-01-11T09:46:45.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.158+0000 7f1b70f9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b6c0b07b0 0x7f1b6c0cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:45.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.158+0000 7f1b70f9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b6c0b07b0 0x7f1b6c0cb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52572/0 (socket says 172.21.15.27:52572) 2024-01-11T09:46:45.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.159+0000 7f1b70f9a700 1 -- 172.21.15.27:0/529541473 learned_addr learned my addr 172.21.15.27:0/529541473 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:45.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.159+0000 7f1b71f9c700 1 --2- 172.21.15.27:0/529541473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b6c0b2b90 0x7f1b6c0cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:45.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.159+0000 7f1b7179b700 1 --2- 172.21.15.27:0/529541473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1b6c0b03e0 0x7f1b6c158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:45.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.159+0000 7f1b70f9a700 1 -- 172.21.15.27:0/529541473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1b6c0b03e0 msgr2=0x7f1b6c158ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:45.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.159+0000 7f1b70f9a700 1 --2- 172.21.15.27:0/529541473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1b6c0b03e0 0x7f1b6c158ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:45.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.159+0000 7f1b70f9a700 1 -- 172.21.15.27:0/529541473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b6c0b2b90 msgr2=0x7f1b6c0cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:45.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.159+0000 7f1b70f9a700 1 --2- 172.21.15.27:0/529541473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b6c0b2b90 0x7f1b6c0cb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:45.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.159+0000 7f1b70f9a700 1 -- 172.21.15.27:0/529541473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b64009ae0 con 0x7f1b6c0b07b0 2024-01-11T09:46:45.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.159+0000 7f1b70f9a700 1 --2- 172.21.15.27:0/529541473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b6c0b07b0 0x7f1b6c0cb400 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c00d860 tx=0x7f1b5c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:45.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.159+0000 7f1b617fa700 1 -- 172.21.15.27:0/529541473 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c0125f0 con 0x7f1b6c0b07b0 2024-01-11T09:46:45.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.159+0000 7f1b617fa700 1 -- 172.21.15.27:0/529541473 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5c012750 con 0x7f1b6c0b07b0 2024-01-11T09:46:45.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.159+0000 7f1b617fa700 1 -- 172.21.15.27:0/529541473 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c012af0 con 0x7f1b6c0b07b0 2024-01-11T09:46:45.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.159+0000 7f1b739ff700 1 -- 172.21.15.27:0/529541473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b6c0c4220 con 0x7f1b6c0b07b0 2024-01-11T09:46:45.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.160+0000 7f1b739ff700 1 -- 172.21.15.27:0/529541473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b6c159220 con 0x7f1b6c0b07b0 2024-01-11T09:46:45.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.161+0000 7f1b739ff700 1 -- 172.21.15.27:0/529541473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b6c0c49e0 con 0x7f1b6c0b07b0 2024-01-11T09:46:45.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.162+0000 7f1b617fa700 1 -- 172.21.15.27:0/529541473 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f1b5c013480 con 0x7f1b6c0b07b0 2024-01-11T09:46:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.163+0000 7f1b617fa700 1 --2- 172.21.15.27:0/529541473 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1b50067170 0x7f1b50069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.163+0000 7f1b617fa700 1 -- 172.21.15.27:0/529541473 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f1b5c084ea0 con 0x7f1b6c0b07b0 2024-01-11T09:46:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.163+0000 7f1b7179b700 1 --2- 172.21.15.27:0/529541473 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1b50067170 0x7f1b50069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.164+0000 7f1b7179b700 1 --2- 172.21.15.27:0/529541473 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1b50067170 0x7f1b50069620 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1b68003a80 tx=0x7f1b68007ec0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:45.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.167+0000 7f1b617fa700 1 -- 172.21.15.27:0/529541473 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1b5c051a10 con 0x7f1b6c0b07b0 2024-01-11T09:46:45.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.421+0000 7f1b739ff700 1 -- 172.21.15.27:0/529541473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b6c0718d0 con 0x7f1b6c0b07b0 2024-01-11T09:46:45.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.422+0000 7f1b617fa700 1 -- 172.21.15.27:0/529541473 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5688 (secure 0 0 0) 0x7f1b5c055030 con 0x7f1b6c0b07b0 2024-01-11T09:46:45.414 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:45.414 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":132,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1568813392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1568813392},{"type":"v1","addr":"172.21.15.77:6834","nonce":1568813392}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122},{"gid":10969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1352491019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1352491019},{"type":"v1","addr":"172.21.15.77:6835","nonce":1352491019}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:46:25.726042+0000","modified":"2024-01-11T09:46:29.257798+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11340":{"gid":11340,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/321681407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":321681407},{"type":"v1","addr":"172.21.15.27:6835","nonce":321681407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:46:39.483941+0000","modified":"2024-01-11T09:46:42.996786+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11331},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11331":{"gid":11331,"name":"a","rank":0,"incarnation":129,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6834/3477661846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3477661846},{"type":"v1","addr":"172.21.15.27:6834","nonce":3477661846}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":16}]} 2024-01-11T09:46:45.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.424+0000 7f1b739ff700 1 -- 172.21.15.27:0/529541473 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1b50067170 msgr2=0x7f1b50069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:45.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.424+0000 7f1b739ff700 1 --2- 172.21.15.27:0/529541473 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1b50067170 0x7f1b50069620 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1b68003a80 tx=0x7f1b68007ec0 comp rx=0 tx=0).stop 2024-01-11T09:46:45.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.424+0000 7f1b739ff700 1 -- 172.21.15.27:0/529541473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b6c0b07b0 msgr2=0x7f1b6c0cb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:45.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.424+0000 7f1b739ff700 1 --2- 172.21.15.27:0/529541473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b6c0b07b0 0x7f1b6c0cb400 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c00d860 tx=0x7f1b5c008040 comp rx=0 tx=0).stop 2024-01-11T09:46:45.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.424+0000 7f1b739ff700 1 -- 172.21.15.27:0/529541473 shutdown_connections 2024-01-11T09:46:45.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.424+0000 7f1b739ff700 1 --2- 172.21.15.27:0/529541473 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1b50067170 0x7f1b50069620 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:45.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.424+0000 7f1b739ff700 1 --2- 172.21.15.27:0/529541473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b6c0b2b90 0x7f1b6c0cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:45.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.424+0000 7f1b739ff700 1 --2- 172.21.15.27:0/529541473 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1b6c0b03e0 0x7f1b6c158ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:45.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.424+0000 7f1b739ff700 1 --2- 172.21.15.27:0/529541473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b6c0b07b0 0x7f1b6c0cb400 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:45.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.424+0000 7f1b739ff700 1 -- 172.21.15.27:0/529541473 >> 172.21.15.27:0/529541473 conn(0x7f1b6c06ed40 msgr2=0x7f1b6c0b3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:45.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.424+0000 7f1b739ff700 1 -- 172.21.15.27:0/529541473 shutdown_connections 2024-01-11T09:46:45.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:45.424+0000 7f1b739ff700 1 -- 172.21.15.27:0/529541473 wait complete. 2024-01-11T09:46:45.418 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 132 2024-01-11T09:46:45.837 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 132, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-11T09:46:39.483941+0000', 'modified': '2024-01-11T09:46:42.996786+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11331}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11331': {'gid': 11331, 'name': 'a', 'rank': 0, 'incarnation': 129, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.27:6834/3477661846', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 3477661846}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 3477661846}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-11T09:46:45.837 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-11T09:46:45.838 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-11T09:46:46.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.050+0000 7fa2613ff700 1 Processor -- start 2024-01-11T09:46:46.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.050+0000 7fa2613ff700 1 -- start start 2024-01-11T09:46:46.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.052+0000 7fa2613ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa25c15ca10 0x7fa25c15a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.052+0000 7fa2613ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa25c068cd0 0x7fa25c0b5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.052+0000 7fa2613ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa25c0b4bd0 con 0x7fa25c15cde0 2024-01-11T09:46:46.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.052+0000 7fa2613ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa25c15ac70 con 0x7fa25c068cd0 2024-01-11T09:46:46.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.052+0000 7fa2613ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c0b3fc0 con 0x7fa25c15ca10 2024-01-11T09:46:46.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.052+0000 7fa25a7fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa25c15cde0 0x7fa25c0685c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35112/0 (socket says 172.21.15.27:35112) 2024-01-11T09:46:46.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.052+0000 7fa25a7fc700 1 -- 172.21.15.27:0/2562260492 learned_addr learned my addr 172.21.15.27:0/2562260492 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:46.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.052+0000 7fa25b7fe700 1 --2- 172.21.15.27:0/2562260492 >> v2:172.21.15.27:3300/0 conn(0x7fa25c068cd0 0x7fa25c0b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.052+0000 7fa25affd700 1 --2- 172.21.15.27:0/2562260492 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa25c15ca10 0x7fa25c15a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.052+0000 7fa25b7fe700 1 -- 172.21.15.27:0/2562260492 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa25c15ca10 msgr2=0x7fa25c15a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:46.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.052+0000 7fa25b7fe700 1 --2- 172.21.15.27:0/2562260492 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa25c15ca10 0x7fa25c15a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.052+0000 7fa25b7fe700 1 -- 172.21.15.27:0/2562260492 >> v1:172.21.15.27:6789/0 conn(0x7fa25c15cde0 legacy=0x7fa25c0685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.052+0000 7fa25b7fe700 1 -- 172.21.15.27:0/2562260492 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c0b42a0 con 0x7fa25c068cd0 2024-01-11T09:46:46.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.052+0000 7fa259ffb700 1 -- 172.21.15.27:0/2562260492 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49462610 0 0) 0x7fa25c0b4bd0 con 0x7fa25c15cde0 2024-01-11T09:46:46.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.053+0000 7fa25b7fe700 1 --2- 172.21.15.27:0/2562260492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa25c068cd0 0x7fa25c0b5060 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fa254009e60 tx=0x7fa254009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d19b9d46b879f15f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:46.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.053+0000 7fa259ffb700 1 -- 172.21.15.27:0/2562260492 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25400e070 con 0x7fa25c068cd0 2024-01-11T09:46:46.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.053+0000 7fa259ffb700 1 -- 172.21.15.27:0/2562260492 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa25400bb60 con 0x7fa25c068cd0 2024-01-11T09:46:46.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.053+0000 7fa259ffb700 1 -- 172.21.15.27:0/2562260492 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa254012430 con 0x7fa25c068cd0 2024-01-11T09:46:46.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.053+0000 7fa2613ff700 1 -- 172.21.15.27:0/2562260492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa25c068cd0 msgr2=0x7fa25c0b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.053+0000 7fa2613ff700 1 --2- 172.21.15.27:0/2562260492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa25c068cd0 0x7fa25c0b5060 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fa254009e60 tx=0x7fa254009780 comp rx=0 tx=0).stop 2024-01-11T09:46:46.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.054+0000 7fa2613ff700 1 -- 172.21.15.27:0/2562260492 shutdown_connections 2024-01-11T09:46:46.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.054+0000 7fa2613ff700 1 --2- 172.21.15.27:0/2562260492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa25c068cd0 0x7fa25c0b5060 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.054+0000 7fa2613ff700 1 --2- 172.21.15.27:0/2562260492 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa25c15ca10 0x7fa25c15a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.054+0000 7fa2613ff700 1 -- 172.21.15.27:0/2562260492 >> 172.21.15.27:0/2562260492 conn(0x7fa25c06ed40 msgr2=0x7fa25c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:46.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.054+0000 7fa2613ff700 1 -- 172.21.15.27:0/2562260492 shutdown_connections 2024-01-11T09:46:46.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.054+0000 7fa2613ff700 1 -- 172.21.15.27:0/2562260492 wait complete. 2024-01-11T09:46:46.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.055+0000 7fa2613ff700 1 Processor -- start 2024-01-11T09:46:46.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.055+0000 7fa2613ff700 1 -- start start 2024-01-11T09:46:46.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.056+0000 7fa2613ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa25c068cd0 0x7fa25c169e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.056+0000 7fa2613ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa25c15ca10 0x7fa25c16a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.056+0000 7fa2613ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa25c15cde0 0x7fa25c166f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.056+0000 7fa2613ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c16aa50 con 0x7fa25c068cd0 2024-01-11T09:46:46.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.056+0000 7fa2613ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c16b890 con 0x7fa25c15cde0 2024-01-11T09:46:46.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.056+0000 7fa2613ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c16ac80 con 0x7fa25c15ca10 2024-01-11T09:46:46.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.057+0000 7fa25affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa25c068cd0 0x7fa25c169e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.057+0000 7fa25affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa25c068cd0 0x7fa25c169e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52586/0 (socket says 172.21.15.27:52586) 2024-01-11T09:46:46.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.057+0000 7fa25affd700 1 -- 172.21.15.27:0/726812385 learned_addr learned my addr 172.21.15.27:0/726812385 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:46.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.057+0000 7fa25a7fc700 1 --2- 172.21.15.27:0/726812385 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa25c15ca10 0x7fa25c16a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.057+0000 7fa25affd700 1 -- 172.21.15.27:0/726812385 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa25c15ca10 msgr2=0x7fa25c16a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.057+0000 7fa25b7fe700 1 --2- 172.21.15.27:0/726812385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa25c15cde0 0x7fa25c166f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.057+0000 7fa25affd700 1 --2- 172.21.15.27:0/726812385 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa25c15ca10 0x7fa25c16a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.057+0000 7fa25affd700 1 -- 172.21.15.27:0/726812385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa25c15cde0 msgr2=0x7fa25c166f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.057+0000 7fa25affd700 1 --2- 172.21.15.27:0/726812385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa25c15cde0 0x7fa25c166f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.057+0000 7fa25affd700 1 -- 172.21.15.27:0/726812385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa254009a20 con 0x7fa25c068cd0 2024-01-11T09:46:46.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.057+0000 7fa25a7fc700 1 --2- 172.21.15.27:0/726812385 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa25c15ca10 0x7fa25c16a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:46:46.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.057+0000 7fa25b7fe700 1 --2- 172.21.15.27:0/726812385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa25c15cde0 0x7fa25c166f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:46:46.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.057+0000 7fa25affd700 1 --2- 172.21.15.27:0/726812385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa25c068cd0 0x7fa25c169e70 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00ec80 tx=0x7fa24c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:46.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.058+0000 7fa242ffd700 1 -- 172.21.15.27:0/726812385 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c00c9d0 con 0x7fa25c068cd0 2024-01-11T09:46:46.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.058+0000 7fa242ffd700 1 -- 172.21.15.27:0/726812385 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24c00cb50 con 0x7fa25c068cd0 2024-01-11T09:46:46.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.058+0000 7fa2613ff700 1 -- 172.21.15.27:0/726812385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25c16af60 con 0x7fa25c068cd0 2024-01-11T09:46:46.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.058+0000 7fa242ffd700 1 -- 172.21.15.27:0/726812385 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c013470 con 0x7fa25c068cd0 2024-01-11T09:46:46.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.058+0000 7fa2613ff700 1 -- 172.21.15.27:0/726812385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25c16b520 con 0x7fa25c068cd0 2024-01-11T09:46:46.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.060+0000 7fa242ffd700 1 -- 172.21.15.27:0/726812385 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fa24c013860 con 0x7fa25c068cd0 2024-01-11T09:46:46.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.061+0000 7fa242ffd700 1 --2- 172.21.15.27:0/726812385 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa248067170 0x7fa248069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.061+0000 7fa242ffd700 1 -- 172.21.15.27:0/726812385 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fa24c085030 con 0x7fa25c068cd0 2024-01-11T09:46:46.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.061+0000 7fa25a7fc700 1 --2- 172.21.15.27:0/726812385 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa248067170 0x7fa248069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.062+0000 7fa25a7fc700 1 --2- 172.21.15.27:0/726812385 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa248067170 0x7fa248069620 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0b4da0 tx=0x7fa250009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:46.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.063+0000 7fa26fa91b80 1 Processor -- start 2024-01-11T09:46:46.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.063+0000 7fa26fa91b80 1 -- start start 2024-01-11T09:46:46.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.064+0000 7fa26fa91b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653fea79df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.064+0000 7fa26fa91b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5653fea743a0 0x5653feaa0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.064+0000 7fa26fa91b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653fea98ef0 0x5653feaa0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.064+0000 7fa26fa91b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653fe991310 con 0x5653fea98b20 2024-01-11T09:46:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.064+0000 7fa26fa91b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653fe9ce130 con 0x5653fea98ef0 2024-01-11T09:46:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.064+0000 7fa26fa91b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x5653fe9cbaf0 con 0x5653fea743a0 2024-01-11T09:46:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.064+0000 7fa25affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653fea79df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.064+0000 7fa25affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653fea79df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52588/0 (socket says 172.21.15.27:52588) 2024-01-11T09:46:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.064+0000 7fa25affd700 1 -- 172.21.15.27:0/349214296 learned_addr learned my addr 172.21.15.27:0/349214296 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.064+0000 7fa25a7fc700 1 --2- 172.21.15.27:0/349214296 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5653fea743a0 0x5653feaa0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.064+0000 7fa25b7fe700 1 --2- 172.21.15.27:0/349214296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653fea98ef0 0x5653feaa0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.064+0000 7fa25affd700 1 -- 172.21.15.27:0/349214296 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5653fea743a0 msgr2=0x5653feaa0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.064+0000 7fa25affd700 1 --2- 172.21.15.27:0/349214296 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5653fea743a0 0x5653feaa0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.064+0000 7fa25affd700 1 -- 172.21.15.27:0/349214296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653fea98ef0 msgr2=0x5653feaa0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.064+0000 7fa25affd700 1 --2- 172.21.15.27:0/349214296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653fea98ef0 0x5653feaa0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.064+0000 7fa25affd700 1 -- 172.21.15.27:0/349214296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653fe8ba0a0 con 0x5653fea98b20 2024-01-11T09:46:46.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.065+0000 7fa25affd700 1 --2- 172.21.15.27:0/349214296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653fea79df0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00ec50 tx=0x7fa24c051cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b79ab5613dae378 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:46.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.065+0000 7fa23bfff700 1 -- 172.21.15.27:0/349214296 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c058b50 con 0x5653fea98b20 2024-01-11T09:46:46.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.065+0000 7fa23bfff700 1 -- 172.21.15.27:0/349214296 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24c05c8a0 con 0x5653fea98b20 2024-01-11T09:46:46.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.065+0000 7fa23bfff700 1 -- 172.21.15.27:0/349214296 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c055b60 con 0x5653fea98b20 2024-01-11T09:46:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.065+0000 7fa26fa91b80 1 -- 172.21.15.27:0/349214296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 msgr2=0x5653fea79df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.065+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/349214296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653fea79df0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00ec50 tx=0x7fa24c051cf0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.066+0000 7fa26fa91b80 1 -- 172.21.15.27:0/349214296 shutdown_connections 2024-01-11T09:46:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.066+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/349214296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653fea98ef0 0x5653feaa0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.066+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/349214296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653fea79df0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.066+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/349214296 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5653fea743a0 0x5653feaa0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.066+0000 7fa26fa91b80 1 -- 172.21.15.27:0/349214296 >> 172.21.15.27:0/349214296 conn(0x5653fea76300 msgr2=0x5653fea78bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.066+0000 7fa26fa91b80 1 -- 172.21.15.27:0/349214296 shutdown_connections 2024-01-11T09:46:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.066+0000 7fa26fa91b80 1 -- 172.21.15.27:0/349214296 wait complete. 2024-01-11T09:46:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.066+0000 7fa26fa91b80 1 -- start start 2024-01-11T09:46:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.067+0000 7fa26fa91b80 1 Processor -- start 2024-01-11T09:46:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.069+0000 7fa26fa91b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653fea743a0 0x5653fea96480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.069+0000 7fa26fa91b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653feaa4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.069+0000 7fa26fa91b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5653fea98ef0 0x5653fea73710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.069+0000 7fa26fa91b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653fe9c78e0 con 0x5653fea98b20 2024-01-11T09:46:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.069+0000 7fa26fa91b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653fe973980 con 0x5653fea743a0 2024-01-11T09:46:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.069+0000 7fa26fa91b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x5653fe973620 con 0x5653fea98ef0 2024-01-11T09:46:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.069+0000 7fa26fa91b80 10 client.0 _ll_register_callbacks cb 0x5653fe7d6a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T09:46:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.069+0000 7fa25a7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653feaa4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.069+0000 7fa25a7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653feaa4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52594/0 (socket says 172.21.15.27:52594) 2024-01-11T09:46:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.069+0000 7fa25a7fc700 1 -- 172.21.15.27:0/2602725060 learned_addr learned my addr 172.21.15.27:0/2602725060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.069+0000 7fa25affd700 1 --2- 172.21.15.27:0/2602725060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653fea743a0 0x5653fea96480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.069+0000 7fa25b7fe700 1 --2- 172.21.15.27:0/2602725060 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5653fea98ef0 0x5653fea73710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.069+0000 7fa25a7fc700 1 -- 172.21.15.27:0/2602725060 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5653fea98ef0 msgr2=0x5653fea73710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.069+0000 7fa25a7fc700 1 --2- 172.21.15.27:0/2602725060 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5653fea98ef0 0x5653fea73710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.069+0000 7fa25a7fc700 1 -- 172.21.15.27:0/2602725060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653fea743a0 msgr2=0x5653fea96480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.069+0000 7fa25a7fc700 1 --2- 172.21.15.27:0/2602725060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653fea743a0 0x5653fea96480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.069+0000 7fa25a7fc700 1 -- 172.21.15.27:0/2602725060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25c16b520 con 0x5653fea98b20 2024-01-11T09:46:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.070+0000 7fa25a7fc700 1 --2- 172.21.15.27:0/2602725060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653feaa4c60 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7fa25c160430 tx=0x7fa250016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=777b4b9421525795 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.070+0000 7fa2397fa700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T09:46:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.070+0000 7fa2397fa700 1 -- 172.21.15.27:0/2602725060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25000dc50 con 0x5653fea98b20 2024-01-11T09:46:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.070+0000 7fa2397fa700 1 -- 172.21.15.27:0/2602725060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(239..239 src has 1..239) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fa25000ddb0 con 0x5653fea98b20 2024-01-11T09:46:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.070+0000 7fa26fa91b80 1 -- 172.21.15.27:0/2602725060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5653fea6c560 con 0x5653fea98b20 2024-01-11T09:46:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.070+0000 7fa2397fa700 1 -- 172.21.15.27:0/2602725060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=132) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa250012620 con 0x5653fea98b20 2024-01-11T09:46:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.070+0000 7fa26fa91b80 10 client.11625 fetch_fsmap learned FSMap version 132 2024-01-11T09:46:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.070+0000 7fa26fa91b80 1 -- 172.21.15.27:0/2602725060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=132,monmap=2+}) v3 -- 0x5653fe9963c0 con 0x5653fea98b20 2024-01-11T09:46:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.070+0000 7fa2397fa700 1 -- 172.21.15.27:0/2602725060 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5653fea6c560 con 0x5653fea98b20 2024-01-11T09:46:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.070+0000 7fa2397fa700 1 -- 172.21.15.27:0/2602725060 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 132) v1 ==== 3653+0+0 (secure 0 0 0) 0x7fa250012780 con 0x5653fea98b20 2024-01-11T09:46:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.071+0000 7fa26fa91b80 10 client.11625 fetch_fsmap finished waiting for FSMap version 132 2024-01-11T09:46:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.071+0000 7fa26fa91b80 10 client.11625 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-11T09:46:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.071+0000 7fa26fa91b80 20 client.11625 populate_metadata read hostname 'smithi027' 2024-01-11T09:46:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.071+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/2602725060 >> [v2:172.21.15.27:6832/3477661846,v1:172.21.15.27:6834/3477661846] conn(0x5653feacdf00 0x5653feace2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.071+0000 7fa26fa91b80 4 client.11625 mds_command: new command op to 11331 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-11T09:46:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.071+0000 7fa26fa91b80 1 -- 172.21.15.27:0/2602725060 --> [v2:172.21.15.27:6832/3477661846,v1:172.21.15.27:6834/3477661846] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5653fea490e0 con 0x5653feacdf00 2024-01-11T09:46:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.071+0000 7fa25affd700 1 --2- 172.21.15.27:0/2602725060 >> [v2:172.21.15.27:6832/3477661846,v1:172.21.15.27:6834/3477661846] conn(0x5653feacdf00 0x5653feace2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.071+0000 7fa25affd700 1 --2- 172.21.15.27:0/2602725060 >> [v2:172.21.15.27:6832/3477661846,v1:172.21.15.27:6834/3477661846] conn(0x5653feacdf00 0x5653feace2d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff9b2a79499a4bf2 server_cookie=fabe80e2f7093df4 in_seq=0 out_seq=0 2024-01-11T09:46:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.071+0000 7fa2397fa700 10 client.11625 ms_handle_connect on v2:172.21.15.27:6832/3477661846 2024-01-11T09:46:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.072+0000 7fa2397fa700 1 -- 172.21.15.27:0/2602725060 <== mds.0 v2:172.21.15.27:6832/3477661846 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x5653fea490e0 con 0x5653feacdf00 2024-01-11T09:46:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.072+0000 7fa2397fa700 10 client.11625 handle_command_reply: tid=0 2024-01-11T09:46:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.072+0000 7fa26fa91b80 1 client.11625 shutdown 2024-01-11T09:46:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.072+0000 7fa26fa91b80 1 -- 172.21.15.27:0/2602725060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 msgr2=0x5653feaa4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.072+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/2602725060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653feaa4c60 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7fa25c160430 tx=0x7fa250016040 comp rx=0 tx=0).stop 2024-01-11T09:46:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.072+0000 7fa26fa91b80 1 -- 172.21.15.27:0/2602725060 shutdown_connections 2024-01-11T09:46:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.072+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/2602725060 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5653fea98ef0 0x5653fea73710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.072+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/2602725060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653fea743a0 0x5653fea96480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.072+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/2602725060 >> [v2:172.21.15.27:6832/3477661846,v1:172.21.15.27:6834/3477661846] conn(0x5653feacdf00 0x5653feace2d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.072+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/2602725060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653feaa4c60 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.072+0000 7fa26fa91b80 1 -- 172.21.15.27:0/2602725060 >> 172.21.15.27:0/2602725060 conn(0x5653fea76300 msgr2=0x5653fea74a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:46.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.072+0000 7fa2397fa700 0 client.11625 ms_handle_reset on v2:172.21.15.27:6832/3477661846 2024-01-11T09:46:46.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.073+0000 7fa26fa91b80 1 -- 172.21.15.27:0/2602725060 shutdown_connections 2024-01-11T09:46:46.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.073+0000 7fa26fa91b80 1 -- 172.21.15.27:0/2602725060 wait complete. 2024-01-11T09:46:46.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.073+0000 7fa26fa91b80 20 client.11625 trim_cache size 0 max 16384 2024-01-11T09:46:46.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.090+0000 7fa26fa91b80 1 Processor -- start 2024-01-11T09:46:46.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.090+0000 7fa26fa91b80 1 -- start start 2024-01-11T09:46:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.091+0000 7fa26fa91b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5653fea743a0 0x5653feaa2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.091+0000 7fa26fa91b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653fea77480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.091+0000 7fa26fa91b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653fea98ef0 0x5653fea779c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.091+0000 7fa26fa91b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653fe85f1e0 con 0x5653fea98b20 2024-01-11T09:46:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.091+0000 7fa26fa91b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653fea770a0 con 0x5653fea98ef0 2024-01-11T09:46:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.091+0000 7fa26fa91b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x5653feaa0b40 con 0x5653fea743a0 2024-01-11T09:46:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.091+0000 7fa25a7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653fea77480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.091+0000 7fa25a7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653fea77480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52596/0 (socket says 172.21.15.27:52596) 2024-01-11T09:46:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.091+0000 7fa25a7fc700 1 -- 172.21.15.27:0/3224542504 learned_addr learned my addr 172.21.15.27:0/3224542504 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.091+0000 7fa25a7fc700 1 -- 172.21.15.27:0/3224542504 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5653fea743a0 msgr2=0x5653feaa2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:46.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.091+0000 7fa25a7fc700 1 --2- 172.21.15.27:0/3224542504 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5653fea743a0 0x5653feaa2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.091+0000 7fa25a7fc700 1 -- 172.21.15.27:0/3224542504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653fea98ef0 msgr2=0x5653fea779c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:46.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.091+0000 7fa25a7fc700 1 --2- 172.21.15.27:0/3224542504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653fea98ef0 0x5653fea779c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.091+0000 7fa25a7fc700 1 -- 172.21.15.27:0/3224542504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653fe82e1d0 con 0x5653fea98b20 2024-01-11T09:46:46.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.092+0000 7fa25a7fc700 1 --2- 172.21.15.27:0/3224542504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653fea77480 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fa250009010 tx=0x7fa250013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c61951c43831eea server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:46.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.092+0000 7fa240ff9700 1 -- 172.21.15.27:0/3224542504 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa250012ce0 con 0x5653fea98b20 2024-01-11T09:46:46.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.092+0000 7fa240ff9700 1 -- 172.21.15.27:0/3224542504 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2500013f0 con 0x5653fea98b20 2024-01-11T09:46:46.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.092+0000 7fa240ff9700 1 -- 172.21.15.27:0/3224542504 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa250012870 con 0x5653fea98b20 2024-01-11T09:46:46.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.092+0000 7fa26fa91b80 1 -- 172.21.15.27:0/3224542504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 msgr2=0x5653fea77480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.092+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/3224542504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653fea77480 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fa250009010 tx=0x7fa250013000 comp rx=0 tx=0).stop 2024-01-11T09:46:46.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.092+0000 7fa26fa91b80 1 -- 172.21.15.27:0/3224542504 shutdown_connections 2024-01-11T09:46:46.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.092+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/3224542504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653fea98ef0 0x5653fea779c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.092+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/3224542504 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5653fea743a0 0x5653feaa2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.092+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/3224542504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653fea77480 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.093+0000 7fa26fa91b80 1 -- 172.21.15.27:0/3224542504 >> 172.21.15.27:0/3224542504 conn(0x5653fea76300 msgr2=0x5653fea736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:46.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.093+0000 7fa26fa91b80 1 -- 172.21.15.27:0/3224542504 shutdown_connections 2024-01-11T09:46:46.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.093+0000 7fa26fa91b80 1 -- 172.21.15.27:0/3224542504 wait complete. 2024-01-11T09:46:46.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.093+0000 7fa26fa91b80 1 -- start start 2024-01-11T09:46:46.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.093+0000 7fa26fa91b80 1 Processor -- start 2024-01-11T09:46:46.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.094+0000 7fa26fa91b80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5653fea743a0 0x5653fea96ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.094+0000 7fa26fa91b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653fea971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.094+0000 7fa26fa91b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653fea98ef0 0x5653fea77480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.094+0000 7fa26fa91b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653feb234d0 con 0x5653fea98b20 2024-01-11T09:46:46.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.094+0000 7fa26fa91b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653feb24480 con 0x5653fea98ef0 2024-01-11T09:46:46.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.094+0000 7fa26fa91b80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x5653feabef20 con 0x5653fea743a0 2024-01-11T09:46:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.094+0000 7fa26fa91b80 10 client.0 _ll_register_callbacks cb 0x5653fe7d6a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T09:46:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.095+0000 7fa25a7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653fea971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.095+0000 7fa25a7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653fea971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52598/0 (socket says 172.21.15.27:52598) 2024-01-11T09:46:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.095+0000 7fa25a7fc700 1 -- 172.21.15.27:0/1334997474 learned_addr learned my addr 172.21.15.27:0/1334997474 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.095+0000 7fa25b7fe700 1 --2- 172.21.15.27:0/1334997474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653fea98ef0 0x5653fea77480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.095+0000 7fa25affd700 1 --2- 172.21.15.27:0/1334997474 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5653fea743a0 0x5653fea96ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.095+0000 7fa25a7fc700 1 -- 172.21.15.27:0/1334997474 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5653fea743a0 msgr2=0x5653fea96ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:46.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.095+0000 7fa25a7fc700 1 --2- 172.21.15.27:0/1334997474 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5653fea743a0 0x5653fea96ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.095+0000 7fa25a7fc700 1 -- 172.21.15.27:0/1334997474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653fea98ef0 msgr2=0x5653fea77480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:46.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.095+0000 7fa25a7fc700 1 --2- 172.21.15.27:0/1334997474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653fea98ef0 0x5653fea77480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.095+0000 7fa25a7fc700 1 -- 172.21.15.27:0/1334997474 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5653fe9963c0 con 0x5653fea98b20 2024-01-11T09:46:46.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.096+0000 7fa25a7fc700 1 --2- 172.21.15.27:0/1334997474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653fea971e0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fa250000fb0 tx=0x7fa250023070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99ce9af72f326d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:46.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.096+0000 7fa2397fa700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T09:46:46.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.096+0000 7fa2397fa700 1 -- 172.21.15.27:0/1334997474 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa250007180 con 0x5653fea98b20 2024-01-11T09:46:46.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.096+0000 7fa26fa91b80 1 -- 172.21.15.27:0/1334997474 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa24c05c8a0 con 0x5653fea98b20 2024-01-11T09:46:46.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.096+0000 7fa2397fa700 1 -- 172.21.15.27:0/1334997474 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(239..239 src has 1..239) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fa2500072e0 con 0x5653fea98b20 2024-01-11T09:46:46.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.096+0000 7fa2397fa700 1 -- 172.21.15.27:0/1334997474 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=132) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa250012830 con 0x5653fea98b20 2024-01-11T09:46:46.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.096+0000 7fa26fa91b80 10 client.11631 fetch_fsmap learned FSMap version 132 2024-01-11T09:46:46.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.096+0000 7fa26fa91b80 1 -- 172.21.15.27:0/1334997474 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=132,monmap=2+}) v3 -- 0x5653fe807d60 con 0x5653fea98b20 2024-01-11T09:46:46.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.096+0000 7fa2397fa700 1 -- 172.21.15.27:0/1334997474 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24c05c8a0 con 0x5653fea98b20 2024-01-11T09:46:46.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.096+0000 7fa2397fa700 1 -- 172.21.15.27:0/1334997474 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 132) v1 ==== 3653+0+0 (secure 0 0 0) 0x7fa250021660 con 0x5653fea98b20 2024-01-11T09:46:46.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.096+0000 7fa26fa91b80 10 client.11631 fetch_fsmap finished waiting for FSMap version 132 2024-01-11T09:46:46.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.096+0000 7fa26fa91b80 10 client.11631 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-11T09:46:46.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.096+0000 7fa26fa91b80 20 client.11631 populate_metadata read hostname 'smithi027' 2024-01-11T09:46:46.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.097+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/1334997474 >> [v2:172.21.15.27:6832/3477661846,v1:172.21.15.27:6834/3477661846] conn(0x5653fea779c0 0x5653feac1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.097+0000 7fa26fa91b80 4 client.11631 mds_command: new command op to 11331 tid=0[{"prefix": "status"}] 2024-01-11T09:46:46.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.097+0000 7fa26fa91b80 1 -- 172.21.15.27:0/1334997474 --> [v2:172.21.15.27:6832/3477661846,v1:172.21.15.27:6834/3477661846] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5653fe96a8d0 con 0x5653fea779c0 2024-01-11T09:46:46.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.097+0000 7fa25affd700 1 --2- 172.21.15.27:0/1334997474 >> [v2:172.21.15.27:6832/3477661846,v1:172.21.15.27:6834/3477661846] conn(0x5653fea779c0 0x5653feac1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.097+0000 7fa25affd700 1 --2- 172.21.15.27:0/1334997474 >> [v2:172.21.15.27:6832/3477661846,v1:172.21.15.27:6834/3477661846] conn(0x5653fea779c0 0x5653feac1db0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae5afaa874936e3e server_cookie=9dc1aab4b45864af in_seq=0 out_seq=0 2024-01-11T09:46:46.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.097+0000 7fa2397fa700 10 client.11631 ms_handle_connect on v2:172.21.15.27:6832/3477661846 2024-01-11T09:46:46.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.097+0000 7fa2397fa700 1 -- 172.21.15.27:0/1334997474 <== mds.0 v2:172.21.15.27:6832/3477661846 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+338 (crc 0 0 0) 0x5653fe96a8d0 con 0x5653fea779c0 2024-01-11T09:46:46.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.097+0000 7fa2397fa700 10 client.11631 handle_command_reply: tid=0 2024-01-11T09:46:46.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.097+0000 7fa26fa91b80 1 client.11631 shutdown 2024-01-11T09:46:46.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.098+0000 7fa26fa91b80 1 -- 172.21.15.27:0/1334997474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 msgr2=0x5653fea971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.098+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/1334997474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653fea971e0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fa250000fb0 tx=0x7fa250023070 comp rx=0 tx=0).stop 2024-01-11T09:46:46.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.098+0000 7fa26fa91b80 1 -- 172.21.15.27:0/1334997474 shutdown_connections 2024-01-11T09:46:46.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.098+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/1334997474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653fea98ef0 0x5653fea77480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.098+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/1334997474 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x5653fea743a0 0x5653fea96ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.098+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/1334997474 >> [v2:172.21.15.27:6832/3477661846,v1:172.21.15.27:6834/3477661846] conn(0x5653fea779c0 0x5653feac1db0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.098+0000 7fa26fa91b80 1 --2- 172.21.15.27:0/1334997474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653fea98b20 0x5653fea971e0 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.098+0000 7fa2397fa700 0 client.11631 ms_handle_reset on v2:172.21.15.27:6832/3477661846 2024-01-11T09:46:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.098+0000 7fa26fa91b80 1 -- 172.21.15.27:0/1334997474 >> 172.21.15.27:0/1334997474 conn(0x5653fea76300 msgr2=0x5653feaa2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.098+0000 7fa26fa91b80 1 -- 172.21.15.27:0/1334997474 shutdown_connections 2024-01-11T09:46:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.098+0000 7fa26fa91b80 1 -- 172.21.15.27:0/1334997474 wait complete. 2024-01-11T09:46:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.098+0000 7fa26fa91b80 20 client.11631 trim_cache size 0 max 16384 2024-01-11T09:46:46.095 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:46:46.095 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "8fddb4a7-9d66-40f1-892d-6fe86c728ee4", 2024-01-11T09:46:46.095 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-11T09:46:46.095 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 11331, 2024-01-11T09:46:46.095 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-11T09:46:46.095 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-11T09:46:46.096 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "backup_fs", 2024-01-11T09:46:46.096 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 6.1249350639999998, 2024-01-11T09:46:46.096 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 132, 2024-01-11T09:46:46.096 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 239, 2024-01-11T09:46:46.096 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 239, 2024-01-11T09:46:46.096 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 25.505209219000001 2024-01-11T09:46:46.096 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:46:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.099+0000 7fa2613ff700 1 -- 172.21.15.27:0/726812385 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa248067170 msgr2=0x7fa248069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.099+0000 7fa2613ff700 1 --2- 172.21.15.27:0/726812385 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa248067170 0x7fa248069620 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0b4da0 tx=0x7fa250009040 comp rx=0 tx=0).stop 2024-01-11T09:46:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.099+0000 7fa2613ff700 1 -- 172.21.15.27:0/726812385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa25c068cd0 msgr2=0x7fa25c169e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.099+0000 7fa2613ff700 1 --2- 172.21.15.27:0/726812385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa25c068cd0 0x7fa25c169e70 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00ec80 tx=0x7fa24c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.099+0000 7fa2613ff700 1 -- 172.21.15.27:0/726812385 shutdown_connections 2024-01-11T09:46:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.099+0000 7fa2613ff700 1 --2- 172.21.15.27:0/726812385 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa248067170 0x7fa248069620 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.099+0000 7fa2613ff700 1 --2- 172.21.15.27:0/726812385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa25c15cde0 0x7fa25c166f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.099+0000 7fa2613ff700 1 --2- 172.21.15.27:0/726812385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa25c068cd0 0x7fa25c169e70 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.099+0000 7fa2613ff700 1 --2- 172.21.15.27:0/726812385 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa25c15ca10 0x7fa25c16a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.099+0000 7fa2613ff700 1 -- 172.21.15.27:0/726812385 >> 172.21.15.27:0/726812385 conn(0x7fa25c06ed40 msgr2=0x7fa25c0b5330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.099+0000 7fa2613ff700 1 -- 172.21.15.27:0/726812385 shutdown_connections 2024-01-11T09:46:46.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.099+0000 7fa2613ff700 1 -- 172.21.15.27:0/726812385 wait complete. 2024-01-11T09:46:46.109 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-11T09:46:46.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.317+0000 7f9284bff700 1 Processor -- start 2024-01-11T09:46:46.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.317+0000 7f9284bff700 1 -- start start 2024-01-11T09:46:46.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.318+0000 7f9284bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9280155e30 0x7f92800afc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.318+0000 7f9284bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92800aeb50 con 0x7f92800b0190 2024-01-11T09:46:46.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.318+0000 7f9284bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92800abf20 con 0x7f9280155a60 2024-01-11T09:46:46.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.318+0000 7f9284bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92800b05d0 con 0x7f9280155e30 2024-01-11T09:46:46.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.319+0000 7f927ed9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f92800b0190 0x7f92800ac3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35124/0 (socket says 172.21.15.27:35124) 2024-01-11T09:46:46.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.319+0000 7f927ed9d700 1 -- 172.21.15.27:0/628661242 learned_addr learned my addr 172.21.15.27:0/628661242 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:46.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.319+0000 7f927dd9b700 1 --2- 172.21.15.27:0/628661242 >> v2:172.21.15.77:3300/0 conn(0x7f9280155e30 0x7f92800afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.319+0000 7f927d59a700 1 -- 172.21.15.27:0/628661242 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072497930 0 0) 0x7f92800aeb50 con 0x7f92800b0190 2024-01-11T09:46:46.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.319+0000 7f927d59a700 1 -- 172.21.15.27:0/628661242 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9268003570 con 0x7f92800b0190 2024-01-11T09:46:46.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.319+0000 7f927d59a700 1 -- 172.21.15.27:0/628661242 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888934974 0 0) 0x7f92800abf20 con 0x7f9280155a60 2024-01-11T09:46:46.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.319+0000 7f927d59a700 1 -- 172.21.15.27:0/628661242 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92800aeb50 con 0x7f9280155a60 2024-01-11T09:46:46.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.319+0000 7f927dd9b700 1 -- 172.21.15.27:0/628661242 >> v1:172.21.15.77:6789/0 conn(0x7f9280155a60 legacy=0x7f92800aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.319+0000 7f927dd9b700 1 -- 172.21.15.27:0/628661242 >> v1:172.21.15.27:6789/0 conn(0x7f92800b0190 legacy=0x7f92800ac3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.319+0000 7f927dd9b700 1 -- 172.21.15.27:0/628661242 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92800ae590 con 0x7f9280155e30 2024-01-11T09:46:46.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.320+0000 7f927dd9b700 1 --2- 172.21.15.27:0/628661242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9280155e30 0x7f92800afc50 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f92740098e0 tx=0x7f92740091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6d51591e3a4baef server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:46.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.320+0000 7f927d59a700 1 -- 172.21.15.27:0/628661242 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927400c690 con 0x7f9280155e30 2024-01-11T09:46:46.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.320+0000 7f927d59a700 1 -- 172.21.15.27:0/628661242 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927400c7f0 con 0x7f9280155e30 2024-01-11T09:46:46.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.320+0000 7f927d59a700 1 -- 172.21.15.27:0/628661242 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927400cb90 con 0x7f9280155e30 2024-01-11T09:46:46.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.320+0000 7f9284bff700 1 -- 172.21.15.27:0/628661242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9280155e30 msgr2=0x7f92800afc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.320+0000 7f9284bff700 1 --2- 172.21.15.27:0/628661242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9280155e30 0x7f92800afc50 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f92740098e0 tx=0x7f92740091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.321+0000 7f9284bff700 1 -- 172.21.15.27:0/628661242 shutdown_connections 2024-01-11T09:46:46.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.321+0000 7f9284bff700 1 --2- 172.21.15.27:0/628661242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9280155e30 0x7f92800afc50 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.321+0000 7f9284bff700 1 -- 172.21.15.27:0/628661242 >> 172.21.15.27:0/628661242 conn(0x7f92800680b0 msgr2=0x7f92800cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:46.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.321+0000 7f9284bff700 1 -- 172.21.15.27:0/628661242 shutdown_connections 2024-01-11T09:46:46.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.321+0000 7f9284bff700 1 -- 172.21.15.27:0/628661242 wait complete. 2024-01-11T09:46:46.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.322+0000 7f9284bff700 1 Processor -- start 2024-01-11T09:46:46.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.322+0000 7f9284bff700 1 -- start start 2024-01-11T09:46:46.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.323+0000 7f9284bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92800b0190 0x7f9280150ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.323+0000 7f9284bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9280155a60 0x7f92800aab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.323+0000 7f9284bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9280155e30 0x7f92800ab040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.323+0000 7f9284bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928016b6b0 con 0x7f9280155e30 2024-01-11T09:46:46.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.323+0000 7f9284bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928016aae0 con 0x7f92800b0190 2024-01-11T09:46:46.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.323+0000 7f9284bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f928016ad80 con 0x7f9280155a60 2024-01-11T09:46:46.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.324+0000 7f927ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9280155e30 0x7f92800ab040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.324+0000 7f927ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9280155e30 0x7f92800ab040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52600/0 (socket says 172.21.15.27:52600) 2024-01-11T09:46:46.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.324+0000 7f927ed9d700 1 -- 172.21.15.27:0/1625401454 learned_addr learned my addr 172.21.15.27:0/1625401454 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:46.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.324+0000 7f927e59c700 1 --2- 172.21.15.27:0/1625401454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92800b0190 0x7f9280150ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.324+0000 7f927dd9b700 1 --2- 172.21.15.27:0/1625401454 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9280155a60 0x7f92800aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.324+0000 7f927ed9d700 1 -- 172.21.15.27:0/1625401454 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9280155a60 msgr2=0x7f92800aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.324+0000 7f927ed9d700 1 --2- 172.21.15.27:0/1625401454 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9280155a60 0x7f92800aab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.324+0000 7f927ed9d700 1 -- 172.21.15.27:0/1625401454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92800b0190 msgr2=0x7f9280150ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.324+0000 7f927ed9d700 1 --2- 172.21.15.27:0/1625401454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92800b0190 0x7f9280150ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.324+0000 7f927ed9d700 1 -- 172.21.15.27:0/1625401454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9274009420 con 0x7f9280155e30 2024-01-11T09:46:46.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.324+0000 7f927ed9d700 1 --2- 172.21.15.27:0/1625401454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9280155e30 0x7f92800ab040 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f927800cc60 tx=0x7f9278001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:46.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.325+0000 7f926e7fc700 1 -- 172.21.15.27:0/1625401454 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927800ce70 con 0x7f9280155e30 2024-01-11T09:46:46.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.325+0000 7f926e7fc700 1 -- 172.21.15.27:0/1625401454 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927800ab70 con 0x7f9280155e30 2024-01-11T09:46:46.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.325+0000 7f926e7fc700 1 -- 172.21.15.27:0/1625401454 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92780070f0 con 0x7f9280155e30 2024-01-11T09:46:46.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.325+0000 7f9284bff700 1 -- 172.21.15.27:0/1625401454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f928016b060 con 0x7f9280155e30 2024-01-11T09:46:46.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.325+0000 7f9284bff700 1 -- 172.21.15.27:0/1625401454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9280151220 con 0x7f9280155e30 2024-01-11T09:46:46.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.326+0000 7f9284bff700 1 -- 172.21.15.27:0/1625401454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f928016b820 con 0x7f9280155e30 2024-01-11T09:46:46.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.327+0000 7f926e7fc700 1 -- 172.21.15.27:0/1625401454 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f92780074a0 con 0x7f9280155e30 2024-01-11T09:46:46.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.328+0000 7f926e7fc700 1 --2- 172.21.15.27:0/1625401454 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9260067170 0x7f9260069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.328+0000 7f926e7fc700 1 -- 172.21.15.27:0/1625401454 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f92780851a0 con 0x7f9280155e30 2024-01-11T09:46:46.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.329+0000 7f927e59c700 1 --2- 172.21.15.27:0/1625401454 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9260067170 0x7f9260069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.329+0000 7f927e59c700 1 --2- 172.21.15.27:0/1625401454 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9260067170 0x7f9260069620 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9270003870 tx=0x7f9270009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:46.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.332+0000 7f926e7fc700 1 -- 172.21.15.27:0/1625401454 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9278055820 con 0x7f9280155e30 2024-01-11T09:46:46.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.565+0000 7f9284bff700 1 -- 172.21.15.27:0/1625401454 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f928004c3c0 con 0x7f9260067170 2024-01-11T09:46:46.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.568+0000 7f926e7fc700 1 -- 172.21.15.27:0/1625401454 <== mgr.9107 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7f928004c3c0 con 0x7f9260067170 2024-01-11T09:46:46.559 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T09:46:46.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:46.559 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-11T09:46:46.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:46:46.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:46:46.560 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "backup_fs" 2024-01-11T09:46:46.560 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:46:46.560 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T09:46:46.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.570+0000 7f9284bff700 1 -- 172.21.15.27:0/1625401454 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9260067170 msgr2=0x7f9260069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.570+0000 7f9284bff700 1 --2- 172.21.15.27:0/1625401454 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9260067170 0x7f9260069620 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9270003870 tx=0x7f9270009040 comp rx=0 tx=0).stop 2024-01-11T09:46:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.570+0000 7f9284bff700 1 -- 172.21.15.27:0/1625401454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9280155e30 msgr2=0x7f92800ab040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.570+0000 7f9284bff700 1 --2- 172.21.15.27:0/1625401454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9280155e30 0x7f92800ab040 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f927800cc60 tx=0x7f9278001ae0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.570+0000 7f9284bff700 1 -- 172.21.15.27:0/1625401454 shutdown_connections 2024-01-11T09:46:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.570+0000 7f9284bff700 1 --2- 172.21.15.27:0/1625401454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9280155e30 0x7f92800ab040 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.570+0000 7f9284bff700 1 --2- 172.21.15.27:0/1625401454 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9260067170 0x7f9260069620 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.570+0000 7f9284bff700 1 --2- 172.21.15.27:0/1625401454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92800b0190 0x7f9280150ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.570+0000 7f9284bff700 1 --2- 172.21.15.27:0/1625401454 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9280155a60 0x7f92800aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.570+0000 7f9284bff700 1 -- 172.21.15.27:0/1625401454 >> 172.21.15.27:0/1625401454 conn(0x7f92800680b0 msgr2=0x7f9280151b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.570+0000 7f9284bff700 1 -- 172.21.15.27:0/1625401454 shutdown_connections 2024-01-11T09:46:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.570+0000 7f9284bff700 1 -- 172.21.15.27:0/1625401454 wait complete. 2024-01-11T09:46:46.579 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-11T09:46:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.789+0000 7fb45dbff700 1 Processor -- start 2024-01-11T09:46:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.790+0000 7fb45dbff700 1 -- start start 2024-01-11T09:46:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.791+0000 7fb45dbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb458142dc0 0x7fb4580cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.791+0000 7fb45dbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb458143190 0x7fb4580cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.791+0000 7fb45dbff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45815d320 con 0x7fb4580cc020 2024-01-11T09:46:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.791+0000 7fb45dbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb45815df30 con 0x7fb458142dc0 2024-01-11T09:46:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.791+0000 7fb45dbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb45815d590 con 0x7fb458143190 2024-01-11T09:46:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.791+0000 7fb456ffd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb458143190 0x7fb4580cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.792+0000 7fb4577fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb458142dc0 0x7fb4580cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.792+0000 7fb457fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb4580cc020 0x7fb4580cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:51290/0 (socket says 172.21.15.27:51290) 2024-01-11T09:46:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.792+0000 7fb457fff700 1 -- 172.21.15.27:0/3674723972 learned_addr learned my addr 172.21.15.27:0/3674723972 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.792+0000 7fb4567fc700 1 -- 172.21.15.27:0/3674723972 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1730683525 0 0) 0x7fb45815d320 con 0x7fb4580cc020 2024-01-11T09:46:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.792+0000 7fb4567fc700 1 -- 172.21.15.27:0/3674723972 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb440003570 con 0x7fb4580cc020 2024-01-11T09:46:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.792+0000 7fb4577fe700 1 -- 172.21.15.27:0/3674723972 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb458143190 msgr2=0x7fb4580cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.792+0000 7fb4577fe700 1 --2- 172.21.15.27:0/3674723972 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb458143190 0x7fb4580cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.792+0000 7fb4577fe700 1 -- 172.21.15.27:0/3674723972 >> v1:172.21.15.77:6789/0 conn(0x7fb4580cc020 legacy=0x7fb4580cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.792+0000 7fb4577fe700 1 -- 172.21.15.27:0/3674723972 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45815d870 con 0x7fb458142dc0 2024-01-11T09:46:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.792+0000 7fb456ffd700 1 --2- 172.21.15.27:0/3674723972 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb458143190 0x7fb4580cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.792+0000 7fb4577fe700 1 --2- 172.21.15.27:0/3674723972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb458142dc0 0x7fb4580cada0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fb4480098e0 tx=0x7fb4480091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b1cbf816c4ff394 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.793+0000 7fb4567fc700 1 -- 172.21.15.27:0/3674723972 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb448011630 con 0x7fb458142dc0 2024-01-11T09:46:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.793+0000 7fb4567fc700 1 -- 172.21.15.27:0/3674723972 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb448011790 con 0x7fb458142dc0 2024-01-11T09:46:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.793+0000 7fb4567fc700 1 -- 172.21.15.27:0/3674723972 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb448011b30 con 0x7fb458142dc0 2024-01-11T09:46:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.793+0000 7fb45dbff700 1 -- 172.21.15.27:0/3674723972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb458142dc0 msgr2=0x7fb4580cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.793+0000 7fb45dbff700 1 --2- 172.21.15.27:0/3674723972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb458142dc0 0x7fb4580cada0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fb4480098e0 tx=0x7fb4480091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.793+0000 7fb45dbff700 1 -- 172.21.15.27:0/3674723972 shutdown_connections 2024-01-11T09:46:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.793+0000 7fb45dbff700 1 --2- 172.21.15.27:0/3674723972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb458142dc0 0x7fb4580cada0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.793+0000 7fb45dbff700 1 --2- 172.21.15.27:0/3674723972 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb458143190 0x7fb4580cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.793+0000 7fb45dbff700 1 -- 172.21.15.27:0/3674723972 >> 172.21.15.27:0/3674723972 conn(0x7fb458063ca0 msgr2=0x7fb45806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.794+0000 7fb45dbff700 1 -- 172.21.15.27:0/3674723972 shutdown_connections 2024-01-11T09:46:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.794+0000 7fb45dbff700 1 -- 172.21.15.27:0/3674723972 wait complete. 2024-01-11T09:46:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.795+0000 7fb45dbff700 1 Processor -- start 2024-01-11T09:46:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.795+0000 7fb45dbff700 1 -- start start 2024-01-11T09:46:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.796+0000 7fb45dbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4580cc020 0x7fb45813f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.796+0000 7fb45dbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb458142dc0 0x7fb458141260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.796+0000 7fb45dbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb458143190 0x7fb4580c84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.796+0000 7fb45dbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45816a120 con 0x7fb458142dc0 2024-01-11T09:46:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.796+0000 7fb45dbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb458141b00 con 0x7fb458143190 2024-01-11T09:46:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.796+0000 7fb45dbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb458141de0 con 0x7fb4580cc020 2024-01-11T09:46:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.796+0000 7fb456ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb458142dc0 0x7fb458141260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.797+0000 7fb4577fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4580cc020 0x7fb45813f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.797+0000 7fb456ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb458142dc0 0x7fb458141260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52612/0 (socket says 172.21.15.27:52612) 2024-01-11T09:46:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.797+0000 7fb456ffd700 1 -- 172.21.15.27:0/1613162036 learned_addr learned my addr 172.21.15.27:0/1613162036 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.797+0000 7fb457fff700 1 --2- 172.21.15.27:0/1613162036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb458143190 0x7fb4580c84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.797+0000 7fb456ffd700 1 -- 172.21.15.27:0/1613162036 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4580cc020 msgr2=0x7fb45813f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.797+0000 7fb456ffd700 1 --2- 172.21.15.27:0/1613162036 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4580cc020 0x7fb45813f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.797+0000 7fb456ffd700 1 -- 172.21.15.27:0/1613162036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb458143190 msgr2=0x7fb4580c84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.797+0000 7fb456ffd700 1 --2- 172.21.15.27:0/1613162036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb458143190 0x7fb4580c84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.797+0000 7fb456ffd700 1 -- 172.21.15.27:0/1613162036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44c009420 con 0x7fb458142dc0 2024-01-11T09:46:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.797+0000 7fb4577fe700 1 --2- 172.21.15.27:0/1613162036 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4580cc020 0x7fb45813f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.797+0000 7fb456ffd700 1 --2- 172.21.15.27:0/1613162036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb458142dc0 0x7fb458141260 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00ec80 tx=0x7fb44c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.797+0000 7fb43f7fe700 1 -- 172.21.15.27:0/1613162036 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c00c850 con 0x7fb458142dc0 2024-01-11T09:46:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.798+0000 7fb43f7fe700 1 -- 172.21.15.27:0/1613162036 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44c00c9b0 con 0x7fb458142dc0 2024-01-11T09:46:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.798+0000 7fb43f7fe700 1 -- 172.21.15.27:0/1613162036 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c00cd30 con 0x7fb458142dc0 2024-01-11T09:46:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.798+0000 7fb45dbff700 1 -- 172.21.15.27:0/1613162036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb448009420 con 0x7fb458142dc0 2024-01-11T09:46:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.798+0000 7fb45dbff700 1 -- 172.21.15.27:0/1613162036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45813fa80 con 0x7fb458142dc0 2024-01-11T09:46:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.799+0000 7fb45dbff700 1 -- 172.21.15.27:0/1613162036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb45816a290 con 0x7fb458142dc0 2024-01-11T09:46:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.800+0000 7fb43f7fe700 1 -- 172.21.15.27:0/1613162036 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 137) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fb44c013680 con 0x7fb458142dc0 2024-01-11T09:46:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.801+0000 7fb43f7fe700 1 --2- 172.21.15.27:0/1613162036 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb444067170 0x7fb444069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.801+0000 7fb43f7fe700 1 -- 172.21.15.27:0/1613162036 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6681+0+0 (secure 0 0 0) 0x7fb44c084f40 con 0x7fb458142dc0 2024-01-11T09:46:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.802+0000 7fb4577fe700 1 --2- 172.21.15.27:0/1613162036 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb444067170 0x7fb444069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.803+0000 7fb4577fe700 1 --2- 172.21.15.27:0/1613162036 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb444067170 0x7fb444069620 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb4480091e0 tx=0x7fb448017000 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:46.806+0000 7fb43f7fe700 1 -- 172.21.15.27:0/1613162036 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb44c055540 con 0x7fb458142dc0 2024-01-11T09:46:47.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:47.070+0000 7fb45dbff700 1 -- 172.21.15.27:0/1613162036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fb45815db50 con 0x7fb458142dc0 2024-01-11T09:46:48.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.018+0000 7fb43f7fe700 1 -- 172.21.15.27:0/1613162036 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 138) v1 ==== 84963+0+0 (secure 0 0 0) 0x7fb44c051e80 con 0x7fb458142dc0 2024-01-11T09:46:48.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.019+0000 7fb43f7fe700 1 -- 172.21.15.27:0/1613162036 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v138) v1 ==== 92+0+0 (secure 0 0 0) 0x7fb44c0552b0 con 0x7fb458142dc0 2024-01-11T09:46:48.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.022+0000 7fb45dbff700 1 -- 172.21.15.27:0/1613162036 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb444067170 msgr2=0x7fb444069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:48.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.022+0000 7fb45dbff700 1 --2- 172.21.15.27:0/1613162036 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb444067170 0x7fb444069620 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb4480091e0 tx=0x7fb448017000 comp rx=0 tx=0).stop 2024-01-11T09:46:48.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.022+0000 7fb45dbff700 1 -- 172.21.15.27:0/1613162036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb458142dc0 msgr2=0x7fb458141260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:48.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.022+0000 7fb45dbff700 1 --2- 172.21.15.27:0/1613162036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb458142dc0 0x7fb458141260 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00ec80 tx=0x7fb44c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.022+0000 7fb45dbff700 1 -- 172.21.15.27:0/1613162036 shutdown_connections 2024-01-11T09:46:48.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.022+0000 7fb45dbff700 1 --2- 172.21.15.27:0/1613162036 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb444067170 0x7fb444069620 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.022+0000 7fb45dbff700 1 --2- 172.21.15.27:0/1613162036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb458143190 0x7fb4580c84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.022+0000 7fb45dbff700 1 --2- 172.21.15.27:0/1613162036 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb4580cc020 0x7fb45813f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.022+0000 7fb45dbff700 1 --2- 172.21.15.27:0/1613162036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb458142dc0 0x7fb458141260 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.022+0000 7fb45dbff700 1 -- 172.21.15.27:0/1613162036 >> 172.21.15.27:0/1613162036 conn(0x7fb458063ca0 msgr2=0x7fb45813fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:48.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.023+0000 7fb45dbff700 1 -- 172.21.15.27:0/1613162036 shutdown_connections 2024-01-11T09:46:48.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.023+0000 7fb45dbff700 1 -- 172.21.15.27:0/1613162036 wait complete. 2024-01-11T09:46:48.036 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-11T09:46:48.223 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:48.231+0000 7f7decdaa1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:46:48.232 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:48.241+0000 7f243bd411c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:46:48.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.255+0000 7f79291ff700 1 Processor -- start 2024-01-11T09:46:48.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.255+0000 7f79291ff700 1 -- start start 2024-01-11T09:46:48.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.256+0000 7f79291ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f792415a1b0 0x7f792415a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:48.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.256+0000 7f79291ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f79240ccd70 0x7f79240cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:48.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.256+0000 7f79291ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79240cf850 con 0x7f79240685c0 2024-01-11T09:46:48.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.257+0000 7f79291ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79240cec40 con 0x7f79240ccd70 2024-01-11T09:46:48.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.257+0000 7f79291ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79240cef20 con 0x7f792415a1b0 2024-01-11T09:46:48.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.257+0000 7f792259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f79240ccd70 0x7f79240cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:48.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.257+0000 7f792359e700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f79240685c0 0x7f7924068a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35136/0 (socket says 172.21.15.27:35136) 2024-01-11T09:46:48.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.257+0000 7f792359e700 1 -- 172.21.15.27:0/2864201073 learned_addr learned my addr 172.21.15.27:0/2864201073 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:48.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.257+0000 7f7922d9d700 1 --2- 172.21.15.27:0/2864201073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f792415a1b0 0x7f792415a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:48.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.257+0000 7f792259c700 1 -- 172.21.15.27:0/2864201073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f792415a1b0 msgr2=0x7f792415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:48.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.257+0000 7f792259c700 1 --2- 172.21.15.27:0/2864201073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f792415a1b0 0x7f792415a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.257+0000 7f792259c700 1 -- 172.21.15.27:0/2864201073 >> v1:172.21.15.27:6789/0 conn(0x7f79240685c0 legacy=0x7f7924068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:48.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.257+0000 7f792259c700 1 -- 172.21.15.27:0/2864201073 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79240cf200 con 0x7f79240ccd70 2024-01-11T09:46:48.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.258+0000 7f792259c700 1 --2- 172.21.15.27:0/2864201073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79240ccd70 0x7f79240cd1e0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f79180098e0 tx=0x7f79180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5094ba516656e950 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:48.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.258+0000 7f7921d9b700 1 -- 172.21.15.27:0/2864201073 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791800b640 con 0x7f79240ccd70 2024-01-11T09:46:48.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.258+0000 7f7921d9b700 1 -- 172.21.15.27:0/2864201073 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f791800b7a0 con 0x7f79240ccd70 2024-01-11T09:46:48.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.258+0000 7f7921d9b700 1 -- 172.21.15.27:0/2864201073 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791800bb40 con 0x7f79240ccd70 2024-01-11T09:46:48.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.258+0000 7f79291ff700 1 -- 172.21.15.27:0/2864201073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79240ccd70 msgr2=0x7f79240cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:48.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.258+0000 7f79291ff700 1 --2- 172.21.15.27:0/2864201073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79240ccd70 0x7f79240cd1e0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f79180098e0 tx=0x7f79180091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.259+0000 7f79291ff700 1 -- 172.21.15.27:0/2864201073 shutdown_connections 2024-01-11T09:46:48.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.259+0000 7f79291ff700 1 --2- 172.21.15.27:0/2864201073 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f792415a1b0 0x7f792415a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.259+0000 7f79291ff700 1 --2- 172.21.15.27:0/2864201073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79240ccd70 0x7f79240cd1e0 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.259+0000 7f79291ff700 1 -- 172.21.15.27:0/2864201073 >> 172.21.15.27:0/2864201073 conn(0x7f792406ed40 msgr2=0x7f792406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:48.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.259+0000 7f79291ff700 1 -- 172.21.15.27:0/2864201073 shutdown_connections 2024-01-11T09:46:48.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.259+0000 7f79291ff700 1 -- 172.21.15.27:0/2864201073 wait complete. 2024-01-11T09:46:48.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.260+0000 7f79291ff700 1 Processor -- start 2024-01-11T09:46:48.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.260+0000 7f79291ff700 1 -- start start 2024-01-11T09:46:48.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.261+0000 7f79291ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79240685c0 0x7f79240b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:48.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.261+0000 7f79291ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79240ccd70 0x7f79240ca420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:48.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.261+0000 7f79291ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f792415a1b0 0x7f79240b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:48.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.261+0000 7f79291ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79240c8760 con 0x7f792415a1b0 2024-01-11T09:46:48.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.261+0000 7f79291ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79240b08f0 con 0x7f79240ccd70 2024-01-11T09:46:48.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.261+0000 7f79291ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79240b0bd0 con 0x7f79240685c0 2024-01-11T09:46:48.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.262+0000 7f792359e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f792415a1b0 0x7f79240b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:48.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.262+0000 7f7922d9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79240685c0 0x7f79240b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:48.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.262+0000 7f792259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79240ccd70 0x7f79240ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:48.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.262+0000 7f7922d9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79240685c0 0x7f79240b3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:45782/0 (socket says 172.21.15.27:45782) 2024-01-11T09:46:48.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.262+0000 7f7922d9d700 1 -- 172.21.15.27:0/2328547113 learned_addr learned my addr 172.21.15.27:0/2328547113 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:48.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.262+0000 7f792359e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f792415a1b0 0x7f79240b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52646/0 (socket says 172.21.15.27:52646) 2024-01-11T09:46:48.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.262+0000 7f7922d9d700 1 -- 172.21.15.27:0/2328547113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79240ccd70 msgr2=0x7f79240ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:48.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.262+0000 7f7922d9d700 1 --2- 172.21.15.27:0/2328547113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79240ccd70 0x7f79240ca420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.262+0000 7f7922d9d700 1 -- 172.21.15.27:0/2328547113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f792415a1b0 msgr2=0x7f79240b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:48.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.262+0000 7f7922d9d700 1 --2- 172.21.15.27:0/2328547113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f792415a1b0 0x7f79240b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.262+0000 7f7922d9d700 1 -- 172.21.15.27:0/2328547113 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7918009420 con 0x7f79240685c0 2024-01-11T09:46:48.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.262+0000 7f792359e700 1 --2- 172.21.15.27:0/2328547113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f792415a1b0 0x7f79240b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:48.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.262+0000 7f7922d9d700 1 --2- 172.21.15.27:0/2328547113 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79240685c0 0x7f79240b3380 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f791400e930 tx=0x7f79140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:48.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.263+0000 7f790affd700 1 -- 172.21.15.27:0/2328547113 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7914013680 con 0x7f79240685c0 2024-01-11T09:46:48.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.263+0000 7f790affd700 1 -- 172.21.15.27:0/2328547113 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79140137e0 con 0x7f79240685c0 2024-01-11T09:46:48.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.263+0000 7f79291ff700 1 -- 172.21.15.27:0/2328547113 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79240b0eb0 con 0x7f79240685c0 2024-01-11T09:46:48.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.263+0000 7f790affd700 1 -- 172.21.15.27:0/2328547113 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7914013b80 con 0x7f79240685c0 2024-01-11T09:46:48.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.263+0000 7f79291ff700 1 -- 172.21.15.27:0/2328547113 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79241552f0 con 0x7f79240685c0 2024-01-11T09:46:48.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.264+0000 7f79291ff700 1 -- 172.21.15.27:0/2328547113 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79240c8530 con 0x7f79240685c0 2024-01-11T09:46:48.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.265+0000 7f790affd700 1 -- 172.21.15.27:0/2328547113 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 138) v1 ==== 84963+0+0 (secure 0 0 0) 0x7f791400c3e0 con 0x7f79240685c0 2024-01-11T09:46:48.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.266+0000 7f790affd700 1 --2- 172.21.15.27:0/2328547113 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f79100671c0 0x7f7910069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:48.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.266+0000 7f790affd700 1 -- 172.21.15.27:0/2328547113 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f7914084d70 con 0x7f79240685c0 2024-01-11T09:46:48.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.266+0000 7f792259c700 1 -- 172.21.15.27:0/2328547113 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f79100671c0 msgr2=0x7f7910069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:48.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.266+0000 7f792259c700 1 --2- 172.21.15.27:0/2328547113 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f79100671c0 0x7f7910069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:46:48.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.270+0000 7f790affd700 1 -- 172.21.15.27:0/2328547113 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7914051980 con 0x7f79240685c0 2024-01-11T09:46:48.403 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:48.411+0000 7f7decdaa1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:46:48.431 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:48.440+0000 7f243bd411c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:46:48.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.467+0000 7f792259c700 1 -- 172.21.15.27:0/2328547113 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f79100671c0 msgr2=0x7f7910069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:48.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.467+0000 7f792259c700 1 --2- 172.21.15.27:0/2328547113 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f79100671c0 0x7f7910069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:46:48.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.517+0000 7f79291ff700 1 -- 172.21.15.27:0/2328547113 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f792404c3c0 con 0x7f79240685c0 2024-01-11T09:46:48.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.522+0000 7f790affd700 1 -- 172.21.15.27:0/2328547113 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v83)=0 v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7914054fa0 con 0x7f79240685c0 2024-01-11T09:46:48.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.524+0000 7f79291ff700 1 -- 172.21.15.27:0/2328547113 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f79100671c0 msgr2=0x7f7910069670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:46:48.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.524+0000 7f79291ff700 1 --2- 172.21.15.27:0/2328547113 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f79100671c0 0x7f7910069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.524+0000 7f79291ff700 1 -- 172.21.15.27:0/2328547113 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79240685c0 msgr2=0x7f79240b3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:48.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.524+0000 7f79291ff700 1 --2- 172.21.15.27:0/2328547113 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79240685c0 0x7f79240b3380 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f791400e930 tx=0x7f79140091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.524+0000 7f79291ff700 1 -- 172.21.15.27:0/2328547113 shutdown_connections 2024-01-11T09:46:48.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.524+0000 7f79291ff700 1 --2- 172.21.15.27:0/2328547113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f792415a1b0 0x7f79240b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.524+0000 7f79291ff700 1 --2- 172.21.15.27:0/2328547113 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79240685c0 0x7f79240b3380 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.524+0000 7f79291ff700 1 --2- 172.21.15.27:0/2328547113 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f79100671c0 0x7f7910069670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.524+0000 7f79291ff700 1 --2- 172.21.15.27:0/2328547113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79240ccd70 0x7f79240ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.524+0000 7f79291ff700 1 -- 172.21.15.27:0/2328547113 >> 172.21.15.27:0/2328547113 conn(0x7f792406ed40 msgr2=0x7f7924155b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:48.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.525+0000 7f79291ff700 1 -- 172.21.15.27:0/2328547113 shutdown_connections 2024-01-11T09:46:48.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.525+0000 7f79291ff700 1 -- 172.21.15.27:0/2328547113 wait complete. 2024-01-11T09:46:48.535 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-11T09:46:48.718 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:48.726+0000 7f7decdaa1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:46:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.755+0000 7efe85dff700 1 Processor -- start 2024-01-11T09:46:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.755+0000 7efe85dff700 1 -- start start 2024-01-11T09:46:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.756+0000 7efe85dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe80069790 0x7efe800c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.756+0000 7efe85dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efe80069b60 0x7efe801598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.756+0000 7efe85dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe800cd0a0 con 0x7efe800c8720 2024-01-11T09:46:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.757+0000 7efe85dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe800c7680 con 0x7efe80069790 2024-01-11T09:46:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.757+0000 7efe85dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe800c7960 con 0x7efe80069b60 2024-01-11T09:46:48.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.757+0000 7efe7f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe80069790 0x7efe800c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:48.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.757+0000 7efe7f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe80069790 0x7efe800c69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52650/0 (socket says 172.21.15.27:52650) 2024-01-11T09:46:48.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.757+0000 7efe7f7fe700 1 -- 172.21.15.27:0/3782770745 learned_addr learned my addr 172.21.15.27:0/3782770745 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:48.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.757+0000 7efe7ffff700 1 --2- 172.21.15.27:0/3782770745 >> v2:172.21.15.77:3300/0 conn(0x7efe80069b60 0x7efe801598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.757+0000 7efe7f7fe700 1 -- 172.21.15.27:0/3782770745 >> v2:172.21.15.77:3300/0 conn(0x7efe80069b60 msgr2=0x7efe801598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.757+0000 7efe7f7fe700 1 --2- 172.21.15.27:0/3782770745 >> v2:172.21.15.77:3300/0 conn(0x7efe80069b60 0x7efe801598b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.757+0000 7efe7f7fe700 1 -- 172.21.15.27:0/3782770745 >> v1:172.21.15.27:6789/0 conn(0x7efe800c8720 legacy=0x7efe801591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.757+0000 7efe7f7fe700 1 -- 172.21.15.27:0/3782770745 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe800c7c40 con 0x7efe80069790 2024-01-11T09:46:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.757+0000 7efe7ffff700 1 --2- 172.21.15.27:0/3782770745 >> v2:172.21.15.77:3300/0 conn(0x7efe80069b60 0x7efe801598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:46:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.758+0000 7efe7f7fe700 1 --2- 172.21.15.27:0/3782770745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe80069790 0x7efe800c69a0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7efe800cb930 tx=0x7efe70009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=867f09ade2aeea27 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.758+0000 7efe7e7fc700 1 -- 172.21.15.27:0/3782770745 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe70005c10 con 0x7efe80069790 2024-01-11T09:46:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.758+0000 7efe7e7fc700 1 -- 172.21.15.27:0/3782770745 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe70005d70 con 0x7efe80069790 2024-01-11T09:46:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.758+0000 7efe7e7fc700 1 -- 172.21.15.27:0/3782770745 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7000b910 con 0x7efe80069790 2024-01-11T09:46:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.759+0000 7efe85dff700 1 -- 172.21.15.27:0/3782770745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe80069790 msgr2=0x7efe800c69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.759+0000 7efe85dff700 1 --2- 172.21.15.27:0/3782770745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe80069790 0x7efe800c69a0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7efe800cb930 tx=0x7efe70009220 comp rx=0 tx=0).stop 2024-01-11T09:46:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.759+0000 7efe85dff700 1 -- 172.21.15.27:0/3782770745 shutdown_connections 2024-01-11T09:46:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.759+0000 7efe85dff700 1 --2- 172.21.15.27:0/3782770745 >> v2:172.21.15.77:3300/0 conn(0x7efe80069b60 0x7efe801598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.759+0000 7efe85dff700 1 --2- 172.21.15.27:0/3782770745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe80069790 0x7efe800c69a0 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.759+0000 7efe85dff700 1 -- 172.21.15.27:0/3782770745 >> 172.21.15.27:0/3782770745 conn(0x7efe800680a0 msgr2=0x7efe800cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.759+0000 7efe85dff700 1 -- 172.21.15.27:0/3782770745 shutdown_connections 2024-01-11T09:46:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.759+0000 7efe85dff700 1 -- 172.21.15.27:0/3782770745 wait complete. 2024-01-11T09:46:48.752 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:46:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.761+0000 7efe85dff700 1 Processor -- start 2024-01-11T09:46:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.762+0000 7efe85dff700 1 -- start start 2024-01-11T09:46:48.759 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:46:48.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.763+0000 7efe85dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe80069790 0x7efe80152dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.763+0000 7efe85dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe80069b60 0x7efe80159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.763+0000 7efe85dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe800c8720 0x7efe80147c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.763+0000 7efe85dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8016a980 con 0x7efe800c8720 2024-01-11T09:46:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.763+0000 7efe85dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8015c9b0 con 0x7efe80069b60 2024-01-11T09:46:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.763+0000 7efe85dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe8015c3f0 con 0x7efe80069790 2024-01-11T09:46:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.763+0000 7efe7ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe800c8720 0x7efe80147c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.764+0000 7efe7f7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe80069790 0x7efe80152dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.764+0000 7efe7ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe800c8720 0x7efe80147c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52652/0 (socket says 172.21.15.27:52652) 2024-01-11T09:46:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.764+0000 7efe7ffff700 1 -- 172.21.15.27:0/1539281111 learned_addr learned my addr 172.21.15.27:0/1539281111 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.764+0000 7efe7effd700 1 --2- 172.21.15.27:0/1539281111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe80069b60 0x7efe80159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.764+0000 7efe7ffff700 1 -- 172.21.15.27:0/1539281111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe80069790 msgr2=0x7efe80152dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.764+0000 7efe7ffff700 1 --2- 172.21.15.27:0/1539281111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe80069790 0x7efe80152dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.764+0000 7efe7ffff700 1 -- 172.21.15.27:0/1539281111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe80069b60 msgr2=0x7efe80159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.764+0000 7efe7ffff700 1 --2- 172.21.15.27:0/1539281111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe80069b60 0x7efe80159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.764+0000 7efe7ffff700 1 -- 172.21.15.27:0/1539281111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe70009480 con 0x7efe800c8720 2024-01-11T09:46:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.764+0000 7efe7ffff700 1 --2- 172.21.15.27:0/1539281111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe800c8720 0x7efe80147c10 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7efe78009fc0 tx=0x7efe780099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.765+0000 7efe677fe700 1 -- 172.21.15.27:0/1539281111 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7800dbe0 con 0x7efe800c8720 2024-01-11T09:46:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.765+0000 7efe677fe700 1 -- 172.21.15.27:0/1539281111 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe7800f040 con 0x7efe800c8720 2024-01-11T09:46:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.765+0000 7efe85dff700 1 -- 172.21.15.27:0/1539281111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe8015c6d0 con 0x7efe800c8720 2024-01-11T09:46:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.765+0000 7efe677fe700 1 -- 172.21.15.27:0/1539281111 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe78015440 con 0x7efe800c8720 2024-01-11T09:46:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.765+0000 7efe85dff700 1 -- 172.21.15.27:0/1539281111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe80148150 con 0x7efe800c8720 2024-01-11T09:46:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.766+0000 7efe85dff700 1 -- 172.21.15.27:0/1539281111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe8004c3c0 con 0x7efe800c8720 2024-01-11T09:46:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.768+0000 7efe677fe700 1 -- 172.21.15.27:0/1539281111 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 138) v1 ==== 84963+0+0 (secure 0 0 0) 0x7efe7801e3b0 con 0x7efe800c8720 2024-01-11T09:46:48.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.769+0000 7efe677fe700 1 --2- 172.21.15.27:0/1539281111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efe6c0671f0 0x7efe6c0696a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:48.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.769+0000 7efe677fe700 1 -- 172.21.15.27:0/1539281111 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6681+0+0 (secure 0 0 0) 0x7efe78085190 con 0x7efe800c8720 2024-01-11T09:46:48.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.769+0000 7efe7f7fe700 1 -- 172.21.15.27:0/1539281111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efe6c0671f0 msgr2=0x7efe6c0696a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:48.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.769+0000 7efe7f7fe700 1 --2- 172.21.15.27:0/1539281111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efe6c0671f0 0x7efe6c0696a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:46:48.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.773+0000 7efe677fe700 1 -- 172.21.15.27:0/1539281111 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efe78051c90 con 0x7efe800c8720 2024-01-11T09:46:48.783 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:48.792+0000 7f243bd411c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:46:48.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.970+0000 7efe7f7fe700 1 -- 172.21.15.27:0/1539281111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efe6c0671f0 msgr2=0x7efe6c0696a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:48.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.970+0000 7efe7f7fe700 1 --2- 172.21.15.27:0/1539281111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efe6c0671f0 0x7efe6c0696a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:46:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.986+0000 7efe85dff700 1 -- 172.21.15.27:0/1539281111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7efe8006a110 con 0x7efe800c8720 2024-01-11T09:46:48.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.990+0000 7efe677fe700 1 -- 172.21.15.27:0/1539281111 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v84)=0 v84) v1 ==== 133+0+0 (secure 0 0 0) 0x7efe780552b0 con 0x7efe800c8720 2024-01-11T09:46:48.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.993+0000 7efe85dff700 1 -- 172.21.15.27:0/1539281111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efe6c0671f0 msgr2=0x7efe6c0696a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:46:48.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.993+0000 7efe85dff700 1 --2- 172.21.15.27:0/1539281111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efe6c0671f0 0x7efe6c0696a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.993+0000 7efe85dff700 1 -- 172.21.15.27:0/1539281111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe800c8720 msgr2=0x7efe80147c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:48.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.993+0000 7efe85dff700 1 --2- 172.21.15.27:0/1539281111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe800c8720 0x7efe80147c10 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7efe78009fc0 tx=0x7efe780099b0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.993+0000 7efe85dff700 1 -- 172.21.15.27:0/1539281111 shutdown_connections 2024-01-11T09:46:48.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.993+0000 7efe85dff700 1 --2- 172.21.15.27:0/1539281111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe800c8720 0x7efe80147c10 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.993+0000 7efe85dff700 1 --2- 172.21.15.27:0/1539281111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7efe80069790 0x7efe80152dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.993+0000 7efe85dff700 1 --2- 172.21.15.27:0/1539281111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7efe6c0671f0 0x7efe6c0696a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.993+0000 7efe85dff700 1 --2- 172.21.15.27:0/1539281111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe80069b60 0x7efe80159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:48.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.993+0000 7efe85dff700 1 -- 172.21.15.27:0/1539281111 >> 172.21.15.27:0/1539281111 conn(0x7efe800680a0 msgr2=0x7efe80153f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:48.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.993+0000 7efe85dff700 1 -- 172.21.15.27:0/1539281111 shutdown_connections 2024-01-11T09:46:48.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:48.993+0000 7efe85dff700 1 -- 172.21.15.27:0/1539281111 wait complete. 2024-01-11T09:46:49.004 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:46:49.004 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2024-01-11T09:46:49.058 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir 1M 2024-01-11T09:46:49.257 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:49.265+0000 7f7decdaa1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:46:49.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.276+0000 7f78bbfff700 1 Processor -- start 2024-01-11T09:46:49.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.276+0000 7f78bbfff700 1 -- start start 2024-01-11T09:46:49.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.277+0000 7f78bbfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f78b4069790 0x7f78b40c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:49.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.277+0000 7f78bbfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f78b4069b60 0x7f78b41598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:49.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.277+0000 7f78bbfff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78b40cd0a0 con 0x7f78b40c8720 2024-01-11T09:46:49.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.277+0000 7f78bbfff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78b40c7680 con 0x7f78b4069790 2024-01-11T09:46:49.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.277+0000 7f78bbfff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78b40c7960 con 0x7f78b4069b60 2024-01-11T09:46:49.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.277+0000 7f78b9d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f78b4069790 0x7f78b40c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:49.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.278+0000 7f78b9d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f78b4069790 0x7f78b40c69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52662/0 (socket says 172.21.15.27:52662) 2024-01-11T09:46:49.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.278+0000 7f78b9d9b700 1 -- 172.21.15.27:0/2342485088 learned_addr learned my addr 172.21.15.27:0/2342485088 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:49.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.278+0000 7f78ba59c700 1 --2- 172.21.15.27:0/2342485088 >> v2:172.21.15.77:3300/0 conn(0x7f78b4069b60 0x7f78b41598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:49.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.278+0000 7f78b9d9b700 1 -- 172.21.15.27:0/2342485088 >> v2:172.21.15.77:3300/0 conn(0x7f78b4069b60 msgr2=0x7f78b41598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:49.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.278+0000 7f78b9d9b700 1 --2- 172.21.15.27:0/2342485088 >> v2:172.21.15.77:3300/0 conn(0x7f78b4069b60 0x7f78b41598b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:49.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.278+0000 7f78b9d9b700 1 -- 172.21.15.27:0/2342485088 >> v1:172.21.15.77:6789/0 conn(0x7f78b40c8720 legacy=0x7f78b41591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:49.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.278+0000 7f78b9d9b700 1 -- 172.21.15.27:0/2342485088 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b40c7c40 con 0x7f78b4069790 2024-01-11T09:46:49.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.278+0000 7f78b9d9b700 1 --2- 172.21.15.27:0/2342485088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78b4069790 0x7f78b40c69a0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f78b40cb930 tx=0x7f78b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e49909f864a39d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:49.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.279+0000 7f78b8d99700 1 -- 172.21.15.27:0/2342485088 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b0009e60 con 0x7f78b4069790 2024-01-11T09:46:49.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.279+0000 7f78b8d99700 1 -- 172.21.15.27:0/2342485088 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b000b8f0 con 0x7f78b4069790 2024-01-11T09:46:49.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.279+0000 7f78b8d99700 1 -- 172.21.15.27:0/2342485088 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b000bc90 con 0x7f78b4069790 2024-01-11T09:46:49.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.279+0000 7f78bbfff700 1 -- 172.21.15.27:0/2342485088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78b4069790 msgr2=0x7f78b40c69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:49.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.279+0000 7f78bbfff700 1 --2- 172.21.15.27:0/2342485088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78b4069790 0x7f78b40c69a0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f78b40cb930 tx=0x7f78b0009250 comp rx=0 tx=0).stop 2024-01-11T09:46:49.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.279+0000 7f78bbfff700 1 -- 172.21.15.27:0/2342485088 shutdown_connections 2024-01-11T09:46:49.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.279+0000 7f78bbfff700 1 --2- 172.21.15.27:0/2342485088 >> v2:172.21.15.77:3300/0 conn(0x7f78b4069b60 0x7f78b41598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:49.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.279+0000 7f78bbfff700 1 --2- 172.21.15.27:0/2342485088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78b4069790 0x7f78b40c69a0 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:49.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.279+0000 7f78bbfff700 1 -- 172.21.15.27:0/2342485088 >> 172.21.15.27:0/2342485088 conn(0x7f78b40680a0 msgr2=0x7f78b40cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:49.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.279+0000 7f78bbfff700 1 -- 172.21.15.27:0/2342485088 shutdown_connections 2024-01-11T09:46:49.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.280+0000 7f78bbfff700 1 -- 172.21.15.27:0/2342485088 wait complete. 2024-01-11T09:46:49.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.281+0000 7f78bbfff700 1 Processor -- start 2024-01-11T09:46:49.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.281+0000 7f78bbfff700 1 -- start start 2024-01-11T09:46:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.282+0000 7f78bbfff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78b4069790 0x7f78b4148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.282+0000 7f78bbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78b4069b60 0x7f78b40b1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.282+0000 7f78bbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78b40c8720 0x7f78b406eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.282+0000 7f78bbfff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b40b0860 con 0x7f78b40c8720 2024-01-11T09:46:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.282+0000 7f78bbfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b4149280 con 0x7f78b4069b60 2024-01-11T09:46:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.282+0000 7f78bbfff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78b40b1d80 con 0x7f78b4069790 2024-01-11T09:46:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.282+0000 7f78ba59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78b40c8720 0x7f78b406eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.282+0000 7f78ba59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78b40c8720 0x7f78b406eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52672/0 (socket says 172.21.15.27:52672) 2024-01-11T09:46:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.282+0000 7f78ba59c700 1 -- 172.21.15.27:0/689816695 learned_addr learned my addr 172.21.15.27:0/689816695 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.282+0000 7f78b9d9b700 1 --2- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78b4069790 0x7f78b4148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.283+0000 7f78b959a700 1 --2- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78b4069b60 0x7f78b40b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.283+0000 7f78ba59c700 1 -- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78b4069790 msgr2=0x7f78b4148d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.283+0000 7f78ba59c700 1 --2- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f78b4069790 0x7f78b4148d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:49.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.283+0000 7f78ba59c700 1 -- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78b4069b60 msgr2=0x7f78b40b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:49.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.283+0000 7f78ba59c700 1 --2- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78b4069b60 0x7f78b40b1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:49.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.283+0000 7f78ba59c700 1 -- 172.21.15.27:0/689816695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b0009490 con 0x7f78b40c8720 2024-01-11T09:46:49.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.283+0000 7f78ba59c700 1 --2- 172.21.15.27:0/689816695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78b40c8720 0x7f78b406eca0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f78ac009e20 tx=0x7f78ac00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:49.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.283+0000 7f78a9ffb700 1 -- 172.21.15.27:0/689816695 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ac00caa0 con 0x7f78b40c8720 2024-01-11T09:46:49.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.284+0000 7f78a9ffb700 1 -- 172.21.15.27:0/689816695 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ac00f040 con 0x7f78b40c8720 2024-01-11T09:46:49.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.284+0000 7f78a9ffb700 1 -- 172.21.15.27:0/689816695 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ac00ce70 con 0x7f78b40c8720 2024-01-11T09:46:49.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.284+0000 7f78bbfff700 1 -- 172.21.15.27:0/689816695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b40aff30 con 0x7f78b40c8720 2024-01-11T09:46:49.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.284+0000 7f78bbfff700 1 -- 172.21.15.27:0/689816695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b40b04f0 con 0x7f78b40c8720 2024-01-11T09:46:49.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.285+0000 7f78bbfff700 1 -- 172.21.15.27:0/689816695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b40b09d0 con 0x7f78b40c8720 2024-01-11T09:46:49.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.286+0000 7f78a9ffb700 1 -- 172.21.15.27:0/689816695 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 138) v1 ==== 84963+0+0 (secure 0 0 0) 0x7f78ac014690 con 0x7f78b40c8720 2024-01-11T09:46:49.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.287+0000 7f78a9ffb700 1 --2- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f78980671c0 0x7f7898069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:49.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.287+0000 7f78a9ffb700 1 -- 172.21.15.27:0/689816695 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6681+0+0 (secure 0 0 0) 0x7f78ac085650 con 0x7f78b40c8720 2024-01-11T09:46:49.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.287+0000 7f78b9d9b700 1 -- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f78980671c0 msgr2=0x7f7898069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:49.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.287+0000 7f78b9d9b700 1 --2- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f78980671c0 0x7f7898069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:46:49.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.291+0000 7f78a9ffb700 1 -- 172.21.15.27:0/689816695 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f78ac055be0 con 0x7f78b40c8720 2024-01-11T09:46:49.357 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:49.366+0000 7f243bd411c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:46:49.429 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:49.437+0000 7f7decdaa1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:46:49.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.488+0000 7f78b9d9b700 1 -- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f78980671c0 msgr2=0x7f7898069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:49.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.488+0000 7f78b9d9b700 1 --2- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f78980671c0 0x7f7898069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:46:49.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.520+0000 7f78bbfff700 1 -- 172.21.15.27:0/689816695 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f78b404c3c0 con 0x7f78980671c0 2024-01-11T09:46:49.538 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:49.546+0000 7f243bd411c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:46:49.841 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:49.849+0000 7f7decdaa1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:46:49.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.889+0000 7f78b9d9b700 1 -- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f78980671c0 msgr2=0x7f7898069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:49.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:49.889+0000 7f78b9d9b700 1 --2- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f78980671c0 0x7f7898069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-11T09:46:49.978 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:49.986+0000 7f243bd411c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:46:50.621 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:50.629+0000 7f7decdaa1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:46:50.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:50.690+0000 7f78b9d9b700 1 -- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f78980671c0 msgr2=0x7f7898069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:50.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:50.690+0000 7f78b9d9b700 1 --2- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f78980671c0 0x7f7898069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-11T09:46:50.710 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:50.718+0000 7f7decdaa1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:46:50.802 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:50.811+0000 7f243bd411c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:46:50.866 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:50.873+0000 7f7decdaa1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:46:50.896 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:50.905+0000 7f243bd411c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:46:50.955 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:50.963+0000 7f7decdaa1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:46:51.062 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:51.071+0000 7f243bd411c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:46:51.156 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:51.165+0000 7f243bd411c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:46:51.173 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:51.181+0000 7f7decdaa1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:46:51.269 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:51.277+0000 7f7decdaa1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:46:51.364 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:51.372+0000 7f7decdaa1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:46:51.383 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:51.392+0000 7f243bd411c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:46:51.459 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:51.467+0000 7f7decdaa1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:46:51.484 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:51.493+0000 7f243bd411c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:46:51.586 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:51.594+0000 7f243bd411c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:46:51.650 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:51.658+0000 7f7decdaa1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:46:51.686 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:51.695+0000 7f243bd411c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:46:51.747 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:51.755+0000 7f7decdaa1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:46:51.886 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:51.895+0000 7f243bd411c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:46:51.990 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:51.999+0000 7f243bd411c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:46:52.090 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:52.098+0000 7f7decdaa1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:46:52.180 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:52.188+0000 7f7decdaa1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:46:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:52.292+0000 7f78b9d9b700 1 -- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f78980671c0 msgr2=0x7f7898069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:52.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:52.292+0000 7f78b9d9b700 1 --2- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f78980671c0 0x7f7898069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-11T09:46:52.352 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:52.361+0000 7f243bd411c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:46:52.447 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:52.456+0000 7f243bd411c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:46:52.720 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:52.728+0000 7f7decdaa1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:46:52.906 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:52.914+0000 7f7decdaa1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:46:53.025 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:53.034+0000 7f243bd411c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:46:53.224 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:53.233+0000 7f243bd411c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:46:53.285 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:53.294+0000 7f7decdaa1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:46:53.622 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:53.630+0000 7f243bd411c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:46:53.707 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:53.714+0000 7f7decdaa1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:46:53.890 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:53.899+0000 7f7decdaa1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:46:54.072 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:54.081+0000 7f243bd411c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:46:54.078 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.087+0000 7f7decdaa1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:46:54.188 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.197+0000 7f7decdaa1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:46:54.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:54.208+0000 7f78a9ffb700 1 -- 172.21.15.27:0/689816695 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 139) v1 ==== 84470+0+0 (secure 0 0 0) 0x7f78ac052520 con 0x7f78b40c8720 2024-01-11T09:46:54.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:54.208+0000 7f78a9ffb700 1 -- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f78980671c0 msgr2=0x7f7898069670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:46:54.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:54.208+0000 7f78a9ffb700 1 --2- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f78980671c0 0x7f7898069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:54.275 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:54.284+0000 7f243bd411c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:46:54.371 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.380+0000 7f7d61c38700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.372 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.380+0000 7f7d61c38700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.372 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.380+0000 7f7d61c38700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.372 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.380+0000 7f7d61c38700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.372 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.380+0000 7f7d61c38700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.372 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.380+0000 7f7d64c3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.372 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.380+0000 7f7d64c3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.372 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.380+0000 7f7d64c3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.372 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.380+0000 7f7d64c3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.372 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.380+0000 7f7d64c3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.385 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.394+0000 7f7d61c38700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.385 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.394+0000 7f7d61c38700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.385 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.394+0000 7f7d61c38700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.385 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.394+0000 7f7d61c38700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.385 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.394+0000 7f7d61c38700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.386 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.395+0000 7f7d64c3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.386 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.395+0000 7f7d64c3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.386 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.395+0000 7f7d64c3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.386 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.395+0000 7f7d64c3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.386 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:54.395+0000 7f7d64c3e700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:46:54.483 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:54.491+0000 7f243bd411c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:46:54.599 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:54.608+0000 7f243bd411c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:46:55.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.210+0000 7f78a9ffb700 1 -- 172.21.15.27:0/689816695 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 140) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f78ac0606d0 con 0x7f78b40c8720 2024-01-11T09:46:55.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.210+0000 7f78a9ffb700 1 --2- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f78980671c0 0x7f7898069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:55.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.210+0000 7f78a9ffb700 1 -- 172.21.15.27:0/689816695 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f78b404c3c0 con 0x7f78980671c0 2024-01-11T09:46:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.210+0000 7f78b9d9b700 1 --2- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f78980671c0 0x7f7898069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:55.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.212+0000 7f78b9d9b700 1 --2- 172.21.15.27:0/689816695 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f78980671c0 0x7f7898069670 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f78b40cb930 tx=0x7f78b0017000 comp rx=0 tx=0).ready entity=mgr.11652 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:55.209 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:55.218+0000 7f7d6bc4c700 -1 mgr.server reply reply (22) Invalid argument filesystem argument is required when there is more than one file system 2024-01-11T09:46:55.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.219+0000 7f78a9ffb700 1 -- 172.21.15.27:0/689816695 <== mgr.11652 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: -22 filesystem argument is required when there is more than one file system) v1 ==== 79+0+0 (secure 0 0 0) 0x7f78b404c3c0 con 0x7f78980671c0 2024-01-11T09:46:55.212 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: filesystem argument is required when there is more than one file system 2024-01-11T09:46:55.221 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T09:46:55.222 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:46:55.222 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2024-01-11T09:46:55.274 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-11T09:46:55.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.485+0000 7fafb97ff700 1 Processor -- start 2024-01-11T09:46:55.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.486+0000 7fafb97ff700 1 -- start start 2024-01-11T09:46:55.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.487+0000 7fafb97ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fafb413ff80 0x7fafb415a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:55.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.487+0000 7fafb97ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fafb415a560 0x7fafb40c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:55.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.487+0000 7fafb97ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb40c6b30 con 0x7fafb413fbb0 2024-01-11T09:46:55.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.487+0000 7fafb97ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb40c6800 con 0x7fafb413ff80 2024-01-11T09:46:55.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.487+0000 7fafb97ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb415a960 con 0x7fafb415a560 2024-01-11T09:46:55.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.487+0000 7fafb27fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fafb413ff80 0x7fafb415a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:55.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.487+0000 7fafb27fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fafb413ff80 0x7fafb415a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41698/0 (socket says 172.21.15.27:41698) 2024-01-11T09:46:55.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.487+0000 7fafb27fc700 1 -- 172.21.15.27:0/2862772380 learned_addr learned my addr 172.21.15.27:0/2862772380 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:55.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.487+0000 7fafb37fe700 1 --2- 172.21.15.27:0/2862772380 >> v2:172.21.15.77:3300/0 conn(0x7fafb415a560 0x7fafb40c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:55.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.488+0000 7fafb27fc700 1 -- 172.21.15.27:0/2862772380 >> v2:172.21.15.77:3300/0 conn(0x7fafb415a560 msgr2=0x7fafb40c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.488+0000 7fafb27fc700 1 --2- 172.21.15.27:0/2862772380 >> v2:172.21.15.77:3300/0 conn(0x7fafb415a560 0x7fafb40c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.488+0000 7fafb27fc700 1 -- 172.21.15.27:0/2862772380 >> v1:172.21.15.77:6789/0 conn(0x7fafb413fbb0 legacy=0x7fafb40ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.488+0000 7fafb27fc700 1 -- 172.21.15.27:0/2862772380 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb40cb240 con 0x7fafb413ff80 2024-01-11T09:46:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.488+0000 7fafb27fc700 1 --2- 172.21.15.27:0/2862772380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb413ff80 0x7fafb415a020 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fafb406fe10 tx=0x7fafa8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a577f27635d5b2f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:55.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.488+0000 7fafb1ffb700 1 -- 172.21.15.27:0/2862772380 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8009e70 con 0x7fafb413ff80 2024-01-11T09:46:55.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.488+0000 7fafb1ffb700 1 -- 172.21.15.27:0/2862772380 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa800b860 con 0x7fafb413ff80 2024-01-11T09:46:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.488+0000 7fafb1ffb700 1 -- 172.21.15.27:0/2862772380 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa800bc00 con 0x7fafb413ff80 2024-01-11T09:46:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.489+0000 7fafb97ff700 1 -- 172.21.15.27:0/2862772380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb413ff80 msgr2=0x7fafb415a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.489+0000 7fafb97ff700 1 --2- 172.21.15.27:0/2862772380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb413ff80 0x7fafb415a020 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fafb406fe10 tx=0x7fafa8009250 comp rx=0 tx=0).stop 2024-01-11T09:46:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.489+0000 7fafb97ff700 1 -- 172.21.15.27:0/2862772380 shutdown_connections 2024-01-11T09:46:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.489+0000 7fafb97ff700 1 --2- 172.21.15.27:0/2862772380 >> v2:172.21.15.77:3300/0 conn(0x7fafb415a560 0x7fafb40c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.489+0000 7fafb97ff700 1 --2- 172.21.15.27:0/2862772380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb413ff80 0x7fafb415a020 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.489+0000 7fafb97ff700 1 -- 172.21.15.27:0/2862772380 >> 172.21.15.27:0/2862772380 conn(0x7fafb40680a0 msgr2=0x7fafb40cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.489+0000 7fafb97ff700 1 -- 172.21.15.27:0/2862772380 shutdown_connections 2024-01-11T09:46:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.489+0000 7fafb97ff700 1 -- 172.21.15.27:0/2862772380 wait complete. 2024-01-11T09:46:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.490+0000 7fafb97ff700 1 Processor -- start 2024-01-11T09:46:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.491+0000 7fafb97ff700 1 -- start start 2024-01-11T09:46:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.492+0000 7fafb97ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb413fbb0 0x7fafb40af800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.492+0000 7fafb97ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafb413ff80 0x7fafb416de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.492+0000 7fafb97ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb415a560 0x7fafb40ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.492+0000 7fafb97ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb40afd40 con 0x7fafb413fbb0 2024-01-11T09:46:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.492+0000 7fafb97ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb416e380 con 0x7fafb415a560 2024-01-11T09:46:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.492+0000 7fafb97ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb40c4440 con 0x7fafb413ff80 2024-01-11T09:46:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.492+0000 7fafb2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb413fbb0 0x7fafb40af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.492+0000 7fafb2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb413fbb0 0x7fafb40af800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41702/0 (socket says 172.21.15.27:41702) 2024-01-11T09:46:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.492+0000 7fafb2ffd700 1 -- 172.21.15.27:0/878407272 learned_addr learned my addr 172.21.15.27:0/878407272 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.492+0000 7fafb27fc700 1 --2- 172.21.15.27:0/878407272 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafb413ff80 0x7fafb416de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.492+0000 7fafb37fe700 1 --2- 172.21.15.27:0/878407272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb415a560 0x7fafb40ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.492+0000 7fafb2ffd700 1 -- 172.21.15.27:0/878407272 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafb413ff80 msgr2=0x7fafb416de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.492+0000 7fafb2ffd700 1 --2- 172.21.15.27:0/878407272 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafb413ff80 0x7fafb416de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.492+0000 7fafb2ffd700 1 -- 172.21.15.27:0/878407272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb415a560 msgr2=0x7fafb40ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.492+0000 7fafb2ffd700 1 --2- 172.21.15.27:0/878407272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb415a560 0x7fafb40ca5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.492+0000 7fafb2ffd700 1 -- 172.21.15.27:0/878407272 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa80094f0 con 0x7fafb413fbb0 2024-01-11T09:46:55.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.493+0000 7fafb2ffd700 1 --2- 172.21.15.27:0/878407272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb413fbb0 0x7fafb40af800 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fafa400c7f0 tx=0x7fafa4007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:55.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.493+0000 7faf9affd700 1 -- 172.21.15.27:0/878407272 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4011590 con 0x7fafb413fbb0 2024-01-11T09:46:55.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.493+0000 7faf9affd700 1 -- 172.21.15.27:0/878407272 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa4011710 con 0x7fafb413fbb0 2024-01-11T09:46:55.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.493+0000 7fafb97ff700 1 -- 172.21.15.27:0/878407272 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb40c3830 con 0x7fafb413fbb0 2024-01-11T09:46:55.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.493+0000 7faf9affd700 1 -- 172.21.15.27:0/878407272 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4011ad0 con 0x7fafb413fbb0 2024-01-11T09:46:55.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.493+0000 7fafb97ff700 1 -- 172.21.15.27:0/878407272 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb40c3df0 con 0x7fafb413fbb0 2024-01-11T09:46:55.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.494+0000 7fafb97ff700 1 -- 172.21.15.27:0/878407272 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb404c3c0 con 0x7fafb413fbb0 2024-01-11T09:46:55.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.495+0000 7faf9affd700 1 -- 172.21.15.27:0/878407272 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 140) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fafa401b3b0 con 0x7fafb413fbb0 2024-01-11T09:46:55.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.496+0000 7faf9affd700 1 --2- 172.21.15.27:0/878407272 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fafa0066dd0 0x7fafa0069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:55.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.497+0000 7faf9affd700 1 -- 172.21.15.27:0/878407272 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 7111+0+0 (secure 0 0 0) 0x7fafa4083be0 con 0x7fafb413fbb0 2024-01-11T09:46:55.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.497+0000 7fafb27fc700 1 --2- 172.21.15.27:0/878407272 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fafa0066dd0 0x7fafa0069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:55.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.498+0000 7fafb27fc700 1 --2- 172.21.15.27:0/878407272 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fafa0066dd0 0x7fafa0069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fafb406fe10 tx=0x7fafa8017000 comp rx=0 tx=0).ready entity=mgr.11652 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:55.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.501+0000 7faf9affd700 1 -- 172.21.15.27:0/878407272 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fafa4050b60 con 0x7fafb413fbb0 2024-01-11T09:46:55.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:55.762+0000 7fafb97ff700 1 -- 172.21.15.27:0/878407272 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fafb40cb520 con 0x7fafb413fbb0 2024-01-11T09:46:56.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.262+0000 7faf9affd700 1 -- 172.21.15.27:0/878407272 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 141) v1 ==== 84947+0+0 (secure 0 0 0) 0x7fafa4050f30 con 0x7fafb413fbb0 2024-01-11T09:46:56.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.264+0000 7faf9affd700 1 -- 172.21.15.27:0/878407272 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v141) v1 ==== 93+0+0 (secure 0 0 0) 0x7fafa406f380 con 0x7fafb413fbb0 2024-01-11T09:46:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.267+0000 7fafb97ff700 1 -- 172.21.15.27:0/878407272 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fafa0066dd0 msgr2=0x7fafa0069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.267+0000 7fafb97ff700 1 --2- 172.21.15.27:0/878407272 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fafa0066dd0 0x7fafa0069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fafb406fe10 tx=0x7fafa8017000 comp rx=0 tx=0).stop 2024-01-11T09:46:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.267+0000 7fafb97ff700 1 -- 172.21.15.27:0/878407272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb413fbb0 msgr2=0x7fafb40af800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.267+0000 7fafb97ff700 1 --2- 172.21.15.27:0/878407272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb413fbb0 0x7fafb40af800 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fafa400c7f0 tx=0x7fafa4007f70 comp rx=0 tx=0).stop 2024-01-11T09:46:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.267+0000 7fafb97ff700 1 -- 172.21.15.27:0/878407272 shutdown_connections 2024-01-11T09:46:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.267+0000 7fafb97ff700 1 --2- 172.21.15.27:0/878407272 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fafa0066dd0 0x7fafa0069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.267+0000 7fafb97ff700 1 --2- 172.21.15.27:0/878407272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb415a560 0x7fafb40ca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.267+0000 7fafb97ff700 1 --2- 172.21.15.27:0/878407272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb413fbb0 0x7fafb40af800 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.267+0000 7fafb97ff700 1 --2- 172.21.15.27:0/878407272 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fafb413ff80 0x7fafb416de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.267+0000 7fafb97ff700 1 -- 172.21.15.27:0/878407272 >> 172.21.15.27:0/878407272 conn(0x7fafb40680a0 msgr2=0x7fafb40c1d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.267+0000 7fafb97ff700 1 -- 172.21.15.27:0/878407272 shutdown_connections 2024-01-11T09:46:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.267+0000 7fafb97ff700 1 -- 172.21.15.27:0/878407272 wait complete. 2024-01-11T09:46:56.279 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T09:46:56.279 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:46:56.396 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:46:56.475 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:56.484+0000 7f923a65f1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:46:56.476 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:56.484+0000 7f95c8a461c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:46:56.489 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T09:46:56.490 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-11T09:46:56.490 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:46:56.490 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-01-11T09:46:56.570 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T09:46:56.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:46:56.571 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-11T09:46:56.656 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:46:56.657 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:56.667+0000 7f923a65f1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:46:56.677 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:56.686+0000 7f95c8a461c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:46:56.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.868+0000 7f1cb8dff700 1 Processor -- start 2024-01-11T09:46:56.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.868+0000 7f1cb8dff700 1 -- start start 2024-01-11T09:46:56.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.869+0000 7f1cb8dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1cb40b03e0 0x7f1cb40b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:56.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.869+0000 7f1cb8dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1cb40b07b0 0x7f1cb40b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:56.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.869+0000 7f1cb8dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb40cd730 con 0x7f1cb4159200 2024-01-11T09:46:56.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.869+0000 7f1cb8dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cb4159d30 con 0x7f1cb40b03e0 2024-01-11T09:46:56.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.869+0000 7f1cb8dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cb40ccb20 con 0x7f1cb40b07b0 2024-01-11T09:46:56.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.869+0000 7f1cb259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1cb40b03e0 0x7f1cb40b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:56.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.870+0000 7f1cb2d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1cb4159200 0x7f1cb4159620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44682/0 (socket says 172.21.15.27:44682) 2024-01-11T09:46:56.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.870+0000 7f1cb259c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1cb40b03e0 0x7f1cb40b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41720/0 (socket says 172.21.15.27:41720) 2024-01-11T09:46:56.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.870+0000 7f1cb2d9d700 1 -- 172.21.15.27:0/722645811 learned_addr learned my addr 172.21.15.27:0/722645811 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:56.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.870+0000 7f1cb1d9b700 1 --2- 172.21.15.27:0/722645811 >> v2:172.21.15.77:3300/0 conn(0x7f1cb40b07b0 0x7f1cb40b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:56.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.870+0000 7f1cb259c700 1 -- 172.21.15.27:0/722645811 >> v2:172.21.15.77:3300/0 conn(0x7f1cb40b07b0 msgr2=0x7f1cb40b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:56.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.870+0000 7f1cb259c700 1 --2- 172.21.15.27:0/722645811 >> v2:172.21.15.77:3300/0 conn(0x7f1cb40b07b0 0x7f1cb40b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:56.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.870+0000 7f1cb259c700 1 -- 172.21.15.27:0/722645811 >> v1:172.21.15.27:6789/0 conn(0x7f1cb4159200 legacy=0x7f1cb4159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:56.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.870+0000 7f1cb259c700 1 -- 172.21.15.27:0/722645811 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb40cce00 con 0x7f1cb40b03e0 2024-01-11T09:46:56.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.870+0000 7f1cb259c700 1 --2- 172.21.15.27:0/722645811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cb40b03e0 0x7f1cb40b2480 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f1cb40cdfc0 tx=0x7f1ca40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e57fbf44b5f6024 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:56.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.870+0000 7f1cb159a700 1 -- 172.21.15.27:0/722645811 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4011630 con 0x7f1cb40b03e0 2024-01-11T09:46:56.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.871+0000 7f1cb159a700 1 -- 172.21.15.27:0/722645811 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca4011790 con 0x7f1cb40b03e0 2024-01-11T09:46:56.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.871+0000 7f1cb159a700 1 -- 172.21.15.27:0/722645811 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4011b30 con 0x7f1cb40b03e0 2024-01-11T09:46:56.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.871+0000 7f1cb8dff700 1 -- 172.21.15.27:0/722645811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cb40b03e0 msgr2=0x7f1cb40b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:56.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.871+0000 7f1cb8dff700 1 --2- 172.21.15.27:0/722645811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cb40b03e0 0x7f1cb40b2480 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f1cb40cdfc0 tx=0x7f1ca40091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:56.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.871+0000 7f1cb8dff700 1 -- 172.21.15.27:0/722645811 shutdown_connections 2024-01-11T09:46:56.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.871+0000 7f1cb8dff700 1 --2- 172.21.15.27:0/722645811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cb40b03e0 0x7f1cb40b2480 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:56.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.871+0000 7f1cb8dff700 1 --2- 172.21.15.27:0/722645811 >> v2:172.21.15.77:3300/0 conn(0x7f1cb40b07b0 0x7f1cb40b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:56.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.871+0000 7f1cb8dff700 1 -- 172.21.15.27:0/722645811 >> 172.21.15.27:0/722645811 conn(0x7f1cb406ed40 msgr2=0x7f1cb406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:56.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.871+0000 7f1cb8dff700 1 -- 172.21.15.27:0/722645811 shutdown_connections 2024-01-11T09:46:56.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.871+0000 7f1cb8dff700 1 -- 172.21.15.27:0/722645811 wait complete. 2024-01-11T09:46:56.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.873+0000 7f1cb8dff700 1 Processor -- start 2024-01-11T09:46:56.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.873+0000 7f1cb8dff700 1 -- start start 2024-01-11T09:46:56.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.874+0000 7f1cb8dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cb40b03e0 0x7f1cb41581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:56.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.874+0000 7f1cb8dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1cb40b07b0 0x7f1cb4169930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:56.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.874+0000 7f1cb8dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cb4159200 0x7f1cb40c6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:56.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.874+0000 7f1cb8dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb4158730 con 0x7f1cb4159200 2024-01-11T09:46:56.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.874+0000 7f1cb8dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb40cfe30 con 0x7f1cb40b03e0 2024-01-11T09:46:56.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.874+0000 7f1cb8dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb4169e70 con 0x7f1cb40b07b0 2024-01-11T09:46:56.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.874+0000 7f1cb2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cb4159200 0x7f1cb40c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:56.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.874+0000 7f1cb259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cb40b03e0 0x7f1cb41581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:56.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.874+0000 7f1cb259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cb40b03e0 0x7f1cb41581f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:42736/0 (socket says 172.21.15.27:42736) 2024-01-11T09:46:56.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.874+0000 7f1cb259c700 1 -- 172.21.15.27:0/2309660496 learned_addr learned my addr 172.21.15.27:0/2309660496 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:56.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.874+0000 7f1cb1d9b700 1 --2- 172.21.15.27:0/2309660496 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1cb40b07b0 0x7f1cb4169930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:56.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.875+0000 7f1cb259c700 1 -- 172.21.15.27:0/2309660496 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1cb40b07b0 msgr2=0x7f1cb4169930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:56.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.875+0000 7f1cb259c700 1 --2- 172.21.15.27:0/2309660496 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1cb40b07b0 0x7f1cb4169930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:56.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.875+0000 7f1cb259c700 1 -- 172.21.15.27:0/2309660496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cb4159200 msgr2=0x7f1cb40c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:56.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.875+0000 7f1cb259c700 1 --2- 172.21.15.27:0/2309660496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cb4159200 0x7f1cb40c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:56.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.875+0000 7f1cb259c700 1 -- 172.21.15.27:0/2309660496 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca4009420 con 0x7f1cb40b03e0 2024-01-11T09:46:56.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.875+0000 7f1cb259c700 1 --2- 172.21.15.27:0/2309660496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cb40b03e0 0x7f1cb41581f0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4069540 tx=0x7f1ca4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:56.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.875+0000 7f1ca27fc700 1 -- 172.21.15.27:0/2309660496 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4000d10 con 0x7f1cb40b03e0 2024-01-11T09:46:56.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.875+0000 7f1ca27fc700 1 -- 172.21.15.27:0/2309660496 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca4011b10 con 0x7f1cb40b03e0 2024-01-11T09:46:56.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.875+0000 7f1ca27fc700 1 -- 172.21.15.27:0/2309660496 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4025640 con 0x7f1cb40b03e0 2024-01-11T09:46:56.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.875+0000 7f1cb8dff700 1 -- 172.21.15.27:0/2309660496 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb40cf220 con 0x7f1cb40b03e0 2024-01-11T09:46:56.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.876+0000 7f1cb8dff700 1 -- 172.21.15.27:0/2309660496 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb40cf7e0 con 0x7f1cb40b03e0 2024-01-11T09:46:56.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.877+0000 7f1cb8dff700 1 -- 172.21.15.27:0/2309660496 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cb40718d0 con 0x7f1cb40b03e0 2024-01-11T09:46:56.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.877+0000 7f1ca27fc700 1 -- 172.21.15.27:0/2309660496 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 141) v1 ==== 84947+0+0 (secure 0 0 0) 0x7f1ca4025a30 con 0x7f1cb40b03e0 2024-01-11T09:46:56.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.879+0000 7f1ca27fc700 1 --2- 172.21.15.27:0/2309660496 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1c94067170 0x7f1c94069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:56.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.879+0000 7f1ca27fc700 1 -- 172.21.15.27:0/2309660496 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 7111+0+0 (secure 0 0 0) 0x7f1ca4086670 con 0x7f1cb40b03e0 2024-01-11T09:46:56.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.879+0000 7f1cb1d9b700 1 -- 172.21.15.27:0/2309660496 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1c94067170 msgr2=0x7f1c94069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:56.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.879+0000 7f1cb1d9b700 1 --2- 172.21.15.27:0/2309660496 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1c94067170 0x7f1c94069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:46:56.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:56.883+0000 7f1ca27fc700 1 -- 172.21.15.27:0/2309660496 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1ca4053030 con 0x7f1cb40b03e0 2024-01-11T09:46:56.973 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:56.982+0000 7f923a65f1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:46:57.017 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:57.025+0000 7f95c8a461c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:46:57.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.080+0000 7f1cb1d9b700 1 -- 172.21.15.27:0/2309660496 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1c94067170 msgr2=0x7f1c94069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:57.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.080+0000 7f1cb1d9b700 1 --2- 172.21.15.27:0/2309660496 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1c94067170 0x7f1c94069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:46:57.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.136+0000 7f1cb8dff700 1 -- 172.21.15.27:0/2309660496 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cb40cd0e0 con 0x7f1cb40b03e0 2024-01-11T09:46:57.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.137+0000 7f1ca27fc700 1 -- 172.21.15.27:0/2309660496 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5688 (secure 0 0 0) 0x7f1ca4056650 con 0x7f1cb40b03e0 2024-01-11T09:46:57.129 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:57.129 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":132,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1568813392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1568813392},{"type":"v1","addr":"172.21.15.77:6834","nonce":1568813392}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122},{"gid":10969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1352491019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1352491019},{"type":"v1","addr":"172.21.15.77:6835","nonce":1352491019}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:46:25.726042+0000","modified":"2024-01-11T09:46:29.257798+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11340":{"gid":11340,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/321681407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":321681407},{"type":"v1","addr":"172.21.15.27:6835","nonce":321681407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:46:39.483941+0000","modified":"2024-01-11T09:46:42.996786+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11331},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11331":{"gid":11331,"name":"a","rank":0,"incarnation":129,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6834/3477661846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3477661846},{"type":"v1","addr":"172.21.15.27:6834","nonce":3477661846}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":16}]} 2024-01-11T09:46:57.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.140+0000 7f1cb8dff700 1 -- 172.21.15.27:0/2309660496 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1c94067170 msgr2=0x7f1c94069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:46:57.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.140+0000 7f1cb8dff700 1 --2- 172.21.15.27:0/2309660496 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1c94067170 0x7f1c94069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.140+0000 7f1cb8dff700 1 -- 172.21.15.27:0/2309660496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cb40b03e0 msgr2=0x7f1cb41581f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:57.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.140+0000 7f1cb8dff700 1 --2- 172.21.15.27:0/2309660496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cb40b03e0 0x7f1cb41581f0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4069540 tx=0x7f1ca4017000 comp rx=0 tx=0).stop 2024-01-11T09:46:57.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.140+0000 7f1cb8dff700 1 -- 172.21.15.27:0/2309660496 shutdown_connections 2024-01-11T09:46:57.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.140+0000 7f1cb8dff700 1 --2- 172.21.15.27:0/2309660496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cb4159200 0x7f1cb40c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.140+0000 7f1cb8dff700 1 --2- 172.21.15.27:0/2309660496 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1c94067170 0x7f1c94069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.140+0000 7f1cb8dff700 1 --2- 172.21.15.27:0/2309660496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cb40b03e0 0x7f1cb41581f0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.140+0000 7f1cb8dff700 1 --2- 172.21.15.27:0/2309660496 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1cb40b07b0 0x7f1cb4169930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.140+0000 7f1cb8dff700 1 -- 172.21.15.27:0/2309660496 >> 172.21.15.27:0/2309660496 conn(0x7f1cb406ed40 msgr2=0x7f1cb415c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:57.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.140+0000 7f1cb8dff700 1 -- 172.21.15.27:0/2309660496 shutdown_connections 2024-01-11T09:46:57.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.140+0000 7f1cb8dff700 1 -- 172.21.15.27:0/2309660496 wait complete. 2024-01-11T09:46:57.136 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 132 2024-01-11T09:46:57.152 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:46:57.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.409+0000 7f0465fff700 1 Processor -- start 2024-01-11T09:46:57.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.409+0000 7f0465fff700 1 -- start start 2024-01-11T09:46:57.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.411+0000 7f0465fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f046015d8e0 0x7f04600aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:57.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.411+0000 7f0465fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f04600af9c0 0x7f04600b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:57.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.411+0000 7f0465fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f046015ec00 con 0x7f046015dcb0 2024-01-11T09:46:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.411+0000 7f0465fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f046015eea0 con 0x7f04600af9c0 2024-01-11T09:46:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.411+0000 7f0465fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f046015f180 con 0x7f046015d8e0 2024-01-11T09:46:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.411+0000 7f045effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f04600af9c0 0x7f04600b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.411+0000 7f045f7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f046015d8e0 0x7f04600aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.411+0000 7f045effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f04600af9c0 0x7f04600b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41740/0 (socket says 172.21.15.27:41740) 2024-01-11T09:46:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.411+0000 7f045ffff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f046015dcb0 0x7f04601616e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:58958/0 (socket says 172.21.15.27:58958) 2024-01-11T09:46:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.411+0000 7f045effd700 1 -- 172.21.15.27:0/462219780 learned_addr learned my addr 172.21.15.27:0/462219780 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.412+0000 7f045f7fe700 1 -- 172.21.15.27:0/462219780 >> v2:172.21.15.27:3300/0 conn(0x7f04600af9c0 msgr2=0x7f04600b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.412+0000 7f045f7fe700 1 --2- 172.21.15.27:0/462219780 >> v2:172.21.15.27:3300/0 conn(0x7f04600af9c0 0x7f04600b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.412+0000 7f045f7fe700 1 -- 172.21.15.27:0/462219780 >> v1:172.21.15.77:6789/0 conn(0x7f046015dcb0 legacy=0x7f04601616e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.412+0000 7f045f7fe700 1 -- 172.21.15.27:0/462219780 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046015f460 con 0x7f046015d8e0 2024-01-11T09:46:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.412+0000 7f045e7fc700 1 -- 172.21.15.27:0/462219780 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652278239 0 0) 0x7f046015ec00 con 0x7f046015dcb0 2024-01-11T09:46:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.412+0000 7f045f7fe700 1 --2- 172.21.15.27:0/462219780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f046015d8e0 0x7f04600aefb0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f0460070d50 tx=0x7f04500091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77e343d4f3695dca server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:57.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.412+0000 7f045e7fc700 1 -- 172.21.15.27:0/462219780 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450011630 con 0x7f046015d8e0 2024-01-11T09:46:57.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.413+0000 7f045e7fc700 1 -- 172.21.15.27:0/462219780 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0450011790 con 0x7f046015d8e0 2024-01-11T09:46:57.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.413+0000 7f045e7fc700 1 -- 172.21.15.27:0/462219780 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450011b30 con 0x7f046015d8e0 2024-01-11T09:46:57.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.413+0000 7f0465fff700 1 -- 172.21.15.27:0/462219780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f046015d8e0 msgr2=0x7f04600aefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:57.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.413+0000 7f0465fff700 1 --2- 172.21.15.27:0/462219780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f046015d8e0 0x7f04600aefb0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f0460070d50 tx=0x7f04500091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.413+0000 7f0465fff700 1 -- 172.21.15.27:0/462219780 shutdown_connections 2024-01-11T09:46:57.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.413+0000 7f0465fff700 1 --2- 172.21.15.27:0/462219780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f046015d8e0 0x7f04600aefb0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.413+0000 7f0465fff700 1 --2- 172.21.15.27:0/462219780 >> v2:172.21.15.27:3300/0 conn(0x7f04600af9c0 0x7f04600b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.413+0000 7f0465fff700 1 -- 172.21.15.27:0/462219780 >> 172.21.15.27:0/462219780 conn(0x7f04600680e0 msgr2=0x7f0460069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:57.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.413+0000 7f0465fff700 1 -- 172.21.15.27:0/462219780 shutdown_connections 2024-01-11T09:46:57.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.413+0000 7f0465fff700 1 -- 172.21.15.27:0/462219780 wait complete. 2024-01-11T09:46:57.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.415+0000 7f0465fff700 1 Processor -- start 2024-01-11T09:46:57.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.416+0000 7f0465fff700 1 -- start start 2024-01-11T09:46:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.417+0000 7f0465fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04600af9c0 0x7f046014d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.417+0000 7f0465fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f046015d8e0 0x7f0460151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.417+0000 7f0465fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f046015dcb0 0x7f046006e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.417+0000 7f0465fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0460150450 con 0x7f046015dcb0 2024-01-11T09:46:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.417+0000 7f0465fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04601536d0 con 0x7f046015d8e0 2024-01-11T09:46:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.417+0000 7f0465fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046014fbf0 con 0x7f04600af9c0 2024-01-11T09:46:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.417+0000 7f045effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f046015d8e0 0x7f0460151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.417+0000 7f045f7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04600af9c0 0x7f046014d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.417+0000 7f045effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f046015d8e0 0x7f0460151240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:42752/0 (socket says 172.21.15.27:42752) 2024-01-11T09:46:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.417+0000 7f045effd700 1 -- 172.21.15.27:0/4166522555 learned_addr learned my addr 172.21.15.27:0/4166522555 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.417+0000 7f045ffff700 1 --2- 172.21.15.27:0/4166522555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f046015dcb0 0x7f046006e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.418+0000 7f045f7fe700 1 -- 172.21.15.27:0/4166522555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f046015d8e0 msgr2=0x7f0460151240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.418+0000 7f045f7fe700 1 --2- 172.21.15.27:0/4166522555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f046015d8e0 0x7f0460151240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.418+0000 7f045f7fe700 1 -- 172.21.15.27:0/4166522555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f046015dcb0 msgr2=0x7f046006e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.418+0000 7f045f7fe700 1 --2- 172.21.15.27:0/4166522555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f046015dcb0 0x7f046006e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.418+0000 7f045f7fe700 1 -- 172.21.15.27:0/4166522555 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0450009420 con 0x7f04600af9c0 2024-01-11T09:46:57.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.418+0000 7f045effd700 1 --2- 172.21.15.27:0/4166522555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f046015d8e0 0x7f0460151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:57.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.418+0000 7f045f7fe700 1 --2- 172.21.15.27:0/4166522555 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04600af9c0 0x7f046014d530 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0450016000 tx=0x7f0450017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:57.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.419+0000 7f04477fe700 1 -- 172.21.15.27:0/4166522555 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450000d10 con 0x7f04600af9c0 2024-01-11T09:46:57.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.419+0000 7f04477fe700 1 -- 172.21.15.27:0/4166522555 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0450011b10 con 0x7f04600af9c0 2024-01-11T09:46:57.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.419+0000 7f0465fff700 1 -- 172.21.15.27:0/4166522555 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04601533a0 con 0x7f04600af9c0 2024-01-11T09:46:57.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.419+0000 7f04477fe700 1 -- 172.21.15.27:0/4166522555 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450025640 con 0x7f04600af9c0 2024-01-11T09:46:57.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.419+0000 7f0465fff700 1 -- 172.21.15.27:0/4166522555 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0460150170 con 0x7f04600af9c0 2024-01-11T09:46:57.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.420+0000 7f0465fff700 1 -- 172.21.15.27:0/4166522555 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04600718d0 con 0x7f04600af9c0 2024-01-11T09:46:57.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.421+0000 7f04477fe700 1 -- 172.21.15.27:0/4166522555 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 141) v1 ==== 84947+0+0 (secure 0 0 0) 0x7f0450025a30 con 0x7f04600af9c0 2024-01-11T09:46:57.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.422+0000 7f04477fe700 1 --2- 172.21.15.27:0/4166522555 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f044c067170 0x7f044c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:57.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.422+0000 7f04477fe700 1 -- 172.21.15.27:0/4166522555 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 7111+0+0 (secure 0 0 0) 0x7f0450086b80 con 0x7f04600af9c0 2024-01-11T09:46:57.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.422+0000 7f045effd700 1 -- 172.21.15.27:0/4166522555 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f044c067170 msgr2=0x7f044c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:57.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.422+0000 7f045effd700 1 --2- 172.21.15.27:0/4166522555 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f044c067170 0x7f044c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:46:57.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.425+0000 7f04477fe700 1 -- 172.21.15.27:0/4166522555 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f04500535c0 con 0x7f04600af9c0 2024-01-11T09:46:57.508 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:57.517+0000 7f923a65f1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:46:57.585 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:57.594+0000 7f95c8a461c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:46:57.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.623+0000 7f045effd700 1 -- 172.21.15.27:0/4166522555 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f044c067170 msgr2=0x7f044c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:57.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.623+0000 7f045effd700 1 --2- 172.21.15.27:0/4166522555 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f044c067170 0x7f044c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:46:57.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.632+0000 7f0465fff700 1 -- 172.21.15.27:0/4166522555 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f046015f740 con 0x7f04600af9c0 2024-01-11T09:46:57.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.634+0000 7f04477fe700 1 -- 172.21.15.27:0/4166522555 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v240) v1 ==== 74+0+19338 (secure 0 0 0) 0x7f045004dba0 con 0x7f04600af9c0 2024-01-11T09:46:57.625 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:46:57.626 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":240,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:46:54.200005+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:46:24.049904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-01-11T09:46:24.737495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-01-11T09:46:37.665603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"238","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-01-11T09:46:38.472134+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"239","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/210673663":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/3835722213":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/696729061":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/206349247":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/3398849669":"2024-01-12T09:46:54.199949+0000","172.21.15.77:6837/59776":"2024-01-12T09:46:54.199949+0000","172.21.15.77:6836/59776":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/1934793537":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/1366958778":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/198451994":"2024-01-12T09:46:54.199949+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:46:57.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.636+0000 7f0465fff700 1 -- 172.21.15.27:0/4166522555 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f044c067170 msgr2=0x7f044c069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:46:57.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.636+0000 7f0465fff700 1 --2- 172.21.15.27:0/4166522555 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f044c067170 0x7f044c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.636+0000 7f0465fff700 1 -- 172.21.15.27:0/4166522555 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04600af9c0 msgr2=0x7f046014d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:57.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.636+0000 7f0465fff700 1 --2- 172.21.15.27:0/4166522555 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04600af9c0 0x7f046014d530 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0450016000 tx=0x7f0450017000 comp rx=0 tx=0).stop 2024-01-11T09:46:57.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.636+0000 7f0465fff700 1 -- 172.21.15.27:0/4166522555 shutdown_connections 2024-01-11T09:46:57.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.636+0000 7f0465fff700 1 --2- 172.21.15.27:0/4166522555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f046015dcb0 0x7f046006e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.636+0000 7f0465fff700 1 --2- 172.21.15.27:0/4166522555 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f04600af9c0 0x7f046014d530 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.636+0000 7f0465fff700 1 --2- 172.21.15.27:0/4166522555 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f044c067170 0x7f044c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.636+0000 7f0465fff700 1 --2- 172.21.15.27:0/4166522555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f046015d8e0 0x7f0460151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.636+0000 7f0465fff700 1 -- 172.21.15.27:0/4166522555 >> 172.21.15.27:0/4166522555 conn(0x7f04600680e0 msgr2=0x7f046014de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:57.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.636+0000 7f0465fff700 1 -- 172.21.15.27:0/4166522555 shutdown_connections 2024-01-11T09:46:57.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.636+0000 7f0465fff700 1 -- 172.21.15.27:0/4166522555 wait complete. 2024-01-11T09:46:57.647 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T09:46:24.049904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '234', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T09:46:24.737495+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '235', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-01-11T09:46:37.665603+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '238', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2024-01-11T09:46:38.472134+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '239', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}}] 2024-01-11T09:46:57.648 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T09:46:57.680 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:57.689+0000 7f923a65f1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:46:57.773 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:57.781+0000 7f95c8a461c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:46:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.863+0000 7fc7d61ff700 1 Processor -- start 2024-01-11T09:46:57.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.863+0000 7fc7d61ff700 1 -- start start 2024-01-11T09:46:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.864+0000 7fc7d61ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7d0142dc0 0x7fc7d00cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.864+0000 7fc7d61ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7d00cb2e0 0x7fc7d00cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.864+0000 7fc7d61ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d00ccc70 con 0x7fc7d0143190 2024-01-11T09:46:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.864+0000 7fc7d61ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d015df30 con 0x7fc7d0142dc0 2024-01-11T09:46:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.864+0000 7fc7d61ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d00cb740 con 0x7fc7d00cb2e0 2024-01-11T09:46:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.865+0000 7fc7cf7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7d0142dc0 0x7fc7d00cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.865+0000 7fc7cf7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7d0142dc0 0x7fc7d00cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41764/0 (socket says 172.21.15.27:41764) 2024-01-11T09:46:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.865+0000 7fc7cf7fe700 1 -- 172.21.15.27:0/1905303548 learned_addr learned my addr 172.21.15.27:0/1905303548 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.865+0000 7fc7cffff700 1 --2- 172.21.15.27:0/1905303548 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7d00cb2e0 0x7fc7d00cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.865+0000 7fc7cf7fe700 1 -- 172.21.15.27:0/1905303548 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7d00cb2e0 msgr2=0x7fc7d00cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.865+0000 7fc7cf7fe700 1 --2- 172.21.15.27:0/1905303548 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7d00cb2e0 0x7fc7d00cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.865+0000 7fc7cf7fe700 1 -- 172.21.15.27:0/1905303548 >> v1:172.21.15.77:6789/0 conn(0x7fc7d0143190 legacy=0x7fc7d00cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.865+0000 7fc7cf7fe700 1 -- 172.21.15.27:0/1905303548 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d015d320 con 0x7fc7d0142dc0 2024-01-11T09:46:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.865+0000 7fc7ce7fc700 1 -- 172.21.15.27:0/1905303548 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267276835 0 0) 0x7fc7d00ccc70 con 0x7fc7d0143190 2024-01-11T09:46:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.865+0000 7fc7cf7fe700 1 --2- 172.21.15.27:0/1905303548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7d0142dc0 0x7fc7d00cada0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00099b0 tx=0x7fc7c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa083d167de16fe server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.866+0000 7fc7ce7fc700 1 -- 172.21.15.27:0/1905303548 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c0009ea0 con 0x7fc7d0142dc0 2024-01-11T09:46:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.866+0000 7fc7ce7fc700 1 -- 172.21.15.27:0/1905303548 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c000b860 con 0x7fc7d0142dc0 2024-01-11T09:46:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.866+0000 7fc7ce7fc700 1 -- 172.21.15.27:0/1905303548 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c000bc00 con 0x7fc7d0142dc0 2024-01-11T09:46:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.866+0000 7fc7d61ff700 1 -- 172.21.15.27:0/1905303548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7d0142dc0 msgr2=0x7fc7d00cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.866+0000 7fc7d61ff700 1 --2- 172.21.15.27:0/1905303548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7d0142dc0 0x7fc7d00cada0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00099b0 tx=0x7fc7c0009250 comp rx=0 tx=0).stop 2024-01-11T09:46:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.866+0000 7fc7d61ff700 1 -- 172.21.15.27:0/1905303548 shutdown_connections 2024-01-11T09:46:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.866+0000 7fc7d61ff700 1 --2- 172.21.15.27:0/1905303548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7d0142dc0 0x7fc7d00cada0 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.866+0000 7fc7d61ff700 1 --2- 172.21.15.27:0/1905303548 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7d00cb2e0 0x7fc7d00cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.866+0000 7fc7d61ff700 1 -- 172.21.15.27:0/1905303548 >> 172.21.15.27:0/1905303548 conn(0x7fc7d0063ca0 msgr2=0x7fc7d006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.866+0000 7fc7d61ff700 1 -- 172.21.15.27:0/1905303548 shutdown_connections 2024-01-11T09:46:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.867+0000 7fc7d61ff700 1 -- 172.21.15.27:0/1905303548 wait complete. 2024-01-11T09:46:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.869+0000 7fc7d61ff700 1 Processor -- start 2024-01-11T09:46:57.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.869+0000 7fc7d61ff700 1 -- start start 2024-01-11T09:46:57.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.870+0000 7fc7d61ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7d00cb2e0 0x7fc7d0169860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:57.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.870+0000 7fc7d61ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7d0142dc0 0x7fc7d016c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.870+0000 7fc7d61ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7d0143190 0x7fc7d0140910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.870+0000 7fc7d61ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d01581b0 con 0x7fc7d00cb2e0 2024-01-11T09:46:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.870+0000 7fc7d61ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d015c280 con 0x7fc7d0143190 2024-01-11T09:46:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.870+0000 7fc7d61ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d0157880 con 0x7fc7d0142dc0 2024-01-11T09:46:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.870+0000 7fc7cf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7d00cb2e0 0x7fc7d0169860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.870+0000 7fc7cf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7d00cb2e0 0x7fc7d0169860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41778/0 (socket says 172.21.15.27:41778) 2024-01-11T09:46:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.870+0000 7fc7cf7fe700 1 -- 172.21.15.27:0/3509052456 learned_addr learned my addr 172.21.15.27:0/3509052456 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.871+0000 7fc7ceffd700 1 --2- 172.21.15.27:0/3509052456 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7d0142dc0 0x7fc7d016c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.871+0000 7fc7cffff700 1 --2- 172.21.15.27:0/3509052456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7d0143190 0x7fc7d0140910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.871+0000 7fc7cf7fe700 1 -- 172.21.15.27:0/3509052456 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7d0142dc0 msgr2=0x7fc7d016c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.871+0000 7fc7cf7fe700 1 --2- 172.21.15.27:0/3509052456 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7d0142dc0 0x7fc7d016c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.871+0000 7fc7cf7fe700 1 -- 172.21.15.27:0/3509052456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7d0143190 msgr2=0x7fc7d0140910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:57.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.871+0000 7fc7cf7fe700 1 --2- 172.21.15.27:0/3509052456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7d0143190 0x7fc7d0140910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:57.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.871+0000 7fc7cf7fe700 1 -- 172.21.15.27:0/3509052456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c00094f0 con 0x7fc7d00cb2e0 2024-01-11T09:46:57.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.871+0000 7fc7cf7fe700 1 --2- 172.21.15.27:0/3509052456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7d00cb2e0 0x7fc7d0169860 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0064a70 tx=0x7fc7c0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:57.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.872+0000 7fc7b77fe700 1 -- 172.21.15.27:0/3509052456 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c0000d10 con 0x7fc7d00cb2e0 2024-01-11T09:46:57.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.872+0000 7fc7b77fe700 1 -- 172.21.15.27:0/3509052456 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c000bbe0 con 0x7fc7d00cb2e0 2024-01-11T09:46:57.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.872+0000 7fc7d61ff700 1 -- 172.21.15.27:0/3509052456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d0157b60 con 0x7fc7d00cb2e0 2024-01-11T09:46:57.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.872+0000 7fc7b77fe700 1 -- 172.21.15.27:0/3509052456 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c0011690 con 0x7fc7d00cb2e0 2024-01-11T09:46:57.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.872+0000 7fc7d61ff700 1 -- 172.21.15.27:0/3509052456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d016cad0 con 0x7fc7d00cb2e0 2024-01-11T09:46:57.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.873+0000 7fc7d61ff700 1 -- 172.21.15.27:0/3509052456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d004c3c0 con 0x7fc7d00cb2e0 2024-01-11T09:46:57.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.874+0000 7fc7b77fe700 1 -- 172.21.15.27:0/3509052456 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 141) v1 ==== 84947+0+0 (secure 0 0 0) 0x7fc7c0011ac0 con 0x7fc7d00cb2e0 2024-01-11T09:46:57.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.875+0000 7fc7b77fe700 1 --2- 172.21.15.27:0/3509052456 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc7bc067170 0x7fc7bc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:57.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.875+0000 7fc7b77fe700 1 -- 172.21.15.27:0/3509052456 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 7111+0+0 (secure 0 0 0) 0x7fc7c0086710 con 0x7fc7d00cb2e0 2024-01-11T09:46:57.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.875+0000 7fc7ceffd700 1 -- 172.21.15.27:0/3509052456 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc7bc067170 msgr2=0x7fc7bc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:57.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.875+0000 7fc7ceffd700 1 --2- 172.21.15.27:0/3509052456 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc7bc067170 0x7fc7bc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:46:57.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:57.879+0000 7fc7b77fe700 1 -- 172.21.15.27:0/3509052456 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc7c00530d0 con 0x7fc7d00cb2e0 2024-01-11T09:46:58.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.076+0000 7fc7ceffd700 1 -- 172.21.15.27:0/3509052456 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc7bc067170 msgr2=0x7fc7bc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:58.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.076+0000 7fc7ceffd700 1 --2- 172.21.15.27:0/3509052456 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc7bc067170 0x7fc7bc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:46:58.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.086+0000 7fc7d61ff700 1 -- 172.21.15.27:0/3509052456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc7d015d590 con 0x7fc7d00cb2e0 2024-01-11T09:46:58.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.087+0000 7fc7b77fe700 1 -- 172.21.15.27:0/3509052456 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v240) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc7c00566f0 con 0x7fc7d00cb2e0 2024-01-11T09:46:58.078 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T09:46:58.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.089+0000 7fc7d61ff700 1 -- 172.21.15.27:0/3509052456 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc7bc067170 msgr2=0x7fc7bc069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:46:58.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.089+0000 7fc7d61ff700 1 --2- 172.21.15.27:0/3509052456 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc7bc067170 0x7fc7bc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.089+0000 7fc7d61ff700 1 -- 172.21.15.27:0/3509052456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7d00cb2e0 msgr2=0x7fc7d0169860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:58.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.089+0000 7fc7d61ff700 1 --2- 172.21.15.27:0/3509052456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7d00cb2e0 0x7fc7d0169860 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0064a70 tx=0x7fc7c0017000 comp rx=0 tx=0).stop 2024-01-11T09:46:58.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.089+0000 7fc7d61ff700 1 -- 172.21.15.27:0/3509052456 shutdown_connections 2024-01-11T09:46:58.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.089+0000 7fc7d61ff700 1 --2- 172.21.15.27:0/3509052456 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc7bc067170 0x7fc7bc069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.089+0000 7fc7d61ff700 1 --2- 172.21.15.27:0/3509052456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7d0143190 0x7fc7d0140910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.089+0000 7fc7d61ff700 1 --2- 172.21.15.27:0/3509052456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7d00cb2e0 0x7fc7d0169860 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.089+0000 7fc7d61ff700 1 --2- 172.21.15.27:0/3509052456 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc7d0142dc0 0x7fc7d016c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.089+0000 7fc7d61ff700 1 -- 172.21.15.27:0/3509052456 >> 172.21.15.27:0/3509052456 conn(0x7fc7d0063ca0 msgr2=0x7fc7d0141cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:58.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.089+0000 7fc7d61ff700 1 -- 172.21.15.27:0/3509052456 shutdown_connections 2024-01-11T09:46:58.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.089+0000 7fc7d61ff700 1 -- 172.21.15.27:0/3509052456 wait complete. 2024-01-11T09:46:58.101 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T09:46:58.101 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:58.110+0000 7f923a65f1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:46:58.232 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:58.240+0000 7f95c8a461c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:46:58.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.316+0000 7febce3ff700 1 Processor -- start 2024-01-11T09:46:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.316+0000 7febce3ff700 1 -- start start 2024-01-11T09:46:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.317+0000 7febce3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7febc8157750 0x7febc80af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.317+0000 7febce3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc80cb3f0 con 0x7febc8068620 2024-01-11T09:46:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.317+0000 7febce3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc80afdf0 con 0x7febc8157380 2024-01-11T09:46:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.317+0000 7febce3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febc806ef70 con 0x7febc8157750 2024-01-11T09:46:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.318+0000 7febc77fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7febc8157750 0x7febc80af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.318+0000 7febcc99c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7febc8068620 0x7febc80cb880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44698/0 (socket says 172.21.15.27:44698) 2024-01-11T09:46:58.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.318+0000 7febcc99c700 1 -- 172.21.15.27:0/2978990059 learned_addr learned my addr 172.21.15.27:0/2978990059 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:58.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.318+0000 7febc77fe700 1 -- 172.21.15.27:0/2978990059 >> v1:172.21.15.77:6789/0 conn(0x7febc8157380 legacy=0x7febc806e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:58.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.318+0000 7febc77fe700 1 -- 172.21.15.27:0/2978990059 >> v1:172.21.15.27:6789/0 conn(0x7febc8068620 legacy=0x7febc80cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:58.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.318+0000 7febc77fe700 1 -- 172.21.15.27:0/2978990059 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc80ca7e0 con 0x7febc8157750 2024-01-11T09:46:58.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.319+0000 7febc77fe700 1 --2- 172.21.15.27:0/2978990059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febc8157750 0x7febc80af8b0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7febbc0098e0 tx=0x7febbc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2b0e4c80c38347a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:58.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.319+0000 7febc6ffd700 1 -- 172.21.15.27:0/2978990059 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febbc009e80 con 0x7febc8157750 2024-01-11T09:46:58.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.319+0000 7febc6ffd700 1 -- 172.21.15.27:0/2978990059 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febbc00b860 con 0x7febc8157750 2024-01-11T09:46:58.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.319+0000 7febc6ffd700 1 -- 172.21.15.27:0/2978990059 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febbc00bbe0 con 0x7febc8157750 2024-01-11T09:46:58.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.320+0000 7febce3ff700 1 -- 172.21.15.27:0/2978990059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febc8157750 msgr2=0x7febc80af8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:58.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.320+0000 7febce3ff700 1 --2- 172.21.15.27:0/2978990059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febc8157750 0x7febc80af8b0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7febbc0098e0 tx=0x7febbc0091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.320+0000 7febce3ff700 1 -- 172.21.15.27:0/2978990059 shutdown_connections 2024-01-11T09:46:58.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.320+0000 7febce3ff700 1 --2- 172.21.15.27:0/2978990059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febc8157750 0x7febc80af8b0 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.320+0000 7febce3ff700 1 -- 172.21.15.27:0/2978990059 >> 172.21.15.27:0/2978990059 conn(0x7febc80c9fb0 msgr2=0x7febc806ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:58.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.320+0000 7febce3ff700 1 -- 172.21.15.27:0/2978990059 shutdown_connections 2024-01-11T09:46:58.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.320+0000 7febce3ff700 1 -- 172.21.15.27:0/2978990059 wait complete. 2024-01-11T09:46:58.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.321+0000 7febce3ff700 1 Processor -- start 2024-01-11T09:46:58.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.321+0000 7febce3ff700 1 -- start start 2024-01-11T09:46:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.322+0000 7febce3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febc8068620 0x7febc816c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.322+0000 7febce3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7febc8157380 0x7febc80addc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.322+0000 7febce3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febc8157750 0x7febc80ae300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.322+0000 7febce3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc81594b0 con 0x7febc8157750 2024-01-11T09:46:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.322+0000 7febce3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc816cec0 con 0x7febc8068620 2024-01-11T09:46:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.322+0000 7febce3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc81588a0 con 0x7febc8157380 2024-01-11T09:46:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.323+0000 7febcc99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febc8157750 0x7febc80ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.323+0000 7febcc99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febc8157750 0x7febc80ae300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41794/0 (socket says 172.21.15.27:41794) 2024-01-11T09:46:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.323+0000 7febcc99c700 1 -- 172.21.15.27:0/551497184 learned_addr learned my addr 172.21.15.27:0/551497184 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.323+0000 7febc7fff700 1 --2- 172.21.15.27:0/551497184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febc8068620 0x7febc816c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:58.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.323+0000 7febc77fe700 1 --2- 172.21.15.27:0/551497184 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7febc8157380 0x7febc80addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.323+0000 7febcc99c700 1 -- 172.21.15.27:0/551497184 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7febc8157380 msgr2=0x7febc80addc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.323+0000 7febcc99c700 1 --2- 172.21.15.27:0/551497184 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7febc8157380 0x7febc80addc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.323+0000 7febcc99c700 1 -- 172.21.15.27:0/551497184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febc8068620 msgr2=0x7febc816c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.323+0000 7febcc99c700 1 --2- 172.21.15.27:0/551497184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febc8068620 0x7febc816c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.323+0000 7febcc99c700 1 -- 172.21.15.27:0/551497184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febbc009420 con 0x7febc8157750 2024-01-11T09:46:58.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.324+0000 7febcc99c700 1 --2- 172.21.15.27:0/551497184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febc8157750 0x7febc80ae300 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7febc000cbc0 tx=0x7febc00019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.324+0000 7febaffff700 1 -- 172.21.15.27:0/551497184 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc000a7e0 con 0x7febc8157750 2024-01-11T09:46:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.324+0000 7febaffff700 1 -- 172.21.15.27:0/551497184 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc000a940 con 0x7febc8157750 2024-01-11T09:46:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.324+0000 7febaffff700 1 -- 172.21.15.27:0/551497184 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc000acc0 con 0x7febc8157750 2024-01-11T09:46:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.324+0000 7febce3ff700 1 -- 172.21.15.27:0/551497184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc8158b80 con 0x7febc8157750 2024-01-11T09:46:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.324+0000 7febce3ff700 1 -- 172.21.15.27:0/551497184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febc8159140 con 0x7febc8157750 2024-01-11T09:46:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.325+0000 7febce3ff700 1 -- 172.21.15.27:0/551497184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febc81595f0 con 0x7febc8157750 2024-01-11T09:46:58.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.326+0000 7febaffff700 1 -- 172.21.15.27:0/551497184 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 141) v1 ==== 84947+0+0 (secure 0 0 0) 0x7febc0007300 con 0x7febc8157750 2024-01-11T09:46:58.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.327+0000 7febaffff700 1 --2- 172.21.15.27:0/551497184 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7febb4067170 0x7febb4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:58.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.327+0000 7febaffff700 1 -- 172.21.15.27:0/551497184 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 7111+0+0 (secure 0 0 0) 0x7febc0085190 con 0x7febc8157750 2024-01-11T09:46:58.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.327+0000 7febc7fff700 1 -- 172.21.15.27:0/551497184 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7febb4067170 msgr2=0x7febb4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:58.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.327+0000 7febc7fff700 1 --2- 172.21.15.27:0/551497184 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7febb4067170 0x7febb4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:46:58.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.331+0000 7febaffff700 1 -- 172.21.15.27:0/551497184 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7febc81595f0 con 0x7febc8157750 2024-01-11T09:46:58.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.528+0000 7febc7fff700 1 -- 172.21.15.27:0/551497184 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7febb4067170 msgr2=0x7febb4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:58.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.528+0000 7febc7fff700 1 --2- 172.21.15.27:0/551497184 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7febb4067170 0x7febb4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:46:58.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.536+0000 7febce3ff700 1 -- 172.21.15.27:0/551497184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7febc81595f0 con 0x7febc8157750 2024-01-11T09:46:58.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.537+0000 7febaffff700 1 -- 172.21.15.27:0/551497184 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v240) v1 ==== 104+0+11 (secure 0 0 0) 0x7febc81595f0 con 0x7febc8157750 2024-01-11T09:46:58.529 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T09:46:58.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.540+0000 7febce3ff700 1 -- 172.21.15.27:0/551497184 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7febb4067170 msgr2=0x7febb4069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:46:58.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.540+0000 7febce3ff700 1 --2- 172.21.15.27:0/551497184 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7febb4067170 0x7febb4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.540+0000 7febce3ff700 1 -- 172.21.15.27:0/551497184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febc8157750 msgr2=0x7febc80ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:58.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.540+0000 7febce3ff700 1 --2- 172.21.15.27:0/551497184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febc8157750 0x7febc80ae300 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7febc000cbc0 tx=0x7febc00019c0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.540+0000 7febce3ff700 1 -- 172.21.15.27:0/551497184 shutdown_connections 2024-01-11T09:46:58.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.540+0000 7febce3ff700 1 --2- 172.21.15.27:0/551497184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febc8157750 0x7febc80ae300 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.540+0000 7febce3ff700 1 --2- 172.21.15.27:0/551497184 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7febb4067170 0x7febb4069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.540+0000 7febce3ff700 1 --2- 172.21.15.27:0/551497184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7febc8068620 0x7febc816c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.540+0000 7febce3ff700 1 --2- 172.21.15.27:0/551497184 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7febc8157380 0x7febc80addc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.540+0000 7febce3ff700 1 -- 172.21.15.27:0/551497184 >> 172.21.15.27:0/551497184 conn(0x7febc80c9fb0 msgr2=0x7febc80cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:58.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.540+0000 7febce3ff700 1 -- 172.21.15.27:0/551497184 shutdown_connections 2024-01-11T09:46:58.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.540+0000 7febce3ff700 1 -- 172.21.15.27:0/551497184 wait complete. 2024-01-11T09:46:58.549 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T09:46:58.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.763+0000 7fb639bff700 1 Processor -- start 2024-01-11T09:46:58.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.763+0000 7fb639bff700 1 -- start start 2024-01-11T09:46:58.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.764+0000 7fb639bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb6340c8720 0x7fb6341591a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:58.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.764+0000 7fb639bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb634069b60 0x7fb6341596e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:58.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.764+0000 7fb639bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6340cd0a0 con 0x7fb634069790 2024-01-11T09:46:58.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.764+0000 7fb639bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6340c7680 con 0x7fb634069b60 2024-01-11T09:46:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.764+0000 7fb639bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6340c7960 con 0x7fb6340c8720 2024-01-11T09:46:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.765+0000 7fb6337fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb634069790 0x7fb6340c69a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44704/0 (socket says 172.21.15.27:44704) 2024-01-11T09:46:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.765+0000 7fb6337fe700 1 -- 172.21.15.27:0/4119372687 learned_addr learned my addr 172.21.15.27:0/4119372687 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.765+0000 7fb633fff700 1 --2- 172.21.15.27:0/4119372687 >> v2:172.21.15.27:3300/0 conn(0x7fb634069b60 0x7fb6341596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.765+0000 7fb632ffd700 1 --2- 172.21.15.27:0/4119372687 >> v2:172.21.15.77:3300/0 conn(0x7fb6340c8720 0x7fb6341591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:58.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.765+0000 7fb633fff700 1 -- 172.21.15.27:0/4119372687 >> v2:172.21.15.77:3300/0 conn(0x7fb6340c8720 msgr2=0x7fb6341591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:58.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.765+0000 7fb633fff700 1 --2- 172.21.15.27:0/4119372687 >> v2:172.21.15.77:3300/0 conn(0x7fb6340c8720 0x7fb6341591a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.765+0000 7fb633fff700 1 -- 172.21.15.27:0/4119372687 >> v1:172.21.15.27:6789/0 conn(0x7fb634069790 legacy=0x7fb6340c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:58.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.765+0000 7fb633fff700 1 -- 172.21.15.27:0/4119372687 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6340c7c40 con 0x7fb634069b60 2024-01-11T09:46:58.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.765+0000 7fb633fff700 1 --2- 172.21.15.27:0/4119372687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb634069b60 0x7fb6341596e0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7fb6340c9d20 tx=0x7fb62c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f1ad58ab920996 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:58.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.766+0000 7fb6327fc700 1 -- 172.21.15.27:0/4119372687 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c00e070 con 0x7fb634069b60 2024-01-11T09:46:58.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.766+0000 7fb6327fc700 1 -- 172.21.15.27:0/4119372687 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62c00bbf0 con 0x7fb634069b60 2024-01-11T09:46:58.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.766+0000 7fb6327fc700 1 -- 172.21.15.27:0/4119372687 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c012430 con 0x7fb634069b60 2024-01-11T09:46:58.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.766+0000 7fb639bff700 1 -- 172.21.15.27:0/4119372687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb634069b60 msgr2=0x7fb6341596e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:58.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.766+0000 7fb639bff700 1 --2- 172.21.15.27:0/4119372687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb634069b60 0x7fb6341596e0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7fb6340c9d20 tx=0x7fb62c009780 comp rx=0 tx=0).stop 2024-01-11T09:46:58.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.766+0000 7fb639bff700 1 -- 172.21.15.27:0/4119372687 shutdown_connections 2024-01-11T09:46:58.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.766+0000 7fb639bff700 1 --2- 172.21.15.27:0/4119372687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb634069b60 0x7fb6341596e0 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.766+0000 7fb639bff700 1 --2- 172.21.15.27:0/4119372687 >> v2:172.21.15.77:3300/0 conn(0x7fb6340c8720 0x7fb6341591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.766+0000 7fb639bff700 1 -- 172.21.15.27:0/4119372687 >> 172.21.15.27:0/4119372687 conn(0x7fb6340680a0 msgr2=0x7fb6340cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:58.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.767+0000 7fb639bff700 1 -- 172.21.15.27:0/4119372687 shutdown_connections 2024-01-11T09:46:58.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.767+0000 7fb639bff700 1 -- 172.21.15.27:0/4119372687 wait complete. 2024-01-11T09:46:58.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.768+0000 7fb639bff700 1 Processor -- start 2024-01-11T09:46:58.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.768+0000 7fb639bff700 1 -- start start 2024-01-11T09:46:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.769+0000 7fb639bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb634069790 0x7fb634169eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.769+0000 7fb639bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb634069b60 0x7fb6340b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.769+0000 7fb639bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6340c8720 0x7fb6340b0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.769+0000 7fb639bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6340b0310 con 0x7fb6340c8720 2024-01-11T09:46:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.769+0000 7fb639bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63416a500 con 0x7fb634069b60 2024-01-11T09:46:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.769+0000 7fb639bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb63416a7e0 con 0x7fb634069790 2024-01-11T09:46:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.769+0000 7fb633fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6340c8720 0x7fb6340b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.769+0000 7fb633fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6340c8720 0x7fb6340b0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41808/0 (socket says 172.21.15.27:41808) 2024-01-11T09:46:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.769+0000 7fb633fff700 1 -- 172.21.15.27:0/242273758 learned_addr learned my addr 172.21.15.27:0/242273758 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.769+0000 7fb632ffd700 1 --2- 172.21.15.27:0/242273758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb634069b60 0x7fb6340b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.770+0000 7fb6337fe700 1 --2- 172.21.15.27:0/242273758 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb634069790 0x7fb634169eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.770+0000 7fb633fff700 1 -- 172.21.15.27:0/242273758 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb634069790 msgr2=0x7fb634169eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.770+0000 7fb633fff700 1 --2- 172.21.15.27:0/242273758 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb634069790 0x7fb634169eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.770+0000 7fb633fff700 1 -- 172.21.15.27:0/242273758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb634069b60 msgr2=0x7fb6340b17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.770+0000 7fb633fff700 1 --2- 172.21.15.27:0/242273758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb634069b60 0x7fb6340b17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.770+0000 7fb633fff700 1 -- 172.21.15.27:0/242273758 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c009a20 con 0x7fb6340c8720 2024-01-11T09:46:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.770+0000 7fb633fff700 1 --2- 172.21.15.27:0/242273758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6340c8720 0x7fb6340b0930 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7fb62c009fd0 tx=0x7fb62c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.771+0000 7fb61b7fe700 1 -- 172.21.15.27:0/242273758 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c027470 con 0x7fb6340c8720 2024-01-11T09:46:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.771+0000 7fb61b7fe700 1 -- 172.21.15.27:0/242273758 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62c0275f0 con 0x7fb6340c8720 2024-01-11T09:46:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.771+0000 7fb639bff700 1 -- 172.21.15.27:0/242273758 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb63416aac0 con 0x7fb6340c8720 2024-01-11T09:46:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.771+0000 7fb61b7fe700 1 -- 172.21.15.27:0/242273758 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c0279b0 con 0x7fb6340c8720 2024-01-11T09:46:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.771+0000 7fb639bff700 1 -- 172.21.15.27:0/242273758 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb63416d5d0 con 0x7fb6340c8720 2024-01-11T09:46:58.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.772+0000 7fb639bff700 1 -- 172.21.15.27:0/242273758 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb63416d8c0 con 0x7fb6340c8720 2024-01-11T09:46:58.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.773+0000 7fb61b7fe700 1 -- 172.21.15.27:0/242273758 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 141) v1 ==== 84947+0+0 (secure 0 0 0) 0x7fb62c024070 con 0x7fb6340c8720 2024-01-11T09:46:58.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.774+0000 7fb61b7fe700 1 --2- 172.21.15.27:0/242273758 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb620067170 0x7fb620069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:58.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.774+0000 7fb61b7fe700 1 -- 172.21.15.27:0/242273758 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 7111+0+0 (secure 0 0 0) 0x7fb62c087640 con 0x7fb6340c8720 2024-01-11T09:46:58.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.774+0000 7fb6337fe700 1 -- 172.21.15.27:0/242273758 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb620067170 msgr2=0x7fb620069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:58.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.774+0000 7fb6337fe700 1 --2- 172.21.15.27:0/242273758 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb620067170 0x7fb620069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:46:58.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.778+0000 7fb61b7fe700 1 -- 172.21.15.27:0/242273758 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb62c0540b0 con 0x7fb6340c8720 2024-01-11T09:46:58.884 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:58.893+0000 7f923a65f1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:46:58.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.975+0000 7fb6337fe700 1 -- 172.21.15.27:0/242273758 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb620067170 msgr2=0x7fb620069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:58.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.975+0000 7fb6337fe700 1 --2- 172.21.15.27:0/242273758 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb620067170 0x7fb620069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:46:58.973 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:58.982+0000 7f923a65f1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:46:58.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.997+0000 7fb639bff700 1 -- 172.21.15.27:0/242273758 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb63406f7f0 con 0x7fb6340c8720 2024-01-11T09:46:58.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:58.998+0000 7fb61b7fe700 1 -- 172.21.15.27:0/242273758 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v240) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb62c0576d0 con 0x7fb6340c8720 2024-01-11T09:46:58.989 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T09:46:58.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.000+0000 7fb639bff700 1 -- 172.21.15.27:0/242273758 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb620067170 msgr2=0x7fb620069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:46:58.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.000+0000 7fb639bff700 1 --2- 172.21.15.27:0/242273758 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb620067170 0x7fb620069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.000+0000 7fb639bff700 1 -- 172.21.15.27:0/242273758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6340c8720 msgr2=0x7fb6340b0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:58.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.000+0000 7fb639bff700 1 --2- 172.21.15.27:0/242273758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6340c8720 0x7fb6340b0930 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7fb62c009fd0 tx=0x7fb62c019000 comp rx=0 tx=0).stop 2024-01-11T09:46:58.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.000+0000 7fb639bff700 1 -- 172.21.15.27:0/242273758 shutdown_connections 2024-01-11T09:46:58.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.000+0000 7fb639bff700 1 --2- 172.21.15.27:0/242273758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6340c8720 0x7fb6340b0930 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.000+0000 7fb639bff700 1 --2- 172.21.15.27:0/242273758 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb634069790 0x7fb634169eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.000+0000 7fb639bff700 1 --2- 172.21.15.27:0/242273758 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb620067170 0x7fb620069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.000+0000 7fb639bff700 1 --2- 172.21.15.27:0/242273758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb634069b60 0x7fb6340b17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:58.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.000+0000 7fb639bff700 1 -- 172.21.15.27:0/242273758 >> 172.21.15.27:0/242273758 conn(0x7fb6340680a0 msgr2=0x7fb634156f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:58.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.000+0000 7fb639bff700 1 -- 172.21.15.27:0/242273758 shutdown_connections 2024-01-11T09:46:58.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.000+0000 7fb639bff700 1 -- 172.21.15.27:0/242273758 wait complete. 2024-01-11T09:46:59.016 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-01-11T09:46:59.096 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:59.105+0000 7f95c8a461c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:46:59.131 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:59.140+0000 7f923a65f1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:46:59.192 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:59.200+0000 7f95c8a461c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:46:59.218 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:59.227+0000 7f923a65f1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:46:59.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.229+0000 7f99da7ff700 1 Processor -- start 2024-01-11T09:46:59.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.230+0000 7f99da7ff700 1 -- start start 2024-01-11T09:46:59.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.231+0000 7f99da7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f99d41577e0 0x7f99d40ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:59.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.231+0000 7f99da7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99d4069540 0x7f99d40ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:59.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.231+0000 7f99da7ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d40699d0 con 0x7f99d4157410 2024-01-11T09:46:59.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.231+0000 7f99da7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d4069cb0 con 0x7f99d41577e0 2024-01-11T09:46:59.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.231+0000 7f99da7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d4069f90 con 0x7f99d4069540 2024-01-11T09:46:59.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.232+0000 7f99d8d9c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99d4069540 0x7f99d40ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:59.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.232+0000 7f99d3fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f99d4157410 0x7f99d40ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:58996/0 (socket says 172.21.15.27:58996) 2024-01-11T09:46:59.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.232+0000 7f99d3fff700 1 -- 172.21.15.27:0/3802151414 learned_addr learned my addr 172.21.15.27:0/3802151414 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:59.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.232+0000 7f99d37fe700 1 --2- 172.21.15.27:0/3802151414 >> v2:172.21.15.77:3300/0 conn(0x7f99d41577e0 0x7f99d40ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:59.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.232+0000 7f99d2ffd700 1 -- 172.21.15.27:0/3802151414 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606043999 0 0) 0x7f99d40699d0 con 0x7f99d4157410 2024-01-11T09:46:59.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.232+0000 7f99d2ffd700 1 -- 172.21.15.27:0/3802151414 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99bc003570 con 0x7f99d4157410 2024-01-11T09:46:59.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.232+0000 7f99d8d9c700 1 -- 172.21.15.27:0/3802151414 >> v2:172.21.15.77:3300/0 conn(0x7f99d41577e0 msgr2=0x7f99d40ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:59.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.232+0000 7f99d8d9c700 1 --2- 172.21.15.27:0/3802151414 >> v2:172.21.15.77:3300/0 conn(0x7f99d41577e0 0x7f99d40ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.232+0000 7f99d8d9c700 1 -- 172.21.15.27:0/3802151414 >> v1:172.21.15.77:6789/0 conn(0x7f99d4157410 legacy=0x7f99d40ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:59.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.232+0000 7f99d8d9c700 1 -- 172.21.15.27:0/3802151414 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d406a270 con 0x7f99d4069540 2024-01-11T09:46:59.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.232+0000 7f99d37fe700 1 --2- 172.21.15.27:0/3802151414 >> v2:172.21.15.77:3300/0 conn(0x7f99d41577e0 0x7f99d40ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:59.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.233+0000 7f99d8d9c700 1 --2- 172.21.15.27:0/3802151414 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99d4069540 0x7f99d40ae540 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f99cc009e60 tx=0x7f99cc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=349a5d563262f7c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:59.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.233+0000 7f99d2ffd700 1 -- 172.21.15.27:0/3802151414 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc00e070 con 0x7f99d4069540 2024-01-11T09:46:59.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.233+0000 7f99d2ffd700 1 -- 172.21.15.27:0/3802151414 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99cc00bab0 con 0x7f99d4069540 2024-01-11T09:46:59.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.233+0000 7f99d2ffd700 1 -- 172.21.15.27:0/3802151414 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc00be70 con 0x7f99d4069540 2024-01-11T09:46:59.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.234+0000 7f99da7ff700 1 -- 172.21.15.27:0/3802151414 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99d4069540 msgr2=0x7f99d40ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:59.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.234+0000 7f99da7ff700 1 --2- 172.21.15.27:0/3802151414 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99d4069540 0x7f99d40ae540 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f99cc009e60 tx=0x7f99cc009780 comp rx=0 tx=0).stop 2024-01-11T09:46:59.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.234+0000 7f99da7ff700 1 -- 172.21.15.27:0/3802151414 shutdown_connections 2024-01-11T09:46:59.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.234+0000 7f99da7ff700 1 --2- 172.21.15.27:0/3802151414 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99d4069540 0x7f99d40ae540 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.234+0000 7f99da7ff700 1 --2- 172.21.15.27:0/3802151414 >> v2:172.21.15.77:3300/0 conn(0x7f99d41577e0 0x7f99d40ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.234+0000 7f99da7ff700 1 -- 172.21.15.27:0/3802151414 >> 172.21.15.27:0/3802151414 conn(0x7f99d40680a0 msgr2=0x7f99d40cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:59.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.234+0000 7f99da7ff700 1 -- 172.21.15.27:0/3802151414 shutdown_connections 2024-01-11T09:46:59.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.234+0000 7f99da7ff700 1 -- 172.21.15.27:0/3802151414 wait complete. 2024-01-11T09:46:59.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.236+0000 7f99da7ff700 1 Processor -- start 2024-01-11T09:46:59.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.236+0000 7f99da7ff700 1 -- start start 2024-01-11T09:46:59.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.237+0000 7f99da7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99d4069540 0x7f99d414b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:59.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.237+0000 7f99da7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99d4157410 0x7f99d40b0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:59.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.237+0000 7f99da7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99d41577e0 0x7f99d414af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:59.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.237+0000 7f99da7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d40b13d0 con 0x7f99d4157410 2024-01-11T09:46:59.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.237+0000 7f99da7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d40b0560 con 0x7f99d4069540 2024-01-11T09:46:59.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.237+0000 7f99da7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d40b07c0 con 0x7f99d41577e0 2024-01-11T09:46:59.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.238+0000 7f99d37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99d4157410 0x7f99d40b0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:59.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.238+0000 7f99d3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99d4069540 0x7f99d414b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:59.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.238+0000 7f99d37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99d4157410 0x7f99d40b0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41822/0 (socket says 172.21.15.27:41822) 2024-01-11T09:46:59.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.238+0000 7f99d37fe700 1 -- 172.21.15.27:0/709060968 learned_addr learned my addr 172.21.15.27:0/709060968 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:59.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.238+0000 7f99d8d9c700 1 --2- 172.21.15.27:0/709060968 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99d41577e0 0x7f99d414af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:59.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.238+0000 7f99d37fe700 1 -- 172.21.15.27:0/709060968 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99d41577e0 msgr2=0x7f99d414af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:59.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.238+0000 7f99d37fe700 1 --2- 172.21.15.27:0/709060968 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99d41577e0 0x7f99d414af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.238+0000 7f99d37fe700 1 -- 172.21.15.27:0/709060968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99d4069540 msgr2=0x7f99d414b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:59.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.238+0000 7f99d37fe700 1 --2- 172.21.15.27:0/709060968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99d4069540 0x7f99d414b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.238+0000 7f99d37fe700 1 -- 172.21.15.27:0/709060968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c8009420 con 0x7f99d4157410 2024-01-11T09:46:59.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.239+0000 7f99d37fe700 1 --2- 172.21.15.27:0/709060968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99d4157410 0x7f99d40b0020 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f99c800ec50 tx=0x7f99c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:59.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.239+0000 7f99bbfff700 1 -- 172.21.15.27:0/709060968 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c800c850 con 0x7f99d4157410 2024-01-11T09:46:59.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.239+0000 7f99bbfff700 1 -- 172.21.15.27:0/709060968 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c800c9b0 con 0x7f99d4157410 2024-01-11T09:46:59.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.239+0000 7f99da7ff700 1 -- 172.21.15.27:0/709060968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99cc009a20 con 0x7f99d4157410 2024-01-11T09:46:59.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.239+0000 7f99bbfff700 1 -- 172.21.15.27:0/709060968 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c800cd50 con 0x7f99d4157410 2024-01-11T09:46:59.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.239+0000 7f99da7ff700 1 -- 172.21.15.27:0/709060968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d40b0d80 con 0x7f99d4157410 2024-01-11T09:46:59.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.240+0000 7f99da7ff700 1 -- 172.21.15.27:0/709060968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d40b1540 con 0x7f99d4157410 2024-01-11T09:46:59.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.241+0000 7f99bbfff700 1 -- 172.21.15.27:0/709060968 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 141) v1 ==== 84947+0+0 (secure 0 0 0) 0x7f99c8013680 con 0x7f99d4157410 2024-01-11T09:46:59.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.243+0000 7f99bbfff700 1 --2- 172.21.15.27:0/709060968 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f99c0067170 0x7f99c0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:59.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.243+0000 7f99bbfff700 1 -- 172.21.15.27:0/709060968 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 7111+0+0 (secure 0 0 0) 0x7f99c80850e0 con 0x7f99d4157410 2024-01-11T09:46:59.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.243+0000 7f99d3fff700 1 -- 172.21.15.27:0/709060968 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f99c0067170 msgr2=0x7f99c0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:59.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.243+0000 7f99d3fff700 1 --2- 172.21.15.27:0/709060968 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f99c0067170 0x7f99c0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:46:59.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.247+0000 7f99bbfff700 1 -- 172.21.15.27:0/709060968 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f99c8051aa0 con 0x7f99d4157410 2024-01-11T09:46:59.362 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:59.371+0000 7f95c8a461c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:46:59.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.444+0000 7f99d3fff700 1 -- 172.21.15.27:0/709060968 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f99c0067170 msgr2=0x7f99c0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:59.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.444+0000 7f99d3fff700 1 --2- 172.21.15.27:0/709060968 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f99c0067170 0x7f99c0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:46:59.439 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:59.448+0000 7f923a65f1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:46:59.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.452+0000 7f99da7ff700 1 -- 172.21.15.27:0/709060968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f99d404c3c0 con 0x7f99d4157410 2024-01-11T09:46:59.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.453+0000 7f99bbfff700 1 -- 172.21.15.27:0/709060968 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v240) v1 ==== 107+0+11 (secure 0 0 0) 0x7f99c80550c0 con 0x7f99d4157410 2024-01-11T09:46:59.444 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T09:46:59.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.455+0000 7f99da7ff700 1 -- 172.21.15.27:0/709060968 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f99c0067170 msgr2=0x7f99c0069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:46:59.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.455+0000 7f99da7ff700 1 --2- 172.21.15.27:0/709060968 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f99c0067170 0x7f99c0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.455+0000 7f99da7ff700 1 -- 172.21.15.27:0/709060968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99d4157410 msgr2=0x7f99d40b0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:59.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.455+0000 7f99da7ff700 1 --2- 172.21.15.27:0/709060968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99d4157410 0x7f99d40b0020 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f99c800ec50 tx=0x7f99c80091a0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.455+0000 7f99da7ff700 1 -- 172.21.15.27:0/709060968 shutdown_connections 2024-01-11T09:46:59.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.455+0000 7f99da7ff700 1 --2- 172.21.15.27:0/709060968 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f99d41577e0 0x7f99d414af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.455+0000 7f99da7ff700 1 --2- 172.21.15.27:0/709060968 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f99c0067170 0x7f99c0069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.455+0000 7f99da7ff700 1 --2- 172.21.15.27:0/709060968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99d4069540 0x7f99d414b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.455+0000 7f99da7ff700 1 --2- 172.21.15.27:0/709060968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99d4157410 0x7f99d40b0020 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.455+0000 7f99da7ff700 1 -- 172.21.15.27:0/709060968 >> 172.21.15.27:0/709060968 conn(0x7f99d40680a0 msgr2=0x7f99d40c6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:59.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.456+0000 7f99da7ff700 1 -- 172.21.15.27:0/709060968 shutdown_connections 2024-01-11T09:46:59.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.456+0000 7f99da7ff700 1 -- 172.21.15.27:0/709060968 wait complete. 2024-01-11T09:46:59.455 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:59.464+0000 7f95c8a461c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:46:59.466 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-01-11T09:46:59.535 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:59.544+0000 7f923a65f1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:46:59.631 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:59.640+0000 7f923a65f1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:46:59.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.683+0000 7f23163ff700 1 Processor -- start 2024-01-11T09:46:59.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.683+0000 7f23163ff700 1 -- start start 2024-01-11T09:46:59.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.685+0000 7f23163ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2310155c50 0x7f23100aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:59.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.685+0000 7f23163ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f23100c74f0 0x7f2310069910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:59.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.685+0000 7f23163ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231006e480 con 0x7f23100694c0 2024-01-11T09:46:59.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.685+0000 7f23163ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23100cb5a0 con 0x7f23100c74f0 2024-01-11T09:46:59.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.685+0000 7f23163ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f231006df00 con 0x7f2310155c50 2024-01-11T09:46:59.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.685+0000 7f230f7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f23100694c0 0x7f23100c6de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44712/0 (socket says 172.21.15.27:44712) 2024-01-11T09:46:59.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.685+0000 7f230f7fe700 1 -- 172.21.15.27:0/1720592080 learned_addr learned my addr 172.21.15.27:0/1720592080 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:46:59.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.685+0000 7f231499c700 1 --2- 172.21.15.27:0/1720592080 >> v2:172.21.15.77:3300/0 conn(0x7f23100c74f0 0x7f2310069910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:59.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.685+0000 7f230ffff700 1 --2- 172.21.15.27:0/1720592080 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2310155c50 0x7f23100aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:59.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.686+0000 7f230effd700 1 -- 172.21.15.27:0/1720592080 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233969939 0 0) 0x7f231006e480 con 0x7f23100694c0 2024-01-11T09:46:59.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.686+0000 7f230effd700 1 -- 172.21.15.27:0/1720592080 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f8003570 con 0x7f23100694c0 2024-01-11T09:46:59.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.686+0000 7f231499c700 1 -- 172.21.15.27:0/1720592080 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2310155c50 msgr2=0x7f23100aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:46:59.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.686+0000 7f231499c700 1 --2- 172.21.15.27:0/1720592080 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2310155c50 0x7f23100aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.686+0000 7f231499c700 1 -- 172.21.15.27:0/1720592080 >> v1:172.21.15.27:6789/0 conn(0x7f23100694c0 legacy=0x7f23100c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:59.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.686+0000 7f231499c700 1 -- 172.21.15.27:0/1720592080 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2310069e50 con 0x7f23100c74f0 2024-01-11T09:46:59.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.686+0000 7f231499c700 1 --2- 172.21.15.27:0/1720592080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23100c74f0 0x7f2310069910 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f2308009e60 tx=0x7f2308009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71d0641f097a8af3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:59.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.686+0000 7f230effd700 1 -- 172.21.15.27:0/1720592080 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230800e070 con 0x7f23100c74f0 2024-01-11T09:46:59.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.687+0000 7f230effd700 1 -- 172.21.15.27:0/1720592080 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f230800bb40 con 0x7f23100c74f0 2024-01-11T09:46:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.687+0000 7f230effd700 1 -- 172.21.15.27:0/1720592080 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2308012430 con 0x7f23100c74f0 2024-01-11T09:46:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.687+0000 7f23163ff700 1 -- 172.21.15.27:0/1720592080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23100c74f0 msgr2=0x7f2310069910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.687+0000 7f23163ff700 1 --2- 172.21.15.27:0/1720592080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23100c74f0 0x7f2310069910 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f2308009e60 tx=0x7f2308009780 comp rx=0 tx=0).stop 2024-01-11T09:46:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.687+0000 7f23163ff700 1 -- 172.21.15.27:0/1720592080 shutdown_connections 2024-01-11T09:46:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.687+0000 7f23163ff700 1 --2- 172.21.15.27:0/1720592080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23100c74f0 0x7f2310069910 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.687+0000 7f23163ff700 1 --2- 172.21.15.27:0/1720592080 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2310155c50 0x7f23100aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.687+0000 7f23163ff700 1 -- 172.21.15.27:0/1720592080 >> 172.21.15.27:0/1720592080 conn(0x7f23100680a0 msgr2=0x7f23100cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.687+0000 7f23163ff700 1 -- 172.21.15.27:0/1720592080 shutdown_connections 2024-01-11T09:46:59.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.687+0000 7f23163ff700 1 -- 172.21.15.27:0/1720592080 wait complete. 2024-01-11T09:46:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.689+0000 7f23163ff700 1 Processor -- start 2024-01-11T09:46:59.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.689+0000 7f23163ff700 1 -- start start 2024-01-11T09:46:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.690+0000 7f23163ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23100694c0 0x7f2310165f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.690+0000 7f23163ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23100c74f0 0x7f23100c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.690+0000 7f23163ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2310155c50 0x7f2310158b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.690+0000 7f23163ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2310166450 con 0x7f2310155c50 2024-01-11T09:46:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.690+0000 7f23163ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2310156370 con 0x7f23100694c0 2024-01-11T09:46:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.690+0000 7f23163ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23100c49a0 con 0x7f23100c74f0 2024-01-11T09:46:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.690+0000 7f231499c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2310155c50 0x7f2310158b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:59.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.690+0000 7f231499c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2310155c50 0x7f2310158b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41826/0 (socket says 172.21.15.27:41826) 2024-01-11T09:46:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.690+0000 7f230ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23100694c0 0x7f2310165f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.690+0000 7f231499c700 1 -- 172.21.15.27:0/1238145335 learned_addr learned my addr 172.21.15.27:0/1238145335 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:46:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.690+0000 7f230f7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23100c74f0 0x7f23100c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:46:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.691+0000 7f230ffff700 1 -- 172.21.15.27:0/1238145335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23100c74f0 msgr2=0x7f23100c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.691+0000 7f230ffff700 1 --2- 172.21.15.27:0/1238145335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23100c74f0 0x7f23100c5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.691+0000 7f230ffff700 1 -- 172.21.15.27:0/1238145335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2310155c50 msgr2=0x7f2310158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.691+0000 7f230ffff700 1 --2- 172.21.15.27:0/1238145335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2310155c50 0x7f2310158b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.691+0000 7f230ffff700 1 -- 172.21.15.27:0/1238145335 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2308009a20 con 0x7f23100694c0 2024-01-11T09:46:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.691+0000 7f231499c700 1 --2- 172.21.15.27:0/1238145335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2310155c50 0x7f2310158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:46:59.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.691+0000 7f230ffff700 1 --2- 172.21.15.27:0/1238145335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23100694c0 0x7f2310165f10 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f230000e9a0 tx=0x7f230000a2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:46:59.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.691+0000 7f22f7fff700 1 -- 172.21.15.27:0/1238145335 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2300010630 con 0x7f23100694c0 2024-01-11T09:46:59.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.691+0000 7f22f7fff700 1 -- 172.21.15.27:0/1238145335 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2300010790 con 0x7f23100694c0 2024-01-11T09:46:59.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.691+0000 7f22f7fff700 1 -- 172.21.15.27:0/1238145335 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2300010b10 con 0x7f23100694c0 2024-01-11T09:46:59.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.691+0000 7f23163ff700 1 -- 172.21.15.27:0/1238145335 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23100c3d90 con 0x7f23100694c0 2024-01-11T09:46:59.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.692+0000 7f23163ff700 1 -- 172.21.15.27:0/1238145335 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23100c4350 con 0x7f23100694c0 2024-01-11T09:46:59.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.693+0000 7f23163ff700 1 -- 172.21.15.27:0/1238145335 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23100c4c80 con 0x7f23100694c0 2024-01-11T09:46:59.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.694+0000 7f22f7fff700 1 -- 172.21.15.27:0/1238145335 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 141) v1 ==== 84947+0+0 (secure 0 0 0) 0x7f23000113f0 con 0x7f23100694c0 2024-01-11T09:46:59.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.694+0000 7f22f7fff700 1 --2- 172.21.15.27:0/1238145335 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f22fc067170 0x7f22fc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:46:59.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.694+0000 7f22f7fff700 1 -- 172.21.15.27:0/1238145335 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 7111+0+0 (secure 0 0 0) 0x7f2300015030 con 0x7f23100694c0 2024-01-11T09:46:59.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.695+0000 7f230f7fe700 1 -- 172.21.15.27:0/1238145335 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f22fc067170 msgr2=0x7f22fc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:59.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.695+0000 7f230f7fe700 1 --2- 172.21.15.27:0/1238145335 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f22fc067170 0x7f22fc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:46:59.689 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:59.698+0000 7f95c8a461c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:46:59.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.698+0000 7f22f7fff700 1 -- 172.21.15.27:0/1238145335 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2300051450 con 0x7f23100694c0 2024-01-11T09:46:59.729 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:59.738+0000 7f923a65f1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:46:59.794 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:59.802+0000 7f95c8a461c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:46:59.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.896+0000 7f230f7fe700 1 -- 172.21.15.27:0/1238145335 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f22fc067170 msgr2=0x7f22fc069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:46:59.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.896+0000 7f230f7fe700 1 --2- 172.21.15.27:0/1238145335 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f22fc067170 0x7f22fc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:46:59.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.904+0000 7f23163ff700 1 -- 172.21.15.27:0/1238145335 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f23100c8220 con 0x7f23100694c0 2024-01-11T09:46:59.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.904+0000 7f22f7fff700 1 -- 172.21.15.27:0/1238145335 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v240) v1 ==== 103+0+11 (secure 0 0 0) 0x7f230001f0a0 con 0x7f23100694c0 2024-01-11T09:46:59.896 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T09:46:59.896 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:46:59.904+0000 7f95c8a461c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:46:59.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.907+0000 7f23163ff700 1 -- 172.21.15.27:0/1238145335 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f22fc067170 msgr2=0x7f22fc069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:46:59.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.907+0000 7f23163ff700 1 --2- 172.21.15.27:0/1238145335 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f22fc067170 0x7f22fc069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.907+0000 7f23163ff700 1 -- 172.21.15.27:0/1238145335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23100694c0 msgr2=0x7f2310165f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:46:59.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.907+0000 7f23163ff700 1 --2- 172.21.15.27:0/1238145335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23100694c0 0x7f2310165f10 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f230000e9a0 tx=0x7f230000a2e0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.907+0000 7f23163ff700 1 -- 172.21.15.27:0/1238145335 shutdown_connections 2024-01-11T09:46:59.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.907+0000 7f23163ff700 1 --2- 172.21.15.27:0/1238145335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2310155c50 0x7f2310158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.907+0000 7f23163ff700 1 --2- 172.21.15.27:0/1238145335 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f22fc067170 0x7f22fc069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.907+0000 7f23163ff700 1 --2- 172.21.15.27:0/1238145335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23100694c0 0x7f2310165f10 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.907+0000 7f23163ff700 1 --2- 172.21.15.27:0/1238145335 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f23100c74f0 0x7f23100c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:46:59.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.907+0000 7f23163ff700 1 -- 172.21.15.27:0/1238145335 >> 172.21.15.27:0/1238145335 conn(0x7f23100680a0 msgr2=0x7f23101538e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:46:59.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.907+0000 7f23163ff700 1 -- 172.21.15.27:0/1238145335 shutdown_connections 2024-01-11T09:46:59.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:46:59.907+0000 7f23163ff700 1 -- 172.21.15.27:0/1238145335 wait complete. 2024-01-11T09:46:59.917 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:46:59.919 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:46:59.929+0000 7f923a65f1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:47:00.001 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:00.010+0000 7f95c8a461c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:47:00.017 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:00.026+0000 7f923a65f1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:47:00.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.128+0000 7fe1243ff700 1 Processor -- start 2024-01-11T09:47:00.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.128+0000 7fe1243ff700 1 -- start start 2024-01-11T09:47:00.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.130+0000 7fe1243ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe11c06c060 0x7fe11c0cd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:00.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.130+0000 7fe1243ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11c15bc30 con 0x7fe11c0c8c70 2024-01-11T09:47:00.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.130+0000 7fe1243ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11c0ccfc0 con 0x7fe11c0c9040 2024-01-11T09:47:00.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.130+0000 7fe1243ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe11c158c40 con 0x7fe11c06c060 2024-01-11T09:47:00.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.130+0000 7fe12219b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe11c0c8c70 0x7fe11c06b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44724/0 (socket says 172.21.15.27:44724) 2024-01-11T09:47:00.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.130+0000 7fe12219b700 1 -- 172.21.15.27:0/4097547764 learned_addr learned my addr 172.21.15.27:0/4097547764 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:00.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.130+0000 7fe12199a700 1 --2- 172.21.15.27:0/4097547764 >> v2:172.21.15.77:3300/0 conn(0x7fe11c06c060 0x7fe11c0cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:00.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.131+0000 7fe121199700 1 -- 172.21.15.27:0/4097547764 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4231354469 0 0) 0x7fe11c15bc30 con 0x7fe11c0c8c70 2024-01-11T09:47:00.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.131+0000 7fe121199700 1 -- 172.21.15.27:0/4097547764 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe108003570 con 0x7fe11c0c8c70 2024-01-11T09:47:00.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.131+0000 7fe121199700 1 -- 172.21.15.27:0/4097547764 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843158296 0 0) 0x7fe11c0ccfc0 con 0x7fe11c0c9040 2024-01-11T09:47:00.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.131+0000 7fe121199700 1 -- 172.21.15.27:0/4097547764 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe11c15bc30 con 0x7fe11c0c9040 2024-01-11T09:47:00.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.131+0000 7fe12199a700 1 -- 172.21.15.27:0/4097547764 >> v1:172.21.15.77:6789/0 conn(0x7fe11c0c9040 legacy=0x7fe11c140550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:00.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.131+0000 7fe121199700 1 -- 172.21.15.27:0/4097547764 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 67616488 0 0) 0x7fe108003570 con 0x7fe11c0c8c70 2024-01-11T09:47:00.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.131+0000 7fe12199a700 1 -- 172.21.15.27:0/4097547764 >> v1:172.21.15.27:6789/0 conn(0x7fe11c0c8c70 legacy=0x7fe11c06b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:00.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.131+0000 7fe12199a700 1 -- 172.21.15.27:0/4097547764 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11c06c5a0 con 0x7fe11c06c060 2024-01-11T09:47:00.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.131+0000 7fe12199a700 1 --2- 172.21.15.27:0/4097547764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c06c060 0x7fe11c0cd410 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0098e0 tx=0x7fe10c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=147944dff950b83 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:00.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.132+0000 7fe121199700 1 -- 172.21.15.27:0/4097547764 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe118001470 con 0x7fe11c06c060 2024-01-11T09:47:00.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.132+0000 7fe121199700 1 -- 172.21.15.27:0/4097547764 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10c011420 con 0x7fe11c06c060 2024-01-11T09:47:00.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.132+0000 7fe121199700 1 -- 172.21.15.27:0/4097547764 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c0117c0 con 0x7fe11c06c060 2024-01-11T09:47:00.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.132+0000 7fe1243ff700 1 -- 172.21.15.27:0/4097547764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c06c060 msgr2=0x7fe11c0cd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:00.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.132+0000 7fe1243ff700 1 --2- 172.21.15.27:0/4097547764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c06c060 0x7fe11c0cd410 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0098e0 tx=0x7fe10c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.132+0000 7fe1243ff700 1 -- 172.21.15.27:0/4097547764 shutdown_connections 2024-01-11T09:47:00.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.132+0000 7fe1243ff700 1 --2- 172.21.15.27:0/4097547764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c06c060 0x7fe11c0cd410 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.132+0000 7fe1243ff700 1 -- 172.21.15.27:0/4097547764 >> 172.21.15.27:0/4097547764 conn(0x7fe11c0ce3e0 msgr2=0x7fe11c0ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:00.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.132+0000 7fe1243ff700 1 -- 172.21.15.27:0/4097547764 shutdown_connections 2024-01-11T09:47:00.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.132+0000 7fe1243ff700 1 -- 172.21.15.27:0/4097547764 wait complete. 2024-01-11T09:47:00.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.134+0000 7fe1243ff700 1 Processor -- start 2024-01-11T09:47:00.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.134+0000 7fe1243ff700 1 -- start start 2024-01-11T09:47:00.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.135+0000 7fe1243ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe11c06c060 0x7fe11c0c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:00.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.135+0000 7fe1243ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c0c8c70 0x7fe11c14fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:00.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.135+0000 7fe1243ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe11c0c9040 0x7fe11c150250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:00.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.135+0000 7fe1243ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c15f2f0 con 0x7fe11c0c9040 2024-01-11T09:47:00.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.135+0000 7fe1243ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c0c5d30 con 0x7fe11c0c8c70 2024-01-11T09:47:00.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.135+0000 7fe1243ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c15e770 con 0x7fe11c06c060 2024-01-11T09:47:00.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.136+0000 7fe12199a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c0c8c70 0x7fe11c14fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:00.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.136+0000 7fe12299c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe11c0c9040 0x7fe11c150250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:00.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.136+0000 7fe12219b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe11c06c060 0x7fe11c0c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:00.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.136+0000 7fe12199a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c0c8c70 0x7fe11c14fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:42832/0 (socket says 172.21.15.27:42832) 2024-01-11T09:47:00.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.136+0000 7fe12199a700 1 -- 172.21.15.27:0/868306429 learned_addr learned my addr 172.21.15.27:0/868306429 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:00.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.136+0000 7fe12219b700 1 -- 172.21.15.27:0/868306429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c0c8c70 msgr2=0x7fe11c14fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:00.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.136+0000 7fe12219b700 1 --2- 172.21.15.27:0/868306429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c0c8c70 0x7fe11c14fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.136+0000 7fe12219b700 1 -- 172.21.15.27:0/868306429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe11c0c9040 msgr2=0x7fe11c150250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:00.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.136+0000 7fe12219b700 1 --2- 172.21.15.27:0/868306429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe11c0c9040 0x7fe11c150250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.136+0000 7fe12219b700 1 -- 172.21.15.27:0/868306429 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c009420 con 0x7fe11c06c060 2024-01-11T09:47:00.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.136+0000 7fe12299c700 1 --2- 172.21.15.27:0/868306429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe11c0c9040 0x7fe11c150250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:00.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.136+0000 7fe12199a700 1 --2- 172.21.15.27:0/868306429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c0c8c70 0x7fe11c14fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:00.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.136+0000 7fe12219b700 1 --2- 172.21.15.27:0/868306429 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe11c06c060 0x7fe11c0c7d00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe1180011e0 tx=0x7fe118001ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:00.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.137+0000 7fe1127fc700 1 -- 172.21.15.27:0/868306429 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe118007150 con 0x7fe11c06c060 2024-01-11T09:47:00.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.137+0000 7fe1127fc700 1 -- 172.21.15.27:0/868306429 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1180072b0 con 0x7fe11c06c060 2024-01-11T09:47:00.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.137+0000 7fe1127fc700 1 -- 172.21.15.27:0/868306429 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe118007670 con 0x7fe11c06c060 2024-01-11T09:47:00.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.137+0000 7fe1243ff700 1 -- 172.21.15.27:0/868306429 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11c15ea50 con 0x7fe11c06c060 2024-01-11T09:47:00.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.137+0000 7fe1243ff700 1 -- 172.21.15.27:0/868306429 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11c15f010 con 0x7fe11c06c060 2024-01-11T09:47:00.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.138+0000 7fe1243ff700 1 -- 172.21.15.27:0/868306429 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe11c0c5ec0 con 0x7fe11c06c060 2024-01-11T09:47:00.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.139+0000 7fe1127fc700 1 -- 172.21.15.27:0/868306429 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 141) v1 ==== 84947+0+0 (secure 0 0 0) 0x7fe118004a40 con 0x7fe11c06c060 2024-01-11T09:47:00.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.140+0000 7fe1127fc700 1 --2- 172.21.15.27:0/868306429 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe100067170 0x7fe100069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:00.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.140+0000 7fe1127fc700 1 -- 172.21.15.27:0/868306429 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 7111+0+0 (secure 0 0 0) 0x7fe1180840e0 con 0x7fe11c06c060 2024-01-11T09:47:00.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.140+0000 7fe12199a700 1 -- 172.21.15.27:0/868306429 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe100067170 msgr2=0x7fe100069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:00.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.140+0000 7fe12199a700 1 --2- 172.21.15.27:0/868306429 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe100067170 0x7fe100069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:00.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.144+0000 7fe1127fc700 1 -- 172.21.15.27:0/868306429 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe11c0c5ec0 con 0x7fe11c06c060 2024-01-11T09:47:00.208 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:00.217+0000 7f95c8a461c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:47:00.314 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:00.323+0000 7f95c8a461c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:47:00.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.341+0000 7fe12199a700 1 -- 172.21.15.27:0/868306429 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe100067170 msgr2=0x7fe100069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:00.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.341+0000 7fe12199a700 1 --2- 172.21.15.27:0/868306429 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe100067170 0x7fe100069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:00.366 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:00.375+0000 7f923a65f1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:47:00.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.394+0000 7fe1243ff700 1 -- 172.21.15.27:0/868306429 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe11c0c5ec0 con 0x7fe11c06c060 2024-01-11T09:47:00.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.395+0000 7fe1127fc700 1 -- 172.21.15.27:0/868306429 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5688 (secure 0 0 0) 0x7fe11c0c5ec0 con 0x7fe11c06c060 2024-01-11T09:47:00.386 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:00.386 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":132,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1568813392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1568813392},{"type":"v1","addr":"172.21.15.77:6834","nonce":1568813392}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122},{"gid":10969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1352491019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1352491019},{"type":"v1","addr":"172.21.15.77:6835","nonce":1352491019}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:46:25.726042+0000","modified":"2024-01-11T09:46:29.257798+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11340":{"gid":11340,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/321681407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":321681407},{"type":"v1","addr":"172.21.15.27:6835","nonce":321681407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:46:39.483941+0000","modified":"2024-01-11T09:46:42.996786+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11331},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11331":{"gid":11331,"name":"a","rank":0,"incarnation":129,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6834/3477661846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3477661846},{"type":"v1","addr":"172.21.15.27:6834","nonce":3477661846}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":16}]} 2024-01-11T09:47:00.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.397+0000 7fe1243ff700 1 -- 172.21.15.27:0/868306429 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe100067170 msgr2=0x7fe100069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:00.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.397+0000 7fe1243ff700 1 --2- 172.21.15.27:0/868306429 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe100067170 0x7fe100069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.397+0000 7fe1243ff700 1 -- 172.21.15.27:0/868306429 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe11c06c060 msgr2=0x7fe11c0c7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:00.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.397+0000 7fe1243ff700 1 --2- 172.21.15.27:0/868306429 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe11c06c060 0x7fe11c0c7d00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe1180011e0 tx=0x7fe118001ad0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.397+0000 7fe1243ff700 1 -- 172.21.15.27:0/868306429 shutdown_connections 2024-01-11T09:47:00.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.397+0000 7fe1243ff700 1 --2- 172.21.15.27:0/868306429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe11c0c9040 0x7fe11c150250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.397+0000 7fe1243ff700 1 --2- 172.21.15.27:0/868306429 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe11c06c060 0x7fe11c0c7d00 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.397+0000 7fe1243ff700 1 --2- 172.21.15.27:0/868306429 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe100067170 0x7fe100069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.397+0000 7fe1243ff700 1 --2- 172.21.15.27:0/868306429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c0c8c70 0x7fe11c14fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.397+0000 7fe1243ff700 1 -- 172.21.15.27:0/868306429 >> 172.21.15.27:0/868306429 conn(0x7fe11c0ce3e0 msgr2=0x7fe11c069fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:00.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.397+0000 7fe1243ff700 1 -- 172.21.15.27:0/868306429 shutdown_connections 2024-01-11T09:47:00.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.397+0000 7fe1243ff700 1 -- 172.21.15.27:0/868306429 wait complete. 2024-01-11T09:47:00.390 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 132 2024-01-11T09:47:00.408 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:47:00.456 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:00.465+0000 7f923a65f1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:47:00.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.623+0000 7fc240dff700 1 Processor -- start 2024-01-11T09:47:00.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.624+0000 7fc240dff700 1 -- start start 2024-01-11T09:47:00.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.626+0000 7fc240dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc23c1599e0 0x7fc23c0b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:00.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.626+0000 7fc240dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc23c159db0 0x7fc23c15a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:00.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.626+0000 7fc240dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23c157e30 con 0x7fc23c14fc20 2024-01-11T09:47:00.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.626+0000 7fc240dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc23c157500 con 0x7fc23c159db0 2024-01-11T09:47:00.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.626+0000 7fc240dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c1577e0 con 0x7fc23c1599e0 2024-01-11T09:47:00.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.626+0000 7fc239d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc23c159db0 0x7fc23c15a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:00.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.626+0000 7fc23ad9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc23c14fc20 0x7fc23c150080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:59022/0 (socket says 172.21.15.27:59022) 2024-01-11T09:47:00.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.626+0000 7fc23ad9d700 1 -- 172.21.15.27:0/4111238579 learned_addr learned my addr 172.21.15.27:0/4111238579 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:00.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.626+0000 7fc239d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc23c159db0 0x7fc23c15a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41840/0 (socket says 172.21.15.27:41840) 2024-01-11T09:47:00.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.626+0000 7fc23a59c700 1 --2- 172.21.15.27:0/4111238579 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc23c1599e0 0x7fc23c0b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:00.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.627+0000 7fc239d9b700 1 -- 172.21.15.27:0/4111238579 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc23c1599e0 msgr2=0x7fc23c0b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:00.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.627+0000 7fc23959a700 1 -- 172.21.15.27:0/4111238579 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710965157 0 0) 0x7fc23c157e30 con 0x7fc23c14fc20 2024-01-11T09:47:00.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.627+0000 7fc239d9b700 1 --2- 172.21.15.27:0/4111238579 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc23c1599e0 0x7fc23c0b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.627+0000 7fc239d9b700 1 -- 172.21.15.27:0/4111238579 >> v1:172.21.15.77:6789/0 conn(0x7fc23c14fc20 legacy=0x7fc23c150080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:00.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.627+0000 7fc239d9b700 1 -- 172.21.15.27:0/4111238579 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23c157ac0 con 0x7fc23c159db0 2024-01-11T09:47:00.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.627+0000 7fc239d9b700 1 --2- 172.21.15.27:0/4111238579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c159db0 0x7fc23c15a220 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7fc2300098e0 tx=0x7fc2300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8476bd170d1a934 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:00.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.627+0000 7fc23959a700 1 -- 172.21.15.27:0/4111238579 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23000b640 con 0x7fc23c159db0 2024-01-11T09:47:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.628+0000 7fc23959a700 1 -- 172.21.15.27:0/4111238579 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc23000b7a0 con 0x7fc23c159db0 2024-01-11T09:47:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.628+0000 7fc23959a700 1 -- 172.21.15.27:0/4111238579 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23000bb40 con 0x7fc23c159db0 2024-01-11T09:47:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.628+0000 7fc240dff700 1 -- 172.21.15.27:0/4111238579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c159db0 msgr2=0x7fc23c15a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.628+0000 7fc240dff700 1 --2- 172.21.15.27:0/4111238579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c159db0 0x7fc23c15a220 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7fc2300098e0 tx=0x7fc2300091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.628+0000 7fc240dff700 1 -- 172.21.15.27:0/4111238579 shutdown_connections 2024-01-11T09:47:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.628+0000 7fc240dff700 1 --2- 172.21.15.27:0/4111238579 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc23c1599e0 0x7fc23c0b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.628+0000 7fc240dff700 1 --2- 172.21.15.27:0/4111238579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c159db0 0x7fc23c15a220 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.628+0000 7fc240dff700 1 -- 172.21.15.27:0/4111238579 >> 172.21.15.27:0/4111238579 conn(0x7fc23c0680e0 msgr2=0x7fc23c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.628+0000 7fc240dff700 1 -- 172.21.15.27:0/4111238579 shutdown_connections 2024-01-11T09:47:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.628+0000 7fc240dff700 1 -- 172.21.15.27:0/4111238579 wait complete. 2024-01-11T09:47:00.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.631+0000 7fc240dff700 1 Processor -- start 2024-01-11T09:47:00.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.631+0000 7fc240dff700 1 -- start start 2024-01-11T09:47:00.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.633+0000 7fc240dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc23c14fc20 0x7fc23c0abf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:00.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.633+0000 7fc240dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c1599e0 0x7fc23c0addb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.633+0000 7fc240dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc23c159db0 0x7fc23c14b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.633+0000 7fc240dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c0cb370 con 0x7fc23c1599e0 2024-01-11T09:47:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.633+0000 7fc240dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c152ec0 con 0x7fc23c159db0 2024-01-11T09:47:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.633+0000 7fc240dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c0ad200 con 0x7fc23c14fc20 2024-01-11T09:47:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.633+0000 7fc239d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c1599e0 0x7fc23c0addb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.633+0000 7fc23a59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc23c14fc20 0x7fc23c0abf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.633+0000 7fc23a59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc23c14fc20 0x7fc23c0abf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:38752/0 (socket says 172.21.15.27:38752) 2024-01-11T09:47:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.633+0000 7fc23a59c700 1 -- 172.21.15.27:0/2278701322 learned_addr learned my addr 172.21.15.27:0/2278701322 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.633+0000 7fc239d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c1599e0 0x7fc23c0addb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41848/0 (socket says 172.21.15.27:41848) 2024-01-11T09:47:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.633+0000 7fc23ad9d700 1 --2- 172.21.15.27:0/2278701322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc23c159db0 0x7fc23c14b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:00.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.633+0000 7fc239d9b700 1 -- 172.21.15.27:0/2278701322 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc23c14fc20 msgr2=0x7fc23c0abf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:00.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.633+0000 7fc239d9b700 1 --2- 172.21.15.27:0/2278701322 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc23c14fc20 0x7fc23c0abf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.633+0000 7fc239d9b700 1 -- 172.21.15.27:0/2278701322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc23c159db0 msgr2=0x7fc23c14b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:00.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.633+0000 7fc239d9b700 1 --2- 172.21.15.27:0/2278701322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc23c159db0 0x7fc23c14b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.633+0000 7fc239d9b700 1 -- 172.21.15.27:0/2278701322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc230009420 con 0x7fc23c1599e0 2024-01-11T09:47:00.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.634+0000 7fc239d9b700 1 --2- 172.21.15.27:0/2278701322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c1599e0 0x7fc23c0addb0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fc230016000 tx=0x7fc230017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:00.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.634+0000 7fc22a7fc700 1 -- 172.21.15.27:0/2278701322 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23000bb20 con 0x7fc23c1599e0 2024-01-11T09:47:00.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.634+0000 7fc22a7fc700 1 -- 172.21.15.27:0/2278701322 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc230000c90 con 0x7fc23c1599e0 2024-01-11T09:47:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.634+0000 7fc240dff700 1 -- 172.21.15.27:0/2278701322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23c0ad690 con 0x7fc23c1599e0 2024-01-11T09:47:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.634+0000 7fc22a7fc700 1 -- 172.21.15.27:0/2278701322 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc230011650 con 0x7fc23c1599e0 2024-01-11T09:47:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.634+0000 7fc240dff700 1 -- 172.21.15.27:0/2278701322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23c0ac8d0 con 0x7fc23c1599e0 2024-01-11T09:47:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.636+0000 7fc240dff700 1 -- 172.21.15.27:0/2278701322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23c0718d0 con 0x7fc23c1599e0 2024-01-11T09:47:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.636+0000 7fc22a7fc700 1 -- 172.21.15.27:0/2278701322 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 141) v1 ==== 84947+0+0 (secure 0 0 0) 0x7fc230011a40 con 0x7fc23c1599e0 2024-01-11T09:47:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.639+0000 7fc22a7fc700 1 --2- 172.21.15.27:0/2278701322 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc21c067170 0x7fc21c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.639+0000 7fc22a7fc700 1 -- 172.21.15.27:0/2278701322 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 7111+0+0 (secure 0 0 0) 0x7fc2300867b0 con 0x7fc23c1599e0 2024-01-11T09:47:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.639+0000 7fc23a59c700 1 -- 172.21.15.27:0/2278701322 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc21c067170 msgr2=0x7fc21c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.639+0000 7fc23a59c700 1 --2- 172.21.15.27:0/2278701322 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc21c067170 0x7fc21c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.642+0000 7fc22a7fc700 1 -- 172.21.15.27:0/2278701322 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc230053170 con 0x7fc23c1599e0 2024-01-11T09:47:00.703 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:00.711+0000 7f95c8a461c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:47:00.798 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:00.807+0000 7f95c8a461c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:47:00.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.840+0000 7fc23a59c700 1 -- 172.21.15.27:0/2278701322 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc21c067170 msgr2=0x7fc21c069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:00.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.840+0000 7fc23a59c700 1 --2- 172.21.15.27:0/2278701322 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc21c067170 0x7fc21c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.862+0000 7fc240dff700 1 -- 172.21.15.27:0/2278701322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc23c0ad340 con 0x7fc23c1599e0 2024-01-11T09:47:00.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.863+0000 7fc22a7fc700 1 -- 172.21.15.27:0/2278701322 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v240) v1 ==== 74+0+19338 (secure 0 0 0) 0x7fc230056790 con 0x7fc23c1599e0 2024-01-11T09:47:00.855 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:00.855 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":240,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:46:54.200005+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:46:24.049904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-01-11T09:46:24.737495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-01-11T09:46:37.665603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"238","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-01-11T09:46:38.472134+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"239","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/210673663":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/3835722213":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/696729061":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/206349247":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/3398849669":"2024-01-12T09:46:54.199949+0000","172.21.15.77:6837/59776":"2024-01-12T09:46:54.199949+0000","172.21.15.77:6836/59776":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/1934793537":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/1366958778":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/198451994":"2024-01-12T09:46:54.199949+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:47:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.866+0000 7fc240dff700 1 -- 172.21.15.27:0/2278701322 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc21c067170 msgr2=0x7fc21c069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.866+0000 7fc240dff700 1 --2- 172.21.15.27:0/2278701322 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc21c067170 0x7fc21c069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.866+0000 7fc240dff700 1 -- 172.21.15.27:0/2278701322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c1599e0 msgr2=0x7fc23c0addb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.866+0000 7fc240dff700 1 --2- 172.21.15.27:0/2278701322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c1599e0 0x7fc23c0addb0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fc230016000 tx=0x7fc230017000 comp rx=0 tx=0).stop 2024-01-11T09:47:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.866+0000 7fc240dff700 1 -- 172.21.15.27:0/2278701322 shutdown_connections 2024-01-11T09:47:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.866+0000 7fc240dff700 1 --2- 172.21.15.27:0/2278701322 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc21c067170 0x7fc21c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.866+0000 7fc240dff700 1 --2- 172.21.15.27:0/2278701322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc23c159db0 0x7fc23c14b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.866+0000 7fc240dff700 1 --2- 172.21.15.27:0/2278701322 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc23c14fc20 0x7fc23c0abf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.866+0000 7fc240dff700 1 --2- 172.21.15.27:0/2278701322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c1599e0 0x7fc23c0addb0 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.866+0000 7fc240dff700 1 -- 172.21.15.27:0/2278701322 >> 172.21.15.27:0/2278701322 conn(0x7fc23c0680e0 msgr2=0x7fc23c150080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:00.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.866+0000 7fc240dff700 1 -- 172.21.15.27:0/2278701322 shutdown_connections 2024-01-11T09:47:00.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:00.866+0000 7fc240dff700 1 -- 172.21.15.27:0/2278701322 wait complete. 2024-01-11T09:47:00.876 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-11T09:47:00.876 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:47:00.998 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:01.007+0000 7f923a65f1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:47:01.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.088+0000 7f7302fff700 1 Processor -- start 2024-01-11T09:47:01.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.088+0000 7f7302fff700 1 -- start start 2024-01-11T09:47:01.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.090+0000 7f7302fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72fc0c8620 0x7f72fc0af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:01.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.090+0000 7f7302fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f72fc0b09e0 0x7f72fc0accc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:01.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.090+0000 7f7302fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc157390 con 0x7f72fc069d40 2024-01-11T09:47:01.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.090+0000 7f7302fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72fc0afeb0 con 0x7f72fc0b09e0 2024-01-11T09:47:01.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.090+0000 7f7302fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72fc06a200 con 0x7f72fc0c8620 2024-01-11T09:47:01.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.090+0000 7f730159c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f72fc0b09e0 0x7f72fc0accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:01.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.090+0000 7f72fbfff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f72fc069d40 0x7f72fc0b02d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44728/0 (socket says 172.21.15.27:44728) 2024-01-11T09:47:01.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.090+0000 7f72fbfff700 1 -- 172.21.15.27:0/2648939559 learned_addr learned my addr 172.21.15.27:0/2648939559 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:01.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.090+0000 7f7300d9b700 1 --2- 172.21.15.27:0/2648939559 >> v2:172.21.15.77:3300/0 conn(0x7f72fc0c8620 0x7f72fc0af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:01.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.090+0000 7f730159c700 1 -- 172.21.15.27:0/2648939559 >> v2:172.21.15.77:3300/0 conn(0x7f72fc0c8620 msgr2=0x7f72fc0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:01.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.091+0000 7f730159c700 1 --2- 172.21.15.27:0/2648939559 >> v2:172.21.15.77:3300/0 conn(0x7f72fc0c8620 0x7f72fc0af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.091+0000 7f730159c700 1 -- 172.21.15.27:0/2648939559 >> v1:172.21.15.27:6789/0 conn(0x7f72fc069d40 legacy=0x7f72fc0b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:01.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.091+0000 7f730159c700 1 -- 172.21.15.27:0/2648939559 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc156d40 con 0x7f72fc0b09e0 2024-01-11T09:47:01.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.091+0000 7f72fb7fe700 1 -- 172.21.15.27:0/2648939559 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433320894 0 0) 0x7f72fc157390 con 0x7f72fc069d40 2024-01-11T09:47:01.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.091+0000 7f730159c700 1 --2- 172.21.15.27:0/2648939559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72fc0b09e0 0x7f72fc0accc0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009e60 tx=0x7f72f4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fbf4083eb4e9126 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:01.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.091+0000 7f72fb7fe700 1 -- 172.21.15.27:0/2648939559 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f400e070 con 0x7f72fc0b09e0 2024-01-11T09:47:01.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.091+0000 7f72fb7fe700 1 -- 172.21.15.27:0/2648939559 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f400bb20 con 0x7f72fc0b09e0 2024-01-11T09:47:01.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.091+0000 7f72fb7fe700 1 -- 172.21.15.27:0/2648939559 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f4012430 con 0x7f72fc0b09e0 2024-01-11T09:47:01.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.092+0000 7f7302fff700 1 -- 172.21.15.27:0/2648939559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72fc0b09e0 msgr2=0x7f72fc0accc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:01.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.092+0000 7f7302fff700 1 --2- 172.21.15.27:0/2648939559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72fc0b09e0 0x7f72fc0accc0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009e60 tx=0x7f72f4009780 comp rx=0 tx=0).stop 2024-01-11T09:47:01.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.092+0000 7f7302fff700 1 -- 172.21.15.27:0/2648939559 shutdown_connections 2024-01-11T09:47:01.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.092+0000 7f7302fff700 1 --2- 172.21.15.27:0/2648939559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72fc0b09e0 0x7f72fc0accc0 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.092+0000 7f7302fff700 1 --2- 172.21.15.27:0/2648939559 >> v2:172.21.15.77:3300/0 conn(0x7f72fc0c8620 0x7f72fc0af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.092+0000 7f7302fff700 1 -- 172.21.15.27:0/2648939559 >> 172.21.15.27:0/2648939559 conn(0x7f72fc0680e0 msgr2=0x7f72fc069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:01.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.092+0000 7f7302fff700 1 -- 172.21.15.27:0/2648939559 shutdown_connections 2024-01-11T09:47:01.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.092+0000 7f7302fff700 1 -- 172.21.15.27:0/2648939559 wait complete. 2024-01-11T09:47:01.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.093+0000 7f7302fff700 1 Processor -- start 2024-01-11T09:47:01.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.094+0000 7f7302fff700 1 -- start start 2024-01-11T09:47:01.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.094+0000 7f7302fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f72fc069d40 0x7f72fc155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:01.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.095+0000 7f7302fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72fc0b09e0 0x7f72fc143ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:01.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.095+0000 7f7302fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72fc0c8620 0x7f72fc0c9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:01.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.095+0000 7f7302fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc155620 con 0x7f72fc0b09e0 2024-01-11T09:47:01.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.095+0000 7f7302fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc154a10 con 0x7f72fc0c8620 2024-01-11T09:47:01.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.095+0000 7f7302fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc154cf0 con 0x7f72fc069d40 2024-01-11T09:47:01.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.095+0000 7f72fbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72fc0b09e0 0x7f72fc143ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:01.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.095+0000 7f72fbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72fc0b09e0 0x7f72fc143ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41856/0 (socket says 172.21.15.27:41856) 2024-01-11T09:47:01.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.095+0000 7f72fbfff700 1 -- 172.21.15.27:0/3048851892 learned_addr learned my addr 172.21.15.27:0/3048851892 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:01.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.095+0000 7f7300d9b700 1 --2- 172.21.15.27:0/3048851892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f72fc069d40 0x7f72fc155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:01.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.095+0000 7f730159c700 1 --2- 172.21.15.27:0/3048851892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72fc0c8620 0x7f72fc0c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:01.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.095+0000 7f72fbfff700 1 -- 172.21.15.27:0/3048851892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f72fc069d40 msgr2=0x7f72fc155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:01.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.095+0000 7f72fbfff700 1 --2- 172.21.15.27:0/3048851892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f72fc069d40 0x7f72fc155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.095+0000 7f72fbfff700 1 -- 172.21.15.27:0/3048851892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72fc0c8620 msgr2=0x7f72fc0c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:01.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.095+0000 7f72fbfff700 1 --2- 172.21.15.27:0/3048851892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72fc0c8620 0x7f72fc0c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.095+0000 7f72fbfff700 1 -- 172.21.15.27:0/3048851892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f4009a20 con 0x7f72fc0b09e0 2024-01-11T09:47:01.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.096+0000 7f72fbfff700 1 --2- 172.21.15.27:0/3048851892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72fc0b09e0 0x7f72fc143ad0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f72e800c7e0 tx=0x7f72e8007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:01.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.096+0000 7f72f8ff9700 1 -- 172.21.15.27:0/3048851892 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8005b30 con 0x7f72fc0b09e0 2024-01-11T09:47:01.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.096+0000 7f72f8ff9700 1 -- 172.21.15.27:0/3048851892 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e8005c90 con 0x7f72fc0b09e0 2024-01-11T09:47:01.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.096+0000 7f72f8ff9700 1 -- 172.21.15.27:0/3048851892 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8007300 con 0x7f72fc0b09e0 2024-01-11T09:47:01.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.096+0000 7f7302fff700 1 -- 172.21.15.27:0/3048851892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72fc154fd0 con 0x7f72fc0b09e0 2024-01-11T09:47:01.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.096+0000 7f7302fff700 1 -- 172.21.15.27:0/3048851892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72fc155ff0 con 0x7f72fc0b09e0 2024-01-11T09:47:01.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.097+0000 7f7302fff700 1 -- 172.21.15.27:0/3048851892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72fc0718d0 con 0x7f72fc0b09e0 2024-01-11T09:47:01.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.098+0000 7f72f8ff9700 1 -- 172.21.15.27:0/3048851892 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 141) v1 ==== 84947+0+0 (secure 0 0 0) 0x7f72e801b3b0 con 0x7f72fc0b09e0 2024-01-11T09:47:01.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.099+0000 7f72f8ff9700 1 --2- 172.21.15.27:0/3048851892 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f72e0067170 0x7f72e0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:01.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.099+0000 7f72f8ff9700 1 -- 172.21.15.27:0/3048851892 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 7111+0+0 (secure 0 0 0) 0x7f72e80844a0 con 0x7f72fc0b09e0 2024-01-11T09:47:01.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.099+0000 7f7300d9b700 1 -- 172.21.15.27:0/3048851892 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f72e0067170 msgr2=0x7f72e0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:01.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.099+0000 7f7300d9b700 1 --2- 172.21.15.27:0/3048851892 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f72e0067170 0x7f72e0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:01.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.103+0000 7f72f8ff9700 1 -- 172.21.15.27:0/3048851892 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f72e8050d90 con 0x7f72fc0b09e0 2024-01-11T09:47:01.187 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:01.196+0000 7f923a65f1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:47:01.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.300+0000 7f7300d9b700 1 -- 172.21.15.27:0/3048851892 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f72e0067170 msgr2=0x7f72e0069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:01.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.300+0000 7f7300d9b700 1 --2- 172.21.15.27:0/3048851892 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f72e0067170 0x7f72e0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:01.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.353+0000 7f7302fff700 1 -- 172.21.15.27:0/3048851892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72fc155470 con 0x7f72fc0b09e0 2024-01-11T09:47:01.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.354+0000 7f72f8ff9700 1 -- 172.21.15.27:0/3048851892 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5688 (secure 0 0 0) 0x7f72e8001d60 con 0x7f72fc0b09e0 2024-01-11T09:47:01.346 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:01.346 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":132,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1568813392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1568813392},{"type":"v1","addr":"172.21.15.77:6834","nonce":1568813392}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122},{"gid":10969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1352491019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1352491019},{"type":"v1","addr":"172.21.15.77:6835","nonce":1352491019}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:46:25.726042+0000","modified":"2024-01-11T09:46:29.257798+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11340":{"gid":11340,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/321681407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":321681407},{"type":"v1","addr":"172.21.15.27:6835","nonce":321681407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:46:39.483941+0000","modified":"2024-01-11T09:46:42.996786+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11331},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11331":{"gid":11331,"name":"a","rank":0,"incarnation":129,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6834/3477661846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3477661846},{"type":"v1","addr":"172.21.15.27:6834","nonce":3477661846}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":16}]} 2024-01-11T09:47:01.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.357+0000 7f7302fff700 1 -- 172.21.15.27:0/3048851892 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f72e0067170 msgr2=0x7f72e0069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:01.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.357+0000 7f7302fff700 1 --2- 172.21.15.27:0/3048851892 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f72e0067170 0x7f72e0069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.357+0000 7f7302fff700 1 -- 172.21.15.27:0/3048851892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72fc0b09e0 msgr2=0x7f72fc143ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:01.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.357+0000 7f7302fff700 1 --2- 172.21.15.27:0/3048851892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72fc0b09e0 0x7f72fc143ad0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f72e800c7e0 tx=0x7f72e8007df0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.357+0000 7f7302fff700 1 -- 172.21.15.27:0/3048851892 shutdown_connections 2024-01-11T09:47:01.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.357+0000 7f7302fff700 1 --2- 172.21.15.27:0/3048851892 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f72e0067170 0x7f72e0069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.357+0000 7f7302fff700 1 --2- 172.21.15.27:0/3048851892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72fc0c8620 0x7f72fc0c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.357+0000 7f7302fff700 1 --2- 172.21.15.27:0/3048851892 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f72fc069d40 0x7f72fc155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.357+0000 7f7302fff700 1 --2- 172.21.15.27:0/3048851892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72fc0b09e0 0x7f72fc143ad0 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.357+0000 7f7302fff700 1 -- 172.21.15.27:0/3048851892 >> 172.21.15.27:0/3048851892 conn(0x7f72fc0680e0 msgr2=0x7f72fc1588a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:01.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.357+0000 7f7302fff700 1 -- 172.21.15.27:0/3048851892 shutdown_connections 2024-01-11T09:47:01.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.357+0000 7f7302fff700 1 -- 172.21.15.27:0/3048851892 wait complete. 2024-01-11T09:47:01.349 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 132 2024-01-11T09:47:01.368 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:47:01.390 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:01.399+0000 7f95c8a461c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:47:01.569 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:01.578+0000 7f923a65f1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:47:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.580+0000 7f2fe75ff700 1 Processor -- start 2024-01-11T09:47:01.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.580+0000 7f2fe75ff700 1 -- start start 2024-01-11T09:47:01.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.581+0000 7f2fe75ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2fe00ca7b0 0x7f2fe00cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:01.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.581+0000 7f2fe75ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe015d640 con 0x7f2fe00ca3e0 2024-01-11T09:47:01.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.582+0000 7f2fe75ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe015a5d0 con 0x7f2fe0159e80 2024-01-11T09:47:01.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.582+0000 7f2fe75ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe015ae70 con 0x7f2fe00ca7b0 2024-01-11T09:47:01.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.582+0000 7f2fe539b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2fe00ca3e0 0x7f2fe00c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44744/0 (socket says 172.21.15.27:44744) 2024-01-11T09:47:01.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.582+0000 7f2fe539b700 1 -- 172.21.15.27:0/1754534159 learned_addr learned my addr 172.21.15.27:0/1754534159 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:01.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.582+0000 7f2fe4b9a700 1 --2- 172.21.15.27:0/1754534159 >> v2:172.21.15.77:3300/0 conn(0x7f2fe00ca7b0 0x7f2fe00cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:01.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.582+0000 7f2fd7fff700 1 -- 172.21.15.27:0/1754534159 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555531830 0 0) 0x7f2fe015d640 con 0x7f2fe00ca3e0 2024-01-11T09:47:01.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.582+0000 7f2fd7fff700 1 -- 172.21.15.27:0/1754534159 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fcc003570 con 0x7f2fe00ca3e0 2024-01-11T09:47:01.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.582+0000 7f2fd7fff700 1 -- 172.21.15.27:0/1754534159 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445442654 0 0) 0x7f2fe015a5d0 con 0x7f2fe0159e80 2024-01-11T09:47:01.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.582+0000 7f2fd7fff700 1 -- 172.21.15.27:0/1754534159 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fe015d640 con 0x7f2fe0159e80 2024-01-11T09:47:01.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.583+0000 7f2fd7fff700 1 -- 172.21.15.27:0/1754534159 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 58050953 0 0) 0x7f2fcc003570 con 0x7f2fe00ca3e0 2024-01-11T09:47:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.583+0000 7f2fe4b9a700 1 -- 172.21.15.27:0/1754534159 >> v1:172.21.15.77:6789/0 conn(0x7f2fe0159e80 legacy=0x7f2fe015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.583+0000 7f2fe4b9a700 1 -- 172.21.15.27:0/1754534159 >> v1:172.21.15.27:6789/0 conn(0x7f2fe00ca3e0 legacy=0x7f2fe00c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.583+0000 7f2fe4b9a700 1 -- 172.21.15.27:0/1754534159 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe015a8b0 con 0x7f2fe00ca7b0 2024-01-11T09:47:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.583+0000 7f2fe4b9a700 1 --2- 172.21.15.27:0/1754534159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe00ca7b0 0x7f2fe00cca20 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009910 tx=0x7f2fd00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c743cad34a6b17e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.583+0000 7f2fd7fff700 1 -- 172.21.15.27:0/1754534159 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc001470 con 0x7f2fe00ca7b0 2024-01-11T09:47:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.583+0000 7f2fd7fff700 1 -- 172.21.15.27:0/1754534159 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd0011420 con 0x7f2fe00ca7b0 2024-01-11T09:47:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.583+0000 7f2fd7fff700 1 -- 172.21.15.27:0/1754534159 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd00117c0 con 0x7f2fe00ca7b0 2024-01-11T09:47:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.584+0000 7f2fe75ff700 1 -- 172.21.15.27:0/1754534159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe00ca7b0 msgr2=0x7f2fe00cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.584+0000 7f2fe75ff700 1 --2- 172.21.15.27:0/1754534159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe00ca7b0 0x7f2fe00cca20 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009910 tx=0x7f2fd00091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.584+0000 7f2fe75ff700 1 -- 172.21.15.27:0/1754534159 shutdown_connections 2024-01-11T09:47:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.584+0000 7f2fe75ff700 1 --2- 172.21.15.27:0/1754534159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe00ca7b0 0x7f2fe00cca20 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.584+0000 7f2fe75ff700 1 -- 172.21.15.27:0/1754534159 >> 172.21.15.27:0/1754534159 conn(0x7f2fe00ce3e0 msgr2=0x7f2fe00ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.584+0000 7f2fe75ff700 1 -- 172.21.15.27:0/1754534159 shutdown_connections 2024-01-11T09:47:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.584+0000 7f2fe75ff700 1 -- 172.21.15.27:0/1754534159 wait complete. 2024-01-11T09:47:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.586+0000 7f2fe75ff700 1 Processor -- start 2024-01-11T09:47:01.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.586+0000 7f2fe75ff700 1 -- start start 2024-01-11T09:47:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.588+0000 7f2fe75ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2fe00ca3e0 0x7f2fe0142d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.588+0000 7f2fe75ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe00ca7b0 0x7f2fe006ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.588+0000 7f2fe75ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fe0159e80 0x7f2fe006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.588+0000 7f2fe75ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe00b36e0 con 0x7f2fe0159e80 2024-01-11T09:47:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.588+0000 7f2fe75ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe00b39c0 con 0x7f2fe00ca7b0 2024-01-11T09:47:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.588+0000 7f2fe75ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe00b3ca0 con 0x7f2fe00ca3e0 2024-01-11T09:47:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.588+0000 7f2fe5b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fe0159e80 0x7f2fe006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.588+0000 7f2fe539b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2fe00ca3e0 0x7f2fe0142d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.588+0000 7f2fe4b9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe00ca7b0 0x7f2fe006ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.588+0000 7f2fe539b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2fe00ca3e0 0x7f2fe0142d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:38778/0 (socket says 172.21.15.27:38778) 2024-01-11T09:47:01.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.588+0000 7f2fe539b700 1 -- 172.21.15.27:0/433207825 learned_addr learned my addr 172.21.15.27:0/433207825 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:01.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.588+0000 7f2fe5b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fe0159e80 0x7f2fe006d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41860/0 (socket says 172.21.15.27:41860) 2024-01-11T09:47:01.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.588+0000 7f2fe539b700 1 -- 172.21.15.27:0/433207825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe00ca7b0 msgr2=0x7f2fe006ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:01.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.588+0000 7f2fe539b700 1 --2- 172.21.15.27:0/433207825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe00ca7b0 0x7f2fe006ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.588+0000 7f2fe539b700 1 -- 172.21.15.27:0/433207825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fe0159e80 msgr2=0x7f2fe006d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:01.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.588+0000 7f2fe539b700 1 --2- 172.21.15.27:0/433207825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fe0159e80 0x7f2fe006d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.588+0000 7f2fe539b700 1 -- 172.21.15.27:0/433207825 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd0009420 con 0x7f2fe00ca3e0 2024-01-11T09:47:01.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.588+0000 7f2fe5b9c700 1 --2- 172.21.15.27:0/433207825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fe0159e80 0x7f2fe006d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:01.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.589+0000 7f2fe539b700 1 --2- 172.21.15.27:0/433207825 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2fe00ca3e0 0x7f2fe0142d50 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00cb90 tx=0x7f2fdc001a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:01.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.589+0000 7f2fd57fa700 1 -- 172.21.15.27:0/433207825 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc007150 con 0x7f2fe00ca3e0 2024-01-11T09:47:01.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.589+0000 7f2fd57fa700 1 -- 172.21.15.27:0/433207825 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fdc0072d0 con 0x7f2fe00ca3e0 2024-01-11T09:47:01.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.589+0000 7f2fd57fa700 1 -- 172.21.15.27:0/433207825 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc007690 con 0x7f2fe00ca3e0 2024-01-11T09:47:01.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.589+0000 7f2fe75ff700 1 -- 172.21.15.27:0/433207825 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe00b3f80 con 0x7f2fe00ca3e0 2024-01-11T09:47:01.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.589+0000 7f2fe75ff700 1 -- 172.21.15.27:0/433207825 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe0143290 con 0x7f2fe00ca3e0 2024-01-11T09:47:01.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.590+0000 7f2fe75ff700 1 -- 172.21.15.27:0/433207825 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe00718d0 con 0x7f2fe00ca3e0 2024-01-11T09:47:01.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.591+0000 7f2fd57fa700 1 -- 172.21.15.27:0/433207825 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 141) v1 ==== 84947+0+0 (secure 0 0 0) 0x7f2fdc004a40 con 0x7f2fe00ca3e0 2024-01-11T09:47:01.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.592+0000 7f2fd57fa700 1 --2- 172.21.15.27:0/433207825 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2fc4067170 0x7f2fc4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:01.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.592+0000 7f2fd57fa700 1 -- 172.21.15.27:0/433207825 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 7111+0+0 (secure 0 0 0) 0x7f2fdc0845b0 con 0x7f2fe00ca3e0 2024-01-11T09:47:01.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.592+0000 7f2fe4b9a700 1 -- 172.21.15.27:0/433207825 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2fc4067170 msgr2=0x7f2fc4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:01.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.592+0000 7f2fe4b9a700 1 --2- 172.21.15.27:0/433207825 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2fc4067170 0x7f2fc4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:01.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.597+0000 7f2fd57fa700 1 -- 172.21.15.27:0/433207825 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2fdc050ff0 con 0x7f2fe00ca3e0 2024-01-11T09:47:01.600 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:01.609+0000 7f95c8a461c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:47:01.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.793+0000 7f2fe4b9a700 1 -- 172.21.15.27:0/433207825 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2fc4067170 msgr2=0x7f2fc4069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:01.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.793+0000 7f2fe4b9a700 1 --2- 172.21.15.27:0/433207825 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2fc4067170 0x7f2fc4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:01.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.846+0000 7f2fe75ff700 1 -- 172.21.15.27:0/433207825 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fe00b3de0 con 0x7f2fe00ca3e0 2024-01-11T09:47:01.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.846+0000 7f2fd57fa700 1 -- 172.21.15.27:0/433207825 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5688 (secure 0 0 0) 0x7f2fe00b3de0 con 0x7f2fe00ca3e0 2024-01-11T09:47:01.838 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:01.838 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":132,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1568813392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1568813392},{"type":"v1","addr":"172.21.15.77:6834","nonce":1568813392}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122},{"gid":10969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1352491019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1352491019},{"type":"v1","addr":"172.21.15.77:6835","nonce":1352491019}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:46:25.726042+0000","modified":"2024-01-11T09:46:29.257798+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11340":{"gid":11340,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/321681407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":321681407},{"type":"v1","addr":"172.21.15.27:6835","nonce":321681407}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:46:39.483941+0000","modified":"2024-01-11T09:46:42.996786+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11331},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11331":{"gid":11331,"name":"a","rank":0,"incarnation":129,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6834/3477661846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3477661846},{"type":"v1","addr":"172.21.15.27:6834","nonce":3477661846}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":16}]} 2024-01-11T09:47:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.849+0000 7f2fe75ff700 1 -- 172.21.15.27:0/433207825 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2fc4067170 msgr2=0x7f2fc4069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.849+0000 7f2fe75ff700 1 --2- 172.21.15.27:0/433207825 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2fc4067170 0x7f2fc4069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.849+0000 7f2fe75ff700 1 -- 172.21.15.27:0/433207825 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2fe00ca3e0 msgr2=0x7f2fe0142d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.849+0000 7f2fe75ff700 1 --2- 172.21.15.27:0/433207825 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2fe00ca3e0 0x7f2fe0142d50 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00cb90 tx=0x7f2fdc001a60 comp rx=0 tx=0).stop 2024-01-11T09:47:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.849+0000 7f2fe75ff700 1 -- 172.21.15.27:0/433207825 shutdown_connections 2024-01-11T09:47:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.849+0000 7f2fe75ff700 1 --2- 172.21.15.27:0/433207825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fe0159e80 0x7f2fe006d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.849+0000 7f2fe75ff700 1 --2- 172.21.15.27:0/433207825 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2fe00ca3e0 0x7f2fe0142d50 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.849+0000 7f2fe75ff700 1 --2- 172.21.15.27:0/433207825 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2fc4067170 0x7f2fc4069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.849+0000 7f2fe75ff700 1 --2- 172.21.15.27:0/433207825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fe00ca7b0 0x7f2fe006ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.849+0000 7f2fe75ff700 1 -- 172.21.15.27:0/433207825 >> 172.21.15.27:0/433207825 conn(0x7f2fe00ce3e0 msgr2=0x7f2fe0151600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.849+0000 7f2fe75ff700 1 -- 172.21.15.27:0/433207825 shutdown_connections 2024-01-11T09:47:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:01.849+0000 7f2fe75ff700 1 -- 172.21.15.27:0/433207825 wait complete. 2024-01-11T09:47:01.842 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 132 2024-01-11T09:47:01.860 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:47:01.993 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.002+0000 7f923a65f1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:47:02.010 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:02.019+0000 7f95c8a461c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:47:02.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.117+0000 7f7f885ff700 1 Processor -- start 2024-01-11T09:47:02.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.117+0000 7f7f885ff700 1 -- start start 2024-01-11T09:47:02.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.119+0000 7f7f885ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f80069d40 0x7f7f800b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:02.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.119+0000 7f7f885ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f80157390 con 0x7f7f800c8620 2024-01-11T09:47:02.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.119+0000 7f7f885ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f8006a200 con 0x7f7f800c89f0 2024-01-11T09:47:02.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.119+0000 7f7f885ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f80156d40 con 0x7f7f80069d40 2024-01-11T09:47:02.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.119+0000 7f7f8639b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7f800c8620 0x7f7f800af970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44750/0 (socket says 172.21.15.27:44750) 2024-01-11T09:47:02.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.119+0000 7f7f8639b700 1 -- 172.21.15.27:0/1263930437 learned_addr learned my addr 172.21.15.27:0/1263930437 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:02.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.119+0000 7f7f85b9a700 1 --2- 172.21.15.27:0/1263930437 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f80069d40 0x7f7f800b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:02.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.119+0000 7f7f85399700 1 -- 172.21.15.27:0/1263930437 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822520734 0 0) 0x7f7f80157390 con 0x7f7f800c8620 2024-01-11T09:47:02.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.120+0000 7f7f85399700 1 -- 172.21.15.27:0/1263930437 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f6c003570 con 0x7f7f800c8620 2024-01-11T09:47:02.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.120+0000 7f7f85399700 1 -- 172.21.15.27:0/1263930437 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416403939 0 0) 0x7f7f8006a200 con 0x7f7f800c89f0 2024-01-11T09:47:02.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.120+0000 7f7f85399700 1 -- 172.21.15.27:0/1263930437 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f80157390 con 0x7f7f800c89f0 2024-01-11T09:47:02.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.120+0000 7f7f85b9a700 1 -- 172.21.15.27:0/1263930437 >> v1:172.21.15.77:6789/0 conn(0x7f7f800c89f0 legacy=0x7f7f800accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:02.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.120+0000 7f7f85b9a700 1 -- 172.21.15.27:0/1263930437 >> v1:172.21.15.27:6789/0 conn(0x7f7f800c8620 legacy=0x7f7f800af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:02.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.120+0000 7f7f85b9a700 1 -- 172.21.15.27:0/1263930437 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f80157020 con 0x7f7f80069d40 2024-01-11T09:47:02.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.120+0000 7f7f85b9a700 1 --2- 172.21.15.27:0/1263930437 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f80069d40 0x7f7f800b02d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7f70009940 tx=0x7f7f700091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbd1c873e43abf80 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:02.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.120+0000 7f7f85399700 1 -- 172.21.15.27:0/1263930437 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f70011630 con 0x7f7f80069d40 2024-01-11T09:47:02.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.120+0000 7f7f85399700 1 -- 172.21.15.27:0/1263930437 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f70011790 con 0x7f7f80069d40 2024-01-11T09:47:02.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.120+0000 7f7f85399700 1 -- 172.21.15.27:0/1263930437 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f70011b10 con 0x7f7f80069d40 2024-01-11T09:47:02.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.121+0000 7f7f885ff700 1 -- 172.21.15.27:0/1263930437 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f80069d40 msgr2=0x7f7f800b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:02.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.121+0000 7f7f885ff700 1 --2- 172.21.15.27:0/1263930437 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f80069d40 0x7f7f800b02d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7f70009940 tx=0x7f7f700091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:02.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.121+0000 7f7f885ff700 1 -- 172.21.15.27:0/1263930437 shutdown_connections 2024-01-11T09:47:02.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.121+0000 7f7f885ff700 1 --2- 172.21.15.27:0/1263930437 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f80069d40 0x7f7f800b02d0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:02.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.121+0000 7f7f885ff700 1 -- 172.21.15.27:0/1263930437 >> 172.21.15.27:0/1263930437 conn(0x7f7f800680e0 msgr2=0x7f7f80069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:02.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.121+0000 7f7f885ff700 1 -- 172.21.15.27:0/1263930437 shutdown_connections 2024-01-11T09:47:02.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.121+0000 7f7f885ff700 1 -- 172.21.15.27:0/1263930437 wait complete. 2024-01-11T09:47:02.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.123+0000 7f7f885ff700 1 Processor -- start 2024-01-11T09:47:02.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.123+0000 7f7f885ff700 1 -- start start 2024-01-11T09:47:02.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.124+0000 7f7f885ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f80069d40 0x7f7f800cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:02.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.124+0000 7f7f885ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f800c8620 0x7f7f80155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:02.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.124+0000 7f7f885ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f800c89f0 0x7f7f80145830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:02.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.124+0000 7f7f885ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f800cb910 con 0x7f7f800c8620 2024-01-11T09:47:02.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.124+0000 7f7f885ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f800accc0 con 0x7f7f800c89f0 2024-01-11T09:47:02.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.124+0000 7f7f885ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f80152ed0 con 0x7f7f80069d40 2024-01-11T09:47:02.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.124+0000 7f7f85b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f800c8620 0x7f7f80155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:02.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.124+0000 7f7f8639b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f80069d40 0x7f7f800cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:02.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.124+0000 7f7f86b9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f800c89f0 0x7f7f80145830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:02.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.124+0000 7f7f8639b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f80069d40 0x7f7f800cb3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:38794/0 (socket says 172.21.15.27:38794) 2024-01-11T09:47:02.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.124+0000 7f7f8639b700 1 -- 172.21.15.27:0/1605060017 learned_addr learned my addr 172.21.15.27:0/1605060017 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:02.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.125+0000 7f7f8639b700 1 -- 172.21.15.27:0/1605060017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f800c89f0 msgr2=0x7f7f80145830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:02.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.125+0000 7f7f8639b700 1 --2- 172.21.15.27:0/1605060017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f800c89f0 0x7f7f80145830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:02.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.125+0000 7f7f8639b700 1 -- 172.21.15.27:0/1605060017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f800c8620 msgr2=0x7f7f80155150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:02.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.125+0000 7f7f8639b700 1 --2- 172.21.15.27:0/1605060017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f800c8620 0x7f7f80155150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:02.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.125+0000 7f7f8639b700 1 -- 172.21.15.27:0/1605060017 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f70009420 con 0x7f7f80069d40 2024-01-11T09:47:02.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.125+0000 7f7f8639b700 1 --2- 172.21.15.27:0/1605060017 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f80069d40 0x7f7f800cb3d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c00c7c0 tx=0x7f7f7c007e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:02.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.125+0000 7f7f767fc700 1 -- 172.21.15.27:0/1605060017 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c005b30 con 0x7f7f80069d40 2024-01-11T09:47:02.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.125+0000 7f7f767fc700 1 -- 172.21.15.27:0/1605060017 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7c005cd0 con 0x7f7f80069d40 2024-01-11T09:47:02.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.125+0000 7f7f885ff700 1 -- 172.21.15.27:0/1605060017 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f800ac0b0 con 0x7f7f80069d40 2024-01-11T09:47:02.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.125+0000 7f7f767fc700 1 -- 172.21.15.27:0/1605060017 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c007320 con 0x7f7f80069d40 2024-01-11T09:47:02.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.125+0000 7f7f885ff700 1 -- 172.21.15.27:0/1605060017 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f800ac670 con 0x7f7f80069d40 2024-01-11T09:47:02.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.126+0000 7f7f885ff700 1 -- 172.21.15.27:0/1605060017 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f800718d0 con 0x7f7f80069d40 2024-01-11T09:47:02.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.127+0000 7f7f767fc700 1 -- 172.21.15.27:0/1605060017 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 141) v1 ==== 84947+0+0 (secure 0 0 0) 0x7f7f7c012400 con 0x7f7f80069d40 2024-01-11T09:47:02.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.129+0000 7f7f767fc700 1 --2- 172.21.15.27:0/1605060017 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7f64067170 0x7f7f64069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:02.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.129+0000 7f7f767fc700 1 -- 172.21.15.27:0/1605060017 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 7111+0+0 (secure 0 0 0) 0x7f7f7c0842a0 con 0x7f7f80069d40 2024-01-11T09:47:02.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.129+0000 7f7f85b9a700 1 -- 172.21.15.27:0/1605060017 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7f64067170 msgr2=0x7f7f64069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:02.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.129+0000 7f7f85b9a700 1 --2- 172.21.15.27:0/1605060017 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7f64067170 0x7f7f64069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:02.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.136+0000 7f7f767fc700 1 -- 172.21.15.27:0/1605060017 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7f7c04b5a0 con 0x7f7f80069d40 2024-01-11T09:47:02.183 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.192+0000 7f923a65f1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:47:02.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.330+0000 7f7f85b9a700 1 -- 172.21.15.27:0/1605060017 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7f64067170 msgr2=0x7f7f64069620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:02.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.330+0000 7f7f85b9a700 1 --2- 172.21.15.27:0/1605060017 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7f64067170 0x7f7f64069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:02.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.345+0000 7f7f885ff700 1 -- 172.21.15.27:0/1605060017 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f800b1150 con 0x7f7f80069d40 2024-01-11T09:47:02.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.346+0000 7f7f767fc700 1 -- 172.21.15.27:0/1605060017 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v240) v1 ==== 74+0+19338 (secure 0 0 0) 0x7f7f7c050d10 con 0x7f7f80069d40 2024-01-11T09:47:02.338 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:02.338 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":240,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:46:54.200005+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:46:24.049904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-01-11T09:46:24.737495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-01-11T09:46:37.665603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"238","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-01-11T09:46:38.472134+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"239","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/210673663":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/3835722213":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/696729061":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/206349247":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/3398849669":"2024-01-12T09:46:54.199949+0000","172.21.15.77:6837/59776":"2024-01-12T09:46:54.199949+0000","172.21.15.77:6836/59776":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/1934793537":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/1366958778":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/198451994":"2024-01-12T09:46:54.199949+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:47:02.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.349+0000 7f7f885ff700 1 -- 172.21.15.27:0/1605060017 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7f64067170 msgr2=0x7f7f64069620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:02.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.349+0000 7f7f885ff700 1 --2- 172.21.15.27:0/1605060017 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7f64067170 0x7f7f64069620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:02.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.349+0000 7f7f885ff700 1 -- 172.21.15.27:0/1605060017 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f80069d40 msgr2=0x7f7f800cb3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:02.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.349+0000 7f7f885ff700 1 --2- 172.21.15.27:0/1605060017 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f80069d40 0x7f7f800cb3d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c00c7c0 tx=0x7f7f7c007e80 comp rx=0 tx=0).stop 2024-01-11T09:47:02.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.349+0000 7f7f885ff700 1 -- 172.21.15.27:0/1605060017 shutdown_connections 2024-01-11T09:47:02.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.349+0000 7f7f885ff700 1 --2- 172.21.15.27:0/1605060017 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7f64067170 0x7f7f64069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:02.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.349+0000 7f7f885ff700 1 --2- 172.21.15.27:0/1605060017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f800c89f0 0x7f7f80145830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:02.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.349+0000 7f7f885ff700 1 --2- 172.21.15.27:0/1605060017 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f80069d40 0x7f7f800cb3d0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:02.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.349+0000 7f7f885ff700 1 --2- 172.21.15.27:0/1605060017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f800c8620 0x7f7f80155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:02.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.349+0000 7f7f885ff700 1 -- 172.21.15.27:0/1605060017 >> 172.21.15.27:0/1605060017 conn(0x7f7f800680e0 msgr2=0x7f7f80153550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:02.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.349+0000 7f7f885ff700 1 -- 172.21.15.27:0/1605060017 shutdown_connections 2024-01-11T09:47:02.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.349+0000 7f7f885ff700 1 -- 172.21.15.27:0/1605060017 wait complete. 2024-01-11T09:47:02.359 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-11T09:47:02.374 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.383+0000 7f923a65f1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:47:02.476 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:02.484+0000 7f95c8a461c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:47:02.483 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.492+0000 7f923a65f1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:47:02.567 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.575+0000 7f91b7c49700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.567 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.575+0000 7f91b7c49700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.567 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.575+0000 7f91b7c49700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.567 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.575+0000 7f91b7c49700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.567 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.575+0000 7f91b7c49700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.567 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.575+0000 7f91b5444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.567 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.575+0000 7f91b5444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.567 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.575+0000 7f91b5444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.567 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.575+0000 7f91b5444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.568 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.575+0000 7f91b5444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.578+0000 7fdaff7ff700 1 Processor -- start 2024-01-11T09:47:02.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.578+0000 7fdaff7ff700 1 -- start start 2024-01-11T09:47:02.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.580+0000 7fdaff7ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdaf8068cd0 0x7fdaf815a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:02.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.580+0000 7fdaff7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf80b4bd0 con 0x7fdaf815ca10 2024-01-11T09:47:02.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.580+0000 7fdaff7ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf815ac70 con 0x7fdaf815cde0 2024-01-11T09:47:02.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.580+0000 7fdaff7ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaf80b3fc0 con 0x7fdaf8068cd0 2024-01-11T09:47:02.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.581+0000 7fdafcd9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdaf8068cd0 0x7fdaf815a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:02.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.581+0000 7fdafcd9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdaf8068cd0 0x7fdaf815a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41874/0 (socket says 172.21.15.27:41874) 2024-01-11T09:47:02.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.581+0000 7fdafcd9a700 1 -- 172.21.15.27:0/2428737449 learned_addr learned my addr 172.21.15.27:0/2428737449 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:02.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.581+0000 7fdafdd9c700 1 --1- 172.21.15.27:0/2428737449 >> v1:172.21.15.77:6789/0 conn(0x7fdaf815cde0 0x7fdaf80b5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:59056/0 (socket says 172.21.15.27:59056) 2024-01-11T09:47:02.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.581+0000 7fdafd59b700 1 --1- 172.21.15.27:0/2428737449 >> v1:172.21.15.27:6789/0 conn(0x7fdaf815ca10 0x7fdaf80685c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44760/0 (socket says 172.21.15.27:44760) 2024-01-11T09:47:02.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.581+0000 7fdafcd9a700 1 -- 172.21.15.27:0/2428737449 >> v1:172.21.15.77:6789/0 conn(0x7fdaf815cde0 legacy=0x7fdaf80b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:02.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.581+0000 7fdafcd9a700 1 -- 172.21.15.27:0/2428737449 >> v1:172.21.15.27:6789/0 conn(0x7fdaf815ca10 legacy=0x7fdaf80685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:02.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.581+0000 7fdafcd9a700 1 -- 172.21.15.27:0/2428737449 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf80b42a0 con 0x7fdaf8068cd0 2024-01-11T09:47:02.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.581+0000 7fdaeffff700 1 -- 172.21.15.27:0/2428737449 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260733256 0 0) 0x7fdaf815ac70 con 0x7fdaf815cde0 2024-01-11T09:47:02.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.582+0000 7fdafcd9a700 1 --2- 172.21.15.27:0/2428737449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaf8068cd0 0x7fdaf815a730 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fdae80099e0 tx=0x7fdae8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f58347d4a5b84ee5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:02.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.582+0000 7fdaeffff700 1 -- 172.21.15.27:0/2428737449 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae800b7e0 con 0x7fdaf8068cd0 2024-01-11T09:47:02.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.582+0000 7fdaeffff700 1 -- 172.21.15.27:0/2428737449 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae800b940 con 0x7fdaf8068cd0 2024-01-11T09:47:02.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.582+0000 7fdaeffff700 1 -- 172.21.15.27:0/2428737449 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae800bce0 con 0x7fdaf8068cd0 2024-01-11T09:47:02.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.582+0000 7fdaff7ff700 1 -- 172.21.15.27:0/2428737449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaf8068cd0 msgr2=0x7fdaf815a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:02.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.582+0000 7fdaff7ff700 1 --2- 172.21.15.27:0/2428737449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaf8068cd0 0x7fdaf815a730 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fdae80099e0 tx=0x7fdae8009250 comp rx=0 tx=0).stop 2024-01-11T09:47:02.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.583+0000 7fdaff7ff700 1 -- 172.21.15.27:0/2428737449 shutdown_connections 2024-01-11T09:47:02.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.583+0000 7fdaff7ff700 1 --2- 172.21.15.27:0/2428737449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaf8068cd0 0x7fdaf815a730 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:02.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.583+0000 7fdaff7ff700 1 -- 172.21.15.27:0/2428737449 >> 172.21.15.27:0/2428737449 conn(0x7fdaf806ed40 msgr2=0x7fdaf806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:02.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.583+0000 7fdaff7ff700 1 -- 172.21.15.27:0/2428737449 shutdown_connections 2024-01-11T09:47:02.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.583+0000 7fdaff7ff700 1 -- 172.21.15.27:0/2428737449 wait complete. 2024-01-11T09:47:02.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.584+0000 7fdaff7ff700 1 Processor -- start 2024-01-11T09:47:02.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.585+0000 7fdaff7ff700 1 -- start start 2024-01-11T09:47:02.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.585+0000 7fdaff7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaf8068cd0 0x7fdaf816ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:02.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.585+0000 7fdaff7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaf815ca10 0x7fdaf816af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:02.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.585+0000 7fdaff7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdaf815cde0 0x7fdaf80b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:02.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.586+0000 7fdaff7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf816c5a0 con 0x7fdaf815ca10 2024-01-11T09:47:02.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.586+0000 7fdaff7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf816b990 con 0x7fdaf8068cd0 2024-01-11T09:47:02.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.586+0000 7fdaff7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf816bc70 con 0x7fdaf815cde0 2024-01-11T09:47:02.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.586+0000 7fdafcd9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaf815ca10 0x7fdaf816af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:02.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.586+0000 7fdafcd9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaf815ca10 0x7fdaf816af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41882/0 (socket says 172.21.15.27:41882) 2024-01-11T09:47:02.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.586+0000 7fdafd59b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaf8068cd0 0x7fdaf816ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:02.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.586+0000 7fdafcd9a700 1 -- 172.21.15.27:0/884703388 learned_addr learned my addr 172.21.15.27:0/884703388 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:02.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.586+0000 7fdafdd9c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdaf815cde0 0x7fdaf80b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:02.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.586+0000 7fdafcd9a700 1 -- 172.21.15.27:0/884703388 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdaf815cde0 msgr2=0x7fdaf80b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:02.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.586+0000 7fdafcd9a700 1 --2- 172.21.15.27:0/884703388 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdaf815cde0 0x7fdaf80b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:02.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.586+0000 7fdafcd9a700 1 -- 172.21.15.27:0/884703388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaf8068cd0 msgr2=0x7fdaf816ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:02.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.586+0000 7fdafcd9a700 1 --2- 172.21.15.27:0/884703388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaf8068cd0 0x7fdaf816ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:02.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.587+0000 7fdafcd9a700 1 -- 172.21.15.27:0/884703388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae80094f0 con 0x7fdaf815ca10 2024-01-11T09:47:02.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.587+0000 7fdafcd9a700 1 --2- 172.21.15.27:0/884703388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaf815ca10 0x7fdaf816af40 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7fdae8009290 tx=0x7fdae8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:02.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.587+0000 7fdaed7fa700 1 -- 172.21.15.27:0/884703388 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae8011c70 con 0x7fdaf815ca10 2024-01-11T09:47:02.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.587+0000 7fdaed7fa700 1 -- 172.21.15.27:0/884703388 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae8020070 con 0x7fdaf815ca10 2024-01-11T09:47:02.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.587+0000 7fdaff7ff700 1 -- 172.21.15.27:0/884703388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaf816bf50 con 0x7fdaf815ca10 2024-01-11T09:47:02.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.587+0000 7fdaed7fa700 1 -- 172.21.15.27:0/884703388 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae801b450 con 0x7fdaf815ca10 2024-01-11T09:47:02.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.587+0000 7fdaff7ff700 1 -- 172.21.15.27:0/884703388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf80b2c90 con 0x7fdaf815ca10 2024-01-11T09:47:02.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.589+0000 7fdaff7ff700 1 -- 172.21.15.27:0/884703388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf804c3c0 con 0x7fdaf815ca10 2024-01-11T09:47:02.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.590+0000 7fdaed7fa700 1 -- 172.21.15.27:0/884703388 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 142) v1 ==== 84453+0+0 (secure 0 0 0) 0x7fdae8022030 con 0x7fdaf815ca10 2024-01-11T09:47:02.581 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.590+0000 7f91b5444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.581 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.590+0000 7f91b5444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.582 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.590+0000 7f91b5444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.582 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.590+0000 7f91b5444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.582 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.590+0000 7f91b5444700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.591+0000 7fdaed7fa700 1 -- 172.21.15.27:0/884703388 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 7455+0+0 (secure 0 0 0) 0x7fdae8086590 con 0x7fdaf815ca10 2024-01-11T09:47:02.583 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.591+0000 7f91b7c49700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.583 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.591+0000 7f91b7c49700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.583 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.591+0000 7f91b7c49700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.583 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.591+0000 7f91b7c49700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.583 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:02.591+0000 7f91b7c49700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:02.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.596+0000 7fdaed7fa700 1 -- 172.21.15.27:0/884703388 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdae80534f0 con 0x7fdaf815ca10 2024-01-11T09:47:02.689 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:02.697+0000 7f95c8a461c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:47:02.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.846+0000 7fdaff7ff700 1 -- 172.21.15.27:0/884703388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fdaf804c3c0 con 0x7fdaf815ca10 2024-01-11T09:47:02.838 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-11T09:47:02.846+0000 7f27065f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-11T09:47:02.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.854+0000 7fdaed7fa700 1 -- 172.21.15.27:0/884703388 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v133) v1 ==== 161+0+0 (secure 0 0 0) 0x7fdae8056b10 con 0x7fdaf815ca10 2024-01-11T09:47:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.856+0000 7fdaff7ff700 1 -- 172.21.15.27:0/884703388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaf815ca10 msgr2=0x7fdaf816af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.856+0000 7fdaff7ff700 1 --2- 172.21.15.27:0/884703388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaf815ca10 0x7fdaf816af40 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7fdae8009290 tx=0x7fdae8016000 comp rx=0 tx=0).stop 2024-01-11T09:47:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.857+0000 7fdaff7ff700 1 -- 172.21.15.27:0/884703388 shutdown_connections 2024-01-11T09:47:02.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.857+0000 7fdaff7ff700 1 --2- 172.21.15.27:0/884703388 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fdaf815cde0 0x7fdaf80b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:02.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.857+0000 7fdaff7ff700 1 --2- 172.21.15.27:0/884703388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdaf8068cd0 0x7fdaf816ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:02.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.857+0000 7fdaff7ff700 1 --2- 172.21.15.27:0/884703388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaf815ca10 0x7fdaf816af40 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:02.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.857+0000 7fdaff7ff700 1 -- 172.21.15.27:0/884703388 >> 172.21.15.27:0/884703388 conn(0x7fdaf806ed40 msgr2=0x7fdaf815a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:02.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.857+0000 7fdaff7ff700 1 -- 172.21.15.27:0/884703388 shutdown_connections 2024-01-11T09:47:02.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:02.857+0000 7fdaff7ff700 1 -- 172.21.15.27:0/884703388 wait complete. 2024-01-11T09:47:02.849 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-11T09:47:02.851 INFO:tasks.ceph.mds.c.smithi027.stderr: -3025> 2024-01-11T09:46:31.204+0000 7f17f05f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T09:47:02.868 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-11T09:47:02.893 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:02.901+0000 7f95c8a461c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:47:02.900 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T09:47:03.017 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:03.026+0000 7f95c8a461c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:47:03.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.089+0000 7fbee55ff700 1 Processor -- start 2024-01-11T09:47:03.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.089+0000 7fbee55ff700 1 -- start start 2024-01-11T09:47:03.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.090+0000 7fbee55ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbee0157410 0x7fbee00ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:03.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.090+0000 7fbee55ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee01577e0 0x7fbee00ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:03.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.090+0000 7fbee55ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbee0069540 0x7fbee00ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:03.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.090+0000 7fbee55ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbee00699d0 con 0x7fbee0069540 2024-01-11T09:47:03.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.090+0000 7fbee55ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbee0069cb0 con 0x7fbee0157410 2024-01-11T09:47:03.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.090+0000 7fbee55ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbee0069f90 con 0x7fbee01577e0 2024-01-11T09:47:03.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.090+0000 7fbedf7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbee0069540 0x7fbee00ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:03.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.090+0000 7fbedf7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbee0069540 0x7fbee00ae540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41908/0 (socket says 172.21.15.27:41908) 2024-01-11T09:47:03.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.090+0000 7fbedf7fe700 1 -- 172.21.15.27:0/3660028386 learned_addr learned my addr 172.21.15.27:0/3660028386 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:03.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.091+0000 7fbedeffd700 1 --2- 172.21.15.27:0/3660028386 >> v2:172.21.15.77:3300/0 conn(0x7fbee0157410 0x7fbee00ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:03.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.091+0000 7fbede7fc700 1 --2- 172.21.15.27:0/3660028386 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee01577e0 0x7fbee00ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:03.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.091+0000 7fbedf7fe700 1 -- 172.21.15.27:0/3660028386 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee01577e0 msgr2=0x7fbee00ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:03.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.091+0000 7fbedf7fe700 1 --2- 172.21.15.27:0/3660028386 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee01577e0 0x7fbee00ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:03.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.091+0000 7fbedf7fe700 1 -- 172.21.15.27:0/3660028386 >> v2:172.21.15.77:3300/0 conn(0x7fbee0157410 msgr2=0x7fbee00ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:03.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.091+0000 7fbedf7fe700 1 --2- 172.21.15.27:0/3660028386 >> v2:172.21.15.77:3300/0 conn(0x7fbee0157410 0x7fbee00ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:03.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.091+0000 7fbedf7fe700 1 -- 172.21.15.27:0/3660028386 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee006a270 con 0x7fbee0069540 2024-01-11T09:47:03.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.091+0000 7fbedf7fe700 1 --2- 172.21.15.27:0/3660028386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee0069540 0x7fbee00ae540 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fbee0070c90 tx=0x7fbed8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53eb395aa0594056 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:03.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.091+0000 7fbeddffb700 1 -- 172.21.15.27:0/3660028386 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed800e070 con 0x7fbee0069540 2024-01-11T09:47:03.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.091+0000 7fbeddffb700 1 -- 172.21.15.27:0/3660028386 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed800b970 con 0x7fbee0069540 2024-01-11T09:47:03.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.091+0000 7fbeddffb700 1 -- 172.21.15.27:0/3660028386 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed80092e0 con 0x7fbee0069540 2024-01-11T09:47:03.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.092+0000 7fbee55ff700 1 -- 172.21.15.27:0/3660028386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee0069540 msgr2=0x7fbee00ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:03.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.092+0000 7fbee55ff700 1 --2- 172.21.15.27:0/3660028386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee0069540 0x7fbee00ae540 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fbee0070c90 tx=0x7fbed8009840 comp rx=0 tx=0).stop 2024-01-11T09:47:03.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.092+0000 7fbee55ff700 1 -- 172.21.15.27:0/3660028386 shutdown_connections 2024-01-11T09:47:03.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.092+0000 7fbee55ff700 1 --2- 172.21.15.27:0/3660028386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee0069540 0x7fbee00ae540 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:03.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.092+0000 7fbee55ff700 1 --2- 172.21.15.27:0/3660028386 >> v2:172.21.15.77:3300/0 conn(0x7fbee0157410 0x7fbee00ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:03.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.092+0000 7fbee55ff700 1 --2- 172.21.15.27:0/3660028386 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee01577e0 0x7fbee00ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:03.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.092+0000 7fbee55ff700 1 -- 172.21.15.27:0/3660028386 >> 172.21.15.27:0/3660028386 conn(0x7fbee00680a0 msgr2=0x7fbee00cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:03.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.092+0000 7fbee55ff700 1 -- 172.21.15.27:0/3660028386 shutdown_connections 2024-01-11T09:47:03.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.092+0000 7fbee55ff700 1 -- 172.21.15.27:0/3660028386 wait complete. 2024-01-11T09:47:03.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.094+0000 7fbee55ff700 1 Processor -- start 2024-01-11T09:47:03.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.094+0000 7fbee55ff700 1 -- start start 2024-01-11T09:47:03.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.095+0000 7fbee55ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee0069540 0x7fbee0153730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:03.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.095+0000 7fbee55ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee0157410 0x7fbee0148b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:03.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.095+0000 7fbee55ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee01577e0 0x7fbee0154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:03.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.095+0000 7fbee55ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee0153c70 con 0x7fbee0157410 2024-01-11T09:47:03.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.095+0000 7fbee55ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee0154ab0 con 0x7fbee0069540 2024-01-11T09:47:03.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.095+0000 7fbee55ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbee0153ea0 con 0x7fbee01577e0 2024-01-11T09:47:03.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.095+0000 7fbede7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee0157410 0x7fbee0148b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:03.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.095+0000 7fbede7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee0157410 0x7fbee0148b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41914/0 (socket says 172.21.15.27:41914) 2024-01-11T09:47:03.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.095+0000 7fbede7fc700 1 -- 172.21.15.27:0/1993448039 learned_addr learned my addr 172.21.15.27:0/1993448039 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:03.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.095+0000 7fbedf7fe700 1 --2- 172.21.15.27:0/1993448039 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee01577e0 0x7fbee0154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:03.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.095+0000 7fbedeffd700 1 --2- 172.21.15.27:0/1993448039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee0069540 0x7fbee0153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:03.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.095+0000 7fbede7fc700 1 -- 172.21.15.27:0/1993448039 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee01577e0 msgr2=0x7fbee0154f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:03.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.095+0000 7fbede7fc700 1 --2- 172.21.15.27:0/1993448039 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee01577e0 0x7fbee0154f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:03.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.095+0000 7fbede7fc700 1 -- 172.21.15.27:0/1993448039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee0069540 msgr2=0x7fbee0153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:03.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.095+0000 7fbede7fc700 1 --2- 172.21.15.27:0/1993448039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee0069540 0x7fbee0153730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:03.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.095+0000 7fbede7fc700 1 -- 172.21.15.27:0/1993448039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed8009ae0 con 0x7fbee0157410 2024-01-11T09:47:03.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.095+0000 7fbedf7fe700 1 --2- 172.21.15.27:0/1993448039 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee01577e0 0x7fbee0154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:03.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.096+0000 7fbede7fc700 1 --2- 172.21.15.27:0/1993448039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee0157410 0x7fbee0148b80 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fbed400e950 tx=0x7fbed40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:03.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.096+0000 7fbec6ffd700 1 -- 172.21.15.27:0/1993448039 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed400ee90 con 0x7fbee0157410 2024-01-11T09:47:03.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.096+0000 7fbec6ffd700 1 -- 172.21.15.27:0/1993448039 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed400c850 con 0x7fbee0157410 2024-01-11T09:47:03.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.096+0000 7fbec6ffd700 1 -- 172.21.15.27:0/1993448039 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed400cbf0 con 0x7fbee0157410 2024-01-11T09:47:03.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.096+0000 7fbee55ff700 1 -- 172.21.15.27:0/1993448039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee0154180 con 0x7fbee0157410 2024-01-11T09:47:03.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.096+0000 7fbee55ff700 1 -- 172.21.15.27:0/1993448039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee0154740 con 0x7fbee0157410 2024-01-11T09:47:03.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.097+0000 7fbee55ff700 1 -- 172.21.15.27:0/1993448039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee0154c60 con 0x7fbee0157410 2024-01-11T09:47:03.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.098+0000 7fbec6ffd700 1 -- 172.21.15.27:0/1993448039 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 142) v1 ==== 84453+0+0 (secure 0 0 0) 0x7fbed4013470 con 0x7fbee0157410 2024-01-11T09:47:03.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.099+0000 7fbec6ffd700 1 -- 172.21.15.27:0/1993448039 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 7541+0+0 (secure 0 0 0) 0x7fbed4084d40 con 0x7fbee0157410 2024-01-11T09:47:03.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.103+0000 7fbec6ffd700 1 -- 172.21.15.27:0/1993448039 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbee0154c60 con 0x7fbee0157410 2024-01-11T09:47:03.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.355+0000 7fbee55ff700 1 -- 172.21.15.27:0/1993448039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fbee0154c20 con 0x7fbee0157410 2024-01-11T09:47:03.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.505+0000 7fbec6ffd700 1 -- 172.21.15.27:0/1993448039 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 143) v1 ==== 84580+0+0 (secure 0 0 0) 0x7fbed4051600 con 0x7fbee0157410 2024-01-11T09:47:03.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.506+0000 7fbec6ffd700 1 --2- 172.21.15.27:0/1993448039 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbecc06f160 0x7fbecc071540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.507+0000 7fbedeffd700 1 --2- 172.21.15.27:0/1993448039 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbecc06f160 0x7fbecc071540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:03.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.512+0000 7fbedeffd700 1 --2- 172.21.15.27:0/1993448039 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbecc06f160 0x7fbecc071540 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009d30 tx=0x7fbed0008040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:03.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.852+0000 7fbec6ffd700 1 -- 172.21.15.27:0/1993448039 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v134) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbed406f5e0 con 0x7fbee0157410 2024-01-11T09:47:03.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.855+0000 7fbee55ff700 1 -- 172.21.15.27:0/1993448039 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbecc06f160 msgr2=0x7fbecc071540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:03.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.855+0000 7fbee55ff700 1 --2- 172.21.15.27:0/1993448039 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbecc06f160 0x7fbecc071540 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009d30 tx=0x7fbed0008040 comp rx=0 tx=0).stop 2024-01-11T09:47:03.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.855+0000 7fbee55ff700 1 -- 172.21.15.27:0/1993448039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee0157410 msgr2=0x7fbee0148b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:03.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.855+0000 7fbee55ff700 1 --2- 172.21.15.27:0/1993448039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee0157410 0x7fbee0148b80 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fbed400e950 tx=0x7fbed40091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:03.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.855+0000 7fbee55ff700 1 -- 172.21.15.27:0/1993448039 shutdown_connections 2024-01-11T09:47:03.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.855+0000 7fbee55ff700 1 --2- 172.21.15.27:0/1993448039 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fbee01577e0 0x7fbee0154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:03.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.855+0000 7fbee55ff700 1 --2- 172.21.15.27:0/1993448039 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fbecc06f160 0x7fbecc071540 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:03.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.855+0000 7fbee55ff700 1 --2- 172.21.15.27:0/1993448039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbee0069540 0x7fbee0153730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:03.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.855+0000 7fbee55ff700 1 --2- 172.21.15.27:0/1993448039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee0157410 0x7fbee0148b80 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:03.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.855+0000 7fbee55ff700 1 -- 172.21.15.27:0/1993448039 >> 172.21.15.27:0/1993448039 conn(0x7fbee00680a0 msgr2=0x7fbee0152090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:03.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.855+0000 7fbee55ff700 1 -- 172.21.15.27:0/1993448039 shutdown_connections 2024-01-11T09:47:03.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:03.855+0000 7fbee55ff700 1 -- 172.21.15.27:0/1993448039 wait complete. 2024-01-11T09:47:03.865 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-11T09:47:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.124+0000 7f17c53ff700 1 Processor -- start 2024-01-11T09:47:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.124+0000 7f17c53ff700 1 -- start start 2024-01-11T09:47:04.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.125+0000 7f17c53ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f17c0069d70 0x7f17c0162190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:04.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.125+0000 7f17c53ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c006d420 con 0x7f17c0157440 2024-01-11T09:47:04.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.125+0000 7f17c53ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c006d700 con 0x7f17c0157810 2024-01-11T09:47:04.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.125+0000 7f17c53ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c006d9e0 con 0x7f17c0069d70 2024-01-11T09:47:04.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.125+0000 7f17bf7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f17c0069d70 0x7f17c0162190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:04.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.126+0000 7f17beffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f17c0157440 0x7f17c00ae430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58858/0 (socket says 172.21.15.27:58858) 2024-01-11T09:47:04.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.126+0000 7f17bf7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f17c0069d70 0x7f17c0162190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42688/0 (socket says 172.21.15.27:42688) 2024-01-11T09:47:04.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.126+0000 7f17beffd700 1 -- 172.21.15.27:0/3143002747 learned_addr learned my addr 172.21.15.27:0/3143002747 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:04.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.126+0000 7f17be7fc700 1 --1- 172.21.15.27:0/3143002747 >> v1:172.21.15.77:6789/0 conn(0x7f17c0157810 0x7f17c0069660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:46526/0 (socket says 172.21.15.27:46526) 2024-01-11T09:47:04.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.126+0000 7f17bf7fe700 1 -- 172.21.15.27:0/3143002747 >> v1:172.21.15.77:6789/0 conn(0x7f17c0157810 legacy=0x7f17c0069660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:04.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.126+0000 7f17bf7fe700 1 -- 172.21.15.27:0/3143002747 >> v1:172.21.15.27:6789/0 conn(0x7f17c0157440 legacy=0x7f17c00ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:04.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.126+0000 7f17bdffb700 1 -- 172.21.15.27:0/3143002747 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151083730 0 0) 0x7f17c006d700 con 0x7f17c0157810 2024-01-11T09:47:04.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.126+0000 7f17bf7fe700 1 -- 172.21.15.27:0/3143002747 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c006dcc0 con 0x7f17c0069d70 2024-01-11T09:47:04.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.126+0000 7f17bf7fe700 1 --2- 172.21.15.27:0/3143002747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c0069d70 0x7f17c0162190 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f17b8009e60 tx=0x7f17b8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf39c1047485017 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:04.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.127+0000 7f17bdffb700 1 -- 172.21.15.27:0/3143002747 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b800e070 con 0x7f17c0069d70 2024-01-11T09:47:04.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.127+0000 7f17bdffb700 1 -- 172.21.15.27:0/3143002747 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b800bb30 con 0x7f17c0069d70 2024-01-11T09:47:04.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.127+0000 7f17bdffb700 1 -- 172.21.15.27:0/3143002747 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b8012430 con 0x7f17c0069d70 2024-01-11T09:47:04.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.127+0000 7f17c53ff700 1 -- 172.21.15.27:0/3143002747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c0069d70 msgr2=0x7f17c0162190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:04.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.127+0000 7f17c53ff700 1 --2- 172.21.15.27:0/3143002747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c0069d70 0x7f17c0162190 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f17b8009e60 tx=0x7f17b8009780 comp rx=0 tx=0).stop 2024-01-11T09:47:04.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.127+0000 7f17c53ff700 1 -- 172.21.15.27:0/3143002747 shutdown_connections 2024-01-11T09:47:04.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.127+0000 7f17c53ff700 1 --2- 172.21.15.27:0/3143002747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c0069d70 0x7f17c0162190 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:04.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.127+0000 7f17c53ff700 1 -- 172.21.15.27:0/3143002747 >> 172.21.15.27:0/3143002747 conn(0x7f17c00680e0 msgr2=0x7f17c00cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:04.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.127+0000 7f17c53ff700 1 -- 172.21.15.27:0/3143002747 shutdown_connections 2024-01-11T09:47:04.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.127+0000 7f17c53ff700 1 -- 172.21.15.27:0/3143002747 wait complete. 2024-01-11T09:47:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.129+0000 7f17c53ff700 1 Processor -- start 2024-01-11T09:47:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.129+0000 7f17c53ff700 1 -- start start 2024-01-11T09:47:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.130+0000 7f17c53ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17c0069d70 0x7f17c00c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.130+0000 7f17c53ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c0157440 0x7f17c014d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.130+0000 7f17c53ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17c0157810 0x7f17c00aa120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.130+0000 7f17c53ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c0157c40 con 0x7f17c0157440 2024-01-11T09:47:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.130+0000 7f17c53ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c00b0ea0 con 0x7f17c0157810 2024-01-11T09:47:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.130+0000 7f17c53ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c00b1180 con 0x7f17c0069d70 2024-01-11T09:47:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.130+0000 7f17be7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c0157440 0x7f17c014d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.130+0000 7f17be7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c0157440 0x7f17c014d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42690/0 (socket says 172.21.15.27:42690) 2024-01-11T09:47:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.130+0000 7f17be7fc700 1 -- 172.21.15.27:0/2758202688 learned_addr learned my addr 172.21.15.27:0/2758202688 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:04.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.130+0000 7f17beffd700 1 --2- 172.21.15.27:0/2758202688 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17c0069d70 0x7f17c00c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:04.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.130+0000 7f17bf7fe700 1 --2- 172.21.15.27:0/2758202688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17c0157810 0x7f17c00aa120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:04.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.130+0000 7f17be7fc700 1 -- 172.21.15.27:0/2758202688 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17c0069d70 msgr2=0x7f17c00c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:04.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.131+0000 7f17be7fc700 1 --2- 172.21.15.27:0/2758202688 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17c0069d70 0x7f17c00c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:04.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.131+0000 7f17be7fc700 1 -- 172.21.15.27:0/2758202688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17c0157810 msgr2=0x7f17c00aa120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:04.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.131+0000 7f17be7fc700 1 --2- 172.21.15.27:0/2758202688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17c0157810 0x7f17c00aa120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:04.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.131+0000 7f17be7fc700 1 -- 172.21.15.27:0/2758202688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b8009a20 con 0x7f17c0157440 2024-01-11T09:47:04.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.131+0000 7f17bf7fe700 1 --2- 172.21.15.27:0/2758202688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17c0157810 0x7f17c00aa120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:04.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.131+0000 7f17be7fc700 1 --2- 172.21.15.27:0/2758202688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c0157440 0x7f17c014d6e0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f17b400c530 tx=0x7f17b4007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:04.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.131+0000 7f17a6ffd700 1 -- 172.21.15.27:0/2758202688 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b4005a70 con 0x7f17c0157440 2024-01-11T09:47:04.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.131+0000 7f17a6ffd700 1 -- 172.21.15.27:0/2758202688 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b4005bd0 con 0x7f17c0157440 2024-01-11T09:47:04.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.131+0000 7f17a6ffd700 1 -- 172.21.15.27:0/2758202688 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b40070d0 con 0x7f17c0157440 2024-01-11T09:47:04.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.131+0000 7f17c53ff700 1 -- 172.21.15.27:0/2758202688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c00b1460 con 0x7f17c0157440 2024-01-11T09:47:04.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.131+0000 7f17c53ff700 1 -- 172.21.15.27:0/2758202688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c014dc20 con 0x7f17c0157440 2024-01-11T09:47:04.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.132+0000 7f17c53ff700 1 -- 172.21.15.27:0/2758202688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c0157df0 con 0x7f17c0157440 2024-01-11T09:47:04.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.133+0000 7f17a6ffd700 1 -- 172.21.15.27:0/2758202688 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 143) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f17b40074c0 con 0x7f17c0157440 2024-01-11T09:47:04.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.135+0000 7f17a6ffd700 1 --2- 172.21.15.27:0/2758202688 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f17ac066d80 0x7f17ac069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:04.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.135+0000 7f17a6ffd700 1 -- 172.21.15.27:0/2758202688 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 7541+0+0 (secure 0 0 0) 0x7f17b4083ef0 con 0x7f17c0157440 2024-01-11T09:47:04.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.135+0000 7f17beffd700 1 --2- 172.21.15.27:0/2758202688 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f17ac066d80 0x7f17ac069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:04.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.136+0000 7f17beffd700 1 --2- 172.21.15.27:0/2758202688 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f17ac066d80 0x7f17ac069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f17b0000ef0 tx=0x7f17b0009040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:04.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.138+0000 7f17a6ffd700 1 -- 172.21.15.27:0/2758202688 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f17c0157df0 con 0x7f17c0157440 2024-01-11T09:47:04.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:04.347+0000 7f17c53ff700 1 -- 172.21.15.27:0/2758202688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f17c0157df0 con 0x7f17c0157440 2024-01-11T09:47:05.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.285+0000 7f17a6ffd700 1 -- 172.21.15.27:0/2758202688 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 144) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f17b4050fe0 con 0x7f17c0157440 2024-01-11T09:47:05.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.287+0000 7f17a6ffd700 1 -- 172.21.15.27:0/2758202688 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v243) v1 ==== 181+0+0 (secure 0 0 0) 0x7f17b406f4e0 con 0x7f17c0157440 2024-01-11T09:47:05.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.293+0000 7f17c53ff700 1 -- 172.21.15.27:0/2758202688 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f17ac066d80 msgr2=0x7f17ac069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:05.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.293+0000 7f17c53ff700 1 --2- 172.21.15.27:0/2758202688 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f17ac066d80 0x7f17ac069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f17b0000ef0 tx=0x7f17b0009040 comp rx=0 tx=0).stop 2024-01-11T09:47:05.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.293+0000 7f17c53ff700 1 -- 172.21.15.27:0/2758202688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c0157440 msgr2=0x7f17c014d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:05.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.293+0000 7f17c53ff700 1 --2- 172.21.15.27:0/2758202688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c0157440 0x7f17c014d6e0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f17b400c530 tx=0x7f17b4007f30 comp rx=0 tx=0).stop 2024-01-11T09:47:05.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.294+0000 7f17c53ff700 1 -- 172.21.15.27:0/2758202688 shutdown_connections 2024-01-11T09:47:05.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.294+0000 7f17c53ff700 1 --2- 172.21.15.27:0/2758202688 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f17ac066d80 0x7f17ac069230 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:05.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.294+0000 7f17c53ff700 1 --2- 172.21.15.27:0/2758202688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17c0157810 0x7f17c00aa120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:05.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.294+0000 7f17c53ff700 1 --2- 172.21.15.27:0/2758202688 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f17c0069d70 0x7f17c00c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:05.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.294+0000 7f17c53ff700 1 --2- 172.21.15.27:0/2758202688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c0157440 0x7f17c014d6e0 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:05.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.294+0000 7f17c53ff700 1 -- 172.21.15.27:0/2758202688 >> 172.21.15.27:0/2758202688 conn(0x7f17c00680e0 msgr2=0x7f17c00cbf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:05.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.294+0000 7f17c53ff700 1 -- 172.21.15.27:0/2758202688 shutdown_connections 2024-01-11T09:47:05.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.294+0000 7f17c53ff700 1 -- 172.21.15.27:0/2758202688 wait complete. 2024-01-11T09:47:05.288 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-01-11T09:47:05.306 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-11T09:47:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.520+0000 7f2a6a7ff700 1 Processor -- start 2024-01-11T09:47:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.521+0000 7f2a6a7ff700 1 -- start start 2024-01-11T09:47:05.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.522+0000 7f2a6a7ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2a640c8f80 0x7f2a6415de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:05.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.522+0000 7f2a6a7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2a6406d080 0x7f2a6406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:05.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.522+0000 7f2a6a7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a6415da00 con 0x7f2a64159d50 2024-01-11T09:47:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.522+0000 7f2a6a7ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a6406da10 con 0x7f2a640c8f80 2024-01-11T09:47:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.522+0000 7f2a6a7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a640b09c0 con 0x7f2a6406d080 2024-01-11T09:47:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.522+0000 7f2a63fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2a640c8f80 0x7f2a6415de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.522+0000 7f2a68d9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2a64159d50 0x7f2a6415a1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58866/0 (socket says 172.21.15.27:58866) 2024-01-11T09:47:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.522+0000 7f2a68d9c700 1 -- 172.21.15.27:0/1158057971 learned_addr learned my addr 172.21.15.27:0/1158057971 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.522+0000 7f2a637fe700 1 --2- 172.21.15.27:0/1158057971 >> v2:172.21.15.77:3300/0 conn(0x7f2a6406d080 0x7f2a6406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.523+0000 7f2a63fff700 1 -- 172.21.15.27:0/1158057971 >> v2:172.21.15.77:3300/0 conn(0x7f2a6406d080 msgr2=0x7f2a6406d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.523+0000 7f2a63fff700 1 --2- 172.21.15.27:0/1158057971 >> v2:172.21.15.77:3300/0 conn(0x7f2a6406d080 0x7f2a6406d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.523+0000 7f2a63fff700 1 -- 172.21.15.27:0/1158057971 >> v1:172.21.15.27:6789/0 conn(0x7f2a64159d50 legacy=0x7f2a6415a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.523+0000 7f2a63fff700 1 -- 172.21.15.27:0/1158057971 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a6415d3f0 con 0x7f2a640c8f80 2024-01-11T09:47:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.523+0000 7f2a62ffd700 1 -- 172.21.15.27:0/1158057971 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670803793 0 0) 0x7f2a6415da00 con 0x7f2a64159d50 2024-01-11T09:47:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.523+0000 7f2a63fff700 1 --2- 172.21.15.27:0/1158057971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a640c8f80 0x7f2a6415de90 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f2a640ce310 tx=0x7f2a540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2185ce31293466b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.523+0000 7f2a62ffd700 1 -- 172.21.15.27:0/1158057971 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5400b6b0 con 0x7f2a640c8f80 2024-01-11T09:47:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.523+0000 7f2a62ffd700 1 -- 172.21.15.27:0/1158057971 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a5400b810 con 0x7f2a640c8f80 2024-01-11T09:47:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.523+0000 7f2a62ffd700 1 -- 172.21.15.27:0/1158057971 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5400bbb0 con 0x7f2a640c8f80 2024-01-11T09:47:05.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.524+0000 7f2a6a7ff700 1 -- 172.21.15.27:0/1158057971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a640c8f80 msgr2=0x7f2a6415de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:05.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.524+0000 7f2a6a7ff700 1 --2- 172.21.15.27:0/1158057971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a640c8f80 0x7f2a6415de90 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f2a640ce310 tx=0x7f2a540091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:05.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.524+0000 7f2a6a7ff700 1 -- 172.21.15.27:0/1158057971 shutdown_connections 2024-01-11T09:47:05.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.524+0000 7f2a6a7ff700 1 --2- 172.21.15.27:0/1158057971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a640c8f80 0x7f2a6415de90 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:05.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.524+0000 7f2a6a7ff700 1 --2- 172.21.15.27:0/1158057971 >> v2:172.21.15.77:3300/0 conn(0x7f2a6406d080 0x7f2a6406d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:05.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.524+0000 7f2a6a7ff700 1 -- 172.21.15.27:0/1158057971 >> 172.21.15.27:0/1158057971 conn(0x7f2a64063d00 msgr2=0x7f2a640ce3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:05.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.524+0000 7f2a6a7ff700 1 -- 172.21.15.27:0/1158057971 shutdown_connections 2024-01-11T09:47:05.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.524+0000 7f2a6a7ff700 1 -- 172.21.15.27:0/1158057971 wait complete. 2024-01-11T09:47:05.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.525+0000 7f2a6a7ff700 1 Processor -- start 2024-01-11T09:47:05.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.525+0000 7f2a6a7ff700 1 -- start start 2024-01-11T09:47:05.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.526+0000 7f2a6a7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a6406d080 0x7f2a640c2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:05.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.526+0000 7f2a6a7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a640c8f80 0x7f2a641527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:05.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.526+0000 7f2a6a7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a64159d50 0x7f2a640caa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:05.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.526+0000 7f2a6a7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a64157230 con 0x7f2a6406d080 2024-01-11T09:47:05.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.526+0000 7f2a6a7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a640c42c0 con 0x7f2a64159d50 2024-01-11T09:47:05.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.526+0000 7f2a6a7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a640c36b0 con 0x7f2a640c8f80 2024-01-11T09:47:05.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.527+0000 7f2a63fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a6406d080 0x7f2a640c2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:05.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.527+0000 7f2a63fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a6406d080 0x7f2a640c2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42694/0 (socket says 172.21.15.27:42694) 2024-01-11T09:47:05.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.527+0000 7f2a63fff700 1 -- 172.21.15.27:0/4254816703 learned_addr learned my addr 172.21.15.27:0/4254816703 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:05.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.527+0000 7f2a68d9c700 1 --2- 172.21.15.27:0/4254816703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a64159d50 0x7f2a640caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:05.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.527+0000 7f2a637fe700 1 --2- 172.21.15.27:0/4254816703 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a640c8f80 0x7f2a641527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:05.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.527+0000 7f2a63fff700 1 -- 172.21.15.27:0/4254816703 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a640c8f80 msgr2=0x7f2a641527a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:05.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.527+0000 7f2a63fff700 1 --2- 172.21.15.27:0/4254816703 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a640c8f80 0x7f2a641527a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:05.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.527+0000 7f2a63fff700 1 -- 172.21.15.27:0/4254816703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a64159d50 msgr2=0x7f2a640caa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:05.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.527+0000 7f2a63fff700 1 --2- 172.21.15.27:0/4254816703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a64159d50 0x7f2a640caa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:05.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.527+0000 7f2a63fff700 1 -- 172.21.15.27:0/4254816703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a54009420 con 0x7f2a6406d080 2024-01-11T09:47:05.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.527+0000 7f2a68d9c700 1 --2- 172.21.15.27:0/4254816703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a64159d50 0x7f2a640caa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:05.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.527+0000 7f2a63fff700 1 --2- 172.21.15.27:0/4254816703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a6406d080 0x7f2a640c2fb0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f2a6406f010 tx=0x7f2a54017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:05.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.528+0000 7f2a4bfff700 1 -- 172.21.15.27:0/4254816703 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5401f070 con 0x7f2a6406d080 2024-01-11T09:47:05.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.528+0000 7f2a4bfff700 1 -- 172.21.15.27:0/4254816703 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a54000c90 con 0x7f2a6406d080 2024-01-11T09:47:05.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.528+0000 7f2a4bfff700 1 -- 172.21.15.27:0/4254816703 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a54025420 con 0x7f2a6406d080 2024-01-11T09:47:05.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.528+0000 7f2a6a7ff700 1 -- 172.21.15.27:0/4254816703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a640c3990 con 0x7f2a6406d080 2024-01-11T09:47:05.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.528+0000 7f2a6a7ff700 1 -- 172.21.15.27:0/4254816703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a640c3f50 con 0x7f2a6406d080 2024-01-11T09:47:05.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.529+0000 7f2a6a7ff700 1 -- 172.21.15.27:0/4254816703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a6404c3c0 con 0x7f2a6406d080 2024-01-11T09:47:05.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.530+0000 7f2a4bfff700 1 -- 172.21.15.27:0/4254816703 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 144) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f2a54023070 con 0x7f2a6406d080 2024-01-11T09:47:05.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.531+0000 7f2a4bfff700 1 --2- 172.21.15.27:0/4254816703 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2a500670e0 0x7f2a50069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:05.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.531+0000 7f2a4bfff700 1 -- 172.21.15.27:0/4254816703 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 7103+0+0 (secure 0 0 0) 0x7f2a54086390 con 0x7f2a6406d080 2024-01-11T09:47:05.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.531+0000 7f2a637fe700 1 --2- 172.21.15.27:0/4254816703 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2a500670e0 0x7f2a50069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.532+0000 7f2a637fe700 1 --2- 172.21.15.27:0/4254816703 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2a500670e0 0x7f2a50069590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2a64156d10 tx=0x7f2a58008040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:05.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.539+0000 7f2a4bfff700 1 -- 172.21.15.27:0/4254816703 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2a54052e10 con 0x7f2a6406d080 2024-01-11T09:47:05.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:05.748+0000 7f2a6a7ff700 1 -- 172.21.15.27:0/4254816703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2a640c8b10 con 0x7f2a6406d080 2024-01-11T09:47:06.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.514+0000 7f2a4bfff700 1 -- 172.21.15.27:0/4254816703 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v244) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2a54056430 con 0x7f2a6406d080 2024-01-11T09:47:06.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.519+0000 7f2a6a7ff700 1 -- 172.21.15.27:0/4254816703 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2a500670e0 msgr2=0x7f2a50069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:06.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.519+0000 7f2a6a7ff700 1 --2- 172.21.15.27:0/4254816703 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2a500670e0 0x7f2a50069590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2a64156d10 tx=0x7f2a58008040 comp rx=0 tx=0).stop 2024-01-11T09:47:06.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.520+0000 7f2a6a7ff700 1 -- 172.21.15.27:0/4254816703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a6406d080 msgr2=0x7f2a640c2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:06.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.520+0000 7f2a6a7ff700 1 --2- 172.21.15.27:0/4254816703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a6406d080 0x7f2a640c2fb0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f2a6406f010 tx=0x7f2a54017000 comp rx=0 tx=0).stop 2024-01-11T09:47:06.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.520+0000 7f2a6a7ff700 1 -- 172.21.15.27:0/4254816703 shutdown_connections 2024-01-11T09:47:06.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.520+0000 7f2a6a7ff700 1 --2- 172.21.15.27:0/4254816703 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2a500670e0 0x7f2a50069590 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:06.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.520+0000 7f2a6a7ff700 1 --2- 172.21.15.27:0/4254816703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a64159d50 0x7f2a640caa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:06.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.520+0000 7f2a6a7ff700 1 --2- 172.21.15.27:0/4254816703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a6406d080 0x7f2a640c2fb0 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:06.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.520+0000 7f2a6a7ff700 1 --2- 172.21.15.27:0/4254816703 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2a640c8f80 0x7f2a641527a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:06.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.520+0000 7f2a6a7ff700 1 -- 172.21.15.27:0/4254816703 >> 172.21.15.27:0/4254816703 conn(0x7f2a64063d00 msgr2=0x7f2a64068680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:06.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.521+0000 7f2a6a7ff700 1 -- 172.21.15.27:0/4254816703 shutdown_connections 2024-01-11T09:47:06.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.521+0000 7f2a6a7ff700 1 -- 172.21.15.27:0/4254816703 wait complete. 2024-01-11T09:47:06.515 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-01-11T09:47:06.537 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:47:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.749+0000 7f8b36dff700 1 Processor -- start 2024-01-11T09:47:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.750+0000 7f8b36dff700 1 -- start start 2024-01-11T09:47:06.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.751+0000 7f8b36dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8b300b03e0 0x7f8b300b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:06.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.751+0000 7f8b36dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b300cd730 con 0x7f8b300b29c0 2024-01-11T09:47:06.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.751+0000 7f8b36dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b300b2de0 con 0x7f8b300b07b0 2024-01-11T09:47:06.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.751+0000 7f8b36dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b300ccb20 con 0x7f8b300b03e0 2024-01-11T09:47:06.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.751+0000 7f8b3539c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8b300b29c0 0x7f8b300aefc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58870/0 (socket says 172.21.15.27:58870) 2024-01-11T09:47:06.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.751+0000 7f8b3539c700 1 -- 172.21.15.27:0/308825868 learned_addr learned my addr 172.21.15.27:0/308825868 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.751+0000 7f8b34b9b700 1 --2- 172.21.15.27:0/308825868 >> v2:172.21.15.77:3300/0 conn(0x7f8b300b03e0 0x7f8b300b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.752+0000 7f8b2f7fe700 1 -- 172.21.15.27:0/308825868 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328422262 0 0) 0x7f8b300cd730 con 0x7f8b300b29c0 2024-01-11T09:47:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.752+0000 7f8b2f7fe700 1 -- 172.21.15.27:0/308825868 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b20003570 con 0x7f8b300b29c0 2024-01-11T09:47:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.752+0000 7f8b2f7fe700 1 -- 172.21.15.27:0/308825868 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299859294 0 0) 0x7f8b300b2de0 con 0x7f8b300b07b0 2024-01-11T09:47:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.752+0000 7f8b2f7fe700 1 -- 172.21.15.27:0/308825868 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b300cd730 con 0x7f8b300b07b0 2024-01-11T09:47:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.752+0000 7f8b34b9b700 1 -- 172.21.15.27:0/308825868 >> v1:172.21.15.77:6789/0 conn(0x7f8b300b07b0 legacy=0x7f8b30159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.752+0000 7f8b2f7fe700 1 -- 172.21.15.27:0/308825868 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4164125574 0 0) 0x7f8b20003570 con 0x7f8b300b29c0 2024-01-11T09:47:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.752+0000 7f8b34b9b700 1 -- 172.21.15.27:0/308825868 >> v1:172.21.15.27:6789/0 conn(0x7f8b300b29c0 legacy=0x7f8b300aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.752+0000 7f8b34b9b700 1 -- 172.21.15.27:0/308825868 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b300cce00 con 0x7f8b300b03e0 2024-01-11T09:47:06.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.752+0000 7f8b34b9b700 1 --2- 172.21.15.27:0/308825868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b300b03e0 0x7f8b300b2480 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f8b240098e0 tx=0x7f8b240091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7a142041967eab4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.753+0000 7f8b2f7fe700 1 -- 172.21.15.27:0/308825868 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b280017f0 con 0x7f8b300b03e0 2024-01-11T09:47:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.753+0000 7f8b2f7fe700 1 -- 172.21.15.27:0/308825868 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b24011420 con 0x7f8b300b03e0 2024-01-11T09:47:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.753+0000 7f8b2f7fe700 1 -- 172.21.15.27:0/308825868 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b240117c0 con 0x7f8b300b03e0 2024-01-11T09:47:06.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.753+0000 7f8b36dff700 1 -- 172.21.15.27:0/308825868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b300b03e0 msgr2=0x7f8b300b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.753+0000 7f8b36dff700 1 --2- 172.21.15.27:0/308825868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b300b03e0 0x7f8b300b2480 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f8b240098e0 tx=0x7f8b240091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.753+0000 7f8b36dff700 1 -- 172.21.15.27:0/308825868 shutdown_connections 2024-01-11T09:47:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.753+0000 7f8b36dff700 1 --2- 172.21.15.27:0/308825868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b300b03e0 0x7f8b300b2480 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.753+0000 7f8b36dff700 1 -- 172.21.15.27:0/308825868 >> 172.21.15.27:0/308825868 conn(0x7f8b3006ed40 msgr2=0x7f8b3006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.753+0000 7f8b36dff700 1 -- 172.21.15.27:0/308825868 shutdown_connections 2024-01-11T09:47:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.753+0000 7f8b36dff700 1 -- 172.21.15.27:0/308825868 wait complete. 2024-01-11T09:47:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.755+0000 7f8b36dff700 1 Processor -- start 2024-01-11T09:47:06.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.755+0000 7f8b36dff700 1 -- start start 2024-01-11T09:47:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.756+0000 7f8b36dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b300b03e0 0x7f8b3016c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.756+0000 7f8b36dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b300b07b0 0x7f8b300d0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.756+0000 7f8b36dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b300b29c0 0x7f8b300d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.756+0000 7f8b36dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b300a9200 con 0x7f8b300b29c0 2024-01-11T09:47:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.756+0000 7f8b36dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3016abb0 con 0x7f8b300b07b0 2024-01-11T09:47:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.756+0000 7f8b36dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3016a8d0 con 0x7f8b300b03e0 2024-01-11T09:47:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.756+0000 7f8b3539c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b300b29c0 0x7f8b300d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.756+0000 7f8b3539c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b300b29c0 0x7f8b300d08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42700/0 (socket says 172.21.15.27:42700) 2024-01-11T09:47:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.756+0000 7f8b3539c700 1 -- 172.21.15.27:0/2106875072 learned_addr learned my addr 172.21.15.27:0/2106875072 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.756+0000 7f8b34b9b700 1 --2- 172.21.15.27:0/2106875072 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b300b03e0 0x7f8b3016c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.756+0000 7f8b2ffff700 1 --2- 172.21.15.27:0/2106875072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b300b07b0 0x7f8b300d0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.757+0000 7f8b3539c700 1 -- 172.21.15.27:0/2106875072 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b300b03e0 msgr2=0x7f8b3016c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.757+0000 7f8b3539c700 1 --2- 172.21.15.27:0/2106875072 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b300b03e0 0x7f8b3016c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:06.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.757+0000 7f8b3539c700 1 -- 172.21.15.27:0/2106875072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b300b07b0 msgr2=0x7f8b300d0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:06.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.757+0000 7f8b3539c700 1 --2- 172.21.15.27:0/2106875072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b300b07b0 0x7f8b300d0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:06.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.757+0000 7f8b3539c700 1 -- 172.21.15.27:0/2106875072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b24009420 con 0x7f8b300b29c0 2024-01-11T09:47:06.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.757+0000 7f8b3539c700 1 --2- 172.21.15.27:0/2106875072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b300b29c0 0x7f8b300d08a0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800cc10 tx=0x7f8b28007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:06.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.757+0000 7f8b2cff9700 1 -- 172.21.15.27:0/2106875072 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b28004dc0 con 0x7f8b300b29c0 2024-01-11T09:47:06.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.757+0000 7f8b2cff9700 1 -- 172.21.15.27:0/2106875072 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2800a5c0 con 0x7f8b300b29c0 2024-01-11T09:47:06.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.757+0000 7f8b2cff9700 1 -- 172.21.15.27:0/2106875072 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2800a980 con 0x7f8b300b29c0 2024-01-11T09:47:06.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.757+0000 7f8b36dff700 1 -- 172.21.15.27:0/2106875072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b300a98e0 con 0x7f8b300b29c0 2024-01-11T09:47:06.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.757+0000 7f8b36dff700 1 -- 172.21.15.27:0/2106875072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b300a9ea0 con 0x7f8b300b29c0 2024-01-11T09:47:06.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.758+0000 7f8b36dff700 1 -- 172.21.15.27:0/2106875072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b300718d0 con 0x7f8b300b29c0 2024-01-11T09:47:06.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.759+0000 7f8b2cff9700 1 -- 172.21.15.27:0/2106875072 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 144) v1 ==== 84890+0+0 (secure 0 0 0) 0x7f8b2800ad30 con 0x7f8b300b29c0 2024-01-11T09:47:06.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.760+0000 7f8b2cff9700 1 --2- 172.21.15.27:0/2106875072 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8b140670e0 0x7f8b14069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:06.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.760+0000 7f8b2cff9700 1 -- 172.21.15.27:0/2106875072 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f8b28084210 con 0x7f8b300b29c0 2024-01-11T09:47:06.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.761+0000 7f8b34b9b700 1 --2- 172.21.15.27:0/2106875072 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8b140670e0 0x7f8b14069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:06.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.761+0000 7f8b34b9b700 1 --2- 172.21.15.27:0/2106875072 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8b140670e0 0x7f8b14069590 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8b240093f0 tx=0x7f8b24016000 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:06.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.765+0000 7f8b2cff9700 1 -- 172.21.15.27:0/2106875072 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8b28050e00 con 0x7f8b300b29c0 2024-01-11T09:47:06.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.972+0000 7f8b36dff700 1 -- 172.21.15.27:0/2106875072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b300cd0e0 con 0x7f8b300b29c0 2024-01-11T09:47:06.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.973+0000 7f8b2cff9700 1 -- 172.21.15.27:0/2106875072 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v244) v1 ==== 74+0+16739 (secure 0 0 0) 0x7f8b28054420 con 0x7f8b300b29c0 2024-01-11T09:47:06.965 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:06.965 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":244,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:47:06.507632+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-01-11T09:46:37.665603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"238","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-01-11T09:46:38.472134+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"239","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/321681407":"2024-01-12T09:47:02.847264+0000","172.21.15.77:0/463475654":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/1391607686":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/2619336301":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/1120758885":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/1857056349":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/210673663":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/3835722213":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/696729061":"2024-01-12T09:46:54.199949+0000","172.21.15.27:6833/321681407":"2024-01-12T09:47:02.847264+0000","172.21.15.77:0/206349247":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/228357086":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/3326555344":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/3398849669":"2024-01-12T09:46:54.199949+0000","172.21.15.77:6837/59776":"2024-01-12T09:46:54.199949+0000","172.21.15.77:6836/59776":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/1934793537":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/842931202":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/1366958778":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/198451994":"2024-01-12T09:46:54.199949+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:47:06.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.975+0000 7f8b36dff700 1 -- 172.21.15.27:0/2106875072 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8b140670e0 msgr2=0x7f8b14069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:06.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.975+0000 7f8b36dff700 1 --2- 172.21.15.27:0/2106875072 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8b140670e0 0x7f8b14069590 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8b240093f0 tx=0x7f8b24016000 comp rx=0 tx=0).stop 2024-01-11T09:47:06.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.976+0000 7f8b36dff700 1 -- 172.21.15.27:0/2106875072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b300b29c0 msgr2=0x7f8b300d08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:06.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.976+0000 7f8b36dff700 1 --2- 172.21.15.27:0/2106875072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b300b29c0 0x7f8b300d08a0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800cc10 tx=0x7f8b28007d20 comp rx=0 tx=0).stop 2024-01-11T09:47:06.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.976+0000 7f8b36dff700 1 -- 172.21.15.27:0/2106875072 shutdown_connections 2024-01-11T09:47:06.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.976+0000 7f8b36dff700 1 --2- 172.21.15.27:0/2106875072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b300b29c0 0x7f8b300d08a0 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:06.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.976+0000 7f8b36dff700 1 --2- 172.21.15.27:0/2106875072 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8b300b03e0 0x7f8b3016c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:06.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.976+0000 7f8b36dff700 1 --2- 172.21.15.27:0/2106875072 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8b140670e0 0x7f8b14069590 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:06.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.976+0000 7f8b36dff700 1 --2- 172.21.15.27:0/2106875072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b300b07b0 0x7f8b300d0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:06.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.976+0000 7f8b36dff700 1 -- 172.21.15.27:0/2106875072 >> 172.21.15.27:0/2106875072 conn(0x7f8b3006ed40 msgr2=0x7f8b300c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:06.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.976+0000 7f8b36dff700 1 -- 172.21.15.27:0/2106875072 shutdown_connections 2024-01-11T09:47:06.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:06.976+0000 7f8b36dff700 1 -- 172.21.15.27:0/2106875072 wait complete. 2024-01-11T09:47:06.988 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-01-11T09:46:37.665603+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '238', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2024-01-11T09:46:38.472134+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '239', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}}] 2024-01-11T09:47:06.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T09:47:07.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.201+0000 7fc9a73ff700 1 Processor -- start 2024-01-11T09:47:07.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.202+0000 7fc9a73ff700 1 -- start start 2024-01-11T09:47:07.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.203+0000 7fc9a73ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc9a0157410 0x7fc9a00ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:07.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.203+0000 7fc9a73ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9a0069540 0x7fc9a00c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:07.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.203+0000 7fc9a73ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a00699d0 con 0x7fc9a01577e0 2024-01-11T09:47:07.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.203+0000 7fc9a73ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a0069cb0 con 0x7fc9a0069540 2024-01-11T09:47:07.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.203+0000 7fc9a73ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a0069f90 con 0x7fc9a0157410 2024-01-11T09:47:07.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.203+0000 7fc9a599c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9a0069540 0x7fc9a00c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:07.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.204+0000 7fc9a599c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9a0069540 0x7fc9a00c6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42704/0 (socket says 172.21.15.27:42704) 2024-01-11T09:47:07.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.204+0000 7fc9a599c700 1 -- 172.21.15.27:0/3810011494 learned_addr learned my addr 172.21.15.27:0/3810011494 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:07.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.204+0000 7fc9a519b700 1 --2- 172.21.15.27:0/3810011494 >> v2:172.21.15.77:3300/0 conn(0x7fc9a0157410 0x7fc9a00ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.204+0000 7fc9a599c700 1 -- 172.21.15.27:0/3810011494 >> v2:172.21.15.77:3300/0 conn(0x7fc9a0157410 msgr2=0x7fc9a00ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.204+0000 7fc9a599c700 1 --2- 172.21.15.27:0/3810011494 >> v2:172.21.15.77:3300/0 conn(0x7fc9a0157410 0x7fc9a00ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.204+0000 7fc9a599c700 1 -- 172.21.15.27:0/3810011494 >> v1:172.21.15.77:6789/0 conn(0x7fc9a01577e0 legacy=0x7fc9a00ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.204+0000 7fc9a599c700 1 -- 172.21.15.27:0/3810011494 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a006a270 con 0x7fc9a0069540 2024-01-11T09:47:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.204+0000 7fc9a599c700 1 --2- 172.21.15.27:0/3810011494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a0069540 0x7fc9a00c6ba0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0070c90 tx=0x7fc998009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cb8f97e95248b3d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.205+0000 7fc997fff700 1 -- 172.21.15.27:0/3810011494 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99800e070 con 0x7fc9a0069540 2024-01-11T09:47:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.205+0000 7fc997fff700 1 -- 172.21.15.27:0/3810011494 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99800ba20 con 0x7fc9a0069540 2024-01-11T09:47:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.205+0000 7fc997fff700 1 -- 172.21.15.27:0/3810011494 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9980092e0 con 0x7fc9a0069540 2024-01-11T09:47:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.205+0000 7fc9a73ff700 1 -- 172.21.15.27:0/3810011494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a0069540 msgr2=0x7fc9a00c6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.205+0000 7fc9a73ff700 1 --2- 172.21.15.27:0/3810011494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a0069540 0x7fc9a00c6ba0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0070c90 tx=0x7fc998009840 comp rx=0 tx=0).stop 2024-01-11T09:47:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.205+0000 7fc9a73ff700 1 -- 172.21.15.27:0/3810011494 shutdown_connections 2024-01-11T09:47:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.205+0000 7fc9a73ff700 1 --2- 172.21.15.27:0/3810011494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a0069540 0x7fc9a00c6ba0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.205+0000 7fc9a73ff700 1 --2- 172.21.15.27:0/3810011494 >> v2:172.21.15.77:3300/0 conn(0x7fc9a0157410 0x7fc9a00ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.205+0000 7fc9a73ff700 1 -- 172.21.15.27:0/3810011494 >> 172.21.15.27:0/3810011494 conn(0x7fc9a00680a0 msgr2=0x7fc9a00cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.205+0000 7fc9a73ff700 1 -- 172.21.15.27:0/3810011494 shutdown_connections 2024-01-11T09:47:07.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.205+0000 7fc9a73ff700 1 -- 172.21.15.27:0/3810011494 wait complete. 2024-01-11T09:47:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.207+0000 7fc9a73ff700 1 Processor -- start 2024-01-11T09:47:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.207+0000 7fc9a73ff700 1 -- start start 2024-01-11T09:47:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.208+0000 7fc9a73ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9a0069540 0x7fc9a0156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.208+0000 7fc9a73ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a0157410 0x7fc9a0146c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.208+0000 7fc9a73ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9a01577e0 0x7fc9a015bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.208+0000 7fc9a73ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a01538c0 con 0x7fc9a0157410 2024-01-11T09:47:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.208+0000 7fc9a73ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a0156c10 con 0x7fc9a01577e0 2024-01-11T09:47:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.208+0000 7fc9a73ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a0156950 con 0x7fc9a0069540 2024-01-11T09:47:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.208+0000 7fc9a499a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a0157410 0x7fc9a0146c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.208+0000 7fc9a499a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a0157410 0x7fc9a0146c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42710/0 (socket says 172.21.15.27:42710) 2024-01-11T09:47:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.208+0000 7fc9a499a700 1 -- 172.21.15.27:0/4090642063 learned_addr learned my addr 172.21.15.27:0/4090642063 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.208+0000 7fc9a519b700 1 --2- 172.21.15.27:0/4090642063 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9a0069540 0x7fc9a0156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.208+0000 7fc9a599c700 1 --2- 172.21.15.27:0/4090642063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9a01577e0 0x7fc9a015bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.209+0000 7fc9a499a700 1 -- 172.21.15.27:0/4090642063 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9a0069540 msgr2=0x7fc9a0156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.209+0000 7fc9a499a700 1 --2- 172.21.15.27:0/4090642063 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9a0069540 0x7fc9a0156410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.209+0000 7fc9a499a700 1 -- 172.21.15.27:0/4090642063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9a01577e0 msgr2=0x7fc9a015bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.209+0000 7fc9a499a700 1 --2- 172.21.15.27:0/4090642063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9a01577e0 0x7fc9a015bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.209+0000 7fc9a499a700 1 -- 172.21.15.27:0/4090642063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc998009ae0 con 0x7fc9a0157410 2024-01-11T09:47:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.209+0000 7fc9a599c700 1 --2- 172.21.15.27:0/4090642063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9a01577e0 0x7fc9a015bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.209+0000 7fc9a499a700 1 --2- 172.21.15.27:0/4090642063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a0157410 0x7fc9a0146c20 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fc99000c7c0 tx=0x7fc990007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.209+0000 7fc9957fa700 1 -- 172.21.15.27:0/4090642063 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc990011590 con 0x7fc9a0157410 2024-01-11T09:47:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.209+0000 7fc9957fa700 1 -- 172.21.15.27:0/4090642063 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9900116f0 con 0x7fc9a0157410 2024-01-11T09:47:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.209+0000 7fc9a73ff700 1 -- 172.21.15.27:0/4090642063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a01699c0 con 0x7fc9a0157410 2024-01-11T09:47:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.209+0000 7fc9957fa700 1 -- 172.21.15.27:0/4090642063 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc990011ab0 con 0x7fc9a0157410 2024-01-11T09:47:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.209+0000 7fc9a73ff700 1 -- 172.21.15.27:0/4090642063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a016a000 con 0x7fc9a0157410 2024-01-11T09:47:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.210+0000 7fc9a73ff700 1 -- 172.21.15.27:0/4090642063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a004c3c0 con 0x7fc9a0157410 2024-01-11T09:47:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.211+0000 7fc9957fa700 1 -- 172.21.15.27:0/4090642063 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 144) v1 ==== 84890+0+0 (secure 0 0 0) 0x7fc99001b3b0 con 0x7fc9a0157410 2024-01-11T09:47:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.212+0000 7fc9957fa700 1 --2- 172.21.15.27:0/4090642063 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc9840670e0 0x7fc984069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.212+0000 7fc9957fa700 1 -- 172.21.15.27:0/4090642063 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6689+0+0 (secure 0 0 0) 0x7fc990084250 con 0x7fc9a0157410 2024-01-11T09:47:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.213+0000 7fc9a519b700 1 --2- 172.21.15.27:0/4090642063 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc9840670e0 0x7fc984069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:07.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.214+0000 7fc9a519b700 1 --2- 172.21.15.27:0/4090642063 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc9840670e0 0x7fc984069590 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc99c009d30 tx=0x7fc99c008040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.216+0000 7fc9957fa700 1 -- 172.21.15.27:0/4090642063 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc990050dc0 con 0x7fc9a0157410 2024-01-11T09:47:07.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.286+0000 7fc9957fa700 1 -- 172.21.15.27:0/4090642063 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fc990051190 con 0x7fc9a0157410 2024-01-11T09:47:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.427+0000 7fc9a73ff700 1 -- 172.21.15.27:0/4090642063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc9a00c9a90 con 0x7fc9a0157410 2024-01-11T09:47:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.428+0000 7fc9957fa700 1 -- 172.21.15.27:0/4090642063 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v244) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc990054850 con 0x7fc9a0157410 2024-01-11T09:47:07.419 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T09:47:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.430+0000 7fc9a73ff700 1 -- 172.21.15.27:0/4090642063 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc9840670e0 msgr2=0x7fc984069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.430+0000 7fc9a73ff700 1 --2- 172.21.15.27:0/4090642063 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc9840670e0 0x7fc984069590 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc99c009d30 tx=0x7fc99c008040 comp rx=0 tx=0).stop 2024-01-11T09:47:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.430+0000 7fc9a73ff700 1 -- 172.21.15.27:0/4090642063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a0157410 msgr2=0x7fc9a0146c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.430+0000 7fc9a73ff700 1 --2- 172.21.15.27:0/4090642063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a0157410 0x7fc9a0146c20 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fc99000c7c0 tx=0x7fc990007f50 comp rx=0 tx=0).stop 2024-01-11T09:47:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.430+0000 7fc9a73ff700 1 -- 172.21.15.27:0/4090642063 shutdown_connections 2024-01-11T09:47:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.430+0000 7fc9a73ff700 1 --2- 172.21.15.27:0/4090642063 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc9840670e0 0x7fc984069590 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.430+0000 7fc9a73ff700 1 --2- 172.21.15.27:0/4090642063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9a01577e0 0x7fc9a015bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.430+0000 7fc9a73ff700 1 --2- 172.21.15.27:0/4090642063 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc9a0069540 0x7fc9a0156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.430+0000 7fc9a73ff700 1 --2- 172.21.15.27:0/4090642063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a0157410 0x7fc9a0146c20 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.430+0000 7fc9a73ff700 1 -- 172.21.15.27:0/4090642063 >> 172.21.15.27:0/4090642063 conn(0x7fc9a00680a0 msgr2=0x7fc9a00af320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.430+0000 7fc9a73ff700 1 -- 172.21.15.27:0/4090642063 shutdown_connections 2024-01-11T09:47:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.430+0000 7fc9a73ff700 1 -- 172.21.15.27:0/4090642063 wait complete. 2024-01-11T09:47:07.439 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-01-11T09:47:07.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.652+0000 7feb327ff700 1 Processor -- start 2024-01-11T09:47:07.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.652+0000 7feb327ff700 1 -- start start 2024-01-11T09:47:07.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.654+0000 7feb327ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c157410 0x7feb2c0ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:07.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.654+0000 7feb327ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feb2c1577e0 0x7feb2c0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:07.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.654+0000 7feb327ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2c0699d0 con 0x7feb2c069540 2024-01-11T09:47:07.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.654+0000 7feb327ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2c069cb0 con 0x7feb2c1577e0 2024-01-11T09:47:07.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.654+0000 7feb327ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c069f90 con 0x7feb2c157410 2024-01-11T09:47:07.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.654+0000 7feb30d9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7feb2c069540 0x7feb2c15b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58878/0 (socket says 172.21.15.27:58878) 2024-01-11T09:47:07.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.654+0000 7feb30d9c700 1 -- 172.21.15.27:0/3752950346 learned_addr learned my addr 172.21.15.27:0/3752950346 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:07.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.654+0000 7feb2b7fe700 1 --2- 172.21.15.27:0/3752950346 >> v2:172.21.15.77:3300/0 conn(0x7feb2c1577e0 0x7feb2c0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:07.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.654+0000 7feb2bfff700 1 --2- 172.21.15.27:0/3752950346 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c157410 0x7feb2c0ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:07.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.654+0000 7feb2affd700 1 -- 172.21.15.27:0/3752950346 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803083995 0 0) 0x7feb2c0699d0 con 0x7feb2c069540 2024-01-11T09:47:07.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.654+0000 7feb2affd700 1 -- 172.21.15.27:0/3752950346 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb14003570 con 0x7feb2c069540 2024-01-11T09:47:07.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.655+0000 7feb2affd700 1 -- 172.21.15.27:0/3752950346 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1819306024 0 0) 0x7feb14003570 con 0x7feb2c069540 2024-01-11T09:47:07.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.655+0000 7feb2affd700 1 -- 172.21.15.27:0/3752950346 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c157410 msgr2=0x7feb2c0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:07.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.655+0000 7feb2affd700 1 --2- 172.21.15.27:0/3752950346 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c157410 0x7feb2c0ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:07.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.655+0000 7feb2affd700 1 -- 172.21.15.27:0/3752950346 >> v2:172.21.15.77:3300/0 conn(0x7feb2c1577e0 msgr2=0x7feb2c0ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:07.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.655+0000 7feb2affd700 1 --2- 172.21.15.27:0/3752950346 >> v2:172.21.15.77:3300/0 conn(0x7feb2c1577e0 0x7feb2c0ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.655+0000 7feb2affd700 1 -- 172.21.15.27:0/3752950346 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c06a270 con 0x7feb2c069540 2024-01-11T09:47:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.655+0000 7feb2affd700 1 -- 172.21.15.27:0/3752950346 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7feb240017f0 con 0x7feb2c069540 2024-01-11T09:47:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.655+0000 7feb2affd700 1 -- 172.21.15.27:0/3752950346 >> v1:172.21.15.27:6789/0 conn(0x7feb2c069540 legacy=0x7feb2c15b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.655+0000 7feb2affd700 1 --2- 172.21.15.27:0/3752950346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb14003ae0 0x7feb14003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.655+0000 7feb2affd700 1 --2- 172.21.15.27:0/3752950346 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c157410 0x7feb14004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.655+0000 7feb2affd700 1 --2- 172.21.15.27:0/3752950346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb14004d50 0x7feb140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.655+0000 7feb2affd700 1 -- 172.21.15.27:0/3752950346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c069cb0 con 0x7feb14004d50 2024-01-11T09:47:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.655+0000 7feb2affd700 1 -- 172.21.15.27:0/3752950346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c069f90 con 0x7feb14003ae0 2024-01-11T09:47:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.655+0000 7feb2affd700 1 -- 172.21.15.27:0/3752950346 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb1400bb10 con 0x7feb2c157410 2024-01-11T09:47:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.655+0000 7feb30d9c700 1 --2- 172.21.15.27:0/3752950346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb14004d50 0x7feb140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.655+0000 7feb2b7fe700 1 --2- 172.21.15.27:0/3752950346 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c157410 0x7feb14004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.655+0000 7feb2bfff700 1 --2- 172.21.15.27:0/3752950346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb14003ae0 0x7feb14003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.656+0000 7feb30d9c700 1 -- 172.21.15.27:0/3752950346 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c157410 msgr2=0x7feb14004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.656+0000 7feb30d9c700 1 --2- 172.21.15.27:0/3752950346 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c157410 0x7feb14004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:07.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.656+0000 7feb30d9c700 1 -- 172.21.15.27:0/3752950346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb14003ae0 msgr2=0x7feb14003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.656+0000 7feb30d9c700 1 --2- 172.21.15.27:0/3752950346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb14003ae0 0x7feb14003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.656+0000 7feb30d9c700 1 -- 172.21.15.27:0/3752950346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb14003570 con 0x7feb14004d50 2024-01-11T09:47:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.656+0000 7feb2b7fe700 1 --2- 172.21.15.27:0/3752950346 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c157410 0x7feb14004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.656+0000 7feb30d9c700 1 --2- 172.21.15.27:0/3752950346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb14004d50 0x7feb140051c0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7feb2400a890 tx=0x7feb2c06a550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4720db2e5fe098f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.656+0000 7feb2affd700 1 -- 172.21.15.27:0/3752950346 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb240070f0 con 0x7feb14004d50 2024-01-11T09:47:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.656+0000 7feb2affd700 1 -- 172.21.15.27:0/3752950346 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb24007250 con 0x7feb14004d50 2024-01-11T09:47:07.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.656+0000 7feb2affd700 1 -- 172.21.15.27:0/3752950346 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24007610 con 0x7feb14004d50 2024-01-11T09:47:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.657+0000 7feb327ff700 1 -- 172.21.15.27:0/3752950346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb14004d50 msgr2=0x7feb140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.657+0000 7feb327ff700 1 --2- 172.21.15.27:0/3752950346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb14004d50 0x7feb140051c0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7feb2400a890 tx=0x7feb2c06a550 comp rx=0 tx=0).stop 2024-01-11T09:47:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.657+0000 7feb30d9c700 1 -- 172.21.15.27:0/3752950346 reap_dead start 2024-01-11T09:47:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.657+0000 7feb327ff700 1 -- 172.21.15.27:0/3752950346 shutdown_connections 2024-01-11T09:47:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.657+0000 7feb327ff700 1 -- 172.21.15.27:0/3752950346 >> 172.21.15.27:0/3752950346 conn(0x7feb2c0680a0 msgr2=0x7feb2c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.657+0000 7feb327ff700 1 -- 172.21.15.27:0/3752950346 shutdown_connections 2024-01-11T09:47:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.657+0000 7feb327ff700 1 -- 172.21.15.27:0/3752950346 wait complete. 2024-01-11T09:47:07.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.659+0000 7feb327ff700 1 Processor -- start 2024-01-11T09:47:07.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.659+0000 7feb327ff700 1 -- start start 2024-01-11T09:47:07.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.660+0000 7feb327ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb14004d50 0x7feb2c150b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.660+0000 7feb327ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb2c1510c0 0x7feb2c157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.660+0000 7feb327ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c1580f0 0x7feb2c0c5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.660+0000 7feb327ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c159a20 con 0x7feb14004d50 2024-01-11T09:47:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.660+0000 7feb327ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c155180 con 0x7feb2c1510c0 2024-01-11T09:47:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.660+0000 7feb327ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c158590 con 0x7feb2c1580f0 2024-01-11T09:47:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.661+0000 7feb2bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb14004d50 0x7feb2c150b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.661+0000 7feb2bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb14004d50 0x7feb2c150b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42728/0 (socket says 172.21.15.27:42728) 2024-01-11T09:47:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.661+0000 7feb2bfff700 1 -- 172.21.15.27:0/2848527226 learned_addr learned my addr 172.21.15.27:0/2848527226 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.661+0000 7feb30d9c700 1 --2- 172.21.15.27:0/2848527226 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c1580f0 0x7feb2c0c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.661+0000 7feb2b7fe700 1 --2- 172.21.15.27:0/2848527226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb2c1510c0 0x7feb2c157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.661+0000 7feb2bfff700 1 -- 172.21.15.27:0/2848527226 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c1580f0 msgr2=0x7feb2c0c5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.661+0000 7feb2bfff700 1 --2- 172.21.15.27:0/2848527226 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c1580f0 0x7feb2c0c5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.661+0000 7feb2bfff700 1 -- 172.21.15.27:0/2848527226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb2c1510c0 msgr2=0x7feb2c157bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:07.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.661+0000 7feb2bfff700 1 --2- 172.21.15.27:0/2848527226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb2c1510c0 0x7feb2c157bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.661+0000 7feb2bfff700 1 -- 172.21.15.27:0/2848527226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c154570 con 0x7feb14004d50 2024-01-11T09:47:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.661+0000 7feb2bfff700 1 --2- 172.21.15.27:0/2848527226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb14004d50 0x7feb2c150b80 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7feb1c011b10 tx=0x7feb1c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.662+0000 7feb13fff700 1 -- 172.21.15.27:0/2848527226 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c005c10 con 0x7feb14004d50 2024-01-11T09:47:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.662+0000 7feb13fff700 1 -- 172.21.15.27:0/2848527226 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1c005d70 con 0x7feb14004d50 2024-01-11T09:47:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.662+0000 7feb327ff700 1 -- 172.21.15.27:0/2848527226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2c154850 con 0x7feb14004d50 2024-01-11T09:47:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.662+0000 7feb13fff700 1 -- 172.21.15.27:0/2848527226 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c00fa10 con 0x7feb14004d50 2024-01-11T09:47:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.662+0000 7feb327ff700 1 -- 172.21.15.27:0/2848527226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2c154e10 con 0x7feb14004d50 2024-01-11T09:47:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.663+0000 7feb327ff700 1 -- 172.21.15.27:0/2848527226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2c04c3c0 con 0x7feb14004d50 2024-01-11T09:47:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.664+0000 7feb13fff700 1 -- 172.21.15.27:0/2848527226 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7feb1c006050 con 0x7feb14004d50 2024-01-11T09:47:07.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.665+0000 7feb13fff700 1 --2- 172.21.15.27:0/2848527226 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7feb18067170 0x7feb18069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:07.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.665+0000 7feb13fff700 1 -- 172.21.15.27:0/2848527226 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6689+0+0 (secure 0 0 0) 0x7feb1c0856c0 con 0x7feb14004d50 2024-01-11T09:47:07.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.665+0000 7feb2b7fe700 1 --2- 172.21.15.27:0/2848527226 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7feb18067170 0x7feb18069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:07.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.666+0000 7feb2b7fe700 1 --2- 172.21.15.27:0/2848527226 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7feb18067170 0x7feb18069620 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7feb20009170 tx=0x7feb20008040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:07.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.669+0000 7feb13fff700 1 -- 172.21.15.27:0/2848527226 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feb1c0522c0 con 0x7feb14004d50 2024-01-11T09:47:07.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.870+0000 7feb327ff700 1 -- 172.21.15.27:0/2848527226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feb2c0cc5a0 con 0x7feb14004d50 2024-01-11T09:47:07.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.870+0000 7feb13fff700 1 -- 172.21.15.27:0/2848527226 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v244) v1 ==== 107+0+11 (secure 0 0 0) 0x7feb1c0558e0 con 0x7feb14004d50 2024-01-11T09:47:07.862 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T09:47:07.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.872+0000 7feb327ff700 1 -- 172.21.15.27:0/2848527226 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7feb18067170 msgr2=0x7feb18069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:07.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.872+0000 7feb327ff700 1 --2- 172.21.15.27:0/2848527226 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7feb18067170 0x7feb18069620 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7feb20009170 tx=0x7feb20008040 comp rx=0 tx=0).stop 2024-01-11T09:47:07.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.872+0000 7feb327ff700 1 -- 172.21.15.27:0/2848527226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb14004d50 msgr2=0x7feb2c150b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:07.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.872+0000 7feb327ff700 1 --2- 172.21.15.27:0/2848527226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb14004d50 0x7feb2c150b80 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7feb1c011b10 tx=0x7feb1c009120 comp rx=0 tx=0).stop 2024-01-11T09:47:07.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.872+0000 7feb327ff700 1 -- 172.21.15.27:0/2848527226 shutdown_connections 2024-01-11T09:47:07.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.872+0000 7feb327ff700 1 --2- 172.21.15.27:0/2848527226 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb2c1580f0 0x7feb2c0c5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:07.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.872+0000 7feb327ff700 1 --2- 172.21.15.27:0/2848527226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb14004d50 0x7feb2c150b80 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:07.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.872+0000 7feb327ff700 1 --2- 172.21.15.27:0/2848527226 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7feb18067170 0x7feb18069620 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:07.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.872+0000 7feb327ff700 1 --2- 172.21.15.27:0/2848527226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb2c1510c0 0x7feb2c157bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:07.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.872+0000 7feb327ff700 1 -- 172.21.15.27:0/2848527226 >> 172.21.15.27:0/2848527226 conn(0x7feb2c0680a0 msgr2=0x7feb2c152b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:07.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.872+0000 7feb327ff700 1 -- 172.21.15.27:0/2848527226 shutdown_connections 2024-01-11T09:47:07.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:07.873+0000 7feb327ff700 1 -- 172.21.15.27:0/2848527226 wait complete. 2024-01-11T09:47:07.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-01-11T09:47:08.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.093+0000 7f8d6d1ff700 1 Processor -- start 2024-01-11T09:47:08.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.094+0000 7f8d6d1ff700 1 -- start start 2024-01-11T09:47:08.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.095+0000 7f8d6d1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8d6813fbb0 0x7f8d680ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:08.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.095+0000 7f8d6d1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d680c6b30 con 0x7f8d6813ff80 2024-01-11T09:47:08.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.095+0000 7f8d6d1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d680c6800 con 0x7f8d6815a730 2024-01-11T09:47:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.095+0000 7f8d6d1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d680cb240 con 0x7f8d6813fbb0 2024-01-11T09:47:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.095+0000 7f8d6659c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8d6813ff80 0x7f8d6815a020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58886/0 (socket says 172.21.15.27:58886) 2024-01-11T09:47:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.095+0000 7f8d6659c700 1 -- 172.21.15.27:0/1119307256 learned_addr learned my addr 172.21.15.27:0/1119307256 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.096+0000 7f8d66d9d700 1 --2- 172.21.15.27:0/1119307256 >> v2:172.21.15.77:3300/0 conn(0x7f8d6813fbb0 0x7f8d680ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.096+0000 7f8d65d9b700 1 -- 172.21.15.27:0/1119307256 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24509962 0 0) 0x7f8d680c6b30 con 0x7f8d6813ff80 2024-01-11T09:47:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.096+0000 7f8d65d9b700 1 -- 172.21.15.27:0/1119307256 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d50003570 con 0x7f8d6813ff80 2024-01-11T09:47:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.096+0000 7f8d65d9b700 1 -- 172.21.15.27:0/1119307256 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 17555403 0 0) 0x7f8d50003570 con 0x7f8d6813ff80 2024-01-11T09:47:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.096+0000 7f8d65d9b700 1 -- 172.21.15.27:0/1119307256 >> v2:172.21.15.77:3300/0 conn(0x7f8d6813fbb0 msgr2=0x7f8d680ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.096+0000 7f8d65d9b700 1 --2- 172.21.15.27:0/1119307256 >> v2:172.21.15.77:3300/0 conn(0x7f8d6813fbb0 0x7f8d680ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.096+0000 7f8d65d9b700 1 -- 172.21.15.27:0/1119307256 >> v1:172.21.15.77:6789/0 conn(0x7f8d6815a730 legacy=0x7f8d680c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.096+0000 7f8d65d9b700 1 -- 172.21.15.27:0/1119307256 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d680cb520 con 0x7f8d6813ff80 2024-01-11T09:47:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.096+0000 7f8d65d9b700 1 -- 172.21.15.27:0/1119307256 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f8d5c001470 con 0x7f8d6813ff80 2024-01-11T09:47:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.096+0000 7f8d65d9b700 1 -- 172.21.15.27:0/1119307256 >> v1:172.21.15.27:6789/0 conn(0x7f8d6813ff80 legacy=0x7f8d6815a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.096+0000 7f8d65d9b700 1 --2- 172.21.15.27:0/1119307256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d50003a60 0x7f8d50003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.096+0000 7f8d65d9b700 1 --2- 172.21.15.27:0/1119307256 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d50004810 0x7f8d50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.096+0000 7f8d65d9b700 1 --2- 172.21.15.27:0/1119307256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d500051c0 0x7f8d5000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.096+0000 7f8d65d9b700 1 -- 172.21.15.27:0/1119307256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d680cb240 con 0x7f8d500051c0 2024-01-11T09:47:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.096+0000 7f8d65d9b700 1 -- 172.21.15.27:0/1119307256 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5000c0b0 con 0x7f8d50003a60 2024-01-11T09:47:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.096+0000 7f8d65d9b700 1 -- 172.21.15.27:0/1119307256 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5000c270 con 0x7f8d50004810 2024-01-11T09:47:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.097+0000 7f8d66d9d700 1 --2- 172.21.15.27:0/1119307256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d50003a60 0x7f8d50003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.097+0000 7f8d6759e700 1 --2- 172.21.15.27:0/1119307256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d500051c0 0x7f8d5000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.097+0000 7f8d6659c700 1 --2- 172.21.15.27:0/1119307256 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d50004810 0x7f8d50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.097+0000 7f8d66d9d700 1 -- 172.21.15.27:0/1119307256 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d50004810 msgr2=0x7f8d50004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.097+0000 7f8d66d9d700 1 --2- 172.21.15.27:0/1119307256 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d50004810 0x7f8d50004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.097+0000 7f8d66d9d700 1 -- 172.21.15.27:0/1119307256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d500051c0 msgr2=0x7f8d5000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.097+0000 7f8d66d9d700 1 --2- 172.21.15.27:0/1119307256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d500051c0 0x7f8d5000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.097+0000 7f8d66d9d700 1 -- 172.21.15.27:0/1119307256 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d50003570 con 0x7f8d50003a60 2024-01-11T09:47:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.097+0000 7f8d6759e700 1 --2- 172.21.15.27:0/1119307256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d500051c0 0x7f8d5000ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:08.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.097+0000 7f8d6759e700 1 -- 172.21.15.27:0/1119307256 reap_dead start 2024-01-11T09:47:08.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.097+0000 7f8d66d9d700 1 --2- 172.21.15.27:0/1119307256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d50003a60 0x7f8d50003e30 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f8d58009bf0 tx=0x7f8d58009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33c9213be42607c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:08.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.098+0000 7f8d65d9b700 1 -- 172.21.15.27:0/1119307256 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d58005a00 con 0x7f8d50003a60 2024-01-11T09:47:08.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.098+0000 7f8d65d9b700 1 -- 172.21.15.27:0/1119307256 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d58005b60 con 0x7f8d50003a60 2024-01-11T09:47:08.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.098+0000 7f8d65d9b700 1 -- 172.21.15.27:0/1119307256 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5800e710 con 0x7f8d50003a60 2024-01-11T09:47:08.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.098+0000 7f8d6d1ff700 1 -- 172.21.15.27:0/1119307256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d50003a60 msgr2=0x7f8d50003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:08.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.098+0000 7f8d6d1ff700 1 --2- 172.21.15.27:0/1119307256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d50003a60 0x7f8d50003e30 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f8d58009bf0 tx=0x7f8d58009120 comp rx=0 tx=0).stop 2024-01-11T09:47:08.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.098+0000 7f8d6d1ff700 1 -- 172.21.15.27:0/1119307256 shutdown_connections 2024-01-11T09:47:08.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.098+0000 7f8d6d1ff700 1 --2- 172.21.15.27:0/1119307256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d50003a60 0x7f8d50003e30 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.098+0000 7f8d6d1ff700 1 -- 172.21.15.27:0/1119307256 >> 172.21.15.27:0/1119307256 conn(0x7f8d680680a0 msgr2=0x7f8d680cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:08.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.098+0000 7f8d6d1ff700 1 -- 172.21.15.27:0/1119307256 shutdown_connections 2024-01-11T09:47:08.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.098+0000 7f8d6d1ff700 1 -- 172.21.15.27:0/1119307256 wait complete. 2024-01-11T09:47:08.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.100+0000 7f8d6d1ff700 1 Processor -- start 2024-01-11T09:47:08.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.100+0000 7f8d6d1ff700 1 -- start start 2024-01-11T09:47:08.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.101+0000 7f8d6d1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d50003a60 0x7f8d6813bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:08.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.101+0000 7f8d6d1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d680aeff0 0x7f8d680af460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:08.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.101+0000 7f8d6d1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d680c8a50 0x7f8d680c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:08.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.101+0000 7f8d6d1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d680c9430 con 0x7f8d680c8a50 2024-01-11T09:47:08.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.101+0000 7f8d6d1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d68157f60 con 0x7f8d50003a60 2024-01-11T09:47:08.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.101+0000 7f8d6d1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d680afa40 con 0x7f8d680aeff0 2024-01-11T09:47:08.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.101+0000 7f8d6759e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d680c8a50 0x7f8d680c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:08.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.101+0000 7f8d6759e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d680c8a50 0x7f8d680c8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42740/0 (socket says 172.21.15.27:42740) 2024-01-11T09:47:08.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.101+0000 7f8d6759e700 1 -- 172.21.15.27:0/3864634205 learned_addr learned my addr 172.21.15.27:0/3864634205 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:08.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.101+0000 7f8d66d9d700 1 --2- 172.21.15.27:0/3864634205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d50003a60 0x7f8d6813bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:08.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.101+0000 7f8d6659c700 1 --2- 172.21.15.27:0/3864634205 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d680aeff0 0x7f8d680af460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:08.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.102+0000 7f8d6759e700 1 -- 172.21.15.27:0/3864634205 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d680aeff0 msgr2=0x7f8d680af460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:08.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.102+0000 7f8d6759e700 1 --2- 172.21.15.27:0/3864634205 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d680aeff0 0x7f8d680af460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.102+0000 7f8d6759e700 1 -- 172.21.15.27:0/3864634205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d50003a60 msgr2=0x7f8d6813bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:08.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.102+0000 7f8d6759e700 1 --2- 172.21.15.27:0/3864634205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d50003a60 0x7f8d6813bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.102+0000 7f8d6759e700 1 -- 172.21.15.27:0/3864634205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d680afd20 con 0x7f8d680c8a50 2024-01-11T09:47:08.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.102+0000 7f8d6759e700 1 --2- 172.21.15.27:0/3864634205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d680c8a50 0x7f8d680c8ef0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f8d68066150 tx=0x7f8d60009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:08.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.102+0000 7f8d4effd700 1 -- 172.21.15.27:0/3864634205 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6000d8f0 con 0x7f8d680c8a50 2024-01-11T09:47:08.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.102+0000 7f8d4effd700 1 -- 172.21.15.27:0/3864634205 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6000da50 con 0x7f8d680c8a50 2024-01-11T09:47:08.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.102+0000 7f8d6d1ff700 1 -- 172.21.15.27:0/3864634205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d680b0000 con 0x7f8d680c8a50 2024-01-11T09:47:08.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.102+0000 7f8d4effd700 1 -- 172.21.15.27:0/3864634205 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6000de10 con 0x7f8d680c8a50 2024-01-11T09:47:08.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.103+0000 7f8d6d1ff700 1 -- 172.21.15.27:0/3864634205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d6813c2c0 con 0x7f8d680c8a50 2024-01-11T09:47:08.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.104+0000 7f8d6d1ff700 1 -- 172.21.15.27:0/3864634205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d68158120 con 0x7f8d680c8a50 2024-01-11T09:47:08.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.105+0000 7f8d4effd700 1 -- 172.21.15.27:0/3864634205 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f8d60003030 con 0x7f8d680c8a50 2024-01-11T09:47:08.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.105+0000 7f8d4effd700 1 --2- 172.21.15.27:0/3864634205 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d54067170 0x7f8d54069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:08.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.105+0000 7f8d4effd700 1 -- 172.21.15.27:0/3864634205 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f8d60084b50 con 0x7f8d680c8a50 2024-01-11T09:47:08.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.106+0000 7f8d66d9d700 1 --2- 172.21.15.27:0/3864634205 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d54067170 0x7f8d54069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:08.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.107+0000 7f8d66d9d700 1 --2- 172.21.15.27:0/3864634205 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d54067170 0x7f8d54069620 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8d58009160 tx=0x7f8d58019000 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:08.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.109+0000 7f8d4effd700 1 -- 172.21.15.27:0/3864634205 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8d68158120 con 0x7f8d680c8a50 2024-01-11T09:47:08.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.311+0000 7f8d6d1ff700 1 -- 172.21.15.27:0/3864634205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f8d68158120 con 0x7f8d680c8a50 2024-01-11T09:47:08.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.312+0000 7f8d4effd700 1 -- 172.21.15.27:0/3864634205 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v244) v1 ==== 103+0+11 (secure 0 0 0) 0x7f8d68158120 con 0x7f8d680c8a50 2024-01-11T09:47:08.303 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T09:47:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.314+0000 7f8d6d1ff700 1 -- 172.21.15.27:0/3864634205 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d54067170 msgr2=0x7f8d54069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.314+0000 7f8d6d1ff700 1 --2- 172.21.15.27:0/3864634205 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d54067170 0x7f8d54069620 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8d58009160 tx=0x7f8d58019000 comp rx=0 tx=0).stop 2024-01-11T09:47:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.314+0000 7f8d6d1ff700 1 -- 172.21.15.27:0/3864634205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d680c8a50 msgr2=0x7f8d680c8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.314+0000 7f8d6d1ff700 1 --2- 172.21.15.27:0/3864634205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d680c8a50 0x7f8d680c8ef0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f8d68066150 tx=0x7f8d60009040 comp rx=0 tx=0).stop 2024-01-11T09:47:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.314+0000 7f8d6d1ff700 1 -- 172.21.15.27:0/3864634205 shutdown_connections 2024-01-11T09:47:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.314+0000 7f8d6d1ff700 1 --2- 172.21.15.27:0/3864634205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d680c8a50 0x7f8d680c8ef0 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.314+0000 7f8d6d1ff700 1 --2- 172.21.15.27:0/3864634205 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8d54067170 0x7f8d54069620 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.314+0000 7f8d6d1ff700 1 --2- 172.21.15.27:0/3864634205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d50003a60 0x7f8d6813bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.314+0000 7f8d6d1ff700 1 --2- 172.21.15.27:0/3864634205 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8d680aeff0 0x7f8d680af460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.314+0000 7f8d6d1ff700 1 -- 172.21.15.27:0/3864634205 >> 172.21.15.27:0/3864634205 conn(0x7f8d680680a0 msgr2=0x7f8d6813ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:08.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.314+0000 7f8d6d1ff700 1 -- 172.21.15.27:0/3864634205 shutdown_connections 2024-01-11T09:47:08.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.315+0000 7f8d6d1ff700 1 -- 172.21.15.27:0/3864634205 wait complete. 2024-01-11T09:47:08.324 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:47:08.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.534+0000 7fd6761ff700 1 Processor -- start 2024-01-11T09:47:08.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.534+0000 7fd6761ff700 1 -- start start 2024-01-11T09:47:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.535+0000 7fd6761ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd670159670 0x7fd67015dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.535+0000 7fd6761ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd67006a700 0x7fd67006ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.535+0000 7fd6761ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd67015d860 con 0x7fd670159a40 2024-01-11T09:47:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.536+0000 7fd6761ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd67006b070 con 0x7fd670159670 2024-01-11T09:47:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.536+0000 7fd6761ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67015e230 con 0x7fd67006a700 2024-01-11T09:47:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.536+0000 7fd66f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd670159670 0x7fd67015dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.536+0000 7fd66f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd670159670 0x7fd67015dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42746/0 (socket says 172.21.15.27:42746) 2024-01-11T09:47:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.536+0000 7fd66f7fe700 1 -- 172.21.15.27:0/144003481 learned_addr learned my addr 172.21.15.27:0/144003481 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.536+0000 7fd66ffff700 1 --2- 172.21.15.27:0/144003481 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd67006a700 0x7fd67006ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.536+0000 7fd66f7fe700 1 -- 172.21.15.27:0/144003481 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd67006a700 msgr2=0x7fd67006ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.536+0000 7fd66f7fe700 1 --2- 172.21.15.27:0/144003481 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd67006a700 0x7fd67006ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.536+0000 7fd66f7fe700 1 -- 172.21.15.27:0/144003481 >> v1:172.21.15.77:6789/0 conn(0x7fd670159a40 legacy=0x7fd67006e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.536+0000 7fd66f7fe700 1 -- 172.21.15.27:0/144003481 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67006ecd0 con 0x7fd670159670 2024-01-11T09:47:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.536+0000 7fd66ffff700 1 --2- 172.21.15.27:0/144003481 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd67006a700 0x7fd67006ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.536+0000 7fd66f7fe700 1 --2- 172.21.15.27:0/144003481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd670159670 0x7fd67015dcf0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fd6600099a0 tx=0x7fd660009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce03b064f0581001 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.537+0000 7fd66e7fc700 1 -- 172.21.15.27:0/144003481 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66000b860 con 0x7fd670159670 2024-01-11T09:47:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.537+0000 7fd66e7fc700 1 -- 172.21.15.27:0/144003481 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66000b9c0 con 0x7fd670159670 2024-01-11T09:47:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.537+0000 7fd66e7fc700 1 -- 172.21.15.27:0/144003481 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66000bd80 con 0x7fd670159670 2024-01-11T09:47:08.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.537+0000 7fd6761ff700 1 -- 172.21.15.27:0/144003481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd670159670 msgr2=0x7fd67015dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:08.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.537+0000 7fd6761ff700 1 --2- 172.21.15.27:0/144003481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd670159670 0x7fd67015dcf0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fd6600099a0 tx=0x7fd660009250 comp rx=0 tx=0).stop 2024-01-11T09:47:08.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.537+0000 7fd6761ff700 1 -- 172.21.15.27:0/144003481 shutdown_connections 2024-01-11T09:47:08.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.537+0000 7fd6761ff700 1 --2- 172.21.15.27:0/144003481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd670159670 0x7fd67015dcf0 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.537+0000 7fd6761ff700 1 --2- 172.21.15.27:0/144003481 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd67006a700 0x7fd67006ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.537+0000 7fd6761ff700 1 -- 172.21.15.27:0/144003481 >> 172.21.15.27:0/144003481 conn(0x7fd6700cd2b0 msgr2=0x7fd6700cd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.537+0000 7fd6761ff700 1 -- 172.21.15.27:0/144003481 shutdown_connections 2024-01-11T09:47:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.537+0000 7fd6761ff700 1 -- 172.21.15.27:0/144003481 wait complete. 2024-01-11T09:47:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.539+0000 7fd6761ff700 1 Processor -- start 2024-01-11T09:47:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.539+0000 7fd6761ff700 1 -- start start 2024-01-11T09:47:08.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.540+0000 7fd6761ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd67006a700 0x7fd67016dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:08.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.540+0000 7fd6761ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd670159670 0x7fd670156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:08.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.540+0000 7fd6761ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd670159a40 0x7fd6701572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:08.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.540+0000 7fd6761ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67016add0 con 0x7fd670159670 2024-01-11T09:47:08.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.540+0000 7fd6761ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6700b1530 con 0x7fd670159a40 2024-01-11T09:47:08.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.540+0000 7fd6761ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67016a030 con 0x7fd67006a700 2024-01-11T09:47:08.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.540+0000 7fd66effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd670159670 0x7fd670156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:08.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.540+0000 7fd66effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd670159670 0x7fd670156db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42760/0 (socket says 172.21.15.27:42760) 2024-01-11T09:47:08.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.540+0000 7fd66effd700 1 -- 172.21.15.27:0/3352460756 learned_addr learned my addr 172.21.15.27:0/3352460756 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:08.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.540+0000 7fd66f7fe700 1 --2- 172.21.15.27:0/3352460756 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd67006a700 0x7fd67016dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:08.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.540+0000 7fd66ffff700 1 --2- 172.21.15.27:0/3352460756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd670159a40 0x7fd6701572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:08.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.541+0000 7fd66effd700 1 -- 172.21.15.27:0/3352460756 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd67006a700 msgr2=0x7fd67016dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:08.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.541+0000 7fd66effd700 1 --2- 172.21.15.27:0/3352460756 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd67006a700 0x7fd67016dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.541+0000 7fd66effd700 1 -- 172.21.15.27:0/3352460756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd670159a40 msgr2=0x7fd6701572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:08.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.541+0000 7fd66effd700 1 --2- 172.21.15.27:0/3352460756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd670159a40 0x7fd6701572f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.541+0000 7fd66effd700 1 -- 172.21.15.27:0/3352460756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd660009490 con 0x7fd670159670 2024-01-11T09:47:08.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.541+0000 7fd66effd700 1 --2- 172.21.15.27:0/3352460756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd670159670 0x7fd670156db0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fd66400c4e0 tx=0x7fd66400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:08.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.541+0000 7fd6577fe700 1 -- 172.21.15.27:0/3352460756 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664011590 con 0x7fd670159670 2024-01-11T09:47:08.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.541+0000 7fd6577fe700 1 -- 172.21.15.27:0/3352460756 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd664011710 con 0x7fd670159670 2024-01-11T09:47:08.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.541+0000 7fd6761ff700 1 -- 172.21.15.27:0/3352460756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd67016a310 con 0x7fd670159670 2024-01-11T09:47:08.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.541+0000 7fd6577fe700 1 -- 172.21.15.27:0/3352460756 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664011ad0 con 0x7fd670159670 2024-01-11T09:47:08.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.541+0000 7fd6761ff700 1 -- 172.21.15.27:0/3352460756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd67016a8d0 con 0x7fd670159670 2024-01-11T09:47:08.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.542+0000 7fd6761ff700 1 -- 172.21.15.27:0/3352460756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6700718d0 con 0x7fd670159670 2024-01-11T09:47:08.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.544+0000 7fd6577fe700 1 -- 172.21.15.27:0/3352460756 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fd66401b3b0 con 0x7fd670159670 2024-01-11T09:47:08.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.544+0000 7fd6577fe700 1 --2- 172.21.15.27:0/3352460756 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd65c067170 0x7fd65c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:08.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.545+0000 7fd6577fe700 1 -- 172.21.15.27:0/3352460756 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6689+0+0 (secure 0 0 0) 0x7fd6640840d0 con 0x7fd670159670 2024-01-11T09:47:08.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.545+0000 7fd66f7fe700 1 --2- 172.21.15.27:0/3352460756 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd65c067170 0x7fd65c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:08.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.546+0000 7fd66f7fe700 1 --2- 172.21.15.27:0/3352460756 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd65c067170 0x7fd65c069620 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd6600099a0 tx=0x7fd660017000 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:08.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.548+0000 7fd6577fe700 1 -- 172.21.15.27:0/3352460756 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd664050c30 con 0x7fd670159670 2024-01-11T09:47:08.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.794+0000 7fd6761ff700 1 -- 172.21.15.27:0/3352460756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd67006efa0 con 0x7fd670159670 2024-01-11T09:47:08.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.795+0000 7fd6577fe700 1 -- 172.21.15.27:0/3352460756 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+4614 (secure 0 0 0) 0x7fd664017020 con 0x7fd670159670 2024-01-11T09:47:08.786 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:08.786 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":134,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1568813392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1568813392},{"type":"v1","addr":"172.21.15.77:6834","nonce":1568813392}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122},{"gid":9740,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3700058093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":3700058093},{"type":"v1","addr":"172.21.15.27:6835","nonce":3700058093}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134},{"gid":10969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1352491019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1352491019},{"type":"v1","addr":"172.21.15.77:6835","nonce":1352491019}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:46:39.483941+0000","modified":"2024-01-11T09:46:42.996786+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11331},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11331":{"gid":11331,"name":"a","rank":0,"incarnation":129,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6834/3477661846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3477661846},{"type":"v1","addr":"172.21.15.27:6834","nonce":3477661846}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":16}]} 2024-01-11T09:47:08.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.797+0000 7fd6761ff700 1 -- 172.21.15.27:0/3352460756 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd65c067170 msgr2=0x7fd65c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:08.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.797+0000 7fd6761ff700 1 --2- 172.21.15.27:0/3352460756 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd65c067170 0x7fd65c069620 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd6600099a0 tx=0x7fd660017000 comp rx=0 tx=0).stop 2024-01-11T09:47:08.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.797+0000 7fd6761ff700 1 -- 172.21.15.27:0/3352460756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd670159670 msgr2=0x7fd670156db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:08.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.797+0000 7fd6761ff700 1 --2- 172.21.15.27:0/3352460756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd670159670 0x7fd670156db0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fd66400c4e0 tx=0x7fd66400c7f0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.797+0000 7fd6761ff700 1 -- 172.21.15.27:0/3352460756 shutdown_connections 2024-01-11T09:47:08.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.797+0000 7fd6761ff700 1 --2- 172.21.15.27:0/3352460756 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd65c067170 0x7fd65c069620 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.797+0000 7fd6761ff700 1 --2- 172.21.15.27:0/3352460756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd670159a40 0x7fd6701572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.797+0000 7fd6761ff700 1 --2- 172.21.15.27:0/3352460756 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd67006a700 0x7fd67016dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.797+0000 7fd6761ff700 1 --2- 172.21.15.27:0/3352460756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd670159670 0x7fd670156db0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:08.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.797+0000 7fd6761ff700 1 -- 172.21.15.27:0/3352460756 >> 172.21.15.27:0/3352460756 conn(0x7fd6700cd2b0 msgr2=0x7fd670158950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:08.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.797+0000 7fd6761ff700 1 -- 172.21.15.27:0/3352460756 shutdown_connections 2024-01-11T09:47:08.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:08.797+0000 7fd6761ff700 1 -- 172.21.15.27:0/3352460756 wait complete. 2024-01-11T09:47:08.790 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 134 2024-01-11T09:47:08.806 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:47:09.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.015+0000 7f8cfc7ff700 1 Processor -- start 2024-01-11T09:47:09.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.015+0000 7f8cfc7ff700 1 -- start start 2024-01-11T09:47:09.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.016+0000 7f8cfc7ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8cf41599e0 0x7f8cf40b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.016+0000 7f8cfc7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8cf4150330 0x7f8cf415a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.016+0000 7f8cfc7ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cf4157e30 con 0x7f8cf4159db0 2024-01-11T09:47:09.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.016+0000 7f8cfc7ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cf4157500 con 0x7f8cf41599e0 2024-01-11T09:47:09.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.016+0000 7f8cfc7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cf41577e0 con 0x7f8cf4150330 2024-01-11T09:47:09.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.016+0000 7f8cfa59b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8cf41599e0 0x7f8cf40b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:09.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.016+0000 7f8cfa59b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8cf41599e0 0x7f8cf40b0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42774/0 (socket says 172.21.15.27:42774) 2024-01-11T09:47:09.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.016+0000 7f8cfa59b700 1 -- 172.21.15.27:0/426564202 learned_addr learned my addr 172.21.15.27:0/426564202 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:09.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.017+0000 7f8cfad9c700 1 --2- 172.21.15.27:0/426564202 >> v2:172.21.15.77:3300/0 conn(0x7f8cf4150330 0x7f8cf415a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:09.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.017+0000 7f8cfa59b700 1 -- 172.21.15.27:0/426564202 >> v2:172.21.15.77:3300/0 conn(0x7f8cf4150330 msgr2=0x7f8cf415a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:09.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.017+0000 7f8cfa59b700 1 --2- 172.21.15.27:0/426564202 >> v2:172.21.15.77:3300/0 conn(0x7f8cf4150330 0x7f8cf415a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.017+0000 7f8cfa59b700 1 -- 172.21.15.27:0/426564202 >> v1:172.21.15.77:6789/0 conn(0x7f8cf4159db0 legacy=0x7f8cf414fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:09.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.017+0000 7f8cfa59b700 1 -- 172.21.15.27:0/426564202 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf4157ac0 con 0x7f8cf41599e0 2024-01-11T09:47:09.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.017+0000 7f8cfa59b700 1 --2- 172.21.15.27:0/426564202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf41599e0 0x7f8cf40b0130 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f8cf00099b0 tx=0x7f8cf0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32dbc7d3aac9b51a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:09.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.017+0000 7f8cf9599700 1 -- 172.21.15.27:0/426564202 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf000b820 con 0x7f8cf41599e0 2024-01-11T09:47:09.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.017+0000 7f8cf9599700 1 -- 172.21.15.27:0/426564202 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf000b980 con 0x7f8cf41599e0 2024-01-11T09:47:09.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.017+0000 7f8cf9599700 1 -- 172.21.15.27:0/426564202 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf000bd20 con 0x7f8cf41599e0 2024-01-11T09:47:09.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.018+0000 7f8cfc7ff700 1 -- 172.21.15.27:0/426564202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf41599e0 msgr2=0x7f8cf40b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:09.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.018+0000 7f8cfc7ff700 1 --2- 172.21.15.27:0/426564202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf41599e0 0x7f8cf40b0130 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f8cf00099b0 tx=0x7f8cf0009250 comp rx=0 tx=0).stop 2024-01-11T09:47:09.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.018+0000 7f8cfc7ff700 1 -- 172.21.15.27:0/426564202 shutdown_connections 2024-01-11T09:47:09.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.018+0000 7f8cfc7ff700 1 --2- 172.21.15.27:0/426564202 >> v2:172.21.15.77:3300/0 conn(0x7f8cf4150330 0x7f8cf415a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.018+0000 7f8cfc7ff700 1 --2- 172.21.15.27:0/426564202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf41599e0 0x7f8cf40b0130 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.018+0000 7f8cfc7ff700 1 -- 172.21.15.27:0/426564202 >> 172.21.15.27:0/426564202 conn(0x7f8cf40680e0 msgr2=0x7f8cf4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:09.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.018+0000 7f8cfc7ff700 1 -- 172.21.15.27:0/426564202 shutdown_connections 2024-01-11T09:47:09.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.018+0000 7f8cfc7ff700 1 -- 172.21.15.27:0/426564202 wait complete. 2024-01-11T09:47:09.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.020+0000 7f8cfc7ff700 1 Processor -- start 2024-01-11T09:47:09.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.020+0000 7f8cfc7ff700 1 -- start start 2024-01-11T09:47:09.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.021+0000 7f8cfc7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8cf4150330 0x7f8cf40c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.021+0000 7f8cfc7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf41599e0 0x7f8cf40ada30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.021+0000 7f8cfc7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cf4159db0 0x7f8cf4069800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.021+0000 7f8cfc7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf40adf70 con 0x7f8cf41599e0 2024-01-11T09:47:09.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.021+0000 7f8cfc7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf40c8fe0 con 0x7f8cf4159db0 2024-01-11T09:47:09.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.021+0000 7f8cfc7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf40c92c0 con 0x7f8cf4150330 2024-01-11T09:47:09.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.021+0000 7f8cf9d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf41599e0 0x7f8cf40ada30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:09.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.021+0000 7f8cf9d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf41599e0 0x7f8cf40ada30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42780/0 (socket says 172.21.15.27:42780) 2024-01-11T09:47:09.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.021+0000 7f8cf9d9a700 1 -- 172.21.15.27:0/1001174174 learned_addr learned my addr 172.21.15.27:0/1001174174 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:09.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.021+0000 7f8cfa59b700 1 --2- 172.21.15.27:0/1001174174 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8cf4150330 0x7f8cf40c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.021+0000 7f8cfad9c700 1 --2- 172.21.15.27:0/1001174174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cf4159db0 0x7f8cf4069800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.022+0000 7f8cf9d9a700 1 -- 172.21.15.27:0/1001174174 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8cf4150330 msgr2=0x7f8cf40c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.022+0000 7f8cf9d9a700 1 --2- 172.21.15.27:0/1001174174 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8cf4150330 0x7f8cf40c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.022+0000 7f8cf9d9a700 1 -- 172.21.15.27:0/1001174174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cf4159db0 msgr2=0x7f8cf4069800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.022+0000 7f8cf9d9a700 1 --2- 172.21.15.27:0/1001174174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cf4159db0 0x7f8cf4069800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.022+0000 7f8cf9d9a700 1 -- 172.21.15.27:0/1001174174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf00094f0 con 0x7f8cf41599e0 2024-01-11T09:47:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.022+0000 7f8cf9d9a700 1 --2- 172.21.15.27:0/1001174174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf41599e0 0x7f8cf40ada30 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f8ce400c4e0 tx=0x7f8ce400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.022+0000 7f8cea7fc700 1 -- 172.21.15.27:0/1001174174 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce4011590 con 0x7f8cf41599e0 2024-01-11T09:47:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.022+0000 7f8cea7fc700 1 -- 172.21.15.27:0/1001174174 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce4011710 con 0x7f8cf41599e0 2024-01-11T09:47:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.022+0000 7f8cea7fc700 1 -- 172.21.15.27:0/1001174174 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce4011ad0 con 0x7f8cf41599e0 2024-01-11T09:47:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.022+0000 7f8cfc7ff700 1 -- 172.21.15.27:0/1001174174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf40c95a0 con 0x7f8cf41599e0 2024-01-11T09:47:09.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.022+0000 7f8cfc7ff700 1 -- 172.21.15.27:0/1001174174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf40c9b60 con 0x7f8cf41599e0 2024-01-11T09:47:09.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.023+0000 7f8cfc7ff700 1 -- 172.21.15.27:0/1001174174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cf40718d0 con 0x7f8cf41599e0 2024-01-11T09:47:09.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.024+0000 7f8cea7fc700 1 -- 172.21.15.27:0/1001174174 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f8ce401b3b0 con 0x7f8cf41599e0 2024-01-11T09:47:09.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.025+0000 7f8cea7fc700 1 --2- 172.21.15.27:0/1001174174 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8cd8067170 0x7f8cd8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.025+0000 7f8cea7fc700 1 -- 172.21.15.27:0/1001174174 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f8ce40841d0 con 0x7f8cf41599e0 2024-01-11T09:47:09.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.025+0000 7f8cfa59b700 1 --2- 172.21.15.27:0/1001174174 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8cd8067170 0x7f8cd8069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:09.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.026+0000 7f8cfa59b700 1 --2- 172.21.15.27:0/1001174174 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8cd8067170 0x7f8cd8069620 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8cf00099b0 tx=0x7f8cf0017000 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:09.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.029+0000 7f8cea7fc700 1 -- 172.21.15.27:0/1001174174 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8ce4050d30 con 0x7f8cf41599e0 2024-01-11T09:47:09.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.232+0000 7f8cfc7ff700 1 -- 172.21.15.27:0/1001174174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cf40b0670 con 0x7f8cf41599e0 2024-01-11T09:47:09.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.232+0000 7f8cea7fc700 1 -- 172.21.15.27:0/1001174174 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v244) v1 ==== 74+0+16739 (secure 0 0 0) 0x7f8ce4017090 con 0x7f8cf41599e0 2024-01-11T09:47:09.224 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:09.224 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":244,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:47:06.507632+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-01-11T09:46:37.665603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"238","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-01-11T09:46:38.472134+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"239","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/321681407":"2024-01-12T09:47:02.847264+0000","172.21.15.77:0/463475654":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/1391607686":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/2619336301":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/1120758885":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/1857056349":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/210673663":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/3835722213":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/696729061":"2024-01-12T09:46:54.199949+0000","172.21.15.27:6833/321681407":"2024-01-12T09:47:02.847264+0000","172.21.15.77:0/206349247":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/228357086":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/3326555344":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/3398849669":"2024-01-12T09:46:54.199949+0000","172.21.15.77:6837/59776":"2024-01-12T09:46:54.199949+0000","172.21.15.77:6836/59776":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/1934793537":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/842931202":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/1366958778":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/198451994":"2024-01-12T09:46:54.199949+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:47:09.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.235+0000 7f8cfc7ff700 1 -- 172.21.15.27:0/1001174174 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8cd8067170 msgr2=0x7f8cd8069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:09.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.235+0000 7f8cfc7ff700 1 --2- 172.21.15.27:0/1001174174 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8cd8067170 0x7f8cd8069620 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8cf00099b0 tx=0x7f8cf0017000 comp rx=0 tx=0).stop 2024-01-11T09:47:09.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.235+0000 7f8cfc7ff700 1 -- 172.21.15.27:0/1001174174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf41599e0 msgr2=0x7f8cf40ada30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:09.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.235+0000 7f8cfc7ff700 1 --2- 172.21.15.27:0/1001174174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf41599e0 0x7f8cf40ada30 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f8ce400c4e0 tx=0x7f8ce400c8a0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.235+0000 7f8cfc7ff700 1 -- 172.21.15.27:0/1001174174 shutdown_connections 2024-01-11T09:47:09.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.235+0000 7f8cfc7ff700 1 --2- 172.21.15.27:0/1001174174 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8cd8067170 0x7f8cd8069620 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.235+0000 7f8cfc7ff700 1 --2- 172.21.15.27:0/1001174174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cf4159db0 0x7f8cf4069800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.235+0000 7f8cfc7ff700 1 --2- 172.21.15.27:0/1001174174 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8cf4150330 0x7f8cf40c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.235+0000 7f8cfc7ff700 1 --2- 172.21.15.27:0/1001174174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf41599e0 0x7f8cf40ada30 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.235+0000 7f8cfc7ff700 1 -- 172.21.15.27:0/1001174174 >> 172.21.15.27:0/1001174174 conn(0x7f8cf40680e0 msgr2=0x7f8cf414f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:09.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.235+0000 7f8cfc7ff700 1 -- 172.21.15.27:0/1001174174 shutdown_connections 2024-01-11T09:47:09.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.235+0000 7f8cfc7ff700 1 -- 172.21.15.27:0/1001174174 wait complete. 2024-01-11T09:47:09.245 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2024-01-11T09:47:09.245 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:47:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.453+0000 7f5ccc3ff700 1 Processor -- start 2024-01-11T09:47:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.454+0000 7f5ccc3ff700 1 -- start start 2024-01-11T09:47:09.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.455+0000 7f5ccc3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5cc4069d40 0x7f5cc40b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.455+0000 7f5ccc3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cc4157390 con 0x7f5cc40c8620 2024-01-11T09:47:09.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.455+0000 7f5ccc3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cc406a200 con 0x7f5cc40c89f0 2024-01-11T09:47:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.455+0000 7f5ccc3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc4156d40 con 0x7f5cc4069d40 2024-01-11T09:47:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.455+0000 7f5cca19b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5cc40c8620 0x7f5cc40af970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58898/0 (socket says 172.21.15.27:58898) 2024-01-11T09:47:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.455+0000 7f5cca19b700 1 -- 172.21.15.27:0/2065916623 learned_addr learned my addr 172.21.15.27:0/2065916623 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.455+0000 7f5cc9199700 1 -- 172.21.15.27:0/2065916623 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209167601 0 0) 0x7f5cc4157390 con 0x7f5cc40c8620 2024-01-11T09:47:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.456+0000 7f5cc9199700 1 -- 172.21.15.27:0/2065916623 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cb0003570 con 0x7f5cc40c8620 2024-01-11T09:47:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.456+0000 7f5cc999a700 1 --2- 172.21.15.27:0/2065916623 >> v2:172.21.15.77:3300/0 conn(0x7f5cc4069d40 0x7f5cc40b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.456+0000 7f5cc9199700 1 -- 172.21.15.27:0/2065916623 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1678471159 0 0) 0x7f5cb0003570 con 0x7f5cc40c8620 2024-01-11T09:47:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.456+0000 7f5cc9199700 1 -- 172.21.15.27:0/2065916623 >> v2:172.21.15.77:3300/0 conn(0x7f5cc4069d40 msgr2=0x7f5cc40b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.456+0000 7f5cc9199700 1 --2- 172.21.15.27:0/2065916623 >> v2:172.21.15.77:3300/0 conn(0x7f5cc4069d40 0x7f5cc40b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.456+0000 7f5cc9199700 1 -- 172.21.15.27:0/2065916623 >> v1:172.21.15.77:6789/0 conn(0x7f5cc40c89f0 legacy=0x7f5cc40accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.456+0000 7f5cc9199700 1 -- 172.21.15.27:0/2065916623 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc4157020 con 0x7f5cc40c8620 2024-01-11T09:47:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.456+0000 7f5cc9199700 1 -- 172.21.15.27:0/2065916623 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f5cc0001470 con 0x7f5cc40c8620 2024-01-11T09:47:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.456+0000 7f5cc999a700 1 --2- 172.21.15.27:0/2065916623 >> v2:172.21.15.77:3300/0 conn(0x7f5cc4069d40 0x7f5cc40b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.456+0000 7f5cc9199700 1 -- 172.21.15.27:0/2065916623 >> v1:172.21.15.27:6789/0 conn(0x7f5cc40c8620 legacy=0x7f5cc40af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.456+0000 7f5cc9199700 1 --2- 172.21.15.27:0/2065916623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cb0003ab0 0x7f5cb0003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.456+0000 7f5cc9199700 1 --2- 172.21.15.27:0/2065916623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb0004810 0x7f5cb0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.456+0000 7f5cc9199700 1 --2- 172.21.15.27:0/2065916623 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cb00052e0 0x7f5cb000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.456+0000 7f5cc9199700 1 -- 172.21.15.27:0/2065916623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc4156d40 con 0x7f5cb0004810 2024-01-11T09:47:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.456+0000 7f5cc9199700 1 -- 172.21.15.27:0/2065916623 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb000c260 con 0x7f5cb0003ab0 2024-01-11T09:47:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.456+0000 7f5cc9199700 1 -- 172.21.15.27:0/2065916623 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb000c420 con 0x7f5cb00052e0 2024-01-11T09:47:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.456+0000 7f5cc999a700 1 --2- 172.21.15.27:0/2065916623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb0004810 0x7f5cb0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.456+0000 7f5cca19b700 1 --2- 172.21.15.27:0/2065916623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cb0003ab0 0x7f5cb0003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.456+0000 7f5cca99c700 1 --2- 172.21.15.27:0/2065916623 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cb00052e0 0x7f5cb000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.457+0000 7f5cc999a700 1 -- 172.21.15.27:0/2065916623 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cb00052e0 msgr2=0x7f5cb000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.457+0000 7f5cc999a700 1 --2- 172.21.15.27:0/2065916623 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cb00052e0 0x7f5cb000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.457+0000 7f5cc999a700 1 -- 172.21.15.27:0/2065916623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cb0003ab0 msgr2=0x7f5cb0003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.457+0000 7f5cc999a700 1 --2- 172.21.15.27:0/2065916623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cb0003ab0 0x7f5cb0003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.457+0000 7f5cc999a700 1 -- 172.21.15.27:0/2065916623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb0003570 con 0x7f5cb0004810 2024-01-11T09:47:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.457+0000 7f5cca19b700 1 --2- 172.21.15.27:0/2065916623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cb0003ab0 0x7f5cb0003ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.457+0000 7f5cca99c700 1 -- 172.21.15.27:0/2065916623 reap_dead start 2024-01-11T09:47:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.457+0000 7f5cc999a700 1 --2- 172.21.15.27:0/2065916623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb0004810 0x7f5cb0004c80 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4009ba0 tx=0x7f5cb4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=829021077e92b423 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.457+0000 7f5cc9199700 1 -- 172.21.15.27:0/2065916623 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb400c070 con 0x7f5cb0004810 2024-01-11T09:47:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.457+0000 7f5cc9199700 1 -- 172.21.15.27:0/2065916623 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb400e5e0 con 0x7f5cb0004810 2024-01-11T09:47:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.457+0000 7f5cc9199700 1 -- 172.21.15.27:0/2065916623 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb4009510 con 0x7f5cb0004810 2024-01-11T09:47:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.458+0000 7f5ccc3ff700 1 -- 172.21.15.27:0/2065916623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb0004810 msgr2=0x7f5cb0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.458+0000 7f5ccc3ff700 1 --2- 172.21.15.27:0/2065916623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb0004810 0x7f5cb0004c80 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4009ba0 tx=0x7f5cb4008040 comp rx=0 tx=0).stop 2024-01-11T09:47:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.458+0000 7f5ccc3ff700 1 -- 172.21.15.27:0/2065916623 shutdown_connections 2024-01-11T09:47:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.458+0000 7f5ccc3ff700 1 --2- 172.21.15.27:0/2065916623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb0004810 0x7f5cb0004c80 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.458+0000 7f5ccc3ff700 1 -- 172.21.15.27:0/2065916623 >> 172.21.15.27:0/2065916623 conn(0x7f5cc40680e0 msgr2=0x7f5cc4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.458+0000 7f5ccc3ff700 1 -- 172.21.15.27:0/2065916623 shutdown_connections 2024-01-11T09:47:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.458+0000 7f5ccc3ff700 1 -- 172.21.15.27:0/2065916623 wait complete. 2024-01-11T09:47:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.459+0000 7f5ccc3ff700 1 Processor -- start 2024-01-11T09:47:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.460+0000 7f5ccc3ff700 1 -- start start 2024-01-11T09:47:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.460+0000 7f5ccc3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb0004810 0x7f5cc40cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.460+0000 7f5ccc3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc4142c60 0x7f5cc4158190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.460+0000 7f5ccc3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cc41586d0 0x7f5cc40a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.461+0000 7f5ccc3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc4158b50 con 0x7f5cb0004810 2024-01-11T09:47:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.461+0000 7f5ccc3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc40ca190 con 0x7f5cc4142c60 2024-01-11T09:47:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.461+0000 7f5ccc3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc40ca430 con 0x7f5cc41586d0 2024-01-11T09:47:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.461+0000 7f5cca19b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb0004810 0x7f5cc40cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.461+0000 7f5cca19b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb0004810 0x7f5cc40cb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42802/0 (socket says 172.21.15.27:42802) 2024-01-11T09:47:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.461+0000 7f5cca19b700 1 -- 172.21.15.27:0/463663537 learned_addr learned my addr 172.21.15.27:0/463663537 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.461+0000 7f5cca99c700 1 --2- 172.21.15.27:0/463663537 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cc41586d0 0x7f5cc40a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:09.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.461+0000 7f5cc999a700 1 --2- 172.21.15.27:0/463663537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc4142c60 0x7f5cc4158190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:09.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.461+0000 7f5cca19b700 1 -- 172.21.15.27:0/463663537 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cc41586d0 msgr2=0x7f5cc40a5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:09.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.461+0000 7f5cca19b700 1 --2- 172.21.15.27:0/463663537 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cc41586d0 0x7f5cc40a5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.461+0000 7f5cca19b700 1 -- 172.21.15.27:0/463663537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc4142c60 msgr2=0x7f5cc4158190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:09.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.461+0000 7f5cca19b700 1 --2- 172.21.15.27:0/463663537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc4142c60 0x7f5cc4158190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.461+0000 7f5cca19b700 1 -- 172.21.15.27:0/463663537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc40ca710 con 0x7f5cb0004810 2024-01-11T09:47:09.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.462+0000 7f5cca19b700 1 --2- 172.21.15.27:0/463663537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb0004810 0x7f5cc40cb1f0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0000fd0 tx=0x7f5cc0001e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:09.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.462+0000 7f5cba7fc700 1 -- 172.21.15.27:0/463663537 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc000ee70 con 0x7f5cb0004810 2024-01-11T09:47:09.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.462+0000 7f5cba7fc700 1 -- 172.21.15.27:0/463663537 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc000c7e0 con 0x7f5cb0004810 2024-01-11T09:47:09.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.462+0000 7f5ccc3ff700 1 -- 172.21.15.27:0/463663537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc40ca9f0 con 0x7f5cb0004810 2024-01-11T09:47:09.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.462+0000 7f5cba7fc700 1 -- 172.21.15.27:0/463663537 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc000cb60 con 0x7f5cb0004810 2024-01-11T09:47:09.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.462+0000 7f5ccc3ff700 1 -- 172.21.15.27:0/463663537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc41430d0 con 0x7f5cb0004810 2024-01-11T09:47:09.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.463+0000 7f5ccc3ff700 1 -- 172.21.15.27:0/463663537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cc40718d0 con 0x7f5cb0004810 2024-01-11T09:47:09.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.464+0000 7f5cba7fc700 1 -- 172.21.15.27:0/463663537 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f5cc001d460 con 0x7f5cb0004810 2024-01-11T09:47:09.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.465+0000 7f5cba7fc700 1 --2- 172.21.15.27:0/463663537 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5ca8067170 0x7f5ca8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.465+0000 7f5cba7fc700 1 -- 172.21.15.27:0/463663537 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f5cc0084c70 con 0x7f5cb0004810 2024-01-11T09:47:09.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.465+0000 7f5cc999a700 1 --2- 172.21.15.27:0/463663537 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5ca8067170 0x7f5ca8069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:09.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.466+0000 7f5cc999a700 1 --2- 172.21.15.27:0/463663537 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5ca8067170 0x7f5ca8069620 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4009ba0 tx=0x7f5cb401a000 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:09.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.469+0000 7f5cba7fc700 1 -- 172.21.15.27:0/463663537 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5cc00517d0 con 0x7f5cb0004810 2024-01-11T09:47:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.721+0000 7f5ccc3ff700 1 -- 172.21.15.27:0/463663537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5cc40b1150 con 0x7f5cb0004810 2024-01-11T09:47:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.722+0000 7f5cba7fc700 1 -- 172.21.15.27:0/463663537 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+4614 (secure 0 0 0) 0x7f5cc0054df0 con 0x7f5cb0004810 2024-01-11T09:47:09.713 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:09.713 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":134,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1568813392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1568813392},{"type":"v1","addr":"172.21.15.77:6834","nonce":1568813392}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122},{"gid":9740,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3700058093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":3700058093},{"type":"v1","addr":"172.21.15.27:6835","nonce":3700058093}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134},{"gid":10969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1352491019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1352491019},{"type":"v1","addr":"172.21.15.77:6835","nonce":1352491019}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:46:39.483941+0000","modified":"2024-01-11T09:46:42.996786+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11331},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11331":{"gid":11331,"name":"a","rank":0,"incarnation":129,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6834/3477661846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3477661846},{"type":"v1","addr":"172.21.15.27:6834","nonce":3477661846}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":16}]} 2024-01-11T09:47:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.724+0000 7f5ccc3ff700 1 -- 172.21.15.27:0/463663537 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5ca8067170 msgr2=0x7f5ca8069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.724+0000 7f5ccc3ff700 1 --2- 172.21.15.27:0/463663537 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5ca8067170 0x7f5ca8069620 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4009ba0 tx=0x7f5cb401a000 comp rx=0 tx=0).stop 2024-01-11T09:47:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.724+0000 7f5ccc3ff700 1 -- 172.21.15.27:0/463663537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb0004810 msgr2=0x7f5cc40cb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.724+0000 7f5ccc3ff700 1 --2- 172.21.15.27:0/463663537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb0004810 0x7f5cc40cb1f0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0000fd0 tx=0x7f5cc0001e00 comp rx=0 tx=0).stop 2024-01-11T09:47:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.724+0000 7f5ccc3ff700 1 -- 172.21.15.27:0/463663537 shutdown_connections 2024-01-11T09:47:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.724+0000 7f5ccc3ff700 1 --2- 172.21.15.27:0/463663537 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5cc41586d0 0x7f5cc40a5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.724+0000 7f5ccc3ff700 1 --2- 172.21.15.27:0/463663537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb0004810 0x7f5cc40cb1f0 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.724+0000 7f5ccc3ff700 1 --2- 172.21.15.27:0/463663537 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5ca8067170 0x7f5ca8069620 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.724+0000 7f5ccc3ff700 1 --2- 172.21.15.27:0/463663537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc4142c60 0x7f5cc4158190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.724+0000 7f5ccc3ff700 1 -- 172.21.15.27:0/463663537 >> 172.21.15.27:0/463663537 conn(0x7f5cc40680e0 msgr2=0x7f5cc4157820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.724+0000 7f5ccc3ff700 1 -- 172.21.15.27:0/463663537 shutdown_connections 2024-01-11T09:47:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.724+0000 7f5ccc3ff700 1 -- 172.21.15.27:0/463663537 wait complete. 2024-01-11T09:47:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 134 2024-01-11T09:47:09.734 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:47:09.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.941+0000 7feb807ff700 1 Processor -- start 2024-01-11T09:47:09.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.941+0000 7feb807ff700 1 -- start start 2024-01-11T09:47:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.943+0000 7feb807ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feb780ca7b0 0x7feb780cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.943+0000 7feb807ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feb78159e80 0x7feb7815dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.943+0000 7feb807ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7815d640 con 0x7feb780ca3e0 2024-01-11T09:47:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.943+0000 7feb807ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb7815ae70 con 0x7feb780ca7b0 2024-01-11T09:47:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.943+0000 7feb807ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb780cc4f0 con 0x7feb78159e80 2024-01-11T09:47:09.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.943+0000 7feb7dd9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feb780ca7b0 0x7feb780cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:09.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.943+0000 7feb7e59b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7feb780ca3e0 0x7feb780c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58912/0 (socket says 172.21.15.27:58912) 2024-01-11T09:47:09.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.943+0000 7feb7e59b700 1 -- 172.21.15.27:0/3400662680 learned_addr learned my addr 172.21.15.27:0/3400662680 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:09.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.943+0000 7feb7dd9a700 1 -- 172.21.15.27:0/3400662680 >> v2:172.21.15.77:3300/0 conn(0x7feb78159e80 msgr2=0x7feb7815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:09.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.943+0000 7feb7dd9a700 1 --2- 172.21.15.27:0/3400662680 >> v2:172.21.15.77:3300/0 conn(0x7feb78159e80 0x7feb7815dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.943+0000 7feb7dd9a700 1 -- 172.21.15.27:0/3400662680 >> v1:172.21.15.27:6789/0 conn(0x7feb780ca3e0 legacy=0x7feb780c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.943+0000 7feb7dd9a700 1 -- 172.21.15.27:0/3400662680 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7815a5d0 con 0x7feb780ca7b0 2024-01-11T09:47:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.943+0000 7feb7d599700 1 -- 172.21.15.27:0/3400662680 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414540664 0 0) 0x7feb7815d640 con 0x7feb780ca3e0 2024-01-11T09:47:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.944+0000 7feb7dd9a700 1 --2- 172.21.15.27:0/3400662680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb780ca7b0 0x7feb780cca20 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7feb68009910 tx=0x7feb680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8102e7b6476b6c7c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.944+0000 7feb7d599700 1 -- 172.21.15.27:0/3400662680 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6800b7e0 con 0x7feb780ca7b0 2024-01-11T09:47:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.944+0000 7feb7d599700 1 -- 172.21.15.27:0/3400662680 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6800b940 con 0x7feb780ca7b0 2024-01-11T09:47:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.944+0000 7feb7d599700 1 -- 172.21.15.27:0/3400662680 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6800bce0 con 0x7feb780ca7b0 2024-01-11T09:47:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.944+0000 7feb807ff700 1 -- 172.21.15.27:0/3400662680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb780ca7b0 msgr2=0x7feb780cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.944+0000 7feb807ff700 1 --2- 172.21.15.27:0/3400662680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb780ca7b0 0x7feb780cca20 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7feb68009910 tx=0x7feb680091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.945+0000 7feb807ff700 1 -- 172.21.15.27:0/3400662680 shutdown_connections 2024-01-11T09:47:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.945+0000 7feb807ff700 1 --2- 172.21.15.27:0/3400662680 >> v2:172.21.15.77:3300/0 conn(0x7feb78159e80 0x7feb7815dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.945+0000 7feb807ff700 1 --2- 172.21.15.27:0/3400662680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb780ca7b0 0x7feb780cca20 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.945+0000 7feb807ff700 1 -- 172.21.15.27:0/3400662680 >> 172.21.15.27:0/3400662680 conn(0x7feb780ce3e0 msgr2=0x7feb780ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.945+0000 7feb807ff700 1 -- 172.21.15.27:0/3400662680 shutdown_connections 2024-01-11T09:47:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.945+0000 7feb807ff700 1 -- 172.21.15.27:0/3400662680 wait complete. 2024-01-11T09:47:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.946+0000 7feb807ff700 1 Processor -- start 2024-01-11T09:47:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.946+0000 7feb807ff700 1 -- start start 2024-01-11T09:47:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.947+0000 7feb807ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb780ca3e0 0x7feb781512e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.947+0000 7feb807ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb780ca7b0 0x7feb780b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.947+0000 7feb807ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb78159e80 0x7feb780c3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.947+0000 7feb807ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb780c9320 con 0x7feb78159e80 2024-01-11T09:47:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.947+0000 7feb807ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb780b20d0 con 0x7feb780ca7b0 2024-01-11T09:47:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.947+0000 7feb807ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb780c8710 con 0x7feb780ca3e0 2024-01-11T09:47:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.947+0000 7feb7ed9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb78159e80 0x7feb780c3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.948+0000 7feb7ed9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb78159e80 0x7feb780c3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42818/0 (socket says 172.21.15.27:42818) 2024-01-11T09:47:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.948+0000 7feb7ed9c700 1 -- 172.21.15.27:0/1473961135 learned_addr learned my addr 172.21.15.27:0/1473961135 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.948+0000 7feb7e59b700 1 --2- 172.21.15.27:0/1473961135 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb780ca3e0 0x7feb781512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.948+0000 7feb7dd9a700 1 --2- 172.21.15.27:0/1473961135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb780ca7b0 0x7feb780b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.948+0000 7feb7ed9c700 1 -- 172.21.15.27:0/1473961135 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb780ca3e0 msgr2=0x7feb781512e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.948+0000 7feb7ed9c700 1 --2- 172.21.15.27:0/1473961135 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb780ca3e0 0x7feb781512e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.948+0000 7feb7ed9c700 1 -- 172.21.15.27:0/1473961135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb780ca7b0 msgr2=0x7feb780b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.948+0000 7feb7ed9c700 1 --2- 172.21.15.27:0/1473961135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb780ca7b0 0x7feb780b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.948+0000 7feb7ed9c700 1 -- 172.21.15.27:0/1473961135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb68009420 con 0x7feb78159e80 2024-01-11T09:47:09.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.948+0000 7feb7ed9c700 1 --2- 172.21.15.27:0/1473961135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb78159e80 0x7feb780c3680 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7feb7000bc30 tx=0x7feb70009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:09.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.948+0000 7feb6e7fc700 1 -- 172.21.15.27:0/1473961135 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb700092e0 con 0x7feb78159e80 2024-01-11T09:47:09.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.949+0000 7feb6e7fc700 1 -- 172.21.15.27:0/1473961135 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7000c630 con 0x7feb78159e80 2024-01-11T09:47:09.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.949+0000 7feb6e7fc700 1 -- 172.21.15.27:0/1473961135 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7000ca10 con 0x7feb78159e80 2024-01-11T09:47:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.949+0000 7feb807ff700 1 -- 172.21.15.27:0/1473961135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb780c89f0 con 0x7feb78159e80 2024-01-11T09:47:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.949+0000 7feb807ff700 1 -- 172.21.15.27:0/1473961135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb780c8fb0 con 0x7feb78159e80 2024-01-11T09:47:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.950+0000 7feb807ff700 1 -- 172.21.15.27:0/1473961135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb780c9490 con 0x7feb78159e80 2024-01-11T09:47:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.951+0000 7feb6e7fc700 1 -- 172.21.15.27:0/1473961135 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7feb70012070 con 0x7feb78159e80 2024-01-11T09:47:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.951+0000 7feb6e7fc700 1 --2- 172.21.15.27:0/1473961135 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7feb5c067170 0x7feb5c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.952+0000 7feb6e7fc700 1 -- 172.21.15.27:0/1473961135 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6689+0+0 (secure 0 0 0) 0x7feb70084850 con 0x7feb78159e80 2024-01-11T09:47:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.952+0000 7feb7e59b700 1 --2- 172.21.15.27:0/1473961135 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7feb5c067170 0x7feb5c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:09.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.953+0000 7feb7e59b700 1 --2- 172.21.15.27:0/1473961135 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7feb5c067170 0x7feb5c069620 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7feb74007fd0 tx=0x7feb74007d30 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:09.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:09.955+0000 7feb6e7fc700 1 -- 172.21.15.27:0/1473961135 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feb700513b0 con 0x7feb78159e80 2024-01-11T09:47:10.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.203+0000 7feb807ff700 1 -- 172.21.15.27:0/1473961135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb780718d0 con 0x7feb78159e80 2024-01-11T09:47:10.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.204+0000 7feb6e7fc700 1 -- 172.21.15.27:0/1473961135 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+4614 (secure 0 0 0) 0x7feb700549d0 con 0x7feb78159e80 2024-01-11T09:47:10.195 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:10.195 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":134,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1568813392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1568813392},{"type":"v1","addr":"172.21.15.77:6834","nonce":1568813392}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122},{"gid":9740,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3700058093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":3700058093},{"type":"v1","addr":"172.21.15.27:6835","nonce":3700058093}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134},{"gid":10969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1352491019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1352491019},{"type":"v1","addr":"172.21.15.77:6835","nonce":1352491019}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:46:39.483941+0000","modified":"2024-01-11T09:46:42.996786+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11331},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11331":{"gid":11331,"name":"a","rank":0,"incarnation":129,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6834/3477661846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3477661846},{"type":"v1","addr":"172.21.15.27:6834","nonce":3477661846}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","standby_count_wanted":0},"id":16}]} 2024-01-11T09:47:10.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.206+0000 7feb807ff700 1 -- 172.21.15.27:0/1473961135 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7feb5c067170 msgr2=0x7feb5c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:10.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.206+0000 7feb807ff700 1 --2- 172.21.15.27:0/1473961135 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7feb5c067170 0x7feb5c069620 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7feb74007fd0 tx=0x7feb74007d30 comp rx=0 tx=0).stop 2024-01-11T09:47:10.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.206+0000 7feb807ff700 1 -- 172.21.15.27:0/1473961135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb78159e80 msgr2=0x7feb780c3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:10.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.206+0000 7feb807ff700 1 --2- 172.21.15.27:0/1473961135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb78159e80 0x7feb780c3680 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7feb7000bc30 tx=0x7feb70009840 comp rx=0 tx=0).stop 2024-01-11T09:47:10.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.206+0000 7feb807ff700 1 -- 172.21.15.27:0/1473961135 shutdown_connections 2024-01-11T09:47:10.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.206+0000 7feb807ff700 1 --2- 172.21.15.27:0/1473961135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb78159e80 0x7feb780c3680 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:10.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.206+0000 7feb807ff700 1 --2- 172.21.15.27:0/1473961135 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feb780ca3e0 0x7feb781512e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:10.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.206+0000 7feb807ff700 1 --2- 172.21.15.27:0/1473961135 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7feb5c067170 0x7feb5c069620 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:10.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.206+0000 7feb807ff700 1 --2- 172.21.15.27:0/1473961135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feb780ca7b0 0x7feb780b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:10.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.206+0000 7feb807ff700 1 -- 172.21.15.27:0/1473961135 >> 172.21.15.27:0/1473961135 conn(0x7feb780ce3e0 msgr2=0x7feb780ce7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:10.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.207+0000 7feb807ff700 1 -- 172.21.15.27:0/1473961135 shutdown_connections 2024-01-11T09:47:10.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.207+0000 7feb807ff700 1 -- 172.21.15.27:0/1473961135 wait complete. 2024-01-11T09:47:10.199 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 134 2024-01-11T09:47:10.216 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:47:10.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.424+0000 7fe395dff700 1 Processor -- start 2024-01-11T09:47:10.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.424+0000 7fe395dff700 1 -- start start 2024-01-11T09:47:10.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.425+0000 7fe395dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe3900c7e70 0x7fe3900694d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:10.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.425+0000 7fe395dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe390069a10 0x7fe390161080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.425+0000 7fe395dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39006e3d0 con 0x7fe390151a20 2024-01-11T09:47:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.425+0000 7fe395dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe390069e90 con 0x7fe390069a10 2024-01-11T09:47:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.425+0000 7fe395dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39006db30 con 0x7fe3900c7e70 2024-01-11T09:47:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.426+0000 7fe38ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe390069a10 0x7fe390161080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.426+0000 7fe38effd700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe3900c7e70 0x7fe3900694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.426+0000 7fe38ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe390069a10 0x7fe390161080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42834/0 (socket says 172.21.15.27:42834) 2024-01-11T09:47:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.426+0000 7fe38ffff700 1 -- 172.21.15.27:0/3209342413 learned_addr learned my addr 172.21.15.27:0/3209342413 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.426+0000 7fe38f7fe700 1 --1- 172.21.15.27:0/3209342413 >> v1:172.21.15.77:6789/0 conn(0x7fe390151a20 0x7fe390158480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:46566/0 (socket says 172.21.15.27:46566) 2024-01-11T09:47:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.426+0000 7fe38effd700 1 -- 172.21.15.27:0/3209342413 >> v2:172.21.15.27:3300/0 conn(0x7fe390069a10 msgr2=0x7fe390161080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.426+0000 7fe38effd700 1 --2- 172.21.15.27:0/3209342413 >> v2:172.21.15.27:3300/0 conn(0x7fe390069a10 0x7fe390161080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.426+0000 7fe38effd700 1 -- 172.21.15.27:0/3209342413 >> v1:172.21.15.77:6789/0 conn(0x7fe390151a20 legacy=0x7fe390158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.426+0000 7fe38e7fc700 1 -- 172.21.15.27:0/3209342413 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364826444 0 0) 0x7fe380001160 con 0x7fe390151a20 2024-01-11T09:47:10.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.426+0000 7fe38effd700 1 -- 172.21.15.27:0/3209342413 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39006de10 con 0x7fe3900c7e70 2024-01-11T09:47:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.426+0000 7fe38ffff700 1 --2- 172.21.15.27:0/3209342413 >> v2:172.21.15.27:3300/0 conn(0x7fe390069a10 0x7fe390161080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.427+0000 7fe38effd700 1 --2- 172.21.15.27:0/3209342413 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe3900c7e70 0x7fe3900694d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe3840098e0 tx=0x7fe3840091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b47d0f342b07587 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.427+0000 7fe38e7fc700 1 -- 172.21.15.27:0/3209342413 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38400b640 con 0x7fe3900c7e70 2024-01-11T09:47:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.427+0000 7fe38e7fc700 1 -- 172.21.15.27:0/3209342413 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38400b7a0 con 0x7fe3900c7e70 2024-01-11T09:47:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.427+0000 7fe38e7fc700 1 -- 172.21.15.27:0/3209342413 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38400bb40 con 0x7fe3900c7e70 2024-01-11T09:47:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.427+0000 7fe395dff700 1 -- 172.21.15.27:0/3209342413 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe3900c7e70 msgr2=0x7fe3900694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.427+0000 7fe395dff700 1 --2- 172.21.15.27:0/3209342413 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe3900c7e70 0x7fe3900694d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe3840098e0 tx=0x7fe3840091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.428+0000 7fe395dff700 1 -- 172.21.15.27:0/3209342413 shutdown_connections 2024-01-11T09:47:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.428+0000 7fe395dff700 1 --2- 172.21.15.27:0/3209342413 >> v2:172.21.15.27:3300/0 conn(0x7fe390069a10 0x7fe390161080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.428+0000 7fe395dff700 1 --2- 172.21.15.27:0/3209342413 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe3900c7e70 0x7fe3900694d0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.428+0000 7fe395dff700 1 -- 172.21.15.27:0/3209342413 >> 172.21.15.27:0/3209342413 conn(0x7fe3900680a0 msgr2=0x7fe3900cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.428+0000 7fe395dff700 1 -- 172.21.15.27:0/3209342413 shutdown_connections 2024-01-11T09:47:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.428+0000 7fe395dff700 1 -- 172.21.15.27:0/3209342413 wait complete. 2024-01-11T09:47:10.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.429+0000 7fe395dff700 1 Processor -- start 2024-01-11T09:47:10.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.429+0000 7fe395dff700 1 -- start start 2024-01-11T09:47:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.430+0000 7fe395dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe390069a10 0x7fe39015a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.430+0000 7fe395dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3900c7e70 0x7fe39016dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.430+0000 7fe395dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe390151a20 0x7fe3900ae5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.430+0000 7fe395dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3900aeb00 con 0x7fe390069a10 2024-01-11T09:47:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.430+0000 7fe395dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39015a830 con 0x7fe3900c7e70 2024-01-11T09:47:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.430+0000 7fe395dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39015ab10 con 0x7fe390151a20 2024-01-11T09:47:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.430+0000 7fe38f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe390069a10 0x7fe39015a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.431+0000 7fe38f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe390069a10 0x7fe39015a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42850/0 (socket says 172.21.15.27:42850) 2024-01-11T09:47:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.431+0000 7fe38f7fe700 1 -- 172.21.15.27:0/1020953375 learned_addr learned my addr 172.21.15.27:0/1020953375 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.431+0000 7fe38effd700 1 --2- 172.21.15.27:0/1020953375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3900c7e70 0x7fe39016dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:10.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.431+0000 7fe38ffff700 1 --2- 172.21.15.27:0/1020953375 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe390151a20 0x7fe3900ae5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:10.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.431+0000 7fe38f7fe700 1 -- 172.21.15.27:0/1020953375 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe390151a20 msgr2=0x7fe3900ae5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:10.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.431+0000 7fe38f7fe700 1 --2- 172.21.15.27:0/1020953375 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe390151a20 0x7fe3900ae5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:10.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.431+0000 7fe38f7fe700 1 -- 172.21.15.27:0/1020953375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3900c7e70 msgr2=0x7fe39016dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:10.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.431+0000 7fe38f7fe700 1 --2- 172.21.15.27:0/1020953375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3900c7e70 0x7fe39016dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:10.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.431+0000 7fe38f7fe700 1 -- 172.21.15.27:0/1020953375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe388009ae0 con 0x7fe390069a10 2024-01-11T09:47:10.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.431+0000 7fe38effd700 1 --2- 172.21.15.27:0/1020953375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3900c7e70 0x7fe39016dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:10.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.431+0000 7fe38f7fe700 1 --2- 172.21.15.27:0/1020953375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe390069a10 0x7fe39015a170 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7fe38000c5e0 tx=0x7fe380001030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:10.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.431+0000 7fe3777fe700 1 -- 172.21.15.27:0/1020953375 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380005ac0 con 0x7fe390069a10 2024-01-11T09:47:10.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.431+0000 7fe3777fe700 1 -- 172.21.15.27:0/1020953375 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe380005c40 con 0x7fe390069a10 2024-01-11T09:47:10.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.431+0000 7fe395dff700 1 -- 172.21.15.27:0/1020953375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe384009420 con 0x7fe390069a10 2024-01-11T09:47:10.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.432+0000 7fe3777fe700 1 -- 172.21.15.27:0/1020953375 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3800070d0 con 0x7fe390069a10 2024-01-11T09:47:10.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.432+0000 7fe395dff700 1 -- 172.21.15.27:0/1020953375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39016d780 con 0x7fe390069a10 2024-01-11T09:47:10.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.433+0000 7fe395dff700 1 -- 172.21.15.27:0/1020953375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe39004c3c0 con 0x7fe390069a10 2024-01-11T09:47:10.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.434+0000 7fe3777fe700 1 -- 172.21.15.27:0/1020953375 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fe380007480 con 0x7fe390069a10 2024-01-11T09:47:10.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.435+0000 7fe3777fe700 1 --2- 172.21.15.27:0/1020953375 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe37c067170 0x7fe37c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:10.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.435+0000 7fe3777fe700 1 -- 172.21.15.27:0/1020953375 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6689+0+0 (secure 0 0 0) 0x7fe380083f90 con 0x7fe390069a10 2024-01-11T09:47:10.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.435+0000 7fe38effd700 1 --2- 172.21.15.27:0/1020953375 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe37c067170 0x7fe37c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:10.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.436+0000 7fe38effd700 1 --2- 172.21.15.27:0/1020953375 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe37c067170 0x7fe37c069620 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe3840093f0 tx=0x7fe384017000 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:10.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.439+0000 7fe3777fe700 1 -- 172.21.15.27:0/1020953375 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe380050b90 con 0x7fe390069a10 2024-01-11T09:47:10.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.646+0000 7fe395dff700 1 -- 172.21.15.27:0/1020953375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe39006e3d0 con 0x7fe390069a10 2024-01-11T09:47:10.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.647+0000 7fe3777fe700 1 -- 172.21.15.27:0/1020953375 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v244) v1 ==== 74+0+16739 (secure 0 0 0) 0x7fe39006e3d0 con 0x7fe390069a10 2024-01-11T09:47:10.639 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:10.639 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":244,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:47:06.507632+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-01-11T09:46:37.665603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"238","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-01-11T09:46:38.472134+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"239","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/321681407":"2024-01-12T09:47:02.847264+0000","172.21.15.77:0/463475654":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/1391607686":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/2619336301":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/1120758885":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/1857056349":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/210673663":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/3835722213":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/696729061":"2024-01-12T09:46:54.199949+0000","172.21.15.27:6833/321681407":"2024-01-12T09:47:02.847264+0000","172.21.15.77:0/206349247":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/228357086":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/3326555344":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/3398849669":"2024-01-12T09:46:54.199949+0000","172.21.15.77:6837/59776":"2024-01-12T09:46:54.199949+0000","172.21.15.77:6836/59776":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/1934793537":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/842931202":"2024-01-12T09:47:02.495942+0000","172.21.15.77:0/1366958778":"2024-01-12T09:46:54.199949+0000","172.21.15.77:0/198451994":"2024-01-12T09:46:54.199949+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:47:10.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.649+0000 7fe395dff700 1 -- 172.21.15.27:0/1020953375 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe37c067170 msgr2=0x7fe37c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:10.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.649+0000 7fe395dff700 1 --2- 172.21.15.27:0/1020953375 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe37c067170 0x7fe37c069620 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe3840093f0 tx=0x7fe384017000 comp rx=0 tx=0).stop 2024-01-11T09:47:10.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.649+0000 7fe395dff700 1 -- 172.21.15.27:0/1020953375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe390069a10 msgr2=0x7fe39015a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:10.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.649+0000 7fe395dff700 1 --2- 172.21.15.27:0/1020953375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe390069a10 0x7fe39015a170 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7fe38000c5e0 tx=0x7fe380001030 comp rx=0 tx=0).stop 2024-01-11T09:47:10.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.649+0000 7fe395dff700 1 -- 172.21.15.27:0/1020953375 shutdown_connections 2024-01-11T09:47:10.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.649+0000 7fe395dff700 1 --2- 172.21.15.27:0/1020953375 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe390151a20 0x7fe3900ae5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:10.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.649+0000 7fe395dff700 1 --2- 172.21.15.27:0/1020953375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe390069a10 0x7fe39015a170 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:10.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.649+0000 7fe395dff700 1 --2- 172.21.15.27:0/1020953375 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe37c067170 0x7fe37c069620 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:10.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.649+0000 7fe395dff700 1 --2- 172.21.15.27:0/1020953375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3900c7e70 0x7fe39016dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:10.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.649+0000 7fe395dff700 1 -- 172.21.15.27:0/1020953375 >> 172.21.15.27:0/1020953375 conn(0x7fe3900680a0 msgr2=0x7fe3900ca2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:10.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.650+0000 7fe395dff700 1 -- 172.21.15.27:0/1020953375 shutdown_connections 2024-01-11T09:47:10.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.650+0000 7fe395dff700 1 -- 172.21.15.27:0/1020953375 wait complete. 2024-01-11T09:47:10.659 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2024-01-11T09:47:10.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.869+0000 7f122b3ff700 1 Processor -- start 2024-01-11T09:47:10.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.869+0000 7f122b3ff700 1 -- start start 2024-01-11T09:47:10.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.870+0000 7f122b3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1224156ab0 0x7f12240ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:10.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.870+0000 7f122b3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f12240b0550 0x7f1224160070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:10.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.870+0000 7f122b3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12240adc80 con 0x7f12240cc9b0 2024-01-11T09:47:10.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.870+0000 7f122b3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12240adf60 con 0x7f12240b0550 2024-01-11T09:47:10.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.870+0000 7f122b3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12240ae240 con 0x7f1224156ab0 2024-01-11T09:47:10.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.870+0000 7f122999c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f12240b0550 0x7f1224160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:10.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.871+0000 7f122899a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f12240cc9b0 0x7f1224156e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58920/0 (socket says 172.21.15.27:58920) 2024-01-11T09:47:10.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.871+0000 7f122899a700 1 -- 172.21.15.27:0/4105410872 learned_addr learned my addr 172.21.15.27:0/4105410872 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:10.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.871+0000 7f122919b700 1 --2- 172.21.15.27:0/4105410872 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1224156ab0 0x7f12240ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:10.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.871+0000 7f122999c700 1 -- 172.21.15.27:0/4105410872 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1224156ab0 msgr2=0x7f12240ad650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:10.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.871+0000 7f122999c700 1 --2- 172.21.15.27:0/4105410872 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1224156ab0 0x7f12240ad650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:10.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.871+0000 7f122999c700 1 -- 172.21.15.27:0/4105410872 >> v1:172.21.15.27:6789/0 conn(0x7f12240cc9b0 legacy=0x7f1224156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:10.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.871+0000 7f122999c700 1 -- 172.21.15.27:0/4105410872 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12240ae520 con 0x7f12240b0550 2024-01-11T09:47:10.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.871+0000 7f122919b700 1 --2- 172.21.15.27:0/4105410872 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1224156ab0 0x7f12240ad650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:10.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.871+0000 7f121bfff700 1 -- 172.21.15.27:0/4105410872 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597708825 0 0) 0x7f12240adc80 con 0x7f12240cc9b0 2024-01-11T09:47:10.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.871+0000 7f122999c700 1 --2- 172.21.15.27:0/4105410872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12240b0550 0x7f1224160070 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f121c009e60 tx=0x7f121c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b54235c547d524 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:10.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.872+0000 7f121bfff700 1 -- 172.21.15.27:0/4105410872 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c00e070 con 0x7f12240b0550 2024-01-11T09:47:10.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.872+0000 7f121bfff700 1 -- 172.21.15.27:0/4105410872 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121c00bc30 con 0x7f12240b0550 2024-01-11T09:47:10.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.872+0000 7f121bfff700 1 -- 172.21.15.27:0/4105410872 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c012430 con 0x7f12240b0550 2024-01-11T09:47:10.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.872+0000 7f122b3ff700 1 -- 172.21.15.27:0/4105410872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12240b0550 msgr2=0x7f1224160070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:10.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.872+0000 7f122b3ff700 1 --2- 172.21.15.27:0/4105410872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12240b0550 0x7f1224160070 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f121c009e60 tx=0x7f121c009780 comp rx=0 tx=0).stop 2024-01-11T09:47:10.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.872+0000 7f122b3ff700 1 -- 172.21.15.27:0/4105410872 shutdown_connections 2024-01-11T09:47:10.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.872+0000 7f122b3ff700 1 --2- 172.21.15.27:0/4105410872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12240b0550 0x7f1224160070 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:10.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.872+0000 7f122b3ff700 1 --2- 172.21.15.27:0/4105410872 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1224156ab0 0x7f12240ad650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:10.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.872+0000 7f122b3ff700 1 -- 172.21.15.27:0/4105410872 >> 172.21.15.27:0/4105410872 conn(0x7f12240680b0 msgr2=0x7f12240cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:10.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.872+0000 7f122b3ff700 1 -- 172.21.15.27:0/4105410872 shutdown_connections 2024-01-11T09:47:10.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.872+0000 7f122b3ff700 1 -- 172.21.15.27:0/4105410872 wait complete. 2024-01-11T09:47:10.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.874+0000 7f122b3ff700 1 Processor -- start 2024-01-11T09:47:10.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.874+0000 7f122b3ff700 1 -- start start 2024-01-11T09:47:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.875+0000 7f122b3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12240b0550 0x7f12240c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.875+0000 7f122b3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12240cc9b0 0x7f122406e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.875+0000 7f122b3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1224156ab0 0x7f1224140b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.875+0000 7f122b3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122406ec70 con 0x7f12240cc9b0 2024-01-11T09:47:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.875+0000 7f122b3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12241572a0 con 0x7f12240b0550 2024-01-11T09:47:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.875+0000 7f122b3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12240c94b0 con 0x7f1224156ab0 2024-01-11T09:47:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.875+0000 7f122899a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12240cc9b0 0x7f122406e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.875+0000 7f122899a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12240cc9b0 0x7f122406e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42882/0 (socket says 172.21.15.27:42882) 2024-01-11T09:47:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.875+0000 7f122899a700 1 -- 172.21.15.27:0/744044005 learned_addr learned my addr 172.21.15.27:0/744044005 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.876+0000 7f122999c700 1 --2- 172.21.15.27:0/744044005 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1224156ab0 0x7f1224140b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.876+0000 7f122919b700 1 --2- 172.21.15.27:0/744044005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12240b0550 0x7f12240c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:10.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.876+0000 7f122899a700 1 -- 172.21.15.27:0/744044005 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1224156ab0 msgr2=0x7f1224140b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:10.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.876+0000 7f122899a700 1 --2- 172.21.15.27:0/744044005 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1224156ab0 0x7f1224140b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:10.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.876+0000 7f122899a700 1 -- 172.21.15.27:0/744044005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12240b0550 msgr2=0x7f12240c8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:10.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.876+0000 7f122899a700 1 --2- 172.21.15.27:0/744044005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12240b0550 0x7f12240c8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:10.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.876+0000 7f122899a700 1 -- 172.21.15.27:0/744044005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121c009a20 con 0x7f12240cc9b0 2024-01-11T09:47:10.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.876+0000 7f122919b700 1 --2- 172.21.15.27:0/744044005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12240b0550 0x7f12240c8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:10.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.876+0000 7f122899a700 1 --2- 172.21.15.27:0/744044005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12240cc9b0 0x7f122406e730 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f121400c5e0 tx=0x7f1214001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:10.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.876+0000 7f12197fa700 1 -- 172.21.15.27:0/744044005 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1214005b30 con 0x7f12240cc9b0 2024-01-11T09:47:10.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.876+0000 7f12197fa700 1 -- 172.21.15.27:0/744044005 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1214005c90 con 0x7f12240cc9b0 2024-01-11T09:47:10.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.876+0000 7f12197fa700 1 -- 172.21.15.27:0/744044005 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12140072e0 con 0x7f12240cc9b0 2024-01-11T09:47:10.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.876+0000 7f122b3ff700 1 -- 172.21.15.27:0/744044005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12240c9790 con 0x7f12240cc9b0 2024-01-11T09:47:10.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.877+0000 7f122b3ff700 1 -- 172.21.15.27:0/744044005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12240c9d50 con 0x7f12240cc9b0 2024-01-11T09:47:10.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.878+0000 7f122b3ff700 1 -- 172.21.15.27:0/744044005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12241573e0 con 0x7f12240cc9b0 2024-01-11T09:47:10.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.878+0000 7f12197fa700 1 -- 172.21.15.27:0/744044005 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f1214012400 con 0x7f12240cc9b0 2024-01-11T09:47:10.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.879+0000 7f12197fa700 1 --2- 172.21.15.27:0/744044005 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1208067140 0x7f12080695f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:10.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.879+0000 7f12197fa700 1 -- 172.21.15.27:0/744044005 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f1214084150 con 0x7f12240cc9b0 2024-01-11T09:47:10.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.880+0000 7f122919b700 1 --2- 172.21.15.27:0/744044005 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1208067140 0x7f12080695f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:10.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.880+0000 7f122919b700 1 --2- 172.21.15.27:0/744044005 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1208067140 0x7f12080695f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1220008b20 tx=0x7f1220008040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:10.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:10.883+0000 7f12197fa700 1 -- 172.21.15.27:0/744044005 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f121404b5a0 con 0x7f12240cc9b0 2024-01-11T09:47:11.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.139+0000 7f122b3ff700 1 -- 172.21.15.27:0/744044005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7f122404c3c0 con 0x7f12240cc9b0 2024-01-11T09:47:11.131 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-11T09:47:11.140+0000 7f27065f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-11T09:47:11.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.144+0000 7f12197fa700 1 -- 172.21.15.27:0/744044005 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v135) v1 ==== 167+0+0 (secure 0 0 0) 0x7f12140547f0 con 0x7f12240cc9b0 2024-01-11T09:47:11.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.148+0000 7f122b3ff700 1 -- 172.21.15.27:0/744044005 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1208067140 msgr2=0x7f12080695f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:11.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.148+0000 7f122b3ff700 1 --2- 172.21.15.27:0/744044005 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1208067140 0x7f12080695f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1220008b20 tx=0x7f1220008040 comp rx=0 tx=0).stop 2024-01-11T09:47:11.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.148+0000 7f122b3ff700 1 -- 172.21.15.27:0/744044005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12240cc9b0 msgr2=0x7f122406e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:11.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.148+0000 7f122b3ff700 1 --2- 172.21.15.27:0/744044005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12240cc9b0 0x7f122406e730 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f121400c5e0 tx=0x7f1214001070 comp rx=0 tx=0).stop 2024-01-11T09:47:11.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.148+0000 7f122b3ff700 1 -- 172.21.15.27:0/744044005 shutdown_connections 2024-01-11T09:47:11.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.148+0000 7f122b3ff700 1 --2- 172.21.15.27:0/744044005 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f1224156ab0 0x7f1224140b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:11.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.148+0000 7f122b3ff700 1 --2- 172.21.15.27:0/744044005 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f1208067140 0x7f12080695f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:11.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.148+0000 7f122b3ff700 1 --2- 172.21.15.27:0/744044005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12240b0550 0x7f12240c8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:11.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.148+0000 7f122b3ff700 1 --2- 172.21.15.27:0/744044005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12240cc9b0 0x7f122406e730 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:11.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.148+0000 7f122b3ff700 1 -- 172.21.15.27:0/744044005 >> 172.21.15.27:0/744044005 conn(0x7f12240680b0 msgr2=0x7f12240c8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:11.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.148+0000 7f122b3ff700 1 -- 172.21.15.27:0/744044005 shutdown_connections 2024-01-11T09:47:11.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.148+0000 7f122b3ff700 1 -- 172.21.15.27:0/744044005 wait complete. 2024-01-11T09:47:11.141 INFO:teuthology.orchestra.run.smithi027.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-11T09:47:11.144 INFO:tasks.ceph.mds.a.smithi027.stderr: -1689> 2024-01-11T09:46:41.205+0000 7fc9719f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T09:47:11.163 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2024-01-11T09:47:11.190 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T09:47:11.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.377+0000 7f6f341ff700 1 Processor -- start 2024-01-11T09:47:11.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.378+0000 7f6f341ff700 1 -- start start 2024-01-11T09:47:11.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.379+0000 7f6f341ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f2c1577e0 0x7f6f2c0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:11.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.379+0000 7f6f341ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6f2c069540 0x7f6f2c0ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:11.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.379+0000 7f6f341ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2c0699d0 con 0x7f6f2c157410 2024-01-11T09:47:11.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.379+0000 7f6f341ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f2c069cb0 con 0x7f6f2c069540 2024-01-11T09:47:11.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.379+0000 7f6f341ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c069f90 con 0x7f6f2c1577e0 2024-01-11T09:47:11.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.379+0000 7f6f3179a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f2c1577e0 0x7f6f2c0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:11.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.379+0000 7f6f3279c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6f2c069540 0x7f6f2c0ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:11.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.379+0000 7f6f3179a700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f2c1577e0 0x7f6f2c0ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:50408/0 (socket says 172.21.15.27:50408) 2024-01-11T09:47:11.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.379+0000 7f6f31f9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6f2c157410 0x7f6f2c0ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:46586/0 (socket says 172.21.15.27:46586) 2024-01-11T09:47:11.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.379+0000 7f6f3179a700 1 -- 172.21.15.27:0/1441579184 learned_addr learned my addr 172.21.15.27:0/1441579184 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:11.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.380+0000 7f6f30f99700 1 -- 172.21.15.27:0/1441579184 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148840800 0 0) 0x7f6f2c0699d0 con 0x7f6f2c157410 2024-01-11T09:47:11.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.380+0000 7f6f3279c700 1 -- 172.21.15.27:0/1441579184 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f2c1577e0 msgr2=0x7f6f2c0ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:11.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.380+0000 7f6f3279c700 1 --2- 172.21.15.27:0/1441579184 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f2c1577e0 0x7f6f2c0ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:11.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.380+0000 7f6f3279c700 1 -- 172.21.15.27:0/1441579184 >> v1:172.21.15.77:6789/0 conn(0x7f6f2c157410 legacy=0x7f6f2c0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:11.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.380+0000 7f6f3279c700 1 -- 172.21.15.27:0/1441579184 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2c06a270 con 0x7f6f2c069540 2024-01-11T09:47:11.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.380+0000 7f6f3179a700 1 --2- 172.21.15.27:0/1441579184 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f2c1577e0 0x7f6f2c0ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:11.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.380+0000 7f6f3279c700 1 --2- 172.21.15.27:0/1441579184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f2c069540 0x7f6f2c0ae540 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f6f24009e60 tx=0x7f6f24009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5830e75484847cfc server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:11.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.381+0000 7f6f30f99700 1 -- 172.21.15.27:0/1441579184 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2400e070 con 0x7f6f2c069540 2024-01-11T09:47:11.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.381+0000 7f6f30f99700 1 -- 172.21.15.27:0/1441579184 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2400bb60 con 0x7f6f2c069540 2024-01-11T09:47:11.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.381+0000 7f6f30f99700 1 -- 172.21.15.27:0/1441579184 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f24012430 con 0x7f6f2c069540 2024-01-11T09:47:11.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.381+0000 7f6f341ff700 1 -- 172.21.15.27:0/1441579184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f2c069540 msgr2=0x7f6f2c0ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:11.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.381+0000 7f6f341ff700 1 --2- 172.21.15.27:0/1441579184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f2c069540 0x7f6f2c0ae540 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f6f24009e60 tx=0x7f6f24009780 comp rx=0 tx=0).stop 2024-01-11T09:47:11.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.381+0000 7f6f341ff700 1 -- 172.21.15.27:0/1441579184 shutdown_connections 2024-01-11T09:47:11.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.381+0000 7f6f341ff700 1 --2- 172.21.15.27:0/1441579184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f2c069540 0x7f6f2c0ae540 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:11.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.381+0000 7f6f341ff700 1 --2- 172.21.15.27:0/1441579184 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f2c1577e0 0x7f6f2c0ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:11.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.381+0000 7f6f341ff700 1 -- 172.21.15.27:0/1441579184 >> 172.21.15.27:0/1441579184 conn(0x7f6f2c0680a0 msgr2=0x7f6f2c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:11.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.381+0000 7f6f341ff700 1 -- 172.21.15.27:0/1441579184 shutdown_connections 2024-01-11T09:47:11.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.381+0000 7f6f341ff700 1 -- 172.21.15.27:0/1441579184 wait complete. 2024-01-11T09:47:11.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.383+0000 7f6f341ff700 1 Processor -- start 2024-01-11T09:47:11.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.383+0000 7f6f341ff700 1 -- start start 2024-01-11T09:47:11.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.384+0000 7f6f341ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f2c069540 0x7f6f2c153980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:11.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.384+0000 7f6f341ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f2c157410 0x7f6f2c1448f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:11.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.384+0000 7f6f341ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f2c1577e0 0x7f6f2c144e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:11.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.384+0000 7f6f341ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c145370 con 0x7f6f2c157410 2024-01-11T09:47:11.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.384+0000 7f6f341ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c0b08e0 con 0x7f6f2c069540 2024-01-11T09:47:11.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.384+0000 7f6f341ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c156c40 con 0x7f6f2c1577e0 2024-01-11T09:47:11.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.384+0000 7f6f3179a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f2c157410 0x7f6f2c1448f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:11.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.384+0000 7f6f3179a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f2c157410 0x7f6f2c1448f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42922/0 (socket says 172.21.15.27:42922) 2024-01-11T09:47:11.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.384+0000 7f6f3179a700 1 -- 172.21.15.27:0/2234172767 learned_addr learned my addr 172.21.15.27:0/2234172767 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:11.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.385+0000 7f6f3279c700 1 --2- 172.21.15.27:0/2234172767 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f2c1577e0 0x7f6f2c144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:11.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.385+0000 7f6f31f9b700 1 --2- 172.21.15.27:0/2234172767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f2c069540 0x7f6f2c153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:11.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.385+0000 7f6f3179a700 1 -- 172.21.15.27:0/2234172767 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f2c1577e0 msgr2=0x7f6f2c144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:11.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.385+0000 7f6f3179a700 1 --2- 172.21.15.27:0/2234172767 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f2c1577e0 0x7f6f2c144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:11.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.385+0000 7f6f3179a700 1 -- 172.21.15.27:0/2234172767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f2c069540 msgr2=0x7f6f2c153980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:11.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.385+0000 7f6f3179a700 1 --2- 172.21.15.27:0/2234172767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f2c069540 0x7f6f2c153980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:11.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.385+0000 7f6f3179a700 1 -- 172.21.15.27:0/2234172767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f1c0094f0 con 0x7f6f2c157410 2024-01-11T09:47:11.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.385+0000 7f6f3279c700 1 --2- 172.21.15.27:0/2234172767 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f2c1577e0 0x7f6f2c144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:11.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.385+0000 7f6f31f9b700 1 --2- 172.21.15.27:0/2234172767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f2c069540 0x7f6f2c153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:11.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.385+0000 7f6f3179a700 1 --2- 172.21.15.27:0/2234172767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f2c157410 0x7f6f2c1448f0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c00ec50 tx=0x7f6f1c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:11.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.385+0000 7f6f21ffb700 1 -- 172.21.15.27:0/2234172767 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c00c9d0 con 0x7f6f2c157410 2024-01-11T09:47:11.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.385+0000 7f6f21ffb700 1 -- 172.21.15.27:0/2234172767 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f1c00cb30 con 0x7f6f2c157410 2024-01-11T09:47:11.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.385+0000 7f6f21ffb700 1 -- 172.21.15.27:0/2234172767 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c013470 con 0x7f6f2c157410 2024-01-11T09:47:11.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.385+0000 7f6f341ff700 1 -- 172.21.15.27:0/2234172767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f24009a20 con 0x7f6f2c157410 2024-01-11T09:47:11.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.386+0000 7f6f341ff700 1 -- 172.21.15.27:0/2234172767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f2c0b0290 con 0x7f6f2c157410 2024-01-11T09:47:11.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.387+0000 7f6f341ff700 1 -- 172.21.15.27:0/2234172767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f2c0b0a50 con 0x7f6f2c157410 2024-01-11T09:47:11.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.388+0000 7f6f21ffb700 1 -- 172.21.15.27:0/2234172767 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f6f1c013820 con 0x7f6f2c157410 2024-01-11T09:47:11.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.388+0000 7f6f21ffb700 1 --2- 172.21.15.27:0/2234172767 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6f10067170 0x7f6f10069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:11.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.388+0000 7f6f21ffb700 1 -- 172.21.15.27:0/2234172767 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6775+0+0 (secure 0 0 0) 0x7f6f1c085120 con 0x7f6f2c157410 2024-01-11T09:47:11.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.389+0000 7f6f31f9b700 1 --2- 172.21.15.27:0/2234172767 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6f10067170 0x7f6f10069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:11.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.389+0000 7f6f31f9b700 1 --2- 172.21.15.27:0/2234172767 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6f10067170 0x7f6f10069620 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6f28000fb0 tx=0x7f6f28007c40 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:11.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.393+0000 7f6f21ffb700 1 -- 172.21.15.27:0/2234172767 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6f1c051be0 con 0x7f6f2c157410 2024-01-11T09:47:11.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:11.640+0000 7f6f341ff700 1 -- 172.21.15.27:0/2234172767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6f2c04c3c0 con 0x7f6f2c157410 2024-01-11T09:47:12.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.145+0000 7f6f21ffb700 1 -- 172.21.15.27:0/2234172767 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v136) v1 ==== 107+0+0 (secure 0 0 0) 0x7f6f1c055200 con 0x7f6f2c157410 2024-01-11T09:47:12.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.148+0000 7f6f341ff700 1 -- 172.21.15.27:0/2234172767 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6f10067170 msgr2=0x7f6f10069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:12.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.148+0000 7f6f341ff700 1 --2- 172.21.15.27:0/2234172767 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6f10067170 0x7f6f10069620 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6f28000fb0 tx=0x7f6f28007c40 comp rx=0 tx=0).stop 2024-01-11T09:47:12.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.148+0000 7f6f341ff700 1 -- 172.21.15.27:0/2234172767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f2c157410 msgr2=0x7f6f2c1448f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:12.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.148+0000 7f6f341ff700 1 --2- 172.21.15.27:0/2234172767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f2c157410 0x7f6f2c1448f0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c00ec50 tx=0x7f6f1c009250 comp rx=0 tx=0).stop 2024-01-11T09:47:12.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.148+0000 7f6f341ff700 1 -- 172.21.15.27:0/2234172767 shutdown_connections 2024-01-11T09:47:12.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.148+0000 7f6f341ff700 1 --2- 172.21.15.27:0/2234172767 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6f2c1577e0 0x7f6f2c144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:12.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.148+0000 7f6f341ff700 1 --2- 172.21.15.27:0/2234172767 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6f10067170 0x7f6f10069620 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:12.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.148+0000 7f6f341ff700 1 --2- 172.21.15.27:0/2234172767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f2c069540 0x7f6f2c153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:12.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.148+0000 7f6f341ff700 1 --2- 172.21.15.27:0/2234172767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f2c157410 0x7f6f2c1448f0 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:12.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.148+0000 7f6f341ff700 1 -- 172.21.15.27:0/2234172767 >> 172.21.15.27:0/2234172767 conn(0x7f6f2c0680a0 msgr2=0x7f6f2c0c1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:12.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.148+0000 7f6f341ff700 1 -- 172.21.15.27:0/2234172767 shutdown_connections 2024-01-11T09:47:12.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.149+0000 7f6f341ff700 1 -- 172.21.15.27:0/2234172767 wait complete. 2024-01-11T09:47:12.159 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2024-01-11T09:47:12.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.414+0000 7f35e09ff700 1 Processor -- start 2024-01-11T09:47:12.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.414+0000 7f35e09ff700 1 -- start start 2024-01-11T09:47:12.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.416+0000 7f35e09ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35dc15a7a0 0x7f35dc15c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:12.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.416+0000 7f35e09ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35dc159e60 0x7f35dc15aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:12.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.416+0000 7f35e09ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35dc15c4b0 con 0x7f35dc15b400 2024-01-11T09:47:12.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.416+0000 7f35e09ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35dc15b8a0 con 0x7f35dc159e60 2024-01-11T09:47:12.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.416+0000 7f35e09ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc15bb80 con 0x7f35dc15a7a0 2024-01-11T09:47:12.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.416+0000 7f35d9d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35dc159e60 0x7f35dc15aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:12.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.416+0000 7f35d9d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35dc159e60 0x7f35dc15aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42926/0 (socket says 172.21.15.27:42926) 2024-01-11T09:47:12.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.416+0000 7f35d9d9b700 1 -- 172.21.15.27:0/2288001904 learned_addr learned my addr 172.21.15.27:0/2288001904 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:12.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.416+0000 7f35da59c700 1 --2- 172.21.15.27:0/2288001904 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35dc15a7a0 0x7f35dc15c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:12.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.416+0000 7f35d9d9b700 1 -- 172.21.15.27:0/2288001904 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35dc15a7a0 msgr2=0x7f35dc15c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:12.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.416+0000 7f35d9d9b700 1 --2- 172.21.15.27:0/2288001904 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35dc15a7a0 0x7f35dc15c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:12.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.416+0000 7f35d9d9b700 1 -- 172.21.15.27:0/2288001904 >> v1:172.21.15.77:6789/0 conn(0x7f35dc15b400 legacy=0x7f35dc0685e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:12.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.417+0000 7f35d9d9b700 1 -- 172.21.15.27:0/2288001904 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35dc15be60 con 0x7f35dc159e60 2024-01-11T09:47:12.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.417+0000 7f35d9d9b700 1 --2- 172.21.15.27:0/2288001904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35dc159e60 0x7f35dc15aec0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f35d00099b0 tx=0x7f35d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9110672353b3f9f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:12.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.417+0000 7f35d959a700 1 -- 172.21.15.27:0/2288001904 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d000b860 con 0x7f35dc159e60 2024-01-11T09:47:12.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.417+0000 7f35d959a700 1 -- 172.21.15.27:0/2288001904 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d000b9c0 con 0x7f35dc159e60 2024-01-11T09:47:12.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.417+0000 7f35d959a700 1 -- 172.21.15.27:0/2288001904 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d000bd60 con 0x7f35dc159e60 2024-01-11T09:47:12.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.418+0000 7f35e09ff700 1 -- 172.21.15.27:0/2288001904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35dc159e60 msgr2=0x7f35dc15aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:12.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.418+0000 7f35e09ff700 1 --2- 172.21.15.27:0/2288001904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35dc159e60 0x7f35dc15aec0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f35d00099b0 tx=0x7f35d0009250 comp rx=0 tx=0).stop 2024-01-11T09:47:12.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.418+0000 7f35e09ff700 1 -- 172.21.15.27:0/2288001904 shutdown_connections 2024-01-11T09:47:12.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.418+0000 7f35e09ff700 1 --2- 172.21.15.27:0/2288001904 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35dc15a7a0 0x7f35dc15c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:12.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.418+0000 7f35e09ff700 1 --2- 172.21.15.27:0/2288001904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35dc159e60 0x7f35dc15aec0 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:12.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.418+0000 7f35e09ff700 1 -- 172.21.15.27:0/2288001904 >> 172.21.15.27:0/2288001904 conn(0x7f35dc06ed40 msgr2=0x7f35dc06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:12.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.418+0000 7f35e09ff700 1 -- 172.21.15.27:0/2288001904 shutdown_connections 2024-01-11T09:47:12.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.418+0000 7f35e09ff700 1 -- 172.21.15.27:0/2288001904 wait complete. 2024-01-11T09:47:12.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.419+0000 7f35e09ff700 1 Processor -- start 2024-01-11T09:47:12.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.420+0000 7f35e09ff700 1 -- start start 2024-01-11T09:47:12.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.420+0000 7f35e09ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35dc15a7a0 0x7f35dc0b2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:12.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.421+0000 7f35e09ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35dc15b400 0x7f35dc145510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:12.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.421+0000 7f35e09ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35dc16dce0 0x7f35dc157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:12.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.421+0000 7f35e09ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc0b2e60 con 0x7f35dc16dce0 2024-01-11T09:47:12.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.421+0000 7f35e09ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc16a930 con 0x7f35dc15b400 2024-01-11T09:47:12.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.421+0000 7f35e09ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc169aa0 con 0x7f35dc15a7a0 2024-01-11T09:47:12.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.421+0000 7f35dad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35dc16dce0 0x7f35dc157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:12.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.421+0000 7f35dad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35dc16dce0 0x7f35dc157910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42938/0 (socket says 172.21.15.27:42938) 2024-01-11T09:47:12.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.421+0000 7f35dad9d700 1 -- 172.21.15.27:0/3410104220 learned_addr learned my addr 172.21.15.27:0/3410104220 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:12.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.421+0000 7f35da59c700 1 --2- 172.21.15.27:0/3410104220 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35dc15a7a0 0x7f35dc0b2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:12.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.421+0000 7f35d9d9b700 1 --2- 172.21.15.27:0/3410104220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35dc15b400 0x7f35dc145510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:12.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.421+0000 7f35dad9d700 1 -- 172.21.15.27:0/3410104220 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35dc15a7a0 msgr2=0x7f35dc0b2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:12.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.421+0000 7f35dad9d700 1 --2- 172.21.15.27:0/3410104220 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35dc15a7a0 0x7f35dc0b2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:12.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.421+0000 7f35dad9d700 1 -- 172.21.15.27:0/3410104220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35dc15b400 msgr2=0x7f35dc145510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:12.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.421+0000 7f35dad9d700 1 --2- 172.21.15.27:0/3410104220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35dc15b400 0x7f35dc145510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:12.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.421+0000 7f35dad9d700 1 -- 172.21.15.27:0/3410104220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d00094f0 con 0x7f35dc16dce0 2024-01-11T09:47:12.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.421+0000 7f35d9d9b700 1 --2- 172.21.15.27:0/3410104220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35dc15b400 0x7f35dc145510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:12.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.422+0000 7f35dad9d700 1 --2- 172.21.15.27:0/3410104220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35dc16dce0 0x7f35dc157910 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f35d400c860 tx=0x7f35d4001690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:12.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.422+0000 7f35ca7fc700 1 -- 172.21.15.27:0/3410104220 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d4012630 con 0x7f35dc16dce0 2024-01-11T09:47:12.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.422+0000 7f35ca7fc700 1 -- 172.21.15.27:0/3410104220 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d4012790 con 0x7f35dc16dce0 2024-01-11T09:47:12.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.422+0000 7f35ca7fc700 1 -- 172.21.15.27:0/3410104220 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d4012b50 con 0x7f35dc16dce0 2024-01-11T09:47:12.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.422+0000 7f35e09ff700 1 -- 172.21.15.27:0/3410104220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35dc169d20 con 0x7f35dc16dce0 2024-01-11T09:47:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.422+0000 7f35e09ff700 1 -- 172.21.15.27:0/3410104220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35dc16a2e0 con 0x7f35dc16dce0 2024-01-11T09:47:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.423+0000 7f35e09ff700 1 -- 172.21.15.27:0/3410104220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35dc04c3c0 con 0x7f35dc16dce0 2024-01-11T09:47:12.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.424+0000 7f35ca7fc700 1 -- 172.21.15.27:0/3410104220 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f35d401a3a0 con 0x7f35dc16dce0 2024-01-11T09:47:12.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.425+0000 7f35ca7fc700 1 --2- 172.21.15.27:0/3410104220 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f35bc067170 0x7f35bc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:12.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.425+0000 7f35ca7fc700 1 -- 172.21.15.27:0/3410104220 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6775+0+0 (secure 0 0 0) 0x7f35d4084f40 con 0x7f35dc16dce0 2024-01-11T09:47:12.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.425+0000 7f35da59c700 1 --2- 172.21.15.27:0/3410104220 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f35bc067170 0x7f35bc069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:12.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.426+0000 7f35da59c700 1 --2- 172.21.15.27:0/3410104220 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f35bc067170 0x7f35bc069620 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f35cc006fd0 tx=0x7f35cc008040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:12.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.429+0000 7f35ca7fc700 1 -- 172.21.15.27:0/3410104220 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f35d4051a50 con 0x7f35dc16dce0 2024-01-11T09:47:12.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:12.632+0000 7f35e09ff700 1 -- 172.21.15.27:0/3410104220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f35dc15c140 con 0x7f35dc16dce0 2024-01-11T09:47:13.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.538+0000 7f35ca7fc700 1 -- 172.21.15.27:0/3410104220 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v246) v1 ==== 190+0+0 (secure 0 0 0) 0x7f35d4055070 con 0x7f35dc16dce0 2024-01-11T09:47:13.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.543+0000 7f35e09ff700 1 -- 172.21.15.27:0/3410104220 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f35bc067170 msgr2=0x7f35bc069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:13.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.543+0000 7f35e09ff700 1 --2- 172.21.15.27:0/3410104220 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f35bc067170 0x7f35bc069620 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f35cc006fd0 tx=0x7f35cc008040 comp rx=0 tx=0).stop 2024-01-11T09:47:13.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.544+0000 7f35e09ff700 1 -- 172.21.15.27:0/3410104220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35dc16dce0 msgr2=0x7f35dc157910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:13.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.544+0000 7f35e09ff700 1 --2- 172.21.15.27:0/3410104220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35dc16dce0 0x7f35dc157910 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f35d400c860 tx=0x7f35d4001690 comp rx=0 tx=0).stop 2024-01-11T09:47:13.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.544+0000 7f35e09ff700 1 -- 172.21.15.27:0/3410104220 shutdown_connections 2024-01-11T09:47:13.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.544+0000 7f35e09ff700 1 --2- 172.21.15.27:0/3410104220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35dc16dce0 0x7f35dc157910 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:13.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.544+0000 7f35e09ff700 1 --2- 172.21.15.27:0/3410104220 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f35dc15a7a0 0x7f35dc0b2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:13.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.544+0000 7f35e09ff700 1 --2- 172.21.15.27:0/3410104220 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f35bc067170 0x7f35bc069620 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:13.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.544+0000 7f35e09ff700 1 --2- 172.21.15.27:0/3410104220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35dc15b400 0x7f35dc145510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:13.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.544+0000 7f35e09ff700 1 -- 172.21.15.27:0/3410104220 >> 172.21.15.27:0/3410104220 conn(0x7f35dc06ed40 msgr2=0x7f35dc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:13.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.545+0000 7f35e09ff700 1 -- 172.21.15.27:0/3410104220 shutdown_connections 2024-01-11T09:47:13.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.545+0000 7f35e09ff700 1 -- 172.21.15.27:0/3410104220 wait complete. 2024-01-11T09:47:13.539 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'backup_fs_metadata' removed 2024-01-11T09:47:13.559 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2024-01-11T09:47:13.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.776+0000 7f793e7ff700 1 Processor -- start 2024-01-11T09:47:13.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.776+0000 7f793e7ff700 1 -- start start 2024-01-11T09:47:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.777+0000 7f793e7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79380b08a0 0x7f79380684d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.778+0000 7f793e7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7938068e10 con 0x7f7938159250 2024-01-11T09:47:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.778+0000 7f793e7ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f793815a440 con 0x7f7938068a10 2024-01-11T09:47:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.778+0000 7f793e7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f793815ace0 con 0x7f79380b08a0 2024-01-11T09:47:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.778+0000 7f7937fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7938159250 0x7f79380cb410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58932/0 (socket says 172.21.15.27:58932) 2024-01-11T09:47:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.778+0000 7f7937fff700 1 -- 172.21.15.27:0/3969151578 learned_addr learned my addr 172.21.15.27:0/3969151578 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.778+0000 7f79377fe700 1 --2- 172.21.15.27:0/3969151578 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79380b08a0 0x7f79380684d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:13.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.778+0000 7f7936ffd700 1 -- 172.21.15.27:0/3969151578 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167636784 0 0) 0x7f7938068e10 con 0x7f7938159250 2024-01-11T09:47:13.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.778+0000 7f7936ffd700 1 -- 172.21.15.27:0/3969151578 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7920003570 con 0x7f7938159250 2024-01-11T09:47:13.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.778+0000 7f7936ffd700 1 -- 172.21.15.27:0/3969151578 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799915002 0 0) 0x7f793815a440 con 0x7f7938068a10 2024-01-11T09:47:13.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.779+0000 7f7936ffd700 1 -- 172.21.15.27:0/3969151578 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7938068e10 con 0x7f7938068a10 2024-01-11T09:47:13.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.779+0000 7f79377fe700 1 -- 172.21.15.27:0/3969151578 >> v1:172.21.15.77:6789/0 conn(0x7f7938068a10 legacy=0x7f79380ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:13.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.779+0000 7f79377fe700 1 -- 172.21.15.27:0/3969151578 >> v1:172.21.15.27:6789/0 conn(0x7f7938159250 legacy=0x7f79380cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:13.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.779+0000 7f79377fe700 1 -- 172.21.15.27:0/3969151578 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f793815a720 con 0x7f79380b08a0 2024-01-11T09:47:13.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.779+0000 7f79377fe700 1 --2- 172.21.15.27:0/3969151578 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79380b08a0 0x7f79380684d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f79380c9f30 tx=0x7f792c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf9a08c655078251 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:13.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.779+0000 7f7936ffd700 1 -- 172.21.15.27:0/3969151578 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792c011630 con 0x7f79380b08a0 2024-01-11T09:47:13.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.779+0000 7f7936ffd700 1 -- 172.21.15.27:0/3969151578 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f792c011790 con 0x7f79380b08a0 2024-01-11T09:47:13.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.779+0000 7f7936ffd700 1 -- 172.21.15.27:0/3969151578 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792c011b50 con 0x7f79380b08a0 2024-01-11T09:47:13.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.780+0000 7f793e7ff700 1 -- 172.21.15.27:0/3969151578 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79380b08a0 msgr2=0x7f79380684d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:13.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.780+0000 7f793e7ff700 1 --2- 172.21.15.27:0/3969151578 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79380b08a0 0x7f79380684d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f79380c9f30 tx=0x7f792c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:13.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.780+0000 7f793e7ff700 1 -- 172.21.15.27:0/3969151578 shutdown_connections 2024-01-11T09:47:13.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.780+0000 7f793e7ff700 1 --2- 172.21.15.27:0/3969151578 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79380b08a0 0x7f79380684d0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:13.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.780+0000 7f793e7ff700 1 -- 172.21.15.27:0/3969151578 >> 172.21.15.27:0/3969151578 conn(0x7f7938067610 msgr2=0x7f7938067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:13.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.780+0000 7f793e7ff700 1 -- 172.21.15.27:0/3969151578 shutdown_connections 2024-01-11T09:47:13.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.780+0000 7f793e7ff700 1 -- 172.21.15.27:0/3969151578 wait complete. 2024-01-11T09:47:13.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.781+0000 7f793e7ff700 1 Processor -- start 2024-01-11T09:47:13.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.781+0000 7f793e7ff700 1 -- start start 2024-01-11T09:47:13.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.782+0000 7f793e7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7938068a10 0x7f7938155e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:13.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.782+0000 7f793e7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79380b08a0 0x7f79380aec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:13.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.782+0000 7f793e7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7938159250 0x7f79380af1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:13.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.782+0000 7f793e7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79380b2c70 con 0x7f7938159250 2024-01-11T09:47:13.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.782+0000 7f793e7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79380c6eb0 con 0x7f7938068a10 2024-01-11T09:47:13.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.782+0000 7f793e7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79380b26b0 con 0x7f79380b08a0 2024-01-11T09:47:13.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.783+0000 7f793cd9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7938159250 0x7f79380af1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:13.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.783+0000 7f793cd9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7938159250 0x7f79380af1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42940/0 (socket says 172.21.15.27:42940) 2024-01-11T09:47:13.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.783+0000 7f793cd9c700 1 -- 172.21.15.27:0/3535568339 learned_addr learned my addr 172.21.15.27:0/3535568339 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:13.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.783+0000 7f7937fff700 1 --2- 172.21.15.27:0/3535568339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7938068a10 0x7f7938155e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:13.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.783+0000 7f79377fe700 1 --2- 172.21.15.27:0/3535568339 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79380b08a0 0x7f79380aec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:13.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.783+0000 7f793cd9c700 1 -- 172.21.15.27:0/3535568339 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79380b08a0 msgr2=0x7f79380aec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:13.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.783+0000 7f793cd9c700 1 --2- 172.21.15.27:0/3535568339 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79380b08a0 0x7f79380aec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:13.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.783+0000 7f793cd9c700 1 -- 172.21.15.27:0/3535568339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7938068a10 msgr2=0x7f7938155e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:13.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.783+0000 7f793cd9c700 1 --2- 172.21.15.27:0/3535568339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7938068a10 0x7f7938155e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:13.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.783+0000 7f793cd9c700 1 -- 172.21.15.27:0/3535568339 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f792c009420 con 0x7f7938159250 2024-01-11T09:47:13.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.784+0000 7f793cd9c700 1 --2- 172.21.15.27:0/3535568339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7938159250 0x7f79380af1c0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f793000c860 tx=0x7f7930002700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:13.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.784+0000 7f791ffff700 1 -- 172.21.15.27:0/3535568339 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7930002030 con 0x7f7938159250 2024-01-11T09:47:13.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.784+0000 7f791ffff700 1 -- 172.21.15.27:0/3535568339 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79300021b0 con 0x7f7938159250 2024-01-11T09:47:13.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.784+0000 7f791ffff700 1 -- 172.21.15.27:0/3535568339 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7930012630 con 0x7f7938159250 2024-01-11T09:47:13.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.784+0000 7f793e7ff700 1 -- 172.21.15.27:0/3535568339 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79380b2990 con 0x7f7938159250 2024-01-11T09:47:13.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.784+0000 7f793e7ff700 1 -- 172.21.15.27:0/3535568339 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7938156370 con 0x7f7938159250 2024-01-11T09:47:13.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.785+0000 7f793e7ff700 1 -- 172.21.15.27:0/3535568339 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79380c7020 con 0x7f7938159250 2024-01-11T09:47:13.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.786+0000 7f791ffff700 1 -- 172.21.15.27:0/3535568339 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f7930012a20 con 0x7f7938159250 2024-01-11T09:47:13.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.787+0000 7f791ffff700 1 --2- 172.21.15.27:0/3535568339 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7924067170 0x7f7924069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:13.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.787+0000 7f791ffff700 1 -- 172.21.15.27:0/3535568339 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f7930084ae0 con 0x7f7938159250 2024-01-11T09:47:13.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.787+0000 7f7937fff700 1 --2- 172.21.15.27:0/3535568339 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7924067170 0x7f7924069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:13.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.788+0000 7f7937fff700 1 --2- 172.21.15.27:0/3535568339 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7924067170 0x7f7924069620 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7928007fd0 tx=0x7f7928009040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.791+0000 7f791ffff700 1 -- 172.21.15.27:0/3535568339 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7930051780 con 0x7f7938159250 2024-01-11T09:47:13.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:13.996+0000 7f793e7ff700 1 -- 172.21.15.27:0/3535568339 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f793804c3c0 con 0x7f7938159250 2024-01-11T09:47:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.549+0000 7f791ffff700 1 -- 172.21.15.27:0/3535568339 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v247) v1 ==== 178+0+0 (secure 0 0 0) 0x7f7930054da0 con 0x7f7938159250 2024-01-11T09:47:14.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.553+0000 7f793e7ff700 1 -- 172.21.15.27:0/3535568339 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7924067170 msgr2=0x7f7924069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:14.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.553+0000 7f793e7ff700 1 --2- 172.21.15.27:0/3535568339 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7924067170 0x7f7924069620 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7928007fd0 tx=0x7f7928009040 comp rx=0 tx=0).stop 2024-01-11T09:47:14.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.553+0000 7f793e7ff700 1 -- 172.21.15.27:0/3535568339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7938159250 msgr2=0x7f79380af1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:14.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.553+0000 7f793e7ff700 1 --2- 172.21.15.27:0/3535568339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7938159250 0x7f79380af1c0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f793000c860 tx=0x7f7930002700 comp rx=0 tx=0).stop 2024-01-11T09:47:14.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.554+0000 7f793e7ff700 1 -- 172.21.15.27:0/3535568339 shutdown_connections 2024-01-11T09:47:14.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.554+0000 7f793e7ff700 1 --2- 172.21.15.27:0/3535568339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7938159250 0x7f79380af1c0 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:14.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.554+0000 7f793e7ff700 1 --2- 172.21.15.27:0/3535568339 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7924067170 0x7f7924069620 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:14.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.554+0000 7f793e7ff700 1 --2- 172.21.15.27:0/3535568339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7938068a10 0x7f7938155e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:14.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.554+0000 7f793e7ff700 1 --2- 172.21.15.27:0/3535568339 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f79380b08a0 0x7f79380aec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:14.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.554+0000 7f793e7ff700 1 -- 172.21.15.27:0/3535568339 >> 172.21.15.27:0/3535568339 conn(0x7f7938067610 msgr2=0x7f793815fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:14.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.554+0000 7f793e7ff700 1 -- 172.21.15.27:0/3535568339 shutdown_connections 2024-01-11T09:47:14.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.554+0000 7f793e7ff700 1 -- 172.21.15.27:0/3535568339 wait complete. 2024-01-11T09:47:14.547 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'backup_fs_data' removed 2024-01-11T09:47:14.569 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-11T09:47:14.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.781+0000 7fd528dff700 1 Processor -- start 2024-01-11T09:47:14.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.781+0000 7fd528dff700 1 -- start start 2024-01-11T09:47:14.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.782+0000 7fd528dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd524140dc0 0x7fd5240adc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:14.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.783+0000 7fd528dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd52415c060 con 0x7fd524141190 2024-01-11T09:47:14.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.783+0000 7fd528dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5240cc9b0 con 0x7fd5240c9130 2024-01-11T09:47:14.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.783+0000 7fd528dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd52415bd80 con 0x7fd524140dc0 2024-01-11T09:47:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.783+0000 7fd521d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd524141190 0x7fd5240c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33692/0 (socket says 172.21.15.27:33692) 2024-01-11T09:47:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.783+0000 7fd521d9b700 1 -- 172.21.15.27:0/1999688091 learned_addr learned my addr 172.21.15.27:0/1999688091 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.783+0000 7fd52259c700 1 --2- 172.21.15.27:0/1999688091 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd524140dc0 0x7fd5240adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.783+0000 7fd52159a700 1 -- 172.21.15.27:0/1999688091 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204053983 0 0) 0x7fd518001110 con 0x7fd524141190 2024-01-11T09:47:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.783+0000 7fd52159a700 1 -- 172.21.15.27:0/1999688091 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd50c003570 con 0x7fd524141190 2024-01-11T09:47:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.784+0000 7fd52159a700 1 -- 172.21.15.27:0/1999688091 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4274405527 0 0) 0x7fd50c003570 con 0x7fd524141190 2024-01-11T09:47:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.784+0000 7fd52159a700 1 -- 172.21.15.27:0/1999688091 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd524140dc0 msgr2=0x7fd5240adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.784+0000 7fd52159a700 1 --2- 172.21.15.27:0/1999688091 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd524140dc0 0x7fd5240adc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.784+0000 7fd52159a700 1 -- 172.21.15.27:0/1999688091 >> v1:172.21.15.77:6789/0 conn(0x7fd5240c9130 legacy=0x7fd52415b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.784+0000 7fd52159a700 1 -- 172.21.15.27:0/1999688091 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5240cc360 con 0x7fd524141190 2024-01-11T09:47:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.784+0000 7fd52159a700 1 -- 172.21.15.27:0/1999688091 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fd518001600 con 0x7fd524141190 2024-01-11T09:47:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.784+0000 7fd52159a700 1 -- 172.21.15.27:0/1999688091 >> v1:172.21.15.27:6789/0 conn(0x7fd524141190 legacy=0x7fd5240c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.784+0000 7fd52159a700 1 --2- 172.21.15.27:0/1999688091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd50c003a90 0x7fd50c003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.784+0000 7fd52159a700 1 --2- 172.21.15.27:0/1999688091 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd524140dc0 0x7fd50c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:14.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.784+0000 7fd52159a700 1 --2- 172.21.15.27:0/1999688091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd50c004d50 0x7fd50c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:14.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.784+0000 7fd52159a700 1 -- 172.21.15.27:0/1999688091 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52415bd80 con 0x7fd50c004d50 2024-01-11T09:47:14.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.784+0000 7fd52159a700 1 -- 172.21.15.27:0/1999688091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c00bbf0 con 0x7fd50c003a90 2024-01-11T09:47:14.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.784+0000 7fd52159a700 1 -- 172.21.15.27:0/1999688091 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c00bd80 con 0x7fd524140dc0 2024-01-11T09:47:14.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.784+0000 7fd52259c700 1 --2- 172.21.15.27:0/1999688091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd50c003a90 0x7fd50c003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:14.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.784+0000 7fd521d9b700 1 --2- 172.21.15.27:0/1999688091 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd524140dc0 0x7fd50c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:14.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.784+0000 7fd522d9d700 1 --2- 172.21.15.27:0/1999688091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd50c004d50 0x7fd50c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:14.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.784+0000 7fd52259c700 1 -- 172.21.15.27:0/1999688091 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd524140dc0 msgr2=0x7fd50c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:14.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.784+0000 7fd52259c700 1 --2- 172.21.15.27:0/1999688091 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd524140dc0 0x7fd50c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:14.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.785+0000 7fd52259c700 1 -- 172.21.15.27:0/1999688091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd50c004d50 msgr2=0x7fd50c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:14.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.785+0000 7fd52259c700 1 --2- 172.21.15.27:0/1999688091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd50c004d50 0x7fd50c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:14.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.785+0000 7fd52259c700 1 -- 172.21.15.27:0/1999688091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c003570 con 0x7fd50c003a90 2024-01-11T09:47:14.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.785+0000 7fd52259c700 1 --2- 172.21.15.27:0/1999688091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd50c003a90 0x7fd50c003ea0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fd514009c00 tx=0x7fd5140091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdc487d98c56f4ed server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:14.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.785+0000 7fd52159a700 1 -- 172.21.15.27:0/1999688091 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51400e710 con 0x7fd50c003a90 2024-01-11T09:47:14.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.785+0000 7fd52159a700 1 -- 172.21.15.27:0/1999688091 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51400e870 con 0x7fd50c003a90 2024-01-11T09:47:14.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.785+0000 7fd52159a700 1 -- 172.21.15.27:0/1999688091 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51400ec30 con 0x7fd50c003a90 2024-01-11T09:47:14.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.786+0000 7fd528dff700 1 -- 172.21.15.27:0/1999688091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd50c003a90 msgr2=0x7fd50c003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:14.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.786+0000 7fd528dff700 1 --2- 172.21.15.27:0/1999688091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd50c003a90 0x7fd50c003ea0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fd514009c00 tx=0x7fd5140091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:14.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.786+0000 7fd522d9d700 1 -- 172.21.15.27:0/1999688091 reap_dead start 2024-01-11T09:47:14.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.786+0000 7fd528dff700 1 -- 172.21.15.27:0/1999688091 shutdown_connections 2024-01-11T09:47:14.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.786+0000 7fd528dff700 1 -- 172.21.15.27:0/1999688091 >> 172.21.15.27:0/1999688091 conn(0x7fd524063ca0 msgr2=0x7fd52406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:14.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.786+0000 7fd528dff700 1 -- 172.21.15.27:0/1999688091 shutdown_connections 2024-01-11T09:47:14.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.786+0000 7fd528dff700 1 -- 172.21.15.27:0/1999688091 wait complete. 2024-01-11T09:47:14.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.787+0000 7fd528dff700 1 Processor -- start 2024-01-11T09:47:14.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.788+0000 7fd528dff700 1 -- start start 2024-01-11T09:47:14.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.788+0000 7fd528dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd50c003a90 0x7fd5240adc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:14.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.788+0000 7fd528dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5240c6ab0 0x7fd5240b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:14.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.788+0000 7fd528dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd5240c6f20 0x7fd5240b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:14.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.789+0000 7fd528dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52413a180 con 0x7fd5240c6ab0 2024-01-11T09:47:14.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.789+0000 7fd528dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52413b110 con 0x7fd50c003a90 2024-01-11T09:47:14.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.789+0000 7fd528dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd52413a500 con 0x7fd5240c6f20 2024-01-11T09:47:14.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.789+0000 7fd521d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5240c6ab0 0x7fd5240b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:14.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.789+0000 7fd521d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5240c6ab0 0x7fd5240b3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44114/0 (socket says 172.21.15.27:44114) 2024-01-11T09:47:14.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.789+0000 7fd521d9b700 1 -- 172.21.15.27:0/3770968386 learned_addr learned my addr 172.21.15.27:0/3770968386 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:14.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.789+0000 7fd52259c700 1 --2- 172.21.15.27:0/3770968386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd50c003a90 0x7fd5240adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:14.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.789+0000 7fd522d9d700 1 --2- 172.21.15.27:0/3770968386 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd5240c6f20 0x7fd5240b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:14.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.789+0000 7fd521d9b700 1 -- 172.21.15.27:0/3770968386 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd5240c6f20 msgr2=0x7fd5240b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:14.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.789+0000 7fd521d9b700 1 --2- 172.21.15.27:0/3770968386 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd5240c6f20 0x7fd5240b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:14.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.789+0000 7fd521d9b700 1 -- 172.21.15.27:0/3770968386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd50c003a90 msgr2=0x7fd5240adc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:14.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.789+0000 7fd521d9b700 1 --2- 172.21.15.27:0/3770968386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd50c003a90 0x7fd5240adc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:14.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.789+0000 7fd521d9b700 1 -- 172.21.15.27:0/3770968386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52413a7e0 con 0x7fd5240c6ab0 2024-01-11T09:47:14.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.789+0000 7fd522d9d700 1 --2- 172.21.15.27:0/3770968386 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd5240c6f20 0x7fd5240b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:14.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.789+0000 7fd521d9b700 1 --2- 172.21.15.27:0/3770968386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5240c6ab0 0x7fd5240b3610 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fd518001e00 tx=0x7fd524157b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:14.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.790+0000 7fd5127fc700 1 -- 172.21.15.27:0/3770968386 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd518018030 con 0x7fd5240c6ab0 2024-01-11T09:47:14.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.790+0000 7fd5127fc700 1 -- 172.21.15.27:0/3770968386 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51800f8a0 con 0x7fd5240c6ab0 2024-01-11T09:47:14.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.790+0000 7fd528dff700 1 -- 172.21.15.27:0/3770968386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd52413aac0 con 0x7fd5240c6ab0 2024-01-11T09:47:14.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.790+0000 7fd5127fc700 1 -- 172.21.15.27:0/3770968386 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51800fc60 con 0x7fd5240c6ab0 2024-01-11T09:47:14.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.790+0000 7fd528dff700 1 -- 172.21.15.27:0/3770968386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5240ae1a0 con 0x7fd5240c6ab0 2024-01-11T09:47:14.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.791+0000 7fd528dff700 1 -- 172.21.15.27:0/3770968386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd52413b2c0 con 0x7fd5240c6ab0 2024-01-11T09:47:14.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.792+0000 7fd5127fc700 1 -- 172.21.15.27:0/3770968386 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fd51801b070 con 0x7fd5240c6ab0 2024-01-11T09:47:14.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.793+0000 7fd5127fc700 1 --2- 172.21.15.27:0/3770968386 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd504067170 0x7fd504069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:14.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.793+0000 7fd5127fc700 1 -- 172.21.15.27:0/3770968386 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fd518019070 con 0x7fd5240c6ab0 2024-01-11T09:47:14.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.793+0000 7fd52259c700 1 --2- 172.21.15.27:0/3770968386 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd504067170 0x7fd504069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:14.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.794+0000 7fd52259c700 1 --2- 172.21.15.27:0/3770968386 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd504067170 0x7fd504069620 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd5140091e0 tx=0x7fd51401a000 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:14.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:14.797+0000 7fd5127fc700 1 -- 172.21.15.27:0/3770968386 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd52413b2c0 con 0x7fd5240c6ab0 2024-01-11T09:47:14.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.001+0000 7fd528dff700 1 -- 172.21.15.27:0/3770968386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fd52415c060 con 0x7fd5240c6ab0 2024-01-11T09:47:14.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.005+0000 7fd5127fc700 1 -- 172.21.15.27:0/3770968386 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v85) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd52415c060 con 0x7fd5240c6ab0 2024-01-11T09:47:14.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.008+0000 7fd528dff700 1 -- 172.21.15.27:0/3770968386 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd504067170 msgr2=0x7fd504069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:14.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.008+0000 7fd528dff700 1 --2- 172.21.15.27:0/3770968386 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd504067170 0x7fd504069620 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd5140091e0 tx=0x7fd51401a000 comp rx=0 tx=0).stop 2024-01-11T09:47:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.008+0000 7fd528dff700 1 -- 172.21.15.27:0/3770968386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5240c6ab0 msgr2=0x7fd5240b3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.008+0000 7fd528dff700 1 --2- 172.21.15.27:0/3770968386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5240c6ab0 0x7fd5240b3610 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fd518001e00 tx=0x7fd524157b80 comp rx=0 tx=0).stop 2024-01-11T09:47:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.008+0000 7fd528dff700 1 -- 172.21.15.27:0/3770968386 shutdown_connections 2024-01-11T09:47:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.008+0000 7fd528dff700 1 --2- 172.21.15.27:0/3770968386 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd5240c6f20 0x7fd5240b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.008+0000 7fd528dff700 1 --2- 172.21.15.27:0/3770968386 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd504067170 0x7fd504069620 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.008+0000 7fd528dff700 1 --2- 172.21.15.27:0/3770968386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd50c003a90 0x7fd5240adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.008+0000 7fd528dff700 1 --2- 172.21.15.27:0/3770968386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5240c6ab0 0x7fd5240b3610 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.008+0000 7fd528dff700 1 -- 172.21.15.27:0/3770968386 >> 172.21.15.27:0/3770968386 conn(0x7fd524063ca0 msgr2=0x7fd5240c7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.008+0000 7fd528dff700 1 -- 172.21.15.27:0/3770968386 shutdown_connections 2024-01-11T09:47:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.008+0000 7fd528dff700 1 -- 172.21.15.27:0/3770968386 wait complete. 2024-01-11T09:47:15.018 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-11T09:47:15.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.237+0000 7feebf7ff700 1 Processor -- start 2024-01-11T09:47:15.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.237+0000 7feebf7ff700 1 -- start start 2024-01-11T09:47:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.238+0000 7feebf7ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feeb815a1b0 0x7feeb815a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.239+0000 7feebf7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb815ae70 0x7feeb815b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.239+0000 7feebf7ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb80cf850 con 0x7feeb80ccd70 2024-01-11T09:47:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.239+0000 7feebf7ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeb80cec40 con 0x7feeb815a1b0 2024-01-11T09:47:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.239+0000 7feebf7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb80cef20 con 0x7feeb815ae70 2024-01-11T09:47:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.239+0000 7feebcd9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7feeb80ccd70 0x7feeb80cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33700/0 (socket says 172.21.15.27:33700) 2024-01-11T09:47:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.239+0000 7feebcd9a700 1 -- 172.21.15.27:0/440922036 learned_addr learned my addr 172.21.15.27:0/440922036 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:15.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.239+0000 7feebd59b700 1 --2- 172.21.15.27:0/440922036 >> v2:172.21.15.77:3300/0 conn(0x7feeb815a1b0 0x7feeb815a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:15.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.239+0000 7feebdd9c700 1 --2- 172.21.15.27:0/440922036 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb815ae70 0x7feeb815b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:15.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.239+0000 7feeaffff700 1 -- 172.21.15.27:0/440922036 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390278377 0 0) 0x7feeb80cf850 con 0x7feeb80ccd70 2024-01-11T09:47:15.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.239+0000 7feeaffff700 1 -- 172.21.15.27:0/440922036 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feea4003570 con 0x7feeb80ccd70 2024-01-11T09:47:15.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.240+0000 7feebd59b700 1 -- 172.21.15.27:0/440922036 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb815ae70 msgr2=0x7feeb815b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:15.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.240+0000 7feebd59b700 1 --2- 172.21.15.27:0/440922036 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb815ae70 0x7feeb815b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.240+0000 7feebd59b700 1 -- 172.21.15.27:0/440922036 >> v1:172.21.15.27:6789/0 conn(0x7feeb80ccd70 legacy=0x7feeb80cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:15.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.240+0000 7feebd59b700 1 -- 172.21.15.27:0/440922036 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb80cf200 con 0x7feeb815a1b0 2024-01-11T09:47:15.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.240+0000 7feebd59b700 1 --2- 172.21.15.27:0/440922036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feeb815a1b0 0x7feeb815a580 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7feeb40098e0 tx=0x7feeb40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8f7f25798b242d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:15.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.240+0000 7feeaffff700 1 -- 172.21.15.27:0/440922036 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb4011630 con 0x7feeb815a1b0 2024-01-11T09:47:15.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.240+0000 7feeaffff700 1 -- 172.21.15.27:0/440922036 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb4011790 con 0x7feeb815a1b0 2024-01-11T09:47:15.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.240+0000 7feeaffff700 1 -- 172.21.15.27:0/440922036 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb4011b30 con 0x7feeb815a1b0 2024-01-11T09:47:15.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.241+0000 7feebf7ff700 1 -- 172.21.15.27:0/440922036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feeb815a1b0 msgr2=0x7feeb815a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:15.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.241+0000 7feebf7ff700 1 --2- 172.21.15.27:0/440922036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feeb815a1b0 0x7feeb815a580 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7feeb40098e0 tx=0x7feeb40091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.241+0000 7feebf7ff700 1 -- 172.21.15.27:0/440922036 shutdown_connections 2024-01-11T09:47:15.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.241+0000 7feebf7ff700 1 --2- 172.21.15.27:0/440922036 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb815ae70 0x7feeb815b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.241+0000 7feebf7ff700 1 --2- 172.21.15.27:0/440922036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feeb815a1b0 0x7feeb815a580 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.241+0000 7feebf7ff700 1 -- 172.21.15.27:0/440922036 >> 172.21.15.27:0/440922036 conn(0x7feeb806ed40 msgr2=0x7feeb806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:15.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.241+0000 7feebf7ff700 1 -- 172.21.15.27:0/440922036 shutdown_connections 2024-01-11T09:47:15.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.241+0000 7feebf7ff700 1 -- 172.21.15.27:0/440922036 wait complete. 2024-01-11T09:47:15.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.242+0000 7feebf7ff700 1 Processor -- start 2024-01-11T09:47:15.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.243+0000 7feebf7ff700 1 -- start start 2024-01-11T09:47:15.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.243+0000 7feebf7ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb80ccd70 0x7feeb80c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:15.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.243+0000 7feebf7ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb815a1b0 0x7feeb80c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.243+0000 7feebf7ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feeb815ae70 0x7feeb814ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.244+0000 7feebf7ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb80c8a10 con 0x7feeb80ccd70 2024-01-11T09:47:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.244+0000 7feebf7ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb80b4df0 con 0x7feeb815ae70 2024-01-11T09:47:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.244+0000 7feebf7ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb80b44c0 con 0x7feeb815a1b0 2024-01-11T09:47:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.244+0000 7feebd59b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb80ccd70 0x7feeb80c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.244+0000 7feebd59b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb80ccd70 0x7feeb80c6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44130/0 (socket says 172.21.15.27:44130) 2024-01-11T09:47:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.244+0000 7feebd59b700 1 -- 172.21.15.27:0/3261204797 learned_addr learned my addr 172.21.15.27:0/3261204797 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.244+0000 7feebdd9c700 1 --2- 172.21.15.27:0/3261204797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feeb815ae70 0x7feeb814ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.244+0000 7feebcd9a700 1 --2- 172.21.15.27:0/3261204797 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb815a1b0 0x7feeb80c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.244+0000 7feebd59b700 1 -- 172.21.15.27:0/3261204797 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb815a1b0 msgr2=0x7feeb80c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.244+0000 7feebd59b700 1 --2- 172.21.15.27:0/3261204797 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb815a1b0 0x7feeb80c8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.244+0000 7feebd59b700 1 -- 172.21.15.27:0/3261204797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feeb815ae70 msgr2=0x7feeb814ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.244+0000 7feebd59b700 1 --2- 172.21.15.27:0/3261204797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feeb815ae70 0x7feeb814ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.244+0000 7feebd59b700 1 -- 172.21.15.27:0/3261204797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb4009420 con 0x7feeb80ccd70 2024-01-11T09:47:15.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.245+0000 7feebd59b700 1 --2- 172.21.15.27:0/3261204797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb80ccd70 0x7feeb80c6ab0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7feeb8069770 tx=0x7feeb4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:15.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.245+0000 7feead7fa700 1 -- 172.21.15.27:0/3261204797 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb4011900 con 0x7feeb80ccd70 2024-01-11T09:47:15.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.245+0000 7feead7fa700 1 -- 172.21.15.27:0/3261204797 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb4011a60 con 0x7feeb80ccd70 2024-01-11T09:47:15.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.245+0000 7feebf7ff700 1 -- 172.21.15.27:0/3261204797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeb80b47a0 con 0x7feeb80ccd70 2024-01-11T09:47:15.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.245+0000 7feead7fa700 1 -- 172.21.15.27:0/3261204797 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb4025640 con 0x7feeb80ccd70 2024-01-11T09:47:15.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.245+0000 7feebf7ff700 1 -- 172.21.15.27:0/3261204797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb81552e0 con 0x7feeb80ccd70 2024-01-11T09:47:15.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.246+0000 7feebf7ff700 1 -- 172.21.15.27:0/3261204797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeb804c3c0 con 0x7feeb80ccd70 2024-01-11T09:47:15.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.247+0000 7feead7fa700 1 -- 172.21.15.27:0/3261204797 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7feeb40259f0 con 0x7feeb80ccd70 2024-01-11T09:47:15.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.249+0000 7feead7fa700 1 --2- 172.21.15.27:0/3261204797 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fee9c067170 0x7fee9c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:15.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.249+0000 7feead7fa700 1 -- 172.21.15.27:0/3261204797 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 5911+0+0 (secure 0 0 0) 0x7feeb4086120 con 0x7feeb80ccd70 2024-01-11T09:47:15.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.249+0000 7feebcd9a700 1 --2- 172.21.15.27:0/3261204797 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fee9c067170 0x7fee9c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:15.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.250+0000 7feebcd9a700 1 --2- 172.21.15.27:0/3261204797 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fee9c067170 0x7fee9c069620 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7feeb8155140 tx=0x7feea8009040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:15.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.253+0000 7feead7fa700 1 -- 172.21.15.27:0/3261204797 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feeb4052f90 con 0x7feeb80ccd70 2024-01-11T09:47:15.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.462+0000 7feebf7ff700 1 -- 172.21.15.27:0/3261204797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7feeb80694d0 con 0x7feeb80ccd70 2024-01-11T09:47:15.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.466+0000 7feead7fa700 1 -- 172.21.15.27:0/3261204797 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v86) v1 ==== 115+0+0 (secure 0 0 0) 0x7feeb40565b0 con 0x7feeb80ccd70 2024-01-11T09:47:15.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.468+0000 7feebf7ff700 1 -- 172.21.15.27:0/3261204797 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fee9c067170 msgr2=0x7fee9c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:15.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.468+0000 7feebf7ff700 1 --2- 172.21.15.27:0/3261204797 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fee9c067170 0x7fee9c069620 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7feeb8155140 tx=0x7feea8009040 comp rx=0 tx=0).stop 2024-01-11T09:47:15.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.468+0000 7feebf7ff700 1 -- 172.21.15.27:0/3261204797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb80ccd70 msgr2=0x7feeb80c6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:15.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.468+0000 7feebf7ff700 1 --2- 172.21.15.27:0/3261204797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb80ccd70 0x7feeb80c6ab0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7feeb8069770 tx=0x7feeb4017000 comp rx=0 tx=0).stop 2024-01-11T09:47:15.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.469+0000 7feebf7ff700 1 -- 172.21.15.27:0/3261204797 shutdown_connections 2024-01-11T09:47:15.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.469+0000 7feebf7ff700 1 --2- 172.21.15.27:0/3261204797 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fee9c067170 0x7fee9c069620 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.469+0000 7feebf7ff700 1 --2- 172.21.15.27:0/3261204797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feeb815ae70 0x7feeb814ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.469+0000 7feebf7ff700 1 --2- 172.21.15.27:0/3261204797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeb80ccd70 0x7feeb80c6ab0 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.469+0000 7feebf7ff700 1 --2- 172.21.15.27:0/3261204797 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7feeb815a1b0 0x7feeb80c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.469+0000 7feebf7ff700 1 -- 172.21.15.27:0/3261204797 >> 172.21.15.27:0/3261204797 conn(0x7feeb806ed40 msgr2=0x7feeb806f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:15.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.469+0000 7feebf7ff700 1 -- 172.21.15.27:0/3261204797 shutdown_connections 2024-01-11T09:47:15.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.469+0000 7feebf7ff700 1 -- 172.21.15.27:0/3261204797 wait complete. 2024-01-11T09:47:15.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-11T09:47:15.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.688+0000 7fabc13ff700 1 Processor -- start 2024-01-11T09:47:15.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.688+0000 7fabc13ff700 1 -- start start 2024-01-11T09:47:15.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.690+0000 7fabc13ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fabbc15a1b0 0x7fabbc15a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:15.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.690+0000 7fabc13ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fabbc0ccd70 0x7fabbc0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:15.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.690+0000 7fabc13ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fabbc0685c0 0x7fabbc068a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:15.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.690+0000 7fabc13ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabbc0cf850 con 0x7fabbc0685c0 2024-01-11T09:47:15.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.690+0000 7fabc13ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabbc0cec40 con 0x7fabbc0ccd70 2024-01-11T09:47:15.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.690+0000 7fabc13ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabbc0cef20 con 0x7fabbc15a1b0 2024-01-11T09:47:15.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.690+0000 7fabbb7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fabbc0685c0 0x7fabbc068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:15.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.690+0000 7fabbb7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fabbc0685c0 0x7fabbc068a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44132/0 (socket says 172.21.15.27:44132) 2024-01-11T09:47:15.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.690+0000 7fabbb7fe700 1 -- 172.21.15.27:0/1035028525 learned_addr learned my addr 172.21.15.27:0/1035028525 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:15.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.690+0000 7fabba7fc700 1 --2- 172.21.15.27:0/1035028525 >> v2:172.21.15.77:3300/0 conn(0x7fabbc0ccd70 0x7fabbc0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:15.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.690+0000 7fabbaffd700 1 --2- 172.21.15.27:0/1035028525 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fabbc15a1b0 0x7fabbc15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:15.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.690+0000 7fabbb7fe700 1 -- 172.21.15.27:0/1035028525 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fabbc15a1b0 msgr2=0x7fabbc15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:15.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.691+0000 7fabbb7fe700 1 --2- 172.21.15.27:0/1035028525 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fabbc15a1b0 0x7fabbc15a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.691+0000 7fabbb7fe700 1 -- 172.21.15.27:0/1035028525 >> v2:172.21.15.77:3300/0 conn(0x7fabbc0ccd70 msgr2=0x7fabbc0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:15.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.691+0000 7fabbb7fe700 1 --2- 172.21.15.27:0/1035028525 >> v2:172.21.15.77:3300/0 conn(0x7fabbc0ccd70 0x7fabbc0cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.691+0000 7fabbb7fe700 1 -- 172.21.15.27:0/1035028525 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabbc0cf200 con 0x7fabbc0685c0 2024-01-11T09:47:15.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.691+0000 7fabbb7fe700 1 --2- 172.21.15.27:0/1035028525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabbc0685c0 0x7fabbc068a40 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fabbc069600 tx=0x7fabb4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9f6a79d794fd32 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:15.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.691+0000 7fabb9ffb700 1 -- 172.21.15.27:0/1035028525 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb400e070 con 0x7fabbc0685c0 2024-01-11T09:47:15.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.691+0000 7fabb9ffb700 1 -- 172.21.15.27:0/1035028525 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb400ba50 con 0x7fabbc0685c0 2024-01-11T09:47:15.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.691+0000 7fabb9ffb700 1 -- 172.21.15.27:0/1035028525 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb40092e0 con 0x7fabbc0685c0 2024-01-11T09:47:15.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.692+0000 7fabc13ff700 1 -- 172.21.15.27:0/1035028525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabbc0685c0 msgr2=0x7fabbc068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:15.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.692+0000 7fabc13ff700 1 --2- 172.21.15.27:0/1035028525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabbc0685c0 0x7fabbc068a40 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fabbc069600 tx=0x7fabb4009840 comp rx=0 tx=0).stop 2024-01-11T09:47:15.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.692+0000 7fabc13ff700 1 -- 172.21.15.27:0/1035028525 shutdown_connections 2024-01-11T09:47:15.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.692+0000 7fabc13ff700 1 --2- 172.21.15.27:0/1035028525 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fabbc15a1b0 0x7fabbc15a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.692+0000 7fabc13ff700 1 --2- 172.21.15.27:0/1035028525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabbc0685c0 0x7fabbc068a40 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.692+0000 7fabc13ff700 1 --2- 172.21.15.27:0/1035028525 >> v2:172.21.15.77:3300/0 conn(0x7fabbc0ccd70 0x7fabbc0cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.692+0000 7fabc13ff700 1 -- 172.21.15.27:0/1035028525 >> 172.21.15.27:0/1035028525 conn(0x7fabbc06ed40 msgr2=0x7fabbc06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:15.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.692+0000 7fabc13ff700 1 -- 172.21.15.27:0/1035028525 shutdown_connections 2024-01-11T09:47:15.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.692+0000 7fabc13ff700 1 -- 172.21.15.27:0/1035028525 wait complete. 2024-01-11T09:47:15.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.693+0000 7fabc13ff700 1 Processor -- start 2024-01-11T09:47:15.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.693+0000 7fabc13ff700 1 -- start start 2024-01-11T09:47:15.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.694+0000 7fabc13ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabbc0685c0 0x7fabbc0c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:15.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.694+0000 7fabc13ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabbc0ccd70 0x7fabbc0c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:15.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.694+0000 7fabc13ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fabbc15a1b0 0x7fabbc0b4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:15.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.694+0000 7fabc13ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabbc0ca860 con 0x7fabbc0685c0 2024-01-11T09:47:15.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.694+0000 7fabc13ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabbc0c94f0 con 0x7fabbc0ccd70 2024-01-11T09:47:15.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.694+0000 7fabc13ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabbc0c97c0 con 0x7fabbc15a1b0 2024-01-11T09:47:15.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.695+0000 7fabbaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabbc0685c0 0x7fabbc0c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:15.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.695+0000 7fabbaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabbc0685c0 0x7fabbc0c8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44146/0 (socket says 172.21.15.27:44146) 2024-01-11T09:47:15.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.695+0000 7fabbaffd700 1 -- 172.21.15.27:0/1149334308 learned_addr learned my addr 172.21.15.27:0/1149334308 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:15.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.695+0000 7fabbb7fe700 1 --2- 172.21.15.27:0/1149334308 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fabbc15a1b0 0x7fabbc0b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:15.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.695+0000 7fabba7fc700 1 --2- 172.21.15.27:0/1149334308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabbc0ccd70 0x7fabbc0c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:15.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.695+0000 7fabbaffd700 1 -- 172.21.15.27:0/1149334308 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fabbc15a1b0 msgr2=0x7fabbc0b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:15.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.695+0000 7fabbaffd700 1 --2- 172.21.15.27:0/1149334308 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fabbc15a1b0 0x7fabbc0b4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.695+0000 7fabbaffd700 1 -- 172.21.15.27:0/1149334308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabbc0ccd70 msgr2=0x7fabbc0c8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:15.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.695+0000 7fabbaffd700 1 --2- 172.21.15.27:0/1149334308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabbc0ccd70 0x7fabbc0c8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.695+0000 7fabbaffd700 1 -- 172.21.15.27:0/1149334308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb4009ae0 con 0x7fabbc0685c0 2024-01-11T09:47:15.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.695+0000 7fabba7fc700 1 --2- 172.21.15.27:0/1149334308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabbc0ccd70 0x7fabbc0c8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:15.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.695+0000 7fabbaffd700 1 --2- 172.21.15.27:0/1149334308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabbc0685c0 0x7fabbc0c8a70 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7fabac00d890 tx=0x7fabac008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:15.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.695+0000 7faba2ffd700 1 -- 172.21.15.27:0/1149334308 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac00b600 con 0x7fabbc0685c0 2024-01-11T09:47:15.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.695+0000 7faba2ffd700 1 -- 172.21.15.27:0/1149334308 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabac00b760 con 0x7fabbc0685c0 2024-01-11T09:47:15.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.695+0000 7fabc13ff700 1 -- 172.21.15.27:0/1149334308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabbc0c9aa0 con 0x7fabbc0685c0 2024-01-11T09:47:15.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.695+0000 7faba2ffd700 1 -- 172.21.15.27:0/1149334308 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac00bae0 con 0x7fabbc0685c0 2024-01-11T09:47:15.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.696+0000 7fabc13ff700 1 -- 172.21.15.27:0/1149334308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabbc0ca060 con 0x7fabbc0685c0 2024-01-11T09:47:15.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.697+0000 7fabc13ff700 1 -- 172.21.15.27:0/1149334308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabbc04c3c0 con 0x7fabbc0685c0 2024-01-11T09:47:15.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.698+0000 7faba2ffd700 1 -- 172.21.15.27:0/1149334308 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fabac01e460 con 0x7fabbc0685c0 2024-01-11T09:47:15.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.698+0000 7faba2ffd700 1 --2- 172.21.15.27:0/1149334308 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7faba8067170 0x7faba8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:15.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.699+0000 7faba2ffd700 1 -- 172.21.15.27:0/1149334308 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fabac0850b0 con 0x7fabbc0685c0 2024-01-11T09:47:15.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.699+0000 7fabba7fc700 1 --2- 172.21.15.27:0/1149334308 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7faba8067170 0x7faba8069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:15.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.700+0000 7fabba7fc700 1 --2- 172.21.15.27:0/1149334308 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7faba8067170 0x7faba8069620 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fabbc158da0 tx=0x7fabb0008040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:15.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.702+0000 7faba2ffd700 1 -- 172.21.15.27:0/1149334308 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fabac051f20 con 0x7fabbc0685c0 2024-01-11T09:47:15.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.904+0000 7fabc13ff700 1 -- 172.21.15.27:0/1149334308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fabbc0b2d60 con 0x7fabbc0685c0 2024-01-11T09:47:15.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.910+0000 7faba2ffd700 1 -- 172.21.15.27:0/1149334308 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v87) v1 ==== 104+0+0 (secure 0 0 0) 0x7fabac055540 con 0x7fabbc0685c0 2024-01-11T09:47:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.912+0000 7fabc13ff700 1 -- 172.21.15.27:0/1149334308 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7faba8067170 msgr2=0x7faba8069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.912+0000 7fabc13ff700 1 --2- 172.21.15.27:0/1149334308 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7faba8067170 0x7faba8069620 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fabbc158da0 tx=0x7fabb0008040 comp rx=0 tx=0).stop 2024-01-11T09:47:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.912+0000 7fabc13ff700 1 -- 172.21.15.27:0/1149334308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabbc0685c0 msgr2=0x7fabbc0c8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.912+0000 7fabc13ff700 1 --2- 172.21.15.27:0/1149334308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabbc0685c0 0x7fabbc0c8a70 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7fabac00d890 tx=0x7fabac008040 comp rx=0 tx=0).stop 2024-01-11T09:47:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.912+0000 7fabc13ff700 1 -- 172.21.15.27:0/1149334308 shutdown_connections 2024-01-11T09:47:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.912+0000 7fabc13ff700 1 --2- 172.21.15.27:0/1149334308 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fabbc15a1b0 0x7fabbc0b4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.912+0000 7fabc13ff700 1 --2- 172.21.15.27:0/1149334308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabbc0685c0 0x7fabbc0c8a70 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.912+0000 7fabc13ff700 1 --2- 172.21.15.27:0/1149334308 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7faba8067170 0x7faba8069620 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.912+0000 7fabc13ff700 1 --2- 172.21.15.27:0/1149334308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabbc0ccd70 0x7fabbc0c8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.912+0000 7fabc13ff700 1 -- 172.21.15.27:0/1149334308 >> 172.21.15.27:0/1149334308 conn(0x7fabbc06ed40 msgr2=0x7fabbc0b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.912+0000 7fabc13ff700 1 -- 172.21.15.27:0/1149334308 shutdown_connections 2024-01-11T09:47:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:15.912+0000 7fabc13ff700 1 -- 172.21.15.27:0/1149334308 wait complete. 2024-01-11T09:47:15.923 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2024-01-11T09:47:16.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.132+0000 7f33a75ff700 1 Processor -- start 2024-01-11T09:47:16.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.132+0000 7f33a75ff700 1 -- start start 2024-01-11T09:47:16.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.134+0000 7f33a75ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33a01596c0 0x7f33a015d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:16.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.134+0000 7f33a75ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f33a00ca670 0x7f33a0068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:16.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.134+0000 7f33a75ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f33a0068b60 0x7f33a0159a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:16.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.134+0000 7f33a75ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33a0068fe0 con 0x7f33a00ca670 2024-01-11T09:47:16.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.134+0000 7f33a75ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33a00a9cb0 con 0x7f33a0068b60 2024-01-11T09:47:16.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.134+0000 7f33a75ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a015e210 con 0x7f33a01596c0 2024-01-11T09:47:16.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.134+0000 7f33a4b9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f33a00ca670 0x7f33a0068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:16.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.134+0000 7f33a4b9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f33a00ca670 0x7f33a0068620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44150/0 (socket says 172.21.15.27:44150) 2024-01-11T09:47:16.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.134+0000 7f33a4b9a700 1 -- 172.21.15.27:0/610461472 learned_addr learned my addr 172.21.15.27:0/610461472 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:16.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.134+0000 7f33a539b700 1 --2- 172.21.15.27:0/610461472 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33a01596c0 0x7f33a015d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:16.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.134+0000 7f33a5b9c700 1 --2- 172.21.15.27:0/610461472 >> v2:172.21.15.77:3300/0 conn(0x7f33a0068b60 0x7f33a0159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:16.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.135+0000 7f33a4b9a700 1 -- 172.21.15.27:0/610461472 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33a01596c0 msgr2=0x7f33a015d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:16.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.135+0000 7f33a4b9a700 1 --2- 172.21.15.27:0/610461472 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33a01596c0 0x7f33a015d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:16.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.135+0000 7f33a4b9a700 1 -- 172.21.15.27:0/610461472 >> v2:172.21.15.77:3300/0 conn(0x7f33a0068b60 msgr2=0x7f33a0159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:16.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.135+0000 7f33a4b9a700 1 --2- 172.21.15.27:0/610461472 >> v2:172.21.15.77:3300/0 conn(0x7f33a0068b60 0x7f33a0159a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:16.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.135+0000 7f33a4b9a700 1 -- 172.21.15.27:0/610461472 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a015dc50 con 0x7f33a00ca670 2024-01-11T09:47:16.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.135+0000 7f33a4b9a700 1 --2- 172.21.15.27:0/610461472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a00ca670 0x7f33a0068620 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f33a006ca60 tx=0x7f3390009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e471a70433b598 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:16.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.135+0000 7f3397fff700 1 -- 172.21.15.27:0/610461472 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339000b640 con 0x7f33a00ca670 2024-01-11T09:47:16.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.135+0000 7f3397fff700 1 -- 172.21.15.27:0/610461472 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f339000b7a0 con 0x7f33a00ca670 2024-01-11T09:47:16.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.135+0000 7f3397fff700 1 -- 172.21.15.27:0/610461472 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339000bb60 con 0x7f33a00ca670 2024-01-11T09:47:16.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.136+0000 7f33a75ff700 1 -- 172.21.15.27:0/610461472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a00ca670 msgr2=0x7f33a0068620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:16.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.136+0000 7f33a75ff700 1 --2- 172.21.15.27:0/610461472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a00ca670 0x7f33a0068620 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f33a006ca60 tx=0x7f3390009250 comp rx=0 tx=0).stop 2024-01-11T09:47:16.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.136+0000 7f33a75ff700 1 -- 172.21.15.27:0/610461472 shutdown_connections 2024-01-11T09:47:16.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.136+0000 7f33a75ff700 1 --2- 172.21.15.27:0/610461472 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33a01596c0 0x7f33a015d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:16.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.136+0000 7f33a75ff700 1 --2- 172.21.15.27:0/610461472 >> v2:172.21.15.77:3300/0 conn(0x7f33a0068b60 0x7f33a0159a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:16.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.136+0000 7f33a75ff700 1 --2- 172.21.15.27:0/610461472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a00ca670 0x7f33a0068620 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:16.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.136+0000 7f33a75ff700 1 -- 172.21.15.27:0/610461472 >> 172.21.15.27:0/610461472 conn(0x7f33a00c9f70 msgr2=0x7f33a006deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:16.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.136+0000 7f33a75ff700 1 -- 172.21.15.27:0/610461472 shutdown_connections 2024-01-11T09:47:16.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.136+0000 7f33a75ff700 1 -- 172.21.15.27:0/610461472 wait complete. 2024-01-11T09:47:16.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.138+0000 7f33a75ff700 1 Processor -- start 2024-01-11T09:47:16.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.138+0000 7f33a75ff700 1 -- start start 2024-01-11T09:47:16.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.139+0000 7f33a75ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0068b60 0x7f33a014b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:16.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.139+0000 7f33a75ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33a00ca670 0x7f33a00ae430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:16.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.139+0000 7f33a75ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33a01596c0 0x7f33a015ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:16.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.139+0000 7f33a75ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a0169a70 con 0x7f33a0068b60 2024-01-11T09:47:16.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.139+0000 7f33a75ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a016a890 con 0x7f33a00ca670 2024-01-11T09:47:16.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.139+0000 7f33a75ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a014bab0 con 0x7f33a01596c0 2024-01-11T09:47:16.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.139+0000 7f33a539b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0068b60 0x7f33a014b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:16.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.139+0000 7f33a539b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0068b60 0x7f33a014b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44156/0 (socket says 172.21.15.27:44156) 2024-01-11T09:47:16.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.139+0000 7f33a539b700 1 -- 172.21.15.27:0/1464929678 learned_addr learned my addr 172.21.15.27:0/1464929678 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:16.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.139+0000 7f33a4b9a700 1 --2- 172.21.15.27:0/1464929678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33a00ca670 0x7f33a00ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:16.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.139+0000 7f33a539b700 1 -- 172.21.15.27:0/1464929678 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33a01596c0 msgr2=0x7f33a015ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:16.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.139+0000 7f33a5b9c700 1 --2- 172.21.15.27:0/1464929678 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33a01596c0 0x7f33a015ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:16.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.139+0000 7f33a539b700 1 --2- 172.21.15.27:0/1464929678 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33a01596c0 0x7f33a015ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:16.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.139+0000 7f33a539b700 1 -- 172.21.15.27:0/1464929678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33a00ca670 msgr2=0x7f33a00ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:16.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.139+0000 7f33a539b700 1 --2- 172.21.15.27:0/1464929678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33a00ca670 0x7f33a00ae430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:16.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.139+0000 7f33a539b700 1 -- 172.21.15.27:0/1464929678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33900094f0 con 0x7f33a0068b60 2024-01-11T09:47:16.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.139+0000 7f33a4b9a700 1 --2- 172.21.15.27:0/1464929678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33a00ca670 0x7f33a00ae430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:16.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.139+0000 7f33a5b9c700 1 --2- 172.21.15.27:0/1464929678 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33a01596c0 0x7f33a015ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:16.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.140+0000 7f33a539b700 1 --2- 172.21.15.27:0/1464929678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0068b60 0x7f33a014b570 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f339c00e930 tx=0x7f339c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:16.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.140+0000 7f33957fa700 1 -- 172.21.15.27:0/1464929678 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339c00eed0 con 0x7f33a0068b60 2024-01-11T09:47:16.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.140+0000 7f33957fa700 1 -- 172.21.15.27:0/1464929678 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f339c00c850 con 0x7f33a0068b60 2024-01-11T09:47:16.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.140+0000 7f33a75ff700 1 -- 172.21.15.27:0/1464929678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a0169c80 con 0x7f33a0068b60 2024-01-11T09:47:16.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.140+0000 7f33957fa700 1 -- 172.21.15.27:0/1464929678 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339c00cbf0 con 0x7f33a0068b60 2024-01-11T09:47:16.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.140+0000 7f33a75ff700 1 -- 172.21.15.27:0/1464929678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a016a010 con 0x7f33a0068b60 2024-01-11T09:47:16.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.141+0000 7f33a75ff700 1 -- 172.21.15.27:0/1464929678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33a00718d0 con 0x7f33a0068b60 2024-01-11T09:47:16.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.142+0000 7f33957fa700 1 -- 172.21.15.27:0/1464929678 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f339c013470 con 0x7f33a0068b60 2024-01-11T09:47:16.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.143+0000 7f33957fa700 1 --2- 172.21.15.27:0/1464929678 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3384067170 0x7f3384069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:16.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.143+0000 7f33957fa700 1 -- 172.21.15.27:0/1464929678 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f339c084b90 con 0x7f33a0068b60 2024-01-11T09:47:16.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.143+0000 7f33a4b9a700 1 --2- 172.21.15.27:0/1464929678 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3384067170 0x7f3384069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:16.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.144+0000 7f33a4b9a700 1 --2- 172.21.15.27:0/1464929678 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3384067170 0x7f3384069620 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f33a006ca60 tx=0x7f3390017000 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:16.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.147+0000 7f33957fa700 1 -- 172.21.15.27:0/1464929678 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f339c051aa0 con 0x7f33a0068b60 2024-01-11T09:47:16.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.432+0000 7f33a75ff700 1 -- 172.21.15.27:0/1464929678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7f33a015df30 con 0x7f33a0068b60 2024-01-11T09:47:16.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.548+0000 7f33957fa700 1 -- 172.21.15.27:0/1464929678 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3184) v1 ==== 187+0+0 (secure 0 0 0) 0x7f339c0550c0 con 0x7f33a0068b60 2024-01-11T09:47:16.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.551+0000 7f33a75ff700 1 -- 172.21.15.27:0/1464929678 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3384067170 msgr2=0x7f3384069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:16.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.551+0000 7f33a75ff700 1 --2- 172.21.15.27:0/1464929678 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3384067170 0x7f3384069620 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f33a006ca60 tx=0x7f3390017000 comp rx=0 tx=0).stop 2024-01-11T09:47:16.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.551+0000 7f33a75ff700 1 -- 172.21.15.27:0/1464929678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0068b60 msgr2=0x7f33a014b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:16.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.551+0000 7f33a75ff700 1 --2- 172.21.15.27:0/1464929678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0068b60 0x7f33a014b570 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f339c00e930 tx=0x7f339c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:16.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.551+0000 7f33a75ff700 1 -- 172.21.15.27:0/1464929678 shutdown_connections 2024-01-11T09:47:16.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.551+0000 7f33a75ff700 1 --2- 172.21.15.27:0/1464929678 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f33a01596c0 0x7f33a015ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:16.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.551+0000 7f33a75ff700 1 --2- 172.21.15.27:0/1464929678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0068b60 0x7f33a014b570 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:16.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.551+0000 7f33a75ff700 1 --2- 172.21.15.27:0/1464929678 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3384067170 0x7f3384069620 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:16.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.551+0000 7f33a75ff700 1 --2- 172.21.15.27:0/1464929678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33a00ca670 0x7f33a00ae430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:16.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.551+0000 7f33a75ff700 1 -- 172.21.15.27:0/1464929678 >> 172.21.15.27:0/1464929678 conn(0x7f33a00c9f70 msgr2=0x7f33a01575e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:16.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.551+0000 7f33a75ff700 1 -- 172.21.15.27:0/1464929678 shutdown_connections 2024-01-11T09:47:16.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.551+0000 7f33a75ff700 1 -- 172.21.15.27:0/1464929678 wait complete. 2024-01-11T09:47:16.561 INFO:tasks.cephfs_test_runner:Test command fails without --fs argument in presence of multiple fs ... ok 2024-01-11T09:47:16.562 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2024-01-11T09:47:16.562 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2024-01-11T09:47:16.562 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2024-01-11T09:47:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.769+0000 7f11a5dff700 1 Processor -- start 2024-01-11T09:47:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.769+0000 7f11a5dff700 1 -- start start 2024-01-11T09:47:16.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.770+0000 7f11a5dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f11a01599e0 0x7f11a00b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:16.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.770+0000 7f11a5dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11a0150330 0x7f11a015a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:16.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.770+0000 7f11a5dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a0157e30 con 0x7f11a0159db0 2024-01-11T09:47:16.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.770+0000 7f11a5dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a0157500 con 0x7f11a01599e0 2024-01-11T09:47:16.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.770+0000 7f11a5dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a01577e0 con 0x7f11a0150330 2024-01-11T09:47:16.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.770+0000 7f119f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f11a01599e0 0x7f11a00b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:16.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.771+0000 7f119f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f11a01599e0 0x7f11a00b0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44170/0 (socket says 172.21.15.27:44170) 2024-01-11T09:47:16.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.771+0000 7f119effd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f11a0159db0 0x7f11a014fc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33708/0 (socket says 172.21.15.27:33708) 2024-01-11T09:47:16.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.771+0000 7f119f7fe700 1 -- 172.21.15.27:0/4053734340 learned_addr learned my addr 172.21.15.27:0/4053734340 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:16.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.771+0000 7f119f7fe700 1 -- 172.21.15.27:0/4053734340 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11a0150330 msgr2=0x7f11a015a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:16.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.771+0000 7f119f7fe700 1 --2- 172.21.15.27:0/4053734340 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11a0150330 0x7f11a015a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:16.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.771+0000 7f119f7fe700 1 -- 172.21.15.27:0/4053734340 >> v1:172.21.15.27:6789/0 conn(0x7f11a0159db0 legacy=0x7f11a014fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:16.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.771+0000 7f119f7fe700 1 -- 172.21.15.27:0/4053734340 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a0157ac0 con 0x7f11a01599e0 2024-01-11T09:47:16.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.771+0000 7f119e7fc700 1 -- 172.21.15.27:0/4053734340 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461539136 0 0) 0x7f11a0157e30 con 0x7f11a0159db0 2024-01-11T09:47:16.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.771+0000 7f119f7fe700 1 --2- 172.21.15.27:0/4053734340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11a01599e0 0x7f11a00b0130 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f11900099b0 tx=0x7f1190009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43f3bd0a5e320f30 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:16.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.772+0000 7f119e7fc700 1 -- 172.21.15.27:0/4053734340 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119000b860 con 0x7f11a01599e0 2024-01-11T09:47:16.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.772+0000 7f119e7fc700 1 -- 172.21.15.27:0/4053734340 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119000b9c0 con 0x7f11a01599e0 2024-01-11T09:47:16.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.772+0000 7f119e7fc700 1 -- 172.21.15.27:0/4053734340 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119000bd60 con 0x7f11a01599e0 2024-01-11T09:47:16.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.772+0000 7f11a5dff700 1 -- 172.21.15.27:0/4053734340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11a01599e0 msgr2=0x7f11a00b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:16.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.772+0000 7f11a5dff700 1 --2- 172.21.15.27:0/4053734340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11a01599e0 0x7f11a00b0130 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f11900099b0 tx=0x7f1190009250 comp rx=0 tx=0).stop 2024-01-11T09:47:16.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.772+0000 7f11a5dff700 1 -- 172.21.15.27:0/4053734340 shutdown_connections 2024-01-11T09:47:16.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.772+0000 7f11a5dff700 1 --2- 172.21.15.27:0/4053734340 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11a0150330 0x7f11a015a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:16.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.772+0000 7f11a5dff700 1 --2- 172.21.15.27:0/4053734340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11a01599e0 0x7f11a00b0130 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:16.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.772+0000 7f11a5dff700 1 -- 172.21.15.27:0/4053734340 >> 172.21.15.27:0/4053734340 conn(0x7f11a00680e0 msgr2=0x7f11a0069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:16.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.772+0000 7f11a5dff700 1 -- 172.21.15.27:0/4053734340 shutdown_connections 2024-01-11T09:47:16.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.772+0000 7f11a5dff700 1 -- 172.21.15.27:0/4053734340 wait complete. 2024-01-11T09:47:16.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.774+0000 7f11a5dff700 1 Processor -- start 2024-01-11T09:47:16.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.774+0000 7f11a5dff700 1 -- start start 2024-01-11T09:47:16.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.775+0000 7f11a5dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11a0150330 0x7f11a00ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:16.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.775+0000 7f11a5dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11a01599e0 0x7f11a01686b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:16.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.775+0000 7f11a5dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11a0159db0 0x7f11a00a7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:16.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.775+0000 7f11a5dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a00a9880 con 0x7f11a0150330 2024-01-11T09:47:16.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.775+0000 7f11a5dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a00a9b20 con 0x7f11a0159db0 2024-01-11T09:47:16.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.775+0000 7f11a5dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a00a9e00 con 0x7f11a01599e0 2024-01-11T09:47:16.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.775+0000 7f119f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11a0150330 0x7f11a00ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:16.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.775+0000 7f119f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11a0150330 0x7f11a00ae4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44184/0 (socket says 172.21.15.27:44184) 2024-01-11T09:47:16.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.775+0000 7f119f7fe700 1 -- 172.21.15.27:0/3821518268 learned_addr learned my addr 172.21.15.27:0/3821518268 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:16.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.775+0000 7f119effd700 1 --2- 172.21.15.27:0/3821518268 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11a01599e0 0x7f11a01686b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:16.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.775+0000 7f119ffff700 1 --2- 172.21.15.27:0/3821518268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11a0159db0 0x7f11a00a7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:16.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.775+0000 7f119f7fe700 1 -- 172.21.15.27:0/3821518268 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11a01599e0 msgr2=0x7f11a01686b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:16.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.775+0000 7f119f7fe700 1 --2- 172.21.15.27:0/3821518268 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11a01599e0 0x7f11a01686b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:16.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.776+0000 7f119f7fe700 1 -- 172.21.15.27:0/3821518268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11a0159db0 msgr2=0x7f11a00a7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:16.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.776+0000 7f119f7fe700 1 --2- 172.21.15.27:0/3821518268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11a0159db0 0x7f11a00a7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:16.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.776+0000 7f119f7fe700 1 -- 172.21.15.27:0/3821518268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11900094f0 con 0x7f11a0150330 2024-01-11T09:47:16.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.776+0000 7f119ffff700 1 --2- 172.21.15.27:0/3821518268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11a0159db0 0x7f11a00a7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:16.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.776+0000 7f119effd700 1 --2- 172.21.15.27:0/3821518268 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11a01599e0 0x7f11a01686b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:16.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.776+0000 7f119f7fe700 1 --2- 172.21.15.27:0/3821518268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11a0150330 0x7f11a00ae4c0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f11900090b0 tx=0x7f1190017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:16.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.776+0000 7f11877fe700 1 -- 172.21.15.27:0/3821518268 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1190026480 con 0x7f11a0150330 2024-01-11T09:47:16.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.776+0000 7f11877fe700 1 -- 172.21.15.27:0/3821518268 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119001f070 con 0x7f11a0150330 2024-01-11T09:47:16.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.776+0000 7f11a5dff700 1 -- 172.21.15.27:0/3821518268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a00aa0e0 con 0x7f11a0150330 2024-01-11T09:47:16.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.776+0000 7f11877fe700 1 -- 172.21.15.27:0/3821518268 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1190026850 con 0x7f11a0150330 2024-01-11T09:47:16.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.776+0000 7f11a5dff700 1 -- 172.21.15.27:0/3821518268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a00aea00 con 0x7f11a0150330 2024-01-11T09:47:16.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.777+0000 7f11a5dff700 1 -- 172.21.15.27:0/3821518268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a00718d0 con 0x7f11a0150330 2024-01-11T09:47:16.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.778+0000 7f11877fe700 1 -- 172.21.15.27:0/3821518268 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f119001a430 con 0x7f11a0150330 2024-01-11T09:47:16.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.779+0000 7f11877fe700 1 --2- 172.21.15.27:0/3821518268 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f118c067170 0x7f118c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:16.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.779+0000 7f11877fe700 1 -- 172.21.15.27:0/3821518268 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f1190021070 con 0x7f11a0150330 2024-01-11T09:47:16.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.780+0000 7f119effd700 1 --2- 172.21.15.27:0/3821518268 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f118c067170 0x7f118c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:16.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.780+0000 7f119effd700 1 --2- 172.21.15.27:0/3821518268 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f118c067170 0x7f118c069620 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1194001810 tx=0x7f1194009040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:16.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:16.784+0000 7f11877fe700 1 -- 172.21.15.27:0/3821518268 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1190053c30 con 0x7f11a0150330 2024-01-11T09:47:17.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.066+0000 7f11a5dff700 1 -- 172.21.15.27:0/3821518268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f11a00b0670 con 0x7f11a0150330 2024-01-11T09:47:17.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.550+0000 7f11877fe700 1 -- 172.21.15.27:0/3821518268 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3185) v1 ==== 158+0+0 (secure 0 0 0) 0x7f11900150e0 con 0x7f11a0150330 2024-01-11T09:47:17.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.553+0000 7f11a5dff700 1 -- 172.21.15.27:0/3821518268 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f118c067170 msgr2=0x7f118c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:17.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.553+0000 7f11a5dff700 1 --2- 172.21.15.27:0/3821518268 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f118c067170 0x7f118c069620 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1194001810 tx=0x7f1194009040 comp rx=0 tx=0).stop 2024-01-11T09:47:17.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.553+0000 7f11a5dff700 1 -- 172.21.15.27:0/3821518268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11a0150330 msgr2=0x7f11a00ae4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:17.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.553+0000 7f11a5dff700 1 --2- 172.21.15.27:0/3821518268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11a0150330 0x7f11a00ae4c0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f11900090b0 tx=0x7f1190017000 comp rx=0 tx=0).stop 2024-01-11T09:47:17.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.553+0000 7f11a5dff700 1 -- 172.21.15.27:0/3821518268 shutdown_connections 2024-01-11T09:47:17.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.553+0000 7f11a5dff700 1 --2- 172.21.15.27:0/3821518268 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f118c067170 0x7f118c069620 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:17.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.553+0000 7f11a5dff700 1 --2- 172.21.15.27:0/3821518268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11a0159db0 0x7f11a00a7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:17.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.553+0000 7f11a5dff700 1 --2- 172.21.15.27:0/3821518268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11a0150330 0x7f11a00ae4c0 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:17.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.553+0000 7f11a5dff700 1 --2- 172.21.15.27:0/3821518268 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f11a01599e0 0x7f11a01686b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:17.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.553+0000 7f11a5dff700 1 -- 172.21.15.27:0/3821518268 >> 172.21.15.27:0/3821518268 conn(0x7f11a00680e0 msgr2=0x7f11a01582c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:17.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.553+0000 7f11a5dff700 1 -- 172.21.15.27:0/3821518268 shutdown_connections 2024-01-11T09:47:17.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.553+0000 7f11a5dff700 1 -- 172.21.15.27:0/3821518268 wait complete. 2024-01-11T09:47:17.564 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-11T09:47:17.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.775+0000 7faae9dff700 1 Processor -- start 2024-01-11T09:47:17.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.775+0000 7faae9dff700 1 -- start start 2024-01-11T09:47:17.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.777+0000 7faae9dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faae4157410 0x7faae40ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:17.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.777+0000 7faae9dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faae4069540 0x7faae40c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:17.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.777+0000 7faae9dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae40699d0 con 0x7faae41577e0 2024-01-11T09:47:17.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.777+0000 7faae9dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faae4069cb0 con 0x7faae4069540 2024-01-11T09:47:17.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.777+0000 7faae9dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae4069f90 con 0x7faae4157410 2024-01-11T09:47:17.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.777+0000 7faae2ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7faae41577e0 0x7faae40ae000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33716/0 (socket says 172.21.15.27:33716) 2024-01-11T09:47:17.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.777+0000 7faae2ffd700 1 -- 172.21.15.27:0/1736821226 learned_addr learned my addr 172.21.15.27:0/1736821226 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:17.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.777+0000 7faae37fe700 1 --2- 172.21.15.27:0/1736821226 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faae4157410 0x7faae40ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.777+0000 7faae3fff700 1 --2- 172.21.15.27:0/1736821226 >> v2:172.21.15.77:3300/0 conn(0x7faae4069540 0x7faae40c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.778+0000 7faae27fc700 1 -- 172.21.15.27:0/1736821226 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180568383 0 0) 0x7faae40699d0 con 0x7faae41577e0 2024-01-11T09:47:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.778+0000 7faae27fc700 1 -- 172.21.15.27:0/1736821226 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faacc003570 con 0x7faae41577e0 2024-01-11T09:47:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.778+0000 7faae37fe700 1 -- 172.21.15.27:0/1736821226 >> v2:172.21.15.77:3300/0 conn(0x7faae4069540 msgr2=0x7faae40c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.778+0000 7faae37fe700 1 --2- 172.21.15.27:0/1736821226 >> v2:172.21.15.77:3300/0 conn(0x7faae4069540 0x7faae40c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.778+0000 7faae37fe700 1 -- 172.21.15.27:0/1736821226 >> v1:172.21.15.27:6789/0 conn(0x7faae41577e0 legacy=0x7faae40ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.778+0000 7faae37fe700 1 -- 172.21.15.27:0/1736821226 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae406a270 con 0x7faae4157410 2024-01-11T09:47:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.778+0000 7faae37fe700 1 --2- 172.21.15.27:0/1736821226 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faae4157410 0x7faae40ccae0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7faad40098d0 tx=0x7faad4009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4673b4d99301312 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.778+0000 7faae27fc700 1 -- 172.21.15.27:0/1736821226 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad4005c10 con 0x7faae4157410 2024-01-11T09:47:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.779+0000 7faae27fc700 1 -- 172.21.15.27:0/1736821226 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad4005d70 con 0x7faae4157410 2024-01-11T09:47:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.779+0000 7faae27fc700 1 -- 172.21.15.27:0/1736821226 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad4006260 con 0x7faae4157410 2024-01-11T09:47:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.779+0000 7faae9dff700 1 -- 172.21.15.27:0/1736821226 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faae4157410 msgr2=0x7faae40ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:17.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.779+0000 7faae9dff700 1 --2- 172.21.15.27:0/1736821226 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faae4157410 0x7faae40ccae0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7faad40098d0 tx=0x7faad4009120 comp rx=0 tx=0).stop 2024-01-11T09:47:17.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.779+0000 7faae9dff700 1 -- 172.21.15.27:0/1736821226 shutdown_connections 2024-01-11T09:47:17.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.779+0000 7faae9dff700 1 --2- 172.21.15.27:0/1736821226 >> v2:172.21.15.77:3300/0 conn(0x7faae4069540 0x7faae40c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:17.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.779+0000 7faae9dff700 1 --2- 172.21.15.27:0/1736821226 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faae4157410 0x7faae40ccae0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:17.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.779+0000 7faae9dff700 1 -- 172.21.15.27:0/1736821226 >> 172.21.15.27:0/1736821226 conn(0x7faae40680a0 msgr2=0x7faae40cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:17.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.779+0000 7faae9dff700 1 -- 172.21.15.27:0/1736821226 shutdown_connections 2024-01-11T09:47:17.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.779+0000 7faae9dff700 1 -- 172.21.15.27:0/1736821226 wait complete. 2024-01-11T09:47:17.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.781+0000 7faae9dff700 1 Processor -- start 2024-01-11T09:47:17.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.781+0000 7faae9dff700 1 -- start start 2024-01-11T09:47:17.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.782+0000 7faae9dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faae4069540 0x7faae4144b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:17.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.782+0000 7faae9dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faae4157410 0x7faae40ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:17.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.782+0000 7faae9dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae41577e0 0x7faae40b10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:17.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.782+0000 7faae9dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae4153730 con 0x7faae41577e0 2024-01-11T09:47:17.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.782+0000 7faae9dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae41539d0 con 0x7faae4157410 2024-01-11T09:47:17.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.782+0000 7faae9dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae40cb180 con 0x7faae4069540 2024-01-11T09:47:17.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.782+0000 7faae3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae41577e0 0x7faae40b10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:17.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.782+0000 7faae3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae41577e0 0x7faae40b10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44200/0 (socket says 172.21.15.27:44200) 2024-01-11T09:47:17.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.782+0000 7faae3fff700 1 -- 172.21.15.27:0/3769357962 learned_addr learned my addr 172.21.15.27:0/3769357962 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:17.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.782+0000 7faae2ffd700 1 --2- 172.21.15.27:0/3769357962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faae4157410 0x7faae40ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:17.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.782+0000 7faae37fe700 1 --2- 172.21.15.27:0/3769357962 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faae4069540 0x7faae4144b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:17.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.783+0000 7faae3fff700 1 -- 172.21.15.27:0/3769357962 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faae4069540 msgr2=0x7faae4144b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:17.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.783+0000 7faae3fff700 1 --2- 172.21.15.27:0/3769357962 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faae4069540 0x7faae4144b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:17.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.783+0000 7faae3fff700 1 -- 172.21.15.27:0/3769357962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faae4157410 msgr2=0x7faae40ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:17.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.783+0000 7faae3fff700 1 --2- 172.21.15.27:0/3769357962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faae4157410 0x7faae40ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:17.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.783+0000 7faae3fff700 1 -- 172.21.15.27:0/3769357962 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad40093c0 con 0x7faae41577e0 2024-01-11T09:47:17.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.783+0000 7faae37fe700 1 --2- 172.21.15.27:0/3769357962 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faae4069540 0x7faae4144b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:17.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.783+0000 7faae3fff700 1 --2- 172.21.15.27:0/3769357962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae41577e0 0x7faae40b10d0 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7faadc009ef0 tx=0x7faadc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:17.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.783+0000 7faacb7fe700 1 -- 172.21.15.27:0/3769357962 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc00cc20 con 0x7faae41577e0 2024-01-11T09:47:17.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.783+0000 7faacb7fe700 1 -- 172.21.15.27:0/3769357962 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faadc00f040 con 0x7faae41577e0 2024-01-11T09:47:17.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.783+0000 7faacb7fe700 1 -- 172.21.15.27:0/3769357962 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc014480 con 0x7faae41577e0 2024-01-11T09:47:17.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.783+0000 7faae9dff700 1 -- 172.21.15.27:0/3769357962 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae40cb460 con 0x7faae41577e0 2024-01-11T09:47:17.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.784+0000 7faae9dff700 1 -- 172.21.15.27:0/3769357962 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae41450b0 con 0x7faae41577e0 2024-01-11T09:47:17.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.784+0000 7faae9dff700 1 -- 172.21.15.27:0/3769357962 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae404c3c0 con 0x7faae41577e0 2024-01-11T09:47:17.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.785+0000 7faacb7fe700 1 -- 172.21.15.27:0/3769357962 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7faadc014830 con 0x7faae41577e0 2024-01-11T09:47:17.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.786+0000 7faacb7fe700 1 --2- 172.21.15.27:0/3769357962 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7faad0067170 0x7faad0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:17.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.786+0000 7faacb7fe700 1 -- 172.21.15.27:0/3769357962 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 5911+0+0 (secure 0 0 0) 0x7faadc085780 con 0x7faae41577e0 2024-01-11T09:47:17.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.787+0000 7faae37fe700 1 --2- 172.21.15.27:0/3769357962 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7faad0067170 0x7faad0069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:17.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.788+0000 7faae37fe700 1 --2- 172.21.15.27:0/3769357962 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7faad0067170 0x7faad0069620 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7faad4009160 tx=0x7faad4016000 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:17.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.790+0000 7faacb7fe700 1 -- 172.21.15.27:0/3769357962 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faadc052580 con 0x7faae41577e0 2024-01-11T09:47:17.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.992+0000 7faae9dff700 1 -- 172.21.15.27:0/3769357962 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7faae40af900 con 0x7faae41577e0 2024-01-11T09:47:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.996+0000 7faacb7fe700 1 -- 172.21.15.27:0/3769357962 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v88)=0 v88) v1 ==== 122+0+0 (secure 0 0 0) 0x7faadc055ba0 con 0x7faae41577e0 2024-01-11T09:47:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.999+0000 7faae9dff700 1 -- 172.21.15.27:0/3769357962 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7faad0067170 msgr2=0x7faad0069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.999+0000 7faae9dff700 1 --2- 172.21.15.27:0/3769357962 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7faad0067170 0x7faad0069620 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7faad4009160 tx=0x7faad4016000 comp rx=0 tx=0).stop 2024-01-11T09:47:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.999+0000 7faae9dff700 1 -- 172.21.15.27:0/3769357962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae41577e0 msgr2=0x7faae40b10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.999+0000 7faae9dff700 1 --2- 172.21.15.27:0/3769357962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae41577e0 0x7faae40b10d0 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7faadc009ef0 tx=0x7faadc00b040 comp rx=0 tx=0).stop 2024-01-11T09:47:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.999+0000 7faae9dff700 1 -- 172.21.15.27:0/3769357962 shutdown_connections 2024-01-11T09:47:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.999+0000 7faae9dff700 1 --2- 172.21.15.27:0/3769357962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faae41577e0 0x7faae40b10d0 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.999+0000 7faae9dff700 1 --2- 172.21.15.27:0/3769357962 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7faae4069540 0x7faae4144b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.999+0000 7faae9dff700 1 --2- 172.21.15.27:0/3769357962 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7faad0067170 0x7faad0069620 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.999+0000 7faae9dff700 1 --2- 172.21.15.27:0/3769357962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faae4157410 0x7faae40ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.999+0000 7faae9dff700 1 -- 172.21.15.27:0/3769357962 >> 172.21.15.27:0/3769357962 conn(0x7faae40680a0 msgr2=0x7faae4152a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.999+0000 7faae9dff700 1 -- 172.21.15.27:0/3769357962 shutdown_connections 2024-01-11T09:47:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:17.999+0000 7faae9dff700 1 -- 172.21.15.27:0/3769357962 wait complete. 2024-01-11T09:47:18.009 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T09:47:18.009 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:47:18.111 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:47:18.209 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T09:47:18.209 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-11T09:47:18.209 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:47:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.416+0000 7f57d65ff700 1 Processor -- start 2024-01-11T09:47:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.416+0000 7f57d65ff700 1 -- start start 2024-01-11T09:47:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.417+0000 7f57d65ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f57d00b03e0 0x7f57d00b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.417+0000 7f57d65ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f57d00b07b0 0x7f57d00b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.417+0000 7f57d65ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d00cd730 con 0x7f57d0159200 2024-01-11T09:47:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.417+0000 7f57d65ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57d0159d50 con 0x7f57d00b07b0 2024-01-11T09:47:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.417+0000 7f57d65ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d00ccb20 con 0x7f57d00b03e0 2024-01-11T09:47:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.418+0000 7f57cffff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f57d00b03e0 0x7f57d00b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.418+0000 7f57cffff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f57d00b03e0 0x7f57d00b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:51632/0 (socket says 172.21.15.27:51632) 2024-01-11T09:47:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.418+0000 7f57cffff700 1 -- 172.21.15.27:0/723153128 learned_addr learned my addr 172.21.15.27:0/723153128 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.418+0000 7f57cf7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f57d00b07b0 0x7f57d00b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:18.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.418+0000 7f57cffff700 1 -- 172.21.15.27:0/723153128 >> v2:172.21.15.77:3300/0 conn(0x7f57d00b07b0 msgr2=0x7f57d00b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:18.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.418+0000 7f57cffff700 1 --2- 172.21.15.27:0/723153128 >> v2:172.21.15.77:3300/0 conn(0x7f57d00b07b0 0x7f57d00b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:18.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.418+0000 7f57cffff700 1 -- 172.21.15.27:0/723153128 >> v1:172.21.15.77:6789/0 conn(0x7f57d0159200 legacy=0x7f57d0159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:18.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.418+0000 7f57cffff700 1 -- 172.21.15.27:0/723153128 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d00cce00 con 0x7f57d00b03e0 2024-01-11T09:47:18.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.418+0000 7f57cffff700 1 --2- 172.21.15.27:0/723153128 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f57d00b03e0 0x7f57d00b2480 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f57c00099b0 tx=0x7f57c0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c899c5922c903075 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:18.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.419+0000 7f57ceffd700 1 -- 172.21.15.27:0/723153128 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c0009ea0 con 0x7f57d00b03e0 2024-01-11T09:47:18.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.419+0000 7f57ceffd700 1 -- 172.21.15.27:0/723153128 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c000b860 con 0x7f57d00b03e0 2024-01-11T09:47:18.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.419+0000 7f57ceffd700 1 -- 172.21.15.27:0/723153128 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c000bc00 con 0x7f57d00b03e0 2024-01-11T09:47:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.419+0000 7f57d65ff700 1 -- 172.21.15.27:0/723153128 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f57d00b03e0 msgr2=0x7f57d00b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.419+0000 7f57d65ff700 1 --2- 172.21.15.27:0/723153128 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f57d00b03e0 0x7f57d00b2480 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f57c00099b0 tx=0x7f57c0009250 comp rx=0 tx=0).stop 2024-01-11T09:47:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.419+0000 7f57d65ff700 1 -- 172.21.15.27:0/723153128 shutdown_connections 2024-01-11T09:47:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.419+0000 7f57d65ff700 1 --2- 172.21.15.27:0/723153128 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f57d00b03e0 0x7f57d00b2480 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.419+0000 7f57d65ff700 1 --2- 172.21.15.27:0/723153128 >> v2:172.21.15.77:3300/0 conn(0x7f57d00b07b0 0x7f57d00b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.419+0000 7f57d65ff700 1 -- 172.21.15.27:0/723153128 >> 172.21.15.27:0/723153128 conn(0x7f57d006ed40 msgr2=0x7f57d006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.419+0000 7f57d65ff700 1 -- 172.21.15.27:0/723153128 shutdown_connections 2024-01-11T09:47:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.419+0000 7f57d65ff700 1 -- 172.21.15.27:0/723153128 wait complete. 2024-01-11T09:47:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.421+0000 7f57d65ff700 1 Processor -- start 2024-01-11T09:47:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.421+0000 7f57d65ff700 1 -- start start 2024-01-11T09:47:18.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.422+0000 7f57d65ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d00b03e0 0x7f57d00ae760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.422+0000 7f57d65ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f57d00b07b0 0x7f57d00c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.422+0000 7f57d65ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57d0159200 0x7f57d00cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.422+0000 7f57d65ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d0169e90 con 0x7f57d00b03e0 2024-01-11T09:47:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.422+0000 7f57d65ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d00c9910 con 0x7f57d0159200 2024-01-11T09:47:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.422+0000 7f57d65ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d00c9bf0 con 0x7f57d00b07b0 2024-01-11T09:47:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.422+0000 7f57cffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d00b03e0 0x7f57d00ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.422+0000 7f57cffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d00b03e0 0x7f57d00ae760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44210/0 (socket says 172.21.15.27:44210) 2024-01-11T09:47:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.422+0000 7f57cffff700 1 -- 172.21.15.27:0/3699540709 learned_addr learned my addr 172.21.15.27:0/3699540709 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.422+0000 7f57cf7fe700 1 --2- 172.21.15.27:0/3699540709 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f57d00b07b0 0x7f57d00c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.423+0000 7f57cffff700 1 -- 172.21.15.27:0/3699540709 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f57d00b07b0 msgr2=0x7f57d00c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.423+0000 7f57d4b9c700 1 --2- 172.21.15.27:0/3699540709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57d0159200 0x7f57d00cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.423+0000 7f57cffff700 1 --2- 172.21.15.27:0/3699540709 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f57d00b07b0 0x7f57d00c8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:18.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.423+0000 7f57cffff700 1 -- 172.21.15.27:0/3699540709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57d0159200 msgr2=0x7f57d00cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:18.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.423+0000 7f57cffff700 1 --2- 172.21.15.27:0/3699540709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57d0159200 0x7f57d00cefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:18.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.423+0000 7f57cffff700 1 -- 172.21.15.27:0/3699540709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c00094f0 con 0x7f57d00b03e0 2024-01-11T09:47:18.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.423+0000 7f57cf7fe700 1 --2- 172.21.15.27:0/3699540709 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f57d00b07b0 0x7f57d00c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:18.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.423+0000 7f57d4b9c700 1 --2- 172.21.15.27:0/3699540709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57d0159200 0x7f57d00cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:18.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.423+0000 7f57cffff700 1 --2- 172.21.15.27:0/3699540709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d00b03e0 0x7f57d00ae760 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f57d006fd20 tx=0x7f57c0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:18.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.423+0000 7f57b7fff700 1 -- 172.21.15.27:0/3699540709 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c000bbe0 con 0x7f57d00b03e0 2024-01-11T09:47:18.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.423+0000 7f57b7fff700 1 -- 172.21.15.27:0/3699540709 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c0011420 con 0x7f57d00b03e0 2024-01-11T09:47:18.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.423+0000 7f57d65ff700 1 -- 172.21.15.27:0/3699540709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d00c9ed0 con 0x7f57d00b03e0 2024-01-11T09:47:18.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.423+0000 7f57b7fff700 1 -- 172.21.15.27:0/3699540709 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c00117c0 con 0x7f57d00b03e0 2024-01-11T09:47:18.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.423+0000 7f57d65ff700 1 -- 172.21.15.27:0/3699540709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d00aeca0 con 0x7f57d00b03e0 2024-01-11T09:47:18.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.424+0000 7f57d65ff700 1 -- 172.21.15.27:0/3699540709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d00718d0 con 0x7f57d00b03e0 2024-01-11T09:47:18.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.425+0000 7f57b7fff700 1 -- 172.21.15.27:0/3699540709 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f57c0011bf0 con 0x7f57d00b03e0 2024-01-11T09:47:18.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.426+0000 7f57b7fff700 1 --2- 172.21.15.27:0/3699540709 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f57bc067170 0x7f57bc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:18.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.426+0000 7f57b7fff700 1 -- 172.21.15.27:0/3699540709 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f57c00863b0 con 0x7f57d00b03e0 2024-01-11T09:47:18.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.427+0000 7f57cf7fe700 1 --2- 172.21.15.27:0/3699540709 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f57bc067170 0x7f57bc069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:18.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.427+0000 7f57cf7fe700 1 --2- 172.21.15.27:0/3699540709 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f57bc067170 0x7f57bc069620 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f57c4006f00 tx=0x7f57c4008040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:18.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.430+0000 7f57b7fff700 1 -- 172.21.15.27:0/3699540709 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f57c00532c0 con 0x7f57d00b03e0 2024-01-11T09:47:18.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.676+0000 7f57d65ff700 1 -- 172.21.15.27:0/3699540709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57d00cd0e0 con 0x7f57d00b03e0 2024-01-11T09:47:18.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.677+0000 7f57b7fff700 1 -- 172.21.15.27:0/3699540709 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+3536 (secure 0 0 0) 0x7f57c0015de0 con 0x7f57d00b03e0 2024-01-11T09:47:18.669 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:18.669 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":136,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1568813392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1568813392},{"type":"v1","addr":"172.21.15.77:6834","nonce":1568813392}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122},{"gid":9740,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3700058093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":3700058093},{"type":"v1","addr":"172.21.15.27:6835","nonce":3700058093}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":134},{"gid":10969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1352491019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1352491019},{"type":"v1","addr":"172.21.15.77:6835","nonce":1352491019}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":122},{"gid":11976,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1809173098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1809173098},{"type":"v1","addr":"172.21.15.27:6834","nonce":1809173098}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":136}],"filesystems":[]} 2024-01-11T09:47:18.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.679+0000 7f57d65ff700 1 -- 172.21.15.27:0/3699540709 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f57bc067170 msgr2=0x7f57bc069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:18.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.679+0000 7f57d65ff700 1 --2- 172.21.15.27:0/3699540709 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f57bc067170 0x7f57bc069620 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f57c4006f00 tx=0x7f57c4008040 comp rx=0 tx=0).stop 2024-01-11T09:47:18.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.679+0000 7f57d65ff700 1 -- 172.21.15.27:0/3699540709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d00b03e0 msgr2=0x7f57d00ae760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:18.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.679+0000 7f57d65ff700 1 --2- 172.21.15.27:0/3699540709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d00b03e0 0x7f57d00ae760 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f57d006fd20 tx=0x7f57c0017000 comp rx=0 tx=0).stop 2024-01-11T09:47:18.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.679+0000 7f57d65ff700 1 -- 172.21.15.27:0/3699540709 shutdown_connections 2024-01-11T09:47:18.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.679+0000 7f57d65ff700 1 --2- 172.21.15.27:0/3699540709 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f57bc067170 0x7f57bc069620 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:18.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.679+0000 7f57d65ff700 1 --2- 172.21.15.27:0/3699540709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57d0159200 0x7f57d00cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:18.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.679+0000 7f57d65ff700 1 --2- 172.21.15.27:0/3699540709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d00b03e0 0x7f57d00ae760 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:18.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.679+0000 7f57d65ff700 1 --2- 172.21.15.27:0/3699540709 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f57d00b07b0 0x7f57d00c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:18.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.679+0000 7f57d65ff700 1 -- 172.21.15.27:0/3699540709 >> 172.21.15.27:0/3699540709 conn(0x7f57d006ed40 msgr2=0x7f57d00b2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:18.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.679+0000 7f57d65ff700 1 -- 172.21.15.27:0/3699540709 shutdown_connections 2024-01-11T09:47:18.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.679+0000 7f57d65ff700 1 -- 172.21.15.27:0/3699540709 wait complete. 2024-01-11T09:47:18.673 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 136 2024-01-11T09:47:18.689 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-11T09:47:18.689 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-11T09:47:18.690 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-11T09:47:18.690 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:47:18.690 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-11T09:47:18.690 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-11T09:47:18.690 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-11T09:47:18.690 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:47:18.691 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-11T09:47:18.691 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-11T09:47:18.691 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-11T09:47:18.691 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:47:18.691 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-11T09:47:18.691 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-11T09:47:18.691 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-11T09:47:18.691 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:47:18.703 INFO:tasks.ceph.mds.c:Stopped 2024-01-11T09:47:18.704 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-11T09:47:18.704 INFO:tasks.ceph.mds.a:Stopped 2024-01-11T09:47:18.705 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-11T09:47:18.707 INFO:tasks.ceph.mds.b:Stopped 2024-01-11T09:47:18.707 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-11T09:47:18.708 INFO:tasks.ceph.mds.d:Stopped 2024-01-11T09:47:18.708 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-11T09:47:18.749 INFO:tasks.ceph.mds.c:Started 2024-01-11T09:47:18.749 DEBUG:teuthology.parallel:result is None 2024-01-11T09:47:18.752 INFO:tasks.ceph.mds.a:Started 2024-01-11T09:47:18.752 DEBUG:teuthology.parallel:result is None 2024-01-11T09:47:18.753 INFO:tasks.ceph.mds.b:Started 2024-01-11T09:47:18.753 DEBUG:teuthology.parallel:result is None 2024-01-11T09:47:18.756 INFO:tasks.ceph.mds.d:Started 2024-01-11T09:47:18.757 DEBUG:teuthology.parallel:result is None 2024-01-11T09:47:18.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-11T09:47:18.903 INFO:tasks.ceph.mds.b.smithi077.stdout:starting mds.b at 2024-01-11T09:47:18.904 INFO:tasks.ceph.mds.d.smithi077.stdout:starting mds.d at 2024-01-11T09:47:18.906 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-11T09:47:18.909 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T09:47:18.918 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:47:18.922 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:47:18.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.982+0000 7f6e265ff700 1 Processor -- start 2024-01-11T09:47:18.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.982+0000 7f6e265ff700 1 -- start start 2024-01-11T09:47:18.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.983+0000 7f6e265ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6e200d0a80 0x7f6e200ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:18.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.983+0000 7f6e265ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e200cf660 con 0x7f6e200d06b0 2024-01-11T09:47:18.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.983+0000 7f6e265ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2015e1f0 con 0x7f6e2015dd70 2024-01-11T09:47:18.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.983+0000 7f6e265ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e200cd650 con 0x7f6e200d0a80 2024-01-11T09:47:18.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.983+0000 7f6e1ffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6e200d06b0 0x7f6e200cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33726/0 (socket says 172.21.15.27:33726) 2024-01-11T09:47:18.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.984+0000 7f6e1ffff700 1 -- 172.21.15.27:0/45243535 learned_addr learned my addr 172.21.15.27:0/45243535 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:18.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.984+0000 7f6e1f7fe700 1 --2- 172.21.15.27:0/45243535 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6e200d0a80 0x7f6e200ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:18.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.984+0000 7f6e1effd700 1 -- 172.21.15.27:0/45243535 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398102571 0 0) 0x7f6e200cf660 con 0x7f6e200d06b0 2024-01-11T09:47:18.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.984+0000 7f6e1effd700 1 -- 172.21.15.27:0/45243535 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e08003570 con 0x7f6e200d06b0 2024-01-11T09:47:18.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.984+0000 7f6e1effd700 1 -- 172.21.15.27:0/45243535 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462577059 0 0) 0x7f6e2015e1f0 con 0x7f6e2015dd70 2024-01-11T09:47:18.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.984+0000 7f6e1f7fe700 1 -- 172.21.15.27:0/45243535 >> v1:172.21.15.77:6789/0 conn(0x7f6e2015dd70 legacy=0x7f6e200b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:18.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.984+0000 7f6e1f7fe700 1 -- 172.21.15.27:0/45243535 >> v1:172.21.15.27:6789/0 conn(0x7f6e200d06b0 legacy=0x7f6e200cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:18.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.984+0000 7f6e1f7fe700 1 -- 172.21.15.27:0/45243535 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e200ced30 con 0x7f6e200d0a80 2024-01-11T09:47:18.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.984+0000 7f6e1f7fe700 1 --2- 172.21.15.27:0/45243535 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6e200d0a80 0x7f6e200ccf50 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009910 tx=0x7f6e140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a435f2c692d67682 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:18.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.985+0000 7f6e1effd700 1 -- 172.21.15.27:0/45243535 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1400b680 con 0x7f6e200d0a80 2024-01-11T09:47:18.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.985+0000 7f6e1effd700 1 -- 172.21.15.27:0/45243535 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1400b7e0 con 0x7f6e200d0a80 2024-01-11T09:47:18.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.985+0000 7f6e1effd700 1 -- 172.21.15.27:0/45243535 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1400bba0 con 0x7f6e200d0a80 2024-01-11T09:47:18.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.985+0000 7f6e265ff700 1 -- 172.21.15.27:0/45243535 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6e200d0a80 msgr2=0x7f6e200ccf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:18.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.985+0000 7f6e265ff700 1 --2- 172.21.15.27:0/45243535 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6e200d0a80 0x7f6e200ccf50 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009910 tx=0x7f6e140091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:18.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.985+0000 7f6e265ff700 1 -- 172.21.15.27:0/45243535 shutdown_connections 2024-01-11T09:47:18.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.985+0000 7f6e265ff700 1 --2- 172.21.15.27:0/45243535 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6e200d0a80 0x7f6e200ccf50 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:18.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.985+0000 7f6e265ff700 1 -- 172.21.15.27:0/45243535 >> 172.21.15.27:0/45243535 conn(0x7f6e2006ed40 msgr2=0x7f6e2006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:18.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.986+0000 7f6e265ff700 1 -- 172.21.15.27:0/45243535 shutdown_connections 2024-01-11T09:47:18.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.986+0000 7f6e265ff700 1 -- 172.21.15.27:0/45243535 wait complete. 2024-01-11T09:47:18.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.987+0000 7f6e265ff700 1 Processor -- start 2024-01-11T09:47:18.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.987+0000 7f6e265ff700 1 -- start start 2024-01-11T09:47:18.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.988+0000 7f6e265ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6e200d06b0 0x7f6e200aef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:18.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.988+0000 7f6e265ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e200d0a80 0x7f6e200a84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.988+0000 7f6e265ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e2015dd70 0x7f6e200aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.988+0000 7f6e265ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e20153cc0 con 0x7f6e2015dd70 2024-01-11T09:47:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.988+0000 7f6e265ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e200aacb0 con 0x7f6e200d0a80 2024-01-11T09:47:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.988+0000 7f6e265ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e200aaa10 con 0x7f6e200d06b0 2024-01-11T09:47:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.989+0000 7f6e1ffff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6e200d06b0 0x7f6e200aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.989+0000 7f6e1ffff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6e200d06b0 0x7f6e200aef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:51692/0 (socket says 172.21.15.27:51692) 2024-01-11T09:47:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.989+0000 7f6e1ffff700 1 -- 172.21.15.27:0/1771756512 learned_addr learned my addr 172.21.15.27:0/1771756512 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.989+0000 7f6e1f7fe700 1 --2- 172.21.15.27:0/1771756512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e200d0a80 0x7f6e200a84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.989+0000 7f6e24b9c700 1 --2- 172.21.15.27:0/1771756512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e2015dd70 0x7f6e200aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.989+0000 7f6e1ffff700 1 -- 172.21.15.27:0/1771756512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e200d0a80 msgr2=0x7f6e200a84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.989+0000 7f6e1ffff700 1 --2- 172.21.15.27:0/1771756512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e200d0a80 0x7f6e200a84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.989+0000 7f6e1ffff700 1 -- 172.21.15.27:0/1771756512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e2015dd70 msgr2=0x7f6e200aa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.989+0000 7f6e1ffff700 1 --2- 172.21.15.27:0/1771756512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e2015dd70 0x7f6e200aa4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.989+0000 7f6e1ffff700 1 -- 172.21.15.27:0/1771756512 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e14009420 con 0x7f6e200d06b0 2024-01-11T09:47:18.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.989+0000 7f6e1ffff700 1 --2- 172.21.15.27:0/1771756512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6e200d06b0 0x7f6e200aef90 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6e100011e0 tx=0x7f6e10007df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:18.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.990+0000 7f6e07fff700 1 -- 172.21.15.27:0/1771756512 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e10005b30 con 0x7f6e200d06b0 2024-01-11T09:47:18.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.990+0000 7f6e07fff700 1 -- 172.21.15.27:0/1771756512 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e10005c90 con 0x7f6e200d06b0 2024-01-11T09:47:18.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.990+0000 7f6e265ff700 1 -- 172.21.15.27:0/1771756512 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e2015b070 con 0x7f6e200d06b0 2024-01-11T09:47:18.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.990+0000 7f6e07fff700 1 -- 172.21.15.27:0/1771756512 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e10007300 con 0x7f6e200d06b0 2024-01-11T09:47:18.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.990+0000 7f6e265ff700 1 -- 172.21.15.27:0/1771756512 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e2015b6b0 con 0x7f6e200d06b0 2024-01-11T09:47:18.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.991+0000 7f6e265ff700 1 -- 172.21.15.27:0/1771756512 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e2004c3c0 con 0x7f6e200d06b0 2024-01-11T09:47:18.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.993+0000 7f6e07fff700 1 -- 172.21.15.27:0/1771756512 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f6e10007670 con 0x7f6e200d06b0 2024-01-11T09:47:18.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.994+0000 7f6e07fff700 1 --2- 172.21.15.27:0/1771756512 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6e0c067170 0x7f6e0c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:18.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.994+0000 7f6e07fff700 1 -- 172.21.15.27:0/1771756512 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f6e10083e10 con 0x7f6e200d06b0 2024-01-11T09:47:18.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.994+0000 7f6e1f7fe700 1 --2- 172.21.15.27:0/1771756512 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6e0c067170 0x7f6e0c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:18.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:18.995+0000 7f6e1f7fe700 1 --2- 172.21.15.27:0/1771756512 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6e0c067170 0x7f6e0c069620 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6e140093f0 tx=0x7f6e14017000 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:18.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:19.001+0000 7f6e07fff700 1 -- 172.21.15.27:0/1771756512 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6e10050c50 con 0x7f6e200d06b0 2024-01-11T09:47:19.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:19.225+0000 7f6e265ff700 1 -- 172.21.15.27:0/1771756512 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f6e200b11c0 con 0x7f6e200d06b0 2024-01-11T09:47:20.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.020+0000 7f6e07fff700 1 -- 172.21.15.27:0/1771756512 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v252) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6e10017020 con 0x7f6e200d06b0 2024-01-11T09:47:20.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.023+0000 7f6e265ff700 1 -- 172.21.15.27:0/1771756512 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6e0c067170 msgr2=0x7f6e0c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:20.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.023+0000 7f6e265ff700 1 --2- 172.21.15.27:0/1771756512 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6e0c067170 0x7f6e0c069620 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6e140093f0 tx=0x7f6e14017000 comp rx=0 tx=0).stop 2024-01-11T09:47:20.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.023+0000 7f6e265ff700 1 -- 172.21.15.27:0/1771756512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6e200d06b0 msgr2=0x7f6e200aef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:20.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.023+0000 7f6e265ff700 1 --2- 172.21.15.27:0/1771756512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6e200d06b0 0x7f6e200aef90 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6e100011e0 tx=0x7f6e10007df0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.023+0000 7f6e265ff700 1 -- 172.21.15.27:0/1771756512 shutdown_connections 2024-01-11T09:47:20.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.023+0000 7f6e265ff700 1 --2- 172.21.15.27:0/1771756512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e2015dd70 0x7f6e200aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.023+0000 7f6e265ff700 1 --2- 172.21.15.27:0/1771756512 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6e200d06b0 0x7f6e200aef90 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.023+0000 7f6e265ff700 1 --2- 172.21.15.27:0/1771756512 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6e0c067170 0x7f6e0c069620 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.023+0000 7f6e265ff700 1 --2- 172.21.15.27:0/1771756512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e200d0a80 0x7f6e200a84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.023+0000 7f6e265ff700 1 -- 172.21.15.27:0/1771756512 >> 172.21.15.27:0/1771756512 conn(0x7f6e2006ed40 msgr2=0x7f6e200b5790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:20.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.023+0000 7f6e265ff700 1 -- 172.21.15.27:0/1771756512 shutdown_connections 2024-01-11T09:47:20.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.024+0000 7f6e265ff700 1 -- 172.21.15.27:0/1771756512 wait complete. 2024-01-11T09:47:20.017 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-11T09:47:20.037 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-11T09:47:20.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.249+0000 7f68741ff700 1 Processor -- start 2024-01-11T09:47:20.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.249+0000 7f68741ff700 1 -- start start 2024-01-11T09:47:20.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.251+0000 7f68741ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f686c069d40 0x7f686c0b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:20.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.251+0000 7f68741ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686c157390 con 0x7f686c0c89f0 2024-01-11T09:47:20.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.251+0000 7f68741ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686c06a200 con 0x7f686c0c8620 2024-01-11T09:47:20.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.251+0000 7f68741ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f686c156d40 con 0x7f686c069d40 2024-01-11T09:47:20.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.251+0000 7f687179a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f686c069d40 0x7f686c0b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:20.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.251+0000 7f687279c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f686c0c89f0 0x7f686c0accc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33730/0 (socket says 172.21.15.27:33730) 2024-01-11T09:47:20.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.251+0000 7f687279c700 1 -- 172.21.15.27:0/3397086736 learned_addr learned my addr 172.21.15.27:0/3397086736 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:20.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.252+0000 7f687179a700 1 -- 172.21.15.27:0/3397086736 >> v1:172.21.15.77:6789/0 conn(0x7f686c0c8620 legacy=0x7f686c0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:20.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.252+0000 7f687179a700 1 -- 172.21.15.27:0/3397086736 >> v1:172.21.15.27:6789/0 conn(0x7f686c0c89f0 legacy=0x7f686c0accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:20.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.252+0000 7f687179a700 1 -- 172.21.15.27:0/3397086736 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686c157020 con 0x7f686c069d40 2024-01-11T09:47:20.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.252+0000 7f6870f99700 1 -- 172.21.15.27:0/3397086736 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403945404 0 0) 0x7f686c157390 con 0x7f686c0c89f0 2024-01-11T09:47:20.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.252+0000 7f687179a700 1 --2- 172.21.15.27:0/3397086736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686c069d40 0x7f686c0b02d0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f685c0098e0 tx=0x7f685c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5dc503aaebfd7bb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:20.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.252+0000 7f6870f99700 1 -- 172.21.15.27:0/3397086736 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c009e80 con 0x7f686c069d40 2024-01-11T09:47:20.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.252+0000 7f6870f99700 1 -- 172.21.15.27:0/3397086736 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685c00b860 con 0x7f686c069d40 2024-01-11T09:47:20.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.252+0000 7f6870f99700 1 -- 172.21.15.27:0/3397086736 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c00bbe0 con 0x7f686c069d40 2024-01-11T09:47:20.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.253+0000 7f68741ff700 1 -- 172.21.15.27:0/3397086736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686c069d40 msgr2=0x7f686c0b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:20.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.253+0000 7f68741ff700 1 --2- 172.21.15.27:0/3397086736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686c069d40 0x7f686c0b02d0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f685c0098e0 tx=0x7f685c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.253+0000 7f68741ff700 1 -- 172.21.15.27:0/3397086736 shutdown_connections 2024-01-11T09:47:20.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.253+0000 7f68741ff700 1 --2- 172.21.15.27:0/3397086736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686c069d40 0x7f686c0b02d0 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.253+0000 7f68741ff700 1 -- 172.21.15.27:0/3397086736 >> 172.21.15.27:0/3397086736 conn(0x7f686c0680e0 msgr2=0x7f686c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:20.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.253+0000 7f68741ff700 1 -- 172.21.15.27:0/3397086736 shutdown_connections 2024-01-11T09:47:20.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.253+0000 7f68741ff700 1 -- 172.21.15.27:0/3397086736 wait complete. 2024-01-11T09:47:20.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.254+0000 7f68741ff700 1 Processor -- start 2024-01-11T09:47:20.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.255+0000 7f68741ff700 1 -- start start 2024-01-11T09:47:20.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.255+0000 7f68741ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f686c069d40 0x7f686c14b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:20.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.255+0000 7f68741ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f686c0c8620 0x7f686c157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:20.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.255+0000 7f68741ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686c0c89f0 0x7f686c1581a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:20.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.255+0000 7f68741ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686c15ab90 con 0x7f686c0c89f0 2024-01-11T09:47:20.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.255+0000 7f68741ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686c159f80 con 0x7f686c069d40 2024-01-11T09:47:20.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.255+0000 7f68741ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f686c15a260 con 0x7f686c0c8620 2024-01-11T09:47:20.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.256+0000 7f687279c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686c0c89f0 0x7f686c1581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:20.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.256+0000 7f687279c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686c0c89f0 0x7f686c1581a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44248/0 (socket says 172.21.15.27:44248) 2024-01-11T09:47:20.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.256+0000 7f687279c700 1 -- 172.21.15.27:0/265644937 learned_addr learned my addr 172.21.15.27:0/265644937 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:20.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.256+0000 7f6871f9b700 1 --2- 172.21.15.27:0/265644937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f686c069d40 0x7f686c14b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:20.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.256+0000 7f687179a700 1 --2- 172.21.15.27:0/265644937 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f686c0c8620 0x7f686c157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:20.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.256+0000 7f687279c700 1 -- 172.21.15.27:0/265644937 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f686c0c8620 msgr2=0x7f686c157c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:20.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.256+0000 7f687279c700 1 --2- 172.21.15.27:0/265644937 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f686c0c8620 0x7f686c157c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.256+0000 7f687279c700 1 -- 172.21.15.27:0/265644937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f686c069d40 msgr2=0x7f686c14b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:20.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.256+0000 7f687279c700 1 --2- 172.21.15.27:0/265644937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f686c069d40 0x7f686c14b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.256+0000 7f687279c700 1 -- 172.21.15.27:0/265644937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f685c009420 con 0x7f686c0c89f0 2024-01-11T09:47:20.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.256+0000 7f687279c700 1 --2- 172.21.15.27:0/265644937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686c0c89f0 0x7f686c1581a0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f686400ccc0 tx=0x7f6864001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:20.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.257+0000 7f6861ffb700 1 -- 172.21.15.27:0/265644937 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686400a880 con 0x7f686c0c89f0 2024-01-11T09:47:20.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.257+0000 7f6861ffb700 1 -- 172.21.15.27:0/265644937 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686400a9e0 con 0x7f686c0c89f0 2024-01-11T09:47:20.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.257+0000 7f68741ff700 1 -- 172.21.15.27:0/265644937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f686c15a540 con 0x7f686c0c89f0 2024-01-11T09:47:20.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.257+0000 7f6861ffb700 1 -- 172.21.15.27:0/265644937 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686400ad80 con 0x7f686c0c89f0 2024-01-11T09:47:20.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.257+0000 7f68741ff700 1 -- 172.21.15.27:0/265644937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f686c14b830 con 0x7f686c0c89f0 2024-01-11T09:47:20.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.258+0000 7f68741ff700 1 -- 172.21.15.27:0/265644937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f686c0718d0 con 0x7f686c0c89f0 2024-01-11T09:47:20.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.259+0000 7f6861ffb700 1 -- 172.21.15.27:0/265644937 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f6864007300 con 0x7f686c0c89f0 2024-01-11T09:47:20.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.260+0000 7f6861ffb700 1 --2- 172.21.15.27:0/265644937 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6850067170 0x7f6850069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:20.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.260+0000 7f6861ffb700 1 -- 172.21.15.27:0/265644937 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f6864084970 con 0x7f686c0c89f0 2024-01-11T09:47:20.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.260+0000 7f6871f9b700 1 --2- 172.21.15.27:0/265644937 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6850067170 0x7f6850069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:20.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.261+0000 7f6871f9b700 1 --2- 172.21.15.27:0/265644937 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6850067170 0x7f6850069620 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6868003a80 tx=0x7f6868007ec0 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:20.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.264+0000 7f6861ffb700 1 -- 172.21.15.27:0/265644937 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6864051c10 con 0x7f686c0c89f0 2024-01-11T09:47:20.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.542+0000 7f68741ff700 1 -- 172.21.15.27:0/265644937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f686c0b1150 con 0x7f686c0c89f0 2024-01-11T09:47:20.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.543+0000 7f6861ffb700 1 -- 172.21.15.27:0/265644937 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v52) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f6864055230 con 0x7f686c0c89f0 2024-01-11T09:47:20.535 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:20.535 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:47:20.535 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-11T09:47:20.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:20.535 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-11T09:47:20.536 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lQm3yLhAAv6erz5hMfat8EUzRA5luMA==", 2024-01-11T09:47:20.536 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:47:20.536 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:47:20.536 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:47:20.536 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:47:20.536 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:47:20.536 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:47:20.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:20.536 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-11T09:47:20.536 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9l+FG8ChAACHndtVLx7qS3sidPniSycA==", 2024-01-11T09:47:20.536 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:47:20.536 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:47:20.536 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:47:20.537 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:47:20.537 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:47:20.537 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:47:20.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:20.537 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-11T09:47:20.537 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l1NnEOhAALHXsc5es7hufCthGdRzkyw==", 2024-01-11T09:47:20.537 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:47:20.537 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:47:20.537 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:47:20.537 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:47:20.537 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:47:20.537 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:47:20.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:20.538 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-11T09:47:20.538 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lruoWFhAAm8AmrZtTJhYI1+19F79RPQ==", 2024-01-11T09:47:20.538 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:47:20.538 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:47:20.538 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:47:20.538 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:47:20.538 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:47:20.538 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:47:20.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:20.538 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-11T09:47:20.538 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA2op9ltZygORAAZr1X1nCN0ce8rIVaGDpFyQ==", 2024-01-11T09:47:20.538 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:47:20.539 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:47:20.539 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:47:20.539 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:47:20.539 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:47:20.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:20.540 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-11T09:47:20.540 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA6op9lqSRPAhAAmtyCsyK5ZVGX2wfhOCMuFQ==", 2024-01-11T09:47:20.540 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:47:20.540 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:47:20.540 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:47:20.540 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:47:20.540 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:47:20.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:20.541 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-11T09:47:20.541 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQA9op9lZcs8BhAA8j+U+jPOoy+fliVWZAPFYg==", 2024-01-11T09:47:20.541 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:47:20.541 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:47:20.541 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:47:20.541 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:47:20.541 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:47:20.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:20.541 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-11T09:47:20.541 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBAop9lFuCUChAAP6si/2W1mxH4ecQJvuumkw==", 2024-01-11T09:47:20.541 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:47:20.542 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:47:20.542 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:47:20.542 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:47:20.542 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:47:20.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:20.542 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-11T09:47:20.542 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBGop9lOpwBIRAAcpcsjwmX2IldcKHJWAgHuA==", 2024-01-11T09:47:20.542 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:47:20.542 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:47:20.542 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:47:20.542 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:47:20.543 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:47:20.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:20.543 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-11T09:47:20.543 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBJop9l4q8hJRAAK8mAYVRQMoWS6wBOU69HmA==", 2024-01-11T09:47:20.543 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:47:20.543 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:47:20.543 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:47:20.543 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:47:20.543 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:47:20.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:20.543 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-11T09:47:20.543 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBMop9lqMF9KBAAFgFFI2GWt9AGif4AoScThw==", 2024-01-11T09:47:20.544 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:47:20.544 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:47:20.544 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:47:20.544 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:47:20.544 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:47:20.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:20.544 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-11T09:47:20.544 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQBPop9lN7eDLBAAKswPZFfs0g2AaSe6V+pbbQ==", 2024-01-11T09:47:20.544 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:47:20.544 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-11T09:47:20.544 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-11T09:47:20.544 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:47:20.545 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:47:20.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:20.545 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-11T09:47:20.545 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lT5okIBAABy8PbJJbA062VNmb1Y8qhg==", 2024-01-11T09:47:20.545 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:47:20.545 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:47:20.545 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-01-11T09:47:20.545 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-11T09:47:20.545 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:47:20.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:20.545 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-11T09:47:20.545 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lDf37LhAAqV+EabKXJEW1sS5xtu1tXw==", 2024-01-11T09:47:20.546 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:47:20.546 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:47:20.546 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T09:47:20.546 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T09:47:20.546 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T09:47:20.546 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:47:20.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:20.546 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-11T09:47:20.546 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lWayxJxAAcalM8QSOGtbCXxjYZGaNQA==", 2024-01-11T09:47:20.546 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:47:20.546 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:47:20.546 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T09:47:20.546 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T09:47:20.547 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T09:47:20.547 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:47:20.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:20.547 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-11T09:47:20.547 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAyop9lHD4tNhAAdmnFBmMrK+QhQYdpbzW0mg==", 2024-01-11T09:47:20.547 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:47:20.547 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-11T09:47:20.547 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-11T09:47:20.547 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-11T09:47:20.547 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-11T09:47:20.547 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:47:20.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:20.547 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-11T09:47:20.548 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAwop9lGPtDMxAAJ8FjrSEdCtQL8AhtAyqnTw==", 2024-01-11T09:47:20.548 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:47:20.548 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T09:47:20.548 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-11T09:47:20.548 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-11T09:47:20.548 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:47:20.548 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:47:20.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:20.548 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-11T09:47:20.548 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9lXBZLJxAAXa1S8Vkm3MiV4igtNbg/1Q==", 2024-01-11T09:47:20.548 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:47:20.548 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T09:47:20.548 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T09:47:20.549 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:47:20.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-11T09:47:20.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:20.549 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-11T09:47:20.549 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQAxop9l3ZGoHxAANqzytTTvWYSomG+RFsFDyg==", 2024-01-11T09:47:20.549 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-11T09:47:20.549 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-11T09:47:20.549 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-11T09:47:20.549 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-11T09:47:20.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:20.549 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-11T09:47:20.549 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:47:20.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.545+0000 7f68741ff700 1 -- 172.21.15.27:0/265644937 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6850067170 msgr2=0x7f6850069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:20.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.545+0000 7f68741ff700 1 --2- 172.21.15.27:0/265644937 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6850067170 0x7f6850069620 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6868003a80 tx=0x7f6868007ec0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.545+0000 7f68741ff700 1 -- 172.21.15.27:0/265644937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686c0c89f0 msgr2=0x7f686c1581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:20.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.545+0000 7f68741ff700 1 --2- 172.21.15.27:0/265644937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686c0c89f0 0x7f686c1581a0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f686400ccc0 tx=0x7f6864001a20 comp rx=0 tx=0).stop 2024-01-11T09:47:20.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.545+0000 7f68741ff700 1 -- 172.21.15.27:0/265644937 shutdown_connections 2024-01-11T09:47:20.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.545+0000 7f68741ff700 1 --2- 172.21.15.27:0/265644937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686c0c89f0 0x7f686c1581a0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.545+0000 7f68741ff700 1 --2- 172.21.15.27:0/265644937 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6850067170 0x7f6850069620 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.545+0000 7f68741ff700 1 --2- 172.21.15.27:0/265644937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f686c069d40 0x7f686c14b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.545+0000 7f68741ff700 1 --2- 172.21.15.27:0/265644937 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f686c0c8620 0x7f686c157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.545+0000 7f68741ff700 1 -- 172.21.15.27:0/265644937 >> 172.21.15.27:0/265644937 conn(0x7f686c0680e0 msgr2=0x7f686c153680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:20.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.545+0000 7f68741ff700 1 -- 172.21.15.27:0/265644937 shutdown_connections 2024-01-11T09:47:20.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.545+0000 7f68741ff700 1 -- 172.21.15.27:0/265644937 wait complete. 2024-01-11T09:47:20.556 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:47:20.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.764+0000 7f4c689ff700 1 Processor -- start 2024-01-11T09:47:20.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.764+0000 7f4c689ff700 1 -- start start 2024-01-11T09:47:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.766+0000 7f4c689ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c640b03e0 0x7f4c640b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.766+0000 7f4c689ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4c640b07b0 0x7f4c640b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.766+0000 7f4c689ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c640cd730 con 0x7f4c64159200 2024-01-11T09:47:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.766+0000 7f4c689ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c64159d50 con 0x7f4c640b07b0 2024-01-11T09:47:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.766+0000 7f4c689ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640ccb20 con 0x7f4c640b03e0 2024-01-11T09:47:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.766+0000 7f4c61d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4c640b07b0 0x7f4c640b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.766+0000 7f4c62d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4c64159200 0x7f4c64159640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33744/0 (socket says 172.21.15.27:33744) 2024-01-11T09:47:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.766+0000 7f4c62d9d700 1 -- 172.21.15.27:0/3650317085 learned_addr learned my addr 172.21.15.27:0/3650317085 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.766+0000 7f4c61d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4c640b07b0 0x7f4c640b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44254/0 (socket says 172.21.15.27:44254) 2024-01-11T09:47:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.766+0000 7f4c6259c700 1 --2- 172.21.15.27:0/3650317085 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c640b03e0 0x7f4c640b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:20.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.767+0000 7f4c61d9b700 1 -- 172.21.15.27:0/3650317085 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c640b03e0 msgr2=0x7f4c640b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.767+0000 7f4c61d9b700 1 --2- 172.21.15.27:0/3650317085 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c640b03e0 0x7f4c640b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.767+0000 7f4c61d9b700 1 -- 172.21.15.27:0/3650317085 >> v1:172.21.15.27:6789/0 conn(0x7f4c64159200 legacy=0x7f4c64159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.767+0000 7f4c61d9b700 1 -- 172.21.15.27:0/3650317085 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c640cce00 con 0x7f4c640b07b0 2024-01-11T09:47:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.767+0000 7f4c6159a700 1 -- 172.21.15.27:0/3650317085 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825939590 0 0) 0x7f4c640cd730 con 0x7f4c64159200 2024-01-11T09:47:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.767+0000 7f4c61d9b700 1 --2- 172.21.15.27:0/3650317085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c640b07b0 0x7f4c640b29c0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f4c580098e0 tx=0x7f4c580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e64fa837d61ec4eb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.767+0000 7f4c6159a700 1 -- 172.21.15.27:0/3650317085 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5800b640 con 0x7f4c640b07b0 2024-01-11T09:47:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.767+0000 7f4c6159a700 1 -- 172.21.15.27:0/3650317085 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5800b7a0 con 0x7f4c640b07b0 2024-01-11T09:47:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.767+0000 7f4c6159a700 1 -- 172.21.15.27:0/3650317085 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5800bb40 con 0x7f4c640b07b0 2024-01-11T09:47:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.768+0000 7f4c689ff700 1 -- 172.21.15.27:0/3650317085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c640b07b0 msgr2=0x7f4c640b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.768+0000 7f4c689ff700 1 --2- 172.21.15.27:0/3650317085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c640b07b0 0x7f4c640b29c0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f4c580098e0 tx=0x7f4c580091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.768+0000 7f4c689ff700 1 -- 172.21.15.27:0/3650317085 shutdown_connections 2024-01-11T09:47:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.768+0000 7f4c689ff700 1 --2- 172.21.15.27:0/3650317085 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c640b03e0 0x7f4c640b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.768+0000 7f4c689ff700 1 --2- 172.21.15.27:0/3650317085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c640b07b0 0x7f4c640b29c0 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.768+0000 7f4c689ff700 1 -- 172.21.15.27:0/3650317085 >> 172.21.15.27:0/3650317085 conn(0x7f4c6406ed40 msgr2=0x7f4c6406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:20.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.768+0000 7f4c689ff700 1 -- 172.21.15.27:0/3650317085 shutdown_connections 2024-01-11T09:47:20.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.768+0000 7f4c689ff700 1 -- 172.21.15.27:0/3650317085 wait complete. 2024-01-11T09:47:20.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.769+0000 7f4c689ff700 1 Processor -- start 2024-01-11T09:47:20.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.770+0000 7f4c689ff700 1 -- start start 2024-01-11T09:47:20.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.770+0000 7f4c689ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c640b03e0 0x7f4c640ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:20.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.770+0000 7f4c689ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c640b07b0 0x7f4c640c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:20.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.770+0000 7f4c689ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c64159200 0x7f4c6416a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:20.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.771+0000 7f4c689ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640d1050 con 0x7f4c640b07b0 2024-01-11T09:47:20.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.771+0000 7f4c689ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640c9fa0 con 0x7f4c64159200 2024-01-11T09:47:20.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.771+0000 7f4c689ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6416a9d0 con 0x7f4c640b03e0 2024-01-11T09:47:20.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.771+0000 7f4c61d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c640b07b0 0x7f4c640c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.771+0000 7f4c61d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c640b07b0 0x7f4c640c9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44256/0 (socket says 172.21.15.27:44256) 2024-01-11T09:47:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.771+0000 7f4c61d9b700 1 -- 172.21.15.27:0/3991406578 learned_addr learned my addr 172.21.15.27:0/3991406578 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.771+0000 7f4c62d9d700 1 --2- 172.21.15.27:0/3991406578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c64159200 0x7f4c6416a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.771+0000 7f4c6259c700 1 --2- 172.21.15.27:0/3991406578 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c640b03e0 0x7f4c640ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.771+0000 7f4c61d9b700 1 -- 172.21.15.27:0/3991406578 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c640b03e0 msgr2=0x7f4c640ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.771+0000 7f4c61d9b700 1 --2- 172.21.15.27:0/3991406578 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c640b03e0 0x7f4c640ca5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.771+0000 7f4c61d9b700 1 -- 172.21.15.27:0/3991406578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c64159200 msgr2=0x7f4c6416a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.771+0000 7f4c61d9b700 1 --2- 172.21.15.27:0/3991406578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c64159200 0x7f4c6416a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.771+0000 7f4c61d9b700 1 -- 172.21.15.27:0/3991406578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c58009420 con 0x7f4c640b07b0 2024-01-11T09:47:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.771+0000 7f4c6259c700 1 --2- 172.21.15.27:0/3991406578 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c640b03e0 0x7f4c640ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.772+0000 7f4c61d9b700 1 --2- 172.21.15.27:0/3991406578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c640b07b0 0x7f4c640c9a60 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f4c64069540 tx=0x7f4c58017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.772+0000 7f4c527fc700 1 -- 172.21.15.27:0/3991406578 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5800bb20 con 0x7f4c640b07b0 2024-01-11T09:47:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.772+0000 7f4c527fc700 1 -- 172.21.15.27:0/3991406578 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c58011420 con 0x7f4c640b07b0 2024-01-11T09:47:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.772+0000 7f4c689ff700 1 -- 172.21.15.27:0/3991406578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c640d07b0 con 0x7f4c640b07b0 2024-01-11T09:47:20.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.772+0000 7f4c527fc700 1 -- 172.21.15.27:0/3991406578 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c580117a0 con 0x7f4c640b07b0 2024-01-11T09:47:20.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.772+0000 7f4c689ff700 1 -- 172.21.15.27:0/3991406578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c640d0d70 con 0x7f4c640b07b0 2024-01-11T09:47:20.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.773+0000 7f4c689ff700 1 -- 172.21.15.27:0/3991406578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c640718d0 con 0x7f4c640b07b0 2024-01-11T09:47:20.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.774+0000 7f4c527fc700 1 -- 172.21.15.27:0/3991406578 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f4c58011b50 con 0x7f4c640b07b0 2024-01-11T09:47:20.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.775+0000 7f4c527fc700 1 --2- 172.21.15.27:0/3991406578 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4c44067170 0x7f4c44069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:20.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.775+0000 7f4c527fc700 1 -- 172.21.15.27:0/3991406578 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f4c58085d80 con 0x7f4c640b07b0 2024-01-11T09:47:20.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.776+0000 7f4c6259c700 1 --2- 172.21.15.27:0/3991406578 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4c44067170 0x7f4c44069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:20.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.776+0000 7f4c6259c700 1 --2- 172.21.15.27:0/3991406578 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4c44067170 0x7f4c44069620 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4c54006f00 tx=0x7f4c54008040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:20.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.779+0000 7f4c527fc700 1 -- 172.21.15.27:0/3991406578 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4c58052fa0 con 0x7f4c640b07b0 2024-01-11T09:47:20.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.984+0000 7f4c689ff700 1 -- 172.21.15.27:0/3991406578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c640cd0e0 con 0x7f4c640b07b0 2024-01-11T09:47:20.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.985+0000 7f4c527fc700 1 -- 172.21.15.27:0/3991406578 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) v1 ==== 74+0+12275 (secure 0 0 0) 0x7f4c580565c0 con 0x7f4c640b07b0 2024-01-11T09:47:20.978 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:20.978 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":252,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:47:20.016579+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:47:20.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.988+0000 7f4c689ff700 1 -- 172.21.15.27:0/3991406578 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4c44067170 msgr2=0x7f4c44069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:20.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.988+0000 7f4c689ff700 1 --2- 172.21.15.27:0/3991406578 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4c44067170 0x7f4c44069620 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4c54006f00 tx=0x7f4c54008040 comp rx=0 tx=0).stop 2024-01-11T09:47:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.988+0000 7f4c689ff700 1 -- 172.21.15.27:0/3991406578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c640b07b0 msgr2=0x7f4c640c9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.988+0000 7f4c689ff700 1 --2- 172.21.15.27:0/3991406578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c640b07b0 0x7f4c640c9a60 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f4c64069540 tx=0x7f4c58017000 comp rx=0 tx=0).stop 2024-01-11T09:47:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.988+0000 7f4c689ff700 1 -- 172.21.15.27:0/3991406578 shutdown_connections 2024-01-11T09:47:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.988+0000 7f4c689ff700 1 --2- 172.21.15.27:0/3991406578 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4c44067170 0x7f4c44069620 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.988+0000 7f4c689ff700 1 --2- 172.21.15.27:0/3991406578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c64159200 0x7f4c6416a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.988+0000 7f4c689ff700 1 --2- 172.21.15.27:0/3991406578 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4c640b03e0 0x7f4c640ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.988+0000 7f4c689ff700 1 --2- 172.21.15.27:0/3991406578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c640b07b0 0x7f4c640c9a60 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.988+0000 7f4c689ff700 1 -- 172.21.15.27:0/3991406578 >> 172.21.15.27:0/3991406578 conn(0x7f4c6406ed40 msgr2=0x7f4c64158a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.988+0000 7f4c689ff700 1 -- 172.21.15.27:0/3991406578 shutdown_connections 2024-01-11T09:47:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:20.988+0000 7f4c689ff700 1 -- 172.21.15.27:0/3991406578 wait complete. 2024-01-11T09:47:20.998 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-11T09:47:20.999 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T09:47:21.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.210+0000 7f6fd89ff700 1 Processor -- start 2024-01-11T09:47:21.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.210+0000 7f6fd89ff700 1 -- start start 2024-01-11T09:47:21.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.211+0000 7f6fd89ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6fd41622d0 0x7f6fd41646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:21.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.211+0000 7f6fd89ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd40cf850 con 0x7f6fd415a1b0 2024-01-11T09:47:21.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.211+0000 7f6fd89ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd40cec40 con 0x7f6fd40ccd70 2024-01-11T09:47:21.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.211+0000 7f6fd89ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd415ac90 con 0x7f6fd41622d0 2024-01-11T09:47:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.211+0000 7f6fd259c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6fd415a1b0 0x7f6fd415a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33760/0 (socket says 172.21.15.27:33760) 2024-01-11T09:47:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.211+0000 7f6fd259c700 1 -- 172.21.15.27:0/158724918 learned_addr learned my addr 172.21.15.27:0/158724918 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.212+0000 7f6fd2d9d700 1 --2- 172.21.15.27:0/158724918 >> v2:172.21.15.77:3300/0 conn(0x7f6fd41622d0 0x7f6fd41646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.212+0000 7f6fd159a700 1 -- 172.21.15.27:0/158724918 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002897813 0 0) 0x7f6fd40cf850 con 0x7f6fd415a1b0 2024-01-11T09:47:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.212+0000 7f6fd159a700 1 -- 172.21.15.27:0/158724918 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fbc003570 con 0x7f6fd415a1b0 2024-01-11T09:47:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.212+0000 7f6fd159a700 1 -- 172.21.15.27:0/158724918 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3774072415 0 0) 0x7f6fbc003570 con 0x7f6fd415a1b0 2024-01-11T09:47:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.212+0000 7f6fd159a700 1 -- 172.21.15.27:0/158724918 >> v2:172.21.15.77:3300/0 conn(0x7f6fd41622d0 msgr2=0x7f6fd41646b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.212+0000 7f6fd159a700 1 --2- 172.21.15.27:0/158724918 >> v2:172.21.15.77:3300/0 conn(0x7f6fd41622d0 0x7f6fd41646b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.212+0000 7f6fd159a700 1 -- 172.21.15.27:0/158724918 >> v1:172.21.15.77:6789/0 conn(0x7f6fd40ccd70 legacy=0x7f6fd40cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.212+0000 7f6fd159a700 1 -- 172.21.15.27:0/158724918 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd40cef20 con 0x7f6fd415a1b0 2024-01-11T09:47:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.212+0000 7f6fd159a700 1 -- 172.21.15.27:0/158724918 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f6fc4001470 con 0x7f6fd415a1b0 2024-01-11T09:47:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.212+0000 7f6fd159a700 1 -- 172.21.15.27:0/158724918 >> v1:172.21.15.27:6789/0 conn(0x7f6fd415a1b0 legacy=0x7f6fd415a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.212+0000 7f6fd159a700 1 --2- 172.21.15.27:0/158724918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fbc003a60 0x7f6fbc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.212+0000 7f6fd159a700 1 --2- 172.21.15.27:0/158724918 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fbc004810 0x7f6fbc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.212+0000 7f6fd159a700 1 --2- 172.21.15.27:0/158724918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fbc0051c0 0x7f6fbc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.212+0000 7f6fd159a700 1 -- 172.21.15.27:0/158724918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd415ac90 con 0x7f6fbc0051c0 2024-01-11T09:47:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.212+0000 7f6fd159a700 1 -- 172.21.15.27:0/158724918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc00c050 con 0x7f6fbc003a60 2024-01-11T09:47:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.212+0000 7f6fd159a700 1 -- 172.21.15.27:0/158724918 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc00c210 con 0x7f6fbc004810 2024-01-11T09:47:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.213+0000 7f6fd2d9d700 1 --2- 172.21.15.27:0/158724918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fbc0051c0 0x7f6fbc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.213+0000 7f6fd259c700 1 --2- 172.21.15.27:0/158724918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fbc003a60 0x7f6fbc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.213+0000 7f6fd1d9b700 1 --2- 172.21.15.27:0/158724918 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fbc004810 0x7f6fbc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.213+0000 7f6fd2d9d700 1 -- 172.21.15.27:0/158724918 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fbc004810 msgr2=0x7f6fbc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.213+0000 7f6fd2d9d700 1 --2- 172.21.15.27:0/158724918 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fbc004810 0x7f6fbc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.213+0000 7f6fd2d9d700 1 -- 172.21.15.27:0/158724918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fbc003a60 msgr2=0x7f6fbc003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:21.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.213+0000 7f6fd2d9d700 1 --2- 172.21.15.27:0/158724918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fbc003a60 0x7f6fbc003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.213+0000 7f6fd2d9d700 1 -- 172.21.15.27:0/158724918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fbc003570 con 0x7f6fbc0051c0 2024-01-11T09:47:21.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.213+0000 7f6fd2d9d700 1 -- 172.21.15.27:0/158724918 reap_dead start 2024-01-11T09:47:21.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.214+0000 7f6fd2d9d700 1 --2- 172.21.15.27:0/158724918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fbc0051c0 0x7f6fbc00ba60 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc00e210 tx=0x7f6fcc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e92eb88e50087cc server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:21.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.214+0000 7f6fd159a700 1 -- 172.21.15.27:0/158724918 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc00eac0 con 0x7f6fbc0051c0 2024-01-11T09:47:21.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.214+0000 7f6fd159a700 1 -- 172.21.15.27:0/158724918 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fcc00f040 con 0x7f6fbc0051c0 2024-01-11T09:47:21.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.214+0000 7f6fd159a700 1 -- 172.21.15.27:0/158724918 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc00ee70 con 0x7f6fbc0051c0 2024-01-11T09:47:21.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.214+0000 7f6fd89ff700 1 -- 172.21.15.27:0/158724918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fbc0051c0 msgr2=0x7f6fbc00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:21.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.214+0000 7f6fd89ff700 1 --2- 172.21.15.27:0/158724918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fbc0051c0 0x7f6fbc00ba60 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc00e210 tx=0x7f6fcc00b040 comp rx=0 tx=0).stop 2024-01-11T09:47:21.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.214+0000 7f6fd89ff700 1 -- 172.21.15.27:0/158724918 shutdown_connections 2024-01-11T09:47:21.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.214+0000 7f6fd89ff700 1 --2- 172.21.15.27:0/158724918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fbc0051c0 0x7f6fbc00ba60 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.214+0000 7f6fd89ff700 1 -- 172.21.15.27:0/158724918 >> 172.21.15.27:0/158724918 conn(0x7f6fd406ed40 msgr2=0x7f6fd406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:21.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.214+0000 7f6fd89ff700 1 -- 172.21.15.27:0/158724918 shutdown_connections 2024-01-11T09:47:21.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.215+0000 7f6fd89ff700 1 -- 172.21.15.27:0/158724918 wait complete. 2024-01-11T09:47:21.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.216+0000 7f6fd89ff700 1 Processor -- start 2024-01-11T09:47:21.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.216+0000 7f6fd89ff700 1 -- start start 2024-01-11T09:47:21.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.217+0000 7f6fd89ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fbc0051c0 0x7f6fd414fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:21.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.217+0000 7f6fd89ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd40b03e0 0x7f6fd40b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:21.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.217+0000 7f6fd89ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd40c7690 0x7f6fd40c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:21.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.217+0000 7f6fd89ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40b2140 con 0x7f6fd40c7690 2024-01-11T09:47:21.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.217+0000 7f6fd89ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40b10a0 con 0x7f6fd40b03e0 2024-01-11T09:47:21.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.217+0000 7f6fd89ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40b1380 con 0x7f6fbc0051c0 2024-01-11T09:47:21.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.217+0000 7f6fd2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd40c7690 0x7f6fd40c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:21.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.217+0000 7f6fd2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd40c7690 0x7f6fd40c7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44278/0 (socket says 172.21.15.27:44278) 2024-01-11T09:47:21.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.218+0000 7f6fd2d9d700 1 -- 172.21.15.27:0/2110763913 learned_addr learned my addr 172.21.15.27:0/2110763913 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:21.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.217+0000 7f6fd259c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fbc0051c0 0x7f6fd414fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:21.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.218+0000 7f6fd1d9b700 1 --2- 172.21.15.27:0/2110763913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd40b03e0 0x7f6fd40b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:21.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.218+0000 7f6fd2d9d700 1 -- 172.21.15.27:0/2110763913 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fbc0051c0 msgr2=0x7f6fd414fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:21.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.218+0000 7f6fd2d9d700 1 --2- 172.21.15.27:0/2110763913 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fbc0051c0 0x7f6fd414fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.218+0000 7f6fd2d9d700 1 -- 172.21.15.27:0/2110763913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd40b03e0 msgr2=0x7f6fd40b0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:21.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.218+0000 7f6fd2d9d700 1 --2- 172.21.15.27:0/2110763913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd40b03e0 0x7f6fd40b0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.218+0000 7f6fd2d9d700 1 -- 172.21.15.27:0/2110763913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd40b1660 con 0x7f6fd40c7690 2024-01-11T09:47:21.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.218+0000 7f6fd2d9d700 1 --2- 172.21.15.27:0/2110763913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd40c7690 0x7f6fd40c7ae0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f6fd40cc220 tx=0x7f6fcc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:21.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.218+0000 7f6fc27fc700 1 -- 172.21.15.27:0/2110763913 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc01d650 con 0x7f6fd40c7690 2024-01-11T09:47:21.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.218+0000 7f6fc27fc700 1 -- 172.21.15.27:0/2110763913 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fcc01d7d0 con 0x7f6fd40c7690 2024-01-11T09:47:21.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.218+0000 7f6fd89ff700 1 -- 172.21.15.27:0/2110763913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd40b1940 con 0x7f6fd40c7690 2024-01-11T09:47:21.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.218+0000 7f6fc27fc700 1 -- 172.21.15.27:0/2110763913 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc01db90 con 0x7f6fd40c7690 2024-01-11T09:47:21.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.219+0000 7f6fd89ff700 1 -- 172.21.15.27:0/2110763913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd40b0d90 con 0x7f6fd40c7690 2024-01-11T09:47:21.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.220+0000 7f6fd89ff700 1 -- 172.21.15.27:0/2110763913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd40b1e40 con 0x7f6fd40c7690 2024-01-11T09:47:21.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.221+0000 7f6fc27fc700 1 -- 172.21.15.27:0/2110763913 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f6fcc003020 con 0x7f6fd40c7690 2024-01-11T09:47:21.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.222+0000 7f6fc27fc700 1 --2- 172.21.15.27:0/2110763913 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6fb4067170 0x7f6fb4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:21.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.222+0000 7f6fc27fc700 1 -- 172.21.15.27:0/2110763913 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f6fcc086110 con 0x7f6fd40c7690 2024-01-11T09:47:21.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.222+0000 7f6fd259c700 1 --2- 172.21.15.27:0/2110763913 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6fb4067170 0x7f6fb4069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:21.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.223+0000 7f6fd259c700 1 --2- 172.21.15.27:0/2110763913 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6fb4067170 0x7f6fb4069620 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4000e50 tx=0x7f6fc4001df0 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:21.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.226+0000 7f6fc27fc700 1 -- 172.21.15.27:0/2110763913 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6fcc0532e0 con 0x7f6fd40c7690 2024-01-11T09:47:21.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.427+0000 7f6fd89ff700 1 -- 172.21.15.27:0/2110763913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6fd404c3c0 con 0x7f6fd40c7690 2024-01-11T09:47:21.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.427+0000 7f6fc27fc700 1 -- 172.21.15.27:0/2110763913 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v252) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6fcc0568c0 con 0x7f6fd40c7690 2024-01-11T09:47:21.420 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T09:47:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.429+0000 7f6fd89ff700 1 -- 172.21.15.27:0/2110763913 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6fb4067170 msgr2=0x7f6fb4069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.429+0000 7f6fd89ff700 1 --2- 172.21.15.27:0/2110763913 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6fb4067170 0x7f6fb4069620 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4000e50 tx=0x7f6fc4001df0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.429+0000 7f6fd89ff700 1 -- 172.21.15.27:0/2110763913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd40c7690 msgr2=0x7f6fd40c7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.429+0000 7f6fd89ff700 1 --2- 172.21.15.27:0/2110763913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd40c7690 0x7f6fd40c7ae0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f6fd40cc220 tx=0x7f6fcc019000 comp rx=0 tx=0).stop 2024-01-11T09:47:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.430+0000 7f6fd89ff700 1 -- 172.21.15.27:0/2110763913 shutdown_connections 2024-01-11T09:47:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.430+0000 7f6fd89ff700 1 --2- 172.21.15.27:0/2110763913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd40c7690 0x7f6fd40c7ae0 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.430+0000 7f6fd89ff700 1 --2- 172.21.15.27:0/2110763913 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6fbc0051c0 0x7f6fd414fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.430+0000 7f6fd89ff700 1 --2- 172.21.15.27:0/2110763913 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6fb4067170 0x7f6fb4069620 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.430+0000 7f6fd89ff700 1 --2- 172.21.15.27:0/2110763913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd40b03e0 0x7f6fd40b0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.430+0000 7f6fd89ff700 1 -- 172.21.15.27:0/2110763913 >> 172.21.15.27:0/2110763913 conn(0x7f6fd406ed40 msgr2=0x7f6fd406f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.430+0000 7f6fd89ff700 1 -- 172.21.15.27:0/2110763913 shutdown_connections 2024-01-11T09:47:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.430+0000 7f6fd89ff700 1 -- 172.21.15.27:0/2110763913 wait complete. 2024-01-11T09:47:21.440 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-11T09:47:21.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.647+0000 7fe2be1ff700 1 Processor -- start 2024-01-11T09:47:21.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.647+0000 7fe2be1ff700 1 -- start start 2024-01-11T09:47:21.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.648+0000 7fe2be1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe2b80b3410 0x7fe2b80b1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:21.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.648+0000 7fe2be1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2b80b1c70 0x7fe2b8159180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:21.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.648+0000 7fe2be1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b80b38f0 con 0x7fe2b80cd880 2024-01-11T09:47:21.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.648+0000 7fe2be1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b80b3bd0 con 0x7fe2b80b3410 2024-01-11T09:47:21.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.648+0000 7fe2be1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b80b3eb0 con 0x7fe2b80b1c70 2024-01-11T09:47:21.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.648+0000 7fe2b77fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe2b80cd880 0x7fe2b8068930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33764/0 (socket says 172.21.15.27:33764) 2024-01-11T09:47:21.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.648+0000 7fe2b77fe700 1 -- 172.21.15.27:0/2976003325 learned_addr learned my addr 172.21.15.27:0/2976003325 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:21.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.649+0000 7fe2b6ffd700 1 --2- 172.21.15.27:0/2976003325 >> v2:172.21.15.77:3300/0 conn(0x7fe2b80b3410 0x7fe2b80b1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:21.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.649+0000 7fe2b7fff700 1 --2- 172.21.15.27:0/2976003325 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2b80b1c70 0x7fe2b8159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:21.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.649+0000 7fe2b67fc700 1 -- 172.21.15.27:0/2976003325 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965383615 0 0) 0x7fe2b80b38f0 con 0x7fe2b80cd880 2024-01-11T09:47:21.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.649+0000 7fe2b67fc700 1 -- 172.21.15.27:0/2976003325 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2a0003570 con 0x7fe2b80cd880 2024-01-11T09:47:21.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.649+0000 7fe2b67fc700 1 -- 172.21.15.27:0/2976003325 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1174186665 0 0) 0x7fe2a0003570 con 0x7fe2b80cd880 2024-01-11T09:47:21.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.649+0000 7fe2b67fc700 1 -- 172.21.15.27:0/2976003325 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2b80b1c70 msgr2=0x7fe2b8159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:21.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.649+0000 7fe2b67fc700 1 --2- 172.21.15.27:0/2976003325 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2b80b1c70 0x7fe2b8159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.649+0000 7fe2b67fc700 1 -- 172.21.15.27:0/2976003325 >> v2:172.21.15.77:3300/0 conn(0x7fe2b80b3410 msgr2=0x7fe2b80b1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:21.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.649+0000 7fe2b67fc700 1 --2- 172.21.15.27:0/2976003325 >> v2:172.21.15.77:3300/0 conn(0x7fe2b80b3410 0x7fe2b80b1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.649+0000 7fe2b67fc700 1 -- 172.21.15.27:0/2976003325 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b80b4190 con 0x7fe2b80cd880 2024-01-11T09:47:21.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.649+0000 7fe2b67fc700 1 -- 172.21.15.27:0/2976003325 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fe2a8001470 con 0x7fe2b80cd880 2024-01-11T09:47:21.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.649+0000 7fe2b6ffd700 1 --2- 172.21.15.27:0/2976003325 >> v2:172.21.15.77:3300/0 conn(0x7fe2b80b3410 0x7fe2b80b1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:21.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.649+0000 7fe2b7fff700 1 --2- 172.21.15.27:0/2976003325 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2b80b1c70 0x7fe2b8159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:21.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.649+0000 7fe2b67fc700 1 -- 172.21.15.27:0/2976003325 >> v1:172.21.15.27:6789/0 conn(0x7fe2b80cd880 legacy=0x7fe2b8068930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:21.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.649+0000 7fe2b67fc700 1 --2- 172.21.15.27:0/2976003325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2a0003ae0 0x7fe2a0003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:21.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.649+0000 7fe2b67fc700 1 --2- 172.21.15.27:0/2976003325 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2b80b1c70 0x7fe2a0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:21.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.649+0000 7fe2b67fc700 1 --2- 172.21.15.27:0/2976003325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2a0004d50 0x7fe2a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:21.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.649+0000 7fe2b67fc700 1 -- 172.21.15.27:0/2976003325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b80b3bd0 con 0x7fe2a0004d50 2024-01-11T09:47:21.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.650+0000 7fe2b67fc700 1 -- 172.21.15.27:0/2976003325 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b80b3eb0 con 0x7fe2a0003ae0 2024-01-11T09:47:21.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.650+0000 7fe2b67fc700 1 -- 172.21.15.27:0/2976003325 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a000bab0 con 0x7fe2b80b1c70 2024-01-11T09:47:21.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.650+0000 7fe2b7fff700 1 --2- 172.21.15.27:0/2976003325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2a0004d50 0x7fe2a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:21.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.650+0000 7fe2b77fe700 1 --2- 172.21.15.27:0/2976003325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2a0003ae0 0x7fe2a0003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:21.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.650+0000 7fe2b6ffd700 1 --2- 172.21.15.27:0/2976003325 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2b80b1c70 0x7fe2a0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:21.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.650+0000 7fe2b7fff700 1 -- 172.21.15.27:0/2976003325 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2b80b1c70 msgr2=0x7fe2a0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:21.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.650+0000 7fe2b7fff700 1 --2- 172.21.15.27:0/2976003325 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2b80b1c70 0x7fe2a0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.650+0000 7fe2b7fff700 1 -- 172.21.15.27:0/2976003325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2a0003ae0 msgr2=0x7fe2a0003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:21.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.650+0000 7fe2b7fff700 1 --2- 172.21.15.27:0/2976003325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2a0003ae0 0x7fe2a0003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.650+0000 7fe2b7fff700 1 -- 172.21.15.27:0/2976003325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b0009a20 con 0x7fe2a0004d50 2024-01-11T09:47:21.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.650+0000 7fe2b7fff700 1 --2- 172.21.15.27:0/2976003325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2a0004d50 0x7fe2a00051c0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fe2b80b2090 tx=0x7fe2b000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0bd8dc7765b892f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:21.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.651+0000 7fe2b67fc700 1 -- 172.21.15.27:0/2976003325 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0009c50 con 0x7fe2a0004d50 2024-01-11T09:47:21.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.651+0000 7fe2b67fc700 1 -- 172.21.15.27:0/2976003325 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b000f040 con 0x7fe2a0004d50 2024-01-11T09:47:21.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.651+0000 7fe2b67fc700 1 -- 172.21.15.27:0/2976003325 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b000ee60 con 0x7fe2a0004d50 2024-01-11T09:47:21.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.651+0000 7fe2be1ff700 1 -- 172.21.15.27:0/2976003325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2a0004d50 msgr2=0x7fe2a00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:21.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.651+0000 7fe2be1ff700 1 --2- 172.21.15.27:0/2976003325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2a0004d50 0x7fe2a00051c0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fe2b80b2090 tx=0x7fe2b000b040 comp rx=0 tx=0).stop 2024-01-11T09:47:21.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.651+0000 7fe2b7fff700 1 -- 172.21.15.27:0/2976003325 reap_dead start 2024-01-11T09:47:21.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.651+0000 7fe2be1ff700 1 -- 172.21.15.27:0/2976003325 shutdown_connections 2024-01-11T09:47:21.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.651+0000 7fe2be1ff700 1 -- 172.21.15.27:0/2976003325 >> 172.21.15.27:0/2976003325 conn(0x7fe2b80ce3a0 msgr2=0x7fe2b80ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:21.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.651+0000 7fe2be1ff700 1 -- 172.21.15.27:0/2976003325 shutdown_connections 2024-01-11T09:47:21.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.651+0000 7fe2be1ff700 1 -- 172.21.15.27:0/2976003325 wait complete. 2024-01-11T09:47:21.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.653+0000 7fe2be1ff700 1 Processor -- start 2024-01-11T09:47:21.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.653+0000 7fe2be1ff700 1 -- start start 2024-01-11T09:47:21.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.654+0000 7fe2be1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2a0004d50 0x7fe2b81506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:21.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.654+0000 7fe2be1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2b815aba0 0x7fe2b814cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:21.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.654+0000 7fe2be1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2b814d880 0x7fe2b806b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:21.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.654+0000 7fe2be1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b815bc20 con 0x7fe2a0004d50 2024-01-11T09:47:21.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.654+0000 7fe2be1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b815b010 con 0x7fe2b814d880 2024-01-11T09:47:21.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.654+0000 7fe2be1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b815b2f0 con 0x7fe2b815aba0 2024-01-11T09:47:21.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.654+0000 7fe2b77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2a0004d50 0x7fe2b81506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:21.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.654+0000 7fe2b77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2a0004d50 0x7fe2b81506b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44296/0 (socket says 172.21.15.27:44296) 2024-01-11T09:47:21.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.654+0000 7fe2b77fe700 1 -- 172.21.15.27:0/4141329098 learned_addr learned my addr 172.21.15.27:0/4141329098 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:21.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.654+0000 7fe2b7fff700 1 --2- 172.21.15.27:0/4141329098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2b814d880 0x7fe2b806b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:21.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.655+0000 7fe2b6ffd700 1 --2- 172.21.15.27:0/4141329098 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2b815aba0 0x7fe2b814cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:21.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.655+0000 7fe2b77fe700 1 -- 172.21.15.27:0/4141329098 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2b815aba0 msgr2=0x7fe2b814cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:21.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.655+0000 7fe2b77fe700 1 --2- 172.21.15.27:0/4141329098 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2b815aba0 0x7fe2b814cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.655+0000 7fe2b77fe700 1 -- 172.21.15.27:0/4141329098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2b814d880 msgr2=0x7fe2b806b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:21.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.655+0000 7fe2b77fe700 1 --2- 172.21.15.27:0/4141329098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2b814d880 0x7fe2b806b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.655+0000 7fe2b77fe700 1 -- 172.21.15.27:0/4141329098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b815b5d0 con 0x7fe2a0004d50 2024-01-11T09:47:21.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.655+0000 7fe2b6ffd700 1 --2- 172.21.15.27:0/4141329098 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2b815aba0 0x7fe2b814cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:21.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.655+0000 7fe2b77fe700 1 --2- 172.21.15.27:0/4141329098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2a0004d50 0x7fe2b81506b0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8007390 tx=0x7fe2b80b29c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:21.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.655+0000 7fe29f7fe700 1 -- 172.21.15.27:0/4141329098 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a8018030 con 0x7fe2a0004d50 2024-01-11T09:47:21.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.655+0000 7fe29f7fe700 1 -- 172.21.15.27:0/4141329098 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a800f8d0 con 0x7fe2a0004d50 2024-01-11T09:47:21.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.655+0000 7fe2be1ff700 1 -- 172.21.15.27:0/4141329098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b815b8b0 con 0x7fe2a0004d50 2024-01-11T09:47:21.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.655+0000 7fe29f7fe700 1 -- 172.21.15.27:0/4141329098 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a800fc70 con 0x7fe2a0004d50 2024-01-11T09:47:21.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.656+0000 7fe2be1ff700 1 -- 172.21.15.27:0/4141329098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b806be40 con 0x7fe2a0004d50 2024-01-11T09:47:21.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.657+0000 7fe2be1ff700 1 -- 172.21.15.27:0/4141329098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b804c3c0 con 0x7fe2a0004d50 2024-01-11T09:47:21.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.657+0000 7fe29f7fe700 1 -- 172.21.15.27:0/4141329098 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fe2a801b070 con 0x7fe2a0004d50 2024-01-11T09:47:21.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.658+0000 7fe29f7fe700 1 --2- 172.21.15.27:0/4141329098 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe2a40671a0 0x7fe2a4069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:21.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.658+0000 7fe29f7fe700 1 -- 172.21.15.27:0/4141329098 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 4965+0+0 (secure 0 0 0) 0x7fe2a8019070 con 0x7fe2a0004d50 2024-01-11T09:47:21.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.659+0000 7fe2b6ffd700 1 --2- 172.21.15.27:0/4141329098 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe2a40671a0 0x7fe2a4069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:21.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.659+0000 7fe2b6ffd700 1 --2- 172.21.15.27:0/4141329098 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe2a40671a0 0x7fe2a4069650 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe2b80ae620 tx=0x7fe2ac008040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:21.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.663+0000 7fe29f7fe700 1 -- 172.21.15.27:0/4141329098 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe2a8051710 con 0x7fe2a0004d50 2024-01-11T09:47:21.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.867+0000 7fe2be1ff700 1 -- 172.21.15.27:0/4141329098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe2b806ce20 con 0x7fe2a0004d50 2024-01-11T09:47:21.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.868+0000 7fe29f7fe700 1 -- 172.21.15.27:0/4141329098 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v252) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe2a8054d30 con 0x7fe2a0004d50 2024-01-11T09:47:21.860 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:21.860 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T09:47:21.861 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:21.861 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-11T09:47:21.861 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-11T09:47:21.861 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:21.861 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T09:47:21.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.870+0000 7fe2be1ff700 1 -- 172.21.15.27:0/4141329098 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe2a40671a0 msgr2=0x7fe2a4069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:21.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.870+0000 7fe2be1ff700 1 --2- 172.21.15.27:0/4141329098 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe2a40671a0 0x7fe2a4069650 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe2b80ae620 tx=0x7fe2ac008040 comp rx=0 tx=0).stop 2024-01-11T09:47:21.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.870+0000 7fe2be1ff700 1 -- 172.21.15.27:0/4141329098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2a0004d50 msgr2=0x7fe2b81506b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:21.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.870+0000 7fe2be1ff700 1 --2- 172.21.15.27:0/4141329098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2a0004d50 0x7fe2b81506b0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8007390 tx=0x7fe2b80b29c0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.870+0000 7fe2be1ff700 1 -- 172.21.15.27:0/4141329098 shutdown_connections 2024-01-11T09:47:21.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.870+0000 7fe2be1ff700 1 --2- 172.21.15.27:0/4141329098 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe2a40671a0 0x7fe2a4069650 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.870+0000 7fe2be1ff700 1 --2- 172.21.15.27:0/4141329098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2b814d880 0x7fe2b806b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.870+0000 7fe2be1ff700 1 --2- 172.21.15.27:0/4141329098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2a0004d50 0x7fe2b81506b0 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.870+0000 7fe2be1ff700 1 --2- 172.21.15.27:0/4141329098 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe2b815aba0 0x7fe2b814cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:21.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.870+0000 7fe2be1ff700 1 -- 172.21.15.27:0/4141329098 >> 172.21.15.27:0/4141329098 conn(0x7fe2b80ce3a0 msgr2=0x7fe2b80adbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:21.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.870+0000 7fe2be1ff700 1 -- 172.21.15.27:0/4141329098 shutdown_connections 2024-01-11T09:47:21.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:21.870+0000 7fe2be1ff700 1 -- 172.21.15.27:0/4141329098 wait complete. 2024-01-11T09:47:21.881 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-11T09:47:21.881 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-11T09:47:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.095+0000 7f26be3ff700 1 Processor -- start 2024-01-11T09:47:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.095+0000 7f26be3ff700 1 -- start start 2024-01-11T09:47:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.096+0000 7f26be3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26b815a7a0 0x7f26b815c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.096+0000 7f26be3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f26b8159e60 0x7f26b815aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.096+0000 7f26be3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b815c4b0 con 0x7f26b815b400 2024-01-11T09:47:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.096+0000 7f26be3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b815b8a0 con 0x7f26b815a7a0 2024-01-11T09:47:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.096+0000 7f26be3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b815bb80 con 0x7f26b8159e60 2024-01-11T09:47:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.096+0000 7f26b7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26b815a7a0 0x7f26b815c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.096+0000 7f26b7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26b815a7a0 0x7f26b815c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44300/0 (socket says 172.21.15.27:44300) 2024-01-11T09:47:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.096+0000 7f26b7fff700 1 -- 172.21.15.27:0/3850222760 learned_addr learned my addr 172.21.15.27:0/3850222760 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:22.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.097+0000 7f26b77fe700 1 --2- 172.21.15.27:0/3850222760 >> v2:172.21.15.77:3300/0 conn(0x7f26b8159e60 0x7f26b815aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:22.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.097+0000 7f26b7fff700 1 -- 172.21.15.27:0/3850222760 >> v2:172.21.15.77:3300/0 conn(0x7f26b8159e60 msgr2=0x7f26b815aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:22.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.097+0000 7f26b7fff700 1 --2- 172.21.15.27:0/3850222760 >> v2:172.21.15.77:3300/0 conn(0x7f26b8159e60 0x7f26b815aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:22.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.097+0000 7f26b7fff700 1 -- 172.21.15.27:0/3850222760 >> v1:172.21.15.27:6789/0 conn(0x7f26b815b400 legacy=0x7f26b80685e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:22.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.097+0000 7f26b7fff700 1 -- 172.21.15.27:0/3850222760 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b815be60 con 0x7f26b815a7a0 2024-01-11T09:47:22.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.097+0000 7f26b6ffd700 1 -- 172.21.15.27:0/3850222760 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068601858 0 0) 0x7f26b815c4b0 con 0x7f26b815b400 2024-01-11T09:47:22.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.097+0000 7f26b7fff700 1 --2- 172.21.15.27:0/3850222760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b815a7a0 0x7f26b815c940 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f26b80ce530 tx=0x7f26a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f43eeec8d09370 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:22.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.097+0000 7f26b6ffd700 1 -- 172.21.15.27:0/3850222760 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a8009e70 con 0x7f26b815a7a0 2024-01-11T09:47:22.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.098+0000 7f26b6ffd700 1 -- 172.21.15.27:0/3850222760 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a800b860 con 0x7f26b815a7a0 2024-01-11T09:47:22.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.098+0000 7f26b6ffd700 1 -- 172.21.15.27:0/3850222760 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a800bc00 con 0x7f26b815a7a0 2024-01-11T09:47:22.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.098+0000 7f26be3ff700 1 -- 172.21.15.27:0/3850222760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b815a7a0 msgr2=0x7f26b815c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:22.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.098+0000 7f26be3ff700 1 --2- 172.21.15.27:0/3850222760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b815a7a0 0x7f26b815c940 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f26b80ce530 tx=0x7f26a8009250 comp rx=0 tx=0).stop 2024-01-11T09:47:22.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.098+0000 7f26be3ff700 1 -- 172.21.15.27:0/3850222760 shutdown_connections 2024-01-11T09:47:22.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.098+0000 7f26be3ff700 1 --2- 172.21.15.27:0/3850222760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b815a7a0 0x7f26b815c940 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:22.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.098+0000 7f26be3ff700 1 --2- 172.21.15.27:0/3850222760 >> v2:172.21.15.77:3300/0 conn(0x7f26b8159e60 0x7f26b815aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:22.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.098+0000 7f26be3ff700 1 -- 172.21.15.27:0/3850222760 >> 172.21.15.27:0/3850222760 conn(0x7f26b806ed40 msgr2=0x7f26b806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:22.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.098+0000 7f26be3ff700 1 -- 172.21.15.27:0/3850222760 shutdown_connections 2024-01-11T09:47:22.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.098+0000 7f26be3ff700 1 -- 172.21.15.27:0/3850222760 wait complete. 2024-01-11T09:47:22.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.100+0000 7f26be3ff700 1 Processor -- start 2024-01-11T09:47:22.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.100+0000 7f26be3ff700 1 -- start start 2024-01-11T09:47:22.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.101+0000 7f26be3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b815a7a0 0x7f26b81521f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:22.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.101+0000 7f26be3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26b815b400 0x7f26b8140110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:22.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.101+0000 7f26be3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26b813d820 0x7f26b8156c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:22.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.101+0000 7f26be3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b81536a0 con 0x7f26b815a7a0 2024-01-11T09:47:22.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.101+0000 7f26be3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b8152a90 con 0x7f26b815b400 2024-01-11T09:47:22.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.101+0000 7f26be3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b8152d70 con 0x7f26b813d820 2024-01-11T09:47:22.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.101+0000 7f26b7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b815a7a0 0x7f26b81521f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:22.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.101+0000 7f26b7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b815a7a0 0x7f26b81521f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44308/0 (socket says 172.21.15.27:44308) 2024-01-11T09:47:22.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.101+0000 7f26b7fff700 1 -- 172.21.15.27:0/3776071380 learned_addr learned my addr 172.21.15.27:0/3776071380 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:22.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.101+0000 7f26b7fff700 1 -- 172.21.15.27:0/3776071380 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26b813d820 msgr2=0x7f26b8156c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:22.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.101+0000 7f26b7fff700 1 --2- 172.21.15.27:0/3776071380 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26b813d820 0x7f26b8156c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:22.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.101+0000 7f26b7fff700 1 -- 172.21.15.27:0/3776071380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26b815b400 msgr2=0x7f26b8140110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:22.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.101+0000 7f26b77fe700 1 --2- 172.21.15.27:0/3776071380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26b815b400 0x7f26b8140110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:22.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.101+0000 7f26b7fff700 1 --2- 172.21.15.27:0/3776071380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26b815b400 0x7f26b8140110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:22.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.101+0000 7f26b7fff700 1 -- 172.21.15.27:0/3776071380 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a80094f0 con 0x7f26b815a7a0 2024-01-11T09:47:22.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.101+0000 7f26b77fe700 1 --2- 172.21.15.27:0/3776071380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26b815b400 0x7f26b8140110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:22.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.102+0000 7f26b7fff700 1 --2- 172.21.15.27:0/3776071380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b815a7a0 0x7f26b81521f0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f26a80094c0 tx=0x7f26a8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:22.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.102+0000 7f269ffff700 1 -- 172.21.15.27:0/3776071380 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a800b9d0 con 0x7f26b815a7a0 2024-01-11T09:47:22.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.102+0000 7f269ffff700 1 -- 172.21.15.27:0/3776071380 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a800bb30 con 0x7f26b815a7a0 2024-01-11T09:47:22.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.102+0000 7f269ffff700 1 -- 172.21.15.27:0/3776071380 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a8025670 con 0x7f26b815a7a0 2024-01-11T09:47:22.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.102+0000 7f26be3ff700 1 -- 172.21.15.27:0/3776071380 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b8153050 con 0x7f26b815a7a0 2024-01-11T09:47:22.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.102+0000 7f26be3ff700 1 -- 172.21.15.27:0/3776071380 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b8152730 con 0x7f26b815a7a0 2024-01-11T09:47:22.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.103+0000 7f26be3ff700 1 -- 172.21.15.27:0/3776071380 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b804c3c0 con 0x7f26b815a7a0 2024-01-11T09:47:22.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.104+0000 7f269ffff700 1 -- 172.21.15.27:0/3776071380 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f26a8023070 con 0x7f26b815a7a0 2024-01-11T09:47:22.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.106+0000 7f269ffff700 1 --2- 172.21.15.27:0/3776071380 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f26a4067170 0x7f26a4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:22.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.106+0000 7f269ffff700 1 -- 172.21.15.27:0/3776071380 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f26a8085bc0 con 0x7f26b815a7a0 2024-01-11T09:47:22.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.106+0000 7f26b77fe700 1 --2- 172.21.15.27:0/3776071380 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f26a4067170 0x7f26a4069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:22.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.107+0000 7f26b77fe700 1 --2- 172.21.15.27:0/3776071380 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f26a4067170 0x7f26a4069620 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f26ac006f00 tx=0x7f26ac008040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:22.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.110+0000 7f269ffff700 1 -- 172.21.15.27:0/3776071380 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f26a8052de0 con 0x7f26b815a7a0 2024-01-11T09:47:22.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.317+0000 7f26be3ff700 1 -- 172.21.15.27:0/3776071380 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f26b815c140 con 0x7f26b815a7a0 2024-01-11T09:47:22.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.956+0000 7f269ffff700 1 -- 172.21.15.27:0/3776071380 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v253) v1 ==== 138+0+0 (secure 0 0 0) 0x7f26a80150b0 con 0x7f26b815a7a0 2024-01-11T09:47:22.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.961+0000 7f26be3ff700 1 -- 172.21.15.27:0/3776071380 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f26a4067170 msgr2=0x7f26a4069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:22.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.961+0000 7f26be3ff700 1 --2- 172.21.15.27:0/3776071380 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f26a4067170 0x7f26a4069620 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f26ac006f00 tx=0x7f26ac008040 comp rx=0 tx=0).stop 2024-01-11T09:47:22.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.961+0000 7f26be3ff700 1 -- 172.21.15.27:0/3776071380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b815a7a0 msgr2=0x7f26b81521f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:22.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.961+0000 7f26be3ff700 1 --2- 172.21.15.27:0/3776071380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b815a7a0 0x7f26b81521f0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f26a80094c0 tx=0x7f26a8017000 comp rx=0 tx=0).stop 2024-01-11T09:47:22.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.961+0000 7f26be3ff700 1 -- 172.21.15.27:0/3776071380 shutdown_connections 2024-01-11T09:47:22.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.961+0000 7f26be3ff700 1 --2- 172.21.15.27:0/3776071380 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f26b813d820 0x7f26b8156c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:22.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.961+0000 7f26be3ff700 1 --2- 172.21.15.27:0/3776071380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b815a7a0 0x7f26b81521f0 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:22.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.961+0000 7f26be3ff700 1 --2- 172.21.15.27:0/3776071380 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f26a4067170 0x7f26a4069620 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:22.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.961+0000 7f26be3ff700 1 --2- 172.21.15.27:0/3776071380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26b815b400 0x7f26b8140110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:22.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.961+0000 7f26be3ff700 1 -- 172.21.15.27:0/3776071380 >> 172.21.15.27:0/3776071380 conn(0x7f26b806ed40 msgr2=0x7f26b8150130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:22.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.963+0000 7f26be3ff700 1 -- 172.21.15.27:0/3776071380 shutdown_connections 2024-01-11T09:47:22.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:22.963+0000 7f26be3ff700 1 -- 172.21.15.27:0/3776071380 wait complete. 2024-01-11T09:47:22.958 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-11T09:47:22.977 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-11T09:47:23.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.193+0000 7f63ea3ff700 1 Processor -- start 2024-01-11T09:47:23.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.193+0000 7f63ea3ff700 1 -- start start 2024-01-11T09:47:23.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.194+0000 7f63ea3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63e4153e90 0x7f63e40accc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:23.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.194+0000 7f63ea3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f63e4069d40 0x7f63e40b0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:23.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.194+0000 7f63ea3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e40af4c0 con 0x7f63e40ab700 2024-01-11T09:47:23.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.194+0000 7f63ea3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63e41574d0 con 0x7f63e4069d40 2024-01-11T09:47:23.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.194+0000 7f63ea3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e41577b0 con 0x7f63e4153e90 2024-01-11T09:47:23.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.195+0000 7f63e37fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f63e4069d40 0x7f63e40b0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:23.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.195+0000 7f63e3fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63e4153e90 0x7f63e40accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:23.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.195+0000 7f63e899c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f63e40ab700 0x7f63e40af950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:44012/0 (socket says 172.21.15.27:44012) 2024-01-11T09:47:23.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.195+0000 7f63e899c700 1 -- 172.21.15.27:0/178582883 learned_addr learned my addr 172.21.15.27:0/178582883 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.195+0000 7f63e2ffd700 1 -- 172.21.15.27:0/178582883 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731951864 0 0) 0x7f63e40af4c0 con 0x7f63e40ab700 2024-01-11T09:47:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.195+0000 7f63e2ffd700 1 -- 172.21.15.27:0/178582883 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63cc003570 con 0x7f63e40ab700 2024-01-11T09:47:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.195+0000 7f63e37fe700 1 -- 172.21.15.27:0/178582883 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63e4153e90 msgr2=0x7f63e40accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.195+0000 7f63e37fe700 1 --2- 172.21.15.27:0/178582883 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63e4153e90 0x7f63e40accc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.195+0000 7f63e37fe700 1 -- 172.21.15.27:0/178582883 >> v1:172.21.15.77:6789/0 conn(0x7f63e40ab700 legacy=0x7f63e40af950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.195+0000 7f63e37fe700 1 -- 172.21.15.27:0/178582883 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e40af150 con 0x7f63e4069d40 2024-01-11T09:47:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.195+0000 7f63e3fff700 1 --2- 172.21.15.27:0/178582883 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63e4153e90 0x7f63e40accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.196+0000 7f63e37fe700 1 --2- 172.21.15.27:0/178582883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63e4069d40 0x7f63e40b0720 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f63d80098e0 tx=0x7f63d80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c096e3bae2299972 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.196+0000 7f63e2ffd700 1 -- 172.21.15.27:0/178582883 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d8011630 con 0x7f63e4069d40 2024-01-11T09:47:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.196+0000 7f63e2ffd700 1 -- 172.21.15.27:0/178582883 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d8011790 con 0x7f63e4069d40 2024-01-11T09:47:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.196+0000 7f63e2ffd700 1 -- 172.21.15.27:0/178582883 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d8011b30 con 0x7f63e4069d40 2024-01-11T09:47:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.197+0000 7f63ea3ff700 1 -- 172.21.15.27:0/178582883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63e4069d40 msgr2=0x7f63e40b0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.197+0000 7f63ea3ff700 1 --2- 172.21.15.27:0/178582883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63e4069d40 0x7f63e40b0720 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f63d80098e0 tx=0x7f63d80091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:23.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.197+0000 7f63ea3ff700 1 -- 172.21.15.27:0/178582883 shutdown_connections 2024-01-11T09:47:23.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.197+0000 7f63ea3ff700 1 --2- 172.21.15.27:0/178582883 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63e4153e90 0x7f63e40accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:23.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.197+0000 7f63ea3ff700 1 --2- 172.21.15.27:0/178582883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63e4069d40 0x7f63e40b0720 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:23.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.197+0000 7f63ea3ff700 1 -- 172.21.15.27:0/178582883 >> 172.21.15.27:0/178582883 conn(0x7f63e40680a0 msgr2=0x7f63e4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:23.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.197+0000 7f63ea3ff700 1 -- 172.21.15.27:0/178582883 shutdown_connections 2024-01-11T09:47:23.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.197+0000 7f63ea3ff700 1 -- 172.21.15.27:0/178582883 wait complete. 2024-01-11T09:47:23.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.198+0000 7f63ea3ff700 1 Processor -- start 2024-01-11T09:47:23.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.199+0000 7f63ea3ff700 1 -- start start 2024-01-11T09:47:23.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.200+0000 7f63ea3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63e4069d40 0x7f63e416a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:23.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.200+0000 7f63ea3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63e40ab700 0x7f63e40a9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:23.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.200+0000 7f63ea3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63e4153e90 0x7f63e4155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:23.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.200+0000 7f63ea3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e40aac40 con 0x7f63e40ab700 2024-01-11T09:47:23.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.200+0000 7f63ea3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e40aa030 con 0x7f63e4153e90 2024-01-11T09:47:23.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.200+0000 7f63ea3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e40aa310 con 0x7f63e4069d40 2024-01-11T09:47:23.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.200+0000 7f63e37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63e40ab700 0x7f63e40a9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:23.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.200+0000 7f63e37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63e40ab700 0x7f63e40a9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44316/0 (socket says 172.21.15.27:44316) 2024-01-11T09:47:23.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.200+0000 7f63e37fe700 1 -- 172.21.15.27:0/2822010387 learned_addr learned my addr 172.21.15.27:0/2822010387 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:23.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.200+0000 7f63e899c700 1 --2- 172.21.15.27:0/2822010387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63e4153e90 0x7f63e4155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:23.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.200+0000 7f63e3fff700 1 --2- 172.21.15.27:0/2822010387 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63e4069d40 0x7f63e416a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:23.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.200+0000 7f63e37fe700 1 -- 172.21.15.27:0/2822010387 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63e4069d40 msgr2=0x7f63e416a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:23.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.200+0000 7f63e37fe700 1 --2- 172.21.15.27:0/2822010387 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63e4069d40 0x7f63e416a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:23.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.200+0000 7f63e37fe700 1 -- 172.21.15.27:0/2822010387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63e4153e90 msgr2=0x7f63e4155680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:23.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.200+0000 7f63e37fe700 1 --2- 172.21.15.27:0/2822010387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63e4153e90 0x7f63e4155680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:23.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.200+0000 7f63e37fe700 1 -- 172.21.15.27:0/2822010387 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d4009420 con 0x7f63e40ab700 2024-01-11T09:47:23.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.201+0000 7f63e37fe700 1 --2- 172.21.15.27:0/2822010387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63e40ab700 0x7f63e40a9950 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f63e40694d0 tx=0x7f63d8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:23.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.201+0000 7f63cbfff700 1 -- 172.21.15.27:0/2822010387 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d8000d10 con 0x7f63e40ab700 2024-01-11T09:47:23.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.201+0000 7f63cbfff700 1 -- 172.21.15.27:0/2822010387 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d8011b10 con 0x7f63e40ab700 2024-01-11T09:47:23.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.201+0000 7f63ea3ff700 1 -- 172.21.15.27:0/2822010387 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d8009420 con 0x7f63e40ab700 2024-01-11T09:47:23.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.201+0000 7f63cbfff700 1 -- 172.21.15.27:0/2822010387 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d8025660 con 0x7f63e40ab700 2024-01-11T09:47:23.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.201+0000 7f63ea3ff700 1 -- 172.21.15.27:0/2822010387 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e40aa8d0 con 0x7f63e40ab700 2024-01-11T09:47:23.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.202+0000 7f63ea3ff700 1 -- 172.21.15.27:0/2822010387 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e40aadf0 con 0x7f63e40ab700 2024-01-11T09:47:23.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.204+0000 7f63cbfff700 1 -- 172.21.15.27:0/2822010387 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f63d8025a50 con 0x7f63e40ab700 2024-01-11T09:47:23.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.204+0000 7f63cbfff700 1 --2- 172.21.15.27:0/2822010387 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f63d0067170 0x7f63d0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:23.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.205+0000 7f63cbfff700 1 -- 172.21.15.27:0/2822010387 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f63d80862e0 con 0x7f63e40ab700 2024-01-11T09:47:23.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.205+0000 7f63e3fff700 1 --2- 172.21.15.27:0/2822010387 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f63d0067170 0x7f63d0069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:23.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.206+0000 7f63e3fff700 1 --2- 172.21.15.27:0/2822010387 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f63d0067170 0x7f63d0069620 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f63d4009e10 tx=0x7f63d40091a0 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:23.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.209+0000 7f63cbfff700 1 -- 172.21.15.27:0/2822010387 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f63d8053430 con 0x7f63e40ab700 2024-01-11T09:47:23.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.414+0000 7f63ea3ff700 1 -- 172.21.15.27:0/2822010387 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f63e404c3c0 con 0x7f63e40ab700 2024-01-11T09:47:23.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.964+0000 7f63cbfff700 1 -- 172.21.15.27:0/2822010387 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v254) v1 ==== 170+0+0 (secure 0 0 0) 0x7f63d8056a50 con 0x7f63e40ab700 2024-01-11T09:47:23.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.968+0000 7f63ea3ff700 1 -- 172.21.15.27:0/2822010387 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f63d0067170 msgr2=0x7f63d0069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:23.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.969+0000 7f63ea3ff700 1 --2- 172.21.15.27:0/2822010387 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f63d0067170 0x7f63d0069620 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f63d4009e10 tx=0x7f63d40091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:23.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.969+0000 7f63ea3ff700 1 -- 172.21.15.27:0/2822010387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63e40ab700 msgr2=0x7f63e40a9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:23.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.969+0000 7f63ea3ff700 1 --2- 172.21.15.27:0/2822010387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63e40ab700 0x7f63e40a9950 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f63e40694d0 tx=0x7f63d8017000 comp rx=0 tx=0).stop 2024-01-11T09:47:23.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.969+0000 7f63ea3ff700 1 -- 172.21.15.27:0/2822010387 shutdown_connections 2024-01-11T09:47:23.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.969+0000 7f63ea3ff700 1 --2- 172.21.15.27:0/2822010387 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f63d0067170 0x7f63d0069620 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f63d4009e10 tx=0x7f63d40091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:23.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.969+0000 7f63ea3ff700 1 --2- 172.21.15.27:0/2822010387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63e4153e90 0x7f63e4155680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:23.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.969+0000 7f63ea3ff700 1 --2- 172.21.15.27:0/2822010387 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f63e4069d40 0x7f63e416a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:23.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.969+0000 7f63ea3ff700 1 --2- 172.21.15.27:0/2822010387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63e40ab700 0x7f63e40a9950 secure :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f63e40694d0 tx=0x7f63d8017000 comp rx=0 tx=0).stop 2024-01-11T09:47:23.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.969+0000 7f63ea3ff700 1 -- 172.21.15.27:0/2822010387 >> 172.21.15.27:0/2822010387 conn(0x7f63e40680a0 msgr2=0x7f63e40b0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:23.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.970+0000 7f63ea3ff700 1 -- 172.21.15.27:0/2822010387 shutdown_connections 2024-01-11T09:47:23.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:23.970+0000 7f63ea3ff700 1 -- 172.21.15.27:0/2822010387 wait complete. 2024-01-11T09:47:23.966 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-11T09:47:23.995 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-11T09:47:24.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.205+0000 7f01e81ff700 1 Processor -- start 2024-01-11T09:47:24.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.205+0000 7f01e81ff700 1 -- start start 2024-01-11T09:47:24.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.207+0000 7f01e81ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01e015a1b0 0x7f01e015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:24.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.207+0000 7f01e81ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f01e00ccd70 0x7f01e00cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:24.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.207+0000 7f01e81ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e00cf850 con 0x7f01e00685c0 2024-01-11T09:47:24.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.207+0000 7f01e81ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01e00cec40 con 0x7f01e00ccd70 2024-01-11T09:47:24.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.207+0000 7f01e81ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e00cef20 con 0x7f01e015a1b0 2024-01-11T09:47:24.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.207+0000 7f01e579a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f01e00ccd70 0x7f01e00cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:24.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.207+0000 7f01e579a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f01e00ccd70 0x7f01e00cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56090/0 (socket says 172.21.15.27:56090) 2024-01-11T09:47:24.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.207+0000 7f01e579a700 1 -- 172.21.15.27:0/2618221941 learned_addr learned my addr 172.21.15.27:0/2618221941 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:24.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.207+0000 7f01e5f9b700 1 --2- 172.21.15.27:0/2618221941 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01e015a1b0 0x7f01e015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:24.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.208+0000 7f01e579a700 1 -- 172.21.15.27:0/2618221941 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01e015a1b0 msgr2=0x7f01e015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:24.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.208+0000 7f01e579a700 1 --2- 172.21.15.27:0/2618221941 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01e015a1b0 0x7f01e015a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:24.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.208+0000 7f01e579a700 1 -- 172.21.15.27:0/2618221941 >> v1:172.21.15.77:6789/0 conn(0x7f01e00685c0 legacy=0x7f01e0068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:24.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.208+0000 7f01e579a700 1 -- 172.21.15.27:0/2618221941 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e00cf200 con 0x7f01e00ccd70 2024-01-11T09:47:24.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.208+0000 7f01e4f99700 1 -- 172.21.15.27:0/2618221941 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891506743 0 0) 0x7f01e00cf850 con 0x7f01e00685c0 2024-01-11T09:47:24.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.208+0000 7f01e579a700 1 --2- 172.21.15.27:0/2618221941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01e00ccd70 0x7f01e00cd1e0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f01d00099b0 tx=0x7f01d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc9019a3dec9b7ba server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:24.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.208+0000 7f01e4f99700 1 -- 172.21.15.27:0/2618221941 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d0009ea0 con 0x7f01e00ccd70 2024-01-11T09:47:24.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.208+0000 7f01e4f99700 1 -- 172.21.15.27:0/2618221941 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d000b860 con 0x7f01e00ccd70 2024-01-11T09:47:24.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.208+0000 7f01e4f99700 1 -- 172.21.15.27:0/2618221941 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d000bc00 con 0x7f01e00ccd70 2024-01-11T09:47:24.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.209+0000 7f01e81ff700 1 -- 172.21.15.27:0/2618221941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01e00ccd70 msgr2=0x7f01e00cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:24.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.209+0000 7f01e81ff700 1 --2- 172.21.15.27:0/2618221941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01e00ccd70 0x7f01e00cd1e0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f01d00099b0 tx=0x7f01d0009250 comp rx=0 tx=0).stop 2024-01-11T09:47:24.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.209+0000 7f01e81ff700 1 -- 172.21.15.27:0/2618221941 shutdown_connections 2024-01-11T09:47:24.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.209+0000 7f01e81ff700 1 --2- 172.21.15.27:0/2618221941 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01e015a1b0 0x7f01e015a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:24.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.209+0000 7f01e81ff700 1 --2- 172.21.15.27:0/2618221941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01e00ccd70 0x7f01e00cd1e0 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:24.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.209+0000 7f01e81ff700 1 -- 172.21.15.27:0/2618221941 >> 172.21.15.27:0/2618221941 conn(0x7f01e006ed40 msgr2=0x7f01e006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:24.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.209+0000 7f01e81ff700 1 -- 172.21.15.27:0/2618221941 shutdown_connections 2024-01-11T09:47:24.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.209+0000 7f01e81ff700 1 -- 172.21.15.27:0/2618221941 wait complete. 2024-01-11T09:47:24.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.211+0000 7f01e81ff700 1 Processor -- start 2024-01-11T09:47:24.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.211+0000 7f01e81ff700 1 -- start start 2024-01-11T09:47:24.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.212+0000 7f01e81ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01e00685c0 0x7f01e0158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:24.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.213+0000 7f01e81ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01e00ccd70 0x7f01e00b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.213+0000 7f01e81ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01e015a1b0 0x7f01e00b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.213+0000 7f01e81ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e00c8060 con 0x7f01e00ccd70 2024-01-11T09:47:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.213+0000 7f01e81ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e00c7730 con 0x7f01e015a1b0 2024-01-11T09:47:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.213+0000 7f01e81ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e00c7a10 con 0x7f01e00685c0 2024-01-11T09:47:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.213+0000 7f01e579a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01e00ccd70 0x7f01e00b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.213+0000 7f01e579a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01e00ccd70 0x7f01e00b54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56094/0 (socket says 172.21.15.27:56094) 2024-01-11T09:47:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.213+0000 7f01e579a700 1 -- 172.21.15.27:0/2138360565 learned_addr learned my addr 172.21.15.27:0/2138360565 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.213+0000 7f01e5f9b700 1 --2- 172.21.15.27:0/2138360565 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01e00685c0 0x7f01e0158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.213+0000 7f01e679c700 1 --2- 172.21.15.27:0/2138360565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01e015a1b0 0x7f01e00b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.213+0000 7f01e579a700 1 -- 172.21.15.27:0/2138360565 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01e00685c0 msgr2=0x7f01e0158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.213+0000 7f01e579a700 1 --2- 172.21.15.27:0/2138360565 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01e00685c0 0x7f01e0158770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.213+0000 7f01e579a700 1 -- 172.21.15.27:0/2138360565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01e015a1b0 msgr2=0x7f01e00b5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.213+0000 7f01e579a700 1 --2- 172.21.15.27:0/2138360565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01e015a1b0 0x7f01e00b5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.213+0000 7f01e579a700 1 -- 172.21.15.27:0/2138360565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d00094f0 con 0x7f01e00ccd70 2024-01-11T09:47:24.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.214+0000 7f01e579a700 1 --2- 172.21.15.27:0/2138360565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01e00ccd70 0x7f01e00b54d0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f01e0069770 tx=0x7f01d0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.214+0000 7f01d5ffb700 1 -- 172.21.15.27:0/2138360565 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d000bbe0 con 0x7f01e00ccd70 2024-01-11T09:47:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.214+0000 7f01d5ffb700 1 -- 172.21.15.27:0/2138360565 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d0011420 con 0x7f01e00ccd70 2024-01-11T09:47:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.214+0000 7f01e81ff700 1 -- 172.21.15.27:0/2138360565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e00c7cf0 con 0x7f01e00ccd70 2024-01-11T09:47:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.214+0000 7f01d5ffb700 1 -- 172.21.15.27:0/2138360565 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d00117e0 con 0x7f01e00ccd70 2024-01-11T09:47:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.214+0000 7f01e81ff700 1 -- 172.21.15.27:0/2138360565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e016d010 con 0x7f01e00ccd70 2024-01-11T09:47:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.215+0000 7f01e81ff700 1 -- 172.21.15.27:0/2138360565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e004c3c0 con 0x7f01e00ccd70 2024-01-11T09:47:24.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.216+0000 7f01d5ffb700 1 -- 172.21.15.27:0/2138360565 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f01d0011c10 con 0x7f01e00ccd70 2024-01-11T09:47:24.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.217+0000 7f01d5ffb700 1 --2- 172.21.15.27:0/2138360565 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f01c4067170 0x7f01c4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:24.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.217+0000 7f01d5ffb700 1 -- 172.21.15.27:0/2138360565 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 5717+0+0 (secure 0 0 0) 0x7f01d0086b80 con 0x7f01e00ccd70 2024-01-11T09:47:24.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.218+0000 7f01e5f9b700 1 --2- 172.21.15.27:0/2138360565 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f01c4067170 0x7f01c4069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:24.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.218+0000 7f01e5f9b700 1 --2- 172.21.15.27:0/2138360565 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f01c4067170 0x7f01c4069620 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f01dc009ca0 tx=0x7f01dc008040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:24.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.221+0000 7f01d5ffb700 1 -- 172.21.15.27:0/2138360565 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f01d0053b30 con 0x7f01e00ccd70 2024-01-11T09:47:24.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.471+0000 7f01e81ff700 1 -- 172.21.15.27:0/2138360565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f01e00c81a0 con 0x7f01e00ccd70 2024-01-11T09:47:24.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.977+0000 7f01d5ffb700 1 -- 172.21.15.27:0/2138360565 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 34 and data pool 35 v138) v1 ==== 174+0+0 (secure 0 0 0) 0x7f01d00330c0 con 0x7f01e00ccd70 2024-01-11T09:47:24.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.983+0000 7f01e81ff700 1 -- 172.21.15.27:0/2138360565 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f01c4067170 msgr2=0x7f01c4069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:24.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.983+0000 7f01e81ff700 1 --2- 172.21.15.27:0/2138360565 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f01c4067170 0x7f01c4069620 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f01dc009ca0 tx=0x7f01dc008040 comp rx=0 tx=0).stop 2024-01-11T09:47:24.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.983+0000 7f01e81ff700 1 -- 172.21.15.27:0/2138360565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01e00ccd70 msgr2=0x7f01e00b54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:24.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.983+0000 7f01e81ff700 1 --2- 172.21.15.27:0/2138360565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01e00ccd70 0x7f01e00b54d0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f01e0069770 tx=0x7f01d0017000 comp rx=0 tx=0).stop 2024-01-11T09:47:24.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.984+0000 7f01e81ff700 1 -- 172.21.15.27:0/2138360565 shutdown_connections 2024-01-11T09:47:24.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.984+0000 7f01e81ff700 1 --2- 172.21.15.27:0/2138360565 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f01c4067170 0x7f01c4069620 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:24.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.984+0000 7f01e81ff700 1 --2- 172.21.15.27:0/2138360565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01e015a1b0 0x7f01e00b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:24.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.984+0000 7f01e81ff700 1 --2- 172.21.15.27:0/2138360565 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f01e00685c0 0x7f01e0158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:24.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.984+0000 7f01e81ff700 1 --2- 172.21.15.27:0/2138360565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01e00ccd70 0x7f01e00b54d0 secure :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f01e0069770 tx=0x7f01d0017000 comp rx=0 tx=0).stop 2024-01-11T09:47:24.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.984+0000 7f01e81ff700 1 -- 172.21.15.27:0/2138360565 >> 172.21.15.27:0/2138360565 conn(0x7f01e006ed40 msgr2=0x7f01e0068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:24.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.984+0000 7f01e81ff700 1 -- 172.21.15.27:0/2138360565 shutdown_connections 2024-01-11T09:47:24.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:24.984+0000 7f01e81ff700 1 -- 172.21.15.27:0/2138360565 wait complete. 2024-01-11T09:47:24.979 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 34 and data pool 35 2024-01-11T09:47:25.008 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:47:25.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.218+0000 7f56623ff700 1 Processor -- start 2024-01-11T09:47:25.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.218+0000 7f56623ff700 1 -- start start 2024-01-11T09:47:25.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.219+0000 7f56623ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f565c0b03e0 0x7f565c0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:25.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.219+0000 7f56623ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f565c0b07b0 0x7f565c0b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:25.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.219+0000 7f56623ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565c0cd730 con 0x7f565c159200 2024-01-11T09:47:25.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.219+0000 7f56623ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f565c159d50 con 0x7f565c0b07b0 2024-01-11T09:47:25.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.219+0000 7f56623ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0ccb20 con 0x7f565c0b03e0 2024-01-11T09:47:25.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.220+0000 7f566099c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f565c159200 0x7f565c159640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51248/0 (socket says 172.21.15.27:51248) 2024-01-11T09:47:25.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.220+0000 7f566099c700 1 -- 172.21.15.27:0/1325318372 learned_addr learned my addr 172.21.15.27:0/1325318372 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:25.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.220+0000 7f565b7fe700 1 --2- 172.21.15.27:0/1325318372 >> v2:172.21.15.77:3300/0 conn(0x7f565c0b07b0 0x7f565c0b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:25.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.220+0000 7f565bfff700 1 --2- 172.21.15.27:0/1325318372 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f565c0b03e0 0x7f565c0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:25.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.220+0000 7f565affd700 1 -- 172.21.15.27:0/1325318372 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328536448 0 0) 0x7f565c0cd730 con 0x7f565c159200 2024-01-11T09:47:25.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.220+0000 7f565affd700 1 -- 172.21.15.27:0/1325318372 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5644003570 con 0x7f565c159200 2024-01-11T09:47:25.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.220+0000 7f565b7fe700 1 -- 172.21.15.27:0/1325318372 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f565c0b03e0 msgr2=0x7f565c0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:25.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.220+0000 7f565b7fe700 1 --2- 172.21.15.27:0/1325318372 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f565c0b03e0 0x7f565c0b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.220+0000 7f565b7fe700 1 -- 172.21.15.27:0/1325318372 >> v1:172.21.15.27:6789/0 conn(0x7f565c159200 legacy=0x7f565c159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:25.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.220+0000 7f565b7fe700 1 -- 172.21.15.27:0/1325318372 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565c0cce00 con 0x7f565c0b07b0 2024-01-11T09:47:25.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.220+0000 7f565bfff700 1 --2- 172.21.15.27:0/1325318372 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f565c0b03e0 0x7f565c0b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:25.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.221+0000 7f565b7fe700 1 --2- 172.21.15.27:0/1325318372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565c0b07b0 0x7f565c0b29c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f56500098e0 tx=0x7f56500091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51b74c8deb2a2577 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:25.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.221+0000 7f565affd700 1 -- 172.21.15.27:0/1325318372 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5650011630 con 0x7f565c0b07b0 2024-01-11T09:47:25.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.221+0000 7f565affd700 1 -- 172.21.15.27:0/1325318372 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5650011790 con 0x7f565c0b07b0 2024-01-11T09:47:25.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.221+0000 7f565affd700 1 -- 172.21.15.27:0/1325318372 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5650011b30 con 0x7f565c0b07b0 2024-01-11T09:47:25.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.221+0000 7f56623ff700 1 -- 172.21.15.27:0/1325318372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565c0b07b0 msgr2=0x7f565c0b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:25.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.221+0000 7f56623ff700 1 --2- 172.21.15.27:0/1325318372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565c0b07b0 0x7f565c0b29c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f56500098e0 tx=0x7f56500091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.221+0000 7f56623ff700 1 -- 172.21.15.27:0/1325318372 shutdown_connections 2024-01-11T09:47:25.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.221+0000 7f56623ff700 1 --2- 172.21.15.27:0/1325318372 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f565c0b03e0 0x7f565c0b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.222+0000 7f56623ff700 1 --2- 172.21.15.27:0/1325318372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565c0b07b0 0x7f565c0b29c0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.222+0000 7f56623ff700 1 -- 172.21.15.27:0/1325318372 >> 172.21.15.27:0/1325318372 conn(0x7f565c06ed40 msgr2=0x7f565c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:25.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.222+0000 7f56623ff700 1 -- 172.21.15.27:0/1325318372 shutdown_connections 2024-01-11T09:47:25.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.222+0000 7f56623ff700 1 -- 172.21.15.27:0/1325318372 wait complete. 2024-01-11T09:47:25.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.223+0000 7f56623ff700 1 Processor -- start 2024-01-11T09:47:25.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.223+0000 7f56623ff700 1 -- start start 2024-01-11T09:47:25.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.224+0000 7f56623ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565c0b03e0 0x7f565c16d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:25.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.224+0000 7f56623ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f565c0b07b0 0x7f565c16db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:25.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.224+0000 7f56623ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f565c159200 0x7f565c0af1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:25.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.224+0000 7f56623ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0c80e0 con 0x7f565c0b07b0 2024-01-11T09:47:25.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.224+0000 7f56623ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0c83c0 con 0x7f565c0b03e0 2024-01-11T09:47:25.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.224+0000 7f56623ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0c86a0 con 0x7f565c159200 2024-01-11T09:47:25.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.225+0000 7f565b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f565c0b07b0 0x7f565c16db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:25.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.225+0000 7f565b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f565c0b07b0 0x7f565c16db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56096/0 (socket says 172.21.15.27:56096) 2024-01-11T09:47:25.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.225+0000 7f565b7fe700 1 -- 172.21.15.27:0/422666985 learned_addr learned my addr 172.21.15.27:0/422666985 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:25.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.225+0000 7f565bfff700 1 --2- 172.21.15.27:0/422666985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565c0b03e0 0x7f565c16d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:25.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.225+0000 7f566099c700 1 --2- 172.21.15.27:0/422666985 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f565c159200 0x7f565c0af1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:25.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.225+0000 7f565b7fe700 1 -- 172.21.15.27:0/422666985 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f565c159200 msgr2=0x7f565c0af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:25.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.225+0000 7f565b7fe700 1 --2- 172.21.15.27:0/422666985 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f565c159200 0x7f565c0af1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.225+0000 7f565b7fe700 1 -- 172.21.15.27:0/422666985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565c0b03e0 msgr2=0x7f565c16d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:25.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.225+0000 7f565b7fe700 1 --2- 172.21.15.27:0/422666985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565c0b03e0 0x7f565c16d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.225+0000 7f565b7fe700 1 -- 172.21.15.27:0/422666985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564c009420 con 0x7f565c0b07b0 2024-01-11T09:47:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.225+0000 7f565b7fe700 1 --2- 172.21.15.27:0/422666985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f565c0b07b0 0x7f565c16db00 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f565c069540 tx=0x7f5650017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.226+0000 7f5643fff700 1 -- 172.21.15.27:0/422666985 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5650011900 con 0x7f565c0b07b0 2024-01-11T09:47:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.226+0000 7f5643fff700 1 -- 172.21.15.27:0/422666985 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5650011a60 con 0x7f565c0b07b0 2024-01-11T09:47:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.226+0000 7f5643fff700 1 -- 172.21.15.27:0/422666985 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5650025640 con 0x7f565c0b07b0 2024-01-11T09:47:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.226+0000 7f56623ff700 1 -- 172.21.15.27:0/422666985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5650009420 con 0x7f565c0b07b0 2024-01-11T09:47:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.226+0000 7f56623ff700 1 -- 172.21.15.27:0/422666985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565c0c8c60 con 0x7f565c0b07b0 2024-01-11T09:47:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.227+0000 7f56623ff700 1 -- 172.21.15.27:0/422666985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565c0c8e80 con 0x7f565c0b07b0 2024-01-11T09:47:25.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.228+0000 7f5643fff700 1 -- 172.21.15.27:0/422666985 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f5650025a30 con 0x7f565c0b07b0 2024-01-11T09:47:25.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.229+0000 7f5643fff700 1 --2- 172.21.15.27:0/422666985 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5648067170 0x7f5648069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:25.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.229+0000 7f5643fff700 1 -- 172.21.15.27:0/422666985 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f5650086090 con 0x7f565c0b07b0 2024-01-11T09:47:25.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.230+0000 7f565bfff700 1 --2- 172.21.15.27:0/422666985 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5648067170 0x7f5648069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:25.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.230+0000 7f565bfff700 1 --2- 172.21.15.27:0/422666985 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5648067170 0x7f5648069620 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f565c0cacb0 tx=0x7f564c0091a0 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:25.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.234+0000 7f5643fff700 1 -- 172.21.15.27:0/422666985 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5650052f60 con 0x7f565c0b07b0 2024-01-11T09:47:25.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.438+0000 7f56623ff700 1 -- 172.21.15.27:0/422666985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f565c0718d0 con 0x7f565c0b07b0 2024-01-11T09:47:25.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.439+0000 7f5643fff700 1 -- 172.21.15.27:0/422666985 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v255) v1 ==== 74+0+15508 (secure 0 0 0) 0x7f5650056580 con 0x7f565c0b07b0 2024-01-11T09:47:25.430 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:25.430 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":255,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:47:24.473723+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:47:22.374454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-01-11T09:47:23.471379+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:47:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.441+0000 7f56623ff700 1 -- 172.21.15.27:0/422666985 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5648067170 msgr2=0x7f5648069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.441+0000 7f56623ff700 1 --2- 172.21.15.27:0/422666985 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5648067170 0x7f5648069620 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f565c0cacb0 tx=0x7f564c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.441+0000 7f56623ff700 1 -- 172.21.15.27:0/422666985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f565c0b07b0 msgr2=0x7f565c16db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.441+0000 7f56623ff700 1 --2- 172.21.15.27:0/422666985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f565c0b07b0 0x7f565c16db00 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f565c069540 tx=0x7f5650017000 comp rx=0 tx=0).stop 2024-01-11T09:47:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.441+0000 7f56623ff700 1 -- 172.21.15.27:0/422666985 shutdown_connections 2024-01-11T09:47:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.441+0000 7f56623ff700 1 --2- 172.21.15.27:0/422666985 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f565c159200 0x7f565c0af1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.441+0000 7f56623ff700 1 --2- 172.21.15.27:0/422666985 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5648067170 0x7f5648069620 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.441+0000 7f56623ff700 1 --2- 172.21.15.27:0/422666985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565c0b03e0 0x7f565c16d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.441+0000 7f56623ff700 1 --2- 172.21.15.27:0/422666985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f565c0b07b0 0x7f565c16db00 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.441+0000 7f56623ff700 1 -- 172.21.15.27:0/422666985 >> 172.21.15.27:0/422666985 conn(0x7f565c06ed40 msgr2=0x7f565c158a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.441+0000 7f56623ff700 1 -- 172.21.15.27:0/422666985 shutdown_connections 2024-01-11T09:47:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.442+0000 7f56623ff700 1 -- 172.21.15.27:0/422666985 wait complete. 2024-01-11T09:47:25.451 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:47:25.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.660+0000 7fcd2ffff700 1 Processor -- start 2024-01-11T09:47:25.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.660+0000 7fcd2ffff700 1 -- start start 2024-01-11T09:47:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.661+0000 7fcd2ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcd28159650 0x7fcd2806bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.661+0000 7fcd2ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcd28159a20 0x7fcd280b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.661+0000 7fcd2ffff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd280b2730 0x7fcd280b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.661+0000 7fcd2ffff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd2806e700 con 0x7fcd28159650 2024-01-11T09:47:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.661+0000 7fcd2ffff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd280b11a0 con 0x7fcd28159a20 2024-01-11T09:47:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.661+0000 7fcd2ffff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2806c360 con 0x7fcd280b2730 2024-01-11T09:47:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.661+0000 7fcd2dd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcd28159650 0x7fcd2806bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.662+0000 7fcd2d59a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcd28159a20 0x7fcd280b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.662+0000 7fcd2e59c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd280b2730 0x7fcd280b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.662+0000 7fcd2dd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcd28159650 0x7fcd2806bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56110/0 (socket says 172.21.15.27:56110) 2024-01-11T09:47:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.662+0000 7fcd2dd9b700 1 -- 172.21.15.27:0/342555398 learned_addr learned my addr 172.21.15.27:0/342555398 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.662+0000 7fcd2e59c700 1 -- 172.21.15.27:0/342555398 >> v2:172.21.15.77:3300/0 conn(0x7fcd28159a20 msgr2=0x7fcd280b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.662+0000 7fcd2e59c700 1 --2- 172.21.15.27:0/342555398 >> v2:172.21.15.77:3300/0 conn(0x7fcd28159a20 0x7fcd280b0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.662+0000 7fcd2e59c700 1 -- 172.21.15.27:0/342555398 >> v2:172.21.15.27:3300/0 conn(0x7fcd28159650 msgr2=0x7fcd2806bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.662+0000 7fcd2e59c700 1 --2- 172.21.15.27:0/342555398 >> v2:172.21.15.27:3300/0 conn(0x7fcd28159650 0x7fcd2806bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.662+0000 7fcd2e59c700 1 -- 172.21.15.27:0/342555398 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd2806c640 con 0x7fcd280b2730 2024-01-11T09:47:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.662+0000 7fcd2e59c700 1 --2- 172.21.15.27:0/342555398 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd280b2730 0x7fcd280b2b90 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fcd28069c00 tx=0x7fcd20009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c00a0cffa791d5d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:25.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.663+0000 7fcd2cd99700 1 -- 172.21.15.27:0/342555398 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2000e070 con 0x7fcd280b2730 2024-01-11T09:47:25.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.663+0000 7fcd2cd99700 1 -- 172.21.15.27:0/342555398 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2000bab0 con 0x7fcd280b2730 2024-01-11T09:47:25.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.663+0000 7fcd2cd99700 1 -- 172.21.15.27:0/342555398 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2000be70 con 0x7fcd280b2730 2024-01-11T09:47:25.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.663+0000 7fcd2ffff700 1 -- 172.21.15.27:0/342555398 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd280b2730 msgr2=0x7fcd280b2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:25.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.663+0000 7fcd2ffff700 1 --2- 172.21.15.27:0/342555398 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd280b2730 0x7fcd280b2b90 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fcd28069c00 tx=0x7fcd20009780 comp rx=0 tx=0).stop 2024-01-11T09:47:25.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.663+0000 7fcd2ffff700 1 -- 172.21.15.27:0/342555398 shutdown_connections 2024-01-11T09:47:25.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.663+0000 7fcd2ffff700 1 --2- 172.21.15.27:0/342555398 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd280b2730 0x7fcd280b2b90 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.663+0000 7fcd2ffff700 1 --2- 172.21.15.27:0/342555398 >> v2:172.21.15.27:3300/0 conn(0x7fcd28159650 0x7fcd2806bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.663+0000 7fcd2ffff700 1 --2- 172.21.15.27:0/342555398 >> v2:172.21.15.77:3300/0 conn(0x7fcd28159a20 0x7fcd280b0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.663+0000 7fcd2ffff700 1 -- 172.21.15.27:0/342555398 >> 172.21.15.27:0/342555398 conn(0x7fcd2806b200 msgr2=0x7fcd280aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:25.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.663+0000 7fcd2ffff700 1 -- 172.21.15.27:0/342555398 shutdown_connections 2024-01-11T09:47:25.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.663+0000 7fcd2ffff700 1 -- 172.21.15.27:0/342555398 wait complete. 2024-01-11T09:47:25.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.665+0000 7fcd2ffff700 1 Processor -- start 2024-01-11T09:47:25.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.665+0000 7fcd2ffff700 1 -- start start 2024-01-11T09:47:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.666+0000 7fcd2ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd280b2730 0x7fcd28142cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.666+0000 7fcd2ffff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd28159650 0x7fcd2816cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.666+0000 7fcd2ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd28159a20 0x7fcd2816d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.666+0000 7fcd2ffff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2815fa60 con 0x7fcd28159a20 2024-01-11T09:47:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.666+0000 7fcd2ffff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2815fd40 con 0x7fcd280b2730 2024-01-11T09:47:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.666+0000 7fcd2ffff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd28160020 con 0x7fcd28159650 2024-01-11T09:47:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.666+0000 7fcd2dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd280b2730 0x7fcd28142cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.666+0000 7fcd2e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd28159a20 0x7fcd2816d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.666+0000 7fcd2dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd280b2730 0x7fcd28142cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:51968/0 (socket says 172.21.15.27:51968) 2024-01-11T09:47:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.666+0000 7fcd2dd9b700 1 -- 172.21.15.27:0/209189033 learned_addr learned my addr 172.21.15.27:0/209189033 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.666+0000 7fcd2d59a700 1 --2- 172.21.15.27:0/209189033 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd28159650 0x7fcd2816cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.667+0000 7fcd2dd9b700 1 -- 172.21.15.27:0/209189033 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd28159650 msgr2=0x7fcd2816cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.667+0000 7fcd2dd9b700 1 --2- 172.21.15.27:0/209189033 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd28159650 0x7fcd2816cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.667+0000 7fcd2dd9b700 1 -- 172.21.15.27:0/209189033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd28159a20 msgr2=0x7fcd2816d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.667+0000 7fcd2dd9b700 1 --2- 172.21.15.27:0/209189033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd28159a20 0x7fcd2816d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.667+0000 7fcd2dd9b700 1 -- 172.21.15.27:0/209189033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd20009a20 con 0x7fcd280b2730 2024-01-11T09:47:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.667+0000 7fcd2dd9b700 1 --2- 172.21.15.27:0/209189033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd280b2730 0x7fcd28142cf0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fcd2400ea50 tx=0x7fcd24009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.667+0000 7fcd1dffb700 1 -- 172.21.15.27:0/209189033 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2400c640 con 0x7fcd280b2730 2024-01-11T09:47:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.667+0000 7fcd1dffb700 1 -- 172.21.15.27:0/209189033 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2400c7a0 con 0x7fcd280b2730 2024-01-11T09:47:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.667+0000 7fcd1dffb700 1 -- 172.21.15.27:0/209189033 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2400cb20 con 0x7fcd280b2730 2024-01-11T09:47:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.667+0000 7fcd2ffff700 1 -- 172.21.15.27:0/209189033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd280aa9c0 con 0x7fcd280b2730 2024-01-11T09:47:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.668+0000 7fcd2ffff700 1 -- 172.21.15.27:0/209189033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd28143230 con 0x7fcd280b2730 2024-01-11T09:47:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.669+0000 7fcd2ffff700 1 -- 172.21.15.27:0/209189033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd280718d0 con 0x7fcd280b2730 2024-01-11T09:47:25.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.670+0000 7fcd1dffb700 1 -- 172.21.15.27:0/209189033 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fcd24013470 con 0x7fcd280b2730 2024-01-11T09:47:25.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.671+0000 7fcd1dffb700 1 --2- 172.21.15.27:0/209189033 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcd0c067170 0x7fcd0c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:25.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.671+0000 7fcd1dffb700 1 -- 172.21.15.27:0/209189033 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 5817+0+0 (secure 0 0 0) 0x7fcd24084b90 con 0x7fcd280b2730 2024-01-11T09:47:25.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.671+0000 7fcd2d59a700 1 --2- 172.21.15.27:0/209189033 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcd0c067170 0x7fcd0c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:25.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.672+0000 7fcd2d59a700 1 --2- 172.21.15.27:0/209189033 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcd0c067170 0x7fcd0c069620 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcd280b0880 tx=0x7fcd180091a0 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:25.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.675+0000 7fcd1dffb700 1 -- 172.21.15.27:0/209189033 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcd24051990 con 0x7fcd280b2730 2024-01-11T09:47:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.879+0000 7fcd2ffff700 1 -- 172.21.15.27:0/209189033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd2806c920 con 0x7fcd280b2730 2024-01-11T09:47:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.880+0000 7fcd1dffb700 1 -- 172.21.15.27:0/209189033 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v255) v1 ==== 74+0+15508 (secure 0 0 0) 0x7fcd24054fb0 con 0x7fcd280b2730 2024-01-11T09:47:25.872 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:25.872 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":255,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:47:24.473723+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:47:22.374454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-01-11T09:47:23.471379+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:47:25.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.883+0000 7fcd2ffff700 1 -- 172.21.15.27:0/209189033 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcd0c067170 msgr2=0x7fcd0c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:25.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.883+0000 7fcd2ffff700 1 --2- 172.21.15.27:0/209189033 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcd0c067170 0x7fcd0c069620 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcd280b0880 tx=0x7fcd180091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.883+0000 7fcd2ffff700 1 -- 172.21.15.27:0/209189033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd280b2730 msgr2=0x7fcd28142cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:25.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.883+0000 7fcd2ffff700 1 --2- 172.21.15.27:0/209189033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd280b2730 0x7fcd28142cf0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fcd2400ea50 tx=0x7fcd24009250 comp rx=0 tx=0).stop 2024-01-11T09:47:25.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.883+0000 7fcd2ffff700 1 -- 172.21.15.27:0/209189033 shutdown_connections 2024-01-11T09:47:25.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.883+0000 7fcd2ffff700 1 --2- 172.21.15.27:0/209189033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd28159a20 0x7fcd2816d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.883+0000 7fcd2ffff700 1 --2- 172.21.15.27:0/209189033 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcd0c067170 0x7fcd0c069620 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.883+0000 7fcd2ffff700 1 --2- 172.21.15.27:0/209189033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd280b2730 0x7fcd28142cf0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.883+0000 7fcd2ffff700 1 --2- 172.21.15.27:0/209189033 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcd28159650 0x7fcd2816cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:25.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.883+0000 7fcd2ffff700 1 -- 172.21.15.27:0/209189033 >> 172.21.15.27:0/209189033 conn(0x7fcd2806b200 msgr2=0x7fcd28159df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:25.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.883+0000 7fcd2ffff700 1 -- 172.21.15.27:0/209189033 shutdown_connections 2024-01-11T09:47:25.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:25.883+0000 7fcd2ffff700 1 -- 172.21.15.27:0/209189033 wait complete. 2024-01-11T09:47:25.892 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-11T09:47:26.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.107+0000 7f31957ff700 1 Processor -- start 2024-01-11T09:47:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.107+0000 7f31957ff700 1 -- start start 2024-01-11T09:47:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.108+0000 7f31957ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f319015a1b0 0x7f319015a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:26.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.108+0000 7f31957ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f31900ccd70 0x7f31900cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.108+0000 7f31957ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31900cf850 con 0x7f31900685c0 2024-01-11T09:47:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.108+0000 7f31957ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31900cec40 con 0x7f31900ccd70 2024-01-11T09:47:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.109+0000 7f31957ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31900cef20 con 0x7f319015a1b0 2024-01-11T09:47:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.109+0000 7f318e7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f31900ccd70 0x7f31900cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.109+0000 7f318e7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f31900ccd70 0x7f31900cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56138/0 (socket says 172.21.15.27:56138) 2024-01-11T09:47:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.109+0000 7f318e7fc700 1 -- 172.21.15.27:0/4000159964 learned_addr learned my addr 172.21.15.27:0/4000159964 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.109+0000 7f318effd700 1 --2- 172.21.15.27:0/4000159964 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f319015a1b0 0x7f319015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.109+0000 7f318e7fc700 1 -- 172.21.15.27:0/4000159964 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f319015a1b0 msgr2=0x7f319015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.109+0000 7f318e7fc700 1 --2- 172.21.15.27:0/4000159964 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f319015a1b0 0x7f319015a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.109+0000 7f318e7fc700 1 -- 172.21.15.27:0/4000159964 >> v1:172.21.15.77:6789/0 conn(0x7f31900685c0 legacy=0x7f3190068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.109+0000 7f318e7fc700 1 -- 172.21.15.27:0/4000159964 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31900cf200 con 0x7f31900ccd70 2024-01-11T09:47:26.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.109+0000 7f318effd700 1 --2- 172.21.15.27:0/4000159964 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f319015a1b0 0x7f319015a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.110+0000 7f318e7fc700 1 --2- 172.21.15.27:0/4000159964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31900ccd70 0x7f31900cd1e0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f31840099b0 tx=0x7f3184009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bddbee2c6c91deb server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.110+0000 7f318dffb700 1 -- 172.21.15.27:0/4000159964 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318400b860 con 0x7f31900ccd70 2024-01-11T09:47:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.110+0000 7f318dffb700 1 -- 172.21.15.27:0/4000159964 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f318400b9c0 con 0x7f31900ccd70 2024-01-11T09:47:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.110+0000 7f318dffb700 1 -- 172.21.15.27:0/4000159964 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318400bd60 con 0x7f31900ccd70 2024-01-11T09:47:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.110+0000 7f31957ff700 1 -- 172.21.15.27:0/4000159964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31900ccd70 msgr2=0x7f31900cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.110+0000 7f31957ff700 1 --2- 172.21.15.27:0/4000159964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31900ccd70 0x7f31900cd1e0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f31840099b0 tx=0x7f3184009250 comp rx=0 tx=0).stop 2024-01-11T09:47:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.111+0000 7f31957ff700 1 -- 172.21.15.27:0/4000159964 shutdown_connections 2024-01-11T09:47:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.111+0000 7f31957ff700 1 --2- 172.21.15.27:0/4000159964 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f319015a1b0 0x7f319015a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.111+0000 7f31957ff700 1 --2- 172.21.15.27:0/4000159964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31900ccd70 0x7f31900cd1e0 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.111+0000 7f31957ff700 1 -- 172.21.15.27:0/4000159964 >> 172.21.15.27:0/4000159964 conn(0x7f319006ed40 msgr2=0x7f319006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:26.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.111+0000 7f31957ff700 1 -- 172.21.15.27:0/4000159964 shutdown_connections 2024-01-11T09:47:26.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.111+0000 7f31957ff700 1 -- 172.21.15.27:0/4000159964 wait complete. 2024-01-11T09:47:26.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.113+0000 7f31957ff700 1 Processor -- start 2024-01-11T09:47:26.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.113+0000 7f31957ff700 1 -- start start 2024-01-11T09:47:26.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.114+0000 7f31957ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31900685c0 0x7f31901540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:26.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.114+0000 7f31957ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31900ccd70 0x7f3190145130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:26.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.114+0000 7f31957ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f319015a1b0 0x7f3190151470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:26.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.114+0000 7f31957ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3190155dc0 con 0x7f31900ccd70 2024-01-11T09:47:26.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.114+0000 7f31957ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3190154720 con 0x7f31900685c0 2024-01-11T09:47:26.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.114+0000 7f31957ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3190155490 con 0x7f319015a1b0 2024-01-11T09:47:26.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.114+0000 7f318e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31900ccd70 0x7f3190145130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:26.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.115+0000 7f318e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31900ccd70 0x7f3190145130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56154/0 (socket says 172.21.15.27:56154) 2024-01-11T09:47:26.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.115+0000 7f318e7fc700 1 -- 172.21.15.27:0/1068741680 learned_addr learned my addr 172.21.15.27:0/1068741680 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:26.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.115+0000 7f318effd700 1 --2- 172.21.15.27:0/1068741680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31900685c0 0x7f31901540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.115+0000 7f318f7fe700 1 --2- 172.21.15.27:0/1068741680 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f319015a1b0 0x7f3190151470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.115+0000 7f318e7fc700 1 -- 172.21.15.27:0/1068741680 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f319015a1b0 msgr2=0x7f3190151470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.115+0000 7f318e7fc700 1 --2- 172.21.15.27:0/1068741680 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f319015a1b0 0x7f3190151470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.115+0000 7f318e7fc700 1 -- 172.21.15.27:0/1068741680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31900685c0 msgr2=0x7f31901540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.115+0000 7f318e7fc700 1 --2- 172.21.15.27:0/1068741680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31900685c0 0x7f31901540b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.115+0000 7f318e7fc700 1 -- 172.21.15.27:0/1068741680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31840094f0 con 0x7f31900ccd70 2024-01-11T09:47:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.115+0000 7f318effd700 1 --2- 172.21.15.27:0/1068741680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31900685c0 0x7f31901540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.115+0000 7f318e7fc700 1 --2- 172.21.15.27:0/1068741680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31900ccd70 0x7f3190145130 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f3184009290 tx=0x7f3184017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.115+0000 7f3176ffd700 1 -- 172.21.15.27:0/1068741680 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3184026480 con 0x7f31900ccd70 2024-01-11T09:47:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.116+0000 7f3176ffd700 1 -- 172.21.15.27:0/1068741680 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f318401f070 con 0x7f31900ccd70 2024-01-11T09:47:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.116+0000 7f3176ffd700 1 -- 172.21.15.27:0/1068741680 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3184026830 con 0x7f31900ccd70 2024-01-11T09:47:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.116+0000 7f31957ff700 1 -- 172.21.15.27:0/1068741680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3190155770 con 0x7f31900ccd70 2024-01-11T09:47:26.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.116+0000 7f31957ff700 1 -- 172.21.15.27:0/1068741680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31901519b0 con 0x7f31900ccd70 2024-01-11T09:47:26.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.117+0000 7f31957ff700 1 -- 172.21.15.27:0/1068741680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3190155f70 con 0x7f31900ccd70 2024-01-11T09:47:26.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.118+0000 7f3176ffd700 1 -- 172.21.15.27:0/1068741680 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f3184026ba0 con 0x7f31900ccd70 2024-01-11T09:47:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.119+0000 7f3176ffd700 1 --2- 172.21.15.27:0/1068741680 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f317c067170 0x7f317c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.119+0000 7f3176ffd700 1 -- 172.21.15.27:0/1068741680 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f3184021070 con 0x7f31900ccd70 2024-01-11T09:47:26.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.119+0000 7f318effd700 1 --2- 172.21.15.27:0/1068741680 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f317c067170 0x7f317c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:26.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.120+0000 7f318effd700 1 --2- 172.21.15.27:0/1068741680 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f317c067170 0x7f317c069620 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3190154040 tx=0x7f3180008040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:26.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.123+0000 7f3176ffd700 1 -- 172.21.15.27:0/1068741680 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f31840536c0 con 0x7f31900ccd70 2024-01-11T09:47:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.374+0000 7f31957ff700 1 -- 172.21.15.27:0/1068741680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f319004c3c0 con 0x7f31900ccd70 2024-01-11T09:47:26.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.986+0000 7f3176ffd700 1 -- 172.21.15.27:0/1068741680 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v140) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3184056ce0 con 0x7f31900ccd70 2024-01-11T09:47:26.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.989+0000 7f31957ff700 1 -- 172.21.15.27:0/1068741680 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f317c067170 msgr2=0x7f317c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.989+0000 7f31957ff700 1 --2- 172.21.15.27:0/1068741680 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f317c067170 0x7f317c069620 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3190154040 tx=0x7f3180008040 comp rx=0 tx=0).stop 2024-01-11T09:47:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.989+0000 7f31957ff700 1 -- 172.21.15.27:0/1068741680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31900ccd70 msgr2=0x7f3190145130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.989+0000 7f31957ff700 1 --2- 172.21.15.27:0/1068741680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31900ccd70 0x7f3190145130 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f3184009290 tx=0x7f3184017000 comp rx=0 tx=0).stop 2024-01-11T09:47:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.989+0000 7f31957ff700 1 -- 172.21.15.27:0/1068741680 shutdown_connections 2024-01-11T09:47:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.989+0000 7f31957ff700 1 --2- 172.21.15.27:0/1068741680 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f319015a1b0 0x7f3190151470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.989+0000 7f31957ff700 1 --2- 172.21.15.27:0/1068741680 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f317c067170 0x7f317c069620 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.989+0000 7f31957ff700 1 --2- 172.21.15.27:0/1068741680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31900685c0 0x7f31901540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.989+0000 7f31957ff700 1 --2- 172.21.15.27:0/1068741680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31900ccd70 0x7f3190145130 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.989+0000 7f31957ff700 1 -- 172.21.15.27:0/1068741680 >> 172.21.15.27:0/1068741680 conn(0x7f319006ed40 msgr2=0x7f31900c5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.989+0000 7f31957ff700 1 -- 172.21.15.27:0/1068741680 shutdown_connections 2024-01-11T09:47:26.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:26.989+0000 7f31957ff700 1 -- 172.21.15.27:0/1068741680 wait complete. 2024-01-11T09:47:27.000 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-11T09:47:27.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.212+0000 7f61727ff700 1 Processor -- start 2024-01-11T09:47:27.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.212+0000 7f61727ff700 1 -- start start 2024-01-11T09:47:27.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.214+0000 7f61727ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f616c0ccd70 0x7f616c0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:27.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.214+0000 7f61727ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f616c15ae70 0x7f616c15b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:27.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.214+0000 7f61727ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616c0cf850 con 0x7f616c15a1b0 2024-01-11T09:47:27.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.214+0000 7f61727ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f616c15ac90 con 0x7f616c15ae70 2024-01-11T09:47:27.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.214+0000 7f61727ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0cec40 con 0x7f616c0ccd70 2024-01-11T09:47:27.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.214+0000 7f616bfff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f616c15a1b0 0x7f616c15a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51260/0 (socket says 172.21.15.27:51260) 2024-01-11T09:47:27.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.214+0000 7f616bfff700 1 -- 172.21.15.27:0/2538951189 learned_addr learned my addr 172.21.15.27:0/2538951189 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:27.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.214+0000 7f6170d9c700 1 --2- 172.21.15.27:0/2538951189 >> v2:172.21.15.27:3300/0 conn(0x7f616c15ae70 0x7f616c15b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:27.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.214+0000 7f616b7fe700 1 --2- 172.21.15.27:0/2538951189 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f616c0ccd70 0x7f616c0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:27.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.214+0000 7f616affd700 1 -- 172.21.15.27:0/2538951189 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967852358 0 0) 0x7f616c0cf850 con 0x7f616c15a1b0 2024-01-11T09:47:27.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.214+0000 7f616affd700 1 -- 172.21.15.27:0/2538951189 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6154003570 con 0x7f616c15a1b0 2024-01-11T09:47:27.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.215+0000 7f6170d9c700 1 -- 172.21.15.27:0/2538951189 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f616c0ccd70 msgr2=0x7f616c0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:27.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.215+0000 7f6170d9c700 1 --2- 172.21.15.27:0/2538951189 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f616c0ccd70 0x7f616c0cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:27.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.215+0000 7f6170d9c700 1 -- 172.21.15.27:0/2538951189 >> v1:172.21.15.27:6789/0 conn(0x7f616c15a1b0 legacy=0x7f616c15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:27.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.215+0000 7f6170d9c700 1 -- 172.21.15.27:0/2538951189 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c0cef20 con 0x7f616c15ae70 2024-01-11T09:47:27.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.215+0000 7f6170d9c700 1 --2- 172.21.15.27:0/2538951189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f616c15ae70 0x7f616c15b2f0 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f6164009e60 tx=0x7f6164009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c8c183dd0fea22 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:27.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.215+0000 7f616affd700 1 -- 172.21.15.27:0/2538951189 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616400e070 con 0x7f616c15ae70 2024-01-11T09:47:27.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.215+0000 7f616affd700 1 -- 172.21.15.27:0/2538951189 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616400bb60 con 0x7f616c15ae70 2024-01-11T09:47:27.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.215+0000 7f616affd700 1 -- 172.21.15.27:0/2538951189 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6164012430 con 0x7f616c15ae70 2024-01-11T09:47:27.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.216+0000 7f61727ff700 1 -- 172.21.15.27:0/2538951189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f616c15ae70 msgr2=0x7f616c15b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:27.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.216+0000 7f61727ff700 1 --2- 172.21.15.27:0/2538951189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f616c15ae70 0x7f616c15b2f0 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f6164009e60 tx=0x7f6164009780 comp rx=0 tx=0).stop 2024-01-11T09:47:27.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.216+0000 7f61727ff700 1 -- 172.21.15.27:0/2538951189 shutdown_connections 2024-01-11T09:47:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.216+0000 7f61727ff700 1 --2- 172.21.15.27:0/2538951189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f616c15ae70 0x7f616c15b2f0 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.216+0000 7f61727ff700 1 --2- 172.21.15.27:0/2538951189 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f616c0ccd70 0x7f616c0cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.216+0000 7f61727ff700 1 -- 172.21.15.27:0/2538951189 >> 172.21.15.27:0/2538951189 conn(0x7f616c06ed40 msgr2=0x7f616c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.216+0000 7f61727ff700 1 -- 172.21.15.27:0/2538951189 shutdown_connections 2024-01-11T09:47:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.216+0000 7f61727ff700 1 -- 172.21.15.27:0/2538951189 wait complete. 2024-01-11T09:47:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.217+0000 7f61727ff700 1 Processor -- start 2024-01-11T09:47:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.218+0000 7f61727ff700 1 -- start start 2024-01-11T09:47:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.218+0000 7f61727ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f616c0ccd70 0x7f616c148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.218+0000 7f61727ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f616c15a1b0 0x7f616c15da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.218+0000 7f61727ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f616c15ae70 0x7f616c15df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.219+0000 7f61727ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c1485c0 con 0x7f616c0ccd70 2024-01-11T09:47:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.219+0000 7f61727ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0b4330 con 0x7f616c15ae70 2024-01-11T09:47:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.219+0000 7f61727ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0b2670 con 0x7f616c15a1b0 2024-01-11T09:47:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.219+0000 7f616bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f616c0ccd70 0x7f616c148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.219+0000 7f616bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f616c0ccd70 0x7f616c148080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56166/0 (socket says 172.21.15.27:56166) 2024-01-11T09:47:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.219+0000 7f616bfff700 1 -- 172.21.15.27:0/3582380199 learned_addr learned my addr 172.21.15.27:0/3582380199 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.219+0000 7f6170d9c700 1 --2- 172.21.15.27:0/3582380199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f616c15ae70 0x7f616c15df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.219+0000 7f616bfff700 1 -- 172.21.15.27:0/3582380199 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f616c15a1b0 msgr2=0x7f616c15da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.219+0000 7f616b7fe700 1 --2- 172.21.15.27:0/3582380199 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f616c15a1b0 0x7f616c15da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.219+0000 7f616bfff700 1 --2- 172.21.15.27:0/3582380199 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f616c15a1b0 0x7f616c15da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.219+0000 7f616bfff700 1 -- 172.21.15.27:0/3582380199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f616c15ae70 msgr2=0x7f616c15df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.219+0000 7f616bfff700 1 --2- 172.21.15.27:0/3582380199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f616c15ae70 0x7f616c15df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.219+0000 7f616bfff700 1 -- 172.21.15.27:0/3582380199 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6164009a20 con 0x7f616c0ccd70 2024-01-11T09:47:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.219+0000 7f6170d9c700 1 --2- 172.21.15.27:0/3582380199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f616c15ae70 0x7f616c15df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:27.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.219+0000 7f616b7fe700 1 --2- 172.21.15.27:0/3582380199 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f616c15a1b0 0x7f616c15da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:27.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.219+0000 7f616bfff700 1 --2- 172.21.15.27:0/3582380199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f616c0ccd70 0x7f616c148080 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f615c00c7c0 tx=0x7f615c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:27.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.220+0000 7f6153fff700 1 -- 172.21.15.27:0/3582380199 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c005b30 con 0x7f616c0ccd70 2024-01-11T09:47:27.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.220+0000 7f6153fff700 1 -- 172.21.15.27:0/3582380199 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615c005c90 con 0x7f616c0ccd70 2024-01-11T09:47:27.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.220+0000 7f61727ff700 1 -- 172.21.15.27:0/3582380199 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616c0b45d0 con 0x7f616c0ccd70 2024-01-11T09:47:27.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.220+0000 7f6153fff700 1 -- 172.21.15.27:0/3582380199 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c007340 con 0x7f616c0ccd70 2024-01-11T09:47:27.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.220+0000 7f61727ff700 1 -- 172.21.15.27:0/3582380199 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616c0b2950 con 0x7f616c0ccd70 2024-01-11T09:47:27.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.221+0000 7f61727ff700 1 -- 172.21.15.27:0/3582380199 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616c0cfe50 con 0x7f616c0ccd70 2024-01-11T09:47:27.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.222+0000 7f6153fff700 1 -- 172.21.15.27:0/3582380199 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f615c01b400 con 0x7f616c0ccd70 2024-01-11T09:47:27.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.223+0000 7f6153fff700 1 --2- 172.21.15.27:0/3582380199 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6158067170 0x7f6158069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:27.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.223+0000 7f6153fff700 1 -- 172.21.15.27:0/3582380199 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f615c084440 con 0x7f616c0ccd70 2024-01-11T09:47:27.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.223+0000 7f616b7fe700 1 --2- 172.21.15.27:0/3582380199 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6158067170 0x7f6158069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.224+0000 7f616b7fe700 1 --2- 172.21.15.27:0/3582380199 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6158067170 0x7f6158069620 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f616c16cd90 tx=0x7f6160008040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:27.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.227+0000 7f6153fff700 1 -- 172.21.15.27:0/3582380199 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f615c051360 con 0x7f616c0ccd70 2024-01-11T09:47:27.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.476+0000 7f61727ff700 1 -- 172.21.15.27:0/3582380199 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f616c04c3c0 con 0x7f616c0ccd70 2024-01-11T09:47:27.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.992+0000 7f6153fff700 1 -- 172.21.15.27:0/3582380199 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v141) v1 ==== 122+0+0 (secure 0 0 0) 0x7f615c054980 con 0x7f616c0ccd70 2024-01-11T09:47:27.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.994+0000 7f61727ff700 1 -- 172.21.15.27:0/3582380199 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6158067170 msgr2=0x7f6158069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:27.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.994+0000 7f61727ff700 1 --2- 172.21.15.27:0/3582380199 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6158067170 0x7f6158069620 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f616c16cd90 tx=0x7f6160008040 comp rx=0 tx=0).stop 2024-01-11T09:47:27.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.994+0000 7f61727ff700 1 -- 172.21.15.27:0/3582380199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f616c0ccd70 msgr2=0x7f616c148080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:27.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.994+0000 7f61727ff700 1 --2- 172.21.15.27:0/3582380199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f616c0ccd70 0x7f616c148080 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f615c00c7c0 tx=0x7f615c007e00 comp rx=0 tx=0).stop 2024-01-11T09:47:27.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.994+0000 7f61727ff700 1 -- 172.21.15.27:0/3582380199 shutdown_connections 2024-01-11T09:47:27.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.994+0000 7f61727ff700 1 --2- 172.21.15.27:0/3582380199 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6158067170 0x7f6158069620 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:27.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.994+0000 7f61727ff700 1 --2- 172.21.15.27:0/3582380199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f616c15ae70 0x7f616c15df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:27.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.994+0000 7f61727ff700 1 --2- 172.21.15.27:0/3582380199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f616c0ccd70 0x7f616c148080 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:27.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.994+0000 7f61727ff700 1 --2- 172.21.15.27:0/3582380199 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f616c15a1b0 0x7f616c15da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:27.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.994+0000 7f61727ff700 1 -- 172.21.15.27:0/3582380199 >> 172.21.15.27:0/3582380199 conn(0x7f616c06ed40 msgr2=0x7f616c155e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:27.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.994+0000 7f61727ff700 1 -- 172.21.15.27:0/3582380199 shutdown_connections 2024-01-11T09:47:27.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:27.995+0000 7f61727ff700 1 -- 172.21.15.27:0/3582380199 wait complete. 2024-01-11T09:47:28.004 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:47:28.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.213+0000 7f96b15ff700 1 Processor -- start 2024-01-11T09:47:28.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.213+0000 7f96b15ff700 1 -- start start 2024-01-11T09:47:28.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.215+0000 7f96b15ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f96ac1596c0 0x7f96ac15d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:28.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.215+0000 7f96b15ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96ac0ca670 0x7f96ac068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:28.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.215+0000 7f96b15ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac0a9cb0 con 0x7f96ac068b60 2024-01-11T09:47:28.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.215+0000 7f96b15ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96ac068fe0 con 0x7f96ac1596c0 2024-01-11T09:47:28.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.215+0000 7f96b15ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac15e210 con 0x7f96ac0ca670 2024-01-11T09:47:28.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.215+0000 7f96ab7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f96ac068b60 0x7f96ac159a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51274/0 (socket says 172.21.15.27:51274) 2024-01-11T09:47:28.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.215+0000 7f96ab7fe700 1 -- 172.21.15.27:0/1727592219 learned_addr learned my addr 172.21.15.27:0/1727592219 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:28.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.215+0000 7f96aaffd700 1 --2- 172.21.15.27:0/1727592219 >> v2:172.21.15.27:3300/0 conn(0x7f96ac1596c0 0x7f96ac15d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:28.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.215+0000 7f96aa7fc700 1 --2- 172.21.15.27:0/1727592219 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96ac0ca670 0x7f96ac068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:28.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.216+0000 7f96a9ffb700 1 -- 172.21.15.27:0/1727592219 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547665380 0 0) 0x7f96ac0a9cb0 con 0x7f96ac068b60 2024-01-11T09:47:28.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.216+0000 7f96a9ffb700 1 -- 172.21.15.27:0/1727592219 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9694003570 con 0x7f96ac068b60 2024-01-11T09:47:28.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.216+0000 7f96aaffd700 1 -- 172.21.15.27:0/1727592219 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96ac0ca670 msgr2=0x7f96ac068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:28.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.216+0000 7f96aaffd700 1 --2- 172.21.15.27:0/1727592219 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96ac0ca670 0x7f96ac068620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:28.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.216+0000 7f96aaffd700 1 -- 172.21.15.27:0/1727592219 >> v1:172.21.15.27:6789/0 conn(0x7f96ac068b60 legacy=0x7f96ac159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:28.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.216+0000 7f96aaffd700 1 -- 172.21.15.27:0/1727592219 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac15dc50 con 0x7f96ac1596c0 2024-01-11T09:47:28.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.216+0000 7f96aaffd700 1 --2- 172.21.15.27:0/1727592219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96ac1596c0 0x7f96ac15d300 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f969c0098e0 tx=0x7f969c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ec0c02b860f7883 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:28.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.216+0000 7f96a9ffb700 1 -- 172.21.15.27:0/1727592219 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c011630 con 0x7f96ac1596c0 2024-01-11T09:47:28.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.216+0000 7f96a9ffb700 1 -- 172.21.15.27:0/1727592219 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969c011790 con 0x7f96ac1596c0 2024-01-11T09:47:28.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.216+0000 7f96a9ffb700 1 -- 172.21.15.27:0/1727592219 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c011b30 con 0x7f96ac1596c0 2024-01-11T09:47:28.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.217+0000 7f96b15ff700 1 -- 172.21.15.27:0/1727592219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96ac1596c0 msgr2=0x7f96ac15d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:28.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.217+0000 7f96b15ff700 1 --2- 172.21.15.27:0/1727592219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96ac1596c0 0x7f96ac15d300 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f969c0098e0 tx=0x7f969c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:28.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.217+0000 7f96b15ff700 1 -- 172.21.15.27:0/1727592219 shutdown_connections 2024-01-11T09:47:28.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.217+0000 7f96b15ff700 1 --2- 172.21.15.27:0/1727592219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96ac1596c0 0x7f96ac15d300 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:28.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.217+0000 7f96b15ff700 1 --2- 172.21.15.27:0/1727592219 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96ac0ca670 0x7f96ac068620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:28.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.217+0000 7f96b15ff700 1 -- 172.21.15.27:0/1727592219 >> 172.21.15.27:0/1727592219 conn(0x7f96ac0c9f70 msgr2=0x7f96ac06deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:28.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.217+0000 7f96b15ff700 1 -- 172.21.15.27:0/1727592219 shutdown_connections 2024-01-11T09:47:28.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.217+0000 7f96b15ff700 1 -- 172.21.15.27:0/1727592219 wait complete. 2024-01-11T09:47:28.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.219+0000 7f96b15ff700 1 Processor -- start 2024-01-11T09:47:28.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.219+0000 7f96b15ff700 1 -- start start 2024-01-11T09:47:28.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.220+0000 7f96b15ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96ac068b60 0x7f96ac0b0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:28.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.220+0000 7f96b15ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96ac0ca670 0x7f96ac155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:28.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.220+0000 7f96b15ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96ac1596c0 0x7f96ac159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:28.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.220+0000 7f96b15ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac0b0300 con 0x7f96ac0ca670 2024-01-11T09:47:28.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.220+0000 7f96b15ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac0b0cd0 con 0x7f96ac1596c0 2024-01-11T09:47:28.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.220+0000 7f96b15ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac0c3200 con 0x7f96ac068b60 2024-01-11T09:47:28.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.220+0000 7f96aa7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96ac0ca670 0x7f96ac155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:28.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.220+0000 7f96aa7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96ac0ca670 0x7f96ac155950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56182/0 (socket says 172.21.15.27:56182) 2024-01-11T09:47:28.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.220+0000 7f96aa7fc700 1 -- 172.21.15.27:0/1773509608 learned_addr learned my addr 172.21.15.27:0/1773509608 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:28.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.220+0000 7f96aaffd700 1 --2- 172.21.15.27:0/1773509608 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96ac068b60 0x7f96ac0b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:28.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.220+0000 7f96ab7fe700 1 --2- 172.21.15.27:0/1773509608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96ac1596c0 0x7f96ac159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:28.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.221+0000 7f96aa7fc700 1 -- 172.21.15.27:0/1773509608 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96ac068b60 msgr2=0x7f96ac0b0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:28.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.221+0000 7f96aa7fc700 1 --2- 172.21.15.27:0/1773509608 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96ac068b60 0x7f96ac0b0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:28.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.221+0000 7f96aa7fc700 1 -- 172.21.15.27:0/1773509608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96ac1596c0 msgr2=0x7f96ac159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:28.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.221+0000 7f96aa7fc700 1 --2- 172.21.15.27:0/1773509608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96ac1596c0 0x7f96ac159a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:28.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.221+0000 7f96aa7fc700 1 -- 172.21.15.27:0/1773509608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c009420 con 0x7f96ac0ca670 2024-01-11T09:47:28.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.221+0000 7f96aa7fc700 1 --2- 172.21.15.27:0/1773509608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96ac0ca670 0x7f96ac155950 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f96a000ec50 tx=0x7f96a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:28.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.221+0000 7f9692ffd700 1 -- 172.21.15.27:0/1773509608 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a000c9d0 con 0x7f96ac0ca670 2024-01-11T09:47:28.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.221+0000 7f9692ffd700 1 -- 172.21.15.27:0/1773509608 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a000cb30 con 0x7f96ac0ca670 2024-01-11T09:47:28.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.221+0000 7f9692ffd700 1 -- 172.21.15.27:0/1773509608 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a0013470 con 0x7f96ac0ca670 2024-01-11T09:47:28.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.221+0000 7f96b15ff700 1 -- 172.21.15.27:0/1773509608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ac0af9d0 con 0x7f96ac0ca670 2024-01-11T09:47:28.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.221+0000 7f96b15ff700 1 -- 172.21.15.27:0/1773509608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ac0aff90 con 0x7f96ac0ca670 2024-01-11T09:47:28.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.222+0000 7f96b15ff700 1 -- 172.21.15.27:0/1773509608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ac0b0470 con 0x7f96ac0ca670 2024-01-11T09:47:28.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.223+0000 7f9692ffd700 1 -- 172.21.15.27:0/1773509608 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f96a0013860 con 0x7f96ac0ca670 2024-01-11T09:47:28.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.225+0000 7f9692ffd700 1 --2- 172.21.15.27:0/1773509608 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9698067170 0x7f9698069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:28.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.225+0000 7f9692ffd700 1 -- 172.21.15.27:0/1773509608 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f96a0084cc0 con 0x7f96ac0ca670 2024-01-11T09:47:28.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.225+0000 7f96aaffd700 1 --2- 172.21.15.27:0/1773509608 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9698067170 0x7f9698069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:28.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.226+0000 7f96aaffd700 1 --2- 172.21.15.27:0/1773509608 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9698067170 0x7f9698069620 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f969c0091e0 tx=0x7f969c017000 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:28.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.228+0000 7f9692ffd700 1 -- 172.21.15.27:0/1773509608 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f96a008b080 con 0x7f96ac0ca670 2024-01-11T09:47:28.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.475+0000 7f96b15ff700 1 -- 172.21.15.27:0/1773509608 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96ac15df30 con 0x7f96ac0ca670 2024-01-11T09:47:28.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.476+0000 7f9692ffd700 1 -- 172.21.15.27:0/1773509608 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+4612 (secure 0 0 0) 0x7f96a0051b90 con 0x7f96ac0ca670 2024-01-11T09:47:28.467 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:28.467 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":141,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9866,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1060058066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1060058066},{"type":"v1","addr":"172.21.15.77:6835","nonce":1060058066}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137},{"gid":11626,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/4169546841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":4169546841},{"type":"v1","addr":"172.21.15.77:6834","nonce":4169546841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137},{"gid":11632,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/2567971522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2567971522},{"type":"v1","addr":"172.21.15.27:6833","nonce":2567971522}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:47:24.474082+0000","modified":"2024-01-11T09:47:27.985712+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12063},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12063":{"gid":12063,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1065634442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1065634442},{"type":"v1","addr":"172.21.15.27:6835","nonce":1065634442}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2024-01-11T09:47:28.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.478+0000 7f96b15ff700 1 -- 172.21.15.27:0/1773509608 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9698067170 msgr2=0x7f9698069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:28.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.478+0000 7f96b15ff700 1 --2- 172.21.15.27:0/1773509608 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9698067170 0x7f9698069620 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f969c0091e0 tx=0x7f969c017000 comp rx=0 tx=0).stop 2024-01-11T09:47:28.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.478+0000 7f96b15ff700 1 -- 172.21.15.27:0/1773509608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96ac0ca670 msgr2=0x7f96ac155950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:28.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.478+0000 7f96b15ff700 1 --2- 172.21.15.27:0/1773509608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96ac0ca670 0x7f96ac155950 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f96a000ec50 tx=0x7f96a00091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:28.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.478+0000 7f96b15ff700 1 -- 172.21.15.27:0/1773509608 shutdown_connections 2024-01-11T09:47:28.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.478+0000 7f96b15ff700 1 --2- 172.21.15.27:0/1773509608 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9698067170 0x7f9698069620 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:28.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.478+0000 7f96b15ff700 1 --2- 172.21.15.27:0/1773509608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96ac1596c0 0x7f96ac159a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:28.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.478+0000 7f96b15ff700 1 --2- 172.21.15.27:0/1773509608 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f96ac068b60 0x7f96ac0b0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:28.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.478+0000 7f96b15ff700 1 --2- 172.21.15.27:0/1773509608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96ac0ca670 0x7f96ac155950 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:28.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.478+0000 7f96b15ff700 1 -- 172.21.15.27:0/1773509608 >> 172.21.15.27:0/1773509608 conn(0x7f96ac0c9f70 msgr2=0x7f96ac158690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:28.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.479+0000 7f96b15ff700 1 -- 172.21.15.27:0/1773509608 shutdown_connections 2024-01-11T09:47:28.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.479+0000 7f96b15ff700 1 -- 172.21.15.27:0/1773509608 wait complete. 2024-01-11T09:47:28.471 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 141 2024-01-11T09:47:28.488 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:47:28.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.696+0000 7f884a9ff700 1 Processor -- start 2024-01-11T09:47:28.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.697+0000 7f884a9ff700 1 -- start start 2024-01-11T09:47:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.698+0000 7f884a9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b03e0 0x7f88440b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.698+0000 7f884a9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88440b2e00 con 0x7f88440b07b0 2024-01-11T09:47:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.699+0000 7f884a9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88440cd730 con 0x7f88440b29c0 2024-01-11T09:47:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.699+0000 7f884a9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88440ccb20 con 0x7f88440b03e0 2024-01-11T09:47:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.699+0000 7f88437fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f88440b07b0 0x7f8844159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51284/0 (socket says 172.21.15.27:51284) 2024-01-11T09:47:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.699+0000 7f88437fe700 1 -- 172.21.15.27:0/3967042406 learned_addr learned my addr 172.21.15.27:0/3967042406 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.699+0000 7f8843fff700 1 --2- 172.21.15.27:0/3967042406 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b03e0 0x7f88440b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:28.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.699+0000 7f8842ffd700 1 -- 172.21.15.27:0/3967042406 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299050101 0 0) 0x7f8838001110 con 0x7f88440b07b0 2024-01-11T09:47:28.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.699+0000 7f8842ffd700 1 -- 172.21.15.27:0/3967042406 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f882c003570 con 0x7f88440b07b0 2024-01-11T09:47:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.699+0000 7f8842ffd700 1 -- 172.21.15.27:0/3967042406 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717844441 0 0) 0x7f88440cd730 con 0x7f88440b29c0 2024-01-11T09:47:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.699+0000 7f8842ffd700 1 -- 172.21.15.27:0/3967042406 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8838001110 con 0x7f88440b29c0 2024-01-11T09:47:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.699+0000 7f8843fff700 1 -- 172.21.15.27:0/3967042406 >> v1:172.21.15.77:6789/0 conn(0x7f88440b29c0 legacy=0x7f88440aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.700+0000 7f8843fff700 1 -- 172.21.15.27:0/3967042406 >> v1:172.21.15.27:6789/0 conn(0x7f88440b07b0 legacy=0x7f8844159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.700+0000 7f8843fff700 1 -- 172.21.15.27:0/3967042406 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88440cce00 con 0x7f88440b03e0 2024-01-11T09:47:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.700+0000 7f8843fff700 1 --2- 172.21.15.27:0/3967042406 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b03e0 0x7f88440b2480 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8834009910 tx=0x7f88340091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d69dcab8c87e8ac server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.700+0000 7f8842ffd700 1 -- 172.21.15.27:0/3967042406 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8834011630 con 0x7f88440b03e0 2024-01-11T09:47:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.700+0000 7f8842ffd700 1 -- 172.21.15.27:0/3967042406 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8834011790 con 0x7f88440b03e0 2024-01-11T09:47:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.700+0000 7f8842ffd700 1 -- 172.21.15.27:0/3967042406 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8834011b30 con 0x7f88440b03e0 2024-01-11T09:47:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.701+0000 7f884a9ff700 1 -- 172.21.15.27:0/3967042406 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b03e0 msgr2=0x7f88440b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.701+0000 7f884a9ff700 1 --2- 172.21.15.27:0/3967042406 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b03e0 0x7f88440b2480 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8834009910 tx=0x7f88340091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.701+0000 7f884a9ff700 1 -- 172.21.15.27:0/3967042406 shutdown_connections 2024-01-11T09:47:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.701+0000 7f884a9ff700 1 --2- 172.21.15.27:0/3967042406 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b03e0 0x7f88440b2480 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:28.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.701+0000 7f884a9ff700 1 -- 172.21.15.27:0/3967042406 >> 172.21.15.27:0/3967042406 conn(0x7f884406ed40 msgr2=0x7f884406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:28.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.701+0000 7f884a9ff700 1 -- 172.21.15.27:0/3967042406 shutdown_connections 2024-01-11T09:47:28.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.701+0000 7f884a9ff700 1 -- 172.21.15.27:0/3967042406 wait complete. 2024-01-11T09:47:28.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.703+0000 7f884a9ff700 1 Processor -- start 2024-01-11T09:47:28.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.703+0000 7f884a9ff700 1 -- start start 2024-01-11T09:47:28.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.704+0000 7f884a9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b03e0 0x7f88440a8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:28.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.704+0000 7f884a9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88440b07b0 0x7f88440a6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:28.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.704+0000 7f884a9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88440b29c0 0x7f8844169b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:28.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.704+0000 7f884a9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88440a8c90 con 0x7f88440b07b0 2024-01-11T09:47:28.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.704+0000 7f884a9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88440a7480 con 0x7f88440b29c0 2024-01-11T09:47:28.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.705+0000 7f884a9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88440aec80 con 0x7f88440b03e0 2024-01-11T09:47:28.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.705+0000 7f88437fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88440b07b0 0x7f88440a6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.705+0000 7f8843fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b03e0 0x7f88440a8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.705+0000 7f88437fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88440b07b0 0x7f88440a6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56194/0 (socket says 172.21.15.27:56194) 2024-01-11T09:47:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.705+0000 7f88437fe700 1 -- 172.21.15.27:0/2175207637 learned_addr learned my addr 172.21.15.27:0/2175207637 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.705+0000 7f8848f9c700 1 --2- 172.21.15.27:0/2175207637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88440b29c0 0x7f8844169b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.705+0000 7f88437fe700 1 -- 172.21.15.27:0/2175207637 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b03e0 msgr2=0x7f88440a8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.705+0000 7f88437fe700 1 --2- 172.21.15.27:0/2175207637 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b03e0 0x7f88440a8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.705+0000 7f88437fe700 1 -- 172.21.15.27:0/2175207637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88440b29c0 msgr2=0x7f8844169b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.705+0000 7f88437fe700 1 --2- 172.21.15.27:0/2175207637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88440b29c0 0x7f8844169b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.705+0000 7f88437fe700 1 -- 172.21.15.27:0/2175207637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8834009420 con 0x7f88440b07b0 2024-01-11T09:47:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.706+0000 7f88437fe700 1 --2- 172.21.15.27:0/2175207637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88440b07b0 0x7f88440a6f40 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f88380058d0 tx=0x7f8838007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.706+0000 7f882bfff700 1 -- 172.21.15.27:0/2175207637 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8838005b30 con 0x7f88440b07b0 2024-01-11T09:47:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.706+0000 7f882bfff700 1 -- 172.21.15.27:0/2175207637 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8838005c90 con 0x7f88440b07b0 2024-01-11T09:47:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.706+0000 7f884a9ff700 1 -- 172.21.15.27:0/2175207637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88440a82c0 con 0x7f88440b07b0 2024-01-11T09:47:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.706+0000 7f882bfff700 1 -- 172.21.15.27:0/2175207637 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8838007300 con 0x7f88440b07b0 2024-01-11T09:47:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.706+0000 7f884a9ff700 1 -- 172.21.15.27:0/2175207637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88440a7990 con 0x7f88440b07b0 2024-01-11T09:47:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.707+0000 7f884a9ff700 1 -- 172.21.15.27:0/2175207637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88440718d0 con 0x7f88440b07b0 2024-01-11T09:47:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.708+0000 7f882bfff700 1 -- 172.21.15.27:0/2175207637 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f883801b400 con 0x7f88440b07b0 2024-01-11T09:47:28.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.709+0000 7f882bfff700 1 --2- 172.21.15.27:0/2175207637 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8830067170 0x7f8830069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:28.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.709+0000 7f882bfff700 1 -- 172.21.15.27:0/2175207637 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f8838084370 con 0x7f88440b07b0 2024-01-11T09:47:28.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.709+0000 7f8843fff700 1 --2- 172.21.15.27:0/2175207637 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8830067170 0x7f8830069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:28.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.710+0000 7f8843fff700 1 --2- 172.21.15.27:0/2175207637 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8830067170 0x7f8830069620 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f88340091e0 tx=0x7f8834017000 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.713+0000 7f882bfff700 1 -- 172.21.15.27:0/2175207637 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f88380512c0 con 0x7f88440b07b0 2024-01-11T09:47:28.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.916+0000 7f884a9ff700 1 -- 172.21.15.27:0/2175207637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88440b2e00 con 0x7f88440b07b0 2024-01-11T09:47:28.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.917+0000 7f882bfff700 1 -- 172.21.15.27:0/2175207637 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v256) v1 ==== 74+0+15495 (secure 0 0 0) 0x7f88440b2e00 con 0x7f88440b07b0 2024-01-11T09:47:28.909 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:28.909 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":256,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:47:25.972775+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:47:22.374454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-01-11T09:47:23.471379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:47:28.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.920+0000 7f884a9ff700 1 -- 172.21.15.27:0/2175207637 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8830067170 msgr2=0x7f8830069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:28.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.920+0000 7f884a9ff700 1 --2- 172.21.15.27:0/2175207637 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8830067170 0x7f8830069620 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f88340091e0 tx=0x7f8834017000 comp rx=0 tx=0).stop 2024-01-11T09:47:28.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.920+0000 7f884a9ff700 1 -- 172.21.15.27:0/2175207637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88440b07b0 msgr2=0x7f88440a6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:28.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.920+0000 7f884a9ff700 1 --2- 172.21.15.27:0/2175207637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88440b07b0 0x7f88440a6f40 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f88380058d0 tx=0x7f8838007e80 comp rx=0 tx=0).stop 2024-01-11T09:47:28.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.920+0000 7f884a9ff700 1 -- 172.21.15.27:0/2175207637 shutdown_connections 2024-01-11T09:47:28.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.920+0000 7f884a9ff700 1 --2- 172.21.15.27:0/2175207637 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8830067170 0x7f8830069620 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:28.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.920+0000 7f884a9ff700 1 --2- 172.21.15.27:0/2175207637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88440b29c0 0x7f8844169b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:28.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.920+0000 7f884a9ff700 1 --2- 172.21.15.27:0/2175207637 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f88440b03e0 0x7f88440a8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:28.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.920+0000 7f884a9ff700 1 --2- 172.21.15.27:0/2175207637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88440b07b0 0x7f88440a6f40 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:28.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.920+0000 7f884a9ff700 1 -- 172.21.15.27:0/2175207637 >> 172.21.15.27:0/2175207637 conn(0x7f884406ed40 msgr2=0x7f88440ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:28.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.920+0000 7f884a9ff700 1 -- 172.21.15.27:0/2175207637 shutdown_connections 2024-01-11T09:47:28.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:28.920+0000 7f884a9ff700 1 -- 172.21.15.27:0/2175207637 wait complete. 2024-01-11T09:47:28.930 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-11T09:47:28.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:47:29.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.140+0000 7fd887dff700 1 Processor -- start 2024-01-11T09:47:29.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.140+0000 7fd887dff700 1 -- start start 2024-01-11T09:47:29.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.141+0000 7fd887dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd8801575e0 0x7fd880158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:29.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.141+0000 7fd887dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd88006b610 0x7fd8800b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:29.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.141+0000 7fd887dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd880158b60 con 0x7fd8800b1800 2024-01-11T09:47:29.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.141+0000 7fd887dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8800cb5e0 con 0x7fd88006b610 2024-01-11T09:47:29.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.141+0000 7fd887dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8800b1c80 con 0x7fd8801575e0 2024-01-11T09:47:29.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.142+0000 7fd885b9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd8801575e0 0x7fd880158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:29.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.142+0000 7fd88539a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd88006b610 0x7fd8800b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:29.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.142+0000 7fd88639c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd8800b1800 0x7fd8800ca1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:51398/0 (socket says 172.21.15.27:51398) 2024-01-11T09:47:29.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.142+0000 7fd88639c700 1 -- 172.21.15.27:0/1446199931 learned_addr learned my addr 172.21.15.27:0/1446199931 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:29.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.143+0000 7fd884b99700 1 -- 172.21.15.27:0/1446199931 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634573077 0 0) 0x7fd878001490 con 0x7fd8800b1800 2024-01-11T09:47:29.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.143+0000 7fd884b99700 1 -- 172.21.15.27:0/1446199931 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd86c003570 con 0x7fd8800b1800 2024-01-11T09:47:29.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.143+0000 7fd88539a700 1 -- 172.21.15.27:0/1446199931 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd8801575e0 msgr2=0x7fd880158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:29.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.143+0000 7fd88539a700 1 --2- 172.21.15.27:0/1446199931 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd8801575e0 0x7fd880158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.143+0000 7fd88539a700 1 -- 172.21.15.27:0/1446199931 >> v1:172.21.15.77:6789/0 conn(0x7fd8800b1800 legacy=0x7fd8800ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:29.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.143+0000 7fd88539a700 1 -- 172.21.15.27:0/1446199931 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8801599b0 con 0x7fd88006b610 2024-01-11T09:47:29.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.143+0000 7fd885b9b700 1 --2- 172.21.15.27:0/1446199931 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd8801575e0 0x7fd880158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:29.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.143+0000 7fd88539a700 1 --2- 172.21.15.27:0/1446199931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd88006b610 0x7fd8800b12c0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fd8700098e0 tx=0x7fd8700091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcb1632c510a45b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:29.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.143+0000 7fd884b99700 1 -- 172.21.15.27:0/1446199931 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd870011630 con 0x7fd88006b610 2024-01-11T09:47:29.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.143+0000 7fd884b99700 1 -- 172.21.15.27:0/1446199931 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd870011790 con 0x7fd88006b610 2024-01-11T09:47:29.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.143+0000 7fd884b99700 1 -- 172.21.15.27:0/1446199931 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd870011b30 con 0x7fd88006b610 2024-01-11T09:47:29.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.144+0000 7fd887dff700 1 -- 172.21.15.27:0/1446199931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd88006b610 msgr2=0x7fd8800b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:29.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.144+0000 7fd887dff700 1 --2- 172.21.15.27:0/1446199931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd88006b610 0x7fd8800b12c0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fd8700098e0 tx=0x7fd8700091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.144+0000 7fd887dff700 1 -- 172.21.15.27:0/1446199931 shutdown_connections 2024-01-11T09:47:29.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.144+0000 7fd887dff700 1 --2- 172.21.15.27:0/1446199931 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd8801575e0 0x7fd880158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.144+0000 7fd887dff700 1 --2- 172.21.15.27:0/1446199931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd88006b610 0x7fd8800b12c0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.144+0000 7fd887dff700 1 -- 172.21.15.27:0/1446199931 >> 172.21.15.27:0/1446199931 conn(0x7fd8800680e0 msgr2=0x7fd880069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:29.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.144+0000 7fd887dff700 1 -- 172.21.15.27:0/1446199931 shutdown_connections 2024-01-11T09:47:29.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.144+0000 7fd887dff700 1 -- 172.21.15.27:0/1446199931 wait complete. 2024-01-11T09:47:29.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.146+0000 7fd887dff700 1 Processor -- start 2024-01-11T09:47:29.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.146+0000 7fd887dff700 1 -- start start 2024-01-11T09:47:29.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.148+0000 7fd887dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd88006b610 0x7fd880144c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:29.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.148+0000 7fd887dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8800b1800 0x7fd8800b0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:29.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.148+0000 7fd887dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8801575e0 0x7fd880154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:29.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.148+0000 7fd887dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8801524a0 con 0x7fd8801575e0 2024-01-11T09:47:29.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.148+0000 7fd887dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8800af490 con 0x7fd8800b1800 2024-01-11T09:47:29.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.148+0000 7fd887dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8800af770 con 0x7fd88006b610 2024-01-11T09:47:29.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.148+0000 7fd88639c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8801575e0 0x7fd880154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:29.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.148+0000 7fd88639c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8801575e0 0x7fd880154680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56196/0 (socket says 172.21.15.27:56196) 2024-01-11T09:47:29.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.148+0000 7fd88639c700 1 -- 172.21.15.27:0/3194618139 learned_addr learned my addr 172.21.15.27:0/3194618139 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:29.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.148+0000 7fd885b9b700 1 --2- 172.21.15.27:0/3194618139 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd88006b610 0x7fd880144c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:29.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.148+0000 7fd88539a700 1 --2- 172.21.15.27:0/3194618139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8800b1800 0x7fd8800b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:29.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.149+0000 7fd88639c700 1 -- 172.21.15.27:0/3194618139 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd88006b610 msgr2=0x7fd880144c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:29.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.149+0000 7fd88639c700 1 --2- 172.21.15.27:0/3194618139 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd88006b610 0x7fd880144c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.149+0000 7fd88639c700 1 -- 172.21.15.27:0/3194618139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8800b1800 msgr2=0x7fd8800b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:29.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.149+0000 7fd88639c700 1 --2- 172.21.15.27:0/3194618139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8800b1800 0x7fd8800b0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.149+0000 7fd88639c700 1 -- 172.21.15.27:0/3194618139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87c009420 con 0x7fd8801575e0 2024-01-11T09:47:29.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.149+0000 7fd88539a700 1 --2- 172.21.15.27:0/3194618139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8800b1800 0x7fd8800b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:29.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.149+0000 7fd88639c700 1 --2- 172.21.15.27:0/3194618139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8801575e0 0x7fd880154680 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fd87800cd00 tx=0x7fd878001ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:29.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.149+0000 7fd875ffb700 1 -- 172.21.15.27:0/3194618139 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87800ce00 con 0x7fd8801575e0 2024-01-11T09:47:29.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.149+0000 7fd875ffb700 1 -- 172.21.15.27:0/3194618139 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87800ab70 con 0x7fd8801575e0 2024-01-11T09:47:29.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.149+0000 7fd875ffb700 1 -- 172.21.15.27:0/3194618139 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8780070f0 con 0x7fd8801575e0 2024-01-11T09:47:29.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.149+0000 7fd887dff700 1 -- 172.21.15.27:0/3194618139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd870009420 con 0x7fd8801575e0 2024-01-11T09:47:29.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.149+0000 7fd887dff700 1 -- 172.21.15.27:0/3194618139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8800afd30 con 0x7fd8801575e0 2024-01-11T09:47:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.150+0000 7fd887dff700 1 -- 172.21.15.27:0/3194618139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8800718d0 con 0x7fd8801575e0 2024-01-11T09:47:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.151+0000 7fd875ffb700 1 -- 172.21.15.27:0/3194618139 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fd8780074e0 con 0x7fd8801575e0 2024-01-11T09:47:29.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.152+0000 7fd875ffb700 1 --2- 172.21.15.27:0/3194618139 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd864067170 0x7fd864069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:29.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.152+0000 7fd875ffb700 1 -- 172.21.15.27:0/3194618139 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 5817+0+0 (secure 0 0 0) 0x7fd878084c00 con 0x7fd8801575e0 2024-01-11T09:47:29.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.153+0000 7fd885b9b700 1 --2- 172.21.15.27:0/3194618139 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd864067170 0x7fd864069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:29.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.154+0000 7fd885b9b700 1 --2- 172.21.15.27:0/3194618139 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd864067170 0x7fd864069620 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009ea0 tx=0x7fd87c0091a0 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:29.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.157+0000 7fd875ffb700 1 -- 172.21.15.27:0/3194618139 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd87808b080 con 0x7fd8801575e0 2024-01-11T09:47:29.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.353+0000 7fd887dff700 1 -- 172.21.15.27:0/3194618139 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd880158b60 con 0x7fd864067170 2024-01-11T09:47:29.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.365+0000 7fd875ffb700 1 -- 172.21.15.27:0/3194618139 <== mgr.9641 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289550 (secure 0 0 0) 0x7fd880158b60 con 0x7fd864067170 2024-01-11T09:47:29.360 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:29.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.368+0000 7fd887dff700 1 -- 172.21.15.27:0/3194618139 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd864067170 msgr2=0x7fd864069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:29.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.368+0000 7fd887dff700 1 --2- 172.21.15.27:0/3194618139 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd864067170 0x7fd864069620 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009ea0 tx=0x7fd87c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.368+0000 7fd887dff700 1 -- 172.21.15.27:0/3194618139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8801575e0 msgr2=0x7fd880154680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:29.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.368+0000 7fd887dff700 1 --2- 172.21.15.27:0/3194618139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8801575e0 0x7fd880154680 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fd87800cd00 tx=0x7fd878001ad0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.369+0000 7fd887dff700 1 -- 172.21.15.27:0/3194618139 shutdown_connections 2024-01-11T09:47:29.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.369+0000 7fd887dff700 1 --2- 172.21.15.27:0/3194618139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8801575e0 0x7fd880154680 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.369+0000 7fd887dff700 1 --2- 172.21.15.27:0/3194618139 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd88006b610 0x7fd880144c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.369+0000 7fd887dff700 1 --2- 172.21.15.27:0/3194618139 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd864067170 0x7fd864069620 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.369+0000 7fd887dff700 1 --2- 172.21.15.27:0/3194618139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8800b1800 0x7fd8800b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.369+0000 7fd887dff700 1 -- 172.21.15.27:0/3194618139 >> 172.21.15.27:0/3194618139 conn(0x7fd8800680e0 msgr2=0x7fd880155b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:29.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.369+0000 7fd887dff700 1 -- 172.21.15.27:0/3194618139 shutdown_connections 2024-01-11T09:47:29.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.369+0000 7fd887dff700 1 -- 172.21.15.27:0/3194618139 wait complete. 2024-01-11T09:47:29.364 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:47:29.378 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-01-11T09:47:28.523570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2654,"num_read_kb":5617,"num_write":731,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":446,"ondisk_log_size":446,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2466248,"kb_used_data":26504,"kb_used_omap":0,"kb_used_meta":2439680,"kb_avail":752508472,"statfs":{"total":773094113280,"available":770568675328,"internally_reserved":0,"allocated":27140096,"data_stored":13732364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2498232320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.565826"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.008397+0000","last_change":"2024-01-11T09:47:24.997406+0000","last_active":"2024-01-11T09:47:25.008397+0000","last_peered":"2024-01-11T09:47:25.008397+0000","last_clean":"2024-01-11T09:47:25.008397+0000","last_became_active":"2024-01-11T09:47:24.997138+0000","last_became_peered":"2024-01-11T09:47:24.997138+0000","last_unstale":"2024-01-11T09:47:25.008397+0000","last_undegraded":"2024-01-11T09:47:25.008397+0000","last_fullsized":"2024-01-11T09:47:25.008397+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:39:31.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.981933+0000","last_change":"2024-01-11T09:47:24.975616+0000","last_active":"2024-01-11T09:47:24.981933+0000","last_peered":"2024-01-11T09:47:24.981933+0000","last_clean":"2024-01-11T09:47:24.981933+0000","last_became_active":"2024-01-11T09:47:24.975334+0000","last_became_peered":"2024-01-11T09:47:24.975334+0000","last_unstale":"2024-01-11T09:47:24.981933+0000","last_undegraded":"2024-01-11T09:47:24.981933+0000","last_fullsized":"2024-01-11T09:47:24.981933+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:42:00.528441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004226+0000","last_change":"2024-01-11T09:47:24.999455+0000","last_active":"2024-01-11T09:47:25.004226+0000","last_peered":"2024-01-11T09:47:25.004226+0000","last_clean":"2024-01-11T09:47:25.004226+0000","last_became_active":"2024-01-11T09:47:24.999172+0000","last_became_peered":"2024-01-11T09:47:24.999172+0000","last_unstale":"2024-01-11T09:47:25.004226+0000","last_undegraded":"2024-01-11T09:47:25.004226+0000","last_fullsized":"2024-01-11T09:47:25.004226+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:39:08.765597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.000678+0000","last_change":"2024-01-11T09:47:24.991535+0000","last_active":"2024-01-11T09:47:25.000678+0000","last_peered":"2024-01-11T09:47:25.000678+0000","last_clean":"2024-01-11T09:47:25.000678+0000","last_became_active":"2024-01-11T09:47:24.991241+0000","last_became_peered":"2024-01-11T09:47:24.991241+0000","last_unstale":"2024-01-11T09:47:25.000678+0000","last_undegraded":"2024-01-11T09:47:25.000678+0000","last_fullsized":"2024-01-11T09:47:25.000678+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:10:02.432500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.981901+0000","last_change":"2024-01-11T09:47:24.977175+0000","last_active":"2024-01-11T09:47:24.981901+0000","last_peered":"2024-01-11T09:47:24.981901+0000","last_clean":"2024-01-11T09:47:24.981901+0000","last_became_active":"2024-01-11T09:47:24.976914+0000","last_became_peered":"2024-01-11T09:47:24.976914+0000","last_unstale":"2024-01-11T09:47:24.981901+0000","last_undegraded":"2024-01-11T09:47:24.981901+0000","last_fullsized":"2024-01-11T09:47:24.981901+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:27:48.271801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.981846+0000","last_change":"2024-01-11T09:47:24.976207+0000","last_active":"2024-01-11T09:47:24.981846+0000","last_peered":"2024-01-11T09:47:24.981846+0000","last_clean":"2024-01-11T09:47:24.981846+0000","last_became_active":"2024-01-11T09:47:24.975903+0000","last_became_peered":"2024-01-11T09:47:24.975903+0000","last_unstale":"2024-01-11T09:47:24.981846+0000","last_undegraded":"2024-01-11T09:47:24.981846+0000","last_fullsized":"2024-01-11T09:47:24.981846+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:17:43.512711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.996431+0000","last_change":"2024-01-11T09:47:24.993021+0000","last_active":"2024-01-11T09:47:24.996431+0000","last_peered":"2024-01-11T09:47:24.996431+0000","last_clean":"2024-01-11T09:47:24.996431+0000","last_became_active":"2024-01-11T09:47:24.992704+0000","last_became_peered":"2024-01-11T09:47:24.992704+0000","last_unstale":"2024-01-11T09:47:24.996431+0000","last_undegraded":"2024-01-11T09:47:24.996431+0000","last_fullsized":"2024-01-11T09:47:24.996431+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:16:26.902792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002366+0000","last_change":"2024-01-11T09:47:24.998877+0000","last_active":"2024-01-11T09:47:25.002366+0000","last_peered":"2024-01-11T09:47:25.002366+0000","last_clean":"2024-01-11T09:47:25.002366+0000","last_became_active":"2024-01-11T09:47:24.998622+0000","last_became_peered":"2024-01-11T09:47:24.998622+0000","last_unstale":"2024-01-11T09:47:25.002366+0000","last_undegraded":"2024-01-11T09:47:25.002366+0000","last_fullsized":"2024-01-11T09:47:25.002366+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:22:49.439958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.001924+0000","last_change":"2024-01-11T09:47:24.994645+0000","last_active":"2024-01-11T09:47:25.001924+0000","last_peered":"2024-01-11T09:47:25.001924+0000","last_clean":"2024-01-11T09:47:25.001924+0000","last_became_active":"2024-01-11T09:47:24.994365+0000","last_became_peered":"2024-01-11T09:47:24.994365+0000","last_unstale":"2024-01-11T09:47:25.001924+0000","last_undegraded":"2024-01-11T09:47:25.001924+0000","last_fullsized":"2024-01-11T09:47:25.001924+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:33:56.394147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002854+0000","last_change":"2024-01-11T09:47:24.996859+0000","last_active":"2024-01-11T09:47:25.002854+0000","last_peered":"2024-01-11T09:47:25.002854+0000","last_clean":"2024-01-11T09:47:25.002854+0000","last_became_active":"2024-01-11T09:47:24.996616+0000","last_became_peered":"2024-01-11T09:47:24.996616+0000","last_unstale":"2024-01-11T09:47:25.002854+0000","last_undegraded":"2024-01-11T09:47:25.002854+0000","last_fullsized":"2024-01-11T09:47:25.002854+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:02:36.455725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.007468+0000","last_change":"2024-01-11T09:47:25.003959+0000","last_active":"2024-01-11T09:47:25.007468+0000","last_peered":"2024-01-11T09:47:25.007468+0000","last_clean":"2024-01-11T09:47:25.007468+0000","last_became_active":"2024-01-11T09:47:25.003688+0000","last_became_peered":"2024-01-11T09:47:25.003688+0000","last_unstale":"2024-01-11T09:47:25.007468+0000","last_undegraded":"2024-01-11T09:47:25.007468+0000","last_fullsized":"2024-01-11T09:47:25.007468+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:49:32.683244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.001960+0000","last_change":"2024-01-11T09:47:24.996977+0000","last_active":"2024-01-11T09:47:25.001960+0000","last_peered":"2024-01-11T09:47:25.001960+0000","last_clean":"2024-01-11T09:47:25.001960+0000","last_became_active":"2024-01-11T09:47:24.995847+0000","last_became_peered":"2024-01-11T09:47:24.995847+0000","last_unstale":"2024-01-11T09:47:25.001960+0000","last_undegraded":"2024-01-11T09:47:25.001960+0000","last_fullsized":"2024-01-11T09:47:25.001960+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:16:51.696701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.009302+0000","last_change":"2024-01-11T09:47:24.999731+0000","last_active":"2024-01-11T09:47:25.009302+0000","last_peered":"2024-01-11T09:47:25.009302+0000","last_clean":"2024-01-11T09:47:25.009302+0000","last_became_active":"2024-01-11T09:47:24.999446+0000","last_became_peered":"2024-01-11T09:47:24.999446+0000","last_unstale":"2024-01-11T09:47:25.009302+0000","last_undegraded":"2024-01-11T09:47:25.009302+0000","last_fullsized":"2024-01-11T09:47:25.009302+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:39:30.172237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004026+0000","last_change":"2024-01-11T09:47:24.993445+0000","last_active":"2024-01-11T09:47:25.004026+0000","last_peered":"2024-01-11T09:47:25.004026+0000","last_clean":"2024-01-11T09:47:25.004026+0000","last_became_active":"2024-01-11T09:47:24.993127+0000","last_became_peered":"2024-01-11T09:47:24.993127+0000","last_unstale":"2024-01-11T09:47:25.004026+0000","last_undegraded":"2024-01-11T09:47:25.004026+0000","last_fullsized":"2024-01-11T09:47:25.004026+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:11:13.244226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.996408+0000","last_change":"2024-01-11T09:47:24.991126+0000","last_active":"2024-01-11T09:47:24.996408+0000","last_peered":"2024-01-11T09:47:24.996408+0000","last_clean":"2024-01-11T09:47:24.996408+0000","last_became_active":"2024-01-11T09:47:24.990346+0000","last_became_peered":"2024-01-11T09:47:24.990346+0000","last_unstale":"2024-01-11T09:47:24.996408+0000","last_undegraded":"2024-01-11T09:47:24.996408+0000","last_fullsized":"2024-01-11T09:47:24.996408+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:03:35.424742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.999691+0000","last_change":"2024-01-11T09:47:24.995534+0000","last_active":"2024-01-11T09:47:24.999691+0000","last_peered":"2024-01-11T09:47:24.999691+0000","last_clean":"2024-01-11T09:47:24.999691+0000","last_became_active":"2024-01-11T09:47:24.995260+0000","last_became_peered":"2024-01-11T09:47:24.995260+0000","last_unstale":"2024-01-11T09:47:24.999691+0000","last_undegraded":"2024-01-11T09:47:24.999691+0000","last_fullsized":"2024-01-11T09:47:24.999691+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:26:00.307598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.999466+0000","last_change":"2024-01-11T09:47:24.989635+0000","last_active":"2024-01-11T09:47:24.999466+0000","last_peered":"2024-01-11T09:47:24.999466+0000","last_clean":"2024-01-11T09:47:24.999466+0000","last_became_active":"2024-01-11T09:47:24.989278+0000","last_became_peered":"2024-01-11T09:47:24.989278+0000","last_unstale":"2024-01-11T09:47:24.999466+0000","last_undegraded":"2024-01-11T09:47:24.999466+0000","last_fullsized":"2024-01-11T09:47:24.999466+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:36.344385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002152+0000","last_change":"2024-01-11T09:47:24.995164+0000","last_active":"2024-01-11T09:47:25.002152+0000","last_peered":"2024-01-11T09:47:25.002152+0000","last_clean":"2024-01-11T09:47:25.002152+0000","last_became_active":"2024-01-11T09:47:24.994859+0000","last_became_peered":"2024-01-11T09:47:24.994859+0000","last_unstale":"2024-01-11T09:47:25.002152+0000","last_undegraded":"2024-01-11T09:47:25.002152+0000","last_fullsized":"2024-01-11T09:47:25.002152+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:40:26.192216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.007663+0000","last_change":"2024-01-11T09:47:25.003579+0000","last_active":"2024-01-11T09:47:25.007663+0000","last_peered":"2024-01-11T09:47:25.007663+0000","last_clean":"2024-01-11T09:47:25.007663+0000","last_became_active":"2024-01-11T09:47:25.002974+0000","last_became_peered":"2024-01-11T09:47:25.002974+0000","last_unstale":"2024-01-11T09:47:25.007663+0000","last_undegraded":"2024-01-11T09:47:25.007663+0000","last_fullsized":"2024-01-11T09:47:25.007663+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:20:42.501759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002743+0000","last_change":"2024-01-11T09:47:24.999637+0000","last_active":"2024-01-11T09:47:25.002743+0000","last_peered":"2024-01-11T09:47:25.002743+0000","last_clean":"2024-01-11T09:47:25.002743+0000","last_became_active":"2024-01-11T09:47:24.999343+0000","last_became_peered":"2024-01-11T09:47:24.999343+0000","last_unstale":"2024-01-11T09:47:25.002743+0000","last_undegraded":"2024-01-11T09:47:25.002743+0000","last_fullsized":"2024-01-11T09:47:25.002743+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:56:49.891176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.009324+0000","last_change":"2024-01-11T09:47:25.000182+0000","last_active":"2024-01-11T09:47:25.009324+0000","last_peered":"2024-01-11T09:47:25.009324+0000","last_clean":"2024-01-11T09:47:25.009324+0000","last_became_active":"2024-01-11T09:47:24.999929+0000","last_became_peered":"2024-01-11T09:47:24.999929+0000","last_unstale":"2024-01-11T09:47:25.009324+0000","last_undegraded":"2024-01-11T09:47:25.009324+0000","last_fullsized":"2024-01-11T09:47:25.009324+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:00:22.287610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.994002+0000","last_change":"2024-01-11T09:47:24.989120+0000","last_active":"2024-01-11T09:47:24.994002+0000","last_peered":"2024-01-11T09:47:24.994002+0000","last_clean":"2024-01-11T09:47:24.994002+0000","last_became_active":"2024-01-11T09:47:24.988849+0000","last_became_peered":"2024-01-11T09:47:24.988849+0000","last_unstale":"2024-01-11T09:47:24.994002+0000","last_undegraded":"2024-01-11T09:47:24.994002+0000","last_fullsized":"2024-01-11T09:47:24.994002+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:21:35.094425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.993982+0000","last_change":"2024-01-11T09:47:24.990469+0000","last_active":"2024-01-11T09:47:24.993982+0000","last_peered":"2024-01-11T09:47:24.993982+0000","last_clean":"2024-01-11T09:47:24.993982+0000","last_became_active":"2024-01-11T09:47:24.990025+0000","last_became_peered":"2024-01-11T09:47:24.990025+0000","last_unstale":"2024-01-11T09:47:24.993982+0000","last_undegraded":"2024-01-11T09:47:24.993982+0000","last_fullsized":"2024-01-11T09:47:24.993982+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:01:36.213583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.000890+0000","last_change":"2024-01-11T09:47:24.992429+0000","last_active":"2024-01-11T09:47:25.000890+0000","last_peered":"2024-01-11T09:47:25.000890+0000","last_clean":"2024-01-11T09:47:25.000890+0000","last_became_active":"2024-01-11T09:47:24.992117+0000","last_became_peered":"2024-01-11T09:47:24.992117+0000","last_unstale":"2024-01-11T09:47:25.000890+0000","last_undegraded":"2024-01-11T09:47:25.000890+0000","last_fullsized":"2024-01-11T09:47:25.000890+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:46:52.592447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.993814+0000","last_change":"2024-01-11T09:47:24.988252+0000","last_active":"2024-01-11T09:47:24.993814+0000","last_peered":"2024-01-11T09:47:24.993814+0000","last_clean":"2024-01-11T09:47:24.993814+0000","last_became_active":"2024-01-11T09:47:24.987962+0000","last_became_peered":"2024-01-11T09:47:24.987962+0000","last_unstale":"2024-01-11T09:47:24.993814+0000","last_undegraded":"2024-01-11T09:47:24.993814+0000","last_fullsized":"2024-01-11T09:47:24.993814+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:11:13.244226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.001890+0000","last_change":"2024-01-11T09:47:24.993862+0000","last_active":"2024-01-11T09:47:25.001890+0000","last_peered":"2024-01-11T09:47:25.001890+0000","last_clean":"2024-01-11T09:47:25.001890+0000","last_became_active":"2024-01-11T09:47:24.992925+0000","last_became_peered":"2024-01-11T09:47:24.992925+0000","last_unstale":"2024-01-11T09:47:25.001890+0000","last_undegraded":"2024-01-11T09:47:25.001890+0000","last_fullsized":"2024-01-11T09:47:25.001890+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:45:09.788829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.990490+0000","last_change":"2024-01-11T09:47:24.989603+0000","last_active":"2024-01-11T09:47:24.990490+0000","last_peered":"2024-01-11T09:47:24.990490+0000","last_clean":"2024-01-11T09:47:24.990490+0000","last_became_active":"2024-01-11T09:47:24.988816+0000","last_became_peered":"2024-01-11T09:47:24.988816+0000","last_unstale":"2024-01-11T09:47:24.990490+0000","last_undegraded":"2024-01-11T09:47:24.990490+0000","last_fullsized":"2024-01-11T09:47:24.990490+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:46:52.592447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.988647+0000","last_change":"2024-01-11T09:47:24.983913+0000","last_active":"2024-01-11T09:47:24.988647+0000","last_peered":"2024-01-11T09:47:24.988647+0000","last_clean":"2024-01-11T09:47:24.988647+0000","last_became_active":"2024-01-11T09:47:24.983628+0000","last_became_peered":"2024-01-11T09:47:24.983628+0000","last_unstale":"2024-01-11T09:47:24.988647+0000","last_undegraded":"2024-01-11T09:47:24.988647+0000","last_fullsized":"2024-01-11T09:47:24.988647+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:05:16.612544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.990474+0000","last_change":"2024-01-11T09:47:24.989376+0000","last_active":"2024-01-11T09:47:24.990474+0000","last_peered":"2024-01-11T09:47:24.990474+0000","last_clean":"2024-01-11T09:47:24.990474+0000","last_became_active":"2024-01-11T09:47:24.989054+0000","last_became_peered":"2024-01-11T09:47:24.989054+0000","last_unstale":"2024-01-11T09:47:24.990474+0000","last_undegraded":"2024-01-11T09:47:24.990474+0000","last_fullsized":"2024-01-11T09:47:24.990474+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:05:40.698838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.991214+0000","last_change":"2024-01-11T09:47:24.988736+0000","last_active":"2024-01-11T09:47:24.991214+0000","last_peered":"2024-01-11T09:47:24.991214+0000","last_clean":"2024-01-11T09:47:24.991214+0000","last_became_active":"2024-01-11T09:47:24.988477+0000","last_became_peered":"2024-01-11T09:47:24.988477+0000","last_unstale":"2024-01-11T09:47:24.991214+0000","last_undegraded":"2024-01-11T09:47:24.991214+0000","last_fullsized":"2024-01-11T09:47:24.991214+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:36:36.349947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.997906+0000","last_change":"2024-01-11T09:47:24.989526+0000","last_active":"2024-01-11T09:47:24.997906+0000","last_peered":"2024-01-11T09:47:24.997906+0000","last_clean":"2024-01-11T09:47:24.997906+0000","last_became_active":"2024-01-11T09:47:24.989259+0000","last_became_peered":"2024-01-11T09:47:24.989259+0000","last_unstale":"2024-01-11T09:47:24.997906+0000","last_undegraded":"2024-01-11T09:47:24.997906+0000","last_fullsized":"2024-01-11T09:47:24.997906+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:36:28.970823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.997807+0000","last_change":"2024-01-11T09:47:24.987338+0000","last_active":"2024-01-11T09:47:24.997807+0000","last_peered":"2024-01-11T09:47:24.997807+0000","last_clean":"2024-01-11T09:47:24.997807+0000","last_became_active":"2024-01-11T09:47:24.987071+0000","last_became_peered":"2024-01-11T09:47:24.987071+0000","last_unstale":"2024-01-11T09:47:24.997807+0000","last_undegraded":"2024-01-11T09:47:24.997807+0000","last_fullsized":"2024-01-11T09:47:24.997807+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:53:51.475245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1f","version":"255'4","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.008715+0000","last_change":"2024-01-11T09:47:23.989599+0000","last_active":"2024-01-11T09:47:25.008715+0000","last_peered":"2024-01-11T09:47:25.008715+0000","last_clean":"2024-01-11T09:47:25.008715+0000","last_became_active":"2024-01-11T09:47:23.989185+0000","last_became_peered":"2024-01-11T09:47:23.989185+0000","last_unstale":"2024-01-11T09:47:25.008715+0000","last_undegraded":"2024-01-11T09:47:25.008715+0000","last_fullsized":"2024-01-11T09:47:25.008715+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:08:15.232479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004946+0000","last_change":"2024-01-11T09:47:25.002124+0000","last_active":"2024-01-11T09:47:25.004946+0000","last_peered":"2024-01-11T09:47:25.004946+0000","last_clean":"2024-01-11T09:47:25.004946+0000","last_became_active":"2024-01-11T09:47:25.001876+0000","last_became_peered":"2024-01-11T09:47:25.001876+0000","last_unstale":"2024-01-11T09:47:25.004946+0000","last_undegraded":"2024-01-11T09:47:25.004946+0000","last_fullsized":"2024-01-11T09:47:25.004946+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:33:54.313596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1e","version":"255'1","reported_seq":11,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004936+0000","last_change":"2024-01-11T09:47:23.983332+0000","last_active":"2024-01-11T09:47:25.004936+0000","last_peered":"2024-01-11T09:47:25.004936+0000","last_clean":"2024-01-11T09:47:25.004936+0000","last_became_active":"2024-01-11T09:47:23.983024+0000","last_became_peered":"2024-01-11T09:47:23.983024+0000","last_unstale":"2024-01-11T09:47:25.004936+0000","last_undegraded":"2024-01-11T09:47:25.004936+0000","last_fullsized":"2024-01-11T09:47:25.004936+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:50:00.125829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.992894+0000","last_change":"2024-01-11T09:47:24.987265+0000","last_active":"2024-01-11T09:47:24.992894+0000","last_peered":"2024-01-11T09:47:24.992894+0000","last_clean":"2024-01-11T09:47:24.992894+0000","last_became_active":"2024-01-11T09:47:24.983227+0000","last_became_peered":"2024-01-11T09:47:24.983227+0000","last_unstale":"2024-01-11T09:47:24.992894+0000","last_undegraded":"2024-01-11T09:47:24.992894+0000","last_fullsized":"2024-01-11T09:47:24.992894+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:45:09.788829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1d","version":"255'2","reported_seq":12,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.005801+0000","last_change":"2024-01-11T09:47:23.980815+0000","last_active":"2024-01-11T09:47:25.005801+0000","last_peered":"2024-01-11T09:47:25.005801+0000","last_clean":"2024-01-11T09:47:25.005801+0000","last_became_active":"2024-01-11T09:47:23.977600+0000","last_became_peered":"2024-01-11T09:47:23.977600+0000","last_unstale":"2024-01-11T09:47:25.005801+0000","last_undegraded":"2024-01-11T09:47:25.005801+0000","last_fullsized":"2024-01-11T09:47:25.005801+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:48:33.759315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002362+0000","last_change":"2024-01-11T09:47:24.998029+0000","last_active":"2024-01-11T09:47:25.002362+0000","last_peered":"2024-01-11T09:47:25.002362+0000","last_clean":"2024-01-11T09:47:25.002362+0000","last_became_active":"2024-01-11T09:47:24.995913+0000","last_became_peered":"2024-01-11T09:47:24.995913+0000","last_unstale":"2024-01-11T09:47:25.002362+0000","last_undegraded":"2024-01-11T09:47:25.002362+0000","last_fullsized":"2024-01-11T09:47:25.002362+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:39:31.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"255'2","reported_seq":12,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.005331+0000","last_change":"2024-01-11T09:47:23.987256+0000","last_active":"2024-01-11T09:47:25.005331+0000","last_peered":"2024-01-11T09:47:25.005331+0000","last_clean":"2024-01-11T09:47:25.005331+0000","last_became_active":"2024-01-11T09:47:23.984657+0000","last_became_peered":"2024-01-11T09:47:23.984657+0000","last_unstale":"2024-01-11T09:47:25.005331+0000","last_undegraded":"2024-01-11T09:47:25.005331+0000","last_fullsized":"2024-01-11T09:47:25.005331+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:20:42.021716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.991859+0000","last_change":"2024-01-11T09:47:24.985214+0000","last_active":"2024-01-11T09:47:24.991859+0000","last_peered":"2024-01-11T09:47:24.991859+0000","last_clean":"2024-01-11T09:47:24.991859+0000","last_became_active":"2024-01-11T09:47:24.983021+0000","last_became_peered":"2024-01-11T09:47:24.983021+0000","last_unstale":"2024-01-11T09:47:24.991859+0000","last_undegraded":"2024-01-11T09:47:24.991859+0000","last_fullsized":"2024-01-11T09:47:24.991859+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:40.944331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1b","version":"255'1","reported_seq":11,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004527+0000","last_change":"2024-01-11T09:47:23.993016+0000","last_active":"2024-01-11T09:47:25.004527+0000","last_peered":"2024-01-11T09:47:25.004527+0000","last_clean":"2024-01-11T09:47:25.004527+0000","last_became_active":"2024-01-11T09:47:23.992681+0000","last_became_peered":"2024-01-11T09:47:23.992681+0000","last_unstale":"2024-01-11T09:47:25.004527+0000","last_undegraded":"2024-01-11T09:47:25.004527+0000","last_fullsized":"2024-01-11T09:47:25.004527+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:54:08.481155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.999814+0000","last_change":"2024-01-11T09:47:24.995779+0000","last_active":"2024-01-11T09:47:24.999814+0000","last_peered":"2024-01-11T09:47:24.999814+0000","last_clean":"2024-01-11T09:47:24.999814+0000","last_became_active":"2024-01-11T09:47:24.995465+0000","last_became_peered":"2024-01-11T09:47:24.995465+0000","last_unstale":"2024-01-11T09:47:24.999814+0000","last_undegraded":"2024-01-11T09:47:24.999814+0000","last_fullsized":"2024-01-11T09:47:24.999814+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:39:08.765597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1a","version":"255'1","reported_seq":11,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.009864+0000","last_change":"2024-01-11T09:47:23.987255+0000","last_active":"2024-01-11T09:47:25.009864+0000","last_peered":"2024-01-11T09:47:25.009864+0000","last_clean":"2024-01-11T09:47:25.009864+0000","last_became_active":"2024-01-11T09:47:23.986953+0000","last_became_peered":"2024-01-11T09:47:23.986953+0000","last_unstale":"2024-01-11T09:47:25.009864+0000","last_undegraded":"2024-01-11T09:47:25.009864+0000","last_fullsized":"2024-01-11T09:47:25.009864+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:58:13.358081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002018+0000","last_change":"2024-01-11T09:47:24.995673+0000","last_active":"2024-01-11T09:47:25.002018+0000","last_peered":"2024-01-11T09:47:25.002018+0000","last_clean":"2024-01-11T09:47:25.002018+0000","last_became_active":"2024-01-11T09:47:24.995421+0000","last_became_peered":"2024-01-11T09:47:24.995421+0000","last_unstale":"2024-01-11T09:47:25.002018+0000","last_undegraded":"2024-01-11T09:47:25.002018+0000","last_fullsized":"2024-01-11T09:47:25.002018+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:46:02.428482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.982524+0000","last_change":"2024-01-11T09:47:23.982683+0000","last_active":"2024-01-11T09:47:24.982524+0000","last_peered":"2024-01-11T09:47:24.982524+0000","last_clean":"2024-01-11T09:47:24.982524+0000","last_became_active":"2024-01-11T09:47:23.981992+0000","last_became_peered":"2024-01-11T09:47:23.981992+0000","last_unstale":"2024-01-11T09:47:24.982524+0000","last_undegraded":"2024-01-11T09:47:24.982524+0000","last_fullsized":"2024-01-11T09:47:24.982524+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:45:16.009410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.995046+0000","last_change":"2024-01-11T09:47:24.991131+0000","last_active":"2024-01-11T09:47:24.995046+0000","last_peered":"2024-01-11T09:47:24.995046+0000","last_clean":"2024-01-11T09:47:24.995046+0000","last_became_active":"2024-01-11T09:47:24.990857+0000","last_became_peered":"2024-01-11T09:47:24.990857+0000","last_unstale":"2024-01-11T09:47:24.995046+0000","last_undegraded":"2024-01-11T09:47:24.995046+0000","last_fullsized":"2024-01-11T09:47:24.995046+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:03:49.193894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.975904+0000","last_change":"2024-01-11T09:47:23.988334+0000","last_active":"2024-01-11T09:47:24.975904+0000","last_peered":"2024-01-11T09:47:24.975904+0000","last_clean":"2024-01-11T09:47:24.975904+0000","last_became_active":"2024-01-11T09:47:23.988049+0000","last_became_peered":"2024-01-11T09:47:23.988049+0000","last_unstale":"2024-01-11T09:47:24.975904+0000","last_undegraded":"2024-01-11T09:47:24.975904+0000","last_fullsized":"2024-01-11T09:47:24.975904+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:54:08.481155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.990622+0000","last_change":"2024-01-11T09:47:24.986224+0000","last_active":"2024-01-11T09:47:24.990622+0000","last_peered":"2024-01-11T09:47:24.990622+0000","last_clean":"2024-01-11T09:47:24.990622+0000","last_became_active":"2024-01-11T09:47:24.985954+0000","last_became_peered":"2024-01-11T09:47:24.985954+0000","last_unstale":"2024-01-11T09:47:24.990622+0000","last_undegraded":"2024-01-11T09:47:24.990622+0000","last_fullsized":"2024-01-11T09:47:24.990622+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:49:23.959042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.978606+0000","last_change":"2024-01-11T09:47:23.982382+0000","last_active":"2024-01-11T09:47:24.978606+0000","last_peered":"2024-01-11T09:47:24.978606+0000","last_clean":"2024-01-11T09:47:24.978606+0000","last_became_active":"2024-01-11T09:47:23.981842+0000","last_became_peered":"2024-01-11T09:47:23.981842+0000","last_unstale":"2024-01-11T09:47:24.978606+0000","last_undegraded":"2024-01-11T09:47:24.978606+0000","last_fullsized":"2024-01-11T09:47:24.978606+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:21.143548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002925+0000","last_change":"2024-01-11T09:47:24.997080+0000","last_active":"2024-01-11T09:47:25.002925+0000","last_peered":"2024-01-11T09:47:25.002925+0000","last_clean":"2024-01-11T09:47:25.002925+0000","last_became_active":"2024-01-11T09:47:24.995601+0000","last_became_peered":"2024-01-11T09:47:24.995601+0000","last_unstale":"2024-01-11T09:47:25.002925+0000","last_undegraded":"2024-01-11T09:47:25.002925+0000","last_fullsized":"2024-01-11T09:47:25.002925+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:40.944331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.972520+0000","last_change":"2024-01-11T09:47:23.985660+0000","last_active":"2024-01-11T09:47:24.972520+0000","last_peered":"2024-01-11T09:47:24.972520+0000","last_clean":"2024-01-11T09:47:24.972520+0000","last_became_active":"2024-01-11T09:47:23.985348+0000","last_became_peered":"2024-01-11T09:47:23.985348+0000","last_unstale":"2024-01-11T09:47:24.972520+0000","last_undegraded":"2024-01-11T09:47:24.972520+0000","last_fullsized":"2024-01-11T09:47:24.972520+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:21:00.064284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.991838+0000","last_change":"2024-01-11T09:47:24.988499+0000","last_active":"2024-01-11T09:47:24.991838+0000","last_peered":"2024-01-11T09:47:24.991838+0000","last_clean":"2024-01-11T09:47:24.991838+0000","last_became_active":"2024-01-11T09:47:24.988226+0000","last_became_peered":"2024-01-11T09:47:24.988226+0000","last_unstale":"2024-01-11T09:47:24.991838+0000","last_undegraded":"2024-01-11T09:47:24.991838+0000","last_fullsized":"2024-01-11T09:47:24.991838+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:48:34.768650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.15","version":"255'3","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004676+0000","last_change":"2024-01-11T09:47:23.992634+0000","last_active":"2024-01-11T09:47:25.004676+0000","last_peered":"2024-01-11T09:47:25.004676+0000","last_clean":"2024-01-11T09:47:25.004676+0000","last_became_active":"2024-01-11T09:47:23.992236+0000","last_became_peered":"2024-01-11T09:47:23.992236+0000","last_unstale":"2024-01-11T09:47:25.004676+0000","last_undegraded":"2024-01-11T09:47:25.004676+0000","last_fullsized":"2024-01-11T09:47:25.004676+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:44:42.790810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.993190+0000","last_change":"2024-01-11T09:47:24.985111+0000","last_active":"2024-01-11T09:47:24.993190+0000","last_peered":"2024-01-11T09:47:24.993190+0000","last_clean":"2024-01-11T09:47:24.993190+0000","last_became_active":"2024-01-11T09:47:24.981918+0000","last_became_peered":"2024-01-11T09:47:24.981918+0000","last_unstale":"2024-01-11T09:47:24.993190+0000","last_undegraded":"2024-01-11T09:47:24.993190+0000","last_fullsized":"2024-01-11T09:47:24.993190+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:33:03.766112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.14","version":"255'6","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.010959+0000","last_change":"2024-01-11T09:47:23.979893+0000","last_active":"2024-01-11T09:47:25.010959+0000","last_peered":"2024-01-11T09:47:25.010959+0000","last_clean":"2024-01-11T09:47:25.010959+0000","last_became_active":"2024-01-11T09:47:23.979651+0000","last_became_peered":"2024-01-11T09:47:23.979651+0000","last_unstale":"2024-01-11T09:47:25.010959+0000","last_undegraded":"2024-01-11T09:47:25.010959+0000","last_fullsized":"2024-01-11T09:47:25.010959+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:43:25.959850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002230+0000","last_change":"2024-01-11T09:47:24.998313+0000","last_active":"2024-01-11T09:47:25.002230+0000","last_peered":"2024-01-11T09:47:25.002230+0000","last_clean":"2024-01-11T09:47:25.002230+0000","last_became_active":"2024-01-11T09:47:24.998032+0000","last_became_peered":"2024-01-11T09:47:24.998032+0000","last_unstale":"2024-01-11T09:47:25.002230+0000","last_undegraded":"2024-01-11T09:47:25.002230+0000","last_fullsized":"2024-01-11T09:47:25.002230+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:20:08.159916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.13","version":"255'2","reported_seq":12,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.010541+0000","last_change":"2024-01-11T09:47:23.985710+0000","last_active":"2024-01-11T09:47:25.010541+0000","last_peered":"2024-01-11T09:47:25.010541+0000","last_clean":"2024-01-11T09:47:25.010541+0000","last_became_active":"2024-01-11T09:47:23.985422+0000","last_became_peered":"2024-01-11T09:47:23.985422+0000","last_unstale":"2024-01-11T09:47:25.010541+0000","last_undegraded":"2024-01-11T09:47:25.010541+0000","last_fullsized":"2024-01-11T09:47:25.010541+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:16:05.292707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.007847+0000","last_change":"2024-01-11T09:47:25.005425+0000","last_active":"2024-01-11T09:47:25.007847+0000","last_peered":"2024-01-11T09:47:25.007847+0000","last_clean":"2024-01-11T09:47:25.007847+0000","last_became_active":"2024-01-11T09:47:25.005066+0000","last_became_peered":"2024-01-11T09:47:25.005066+0000","last_unstale":"2024-01-11T09:47:25.007847+0000","last_undegraded":"2024-01-11T09:47:25.007847+0000","last_fullsized":"2024-01-11T09:47:25.007847+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:16:51.113785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.974732+0000","last_change":"2024-01-11T09:47:23.983718+0000","last_active":"2024-01-11T09:47:24.974732+0000","last_peered":"2024-01-11T09:47:24.974732+0000","last_clean":"2024-01-11T09:47:24.974732+0000","last_became_active":"2024-01-11T09:47:23.980334+0000","last_became_peered":"2024-01-11T09:47:23.980334+0000","last_unstale":"2024-01-11T09:47:24.974732+0000","last_undegraded":"2024-01-11T09:47:24.974732+0000","last_fullsized":"2024-01-11T09:47:24.974732+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:07:38.872375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.990746+0000","last_change":"2024-01-11T09:47:24.986626+0000","last_active":"2024-01-11T09:47:24.990746+0000","last_peered":"2024-01-11T09:47:24.990746+0000","last_clean":"2024-01-11T09:47:24.990746+0000","last_became_active":"2024-01-11T09:47:24.984969+0000","last_became_peered":"2024-01-11T09:47:24.984969+0000","last_unstale":"2024-01-11T09:47:24.990746+0000","last_undegraded":"2024-01-11T09:47:24.990746+0000","last_fullsized":"2024-01-11T09:47:24.990746+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:13:19.226196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.975742+0000","last_change":"2024-01-11T09:47:23.987106+0000","last_active":"2024-01-11T09:47:24.975742+0000","last_peered":"2024-01-11T09:47:24.975742+0000","last_clean":"2024-01-11T09:47:24.975742+0000","last_became_active":"2024-01-11T09:47:23.986838+0000","last_became_peered":"2024-01-11T09:47:23.986838+0000","last_unstale":"2024-01-11T09:47:24.975742+0000","last_undegraded":"2024-01-11T09:47:24.975742+0000","last_fullsized":"2024-01-11T09:47:24.975742+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:31:39.274165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.999673+0000","last_change":"2024-01-11T09:47:24.995245+0000","last_active":"2024-01-11T09:47:24.999673+0000","last_peered":"2024-01-11T09:47:24.999673+0000","last_clean":"2024-01-11T09:47:24.999673+0000","last_became_active":"2024-01-11T09:47:24.994109+0000","last_became_peered":"2024-01-11T09:47:24.994109+0000","last_unstale":"2024-01-11T09:47:24.999673+0000","last_undegraded":"2024-01-11T09:47:24.999673+0000","last_fullsized":"2024-01-11T09:47:24.999673+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:26:09.368076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.10","version":"255'8","reported_seq":18,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.009227+0000","last_change":"2024-01-11T09:47:23.992629+0000","last_active":"2024-01-11T09:47:25.009227+0000","last_peered":"2024-01-11T09:47:25.009227+0000","last_clean":"2024-01-11T09:47:25.009227+0000","last_became_active":"2024-01-11T09:47:23.992344+0000","last_became_peered":"2024-01-11T09:47:23.992344+0000","last_unstale":"2024-01-11T09:47:25.009227+0000","last_undegraded":"2024-01-11T09:47:25.009227+0000","last_fullsized":"2024-01-11T09:47:25.009227+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:16:18.912554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.987383+0000","last_change":"2024-01-11T09:47:24.984069+0000","last_active":"2024-01-11T09:47:24.987383+0000","last_peered":"2024-01-11T09:47:24.987383+0000","last_clean":"2024-01-11T09:47:24.987383+0000","last_became_active":"2024-01-11T09:47:24.983802+0000","last_became_peered":"2024-01-11T09:47:24.983802+0000","last_unstale":"2024-01-11T09:47:24.987383+0000","last_undegraded":"2024-01-11T09:47:24.987383+0000","last_fullsized":"2024-01-11T09:47:24.987383+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:04:45.182025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"255'1","reported_seq":11,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.009813+0000","last_change":"2024-01-11T09:47:23.985751+0000","last_active":"2024-01-11T09:47:25.009813+0000","last_peered":"2024-01-11T09:47:25.009813+0000","last_clean":"2024-01-11T09:47:25.009813+0000","last_became_active":"2024-01-11T09:47:23.985422+0000","last_became_peered":"2024-01-11T09:47:23.985422+0000","last_unstale":"2024-01-11T09:47:25.009813+0000","last_undegraded":"2024-01-11T09:47:25.009813+0000","last_fullsized":"2024-01-11T09:47:25.009813+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:13:38.339268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.986787+0000","last_change":"2024-01-11T09:47:24.982411+0000","last_active":"2024-01-11T09:47:24.986787+0000","last_peered":"2024-01-11T09:47:24.986787+0000","last_clean":"2024-01-11T09:47:24.986787+0000","last_became_active":"2024-01-11T09:47:24.980530+0000","last_became_peered":"2024-01-11T09:47:24.980530+0000","last_unstale":"2024-01-11T09:47:24.986787+0000","last_undegraded":"2024-01-11T09:47:24.986787+0000","last_fullsized":"2024-01-11T09:47:24.986787+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:26:00.307598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.e","version":"255'2","reported_seq":12,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.006007+0000","last_change":"2024-01-11T09:47:23.974627+0000","last_active":"2024-01-11T09:47:25.006007+0000","last_peered":"2024-01-11T09:47:25.006007+0000","last_clean":"2024-01-11T09:47:25.006007+0000","last_became_active":"2024-01-11T09:47:23.973839+0000","last_became_peered":"2024-01-11T09:47:23.973839+0000","last_unstale":"2024-01-11T09:47:25.006007+0000","last_undegraded":"2024-01-11T09:47:25.006007+0000","last_fullsized":"2024-01-11T09:47:25.006007+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:57:20.246338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004188+0000","last_change":"2024-01-11T09:47:24.997271+0000","last_active":"2024-01-11T09:47:25.004188+0000","last_peered":"2024-01-11T09:47:25.004188+0000","last_clean":"2024-01-11T09:47:25.004188+0000","last_became_active":"2024-01-11T09:47:24.995852+0000","last_became_peered":"2024-01-11T09:47:24.995852+0000","last_unstale":"2024-01-11T09:47:25.004188+0000","last_undegraded":"2024-01-11T09:47:25.004188+0000","last_fullsized":"2024-01-11T09:47:25.004188+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:48:39.698634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.d","version":"255'1","reported_seq":11,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.007572+0000","last_change":"2024-01-11T09:47:23.989553+0000","last_active":"2024-01-11T09:47:25.007572+0000","last_peered":"2024-01-11T09:47:25.007572+0000","last_clean":"2024-01-11T09:47:25.007572+0000","last_became_active":"2024-01-11T09:47:23.989285+0000","last_became_peered":"2024-01-11T09:47:23.989285+0000","last_unstale":"2024-01-11T09:47:25.007572+0000","last_undegraded":"2024-01-11T09:47:25.007572+0000","last_fullsized":"2024-01-11T09:47:25.007572+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:36:27.961488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.980678+0000","last_change":"2024-01-11T09:47:24.976990+0000","last_active":"2024-01-11T09:47:24.980678+0000","last_peered":"2024-01-11T09:47:24.980678+0000","last_clean":"2024-01-11T09:47:24.980678+0000","last_became_active":"2024-01-11T09:47:24.976691+0000","last_became_peered":"2024-01-11T09:47:24.976691+0000","last_unstale":"2024-01-11T09:47:24.980678+0000","last_undegraded":"2024-01-11T09:47:24.980678+0000","last_fullsized":"2024-01-11T09:47:24.980678+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:03:19.459500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.980699+0000","last_change":"2024-01-11T09:47:23.980634+0000","last_active":"2024-01-11T09:47:24.980699+0000","last_peered":"2024-01-11T09:47:24.980699+0000","last_clean":"2024-01-11T09:47:24.980699+0000","last_became_active":"2024-01-11T09:47:23.980274+0000","last_became_peered":"2024-01-11T09:47:23.980274+0000","last_unstale":"2024-01-11T09:47:24.980699+0000","last_undegraded":"2024-01-11T09:47:24.980699+0000","last_fullsized":"2024-01-11T09:47:24.980699+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:34:28.082927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002123+0000","last_change":"2024-01-11T09:47:24.994868+0000","last_active":"2024-01-11T09:47:25.002123+0000","last_peered":"2024-01-11T09:47:25.002123+0000","last_clean":"2024-01-11T09:47:25.002123+0000","last_became_active":"2024-01-11T09:47:24.994389+0000","last_became_peered":"2024-01-11T09:47:24.994389+0000","last_unstale":"2024-01-11T09:47:25.002123+0000","last_undegraded":"2024-01-11T09:47:25.002123+0000","last_fullsized":"2024-01-11T09:47:25.002123+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:22.152883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.b","version":"255'2","reported_seq":12,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.008312+0000","last_change":"2024-01-11T09:47:23.990231+0000","last_active":"2024-01-11T09:47:25.008312+0000","last_peered":"2024-01-11T09:47:25.008312+0000","last_clean":"2024-01-11T09:47:25.008312+0000","last_became_active":"2024-01-11T09:47:23.989904+0000","last_became_peered":"2024-01-11T09:47:23.989904+0000","last_unstale":"2024-01-11T09:47:25.008312+0000","last_undegraded":"2024-01-11T09:47:25.008312+0000","last_fullsized":"2024-01-11T09:47:25.008312+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:32:21.144708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.990629+0000","last_change":"2024-01-11T09:47:24.986224+0000","last_active":"2024-01-11T09:47:24.990629+0000","last_peered":"2024-01-11T09:47:24.990629+0000","last_clean":"2024-01-11T09:47:24.990629+0000","last_became_active":"2024-01-11T09:47:24.985980+0000","last_became_peered":"2024-01-11T09:47:24.985980+0000","last_unstale":"2024-01-11T09:47:24.990629+0000","last_undegraded":"2024-01-11T09:47:24.990629+0000","last_fullsized":"2024-01-11T09:47:24.990629+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:21:42.087560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.970334+0000","last_change":"2024-01-11T09:47:23.985032+0000","last_active":"2024-01-11T09:47:24.970334+0000","last_peered":"2024-01-11T09:47:24.970334+0000","last_clean":"2024-01-11T09:47:24.970334+0000","last_became_active":"2024-01-11T09:47:23.984242+0000","last_became_peered":"2024-01-11T09:47:23.984242+0000","last_unstale":"2024-01-11T09:47:24.970334+0000","last_undegraded":"2024-01-11T09:47:24.970334+0000","last_fullsized":"2024-01-11T09:47:24.970334+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:45:16.009410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004925+0000","last_change":"2024-01-11T09:47:25.001180+0000","last_active":"2024-01-11T09:47:25.004925+0000","last_peered":"2024-01-11T09:47:25.004925+0000","last_clean":"2024-01-11T09:47:25.004925+0000","last_became_active":"2024-01-11T09:47:24.997474+0000","last_became_peered":"2024-01-11T09:47:24.997474+0000","last_unstale":"2024-01-11T09:47:25.004925+0000","last_undegraded":"2024-01-11T09:47:25.004925+0000","last_fullsized":"2024-01-11T09:47:25.004925+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:15:03.907181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.974098+0000","last_change":"2024-01-11T09:47:23.981388+0000","last_active":"2024-01-11T09:47:24.974098+0000","last_peered":"2024-01-11T09:47:24.974098+0000","last_clean":"2024-01-11T09:47:24.974098+0000","last_became_active":"2024-01-11T09:47:23.981006+0000","last_became_peered":"2024-01-11T09:47:23.981006+0000","last_unstale":"2024-01-11T09:47:24.974098+0000","last_undegraded":"2024-01-11T09:47:24.974098+0000","last_fullsized":"2024-01-11T09:47:24.974098+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:44:11.182427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.001816+0000","last_change":"2024-01-11T09:47:24.995629+0000","last_active":"2024-01-11T09:47:25.001816+0000","last_peered":"2024-01-11T09:47:25.001816+0000","last_clean":"2024-01-11T09:47:25.001816+0000","last_became_active":"2024-01-11T09:47:24.994627+0000","last_became_peered":"2024-01-11T09:47:24.994627+0000","last_unstale":"2024-01-11T09:47:25.001816+0000","last_undegraded":"2024-01-11T09:47:25.001816+0000","last_fullsized":"2024-01-11T09:47:25.001816+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:23:57.820334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.982072+0000","last_change":"2024-01-11T09:47:23.989028+0000","last_active":"2024-01-11T09:47:24.982072+0000","last_peered":"2024-01-11T09:47:24.982072+0000","last_clean":"2024-01-11T09:47:24.982072+0000","last_became_active":"2024-01-11T09:47:23.988482+0000","last_became_peered":"2024-01-11T09:47:23.988482+0000","last_unstale":"2024-01-11T09:47:24.982072+0000","last_undegraded":"2024-01-11T09:47:24.982072+0000","last_fullsized":"2024-01-11T09:47:24.982072+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:53:24.155769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002478+0000","last_change":"2024-01-11T09:47:24.998398+0000","last_active":"2024-01-11T09:47:25.002478+0000","last_peered":"2024-01-11T09:47:25.002478+0000","last_clean":"2024-01-11T09:47:25.002478+0000","last_became_active":"2024-01-11T09:47:24.998044+0000","last_became_peered":"2024-01-11T09:47:24.998044+0000","last_unstale":"2024-01-11T09:47:25.002478+0000","last_undegraded":"2024-01-11T09:47:25.002478+0000","last_fullsized":"2024-01-11T09:47:25.002478+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:24:05.147676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.7","version":"255'2","reported_seq":12,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004581+0000","last_change":"2024-01-11T09:47:23.983465+0000","last_active":"2024-01-11T09:47:25.004581+0000","last_peered":"2024-01-11T09:47:25.004581+0000","last_clean":"2024-01-11T09:47:25.004581+0000","last_became_active":"2024-01-11T09:47:23.982586+0000","last_became_peered":"2024-01-11T09:47:23.982586+0000","last_unstale":"2024-01-11T09:47:25.004581+0000","last_undegraded":"2024-01-11T09:47:25.004581+0000","last_fullsized":"2024-01-11T09:47:25.004581+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:48:21.245464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.993191+0000","last_change":"2024-01-11T09:47:24.990311+0000","last_active":"2024-01-11T09:47:24.993191+0000","last_peered":"2024-01-11T09:47:24.993191+0000","last_clean":"2024-01-11T09:47:24.993191+0000","last_became_active":"2024-01-11T09:47:24.990039+0000","last_became_peered":"2024-01-11T09:47:24.990039+0000","last_unstale":"2024-01-11T09:47:24.993191+0000","last_undegraded":"2024-01-11T09:47:24.993191+0000","last_fullsized":"2024-01-11T09:47:24.993191+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:36:54.837036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.983322+0000","last_change":"2024-01-11T09:47:23.984889+0000","last_active":"2024-01-11T09:47:24.983322+0000","last_peered":"2024-01-11T09:47:24.983322+0000","last_clean":"2024-01-11T09:47:24.983322+0000","last_became_active":"2024-01-11T09:47:23.984422+0000","last_became_peered":"2024-01-11T09:47:23.984422+0000","last_unstale":"2024-01-11T09:47:24.983322+0000","last_undegraded":"2024-01-11T09:47:24.983322+0000","last_fullsized":"2024-01-11T09:47:24.983322+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:55:32.480267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.996505+0000","last_change":"2024-01-11T09:47:24.992441+0000","last_active":"2024-01-11T09:47:24.996505+0000","last_peered":"2024-01-11T09:47:24.996505+0000","last_clean":"2024-01-11T09:47:24.996505+0000","last_became_active":"2024-01-11T09:47:24.992125+0000","last_became_peered":"2024-01-11T09:47:24.992125+0000","last_unstale":"2024-01-11T09:47:24.996505+0000","last_undegraded":"2024-01-11T09:47:24.996505+0000","last_fullsized":"2024-01-11T09:47:24.996505+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:17:43.512711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.973292+0000","last_change":"2024-01-11T09:47:23.984077+0000","last_active":"2024-01-11T09:47:24.973292+0000","last_peered":"2024-01-11T09:47:24.973292+0000","last_clean":"2024-01-11T09:47:24.973292+0000","last_became_active":"2024-01-11T09:47:23.982674+0000","last_became_peered":"2024-01-11T09:47:23.982674+0000","last_unstale":"2024-01-11T09:47:24.973292+0000","last_undegraded":"2024-01-11T09:47:24.973292+0000","last_fullsized":"2024-01-11T09:47:24.973292+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:38:13.535607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002340+0000","last_change":"2024-01-11T09:47:24.997499+0000","last_active":"2024-01-11T09:47:25.002340+0000","last_peered":"2024-01-11T09:47:25.002340+0000","last_clean":"2024-01-11T09:47:25.002340+0000","last_became_active":"2024-01-11T09:47:24.993811+0000","last_became_peered":"2024-01-11T09:47:24.993811+0000","last_unstale":"2024-01-11T09:47:25.002340+0000","last_undegraded":"2024-01-11T09:47:25.002340+0000","last_fullsized":"2024-01-11T09:47:25.002340+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:39:30.172237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.972809+0000","last_change":"2024-01-11T09:47:23.990863+0000","last_active":"2024-01-11T09:47:24.972809+0000","last_peered":"2024-01-11T09:47:24.972809+0000","last_clean":"2024-01-11T09:47:24.972809+0000","last_became_active":"2024-01-11T09:47:23.990051+0000","last_became_peered":"2024-01-11T09:47:23.990051+0000","last_unstale":"2024-01-11T09:47:24.972809+0000","last_undegraded":"2024-01-11T09:47:24.972809+0000","last_fullsized":"2024-01-11T09:47:24.972809+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:30:02.583920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002102+0000","last_change":"2024-01-11T09:47:24.994678+0000","last_active":"2024-01-11T09:47:25.002102+0000","last_peered":"2024-01-11T09:47:25.002102+0000","last_clean":"2024-01-11T09:47:25.002102+0000","last_became_active":"2024-01-11T09:47:24.994225+0000","last_became_peered":"2024-01-11T09:47:24.994225+0000","last_unstale":"2024-01-11T09:47:25.002102+0000","last_undegraded":"2024-01-11T09:47:25.002102+0000","last_fullsized":"2024-01-11T09:47:25.002102+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:44:12.191761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.979025+0000","last_change":"2024-01-11T09:47:23.988752+0000","last_active":"2024-01-11T09:47:24.979025+0000","last_peered":"2024-01-11T09:47:24.979025+0000","last_clean":"2024-01-11T09:47:24.979025+0000","last_became_active":"2024-01-11T09:47:23.988477+0000","last_became_peered":"2024-01-11T09:47:23.988477+0000","last_unstale":"2024-01-11T09:47:24.979025+0000","last_undegraded":"2024-01-11T09:47:24.979025+0000","last_fullsized":"2024-01-11T09:47:24.979025+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:39:41.499725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.991757+0000","last_change":"2024-01-11T09:47:24.989860+0000","last_active":"2024-01-11T09:47:24.991757+0000","last_peered":"2024-01-11T09:47:24.991757+0000","last_clean":"2024-01-11T09:47:24.991757+0000","last_became_active":"2024-01-11T09:47:24.988915+0000","last_became_peered":"2024-01-11T09:47:24.988915+0000","last_unstale":"2024-01-11T09:47:24.991757+0000","last_undegraded":"2024-01-11T09:47:24.991757+0000","last_fullsized":"2024-01-11T09:47:24.991757+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:20:16.704821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.985174+0000","last_change":"2024-01-11T09:47:23.982214+0000","last_active":"2024-01-11T09:47:24.985174+0000","last_peered":"2024-01-11T09:47:24.985174+0000","last_clean":"2024-01-11T09:47:24.985174+0000","last_became_active":"2024-01-11T09:47:23.981616+0000","last_became_peered":"2024-01-11T09:47:23.981616+0000","last_unstale":"2024-01-11T09:47:24.985174+0000","last_undegraded":"2024-01-11T09:47:24.985174+0000","last_fullsized":"2024-01-11T09:47:24.985174+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:43:25.959850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004904+0000","last_change":"2024-01-11T09:47:25.001162+0000","last_active":"2024-01-11T09:47:25.004904+0000","last_peered":"2024-01-11T09:47:25.004904+0000","last_clean":"2024-01-11T09:47:25.004904+0000","last_became_active":"2024-01-11T09:47:24.997121+0000","last_became_peered":"2024-01-11T09:47:24.997121+0000","last_unstale":"2024-01-11T09:47:25.004904+0000","last_undegraded":"2024-01-11T09:47:25.004904+0000","last_fullsized":"2024-01-11T09:47:25.004904+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:42:18.444395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.982605+0000","last_change":"2024-01-11T09:47:23.981247+0000","last_active":"2024-01-11T09:47:24.982605+0000","last_peered":"2024-01-11T09:47:24.982605+0000","last_clean":"2024-01-11T09:47:24.982605+0000","last_became_active":"2024-01-11T09:47:23.980209+0000","last_became_peered":"2024-01-11T09:47:23.980209+0000","last_unstale":"2024-01-11T09:47:24.982605+0000","last_undegraded":"2024-01-11T09:47:24.982605+0000","last_fullsized":"2024-01-11T09:47:24.982605+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:57:20.246338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004798+0000","last_change":"2024-01-11T09:47:24.999219+0000","last_active":"2024-01-11T09:47:25.004798+0000","last_peered":"2024-01-11T09:47:25.004798+0000","last_clean":"2024-01-11T09:47:25.004798+0000","last_became_active":"2024-01-11T09:47:24.994240+0000","last_became_peered":"2024-01-11T09:47:24.994240+0000","last_unstale":"2024-01-11T09:47:25.004798+0000","last_undegraded":"2024-01-11T09:47:25.004798+0000","last_fullsized":"2024-01-11T09:47:25.004798+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:04:45.182025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.969020+0000","last_change":"2024-01-11T09:47:23.983239+0000","last_active":"2024-01-11T09:47:24.969020+0000","last_peered":"2024-01-11T09:47:24.969020+0000","last_clean":"2024-01-11T09:47:24.969020+0000","last_became_active":"2024-01-11T09:47:23.982982+0000","last_became_peered":"2024-01-11T09:47:23.982982+0000","last_unstale":"2024-01-11T09:47:24.969020+0000","last_undegraded":"2024-01-11T09:47:24.969020+0000","last_fullsized":"2024-01-11T09:47:24.969020+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:53:24.155769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.990534+0000","last_change":"2024-01-11T09:47:24.987204+0000","last_active":"2024-01-11T09:47:24.990534+0000","last_peered":"2024-01-11T09:47:24.990534+0000","last_clean":"2024-01-11T09:47:24.990534+0000","last_became_active":"2024-01-11T09:47:24.984730+0000","last_became_peered":"2024-01-11T09:47:24.984730+0000","last_unstale":"2024-01-11T09:47:24.990534+0000","last_undegraded":"2024-01-11T09:47:24.990534+0000","last_fullsized":"2024-01-11T09:47:24.990534+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:46:15.517759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"242'408","reported_seq":1383,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.972278+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:47:24.972278+0000","last_peered":"2024-01-11T09:47:24.972278+0000","last_clean":"2024-01-11T09:47:24.972278+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:47:24.972278+0000","last_undegraded":"2024-01-11T09:47:24.972278+0000","last_fullsized":"2024-01-11T09:47:24.972278+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":408,"log_dups_size":0,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2654,"num_read_kb":5617,"num_write":687,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2654,"num_read_kb":5617,"num_write":687,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":408,"ondisk_log_size":408,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575992,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305124,"kb_used_data":3740,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94066716,"statfs":{"total":96636764160,"available":96324317184,"internally_reserved":0,"allocated":3829760,"data_stored":2158745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.68899999999999995,"15min":0.72799999999999998},"min":{"1min":0.39100000000000001,"5min":0.376,"15min":0.36599999999999999},"max":{"1min":0.89200000000000002,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.71499999999999997,"15min":0.74299999999999999},"min":{"1min":0.40400000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83799999999999997,"5min":1.0609999999999999,"15min":1.0669999999999999},"last":0.55600000000000005}]},{"osd":1,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.69799999999999995,"15min":0.72499999999999998},"min":{"1min":0.48099999999999998,"5min":0.41799999999999998,"15min":0.41599999999999998},"max":{"1min":0.82099999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.70499999999999996,"15min":0.73599999999999999},"min":{"1min":0.504,"5min":0.434,"15min":0.434},"max":{"1min":0.89400000000000002,"5min":0.999,"15min":1.0820000000000001},"last":0.69499999999999995}]},{"osd":2,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.71799999999999997,"15min":0.73499999999999999},"min":{"1min":0.46899999999999997,"5min":0.443,"15min":0.443},"max":{"1min":0.875,"5min":0.94399999999999995,"15min":1.0600000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.71099999999999997,"15min":0.747},"min":{"1min":0.48499999999999999,"5min":0.44800000000000001,"15min":0.44500000000000001},"max":{"1min":0.86099999999999999,"5min":0.96999999999999997,"15min":1.153},"last":0.67900000000000005}]},{"osd":3,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73599999999999999,"15min":0.76200000000000001},"min":{"1min":0.51500000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":1.0660000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.72299999999999998,"15min":0.747},"min":{"1min":0.51000000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.77100000000000002,"5min":1,"15min":1.0880000000000001},"last":0.63900000000000001}]},{"osd":4,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61099999999999999,"15min":0.60599999999999998},"min":{"1min":0.45900000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79600000000000004,"5min":1.0860000000000001,"15min":1.107},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64000000000000001,"15min":0.63900000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39700000000000002},"max":{"1min":0.871,"5min":0.995,"15min":1.0960000000000001},"last":0.84199999999999997}]},{"osd":5,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.622,"15min":0.64100000000000001},"min":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86099999999999999,"5min":0.86699999999999999,"15min":1.0109999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.69199999999999995,"15min":0.68799999999999994},"min":{"1min":0.41499999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.64900000000000002}]},{"osd":6,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.78000000000000003},"min":{"1min":0.56000000000000005,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.96899999999999997,"5min":1.232,"15min":1.232},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.002,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.88300000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575991,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312740,"kb_used_data":3164,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059100,"statfs":{"total":96636764160,"available":96316518400,"internally_reserved":0,"allocated":3239936,"data_stored":1568467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.63200000000000001,"15min":0.69499999999999995},"min":{"1min":0.438,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.79800000000000004,"5min":0.94199999999999995,"15min":0.999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.68400000000000005,"15min":0.72799999999999998},"min":{"1min":0.46100000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.93400000000000005,"5min":0.97099999999999997,"15min":1.216},"last":0.67500000000000004}]},{"osd":1,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.66000000000000003,"15min":0.72199999999999998},"min":{"1min":0.46200000000000002,"5min":0.42799999999999999,"15min":0.41699999999999998},"max":{"1min":0.91000000000000003,"5min":0.98499999999999999,"15min":1.3129999999999999},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.67000000000000004,"15min":0.71599999999999997},"min":{"1min":0.46500000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.91100000000000003,"5min":0.95999999999999996,"15min":1.0840000000000001},"last":0.81499999999999995}]},{"osd":2,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.68700000000000006,"15min":0.73499999999999999},"min":{"1min":0.44600000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.83799999999999997,"5min":1.0069999999999999,"15min":1.4179999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.69999999999999996,"15min":0.73999999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.92800000000000005,"5min":1.796,"15min":1.796},"last":0.90400000000000003}]},{"osd":3,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.71299999999999997,"15min":0.746},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89000000000000001,"5min":1.012,"15min":1.014},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.70099999999999996,"15min":0.754},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":1.165},"last":0.86199999999999999}]},{"osd":4,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.59399999999999997,"15min":0.56999999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.86599999999999999,"5min":0.91700000000000004,"15min":1.0309999999999999},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.64400000000000002,"15min":0.622},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":1.1259999999999999},"last":0.95699999999999996}]},{"osd":5,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.077,"5min":1.077,"15min":1.369},"last":1.0880000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.71699999999999997,"15min":0.70899999999999996},"min":{"1min":0.495,"5min":0.49099999999999999,"15min":0.45400000000000001},"max":{"1min":0.94399999999999995,"5min":0.999,"15min":1.0960000000000001},"last":1.008}]},{"osd":7,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.72099999999999997,"15min":0.71399999999999997},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.45100000000000001},"max":{"1min":1.0089999999999999,"5min":1.089,"15min":1.484},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.77500000000000002,"15min":0.77200000000000002},"min":{"1min":0.53700000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.123,"5min":1.1379999999999999,"15min":1.603},"last":1.1299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575993,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308644,"kb_used_data":3164,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063196,"statfs":{"total":96636764160,"available":96320712704,"internally_reserved":0,"allocated":3239936,"data_stored":1568411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:46 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.504,"15min":0.50800000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.65500000000000003,"5min":0.73399999999999999,"15min":0.93999999999999995},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.58199999999999996,"15min":0.60799999999999998},"min":{"1min":0.41799999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.69999999999999996,"5min":0.85199999999999998,"15min":0.95099999999999996},"last":0.41899999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:46:46 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.48899999999999999,"15min":0.504},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.80100000000000005,"5min":0.82399999999999995,"15min":0.82699999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.57499999999999996,"15min":0.58599999999999997},"min":{"1min":0.38400000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.67400000000000004,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.39300000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:46:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.81299999999999994,"5min":0.91600000000000004,"15min":0.96699999999999997},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.57099999999999995,"15min":0.57599999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.34599999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:46:46 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.79100000000000004,"15min":0.81200000000000006},"min":{"1min":0.54900000000000004,"5min":0.51400000000000001,"15min":0.505},"max":{"1min":0.94699999999999995,"5min":1.214,"15min":1.2969999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.82899999999999996,"15min":0.85199999999999998},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.008,"5min":1.1879999999999999,"15min":1.1890000000000001},"last":0.72499999999999998}]},{"osd":5,"last update":"Thu Jan 11 09:46:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.80800000000000005,"15min":0.81999999999999995},"min":{"1min":0.55900000000000005,"5min":0.443,"15min":0.443},"max":{"1min":0.98399999999999999,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.79500000000000004,"15min":0.80300000000000005},"min":{"1min":0.56499999999999995,"5min":0.54200000000000004,"15min":0.51300000000000001},"max":{"1min":0.99399999999999999,"5min":1.341,"15min":1.341},"last":0.68600000000000005}]},{"osd":6,"last update":"Thu Jan 11 09:46:46 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.81799999999999995,"15min":0.83799999999999997},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.56799999999999995},"max":{"1min":0.97999999999999998,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.748,"5min":0.82899999999999996,"15min":0.84699999999999998},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.94199999999999995,"5min":1.177,"15min":1.177},"last":0.57299999999999995}]},{"osd":7,"last update":"Thu Jan 11 09:46:46 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.87,"15min":0.879},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.1279999999999999,"5min":1.421,"15min":1.421},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.84499999999999997,"15min":0.85199999999999998},"min":{"1min":0.623,"5min":0.60099999999999998,"15min":0.59299999999999997},"max":{"1min":1.163,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.77400000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575992,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309232,"kb_used_data":3752,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062608,"statfs":{"total":96636764160,"available":96320110592,"internally_reserved":0,"allocated":3842048,"data_stored":2159355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.48799999999999999,"15min":0.51300000000000001},"min":{"1min":0.34899999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.73099999999999998,"5min":0.754,"15min":0.84699999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.56299999999999994,"15min":0.59099999999999997},"min":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.59599999999999997,"5min":0.81599999999999995,"15min":0.874},"last":0.70499999999999996}]},{"osd":2,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.53300000000000003,"15min":0.56200000000000006},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.68200000000000005,"5min":0.83099999999999996,"15min":0.97499999999999998},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55400000000000005,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.86399999999999999},"last":0.59099999999999997}]},{"osd":3,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51100000000000001,"15min":0.53800000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.95399999999999996},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60699999999999998,"15min":0.64400000000000002},"min":{"1min":0.42499999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.87,"5min":0.90300000000000002,"15min":1.0389999999999999},"last":0.81200000000000006}]},{"osd":4,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.78600000000000003,"15min":0.81100000000000005},"min":{"1min":0.46100000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.304},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.78500000000000003,"15min":0.80100000000000005},"min":{"1min":0.48799999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.95399999999999996,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":1.0049999999999999}]},{"osd":5,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.80800000000000005,"15min":0.83599999999999997},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.97599999999999998,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.81399999999999995,"15min":0.83699999999999997},"min":{"1min":0.60399999999999998,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.02,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":1.139}]},{"osd":6,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.79700000000000004,"15min":0.82099999999999995},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.95899999999999996,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.82299999999999995,"15min":0.85999999999999999},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":0.97699999999999998,"5min":1.1699999999999999,"15min":1.256},"last":1.1220000000000001}]},{"osd":7,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.83699999999999997,"15min":0.86299999999999999},"min":{"1min":0.64100000000000001,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1000000000000001,"5min":1.226,"15min":1.28},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.84499999999999997,"15min":0.86899999999999999},"min":{"1min":0.63,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0369999999999999,"5min":1.238,"15min":1.2410000000000001},"last":1.073}]}]},{"osd":2,"up_from":13,"seq":55834575991,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300452,"kb_used_data":3164,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071388,"statfs":{"total":96636764160,"available":96329101312,"internally_reserved":0,"allocated":3239936,"data_stored":1568467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.46999999999999997,"15min":0.5},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.27900000000000003},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.85399999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.53400000000000003,"15min":0.56000000000000005},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66900000000000004,"5min":0.77300000000000002,"15min":1.6359999999999999},"last":0.82299999999999995}]},{"osd":1,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56499999999999995,"15min":0.57999999999999996},"min":{"1min":0.40000000000000002,"5min":0.38200000000000001,"15min":0.33500000000000002},"max":{"1min":0.70099999999999996,"5min":0.88800000000000001,"15min":0.93000000000000005},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.56399999999999995,"15min":0.59299999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.64800000000000002,"5min":0.97299999999999998,"15min":1.145},"last":0.71799999999999997}]},{"osd":3,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52900000000000003,"15min":0.55000000000000004},"min":{"1min":0.376,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81499999999999995,"5min":0.83099999999999996,"15min":0.96399999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.61199999999999999,"15min":0.65000000000000002},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.75600000000000001,"5min":0.876,"15min":1.3620000000000001},"last":0.498}]},{"osd":4,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.75700000000000001,"15min":0.79100000000000004},"min":{"1min":0.56299999999999994,"5min":0.55800000000000005,"15min":0.54300000000000004},"max":{"1min":0.96499999999999997,"5min":1.071,"15min":1.1519999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.76400000000000001,"15min":0.79300000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0209999999999999,"5min":1.161,"15min":1.161},"last":0.63}]},{"osd":5,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.79900000000000004,"15min":0.85599999999999998},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.94099999999999995,"5min":1.228,"15min":1.429},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.82799999999999996,"15min":0.85299999999999998},"min":{"1min":0.61499999999999999,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":0.94699999999999995,"5min":1.1850000000000001,"15min":1.2170000000000001},"last":0.90800000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.77600000000000002,"15min":0.81799999999999995},"min":{"1min":0.56000000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96499999999999997,"5min":1.1659999999999999,"15min":1.4139999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.80400000000000005,"15min":0.85599999999999998},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.94999999999999996,"5min":1.2010000000000001,"15min":1.593},"last":0.55800000000000005}]},{"osd":7,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.84399999999999997,"15min":0.88400000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.57299999999999995},"max":{"1min":1.004,"5min":1.1339999999999999,"15min":2.0539999999999998},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.83499999999999996,"15min":0.872},"min":{"1min":0.624,"5min":0.60799999999999998,"15min":0.55500000000000005},"max":{"1min":1.026,"5min":1.2270000000000001,"15min":1.2490000000000001},"last":0.52900000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575991,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304556,"kb_used_data":3172,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067284,"statfs":{"total":96636764160,"available":96324898816,"internally_reserved":0,"allocated":3248128,"data_stored":1569913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.48999999999999999,"15min":0.51400000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.621,"5min":0.89700000000000002,"15min":0.90000000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55000000000000004,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.38600000000000001}]},{"osd":1,"last update":"Thu Jan 11 09:46:42 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.48199999999999998,"15min":0.50900000000000001},"min":{"1min":0.34799999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.304},"last":0.443},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57799999999999996,"15min":0.60599999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":1.4590000000000001},"last":0.40600000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:46:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.56899999999999995,"15min":0.60299999999999998},"min":{"1min":0.42999999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.628,"15min":0.65500000000000003},"min":{"1min":0.45200000000000001,"5min":0.44800000000000001,"15min":0.44700000000000001},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.502}]},{"osd":4,"last update":"Thu Jan 11 09:46:42 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.74099999999999999,"15min":0.78500000000000003},"min":{"1min":0.502,"5min":0.442,"15min":0.442},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.76000000000000001,"15min":0.80600000000000005},"min":{"1min":0.57799999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.98999999999999999,"5min":1.171,"15min":1.1830000000000001},"last":0.70299999999999996}]},{"osd":5,"last update":"Thu Jan 11 09:46:42 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.77500000000000002,"15min":0.81299999999999994},"min":{"1min":0.56599999999999995,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.747},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.79500000000000004,"15min":0.84299999999999997},"min":{"1min":0.56100000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.3440000000000001},"last":0.84799999999999998}]},{"osd":6,"last update":"Thu Jan 11 09:46:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.81299999999999994,"15min":0.85299999999999998},"min":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.181,"5min":1.181,"15min":1.2170000000000001},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.79300000000000004,"15min":0.82499999999999996},"min":{"1min":0.57399999999999995,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.883,"5min":1.883,"15min":1.883},"last":0.78900000000000003}]},{"osd":7,"last update":"Thu Jan 11 09:46:42 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.82099999999999995,"15min":0.86299999999999999},"min":{"1min":0.54800000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.9319999999999999,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.83799999999999997,"15min":0.88800000000000001},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.877}]}]},{"osd":4,"up_from":13,"seq":55834575992,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312744,"kb_used_data":3168,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059096,"statfs":{"total":96636764160,"available":96316514304,"internally_reserved":0,"allocated":3244032,"data_stored":1568469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.72299999999999998,"15min":0.76000000000000001},"min":{"1min":0.42099999999999999,"5min":0.41199999999999998,"15min":0.39900000000000002},"max":{"1min":0.85799999999999998,"5min":0.97799999999999998,"15min":1.119},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.73799999999999999,"15min":0.76700000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86399999999999999,"5min":1.853,"15min":1.853},"last":0.70299999999999996}]},{"osd":1,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.69899999999999995,"15min":0.73499999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.83599999999999997,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.69799999999999995,"15min":0.72699999999999998},"min":{"1min":0.438,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81399999999999995,"5min":1.639,"15min":1.639},"last":0.72199999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.71999999999999997,"15min":0.746},"min":{"1min":0.50600000000000001,"5min":0.46500000000000002,"15min":0.44700000000000001},"max":{"1min":0.76400000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.71999999999999997,"15min":0.74199999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.93300000000000005,"5min":1.0249999999999999,"15min":1.0700000000000001},"last":0.60399999999999998}]},{"osd":3,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.70199999999999996,"15min":0.73299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.628},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.75900000000000001,"15min":0.77300000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.97499999999999998,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.81000000000000005}]},{"osd":5,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63500000000000001,"15min":0.623},"min":{"1min":0.41999999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.89900000000000002,"5min":0.91500000000000004,"15min":0.97199999999999998},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.66700000000000004,"15min":0.64700000000000002},"min":{"1min":0.45900000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95099999999999996,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.56799999999999995}]},{"osd":6,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.68999999999999995,"15min":0.67300000000000004},"min":{"1min":0.441,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.012,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.71899999999999997,"15min":0.70499999999999996},"min":{"1min":0.46000000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0609999999999999,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.754}]},{"osd":7,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.72899999999999998,"15min":0.70199999999999996},"min":{"1min":0.495,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.075,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.752,"5min":0.77900000000000003,"15min":0.754},"min":{"1min":0.56699999999999995,"5min":0.436,"15min":0.41699999999999998},"max":{"1min":1.0409999999999999,"5min":2.052,"15min":2.052},"last":0.73799999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575992,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312756,"kb_used_data":3180,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059084,"statfs":{"total":96636764160,"available":96316502016,"internally_reserved":0,"allocated":3256320,"data_stored":1570537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.71399999999999997,"15min":0.72999999999999998},"min":{"1min":0.42899999999999999,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.72599999999999998,"15min":0.72999999999999998},"min":{"1min":0.46400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.9900000000000002,"5min":2.9900000000000002,"15min":2.9900000000000002},"last":0.68700000000000006}]},{"osd":1,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.71799999999999997,"15min":0.73499999999999999},"min":{"1min":0.46000000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.873,"5min":0.75600000000000001,"15min":0.748},"min":{"1min":0.47699999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":5.4039999999999999,"5min":5.4039999999999999,"15min":5.4039999999999999},"last":0.69999999999999996}]},{"osd":2,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.54300000000000004,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.7120000000000002,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.627},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.73999999999999999,"15min":0.74299999999999999},"min":{"1min":0.51700000000000002,"5min":0.439,"15min":0.439},"max":{"1min":2.5099999999999998,"5min":2.5099999999999998,"15min":2.5099999999999998},"last":0.76900000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.745,"15min":0.75700000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.7799999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":0.752},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.79900000000000004,"15min":0.78300000000000003},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":5.5570000000000004,"5min":5.5570000000000004,"15min":5.5570000000000004},"last":0.73399999999999999}]},{"osd":4,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.64600000000000002,"15min":0.61699999999999999},"min":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41699999999999998},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.66500000000000004,"15min":0.63800000000000001},"min":{"1min":0.40400000000000003,"5min":0.40200000000000002,"15min":0.375},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":0.64600000000000002}]},{"osd":6,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.67400000000000004,"15min":0.65000000000000002},"min":{"1min":0.42899999999999999,"5min":0.42099999999999999,"15min":0.40999999999999998},"max":{"1min":2.9500000000000002,"5min":2.9500000000000002,"15min":2.9500000000000002},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.6280000000000001,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":0.71699999999999997}]},{"osd":7,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.748,"15min":0.73199999999999998},"min":{"1min":0.52700000000000002,"5min":0.47299999999999998,"15min":0.434},"max":{"1min":2.399,"5min":2.399,"15min":2.399},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.80400000000000005,"15min":0.78200000000000003},"min":{"1min":0.61399999999999999,"5min":0.48799999999999999,"15min":0.47599999999999998},"max":{"1min":2.7570000000000001,"5min":2.7570000000000001,"15min":2.7570000000000001},"last":0.84099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:47:29.384 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:47:29.551 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-01-11T09:47:29.560+0000 7feaee9f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T09:47:29.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.626+0000 7f3d2f5ff700 1 Processor -- start 2024-01-11T09:47:29.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.626+0000 7f3d2f5ff700 1 -- start start 2024-01-11T09:47:29.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.627+0000 7f3d2f5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d280b2b90 0x7f3d28159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:29.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.627+0000 7f3d2f5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3d280b07b0 0x7f3d28159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:29.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.627+0000 7f3d2f5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d280cd730 con 0x7f3d280b03e0 2024-01-11T09:47:29.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.627+0000 7f3d2f5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d280ccb20 con 0x7f3d280b2b90 2024-01-11T09:47:29.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.627+0000 7f3d2f5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d280cce00 con 0x7f3d280b07b0 2024-01-11T09:47:29.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.627+0000 7f3d2cb9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d280b2b90 0x7f3d28159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:29.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.627+0000 7f3d2d39b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3d280b03e0 0x7f3d280b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51286/0 (socket says 172.21.15.27:51286) 2024-01-11T09:47:29.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.627+0000 7f3d2d39b700 1 -- 172.21.15.27:0/3445237291 learned_addr learned my addr 172.21.15.27:0/3445237291 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:29.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.627+0000 7f3d2db9c700 1 --2- 172.21.15.27:0/3445237291 >> v2:172.21.15.77:3300/0 conn(0x7f3d280b07b0 0x7f3d28159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:29.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.628+0000 7f3d2cb9a700 1 -- 172.21.15.27:0/3445237291 >> v2:172.21.15.77:3300/0 conn(0x7f3d280b07b0 msgr2=0x7f3d28159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:29.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.628+0000 7f3d2cb9a700 1 --2- 172.21.15.27:0/3445237291 >> v2:172.21.15.77:3300/0 conn(0x7f3d280b07b0 0x7f3d28159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.628+0000 7f3d2cb9a700 1 -- 172.21.15.27:0/3445237291 >> v1:172.21.15.27:6789/0 conn(0x7f3d280b03e0 legacy=0x7f3d280b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:29.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.628+0000 7f3d2cb9a700 1 -- 172.21.15.27:0/3445237291 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d280cd0e0 con 0x7f3d280b2b90 2024-01-11T09:47:29.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.628+0000 7f3d2cb9a700 1 --2- 172.21.15.27:0/3445237291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d280b2b90 0x7f3d28159200 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f3d280b1d10 tx=0x7f3d180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60a78b89acd96ae server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:29.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.628+0000 7f3d1ffff700 1 -- 172.21.15.27:0/3445237291 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d18011630 con 0x7f3d280b2b90 2024-01-11T09:47:29.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.629+0000 7f3d1ffff700 1 -- 172.21.15.27:0/3445237291 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d18011790 con 0x7f3d280b2b90 2024-01-11T09:47:29.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.629+0000 7f3d1ffff700 1 -- 172.21.15.27:0/3445237291 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d18011b30 con 0x7f3d280b2b90 2024-01-11T09:47:29.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.629+0000 7f3d2f5ff700 1 -- 172.21.15.27:0/3445237291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d280b2b90 msgr2=0x7f3d28159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:29.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.629+0000 7f3d2f5ff700 1 --2- 172.21.15.27:0/3445237291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d280b2b90 0x7f3d28159200 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f3d280b1d10 tx=0x7f3d180091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.629+0000 7f3d2f5ff700 1 -- 172.21.15.27:0/3445237291 shutdown_connections 2024-01-11T09:47:29.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.629+0000 7f3d2f5ff700 1 --2- 172.21.15.27:0/3445237291 >> v2:172.21.15.77:3300/0 conn(0x7f3d280b07b0 0x7f3d28159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.629+0000 7f3d2f5ff700 1 --2- 172.21.15.27:0/3445237291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d280b2b90 0x7f3d28159200 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.629+0000 7f3d2f5ff700 1 -- 172.21.15.27:0/3445237291 >> 172.21.15.27:0/3445237291 conn(0x7f3d2806ed40 msgr2=0x7f3d2806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:29.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.629+0000 7f3d2f5ff700 1 -- 172.21.15.27:0/3445237291 shutdown_connections 2024-01-11T09:47:29.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.629+0000 7f3d2f5ff700 1 -- 172.21.15.27:0/3445237291 wait complete. 2024-01-11T09:47:29.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.631+0000 7f3d2f5ff700 1 Processor -- start 2024-01-11T09:47:29.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.631+0000 7f3d2f5ff700 1 -- start start 2024-01-11T09:47:29.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.632+0000 7f3d2f5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d280b03e0 0x7f3d281589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:29.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.632+0000 7f3d2f5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3d280b07b0 0x7f3d280c7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:29.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.632+0000 7f3d2f5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d280b2b90 0x7f3d28169f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.632+0000 7f3d2f5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d280a70e0 con 0x7f3d280b2b90 2024-01-11T09:47:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.632+0000 7f3d2f5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2816a9c0 con 0x7f3d280b03e0 2024-01-11T09:47:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.632+0000 7f3d2f5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2816aca0 con 0x7f3d280b07b0 2024-01-11T09:47:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.632+0000 7f3d2db9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d280b2b90 0x7f3d28169f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.632+0000 7f3d2db9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d280b2b90 0x7f3d28169f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56206/0 (socket says 172.21.15.27:56206) 2024-01-11T09:47:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.632+0000 7f3d2db9c700 1 -- 172.21.15.27:0/4279376375 learned_addr learned my addr 172.21.15.27:0/4279376375 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.632+0000 7f3d2d39b700 1 --2- 172.21.15.27:0/4279376375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d280b03e0 0x7f3d281589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.632+0000 7f3d2cb9a700 1 --2- 172.21.15.27:0/4279376375 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3d280b07b0 0x7f3d280c7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.633+0000 7f3d2db9c700 1 -- 172.21.15.27:0/4279376375 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3d280b07b0 msgr2=0x7f3d280c7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.633+0000 7f3d2db9c700 1 --2- 172.21.15.27:0/4279376375 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3d280b07b0 0x7f3d280c7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.633+0000 7f3d2db9c700 1 -- 172.21.15.27:0/4279376375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d280b03e0 msgr2=0x7f3d281589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.633+0000 7f3d2db9c700 1 --2- 172.21.15.27:0/4279376375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d280b03e0 0x7f3d281589b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.633+0000 7f3d2db9c700 1 -- 172.21.15.27:0/4279376375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d18009420 con 0x7f3d280b2b90 2024-01-11T09:47:29.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.633+0000 7f3d2cb9a700 1 --2- 172.21.15.27:0/4279376375 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3d280b07b0 0x7f3d280c7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:29.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.633+0000 7f3d2d39b700 1 --2- 172.21.15.27:0/4279376375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d280b03e0 0x7f3d281589b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:29.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.633+0000 7f3d2db9c700 1 --2- 172.21.15.27:0/4279376375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d280b2b90 0x7f3d28169f20 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f3d20009e90 tx=0x7f3d2000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:29.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.633+0000 7f3d1d7fa700 1 -- 172.21.15.27:0/4279376375 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2000cc20 con 0x7f3d280b2b90 2024-01-11T09:47:29.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.633+0000 7f3d1d7fa700 1 -- 172.21.15.27:0/4279376375 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2000f040 con 0x7f3d280b2b90 2024-01-11T09:47:29.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.633+0000 7f3d1d7fa700 1 -- 172.21.15.27:0/4279376375 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20014480 con 0x7f3d280b2b90 2024-01-11T09:47:29.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.633+0000 7f3d2f5ff700 1 -- 172.21.15.27:0/4279376375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d280c7760 con 0x7f3d280b2b90 2024-01-11T09:47:29.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.633+0000 7f3d2f5ff700 1 -- 172.21.15.27:0/4279376375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d28158ef0 con 0x7f3d280b2b90 2024-01-11T09:47:29.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.634+0000 7f3d2f5ff700 1 -- 172.21.15.27:0/4279376375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d280c75c0 con 0x7f3d280b2b90 2024-01-11T09:47:29.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.635+0000 7f3d1d7fa700 1 -- 172.21.15.27:0/4279376375 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f3d20014830 con 0x7f3d280b2b90 2024-01-11T09:47:29.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.636+0000 7f3d1d7fa700 1 --2- 172.21.15.27:0/4279376375 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3d0c067170 0x7f3d0c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:29.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.636+0000 7f3d1d7fa700 1 -- 172.21.15.27:0/4279376375 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f3d200854d0 con 0x7f3d280b2b90 2024-01-11T09:47:29.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.637+0000 7f3d2d39b700 1 --2- 172.21.15.27:0/4279376375 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3d0c067170 0x7f3d0c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:29.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.637+0000 7f3d2d39b700 1 --2- 172.21.15.27:0/4279376375 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3d0c067170 0x7f3d0c069620 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3d24007fd0 tx=0x7f3d24009040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:29.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.640+0000 7f3d1d7fa700 1 -- 172.21.15.27:0/4279376375 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3d280c75c0 con 0x7f3d280b2b90 2024-01-11T09:47:29.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.840+0000 7f3d2f5ff700 1 -- 172.21.15.27:0/4279376375 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3d280c75c0 con 0x7f3d0c067170 2024-01-11T09:47:29.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.849+0000 7f3d1d7fa700 1 -- 172.21.15.27:0/4279376375 <== mgr.9641 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289550 (secure 0 0 0) 0x7f3d280c75c0 con 0x7f3d0c067170 2024-01-11T09:47:29.845 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:29.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.854+0000 7f3d2f5ff700 1 -- 172.21.15.27:0/4279376375 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3d0c067170 msgr2=0x7f3d0c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:29.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.854+0000 7f3d2f5ff700 1 --2- 172.21.15.27:0/4279376375 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3d0c067170 0x7f3d0c069620 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3d24007fd0 tx=0x7f3d24009040 comp rx=0 tx=0).stop 2024-01-11T09:47:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.854+0000 7f3d2f5ff700 1 -- 172.21.15.27:0/4279376375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d280b2b90 msgr2=0x7f3d28169f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.854+0000 7f3d2f5ff700 1 --2- 172.21.15.27:0/4279376375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d280b2b90 0x7f3d28169f20 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f3d20009e90 tx=0x7f3d2000b040 comp rx=0 tx=0).stop 2024-01-11T09:47:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.854+0000 7f3d2f5ff700 1 -- 172.21.15.27:0/4279376375 shutdown_connections 2024-01-11T09:47:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.854+0000 7f3d2f5ff700 1 --2- 172.21.15.27:0/4279376375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d280b2b90 0x7f3d28169f20 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.854+0000 7f3d2f5ff700 1 --2- 172.21.15.27:0/4279376375 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3d0c067170 0x7f3d0c069620 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.854+0000 7f3d2f5ff700 1 --2- 172.21.15.27:0/4279376375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d280b03e0 0x7f3d281589b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.854+0000 7f3d2f5ff700 1 --2- 172.21.15.27:0/4279376375 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3d280b07b0 0x7f3d280c7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.854+0000 7f3d2f5ff700 1 -- 172.21.15.27:0/4279376375 >> 172.21.15.27:0/4279376375 conn(0x7f3d2806ed40 msgr2=0x7f3d280b4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.854+0000 7f3d2f5ff700 1 -- 172.21.15.27:0/4279376375 shutdown_connections 2024-01-11T09:47:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:29.854+0000 7f3d2f5ff700 1 -- 172.21.15.27:0/4279376375 wait complete. 2024-01-11T09:47:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:47:29.863 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-01-11T09:47:28.523570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2654,"num_read_kb":5617,"num_write":731,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":446,"ondisk_log_size":446,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2466248,"kb_used_data":26504,"kb_used_omap":0,"kb_used_meta":2439680,"kb_avail":752508472,"statfs":{"total":773094113280,"available":770568675328,"internally_reserved":0,"allocated":27140096,"data_stored":13732364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2498232320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.565826"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.008397+0000","last_change":"2024-01-11T09:47:24.997406+0000","last_active":"2024-01-11T09:47:25.008397+0000","last_peered":"2024-01-11T09:47:25.008397+0000","last_clean":"2024-01-11T09:47:25.008397+0000","last_became_active":"2024-01-11T09:47:24.997138+0000","last_became_peered":"2024-01-11T09:47:24.997138+0000","last_unstale":"2024-01-11T09:47:25.008397+0000","last_undegraded":"2024-01-11T09:47:25.008397+0000","last_fullsized":"2024-01-11T09:47:25.008397+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:39:31.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.981933+0000","last_change":"2024-01-11T09:47:24.975616+0000","last_active":"2024-01-11T09:47:24.981933+0000","last_peered":"2024-01-11T09:47:24.981933+0000","last_clean":"2024-01-11T09:47:24.981933+0000","last_became_active":"2024-01-11T09:47:24.975334+0000","last_became_peered":"2024-01-11T09:47:24.975334+0000","last_unstale":"2024-01-11T09:47:24.981933+0000","last_undegraded":"2024-01-11T09:47:24.981933+0000","last_fullsized":"2024-01-11T09:47:24.981933+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:42:00.528441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004226+0000","last_change":"2024-01-11T09:47:24.999455+0000","last_active":"2024-01-11T09:47:25.004226+0000","last_peered":"2024-01-11T09:47:25.004226+0000","last_clean":"2024-01-11T09:47:25.004226+0000","last_became_active":"2024-01-11T09:47:24.999172+0000","last_became_peered":"2024-01-11T09:47:24.999172+0000","last_unstale":"2024-01-11T09:47:25.004226+0000","last_undegraded":"2024-01-11T09:47:25.004226+0000","last_fullsized":"2024-01-11T09:47:25.004226+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:39:08.765597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.000678+0000","last_change":"2024-01-11T09:47:24.991535+0000","last_active":"2024-01-11T09:47:25.000678+0000","last_peered":"2024-01-11T09:47:25.000678+0000","last_clean":"2024-01-11T09:47:25.000678+0000","last_became_active":"2024-01-11T09:47:24.991241+0000","last_became_peered":"2024-01-11T09:47:24.991241+0000","last_unstale":"2024-01-11T09:47:25.000678+0000","last_undegraded":"2024-01-11T09:47:25.000678+0000","last_fullsized":"2024-01-11T09:47:25.000678+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:10:02.432500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.981901+0000","last_change":"2024-01-11T09:47:24.977175+0000","last_active":"2024-01-11T09:47:24.981901+0000","last_peered":"2024-01-11T09:47:24.981901+0000","last_clean":"2024-01-11T09:47:24.981901+0000","last_became_active":"2024-01-11T09:47:24.976914+0000","last_became_peered":"2024-01-11T09:47:24.976914+0000","last_unstale":"2024-01-11T09:47:24.981901+0000","last_undegraded":"2024-01-11T09:47:24.981901+0000","last_fullsized":"2024-01-11T09:47:24.981901+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:27:48.271801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.981846+0000","last_change":"2024-01-11T09:47:24.976207+0000","last_active":"2024-01-11T09:47:24.981846+0000","last_peered":"2024-01-11T09:47:24.981846+0000","last_clean":"2024-01-11T09:47:24.981846+0000","last_became_active":"2024-01-11T09:47:24.975903+0000","last_became_peered":"2024-01-11T09:47:24.975903+0000","last_unstale":"2024-01-11T09:47:24.981846+0000","last_undegraded":"2024-01-11T09:47:24.981846+0000","last_fullsized":"2024-01-11T09:47:24.981846+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:17:43.512711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.996431+0000","last_change":"2024-01-11T09:47:24.993021+0000","last_active":"2024-01-11T09:47:24.996431+0000","last_peered":"2024-01-11T09:47:24.996431+0000","last_clean":"2024-01-11T09:47:24.996431+0000","last_became_active":"2024-01-11T09:47:24.992704+0000","last_became_peered":"2024-01-11T09:47:24.992704+0000","last_unstale":"2024-01-11T09:47:24.996431+0000","last_undegraded":"2024-01-11T09:47:24.996431+0000","last_fullsized":"2024-01-11T09:47:24.996431+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:16:26.902792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002366+0000","last_change":"2024-01-11T09:47:24.998877+0000","last_active":"2024-01-11T09:47:25.002366+0000","last_peered":"2024-01-11T09:47:25.002366+0000","last_clean":"2024-01-11T09:47:25.002366+0000","last_became_active":"2024-01-11T09:47:24.998622+0000","last_became_peered":"2024-01-11T09:47:24.998622+0000","last_unstale":"2024-01-11T09:47:25.002366+0000","last_undegraded":"2024-01-11T09:47:25.002366+0000","last_fullsized":"2024-01-11T09:47:25.002366+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:22:49.439958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.001924+0000","last_change":"2024-01-11T09:47:24.994645+0000","last_active":"2024-01-11T09:47:25.001924+0000","last_peered":"2024-01-11T09:47:25.001924+0000","last_clean":"2024-01-11T09:47:25.001924+0000","last_became_active":"2024-01-11T09:47:24.994365+0000","last_became_peered":"2024-01-11T09:47:24.994365+0000","last_unstale":"2024-01-11T09:47:25.001924+0000","last_undegraded":"2024-01-11T09:47:25.001924+0000","last_fullsized":"2024-01-11T09:47:25.001924+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:33:56.394147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002854+0000","last_change":"2024-01-11T09:47:24.996859+0000","last_active":"2024-01-11T09:47:25.002854+0000","last_peered":"2024-01-11T09:47:25.002854+0000","last_clean":"2024-01-11T09:47:25.002854+0000","last_became_active":"2024-01-11T09:47:24.996616+0000","last_became_peered":"2024-01-11T09:47:24.996616+0000","last_unstale":"2024-01-11T09:47:25.002854+0000","last_undegraded":"2024-01-11T09:47:25.002854+0000","last_fullsized":"2024-01-11T09:47:25.002854+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:02:36.455725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.007468+0000","last_change":"2024-01-11T09:47:25.003959+0000","last_active":"2024-01-11T09:47:25.007468+0000","last_peered":"2024-01-11T09:47:25.007468+0000","last_clean":"2024-01-11T09:47:25.007468+0000","last_became_active":"2024-01-11T09:47:25.003688+0000","last_became_peered":"2024-01-11T09:47:25.003688+0000","last_unstale":"2024-01-11T09:47:25.007468+0000","last_undegraded":"2024-01-11T09:47:25.007468+0000","last_fullsized":"2024-01-11T09:47:25.007468+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:49:32.683244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.001960+0000","last_change":"2024-01-11T09:47:24.996977+0000","last_active":"2024-01-11T09:47:25.001960+0000","last_peered":"2024-01-11T09:47:25.001960+0000","last_clean":"2024-01-11T09:47:25.001960+0000","last_became_active":"2024-01-11T09:47:24.995847+0000","last_became_peered":"2024-01-11T09:47:24.995847+0000","last_unstale":"2024-01-11T09:47:25.001960+0000","last_undegraded":"2024-01-11T09:47:25.001960+0000","last_fullsized":"2024-01-11T09:47:25.001960+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:16:51.696701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.009302+0000","last_change":"2024-01-11T09:47:24.999731+0000","last_active":"2024-01-11T09:47:25.009302+0000","last_peered":"2024-01-11T09:47:25.009302+0000","last_clean":"2024-01-11T09:47:25.009302+0000","last_became_active":"2024-01-11T09:47:24.999446+0000","last_became_peered":"2024-01-11T09:47:24.999446+0000","last_unstale":"2024-01-11T09:47:25.009302+0000","last_undegraded":"2024-01-11T09:47:25.009302+0000","last_fullsized":"2024-01-11T09:47:25.009302+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:39:30.172237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004026+0000","last_change":"2024-01-11T09:47:24.993445+0000","last_active":"2024-01-11T09:47:25.004026+0000","last_peered":"2024-01-11T09:47:25.004026+0000","last_clean":"2024-01-11T09:47:25.004026+0000","last_became_active":"2024-01-11T09:47:24.993127+0000","last_became_peered":"2024-01-11T09:47:24.993127+0000","last_unstale":"2024-01-11T09:47:25.004026+0000","last_undegraded":"2024-01-11T09:47:25.004026+0000","last_fullsized":"2024-01-11T09:47:25.004026+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:11:13.244226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.996408+0000","last_change":"2024-01-11T09:47:24.991126+0000","last_active":"2024-01-11T09:47:24.996408+0000","last_peered":"2024-01-11T09:47:24.996408+0000","last_clean":"2024-01-11T09:47:24.996408+0000","last_became_active":"2024-01-11T09:47:24.990346+0000","last_became_peered":"2024-01-11T09:47:24.990346+0000","last_unstale":"2024-01-11T09:47:24.996408+0000","last_undegraded":"2024-01-11T09:47:24.996408+0000","last_fullsized":"2024-01-11T09:47:24.996408+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:03:35.424742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.999691+0000","last_change":"2024-01-11T09:47:24.995534+0000","last_active":"2024-01-11T09:47:24.999691+0000","last_peered":"2024-01-11T09:47:24.999691+0000","last_clean":"2024-01-11T09:47:24.999691+0000","last_became_active":"2024-01-11T09:47:24.995260+0000","last_became_peered":"2024-01-11T09:47:24.995260+0000","last_unstale":"2024-01-11T09:47:24.999691+0000","last_undegraded":"2024-01-11T09:47:24.999691+0000","last_fullsized":"2024-01-11T09:47:24.999691+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:26:00.307598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.999466+0000","last_change":"2024-01-11T09:47:24.989635+0000","last_active":"2024-01-11T09:47:24.999466+0000","last_peered":"2024-01-11T09:47:24.999466+0000","last_clean":"2024-01-11T09:47:24.999466+0000","last_became_active":"2024-01-11T09:47:24.989278+0000","last_became_peered":"2024-01-11T09:47:24.989278+0000","last_unstale":"2024-01-11T09:47:24.999466+0000","last_undegraded":"2024-01-11T09:47:24.999466+0000","last_fullsized":"2024-01-11T09:47:24.999466+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:36.344385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002152+0000","last_change":"2024-01-11T09:47:24.995164+0000","last_active":"2024-01-11T09:47:25.002152+0000","last_peered":"2024-01-11T09:47:25.002152+0000","last_clean":"2024-01-11T09:47:25.002152+0000","last_became_active":"2024-01-11T09:47:24.994859+0000","last_became_peered":"2024-01-11T09:47:24.994859+0000","last_unstale":"2024-01-11T09:47:25.002152+0000","last_undegraded":"2024-01-11T09:47:25.002152+0000","last_fullsized":"2024-01-11T09:47:25.002152+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:40:26.192216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.007663+0000","last_change":"2024-01-11T09:47:25.003579+0000","last_active":"2024-01-11T09:47:25.007663+0000","last_peered":"2024-01-11T09:47:25.007663+0000","last_clean":"2024-01-11T09:47:25.007663+0000","last_became_active":"2024-01-11T09:47:25.002974+0000","last_became_peered":"2024-01-11T09:47:25.002974+0000","last_unstale":"2024-01-11T09:47:25.007663+0000","last_undegraded":"2024-01-11T09:47:25.007663+0000","last_fullsized":"2024-01-11T09:47:25.007663+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:20:42.501759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002743+0000","last_change":"2024-01-11T09:47:24.999637+0000","last_active":"2024-01-11T09:47:25.002743+0000","last_peered":"2024-01-11T09:47:25.002743+0000","last_clean":"2024-01-11T09:47:25.002743+0000","last_became_active":"2024-01-11T09:47:24.999343+0000","last_became_peered":"2024-01-11T09:47:24.999343+0000","last_unstale":"2024-01-11T09:47:25.002743+0000","last_undegraded":"2024-01-11T09:47:25.002743+0000","last_fullsized":"2024-01-11T09:47:25.002743+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:56:49.891176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.009324+0000","last_change":"2024-01-11T09:47:25.000182+0000","last_active":"2024-01-11T09:47:25.009324+0000","last_peered":"2024-01-11T09:47:25.009324+0000","last_clean":"2024-01-11T09:47:25.009324+0000","last_became_active":"2024-01-11T09:47:24.999929+0000","last_became_peered":"2024-01-11T09:47:24.999929+0000","last_unstale":"2024-01-11T09:47:25.009324+0000","last_undegraded":"2024-01-11T09:47:25.009324+0000","last_fullsized":"2024-01-11T09:47:25.009324+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:00:22.287610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.994002+0000","last_change":"2024-01-11T09:47:24.989120+0000","last_active":"2024-01-11T09:47:24.994002+0000","last_peered":"2024-01-11T09:47:24.994002+0000","last_clean":"2024-01-11T09:47:24.994002+0000","last_became_active":"2024-01-11T09:47:24.988849+0000","last_became_peered":"2024-01-11T09:47:24.988849+0000","last_unstale":"2024-01-11T09:47:24.994002+0000","last_undegraded":"2024-01-11T09:47:24.994002+0000","last_fullsized":"2024-01-11T09:47:24.994002+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:21:35.094425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.993982+0000","last_change":"2024-01-11T09:47:24.990469+0000","last_active":"2024-01-11T09:47:24.993982+0000","last_peered":"2024-01-11T09:47:24.993982+0000","last_clean":"2024-01-11T09:47:24.993982+0000","last_became_active":"2024-01-11T09:47:24.990025+0000","last_became_peered":"2024-01-11T09:47:24.990025+0000","last_unstale":"2024-01-11T09:47:24.993982+0000","last_undegraded":"2024-01-11T09:47:24.993982+0000","last_fullsized":"2024-01-11T09:47:24.993982+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:01:36.213583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.000890+0000","last_change":"2024-01-11T09:47:24.992429+0000","last_active":"2024-01-11T09:47:25.000890+0000","last_peered":"2024-01-11T09:47:25.000890+0000","last_clean":"2024-01-11T09:47:25.000890+0000","last_became_active":"2024-01-11T09:47:24.992117+0000","last_became_peered":"2024-01-11T09:47:24.992117+0000","last_unstale":"2024-01-11T09:47:25.000890+0000","last_undegraded":"2024-01-11T09:47:25.000890+0000","last_fullsized":"2024-01-11T09:47:25.000890+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:46:52.592447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.993814+0000","last_change":"2024-01-11T09:47:24.988252+0000","last_active":"2024-01-11T09:47:24.993814+0000","last_peered":"2024-01-11T09:47:24.993814+0000","last_clean":"2024-01-11T09:47:24.993814+0000","last_became_active":"2024-01-11T09:47:24.987962+0000","last_became_peered":"2024-01-11T09:47:24.987962+0000","last_unstale":"2024-01-11T09:47:24.993814+0000","last_undegraded":"2024-01-11T09:47:24.993814+0000","last_fullsized":"2024-01-11T09:47:24.993814+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:11:13.244226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.001890+0000","last_change":"2024-01-11T09:47:24.993862+0000","last_active":"2024-01-11T09:47:25.001890+0000","last_peered":"2024-01-11T09:47:25.001890+0000","last_clean":"2024-01-11T09:47:25.001890+0000","last_became_active":"2024-01-11T09:47:24.992925+0000","last_became_peered":"2024-01-11T09:47:24.992925+0000","last_unstale":"2024-01-11T09:47:25.001890+0000","last_undegraded":"2024-01-11T09:47:25.001890+0000","last_fullsized":"2024-01-11T09:47:25.001890+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:45:09.788829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.990490+0000","last_change":"2024-01-11T09:47:24.989603+0000","last_active":"2024-01-11T09:47:24.990490+0000","last_peered":"2024-01-11T09:47:24.990490+0000","last_clean":"2024-01-11T09:47:24.990490+0000","last_became_active":"2024-01-11T09:47:24.988816+0000","last_became_peered":"2024-01-11T09:47:24.988816+0000","last_unstale":"2024-01-11T09:47:24.990490+0000","last_undegraded":"2024-01-11T09:47:24.990490+0000","last_fullsized":"2024-01-11T09:47:24.990490+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:46:52.592447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.988647+0000","last_change":"2024-01-11T09:47:24.983913+0000","last_active":"2024-01-11T09:47:24.988647+0000","last_peered":"2024-01-11T09:47:24.988647+0000","last_clean":"2024-01-11T09:47:24.988647+0000","last_became_active":"2024-01-11T09:47:24.983628+0000","last_became_peered":"2024-01-11T09:47:24.983628+0000","last_unstale":"2024-01-11T09:47:24.988647+0000","last_undegraded":"2024-01-11T09:47:24.988647+0000","last_fullsized":"2024-01-11T09:47:24.988647+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:05:16.612544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.990474+0000","last_change":"2024-01-11T09:47:24.989376+0000","last_active":"2024-01-11T09:47:24.990474+0000","last_peered":"2024-01-11T09:47:24.990474+0000","last_clean":"2024-01-11T09:47:24.990474+0000","last_became_active":"2024-01-11T09:47:24.989054+0000","last_became_peered":"2024-01-11T09:47:24.989054+0000","last_unstale":"2024-01-11T09:47:24.990474+0000","last_undegraded":"2024-01-11T09:47:24.990474+0000","last_fullsized":"2024-01-11T09:47:24.990474+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:05:40.698838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.991214+0000","last_change":"2024-01-11T09:47:24.988736+0000","last_active":"2024-01-11T09:47:24.991214+0000","last_peered":"2024-01-11T09:47:24.991214+0000","last_clean":"2024-01-11T09:47:24.991214+0000","last_became_active":"2024-01-11T09:47:24.988477+0000","last_became_peered":"2024-01-11T09:47:24.988477+0000","last_unstale":"2024-01-11T09:47:24.991214+0000","last_undegraded":"2024-01-11T09:47:24.991214+0000","last_fullsized":"2024-01-11T09:47:24.991214+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:36:36.349947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.997906+0000","last_change":"2024-01-11T09:47:24.989526+0000","last_active":"2024-01-11T09:47:24.997906+0000","last_peered":"2024-01-11T09:47:24.997906+0000","last_clean":"2024-01-11T09:47:24.997906+0000","last_became_active":"2024-01-11T09:47:24.989259+0000","last_became_peered":"2024-01-11T09:47:24.989259+0000","last_unstale":"2024-01-11T09:47:24.997906+0000","last_undegraded":"2024-01-11T09:47:24.997906+0000","last_fullsized":"2024-01-11T09:47:24.997906+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:36:28.970823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.997807+0000","last_change":"2024-01-11T09:47:24.987338+0000","last_active":"2024-01-11T09:47:24.997807+0000","last_peered":"2024-01-11T09:47:24.997807+0000","last_clean":"2024-01-11T09:47:24.997807+0000","last_became_active":"2024-01-11T09:47:24.987071+0000","last_became_peered":"2024-01-11T09:47:24.987071+0000","last_unstale":"2024-01-11T09:47:24.997807+0000","last_undegraded":"2024-01-11T09:47:24.997807+0000","last_fullsized":"2024-01-11T09:47:24.997807+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:53:51.475245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1f","version":"255'4","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.008715+0000","last_change":"2024-01-11T09:47:23.989599+0000","last_active":"2024-01-11T09:47:25.008715+0000","last_peered":"2024-01-11T09:47:25.008715+0000","last_clean":"2024-01-11T09:47:25.008715+0000","last_became_active":"2024-01-11T09:47:23.989185+0000","last_became_peered":"2024-01-11T09:47:23.989185+0000","last_unstale":"2024-01-11T09:47:25.008715+0000","last_undegraded":"2024-01-11T09:47:25.008715+0000","last_fullsized":"2024-01-11T09:47:25.008715+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:08:15.232479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004946+0000","last_change":"2024-01-11T09:47:25.002124+0000","last_active":"2024-01-11T09:47:25.004946+0000","last_peered":"2024-01-11T09:47:25.004946+0000","last_clean":"2024-01-11T09:47:25.004946+0000","last_became_active":"2024-01-11T09:47:25.001876+0000","last_became_peered":"2024-01-11T09:47:25.001876+0000","last_unstale":"2024-01-11T09:47:25.004946+0000","last_undegraded":"2024-01-11T09:47:25.004946+0000","last_fullsized":"2024-01-11T09:47:25.004946+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:33:54.313596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1e","version":"255'1","reported_seq":11,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004936+0000","last_change":"2024-01-11T09:47:23.983332+0000","last_active":"2024-01-11T09:47:25.004936+0000","last_peered":"2024-01-11T09:47:25.004936+0000","last_clean":"2024-01-11T09:47:25.004936+0000","last_became_active":"2024-01-11T09:47:23.983024+0000","last_became_peered":"2024-01-11T09:47:23.983024+0000","last_unstale":"2024-01-11T09:47:25.004936+0000","last_undegraded":"2024-01-11T09:47:25.004936+0000","last_fullsized":"2024-01-11T09:47:25.004936+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:50:00.125829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.992894+0000","last_change":"2024-01-11T09:47:24.987265+0000","last_active":"2024-01-11T09:47:24.992894+0000","last_peered":"2024-01-11T09:47:24.992894+0000","last_clean":"2024-01-11T09:47:24.992894+0000","last_became_active":"2024-01-11T09:47:24.983227+0000","last_became_peered":"2024-01-11T09:47:24.983227+0000","last_unstale":"2024-01-11T09:47:24.992894+0000","last_undegraded":"2024-01-11T09:47:24.992894+0000","last_fullsized":"2024-01-11T09:47:24.992894+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:45:09.788829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1d","version":"255'2","reported_seq":12,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.005801+0000","last_change":"2024-01-11T09:47:23.980815+0000","last_active":"2024-01-11T09:47:25.005801+0000","last_peered":"2024-01-11T09:47:25.005801+0000","last_clean":"2024-01-11T09:47:25.005801+0000","last_became_active":"2024-01-11T09:47:23.977600+0000","last_became_peered":"2024-01-11T09:47:23.977600+0000","last_unstale":"2024-01-11T09:47:25.005801+0000","last_undegraded":"2024-01-11T09:47:25.005801+0000","last_fullsized":"2024-01-11T09:47:25.005801+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:48:33.759315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002362+0000","last_change":"2024-01-11T09:47:24.998029+0000","last_active":"2024-01-11T09:47:25.002362+0000","last_peered":"2024-01-11T09:47:25.002362+0000","last_clean":"2024-01-11T09:47:25.002362+0000","last_became_active":"2024-01-11T09:47:24.995913+0000","last_became_peered":"2024-01-11T09:47:24.995913+0000","last_unstale":"2024-01-11T09:47:25.002362+0000","last_undegraded":"2024-01-11T09:47:25.002362+0000","last_fullsized":"2024-01-11T09:47:25.002362+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:39:31.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"255'2","reported_seq":12,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.005331+0000","last_change":"2024-01-11T09:47:23.987256+0000","last_active":"2024-01-11T09:47:25.005331+0000","last_peered":"2024-01-11T09:47:25.005331+0000","last_clean":"2024-01-11T09:47:25.005331+0000","last_became_active":"2024-01-11T09:47:23.984657+0000","last_became_peered":"2024-01-11T09:47:23.984657+0000","last_unstale":"2024-01-11T09:47:25.005331+0000","last_undegraded":"2024-01-11T09:47:25.005331+0000","last_fullsized":"2024-01-11T09:47:25.005331+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:20:42.021716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.991859+0000","last_change":"2024-01-11T09:47:24.985214+0000","last_active":"2024-01-11T09:47:24.991859+0000","last_peered":"2024-01-11T09:47:24.991859+0000","last_clean":"2024-01-11T09:47:24.991859+0000","last_became_active":"2024-01-11T09:47:24.983021+0000","last_became_peered":"2024-01-11T09:47:24.983021+0000","last_unstale":"2024-01-11T09:47:24.991859+0000","last_undegraded":"2024-01-11T09:47:24.991859+0000","last_fullsized":"2024-01-11T09:47:24.991859+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:40.944331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1b","version":"255'1","reported_seq":11,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004527+0000","last_change":"2024-01-11T09:47:23.993016+0000","last_active":"2024-01-11T09:47:25.004527+0000","last_peered":"2024-01-11T09:47:25.004527+0000","last_clean":"2024-01-11T09:47:25.004527+0000","last_became_active":"2024-01-11T09:47:23.992681+0000","last_became_peered":"2024-01-11T09:47:23.992681+0000","last_unstale":"2024-01-11T09:47:25.004527+0000","last_undegraded":"2024-01-11T09:47:25.004527+0000","last_fullsized":"2024-01-11T09:47:25.004527+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:54:08.481155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.999814+0000","last_change":"2024-01-11T09:47:24.995779+0000","last_active":"2024-01-11T09:47:24.999814+0000","last_peered":"2024-01-11T09:47:24.999814+0000","last_clean":"2024-01-11T09:47:24.999814+0000","last_became_active":"2024-01-11T09:47:24.995465+0000","last_became_peered":"2024-01-11T09:47:24.995465+0000","last_unstale":"2024-01-11T09:47:24.999814+0000","last_undegraded":"2024-01-11T09:47:24.999814+0000","last_fullsized":"2024-01-11T09:47:24.999814+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:39:08.765597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1a","version":"255'1","reported_seq":11,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.009864+0000","last_change":"2024-01-11T09:47:23.987255+0000","last_active":"2024-01-11T09:47:25.009864+0000","last_peered":"2024-01-11T09:47:25.009864+0000","last_clean":"2024-01-11T09:47:25.009864+0000","last_became_active":"2024-01-11T09:47:23.986953+0000","last_became_peered":"2024-01-11T09:47:23.986953+0000","last_unstale":"2024-01-11T09:47:25.009864+0000","last_undegraded":"2024-01-11T09:47:25.009864+0000","last_fullsized":"2024-01-11T09:47:25.009864+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:58:13.358081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002018+0000","last_change":"2024-01-11T09:47:24.995673+0000","last_active":"2024-01-11T09:47:25.002018+0000","last_peered":"2024-01-11T09:47:25.002018+0000","last_clean":"2024-01-11T09:47:25.002018+0000","last_became_active":"2024-01-11T09:47:24.995421+0000","last_became_peered":"2024-01-11T09:47:24.995421+0000","last_unstale":"2024-01-11T09:47:25.002018+0000","last_undegraded":"2024-01-11T09:47:25.002018+0000","last_fullsized":"2024-01-11T09:47:25.002018+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:46:02.428482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.982524+0000","last_change":"2024-01-11T09:47:23.982683+0000","last_active":"2024-01-11T09:47:24.982524+0000","last_peered":"2024-01-11T09:47:24.982524+0000","last_clean":"2024-01-11T09:47:24.982524+0000","last_became_active":"2024-01-11T09:47:23.981992+0000","last_became_peered":"2024-01-11T09:47:23.981992+0000","last_unstale":"2024-01-11T09:47:24.982524+0000","last_undegraded":"2024-01-11T09:47:24.982524+0000","last_fullsized":"2024-01-11T09:47:24.982524+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:45:16.009410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.995046+0000","last_change":"2024-01-11T09:47:24.991131+0000","last_active":"2024-01-11T09:47:24.995046+0000","last_peered":"2024-01-11T09:47:24.995046+0000","last_clean":"2024-01-11T09:47:24.995046+0000","last_became_active":"2024-01-11T09:47:24.990857+0000","last_became_peered":"2024-01-11T09:47:24.990857+0000","last_unstale":"2024-01-11T09:47:24.995046+0000","last_undegraded":"2024-01-11T09:47:24.995046+0000","last_fullsized":"2024-01-11T09:47:24.995046+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:03:49.193894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.975904+0000","last_change":"2024-01-11T09:47:23.988334+0000","last_active":"2024-01-11T09:47:24.975904+0000","last_peered":"2024-01-11T09:47:24.975904+0000","last_clean":"2024-01-11T09:47:24.975904+0000","last_became_active":"2024-01-11T09:47:23.988049+0000","last_became_peered":"2024-01-11T09:47:23.988049+0000","last_unstale":"2024-01-11T09:47:24.975904+0000","last_undegraded":"2024-01-11T09:47:24.975904+0000","last_fullsized":"2024-01-11T09:47:24.975904+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:54:08.481155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.990622+0000","last_change":"2024-01-11T09:47:24.986224+0000","last_active":"2024-01-11T09:47:24.990622+0000","last_peered":"2024-01-11T09:47:24.990622+0000","last_clean":"2024-01-11T09:47:24.990622+0000","last_became_active":"2024-01-11T09:47:24.985954+0000","last_became_peered":"2024-01-11T09:47:24.985954+0000","last_unstale":"2024-01-11T09:47:24.990622+0000","last_undegraded":"2024-01-11T09:47:24.990622+0000","last_fullsized":"2024-01-11T09:47:24.990622+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:49:23.959042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.978606+0000","last_change":"2024-01-11T09:47:23.982382+0000","last_active":"2024-01-11T09:47:24.978606+0000","last_peered":"2024-01-11T09:47:24.978606+0000","last_clean":"2024-01-11T09:47:24.978606+0000","last_became_active":"2024-01-11T09:47:23.981842+0000","last_became_peered":"2024-01-11T09:47:23.981842+0000","last_unstale":"2024-01-11T09:47:24.978606+0000","last_undegraded":"2024-01-11T09:47:24.978606+0000","last_fullsized":"2024-01-11T09:47:24.978606+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:21.143548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002925+0000","last_change":"2024-01-11T09:47:24.997080+0000","last_active":"2024-01-11T09:47:25.002925+0000","last_peered":"2024-01-11T09:47:25.002925+0000","last_clean":"2024-01-11T09:47:25.002925+0000","last_became_active":"2024-01-11T09:47:24.995601+0000","last_became_peered":"2024-01-11T09:47:24.995601+0000","last_unstale":"2024-01-11T09:47:25.002925+0000","last_undegraded":"2024-01-11T09:47:25.002925+0000","last_fullsized":"2024-01-11T09:47:25.002925+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:35:40.944331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.972520+0000","last_change":"2024-01-11T09:47:23.985660+0000","last_active":"2024-01-11T09:47:24.972520+0000","last_peered":"2024-01-11T09:47:24.972520+0000","last_clean":"2024-01-11T09:47:24.972520+0000","last_became_active":"2024-01-11T09:47:23.985348+0000","last_became_peered":"2024-01-11T09:47:23.985348+0000","last_unstale":"2024-01-11T09:47:24.972520+0000","last_undegraded":"2024-01-11T09:47:24.972520+0000","last_fullsized":"2024-01-11T09:47:24.972520+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:21:00.064284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.991838+0000","last_change":"2024-01-11T09:47:24.988499+0000","last_active":"2024-01-11T09:47:24.991838+0000","last_peered":"2024-01-11T09:47:24.991838+0000","last_clean":"2024-01-11T09:47:24.991838+0000","last_became_active":"2024-01-11T09:47:24.988226+0000","last_became_peered":"2024-01-11T09:47:24.988226+0000","last_unstale":"2024-01-11T09:47:24.991838+0000","last_undegraded":"2024-01-11T09:47:24.991838+0000","last_fullsized":"2024-01-11T09:47:24.991838+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:48:34.768650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.15","version":"255'3","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004676+0000","last_change":"2024-01-11T09:47:23.992634+0000","last_active":"2024-01-11T09:47:25.004676+0000","last_peered":"2024-01-11T09:47:25.004676+0000","last_clean":"2024-01-11T09:47:25.004676+0000","last_became_active":"2024-01-11T09:47:23.992236+0000","last_became_peered":"2024-01-11T09:47:23.992236+0000","last_unstale":"2024-01-11T09:47:25.004676+0000","last_undegraded":"2024-01-11T09:47:25.004676+0000","last_fullsized":"2024-01-11T09:47:25.004676+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:44:42.790810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.993190+0000","last_change":"2024-01-11T09:47:24.985111+0000","last_active":"2024-01-11T09:47:24.993190+0000","last_peered":"2024-01-11T09:47:24.993190+0000","last_clean":"2024-01-11T09:47:24.993190+0000","last_became_active":"2024-01-11T09:47:24.981918+0000","last_became_peered":"2024-01-11T09:47:24.981918+0000","last_unstale":"2024-01-11T09:47:24.993190+0000","last_undegraded":"2024-01-11T09:47:24.993190+0000","last_fullsized":"2024-01-11T09:47:24.993190+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:33:03.766112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.14","version":"255'6","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.010959+0000","last_change":"2024-01-11T09:47:23.979893+0000","last_active":"2024-01-11T09:47:25.010959+0000","last_peered":"2024-01-11T09:47:25.010959+0000","last_clean":"2024-01-11T09:47:25.010959+0000","last_became_active":"2024-01-11T09:47:23.979651+0000","last_became_peered":"2024-01-11T09:47:23.979651+0000","last_unstale":"2024-01-11T09:47:25.010959+0000","last_undegraded":"2024-01-11T09:47:25.010959+0000","last_fullsized":"2024-01-11T09:47:25.010959+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:43:25.959850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002230+0000","last_change":"2024-01-11T09:47:24.998313+0000","last_active":"2024-01-11T09:47:25.002230+0000","last_peered":"2024-01-11T09:47:25.002230+0000","last_clean":"2024-01-11T09:47:25.002230+0000","last_became_active":"2024-01-11T09:47:24.998032+0000","last_became_peered":"2024-01-11T09:47:24.998032+0000","last_unstale":"2024-01-11T09:47:25.002230+0000","last_undegraded":"2024-01-11T09:47:25.002230+0000","last_fullsized":"2024-01-11T09:47:25.002230+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:20:08.159916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.13","version":"255'2","reported_seq":12,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.010541+0000","last_change":"2024-01-11T09:47:23.985710+0000","last_active":"2024-01-11T09:47:25.010541+0000","last_peered":"2024-01-11T09:47:25.010541+0000","last_clean":"2024-01-11T09:47:25.010541+0000","last_became_active":"2024-01-11T09:47:23.985422+0000","last_became_peered":"2024-01-11T09:47:23.985422+0000","last_unstale":"2024-01-11T09:47:25.010541+0000","last_undegraded":"2024-01-11T09:47:25.010541+0000","last_fullsized":"2024-01-11T09:47:25.010541+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:16:05.292707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.007847+0000","last_change":"2024-01-11T09:47:25.005425+0000","last_active":"2024-01-11T09:47:25.007847+0000","last_peered":"2024-01-11T09:47:25.007847+0000","last_clean":"2024-01-11T09:47:25.007847+0000","last_became_active":"2024-01-11T09:47:25.005066+0000","last_became_peered":"2024-01-11T09:47:25.005066+0000","last_unstale":"2024-01-11T09:47:25.007847+0000","last_undegraded":"2024-01-11T09:47:25.007847+0000","last_fullsized":"2024-01-11T09:47:25.007847+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:16:51.113785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.974732+0000","last_change":"2024-01-11T09:47:23.983718+0000","last_active":"2024-01-11T09:47:24.974732+0000","last_peered":"2024-01-11T09:47:24.974732+0000","last_clean":"2024-01-11T09:47:24.974732+0000","last_became_active":"2024-01-11T09:47:23.980334+0000","last_became_peered":"2024-01-11T09:47:23.980334+0000","last_unstale":"2024-01-11T09:47:24.974732+0000","last_undegraded":"2024-01-11T09:47:24.974732+0000","last_fullsized":"2024-01-11T09:47:24.974732+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:07:38.872375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.990746+0000","last_change":"2024-01-11T09:47:24.986626+0000","last_active":"2024-01-11T09:47:24.990746+0000","last_peered":"2024-01-11T09:47:24.990746+0000","last_clean":"2024-01-11T09:47:24.990746+0000","last_became_active":"2024-01-11T09:47:24.984969+0000","last_became_peered":"2024-01-11T09:47:24.984969+0000","last_unstale":"2024-01-11T09:47:24.990746+0000","last_undegraded":"2024-01-11T09:47:24.990746+0000","last_fullsized":"2024-01-11T09:47:24.990746+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:13:19.226196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.975742+0000","last_change":"2024-01-11T09:47:23.987106+0000","last_active":"2024-01-11T09:47:24.975742+0000","last_peered":"2024-01-11T09:47:24.975742+0000","last_clean":"2024-01-11T09:47:24.975742+0000","last_became_active":"2024-01-11T09:47:23.986838+0000","last_became_peered":"2024-01-11T09:47:23.986838+0000","last_unstale":"2024-01-11T09:47:24.975742+0000","last_undegraded":"2024-01-11T09:47:24.975742+0000","last_fullsized":"2024-01-11T09:47:24.975742+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:31:39.274165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.999673+0000","last_change":"2024-01-11T09:47:24.995245+0000","last_active":"2024-01-11T09:47:24.999673+0000","last_peered":"2024-01-11T09:47:24.999673+0000","last_clean":"2024-01-11T09:47:24.999673+0000","last_became_active":"2024-01-11T09:47:24.994109+0000","last_became_peered":"2024-01-11T09:47:24.994109+0000","last_unstale":"2024-01-11T09:47:24.999673+0000","last_undegraded":"2024-01-11T09:47:24.999673+0000","last_fullsized":"2024-01-11T09:47:24.999673+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:26:09.368076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.10","version":"255'8","reported_seq":18,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.009227+0000","last_change":"2024-01-11T09:47:23.992629+0000","last_active":"2024-01-11T09:47:25.009227+0000","last_peered":"2024-01-11T09:47:25.009227+0000","last_clean":"2024-01-11T09:47:25.009227+0000","last_became_active":"2024-01-11T09:47:23.992344+0000","last_became_peered":"2024-01-11T09:47:23.992344+0000","last_unstale":"2024-01-11T09:47:25.009227+0000","last_undegraded":"2024-01-11T09:47:25.009227+0000","last_fullsized":"2024-01-11T09:47:25.009227+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:16:18.912554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.987383+0000","last_change":"2024-01-11T09:47:24.984069+0000","last_active":"2024-01-11T09:47:24.987383+0000","last_peered":"2024-01-11T09:47:24.987383+0000","last_clean":"2024-01-11T09:47:24.987383+0000","last_became_active":"2024-01-11T09:47:24.983802+0000","last_became_peered":"2024-01-11T09:47:24.983802+0000","last_unstale":"2024-01-11T09:47:24.987383+0000","last_undegraded":"2024-01-11T09:47:24.987383+0000","last_fullsized":"2024-01-11T09:47:24.987383+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:04:45.182025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"255'1","reported_seq":11,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.009813+0000","last_change":"2024-01-11T09:47:23.985751+0000","last_active":"2024-01-11T09:47:25.009813+0000","last_peered":"2024-01-11T09:47:25.009813+0000","last_clean":"2024-01-11T09:47:25.009813+0000","last_became_active":"2024-01-11T09:47:23.985422+0000","last_became_peered":"2024-01-11T09:47:23.985422+0000","last_unstale":"2024-01-11T09:47:25.009813+0000","last_undegraded":"2024-01-11T09:47:25.009813+0000","last_fullsized":"2024-01-11T09:47:25.009813+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:13:38.339268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.986787+0000","last_change":"2024-01-11T09:47:24.982411+0000","last_active":"2024-01-11T09:47:24.986787+0000","last_peered":"2024-01-11T09:47:24.986787+0000","last_clean":"2024-01-11T09:47:24.986787+0000","last_became_active":"2024-01-11T09:47:24.980530+0000","last_became_peered":"2024-01-11T09:47:24.980530+0000","last_unstale":"2024-01-11T09:47:24.986787+0000","last_undegraded":"2024-01-11T09:47:24.986787+0000","last_fullsized":"2024-01-11T09:47:24.986787+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:26:00.307598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.e","version":"255'2","reported_seq":12,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.006007+0000","last_change":"2024-01-11T09:47:23.974627+0000","last_active":"2024-01-11T09:47:25.006007+0000","last_peered":"2024-01-11T09:47:25.006007+0000","last_clean":"2024-01-11T09:47:25.006007+0000","last_became_active":"2024-01-11T09:47:23.973839+0000","last_became_peered":"2024-01-11T09:47:23.973839+0000","last_unstale":"2024-01-11T09:47:25.006007+0000","last_undegraded":"2024-01-11T09:47:25.006007+0000","last_fullsized":"2024-01-11T09:47:25.006007+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:57:20.246338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004188+0000","last_change":"2024-01-11T09:47:24.997271+0000","last_active":"2024-01-11T09:47:25.004188+0000","last_peered":"2024-01-11T09:47:25.004188+0000","last_clean":"2024-01-11T09:47:25.004188+0000","last_became_active":"2024-01-11T09:47:24.995852+0000","last_became_peered":"2024-01-11T09:47:24.995852+0000","last_unstale":"2024-01-11T09:47:25.004188+0000","last_undegraded":"2024-01-11T09:47:25.004188+0000","last_fullsized":"2024-01-11T09:47:25.004188+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:48:39.698634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.d","version":"255'1","reported_seq":11,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.007572+0000","last_change":"2024-01-11T09:47:23.989553+0000","last_active":"2024-01-11T09:47:25.007572+0000","last_peered":"2024-01-11T09:47:25.007572+0000","last_clean":"2024-01-11T09:47:25.007572+0000","last_became_active":"2024-01-11T09:47:23.989285+0000","last_became_peered":"2024-01-11T09:47:23.989285+0000","last_unstale":"2024-01-11T09:47:25.007572+0000","last_undegraded":"2024-01-11T09:47:25.007572+0000","last_fullsized":"2024-01-11T09:47:25.007572+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T17:36:27.961488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.980678+0000","last_change":"2024-01-11T09:47:24.976990+0000","last_active":"2024-01-11T09:47:24.980678+0000","last_peered":"2024-01-11T09:47:24.980678+0000","last_clean":"2024-01-11T09:47:24.980678+0000","last_became_active":"2024-01-11T09:47:24.976691+0000","last_became_peered":"2024-01-11T09:47:24.976691+0000","last_unstale":"2024-01-11T09:47:24.980678+0000","last_undegraded":"2024-01-11T09:47:24.980678+0000","last_fullsized":"2024-01-11T09:47:24.980678+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:03:19.459500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.980699+0000","last_change":"2024-01-11T09:47:23.980634+0000","last_active":"2024-01-11T09:47:24.980699+0000","last_peered":"2024-01-11T09:47:24.980699+0000","last_clean":"2024-01-11T09:47:24.980699+0000","last_became_active":"2024-01-11T09:47:23.980274+0000","last_became_peered":"2024-01-11T09:47:23.980274+0000","last_unstale":"2024-01-11T09:47:24.980699+0000","last_undegraded":"2024-01-11T09:47:24.980699+0000","last_fullsized":"2024-01-11T09:47:24.980699+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:34:28.082927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002123+0000","last_change":"2024-01-11T09:47:24.994868+0000","last_active":"2024-01-11T09:47:25.002123+0000","last_peered":"2024-01-11T09:47:25.002123+0000","last_clean":"2024-01-11T09:47:25.002123+0000","last_became_active":"2024-01-11T09:47:24.994389+0000","last_became_peered":"2024-01-11T09:47:24.994389+0000","last_unstale":"2024-01-11T09:47:25.002123+0000","last_undegraded":"2024-01-11T09:47:25.002123+0000","last_fullsized":"2024-01-11T09:47:25.002123+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:57:22.152883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.b","version":"255'2","reported_seq":12,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.008312+0000","last_change":"2024-01-11T09:47:23.990231+0000","last_active":"2024-01-11T09:47:25.008312+0000","last_peered":"2024-01-11T09:47:25.008312+0000","last_clean":"2024-01-11T09:47:25.008312+0000","last_became_active":"2024-01-11T09:47:23.989904+0000","last_became_peered":"2024-01-11T09:47:23.989904+0000","last_unstale":"2024-01-11T09:47:25.008312+0000","last_undegraded":"2024-01-11T09:47:25.008312+0000","last_fullsized":"2024-01-11T09:47:25.008312+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:32:21.144708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.990629+0000","last_change":"2024-01-11T09:47:24.986224+0000","last_active":"2024-01-11T09:47:24.990629+0000","last_peered":"2024-01-11T09:47:24.990629+0000","last_clean":"2024-01-11T09:47:24.990629+0000","last_became_active":"2024-01-11T09:47:24.985980+0000","last_became_peered":"2024-01-11T09:47:24.985980+0000","last_unstale":"2024-01-11T09:47:24.990629+0000","last_undegraded":"2024-01-11T09:47:24.990629+0000","last_fullsized":"2024-01-11T09:47:24.990629+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:21:42.087560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.970334+0000","last_change":"2024-01-11T09:47:23.985032+0000","last_active":"2024-01-11T09:47:24.970334+0000","last_peered":"2024-01-11T09:47:24.970334+0000","last_clean":"2024-01-11T09:47:24.970334+0000","last_became_active":"2024-01-11T09:47:23.984242+0000","last_became_peered":"2024-01-11T09:47:23.984242+0000","last_unstale":"2024-01-11T09:47:24.970334+0000","last_undegraded":"2024-01-11T09:47:24.970334+0000","last_fullsized":"2024-01-11T09:47:24.970334+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:45:16.009410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004925+0000","last_change":"2024-01-11T09:47:25.001180+0000","last_active":"2024-01-11T09:47:25.004925+0000","last_peered":"2024-01-11T09:47:25.004925+0000","last_clean":"2024-01-11T09:47:25.004925+0000","last_became_active":"2024-01-11T09:47:24.997474+0000","last_became_peered":"2024-01-11T09:47:24.997474+0000","last_unstale":"2024-01-11T09:47:25.004925+0000","last_undegraded":"2024-01-11T09:47:25.004925+0000","last_fullsized":"2024-01-11T09:47:25.004925+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:15:03.907181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.974098+0000","last_change":"2024-01-11T09:47:23.981388+0000","last_active":"2024-01-11T09:47:24.974098+0000","last_peered":"2024-01-11T09:47:24.974098+0000","last_clean":"2024-01-11T09:47:24.974098+0000","last_became_active":"2024-01-11T09:47:23.981006+0000","last_became_peered":"2024-01-11T09:47:23.981006+0000","last_unstale":"2024-01-11T09:47:24.974098+0000","last_undegraded":"2024-01-11T09:47:24.974098+0000","last_fullsized":"2024-01-11T09:47:24.974098+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:44:11.182427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.001816+0000","last_change":"2024-01-11T09:47:24.995629+0000","last_active":"2024-01-11T09:47:25.001816+0000","last_peered":"2024-01-11T09:47:25.001816+0000","last_clean":"2024-01-11T09:47:25.001816+0000","last_became_active":"2024-01-11T09:47:24.994627+0000","last_became_peered":"2024-01-11T09:47:24.994627+0000","last_unstale":"2024-01-11T09:47:25.001816+0000","last_undegraded":"2024-01-11T09:47:25.001816+0000","last_fullsized":"2024-01-11T09:47:25.001816+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T21:23:57.820334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.982072+0000","last_change":"2024-01-11T09:47:23.989028+0000","last_active":"2024-01-11T09:47:24.982072+0000","last_peered":"2024-01-11T09:47:24.982072+0000","last_clean":"2024-01-11T09:47:24.982072+0000","last_became_active":"2024-01-11T09:47:23.988482+0000","last_became_peered":"2024-01-11T09:47:23.988482+0000","last_unstale":"2024-01-11T09:47:24.982072+0000","last_undegraded":"2024-01-11T09:47:24.982072+0000","last_fullsized":"2024-01-11T09:47:24.982072+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:53:24.155769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002478+0000","last_change":"2024-01-11T09:47:24.998398+0000","last_active":"2024-01-11T09:47:25.002478+0000","last_peered":"2024-01-11T09:47:25.002478+0000","last_clean":"2024-01-11T09:47:25.002478+0000","last_became_active":"2024-01-11T09:47:24.998044+0000","last_became_peered":"2024-01-11T09:47:24.998044+0000","last_unstale":"2024-01-11T09:47:25.002478+0000","last_undegraded":"2024-01-11T09:47:25.002478+0000","last_fullsized":"2024-01-11T09:47:25.002478+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:24:05.147676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.7","version":"255'2","reported_seq":12,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004581+0000","last_change":"2024-01-11T09:47:23.983465+0000","last_active":"2024-01-11T09:47:25.004581+0000","last_peered":"2024-01-11T09:47:25.004581+0000","last_clean":"2024-01-11T09:47:25.004581+0000","last_became_active":"2024-01-11T09:47:23.982586+0000","last_became_peered":"2024-01-11T09:47:23.982586+0000","last_unstale":"2024-01-11T09:47:25.004581+0000","last_undegraded":"2024-01-11T09:47:25.004581+0000","last_fullsized":"2024-01-11T09:47:25.004581+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:48:21.245464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.993191+0000","last_change":"2024-01-11T09:47:24.990311+0000","last_active":"2024-01-11T09:47:24.993191+0000","last_peered":"2024-01-11T09:47:24.993191+0000","last_clean":"2024-01-11T09:47:24.993191+0000","last_became_active":"2024-01-11T09:47:24.990039+0000","last_became_peered":"2024-01-11T09:47:24.990039+0000","last_unstale":"2024-01-11T09:47:24.993191+0000","last_undegraded":"2024-01-11T09:47:24.993191+0000","last_fullsized":"2024-01-11T09:47:24.993191+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T18:36:54.837036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.983322+0000","last_change":"2024-01-11T09:47:23.984889+0000","last_active":"2024-01-11T09:47:24.983322+0000","last_peered":"2024-01-11T09:47:24.983322+0000","last_clean":"2024-01-11T09:47:24.983322+0000","last_became_active":"2024-01-11T09:47:23.984422+0000","last_became_peered":"2024-01-11T09:47:23.984422+0000","last_unstale":"2024-01-11T09:47:24.983322+0000","last_undegraded":"2024-01-11T09:47:24.983322+0000","last_fullsized":"2024-01-11T09:47:24.983322+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T10:55:32.480267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.996505+0000","last_change":"2024-01-11T09:47:24.992441+0000","last_active":"2024-01-11T09:47:24.996505+0000","last_peered":"2024-01-11T09:47:24.996505+0000","last_clean":"2024-01-11T09:47:24.996505+0000","last_became_active":"2024-01-11T09:47:24.992125+0000","last_became_peered":"2024-01-11T09:47:24.992125+0000","last_unstale":"2024-01-11T09:47:24.996505+0000","last_undegraded":"2024-01-11T09:47:24.996505+0000","last_fullsized":"2024-01-11T09:47:24.996505+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T19:17:43.512711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.973292+0000","last_change":"2024-01-11T09:47:23.984077+0000","last_active":"2024-01-11T09:47:24.973292+0000","last_peered":"2024-01-11T09:47:24.973292+0000","last_clean":"2024-01-11T09:47:24.973292+0000","last_became_active":"2024-01-11T09:47:23.982674+0000","last_became_peered":"2024-01-11T09:47:23.982674+0000","last_unstale":"2024-01-11T09:47:24.973292+0000","last_undegraded":"2024-01-11T09:47:24.973292+0000","last_fullsized":"2024-01-11T09:47:24.973292+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:38:13.535607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002340+0000","last_change":"2024-01-11T09:47:24.997499+0000","last_active":"2024-01-11T09:47:25.002340+0000","last_peered":"2024-01-11T09:47:25.002340+0000","last_clean":"2024-01-11T09:47:25.002340+0000","last_became_active":"2024-01-11T09:47:24.993811+0000","last_became_peered":"2024-01-11T09:47:24.993811+0000","last_unstale":"2024-01-11T09:47:25.002340+0000","last_undegraded":"2024-01-11T09:47:25.002340+0000","last_fullsized":"2024-01-11T09:47:25.002340+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:39:30.172237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.972809+0000","last_change":"2024-01-11T09:47:23.990863+0000","last_active":"2024-01-11T09:47:24.972809+0000","last_peered":"2024-01-11T09:47:24.972809+0000","last_clean":"2024-01-11T09:47:24.972809+0000","last_became_active":"2024-01-11T09:47:23.990051+0000","last_became_peered":"2024-01-11T09:47:23.990051+0000","last_unstale":"2024-01-11T09:47:24.972809+0000","last_undegraded":"2024-01-11T09:47:24.972809+0000","last_fullsized":"2024-01-11T09:47:24.972809+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T15:30:02.583920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.002102+0000","last_change":"2024-01-11T09:47:24.994678+0000","last_active":"2024-01-11T09:47:25.002102+0000","last_peered":"2024-01-11T09:47:25.002102+0000","last_clean":"2024-01-11T09:47:25.002102+0000","last_became_active":"2024-01-11T09:47:24.994225+0000","last_became_peered":"2024-01-11T09:47:24.994225+0000","last_unstale":"2024-01-11T09:47:25.002102+0000","last_undegraded":"2024-01-11T09:47:25.002102+0000","last_fullsized":"2024-01-11T09:47:25.002102+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:44:12.191761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.979025+0000","last_change":"2024-01-11T09:47:23.988752+0000","last_active":"2024-01-11T09:47:24.979025+0000","last_peered":"2024-01-11T09:47:24.979025+0000","last_clean":"2024-01-11T09:47:24.979025+0000","last_became_active":"2024-01-11T09:47:23.988477+0000","last_became_peered":"2024-01-11T09:47:23.988477+0000","last_unstale":"2024-01-11T09:47:24.979025+0000","last_undegraded":"2024-01-11T09:47:24.979025+0000","last_fullsized":"2024-01-11T09:47:24.979025+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:39:41.499725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.991757+0000","last_change":"2024-01-11T09:47:24.989860+0000","last_active":"2024-01-11T09:47:24.991757+0000","last_peered":"2024-01-11T09:47:24.991757+0000","last_clean":"2024-01-11T09:47:24.991757+0000","last_became_active":"2024-01-11T09:47:24.988915+0000","last_became_peered":"2024-01-11T09:47:24.988915+0000","last_unstale":"2024-01-11T09:47:24.991757+0000","last_undegraded":"2024-01-11T09:47:24.991757+0000","last_fullsized":"2024-01-11T09:47:24.991757+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T12:20:16.704821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.985174+0000","last_change":"2024-01-11T09:47:23.982214+0000","last_active":"2024-01-11T09:47:24.985174+0000","last_peered":"2024-01-11T09:47:24.985174+0000","last_clean":"2024-01-11T09:47:24.985174+0000","last_became_active":"2024-01-11T09:47:23.981616+0000","last_became_peered":"2024-01-11T09:47:23.981616+0000","last_unstale":"2024-01-11T09:47:24.985174+0000","last_undegraded":"2024-01-11T09:47:24.985174+0000","last_fullsized":"2024-01-11T09:47:24.985174+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T20:43:25.959850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004904+0000","last_change":"2024-01-11T09:47:25.001162+0000","last_active":"2024-01-11T09:47:25.004904+0000","last_peered":"2024-01-11T09:47:25.004904+0000","last_clean":"2024-01-11T09:47:25.004904+0000","last_became_active":"2024-01-11T09:47:24.997121+0000","last_became_peered":"2024-01-11T09:47:24.997121+0000","last_unstale":"2024-01-11T09:47:25.004904+0000","last_undegraded":"2024-01-11T09:47:25.004904+0000","last_fullsized":"2024-01-11T09:47:25.004904+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:42:18.444395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.982605+0000","last_change":"2024-01-11T09:47:23.981247+0000","last_active":"2024-01-11T09:47:24.982605+0000","last_peered":"2024-01-11T09:47:24.982605+0000","last_clean":"2024-01-11T09:47:24.982605+0000","last_became_active":"2024-01-11T09:47:23.980209+0000","last_became_peered":"2024-01-11T09:47:23.980209+0000","last_unstale":"2024-01-11T09:47:24.982605+0000","last_undegraded":"2024-01-11T09:47:24.982605+0000","last_fullsized":"2024-01-11T09:47:24.982605+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T11:57:20.246338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:25.004798+0000","last_change":"2024-01-11T09:47:24.999219+0000","last_active":"2024-01-11T09:47:25.004798+0000","last_peered":"2024-01-11T09:47:25.004798+0000","last_clean":"2024-01-11T09:47:25.004798+0000","last_became_active":"2024-01-11T09:47:24.994240+0000","last_became_peered":"2024-01-11T09:47:24.994240+0000","last_unstale":"2024-01-11T09:47:25.004798+0000","last_undegraded":"2024-01-11T09:47:25.004798+0000","last_fullsized":"2024-01-11T09:47:25.004798+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:04:45.182025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":10,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.969020+0000","last_change":"2024-01-11T09:47:23.983239+0000","last_active":"2024-01-11T09:47:24.969020+0000","last_peered":"2024-01-11T09:47:24.969020+0000","last_clean":"2024-01-11T09:47:24.969020+0000","last_became_active":"2024-01-11T09:47:23.982982+0000","last_became_peered":"2024-01-11T09:47:23.982982+0000","last_unstale":"2024-01-11T09:47:24.969020+0000","last_undegraded":"2024-01-11T09:47:24.969020+0000","last_fullsized":"2024-01-11T09:47:24.969020+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:22.947609+0000","last_clean_scrub_stamp":"2024-01-11T09:47:22.947609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T09:53:24.155769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.990534+0000","last_change":"2024-01-11T09:47:24.987204+0000","last_active":"2024-01-11T09:47:24.990534+0000","last_peered":"2024-01-11T09:47:24.990534+0000","last_clean":"2024-01-11T09:47:24.990534+0000","last_became_active":"2024-01-11T09:47:24.984730+0000","last_became_peered":"2024-01-11T09:47:24.984730+0000","last_unstale":"2024-01-11T09:47:24.990534+0000","last_undegraded":"2024-01-11T09:47:24.990534+0000","last_fullsized":"2024-01-11T09:47:24.990534+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T09:47:23.956944+0000","last_clean_scrub_stamp":"2024-01-11T09:47:23.956944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T13:46:15.517759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"242'408","reported_seq":1383,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-01-11T09:47:24.972278+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:47:24.972278+0000","last_peered":"2024-01-11T09:47:24.972278+0000","last_clean":"2024-01-11T09:47:24.972278+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:47:24.972278+0000","last_undegraded":"2024-01-11T09:47:24.972278+0000","last_fullsized":"2024-01-11T09:47:24.972278+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":408,"log_dups_size":0,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2654,"num_read_kb":5617,"num_write":687,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2654,"num_read_kb":5617,"num_write":687,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":408,"ondisk_log_size":408,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575992,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305124,"kb_used_data":3740,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94066716,"statfs":{"total":96636764160,"available":96324317184,"internally_reserved":0,"allocated":3829760,"data_stored":2158745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.68899999999999995,"15min":0.72799999999999998},"min":{"1min":0.39100000000000001,"5min":0.376,"15min":0.36599999999999999},"max":{"1min":0.89200000000000002,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.71499999999999997,"15min":0.74299999999999999},"min":{"1min":0.40400000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83799999999999997,"5min":1.0609999999999999,"15min":1.0669999999999999},"last":0.55600000000000005}]},{"osd":1,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.69799999999999995,"15min":0.72499999999999998},"min":{"1min":0.48099999999999998,"5min":0.41799999999999998,"15min":0.41599999999999998},"max":{"1min":0.82099999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.70499999999999996,"15min":0.73599999999999999},"min":{"1min":0.504,"5min":0.434,"15min":0.434},"max":{"1min":0.89400000000000002,"5min":0.999,"15min":1.0820000000000001},"last":0.69499999999999995}]},{"osd":2,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.71799999999999997,"15min":0.73499999999999999},"min":{"1min":0.46899999999999997,"5min":0.443,"15min":0.443},"max":{"1min":0.875,"5min":0.94399999999999995,"15min":1.0600000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.71099999999999997,"15min":0.747},"min":{"1min":0.48499999999999999,"5min":0.44800000000000001,"15min":0.44500000000000001},"max":{"1min":0.86099999999999999,"5min":0.96999999999999997,"15min":1.153},"last":0.67900000000000005}]},{"osd":3,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73599999999999999,"15min":0.76200000000000001},"min":{"1min":0.51500000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":1.0660000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.72299999999999998,"15min":0.747},"min":{"1min":0.51000000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.77100000000000002,"5min":1,"15min":1.0880000000000001},"last":0.63900000000000001}]},{"osd":4,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61099999999999999,"15min":0.60599999999999998},"min":{"1min":0.45900000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79600000000000004,"5min":1.0860000000000001,"15min":1.107},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64000000000000001,"15min":0.63900000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.39700000000000002},"max":{"1min":0.871,"5min":0.995,"15min":1.0960000000000001},"last":0.84199999999999997}]},{"osd":5,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.622,"15min":0.64100000000000001},"min":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86099999999999999,"5min":0.86699999999999999,"15min":1.0109999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.69199999999999995,"15min":0.68799999999999994},"min":{"1min":0.41499999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.64900000000000002}]},{"osd":6,"last update":"Thu Jan 11 09:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.78000000000000003},"min":{"1min":0.56000000000000005,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.96899999999999997,"5min":1.232,"15min":1.232},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.002,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.88300000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575991,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312740,"kb_used_data":3164,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059100,"statfs":{"total":96636764160,"available":96316518400,"internally_reserved":0,"allocated":3239936,"data_stored":1568467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.63200000000000001,"15min":0.69499999999999995},"min":{"1min":0.438,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.79800000000000004,"5min":0.94199999999999995,"15min":0.999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.68400000000000005,"15min":0.72799999999999998},"min":{"1min":0.46100000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.93400000000000005,"5min":0.97099999999999997,"15min":1.216},"last":0.67500000000000004}]},{"osd":1,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.66000000000000003,"15min":0.72199999999999998},"min":{"1min":0.46200000000000002,"5min":0.42799999999999999,"15min":0.41699999999999998},"max":{"1min":0.91000000000000003,"5min":0.98499999999999999,"15min":1.3129999999999999},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.67000000000000004,"15min":0.71599999999999997},"min":{"1min":0.46500000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.91100000000000003,"5min":0.95999999999999996,"15min":1.0840000000000001},"last":0.81499999999999995}]},{"osd":2,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.68700000000000006,"15min":0.73499999999999999},"min":{"1min":0.44600000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.83799999999999997,"5min":1.0069999999999999,"15min":1.4179999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.69999999999999996,"15min":0.73999999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.92800000000000005,"5min":1.796,"15min":1.796},"last":0.90400000000000003}]},{"osd":3,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.71299999999999997,"15min":0.746},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89000000000000001,"5min":1.012,"15min":1.014},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.70099999999999996,"15min":0.754},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":1.165},"last":0.86199999999999999}]},{"osd":4,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.59399999999999997,"15min":0.56999999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.86599999999999999,"5min":0.91700000000000004,"15min":1.0309999999999999},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.64400000000000002,"15min":0.622},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":1.1259999999999999},"last":0.95699999999999996}]},{"osd":5,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.077,"5min":1.077,"15min":1.369},"last":1.0880000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.71699999999999997,"15min":0.70899999999999996},"min":{"1min":0.495,"5min":0.49099999999999999,"15min":0.45400000000000001},"max":{"1min":0.94399999999999995,"5min":0.999,"15min":1.0960000000000001},"last":1.008}]},{"osd":7,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.72099999999999997,"15min":0.71399999999999997},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.45100000000000001},"max":{"1min":1.0089999999999999,"5min":1.089,"15min":1.484},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.77500000000000002,"15min":0.77200000000000002},"min":{"1min":0.53700000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.123,"5min":1.1379999999999999,"15min":1.603},"last":1.1299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575993,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308644,"kb_used_data":3164,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063196,"statfs":{"total":96636764160,"available":96320712704,"internally_reserved":0,"allocated":3239936,"data_stored":1568411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:46 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.504,"15min":0.50800000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.65500000000000003,"5min":0.73399999999999999,"15min":0.93999999999999995},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.58199999999999996,"15min":0.60799999999999998},"min":{"1min":0.41799999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.69999999999999996,"5min":0.85199999999999998,"15min":0.95099999999999996},"last":0.41899999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:46:46 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.48899999999999999,"15min":0.504},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.80100000000000005,"5min":0.82399999999999995,"15min":0.82699999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.57499999999999996,"15min":0.58599999999999997},"min":{"1min":0.38400000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.67400000000000004,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.39300000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:46:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.81299999999999994,"5min":0.91600000000000004,"15min":0.96699999999999997},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.57099999999999995,"15min":0.57599999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.34599999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:46:46 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.79100000000000004,"15min":0.81200000000000006},"min":{"1min":0.54900000000000004,"5min":0.51400000000000001,"15min":0.505},"max":{"1min":0.94699999999999995,"5min":1.214,"15min":1.2969999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.82899999999999996,"15min":0.85199999999999998},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.008,"5min":1.1879999999999999,"15min":1.1890000000000001},"last":0.72499999999999998}]},{"osd":5,"last update":"Thu Jan 11 09:46:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.80800000000000005,"15min":0.81999999999999995},"min":{"1min":0.55900000000000005,"5min":0.443,"15min":0.443},"max":{"1min":0.98399999999999999,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.79500000000000004,"15min":0.80300000000000005},"min":{"1min":0.56499999999999995,"5min":0.54200000000000004,"15min":0.51300000000000001},"max":{"1min":0.99399999999999999,"5min":1.341,"15min":1.341},"last":0.68600000000000005}]},{"osd":6,"last update":"Thu Jan 11 09:46:46 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.81799999999999995,"15min":0.83799999999999997},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.56799999999999995},"max":{"1min":0.97999999999999998,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.748,"5min":0.82899999999999996,"15min":0.84699999999999998},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.94199999999999995,"5min":1.177,"15min":1.177},"last":0.57299999999999995}]},{"osd":7,"last update":"Thu Jan 11 09:46:46 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.87,"15min":0.879},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.1279999999999999,"5min":1.421,"15min":1.421},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.84499999999999997,"15min":0.85199999999999998},"min":{"1min":0.623,"5min":0.60099999999999998,"15min":0.59299999999999997},"max":{"1min":1.163,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.77400000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575992,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309232,"kb_used_data":3752,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062608,"statfs":{"total":96636764160,"available":96320110592,"internally_reserved":0,"allocated":3842048,"data_stored":2159355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.48799999999999999,"15min":0.51300000000000001},"min":{"1min":0.34899999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.73099999999999998,"5min":0.754,"15min":0.84699999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.56299999999999994,"15min":0.59099999999999997},"min":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.59599999999999997,"5min":0.81599999999999995,"15min":0.874},"last":0.70499999999999996}]},{"osd":2,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.53300000000000003,"15min":0.56200000000000006},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.68200000000000005,"5min":0.83099999999999996,"15min":0.97499999999999998},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55400000000000005,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.86399999999999999},"last":0.59099999999999997}]},{"osd":3,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51100000000000001,"15min":0.53800000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.95399999999999996},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60699999999999998,"15min":0.64400000000000002},"min":{"1min":0.42499999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.87,"5min":0.90300000000000002,"15min":1.0389999999999999},"last":0.81200000000000006}]},{"osd":4,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.78600000000000003,"15min":0.81100000000000005},"min":{"1min":0.46100000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.304},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.78500000000000003,"15min":0.80100000000000005},"min":{"1min":0.48799999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.95399999999999996,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":1.0049999999999999}]},{"osd":5,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.80800000000000005,"15min":0.83599999999999997},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.97599999999999998,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.81399999999999995,"15min":0.83699999999999997},"min":{"1min":0.60399999999999998,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.02,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":1.139}]},{"osd":6,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.79700000000000004,"15min":0.82099999999999995},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.95899999999999996,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.82299999999999995,"15min":0.85999999999999999},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":0.97699999999999998,"5min":1.1699999999999999,"15min":1.256},"last":1.1220000000000001}]},{"osd":7,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.83699999999999997,"15min":0.86299999999999999},"min":{"1min":0.64100000000000001,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1000000000000001,"5min":1.226,"15min":1.28},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.84499999999999997,"15min":0.86899999999999999},"min":{"1min":0.63,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0369999999999999,"5min":1.238,"15min":1.2410000000000001},"last":1.073}]}]},{"osd":2,"up_from":13,"seq":55834575991,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300452,"kb_used_data":3164,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071388,"statfs":{"total":96636764160,"available":96329101312,"internally_reserved":0,"allocated":3239936,"data_stored":1568467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.46999999999999997,"15min":0.5},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.27900000000000003},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.85399999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.53400000000000003,"15min":0.56000000000000005},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66900000000000004,"5min":0.77300000000000002,"15min":1.6359999999999999},"last":0.82299999999999995}]},{"osd":1,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56499999999999995,"15min":0.57999999999999996},"min":{"1min":0.40000000000000002,"5min":0.38200000000000001,"15min":0.33500000000000002},"max":{"1min":0.70099999999999996,"5min":0.88800000000000001,"15min":0.93000000000000005},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.56399999999999995,"15min":0.59299999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.64800000000000002,"5min":0.97299999999999998,"15min":1.145},"last":0.71799999999999997}]},{"osd":3,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52900000000000003,"15min":0.55000000000000004},"min":{"1min":0.376,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81499999999999995,"5min":0.83099999999999996,"15min":0.96399999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.61199999999999999,"15min":0.65000000000000002},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.75600000000000001,"5min":0.876,"15min":1.3620000000000001},"last":0.498}]},{"osd":4,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.75700000000000001,"15min":0.79100000000000004},"min":{"1min":0.56299999999999994,"5min":0.55800000000000005,"15min":0.54300000000000004},"max":{"1min":0.96499999999999997,"5min":1.071,"15min":1.1519999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.76400000000000001,"15min":0.79300000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0209999999999999,"5min":1.161,"15min":1.161},"last":0.63}]},{"osd":5,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.79900000000000004,"15min":0.85599999999999998},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.94099999999999995,"5min":1.228,"15min":1.429},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.82799999999999996,"15min":0.85299999999999998},"min":{"1min":0.61499999999999999,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":0.94699999999999995,"5min":1.1850000000000001,"15min":1.2170000000000001},"last":0.90800000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.77600000000000002,"15min":0.81799999999999995},"min":{"1min":0.56000000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96499999999999997,"5min":1.1659999999999999,"15min":1.4139999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.80400000000000005,"15min":0.85599999999999998},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.94999999999999996,"5min":1.2010000000000001,"15min":1.593},"last":0.55800000000000005}]},{"osd":7,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.84399999999999997,"15min":0.88400000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.57299999999999995},"max":{"1min":1.004,"5min":1.1339999999999999,"15min":2.0539999999999998},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.83499999999999996,"15min":0.872},"min":{"1min":0.624,"5min":0.60799999999999998,"15min":0.55500000000000005},"max":{"1min":1.026,"5min":1.2270000000000001,"15min":1.2490000000000001},"last":0.52900000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575991,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304556,"kb_used_data":3172,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067284,"statfs":{"total":96636764160,"available":96324898816,"internally_reserved":0,"allocated":3248128,"data_stored":1569913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:42 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.48999999999999999,"15min":0.51400000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.621,"5min":0.89700000000000002,"15min":0.90000000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55000000000000004,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.38600000000000001}]},{"osd":1,"last update":"Thu Jan 11 09:46:42 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.48199999999999998,"15min":0.50900000000000001},"min":{"1min":0.34799999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.304},"last":0.443},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57799999999999996,"15min":0.60599999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":1.4590000000000001},"last":0.40600000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:46:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.56899999999999995,"15min":0.60299999999999998},"min":{"1min":0.42999999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.628,"15min":0.65500000000000003},"min":{"1min":0.45200000000000001,"5min":0.44800000000000001,"15min":0.44700000000000001},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.502}]},{"osd":4,"last update":"Thu Jan 11 09:46:42 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.74099999999999999,"15min":0.78500000000000003},"min":{"1min":0.502,"5min":0.442,"15min":0.442},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.76000000000000001,"15min":0.80600000000000005},"min":{"1min":0.57799999999999996,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.98999999999999999,"5min":1.171,"15min":1.1830000000000001},"last":0.70299999999999996}]},{"osd":5,"last update":"Thu Jan 11 09:46:42 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.77500000000000002,"15min":0.81299999999999994},"min":{"1min":0.56599999999999995,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.747},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.79500000000000004,"15min":0.84299999999999997},"min":{"1min":0.56100000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.3440000000000001},"last":0.84799999999999998}]},{"osd":6,"last update":"Thu Jan 11 09:46:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.81299999999999994,"15min":0.85299999999999998},"min":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.181,"5min":1.181,"15min":1.2170000000000001},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.79300000000000004,"15min":0.82499999999999996},"min":{"1min":0.57399999999999995,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.883,"5min":1.883,"15min":1.883},"last":0.78900000000000003}]},{"osd":7,"last update":"Thu Jan 11 09:46:42 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.82099999999999995,"15min":0.86299999999999999},"min":{"1min":0.54800000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.9319999999999999,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.83799999999999997,"15min":0.88800000000000001},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.877}]}]},{"osd":4,"up_from":13,"seq":55834575992,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312744,"kb_used_data":3168,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059096,"statfs":{"total":96636764160,"available":96316514304,"internally_reserved":0,"allocated":3244032,"data_stored":1568469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.72299999999999998,"15min":0.76000000000000001},"min":{"1min":0.42099999999999999,"5min":0.41199999999999998,"15min":0.39900000000000002},"max":{"1min":0.85799999999999998,"5min":0.97799999999999998,"15min":1.119},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.73799999999999999,"15min":0.76700000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86399999999999999,"5min":1.853,"15min":1.853},"last":0.70299999999999996}]},{"osd":1,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.69899999999999995,"15min":0.73499999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.83599999999999997,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.69799999999999995,"15min":0.72699999999999998},"min":{"1min":0.438,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81399999999999995,"5min":1.639,"15min":1.639},"last":0.72199999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.71999999999999997,"15min":0.746},"min":{"1min":0.50600000000000001,"5min":0.46500000000000002,"15min":0.44700000000000001},"max":{"1min":0.76400000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.71999999999999997,"15min":0.74199999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.93300000000000005,"5min":1.0249999999999999,"15min":1.0700000000000001},"last":0.60399999999999998}]},{"osd":3,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.70199999999999996,"15min":0.73299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.628},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.75900000000000001,"15min":0.77300000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.97499999999999998,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.81000000000000005}]},{"osd":5,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63500000000000001,"15min":0.623},"min":{"1min":0.41999999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.89900000000000002,"5min":0.91500000000000004,"15min":0.97199999999999998},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.66700000000000004,"15min":0.64700000000000002},"min":{"1min":0.45900000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95099999999999996,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.56799999999999995}]},{"osd":6,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.68999999999999995,"15min":0.67300000000000004},"min":{"1min":0.441,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.012,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.71899999999999997,"15min":0.70499999999999996},"min":{"1min":0.46000000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0609999999999999,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.754}]},{"osd":7,"last update":"Thu Jan 11 09:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.72899999999999998,"15min":0.70199999999999996},"min":{"1min":0.495,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.075,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.752,"5min":0.77900000000000003,"15min":0.754},"min":{"1min":0.56699999999999995,"5min":0.436,"15min":0.41699999999999998},"max":{"1min":1.0409999999999999,"5min":2.052,"15min":2.052},"last":0.73799999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575992,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312756,"kb_used_data":3180,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059084,"statfs":{"total":96636764160,"available":96316502016,"internally_reserved":0,"allocated":3256320,"data_stored":1570537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.71399999999999997,"15min":0.72999999999999998},"min":{"1min":0.42899999999999999,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.72599999999999998,"15min":0.72999999999999998},"min":{"1min":0.46400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.9900000000000002,"5min":2.9900000000000002,"15min":2.9900000000000002},"last":0.68700000000000006}]},{"osd":1,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.71799999999999997,"15min":0.73499999999999999},"min":{"1min":0.46000000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.873,"5min":0.75600000000000001,"15min":0.748},"min":{"1min":0.47699999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":5.4039999999999999,"5min":5.4039999999999999,"15min":5.4039999999999999},"last":0.69999999999999996}]},{"osd":2,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.54300000000000004,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.7120000000000002,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.627},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.73999999999999999,"15min":0.74299999999999999},"min":{"1min":0.51700000000000002,"5min":0.439,"15min":0.439},"max":{"1min":2.5099999999999998,"5min":2.5099999999999998,"15min":2.5099999999999998},"last":0.76900000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.745,"15min":0.75700000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.7799999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":0.752},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.79900000000000004,"15min":0.78300000000000003},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":5.5570000000000004,"5min":5.5570000000000004,"15min":5.5570000000000004},"last":0.73399999999999999}]},{"osd":4,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.64600000000000002,"15min":0.61699999999999999},"min":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41699999999999998},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.66500000000000004,"15min":0.63800000000000001},"min":{"1min":0.40400000000000003,"5min":0.40200000000000002,"15min":0.375},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":0.64600000000000002}]},{"osd":6,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.67400000000000004,"15min":0.65000000000000002},"min":{"1min":0.42899999999999999,"5min":0.42099999999999999,"15min":0.40999999999999998},"max":{"1min":2.9500000000000002,"5min":2.9500000000000002,"15min":2.9500000000000002},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.6280000000000001,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":0.71699999999999997}]},{"osd":7,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.748,"15min":0.73199999999999998},"min":{"1min":0.52700000000000002,"5min":0.47299999999999998,"15min":0.434},"max":{"1min":2.399,"5min":2.399,"15min":2.399},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.80400000000000005,"15min":0.78200000000000003},"min":{"1min":0.61399999999999999,"5min":0.48799999999999999,"15min":0.47599999999999998},"max":{"1min":2.7570000000000001,"5min":2.7570000000000001,"15min":2.7570000000000001},"last":0.84099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:47:29.868 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-11T09:47:29.868 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-11T09:47:30.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.120+0000 7f7f1efff700 1 Processor -- start 2024-01-11T09:47:30.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.120+0000 7f7f1efff700 1 -- start start 2024-01-11T09:47:30.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.121+0000 7f7f1efff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7f1806bba0 0x7f7f180cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:30.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.121+0000 7f7f1efff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7f1806bf70 0x7f7f18159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:30.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.121+0000 7f7f1efff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f1815db00 0x7f7f1815df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:30.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.122+0000 7f7f1efff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f18069fc0 con 0x7f7f1806bf70 2024-01-11T09:47:30.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.122+0000 7f7f1efff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f180cce10 con 0x7f7f1806bba0 2024-01-11T09:47:30.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.122+0000 7f7f1efff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1815aea0 con 0x7f7f1815db00 2024-01-11T09:47:30.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.122+0000 7f7f17fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7f1806bf70 0x7f7f18159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:30.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.122+0000 7f7f17fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7f1806bf70 0x7f7f18159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56214/0 (socket says 172.21.15.27:56214) 2024-01-11T09:47:30.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.122+0000 7f7f17fff700 1 -- 172.21.15.27:0/2076833402 learned_addr learned my addr 172.21.15.27:0/2076833402 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:30.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.122+0000 7f7f1d59c700 1 --2- 172.21.15.27:0/2076833402 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f1815db00 0x7f7f1815df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:30.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.122+0000 7f7f1cd9b700 1 --2- 172.21.15.27:0/2076833402 >> v2:172.21.15.77:3300/0 conn(0x7f7f1806bba0 0x7f7f180cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:30.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.122+0000 7f7f17fff700 1 -- 172.21.15.27:0/2076833402 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f1815db00 msgr2=0x7f7f1815df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:30.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.122+0000 7f7f17fff700 1 --2- 172.21.15.27:0/2076833402 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f1815db00 0x7f7f1815df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:30.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.122+0000 7f7f17fff700 1 -- 172.21.15.27:0/2076833402 >> v2:172.21.15.77:3300/0 conn(0x7f7f1806bba0 msgr2=0x7f7f180cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:30.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.122+0000 7f7f17fff700 1 --2- 172.21.15.27:0/2076833402 >> v2:172.21.15.77:3300/0 conn(0x7f7f1806bba0 0x7f7f180cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:30.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.122+0000 7f7f17fff700 1 -- 172.21.15.27:0/2076833402 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f1815d670 con 0x7f7f1806bf70 2024-01-11T09:47:30.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.122+0000 7f7f1d59c700 1 --2- 172.21.15.27:0/2076833402 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f1815db00 0x7f7f1815df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:30.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.122+0000 7f7f1cd9b700 1 --2- 172.21.15.27:0/2076833402 >> v2:172.21.15.77:3300/0 conn(0x7f7f1806bba0 0x7f7f180cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:30.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.123+0000 7f7f17fff700 1 --2- 172.21.15.27:0/2076833402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f1806bf70 0x7f7f18159e40 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f7f180adc70 tx=0x7f7f04009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770b397227a5f4f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:30.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.123+0000 7f7f177fe700 1 -- 172.21.15.27:0/2076833402 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0400b820 con 0x7f7f1806bf70 2024-01-11T09:47:30.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.123+0000 7f7f177fe700 1 -- 172.21.15.27:0/2076833402 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0400b980 con 0x7f7f1806bf70 2024-01-11T09:47:30.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.123+0000 7f7f177fe700 1 -- 172.21.15.27:0/2076833402 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0400bd20 con 0x7f7f1806bf70 2024-01-11T09:47:30.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.123+0000 7f7f1efff700 1 -- 172.21.15.27:0/2076833402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f1806bf70 msgr2=0x7f7f18159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:30.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.123+0000 7f7f1efff700 1 --2- 172.21.15.27:0/2076833402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f1806bf70 0x7f7f18159e40 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f7f180adc70 tx=0x7f7f04009250 comp rx=0 tx=0).stop 2024-01-11T09:47:30.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.123+0000 7f7f1efff700 1 -- 172.21.15.27:0/2076833402 shutdown_connections 2024-01-11T09:47:30.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.123+0000 7f7f1efff700 1 --2- 172.21.15.27:0/2076833402 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f1815db00 0x7f7f1815df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:30.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.123+0000 7f7f1efff700 1 --2- 172.21.15.27:0/2076833402 >> v2:172.21.15.77:3300/0 conn(0x7f7f1806bba0 0x7f7f180cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:30.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.123+0000 7f7f1efff700 1 --2- 172.21.15.27:0/2076833402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f1806bf70 0x7f7f18159e40 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:30.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.123+0000 7f7f1efff700 1 -- 172.21.15.27:0/2076833402 >> 172.21.15.27:0/2076833402 conn(0x7f7f180ce3e0 msgr2=0x7f7f180ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:30.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.124+0000 7f7f1efff700 1 -- 172.21.15.27:0/2076833402 shutdown_connections 2024-01-11T09:47:30.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.124+0000 7f7f1efff700 1 -- 172.21.15.27:0/2076833402 wait complete. 2024-01-11T09:47:30.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.126+0000 7f7f1efff700 1 Processor -- start 2024-01-11T09:47:30.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.126+0000 7f7f1efff700 1 -- start start 2024-01-11T09:47:30.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.127+0000 7f7f1efff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f1806bba0 0x7f7f180b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:30.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.127+0000 7f7f1efff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f1806bf70 0x7f7f180cad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:30.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.127+0000 7f7f1efff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f1815db00 0x7f7f18157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:30.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.127+0000 7f7f1efff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f180ca270 con 0x7f7f1815db00 2024-01-11T09:47:30.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.127+0000 7f7f1efff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f180c93e0 con 0x7f7f1806bf70 2024-01-11T09:47:30.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.127+0000 7f7f1efff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f180c9660 con 0x7f7f1806bba0 2024-01-11T09:47:30.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.127+0000 7f7f1d59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f1815db00 0x7f7f18157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:30.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.127+0000 7f7f17fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f1806bf70 0x7f7f180cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:30.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.128+0000 7f7f1cd9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f1806bba0 0x7f7f180b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:30.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.128+0000 7f7f17fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f1806bf70 0x7f7f180cad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:52074/0 (socket says 172.21.15.27:52074) 2024-01-11T09:47:30.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.128+0000 7f7f1d59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f1815db00 0x7f7f18157440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56226/0 (socket says 172.21.15.27:56226) 2024-01-11T09:47:30.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.128+0000 7f7f17fff700 1 -- 172.21.15.27:0/2807971769 learned_addr learned my addr 172.21.15.27:0/2807971769 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:30.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.128+0000 7f7f1d59c700 1 -- 172.21.15.27:0/2807971769 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f1806bba0 msgr2=0x7f7f180b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:30.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.128+0000 7f7f1d59c700 1 --2- 172.21.15.27:0/2807971769 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f1806bba0 0x7f7f180b39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:30.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.128+0000 7f7f1d59c700 1 -- 172.21.15.27:0/2807971769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f1806bf70 msgr2=0x7f7f180cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:30.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.128+0000 7f7f1d59c700 1 --2- 172.21.15.27:0/2807971769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f1806bf70 0x7f7f180cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:30.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.128+0000 7f7f1d59c700 1 -- 172.21.15.27:0/2807971769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f040094f0 con 0x7f7f1815db00 2024-01-11T09:47:30.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.128+0000 7f7f1d59c700 1 --2- 172.21.15.27:0/2807971769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f1815db00 0x7f7f18157440 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f7f1000cf30 tx=0x7f7f10007840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:30.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.128+0000 7f7f14ff9700 1 -- 172.21.15.27:0/2807971769 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1000ae40 con 0x7f7f1815db00 2024-01-11T09:47:30.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.128+0000 7f7f14ff9700 1 -- 172.21.15.27:0/2807971769 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f10015440 con 0x7f7f1815db00 2024-01-11T09:47:30.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.128+0000 7f7f14ff9700 1 -- 172.21.15.27:0/2807971769 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f10015800 con 0x7f7f1815db00 2024-01-11T09:47:30.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.129+0000 7f7f1efff700 1 -- 172.21.15.27:0/2807971769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f180c9940 con 0x7f7f1815db00 2024-01-11T09:47:30.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.129+0000 7f7f1efff700 1 -- 172.21.15.27:0/2807971769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f180c9f00 con 0x7f7f1815db00 2024-01-11T09:47:30.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.130+0000 7f7f1efff700 1 -- 172.21.15.27:0/2807971769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f1804c3c0 con 0x7f7f1815db00 2024-01-11T09:47:30.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.131+0000 7f7f14ff9700 1 -- 172.21.15.27:0/2807971769 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f7f1000d440 con 0x7f7f1815db00 2024-01-11T09:47:30.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.131+0000 7f7f14ff9700 1 --2- 172.21.15.27:0/2807971769 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7efc067170 0x7f7efc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:30.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.132+0000 7f7f14ff9700 1 -- 172.21.15.27:0/2807971769 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f7f10085550 con 0x7f7f1815db00 2024-01-11T09:47:30.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.132+0000 7f7f1cd9b700 1 --2- 172.21.15.27:0/2807971769 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7efc067170 0x7f7efc069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:30.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.133+0000 7f7f1cd9b700 1 --2- 172.21.15.27:0/2807971769 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7efc067170 0x7f7efc069620 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00ac20 tx=0x7f7f0c008040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:30.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.135+0000 7f7f14ff9700 1 -- 172.21.15.27:0/2807971769 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7f10052420 con 0x7f7f1815db00 2024-01-11T09:47:30.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.417+0000 7f7f1efff700 1 -- 172.21.15.27:0/2807971769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f7f1806c7a0 con 0x7f7f1815db00 2024-01-11T09:47:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.422+0000 7f7f14ff9700 1 -- 172.21.15.27:0/2807971769 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v53) v1 ==== 190+0+0 (secure 0 0 0) 0x7f7f10055a40 con 0x7f7f1815db00 2024-01-11T09:47:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.424+0000 7f7f1efff700 1 -- 172.21.15.27:0/2807971769 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7efc067170 msgr2=0x7f7efc069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.424+0000 7f7f1efff700 1 --2- 172.21.15.27:0/2807971769 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7efc067170 0x7f7efc069620 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00ac20 tx=0x7f7f0c008040 comp rx=0 tx=0).stop 2024-01-11T09:47:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.424+0000 7f7f1efff700 1 -- 172.21.15.27:0/2807971769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f1815db00 msgr2=0x7f7f18157440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.424+0000 7f7f1efff700 1 --2- 172.21.15.27:0/2807971769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f1815db00 0x7f7f18157440 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f7f1000cf30 tx=0x7f7f10007840 comp rx=0 tx=0).stop 2024-01-11T09:47:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.424+0000 7f7f1efff700 1 -- 172.21.15.27:0/2807971769 shutdown_connections 2024-01-11T09:47:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.424+0000 7f7f1efff700 1 --2- 172.21.15.27:0/2807971769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f1815db00 0x7f7f18157440 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.424+0000 7f7f1efff700 1 --2- 172.21.15.27:0/2807971769 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7f1806bba0 0x7f7f180b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.424+0000 7f7f1efff700 1 --2- 172.21.15.27:0/2807971769 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7efc067170 0x7f7efc069620 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:30.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.424+0000 7f7f1efff700 1 --2- 172.21.15.27:0/2807971769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f1806bf70 0x7f7f180cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:30.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.424+0000 7f7f1efff700 1 -- 172.21.15.27:0/2807971769 >> 172.21.15.27:0/2807971769 conn(0x7f7f180ce3e0 msgr2=0x7f7f1815b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:30.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.424+0000 7f7f1efff700 1 -- 172.21.15.27:0/2807971769 shutdown_connections 2024-01-11T09:47:30.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.425+0000 7f7f1efff700 1 -- 172.21.15.27:0/2807971769 wait complete. 2024-01-11T09:47:30.417 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-11T09:47:30.434 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:47:30.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.642+0000 7ff2be3ff700 1 Processor -- start 2024-01-11T09:47:30.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.643+0000 7ff2be3ff700 1 -- start start 2024-01-11T09:47:30.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.644+0000 7ff2be3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff2b80ca3e0 0x7ff2b80cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:30.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.644+0000 7ff2be3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b815d640 con 0x7ff2b80ca7b0 2024-01-11T09:47:30.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.644+0000 7ff2be3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b815a5d0 con 0x7ff2b8159e80 2024-01-11T09:47:30.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.644+0000 7ff2be3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2b815ae70 con 0x7ff2b80ca3e0 2024-01-11T09:47:30.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.644+0000 7ff2b7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff2b80ca3e0 0x7ff2b80cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:30.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.644+0000 7ff2b77fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff2b80ca7b0 0x7ff2b80c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51302/0 (socket says 172.21.15.27:51302) 2024-01-11T09:47:30.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.644+0000 7ff2b77fe700 1 -- 172.21.15.27:0/1117332912 learned_addr learned my addr 172.21.15.27:0/1117332912 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:30.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.645+0000 7ff2b7fff700 1 -- 172.21.15.27:0/1117332912 >> v1:172.21.15.77:6789/0 conn(0x7ff2b8159e80 legacy=0x7ff2b815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:30.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.645+0000 7ff2b7fff700 1 -- 172.21.15.27:0/1117332912 >> v1:172.21.15.27:6789/0 conn(0x7ff2b80ca7b0 legacy=0x7ff2b80c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:30.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.645+0000 7ff2b7fff700 1 -- 172.21.15.27:0/1117332912 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b815a8b0 con 0x7ff2b80ca3e0 2024-01-11T09:47:30.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.645+0000 7ff2b6ffd700 1 -- 172.21.15.27:0/1117332912 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708578485 0 0) 0x7ff2b815d640 con 0x7ff2b80ca7b0 2024-01-11T09:47:30.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.645+0000 7ff2b7fff700 1 --2- 172.21.15.27:0/1117332912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2b80ca3e0 0x7ff2b80cca20 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8009910 tx=0x7ff2a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8d2f45d2009ea6d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:30.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.645+0000 7ff2b6ffd700 1 -- 172.21.15.27:0/1117332912 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a800b820 con 0x7ff2b80ca3e0 2024-01-11T09:47:30.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.645+0000 7ff2b6ffd700 1 -- 172.21.15.27:0/1117332912 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2a800b980 con 0x7ff2b80ca3e0 2024-01-11T09:47:30.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.645+0000 7ff2b6ffd700 1 -- 172.21.15.27:0/1117332912 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a800bd20 con 0x7ff2b80ca3e0 2024-01-11T09:47:30.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.646+0000 7ff2be3ff700 1 -- 172.21.15.27:0/1117332912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2b80ca3e0 msgr2=0x7ff2b80cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:30.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.646+0000 7ff2be3ff700 1 --2- 172.21.15.27:0/1117332912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2b80ca3e0 0x7ff2b80cca20 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8009910 tx=0x7ff2a80091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:30.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.646+0000 7ff2be3ff700 1 -- 172.21.15.27:0/1117332912 shutdown_connections 2024-01-11T09:47:30.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.646+0000 7ff2be3ff700 1 --2- 172.21.15.27:0/1117332912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2b80ca3e0 0x7ff2b80cca20 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:30.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.646+0000 7ff2be3ff700 1 -- 172.21.15.27:0/1117332912 >> 172.21.15.27:0/1117332912 conn(0x7ff2b80ce3e0 msgr2=0x7ff2b80ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:30.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.646+0000 7ff2be3ff700 1 -- 172.21.15.27:0/1117332912 shutdown_connections 2024-01-11T09:47:30.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.646+0000 7ff2be3ff700 1 -- 172.21.15.27:0/1117332912 wait complete. 2024-01-11T09:47:30.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.647+0000 7ff2be3ff700 1 Processor -- start 2024-01-11T09:47:30.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.647+0000 7ff2be3ff700 1 -- start start 2024-01-11T09:47:30.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.648+0000 7ff2be3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2b80ca3e0 0x7ff2b80b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:30.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.648+0000 7ff2be3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2b80ca7b0 0x7ff2b8150720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:30.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.648+0000 7ff2be3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2b8159e80 0x7ff2b815fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:30.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.648+0000 7ff2be3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b815dee0 con 0x7ff2b8159e80 2024-01-11T09:47:30.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.648+0000 7ff2be3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b8150c60 con 0x7ff2b80ca3e0 2024-01-11T09:47:30.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.648+0000 7ff2be3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b80b3b40 con 0x7ff2b80ca7b0 2024-01-11T09:47:30.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.649+0000 7ff2bc99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2b8159e80 0x7ff2b815fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:30.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.649+0000 7ff2bc99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2b8159e80 0x7ff2b815fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56246/0 (socket says 172.21.15.27:56246) 2024-01-11T09:47:30.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.649+0000 7ff2bc99c700 1 -- 172.21.15.27:0/3038306015 learned_addr learned my addr 172.21.15.27:0/3038306015 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:30.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.649+0000 7ff2b7fff700 1 --2- 172.21.15.27:0/3038306015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2b80ca3e0 0x7ff2b80b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:30.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.649+0000 7ff2b77fe700 1 --2- 172.21.15.27:0/3038306015 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2b80ca7b0 0x7ff2b8150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:30.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.649+0000 7ff2bc99c700 1 -- 172.21.15.27:0/3038306015 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2b80ca7b0 msgr2=0x7ff2b8150720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:30.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.649+0000 7ff2bc99c700 1 --2- 172.21.15.27:0/3038306015 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2b80ca7b0 0x7ff2b8150720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:30.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.649+0000 7ff2bc99c700 1 -- 172.21.15.27:0/3038306015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2b80ca3e0 msgr2=0x7ff2b80b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:30.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.649+0000 7ff2bc99c700 1 --2- 172.21.15.27:0/3038306015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2b80ca3e0 0x7ff2b80b1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:30.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.649+0000 7ff2bc99c700 1 -- 172.21.15.27:0/3038306015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a8009420 con 0x7ff2b8159e80 2024-01-11T09:47:30.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.649+0000 7ff2b7fff700 1 --2- 172.21.15.27:0/3038306015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2b80ca3e0 0x7ff2b80b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:30.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.649+0000 7ff2b77fe700 1 --2- 172.21.15.27:0/3038306015 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2b80ca7b0 0x7ff2b8150720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:30.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.649+0000 7ff2bc99c700 1 --2- 172.21.15.27:0/3038306015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2b8159e80 0x7ff2b815fa30 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7ff2b000c910 tx=0x7ff2b0001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:30.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.650+0000 7ff29ffff700 1 -- 172.21.15.27:0/3038306015 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b000a7a0 con 0x7ff2b8159e80 2024-01-11T09:47:30.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.650+0000 7ff29ffff700 1 -- 172.21.15.27:0/3038306015 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b000a900 con 0x7ff2b8159e80 2024-01-11T09:47:30.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.650+0000 7ff2be3ff700 1 -- 172.21.15.27:0/3038306015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b80b40c0 con 0x7ff2b8159e80 2024-01-11T09:47:30.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.650+0000 7ff29ffff700 1 -- 172.21.15.27:0/3038306015 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b000acc0 con 0x7ff2b8159e80 2024-01-11T09:47:30.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.650+0000 7ff2be3ff700 1 -- 172.21.15.27:0/3038306015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b815db70 con 0x7ff2b8159e80 2024-01-11T09:47:30.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.651+0000 7ff2be3ff700 1 -- 172.21.15.27:0/3038306015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b815e050 con 0x7ff2b8159e80 2024-01-11T09:47:30.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.652+0000 7ff29ffff700 1 -- 172.21.15.27:0/3038306015 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7ff2b0012630 con 0x7ff2b8159e80 2024-01-11T09:47:30.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.653+0000 7ff29ffff700 1 --2- 172.21.15.27:0/3038306015 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff2a4067170 0x7ff2a4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:30.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.653+0000 7ff29ffff700 1 -- 172.21.15.27:0/3038306015 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 5817+0+0 (secure 0 0 0) 0x7ff2b0084bf0 con 0x7ff2b8159e80 2024-01-11T09:47:30.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.653+0000 7ff2b7fff700 1 --2- 172.21.15.27:0/3038306015 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff2a4067170 0x7ff2a4069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:30.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.654+0000 7ff2b7fff700 1 --2- 172.21.15.27:0/3038306015 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff2a4067170 0x7ff2a4069620 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80093f0 tx=0x7ff2a8017000 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:30.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.657+0000 7ff29ffff700 1 -- 172.21.15.27:0/3038306015 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff2b00555f0 con 0x7ff2b8159e80 2024-01-11T09:47:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.904+0000 7ff2be3ff700 1 -- 172.21.15.27:0/3038306015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2b80718d0 con 0x7ff2b8159e80 2024-01-11T09:47:30.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.905+0000 7ff29ffff700 1 -- 172.21.15.27:0/3038306015 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+4612 (secure 0 0 0) 0x7ff2b0055180 con 0x7ff2b8159e80 2024-01-11T09:47:30.896 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:30.896 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":141,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9866,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1060058066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1060058066},{"type":"v1","addr":"172.21.15.77:6835","nonce":1060058066}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137},{"gid":11626,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/4169546841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":4169546841},{"type":"v1","addr":"172.21.15.77:6834","nonce":4169546841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137},{"gid":11632,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/2567971522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2567971522},{"type":"v1","addr":"172.21.15.27:6833","nonce":2567971522}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:47:24.474082+0000","modified":"2024-01-11T09:47:27.985712+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12063},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12063":{"gid":12063,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1065634442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1065634442},{"type":"v1","addr":"172.21.15.27:6835","nonce":1065634442}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2024-01-11T09:47:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.907+0000 7ff2be3ff700 1 -- 172.21.15.27:0/3038306015 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff2a4067170 msgr2=0x7ff2a4069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.907+0000 7ff2be3ff700 1 --2- 172.21.15.27:0/3038306015 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff2a4067170 0x7ff2a4069620 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80093f0 tx=0x7ff2a8017000 comp rx=0 tx=0).stop 2024-01-11T09:47:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.907+0000 7ff2be3ff700 1 -- 172.21.15.27:0/3038306015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2b8159e80 msgr2=0x7ff2b815fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.907+0000 7ff2be3ff700 1 --2- 172.21.15.27:0/3038306015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2b8159e80 0x7ff2b815fa30 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7ff2b000c910 tx=0x7ff2b0001710 comp rx=0 tx=0).stop 2024-01-11T09:47:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.907+0000 7ff2be3ff700 1 -- 172.21.15.27:0/3038306015 shutdown_connections 2024-01-11T09:47:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.907+0000 7ff2be3ff700 1 --2- 172.21.15.27:0/3038306015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2b8159e80 0x7ff2b815fa30 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.907+0000 7ff2be3ff700 1 --2- 172.21.15.27:0/3038306015 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff2a4067170 0x7ff2a4069620 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.907+0000 7ff2be3ff700 1 --2- 172.21.15.27:0/3038306015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2b80ca3e0 0x7ff2b80b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.907+0000 7ff2be3ff700 1 --2- 172.21.15.27:0/3038306015 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2b80ca7b0 0x7ff2b8150720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.907+0000 7ff2be3ff700 1 -- 172.21.15.27:0/3038306015 >> 172.21.15.27:0/3038306015 conn(0x7ff2b80ce3e0 msgr2=0x7ff2b80c3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.907+0000 7ff2be3ff700 1 -- 172.21.15.27:0/3038306015 shutdown_connections 2024-01-11T09:47:30.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:30.907+0000 7ff2be3ff700 1 -- 172.21.15.27:0/3038306015 wait complete. 2024-01-11T09:47:30.900 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 141 2024-01-11T09:47:31.320 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 141, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-11T09:47:24.474082+0000', 'modified': '2024-01-11T09:47:27.985712+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12063}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12063': {'gid': 12063, 'name': 'c', 'rank': 0, 'incarnation': 138, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6835/1065634442', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6834', 'nonce': 1065634442}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 1065634442}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2024-01-11T09:47:31.320 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-11T09:47:31.320 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-11T09:47:31.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.532+0000 7fc4d8bff700 1 Processor -- start 2024-01-11T09:47:31.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.533+0000 7fc4d8bff700 1 -- start start 2024-01-11T09:47:31.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.534+0000 7fc4d8bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4d4068720 0x7fc4d4068b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.534+0000 7fc4d8bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d415d290 con 0x7fc4d406bf10 2024-01-11T09:47:31.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.534+0000 7fc4d8bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d415d570 con 0x7fc4d40b0a40 2024-01-11T09:47:31.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.534+0000 7fc4d8bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d4069080 con 0x7fc4d4068720 2024-01-11T09:47:31.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.534+0000 7fc4d259c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc4d406bf10 0x7fc4d4159c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51312/0 (socket says 172.21.15.27:51312) 2024-01-11T09:47:31.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.534+0000 7fc4d259c700 1 -- 172.21.15.27:0/1585233322 learned_addr learned my addr 172.21.15.27:0/1585233322 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:31.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.534+0000 7fc4d2d9d700 1 --2- 172.21.15.27:0/1585233322 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4d4068720 0x7fc4d4068b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.535+0000 7fc4d159a700 1 -- 172.21.15.27:0/1585233322 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707857751 0 0) 0x7fc4d415d290 con 0x7fc4d406bf10 2024-01-11T09:47:31.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.535+0000 7fc4d159a700 1 -- 172.21.15.27:0/1585233322 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4bc003570 con 0x7fc4d406bf10 2024-01-11T09:47:31.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.535+0000 7fc4d2d9d700 1 -- 172.21.15.27:0/1585233322 >> v1:172.21.15.77:6789/0 conn(0x7fc4d40b0a40 legacy=0x7fc4d415da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:31.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.535+0000 7fc4d159a700 1 -- 172.21.15.27:0/1585233322 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910090761 0 0) 0x7fc4d415d570 con 0x7fc4d40b0a40 2024-01-11T09:47:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.535+0000 7fc4d2d9d700 1 -- 172.21.15.27:0/1585233322 >> v1:172.21.15.27:6789/0 conn(0x7fc4d406bf10 legacy=0x7fc4d4159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.535+0000 7fc4d2d9d700 1 -- 172.21.15.27:0/1585233322 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d4154180 con 0x7fc4d4068720 2024-01-11T09:47:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.535+0000 7fc4d2d9d700 1 --2- 172.21.15.27:0/1585233322 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4d4068720 0x7fc4d4068b40 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc009e90 tx=0x7fc4cc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7afa48599f628a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.535+0000 7fc4d159a700 1 -- 172.21.15.27:0/1585233322 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc00e070 con 0x7fc4d4068720 2024-01-11T09:47:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.535+0000 7fc4d159a700 1 -- 172.21.15.27:0/1585233322 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4cc00bc30 con 0x7fc4d4068720 2024-01-11T09:47:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.535+0000 7fc4d159a700 1 -- 172.21.15.27:0/1585233322 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc012430 con 0x7fc4d4068720 2024-01-11T09:47:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.536+0000 7fc4d8bff700 1 -- 172.21.15.27:0/1585233322 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4d4068720 msgr2=0x7fc4d4068b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.536+0000 7fc4d8bff700 1 --2- 172.21.15.27:0/1585233322 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4d4068720 0x7fc4d4068b40 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc009e90 tx=0x7fc4cc009780 comp rx=0 tx=0).stop 2024-01-11T09:47:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.536+0000 7fc4d8bff700 1 -- 172.21.15.27:0/1585233322 shutdown_connections 2024-01-11T09:47:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.536+0000 7fc4d8bff700 1 --2- 172.21.15.27:0/1585233322 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4d4068720 0x7fc4d4068b40 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.536+0000 7fc4d8bff700 1 -- 172.21.15.27:0/1585233322 >> 172.21.15.27:0/1585233322 conn(0x7fc4d40ce3c0 msgr2=0x7fc4d40ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.536+0000 7fc4d8bff700 1 -- 172.21.15.27:0/1585233322 shutdown_connections 2024-01-11T09:47:31.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.536+0000 7fc4d8bff700 1 -- 172.21.15.27:0/1585233322 wait complete. 2024-01-11T09:47:31.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.538+0000 7fc4d8bff700 1 Processor -- start 2024-01-11T09:47:31.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.538+0000 7fc4d8bff700 1 -- start start 2024-01-11T09:47:31.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.539+0000 7fc4d8bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4d4068720 0x7fc4d40c4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.539+0000 7fc4d8bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4d406bf10 0x7fc4d40b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.539+0000 7fc4d8bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d40b0a40 0x7fc4d40c2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.539+0000 7fc4d8bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d40b3600 con 0x7fc4d406bf10 2024-01-11T09:47:31.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.539+0000 7fc4d8bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d40b16b0 con 0x7fc4d40b0a40 2024-01-11T09:47:31.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.539+0000 7fc4d8bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d40b37f0 con 0x7fc4d4068720 2024-01-11T09:47:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.539+0000 7fc4d1d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4d406bf10 0x7fc4d40b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.539+0000 7fc4d1d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4d406bf10 0x7fc4d40b1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56260/0 (socket says 172.21.15.27:56260) 2024-01-11T09:47:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.539+0000 7fc4d1d9b700 1 -- 172.21.15.27:0/1101449111 learned_addr learned my addr 172.21.15.27:0/1101449111 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.539+0000 7fc4d259c700 1 --2- 172.21.15.27:0/1101449111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4d4068720 0x7fc4d40c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.539+0000 7fc4d2d9d700 1 --2- 172.21.15.27:0/1101449111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d40b0a40 0x7fc4d40c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.539+0000 7fc4d1d9b700 1 -- 172.21.15.27:0/1101449111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4d4068720 msgr2=0x7fc4d40c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.539+0000 7fc4d1d9b700 1 --2- 172.21.15.27:0/1101449111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4d4068720 0x7fc4d40c4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.539+0000 7fc4d1d9b700 1 -- 172.21.15.27:0/1101449111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d40b0a40 msgr2=0x7fc4d40c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.539+0000 7fc4d1d9b700 1 --2- 172.21.15.27:0/1101449111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d40b0a40 0x7fc4d40c2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.539+0000 7fc4d1d9b700 1 -- 172.21.15.27:0/1101449111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4cc009a20 con 0x7fc4d406bf10 2024-01-11T09:47:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.539+0000 7fc4d259c700 1 --2- 172.21.15.27:0/1101449111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4d4068720 0x7fc4d40c4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.539+0000 7fc4d2d9d700 1 --2- 172.21.15.27:0/1101449111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d40b0a40 0x7fc4d40c2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.540+0000 7fc4d1d9b700 1 --2- 172.21.15.27:0/1101449111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4d406bf10 0x7fc4d40b1170 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fc4c800c7e0 tx=0x7fc4c8007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.540+0000 7fc4c27fc700 1 -- 172.21.15.27:0/1101449111 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8012700 con 0x7fc4d406bf10 2024-01-11T09:47:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.540+0000 7fc4c27fc700 1 -- 172.21.15.27:0/1101449111 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c8012880 con 0x7fc4d406bf10 2024-01-11T09:47:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.540+0000 7fc4d8bff700 1 -- 172.21.15.27:0/1101449111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d40b3ad0 con 0x7fc4d406bf10 2024-01-11T09:47:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.540+0000 7fc4c27fc700 1 -- 172.21.15.27:0/1101449111 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8012c40 con 0x7fc4d406bf10 2024-01-11T09:47:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.540+0000 7fc4d8bff700 1 -- 172.21.15.27:0/1101449111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d4153b50 con 0x7fc4d406bf10 2024-01-11T09:47:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.542+0000 7fc4c27fc700 1 -- 172.21.15.27:0/1101449111 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7fc4c801b400 con 0x7fc4d406bf10 2024-01-11T09:47:31.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.543+0000 7fc4c27fc700 1 --2- 172.21.15.27:0/1101449111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc4b4067170 0x7fc4b4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.543+0000 7fc4c27fc700 1 -- 172.21.15.27:0/1101449111 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 5817+0+0 (secure 0 0 0) 0x7fc4c8084570 con 0x7fc4d406bf10 2024-01-11T09:47:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.544+0000 7fc4d259c700 1 --2- 172.21.15.27:0/1101449111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc4b4067170 0x7fc4b4069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.545+0000 7fc4d259c700 1 --2- 172.21.15.27:0/1101449111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc4b4067170 0x7fc4b4069620 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40058d0 tx=0x7fc4c4009040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.545+0000 7fc4e72dab80 1 Processor -- start 2024-01-11T09:47:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.545+0000 7fc4e72dab80 1 -- start start 2024-01-11T09:47:31.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.546+0000 7fc4e72dab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784febeb20 0x55784fe9fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.546+0000 7fc4e72dab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784fe9a3a0 0x55784fec6860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.546+0000 7fc4e72dab80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeef0 0x55784fec6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.546+0000 7fc4e72dab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55784fdb7310 con 0x55784febeb20 2024-01-11T09:47:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.546+0000 7fc4e72dab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55784fdf4130 con 0x55784fe9a3a0 2024-01-11T09:47:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.546+0000 7fc4e72dab80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55784fdf1af0 con 0x55784febeef0 2024-01-11T09:47:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.546+0000 7fc4d259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784febeb20 0x55784fe9fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.546+0000 7fc4d259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784febeb20 0x55784fe9fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56270/0 (socket says 172.21.15.27:56270) 2024-01-11T09:47:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.546+0000 7fc4d259c700 1 -- 172.21.15.27:0/868114249 learned_addr learned my addr 172.21.15.27:0/868114249 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.546+0000 7fc4d2d9d700 1 --2- 172.21.15.27:0/868114249 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeef0 0x55784fec6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.546+0000 7fc4d1d9b700 1 --2- 172.21.15.27:0/868114249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784fe9a3a0 0x55784fec6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.547+0000 7fc4d259c700 1 -- 172.21.15.27:0/868114249 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeef0 msgr2=0x55784fec6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.547+0000 7fc4d259c700 1 --2- 172.21.15.27:0/868114249 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeef0 0x55784fec6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.547+0000 7fc4d259c700 1 -- 172.21.15.27:0/868114249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784fe9a3a0 msgr2=0x55784fec6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.547+0000 7fc4d259c700 1 --2- 172.21.15.27:0/868114249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784fe9a3a0 0x55784fec6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.547+0000 7fc4d259c700 1 -- 172.21.15.27:0/868114249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55784fce00a0 con 0x55784febeb20 2024-01-11T09:47:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.547+0000 7fc4d1d9b700 1 --2- 172.21.15.27:0/868114249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784fe9a3a0 0x55784fec6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.547+0000 7fc4d259c700 1 --2- 172.21.15.27:0/868114249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784febeb20 0x55784fe9fdf0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4005b60 tx=0x7fc4c4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef04fd64251f3237 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.547+0000 7fc4b37fe700 1 -- 172.21.15.27:0/868114249 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4001520 con 0x55784febeb20 2024-01-11T09:47:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.547+0000 7fc4b37fe700 1 -- 172.21.15.27:0/868114249 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c4007740 con 0x55784febeb20 2024-01-11T09:47:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.547+0000 7fc4b37fe700 1 -- 172.21.15.27:0/868114249 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c400d8c0 con 0x55784febeb20 2024-01-11T09:47:31.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.548+0000 7fc4e72dab80 1 -- 172.21.15.27:0/868114249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784febeb20 msgr2=0x55784fe9fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:31.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.548+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/868114249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784febeb20 0x55784fe9fdf0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4005b60 tx=0x7fc4c4012000 comp rx=0 tx=0).stop 2024-01-11T09:47:31.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.548+0000 7fc4e72dab80 1 -- 172.21.15.27:0/868114249 shutdown_connections 2024-01-11T09:47:31.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.548+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/868114249 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeef0 0x55784fec6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.548+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/868114249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784febeb20 0x55784fe9fdf0 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.548+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/868114249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784fe9a3a0 0x55784fec6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.548+0000 7fc4e72dab80 1 -- 172.21.15.27:0/868114249 >> 172.21.15.27:0/868114249 conn(0x55784fe9c300 msgr2=0x55784fe9ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:31.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.548+0000 7fc4e72dab80 1 -- 172.21.15.27:0/868114249 shutdown_connections 2024-01-11T09:47:31.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.548+0000 7fc4e72dab80 1 -- 172.21.15.27:0/868114249 wait complete. 2024-01-11T09:47:31.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.549+0000 7fc4e72dab80 1 -- start start 2024-01-11T09:47:31.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.549+0000 7fc4e72dab80 1 Processor -- start 2024-01-11T09:47:31.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.550+0000 7fc4e72dab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784fe9a3a0 0x55784fec1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.550+0000 7fc4e72dab80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeb20 0x55784fec16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.550+0000 7fc4e72dab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784febeef0 0x55784fe99710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.550+0000 7fc4e72dab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55784fded8e0 con 0x55784febeef0 2024-01-11T09:47:31.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.550+0000 7fc4e72dab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55784fd99980 con 0x55784fe9a3a0 2024-01-11T09:47:31.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.550+0000 7fc4e72dab80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55784fd99620 con 0x55784febeb20 2024-01-11T09:47:31.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.550+0000 7fc4e72dab80 10 client.0 _ll_register_callbacks cb 0x55784fbfca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T09:47:31.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.550+0000 7fc4d2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784febeef0 0x55784fe99710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.550+0000 7fc4d2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784febeef0 0x55784fe99710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56286/0 (socket says 172.21.15.27:56286) 2024-01-11T09:47:31.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.550+0000 7fc4d2d9d700 1 -- 172.21.15.27:0/2282671695 learned_addr learned my addr 172.21.15.27:0/2282671695 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:31.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.550+0000 7fc4d1d9b700 1 --2- 172.21.15.27:0/2282671695 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeb20 0x55784fec16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.550+0000 7fc4d259c700 1 --2- 172.21.15.27:0/2282671695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784fe9a3a0 0x55784fec1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.551+0000 7fc4d2d9d700 1 -- 172.21.15.27:0/2282671695 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeb20 msgr2=0x55784fec16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:31.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.551+0000 7fc4d2d9d700 1 --2- 172.21.15.27:0/2282671695 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeb20 0x55784fec16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.551+0000 7fc4d2d9d700 1 -- 172.21.15.27:0/2282671695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784fe9a3a0 msgr2=0x55784fec1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.551+0000 7fc4d2d9d700 1 --2- 172.21.15.27:0/2282671695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784fe9a3a0 0x55784fec1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.551+0000 7fc4d2d9d700 1 -- 172.21.15.27:0/2282671695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4bc003570 con 0x55784febeef0 2024-01-11T09:47:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.551+0000 7fc4d259c700 1 --2- 172.21.15.27:0/2282671695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784fe9a3a0 0x55784fec1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.551+0000 7fc4d2d9d700 1 --2- 172.21.15.27:0/2282671695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784febeef0 0x55784fe99710 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7fc4d406b3c0 tx=0x7fc4cc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58257a0928498443 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.551+0000 7fc4b0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T09:47:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.551+0000 7fc4b0ff9700 1 -- 172.21.15.27:0/2282671695 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc022070 con 0x55784febeef0 2024-01-11T09:47:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.551+0000 7fc4b0ff9700 1 -- 172.21.15.27:0/2282671695 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(256..256 src has 1..256) v4 ==== 5817+0+0 (secure 0 0 0) 0x7fc4cc027b20 con 0x55784febeef0 2024-01-11T09:47:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.551+0000 7fc4e72dab80 1 -- 172.21.15.27:0/2282671695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55784fe92560 con 0x55784febeef0 2024-01-11T09:47:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.552+0000 7fc4b0ff9700 1 -- 172.21.15.27:0/2282671695 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=141) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4cc012430 con 0x55784febeef0 2024-01-11T09:47:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.552+0000 7fc4e72dab80 10 client.12198 fetch_fsmap learned FSMap version 141 2024-01-11T09:47:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.552+0000 7fc4e72dab80 1 -- 172.21.15.27:0/2282671695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=141,monmap=2+}) v3 -- 0x55784fdbc3c0 con 0x55784febeef0 2024-01-11T09:47:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.552+0000 7fc4b0ff9700 1 -- 172.21.15.27:0/2282671695 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55784fe92560 con 0x55784febeef0 2024-01-11T09:47:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.552+0000 7fc4b0ff9700 1 -- 172.21.15.27:0/2282671695 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 141) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fc4cc027020 con 0x55784febeef0 2024-01-11T09:47:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.552+0000 7fc4e72dab80 10 client.12198 fetch_fsmap finished waiting for FSMap version 141 2024-01-11T09:47:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.552+0000 7fc4e72dab80 10 client.12198 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-11T09:47:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.552+0000 7fc4e72dab80 20 client.12198 populate_metadata read hostname 'smithi027' 2024-01-11T09:47:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.552+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/2282671695 >> [v2:172.21.15.27:6834/1065634442,v1:172.21.15.27:6835/1065634442] conn(0x55784fecb470 0x55784fecb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.552+0000 7fc4e72dab80 4 client.12198 mds_command: new command op to 12063 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-11T09:47:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.552+0000 7fc4e72dab80 1 -- 172.21.15.27:0/2282671695 --> [v2:172.21.15.27:6834/1065634442,v1:172.21.15.27:6835/1065634442] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55784fe6f0e0 con 0x55784fecb470 2024-01-11T09:47:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.552+0000 7fc4d259c700 1 --2- 172.21.15.27:0/2282671695 >> [v2:172.21.15.27:6834/1065634442,v1:172.21.15.27:6835/1065634442] conn(0x55784fecb470 0x55784fecb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.553+0000 7fc4d259c700 1 --2- 172.21.15.27:0/2282671695 >> [v2:172.21.15.27:6834/1065634442,v1:172.21.15.27:6835/1065634442] conn(0x55784fecb470 0x55784fecb840 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e082ea5d63c90e4f server_cookie=3762f753a72c3665 in_seq=0 out_seq=0 2024-01-11T09:47:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.553+0000 7fc4b0ff9700 10 client.12198 ms_handle_connect on v2:172.21.15.27:6834/1065634442 2024-01-11T09:47:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.553+0000 7fc4b0ff9700 1 -- 172.21.15.27:0/2282671695 <== mds.0 v2:172.21.15.27:6834/1065634442 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x55784fe6f0e0 con 0x55784fecb470 2024-01-11T09:47:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.553+0000 7fc4b0ff9700 10 client.12198 handle_command_reply: tid=0 2024-01-11T09:47:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.554+0000 7fc4e72dab80 1 client.12198 shutdown 2024-01-11T09:47:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.554+0000 7fc4e72dab80 1 -- 172.21.15.27:0/2282671695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784febeef0 msgr2=0x55784fe99710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.554+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/2282671695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784febeef0 0x55784fe99710 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7fc4d406b3c0 tx=0x7fc4cc019000 comp rx=0 tx=0).stop 2024-01-11T09:47:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.554+0000 7fc4e72dab80 1 -- 172.21.15.27:0/2282671695 shutdown_connections 2024-01-11T09:47:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.554+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/2282671695 >> [v2:172.21.15.27:6834/1065634442,v1:172.21.15.27:6835/1065634442] conn(0x55784fecb470 0x55784fecb840 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.554+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/2282671695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784febeef0 0x55784fe99710 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.554+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/2282671695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784fe9a3a0 0x55784fec1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.554+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/2282671695 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeb20 0x55784fec16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.554+0000 7fc4b0ff9700 0 client.12198 ms_handle_reset on v2:172.21.15.27:6834/1065634442 2024-01-11T09:47:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.554+0000 7fc4e72dab80 1 -- 172.21.15.27:0/2282671695 >> 172.21.15.27:0/2282671695 conn(0x55784fe9c300 msgr2=0x55784fec8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.554+0000 7fc4e72dab80 1 -- 172.21.15.27:0/2282671695 shutdown_connections 2024-01-11T09:47:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.554+0000 7fc4e72dab80 1 -- 172.21.15.27:0/2282671695 wait complete. 2024-01-11T09:47:31.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.554+0000 7fc4e72dab80 20 client.12198 trim_cache size 0 max 16384 2024-01-11T09:47:31.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.572+0000 7fc4e72dab80 1 Processor -- start 2024-01-11T09:47:31.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.572+0000 7fc4e72dab80 1 -- start start 2024-01-11T09:47:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.573+0000 7fc4e72dab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784fe9a3a0 0x55784fec1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.573+0000 7fc4e72dab80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeb20 0x55784fe9a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.573+0000 7fc4e72dab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784febeef0 0x55784fec4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.573+0000 7fc4e72dab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55784fc851e0 con 0x55784fe9a3a0 2024-01-11T09:47:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.573+0000 7fc4e72dab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55784fe96390 con 0x55784febeef0 2024-01-11T09:47:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.573+0000 7fc4e72dab80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55784fef4980 con 0x55784febeb20 2024-01-11T09:47:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.573+0000 7fc4d259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784fe9a3a0 0x55784fec1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.573+0000 7fc4d259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784fe9a3a0 0x55784fec1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56292/0 (socket says 172.21.15.27:56292) 2024-01-11T09:47:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.573+0000 7fc4d259c700 1 -- 172.21.15.27:0/546103110 learned_addr learned my addr 172.21.15.27:0/546103110 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.573+0000 7fc4d1d9b700 1 --2- 172.21.15.27:0/546103110 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeb20 0x55784fe9a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.573+0000 7fc4d2d9d700 1 --2- 172.21.15.27:0/546103110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784febeef0 0x55784fec4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.573+0000 7fc4d259c700 1 -- 172.21.15.27:0/546103110 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeb20 msgr2=0x55784fe9a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.573+0000 7fc4d259c700 1 --2- 172.21.15.27:0/546103110 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeb20 0x55784fe9a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.573+0000 7fc4d259c700 1 -- 172.21.15.27:0/546103110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784febeef0 msgr2=0x55784fec4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.573+0000 7fc4d259c700 1 --2- 172.21.15.27:0/546103110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784febeef0 0x55784fec4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.573+0000 7fc4d259c700 1 -- 172.21.15.27:0/546103110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55784fe9b8b0 con 0x55784fe9a3a0 2024-01-11T09:47:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.574+0000 7fc4d259c700 1 --2- 172.21.15.27:0/546103110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784fe9a3a0 0x55784fec1280 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4005b30 tx=0x7fc4c4023040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bba5230dbed4072 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:31.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.574+0000 7fc4b3fff700 1 -- 172.21.15.27:0/546103110 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4000c30 con 0x55784fe9a3a0 2024-01-11T09:47:31.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.574+0000 7fc4b3fff700 1 -- 172.21.15.27:0/546103110 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c4001520 con 0x55784fe9a3a0 2024-01-11T09:47:31.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.574+0000 7fc4b3fff700 1 -- 172.21.15.27:0/546103110 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c400dcb0 con 0x55784fe9a3a0 2024-01-11T09:47:31.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.574+0000 7fc4e72dab80 1 -- 172.21.15.27:0/546103110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784fe9a3a0 msgr2=0x55784fec1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:31.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.574+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/546103110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784fe9a3a0 0x55784fec1280 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4005b30 tx=0x7fc4c4023040 comp rx=0 tx=0).stop 2024-01-11T09:47:31.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.574+0000 7fc4e72dab80 1 -- 172.21.15.27:0/546103110 shutdown_connections 2024-01-11T09:47:31.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.574+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/546103110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784febeef0 0x55784fec4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.574+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/546103110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784fe9a3a0 0x55784fec1280 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.574+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/546103110 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeb20 0x55784fe9a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.574+0000 7fc4e72dab80 1 -- 172.21.15.27:0/546103110 >> 172.21.15.27:0/546103110 conn(0x55784fe9c300 msgr2=0x55784fec2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:31.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.575+0000 7fc4e72dab80 1 -- 172.21.15.27:0/546103110 shutdown_connections 2024-01-11T09:47:31.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.575+0000 7fc4e72dab80 1 -- 172.21.15.27:0/546103110 wait complete. 2024-01-11T09:47:31.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.575+0000 7fc4e72dab80 1 -- start start 2024-01-11T09:47:31.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.575+0000 7fc4e72dab80 1 Processor -- start 2024-01-11T09:47:31.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.576+0000 7fc4e72dab80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784fe9a3a0 0x55784fe9e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.576+0000 7fc4e72dab80 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeb20 0x55784fefd050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.576+0000 7fc4e72dab80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784febeef0 0x55784ff49180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.576+0000 7fc4e72dab80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55784fec97a0 con 0x55784fe9a3a0 2024-01-11T09:47:31.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.576+0000 7fc4e72dab80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55784fee5200 con 0x55784febeef0 2024-01-11T09:47:31.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.576+0000 7fc4e72dab80 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x55784fefda60 con 0x55784febeb20 2024-01-11T09:47:31.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.576+0000 7fc4e72dab80 10 client.0 _ll_register_callbacks cb 0x55784fbfca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-11T09:47:31.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.577+0000 7fc4d259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784fe9a3a0 0x55784fe9e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.577+0000 7fc4d259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784fe9a3a0 0x55784fe9e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56300/0 (socket says 172.21.15.27:56300) 2024-01-11T09:47:31.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.577+0000 7fc4d259c700 1 -- 172.21.15.27:0/1251578764 learned_addr learned my addr 172.21.15.27:0/1251578764 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:31.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.577+0000 7fc4d1d9b700 1 --2- 172.21.15.27:0/1251578764 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeb20 0x55784fefd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.577+0000 7fc4d2d9d700 1 --2- 172.21.15.27:0/1251578764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784febeef0 0x55784ff49180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.577+0000 7fc4d259c700 1 -- 172.21.15.27:0/1251578764 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeb20 msgr2=0x55784fefd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:31.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.577+0000 7fc4d259c700 1 --2- 172.21.15.27:0/1251578764 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeb20 0x55784fefd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.577+0000 7fc4d259c700 1 -- 172.21.15.27:0/1251578764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784febeef0 msgr2=0x55784ff49180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:31.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.577+0000 7fc4d259c700 1 --2- 172.21.15.27:0/1251578764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784febeef0 0x55784ff49180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.577+0000 7fc4d259c700 1 -- 172.21.15.27:0/1251578764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c4028040 con 0x55784fe9a3a0 2024-01-11T09:47:31.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.577+0000 7fc4d2d9d700 1 --2- 172.21.15.27:0/1251578764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784febeef0 0x55784ff49180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:31.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.577+0000 7fc4d259c700 1 --2- 172.21.15.27:0/1251578764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784fe9a3a0 0x55784fe9e960 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009010 tx=0x7fc4c402f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e506e734bcd8d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:31.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.577+0000 7fc4b0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-11T09:47:31.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.578+0000 7fc4b0ff9700 1 -- 172.21.15.27:0/1251578764 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c403c070 con 0x55784fe9a3a0 2024-01-11T09:47:31.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.578+0000 7fc4b0ff9700 1 -- 172.21.15.27:0/1251578764 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(256..256 src has 1..256) v4 ==== 5817+0+0 (secure 0 0 0) 0x7fc4c4007580 con 0x55784fe9a3a0 2024-01-11T09:47:31.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.578+0000 7fc4e72dab80 1 -- 172.21.15.27:0/1251578764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc4c4007740 con 0x55784fe9a3a0 2024-01-11T09:47:31.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.578+0000 7fc4b0ff9700 1 -- 172.21.15.27:0/1251578764 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=141) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4c400dec0 con 0x55784fe9a3a0 2024-01-11T09:47:31.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.578+0000 7fc4e72dab80 10 client.12204 fetch_fsmap learned FSMap version 141 2024-01-11T09:47:31.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.578+0000 7fc4e72dab80 1 -- 172.21.15.27:0/1251578764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=141,monmap=2+}) v3 -- 0x55784fc541d0 con 0x55784fe9a3a0 2024-01-11T09:47:31.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.578+0000 7fc4b0ff9700 1 -- 172.21.15.27:0/1251578764 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c4007740 con 0x55784fe9a3a0 2024-01-11T09:47:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.578+0000 7fc4b0ff9700 1 -- 172.21.15.27:0/1251578764 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 141) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fc4c402b050 con 0x55784fe9a3a0 2024-01-11T09:47:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.578+0000 7fc4e72dab80 10 client.12204 fetch_fsmap finished waiting for FSMap version 141 2024-01-11T09:47:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.578+0000 7fc4e72dab80 10 client.12204 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-11T09:47:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.578+0000 7fc4e72dab80 20 client.12204 populate_metadata read hostname 'smithi027' 2024-01-11T09:47:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.578+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/1251578764 >> [v2:172.21.15.27:6834/1065634442,v1:172.21.15.27:6835/1065634442] conn(0x55784febf570 0x55784fef2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.578+0000 7fc4e72dab80 4 client.12204 mds_command: new command op to 12063 tid=0[{"prefix": "status"}] 2024-01-11T09:47:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.578+0000 7fc4e72dab80 1 -- 172.21.15.27:0/1251578764 --> [v2:172.21.15.27:6834/1065634442,v1:172.21.15.27:6835/1065634442] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55784fd908d0 con 0x55784febf570 2024-01-11T09:47:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.579+0000 7fc4d1d9b700 1 --2- 172.21.15.27:0/1251578764 >> [v2:172.21.15.27:6834/1065634442,v1:172.21.15.27:6835/1065634442] conn(0x55784febf570 0x55784fef2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.579+0000 7fc4d1d9b700 1 --2- 172.21.15.27:0/1251578764 >> [v2:172.21.15.27:6834/1065634442,v1:172.21.15.27:6835/1065634442] conn(0x55784febf570 0x55784fef2e80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb0a97a2dcac0159 server_cookie=c8ad76cf45f3a32b in_seq=0 out_seq=0 2024-01-11T09:47:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.579+0000 7fc4b0ff9700 10 client.12204 ms_handle_connect on v2:172.21.15.27:6834/1065634442 2024-01-11T09:47:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.579+0000 7fc4b0ff9700 1 -- 172.21.15.27:0/1251578764 <== mds.0 v2:172.21.15.27:6834/1065634442 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55784fd908d0 con 0x55784febf570 2024-01-11T09:47:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.579+0000 7fc4b0ff9700 10 client.12204 handle_command_reply: tid=0 2024-01-11T09:47:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.579+0000 7fc4e72dab80 1 client.12204 shutdown 2024-01-11T09:47:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.579+0000 7fc4e72dab80 1 -- 172.21.15.27:0/1251578764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784fe9a3a0 msgr2=0x55784fe9e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.579+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/1251578764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784fe9a3a0 0x55784fe9e960 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009010 tx=0x7fc4c402f000 comp rx=0 tx=0).stop 2024-01-11T09:47:31.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.580+0000 7fc4e72dab80 1 -- 172.21.15.27:0/1251578764 shutdown_connections 2024-01-11T09:47:31.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.580+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/1251578764 >> [v2:172.21.15.27:6834/1065634442,v1:172.21.15.27:6835/1065634442] conn(0x55784febf570 0x55784fef2e80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.580+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/1251578764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55784febeef0 0x55784ff49180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.580+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/1251578764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55784fe9a3a0 0x55784fe9e960 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.580+0000 7fc4e72dab80 1 --2- 172.21.15.27:0/1251578764 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x55784febeb20 0x55784fefd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.580+0000 7fc4b0ff9700 0 client.12204 ms_handle_reset on v2:172.21.15.27:6834/1065634442 2024-01-11T09:47:31.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.580+0000 7fc4e72dab80 1 -- 172.21.15.27:0/1251578764 >> 172.21.15.27:0/1251578764 conn(0x55784fe9c300 msgr2=0x55784fec1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:31.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.580+0000 7fc4e72dab80 1 -- 172.21.15.27:0/1251578764 shutdown_connections 2024-01-11T09:47:31.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.580+0000 7fc4e72dab80 1 -- 172.21.15.27:0/1251578764 wait complete. 2024-01-11T09:47:31.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.580+0000 7fc4e72dab80 20 client.12204 trim_cache size 0 max 16384 2024-01-11T09:47:31.575 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:47:31.575 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "8fddb4a7-9d66-40f1-892d-6fe86c728ee4", 2024-01-11T09:47:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-11T09:47:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 12063, 2024-01-11T09:47:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-11T09:47:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-11T09:47:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-11T09:47:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 6.6063888500000001, 2024-01-11T09:47:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 141, 2024-01-11T09:47:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 256, 2024-01-11T09:47:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 256, 2024-01-11T09:47:31.576 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 12.655547067000001 2024-01-11T09:47:31.576 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:47:31.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.580+0000 7fc4d8bff700 1 -- 172.21.15.27:0/1101449111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc4b4067170 msgr2=0x7fc4b4069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:31.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.580+0000 7fc4d8bff700 1 --2- 172.21.15.27:0/1101449111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc4b4067170 0x7fc4b4069620 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40058d0 tx=0x7fc4c4009040 comp rx=0 tx=0).stop 2024-01-11T09:47:31.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.580+0000 7fc4d8bff700 1 -- 172.21.15.27:0/1101449111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4d406bf10 msgr2=0x7fc4d40b1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:31.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.580+0000 7fc4d8bff700 1 --2- 172.21.15.27:0/1101449111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4d406bf10 0x7fc4d40b1170 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fc4c800c7e0 tx=0x7fc4c8007df0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.580+0000 7fc4d8bff700 1 -- 172.21.15.27:0/1101449111 shutdown_connections 2024-01-11T09:47:31.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.581+0000 7fc4d8bff700 1 --2- 172.21.15.27:0/1101449111 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fc4b4067170 0x7fc4b4069620 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.581+0000 7fc4d8bff700 1 --2- 172.21.15.27:0/1101449111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d40b0a40 0x7fc4d40c2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.581+0000 7fc4d8bff700 1 --2- 172.21.15.27:0/1101449111 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fc4d4068720 0x7fc4d40c4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.581+0000 7fc4d8bff700 1 --2- 172.21.15.27:0/1101449111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4d406bf10 0x7fc4d40b1170 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.581+0000 7fc4d8bff700 1 -- 172.21.15.27:0/1101449111 >> 172.21.15.27:0/1101449111 conn(0x7fc4d40ce3c0 msgr2=0x7fc4d40ce790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:31.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.581+0000 7fc4d8bff700 1 -- 172.21.15.27:0/1101449111 shutdown_connections 2024-01-11T09:47:31.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.581+0000 7fc4d8bff700 1 -- 172.21.15.27:0/1101449111 wait complete. 2024-01-11T09:47:31.589 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-11T09:47:31.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.798+0000 7f80c77ff700 1 Processor -- start 2024-01-11T09:47:31.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.798+0000 7f80c77ff700 1 -- start start 2024-01-11T09:47:31.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.799+0000 7f80c77ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f80c01596c0 0x7f80c015d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.799+0000 7f80c77ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f80c00ca670 0x7f80c0068620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.799+0000 7f80c77ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80c00a9cb0 con 0x7f80c0068b60 2024-01-11T09:47:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.799+0000 7f80c77ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80c0068fe0 con 0x7f80c01596c0 2024-01-11T09:47:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.799+0000 7f80c77ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c015e210 con 0x7f80c00ca670 2024-01-11T09:47:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.800+0000 7f80c559b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f80c01596c0 0x7f80c015d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.800+0000 7f80c559b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f80c01596c0 0x7f80c015d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56310/0 (socket says 172.21.15.27:56310) 2024-01-11T09:47:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.800+0000 7f80c559b700 1 -- 172.21.15.27:0/3058184435 learned_addr learned my addr 172.21.15.27:0/3058184435 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.800+0000 7f80c4d9a700 1 --2- 172.21.15.27:0/3058184435 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f80c00ca670 0x7f80c0068620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.800+0000 7f80c559b700 1 -- 172.21.15.27:0/3058184435 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f80c00ca670 msgr2=0x7f80c0068620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.800+0000 7f80c559b700 1 --2- 172.21.15.27:0/3058184435 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f80c00ca670 0x7f80c0068620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.800+0000 7f80c559b700 1 -- 172.21.15.27:0/3058184435 >> v1:172.21.15.77:6789/0 conn(0x7f80c0068b60 legacy=0x7f80c0159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.800+0000 7f80c559b700 1 -- 172.21.15.27:0/3058184435 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c015dc50 con 0x7f80c01596c0 2024-01-11T09:47:31.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.800+0000 7f80c4d9a700 1 --2- 172.21.15.27:0/3058184435 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f80c00ca670 0x7f80c0068620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:31.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.801+0000 7f80c559b700 1 --2- 172.21.15.27:0/3058184435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c01596c0 0x7f80c015d300 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0099b0 tx=0x7f80bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc5f2b4726c3c78 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:31.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.801+0000 7f80b7fff700 1 -- 172.21.15.27:0/3058184435 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80bc00b820 con 0x7f80c01596c0 2024-01-11T09:47:31.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.801+0000 7f80b7fff700 1 -- 172.21.15.27:0/3058184435 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80bc00b980 con 0x7f80c01596c0 2024-01-11T09:47:31.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.801+0000 7f80b7fff700 1 -- 172.21.15.27:0/3058184435 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80bc00bd20 con 0x7f80c01596c0 2024-01-11T09:47:31.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.801+0000 7f80c77ff700 1 -- 172.21.15.27:0/3058184435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c01596c0 msgr2=0x7f80c015d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:31.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.801+0000 7f80c77ff700 1 --2- 172.21.15.27:0/3058184435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c01596c0 0x7f80c015d300 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0099b0 tx=0x7f80bc009250 comp rx=0 tx=0).stop 2024-01-11T09:47:31.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.801+0000 7f80c77ff700 1 -- 172.21.15.27:0/3058184435 shutdown_connections 2024-01-11T09:47:31.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.801+0000 7f80c77ff700 1 --2- 172.21.15.27:0/3058184435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c01596c0 0x7f80c015d300 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.801+0000 7f80c77ff700 1 --2- 172.21.15.27:0/3058184435 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f80c00ca670 0x7f80c0068620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.801+0000 7f80c77ff700 1 -- 172.21.15.27:0/3058184435 >> 172.21.15.27:0/3058184435 conn(0x7f80c00c9f70 msgr2=0x7f80c006deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:31.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.801+0000 7f80c77ff700 1 -- 172.21.15.27:0/3058184435 shutdown_connections 2024-01-11T09:47:31.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.802+0000 7f80c77ff700 1 -- 172.21.15.27:0/3058184435 wait complete. 2024-01-11T09:47:31.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.803+0000 7f80c77ff700 1 Processor -- start 2024-01-11T09:47:31.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.803+0000 7f80c77ff700 1 -- start start 2024-01-11T09:47:31.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.804+0000 7f80c77ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c0068b60 0x7f80c014b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.804+0000 7f80c77ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c00ca670 0x7f80c0159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.804+0000 7f80c77ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f80c01596c0 0x7f80c0159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.804+0000 7f80c77ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c00af930 con 0x7f80c00ca670 2024-01-11T09:47:31.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.804+0000 7f80c77ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c00b04b0 con 0x7f80c0068b60 2024-01-11T09:47:31.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.804+0000 7f80c77ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c00afb80 con 0x7f80c01596c0 2024-01-11T09:47:31.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.804+0000 7f80c4d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c00ca670 0x7f80c0159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.804+0000 7f80c559b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c0068b60 0x7f80c014b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.804+0000 7f80c4d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c00ca670 0x7f80c0159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56312/0 (socket says 172.21.15.27:56312) 2024-01-11T09:47:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.804+0000 7f80c4d9a700 1 -- 172.21.15.27:0/2160219994 learned_addr learned my addr 172.21.15.27:0/2160219994 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.804+0000 7f80c5d9c700 1 --2- 172.21.15.27:0/2160219994 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f80c01596c0 0x7f80c0159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.805+0000 7f80c4d9a700 1 -- 172.21.15.27:0/2160219994 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f80c01596c0 msgr2=0x7f80c0159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.805+0000 7f80c4d9a700 1 --2- 172.21.15.27:0/2160219994 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f80c01596c0 0x7f80c0159fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.805+0000 7f80c4d9a700 1 -- 172.21.15.27:0/2160219994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c0068b60 msgr2=0x7f80c014b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.805+0000 7f80c4d9a700 1 --2- 172.21.15.27:0/2160219994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c0068b60 0x7f80c014b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.805+0000 7f80c4d9a700 1 -- 172.21.15.27:0/2160219994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80bc0094f0 con 0x7f80c00ca670 2024-01-11T09:47:31.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.805+0000 7f80c4d9a700 1 --2- 172.21.15.27:0/2160219994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c00ca670 0x7f80c0159a90 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f80b000c880 tx=0x7f80b0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:31.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.805+0000 7f80b57fa700 1 -- 172.21.15.27:0/2160219994 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b0011600 con 0x7f80c00ca670 2024-01-11T09:47:31.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.805+0000 7f80b57fa700 1 -- 172.21.15.27:0/2160219994 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80b0011760 con 0x7f80c00ca670 2024-01-11T09:47:31.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.805+0000 7f80b57fa700 1 -- 172.21.15.27:0/2160219994 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b0011ae0 con 0x7f80c00ca670 2024-01-11T09:47:31.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.805+0000 7f80c77ff700 1 -- 172.21.15.27:0/2160219994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c00afe60 con 0x7f80c00ca670 2024-01-11T09:47:31.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.806+0000 7f80c77ff700 1 -- 172.21.15.27:0/2160219994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c00b0940 con 0x7f80c00ca670 2024-01-11T09:47:31.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.807+0000 7f80c77ff700 1 -- 172.21.15.27:0/2160219994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c00689a0 con 0x7f80c00ca670 2024-01-11T09:47:31.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.808+0000 7f80b57fa700 1 -- 172.21.15.27:0/2160219994 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f80b00193b0 con 0x7f80c00ca670 2024-01-11T09:47:31.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.808+0000 7f80b57fa700 1 --2- 172.21.15.27:0/2160219994 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f80a4067170 0x7f80a4069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:31.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.809+0000 7f80b57fa700 1 -- 172.21.15.27:0/2160219994 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f80b0084930 con 0x7f80c00ca670 2024-01-11T09:47:31.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.809+0000 7f80c559b700 1 --2- 172.21.15.27:0/2160219994 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f80a4067170 0x7f80a4069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:31.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.810+0000 7f80c559b700 1 --2- 172.21.15.27:0/2160219994 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f80a4067170 0x7f80a4069620 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0099b0 tx=0x7f80bc016000 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:31.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:31.813+0000 7f80b57fa700 1 -- 172.21.15.27:0/2160219994 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f80b0051790 con 0x7f80c00ca670 2024-01-11T09:47:32.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:32.069+0000 7f80c77ff700 1 -- 172.21.15.27:0/2160219994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f80c00b05f0 con 0x7f80c00ca670 2024-01-11T09:47:32.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:32.069+0000 7f80b57fa700 1 -- 172.21.15.27:0/2160219994 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v141) v1 ==== 53+0+73 (secure 0 0 0) 0x7f80b0015070 con 0x7f80c00ca670 2024-01-11T09:47:32.061 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-11T09:47:32.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:32.072+0000 7f80c77ff700 1 -- 172.21.15.27:0/2160219994 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f80a4067170 msgr2=0x7f80a4069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:32.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:32.072+0000 7f80c77ff700 1 --2- 172.21.15.27:0/2160219994 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f80a4067170 0x7f80a4069620 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0099b0 tx=0x7f80bc016000 comp rx=0 tx=0).stop 2024-01-11T09:47:32.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:32.072+0000 7f80c77ff700 1 -- 172.21.15.27:0/2160219994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c00ca670 msgr2=0x7f80c0159a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:32.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:32.072+0000 7f80c77ff700 1 --2- 172.21.15.27:0/2160219994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c00ca670 0x7f80c0159a90 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f80b000c880 tx=0x7f80b0008040 comp rx=0 tx=0).stop 2024-01-11T09:47:32.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:32.072+0000 7f80c77ff700 1 -- 172.21.15.27:0/2160219994 shutdown_connections 2024-01-11T09:47:32.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:32.072+0000 7f80c77ff700 1 --2- 172.21.15.27:0/2160219994 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f80c01596c0 0x7f80c0159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:32.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:32.072+0000 7f80c77ff700 1 --2- 172.21.15.27:0/2160219994 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f80a4067170 0x7f80a4069620 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:32.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:32.072+0000 7f80c77ff700 1 --2- 172.21.15.27:0/2160219994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c0068b60 0x7f80c014b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:32.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:32.072+0000 7f80c77ff700 1 --2- 172.21.15.27:0/2160219994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80c00ca670 0x7f80c0159a90 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:32.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:32.072+0000 7f80c77ff700 1 -- 172.21.15.27:0/2160219994 >> 172.21.15.27:0/2160219994 conn(0x7f80c00c9f70 msgr2=0x7f80c0157410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:32.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:32.072+0000 7f80c77ff700 1 -- 172.21.15.27:0/2160219994 shutdown_connections 2024-01-11T09:47:32.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:32.072+0000 7f80c77ff700 1 -- 172.21.15.27:0/2160219994 wait complete. 2024-01-11T09:47:32.081 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-11T09:47:32.081 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-11T09:47:32.081 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-11T09:47:32.081 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-11T09:47:32.082 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-11T09:47:32.082 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-11T09:47:32.082 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-11T09:47:32.082 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-11T09:47:32.082 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T09:47:32.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:32.082 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-11T09:47:32.131 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-11T09:47:32.131 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-11T09:47:32.131 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-11T09:47:32.131 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:47:32.131 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-11T09:47:32.131 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:47:32.131 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T09:47:32.131 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-11T09:47:32.131 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-11T09:47:32.131 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T09:47:32.131 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-11T09:47:32.132 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-11T09:47:32.132 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-11T09:47:32.132 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-11T09:47:32.132 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-11T09:47:32.132 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-11T09:47:32.132 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 36670sec preferred_lft 36670sec 2024-01-11T09:47:32.132 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-11T09:47:32.132 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:47:32.132 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-11T09:47:32.132 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-11T09:47:32.132 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-11T09:47:32.132 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-11T09:47:32.132 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff 2024-01-11T09:47:32.133 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-11T09:47:32.133 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:47:32.133 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::acd2:b3ff:fe6a:db4b/64 scope link 2024-01-11T09:47:32.133 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:47:32.133 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T09:47:32.133 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:fa:85:1b:3a:bc brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-11T09:47:32.133 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ecfa:85ff:fe1b:3abc/64 scope link 2024-01-11T09:47:32.133 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:47:32.133 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-11T09:47:32.133 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ee:88:fa:17:41:aa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-11T09:47:32.133 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ec88:faff:fe17:41aa/64 scope link 2024-01-11T09:47:32.133 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-11T09:47:32.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:32.134 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-11T09:47:32.204 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-11T09:47:32.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:32.207 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-11T09:47:32.257 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-11T09:47:32.257 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-11T09:47:32.257 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-11T09:47:32.257 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-11T09:47:32.257 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-11T09:47:32.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:32.259 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-11T09:47:32.259 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-11T09:47:32.259 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-11T09:47:32.259 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-11T09:47:32.259 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-11T09:47:32.259 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-11T09:47:32.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:32.436 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T09:47:32.485 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-11T09:47:32.485 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-11T09:47:32.485 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-11T09:47:32.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:32.485 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-11T09:47:32.558 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T09:47:32.559 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-11T09:47:32.606 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-11T09:47:32.606 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-11T09:47:32.606 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-11T09:47:32.653 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-11T09:47:32.653 INFO:teuthology.orchestra.run:Running command with timeout 1800 2024-01-11T09:47:32.653 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:47:32.757 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:47:32.757 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T09:47:32.757 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T09:47:32.771 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:47:32.771 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T09:47:32.771 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T09:47:32.772 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T09:47:32.772 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T09:47:32.793 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-11T09:47:32.793 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-11T09:47:32.793 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2024-01-11T09:47:32.793 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.77:3300,172.21.15.77:3301" 2024-01-11T09:47:32.793 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-11T09:47:32.793 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790" 2024-01-11T09:47:32.793 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.77:6789,172.21.15.77:6790:/ 2024-01-11T09:47:32.794 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=8fddb4a7-9d66-40f1-892d-6fe86c728ee4" will pass to kernel 2024-01-11T09:47:32.797 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-11T09:47:32.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:32.797 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-11T09:47:32.870 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-11T09:47:33.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.083+0000 7f851c1ff700 1 Processor -- start 2024-01-11T09:47:33.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.083+0000 7f851c1ff700 1 -- start start 2024-01-11T09:47:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.084+0000 7f851c1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8514068cd0 0x7f851415a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.084+0000 7f851c1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f851415cde0 0x7f85140b5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.084+0000 7f851c1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85140b4bd0 con 0x7f851415ca10 2024-01-11T09:47:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.084+0000 7f851c1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f851415ac70 con 0x7f8514068cd0 2024-01-11T09:47:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.084+0000 7f851c1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85140b3fc0 con 0x7f851415cde0 2024-01-11T09:47:33.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.085+0000 7f851a79c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f851415cde0 0x7f85140b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.085+0000 7f851979a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8514068cd0 0x7f851415a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.085+0000 7f851a79c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f851415cde0 0x7f85140b5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:59702/0 (socket says 172.21.15.27:59702) 2024-01-11T09:47:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.085+0000 7f851a79c700 1 -- 172.21.15.27:0/2483969443 learned_addr learned my addr 172.21.15.27:0/2483969443 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.085+0000 7f851979a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8514068cd0 0x7f851415a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:52152/0 (socket says 172.21.15.27:52152) 2024-01-11T09:47:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.085+0000 7f8519f9b700 1 --1- 172.21.15.27:0/2483969443 >> v1:172.21.15.77:6789/0 conn(0x7f851415ca10 0x7f85140685c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:51418/0 (socket says 172.21.15.27:51418) 2024-01-11T09:47:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.085+0000 7f851a79c700 1 -- 172.21.15.27:0/2483969443 >> v2:172.21.15.77:3300/0 conn(0x7f8514068cd0 msgr2=0x7f851415a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.085+0000 7f851a79c700 1 --2- 172.21.15.27:0/2483969443 >> v2:172.21.15.77:3300/0 conn(0x7f8514068cd0 0x7f851415a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.085+0000 7f851a79c700 1 -- 172.21.15.27:0/2483969443 >> v1:172.21.15.77:6789/0 conn(0x7f851415ca10 legacy=0x7f85140685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.085+0000 7f851a79c700 1 -- 172.21.15.27:0/2483969443 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85140b42a0 con 0x7f851415cde0 2024-01-11T09:47:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.085+0000 7f8518f99700 1 -- 172.21.15.27:0/2483969443 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159889924 0 0) 0x7f85140b4bd0 con 0x7f851415ca10 2024-01-11T09:47:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.086+0000 7f851a79c700 1 --2- 172.21.15.27:0/2483969443 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f851415cde0 0x7f85140b5060 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f850c009f50 tx=0x7f850c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccf416eda64d8bdf server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.086+0000 7f8518f99700 1 -- 172.21.15.27:0/2483969443 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c00e070 con 0x7f851415cde0 2024-01-11T09:47:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.086+0000 7f8518f99700 1 -- 172.21.15.27:0/2483969443 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850c00b9c0 con 0x7f851415cde0 2024-01-11T09:47:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.086+0000 7f8518f99700 1 -- 172.21.15.27:0/2483969443 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c0092e0 con 0x7f851415cde0 2024-01-11T09:47:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.087+0000 7f851c1ff700 1 -- 172.21.15.27:0/2483969443 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f851415cde0 msgr2=0x7f85140b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.087+0000 7f851c1ff700 1 --2- 172.21.15.27:0/2483969443 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f851415cde0 0x7f85140b5060 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f850c009f50 tx=0x7f850c009840 comp rx=0 tx=0).stop 2024-01-11T09:47:33.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.087+0000 7f851c1ff700 1 -- 172.21.15.27:0/2483969443 shutdown_connections 2024-01-11T09:47:33.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.087+0000 7f851c1ff700 1 --2- 172.21.15.27:0/2483969443 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f851415cde0 0x7f85140b5060 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:33.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.087+0000 7f851c1ff700 1 --2- 172.21.15.27:0/2483969443 >> v2:172.21.15.77:3300/0 conn(0x7f8514068cd0 0x7f851415a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:33.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.087+0000 7f851c1ff700 1 -- 172.21.15.27:0/2483969443 >> 172.21.15.27:0/2483969443 conn(0x7f851406ed40 msgr2=0x7f851406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:33.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.087+0000 7f851c1ff700 1 -- 172.21.15.27:0/2483969443 shutdown_connections 2024-01-11T09:47:33.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.087+0000 7f851c1ff700 1 -- 172.21.15.27:0/2483969443 wait complete. 2024-01-11T09:47:33.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.088+0000 7f851c1ff700 1 Processor -- start 2024-01-11T09:47:33.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.088+0000 7f851c1ff700 1 -- start start 2024-01-11T09:47:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.089+0000 7f851c1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8514068cd0 0x7f8514152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.089+0000 7f851c1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f851415ca10 0x7f851415f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.089+0000 7f851c1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f851415cde0 0x7f85140cf760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.089+0000 7f851c1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851416c9e0 con 0x7f851415ca10 2024-01-11T09:47:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.089+0000 7f851c1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851416cc30 con 0x7f8514068cd0 2024-01-11T09:47:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.089+0000 7f851c1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f851416cf10 con 0x7f851415cde0 2024-01-11T09:47:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.090+0000 7f851979a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f851415ca10 0x7f851415f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.090+0000 7f851979a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f851415ca10 0x7f851415f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56324/0 (socket says 172.21.15.27:56324) 2024-01-11T09:47:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.090+0000 7f851979a700 1 -- 172.21.15.27:0/3784066032 learned_addr learned my addr 172.21.15.27:0/3784066032 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.090+0000 7f8519f9b700 1 --2- 172.21.15.27:0/3784066032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8514068cd0 0x7f8514152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:33.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.090+0000 7f851a79c700 1 --2- 172.21.15.27:0/3784066032 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f851415cde0 0x7f85140cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.090+0000 7f851979a700 1 -- 172.21.15.27:0/3784066032 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f851415cde0 msgr2=0x7f85140cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.090+0000 7f851979a700 1 --2- 172.21.15.27:0/3784066032 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f851415cde0 0x7f85140cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.090+0000 7f851979a700 1 -- 172.21.15.27:0/3784066032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8514068cd0 msgr2=0x7f8514152db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.090+0000 7f851979a700 1 --2- 172.21.15.27:0/3784066032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8514068cd0 0x7f8514152db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.090+0000 7f851979a700 1 -- 172.21.15.27:0/3784066032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c009ae0 con 0x7f851415ca10 2024-01-11T09:47:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.091+0000 7f851979a700 1 --2- 172.21.15.27:0/3784066032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f851415ca10 0x7f851415f310 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f850400ea50 tx=0x7f85040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.091+0000 7f8509ffb700 1 -- 172.21.15.27:0/3784066032 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850400ee90 con 0x7f851415ca10 2024-01-11T09:47:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.091+0000 7f8509ffb700 1 -- 172.21.15.27:0/3784066032 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850400c8e0 con 0x7f851415ca10 2024-01-11T09:47:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.091+0000 7f851c1ff700 1 -- 172.21.15.27:0/3784066032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f851416d1f0 con 0x7f851415ca10 2024-01-11T09:47:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.091+0000 7f8509ffb700 1 -- 172.21.15.27:0/3784066032 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850400cc80 con 0x7f851415ca10 2024-01-11T09:47:33.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.091+0000 7f851c1ff700 1 -- 172.21.15.27:0/3784066032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f851416d840 con 0x7f851415ca10 2024-01-11T09:47:33.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.092+0000 7f851c1ff700 1 -- 172.21.15.27:0/3784066032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f851404c3c0 con 0x7f851415ca10 2024-01-11T09:47:33.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.093+0000 7f8509ffb700 1 -- 172.21.15.27:0/3784066032 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f8504013680 con 0x7f851415ca10 2024-01-11T09:47:33.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.094+0000 7f8509ffb700 1 --2- 172.21.15.27:0/3784066032 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f84f8067170 0x7f84f8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:33.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.094+0000 7f8509ffb700 1 -- 172.21.15.27:0/3784066032 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f8504084c90 con 0x7f851415ca10 2024-01-11T09:47:33.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.094+0000 7f8519f9b700 1 --2- 172.21.15.27:0/3784066032 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f84f8067170 0x7f84f8069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:33.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.095+0000 7f8519f9b700 1 --2- 172.21.15.27:0/3784066032 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f84f8067170 0x7f84f8069620 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8510007fd0 tx=0x7f8510007d50 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:33.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.098+0000 7f8509ffb700 1 -- 172.21.15.27:0/3784066032 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8504051b60 con 0x7f851415ca10 2024-01-11T09:47:33.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.329+0000 7f851c1ff700 1 -- 172.21.15.27:0/3784066032 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f85140b4580 con 0x7f84f8067170 2024-01-11T09:47:33.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.331+0000 7f8509ffb700 1 -- 172.21.15.27:0/3784066032 <== mgr.9641 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f85140b4580 con 0x7f84f8067170 2024-01-11T09:47:33.322 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-11T09:47:33.323 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-11T09:47:33.323 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-11T09:47:33.323 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-11T09:47:33.323 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-11T09:47:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.333+0000 7f851c1ff700 1 -- 172.21.15.27:0/3784066032 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f84f8067170 msgr2=0x7f84f8069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.333+0000 7f851c1ff700 1 --2- 172.21.15.27:0/3784066032 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f84f8067170 0x7f84f8069620 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8510007fd0 tx=0x7f8510007d50 comp rx=0 tx=0).stop 2024-01-11T09:47:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.333+0000 7f851c1ff700 1 -- 172.21.15.27:0/3784066032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f851415ca10 msgr2=0x7f851415f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.333+0000 7f851c1ff700 1 --2- 172.21.15.27:0/3784066032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f851415ca10 0x7f851415f310 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f850400ea50 tx=0x7f85040091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.333+0000 7f851c1ff700 1 -- 172.21.15.27:0/3784066032 shutdown_connections 2024-01-11T09:47:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.334+0000 7f851c1ff700 1 --2- 172.21.15.27:0/3784066032 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f851415cde0 0x7f85140cf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:33.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.334+0000 7f851c1ff700 1 --2- 172.21.15.27:0/3784066032 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f84f8067170 0x7f84f8069620 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:33.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.334+0000 7f851c1ff700 1 --2- 172.21.15.27:0/3784066032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8514068cd0 0x7f8514152db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:33.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.334+0000 7f851c1ff700 1 --2- 172.21.15.27:0/3784066032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f851415ca10 0x7f851415f310 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:33.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.334+0000 7f851c1ff700 1 -- 172.21.15.27:0/3784066032 >> 172.21.15.27:0/3784066032 conn(0x7f851406ed40 msgr2=0x7f85140d06d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:33.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.334+0000 7f851c1ff700 1 -- 172.21.15.27:0/3784066032 shutdown_connections 2024-01-11T09:47:33.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.334+0000 7f851c1ff700 1 -- 172.21.15.27:0/3784066032 wait complete. 2024-01-11T09:47:33.343 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-01-11T09:47:33.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.554+0000 7f8614dff700 1 Processor -- start 2024-01-11T09:47:33.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.554+0000 7f8614dff700 1 -- start start 2024-01-11T09:47:33.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.555+0000 7f8614dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f86100ccd70 0x7f86100cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:33.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.556+0000 7f8614dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f861015ae70 0x7f861015b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:33.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.556+0000 7f8614dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86100cf850 con 0x7f861015a1b0 2024-01-11T09:47:33.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.556+0000 7f8614dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f861015ac90 con 0x7f86100ccd70 2024-01-11T09:47:33.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.556+0000 7f8614dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86100cec40 con 0x7f861015ae70 2024-01-11T09:47:33.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.556+0000 7f860e59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f861015a1b0 0x7f861015a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51326/0 (socket says 172.21.15.27:51326) 2024-01-11T09:47:33.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.556+0000 7f860e59c700 1 -- 172.21.15.27:0/2354691271 learned_addr learned my addr 172.21.15.27:0/2354691271 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:33.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.556+0000 7f860dd9b700 1 --2- 172.21.15.27:0/2354691271 >> v2:172.21.15.77:3300/0 conn(0x7f86100ccd70 0x7f86100cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:33.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.556+0000 7f860ed9d700 1 --2- 172.21.15.27:0/2354691271 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f861015ae70 0x7f861015b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:33.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.556+0000 7f860d59a700 1 -- 172.21.15.27:0/2354691271 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157685659 0 0) 0x7f86100cf850 con 0x7f861015a1b0 2024-01-11T09:47:33.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.556+0000 7f860d59a700 1 -- 172.21.15.27:0/2354691271 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f8003570 con 0x7f861015a1b0 2024-01-11T09:47:33.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.557+0000 7f860dd9b700 1 -- 172.21.15.27:0/2354691271 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f861015ae70 msgr2=0x7f861015b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:33.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.557+0000 7f860dd9b700 1 --2- 172.21.15.27:0/2354691271 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f861015ae70 0x7f861015b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:33.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.557+0000 7f860dd9b700 1 -- 172.21.15.27:0/2354691271 >> v1:172.21.15.27:6789/0 conn(0x7f861015a1b0 legacy=0x7f861015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:33.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.557+0000 7f860dd9b700 1 -- 172.21.15.27:0/2354691271 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86100cef20 con 0x7f86100ccd70 2024-01-11T09:47:33.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.557+0000 7f860ed9d700 1 --2- 172.21.15.27:0/2354691271 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f861015ae70 0x7f861015b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:33.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.557+0000 7f860dd9b700 1 --2- 172.21.15.27:0/2354691271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86100ccd70 0x7f86100cd210 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f86040098e0 tx=0x7f86040091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e65fefdf2f7e31e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:33.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.557+0000 7f860d59a700 1 -- 172.21.15.27:0/2354691271 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8604011630 con 0x7f86100ccd70 2024-01-11T09:47:33.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.557+0000 7f860d59a700 1 -- 172.21.15.27:0/2354691271 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8604011790 con 0x7f86100ccd70 2024-01-11T09:47:33.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.557+0000 7f860d59a700 1 -- 172.21.15.27:0/2354691271 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8604011b10 con 0x7f86100ccd70 2024-01-11T09:47:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.558+0000 7f8614dff700 1 -- 172.21.15.27:0/2354691271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86100ccd70 msgr2=0x7f86100cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.558+0000 7f8614dff700 1 --2- 172.21.15.27:0/2354691271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86100ccd70 0x7f86100cd210 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f86040098e0 tx=0x7f86040091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.558+0000 7f8614dff700 1 -- 172.21.15.27:0/2354691271 shutdown_connections 2024-01-11T09:47:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.558+0000 7f8614dff700 1 --2- 172.21.15.27:0/2354691271 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f861015ae70 0x7f861015b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.558+0000 7f8614dff700 1 --2- 172.21.15.27:0/2354691271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86100ccd70 0x7f86100cd210 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.558+0000 7f8614dff700 1 -- 172.21.15.27:0/2354691271 >> 172.21.15.27:0/2354691271 conn(0x7f861006ed40 msgr2=0x7f861006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.558+0000 7f8614dff700 1 -- 172.21.15.27:0/2354691271 shutdown_connections 2024-01-11T09:47:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.558+0000 7f8614dff700 1 -- 172.21.15.27:0/2354691271 wait complete. 2024-01-11T09:47:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.559+0000 7f8614dff700 1 Processor -- start 2024-01-11T09:47:33.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.560+0000 7f8614dff700 1 -- start start 2024-01-11T09:47:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.560+0000 7f8614dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f86100ccd70 0x7f86100c81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.560+0000 7f8614dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f861015a1b0 0x7f86101585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.560+0000 7f8614dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f861015ae70 0x7f861015f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.560+0000 7f8614dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86100c7d10 con 0x7f861015ae70 2024-01-11T09:47:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.560+0000 7f8614dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86100c7730 con 0x7f861015a1b0 2024-01-11T09:47:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.561+0000 7f8614dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861015fe80 con 0x7f86100ccd70 2024-01-11T09:47:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.561+0000 7f860ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f861015ae70 0x7f861015f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.561+0000 7f860ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f861015ae70 0x7f861015f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56330/0 (socket says 172.21.15.27:56330) 2024-01-11T09:47:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.561+0000 7f860ed9d700 1 -- 172.21.15.27:0/2110067314 learned_addr learned my addr 172.21.15.27:0/2110067314 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:33.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.561+0000 7f860dd9b700 1 --2- 172.21.15.27:0/2110067314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f861015a1b0 0x7f86101585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.561+0000 7f860e59c700 1 --2- 172.21.15.27:0/2110067314 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f86100ccd70 0x7f86100c81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.561+0000 7f860ed9d700 1 -- 172.21.15.27:0/2110067314 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f86100ccd70 msgr2=0x7f86100c81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.561+0000 7f860ed9d700 1 --2- 172.21.15.27:0/2110067314 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f86100ccd70 0x7f86100c81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.561+0000 7f860ed9d700 1 -- 172.21.15.27:0/2110067314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f861015a1b0 msgr2=0x7f86101585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.561+0000 7f860ed9d700 1 --2- 172.21.15.27:0/2110067314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f861015a1b0 0x7f86101585b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.561+0000 7f860ed9d700 1 -- 172.21.15.27:0/2110067314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8608009a20 con 0x7f861015ae70 2024-01-11T09:47:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.561+0000 7f860dd9b700 1 --2- 172.21.15.27:0/2110067314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f861015a1b0 0x7f86101585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.561+0000 7f860ed9d700 1 --2- 172.21.15.27:0/2110067314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f861015ae70 0x7f861015f3c0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f8608009fc0 tx=0x7f860800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.562+0000 7f85fe7fc700 1 -- 172.21.15.27:0/2110067314 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860800cd30 con 0x7f861015ae70 2024-01-11T09:47:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.562+0000 7f85fe7fc700 1 -- 172.21.15.27:0/2110067314 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860800f040 con 0x7f861015ae70 2024-01-11T09:47:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.562+0000 7f85fe7fc700 1 -- 172.21.15.27:0/2110067314 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86080146d0 con 0x7f861015ae70 2024-01-11T09:47:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.562+0000 7f8614dff700 1 -- 172.21.15.27:0/2110067314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8604009420 con 0x7f861015ae70 2024-01-11T09:47:33.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.562+0000 7f8614dff700 1 -- 172.21.15.27:0/2110067314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8610158af0 con 0x7f861015ae70 2024-01-11T09:47:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.563+0000 7f8614dff700 1 -- 172.21.15.27:0/2110067314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86100c7e80 con 0x7f861015ae70 2024-01-11T09:47:33.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.564+0000 7f85fe7fc700 1 -- 172.21.15.27:0/2110067314 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 145) v1 ==== 84946+0+0 (secure 0 0 0) 0x7f8608003020 con 0x7f861015ae70 2024-01-11T09:47:33.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.566+0000 7f85fe7fc700 1 --2- 172.21.15.27:0/2110067314 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f85f0067170 0x7f85f0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:33.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.566+0000 7f85fe7fc700 1 -- 172.21.15.27:0/2110067314 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f8608085570 con 0x7f861015ae70 2024-01-11T09:47:33.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.566+0000 7f860e59c700 1 --2- 172.21.15.27:0/2110067314 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f85f0067170 0x7f85f0069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:33.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.567+0000 7f860e59c700 1 --2- 172.21.15.27:0/2110067314 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f85f0067170 0x7f85f0069620 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f86000058d0 tx=0x7f8600009040 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:33.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.570+0000 7f85fe7fc700 1 -- 172.21.15.27:0/2110067314 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8608052300 con 0x7f861015ae70 2024-01-11T09:47:33.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:33.829+0000 7f8614dff700 1 -- 172.21.15.27:0/2110067314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f861004c3c0 con 0x7f861015ae70 2024-01-11T09:47:34.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.016+0000 7f85fe7fc700 1 -- 172.21.15.27:0/2110067314 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 146) v1 ==== 84963+0+0 (secure 0 0 0) 0x7f86080526d0 con 0x7f861015ae70 2024-01-11T09:47:34.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.018+0000 7f85fe7fc700 1 -- 172.21.15.27:0/2110067314 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v146) v1 ==== 92+0+0 (secure 0 0 0) 0x7f8608055d90 con 0x7f861015ae70 2024-01-11T09:47:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.021+0000 7f8614dff700 1 -- 172.21.15.27:0/2110067314 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f85f0067170 msgr2=0x7f85f0069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.021+0000 7f8614dff700 1 --2- 172.21.15.27:0/2110067314 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f85f0067170 0x7f85f0069620 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f86000058d0 tx=0x7f8600009040 comp rx=0 tx=0).stop 2024-01-11T09:47:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.021+0000 7f8614dff700 1 -- 172.21.15.27:0/2110067314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f861015ae70 msgr2=0x7f861015f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.021+0000 7f8614dff700 1 --2- 172.21.15.27:0/2110067314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f861015ae70 0x7f861015f3c0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f8608009fc0 tx=0x7f860800b040 comp rx=0 tx=0).stop 2024-01-11T09:47:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.021+0000 7f8614dff700 1 -- 172.21.15.27:0/2110067314 shutdown_connections 2024-01-11T09:47:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.021+0000 7f8614dff700 1 --2- 172.21.15.27:0/2110067314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f861015ae70 0x7f861015f3c0 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.021+0000 7f8614dff700 1 --2- 172.21.15.27:0/2110067314 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f86100ccd70 0x7f86100c81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.021+0000 7f8614dff700 1 --2- 172.21.15.27:0/2110067314 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f85f0067170 0x7f85f0069620 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.021+0000 7f8614dff700 1 --2- 172.21.15.27:0/2110067314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f861015a1b0 0x7f86101585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:34.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.021+0000 7f8614dff700 1 -- 172.21.15.27:0/2110067314 >> 172.21.15.27:0/2110067314 conn(0x7f861006ed40 msgr2=0x7f86100c6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:34.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.021+0000 7f8614dff700 1 -- 172.21.15.27:0/2110067314 shutdown_connections 2024-01-11T09:47:34.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.021+0000 7f8614dff700 1 -- 172.21.15.27:0/2110067314 wait complete. 2024-01-11T09:47:34.032 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-01-11T09:47:34.226 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:34.234+0000 7f07bbebe1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:47:34.226 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:34.235+0000 7faeab6861c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:47:34.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.249+0000 7fb0e5fff700 1 Processor -- start 2024-01-11T09:47:34.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.249+0000 7fb0e5fff700 1 -- start start 2024-01-11T09:47:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.250+0000 7fb0e5fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0e0069b60 0x7fb0e01598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.250+0000 7fb0e5fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e00cd0a0 con 0x7fb0e0069790 2024-01-11T09:47:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.250+0000 7fb0e5fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e00c7680 con 0x7fb0e00c8720 2024-01-11T09:47:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.250+0000 7fb0e5fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e00c7960 con 0x7fb0e0069b60 2024-01-11T09:47:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.251+0000 7fb0df7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb0e0069790 0x7fb0e00c69a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34506/0 (socket says 172.21.15.27:34506) 2024-01-11T09:47:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.251+0000 7fb0df7fe700 1 -- 172.21.15.27:0/1129621604 learned_addr learned my addr 172.21.15.27:0/1129621604 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.251+0000 7fb0dffff700 1 --2- 172.21.15.27:0/1129621604 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0e0069b60 0x7fb0e01598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.251+0000 7fb0dffff700 1 -- 172.21.15.27:0/1129621604 >> v1:172.21.15.77:6789/0 conn(0x7fb0e00c8720 legacy=0x7fb0e01591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.251+0000 7fb0dffff700 1 -- 172.21.15.27:0/1129621604 >> v1:172.21.15.27:6789/0 conn(0x7fb0e0069790 legacy=0x7fb0e00c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.251+0000 7fb0de7fc700 1 -- 172.21.15.27:0/1129621604 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799321624 0 0) 0x7fb0e00c7680 con 0x7fb0e00c8720 2024-01-11T09:47:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.251+0000 7fb0dffff700 1 -- 172.21.15.27:0/1129621604 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e00c7c40 con 0x7fb0e0069b60 2024-01-11T09:47:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.251+0000 7fb0dffff700 1 --2- 172.21.15.27:0/1129621604 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0e0069b60 0x7fb0e01598b0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009e90 tx=0x7fb0d8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be2e7ad7c24edab5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.252+0000 7fb0de7fc700 1 -- 172.21.15.27:0/1129621604 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d800e070 con 0x7fb0e0069b60 2024-01-11T09:47:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.252+0000 7fb0de7fc700 1 -- 172.21.15.27:0/1129621604 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d800bb30 con 0x7fb0e0069b60 2024-01-11T09:47:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.252+0000 7fb0de7fc700 1 -- 172.21.15.27:0/1129621604 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8012430 con 0x7fb0e0069b60 2024-01-11T09:47:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.252+0000 7fb0e5fff700 1 -- 172.21.15.27:0/1129621604 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0e0069b60 msgr2=0x7fb0e01598b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:34.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.252+0000 7fb0e5fff700 1 --2- 172.21.15.27:0/1129621604 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0e0069b60 0x7fb0e01598b0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009e90 tx=0x7fb0d8009780 comp rx=0 tx=0).stop 2024-01-11T09:47:34.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.252+0000 7fb0e5fff700 1 -- 172.21.15.27:0/1129621604 shutdown_connections 2024-01-11T09:47:34.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.252+0000 7fb0e5fff700 1 --2- 172.21.15.27:0/1129621604 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0e0069b60 0x7fb0e01598b0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:34.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.252+0000 7fb0e5fff700 1 -- 172.21.15.27:0/1129621604 >> 172.21.15.27:0/1129621604 conn(0x7fb0e00680a0 msgr2=0x7fb0e00cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:34.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.252+0000 7fb0e5fff700 1 -- 172.21.15.27:0/1129621604 shutdown_connections 2024-01-11T09:47:34.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.252+0000 7fb0e5fff700 1 -- 172.21.15.27:0/1129621604 wait complete. 2024-01-11T09:47:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.254+0000 7fb0e5fff700 1 Processor -- start 2024-01-11T09:47:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.254+0000 7fb0e5fff700 1 -- start start 2024-01-11T09:47:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.255+0000 7fb0e5fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0e0069790 0x7fb0e016c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.255+0000 7fb0e5fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0e0069b60 0x7fb0e006f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.255+0000 7fb0e5fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0e00c8720 0x7fb0e00c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.255+0000 7fb0e5fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e0169550 con 0x7fb0e0069b60 2024-01-11T09:47:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.255+0000 7fb0e5fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e00c4e90 con 0x7fb0e00c8720 2024-01-11T09:47:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.255+0000 7fb0e5fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e016cf90 con 0x7fb0e0069790 2024-01-11T09:47:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.255+0000 7fb0deffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0e0069b60 0x7fb0e006f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.255+0000 7fb0df7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0e0069790 0x7fb0e016c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.255+0000 7fb0df7fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0e0069790 0x7fb0e016c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:47180/0 (socket says 172.21.15.27:47180) 2024-01-11T09:47:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.255+0000 7fb0df7fe700 1 -- 172.21.15.27:0/1583193014 learned_addr learned my addr 172.21.15.27:0/1583193014 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:34.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.255+0000 7fb0dffff700 1 --2- 172.21.15.27:0/1583193014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0e00c8720 0x7fb0e00c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.256+0000 7fb0df7fe700 1 -- 172.21.15.27:0/1583193014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0e00c8720 msgr2=0x7fb0e00c3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.256+0000 7fb0df7fe700 1 --2- 172.21.15.27:0/1583193014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0e00c8720 0x7fb0e00c3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.256+0000 7fb0df7fe700 1 -- 172.21.15.27:0/1583193014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0e0069b60 msgr2=0x7fb0e006f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.256+0000 7fb0df7fe700 1 --2- 172.21.15.27:0/1583193014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0e0069b60 0x7fb0e006f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.256+0000 7fb0df7fe700 1 -- 172.21.15.27:0/1583193014 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d8009a20 con 0x7fb0e0069790 2024-01-11T09:47:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.256+0000 7fb0df7fe700 1 --2- 172.21.15.27:0/1583193014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0e0069790 0x7fb0e016c9b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb0d000c760 tx=0x7fb0d0007da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.256+0000 7fb0c77fe700 1 -- 172.21.15.27:0/1583193014 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d00073b0 con 0x7fb0e0069790 2024-01-11T09:47:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.256+0000 7fb0c77fe700 1 -- 172.21.15.27:0/1583193014 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d0007510 con 0x7fb0e0069790 2024-01-11T09:47:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.256+0000 7fb0c77fe700 1 -- 172.21.15.27:0/1583193014 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d0005a20 con 0x7fb0e0069790 2024-01-11T09:47:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.256+0000 7fb0e5fff700 1 -- 172.21.15.27:0/1583193014 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e016d270 con 0x7fb0e0069790 2024-01-11T09:47:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.257+0000 7fb0e5fff700 1 -- 172.21.15.27:0/1583193014 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e016d830 con 0x7fb0e0069790 2024-01-11T09:47:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.258+0000 7fb0e5fff700 1 -- 172.21.15.27:0/1583193014 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e004c3c0 con 0x7fb0e0069790 2024-01-11T09:47:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.258+0000 7fb0c77fe700 1 -- 172.21.15.27:0/1583193014 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 146) v1 ==== 84963+0+0 (secure 0 0 0) 0x7fb0d001b3b0 con 0x7fb0e0069790 2024-01-11T09:47:34.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.259+0000 7fb0c77fe700 1 --2- 172.21.15.27:0/1583193014 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0cc0671c0 0x7fb0cc069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:34.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.259+0000 7fb0c77fe700 1 -- 172.21.15.27:0/1583193014 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 5817+0+0 (secure 0 0 0) 0x7fb0d0083f50 con 0x7fb0e0069790 2024-01-11T09:47:34.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.259+0000 7fb0deffd700 1 -- 172.21.15.27:0/1583193014 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0cc0671c0 msgr2=0x7fb0cc069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:34.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.260+0000 7fb0deffd700 1 --2- 172.21.15.27:0/1583193014 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0cc0671c0 0x7fb0cc069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:34.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.264+0000 7fb0c77fe700 1 -- 172.21.15.27:0/1583193014 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb0d0050ea0 con 0x7fb0e0069790 2024-01-11T09:47:34.404 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:34.412+0000 7f07bbebe1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:47:34.423 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:34.430+0000 7faeab6861c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:47:34.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.461+0000 7fb0deffd700 1 -- 172.21.15.27:0/1583193014 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0cc0671c0 msgr2=0x7fb0cc069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:34.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.461+0000 7fb0deffd700 1 --2- 172.21.15.27:0/1583193014 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0cc0671c0 0x7fb0cc069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:34.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.509+0000 7fb0e5fff700 1 -- 172.21.15.27:0/1583193014 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fb0e006a110 con 0x7fb0e0069790 2024-01-11T09:47:34.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.514+0000 7fb0c77fe700 1 -- 172.21.15.27:0/1583193014 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v89)=0 v89) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb0d0017070 con 0x7fb0e0069790 2024-01-11T09:47:34.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.517+0000 7fb0e5fff700 1 -- 172.21.15.27:0/1583193014 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0cc0671c0 msgr2=0x7fb0cc069670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:34.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.517+0000 7fb0e5fff700 1 --2- 172.21.15.27:0/1583193014 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0cc0671c0 0x7fb0cc069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:34.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.517+0000 7fb0e5fff700 1 -- 172.21.15.27:0/1583193014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0e0069790 msgr2=0x7fb0e016c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:34.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.517+0000 7fb0e5fff700 1 --2- 172.21.15.27:0/1583193014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0e0069790 0x7fb0e016c9b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb0d000c760 tx=0x7fb0d0007da0 comp rx=0 tx=0).stop 2024-01-11T09:47:34.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.517+0000 7fb0e5fff700 1 -- 172.21.15.27:0/1583193014 shutdown_connections 2024-01-11T09:47:34.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.517+0000 7fb0e5fff700 1 --2- 172.21.15.27:0/1583193014 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0cc0671c0 0x7fb0cc069670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:34.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.517+0000 7fb0e5fff700 1 --2- 172.21.15.27:0/1583193014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0e00c8720 0x7fb0e00c3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:34.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.517+0000 7fb0e5fff700 1 --2- 172.21.15.27:0/1583193014 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0e0069790 0x7fb0e016c9b0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:34.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.517+0000 7fb0e5fff700 1 --2- 172.21.15.27:0/1583193014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0e0069b60 0x7fb0e006f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:34.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.517+0000 7fb0e5fff700 1 -- 172.21.15.27:0/1583193014 >> 172.21.15.27:0/1583193014 conn(0x7fb0e00680a0 msgr2=0x7fb0e00c1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:34.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.517+0000 7fb0e5fff700 1 -- 172.21.15.27:0/1583193014 shutdown_connections 2024-01-11T09:47:34.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.517+0000 7fb0e5fff700 1 -- 172.21.15.27:0/1583193014 wait complete. 2024-01-11T09:47:34.527 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-01-11T09:47:34.718 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:34.725+0000 7f07bbebe1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:47:35.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.741+0000 7fb25d3ff700 1 Processor -- start 2024-01-11T09:47:35.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.741+0000 7fb25d3ff700 1 -- start start 2024-01-11T09:47:35.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.742+0000 7fb25d3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb2580cb4b0 0x7fb2580cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:35.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.742+0000 7fb25d3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2580ccc70 con 0x7fb258142dc0 2024-01-11T09:47:35.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.742+0000 7fb25d3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb25815d320 con 0x7fb258143190 2024-01-11T09:47:35.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.742+0000 7fb25d3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb25815df30 con 0x7fb2580cb4b0 2024-01-11T09:47:35.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.742+0000 7fb256ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb258142dc0 0x7fb2580cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34512/0 (socket says 172.21.15.27:34512) 2024-01-11T09:47:35.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.742+0000 7fb256ffd700 1 -- 172.21.15.27:0/1149618644 learned_addr learned my addr 172.21.15.27:0/1149618644 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:35.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.743+0000 7fb2577fe700 1 --2- 172.21.15.27:0/1149618644 >> v2:172.21.15.77:3300/0 conn(0x7fb2580cb4b0 0x7fb2580cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:35.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.743+0000 7fb255ffb700 1 -- 172.21.15.27:0/1149618644 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138290440 0 0) 0x7fb2580ccc70 con 0x7fb258142dc0 2024-01-11T09:47:35.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.743+0000 7fb255ffb700 1 -- 172.21.15.27:0/1149618644 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb240003570 con 0x7fb258142dc0 2024-01-11T09:47:35.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.743+0000 7fb255ffb700 1 -- 172.21.15.27:0/1149618644 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375493598 0 0) 0x7fb25815d320 con 0x7fb258143190 2024-01-11T09:47:35.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.743+0000 7fb255ffb700 1 -- 172.21.15.27:0/1149618644 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2580ccc70 con 0x7fb258143190 2024-01-11T09:47:35.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.743+0000 7fb255ffb700 1 -- 172.21.15.27:0/1149618644 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2268418536 0 0) 0x7fb240003570 con 0x7fb258142dc0 2024-01-11T09:47:35.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.743+0000 7fb2577fe700 1 -- 172.21.15.27:0/1149618644 >> v1:172.21.15.77:6789/0 conn(0x7fb258143190 legacy=0x7fb2580cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:35.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.743+0000 7fb2577fe700 1 -- 172.21.15.27:0/1149618644 >> v1:172.21.15.27:6789/0 conn(0x7fb258142dc0 legacy=0x7fb2580cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:35.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.743+0000 7fb2577fe700 1 -- 172.21.15.27:0/1149618644 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25815d590 con 0x7fb2580cb4b0 2024-01-11T09:47:35.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.743+0000 7fb2577fe700 1 --2- 172.21.15.27:0/1149618644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2580cb4b0 0x7fb2580cc730 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fb250009e60 tx=0x7fb250009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca58b5cf62b11c87 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:35.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.744+0000 7fb255ffb700 1 -- 172.21.15.27:0/1149618644 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25000e070 con 0x7fb2580cb4b0 2024-01-11T09:47:35.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.744+0000 7fb255ffb700 1 -- 172.21.15.27:0/1149618644 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25000b7a0 con 0x7fb2580cb4b0 2024-01-11T09:47:35.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.744+0000 7fb255ffb700 1 -- 172.21.15.27:0/1149618644 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb248001470 con 0x7fb2580cb4b0 2024-01-11T09:47:35.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.744+0000 7fb25d3ff700 1 -- 172.21.15.27:0/1149618644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2580cb4b0 msgr2=0x7fb2580cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:35.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.744+0000 7fb25d3ff700 1 --2- 172.21.15.27:0/1149618644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2580cb4b0 0x7fb2580cc730 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fb250009e60 tx=0x7fb250009780 comp rx=0 tx=0).stop 2024-01-11T09:47:35.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.744+0000 7fb25d3ff700 1 -- 172.21.15.27:0/1149618644 shutdown_connections 2024-01-11T09:47:35.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.744+0000 7fb25d3ff700 1 --2- 172.21.15.27:0/1149618644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb2580cb4b0 0x7fb2580cc730 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:35.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.744+0000 7fb25d3ff700 1 -- 172.21.15.27:0/1149618644 >> 172.21.15.27:0/1149618644 conn(0x7fb258063ca0 msgr2=0x7fb25806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:35.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.744+0000 7fb25d3ff700 1 -- 172.21.15.27:0/1149618644 shutdown_connections 2024-01-11T09:47:35.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.745+0000 7fb25d3ff700 1 -- 172.21.15.27:0/1149618644 wait complete. 2024-01-11T09:47:35.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.746+0000 7fb25d3ff700 1 Processor -- start 2024-01-11T09:47:35.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.746+0000 7fb25d3ff700 1 -- start start 2024-01-11T09:47:35.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.747+0000 7fb25d3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2580cb4b0 0x7fb2580c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:35.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.747+0000 7fb25d3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb258142dc0 0x7fb2581409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:35.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.747+0000 7fb25d3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb258143190 0x7fb258169bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:35.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.747+0000 7fb25d3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2580c7690 con 0x7fb2580cb4b0 2024-01-11T09:47:35.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.747+0000 7fb25d3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2580c6a80 con 0x7fb258142dc0 2024-01-11T09:47:35.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.747+0000 7fb25d3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2580c6d60 con 0x7fb258143190 2024-01-11T09:47:35.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.747+0000 7fb256ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2580cb4b0 0x7fb2580c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:35.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.747+0000 7fb256ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2580cb4b0 0x7fb2580c5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36898/0 (socket says 172.21.15.27:36898) 2024-01-11T09:47:35.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.747+0000 7fb256ffd700 1 -- 172.21.15.27:0/788996818 learned_addr learned my addr 172.21.15.27:0/788996818 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:35.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.747+0000 7fb2577fe700 1 --2- 172.21.15.27:0/788996818 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb258143190 0x7fb258169bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.747+0000 7fb2567fc700 1 --2- 172.21.15.27:0/788996818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb258142dc0 0x7fb2581409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.748+0000 7fb256ffd700 1 -- 172.21.15.27:0/788996818 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb258143190 msgr2=0x7fb258169bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.748+0000 7fb256ffd700 1 --2- 172.21.15.27:0/788996818 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb258143190 0x7fb258169bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.748+0000 7fb256ffd700 1 -- 172.21.15.27:0/788996818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb258142dc0 msgr2=0x7fb2581409d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.748+0000 7fb256ffd700 1 --2- 172.21.15.27:0/788996818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb258142dc0 0x7fb2581409d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.748+0000 7fb256ffd700 1 -- 172.21.15.27:0/788996818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb250009a20 con 0x7fb2580cb4b0 2024-01-11T09:47:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.748+0000 7fb256ffd700 1 --2- 172.21.15.27:0/788996818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2580cb4b0 0x7fb2580c5ea0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fb24800cb70 tx=0x7fb248001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.748+0000 7fb23effd700 1 -- 172.21.15.27:0/788996818 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24800ce80 con 0x7fb2580cb4b0 2024-01-11T09:47:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.748+0000 7fb23effd700 1 -- 172.21.15.27:0/788996818 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb248007360 con 0x7fb2580cb4b0 2024-01-11T09:47:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.748+0000 7fb23effd700 1 -- 172.21.15.27:0/788996818 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb248007700 con 0x7fb2580cb4b0 2024-01-11T09:47:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.748+0000 7fb25d3ff700 1 -- 172.21.15.27:0/788996818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2580c7040 con 0x7fb2580cb4b0 2024-01-11T09:47:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.749+0000 7fb25d3ff700 1 -- 172.21.15.27:0/788996818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2580c63e0 con 0x7fb2580cb4b0 2024-01-11T09:47:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.750+0000 7fb25d3ff700 1 -- 172.21.15.27:0/788996818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb25804c3c0 con 0x7fb2580cb4b0 2024-01-11T09:47:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.751+0000 7fb23effd700 1 -- 172.21.15.27:0/788996818 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 146) v1 ==== 84963+0+0 (secure 0 0 0) 0x7fb248004a40 con 0x7fb2580cb4b0 2024-01-11T09:47:35.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.752+0000 7fb23effd700 1 --2- 172.21.15.27:0/788996818 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb2440671c0 0x7fb244069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:35.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.752+0000 7fb23effd700 1 -- 172.21.15.27:0/788996818 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 5817+0+0 (secure 0 0 0) 0x7fb248083d60 con 0x7fb2580cb4b0 2024-01-11T09:47:35.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.752+0000 7fb2567fc700 1 -- 172.21.15.27:0/788996818 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb2440671c0 msgr2=0x7fb244069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:35.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.752+0000 7fb2567fc700 1 --2- 172.21.15.27:0/788996818 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb2440671c0 0x7fb244069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:35.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.756+0000 7fb23effd700 1 -- 172.21.15.27:0/788996818 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb248050d50 con 0x7fb2580cb4b0 2024-01-11T09:47:35.043 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:34.771+0000 7faeab6861c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:47:35.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.953+0000 7fb2567fc700 1 -- 172.21.15.27:0/788996818 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb2440671c0 msgr2=0x7fb244069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:35.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.953+0000 7fb2567fc700 1 --2- 172.21.15.27:0/788996818 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb2440671c0 0x7fb244069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:35.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.963+0000 7fb25d3ff700 1 -- 172.21.15.27:0/788996818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fb25815d870 con 0x7fb2580cb4b0 2024-01-11T09:47:35.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.967+0000 7fb23effd700 1 -- 172.21.15.27:0/788996818 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v90)=0 v90) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb248018020 con 0x7fb2580cb4b0 2024-01-11T09:47:35.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.970+0000 7fb25d3ff700 1 -- 172.21.15.27:0/788996818 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb2440671c0 msgr2=0x7fb244069670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:35.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.970+0000 7fb25d3ff700 1 --2- 172.21.15.27:0/788996818 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb2440671c0 0x7fb244069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:35.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.970+0000 7fb25d3ff700 1 -- 172.21.15.27:0/788996818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2580cb4b0 msgr2=0x7fb2580c5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:35.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.970+0000 7fb25d3ff700 1 --2- 172.21.15.27:0/788996818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2580cb4b0 0x7fb2580c5ea0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fb24800cb70 tx=0x7fb248001b50 comp rx=0 tx=0).stop 2024-01-11T09:47:35.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.970+0000 7fb25d3ff700 1 -- 172.21.15.27:0/788996818 shutdown_connections 2024-01-11T09:47:35.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.970+0000 7fb25d3ff700 1 --2- 172.21.15.27:0/788996818 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb258143190 0x7fb258169bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:35.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.970+0000 7fb25d3ff700 1 --2- 172.21.15.27:0/788996818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2580cb4b0 0x7fb2580c5ea0 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:35.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.970+0000 7fb25d3ff700 1 --2- 172.21.15.27:0/788996818 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb2440671c0 0x7fb244069670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:35.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.970+0000 7fb25d3ff700 1 --2- 172.21.15.27:0/788996818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb258142dc0 0x7fb2581409d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:35.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.970+0000 7fb25d3ff700 1 -- 172.21.15.27:0/788996818 >> 172.21.15.27:0/788996818 conn(0x7fb258063ca0 msgr2=0x7fb2580cbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:35.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.970+0000 7fb25d3ff700 1 -- 172.21.15.27:0/788996818 shutdown_connections 2024-01-11T09:47:35.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:34.970+0000 7fb25d3ff700 1 -- 172.21.15.27:0/788996818 wait complete. 2024-01-11T09:47:35.047 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:47:35.047 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-01-11T09:47:35.101 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-01-11T09:47:35.245 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:35.252+0000 7f07bbebe1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:47:35.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.316+0000 7f06bd1ff700 1 Processor -- start 2024-01-11T09:47:35.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.317+0000 7f06bd1ff700 1 -- start start 2024-01-11T09:47:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.318+0000 7f06bd1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f06b813fbb0 0x7f06b80ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.318+0000 7f06bd1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b80c6b30 con 0x7f06b815a730 2024-01-11T09:47:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.318+0000 7f06bd1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b80c6800 con 0x7f06b813ff80 2024-01-11T09:47:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.318+0000 7f06bd1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b80cb240 con 0x7f06b813fbb0 2024-01-11T09:47:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.318+0000 7f06b759e700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f06b815a730 0x7f06b80c6fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34514/0 (socket says 172.21.15.27:34514) 2024-01-11T09:47:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.318+0000 7f06b759e700 1 -- 172.21.15.27:0/1088830454 learned_addr learned my addr 172.21.15.27:0/1088830454 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.318+0000 7f06b6d9d700 1 --2- 172.21.15.27:0/1088830454 >> v2:172.21.15.77:3300/0 conn(0x7f06b813fbb0 0x7f06b80ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:35.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.319+0000 7f06b5d9b700 1 -- 172.21.15.27:0/1088830454 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115717023 0 0) 0x7f06b80c6800 con 0x7f06b813ff80 2024-01-11T09:47:35.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.319+0000 7f06b6d9d700 1 -- 172.21.15.27:0/1088830454 >> v1:172.21.15.77:6789/0 conn(0x7f06b813ff80 legacy=0x7f06b815a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:35.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.319+0000 7f06b6d9d700 1 -- 172.21.15.27:0/1088830454 >> v1:172.21.15.27:6789/0 conn(0x7f06b815a730 legacy=0x7f06b80c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:35.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.319+0000 7f06b6d9d700 1 -- 172.21.15.27:0/1088830454 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b80cb520 con 0x7f06b813fbb0 2024-01-11T09:47:35.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.319+0000 7f06b6d9d700 1 --2- 172.21.15.27:0/1088830454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06b813fbb0 0x7f06b80ca7a0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f06a80098d0 tx=0x7f06a8009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ccb31f3ad4e66d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:35.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.320+0000 7f06b5d9b700 1 -- 172.21.15.27:0/1088830454 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a8009e10 con 0x7f06b813fbb0 2024-01-11T09:47:35.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.320+0000 7f06b5d9b700 1 -- 172.21.15.27:0/1088830454 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a800b760 con 0x7f06b813fbb0 2024-01-11T09:47:35.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.320+0000 7f06b5d9b700 1 -- 172.21.15.27:0/1088830454 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a800bb00 con 0x7f06b813fbb0 2024-01-11T09:47:35.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.320+0000 7f06bd1ff700 1 -- 172.21.15.27:0/1088830454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06b813fbb0 msgr2=0x7f06b80ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:35.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.320+0000 7f06bd1ff700 1 --2- 172.21.15.27:0/1088830454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06b813fbb0 0x7f06b80ca7a0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f06a80098d0 tx=0x7f06a8009120 comp rx=0 tx=0).stop 2024-01-11T09:47:35.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.320+0000 7f06bd1ff700 1 -- 172.21.15.27:0/1088830454 shutdown_connections 2024-01-11T09:47:35.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.320+0000 7f06bd1ff700 1 --2- 172.21.15.27:0/1088830454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06b813fbb0 0x7f06b80ca7a0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:35.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.320+0000 7f06bd1ff700 1 -- 172.21.15.27:0/1088830454 >> 172.21.15.27:0/1088830454 conn(0x7f06b80680a0 msgr2=0x7f06b80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:35.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.320+0000 7f06bd1ff700 1 -- 172.21.15.27:0/1088830454 shutdown_connections 2024-01-11T09:47:35.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.320+0000 7f06bd1ff700 1 -- 172.21.15.27:0/1088830454 wait complete. 2024-01-11T09:47:35.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.322+0000 7f06bd1ff700 1 Processor -- start 2024-01-11T09:47:35.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.322+0000 7f06bd1ff700 1 -- start start 2024-01-11T09:47:35.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.323+0000 7f06bd1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06b813fbb0 0x7f06b80c0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:35.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.323+0000 7f06bd1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b813ff80 0x7f06b8159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:35.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.323+0000 7f06bd1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f06b815a730 0x7f06b80b18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:35.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.323+0000 7f06bd1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b80c21e0 con 0x7f06b813ff80 2024-01-11T09:47:35.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.323+0000 7f06bd1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b81598b0 con 0x7f06b813fbb0 2024-01-11T09:47:35.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.323+0000 7f06bd1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b80c15d0 con 0x7f06b815a730 2024-01-11T09:47:35.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.324+0000 7f06b6d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06b813fbb0 0x7f06b80c0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:35.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.324+0000 7f06b659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b813ff80 0x7f06b8159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:35.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.324+0000 7f06b6d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06b813fbb0 0x7f06b80c0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:55122/0 (socket says 172.21.15.27:55122) 2024-01-11T09:47:35.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.324+0000 7f06b6d9d700 1 -- 172.21.15.27:0/836479866 learned_addr learned my addr 172.21.15.27:0/836479866 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:35.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.324+0000 7f06b759e700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f06b815a730 0x7f06b80b18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:35.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.324+0000 7f06b6d9d700 1 -- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f06b815a730 msgr2=0x7f06b80b18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:35.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.324+0000 7f06b6d9d700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f06b815a730 0x7f06b80b18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:35.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.324+0000 7f06b6d9d700 1 -- 172.21.15.27:0/836479866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b813ff80 msgr2=0x7f06b8159370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:35.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.324+0000 7f06b6d9d700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b813ff80 0x7f06b8159370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:35.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.324+0000 7f06b6d9d700 1 -- 172.21.15.27:0/836479866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a80093c0 con 0x7f06b813fbb0 2024-01-11T09:47:35.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.324+0000 7f06b6d9d700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06b813fbb0 0x7f06b80c0f60 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f06a8009160 tx=0x7f06a8016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:35.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.325+0000 7f069effd700 1 -- 172.21.15.27:0/836479866 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a8000d10 con 0x7f06b813fbb0 2024-01-11T09:47:35.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.325+0000 7f069effd700 1 -- 172.21.15.27:0/836479866 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a801c460 con 0x7f06b813fbb0 2024-01-11T09:47:35.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.325+0000 7f06bd1ff700 1 -- 172.21.15.27:0/836479866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b80c18b0 con 0x7f06b813fbb0 2024-01-11T09:47:35.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.325+0000 7f069effd700 1 -- 172.21.15.27:0/836479866 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a801c820 con 0x7f06b813fbb0 2024-01-11T09:47:35.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.325+0000 7f06bd1ff700 1 -- 172.21.15.27:0/836479866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b80c1e70 con 0x7f06b813fbb0 2024-01-11T09:47:35.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.326+0000 7f06bd1ff700 1 -- 172.21.15.27:0/836479866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b804c3c0 con 0x7f06b813fbb0 2024-01-11T09:47:35.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.328+0000 7f069effd700 1 -- 172.21.15.27:0/836479866 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 146) v1 ==== 84963+0+0 (secure 0 0 0) 0x7f06a801cc10 con 0x7f06b813fbb0 2024-01-11T09:47:35.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.329+0000 7f069effd700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f06a40671c0 0x7f06a4069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:35.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.329+0000 7f069effd700 1 -- 172.21.15.27:0/836479866 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 5817+0+0 (secure 0 0 0) 0x7f06a8086d50 con 0x7f06b813fbb0 2024-01-11T09:47:35.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.329+0000 7f06b659c700 1 -- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f06a40671c0 msgr2=0x7f06a4069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:35.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.329+0000 7f06b659c700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f06a40671c0 0x7f06a4069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.338+0000 7f069effd700 1 -- 172.21.15.27:0/836479866 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f06a808a030 con 0x7f06b813fbb0 2024-01-11T09:47:35.342 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:35.350+0000 7faeab6861c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:47:35.420 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:35.428+0000 7f07bbebe1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:47:35.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.530+0000 7f06b659c700 1 -- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f06a40671c0 msgr2=0x7f06a4069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:35.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.530+0000 7f06b659c700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f06a40671c0 0x7f06a4069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:35.523 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:35.532+0000 7faeab6861c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:47:35.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.569+0000 7f06bd1ff700 1 -- 172.21.15.27:0/836479866 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f06b80c2320 con 0x7f06a40671c0 2024-01-11T09:47:35.838 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:35.847+0000 7f07bbebe1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:47:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.931+0000 7f06b659c700 1 -- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f06a40671c0 msgr2=0x7f06a4069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:35.931+0000 7f06b659c700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f06a40671c0 0x7f06a4069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-11T09:47:35.958 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:35.967+0000 7faeab6861c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:47:36.617 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:36.625+0000 7f07bbebe1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:47:36.706 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:36.714+0000 7f07bbebe1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:47:36.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:36.732+0000 7f06b659c700 1 -- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f06a40671c0 msgr2=0x7f06a4069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:36.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:36.732+0000 7f06b659c700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f06a40671c0 0x7f06a4069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-11T09:47:36.781 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:36.790+0000 7faeab6861c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:47:36.862 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:36.870+0000 7f07bbebe1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:47:36.874 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:36.883+0000 7faeab6861c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:47:36.950 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:36.958+0000 7f07bbebe1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:47:37.039 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:37.047+0000 7faeab6861c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:47:37.132 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:37.140+0000 7faeab6861c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:47:37.171 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:37.179+0000 7f07bbebe1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:47:37.267 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:37.275+0000 7f07bbebe1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:47:37.360 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:37.369+0000 7faeab6861c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:47:37.363 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:37.371+0000 7f07bbebe1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:47:37.460 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:37.468+0000 7f07bbebe1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:47:37.462 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:37.470+0000 7faeab6861c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:47:37.564 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:37.572+0000 7faeab6861c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:47:37.652 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:37.660+0000 7f07bbebe1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:47:37.664 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:37.673+0000 7faeab6861c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:47:37.750 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:37.759+0000 7f07bbebe1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:47:37.863 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:37.872+0000 7faeab6861c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:47:37.968 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:37.977+0000 7faeab6861c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:47:38.097 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:38.105+0000 7f07bbebe1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:47:38.187 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:38.196+0000 7f07bbebe1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:47:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:38.333+0000 7f06b659c700 1 -- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f06a40671c0 msgr2=0x7f06a4069670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:38.333+0000 7f06b659c700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f06a40671c0 0x7f06a4069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-11T09:47:38.328 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:38.337+0000 7faeab6861c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:47:38.423 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:38.432+0000 7faeab6861c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:47:38.729 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:38.738+0000 7f07bbebe1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:47:38.917 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:38.926+0000 7f07bbebe1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:47:38.999 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:39.008+0000 7faeab6861c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:47:39.197 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:39.205+0000 7faeab6861c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:47:39.300 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:39.309+0000 7f07bbebe1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:47:39.594 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:39.602+0000 7faeab6861c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:47:39.724 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:39.733+0000 7f07bbebe1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:47:39.907 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:39.916+0000 7f07bbebe1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:47:40.045 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:40.054+0000 7faeab6861c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:47:40.102 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:40.109+0000 7f07bbebe1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:47:40.209 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:40.218+0000 7f07bbebe1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:47:40.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:40.230+0000 7f069effd700 1 -- 172.21.15.27:0/836479866 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 147) v1 ==== 84470+0+0 (secure 0 0 0) 0x7f06a8054140 con 0x7f06b813fbb0 2024-01-11T09:47:40.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:40.230+0000 7f069effd700 1 -- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f06a40671c0 msgr2=0x7f06a4069670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:40.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:40.230+0000 7f069effd700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f06a40671c0 0x7f06a4069670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:40.242 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:40.250+0000 7faeab6861c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:47:40.340 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:40.347+0000 7f07344bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:40.340 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:40.347+0000 7f07344bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:40.340 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:40.347+0000 7f07344bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:40.340 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:40.347+0000 7f07344bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:40.340 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:40.347+0000 7f07344bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:40.341 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:40.347+0000 7f07314b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:40.342 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:40.347+0000 7f07314b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:40.342 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:40.347+0000 7f07314b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:40.342 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:40.347+0000 7f07314b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:40.342 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:40.347+0000 7f07314b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:40.445 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:40.454+0000 7faeab6861c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:47:40.562 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:40.571+0000 7faeab6861c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:47:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.243+0000 7f069effd700 1 -- 172.21.15.27:0/836479866 <== mon.1 v2:172.21.15.77:3300/0 8 ==== mgrmap(e 148) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f06a805a750 con 0x7f06b813fbb0 2024-01-11T09:47:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.243+0000 7f069effd700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f06a40671c0 0x7f06a4069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.243+0000 7f069effd700 1 -- 172.21.15.27:0/836479866 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f06a8000d10 con 0x7f06a40671c0 2024-01-11T09:47:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.244+0000 7f06b659c700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f06a40671c0 0x7f06a4069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.245+0000 7f06b659c700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f06a40671c0 0x7f06a4069670 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f06b80af420 tx=0x7f06ac007d30 comp rx=0 tx=0).ready entity=mgr.9995 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:41.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.310+0000 7f069effd700 1 -- 172.21.15.27:0/836479866 <== mgr.9995 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f06a8000d10 con 0x7f06a40671c0 2024-01-11T09:47:41.301 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule set for path snap_test_dir1 2024-01-11T09:47:41.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.312+0000 7f06bd1ff700 1 -- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f06a40671c0 msgr2=0x7f06a4069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:41.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.312+0000 7f06bd1ff700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f06a40671c0 0x7f06a4069670 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f06b80af420 tx=0x7f06ac007d30 comp rx=0 tx=0).stop 2024-01-11T09:47:41.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.312+0000 7f06bd1ff700 1 -- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06b813fbb0 msgr2=0x7f06b80c0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:41.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.312+0000 7f06bd1ff700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06b813fbb0 0x7f06b80c0f60 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f06a8009160 tx=0x7f06a8016000 comp rx=0 tx=0).stop 2024-01-11T09:47:41.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.312+0000 7f06bd1ff700 1 -- 172.21.15.27:0/836479866 shutdown_connections 2024-01-11T09:47:41.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.312+0000 7f06bd1ff700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f06b815a730 0x7f06b80b18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:41.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.312+0000 7f06bd1ff700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f06a40671c0 0x7f06a4069670 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:41.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.312+0000 7f06bd1ff700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06b813fbb0 0x7f06b80c0f60 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:41.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.312+0000 7f06bd1ff700 1 --2- 172.21.15.27:0/836479866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b813ff80 0x7f06b8159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:41.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.312+0000 7f06bd1ff700 1 -- 172.21.15.27:0/836479866 >> 172.21.15.27:0/836479866 conn(0x7f06b80680a0 msgr2=0x7f06b8069880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:41.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.313+0000 7f06bd1ff700 1 -- 172.21.15.27:0/836479866 shutdown_connections 2024-01-11T09:47:41.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.313+0000 7f06bd1ff700 1 -- 172.21.15.27:0/836479866 wait complete. 2024-01-11T09:47:41.323 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-01-11T09:47:41.324 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2024-01-11T09:47:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.535+0000 7fcf6e1ff700 1 Processor -- start 2024-01-11T09:47:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.536+0000 7fcf6e1ff700 1 -- start start 2024-01-11T09:47:41.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.537+0000 7fcf6e1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf68068be0 0x7fcf680ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:41.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.537+0000 7fcf6e1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680b0d20 con 0x7fcf68159250 2024-01-11T09:47:41.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.537+0000 7fcf6e1ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf6815a440 con 0x7fcf680b08a0 2024-01-11T09:47:41.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.537+0000 7fcf6e1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6815ace0 con 0x7fcf68068be0 2024-01-11T09:47:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.537+0000 7fcf677fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcf68159250 0x7fcf680cb410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34522/0 (socket says 172.21.15.27:34522) 2024-01-11T09:47:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.537+0000 7fcf677fe700 1 -- 172.21.15.27:0/2786699562 learned_addr learned my addr 172.21.15.27:0/2786699562 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.537+0000 7fcf67fff700 1 --2- 172.21.15.27:0/2786699562 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf68068be0 0x7fcf680ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.538+0000 7fcf667fc700 1 -- 172.21.15.27:0/2786699562 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177267926 0 0) 0x7fcf680b0d20 con 0x7fcf68159250 2024-01-11T09:47:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.538+0000 7fcf667fc700 1 -- 172.21.15.27:0/2786699562 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf50003570 con 0x7fcf68159250 2024-01-11T09:47:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.538+0000 7fcf667fc700 1 -- 172.21.15.27:0/2786699562 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755370162 0 0) 0x7fcf6815a440 con 0x7fcf680b08a0 2024-01-11T09:47:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.538+0000 7fcf667fc700 1 -- 172.21.15.27:0/2786699562 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf680b0d20 con 0x7fcf680b08a0 2024-01-11T09:47:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.538+0000 7fcf67fff700 1 -- 172.21.15.27:0/2786699562 >> v1:172.21.15.77:6789/0 conn(0x7fcf680b08a0 legacy=0x7fcf680684d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.538+0000 7fcf67fff700 1 -- 172.21.15.27:0/2786699562 >> v1:172.21.15.27:6789/0 conn(0x7fcf68159250 legacy=0x7fcf680cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.538+0000 7fcf67fff700 1 -- 172.21.15.27:0/2786699562 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6815a720 con 0x7fcf68068be0 2024-01-11T09:47:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.538+0000 7fcf67fff700 1 --2- 172.21.15.27:0/2786699562 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf68068be0 0x7fcf680ce2d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009e90 tx=0x7fcf60009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fab8b325fd140e4c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.538+0000 7fcf667fc700 1 -- 172.21.15.27:0/2786699562 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6000e070 con 0x7fcf68068be0 2024-01-11T09:47:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.538+0000 7fcf667fc700 1 -- 172.21.15.27:0/2786699562 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf6000bb80 con 0x7fcf68068be0 2024-01-11T09:47:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.538+0000 7fcf667fc700 1 -- 172.21.15.27:0/2786699562 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf60012430 con 0x7fcf68068be0 2024-01-11T09:47:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.539+0000 7fcf6e1ff700 1 -- 172.21.15.27:0/2786699562 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf68068be0 msgr2=0x7fcf680ce2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.539+0000 7fcf6e1ff700 1 --2- 172.21.15.27:0/2786699562 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf68068be0 0x7fcf680ce2d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009e90 tx=0x7fcf60009780 comp rx=0 tx=0).stop 2024-01-11T09:47:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.539+0000 7fcf6e1ff700 1 -- 172.21.15.27:0/2786699562 shutdown_connections 2024-01-11T09:47:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.539+0000 7fcf6e1ff700 1 --2- 172.21.15.27:0/2786699562 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf68068be0 0x7fcf680ce2d0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.539+0000 7fcf6e1ff700 1 -- 172.21.15.27:0/2786699562 >> 172.21.15.27:0/2786699562 conn(0x7fcf68067610 msgr2=0x7fcf68067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.539+0000 7fcf6e1ff700 1 -- 172.21.15.27:0/2786699562 shutdown_connections 2024-01-11T09:47:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.539+0000 7fcf6e1ff700 1 -- 172.21.15.27:0/2786699562 wait complete. 2024-01-11T09:47:41.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.541+0000 7fcf6e1ff700 1 Processor -- start 2024-01-11T09:47:41.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.541+0000 7fcf6e1ff700 1 -- start start 2024-01-11T09:47:41.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.542+0000 7fcf6e1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf68068be0 0x7fcf680c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:41.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.542+0000 7fcf6e1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf680b08a0 0x7fcf6816a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:41.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.542+0000 7fcf6e1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf68159250 0x7fcf680c1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:41.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.542+0000 7fcf6e1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf680c4eb0 con 0x7fcf680b08a0 2024-01-11T09:47:41.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.542+0000 7fcf6e1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf680c42a0 con 0x7fcf68159250 2024-01-11T09:47:41.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.542+0000 7fcf6e1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf680c4580 con 0x7fcf68068be0 2024-01-11T09:47:41.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.542+0000 7fcf66ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf680b08a0 0x7fcf6816a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:41.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.542+0000 7fcf677fe700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf68068be0 0x7fcf680c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:41.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.542+0000 7fcf66ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf680b08a0 0x7fcf6816a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36912/0 (socket says 172.21.15.27:36912) 2024-01-11T09:47:41.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.542+0000 7fcf66ffd700 1 -- 172.21.15.27:0/3972232182 learned_addr learned my addr 172.21.15.27:0/3972232182 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:41.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.542+0000 7fcf67fff700 1 --2- 172.21.15.27:0/3972232182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf68159250 0x7fcf680c1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:41.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.542+0000 7fcf66ffd700 1 -- 172.21.15.27:0/3972232182 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf68068be0 msgr2=0x7fcf680c3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:41.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.542+0000 7fcf66ffd700 1 --2- 172.21.15.27:0/3972232182 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf68068be0 0x7fcf680c3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:41.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.542+0000 7fcf66ffd700 1 -- 172.21.15.27:0/3972232182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf68159250 msgr2=0x7fcf680c1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:41.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.542+0000 7fcf66ffd700 1 --2- 172.21.15.27:0/3972232182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf68159250 0x7fcf680c1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:41.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.542+0000 7fcf66ffd700 1 -- 172.21.15.27:0/3972232182 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf60009a20 con 0x7fcf680b08a0 2024-01-11T09:47:41.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.543+0000 7fcf677fe700 1 --2- 172.21.15.27:0/3972232182 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf68068be0 0x7fcf680c3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:41.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.543+0000 7fcf66ffd700 1 --2- 172.21.15.27:0/3972232182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf680b08a0 0x7fcf6816a0c0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00c790 tx=0x7fcf5c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:41.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.543+0000 7fcf4f7fe700 1 -- 172.21.15.27:0/3972232182 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c010510 con 0x7fcf680b08a0 2024-01-11T09:47:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.543+0000 7fcf4f7fe700 1 -- 172.21.15.27:0/3972232182 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf5c010670 con 0x7fcf680b08a0 2024-01-11T09:47:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.543+0000 7fcf6e1ff700 1 -- 172.21.15.27:0/3972232182 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf680c4860 con 0x7fcf680b08a0 2024-01-11T09:47:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.543+0000 7fcf4f7fe700 1 -- 172.21.15.27:0/3972232182 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c010a10 con 0x7fcf680b08a0 2024-01-11T09:47:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.543+0000 7fcf6e1ff700 1 -- 172.21.15.27:0/3972232182 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf6816a600 con 0x7fcf680b08a0 2024-01-11T09:47:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.544+0000 7fcf6e1ff700 1 -- 172.21.15.27:0/3972232182 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf680c5060 con 0x7fcf680b08a0 2024-01-11T09:47:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.545+0000 7fcf4f7fe700 1 -- 172.21.15.27:0/3972232182 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 148) v1 ==== 84597+0+0 (secure 0 0 0) 0x7fcf5c01b3b0 con 0x7fcf680b08a0 2024-01-11T09:47:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.546+0000 7fcf4f7fe700 1 --2- 172.21.15.27:0/3972232182 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcf54066dd0 0x7fcf54069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:41.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.546+0000 7fcf4f7fe700 1 -- 172.21.15.27:0/3972232182 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6247+0+0 (secure 0 0 0) 0x7fcf5c083da0 con 0x7fcf680b08a0 2024-01-11T09:47:41.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.547+0000 7fcf677fe700 1 --2- 172.21.15.27:0/3972232182 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcf54066dd0 0x7fcf54069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:41.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.547+0000 7fcf677fe700 1 --2- 172.21.15.27:0/3972232182 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcf54066dd0 0x7fcf54069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcf58007fd0 tx=0x7fcf58009040 comp rx=0 tx=0).ready entity=mgr.9995 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.551+0000 7fcf4f7fe700 1 -- 172.21.15.27:0/3972232182 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcf680c5060 con 0x7fcf680b08a0 2024-01-11T09:47:41.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.778+0000 7fcf6e1ff700 1 -- 172.21.15.27:0/3972232182 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf680c5060 con 0x7fcf54066dd0 2024-01-11T09:47:41.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.831+0000 7fcf4f7fe700 1 -- 172.21.15.27:0/3972232182 <== mgr.9995 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7fcf680c5060 con 0x7fcf54066dd0 2024-01-11T09:47:41.823 INFO:teuthology.orchestra.run.smithi027.stdout:Retention added to path snap_test_dir1 2024-01-11T09:47:41.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.834+0000 7fcf6e1ff700 1 -- 172.21.15.27:0/3972232182 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcf54066dd0 msgr2=0x7fcf54069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:41.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.834+0000 7fcf6e1ff700 1 --2- 172.21.15.27:0/3972232182 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcf54066dd0 0x7fcf54069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcf58007fd0 tx=0x7fcf58009040 comp rx=0 tx=0).stop 2024-01-11T09:47:41.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.834+0000 7fcf6e1ff700 1 -- 172.21.15.27:0/3972232182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf680b08a0 msgr2=0x7fcf6816a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:41.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.834+0000 7fcf6e1ff700 1 --2- 172.21.15.27:0/3972232182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf680b08a0 0x7fcf6816a0c0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00c790 tx=0x7fcf5c007e10 comp rx=0 tx=0).stop 2024-01-11T09:47:41.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.834+0000 7fcf6e1ff700 1 -- 172.21.15.27:0/3972232182 shutdown_connections 2024-01-11T09:47:41.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.834+0000 7fcf6e1ff700 1 --2- 172.21.15.27:0/3972232182 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fcf54066dd0 0x7fcf54069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:41.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.834+0000 7fcf6e1ff700 1 --2- 172.21.15.27:0/3972232182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf68159250 0x7fcf680c1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:41.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.834+0000 7fcf6e1ff700 1 --2- 172.21.15.27:0/3972232182 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fcf68068be0 0x7fcf680c3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:41.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.834+0000 7fcf6e1ff700 1 --2- 172.21.15.27:0/3972232182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf680b08a0 0x7fcf6816a0c0 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:41.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.834+0000 7fcf6e1ff700 1 -- 172.21.15.27:0/3972232182 >> 172.21.15.27:0/3972232182 conn(0x7fcf68067610 msgr2=0x7fcf68158840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:41.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.834+0000 7fcf6e1ff700 1 -- 172.21.15.27:0/3972232182 shutdown_connections 2024-01-11T09:47:41.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:41.834+0000 7fcf6e1ff700 1 -- 172.21.15.27:0/3972232182 wait complete. 2024-01-11T09:47:41.844 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2024-01-11T09:47:41.848 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-01-11-09_48_00 in ~19s... 2024-01-11T09:47:41.848 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2024-01-11T09:47:41.848 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-01-11T09:47:42.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.059+0000 7f20af1ff700 1 Processor -- start 2024-01-11T09:47:42.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.059+0000 7f20af1ff700 1 -- start start 2024-01-11T09:47:42.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.060+0000 7f20af1ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f20a813fbb0 0x7f20a80ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:42.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.060+0000 7f20af1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f20a813ff80 0x7f20a815a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:42.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.060+0000 7f20af1ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a80c6b30 con 0x7f20a80cace0 2024-01-11T09:47:42.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.060+0000 7f20af1ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a80c6800 con 0x7f20a813fbb0 2024-01-11T09:47:42.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.060+0000 7f20af1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a80cb240 con 0x7f20a813ff80 2024-01-11T09:47:42.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.061+0000 7f20ad79c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f20a80cace0 0x7f20a806e6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34534/0 (socket says 172.21.15.27:34534) 2024-01-11T09:47:42.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.061+0000 7f20ad79c700 1 -- 172.21.15.27:0/1826510453 learned_addr learned my addr 172.21.15.27:0/1826510453 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:42.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.061+0000 7f20acf9b700 1 --2- 172.21.15.27:0/1826510453 >> v2:172.21.15.77:3300/0 conn(0x7f20a813fbb0 0x7f20a80ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:42.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.061+0000 7f20a7fff700 1 --2- 172.21.15.27:0/1826510453 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f20a813ff80 0x7f20a815a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:42.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.061+0000 7f20a77fe700 1 -- 172.21.15.27:0/1826510453 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1600853241 0 0) 0x7f20a80c6b30 con 0x7f20a80cace0 2024-01-11T09:47:42.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.061+0000 7f20a77fe700 1 -- 172.21.15.27:0/1826510453 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2098003570 con 0x7f20a80cace0 2024-01-11T09:47:42.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.061+0000 7f20a77fe700 1 -- 172.21.15.27:0/1826510453 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3926639352 0 0) 0x7f2098003570 con 0x7f20a80cace0 2024-01-11T09:47:42.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.061+0000 7f20a77fe700 1 -- 172.21.15.27:0/1826510453 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f20a813ff80 msgr2=0x7f20a815a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:42.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.061+0000 7f20a77fe700 1 --2- 172.21.15.27:0/1826510453 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f20a813ff80 0x7f20a815a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.061+0000 7f20a77fe700 1 -- 172.21.15.27:0/1826510453 >> v2:172.21.15.77:3300/0 conn(0x7f20a813fbb0 msgr2=0x7f20a80ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.061+0000 7f20a77fe700 1 --2- 172.21.15.27:0/1826510453 >> v2:172.21.15.77:3300/0 conn(0x7f20a813fbb0 0x7f20a80ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.061+0000 7f20a77fe700 1 -- 172.21.15.27:0/1826510453 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a80cb520 con 0x7f20a80cace0 2024-01-11T09:47:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.061+0000 7f20a77fe700 1 -- 172.21.15.27:0/1826510453 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f20a00017f0 con 0x7f20a80cace0 2024-01-11T09:47:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.061+0000 7f20acf9b700 1 --2- 172.21.15.27:0/1826510453 >> v2:172.21.15.77:3300/0 conn(0x7f20a813fbb0 0x7f20a80ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.061+0000 7f20a77fe700 1 -- 172.21.15.27:0/1826510453 >> v1:172.21.15.27:6789/0 conn(0x7f20a80cace0 legacy=0x7f20a806e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.062+0000 7f20a77fe700 1 --2- 172.21.15.27:0/1826510453 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f20a813ff80 0x7f2098003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.062+0000 7f20a77fe700 1 --2- 172.21.15.27:0/1826510453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2098003fe0 0x7f2098004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.062+0000 7f20a77fe700 1 --2- 172.21.15.27:0/1826510453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2098004d50 0x7f20980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.062+0000 7f20a77fe700 1 -- 172.21.15.27:0/1826510453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a80c6800 con 0x7f2098004d50 2024-01-11T09:47:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.062+0000 7f20a77fe700 1 -- 172.21.15.27:0/1826510453 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a80cb240 con 0x7f2098003fe0 2024-01-11T09:47:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.062+0000 7f20a77fe700 1 -- 172.21.15.27:0/1826510453 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209800ba90 con 0x7f20a813ff80 2024-01-11T09:47:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.062+0000 7f20acf9b700 1 --2- 172.21.15.27:0/1826510453 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f20a813ff80 0x7f2098003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.062+0000 7f20ad79c700 1 --2- 172.21.15.27:0/1826510453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2098004d50 0x7f20980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.062+0000 7f20a7fff700 1 --2- 172.21.15.27:0/1826510453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2098003fe0 0x7f2098004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.062+0000 7f20acf9b700 1 -- 172.21.15.27:0/1826510453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2098003fe0 msgr2=0x7f2098004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.062+0000 7f20acf9b700 1 --2- 172.21.15.27:0/1826510453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2098003fe0 0x7f2098004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.062+0000 7f20acf9b700 1 -- 172.21.15.27:0/1826510453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2098004d50 msgr2=0x7f20980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.062+0000 7f20acf9b700 1 --2- 172.21.15.27:0/1826510453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2098004d50 0x7f20980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.062+0000 7f20acf9b700 1 -- 172.21.15.27:0/1826510453 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2098003570 con 0x7f20a813ff80 2024-01-11T09:47:42.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.063+0000 7f20acf9b700 1 --2- 172.21.15.27:0/1826510453 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f20a813ff80 0x7f2098003a60 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f209c009de0 tx=0x7f209c009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f4fa3884db68dea server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:42.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.063+0000 7f20a77fe700 1 -- 172.21.15.27:0/1826510453 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c010ad0 con 0x7f20a813ff80 2024-01-11T09:47:42.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.063+0000 7f20a77fe700 1 -- 172.21.15.27:0/1826510453 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209c010c30 con 0x7f20a813ff80 2024-01-11T09:47:42.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.063+0000 7f20a77fe700 1 -- 172.21.15.27:0/1826510453 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c005a00 con 0x7f20a813ff80 2024-01-11T09:47:42.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.063+0000 7f20af1ff700 1 -- 172.21.15.27:0/1826510453 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f20a813ff80 msgr2=0x7f2098003a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:42.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.063+0000 7f20af1ff700 1 --2- 172.21.15.27:0/1826510453 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f20a813ff80 0x7f2098003a60 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f209c009de0 tx=0x7f209c009120 comp rx=0 tx=0).stop 2024-01-11T09:47:42.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.063+0000 7f20ad79c700 1 -- 172.21.15.27:0/1826510453 reap_dead start 2024-01-11T09:47:42.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.063+0000 7f20af1ff700 1 -- 172.21.15.27:0/1826510453 shutdown_connections 2024-01-11T09:47:42.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.063+0000 7f20af1ff700 1 -- 172.21.15.27:0/1826510453 >> 172.21.15.27:0/1826510453 conn(0x7f20a80680a0 msgr2=0x7f20a80cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:42.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.063+0000 7f20af1ff700 1 -- 172.21.15.27:0/1826510453 shutdown_connections 2024-01-11T09:47:42.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.064+0000 7f20af1ff700 1 -- 172.21.15.27:0/1826510453 wait complete. 2024-01-11T09:47:42.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.065+0000 7f20af1ff700 1 Processor -- start 2024-01-11T09:47:42.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.065+0000 7f20af1ff700 1 -- start start 2024-01-11T09:47:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.066+0000 7f20af1ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20a813ff80 0x7f20a80b1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.066+0000 7f20af1ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f20a813d9d0 0x7f20a80b0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.066+0000 7f20af1ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20a8154190 0x7f20a80c1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.066+0000 7f20af1ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a80b17c0 con 0x7f20a813ff80 2024-01-11T09:47:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.066+0000 7f20af1ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a80b0bb0 con 0x7f20a8154190 2024-01-11T09:47:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.066+0000 7f20af1ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a80b0e90 con 0x7f20a813d9d0 2024-01-11T09:47:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.066+0000 7f20acf9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20a813ff80 0x7f20a80b1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.066+0000 7f20acf9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20a813ff80 0x7f20a80b1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36944/0 (socket says 172.21.15.27:36944) 2024-01-11T09:47:42.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.066+0000 7f20acf9b700 1 -- 172.21.15.27:0/1713754583 learned_addr learned my addr 172.21.15.27:0/1713754583 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.067+0000 7f20a7fff700 1 --2- 172.21.15.27:0/1713754583 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f20a813d9d0 0x7f20a80b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.067+0000 7f20ad79c700 1 --2- 172.21.15.27:0/1713754583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20a8154190 0x7f20a80c1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.067+0000 7f20acf9b700 1 -- 172.21.15.27:0/1713754583 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f20a813d9d0 msgr2=0x7f20a80b0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.067+0000 7f20acf9b700 1 --2- 172.21.15.27:0/1713754583 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f20a813d9d0 0x7f20a80b0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.067+0000 7f20acf9b700 1 -- 172.21.15.27:0/1713754583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20a8154190 msgr2=0x7f20a80c1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.067+0000 7f20acf9b700 1 --2- 172.21.15.27:0/1713754583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20a8154190 0x7f20a80c1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.067+0000 7f20acf9b700 1 -- 172.21.15.27:0/1713754583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a80b1170 con 0x7f20a813ff80 2024-01-11T09:47:42.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.067+0000 7f20acf9b700 1 --2- 172.21.15.27:0/1713754583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20a813ff80 0x7f20a80b1c50 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f209800bc20 tx=0x7f209c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.067+0000 7f20a4ff9700 1 -- 172.21.15.27:0/1713754583 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c005d70 con 0x7f20a813ff80 2024-01-11T09:47:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.068+0000 7f20a4ff9700 1 -- 172.21.15.27:0/1713754583 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209c010da0 con 0x7f20a813ff80 2024-01-11T09:47:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.068+0000 7f20a4ff9700 1 -- 172.21.15.27:0/1713754583 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c0236a0 con 0x7f20a813ff80 2024-01-11T09:47:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.068+0000 7f20af1ff700 1 -- 172.21.15.27:0/1713754583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a80b1450 con 0x7f20a813ff80 2024-01-11T09:47:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.068+0000 7f20af1ff700 1 -- 172.21.15.27:0/1713754583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a8154600 con 0x7f20a813ff80 2024-01-11T09:47:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.069+0000 7f20af1ff700 1 -- 172.21.15.27:0/1713754583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a813dda0 con 0x7f20a813ff80 2024-01-11T09:47:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.070+0000 7f20a4ff9700 1 -- 172.21.15.27:0/1713754583 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 148) v1 ==== 84597+0+0 (secure 0 0 0) 0x7f209c023a50 con 0x7f20a813ff80 2024-01-11T09:47:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.070+0000 7f20a4ff9700 1 --2- 172.21.15.27:0/1713754583 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f208c066dd0 0x7f208c069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:42.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.071+0000 7f20a4ff9700 1 -- 172.21.15.27:0/1713754583 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6247+0+0 (secure 0 0 0) 0x7f209c0864a0 con 0x7f20a813ff80 2024-01-11T09:47:42.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.071+0000 7f20a7fff700 1 --2- 172.21.15.27:0/1713754583 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f208c066dd0 0x7f208c069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:42.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.072+0000 7f20a7fff700 1 --2- 172.21.15.27:0/1713754583 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f208c066dd0 0x7f208c069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f20a8155f30 tx=0x7f20940091a0 comp rx=0 tx=0).ready entity=mgr.9995 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:42.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.075+0000 7f20a4ff9700 1 -- 172.21.15.27:0/1713754583 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f209c053790 con 0x7f20a813ff80 2024-01-11T09:47:42.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.317+0000 7f20a4ff9700 1 -- 172.21.15.27:0/1713754583 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 149) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f209c053b60 con 0x7f20a813ff80 2024-01-11T09:47:42.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.332+0000 7f20af1ff700 1 -- 172.21.15.27:0/1713754583 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f20a80b1900 con 0x7f208c066dd0 2024-01-11T09:47:42.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.343+0000 7f20a4ff9700 1 -- 172.21.15.27:0/1713754583 <== mgr.9995 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f2094014940 con 0x7f208c066dd0 2024-01-11T09:47:42.335 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:42.335 INFO:teuthology.orchestra.run.smithi027.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-01-11T09:47:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.346+0000 7f20af1ff700 1 -- 172.21.15.27:0/1713754583 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f208c066dd0 msgr2=0x7f208c069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.346+0000 7f20af1ff700 1 --2- 172.21.15.27:0/1713754583 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f208c066dd0 0x7f208c069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f20a8155f30 tx=0x7f20940091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.346+0000 7f20af1ff700 1 -- 172.21.15.27:0/1713754583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20a813ff80 msgr2=0x7f20a80b1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.346+0000 7f20af1ff700 1 --2- 172.21.15.27:0/1713754583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20a813ff80 0x7f20a80b1c50 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f209800bc20 tx=0x7f209c018000 comp rx=0 tx=0).stop 2024-01-11T09:47:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.346+0000 7f20af1ff700 1 -- 172.21.15.27:0/1713754583 shutdown_connections 2024-01-11T09:47:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.346+0000 7f20af1ff700 1 --2- 172.21.15.27:0/1713754583 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f208c066dd0 0x7f208c069280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.346+0000 7f20af1ff700 1 --2- 172.21.15.27:0/1713754583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20a8154190 0x7f20a80c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.346+0000 7f20af1ff700 1 --2- 172.21.15.27:0/1713754583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20a813ff80 0x7f20a80b1c50 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.346+0000 7f20af1ff700 1 --2- 172.21.15.27:0/1713754583 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f20a813d9d0 0x7f20a80b0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.346+0000 7f20af1ff700 1 -- 172.21.15.27:0/1713754583 >> 172.21.15.27:0/1713754583 conn(0x7f20a80680a0 msgr2=0x7f20a80c80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.346+0000 7f20af1ff700 1 -- 172.21.15.27:0/1713754583 shutdown_connections 2024-01-11T09:47:42.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.346+0000 7f20af1ff700 1 -- 172.21.15.27:0/1713754583 wait complete. 2024-01-11T09:47:42.356 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-01-11T09:47:42.356 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2024-01-11T09:47:42.357 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-01-11T09:47:42.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.568+0000 7f77901ff700 1 Processor -- start 2024-01-11T09:47:42.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.568+0000 7f77901ff700 1 -- start start 2024-01-11T09:47:42.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.569+0000 7f77901ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77880ccd70 0x7f77880cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:42.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.569+0000 7f77901ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f778815ae70 0x7f778815b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:42.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.569+0000 7f77901ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77880cf850 con 0x7f778815a1b0 2024-01-11T09:47:42.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.569+0000 7f77901ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f778815ac90 con 0x7f778815ae70 2024-01-11T09:47:42.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.569+0000 7f77901ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77880cec40 con 0x7f77880ccd70 2024-01-11T09:47:42.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.569+0000 7f778df9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f778815a1b0 0x7f778815a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34546/0 (socket says 172.21.15.27:34546) 2024-01-11T09:47:42.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.569+0000 7f778df9b700 1 -- 172.21.15.27:0/1434899643 learned_addr learned my addr 172.21.15.27:0/1434899643 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:42.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.570+0000 7f778d79a700 1 --2- 172.21.15.27:0/1434899643 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77880ccd70 0x7f77880cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:42.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.570+0000 7f778e79c700 1 --2- 172.21.15.27:0/1434899643 >> v2:172.21.15.77:3300/0 conn(0x7f778815ae70 0x7f778815b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:42.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.570+0000 7f778cf99700 1 -- 172.21.15.27:0/1434899643 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161921217 0 0) 0x7f77880cf850 con 0x7f778815a1b0 2024-01-11T09:47:42.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.570+0000 7f778cf99700 1 -- 172.21.15.27:0/1434899643 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7774003570 con 0x7f778815a1b0 2024-01-11T09:47:42.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.570+0000 7f778cf99700 1 -- 172.21.15.27:0/1434899643 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 357574527 0 0) 0x7f7774003570 con 0x7f778815a1b0 2024-01-11T09:47:42.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.570+0000 7f778cf99700 1 -- 172.21.15.27:0/1434899643 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77880ccd70 msgr2=0x7f77880cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:42.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.570+0000 7f778cf99700 1 --2- 172.21.15.27:0/1434899643 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77880ccd70 0x7f77880cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.570+0000 7f778cf99700 1 -- 172.21.15.27:0/1434899643 >> v2:172.21.15.77:3300/0 conn(0x7f778815ae70 msgr2=0x7f778815b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:42.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.570+0000 7f778cf99700 1 --2- 172.21.15.27:0/1434899643 >> v2:172.21.15.77:3300/0 conn(0x7f778815ae70 0x7f778815b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.570+0000 7f778cf99700 1 -- 172.21.15.27:0/1434899643 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77880cef20 con 0x7f778815a1b0 2024-01-11T09:47:42.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.570+0000 7f778cf99700 1 -- 172.21.15.27:0/1434899643 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f7784001470 con 0x7f778815a1b0 2024-01-11T09:47:42.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.570+0000 7f778cf99700 1 -- 172.21.15.27:0/1434899643 >> v1:172.21.15.27:6789/0 conn(0x7f778815a1b0 legacy=0x7f778815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:42.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.570+0000 7f778cf99700 1 --2- 172.21.15.27:0/1434899643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7774003ae0 0x7f7774003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:42.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.570+0000 7f778cf99700 1 --2- 172.21.15.27:0/1434899643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7774004810 0x7f7774004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:42.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.570+0000 7f778cf99700 1 --2- 172.21.15.27:0/1434899643 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77880ccd70 0x7f77740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:42.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.570+0000 7f778cf99700 1 -- 172.21.15.27:0/1434899643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778815ac90 con 0x7f7774003ae0 2024-01-11T09:47:42.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.570+0000 7f778cf99700 1 -- 172.21.15.27:0/1434899643 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77880cec40 con 0x7f7774004810 2024-01-11T09:47:42.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.570+0000 7f778cf99700 1 -- 172.21.15.27:0/1434899643 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f777400bb10 con 0x7f77880ccd70 2024-01-11T09:47:42.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.571+0000 7f778df9b700 1 --2- 172.21.15.27:0/1434899643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7774003ae0 0x7f7774003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:42.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.571+0000 7f778d79a700 1 --2- 172.21.15.27:0/1434899643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7774004810 0x7f7774004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:42.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.571+0000 7f778e79c700 1 --2- 172.21.15.27:0/1434899643 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77880ccd70 0x7f77740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:42.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.571+0000 7f778df9b700 1 -- 172.21.15.27:0/1434899643 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77880ccd70 msgr2=0x7f77740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:42.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.571+0000 7f778df9b700 1 --2- 172.21.15.27:0/1434899643 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77880ccd70 0x7f77740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.571+0000 7f778df9b700 1 -- 172.21.15.27:0/1434899643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7774004810 msgr2=0x7f7774004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:42.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.571+0000 7f778df9b700 1 --2- 172.21.15.27:0/1434899643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7774004810 0x7f7774004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.571+0000 7f778df9b700 1 -- 172.21.15.27:0/1434899643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7774003570 con 0x7f7774003ae0 2024-01-11T09:47:42.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.571+0000 7f778d79a700 1 --2- 172.21.15.27:0/1434899643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7774004810 0x7f7774004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:42.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.571+0000 7f778df9b700 1 --2- 172.21.15.27:0/1434899643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7774003ae0 0x7f7774003ef0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f778400c2d0 tx=0x7f7784001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb81c68e3c553d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:42.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.572+0000 7f778cf99700 1 -- 172.21.15.27:0/1434899643 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778400cba0 con 0x7f7774003ae0 2024-01-11T09:47:42.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.572+0000 7f778cf99700 1 -- 172.21.15.27:0/1434899643 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778400cd00 con 0x7f7774003ae0 2024-01-11T09:47:42.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.572+0000 7f778cf99700 1 -- 172.21.15.27:0/1434899643 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7784005b50 con 0x7f7774003ae0 2024-01-11T09:47:42.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.572+0000 7f77901ff700 1 -- 172.21.15.27:0/1434899643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7774003ae0 msgr2=0x7f7774003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:42.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.572+0000 7f77901ff700 1 --2- 172.21.15.27:0/1434899643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7774003ae0 0x7f7774003ef0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f778400c2d0 tx=0x7f7784001dc0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.572+0000 7f778e79c700 1 -- 172.21.15.27:0/1434899643 reap_dead start 2024-01-11T09:47:42.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.572+0000 7f77901ff700 1 -- 172.21.15.27:0/1434899643 shutdown_connections 2024-01-11T09:47:42.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.572+0000 7f77901ff700 1 -- 172.21.15.27:0/1434899643 >> 172.21.15.27:0/1434899643 conn(0x7f778806ed40 msgr2=0x7f778806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:42.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.572+0000 7f77901ff700 1 -- 172.21.15.27:0/1434899643 shutdown_connections 2024-01-11T09:47:42.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.572+0000 7f77901ff700 1 -- 172.21.15.27:0/1434899643 wait complete. 2024-01-11T09:47:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.574+0000 7f77901ff700 1 Processor -- start 2024-01-11T09:47:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.574+0000 7f77901ff700 1 -- start start 2024-01-11T09:47:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.575+0000 7f77901ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7774003ae0 0x7f77880c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.575+0000 7f77901ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77880c9930 0x7f7788145f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.575+0000 7f77901ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77880b2290 0x7f77880b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.575+0000 7f77901ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77880c8930 con 0x7f77880c9930 2024-01-11T09:47:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.575+0000 7f77901ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77880b2c00 con 0x7f7774003ae0 2024-01-11T09:47:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.575+0000 7f77901ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77880c9da0 con 0x7f77880b2290 2024-01-11T09:47:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.575+0000 7f778d79a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77880c9930 0x7f7788145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.575+0000 7f778d79a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77880c9930 0x7f7788145f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36962/0 (socket says 172.21.15.27:36962) 2024-01-11T09:47:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.575+0000 7f778d79a700 1 -- 172.21.15.27:0/3816756832 learned_addr learned my addr 172.21.15.27:0/3816756832 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.575+0000 7f778e79c700 1 --2- 172.21.15.27:0/3816756832 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77880b2290 0x7f77880b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.575+0000 7f778df9b700 1 --2- 172.21.15.27:0/3816756832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7774003ae0 0x7f77880c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.576+0000 7f778d79a700 1 -- 172.21.15.27:0/3816756832 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77880b2290 msgr2=0x7f77880b26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:42.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.576+0000 7f778d79a700 1 --2- 172.21.15.27:0/3816756832 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77880b2290 0x7f77880b26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.576+0000 7f778d79a700 1 -- 172.21.15.27:0/3816756832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7774003ae0 msgr2=0x7f77880c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:42.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.576+0000 7f778d79a700 1 --2- 172.21.15.27:0/3816756832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7774003ae0 0x7f77880c7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.576+0000 7f778d79a700 1 -- 172.21.15.27:0/3816756832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77880c7d20 con 0x7f77880c9930 2024-01-11T09:47:42.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.576+0000 7f778d79a700 1 --2- 172.21.15.27:0/3816756832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77880c9930 0x7f7788145f30 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f7778011d70 tx=0x7f77780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:42.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.576+0000 7f777dffb700 1 -- 172.21.15.27:0/3816756832 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778011ed0 con 0x7f77880c9930 2024-01-11T09:47:42.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.576+0000 7f777dffb700 1 -- 172.21.15.27:0/3816756832 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777800fbf0 con 0x7f77880c9930 2024-01-11T09:47:42.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.576+0000 7f77901ff700 1 -- 172.21.15.27:0/3816756832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77880c8000 con 0x7f77880c9930 2024-01-11T09:47:42.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.576+0000 7f777dffb700 1 -- 172.21.15.27:0/3816756832 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777801d480 con 0x7f77880c9930 2024-01-11T09:47:42.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.576+0000 7f77901ff700 1 -- 172.21.15.27:0/3816756832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77880c85c0 con 0x7f77880c9930 2024-01-11T09:47:42.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.577+0000 7f77901ff700 1 -- 172.21.15.27:0/3816756832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f778804c3c0 con 0x7f77880c9930 2024-01-11T09:47:42.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.578+0000 7f777dffb700 1 -- 172.21.15.27:0/3816756832 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 149) v1 ==== 84862+0+0 (secure 0 0 0) 0x7f777801d870 con 0x7f77880c9930 2024-01-11T09:47:42.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.579+0000 7f777dffb700 1 --2- 172.21.15.27:0/3816756832 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f776c0670a0 0x7f776c069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:42.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.579+0000 7f777dffb700 1 -- 172.21.15.27:0/3816756832 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6247+0+0 (secure 0 0 0) 0x7f7778018070 con 0x7f77880c9930 2024-01-11T09:47:42.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.580+0000 7f778df9b700 1 --2- 172.21.15.27:0/3816756832 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f776c0670a0 0x7f776c069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:42.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.580+0000 7f778df9b700 1 --2- 172.21.15.27:0/3816756832 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f776c0670a0 0x7f776c069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7784015040 tx=0x7f7784012000 comp rx=0 tx=0).ready entity=mgr.9995 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:42.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.583+0000 7f777dffb700 1 -- 172.21.15.27:0/3816756832 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f77780561f0 con 0x7f77880c9930 2024-01-11T09:47:42.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.812+0000 7f77901ff700 1 -- 172.21.15.27:0/3816756832 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7788068940 con 0x7f776c0670a0 2024-01-11T09:47:42.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.874+0000 7f777dffb700 1 -- 172.21.15.27:0/3816756832 <== mgr.9995 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f7788068940 con 0x7f776c0670a0 2024-01-11T09:47:42.865 INFO:teuthology.orchestra.run.smithi027.stdout:Schedule removed for path snap_test_dir1 2024-01-11T09:47:42.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.876+0000 7f77901ff700 1 -- 172.21.15.27:0/3816756832 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f776c0670a0 msgr2=0x7f776c069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:42.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.876+0000 7f77901ff700 1 --2- 172.21.15.27:0/3816756832 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f776c0670a0 0x7f776c069550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7784015040 tx=0x7f7784012000 comp rx=0 tx=0).stop 2024-01-11T09:47:42.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.876+0000 7f77901ff700 1 -- 172.21.15.27:0/3816756832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77880c9930 msgr2=0x7f7788145f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:42.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.876+0000 7f77901ff700 1 --2- 172.21.15.27:0/3816756832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77880c9930 0x7f7788145f30 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f7778011d70 tx=0x7f77780091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.876+0000 7f77901ff700 1 -- 172.21.15.27:0/3816756832 shutdown_connections 2024-01-11T09:47:42.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.876+0000 7f77901ff700 1 --2- 172.21.15.27:0/3816756832 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77880b2290 0x7f77880b26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.876+0000 7f77901ff700 1 --2- 172.21.15.27:0/3816756832 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f776c0670a0 0x7f776c069550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.876+0000 7f77901ff700 1 --2- 172.21.15.27:0/3816756832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7774003ae0 0x7f77880c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.876+0000 7f77901ff700 1 --2- 172.21.15.27:0/3816756832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77880c9930 0x7f7788145f30 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:42.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.877+0000 7f77901ff700 1 -- 172.21.15.27:0/3816756832 >> 172.21.15.27:0/3816756832 conn(0x7f778806ed40 msgr2=0x7f77880c5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:42.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.877+0000 7f77901ff700 1 -- 172.21.15.27:0/3816756832 shutdown_connections 2024-01-11T09:47:42.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:42.877+0000 7f77901ff700 1 -- 172.21.15.27:0/3816756832 wait complete. 2024-01-11T09:47:42.886 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-01-11T09:47:42.886 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:47:42.886 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-01-11T09:47:42.936 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:47:42.936 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-01-11T09:47:42.986 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-01-11T09:47:43.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.197+0000 7ff5a45ff700 1 Processor -- start 2024-01-11T09:47:43.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.197+0000 7ff5a45ff700 1 -- start start 2024-01-11T09:47:43.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.199+0000 7ff5a45ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff59c155c50 0x7ff59c0aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:43.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.199+0000 7ff5a45ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff59c0c74f0 0x7ff59c069910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:43.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.199+0000 7ff5a45ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c06e480 con 0x7ff59c0694c0 2024-01-11T09:47:43.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.199+0000 7ff5a45ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff59c0cb5a0 con 0x7ff59c0c74f0 2024-01-11T09:47:43.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.199+0000 7ff5a45ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c06df00 con 0x7ff59c155c50 2024-01-11T09:47:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.199+0000 7ff5a239b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff59c155c50 0x7ff59c0aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.199+0000 7ff5a239b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff59c155c50 0x7ff59c0aef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:47252/0 (socket says 172.21.15.27:47252) 2024-01-11T09:47:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.199+0000 7ff5a239b700 1 -- 172.21.15.27:0/1641392154 learned_addr learned my addr 172.21.15.27:0/1641392154 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.199+0000 7ff5a1b9a700 1 --1- 172.21.15.27:0/1641392154 >> v1:172.21.15.77:6789/0 conn(0x7ff59c0694c0 0x7ff59c0c6de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:49532/0 (socket says 172.21.15.27:49532) 2024-01-11T09:47:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.199+0000 7ff5a2b9c700 1 --2- 172.21.15.27:0/1641392154 >> v2:172.21.15.77:3300/0 conn(0x7ff59c0c74f0 0x7ff59c069910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.200+0000 7ff5a239b700 1 -- 172.21.15.27:0/1641392154 >> v2:172.21.15.77:3300/0 conn(0x7ff59c0c74f0 msgr2=0x7ff59c069910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.200+0000 7ff5a239b700 1 --2- 172.21.15.27:0/1641392154 >> v2:172.21.15.77:3300/0 conn(0x7ff59c0c74f0 0x7ff59c069910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.200+0000 7ff5a239b700 1 -- 172.21.15.27:0/1641392154 >> v1:172.21.15.77:6789/0 conn(0x7ff59c0694c0 legacy=0x7ff59c0c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.200+0000 7ff5a1399700 1 -- 172.21.15.27:0/1641392154 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445471424 0 0) 0x7ff59c06e480 con 0x7ff59c0694c0 2024-01-11T09:47:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.200+0000 7ff5a239b700 1 -- 172.21.15.27:0/1641392154 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c069e50 con 0x7ff59c155c50 2024-01-11T09:47:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.200+0000 7ff5a239b700 1 --2- 172.21.15.27:0/1641392154 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff59c155c50 0x7ff59c0aef80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff59800bc30 tx=0x7ff59800b4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=639ebeca87867477 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.200+0000 7ff5a1399700 1 -- 172.21.15.27:0/1641392154 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59800dc30 con 0x7ff59c155c50 2024-01-11T09:47:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.200+0000 7ff5a1399700 1 -- 172.21.15.27:0/1641392154 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59800dd90 con 0x7ff59c155c50 2024-01-11T09:47:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.201+0000 7ff5a1399700 1 -- 172.21.15.27:0/1641392154 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff598005c30 con 0x7ff59c155c50 2024-01-11T09:47:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.201+0000 7ff5a45ff700 1 -- 172.21.15.27:0/1641392154 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff59c155c50 msgr2=0x7ff59c0aef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.201+0000 7ff5a45ff700 1 --2- 172.21.15.27:0/1641392154 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff59c155c50 0x7ff59c0aef80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff59800bc30 tx=0x7ff59800b4c0 comp rx=0 tx=0).stop 2024-01-11T09:47:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.201+0000 7ff5a45ff700 1 -- 172.21.15.27:0/1641392154 shutdown_connections 2024-01-11T09:47:43.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.201+0000 7ff5a45ff700 1 --2- 172.21.15.27:0/1641392154 >> v2:172.21.15.77:3300/0 conn(0x7ff59c0c74f0 0x7ff59c069910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:43.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.201+0000 7ff5a45ff700 1 --2- 172.21.15.27:0/1641392154 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff59c155c50 0x7ff59c0aef80 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:43.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.201+0000 7ff5a45ff700 1 -- 172.21.15.27:0/1641392154 >> 172.21.15.27:0/1641392154 conn(0x7ff59c0680a0 msgr2=0x7ff59c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:43.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.201+0000 7ff5a45ff700 1 -- 172.21.15.27:0/1641392154 shutdown_connections 2024-01-11T09:47:43.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.201+0000 7ff5a45ff700 1 -- 172.21.15.27:0/1641392154 wait complete. 2024-01-11T09:47:43.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.203+0000 7ff5a45ff700 1 Processor -- start 2024-01-11T09:47:43.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.203+0000 7ff5a45ff700 1 -- start start 2024-01-11T09:47:43.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.204+0000 7ff5a45ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff59c0694c0 0x7ff59c0c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:43.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.204+0000 7ff5a45ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff59c0c74f0 0x7ff59c157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:43.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.204+0000 7ff5a45ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff59c155c50 0x7ff59c14abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:43.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.204+0000 7ff5a45ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c1694e0 con 0x7ff59c0c74f0 2024-01-11T09:47:43.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.204+0000 7ff5a45ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c16a020 con 0x7ff59c0694c0 2024-01-11T09:47:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.204+0000 7ff5a45ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c157bb0 con 0x7ff59c155c50 2024-01-11T09:47:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.204+0000 7ff5a1b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff59c0c74f0 0x7ff59c157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.204+0000 7ff5a1b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff59c0c74f0 0x7ff59c157220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36970/0 (socket says 172.21.15.27:36970) 2024-01-11T09:47:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.204+0000 7ff5a1b9a700 1 -- 172.21.15.27:0/1178325168 learned_addr learned my addr 172.21.15.27:0/1178325168 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.204+0000 7ff5a239b700 1 --2- 172.21.15.27:0/1178325168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff59c0694c0 0x7ff59c0c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.204+0000 7ff5a2b9c700 1 --2- 172.21.15.27:0/1178325168 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff59c155c50 0x7ff59c14abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.204+0000 7ff5a1b9a700 1 -- 172.21.15.27:0/1178325168 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff59c155c50 msgr2=0x7ff59c14abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.204+0000 7ff5a1b9a700 1 --2- 172.21.15.27:0/1178325168 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff59c155c50 0x7ff59c14abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.204+0000 7ff5a1b9a700 1 -- 172.21.15.27:0/1178325168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff59c0694c0 msgr2=0x7ff59c0c5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.204+0000 7ff5a1b9a700 1 --2- 172.21.15.27:0/1178325168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff59c0694c0 0x7ff59c0c5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.204+0000 7ff5a1b9a700 1 -- 172.21.15.27:0/1178325168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59800b720 con 0x7ff59c0c74f0 2024-01-11T09:47:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.204+0000 7ff5a239b700 1 --2- 172.21.15.27:0/1178325168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff59c0694c0 0x7ff59c0c5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.205+0000 7ff5a1b9a700 1 --2- 172.21.15.27:0/1178325168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff59c0c74f0 0x7ff59c157220 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7ff58c0011e0 tx=0x7ff58c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.205+0000 7ff5927fc700 1 -- 172.21.15.27:0/1178325168 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c007300 con 0x7ff59c0c74f0 2024-01-11T09:47:43.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.205+0000 7ff5927fc700 1 -- 172.21.15.27:0/1178325168 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58c007460 con 0x7ff59c0c74f0 2024-01-11T09:47:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.205+0000 7ff5a45ff700 1 -- 172.21.15.27:0/1178325168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff59c157760 con 0x7ff59c0c74f0 2024-01-11T09:47:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.205+0000 7ff5927fc700 1 -- 172.21.15.27:0/1178325168 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c004830 con 0x7ff59c0c74f0 2024-01-11T09:47:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.205+0000 7ff5a45ff700 1 -- 172.21.15.27:0/1178325168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff59c1699d0 con 0x7ff59c0c74f0 2024-01-11T09:47:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.206+0000 7ff5a45ff700 1 -- 172.21.15.27:0/1178325168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff59c16a190 con 0x7ff59c0c74f0 2024-01-11T09:47:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.207+0000 7ff5927fc700 1 -- 172.21.15.27:0/1178325168 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 149) v1 ==== 84862+0+0 (secure 0 0 0) 0x7ff58c004be0 con 0x7ff59c0c74f0 2024-01-11T09:47:43.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.208+0000 7ff5927fc700 1 --2- 172.21.15.27:0/1178325168 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff5800670a0 0x7ff580069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:43.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.208+0000 7ff5927fc700 1 -- 172.21.15.27:0/1178325168 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6247+0+0 (secure 0 0 0) 0x7ff58c083f00 con 0x7ff59c0c74f0 2024-01-11T09:47:43.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.209+0000 7ff5a239b700 1 --2- 172.21.15.27:0/1178325168 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff5800670a0 0x7ff580069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:43.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.209+0000 7ff5a239b700 1 --2- 172.21.15.27:0/1178325168 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff5800670a0 0x7ff580069550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff59800b500 tx=0x7ff59801a000 comp rx=0 tx=0).ready entity=mgr.9995 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:43.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.212+0000 7ff5927fc700 1 -- 172.21.15.27:0/1178325168 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff58c050e20 con 0x7ff59c0c74f0 2024-01-11T09:47:43.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:43.476+0000 7ff5a45ff700 1 -- 172.21.15.27:0/1178325168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7ff59c04c3c0 con 0x7ff59c0c74f0 2024-01-11T09:47:44.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.335+0000 7ff5927fc700 1 -- 172.21.15.27:0/1178325168 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7ff58c0511f0 con 0x7ff59c0c74f0 2024-01-11T09:47:44.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.337+0000 7ff5927fc700 1 -- 172.21.15.27:0/1178325168 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v150) v1 ==== 93+0+0 (secure 0 0 0) 0x7ff58c050e20 con 0x7ff59c0c74f0 2024-01-11T09:47:44.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.341+0000 7ff5a45ff700 1 -- 172.21.15.27:0/1178325168 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff5800670a0 msgr2=0x7ff580069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.341+0000 7ff5a45ff700 1 --2- 172.21.15.27:0/1178325168 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff5800670a0 0x7ff580069550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff59800b500 tx=0x7ff59801a000 comp rx=0 tx=0).stop 2024-01-11T09:47:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.341+0000 7ff5a45ff700 1 -- 172.21.15.27:0/1178325168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff59c0c74f0 msgr2=0x7ff59c157220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.341+0000 7ff5a45ff700 1 --2- 172.21.15.27:0/1178325168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff59c0c74f0 0x7ff59c157220 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7ff58c0011e0 tx=0x7ff58c007e10 comp rx=0 tx=0).stop 2024-01-11T09:47:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.341+0000 7ff5a45ff700 1 -- 172.21.15.27:0/1178325168 shutdown_connections 2024-01-11T09:47:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.341+0000 7ff5a45ff700 1 --2- 172.21.15.27:0/1178325168 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff59c155c50 0x7ff59c14abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.341+0000 7ff5a45ff700 1 --2- 172.21.15.27:0/1178325168 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff5800670a0 0x7ff580069550 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.341+0000 7ff5a45ff700 1 --2- 172.21.15.27:0/1178325168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff59c0694c0 0x7ff59c0c5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.341+0000 7ff5a45ff700 1 --2- 172.21.15.27:0/1178325168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff59c0c74f0 0x7ff59c157220 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.341+0000 7ff5a45ff700 1 -- 172.21.15.27:0/1178325168 >> 172.21.15.27:0/1178325168 conn(0x7ff59c0680a0 msgr2=0x7ff59c06e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.341+0000 7ff5a45ff700 1 -- 172.21.15.27:0/1178325168 shutdown_connections 2024-01-11T09:47:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.341+0000 7ff5a45ff700 1 -- 172.21.15.27:0/1178325168 wait complete. 2024-01-11T09:47:44.354 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-11T09:47:44.355 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:47:44.453 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-11T09:47:44.549 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-11T09:47:44.549 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-11T09:47:44.549 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-01-11T09:47:44.549 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-01-11T09:47:44.555 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:44.564+0000 7fef03e9e1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:47:44.556 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:44.561+0000 7f40ad6511c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T09:47:44.634 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T09:47:44.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:44.634 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-11T09:47:44.703 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:47:44.734 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:44.743+0000 7fef03e9e1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:47:44.759 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:44.767+0000 7f40ad6511c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T09:47:44.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.939+0000 7f7b2adff700 1 Processor -- start 2024-01-11T09:47:44.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.939+0000 7f7b2adff700 1 -- start start 2024-01-11T09:47:44.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.940+0000 7f7b2adff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7b240ca7b0 0x7f7b240cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:44.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.940+0000 7f7b2adff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7b24159e80 0x7f7b2415dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:44.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.940+0000 7f7b2adff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2415d640 con 0x7f7b240ca3e0 2024-01-11T09:47:44.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.940+0000 7f7b2adff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b2415ae70 con 0x7f7b24159e80 2024-01-11T09:47:44.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.940+0000 7f7b2adff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b240cc4f0 con 0x7f7b240ca7b0 2024-01-11T09:47:44.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.941+0000 7f7b2939c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7b24159e80 0x7f7b2415dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:44.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.941+0000 7f7b23fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7b240ca7b0 0x7f7b240cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:44.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.941+0000 7f7b2939c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7b24159e80 0x7f7b2415dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60482/0 (socket says 172.21.15.27:60482) 2024-01-11T09:47:44.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.941+0000 7f7b23fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7b240ca7b0 0x7f7b240cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:49002/0 (socket says 172.21.15.27:49002) 2024-01-11T09:47:44.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.941+0000 7f7b2939c700 1 -- 172.21.15.27:0/3606916472 learned_addr learned my addr 172.21.15.27:0/3606916472 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:44.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.941+0000 7f7b2939c700 1 -- 172.21.15.27:0/3606916472 >> v2:172.21.15.77:3300/0 conn(0x7f7b240ca7b0 msgr2=0x7f7b240cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:44.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.941+0000 7f7b2939c700 1 --2- 172.21.15.27:0/3606916472 >> v2:172.21.15.77:3300/0 conn(0x7f7b240ca7b0 0x7f7b240cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:44.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.941+0000 7f7b2939c700 1 -- 172.21.15.27:0/3606916472 >> v1:172.21.15.77:6789/0 conn(0x7f7b240ca3e0 legacy=0x7f7b240c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:44.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.941+0000 7f7b2939c700 1 -- 172.21.15.27:0/3606916472 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2415a5d0 con 0x7f7b24159e80 2024-01-11T09:47:44.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.941+0000 7f7b23fff700 1 --2- 172.21.15.27:0/3606916472 >> v2:172.21.15.77:3300/0 conn(0x7f7b240ca7b0 0x7f7b240cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:44.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.942+0000 7f7b2939c700 1 --2- 172.21.15.27:0/3606916472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b24159e80 0x7f7b2415dad0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c009f50 tx=0x7f7b1c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0fc50dcc6544036 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:44.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.942+0000 7f7b237fe700 1 -- 172.21.15.27:0/3606916472 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c00e070 con 0x7f7b24159e80 2024-01-11T09:47:44.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.942+0000 7f7b237fe700 1 -- 172.21.15.27:0/3606916472 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1c00b980 con 0x7f7b24159e80 2024-01-11T09:47:44.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.942+0000 7f7b237fe700 1 -- 172.21.15.27:0/3606916472 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c0092e0 con 0x7f7b24159e80 2024-01-11T09:47:44.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.943+0000 7f7b2adff700 1 -- 172.21.15.27:0/3606916472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b24159e80 msgr2=0x7f7b2415dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:44.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.943+0000 7f7b2adff700 1 --2- 172.21.15.27:0/3606916472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b24159e80 0x7f7b2415dad0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c009f50 tx=0x7f7b1c009840 comp rx=0 tx=0).stop 2024-01-11T09:47:44.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.943+0000 7f7b2adff700 1 -- 172.21.15.27:0/3606916472 shutdown_connections 2024-01-11T09:47:44.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.943+0000 7f7b2adff700 1 --2- 172.21.15.27:0/3606916472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b24159e80 0x7f7b2415dad0 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:44.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.943+0000 7f7b2adff700 1 --2- 172.21.15.27:0/3606916472 >> v2:172.21.15.77:3300/0 conn(0x7f7b240ca7b0 0x7f7b240cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:44.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.943+0000 7f7b2adff700 1 -- 172.21.15.27:0/3606916472 >> 172.21.15.27:0/3606916472 conn(0x7f7b240ce3e0 msgr2=0x7f7b240ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:44.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.943+0000 7f7b2adff700 1 -- 172.21.15.27:0/3606916472 shutdown_connections 2024-01-11T09:47:44.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.943+0000 7f7b2adff700 1 -- 172.21.15.27:0/3606916472 wait complete. 2024-01-11T09:47:44.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.944+0000 7f7b2adff700 1 Processor -- start 2024-01-11T09:47:44.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.945+0000 7f7b2adff700 1 -- start start 2024-01-11T09:47:44.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.945+0000 7f7b2adff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b240ca3e0 0x7f7b240c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:44.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.945+0000 7f7b2adff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b240ca7b0 0x7f7b2413ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:44.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.946+0000 7f7b2adff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b24159e80 0x7f7b2416d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:44.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.946+0000 7f7b2adff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b240c7a00 con 0x7f7b240ca3e0 2024-01-11T09:47:44.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.946+0000 7f7b2adff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b240c6e30 con 0x7f7b240ca7b0 2024-01-11T09:47:44.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.946+0000 7f7b2adff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b240c70d0 con 0x7f7b24159e80 2024-01-11T09:47:44.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.946+0000 7f7b28b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b240ca3e0 0x7f7b240c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:44.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.946+0000 7f7b28b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b240ca3e0 0x7f7b240c63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60484/0 (socket says 172.21.15.27:60484) 2024-01-11T09:47:44.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.946+0000 7f7b28b9b700 1 -- 172.21.15.27:0/2506724147 learned_addr learned my addr 172.21.15.27:0/2506724147 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:44.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.946+0000 7f7b2939c700 1 --2- 172.21.15.27:0/2506724147 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b24159e80 0x7f7b2416d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:44.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.946+0000 7f7b23fff700 1 --2- 172.21.15.27:0/2506724147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b240ca7b0 0x7f7b2413ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:44.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.946+0000 7f7b28b9b700 1 -- 172.21.15.27:0/2506724147 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b24159e80 msgr2=0x7f7b2416d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:44.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.946+0000 7f7b28b9b700 1 --2- 172.21.15.27:0/2506724147 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b24159e80 0x7f7b2416d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:44.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.946+0000 7f7b28b9b700 1 -- 172.21.15.27:0/2506724147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b240ca7b0 msgr2=0x7f7b2413ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:44.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.946+0000 7f7b28b9b700 1 --2- 172.21.15.27:0/2506724147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b240ca7b0 0x7f7b2413ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:44.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.946+0000 7f7b28b9b700 1 -- 172.21.15.27:0/2506724147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b10009420 con 0x7f7b240ca3e0 2024-01-11T09:47:44.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.947+0000 7f7b28b9b700 1 --2- 172.21.15.27:0/2506724147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b240ca3e0 0x7f7b240c63f0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f7b1800c590 tx=0x7f7b1800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:44.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.947+0000 7f7b20ff9700 1 -- 172.21.15.27:0/2506724147 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b18011590 con 0x7f7b240ca3e0 2024-01-11T09:47:44.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.947+0000 7f7b20ff9700 1 -- 172.21.15.27:0/2506724147 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b18011710 con 0x7f7b240ca3e0 2024-01-11T09:47:44.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.947+0000 7f7b20ff9700 1 -- 172.21.15.27:0/2506724147 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b18011ad0 con 0x7f7b240ca3e0 2024-01-11T09:47:44.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.947+0000 7f7b2adff700 1 -- 172.21.15.27:0/2506724147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b1c009ae0 con 0x7f7b240ca3e0 2024-01-11T09:47:44.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.947+0000 7f7b2adff700 1 -- 172.21.15.27:0/2506724147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b240c7690 con 0x7f7b240ca3e0 2024-01-11T09:47:44.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.948+0000 7f7b2adff700 1 -- 172.21.15.27:0/2506724147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b240718d0 con 0x7f7b240ca3e0 2024-01-11T09:47:44.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.949+0000 7f7b20ff9700 1 -- 172.21.15.27:0/2506724147 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f7b1801b3b0 con 0x7f7b240ca3e0 2024-01-11T09:47:44.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.950+0000 7f7b20ff9700 1 --2- 172.21.15.27:0/2506724147 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7b080670e0 0x7f7b08069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:44.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.950+0000 7f7b20ff9700 1 -- 172.21.15.27:0/2506724147 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6247+0+0 (secure 0 0 0) 0x7f7b18083ee0 con 0x7f7b240ca3e0 2024-01-11T09:47:44.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.950+0000 7f7b23fff700 1 -- 172.21.15.27:0/2506724147 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7b080670e0 msgr2=0x7f7b08069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:44.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.950+0000 7f7b23fff700 1 --2- 172.21.15.27:0/2506724147 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7b080670e0 0x7f7b08069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:44.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:44.954+0000 7f7b20ff9700 1 -- 172.21.15.27:0/2506724147 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7b18050cb0 con 0x7f7b240ca3e0 2024-01-11T09:47:45.048 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:45.057+0000 7fef03e9e1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:47:45.102 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:45.111+0000 7f40ad6511c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T09:47:45.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.151+0000 7f7b23fff700 1 -- 172.21.15.27:0/2506724147 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7b080670e0 msgr2=0x7f7b08069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:45.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.151+0000 7f7b23fff700 1 --2- 172.21.15.27:0/2506724147 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7b080670e0 0x7f7b08069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:45.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.213+0000 7f7b2adff700 1 -- 172.21.15.27:0/2506724147 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b2415a8b0 con 0x7f7b240ca3e0 2024-01-11T09:47:45.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.213+0000 7f7b20ff9700 1 -- 172.21.15.27:0/2506724147 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+4612 (secure 0 0 0) 0x7f7b18017070 con 0x7f7b240ca3e0 2024-01-11T09:47:45.205 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:45.205 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":141,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9866,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1060058066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1060058066},{"type":"v1","addr":"172.21.15.77:6835","nonce":1060058066}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137},{"gid":11626,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/4169546841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":4169546841},{"type":"v1","addr":"172.21.15.77:6834","nonce":4169546841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137},{"gid":11632,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/2567971522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2567971522},{"type":"v1","addr":"172.21.15.27:6833","nonce":2567971522}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:47:24.474082+0000","modified":"2024-01-11T09:47:27.985712+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12063},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12063":{"gid":12063,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1065634442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1065634442},{"type":"v1","addr":"172.21.15.27:6835","nonce":1065634442}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2024-01-11T09:47:45.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.216+0000 7f7b2adff700 1 -- 172.21.15.27:0/2506724147 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7b080670e0 msgr2=0x7f7b08069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:45.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.216+0000 7f7b2adff700 1 --2- 172.21.15.27:0/2506724147 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7b080670e0 0x7f7b08069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.216+0000 7f7b2adff700 1 -- 172.21.15.27:0/2506724147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b240ca3e0 msgr2=0x7f7b240c63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:45.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.216+0000 7f7b2adff700 1 --2- 172.21.15.27:0/2506724147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b240ca3e0 0x7f7b240c63f0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f7b1800c590 tx=0x7f7b1800c8a0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.216+0000 7f7b2adff700 1 -- 172.21.15.27:0/2506724147 shutdown_connections 2024-01-11T09:47:45.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.216+0000 7f7b2adff700 1 --2- 172.21.15.27:0/2506724147 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7b24159e80 0x7f7b2416d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.216+0000 7f7b2adff700 1 --2- 172.21.15.27:0/2506724147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b240ca3e0 0x7f7b240c63f0 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.216+0000 7f7b2adff700 1 --2- 172.21.15.27:0/2506724147 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7b080670e0 0x7f7b08069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.216+0000 7f7b2adff700 1 --2- 172.21.15.27:0/2506724147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b240ca7b0 0x7f7b2413ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.216+0000 7f7b2adff700 1 -- 172.21.15.27:0/2506724147 >> 172.21.15.27:0/2506724147 conn(0x7f7b240ce3e0 msgr2=0x7f7b240ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:45.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.216+0000 7f7b2adff700 1 -- 172.21.15.27:0/2506724147 shutdown_connections 2024-01-11T09:47:45.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.216+0000 7f7b2adff700 1 -- 172.21.15.27:0/2506724147 wait complete. 2024-01-11T09:47:45.210 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 141 2024-01-11T09:47:45.226 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:47:45.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.437+0000 7f7dc2dff700 1 Processor -- start 2024-01-11T09:47:45.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.437+0000 7f7dc2dff700 1 -- start start 2024-01-11T09:47:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.439+0000 7f7dc2dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dbc0c8620 0x7f7dbc0af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.439+0000 7f7dc2dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7dbc069d40 0x7f7dbc0b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.439+0000 7f7dc2dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dbc157390 con 0x7f7dbc0c89f0 2024-01-11T09:47:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.439+0000 7f7dc2dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dbc0adcd0 con 0x7f7dbc069d40 2024-01-11T09:47:45.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.439+0000 7f7dc2dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc0afeb0 con 0x7f7dbc0c8620 2024-01-11T09:47:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.439+0000 7f7dbbfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7dbc069d40 0x7f7dbc0b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.439+0000 7f7dc139c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7dbc0c89f0 0x7f7dbc159ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37754/0 (socket says 172.21.15.27:37754) 2024-01-11T09:47:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.439+0000 7f7dc139c700 1 -- 172.21.15.27:0/878753996 learned_addr learned my addr 172.21.15.27:0/878753996 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.439+0000 7f7dc0b9b700 1 --2- 172.21.15.27:0/878753996 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dbc0c8620 0x7f7dbc0af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.440+0000 7f7dbbfff700 1 -- 172.21.15.27:0/878753996 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dbc0c8620 msgr2=0x7f7dbc0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.440+0000 7f7dbbfff700 1 --2- 172.21.15.27:0/878753996 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dbc0c8620 0x7f7dbc0af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.440+0000 7f7dbbfff700 1 -- 172.21.15.27:0/878753996 >> v1:172.21.15.27:6789/0 conn(0x7f7dbc0c89f0 legacy=0x7f7dbc159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.440+0000 7f7dbbfff700 1 -- 172.21.15.27:0/878753996 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dbc06a200 con 0x7f7dbc069d40 2024-01-11T09:47:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.440+0000 7f7dbb7fe700 1 -- 172.21.15.27:0/878753996 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448442259 0 0) 0x7f7dbc157390 con 0x7f7dbc0c89f0 2024-01-11T09:47:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.440+0000 7f7dbbfff700 1 --2- 172.21.15.27:0/878753996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dbc069d40 0x7f7dbc0b02d0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f7da80098e0 tx=0x7f7da80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6a19d6f7e8512c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.440+0000 7f7dbb7fe700 1 -- 172.21.15.27:0/878753996 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da800b640 con 0x7f7dbc069d40 2024-01-11T09:47:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.440+0000 7f7dbb7fe700 1 -- 172.21.15.27:0/878753996 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da800b7a0 con 0x7f7dbc069d40 2024-01-11T09:47:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.440+0000 7f7dbb7fe700 1 -- 172.21.15.27:0/878753996 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da800bb40 con 0x7f7dbc069d40 2024-01-11T09:47:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.441+0000 7f7dc2dff700 1 -- 172.21.15.27:0/878753996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dbc069d40 msgr2=0x7f7dbc0b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.441+0000 7f7dc2dff700 1 --2- 172.21.15.27:0/878753996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dbc069d40 0x7f7dbc0b02d0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f7da80098e0 tx=0x7f7da80091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.441+0000 7f7dc2dff700 1 -- 172.21.15.27:0/878753996 shutdown_connections 2024-01-11T09:47:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.441+0000 7f7dc2dff700 1 --2- 172.21.15.27:0/878753996 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dbc0c8620 0x7f7dbc0af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.441+0000 7f7dc2dff700 1 --2- 172.21.15.27:0/878753996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dbc069d40 0x7f7dbc0b02d0 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.441+0000 7f7dc2dff700 1 -- 172.21.15.27:0/878753996 >> 172.21.15.27:0/878753996 conn(0x7f7dbc0680e0 msgr2=0x7f7dbc069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.441+0000 7f7dc2dff700 1 -- 172.21.15.27:0/878753996 shutdown_connections 2024-01-11T09:47:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.441+0000 7f7dc2dff700 1 -- 172.21.15.27:0/878753996 wait complete. 2024-01-11T09:47:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.444+0000 7f7dc2dff700 1 Processor -- start 2024-01-11T09:47:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.444+0000 7f7dc2dff700 1 -- start start 2024-01-11T09:47:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.446+0000 7f7dc2dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dbc069d40 0x7f7dbc1560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.446+0000 7f7dc2dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dbc0c8620 0x7f7dbc144ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.446+0000 7f7dc2dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dbc0c89f0 0x7f7dbc0b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.446+0000 7f7dc2dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc155a90 con 0x7f7dbc0c8620 2024-01-11T09:47:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.446+0000 7f7dc2dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc156600 con 0x7f7dbc069d40 2024-01-11T09:47:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.446+0000 7f7dc2dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc15a240 con 0x7f7dbc0c89f0 2024-01-11T09:47:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.446+0000 7f7dbbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dbc0c8620 0x7f7dbc144ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.446+0000 7f7dc0b9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dbc069d40 0x7f7dbc1560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.446+0000 7f7dc0b9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dbc069d40 0x7f7dbc1560c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:49030/0 (socket says 172.21.15.27:49030) 2024-01-11T09:47:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.446+0000 7f7dc0b9b700 1 -- 172.21.15.27:0/1121303476 learned_addr learned my addr 172.21.15.27:0/1121303476 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.446+0000 7f7dc139c700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dbc0c89f0 0x7f7dbc0b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.446+0000 7f7dc0b9b700 1 -- 172.21.15.27:0/1121303476 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dbc0c89f0 msgr2=0x7f7dbc0b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.446+0000 7f7dc0b9b700 1 --2- 172.21.15.27:0/1121303476 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dbc0c89f0 0x7f7dbc0b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.446+0000 7f7dc0b9b700 1 -- 172.21.15.27:0/1121303476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dbc0c8620 msgr2=0x7f7dbc144ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.446+0000 7f7dc0b9b700 1 --2- 172.21.15.27:0/1121303476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dbc0c8620 0x7f7dbc144ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.446+0000 7f7dc0b9b700 1 -- 172.21.15.27:0/1121303476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da8009420 con 0x7f7dbc069d40 2024-01-11T09:47:45.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.447+0000 7f7dc0b9b700 1 --2- 172.21.15.27:0/1121303476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dbc069d40 0x7f7dbc1560c0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f7db000ecb0 tx=0x7f7db00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:45.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.447+0000 7f7db8ff9700 1 -- 172.21.15.27:0/1121303476 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db000c850 con 0x7f7dbc069d40 2024-01-11T09:47:45.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.447+0000 7f7db8ff9700 1 -- 172.21.15.27:0/1121303476 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db000c9b0 con 0x7f7dbc069d40 2024-01-11T09:47:45.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.447+0000 7f7dc2dff700 1 -- 172.21.15.27:0/1121303476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dbc1451e0 con 0x7f7dbc069d40 2024-01-11T09:47:45.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.447+0000 7f7db8ff9700 1 -- 172.21.15.27:0/1121303476 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db000cd50 con 0x7f7dbc069d40 2024-01-11T09:47:45.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.447+0000 7f7dc2dff700 1 -- 172.21.15.27:0/1121303476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dbc154cd0 con 0x7f7dbc069d40 2024-01-11T09:47:45.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.448+0000 7f7dc2dff700 1 -- 172.21.15.27:0/1121303476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dbc1558e0 con 0x7f7dbc069d40 2024-01-11T09:47:45.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.449+0000 7f7db8ff9700 1 -- 172.21.15.27:0/1121303476 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f7db0013680 con 0x7f7dbc069d40 2024-01-11T09:47:45.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.450+0000 7f7db8ff9700 1 --2- 172.21.15.27:0/1121303476 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7da00670e0 0x7f7da0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:45.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.450+0000 7f7db8ff9700 1 -- 172.21.15.27:0/1121303476 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6247+0+0 (secure 0 0 0) 0x7f7db00850f0 con 0x7f7dbc069d40 2024-01-11T09:47:45.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.450+0000 7f7dbbfff700 1 -- 172.21.15.27:0/1121303476 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7da00670e0 msgr2=0x7f7da0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:45.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.450+0000 7f7dbbfff700 1 --2- 172.21.15.27:0/1121303476 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7da00670e0 0x7f7da0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:45.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.454+0000 7f7db8ff9700 1 -- 172.21.15.27:0/1121303476 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7db0051e20 con 0x7f7dbc069d40 2024-01-11T09:47:45.569 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:45.578+0000 7fef03e9e1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:47:45.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.651+0000 7f7dbbfff700 1 -- 172.21.15.27:0/1121303476 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7da00670e0 msgr2=0x7f7da0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:45.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.651+0000 7f7dbbfff700 1 --2- 172.21.15.27:0/1121303476 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7da00670e0 0x7f7da0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:45.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.659+0000 7f7dc2dff700 1 -- 172.21.15.27:0/1121303476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dbc0718d0 con 0x7f7dbc069d40 2024-01-11T09:47:45.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.660+0000 7f7db8ff9700 1 -- 172.21.15.27:0/1121303476 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v257) v1 ==== 74+0+16107 (secure 0 0 0) 0x7f7db0055440 con 0x7f7dbc069d40 2024-01-11T09:47:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:45.652 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":257,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:47:40.221837+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:47:22.374454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-01-11T09:47:23.471379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/151121862":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/3962343616":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/710006972":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/2450284170":"2024-01-12T09:47:40.221791+0000","172.21.15.77:6837/59776":"2024-01-12T09:47:40.221791+0000","172.21.15.77:6836/59776":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/447393830":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/1295621369":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/1441504549":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/1838517028":"2024-01-12T09:47:40.221791+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:47:45.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.662+0000 7f7dc2dff700 1 -- 172.21.15.27:0/1121303476 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7da00670e0 msgr2=0x7f7da0069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:45.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.662+0000 7f7dc2dff700 1 --2- 172.21.15.27:0/1121303476 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7da00670e0 0x7f7da0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.662+0000 7f7dc2dff700 1 -- 172.21.15.27:0/1121303476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dbc069d40 msgr2=0x7f7dbc1560c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:45.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.662+0000 7f7dc2dff700 1 --2- 172.21.15.27:0/1121303476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dbc069d40 0x7f7dbc1560c0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f7db000ecb0 tx=0x7f7db00091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.662+0000 7f7dc2dff700 1 -- 172.21.15.27:0/1121303476 shutdown_connections 2024-01-11T09:47:45.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.663+0000 7f7dc2dff700 1 --2- 172.21.15.27:0/1121303476 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7dbc0c89f0 0x7f7dbc0b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.663+0000 7f7dc2dff700 1 --2- 172.21.15.27:0/1121303476 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7da00670e0 0x7f7da0069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.663+0000 7f7dc2dff700 1 --2- 172.21.15.27:0/1121303476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dbc069d40 0x7f7dbc1560c0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.663+0000 7f7dc2dff700 1 --2- 172.21.15.27:0/1121303476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7dbc0c8620 0x7f7dbc144ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.663+0000 7f7dc2dff700 1 -- 172.21.15.27:0/1121303476 >> 172.21.15.27:0/1121303476 conn(0x7f7dbc0680e0 msgr2=0x7f7dbc06de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:45.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.663+0000 7f7dc2dff700 1 -- 172.21.15.27:0/1121303476 shutdown_connections 2024-01-11T09:47:45.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.663+0000 7f7dc2dff700 1 -- 172.21.15.27:0/1121303476 wait complete. 2024-01-11T09:47:45.669 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:45.678+0000 7f40ad6511c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T09:47:45.674 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-11T08:10:19.576658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-11T09:47:22.374454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '255', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2024-01-11T09:47:23.471379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '256', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-11T09:47:45.675 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-11T09:47:45.740 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:45.749+0000 7fef03e9e1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:47:45.853 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:45.861+0000 7f40ad6511c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T09:47:45.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.889+0000 7fedf8fff700 1 Processor -- start 2024-01-11T09:47:45.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.889+0000 7fedf8fff700 1 -- start start 2024-01-11T09:47:45.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.890+0000 7fedf8fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fedf4157410 0x7fedf40ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:45.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.890+0000 7fedf8fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fedf4069540 0x7fedf40c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:45.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.890+0000 7fedf8fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf40699d0 con 0x7fedf41577e0 2024-01-11T09:47:45.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.890+0000 7fedf8fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf4069cb0 con 0x7fedf4069540 2024-01-11T09:47:45.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.890+0000 7fedf8fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf4069f90 con 0x7fedf4157410 2024-01-11T09:47:45.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.891+0000 7fedf1d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fedf41577e0 0x7fedf40ae000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37762/0 (socket says 172.21.15.27:37762) 2024-01-11T09:47:45.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.891+0000 7fedf1d9b700 1 -- 172.21.15.27:0/2244784582 learned_addr learned my addr 172.21.15.27:0/2244784582 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:45.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.891+0000 7fedf259c700 1 --2- 172.21.15.27:0/2244784582 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fedf4157410 0x7fedf40ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:45.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.891+0000 7fedf2d9d700 1 --2- 172.21.15.27:0/2244784582 >> v2:172.21.15.77:3300/0 conn(0x7fedf4069540 0x7fedf40c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:45.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.891+0000 7fedf159a700 1 -- 172.21.15.27:0/2244784582 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111264390 0 0) 0x7fedf40699d0 con 0x7fedf41577e0 2024-01-11T09:47:45.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.891+0000 7fedf159a700 1 -- 172.21.15.27:0/2244784582 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feddc003570 con 0x7fedf41577e0 2024-01-11T09:47:45.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.891+0000 7fedf2d9d700 1 -- 172.21.15.27:0/2244784582 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fedf4157410 msgr2=0x7fedf40ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:45.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.891+0000 7fedf2d9d700 1 --2- 172.21.15.27:0/2244784582 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fedf4157410 0x7fedf40ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.891+0000 7fedf2d9d700 1 -- 172.21.15.27:0/2244784582 >> v1:172.21.15.27:6789/0 conn(0x7fedf41577e0 legacy=0x7fedf40ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:45.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.891+0000 7fedf2d9d700 1 -- 172.21.15.27:0/2244784582 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf406a270 con 0x7fedf4069540 2024-01-11T09:47:45.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.892+0000 7fedf2d9d700 1 --2- 172.21.15.27:0/2244784582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedf4069540 0x7fedf40c6ba0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fedec009e60 tx=0x7fedec009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b4beeacbf16e941 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:45.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.892+0000 7fedf159a700 1 -- 172.21.15.27:0/2244784582 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec00e070 con 0x7fedf4069540 2024-01-11T09:47:45.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.892+0000 7fedf159a700 1 -- 172.21.15.27:0/2244784582 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedec00bab0 con 0x7fedf4069540 2024-01-11T09:47:45.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.892+0000 7fedf159a700 1 -- 172.21.15.27:0/2244784582 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec00be70 con 0x7fedf4069540 2024-01-11T09:47:45.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.892+0000 7fedf8fff700 1 -- 172.21.15.27:0/2244784582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedf4069540 msgr2=0x7fedf40c6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:45.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.892+0000 7fedf8fff700 1 --2- 172.21.15.27:0/2244784582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedf4069540 0x7fedf40c6ba0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fedec009e60 tx=0x7fedec009780 comp rx=0 tx=0).stop 2024-01-11T09:47:45.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.893+0000 7fedf8fff700 1 -- 172.21.15.27:0/2244784582 shutdown_connections 2024-01-11T09:47:45.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.893+0000 7fedf8fff700 1 --2- 172.21.15.27:0/2244784582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedf4069540 0x7fedf40c6ba0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.893+0000 7fedf8fff700 1 --2- 172.21.15.27:0/2244784582 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fedf4157410 0x7fedf40ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.893+0000 7fedf8fff700 1 -- 172.21.15.27:0/2244784582 >> 172.21.15.27:0/2244784582 conn(0x7fedf40680a0 msgr2=0x7fedf40cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:45.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.893+0000 7fedf8fff700 1 -- 172.21.15.27:0/2244784582 shutdown_connections 2024-01-11T09:47:45.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.893+0000 7fedf8fff700 1 -- 172.21.15.27:0/2244784582 wait complete. 2024-01-11T09:47:45.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.894+0000 7fedf8fff700 1 Processor -- start 2024-01-11T09:47:45.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.894+0000 7fedf8fff700 1 -- start start 2024-01-11T09:47:45.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.895+0000 7fedf8fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedf4069540 0x7fedf414be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:45.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.895+0000 7fedf8fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf4157410 0x7fedf415d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:45.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.895+0000 7fedf8fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fedf41577e0 0x7fedf416cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:45.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.895+0000 7fedf8fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf415bfc0 con 0x7fedf4157410 2024-01-11T09:47:45.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.895+0000 7fedf8fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf415ce20 con 0x7fedf4069540 2024-01-11T09:47:45.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.895+0000 7fedf8fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf415c210 con 0x7fedf41577e0 2024-01-11T09:47:45.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.896+0000 7fedf1d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf4157410 0x7fedf415d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:45.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.896+0000 7fedf259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedf4069540 0x7fedf414be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:45.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.896+0000 7fedf259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedf4069540 0x7fedf414be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:49054/0 (socket says 172.21.15.27:49054) 2024-01-11T09:47:45.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.896+0000 7fedf259c700 1 -- 172.21.15.27:0/110199696 learned_addr learned my addr 172.21.15.27:0/110199696 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:45.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.896+0000 7fedf2d9d700 1 --2- 172.21.15.27:0/110199696 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fedf41577e0 0x7fedf416cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.896+0000 7fedf1d9b700 1 -- 172.21.15.27:0/110199696 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fedf41577e0 msgr2=0x7fedf416cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.896+0000 7fedf1d9b700 1 --2- 172.21.15.27:0/110199696 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fedf41577e0 0x7fedf416cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.896+0000 7fedf1d9b700 1 -- 172.21.15.27:0/110199696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedf4069540 msgr2=0x7fedf414be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.896+0000 7fedf1d9b700 1 --2- 172.21.15.27:0/110199696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedf4069540 0x7fedf414be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.896+0000 7fedf1d9b700 1 -- 172.21.15.27:0/110199696 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedec009a20 con 0x7fedf4157410 2024-01-11T09:47:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.896+0000 7fedf259c700 1 --2- 172.21.15.27:0/110199696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedf4069540 0x7fedf414be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.896+0000 7fedf1d9b700 1 --2- 172.21.15.27:0/110199696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf4157410 0x7fedf415d2b0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7fedf406d560 tx=0x7fede8007d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.897+0000 7fede27fc700 1 -- 172.21.15.27:0/110199696 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede8005b30 con 0x7fedf4157410 2024-01-11T09:47:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.897+0000 7fede27fc700 1 -- 172.21.15.27:0/110199696 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede8005cb0 con 0x7fedf4157410 2024-01-11T09:47:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.897+0000 7fede27fc700 1 -- 172.21.15.27:0/110199696 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede8007320 con 0x7fedf4157410 2024-01-11T09:47:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.897+0000 7fedf8fff700 1 -- 172.21.15.27:0/110199696 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf415c4f0 con 0x7fedf4157410 2024-01-11T09:47:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.897+0000 7fedf8fff700 1 -- 172.21.15.27:0/110199696 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf415cab0 con 0x7fedf4157410 2024-01-11T09:47:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.898+0000 7fedf8fff700 1 -- 172.21.15.27:0/110199696 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf415cf90 con 0x7fedf4157410 2024-01-11T09:47:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.899+0000 7fede27fc700 1 -- 172.21.15.27:0/110199696 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7fede801b3b0 con 0x7fedf4157410 2024-01-11T09:47:45.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.900+0000 7fede27fc700 1 --2- 172.21.15.27:0/110199696 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fedd40670e0 0x7fedd4069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:45.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.900+0000 7fede27fc700 1 -- 172.21.15.27:0/110199696 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6247+0+0 (secure 0 0 0) 0x7fede8083e50 con 0x7fedf4157410 2024-01-11T09:47:45.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.900+0000 7fedf259c700 1 -- 172.21.15.27:0/110199696 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fedd40670e0 msgr2=0x7fedd4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:45.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.900+0000 7fedf259c700 1 --2- 172.21.15.27:0/110199696 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fedd40670e0 0x7fedd4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:45.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:45.907+0000 7fede27fc700 1 -- 172.21.15.27:0/110199696 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fede8050b80 con 0x7fedf4157410 2024-01-11T09:47:46.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.101+0000 7fedf259c700 1 -- 172.21.15.27:0/110199696 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fedd40670e0 msgr2=0x7fedd4069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:46.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.101+0000 7fedf259c700 1 --2- 172.21.15.27:0/110199696 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fedd40670e0 0x7fedd4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:46.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.115+0000 7fedf8fff700 1 -- 172.21.15.27:0/110199696 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fedf404c3c0 con 0x7fedf4157410 2024-01-11T09:47:46.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.115+0000 7fede27fc700 1 -- 172.21.15.27:0/110199696 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v257) v1 ==== 93+0+10 (secure 0 0 0) 0x7fede8017020 con 0x7fedf4157410 2024-01-11T09:47:46.107 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-11T09:47:46.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.118+0000 7fedf8fff700 1 -- 172.21.15.27:0/110199696 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fedd40670e0 msgr2=0x7fedd4069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:46.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.118+0000 7fedf8fff700 1 --2- 172.21.15.27:0/110199696 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fedd40670e0 0x7fedd4069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:46.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.118+0000 7fedf8fff700 1 -- 172.21.15.27:0/110199696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf4157410 msgr2=0x7fedf415d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:46.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.118+0000 7fedf8fff700 1 --2- 172.21.15.27:0/110199696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf4157410 0x7fedf415d2b0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7fedf406d560 tx=0x7fede8007d50 comp rx=0 tx=0).stop 2024-01-11T09:47:46.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.118+0000 7fedf8fff700 1 -- 172.21.15.27:0/110199696 shutdown_connections 2024-01-11T09:47:46.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.118+0000 7fedf8fff700 1 --2- 172.21.15.27:0/110199696 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fedf41577e0 0x7fedf416cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:46.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.118+0000 7fedf8fff700 1 --2- 172.21.15.27:0/110199696 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fedd40670e0 0x7fedd4069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:46.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.118+0000 7fedf8fff700 1 --2- 172.21.15.27:0/110199696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedf4069540 0x7fedf414be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:46.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.118+0000 7fedf8fff700 1 --2- 172.21.15.27:0/110199696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf4157410 0x7fedf415d2b0 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:46.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.118+0000 7fedf8fff700 1 -- 172.21.15.27:0/110199696 >> 172.21.15.27:0/110199696 conn(0x7fedf40680a0 msgr2=0x7fedf40c29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:46.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.118+0000 7fedf8fff700 1 -- 172.21.15.27:0/110199696 shutdown_connections 2024-01-11T09:47:46.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.118+0000 7fedf8fff700 1 -- 172.21.15.27:0/110199696 wait complete. 2024-01-11T09:47:46.128 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-11T09:47:46.156 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:46.165+0000 7fef03e9e1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:47:46.315 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:46.323+0000 7f40ad6511c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T09:47:46.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.353+0000 7f491b5ff700 1 Processor -- start 2024-01-11T09:47:46.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.354+0000 7f491b5ff700 1 -- start start 2024-01-11T09:47:46.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.355+0000 7f491b5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4914157410 0x7f49140ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:46.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.355+0000 7f491b5ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49140699d0 con 0x7f49141577e0 2024-01-11T09:47:46.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.355+0000 7f491b5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4914069cb0 con 0x7f4914069540 2024-01-11T09:47:46.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.355+0000 7f491b5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4914069f90 con 0x7f4914157410 2024-01-11T09:47:46.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.355+0000 7f4918b9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f49141577e0 0x7f49140ae000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37766/0 (socket says 172.21.15.27:37766) 2024-01-11T09:47:46.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.355+0000 7f4918b9a700 1 -- 172.21.15.27:0/372803718 learned_addr learned my addr 172.21.15.27:0/372803718 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:46.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.355+0000 7f491939b700 1 --2- 172.21.15.27:0/372803718 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4914157410 0x7f49140ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:46.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.356+0000 7f491939b700 1 -- 172.21.15.27:0/372803718 >> v1:172.21.15.77:6789/0 conn(0x7f4914069540 legacy=0x7f49140c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:46.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.356+0000 7f490bfff700 1 -- 172.21.15.27:0/372803718 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3747635078 0 0) 0x7f4914069cb0 con 0x7f4914069540 2024-01-11T09:47:46.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.356+0000 7f491939b700 1 -- 172.21.15.27:0/372803718 >> v1:172.21.15.27:6789/0 conn(0x7f49141577e0 legacy=0x7f49140ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:46.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.356+0000 7f491939b700 1 -- 172.21.15.27:0/372803718 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491406a270 con 0x7f4914157410 2024-01-11T09:47:46.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.356+0000 7f491939b700 1 --2- 172.21.15.27:0/372803718 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4914157410 0x7f49140ccae0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f4910009900 tx=0x7f4910009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af11795c0c29e95c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:46.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.356+0000 7f490bfff700 1 -- 172.21.15.27:0/372803718 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4910009e40 con 0x7f4914157410 2024-01-11T09:47:46.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.356+0000 7f490bfff700 1 -- 172.21.15.27:0/372803718 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491000b760 con 0x7f4914157410 2024-01-11T09:47:46.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.356+0000 7f490bfff700 1 -- 172.21.15.27:0/372803718 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491000bb00 con 0x7f4914157410 2024-01-11T09:47:46.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.357+0000 7f491b5ff700 1 -- 172.21.15.27:0/372803718 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4914157410 msgr2=0x7f49140ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:46.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.357+0000 7f491b5ff700 1 --2- 172.21.15.27:0/372803718 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4914157410 0x7f49140ccae0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f4910009900 tx=0x7f4910009120 comp rx=0 tx=0).stop 2024-01-11T09:47:46.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.357+0000 7f491b5ff700 1 -- 172.21.15.27:0/372803718 shutdown_connections 2024-01-11T09:47:46.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.357+0000 7f491b5ff700 1 --2- 172.21.15.27:0/372803718 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4914157410 0x7f49140ccae0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:46.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.357+0000 7f491b5ff700 1 -- 172.21.15.27:0/372803718 >> 172.21.15.27:0/372803718 conn(0x7f49140680a0 msgr2=0x7f49140cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:46.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.357+0000 7f491b5ff700 1 -- 172.21.15.27:0/372803718 shutdown_connections 2024-01-11T09:47:46.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.357+0000 7f491b5ff700 1 -- 172.21.15.27:0/372803718 wait complete. 2024-01-11T09:47:46.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.359+0000 7f491b5ff700 1 Processor -- start 2024-01-11T09:47:46.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.359+0000 7f491b5ff700 1 -- start start 2024-01-11T09:47:46.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.360+0000 7f491b5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4914069540 0x7f49141530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:46.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.360+0000 7f491b5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4914157410 0x7f49140b10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:46.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.360+0000 7f491b5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49141577e0 0x7f49140b1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:46.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.360+0000 7f491b5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491415e190 con 0x7f4914157410 2024-01-11T09:47:46.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.360+0000 7f491b5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49141535e0 con 0x7f49141577e0 2024-01-11T09:47:46.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.360+0000 7f491b5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f491415dbd0 con 0x7f4914069540 2024-01-11T09:47:46.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.360+0000 7f4918b9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4914157410 0x7f49140b10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:46.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.360+0000 7f491939b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4914069540 0x7f49141530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:46.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.360+0000 7f491939b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4914069540 0x7f49141530a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:51388/0 (socket says 172.21.15.27:51388) 2024-01-11T09:47:46.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.360+0000 7f491939b700 1 -- 172.21.15.27:0/3504751563 learned_addr learned my addr 172.21.15.27:0/3504751563 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:46.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.360+0000 7f4919b9c700 1 --2- 172.21.15.27:0/3504751563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49141577e0 0x7f49140b1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:46.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.360+0000 7f491939b700 1 -- 172.21.15.27:0/3504751563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49141577e0 msgr2=0x7f49140b1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:46.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.360+0000 7f491939b700 1 --2- 172.21.15.27:0/3504751563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49141577e0 0x7f49140b1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:46.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.360+0000 7f491939b700 1 -- 172.21.15.27:0/3504751563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4914157410 msgr2=0x7f49140b10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:46.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.360+0000 7f491939b700 1 --2- 172.21.15.27:0/3504751563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4914157410 0x7f49140b10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:46.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.360+0000 7f491939b700 1 -- 172.21.15.27:0/3504751563 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49100093c0 con 0x7f4914069540 2024-01-11T09:47:46.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.361+0000 7f491939b700 1 --2- 172.21.15.27:0/3504751563 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4914069540 0x7f49141530a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f491000c010 tx=0x7f4910016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:46.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.361+0000 7f49097fa700 1 -- 172.21.15.27:0/3504751563 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4910011ea0 con 0x7f4914069540 2024-01-11T09:47:46.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.361+0000 7f49097fa700 1 -- 172.21.15.27:0/3504751563 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491001c460 con 0x7f4914069540 2024-01-11T09:47:46.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.361+0000 7f49097fa700 1 -- 172.21.15.27:0/3504751563 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491001c800 con 0x7f4914069540 2024-01-11T09:47:46.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.361+0000 7f491b5ff700 1 -- 172.21.15.27:0/3504751563 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f491415deb0 con 0x7f4914069540 2024-01-11T09:47:46.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.361+0000 7f491b5ff700 1 -- 172.21.15.27:0/3504751563 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49140c36f0 con 0x7f4914069540 2024-01-11T09:47:46.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.362+0000 7f491b5ff700 1 -- 172.21.15.27:0/3504751563 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4914068f30 con 0x7f4914069540 2024-01-11T09:47:46.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.364+0000 7f49097fa700 1 -- 172.21.15.27:0/3504751563 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f491001cbf0 con 0x7f4914069540 2024-01-11T09:47:46.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.365+0000 7f49097fa700 1 --2- 172.21.15.27:0/3504751563 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f48f80670e0 0x7f48f8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:46.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.365+0000 7f49097fa700 1 -- 172.21.15.27:0/3504751563 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6247+0+0 (secure 0 0 0) 0x7f4910086e90 con 0x7f4914069540 2024-01-11T09:47:46.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.365+0000 7f4918b9a700 1 -- 172.21.15.27:0/3504751563 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f48f80670e0 msgr2=0x7f48f8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:46.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.365+0000 7f4918b9a700 1 --2- 172.21.15.27:0/3504751563 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f48f80670e0 0x7f48f8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:46.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.370+0000 7f49097fa700 1 -- 172.21.15.27:0/3504751563 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4910053c60 con 0x7f4914069540 2024-01-11T09:47:46.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.566+0000 7f4918b9a700 1 -- 172.21.15.27:0/3504751563 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f48f80670e0 msgr2=0x7f48f8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:46.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.566+0000 7f4918b9a700 1 --2- 172.21.15.27:0/3504751563 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f48f80670e0 0x7f48f8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:46.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.574+0000 7f491b5ff700 1 -- 172.21.15.27:0/3504751563 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4914068f30 con 0x7f4914069540 2024-01-11T09:47:46.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.575+0000 7f49097fa700 1 -- 172.21.15.27:0/3504751563 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v257) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4910057280 con 0x7f4914069540 2024-01-11T09:47:46.566 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-11T09:47:46.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.577+0000 7f491b5ff700 1 -- 172.21.15.27:0/3504751563 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f48f80670e0 msgr2=0x7f48f8069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:46.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.577+0000 7f491b5ff700 1 --2- 172.21.15.27:0/3504751563 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f48f80670e0 0x7f48f8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:46.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.577+0000 7f491b5ff700 1 -- 172.21.15.27:0/3504751563 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4914069540 msgr2=0x7f49141530a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:46.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.577+0000 7f491b5ff700 1 --2- 172.21.15.27:0/3504751563 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4914069540 0x7f49141530a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f491000c010 tx=0x7f4910016000 comp rx=0 tx=0).stop 2024-01-11T09:47:46.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.577+0000 7f491b5ff700 1 -- 172.21.15.27:0/3504751563 shutdown_connections 2024-01-11T09:47:46.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.577+0000 7f491b5ff700 1 --2- 172.21.15.27:0/3504751563 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f48f80670e0 0x7f48f8069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:46.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.577+0000 7f491b5ff700 1 --2- 172.21.15.27:0/3504751563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49141577e0 0x7f49140b1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:46.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.577+0000 7f491b5ff700 1 --2- 172.21.15.27:0/3504751563 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4914069540 0x7f49141530a0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:46.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.577+0000 7f491b5ff700 1 --2- 172.21.15.27:0/3504751563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4914157410 0x7f49140b10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:46.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.577+0000 7f491b5ff700 1 -- 172.21.15.27:0/3504751563 >> 172.21.15.27:0/3504751563 conn(0x7f49140680a0 msgr2=0x7f49140c1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:46.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.577+0000 7f491b5ff700 1 -- 172.21.15.27:0/3504751563 shutdown_connections 2024-01-11T09:47:46.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.577+0000 7f491b5ff700 1 -- 172.21.15.27:0/3504751563 wait complete. 2024-01-11T09:47:46.589 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-11T09:47:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.801+0000 7fa753dff700 1 Processor -- start 2024-01-11T09:47:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.802+0000 7fa753dff700 1 -- start start 2024-01-11T09:47:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.803+0000 7fa753dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa74c155c50 0x7fa74c0aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.803+0000 7fa753dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa74c0c74f0 0x7fa74c069910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.803+0000 7fa753dff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa74c06e480 con 0x7fa74c0694c0 2024-01-11T09:47:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.804+0000 7fa753dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa74c0cb5a0 con 0x7fa74c155c50 2024-01-11T09:47:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.804+0000 7fa753dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa74c06df00 con 0x7fa74c0c74f0 2024-01-11T09:47:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.804+0000 7fa751b9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa74c155c50 0x7fa74c0aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.804+0000 7fa751b9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa74c155c50 0x7fa74c0aef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60544/0 (socket says 172.21.15.27:60544) 2024-01-11T09:47:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.804+0000 7fa751b9b700 1 -- 172.21.15.27:0/4051029421 learned_addr learned my addr 172.21.15.27:0/4051029421 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.804+0000 7fa75239c700 1 --2- 172.21.15.27:0/4051029421 >> v2:172.21.15.77:3300/0 conn(0x7fa74c0c74f0 0x7fa74c069910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.804+0000 7fa751b9b700 1 -- 172.21.15.27:0/4051029421 >> v2:172.21.15.77:3300/0 conn(0x7fa74c0c74f0 msgr2=0x7fa74c069910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.804+0000 7fa751b9b700 1 --2- 172.21.15.27:0/4051029421 >> v2:172.21.15.77:3300/0 conn(0x7fa74c0c74f0 0x7fa74c069910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.804+0000 7fa751b9b700 1 -- 172.21.15.27:0/4051029421 >> v1:172.21.15.77:6789/0 conn(0x7fa74c0694c0 legacy=0x7fa74c0c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.804+0000 7fa751b9b700 1 -- 172.21.15.27:0/4051029421 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74c069e50 con 0x7fa74c155c50 2024-01-11T09:47:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.805+0000 7fa751b9b700 1 --2- 172.21.15.27:0/4051029421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa74c155c50 0x7fa74c0aef80 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7fa74c06d540 tx=0x7fa748009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3ea0d036658c65 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.805+0000 7fa750b99700 1 -- 172.21.15.27:0/4051029421 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa748009e60 con 0x7fa74c155c50 2024-01-11T09:47:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.805+0000 7fa750b99700 1 -- 172.21.15.27:0/4051029421 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74800b8f0 con 0x7fa74c155c50 2024-01-11T09:47:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.805+0000 7fa750b99700 1 -- 172.21.15.27:0/4051029421 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74800bc90 con 0x7fa74c155c50 2024-01-11T09:47:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.805+0000 7fa753dff700 1 -- 172.21.15.27:0/4051029421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa74c155c50 msgr2=0x7fa74c0aef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.805+0000 7fa753dff700 1 --2- 172.21.15.27:0/4051029421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa74c155c50 0x7fa74c0aef80 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7fa74c06d540 tx=0x7fa748009250 comp rx=0 tx=0).stop 2024-01-11T09:47:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.806+0000 7fa753dff700 1 -- 172.21.15.27:0/4051029421 shutdown_connections 2024-01-11T09:47:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.806+0000 7fa753dff700 1 --2- 172.21.15.27:0/4051029421 >> v2:172.21.15.77:3300/0 conn(0x7fa74c0c74f0 0x7fa74c069910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.806+0000 7fa753dff700 1 --2- 172.21.15.27:0/4051029421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa74c155c50 0x7fa74c0aef80 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.806+0000 7fa753dff700 1 -- 172.21.15.27:0/4051029421 >> 172.21.15.27:0/4051029421 conn(0x7fa74c0680a0 msgr2=0x7fa74c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.806+0000 7fa753dff700 1 -- 172.21.15.27:0/4051029421 shutdown_connections 2024-01-11T09:47:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.806+0000 7fa753dff700 1 -- 172.21.15.27:0/4051029421 wait complete. 2024-01-11T09:47:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.808+0000 7fa753dff700 1 Processor -- start 2024-01-11T09:47:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.808+0000 7fa753dff700 1 -- start start 2024-01-11T09:47:46.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.810+0000 7fa753dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa74c0694c0 0x7fa74c167c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:46.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.810+0000 7fa753dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa74c0c74f0 0x7fa74c149c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:46.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.810+0000 7fa753dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa74c155c50 0x7fa74c0c3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.810+0000 7fa753dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c1681b0 con 0x7fa74c155c50 2024-01-11T09:47:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.810+0000 7fa753dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c168fd0 con 0x7fa74c0694c0 2024-01-11T09:47:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.810+0000 7fa753dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c156020 con 0x7fa74c0c74f0 2024-01-11T09:47:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.810+0000 7fa75239c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa74c155c50 0x7fa74c0c3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.810+0000 7fa75239c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa74c155c50 0x7fa74c0c3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60560/0 (socket says 172.21.15.27:60560) 2024-01-11T09:47:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.810+0000 7fa75239c700 1 -- 172.21.15.27:0/1260630128 learned_addr learned my addr 172.21.15.27:0/1260630128 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.810+0000 7fa751b9b700 1 --2- 172.21.15.27:0/1260630128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa74c0694c0 0x7fa74c167c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.810+0000 7fa75139a700 1 --2- 172.21.15.27:0/1260630128 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa74c0c74f0 0x7fa74c149c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.811+0000 7fa75239c700 1 -- 172.21.15.27:0/1260630128 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa74c0c74f0 msgr2=0x7fa74c149c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.811+0000 7fa75239c700 1 --2- 172.21.15.27:0/1260630128 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa74c0c74f0 0x7fa74c149c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.811+0000 7fa75239c700 1 -- 172.21.15.27:0/1260630128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa74c0694c0 msgr2=0x7fa74c167c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:46.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.811+0000 7fa75239c700 1 --2- 172.21.15.27:0/1260630128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa74c0694c0 0x7fa74c167c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:46.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.811+0000 7fa75239c700 1 -- 172.21.15.27:0/1260630128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa748009490 con 0x7fa74c155c50 2024-01-11T09:47:46.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.811+0000 7fa75239c700 1 --2- 172.21.15.27:0/1260630128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa74c155c50 0x7fa74c0c3950 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7fa744009e90 tx=0x7fa74400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:46.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.811+0000 7fa741ffb700 1 -- 172.21.15.27:0/1260630128 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74400cb70 con 0x7fa74c155c50 2024-01-11T09:47:46.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.811+0000 7fa741ffb700 1 -- 172.21.15.27:0/1260630128 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74400f040 con 0x7fa74c155c50 2024-01-11T09:47:46.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.811+0000 7fa753dff700 1 -- 172.21.15.27:0/1260630128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa74c0c4d50 con 0x7fa74c155c50 2024-01-11T09:47:46.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.812+0000 7fa741ffb700 1 -- 172.21.15.27:0/1260630128 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa744014480 con 0x7fa74c155c50 2024-01-11T09:47:46.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.812+0000 7fa753dff700 1 -- 172.21.15.27:0/1260630128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74c168720 con 0x7fa74c155c50 2024-01-11T09:47:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.813+0000 7fa753dff700 1 -- 172.21.15.27:0/1260630128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa74c04c3c0 con 0x7fa74c155c50 2024-01-11T09:47:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.814+0000 7fa741ffb700 1 -- 172.21.15.27:0/1260630128 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7fa744014830 con 0x7fa74c155c50 2024-01-11T09:47:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.815+0000 7fa741ffb700 1 --2- 172.21.15.27:0/1260630128 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa7300670e0 0x7fa730069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.815+0000 7fa741ffb700 1 -- 172.21.15.27:0/1260630128 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6247+0+0 (secure 0 0 0) 0x7fa744085650 con 0x7fa74c155c50 2024-01-11T09:47:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.815+0000 7fa751b9b700 1 -- 172.21.15.27:0/1260630128 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa7300670e0 msgr2=0x7fa730069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.815+0000 7fa751b9b700 1 --2- 172.21.15.27:0/1260630128 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa7300670e0 0x7fa730069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:46.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:46.823+0000 7fa741ffb700 1 -- 172.21.15.27:0/1260630128 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa744052270 con 0x7fa74c155c50 2024-01-11T09:47:46.935 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:46.944+0000 7fef03e9e1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:47:47.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.016+0000 7fa751b9b700 1 -- 172.21.15.27:0/1260630128 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa7300670e0 msgr2=0x7fa730069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:47.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.016+0000 7fa751b9b700 1 --2- 172.21.15.27:0/1260630128 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa7300670e0 0x7fa730069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:47.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.026+0000 7fa753dff700 1 -- 172.21.15.27:0/1260630128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa74c169110 con 0x7fa74c155c50 2024-01-11T09:47:47.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.027+0000 7fa741ffb700 1 -- 172.21.15.27:0/1260630128 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v257) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa744055890 con 0x7fa74c155c50 2024-01-11T09:47:47.018 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-11T09:47:47.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.029+0000 7fa753dff700 1 -- 172.21.15.27:0/1260630128 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa7300670e0 msgr2=0x7fa730069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:47.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.029+0000 7fa753dff700 1 --2- 172.21.15.27:0/1260630128 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa7300670e0 0x7fa730069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.029+0000 7fa753dff700 1 -- 172.21.15.27:0/1260630128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa74c155c50 msgr2=0x7fa74c0c3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:47.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.029+0000 7fa753dff700 1 --2- 172.21.15.27:0/1260630128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa74c155c50 0x7fa74c0c3950 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7fa744009e90 tx=0x7fa74400b040 comp rx=0 tx=0).stop 2024-01-11T09:47:47.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.030+0000 7fa753dff700 1 -- 172.21.15.27:0/1260630128 shutdown_connections 2024-01-11T09:47:47.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.030+0000 7fa753dff700 1 --2- 172.21.15.27:0/1260630128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa74c155c50 0x7fa74c0c3950 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.030+0000 7fa753dff700 1 --2- 172.21.15.27:0/1260630128 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa7300670e0 0x7fa730069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.030+0000 7fa753dff700 1 --2- 172.21.15.27:0/1260630128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa74c0694c0 0x7fa74c167c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.030+0000 7fa753dff700 1 --2- 172.21.15.27:0/1260630128 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa74c0c74f0 0x7fa74c149c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.030+0000 7fa753dff700 1 -- 172.21.15.27:0/1260630128 >> 172.21.15.27:0/1260630128 conn(0x7fa74c0680a0 msgr2=0x7fa74c14db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:47.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.030+0000 7fa753dff700 1 -- 172.21.15.27:0/1260630128 shutdown_connections 2024-01-11T09:47:47.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.030+0000 7fa753dff700 1 -- 172.21.15.27:0/1260630128 wait complete. 2024-01-11T09:47:47.023 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:47.032+0000 7fef03e9e1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:47:47.041 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:47:47.172 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:47.181+0000 7f40ad6511c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T09:47:47.180 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:47.189+0000 7fef03e9e1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:47:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.259+0000 7f37965ff700 1 Processor -- start 2024-01-11T09:47:47.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.259+0000 7f37965ff700 1 -- start start 2024-01-11T09:47:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.261+0000 7f37965ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f37900ad020 0x7f37900af320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.261+0000 7f37965ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37900aa1d0 con 0x7f37900ac370 2024-01-11T09:47:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.261+0000 7f37965ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37900af860 con 0x7f3790159890 2024-01-11T09:47:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.261+0000 7f37965ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37900ac8d0 con 0x7f37900ad020 2024-01-11T09:47:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.261+0000 7f378ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f37900ad020 0x7f37900af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.261+0000 7f378f7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f37900ac370 0x7f3790159180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37774/0 (socket says 172.21.15.27:37774) 2024-01-11T09:47:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.261+0000 7f378f7fe700 1 -- 172.21.15.27:0/628074063 learned_addr learned my addr 172.21.15.27:0/628074063 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.262+0000 7f378ffff700 1 -- 172.21.15.27:0/628074063 >> v1:172.21.15.77:6789/0 conn(0x7f3790159890 legacy=0x7f37900aa660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.262+0000 7f378ffff700 1 -- 172.21.15.27:0/628074063 >> v1:172.21.15.27:6789/0 conn(0x7f37900ac370 legacy=0x7f3790159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.262+0000 7f378ffff700 1 -- 172.21.15.27:0/628074063 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37900a98a0 con 0x7f37900ad020 2024-01-11T09:47:47.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.262+0000 7f378ffff700 1 --2- 172.21.15.27:0/628074063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37900ad020 0x7f37900af320 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f3780009910 tx=0x7f37800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926786a2cbd4b5d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:47.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.262+0000 7f378effd700 1 -- 172.21.15.27:0/628074063 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378000b820 con 0x7f37900ad020 2024-01-11T09:47:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.262+0000 7f378effd700 1 -- 172.21.15.27:0/628074063 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378000b980 con 0x7f37900ad020 2024-01-11T09:47:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.262+0000 7f378effd700 1 -- 172.21.15.27:0/628074063 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378000bd20 con 0x7f37900ad020 2024-01-11T09:47:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.263+0000 7f37965ff700 1 -- 172.21.15.27:0/628074063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37900ad020 msgr2=0x7f37900af320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.263+0000 7f37965ff700 1 --2- 172.21.15.27:0/628074063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37900ad020 0x7f37900af320 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f3780009910 tx=0x7f37800091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.263+0000 7f37965ff700 1 -- 172.21.15.27:0/628074063 shutdown_connections 2024-01-11T09:47:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.263+0000 7f37965ff700 1 --2- 172.21.15.27:0/628074063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37900ad020 0x7f37900af320 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.263+0000 7f37965ff700 1 -- 172.21.15.27:0/628074063 >> 172.21.15.27:0/628074063 conn(0x7f37900680e0 msgr2=0x7f3790069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.263+0000 7f37965ff700 1 -- 172.21.15.27:0/628074063 shutdown_connections 2024-01-11T09:47:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.263+0000 7f37965ff700 1 -- 172.21.15.27:0/628074063 wait complete. 2024-01-11T09:47:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.264+0000 7f37965ff700 1 Processor -- start 2024-01-11T09:47:47.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.265+0000 7f37965ff700 1 -- start start 2024-01-11T09:47:47.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.265+0000 7f37965ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37900ac370 0x7f37901576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:47.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.265+0000 7f37965ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37900ad020 0x7f37901550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:47.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.266+0000 7f37965ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3790159890 0x7f37900cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:47.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.266+0000 7f37965ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37901566d0 con 0x7f37900ad020 2024-01-11T09:47:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.266+0000 7f37965ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3790155ac0 con 0x7f37900ac370 2024-01-11T09:47:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.266+0000 7f37965ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3790155da0 con 0x7f3790159890 2024-01-11T09:47:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.266+0000 7f378f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37900ad020 0x7f37901550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.266+0000 7f378f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37900ad020 0x7f37901550e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60566/0 (socket says 172.21.15.27:60566) 2024-01-11T09:47:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.266+0000 7f378f7fe700 1 -- 172.21.15.27:0/2580903114 learned_addr learned my addr 172.21.15.27:0/2580903114 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.266+0000 7f378ffff700 1 --2- 172.21.15.27:0/2580903114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37900ac370 0x7f37901576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.266+0000 7f3794b9c700 1 --2- 172.21.15.27:0/2580903114 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3790159890 0x7f37900cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.266+0000 7f378f7fe700 1 -- 172.21.15.27:0/2580903114 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3790159890 msgr2=0x7f37900cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.266+0000 7f378f7fe700 1 --2- 172.21.15.27:0/2580903114 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3790159890 0x7f37900cbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.266+0000 7f378f7fe700 1 -- 172.21.15.27:0/2580903114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37900ac370 msgr2=0x7f37901576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.266+0000 7f378f7fe700 1 --2- 172.21.15.27:0/2580903114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37900ac370 0x7f37901576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.266+0000 7f378f7fe700 1 -- 172.21.15.27:0/2580903114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3780009420 con 0x7f37900ad020 2024-01-11T09:47:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.266+0000 7f378f7fe700 1 --2- 172.21.15.27:0/2580903114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37900ad020 0x7f37901550e0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f378400c560 tx=0x7f3784001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.267+0000 7f3777fff700 1 -- 172.21.15.27:0/2580903114 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3784005b30 con 0x7f37900ad020 2024-01-11T09:47:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.267+0000 7f3777fff700 1 -- 172.21.15.27:0/2580903114 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3784005c90 con 0x7f37900ad020 2024-01-11T09:47:47.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.267+0000 7f3777fff700 1 -- 172.21.15.27:0/2580903114 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3784007320 con 0x7f37900ad020 2024-01-11T09:47:47.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.267+0000 7f37965ff700 1 -- 172.21.15.27:0/2580903114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3790156080 con 0x7f37900ad020 2024-01-11T09:47:47.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.267+0000 7f37965ff700 1 -- 172.21.15.27:0/2580903114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3790157bf0 con 0x7f37900ad020 2024-01-11T09:47:47.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.268+0000 7f37965ff700 1 -- 172.21.15.27:0/2580903114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37900718d0 con 0x7f37900ad020 2024-01-11T09:47:47.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.269+0000 7f3777fff700 1 -- 172.21.15.27:0/2580903114 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f378401b3b0 con 0x7f37900ad020 2024-01-11T09:47:47.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.270+0000 7f3777fff700 1 --2- 172.21.15.27:0/2580903114 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f377c0670e0 0x7f377c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:47.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.270+0000 7f3777fff700 1 -- 172.21.15.27:0/2580903114 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6247+0+0 (secure 0 0 0) 0x7f3784083e80 con 0x7f37900ad020 2024-01-11T09:47:47.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.270+0000 7f378ffff700 1 -- 172.21.15.27:0/2580903114 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f377c0670e0 msgr2=0x7f377c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:47.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.271+0000 7f378ffff700 1 --2- 172.21.15.27:0/2580903114 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f377c0670e0 0x7f377c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:47.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.274+0000 7f3777fff700 1 -- 172.21.15.27:0/2580903114 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3784050bb0 con 0x7f37900ad020 2024-01-11T09:47:47.268 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:47.277+0000 7fef03e9e1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:47:47.270 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:47.279+0000 7f40ad6511c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T09:47:47.440 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:47.449+0000 7f40ad6511c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T09:47:47.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.472+0000 7f378ffff700 1 -- 172.21.15.27:0/2580903114 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f377c0670e0 msgr2=0x7f377c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:47.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.472+0000 7f378ffff700 1 --2- 172.21.15.27:0/2580903114 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f377c0670e0 0x7f377c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:47.487 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:47.497+0000 7fef03e9e1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:47:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.525+0000 7f37965ff700 1 -- 172.21.15.27:0/2580903114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37900a9b80 con 0x7f37900ad020 2024-01-11T09:47:47.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.526+0000 7f3777fff700 1 -- 172.21.15.27:0/2580903114 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+4612 (secure 0 0 0) 0x7f3784017090 con 0x7f37900ad020 2024-01-11T09:47:47.517 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:47.517 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":141,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9866,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1060058066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1060058066},{"type":"v1","addr":"172.21.15.77:6835","nonce":1060058066}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137},{"gid":11626,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/4169546841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":4169546841},{"type":"v1","addr":"172.21.15.77:6834","nonce":4169546841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137},{"gid":11632,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/2567971522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2567971522},{"type":"v1","addr":"172.21.15.27:6833","nonce":2567971522}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:47:24.474082+0000","modified":"2024-01-11T09:47:27.985712+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12063},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12063":{"gid":12063,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1065634442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1065634442},{"type":"v1","addr":"172.21.15.27:6835","nonce":1065634442}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2024-01-11T09:47:47.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.528+0000 7f37965ff700 1 -- 172.21.15.27:0/2580903114 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f377c0670e0 msgr2=0x7f377c069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:47.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.528+0000 7f37965ff700 1 --2- 172.21.15.27:0/2580903114 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f377c0670e0 0x7f377c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.528+0000 7f37965ff700 1 -- 172.21.15.27:0/2580903114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37900ad020 msgr2=0x7f37901550e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:47.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.528+0000 7f37965ff700 1 --2- 172.21.15.27:0/2580903114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37900ad020 0x7f37901550e0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f378400c560 tx=0x7f3784001020 comp rx=0 tx=0).stop 2024-01-11T09:47:47.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.528+0000 7f37965ff700 1 -- 172.21.15.27:0/2580903114 shutdown_connections 2024-01-11T09:47:47.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.528+0000 7f37965ff700 1 --2- 172.21.15.27:0/2580903114 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f3790159890 0x7f37900cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.528+0000 7f37965ff700 1 --2- 172.21.15.27:0/2580903114 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f377c0670e0 0x7f377c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.528+0000 7f37965ff700 1 --2- 172.21.15.27:0/2580903114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37900ac370 0x7f37901576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.528+0000 7f37965ff700 1 --2- 172.21.15.27:0/2580903114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37900ad020 0x7f37901550e0 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.528+0000 7f37965ff700 1 -- 172.21.15.27:0/2580903114 >> 172.21.15.27:0/2580903114 conn(0x7f37900680e0 msgr2=0x7f379006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:47.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.529+0000 7f37965ff700 1 -- 172.21.15.27:0/2580903114 shutdown_connections 2024-01-11T09:47:47.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.529+0000 7f37965ff700 1 -- 172.21.15.27:0/2580903114 wait complete. 2024-01-11T09:47:47.521 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 141 2024-01-11T09:47:47.534 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:47.543+0000 7f40ad6511c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T09:47:47.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:47:47.583 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:47.592+0000 7fef03e9e1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:47:47.678 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:47.687+0000 7fef03e9e1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:47:47.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.750+0000 7f779f3ff700 1 Processor -- start 2024-01-11T09:47:47.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.750+0000 7f779f3ff700 1 -- start start 2024-01-11T09:47:47.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.752+0000 7f779f3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77980cc410 0x7f7798159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:47.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.752+0000 7f779f3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77980cd730 con 0x7f77980b03e0 2024-01-11T09:47:47.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.752+0000 7f779f3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77980ccb20 con 0x7f77980b2b90 2024-01-11T09:47:47.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.752+0000 7f779f3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77980cce00 con 0x7f77980cc410 2024-01-11T09:47:47.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.752+0000 7f779d19b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f77980b03e0 0x7f77980b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37786/0 (socket says 172.21.15.27:37786) 2024-01-11T09:47:47.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.752+0000 7f779d19b700 1 -- 172.21.15.27:0/1842865478 learned_addr learned my addr 172.21.15.27:0/1842865478 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.752+0000 7f779d99c700 1 --2- 172.21.15.27:0/1842865478 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77980cc410 0x7f7798159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.752+0000 7f778ffff700 1 -- 172.21.15.27:0/1842865478 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508929256 0 0) 0x7f77980cd730 con 0x7f77980b03e0 2024-01-11T09:47:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.753+0000 7f778ffff700 1 -- 172.21.15.27:0/1842865478 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7784003570 con 0x7f77980b03e0 2024-01-11T09:47:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.753+0000 7f778ffff700 1 -- 172.21.15.27:0/1842865478 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084403663 0 0) 0x7f77980ccb20 con 0x7f77980b2b90 2024-01-11T09:47:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.753+0000 7f778ffff700 1 -- 172.21.15.27:0/1842865478 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77980cd730 con 0x7f77980b2b90 2024-01-11T09:47:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.753+0000 7f779d99c700 1 -- 172.21.15.27:0/1842865478 >> v1:172.21.15.77:6789/0 conn(0x7f77980b2b90 legacy=0x7f7798159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.753+0000 7f779d99c700 1 -- 172.21.15.27:0/1842865478 >> v1:172.21.15.27:6789/0 conn(0x7f77980b03e0 legacy=0x7f77980b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.753+0000 7f779d99c700 1 -- 172.21.15.27:0/1842865478 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77980cd0e0 con 0x7f77980cc410 2024-01-11T09:47:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.753+0000 7f779d99c700 1 --2- 172.21.15.27:0/1842865478 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77980cc410 0x7f7798159910 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f7790009e90 tx=0x7f7790009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a80d8c2782b82be server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:47.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.753+0000 7f778ffff700 1 -- 172.21.15.27:0/1842865478 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779000e070 con 0x7f77980cc410 2024-01-11T09:47:47.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.753+0000 7f778ffff700 1 -- 172.21.15.27:0/1842865478 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779000bb80 con 0x7f77980cc410 2024-01-11T09:47:47.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.753+0000 7f778ffff700 1 -- 172.21.15.27:0/1842865478 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7790012430 con 0x7f77980cc410 2024-01-11T09:47:47.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.754+0000 7f779f3ff700 1 -- 172.21.15.27:0/1842865478 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77980cc410 msgr2=0x7f7798159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:47.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.754+0000 7f779f3ff700 1 --2- 172.21.15.27:0/1842865478 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77980cc410 0x7f7798159910 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f7790009e90 tx=0x7f7790009780 comp rx=0 tx=0).stop 2024-01-11T09:47:47.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.754+0000 7f779f3ff700 1 -- 172.21.15.27:0/1842865478 shutdown_connections 2024-01-11T09:47:47.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.754+0000 7f779f3ff700 1 --2- 172.21.15.27:0/1842865478 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77980cc410 0x7f7798159910 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.754+0000 7f779f3ff700 1 -- 172.21.15.27:0/1842865478 >> 172.21.15.27:0/1842865478 conn(0x7f779806ed40 msgr2=0x7f779806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:47.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.754+0000 7f779f3ff700 1 -- 172.21.15.27:0/1842865478 shutdown_connections 2024-01-11T09:47:47.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.754+0000 7f779f3ff700 1 -- 172.21.15.27:0/1842865478 wait complete. 2024-01-11T09:47:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.756+0000 7f779f3ff700 1 Processor -- start 2024-01-11T09:47:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.756+0000 7f779f3ff700 1 -- start start 2024-01-11T09:47:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.757+0000 7f779f3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77980b03e0 0x7f779814da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.757+0000 7f779f3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77980b2b90 0x7f77980d0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.757+0000 7f779f3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77980cc410 0x7f77980c9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.757+0000 7f779f3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77980cfba0 con 0x7f77980cc410 2024-01-11T09:47:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.757+0000 7f779f3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77981696c0 con 0x7f77980b03e0 2024-01-11T09:47:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.757+0000 7f779f3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779814df40 con 0x7f77980b2b90 2024-01-11T09:47:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.757+0000 7f779d99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77980cc410 0x7f77980c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.757+0000 7f779d99c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77980cc410 0x7f77980c9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60570/0 (socket says 172.21.15.27:60570) 2024-01-11T09:47:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.757+0000 7f779d99c700 1 -- 172.21.15.27:0/1048546070 learned_addr learned my addr 172.21.15.27:0/1048546070 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.757+0000 7f779d19b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77980b03e0 0x7f779814da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.757+0000 7f779c99a700 1 --2- 172.21.15.27:0/1048546070 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77980b2b90 0x7f77980d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.758+0000 7f779c99a700 1 -- 172.21.15.27:0/1048546070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77980b03e0 msgr2=0x7f779814da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.758+0000 7f779c99a700 1 --2- 172.21.15.27:0/1048546070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77980b03e0 0x7f779814da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.758+0000 7f779c99a700 1 -- 172.21.15.27:0/1048546070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77980cc410 msgr2=0x7f77980c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.758+0000 7f779c99a700 1 --2- 172.21.15.27:0/1048546070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77980cc410 0x7f77980c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.758+0000 7f779c99a700 1 -- 172.21.15.27:0/1048546070 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7790009a20 con 0x7f77980b2b90 2024-01-11T09:47:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.758+0000 7f779c99a700 1 --2- 172.21.15.27:0/1048546070 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77980b2b90 0x7f77980d0950 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f778800c560 tx=0x7f7788007d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.758+0000 7f778d7fa700 1 -- 172.21.15.27:0/1048546070 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7788011590 con 0x7f77980b2b90 2024-01-11T09:47:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.759+0000 7f778d7fa700 1 -- 172.21.15.27:0/1048546070 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77880116f0 con 0x7f77980b2b90 2024-01-11T09:47:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.759+0000 7f779f3ff700 1 -- 172.21.15.27:0/1048546070 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77980cf020 con 0x7f77980b2b90 2024-01-11T09:47:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.759+0000 7f778d7fa700 1 -- 172.21.15.27:0/1048546070 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7788011ab0 con 0x7f77980b2b90 2024-01-11T09:47:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.759+0000 7f779f3ff700 1 -- 172.21.15.27:0/1048546070 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77980cf5e0 con 0x7f77980b2b90 2024-01-11T09:47:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.760+0000 7f779f3ff700 1 -- 172.21.15.27:0/1048546070 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7798169870 con 0x7f77980b2b90 2024-01-11T09:47:47.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.761+0000 7f778d7fa700 1 -- 172.21.15.27:0/1048546070 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f778801c460 con 0x7f77980b2b90 2024-01-11T09:47:47.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.761+0000 7f778d7fa700 1 --2- 172.21.15.27:0/1048546070 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f777c0670e0 0x7f777c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:47.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.762+0000 7f778d7fa700 1 -- 172.21.15.27:0/1048546070 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6247+0+0 (secure 0 0 0) 0x7f7788084650 con 0x7f77980b2b90 2024-01-11T09:47:47.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.762+0000 7f779d19b700 1 -- 172.21.15.27:0/1048546070 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f777c0670e0 msgr2=0x7f777c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:47.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.762+0000 7f779d19b700 1 --2- 172.21.15.27:0/1048546070 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f777c0670e0 0x7f777c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:47.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.766+0000 7f778d7fa700 1 -- 172.21.15.27:0/1048546070 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7798169870 con 0x7f77980b2b90 2024-01-11T09:47:47.772 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:47.781+0000 7f40ad6511c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T09:47:47.774 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:47.783+0000 7fef03e9e1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:47:47.875 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:47.884+0000 7f40ad6511c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T09:47:47.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.963+0000 7f779d19b700 1 -- 172.21.15.27:0/1048546070 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f777c0670e0 msgr2=0x7f777c069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:47.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.963+0000 7f779d19b700 1 --2- 172.21.15.27:0/1048546070 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f777c0670e0 0x7f777c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:47.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.971+0000 7f779f3ff700 1 -- 172.21.15.27:0/1048546070 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77980b1440 con 0x7f77980b2b90 2024-01-11T09:47:47.963 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:47.972+0000 7fef03e9e1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:47:47.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.972+0000 7f778d7fa700 1 -- 172.21.15.27:0/1048546070 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v257) v1 ==== 74+0+16107 (secure 0 0 0) 0x7f7788051420 con 0x7f77980b2b90 2024-01-11T09:47:47.964 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:47.964 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":257,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:47:40.221837+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:47:22.374454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-01-11T09:47:23.471379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/151121862":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/3962343616":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/710006972":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/2450284170":"2024-01-12T09:47:40.221791+0000","172.21.15.77:6837/59776":"2024-01-12T09:47:40.221791+0000","172.21.15.77:6836/59776":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/447393830":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/1295621369":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/1441504549":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/1838517028":"2024-01-12T09:47:40.221791+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:47:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.974+0000 7f779f3ff700 1 -- 172.21.15.27:0/1048546070 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f777c0670e0 msgr2=0x7f777c069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.974+0000 7f779f3ff700 1 --2- 172.21.15.27:0/1048546070 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f777c0670e0 0x7f777c069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.975+0000 7f779f3ff700 1 -- 172.21.15.27:0/1048546070 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77980b2b90 msgr2=0x7f77980d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.975+0000 7f779f3ff700 1 --2- 172.21.15.27:0/1048546070 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77980b2b90 0x7f77980d0950 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f778800c560 tx=0x7f7788007d50 comp rx=0 tx=0).stop 2024-01-11T09:47:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.975+0000 7f779f3ff700 1 -- 172.21.15.27:0/1048546070 shutdown_connections 2024-01-11T09:47:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.975+0000 7f779f3ff700 1 --2- 172.21.15.27:0/1048546070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77980cc410 0x7f77980c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.975+0000 7f779f3ff700 1 --2- 172.21.15.27:0/1048546070 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f777c0670e0 0x7f777c069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.975+0000 7f779f3ff700 1 --2- 172.21.15.27:0/1048546070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77980b03e0 0x7f779814da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.975+0000 7f779f3ff700 1 --2- 172.21.15.27:0/1048546070 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f77980b2b90 0x7f77980d0950 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.975+0000 7f779f3ff700 1 -- 172.21.15.27:0/1048546070 >> 172.21.15.27:0/1048546070 conn(0x7f779806ed40 msgr2=0x7f77980b2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.975+0000 7f779f3ff700 1 -- 172.21.15.27:0/1048546070 shutdown_connections 2024-01-11T09:47:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:47.975+0000 7f779f3ff700 1 -- 172.21.15.27:0/1048546070 wait complete. 2024-01-11T09:47:47.979 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:47.988+0000 7f40ad6511c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T09:47:47.987 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-11T09:47:47.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:47:48.064 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:48.073+0000 7fef03e9e1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:47:48.087 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:48.096+0000 7f40ad6511c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T09:47:48.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.199+0000 7f7101dff700 1 Processor -- start 2024-01-11T09:47:48.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.200+0000 7f7101dff700 1 -- start start 2024-01-11T09:47:48.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.201+0000 7f7101dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f70fc1575e0 0x7f70fc158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:48.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.201+0000 7f7101dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f70fc0b19d0 0x7f70fc159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:48.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.201+0000 7f7101dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70fc158b60 con 0x7f70fc06b610 2024-01-11T09:47:48.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.201+0000 7f7101dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70fc0cb5e0 con 0x7f70fc0b19d0 2024-01-11T09:47:48.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.201+0000 7f7101dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70fc1599b0 con 0x7f70fc1575e0 2024-01-11T09:47:48.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.201+0000 7f70faffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f70fc06b610 0x7f70fc0b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37794/0 (socket says 172.21.15.27:37794) 2024-01-11T09:47:48.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.201+0000 7f70faffd700 1 -- 172.21.15.27:0/2367645143 learned_addr learned my addr 172.21.15.27:0/2367645143 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:48.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.201+0000 7f70fbfff700 1 --2- 172.21.15.27:0/2367645143 >> v2:172.21.15.27:3300/0 conn(0x7f70fc0b19d0 0x7f70fc159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:48.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.201+0000 7f70fb7fe700 1 --2- 172.21.15.27:0/2367645143 >> v2:172.21.15.77:3300/0 conn(0x7f70fc1575e0 0x7f70fc158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:48.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.202+0000 7f70fa7fc700 1 -- 172.21.15.27:0/2367645143 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271575880 0 0) 0x7f70f0001110 con 0x7f70fc06b610 2024-01-11T09:47:48.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.202+0000 7f70fa7fc700 1 -- 172.21.15.27:0/2367645143 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70e4003570 con 0x7f70fc06b610 2024-01-11T09:47:48.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.202+0000 7f70fb7fe700 1 -- 172.21.15.27:0/2367645143 >> v2:172.21.15.27:3300/0 conn(0x7f70fc0b19d0 msgr2=0x7f70fc159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:48.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.202+0000 7f70fb7fe700 1 --2- 172.21.15.27:0/2367645143 >> v2:172.21.15.27:3300/0 conn(0x7f70fc0b19d0 0x7f70fc159e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:48.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.202+0000 7f70fb7fe700 1 -- 172.21.15.27:0/2367645143 >> v1:172.21.15.27:6789/0 conn(0x7f70fc06b610 legacy=0x7f70fc0b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:48.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.202+0000 7f70fb7fe700 1 -- 172.21.15.27:0/2367645143 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc0c0f90 con 0x7f70fc1575e0 2024-01-11T09:47:48.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.202+0000 7f70fbfff700 1 --2- 172.21.15.27:0/2367645143 >> v2:172.21.15.27:3300/0 conn(0x7f70fc0b19d0 0x7f70fc159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:48.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.202+0000 7f70fb7fe700 1 --2- 172.21.15.27:0/2367645143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70fc1575e0 0x7f70fc158620 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f70fc064b00 tx=0x7f70ec0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14f4a26f3dd659e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:48.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.202+0000 7f70fa7fc700 1 -- 172.21.15.27:0/2367645143 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec011630 con 0x7f70fc1575e0 2024-01-11T09:47:48.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.203+0000 7f70fa7fc700 1 -- 172.21.15.27:0/2367645143 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ec0117b0 con 0x7f70fc1575e0 2024-01-11T09:47:48.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.203+0000 7f70fa7fc700 1 -- 172.21.15.27:0/2367645143 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec011b70 con 0x7f70fc1575e0 2024-01-11T09:47:48.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.203+0000 7f7101dff700 1 -- 172.21.15.27:0/2367645143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70fc1575e0 msgr2=0x7f70fc158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:48.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.203+0000 7f7101dff700 1 --2- 172.21.15.27:0/2367645143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70fc1575e0 0x7f70fc158620 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f70fc064b00 tx=0x7f70ec0091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:48.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.203+0000 7f7101dff700 1 -- 172.21.15.27:0/2367645143 shutdown_connections 2024-01-11T09:47:48.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.203+0000 7f7101dff700 1 --2- 172.21.15.27:0/2367645143 >> v2:172.21.15.27:3300/0 conn(0x7f70fc0b19d0 0x7f70fc159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:48.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.203+0000 7f7101dff700 1 --2- 172.21.15.27:0/2367645143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70fc1575e0 0x7f70fc158620 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:48.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.203+0000 7f7101dff700 1 -- 172.21.15.27:0/2367645143 >> 172.21.15.27:0/2367645143 conn(0x7f70fc0680e0 msgr2=0x7f70fc069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:48.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.203+0000 7f7101dff700 1 -- 172.21.15.27:0/2367645143 shutdown_connections 2024-01-11T09:47:48.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.203+0000 7f7101dff700 1 -- 172.21.15.27:0/2367645143 wait complete. 2024-01-11T09:47:48.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.205+0000 7f7101dff700 1 Processor -- start 2024-01-11T09:47:48.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.205+0000 7f7101dff700 1 -- start start 2024-01-11T09:47:48.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.206+0000 7f7101dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70fc06b610 0x7f70fc152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:48.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.206+0000 7f7101dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70fc0b19d0 0x7f70fc16c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:48.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.206+0000 7f7101dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70fc1575e0 0x7f70fc16cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:48.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.206+0000 7f7101dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc0b00f0 con 0x7f70fc1575e0 2024-01-11T09:47:48.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.206+0000 7f7101dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc0c6090 con 0x7f70fc0b19d0 2024-01-11T09:47:48.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.206+0000 7f7101dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc0c5760 con 0x7f70fc06b610 2024-01-11T09:47:48.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.206+0000 7f70fbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70fc1575e0 0x7f70fc16cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:48.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.206+0000 7f70fbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70fc1575e0 0x7f70fc16cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60592/0 (socket says 172.21.15.27:60592) 2024-01-11T09:47:48.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.206+0000 7f70fbfff700 1 -- 172.21.15.27:0/3979726478 learned_addr learned my addr 172.21.15.27:0/3979726478 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:48.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.206+0000 7f70faffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70fc0b19d0 0x7f70fc16c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:48.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.206+0000 7f70fb7fe700 1 --2- 172.21.15.27:0/3979726478 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70fc06b610 0x7f70fc152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:48.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.206+0000 7f70fbfff700 1 -- 172.21.15.27:0/3979726478 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70fc06b610 msgr2=0x7f70fc152a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:48.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.207+0000 7f70fbfff700 1 --2- 172.21.15.27:0/3979726478 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70fc06b610 0x7f70fc152a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:48.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.207+0000 7f70fbfff700 1 -- 172.21.15.27:0/3979726478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70fc0b19d0 msgr2=0x7f70fc16c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:48.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.207+0000 7f70fbfff700 1 --2- 172.21.15.27:0/3979726478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70fc0b19d0 0x7f70fc16c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:48.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.207+0000 7f70fbfff700 1 -- 172.21.15.27:0/3979726478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f4009a20 con 0x7f70fc1575e0 2024-01-11T09:47:48.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.207+0000 7f70faffd700 1 --2- 172.21.15.27:0/3979726478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70fc0b19d0 0x7f70fc16c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:48.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.207+0000 7f70fbfff700 1 --2- 172.21.15.27:0/3979726478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70fc1575e0 0x7f70fc16cc20 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f70f4009fc0 tx=0x7f70f400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:48.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.207+0000 7f70e37fe700 1 -- 172.21.15.27:0/3979726478 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f400cbd0 con 0x7f70fc1575e0 2024-01-11T09:47:48.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.207+0000 7f70e37fe700 1 -- 172.21.15.27:0/3979726478 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f400f040 con 0x7f70fc1575e0 2024-01-11T09:47:48.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.207+0000 7f7101dff700 1 -- 172.21.15.27:0/3979726478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70ec009420 con 0x7f70fc1575e0 2024-01-11T09:47:48.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.207+0000 7f70e37fe700 1 -- 172.21.15.27:0/3979726478 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f4014480 con 0x7f70fc1575e0 2024-01-11T09:47:48.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.207+0000 7f7101dff700 1 -- 172.21.15.27:0/3979726478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70fc0c5d20 con 0x7f70fc1575e0 2024-01-11T09:47:48.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.208+0000 7f7101dff700 1 -- 172.21.15.27:0/3979726478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70fc0c6220 con 0x7f70fc1575e0 2024-01-11T09:47:48.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.210+0000 7f70e37fe700 1 -- 172.21.15.27:0/3979726478 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f70f4003020 con 0x7f70fc1575e0 2024-01-11T09:47:48.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.211+0000 7f70e37fe700 1 --2- 172.21.15.27:0/3979726478 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f70e80670e0 0x7f70e8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:48.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.211+0000 7f70e37fe700 1 -- 172.21.15.27:0/3979726478 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6247+0+0 (secure 0 0 0) 0x7f70f4085500 con 0x7f70fc1575e0 2024-01-11T09:47:48.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.211+0000 7f70fb7fe700 1 -- 172.21.15.27:0/3979726478 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f70e80670e0 msgr2=0x7f70e8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:48.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.211+0000 7f70fb7fe700 1 --2- 172.21.15.27:0/3979726478 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f70e80670e0 0x7f70e8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:48.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.215+0000 7f70e37fe700 1 -- 172.21.15.27:0/3979726478 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f70f40521c0 con 0x7f70fc1575e0 2024-01-11T09:47:48.294 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:48.303+0000 7f40ad6511c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T09:47:48.400 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:48.409+0000 7f40ad6511c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T09:47:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.412+0000 7f70fb7fe700 1 -- 172.21.15.27:0/3979726478 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f70e80670e0 msgr2=0x7f70e8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.412+0000 7f70fb7fe700 1 --2- 172.21.15.27:0/3979726478 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f70e80670e0 0x7f70e8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:48.407 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:48.417+0000 7fef03e9e1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:47:48.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.467+0000 7f7101dff700 1 -- 172.21.15.27:0/3979726478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70fc0718d0 con 0x7f70fc1575e0 2024-01-11T09:47:48.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.468+0000 7f70e37fe700 1 -- 172.21.15.27:0/3979726478 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+4612 (secure 0 0 0) 0x7f70f40557e0 con 0x7f70fc1575e0 2024-01-11T09:47:48.459 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:48.460 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":141,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9866,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1060058066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1060058066},{"type":"v1","addr":"172.21.15.77:6835","nonce":1060058066}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137},{"gid":11626,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/4169546841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":4169546841},{"type":"v1","addr":"172.21.15.77:6834","nonce":4169546841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137},{"gid":11632,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/2567971522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2567971522},{"type":"v1","addr":"172.21.15.27:6833","nonce":2567971522}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:47:24.474082+0000","modified":"2024-01-11T09:47:27.985712+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12063},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12063":{"gid":12063,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1065634442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1065634442},{"type":"v1","addr":"172.21.15.27:6835","nonce":1065634442}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2024-01-11T09:47:48.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.470+0000 7f7101dff700 1 -- 172.21.15.27:0/3979726478 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f70e80670e0 msgr2=0x7f70e8069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:48.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.470+0000 7f7101dff700 1 --2- 172.21.15.27:0/3979726478 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f70e80670e0 0x7f70e8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:48.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.470+0000 7f7101dff700 1 -- 172.21.15.27:0/3979726478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70fc1575e0 msgr2=0x7f70fc16cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:48.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.471+0000 7f7101dff700 1 --2- 172.21.15.27:0/3979726478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70fc1575e0 0x7f70fc16cc20 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f70f4009fc0 tx=0x7f70f400b040 comp rx=0 tx=0).stop 2024-01-11T09:47:48.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.471+0000 7f7101dff700 1 -- 172.21.15.27:0/3979726478 shutdown_connections 2024-01-11T09:47:48.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.471+0000 7f7101dff700 1 --2- 172.21.15.27:0/3979726478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70fc1575e0 0x7f70fc16cc20 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:48.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.471+0000 7f7101dff700 1 --2- 172.21.15.27:0/3979726478 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70fc06b610 0x7f70fc152a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:48.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.471+0000 7f7101dff700 1 --2- 172.21.15.27:0/3979726478 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f70e80670e0 0x7f70e8069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:48.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.471+0000 7f7101dff700 1 --2- 172.21.15.27:0/3979726478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70fc0b19d0 0x7f70fc16c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:48.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.471+0000 7f7101dff700 1 -- 172.21.15.27:0/3979726478 >> 172.21.15.27:0/3979726478 conn(0x7f70fc0680e0 msgr2=0x7f70fc155e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:48.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.471+0000 7f7101dff700 1 -- 172.21.15.27:0/3979726478 shutdown_connections 2024-01-11T09:47:48.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.471+0000 7f7101dff700 1 -- 172.21.15.27:0/3979726478 wait complete. 2024-01-11T09:47:48.463 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 141 2024-01-11T09:47:48.481 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:47:48.497 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:48.506+0000 7fef03e9e1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:47:48.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.702+0000 7f50037ff700 1 Processor -- start 2024-01-11T09:47:48.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.702+0000 7f50037ff700 1 -- start start 2024-01-11T09:47:48.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.703+0000 7f50037ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4ffc0cdb20 0x7f4ffc140550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:48.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.703+0000 7f50037ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ffc15bc30 con 0x7f4ffc06c060 2024-01-11T09:47:48.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.703+0000 7f50037ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ffc0ccfc0 con 0x7f4ffc0c8c70 2024-01-11T09:47:48.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.703+0000 7f50037ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ffc158c40 con 0x7f4ffc0cdb20 2024-01-11T09:47:48.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.704+0000 7f5000d9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4ffc06c060 0x7f4ffc0cd410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37804/0 (socket says 172.21.15.27:37804) 2024-01-11T09:47:48.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.704+0000 7f5000d9a700 1 -- 172.21.15.27:0/3839637096 learned_addr learned my addr 172.21.15.27:0/3839637096 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:48.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.704+0000 7f5001d9c700 1 --2- 172.21.15.27:0/3839637096 >> v2:172.21.15.77:3300/0 conn(0x7f4ffc0cdb20 0x7f4ffc140550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:48.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.704+0000 7f4ff3fff700 1 -- 172.21.15.27:0/3839637096 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791857334 0 0) 0x7f4ffc15bc30 con 0x7f4ffc06c060 2024-01-11T09:47:48.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.704+0000 7f4ff3fff700 1 -- 172.21.15.27:0/3839637096 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe8003570 con 0x7f4ffc06c060 2024-01-11T09:47:48.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.704+0000 7f5001d9c700 1 -- 172.21.15.27:0/3839637096 >> v1:172.21.15.77:6789/0 conn(0x7f4ffc0c8c70 legacy=0x7f4ffc06b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:48.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.704+0000 7f5001d9c700 1 -- 172.21.15.27:0/3839637096 >> v1:172.21.15.27:6789/0 conn(0x7f4ffc06c060 legacy=0x7f4ffc0cd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:48.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.704+0000 7f5001d9c700 1 -- 172.21.15.27:0/3839637096 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc06c5a0 con 0x7f4ffc0cdb20 2024-01-11T09:47:48.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.705+0000 7f5001d9c700 1 --2- 172.21.15.27:0/3839637096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ffc0cdb20 0x7f4ffc140550 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4009e60 tx=0x7f4ff4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83d727b535856c19 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:48.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.705+0000 7f4ff3fff700 1 -- 172.21.15.27:0/3839637096 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff400e070 con 0x7f4ffc0cdb20 2024-01-11T09:47:48.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.705+0000 7f4ff3fff700 1 -- 172.21.15.27:0/3839637096 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff400bb60 con 0x7f4ffc0cdb20 2024-01-11T09:47:48.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.705+0000 7f4ff3fff700 1 -- 172.21.15.27:0/3839637096 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4012430 con 0x7f4ffc0cdb20 2024-01-11T09:47:48.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.705+0000 7f50037ff700 1 -- 172.21.15.27:0/3839637096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ffc0cdb20 msgr2=0x7f4ffc140550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:48.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.705+0000 7f50037ff700 1 --2- 172.21.15.27:0/3839637096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ffc0cdb20 0x7f4ffc140550 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4009e60 tx=0x7f4ff4009780 comp rx=0 tx=0).stop 2024-01-11T09:47:48.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.706+0000 7f50037ff700 1 -- 172.21.15.27:0/3839637096 shutdown_connections 2024-01-11T09:47:48.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.706+0000 7f50037ff700 1 --2- 172.21.15.27:0/3839637096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ffc0cdb20 0x7f4ffc140550 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:48.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.706+0000 7f50037ff700 1 -- 172.21.15.27:0/3839637096 >> 172.21.15.27:0/3839637096 conn(0x7f4ffc0ce3e0 msgr2=0x7f4ffc0ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:48.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.706+0000 7f50037ff700 1 -- 172.21.15.27:0/3839637096 shutdown_connections 2024-01-11T09:47:48.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.706+0000 7f50037ff700 1 -- 172.21.15.27:0/3839637096 wait complete. 2024-01-11T09:47:48.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.707+0000 7f50037ff700 1 Processor -- start 2024-01-11T09:47:48.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.707+0000 7f50037ff700 1 -- start start 2024-01-11T09:47:48.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.708+0000 7f50037ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ffc06c060 0x7f4ffc0b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:48.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.708+0000 7f50037ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4ffc0c8c70 0x7f4ffc0b1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:48.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.708+0000 7f50037ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ffc0cdb20 0x7f4ffc13f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.708+0000 7f50037ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc0c5c20 con 0x7f4ffc0cdb20 2024-01-11T09:47:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.708+0000 7f50037ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc0c5f00 con 0x7f4ffc06c060 2024-01-11T09:47:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.708+0000 7f50037ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc0c61e0 con 0x7f4ffc0c8c70 2024-01-11T09:47:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.709+0000 7f5001d9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ffc0cdb20 0x7f4ffc13f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.709+0000 7f500159b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ffc06c060 0x7f4ffc0b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.709+0000 7f5001d9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ffc0cdb20 0x7f4ffc13f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60594/0 (socket says 172.21.15.27:60594) 2024-01-11T09:47:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.709+0000 7f5001d9c700 1 -- 172.21.15.27:0/2078373037 learned_addr learned my addr 172.21.15.27:0/2078373037 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.709+0000 7f5000d9a700 1 --2- 172.21.15.27:0/2078373037 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4ffc0c8c70 0x7f4ffc0b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.709+0000 7f5001d9c700 1 -- 172.21.15.27:0/2078373037 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4ffc0c8c70 msgr2=0x7f4ffc0b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.709+0000 7f5001d9c700 1 --2- 172.21.15.27:0/2078373037 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4ffc0c8c70 0x7f4ffc0b1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.709+0000 7f5001d9c700 1 -- 172.21.15.27:0/2078373037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ffc06c060 msgr2=0x7f4ffc0b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.709+0000 7f5001d9c700 1 --2- 172.21.15.27:0/2078373037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ffc06c060 0x7f4ffc0b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.709+0000 7f5001d9c700 1 -- 172.21.15.27:0/2078373037 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff4009a20 con 0x7f4ffc0cdb20 2024-01-11T09:47:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.709+0000 7f500159b700 1 --2- 172.21.15.27:0/2078373037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ffc06c060 0x7f4ffc0b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:48.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.709+0000 7f5001d9c700 1 --2- 172.21.15.27:0/2078373037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ffc0cdb20 0x7f4ffc13f9b0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc064980 tx=0x7f4ff4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:48.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.710+0000 7f4ff17fa700 1 -- 172.21.15.27:0/2078373037 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4027470 con 0x7f4ffc0cdb20 2024-01-11T09:47:48.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.710+0000 7f4ff17fa700 1 -- 172.21.15.27:0/2078373037 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff40275f0 con 0x7f4ffc0cdb20 2024-01-11T09:47:48.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.710+0000 7f4ff17fa700 1 -- 172.21.15.27:0/2078373037 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff40279b0 con 0x7f4ffc0cdb20 2024-01-11T09:47:48.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.710+0000 7f50037ff700 1 -- 172.21.15.27:0/2078373037 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ffc0c64c0 con 0x7f4ffc0cdb20 2024-01-11T09:47:48.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.710+0000 7f50037ff700 1 -- 172.21.15.27:0/2078373037 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ffc0c5880 con 0x7f4ffc0cdb20 2024-01-11T09:47:48.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.711+0000 7f50037ff700 1 -- 172.21.15.27:0/2078373037 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ffc0718d0 con 0x7f4ffc0cdb20 2024-01-11T09:47:48.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.713+0000 7f4ff17fa700 1 -- 172.21.15.27:0/2078373037 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f4ff4024070 con 0x7f4ffc0cdb20 2024-01-11T09:47:48.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.713+0000 7f4ff17fa700 1 --2- 172.21.15.27:0/2078373037 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4fe00670e0 0x7f4fe0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:48.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.714+0000 7f4ff17fa700 1 -- 172.21.15.27:0/2078373037 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6247+0+0 (secure 0 0 0) 0x7f4ff4087420 con 0x7f4ffc0cdb20 2024-01-11T09:47:48.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.714+0000 7f500159b700 1 -- 172.21.15.27:0/2078373037 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4fe00670e0 msgr2=0x7f4fe0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:48.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.714+0000 7f500159b700 1 --2- 172.21.15.27:0/2078373037 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4fe00670e0 0x7f4fe0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:48.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.718+0000 7f4ff17fa700 1 -- 172.21.15.27:0/2078373037 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4ff40541f0 con 0x7f4ffc0cdb20 2024-01-11T09:47:48.779 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:48.788+0000 7f40ad6511c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T09:47:48.875 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:48.883+0000 7f40ad6511c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T09:47:48.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.915+0000 7f500159b700 1 -- 172.21.15.27:0/2078373037 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4fe00670e0 msgr2=0x7f4fe0069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:48.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.915+0000 7f500159b700 1 --2- 172.21.15.27:0/2078373037 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4fe00670e0 0x7f4fe0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:48.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.970+0000 7f50037ff700 1 -- 172.21.15.27:0/2078373037 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ffc15b300 con 0x7f4ffc0cdb20 2024-01-11T09:47:48.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.971+0000 7f4ff17fa700 1 -- 172.21.15.27:0/2078373037 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+4612 (secure 0 0 0) 0x7f4ff4057810 con 0x7f4ffc0cdb20 2024-01-11T09:47:48.962 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:48.963 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":141,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9866,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1060058066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1060058066},{"type":"v1","addr":"172.21.15.77:6835","nonce":1060058066}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137},{"gid":11626,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/4169546841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":4169546841},{"type":"v1","addr":"172.21.15.77:6834","nonce":4169546841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137},{"gid":11632,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/2567971522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2567971522},{"type":"v1","addr":"172.21.15.27:6833","nonce":2567971522}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-11T09:47:24.474082+0000","modified":"2024-01-11T09:47:27.985712+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12063},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12063":{"gid":12063,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1065634442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1065634442},{"type":"v1","addr":"172.21.15.27:6835","nonce":1065634442}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2024-01-11T09:47:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.973+0000 7f50037ff700 1 -- 172.21.15.27:0/2078373037 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4fe00670e0 msgr2=0x7f4fe0069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.973+0000 7f50037ff700 1 --2- 172.21.15.27:0/2078373037 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4fe00670e0 0x7f4fe0069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.973+0000 7f50037ff700 1 -- 172.21.15.27:0/2078373037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ffc0cdb20 msgr2=0x7f4ffc13f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.973+0000 7f50037ff700 1 --2- 172.21.15.27:0/2078373037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ffc0cdb20 0x7f4ffc13f9b0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc064980 tx=0x7f4ff4019000 comp rx=0 tx=0).stop 2024-01-11T09:47:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.974+0000 7f50037ff700 1 -- 172.21.15.27:0/2078373037 shutdown_connections 2024-01-11T09:47:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.974+0000 7f50037ff700 1 --2- 172.21.15.27:0/2078373037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ffc0cdb20 0x7f4ffc13f9b0 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.974+0000 7f50037ff700 1 --2- 172.21.15.27:0/2078373037 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4fe00670e0 0x7f4fe0069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.974+0000 7f50037ff700 1 --2- 172.21.15.27:0/2078373037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ffc06c060 0x7f4ffc0b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.974+0000 7f50037ff700 1 --2- 172.21.15.27:0/2078373037 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4ffc0c8c70 0x7f4ffc0b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.974+0000 7f50037ff700 1 -- 172.21.15.27:0/2078373037 >> 172.21.15.27:0/2078373037 conn(0x7f4ffc0ce3e0 msgr2=0x7f4ffc0c3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.974+0000 7f50037ff700 1 -- 172.21.15.27:0/2078373037 shutdown_connections 2024-01-11T09:47:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:48.974+0000 7f50037ff700 1 -- 172.21.15.27:0/2078373037 wait complete. 2024-01-11T09:47:48.967 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 141 2024-01-11T09:47:48.986 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:47:49.042 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:49.051+0000 7fef03e9e1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:47:49.091 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:47:49.096 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:47:49.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.199+0000 7fac0afff700 1 Processor -- start 2024-01-11T09:47:49.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.200+0000 7fac0afff700 1 -- start start 2024-01-11T09:47:49.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.201+0000 7fac0afff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac041575e0 0x7fac04158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:49.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.201+0000 7fac0afff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fac0406b610 0x7fac040b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:49.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.201+0000 7fac0afff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac04158b60 con 0x7fac040b1800 2024-01-11T09:47:49.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.201+0000 7fac0afff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac040cb5e0 con 0x7fac0406b610 2024-01-11T09:47:49.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.201+0000 7fac0afff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac040b1c80 con 0x7fac041575e0 2024-01-11T09:47:49.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.201+0000 7fac08d9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac041575e0 0x7fac04158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:49.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.201+0000 7fac03fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fac0406b610 0x7fac040b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:49.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.202+0000 7fac0959c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fac040b1800 0x7fac040ca1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:34262/0 (socket says 172.21.15.27:34262) 2024-01-11T09:47:49.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.202+0000 7fac0959c700 1 -- 172.21.15.27:0/629292659 learned_addr learned my addr 172.21.15.27:0/629292659 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:49.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.202+0000 7fac037fe700 1 -- 172.21.15.27:0/629292659 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277676404 0 0) 0x7fabfc001490 con 0x7fac040b1800 2024-01-11T09:47:49.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.202+0000 7fac08d9b700 1 -- 172.21.15.27:0/629292659 >> v2:172.21.15.77:3300/0 conn(0x7fac0406b610 msgr2=0x7fac040b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:49.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.202+0000 7fac08d9b700 1 --2- 172.21.15.27:0/629292659 >> v2:172.21.15.77:3300/0 conn(0x7fac0406b610 0x7fac040b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.202+0000 7fac08d9b700 1 -- 172.21.15.27:0/629292659 >> v1:172.21.15.77:6789/0 conn(0x7fac040b1800 legacy=0x7fac040ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:49.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.202+0000 7fac08d9b700 1 -- 172.21.15.27:0/629292659 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac041599b0 con 0x7fac041575e0 2024-01-11T09:47:49.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.202+0000 7fac03fff700 1 --2- 172.21.15.27:0/629292659 >> v2:172.21.15.77:3300/0 conn(0x7fac0406b610 0x7fac040b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:49.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.202+0000 7fac08d9b700 1 --2- 172.21.15.27:0/629292659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac041575e0 0x7fac04158620 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fabf80098e0 tx=0x7fabf80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20336e05202b00c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:49.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.203+0000 7fac037fe700 1 -- 172.21.15.27:0/629292659 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf800b640 con 0x7fac041575e0 2024-01-11T09:47:49.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.203+0000 7fac037fe700 1 -- 172.21.15.27:0/629292659 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf800b7a0 con 0x7fac041575e0 2024-01-11T09:47:49.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.203+0000 7fac037fe700 1 -- 172.21.15.27:0/629292659 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf800bb40 con 0x7fac041575e0 2024-01-11T09:47:49.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.203+0000 7fac0afff700 1 -- 172.21.15.27:0/629292659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac041575e0 msgr2=0x7fac04158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:49.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.203+0000 7fac0afff700 1 --2- 172.21.15.27:0/629292659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac041575e0 0x7fac04158620 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fabf80098e0 tx=0x7fabf80091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.203+0000 7fac0afff700 1 -- 172.21.15.27:0/629292659 shutdown_connections 2024-01-11T09:47:49.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.203+0000 7fac0afff700 1 --2- 172.21.15.27:0/629292659 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac041575e0 0x7fac04158620 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.203+0000 7fac0afff700 1 --2- 172.21.15.27:0/629292659 >> v2:172.21.15.77:3300/0 conn(0x7fac0406b610 0x7fac040b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.203+0000 7fac0afff700 1 -- 172.21.15.27:0/629292659 >> 172.21.15.27:0/629292659 conn(0x7fac040680e0 msgr2=0x7fac04069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:49.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.203+0000 7fac0afff700 1 -- 172.21.15.27:0/629292659 shutdown_connections 2024-01-11T09:47:49.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.203+0000 7fac0afff700 1 -- 172.21.15.27:0/629292659 wait complete. 2024-01-11T09:47:49.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.205+0000 7fac0afff700 1 Processor -- start 2024-01-11T09:47:49.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.205+0000 7fac0afff700 1 -- start start 2024-01-11T09:47:49.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.206+0000 7fac0afff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac0406b610 0x7fac0414d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:49.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.206+0000 7fac0afff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac040b1800 0x7fac0414d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:49.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.206+0000 7fac0afff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac041575e0 0x7fac0415d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:49.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.206+0000 7fac0afff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0415e0b0 con 0x7fac041575e0 2024-01-11T09:47:49.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.206+0000 7fac0afff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0414dae0 con 0x7fac040b1800 2024-01-11T09:47:49.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.206+0000 7fac0afff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac0406e7a0 con 0x7fac0406b610 2024-01-11T09:47:49.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.206+0000 7fac08d9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac0406b610 0x7fac0414d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:49.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.206+0000 7fac08d9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac0406b610 0x7fac0414d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:51470/0 (socket says 172.21.15.27:51470) 2024-01-11T09:47:49.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.207+0000 7fac08d9b700 1 -- 172.21.15.27:0/2200278137 learned_addr learned my addr 172.21.15.27:0/2200278137 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:49.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.207+0000 7fac0959c700 1 --2- 172.21.15.27:0/2200278137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac041575e0 0x7fac0415d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:49.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.207+0000 7fac03fff700 1 --2- 172.21.15.27:0/2200278137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac040b1800 0x7fac0414d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:49.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.207+0000 7fac08d9b700 1 -- 172.21.15.27:0/2200278137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac040b1800 msgr2=0x7fac0414d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:49.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.207+0000 7fac08d9b700 1 --2- 172.21.15.27:0/2200278137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac040b1800 0x7fac0414d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.207+0000 7fac08d9b700 1 -- 172.21.15.27:0/2200278137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac041575e0 msgr2=0x7fac0415d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:49.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.207+0000 7fac08d9b700 1 --2- 172.21.15.27:0/2200278137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac041575e0 0x7fac0415d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.207+0000 7fac08d9b700 1 -- 172.21.15.27:0/2200278137 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf0009420 con 0x7fac0406b610 2024-01-11T09:47:49.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.207+0000 7fac08d9b700 1 --2- 172.21.15.27:0/2200278137 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac0406b610 0x7fac0414d060 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fac040aed30 tx=0x7fabf8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:49.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.207+0000 7fac00ff9700 1 -- 172.21.15.27:0/2200278137 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf800bb20 con 0x7fac0406b610 2024-01-11T09:47:49.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.208+0000 7fac00ff9700 1 -- 172.21.15.27:0/2200278137 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf8000c90 con 0x7fac0406b610 2024-01-11T09:47:49.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.208+0000 7fac00ff9700 1 -- 172.21.15.27:0/2200278137 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf8011630 con 0x7fac0406b610 2024-01-11T09:47:49.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.208+0000 7fac0afff700 1 -- 172.21.15.27:0/2200278137 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf8009420 con 0x7fac0406b610 2024-01-11T09:47:49.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.208+0000 7fac0afff700 1 -- 172.21.15.27:0/2200278137 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac0406ed60 con 0x7fac0406b610 2024-01-11T09:47:49.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.209+0000 7fac0afff700 1 -- 172.21.15.27:0/2200278137 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac040718d0 con 0x7fac0406b610 2024-01-11T09:47:49.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.210+0000 7fac00ff9700 1 -- 172.21.15.27:0/2200278137 <== mon.2 v2:172.21.15.77:3301/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7fabf80119a0 con 0x7fac0406b610 2024-01-11T09:47:49.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.211+0000 7fac00ff9700 1 --2- 172.21.15.27:0/2200278137 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fabe80670e0 0x7fabe8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:49.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.211+0000 7fac00ff9700 1 -- 172.21.15.27:0/2200278137 <== mon.2 v2:172.21.15.77:3301/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6247+0+0 (secure 0 0 0) 0x7fabf80860f0 con 0x7fac0406b610 2024-01-11T09:47:49.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.211+0000 7fac03fff700 1 -- 172.21.15.27:0/2200278137 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fabe80670e0 msgr2=0x7fabe8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:49.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.211+0000 7fac03fff700 1 --2- 172.21.15.27:0/2200278137 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fabe80670e0 0x7fabe8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:49.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.214+0000 7fac00ff9700 1 -- 172.21.15.27:0/2200278137 <== mon.2 v2:172.21.15.77:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fabf8052ea0 con 0x7fac0406b610 2024-01-11T09:47:49.238 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:49.247+0000 7fef03e9e1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:47:49.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.412+0000 7fac03fff700 1 -- 172.21.15.27:0/2200278137 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fabe80670e0 msgr2=0x7fabe8069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:49.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.412+0000 7fac03fff700 1 --2- 172.21.15.27:0/2200278137 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fabe80670e0 0x7fabe8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.423+0000 7fac0afff700 1 -- 172.21.15.27:0/2200278137 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac04069b20 con 0x7fac0406b610 2024-01-11T09:47:49.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.424+0000 7fac00ff9700 1 -- 172.21.15.27:0/2200278137 <== mon.2 v2:172.21.15.77:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v257) v1 ==== 74+0+16107 (secure 0 0 0) 0x7fabf80564c0 con 0x7fac0406b610 2024-01-11T09:47:49.415 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:49.416 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":257,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:47:40.221837+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-01-11T09:47:22.374454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-01-11T09:47:23.471379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/151121862":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/3962343616":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/710006972":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/2450284170":"2024-01-12T09:47:40.221791+0000","172.21.15.77:6837/59776":"2024-01-12T09:47:40.221791+0000","172.21.15.77:6836/59776":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/447393830":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/1295621369":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/1441504549":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/1838517028":"2024-01-12T09:47:40.221791+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:47:49.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.428+0000 7fac0afff700 1 -- 172.21.15.27:0/2200278137 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fabe80670e0 msgr2=0x7fabe8069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:49.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.428+0000 7fac0afff700 1 --2- 172.21.15.27:0/2200278137 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fabe80670e0 0x7fabe8069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.428+0000 7fac0afff700 1 -- 172.21.15.27:0/2200278137 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac0406b610 msgr2=0x7fac0414d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:49.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.428+0000 7fac0afff700 1 --2- 172.21.15.27:0/2200278137 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac0406b610 0x7fac0414d060 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fac040aed30 tx=0x7fabf8017000 comp rx=0 tx=0).stop 2024-01-11T09:47:49.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.428+0000 7fac0afff700 1 -- 172.21.15.27:0/2200278137 shutdown_connections 2024-01-11T09:47:49.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.428+0000 7fac0afff700 1 --2- 172.21.15.27:0/2200278137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac041575e0 0x7fac0415d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.428+0000 7fac0afff700 1 --2- 172.21.15.27:0/2200278137 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fac0406b610 0x7fac0414d060 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.428+0000 7fac0afff700 1 --2- 172.21.15.27:0/2200278137 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fabe80670e0 0x7fabe8069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.428+0000 7fac0afff700 1 --2- 172.21.15.27:0/2200278137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac040b1800 0x7fac0414d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.428+0000 7fac0afff700 1 -- 172.21.15.27:0/2200278137 >> 172.21.15.27:0/2200278137 conn(0x7fac040680e0 msgr2=0x7fac04154a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:49.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.428+0000 7fac0afff700 1 -- 172.21.15.27:0/2200278137 shutdown_connections 2024-01-11T09:47:49.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.428+0000 7fac0afff700 1 -- 172.21.15.27:0/2200278137 wait complete. 2024-01-11T09:47:49.439 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-11T09:47:49.477 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:49.486+0000 7f40ad6511c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T09:47:49.618 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:49.627+0000 7fef03e9e1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:47:49.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.689+0000 7ff2d01ff700 1 Processor -- start 2024-01-11T09:47:49.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.689+0000 7ff2d01ff700 1 -- start start 2024-01-11T09:47:49.681 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:49.690+0000 7f40ad6511c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T09:47:49.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.691+0000 7ff2d01ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff2c806bf10 0x7ff2c8159c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:49.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.691+0000 7ff2d01ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2c8068720 0x7ff2c8068b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:49.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.691+0000 7ff2d01ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c815d290 con 0x7ff2c80b0a40 2024-01-11T09:47:49.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.691+0000 7ff2d01ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2c8069080 con 0x7ff2c806bf10 2024-01-11T09:47:49.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.691+0000 7ff2d01ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c815d570 con 0x7ff2c8068720 2024-01-11T09:47:49.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.691+0000 7ff2cdf9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff2c806bf10 0x7ff2c8159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:49.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.691+0000 7ff2cdf9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff2c806bf10 0x7ff2c8159c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60608/0 (socket says 172.21.15.27:60608) 2024-01-11T09:47:49.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.691+0000 7ff2cdf9b700 1 -- 172.21.15.27:0/3335785348 learned_addr learned my addr 172.21.15.27:0/3335785348 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:49.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.691+0000 7ff2ce79c700 1 --2- 172.21.15.27:0/3335785348 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2c8068720 0x7ff2c8068b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:49.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.691+0000 7ff2cdf9b700 1 -- 172.21.15.27:0/3335785348 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2c8068720 msgr2=0x7ff2c8068b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:49.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.691+0000 7ff2cdf9b700 1 --2- 172.21.15.27:0/3335785348 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2c8068720 0x7ff2c8068b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.691+0000 7ff2cdf9b700 1 -- 172.21.15.27:0/3335785348 >> v1:172.21.15.77:6789/0 conn(0x7ff2c80b0a40 legacy=0x7ff2c815da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:49.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.691+0000 7ff2cdf9b700 1 -- 172.21.15.27:0/3335785348 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c8154180 con 0x7ff2c806bf10 2024-01-11T09:47:49.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.691+0000 7ff2ccf99700 1 -- 172.21.15.27:0/3335785348 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1829842781 0 0) 0x7ff2c815d290 con 0x7ff2c80b0a40 2024-01-11T09:47:49.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.692+0000 7ff2cdf9b700 1 --2- 172.21.15.27:0/3335785348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c806bf10 0x7ff2c8159c50 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7ff2c40099b0 tx=0x7ff2c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab2b1fa68bac1665 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:49.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.692+0000 7ff2ccf99700 1 -- 172.21.15.27:0/3335785348 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4009ea0 con 0x7ff2c806bf10 2024-01-11T09:47:49.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.692+0000 7ff2ccf99700 1 -- 172.21.15.27:0/3335785348 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c400b860 con 0x7ff2c806bf10 2024-01-11T09:47:49.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.692+0000 7ff2ccf99700 1 -- 172.21.15.27:0/3335785348 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c400bc00 con 0x7ff2c806bf10 2024-01-11T09:47:49.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.693+0000 7ff2d01ff700 1 -- 172.21.15.27:0/3335785348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c806bf10 msgr2=0x7ff2c8159c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:49.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.693+0000 7ff2d01ff700 1 --2- 172.21.15.27:0/3335785348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c806bf10 0x7ff2c8159c50 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7ff2c40099b0 tx=0x7ff2c4009250 comp rx=0 tx=0).stop 2024-01-11T09:47:49.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.693+0000 7ff2d01ff700 1 -- 172.21.15.27:0/3335785348 shutdown_connections 2024-01-11T09:47:49.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.693+0000 7ff2d01ff700 1 --2- 172.21.15.27:0/3335785348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c806bf10 0x7ff2c8159c50 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.693+0000 7ff2d01ff700 1 --2- 172.21.15.27:0/3335785348 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2c8068720 0x7ff2c8068b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.693+0000 7ff2d01ff700 1 -- 172.21.15.27:0/3335785348 >> 172.21.15.27:0/3335785348 conn(0x7ff2c80ce3c0 msgr2=0x7ff2c80ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:49.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.693+0000 7ff2d01ff700 1 -- 172.21.15.27:0/3335785348 shutdown_connections 2024-01-11T09:47:49.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.693+0000 7ff2d01ff700 1 -- 172.21.15.27:0/3335785348 wait complete. 2024-01-11T09:47:49.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.694+0000 7ff2d01ff700 1 Processor -- start 2024-01-11T09:47:49.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.695+0000 7ff2d01ff700 1 -- start start 2024-01-11T09:47:49.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.695+0000 7ff2d01ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c8068720 0x7ff2c8155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:49.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.695+0000 7ff2d01ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2c806bf10 0x7ff2c8155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:49.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.695+0000 7ff2d01ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2c80b0a40 0x7ff2c80b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:49.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.695+0000 7ff2d01ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c8155d90 con 0x7ff2c8068720 2024-01-11T09:47:49.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.695+0000 7ff2d01ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c8156040 con 0x7ff2c806bf10 2024-01-11T09:47:49.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.695+0000 7ff2d01ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c8156320 con 0x7ff2c80b0a40 2024-01-11T09:47:49.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.696+0000 7ff2cdf9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c8068720 0x7ff2c8155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:49.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.696+0000 7ff2cdf9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c8068720 0x7ff2c8155310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60620/0 (socket says 172.21.15.27:60620) 2024-01-11T09:47:49.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.696+0000 7ff2cdf9b700 1 -- 172.21.15.27:0/2298959699 learned_addr learned my addr 172.21.15.27:0/2298959699 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:49.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.696+0000 7ff2cd79a700 1 --2- 172.21.15.27:0/2298959699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2c806bf10 0x7ff2c8155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:49.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.696+0000 7ff2ce79c700 1 --2- 172.21.15.27:0/2298959699 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2c80b0a40 0x7ff2c80b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:49.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.696+0000 7ff2cdf9b700 1 -- 172.21.15.27:0/2298959699 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2c80b0a40 msgr2=0x7ff2c80b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:49.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.696+0000 7ff2cdf9b700 1 --2- 172.21.15.27:0/2298959699 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2c80b0a40 0x7ff2c80b2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.696+0000 7ff2cdf9b700 1 -- 172.21.15.27:0/2298959699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2c806bf10 msgr2=0x7ff2c8155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:49.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.696+0000 7ff2cdf9b700 1 --2- 172.21.15.27:0/2298959699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2c806bf10 0x7ff2c8155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.696+0000 7ff2cdf9b700 1 -- 172.21.15.27:0/2298959699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c40094f0 con 0x7ff2c8068720 2024-01-11T09:47:49.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.697+0000 7ff2cdf9b700 1 --2- 172.21.15.27:0/2298959699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c8068720 0x7ff2c8155310 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7ff2c806b3c0 tx=0x7ff2c4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:49.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.697+0000 7ff2bdffb700 1 -- 172.21.15.27:0/2298959699 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4000d10 con 0x7ff2c8068720 2024-01-11T09:47:49.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.697+0000 7ff2bdffb700 1 -- 172.21.15.27:0/2298959699 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c400bbe0 con 0x7ff2c8068720 2024-01-11T09:47:49.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.697+0000 7ff2bdffb700 1 -- 172.21.15.27:0/2298959699 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4011670 con 0x7ff2c8068720 2024-01-11T09:47:49.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.697+0000 7ff2d01ff700 1 -- 172.21.15.27:0/2298959699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c8156600 con 0x7ff2c8068720 2024-01-11T09:47:49.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.697+0000 7ff2d01ff700 1 -- 172.21.15.27:0/2298959699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c8156bc0 con 0x7ff2c8068720 2024-01-11T09:47:49.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.698+0000 7ff2d01ff700 1 -- 172.21.15.27:0/2298959699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2c80c5960 con 0x7ff2c8068720 2024-01-11T09:47:49.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.699+0000 7ff2bdffb700 1 -- 172.21.15.27:0/2298959699 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7ff2c4011aa0 con 0x7ff2c8068720 2024-01-11T09:47:49.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.700+0000 7ff2bdffb700 1 --2- 172.21.15.27:0/2298959699 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff2ac0670e0 0x7ff2ac069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:49.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.700+0000 7ff2bdffb700 1 -- 172.21.15.27:0/2298959699 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6247+0+0 (secure 0 0 0) 0x7ff2c4085e70 con 0x7ff2c8068720 2024-01-11T09:47:49.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.700+0000 7ff2cd79a700 1 -- 172.21.15.27:0/2298959699 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff2ac0670e0 msgr2=0x7ff2ac069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:49.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.700+0000 7ff2cd79a700 1 --2- 172.21.15.27:0/2298959699 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff2ac0670e0 0x7ff2ac069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:49.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.704+0000 7ff2bdffb700 1 -- 172.21.15.27:0/2298959699 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff2c80c5960 con 0x7ff2c8068720 2024-01-11T09:47:49.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.901+0000 7ff2cd79a700 1 -- 172.21.15.27:0/2298959699 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff2ac0670e0 msgr2=0x7ff2ac069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:49.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.901+0000 7ff2cd79a700 1 --2- 172.21.15.27:0/2298959699 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff2ac0670e0 0x7ff2ac069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:49.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.954+0000 7ff2d01ff700 1 -- 172.21.15.27:0/2298959699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff2c80c5960 con 0x7ff2c8068720 2024-01-11T09:47:49.946 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-11T09:47:49.955+0000 7f27065f1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-11T09:47:49.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.960+0000 7ff2bdffb700 1 -- 172.21.15.27:0/2298959699 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v142) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff2c80c5960 con 0x7ff2c8068720 2024-01-11T09:47:49.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.963+0000 7ff2d01ff700 1 -- 172.21.15.27:0/2298959699 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff2ac0670e0 msgr2=0x7ff2ac069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:49.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.963+0000 7ff2d01ff700 1 --2- 172.21.15.27:0/2298959699 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff2ac0670e0 0x7ff2ac069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.963+0000 7ff2d01ff700 1 -- 172.21.15.27:0/2298959699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c8068720 msgr2=0x7ff2c8155310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:49.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.963+0000 7ff2d01ff700 1 --2- 172.21.15.27:0/2298959699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c8068720 0x7ff2c8155310 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7ff2c806b3c0 tx=0x7ff2c4017000 comp rx=0 tx=0).stop 2024-01-11T09:47:49.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.963+0000 7ff2d01ff700 1 -- 172.21.15.27:0/2298959699 shutdown_connections 2024-01-11T09:47:49.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.963+0000 7ff2d01ff700 1 --2- 172.21.15.27:0/2298959699 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff2c80b0a40 0x7ff2c80b2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.963+0000 7ff2d01ff700 1 --2- 172.21.15.27:0/2298959699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c8068720 0x7ff2c8155310 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.963+0000 7ff2d01ff700 1 --2- 172.21.15.27:0/2298959699 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff2ac0670e0 0x7ff2ac069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.963+0000 7ff2d01ff700 1 --2- 172.21.15.27:0/2298959699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2c806bf10 0x7ff2c8155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:49.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.963+0000 7ff2d01ff700 1 -- 172.21.15.27:0/2298959699 >> 172.21.15.27:0/2298959699 conn(0x7ff2c80ce3c0 msgr2=0x7ff2c806d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:49.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.963+0000 7ff2d01ff700 1 -- 172.21.15.27:0/2298959699 shutdown_connections 2024-01-11T09:47:49.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:49.963+0000 7ff2d01ff700 1 -- 172.21.15.27:0/2298959699 wait complete. 2024-01-11T09:47:49.956 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-11T09:47:49.959 INFO:tasks.ceph.mds.c.smithi027.stderr: -2454> 2024-01-11T09:47:29.560+0000 7feaee9f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-11T09:47:49.974 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-11T09:47:50.004 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-11T09:47:50.046 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:50.055+0000 7fef03e9e1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:47:50.094 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:50.103+0000 7f40ad6511c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:47:50.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.194+0000 7f44711ff700 1 Processor -- start 2024-01-11T09:47:50.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.194+0000 7f44711ff700 1 -- start start 2024-01-11T09:47:50.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.195+0000 7f44711ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f446c157410 0x7f446c0ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:50.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.195+0000 7f44711ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f446c1577e0 0x7f446c0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:50.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.195+0000 7f44711ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f446c0699d0 con 0x7f446c069540 2024-01-11T09:47:50.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.195+0000 7f44711ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f446c069cb0 con 0x7f446c1577e0 2024-01-11T09:47:50.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.195+0000 7f44711ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f446c069f90 con 0x7f446c157410 2024-01-11T09:47:50.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.195+0000 7f446ad9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f446c157410 0x7f446c0ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:50.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.196+0000 7f446ad9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f446c157410 0x7f446c0ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:51508/0 (socket says 172.21.15.27:51508) 2024-01-11T09:47:50.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.196+0000 7f446ad9d700 1 -- 172.21.15.27:0/3106839222 learned_addr learned my addr 172.21.15.27:0/3106839222 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:50.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.196+0000 7f446b59e700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f446c069540 0x7f446c15b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:34294/0 (socket says 172.21.15.27:34294) 2024-01-11T09:47:50.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.196+0000 7f446a59c700 1 --2- 172.21.15.27:0/3106839222 >> v2:172.21.15.77:3300/0 conn(0x7f446c1577e0 0x7f446c0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:50.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.196+0000 7f446ad9d700 1 -- 172.21.15.27:0/3106839222 >> v2:172.21.15.77:3300/0 conn(0x7f446c1577e0 msgr2=0x7f446c0ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:50.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.196+0000 7f446ad9d700 1 --2- 172.21.15.27:0/3106839222 >> v2:172.21.15.77:3300/0 conn(0x7f446c1577e0 0x7f446c0ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:50.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.196+0000 7f446ad9d700 1 -- 172.21.15.27:0/3106839222 >> v1:172.21.15.77:6789/0 conn(0x7f446c069540 legacy=0x7f446c15b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:50.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.196+0000 7f446ad9d700 1 -- 172.21.15.27:0/3106839222 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446c06a270 con 0x7f446c157410 2024-01-11T09:47:50.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.196+0000 7f4469d9b700 1 -- 172.21.15.27:0/3106839222 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2317009768 0 0) 0x7f446c0699d0 con 0x7f446c069540 2024-01-11T09:47:50.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.196+0000 7f446ad9d700 1 --2- 172.21.15.27:0/3106839222 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f446c157410 0x7f446c0ccae0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f445c009990 tx=0x7f445c009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd94ad74a1c07699 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:50.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.197+0000 7f4469d9b700 1 -- 172.21.15.27:0/3106839222 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445c009e80 con 0x7f446c157410 2024-01-11T09:47:50.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.197+0000 7f4469d9b700 1 -- 172.21.15.27:0/3106839222 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445c005c10 con 0x7f446c157410 2024-01-11T09:47:50.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.197+0000 7f4469d9b700 1 -- 172.21.15.27:0/3106839222 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445c00b650 con 0x7f446c157410 2024-01-11T09:47:50.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.197+0000 7f44711ff700 1 -- 172.21.15.27:0/3106839222 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f446c157410 msgr2=0x7f446c0ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:50.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.197+0000 7f44711ff700 1 --2- 172.21.15.27:0/3106839222 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f446c157410 0x7f446c0ccae0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f445c009990 tx=0x7f445c009220 comp rx=0 tx=0).stop 2024-01-11T09:47:50.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.197+0000 7f44711ff700 1 -- 172.21.15.27:0/3106839222 shutdown_connections 2024-01-11T09:47:50.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.197+0000 7f44711ff700 1 --2- 172.21.15.27:0/3106839222 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f446c157410 0x7f446c0ccae0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:50.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.197+0000 7f44711ff700 1 --2- 172.21.15.27:0/3106839222 >> v2:172.21.15.77:3300/0 conn(0x7f446c1577e0 0x7f446c0ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:50.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.197+0000 7f44711ff700 1 -- 172.21.15.27:0/3106839222 >> 172.21.15.27:0/3106839222 conn(0x7f446c0680a0 msgr2=0x7f446c0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:50.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.197+0000 7f44711ff700 1 -- 172.21.15.27:0/3106839222 shutdown_connections 2024-01-11T09:47:50.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.197+0000 7f44711ff700 1 -- 172.21.15.27:0/3106839222 wait complete. 2024-01-11T09:47:50.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.199+0000 7f44711ff700 1 Processor -- start 2024-01-11T09:47:50.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.199+0000 7f44711ff700 1 -- start start 2024-01-11T09:47:50.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.200+0000 7f44711ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f446c069540 0x7f446c0c5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.200+0000 7f44711ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f446c157410 0x7f446c157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.200+0000 7f44711ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f446c1577e0 0x7f446c151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.200+0000 7f44711ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446c16acd0 con 0x7f446c157410 2024-01-11T09:47:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.200+0000 7f44711ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446c167bd0 con 0x7f446c1577e0 2024-01-11T09:47:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.200+0000 7f44711ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f446c167eb0 con 0x7f446c069540 2024-01-11T09:47:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.200+0000 7f446ad9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f446c069540 0x7f446c0c5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.200+0000 7f446a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f446c157410 0x7f446c157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.200+0000 7f446a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f446c157410 0x7f446c157bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60642/0 (socket says 172.21.15.27:60642) 2024-01-11T09:47:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.200+0000 7f446a59c700 1 -- 172.21.15.27:0/601751769 learned_addr learned my addr 172.21.15.27:0/601751769 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.200+0000 7f446b59e700 1 --2- 172.21.15.27:0/601751769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f446c1577e0 0x7f446c151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:50.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.201+0000 7f446a59c700 1 -- 172.21.15.27:0/601751769 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f446c069540 msgr2=0x7f446c0c5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.201+0000 7f446a59c700 1 --2- 172.21.15.27:0/601751769 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f446c069540 0x7f446c0c5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.201+0000 7f446a59c700 1 -- 172.21.15.27:0/601751769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f446c1577e0 msgr2=0x7f446c151c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.201+0000 7f446a59c700 1 --2- 172.21.15.27:0/601751769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f446c1577e0 0x7f446c151c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.201+0000 7f446a59c700 1 -- 172.21.15.27:0/601751769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445c009480 con 0x7f446c157410 2024-01-11T09:47:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.201+0000 7f446b59e700 1 --2- 172.21.15.27:0/601751769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f446c1577e0 0x7f446c151c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.201+0000 7f446a59c700 1 --2- 172.21.15.27:0/601751769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f446c157410 0x7f446c157bb0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f446000e900 tx=0x7f44600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.201+0000 7f4452ffd700 1 -- 172.21.15.27:0/601751769 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4460013680 con 0x7f446c157410 2024-01-11T09:47:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.201+0000 7f4452ffd700 1 -- 172.21.15.27:0/601751769 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44600137e0 con 0x7f446c157410 2024-01-11T09:47:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.201+0000 7f4452ffd700 1 -- 172.21.15.27:0/601751769 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4460013ba0 con 0x7f446c157410 2024-01-11T09:47:50.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.201+0000 7f44711ff700 1 -- 172.21.15.27:0/601751769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446c168190 con 0x7f446c157410 2024-01-11T09:47:50.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.202+0000 7f44711ff700 1 -- 172.21.15.27:0/601751769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f446c0c55b0 con 0x7f446c157410 2024-01-11T09:47:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.203+0000 7f44711ff700 1 -- 172.21.15.27:0/601751769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f446c168630 con 0x7f446c157410 2024-01-11T09:47:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.203+0000 7f4452ffd700 1 -- 172.21.15.27:0/601751769 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 150) v1 ==== 84901+0+0 (secure 0 0 0) 0x7f44600123d0 con 0x7f446c157410 2024-01-11T09:47:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.205+0000 7f4452ffd700 1 --2- 172.21.15.27:0/601751769 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f44580670e0 0x7f4458069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.205+0000 7f4452ffd700 1 -- 172.21.15.27:0/601751769 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f4460084960 con 0x7f446c157410 2024-01-11T09:47:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.205+0000 7f446ad9d700 1 -- 172.21.15.27:0/601751769 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f44580670e0 msgr2=0x7f4458069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:50.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.205+0000 7f446ad9d700 1 --2- 172.21.15.27:0/601751769 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f44580670e0 0x7f4458069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-11T09:47:50.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.209+0000 7f4452ffd700 1 -- 172.21.15.27:0/601751769 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4460051640 con 0x7f446c157410 2024-01-11T09:47:50.232 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:50.241+0000 7fef03e9e1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:47:50.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.406+0000 7f446ad9d700 1 -- 172.21.15.27:0/601751769 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f44580670e0 msgr2=0x7f4458069590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6836/59776 2024-01-11T09:47:50.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.406+0000 7f446ad9d700 1 --2- 172.21.15.27:0/601751769 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f44580670e0 0x7f4458069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-11T09:47:50.421 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:50.430+0000 7fef03e9e1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:47:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.458+0000 7f44711ff700 1 -- 172.21.15.27:0/601751769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f446c04c3c0 con 0x7f446c157410 2024-01-11T09:47:50.532 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:50.541+0000 7fef03e9e1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:47:50.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.549+0000 7f4452ffd700 1 -- 172.21.15.27:0/601751769 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 151) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f4460051a10 con 0x7f446c157410 2024-01-11T09:47:50.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.549+0000 7f4452ffd700 1 -- 172.21.15.27:0/601751769 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f44580670e0 msgr2=0x7f4458069590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T09:47:50.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.549+0000 7f4452ffd700 1 --2- 172.21.15.27:0/601751769 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f44580670e0 0x7f4458069590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:50.556 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:50.565+0000 7f40ad6511c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T09:47:50.610 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:50.619+0000 7fee7ecc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:50.610 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:50.619+0000 7fee7ecc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:50.610 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:50.619+0000 7fee7ecc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:50.610 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:50.619+0000 7fee7ecc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:50.610 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:50.619+0000 7fee7ecc4700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:50.611 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:50.620+0000 7fee81cca700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:50.611 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:50.620+0000 7fee81cca700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:50.611 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:50.620+0000 7fee81cca700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:50.611 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:50.620+0000 7fee81cca700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:50.611 INFO:tasks.ceph.mgr.x.smithi077.stderr:2024-01-11T09:47:50.620+0000 7fee81cca700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-11T09:47:50.752 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:50.761+0000 7f40ad6511c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T09:47:50.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.959+0000 7f4452ffd700 1 -- 172.21.15.27:0/601751769 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v143) v1 ==== 104+0+0 (secure 0 0 0) 0x7f446006fef0 con 0x7f446c157410 2024-01-11T09:47:50.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.962+0000 7f44711ff700 1 -- 172.21.15.27:0/601751769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f446c157410 msgr2=0x7f446c157bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:50.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.962+0000 7f44711ff700 1 --2- 172.21.15.27:0/601751769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f446c157410 0x7f446c157bb0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f446000e900 tx=0x7f44600091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:50.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.962+0000 7f44711ff700 1 -- 172.21.15.27:0/601751769 shutdown_connections 2024-01-11T09:47:50.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.962+0000 7f44711ff700 1 --2- 172.21.15.27:0/601751769 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f44580670e0 0x7f4458069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:50.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.962+0000 7f44711ff700 1 --2- 172.21.15.27:0/601751769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f446c1577e0 0x7f446c151c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:50.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.962+0000 7f44711ff700 1 --2- 172.21.15.27:0/601751769 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f446c069540 0x7f446c0c5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:50.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.962+0000 7f44711ff700 1 --2- 172.21.15.27:0/601751769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f446c157410 0x7f446c157bb0 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:50.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.962+0000 7f44711ff700 1 -- 172.21.15.27:0/601751769 >> 172.21.15.27:0/601751769 conn(0x7f446c0680a0 msgr2=0x7f446c152ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:50.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.962+0000 7f44711ff700 1 -- 172.21.15.27:0/601751769 shutdown_connections 2024-01-11T09:47:50.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:50.962+0000 7f44711ff700 1 -- 172.21.15.27:0/601751769 wait complete. 2024-01-11T09:47:50.956 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:50.965+0000 7f40ad6511c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T09:47:50.972 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-11T09:47:51.078 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-11T09:47:51.087+0000 7f40ad6511c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T09:47:51.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.229+0000 7f6afe3ff700 1 Processor -- start 2024-01-11T09:47:51.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.229+0000 7f6afe3ff700 1 -- start start 2024-01-11T09:47:51.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.230+0000 7f6afe3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6af8159250 0x7f6af80b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:51.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.230+0000 7f6afe3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6af8068be0 0x7f6af80cb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:51.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.230+0000 7f6afe3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6af815a440 con 0x7f6af80cb410 2024-01-11T09:47:51.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.230+0000 7f6afe3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6af815ace0 con 0x7f6af8159250 2024-01-11T09:47:51.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.230+0000 7f6afe3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af80b0de0 con 0x7f6af8068be0 2024-01-11T09:47:51.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.230+0000 7f6af7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6af8159250 0x7f6af80b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:51.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.230+0000 7f6af7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6af8159250 0x7f6af80b08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60658/0 (socket says 172.21.15.27:60658) 2024-01-11T09:47:51.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.230+0000 7f6af7fff700 1 -- 172.21.15.27:0/3361824233 learned_addr learned my addr 172.21.15.27:0/3361824233 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:51.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.231+0000 7f6afc99c700 1 --2- 172.21.15.27:0/3361824233 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6af8068be0 0x7f6af80cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:51.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.231+0000 7f6af7fff700 1 -- 172.21.15.27:0/3361824233 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6af8068be0 msgr2=0x7f6af80cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:51.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.231+0000 7f6af7fff700 1 --2- 172.21.15.27:0/3361824233 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6af8068be0 0x7f6af80cb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:51.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.231+0000 7f6af7fff700 1 -- 172.21.15.27:0/3361824233 >> v1:172.21.15.77:6789/0 conn(0x7f6af80cb410 legacy=0x7f6af80684d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:51.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.231+0000 7f6af7fff700 1 -- 172.21.15.27:0/3361824233 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af815a720 con 0x7f6af8159250 2024-01-11T09:47:51.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.231+0000 7f6af7fff700 1 --2- 172.21.15.27:0/3361824233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8159250 0x7f6af80b08a0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f6ae80099b0 tx=0x7f6ae8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=432b5b179a26a6ca server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:51.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.231+0000 7f6af6ffd700 1 -- 172.21.15.27:0/3361824233 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae800b860 con 0x7f6af8159250 2024-01-11T09:47:51.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.231+0000 7f6af6ffd700 1 -- 172.21.15.27:0/3361824233 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae800b9c0 con 0x7f6af8159250 2024-01-11T09:47:51.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.231+0000 7f6af6ffd700 1 -- 172.21.15.27:0/3361824233 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae800bd60 con 0x7f6af8159250 2024-01-11T09:47:51.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.232+0000 7f6afe3ff700 1 -- 172.21.15.27:0/3361824233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8159250 msgr2=0x7f6af80b08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:51.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.232+0000 7f6afe3ff700 1 --2- 172.21.15.27:0/3361824233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8159250 0x7f6af80b08a0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f6ae80099b0 tx=0x7f6ae8009250 comp rx=0 tx=0).stop 2024-01-11T09:47:51.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.232+0000 7f6afe3ff700 1 -- 172.21.15.27:0/3361824233 shutdown_connections 2024-01-11T09:47:51.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.232+0000 7f6afe3ff700 1 --2- 172.21.15.27:0/3361824233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8159250 0x7f6af80b08a0 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:51.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.232+0000 7f6afe3ff700 1 --2- 172.21.15.27:0/3361824233 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6af8068be0 0x7f6af80cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:51.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.232+0000 7f6afe3ff700 1 -- 172.21.15.27:0/3361824233 >> 172.21.15.27:0/3361824233 conn(0x7f6af8067610 msgr2=0x7f6af8067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:51.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.232+0000 7f6afe3ff700 1 -- 172.21.15.27:0/3361824233 shutdown_connections 2024-01-11T09:47:51.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.232+0000 7f6afe3ff700 1 -- 172.21.15.27:0/3361824233 wait complete. 2024-01-11T09:47:51.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.234+0000 7f6afe3ff700 1 Processor -- start 2024-01-11T09:47:51.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.234+0000 7f6afe3ff700 1 -- start start 2024-01-11T09:47:51.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.235+0000 7f6afe3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8068be0 0x7f6af80b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:51.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.235+0000 7f6afe3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6af80cb410 0x7f6af8154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:51.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.235+0000 7f6afe3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6af8159250 0x7f6af80aead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:51.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.235+0000 7f6afe3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af80b2200 con 0x7f6af8068be0 2024-01-11T09:47:51.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.235+0000 7f6afe3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af815c970 con 0x7f6af8159250 2024-01-11T09:47:51.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.235+0000 7f6afe3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af80b0080 con 0x7f6af80cb410 2024-01-11T09:47:51.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.235+0000 7f6af7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8068be0 0x7f6af80b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:51.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.235+0000 7f6af7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8068be0 0x7f6af80b1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60674/0 (socket says 172.21.15.27:60674) 2024-01-11T09:47:51.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.235+0000 7f6af7fff700 1 -- 172.21.15.27:0/3161567287 learned_addr learned my addr 172.21.15.27:0/3161567287 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:51.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.235+0000 7f6af77fe700 1 --2- 172.21.15.27:0/3161567287 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6af80cb410 0x7f6af8154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.235+0000 7f6afc99c700 1 --2- 172.21.15.27:0/3161567287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6af8159250 0x7f6af80aead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.235+0000 7f6af7fff700 1 -- 172.21.15.27:0/3161567287 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6af80cb410 msgr2=0x7f6af8154fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.235+0000 7f6af7fff700 1 --2- 172.21.15.27:0/3161567287 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6af80cb410 0x7f6af8154fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.235+0000 7f6af7fff700 1 -- 172.21.15.27:0/3161567287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6af8159250 msgr2=0x7f6af80aead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.235+0000 7f6af7fff700 1 --2- 172.21.15.27:0/3161567287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6af8159250 0x7f6af80aead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.235+0000 7f6af7fff700 1 -- 172.21.15.27:0/3161567287 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae80094f0 con 0x7f6af8068be0 2024-01-11T09:47:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.235+0000 7f6afc99c700 1 --2- 172.21.15.27:0/3161567287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6af8159250 0x7f6af80aead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.236+0000 7f6af7fff700 1 --2- 172.21.15.27:0/3161567287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8068be0 0x7f6af80b1b70 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f6af80ceef0 tx=0x7f6ae8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.236+0000 7f6adffff700 1 -- 172.21.15.27:0/3161567287 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8024440 con 0x7f6af8068be0 2024-01-11T09:47:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.236+0000 7f6adffff700 1 -- 172.21.15.27:0/3161567287 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae80245a0 con 0x7f6af8068be0 2024-01-11T09:47:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.236+0000 7f6adffff700 1 -- 172.21.15.27:0/3161567287 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8024940 con 0x7f6af8068be0 2024-01-11T09:47:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.236+0000 7f6afe3ff700 1 -- 172.21.15.27:0/3161567287 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af815c040 con 0x7f6af8068be0 2024-01-11T09:47:51.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.236+0000 7f6afe3ff700 1 -- 172.21.15.27:0/3161567287 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af815c600 con 0x7f6af8068be0 2024-01-11T09:47:51.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.237+0000 7f6afe3ff700 1 -- 172.21.15.27:0/3161567287 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af804c3c0 con 0x7f6af8068be0 2024-01-11T09:47:51.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.238+0000 7f6adffff700 1 -- 172.21.15.27:0/3161567287 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 151) v1 ==== 84453+0+0 (secure 0 0 0) 0x7f6ae801a420 con 0x7f6af8068be0 2024-01-11T09:47:51.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.240+0000 7f6adffff700 1 -- 172.21.15.27:0/3161567287 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 6634+0+0 (secure 0 0 0) 0x7f6ae8086230 con 0x7f6af8068be0 2024-01-11T09:47:51.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.244+0000 7f6adffff700 1 -- 172.21.15.27:0/3161567287 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6ae8053450 con 0x7f6af8068be0 2024-01-11T09:47:51.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.448+0000 7f6afe3ff700 1 -- 172.21.15.27:0/3161567287 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6af815aa00 con 0x7f6af8068be0 2024-01-11T09:47:51.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.552+0000 7f6adffff700 1 -- 172.21.15.27:0/3161567287 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 152) v1 ==== 84580+0+0 (secure 0 0 0) 0x7f6ae8052f00 con 0x7f6af8068be0 2024-01-11T09:47:51.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.552+0000 7f6adffff700 1 --2- 172.21.15.27:0/3161567287 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6ae406ca40 0x7f6ae406ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:51.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.553+0000 7f6af77fe700 1 --2- 172.21.15.27:0/3161567287 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6ae406ca40 0x7f6ae406ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:51.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:51.553+0000 7f6af77fe700 1 --2- 172.21.15.27:0/3161567287 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6ae406ca40 0x7f6ae406ee20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6af816d060 tx=0x7f6aec007ec0 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:52.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.350+0000 7f6adffff700 1 -- 172.21.15.27:0/3161567287 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v260) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6ae8069960 con 0x7f6af8068be0 2024-01-11T09:47:52.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.355+0000 7f6afe3ff700 1 -- 172.21.15.27:0/3161567287 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6ae406ca40 msgr2=0x7f6ae406ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:52.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.355+0000 7f6afe3ff700 1 --2- 172.21.15.27:0/3161567287 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6ae406ca40 0x7f6ae406ee20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6af816d060 tx=0x7f6aec007ec0 comp rx=0 tx=0).stop 2024-01-11T09:47:52.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.355+0000 7f6afe3ff700 1 -- 172.21.15.27:0/3161567287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8068be0 msgr2=0x7f6af80b1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:52.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.355+0000 7f6afe3ff700 1 --2- 172.21.15.27:0/3161567287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8068be0 0x7f6af80b1b70 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f6af80ceef0 tx=0x7f6ae8017000 comp rx=0 tx=0).stop 2024-01-11T09:47:52.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.355+0000 7f6afe3ff700 1 -- 172.21.15.27:0/3161567287 shutdown_connections 2024-01-11T09:47:52.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.355+0000 7f6afe3ff700 1 --2- 172.21.15.27:0/3161567287 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f6ae406ca40 0x7f6ae406ee20 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:52.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.355+0000 7f6afe3ff700 1 --2- 172.21.15.27:0/3161567287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6af8159250 0x7f6af80aead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:52.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.355+0000 7f6afe3ff700 1 --2- 172.21.15.27:0/3161567287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8068be0 0x7f6af80b1b70 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:52.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.355+0000 7f6afe3ff700 1 --2- 172.21.15.27:0/3161567287 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f6af80cb410 0x7f6af8154fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:52.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.355+0000 7f6afe3ff700 1 -- 172.21.15.27:0/3161567287 >> 172.21.15.27:0/3161567287 conn(0x7f6af8067610 msgr2=0x7f6af8158810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:52.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.355+0000 7f6afe3ff700 1 -- 172.21.15.27:0/3161567287 shutdown_connections 2024-01-11T09:47:52.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.356+0000 7f6afe3ff700 1 -- 172.21.15.27:0/3161567287 wait complete. 2024-01-11T09:47:52.348 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-01-11T09:47:52.370 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-11T09:47:52.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.589+0000 7fb0cedff700 1 Processor -- start 2024-01-11T09:47:52.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.590+0000 7fb0cedff700 1 -- start start 2024-01-11T09:47:52.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.591+0000 7fb0cedff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb0c8157810 0x7fb0c8069660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.591+0000 7fb0cedff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c806f4d0 0x7fb0c8069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.591+0000 7fb0cedff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c806d420 con 0x7fb0c8157440 2024-01-11T09:47:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.591+0000 7fb0cedff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c806d700 con 0x7fb0c8157810 2024-01-11T09:47:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.591+0000 7fb0cedff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c806d9e0 con 0x7fb0c806f4d0 2024-01-11T09:47:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.592+0000 7fb0ccb9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb0c8157440 0x7fb0c80ae430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37812/0 (socket says 172.21.15.27:37812) 2024-01-11T09:47:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.592+0000 7fb0ccb9b700 1 -- 172.21.15.27:0/4094040577 learned_addr learned my addr 172.21.15.27:0/4094040577 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.592+0000 7fb0c7fff700 1 --2- 172.21.15.27:0/4094040577 >> v2:172.21.15.77:3300/0 conn(0x7fb0c8157810 0x7fb0c8069660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.592+0000 7fb0cd39c700 1 --2- 172.21.15.27:0/4094040577 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c806f4d0 0x7fb0c8069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.592+0000 7fb0c77fe700 1 -- 172.21.15.27:0/4094040577 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950836931 0 0) 0x7fb0c806d420 con 0x7fb0c8157440 2024-01-11T09:47:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.592+0000 7fb0c77fe700 1 -- 172.21.15.27:0/4094040577 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0b8003570 con 0x7fb0c8157440 2024-01-11T09:47:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.592+0000 7fb0c7fff700 1 -- 172.21.15.27:0/4094040577 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c806f4d0 msgr2=0x7fb0c8069ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.593+0000 7fb0c7fff700 1 --2- 172.21.15.27:0/4094040577 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c806f4d0 0x7fb0c8069ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.593+0000 7fb0c7fff700 1 -- 172.21.15.27:0/4094040577 >> v1:172.21.15.27:6789/0 conn(0x7fb0c8157440 legacy=0x7fb0c80ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.593+0000 7fb0c7fff700 1 -- 172.21.15.27:0/4094040577 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c806dcc0 con 0x7fb0c8157810 2024-01-11T09:47:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.593+0000 7fb0c7fff700 1 --2- 172.21.15.27:0/4094040577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c8157810 0x7fb0c8069660 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fb0b40098e0 tx=0x7fb0b40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f83bdc0b62eb4e96 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.593+0000 7fb0c77fe700 1 -- 172.21.15.27:0/4094040577 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4011630 con 0x7fb0c8157810 2024-01-11T09:47:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.593+0000 7fb0c77fe700 1 -- 172.21.15.27:0/4094040577 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b4011790 con 0x7fb0c8157810 2024-01-11T09:47:52.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.593+0000 7fb0c77fe700 1 -- 172.21.15.27:0/4094040577 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4011b30 con 0x7fb0c8157810 2024-01-11T09:47:52.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.594+0000 7fb0cedff700 1 -- 172.21.15.27:0/4094040577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c8157810 msgr2=0x7fb0c8069660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:52.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.594+0000 7fb0cedff700 1 --2- 172.21.15.27:0/4094040577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c8157810 0x7fb0c8069660 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fb0b40098e0 tx=0x7fb0b40091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:52.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.594+0000 7fb0cedff700 1 -- 172.21.15.27:0/4094040577 shutdown_connections 2024-01-11T09:47:52.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.594+0000 7fb0cedff700 1 --2- 172.21.15.27:0/4094040577 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c806f4d0 0x7fb0c8069ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:52.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.594+0000 7fb0cedff700 1 --2- 172.21.15.27:0/4094040577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c8157810 0x7fb0c8069660 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:52.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.594+0000 7fb0cedff700 1 -- 172.21.15.27:0/4094040577 >> 172.21.15.27:0/4094040577 conn(0x7fb0c80680e0 msgr2=0x7fb0c80cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:52.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.594+0000 7fb0cedff700 1 -- 172.21.15.27:0/4094040577 shutdown_connections 2024-01-11T09:47:52.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.594+0000 7fb0cedff700 1 -- 172.21.15.27:0/4094040577 wait complete. 2024-01-11T09:47:52.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.596+0000 7fb0cedff700 1 Processor -- start 2024-01-11T09:47:52.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.596+0000 7fb0cedff700 1 -- start start 2024-01-11T09:47:52.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.597+0000 7fb0cedff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c806f4d0 0x7fb0c80c5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:52.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.597+0000 7fb0cedff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c8157440 0x7fb0c8168250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:52.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.597+0000 7fb0cedff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c8157810 0x7fb0c814d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:52.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.597+0000 7fb0cedff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c80c6090 con 0x7fb0c8157440 2024-01-11T09:47:52.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.597+0000 7fb0cedff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c814dd00 con 0x7fb0c8157810 2024-01-11T09:47:52.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.597+0000 7fb0cedff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c80c56c0 con 0x7fb0c806f4d0 2024-01-11T09:47:52.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.597+0000 7fb0c7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c8157440 0x7fb0c8168250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:52.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.597+0000 7fb0c7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c8157440 0x7fb0c8168250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60680/0 (socket says 172.21.15.27:60680) 2024-01-11T09:47:52.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.597+0000 7fb0c7fff700 1 -- 172.21.15.27:0/4191815489 learned_addr learned my addr 172.21.15.27:0/4191815489 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.597+0000 7fb0ccb9b700 1 --2- 172.21.15.27:0/4191815489 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c806f4d0 0x7fb0c80c5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.597+0000 7fb0cd39c700 1 --2- 172.21.15.27:0/4191815489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c8157810 0x7fb0c814d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.597+0000 7fb0c7fff700 1 -- 172.21.15.27:0/4191815489 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c806f4d0 msgr2=0x7fb0c80c5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.597+0000 7fb0c7fff700 1 --2- 172.21.15.27:0/4191815489 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c806f4d0 0x7fb0c80c5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.597+0000 7fb0c7fff700 1 -- 172.21.15.27:0/4191815489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c8157810 msgr2=0x7fb0c814d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.597+0000 7fb0c7fff700 1 --2- 172.21.15.27:0/4191815489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c8157810 0x7fb0c814d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.597+0000 7fb0c7fff700 1 -- 172.21.15.27:0/4191815489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b4009420 con 0x7fb0c8157440 2024-01-11T09:47:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.598+0000 7fb0c7fff700 1 --2- 172.21.15.27:0/4191815489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c8157440 0x7fb0c8168250 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7fb0c806cfd0 tx=0x7fb0b4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.598+0000 7fb0c4ff9700 1 -- 172.21.15.27:0/4191815489 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4011900 con 0x7fb0c8157440 2024-01-11T09:47:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.598+0000 7fb0c4ff9700 1 -- 172.21.15.27:0/4191815489 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b4011a60 con 0x7fb0c8157440 2024-01-11T09:47:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.598+0000 7fb0c4ff9700 1 -- 172.21.15.27:0/4191815489 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4025620 con 0x7fb0c8157440 2024-01-11T09:47:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.598+0000 7fb0cedff700 1 -- 172.21.15.27:0/4191815489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c8168790 con 0x7fb0c8157440 2024-01-11T09:47:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.598+0000 7fb0cedff700 1 -- 172.21.15.27:0/4191815489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c80c5070 con 0x7fb0c8157440 2024-01-11T09:47:52.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.599+0000 7fb0cedff700 1 -- 172.21.15.27:0/4191815489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c804c3c0 con 0x7fb0c8157440 2024-01-11T09:47:52.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.600+0000 7fb0c4ff9700 1 -- 172.21.15.27:0/4191815489 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 152) v1 ==== 84580+0+0 (secure 0 0 0) 0x7fb0b4025a10 con 0x7fb0c8157440 2024-01-11T09:47:52.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.601+0000 7fb0c4ff9700 1 --2- 172.21.15.27:0/4191815489 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0ac066d80 0x7fb0ac069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:52.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.601+0000 7fb0c4ff9700 1 -- 172.21.15.27:0/4191815489 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 6196+0+0 (secure 0 0 0) 0x7fb0b40859f0 con 0x7fb0c8157440 2024-01-11T09:47:52.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.602+0000 7fb0ccb9b700 1 --2- 172.21.15.27:0/4191815489 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0ac066d80 0x7fb0ac069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:52.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.602+0000 7fb0ccb9b700 1 --2- 172.21.15.27:0/4191815489 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0ac066d80 0x7fb0ac069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb0c815add0 tx=0x7fb0bc009040 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:52.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.605+0000 7fb0c4ff9700 1 -- 172.21.15.27:0/4191815489 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb0b4052d20 con 0x7fb0c8157440 2024-01-11T09:47:52.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:52.812+0000 7fb0cedff700 1 -- 172.21.15.27:0/4191815489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb0c806e030 con 0x7fb0c8157440 2024-01-11T09:47:53.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.364+0000 7fb0c4ff9700 1 -- 172.21.15.27:0/4191815489 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 153) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fb0b40530f0 con 0x7fb0c8157440 2024-01-11T09:47:53.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.558+0000 7fb0c4ff9700 1 -- 172.21.15.27:0/4191815489 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v261) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb0b40567b0 con 0x7fb0c8157440 2024-01-11T09:47:53.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.563+0000 7fb0cedff700 1 -- 172.21.15.27:0/4191815489 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0ac066d80 msgr2=0x7fb0ac069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:53.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.563+0000 7fb0cedff700 1 --2- 172.21.15.27:0/4191815489 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0ac066d80 0x7fb0ac069230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb0c815add0 tx=0x7fb0bc009040 comp rx=0 tx=0).stop 2024-01-11T09:47:53.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.563+0000 7fb0cedff700 1 -- 172.21.15.27:0/4191815489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c8157440 msgr2=0x7fb0c8168250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:53.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.563+0000 7fb0cedff700 1 --2- 172.21.15.27:0/4191815489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c8157440 0x7fb0c8168250 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7fb0c806cfd0 tx=0x7fb0b4017000 comp rx=0 tx=0).stop 2024-01-11T09:47:53.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.563+0000 7fb0cedff700 1 -- 172.21.15.27:0/4191815489 shutdown_connections 2024-01-11T09:47:53.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.563+0000 7fb0cedff700 1 --2- 172.21.15.27:0/4191815489 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fb0ac066d80 0x7fb0ac069230 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:53.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.563+0000 7fb0cedff700 1 --2- 172.21.15.27:0/4191815489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0c8157810 0x7fb0c814d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:53.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.563+0000 7fb0cedff700 1 --2- 172.21.15.27:0/4191815489 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fb0c806f4d0 0x7fb0c80c5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:53.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.563+0000 7fb0cedff700 1 --2- 172.21.15.27:0/4191815489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c8157440 0x7fb0c8168250 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:53.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.563+0000 7fb0cedff700 1 -- 172.21.15.27:0/4191815489 >> 172.21.15.27:0/4191815489 conn(0x7fb0c80680e0 msgr2=0x7fb0c80c8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:53.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.564+0000 7fb0cedff700 1 -- 172.21.15.27:0/4191815489 shutdown_connections 2024-01-11T09:47:53.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.564+0000 7fb0cedff700 1 -- 172.21.15.27:0/4191815489 wait complete. 2024-01-11T09:47:53.558 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-01-11T09:47:53.578 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-01-11T09:47:53.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.787+0000 7fe5591ff700 1 Processor -- start 2024-01-11T09:47:53.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.788+0000 7fe5591ff700 1 -- start start 2024-01-11T09:47:53.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.789+0000 7fe5591ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe554143190 0x7fe5540cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:53.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.789+0000 7fe5591ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5540ccc70 con 0x7fe554142dc0 2024-01-11T09:47:53.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.789+0000 7fe5591ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55415d320 con 0x7fe5540cb4b0 2024-01-11T09:47:53.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.789+0000 7fe5591ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe55415df30 con 0x7fe554143190 2024-01-11T09:47:53.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.789+0000 7fe552d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe554142dc0 0x7fe5540cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37824/0 (socket says 172.21.15.27:37824) 2024-01-11T09:47:53.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.789+0000 7fe552d9d700 1 -- 172.21.15.27:0/74305524 learned_addr learned my addr 172.21.15.27:0/74305524 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:53.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.790+0000 7fe55259c700 1 --2- 172.21.15.27:0/74305524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe554143190 0x7fe5540cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:53.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.790+0000 7fe551d9b700 1 -- 172.21.15.27:0/74305524 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807028900 0 0) 0x7fe5540ccc70 con 0x7fe554142dc0 2024-01-11T09:47:53.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.790+0000 7fe551d9b700 1 -- 172.21.15.27:0/74305524 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe53c003570 con 0x7fe554142dc0 2024-01-11T09:47:53.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.790+0000 7fe551d9b700 1 -- 172.21.15.27:0/74305524 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258752746 0 0) 0x7fe55415d320 con 0x7fe5540cb4b0 2024-01-11T09:47:53.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.790+0000 7fe551d9b700 1 -- 172.21.15.27:0/74305524 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5540ccc70 con 0x7fe5540cb4b0 2024-01-11T09:47:53.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.790+0000 7fe551d9b700 1 -- 172.21.15.27:0/74305524 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1546186126 0 0) 0x7fe53c003570 con 0x7fe554142dc0 2024-01-11T09:47:53.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.790+0000 7fe551d9b700 1 -- 172.21.15.27:0/74305524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe554143190 msgr2=0x7fe5540cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:53.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.790+0000 7fe551d9b700 1 --2- 172.21.15.27:0/74305524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe554143190 0x7fe5540cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:53.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.790+0000 7fe551d9b700 1 -- 172.21.15.27:0/74305524 >> v1:172.21.15.77:6789/0 conn(0x7fe5540cb4b0 legacy=0x7fe5540cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:53.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.790+0000 7fe551d9b700 1 -- 172.21.15.27:0/74305524 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55415d590 con 0x7fe554142dc0 2024-01-11T09:47:53.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.790+0000 7fe551d9b700 1 -- 172.21.15.27:0/74305524 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fe544001470 con 0x7fe554142dc0 2024-01-11T09:47:53.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.790+0000 7fe551d9b700 1 -- 172.21.15.27:0/74305524 >> v1:172.21.15.27:6789/0 conn(0x7fe554142dc0 legacy=0x7fe5540cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:53.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.790+0000 7fe551d9b700 1 --2- 172.21.15.27:0/74305524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53c003c20 0x7fe53c004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:53.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.790+0000 7fe551d9b700 1 --2- 172.21.15.27:0/74305524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe53c004810 0x7fe53c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:53.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.790+0000 7fe551d9b700 1 --2- 172.21.15.27:0/74305524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe554143190 0x7fe53c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:53.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.790+0000 7fe551d9b700 1 -- 172.21.15.27:0/74305524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55415df30 con 0x7fe53c003c20 2024-01-11T09:47:53.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.790+0000 7fe551d9b700 1 -- 172.21.15.27:0/74305524 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c00bb40 con 0x7fe53c004810 2024-01-11T09:47:53.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.790+0000 7fe551d9b700 1 -- 172.21.15.27:0/74305524 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c00bd00 con 0x7fe554143190 2024-01-11T09:47:53.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.791+0000 7fe552d9d700 1 --2- 172.21.15.27:0/74305524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53c003c20 0x7fe53c004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:53.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.791+0000 7fe55259c700 1 --2- 172.21.15.27:0/74305524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe53c004810 0x7fe53c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:53.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.791+0000 7fe55359e700 1 --2- 172.21.15.27:0/74305524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe554143190 0x7fe53c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:53.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.791+0000 7fe552d9d700 1 -- 172.21.15.27:0/74305524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe554143190 msgr2=0x7fe53c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:53.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.791+0000 7fe552d9d700 1 --2- 172.21.15.27:0/74305524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe554143190 0x7fe53c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:53.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.791+0000 7fe552d9d700 1 -- 172.21.15.27:0/74305524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe53c004810 msgr2=0x7fe53c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:53.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.791+0000 7fe552d9d700 1 --2- 172.21.15.27:0/74305524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe53c004810 0x7fe53c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:53.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.791+0000 7fe552d9d700 1 -- 172.21.15.27:0/74305524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c003570 con 0x7fe53c003c20 2024-01-11T09:47:53.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.791+0000 7fe55359e700 1 --2- 172.21.15.27:0/74305524 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe554143190 0x7fe53c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:53.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.791+0000 7fe552d9d700 1 --2- 172.21.15.27:0/74305524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53c003c20 0x7fe53c004010 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7fe54400c2d0 tx=0x7fe544001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17bfc75f00759879 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:53.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.792+0000 7fe551d9b700 1 -- 172.21.15.27:0/74305524 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54400cba0 con 0x7fe53c003c20 2024-01-11T09:47:53.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.792+0000 7fe551d9b700 1 -- 172.21.15.27:0/74305524 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54400cd00 con 0x7fe53c003c20 2024-01-11T09:47:53.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.792+0000 7fe551d9b700 1 -- 172.21.15.27:0/74305524 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe544005b70 con 0x7fe53c003c20 2024-01-11T09:47:53.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.792+0000 7fe5591ff700 1 -- 172.21.15.27:0/74305524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53c003c20 msgr2=0x7fe53c004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:53.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.792+0000 7fe5591ff700 1 --2- 172.21.15.27:0/74305524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53c003c20 0x7fe53c004010 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7fe54400c2d0 tx=0x7fe544001e10 comp rx=0 tx=0).stop 2024-01-11T09:47:53.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.792+0000 7fe55359e700 1 -- 172.21.15.27:0/74305524 reap_dead start 2024-01-11T09:47:53.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.792+0000 7fe5591ff700 1 -- 172.21.15.27:0/74305524 shutdown_connections 2024-01-11T09:47:53.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.792+0000 7fe5591ff700 1 -- 172.21.15.27:0/74305524 >> 172.21.15.27:0/74305524 conn(0x7fe554063ca0 msgr2=0x7fe55406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:53.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.792+0000 7fe5591ff700 1 -- 172.21.15.27:0/74305524 shutdown_connections 2024-01-11T09:47:53.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.792+0000 7fe5591ff700 1 -- 172.21.15.27:0/74305524 wait complete. 2024-01-11T09:47:53.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.794+0000 7fe5591ff700 1 Processor -- start 2024-01-11T09:47:53.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.794+0000 7fe5591ff700 1 -- start start 2024-01-11T09:47:53.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.795+0000 7fe5591ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53c003c20 0x7fe5540c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:53.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.795+0000 7fe5591ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe554140150 0x7fe55415c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:53.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.795+0000 7fe5591ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe55415c7b0 0x7fe5540b3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:53.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.795+0000 7fe5591ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55415cfe0 con 0x7fe53c003c20 2024-01-11T09:47:53.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.795+0000 7fe5591ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5540b2db0 con 0x7fe554140150 2024-01-11T09:47:53.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.795+0000 7fe5591ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5540b2480 con 0x7fe55415c7b0 2024-01-11T09:47:53.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.795+0000 7fe552d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53c003c20 0x7fe5540c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:53.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.795+0000 7fe552d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53c003c20 0x7fe5540c7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60694/0 (socket says 172.21.15.27:60694) 2024-01-11T09:47:53.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.795+0000 7fe552d9d700 1 -- 172.21.15.27:0/4062352159 learned_addr learned my addr 172.21.15.27:0/4062352159 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:53.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.795+0000 7fe55259c700 1 --2- 172.21.15.27:0/4062352159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe554140150 0x7fe55415c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:53.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.795+0000 7fe55359e700 1 --2- 172.21.15.27:0/4062352159 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe55415c7b0 0x7fe5540b3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:53.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.795+0000 7fe552d9d700 1 -- 172.21.15.27:0/4062352159 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe55415c7b0 msgr2=0x7fe5540b3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:53.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.795+0000 7fe552d9d700 1 --2- 172.21.15.27:0/4062352159 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe55415c7b0 0x7fe5540b3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:53.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.795+0000 7fe552d9d700 1 -- 172.21.15.27:0/4062352159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe554140150 msgr2=0x7fe55415c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:53.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.795+0000 7fe552d9d700 1 --2- 172.21.15.27:0/4062352159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe554140150 0x7fe55415c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:53.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.795+0000 7fe552d9d700 1 -- 172.21.15.27:0/4062352159 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5540b2760 con 0x7fe53c003c20 2024-01-11T09:47:53.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.795+0000 7fe55259c700 1 --2- 172.21.15.27:0/4062352159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe554140150 0x7fe55415c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:53.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.796+0000 7fe552d9d700 1 --2- 172.21.15.27:0/4062352159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53c003c20 0x7fe5540c7ed0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fe544007c90 tx=0x7fe544015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:53.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.796+0000 7fe53affd700 1 -- 172.21.15.27:0/4062352159 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe544000cb0 con 0x7fe53c003c20 2024-01-11T09:47:53.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.796+0000 7fe53affd700 1 -- 172.21.15.27:0/4062352159 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5440070d0 con 0x7fe53c003c20 2024-01-11T09:47:53.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.796+0000 7fe53affd700 1 -- 172.21.15.27:0/4062352159 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe544007490 con 0x7fe53c003c20 2024-01-11T09:47:53.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.796+0000 7fe5591ff700 1 -- 172.21.15.27:0/4062352159 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5540b2a40 con 0x7fe53c003c20 2024-01-11T09:47:53.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.796+0000 7fe5591ff700 1 -- 172.21.15.27:0/4062352159 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5540b3780 con 0x7fe53c003c20 2024-01-11T09:47:53.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.797+0000 7fe5591ff700 1 -- 172.21.15.27:0/4062352159 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5540b3090 con 0x7fe53c003c20 2024-01-11T09:47:53.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.798+0000 7fe53affd700 1 -- 172.21.15.27:0/4062352159 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 153) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fe544019030 con 0x7fe53c003c20 2024-01-11T09:47:53.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.800+0000 7fe53affd700 1 --2- 172.21.15.27:0/4062352159 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe540066ea0 0x7fe540069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:53.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.800+0000 7fe53affd700 1 -- 172.21.15.27:0/4062352159 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7fe5440853d0 con 0x7fe53c003c20 2024-01-11T09:47:53.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.801+0000 7fe55259c700 1 --2- 172.21.15.27:0/4062352159 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe540066ea0 0x7fe540069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:53.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.801+0000 7fe55259c700 1 --2- 172.21.15.27:0/4062352159 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe540066ea0 0x7fe540069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe55416d2b0 tx=0x7fe548008040 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:53.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:53.804+0000 7fe53affd700 1 -- 172.21.15.27:0/4062352159 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe5440526d0 con 0x7fe53c003c20 2024-01-11T09:47:53.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.005+0000 7fe5591ff700 1 -- 172.21.15.27:0/4062352159 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fe55415cfe0 con 0x7fe53c003c20 2024-01-11T09:47:54.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.009+0000 7fe53affd700 1 -- 172.21.15.27:0/4062352159 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v91) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe544055cf0 con 0x7fe53c003c20 2024-01-11T09:47:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.011+0000 7fe5591ff700 1 -- 172.21.15.27:0/4062352159 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe540066ea0 msgr2=0x7fe540069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.011+0000 7fe5591ff700 1 --2- 172.21.15.27:0/4062352159 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe540066ea0 0x7fe540069350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe55416d2b0 tx=0x7fe548008040 comp rx=0 tx=0).stop 2024-01-11T09:47:54.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.012+0000 7fe5591ff700 1 -- 172.21.15.27:0/4062352159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53c003c20 msgr2=0x7fe5540c7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:54.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.012+0000 7fe5591ff700 1 --2- 172.21.15.27:0/4062352159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53c003c20 0x7fe5540c7ed0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fe544007c90 tx=0x7fe544015000 comp rx=0 tx=0).stop 2024-01-11T09:47:54.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.012+0000 7fe5591ff700 1 -- 172.21.15.27:0/4062352159 shutdown_connections 2024-01-11T09:47:54.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.012+0000 7fe5591ff700 1 --2- 172.21.15.27:0/4062352159 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe55415c7b0 0x7fe5540b3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.012+0000 7fe5591ff700 1 --2- 172.21.15.27:0/4062352159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53c003c20 0x7fe5540c7ed0 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.012+0000 7fe5591ff700 1 --2- 172.21.15.27:0/4062352159 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe540066ea0 0x7fe540069350 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.012+0000 7fe5591ff700 1 --2- 172.21.15.27:0/4062352159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe554140150 0x7fe55415c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.012+0000 7fe5591ff700 1 -- 172.21.15.27:0/4062352159 >> 172.21.15.27:0/4062352159 conn(0x7fe554063ca0 msgr2=0x7fe554141f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:54.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.012+0000 7fe5591ff700 1 -- 172.21.15.27:0/4062352159 shutdown_connections 2024-01-11T09:47:54.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.012+0000 7fe5591ff700 1 -- 172.21.15.27:0/4062352159 wait complete. 2024-01-11T09:47:54.022 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-01-11T09:47:54.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.231+0000 7f61d33ff700 1 Processor -- start 2024-01-11T09:47:54.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.231+0000 7f61d33ff700 1 -- start start 2024-01-11T09:47:54.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.233+0000 7f61d33ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f61cc069bd0 0x7f61cc0aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:54.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.233+0000 7f61d33ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61cc06e480 con 0x7f61cc0c6de0 2024-01-11T09:47:54.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.233+0000 7f61d33ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61cc0c7a60 con 0x7f61cc155c50 2024-01-11T09:47:54.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.233+0000 7f61d33ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc0cb5a0 con 0x7f61cc069bd0 2024-01-11T09:47:54.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.233+0000 7f61d199c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f61cc0c6de0 0x7f61cc0c7280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46274/0 (socket says 172.21.15.27:46274) 2024-01-11T09:47:54.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.233+0000 7f61d199c700 1 -- 172.21.15.27:0/3452758009 learned_addr learned my addr 172.21.15.27:0/3452758009 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:54.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.233+0000 7f61d099a700 1 --2- 172.21.15.27:0/3452758009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f61cc069bd0 0x7f61cc0aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:54.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.234+0000 7f61c3fff700 1 -- 172.21.15.27:0/3452758009 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506593698 0 0) 0x7f61cc06e480 con 0x7f61cc0c6de0 2024-01-11T09:47:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.234+0000 7f61c3fff700 1 -- 172.21.15.27:0/3452758009 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61b8003570 con 0x7f61cc0c6de0 2024-01-11T09:47:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.234+0000 7f61c3fff700 1 -- 172.21.15.27:0/3452758009 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571173648 0 0) 0x7f61cc0c7a60 con 0x7f61cc155c50 2024-01-11T09:47:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.234+0000 7f61c3fff700 1 -- 172.21.15.27:0/3452758009 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61cc06e480 con 0x7f61cc155c50 2024-01-11T09:47:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.234+0000 7f61d099a700 1 -- 172.21.15.27:0/3452758009 >> v1:172.21.15.77:6789/0 conn(0x7f61cc155c50 legacy=0x7f61cc0694c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.234+0000 7f61d099a700 1 -- 172.21.15.27:0/3452758009 >> v1:172.21.15.27:6789/0 conn(0x7f61cc0c6de0 legacy=0x7f61cc0c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.234+0000 7f61d099a700 1 -- 172.21.15.27:0/3452758009 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61cc0c7d40 con 0x7f61cc069bd0 2024-01-11T09:47:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.234+0000 7f61d099a700 1 --2- 172.21.15.27:0/3452758009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f61cc069bd0 0x7f61cc0aef80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f61bc00aac0 tx=0x7f61bc00a330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fbcf45f3dad9f71 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.235+0000 7f61c3fff700 1 -- 172.21.15.27:0/3452758009 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc00ca00 con 0x7f61cc069bd0 2024-01-11T09:47:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.235+0000 7f61c3fff700 1 -- 172.21.15.27:0/3452758009 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61bc00cb60 con 0x7f61cc069bd0 2024-01-11T09:47:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.235+0000 7f61c3fff700 1 -- 172.21.15.27:0/3452758009 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc018400 con 0x7f61cc069bd0 2024-01-11T09:47:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.235+0000 7f61d33ff700 1 -- 172.21.15.27:0/3452758009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f61cc069bd0 msgr2=0x7f61cc0aef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.235+0000 7f61d33ff700 1 --2- 172.21.15.27:0/3452758009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f61cc069bd0 0x7f61cc0aef80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f61bc00aac0 tx=0x7f61bc00a330 comp rx=0 tx=0).stop 2024-01-11T09:47:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.235+0000 7f61d33ff700 1 -- 172.21.15.27:0/3452758009 shutdown_connections 2024-01-11T09:47:54.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.235+0000 7f61d33ff700 1 --2- 172.21.15.27:0/3452758009 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f61cc069bd0 0x7f61cc0aef80 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.235+0000 7f61d33ff700 1 -- 172.21.15.27:0/3452758009 >> 172.21.15.27:0/3452758009 conn(0x7f61cc0680a0 msgr2=0x7f61cc0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:54.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.235+0000 7f61d33ff700 1 -- 172.21.15.27:0/3452758009 shutdown_connections 2024-01-11T09:47:54.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.235+0000 7f61d33ff700 1 -- 172.21.15.27:0/3452758009 wait complete. 2024-01-11T09:47:54.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.237+0000 7f61d33ff700 1 Processor -- start 2024-01-11T09:47:54.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.237+0000 7f61d33ff700 1 -- start start 2024-01-11T09:47:54.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.238+0000 7f61d33ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f61cc069bd0 0x7f61cc145bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:54.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.238+0000 7f61d33ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61cc0c6de0 0x7f61cc0c3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:54.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.238+0000 7f61d33ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61cc155c50 0x7f61cc0ab020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:54.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.238+0000 7f61d33ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc0ad200 con 0x7f61cc155c50 2024-01-11T09:47:54.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.238+0000 7f61d33ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc0add60 con 0x7f61cc0c6de0 2024-01-11T09:47:54.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.238+0000 7f61d33ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc0ad430 con 0x7f61cc069bd0 2024-01-11T09:47:54.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.238+0000 7f61d199c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61cc155c50 0x7f61cc0ab020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:54.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.238+0000 7f61d199c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61cc155c50 0x7f61cc0ab020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42236/0 (socket says 172.21.15.27:42236) 2024-01-11T09:47:54.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.238+0000 7f61d199c700 1 -- 172.21.15.27:0/1245539100 learned_addr learned my addr 172.21.15.27:0/1245539100 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:54.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.238+0000 7f61d099a700 1 --2- 172.21.15.27:0/1245539100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61cc0c6de0 0x7f61cc0c3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:54.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.238+0000 7f61d119b700 1 --2- 172.21.15.27:0/1245539100 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f61cc069bd0 0x7f61cc145bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:54.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.239+0000 7f61d199c700 1 -- 172.21.15.27:0/1245539100 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f61cc069bd0 msgr2=0x7f61cc145bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:54.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.239+0000 7f61d199c700 1 --2- 172.21.15.27:0/1245539100 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f61cc069bd0 0x7f61cc145bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.239+0000 7f61d199c700 1 -- 172.21.15.27:0/1245539100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61cc0c6de0 msgr2=0x7f61cc0c3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:54.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.239+0000 7f61d199c700 1 --2- 172.21.15.27:0/1245539100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61cc0c6de0 0x7f61cc0c3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.239+0000 7f61d199c700 1 -- 172.21.15.27:0/1245539100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61bc00a5d0 con 0x7f61cc155c50 2024-01-11T09:47:54.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.239+0000 7f61d199c700 1 --2- 172.21.15.27:0/1245539100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61cc155c50 0x7f61cc0ab020 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f61c4001460 tx=0x7f61c4001ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:54.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.239+0000 7f61c17fa700 1 -- 172.21.15.27:0/1245539100 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c400a8b0 con 0x7f61cc155c50 2024-01-11T09:47:54.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.239+0000 7f61c17fa700 1 -- 172.21.15.27:0/1245539100 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c400aa10 con 0x7f61cc155c50 2024-01-11T09:47:54.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.239+0000 7f61d33ff700 1 -- 172.21.15.27:0/1245539100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61cc0ad710 con 0x7f61cc155c50 2024-01-11T09:47:54.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.239+0000 7f61c17fa700 1 -- 172.21.15.27:0/1245539100 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c400adb0 con 0x7f61cc155c50 2024-01-11T09:47:54.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.239+0000 7f61d33ff700 1 -- 172.21.15.27:0/1245539100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61cc146130 con 0x7f61cc155c50 2024-01-11T09:47:54.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.241+0000 7f61d33ff700 1 -- 172.21.15.27:0/1245539100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61cc04c3c0 con 0x7f61cc155c50 2024-01-11T09:47:54.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.241+0000 7f61c17fa700 1 -- 172.21.15.27:0/1245539100 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 153) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f61c4007300 con 0x7f61cc155c50 2024-01-11T09:47:54.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.242+0000 7f61c17fa700 1 --2- 172.21.15.27:0/1245539100 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f61b0066ea0 0x7f61b0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:54.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.242+0000 7f61c17fa700 1 -- 172.21.15.27:0/1245539100 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f61c40847b0 con 0x7f61cc155c50 2024-01-11T09:47:54.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.243+0000 7f61d119b700 1 --2- 172.21.15.27:0/1245539100 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f61b0066ea0 0x7f61b0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:54.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.243+0000 7f61d119b700 1 --2- 172.21.15.27:0/1245539100 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f61b0066ea0 0x7f61b0069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f61c8003a40 tx=0x7f61c8007d40 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:54.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.246+0000 7f61c17fa700 1 -- 172.21.15.27:0/1245539100 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f61c4051b30 con 0x7f61cc155c50 2024-01-11T09:47:54.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.448+0000 7f61d33ff700 1 -- 172.21.15.27:0/1245539100 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f61cc0ccaa0 con 0x7f61cc155c50 2024-01-11T09:47:54.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.452+0000 7f61c17fa700 1 -- 172.21.15.27:0/1245539100 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v92) v1 ==== 115+0+0 (secure 0 0 0) 0x7f61c4055150 con 0x7f61cc155c50 2024-01-11T09:47:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.454+0000 7f61d33ff700 1 -- 172.21.15.27:0/1245539100 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f61b0066ea0 msgr2=0x7f61b0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.454+0000 7f61d33ff700 1 --2- 172.21.15.27:0/1245539100 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f61b0066ea0 0x7f61b0069350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f61c8003a40 tx=0x7f61c8007d40 comp rx=0 tx=0).stop 2024-01-11T09:47:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.454+0000 7f61d33ff700 1 -- 172.21.15.27:0/1245539100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61cc155c50 msgr2=0x7f61cc0ab020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.454+0000 7f61d33ff700 1 --2- 172.21.15.27:0/1245539100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61cc155c50 0x7f61cc0ab020 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f61c4001460 tx=0x7f61c4001ac0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.454+0000 7f61d33ff700 1 -- 172.21.15.27:0/1245539100 shutdown_connections 2024-01-11T09:47:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.454+0000 7f61d33ff700 1 --2- 172.21.15.27:0/1245539100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61cc155c50 0x7f61cc0ab020 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.454+0000 7f61d33ff700 1 --2- 172.21.15.27:0/1245539100 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f61cc069bd0 0x7f61cc145bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.454+0000 7f61d33ff700 1 --2- 172.21.15.27:0/1245539100 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f61b0066ea0 0x7f61b0069350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.454+0000 7f61d33ff700 1 --2- 172.21.15.27:0/1245539100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61cc0c6de0 0x7f61cc0c3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.454+0000 7f61d33ff700 1 -- 172.21.15.27:0/1245539100 >> 172.21.15.27:0/1245539100 conn(0x7f61cc0680a0 msgr2=0x7f61cc0c2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.454+0000 7f61d33ff700 1 -- 172.21.15.27:0/1245539100 shutdown_connections 2024-01-11T09:47:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.454+0000 7f61d33ff700 1 -- 172.21.15.27:0/1245539100 wait complete. 2024-01-11T09:47:54.464 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-11T09:47:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.673+0000 7f5d6d3ff700 1 Processor -- start 2024-01-11T09:47:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.674+0000 7f5d6d3ff700 1 -- start start 2024-01-11T09:47:54.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.675+0000 7f5d6d3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d6815a1b0 0x7f5d6815a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.675+0000 7f5d6d3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5d680ccd70 0x7f5d680cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.675+0000 7f5d6d3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d680cf850 con 0x7f5d680685c0 2024-01-11T09:47:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.675+0000 7f5d6d3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d680cec40 con 0x7f5d680ccd70 2024-01-11T09:47:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.675+0000 7f5d6d3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d680cef20 con 0x7f5d6815a1b0 2024-01-11T09:47:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.676+0000 7f5d667fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5d680ccd70 0x7f5d680cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.676+0000 7f5d667fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5d680ccd70 0x7f5d680cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42252/0 (socket says 172.21.15.27:42252) 2024-01-11T09:47:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.676+0000 7f5d667fc700 1 -- 172.21.15.27:0/229034327 learned_addr learned my addr 172.21.15.27:0/229034327 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.676+0000 7f5d66ffd700 1 --2- 172.21.15.27:0/229034327 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d6815a1b0 0x7f5d6815a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.676+0000 7f5d667fc700 1 -- 172.21.15.27:0/229034327 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d6815a1b0 msgr2=0x7f5d6815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.676+0000 7f5d667fc700 1 --2- 172.21.15.27:0/229034327 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d6815a1b0 0x7f5d6815a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.676+0000 7f5d667fc700 1 -- 172.21.15.27:0/229034327 >> v1:172.21.15.77:6789/0 conn(0x7f5d680685c0 legacy=0x7f5d68068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.676+0000 7f5d667fc700 1 -- 172.21.15.27:0/229034327 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d680cf200 con 0x7f5d680ccd70 2024-01-11T09:47:54.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.676+0000 7f5d66ffd700 1 --2- 172.21.15.27:0/229034327 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d6815a1b0 0x7f5d6815a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:54.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.677+0000 7f5d667fc700 1 --2- 172.21.15.27:0/229034327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d680ccd70 0x7f5d680cd1e0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c0099b0 tx=0x7f5d5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c466ed3971285912 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:54.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.677+0000 7f5d65ffb700 1 -- 172.21.15.27:0/229034327 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c00b860 con 0x7f5d680ccd70 2024-01-11T09:47:54.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.677+0000 7f5d65ffb700 1 -- 172.21.15.27:0/229034327 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d5c00b9c0 con 0x7f5d680ccd70 2024-01-11T09:47:54.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.677+0000 7f5d65ffb700 1 -- 172.21.15.27:0/229034327 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c00bd60 con 0x7f5d680ccd70 2024-01-11T09:47:54.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.677+0000 7f5d6d3ff700 1 -- 172.21.15.27:0/229034327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d680ccd70 msgr2=0x7f5d680cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:54.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.677+0000 7f5d6d3ff700 1 --2- 172.21.15.27:0/229034327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d680ccd70 0x7f5d680cd1e0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c0099b0 tx=0x7f5d5c009250 comp rx=0 tx=0).stop 2024-01-11T09:47:54.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.677+0000 7f5d6d3ff700 1 -- 172.21.15.27:0/229034327 shutdown_connections 2024-01-11T09:47:54.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.677+0000 7f5d6d3ff700 1 --2- 172.21.15.27:0/229034327 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d6815a1b0 0x7f5d6815a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.677+0000 7f5d6d3ff700 1 --2- 172.21.15.27:0/229034327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d680ccd70 0x7f5d680cd1e0 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.677+0000 7f5d6d3ff700 1 -- 172.21.15.27:0/229034327 >> 172.21.15.27:0/229034327 conn(0x7f5d6806ed40 msgr2=0x7f5d6806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:54.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.677+0000 7f5d6d3ff700 1 -- 172.21.15.27:0/229034327 shutdown_connections 2024-01-11T09:47:54.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.677+0000 7f5d6d3ff700 1 -- 172.21.15.27:0/229034327 wait complete. 2024-01-11T09:47:54.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.679+0000 7f5d6d3ff700 1 Processor -- start 2024-01-11T09:47:54.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.679+0000 7f5d6d3ff700 1 -- start start 2024-01-11T09:47:54.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.680+0000 7f5d6d3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d680685c0 0x7f5d68158f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:54.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.680+0000 7f5d6d3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d680ccd70 0x7f5d680c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:54.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.680+0000 7f5d6d3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d6815a1b0 0x7f5d68152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.680+0000 7f5d6d3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d68159460 con 0x7f5d680685c0 2024-01-11T09:47:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.680+0000 7f5d6d3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d68158a90 con 0x7f5d6815a1b0 2024-01-11T09:47:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.680+0000 7f5d6d3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d68157ba0 con 0x7f5d680ccd70 2024-01-11T09:47:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.680+0000 7f5d66ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d680685c0 0x7f5d68158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.680+0000 7f5d66ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d680685c0 0x7f5d68158f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42256/0 (socket says 172.21.15.27:42256) 2024-01-11T09:47:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.680+0000 7f5d66ffd700 1 -- 172.21.15.27:0/1974759146 learned_addr learned my addr 172.21.15.27:0/1974759146 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.681+0000 7f5d667fc700 1 --2- 172.21.15.27:0/1974759146 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d680ccd70 0x7f5d680c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.681+0000 7f5d677fe700 1 --2- 172.21.15.27:0/1974759146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d6815a1b0 0x7f5d68152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.681+0000 7f5d66ffd700 1 -- 172.21.15.27:0/1974759146 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d680ccd70 msgr2=0x7f5d680c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.681+0000 7f5d66ffd700 1 --2- 172.21.15.27:0/1974759146 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d680ccd70 0x7f5d680c8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.681+0000 7f5d66ffd700 1 -- 172.21.15.27:0/1974759146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d6815a1b0 msgr2=0x7f5d68152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.681+0000 7f5d66ffd700 1 --2- 172.21.15.27:0/1974759146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d6815a1b0 0x7f5d68152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.681+0000 7f5d66ffd700 1 -- 172.21.15.27:0/1974759146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5c0094f0 con 0x7f5d680685c0 2024-01-11T09:47:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.681+0000 7f5d66ffd700 1 --2- 172.21.15.27:0/1974759146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d680685c0 0x7f5d68158f20 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f5d5800e8d0 tx=0x7f5d58008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.681+0000 7f5d4effd700 1 -- 172.21.15.27:0/1974759146 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d58013410 con 0x7f5d680685c0 2024-01-11T09:47:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.682+0000 7f5d4effd700 1 -- 172.21.15.27:0/1974759146 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d58013570 con 0x7f5d680685c0 2024-01-11T09:47:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.682+0000 7f5d4effd700 1 -- 172.21.15.27:0/1974759146 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d58013910 con 0x7f5d680685c0 2024-01-11T09:47:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.682+0000 7f5d6d3ff700 1 -- 172.21.15.27:0/1974759146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d68157e80 con 0x7f5d680685c0 2024-01-11T09:47:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.682+0000 7f5d6d3ff700 1 -- 172.21.15.27:0/1974759146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d68158440 con 0x7f5d680685c0 2024-01-11T09:47:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.683+0000 7f5d6d3ff700 1 -- 172.21.15.27:0/1974759146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d6804c3c0 con 0x7f5d680685c0 2024-01-11T09:47:54.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.684+0000 7f5d4effd700 1 -- 172.21.15.27:0/1974759146 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 153) v1 ==== 84686+0+0 (secure 0 0 0) 0x7f5d58013cc0 con 0x7f5d680685c0 2024-01-11T09:47:54.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.685+0000 7f5d4effd700 1 --2- 172.21.15.27:0/1974759146 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5d54066ea0 0x7f5d54069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:54.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.685+0000 7f5d4effd700 1 -- 172.21.15.27:0/1974759146 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f5d58084150 con 0x7f5d680685c0 2024-01-11T09:47:54.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.685+0000 7f5d667fc700 1 --2- 172.21.15.27:0/1974759146 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5d54066ea0 0x7f5d54069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:54.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.686+0000 7f5d667fc700 1 --2- 172.21.15.27:0/1974759146 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5d54066ea0 0x7f5d54069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c0099b0 tx=0x7f5d5c017000 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:54.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.689+0000 7f5d4effd700 1 -- 172.21.15.27:0/1974759146 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5d58051450 con 0x7f5d680685c0 2024-01-11T09:47:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.896+0000 7f5d6d3ff700 1 -- 172.21.15.27:0/1974759146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f5d680c6590 con 0x7f5d680685c0 2024-01-11T09:47:54.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.901+0000 7f5d4effd700 1 -- 172.21.15.27:0/1974759146 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v93) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5d58054a70 con 0x7f5d680685c0 2024-01-11T09:47:54.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.903+0000 7f5d6d3ff700 1 -- 172.21.15.27:0/1974759146 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5d54066ea0 msgr2=0x7f5d54069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:54.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.903+0000 7f5d6d3ff700 1 --2- 172.21.15.27:0/1974759146 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5d54066ea0 0x7f5d54069350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c0099b0 tx=0x7f5d5c017000 comp rx=0 tx=0).stop 2024-01-11T09:47:54.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.903+0000 7f5d6d3ff700 1 -- 172.21.15.27:0/1974759146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d680685c0 msgr2=0x7f5d68158f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:54.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.903+0000 7f5d6d3ff700 1 --2- 172.21.15.27:0/1974759146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d680685c0 0x7f5d68158f20 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f5d5800e8d0 tx=0x7f5d58008040 comp rx=0 tx=0).stop 2024-01-11T09:47:54.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.903+0000 7f5d6d3ff700 1 -- 172.21.15.27:0/1974759146 shutdown_connections 2024-01-11T09:47:54.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.903+0000 7f5d6d3ff700 1 --2- 172.21.15.27:0/1974759146 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f5d54066ea0 0x7f5d54069350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.903+0000 7f5d6d3ff700 1 --2- 172.21.15.27:0/1974759146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d6815a1b0 0x7f5d68152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.903+0000 7f5d6d3ff700 1 --2- 172.21.15.27:0/1974759146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d680685c0 0x7f5d68158f20 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.903+0000 7f5d6d3ff700 1 --2- 172.21.15.27:0/1974759146 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f5d680ccd70 0x7f5d680c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:54.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.903+0000 7f5d6d3ff700 1 -- 172.21.15.27:0/1974759146 >> 172.21.15.27:0/1974759146 conn(0x7f5d6806ed40 msgr2=0x7f5d68156080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:54.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.903+0000 7f5d6d3ff700 1 -- 172.21.15.27:0/1974759146 shutdown_connections 2024-01-11T09:47:54.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:54.903+0000 7f5d6d3ff700 1 -- 172.21.15.27:0/1974759146 wait complete. 2024-01-11T09:47:54.913 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2024-01-11T09:47:55.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.121+0000 7fea31d9d700 1 Processor -- start 2024-01-11T09:47:55.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.122+0000 7fea31d9d700 1 -- start start 2024-01-11T09:47:55.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.123+0000 7fea31d9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fea2c0aee70 0x7fea2c0ad720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:55.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.123+0000 7fea31d9d700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea2c06e5c0 con 0x7fea2c1572b0 2024-01-11T09:47:55.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.123+0000 7fea31d9d700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea2c06d9b0 con 0x7fea2c157680 2024-01-11T09:47:55.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.123+0000 7fea31d9d700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea2c06dc90 con 0x7fea2c0aee70 2024-01-11T09:47:55.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.123+0000 7fea30d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fea2c1572b0 0x7fea2c069680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46284/0 (socket says 172.21.15.27:46284) 2024-01-11T09:47:55.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.123+0000 7fea30d9b700 1 -- 172.21.15.27:0/174232255 learned_addr learned my addr 172.21.15.27:0/174232255 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:55.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.123+0000 7fea2b7fe700 1 -- 172.21.15.27:0/174232255 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2872633696 0 0) 0x7fea2c06e5c0 con 0x7fea2c1572b0 2024-01-11T09:47:55.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.124+0000 7fea2b7fe700 1 -- 172.21.15.27:0/174232255 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea1c003570 con 0x7fea2c1572b0 2024-01-11T09:47:55.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.124+0000 7fea2bfff700 1 --2- 172.21.15.27:0/174232255 >> v2:172.21.15.77:3300/0 conn(0x7fea2c0aee70 0x7fea2c0ad720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.124+0000 7fea2b7fe700 1 -- 172.21.15.27:0/174232255 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2138068428 0 0) 0x7fea1c003570 con 0x7fea2c1572b0 2024-01-11T09:47:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.124+0000 7fea2b7fe700 1 -- 172.21.15.27:0/174232255 >> v2:172.21.15.77:3300/0 conn(0x7fea2c0aee70 msgr2=0x7fea2c0ad720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.124+0000 7fea2b7fe700 1 --2- 172.21.15.27:0/174232255 >> v2:172.21.15.77:3300/0 conn(0x7fea2c0aee70 0x7fea2c0ad720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.124+0000 7fea2b7fe700 1 -- 172.21.15.27:0/174232255 >> v1:172.21.15.77:6789/0 conn(0x7fea2c157680 legacy=0x7fea2c0adc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.124+0000 7fea2b7fe700 1 -- 172.21.15.27:0/174232255 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea2c06df70 con 0x7fea2c1572b0 2024-01-11T09:47:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.124+0000 7fea2b7fe700 1 -- 172.21.15.27:0/174232255 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fea20001470 con 0x7fea2c1572b0 2024-01-11T09:47:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.124+0000 7fea2bfff700 1 --2- 172.21.15.27:0/174232255 >> v2:172.21.15.77:3300/0 conn(0x7fea2c0aee70 0x7fea2c0ad720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.124+0000 7fea2b7fe700 1 -- 172.21.15.27:0/174232255 >> v1:172.21.15.27:6789/0 conn(0x7fea2c1572b0 legacy=0x7fea2c069680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.124+0000 7fea2b7fe700 1 --2- 172.21.15.27:0/174232255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1c003a60 0x7fea1c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.124+0000 7fea2b7fe700 1 --2- 172.21.15.27:0/174232255 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea1c004810 0x7fea1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.124+0000 7fea2b7fe700 1 --2- 172.21.15.27:0/174232255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea1c0053c0 0x7fea1c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.124+0000 7fea2b7fe700 1 -- 172.21.15.27:0/174232255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2c06dc90 con 0x7fea1c003a60 2024-01-11T09:47:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.124+0000 7fea2b7fe700 1 -- 172.21.15.27:0/174232255 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c00c2d0 con 0x7fea1c0053c0 2024-01-11T09:47:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.124+0000 7fea2b7fe700 1 -- 172.21.15.27:0/174232255 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c00c490 con 0x7fea1c004810 2024-01-11T09:47:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.124+0000 7fea30d9b700 1 --2- 172.21.15.27:0/174232255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1c003a60 0x7fea1c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.124+0000 7fea2bfff700 1 --2- 172.21.15.27:0/174232255 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea1c004810 0x7fea1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.125+0000 7fea3159c700 1 --2- 172.21.15.27:0/174232255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea1c0053c0 0x7fea1c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.125+0000 7fea30d9b700 1 -- 172.21.15.27:0/174232255 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea1c004810 msgr2=0x7fea1c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.125+0000 7fea30d9b700 1 --2- 172.21.15.27:0/174232255 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea1c004810 0x7fea1c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.125+0000 7fea30d9b700 1 -- 172.21.15.27:0/174232255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea1c0053c0 msgr2=0x7fea1c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.125+0000 7fea30d9b700 1 --2- 172.21.15.27:0/174232255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea1c0053c0 0x7fea1c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.125+0000 7fea3159c700 1 -- 172.21.15.27:0/174232255 reap_dead start 2024-01-11T09:47:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.125+0000 7fea30d9b700 1 -- 172.21.15.27:0/174232255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1c003570 con 0x7fea1c003a60 2024-01-11T09:47:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.125+0000 7fea30d9b700 1 --2- 172.21.15.27:0/174232255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1c003a60 0x7fea1c003e70 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fea2000c3c0 tx=0x7fea20001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbefe0e2b36e4aa6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.125+0000 7fea2b7fe700 1 -- 172.21.15.27:0/174232255 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2000cc90 con 0x7fea1c003a60 2024-01-11T09:47:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.125+0000 7fea2b7fe700 1 -- 172.21.15.27:0/174232255 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2000cdf0 con 0x7fea1c003a60 2024-01-11T09:47:55.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.125+0000 7fea2b7fe700 1 -- 172.21.15.27:0/174232255 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea20005b50 con 0x7fea1c003a60 2024-01-11T09:47:55.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.126+0000 7fea31d9d700 1 -- 172.21.15.27:0/174232255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1c003a60 msgr2=0x7fea1c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:55.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.126+0000 7fea31d9d700 1 --2- 172.21.15.27:0/174232255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1c003a60 0x7fea1c003e70 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fea2000c3c0 tx=0x7fea20001e50 comp rx=0 tx=0).stop 2024-01-11T09:47:55.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.126+0000 7fea31d9d700 1 -- 172.21.15.27:0/174232255 shutdown_connections 2024-01-11T09:47:55.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.126+0000 7fea31d9d700 1 --2- 172.21.15.27:0/174232255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1c003a60 0x7fea1c003e70 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:55.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.126+0000 7fea31d9d700 1 -- 172.21.15.27:0/174232255 >> 172.21.15.27:0/174232255 conn(0x7fea2c0680a0 msgr2=0x7fea2c0cc020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:55.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.126+0000 7fea31d9d700 1 -- 172.21.15.27:0/174232255 shutdown_connections 2024-01-11T09:47:55.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.126+0000 7fea31d9d700 1 -- 172.21.15.27:0/174232255 wait complete. 2024-01-11T09:47:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.128+0000 7fea31d9d700 1 Processor -- start 2024-01-11T09:47:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.128+0000 7fea31d9d700 1 -- start start 2024-01-11T09:47:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.129+0000 7fea31d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea1c003a60 0x7fea2c15a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.129+0000 7fea31d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea2c155270 0x7fea2c1556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.129+0000 7fea31d9d700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea2c0c0980 0x7fea2c0c0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.129+0000 7fea31d9d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2c16db60 con 0x7fea2c155270 2024-01-11T09:47:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.129+0000 7fea31d9d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2c16cf90 con 0x7fea1c003a60 2024-01-11T09:47:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.129+0000 7fea31d9d700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea2c16d230 con 0x7fea2c0c0980 2024-01-11T09:47:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.129+0000 7fea2bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea2c155270 0x7fea2c1556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.129+0000 7fea2bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea2c155270 0x7fea2c1556e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42280/0 (socket says 172.21.15.27:42280) 2024-01-11T09:47:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.129+0000 7fea2bfff700 1 -- 172.21.15.27:0/2905916902 learned_addr learned my addr 172.21.15.27:0/2905916902 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.129+0000 7fea30d9b700 1 --2- 172.21.15.27:0/2905916902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea1c003a60 0x7fea2c15a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.129+0000 7fea3159c700 1 --2- 172.21.15.27:0/2905916902 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea2c0c0980 0x7fea2c0c0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.129+0000 7fea2bfff700 1 -- 172.21.15.27:0/2905916902 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea2c0c0980 msgr2=0x7fea2c0c0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.129+0000 7fea2bfff700 1 --2- 172.21.15.27:0/2905916902 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea2c0c0980 0x7fea2c0c0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.129+0000 7fea2bfff700 1 -- 172.21.15.27:0/2905916902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea1c003a60 msgr2=0x7fea2c15a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.129+0000 7fea2bfff700 1 --2- 172.21.15.27:0/2905916902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea1c003a60 0x7fea2c15a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.129+0000 7fea2bfff700 1 -- 172.21.15.27:0/2905916902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea2c16d510 con 0x7fea2c155270 2024-01-11T09:47:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.129+0000 7fea30d9b700 1 --2- 172.21.15.27:0/2905916902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea1c003a60 0x7fea2c15a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.129+0000 7fea3159c700 1 --2- 172.21.15.27:0/2905916902 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea2c0c0980 0x7fea2c0c0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.130+0000 7fea2bfff700 1 --2- 172.21.15.27:0/2905916902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea2c155270 0x7fea2c1556e0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7fea180116e0 tx=0x7fea18008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.130+0000 7fea28ff9700 1 -- 172.21.15.27:0/2905916902 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1800c070 con 0x7fea2c155270 2024-01-11T09:47:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.130+0000 7fea28ff9700 1 -- 172.21.15.27:0/2905916902 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea18015740 con 0x7fea2c155270 2024-01-11T09:47:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.130+0000 7fea28ff9700 1 -- 172.21.15.27:0/2905916902 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea18015b40 con 0x7fea2c155270 2024-01-11T09:47:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.130+0000 7fea31d9d700 1 -- 172.21.15.27:0/2905916902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea2c16d7f0 con 0x7fea2c155270 2024-01-11T09:47:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.130+0000 7fea31d9d700 1 -- 172.21.15.27:0/2905916902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea2c0c30e0 con 0x7fea2c155270 2024-01-11T09:47:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.131+0000 7fea31d9d700 1 -- 172.21.15.27:0/2905916902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea2c04c3c0 con 0x7fea2c155270 2024-01-11T09:47:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.132+0000 7fea28ff9700 1 -- 172.21.15.27:0/2905916902 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 153) v1 ==== 84686+0+0 (secure 0 0 0) 0x7fea18013410 con 0x7fea2c155270 2024-01-11T09:47:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.133+0000 7fea28ff9700 1 --2- 172.21.15.27:0/2905916902 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fea10066ed0 0x7fea10069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.133+0000 7fea28ff9700 1 -- 172.21.15.27:0/2905916902 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7fea18084760 con 0x7fea2c155270 2024-01-11T09:47:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.134+0000 7fea30d9b700 1 --2- 172.21.15.27:0/2905916902 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fea10066ed0 0x7fea10069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.134+0000 7fea30d9b700 1 --2- 172.21.15.27:0/2905916902 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fea10066ed0 0x7fea10069380 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fea20001ed0 tx=0x7fea20015000 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:55.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.138+0000 7fea28ff9700 1 -- 172.21.15.27:0/2905916902 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fea180519c0 con 0x7fea2c155270 2024-01-11T09:47:55.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.425+0000 7fea31d9d700 1 -- 172.21.15.27:0/2905916902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7fea2c06e250 con 0x7fea2c155270 2024-01-11T09:47:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:55.458+0000 7fea28ff9700 1 -- 172.21.15.27:0/2905916902 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fea18051d90 con 0x7fea2c155270 2024-01-11T09:47:56.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:56.370+0000 7fea28ff9700 1 -- 172.21.15.27:0/2905916902 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3219) v1 ==== 155+0+0 (secure 0 0 0) 0x7fea180519c0 con 0x7fea2c155270 2024-01-11T09:47:56.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:56.373+0000 7fea31d9d700 1 -- 172.21.15.27:0/2905916902 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fea10066ed0 msgr2=0x7fea10069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:56.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:56.373+0000 7fea31d9d700 1 --2- 172.21.15.27:0/2905916902 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fea10066ed0 0x7fea10069380 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fea20001ed0 tx=0x7fea20015000 comp rx=0 tx=0).stop 2024-01-11T09:47:56.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:56.373+0000 7fea31d9d700 1 -- 172.21.15.27:0/2905916902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea2c155270 msgr2=0x7fea2c1556e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:56.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:56.373+0000 7fea31d9d700 1 --2- 172.21.15.27:0/2905916902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea2c155270 0x7fea2c1556e0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7fea180116e0 tx=0x7fea18008040 comp rx=0 tx=0).stop 2024-01-11T09:47:56.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:56.373+0000 7fea31d9d700 1 -- 172.21.15.27:0/2905916902 shutdown_connections 2024-01-11T09:47:56.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:56.373+0000 7fea31d9d700 1 --2- 172.21.15.27:0/2905916902 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fea2c0c0980 0x7fea2c0c0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:56.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:56.373+0000 7fea31d9d700 1 --2- 172.21.15.27:0/2905916902 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fea10066ed0 0x7fea10069380 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:56.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:56.373+0000 7fea31d9d700 1 --2- 172.21.15.27:0/2905916902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea1c003a60 0x7fea2c15a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:56.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:56.373+0000 7fea31d9d700 1 --2- 172.21.15.27:0/2905916902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea2c155270 0x7fea2c1556e0 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:56.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:56.373+0000 7fea31d9d700 1 -- 172.21.15.27:0/2905916902 >> 172.21.15.27:0/2905916902 conn(0x7fea2c0680a0 msgr2=0x7fea2c156510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:56.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:56.373+0000 7fea31d9d700 1 -- 172.21.15.27:0/2905916902 shutdown_connections 2024-01-11T09:47:56.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:56.373+0000 7fea31d9d700 1 -- 172.21.15.27:0/2905916902 wait complete. 2024-01-11T09:47:56.386 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2024-01-11T09:47:56.387 INFO:tasks.cephfs_test_runner: 2024-01-11T09:47:56.387 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-11T09:47:56.387 INFO:tasks.cephfs_test_runner:Ran 14 tests in 5811.227s 2024-01-11T09:47:56.387 INFO:tasks.cephfs_test_runner: 2024-01-11T09:47:56.387 INFO:tasks.cephfs_test_runner:OK 2024-01-11T09:47:56.387 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-01-11T09:47:56.397 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-01-11T09:47:56.442 INFO:tasks.kclient:Unmounting kernel clients... 2024-01-11T09:47:56.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:56.442 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T09:47:56.493 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-11T09:47:56.493 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-11T09:47:56.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:56.494 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-11T09:47:56.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:56.570 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-11T09:47:56.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:56.647 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2024-01-11T09:47:56.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:56.769 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-11T09:47:56.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:56.823 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2024-01-11T09:47:56.902 INFO:teuthology.orchestra.run.smithi027.stderr:Cannot find device "ceph-brx" 2024-01-11T09:47:56.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T09:47:56.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:56.905 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-01-11T09:47:56.958 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-11T09:47:56.958 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-11T09:47:56.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:56.959 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-11T09:47:57.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:57.033 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-11T09:47:57.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:57.122 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2024-01-11T09:47:57.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:57.252 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-01-11T09:47:57.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-11T09:47:57.301 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2024-01-11T09:47:57.376 INFO:teuthology.orchestra.run.smithi077.stderr:Cannot find device "ceph-brx" 2024-01-11T09:47:57.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T09:47:57.379 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-11T09:47:57.389 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-01-11T09:47:57.389 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:47:57.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.599+0000 7f27a7bff700 1 Processor -- start 2024-01-11T09:47:57.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.599+0000 7f27a7bff700 1 -- start start 2024-01-11T09:47:57.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.600+0000 7f27a7bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27a0069d40 0x7f27a00b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:57.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.600+0000 7f27a7bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f27a00c89f0 0x7f27a00b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:57.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.600+0000 7f27a7bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a0157390 con 0x7f27a00c8620 2024-01-11T09:47:57.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.600+0000 7f27a7bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a00adcd0 con 0x7f27a00c89f0 2024-01-11T09:47:57.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.600+0000 7f27a7bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a006a200 con 0x7f27a0069d40 2024-01-11T09:47:57.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.601+0000 7f27a599b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f27a00c8620 0x7f27a00af970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46294/0 (socket says 172.21.15.27:46294) 2024-01-11T09:47:57.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.601+0000 7f27a599b700 1 -- 172.21.15.27:0/1579089931 learned_addr learned my addr 172.21.15.27:0/1579089931 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:57.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.601+0000 7f27a619c700 1 --2- 172.21.15.27:0/1579089931 >> v2:172.21.15.27:3300/0 conn(0x7f27a00c89f0 0x7f27a00b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:57.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.601+0000 7f27a519a700 1 --2- 172.21.15.27:0/1579089931 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27a0069d40 0x7f27a00b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:57.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.601+0000 7f27a4999700 1 -- 172.21.15.27:0/1579089931 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647736767 0 0) 0x7f27a0157390 con 0x7f27a00c8620 2024-01-11T09:47:57.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.601+0000 7f27a4999700 1 -- 172.21.15.27:0/1579089931 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f278c003570 con 0x7f27a00c8620 2024-01-11T09:47:57.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.601+0000 7f27a519a700 1 -- 172.21.15.27:0/1579089931 >> v2:172.21.15.27:3300/0 conn(0x7f27a00c89f0 msgr2=0x7f27a00b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:57.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.601+0000 7f27a519a700 1 --2- 172.21.15.27:0/1579089931 >> v2:172.21.15.27:3300/0 conn(0x7f27a00c89f0 0x7f27a00b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:57.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.601+0000 7f27a519a700 1 -- 172.21.15.27:0/1579089931 >> v1:172.21.15.27:6789/0 conn(0x7f27a00c8620 legacy=0x7f27a00af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:57.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.601+0000 7f27a519a700 1 -- 172.21.15.27:0/1579089931 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a0156d40 con 0x7f27a0069d40 2024-01-11T09:47:57.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.601+0000 7f27a619c700 1 --2- 172.21.15.27:0/1579089931 >> v2:172.21.15.27:3300/0 conn(0x7f27a00c89f0 0x7f27a00b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:57.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.602+0000 7f27a519a700 1 --2- 172.21.15.27:0/1579089931 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27a0069d40 0x7f27a00b02d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2790009910 tx=0x7f27900091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9810998873a97e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:57.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.602+0000 7f27a4999700 1 -- 172.21.15.27:0/1579089931 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2790011630 con 0x7f27a0069d40 2024-01-11T09:47:57.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.602+0000 7f27a4999700 1 -- 172.21.15.27:0/1579089931 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2790011790 con 0x7f27a0069d40 2024-01-11T09:47:57.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.602+0000 7f27a4999700 1 -- 172.21.15.27:0/1579089931 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2790011b30 con 0x7f27a0069d40 2024-01-11T09:47:57.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.602+0000 7f27a7bff700 1 -- 172.21.15.27:0/1579089931 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27a0069d40 msgr2=0x7f27a00b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:57.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.602+0000 7f27a7bff700 1 --2- 172.21.15.27:0/1579089931 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27a0069d40 0x7f27a00b02d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2790009910 tx=0x7f27900091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:57.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.603+0000 7f27a7bff700 1 -- 172.21.15.27:0/1579089931 shutdown_connections 2024-01-11T09:47:57.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.603+0000 7f27a7bff700 1 --2- 172.21.15.27:0/1579089931 >> v2:172.21.15.27:3300/0 conn(0x7f27a00c89f0 0x7f27a00b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:57.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.603+0000 7f27a7bff700 1 --2- 172.21.15.27:0/1579089931 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27a0069d40 0x7f27a00b02d0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:57.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.603+0000 7f27a7bff700 1 -- 172.21.15.27:0/1579089931 >> 172.21.15.27:0/1579089931 conn(0x7f27a00680e0 msgr2=0x7f27a0069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:57.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.603+0000 7f27a7bff700 1 -- 172.21.15.27:0/1579089931 shutdown_connections 2024-01-11T09:47:57.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.603+0000 7f27a7bff700 1 -- 172.21.15.27:0/1579089931 wait complete. 2024-01-11T09:47:57.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.604+0000 7f27a7bff700 1 Processor -- start 2024-01-11T09:47:57.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.604+0000 7f27a7bff700 1 -- start start 2024-01-11T09:47:57.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.605+0000 7f27a7bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27a0069d40 0x7f27a016d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:57.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.605+0000 7f27a7bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27a00c8620 0x7f27a016a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:57.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.605+0000 7f27a7bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27a00c89f0 0x7f27a0158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:57.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.605+0000 7f27a7bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a016a580 con 0x7f27a00c8620 2024-01-11T09:47:57.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.605+0000 7f27a7bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a00c1f20 con 0x7f27a0069d40 2024-01-11T09:47:57.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.605+0000 7f27a7bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a00c1960 con 0x7f27a00c89f0 2024-01-11T09:47:57.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.605+0000 7f27a519a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27a00c8620 0x7f27a016a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:57.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.605+0000 7f27a519a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27a00c8620 0x7f27a016a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42288/0 (socket says 172.21.15.27:42288) 2024-01-11T09:47:57.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.605+0000 7f27a519a700 1 -- 172.21.15.27:0/3766918151 learned_addr learned my addr 172.21.15.27:0/3766918151 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:57.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.606+0000 7f27a599b700 1 --2- 172.21.15.27:0/3766918151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27a0069d40 0x7f27a016d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:57.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.606+0000 7f27a619c700 1 --2- 172.21.15.27:0/3766918151 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27a00c89f0 0x7f27a0158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:57.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.606+0000 7f27a519a700 1 -- 172.21.15.27:0/3766918151 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27a00c89f0 msgr2=0x7f27a0158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:57.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.606+0000 7f27a519a700 1 --2- 172.21.15.27:0/3766918151 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27a00c89f0 0x7f27a0158a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.606+0000 7f27a519a700 1 -- 172.21.15.27:0/3766918151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27a0069d40 msgr2=0x7f27a016d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.606+0000 7f27a519a700 1 --2- 172.21.15.27:0/3766918151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27a0069d40 0x7f27a016d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.606+0000 7f27a519a700 1 -- 172.21.15.27:0/3766918151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2798009a20 con 0x7f27a00c8620 2024-01-11T09:47:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.606+0000 7f27a519a700 1 --2- 172.21.15.27:0/3766918151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27a00c8620 0x7f27a016a040 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f2790013000 tx=0x7f2790017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.606+0000 7f2795ffb700 1 -- 172.21.15.27:0/3766918151 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2790011900 con 0x7f27a00c8620 2024-01-11T09:47:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.607+0000 7f2795ffb700 1 -- 172.21.15.27:0/3766918151 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2790011a80 con 0x7f27a00c8620 2024-01-11T09:47:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.607+0000 7f2795ffb700 1 -- 172.21.15.27:0/3766918151 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2790025670 con 0x7f27a00c8620 2024-01-11T09:47:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.607+0000 7f27a7bff700 1 -- 172.21.15.27:0/3766918151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2790009420 con 0x7f27a00c8620 2024-01-11T09:47:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.607+0000 7f27a7bff700 1 -- 172.21.15.27:0/3766918151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a016cc40 con 0x7f27a00c8620 2024-01-11T09:47:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.608+0000 7f27a7bff700 1 -- 172.21.15.27:0/3766918151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a00718d0 con 0x7f27a00c8620 2024-01-11T09:47:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.609+0000 7f2795ffb700 1 -- 172.21.15.27:0/3766918151 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2790023070 con 0x7f27a00c8620 2024-01-11T09:47:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.609+0000 7f2795ffb700 1 --2- 172.21.15.27:0/3766918151 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2784066f30 0x7f27840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.610+0000 7f2795ffb700 1 -- 172.21.15.27:0/3766918151 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f2790085a70 con 0x7f27a00c8620 2024-01-11T09:47:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.610+0000 7f27a599b700 1 --2- 172.21.15.27:0/3766918151 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2784066f30 0x7f27840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:57.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.610+0000 7f27a599b700 1 --2- 172.21.15.27:0/3766918151 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2784066f30 0x7f27840693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f279c007fd0 tx=0x7f279c009040 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:57.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.614+0000 7f2795ffb700 1 -- 172.21.15.27:0/3766918151 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2790052cb0 con 0x7f27a00c8620 2024-01-11T09:47:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.818+0000 7f27a7bff700 1 -- 172.21.15.27:0/3766918151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27a0157020 con 0x7f27a00c8620 2024-01-11T09:47:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.819+0000 7f2795ffb700 1 -- 172.21.15.27:0/3766918151 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v261) v1 ==== 74+0+13450 (secure 0 0 0) 0x7f279002a090 con 0x7f27a00c8620 2024-01-11T09:47:57.811 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:57.812 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":261,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:47:53.552822+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/2199022814":"2024-01-12T09:47:50.543987+0000","172.21.15.77:0/886501039":"2024-01-12T09:47:50.543987+0000","172.21.15.77:0/3554846888":"2024-01-12T09:47:50.543987+0000","172.21.15.77:0/1909773939":"2024-01-12T09:47:50.543987+0000","172.21.15.77:0/1544835440":"2024-01-12T09:47:50.543987+0000","172.21.15.77:0/2302937394":"2024-01-12T09:47:50.543987+0000","172.21.15.27:6835/1065634442":"2024-01-12T09:47:49.956028+0000","172.21.15.77:0/151121862":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/1427717667":"2024-01-12T09:47:50.543987+0000","172.21.15.77:0/3962343616":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/710006972":"2024-01-12T09:47:40.221791+0000","172.21.15.27:6834/1065634442":"2024-01-12T09:47:49.956028+0000","172.21.15.77:0/2450284170":"2024-01-12T09:47:40.221791+0000","172.21.15.77:6837/59776":"2024-01-12T09:47:40.221791+0000","172.21.15.77:6836/59776":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/447393830":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/1295621369":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/1441504549":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/1838517028":"2024-01-12T09:47:40.221791+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:47:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.821+0000 7f27a7bff700 1 -- 172.21.15.27:0/3766918151 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2784066f30 msgr2=0x7f27840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.821+0000 7f27a7bff700 1 --2- 172.21.15.27:0/3766918151 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2784066f30 0x7f27840693e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f279c007fd0 tx=0x7f279c009040 comp rx=0 tx=0).stop 2024-01-11T09:47:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.821+0000 7f27a7bff700 1 -- 172.21.15.27:0/3766918151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27a00c8620 msgr2=0x7f27a016a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.821+0000 7f27a7bff700 1 --2- 172.21.15.27:0/3766918151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27a00c8620 0x7f27a016a040 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f2790013000 tx=0x7f2790017000 comp rx=0 tx=0).stop 2024-01-11T09:47:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.821+0000 7f27a7bff700 1 -- 172.21.15.27:0/3766918151 shutdown_connections 2024-01-11T09:47:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.821+0000 7f27a7bff700 1 --2- 172.21.15.27:0/3766918151 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f27a00c89f0 0x7f27a0158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.821+0000 7f27a7bff700 1 --2- 172.21.15.27:0/3766918151 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2784066f30 0x7f27840693e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.821+0000 7f27a7bff700 1 --2- 172.21.15.27:0/3766918151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27a0069d40 0x7f27a016d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.821+0000 7f27a7bff700 1 --2- 172.21.15.27:0/3766918151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27a00c8620 0x7f27a016a040 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.821+0000 7f27a7bff700 1 -- 172.21.15.27:0/3766918151 >> 172.21.15.27:0/3766918151 conn(0x7f27a00680e0 msgr2=0x7f27a00cca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.821+0000 7f27a7bff700 1 -- 172.21.15.27:0/3766918151 shutdown_connections 2024-01-11T09:47:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:57.822+0000 7f27a7bff700 1 -- 172.21.15.27:0/3766918151 wait complete. 2024-01-11T09:47:57.832 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-11T09:47:57.832 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:47:58.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.041+0000 7fa583dff700 1 Processor -- start 2024-01-11T09:47:58.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.041+0000 7fa583dff700 1 -- start start 2024-01-11T09:47:58.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.043+0000 7fa583dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa57c0ca7b0 0x7fa57c0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:58.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.043+0000 7fa583dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa57c159e80 0x7fa57c15dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:58.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.043+0000 7fa583dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa57c15d640 con 0x7fa57c0ca3e0 2024-01-11T09:47:58.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.043+0000 7fa583dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa57c15ae70 con 0x7fa57c0ca7b0 2024-01-11T09:47:58.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.043+0000 7fa583dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa57c0cc4f0 con 0x7fa57c159e80 2024-01-11T09:47:58.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.043+0000 7fa581b9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa57c0ca3e0 0x7fa57c0c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46300/0 (socket says 172.21.15.27:46300) 2024-01-11T09:47:58.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.043+0000 7fa581b9b700 1 -- 172.21.15.27:0/2322668211 learned_addr learned my addr 172.21.15.27:0/2322668211 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:58.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.043+0000 7fa58239c700 1 --2- 172.21.15.27:0/2322668211 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa57c159e80 0x7fa57c15dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:58.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.043+0000 7fa58139a700 1 --2- 172.21.15.27:0/2322668211 >> v2:172.21.15.77:3300/0 conn(0x7fa57c0ca7b0 0x7fa57c0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:58.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.043+0000 7fa580b99700 1 -- 172.21.15.27:0/2322668211 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573414909 0 0) 0x7fa57c15d640 con 0x7fa57c0ca3e0 2024-01-11T09:47:58.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.044+0000 7fa580b99700 1 -- 172.21.15.27:0/2322668211 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa568003570 con 0x7fa57c0ca3e0 2024-01-11T09:47:58.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.044+0000 7fa58239c700 1 -- 172.21.15.27:0/2322668211 >> v2:172.21.15.77:3300/0 conn(0x7fa57c0ca7b0 msgr2=0x7fa57c0cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:58.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.044+0000 7fa58239c700 1 --2- 172.21.15.27:0/2322668211 >> v2:172.21.15.77:3300/0 conn(0x7fa57c0ca7b0 0x7fa57c0cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.044+0000 7fa58239c700 1 -- 172.21.15.27:0/2322668211 >> v1:172.21.15.27:6789/0 conn(0x7fa57c0ca3e0 legacy=0x7fa57c0c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:58.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.044+0000 7fa58239c700 1 -- 172.21.15.27:0/2322668211 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57c15a5d0 con 0x7fa57c159e80 2024-01-11T09:47:58.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.044+0000 7fa58239c700 1 --2- 172.21.15.27:0/2322668211 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa57c159e80 0x7fa57c15dad0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fa574009e90 tx=0x7fa574009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38fac77dcc93e22b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:58.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.044+0000 7fa580b99700 1 -- 172.21.15.27:0/2322668211 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57400e070 con 0x7fa57c159e80 2024-01-11T09:47:58.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.044+0000 7fa580b99700 1 -- 172.21.15.27:0/2322668211 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57400bb40 con 0x7fa57c159e80 2024-01-11T09:47:58.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.044+0000 7fa580b99700 1 -- 172.21.15.27:0/2322668211 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa574012430 con 0x7fa57c159e80 2024-01-11T09:47:58.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.045+0000 7fa583dff700 1 -- 172.21.15.27:0/2322668211 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa57c159e80 msgr2=0x7fa57c15dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:58.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.045+0000 7fa583dff700 1 --2- 172.21.15.27:0/2322668211 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa57c159e80 0x7fa57c15dad0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fa574009e90 tx=0x7fa574009780 comp rx=0 tx=0).stop 2024-01-11T09:47:58.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.045+0000 7fa583dff700 1 -- 172.21.15.27:0/2322668211 shutdown_connections 2024-01-11T09:47:58.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.045+0000 7fa583dff700 1 --2- 172.21.15.27:0/2322668211 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa57c159e80 0x7fa57c15dad0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.045+0000 7fa583dff700 1 --2- 172.21.15.27:0/2322668211 >> v2:172.21.15.77:3300/0 conn(0x7fa57c0ca7b0 0x7fa57c0cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.045+0000 7fa583dff700 1 -- 172.21.15.27:0/2322668211 >> 172.21.15.27:0/2322668211 conn(0x7fa57c0ce3e0 msgr2=0x7fa57c0ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:58.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.045+0000 7fa583dff700 1 -- 172.21.15.27:0/2322668211 shutdown_connections 2024-01-11T09:47:58.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.045+0000 7fa583dff700 1 -- 172.21.15.27:0/2322668211 wait complete. 2024-01-11T09:47:58.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.046+0000 7fa583dff700 1 Processor -- start 2024-01-11T09:47:58.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.047+0000 7fa583dff700 1 -- start start 2024-01-11T09:47:58.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.047+0000 7fa583dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa57c0ca3e0 0x7fa57c0b2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:58.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.048+0000 7fa583dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa57c0ca7b0 0x7fa57c16aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:58.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.048+0000 7fa583dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa57c159e80 0x7fa57c1674e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.048+0000 7fa583dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57c16be60 con 0x7fa57c0ca3e0 2024-01-11T09:47:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.048+0000 7fa583dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57c16afb0 con 0x7fa57c0ca7b0 2024-01-11T09:47:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.048+0000 7fa583dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa57c13fe20 con 0x7fa57c159e80 2024-01-11T09:47:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.048+0000 7fa581b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa57c0ca3e0 0x7fa57c0b2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.048+0000 7fa581b9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa57c0ca3e0 0x7fa57c0b2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42302/0 (socket says 172.21.15.27:42302) 2024-01-11T09:47:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.048+0000 7fa581b9b700 1 -- 172.21.15.27:0/2153779930 learned_addr learned my addr 172.21.15.27:0/2153779930 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.048+0000 7fa58239c700 1 --2- 172.21.15.27:0/2153779930 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa57c159e80 0x7fa57c1674e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.048+0000 7fa58139a700 1 --2- 172.21.15.27:0/2153779930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa57c0ca7b0 0x7fa57c16aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.048+0000 7fa581b9b700 1 -- 172.21.15.27:0/2153779930 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa57c159e80 msgr2=0x7fa57c1674e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.048+0000 7fa581b9b700 1 --2- 172.21.15.27:0/2153779930 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa57c159e80 0x7fa57c1674e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.048+0000 7fa581b9b700 1 -- 172.21.15.27:0/2153779930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa57c0ca7b0 msgr2=0x7fa57c16aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.048+0000 7fa581b9b700 1 --2- 172.21.15.27:0/2153779930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa57c0ca7b0 0x7fa57c16aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.048+0000 7fa581b9b700 1 -- 172.21.15.27:0/2153779930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa574009a20 con 0x7fa57c0ca3e0 2024-01-11T09:47:58.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.048+0000 7fa58139a700 1 --2- 172.21.15.27:0/2153779930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa57c0ca7b0 0x7fa57c16aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:47:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.049+0000 7fa581b9b700 1 --2- 172.21.15.27:0/2153779930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa57c0ca3e0 0x7fa57c0b2a80 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7fa57800c7c0 tx=0x7fa578007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.049+0000 7fa571ffb700 1 -- 172.21.15.27:0/2153779930 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa578005b30 con 0x7fa57c0ca3e0 2024-01-11T09:47:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.049+0000 7fa571ffb700 1 -- 172.21.15.27:0/2153779930 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa578005c90 con 0x7fa57c0ca3e0 2024-01-11T09:47:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.049+0000 7fa583dff700 1 -- 172.21.15.27:0/2153779930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa57c16b250 con 0x7fa57c0ca3e0 2024-01-11T09:47:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.049+0000 7fa571ffb700 1 -- 172.21.15.27:0/2153779930 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa578007340 con 0x7fa57c0ca3e0 2024-01-11T09:47:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.049+0000 7fa583dff700 1 -- 172.21.15.27:0/2153779930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57c16b810 con 0x7fa57c0ca3e0 2024-01-11T09:47:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.050+0000 7fa583dff700 1 -- 172.21.15.27:0/2153779930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57c0718d0 con 0x7fa57c0ca3e0 2024-01-11T09:47:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.051+0000 7fa571ffb700 1 -- 172.21.15.27:0/2153779930 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa57801b400 con 0x7fa57c0ca3e0 2024-01-11T09:47:58.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.052+0000 7fa571ffb700 1 --2- 172.21.15.27:0/2153779930 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa560066f30 0x7fa5600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:58.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.052+0000 7fa571ffb700 1 -- 172.21.15.27:0/2153779930 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7fa578084120 con 0x7fa57c0ca3e0 2024-01-11T09:47:58.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.053+0000 7fa58139a700 1 --2- 172.21.15.27:0/2153779930 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa560066f30 0x7fa5600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:58.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.053+0000 7fa58139a700 1 --2- 172.21.15.27:0/2153779930 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa560066f30 0x7fa5600693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa56c006f00 tx=0x7fa56c008040 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:58.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.056+0000 7fa571ffb700 1 -- 172.21.15.27:0/2153779930 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa578051340 con 0x7fa57c0ca3e0 2024-01-11T09:47:58.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.258+0000 7fa583dff700 1 -- 172.21.15.27:0/2153779930 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa57c16bfa0 con 0x7fa560066f30 2024-01-11T09:47:58.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.261+0000 7fa571ffb700 1 -- 172.21.15.27:0/2153779930 <== mgr.11878 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50727 (secure 0 0 0) 0x7fa56c009e40 con 0x7fa560066f30 2024-01-11T09:47:58.254 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.263+0000 7fa583dff700 1 -- 172.21.15.27:0/2153779930 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa560066f30 msgr2=0x7fa5600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.263+0000 7fa583dff700 1 --2- 172.21.15.27:0/2153779930 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa560066f30 0x7fa5600693e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa56c006f00 tx=0x7fa56c008040 comp rx=0 tx=0).stop 2024-01-11T09:47:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.263+0000 7fa583dff700 1 -- 172.21.15.27:0/2153779930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa57c0ca3e0 msgr2=0x7fa57c0b2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.263+0000 7fa583dff700 1 --2- 172.21.15.27:0/2153779930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa57c0ca3e0 0x7fa57c0b2a80 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7fa57800c7c0 tx=0x7fa578007e00 comp rx=0 tx=0).stop 2024-01-11T09:47:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.263+0000 7fa583dff700 1 -- 172.21.15.27:0/2153779930 shutdown_connections 2024-01-11T09:47:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.263+0000 7fa583dff700 1 --2- 172.21.15.27:0/2153779930 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa57c159e80 0x7fa57c1674e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.263+0000 7fa583dff700 1 --2- 172.21.15.27:0/2153779930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa57c0ca3e0 0x7fa57c0b2a80 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.263+0000 7fa583dff700 1 --2- 172.21.15.27:0/2153779930 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa560066f30 0x7fa5600693e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.263+0000 7fa583dff700 1 --2- 172.21.15.27:0/2153779930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa57c0ca7b0 0x7fa57c16aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.263+0000 7fa583dff700 1 -- 172.21.15.27:0/2153779930 >> 172.21.15.27:0/2153779930 conn(0x7fa57c0ce3e0 msgr2=0x7fa57c0689e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.263+0000 7fa583dff700 1 -- 172.21.15.27:0/2153779930 shutdown_connections 2024-01-11T09:47:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.264+0000 7fa583dff700 1 -- 172.21.15.27:0/2153779930 wait complete. 2024-01-11T09:47:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:47:58.274 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-01-11T09:47:56.566105+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2846,"num_read_kb":6029,"num_write":727,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":430,"ondisk_log_size":430,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":68,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":754974720,"kb_used":2471112,"kb_used_data":27272,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752503608,"statfs":{"total":773094113280,"available":770563694592,"internally_reserved":0,"allocated":27926528,"data_stored":14307017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-668747,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-221,"num_read_kb":-75,"num_write":-261,"num_write_kb":-2052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.998876"},"pg_stats":[{"pgid":"1.0","version":"260'430","reported_seq":1465,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-01-11T09:47:52.627901+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:47:52.627901+0000","last_peered":"2024-01-11T09:47:52.627901+0000","last_clean":"2024-01-11T09:47:52.627901+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:47:52.627901+0000","last_undegraded":"2024-01-11T09:47:52.627901+0000","last_fullsized":"2024-01-11T09:47:52.627901+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":430,"log_dups_size":0,"ondisk_log_size":430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2846,"num_read_kb":6029,"num_write":727,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2846,"num_read_kb":6029,"num_write":727,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":430,"ondisk_log_size":430,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305192,"kb_used_data":3808,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94066648,"statfs":{"total":96636764160,"available":96324247552,"internally_reserved":0,"allocated":3899392,"data_stored":2192668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.66500000000000004,"15min":0.71899999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36599999999999999},"max":{"1min":0.81799999999999995,"5min":0.95699999999999996,"15min":1.0529999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.69299999999999995,"15min":0.73399999999999999},"min":{"1min":0.41699999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.97999999999999998,"5min":1.026,"15min":1.0669999999999999},"last":0.84599999999999997}]},{"osd":1,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.71999999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.0209999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69699999999999995,"15min":0.73099999999999998},"min":{"1min":0.44800000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.91800000000000004,"5min":0.95899999999999996,"15min":1.0820000000000001},"last":0.65500000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.69999999999999996,"15min":0.72899999999999998},"min":{"1min":0.48899999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.85399999999999998,"5min":0.93799999999999994,"15min":1.0600000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70099999999999996,"15min":0.74399999999999999},"min":{"1min":0.52200000000000002,"5min":0.44800000000000001,"15min":0.44500000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":1.153},"last":0.88100000000000001}]},{"osd":3,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73499999999999999,"15min":0.76000000000000001},"min":{"1min":0.5,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.238,"5min":1.238,"15min":1.238},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70899999999999996,"15min":0.74199999999999999},"min":{"1min":0.47699999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.89400000000000002,"5min":0.94099999999999995,"15min":1.0880000000000001},"last":1.024}]},{"osd":4,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.61899999999999999,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0209999999999999,"5min":1.0860000000000001,"15min":1.107},"last":1.004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.65900000000000003,"15min":0.64500000000000002},"min":{"1min":0.443,"5min":0.437,"15min":0.39700000000000002},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.90800000000000003}]},{"osd":5,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63200000000000001,"15min":0.64200000000000002},"min":{"1min":0.50900000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.90800000000000003},"last":1.101},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.70799999999999996,"15min":0.69399999999999995},"min":{"1min":0.495,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":1.1160000000000001}]},{"osd":6,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.77700000000000002,"15min":0.78300000000000003},"min":{"1min":0.54900000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0649999999999999,"5min":1.232,"15min":1.232},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.76800000000000002,"15min":0.76700000000000002},"min":{"1min":0.52300000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":1.1299999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.63200000000000001,"15min":0.69499999999999995},"min":{"1min":0.438,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.79800000000000004,"5min":0.94199999999999995,"15min":0.999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.621,"5min":0.68400000000000005,"15min":0.72799999999999998},"min":{"1min":0.46100000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.93400000000000005,"5min":0.97099999999999997,"15min":1.216},"last":0.72099999999999997}]},{"osd":1,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.66000000000000003,"15min":0.72199999999999998},"min":{"1min":0.46200000000000002,"5min":0.42799999999999999,"15min":0.41699999999999998},"max":{"1min":0.91000000000000003,"5min":0.98499999999999999,"15min":1.3129999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.67000000000000004,"15min":0.71599999999999997},"min":{"1min":0.46500000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.91100000000000003,"5min":0.95999999999999996,"15min":1.0840000000000001},"last":0.755}]},{"osd":2,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.68700000000000006,"15min":0.73499999999999999},"min":{"1min":0.44600000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.83799999999999997,"5min":1.0069999999999999,"15min":1.4179999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.69999999999999996,"15min":0.73999999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.92800000000000005,"5min":1.796,"15min":1.796},"last":0.63700000000000001}]},{"osd":3,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.71299999999999997,"15min":0.746},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89000000000000001,"5min":1.012,"15min":1.014},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.70099999999999996,"15min":0.754},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":1.165},"last":0.74099999999999999}]},{"osd":4,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.59399999999999997,"15min":0.56999999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.86599999999999999,"5min":0.91700000000000004,"15min":1.0309999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.64400000000000002,"15min":0.622},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":1.1259999999999999},"last":0.56200000000000006}]},{"osd":5,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.077,"5min":1.077,"15min":1.369},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.71699999999999997,"15min":0.70899999999999996},"min":{"1min":0.495,"5min":0.49099999999999999,"15min":0.45400000000000001},"max":{"1min":0.94399999999999995,"5min":0.999,"15min":1.0960000000000001},"last":0.82099999999999995}]},{"osd":7,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.72099999999999997,"15min":0.71399999999999997},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.45100000000000001},"max":{"1min":1.0089999999999999,"5min":1.089,"15min":1.484},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.77500000000000002,"15min":0.77200000000000002},"min":{"1min":0.53700000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.123,"5min":1.1379999999999999,"15min":1.603},"last":0.83499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575997,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.502,"15min":0.51000000000000001},"min":{"1min":0.36299999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.70699999999999996,"5min":0.73399999999999999,"15min":0.93999999999999995},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.60699999999999998},"min":{"1min":0.41899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73499999999999999,"5min":0.85199999999999998,"15min":0.95099999999999996},"last":0.495}]},{"osd":2,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.47199999999999998,"15min":0.501},"min":{"1min":0.35199999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.63800000000000001,"5min":0.82399999999999995,"15min":0.82699999999999996},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56599999999999995,"15min":0.58399999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.69999999999999996,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.62,"15min":0.64300000000000002},"min":{"1min":0.45500000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.76000000000000001,"5min":0.85199999999999998,"15min":0.96699999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55500000000000005,"15min":0.57199999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.44700000000000001}]},{"osd":4,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.76400000000000001,"15min":0.80400000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.159,"5min":1.214,"15min":1.2969999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.81799999999999995,"15min":0.84999999999999998},"min":{"1min":0.58699999999999997,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.169,"5min":1.1879999999999999,"15min":1.1890000000000001},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.79100000000000004,"15min":0.81999999999999995},"min":{"1min":0.55600000000000005,"5min":0.443,"15min":0.443},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.78800000000000003,"15min":0.80200000000000005},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.51300000000000001},"max":{"1min":1.1539999999999999,"5min":1.341,"15min":1.341},"last":0.624}]},{"osd":6,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.80400000000000005,"15min":0.83799999999999997},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.1830000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.81799999999999995,"15min":0.84599999999999997},"min":{"1min":0.57299999999999995,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.61099999999999999}]},{"osd":7,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.86599999999999999,"15min":0.88200000000000001},"min":{"1min":0.63900000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.84299999999999997,"15min":0.85399999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.391,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.68999999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575996,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309288,"kb_used_data":3808,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062552,"statfs":{"total":96636764160,"available":96320053248,"internally_reserved":0,"allocated":3899392,"data_stored":2192668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.48799999999999999,"15min":0.51300000000000001},"min":{"1min":0.34899999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.73099999999999998,"5min":0.754,"15min":0.84699999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.56299999999999994,"15min":0.59099999999999997},"min":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.59599999999999997,"5min":0.81599999999999995,"15min":0.874},"last":0.50600000000000001}]},{"osd":2,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.53300000000000003,"15min":0.56200000000000006},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.68200000000000005,"5min":0.83099999999999996,"15min":0.97499999999999998},"last":0.379},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55400000000000005,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.86399999999999999},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51100000000000001,"15min":0.53800000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.95399999999999996},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60699999999999998,"15min":0.64400000000000002},"min":{"1min":0.42499999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.87,"5min":0.90300000000000002,"15min":1.0389999999999999},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.78600000000000003,"15min":0.81100000000000005},"min":{"1min":0.46100000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.304},"last":1.091},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.78500000000000003,"15min":0.80100000000000005},"min":{"1min":0.48799999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.95399999999999996,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":1.1160000000000001}]},{"osd":5,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.80800000000000005,"15min":0.83599999999999997},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.97599999999999998,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.81399999999999995,"15min":0.83699999999999997},"min":{"1min":0.60399999999999998,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.02,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.77300000000000002}]},{"osd":6,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.79700000000000004,"15min":0.82099999999999995},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.95899999999999996,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.82299999999999995,"15min":0.85999999999999999},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":0.97699999999999998,"5min":1.1699999999999999,"15min":1.256},"last":1.048}]},{"osd":7,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.83699999999999997,"15min":0.86299999999999999},"min":{"1min":0.64100000000000001,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1000000000000001,"5min":1.226,"15min":1.28},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.84499999999999997,"15min":0.86899999999999999},"min":{"1min":0.63,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0369999999999999,"5min":1.238,"15min":1.2410000000000001},"last":1.173}]}]},{"osd":2,"up_from":13,"seq":55834575995,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":300516,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071324,"statfs":{"total":96636764160,"available":96329035776,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.46999999999999997,"15min":0.5},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.27900000000000003},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.85399999999999998},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.53400000000000003,"15min":0.56000000000000005},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66900000000000004,"5min":0.77300000000000002,"15min":1.6359999999999999},"last":0.58099999999999996}]},{"osd":1,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56499999999999995,"15min":0.57999999999999996},"min":{"1min":0.40000000000000002,"5min":0.38200000000000001,"15min":0.33500000000000002},"max":{"1min":0.70099999999999996,"5min":0.88800000000000001,"15min":0.93000000000000005},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.56399999999999995,"15min":0.59299999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.64800000000000002,"5min":0.97299999999999998,"15min":1.145},"last":0.76500000000000001}]},{"osd":3,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52900000000000003,"15min":0.55000000000000004},"min":{"1min":0.376,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81499999999999995,"5min":0.83099999999999996,"15min":0.96399999999999997},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.61199999999999999,"15min":0.65000000000000002},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.75600000000000001,"5min":0.876,"15min":1.3620000000000001},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.75700000000000001,"15min":0.79100000000000004},"min":{"1min":0.56299999999999994,"5min":0.55800000000000005,"15min":0.54300000000000004},"max":{"1min":0.96499999999999997,"5min":1.071,"15min":1.1519999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.76400000000000001,"15min":0.79300000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0209999999999999,"5min":1.161,"15min":1.161},"last":0.77900000000000003}]},{"osd":5,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.79900000000000004,"15min":0.85599999999999998},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.94099999999999995,"5min":1.228,"15min":1.429},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.82799999999999996,"15min":0.85299999999999998},"min":{"1min":0.61499999999999999,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":0.94699999999999995,"5min":1.1850000000000001,"15min":1.2170000000000001},"last":0.81899999999999995}]},{"osd":6,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.77600000000000002,"15min":0.81799999999999995},"min":{"1min":0.56000000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96499999999999997,"5min":1.1659999999999999,"15min":1.4139999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.80400000000000005,"15min":0.85599999999999998},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.94999999999999996,"5min":1.2010000000000001,"15min":1.593},"last":0.79800000000000004}]},{"osd":7,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.84399999999999997,"15min":0.88400000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.57299999999999995},"max":{"1min":1.004,"5min":1.1339999999999999,"15min":2.0539999999999998},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.83499999999999996,"15min":0.872},"min":{"1min":0.624,"5min":0.60799999999999998,"15min":0.55500000000000005},"max":{"1min":1.026,"5min":1.2270000000000001,"15min":1.2490000000000001},"last":0.85499999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":304612,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067228,"statfs":{"total":96636764160,"available":96324841472,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.47699999999999998,"15min":0.50900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.61699999999999999,"5min":0.89700000000000002,"15min":0.90000000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.52200000000000002,"15min":0.57499999999999996},"min":{"1min":0.35799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69999999999999996,"5min":1.448,"15min":1.448},"last":0.52800000000000002}]},{"osd":1,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46800000000000003,"15min":0.503},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73199999999999998,"5min":0.90900000000000003,"15min":1.304},"last":0.497},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.55500000000000005,"15min":0.59599999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.71499999999999997,"5min":0.84599999999999997,"15min":1.4590000000000001},"last":0.38}]},{"osd":2,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.54600000000000004,"15min":0.59499999999999997},"min":{"1min":0.34899999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.78600000000000003,"5min":1.054,"15min":1.054},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.61399999999999999,"15min":0.64900000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.76100000000000001,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.70899999999999996}]},{"osd":4,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72299999999999998,"15min":0.77600000000000002},"min":{"1min":0.52600000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.93400000000000005,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.746,"15min":0.79600000000000004},"min":{"1min":0.51800000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.88800000000000001,"5min":1.171,"15min":1.1830000000000001},"last":0.623}]},{"osd":5,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.755,"15min":0.80600000000000005},"min":{"1min":0.497,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.02,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.75},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78500000000000003,"15min":0.83399999999999996},"min":{"1min":0.55600000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.054,"5min":1.2669999999999999,"15min":1.3440000000000001},"last":0.66300000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.79900000000000004,"15min":0.84299999999999997},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.97799999999999998,"5min":1.181,"15min":1.2170000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.77800000000000002,"15min":0.81799999999999995},"min":{"1min":0.59599999999999997,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.81100000000000005,"5min":1.883,"15min":1.883},"last":0.64300000000000002}]},{"osd":7,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.79700000000000004,"15min":0.85399999999999998},"min":{"1min":0.51600000000000001,"5min":0.505,"15min":0.505},"max":{"1min":0.89600000000000002,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.746,"5min":0.81200000000000006,"15min":0.876},"min":{"1min":0.59899999999999998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.97599999999999998,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.65600000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575995,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313092,"kb_used_data":3516,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94058748,"statfs":{"total":96636764160,"available":96316157952,"internally_reserved":0,"allocated":3600384,"data_stored":1910181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.68700000000000006,"15min":0.748},"min":{"1min":0.47299999999999998,"5min":0.42099999999999999,"15min":0.39900000000000002},"max":{"1min":0.86899999999999999,"5min":0.91200000000000003,"15min":1.0760000000000001},"last":0.435},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.70699999999999996,"15min":0.75700000000000001},"min":{"1min":0.52800000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84599999999999997,"5min":1.853,"15min":1.853},"last":0.68899999999999995}]},{"osd":1,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.67600000000000005,"15min":0.72599999999999998},"min":{"1min":0.42299999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.91500000000000004,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.67000000000000004,"15min":0.71999999999999997},"min":{"1min":0.44,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.93100000000000005,"5min":1.639,"15min":1.639},"last":0.629}]},{"osd":2,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.68600000000000005,"15min":0.73599999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.44700000000000001},"max":{"1min":0.94099999999999995,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.68700000000000006,"15min":0.73199999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.014,"5min":1.014,"15min":1.0449999999999999},"last":0.66500000000000004}]},{"osd":3,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.72499999999999998},"min":{"1min":0.47799999999999998,"5min":0.436,"15min":0.436},"max":{"1min":0.91900000000000004,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73699999999999999,"15min":0.76800000000000002},"min":{"1min":0.59799999999999998,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.98899999999999999,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.80100000000000005}]},{"osd":5,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.629,"15min":0.624},"min":{"1min":0.35299999999999998,"5min":0.311,"15min":0.311},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.66300000000000003,"15min":0.64500000000000002},"min":{"1min":0.47499999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0580000000000001,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.78900000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68999999999999995,"15min":0.67200000000000004},"min":{"1min":0.441,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0680000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.72399999999999998,"15min":0.70499999999999996},"min":{"1min":0.39400000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.1899999999999999,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.59199999999999997}]},{"osd":7,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.72799999999999998,"15min":0.70199999999999996},"min":{"1min":0.44700000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0409999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.78200000000000003,"15min":0.755},"min":{"1min":0.51300000000000001,"5min":0.436,"15min":0.41699999999999998},"max":{"1min":1.109,"5min":2.052,"15min":2.052},"last":0.74399999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.71399999999999997,"15min":0.72999999999999998},"min":{"1min":0.42899999999999999,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.72599999999999998,"15min":0.72999999999999998},"min":{"1min":0.46400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.9900000000000002,"5min":2.9900000000000002,"15min":2.9900000000000002},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.71799999999999997,"15min":0.73499999999999999},"min":{"1min":0.46000000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.873,"5min":0.75600000000000001,"15min":0.748},"min":{"1min":0.47699999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":5.4039999999999999,"5min":5.4039999999999999,"15min":5.4039999999999999},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.54300000000000004,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.7120000000000002,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.73999999999999999,"15min":0.74299999999999999},"min":{"1min":0.51700000000000002,"5min":0.439,"15min":0.439},"max":{"1min":2.5099999999999998,"5min":2.5099999999999998,"15min":2.5099999999999998},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.745,"15min":0.75700000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.7799999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":0.628},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.79900000000000004,"15min":0.78300000000000003},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":5.5570000000000004,"5min":5.5570000000000004,"15min":5.5570000000000004},"last":0.71499999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.64600000000000002,"15min":0.61699999999999999},"min":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41699999999999998},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.66500000000000004,"15min":0.63800000000000001},"min":{"1min":0.40400000000000003,"5min":0.40200000000000002,"15min":0.375},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":0.69799999999999995}]},{"osd":6,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.67400000000000004,"15min":0.65000000000000002},"min":{"1min":0.42899999999999999,"5min":0.42099999999999999,"15min":0.40999999999999998},"max":{"1min":2.9500000000000002,"5min":2.9500000000000002,"15min":2.9500000000000002},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.6280000000000001,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.748,"15min":0.73199999999999998},"min":{"1min":0.52700000000000002,"5min":0.47299999999999998,"15min":0.434},"max":{"1min":2.399,"5min":2.399,"15min":2.399},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.80400000000000005,"15min":0.78200000000000003},"min":{"1min":0.61399999999999999,"5min":0.48799999999999999,"15min":0.47599999999999998},"max":{"1min":2.7570000000000001,"5min":2.7570000000000001,"15min":2.7570000000000001},"last":0.90700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:47:58.276 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:47:58.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.484+0000 7f749fdff700 1 Processor -- start 2024-01-11T09:47:58.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.485+0000 7f749fdff700 1 -- start start 2024-01-11T09:47:58.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.486+0000 7f749fdff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74980ca3e0 0x7f74980cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:58.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.486+0000 7f749fdff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749815d640 con 0x7f74980ca7b0 2024-01-11T09:47:58.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.486+0000 7f749fdff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749815a5d0 con 0x7f7498159e80 2024-01-11T09:47:58.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.486+0000 7f749fdff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f749815ae70 con 0x7f74980ca3e0 2024-01-11T09:47:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.486+0000 7f749d39a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f74980ca7b0 0x7f74980c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46316/0 (socket says 172.21.15.27:46316) 2024-01-11T09:47:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.486+0000 7f749d39a700 1 -- 172.21.15.27:0/564545092 learned_addr learned my addr 172.21.15.27:0/564545092 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.486+0000 7f749db9b700 1 --2- 172.21.15.27:0/564545092 >> v2:172.21.15.27:3300/0 conn(0x7f74980ca3e0 0x7f74980cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.487+0000 7f749cb99700 1 -- 172.21.15.27:0/564545092 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224081672 0 0) 0x7f749815d640 con 0x7f74980ca7b0 2024-01-11T09:47:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.487+0000 7f749cb99700 1 -- 172.21.15.27:0/564545092 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7484003570 con 0x7f74980ca7b0 2024-01-11T09:47:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.487+0000 7f749db9b700 1 -- 172.21.15.27:0/564545092 >> v1:172.21.15.77:6789/0 conn(0x7f7498159e80 legacy=0x7f749815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.487+0000 7f749db9b700 1 -- 172.21.15.27:0/564545092 >> v1:172.21.15.27:6789/0 conn(0x7f74980ca7b0 legacy=0x7f74980c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.487+0000 7f749db9b700 1 -- 172.21.15.27:0/564545092 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749815a8b0 con 0x7f74980ca3e0 2024-01-11T09:47:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.487+0000 7f749db9b700 1 --2- 172.21.15.27:0/564545092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74980ca3e0 0x7f74980cca20 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f7494009910 tx=0x7f74940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a60fc3f291d67320 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.487+0000 7f749cb99700 1 -- 172.21.15.27:0/564545092 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749400b640 con 0x7f74980ca3e0 2024-01-11T09:47:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.487+0000 7f749cb99700 1 -- 172.21.15.27:0/564545092 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749400b7a0 con 0x7f74980ca3e0 2024-01-11T09:47:58.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.487+0000 7f749cb99700 1 -- 172.21.15.27:0/564545092 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749400bb60 con 0x7f74980ca3e0 2024-01-11T09:47:58.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.488+0000 7f749fdff700 1 -- 172.21.15.27:0/564545092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74980ca3e0 msgr2=0x7f74980cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:58.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.488+0000 7f749fdff700 1 --2- 172.21.15.27:0/564545092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74980ca3e0 0x7f74980cca20 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f7494009910 tx=0x7f74940091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.488+0000 7f749fdff700 1 -- 172.21.15.27:0/564545092 shutdown_connections 2024-01-11T09:47:58.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.488+0000 7f749fdff700 1 --2- 172.21.15.27:0/564545092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74980ca3e0 0x7f74980cca20 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.488+0000 7f749fdff700 1 -- 172.21.15.27:0/564545092 >> 172.21.15.27:0/564545092 conn(0x7f74980ce3e0 msgr2=0x7f74980ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:58.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.488+0000 7f749fdff700 1 -- 172.21.15.27:0/564545092 shutdown_connections 2024-01-11T09:47:58.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.488+0000 7f749fdff700 1 -- 172.21.15.27:0/564545092 wait complete. 2024-01-11T09:47:58.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.490+0000 7f749fdff700 1 Processor -- start 2024-01-11T09:47:58.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.490+0000 7f749fdff700 1 -- start start 2024-01-11T09:47:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.491+0000 7f749fdff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74980ca3e0 0x7f749816a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.491+0000 7f749fdff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74980ca7b0 0x7f749816d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.491+0000 7f749fdff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7498159e80 0x7f7498150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.491+0000 7f749fdff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74980b2870 con 0x7f7498159e80 2024-01-11T09:47:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.491+0000 7f749fdff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74980b1c80 con 0x7f74980ca7b0 2024-01-11T09:47:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.491+0000 7f749fdff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74980b1f40 con 0x7f74980ca3e0 2024-01-11T09:47:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.491+0000 7f749e39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7498159e80 0x7f7498150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.491+0000 7f749e39c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7498159e80 0x7f7498150960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42318/0 (socket says 172.21.15.27:42318) 2024-01-11T09:47:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.491+0000 7f749e39c700 1 -- 172.21.15.27:0/3389226253 learned_addr learned my addr 172.21.15.27:0/3389226253 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.491+0000 7f749d39a700 1 --2- 172.21.15.27:0/3389226253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74980ca7b0 0x7f749816d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.491+0000 7f749db9b700 1 --2- 172.21.15.27:0/3389226253 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74980ca3e0 0x7f749816a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.492+0000 7f749e39c700 1 -- 172.21.15.27:0/3389226253 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74980ca3e0 msgr2=0x7f749816a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.492+0000 7f749e39c700 1 --2- 172.21.15.27:0/3389226253 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74980ca3e0 0x7f749816a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.492+0000 7f749e39c700 1 -- 172.21.15.27:0/3389226253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74980ca7b0 msgr2=0x7f749816d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:58.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.492+0000 7f749e39c700 1 --2- 172.21.15.27:0/3389226253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74980ca7b0 0x7f749816d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.492+0000 7f749e39c700 1 -- 172.21.15.27:0/3389226253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7494009420 con 0x7f7498159e80 2024-01-11T09:47:58.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.492+0000 7f749e39c700 1 --2- 172.21.15.27:0/3389226253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7498159e80 0x7f7498150960 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f749000c8e0 tx=0x7f7490001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.492+0000 7f748dffb700 1 -- 172.21.15.27:0/3389226253 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7490012630 con 0x7f7498159e80 2024-01-11T09:47:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.492+0000 7f748dffb700 1 -- 172.21.15.27:0/3389226253 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7490012790 con 0x7f7498159e80 2024-01-11T09:47:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.492+0000 7f749fdff700 1 -- 172.21.15.27:0/3389226253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74980b2220 con 0x7f7498159e80 2024-01-11T09:47:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.492+0000 7f748dffb700 1 -- 172.21.15.27:0/3389226253 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7490012b50 con 0x7f7498159e80 2024-01-11T09:47:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.492+0000 7f749fdff700 1 -- 172.21.15.27:0/3389226253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7498150ea0 con 0x7f7498159e80 2024-01-11T09:47:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.493+0000 7f749fdff700 1 -- 172.21.15.27:0/3389226253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74980b2a20 con 0x7f7498159e80 2024-01-11T09:47:58.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.494+0000 7f748dffb700 1 -- 172.21.15.27:0/3389226253 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f749001a3a0 con 0x7f7498159e80 2024-01-11T09:47:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.495+0000 7f748dffb700 1 --2- 172.21.15.27:0/3389226253 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f747c066f30 0x7f747c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.495+0000 7f748dffb700 1 -- 172.21.15.27:0/3389226253 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f74900846c0 con 0x7f7498159e80 2024-01-11T09:47:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.496+0000 7f749db9b700 1 --2- 172.21.15.27:0/3389226253 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f747c066f30 0x7f747c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.496+0000 7f749db9b700 1 --2- 172.21.15.27:0/3389226253 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f747c066f30 0x7f747c0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f74940093f0 tx=0x7f7494017000 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.499+0000 7f748dffb700 1 -- 172.21.15.27:0/3389226253 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f74980b2a20 con 0x7f7498159e80 2024-01-11T09:47:58.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.699+0000 7f749fdff700 1 -- 172.21.15.27:0/3389226253 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f74980718d0 con 0x7f747c066f30 2024-01-11T09:47:58.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.701+0000 7f748dffb700 1 -- 172.21.15.27:0/3389226253 <== mgr.11878 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50741 (secure 0 0 0) 0x7f74980718d0 con 0x7f747c066f30 2024-01-11T09:47:58.694 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:58.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.704+0000 7f749fdff700 1 -- 172.21.15.27:0/3389226253 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f747c066f30 msgr2=0x7f747c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:58.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.704+0000 7f749fdff700 1 --2- 172.21.15.27:0/3389226253 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f747c066f30 0x7f747c0693e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f74940093f0 tx=0x7f7494017000 comp rx=0 tx=0).stop 2024-01-11T09:47:58.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.704+0000 7f749fdff700 1 -- 172.21.15.27:0/3389226253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7498159e80 msgr2=0x7f7498150960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:58.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.704+0000 7f749fdff700 1 --2- 172.21.15.27:0/3389226253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7498159e80 0x7f7498150960 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f749000c8e0 tx=0x7f7490001710 comp rx=0 tx=0).stop 2024-01-11T09:47:58.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.704+0000 7f749fdff700 1 -- 172.21.15.27:0/3389226253 shutdown_connections 2024-01-11T09:47:58.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.704+0000 7f749fdff700 1 --2- 172.21.15.27:0/3389226253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7498159e80 0x7f7498150960 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.704+0000 7f749fdff700 1 --2- 172.21.15.27:0/3389226253 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f74980ca3e0 0x7f749816a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.704+0000 7f749fdff700 1 --2- 172.21.15.27:0/3389226253 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f747c066f30 0x7f747c0693e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.704+0000 7f749fdff700 1 --2- 172.21.15.27:0/3389226253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74980ca7b0 0x7f749816d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.704+0000 7f749fdff700 1 -- 172.21.15.27:0/3389226253 >> 172.21.15.27:0/3389226253 conn(0x7f74980ce3e0 msgr2=0x7f749806ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:58.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.704+0000 7f749fdff700 1 -- 172.21.15.27:0/3389226253 shutdown_connections 2024-01-11T09:47:58.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.704+0000 7f749fdff700 1 -- 172.21.15.27:0/3389226253 wait complete. 2024-01-11T09:47:58.698 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:47:58.714 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-01-11T09:47:58.566494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2846,"num_read_kb":6029,"num_write":727,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":430,"ondisk_log_size":430,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":2470824,"kb_used_data":26984,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752503896,"statfs":{"total":773094113280,"available":770563989504,"internally_reserved":0,"allocated":27631616,"data_stored":13999136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-668747,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-221,"num_read_kb":-75,"num_write":-261,"num_write_kb":-2052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.999265"},"pg_stats":[{"pgid":"1.0","version":"260'430","reported_seq":1465,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-01-11T09:47:52.627901+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:47:52.627901+0000","last_peered":"2024-01-11T09:47:52.627901+0000","last_clean":"2024-01-11T09:47:52.627901+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:47:52.627901+0000","last_undegraded":"2024-01-11T09:47:52.627901+0000","last_fullsized":"2024-01-11T09:47:52.627901+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":430,"log_dups_size":0,"ondisk_log_size":430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2846,"num_read_kb":6029,"num_write":727,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2846,"num_read_kb":6029,"num_write":727,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":430,"ondisk_log_size":430,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305192,"kb_used_data":3808,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94066648,"statfs":{"total":96636764160,"available":96324247552,"internally_reserved":0,"allocated":3899392,"data_stored":2192668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.66500000000000004,"15min":0.71899999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36599999999999999},"max":{"1min":0.81799999999999995,"5min":0.95699999999999996,"15min":1.0529999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.69299999999999995,"15min":0.73399999999999999},"min":{"1min":0.41699999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.97999999999999998,"5min":1.026,"15min":1.0669999999999999},"last":0.84599999999999997}]},{"osd":1,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.71999999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.0209999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69699999999999995,"15min":0.73099999999999998},"min":{"1min":0.44800000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.91800000000000004,"5min":0.95899999999999996,"15min":1.0820000000000001},"last":0.65500000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.69999999999999996,"15min":0.72899999999999998},"min":{"1min":0.48899999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.85399999999999998,"5min":0.93799999999999994,"15min":1.0600000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70099999999999996,"15min":0.74399999999999999},"min":{"1min":0.52200000000000002,"5min":0.44800000000000001,"15min":0.44500000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":1.153},"last":0.88100000000000001}]},{"osd":3,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73499999999999999,"15min":0.76000000000000001},"min":{"1min":0.5,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.238,"5min":1.238,"15min":1.238},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70899999999999996,"15min":0.74199999999999999},"min":{"1min":0.47699999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.89400000000000002,"5min":0.94099999999999995,"15min":1.0880000000000001},"last":1.024}]},{"osd":4,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.61899999999999999,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0209999999999999,"5min":1.0860000000000001,"15min":1.107},"last":1.004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.65900000000000003,"15min":0.64500000000000002},"min":{"1min":0.443,"5min":0.437,"15min":0.39700000000000002},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.90800000000000003}]},{"osd":5,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63200000000000001,"15min":0.64200000000000002},"min":{"1min":0.50900000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.90800000000000003},"last":1.101},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.70799999999999996,"15min":0.69399999999999995},"min":{"1min":0.495,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":1.1160000000000001}]},{"osd":6,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.77700000000000002,"15min":0.78300000000000003},"min":{"1min":0.54900000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0649999999999999,"5min":1.232,"15min":1.232},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.76800000000000002,"15min":0.76700000000000002},"min":{"1min":0.52300000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":1.1299999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.63200000000000001,"15min":0.69499999999999995},"min":{"1min":0.438,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.79800000000000004,"5min":0.94199999999999995,"15min":0.999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.621,"5min":0.68400000000000005,"15min":0.72799999999999998},"min":{"1min":0.46100000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.93400000000000005,"5min":0.97099999999999997,"15min":1.216},"last":0.72099999999999997}]},{"osd":1,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.66000000000000003,"15min":0.72199999999999998},"min":{"1min":0.46200000000000002,"5min":0.42799999999999999,"15min":0.41699999999999998},"max":{"1min":0.91000000000000003,"5min":0.98499999999999999,"15min":1.3129999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.67000000000000004,"15min":0.71599999999999997},"min":{"1min":0.46500000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.91100000000000003,"5min":0.95999999999999996,"15min":1.0840000000000001},"last":0.755}]},{"osd":2,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.68700000000000006,"15min":0.73499999999999999},"min":{"1min":0.44600000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.83799999999999997,"5min":1.0069999999999999,"15min":1.4179999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.69999999999999996,"15min":0.73999999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.92800000000000005,"5min":1.796,"15min":1.796},"last":0.63700000000000001}]},{"osd":3,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.71299999999999997,"15min":0.746},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89000000000000001,"5min":1.012,"15min":1.014},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.70099999999999996,"15min":0.754},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":1.165},"last":0.74099999999999999}]},{"osd":4,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.59399999999999997,"15min":0.56999999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.86599999999999999,"5min":0.91700000000000004,"15min":1.0309999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.64400000000000002,"15min":0.622},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":1.1259999999999999},"last":0.56200000000000006}]},{"osd":5,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.077,"5min":1.077,"15min":1.369},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.71699999999999997,"15min":0.70899999999999996},"min":{"1min":0.495,"5min":0.49099999999999999,"15min":0.45400000000000001},"max":{"1min":0.94399999999999995,"5min":0.999,"15min":1.0960000000000001},"last":0.82099999999999995}]},{"osd":7,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.72099999999999997,"15min":0.71399999999999997},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.45100000000000001},"max":{"1min":1.0089999999999999,"5min":1.089,"15min":1.484},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.77500000000000002,"15min":0.77200000000000002},"min":{"1min":0.53700000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.123,"5min":1.1379999999999999,"15min":1.603},"last":0.83499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575997,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.502,"15min":0.51000000000000001},"min":{"1min":0.36299999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.70699999999999996,"5min":0.73399999999999999,"15min":0.93999999999999995},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.60699999999999998},"min":{"1min":0.41899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73499999999999999,"5min":0.85199999999999998,"15min":0.95099999999999996},"last":0.495}]},{"osd":2,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.47199999999999998,"15min":0.501},"min":{"1min":0.35199999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.63800000000000001,"5min":0.82399999999999995,"15min":0.82699999999999996},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56599999999999995,"15min":0.58399999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.69999999999999996,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.62,"15min":0.64300000000000002},"min":{"1min":0.45500000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.76000000000000001,"5min":0.85199999999999998,"15min":0.96699999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55500000000000005,"15min":0.57199999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.44700000000000001}]},{"osd":4,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.76400000000000001,"15min":0.80400000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.159,"5min":1.214,"15min":1.2969999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.81799999999999995,"15min":0.84999999999999998},"min":{"1min":0.58699999999999997,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.169,"5min":1.1879999999999999,"15min":1.1890000000000001},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.79100000000000004,"15min":0.81999999999999995},"min":{"1min":0.55600000000000005,"5min":0.443,"15min":0.443},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.78800000000000003,"15min":0.80200000000000005},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.51300000000000001},"max":{"1min":1.1539999999999999,"5min":1.341,"15min":1.341},"last":0.624}]},{"osd":6,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.80400000000000005,"15min":0.83799999999999997},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.1830000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.81799999999999995,"15min":0.84599999999999997},"min":{"1min":0.57299999999999995,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.61099999999999999}]},{"osd":7,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.86599999999999999,"15min":0.88200000000000001},"min":{"1min":0.63900000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.84299999999999997,"15min":0.85399999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.391,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.68999999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575996,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309288,"kb_used_data":3808,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062552,"statfs":{"total":96636764160,"available":96320053248,"internally_reserved":0,"allocated":3899392,"data_stored":2192668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.48799999999999999,"15min":0.51300000000000001},"min":{"1min":0.34899999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.73099999999999998,"5min":0.754,"15min":0.84699999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.56299999999999994,"15min":0.59099999999999997},"min":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.59599999999999997,"5min":0.81599999999999995,"15min":0.874},"last":0.50600000000000001}]},{"osd":2,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.53300000000000003,"15min":0.56200000000000006},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.68200000000000005,"5min":0.83099999999999996,"15min":0.97499999999999998},"last":0.379},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55400000000000005,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.86399999999999999},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51100000000000001,"15min":0.53800000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.95399999999999996},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60699999999999998,"15min":0.64400000000000002},"min":{"1min":0.42499999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.87,"5min":0.90300000000000002,"15min":1.0389999999999999},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.78600000000000003,"15min":0.81100000000000005},"min":{"1min":0.46100000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.304},"last":1.091},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.78500000000000003,"15min":0.80100000000000005},"min":{"1min":0.48799999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.95399999999999996,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":1.1160000000000001}]},{"osd":5,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.80800000000000005,"15min":0.83599999999999997},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.97599999999999998,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.81399999999999995,"15min":0.83699999999999997},"min":{"1min":0.60399999999999998,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.02,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.77300000000000002}]},{"osd":6,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.79700000000000004,"15min":0.82099999999999995},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.95899999999999996,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.82299999999999995,"15min":0.85999999999999999},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":0.97699999999999998,"5min":1.1699999999999999,"15min":1.256},"last":1.048}]},{"osd":7,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.83699999999999997,"15min":0.86299999999999999},"min":{"1min":0.64100000000000001,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1000000000000001,"5min":1.226,"15min":1.28},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.84499999999999997,"15min":0.86899999999999999},"min":{"1min":0.63,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0369999999999999,"5min":1.238,"15min":1.2410000000000001},"last":1.173}]}]},{"osd":2,"up_from":13,"seq":55834575995,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":300516,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071324,"statfs":{"total":96636764160,"available":96329035776,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.46999999999999997,"15min":0.5},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.27900000000000003},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.85399999999999998},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.53400000000000003,"15min":0.56000000000000005},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66900000000000004,"5min":0.77300000000000002,"15min":1.6359999999999999},"last":0.58099999999999996}]},{"osd":1,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56499999999999995,"15min":0.57999999999999996},"min":{"1min":0.40000000000000002,"5min":0.38200000000000001,"15min":0.33500000000000002},"max":{"1min":0.70099999999999996,"5min":0.88800000000000001,"15min":0.93000000000000005},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.56399999999999995,"15min":0.59299999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.64800000000000002,"5min":0.97299999999999998,"15min":1.145},"last":0.76500000000000001}]},{"osd":3,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52900000000000003,"15min":0.55000000000000004},"min":{"1min":0.376,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81499999999999995,"5min":0.83099999999999996,"15min":0.96399999999999997},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.61199999999999999,"15min":0.65000000000000002},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.75600000000000001,"5min":0.876,"15min":1.3620000000000001},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.75700000000000001,"15min":0.79100000000000004},"min":{"1min":0.56299999999999994,"5min":0.55800000000000005,"15min":0.54300000000000004},"max":{"1min":0.96499999999999997,"5min":1.071,"15min":1.1519999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.76400000000000001,"15min":0.79300000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0209999999999999,"5min":1.161,"15min":1.161},"last":0.77900000000000003}]},{"osd":5,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.79900000000000004,"15min":0.85599999999999998},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.94099999999999995,"5min":1.228,"15min":1.429},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.82799999999999996,"15min":0.85299999999999998},"min":{"1min":0.61499999999999999,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":0.94699999999999995,"5min":1.1850000000000001,"15min":1.2170000000000001},"last":0.81899999999999995}]},{"osd":6,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.77600000000000002,"15min":0.81799999999999995},"min":{"1min":0.56000000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96499999999999997,"5min":1.1659999999999999,"15min":1.4139999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.80400000000000005,"15min":0.85599999999999998},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.94999999999999996,"5min":1.2010000000000001,"15min":1.593},"last":0.79800000000000004}]},{"osd":7,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.84399999999999997,"15min":0.88400000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.57299999999999995},"max":{"1min":1.004,"5min":1.1339999999999999,"15min":2.0539999999999998},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.83499999999999996,"15min":0.872},"min":{"1min":0.624,"5min":0.60799999999999998,"15min":0.55500000000000005},"max":{"1min":1.026,"5min":1.2270000000000001,"15min":1.2490000000000001},"last":0.85499999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":304612,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067228,"statfs":{"total":96636764160,"available":96324841472,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.47699999999999998,"15min":0.50900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.61699999999999999,"5min":0.89700000000000002,"15min":0.90000000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.52200000000000002,"15min":0.57499999999999996},"min":{"1min":0.35799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69999999999999996,"5min":1.448,"15min":1.448},"last":0.52800000000000002}]},{"osd":1,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46800000000000003,"15min":0.503},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73199999999999998,"5min":0.90900000000000003,"15min":1.304},"last":0.497},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.55500000000000005,"15min":0.59599999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.71499999999999997,"5min":0.84599999999999997,"15min":1.4590000000000001},"last":0.38}]},{"osd":2,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.54600000000000004,"15min":0.59499999999999997},"min":{"1min":0.34899999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.78600000000000003,"5min":1.054,"15min":1.054},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.61399999999999999,"15min":0.64900000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.76100000000000001,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.70899999999999996}]},{"osd":4,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72299999999999998,"15min":0.77600000000000002},"min":{"1min":0.52600000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.93400000000000005,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.746,"15min":0.79600000000000004},"min":{"1min":0.51800000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.88800000000000001,"5min":1.171,"15min":1.1830000000000001},"last":0.623}]},{"osd":5,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.755,"15min":0.80600000000000005},"min":{"1min":0.497,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.02,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.75},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78500000000000003,"15min":0.83399999999999996},"min":{"1min":0.55600000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.054,"5min":1.2669999999999999,"15min":1.3440000000000001},"last":0.66300000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.79900000000000004,"15min":0.84299999999999997},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.97799999999999998,"5min":1.181,"15min":1.2170000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.77800000000000002,"15min":0.81799999999999995},"min":{"1min":0.59599999999999997,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.81100000000000005,"5min":1.883,"15min":1.883},"last":0.64300000000000002}]},{"osd":7,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.79700000000000004,"15min":0.85399999999999998},"min":{"1min":0.51600000000000001,"5min":0.505,"15min":0.505},"max":{"1min":0.89600000000000002,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.746,"5min":0.81200000000000006,"15min":0.876},"min":{"1min":0.59899999999999998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.97599999999999998,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.65600000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575996,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.68700000000000006,"15min":0.748},"min":{"1min":0.47299999999999998,"5min":0.42099999999999999,"15min":0.39900000000000002},"max":{"1min":0.86899999999999999,"5min":0.91200000000000003,"15min":1.0760000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.70699999999999996,"15min":0.75700000000000001},"min":{"1min":0.52800000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84599999999999997,"5min":1.853,"15min":1.853},"last":0.77500000000000002}]},{"osd":1,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.67600000000000005,"15min":0.72599999999999998},"min":{"1min":0.42299999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.91500000000000004,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.67000000000000004,"15min":0.71999999999999997},"min":{"1min":0.44,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.93100000000000005,"5min":1.639,"15min":1.639},"last":0.504}]},{"osd":2,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.68600000000000005,"15min":0.73599999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.44700000000000001},"max":{"1min":0.94099999999999995,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.68700000000000006,"15min":0.73199999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.014,"5min":1.014,"15min":1.0449999999999999},"last":0.56599999999999995}]},{"osd":3,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.72499999999999998},"min":{"1min":0.47799999999999998,"5min":0.436,"15min":0.436},"max":{"1min":0.91900000000000004,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73699999999999999,"15min":0.76800000000000002},"min":{"1min":0.59799999999999998,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.98899999999999999,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.72499999999999998}]},{"osd":5,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.629,"15min":0.624},"min":{"1min":0.35299999999999998,"5min":0.311,"15min":0.311},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.66300000000000003,"15min":0.64500000000000002},"min":{"1min":0.47499999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0580000000000001,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.82199999999999995}]},{"osd":6,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68999999999999995,"15min":0.67200000000000004},"min":{"1min":0.441,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0680000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.72399999999999998,"15min":0.70499999999999996},"min":{"1min":0.39400000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.1899999999999999,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.72799999999999998,"15min":0.70199999999999996},"min":{"1min":0.44700000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0409999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.78200000000000003,"15min":0.755},"min":{"1min":0.51300000000000001,"5min":0.436,"15min":0.41699999999999998},"max":{"1min":1.109,"5min":2.052,"15min":2.052},"last":0.92300000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.71399999999999997,"15min":0.72999999999999998},"min":{"1min":0.42899999999999999,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.72599999999999998,"15min":0.72999999999999998},"min":{"1min":0.46400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.9900000000000002,"5min":2.9900000000000002,"15min":2.9900000000000002},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.71799999999999997,"15min":0.73499999999999999},"min":{"1min":0.46000000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.873,"5min":0.75600000000000001,"15min":0.748},"min":{"1min":0.47699999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":5.4039999999999999,"5min":5.4039999999999999,"15min":5.4039999999999999},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.54300000000000004,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.7120000000000002,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.73999999999999999,"15min":0.74299999999999999},"min":{"1min":0.51700000000000002,"5min":0.439,"15min":0.439},"max":{"1min":2.5099999999999998,"5min":2.5099999999999998,"15min":2.5099999999999998},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.745,"15min":0.75700000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.7799999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":0.628},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.79900000000000004,"15min":0.78300000000000003},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":5.5570000000000004,"5min":5.5570000000000004,"15min":5.5570000000000004},"last":0.71499999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.64600000000000002,"15min":0.61699999999999999},"min":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41699999999999998},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.66500000000000004,"15min":0.63800000000000001},"min":{"1min":0.40400000000000003,"5min":0.40200000000000002,"15min":0.375},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":0.69799999999999995}]},{"osd":6,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.67400000000000004,"15min":0.65000000000000002},"min":{"1min":0.42899999999999999,"5min":0.42099999999999999,"15min":0.40999999999999998},"max":{"1min":2.9500000000000002,"5min":2.9500000000000002,"15min":2.9500000000000002},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.6280000000000001,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.748,"15min":0.73199999999999998},"min":{"1min":0.52700000000000002,"5min":0.47299999999999998,"15min":0.434},"max":{"1min":2.399,"5min":2.399,"15min":2.399},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.80400000000000005,"15min":0.78200000000000003},"min":{"1min":0.61399999999999999,"5min":0.48799999999999999,"15min":0.47599999999999998},"max":{"1min":2.7570000000000001,"5min":2.7570000000000001,"15min":2.7570000000000001},"last":0.90700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:47:58.715 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-11T09:47:58.716 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:47:58.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.923+0000 7f2e9f3ff700 1 Processor -- start 2024-01-11T09:47:58.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.923+0000 7f2e9f3ff700 1 -- start start 2024-01-11T09:47:58.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.924+0000 7f2e9f3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2e980ca7b0 0x7f2e980cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.924+0000 7f2e9f3ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e9815d640 con 0x7f2e980ca3e0 2024-01-11T09:47:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.924+0000 7f2e9f3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e9815a5d0 con 0x7f2e98159e80 2024-01-11T09:47:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.924+0000 7f2e9f3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e9815ae70 con 0x7f2e980ca7b0 2024-01-11T09:47:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.924+0000 7f2e9d19b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2e980ca3e0 0x7f2e980c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46320/0 (socket says 172.21.15.27:46320) 2024-01-11T09:47:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.924+0000 7f2e9d19b700 1 -- 172.21.15.27:0/1890806252 learned_addr learned my addr 172.21.15.27:0/1890806252 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.925+0000 7f2e9c99a700 1 --2- 172.21.15.27:0/1890806252 >> v2:172.21.15.77:3300/0 conn(0x7f2e980ca7b0 0x7f2e980cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.925+0000 7f2e8ffff700 1 -- 172.21.15.27:0/1890806252 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3694104963 0 0) 0x7f2e9815d640 con 0x7f2e980ca3e0 2024-01-11T09:47:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.925+0000 7f2e8ffff700 1 -- 172.21.15.27:0/1890806252 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e84003570 con 0x7f2e980ca3e0 2024-01-11T09:47:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.925+0000 7f2e8ffff700 1 -- 172.21.15.27:0/1890806252 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182780783 0 0) 0x7f2e9815a5d0 con 0x7f2e98159e80 2024-01-11T09:47:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.925+0000 7f2e8ffff700 1 -- 172.21.15.27:0/1890806252 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e9815d640 con 0x7f2e98159e80 2024-01-11T09:47:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.925+0000 7f2e8ffff700 1 -- 172.21.15.27:0/1890806252 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 136220812 0 0) 0x7f2e84003570 con 0x7f2e980ca3e0 2024-01-11T09:47:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.925+0000 7f2e8ffff700 1 -- 172.21.15.27:0/1890806252 >> v2:172.21.15.77:3300/0 conn(0x7f2e980ca7b0 msgr2=0x7f2e980cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.925+0000 7f2e8ffff700 1 --2- 172.21.15.27:0/1890806252 >> v2:172.21.15.77:3300/0 conn(0x7f2e980ca7b0 0x7f2e980cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.925+0000 7f2e8ffff700 1 -- 172.21.15.27:0/1890806252 >> v1:172.21.15.77:6789/0 conn(0x7f2e98159e80 legacy=0x7f2e9815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.925+0000 7f2e8ffff700 1 -- 172.21.15.27:0/1890806252 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9815a8b0 con 0x7f2e980ca3e0 2024-01-11T09:47:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.925+0000 7f2e8ffff700 1 -- 172.21.15.27:0/1890806252 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f2e94001470 con 0x7f2e980ca3e0 2024-01-11T09:47:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.925+0000 7f2e8ffff700 1 -- 172.21.15.27:0/1890806252 >> v1:172.21.15.27:6789/0 conn(0x7f2e980ca3e0 legacy=0x7f2e980c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.925+0000 7f2e8ffff700 1 --2- 172.21.15.27:0/1890806252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e84003c70 0x7f2e84004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.925+0000 7f2e8ffff700 1 --2- 172.21.15.27:0/1890806252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e84004810 0x7f2e84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.926+0000 7f2e8ffff700 1 --2- 172.21.15.27:0/1890806252 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e84005210 0x7f2e8400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.926+0000 7f2e8ffff700 1 -- 172.21.15.27:0/1890806252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9815ae70 con 0x7f2e84003c70 2024-01-11T09:47:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.926+0000 7f2e8ffff700 1 -- 172.21.15.27:0/1890806252 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8400c1e0 con 0x7f2e84004810 2024-01-11T09:47:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.926+0000 7f2e8ffff700 1 -- 172.21.15.27:0/1890806252 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8400c3a0 con 0x7f2e84005210 2024-01-11T09:47:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.926+0000 7f2e9d19b700 1 --2- 172.21.15.27:0/1890806252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e84003c70 0x7f2e84004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.926+0000 7f2e9c99a700 1 --2- 172.21.15.27:0/1890806252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e84004810 0x7f2e84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.926+0000 7f2e9d99c700 1 --2- 172.21.15.27:0/1890806252 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e84005210 0x7f2e8400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.926+0000 7f2e9d19b700 1 -- 172.21.15.27:0/1890806252 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e84005210 msgr2=0x7f2e8400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.926+0000 7f2e9d19b700 1 --2- 172.21.15.27:0/1890806252 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e84005210 0x7f2e8400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.926+0000 7f2e9d19b700 1 -- 172.21.15.27:0/1890806252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e84004810 msgr2=0x7f2e84004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.926+0000 7f2e9d19b700 1 --2- 172.21.15.27:0/1890806252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e84004810 0x7f2e84004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.926+0000 7f2e9d19b700 1 -- 172.21.15.27:0/1890806252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e84003570 con 0x7f2e84003c70 2024-01-11T09:47:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.926+0000 7f2e9d99c700 1 -- 172.21.15.27:0/1890806252 reap_dead start 2024-01-11T09:47:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.926+0000 7f2e9d19b700 1 --2- 172.21.15.27:0/1890806252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e84003c70 0x7f2e84004060 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f2e840057f0 tx=0x7f2e94001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c525fcdaed628305 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.927+0000 7f2e8ffff700 1 -- 172.21.15.27:0/1890806252 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9400cbb0 con 0x7f2e84003c70 2024-01-11T09:47:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.927+0000 7f2e8ffff700 1 -- 172.21.15.27:0/1890806252 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e9400cd10 con 0x7f2e84003c70 2024-01-11T09:47:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.927+0000 7f2e8ffff700 1 -- 172.21.15.27:0/1890806252 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e94005b70 con 0x7f2e84003c70 2024-01-11T09:47:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.927+0000 7f2e9f3ff700 1 -- 172.21.15.27:0/1890806252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e84003c70 msgr2=0x7f2e84004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.927+0000 7f2e9f3ff700 1 --2- 172.21.15.27:0/1890806252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e84003c70 0x7f2e84004060 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f2e840057f0 tx=0x7f2e94001d40 comp rx=0 tx=0).stop 2024-01-11T09:47:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.927+0000 7f2e9f3ff700 1 -- 172.21.15.27:0/1890806252 shutdown_connections 2024-01-11T09:47:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.927+0000 7f2e9f3ff700 1 --2- 172.21.15.27:0/1890806252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e84003c70 0x7f2e84004060 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.927+0000 7f2e9f3ff700 1 -- 172.21.15.27:0/1890806252 >> 172.21.15.27:0/1890806252 conn(0x7f2e980ce3e0 msgr2=0x7f2e980ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.927+0000 7f2e9f3ff700 1 -- 172.21.15.27:0/1890806252 shutdown_connections 2024-01-11T09:47:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.927+0000 7f2e9f3ff700 1 -- 172.21.15.27:0/1890806252 wait complete. 2024-01-11T09:47:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.929+0000 7f2e9f3ff700 1 Processor -- start 2024-01-11T09:47:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.929+0000 7f2e9f3ff700 1 -- start start 2024-01-11T09:47:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.930+0000 7f2e9f3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e84003c70 0x7f2e980b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.930+0000 7f2e9f3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e98151500 0x7f2e9815f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.930+0000 7f2e9f3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e9815fe20 0x7f2e9816d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.930+0000 7f2e9f3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980b4160 con 0x7f2e98151500 2024-01-11T09:47:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.930+0000 7f2e9f3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e98160290 con 0x7f2e9815fe20 2024-01-11T09:47:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.930+0000 7f2e9f3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9815eac0 con 0x7f2e84003c70 2024-01-11T09:47:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.930+0000 7f2e9c99a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e98151500 0x7f2e9815f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.930+0000 7f2e9c99a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e98151500 0x7f2e9815f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42350/0 (socket says 172.21.15.27:42350) 2024-01-11T09:47:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.930+0000 7f2e9c99a700 1 -- 172.21.15.27:0/1968734330 learned_addr learned my addr 172.21.15.27:0/1968734330 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.930+0000 7f2e9d19b700 1 --2- 172.21.15.27:0/1968734330 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e84003c70 0x7f2e980b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.930+0000 7f2e9d99c700 1 --2- 172.21.15.27:0/1968734330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e9815fe20 0x7f2e9816d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.931+0000 7f2e9c99a700 1 -- 172.21.15.27:0/1968734330 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e84003c70 msgr2=0x7f2e980b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.931+0000 7f2e9c99a700 1 --2- 172.21.15.27:0/1968734330 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e84003c70 0x7f2e980b3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.931+0000 7f2e9c99a700 1 -- 172.21.15.27:0/1968734330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e9815fe20 msgr2=0x7f2e9816d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.931+0000 7f2e9c99a700 1 --2- 172.21.15.27:0/1968734330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e9815fe20 0x7f2e9816d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.931+0000 7f2e9c99a700 1 -- 172.21.15.27:0/1968734330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9815deb0 con 0x7f2e98151500 2024-01-11T09:47:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.931+0000 7f2e9d99c700 1 --2- 172.21.15.27:0/1968734330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e9815fe20 0x7f2e9816d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.931+0000 7f2e9d19b700 1 --2- 172.21.15.27:0/1968734330 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e84003c70 0x7f2e980b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.931+0000 7f2e9c99a700 1 --2- 172.21.15.27:0/1968734330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e98151500 0x7f2e9815f8e0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f2e88011a40 tx=0x7f2e880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.931+0000 7f2e8d7fa700 1 -- 172.21.15.27:0/1968734330 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8800f940 con 0x7f2e98151500 2024-01-11T09:47:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.931+0000 7f2e8d7fa700 1 -- 172.21.15.27:0/1968734330 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8800faa0 con 0x7f2e98151500 2024-01-11T09:47:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.931+0000 7f2e9f3ff700 1 -- 172.21.15.27:0/1968734330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e9815e190 con 0x7f2e98151500 2024-01-11T09:47:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.931+0000 7f2e8d7fa700 1 -- 172.21.15.27:0/1968734330 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8800fe20 con 0x7f2e98151500 2024-01-11T09:47:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.931+0000 7f2e9f3ff700 1 -- 172.21.15.27:0/1968734330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e9815e750 con 0x7f2e98151500 2024-01-11T09:47:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.933+0000 7f2e9f3ff700 1 -- 172.21.15.27:0/1968734330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e980c9560 con 0x7f2e98151500 2024-01-11T09:47:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.933+0000 7f2e8d7fa700 1 -- 172.21.15.27:0/1968734330 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f2e88014650 con 0x7f2e98151500 2024-01-11T09:47:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.934+0000 7f2e8d7fa700 1 --2- 172.21.15.27:0/1968734330 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2e7c066f30 0x7f2e7c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.934+0000 7f2e8d7fa700 1 -- 172.21.15.27:0/1968734330 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f2e88018030 con 0x7f2e98151500 2024-01-11T09:47:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.935+0000 7f2e9d19b700 1 --2- 172.21.15.27:0/1968734330 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2e7c066f30 0x7f2e7c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:58.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.935+0000 7f2e9d19b700 1 --2- 172.21.15.27:0/1968734330 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2e7c066f30 0x7f2e7c0693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2e98160d50 tx=0x7f2e94015000 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:58.938+0000 7f2e8d7fa700 1 -- 172.21.15.27:0/1968734330 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2e88055c50 con 0x7f2e98151500 2024-01-11T09:47:59.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.139+0000 7f2e9f3ff700 1 -- 172.21.15.27:0/1968734330 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2e980718d0 con 0x7f2e7c066f30 2024-01-11T09:47:59.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.142+0000 7f2e8d7fa700 1 -- 172.21.15.27:0/1968734330 <== mgr.11878 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50741 (secure 0 0 0) 0x7f2e980718d0 con 0x7f2e7c066f30 2024-01-11T09:47:59.135 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.144+0000 7f2e9f3ff700 1 -- 172.21.15.27:0/1968734330 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2e7c066f30 msgr2=0x7f2e7c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.144+0000 7f2e9f3ff700 1 --2- 172.21.15.27:0/1968734330 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2e7c066f30 0x7f2e7c0693e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2e98160d50 tx=0x7f2e94015000 comp rx=0 tx=0).stop 2024-01-11T09:47:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.144+0000 7f2e9f3ff700 1 -- 172.21.15.27:0/1968734330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e98151500 msgr2=0x7f2e9815f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.144+0000 7f2e9f3ff700 1 --2- 172.21.15.27:0/1968734330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e98151500 0x7f2e9815f8e0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f2e88011a40 tx=0x7f2e880091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.144+0000 7f2e9f3ff700 1 -- 172.21.15.27:0/1968734330 shutdown_connections 2024-01-11T09:47:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.144+0000 7f2e9f3ff700 1 --2- 172.21.15.27:0/1968734330 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f2e7c066f30 0x7f2e7c0693e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.144+0000 7f2e9f3ff700 1 --2- 172.21.15.27:0/1968734330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e9815fe20 0x7f2e9816d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.144+0000 7f2e9f3ff700 1 --2- 172.21.15.27:0/1968734330 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f2e84003c70 0x7f2e980b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.144+0000 7f2e9f3ff700 1 --2- 172.21.15.27:0/1968734330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e98151500 0x7f2e9815f8e0 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.144+0000 7f2e9f3ff700 1 -- 172.21.15.27:0/1968734330 >> 172.21.15.27:0/1968734330 conn(0x7f2e980ce3e0 msgr2=0x7f2e980c4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.144+0000 7f2e9f3ff700 1 -- 172.21.15.27:0/1968734330 shutdown_connections 2024-01-11T09:47:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.144+0000 7f2e9f3ff700 1 -- 172.21.15.27:0/1968734330 wait complete. 2024-01-11T09:47:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:47:59.155 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-01-11T09:47:58.566494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2846,"num_read_kb":6029,"num_write":727,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":430,"ondisk_log_size":430,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":2470824,"kb_used_data":26984,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752503896,"statfs":{"total":773094113280,"available":770563989504,"internally_reserved":0,"allocated":27631616,"data_stored":13999136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-668747,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-221,"num_read_kb":-75,"num_write":-261,"num_write_kb":-2052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.999265"},"pg_stats":[{"pgid":"1.0","version":"260'430","reported_seq":1465,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-01-11T09:47:52.627901+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:47:52.627901+0000","last_peered":"2024-01-11T09:47:52.627901+0000","last_clean":"2024-01-11T09:47:52.627901+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:47:52.627901+0000","last_undegraded":"2024-01-11T09:47:52.627901+0000","last_fullsized":"2024-01-11T09:47:52.627901+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":430,"log_dups_size":0,"ondisk_log_size":430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2846,"num_read_kb":6029,"num_write":727,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2846,"num_read_kb":6029,"num_write":727,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":430,"ondisk_log_size":430,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305192,"kb_used_data":3808,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94066648,"statfs":{"total":96636764160,"available":96324247552,"internally_reserved":0,"allocated":3899392,"data_stored":2192668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.66500000000000004,"15min":0.71899999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36599999999999999},"max":{"1min":0.81799999999999995,"5min":0.95699999999999996,"15min":1.0529999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.69299999999999995,"15min":0.73399999999999999},"min":{"1min":0.41699999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.97999999999999998,"5min":1.026,"15min":1.0669999999999999},"last":0.84599999999999997}]},{"osd":1,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.71999999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.0209999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69699999999999995,"15min":0.73099999999999998},"min":{"1min":0.44800000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.91800000000000004,"5min":0.95899999999999996,"15min":1.0820000000000001},"last":0.65500000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.69999999999999996,"15min":0.72899999999999998},"min":{"1min":0.48899999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.85399999999999998,"5min":0.93799999999999994,"15min":1.0600000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70099999999999996,"15min":0.74399999999999999},"min":{"1min":0.52200000000000002,"5min":0.44800000000000001,"15min":0.44500000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":1.153},"last":0.88100000000000001}]},{"osd":3,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73499999999999999,"15min":0.76000000000000001},"min":{"1min":0.5,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.238,"5min":1.238,"15min":1.238},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70899999999999996,"15min":0.74199999999999999},"min":{"1min":0.47699999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.89400000000000002,"5min":0.94099999999999995,"15min":1.0880000000000001},"last":1.024}]},{"osd":4,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.61899999999999999,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0209999999999999,"5min":1.0860000000000001,"15min":1.107},"last":1.004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.65900000000000003,"15min":0.64500000000000002},"min":{"1min":0.443,"5min":0.437,"15min":0.39700000000000002},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.90800000000000003}]},{"osd":5,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63200000000000001,"15min":0.64200000000000002},"min":{"1min":0.50900000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.90800000000000003},"last":1.101},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.70799999999999996,"15min":0.69399999999999995},"min":{"1min":0.495,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":1.1160000000000001}]},{"osd":6,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.77700000000000002,"15min":0.78300000000000003},"min":{"1min":0.54900000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0649999999999999,"5min":1.232,"15min":1.232},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.76800000000000002,"15min":0.76700000000000002},"min":{"1min":0.52300000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":1.1299999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.63200000000000001,"15min":0.69499999999999995},"min":{"1min":0.438,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.79800000000000004,"5min":0.94199999999999995,"15min":0.999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.621,"5min":0.68400000000000005,"15min":0.72799999999999998},"min":{"1min":0.46100000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.93400000000000005,"5min":0.97099999999999997,"15min":1.216},"last":0.72099999999999997}]},{"osd":1,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.66000000000000003,"15min":0.72199999999999998},"min":{"1min":0.46200000000000002,"5min":0.42799999999999999,"15min":0.41699999999999998},"max":{"1min":0.91000000000000003,"5min":0.98499999999999999,"15min":1.3129999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.67000000000000004,"15min":0.71599999999999997},"min":{"1min":0.46500000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.91100000000000003,"5min":0.95999999999999996,"15min":1.0840000000000001},"last":0.755}]},{"osd":2,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.68700000000000006,"15min":0.73499999999999999},"min":{"1min":0.44600000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.83799999999999997,"5min":1.0069999999999999,"15min":1.4179999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.69999999999999996,"15min":0.73999999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.92800000000000005,"5min":1.796,"15min":1.796},"last":0.63700000000000001}]},{"osd":3,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.71299999999999997,"15min":0.746},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89000000000000001,"5min":1.012,"15min":1.014},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.70099999999999996,"15min":0.754},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":1.165},"last":0.74099999999999999}]},{"osd":4,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.59399999999999997,"15min":0.56999999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.86599999999999999,"5min":0.91700000000000004,"15min":1.0309999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.64400000000000002,"15min":0.622},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":1.1259999999999999},"last":0.56200000000000006}]},{"osd":5,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.077,"5min":1.077,"15min":1.369},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.71699999999999997,"15min":0.70899999999999996},"min":{"1min":0.495,"5min":0.49099999999999999,"15min":0.45400000000000001},"max":{"1min":0.94399999999999995,"5min":0.999,"15min":1.0960000000000001},"last":0.82099999999999995}]},{"osd":7,"last update":"Thu Jan 11 09:47:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.72099999999999997,"15min":0.71399999999999997},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.45100000000000001},"max":{"1min":1.0089999999999999,"5min":1.089,"15min":1.484},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.77500000000000002,"15min":0.77200000000000002},"min":{"1min":0.53700000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.123,"5min":1.1379999999999999,"15min":1.603},"last":0.83499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575997,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.502,"15min":0.51000000000000001},"min":{"1min":0.36299999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.70699999999999996,"5min":0.73399999999999999,"15min":0.93999999999999995},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.60699999999999998},"min":{"1min":0.41899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73499999999999999,"5min":0.85199999999999998,"15min":0.95099999999999996},"last":0.495}]},{"osd":2,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.47199999999999998,"15min":0.501},"min":{"1min":0.35199999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.63800000000000001,"5min":0.82399999999999995,"15min":0.82699999999999996},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56599999999999995,"15min":0.58399999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.69999999999999996,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.62,"15min":0.64300000000000002},"min":{"1min":0.45500000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.76000000000000001,"5min":0.85199999999999998,"15min":0.96699999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55500000000000005,"15min":0.57199999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.44700000000000001}]},{"osd":4,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.76400000000000001,"15min":0.80400000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.159,"5min":1.214,"15min":1.2969999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.81799999999999995,"15min":0.84999999999999998},"min":{"1min":0.58699999999999997,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.169,"5min":1.1879999999999999,"15min":1.1890000000000001},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.79100000000000004,"15min":0.81999999999999995},"min":{"1min":0.55600000000000005,"5min":0.443,"15min":0.443},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.78800000000000003,"15min":0.80200000000000005},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.51300000000000001},"max":{"1min":1.1539999999999999,"5min":1.341,"15min":1.341},"last":0.624}]},{"osd":6,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.80400000000000005,"15min":0.83799999999999997},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.1830000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.81799999999999995,"15min":0.84599999999999997},"min":{"1min":0.57299999999999995,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.61099999999999999}]},{"osd":7,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.86599999999999999,"15min":0.88200000000000001},"min":{"1min":0.63900000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.84299999999999997,"15min":0.85399999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.391,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.68999999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575996,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309288,"kb_used_data":3808,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062552,"statfs":{"total":96636764160,"available":96320053248,"internally_reserved":0,"allocated":3899392,"data_stored":2192668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.48799999999999999,"15min":0.51300000000000001},"min":{"1min":0.34899999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.73099999999999998,"5min":0.754,"15min":0.84699999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.56299999999999994,"15min":0.59099999999999997},"min":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.59599999999999997,"5min":0.81599999999999995,"15min":0.874},"last":0.50600000000000001}]},{"osd":2,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.53300000000000003,"15min":0.56200000000000006},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.68200000000000005,"5min":0.83099999999999996,"15min":0.97499999999999998},"last":0.379},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55400000000000005,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.86399999999999999},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51100000000000001,"15min":0.53800000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.95399999999999996},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60699999999999998,"15min":0.64400000000000002},"min":{"1min":0.42499999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.87,"5min":0.90300000000000002,"15min":1.0389999999999999},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.78600000000000003,"15min":0.81100000000000005},"min":{"1min":0.46100000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.304},"last":1.091},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.78500000000000003,"15min":0.80100000000000005},"min":{"1min":0.48799999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.95399999999999996,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":1.1160000000000001}]},{"osd":5,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.80800000000000005,"15min":0.83599999999999997},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.97599999999999998,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.81399999999999995,"15min":0.83699999999999997},"min":{"1min":0.60399999999999998,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.02,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.77300000000000002}]},{"osd":6,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.79700000000000004,"15min":0.82099999999999995},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.95899999999999996,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.82299999999999995,"15min":0.85999999999999999},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":0.97699999999999998,"5min":1.1699999999999999,"15min":1.256},"last":1.048}]},{"osd":7,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.83699999999999997,"15min":0.86299999999999999},"min":{"1min":0.64100000000000001,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1000000000000001,"5min":1.226,"15min":1.28},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.84499999999999997,"15min":0.86899999999999999},"min":{"1min":0.63,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0369999999999999,"5min":1.238,"15min":1.2410000000000001},"last":1.173}]}]},{"osd":2,"up_from":13,"seq":55834575995,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":300516,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071324,"statfs":{"total":96636764160,"available":96329035776,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.46999999999999997,"15min":0.5},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.27900000000000003},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.85399999999999998},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.53400000000000003,"15min":0.56000000000000005},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66900000000000004,"5min":0.77300000000000002,"15min":1.6359999999999999},"last":0.58099999999999996}]},{"osd":1,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56499999999999995,"15min":0.57999999999999996},"min":{"1min":0.40000000000000002,"5min":0.38200000000000001,"15min":0.33500000000000002},"max":{"1min":0.70099999999999996,"5min":0.88800000000000001,"15min":0.93000000000000005},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.56399999999999995,"15min":0.59299999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.64800000000000002,"5min":0.97299999999999998,"15min":1.145},"last":0.76500000000000001}]},{"osd":3,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52900000000000003,"15min":0.55000000000000004},"min":{"1min":0.376,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81499999999999995,"5min":0.83099999999999996,"15min":0.96399999999999997},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.61199999999999999,"15min":0.65000000000000002},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.75600000000000001,"5min":0.876,"15min":1.3620000000000001},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.75700000000000001,"15min":0.79100000000000004},"min":{"1min":0.56299999999999994,"5min":0.55800000000000005,"15min":0.54300000000000004},"max":{"1min":0.96499999999999997,"5min":1.071,"15min":1.1519999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.76400000000000001,"15min":0.79300000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0209999999999999,"5min":1.161,"15min":1.161},"last":0.77900000000000003}]},{"osd":5,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.79900000000000004,"15min":0.85599999999999998},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.94099999999999995,"5min":1.228,"15min":1.429},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.82799999999999996,"15min":0.85299999999999998},"min":{"1min":0.61499999999999999,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":0.94699999999999995,"5min":1.1850000000000001,"15min":1.2170000000000001},"last":0.81899999999999995}]},{"osd":6,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.77600000000000002,"15min":0.81799999999999995},"min":{"1min":0.56000000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96499999999999997,"5min":1.1659999999999999,"15min":1.4139999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.80400000000000005,"15min":0.85599999999999998},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.94999999999999996,"5min":1.2010000000000001,"15min":1.593},"last":0.79800000000000004}]},{"osd":7,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.84399999999999997,"15min":0.88400000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.57299999999999995},"max":{"1min":1.004,"5min":1.1339999999999999,"15min":2.0539999999999998},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.83499999999999996,"15min":0.872},"min":{"1min":0.624,"5min":0.60799999999999998,"15min":0.55500000000000005},"max":{"1min":1.026,"5min":1.2270000000000001,"15min":1.2490000000000001},"last":0.85499999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":304612,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067228,"statfs":{"total":96636764160,"available":96324841472,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.47699999999999998,"15min":0.50900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.61699999999999999,"5min":0.89700000000000002,"15min":0.90000000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.52200000000000002,"15min":0.57499999999999996},"min":{"1min":0.35799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69999999999999996,"5min":1.448,"15min":1.448},"last":0.52800000000000002}]},{"osd":1,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46800000000000003,"15min":0.503},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73199999999999998,"5min":0.90900000000000003,"15min":1.304},"last":0.497},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.55500000000000005,"15min":0.59599999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.71499999999999997,"5min":0.84599999999999997,"15min":1.4590000000000001},"last":0.38}]},{"osd":2,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.54600000000000004,"15min":0.59499999999999997},"min":{"1min":0.34899999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.78600000000000003,"5min":1.054,"15min":1.054},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.61399999999999999,"15min":0.64900000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.76100000000000001,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.70899999999999996}]},{"osd":4,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72299999999999998,"15min":0.77600000000000002},"min":{"1min":0.52600000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.93400000000000005,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.746,"15min":0.79600000000000004},"min":{"1min":0.51800000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.88800000000000001,"5min":1.171,"15min":1.1830000000000001},"last":0.623}]},{"osd":5,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.755,"15min":0.80600000000000005},"min":{"1min":0.497,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.02,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.75},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78500000000000003,"15min":0.83399999999999996},"min":{"1min":0.55600000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.054,"5min":1.2669999999999999,"15min":1.3440000000000001},"last":0.66300000000000003}]},{"osd":6,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.79900000000000004,"15min":0.84299999999999997},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.97799999999999998,"5min":1.181,"15min":1.2170000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.77800000000000002,"15min":0.81799999999999995},"min":{"1min":0.59599999999999997,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.81100000000000005,"5min":1.883,"15min":1.883},"last":0.64300000000000002}]},{"osd":7,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.79700000000000004,"15min":0.85399999999999998},"min":{"1min":0.51600000000000001,"5min":0.505,"15min":0.505},"max":{"1min":0.89600000000000002,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.746,"5min":0.81200000000000006,"15min":0.876},"min":{"1min":0.59899999999999998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.97599999999999998,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.65600000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575996,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.68700000000000006,"15min":0.748},"min":{"1min":0.47299999999999998,"5min":0.42099999999999999,"15min":0.39900000000000002},"max":{"1min":0.86899999999999999,"5min":0.91200000000000003,"15min":1.0760000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.70699999999999996,"15min":0.75700000000000001},"min":{"1min":0.52800000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84599999999999997,"5min":1.853,"15min":1.853},"last":0.77500000000000002}]},{"osd":1,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.67600000000000005,"15min":0.72599999999999998},"min":{"1min":0.42299999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.91500000000000004,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.67000000000000004,"15min":0.71999999999999997},"min":{"1min":0.44,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.93100000000000005,"5min":1.639,"15min":1.639},"last":0.504}]},{"osd":2,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.68600000000000005,"15min":0.73599999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.44700000000000001},"max":{"1min":0.94099999999999995,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.68700000000000006,"15min":0.73199999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.014,"5min":1.014,"15min":1.0449999999999999},"last":0.56599999999999995}]},{"osd":3,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.72499999999999998},"min":{"1min":0.47799999999999998,"5min":0.436,"15min":0.436},"max":{"1min":0.91900000000000004,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73699999999999999,"15min":0.76800000000000002},"min":{"1min":0.59799999999999998,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.98899999999999999,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.72499999999999998}]},{"osd":5,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.629,"15min":0.624},"min":{"1min":0.35299999999999998,"5min":0.311,"15min":0.311},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.66300000000000003,"15min":0.64500000000000002},"min":{"1min":0.47499999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0580000000000001,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.82199999999999995}]},{"osd":6,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68999999999999995,"15min":0.67200000000000004},"min":{"1min":0.441,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0680000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.72399999999999998,"15min":0.70499999999999996},"min":{"1min":0.39400000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.1899999999999999,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.72799999999999998,"15min":0.70199999999999996},"min":{"1min":0.44700000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0409999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.78200000000000003,"15min":0.755},"min":{"1min":0.51300000000000001,"5min":0.436,"15min":0.41699999999999998},"max":{"1min":1.109,"5min":2.052,"15min":2.052},"last":0.92300000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.71399999999999997,"15min":0.72999999999999998},"min":{"1min":0.42899999999999999,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.72599999999999998,"15min":0.72999999999999998},"min":{"1min":0.46400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.9900000000000002,"5min":2.9900000000000002,"15min":2.9900000000000002},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.71799999999999997,"15min":0.73499999999999999},"min":{"1min":0.46000000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.873,"5min":0.75600000000000001,"15min":0.748},"min":{"1min":0.47699999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":5.4039999999999999,"5min":5.4039999999999999,"15min":5.4039999999999999},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.54300000000000004,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.7120000000000002,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.73999999999999999,"15min":0.74299999999999999},"min":{"1min":0.51700000000000002,"5min":0.439,"15min":0.439},"max":{"1min":2.5099999999999998,"5min":2.5099999999999998,"15min":2.5099999999999998},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.745,"15min":0.75700000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.7799999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":0.628},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.79900000000000004,"15min":0.78300000000000003},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":5.5570000000000004,"5min":5.5570000000000004,"15min":5.5570000000000004},"last":0.71499999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.64600000000000002,"15min":0.61699999999999999},"min":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41699999999999998},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.66500000000000004,"15min":0.63800000000000001},"min":{"1min":0.40400000000000003,"5min":0.40200000000000002,"15min":0.375},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":0.69799999999999995}]},{"osd":6,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.67400000000000004,"15min":0.65000000000000002},"min":{"1min":0.42899999999999999,"5min":0.42099999999999999,"15min":0.40999999999999998},"max":{"1min":2.9500000000000002,"5min":2.9500000000000002,"15min":2.9500000000000002},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.6280000000000001,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.748,"15min":0.73199999999999998},"min":{"1min":0.52700000000000002,"5min":0.47299999999999998,"15min":0.434},"max":{"1min":2.399,"5min":2.399,"15min":2.399},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.80400000000000005,"15min":0.78200000000000003},"min":{"1min":0.61399999999999999,"5min":0.48799999999999999,"15min":0.47599999999999998},"max":{"1min":2.7570000000000001,"5min":2.7570000000000001,"15min":2.7570000000000001},"last":0.90700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:47:59.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T09:47:59.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.363+0000 7f3755fff700 1 Processor -- start 2024-01-11T09:47:59.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.363+0000 7f3755fff700 1 -- start start 2024-01-11T09:47:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.364+0000 7f3755fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f37500b2b90 0x7f3750159200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.364+0000 7f3755fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f37500b07b0 0x7f3750159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.364+0000 7f3755fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37500cd730 con 0x7f37500b03e0 2024-01-11T09:47:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.364+0000 7f3755fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37500ccb20 con 0x7f37500b2b90 2024-01-11T09:47:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.364+0000 7f3755fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37500cce00 con 0x7f37500b07b0 2024-01-11T09:47:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.364+0000 7f374f7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f37500b03e0 0x7f37500b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46330/0 (socket says 172.21.15.27:46330) 2024-01-11T09:47:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.364+0000 7f374f7fe700 1 -- 172.21.15.27:0/3093614131 learned_addr learned my addr 172.21.15.27:0/3093614131 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:47:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.365+0000 7f374effd700 1 --2- 172.21.15.27:0/3093614131 >> v2:172.21.15.77:3300/0 conn(0x7f37500b2b90 0x7f3750159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.365+0000 7f374ffff700 1 --2- 172.21.15.27:0/3093614131 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f37500b07b0 0x7f3750159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.365+0000 7f374e7fc700 1 -- 172.21.15.27:0/3093614131 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434863855 0 0) 0x7f37500cd730 con 0x7f37500b03e0 2024-01-11T09:47:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.365+0000 7f374e7fc700 1 -- 172.21.15.27:0/3093614131 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3738003570 con 0x7f37500b03e0 2024-01-11T09:47:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.365+0000 7f374effd700 1 -- 172.21.15.27:0/3093614131 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f37500b07b0 msgr2=0x7f3750159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:47:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.365+0000 7f374e7fc700 1 -- 172.21.15.27:0/3093614131 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1119720737 0 0) 0x7f3738003570 con 0x7f37500b03e0 2024-01-11T09:47:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.365+0000 7f374effd700 1 --2- 172.21.15.27:0/3093614131 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f37500b07b0 0x7f3750159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.365+0000 7f374effd700 1 -- 172.21.15.27:0/3093614131 >> v1:172.21.15.27:6789/0 conn(0x7f37500b03e0 legacy=0x7f37500b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.365+0000 7f374effd700 1 -- 172.21.15.27:0/3093614131 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37500cd0e0 con 0x7f37500b2b90 2024-01-11T09:47:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.365+0000 7f374ffff700 1 --2- 172.21.15.27:0/3093614131 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f37500b07b0 0x7f3750159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:47:59.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.365+0000 7f374effd700 1 --2- 172.21.15.27:0/3093614131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37500b2b90 0x7f3750159200 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f37440098e0 tx=0x7f37440091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5ccc4b155da777b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:59.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.366+0000 7f374e7fc700 1 -- 172.21.15.27:0/3093614131 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3740001470 con 0x7f37500b2b90 2024-01-11T09:47:59.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.366+0000 7f374e7fc700 1 -- 172.21.15.27:0/3093614131 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3744011420 con 0x7f37500b2b90 2024-01-11T09:47:59.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.366+0000 7f374e7fc700 1 -- 172.21.15.27:0/3093614131 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37440117c0 con 0x7f37500b2b90 2024-01-11T09:47:59.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.366+0000 7f3755fff700 1 -- 172.21.15.27:0/3093614131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37500b2b90 msgr2=0x7f3750159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:59.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.366+0000 7f3755fff700 1 --2- 172.21.15.27:0/3093614131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37500b2b90 0x7f3750159200 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f37440098e0 tx=0x7f37440091a0 comp rx=0 tx=0).stop 2024-01-11T09:47:59.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.366+0000 7f3755fff700 1 -- 172.21.15.27:0/3093614131 shutdown_connections 2024-01-11T09:47:59.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.366+0000 7f3755fff700 1 --2- 172.21.15.27:0/3093614131 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f37500b07b0 0x7f3750159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:59.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.366+0000 7f3755fff700 1 --2- 172.21.15.27:0/3093614131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37500b2b90 0x7f3750159200 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:59.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.366+0000 7f3755fff700 1 -- 172.21.15.27:0/3093614131 >> 172.21.15.27:0/3093614131 conn(0x7f375006ed40 msgr2=0x7f375006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:59.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.366+0000 7f3755fff700 1 -- 172.21.15.27:0/3093614131 shutdown_connections 2024-01-11T09:47:59.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.366+0000 7f3755fff700 1 -- 172.21.15.27:0/3093614131 wait complete. 2024-01-11T09:47:59.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.368+0000 7f3755fff700 1 Processor -- start 2024-01-11T09:47:59.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.368+0000 7f3755fff700 1 -- start start 2024-01-11T09:47:59.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.369+0000 7f3755fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37500b03e0 0x7f37501476a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:59.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.369+0000 7f3755fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37500b07b0 0x7f375015c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:59.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.369+0000 7f3755fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f37500b2b90 0x7f375015c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:59.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.369+0000 7f3755fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37500ca6b0 con 0x7f37500b07b0 2024-01-11T09:47:59.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.369+0000 7f3755fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37500ca990 con 0x7f37500b03e0 2024-01-11T09:47:59.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.369+0000 7f3755fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37500c3f40 con 0x7f37500b2b90 2024-01-11T09:47:59.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.369+0000 7f374effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37500b07b0 0x7f375015c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:59.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.369+0000 7f374effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37500b07b0 0x7f375015c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42352/0 (socket says 172.21.15.27:42352) 2024-01-11T09:47:59.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.369+0000 7f374effd700 1 -- 172.21.15.27:0/1862475853 learned_addr learned my addr 172.21.15.27:0/1862475853 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:47:59.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.370+0000 7f374f7fe700 1 --2- 172.21.15.27:0/1862475853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37500b03e0 0x7f37501476a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:59.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.370+0000 7f374ffff700 1 --2- 172.21.15.27:0/1862475853 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f37500b2b90 0x7f375015c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:59.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.370+0000 7f374effd700 1 -- 172.21.15.27:0/1862475853 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f37500b2b90 msgr2=0x7f375015c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:59.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.370+0000 7f374effd700 1 --2- 172.21.15.27:0/1862475853 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f37500b2b90 0x7f375015c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:59.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.370+0000 7f374effd700 1 -- 172.21.15.27:0/1862475853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37500b03e0 msgr2=0x7f37501476a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:59.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.370+0000 7f374effd700 1 --2- 172.21.15.27:0/1862475853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37500b03e0 0x7f37501476a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:59.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.370+0000 7f374effd700 1 -- 172.21.15.27:0/1862475853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3748009a20 con 0x7f37500b07b0 2024-01-11T09:47:59.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.370+0000 7f374ffff700 1 --2- 172.21.15.27:0/1862475853 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f37500b2b90 0x7f375015c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:47:59.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.370+0000 7f374effd700 1 --2- 172.21.15.27:0/1862475853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37500b07b0 0x7f375015c230 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f37440093f0 tx=0x7f3744016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:59.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.370+0000 7f37377fe700 1 -- 172.21.15.27:0/1862475853 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374401cc90 con 0x7f37500b07b0 2024-01-11T09:47:59.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.370+0000 7f37377fe700 1 -- 172.21.15.27:0/1862475853 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f374401cdf0 con 0x7f37500b07b0 2024-01-11T09:47:59.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.370+0000 7f37377fe700 1 -- 172.21.15.27:0/1862475853 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374401a660 con 0x7f37500b07b0 2024-01-11T09:47:59.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.370+0000 7f3755fff700 1 -- 172.21.15.27:0/1862475853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3744009420 con 0x7f37500b07b0 2024-01-11T09:47:59.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.371+0000 7f3755fff700 1 -- 172.21.15.27:0/1862475853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37500c4500 con 0x7f37500b07b0 2024-01-11T09:47:59.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.372+0000 7f3755fff700 1 -- 172.21.15.27:0/1862475853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37500c4740 con 0x7f37500b07b0 2024-01-11T09:47:59.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.372+0000 7f37377fe700 1 -- 172.21.15.27:0/1862475853 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f374401aa90 con 0x7f37500b07b0 2024-01-11T09:47:59.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.374+0000 7f37377fe700 1 --2- 172.21.15.27:0/1862475853 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f373c066f30 0x7f373c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:47:59.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.374+0000 7f37377fe700 1 -- 172.21.15.27:0/1862475853 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f3744085e30 con 0x7f37500b07b0 2024-01-11T09:47:59.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.375+0000 7f374f7fe700 1 --2- 172.21.15.27:0/1862475853 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f373c066f30 0x7f373c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:47:59.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.375+0000 7f374f7fe700 1 --2- 172.21.15.27:0/1862475853 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f373c066f30 0x7f373c0693e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3740005c20 tx=0x7f3740001120 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:47:59.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.378+0000 7f37377fe700 1 -- 172.21.15.27:0/1862475853 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f37500c4740 con 0x7f37500b07b0 2024-01-11T09:47:59.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.580+0000 7f3755fff700 1 -- 172.21.15.27:0/1862475853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37500c4740 con 0x7f37500b07b0 2024-01-11T09:47:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.581+0000 7f37377fe700 1 -- 172.21.15.27:0/1862475853 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v261) v1 ==== 74+0+13450 (secure 0 0 0) 0x7f37500c4740 con 0x7f37500b07b0 2024-01-11T09:47:59.574 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:47:59.574 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":261,"fsid":"8fddb4a7-9d66-40f1-892d-6fe86c728ee4","created":"2024-01-11T08:10:04.682726+0000","modified":"2024-01-11T09:47:53.552822+0000","last_up_change":"2024-01-11T08:10:17.982537+0000","last_in_change":"2024-01-11T08:10:08.812242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T08:10:19.576658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"21720a56-9ef9-4541-acb7-dcb4fcfbd10d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":41829},{"type":"v1","addr":"172.21.15.27:6801","nonce":41829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":41829},{"type":"v1","addr":"172.21.15.27:6803","nonce":41829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":41829},{"type":"v1","addr":"172.21.15.27:6807","nonce":41829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":41829},{"type":"v1","addr":"172.21.15.27:6805","nonce":41829}]},"public_addr":"172.21.15.27:6801/41829","cluster_addr":"172.21.15.27:6803/41829","heartbeat_back_addr":"172.21.15.27:6807/41829","heartbeat_front_addr":"172.21.15.27:6805/41829","state":["exists","up"]},{"osd":1,"uuid":"2de9c7c8-0153-4483-a1c5-0a70f8d6660b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":41835},{"type":"v1","addr":"172.21.15.27:6816","nonce":41835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6819","nonce":41835},{"type":"v1","addr":"172.21.15.27:6821","nonce":41835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6829","nonce":41835},{"type":"v1","addr":"172.21.15.27:6831","nonce":41835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":41835},{"type":"v1","addr":"172.21.15.27:6827","nonce":41835}]},"public_addr":"172.21.15.27:6816/41835","cluster_addr":"172.21.15.27:6821/41835","heartbeat_back_addr":"172.21.15.27:6831/41835","heartbeat_front_addr":"172.21.15.27:6827/41835","state":["exists","up"]},{"osd":2,"uuid":"bb18c833-73e9-4afd-96a2-103eb308745b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":41833},{"type":"v1","addr":"172.21.15.27:6811","nonce":41833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":41833},{"type":"v1","addr":"172.21.15.27:6817","nonce":41833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":41833},{"type":"v1","addr":"172.21.15.27:6830","nonce":41833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":41833},{"type":"v1","addr":"172.21.15.27:6826","nonce":41833}]},"public_addr":"172.21.15.27:6811/41833","cluster_addr":"172.21.15.27:6817/41833","heartbeat_back_addr":"172.21.15.27:6830/41833","heartbeat_front_addr":"172.21.15.27:6826/41833","state":["exists","up"]},{"osd":3,"uuid":"cef59f93-06ff-469c-a433-41cea02e7c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":41831},{"type":"v1","addr":"172.21.15.27:6810","nonce":41831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":41831},{"type":"v1","addr":"172.21.15.27:6815","nonce":41831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6823","nonce":41831},{"type":"v1","addr":"172.21.15.27:6825","nonce":41831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":41831},{"type":"v1","addr":"172.21.15.27:6820","nonce":41831}]},"public_addr":"172.21.15.27:6810/41831","cluster_addr":"172.21.15.27:6815/41831","heartbeat_back_addr":"172.21.15.27:6825/41831","heartbeat_front_addr":"172.21.15.27:6820/41831","state":["exists","up"]},{"osd":4,"uuid":"4cfd29cd-a247-49ca-b1d9-a56b522d54a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":42095},{"type":"v1","addr":"172.21.15.77:6801","nonce":42095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":42095},{"type":"v1","addr":"172.21.15.77:6803","nonce":42095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":42095},{"type":"v1","addr":"172.21.15.77:6807","nonce":42095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":42095},{"type":"v1","addr":"172.21.15.77:6805","nonce":42095}]},"public_addr":"172.21.15.77:6801/42095","cluster_addr":"172.21.15.77:6803/42095","heartbeat_back_addr":"172.21.15.77:6807/42095","heartbeat_front_addr":"172.21.15.77:6805/42095","state":["exists","up"]},{"osd":5,"uuid":"ea8d9836-29b2-4a44-8510-0373c8be8878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":42096},{"type":"v1","addr":"172.21.15.77:6812","nonce":42096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":42096},{"type":"v1","addr":"172.21.15.77:6816","nonce":42096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":42096},{"type":"v1","addr":"172.21.15.77:6823","nonce":42096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":42096},{"type":"v1","addr":"172.21.15.77:6820","nonce":42096}]},"public_addr":"172.21.15.77:6812/42096","cluster_addr":"172.21.15.77:6816/42096","heartbeat_back_addr":"172.21.15.77:6823/42096","heartbeat_front_addr":"172.21.15.77:6820/42096","state":["exists","up"]},{"osd":6,"uuid":"9f418630-4faf-41b2-9a23-ac50e6177719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":42097},{"type":"v1","addr":"172.21.15.77:6809","nonce":42097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":42097},{"type":"v1","addr":"172.21.15.77:6813","nonce":42097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6819","nonce":42097},{"type":"v1","addr":"172.21.15.77:6821","nonce":42097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6815","nonce":42097},{"type":"v1","addr":"172.21.15.77:6817","nonce":42097}]},"public_addr":"172.21.15.77:6809/42097","cluster_addr":"172.21.15.77:6813/42097","heartbeat_back_addr":"172.21.15.77:6821/42097","heartbeat_front_addr":"172.21.15.77:6817/42097","state":["exists","up"]},{"osd":7,"uuid":"90fddfc3-73d2-4217-a305-73658fd056bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":42140},{"type":"v1","addr":"172.21.15.77:6825","nonce":42140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":42140},{"type":"v1","addr":"172.21.15.77:6827","nonce":42140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":42140},{"type":"v1","addr":"172.21.15.77:6831","nonce":42140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":42140},{"type":"v1","addr":"172.21.15.77:6829","nonce":42140}]},"public_addr":"172.21.15.77:6825/42140","cluster_addr":"172.21.15.77:6827/42140","heartbeat_back_addr":"172.21.15.77:6831/42140","heartbeat_front_addr":"172.21.15.77:6829/42140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.071062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.161967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.141651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.211342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.077866+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.200851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.126483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-11T08:10:14.201914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/2199022814":"2024-01-12T09:47:50.543987+0000","172.21.15.77:0/886501039":"2024-01-12T09:47:50.543987+0000","172.21.15.77:0/3554846888":"2024-01-12T09:47:50.543987+0000","172.21.15.77:0/1909773939":"2024-01-12T09:47:50.543987+0000","172.21.15.77:0/1544835440":"2024-01-12T09:47:50.543987+0000","172.21.15.77:0/2302937394":"2024-01-12T09:47:50.543987+0000","172.21.15.27:6835/1065634442":"2024-01-12T09:47:49.956028+0000","172.21.15.77:0/151121862":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/1427717667":"2024-01-12T09:47:50.543987+0000","172.21.15.77:0/3962343616":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/710006972":"2024-01-12T09:47:40.221791+0000","172.21.15.27:6834/1065634442":"2024-01-12T09:47:49.956028+0000","172.21.15.77:0/2450284170":"2024-01-12T09:47:40.221791+0000","172.21.15.77:6837/59776":"2024-01-12T09:47:40.221791+0000","172.21.15.77:6836/59776":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/447393830":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/1295621369":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/1441504549":"2024-01-12T09:47:40.221791+0000","172.21.15.77:0/1838517028":"2024-01-12T09:47:40.221791+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T09:47:59.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.584+0000 7f3755fff700 1 -- 172.21.15.27:0/1862475853 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f373c066f30 msgr2=0x7f373c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.584+0000 7f3755fff700 1 --2- 172.21.15.27:0/1862475853 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f373c066f30 0x7f373c0693e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3740005c20 tx=0x7f3740001120 comp rx=0 tx=0).stop 2024-01-11T09:47:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.584+0000 7f3755fff700 1 -- 172.21.15.27:0/1862475853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37500b07b0 msgr2=0x7f375015c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:47:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.584+0000 7f3755fff700 1 --2- 172.21.15.27:0/1862475853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37500b07b0 0x7f375015c230 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f37440093f0 tx=0x7f3744016000 comp rx=0 tx=0).stop 2024-01-11T09:47:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.584+0000 7f3755fff700 1 -- 172.21.15.27:0/1862475853 shutdown_connections 2024-01-11T09:47:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.584+0000 7f3755fff700 1 --2- 172.21.15.27:0/1862475853 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f37500b2b90 0x7f375015c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.584+0000 7f3755fff700 1 --2- 172.21.15.27:0/1862475853 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f373c066f30 0x7f373c0693e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.584+0000 7f3755fff700 1 --2- 172.21.15.27:0/1862475853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37500b03e0 0x7f37501476a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.584+0000 7f3755fff700 1 --2- 172.21.15.27:0/1862475853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37500b07b0 0x7f375015c230 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:47:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.584+0000 7f3755fff700 1 -- 172.21.15.27:0/1862475853 >> 172.21.15.27:0/1862475853 conn(0x7f375006ed40 msgr2=0x7f37500d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:47:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.584+0000 7f3755fff700 1 -- 172.21.15.27:0/1862475853 shutdown_connections 2024-01-11T09:47:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:47:59.584+0000 7f3755fff700 1 -- 172.21.15.27:0/1862475853 wait complete. 2024-01-11T09:48:00.596 INFO:tasks.ceph:Scrubbing osd.0 2024-01-11T09:48:00.596 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-01-11T09:48:00.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.818+0000 7f8c5b5ff700 1 Processor -- start 2024-01-11T09:48:00.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.818+0000 7f8c5b5ff700 1 -- start start 2024-01-11T09:48:00.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.819+0000 7f8c5b5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c541596a0 0x7f8c540c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:00.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.819+0000 7f8c5b5ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8c540c5490 0x7f8c540ceb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:00.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.819+0000 7f8c5b5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c540cb900 con 0x7f8c540cf0c0 2024-01-11T09:48:00.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.819+0000 7f8c5b5ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c540cf4e0 con 0x7f8c540c5490 2024-01-11T09:48:00.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.819+0000 7f8c5b5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c540c6bd0 con 0x7f8c541596a0 2024-01-11T09:48:00.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.820+0000 7f8c5939b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c541596a0 0x7f8c540c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:00.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.820+0000 7f8c5939b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c541596a0 0x7f8c540c62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:46912/0 (socket says 172.21.15.27:46912) 2024-01-11T09:48:00.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.820+0000 7f8c5939b700 1 -- 172.21.15.27:0/3434832017 learned_addr learned my addr 172.21.15.27:0/3434832017 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:00.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.820+0000 7f8c58b9a700 1 --2- 172.21.15.27:0/3434832017 >> v2:172.21.15.77:3300/0 conn(0x7f8c540c5490 0x7f8c540ceb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:00.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.820+0000 7f8c4bfff700 1 -- 172.21.15.27:0/3434832017 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327281122 0 0) 0x7f8c540cb900 con 0x7f8c540cf0c0 2024-01-11T09:48:00.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.820+0000 7f8c4bfff700 1 -- 172.21.15.27:0/3434832017 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c40003570 con 0x7f8c540cf0c0 2024-01-11T09:48:00.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.820+0000 7f8c5939b700 1 -- 172.21.15.27:0/3434832017 >> v2:172.21.15.77:3300/0 conn(0x7f8c540c5490 msgr2=0x7f8c540ceb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:00.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.820+0000 7f8c5939b700 1 --2- 172.21.15.27:0/3434832017 >> v2:172.21.15.77:3300/0 conn(0x7f8c540c5490 0x7f8c540ceb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:00.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.820+0000 7f8c5939b700 1 -- 172.21.15.27:0/3434832017 >> v1:172.21.15.77:6789/0 conn(0x7f8c540cf0c0 legacy=0x7f8c540cbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:00.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.820+0000 7f8c5939b700 1 -- 172.21.15.27:0/3434832017 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c540c6eb0 con 0x7f8c541596a0 2024-01-11T09:48:00.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.820+0000 7f8c58b9a700 1 --2- 172.21.15.27:0/3434832017 >> v2:172.21.15.77:3300/0 conn(0x7f8c540c5490 0x7f8c540ceb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:48:00.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.821+0000 7f8c5939b700 1 --2- 172.21.15.27:0/3434832017 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c541596a0 0x7f8c540c62d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8c50009990 tx=0x7f8c50009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2140cdb1c5fdef37 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:00.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.821+0000 7f8c4bfff700 1 -- 172.21.15.27:0/3434832017 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c50005c10 con 0x7f8c541596a0 2024-01-11T09:48:00.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.821+0000 7f8c4bfff700 1 -- 172.21.15.27:0/3434832017 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c50005d70 con 0x7f8c541596a0 2024-01-11T09:48:00.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.821+0000 7f8c4bfff700 1 -- 172.21.15.27:0/3434832017 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c50006260 con 0x7f8c541596a0 2024-01-11T09:48:00.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.821+0000 7f8c5b5ff700 1 -- 172.21.15.27:0/3434832017 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c541596a0 msgr2=0x7f8c540c62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:00.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.821+0000 7f8c5b5ff700 1 --2- 172.21.15.27:0/3434832017 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c541596a0 0x7f8c540c62d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8c50009990 tx=0x7f8c50009220 comp rx=0 tx=0).stop 2024-01-11T09:48:00.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.822+0000 7f8c5b5ff700 1 -- 172.21.15.27:0/3434832017 shutdown_connections 2024-01-11T09:48:00.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.822+0000 7f8c5b5ff700 1 --2- 172.21.15.27:0/3434832017 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c541596a0 0x7f8c540c62d0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:00.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.822+0000 7f8c5b5ff700 1 --2- 172.21.15.27:0/3434832017 >> v2:172.21.15.77:3300/0 conn(0x7f8c540c5490 0x7f8c540ceb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:00.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.822+0000 7f8c5b5ff700 1 -- 172.21.15.27:0/3434832017 >> 172.21.15.27:0/3434832017 conn(0x7f8c540998c0 msgr2=0x7f8c5406f520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:00.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.822+0000 7f8c5b5ff700 1 -- 172.21.15.27:0/3434832017 shutdown_connections 2024-01-11T09:48:00.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.822+0000 7f8c5b5ff700 1 -- 172.21.15.27:0/3434832017 wait complete. 2024-01-11T09:48:00.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.823+0000 7f8c5b5ff700 1 Processor -- start 2024-01-11T09:48:00.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.823+0000 7f8c5b5ff700 1 -- start start 2024-01-11T09:48:00.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.824+0000 7f8c5b5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c540c5490 0x7f8c541564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:00.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.824+0000 7f8c5b5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c540cf0c0 0x7f8c5415bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:00.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.824+0000 7f8c5b5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c541596a0 0x7f8c5415c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:00.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.824+0000 7f8c5b5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c54069da0 con 0x7f8c541596a0 2024-01-11T09:48:00.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.824+0000 7f8c5b5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c54168ff0 con 0x7f8c540cf0c0 2024-01-11T09:48:00.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.824+0000 7f8c5b5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5415cac0 con 0x7f8c540c5490 2024-01-11T09:48:00.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.825+0000 7f8c59b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c541596a0 0x7f8c5415c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:00.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.825+0000 7f8c5939b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c540c5490 0x7f8c541564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:00.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.825+0000 7f8c59b9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c541596a0 0x7f8c5415c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42356/0 (socket says 172.21.15.27:42356) 2024-01-11T09:48:00.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.825+0000 7f8c59b9c700 1 -- 172.21.15.27:0/1020818464 learned_addr learned my addr 172.21.15.27:0/1020818464 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:00.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.825+0000 7f8c5939b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c540c5490 0x7f8c541564b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3301/0 says I am v2:172.21.15.27:46924/0 (socket says 172.21.15.27:46924) 2024-01-11T09:48:00.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.825+0000 7f8c58b9a700 1 --2- 172.21.15.27:0/1020818464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c540cf0c0 0x7f8c5415bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:00.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.825+0000 7f8c59b9c700 1 -- 172.21.15.27:0/1020818464 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c540c5490 msgr2=0x7f8c541564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:00.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.825+0000 7f8c59b9c700 1 --2- 172.21.15.27:0/1020818464 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f8c540c5490 0x7f8c541564b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:00.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.825+0000 7f8c59b9c700 1 -- 172.21.15.27:0/1020818464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c540cf0c0 msgr2=0x7f8c5415bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:00.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.825+0000 7f8c59b9c700 1 --2- 172.21.15.27:0/1020818464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c540cf0c0 0x7f8c5415bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:00.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.825+0000 7f8c59b9c700 1 -- 172.21.15.27:0/1020818464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c44009420 con 0x7f8c541596a0 2024-01-11T09:48:00.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.826+0000 7f8c59b9c700 1 --2- 172.21.15.27:0/1020818464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c541596a0 0x7f8c5415c1b0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c00c910 tx=0x7f8c4c0027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:00.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.826+0000 7f8c497fa700 1 -- 172.21.15.27:0/1020818464 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c012630 con 0x7f8c541596a0 2024-01-11T09:48:00.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.826+0000 7f8c497fa700 1 -- 172.21.15.27:0/1020818464 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c4c0127b0 con 0x7f8c541596a0 2024-01-11T09:48:00.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.826+0000 7f8c497fa700 1 -- 172.21.15.27:0/1020818464 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c012b70 con 0x7f8c541596a0 2024-01-11T09:48:00.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.826+0000 7f8c5b5ff700 1 -- 172.21.15.27:0/1020818464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c50009480 con 0x7f8c541596a0 2024-01-11T09:48:00.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.826+0000 7f8c5b5ff700 1 -- 172.21.15.27:0/1020818464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c541694e0 con 0x7f8c541596a0 2024-01-11T09:48:00.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.827+0000 7f8c5b5ff700 1 -- 172.21.15.27:0/1020818464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8c54169370 con 0x7f8c541596a0 2024-01-11T09:48:00.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.828+0000 7f8c497fa700 1 -- 172.21.15.27:0/1020818464 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f8c4c01b400 con 0x7f8c541596a0 2024-01-11T09:48:00.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.829+0000 7f8c497fa700 1 --2- 172.21.15.27:0/1020818464 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8c38066f30 0x7f8c380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.829+0000 7f8c497fa700 1 -- 172.21.15.27:0/1020818464 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f8c4c084930 con 0x7f8c541596a0 2024-01-11T09:48:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.829+0000 7f8c497fa700 1 --2- 172.21.15.27:0/1020818464 >> [v2:172.21.15.27:6800/41829,v1:172.21.15.27:6801/41829] conn(0x7f8c3806d4c0 0x7f8c3806f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.829+0000 7f8c497fa700 1 -- 172.21.15.27:0/1020818464 --> [v2:172.21.15.27:6800/41829,v1:172.21.15.27:6801/41829] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c38070000 con 0x7f8c3806d4c0 2024-01-11T09:48:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.829+0000 7f8c497fa700 1 -- 172.21.15.27:0/1020818464 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=261) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c4c089220 con 0x7f8c541596a0 2024-01-11T09:48:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.829+0000 7f8c58b9a700 1 --2- 172.21.15.27:0/1020818464 >> [v2:172.21.15.27:6800/41829,v1:172.21.15.27:6801/41829] conn(0x7f8c3806d4c0 0x7f8c3806f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.829+0000 7f8c5939b700 1 --2- 172.21.15.27:0/1020818464 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8c38066f30 0x7f8c380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.830+0000 7f8c58b9a700 1 --2- 172.21.15.27:0/1020818464 >> [v2:172.21.15.27:6800/41829,v1:172.21.15.27:6801/41829] conn(0x7f8c3806d4c0 0x7f8c3806f930 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:00.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.830+0000 7f8c5939b700 1 --2- 172.21.15.27:0/1020818464 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8c38066f30 0x7f8c380693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8c5000a010 tx=0x7f8c50016000 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:00.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.831+0000 7f8c497fa700 1 -- 172.21.15.27:0/1020818464 <== osd.0 v2:172.21.15.27:6800/41829 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f8c38070000 con 0x7f8c3806d4c0 2024-01-11T09:48:00.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.846+0000 7f8c5b5ff700 1 -- 172.21.15.27:0/1020818464 --> [v2:172.21.15.27:6800/41829,v1:172.21.15.27:6801/41829] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8c5404c3c0 con 0x7f8c3806d4c0 2024-01-11T09:48:00.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.846+0000 7f8c497fa700 1 -- 172.21.15.27:0/1020818464 <== osd.0 v2:172.21.15.27:6800/41829 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f8c5404c3c0 con 0x7f8c3806d4c0 2024-01-11T09:48:00.838 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:48:00.838 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-11T09:48:00.839 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:48:00.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.846+0000 7f8c5b5ff700 1 -- 172.21.15.27:0/1020818464 >> [v2:172.21.15.27:6800/41829,v1:172.21.15.27:6801/41829] conn(0x7f8c3806d4c0 msgr2=0x7f8c3806f930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:00.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.846+0000 7f8c5b5ff700 1 --2- 172.21.15.27:0/1020818464 >> [v2:172.21.15.27:6800/41829,v1:172.21.15.27:6801/41829] conn(0x7f8c3806d4c0 0x7f8c3806f930 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:00.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.846+0000 7f8c5b5ff700 1 -- 172.21.15.27:0/1020818464 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8c38066f30 msgr2=0x7f8c380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:00.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.846+0000 7f8c5b5ff700 1 --2- 172.21.15.27:0/1020818464 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f8c38066f30 0x7f8c380693e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8c5000a010 tx=0x7f8c50016000 comp rx=0 tx=0).stop 2024-01-11T09:48:00.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.846+0000 7f8c5b5ff700 1 -- 172.21.15.27:0/1020818464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c541596a0 msgr2=0x7f8c5415c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:00.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.846+0000 7f8c5b5ff700 1 --2- 172.21.15.27:0/1020818464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c541596a0 0x7f8c5415c1b0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c00c910 tx=0x7f8c4c0027a0 comp rx=0 tx=0).stop 2024-01-11T09:48:00.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.846+0000 7f8c59b9c700 1 -- 172.21.15.27:0/1020818464 reap_dead start 2024-01-11T09:48:00.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.847+0000 7f8c5b5ff700 1 -- 172.21.15.27:0/1020818464 shutdown_connections 2024-01-11T09:48:00.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.847+0000 7f8c5b5ff700 1 -- 172.21.15.27:0/1020818464 >> 172.21.15.27:0/1020818464 conn(0x7f8c540998c0 msgr2=0x7f8c54158420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:00.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.847+0000 7f8c5b5ff700 1 -- 172.21.15.27:0/1020818464 shutdown_connections 2024-01-11T09:48:00.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:00.847+0000 7f8c5b5ff700 1 -- 172.21.15.27:0/1020818464 wait complete. 2024-01-11T09:48:00.856 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-01-11T09:48:01.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.066+0000 7f40f87ff700 1 Processor -- start 2024-01-11T09:48:01.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.066+0000 7f40f87ff700 1 -- start start 2024-01-11T09:48:01.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.067+0000 7f40f87ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f40f0068ba0 0x7f40f00abd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:01.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.067+0000 7f40f87ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f40f0067f90 0x7f40f0068410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:01.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.067+0000 7f40f87ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f00ac250 con 0x7f40f01596a0 2024-01-11T09:48:01.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.067+0000 7f40f87ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f0068950 con 0x7f40f0068ba0 2024-01-11T09:48:01.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.067+0000 7f40f87ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f0067b00 con 0x7f40f0067f90 2024-01-11T09:48:01.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.068+0000 7f40f5d9a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f40f0068ba0 0x7f40f00abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:01.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.068+0000 7f40f659b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f40f01596a0 0x7f40f00af8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46336/0 (socket says 172.21.15.27:46336) 2024-01-11T09:48:01.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.068+0000 7f40f659b700 1 -- 172.21.15.27:0/2163520764 learned_addr learned my addr 172.21.15.27:0/2163520764 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:48:01.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.068+0000 7f40f6d9c700 1 --2- 172.21.15.27:0/2163520764 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f40f0067f90 0x7f40f0068410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:01.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.068+0000 7f40f5d9a700 1 -- 172.21.15.27:0/2163520764 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f40f0067f90 msgr2=0x7f40f0068410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:01.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.068+0000 7f40f5d9a700 1 --2- 172.21.15.27:0/2163520764 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f40f0067f90 0x7f40f0068410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.068+0000 7f40f5d9a700 1 -- 172.21.15.27:0/2163520764 >> v1:172.21.15.27:6789/0 conn(0x7f40f01596a0 legacy=0x7f40f00af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.068+0000 7f40f5d9a700 1 -- 172.21.15.27:0/2163520764 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f006bf60 con 0x7f40f0068ba0 2024-01-11T09:48:01.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.068+0000 7f40f5599700 1 -- 172.21.15.27:0/2163520764 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540494874 0 0) 0x7f40f00ac250 con 0x7f40f01596a0 2024-01-11T09:48:01.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.068+0000 7f40f5d9a700 1 --2- 172.21.15.27:0/2163520764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40f0068ba0 0x7f40f00abd10 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f40e00098e0 tx=0x7f40e00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e3da798aa97d907 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:01.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.069+0000 7f40f5599700 1 -- 172.21.15.27:0/2163520764 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e0009e80 con 0x7f40f0068ba0 2024-01-11T09:48:01.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.069+0000 7f40f5599700 1 -- 172.21.15.27:0/2163520764 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e000b860 con 0x7f40f0068ba0 2024-01-11T09:48:01.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.069+0000 7f40f5599700 1 -- 172.21.15.27:0/2163520764 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e000bbe0 con 0x7f40f0068ba0 2024-01-11T09:48:01.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.069+0000 7f40f87ff700 1 -- 172.21.15.27:0/2163520764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40f0068ba0 msgr2=0x7f40f00abd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.069+0000 7f40f87ff700 1 --2- 172.21.15.27:0/2163520764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40f0068ba0 0x7f40f00abd10 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f40e00098e0 tx=0x7f40e00091a0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.069+0000 7f40f87ff700 1 -- 172.21.15.27:0/2163520764 shutdown_connections 2024-01-11T09:48:01.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.069+0000 7f40f87ff700 1 --2- 172.21.15.27:0/2163520764 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f40f0067f90 0x7f40f0068410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.069+0000 7f40f87ff700 1 --2- 172.21.15.27:0/2163520764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40f0068ba0 0x7f40f00abd10 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.069+0000 7f40f87ff700 1 -- 172.21.15.27:0/2163520764 >> 172.21.15.27:0/2163520764 conn(0x7f40f006a8b0 msgr2=0x7f40f00ae520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:01.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.069+0000 7f40f87ff700 1 -- 172.21.15.27:0/2163520764 shutdown_connections 2024-01-11T09:48:01.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.069+0000 7f40f87ff700 1 -- 172.21.15.27:0/2163520764 wait complete. 2024-01-11T09:48:01.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.071+0000 7f40f87ff700 1 Processor -- start 2024-01-11T09:48:01.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.072+0000 7f40f87ff700 1 -- start start 2024-01-11T09:48:01.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.073+0000 7f40f87ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f40f0067f90 0x7f40f015eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:01.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.073+0000 7f40f87ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40f0068ba0 0x7f40f00c1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:01.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.073+0000 7f40f87ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40f01596a0 0x7f40f01552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:01.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.073+0000 7f40f87ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f0169a00 con 0x7f40f0068ba0 2024-01-11T09:48:01.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.073+0000 7f40f87ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f015f440 con 0x7f40f01596a0 2024-01-11T09:48:01.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.073+0000 7f40f87ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f015f720 con 0x7f40f0067f90 2024-01-11T09:48:01.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.073+0000 7f40f5d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40f0068ba0 0x7f40f00c1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:01.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.073+0000 7f40f5d9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40f0068ba0 0x7f40f00c1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42380/0 (socket says 172.21.15.27:42380) 2024-01-11T09:48:01.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.073+0000 7f40f5d9a700 1 -- 172.21.15.27:0/959181735 learned_addr learned my addr 172.21.15.27:0/959181735 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:01.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.073+0000 7f40f659b700 1 --2- 172.21.15.27:0/959181735 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f40f0067f90 0x7f40f015eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:01.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.073+0000 7f40f6d9c700 1 --2- 172.21.15.27:0/959181735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40f01596a0 0x7f40f01552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:01.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.074+0000 7f40f5d9a700 1 -- 172.21.15.27:0/959181735 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f40f0067f90 msgr2=0x7f40f015eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.074+0000 7f40f5d9a700 1 --2- 172.21.15.27:0/959181735 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f40f0067f90 0x7f40f015eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.074+0000 7f40f5d9a700 1 -- 172.21.15.27:0/959181735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40f01596a0 msgr2=0x7f40f01552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.074+0000 7f40f5d9a700 1 --2- 172.21.15.27:0/959181735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40f01596a0 0x7f40f01552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.074+0000 7f40f5d9a700 1 -- 172.21.15.27:0/959181735 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e0009420 con 0x7f40f0068ba0 2024-01-11T09:48:01.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.074+0000 7f40f5d9a700 1 --2- 172.21.15.27:0/959181735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40f0068ba0 0x7f40f00c1a60 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f40f006d500 tx=0x7f40e0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:01.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.074+0000 7f40e67fc700 1 -- 172.21.15.27:0/959181735 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e000bbe0 con 0x7f40f0068ba0 2024-01-11T09:48:01.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.074+0000 7f40e67fc700 1 -- 172.21.15.27:0/959181735 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e0011420 con 0x7f40f0068ba0 2024-01-11T09:48:01.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.074+0000 7f40f87ff700 1 -- 172.21.15.27:0/959181735 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f015fa00 con 0x7f40f0068ba0 2024-01-11T09:48:01.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.074+0000 7f40e67fc700 1 -- 172.21.15.27:0/959181735 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e00117e0 con 0x7f40f0068ba0 2024-01-11T09:48:01.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.075+0000 7f40f87ff700 1 -- 172.21.15.27:0/959181735 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f00c1fa0 con 0x7f40f0068ba0 2024-01-11T09:48:01.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.075+0000 7f40f87ff700 1 -- 172.21.15.27:0/959181735 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f0169c20 con 0x7f40f0068ba0 2024-01-11T09:48:01.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.076+0000 7f40e67fc700 1 -- 172.21.15.27:0/959181735 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f40e0011bd0 con 0x7f40f0068ba0 2024-01-11T09:48:01.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.077+0000 7f40e67fc700 1 --2- 172.21.15.27:0/959181735 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f40d4066f30 0x7f40d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:01.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.077+0000 7f40e67fc700 1 -- 172.21.15.27:0/959181735 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f40e00858c0 con 0x7f40f0068ba0 2024-01-11T09:48:01.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.078+0000 7f40f659b700 1 --2- 172.21.15.27:0/959181735 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f40d4066f30 0x7f40d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:01.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.078+0000 7f40f659b700 1 --2- 172.21.15.27:0/959181735 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f40d4066f30 0x7f40d40693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f40ec007160 tx=0x7f40ec007f50 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:01.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.082+0000 7f40e67fc700 1 -- 172.21.15.27:0/959181735 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f40f0169c20 con 0x7f40f0068ba0 2024-01-11T09:48:01.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.283+0000 7f40f87ff700 1 -- 172.21.15.27:0/959181735 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f40f0169c20 con 0x7f40d4066f30 2024-01-11T09:48:01.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.283+0000 7f40e67fc700 1 -- 172.21.15.27:0/959181735 <== mgr.11878 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f40f0169c20 con 0x7f40d4066f30 2024-01-11T09:48:01.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.286+0000 7f40f87ff700 1 -- 172.21.15.27:0/959181735 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f40d4066f30 msgr2=0x7f40d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.286+0000 7f40f87ff700 1 --2- 172.21.15.27:0/959181735 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f40d4066f30 0x7f40d40693e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f40ec007160 tx=0x7f40ec007f50 comp rx=0 tx=0).stop 2024-01-11T09:48:01.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.286+0000 7f40f87ff700 1 -- 172.21.15.27:0/959181735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40f0068ba0 msgr2=0x7f40f00c1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.286+0000 7f40f87ff700 1 --2- 172.21.15.27:0/959181735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40f0068ba0 0x7f40f00c1a60 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f40f006d500 tx=0x7f40e0017000 comp rx=0 tx=0).stop 2024-01-11T09:48:01.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.286+0000 7f40f87ff700 1 -- 172.21.15.27:0/959181735 shutdown_connections 2024-01-11T09:48:01.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.286+0000 7f40f87ff700 1 --2- 172.21.15.27:0/959181735 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f40d4066f30 0x7f40d40693e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.286+0000 7f40f87ff700 1 --2- 172.21.15.27:0/959181735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40f01596a0 0x7f40f01552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.286+0000 7f40f87ff700 1 --2- 172.21.15.27:0/959181735 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f40f0067f90 0x7f40f015eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.286+0000 7f40f87ff700 1 --2- 172.21.15.27:0/959181735 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40f0068ba0 0x7f40f00c1a60 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.286+0000 7f40f87ff700 1 -- 172.21.15.27:0/959181735 >> 172.21.15.27:0/959181735 conn(0x7f40f006a8b0 msgr2=0x7f40f0156cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:01.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.286+0000 7f40f87ff700 1 -- 172.21.15.27:0/959181735 shutdown_connections 2024-01-11T09:48:01.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.286+0000 7f40f87ff700 1 -- 172.21.15.27:0/959181735 wait complete. 2024-01-11T09:48:01.280 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 0 to deep-scrub 2024-01-11T09:48:01.296 INFO:tasks.ceph:Scrubbing osd.1 2024-01-11T09:48:01.297 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-01-11T09:48:01.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.508+0000 7f7c69bff700 1 Processor -- start 2024-01-11T09:48:01.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.508+0000 7f7c69bff700 1 -- start start 2024-01-11T09:48:01.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.509+0000 7f7c69bff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c64159e60 0x7f7c6415aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:01.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.510+0000 7f7c69bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c6415c4b0 con 0x7f7c6415b400 2024-01-11T09:48:01.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.510+0000 7f7c69bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c6415d050 con 0x7f7c6415a7a0 2024-01-11T09:48:01.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.510+0000 7f7c69bff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c6415b8a0 con 0x7f7c64159e60 2024-01-11T09:48:01.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.510+0000 7f7c62ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c64159e60 0x7f7c6415aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:01.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.510+0000 7f7c63fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7c6415b400 0x7f7c640685e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46340/0 (socket says 172.21.15.27:46340) 2024-01-11T09:48:01.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.510+0000 7f7c63fff700 1 -- 172.21.15.27:0/2352613221 learned_addr learned my addr 172.21.15.27:0/2352613221 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:48:01.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.510+0000 7f7c62ffd700 1 -- 172.21.15.27:0/2352613221 >> v1:172.21.15.77:6789/0 conn(0x7f7c6415a7a0 legacy=0x7f7c6415c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.510+0000 7f7c62ffd700 1 -- 172.21.15.27:0/2352613221 >> v1:172.21.15.27:6789/0 conn(0x7f7c6415b400 legacy=0x7f7c640685e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.510+0000 7f7c62ffd700 1 -- 172.21.15.27:0/2352613221 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c6415bb80 con 0x7f7c64159e60 2024-01-11T09:48:01.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.511+0000 7f7c62ffd700 1 --2- 172.21.15.27:0/2352613221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c64159e60 0x7f7c6415aec0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f7c580098e0 tx=0x7f7c580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=807adee10393df5f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:01.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.511+0000 7f7c627fc700 1 -- 172.21.15.27:0/2352613221 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c58009e80 con 0x7f7c64159e60 2024-01-11T09:48:01.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.511+0000 7f7c627fc700 1 -- 172.21.15.27:0/2352613221 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5800b860 con 0x7f7c64159e60 2024-01-11T09:48:01.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.511+0000 7f7c627fc700 1 -- 172.21.15.27:0/2352613221 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5800bbe0 con 0x7f7c64159e60 2024-01-11T09:48:01.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.511+0000 7f7c69bff700 1 -- 172.21.15.27:0/2352613221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c64159e60 msgr2=0x7f7c6415aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.511+0000 7f7c69bff700 1 --2- 172.21.15.27:0/2352613221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c64159e60 0x7f7c6415aec0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f7c580098e0 tx=0x7f7c580091a0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.512+0000 7f7c69bff700 1 -- 172.21.15.27:0/2352613221 shutdown_connections 2024-01-11T09:48:01.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.512+0000 7f7c69bff700 1 --2- 172.21.15.27:0/2352613221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c64159e60 0x7f7c6415aec0 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.512+0000 7f7c69bff700 1 -- 172.21.15.27:0/2352613221 >> 172.21.15.27:0/2352613221 conn(0x7f7c6406ed40 msgr2=0x7f7c6406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:01.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.512+0000 7f7c69bff700 1 -- 172.21.15.27:0/2352613221 shutdown_connections 2024-01-11T09:48:01.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.512+0000 7f7c69bff700 1 -- 172.21.15.27:0/2352613221 wait complete. 2024-01-11T09:48:01.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.513+0000 7f7c69bff700 1 Processor -- start 2024-01-11T09:48:01.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.513+0000 7f7c69bff700 1 -- start start 2024-01-11T09:48:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.514+0000 7f7c69bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c6415a7a0 0x7f7c64169a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.514+0000 7f7c69bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c6415b400 0x7f7c64156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.514+0000 7f7c69bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c64152f30 0x7f7c641578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.514+0000 7f7c69bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c640b0930 con 0x7f7c6415a7a0 2024-01-11T09:48:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.514+0000 7f7c69bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c641533a0 con 0x7f7c6415b400 2024-01-11T09:48:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.514+0000 7f7c69bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c64153680 con 0x7f7c64152f30 2024-01-11T09:48:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.515+0000 7f7c637fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c6415a7a0 0x7f7c64169a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.515+0000 7f7c637fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c6415a7a0 0x7f7c64169a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42388/0 (socket says 172.21.15.27:42388) 2024-01-11T09:48:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.515+0000 7f7c637fe700 1 -- 172.21.15.27:0/1164669222 learned_addr learned my addr 172.21.15.27:0/1164669222 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:01.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.515+0000 7f7c62ffd700 1 --2- 172.21.15.27:0/1164669222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c6415b400 0x7f7c64156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:01.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.515+0000 7f7c637fe700 1 -- 172.21.15.27:0/1164669222 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c64152f30 msgr2=0x7f7c641578f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.515+0000 7f7c637fe700 1 --2- 172.21.15.27:0/1164669222 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f7c64152f30 0x7f7c641578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.515+0000 7f7c637fe700 1 -- 172.21.15.27:0/1164669222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c6415b400 msgr2=0x7f7c64156cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.515+0000 7f7c637fe700 1 --2- 172.21.15.27:0/1164669222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c6415b400 0x7f7c64156cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.515+0000 7f7c637fe700 1 -- 172.21.15.27:0/1164669222 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c58009420 con 0x7f7c6415a7a0 2024-01-11T09:48:01.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.515+0000 7f7c62ffd700 1 --2- 172.21.15.27:0/1164669222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c6415b400 0x7f7c64156cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:48:01.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.515+0000 7f7c637fe700 1 --2- 172.21.15.27:0/1164669222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c6415a7a0 0x7f7c64169a60 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f7c5400c590 tx=0x7f7c5400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:01.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.515+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/1164669222 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c54011590 con 0x7f7c6415a7a0 2024-01-11T09:48:01.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.516+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/1164669222 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c540116f0 con 0x7f7c6415a7a0 2024-01-11T09:48:01.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.516+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/1164669222 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c54011ab0 con 0x7f7c6415a7a0 2024-01-11T09:48:01.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.516+0000 7f7c69bff700 1 -- 172.21.15.27:0/1164669222 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c64153960 con 0x7f7c6415a7a0 2024-01-11T09:48:01.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.516+0000 7f7c69bff700 1 -- 172.21.15.27:0/1164669222 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c64169fa0 con 0x7f7c6415a7a0 2024-01-11T09:48:01.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.517+0000 7f7c69bff700 1 -- 172.21.15.27:0/1164669222 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7c6416a190 con 0x7f7c6415a7a0 2024-01-11T09:48:01.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.518+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/1164669222 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7c5401b3b0 con 0x7f7c6415a7a0 2024-01-11T09:48:01.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.519+0000 7f7c4b7fe700 1 --2- 172.21.15.27:0/1164669222 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7c50066f30 0x7f7c500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:01.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.519+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/1164669222 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f7c54083bd0 con 0x7f7c6415a7a0 2024-01-11T09:48:01.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.519+0000 7f7c4b7fe700 1 --2- 172.21.15.27:0/1164669222 >> [v2:172.21.15.27:6813/41835,v1:172.21.15.27:6816/41835] conn(0x7f7c5006d4c0 0x7f7c5006f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:01.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.519+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/1164669222 --> [v2:172.21.15.27:6813/41835,v1:172.21.15.27:6816/41835] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7c50070000 con 0x7f7c5006d4c0 2024-01-11T09:48:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.519+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/1164669222 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=261) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c54088220 con 0x7f7c6415a7a0 2024-01-11T09:48:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.519+0000 7f7c62ffd700 1 --2- 172.21.15.27:0/1164669222 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7c50066f30 0x7f7c500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.519+0000 7f7c63fff700 1 --2- 172.21.15.27:0/1164669222 >> [v2:172.21.15.27:6813/41835,v1:172.21.15.27:6816/41835] conn(0x7f7c5006d4c0 0x7f7c5006f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.520+0000 7f7c62ffd700 1 --2- 172.21.15.27:0/1164669222 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7c50066f30 0x7f7c500693e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7c640b60d0 tx=0x7f7c58017000 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.520+0000 7f7c63fff700 1 --2- 172.21.15.27:0/1164669222 >> [v2:172.21.15.27:6813/41835,v1:172.21.15.27:6816/41835] conn(0x7f7c5006d4c0 0x7f7c5006f930 crc :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.521+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/1164669222 <== osd.1 v2:172.21.15.27:6813/41835 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f7c50070000 con 0x7f7c5006d4c0 2024-01-11T09:48:01.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.535+0000 7f7c69bff700 1 -- 172.21.15.27:0/1164669222 --> [v2:172.21.15.27:6813/41835,v1:172.21.15.27:6816/41835] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7c64150700 con 0x7f7c5006d4c0 2024-01-11T09:48:01.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.536+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/1164669222 <== osd.1 v2:172.21.15.27:6813/41835 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f7c64150700 con 0x7f7c5006d4c0 2024-01-11T09:48:01.528 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:48:01.528 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-11T09:48:01.528 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:48:01.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.536+0000 7f7c69bff700 1 -- 172.21.15.27:0/1164669222 >> [v2:172.21.15.27:6813/41835,v1:172.21.15.27:6816/41835] conn(0x7f7c5006d4c0 msgr2=0x7f7c5006f930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.536+0000 7f7c69bff700 1 --2- 172.21.15.27:0/1164669222 >> [v2:172.21.15.27:6813/41835,v1:172.21.15.27:6816/41835] conn(0x7f7c5006d4c0 0x7f7c5006f930 crc :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.536+0000 7f7c69bff700 1 -- 172.21.15.27:0/1164669222 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7c50066f30 msgr2=0x7f7c500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.536+0000 7f7c69bff700 1 --2- 172.21.15.27:0/1164669222 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f7c50066f30 0x7f7c500693e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7c640b60d0 tx=0x7f7c58017000 comp rx=0 tx=0).stop 2024-01-11T09:48:01.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.536+0000 7f7c69bff700 1 -- 172.21.15.27:0/1164669222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c6415a7a0 msgr2=0x7f7c64169a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.536+0000 7f7c69bff700 1 --2- 172.21.15.27:0/1164669222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c6415a7a0 0x7f7c64169a60 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f7c5400c590 tx=0x7f7c5400c950 comp rx=0 tx=0).stop 2024-01-11T09:48:01.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.536+0000 7f7c63fff700 1 -- 172.21.15.27:0/1164669222 reap_dead start 2024-01-11T09:48:01.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.536+0000 7f7c69bff700 1 -- 172.21.15.27:0/1164669222 shutdown_connections 2024-01-11T09:48:01.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.536+0000 7f7c69bff700 1 -- 172.21.15.27:0/1164669222 >> 172.21.15.27:0/1164669222 conn(0x7f7c6406ed40 msgr2=0x7f7c64144cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:01.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.536+0000 7f7c69bff700 1 -- 172.21.15.27:0/1164669222 shutdown_connections 2024-01-11T09:48:01.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.536+0000 7f7c69bff700 1 -- 172.21.15.27:0/1164669222 wait complete. 2024-01-11T09:48:01.545 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-01-11T09:48:01.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.754+0000 7fe79fbff700 1 Processor -- start 2024-01-11T09:48:01.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.754+0000 7fe79fbff700 1 -- start start 2024-01-11T09:48:01.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.755+0000 7fe79fbff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7980b0550 0x7fe7980ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:01.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.755+0000 7fe79fbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe798156e80 0x7fe7981572c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:01.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.755+0000 7fe79fbff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7980adc80 con 0x7fe798156ab0 2024-01-11T09:48:01.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.755+0000 7fe79fbff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7980adf60 con 0x7fe7980b0550 2024-01-11T09:48:01.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.755+0000 7fe79fbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7980ae240 con 0x7fe798156e80 2024-01-11T09:48:01.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.756+0000 7fe79d19a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7980b0550 0x7fe7980ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:01.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.756+0000 7fe79d99b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe798156ab0 0x7fe7980cc9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46350/0 (socket says 172.21.15.27:46350) 2024-01-11T09:48:01.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.756+0000 7fe79d99b700 1 -- 172.21.15.27:0/569301004 learned_addr learned my addr 172.21.15.27:0/569301004 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:48:01.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.756+0000 7fe79e19c700 1 --2- 172.21.15.27:0/569301004 >> v2:172.21.15.77:3300/0 conn(0x7fe798156e80 0x7fe7981572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:01.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.756+0000 7fe79d19a700 1 -- 172.21.15.27:0/569301004 >> v2:172.21.15.77:3300/0 conn(0x7fe798156e80 msgr2=0x7fe7981572c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:01.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.756+0000 7fe79d19a700 1 --2- 172.21.15.27:0/569301004 >> v2:172.21.15.77:3300/0 conn(0x7fe798156e80 0x7fe7981572c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.756+0000 7fe79d19a700 1 -- 172.21.15.27:0/569301004 >> v1:172.21.15.27:6789/0 conn(0x7fe798156ab0 legacy=0x7fe7980cc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.756+0000 7fe79d19a700 1 -- 172.21.15.27:0/569301004 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7980ae520 con 0x7fe7980b0550 2024-01-11T09:48:01.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.756+0000 7fe79e19c700 1 --2- 172.21.15.27:0/569301004 >> v2:172.21.15.77:3300/0 conn(0x7fe798156e80 0x7fe7981572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:48:01.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.756+0000 7fe79c999700 1 -- 172.21.15.27:0/569301004 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287446811 0 0) 0x7fe7980adc80 con 0x7fe798156ab0 2024-01-11T09:48:01.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.756+0000 7fe79d19a700 1 --2- 172.21.15.27:0/569301004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7980b0550 0x7fe7980ad650 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fe7980685a0 tx=0x7fe7880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5befe443d5282c8f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.757+0000 7fe79c999700 1 -- 172.21.15.27:0/569301004 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78800b6b0 con 0x7fe7980b0550 2024-01-11T09:48:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.757+0000 7fe79c999700 1 -- 172.21.15.27:0/569301004 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe78800b810 con 0x7fe7980b0550 2024-01-11T09:48:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.757+0000 7fe79c999700 1 -- 172.21.15.27:0/569301004 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78800bbb0 con 0x7fe7980b0550 2024-01-11T09:48:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.757+0000 7fe79fbff700 1 -- 172.21.15.27:0/569301004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7980b0550 msgr2=0x7fe7980ad650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.757+0000 7fe79fbff700 1 --2- 172.21.15.27:0/569301004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7980b0550 0x7fe7980ad650 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fe7980685a0 tx=0x7fe7880091a0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.757+0000 7fe79fbff700 1 -- 172.21.15.27:0/569301004 shutdown_connections 2024-01-11T09:48:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.757+0000 7fe79fbff700 1 --2- 172.21.15.27:0/569301004 >> v2:172.21.15.77:3300/0 conn(0x7fe798156e80 0x7fe7981572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.757+0000 7fe79fbff700 1 --2- 172.21.15.27:0/569301004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7980b0550 0x7fe7980ad650 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.757+0000 7fe79fbff700 1 -- 172.21.15.27:0/569301004 >> 172.21.15.27:0/569301004 conn(0x7fe7980680b0 msgr2=0x7fe7980cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.757+0000 7fe79fbff700 1 -- 172.21.15.27:0/569301004 shutdown_connections 2024-01-11T09:48:01.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.757+0000 7fe79fbff700 1 -- 172.21.15.27:0/569301004 wait complete. 2024-01-11T09:48:01.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.759+0000 7fe79fbff700 1 Processor -- start 2024-01-11T09:48:01.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.759+0000 7fe79fbff700 1 -- start start 2024-01-11T09:48:01.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.760+0000 7fe79fbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7980b0550 0x7fe798151de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:01.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.760+0000 7fe79fbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe798156ab0 0x7fe798152b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:01.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.760+0000 7fe79fbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe798156e80 0x7fe79813e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:01.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.760+0000 7fe79fbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7981510f0 con 0x7fe798156ab0 2024-01-11T09:48:01.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.760+0000 7fe79fbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7980cba70 con 0x7fe7980b0550 2024-01-11T09:48:01.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.760+0000 7fe79fbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe798151300 con 0x7fe798156e80 2024-01-11T09:48:01.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.760+0000 7fe79d19a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe798156ab0 0x7fe798152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:01.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.760+0000 7fe79d19a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe798156ab0 0x7fe798152b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42400/0 (socket says 172.21.15.27:42400) 2024-01-11T09:48:01.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.760+0000 7fe79d19a700 1 -- 172.21.15.27:0/3860682932 learned_addr learned my addr 172.21.15.27:0/3860682932 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:01.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.760+0000 7fe79d99b700 1 --2- 172.21.15.27:0/3860682932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7980b0550 0x7fe798151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:01.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.760+0000 7fe79e19c700 1 --2- 172.21.15.27:0/3860682932 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe798156e80 0x7fe79813e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.761+0000 7fe79d19a700 1 -- 172.21.15.27:0/3860682932 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe798156e80 msgr2=0x7fe79813e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.761+0000 7fe79d19a700 1 --2- 172.21.15.27:0/3860682932 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe798156e80 0x7fe79813e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.761+0000 7fe79d19a700 1 -- 172.21.15.27:0/3860682932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7980b0550 msgr2=0x7fe798151de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.761+0000 7fe79d19a700 1 --2- 172.21.15.27:0/3860682932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7980b0550 0x7fe798151de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.761+0000 7fe79d19a700 1 -- 172.21.15.27:0/3860682932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe788009420 con 0x7fe798156ab0 2024-01-11T09:48:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.761+0000 7fe79d19a700 1 --2- 172.21.15.27:0/3860682932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe798156ab0 0x7fe798152b60 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7fe79806fe90 tx=0x7fe788017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.761+0000 7fe78dffb700 1 -- 172.21.15.27:0/3860682932 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78800bb90 con 0x7fe798156ab0 2024-01-11T09:48:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.761+0000 7fe78dffb700 1 -- 172.21.15.27:0/3860682932 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe788011420 con 0x7fe798156ab0 2024-01-11T09:48:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.761+0000 7fe79fbff700 1 -- 172.21.15.27:0/3860682932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7981515e0 con 0x7fe798156ab0 2024-01-11T09:48:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.761+0000 7fe78dffb700 1 -- 172.21.15.27:0/3860682932 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7880117e0 con 0x7fe798156ab0 2024-01-11T09:48:01.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.762+0000 7fe79fbff700 1 -- 172.21.15.27:0/3860682932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7981530a0 con 0x7fe798156ab0 2024-01-11T09:48:01.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.762+0000 7fe79fbff700 1 -- 172.21.15.27:0/3860682932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe798151b00 con 0x7fe798156ab0 2024-01-11T09:48:01.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.763+0000 7fe78dffb700 1 -- 172.21.15.27:0/3860682932 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe788011bd0 con 0x7fe798156ab0 2024-01-11T09:48:01.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.764+0000 7fe78dffb700 1 --2- 172.21.15.27:0/3860682932 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe77c067000 0x7fe77c0694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:01.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.764+0000 7fe78dffb700 1 -- 172.21.15.27:0/3860682932 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7fe788085e10 con 0x7fe798156ab0 2024-01-11T09:48:01.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.765+0000 7fe79d99b700 1 --2- 172.21.15.27:0/3860682932 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe77c067000 0x7fe77c0694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:01.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.765+0000 7fe79d99b700 1 --2- 172.21.15.27:0/3860682932 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe77c067000 0x7fe77c0694b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe794007fd0 tx=0x7fe794007d30 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:01.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.769+0000 7fe78dffb700 1 -- 172.21.15.27:0/3860682932 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe798151b00 con 0x7fe798156ab0 2024-01-11T09:48:01.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.970+0000 7fe79fbff700 1 -- 172.21.15.27:0/3860682932 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe798151b00 con 0x7fe77c067000 2024-01-11T09:48:01.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.972+0000 7fe78dffb700 1 -- 172.21.15.27:0/3860682932 <== mgr.11878 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe798151b00 con 0x7fe77c067000 2024-01-11T09:48:01.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.973+0000 7fe79fbff700 1 -- 172.21.15.27:0/3860682932 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe77c067000 msgr2=0x7fe77c0694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.973+0000 7fe79fbff700 1 --2- 172.21.15.27:0/3860682932 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe77c067000 0x7fe77c0694b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe794007fd0 tx=0x7fe794007d30 comp rx=0 tx=0).stop 2024-01-11T09:48:01.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.973+0000 7fe79fbff700 1 -- 172.21.15.27:0/3860682932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe798156ab0 msgr2=0x7fe798152b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:01.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.973+0000 7fe79fbff700 1 --2- 172.21.15.27:0/3860682932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe798156ab0 0x7fe798152b60 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7fe79806fe90 tx=0x7fe788017000 comp rx=0 tx=0).stop 2024-01-11T09:48:01.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.973+0000 7fe79fbff700 1 -- 172.21.15.27:0/3860682932 shutdown_connections 2024-01-11T09:48:01.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.973+0000 7fe79fbff700 1 --2- 172.21.15.27:0/3860682932 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe798156e80 0x7fe79813e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.973+0000 7fe79fbff700 1 --2- 172.21.15.27:0/3860682932 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe77c067000 0x7fe77c0694b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.973+0000 7fe79fbff700 1 --2- 172.21.15.27:0/3860682932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7980b0550 0x7fe798151de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.973+0000 7fe79fbff700 1 --2- 172.21.15.27:0/3860682932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe798156ab0 0x7fe798152b60 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:01.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.973+0000 7fe79fbff700 1 -- 172.21.15.27:0/3860682932 >> 172.21.15.27:0/3860682932 conn(0x7fe7980680b0 msgr2=0x7fe79814eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:01.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.974+0000 7fe79fbff700 1 -- 172.21.15.27:0/3860682932 shutdown_connections 2024-01-11T09:48:01.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:01.974+0000 7fe79fbff700 1 -- 172.21.15.27:0/3860682932 wait complete. 2024-01-11T09:48:01.967 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 1 to deep-scrub 2024-01-11T09:48:01.983 INFO:tasks.ceph:Scrubbing osd.2 2024-01-11T09:48:01.984 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-01-11T09:48:02.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.196+0000 7f54137ff700 1 Processor -- start 2024-01-11T09:48:02.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.196+0000 7f54137ff700 1 -- start start 2024-01-11T09:48:02.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.197+0000 7f54137ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f540c157440 0x7f540c0ae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.197+0000 7f54137ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f540c069d70 0x7f540c161180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.197+0000 7f54137ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540c06d420 con 0x7f540c157810 2024-01-11T09:48:02.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.197+0000 7f54137ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f540c06d700 con 0x7f540c069d70 2024-01-11T09:48:02.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.197+0000 7f54137ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540c06d9e0 con 0x7f540c157440 2024-01-11T09:48:02.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.198+0000 7f5410d9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f540c157810 0x7f540c069660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46352/0 (socket says 172.21.15.27:46352) 2024-01-11T09:48:02.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.198+0000 7f5410d9a700 1 -- 172.21.15.27:0/2775340484 learned_addr learned my addr 172.21.15.27:0/2775340484 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:48:02.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.197+0000 7f5411d9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f540c069d70 0x7f540c161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.198+0000 7f541159b700 1 --2- 172.21.15.27:0/2775340484 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f540c157440 0x7f540c0ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.198+0000 7f5403fff700 1 -- 172.21.15.27:0/2775340484 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216835494 0 0) 0x7f540c06d420 con 0x7f540c157810 2024-01-11T09:48:02.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.198+0000 7f5403fff700 1 -- 172.21.15.27:0/2775340484 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f8003570 con 0x7f540c157810 2024-01-11T09:48:02.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.198+0000 7f5411d9c700 1 -- 172.21.15.27:0/2775340484 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f540c157440 msgr2=0x7f540c0ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:02.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.198+0000 7f5411d9c700 1 --2- 172.21.15.27:0/2775340484 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f540c157440 0x7f540c0ae430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.198+0000 7f5411d9c700 1 -- 172.21.15.27:0/2775340484 >> v1:172.21.15.27:6789/0 conn(0x7f540c157810 legacy=0x7f540c069660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.198+0000 7f5411d9c700 1 -- 172.21.15.27:0/2775340484 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540c06dcc0 con 0x7f540c069d70 2024-01-11T09:48:02.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.198+0000 7f541159b700 1 --2- 172.21.15.27:0/2775340484 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f540c157440 0x7f540c0ae430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:48:02.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.198+0000 7f5411d9c700 1 --2- 172.21.15.27:0/2775340484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c069d70 0x7f540c161180 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f5404009e60 tx=0x7f5404009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9369b7edb5daa79 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:02.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.199+0000 7f5403fff700 1 -- 172.21.15.27:0/2775340484 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540400e070 con 0x7f540c069d70 2024-01-11T09:48:02.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.199+0000 7f5403fff700 1 -- 172.21.15.27:0/2775340484 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540400bb60 con 0x7f540c069d70 2024-01-11T09:48:02.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.199+0000 7f5403fff700 1 -- 172.21.15.27:0/2775340484 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5404012430 con 0x7f540c069d70 2024-01-11T09:48:02.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.199+0000 7f54137ff700 1 -- 172.21.15.27:0/2775340484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c069d70 msgr2=0x7f540c161180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.199+0000 7f54137ff700 1 --2- 172.21.15.27:0/2775340484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c069d70 0x7f540c161180 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f5404009e60 tx=0x7f5404009780 comp rx=0 tx=0).stop 2024-01-11T09:48:02.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.199+0000 7f54137ff700 1 -- 172.21.15.27:0/2775340484 shutdown_connections 2024-01-11T09:48:02.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.199+0000 7f54137ff700 1 --2- 172.21.15.27:0/2775340484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c069d70 0x7f540c161180 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.199+0000 7f54137ff700 1 --2- 172.21.15.27:0/2775340484 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f540c157440 0x7f540c0ae430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.199+0000 7f54137ff700 1 -- 172.21.15.27:0/2775340484 >> 172.21.15.27:0/2775340484 conn(0x7f540c0680e0 msgr2=0x7f540c0cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:02.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.199+0000 7f54137ff700 1 -- 172.21.15.27:0/2775340484 shutdown_connections 2024-01-11T09:48:02.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.199+0000 7f54137ff700 1 -- 172.21.15.27:0/2775340484 wait complete. 2024-01-11T09:48:02.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.201+0000 7f54137ff700 1 Processor -- start 2024-01-11T09:48:02.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.201+0000 7f54137ff700 1 -- start start 2024-01-11T09:48:02.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.202+0000 7f54137ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c069d70 0x7f540c0cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.202+0000 7f54137ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f540c157440 0x7f540c1692e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.202+0000 7f54137ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f540c157810 0x7f540c0c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.202+0000 7f54137ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540c169820 con 0x7f540c069d70 2024-01-11T09:48:02.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.202+0000 7f54137ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540c0c87c0 con 0x7f540c157440 2024-01-11T09:48:02.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.202+0000 7f54137ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540c0bfb10 con 0x7f540c157810 2024-01-11T09:48:02.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.202+0000 7f541159b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c069d70 0x7f540c0cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.202+0000 7f541159b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c069d70 0x7f540c0cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42418/0 (socket says 172.21.15.27:42418) 2024-01-11T09:48:02.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.202+0000 7f541159b700 1 -- 172.21.15.27:0/3272881702 learned_addr learned my addr 172.21.15.27:0/3272881702 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:02.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.202+0000 7f5411d9c700 1 --2- 172.21.15.27:0/3272881702 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f540c157810 0x7f540c0c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.202+0000 7f5410d9a700 1 --2- 172.21.15.27:0/3272881702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f540c157440 0x7f540c1692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.203+0000 7f541159b700 1 -- 172.21.15.27:0/3272881702 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f540c157810 msgr2=0x7f540c0c8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.203+0000 7f541159b700 1 --2- 172.21.15.27:0/3272881702 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f540c157810 0x7f540c0c8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.203+0000 7f541159b700 1 -- 172.21.15.27:0/3272881702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f540c157440 msgr2=0x7f540c1692e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.203+0000 7f541159b700 1 --2- 172.21.15.27:0/3272881702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f540c157440 0x7f540c1692e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.203+0000 7f541159b700 1 -- 172.21.15.27:0/3272881702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5404009a20 con 0x7f540c069d70 2024-01-11T09:48:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.203+0000 7f541159b700 1 --2- 172.21.15.27:0/3272881702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c069d70 0x7f540c0cc4b0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f540800e950 tx=0x7f5408008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.203+0000 7f54017fa700 1 -- 172.21.15.27:0/3272881702 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5408005a00 con 0x7f540c069d70 2024-01-11T09:48:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.203+0000 7f54017fa700 1 -- 172.21.15.27:0/3272881702 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5408005b60 con 0x7f540c069d70 2024-01-11T09:48:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.203+0000 7f54137ff700 1 -- 172.21.15.27:0/3272881702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f540c0ccc20 con 0x7f540c069d70 2024-01-11T09:48:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.203+0000 7f54017fa700 1 -- 172.21.15.27:0/3272881702 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5408006020 con 0x7f540c069d70 2024-01-11T09:48:02.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.203+0000 7f54137ff700 1 -- 172.21.15.27:0/3272881702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f540c169d10 con 0x7f540c069d70 2024-01-11T09:48:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.204+0000 7f54137ff700 1 -- 172.21.15.27:0/3272881702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f540c04c3c0 con 0x7f540c069d70 2024-01-11T09:48:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.205+0000 7f54017fa700 1 -- 172.21.15.27:0/3272881702 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f5408006410 con 0x7f540c069d70 2024-01-11T09:48:02.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.206+0000 7f54017fa700 1 --2- 172.21.15.27:0/3272881702 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f53f0066f30 0x7f53f00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.206+0000 7f54017fa700 1 -- 172.21.15.27:0/3272881702 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f5408084570 con 0x7f540c069d70 2024-01-11T09:48:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.206+0000 7f54017fa700 1 --2- 172.21.15.27:0/3272881702 >> [v2:172.21.15.27:6809/41833,v1:172.21.15.27:6811/41833] conn(0x7f53f006d4c0 0x7f53f006f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.207+0000 7f54017fa700 1 -- 172.21.15.27:0/3272881702 --> [v2:172.21.15.27:6809/41833,v1:172.21.15.27:6811/41833] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53f0070000 con 0x7f53f006d4c0 2024-01-11T09:48:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.207+0000 7f54017fa700 1 -- 172.21.15.27:0/3272881702 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=261) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5408086bb0 con 0x7f540c069d70 2024-01-11T09:48:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.207+0000 7f5410d9a700 1 --2- 172.21.15.27:0/3272881702 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f53f0066f30 0x7f53f00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.207+0000 7f5411d9c700 1 --2- 172.21.15.27:0/3272881702 >> [v2:172.21.15.27:6809/41833,v1:172.21.15.27:6811/41833] conn(0x7f53f006d4c0 0x7f53f006f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.207+0000 7f5411d9c700 1 --2- 172.21.15.27:0/3272881702 >> [v2:172.21.15.27:6809/41833,v1:172.21.15.27:6811/41833] conn(0x7f53f006d4c0 0x7f53f006f930 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.207+0000 7f5410d9a700 1 --2- 172.21.15.27:0/3272881702 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f53f0066f30 0x7f53f00693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f540c151b00 tx=0x7f53fc009040 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:02.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.208+0000 7f54017fa700 1 -- 172.21.15.27:0/3272881702 <== osd.2 v2:172.21.15.27:6809/41833 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f53f0070000 con 0x7f53f006d4c0 2024-01-11T09:48:02.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.223+0000 7f54137ff700 1 -- 172.21.15.27:0/3272881702 --> [v2:172.21.15.27:6809/41833,v1:172.21.15.27:6811/41833] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f540c06dfa0 con 0x7f53f006d4c0 2024-01-11T09:48:02.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.223+0000 7f54017fa700 1 -- 172.21.15.27:0/3272881702 <== osd.2 v2:172.21.15.27:6809/41833 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f540c06dfa0 con 0x7f53f006d4c0 2024-01-11T09:48:02.216 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:48:02.216 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-11T09:48:02.216 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:48:02.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.223+0000 7f54137ff700 1 -- 172.21.15.27:0/3272881702 >> [v2:172.21.15.27:6809/41833,v1:172.21.15.27:6811/41833] conn(0x7f53f006d4c0 msgr2=0x7f53f006f930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.223+0000 7f54137ff700 1 --2- 172.21.15.27:0/3272881702 >> [v2:172.21.15.27:6809/41833,v1:172.21.15.27:6811/41833] conn(0x7f53f006d4c0 0x7f53f006f930 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.224+0000 7f54137ff700 1 -- 172.21.15.27:0/3272881702 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f53f0066f30 msgr2=0x7f53f00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.224+0000 7f54137ff700 1 --2- 172.21.15.27:0/3272881702 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f53f0066f30 0x7f53f00693e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f540c151b00 tx=0x7f53fc009040 comp rx=0 tx=0).stop 2024-01-11T09:48:02.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.224+0000 7f54137ff700 1 -- 172.21.15.27:0/3272881702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c069d70 msgr2=0x7f540c0cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.224+0000 7f54137ff700 1 --2- 172.21.15.27:0/3272881702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c069d70 0x7f540c0cc4b0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f540800e950 tx=0x7f5408008040 comp rx=0 tx=0).stop 2024-01-11T09:48:02.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.224+0000 7f5411d9c700 1 -- 172.21.15.27:0/3272881702 reap_dead start 2024-01-11T09:48:02.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.224+0000 7f54137ff700 1 -- 172.21.15.27:0/3272881702 shutdown_connections 2024-01-11T09:48:02.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.224+0000 7f54137ff700 1 -- 172.21.15.27:0/3272881702 >> 172.21.15.27:0/3272881702 conn(0x7f540c0680e0 msgr2=0x7f540c152d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:02.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.224+0000 7f54137ff700 1 -- 172.21.15.27:0/3272881702 shutdown_connections 2024-01-11T09:48:02.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.224+0000 7f54137ff700 1 -- 172.21.15.27:0/3272881702 wait complete. 2024-01-11T09:48:02.233 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-01-11T09:48:02.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.443+0000 7fd057fff700 1 Processor -- start 2024-01-11T09:48:02.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.443+0000 7fd057fff700 1 -- start start 2024-01-11T09:48:02.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.444+0000 7fd057fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd050155a60 0x7fd050155e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.444+0000 7fd057fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd0500b0360 0x7fd0500ac3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.444+0000 7fd057fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0500aeb50 con 0x7fd0500aef80 2024-01-11T09:48:02.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.444+0000 7fd057fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0500abf20 con 0x7fd0500b0360 2024-01-11T09:48:02.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.444+0000 7fd057fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0500ae590 con 0x7fd050155a60 2024-01-11T09:48:02.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.445+0000 7fd055d9b700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd050155a60 0x7fd050155e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.445+0000 7fd05559a700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd0500aef80 0x7fd0500afc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:57052/0 (socket says 172.21.15.27:57052) 2024-01-11T09:48:02.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.445+0000 7fd05559a700 1 -- 172.21.15.27:0/3998506827 learned_addr learned my addr 172.21.15.27:0/3998506827 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:48:02.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.445+0000 7fd05659c700 1 --2- 172.21.15.27:0/3998506827 >> v2:172.21.15.77:3300/0 conn(0x7fd0500b0360 0x7fd0500ac3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.446+0000 7fd05659c700 1 -- 172.21.15.27:0/3998506827 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd050155a60 msgr2=0x7fd050155e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:02.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.446+0000 7fd05659c700 1 --2- 172.21.15.27:0/3998506827 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd050155a60 0x7fd050155e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.446+0000 7fd05659c700 1 -- 172.21.15.27:0/3998506827 >> v1:172.21.15.77:6789/0 conn(0x7fd0500aef80 legacy=0x7fd0500afc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.446+0000 7fd05659c700 1 -- 172.21.15.27:0/3998506827 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0500ae870 con 0x7fd0500b0360 2024-01-11T09:48:02.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.446+0000 7fd055d9b700 1 --2- 172.21.15.27:0/3998506827 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd050155a60 0x7fd050155e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:48:02.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.446+0000 7fd05659c700 1 --2- 172.21.15.27:0/3998506827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0500b0360 0x7fd0500ac3b0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fd048009e60 tx=0x7fd048009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de930ed5171cff03 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:02.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.446+0000 7fd054d99700 1 -- 172.21.15.27:0/3998506827 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04800bb60 con 0x7fd0500b0360 2024-01-11T09:48:02.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.446+0000 7fd054d99700 1 -- 172.21.15.27:0/3998506827 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd04800bcc0 con 0x7fd0500b0360 2024-01-11T09:48:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.446+0000 7fd054d99700 1 -- 172.21.15.27:0/3998506827 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0480116b0 con 0x7fd0500b0360 2024-01-11T09:48:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.447+0000 7fd057fff700 1 -- 172.21.15.27:0/3998506827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0500b0360 msgr2=0x7fd0500ac3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.447+0000 7fd057fff700 1 --2- 172.21.15.27:0/3998506827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0500b0360 0x7fd0500ac3b0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fd048009e60 tx=0x7fd048009780 comp rx=0 tx=0).stop 2024-01-11T09:48:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.447+0000 7fd057fff700 1 -- 172.21.15.27:0/3998506827 shutdown_connections 2024-01-11T09:48:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.447+0000 7fd057fff700 1 --2- 172.21.15.27:0/3998506827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0500b0360 0x7fd0500ac3b0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.447+0000 7fd057fff700 1 --2- 172.21.15.27:0/3998506827 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd050155a60 0x7fd050155e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.447+0000 7fd057fff700 1 -- 172.21.15.27:0/3998506827 >> 172.21.15.27:0/3998506827 conn(0x7fd0500680b0 msgr2=0x7fd0500cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.447+0000 7fd057fff700 1 -- 172.21.15.27:0/3998506827 shutdown_connections 2024-01-11T09:48:02.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.447+0000 7fd057fff700 1 -- 172.21.15.27:0/3998506827 wait complete. 2024-01-11T09:48:02.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.448+0000 7fd057fff700 1 Processor -- start 2024-01-11T09:48:02.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.449+0000 7fd057fff700 1 -- start start 2024-01-11T09:48:02.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.449+0000 7fd057fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0500aef80 0x7fd050157ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.449+0000 7fd057fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0500b0360 0x7fd050156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.450+0000 7fd057fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd050155a60 0x7fd0501698a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.450+0000 7fd057fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0501563a0 con 0x7fd050155a60 2024-01-11T09:48:02.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.450+0000 7fd057fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd050156960 con 0x7fd0500aef80 2024-01-11T09:48:02.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.450+0000 7fd057fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05016a3f0 con 0x7fd0500b0360 2024-01-11T09:48:02.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.450+0000 7fd055d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0500aef80 0x7fd050157ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.450+0000 7fd05659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd050155a60 0x7fd0501698a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.450+0000 7fd055d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0500aef80 0x7fd050157ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.27:52922/0 (socket says 172.21.15.27:52922) 2024-01-11T09:48:02.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.450+0000 7fd055d9b700 1 -- 172.21.15.27:0/102132223 learned_addr learned my addr 172.21.15.27:0/102132223 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:02.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.450+0000 7fd05559a700 1 --2- 172.21.15.27:0/102132223 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0500b0360 0x7fd050156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.450+0000 7fd05659c700 1 -- 172.21.15.27:0/102132223 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0500b0360 msgr2=0x7fd050156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.450+0000 7fd05659c700 1 --2- 172.21.15.27:0/102132223 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0500b0360 0x7fd050156df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.450+0000 7fd05659c700 1 -- 172.21.15.27:0/102132223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0500aef80 msgr2=0x7fd050157ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.450+0000 7fd05659c700 1 --2- 172.21.15.27:0/102132223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0500aef80 0x7fd050157ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.450+0000 7fd05659c700 1 -- 172.21.15.27:0/102132223 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c009420 con 0x7fd050155a60 2024-01-11T09:48:02.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.451+0000 7fd05659c700 1 --2- 172.21.15.27:0/102132223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd050155a60 0x7fd0501698a0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fd05006c9b0 tx=0x7fd04801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:02.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.451+0000 7fd045ffb700 1 -- 172.21.15.27:0/102132223 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd048019610 con 0x7fd050155a60 2024-01-11T09:48:02.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.451+0000 7fd045ffb700 1 -- 172.21.15.27:0/102132223 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd048019790 con 0x7fd050155a60 2024-01-11T09:48:02.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.451+0000 7fd045ffb700 1 -- 172.21.15.27:0/102132223 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd048019b50 con 0x7fd050155a60 2024-01-11T09:48:02.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.451+0000 7fd057fff700 1 -- 172.21.15.27:0/102132223 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd048009a20 con 0x7fd050155a60 2024-01-11T09:48:02.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.451+0000 7fd057fff700 1 -- 172.21.15.27:0/102132223 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0501565f0 con 0x7fd050155a60 2024-01-11T09:48:02.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.452+0000 7fd057fff700 1 -- 172.21.15.27:0/102132223 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05004c3c0 con 0x7fd050155a60 2024-01-11T09:48:02.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.453+0000 7fd045ffb700 1 -- 172.21.15.27:0/102132223 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd048003040 con 0x7fd050155a60 2024-01-11T09:48:02.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.454+0000 7fd045ffb700 1 --2- 172.21.15.27:0/102132223 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd034066f30 0x7fd0340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.454+0000 7fd045ffb700 1 -- 172.21.15.27:0/102132223 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7fd048085f00 con 0x7fd050155a60 2024-01-11T09:48:02.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.454+0000 7fd055d9b700 1 --2- 172.21.15.27:0/102132223 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd034066f30 0x7fd0340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.455+0000 7fd055d9b700 1 --2- 172.21.15.27:0/102132223 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd034066f30 0x7fd0340693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd04c009fa0 tx=0x7fd04c0091a0 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:02.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.458+0000 7fd045ffb700 1 -- 172.21.15.27:0/102132223 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd0480531d0 con 0x7fd050155a60 2024-01-11T09:48:02.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.660+0000 7fd057fff700 1 -- 172.21.15.27:0/102132223 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0500abbb0 con 0x7fd034066f30 2024-01-11T09:48:02.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.660+0000 7fd045ffb700 1 -- 172.21.15.27:0/102132223 <== mgr.11878 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd0500abbb0 con 0x7fd034066f30 2024-01-11T09:48:02.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.663+0000 7fd057fff700 1 -- 172.21.15.27:0/102132223 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd034066f30 msgr2=0x7fd0340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.663+0000 7fd057fff700 1 --2- 172.21.15.27:0/102132223 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd034066f30 0x7fd0340693e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd04c009fa0 tx=0x7fd04c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.663+0000 7fd057fff700 1 -- 172.21.15.27:0/102132223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd050155a60 msgr2=0x7fd0501698a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.663+0000 7fd057fff700 1 --2- 172.21.15.27:0/102132223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd050155a60 0x7fd0501698a0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fd05006c9b0 tx=0x7fd04801b000 comp rx=0 tx=0).stop 2024-01-11T09:48:02.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.663+0000 7fd057fff700 1 -- 172.21.15.27:0/102132223 shutdown_connections 2024-01-11T09:48:02.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.663+0000 7fd057fff700 1 --2- 172.21.15.27:0/102132223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd050155a60 0x7fd0501698a0 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.663+0000 7fd057fff700 1 --2- 172.21.15.27:0/102132223 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd034066f30 0x7fd0340693e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.663+0000 7fd057fff700 1 --2- 172.21.15.27:0/102132223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0500aef80 0x7fd050157ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.663+0000 7fd057fff700 1 --2- 172.21.15.27:0/102132223 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd0500b0360 0x7fd050156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.663+0000 7fd057fff700 1 -- 172.21.15.27:0/102132223 >> 172.21.15.27:0/102132223 conn(0x7fd0500680b0 msgr2=0x7fd0501542c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:02.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.663+0000 7fd057fff700 1 -- 172.21.15.27:0/102132223 shutdown_connections 2024-01-11T09:48:02.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.663+0000 7fd057fff700 1 -- 172.21.15.27:0/102132223 wait complete. 2024-01-11T09:48:02.657 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 2 to deep-scrub 2024-01-11T09:48:02.673 INFO:tasks.ceph:Scrubbing osd.3 2024-01-11T09:48:02.673 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-01-11T09:48:02.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.885+0000 7f38ab3ff700 1 Processor -- start 2024-01-11T09:48:02.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.885+0000 7f38ab3ff700 1 -- start start 2024-01-11T09:48:02.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.886+0000 7f38ab3ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f38a4159e60 0x7f38a415aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.886+0000 7f38ab3ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f38a415b400 0x7f38a40685e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.886+0000 7f38ab3ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a415c4b0 con 0x7f38a415a7a0 2024-01-11T09:48:02.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.886+0000 7f38ab3ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a415d050 con 0x7f38a415b400 2024-01-11T09:48:02.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.887+0000 7f38ab3ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a415b8a0 con 0x7f38a4159e60 2024-01-11T09:48:02.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.887+0000 7f38a999c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f38a415b400 0x7f38a40685e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.887+0000 7f38a919b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f38a415a7a0 0x7f38a415c940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.27:57062/0 (socket says 172.21.15.27:57062) 2024-01-11T09:48:02.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.887+0000 7f38a919b700 1 -- 172.21.15.27:0/3675927375 learned_addr learned my addr 172.21.15.27:0/3675927375 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:48:02.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.887+0000 7f38a899a700 1 --2- 172.21.15.27:0/3675927375 >> v2:172.21.15.77:3300/0 conn(0x7f38a4159e60 0x7f38a415aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.887+0000 7f38a999c700 1 -- 172.21.15.27:0/3675927375 >> v2:172.21.15.77:3300/0 conn(0x7f38a4159e60 msgr2=0x7f38a415aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:02.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.887+0000 7f38a999c700 1 --2- 172.21.15.27:0/3675927375 >> v2:172.21.15.77:3300/0 conn(0x7f38a4159e60 0x7f38a415aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.887+0000 7f38a999c700 1 -- 172.21.15.27:0/3675927375 >> v1:172.21.15.77:6789/0 conn(0x7f38a415a7a0 legacy=0x7f38a415c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.887+0000 7f389bfff700 1 -- 172.21.15.27:0/3675927375 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498762445 0 0) 0x7f38a415c4b0 con 0x7f38a415a7a0 2024-01-11T09:48:02.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.887+0000 7f38a999c700 1 -- 172.21.15.27:0/3675927375 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a415bb80 con 0x7f38a415b400 2024-01-11T09:48:02.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.888+0000 7f38a999c700 1 --2- 172.21.15.27:0/3675927375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38a415b400 0x7f38a40685e0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f38a40ce530 tx=0x7f389c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3234b2957e39d server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:02.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.888+0000 7f389bfff700 1 -- 172.21.15.27:0/3675927375 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389c00e070 con 0x7f38a415b400 2024-01-11T09:48:02.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.888+0000 7f389bfff700 1 -- 172.21.15.27:0/3675927375 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f389c00bb60 con 0x7f38a415b400 2024-01-11T09:48:02.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.888+0000 7f389bfff700 1 -- 172.21.15.27:0/3675927375 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389c012430 con 0x7f38a415b400 2024-01-11T09:48:02.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.888+0000 7f38ab3ff700 1 -- 172.21.15.27:0/3675927375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38a415b400 msgr2=0x7f38a40685e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.888+0000 7f38ab3ff700 1 --2- 172.21.15.27:0/3675927375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38a415b400 0x7f38a40685e0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f38a40ce530 tx=0x7f389c009780 comp rx=0 tx=0).stop 2024-01-11T09:48:02.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.889+0000 7f38ab3ff700 1 -- 172.21.15.27:0/3675927375 shutdown_connections 2024-01-11T09:48:02.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.889+0000 7f38ab3ff700 1 --2- 172.21.15.27:0/3675927375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38a415b400 0x7f38a40685e0 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.889+0000 7f38ab3ff700 1 --2- 172.21.15.27:0/3675927375 >> v2:172.21.15.77:3300/0 conn(0x7f38a4159e60 0x7f38a415aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.889+0000 7f38ab3ff700 1 -- 172.21.15.27:0/3675927375 >> 172.21.15.27:0/3675927375 conn(0x7f38a406ed40 msgr2=0x7f38a406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:02.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.889+0000 7f38ab3ff700 1 -- 172.21.15.27:0/3675927375 shutdown_connections 2024-01-11T09:48:02.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.889+0000 7f38ab3ff700 1 -- 172.21.15.27:0/3675927375 wait complete. 2024-01-11T09:48:02.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.891+0000 7f38ab3ff700 1 Processor -- start 2024-01-11T09:48:02.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.891+0000 7f38ab3ff700 1 -- start start 2024-01-11T09:48:02.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.892+0000 7f38ab3ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f38a415a7a0 0x7f38a40b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.892+0000 7f38ab3ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38a415b400 0x7f38a413bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.892+0000 7f38ab3ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38a415fb30 0x7f38a4151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.892+0000 7f38ab3ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a40b2e00 con 0x7f38a415b400 2024-01-11T09:48:02.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.892+0000 7f38ab3ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a415f6a0 con 0x7f38a415fb30 2024-01-11T09:48:02.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.892+0000 7f38ab3ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a40b1380 con 0x7f38a415a7a0 2024-01-11T09:48:02.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.893+0000 7f38a899a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38a415b400 0x7f38a413bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.893+0000 7f38a899a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38a415b400 0x7f38a413bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42458/0 (socket says 172.21.15.27:42458) 2024-01-11T09:48:02.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.893+0000 7f38a899a700 1 -- 172.21.15.27:0/468877066 learned_addr learned my addr 172.21.15.27:0/468877066 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:02.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.893+0000 7f38a919b700 1 --2- 172.21.15.27:0/468877066 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f38a415a7a0 0x7f38a40b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.893+0000 7f38a999c700 1 --2- 172.21.15.27:0/468877066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38a415fb30 0x7f38a4151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.893+0000 7f38a899a700 1 -- 172.21.15.27:0/468877066 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f38a415a7a0 msgr2=0x7f38a40b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.893+0000 7f38a899a700 1 --2- 172.21.15.27:0/468877066 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f38a415a7a0 0x7f38a40b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.893+0000 7f38a899a700 1 -- 172.21.15.27:0/468877066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38a415fb30 msgr2=0x7f38a4151720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.893+0000 7f38a899a700 1 --2- 172.21.15.27:0/468877066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38a415fb30 0x7f38a4151720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.893+0000 7f38a899a700 1 -- 172.21.15.27:0/468877066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389c009a20 con 0x7f38a415b400 2024-01-11T09:48:02.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.893+0000 7f38a899a700 1 --2- 172.21.15.27:0/468877066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38a415b400 0x7f38a413bf70 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f389400e930 tx=0x7f38940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:02.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.894+0000 7f38997fa700 1 -- 172.21.15.27:0/468877066 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3894013680 con 0x7f38a415b400 2024-01-11T09:48:02.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.894+0000 7f38997fa700 1 -- 172.21.15.27:0/468877066 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38940137e0 con 0x7f38a415b400 2024-01-11T09:48:02.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.894+0000 7f38997fa700 1 -- 172.21.15.27:0/468877066 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3894013b80 con 0x7f38a415b400 2024-01-11T09:48:02.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.894+0000 7f38ab3ff700 1 -- 172.21.15.27:0/468877066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a40b2840 con 0x7f38a415b400 2024-01-11T09:48:02.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.894+0000 7f38ab3ff700 1 -- 172.21.15.27:0/468877066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a415f330 con 0x7f38a415b400 2024-01-11T09:48:02.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.895+0000 7f38ab3ff700 1 -- 172.21.15.27:0/468877066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f38a404c3c0 con 0x7f38a415b400 2024-01-11T09:48:02.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.896+0000 7f38997fa700 1 -- 172.21.15.27:0/468877066 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f389400c3e0 con 0x7f38a415b400 2024-01-11T09:48:02.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.897+0000 7f38997fa700 1 --2- 172.21.15.27:0/468877066 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3888066f30 0x7f38880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.897+0000 7f38997fa700 1 -- 172.21.15.27:0/468877066 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f38940846e0 con 0x7f38a415b400 2024-01-11T09:48:02.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.897+0000 7f38997fa700 1 --2- 172.21.15.27:0/468877066 >> [v2:172.21.15.27:6808/41831,v1:172.21.15.27:6810/41831] conn(0x7f388806d4c0 0x7f388806f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:02.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.897+0000 7f38997fa700 1 -- 172.21.15.27:0/468877066 --> [v2:172.21.15.27:6808/41831,v1:172.21.15.27:6810/41831] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3888070000 con 0x7f388806d4c0 2024-01-11T09:48:02.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.897+0000 7f38997fa700 1 -- 172.21.15.27:0/468877066 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=261) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3894086be0 con 0x7f38a415b400 2024-01-11T09:48:02.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.897+0000 7f38a919b700 1 --2- 172.21.15.27:0/468877066 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3888066f30 0x7f38880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.897+0000 7f38a999c700 1 --2- 172.21.15.27:0/468877066 >> [v2:172.21.15.27:6808/41831,v1:172.21.15.27:6810/41831] conn(0x7f388806d4c0 0x7f388806f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:02.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.898+0000 7f38a999c700 1 --2- 172.21.15.27:0/468877066 >> [v2:172.21.15.27:6808/41831,v1:172.21.15.27:6810/41831] conn(0x7f388806d4c0 0x7f388806f930 crc :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:02.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.898+0000 7f38a919b700 1 --2- 172.21.15.27:0/468877066 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3888066f30 0x7f38880693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f38a41604d0 tx=0x7f38a0007f50 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:02.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.899+0000 7f38997fa700 1 -- 172.21.15.27:0/468877066 <== osd.3 v2:172.21.15.27:6808/41831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f3888070000 con 0x7f388806d4c0 2024-01-11T09:48:02.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.913+0000 7f38ab3ff700 1 -- 172.21.15.27:0/468877066 --> [v2:172.21.15.27:6808/41831,v1:172.21.15.27:6810/41831] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f38a415be60 con 0x7f388806d4c0 2024-01-11T09:48:02.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.914+0000 7f38997fa700 1 -- 172.21.15.27:0/468877066 <== osd.3 v2:172.21.15.27:6808/41831 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f38a415be60 con 0x7f388806d4c0 2024-01-11T09:48:02.906 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:48:02.906 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-11T09:48:02.906 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:48:02.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.914+0000 7f38ab3ff700 1 -- 172.21.15.27:0/468877066 >> [v2:172.21.15.27:6808/41831,v1:172.21.15.27:6810/41831] conn(0x7f388806d4c0 msgr2=0x7f388806f930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.914+0000 7f38ab3ff700 1 --2- 172.21.15.27:0/468877066 >> [v2:172.21.15.27:6808/41831,v1:172.21.15.27:6810/41831] conn(0x7f388806d4c0 0x7f388806f930 crc :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.914+0000 7f38ab3ff700 1 -- 172.21.15.27:0/468877066 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3888066f30 msgr2=0x7f38880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.914+0000 7f38ab3ff700 1 --2- 172.21.15.27:0/468877066 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f3888066f30 0x7f38880693e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f38a41604d0 tx=0x7f38a0007f50 comp rx=0 tx=0).stop 2024-01-11T09:48:02.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.914+0000 7f38ab3ff700 1 -- 172.21.15.27:0/468877066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38a415b400 msgr2=0x7f38a413bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:02.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.914+0000 7f38ab3ff700 1 --2- 172.21.15.27:0/468877066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38a415b400 0x7f38a413bf70 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f389400e930 tx=0x7f38940091a0 comp rx=0 tx=0).stop 2024-01-11T09:48:02.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.914+0000 7f38a999c700 1 -- 172.21.15.27:0/468877066 reap_dead start 2024-01-11T09:48:02.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.914+0000 7f38ab3ff700 1 -- 172.21.15.27:0/468877066 shutdown_connections 2024-01-11T09:48:02.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.914+0000 7f38ab3ff700 1 -- 172.21.15.27:0/468877066 >> 172.21.15.27:0/468877066 conn(0x7f38a406ed40 msgr2=0x7f38a40af020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:02.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.914+0000 7f38ab3ff700 1 -- 172.21.15.27:0/468877066 shutdown_connections 2024-01-11T09:48:02.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:02.915+0000 7f38ab3ff700 1 -- 172.21.15.27:0/468877066 wait complete. 2024-01-11T09:48:02.924 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-01-11T09:48:03.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.135+0000 7f003a9ff700 1 Processor -- start 2024-01-11T09:48:03.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.135+0000 7f003a9ff700 1 -- start start 2024-01-11T09:48:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.136+0000 7f003a9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f003406bf10 0x7f0034159c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.136+0000 7f003a9ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0034068720 0x7f0034068b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.136+0000 7f003a9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003415d290 con 0x7f00340b0a40 2024-01-11T09:48:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.136+0000 7f003a9ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00340690a0 con 0x7f0034068720 2024-01-11T09:48:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.137+0000 7f003a9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f003415d570 con 0x7f003406bf10 2024-01-11T09:48:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.137+0000 7f00337fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f00340b0a40 0x7f003415da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46358/0 (socket says 172.21.15.27:46358) 2024-01-11T09:48:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.137+0000 7f00337fe700 1 -- 172.21.15.27:0/2833825246 learned_addr learned my addr 172.21.15.27:0/2833825246 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:48:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.137+0000 7f0038f9c700 1 --2- 172.21.15.27:0/2833825246 >> v2:172.21.15.27:3300/0 conn(0x7f0034068720 0x7f0034068b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.137+0000 7f0033fff700 1 --2- 172.21.15.27:0/2833825246 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f003406bf10 0x7f0034159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.137+0000 7f0038f9c700 1 -- 172.21.15.27:0/2833825246 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f003406bf10 msgr2=0x7f0034159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.137+0000 7f0038f9c700 1 --2- 172.21.15.27:0/2833825246 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f003406bf10 0x7f0034159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.137+0000 7f0038f9c700 1 -- 172.21.15.27:0/2833825246 >> v1:172.21.15.27:6789/0 conn(0x7f00340b0a40 legacy=0x7f003415da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:03.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.137+0000 7f0038f9c700 1 -- 172.21.15.27:0/2833825246 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0034154180 con 0x7f0034068720 2024-01-11T09:48:03.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.138+0000 7f0038f9c700 1 --2- 172.21.15.27:0/2833825246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0034068720 0x7f0034068b60 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f002c009e60 tx=0x7f002c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60508fe110eb6514 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:03.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.138+0000 7f0032ffd700 1 -- 172.21.15.27:0/2833825246 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c00e070 con 0x7f0034068720 2024-01-11T09:48:03.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.138+0000 7f0032ffd700 1 -- 172.21.15.27:0/2833825246 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002c00bb60 con 0x7f0034068720 2024-01-11T09:48:03.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.138+0000 7f0032ffd700 1 -- 172.21.15.27:0/2833825246 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c012430 con 0x7f0034068720 2024-01-11T09:48:03.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.138+0000 7f003a9ff700 1 -- 172.21.15.27:0/2833825246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0034068720 msgr2=0x7f0034068b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:03.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.138+0000 7f003a9ff700 1 --2- 172.21.15.27:0/2833825246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0034068720 0x7f0034068b60 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f002c009e60 tx=0x7f002c009780 comp rx=0 tx=0).stop 2024-01-11T09:48:03.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.138+0000 7f003a9ff700 1 -- 172.21.15.27:0/2833825246 shutdown_connections 2024-01-11T09:48:03.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.138+0000 7f003a9ff700 1 --2- 172.21.15.27:0/2833825246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0034068720 0x7f0034068b60 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.138+0000 7f003a9ff700 1 --2- 172.21.15.27:0/2833825246 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f003406bf10 0x7f0034159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.138+0000 7f003a9ff700 1 -- 172.21.15.27:0/2833825246 >> 172.21.15.27:0/2833825246 conn(0x7f00340ce3c0 msgr2=0x7f00340ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:03.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.139+0000 7f003a9ff700 1 -- 172.21.15.27:0/2833825246 shutdown_connections 2024-01-11T09:48:03.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.139+0000 7f003a9ff700 1 -- 172.21.15.27:0/2833825246 wait complete. 2024-01-11T09:48:03.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.141+0000 7f003a9ff700 1 Processor -- start 2024-01-11T09:48:03.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.141+0000 7f003a9ff700 1 -- start start 2024-01-11T09:48:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.142+0000 7f003a9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0034068720 0x7f00340b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.142+0000 7f003a9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f003406bf10 0x7f00340b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.142+0000 7f003a9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00340b0a40 0x7f00340b3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.142+0000 7f003a9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00340c6ee0 con 0x7f00340b0a40 2024-01-11T09:48:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.142+0000 7f003a9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00340c71c0 con 0x7f003406bf10 2024-01-11T09:48:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.142+0000 7f003a9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00340c74a0 con 0x7f0034068720 2024-01-11T09:48:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.142+0000 7f0038f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00340b0a40 0x7f00340b3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.143+0000 7f0038f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00340b0a40 0x7f00340b3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42466/0 (socket says 172.21.15.27:42466) 2024-01-11T09:48:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.143+0000 7f0038f9c700 1 -- 172.21.15.27:0/1533667247 learned_addr learned my addr 172.21.15.27:0/1533667247 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.143+0000 7f0033fff700 1 --2- 172.21.15.27:0/1533667247 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0034068720 0x7f00340b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.143+0000 7f00337fe700 1 --2- 172.21.15.27:0/1533667247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f003406bf10 0x7f00340b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.143+0000 7f0038f9c700 1 -- 172.21.15.27:0/1533667247 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0034068720 msgr2=0x7f00340b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.143+0000 7f0038f9c700 1 --2- 172.21.15.27:0/1533667247 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0034068720 0x7f00340b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.143+0000 7f0038f9c700 1 -- 172.21.15.27:0/1533667247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f003406bf10 msgr2=0x7f00340b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.143+0000 7f0038f9c700 1 --2- 172.21.15.27:0/1533667247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f003406bf10 0x7f00340b2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.143+0000 7f0038f9c700 1 -- 172.21.15.27:0/1533667247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002c009a20 con 0x7f00340b0a40 2024-01-11T09:48:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.143+0000 7f0038f9c700 1 --2- 172.21.15.27:0/1533667247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00340b0a40 0x7f00340b3210 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f002c00b6c0 tx=0x7f002c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.144+0000 7f001bfff700 1 -- 172.21.15.27:0/1533667247 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c025440 con 0x7f00340b0a40 2024-01-11T09:48:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.144+0000 7f001bfff700 1 -- 172.21.15.27:0/1533667247 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002c0255c0 con 0x7f00340b0a40 2024-01-11T09:48:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.144+0000 7f001bfff700 1 -- 172.21.15.27:0/1533667247 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c025980 con 0x7f00340b0a40 2024-01-11T09:48:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.144+0000 7f003a9ff700 1 -- 172.21.15.27:0/1533667247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00340c7780 con 0x7f00340b0a40 2024-01-11T09:48:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.144+0000 7f003a9ff700 1 -- 172.21.15.27:0/1533667247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00340b3f30 con 0x7f00340b0a40 2024-01-11T09:48:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.145+0000 7f003a9ff700 1 -- 172.21.15.27:0/1533667247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00340b4120 con 0x7f00340b0a40 2024-01-11T09:48:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.146+0000 7f001bfff700 1 -- 172.21.15.27:0/1533667247 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f002c003050 con 0x7f00340b0a40 2024-01-11T09:48:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.147+0000 7f001bfff700 1 --2- 172.21.15.27:0/1533667247 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0020066f30 0x7f00200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.147+0000 7f001bfff700 1 -- 172.21.15.27:0/1533667247 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f002c086600 con 0x7f00340b0a40 2024-01-11T09:48:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.147+0000 7f0033fff700 1 --2- 172.21.15.27:0/1533667247 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0020066f30 0x7f00200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.148+0000 7f0033fff700 1 --2- 172.21.15.27:0/1533667247 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0020066f30 0x7f00200693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0024009db0 tx=0x7f00240091a0 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:03.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.151+0000 7f001bfff700 1 -- 172.21.15.27:0/1533667247 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f00340b4120 con 0x7f00340b0a40 2024-01-11T09:48:03.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.354+0000 7f003a9ff700 1 -- 172.21.15.27:0/1533667247 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f00340c7330 con 0x7f0020066f30 2024-01-11T09:48:03.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.355+0000 7f001bfff700 1 -- 172.21.15.27:0/1533667247 <== mgr.11878 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f00340c7330 con 0x7f0020066f30 2024-01-11T09:48:03.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.357+0000 7f003a9ff700 1 -- 172.21.15.27:0/1533667247 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0020066f30 msgr2=0x7f00200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:03.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.357+0000 7f003a9ff700 1 --2- 172.21.15.27:0/1533667247 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0020066f30 0x7f00200693e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0024009db0 tx=0x7f00240091a0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.357+0000 7f003a9ff700 1 -- 172.21.15.27:0/1533667247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00340b0a40 msgr2=0x7f00340b3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:03.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.357+0000 7f003a9ff700 1 --2- 172.21.15.27:0/1533667247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00340b0a40 0x7f00340b3210 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f002c00b6c0 tx=0x7f002c019000 comp rx=0 tx=0).stop 2024-01-11T09:48:03.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.357+0000 7f003a9ff700 1 -- 172.21.15.27:0/1533667247 shutdown_connections 2024-01-11T09:48:03.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.357+0000 7f003a9ff700 1 --2- 172.21.15.27:0/1533667247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00340b0a40 0x7f00340b3210 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.357+0000 7f003a9ff700 1 --2- 172.21.15.27:0/1533667247 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0034068720 0x7f00340b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.357+0000 7f003a9ff700 1 --2- 172.21.15.27:0/1533667247 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0020066f30 0x7f00200693e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.357+0000 7f003a9ff700 1 --2- 172.21.15.27:0/1533667247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f003406bf10 0x7f00340b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.357+0000 7f003a9ff700 1 -- 172.21.15.27:0/1533667247 >> 172.21.15.27:0/1533667247 conn(0x7f00340ce3c0 msgr2=0x7f00341598a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:03.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.357+0000 7f003a9ff700 1 -- 172.21.15.27:0/1533667247 shutdown_connections 2024-01-11T09:48:03.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.357+0000 7f003a9ff700 1 -- 172.21.15.27:0/1533667247 wait complete. 2024-01-11T09:48:03.350 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 3 to deep-scrub 2024-01-11T09:48:03.366 INFO:tasks.ceph:Scrubbing osd.4 2024-01-11T09:48:03.366 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-01-11T09:48:03.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.579+0000 7fa7e7dff700 1 Processor -- start 2024-01-11T09:48:03.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.579+0000 7fa7e7dff700 1 -- start start 2024-01-11T09:48:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.580+0000 7fa7e7dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa7e015a7a0 0x7fa7e015c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.580+0000 7fa7e7dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa7e00685e0 0x7fa7e0068a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.580+0000 7fa7e7dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e015c4b0 con 0x7fa7e0159e60 2024-01-11T09:48:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.580+0000 7fa7e7dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e0068f40 con 0x7fa7e00685e0 2024-01-11T09:48:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.580+0000 7fa7e7dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e015b5d0 con 0x7fa7e015a7a0 2024-01-11T09:48:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.581+0000 7fa7e539a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa7e0159e60 0x7fa7e015aec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46374/0 (socket says 172.21.15.27:46374) 2024-01-11T09:48:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.581+0000 7fa7e539a700 1 -- 172.21.15.27:0/2856028438 learned_addr learned my addr 172.21.15.27:0/2856028438 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:48:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.581+0000 7fa7e639c700 1 --2- 172.21.15.27:0/2856028438 >> v2:172.21.15.27:3300/0 conn(0x7fa7e00685e0 0x7fa7e0068a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:03.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.581+0000 7fa7e5b9b700 1 --2- 172.21.15.27:0/2856028438 >> v2:172.21.15.77:3300/0 conn(0x7fa7e015a7a0 0x7fa7e015c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:03.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.581+0000 7fa7e4b99700 1 -- 172.21.15.27:0/2856028438 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759009275 0 0) 0x7fa7e015c4b0 con 0x7fa7e0159e60 2024-01-11T09:48:03.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.581+0000 7fa7e4b99700 1 -- 172.21.15.27:0/2856028438 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7cc003570 con 0x7fa7e0159e60 2024-01-11T09:48:03.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.581+0000 7fa7e639c700 1 -- 172.21.15.27:0/2856028438 >> v2:172.21.15.77:3300/0 conn(0x7fa7e015a7a0 msgr2=0x7fa7e015c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:03.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.581+0000 7fa7e639c700 1 --2- 172.21.15.27:0/2856028438 >> v2:172.21.15.77:3300/0 conn(0x7fa7e015a7a0 0x7fa7e015c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.581+0000 7fa7e639c700 1 -- 172.21.15.27:0/2856028438 >> v1:172.21.15.27:6789/0 conn(0x7fa7e0159e60 legacy=0x7fa7e015aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:03.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.581+0000 7fa7e639c700 1 -- 172.21.15.27:0/2856028438 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e015b8a0 con 0x7fa7e00685e0 2024-01-11T09:48:03.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.582+0000 7fa7e639c700 1 --2- 172.21.15.27:0/2856028438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e00685e0 0x7fa7e0068a00 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0069650 tx=0x7fa7d8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5f0d294b39b06 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:03.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.582+0000 7fa7e4b99700 1 -- 172.21.15.27:0/2856028438 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d800e070 con 0x7fa7e00685e0 2024-01-11T09:48:03.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.582+0000 7fa7e4b99700 1 -- 172.21.15.27:0/2856028438 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d800bab0 con 0x7fa7e00685e0 2024-01-11T09:48:03.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.582+0000 7fa7e4b99700 1 -- 172.21.15.27:0/2856028438 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d800be70 con 0x7fa7e00685e0 2024-01-11T09:48:03.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.582+0000 7fa7e7dff700 1 -- 172.21.15.27:0/2856028438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e00685e0 msgr2=0x7fa7e0068a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:03.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.582+0000 7fa7e7dff700 1 --2- 172.21.15.27:0/2856028438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e00685e0 0x7fa7e0068a00 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0069650 tx=0x7fa7d8009780 comp rx=0 tx=0).stop 2024-01-11T09:48:03.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.583+0000 7fa7e7dff700 1 -- 172.21.15.27:0/2856028438 shutdown_connections 2024-01-11T09:48:03.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.583+0000 7fa7e7dff700 1 --2- 172.21.15.27:0/2856028438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e00685e0 0x7fa7e0068a00 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.583+0000 7fa7e7dff700 1 --2- 172.21.15.27:0/2856028438 >> v2:172.21.15.77:3300/0 conn(0x7fa7e015a7a0 0x7fa7e015c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.583+0000 7fa7e7dff700 1 -- 172.21.15.27:0/2856028438 >> 172.21.15.27:0/2856028438 conn(0x7fa7e006ed40 msgr2=0x7fa7e006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:03.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.583+0000 7fa7e7dff700 1 -- 172.21.15.27:0/2856028438 shutdown_connections 2024-01-11T09:48:03.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.583+0000 7fa7e7dff700 1 -- 172.21.15.27:0/2856028438 wait complete. 2024-01-11T09:48:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.584+0000 7fa7e7dff700 1 Processor -- start 2024-01-11T09:48:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.584+0000 7fa7e7dff700 1 -- start start 2024-01-11T09:48:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.585+0000 7fa7e7dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e00685e0 0x7fa7e0150020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.585+0000 7fa7e7dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7e015a7a0 0x7fa7e014f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.585+0000 7fa7e7dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e014fae0 0x7fa7e016cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.585+0000 7fa7e7dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e016d970 con 0x7fa7e014fae0 2024-01-11T09:48:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.585+0000 7fa7e7dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e0155c10 con 0x7fa7e00685e0 2024-01-11T09:48:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.585+0000 7fa7e7dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e01557b0 con 0x7fa7e015a7a0 2024-01-11T09:48:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.586+0000 7fa7e639c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e014fae0 0x7fa7e016cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.586+0000 7fa7e639c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e014fae0 0x7fa7e016cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42488/0 (socket says 172.21.15.27:42488) 2024-01-11T09:48:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.586+0000 7fa7e639c700 1 -- 172.21.15.27:0/3397226449 learned_addr learned my addr 172.21.15.27:0/3397226449 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:03.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.586+0000 7fa7e539a700 1 --2- 172.21.15.27:0/3397226449 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7e015a7a0 0x7fa7e014f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.586+0000 7fa7e5b9b700 1 --2- 172.21.15.27:0/3397226449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e00685e0 0x7fa7e0150020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.586+0000 7fa7e639c700 1 -- 172.21.15.27:0/3397226449 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7e015a7a0 msgr2=0x7fa7e014f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.586+0000 7fa7e639c700 1 --2- 172.21.15.27:0/3397226449 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7e015a7a0 0x7fa7e014f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.586+0000 7fa7e639c700 1 -- 172.21.15.27:0/3397226449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e00685e0 msgr2=0x7fa7e0150020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.586+0000 7fa7e639c700 1 --2- 172.21.15.27:0/3397226449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e00685e0 0x7fa7e0150020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.586+0000 7fa7e639c700 1 -- 172.21.15.27:0/3397226449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d8009a20 con 0x7fa7e014fae0 2024-01-11T09:48:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.586+0000 7fa7e5b9b700 1 --2- 172.21.15.27:0/3397226449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e00685e0 0x7fa7e0150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:48:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.586+0000 7fa7e539a700 1 --2- 172.21.15.27:0/3397226449 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fa7e015a7a0 0x7fa7e014f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:48:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.586+0000 7fa7e639c700 1 --2- 172.21.15.27:0/3397226449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e014fae0 0x7fa7e016cae0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800b610 tx=0x7fa7d8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.586+0000 7fa7d5ffb700 1 -- 172.21.15.27:0/3397226449 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8027470 con 0x7fa7e014fae0 2024-01-11T09:48:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.587+0000 7fa7d5ffb700 1 -- 172.21.15.27:0/3397226449 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d80275d0 con 0x7fa7e014fae0 2024-01-11T09:48:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.587+0000 7fa7d5ffb700 1 -- 172.21.15.27:0/3397226449 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8027990 con 0x7fa7e014fae0 2024-01-11T09:48:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.587+0000 7fa7e7dff700 1 -- 172.21.15.27:0/3397226449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e01551f0 con 0x7fa7e014fae0 2024-01-11T09:48:03.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.587+0000 7fa7e7dff700 1 -- 172.21.15.27:0/3397226449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e014f7f0 con 0x7fa7e014fae0 2024-01-11T09:48:03.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.587+0000 7fa7e7dff700 1 -- 172.21.15.27:0/3397226449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa7e01559d0 con 0x7fa7e014fae0 2024-01-11T09:48:03.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.589+0000 7fa7d5ffb700 1 -- 172.21.15.27:0/3397226449 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fa7d8024070 con 0x7fa7e014fae0 2024-01-11T09:48:03.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.590+0000 7fa7d5ffb700 1 --2- 172.21.15.27:0/3397226449 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa7c4066f30 0x7fa7c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:03.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.590+0000 7fa7d5ffb700 1 -- 172.21.15.27:0/3397226449 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7fa7d8086fe0 con 0x7fa7e014fae0 2024-01-11T09:48:03.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.590+0000 7fa7d5ffb700 1 --2- 172.21.15.27:0/3397226449 >> [v2:172.21.15.77:6800/42095,v1:172.21.15.77:6801/42095] conn(0x7fa7c406d4c0 0x7fa7c406f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:03.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.590+0000 7fa7d5ffb700 1 -- 172.21.15.27:0/3397226449 --> [v2:172.21.15.77:6800/42095,v1:172.21.15.77:6801/42095] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa7c4070000 con 0x7fa7c406d4c0 2024-01-11T09:48:03.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.590+0000 7fa7d5ffb700 1 -- 172.21.15.27:0/3397226449 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=261) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7d8089500 con 0x7fa7e014fae0 2024-01-11T09:48:03.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.590+0000 7fa7e5b9b700 1 --2- 172.21.15.27:0/3397226449 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa7c4066f30 0x7fa7c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:03.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.590+0000 7fa7e539a700 1 --2- 172.21.15.27:0/3397226449 >> [v2:172.21.15.77:6800/42095,v1:172.21.15.77:6801/42095] conn(0x7fa7c406d4c0 0x7fa7c406f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:03.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.591+0000 7fa7e539a700 1 --2- 172.21.15.27:0/3397226449 >> [v2:172.21.15.77:6800/42095,v1:172.21.15.77:6801/42095] conn(0x7fa7c406d4c0 0x7fa7c406f930 crc :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:03.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.591+0000 7fa7e5b9b700 1 --2- 172.21.15.27:0/3397226449 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa7c4066f30 0x7fa7c40693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc009d80 tx=0x7fa7dc008040 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:03.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.592+0000 7fa7d5ffb700 1 -- 172.21.15.27:0/3397226449 <== osd.4 v2:172.21.15.77:6800/42095 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa7c4070000 con 0x7fa7c406d4c0 2024-01-11T09:48:03.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.607+0000 7fa7e7dff700 1 -- 172.21.15.27:0/3397226449 --> [v2:172.21.15.77:6800/42095,v1:172.21.15.77:6801/42095] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa7e004c3c0 con 0x7fa7c406d4c0 2024-01-11T09:48:03.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.607+0000 7fa7d5ffb700 1 -- 172.21.15.27:0/3397226449 <== osd.4 v2:172.21.15.77:6800/42095 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fa7e004c3c0 con 0x7fa7c406d4c0 2024-01-11T09:48:03.599 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:48:03.599 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-11T09:48:03.599 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:48:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.608+0000 7fa7e7dff700 1 -- 172.21.15.27:0/3397226449 >> [v2:172.21.15.77:6800/42095,v1:172.21.15.77:6801/42095] conn(0x7fa7c406d4c0 msgr2=0x7fa7c406f930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.608+0000 7fa7e7dff700 1 --2- 172.21.15.27:0/3397226449 >> [v2:172.21.15.77:6800/42095,v1:172.21.15.77:6801/42095] conn(0x7fa7c406d4c0 0x7fa7c406f930 crc :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.608+0000 7fa7e7dff700 1 -- 172.21.15.27:0/3397226449 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa7c4066f30 msgr2=0x7fa7c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.608+0000 7fa7e7dff700 1 --2- 172.21.15.27:0/3397226449 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fa7c4066f30 0x7fa7c40693e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc009d80 tx=0x7fa7dc008040 comp rx=0 tx=0).stop 2024-01-11T09:48:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.608+0000 7fa7e7dff700 1 -- 172.21.15.27:0/3397226449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e014fae0 msgr2=0x7fa7e016cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.608+0000 7fa7e7dff700 1 --2- 172.21.15.27:0/3397226449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7e014fae0 0x7fa7e016cae0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800b610 tx=0x7fa7d8019000 comp rx=0 tx=0).stop 2024-01-11T09:48:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.608+0000 7fa7e639c700 1 -- 172.21.15.27:0/3397226449 reap_dead start 2024-01-11T09:48:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.608+0000 7fa7e7dff700 1 -- 172.21.15.27:0/3397226449 shutdown_connections 2024-01-11T09:48:03.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.608+0000 7fa7e7dff700 1 -- 172.21.15.27:0/3397226449 >> 172.21.15.27:0/3397226449 conn(0x7fa7e006ed40 msgr2=0x7fa7e00b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:03.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.608+0000 7fa7e7dff700 1 -- 172.21.15.27:0/3397226449 shutdown_connections 2024-01-11T09:48:03.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.608+0000 7fa7e7dff700 1 -- 172.21.15.27:0/3397226449 wait complete. 2024-01-11T09:48:03.616 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-01-11T09:48:03.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.825+0000 7f9d6dbff700 1 Processor -- start 2024-01-11T09:48:03.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.825+0000 7f9d6dbff700 1 -- start start 2024-01-11T09:48:03.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.826+0000 7f9d6dbff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9d68068cd0 0x7f9d6815a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:03.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.826+0000 7f9d6dbff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d680b4bd0 con 0x7f9d6815cde0 2024-01-11T09:48:03.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.826+0000 7f9d6dbff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6815ac70 con 0x7f9d6815ca10 2024-01-11T09:48:03.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.826+0000 7f9d6dbff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d680b3fc0 con 0x7f9d68068cd0 2024-01-11T09:48:03.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.827+0000 7f9d66ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9d68068cd0 0x7f9d6815a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:03.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.827+0000 7f9d67fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9d6815cde0 0x7f9d680b5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41536/0 (socket says 172.21.15.27:41536) 2024-01-11T09:48:03.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.827+0000 7f9d67fff700 1 -- 172.21.15.27:0/3226196782 learned_addr learned my addr 172.21.15.27:0/3226196782 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:48:03.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.827+0000 7f9d667fc700 1 -- 172.21.15.27:0/3226196782 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156333279 0 0) 0x7f9d680b4bd0 con 0x7f9d6815cde0 2024-01-11T09:48:03.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.827+0000 7f9d66ffd700 1 -- 172.21.15.27:0/3226196782 >> v1:172.21.15.77:6789/0 conn(0x7f9d6815ca10 legacy=0x7f9d680685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:03.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.827+0000 7f9d66ffd700 1 -- 172.21.15.27:0/3226196782 >> v1:172.21.15.27:6789/0 conn(0x7f9d6815cde0 legacy=0x7f9d680b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:03.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.827+0000 7f9d66ffd700 1 -- 172.21.15.27:0/3226196782 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d680b42a0 con 0x7f9d68068cd0 2024-01-11T09:48:03.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.827+0000 7f9d66ffd700 1 --2- 172.21.15.27:0/3226196782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d68068cd0 0x7f9d6815a730 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c009910 tx=0x7f9d5c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4307833b38546b68 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:03.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.828+0000 7f9d667fc700 1 -- 172.21.15.27:0/3226196782 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c00b820 con 0x7f9d68068cd0 2024-01-11T09:48:03.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.828+0000 7f9d667fc700 1 -- 172.21.15.27:0/3226196782 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5c00b980 con 0x7f9d68068cd0 2024-01-11T09:48:03.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.828+0000 7f9d667fc700 1 -- 172.21.15.27:0/3226196782 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c00bd20 con 0x7f9d68068cd0 2024-01-11T09:48:03.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.828+0000 7f9d6dbff700 1 -- 172.21.15.27:0/3226196782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d68068cd0 msgr2=0x7f9d6815a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:03.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.828+0000 7f9d6dbff700 1 --2- 172.21.15.27:0/3226196782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d68068cd0 0x7f9d6815a730 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c009910 tx=0x7f9d5c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.828+0000 7f9d6dbff700 1 -- 172.21.15.27:0/3226196782 shutdown_connections 2024-01-11T09:48:03.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.828+0000 7f9d6dbff700 1 --2- 172.21.15.27:0/3226196782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d68068cd0 0x7f9d6815a730 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.828+0000 7f9d6dbff700 1 -- 172.21.15.27:0/3226196782 >> 172.21.15.27:0/3226196782 conn(0x7f9d6806ed40 msgr2=0x7f9d6806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:03.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.828+0000 7f9d6dbff700 1 -- 172.21.15.27:0/3226196782 shutdown_connections 2024-01-11T09:48:03.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.829+0000 7f9d6dbff700 1 -- 172.21.15.27:0/3226196782 wait complete. 2024-01-11T09:48:03.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.830+0000 7f9d6dbff700 1 Processor -- start 2024-01-11T09:48:03.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.830+0000 7f9d6dbff700 1 -- start start 2024-01-11T09:48:03.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.831+0000 7f9d6dbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9d68068cd0 0x7f9d6814a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.831+0000 7f9d6dbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d6815ca10 0x7f9d680ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.831+0000 7f9d6dbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d6815cde0 0x7f9d680c5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.831+0000 7f9d6dbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6814a650 con 0x7f9d6815cde0 2024-01-11T09:48:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.831+0000 7f9d6dbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d680c6990 con 0x7f9d6815ca10 2024-01-11T09:48:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.831+0000 7f9d6dbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d680cad20 con 0x7f9d68068cd0 2024-01-11T09:48:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.831+0000 7f9d67fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d6815cde0 0x7f9d680c5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.832+0000 7f9d67fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d6815cde0 0x7f9d680c5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53402/0 (socket says 172.21.15.27:53402) 2024-01-11T09:48:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.832+0000 7f9d67fff700 1 -- 172.21.15.27:0/1799600371 learned_addr learned my addr 172.21.15.27:0/1799600371 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.832+0000 7f9d66ffd700 1 --2- 172.21.15.27:0/1799600371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d6815ca10 0x7f9d680ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.832+0000 7f9d677fe700 1 --2- 172.21.15.27:0/1799600371 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9d68068cd0 0x7f9d6814a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.832+0000 7f9d67fff700 1 -- 172.21.15.27:0/1799600371 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9d68068cd0 msgr2=0x7f9d6814a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.832+0000 7f9d67fff700 1 --2- 172.21.15.27:0/1799600371 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9d68068cd0 0x7f9d6814a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.832+0000 7f9d67fff700 1 -- 172.21.15.27:0/1799600371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d6815ca10 msgr2=0x7f9d680ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.832+0000 7f9d67fff700 1 --2- 172.21.15.27:0/1799600371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d6815ca10 0x7f9d680ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.832+0000 7f9d67fff700 1 -- 172.21.15.27:0/1799600371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5c009420 con 0x7f9d6815cde0 2024-01-11T09:48:03.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.832+0000 7f9d67fff700 1 --2- 172.21.15.27:0/1799600371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d6815cde0 0x7f9d680c5380 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f9d6000cbf0 tx=0x7f9d60001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:03.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.832+0000 7f9d4f7fe700 1 -- 172.21.15.27:0/1799600371 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6000a870 con 0x7f9d6815cde0 2024-01-11T09:48:03.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.832+0000 7f9d4f7fe700 1 -- 172.21.15.27:0/1799600371 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6000a9d0 con 0x7f9d6815cde0 2024-01-11T09:48:03.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.832+0000 7f9d4f7fe700 1 -- 172.21.15.27:0/1799600371 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6000ad90 con 0x7f9d6815cde0 2024-01-11T09:48:03.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.832+0000 7f9d6dbff700 1 -- 172.21.15.27:0/1799600371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d680c5d60 con 0x7f9d6815cde0 2024-01-11T09:48:03.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.833+0000 7f9d6dbff700 1 -- 172.21.15.27:0/1799600371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d680c6320 con 0x7f9d6815cde0 2024-01-11T09:48:03.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.834+0000 7f9d6dbff700 1 -- 172.21.15.27:0/1799600371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6804c3c0 con 0x7f9d6815cde0 2024-01-11T09:48:03.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.834+0000 7f9d4f7fe700 1 -- 172.21.15.27:0/1799600371 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f9d60007300 con 0x7f9d6815cde0 2024-01-11T09:48:03.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.835+0000 7f9d4f7fe700 1 --2- 172.21.15.27:0/1799600371 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9d54066f30 0x7f9d540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:03.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.835+0000 7f9d4f7fe700 1 -- 172.21.15.27:0/1799600371 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f9d60084860 con 0x7f9d6815cde0 2024-01-11T09:48:03.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.836+0000 7f9d677fe700 1 --2- 172.21.15.27:0/1799600371 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9d54066f30 0x7f9d540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:03.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.836+0000 7f9d677fe700 1 --2- 172.21.15.27:0/1799600371 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9d54066f30 0x7f9d540693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9d680b5a60 tx=0x7f9d58009040 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:03.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:03.839+0000 7f9d4f7fe700 1 -- 172.21.15.27:0/1799600371 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9d60051a60 con 0x7f9d6815cde0 2024-01-11T09:48:04.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.042+0000 7f9d6dbff700 1 -- 172.21.15.27:0/1799600371 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d680b45b0 con 0x7f9d54066f30 2024-01-11T09:48:04.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.043+0000 7f9d4f7fe700 1 -- 172.21.15.27:0/1799600371 <== mgr.11878 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9d680b45b0 con 0x7f9d54066f30 2024-01-11T09:48:04.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.045+0000 7f9d6dbff700 1 -- 172.21.15.27:0/1799600371 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9d54066f30 msgr2=0x7f9d540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.045+0000 7f9d6dbff700 1 --2- 172.21.15.27:0/1799600371 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9d54066f30 0x7f9d540693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9d680b5a60 tx=0x7f9d58009040 comp rx=0 tx=0).stop 2024-01-11T09:48:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.045+0000 7f9d6dbff700 1 -- 172.21.15.27:0/1799600371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d6815cde0 msgr2=0x7f9d680c5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.045+0000 7f9d6dbff700 1 --2- 172.21.15.27:0/1799600371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d6815cde0 0x7f9d680c5380 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f9d6000cbf0 tx=0x7f9d60001980 comp rx=0 tx=0).stop 2024-01-11T09:48:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.045+0000 7f9d6dbff700 1 -- 172.21.15.27:0/1799600371 shutdown_connections 2024-01-11T09:48:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.045+0000 7f9d6dbff700 1 --2- 172.21.15.27:0/1799600371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d6815cde0 0x7f9d680c5380 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.045+0000 7f9d6dbff700 1 --2- 172.21.15.27:0/1799600371 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f9d68068cd0 0x7f9d6814a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.045+0000 7f9d6dbff700 1 --2- 172.21.15.27:0/1799600371 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f9d54066f30 0x7f9d540693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.045+0000 7f9d6dbff700 1 --2- 172.21.15.27:0/1799600371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d6815ca10 0x7f9d680ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.045+0000 7f9d6dbff700 1 -- 172.21.15.27:0/1799600371 >> 172.21.15.27:0/1799600371 conn(0x7f9d6806ed40 msgr2=0x7f9d6815a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.045+0000 7f9d6dbff700 1 -- 172.21.15.27:0/1799600371 shutdown_connections 2024-01-11T09:48:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.045+0000 7f9d6dbff700 1 -- 172.21.15.27:0/1799600371 wait complete. 2024-01-11T09:48:04.038 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 4 to deep-scrub 2024-01-11T09:48:04.055 INFO:tasks.ceph:Scrubbing osd.5 2024-01-11T09:48:04.055 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-01-11T09:48:04.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.267+0000 7fe830dff700 1 Processor -- start 2024-01-11T09:48:04.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.268+0000 7fe830dff700 1 -- start start 2024-01-11T09:48:04.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.269+0000 7fe830dff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe82c159250 0x7fe82c0b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.269+0000 7fe830dff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe82c0cb410 0x7fe82c0cb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.270+0000 7fe830dff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe82c15a440 con 0x7fe82c0684d0 2024-01-11T09:48:04.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.270+0000 7fe830dff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe82c15ace0 con 0x7fe82c0cb410 2024-01-11T09:48:04.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.270+0000 7fe830dff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe82c0b0de0 con 0x7fe82c159250 2024-01-11T09:48:04.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.270+0000 7fe829d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe82c0cb410 0x7fe82c0cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.270+0000 7fe82ad9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe82c0684d0 0x7fe82c0ce2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41538/0 (socket says 172.21.15.27:41538) 2024-01-11T09:48:04.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.270+0000 7fe82ad9d700 1 -- 172.21.15.27:0/475385990 learned_addr learned my addr 172.21.15.27:0/475385990 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:48:04.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.270+0000 7fe82a59c700 1 --2- 172.21.15.27:0/475385990 >> v2:172.21.15.77:3300/0 conn(0x7fe82c159250 0x7fe82c0b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.270+0000 7fe82959a700 1 -- 172.21.15.27:0/475385990 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419643965 0 0) 0x7fe82c15a440 con 0x7fe82c0684d0 2024-01-11T09:48:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.270+0000 7fe82959a700 1 -- 172.21.15.27:0/475385990 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe814003570 con 0x7fe82c0684d0 2024-01-11T09:48:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.270+0000 7fe829d9b700 1 -- 172.21.15.27:0/475385990 >> v2:172.21.15.77:3300/0 conn(0x7fe82c159250 msgr2=0x7fe82c0b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.270+0000 7fe829d9b700 1 --2- 172.21.15.27:0/475385990 >> v2:172.21.15.77:3300/0 conn(0x7fe82c159250 0x7fe82c0b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.270+0000 7fe829d9b700 1 -- 172.21.15.27:0/475385990 >> v1:172.21.15.27:6789/0 conn(0x7fe82c0684d0 legacy=0x7fe82c0ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.270+0000 7fe829d9b700 1 -- 172.21.15.27:0/475385990 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82c15a720 con 0x7fe82c0cb410 2024-01-11T09:48:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.271+0000 7fe829d9b700 1 --2- 172.21.15.27:0/475385990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe82c0cb410 0x7fe82c0cb890 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7fe82c0ce0c0 tx=0x7fe8200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d069a6aaced714b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.271+0000 7fe82959a700 1 -- 172.21.15.27:0/475385990 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820011630 con 0x7fe82c0cb410 2024-01-11T09:48:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.271+0000 7fe82959a700 1 -- 172.21.15.27:0/475385990 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8200117b0 con 0x7fe82c0cb410 2024-01-11T09:48:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.271+0000 7fe82959a700 1 -- 172.21.15.27:0/475385990 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820011b70 con 0x7fe82c0cb410 2024-01-11T09:48:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.271+0000 7fe830dff700 1 -- 172.21.15.27:0/475385990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe82c0cb410 msgr2=0x7fe82c0cb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.271+0000 7fe830dff700 1 --2- 172.21.15.27:0/475385990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe82c0cb410 0x7fe82c0cb890 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7fe82c0ce0c0 tx=0x7fe8200091a0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.272+0000 7fe830dff700 1 -- 172.21.15.27:0/475385990 shutdown_connections 2024-01-11T09:48:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.272+0000 7fe830dff700 1 --2- 172.21.15.27:0/475385990 >> v2:172.21.15.77:3300/0 conn(0x7fe82c159250 0x7fe82c0b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.272+0000 7fe830dff700 1 --2- 172.21.15.27:0/475385990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe82c0cb410 0x7fe82c0cb890 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.272+0000 7fe830dff700 1 -- 172.21.15.27:0/475385990 >> 172.21.15.27:0/475385990 conn(0x7fe82c067610 msgr2=0x7fe82c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.272+0000 7fe830dff700 1 -- 172.21.15.27:0/475385990 shutdown_connections 2024-01-11T09:48:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.272+0000 7fe830dff700 1 -- 172.21.15.27:0/475385990 wait complete. 2024-01-11T09:48:04.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.273+0000 7fe830dff700 1 Processor -- start 2024-01-11T09:48:04.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.273+0000 7fe830dff700 1 -- start start 2024-01-11T09:48:04.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.274+0000 7fe830dff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe82c0684d0 0x7fe82c0c53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.274+0000 7fe830dff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe82c0cb410 0x7fe82c0c2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.274+0000 7fe830dff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe82c159250 0x7fe82c0c34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.274+0000 7fe830dff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c1569f0 con 0x7fe82c0cb410 2024-01-11T09:48:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.274+0000 7fe830dff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c0c6db0 con 0x7fe82c159250 2024-01-11T09:48:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.274+0000 7fe830dff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c0c6920 con 0x7fe82c0684d0 2024-01-11T09:48:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.274+0000 7fe829d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe82c0cb410 0x7fe82c0c2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.275+0000 7fe829d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe82c0cb410 0x7fe82c0c2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53420/0 (socket says 172.21.15.27:53420) 2024-01-11T09:48:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.275+0000 7fe829d9b700 1 -- 172.21.15.27:0/3451699673 learned_addr learned my addr 172.21.15.27:0/3451699673 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.275+0000 7fe82ad9d700 1 --2- 172.21.15.27:0/3451699673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe82c159250 0x7fe82c0c34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.275+0000 7fe82a59c700 1 --2- 172.21.15.27:0/3451699673 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe82c0684d0 0x7fe82c0c53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.275+0000 7fe829d9b700 1 -- 172.21.15.27:0/3451699673 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe82c0684d0 msgr2=0x7fe82c0c53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.275+0000 7fe829d9b700 1 --2- 172.21.15.27:0/3451699673 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fe82c0684d0 0x7fe82c0c53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.275+0000 7fe829d9b700 1 -- 172.21.15.27:0/3451699673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe82c159250 msgr2=0x7fe82c0c34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.275+0000 7fe829d9b700 1 --2- 172.21.15.27:0/3451699673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe82c159250 0x7fe82c0c34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.275+0000 7fe829d9b700 1 -- 172.21.15.27:0/3451699673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe820009420 con 0x7fe82c0cb410 2024-01-11T09:48:04.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.275+0000 7fe829d9b700 1 --2- 172.21.15.27:0/3451699673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe82c0cb410 0x7fe82c0c2fa0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7fe82c0ceef0 tx=0x7fe820017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:04.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.275+0000 7fe81a7fc700 1 -- 172.21.15.27:0/3451699673 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82001f070 con 0x7fe82c0cb410 2024-01-11T09:48:04.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.276+0000 7fe81a7fc700 1 -- 172.21.15.27:0/3451699673 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe820011920 con 0x7fe82c0cb410 2024-01-11T09:48:04.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.276+0000 7fe81a7fc700 1 -- 172.21.15.27:0/3451699673 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820023410 con 0x7fe82c0cb410 2024-01-11T09:48:04.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.276+0000 7fe830dff700 1 -- 172.21.15.27:0/3451699673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe82c0c5d10 con 0x7fe82c0cb410 2024-01-11T09:48:04.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.276+0000 7fe830dff700 1 -- 172.21.15.27:0/3451699673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe82c0c62d0 con 0x7fe82c0cb410 2024-01-11T09:48:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.277+0000 7fe830dff700 1 -- 172.21.15.27:0/3451699673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe82c04c3c0 con 0x7fe82c0cb410 2024-01-11T09:48:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.278+0000 7fe81a7fc700 1 -- 172.21.15.27:0/3451699673 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fe82002c3b0 con 0x7fe82c0cb410 2024-01-11T09:48:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.278+0000 7fe81a7fc700 1 --2- 172.21.15.27:0/3451699673 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe80c066f30 0x7fe80c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.279+0000 7fe81a7fc700 1 -- 172.21.15.27:0/3451699673 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7fe8200857c0 con 0x7fe82c0cb410 2024-01-11T09:48:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.279+0000 7fe81a7fc700 1 --2- 172.21.15.27:0/3451699673 >> [v2:172.21.15.77:6810/42096,v1:172.21.15.77:6812/42096] conn(0x7fe80c06d4c0 0x7fe80c06f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.279+0000 7fe81a7fc700 1 -- 172.21.15.27:0/3451699673 --> [v2:172.21.15.77:6810/42096,v1:172.21.15.77:6812/42096] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe80c070000 con 0x7fe80c06d4c0 2024-01-11T09:48:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.279+0000 7fe81a7fc700 1 -- 172.21.15.27:0/3451699673 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=261) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe820087da0 con 0x7fe82c0cb410 2024-01-11T09:48:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.279+0000 7fe82a59c700 1 --2- 172.21.15.27:0/3451699673 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe80c066f30 0x7fe80c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.279+0000 7fe82ad9d700 1 --2- 172.21.15.27:0/3451699673 >> [v2:172.21.15.77:6810/42096,v1:172.21.15.77:6812/42096] conn(0x7fe80c06d4c0 0x7fe80c06f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.280+0000 7fe82a59c700 1 --2- 172.21.15.27:0/3451699673 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe80c066f30 0x7fe80c0693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009db0 tx=0x7fe81c0091a0 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:04.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.280+0000 7fe82ad9d700 1 --2- 172.21.15.27:0/3451699673 >> [v2:172.21.15.77:6810/42096,v1:172.21.15.77:6812/42096] conn(0x7fe80c06d4c0 0x7fe80c06f930 crc :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:04.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.281+0000 7fe81a7fc700 1 -- 172.21.15.27:0/3451699673 <== osd.5 v2:172.21.15.77:6810/42096 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fe80c070000 con 0x7fe80c06d4c0 2024-01-11T09:48:04.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.295+0000 7fe830dff700 1 -- 172.21.15.27:0/3451699673 --> [v2:172.21.15.77:6810/42096,v1:172.21.15.77:6812/42096] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe82c15aa00 con 0x7fe80c06d4c0 2024-01-11T09:48:04.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.296+0000 7fe81a7fc700 1 -- 172.21.15.27:0/3451699673 <== osd.5 v2:172.21.15.77:6810/42096 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fe82c15aa00 con 0x7fe80c06d4c0 2024-01-11T09:48:04.288 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:48:04.288 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-11T09:48:04.288 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:48:04.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.297+0000 7fe830dff700 1 -- 172.21.15.27:0/3451699673 >> [v2:172.21.15.77:6810/42096,v1:172.21.15.77:6812/42096] conn(0x7fe80c06d4c0 msgr2=0x7fe80c06f930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.297+0000 7fe830dff700 1 --2- 172.21.15.27:0/3451699673 >> [v2:172.21.15.77:6810/42096,v1:172.21.15.77:6812/42096] conn(0x7fe80c06d4c0 0x7fe80c06f930 crc :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.297+0000 7fe830dff700 1 -- 172.21.15.27:0/3451699673 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe80c066f30 msgr2=0x7fe80c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.297+0000 7fe830dff700 1 --2- 172.21.15.27:0/3451699673 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fe80c066f30 0x7fe80c0693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009db0 tx=0x7fe81c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.297+0000 7fe830dff700 1 -- 172.21.15.27:0/3451699673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe82c0cb410 msgr2=0x7fe82c0c2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.297+0000 7fe830dff700 1 --2- 172.21.15.27:0/3451699673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe82c0cb410 0x7fe82c0c2fa0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7fe82c0ceef0 tx=0x7fe820017000 comp rx=0 tx=0).stop 2024-01-11T09:48:04.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.297+0000 7fe82ad9d700 1 -- 172.21.15.27:0/3451699673 reap_dead start 2024-01-11T09:48:04.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.297+0000 7fe830dff700 1 -- 172.21.15.27:0/3451699673 shutdown_connections 2024-01-11T09:48:04.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.297+0000 7fe830dff700 1 -- 172.21.15.27:0/3451699673 >> 172.21.15.27:0/3451699673 conn(0x7fe82c067610 msgr2=0x7fe82c06a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:04.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.297+0000 7fe830dff700 1 -- 172.21.15.27:0/3451699673 shutdown_connections 2024-01-11T09:48:04.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.297+0000 7fe830dff700 1 -- 172.21.15.27:0/3451699673 wait complete. 2024-01-11T09:48:04.305 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-01-11T09:48:04.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.517+0000 7f057e5ff700 1 Processor -- start 2024-01-11T09:48:04.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.517+0000 7f057e5ff700 1 -- start start 2024-01-11T09:48:04.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.518+0000 7f057e5ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0578157360 0x7f057806f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.518+0000 7f057e5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f05780b19a0 0x7f05780cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.518+0000 7f057e5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05780cce10 con 0x7f0578157730 2024-01-11T09:48:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.518+0000 7f057e5ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05780cb540 con 0x7f0578157360 2024-01-11T09:48:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.518+0000 7f057e5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05780ca960 con 0x7f05780b19a0 2024-01-11T09:48:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.518+0000 7f0577fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0578157360 0x7f057806f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.518+0000 7f0577fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0578157360 0x7f057806f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53434/0 (socket says 172.21.15.27:53434) 2024-01-11T09:48:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.518+0000 7f0577fff700 1 -- 172.21.15.27:0/2666781289 learned_addr learned my addr 172.21.15.27:0/2666781289 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.519+0000 7f057cb9c700 1 --2- 172.21.15.27:0/2666781289 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f05780b19a0 0x7f05780cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.519+0000 7f0577fff700 1 -- 172.21.15.27:0/2666781289 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f05780b19a0 msgr2=0x7f05780cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.519+0000 7f0577fff700 1 --2- 172.21.15.27:0/2666781289 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f05780b19a0 0x7f05780cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.519+0000 7f0577fff700 1 -- 172.21.15.27:0/2666781289 >> v1:172.21.15.77:6789/0 conn(0x7f0578157730 legacy=0x7f05780b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.519+0000 7f0577fff700 1 -- 172.21.15.27:0/2666781289 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05780cac10 con 0x7f0578157360 2024-01-11T09:48:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.519+0000 7f057cb9c700 1 --2- 172.21.15.27:0/2666781289 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f05780b19a0 0x7f05780cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:48:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.519+0000 7f0577fff700 1 --2- 172.21.15.27:0/2666781289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0578157360 0x7f057806f570 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f05680099b0 tx=0x7f0568009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a492205e58f4120 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:04.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.519+0000 7f0576ffd700 1 -- 172.21.15.27:0/2666781289 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056800b860 con 0x7f0578157360 2024-01-11T09:48:04.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.519+0000 7f0576ffd700 1 -- 172.21.15.27:0/2666781289 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056800b9c0 con 0x7f0578157360 2024-01-11T09:48:04.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.519+0000 7f0576ffd700 1 -- 172.21.15.27:0/2666781289 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056800bd60 con 0x7f0578157360 2024-01-11T09:48:04.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.520+0000 7f057e5ff700 1 -- 172.21.15.27:0/2666781289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0578157360 msgr2=0x7f057806f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.520+0000 7f057e5ff700 1 --2- 172.21.15.27:0/2666781289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0578157360 0x7f057806f570 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f05680099b0 tx=0x7f0568009250 comp rx=0 tx=0).stop 2024-01-11T09:48:04.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.520+0000 7f057e5ff700 1 -- 172.21.15.27:0/2666781289 shutdown_connections 2024-01-11T09:48:04.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.520+0000 7f057e5ff700 1 --2- 172.21.15.27:0/2666781289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0578157360 0x7f057806f570 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.520+0000 7f057e5ff700 1 --2- 172.21.15.27:0/2666781289 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f05780b19a0 0x7f05780cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.520+0000 7f057e5ff700 1 -- 172.21.15.27:0/2666781289 >> 172.21.15.27:0/2666781289 conn(0x7f05780680b0 msgr2=0x7f05780cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:04.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.520+0000 7f057e5ff700 1 -- 172.21.15.27:0/2666781289 shutdown_connections 2024-01-11T09:48:04.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.520+0000 7f057e5ff700 1 -- 172.21.15.27:0/2666781289 wait complete. 2024-01-11T09:48:04.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.522+0000 7f057e5ff700 1 Processor -- start 2024-01-11T09:48:04.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.522+0000 7f057e5ff700 1 -- start start 2024-01-11T09:48:04.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.523+0000 7f057e5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05780b19a0 0x7f057814c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.523+0000 7f057e5ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0578157360 0x7f0578154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.523+0000 7f057e5ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0578157730 0x7f057806e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.523+0000 7f057e5ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05781535a0 con 0x7f0578157360 2024-01-11T09:48:04.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.523+0000 7f057e5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0578150c90 con 0x7f05780b19a0 2024-01-11T09:48:04.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.523+0000 7f057e5ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0578150360 con 0x7f0578157730 2024-01-11T09:48:04.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.523+0000 7f05777fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0578157360 0x7f0578154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.523+0000 7f05777fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0578157360 0x7f0578154190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53450/0 (socket says 172.21.15.27:53450) 2024-01-11T09:48:04.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.523+0000 7f05777fe700 1 -- 172.21.15.27:0/3789130076 learned_addr learned my addr 172.21.15.27:0/3789130076 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:04.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.523+0000 7f0577fff700 1 --2- 172.21.15.27:0/3789130076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05780b19a0 0x7f057814c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.523+0000 7f057cb9c700 1 --2- 172.21.15.27:0/3789130076 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0578157730 0x7f057806e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.523+0000 7f05777fe700 1 -- 172.21.15.27:0/3789130076 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0578157730 msgr2=0x7f057806e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.523+0000 7f05777fe700 1 --2- 172.21.15.27:0/3789130076 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0578157730 0x7f057806e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.523+0000 7f05777fe700 1 -- 172.21.15.27:0/3789130076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05780b19a0 msgr2=0x7f057814c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.523+0000 7f05777fe700 1 --2- 172.21.15.27:0/3789130076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05780b19a0 0x7f057814c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.524+0000 7f05777fe700 1 -- 172.21.15.27:0/3789130076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05680094f0 con 0x7f0578157360 2024-01-11T09:48:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.524+0000 7f0577fff700 1 --2- 172.21.15.27:0/3789130076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05780b19a0 0x7f057814c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:48:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.524+0000 7f05777fe700 1 --2- 172.21.15.27:0/3789130076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0578157360 0x7f0578154190 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f056c00c590 tx=0x7f056c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.524+0000 7f055ffff700 1 -- 172.21.15.27:0/3789130076 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c011590 con 0x7f0578157360 2024-01-11T09:48:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.524+0000 7f055ffff700 1 -- 172.21.15.27:0/3789130076 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056c0116f0 con 0x7f0578157360 2024-01-11T09:48:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.524+0000 7f055ffff700 1 -- 172.21.15.27:0/3789130076 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c011ab0 con 0x7f0578157360 2024-01-11T09:48:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.524+0000 7f057e5ff700 1 -- 172.21.15.27:0/3789130076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0578150640 con 0x7f0578157360 2024-01-11T09:48:04.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.524+0000 7f057e5ff700 1 -- 172.21.15.27:0/3789130076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0578154c30 con 0x7f0578157360 2024-01-11T09:48:04.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.525+0000 7f057e5ff700 1 -- 172.21.15.27:0/3789130076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f057804c3c0 con 0x7f0578157360 2024-01-11T09:48:04.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.527+0000 7f055ffff700 1 -- 172.21.15.27:0/3789130076 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f056c01b3b0 con 0x7f0578157360 2024-01-11T09:48:04.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.528+0000 7f055ffff700 1 --2- 172.21.15.27:0/3789130076 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0564066f30 0x7f05640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.528+0000 7f055ffff700 1 -- 172.21.15.27:0/3789130076 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f056c083b60 con 0x7f0578157360 2024-01-11T09:48:04.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.529+0000 7f0577fff700 1 --2- 172.21.15.27:0/3789130076 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0564066f30 0x7f05640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.529+0000 7f0577fff700 1 --2- 172.21.15.27:0/3789130076 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0564066f30 0x7f05640693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f05680099b0 tx=0x7f0568017000 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:04.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.532+0000 7f055ffff700 1 -- 172.21.15.27:0/3789130076 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f056c050ce0 con 0x7f0578157360 2024-01-11T09:48:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.736+0000 7f057e5ff700 1 -- 172.21.15.27:0/3789130076 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f057806cdf0 con 0x7f0564066f30 2024-01-11T09:48:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.736+0000 7f055ffff700 1 -- 172.21.15.27:0/3789130076 <== mgr.11878 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f057806cdf0 con 0x7f0564066f30 2024-01-11T09:48:04.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.739+0000 7f057e5ff700 1 -- 172.21.15.27:0/3789130076 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0564066f30 msgr2=0x7f05640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.739+0000 7f057e5ff700 1 --2- 172.21.15.27:0/3789130076 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0564066f30 0x7f05640693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f05680099b0 tx=0x7f0568017000 comp rx=0 tx=0).stop 2024-01-11T09:48:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.739+0000 7f057e5ff700 1 -- 172.21.15.27:0/3789130076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0578157360 msgr2=0x7f0578154190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.739+0000 7f057e5ff700 1 --2- 172.21.15.27:0/3789130076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0578157360 0x7f0578154190 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f056c00c590 tx=0x7f056c00c950 comp rx=0 tx=0).stop 2024-01-11T09:48:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.739+0000 7f057e5ff700 1 -- 172.21.15.27:0/3789130076 shutdown_connections 2024-01-11T09:48:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.739+0000 7f057e5ff700 1 --2- 172.21.15.27:0/3789130076 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0578157730 0x7f057806e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.739+0000 7f057e5ff700 1 --2- 172.21.15.27:0/3789130076 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0564066f30 0x7f05640693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.739+0000 7f057e5ff700 1 --2- 172.21.15.27:0/3789130076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05780b19a0 0x7f057814c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.739+0000 7f057e5ff700 1 --2- 172.21.15.27:0/3789130076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0578157360 0x7f0578154190 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.739+0000 7f057e5ff700 1 -- 172.21.15.27:0/3789130076 >> 172.21.15.27:0/3789130076 conn(0x7f05780680b0 msgr2=0x7f05780ac0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.739+0000 7f057e5ff700 1 -- 172.21.15.27:0/3789130076 shutdown_connections 2024-01-11T09:48:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.739+0000 7f057e5ff700 1 -- 172.21.15.27:0/3789130076 wait complete. 2024-01-11T09:48:04.732 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 5 to deep-scrub 2024-01-11T09:48:04.748 INFO:tasks.ceph:Scrubbing osd.6 2024-01-11T09:48:04.748 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-01-11T09:48:04.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.961+0000 7f19ccbff700 1 Processor -- start 2024-01-11T09:48:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.961+0000 7f19ccbff700 1 -- start start 2024-01-11T09:48:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.962+0000 7f19ccbff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f19c815a7a0 0x7f19c815c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.962+0000 7f19ccbff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c815c4b0 con 0x7f19c80685e0 2024-01-11T09:48:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.962+0000 7f19ccbff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c815b5d0 con 0x7f19c8159e60 2024-01-11T09:48:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.962+0000 7f19ccbff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19c815b8a0 con 0x7f19c815a7a0 2024-01-11T09:48:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.963+0000 7f19c6d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f19c80685e0 0x7f19c8068a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41544/0 (socket says 172.21.15.27:41544) 2024-01-11T09:48:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.963+0000 7f19c6d9d700 1 -- 172.21.15.27:0/1580566753 learned_addr learned my addr 172.21.15.27:0/1580566753 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:48:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.963+0000 7f19c659c700 1 --2- 172.21.15.27:0/1580566753 >> v2:172.21.15.77:3300/0 conn(0x7f19c815a7a0 0x7f19c815c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.963+0000 7f19c559a700 1 -- 172.21.15.27:0/1580566753 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703523220 0 0) 0x7f19c815c4b0 con 0x7f19c80685e0 2024-01-11T09:48:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.963+0000 7f19c559a700 1 -- 172.21.15.27:0/1580566753 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19b0003570 con 0x7f19c80685e0 2024-01-11T09:48:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.963+0000 7f19c559a700 1 -- 172.21.15.27:0/1580566753 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2837858947 0 0) 0x7f19b0003570 con 0x7f19c80685e0 2024-01-11T09:48:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c559a700 1 -- 172.21.15.27:0/1580566753 >> v2:172.21.15.77:3300/0 conn(0x7f19c815a7a0 msgr2=0x7f19c815c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c559a700 1 --2- 172.21.15.27:0/1580566753 >> v2:172.21.15.77:3300/0 conn(0x7f19c815a7a0 0x7f19c815c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c559a700 1 -- 172.21.15.27:0/1580566753 >> v1:172.21.15.77:6789/0 conn(0x7f19c8159e60 legacy=0x7f19c815aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c559a700 1 -- 172.21.15.27:0/1580566753 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c815bb80 con 0x7f19c80685e0 2024-01-11T09:48:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c559a700 1 -- 172.21.15.27:0/1580566753 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f19c00017f0 con 0x7f19c80685e0 2024-01-11T09:48:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c559a700 1 -- 172.21.15.27:0/1580566753 >> v1:172.21.15.27:6789/0 conn(0x7f19c80685e0 legacy=0x7f19c8068a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c559a700 1 --2- 172.21.15.27:0/1580566753 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19b0003a60 0x7f19b0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c559a700 1 --2- 172.21.15.27:0/1580566753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19b0004810 0x7f19b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c559a700 1 --2- 172.21.15.27:0/1580566753 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19b00051c0 0x7f19b000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c559a700 1 -- 172.21.15.27:0/1580566753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c815b8a0 con 0x7f19b0004810 2024-01-11T09:48:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c559a700 1 -- 172.21.15.27:0/1580566753 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b000c0b0 con 0x7f19b0003a60 2024-01-11T09:48:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c559a700 1 -- 172.21.15.27:0/1580566753 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19b000c270 con 0x7f19b00051c0 2024-01-11T09:48:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c5d9b700 1 --2- 172.21.15.27:0/1580566753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19b0004810 0x7f19b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c659c700 1 --2- 172.21.15.27:0/1580566753 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19b0003a60 0x7f19b0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c6d9d700 1 --2- 172.21.15.27:0/1580566753 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19b00051c0 0x7f19b000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c5d9b700 1 -- 172.21.15.27:0/1580566753 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19b00051c0 msgr2=0x7f19b000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c5d9b700 1 --2- 172.21.15.27:0/1580566753 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19b00051c0 0x7f19b000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c5d9b700 1 -- 172.21.15.27:0/1580566753 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19b0003a60 msgr2=0x7f19b0003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c5d9b700 1 --2- 172.21.15.27:0/1580566753 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19b0003a60 0x7f19b0003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.964+0000 7f19c5d9b700 1 -- 172.21.15.27:0/1580566753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b0003570 con 0x7f19b0004810 2024-01-11T09:48:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.965+0000 7f19c6d9d700 1 -- 172.21.15.27:0/1580566753 reap_dead start 2024-01-11T09:48:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.965+0000 7f19c5d9b700 1 --2- 172.21.15.27:0/1580566753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19b0004810 0x7f19b0004c80 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f19bc0019a0 tx=0x7f19bc007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a9f4a0335042ac server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.965+0000 7f19c559a700 1 -- 172.21.15.27:0/1580566753 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc00c5d0 con 0x7f19b0004810 2024-01-11T09:48:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.965+0000 7f19c559a700 1 -- 172.21.15.27:0/1580566753 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19bc00c730 con 0x7f19b0004810 2024-01-11T09:48:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.965+0000 7f19c559a700 1 -- 172.21.15.27:0/1580566753 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc00cad0 con 0x7f19b0004810 2024-01-11T09:48:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.965+0000 7f19ccbff700 1 -- 172.21.15.27:0/1580566753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19b0004810 msgr2=0x7f19b0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.965+0000 7f19ccbff700 1 --2- 172.21.15.27:0/1580566753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19b0004810 0x7f19b0004c80 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f19bc0019a0 tx=0x7f19bc007d10 comp rx=0 tx=0).stop 2024-01-11T09:48:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.965+0000 7f19ccbff700 1 -- 172.21.15.27:0/1580566753 shutdown_connections 2024-01-11T09:48:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.965+0000 7f19ccbff700 1 --2- 172.21.15.27:0/1580566753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19b0004810 0x7f19b0004c80 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.965+0000 7f19ccbff700 1 -- 172.21.15.27:0/1580566753 >> 172.21.15.27:0/1580566753 conn(0x7f19c806ed40 msgr2=0x7f19c806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.966+0000 7f19ccbff700 1 -- 172.21.15.27:0/1580566753 shutdown_connections 2024-01-11T09:48:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.966+0000 7f19ccbff700 1 -- 172.21.15.27:0/1580566753 wait complete. 2024-01-11T09:48:04.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.967+0000 7f19ccbff700 1 Processor -- start 2024-01-11T09:48:04.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.967+0000 7f19ccbff700 1 -- start start 2024-01-11T09:48:04.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.968+0000 7f19ccbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19b0004810 0x7f19c814b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.968+0000 7f19ccbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19c8151800 0x7f19c8151c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.968+0000 7f19ccbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c80689e0 0x7f19c8068e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.968+0000 7f19ccbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c8069390 con 0x7f19c80689e0 2024-01-11T09:48:04.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.968+0000 7f19ccbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c8152e90 con 0x7f19c8151800 2024-01-11T09:48:04.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.968+0000 7f19ccbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c814b660 con 0x7f19b0004810 2024-01-11T09:48:04.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.968+0000 7f19c6d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c80689e0 0x7f19c8068e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.968+0000 7f19c6d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c80689e0 0x7f19c8068e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53470/0 (socket says 172.21.15.27:53470) 2024-01-11T09:48:04.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.968+0000 7f19c6d9d700 1 -- 172.21.15.27:0/2876906449 learned_addr learned my addr 172.21.15.27:0/2876906449 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:04.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.969+0000 7f19c659c700 1 --2- 172.21.15.27:0/2876906449 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19b0004810 0x7f19c814b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.969+0000 7f19c5d9b700 1 --2- 172.21.15.27:0/2876906449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19c8151800 0x7f19c8151c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.969+0000 7f19c6d9d700 1 -- 172.21.15.27:0/2876906449 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19b0004810 msgr2=0x7f19c814b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.969+0000 7f19c6d9d700 1 --2- 172.21.15.27:0/2876906449 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f19b0004810 0x7f19c814b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.969+0000 7f19c6d9d700 1 -- 172.21.15.27:0/2876906449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19c8151800 msgr2=0x7f19c8151c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.969+0000 7f19c6d9d700 1 --2- 172.21.15.27:0/2876906449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19c8151800 0x7f19c8151c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.969+0000 7f19c6d9d700 1 -- 172.21.15.27:0/2876906449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c8152280 con 0x7f19c80689e0 2024-01-11T09:48:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.969+0000 7f19c6d9d700 1 --2- 172.21.15.27:0/2876906449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c80689e0 0x7f19c8068e50 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f19c80899d0 tx=0x7f19c0001370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.969+0000 7f19b67fc700 1 -- 172.21.15.27:0/2876906449 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c000eb10 con 0x7f19c80689e0 2024-01-11T09:48:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.969+0000 7f19b67fc700 1 -- 172.21.15.27:0/2876906449 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c000ec90 con 0x7f19c80689e0 2024-01-11T09:48:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.969+0000 7f19b67fc700 1 -- 172.21.15.27:0/2876906449 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c0013420 con 0x7f19c80689e0 2024-01-11T09:48:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.969+0000 7f19ccbff700 1 -- 172.21.15.27:0/2876906449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c8152560 con 0x7f19c80689e0 2024-01-11T09:48:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.970+0000 7f19ccbff700 1 -- 172.21.15.27:0/2876906449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c8152b20 con 0x7f19c80689e0 2024-01-11T09:48:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.970+0000 7f19ccbff700 1 -- 172.21.15.27:0/2876906449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f19c8169a40 con 0x7f19c80689e0 2024-01-11T09:48:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.972+0000 7f19b67fc700 1 -- 172.21.15.27:0/2876906449 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f19c00137b0 con 0x7f19c80689e0 2024-01-11T09:48:04.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.973+0000 7f19b67fc700 1 --2- 172.21.15.27:0/2876906449 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f19a8067000 0x7f19a80694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.973+0000 7f19b67fc700 1 -- 172.21.15.27:0/2876906449 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f19c0084a40 con 0x7f19c80689e0 2024-01-11T09:48:04.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.973+0000 7f19b67fc700 1 --2- 172.21.15.27:0/2876906449 >> [v2:172.21.15.77:6808/42097,v1:172.21.15.77:6809/42097] conn(0x7f19a806d590 0x7f19a806fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:04.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.973+0000 7f19b67fc700 1 -- 172.21.15.27:0/2876906449 --> [v2:172.21.15.77:6808/42097,v1:172.21.15.77:6809/42097] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19a80700d0 con 0x7f19a806d590 2024-01-11T09:48:04.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.973+0000 7f19b67fc700 1 -- 172.21.15.27:0/2876906449 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=261) v2 ==== 24+0+0 (secure 0 0 0) 0x7f19c0089220 con 0x7f19c80689e0 2024-01-11T09:48:04.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.973+0000 7f19c659c700 1 --2- 172.21.15.27:0/2876906449 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f19a8067000 0x7f19a80694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.974+0000 7f19c5d9b700 1 --2- 172.21.15.27:0/2876906449 >> [v2:172.21.15.77:6808/42097,v1:172.21.15.77:6809/42097] conn(0x7f19a806d590 0x7f19a806fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:04.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.974+0000 7f19c659c700 1 --2- 172.21.15.27:0/2876906449 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f19a8067000 0x7f19a80694b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f19b8006f00 tx=0x7f19b80091a0 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:04.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.974+0000 7f19c5d9b700 1 --2- 172.21.15.27:0/2876906449 >> [v2:172.21.15.77:6808/42097,v1:172.21.15.77:6809/42097] conn(0x7f19a806d590 0x7f19a806fa00 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:04.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.975+0000 7f19b67fc700 1 -- 172.21.15.27:0/2876906449 <== osd.6 v2:172.21.15.77:6808/42097 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f19a80700d0 con 0x7f19a806d590 2024-01-11T09:48:04.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.990+0000 7f19ccbff700 1 -- 172.21.15.27:0/2876906449 --> [v2:172.21.15.77:6808/42097,v1:172.21.15.77:6809/42097] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f19c804c3c0 con 0x7f19a806d590 2024-01-11T09:48:04.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.990+0000 7f19b67fc700 1 -- 172.21.15.27:0/2876906449 <== osd.6 v2:172.21.15.77:6808/42097 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f19c804c3c0 con 0x7f19a806d590 2024-01-11T09:48:04.982 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:48:04.982 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-11T09:48:04.982 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:48:04.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.991+0000 7f19ccbff700 1 -- 172.21.15.27:0/2876906449 >> [v2:172.21.15.77:6808/42097,v1:172.21.15.77:6809/42097] conn(0x7f19a806d590 msgr2=0x7f19a806fa00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.991+0000 7f19ccbff700 1 --2- 172.21.15.27:0/2876906449 >> [v2:172.21.15.77:6808/42097,v1:172.21.15.77:6809/42097] conn(0x7f19a806d590 0x7f19a806fa00 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.991+0000 7f19ccbff700 1 -- 172.21.15.27:0/2876906449 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f19a8067000 msgr2=0x7f19a80694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.991+0000 7f19ccbff700 1 --2- 172.21.15.27:0/2876906449 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f19a8067000 0x7f19a80694b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f19b8006f00 tx=0x7f19b80091a0 comp rx=0 tx=0).stop 2024-01-11T09:48:04.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.991+0000 7f19ccbff700 1 -- 172.21.15.27:0/2876906449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c80689e0 msgr2=0x7f19c8068e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:04.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.991+0000 7f19ccbff700 1 --2- 172.21.15.27:0/2876906449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c80689e0 0x7f19c8068e50 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f19c80899d0 tx=0x7f19c0001370 comp rx=0 tx=0).stop 2024-01-11T09:48:04.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.991+0000 7f19c6d9d700 1 -- 172.21.15.27:0/2876906449 reap_dead start 2024-01-11T09:48:04.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.991+0000 7f19ccbff700 1 -- 172.21.15.27:0/2876906449 shutdown_connections 2024-01-11T09:48:04.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.991+0000 7f19ccbff700 1 -- 172.21.15.27:0/2876906449 >> 172.21.15.27:0/2876906449 conn(0x7f19c806ed40 msgr2=0x7f19c814fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:04.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.991+0000 7f19ccbff700 1 -- 172.21.15.27:0/2876906449 shutdown_connections 2024-01-11T09:48:04.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:04.991+0000 7f19ccbff700 1 -- 172.21.15.27:0/2876906449 wait complete. 2024-01-11T09:48:04.999 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-01-11T09:48:05.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.210+0000 7f0b90fff700 1 Processor -- start 2024-01-11T09:48:05.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.210+0000 7f0b90fff700 1 -- start start 2024-01-11T09:48:05.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.211+0000 7f0b90fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b8c15ca10 0x7f0b8c15a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:05.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.211+0000 7f0b90fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b8c068cd0 0x7f0b8c0b5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:05.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.211+0000 7f0b90fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8c0b4bd0 con 0x7f0b8c15cde0 2024-01-11T09:48:05.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.211+0000 7f0b90fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b8c15ac70 con 0x7f0b8c068cd0 2024-01-11T09:48:05.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.211+0000 7f0b90fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8c0b3fc0 con 0x7f0b8c15ca10 2024-01-11T09:48:05.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.211+0000 7f0b89d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0b8c15cde0 0x7f0b8c0685c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41556/0 (socket says 172.21.15.27:41556) 2024-01-11T09:48:05.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.211+0000 7f0b89d9b700 1 -- 172.21.15.27:0/2051193706 learned_addr learned my addr 172.21.15.27:0/2051193706 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:48:05.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.212+0000 7f0b8ad9d700 1 --2- 172.21.15.27:0/2051193706 >> v2:172.21.15.27:3300/0 conn(0x7f0b8c068cd0 0x7f0b8c0b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:05.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.212+0000 7f0b8a59c700 1 --2- 172.21.15.27:0/2051193706 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b8c15ca10 0x7f0b8c15a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:05.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.212+0000 7f0b8959a700 1 -- 172.21.15.27:0/2051193706 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117032809 0 0) 0x7f0b8c0b4bd0 con 0x7f0b8c15cde0 2024-01-11T09:48:05.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.212+0000 7f0b8959a700 1 -- 172.21.15.27:0/2051193706 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b74003570 con 0x7f0b8c15cde0 2024-01-11T09:48:05.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.212+0000 7f0b8ad9d700 1 -- 172.21.15.27:0/2051193706 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b8c15ca10 msgr2=0x7f0b8c15a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:05.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.212+0000 7f0b8ad9d700 1 --2- 172.21.15.27:0/2051193706 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b8c15ca10 0x7f0b8c15a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:05.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.212+0000 7f0b8ad9d700 1 -- 172.21.15.27:0/2051193706 >> v1:172.21.15.27:6789/0 conn(0x7f0b8c15cde0 legacy=0x7f0b8c0685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:05.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.212+0000 7f0b8ad9d700 1 -- 172.21.15.27:0/2051193706 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8c0b42a0 con 0x7f0b8c068cd0 2024-01-11T09:48:05.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.212+0000 7f0b8ad9d700 1 --2- 172.21.15.27:0/2051193706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c068cd0 0x7f0b8c0b5060 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7f0b84009e60 tx=0x7f0b84009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd3aeb687981e8b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:05.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.213+0000 7f0b8959a700 1 -- 172.21.15.27:0/2051193706 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8400e070 con 0x7f0b8c068cd0 2024-01-11T09:48:05.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.213+0000 7f0b8959a700 1 -- 172.21.15.27:0/2051193706 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8400bab0 con 0x7f0b8c068cd0 2024-01-11T09:48:05.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.213+0000 7f0b8959a700 1 -- 172.21.15.27:0/2051193706 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8400be70 con 0x7f0b8c068cd0 2024-01-11T09:48:05.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.213+0000 7f0b90fff700 1 -- 172.21.15.27:0/2051193706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c068cd0 msgr2=0x7f0b8c0b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:05.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.213+0000 7f0b90fff700 1 --2- 172.21.15.27:0/2051193706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c068cd0 0x7f0b8c0b5060 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7f0b84009e60 tx=0x7f0b84009780 comp rx=0 tx=0).stop 2024-01-11T09:48:05.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.213+0000 7f0b90fff700 1 -- 172.21.15.27:0/2051193706 shutdown_connections 2024-01-11T09:48:05.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.213+0000 7f0b90fff700 1 --2- 172.21.15.27:0/2051193706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c068cd0 0x7f0b8c0b5060 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:05.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.213+0000 7f0b90fff700 1 --2- 172.21.15.27:0/2051193706 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b8c15ca10 0x7f0b8c15a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:05.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.213+0000 7f0b90fff700 1 -- 172.21.15.27:0/2051193706 >> 172.21.15.27:0/2051193706 conn(0x7f0b8c06ed40 msgr2=0x7f0b8c06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:05.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.213+0000 7f0b90fff700 1 -- 172.21.15.27:0/2051193706 shutdown_connections 2024-01-11T09:48:05.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.213+0000 7f0b90fff700 1 -- 172.21.15.27:0/2051193706 wait complete. 2024-01-11T09:48:05.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.215+0000 7f0b90fff700 1 Processor -- start 2024-01-11T09:48:05.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.215+0000 7f0b90fff700 1 -- start start 2024-01-11T09:48:05.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.216+0000 7f0b90fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b8c068cd0 0x7f0b8c154d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:05.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.216+0000 7f0b90fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c15ca10 0x7f0b8c157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:05.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.216+0000 7f0b90fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b8c15cde0 0x7f0b8c1553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:05.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.216+0000 7f0b90fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8c158180 con 0x7f0b8c15ca10 2024-01-11T09:48:05.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.216+0000 7f0b90fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8c0b1d80 con 0x7f0b8c15cde0 2024-01-11T09:48:05.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.216+0000 7f0b90fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8c0b2060 con 0x7f0b8c068cd0 2024-01-11T09:48:05.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.216+0000 7f0b89d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c15ca10 0x7f0b8c157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:05.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.216+0000 7f0b89d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c15ca10 0x7f0b8c157580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53492/0 (socket says 172.21.15.27:53492) 2024-01-11T09:48:05.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.216+0000 7f0b89d9b700 1 -- 172.21.15.27:0/3329675636 learned_addr learned my addr 172.21.15.27:0/3329675636 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:05.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.216+0000 7f0b8a59c700 1 --2- 172.21.15.27:0/3329675636 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b8c068cd0 0x7f0b8c154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:05.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.217+0000 7f0b8ad9d700 1 --2- 172.21.15.27:0/3329675636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b8c15cde0 0x7f0b8c1553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:05.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.217+0000 7f0b89d9b700 1 -- 172.21.15.27:0/3329675636 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b8c068cd0 msgr2=0x7f0b8c154d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:05.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.217+0000 7f0b89d9b700 1 --2- 172.21.15.27:0/3329675636 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b8c068cd0 0x7f0b8c154d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:05.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.217+0000 7f0b89d9b700 1 -- 172.21.15.27:0/3329675636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b8c15cde0 msgr2=0x7f0b8c1553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:05.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.217+0000 7f0b89d9b700 1 --2- 172.21.15.27:0/3329675636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b8c15cde0 0x7f0b8c1553b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:05.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.217+0000 7f0b89d9b700 1 -- 172.21.15.27:0/3329675636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b84009a20 con 0x7f0b8c15ca10 2024-01-11T09:48:05.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.217+0000 7f0b89d9b700 1 --2- 172.21.15.27:0/3329675636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c15ca10 0x7f0b8c157580 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f0b8000c590 tx=0x7f0b80007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:05.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.217+0000 7f0b7a7fc700 1 -- 172.21.15.27:0/3329675636 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b80005b30 con 0x7f0b8c15ca10 2024-01-11T09:48:05.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.217+0000 7f0b7a7fc700 1 -- 172.21.15.27:0/3329675636 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b80005cb0 con 0x7f0b8c15ca10 2024-01-11T09:48:05.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.217+0000 7f0b90fff700 1 -- 172.21.15.27:0/3329675636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b8c0b2340 con 0x7f0b8c15ca10 2024-01-11T09:48:05.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.217+0000 7f0b7a7fc700 1 -- 172.21.15.27:0/3329675636 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b80007320 con 0x7f0b8c15ca10 2024-01-11T09:48:05.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.217+0000 7f0b90fff700 1 -- 172.21.15.27:0/3329675636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b8c0c53c0 con 0x7f0b8c15ca10 2024-01-11T09:48:05.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.218+0000 7f0b90fff700 1 -- 172.21.15.27:0/3329675636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b8c04c3c0 con 0x7f0b8c15ca10 2024-01-11T09:48:05.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.219+0000 7f0b7a7fc700 1 -- 172.21.15.27:0/3329675636 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f0b8001b3b0 con 0x7f0b8c15ca10 2024-01-11T09:48:05.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.220+0000 7f0b7a7fc700 1 --2- 172.21.15.27:0/3329675636 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0b6c066f30 0x7f0b6c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:05.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.220+0000 7f0b7a7fc700 1 -- 172.21.15.27:0/3329675636 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f0b80083be0 con 0x7f0b8c15ca10 2024-01-11T09:48:05.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.221+0000 7f0b8a59c700 1 --2- 172.21.15.27:0/3329675636 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0b6c066f30 0x7f0b6c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:05.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.221+0000 7f0b8a59c700 1 --2- 172.21.15.27:0/3329675636 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0b6c066f30 0x7f0b6c0693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c009d40 tx=0x7f0b7c008040 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:05.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.224+0000 7f0b7a7fc700 1 -- 172.21.15.27:0/3329675636 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0b80050d60 con 0x7f0b8c15ca10 2024-01-11T09:48:05.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.428+0000 7f0b90fff700 1 -- 172.21.15.27:0/3329675636 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b8c0b4580 con 0x7f0b6c066f30 2024-01-11T09:48:05.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.429+0000 7f0b7a7fc700 1 -- 172.21.15.27:0/3329675636 <== mgr.11878 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0b8c0b4580 con 0x7f0b6c066f30 2024-01-11T09:48:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.431+0000 7f0b90fff700 1 -- 172.21.15.27:0/3329675636 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0b6c066f30 msgr2=0x7f0b6c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.431+0000 7f0b90fff700 1 --2- 172.21.15.27:0/3329675636 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0b6c066f30 0x7f0b6c0693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c009d40 tx=0x7f0b7c008040 comp rx=0 tx=0).stop 2024-01-11T09:48:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.431+0000 7f0b90fff700 1 -- 172.21.15.27:0/3329675636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c15ca10 msgr2=0x7f0b8c157580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.431+0000 7f0b90fff700 1 --2- 172.21.15.27:0/3329675636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c15ca10 0x7f0b8c157580 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f0b8000c590 tx=0x7f0b80007e00 comp rx=0 tx=0).stop 2024-01-11T09:48:05.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.431+0000 7f0b90fff700 1 -- 172.21.15.27:0/3329675636 shutdown_connections 2024-01-11T09:48:05.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.431+0000 7f0b90fff700 1 --2- 172.21.15.27:0/3329675636 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f0b6c066f30 0x7f0b6c0693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:05.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.431+0000 7f0b90fff700 1 --2- 172.21.15.27:0/3329675636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b8c15cde0 0x7f0b8c1553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:05.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.431+0000 7f0b90fff700 1 --2- 172.21.15.27:0/3329675636 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f0b8c068cd0 0x7f0b8c154d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:05.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.431+0000 7f0b90fff700 1 --2- 172.21.15.27:0/3329675636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b8c15ca10 0x7f0b8c157580 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:05.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.432+0000 7f0b90fff700 1 -- 172.21.15.27:0/3329675636 >> 172.21.15.27:0/3329675636 conn(0x7f0b8c06ed40 msgr2=0x7f0b8c0afe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:05.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.432+0000 7f0b90fff700 1 -- 172.21.15.27:0/3329675636 shutdown_connections 2024-01-11T09:48:05.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.432+0000 7f0b90fff700 1 -- 172.21.15.27:0/3329675636 wait complete. 2024-01-11T09:48:05.424 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 6 to deep-scrub 2024-01-11T09:48:05.441 INFO:tasks.ceph:Scrubbing osd.7 2024-01-11T09:48:05.441 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-01-11T09:48:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.653+0000 7ff5cbbff700 1 Processor -- start 2024-01-11T09:48:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.653+0000 7ff5cbbff700 1 -- start start 2024-01-11T09:48:05.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.654+0000 7ff5cbbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5c4067fd0 0x7ff5c40c7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.654+0000 7ff5cbbff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c406b390 con 0x7ff5c40aedc0 2024-01-11T09:48:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.654+0000 7ff5cbbff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c4068520 con 0x7ff5c40c1c60 2024-01-11T09:48:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.654+0000 7ff5cbbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c4068ae0 con 0x7ff5c4067fd0 2024-01-11T09:48:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.655+0000 7ff5ca19c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff5c40aedc0 0x7ff5c406ac80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41566/0 (socket says 172.21.15.27:41566) 2024-01-11T09:48:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.655+0000 7ff5ca19c700 1 -- 172.21.15.27:0/819986736 learned_addr learned my addr 172.21.15.27:0/819986736 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:48:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.655+0000 7ff5c919a700 1 --2- 172.21.15.27:0/819986736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5c4067fd0 0x7ff5c40c7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.655+0000 7ff5c8999700 1 -- 172.21.15.27:0/819986736 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1593462882 0 0) 0x7ff5c406b390 con 0x7ff5c40aedc0 2024-01-11T09:48:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.655+0000 7ff5c8999700 1 -- 172.21.15.27:0/819986736 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5b0003570 con 0x7ff5c40aedc0 2024-01-11T09:48:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.655+0000 7ff5c8999700 1 -- 172.21.15.27:0/819986736 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173989742 0 0) 0x7ff5c4068520 con 0x7ff5c40c1c60 2024-01-11T09:48:05.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.655+0000 7ff5c8999700 1 -- 172.21.15.27:0/819986736 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5c406b390 con 0x7ff5c40c1c60 2024-01-11T09:48:05.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.655+0000 7ff5c8999700 1 -- 172.21.15.27:0/819986736 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2702062707 0 0) 0x7ff5b0003570 con 0x7ff5c40aedc0 2024-01-11T09:48:05.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.655+0000 7ff5c919a700 1 -- 172.21.15.27:0/819986736 >> v1:172.21.15.77:6789/0 conn(0x7ff5c40c1c60 legacy=0x7ff5c406a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:05.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.655+0000 7ff5c919a700 1 -- 172.21.15.27:0/819986736 >> v1:172.21.15.27:6789/0 conn(0x7ff5c40aedc0 legacy=0x7ff5c406ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:05.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.655+0000 7ff5c919a700 1 -- 172.21.15.27:0/819986736 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c4068800 con 0x7ff5c4067fd0 2024-01-11T09:48:05.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.656+0000 7ff5c919a700 1 --2- 172.21.15.27:0/819986736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5c4067fd0 0x7ff5c40c7010 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4009910 tx=0x7ff5b40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8c3bd121257a7ea server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:05.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.656+0000 7ff5c8999700 1 -- 172.21.15.27:0/819986736 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b4011630 con 0x7ff5c4067fd0 2024-01-11T09:48:05.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.656+0000 7ff5c8999700 1 -- 172.21.15.27:0/819986736 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b4011790 con 0x7ff5c4067fd0 2024-01-11T09:48:05.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.656+0000 7ff5c8999700 1 -- 172.21.15.27:0/819986736 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b4011b30 con 0x7ff5c4067fd0 2024-01-11T09:48:05.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.657+0000 7ff5cbbff700 1 -- 172.21.15.27:0/819986736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5c4067fd0 msgr2=0x7ff5c40c7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:05.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.657+0000 7ff5cbbff700 1 --2- 172.21.15.27:0/819986736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5c4067fd0 0x7ff5c40c7010 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4009910 tx=0x7ff5b40091a0 comp rx=0 tx=0).stop 2024-01-11T09:48:05.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.657+0000 7ff5cbbff700 1 -- 172.21.15.27:0/819986736 shutdown_connections 2024-01-11T09:48:05.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.657+0000 7ff5cbbff700 1 --2- 172.21.15.27:0/819986736 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5c4067fd0 0x7ff5c40c7010 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:05.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.657+0000 7ff5cbbff700 1 -- 172.21.15.27:0/819986736 >> 172.21.15.27:0/819986736 conn(0x7ff5c40cd490 msgr2=0x7ff5c40c9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:05.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.657+0000 7ff5cbbff700 1 -- 172.21.15.27:0/819986736 shutdown_connections 2024-01-11T09:48:05.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.657+0000 7ff5cbbff700 1 -- 172.21.15.27:0/819986736 wait complete. 2024-01-11T09:48:05.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.658+0000 7ff5cbbff700 1 Processor -- start 2024-01-11T09:48:05.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.658+0000 7ff5cbbff700 1 -- start start 2024-01-11T09:48:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.659+0000 7ff5cbbff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5c4067fd0 0x7ff5c415c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.659+0000 7ff5cbbff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5c40aedc0 0x7ff5c414d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.659+0000 7ff5cbbff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5c40c1c60 0x7ff5c4169a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.659+0000 7ff5cbbff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c40c70c0 con 0x7ff5c40c1c60 2024-01-11T09:48:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.659+0000 7ff5cbbff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c40c4440 con 0x7ff5c4067fd0 2024-01-11T09:48:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.659+0000 7ff5cbbff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c40c4c80 con 0x7ff5c40aedc0 2024-01-11T09:48:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.660+0000 7ff5ca19c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5c40c1c60 0x7ff5c4169a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.660+0000 7ff5ca19c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5c40c1c60 0x7ff5c4169a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53502/0 (socket says 172.21.15.27:53502) 2024-01-11T09:48:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.660+0000 7ff5ca19c700 1 -- 172.21.15.27:0/633250909 learned_addr learned my addr 172.21.15.27:0/633250909 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.660+0000 7ff5c999b700 1 --2- 172.21.15.27:0/633250909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5c4067fd0 0x7ff5c415c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.660+0000 7ff5c919a700 1 --2- 172.21.15.27:0/633250909 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5c40aedc0 0x7ff5c414d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.660+0000 7ff5ca19c700 1 -- 172.21.15.27:0/633250909 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5c40aedc0 msgr2=0x7ff5c414d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.660+0000 7ff5ca19c700 1 --2- 172.21.15.27:0/633250909 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5c40aedc0 0x7ff5c414d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.660+0000 7ff5ca19c700 1 -- 172.21.15.27:0/633250909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5c4067fd0 msgr2=0x7ff5c415c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.660+0000 7ff5ca19c700 1 --2- 172.21.15.27:0/633250909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5c4067fd0 0x7ff5c415c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.660+0000 7ff5ca19c700 1 -- 172.21.15.27:0/633250909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b4009420 con 0x7ff5c40c1c60 2024-01-11T09:48:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.660+0000 7ff5c999b700 1 --2- 172.21.15.27:0/633250909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5c4067fd0 0x7ff5c415c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:48:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.660+0000 7ff5c919a700 1 --2- 172.21.15.27:0/633250909 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7ff5c40aedc0 0x7ff5c414d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:48:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.660+0000 7ff5ca19c700 1 --2- 172.21.15.27:0/633250909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5c40c1c60 0x7ff5c4169a30 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00c950 tx=0x7ff5bc007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.661+0000 7ff5b9ffb700 1 -- 172.21.15.27:0/633250909 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc004dc0 con 0x7ff5c40c1c60 2024-01-11T09:48:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.661+0000 7ff5b9ffb700 1 -- 172.21.15.27:0/633250909 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5bc00a5d0 con 0x7ff5c40c1c60 2024-01-11T09:48:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.661+0000 7ff5cbbff700 1 -- 172.21.15.27:0/633250909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c415cdb0 con 0x7ff5c40c1c60 2024-01-11T09:48:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.661+0000 7ff5b9ffb700 1 -- 172.21.15.27:0/633250909 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc00a990 con 0x7ff5c40c1c60 2024-01-11T09:48:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.661+0000 7ff5cbbff700 1 -- 172.21.15.27:0/633250909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c40c4630 con 0x7ff5c40c1c60 2024-01-11T09:48:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.662+0000 7ff5cbbff700 1 -- 172.21.15.27:0/633250909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff5c404c3c0 con 0x7ff5c40c1c60 2024-01-11T09:48:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.663+0000 7ff5b9ffb700 1 -- 172.21.15.27:0/633250909 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7ff5bc00ad40 con 0x7ff5c40c1c60 2024-01-11T09:48:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.664+0000 7ff5b9ffb700 1 --2- 172.21.15.27:0/633250909 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff5a8066f30 0x7ff5a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.664+0000 7ff5b9ffb700 1 -- 172.21.15.27:0/633250909 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7ff5bc083e90 con 0x7ff5c40c1c60 2024-01-11T09:48:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.664+0000 7ff5b9ffb700 1 --2- 172.21.15.27:0/633250909 >> [v2:172.21.15.77:6824/42140,v1:172.21.15.77:6825/42140] conn(0x7ff5a806d4c0 0x7ff5a806f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.664+0000 7ff5b9ffb700 1 -- 172.21.15.27:0/633250909 --> [v2:172.21.15.77:6824/42140,v1:172.21.15.77:6825/42140] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff5a8070000 con 0x7ff5a806d4c0 2024-01-11T09:48:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.664+0000 7ff5b9ffb700 1 -- 172.21.15.27:0/633250909 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=261) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5bc086470 con 0x7ff5c40c1c60 2024-01-11T09:48:05.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.664+0000 7ff5c999b700 1 --2- 172.21.15.27:0/633250909 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff5a8066f30 0x7ff5a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:05.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.665+0000 7ff5c919a700 1 --2- 172.21.15.27:0/633250909 >> [v2:172.21.15.77:6824/42140,v1:172.21.15.77:6825/42140] conn(0x7ff5a806d4c0 0x7ff5a806f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:05.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.665+0000 7ff5c999b700 1 --2- 172.21.15.27:0/633250909 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff5a8066f30 0x7ff5a80693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0003990 tx=0x7ff5c0009040 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:05.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.665+0000 7ff5c919a700 1 --2- 172.21.15.27:0/633250909 >> [v2:172.21.15.77:6824/42140,v1:172.21.15.77:6825/42140] conn(0x7ff5a806d4c0 0x7ff5a806f930 crc :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:05.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.666+0000 7ff5b9ffb700 1 -- 172.21.15.27:0/633250909 <== osd.7 v2:172.21.15.77:6824/42140 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7ff5a8070000 con 0x7ff5a806d4c0 2024-01-11T09:48:05.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.681+0000 7ff5cbbff700 1 -- 172.21.15.27:0/633250909 --> [v2:172.21.15.77:6824/42140,v1:172.21.15.77:6825/42140] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff5c4063810 con 0x7ff5a806d4c0 2024-01-11T09:48:05.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.681+0000 7ff5b9ffb700 1 -- 172.21.15.27:0/633250909 <== osd.7 v2:172.21.15.77:6824/42140 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7ff5c4063810 con 0x7ff5a806d4c0 2024-01-11T09:48:05.673 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-11T09:48:05.673 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-11T09:48:05.673 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-11T09:48:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.682+0000 7ff5cbbff700 1 -- 172.21.15.27:0/633250909 >> [v2:172.21.15.77:6824/42140,v1:172.21.15.77:6825/42140] conn(0x7ff5a806d4c0 msgr2=0x7ff5a806f930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.682+0000 7ff5cbbff700 1 --2- 172.21.15.27:0/633250909 >> [v2:172.21.15.77:6824/42140,v1:172.21.15.77:6825/42140] conn(0x7ff5a806d4c0 0x7ff5a806f930 crc :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.682+0000 7ff5cbbff700 1 -- 172.21.15.27:0/633250909 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff5a8066f30 msgr2=0x7ff5a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.682+0000 7ff5cbbff700 1 --2- 172.21.15.27:0/633250909 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7ff5a8066f30 0x7ff5a80693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0003990 tx=0x7ff5c0009040 comp rx=0 tx=0).stop 2024-01-11T09:48:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.682+0000 7ff5cbbff700 1 -- 172.21.15.27:0/633250909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5c40c1c60 msgr2=0x7ff5c4169a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.682+0000 7ff5cbbff700 1 --2- 172.21.15.27:0/633250909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5c40c1c60 0x7ff5c4169a30 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00c950 tx=0x7ff5bc007ed0 comp rx=0 tx=0).stop 2024-01-11T09:48:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.682+0000 7ff5ca19c700 1 -- 172.21.15.27:0/633250909 reap_dead start 2024-01-11T09:48:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.682+0000 7ff5cbbff700 1 -- 172.21.15.27:0/633250909 shutdown_connections 2024-01-11T09:48:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.682+0000 7ff5cbbff700 1 -- 172.21.15.27:0/633250909 >> 172.21.15.27:0/633250909 conn(0x7ff5c40cd490 msgr2=0x7ff5c41596a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.682+0000 7ff5cbbff700 1 -- 172.21.15.27:0/633250909 shutdown_connections 2024-01-11T09:48:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.682+0000 7ff5cbbff700 1 -- 172.21.15.27:0/633250909 wait complete. 2024-01-11T09:48:05.690 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-01-11T09:48:05.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.900+0000 7f31df9ff700 1 Processor -- start 2024-01-11T09:48:05.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.900+0000 7f31df9ff700 1 -- start start 2024-01-11T09:48:05.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.901+0000 7f31df9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31d815ca10 0x7f31d815a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:05.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.901+0000 7f31df9ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d80b4bd0 con 0x7f31d8068cd0 2024-01-11T09:48:05.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.901+0000 7f31df9ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d815ac70 con 0x7f31d815cde0 2024-01-11T09:48:05.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.901+0000 7f31df9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d80b3fc0 con 0x7f31d815ca10 2024-01-11T09:48:05.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.901+0000 7f31ddf9c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f31d8068cd0 0x7f31d80b5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41574/0 (socket says 172.21.15.27:41574) 2024-01-11T09:48:05.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.901+0000 7f31ddf9c700 1 -- 172.21.15.27:0/2927266814 learned_addr learned my addr 172.21.15.27:0/2927266814 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:48:05.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.902+0000 7f31dd79b700 1 --2- 172.21.15.27:0/2927266814 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31d815ca10 0x7f31d815a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:05.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.902+0000 7f31cffff700 1 -- 172.21.15.27:0/2927266814 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858563335 0 0) 0x7f31d80b4bd0 con 0x7f31d8068cd0 2024-01-11T09:48:05.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.902+0000 7f31cffff700 1 -- 172.21.15.27:0/2927266814 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31c4003570 con 0x7f31d8068cd0 2024-01-11T09:48:05.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.902+0000 7f31cffff700 1 -- 172.21.15.27:0/2927266814 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 76449553 0 0) 0x7f31d815ac70 con 0x7f31d815cde0 2024-01-11T09:48:05.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.902+0000 7f31cffff700 1 -- 172.21.15.27:0/2927266814 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31d80b4bd0 con 0x7f31d815cde0 2024-01-11T09:48:05.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.902+0000 7f31dd79b700 1 -- 172.21.15.27:0/2927266814 >> v1:172.21.15.77:6789/0 conn(0x7f31d815cde0 legacy=0x7f31d80685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:05.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.902+0000 7f31dd79b700 1 -- 172.21.15.27:0/2927266814 >> v1:172.21.15.27:6789/0 conn(0x7f31d8068cd0 legacy=0x7f31d80b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:05.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.902+0000 7f31dd79b700 1 -- 172.21.15.27:0/2927266814 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d80b42a0 con 0x7f31d815ca10 2024-01-11T09:48:05.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.902+0000 7f31dd79b700 1 --2- 172.21.15.27:0/2927266814 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31d815ca10 0x7f31d815a730 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009910 tx=0x7f31d40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7046df26b8aef00a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:05.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.903+0000 7f31cffff700 1 -- 172.21.15.27:0/2927266814 <== mon.2 v2:172.21.15.77:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d4011630 con 0x7f31d815ca10 2024-01-11T09:48:05.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.903+0000 7f31cffff700 1 -- 172.21.15.27:0/2927266814 <== mon.2 v2:172.21.15.77:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d4011790 con 0x7f31d815ca10 2024-01-11T09:48:05.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.903+0000 7f31cffff700 1 -- 172.21.15.27:0/2927266814 <== mon.2 v2:172.21.15.77:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d4011b30 con 0x7f31d815ca10 2024-01-11T09:48:05.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.903+0000 7f31df9ff700 1 -- 172.21.15.27:0/2927266814 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31d815ca10 msgr2=0x7f31d815a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:05.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.903+0000 7f31df9ff700 1 --2- 172.21.15.27:0/2927266814 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31d815ca10 0x7f31d815a730 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009910 tx=0x7f31d40091a0 comp rx=0 tx=0).stop 2024-01-11T09:48:05.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.903+0000 7f31df9ff700 1 -- 172.21.15.27:0/2927266814 shutdown_connections 2024-01-11T09:48:05.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.903+0000 7f31df9ff700 1 --2- 172.21.15.27:0/2927266814 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31d815ca10 0x7f31d815a730 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:05.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.903+0000 7f31df9ff700 1 -- 172.21.15.27:0/2927266814 >> 172.21.15.27:0/2927266814 conn(0x7f31d806ed40 msgr2=0x7f31d806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:05.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.904+0000 7f31df9ff700 1 -- 172.21.15.27:0/2927266814 shutdown_connections 2024-01-11T09:48:05.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.904+0000 7f31df9ff700 1 -- 172.21.15.27:0/2927266814 wait complete. 2024-01-11T09:48:05.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.905+0000 7f31df9ff700 1 Processor -- start 2024-01-11T09:48:05.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.905+0000 7f31df9ff700 1 -- start start 2024-01-11T09:48:05.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.906+0000 7f31df9ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31d8068cd0 0x7f31d8150a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:05.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.906+0000 7f31df9ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d815ca10 0x7f31d8150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:05.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.906+0000 7f31df9ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31d815cde0 0x7f31d8156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:05.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.906+0000 7f31df9ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d81521e0 con 0x7f31d815ca10 2024-01-11T09:48:05.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.906+0000 7f31df9ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d8169ef0 con 0x7f31d8068cd0 2024-01-11T09:48:05.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.906+0000 7f31df9ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d81515d0 con 0x7f31d815cde0 2024-01-11T09:48:05.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.906+0000 7f31dcf9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d815ca10 0x7f31d8150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:05.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.906+0000 7f31dcf9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d815ca10 0x7f31d8150fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53516/0 (socket says 172.21.15.27:53516) 2024-01-11T09:48:05.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.906+0000 7f31dcf9a700 1 -- 172.21.15.27:0/3456732424 learned_addr learned my addr 172.21.15.27:0/3456732424 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:05.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.907+0000 7f31ddf9c700 1 --2- 172.21.15.27:0/3456732424 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31d815cde0 0x7f31d8156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:05.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.907+0000 7f31dd79b700 1 --2- 172.21.15.27:0/3456732424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31d8068cd0 0x7f31d8150a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:05.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.907+0000 7f31dcf9a700 1 -- 172.21.15.27:0/3456732424 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31d815cde0 msgr2=0x7f31d8156700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:05.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.907+0000 7f31dcf9a700 1 --2- 172.21.15.27:0/3456732424 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31d815cde0 0x7f31d8156700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:05.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.907+0000 7f31dcf9a700 1 -- 172.21.15.27:0/3456732424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31d8068cd0 msgr2=0x7f31d8150a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:05.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.907+0000 7f31dcf9a700 1 --2- 172.21.15.27:0/3456732424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31d8068cd0 0x7f31d8150a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:05.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.907+0000 7f31dcf9a700 1 -- 172.21.15.27:0/3456732424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d4009420 con 0x7f31d815ca10 2024-01-11T09:48:05.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.907+0000 7f31dd79b700 1 --2- 172.21.15.27:0/3456732424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31d8068cd0 0x7f31d8150a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:48:05.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.907+0000 7f31dcf9a700 1 --2- 172.21.15.27:0/3456732424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d815ca10 0x7f31d8150fa0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f31c800c580 tx=0x7f31c8007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:05.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.907+0000 7f31cd7fa700 1 -- 172.21.15.27:0/3456732424 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c8010510 con 0x7f31d815ca10 2024-01-11T09:48:05.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.907+0000 7f31cd7fa700 1 -- 172.21.15.27:0/3456732424 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c8010670 con 0x7f31d815ca10 2024-01-11T09:48:05.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.907+0000 7f31cd7fa700 1 -- 172.21.15.27:0/3456732424 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c8010a30 con 0x7f31d815ca10 2024-01-11T09:48:05.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.908+0000 7f31df9ff700 1 -- 172.21.15.27:0/3456732424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d81518b0 con 0x7f31d815ca10 2024-01-11T09:48:05.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.908+0000 7f31df9ff700 1 -- 172.21.15.27:0/3456732424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d8151e70 con 0x7f31d815ca10 2024-01-11T09:48:05.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.909+0000 7f31df9ff700 1 -- 172.21.15.27:0/3456732424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d804c3c0 con 0x7f31d815ca10 2024-01-11T09:48:05.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.910+0000 7f31cd7fa700 1 -- 172.21.15.27:0/3456732424 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f31c801b400 con 0x7f31d815ca10 2024-01-11T09:48:05.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.911+0000 7f31cd7fa700 1 --2- 172.21.15.27:0/3456732424 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f31bc066f30 0x7f31bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:05.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.911+0000 7f31cd7fa700 1 -- 172.21.15.27:0/3456732424 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f31c8084270 con 0x7f31d815ca10 2024-01-11T09:48:05.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.911+0000 7f31dd79b700 1 --2- 172.21.15.27:0/3456732424 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f31bc066f30 0x7f31bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:05.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.912+0000 7f31dd79b700 1 --2- 172.21.15.27:0/3456732424 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f31bc066f30 0x7f31bc0693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f31d40091e0 tx=0x7f31d4017000 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:05.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:05.915+0000 7f31cd7fa700 1 -- 172.21.15.27:0/3456732424 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f31c8051490 con 0x7f31d815ca10 2024-01-11T09:48:06.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.115+0000 7f31df9ff700 1 -- 172.21.15.27:0/3456732424 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f31d8152320 con 0x7f31bc066f30 2024-01-11T09:48:06.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.116+0000 7f31cd7fa700 1 -- 172.21.15.27:0/3456732424 <== mgr.11878 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f31d8152320 con 0x7f31bc066f30 2024-01-11T09:48:06.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.119+0000 7f31df9ff700 1 -- 172.21.15.27:0/3456732424 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f31bc066f30 msgr2=0x7f31bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:06.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.119+0000 7f31df9ff700 1 --2- 172.21.15.27:0/3456732424 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f31bc066f30 0x7f31bc0693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f31d40091e0 tx=0x7f31d4017000 comp rx=0 tx=0).stop 2024-01-11T09:48:06.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.119+0000 7f31df9ff700 1 -- 172.21.15.27:0/3456732424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d815ca10 msgr2=0x7f31d8150fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:06.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.119+0000 7f31df9ff700 1 --2- 172.21.15.27:0/3456732424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d815ca10 0x7f31d8150fa0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f31c800c580 tx=0x7f31c8007e10 comp rx=0 tx=0).stop 2024-01-11T09:48:06.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.119+0000 7f31df9ff700 1 -- 172.21.15.27:0/3456732424 shutdown_connections 2024-01-11T09:48:06.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.119+0000 7f31df9ff700 1 --2- 172.21.15.27:0/3456732424 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f31d815cde0 0x7f31d8156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:06.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.119+0000 7f31df9ff700 1 --2- 172.21.15.27:0/3456732424 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f31bc066f30 0x7f31bc0693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:06.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.119+0000 7f31df9ff700 1 --2- 172.21.15.27:0/3456732424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31d8068cd0 0x7f31d8150a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:06.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.119+0000 7f31df9ff700 1 --2- 172.21.15.27:0/3456732424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d815ca10 0x7f31d8150fa0 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:06.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.119+0000 7f31df9ff700 1 -- 172.21.15.27:0/3456732424 >> 172.21.15.27:0/3456732424 conn(0x7f31d806ed40 msgr2=0x7f31d8150170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:06.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.119+0000 7f31df9ff700 1 -- 172.21.15.27:0/3456732424 shutdown_connections 2024-01-11T09:48:06.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.119+0000 7f31df9ff700 1 -- 172.21.15.27:0/3456732424 wait complete. 2024-01-11T09:48:06.112 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 7 to deep-scrub 2024-01-11T09:48:06.128 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:48:06.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.336+0000 7f4a07fff700 1 Processor -- start 2024-01-11T09:48:06.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.337+0000 7f4a07fff700 1 -- start start 2024-01-11T09:48:06.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.338+0000 7f4a07fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4a001575e0 0x7f4a00158620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:06.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.338+0000 7f4a07fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4a000b19d0 0x7f4a00159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:06.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.338+0000 7f4a07fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a00158b60 con 0x7f4a0006b610 2024-01-11T09:48:06.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.338+0000 7f4a07fff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a000cb5e0 con 0x7f4a000b19d0 2024-01-11T09:48:06.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.338+0000 7f4a07fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a001599b0 con 0x7f4a001575e0 2024-01-11T09:48:06.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.338+0000 7f4a0559a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4a0006b610 0x7f4a000b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41580/0 (socket says 172.21.15.27:41580) 2024-01-11T09:48:06.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.338+0000 7f4a0559a700 1 -- 172.21.15.27:0/862937890 learned_addr learned my addr 172.21.15.27:0/862937890 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:48:06.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.338+0000 7f4a0659c700 1 --2- 172.21.15.27:0/862937890 >> v2:172.21.15.27:3300/0 conn(0x7f4a000b19d0 0x7f4a00159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:06.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.338+0000 7f4a05d9b700 1 --2- 172.21.15.27:0/862937890 >> v2:172.21.15.77:3300/0 conn(0x7f4a001575e0 0x7f4a00158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:06.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.339+0000 7f4a04d99700 1 -- 172.21.15.27:0/862937890 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175931426 0 0) 0x7f49f0001110 con 0x7f4a0006b610 2024-01-11T09:48:06.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.339+0000 7f4a04d99700 1 -- 172.21.15.27:0/862937890 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49ec003570 con 0x7f4a0006b610 2024-01-11T09:48:06.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.339+0000 7f4a0659c700 1 -- 172.21.15.27:0/862937890 >> v2:172.21.15.77:3300/0 conn(0x7f4a001575e0 msgr2=0x7f4a00158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:06.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.339+0000 7f4a0659c700 1 --2- 172.21.15.27:0/862937890 >> v2:172.21.15.77:3300/0 conn(0x7f4a001575e0 0x7f4a00158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:06.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.339+0000 7f4a0659c700 1 -- 172.21.15.27:0/862937890 >> v1:172.21.15.27:6789/0 conn(0x7f4a0006b610 legacy=0x7f4a000b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:06.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.339+0000 7f4a0659c700 1 -- 172.21.15.27:0/862937890 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a000c0f90 con 0x7f4a000b19d0 2024-01-11T09:48:06.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.339+0000 7f4a0659c700 1 --2- 172.21.15.27:0/862937890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a000b19d0 0x7f4a00159e40 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f4a00064b00 tx=0x7f49f8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed81a4017e69f2b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:06.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.339+0000 7f4a04d99700 1 -- 172.21.15.27:0/862937890 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f800e070 con 0x7f4a000b19d0 2024-01-11T09:48:06.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.339+0000 7f4a04d99700 1 -- 172.21.15.27:0/862937890 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f800bab0 con 0x7f4a000b19d0 2024-01-11T09:48:06.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.339+0000 7f4a04d99700 1 -- 172.21.15.27:0/862937890 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f800be70 con 0x7f4a000b19d0 2024-01-11T09:48:06.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.340+0000 7f4a07fff700 1 -- 172.21.15.27:0/862937890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a000b19d0 msgr2=0x7f4a00159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:06.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.340+0000 7f4a07fff700 1 --2- 172.21.15.27:0/862937890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a000b19d0 0x7f4a00159e40 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f4a00064b00 tx=0x7f49f8009780 comp rx=0 tx=0).stop 2024-01-11T09:48:06.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.340+0000 7f4a07fff700 1 -- 172.21.15.27:0/862937890 shutdown_connections 2024-01-11T09:48:06.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.340+0000 7f4a07fff700 1 --2- 172.21.15.27:0/862937890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a000b19d0 0x7f4a00159e40 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:06.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.340+0000 7f4a07fff700 1 --2- 172.21.15.27:0/862937890 >> v2:172.21.15.77:3300/0 conn(0x7f4a001575e0 0x7f4a00158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:06.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.340+0000 7f4a07fff700 1 -- 172.21.15.27:0/862937890 >> 172.21.15.27:0/862937890 conn(0x7f4a000680e0 msgr2=0x7f4a00069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:06.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.340+0000 7f4a07fff700 1 -- 172.21.15.27:0/862937890 shutdown_connections 2024-01-11T09:48:06.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.340+0000 7f4a07fff700 1 -- 172.21.15.27:0/862937890 wait complete. 2024-01-11T09:48:06.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.342+0000 7f4a07fff700 1 Processor -- start 2024-01-11T09:48:06.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.342+0000 7f4a07fff700 1 -- start start 2024-01-11T09:48:06.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.343+0000 7f4a07fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a0006b610 0x7f4a00154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:06.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.343+0000 7f4a07fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4a000b19d0 0x7f4a00159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.343+0000 7f4a07fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a001575e0 0x7f4a0015a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.343+0000 7f4a07fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0015b650 con 0x7f4a0006b610 2024-01-11T09:48:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.343+0000 7f4a07fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0015ce90 con 0x7f4a001575e0 2024-01-11T09:48:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.343+0000 7f4a07fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a00169910 con 0x7f4a000b19d0 2024-01-11T09:48:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.343+0000 7f4a05d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a0006b610 0x7f4a00154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.343+0000 7f4a05d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a0006b610 0x7f4a00154280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53532/0 (socket says 172.21.15.27:53532) 2024-01-11T09:48:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.343+0000 7f4a05d9b700 1 -- 172.21.15.27:0/1391680844 learned_addr learned my addr 172.21.15.27:0/1391680844 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.343+0000 7f4a05d9b700 1 -- 172.21.15.27:0/1391680844 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4a000b19d0 msgr2=0x7f4a00159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.343+0000 7f4a05d9b700 1 --2- 172.21.15.27:0/1391680844 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4a000b19d0 0x7f4a00159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.344+0000 7f4a05d9b700 1 -- 172.21.15.27:0/1391680844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a001575e0 msgr2=0x7f4a0015a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.344+0000 7f4a05d9b700 1 --2- 172.21.15.27:0/1391680844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a001575e0 0x7f4a0015a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.344+0000 7f4a05d9b700 1 -- 172.21.15.27:0/1391680844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f8009a20 con 0x7f4a0006b610 2024-01-11T09:48:06.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.344+0000 7f4a05d9b700 1 --2- 172.21.15.27:0/1391680844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a0006b610 0x7f4a00154280 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00e960 tx=0x7f49fc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:06.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.344+0000 7f49f5ffb700 1 -- 172.21.15.27:0/1391680844 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc00c850 con 0x7f4a0006b610 2024-01-11T09:48:06.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.344+0000 7f49f5ffb700 1 -- 172.21.15.27:0/1391680844 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49fc00c9b0 con 0x7f4a0006b610 2024-01-11T09:48:06.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.344+0000 7f49f5ffb700 1 -- 172.21.15.27:0/1391680844 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc00cd50 con 0x7f4a0006b610 2024-01-11T09:48:06.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.344+0000 7f4a07fff700 1 -- 172.21.15.27:0/1391680844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a001695e0 con 0x7f4a0006b610 2024-01-11T09:48:06.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.344+0000 7f4a07fff700 1 -- 172.21.15.27:0/1391680844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a0015cbb0 con 0x7f4a0006b610 2024-01-11T09:48:06.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.345+0000 7f4a07fff700 1 -- 172.21.15.27:0/1391680844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a000718d0 con 0x7f4a0006b610 2024-01-11T09:48:06.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.346+0000 7f49f5ffb700 1 -- 172.21.15.27:0/1391680844 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f49fc013680 con 0x7f4a0006b610 2024-01-11T09:48:06.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.347+0000 7f49f5ffb700 1 --2- 172.21.15.27:0/1391680844 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f49e4066f30 0x7f49e40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:06.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.347+0000 7f49f5ffb700 1 -- 172.21.15.27:0/1391680844 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f49fc084af0 con 0x7f4a0006b610 2024-01-11T09:48:06.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.348+0000 7f4a0559a700 1 --2- 172.21.15.27:0/1391680844 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f49e4066f30 0x7f49e40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:06.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.349+0000 7f4a0559a700 1 --2- 172.21.15.27:0/1391680844 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f49e4066f30 0x7f49e40693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f49f0007fd0 tx=0x7f49f00011a0 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:06.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.352+0000 7f49f5ffb700 1 -- 172.21.15.27:0/1391680844 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f49fc051d10 con 0x7f4a0006b610 2024-01-11T09:48:06.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.549+0000 7f4a07fff700 1 -- 172.21.15.27:0/1391680844 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4a00158da0 con 0x7f49e4066f30 2024-01-11T09:48:06.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.552+0000 7f49f5ffb700 1 -- 172.21.15.27:0/1391680844 <== mgr.11878 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50617 (secure 0 0 0) 0x7f4a00158da0 con 0x7f49e4066f30 2024-01-11T09:48:06.544 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:48:06.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.554+0000 7f4a07fff700 1 -- 172.21.15.27:0/1391680844 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f49e4066f30 msgr2=0x7f49e40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:06.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.554+0000 7f4a07fff700 1 --2- 172.21.15.27:0/1391680844 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f49e4066f30 0x7f49e40693e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f49f0007fd0 tx=0x7f49f00011a0 comp rx=0 tx=0).stop 2024-01-11T09:48:06.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.554+0000 7f4a07fff700 1 -- 172.21.15.27:0/1391680844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a0006b610 msgr2=0x7f4a00154280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:06.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.554+0000 7f4a07fff700 1 --2- 172.21.15.27:0/1391680844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a0006b610 0x7f4a00154280 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00e960 tx=0x7f49fc0091a0 comp rx=0 tx=0).stop 2024-01-11T09:48:06.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.554+0000 7f4a07fff700 1 -- 172.21.15.27:0/1391680844 shutdown_connections 2024-01-11T09:48:06.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.555+0000 7f4a07fff700 1 --2- 172.21.15.27:0/1391680844 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f49e4066f30 0x7f49e40693e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:06.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.555+0000 7f4a07fff700 1 --2- 172.21.15.27:0/1391680844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a001575e0 0x7f4a0015a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:06.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.555+0000 7f4a07fff700 1 --2- 172.21.15.27:0/1391680844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a0006b610 0x7f4a00154280 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:06.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.555+0000 7f4a07fff700 1 --2- 172.21.15.27:0/1391680844 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4a000b19d0 0x7f4a00159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:06.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.555+0000 7f4a07fff700 1 -- 172.21.15.27:0/1391680844 >> 172.21.15.27:0/1391680844 conn(0x7f4a000680e0 msgr2=0x7f4a00154990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:06.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.555+0000 7f4a07fff700 1 -- 172.21.15.27:0/1391680844 shutdown_connections 2024-01-11T09:48:06.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:06.555+0000 7f4a07fff700 1 -- 172.21.15.27:0/1391680844 wait complete. 2024-01-11T09:48:06.547 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:48:06.564 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-01-11T09:48:04.567719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2846,"num_read_kb":6029,"num_write":727,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":430,"ondisk_log_size":430,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":2470824,"kb_used_data":26984,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752503896,"statfs":{"total":773094113280,"available":770563989504,"internally_reserved":0,"allocated":27631616,"data_stored":13999136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.009158"},"pg_stats":[{"pgid":"1.0","version":"260'430","reported_seq":1465,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-01-11T09:47:52.627901+0000","last_change":"2024-01-11T08:10:21.917122+0000","last_active":"2024-01-11T09:47:52.627901+0000","last_peered":"2024-01-11T09:47:52.627901+0000","last_clean":"2024-01-11T09:47:52.627901+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:47:52.627901+0000","last_undegraded":"2024-01-11T09:47:52.627901+0000","last_fullsized":"2024-01-11T09:47:52.627901+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T08:10:20.459836+0000","last_clean_scrub_stamp":"2024-01-11T08:10:20.459836+0000","objects_scrubbed":0,"log_size":430,"log_dups_size":0,"ondisk_log_size":430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T16:01:25.005494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2846,"num_read_kb":6029,"num_write":727,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2846,"num_read_kb":6029,"num_write":727,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":430,"ondisk_log_size":430,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575997,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305192,"kb_used_data":3808,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94066648,"statfs":{"total":96636764160,"available":96324247552,"internally_reserved":0,"allocated":3899392,"data_stored":2192668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.66500000000000004,"15min":0.71899999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36599999999999999},"max":{"1min":0.81799999999999995,"5min":0.95699999999999996,"15min":1.0529999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.69299999999999995,"15min":0.73399999999999999},"min":{"1min":0.41699999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.97999999999999998,"5min":1.026,"15min":1.0669999999999999},"last":0.72499999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.71999999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.0209999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69699999999999995,"15min":0.73099999999999998},"min":{"1min":0.44800000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.91800000000000004,"5min":0.95899999999999996,"15min":1.0820000000000001},"last":0.66100000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.69999999999999996,"15min":0.72899999999999998},"min":{"1min":0.48899999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.85399999999999998,"5min":0.93799999999999994,"15min":1.0600000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70099999999999996,"15min":0.74399999999999999},"min":{"1min":0.52200000000000002,"5min":0.44800000000000001,"15min":0.44500000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":1.153},"last":0.76500000000000001}]},{"osd":3,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73499999999999999,"15min":0.76000000000000001},"min":{"1min":0.5,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.238,"5min":1.238,"15min":1.238},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70899999999999996,"15min":0.74199999999999999},"min":{"1min":0.47699999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.89400000000000002,"5min":0.94099999999999995,"15min":1.0880000000000001},"last":0.68999999999999995}]},{"osd":4,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.61899999999999999,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0209999999999999,"5min":1.0860000000000001,"15min":1.107},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.65900000000000003,"15min":0.64500000000000002},"min":{"1min":0.443,"5min":0.437,"15min":0.39700000000000002},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.95699999999999996}]},{"osd":5,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63200000000000001,"15min":0.64200000000000002},"min":{"1min":0.50900000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.90800000000000003},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.70799999999999996,"15min":0.69399999999999995},"min":{"1min":0.495,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.875}]},{"osd":6,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.77700000000000002,"15min":0.78300000000000003},"min":{"1min":0.54900000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0649999999999999,"5min":1.232,"15min":1.232},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.76800000000000002,"15min":0.76700000000000002},"min":{"1min":0.52300000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":1.0980000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.627,"15min":0.69099999999999995},"min":{"1min":0.44,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.88900000000000001,"5min":0.89100000000000001,"15min":0.999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66300000000000003,"15min":0.72099999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92800000000000005,"5min":0.97099999999999997,"15min":1.216},"last":0.54000000000000004}]},{"osd":1,"last update":"Thu Jan 11 09:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.64600000000000002,"15min":0.71799999999999997},"min":{"1min":0.439,"5min":0.42799999999999999,"15min":0.41699999999999998},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.3129999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65000000000000002,"15min":0.71299999999999997},"min":{"1min":0.48499999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.95399999999999996,"5min":0.95999999999999996,"15min":1.0840000000000001},"last":0.83099999999999996}]},{"osd":2,"last update":"Thu Jan 11 09:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.67100000000000004,"15min":0.73199999999999998},"min":{"1min":0.501,"5min":0.44,"15min":0.44},"max":{"1min":0.92500000000000004,"5min":1.0069999999999999,"15min":1.4179999999999999},"last":0.623},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68000000000000005,"15min":0.73699999999999999},"min":{"1min":0.45400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.90400000000000003,"5min":1.796,"15min":1.796},"last":0.60099999999999998}]},{"osd":3,"last update":"Thu Jan 11 09:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.69299999999999995,"15min":0.745},"min":{"1min":0.49399999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":1.014},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.69699999999999995,"15min":0.753},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.92200000000000004,"5min":0.94999999999999996,"15min":1.165},"last":0.73099999999999998}]},{"osd":4,"last update":"Thu Jan 11 09:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59799999999999998,"15min":0.57399999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":1.0309999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.624},"min":{"1min":0.41599999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.998,"5min":0.998,"15min":1.1259999999999999},"last":0.77500000000000002}]},{"osd":5,"last update":"Thu Jan 11 09:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.67700000000000005,"15min":0.67000000000000004},"min":{"1min":0.51500000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.369},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.71799999999999997,"15min":0.71299999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.0960000000000001},"last":0.85699999999999998}]},{"osd":7,"last update":"Thu Jan 11 09:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.73299999999999998,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.47199999999999998,"15min":0.45100000000000001},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.484},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.78500000000000003,"15min":0.77800000000000002},"min":{"1min":0.53500000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.603},"last":0.76400000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.502,"15min":0.51000000000000001},"min":{"1min":0.36299999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.70699999999999996,"5min":0.73399999999999999,"15min":0.93999999999999995},"last":0.441},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.60699999999999998},"min":{"1min":0.41899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73499999999999999,"5min":0.85199999999999998,"15min":0.95099999999999996},"last":0.626}]},{"osd":2,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.47199999999999998,"15min":0.501},"min":{"1min":0.35199999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.63800000000000001,"5min":0.82399999999999995,"15min":0.82699999999999996},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56599999999999995,"15min":0.58399999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.69999999999999996,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.58799999999999997}]},{"osd":3,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.62,"15min":0.64300000000000002},"min":{"1min":0.45500000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.76000000000000001,"5min":0.85199999999999998,"15min":0.96699999999999997},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55500000000000005,"15min":0.57199999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.56599999999999995}]},{"osd":4,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.76400000000000001,"15min":0.80400000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.159,"5min":1.214,"15min":1.2969999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.81799999999999995,"15min":0.84999999999999998},"min":{"1min":0.58699999999999997,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.169,"5min":1.1879999999999999,"15min":1.1890000000000001},"last":0.91400000000000003}]},{"osd":5,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.79100000000000004,"15min":0.81999999999999995},"min":{"1min":0.55600000000000005,"5min":0.443,"15min":0.443},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.78800000000000003,"15min":0.80200000000000005},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.51300000000000001},"max":{"1min":1.1539999999999999,"5min":1.341,"15min":1.341},"last":0.85499999999999998}]},{"osd":6,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.80400000000000005,"15min":0.83799999999999997},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.1830000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.873},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.81799999999999995,"15min":0.84599999999999997},"min":{"1min":0.57299999999999995,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.93000000000000005}]},{"osd":7,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.86599999999999999,"15min":0.88200000000000001},"min":{"1min":0.63900000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.84299999999999997,"15min":0.85399999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.391,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.81999999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575997,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309288,"kb_used_data":3808,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062552,"statfs":{"total":96636764160,"available":96320053248,"internally_reserved":0,"allocated":3899392,"data_stored":2192668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.48799999999999999,"15min":0.51300000000000001},"min":{"1min":0.34899999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.73099999999999998,"5min":0.754,"15min":0.84699999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.56299999999999994,"15min":0.59099999999999997},"min":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.59599999999999997,"5min":0.81599999999999995,"15min":0.874},"last":0.50600000000000001}]},{"osd":2,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.53300000000000003,"15min":0.56200000000000006},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.68200000000000005,"5min":0.83099999999999996,"15min":0.97499999999999998},"last":0.379},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55400000000000005,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.86399999999999999},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51100000000000001,"15min":0.53800000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.95399999999999996},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60699999999999998,"15min":0.64400000000000002},"min":{"1min":0.42499999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.87,"5min":0.90300000000000002,"15min":1.0389999999999999},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.78600000000000003,"15min":0.81100000000000005},"min":{"1min":0.46100000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.304},"last":1.091},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.78500000000000003,"15min":0.80100000000000005},"min":{"1min":0.48799999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.95399999999999996,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":1.1160000000000001}]},{"osd":5,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.80800000000000005,"15min":0.83599999999999997},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.97599999999999998,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.81399999999999995,"15min":0.83699999999999997},"min":{"1min":0.60399999999999998,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.02,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.77300000000000002}]},{"osd":6,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.79700000000000004,"15min":0.82099999999999995},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.95899999999999996,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.82299999999999995,"15min":0.85999999999999999},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":0.97699999999999998,"5min":1.1699999999999999,"15min":1.256},"last":1.048}]},{"osd":7,"last update":"Thu Jan 11 09:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.83699999999999997,"15min":0.86299999999999999},"min":{"1min":0.64100000000000001,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1000000000000001,"5min":1.226,"15min":1.28},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.84499999999999997,"15min":0.86899999999999999},"min":{"1min":0.63,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0369999999999999,"5min":1.238,"15min":1.2410000000000001},"last":1.173}]}]},{"osd":2,"up_from":13,"seq":55834575996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":300516,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071324,"statfs":{"total":96636764160,"available":96329035776,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.46999999999999997,"15min":0.5},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.27900000000000003},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.85399999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.53400000000000003,"15min":0.56000000000000005},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66900000000000004,"5min":0.77300000000000002,"15min":1.6359999999999999},"last":0.49299999999999999}]},{"osd":1,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56499999999999995,"15min":0.57999999999999996},"min":{"1min":0.40000000000000002,"5min":0.38200000000000001,"15min":0.33500000000000002},"max":{"1min":0.70099999999999996,"5min":0.88800000000000001,"15min":0.93000000000000005},"last":0.433},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.56399999999999995,"15min":0.59299999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.64800000000000002,"5min":0.97299999999999998,"15min":1.145},"last":0.54700000000000004}]},{"osd":3,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52900000000000003,"15min":0.55000000000000004},"min":{"1min":0.376,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81499999999999995,"5min":0.83099999999999996,"15min":0.96399999999999997},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.61199999999999999,"15min":0.65000000000000002},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.75600000000000001,"5min":0.876,"15min":1.3620000000000001},"last":0.61199999999999999}]},{"osd":4,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.75700000000000001,"15min":0.79100000000000004},"min":{"1min":0.56299999999999994,"5min":0.55800000000000005,"15min":0.54300000000000004},"max":{"1min":0.96499999999999997,"5min":1.071,"15min":1.1519999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.76400000000000001,"15min":0.79300000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0209999999999999,"5min":1.161,"15min":1.161},"last":0.56599999999999995}]},{"osd":5,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.79900000000000004,"15min":0.85599999999999998},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.94099999999999995,"5min":1.228,"15min":1.429},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.82799999999999996,"15min":0.85299999999999998},"min":{"1min":0.61499999999999999,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":0.94699999999999995,"5min":1.1850000000000001,"15min":1.2170000000000001},"last":0.69599999999999995}]},{"osd":6,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.77600000000000002,"15min":0.81799999999999995},"min":{"1min":0.56000000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.96499999999999997,"5min":1.1659999999999999,"15min":1.4139999999999999},"last":0.624},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.80400000000000005,"15min":0.85599999999999998},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.94999999999999996,"5min":1.2010000000000001,"15min":1.593},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 11 09:46:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.84399999999999997,"15min":0.88400000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.57299999999999995},"max":{"1min":1.004,"5min":1.1339999999999999,"15min":2.0539999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.83499999999999996,"15min":0.872},"min":{"1min":0.624,"5min":0.60799999999999998,"15min":0.55500000000000005},"max":{"1min":1.026,"5min":1.2270000000000001,"15min":1.2490000000000001},"last":0.66700000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":304612,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067228,"statfs":{"total":96636764160,"available":96324841472,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.47699999999999998,"15min":0.50900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.61699999999999999,"5min":0.89700000000000002,"15min":0.90000000000000002},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.52200000000000002,"15min":0.57499999999999996},"min":{"1min":0.35799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69999999999999996,"5min":1.448,"15min":1.448},"last":0.47999999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46800000000000003,"15min":0.503},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73199999999999998,"5min":0.90900000000000003,"15min":1.304},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.55500000000000005,"15min":0.59599999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.71499999999999997,"5min":0.84599999999999997,"15min":1.4590000000000001},"last":0.41999999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.54600000000000004,"15min":0.59499999999999997},"min":{"1min":0.34899999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.78600000000000003,"5min":1.054,"15min":1.054},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.61399999999999999,"15min":0.64900000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.76100000000000001,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.497}]},{"osd":4,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72299999999999998,"15min":0.77600000000000002},"min":{"1min":0.52600000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.93400000000000005,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.746,"15min":0.79600000000000004},"min":{"1min":0.51800000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.88800000000000001,"5min":1.171,"15min":1.1830000000000001},"last":0.71499999999999997}]},{"osd":5,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.755,"15min":0.80600000000000005},"min":{"1min":0.497,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.02,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78500000000000003,"15min":0.83399999999999996},"min":{"1min":0.55600000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.054,"5min":1.2669999999999999,"15min":1.3440000000000001},"last":0.79900000000000004}]},{"osd":6,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.79900000000000004,"15min":0.84299999999999997},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.97799999999999998,"5min":1.181,"15min":1.2170000000000001},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.77800000000000002,"15min":0.81799999999999995},"min":{"1min":0.59599999999999997,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.81100000000000005,"5min":1.883,"15min":1.883},"last":0.753}]},{"osd":7,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.79700000000000004,"15min":0.85399999999999998},"min":{"1min":0.51600000000000001,"5min":0.505,"15min":0.505},"max":{"1min":0.89600000000000002,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.746,"5min":0.81200000000000006,"15min":0.876},"min":{"1min":0.59899999999999998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.97599999999999998,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.78200000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575997,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.68700000000000006,"15min":0.748},"min":{"1min":0.47299999999999998,"5min":0.42099999999999999,"15min":0.39900000000000002},"max":{"1min":0.86899999999999999,"5min":0.91200000000000003,"15min":1.0760000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.70699999999999996,"15min":0.75700000000000001},"min":{"1min":0.52800000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84599999999999997,"5min":1.853,"15min":1.853},"last":0.55300000000000005}]},{"osd":1,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.67600000000000005,"15min":0.72599999999999998},"min":{"1min":0.42299999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.91500000000000004,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.67000000000000004,"15min":0.71999999999999997},"min":{"1min":0.44,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.93100000000000005,"5min":1.639,"15min":1.639},"last":0.57899999999999996}]},{"osd":2,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.68600000000000005,"15min":0.73599999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.44700000000000001},"max":{"1min":0.94099999999999995,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.623},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.68700000000000006,"15min":0.73199999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.014,"5min":1.014,"15min":1.0449999999999999},"last":0.53700000000000003}]},{"osd":3,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.72499999999999998},"min":{"1min":0.47799999999999998,"5min":0.436,"15min":0.436},"max":{"1min":0.91900000000000004,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73699999999999999,"15min":0.76800000000000002},"min":{"1min":0.59799999999999998,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.98899999999999999,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.498}]},{"osd":5,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.629,"15min":0.624},"min":{"1min":0.35299999999999998,"5min":0.311,"15min":0.311},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.439},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.66300000000000003,"15min":0.64500000000000002},"min":{"1min":0.47499999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0580000000000001,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.59999999999999998}]},{"osd":6,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68999999999999995,"15min":0.67200000000000004},"min":{"1min":0.441,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0680000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.72399999999999998,"15min":0.70499999999999996},"min":{"1min":0.39400000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.1899999999999999,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.68300000000000005}]},{"osd":7,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.72799999999999998,"15min":0.70199999999999996},"min":{"1min":0.44700000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0409999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.78200000000000003,"15min":0.755},"min":{"1min":0.51300000000000001,"5min":0.436,"15min":0.41699999999999998},"max":{"1min":1.109,"5min":2.052,"15min":2.052},"last":0.69599999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575997,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.71399999999999997,"15min":0.72999999999999998},"min":{"1min":0.42899999999999999,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.72599999999999998,"15min":0.72999999999999998},"min":{"1min":0.46400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.9900000000000002,"5min":2.9900000000000002,"15min":2.9900000000000002},"last":0.68799999999999994}]},{"osd":1,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.71799999999999997,"15min":0.73499999999999999},"min":{"1min":0.46000000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.873,"5min":0.75600000000000001,"15min":0.748},"min":{"1min":0.47699999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":5.4039999999999999,"5min":5.4039999999999999,"15min":5.4039999999999999},"last":0.621}]},{"osd":2,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.54300000000000004,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.7120000000000002,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.73999999999999999,"15min":0.74299999999999999},"min":{"1min":0.51700000000000002,"5min":0.439,"15min":0.439},"max":{"1min":2.5099999999999998,"5min":2.5099999999999998,"15min":2.5099999999999998},"last":0.70999999999999996}]},{"osd":3,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.745,"15min":0.75700000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.7799999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.79900000000000004,"15min":0.78300000000000003},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":5.5570000000000004,"5min":5.5570000000000004,"15min":5.5570000000000004},"last":0.754}]},{"osd":4,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.64600000000000002,"15min":0.61699999999999999},"min":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41699999999999998},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.66500000000000004,"15min":0.63800000000000001},"min":{"1min":0.40400000000000003,"5min":0.40200000000000002,"15min":0.375},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":0.77000000000000002}]},{"osd":6,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.67400000000000004,"15min":0.65000000000000002},"min":{"1min":0.42899999999999999,"5min":0.42099999999999999,"15min":0.40999999999999998},"max":{"1min":2.9500000000000002,"5min":2.9500000000000002,"15min":2.9500000000000002},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.6280000000000001,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":0.82299999999999995}]},{"osd":7,"last update":"Thu Jan 11 09:47:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.748,"15min":0.73199999999999998},"min":{"1min":0.52700000000000002,"5min":0.47299999999999998,"15min":0.434},"max":{"1min":2.399,"5min":2.399,"15min":2.399},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.80400000000000005,"15min":0.78200000000000003},"min":{"1min":0.61399999999999999,"5min":0.48799999999999999,"15min":0.47599999999999998},"max":{"1min":2.7570000000000001,"5min":2.7570000000000001,"15min":2.7570000000000001},"last":0.84499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:48:06.568 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-11T08:10:20.459836+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=8, tm_min=10, tm_sec=20, tm_wday=3, tm_yday=11, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=11, tm_hour=9, tm_min=47, tm_sec=59, tm_wday=3, tm_yday=11, tm_isdst=0) 2024-01-11T09:48:06.569 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-11T09:48:19.239 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:48:19.244 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:48:26.569 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T09:48:26.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.788+0000 7f70951ff700 1 Processor -- start 2024-01-11T09:48:26.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.789+0000 7f70951ff700 1 -- start start 2024-01-11T09:48:26.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.790+0000 7f70951ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f70900ca3e0 0x7f70900cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:26.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.790+0000 7f70951ff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f70900ca7b0 0x7f7090159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:26.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.790+0000 7f70951ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f709015d640 con 0x7f70900c8a20 2024-01-11T09:48:26.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.790+0000 7f70951ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f709015ae70 con 0x7f70900ca3e0 2024-01-11T09:48:26.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.790+0000 7f70951ff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f709015a3c0 con 0x7f70900ca7b0 2024-01-11T09:48:26.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.790+0000 7f708f59e700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f70900c8a20 0x7f709015dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51628/0 (socket says 172.21.15.27:51628) 2024-01-11T09:48:26.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.790+0000 7f708f59e700 1 -- 172.21.15.27:0/148525261 learned_addr learned my addr 172.21.15.27:0/148525261 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:48:26.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.790+0000 7f708ed9d700 1 --2- 172.21.15.27:0/148525261 >> v2:172.21.15.27:3300/0 conn(0x7f70900ca3e0 0x7f70900cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:26.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.791+0000 7f708e59c700 1 --2- 172.21.15.27:0/148525261 >> v2:172.21.15.77:3300/0 conn(0x7f70900ca7b0 0x7f7090159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:26.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.791+0000 7f708ed9d700 1 -- 172.21.15.27:0/148525261 >> v2:172.21.15.77:3300/0 conn(0x7f70900ca7b0 msgr2=0x7f7090159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:26.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.791+0000 7f708ed9d700 1 --2- 172.21.15.27:0/148525261 >> v2:172.21.15.77:3300/0 conn(0x7f70900ca7b0 0x7f7090159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:26.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.791+0000 7f708ed9d700 1 -- 172.21.15.27:0/148525261 >> v1:172.21.15.27:6789/0 conn(0x7f70900c8a20 legacy=0x7f709015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:26.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.791+0000 7f708ed9d700 1 -- 172.21.15.27:0/148525261 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f709015a5d0 con 0x7f70900ca3e0 2024-01-11T09:48:26.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.791+0000 7f708dd9b700 1 -- 172.21.15.27:0/148525261 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1974285288 0 0) 0x7f709015d640 con 0x7f70900c8a20 2024-01-11T09:48:26.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.791+0000 7f708ed9d700 1 --2- 172.21.15.27:0/148525261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70900ca3e0 0x7f70900cca20 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f70800098e0 tx=0x7f70800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=471b39677c3fddf4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:26.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.792+0000 7f708dd9b700 1 -- 172.21.15.27:0/148525261 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7080011630 con 0x7f70900ca3e0 2024-01-11T09:48:26.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.792+0000 7f708dd9b700 1 -- 172.21.15.27:0/148525261 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7080011790 con 0x7f70900ca3e0 2024-01-11T09:48:26.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.792+0000 7f708dd9b700 1 -- 172.21.15.27:0/148525261 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7080011b30 con 0x7f70900ca3e0 2024-01-11T09:48:26.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.792+0000 7f70951ff700 1 -- 172.21.15.27:0/148525261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70900ca3e0 msgr2=0x7f70900cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:26.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.792+0000 7f70951ff700 1 --2- 172.21.15.27:0/148525261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70900ca3e0 0x7f70900cca20 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f70800098e0 tx=0x7f70800091a0 comp rx=0 tx=0).stop 2024-01-11T09:48:26.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.792+0000 7f70951ff700 1 -- 172.21.15.27:0/148525261 shutdown_connections 2024-01-11T09:48:26.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.792+0000 7f70951ff700 1 --2- 172.21.15.27:0/148525261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70900ca3e0 0x7f70900cca20 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:26.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.792+0000 7f70951ff700 1 --2- 172.21.15.27:0/148525261 >> v2:172.21.15.77:3300/0 conn(0x7f70900ca7b0 0x7f7090159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:26.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.792+0000 7f70951ff700 1 -- 172.21.15.27:0/148525261 >> 172.21.15.27:0/148525261 conn(0x7f70900ce3e0 msgr2=0x7f70900ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:26.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.792+0000 7f70951ff700 1 -- 172.21.15.27:0/148525261 shutdown_connections 2024-01-11T09:48:26.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.792+0000 7f70951ff700 1 -- 172.21.15.27:0/148525261 wait complete. 2024-01-11T09:48:26.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.794+0000 7f70951ff700 1 Processor -- start 2024-01-11T09:48:26.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.795+0000 7f70951ff700 1 -- start start 2024-01-11T09:48:26.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.796+0000 7f70951ff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70900c8a20 0x7f70901698b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.796+0000 7f70951ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70900ca3e0 0x7f70900b2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.796+0000 7f70951ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70900ca7b0 0x7f7090151c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.796+0000 7f70951ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7090169df0 con 0x7f70900ca3e0 2024-01-11T09:48:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.796+0000 7f70951ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7090148800 con 0x7f70900ca7b0 2024-01-11T09:48:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.796+0000 7f70951ff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70900b2a60 con 0x7f70900c8a20 2024-01-11T09:48:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.796+0000 7f708e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70900ca3e0 0x7f70900b2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.796+0000 7f708e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70900ca3e0 0x7f70900b2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47084/0 (socket says 172.21.15.27:47084) 2024-01-11T09:48:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.796+0000 7f708e59c700 1 -- 172.21.15.27:0/291957243 learned_addr learned my addr 172.21.15.27:0/291957243 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.797+0000 7f708ed9d700 1 --2- 172.21.15.27:0/291957243 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70900c8a20 0x7f70901698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.797+0000 7f708f59e700 1 --2- 172.21.15.27:0/291957243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70900ca7b0 0x7f7090151c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.797+0000 7f708e59c700 1 -- 172.21.15.27:0/291957243 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70900c8a20 msgr2=0x7f70901698b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.797+0000 7f708e59c700 1 --2- 172.21.15.27:0/291957243 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70900c8a20 0x7f70901698b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.797+0000 7f708e59c700 1 -- 172.21.15.27:0/291957243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70900ca7b0 msgr2=0x7f7090151c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.797+0000 7f708e59c700 1 --2- 172.21.15.27:0/291957243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70900ca7b0 0x7f7090151c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.797+0000 7f708e59c700 1 -- 172.21.15.27:0/291957243 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7080009420 con 0x7f70900ca3e0 2024-01-11T09:48:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.797+0000 7f708ed9d700 1 --2- 172.21.15.27:0/291957243 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70900c8a20 0x7f70901698b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:48:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.797+0000 7f708f59e700 1 --2- 172.21.15.27:0/291957243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70900ca7b0 0x7f7090151c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-11T09:48:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.797+0000 7f708e59c700 1 --2- 172.21.15.27:0/291957243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70900ca3e0 0x7f70900b2520 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f708400e930 tx=0x7f70840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.797+0000 7f7076ffd700 1 -- 172.21.15.27:0/291957243 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708400eed0 con 0x7f70900ca3e0 2024-01-11T09:48:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.797+0000 7f7076ffd700 1 -- 172.21.15.27:0/291957243 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708400c850 con 0x7f70900ca3e0 2024-01-11T09:48:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.797+0000 7f70951ff700 1 -- 172.21.15.27:0/291957243 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7090148e20 con 0x7f70900ca3e0 2024-01-11T09:48:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.797+0000 7f7076ffd700 1 -- 172.21.15.27:0/291957243 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708400cbf0 con 0x7f70900ca3e0 2024-01-11T09:48:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.797+0000 7f70951ff700 1 -- 172.21.15.27:0/291957243 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70900b2f50 con 0x7f70900ca3e0 2024-01-11T09:48:26.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.798+0000 7f70951ff700 1 -- 172.21.15.27:0/291957243 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7090148fb0 con 0x7f70900ca3e0 2024-01-11T09:48:26.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.799+0000 7f7076ffd700 1 -- 172.21.15.27:0/291957243 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f7084013470 con 0x7f70900ca3e0 2024-01-11T09:48:26.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.800+0000 7f7076ffd700 1 --2- 172.21.15.27:0/291957243 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f707c066fb0 0x7f707c069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:26.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.800+0000 7f7076ffd700 1 -- 172.21.15.27:0/291957243 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f70840847e0 con 0x7f70900ca3e0 2024-01-11T09:48:26.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.801+0000 7f708ed9d700 1 --2- 172.21.15.27:0/291957243 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f707c066fb0 0x7f707c069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:26.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.801+0000 7f708ed9d700 1 --2- 172.21.15.27:0/291957243 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f707c066fb0 0x7f707c069460 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7080003080 tx=0x7f7080017000 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:26.805+0000 7f7076ffd700 1 -- 172.21.15.27:0/291957243 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7084051960 con 0x7f70900ca3e0 2024-01-11T09:48:26.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.007+0000 7f70951ff700 1 -- 172.21.15.27:0/291957243 --> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f70900718d0 con 0x7f707c066fb0 2024-01-11T09:48:27.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.010+0000 7f7076ffd700 1 -- 172.21.15.27:0/291957243 <== mgr.11878 v2:172.21.15.77:6836/59776 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50638 (secure 0 0 0) 0x7f70900718d0 con 0x7f707c066fb0 2024-01-11T09:48:27.002 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:48:27.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.013+0000 7f70951ff700 1 -- 172.21.15.27:0/291957243 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f707c066fb0 msgr2=0x7f707c069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:27.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.013+0000 7f70951ff700 1 --2- 172.21.15.27:0/291957243 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f707c066fb0 0x7f707c069460 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7080003080 tx=0x7f7080017000 comp rx=0 tx=0).stop 2024-01-11T09:48:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.013+0000 7f70951ff700 1 -- 172.21.15.27:0/291957243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70900ca3e0 msgr2=0x7f70900b2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.013+0000 7f70951ff700 1 --2- 172.21.15.27:0/291957243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70900ca3e0 0x7f70900b2520 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f708400e930 tx=0x7f70840091a0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.013+0000 7f70951ff700 1 -- 172.21.15.27:0/291957243 shutdown_connections 2024-01-11T09:48:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.013+0000 7f70951ff700 1 --2- 172.21.15.27:0/291957243 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f707c066fb0 0x7f707c069460 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.013+0000 7f70951ff700 1 --2- 172.21.15.27:0/291957243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70900ca7b0 0x7f7090151c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.013+0000 7f70951ff700 1 --2- 172.21.15.27:0/291957243 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f70900c8a20 0x7f70901698b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.013+0000 7f70951ff700 1 --2- 172.21.15.27:0/291957243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70900ca3e0 0x7f70900b2520 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.013+0000 7f70951ff700 1 -- 172.21.15.27:0/291957243 >> 172.21.15.27:0/291957243 conn(0x7f70900ce3e0 msgr2=0x7f7090068780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.013+0000 7f70951ff700 1 -- 172.21.15.27:0/291957243 shutdown_connections 2024-01-11T09:48:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.013+0000 7f70951ff700 1 -- 172.21.15.27:0/291957243 wait complete. 2024-01-11T09:48:27.006 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-11T09:48:27.021 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-11T09:48:26.572617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2846,"num_read_kb":6029,"num_write":727,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":431,"ondisk_log_size":431,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":2470824,"kb_used_data":26984,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752503896,"statfs":{"total":773094113280,"available":770563989504,"internally_reserved":0,"allocated":27631616,"data_stored":13999136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002750"},"pg_stats":[{"pgid":"1.0","version":"261'431","reported_seq":1477,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-01-11T09:48:08.673969+0000","last_change":"2024-01-11T09:48:08.673936+0000","last_active":"2024-01-11T09:48:08.673969+0000","last_peered":"2024-01-11T09:48:08.673969+0000","last_clean":"2024-01-11T09:48:08.673969+0000","last_became_active":"2024-01-11T08:10:21.915943+0000","last_became_peered":"2024-01-11T08:10:21.915943+0000","last_unstale":"2024-01-11T09:48:08.673969+0000","last_undegraded":"2024-01-11T09:48:08.673969+0000","last_fullsized":"2024-01-11T09:48:08.673969+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'431","last_scrub_stamp":"2024-01-11T09:48:08.673836+0000","last_deep_scrub":"261'431","last_deep_scrub_stamp":"2024-01-11T09:48:08.673836+0000","last_clean_scrub_stamp":"2024-01-11T09:48:08.673836+0000","objects_scrubbed":2,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-12T14:04:04.176392+0000","scrub_duration":0.0058418619999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2846,"num_read_kb":6029,"num_write":727,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2846,"num_read_kb":6029,"num_write":727,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":431,"ondisk_log_size":431,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576002,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305192,"kb_used_data":3808,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94066648,"statfs":{"total":96636764160,"available":96324247552,"internally_reserved":0,"allocated":3899392,"data_stored":2192668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.66500000000000004,"15min":0.71899999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36599999999999999},"max":{"1min":0.81799999999999995,"5min":0.95699999999999996,"15min":1.0529999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.69299999999999995,"15min":0.73399999999999999},"min":{"1min":0.41699999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.97999999999999998,"5min":1.026,"15min":1.0669999999999999},"last":0.68100000000000005}]},{"osd":1,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.71999999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.0209999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69699999999999995,"15min":0.73099999999999998},"min":{"1min":0.44800000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.91800000000000004,"5min":0.95899999999999996,"15min":1.0820000000000001},"last":0.72799999999999998}]},{"osd":2,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.69999999999999996,"15min":0.72899999999999998},"min":{"1min":0.48899999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.85399999999999998,"5min":0.93799999999999994,"15min":1.0600000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70099999999999996,"15min":0.74399999999999999},"min":{"1min":0.52200000000000002,"5min":0.44800000000000001,"15min":0.44500000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":1.153},"last":0.54100000000000004}]},{"osd":3,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73499999999999999,"15min":0.76000000000000001},"min":{"1min":0.5,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.238,"5min":1.238,"15min":1.238},"last":0.747},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70899999999999996,"15min":0.74199999999999999},"min":{"1min":0.47699999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.89400000000000002,"5min":0.94099999999999995,"15min":1.0880000000000001},"last":0.65500000000000003}]},{"osd":4,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.61899999999999999,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0209999999999999,"5min":1.0860000000000001,"15min":1.107},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.65900000000000003,"15min":0.64500000000000002},"min":{"1min":0.443,"5min":0.437,"15min":0.39700000000000002},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.70099999999999996}]},{"osd":5,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63200000000000001,"15min":0.64200000000000002},"min":{"1min":0.50900000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.90800000000000003},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.70799999999999996,"15min":0.69399999999999995},"min":{"1min":0.495,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.60799999999999998}]},{"osd":6,"last update":"Thu Jan 11 09:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.77700000000000002,"15min":0.78300000000000003},"min":{"1min":0.54900000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0649999999999999,"5min":1.232,"15min":1.232},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.76800000000000002,"15min":0.76700000000000002},"min":{"1min":0.52300000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":0.79000000000000004}]}]},{"osd":6,"up_from":13,"seq":55834576001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.627,"15min":0.69099999999999995},"min":{"1min":0.44,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.88900000000000001,"5min":0.89100000000000001,"15min":0.999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66300000000000003,"15min":0.72099999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92800000000000005,"5min":0.97099999999999997,"15min":1.216},"last":0.93899999999999995}]},{"osd":1,"last update":"Thu Jan 11 09:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.64600000000000002,"15min":0.71799999999999997},"min":{"1min":0.439,"5min":0.42799999999999999,"15min":0.41699999999999998},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.3129999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65000000000000002,"15min":0.71299999999999997},"min":{"1min":0.48499999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.95399999999999996,"5min":0.95999999999999996,"15min":1.0840000000000001},"last":0.874}]},{"osd":2,"last update":"Thu Jan 11 09:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.67100000000000004,"15min":0.73199999999999998},"min":{"1min":0.501,"5min":0.44,"15min":0.44},"max":{"1min":0.92500000000000004,"5min":1.0069999999999999,"15min":1.4179999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68000000000000005,"15min":0.73699999999999999},"min":{"1min":0.45400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.90400000000000003,"5min":1.796,"15min":1.796},"last":0.77400000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.69299999999999995,"15min":0.745},"min":{"1min":0.49399999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":1.014},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.69699999999999995,"15min":0.753},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.92200000000000004,"5min":0.94999999999999996,"15min":1.165},"last":0.71699999999999997}]},{"osd":4,"last update":"Thu Jan 11 09:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59799999999999998,"15min":0.57399999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":1.0309999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.624},"min":{"1min":0.41599999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.998,"5min":0.998,"15min":1.1259999999999999},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Jan 11 09:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.67700000000000005,"15min":0.67000000000000004},"min":{"1min":0.51500000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.369},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.71799999999999997,"15min":0.71299999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.0960000000000001},"last":0.92200000000000004}]},{"osd":7,"last update":"Thu Jan 11 09:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.73299999999999998,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.47199999999999998,"15min":0.45100000000000001},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.484},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.78500000000000003,"15min":0.77800000000000002},"min":{"1min":0.53500000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.603},"last":0.83599999999999997}]}]},{"osd":1,"up_from":13,"seq":55834576003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.502,"15min":0.51000000000000001},"min":{"1min":0.36299999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.70699999999999996,"5min":0.73399999999999999,"15min":0.93999999999999995},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.60699999999999998},"min":{"1min":0.41899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73499999999999999,"5min":0.85199999999999998,"15min":0.95099999999999996},"last":0.61499999999999999}]},{"osd":2,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.47199999999999998,"15min":0.501},"min":{"1min":0.35199999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.63800000000000001,"5min":0.82399999999999995,"15min":0.82699999999999996},"last":0.372},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56599999999999995,"15min":0.58399999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.69999999999999996,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.65200000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.62,"15min":0.64300000000000002},"min":{"1min":0.45500000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.76000000000000001,"5min":0.85199999999999998,"15min":0.96699999999999997},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55500000000000005,"15min":0.57199999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.746}]},{"osd":4,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.76400000000000001,"15min":0.80400000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.159,"5min":1.214,"15min":1.2969999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.81799999999999995,"15min":0.84999999999999998},"min":{"1min":0.58699999999999997,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.169,"5min":1.1879999999999999,"15min":1.1890000000000001},"last":0.71899999999999997}]},{"osd":5,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.79100000000000004,"15min":0.81999999999999995},"min":{"1min":0.55600000000000005,"5min":0.443,"15min":0.443},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.78800000000000003,"15min":0.80200000000000005},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.51300000000000001},"max":{"1min":1.1539999999999999,"5min":1.341,"15min":1.341},"last":0.59899999999999998}]},{"osd":6,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.80400000000000005,"15min":0.83799999999999997},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.1830000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.81799999999999995,"15min":0.84599999999999997},"min":{"1min":0.57299999999999995,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.69899999999999995}]},{"osd":7,"last update":"Thu Jan 11 09:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.86599999999999999,"15min":0.88200000000000001},"min":{"1min":0.63900000000000001,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.84299999999999997,"15min":0.85399999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.391,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.67600000000000005}]}]},{"osd":0,"up_from":13,"seq":55834576002,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309288,"kb_used_data":3808,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94062552,"statfs":{"total":96636764160,"available":96320053248,"internally_reserved":0,"allocated":3899392,"data_stored":2192668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 11 09:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48499999999999999,"15min":0.51000000000000001},"min":{"1min":0.311,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.84699999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55000000000000004,"15min":0.58699999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73899999999999999,"5min":0.81599999999999995,"15min":0.874},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Jan 11 09:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52900000000000003,"15min":0.56100000000000005},"min":{"1min":0.35799999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.75700000000000001,"5min":0.79900000000000004,"15min":0.97499999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.497,"5min":0.53100000000000003,"15min":0.56899999999999995},"min":{"1min":0.38300000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70299999999999996,"5min":0.84699999999999998,"15min":0.86399999999999999},"last":0.46300000000000002}]},{"osd":3,"last update":"Thu Jan 11 09:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.50900000000000001,"15min":0.53600000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.747,"5min":0.91100000000000003,"15min":0.95399999999999996},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.64000000000000001},"min":{"1min":0.44900000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.82299999999999995,"5min":0.87,"15min":1.0389999999999999},"last":0.502}]},{"osd":4,"last update":"Thu Jan 11 09:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.77700000000000002,"15min":0.81100000000000005},"min":{"1min":0.497,"5min":0.439,"15min":0.439},"max":{"1min":1.091,"5min":1.1399999999999999,"15min":1.224},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.77800000000000002,"15min":0.80400000000000005},"min":{"1min":0.57899999999999996,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.1160000000000001,"5min":1.272,"15min":1.3129999999999999},"last":0.76800000000000002}]},{"osd":5,"last update":"Thu Jan 11 09:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.79600000000000004,"15min":0.83199999999999996},"min":{"1min":0.54900000000000004,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.038,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.80100000000000005,"15min":0.83499999999999996},"min":{"1min":0.56999999999999995,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.139,"5min":1.208,"15min":1.2150000000000001},"last":0.84399999999999997}]},{"osd":6,"last update":"Thu Jan 11 09:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.79100000000000004,"15min":0.82099999999999995},"min":{"1min":0.54200000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.042,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.81999999999999995,"15min":0.85799999999999998},"min":{"1min":0.59199999999999997,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.1220000000000001,"5min":1.1699999999999999,"15min":1.256},"last":0.73399999999999999}]},{"osd":7,"last update":"Thu Jan 11 09:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.82099999999999995,"15min":0.85999999999999999},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.99099999999999999,"5min":1.226,"15min":1.28},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.83699999999999997,"15min":0.86899999999999999},"min":{"1min":0.54300000000000004,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.173,"5min":1.173,"15min":1.2410000000000001},"last":0.80400000000000005}]}]},{"osd":2,"up_from":13,"seq":55834576001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":300516,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071324,"statfs":{"total":96636764160,"available":96329035776,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:48:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47199999999999998,"15min":0.499},"min":{"1min":0.33600000000000002,"5min":0.29499999999999998,"15min":0.27900000000000003},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.85399999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52300000000000002,"15min":0.55500000000000005},"min":{"1min":0.317,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":1.6359999999999999},"last":0.72899999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:48:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55900000000000005,"15min":0.57699999999999996},"min":{"1min":0.38600000000000001,"5min":0.38200000000000001,"15min":0.33500000000000002},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54900000000000004,"15min":0.59099999999999997},"min":{"1min":0.36099999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76500000000000001,"5min":0.81599999999999995,"15min":1.145},"last":0.66600000000000004}]},{"osd":3,"last update":"Thu Jan 11 09:48:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51800000000000002,"15min":0.54700000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.59799999999999998,"15min":0.64400000000000002},"min":{"1min":0.439,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85099999999999998,"5min":0.876,"15min":1.3620000000000001},"last":0.82199999999999995}]},{"osd":4,"last update":"Thu Jan 11 09:48:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.72899999999999998,"15min":0.78400000000000003},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.84899999999999998,"5min":0.97199999999999998,"15min":1.1519999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.74299999999999999,"15min":0.78700000000000003},"min":{"1min":0.55700000000000005,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.095,"5min":1.1439999999999999,"15min":1.161},"last":0.99399999999999999}]},{"osd":5,"last update":"Thu Jan 11 09:48:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.77800000000000002,"15min":0.84899999999999998},"min":{"1min":0.54800000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.80200000000000005,"15min":0.84499999999999997},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.90800000000000003,"5min":1.0629999999999999,"15min":1.2170000000000001},"last":0.93999999999999995}]},{"osd":6,"last update":"Thu Jan 11 09:48:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.76000000000000001,"15min":0.81200000000000006},"min":{"1min":0.55400000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.133,"5min":1.133,"15min":1.4139999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.84999999999999998},"min":{"1min":0.55800000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.89700000000000002}]},{"osd":7,"last update":"Thu Jan 11 09:48:02 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.81899999999999995,"15min":0.877},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57299999999999995},"max":{"1min":0.95799999999999996,"5min":1.111,"15min":2.0539999999999998},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.81200000000000006,"15min":0.86299999999999999},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.0649999999999999,"5min":1.1919999999999999,"15min":1.2490000000000001},"last":0.95199999999999996}]}]},{"osd":3,"up_from":13,"seq":55834576001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":304612,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067228,"statfs":{"total":96636764160,"available":96324841472,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.47699999999999998,"15min":0.50900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.61699999999999999,"5min":0.89700000000000002,"15min":0.90000000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.52200000000000002,"15min":0.57499999999999996},"min":{"1min":0.35799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69999999999999996,"5min":1.448,"15min":1.448},"last":0.57999999999999996}]},{"osd":1,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46800000000000003,"15min":0.503},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73199999999999998,"5min":0.90900000000000003,"15min":1.304},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.55500000000000005,"15min":0.59599999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.71499999999999997,"5min":0.84599999999999997,"15min":1.4590000000000001},"last":0.45700000000000002}]},{"osd":2,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.54600000000000004,"15min":0.59499999999999997},"min":{"1min":0.34899999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.78600000000000003,"5min":1.054,"15min":1.054},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.61399999999999999,"15min":0.64900000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.76100000000000001,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.625}]},{"osd":4,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72299999999999998,"15min":0.77600000000000002},"min":{"1min":0.52600000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.93400000000000005,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.746,"15min":0.79600000000000004},"min":{"1min":0.51800000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.88800000000000001,"5min":1.171,"15min":1.1830000000000001},"last":0.90700000000000003}]},{"osd":5,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.755,"15min":0.80600000000000005},"min":{"1min":0.497,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.02,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78500000000000003,"15min":0.83399999999999996},"min":{"1min":0.55600000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.054,"5min":1.2669999999999999,"15min":1.3440000000000001},"last":0.82899999999999996}]},{"osd":6,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.79900000000000004,"15min":0.84299999999999997},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.97799999999999998,"5min":1.181,"15min":1.2170000000000001},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.77800000000000002,"15min":0.81799999999999995},"min":{"1min":0.59599999999999997,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.81100000000000005,"5min":1.883,"15min":1.883},"last":1.0089999999999999}]},{"osd":7,"last update":"Thu Jan 11 09:47:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.79700000000000004,"15min":0.85399999999999998},"min":{"1min":0.51600000000000001,"5min":0.505,"15min":0.505},"max":{"1min":0.89600000000000002,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.746,"5min":0.81200000000000006,"15min":0.876},"min":{"1min":0.59899999999999998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.97599999999999998,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.95299999999999996}]}]},{"osd":4,"up_from":13,"seq":55834576002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.68700000000000006,"15min":0.748},"min":{"1min":0.47299999999999998,"5min":0.42099999999999999,"15min":0.39900000000000002},"max":{"1min":0.86899999999999999,"5min":0.91200000000000003,"15min":1.0760000000000001},"last":0.876},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.70699999999999996,"15min":0.75700000000000001},"min":{"1min":0.52800000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84599999999999997,"5min":1.853,"15min":1.853},"last":0.85699999999999998}]},{"osd":1,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.67600000000000005,"15min":0.72599999999999998},"min":{"1min":0.42299999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.91500000000000004,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.67000000000000004,"15min":0.71999999999999997},"min":{"1min":0.44,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.93100000000000005,"5min":1.639,"15min":1.639},"last":0.77400000000000002}]},{"osd":2,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.68600000000000005,"15min":0.73599999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.44700000000000001},"max":{"1min":0.94099999999999995,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.68700000000000006,"15min":0.73199999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.014,"5min":1.014,"15min":1.0449999999999999},"last":0.83999999999999997}]},{"osd":3,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.72499999999999998},"min":{"1min":0.47799999999999998,"5min":0.436,"15min":0.436},"max":{"1min":0.91900000000000004,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73699999999999999,"15min":0.76800000000000002},"min":{"1min":0.59799999999999998,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.98899999999999999,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.754}]},{"osd":5,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.629,"15min":0.624},"min":{"1min":0.35299999999999998,"5min":0.311,"15min":0.311},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.66300000000000003,"15min":0.64500000000000002},"min":{"1min":0.47499999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0580000000000001,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.61699999999999999}]},{"osd":6,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68999999999999995,"15min":0.67200000000000004},"min":{"1min":0.441,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0680000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.72399999999999998,"15min":0.70499999999999996},"min":{"1min":0.39400000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.1899999999999999,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.67400000000000004}]},{"osd":7,"last update":"Thu Jan 11 09:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.72799999999999998,"15min":0.70199999999999996},"min":{"1min":0.44700000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0409999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.78200000000000003,"15min":0.755},"min":{"1min":0.51300000000000001,"5min":0.436,"15min":0.41699999999999998},"max":{"1min":1.109,"5min":2.052,"15min":2.052},"last":0.65100000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312804,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059036,"statfs":{"total":96636764160,"available":96316452864,"internally_reserved":0,"allocated":3305472,"data_stored":1602300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 11 09:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69399999999999995,"15min":0.72699999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.35799999999999998},"max":{"1min":0.88400000000000001,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.71099999999999997,"15min":0.72699999999999998},"min":{"1min":0.45700000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.94499999999999995,"5min":2.9900000000000002,"15min":2.9900000000000002},"last":0.84399999999999997}]},{"osd":1,"last update":"Thu Jan 11 09:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.70399999999999996,"15min":0.73199999999999998},"min":{"1min":0.495,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.95299999999999996,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.73899999999999999,"15min":0.745},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.88500000000000001,"5min":5.4039999999999999,"15min":5.4039999999999999},"last":0.93300000000000005}]},{"osd":2,"last update":"Thu Jan 11 09:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71799999999999997,"15min":0.72899999999999998},"min":{"1min":0.50800000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.95899999999999996,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.72099999999999997,"15min":0.73899999999999999},"min":{"1min":0.52000000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.89000000000000001,"5min":2.5099999999999998,"15min":2.5099999999999998},"last":0.91300000000000003}]},{"osd":3,"last update":"Thu Jan 11 09:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74099999999999999,"15min":0.75800000000000001},"min":{"1min":0.48299999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.97599999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.78500000000000003,"15min":0.78000000000000003},"min":{"1min":0.59799999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0169999999999999,"5min":5.5570000000000004,"15min":5.5570000000000004},"last":0.81599999999999995}]},{"osd":4,"last update":"Thu Jan 11 09:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64500000000000002,"15min":0.624},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.82799999999999996,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67500000000000004,"15min":0.64200000000000002},"min":{"1min":0.40600000000000003,"5min":0.40200000000000002,"15min":0.375},"max":{"1min":0.96199999999999997,"5min":2.359,"15min":2.359},"last":0.72299999999999998}]},{"osd":6,"last update":"Thu Jan 11 09:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.68600000000000005,"15min":0.65500000000000003},"min":{"1min":0.497,"5min":0.42099999999999999,"15min":0.40999999999999998},"max":{"1min":0.996,"5min":2.9500000000000002,"15min":2.9500000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.70699999999999996,"15min":0.68899999999999995},"min":{"1min":0.46899999999999997,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.84599999999999997,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":0.79900000000000004}]},{"osd":7,"last update":"Thu Jan 11 09:48:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.755,"15min":0.73499999999999999},"min":{"1min":0.53500000000000003,"5min":0.47299999999999998,"15min":0.434},"max":{"1min":0.95799999999999996,"5min":2.399,"15min":2.399},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.80400000000000005,"15min":0.78400000000000003},"min":{"1min":0.60599999999999998,"5min":0.48799999999999999,"15min":0.47599999999999998},"max":{"1min":1.0600000000000001,"5min":2.7570000000000001,"15min":2.7570000000000001},"last":0.75900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-11T09:48:27.023 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-11T09:48:27.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.227+0000 7f4f94bff700 1 Processor -- start 2024-01-11T09:48:27.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.227+0000 7f4f94bff700 1 -- start start 2024-01-11T09:48:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.228+0000 7f4f94bff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4f901622d0 0x7f4f901646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.228+0000 7f4f94bff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f900cf850 con 0x7f4f9015a1b0 2024-01-11T09:48:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.228+0000 7f4f94bff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f900cec40 con 0x7f4f900ccd70 2024-01-11T09:48:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.228+0000 7f4f94bff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f9015ac90 con 0x7f4f901622d0 2024-01-11T09:48:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.229+0000 7f4f8e59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4f9015a1b0 0x7f4f9015a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51630/0 (socket says 172.21.15.27:51630) 2024-01-11T09:48:27.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.229+0000 7f4f8e59c700 1 -- 172.21.15.27:0/3316678225 learned_addr learned my addr 172.21.15.27:0/3316678225 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:48:27.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.229+0000 7f4f8ed9d700 1 --2- 172.21.15.27:0/3316678225 >> v2:172.21.15.77:3300/0 conn(0x7f4f901622d0 0x7f4f901646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:27.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.229+0000 7f4f8d59a700 1 -- 172.21.15.27:0/3316678225 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115195990 0 0) 0x7f4f900cf850 con 0x7f4f9015a1b0 2024-01-11T09:48:27.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.229+0000 7f4f8d59a700 1 -- 172.21.15.27:0/3316678225 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f78003570 con 0x7f4f9015a1b0 2024-01-11T09:48:27.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.229+0000 7f4f8d59a700 1 -- 172.21.15.27:0/3316678225 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1464291285 0 0) 0x7f4f78003570 con 0x7f4f9015a1b0 2024-01-11T09:48:27.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.229+0000 7f4f8d59a700 1 -- 172.21.15.27:0/3316678225 >> v2:172.21.15.77:3300/0 conn(0x7f4f901622d0 msgr2=0x7f4f901646b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:27.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.229+0000 7f4f8d59a700 1 --2- 172.21.15.27:0/3316678225 >> v2:172.21.15.77:3300/0 conn(0x7f4f901622d0 0x7f4f901646b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.229+0000 7f4f8d59a700 1 -- 172.21.15.27:0/3316678225 >> v1:172.21.15.77:6789/0 conn(0x7f4f900ccd70 legacy=0x7f4f900cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:27.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.229+0000 7f4f8d59a700 1 -- 172.21.15.27:0/3316678225 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f900cef20 con 0x7f4f9015a1b0 2024-01-11T09:48:27.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.229+0000 7f4f8d59a700 1 -- 172.21.15.27:0/3316678225 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7f4f80001470 con 0x7f4f9015a1b0 2024-01-11T09:48:27.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.230+0000 7f4f8d59a700 1 -- 172.21.15.27:0/3316678225 >> v1:172.21.15.27:6789/0 conn(0x7f4f9015a1b0 legacy=0x7f4f9015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:27.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.230+0000 7f4f8d59a700 1 --2- 172.21.15.27:0/3316678225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f78003a60 0x7f4f78003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:27.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.230+0000 7f4f8d59a700 1 --2- 172.21.15.27:0/3316678225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f78004810 0x7f4f78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:27.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.230+0000 7f4f8d59a700 1 --2- 172.21.15.27:0/3316678225 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4f780051c0 0x7f4f7800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:27.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.230+0000 7f4f8d59a700 1 -- 172.21.15.27:0/3316678225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9015ac90 con 0x7f4f78004810 2024-01-11T09:48:27.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.230+0000 7f4f8d59a700 1 -- 172.21.15.27:0/3316678225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7800c0d0 con 0x7f4f78003a60 2024-01-11T09:48:27.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.230+0000 7f4f8d59a700 1 -- 172.21.15.27:0/3316678225 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7800c290 con 0x7f4f780051c0 2024-01-11T09:48:27.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.230+0000 7f4f8dd9b700 1 --2- 172.21.15.27:0/3316678225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f78004810 0x7f4f78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:27.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.230+0000 7f4f8e59c700 1 --2- 172.21.15.27:0/3316678225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f78003a60 0x7f4f78003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:27.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.230+0000 7f4f8ed9d700 1 --2- 172.21.15.27:0/3316678225 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4f780051c0 0x7f4f7800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:27.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.230+0000 7f4f8dd9b700 1 -- 172.21.15.27:0/3316678225 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4f780051c0 msgr2=0x7f4f7800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:27.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.230+0000 7f4f8dd9b700 1 --2- 172.21.15.27:0/3316678225 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4f780051c0 0x7f4f7800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.230+0000 7f4f8dd9b700 1 -- 172.21.15.27:0/3316678225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f78003a60 msgr2=0x7f4f78003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:27.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.230+0000 7f4f8dd9b700 1 --2- 172.21.15.27:0/3316678225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f78003a60 0x7f4f78003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.230+0000 7f4f8dd9b700 1 -- 172.21.15.27:0/3316678225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f78003570 con 0x7f4f78004810 2024-01-11T09:48:27.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.230+0000 7f4f8ed9d700 1 --2- 172.21.15.27:0/3316678225 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4f780051c0 0x7f4f7800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-11T09:48:27.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.230+0000 7f4f8ed9d700 1 -- 172.21.15.27:0/3316678225 reap_dead start 2024-01-11T09:48:27.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.231+0000 7f4f8dd9b700 1 --2- 172.21.15.27:0/3316678225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f78004810 0x7f4f78004c80 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f4f840018b0 tx=0x7f4f84007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7870e2169454a1ca server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:27.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.231+0000 7f4f8d59a700 1 -- 172.21.15.27:0/3316678225 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f84012420 con 0x7f4f78004810 2024-01-11T09:48:27.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.231+0000 7f4f8d59a700 1 -- 172.21.15.27:0/3316678225 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f840125a0 con 0x7f4f78004810 2024-01-11T09:48:27.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.231+0000 7f4f8d59a700 1 -- 172.21.15.27:0/3316678225 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f84012960 con 0x7f4f78004810 2024-01-11T09:48:27.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.231+0000 7f4f94bff700 1 -- 172.21.15.27:0/3316678225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f78004810 msgr2=0x7f4f78004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:27.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.231+0000 7f4f94bff700 1 --2- 172.21.15.27:0/3316678225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f78004810 0x7f4f78004c80 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f4f840018b0 tx=0x7f4f84007d30 comp rx=0 tx=0).stop 2024-01-11T09:48:27.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.232+0000 7f4f94bff700 1 -- 172.21.15.27:0/3316678225 shutdown_connections 2024-01-11T09:48:27.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.232+0000 7f4f94bff700 1 --2- 172.21.15.27:0/3316678225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f78004810 0x7f4f78004c80 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.232+0000 7f4f94bff700 1 -- 172.21.15.27:0/3316678225 >> 172.21.15.27:0/3316678225 conn(0x7f4f9006ed40 msgr2=0x7f4f9006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:27.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.232+0000 7f4f94bff700 1 -- 172.21.15.27:0/3316678225 shutdown_connections 2024-01-11T09:48:27.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.232+0000 7f4f94bff700 1 -- 172.21.15.27:0/3316678225 wait complete. 2024-01-11T09:48:27.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.233+0000 7f4f94bff700 1 Processor -- start 2024-01-11T09:48:27.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.233+0000 7f4f94bff700 1 -- start start 2024-01-11T09:48:27.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.234+0000 7f4f94bff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f78004810 0x7f4f9014e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:27.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.234+0000 7f4f94bff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f900c66e0 0x7f4f900c6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:27.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.234+0000 7f4f94bff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4f900c9990 0x7f4f900c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:27.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.234+0000 7f4f94bff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f900c7090 con 0x7f4f78004810 2024-01-11T09:48:27.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.234+0000 7f4f94bff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f900b5090 con 0x7f4f900c66e0 2024-01-11T09:48:27.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.234+0000 7f4f94bff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f900b4480 con 0x7f4f900c9990 2024-01-11T09:48:27.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.234+0000 7f4f8e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f78004810 0x7f4f9014e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:27.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.234+0000 7f4f8e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f78004810 0x7f4f9014e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47106/0 (socket says 172.21.15.27:47106) 2024-01-11T09:48:27.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.234+0000 7f4f8e59c700 1 -- 172.21.15.27:0/1040017911 learned_addr learned my addr 172.21.15.27:0/1040017911 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:27.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.235+0000 7f4f8dd9b700 1 --2- 172.21.15.27:0/1040017911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f900c66e0 0x7f4f900c6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:27.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.235+0000 7f4f8ed9d700 1 --2- 172.21.15.27:0/1040017911 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4f900c9990 0x7f4f900c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:27.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.235+0000 7f4f8e59c700 1 -- 172.21.15.27:0/1040017911 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4f900c9990 msgr2=0x7f4f900c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:27.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.235+0000 7f4f8e59c700 1 --2- 172.21.15.27:0/1040017911 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4f900c9990 0x7f4f900c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.235+0000 7f4f8e59c700 1 -- 172.21.15.27:0/1040017911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f900c66e0 msgr2=0x7f4f900c6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:27.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.235+0000 7f4f8e59c700 1 --2- 172.21.15.27:0/1040017911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f900c66e0 0x7f4f900c6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.235+0000 7f4f8e59c700 1 -- 172.21.15.27:0/1040017911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f900b4760 con 0x7f4f78004810 2024-01-11T09:48:27.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.235+0000 7f4f8e59c700 1 --2- 172.21.15.27:0/1040017911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f78004810 0x7f4f9014e870 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f4f80001050 tx=0x7f4f80001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:27.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.235+0000 7f4f7e7fc700 1 -- 172.21.15.27:0/1040017911 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f80018030 con 0x7f4f78004810 2024-01-11T09:48:27.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.235+0000 7f4f7e7fc700 1 -- 172.21.15.27:0/1040017911 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8000f8c0 con 0x7f4f78004810 2024-01-11T09:48:27.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.235+0000 7f4f94bff700 1 -- 172.21.15.27:0/1040017911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f900b4a40 con 0x7f4f78004810 2024-01-11T09:48:27.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.235+0000 7f4f7e7fc700 1 -- 172.21.15.27:0/1040017911 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8000fc80 con 0x7f4f78004810 2024-01-11T09:48:27.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.235+0000 7f4f94bff700 1 -- 172.21.15.27:0/1040017911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f900ca340 con 0x7f4f78004810 2024-01-11T09:48:27.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.236+0000 7f4f94bff700 1 -- 172.21.15.27:0/1040017911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f900b5370 con 0x7f4f78004810 2024-01-11T09:48:27.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.237+0000 7f4f7e7fc700 1 -- 172.21.15.27:0/1040017911 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7f4f8001b070 con 0x7f4f78004810 2024-01-11T09:48:27.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.239+0000 7f4f7e7fc700 1 --2- 172.21.15.27:0/1040017911 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4f70066f30 0x7f4f700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:27.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.239+0000 7f4f7e7fc700 1 -- 172.21.15.27:0/1040017911 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7f4f80019070 con 0x7f4f78004810 2024-01-11T09:48:27.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.239+0000 7f4f8dd9b700 1 --2- 172.21.15.27:0/1040017911 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4f70066f30 0x7f4f700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:27.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.240+0000 7f4f8dd9b700 1 --2- 172.21.15.27:0/1040017911 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4f70066f30 0x7f4f700693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4f900c6290 tx=0x7f4f84015000 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:27.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.243+0000 7f4f7e7fc700 1 -- 172.21.15.27:0/1040017911 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4f80051800 con 0x7f4f78004810 2024-01-11T09:48:27.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.448+0000 7f4f94bff700 1 -- 172.21.15.27:0/1040017911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f4f9004c3c0 con 0x7f4f78004810 2024-01-11T09:48:27.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.451+0000 7f4f7e7fc700 1 -- 172.21.15.27:0/1040017911 <== mon.0 v2:172.21.15.27:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f4f80051800 con 0x7f4f78004810 2024-01-11T09:48:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.452+0000 7f4f7e7fc700 1 -- 172.21.15.27:0/1040017911 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v94)=0 v94) v1 ==== 123+0+0 (secure 0 0 0) 0x7f4f80054e20 con 0x7f4f78004810 2024-01-11T09:48:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.455+0000 7f4f94bff700 1 -- 172.21.15.27:0/1040017911 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4f70066f30 msgr2=0x7f4f700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.455+0000 7f4f94bff700 1 --2- 172.21.15.27:0/1040017911 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4f70066f30 0x7f4f700693e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4f900c6290 tx=0x7f4f84015000 comp rx=0 tx=0).stop 2024-01-11T09:48:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.455+0000 7f4f94bff700 1 -- 172.21.15.27:0/1040017911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f78004810 msgr2=0x7f4f9014e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.455+0000 7f4f94bff700 1 --2- 172.21.15.27:0/1040017911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f78004810 0x7f4f9014e870 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f4f80001050 tx=0x7f4f80001e50 comp rx=0 tx=0).stop 2024-01-11T09:48:27.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.455+0000 7f4f94bff700 1 -- 172.21.15.27:0/1040017911 shutdown_connections 2024-01-11T09:48:27.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.455+0000 7f4f94bff700 1 --2- 172.21.15.27:0/1040017911 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7f4f900c9990 0x7f4f900c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.455+0000 7f4f94bff700 1 --2- 172.21.15.27:0/1040017911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f78004810 0x7f4f9014e870 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.455+0000 7f4f94bff700 1 --2- 172.21.15.27:0/1040017911 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7f4f70066f30 0x7f4f700693e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.455+0000 7f4f94bff700 1 --2- 172.21.15.27:0/1040017911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f900c66e0 0x7f4f900c6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.455+0000 7f4f94bff700 1 -- 172.21.15.27:0/1040017911 >> 172.21.15.27:0/1040017911 conn(0x7f4f9006ed40 msgr2=0x7f4f9015ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:27.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.455+0000 7f4f94bff700 1 -- 172.21.15.27:0/1040017911 shutdown_connections 2024-01-11T09:48:27.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.455+0000 7f4f94bff700 1 -- 172.21.15.27:0/1040017911 wait complete. 2024-01-11T09:48:27.463 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-11T09:48:27.464 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-11T09:48:27.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.672+0000 7fd6b1fff700 1 Processor -- start 2024-01-11T09:48:27.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.673+0000 7fd6b1fff700 1 -- start start 2024-01-11T09:48:27.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.674+0000 7fd6b1fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd6ac0b03e0 0x7fd6ac0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:27.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.674+0000 7fd6b1fff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ac0cd730 con 0x7fd6ac0b07b0 2024-01-11T09:48:27.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.674+0000 7fd6b1fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ac0b2de0 con 0x7fd6ac0b29c0 2024-01-11T09:48:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.674+0000 7fd6b1fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6ac0ccb20 con 0x7fd6ac0b03e0 2024-01-11T09:48:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.674+0000 7fd6aaffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd6ac0b07b0 0x7fd6ac159200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51646/0 (socket says 172.21.15.27:51646) 2024-01-11T09:48:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.674+0000 7fd6aaffd700 1 -- 172.21.15.27:0/2466403289 learned_addr learned my addr 172.21.15.27:0/2466403289 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-11T09:48:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.675+0000 7fd6ab7fe700 1 --2- 172.21.15.27:0/2466403289 >> v2:172.21.15.77:3300/0 conn(0x7fd6ac0b03e0 0x7fd6ac0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.675+0000 7fd6aa7fc700 1 -- 172.21.15.27:0/2466403289 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7095481 0 0) 0x7fd6ac0cd730 con 0x7fd6ac0b07b0 2024-01-11T09:48:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.675+0000 7fd6aa7fc700 1 -- 172.21.15.27:0/2466403289 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd694003570 con 0x7fd6ac0b07b0 2024-01-11T09:48:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.675+0000 7fd6aa7fc700 1 -- 172.21.15.27:0/2466403289 <== mon.1 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802802464 0 0) 0x7fd6ac0b2de0 con 0x7fd6ac0b29c0 2024-01-11T09:48:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.675+0000 7fd6aa7fc700 1 -- 172.21.15.27:0/2466403289 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6ac0cd730 con 0x7fd6ac0b29c0 2024-01-11T09:48:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.675+0000 7fd6aa7fc700 1 -- 172.21.15.27:0/2466403289 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2500589015 0 0) 0x7fd694003570 con 0x7fd6ac0b07b0 2024-01-11T09:48:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.675+0000 7fd6aa7fc700 1 -- 172.21.15.27:0/2466403289 >> v2:172.21.15.77:3300/0 conn(0x7fd6ac0b03e0 msgr2=0x7fd6ac0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.675+0000 7fd6aa7fc700 1 --2- 172.21.15.27:0/2466403289 >> v2:172.21.15.77:3300/0 conn(0x7fd6ac0b03e0 0x7fd6ac0b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.675+0000 7fd6aa7fc700 1 -- 172.21.15.27:0/2466403289 >> v1:172.21.15.77:6789/0 conn(0x7fd6ac0b29c0 legacy=0x7fd6ac0aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:27.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.675+0000 7fd6aa7fc700 1 -- 172.21.15.27:0/2466403289 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ac0cce00 con 0x7fd6ac0b07b0 2024-01-11T09:48:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.675+0000 7fd6aa7fc700 1 -- 172.21.15.27:0/2466403289 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2134012808 0 0) 0x7fd6a0001470 con 0x7fd6ac0b07b0 2024-01-11T09:48:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.675+0000 7fd6aa7fc700 1 -- 172.21.15.27:0/2466403289 >> v1:172.21.15.27:6789/0 conn(0x7fd6ac0b07b0 legacy=0x7fd6ac159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.675+0000 7fd6aa7fc700 1 --2- 172.21.15.27:0/2466403289 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd694003bf0 0x7fd694003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.675+0000 7fd6aa7fc700 1 --2- 172.21.15.27:0/2466403289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd694004810 0x7fd694004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.675+0000 7fd6aa7fc700 1 --2- 172.21.15.27:0/2466403289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6940051c0 0x7fd69400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.675+0000 7fd6aa7fc700 1 -- 172.21.15.27:0/2466403289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac0ccb20 con 0x7fd694004810 2024-01-11T09:48:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.675+0000 7fd6aa7fc700 1 -- 172.21.15.27:0/2466403289 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69400c0b0 con 0x7fd6940051c0 2024-01-11T09:48:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.675+0000 7fd6aa7fc700 1 -- 172.21.15.27:0/2466403289 --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69400c270 con 0x7fd694003bf0 2024-01-11T09:48:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.676+0000 7fd6aaffd700 1 --2- 172.21.15.27:0/2466403289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd694004810 0x7fd694004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.676+0000 7fd6abfff700 1 --2- 172.21.15.27:0/2466403289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6940051c0 0x7fd69400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.676+0000 7fd6ab7fe700 1 --2- 172.21.15.27:0/2466403289 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd694003bf0 0x7fd694003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.676+0000 7fd6aaffd700 1 -- 172.21.15.27:0/2466403289 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd694003bf0 msgr2=0x7fd694003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:27.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.676+0000 7fd6aaffd700 1 --2- 172.21.15.27:0/2466403289 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd694003bf0 0x7fd694003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.676+0000 7fd6aaffd700 1 -- 172.21.15.27:0/2466403289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6940051c0 msgr2=0x7fd69400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-11T09:48:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.676+0000 7fd6aaffd700 1 --2- 172.21.15.27:0/2466403289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6940051c0 0x7fd69400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.676+0000 7fd6aaffd700 1 -- 172.21.15.27:0/2466403289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd694003570 con 0x7fd694004810 2024-01-11T09:48:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.676+0000 7fd6abfff700 1 --2- 172.21.15.27:0/2466403289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6940051c0 0x7fd69400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-11T09:48:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.676+0000 7fd6abfff700 1 -- 172.21.15.27:0/2466403289 reap_dead start 2024-01-11T09:48:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.676+0000 7fd6aaffd700 1 --2- 172.21.15.27:0/2466403289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd694004810 0x7fd694004c80 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7fd6a000c2d0 tx=0x7fd6a0001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2882870e80d8228b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.677+0000 7fd6aa7fc700 1 -- 172.21.15.27:0/2466403289 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a000cbc0 con 0x7fd694004810 2024-01-11T09:48:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.677+0000 7fd6aa7fc700 1 -- 172.21.15.27:0/2466403289 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fd6a000cd20 con 0x7fd694004810 2024-01-11T09:48:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.677+0000 7fd6aa7fc700 1 -- 172.21.15.27:0/2466403289 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0005b90 con 0x7fd694004810 2024-01-11T09:48:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.677+0000 7fd6b1fff700 1 -- 172.21.15.27:0/2466403289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd694004810 msgr2=0x7fd694004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.677+0000 7fd6b1fff700 1 --2- 172.21.15.27:0/2466403289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd694004810 0x7fd694004c80 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7fd6a000c2d0 tx=0x7fd6a0001e10 comp rx=0 tx=0).stop 2024-01-11T09:48:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.677+0000 7fd6b1fff700 1 -- 172.21.15.27:0/2466403289 shutdown_connections 2024-01-11T09:48:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.677+0000 7fd6b1fff700 1 --2- 172.21.15.27:0/2466403289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd694004810 0x7fd694004c80 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.677+0000 7fd6b1fff700 1 -- 172.21.15.27:0/2466403289 >> 172.21.15.27:0/2466403289 conn(0x7fd6ac06ed40 msgr2=0x7fd6ac06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.677+0000 7fd6b1fff700 1 -- 172.21.15.27:0/2466403289 shutdown_connections 2024-01-11T09:48:27.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.677+0000 7fd6b1fff700 1 -- 172.21.15.27:0/2466403289 wait complete. 2024-01-11T09:48:27.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.679+0000 7fd6b1fff700 1 Processor -- start 2024-01-11T09:48:27.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.679+0000 7fd6b1fff700 1 -- start start 2024-01-11T09:48:27.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.680+0000 7fd6b1fff700 1 --2- >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd694004810 0x7fd6ac0c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:27.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.680+0000 7fd6b1fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6ac0a6f40 0x7fd6ac16d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:27.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.680+0000 7fd6b1fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6ac16daf0 0x7fd6ac0a8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:27.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.680+0000 7fd6b1fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac16df90 con 0x7fd6ac16daf0 2024-01-11T09:48:27.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.680+0000 7fd6b1fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac0a7fc0 con 0x7fd6ac0a6f40 2024-01-11T09:48:27.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.680+0000 7fd6b1fff700 1 -- --> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac0a73b0 con 0x7fd694004810 2024-01-11T09:48:27.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.680+0000 7fd6abfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6ac16daf0 0x7fd6ac0a8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:27.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.680+0000 7fd6abfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6ac16daf0 0x7fd6ac0a8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47118/0 (socket says 172.21.15.27:47118) 2024-01-11T09:48:27.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.680+0000 7fd6abfff700 1 -- 172.21.15.27:0/1151214679 learned_addr learned my addr 172.21.15.27:0/1151214679 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-11T09:48:27.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.680+0000 7fd6ab7fe700 1 --2- 172.21.15.27:0/1151214679 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd694004810 0x7fd6ac0c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:27.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.680+0000 7fd6aaffd700 1 --2- 172.21.15.27:0/1151214679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6ac0a6f40 0x7fd6ac16d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:27.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.681+0000 7fd6abfff700 1 -- 172.21.15.27:0/1151214679 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd694004810 msgr2=0x7fd6ac0c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:27.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.681+0000 7fd6abfff700 1 --2- 172.21.15.27:0/1151214679 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd694004810 0x7fd6ac0c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.681+0000 7fd6abfff700 1 -- 172.21.15.27:0/1151214679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6ac0a6f40 msgr2=0x7fd6ac16d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:27.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.681+0000 7fd6abfff700 1 --2- 172.21.15.27:0/1151214679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6ac0a6f40 0x7fd6ac16d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.681+0000 7fd6abfff700 1 -- 172.21.15.27:0/1151214679 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ac0a7690 con 0x7fd6ac16daf0 2024-01-11T09:48:27.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.681+0000 7fd6abfff700 1 --2- 172.21.15.27:0/1151214679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6ac16daf0 0x7fd6ac0a8450 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac06fd20 tx=0x7fd6a40017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:27.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.681+0000 7fd6937fe700 1 -- 172.21.15.27:0/1151214679 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4006ba0 con 0x7fd6ac16daf0 2024-01-11T09:48:27.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.681+0000 7fd6937fe700 1 -- 172.21.15.27:0/1151214679 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fd6a4006d00 con 0x7fd6ac16daf0 2024-01-11T09:48:27.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.681+0000 7fd6b1fff700 1 -- 172.21.15.27:0/1151214679 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6ac0a7970 con 0x7fd6ac16daf0 2024-01-11T09:48:27.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.681+0000 7fd6937fe700 1 -- 172.21.15.27:0/1151214679 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a400cd30 con 0x7fd6ac16daf0 2024-01-11T09:48:27.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.681+0000 7fd6b1fff700 1 -- 172.21.15.27:0/1151214679 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ac0c8e20 con 0x7fd6ac16daf0 2024-01-11T09:48:27.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.682+0000 7fd6b1fff700 1 -- 172.21.15.27:0/1151214679 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6ac0c9010 con 0x7fd6ac16daf0 2024-01-11T09:48:27.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.683+0000 7fd6937fe700 1 -- 172.21.15.27:0/1151214679 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 154) v1 ==== 84742+0+0 (secure 0 0 0) 0x7fd6a4007230 con 0x7fd6ac16daf0 2024-01-11T09:48:27.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.684+0000 7fd6937fe700 1 --2- 172.21.15.27:0/1151214679 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd698066f30 0x7fd6980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-11T09:48:27.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.684+0000 7fd6937fe700 1 -- 172.21.15.27:0/1151214679 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5782+0+0 (secure 0 0 0) 0x7fd6a4083d90 con 0x7fd6ac16daf0 2024-01-11T09:48:27.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.685+0000 7fd6ab7fe700 1 --2- 172.21.15.27:0/1151214679 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd698066f30 0x7fd6980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-11T09:48:27.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.685+0000 7fd6ab7fe700 1 --2- 172.21.15.27:0/1151214679 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd698066f30 0x7fd6980693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac04cc40 tx=0x7fd69c0091a0 comp rx=0 tx=0).ready entity=mgr.11878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T09:48:27.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.689+0000 7fd6937fe700 1 -- 172.21.15.27:0/1151214679 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd6a404b5a0 con 0x7fd6ac16daf0 2024-01-11T09:48:27.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.934+0000 7fd6b1fff700 1 -- 172.21.15.27:0/1151214679 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6ac0a8150 con 0x7fd6ac16daf0 2024-01-11T09:48:27.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.934+0000 7fd6937fe700 1 -- 172.21.15.27:0/1151214679 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+3537 (secure 0 0 0) 0x7fd6a4051030 con 0x7fd6ac16daf0 2024-01-11T09:48:27.926 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-11T09:48:27.926 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":143,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9866,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1060058066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1060058066},{"type":"v1","addr":"172.21.15.77:6835","nonce":1060058066}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137},{"gid":11626,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/4169546841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":4169546841},{"type":"v1","addr":"172.21.15.77:6834","nonce":4169546841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137},{"gid":11632,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/2567971522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2567971522},{"type":"v1","addr":"172.21.15.27:6833","nonce":2567971522}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":137},{"gid":11962,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2417358034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2417358034},{"type":"v1","addr":"172.21.15.27:6835","nonce":2417358034}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":143}],"filesystems":[]} 2024-01-11T09:48:27.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.937+0000 7fd6b1fff700 1 -- 172.21.15.27:0/1151214679 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd698066f30 msgr2=0x7fd6980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:27.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.937+0000 7fd6b1fff700 1 --2- 172.21.15.27:0/1151214679 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd698066f30 0x7fd6980693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac04cc40 tx=0x7fd69c0091a0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.937+0000 7fd6b1fff700 1 -- 172.21.15.27:0/1151214679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6ac16daf0 msgr2=0x7fd6ac0a8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T09:48:27.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.937+0000 7fd6b1fff700 1 --2- 172.21.15.27:0/1151214679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6ac16daf0 0x7fd6ac0a8450 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac06fd20 tx=0x7fd6a40017a0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.937+0000 7fd6b1fff700 1 -- 172.21.15.27:0/1151214679 shutdown_connections 2024-01-11T09:48:27.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.937+0000 7fd6b1fff700 1 --2- 172.21.15.27:0/1151214679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6ac16daf0 0x7fd6ac0a8450 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.937+0000 7fd6b1fff700 1 --2- 172.21.15.27:0/1151214679 >> [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] conn(0x7fd694004810 0x7fd6ac0c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.937+0000 7fd6b1fff700 1 --2- 172.21.15.27:0/1151214679 >> [v2:172.21.15.77:6836/59776,v1:172.21.15.77:6837/59776] conn(0x7fd698066f30 0x7fd6980693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.937+0000 7fd6b1fff700 1 --2- 172.21.15.27:0/1151214679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6ac0a6f40 0x7fd6ac16d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-11T09:48:27.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.937+0000 7fd6b1fff700 1 -- 172.21.15.27:0/1151214679 >> 172.21.15.27:0/1151214679 conn(0x7fd6ac06ed40 msgr2=0x7fd6ac0ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T09:48:27.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.937+0000 7fd6b1fff700 1 -- 172.21.15.27:0/1151214679 shutdown_connections 2024-01-11T09:48:27.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-11T09:48:27.937+0000 7fd6b1fff700 1 -- 172.21.15.27:0/1151214679 wait complete. 2024-01-11T09:48:27.930 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 143 2024-01-11T09:48:27.946 DEBUG:tasks.cephfs.filesystem:already dead... 2024-01-11T09:48:27.946 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-11T09:48:27.946 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-11T09:48:27.946 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:48:27.959 INFO:tasks.ceph.mds.a:Stopped 2024-01-11T09:48:27.960 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-11T09:48:27.960 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:48:28.013 INFO:tasks.ceph.mds.c:Stopped 2024-01-11T09:48:28.014 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-11T09:48:28.014 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:48:28.030 INFO:tasks.ceph.mds.b:Stopped 2024-01-11T09:48:28.030 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-11T09:48:28.030 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:48:28.087 INFO:tasks.ceph.mds.d:Stopped 2024-01-11T09:48:28.087 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-11T09:48:28.087 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-11T09:48:28.087 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:48:28.139 INFO:tasks.ceph.osd.0:Stopped 2024-01-11T09:48:28.139 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-11T09:48:28.140 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:48:28.193 INFO:tasks.ceph.osd.1:Stopped 2024-01-11T09:48:28.194 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-11T09:48:28.194 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:48:28.244 INFO:tasks.ceph.osd.2:Stopped 2024-01-11T09:48:28.245 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-01-11T09:48:28.245 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:48:28.289 INFO:tasks.ceph.osd.3:Stopped 2024-01-11T09:48:28.289 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-01-11T09:48:28.289 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:48:28.339 INFO:tasks.ceph.osd.4:Stopped 2024-01-11T09:48:28.340 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-01-11T09:48:28.340 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:48:28.405 INFO:tasks.ceph.osd.5:Stopped 2024-01-11T09:48:28.405 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-01-11T09:48:28.405 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:48:28.455 INFO:tasks.ceph.osd.6:Stopped 2024-01-11T09:48:28.455 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-01-11T09:48:28.455 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:48:28.514 INFO:tasks.ceph.osd.7:Stopped 2024-01-11T09:48:28.514 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-11T09:48:28.514 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-01-11T09:48:28.514 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:48:28.578 INFO:tasks.ceph.mgr.y:Stopped 2024-01-11T09:48:28.579 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-11T09:48:28.579 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:48:28.647 INFO:tasks.ceph.mgr.x:Stopped 2024-01-11T09:48:28.648 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-11T09:48:28.648 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-11T09:48:28.648 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:48:28.680 INFO:tasks.ceph.mon.a:Stopped 2024-01-11T09:48:28.680 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-01-11T09:48:28.680 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:48:28.705 INFO:tasks.ceph.mon.b:Stopped 2024-01-11T09:48:28.705 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-01-11T09:48:28.705 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T09:48:28.771 INFO:tasks.ceph.mon.c:Stopped 2024-01-11T09:48:28.772 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-11T09:48:28.772 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-11T09:48:28.856 INFO:teuthology.orchestra.run.smithi027.stdout:1704961464.7739727 mon.a (mon.0) 551 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-01-11T09:48:28.857 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-01-11T09:48:28.857 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-11T09:48:28.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-11T09:48:29.028 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-11T09:48:29.113 INFO:teuthology.orchestra.run.smithi027.stdout:1704961464.7739727 mon.a (mon.0) 551 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-01-11T09:48:29.114 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T09:48:29.114 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-11T09:48:29.373 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T09:48:29.373 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-11T09:48:29.520 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T09:48:29.520 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-11T09:48:29.653 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T09:48:29.654 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-01-11T09:48:29.804 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi077.front.sepia.ceph.com 2024-01-11T09:48:29.805 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-01-11T09:48:30.070 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi077.front.sepia.ceph.com 2024-01-11T09:48:30.070 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-01-11T09:48:30.190 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi077.front.sepia.ceph.com 2024-01-11T09:48:30.190 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-01-11T09:48:30.305 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi077.front.sepia.ceph.com 2024-01-11T09:48:30.306 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-01-11T09:48:30.442 INFO:tasks.ceph:Archiving mon data... 2024-01-11T09:48:30.444 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512563/data/mon.a.tgz 2024-01-11T09:48:30.444 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-01-11T09:48:30.494 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.i4L4JfDpf8 2024-01-11T09:48:30.494 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.i4L4JfDpf8 2024-01-11T09:48:31.128 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.i4L4JfDpf8 2024-01-11T09:48:31.251 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.i4L4JfDpf8 is 3MB 2024-01-11T09:48:31.410 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.i4L4JfDpf8 2024-01-11T09:48:31.459 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512563/data/mon.b.tgz 2024-01-11T09:48:31.459 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2024-01-11T09:48:31.508 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.HHobAeTVzp 2024-01-11T09:48:31.509 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.HHobAeTVzp 2024-01-11T09:48:32.130 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.HHobAeTVzp 2024-01-11T09:48:32.248 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.HHobAeTVzp is 4MB 2024-01-11T09:48:32.386 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.HHobAeTVzp 2024-01-11T09:48:32.434 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512563/data/mon.c.tgz 2024-01-11T09:48:32.434 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2024-01-11T09:48:32.520 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.qsQ5PGJia7 2024-01-11T09:48:32.520 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.qsQ5PGJia7 2024-01-11T09:48:33.176 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.qsQ5PGJia7 2024-01-11T09:48:33.294 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.qsQ5PGJia7 is 4MB 2024-01-11T09:48:33.449 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.qsQ5PGJia7 2024-01-11T09:48:33.496 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-11T09:48:33.497 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-11T09:48:33.502 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-11T09:48:33.575 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-01-11T09:48:33.575 DEBUG:teuthology.orchestra.run.smithi027:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-01-11T09:48:33.580 DEBUG:teuthology.orchestra.run.smithi077:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-01-11T09:48:33.661 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-01-11T09:48:33.664 INFO:teuthology.orchestra.run.smithi077.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-01-11T09:48:33.680 INFO:tasks.ceph:Archiving crash dumps... 2024-01-11T09:48:33.682 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512563/remote/smithi027/crash 2024-01-11T09:48:33.683 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-11T09:48:33.754 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512563/remote/smithi077/crash 2024-01-11T09:48:33.755 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-11T09:48:33.825 INFO:tasks.ceph:Shutting down logrotate 2024-01-11T09:48:33.825 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:48:33.830 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-11T09:48:33.931 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-01-11T09:48:34.002 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-01-11T09:48:34.072 INFO:tasks.ceph:Compressing logs... 2024-01-11T09:48:34.072 DEBUG:teuthology.orchestra.run.smithi027:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-11T09:48:34.077 DEBUG:teuthology.orchestra.run.smithi077:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-11T09:48:34.144 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.79061.log 2024-01-11T09:48:34.145 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110307.log 2024-01-11T09:48:34.145 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.79061.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.79061.log.gz 2024-01-11T09:48:34.146 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71716.log 2024-01-11T09:48:34.146 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41967.log 2024-01-11T09:48:34.147 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.48400.log 2024-01-11T09:48:34.147 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.110307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90479.log 2024-01-11T09:48:34.148 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.110307.log.gz 2024-01-11T09:48:34.149 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71716.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.71716.log.gz 2024-01-11T09:48:34.149 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119291.log 2024-01-11T09:48:34.150 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90479.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90479.log.gz 2024-01-11T09:48:34.150 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73605.log 2024-01-11T09:48:34.150 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.41967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.48898.log 2024-01-11T09:48:34.150 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41967.log.gz 2024-01-11T09:48:34.150 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.1.48400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.48400.log.gz 2024-01-11T09:48:34.150 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-01-11T09:48:34.150 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.3.48898.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.48898.log.gz 2024-01-11T09:48:34.150 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41868.log 2024-01-11T09:48:34.151 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-11T09:48:34.152 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119291.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.119291.log.gz 2024-01-11T09:48:34.152 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73037.log 2024-01-11T09:48:34.152 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73605.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73605.log.gz 2024-01-11T09:48:34.153 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92362.log 2024-01-11T09:48:34.153 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73037.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73037.log.gz 2024-01-11T09:48:34.153 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76086.log 2024-01-11T09:48:34.154 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92362.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.92362.log.gz 2024-01-11T09:48:34.154 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.94497.log 2024-01-11T09:48:34.155 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.41868.log: gzip -5 --verbose -- 87.9%/var/log/ceph/ceph-client.admin.41771.log -- replaced with /var/log/ceph/ceph-client.admin.41868.log.gz 2024-01-11T09:48:34.155 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-01-11T09:48:34.155 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41819.log 2024-01-11T09:48:34.155 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.41771.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41771.log.gz 2024-01-11T09:48:34.155 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41622.log 2024-01-11T09:48:34.157 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.41819.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41819.log.gz 2024-01-11T09:48:34.157 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.48894.log 2024-01-11T09:48:34.157 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.41622.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41622.log.gz 2024-01-11T09:48:34.157 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76086.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.76086.log.gz 2024-01-11T09:48:34.158 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105070.log 2024-01-11T09:48:34.158 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.94497.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.94497.log.gz 2024-01-11T09:48:34.158 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86830.log 2024-01-11T09:48:34.158 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.105070.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.105070.log.gz 2024-01-11T09:48:34.158 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73788.log 2024-01-11T09:48:34.158 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86830.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.86830.log.gz 2024-01-11T09:48:34.158 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52579.log 2024-01-11T09:48:34.160 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73788.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.73788.log.gz 2024-01-11T09:48:34.160 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111153.log 2024-01-11T09:48:34.160 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.107074.log 2024-01-11T09:48:34.160 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52579.log.gz 2024-01-11T09:48:34.161 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-01-11T09:48:34.161 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.3.48894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.48894.log.gz 2024-01-11T09:48:34.161 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41917.log 2024-01-11T09:48:34.161 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-11T09:48:34.161 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.41917.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41917.log.gz 2024-01-11T09:48:34.161 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.48414.log 2024-01-11T09:48:34.162 INFO:teuthology.orchestra.run.smithi077.stderr: 93.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-11T09:48:34.163 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.111153.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.111153.log.gz 2024-01-11T09:48:34.163 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77999.log 2024-01-11T09:48:34.163 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.107074.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.107074.log.gz 2024-01-11T09:48:34.163 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101547.log 2024-01-11T09:48:34.163 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77999.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77999.log.gz 2024-01-11T09:48:34.163 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114627.log 2024-01-11T09:48:34.165 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.101547.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.101547.log.gz 2024-01-11T09:48:34.165 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.117057.log 2024-01-11T09:48:34.165 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.114627.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.114627.log.gz 2024-01-11T09:48:34.165 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117235.log 2024-01-11T09:48:34.165 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-01-11T09:48:34.166 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.1.48414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.48414.log.gz 2024-01-11T09:48:34.166 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-01-11T09:48:34.166 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-01-11T09:48:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.117057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.117057.log.gz 2024-01-11T09:48:34.167 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110571.log 2024-01-11T09:48:34.167 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.117235.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.117235.log.gz 2024-01-11T09:48:34.167 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114295.log 2024-01-11T09:48:34.167 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-01-11T09:48:34.168 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.110571.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.110571.log.gz 2024-01-11T09:48:34.168 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108030.log 2024-01-11T09:48:34.169 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.114295.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.114295.log.gz 2024-01-11T09:48:34.169 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102456.log 2024-01-11T09:48:34.169 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-11T09:48:34.170 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.108030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104912.log 2024-01-11T09:48:34.170 INFO:teuthology.orchestra.run.smithi027.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.108030.log.gz 2024-01-11T09:48:34.171 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.102456.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.102456.log.gz 2024-01-11T09:48:34.171 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56884.log 2024-01-11T09:48:34.172 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.104912.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.104912.log.gz 2024-01-11T09:48:34.172 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105557.log 2024-01-11T09:48:34.173 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-01-11T09:48:34.174 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56884.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56884.log.gz 2024-01-11T09:48:34.174 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117816.log 2024-01-11T09:48:34.174 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.105557.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105557.log.gz 2024-01-11T09:48:34.174 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107981.log 2024-01-11T09:48:34.175 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41723.log 2024-01-11T09:48:34.176 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.117816.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.117816.log.gz 2024-01-11T09:48:34.176 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102063.log 2024-01-11T09:48:34.176 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.107981.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.107981.log.gz 2024-01-11T09:48:34.176 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48621.log 2024-01-11T09:48:34.177 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.102063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114189.log 2024-01-11T09:48:34.177 INFO:teuthology.orchestra.run.smithi027.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.102063.log.gz 2024-01-11T09:48:34.178 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48621.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48621.log.gz 2024-01-11T09:48:34.178 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114243.log 2024-01-11T09:48:34.179 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.114189.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.114189.log.gz 2024-01-11T09:48:34.180 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51615.log 2024-01-11T09:48:34.181 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.114243.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.114243.log.gz 2024-01-11T09:48:34.181 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109635.log 2024-01-11T09:48:34.182 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.41723.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41723.log.gz 2024-01-11T09:48:34.182 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41673.log 2024-01-11T09:48:34.183 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101843.log 2024-01-11T09:48:34.183 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51615.log.gz 2024-01-11T09:48:34.183 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109635.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.109635.log.gz 2024-01-11T09:48:34.183 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87011.log 2024-01-11T09:48:34.183 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.101843.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.101843.log.gz 2024-01-11T09:48:34.184 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102136.log 2024-01-11T09:48:34.185 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87011.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87011.log.gz 2024-01-11T09:48:34.185 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110718.log 2024-01-11T09:48:34.186 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-01-11T09:48:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.102136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115536.log 2024-01-11T09:48:34.187 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.102136.log.gz 2024-01-11T09:48:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.110718.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.110718.log.gz 2024-01-11T09:48:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77237.log 2024-01-11T09:48:34.188 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.41673.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41673.log.gz 2024-01-11T09:48:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.115536.log: gzip 88.2%-5 -- replaced with /var/log/ceph/ceph-client.admin.115536.log.gz--verbose 2024-01-11T09:48:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:-- /var/log/ceph/ceph-client.admin.113980.log 2024-01-11T09:48:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77237.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77237.log.gz 2024-01-11T09:48:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62634.log 2024-01-11T09:48:34.189 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.113980.log: gzip 87.8%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.113980.log.gz-- 2024-01-11T09:48:34.189 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81257.log 2024-01-11T09:48:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62634.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62634.log.gz 2024-01-11T09:48:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83187.log 2024-01-11T09:48:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81257.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81257.log.gz 2024-01-11T09:48:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118028.log 2024-01-11T09:48:34.193 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83187.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83187.log.gz 2024-01-11T09:48:34.193 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56255.log 2024-01-11T09:48:34.193 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-osd.5.log: 93.8% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-01-11T09:48:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.118028.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.118028.log.gz 2024-01-11T09:48:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110152.log 2024-01-11T09:48:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56255.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56255.log.gz 2024-01-11T09:48:34.195 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97906.log 2024-01-11T09:48:34.196 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.110152.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.110152.log.gz 2024-01-11T09:48:34.196 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54766.log 2024-01-11T09:48:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.97906.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97906.log.gz 2024-01-11T09:48:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112749.log 2024-01-11T09:48:34.198 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54766.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54766.log.gz 2024-01-11T09:48:34.198 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82981.log 2024-01-11T09:48:34.199 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.112749.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112749.log.gz 2024-01-11T09:48:34.199 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112641.log 2024-01-11T09:48:34.200 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82981.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.82981.log.gz 2024-01-11T09:48:34.200 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96840.log 2024-01-11T09:48:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.112641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69814.log 2024-01-11T09:48:34.201 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.112641.log.gz 2024-01-11T09:48:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.96840.log: 88.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.96840.log.gz--verbose 2024-01-11T09:48:34.202 INFO:teuthology.orchestra.run.smithi027.stderr:-- /var/log/ceph/ceph-client.admin.96396.log 2024-01-11T09:48:34.203 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69814.log: gzip 88.1%-5 -- replaced with /var/log/ceph/ceph-client.admin.69814.log.gz--verbose 2024-01-11T09:48:34.203 INFO:teuthology.orchestra.run.smithi027.stderr:-- /var/log/ceph/ceph-client.admin.108846.log 2024-01-11T09:48:34.204 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.96396.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.96396.log.gz 2024-01-11T09:48:34.204 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85625.log 2024-01-11T09:48:34.204 INFO:teuthology.orchestra.run.smithi077.stderr: 93.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-11T09:48:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.108846.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108846.log.gz 2024-01-11T09:48:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116426.log 2024-01-11T09:48:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85625.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85625.log.gz 2024-01-11T09:48:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119389.log 2024-01-11T09:48:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.116426.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.116426.log.gz 2024-01-11T09:48:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52726.log 2024-01-11T09:48:34.208 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.82872.log 2024-01-11T09:48:34.208 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.119389.log.gz 2024-01-11T09:48:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52726.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52726.log.gz 2024-01-11T09:48:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57153.log 2024-01-11T09:48:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.82872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.82872.log.gz 2024-01-11T09:48:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96989.log 2024-01-11T09:48:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57153.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57153.log.gz 2024-01-11T09:48:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71430.log 2024-01-11T09:48:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.96989.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.96989.log.gz 2024-01-11T09:48:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118078.log 2024-01-11T09:48:34.213 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71430.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.71430.log.gz 2024-01-11T09:48:34.213 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49861.log 2024-01-11T09:48:34.214 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.118078.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.118078.log.gz 2024-01-11T09:48:34.214 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49011.log 2024-01-11T09:48:34.215 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102505.log 2024-01-11T09:48:34.215 INFO:teuthology.orchestra.run.smithi027.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49861.log.gz 2024-01-11T09:48:34.216 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49011.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49011.log.gz 2024-01-11T09:48:34.216 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92980.log 2024-01-11T09:48:34.217 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.102505.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.102505.log.gz 2024-01-11T09:48:34.217 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54710.log 2024-01-11T09:48:34.218 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92980.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.92980.log.gz 2024-01-11T09:48:34.218 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67548.log 2024-01-11T09:48:34.219 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.54710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.54710.log.gz 2024-01-11T09:48:34.219 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79373.log 2024-01-11T09:48:34.220 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67548.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67548.log.gz 2024-01-11T09:48:34.220 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97038.log 2024-01-11T09:48:34.221 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79373.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79373.log.gz 2024-01-11T09:48:34.221 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73554.log 2024-01-11T09:48:34.222 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.97038.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.97038.log.gz 2024-01-11T09:48:34.223 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63802.log 2024-01-11T09:48:34.223 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73554.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73554.log.gz 2024-01-11T09:48:34.224 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119879.log 2024-01-11T09:48:34.224 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63802.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63802.log.gz 2024-01-11T09:48:34.225 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60902.log 2024-01-11T09:48:34.226 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57853.log 2024-01-11T09:48:34.226 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.119879.log.gz 2024-01-11T09:48:34.227 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60902.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60902.log.gz 2024-01-11T09:48:34.227 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78099.log 2024-01-11T09:48:34.228 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57853.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.57853.log.gz 2024-01-11T09:48:34.228 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74055.log 2024-01-11T09:48:34.229 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78099.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78099.log.gz 2024-01-11T09:48:34.229 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49616.log 2024-01-11T09:48:34.229 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74055.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74055.log.gz 2024-01-11T09:48:34.230 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71529.log 2024-01-11T09:48:34.231 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49616.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49616.log.gz 2024-01-11T09:48:34.231 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113446.log 2024-01-11T09:48:34.232 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71529.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.71529.log.gz 2024-01-11T09:48:34.232 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105195.log 2024-01-11T09:48:34.233 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.113446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110258.log 2024-01-11T09:48:34.233 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.113446.log.gz 2024-01-11T09:48:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.105195.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.105195.log.gz 2024-01-11T09:48:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101745.log 2024-01-11T09:48:34.235 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.110258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63292.log 2024-01-11T09:48:34.235 INFO:teuthology.orchestra.run.smithi027.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.110258.log.gz 2024-01-11T09:48:34.236 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.101745.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.101745.log.gz 2024-01-11T09:48:34.236 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113143.log 2024-01-11T09:48:34.237 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64474.log 2024-01-11T09:48:34.237 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63292.log.gz 2024-01-11T09:48:34.238 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.113143.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.113143.log.gz 2024-01-11T09:48:34.238 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83716.log 2024-01-11T09:48:34.239 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94237.log 2024-01-11T09:48:34.239 INFO:teuthology.orchestra.run.smithi027.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.64474.log.gz 2024-01-11T09:48:34.240 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83716.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.83716.log.gz 2024-01-11T09:48:34.240 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101596.log 2024-01-11T09:48:34.241 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.94237.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.94237.log.gz 2024-01-11T09:48:34.241 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64415.log 2024-01-11T09:48:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.101596.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.101596.log.gz 2024-01-11T09:48:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69126.log 2024-01-11T09:48:34.243 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64415.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64415.log.gz 2024-01-11T09:48:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53979.log 2024-01-11T09:48:34.245 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69126.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69126.log.gz 2024-01-11T09:48:34.245 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53773.log 2024-01-11T09:48:34.246 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108725.log 2024-01-11T09:48:34.246 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53979.log.gz 2024-01-11T09:48:34.246 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53773.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53773.log.gz 2024-01-11T09:48:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80446.log 2024-01-11T09:48:34.247 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.108725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119193.log 2024-01-11T09:48:34.248 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108725.log.gz 2024-01-11T09:48:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80446.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.80446.log.gz 2024-01-11T09:48:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70451.log 2024-01-11T09:48:34.250 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62018.log 2024-01-11T09:48:34.250 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.119193.log.gz 2024-01-11T09:48:34.251 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70451.log: gzip 89.4%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.70451.log.gz-- 2024-01-11T09:48:34.251 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.97087.log 2024-01-11T09:48:34.252 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77335.log 2024-01-11T09:48:34.252 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62018.log.gz 2024-01-11T09:48:34.253 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.97087.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.97087.log.gz 2024-01-11T09:48:34.253 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60554.log 2024-01-11T09:48:34.254 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47701.log 2024-01-11T09:48:34.254 INFO:teuthology.orchestra.run.smithi027.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77335.log.gz 2024-01-11T09:48:34.255 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60554.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60554.log.gz 2024-01-11T09:48:34.255 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102809.log 2024-01-11T09:48:34.256 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119585.log 2024-01-11T09:48:34.256 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47701.log.gz 2024-01-11T09:48:34.257 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.102809.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.102809.log.gz 2024-01-11T09:48:34.257 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108284.log 2024-01-11T09:48:34.258 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119585.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.119585.log.gz 2024-01-11T09:48:34.258 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.50668.log 2024-01-11T09:48:34.259 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.108284.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108284.log.gz 2024-01-11T09:48:34.259 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72406.log 2024-01-11T09:48:34.260 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.50668.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.50668.log.gz 2024-01-11T09:48:34.260 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106141.log 2024-01-11T09:48:34.261 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72406.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72406.log.gz 2024-01-11T09:48:34.261 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71765.log 2024-01-11T09:48:34.262 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.106141.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.106141.log.gz 2024-01-11T09:48:34.262 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112590.log 2024-01-11T09:48:34.263 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71765.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71765.log.gz 2024-01-11T09:48:34.263 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96297.log 2024-01-11T09:48:34.264 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.112590.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.112590.log.gz 2024-01-11T09:48:34.264 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117975.log 2024-01-11T09:48:34.265 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.96297.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.96297.log.gz 2024-01-11T09:48:34.265 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94164.log 2024-01-11T09:48:34.267 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.117975.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.117975.log.gz 2024-01-11T09:48:34.267 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85923.log 2024-01-11T09:48:34.268 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.94164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69028.log 2024-01-11T09:48:34.268 INFO:teuthology.orchestra.run.smithi027.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.94164.log.gz 2024-01-11T09:48:34.269 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85923.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.85923.log.gz 2024-01-11T09:48:34.269 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101439.log 2024-01-11T09:48:34.270 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69028.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69028.log.gz 2024-01-11T09:48:34.270 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111104.log 2024-01-11T09:48:34.271 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.101439.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.101439.log.gz 2024-01-11T09:48:34.271 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100028.log 2024-01-11T09:48:34.272 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.111104.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.111104.log.gz 2024-01-11T09:48:34.272 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84141.log 2024-01-11T09:48:34.273 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.100028.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.100028.log.gz 2024-01-11T09:48:34.273 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71305.log 2024-01-11T09:48:34.274 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84141.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84141.log.gz 2024-01-11T09:48:34.274 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92308.log 2024-01-11T09:48:34.275 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71305.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71305.log.gz 2024-01-11T09:48:34.275 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63973.log 2024-01-11T09:48:34.276 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92308.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.92308.log.gz 2024-01-11T09:48:34.277 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115117.log 2024-01-11T09:48:34.278 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63973.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63973.log.gz 2024-01-11T09:48:34.278 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68930.log 2024-01-11T09:48:34.279 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.115117.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.115117.log.gz 2024-01-11T09:48:34.279 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80237.log 2024-01-11T09:48:34.280 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68930.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68930.log.gz 2024-01-11T09:48:34.280 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.75798.log 2024-01-11T09:48:34.281 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80237.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.80237.log.gz 2024-01-11T09:48:34.281 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85772.log 2024-01-11T09:48:34.282 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.75798.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.75798.log.gz 2024-01-11T09:48:34.282 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68470.log 2024-01-11T09:48:34.283 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85772.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85772.log.gz 2024-01-11T09:48:34.283 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102578.log 2024-01-11T09:48:34.284 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68470.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68470.log.gz 2024-01-11T09:48:34.284 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68099.log 2024-01-11T09:48:34.285 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.102578.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.102578.log.gz 2024-01-11T09:48:34.285 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63094.log 2024-01-11T09:48:34.286 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68099.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68099.log.gz 2024-01-11T09:48:34.286 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92258.log 2024-01-11T09:48:34.287 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63094.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63094.log.gz 2024-01-11T09:48:34.288 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73185.log 2024-01-11T09:48:34.289 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92258.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.92258.log.gz 2024-01-11T09:48:34.289 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120107.log 2024-01-11T09:48:34.290 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73185.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73185.log.gz 2024-01-11T09:48:34.290 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111410.log 2024-01-11T09:48:34.291 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.120107.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.120107.log.gz 2024-01-11T09:48:34.291 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99303.log 2024-01-11T09:48:34.292 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.111410.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.111410.log.gz 2024-01-11T09:48:34.292 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101241.log 2024-01-11T09:48:34.293 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.99303.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.99303.log.gz 2024-01-11T09:48:34.293 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111804.log 2024-01-11T09:48:34.294 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.101241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75412.log 2024-01-11T09:48:34.294 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.101241.log.gz 2024-01-11T09:48:34.295 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.111804.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.111804.log.gz 2024-01-11T09:48:34.295 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85157.log 2024-01-11T09:48:34.296 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75412.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.75412.log.gz 2024-01-11T09:48:34.296 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71232.log 2024-01-11T09:48:34.297 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85157.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.85157.log.gz 2024-01-11T09:48:34.297 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119977.log 2024-01-11T09:48:34.298 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71232.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71232.log.gz 2024-01-11T09:48:34.298 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116228.log 2024-01-11T09:48:34.299 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119977.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.119977.log.gz 2024-01-11T09:48:34.299 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68050.log 2024-01-11T09:48:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.116228.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.116228.log.gz 2024-01-11T09:48:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64670.log 2024-01-11T09:48:34.301 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68050.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68050.log.gz 2024-01-11T09:48:34.301 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56206.log 2024-01-11T09:48:34.302 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64670.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64670.log.gz 2024-01-11T09:48:34.303 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107587.log 2024-01-11T09:48:34.304 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56206.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56206.log.gz 2024-01-11T09:48:34.304 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77850.log 2024-01-11T09:48:34.305 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.107587.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.107587.log.gz 2024-01-11T09:48:34.305 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86070.log 2024-01-11T09:48:34.306 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77850.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.77850.log.gz 2024-01-11T09:48:34.306 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49008.log 2024-01-11T09:48:34.307 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86070.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.86070.log.gz 2024-01-11T09:48:34.307 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62438.log 2024-01-11T09:48:34.308 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49008.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49008.log.gz 2024-01-11T09:48:34.308 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119732.log 2024-01-11T09:48:34.309 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62438.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62438.log.gz 2024-01-11T09:48:34.309 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70347.log 2024-01-11T09:48:34.310 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119732.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.119732.log.gz 2024-01-11T09:48:34.310 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-11T09:48:34.311 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70347.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70347.log.gz 2024-01-11T09:48:34.311 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108578.log 2024-01-11T09:48:34.312 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92149.log 2024-01-11T09:48:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.108578.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.108578.log.gz 2024-01-11T09:48:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62177.log 2024-01-11T09:48:34.314 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92149.log: gzip -5 88.6%--verbose -- replaced with /var/log/ceph/ceph-client.admin.92149.log.gz-- 2024-01-11T09:48:34.314 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.95346.log 2024-01-11T09:48:34.315 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62177.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62177.log.gz 2024-01-11T09:48:34.316 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51271.log 2024-01-11T09:48:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.95346.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.95346.log.gz 2024-01-11T09:48:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97136.log 2024-01-11T09:48:34.318 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51271.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51271.log.gz 2024-01-11T09:48:34.318 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49371.log 2024-01-11T09:48:34.319 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.97136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96485.log 2024-01-11T09:48:34.319 INFO:teuthology.orchestra.run.smithi027.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.97136.log.gz 2024-01-11T09:48:34.320 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49371.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49371.log.gz 2024-01-11T09:48:34.320 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95450.log 2024-01-11T09:48:34.322 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.96485.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96485.log.gz 2024-01-11T09:48:34.322 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105361.log 2024-01-11T09:48:34.323 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.95450.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95450.log.gz 2024-01-11T09:48:34.323 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116475.log 2024-01-11T09:48:34.324 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.105361.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.105361.log.gz 2024-01-11T09:48:34.324 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51811.log 2024-01-11T09:48:34.325 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.116475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52873.log 2024-01-11T09:48:34.325 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.116475.log.gz 2024-01-11T09:48:34.326 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51811.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51811.log.gz 2024-01-11T09:48:34.326 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114402.log 2024-01-11T09:48:34.328 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52873.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52873.log.gz 2024-01-11T09:48:34.328 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69273.log 2024-01-11T09:48:34.329 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.114402.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.114402.log.gz 2024-01-11T09:48:34.329 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87232.log 2024-01-11T09:48:34.330 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69273.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.69273.log.gz 2024-01-11T09:48:34.330 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75216.log 2024-01-11T09:48:34.331 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87232.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.87232.log.gz 2024-01-11T09:48:34.331 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82343.log 2024-01-11T09:48:34.333 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75216.log: gzip 89.1%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.75216.log.gz-- 2024-01-11T09:48:34.333 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.93968.log 2024-01-11T09:48:34.333 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82343.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82343.log.gz 2024-01-11T09:48:34.334 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114347.log 2024-01-11T09:48:34.335 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.93968.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.93968.log.gz 2024-01-11T09:48:34.335 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53477.log 2024-01-11T09:48:34.343 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.114347.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.114347.log.gz 2024-01-11T09:48:34.343 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51370.log 2024-01-11T09:48:34.343 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53477.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53477.log.gz 2024-01-11T09:48:34.343 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118132.log 2024-01-11T09:48:34.344 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51370.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51370.log.gz 2024-01-11T09:48:34.344 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74712.log 2024-01-11T09:48:34.344 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.118132.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.118132.log.gzgzip 2024-01-11T09:48:34.344 INFO:teuthology.orchestra.run.smithi027.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.62857.log 2024-01-11T09:48:34.344 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74712.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74712.log.gz 2024-01-11T09:48:34.344 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113664.log 2024-01-11T09:48:34.344 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62857.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62857.log.gz 2024-01-11T09:48:34.344 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109003.log 2024-01-11T09:48:34.345 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.113664.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.113664.log.gz 2024-01-11T09:48:34.345 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74251.log 2024-01-11T09:48:34.345 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109003.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.109003.log.gz 2024-01-11T09:48:34.345 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-01-11T09:48:34.346 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74251.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74251.log.gz 2024-01-11T09:48:34.346 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112849.log 2024-01-11T09:48:34.348 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92931.log 2024-01-11T09:48:34.349 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.112849.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.112849.log.gz 2024-01-11T09:48:34.349 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100240.log 2024-01-11T09:48:34.350 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92931.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.92931.log.gz 2024-01-11T09:48:34.350 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74525.log 2024-01-11T09:48:34.351 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.100240.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.100240.log.gz 2024-01-11T09:48:34.351 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49014.log 2024-01-11T09:48:34.352 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74525.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.74525.log.gz 2024-01-11T09:48:34.353 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49273.log 2024-01-11T09:48:34.354 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49014.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49014.log.gz 2024-01-11T09:48:34.354 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117868.log 2024-01-11T09:48:34.355 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49273.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49273.log.gz 2024-01-11T09:48:34.356 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86315.log 2024-01-11T09:48:34.357 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.117868.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.117868.log.gz 2024-01-11T09:48:34.357 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105992.log 2024-01-11T09:48:34.358 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86315.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86315.log.gz 2024-01-11T09:48:34.358 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97283.log 2024-01-11T09:48:34.360 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.105992.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.105992.log.gz 2024-01-11T09:48:34.360 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74426.log 2024-01-11T09:48:34.361 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.97283.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97283.log.gz 2024-01-11T09:48:34.361 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85255.log 2024-01-11T09:48:34.362 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74426.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74426.log.gz 2024-01-11T09:48:34.362 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120060.log 2024-01-11T09:48:34.363 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85255.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85255.log.gz 2024-01-11T09:48:34.363 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83054.log 2024-01-11T09:48:34.364 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.120060.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.120060.log.gz 2024-01-11T09:48:34.365 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63753.log 2024-01-11T09:48:34.366 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83054.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83054.log.gz 2024-01-11T09:48:34.366 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70189.log 2024-01-11T09:48:34.367 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63753.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63753.log.gz 2024-01-11T09:48:34.367 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105682.log 2024-01-11T09:48:34.369 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70189.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70189.log.gz 2024-01-11T09:48:34.369 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63045.log 2024-01-11T09:48:34.370 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.105682.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.105682.log.gz 2024-01-11T09:48:34.370 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69995.log 2024-01-11T09:48:34.371 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63045.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63045.log.gz 2024-01-11T09:48:34.371 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74810.log 2024-01-11T09:48:34.372 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69995.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69995.log.gz 2024-01-11T09:48:34.373 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78406.log 2024-01-11T09:48:34.373 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74810.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74810.log.gz 2024-01-11T09:48:34.374 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115291.log 2024-01-11T09:48:34.375 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78406.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.78406.log.gz 2024-01-11T09:48:34.375 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106496.log 2024-01-11T09:48:34.376 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.115291.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.115291.log.gz 2024-01-11T09:48:34.376 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115068.log 2024-01-11T09:48:34.377 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.106496.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.106496.log.gz 2024-01-11T09:48:34.377 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48606.log 2024-01-11T09:48:34.378 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.115068.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.115068.log.gz 2024-01-11T09:48:34.378 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99254.log 2024-01-11T09:48:34.380 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48606.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48606.log.gz 2024-01-11T09:48:34.380 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101103.log 2024-01-11T09:48:34.381 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.99254.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.99254.log.gz 2024-01-11T09:48:34.381 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86168.log 2024-01-11T09:48:34.382 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.101103.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.101103.log.gz 2024-01-11T09:48:34.382 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78257.log 2024-01-11T09:48:34.383 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86168.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.86168.log.gz 2024-01-11T09:48:34.383 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57709.log 2024-01-11T09:48:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78257.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78257.log.gz 2024-01-11T09:48:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52775.log 2024-01-11T09:48:34.386 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57709.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.57709.log.gz 2024-01-11T09:48:34.386 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117466.log 2024-01-11T09:48:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52775.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52775.log.gz 2024-01-11T09:48:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67444.log 2024-01-11T09:48:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.117466.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.117466.log.gz 2024-01-11T09:48:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74202.log 2024-01-11T09:48:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67444.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67444.log.gz 2024-01-11T09:48:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116671.log 2024-01-11T09:48:34.390 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74202.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74202.log.gz 2024-01-11T09:48:34.391 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76035.log 2024-01-11T09:48:34.392 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.116671.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.116671.log.gz 2024-01-11T09:48:34.392 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100732.log 2024-01-11T09:48:34.393 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76035.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.76035.log.gz 2024-01-11T09:48:34.394 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106298.log 2024-01-11T09:48:34.395 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.100732.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.100732.log.gz 2024-01-11T09:48:34.395 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55790.log 2024-01-11T09:48:34.396 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.106298.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.106298.log.gz 2024-01-11T09:48:34.396 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119928.log 2024-01-11T09:48:34.397 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55790.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55790.log.gz 2024-01-11T09:48:34.398 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63192.log 2024-01-11T09:48:34.398 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119928.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.119928.log.gz 2024-01-11T09:48:34.399 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104648.log 2024-01-11T09:48:34.399 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63192.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63192.log.gz 2024-01-11T09:48:34.400 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49812.log 2024-01-11T09:48:34.401 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.104648.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.104648.log.gz 2024-01-11T09:48:34.401 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115019.log 2024-01-11T09:48:34.402 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49812.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49812.log.gz 2024-01-11T09:48:34.402 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62683.log 2024-01-11T09:48:34.403 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.115019.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.115019.log.gz 2024-01-11T09:48:34.404 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112692.log 2024-01-11T09:48:34.405 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62683.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62683.log.gz 2024-01-11T09:48:34.405 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78208.log 2024-01-11T09:48:34.406 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.112692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75265.log 2024-01-11T09:48:34.406 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.112692.log.gz 2024-01-11T09:48:34.407 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78208.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.78208.log.gz 2024-01-11T09:48:34.407 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52432.log 2024-01-11T09:48:34.409 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75265.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75265.log.gz 2024-01-11T09:48:34.409 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56996.log 2024-01-11T09:48:34.409 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52432.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52432.log.gz 2024-01-11T09:48:34.410 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72905.log 2024-01-11T09:48:34.411 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56996.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56996.log.gz 2024-01-11T09:48:34.411 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106041.log 2024-01-11T09:48:34.412 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72905.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72905.log.gz 2024-01-11T09:48:34.412 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106692.log 2024-01-11T09:48:34.413 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.106041.log: gzip -5 88.2%--verbose -- replaced with /var/log/ceph/ceph-client.admin.106041.log.gz-- 2024-01-11T09:48:34.413 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.95399.log 2024-01-11T09:48:34.414 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.106692.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.106692.log.gz 2024-01-11T09:48:34.415 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71085.log 2024-01-11T09:48:34.416 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.95399.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95399.log.gz 2024-01-11T09:48:34.416 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84720.log 2024-01-11T09:48:34.417 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71085.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71085.log.gz 2024-01-11T09:48:34.417 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77104.log 2024-01-11T09:48:34.418 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84720.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84720.log.gz 2024-01-11T09:48:34.418 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68606.log 2024-01-11T09:48:34.419 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77104.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.77104.log.gz 2024-01-11T09:48:34.419 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.86774.log 2024-01-11T09:48:34.421 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97234.log 2024-01-11T09:48:34.421 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68606.log.gz 2024-01-11T09:48:34.421 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.86774.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.86774.log.gz 2024-01-11T09:48:34.422 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52922.log 2024-01-11T09:48:34.423 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.97234.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.97234.log.gz 2024-01-11T09:48:34.423 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119340.log 2024-01-11T09:48:34.424 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52922.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52922.log.gz 2024-01-11T09:48:34.424 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71134.log 2024-01-11T09:48:34.425 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119340.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.119340.log.gz 2024-01-11T09:48:34.426 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113265.log 2024-01-11T09:48:34.427 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71134.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71134.log.gz 2024-01-11T09:48:34.427 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70069.log 2024-01-11T09:48:34.428 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.113265.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113265.log.gz 2024-01-11T09:48:34.428 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67389.log 2024-01-11T09:48:34.429 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70069.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70069.log.gz 2024-01-11T09:48:34.429 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107883.log 2024-01-11T09:48:34.431 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67389.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67389.log.gz 2024-01-11T09:48:34.431 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49469.log 2024-01-11T09:48:34.432 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.107883.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.107883.log.gz 2024-01-11T09:48:34.432 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93029.log 2024-01-11T09:48:34.433 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49469.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49469.log.gz 2024-01-11T09:48:34.433 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51762.log 2024-01-11T09:48:34.434 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.93029.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.93029.log.gz 2024-01-11T09:48:34.434 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98797.log 2024-01-11T09:48:34.435 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83812.log 2024-01-11T09:48:34.435 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51762.log.gz 2024-01-11T09:48:34.436 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.98797.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.98797.log.gz 2024-01-11T09:48:34.437 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41834.log 2024-01-11T09:48:34.438 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83812.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.83812.log.gz 2024-01-11T09:48:34.438 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59858.log 2024-01-11T09:48:34.439 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41834.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41834.log.gz 2024-01-11T09:48:34.439 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96050.log 2024-01-11T09:48:34.440 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107736.log 2024-01-11T09:48:34.440 INFO:teuthology.orchestra.run.smithi027.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59858.log.gz 2024-01-11T09:48:34.441 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.96050.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96050.log.gz 2024-01-11T09:48:34.441 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72259.log 2024-01-11T09:48:34.443 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.107736.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.107736.log.gz 2024-01-11T09:48:34.443 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60206.log 2024-01-11T09:48:34.444 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72259.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72259.log.gz 2024-01-11T09:48:34.444 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61598.log 2024-01-11T09:48:34.445 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54177.log 2024-01-11T09:48:34.445 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60206.log.gz 2024-01-11T09:48:34.446 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61598.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61598.log.gz 2024-01-11T09:48:34.446 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.75794.log 2024-01-11T09:48:34.447 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54177.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54177.log.gz 2024-01-11T09:48:34.448 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54226.log 2024-01-11T09:48:34.448 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.75794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.75794.log.gz 2024-01-11T09:48:34.449 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113873.log 2024-01-11T09:48:34.450 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54226.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54226.log.gz 2024-01-11T09:48:34.450 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.112236.log 2024-01-11T09:48:34.451 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.113873.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.113873.log.gz 2024-01-11T09:48:34.451 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75118.log 2024-01-11T09:48:34.452 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.112236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.112236.log.gz 2024-01-11T09:48:34.452 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86879.log 2024-01-11T09:48:34.453 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75118.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75118.log.gz 2024-01-11T09:48:34.454 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106349.log 2024-01-11T09:48:34.455 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73735.log 2024-01-11T09:48:34.455 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86879.log.gz 2024-01-11T09:48:34.456 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.106349.log: 93.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-01-11T09:48:34.456 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.106349.log.gz 2024-01-11T09:48:34.456 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69224.log 2024-01-11T09:48:34.457 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73735.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73735.log.gz 2024-01-11T09:48:34.457 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73342.log 2024-01-11T09:48:34.458 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69224.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69224.log.gz 2024-01-11T09:48:34.459 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114455.log 2024-01-11T09:48:34.460 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73342.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73342.log.gz 2024-01-11T09:48:34.460 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-11T09:48:34.464 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.114455.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.114455.log.gz 2024-01-11T09:48:34.464 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111302.log 2024-01-11T09:48:34.464 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49322.log 2024-01-11T09:48:34.464 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.111302.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.111302.log.gz 2024-01-11T09:48:34.464 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113821.log 2024-01-11T09:48:34.465 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49322.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49322.log.gz 2024-01-11T09:48:34.465 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114872.log 2024-01-11T09:48:34.466 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.113821.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.113821.log.gz 2024-01-11T09:48:34.467 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85108.log 2024-01-11T09:48:34.467 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.114872.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114872.log.gz 2024-01-11T09:48:34.468 INFO:teuthology.orchestra.run.smithi027.stderr: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-11T09:48:34.468 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72210.log 2024-01-11T09:48:34.469 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85108.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.85108.log.gz 2024-01-11T09:48:34.469 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108529.log 2024-01-11T09:48:34.470 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72210.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72210.log.gz 2024-01-11T09:48:34.470 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55525.log 2024-01-11T09:48:34.471 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.108529.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.108529.log.gz 2024-01-11T09:48:34.472 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78724.log 2024-01-11T09:48:34.473 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55525.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55525.log.gz 2024-01-11T09:48:34.473 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107232.log 2024-01-11T09:48:34.474 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75069.log 2024-01-11T09:48:34.474 INFO:teuthology.orchestra.run.smithi027.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.78724.log.gz 2024-01-11T09:48:34.475 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.107232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101192.log 2024-01-11T09:48:34.475 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.107232.log.gz 2024-01-11T09:48:34.476 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75069.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.75069.log.gz 2024-01-11T09:48:34.476 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86119.log 2024-01-11T09:48:34.478 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.101192.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.101192.log.gz 2024-01-11T09:48:34.478 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119830.log 2024-01-11T09:48:34.479 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86119.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86119.log.gz 2024-01-11T09:48:34.479 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110966.log 2024-01-11T09:48:34.480 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51419.log 2024-01-11T09:48:34.480 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.119830.log.gz 2024-01-11T09:48:34.481 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.110966.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.110966.log.gz 2024-01-11T09:48:34.481 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49420.log 2024-01-11T09:48:34.483 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51419.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51419.log.gz 2024-01-11T09:48:34.483 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100681.log 2024-01-11T09:48:34.484 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49420.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49420.log.gz 2024-01-11T09:48:34.484 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99182.log 2024-01-11T09:48:34.485 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.100681.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.100681.log.gz 2024-01-11T09:48:34.485 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.86770.log 2024-01-11T09:48:34.486 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.99182.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.99182.log.gz 2024-01-11T09:48:34.486 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77490.log 2024-01-11T09:48:34.487 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.86770.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.86770.log.gz 2024-01-11T09:48:34.487 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80119.log 2024-01-11T09:48:34.489 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77490.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.77490.log.gz 2024-01-11T09:48:34.489 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75976.log 2024-01-11T09:48:34.490 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81790.log 2024-01-11T09:48:34.490 INFO:teuthology.orchestra.run.smithi027.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80119.log.gz 2024-01-11T09:48:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75976.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75976.log.gz 2024-01-11T09:48:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99766.log 2024-01-11T09:48:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81790.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81790.log.gz 2024-01-11T09:48:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85059.log 2024-01-11T09:48:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.99766.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.99766.log.gz 2024-01-11T09:48:34.494 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92046.log 2024-01-11T09:48:34.495 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85059.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.85059.log.gz 2024-01-11T09:48:34.495 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109107.log 2024-01-11T09:48:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92046.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.92046.log.gz 2024-01-11T09:48:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47295.log 2024-01-11T09:48:34.497 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109107.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109107.log.gz 2024-01-11T09:48:34.498 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110473.log 2024-01-11T09:48:34.498 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47295.log: gzip -5 88.1%--verbose -- replaced with /var/log/ceph/ceph-client.admin.47295.log.gz-- 2024-01-11T09:48:34.499 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62585.log 2024-01-11T09:48:34.500 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.110473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97774.log 2024-01-11T09:48:34.500 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.110473.log.gz 2024-01-11T09:48:34.501 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62585.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62585.log.gz 2024-01-11T09:48:34.501 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47901.log 2024-01-11T09:48:34.502 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.97774.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.97774.log.gz 2024-01-11T09:48:34.502 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51320.log 2024-01-11T09:48:34.503 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47901.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47901.log.gz 2024-01-11T09:48:34.503 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118997.log 2024-01-11T09:48:34.505 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51320.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51320.log.gz 2024-01-11T09:48:34.505 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101696.log 2024-01-11T09:48:34.506 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.118997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81643.log 2024-01-11T09:48:34.506 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.118997.log.gz 2024-01-11T09:48:34.507 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.101696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104859.log 2024-01-11T09:48:34.507 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.101696.log.gz 2024-01-11T09:48:34.508 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81643.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.81643.log.gz 2024-01-11T09:48:34.508 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94738.log 2024-01-11T09:48:34.510 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.104859.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.104859.log.gz 2024-01-11T09:48:34.510 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71814.log 2024-01-11T09:48:34.511 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.94738.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.94738.log.gz 2024-01-11T09:48:34.511 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52007.log 2024-01-11T09:48:34.512 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71814.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71814.log.gz 2024-01-11T09:48:34.512 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111353.log 2024-01-11T09:48:34.513 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52007.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52007.log.gz 2024-01-11T09:48:34.513 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48244.log 2024-01-11T09:48:34.515 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.111353.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.111353.log.gz 2024-01-11T09:48:34.515 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80606.log 2024-01-11T09:48:34.516 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48244.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48244.log.gz 2024-01-11T09:48:34.516 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51664.log 2024-01-11T09:48:34.517 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80606.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80606.log.gz 2024-01-11T09:48:34.517 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118744.log 2024-01-11T09:48:34.518 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51664.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51664.log.gz 2024-01-11T09:48:34.518 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112898.log 2024-01-11T09:48:34.519 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.118744.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.118744.log.gz 2024-01-11T09:48:34.519 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68881.log 2024-01-11T09:48:34.520 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.112898.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112898.log.gz 2024-01-11T09:48:34.520 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110620.log 2024-01-11T09:48:34.522 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68881.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68881.log.gz 2024-01-11T09:48:34.522 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67497.log 2024-01-11T09:48:34.523 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.110620.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.110620.log.gz 2024-01-11T09:48:34.523 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95030.log 2024-01-11T09:48:34.524 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67497.log: gzip -5 88.7%--verbose -- replaced with /var/log/ceph/ceph-client.admin.67497.log.gz-- 2024-01-11T09:48:34.524 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.107685.log 2024-01-11T09:48:34.525 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.95030.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95030.log.gz 2024-01-11T09:48:34.525 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67767.log 2024-01-11T09:48:34.527 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.107685.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.107685.log.gz 2024-01-11T09:48:34.527 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50738.log 2024-01-11T09:48:34.528 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67767.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67767.log.gz 2024-01-11T09:48:34.528 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74861.log 2024-01-11T09:48:34.529 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100781.log 2024-01-11T09:48:34.529 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50738.log.gz 2024-01-11T09:48:34.530 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74861.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74861.log.gz 2024-01-11T09:48:34.530 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84614.log 2024-01-11T09:48:34.532 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.100781.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.100781.log.gz 2024-01-11T09:48:34.532 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62282.log 2024-01-11T09:48:34.533 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84614.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84614.log.gz 2024-01-11T09:48:34.533 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49763.log 2024-01-11T09:48:34.534 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62282.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62282.log.gz 2024-01-11T09:48:34.534 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104699.log 2024-01-11T09:48:34.535 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49763.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49763.log.gz 2024-01-11T09:48:34.535 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49004.log 2024-01-11T09:48:34.536 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.104699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96681.log 2024-01-11T09:48:34.537 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.104699.log.gz 2024-01-11T09:48:34.537 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49004.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49004.log.gz 2024-01-11T09:48:34.538 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109425.log 2024-01-11T09:48:34.539 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.96681.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96681.log.gz 2024-01-11T09:48:34.539 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86952.log 2024-01-11T09:48:34.540 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109425.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.109425.log.gz 2024-01-11T09:48:34.540 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83113.log 2024-01-11T09:48:34.541 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86952.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86952.log.gz 2024-01-11T09:48:34.541 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92686.log 2024-01-11T09:48:34.542 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83113.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83113.log.gz 2024-01-11T09:48:34.543 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74969.log 2024-01-11T09:48:34.544 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92686.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.92686.log.gz 2024-01-11T09:48:34.544 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85388.log 2024-01-11T09:48:34.545 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74969.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74969.log.gz 2024-01-11T09:48:34.545 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52824.log 2024-01-11T09:48:34.546 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67285.log 2024-01-11T09:48:34.546 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85388.log.gz 2024-01-11T09:48:34.547 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52824.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52824.log.gz 2024-01-11T09:48:34.548 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75363.log 2024-01-11T09:48:34.549 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67285.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67285.log.gz 2024-01-11T09:48:34.549 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118646.log 2024-01-11T09:48:34.550 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75363.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75363.log.gz 2024-01-11T09:48:34.550 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78357.log 2024-01-11T09:48:34.551 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.118646.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.118646.log.gz 2024-01-11T09:48:34.551 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48618.log 2024-01-11T09:48:34.552 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78357.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78357.log.gz 2024-01-11T09:48:34.552 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84298.log 2024-01-11T09:48:34.554 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48618.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48618.log.gz 2024-01-11T09:48:34.554 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55741.log 2024-01-11T09:48:34.555 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84298.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84298.log.gz 2024-01-11T09:48:34.555 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-01-11T09:48:34.556 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55741.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55741.log.gz 2024-01-11T09:48:34.556 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72014.log 2024-01-11T09:48:34.557 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76313.log 2024-01-11T09:48:34.559 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72014.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72014.log.gz 2024-01-11T09:48:34.559 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56010.log 2024-01-11T09:48:34.560 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92468.log 2024-01-11T09:48:34.560 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76313.log.gz 2024-01-11T09:48:34.561 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56010.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56010.log.gz 2024-01-11T09:48:34.561 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67952.log 2024-01-11T09:48:34.562 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92468.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.92468.log.gz 2024-01-11T09:48:34.562 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106643.log 2024-01-11T09:48:34.564 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67952.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.67952.log.gz 2024-01-11T09:48:34.564 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84827.log 2024-01-11T09:48:34.565 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.106643.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.106643.log.gz 2024-01-11T09:48:34.565 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48993.log 2024-01-11T09:48:34.566 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84827.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.84827.log.gz 2024-01-11T09:48:34.566 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100523.log 2024-01-11T09:48:34.567 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48993.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48993.log.gz 2024-01-11T09:48:34.567 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115922.log 2024-01-11T09:48:34.569 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.100523.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.100523.log.gz 2024-01-11T09:48:34.569 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85576.log 2024-01-11T09:48:34.570 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.115922.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.115922.log.gz 2024-01-11T09:48:34.570 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72528.log 2024-01-11T09:48:34.571 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84350.log 2024-01-11T09:48:34.571 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85576.log.gz 2024-01-11T09:48:34.572 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72528.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72528.log.gz 2024-01-11T09:48:34.572 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82441.log 2024-01-11T09:48:34.574 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84350.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.84350.log.gz 2024-01-11T09:48:34.574 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55237.log 2024-01-11T09:48:34.575 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82441.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82441.log.gz 2024-01-11T09:48:34.575 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120156.log 2024-01-11T09:48:34.576 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55237.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55237.log.gz 2024-01-11T09:48:34.576 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53206.log 2024-01-11T09:48:34.577 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.120156.log: gzip -5 88.9%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.120156.log.gz/var/log/ceph/ceph-client.admin.105459.log 2024-01-11T09:48:34.577 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-01-11T09:48:34.578 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53206.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53206.log.gz 2024-01-11T09:48:34.578 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84666.log 2024-01-11T09:48:34.579 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.105459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113387.log 89.1% 2024-01-11T09:48:34.580 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.105459.log.gz 2024-01-11T09:48:34.581 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84666.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.84666.log.gz 2024-01-11T09:48:34.581 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56157.log 2024-01-11T09:48:34.582 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.113387.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.113387.log.gz 2024-01-11T09:48:34.582 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93440.log 2024-01-11T09:48:34.583 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56157.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56157.log.gz 2024-01-11T09:48:34.583 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100398.log 2024-01-11T09:48:34.584 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.93440.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.93440.log.gz 2024-01-11T09:48:34.584 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107183.log 2024-01-11T09:48:34.586 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.100398.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.100398.log.gz 2024-01-11T09:48:34.586 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77384.log 2024-01-11T09:48:34.587 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.107183.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.107183.log.gz 2024-01-11T09:48:34.587 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78151.log 2024-01-11T09:48:34.588 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77384.log: gzip -5 88.1%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.77384.log.gz/var/log/ceph/ceph-client.admin.48097.log 2024-01-11T09:48:34.588 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-01-11T09:48:34.589 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78151.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.78151.log.gz 2024-01-11T09:48:34.590 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.112240.log 2024-01-11T09:48:34.591 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48097.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48097.log.gz 2024-01-11T09:48:34.591 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114676.log 2024-01-11T09:48:34.592 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.112240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.112240.log.gz 2024-01-11T09:48:34.592 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80710.log 2024-01-11T09:48:34.593 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.114676.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.114676.log.gz 2024-01-11T09:48:34.594 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118695.log 2024-01-11T09:48:34.595 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80710.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80710.log.gz 2024-01-11T09:48:34.595 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76946.log 2024-01-11T09:48:34.596 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.118695.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.118695.log.gz 2024-01-11T09:48:34.596 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82047.log 2024-01-11T09:48:34.597 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76946.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76946.log.gz 2024-01-11T09:48:34.597 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85723.log 2024-01-11T09:48:34.598 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82047.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.82047.log.gz 2024-01-11T09:48:34.598 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109907.log 2024-01-11T09:48:34.599 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85723.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.85723.log.gz 2024-01-11T09:48:34.599 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77177.log 2024-01-11T09:48:34.601 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109319.log 2024-01-11T09:48:34.601 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.109907.log.gz 2024-01-11T09:48:34.602 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77177.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.77177.log.gz 2024-01-11T09:48:34.602 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116720.log 2024-01-11T09:48:34.603 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109319.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.109319.log.gz 2024-01-11T09:48:34.603 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-11T09:48:34.604 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.116720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110005.log 2024-01-11T09:48:34.604 INFO:teuthology.orchestra.run.smithi027.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.116720.log.gz 2024-01-11T09:48:34.605 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118548.log 2024-01-11T09:48:34.607 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.110005.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.110005.log.gz 2024-01-11T09:48:34.607 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117564.log 2024-01-11T09:48:34.608 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.118548.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.118548.log.gz 2024-01-11T09:48:34.609 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78504.log 2024-01-11T09:48:34.610 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.117564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116277.log 2024-01-11T09:48:34.610 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.117564.log.gz 2024-01-11T09:48:34.611 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78504.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78504.log.gz 2024-01-11T09:48:34.611 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109266.log 2024-01-11T09:48:34.615 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.116277.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.116277.log.gz 2024-01-11T09:48:34.615 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109215.log 2024-01-11T09:48:34.615 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109266.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.109266.log.gz 2024-01-11T09:48:34.615 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54128.log 2024-01-11T09:48:34.616 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109215.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.109215.log.gz 2024-01-11T09:48:34.617 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108480.log 2024-01-11T09:48:34.617 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54128.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54128.log.gz 2024-01-11T09:48:34.618 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93919.log 2024-01-11T09:48:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.108480.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108480.log.gz 2024-01-11T09:48:34.620 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84402.log 2024-01-11T09:48:34.621 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.93919.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.93919.log.gz 2024-01-11T09:48:34.621 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115340.log 2024-01-11T09:48:34.623 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84402.log: gzip 87.6%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.84402.log.gz-- 2024-01-11T09:48:34.623 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109858.log 2024-01-11T09:48:34.624 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.115340.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.115340.log.gz 2024-01-11T09:48:34.624 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41475.log 2024-01-11T09:48:34.626 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84901.log 2024-01-11T09:48:34.626 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.109858.log.gz 2024-01-11T09:48:34.628 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41475.log: gzip 92.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.41475.log.gz--verbose 2024-01-11T09:48:34.628 INFO:teuthology.orchestra.run.smithi027.stderr:-- /var/log/ceph/ceph-client.admin.74349.log 2024-01-11T09:48:34.629 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84901.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84901.log.gz 2024-01-11T09:48:34.630 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118290.log 2024-01-11T09:48:34.630 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74349.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74349.log.gz 2024-01-11T09:48:34.631 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115438.log 2024-01-11T09:48:34.632 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.118290.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.118290.log.gz 2024-01-11T09:48:34.632 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101339.log 2024-01-11T09:48:34.634 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.115438.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.115438.log.gz 2024-01-11T09:48:34.634 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119046.log 2024-01-11T09:48:34.635 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.101339.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.101339.log.gz 2024-01-11T09:48:34.636 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109055.log 2024-01-11T09:48:34.637 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119046.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.119046.log.gz 2024-01-11T09:48:34.637 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53930.log 2024-01-11T09:48:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109055.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.109055.log.gz 2024-01-11T09:48:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85874.log 2024-01-11T09:48:34.641 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78553.log 2024-01-11T09:48:34.641 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53930.log.gz 2024-01-11T09:48:34.642 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85874.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.85874.log.gz 2024-01-11T09:48:34.642 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95503.log 2024-01-11T09:48:34.644 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78553.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.78553.log.gz 2024-01-11T09:48:34.644 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99976.log 2024-01-11T09:48:34.645 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.95503.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.95503.log.gz 2024-01-11T09:48:34.645 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76569.log 2024-01-11T09:48:34.647 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.99976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109371.log 2024-01-11T09:48:34.647 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.99976.log.gz 2024-01-11T09:48:34.648 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76569.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.76569.log.gz 2024-01-11T09:48:34.648 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112394.log 2024-01-11T09:48:34.650 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67707.log 2024-01-11T09:48:34.650 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.109371.log.gz 2024-01-11T09:48:34.650 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.112394.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112394.log.gz 2024-01-11T09:48:34.651 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86217.log 2024-01-11T09:48:34.652 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91889.log 2024-01-11T09:48:34.652 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67707.log.gz 2024-01-11T09:48:34.653 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86217.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86217.log.gz 2024-01-11T09:48:34.654 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104439.log 2024-01-11T09:48:34.655 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.91889.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.91889.log.gz 2024-01-11T09:48:34.655 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57421.log 2024-01-11T09:48:34.657 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.104439.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.104439.log.gz 2024-01-11T09:48:34.657 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54706.log 2024-01-11T09:48:34.658 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57421.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57421.log.gz 2024-01-11T09:48:34.658 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105254.log 2024-01-11T09:48:34.660 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.54706.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.54706.log.gz 2024-01-11T09:48:34.660 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105122.log 2024-01-11T09:48:34.661 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.105254.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.105254.log.gz 2024-01-11T09:48:34.661 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84457.log 2024-01-11T09:48:34.662 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.105122.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.105122.log.gz 2024-01-11T09:48:34.663 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83908.log 2024-01-11T09:48:34.664 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68001.log 2024-01-11T09:48:34.668 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.84457.log.gz 2024-01-11T09:48:34.668 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83908.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.83908.log.gz 2024-01-11T09:48:34.669 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114033.log 2024-01-11T09:48:34.669 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68001.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68001.log.gz 2024-01-11T09:48:34.669 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57300.log 2024-01-11T09:48:34.669 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.114033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119242.log 2024-01-11T09:48:34.669 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.114033.log.gz 2024-01-11T09:48:34.670 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57300.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57300.log.gz 2024-01-11T09:48:34.670 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.50682.log 2024-01-11T09:48:34.672 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48614.log 2024-01-11T09:48:34.672 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.119242.log.gz 2024-01-11T09:48:34.673 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.50682.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.50682.log.gz 2024-01-11T09:48:34.673 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100292.log 2024-01-11T09:48:34.674 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48614.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48614.log.gz 2024-01-11T09:48:34.675 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81480.log 2024-01-11T09:48:34.676 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.100292.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.100292.log.gz 2024-01-11T09:48:34.677 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53824.log 2024-01-11T09:48:34.678 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81480.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81480.log.gz 2024-01-11T09:48:34.678 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97332.log 2024-01-11T09:48:34.679 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53824.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53824.log.gz 2024-01-11T09:48:34.680 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118238.log 2024-01-11T09:48:34.682 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.97332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63851.log 2024-01-11T09:48:34.682 INFO:teuthology.orchestra.run.smithi027.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.97332.log.gz 2024-01-11T09:48:34.683 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.118238.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.118238.log.gz 2024-01-11T09:48:34.683 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47999.log 2024-01-11T09:48:34.684 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63851.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63851.log.gz 2024-01-11T09:48:34.685 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61860.log 2024-01-11T09:48:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47999.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47999.log.gz 2024-01-11T09:48:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68657.log 2024-01-11T09:48:34.688 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82392.log 2024-01-11T09:48:34.688 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61860.log.gz 2024-01-11T09:48:34.689 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68657.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68657.log.gz 2024-01-11T09:48:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118367.log 2024-01-11T09:48:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82392.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82392.log.gz 2024-01-11T09:48:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47750.log 2024-01-11T09:48:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.118367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108225.log 2024-01-11T09:48:34.693 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.118367.log.gz 2024-01-11T09:48:34.694 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47750.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47750.log.gz 2024-01-11T09:48:34.694 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.51215.log 2024-01-11T09:48:34.695 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.108225.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.108225.log.gz 2024-01-11T09:48:34.696 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116793.log 2024-01-11T09:48:34.696 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.2.51215.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.51215.log.gz 2024-01-11T09:48:34.697 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57493.log 2024-01-11T09:48:34.698 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.116793.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.116793.log.gz 2024-01-11T09:48:34.699 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106192.log 2024-01-11T09:48:34.700 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57493.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57493.log.gz 2024-01-11T09:48:34.700 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105805.log 2024-01-11T09:48:34.702 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.106192.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.106192.log.gz 2024-01-11T09:48:34.702 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80553.log 2024-01-11T09:48:34.703 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.105805.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.105805.log.gz 2024-01-11T09:48:34.704 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92784.log 2024-01-11T09:48:34.705 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80553.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80553.log.gz 2024-01-11T09:48:34.705 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86021.log 2024-01-11T09:48:34.707 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92784.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.92784.log.gz 2024-01-11T09:48:34.707 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63400.log 2024-01-11T09:48:34.708 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86021.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86021.log.gz 2024-01-11T09:48:34.709 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48609.log 2024-01-11T09:48:34.710 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63400.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63400.log.gz 2024-01-11T09:48:34.710 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.107078.log 2024-01-11T09:48:34.712 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48609.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48609.log.gz 2024-01-11T09:48:34.712 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96940.log 2024-01-11T09:48:34.713 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.107078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.107078.log.gz 2024-01-11T09:48:34.714 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111903.log 2024-01-11T09:48:34.715 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.96940.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.96940.log.gz 2024-01-11T09:48:34.715 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85972.log 2024-01-11T09:48:34.717 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.111903.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.111903.log.gz 2024-01-11T09:48:34.717 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114084.log 2024-01-11T09:48:34.719 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85972.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85972.log.gz 2024-01-11T09:48:34.719 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67892.log 2024-01-11T09:48:34.720 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.114084.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.114084.log.gz 2024-01-11T09:48:34.720 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49665.log 2024-01-11T09:48:34.722 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67892.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67892.log.gz 2024-01-11T09:48:34.722 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105941.log 2024-01-11T09:48:34.723 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49665.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49665.log.gz 2024-01-11T09:48:34.724 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111508.log 2024-01-11T09:48:34.725 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.105941.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.105941.log.gz 2024-01-11T09:48:34.725 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77950.log 2024-01-11T09:48:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.111508.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.111508.log.gz 2024-01-11T09:48:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95609.log 2024-01-11T09:48:34.728 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77950.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.77950.log.gz 2024-01-11T09:48:34.728 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85674.log 2024-01-11T09:48:34.730 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.95609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101290.log 2024-01-11T09:48:34.730 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95609.log.gz 2024-01-11T09:48:34.731 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85674.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85674.log.gz 2024-01-11T09:48:34.731 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115873.log 2024-01-11T09:48:34.733 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.101290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107932.log 2024-01-11T09:48:34.733 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.101290.log.gz 2024-01-11T09:48:34.734 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.115873.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.115873.log.gz 2024-01-11T09:48:34.734 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119536.log 2024-01-11T09:48:34.735 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.107932.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.107932.log.gz 2024-01-11T09:48:34.736 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109956.log 2024-01-11T09:48:34.737 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119536.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.119536.log.gz 2024-01-11T09:48:34.737 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110791.log 2024-01-11T09:48:34.739 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109956.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109956.log.gz 2024-01-11T09:48:34.739 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81208.log 2024-01-11T09:48:34.740 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.110791.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.110791.log.gz 2024-01-11T09:48:34.741 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95555.log 2024-01-11T09:48:34.742 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81208.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81208.log.gz 2024-01-11T09:48:34.742 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57349.log 2024-01-11T09:48:34.744 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.95555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53624.log 2024-01-11T09:48:34.744 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95555.log.gz 2024-01-11T09:48:34.745 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57349.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57349.log.gz 2024-01-11T09:48:34.745 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72308.log 2024-01-11T09:48:34.747 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53624.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53624.log.gz 2024-01-11T09:48:34.747 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110209.log 2024-01-11T09:48:34.748 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72308.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.72308.log.gz 2024-01-11T09:48:34.749 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79299.log 2024-01-11T09:48:34.750 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.110209.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.110209.log.gz 2024-01-11T09:48:34.750 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83860.log 2024-01-11T09:48:34.752 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79299.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.79299.log.gz 2024-01-11T09:48:34.752 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76411.log 2024-01-11T09:48:34.753 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83860.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.83860.log.gz 2024-01-11T09:48:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53306.log 2024-01-11T09:48:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76411.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.76411.log.gz 2024-01-11T09:48:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116524.log 2024-01-11T09:48:34.757 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53306.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53306.log.gz 2024-01-11T09:48:34.757 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101490.log 2024-01-11T09:48:34.758 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.116524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104755.log 2024-01-11T09:48:34.759 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.116524.log.gz 2024-01-11T09:48:34.760 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.101490.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.101490.log.gz 2024-01-11T09:48:34.760 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112345.log 2024-01-11T09:48:34.761 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.104755.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.104755.log.gz 2024-01-11T09:48:34.762 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.82876.log 2024-01-11T09:48:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.112345.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.112345.log.gz 2024-01-11T09:48:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105312.log 2024-01-11T09:48:34.764 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.82876.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.82876.log.gz 2024-01-11T09:48:34.765 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73501.log 2024-01-11T09:48:34.766 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.105312.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.105312.log.gz 2024-01-11T09:48:34.767 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117659.log 2024-01-11T09:48:34.768 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73501.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73501.log.gz 2024-01-11T09:48:34.768 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75018.log 2024-01-11T09:48:34.769 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.117659.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.117659.log.gz 2024-01-11T09:48:34.770 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98003.log 2024-01-11T09:48:34.771 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75018.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75018.log.gz 2024-01-11T09:48:34.772 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100632.log 2024-01-11T09:48:34.773 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.98003.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.98003.log.gz 2024-01-11T09:48:34.773 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64293.log 2024-01-11T09:48:34.774 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.100632.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100632.log.gz 2024-01-11T09:48:34.775 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114921.log 2024-01-11T09:48:34.776 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64293.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64293.log.gz 2024-01-11T09:48:34.777 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82539.log 2024-01-11T09:48:34.778 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.114921.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114921.log.gz 2024-01-11T09:48:34.778 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100928.log 2024-01-11T09:48:34.780 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108627.log 2024-01-11T09:48:34.780 INFO:teuthology.orchestra.run.smithi027.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.82539.log.gz 2024-01-11T09:48:34.781 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.100928.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.100928.log.gz 2024-01-11T09:48:34.782 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96583.log 2024-01-11T09:48:34.783 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.108627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93870.log 2024-01-11T09:48:34.783 INFO:teuthology.orchestra.run.smithi027.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.108627.log.gz 2024-01-11T09:48:34.784 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.96583.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.96583.log.gz 2024-01-11T09:48:34.785 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116020.log 2024-01-11T09:48:34.786 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.93870.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.93870.log.gz 2024-01-11T09:48:34.786 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47603.log 2024-01-11T09:48:34.787 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.116020.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.116020.log.gz 2024-01-11T09:48:34.788 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52971.log 2024-01-11T09:48:34.789 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47603.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47603.log.gz 2024-01-11T09:48:34.790 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76208.log 2024-01-11T09:48:34.791 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52971.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52971.log.gz 2024-01-11T09:48:34.791 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101647.log 2024-01-11T09:48:34.793 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76208.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.76208.log.gz 2024-01-11T09:48:34.793 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94910.log 2024-01-11T09:48:34.794 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.101647.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.101647.log.gz 2024-01-11T09:48:34.795 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47950.log 2024-01-11T09:48:34.796 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.94910.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.94910.log.gz 2024-01-11T09:48:34.796 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76672.log 2024-01-11T09:48:34.797 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47950.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47950.log.gz 2024-01-11T09:48:34.798 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108152.log 2024-01-11T09:48:34.799 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76672.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76672.log.gz 2024-01-11T09:48:34.799 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114137.log 2024-01-11T09:48:34.801 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.108152.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108152.log.gz 2024-01-11T09:48:34.801 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69421.log 2024-01-11T09:48:34.802 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.114137.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.114137.log.gz 2024-01-11T09:48:34.803 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77588.log 2024-01-11T09:48:34.805 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75314.log 2024-01-11T09:48:34.805 INFO:teuthology.orchestra.run.smithi027.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.69421.log.gz 2024-01-11T09:48:34.806 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77588.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.77588.log.gz 2024-01-11T09:48:34.806 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67820.log 2024-01-11T09:48:34.808 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75314.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75314.log.gz 2024-01-11T09:48:34.808 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67601.log 2024-01-11T09:48:34.809 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67820.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67820.log.gz 2024-01-11T09:48:34.810 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83764.log 2024-01-11T09:48:34.811 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67601.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67601.log.gz 2024-01-11T09:48:34.811 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69077.log 2024-01-11T09:48:34.812 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83764.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.83764.log.gz 2024-01-11T09:48:34.813 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48342.log 2024-01-11T09:48:34.814 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69077.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.69077.log.gz 2024-01-11T09:48:34.814 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77899.log 2024-01-11T09:48:34.816 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48342.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48342.log.gz 2024-01-11T09:48:34.816 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109162.log 2024-01-11T09:48:34.817 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77899.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77899.log.gz 2024-01-11T09:48:34.818 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69371.log 2024-01-11T09:48:34.819 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109162.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.109162.log.gz 2024-01-11T09:48:34.819 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119683.log 2024-01-11T09:48:34.820 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69371.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69371.log.gz 2024-01-11T09:48:34.821 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66987.log 2024-01-11T09:48:34.822 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119683.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.119683.log.gz 2024-01-11T09:48:34.823 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119781.log 2024-01-11T09:48:34.824 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.66987.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66987.log.gz 2024-01-11T09:48:34.824 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110669.log 2024-01-11T09:48:34.826 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119781.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.119781.log.gz 2024-01-11T09:48:34.826 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61915.log 2024-01-11T09:48:34.827 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.110669.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.110669.log.gz 2024-01-11T09:48:34.828 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48615.log 2024-01-11T09:48:34.829 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61915.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61915.log.gz 2024-01-11T09:48:34.829 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70715.log 2024-01-11T09:48:34.831 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48615.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48615.log.gz 2024-01-11T09:48:34.831 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106545.log 2024-01-11T09:48:34.832 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70715.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70715.log.gz 2024-01-11T09:48:34.833 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94812.log 2024-01-11T09:48:34.834 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.106545.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.106545.log.gz 2024-01-11T09:48:34.834 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95188.log 2024-01-11T09:48:34.836 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.94812.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.94812.log.gz 2024-01-11T09:48:34.836 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94606.log 2024-01-11T09:48:34.837 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.95188.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95188.log.gz 2024-01-11T09:48:34.838 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84561.log 2024-01-11T09:48:34.839 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.94606.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.94606.log.gz 2024-01-11T09:48:34.839 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112443.log 2024-01-11T09:48:34.840 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84561.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84561.log.gz 2024-01-11T09:48:34.841 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51958.log 2024-01-11T09:48:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.112443.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.112443.log.gz 2024-01-11T09:48:34.843 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101941.log 2024-01-11T09:48:34.844 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51958.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51958.log.gz 2024-01-11T09:48:34.844 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53673.log 2024-01-11T09:48:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.101941.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.101941.log.gz 2024-01-11T09:48:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68246.log 2024-01-11T09:48:34.847 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53673.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53673.log.gz 2024-01-11T09:48:34.848 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96534.log 2024-01-11T09:48:34.849 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68246.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68246.log.gz 2024-01-11T09:48:34.849 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70769.log 2024-01-11T09:48:34.850 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.96534.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96534.log.gz 2024-01-11T09:48:34.851 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112996.log 2024-01-11T09:48:34.852 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70769.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70769.log.gz 2024-01-11T09:48:34.852 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117920.log 2024-01-11T09:48:34.854 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.112996.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.112996.log.gz 2024-01-11T09:48:34.854 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118438.log 2024-01-11T09:48:34.855 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.117920.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.117920.log.gz 2024-01-11T09:48:34.856 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76781.log 2024-01-11T09:48:34.857 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.118438.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.118438.log.gz 2024-01-11T09:48:34.857 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114823.log 2024-01-11T09:48:34.859 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70399.log 2024-01-11T09:48:34.859 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76781.log.gz 2024-01-11T09:48:34.860 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.114823.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.114823.log.gz 2024-01-11T09:48:34.861 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73446.log 2024-01-11T09:48:34.862 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70399.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70399.log.gz 2024-01-11T09:48:34.862 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109709.log 2024-01-11T09:48:34.863 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73446.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.73446.log.gz 2024-01-11T09:48:34.864 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62124.log 2024-01-11T09:48:34.865 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109709.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.109709.log.gz 2024-01-11T09:48:34.865 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94017.log 2024-01-11T09:48:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62124.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62124.log.gz 2024-01-11T09:48:34.867 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52105.log 2024-01-11T09:48:34.868 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.94017.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.94017.log.gz 2024-01-11T09:48:34.868 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.72800.log 2024-01-11T09:48:34.870 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52105.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52105.log.gz 2024-01-11T09:48:34.870 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116328.log 2024-01-11T09:48:34.871 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.72800.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.72800.log.gz 2024-01-11T09:48:34.872 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110424.log 2024-01-11T09:48:34.876 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.116328.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.116328.log.gz 2024-01-11T09:48:34.876 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85010.log 2024-01-11T09:48:34.876 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.110424.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.110424.log.gz 2024-01-11T09:48:34.876 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62536.log 2024-01-11T09:48:34.876 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85010.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.85010.log.gz 2024-01-11T09:48:34.877 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54446.log 2024-01-11T09:48:34.878 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62536.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62536.log.gz 2024-01-11T09:48:34.878 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61967.log 2024-01-11T09:48:34.879 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54446.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54446.log.gz 2024-01-11T09:48:34.880 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62956.log 2024-01-11T09:48:34.881 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61967.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61967.log.gz 2024-01-11T09:48:34.882 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48146.log 2024-01-11T09:48:34.883 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62956.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62956.log.gz 2024-01-11T09:48:34.883 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108103.log 2024-01-11T09:48:34.884 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48146.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48146.log.gz 2024-01-11T09:48:34.885 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79167.log 2024-01-11T09:48:34.886 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.108103.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.108103.log.gz 2024-01-11T09:48:34.886 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83452.log 2024-01-11T09:48:34.887 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79167.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79167.log.gz 2024-01-11T09:48:34.888 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78455.log 2024-01-11T09:48:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83452.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83452.log.gz 2024-01-11T09:48:34.890 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85823.log 2024-01-11T09:48:34.891 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78455.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.78455.log.gz 2024-01-11T09:48:34.892 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83308.log 2024-01-11T09:48:34.893 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85823.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85823.log.gz 2024-01-11T09:48:34.894 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62487.log 2024-01-11T09:48:34.894 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83308.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.83308.log.gz 2024-01-11T09:48:34.895 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83668.log 2024-01-11T09:48:34.896 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62487.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62487.log.gz 2024-01-11T09:48:34.896 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52154.log 2024-01-11T09:48:34.897 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83668.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.83668.log.gz 2024-01-11T09:48:34.898 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92205.log 2024-01-11T09:48:34.899 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52154.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52154.log.gz 2024-01-11T09:48:34.899 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118948.log 2024-01-11T09:48:34.901 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92205.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.92205.log.gz 2024-01-11T09:48:34.901 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54079.log 2024-01-11T09:48:34.902 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.118948.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.118948.log.gz 2024-01-11T09:48:34.903 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.117053.log 2024-01-11T09:48:34.904 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54079.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54079.log.gz 2024-01-11T09:48:34.904 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100346.log 2024-01-11T09:48:34.905 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.117053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.117053.log.gz 2024-01-11T09:48:34.906 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72856.log 2024-01-11T09:48:34.907 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.100346.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.100346.log.gz 2024-01-11T09:48:34.907 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68806.log 2024-01-11T09:48:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72856.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.72856.log.gz 2024-01-11T09:48:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114774.log 2024-01-11T09:48:34.910 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68806.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.68806.log.gz 2024-01-11T09:48:34.911 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110103.log 2024-01-11T09:48:34.912 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.114774.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.114774.log.gz 2024-01-11T09:48:34.912 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93391.log 2024-01-11T09:48:34.913 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.110103.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.110103.log.gz 2024-01-11T09:48:34.914 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109477.log 2024-01-11T09:48:34.916 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.93391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48195.log 2024-01-11T09:48:34.916 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.93391.log.gz 2024-01-11T09:48:34.916 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109477.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.109477.log.gz 2024-01-11T09:48:34.917 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47183.log 2024-01-11T09:48:34.918 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82245.log 2024-01-11T09:48:34.919 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48195.log.gz 2024-01-11T09:48:34.919 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47183.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47183.log.gz 2024-01-11T09:48:34.920 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51517.log 2024-01-11T09:48:34.921 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81990.log 2024-01-11T09:48:34.921 INFO:teuthology.orchestra.run.smithi027.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82245.log.gz 2024-01-11T09:48:34.922 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51517.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51517.log.gz 2024-01-11T09:48:34.923 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107530.log 2024-01-11T09:48:34.924 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117113.log 2024-01-11T09:48:34.925 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81990.log.gz 2024-01-11T09:48:34.925 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.107530.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.107530.log.gz 2024-01-11T09:48:34.926 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56108.log 2024-01-11T09:48:34.927 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.117113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71354.log 2024-01-11T09:48:34.927 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.117113.log.gz 2024-01-11T09:48:34.928 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56108.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56108.log.gz 2024-01-11T09:48:34.929 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97847.log 2024-01-11T09:48:34.930 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.102400.log 2024-01-11T09:48:34.930 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71354.log.gz 2024-01-11T09:48:34.931 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.97847.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97847.log.gz 2024-01-11T09:48:34.932 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83380.log 2024-01-11T09:48:34.933 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.102400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.102400.log.gz 2024-01-11T09:48:34.933 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112947.log 2024-01-11T09:48:34.934 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83380.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.83380.log.gz 2024-01-11T09:48:34.935 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115192.log 2024-01-11T09:48:34.936 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.112947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110867.log 2024-01-11T09:48:34.936 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.112947.log.gz 2024-01-11T09:48:34.937 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.115192.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.115192.log.gz 2024-01-11T09:48:34.938 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81306.log 2024-01-11T09:48:34.939 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.110867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54947.log 2024-01-11T09:48:34.939 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.110867.log.gz 2024-01-11T09:48:34.940 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81306.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81306.log.gz 2024-01-11T09:48:34.941 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77052.log 2024-01-11T09:48:34.941 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54947.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54947.log.gz 2024-01-11T09:48:34.942 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96889.log 2024-01-11T09:48:34.943 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101892.log 2024-01-11T09:48:34.944 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.77052.log.gz 2024-01-11T09:48:34.945 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.96889.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.96889.log.gz 2024-01-11T09:48:34.945 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47553.log 2024-01-11T09:48:34.946 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.101892.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.101892.log.gz 2024-01-11T09:48:34.946 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74661.log 2024-01-11T09:48:34.948 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97725.log 2024-01-11T09:48:34.948 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47553.log.gz 2024-01-11T09:48:34.949 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74661.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.74661.log.gz 2024-01-11T09:48:34.950 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81159.log 2024-01-11T09:48:34.951 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.97725.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.97725.log.gz 2024-01-11T09:48:34.951 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117162.log 2024-01-11T09:48:34.952 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90528.log 2024-01-11T09:48:34.952 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81159.log.gz 2024-01-11T09:48:34.954 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.117162.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.117162.log.gz 2024-01-11T09:48:34.954 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92833.log 2024-01-11T09:48:34.955 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.90528.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90528.log.gz 2024-01-11T09:48:34.956 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116069.log 2024-01-11T09:48:34.957 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92833.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.92833.log.gz 2024-01-11T09:48:34.957 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51713.log 2024-01-11T09:48:34.958 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.116069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110054.log 2024-01-11T09:48:34.959 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.116069.log.gz 2024-01-11T09:48:34.960 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51713.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51713.log.gz 2024-01-11T09:48:34.960 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74912.log 2024-01-11T09:48:34.961 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.110054.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.110054.log.gz 2024-01-11T09:48:34.962 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73394.log 2024-01-11T09:48:34.963 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74912.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74912.log.gz 2024-01-11T09:48:34.963 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64342.log 2024-01-11T09:48:34.965 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73394.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73394.log.gz 2024-01-11T09:48:34.965 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75903.log 2024-01-11T09:48:34.966 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72112.log 2024-01-11T09:48:34.967 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64342.log.gz 2024-01-11T09:48:34.968 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75903.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.75903.log.gz 2024-01-11T09:48:34.968 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64548.log 2024-01-11T09:48:34.969 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72112.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72112.log.gz 2024-01-11T09:48:34.970 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47416.log 2024-01-11T09:48:34.971 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64548.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.64548.log.gz 2024-01-11T09:48:34.972 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86413.log 2024-01-11T09:48:34.973 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94557.log 2024-01-11T09:48:34.973 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47416.log.gz 2024-01-11T09:48:34.975 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67128.log 2024-01-11T09:48:34.975 INFO:teuthology.orchestra.run.smithi027.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.86413.log.gz 2024-01-11T09:48:34.976 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.94557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68197.log 2024-01-11T09:48:34.976 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.94557.log.gz 2024-01-11T09:48:34.977 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67128.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.67128.log.gz 2024-01-11T09:48:34.978 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74761.log 2024-01-11T09:48:34.979 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68197.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68197.log.gz 2024-01-11T09:48:34.979 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86266.log 2024-01-11T09:48:34.981 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96221.log 2024-01-11T09:48:34.981 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74761.log.gz 2024-01-11T09:48:34.982 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86266.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86266.log.gz 2024-01-11T09:48:34.982 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57997.log 2024-01-11T09:48:34.983 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.96221.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.96221.log.gz 2024-01-11T09:48:34.984 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80876.log 2024-01-11T09:48:34.985 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57997.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57997.log.gz 2024-01-11T09:48:34.985 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55044.log 2024-01-11T09:48:34.987 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80876.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.80876.log.gz 2024-01-11T09:48:34.987 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119095.log 2024-01-11T09:48:34.988 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55044.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55044.log.gz 2024-01-11T09:48:34.989 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107134.log 2024-01-11T09:48:34.990 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119095.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.119095.log.gz 2024-01-11T09:48:34.990 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77664.log 2024-01-11T09:48:34.991 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.107134.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.107134.log.gz 2024-01-11T09:48:34.992 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78602.log 2024-01-11T09:48:34.993 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77664.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77664.log.gz 2024-01-11T09:48:34.993 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98101.log 2024-01-11T09:48:34.995 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109582.log 2024-01-11T09:48:34.995 INFO:teuthology.orchestra.run.smithi027.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78602.log.gz 2024-01-11T09:48:34.996 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.98101.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.98101.log.gz 2024-01-11T09:48:34.997 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117368.log 2024-01-11T09:48:34.997 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109582.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.109582.log.gz 2024-01-11T09:48:34.998 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111706.log 2024-01-11T09:48:34.999 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.117368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49959.log 2024-01-11T09:48:34.999 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.117368.log.gz 2024-01-11T09:48:35.000 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.111706.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.111706.log.gz 2024-01-11T09:48:35.001 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68295.log 2024-01-11T09:48:35.003 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49959.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.49959.log.gz 2024-01-11T09:48:35.003 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81939.log 2024-01-11T09:48:35.004 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68295.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68295.log.gz 2024-01-11T09:48:35.004 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.94501.log 2024-01-11T09:48:35.005 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81939.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.81939.log.gz 2024-01-11T09:48:35.006 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55669.log 2024-01-11T09:48:35.007 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.94501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.94501.log.gz 2024-01-11T09:48:35.007 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119487.log 2024-01-11T09:48:35.009 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55669.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55669.log.gz 2024-01-11T09:48:35.009 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117294.log 2024-01-11T09:48:35.010 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58118.log 2024-01-11T09:48:35.011 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.119487.log.gz 2024-01-11T09:48:35.012 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.117294.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.117294.log.gz 2024-01-11T09:48:35.012 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110522.log 2024-01-11T09:48:35.013 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58118.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58118.log.gz 2024-01-11T09:48:35.013 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113045.log 2024-01-11T09:48:35.015 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.110522.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.110522.log.gz 2024-01-11T09:48:35.015 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57565.log 2024-01-11T09:48:35.016 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.113045.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.113045.log.gz 2024-01-11T09:48:35.017 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74104.log 2024-01-11T09:48:35.018 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57565.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57565.log.gz 2024-01-11T09:48:35.018 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102637.log 2024-01-11T09:48:35.020 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74104.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74104.log.gz 2024-01-11T09:48:35.020 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108676.log 2024-01-11T09:48:35.021 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.102637.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102637.log.gz 2024-01-11T09:48:35.022 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105018.log 2024-01-11T09:48:35.023 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.108676.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.108676.log.gz 2024-01-11T09:48:35.023 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104808.log 2024-01-11T09:48:35.025 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.105018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107785.log 2024-01-11T09:48:35.025 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.105018.log.gz 2024-01-11T09:48:35.026 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.104808.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.104808.log.gz 2024-01-11T09:48:35.027 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92414.log 2024-01-11T09:48:35.027 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.107785.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.107785.log.gz 2024-01-11T09:48:35.028 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81110.log 2024-01-11T09:48:35.029 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92414.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.92414.log.gz 2024-01-11T09:48:35.030 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.79057.log 2024-01-11T09:48:35.031 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81110.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.81110.log.gz 2024-01-11T09:48:35.031 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73658.log 2024-01-11T09:48:35.032 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.79057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.79057.log.gz 2024-01-11T09:48:35.033 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119634.log 2024-01-11T09:48:35.034 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73658.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73658.log.gz 2024-01-11T09:48:35.034 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107330.log 2024-01-11T09:48:35.035 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119634.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.119634.log.gz 2024-01-11T09:48:35.036 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53428.log 2024-01-11T09:48:35.037 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.107330.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.107330.log.gz 2024-01-11T09:48:35.037 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79118.log 2024-01-11T09:48:35.038 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53428.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53428.log.gz 2024-01-11T09:48:35.039 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74153.log 2024-01-11T09:48:35.040 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79118.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79118.log.gz 2024-01-11T09:48:35.041 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81061.log 2024-01-11T09:48:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74153.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.74153.log.gz 2024-01-11T09:48:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87085.log 2024-01-11T09:48:35.043 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81061.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-01-11T09:48:35.044 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.81061.log.gz 2024-01-11T09:48:35.044 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.87085.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87085.log.gz 2024-01-11T09:48:35.045 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-11T09:48:35.046 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69936.log 2024-01-11T09:48:35.047 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96632.log 2024-01-11T09:48:35.049 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69936.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69936.log.gz 2024-01-11T09:48:35.049 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81355.log 2024-01-11T09:48:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.96632.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96632.log.gz 2024-01-11T09:48:35.051 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82147.log 2024-01-11T09:48:35.053 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81355.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81355.log.gz 2024-01-11T09:48:35.053 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86486.log 2024-01-11T09:48:35.054 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82147.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82147.log.gz 2024-01-11T09:48:35.056 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86486.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86486.log.gz 2024-01-11T09:48:35.056 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77539.log 2024-01-11T09:48:35.059 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86364.log 2024-01-11T09:48:35.060 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77539.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.77539.log.gz 2024-01-11T09:48:35.060 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74612.log 2024-01-11T09:48:35.061 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.86364.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86364.log.gz 2024-01-11T09:48:35.061 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79240.log 2024-01-11T09:48:35.062 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74612.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.74612.log.gz 2024-01-11T09:48:35.063 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104320.log 2024-01-11T09:48:35.065 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79240.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79240.log.gz 2024-01-11T09:48:35.065 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54324.log 2024-01-11T09:48:35.066 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.104320.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.104320.log.gz 2024-01-11T09:48:35.066 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.34941.log 2024-01-11T09:48:35.068 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54324.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54324.log.gz 2024-01-11T09:48:35.068 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75167.log 2024-01-11T09:48:35.069 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph.tmp-client.admin.34941.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.34941.log.gz 2024-01-11T09:48:35.070 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49714.log 2024-01-11T09:48:35.072 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75167.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75167.log.gz 2024-01-11T09:48:35.072 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115487.log 2024-01-11T09:48:35.073 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49714.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49714.log.gz 2024-01-11T09:48:35.075 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.115487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111608.log 2024-01-11T09:48:35.075 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.115487.log.gz 2024-01-11T09:48:35.080 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116622.log 2024-01-11T09:48:35.080 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.111608.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.111608.log.gz 2024-01-11T09:48:35.080 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.116622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118597.log 2024-01-11T09:48:35.080 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.116622.log.gz 2024-01-11T09:48:35.080 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111055.log 2024-01-11T09:48:35.082 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.118597.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.118597.log.gz 2024-01-11T09:48:35.083 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.111055.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.111055.log.gz 2024-01-11T09:48:35.083 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.51211.log 2024-01-11T09:48:35.087 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105410.log 2024-01-11T09:48:35.087 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.2.51211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.51211.log.gz 2024-01-11T09:48:35.088 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.105410.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.105410.log.gz 2024-01-11T09:48:35.088 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93105.log 2024-01-11T09:48:35.092 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50128.log 2024-01-11T09:48:35.092 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.93105.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.93105.log.gz 2024-01-11T09:48:35.093 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50128.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50128.log.gz 2024-01-11T09:48:35.093 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100136.log 2024-01-11T09:48:35.097 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69494.log 2024-01-11T09:48:35.097 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.100136.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.100136.log.gz 2024-01-11T09:48:35.098 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69494.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69494.log.gz 2024-01-11T09:48:35.098 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73088.log 2024-01-11T09:48:35.102 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48293.log 2024-01-11T09:48:35.102 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73088.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73088.log.gz 2024-01-11T09:48:35.103 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48293.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48293.log.gz 2024-01-11T09:48:35.103 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116120.log 2024-01-11T09:48:35.107 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102858.log 2024-01-11T09:48:35.107 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.116120.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.116120.log.gz 2024-01-11T09:48:35.107 INFO:teuthology.orchestra.run.smithi077.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-01-11T09:48:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.102858.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.102858.log.gz 2024-01-11T09:48:35.108 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55165.log 2024-01-11T09:48:35.112 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112541.log 2024-01-11T09:48:35.112 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55165.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55165.log.gz 2024-01-11T09:48:35.113 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.112541.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.112541.log.gz 2024-01-11T09:48:35.113 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109760.log 2024-01-11T09:48:35.117 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55597.log 2024-01-11T09:48:35.117 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109760.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.109760.log.gz 2024-01-11T09:48:35.118 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55597.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55597.log.gz 2024-01-11T09:48:35.118 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53020.log 2024-01-11T09:48:35.122 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116171.log 2024-01-11T09:48:35.122 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53020.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53020.log.gz 2024-01-11T09:48:35.123 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.116171.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.116171.log.gz 2024-01-11T09:48:35.123 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55839.log 2024-01-11T09:48:35.127 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80394.log 2024-01-11T09:48:35.127 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55839.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55839.log.gz 2024-01-11T09:48:35.127 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58814.log 2024-01-11T09:48:35.130 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80394.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.80394.log.gz 2024-01-11T09:48:35.130 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-01-11T09:48:35.130 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58814.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58814.log.gz 2024-01-11T09:48:35.132 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69863.log 2024-01-11T09:48:35.137 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53881.log 2024-01-11T09:48:35.137 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118184.log 2024-01-11T09:48:35.137 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69863.log.gz 2024-01-11T09:48:35.139 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63343.log 2024-01-11T09:48:35.139 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53881.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53881.log.gz 2024-01-11T09:48:35.139 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.118184.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.118184.log.gz 2024-01-11T09:48:35.141 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93821.log 2024-01-11T09:48:35.141 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63343.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63343.log.gz 2024-01-11T09:48:35.142 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116377.log 2024-01-11T09:48:35.143 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.93821.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.93821.log.gz 2024-01-11T09:48:35.144 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71036.log 2024-01-11T09:48:35.145 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.116377.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.116377.log.gz 2024-01-11T09:48:35.146 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73893.log 2024-01-11T09:48:35.148 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71036.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.71036.log.gz 2024-01-11T09:48:35.149 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72978.log 2024-01-11T09:48:35.149 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73893.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.73893.log.gz 2024-01-11T09:48:35.150 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58069.log 2024-01-11T09:48:35.153 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72978.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72978.log.gz 2024-01-11T09:48:35.153 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52383.log 2024-01-11T09:48:35.153 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58069.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58069.log.gz 2024-01-11T09:48:35.154 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87183.log 2024-01-11T09:48:35.156 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52383.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52383.log.gz 2024-01-11T09:48:35.157 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.87183.log: -- /var/log/ceph/ceph-client.0.69754.log 2024-01-11T09:48:35.157 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.87183.log.gz 2024-01-11T09:48:35.158 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81692.log 2024-01-11T09:48:35.160 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.69754.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.69754.log.gz 2024-01-11T09:48:35.162 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105892.log 2024-01-11T09:48:35.162 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.81692.log.gz 2024-01-11T09:48:35.163 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94066.log 2024-01-11T09:48:35.165 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.105892.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.105892.log.gz 2024-01-11T09:48:35.166 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.94066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92882.log 2024-01-11T09:48:35.166 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.94066.log.gz 2024-01-11T09:48:35.167 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49013.log 2024-01-11T09:48:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92882.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.92882.log.gz 2024-01-11T09:48:35.170 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49013.log: /var/log/ceph/ceph-client.admin.57637.log 2024-01-11T09:48:35.170 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49013.log.gz 2024-01-11T09:48:35.172 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68755.log 2024-01-11T09:48:35.173 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57637.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57637.log.gz 2024-01-11T09:48:35.174 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56059.log 2024-01-11T09:48:35.176 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68755.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68755.log.gz 2024-01-11T09:48:35.177 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84510.log 2024-01-11T09:48:35.177 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56059.log.gz 2024-01-11T09:48:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54030.log 2024-01-11T09:48:35.180 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84510.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84510.log.gz 2024-01-11T09:48:35.181 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63449.log 2024-01-11T09:48:35.181 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54030.log.gz 2024-01-11T09:48:35.182 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48622.log 2024-01-11T09:48:35.184 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63449.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63449.log.gz 2024-01-11T09:48:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68557.log 2024-01-11T09:48:35.186 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48622.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48622.log.gz 2024-01-11T09:48:35.187 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114578.log 2024-01-11T09:48:35.188 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68557.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68557.log.gz 2024-01-11T09:48:35.190 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.114578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80816.log 2024-01-11T09:48:35.190 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.114578.log.gz 2024-01-11T09:48:35.192 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78675.log 2024-01-11T09:48:35.193 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80816.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80816.log.gz 2024-01-11T09:48:35.193 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115661.log 2024-01-11T09:48:35.195 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78675.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.78675.log.gz 2024-01-11T09:48:35.196 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.115661.log: -- /var/log/ceph/ceph-client.admin.58466.log 2024-01-11T09:48:35.196 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.115661.log.gz 2024-01-11T09:48:35.198 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77286.log 2024-01-11T09:48:35.199 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58466.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58466.log.gz 2024-01-11T09:48:35.201 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74300.log 2024-01-11T09:48:35.201 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.77286.log.gz 2024-01-11T09:48:35.202 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113544.log 2024-01-11T09:48:35.204 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74300.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.74300.log.gz 2024-01-11T09:48:35.205 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.113544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82932.log 2024-01-11T09:48:35.205 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.113544.log.gz 2024-01-11T09:48:35.206 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107636.log 2024-01-11T09:48:35.208 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82932.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.82932.log.gz 2024-01-11T09:48:35.209 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.107636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71865.log 2024-01-11T09:48:35.209 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.107636.log.gz 2024-01-11T09:48:35.211 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100451.log 2024-01-11T09:48:35.212 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71865.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71865.log.gz 2024-01-11T09:48:35.214 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.100451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83524.log 2024-01-11T09:48:35.214 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100451.log.gz 2024-01-11T09:48:35.215 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71617.log 2024-01-11T09:48:35.217 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83524.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.83524.log.gz 2024-01-11T09:48:35.218 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93292.log 2024-01-11T09:48:35.218 INFO:teuthology.orchestra.run.smithi027.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.71617.log.gz 2024-01-11T09:48:35.220 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113925.log 2024-01-11T09:48:35.221 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.93292.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.93292.log.gz 2024-01-11T09:48:35.222 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106447.log 2024-01-11T09:48:35.225 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.113925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93489.log 2024-01-11T09:48:35.225 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.113925.log.gz 2024-01-11T09:48:35.226 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.106447.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.106447.log.gz 2024-01-11T09:48:35.227 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54815.log 2024-01-11T09:48:35.228 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.93489.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.93489.log.gz 2024-01-11T09:48:35.229 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48610.log 2024-01-11T09:48:35.231 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54815.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54815.log.gz 2024-01-11T09:48:35.232 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109529.log 2024-01-11T09:48:35.233 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48610.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48610.log.gz 2024-01-11T09:48:35.234 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107834.log 2024-01-11T09:48:35.235 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109529.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.109529.log.gz 2024-01-11T09:48:35.236 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93341.log 2024-01-11T09:48:35.237 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.107834.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.107834.log.gz 2024-01-11T09:48:35.238 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70874.log 2024-01-11T09:48:35.239 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.93341.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.93341.log.gz 2024-01-11T09:48:35.240 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68371.log 2024-01-11T09:48:35.241 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70874.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70874.log.gz 2024-01-11T09:48:35.243 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100830.log 2024-01-11T09:48:35.245 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68371.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.68371.log.gz 2024-01-11T09:48:35.245 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52628.log 2024-01-11T09:48:35.246 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.100830.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100830.log.gz 2024-01-11T09:48:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57925.log 2024-01-11T09:48:35.248 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52628.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52628.log.gz 2024-01-11T09:48:35.249 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-11T09:48:35.250 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57925.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57925.log.gz 2024-01-11T09:48:35.251 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57104.log 2024-01-11T09:48:35.255 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.57104.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57104.log.gz 2024-01-11T09:48:35.255 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106249.log 2024-01-11T09:48:35.255 INFO:teuthology.orchestra.run.smithi027.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-01-11T09:48:35.258 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.106249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80498.log 2024-01-11T09:48:35.259 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.106249.log.gz 2024-01-11T09:48:35.261 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112798.log 2024-01-11T09:48:35.262 INFO:teuthology.orchestra.run.smithi027.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.80498.log.gz 2024-01-11T09:48:35.263 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62354.log 2024-01-11T09:48:35.264 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-11T09:48:35.265 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.112798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57781.log 2024-01-11T09:48:35.266 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112798.log.gz 2024-01-11T09:48:35.267 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62354.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62354.log.gz 2024-01-11T09:48:35.267 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84774.log 2024-01-11T09:48:35.269 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57781.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.57781.log.gz 2024-01-11T09:48:35.270 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63606.log 2024-01-11T09:48:35.270 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84774.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.84774.log.gz 2024-01-11T09:48:35.273 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49005.log 2024-01-11T09:48:35.273 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63606.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63606.log.gz 2024-01-11T09:48:35.274 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49005.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49005.log.gz 2024-01-11T09:48:35.275 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68979.log 2024-01-11T09:48:35.278 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93772.log 2024-01-11T09:48:35.278 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68979.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68979.log.gz 2024-01-11T09:48:35.279 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.93772.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.93772.log.gz 2024-01-11T09:48:35.280 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53722.log 2024-01-11T09:48:35.283 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105508.log 2024-01-11T09:48:35.283 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53722.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53722.log.gz 2024-01-11T09:48:35.284 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.105508.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.105508.log.gz 2024-01-11T09:48:35.285 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74006.log 2024-01-11T09:48:35.286 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100083.log 2024-01-11T09:48:35.288 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.74006.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74006.log.gz 2024-01-11T09:48:35.289 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.100083.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.100083.log.gz 2024-01-11T09:48:35.290 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95845.log 2024-01-11T09:48:35.292 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67337.log 2024-01-11T09:48:35.293 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.95845.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.95845.log.gz 2024-01-11T09:48:35.295 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67337.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67337.log.gz 2024-01-11T09:48:35.295 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85206.log 2024-01-11T09:48:35.298 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54275.log 2024-01-11T09:48:35.298 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85206.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.85206.log.gz 2024-01-11T09:48:35.300 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71183.log 2024-01-11T09:48:35.300 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54275.log.gz 2024-01-11T09:48:35.301 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75534.log 2024-01-11T09:48:35.303 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71183.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.71183.log.gz 2024-01-11T09:48:35.304 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75534.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75534.log.gz 2024-01-11T09:48:35.305 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99924.log 2024-01-11T09:48:35.306 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63704.log 2024-01-11T09:48:35.308 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.99924.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.99924.log.gz 2024-01-11T09:48:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63704.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63704.log.gz 2024-01-11T09:48:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93591.log 2024-01-11T09:48:35.311 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84961.log 2024-01-11T09:48:35.313 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.93591.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.93591.log.gz 2024-01-11T09:48:35.315 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.84961.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.84961.log.gz 2024-01-11T09:48:35.315 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52056.log 2024-01-11T09:48:35.316 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55309.log 2024-01-11T09:48:35.318 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52056.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52056.log.gz 2024-01-11T09:48:35.320 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55309.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55309.log.gz 2024-01-11T09:48:35.320 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48391.log 2024-01-11T09:48:35.323 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106398.log 2024-01-11T09:48:35.323 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48391.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48391.log.gz 2024-01-11T09:48:35.325 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.106398.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.106398.log.gz 2024-01-11T09:48:35.325 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62732.log 2024-01-11T09:48:35.328 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104596.log 2024-01-11T09:48:35.328 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62732.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62732.log.gz 2024-01-11T09:48:35.330 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.104596.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.104596.log.gz 2024-01-11T09:48:35.330 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.69758.log 2024-01-11T09:48:35.333 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111251.log 2024-01-11T09:48:35.333 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.69758.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.69758.log.gz 2024-01-11T09:48:35.333 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92627.log 2024-01-11T09:48:35.335 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.111251.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.111251.log.gz 2024-01-11T09:48:35.335 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112296.log 2024-01-11T09:48:35.337 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92627.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.92627.log.gz 2024-01-11T09:48:35.338 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.112296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80657.log 2024-01-11T09:48:35.338 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.112296.log.gz 2024-01-11T09:48:35.339 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107428.log 2024-01-11T09:48:35.341 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80657.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80657.log.gz 2024-01-11T09:48:35.342 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.107428.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.107428.log.gz 2024-01-11T09:48:35.343 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77433.log 2024-01-11T09:48:35.344 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62231.log 2024-01-11T09:48:35.346 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77433.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77433.log.gz 2024-01-11T09:48:35.347 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62231.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62231.log.gz 2024-01-11T09:48:35.348 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107281.log 2024-01-11T09:48:35.349 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55381.log 2024-01-11T09:48:35.351 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.107281.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107281.log.gz 2024-01-11T09:48:35.352 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55381.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55381.log.gz 2024-01-11T09:48:35.353 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95239.log 2024-01-11T09:48:35.354 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64237.log 2024-01-11T09:48:35.356 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.95239.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.95239.log.gz 2024-01-11T09:48:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.64237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.64237.log.gz 2024-01-11T09:48:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62071.log 2024-01-11T09:48:35.359 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70559.log 2024-01-11T09:48:35.361 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62071.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62071.log.gz 2024-01-11T09:48:35.362 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70559.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70559.log.gz 2024-01-11T09:48:35.362 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64233.log 2024-01-11T09:48:35.364 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54888.log 2024-01-11T09:48:35.366 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.64233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.64233.log.gz 2024-01-11T09:48:35.367 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54888.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54888.log.gz 2024-01-11T09:48:35.368 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59510.log 2024-01-11T09:48:35.369 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92735.log 2024-01-11T09:48:35.371 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59510.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59510.log.gz 2024-01-11T09:48:35.372 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92735.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.92735.log.gz 2024-01-11T09:48:35.373 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51909.log 2024-01-11T09:48:35.374 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106594.log 2024-01-11T09:48:35.376 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51909.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51909.log.gz 2024-01-11T09:48:35.377 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.106594.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.106594.log.gz 2024-01-11T09:48:35.378 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54373.log 2024-01-11T09:48:35.379 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105606.log 2024-01-11T09:48:35.381 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54373.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.54373.log.gz 2024-01-11T09:48:35.382 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.105606.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.105606.log.gz 2024-01-11T09:48:35.383 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.72796.log 2024-01-11T09:48:35.384 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93204.log 2024-01-11T09:48:35.385 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.72796.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.72796.log.gz 2024-01-11T09:48:35.387 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.93204.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.93204.log.gz 2024-01-11T09:48:35.388 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107379.log 2024-01-11T09:48:35.389 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96172.log 2024-01-11T09:48:35.391 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.107379.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.107379.log.gz 2024-01-11T09:48:35.392 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.96172.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.96172.log.gz 2024-01-11T09:48:35.393 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111853.log 2024-01-11T09:48:35.394 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49224.log 2024-01-11T09:48:35.396 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.111853.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.111853.log.gz 2024-01-11T09:48:35.397 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49224.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49224.log.gz 2024-01-11T09:48:35.398 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96099.log 2024-01-11T09:48:35.399 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85487.log 2024-01-11T09:48:35.401 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.96099.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.96099.log.gz 2024-01-11T09:48:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85487.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85487.log.gz 2024-01-11T09:48:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95291.log 2024-01-11T09:48:35.404 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78306.log 2024-01-11T09:48:35.406 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.95291.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95291.log.gz 2024-01-11T09:48:35.408 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112492.log 2024-01-11T09:48:35.408 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78306.log.gz 2024-01-11T09:48:35.409 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82096.log 2024-01-11T09:48:35.411 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.112492.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.112492.log.gz 2024-01-11T09:48:35.412 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82096.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.82096.log.gz 2024-01-11T09:48:35.413 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80929.log 2024-01-11T09:48:35.414 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71916.log 2024-01-11T09:48:35.416 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80929.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.80929.log.gz 2024-01-11T09:48:35.417 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71916.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71916.log.gz 2024-01-11T09:48:35.418 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56933.log 2024-01-11T09:48:35.419 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81741.log 2024-01-11T09:48:35.421 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56933.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56933.log.gz 2024-01-11T09:48:35.422 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81741.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81741.log.gz 2024-01-11T09:48:35.423 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94115.log 2024-01-11T09:48:35.424 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81839.log 2024-01-11T09:48:35.426 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.94115.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.94115.log.gz 2024-01-11T09:48:35.427 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81839.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81839.log.gz 2024-01-11T09:48:35.428 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79471.log 2024-01-11T09:48:35.429 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69175.log 2024-01-11T09:48:35.431 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.79471.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.79471.log.gz 2024-01-11T09:48:35.432 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69175.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69175.log.gz 2024-01-11T09:48:35.433 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70927.log 2024-01-11T09:48:35.434 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72063.log 2024-01-11T09:48:35.436 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70927.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70927.log.gz 2024-01-11T09:48:35.437 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114725.log 2024-01-11T09:48:35.438 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72063.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72063.log.gz 2024-01-11T09:48:35.438 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117515.log 2024-01-11T09:48:35.440 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.114725.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.114725.log.gz 2024-01-11T09:48:35.442 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.117515.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.117515.log.gz 2024-01-11T09:48:35.442 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76728.log 2024-01-11T09:48:35.444 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106090.log 2024-01-11T09:48:35.445 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76728.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76728.log.gz 2024-01-11T09:48:35.447 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.106090.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.106090.log.gz 2024-01-11T09:48:35.447 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48048.log 2024-01-11T09:48:35.449 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119144.log 2024-01-11T09:48:35.450 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48048.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48048.log.gz 2024-01-11T09:48:35.452 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119144.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.119144.log.gz 2024-01-11T09:48:35.452 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63655.log 2024-01-11T09:48:35.454 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72161.log 2024-01-11T09:48:35.455 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63655.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63655.log.gz 2024-01-11T09:48:35.457 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72161.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72161.log.gz 2024-01-11T09:48:35.457 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77000.log 2024-01-11T09:48:35.459 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81888.log 2024-01-11T09:48:35.460 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77000.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.77000.log.gz 2024-01-11T09:48:35.462 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81888.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81888.log.gz 2024-01-11T09:48:35.462 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115389.log 2024-01-11T09:48:35.464 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71666.log 2024-01-11T09:48:35.465 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.115389.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.115389.log.gz 2024-01-11T09:48:35.467 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71666.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.71666.log.gz 2024-01-11T09:48:35.467 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62781.log 2024-01-11T09:48:35.469 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95895.log 2024-01-11T09:48:35.470 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62781.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62781.log.gz 2024-01-11T09:48:35.472 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.95895.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.95895.log.gz 2024-01-11T09:48:35.472 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113314.log 2024-01-11T09:48:35.474 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70506.log 2024-01-11T09:48:35.476 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.113314.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.113314.log.gz 2024-01-11T09:48:35.477 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70506.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70506.log.gz 2024-01-11T09:48:35.477 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59162.log 2024-01-11T09:48:35.479 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100187.log 2024-01-11T09:48:35.481 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59162.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59162.log.gz 2024-01-11T09:48:35.481 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111559.log 2024-01-11T09:48:35.483 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.100187.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.100187.log.gz 2024-01-11T09:48:35.483 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63241.log 2024-01-11T09:48:35.484 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.111559.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.111559.log.gz 2024-01-11T09:48:35.486 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63241.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63241.log.gz 2024-01-11T09:48:35.486 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80762.log 2024-01-11T09:48:35.488 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109809.log 2024-01-11T09:48:35.490 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.80762.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80762.log.gz 2024-01-11T09:48:35.490 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76620.log 2024-01-11T09:48:35.491 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.109809.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.109809.log.gz 2024-01-11T09:48:35.492 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115971.log 2024-01-11T09:48:35.493 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76620.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76620.log.gz 2024-01-11T09:48:35.494 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78797.log 2024-01-11T09:48:35.495 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.115971.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.115971.log.gz 2024-01-11T09:48:35.495 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61755.log 2024-01-11T09:48:35.497 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78797.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.78797.log.gz 2024-01-11T09:48:35.497 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70987.log 2024-01-11T09:48:35.498 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61755.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61755.log.gz 2024-01-11T09:48:35.499 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69322.log 2024-01-11T09:48:35.500 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70987.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70987.log.gz 2024-01-11T09:48:35.501 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95661.log 2024-01-11T09:48:35.501 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.69322.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.69322.log.gz 2024-01-11T09:48:35.502 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76362.log 2024-01-11T09:48:35.504 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.95661.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.95661.log.gz 2024-01-11T09:48:35.504 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97185.log 2024-01-11T09:48:35.505 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76362.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.76362.log.gz 2024-01-11T09:48:35.505 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113094.log 2024-01-11T09:48:35.507 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.97185.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.97185.log.gz 2024-01-11T09:48:35.508 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63498.log 2024-01-11T09:48:35.508 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.113094.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.113094.log.gz 2024-01-11T09:48:35.509 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107479.log 2024-01-11T09:48:35.510 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63498.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63498.log.gz 2024-01-11T09:48:35.511 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63549.log 2024-01-11T09:48:35.513 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.107479.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.107479.log.gz 2024-01-11T09:48:35.513 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106741.log 2024-01-11T09:48:35.514 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63549.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63549.log.gz 2024-01-11T09:48:35.514 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111202.log 2024-01-11T09:48:35.516 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.106741.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.106741.log.gz 2024-01-11T09:48:35.517 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111657.log 2024-01-11T09:48:35.517 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.111202.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.111202.log.gz 2024-01-11T09:48:35.518 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63900.log 2024-01-11T09:48:35.519 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.111657.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.111657.log.gz 2024-01-11T09:48:35.520 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49012.log 2024-01-11T09:48:35.522 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106814.log 2024-01-11T09:48:35.522 INFO:teuthology.orchestra.run.smithi027.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.63900.log.gz 2024-01-11T09:48:35.523 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49012.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49012.log.gz 2024-01-11T09:48:35.523 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85304.log 2024-01-11T09:48:35.524 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.106814.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.106814.log.gz 2024-01-11T09:48:35.526 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83596.log 2024-01-11T09:48:35.526 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.85304.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85304.log.gz 2024-01-11T09:48:35.527 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61807.log 2024-01-11T09:48:35.529 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.83596.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.83596.log.gz 2024-01-11T09:48:35.529 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53575.log 2024-01-11T09:48:35.530 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61807.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61807.log.gz 2024-01-11T09:48:35.532 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53575.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53575.log.gz 2024-01-11T09:48:35.532 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97405.log 2024-01-11T09:48:35.534 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73842.log 2024-01-11T09:48:35.535 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.97405.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97405.log.gz 2024-01-11T09:48:35.537 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73842.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73842.log.gz 2024-01-11T09:48:35.537 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52677.log 2024-01-11T09:48:35.539 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115760.log 2024-01-11T09:48:35.541 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52677.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52677.log.gz 2024-01-11T09:48:35.542 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.115760.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.115760.log.gz 2024-01-11T09:48:35.542 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94679.log 2024-01-11T09:48:35.544 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57202.log 2024-01-11T09:48:35.546 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.94679.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.94679.log.gz 2024-01-11T09:48:35.547 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57202.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57202.log.gz 2024-01-11T09:48:35.548 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.97665.log 2024-01-11T09:48:35.549 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101388.log 2024-01-11T09:48:35.550 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.97665.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.97665.log.gz 2024-01-11T09:48:35.552 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.101388.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.101388.log.gz 2024-01-11T09:48:35.553 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77763.log 2024-01-11T09:48:35.554 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51468.log 2024-01-11T09:48:35.556 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.77763.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77763.log.gz 2024-01-11T09:48:35.557 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51468.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51468.log.gz 2024-01-11T09:48:35.558 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49910.log 2024-01-11T09:48:35.560 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55888.log 2024-01-11T09:48:35.561 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49910.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49910.log.gz 2024-01-11T09:48:35.561 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64719.log 2024-01-11T09:48:35.562 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55888.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55888.log.gz 2024-01-11T09:48:35.564 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63143.log 2024-01-11T09:48:35.564 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.64719.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64719.log.gz 2024-01-11T09:48:35.565 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47852.log 2024-01-11T09:48:35.566 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.63143.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63143.log.gz 2024-01-11T09:48:35.567 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71965.log 2024-01-11T09:48:35.568 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47852.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47852.log.gz 2024-01-11T09:48:35.568 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81001.log 2024-01-11T09:48:35.570 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.71965.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.71965.log.gz 2024-01-11T09:48:35.570 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52203.log 2024-01-11T09:48:35.571 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81001.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81001.log.gz 2024-01-11T09:48:35.573 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52203.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52203.log.gz 2024-01-11T09:48:35.574 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68706.log 2024-01-11T09:48:35.575 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70821.log 2024-01-11T09:48:35.577 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68706.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68706.log.gz 2024-01-11T09:48:35.579 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70821.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70821.log.gz 2024-01-11T09:48:35.580 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93672.log 2024-01-11T09:48:35.580 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116573.log 2024-01-11T09:48:35.582 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.93672.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.93672.log.gz 2024-01-11T09:48:35.584 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.116573.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.116573.log.gz 2024-01-11T09:48:35.584 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57251.log 2024-01-11T09:48:35.585 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101794.log 2024-01-11T09:48:35.587 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57251.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.57251.log.gz 2024-01-11T09:48:35.589 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.101794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49518.log 2024-01-11T09:48:35.589 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.101794.log.gz 2024-01-11T09:48:35.590 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70610.log 2024-01-11T09:48:35.592 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49518.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49518.log.gz 2024-01-11T09:48:35.593 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70610.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70610.log.gz 2024-01-11T09:48:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47504.log 2024-01-11T09:48:35.596 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96732.log 2024-01-11T09:48:35.597 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47504.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47504.log.gz 2024-01-11T09:48:35.598 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.96732.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96732.log.gz 2024-01-11T09:48:35.599 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111459.log 2024-01-11T09:48:35.601 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41957.log 2024-01-11T09:48:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.111459.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.111459.log.gz 2024-01-11T09:48:35.604 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41957.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41957.log.gz 2024-01-11T09:48:35.604 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100879.log 2024-01-11T09:48:35.606 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111976.log 2024-01-11T09:48:35.607 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.100879.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.100879.log.gz 2024-01-11T09:48:35.608 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.111976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61250.log 2024-01-11T09:48:35.609 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.111976.log.gz 2024-01-11T09:48:35.610 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100583.log 2024-01-11T09:48:35.612 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61250.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61250.log.gz 2024-01-11T09:48:35.613 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.100583.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100583.log.gz 2024-01-11T09:48:35.613 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76885.log 2024-01-11T09:48:35.615 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101990.log 2024-01-11T09:48:35.616 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76885.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.76885.log.gz 2024-01-11T09:48:35.618 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.101990.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.101990.log.gz 2024-01-11T09:48:35.618 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70663.log 2024-01-11T09:48:35.620 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53119.log 2024-01-11T09:48:35.621 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.70663.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.70663.log.gz 2024-01-11T09:48:35.623 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53119.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.53119.log.gz 2024-01-11T09:48:35.623 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67653.log 2024-01-11T09:48:35.625 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96001.log 2024-01-11T09:48:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.67653.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67653.log.gz 2024-01-11T09:48:35.628 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.96001.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.96001.log.gz 2024-01-11T09:48:35.628 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96783.log 2024-01-11T09:48:35.630 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53355.log 2024-01-11T09:48:35.632 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.96783.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.96783.log.gz 2024-01-11T09:48:35.633 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53355.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53355.log.gz 2024-01-11T09:48:35.633 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93721.log 2024-01-11T09:48:35.635 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-11T09:48:35.637 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.93721.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.93721.log.gz 2024-01-11T09:48:35.639 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53526.log 2024-01-11T09:48:35.640 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52481.log 2024-01-11T09:48:35.644 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111755.log 2024-01-11T09:48:35.645 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53526.log.gz 2024-01-11T09:48:35.645 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47652.log 2024-01-11T09:48:35.645 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.111755.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52481.log.gz 2024-01-11T09:48:35.645 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.111755.log.gz 2024-01-11T09:48:35.647 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47652.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47652.log.gz 2024-01-11T09:48:35.648 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95713.log 2024-01-11T09:48:35.649 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98846.log 2024-01-11T09:48:35.651 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.95713.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.95713.log.gz 2024-01-11T09:48:35.652 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.98846.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.98846.log.gz 2024-01-11T09:48:35.653 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92573.log 2024-01-11T09:48:35.654 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78048.log 2024-01-11T09:48:35.656 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92573.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.92573.log.gz 2024-01-11T09:48:35.657 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76831.log 2024-01-11T09:48:35.658 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.78048.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.78048.log.gz 2024-01-11T09:48:35.658 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55937.log 2024-01-11T09:48:35.660 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.76831.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76831.log.gz 2024-01-11T09:48:35.661 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55937.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55937.log.gz 2024-01-11T09:48:35.662 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114527.log 2024-01-11T09:48:35.663 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82612.log 2024-01-11T09:48:35.665 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.114527.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.114527.log.gz 2024-01-11T09:48:35.667 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82612.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.82612.log.gz 2024-01-11T09:48:35.667 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68148.log 2024-01-11T09:48:35.669 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51860.log 2024-01-11T09:48:35.670 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.68148.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68148.log.gz 2024-01-11T09:48:35.672 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51860.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51860.log.gz 2024-01-11T09:48:35.672 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95944.log 2024-01-11T09:48:35.675 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.95944.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.95944.log.gz 2024-01-11T09:48:35.676 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72357.log 2024-01-11T09:48:35.677 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108774.log 2024-01-11T09:48:35.679 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72357.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72357.log.gz 2024-01-11T09:48:35.680 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.108774.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.108774.log.gz 2024-01-11T09:48:35.681 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.97669.log 2024-01-11T09:48:35.682 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82490.log 2024-01-11T09:48:35.683 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.97669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.97669.log.gz 2024-01-11T09:48:35.685 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82294.log 2024-01-11T09:48:35.686 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82490.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82490.log.gz 2024-01-11T09:48:35.686 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104964.log 2024-01-11T09:48:35.688 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82294.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82294.log.gz 2024-01-11T09:48:35.689 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.104964.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.104964.log.gz 2024-01-11T09:48:35.690 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115242.log 2024-01-11T09:48:35.691 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51566.log 2024-01-11T09:48:35.693 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.115242.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.115242.log.gz 2024-01-11T09:48:35.695 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51566.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51566.log.gz 2024-01-11T09:48:35.695 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92098.log 2024-01-11T09:48:35.696 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118499.log 2024-01-11T09:48:35.698 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.92098.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.92098.log.gz 2024-01-11T09:48:35.699 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53255.log 2024-01-11T09:48:35.700 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.118499.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.118499.log.gz 2024-01-11T09:48:35.701 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75854.log 2024-01-11T09:48:35.702 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53255.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53255.log.gz 2024-01-11T09:48:35.704 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75854.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.75854.log.gz 2024-01-11T09:48:35.704 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55453.log 2024-01-11T09:48:35.706 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75461.log 2024-01-11T09:48:35.707 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55453.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55453.log.gz 2024-01-11T09:48:35.710 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.75461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72455.log 2024-01-11T09:48:35.710 INFO:teuthology.orchestra.run.smithi027.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.75461.log.gz 2024-01-11T09:48:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49567.log 2024-01-11T09:48:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.72455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81404.log 2024-01-11T09:48:35.714 INFO:teuthology.orchestra.run.smithi027.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.72455.log.gz 2024-01-11T09:48:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.81404.log: /var/log/ceph/ceph-client.admin.49567.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81404.log.gz 2024-01-11T09:48:35.717 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108358.log 2024-01-11T09:48:35.717 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49567.log.gz 2024-01-11T09:48:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.108358.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.108358.log.gz 2024-01-11T09:48:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52530.log 2024-01-11T09:48:35.723 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52530.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52530.log.gz 2024-01-11T09:48:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102711.log 2024-01-11T09:48:35.727 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.102711.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.102711.log.gz 2024-01-11T09:48:35.727 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93540.log 2024-01-11T09:48:35.730 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.93540.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.93540.log.gz 2024-01-11T09:48:35.731 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57055.log 2024-01-11T09:48:35.733 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57055.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57055.log.gz 2024-01-11T09:48:35.734 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47801.log 2024-01-11T09:48:35.735 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94959.log 2024-01-11T09:48:35.737 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92520.log 2024-01-11T09:48:35.737 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47801.log.gz 2024-01-11T09:48:35.740 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.94959.log: /var/log/ceph/ceph-client.admin.92520.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.94959.log.gz 2024-01-11T09:48:35.740 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.92520.log.gz 2024-01-11T09:48:35.740 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82196.log 2024-01-11T09:48:35.744 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.82196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.102396.log 2024-01-11T09:48:35.744 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82196.log.gz 2024-01-11T09:48:35.745 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101004.log 2024-01-11T09:48:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.102396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115585.log 2024-01-11T09:48:35.747 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.102396.log.gz 2024-01-11T09:48:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.101004.log: /var/log/ceph/ceph-client.admin.115585.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.101004.log.gz 2024-01-11T09:48:35.750 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.115585.log.gz 2024-01-11T09:48:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73947.log 2024-01-11T09:48:35.754 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.73947.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.73947.log.gz 2024-01-11T09:48:35.754 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114970.log 2024-01-11T09:48:35.757 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.114970.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.114970.log.gz 2024-01-11T09:48:35.758 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119438.log 2024-01-11T09:48:35.761 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.119438.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.119438.log.gz 2024-01-11T09:48:35.761 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113192.log 2024-01-11T09:48:35.764 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95785.log 2024-01-11T09:48:35.765 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.113192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110374.log 2024-01-11T09:48:35.766 INFO:teuthology.orchestra.run.smithi027.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.113192.log.gz 2024-01-11T09:48:35.767 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.95785.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95785.log.gz 2024-01-11T09:48:35.769 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.110374.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.110374.log.gz 2024-01-11T09:48:35.834 INFO:teuthology.orchestra.run.smithi077.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-01-11T09:48:36.195 INFO:teuthology.orchestra.run.smithi077.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-11T09:48:36.416 INFO:teuthology.orchestra.run.smithi027.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-01-11T09:48:36.549 INFO:teuthology.orchestra.run.smithi077.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-01-11T09:48:37.054 INFO:teuthology.orchestra.run.smithi027.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-11T09:48:37.343 INFO:teuthology.orchestra.run.smithi077.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-01-11T09:48:37.431 INFO:teuthology.orchestra.run.smithi077.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-01-11T09:48:37.838 INFO:teuthology.orchestra.run.smithi077.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-01-11T09:48:37.853 INFO:teuthology.orchestra.run.smithi027.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-01-11T09:48:37.892 INFO:teuthology.orchestra.run.smithi077.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-01-11T09:48:37.895 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-01-11T09:48:37.895 INFO:teuthology.orchestra.run.smithi077.stderr:real 0m3.774s 2024-01-11T09:48:37.895 INFO:teuthology.orchestra.run.smithi077.stderr:user 0m19.360s 2024-01-11T09:48:37.895 INFO:teuthology.orchestra.run.smithi077.stderr:sys 0m1.614s 2024-01-11T09:48:38.413 INFO:teuthology.orchestra.run.smithi027.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-11T09:48:38.433 INFO:teuthology.orchestra.run.smithi027.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-11T09:48:38.897 INFO:teuthology.orchestra.run.smithi027.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-11T09:48:38.900 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-01-11T09:48:38.900 INFO:teuthology.orchestra.run.smithi027.stderr:real 0m4.784s 2024-01-11T09:48:38.900 INFO:teuthology.orchestra.run.smithi027.stderr:user 0m17.085s 2024-01-11T09:48:38.900 INFO:teuthology.orchestra.run.smithi027.stderr:sys 0m3.957s 2024-01-11T09:48:38.901 INFO:tasks.ceph:Archiving logs... 2024-01-11T09:48:38.902 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512563/remote/smithi027/log 2024-01-11T09:48:38.903 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-11T09:48:41.906 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512563/remote/smithi077/log 2024-01-11T09:48:41.908 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-11T09:48:44.309 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-11T09:48:44.318 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-11T09:48:44.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-11T09:48:44.324 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-11T09:48:44.445 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-11T09:48:44.456 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-11T09:48:44.456 DEBUG:teuthology.orchestra.run.smithi027:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-11T09:48:44.462 DEBUG:teuthology.orchestra.run.smithi077:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-11T09:48:44.504 INFO:teuthology.orchestra.run.smithi027.stderr:bash: ntpq: command not found 2024-01-11T09:48:44.506 INFO:teuthology.orchestra.run.smithi077.stderr:bash: ntpq: command not found 2024-01-11T09:48:44.512 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-11T09:48:44.512 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-01-11T09:48:44.512 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv01.front.sepia.ceph.com 2 8 377 115 +730us[ +739us] +/- 73ms 2024-01-11T09:48:44.512 INFO:teuthology.orchestra.run.smithi027.stdout:^* hv02.front.sepia.ceph.com 3 7 377 55 -478us[ -469us] +/- 72ms 2024-01-11T09:48:44.512 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 47 -343us[ -343us] +/- 79ms 2024-01-11T09:48:44.512 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T09:48:44.513 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-11T09:48:44.513 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-01-11T09:48:44.513 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 +782us[ +782us] +/- 74ms 2024-01-11T09:48:44.513 INFO:teuthology.orchestra.run.smithi077.stdout:^* hv02.front.sepia.ceph.com 3 7 377 57 -396us[ -389us] +/- 72ms 2024-01-11T09:48:44.513 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv03.front.sepia.ceph.com 3 9 377 182 -229us[ -222us] +/- 77ms 2024-01-11T09:48:44.513 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T09:48:44.514 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-11T09:48:44.524 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-11T09:48:44.524 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-11T09:48:44.545 DEBUG:teuthology.orchestra.run.smithi027:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-11T09:48:44.727 DEBUG:teuthology.orchestra.run.smithi077:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-11T09:48:44.863 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-11T09:48:44.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T09:48:44.943 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-11T09:48:45.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T09:48:45.018 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-11T09:48:45.028 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-11T09:48:45.053 INFO:teuthology.task.internal:Duration was 6595.251711 seconds 2024-01-11T09:48:45.054 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-11T09:48:45.096 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-11T09:48:45.096 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-11T09:48:45.101 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-11T09:48:45.218 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-11T09:48:45.218 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-11T09:48:45.690 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-11T09:48:45.690 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi027.front.sepia.ceph.com 2024-01-11T09:48:45.690 DEBUG:teuthology.orchestra.run.smithi027:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-11T09:48:45.755 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi077.front.sepia.ceph.com 2024-01-11T09:48:45.755 DEBUG:teuthology.orchestra.run.smithi077:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-11T09:48:45.817 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-11T09:48:45.817 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-11T09:48:45.821 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-11T09:48:45.914 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-11T09:48:45.915 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-11T09:48:45.919 DEBUG:teuthology.orchestra.run.smithi077:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-11T09:48:46.065 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-11T09:48:46.075 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-11T09:48:46.075 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-11T09:48:46.147 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-11T09:48:46.217 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-11T09:48:46.228 DEBUG:teuthology.orchestra.run.smithi027:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-11T09:48:46.233 DEBUG:teuthology.orchestra.run.smithi077:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-11T09:48:46.300 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = core 2024-01-11T09:48:46.301 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = core 2024-01-11T09:48:46.336 DEBUG:teuthology.orchestra.run.smithi027:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-11T09:48:46.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T09:48:46.381 DEBUG:teuthology.orchestra.run.smithi077:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-11T09:48:46.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T09:48:46.425 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-11T09:48:46.436 INFO:teuthology.task.internal:Transferring archived files... 2024-01-11T09:48:46.437 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512563/remote/smithi027 2024-01-11T09:48:46.437 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-11T09:48:46.522 DEBUG:teuthology.misc:Transferring archived files from smithi077:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512563/remote/smithi077 2024-01-11T09:48:46.522 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-11T09:48:46.603 INFO:teuthology.task.internal:Removing archive directory... 2024-01-11T09:48:46.603 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-11T09:48:46.608 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-11T09:48:46.653 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-11T09:48:46.665 INFO:teuthology.task.internal:Not uploading archives. 2024-01-11T09:48:46.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-11T09:48:46.687 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-11T09:48:46.687 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-11T09:48:46.691 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-11T09:48:46.734 INFO:teuthology.orchestra.run.smithi027.stdout: 524301 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 11 09:48 /home/ubuntu/cephtest 2024-01-11T09:48:46.738 INFO:teuthology.orchestra.run.smithi077.stdout: 524293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 11 09:48 /home/ubuntu/cephtest 2024-01-11T09:48:46.741 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-11T09:48:46.751 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-11T09:48:46.815 INFO:teuthology.nuke:Checking targets against current locks 2024-01-11T09:48:46.836 DEBUG:teuthology.nuke:shortname: smithi027 2024-01-11T09:48:46.836 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-11T09:48:46.859 DEBUG:teuthology.nuke:shortname: smithi077 2024-01-11T09:48:46.859 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-11T09:48:46.862 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512563', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-11 07:48:12.485847', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEunaG6cGUEWWP4tba6VzAu7nUPYDutCiMscx2EbA8khWok9fdqlJ8otWGFY17YR4vwhjOiDGke6+1VbYkIjus8='} 2024-01-11T09:48:46.885 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512563', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-11 07:48:12.487316', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNi4VBy2oxJo5MOLjqpm4HK6B6up8DRtTrBVz5TABoJt5pvN7mongi/6XvIYziSsfdzl9BonN62qXSr5Gn3hSHk='} 2024-01-11T09:48:46.911 INFO:teuthology.orchestra.console.smithi027:Power off 2024-01-11T09:48:46.911 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-11T09:48:46.933 DEBUG:teuthology.orchestra.console.smithi027:power off output: Chassis Power Control: Down/Off 2024-01-11T09:48:46.933 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T09:48:46.950 INFO:teuthology.orchestra.console.smithi077:Power off 2024-01-11T09:48:46.950 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-11T09:48:46.958 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-01-11T09:48:46.971 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off 2024-01-11T09:48:46.971 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T09:48:46.995 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-01-11T09:48:50.959 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T09:48:50.996 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T09:48:51.071 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-01-11T09:48:51.105 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-01-11T09:48:55.073 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T09:48:55.106 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T09:48:55.184 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-01-11T09:48:55.217 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2024-01-11T09:48:55.317 INFO:teuthology.orchestra.console.smithi077:Power off completed 2024-01-11T09:48:59.188 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T09:48:59.297 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off 2024-01-11T09:48:59.398 INFO:teuthology.orchestra.console.smithi027:Power off completed 2024-01-11T09:48:59.503 INFO:teuthology.run:Summary data: client.0-kernel-sha1: c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b client.1-kernel-sha1: c719ee96ba7847b7d8dd469c35de92fb2f0dfd6b description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} duration: 6595.251710653305 failure_reason: '"1704961464.7739727 mon.a (mon.0) 551 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2024-01-11T09:48:59.503 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-11T09:48:59.573 INFO:teuthology.run:FAIL